One-way ANOVA, Welch ANOVA or Kruskal-Wallis?

For comparing the means of three or more independent groups, use Welch’s ANOVA by default, and the classic one-way ANOVA only when equal variances are plausible by design. Use the Kruskal-Wallis test when a rank-based question is what you want, as with an ordinal outcome, or when a small sample is badly skewed and a comparison of means is not meaningful anyway.

When you have three or more independent groups and one numeric outcome, three tests compete for the job: the classic one-way ANOVA, Welch’s ANOVA and the Kruskal-Wallis test. The usual advice is “ANOVA if the data are normal, Kruskal-Wallis if not”. That framing hides the more useful distinction. The two ANOVAs compare group means and differ in what they assume about the spread. Kruskal-Wallis asks a different question, about ranks. Pick the test whose question you want answered, then check that its assumptions are reasonable.

One-way ANOVA, Welch’s ANOVA and Kruskal-Wallis ask different questions

All three assume the groups are independent: different animals, plots or participants in each group, one value per unit. If the same units were measured repeatedly, see repeated-measures ANOVA vs a mixed model instead. With exactly two groups the family collapses to the t-test versus Mann-Whitney choice, and the same logic applies.

One-way ANOVA

Null hypothesis: all the population means are equal. The test compares the variation between group means with the variation inside groups, and the “inside” part is a single pooled variance. It assumes the residuals are normally distributed and that every group has the same variance. When the variances differ and the group sizes differ too, that pooled estimate is wrong in a predictable direction. If the smallest group is the most variable one, the F-test rejects far too often. If the largest group is the most variable, it becomes conservative.

Welch’s ANOVA

Same null hypothesis, equal means. Welch’s version (Welch, 1951) estimates each group’s variance separately, weights the groups accordingly and adjusts the denominator degrees of freedom. When the variances really are equal, it loses very little power compared with the classic test. When they are not, it keeps the false-positive rate close to the nominal level. Delacre and colleagues (2019) argue for exactly this trade and recommend it as the default for one-way designs.

Testing for equal variances first, with Levene’s test say, and then choosing between the classic and Welch versions is not a good fix. The pre-test has low power in small samples, which is where the choice matters, and it adds its own error to the procedure. Using Welch from the start avoids the problem. Normality matters less than it seems for the same reason it does in the t-test: the test depends on the sampling distribution of the means, which approaches normality as groups grow.

Kruskal-Wallis test

The Kruskal-Wallis test (Kruskal and Wallis, 1952) pools all observations, ranks them, and asks whether the average rank differs between groups. Its exact null hypothesis is that all groups come from the same distribution, and it has power against alternatives where values in some group tend to be larger than values in the others. It does not assume normality.

It does not, however, test whether the medians are equal, and it does not escape the unequal-spread problem. It is a comparison of medians only under a shift model, where the groups have the same shape and spread and differ only in location. Outside that model, groups with identical medians can produce a significant result, and groups that differ in spread can produce one on their own. Fay and Proschan (2010) set out the different hypotheses the same rank-based decision rule can be read as testing. Describe the result as “values in some group tend to be larger”, not as “the medians differ”.

Worked example: enzyme activity across four soil sites

A field team measures enzyme activity (one value per soil core) at four sites. The sample sizes are unequal (20, 18, 15 and 8 cores), and the smallest group, site D, is by far the most variable. That is the case where pooling hurts most. The data are synthetic, generated in R 4.6.1.

Site	n	Mean	SD
A	20	50.5	3.6
B	18	54.2	4.2
C	15	51.1	6.9
D	8	63.5	11.0

Test	Statistic	df	p	Effect size
One-way ANOVA	F = 9.93	3, 57	0.00002	eta-squared 0.34
Welch’s ANOVA	F = 5.49	3, 21.7	0.0058	eta-squared 0.34
Kruskal-Wallis	H = 14.26	3	0.0026	rank epsilon-squared 0.24

All three reject, but not with the same confidence. The classic F-test reports a p-value about 250 times smaller than Welch’s. The reason is visible in the first table: the group with the largest spread is also the smallest, so the pooled variance is dominated by the three tighter groups and understates the noise in the comparison that matters. Levene’s test confirms the unequal spread (F = 4.10 on 3 and 57 degrees of freedom, p = 0.011), but you should not need that test to choose, since Welch’s version is the better default either way. The eta-squared is the same for both ANOVAs because it describes the fitted differences between means, not the test used to judge them: about a third of the variation in the outcome lies between sites.

Kruskal-Wallis lands in between at p = 0.0026. Its rank epsilon-squared of 0.24 is on a different scale from eta-squared, so the two should not be compared number against number. It answers a different question: are values at some site systematically larger than at the others?

After the omnibus test

A significant omnibus result says only that the groups are not all alike. To find out which differ, follow it with the matching pairwise procedure: Tukey’s HSD after the classic ANOVA, Games-Howell (Games and Howell, 1976) after Welch’s ANOVA, and Dunn’s test with a Holm adjustment after Kruskal-Wallis. The pairing matters. In this example, Tukey’s pooled comparisons call site D clearly different from A, B and C (adjusted p = 0.00002, 0.003 and 0.0001), while Games-Howell, which does not pool the variances, is far more cautious about the same pairs (0.049, 0.177 and 0.066). The procedures, the adjustments and a worked example are covered in multiple comparisons.

Which one to report

• You want to compare means across three or more groups: Welch’s ANOVA, with eta-squared and the group means with intervals. It is the safe default and costs almost nothing when variances turn out equal.
• Equal variances are plausible by design and group sizes are balanced: the classic one-way ANOVA is fine, and the two tests will agree closely.
• The outcome is ordinal, or you want to know which groups tend to score higher: Kruskal-Wallis, reported with an effect size and pairwise comparisons, and not described as a comparison of medians.
• Small sample, heavy skew: consider whether the mean is the right summary at all. A log transformation often makes a ratio of geometric means the natural quantity.

Avoid running all three and reporting whichever has the smallest p-value, and decide the test from the question before looking at results. If the groups are crossed with a second factor, use a two-way ANOVA rather than running one-way tests on each factor separately. For the wider decision tree, see how to choose a statistical test.

How Celeus handles this

When you compare three or more groups on a continuous outcome, Celeus grades Welch’s ANOVA as the recommended default, checks normality within each group and equal variances, and suggests Welch’s ANOVA or Kruskal-Wallis as a re-run when the corresponding check fails. It reports eta-squared for the ANOVAs and a rank effect size for Kruskal-Wallis. It then runs the matching pairwise comparisons automatically (Tukey, Games-Howell or Dunn), each pair with an effect size.