ANOVA post hoc tests: Tukey, Games-Howell, Dunn

A post hoc test tells you which groups differ after an omnibus test across three or more groups, and the right one is fixed by the omnibus test you chose: Tukey’s HSD after the classic one-way ANOVA, Games-Howell after Welch’s ANOVA, and Dunn’s test with a stated p-value adjustment after Kruskal-Wallis. Picking a post hoc procedure separately, or by whichever gives the most significant pairs, breaks the logic that makes its p-values trustworthy.

Each procedure is explained below and then run in R on R’s built-in chickwts data. Side by side, they show what definitions alone do not: on the same data, the choice changes which pairs come out significant.

What a post hoc test is for

An omnibus test such as the one-way ANOVA answers one question: are all the group means equal? A significant result says they are not, and nothing more. It does not say which groups differ, by how much, or in which direction.

With k groups there are k(k - 1)/2 pairs, so six groups give 15. Fifteen ordinary t-tests at the 0.05 level carry a far higher chance than 5% of at least one false positive when no pair truly differs. A post hoc procedure holds the family-wise error rate, the chance of one or more false positives across the whole set, at the stated level. The multiple comparisons guide covers the generic corrections; this article covers the three procedures built for the pairwise follow-up. Two groups need no post hoc test at all, since the one comparison is the test. And comparisons named in the plan before the data were seen, such as each treatment against a control, are planned contrasts, which need a smaller correction than a data-driven search of every pair.

Match the post hoc to the omnibus test

Tukey’s HSD, after the classic one-way ANOVA

Tukey’s honestly significant difference compares every pair of means using the pooled within-group variance from the ANOVA, and refers each difference to the studentized range distribution, the distribution of the largest standardised gap among k means. Its adjustment is single-step, and exact for all pairs when the groups are the same size and its assumptions hold. Its main output is a set of simultaneous confidence intervals: across all 15 together, the chance that every one covers its true difference is 95%.

With unequal group sizes, R’s TukeyHSD() uses the Tukey-Kramer form, which Hayter proved in 1984 to be conservative: its family-wise error rate stays at or below the stated level. The assumption that matters is the pooled variance. Like the classic ANOVA, Tukey’s procedure can mislead when the groups have different spreads, above all when the smallest group is the most variable.

Games-Howell, after Welch’s ANOVA

Games-Howell gives each pair its own standard error from the two groups’ own variances, with its own Welch-type degrees of freedom, and refers the result to the same studentized range distribution. It is the pairwise counterpart of Welch’s ANOVA (Welch, 1951), which also drops the equal-variance assumption. In Games and Howell’s 1976 Monte Carlo study, this separate-variances approach was the only one of three procedures that kept both the family-wise and the per-comparison error rates under control with unequal group sizes of six or more. With groups as small as three, they found its control doubtful.

Dunn’s test, after Kruskal-Wallis

Kruskal-Wallis (Kruskal and Wallis, 1952) ranks all observations together and asks whether the mean rank differs between groups. Dunn’s 1964 procedure compares the groups’ mean ranks from that same joint ranking, with a normal approximation and a correction for ties. Separate Mann-Whitney tests for each pair re-rank the data every time and no longer test what the omnibus test tested.

Dunn’s test has no built-in correction, so you choose one and say which. Holm’s 1979 step-down procedure controls the family-wise error rate for any combination of true and false hypotheses and never rejects fewer hypotheses than Bonferroni, so it is a sound default. Like Kruskal-Wallis, Dunn’s test compares ranks, not means or medians.

Dunnett’s test, when only a control matters

If the question is “which treatments differ from the control”, not “which pairs differ”, Dunnett’s 1955 procedure compares each group with the control only. With six groups that is 5 comparisons instead of 15, so the correction is lighter and power higher for those questions. Fix the control before the analysis; the test says nothing about treatment-versus-treatment pairs. It is not run here, because chickwts has no control feed.

How to choose: the design picks the omnibus, the omnibus picks the post hoc

The decision happens once, before the data are analysed:

1. Decide what you want to compare. Means, or a tendency for values in some groups to be larger? That separates the two ANOVAs from Kruskal-Wallis.
2. For means, use Welch’s ANOVA unless equal variances follow from the design, for the reasons the guide to one-way ANOVA, Welch’s ANOVA and Kruskal-Wallis gives. The post hoc test is then Games-Howell, or Tukey if you chose the classic ANOVA.
3. For ranks, use Kruskal-Wallis followed by Dunn’s test, and write the adjustment into the plan.

When the same subjects are measured under every condition, the groups are not independent and none of these three applies as written; use a repeated-measures analysis and adjust its paired follow-up comparisons, for example with Holm.

The variance and normality checks are not switches in this chain. Choosing Tukey or Games-Howell from a Levene result is the two-stage procedure that fails for the t-test, as the t-tests overview explains: the pre-test has little power in small samples, where unequal variances do the most damage, and it changes the error rate of what follows. Read the checks as diagnostics, and label any second procedure a sensitivity analysis.

Worked example: Welch’s ANOVA and Games-Howell on chickwts

R ships a dataset called chickwts: 71 newly hatched chicks, each given one of six feed supplements (casein, horsebean, linseed, meatmeal, soybean, sunflower), with weight in grams recorded once per chick at six weeks. Each chick is in one group only, so the groups are independent, and the sizes are unequal.

Seeded and run in R; Welch’s ANOVA was chosen before the run, and the engine attaches Games-Howell comparisons to it.

The omnibus test

Welch’s F was 19.66 on 5 and 29.95 degrees of freedom, p < 0.001, so the six feed means are not all equal. The effect size is eta-squared, 0.5417 (95% CI 0.3549 to 0.6513): feed accounts for about half the variation in weight. The engine computes eta-squared from the classic ANOVA sums of squares, and its interval rests on the classic F distribution, so read the interval as approximate when spreads differ.

The assumption checks raised nothing. Shapiro-Wilk did not detect a departure from normality in any of the six groups, and Levene’s test did not detect unequal variances (statistic 0.7493, p = 0.5896). Neither result shows that the data are normal or the variances equal: with roughly a dozen chicks per group these tests have little power. Every group exceeds the sizes at which Games and Howell found their procedure unreliable.

The pairwise comparisons, three ways

The same 71 chicks were run twice more through the engine, once as a classic one-way ANOVA (F = 15.36 on 5 and 65 degrees of freedom, p < 0.001), which brings Tukey’s HSD, and once as Kruskal-Wallis (H = 37.34 on 5 degrees of freedom, p < 0.001, epsilon-squared 0.5335), which brings Dunn’s test with the Holm adjustment. The extra columns are for teaching; a real analysis reports the one procedure the plan named.

Pair	Hedges’ g (unadjusted 95% CI)	Games-Howell adj. p	Tukey adj. p	Dunn (Holm) adj. p
casein vs horsebean	2.89 (1.67 to 4.08)	< 0.001	< 0.001	< 0.001
casein vs linseed	1.73 (0.79 to 2.64)	0.0031	< 0.001	0.010
casein vs meatmeal	0.70 (-0.13 to 1.50)	0.53	0.33	0.63
casein vs soybean	1.26 (0.43 to 2.08)	0.036	0.0084	0.099
casein vs sunflower	-0.09 (-0.86 to 0.68)	0.9999	0.9999	0.99
horsebean vs linseed	-1.21 (-2.08 to -0.31)	0.065	0.14	0.58
horsebean vs meatmeal	-2.07 (-3.11 to -1.00)	0.0012	< 0.001	0.0092
horsebean vs soybean	-1.72 (-2.64 to -0.77)	0.0019	0.0042	0.083
horsebean vs sunflower	-3.64 (-5.01 to -2.25)	< 0.001	< 0.001	< 0.001
linseed vs meatmeal	-0.96 (-1.79 to -0.11)	0.22	0.13	0.48
linseed vs soybean	-0.50 (-1.26 to 0.26)	0.77	0.79	0.99
linseed vs sunflower	-2.10 (-3.08 to -1.10)	< 0.001	< 0.001	0.0062
meatmeal vs soybean	0.50 (-0.28 to 1.27)	0.81	0.74	0.99
meatmeal vs sunflower	-0.88 (-1.70 to -0.04)	0.30	0.22	0.58
soybean vs sunflower	-1.54 (-2.39 to -0.67)	0.0051	0.0039	0.071

Hedges’ g is the difference between the two feeds’ means in units of their pooled standard deviation, with a small-sample correction; a positive value means the first-named feed was heavier. Its interval assumes equal variances within the pair and covers that one pair at 95%. It is not adjusted for the 15 comparisons.

Games-Howell finds 8 of the 15 pairs different at an adjusted p below 0.05. The engine summarises them as a compact letter display, where feeds sharing a letter were not found to differ: casein (a), horsebean (b), linseed (bc), meatmeal (ac), soybean (c), sunflower (a). Casein and sunflower sit at the top, horsebean at the bottom, the rest between.

A pair that does not reach adjusted significance is not evidence that the feeds are equal. Casein against sunflower has a Games-Howell adjusted p of 0.9999, yet the interval for g runs from -0.86 to 0.68, so moderate differences either way remain compatible with these data.

What changes with the procedure

Tukey and Games-Howell flag the same eight pairs. That fits data where no difference in spread was detected, but it is a property of these data, not of the procedures. With unequal spreads and unequal sizes they can disagree sharply, as the worked example in the one-way ANOVA guide shows.

Dunn’s test with Holm flags five. Three pairs that both mean-based procedures call different (casein vs soybean, horsebean vs soybean, soybean vs sunflower) have Holm-adjusted p-values between 0.071 and 0.099. The table cannot separate the two reasons: Dunn’s test asks a different question, about ranks, and it adjusts differently, with Holm’s step-down over 15 normal-approximation tests instead of a single-step studentized range. A procedure picked because it produced more significant pairs no longer has the error rate it advertises.

Tukey also reports what the other two do not: simultaneous intervals for each difference in grams. Sunflower chicks averaged 168.7 g more than horsebean chicks (95% simultaneous CI 99.75 to 237.7), and soybean chicks 77.15 g less than casein chicks (-140.5 to -13.79). For sunflower minus casein the estimate is 5.333 g, with an interval from -60.42 to 71.09: compatible with no difference, with sunflower chicks about 60 g lighter, and with them about 71 g heavier.

Adjusted p-values and unadjusted intervals can disagree

Take horsebean against linseed. The Games-Howell adjusted p is 0.065, above 0.05, yet the unadjusted 95% interval for Hedges’ g, -2.08 to -0.31, excludes zero. Linseed against meatmeal (adjusted p 0.22) and meatmeal against sunflower (0.30) show the same pattern.

This is not a contradiction. The adjusted p-value accounts for all 15 comparisons; the g interval covers one pair at 95% and nothing else. Tukey’s simultaneous interval for the same pair (linseed minus horsebean) runs from -10.41 to 127.5 g and includes zero, consistent with its adjusted p of 0.14.

Report the per-pair effect size for its size and direction, and read significance from the adjusted p-value or a simultaneous interval, never an unadjusted interval.

How to report the follow-up

Name the omnibus test, the post hoc procedure and its adjustment, and give the pairs with their estimates and intervals. For the worked example:

Six-week weight differed between feeds (Welch’s ANOVA, F(5, 29.95) = 19.66, p < 0.001, eta-squared = 0.54, 95% CI 0.35 to 0.65). Games-Howell comparisons found 8 of 15 pairs different at an adjusted p < 0.05; for example, horsebean-fed chicks were lighter than sunflower-fed chicks (adjusted p < 0.001, Hedges’ g = -3.64, 95% CI -5.01 to -2.25; intervals for g are not adjusted for multiplicity).

Put the full pairwise table in a supplement. For Dunn’s test, state the adjustment: “Dunn’s test” alone does not say which was used.

Common mistakes

• Mixing pairs, such as Tukey after Welch’s ANOVA, so the follow-up rests on different assumptions from the test it follows.
• Shopping among procedures for the most significant pairs.
• Running separate t-tests or Mann-Whitney tests without an adjustment.
• Calling a non-significant pair “equal”, or saying a test “proves” a difference. A non-significant result never lets you “accept the null”.
• Forgetting a second factor. With two crossed factors, which pairs to compare depends on the interaction; see the two-way ANOVA guide.

Rerunning the example

Below the article are the R script the engine generated for the Welch run, with its seed and bundled data path, and the output it printed. On the same package versions, it reproduces the Welch test, eta-squared with its interval, the Games-Howell p-values and the sensitivity figure; it does not rerun the Shapiro-Wilk and Levene checks or the letter display, nor print Hedges’ g, which effectsize::hedges_g() gives for each pair of feeds. PMCMRplus::gamesHowellTest() prints “P value adjustment method: none”, but its p-values are already adjusted: each one comes from the studentized range distribution for all six means (ptukey() with the pair’s Welch-type degrees of freedom), which carries the adjustment, so do not adjust them again.

The Tukey and Dunn columns come from two further engine runs on the same data and seed, whose scripts are not shown here: TukeyHSD() after the classic one-way ANOVA, and PMCMRplus::kwAllPairsDunnTest() with p.adjust.method = "holm" after kruskal.test(). The guide to reproducible statistical analysis explains what a reviewer needs to rerun a result like this.

The engine’s sensitivity figure describes the design, not this result: at the average group size, a balanced classic ANOVA reached 80% power at the 0.05 level only for effects of Cohen’s f = 0.4434 or larger. Pairwise comparisons have less power still, so non-significant pairs are better read as unresolved than absent.

# Results

- Test: Welch's ANOVA
- n: 71
- statistic: 19.66
- p-value: < 0.001
- effect size: eta-squared = 0.5417 (95% CI 0.3549 to 0.6513)
  (two-sided CI at the run's confidence level)
- seed: 20261021

## Assumption checks
- normality by group (Shapiro-Wilk): passed=TRUE
- variance homogeneity (Levene): passed=TRUE (no evidence of unequal variances)

## Post-hoc (Games-Howell; correction: single-step (Games-Howell))
- casein vs horsebean: adj p = 9.436e-06 | Hedges' g 2.891 [1.673, 4.076]
- casein vs linseed: adj p = 0.003102 | Hedges' g 1.726 [0.7879, 2.636]
- horsebean vs linseed: adj p = 0.06494 | Hedges' g -1.208 [-2.085, -0.3071]
- casein vs meatmeal: adj p = 0.5293 | Hedges' g 0.6957 [-0.1277, 1.504]
- horsebean vs meatmeal: adj p = 0.001237 | Hedges' g -2.072 [-3.106, -1.005]
- linseed vs meatmeal: adj p = 0.2209 | Hedges' g -0.9564 [-1.786, -0.1071]
- casein vs soybean: adj p = 0.03604 | Hedges' g 1.265 [0.4269, 2.081]
- horsebean vs soybean: adj p = 0.001901 | Hedges' g -1.72 [-2.639, -0.7742]
- linseed vs soybean: adj p = 0.7689 | Hedges' g -0.5032 [-1.258, 0.2618]
- meatmeal vs soybean: adj p = 0.806 | Hedges' g 0.4991 [-0.2831, 1.271]
- casein vs sunflower: adj p = 0.9999 | Hedges' g -0.09007 [-0.862, 0.6839]
- horsebean vs sunflower: adj p = 2.307e-07 | Hedges' g -3.644 [-5.01, -2.247]
- linseed vs sunflower: adj p = 0.0003042 | Hedges' g -2.103 [-3.078, -1.1]
- meatmeal vs sunflower: adj p = 0.303 | Hedges' g -0.8786 [-1.701, -0.03763]
- soybean vs sunflower: adj p = 0.005088 | Hedges' g -1.543 [-2.395, -0.6672]

groups: casein (a), horsebean (b), linseed (bc), meatmeal (ac), soybean (c), sunflower (a) - groups sharing a letter are not significantly different at the adjusted threshold.

## Power (sensitivity, not observed power)
- minimum detectable effect at n=71: Cohen's f = 0.4434 (severely underpowered)
- to detect a conventional medium effect: 36 per group to detect f=0.25
- alpha = 0.05, target power = 0.8

# Celeus - reproduce.R  (run from this package folder; bundled data in data/)
# Test: Welch's ANOVA  |  engine 0.30.0  |  catalog c39-d3cd
# Research context: not provided for this run.
# Reproduces the recorded result deterministically.

set.seed(20261021)
.data_path <- "data/post-hoc-tests-after-anova.csv"
if (!file.exists(.data_path)) stop("Run reproduce.R from the package folder (the bundled data is in data/).")
data <- readr::read_csv(.data_path, show_col_types = FALSE)  # needs the readr package
clean_cells <- function (df) 
{
    ctrl_collapse <- "[\r\n\t\v\f]+"
    zero_width_chars <- intToUtf8(c(8203L, 65279L), multiple = TRUE)
    uni_space_chars <- intToUtf8(c(160L, 8239L, 8199L), multiple = TRUE)
    clean_text <- function(x) {
        out <- x
        for (ch in zero_width_chars) out <- gsub(ch, "", out, 
            fixed = TRUE)
        out <- gsub(ctrl_collapse, " ", out)
        for (ch in uni_space_chars) out <- gsub(ch, " ", out, 
            fixed = TRUE)
        trimws(gsub(" +", " ", out))
    }
    changed_cols <- character(0)
    cells_cleaned <- 0L
    retyped <- character(0)
    for (j in seq_along(df)) {
        col <- df[[j]]
        if (!is.character(col)) 
            next
        cleaned <- clean_text(col)
        n_changed <- sum(!is.na(col) & !is.na(cleaned) & col != 
            cleaned, na.rm = TRUE)
        if (n_changed > 0L) {
            changed_cols <- c(changed_cols, names(df)[j])
            cells_cleaned <- cells_cleaned + n_changed
        }
        df[[j]] <- cleaned
        non_blank <- cleaned[!is.na(cleaned) & nzchar(cleaned)]
        if (length(non_blank) && all(grepl("^[+-]?(0|[1-9][0-9]*)([.][0-9]+)?$", 
            non_blank))) {
            parsed <- suppressWarnings(as.numeric(cleaned))
            was_blank <- is.na(cleaned) | !nzchar(cleaned)
            lost <- (is.na(parsed) | is.infinite(parsed)) & !was_blank
            if (!any(lost)) {
                finite <- parsed[!is.na(parsed)]
                all_whole <- all(finite == floor(finite))
                in_range <- all(abs(finite) <= .Machine$integer.max)
                df[[j]] <- if (all_whole && in_range) 
                  as.integer(parsed)
                else parsed
                retyped <- c(retyped, names(df)[j])
            }
        }
    }
    names(df) <- clean_text(names(df))
    if (length(changed_cols)) {
        attr(df, "ingest_clean_columns") <- changed_cols
        attr(df, "ingest_clean_cells_cleaned") <- cells_cleaned
    }
    if (length(retyped)) 
        attr(df, "ingest_clean_retyped") <- retyped
    df
}
data <- clean_cells(data)
# Complete-case analysis: drop rows missing any analysis value.
.acols <- c("weight", "feed")
data <- data[stats::complete.cases(data[, .acols, drop = FALSE]), , drop = FALSE]
result <- oneway.test(`weight` ~ `feed`, data = data, var.equal = FALSE)
print(result)

# Effect size (Eta2) and its confidence interval.
effect <- { .es_d <- data; .es_d[["feed"]] <- as.factor(.es_d[["feed"]]); .es_d[["weight"]] <- .es_d[["weight"]] - stats::median(.es_d[["weight"]], na.rm = TRUE); effectsize::eta_squared(stats::aov(`weight` ~ `feed`, data = .es_d), partial = FALSE, ci = 0.95, alternative = "two.sided") }
print(effect)

# Post-hoc pairwise comparisons (Games-Howell; correction: single-step (Games-Howell)).
data[["feed"]] <- as.factor(data[["feed"]])
posthoc <- { .ph_d <- data; .ph_d[["feed"]] <- as.factor(.ph_d[["feed"]]); PMCMRplus::gamesHowellTest(`weight` ~ `feed`, data = .ph_d) }
print(posthoc)

# Sensitivity: minimum detectable effect at n=71 (power=0.8, alpha=0.05). NOT observed power.
power <- pwr::pwr.anova.test(k = 6, n = 11.83, sig.level = 0.05, power = 0.8)
print(power)