Dunn's test after Kruskal-Wallis, explained

Dunn’s test tells you which pairs of groups differ after a Kruskal-Wallis test. It compares the groups’ mean ranks from the joint ranking the Kruskal-Wallis test already built, and it needs a p-value adjustment that you choose and name, usually Holm.

This post runs the whole sequence once on R’s built-in InsectSprays data through a deterministic engine, then reruns it with the Bonferroni adjustment so the two can be compared on the same pairs.

What Dunn’s test compares

The Kruskal-Wallis test (Kruskal and Wallis, 1952) replaces every observation by its rank among all observations, from smallest to largest, and asks whether the groups’ average ranks differ by more than chance would allow. A significant result says the groups are not all alike. It does not say which ones differ.

Dunn’s procedure (Dunn, 1964) answers that pair by pair. For two groups it takes the difference between their mean ranks, from that same joint ranking, and divides it by a standard error built from the variance of all the ranks and the two group sizes. The implementation used here corrects that standard error for tied values. The result is referred to the normal distribution, which gives one z statistic and one two-sided p-value per pair. With k groups there are k(k - 1)/2 pairs: 15 for six groups.

Three consequences follow.

It compares ranks, not means or medians. A difference in mean ranks says that values in one group tend to be larger than values in the other. It becomes a statement about medians only when the groups’ distributions have the same shape and spread and differ by a shift.

A pair’s result depends on the other groups. Because the ranks come from all groups together, the mean ranks of groups A and B depend on where the other groups’ observations fall, and the standard error uses the spread of all the ranks.

It is not a set of Mann-Whitney tests. A Mann-Whitney test on one pair re-ranks only those two groups, which answers a different question.

Holm or Bonferroni: the adjustment you name

Dunn’s test has no built-in multiplicity correction. You pick the adjustment, and the report must name it: “Dunn’s test” alone does not say whether the p-values were adjusted.

Bonferroni multiplies every raw p-value by the number of comparisons, capped at 1. Dunn (1961) used the Bonferroni inequality to build simultaneous confidence intervals for a set of comparisons among means chosen in advance, and the correction is sometimes called Dunn-Bonferroni after that work. It controls the family-wise error rate, the chance of at least one false positive across all pairs, whatever the dependence between the tests.

Holm (Holm, 1979) is a step-down version. Sort the raw p-values from smallest to largest. Multiply the smallest by the number of comparisons, the next by one fewer, and so on, then carry the running maximum forward so the adjusted values never decrease. It controls the same family-wise error rate under the same conditions, and each adjusted p-value is less than or equal to Bonferroni’s. Holm therefore never rejects fewer hypotheses than Bonferroni, which makes it the better default.

A third option, Benjamini-Hochberg (Benjamini and Hochberg, 1995), controls the false discovery rate instead: the expected share of false positives among the pairs you call significant. It suits screening, not confirmatory claims about specific pairs. The multiple comparisons guide covers when each error rate fits.

Worked example: six insecticides

InsectSprays records the number of insects on 72 field plots, 12 plots for each of six sprays labelled A to F. Each plot received one spray, so the six groups are independent, and the outcome is a count. The data come from Beall (1942), a paper on transforming insect counts so that a classic analysis of variance applies. A rank-based test sidesteps that: it needs only that larger counts mean more insects.

The engine ran a Kruskal-Wallis test, then Dunn’s test with the Holm adjustment.

The omnibus result

The Kruskal-Wallis statistic is H = 54.69 on 5 degrees of freedom, p < 0.001. The rank-based effect size, epsilon-squared, is 0.7703 with a 95% percentile-bootstrap interval from 0.7327 to 0.837. That is a share of the variation in the ranks, not in the raw counts, and it is large. The engine noted that the data contain tied counts and that the z statistics were corrected for ties.

The pairwise results

Each row gives Dunn’s adjusted p-value under Holm (the main run) and under Bonferroni (the companion run), with the rank-biserial correlation for the pair. A positive r means plots under the first spray tend to have more insects than plots under the second.

Pair	Holm adj. p	Bonferroni adj. p	Rank-biserial r (95% CI, unadjusted)
A vs B	1	1	-0.1389 (-0.545 to 0.32)
A vs C	2.347e-05	2.708e-05	0.9931 (0.9823 to 0.9973)
A vs D	0.01276	0.02735	0.9514 (0.8798 to 0.9808)
A vs E	0.001179	0.001768	1 (1 to 1)
A vs F	1	1	-0.1944 (-0.584 to 0.2678)
B vs C	5.098e-06	5.462e-06	0.9931 (0.9823 to 0.9973)
B vs D	0.004823	0.009044	0.9583 (0.8964 to 0.9836)
B vs E	0.0003451	0.0004706	1 (1 to 1)
B vs F	1	1	-0.03472 (-0.467 to 0.411)
C vs D	0.5857	1	-0.7222 (-0.8818 to -0.4143)
C vs E	1	1	-0.4653 (-0.7511 to -0.03252)
C vs F	3.334e-06	3.334e-06	-1 (-1 to -1)
D vs E	1	1	0.3264 (-0.1319 to 0.6697)
D vs F	0.003838	0.006397	-0.9583 (-0.9836 to -0.8964)
E vs F	0.0002496	0.000312	-1 (-1 to -1)

At the 0.05 level, both adjustments flag the same nine pairs: each of A, B and F against each of C, D and E. Read in the direction of r, plots under C, D and E tend to carry fewer insects than plots under A, B and F, and within each set of three the test does not separate the sprays.

What the comparison shows

Holm’s adjusted p-values are never larger than Bonferroni’s. For C vs F the two are identical, 3.334e-06, because Holm multiplies the smallest raw p-value by the full number of comparisons, exactly as Bonferroni does. From the second step on, Holm multiplies by fewer, so its values are smaller: A vs D is 0.01276 under Holm and 0.02735 under Bonferroni, and C vs D is 0.5857 against 1.

Here the choice changed no decision. The separation between the two sets of sprays is so strong that every pair is either far below 0.05 or far above it under both methods. With pairs nearer the threshold, Holm can call a pair significant that Bonferroni misses, and the reverse cannot happen. Either way, fix the adjustment in the plan before you look.

The effect-size intervals answer a different question. For C vs D the rank-biserial r is -0.7222 with an interval from -0.8818 to -0.4143, which excludes zero, yet Dunn’s adjusted p-value is 0.5857. The main reason is the ranking. r re-ranks only the 24 plots of C and D, where D’s plots mostly sit above C’s. Dunn’s test uses all 72 ranks: C, D and E share the bottom half, E falls between C and D, and the standard error comes from the spread of all 72 ranks, which is wide next to the gap C and D can open inside the bottom half. The adjustment matters less: with p.adjust.method = "none" in the script, Dunn’s p-value for C vs D is still above 0.05. The interval is also unadjusted for the 15 comparisons.

An interval of 1 to 1 is not certainty. For A vs E, B vs E, C vs F and E vs F, every plot in one group had more insects than every plot in the other, so r is 1 or -1. The interval is built on Fisher’s z scale, where an r of exactly 1 or -1 maps to infinity, so both limits map back to the same bound and the interval collapses to a single point. Report the separation, and do not read a zero-width interval as perfect precision.

Assumptions, and what the check does not cover

Kruskal-Wallis and Dunn’s test assume independent observations, here one spray per plot, and an outcome that can be ordered. The engine’s only assumption check for this test was the sample size, which passed. That is a floor for running the test, not an assessment of whether the design suits it.

No check covers shape. If the sprays’ counts differ in spread as well as in typical level, the rank comparison is still valid as a test of whether one group tends to be higher. It just cannot be reported as a difference in medians. The engine reports no power figure, and this post computes none after the fact: power derived from the observed effect adds nothing that the p-values and effect sizes do not already say.

Should Kruskal-Wallis be significant first?

Not strictly. Dunn’s test with Holm or Bonferroni controls the family-wise error rate across all 15 pairs by itself, approximately, since its z statistics rest on a large-sample normal approximation. Requiring a significant omnibus test first is common practice and is allowed, but it makes the combined procedure more conservative. What matters is that the rule is in the plan before the data are seen.

For a question about means rather than ranks, use Welch’s ANOVA followed by Games-Howell. The post on post hoc tests after ANOVA runs Tukey, Games-Howell and Dunn side by side and covers how to pick one. The guide to one-way ANOVA, Welch’s ANOVA and Kruskal-Wallis compares the three omnibus tests, and the t-tests explainer covers the two-group case.

Alternatives, other software, and when not to use it

PMCMRplus, the package behind this run, also offers other rank-based follow-ups: the Nemenyi test (joint mean ranks referred to the studentized range distribution), the Conover test (a t statistic) and the Dwass-Steel-Critchlow-Fligner test, which re-ranks each pair. Against one control group, kwManyOneDunnTest() runs five comparisons here, not 15. Defaults differ: kwAllPairsDunnTest() and FSA::dunnTest() use Holm, while Python’s scikit-posthocs posthoc_dunn() adjusts nothing unless asked. Stata’s dunntest is described by Dinno (2015). For repeated measures on the same units, use the Friedman test and its follow-ups.

How to report Dunn’s test

Name the omnibus test, its statistic and effect size, the follow-up, and the adjustment. For this example:

Insect counts differed between the six sprays (Kruskal-Wallis H = 54.69, df = 5, p < 0.001; epsilon-squared = 0.7703, 95% CI 0.7327 to 0.837). Dunn’s pairwise comparisons with Holm-adjusted p-values showed that sprays C, D and E each differed from sprays A, B and F (adjusted p from < 0.001 to 0.01276), with plots under C, D and E tending to carry fewer insects. No pair within either set differed (adjusted p from 0.5857 to 1).

Put the full pairwise table, with the effect sizes, in a table or supplement.

Rerunning the example

The R script below the article is the one the engine generated. On the same package versions it reproduces the Kruskal-Wallis test, the epsilon-squared estimate with its bootstrap interval, and Dunn’s test with the Holm adjustment through PMCMRplus::kwAllPairsDunnTest(). The rank-biserial values and their intervals come from effectsize::rank_biserial() on each pair and are not in the script. For the Bonferroni column, change p.adjust.method = "holm" to "bonferroni".

The guide to reproducible statistical analysis explains what a reviewer needs to rerun a result like this.

# Results

- Test: Kruskal-Wallis test
- n: 72
- statistic: 54.69
- p-value: < 0.001
- effect size: epsilon-squared = 0.7703 (95% CI 0.7327 to 0.837)
  (two-sided percentile-bootstrap CI at the run's confidence level)
- seed: 20261022

## Assumption checks
- sample size: passed=TRUE (adequate n)

## Post-hoc (Dunn's test; correction: Holm)
- A vs B: adj p = 1 | rank-biserial r -0.1389 [-0.545, 0.32]
- A vs C: adj p = 2.347e-05 | rank-biserial r 0.9931 [0.9823, 0.9973]
- B vs C: adj p = 5.098e-06 | rank-biserial r 0.9931 [0.9823, 0.9973]
- A vs D: adj p = 0.01276 | rank-biserial r 0.9514 [0.8798, 0.9808]
- B vs D: adj p = 0.004823 | rank-biserial r 0.9583 [0.8964, 0.9836]
- C vs D: adj p = 0.5857 | rank-biserial r -0.7222 [-0.8818, -0.4143]
- A vs E: adj p = 0.001179 | rank-biserial r 1 [1, 1]
- B vs E: adj p = 0.0003451 | rank-biserial r 1 [1, 1]
- C vs E: adj p = 1 | rank-biserial r -0.4653 [-0.7511, -0.03252]
- D vs E: adj p = 1 | rank-biserial r 0.3264 [-0.1319, 0.6697]
- A vs F: adj p = 1 | rank-biserial r -0.1944 [-0.584, 0.2678]
- B vs F: adj p = 1 | rank-biserial r -0.03472 [-0.467, 0.411]
- C vs F: adj p = 3.334e-06 | rank-biserial r -1 [-1, -1]
- D vs F: adj p = 0.003838 | rank-biserial r -0.9583 [-0.9836, -0.8964]
- E vs F: adj p = 0.0002496 | rank-biserial r -1 [-1, -1]

groups: A (a), B (a), C (b), D (b), E (b), F (a) - groups sharing a letter are not significantly different at the adjusted threshold.

## Power
- Power is not calculated for this test. The matching parametric test gives a rough guide.

# Celeus - reproduce.R  (run from this package folder; bundled data in data/)
# Test: Kruskal-Wallis test  |  engine 0.30.0  |  catalog c39-d3cd
# Research context: not provided for this run.
# Reproduces the recorded result deterministically.

set.seed(20261022)
.data_path <- "data/dunns-test.csv"
if (!file.exists(.data_path)) stop("Run reproduce.R from the package folder (the bundled data is in data/).")
data <- readr::read_csv(.data_path, show_col_types = FALSE)  # needs the readr package
clean_cells <- function (df) 
{
    ctrl_collapse <- "[\r\n\t\v\f]+"
    zero_width_chars <- intToUtf8(c(8203L, 65279L), multiple = TRUE)
    uni_space_chars <- intToUtf8(c(160L, 8239L, 8199L), multiple = TRUE)
    clean_text <- function(x) {
        out <- x
        for (ch in zero_width_chars) out <- gsub(ch, "", out, 
            fixed = TRUE)
        out <- gsub(ctrl_collapse, " ", out)
        for (ch in uni_space_chars) out <- gsub(ch, " ", out, 
            fixed = TRUE)
        trimws(gsub(" +", " ", out))
    }
    changed_cols <- character(0)
    cells_cleaned <- 0L
    retyped <- character(0)
    for (j in seq_along(df)) {
        col <- df[[j]]
        if (!is.character(col)) 
            next
        cleaned <- clean_text(col)
        n_changed <- sum(!is.na(col) & !is.na(cleaned) & col != 
            cleaned, na.rm = TRUE)
        if (n_changed > 0L) {
            changed_cols <- c(changed_cols, names(df)[j])
            cells_cleaned <- cells_cleaned + n_changed
        }
        df[[j]] <- cleaned
        non_blank <- cleaned[!is.na(cleaned) & nzchar(cleaned)]
        if (length(non_blank) && all(grepl("^[+-]?(0|[1-9][0-9]*)([.][0-9]+)?$", 
            non_blank))) {
            parsed <- suppressWarnings(as.numeric(cleaned))
            was_blank <- is.na(cleaned) | !nzchar(cleaned)
            lost <- (is.na(parsed) | is.infinite(parsed)) & !was_blank
            if (!any(lost)) {
                finite <- parsed[!is.na(parsed)]
                all_whole <- all(finite == floor(finite))
                in_range <- all(abs(finite) <= .Machine$integer.max)
                df[[j]] <- if (all_whole && in_range) 
                  as.integer(parsed)
                else parsed
                retyped <- c(retyped, names(df)[j])
            }
        }
    }
    names(df) <- clean_text(names(df))
    if (length(changed_cols)) {
        attr(df, "ingest_clean_columns") <- changed_cols
        attr(df, "ingest_clean_cells_cleaned") <- cells_cleaned
    }
    if (length(retyped)) 
        attr(df, "ingest_clean_retyped") <- retyped
    df
}
data <- clean_cells(data)
# Complete-case analysis: drop rows missing any analysis value.
.acols <- c("count", "spray")
data <- data[stats::complete.cases(data[, .acols, drop = FALSE]), , drop = FALSE]
result <- kruskal.test(`count` ~ `spray`, data = data)
print(result)

# Effect size (epsilon_squared) and its confidence interval.
effect <- { set.seed(20261022); .es_d <- data; .es_d[["spray"]] <- as.factor(.es_d[["spray"]]); effectsize::rank_epsilon_squared(.es_d[["count"]], .es_d[["spray"]], ci = 0.95, alternative = "two.sided", iterations = 1000L) }
print(effect)

# Post-hoc pairwise comparisons (Dunn's test; correction: Holm).
data[["spray"]] <- as.factor(data[["spray"]])
posthoc <- { .ph_d <- data; .ph_d[["spray"]] <- as.factor(.ph_d[["spray"]]); PMCMRplus::kwAllPairsDunnTest(`count` ~ `spray`, data = .ph_d, p.adjust.method = "holm") }
print(posthoc)