Homogeneity of variance and what to do about it

Homogeneity of variance is the assumption that every group you compare has the same spread around its own mean. Student’s t-test and the classic one-way ANOVA both rely on it. The practical answer to “what should I do about it” is shorter than most textbooks suggest: use the versions of those tests that do not need it, and spend your attention on why the spread differs.

This post covers what unequal variances do, how to look for them, why testing first is the wrong order, and what to do instead, ending with Welch’s ANOVA on R’s built-in InsectSprays data, run through a deterministic engine. It sits under the guide to parametric test assumptions, which covers the other assumptions these tests make.

What the assumption says

A comparison of means has two ingredients: the difference between the group means, and an estimate of how noisy each mean is. Student’s t-test and the classic ANOVA F test estimate that noise once, by pooling the variance of every group into a single number. Pooling is only a fair summary if the groups really share one variance. That is the assumption.

It concerns the populations, not the samples: two samples of 12 will rarely have identical standard deviations even when the populations do.

Several common methods carry it: Student’s two-sample t-test, the classic one-way and factorial ANOVA, Tukey’s HSD comparisons after ANOVA, and ordinary least squares regression, where the same idea is called homoscedasticity (constant residual variance across the range of the predictors). Welch’s t-test and Welch’s ANOVA do not. They estimate each group’s variance separately and adjust the degrees of freedom to match, which is why they appear throughout this post.

What unequal spread does to the error rate

When the assumption fails, the pooled variance is a weighted average of variances that are not the same, and the weights are the group sizes. That is where the damage comes from.

Unequal sizes, smaller group more variable. The pooled estimate is dominated by the large, quiet group, so it understates the noise in the small, noisy one. Test statistics come out too large, and the test rejects a true null hypothesis more often than the 5% you asked for. The test is liberal.

Unequal sizes, larger group more variable. The pooled estimate is dominated by the noisy group, so it overstates the noise. The test rejects less often than 5% under the null, which sounds safe, but the same inflation costs power when there is a real difference. The test is conservative.

Equal sizes. The weights are equal, so the two effects largely cancel and the classic tests drift only modestly from their nominal level. This is the condition behind the common claim that ANOVA is robust to unequal variances. It holds for balanced designs and moderate differences in spread; it is not a general licence. How much the error rate moves depends on two things together: how large the ratio of the biggest to the smallest variance is, and how unequal the group sizes are. Equal sizes soften a large ratio; sharply unequal sizes make even a modest ratio costly.

Delacre and colleagues showed this by simulation for the t-test in 2017 and for one-way ANOVA in 2019. In both papers the classic test gave biased error rates under realistic differences in spread and Welch’s version controlled the error rate better; the t-test paper adds that Welch’s test loses little when the variances are in fact equal.

The script below draws data where every group has the same true mean, so every rejection is a false positive, and counts how often each test rejects at the 0.05 level:

set.seed(20261021)
false_positive_rate <- function(n, sd, reps = 5000) {
  group <- factor(rep(seq_along(n), times = n))
  hits <- replicate(reps, {
    y <- rnorm(sum(n), mean = 0, sd = rep(sd, times = n))
    c(classic = oneway.test(y ~ group, var.equal = TRUE)$p.value < 0.05,
      welch   = oneway.test(y ~ group, var.equal = FALSE)$p.value < 0.05)
  })
  rowMeans(hits)
}
false_positive_rate(n = c(10, 20, 30), sd = c(3, 2, 1))  # smallest group most variable
false_positive_rate(n = c(10, 20, 30), sd = c(1, 2, 3))  # largest group most variable
false_positive_rate(n = c(20, 20, 20), sd = c(3, 2, 1))  # equal sizes, unequal spread
false_positive_rate(n = c(20, 20, 20), sd = c(1, 1, 1))  # equal sizes, equal spread

In the first scenario the classic F test rejects far more often than it should, in the second less often than 5%, and in the third a little above 5%. Welch’s test stays near 5% in all four.

Checking it: look first

The most useful check is the plainest one. Plot the outcome by group, with every point visible, and put each group’s standard deviation next to its mean:

with(InsectSprays, tapply(count, spray, mean))
with(InsectSprays, tapply(count, spray, sd))
boxplot(count ~ spray, data = InsectSprays)

Look for two things: how different the spreads are, and whether the spread rises with the mean. A variance that grows with the mean is a property of the outcome, not bad luck, and it is common with counts, concentrations, times and anything else that cannot go below zero.

For a regression, the equivalent is a plot of residuals against fitted values: a funnel that widens to the right is the same problem. A formal check there, such as the Breusch-Pagan test, shares the weakness described next: it tells you little in small samples and flags trivia in large ones.

Levene’s test, and what its p-value means

Formal tests exist. Levene’s test runs an ANOVA on the absolute distance of each observation from its group centre; if the groups spread differently, those distances differ. The version most software uses by default centres on the median rather than the mean. Brown and Forsythe proposed replacing the mean with a more robust centre such as the median, and found these versions held up better than the original when the data are not normal. In R, car::leveneTest() uses the median by default. Bartlett’s test is the older alternative and is sensitive to non-normality, so a rejection may reflect skew rather than unequal spread; bartlett.test() is only safe for close-to-normal data. Fligner-Killeen’s test, fligner.test(), works on ranks and is a nonparametric option. For exactly two groups there is also the F test of a variance ratio, var.test(), which assumes normal data and is very sensitive to departures from it.

Read the result with care:

• A small p-value is evidence that the population variances differ.
• A large p-value means a difference was not detected. It is not evidence that the variances are equal. With 10 or 12 observations per group, Levene’s test can miss differences large enough to bias Student’s t.
• With large samples, the test detects differences too small to matter.

So the test is weakest when the bias is worst and most sensitive when it matters least. Treat it as a description of the data, not a switch.

Why testing first and then choosing is the wrong order

A common workflow runs Levene’s test, then uses Student’s test if it passes and Welch’s if it fails. It feels careful but is not.

Zimmerman’s 2004 simulations studied exactly this two-stage procedure with unequal group sizes and unequal variances. It failed to protect the significance level and usually made the situation worse, and Welch’s test itself performed worse when it was used only on the occasions a preliminary test called for it. His conclusion was to use the separate-variances test unconditionally whenever sample sizes are unequal.

The cost of that default is small: when the variances truly are equal, Welch’s test gives nearly the same answer as Student’s. The cost of the opposite mistake, Student’s test when variances and group sizes both differ, can be a false positive rate well above the nominal one, as the simulation shows. The decision is therefore asymmetric, and it should be made before the data are seen: Welch’s t-test for two groups, Welch’s ANOVA for three or more.

What to do when groups differ

Unequal spread is information, not only a nuisance.

1. Compare the means with a method that does not assume equal variances. Welch’s t-test for two groups and Welch’s ANOVA for three or more. For pairwise comparisons after Welch’s ANOVA, use Games-Howell, which was developed for unequal sizes and variances. Tukey’s HSD pools the variance and inherits the problem. The guide to one-way ANOVA, Welch and Kruskal-Wallis covers the multi-group choice in more depth, and t-tests explained covers the two-group case.
2. Ask why the spread differs. If the variance rises with the mean, the outcome’s scale may be the issue. A log or square-root transformation can make spread more even, at the price of changing the question to one about the transformed scale. For counts, a model built for counts, such as Poisson or negative binomial regression, models the mean-variance relationship directly instead of working around it.
3. Do not switch to a rank test to fix it. Kruskal-Wallis and Mann-Whitney compare whole distributions. When the groups have clearly different spreads, a significant rank test cannot be read as a shift in location, so it does not answer the question about means that you started with. The t-test vs Mann-Whitney guide explains what those tests actually compare.
4. Report the spread as a result. If one treatment makes outcomes more variable, that may matter as much as its effect on the average.

Worked example: Welch’s ANOVA on InsectSprays

R’s InsectSprays data are cited to a 1942 paper by Beall about transforming counts from entomological field experiments so that analysis of variance becomes applicable. There are 72 experimental units, 12 per insecticide, labelled A to F, and the outcome is the count of insects in each unit. Each unit received one spray, so the six groups are independent. The question for the example is the plain one: does the mean count per unit differ between the sprays?

Seeded and run in R; Welch’s ANOVA was chosen before the run. Its output and script appear below the article.

The result

Welch’s ANOVA gave F(5, 30.04) = 36.07, p < 0.001, with n = 72. The denominator degrees of freedom, 30.04, are far below the 66 that the classic test would use (72 units minus 6 groups), because Welch’s method gives less weight to the noisier groups.

The effect size is eta squared = 0.72 (95% CI 0.60 to 0.79): in this sample, the spray accounts for about 72% of the variation in counts. Two caveats about that figure. It is computed from the classic ANOVA sums of squares, because Welch’s method changes the test, not this ratio. And its interval comes from the classic F distribution, which assumes equal variances, so with the spreads seen here treat the interval as approximate.

The assumption checks

Variance homogeneity (Levene, median-centred): failed. Levene’s statistic was 3.82 with p = 0.0042, evidence that the spread differs between sprays. By the argument above, the example would have used Welch’s test either way.

Normality by group (Shapiro-Wilk): failed in two groups. Spray C gave W = 0.859, p = 0.0476, and spray D gave W = 0.751, p = 0.0027. In A, B, E and F no departure was detected (p = 0.749, 0.641, 0.297 and 0.101). With 12 counts per group, “not detected” is weak information; it does not show those groups are normal.

Because two groups failed, the engine flagged the outcome as non-normal and suggested Kruskal-Wallis. The example keeps Welch’s ANOVA, for two reasons. The question is about means, and a rank test answers a different question, one that is hard to read when spreads differ, as they clearly do here. And counts are discrete and bounded at zero, which on its own can produce a Shapiro-Wilk rejection in a small group without the group means being badly estimated. That is a reason for caution, not a guarantee: with 12 per group, Welch’s test relies on the group means being close to normal, and strong skew would weaken that. It compares means without modelling the spread. For counts, whose variance usually grows with the mean, a count model, as in step 2 above, takes both the skew and the unequal spread seriously.

Which sprays differ

Games-Howell comparisons split the sprays into two sets. All nine pairs that cross the sets differed with adjusted p < 0.001, and the signs of the effect sizes show that A, B and F had the higher mean counts. Within the sets, no pair reached significance after adjustment: A against B gave adjusted p = 0.997, A against F 0.925, B against F 0.989, C against E 0.447 and D against E 0.601.

The effect sizes are Hedges’ g, the mean difference divided by the pooled standard deviation of the two groups in that pair. The largest was B against C, g = 3.84 (95% CI 2.46 to 5.20); the smallest of the cross-set pairs was D against F, g = -2.39 (95% CI -3.42 to -1.34). Two points about these numbers. The pooled denominator and the interval around it assume the two groups in the pair share a variance, so with unequal spread each g and its interval are approximate. And each interval is for that pair alone, not adjusted for the 15 comparisons, while the Games-Howell p-value is. That mainly explains C against D, where the adjusted p-value was 0.0557 but the unadjusted interval for g, -2.05 to -0.35, excludes zero. Read that pair as uncertain, not as a contradiction. The guide to multiple comparisons explains why adjusted and unadjusted results can disagree.

What the design could detect

The engine also reports a sensitivity figure for the design, not observed power computed from the result. With six groups of 12 and a two-sided test at the 0.05 level, the design had 80% power only for effects of Cohen’s f = 0.44 or larger; detecting f = 0.25, a conventional medium effect, with 80% power would need 36 units per group. The calculation assumes equal variances, so treat it as a guide. This result is far from the margin, but a study this size could easily miss a modest difference.

How to report it

A results sentence for the example might read:

Mean insect counts differed between sprays (Welch’s ANOVA, F(5, 30.04) = 36.07, p < 0.001; eta squared = 0.72, 95% CI 0.60 to 0.79). Variances differed between sprays (Levene’s test, median-centred, p = 0.0042); pairwise comparisons used Games-Howell tests, which do not assume equal variances. Sprays A, B and F each had higher mean counts than C, D and E (all adjusted p < 0.001); no pair within those two sets differed after adjustment.

Say in the methods section that Welch’s test was chosen in advance, not because Levene’s test failed. Give Welch’s degrees of freedom with their decimals, since they are what tells a reader which test was used.

Rerunning the example

The script below the article is the R script exactly as the engine generated it, with the seed and the bundled data path. It reads its data from a data/ folder. On the same package versions it reproduces Welch’s ANOVA (oneway.test() with var.equal = FALSE), eta squared and its interval, the Games-Howell p-values and the sensitivity figure. The assumption checks and pairwise effect sizes come from the engine itself: car::leveneTest(count ~ factor(spray), data = data), shapiro.test() within each spray and effectsize::hedges_g() on each pair reproduce them. The guide to reproducible statistical analysis explains what a reviewer needs in order to rerun a result like this one.

The example is about the method. It makes no claim about which insecticide to use.

# Results

- Test: Welch's ANOVA
- n: 72
- statistic: 36.07
- p-value: < 0.001
- effect size: eta-squared = 0.7244 (95% CI 0.6017 to 0.795)
  (two-sided CI at the run's confidence level)
- seed: 20261021

## Assumption checks
- normality by group (Shapiro-Wilk): passed=FALSE
- variance homogeneity (Levene): passed=FALSE (evidence of unequal variances)

## Post-hoc (Games-Howell; correction: single-step (Games-Howell))
- A vs B: adj p = 0.9972 | Hedges' g -0.1788 [-0.951, 0.5975]
- A vs C: adj p = 6.592e-06 | Hedges' g 3.314 [2.057, 4.541]
- B vs C: adj p = 6.653e-07 | Hedges' g 3.844 [2.462, 5.197]
- A vs D: adj p = 0.0001256 | Hedges' g 2.449 [1.379, 3.49]
- B vs D: adj p = 1.263e-05 | Hedges' g 2.873 [1.714, 4.002]
- C vs D: adj p = 0.05567 | Hedges' g -1.213 [-2.053, -0.3511]
- A vs E: adj p = 3.209e-05 | Hedges' g 2.988 [1.804, 4.141]
- B vs E: adj p = 3.675e-06 | Hedges' g 3.506 [2.204, 4.777]
- C vs E: adj p = 0.4466 | Hedges' g -0.7363 [-1.531, 0.07366]
- D vs E: adj p = 0.6006 | Hedges' g 0.6355 [-0.1661, 1.423]
- A vs F: adj p = 0.9247 | Hedges' g -0.3791 [-1.155, 0.4055]
- B vs F: adj p = 0.9888 | Hedges' g -0.2414 [-1.014, 0.537]
- C vs F: adj p = 3.421e-05 | Hedges' g -3.054 [-4.222, -1.855]
- D vs F: adj p = 0.0002914 | Hedges' g -2.395 [-3.425, -1.335]
- E vs F: adj p = 0.0001104 | Hedges' g -2.787 [-3.897, -1.646]

groups: A (a), B (a), C (b), D (b), E (b), F (a) - groups sharing a letter are not significantly different at the adjusted threshold.

## Power (sensitivity, not observed power)
- minimum detectable effect at n=72: Cohen's f = 0.44 (severely underpowered)
- to detect a conventional medium effect: 36 per group to detect f=0.25
- alpha = 0.05, target power = 0.8

# Celeus - reproduce.R  (run from this package folder; bundled data in data/)
# Test: Welch's ANOVA  |  engine 0.30.0  |  catalog c39-d3cd
# Research context: not provided for this run.
# Reproduces the recorded result deterministically.

set.seed(20261021)
.data_path <- "data/homogeneity-of-variance.csv"
if (!file.exists(.data_path)) stop("Run reproduce.R from the package folder (the bundled data is in data/).")
data <- readr::read_csv(.data_path, show_col_types = FALSE)  # needs the readr package
clean_cells <- function (df) 
{
    ctrl_collapse <- "[\r\n\t\v\f]+"
    zero_width_chars <- intToUtf8(c(8203L, 65279L), multiple = TRUE)
    uni_space_chars <- intToUtf8(c(160L, 8239L, 8199L), multiple = TRUE)
    clean_text <- function(x) {
        out <- x
        for (ch in zero_width_chars) out <- gsub(ch, "", out, 
            fixed = TRUE)
        out <- gsub(ctrl_collapse, " ", out)
        for (ch in uni_space_chars) out <- gsub(ch, " ", out, 
            fixed = TRUE)
        trimws(gsub(" +", " ", out))
    }
    changed_cols <- character(0)
    cells_cleaned <- 0L
    retyped <- character(0)
    for (j in seq_along(df)) {
        col <- df[[j]]
        if (!is.character(col)) 
            next
        cleaned <- clean_text(col)
        n_changed <- sum(!is.na(col) & !is.na(cleaned) & col != 
            cleaned, na.rm = TRUE)
        if (n_changed > 0L) {
            changed_cols <- c(changed_cols, names(df)[j])
            cells_cleaned <- cells_cleaned + n_changed
        }
        df[[j]] <- cleaned
        non_blank <- cleaned[!is.na(cleaned) & nzchar(cleaned)]
        if (length(non_blank) && all(grepl("^[+-]?(0|[1-9][0-9]*)([.][0-9]+)?$", 
            non_blank))) {
            parsed <- suppressWarnings(as.numeric(cleaned))
            was_blank <- is.na(cleaned) | !nzchar(cleaned)
            lost <- (is.na(parsed) | is.infinite(parsed)) & !was_blank
            if (!any(lost)) {
                finite <- parsed[!is.na(parsed)]
                all_whole <- all(finite == floor(finite))
                in_range <- all(abs(finite) <= .Machine$integer.max)
                df[[j]] <- if (all_whole && in_range) 
                  as.integer(parsed)
                else parsed
                retyped <- c(retyped, names(df)[j])
            }
        }
    }
    names(df) <- clean_text(names(df))
    if (length(changed_cols)) {
        attr(df, "ingest_clean_columns") <- changed_cols
        attr(df, "ingest_clean_cells_cleaned") <- cells_cleaned
    }
    if (length(retyped)) 
        attr(df, "ingest_clean_retyped") <- retyped
    df
}
data <- clean_cells(data)
# Complete-case analysis: drop rows missing any analysis value.
.acols <- c("count", "spray")
data <- data[stats::complete.cases(data[, .acols, drop = FALSE]), , drop = FALSE]
result <- oneway.test(`count` ~ `spray`, data = data, var.equal = FALSE)
print(result)

# Effect size (Eta2) and its confidence interval.
effect <- { .es_d <- data; .es_d[["spray"]] <- as.factor(.es_d[["spray"]]); .es_d[["count"]] <- .es_d[["count"]] - stats::median(.es_d[["count"]], na.rm = TRUE); effectsize::eta_squared(stats::aov(`count` ~ `spray`, data = .es_d), partial = FALSE, ci = 0.95, alternative = "two.sided") }
print(effect)

# Post-hoc pairwise comparisons (Games-Howell; correction: single-step (Games-Howell)).
data[["spray"]] <- as.factor(data[["spray"]])
posthoc <- { .ph_d <- data; .ph_d[["spray"]] <- as.factor(.ph_d[["spray"]]); PMCMRplus::gamesHowellTest(`count` ~ `spray`, data = .ph_d) }
print(posthoc)

# Sensitivity: minimum detectable effect at n=72 (power=0.8, alpha=0.05). NOT observed power.
power <- pwr::pwr.anova.test(k = 6, n = 12, sig.level = 0.05, power = 0.8)
print(power)