Welch's t-test: when to use it instead of Student's

Use Welch’s t-test whenever you compare the means of two independent groups and cannot justify, from how the study was designed, that both populations have the same variance. Student’s version is only reliable when that assumption holds, and reasonably robust when the groups are the same size; otherwise it fails in a predictable direction when it does not: it overstates precision when the smaller group is the more variable one, and loses power when the larger group is.

This post goes one level below the general guide to t-tests. The only question here is Welch against Student: where they differ, why Welch’s degrees of freedom are fractional, why a variance test is the wrong way to choose, and a worked example where group sizes and spreads both differ, so the contrast is real.

The only thing that differs: the standard error

Both tests divide the same estimate, the difference between two group means, by a standard error. Everything that separates them sits in that denominator.

Student’s test assumes the two populations share one variance and estimates it by pooling the two sample variances, each weighted by its degrees of freedom. The standard error is that pooled variance scaled by the two group sizes:

s_pooled^2 = ((n1 - 1) * s1^2 + (n2 - 1) * s2^2) / (n1 + n2 - 2)
SE_student = sqrt(s_pooled^2 * (1/n1 + 1/n2))

Welch’s test drops the shared-variance assumption and lets each group contribute its own variance, divided by its own size:

SE_welch = sqrt(s1^2 / n1 + s2^2 / n2)

When the group sizes are equal, these two standard errors are exactly the same number, whatever the variances. That is why balanced designs are fairly forgiving: Student’s test then differs from Welch’s only in its degrees of freedom.

When the sizes differ, the pooled variance is pulled towards the variance of the larger group, because that group has more degrees of freedom. Two cases follow, and they go in opposite directions.

Design	What pooling does	Effect on Student’s test
Equal sizes, any variances	Same standard error as Welch	Close to Welch; only the degrees of freedom differ
Unequal sizes, equal variances	Pooling is correct	Fine; Welch gives almost the same answer
Smaller group has the larger variance	Pooled variance too small for the noisy group	Standard error too small: too many false positives, intervals too narrow
Smaller group has the smaller variance	Pooled variance too large for the precise group	Standard error too large: too few rejections, lost power

Ruxton’s 2006 review summarises simulations showing exactly this pattern: Student’s error rate was several times the nominal 5% when the larger variance went with the smaller sample, and a fraction of it in the reverse case, while Welch’s stayed close to 5% throughout, including when the variances were equal. Delacre, Lakens and Leys reached the same conclusion in 2017 and recommend Welch as the default for that reason. The asymmetry is the whole argument: Welch costs very little when it is not needed and protects you when it is.

Welch-Satterthwaite degrees of freedom

Using a separate-variances standard error raises a new problem. The ratio of the difference to that standard error does not follow an exact t distribution, because the standard error now mixes two variance estimates of different precision. The approximation used in practice, published by Welch and by Satterthwaite in the 1940s (Satterthwaite for combinations of variance estimates in general), is to use a t distribution whose degrees of freedom match how much information the standard error actually carries:

df = (s1^2/n1 + s2^2/n2)^2 / ( (s1^2/n1)^2 / (n1 - 1) + (s2^2/n2)^2 / (n2 - 1) )

Three things are worth knowing about this number.

It is usually not a whole number. That is the quickest way to recognise Welch output, and it is why a report should give it with decimals.

It always sits between two bounds. It is never smaller than the smaller group’s size minus one and never larger than the total size minus two, which is Student’s value. It reaches the upper bound only when the two variance terms, each variance divided by its group size, are in a particular balance; when one group’s term dominates the standard error, the degrees of freedom fall towards that group’s size minus one.

It tells you which group drives the uncertainty. A Welch df far below Student’s says the standard error is mostly the noisier group’s variance term, so the precision of the whole comparison rests on that group. The smaller df also means a larger critical value, Welch’s honest response to having less information than the total sample size suggests.

Worked example: unequal sizes, unequal spreads

To make the contrast concrete, the example uses seeded synthetic data because the example needs sizes and spreads that both differ by a known amount. Two independent groups were generated from normal distributions: a small “pilot” group of 16 units drawn with a standard deviation of 12, and a larger “standard” group of 48 units drawn with a standard deviation of 4. The true means were 55 and 50. These settings describe no real study; they were chosen so that the smaller group is the more variable one, the case where Student’s test goes wrong. The example illustrates the method and is not evidence about any population. Each unit contributes one score, so the groups are independent and there is no pairing.

Seeded and run in R; Welch’s t-test was chosen before the run.

Quantity	Value
Units analysed	64 (16 pilot, 48 standard)
Mean score, pilot	56.67
Mean score, standard	50.43
Difference in means, pilot minus standard	6.24
95% confidence interval for the difference	0.85 to 11.64
Welch’s t	2.446 on 16.45 degrees of freedom
p-value (two-sided)	0.026
Cohen’s d (pooled SD) with its 95% interval	1.06 (0.46 to 1.65)

Reading it

The pilot group’s mean was 6.24 points higher than the standard group’s. The 95% confidence interval runs from 0.85 to 11.64: the data are compatible with a small positive difference and with one nearly twice the size of the estimate. With p = 0.026, the difference is statistically significant at the 0.05 level, and the interval excludes zero, which is the same information.

Now look at the degrees of freedom. There are 64 units, so Student’s test would use 62 degrees of freedom. Welch’s test used 16.45, close to the lower bound of 15 set by the pilot group. That is the Welch-Satterthwaite formula telling you that the standard error is dominated by the 16 noisy pilot values, and the 48 tightly clustered standard values add little to its uncertainty. In effect, the standard error is about as well estimated as it would be from a single sample of 17 or so, not from 64 units.

What Student’s test would have said

Run the same data with var.equal = TRUE and Student’s test reports a larger t on 62 degrees of freedom, a narrower interval and a much smaller p-value. Both tests reject a zero difference here, so the direction does not change. The claimed precision does: the pooled variance is dominated by the 48 standard values, so Student’s standard error treats the pilot group as if it were as consistent as the standard group. Its interval is narrower because the standard error is wrong, not because the data hold more information.

# On the bundled data from the example (run from the package folder):
data <- read.csv("data/welchs-t-test.csv")
t.test(score ~ group, data = data)                    # Welch, R's default
t.test(score ~ group, data = data, var.equal = TRUE)  # Student, for comparison only

In a closer case the same overstatement turns a non-significant Welch result into a significant Student one. Over many studies, that is the inflated false-positive rate in the table above.

The assumption checks, read in context

Variance homogeneity (Levene): failed. Levene’s statistic was 22.23 with p < 0.001, so unequal variances were detected. Here the check agrees with the design, which is no surprise because the data were generated that way. It did not choose the test: Welch was specified before the run and would have been used whatever Levene said.

Normality by group (Shapiro-Wilk): passed. W was 0.966 (p = 0.779) in the pilot group and 0.976 (p = 0.425) in the standard group, so no departure from normality was detected. That is not evidence that the scores are normal, especially in a group of 16, where the test has little power. Note also that Welch’s test addresses unequal variances, not non-normality: with small, strongly skewed groups it inherits the same concerns as Student’s. If small groups are heavy-tailed or prone to outliers, Yuen’s trimmed-means version, built for unequal variances, is a standard option; Yuen’s 1974 comparison with Welch recommends it for that case.

Values outside Tukey’s fence. The engine flagged three scores outside Tukey’s interquartile-range fence. That fence is computed over both groups together, so values from the wider group can fall outside it simply because that group is wider. The data were drawn from normal distributions, so these are not errors, and nothing was removed.

Sensitivity. With 16 and 48 units, a two-sided test at the 0.05 level would reach 80% power only for effects of d = 0.82 or larger. That calculation assumes equal variances, so it is a rough guide here, and it describes the design, not this result. A separate design point follows from Welch’s standard error rather than from this calculation: for a fixed total, s1^2/n1 + s2^2/n2 is smallest when each group’s size is proportional to its standard deviation, so a group expected to be more variable should get more units, not fewer.

Which effect size, and with which SD

Cohen’s d was 1.06 (95% interval 0.46 to 1.65). It divides the difference by the pooled standard deviation, which carries the same weighting problem as Student’s standard error: with 48 standard units against 16 pilot units, it leans towards the standard group’s smaller spread. Divided by the pilot group’s own SD, the same difference would look smaller; by the standard group’s SD, larger. No single standardizer is correct when spreads clearly differ, so say which one you used.

The interval for d is also computed the standard way, which assumes equal variances. In this design that assumption is clearly false, so treat the interval as approximate.

Two practical rules follow:

• Lead with the raw difference and Welch’s interval. In the example, 6.24 points with an interval from 0.85 to 11.64 needs no assumption about which SD to divide by.
• If you standardize, name the denominator. Lakens’ 2013 primer recommends Glass’s delta, which uses one group’s SD, typically a control or reference group, whenever the standard deviations differ substantially between conditions.

Why not run Levene’s test first

Running Levene’s test and then picking Student or Welch from its p-value feels careful. It is not, for three reasons.

It has the least power where it matters most. Unequal variances do the most damage when one group is small, exactly where a variance test is least able to detect them. In large samples it flags trivial differences instead.

It changes the error rate of the test that follows. A two-step procedure is a different test from either of its parts. Zimmerman’s 2004 simulations found that the Levene-then-t procedure fails to protect the significance level and usually makes the situation worse, and that using the separate-variances test unconditionally gives the best protection when sample sizes are unequal.

“Not detected” is not “equal”. A non-significant Levene result says the data could not distinguish the variances, not that they are the same. Using it as permission to pool treats absence of evidence as evidence.

In the example, Levene’s test happened to detect the difference. That does not rescue the procedure. Choose Welch before the data arrive.

Check it yourself: a short simulation

You do not have to take the error-rate claims on trust. The script below generates many datasets in which the true means are equal, so every significant result is a false positive, and counts how often each test rejects at the 0.05 level. It uses the same sizes and spreads as the worked example, then swaps which group is the more variable.

set.seed(20261021)
reps <- 10000
false_positive_rate <- function(n1, n2, sd1, sd2) {
  hits <- replicate(reps, {
    x <- rnorm(n1, mean = 0, sd = sd1)
    y <- rnorm(n2, mean = 0, sd = sd2)
    c(student = t.test(x, y, var.equal = TRUE)$p.value < 0.05,
      welch   = t.test(x, y)$p.value < 0.05)
  })
  rowMeans(hits)
}
false_positive_rate(16, 48, sd1 = 12, sd2 = 4)  # smaller group more variable
false_positive_rate(16, 48, sd1 = 4, sd2 = 12)  # smaller group less variable
false_positive_rate(16, 48, sd1 = 8, sd2 = 8)   # equal variances

With the smaller group more variable, Student’s rate lands far above 5%. With the roles swapped it falls far below, so real effects are missed. With equal variances both sit close to 5%, and Welch stays close to 5% in all three runs. Change the sizes and spreads to match your own design.

When Student’s test is still fine

Student’s test is a special case, not a wrong one. It fits when equal variances follow from the design rather than from a test: the same measurement process, randomized units, and a strong reason to expect the treatment to shift the mean without changing the spread. Even then Welch gives almost the same answer, so there is little to gain from switching.

Two adjacent decisions are often confused with this one. If your question is about ranks rather than means, the t-test or Mann-Whitney guide explains why that is a different question, not a fix for unequal variances. With three or more groups, the same reasoning carries over to Welch’s ANOVA, with Games-Howell comparisons for the pairwise follow-ups. If you are not sure a two-group comparison fits your design at all, start from the guide to choosing a statistical test.

Reporting Welch’s t-test

Name the test, give the means and group sizes, the difference with its interval, t with its fractional degrees of freedom, the exact p-value, and an effect size with its denominator named. Ruxton also recommends giving each group’s variance or SD, so a reader can see why Welch was needed. For the worked example:

The pilot group (16 units) had a mean score of 56.67 and the standard group (48 units) a mean of 50.43. Welch’s t-test, chosen in advance because the groups differed in size and were expected to differ in spread, gave a difference of 6.24 (95% CI 0.85 to 11.64), t(16.45) = 2.45, p = 0.026. Cohen’s d using the pooled SD was 1.06 (95% CI 0.46 to 1.65); because the spreads differed, this standardized value depends on the choice of SD and its interval is approximate.

Report the effect as a difference in this sample, without “proves” or causal language, unless the design supports it.

Rerunning the example

Below the article are the R script exactly as the engine generated it and the output it printed. On the same package versions, the script reproduces the Welch test and the sensitivity figure. The effect size and assumption checks come from the engine itself; effectsize::cohens_d(score ~ group, data = data), car::leveneTest(score ~ factor(group), data = data) and shapiro.test() within each group reproduce them. The script reads the bundled data file rather than regenerating it, and it does not run the Student comparison or the simulation; those are the separate snippets above.

The engine’s Welch’s t-test reference and Student’s t-test reference list what each method checks and reports.

# Results

- Test: Welch's t-test
- n: 64
- statistic: 2.446
- p-value: 0.026
- effect size: Cohen's d = 1.058 (95% CI 0.4583 to 1.649)
- 95% CI: [0.8451, 11.64]
- seed: 20261021

## Assumption checks
- normality by group (Shapiro-Wilk): passed=TRUE
- variance homogeneity (Levene): passed=FALSE (evidence of unequal variances)
- sample size: passed=TRUE (adequate n)

## Power (sensitivity, not observed power)
- minimum detectable effect at n=64: Cohen's d = 0.8215 (severely underpowered)
- to detect a conventional medium effect: 64 per group to detect d=0.50
- alpha = 0.05, target power = 0.8

# Celeus - reproduce.R  (run from this package folder; bundled data in data/)
# Test: Welch's t-test  |  engine 0.30.0  |  catalog c39-d3cd
# Research context: not provided for this run.
# Reproduces the recorded result deterministically.

set.seed(20261021)
.data_path <- "data/welchs-t-test.csv"
if (!file.exists(.data_path)) stop("Run reproduce.R from the package folder (the bundled data is in data/).")
data <- readr::read_csv(.data_path, show_col_types = FALSE)  # needs the readr package
clean_cells <- function (df) 
{
    ctrl_collapse <- "[\r\n\t\v\f]+"
    zero_width_chars <- intToUtf8(c(8203L, 65279L), multiple = TRUE)
    uni_space_chars <- intToUtf8(c(160L, 8239L, 8199L), multiple = TRUE)
    clean_text <- function(x) {
        out <- x
        for (ch in zero_width_chars) out <- gsub(ch, "", out, 
            fixed = TRUE)
        out <- gsub(ctrl_collapse, " ", out)
        for (ch in uni_space_chars) out <- gsub(ch, " ", out, 
            fixed = TRUE)
        trimws(gsub(" +", " ", out))
    }
    changed_cols <- character(0)
    cells_cleaned <- 0L
    retyped <- character(0)
    for (j in seq_along(df)) {
        col <- df[[j]]
        if (!is.character(col)) 
            next
        cleaned <- clean_text(col)
        n_changed <- sum(!is.na(col) & !is.na(cleaned) & col != 
            cleaned, na.rm = TRUE)
        if (n_changed > 0L) {
            changed_cols <- c(changed_cols, names(df)[j])
            cells_cleaned <- cells_cleaned + n_changed
        }
        df[[j]] <- cleaned
        non_blank <- cleaned[!is.na(cleaned) & nzchar(cleaned)]
        if (length(non_blank) && all(grepl("^[+-]?(0|[1-9][0-9]*)([.][0-9]+)?$", 
            non_blank))) {
            parsed <- suppressWarnings(as.numeric(cleaned))
            was_blank <- is.na(cleaned) | !nzchar(cleaned)
            lost <- (is.na(parsed) | is.infinite(parsed)) & !was_blank
            if (!any(lost)) {
                finite <- parsed[!is.na(parsed)]
                all_whole <- all(finite == floor(finite))
                in_range <- all(abs(finite) <= .Machine$integer.max)
                df[[j]] <- if (all_whole && in_range) 
                  as.integer(parsed)
                else parsed
                retyped <- c(retyped, names(df)[j])
            }
        }
    }
    names(df) <- clean_text(names(df))
    if (length(changed_cols)) {
        attr(df, "ingest_clean_columns") <- changed_cols
        attr(df, "ingest_clean_cells_cleaned") <- cells_cleaned
    }
    if (length(retyped)) 
        attr(df, "ingest_clean_retyped") <- retyped
    df
}
data <- clean_cells(data)
# Complete-case analysis: drop rows missing any analysis value.
.acols <- c("score", "group")
data <- data[stats::complete.cases(data[, .acols, drop = FALSE]), , drop = FALSE]
result <- t.test(`score` ~ `group`, data = data, var.equal = FALSE)
print(result)

# Sensitivity: minimum detectable effect at n=64 (power=0.8, alpha=0.05). NOT observed power.
power <- pwr::pwr.t2n.test(n1 = 16, n2 = 48, sig.level = 0.05, power = 0.8)
print(power)