Paired t-test vs Wilcoxon signed-rank: which one?

Use the paired t-test when the within-pair differences look roughly symmetric and the mean change is what you want to report; with a reasonable number of pairs, moderate skew in the differences is not a problem. Use the Wilcoxon signed-rank test when there are few pairs and the differences are clearly heavy-tailed, and report the Hodges-Lehmann estimate with its interval. Check these properties on the differences, not on the before and after columns.

When each unit is measured twice (before and after an intervention, left and right eye, two assays on the same sample), the analysis is about one column of numbers: the within-unit differences. Both the paired t-test and the Wilcoxon signed-rank test reduce the data to those differences first. Everything that matters for choosing between them, including the assumptions people usually check on the wrong variables, is a property of the differences.

What each test does with the differences

Call the difference for unit i d_i = after_i - before_i.

The paired t-test is a one-sample t-test on the d_i. Its null hypothesis is that the population mean difference is zero. It estimates that mean difference and gives a confidence interval for it. Its assumptions are that the pairs are independent of each other and that the differences are approximately normally distributed, or that there are enough pairs for the sampling distribution of the mean difference to be close to normal. That is the whole list. It makes no assumption about the before and after columns being normal, and none about their variances being equal. The underlying distribution is the one Student (1908) derived for the mean of a small sample.

The Wilcoxon signed-rank test ranks the absolute differences |d_i| from smallest to largest, then sums the ranks belonging to the positive differences (Wilcoxon, 1945). Its null hypothesis is that the distribution of the differences is symmetric about zero. It assumes the pairs are independent, and that the differences are measured on a scale where their sizes can be meaningfully ordered, since it ranks magnitudes and not just signs. If the differences are symmetric, rejecting the null means their median (which then equals their mean) is not zero. If they are clearly asymmetric, the test can reject even when the median difference is zero, so “the median difference is non-zero” is a claim that needs the symmetry assumption.

The natural estimate that goes with the signed-rank test is the Hodges-Lehmann estimator: the median of all pairwise averages (d_i + d_j) / 2, including each difference with itself (Hodges and Lehmann, 1963). R labels it the “(pseudo)median”. It equals the median of the differences only when their distribution is symmetric.

If the differences are only reliable in sign (who improved and who got worse, with no trust in the size), neither test fits and the sign test is the honest choice.

Worked example: a teacher training workshop

An education researcher gives 30 teachers a 40-point assessment of assessment literacy before and after a workshop. Half are novices and half are experienced, so baseline scores fall into two groups. The data are synthetic, generated in R 4.6.1.

Look at the right column

The before scores are bimodal, one peak per experience level. A Shapiro-Wilk test on them gives p = 0.005, and someone checking “normality of the data” would reach for a rank test. The differences tell a different story. They range from -1 to +6 points, with 18 of the 30 teachers gaining 1, 2 or 3, and the Shapiro-Wilk p-value for them is 0.30. The bimodality belongs to the teachers, not to the change, and the pairing removes it.

A normality test is a weak tool for this decision either way: it has little power with few pairs and flags trivial departures with many. A dot plot or histogram of the 30 differences is more informative than any p-value about them. The example is here to show which variable to look at.

Results

Both tests say the same thing: scores went up by roughly two points on a 40-point scale, and the interval excludes zero comfortably. The Wilcoxon interval is reported at 98.3% rather than 95% because the test statistic is discrete, so only certain confidence levels are exactly attainable, and R reports the level it achieved.

Here the Hodges-Lehmann estimate (2) equals the sample median of the differences (2), as expected when the differences are roughly symmetric. The two are still different estimators, since the estimate is the median of the pairwise averages of the differences, and they need not agree, particularly on integer scores where many differences tie.

For contrast, analysing the same scores as two independent groups (a Welch t-test on after versus before) gives t = 1.13, p = 0.26, and a 95% CI from -1.59 to 5.72. Ignoring the pairing throws away the fact that each teacher is compared with themselves, and the large differences between teachers swamp a consistent two-point gain.

Zeros and ties

Four teachers scored exactly the same both times, and the differences are integers, so many share a value. Both situations need a decision in the signed-rank test.

A zero difference has no sign. The conventional procedure drops zeros and ranks the rest. Pratt (1959) proposed ranking the zeros along with everything else and then leaving them out of the sum, which he argued behaves more sensibly. Software differs on this, and so can two code paths in the same package: in R 4.6.1, the exact calculation used for these results ranks the zeros (V = 437), while the large-sample approximation removes them before ranking (which would give V = 341 for these data).

Earlier versions of R could not compute an exact p-value with ties or zeros and fell back to the normal approximation with a warning. None of this changes the conclusion here, but it does change the reported statistic, which is one reason to record the software version with the result.

The paired t-test has no special handling: a zero is just a difference of zero.

Effect sizes that match the design

The raw mean difference, in the units of the scale, is usually the most useful effect size: “2.07 points on a 40-point test” is something a reader can judge. If you need a standardized measure, be explicit about which one. For these data, dividing the mean difference by the standard deviation of the differences (1.80) gives d_z = 1.15. That figure depends on how strongly before and after correlate, so it is not comparable with a Cohen’s d from a between-groups design. Lakens (2013) sets out the variants (d_z, and versions standardized by the average of the before and after standard deviations) and when each is appropriate. For the signed-rank test, the matched-pairs rank-biserial correlation is the usual companion.

Common mistakes with paired data

• Checking the wrong column. Normality and symmetry are properties of the differences, not of the before or after scores.
• Running an unpaired test. It treats 30 teachers as 60 independent people and, as the contrast above shows, can hide a consistent gain.
• Choosing the test from the p-value. Decide from the shape of the differences and the estimate you want to report, then run that one test.
• Losing track of the zeros. Report the number of pairs (30 here) and say how the four zero differences were handled, because the signed-rank statistic depends on it.
• Reporting a p-value alone. Give the estimate and its interval in the units of the scale.

Choosing between them

• The differences look roughly symmetric and the mean change is what you want to report: paired t-test. With a reasonable number of pairs, moderate skew in the differences is not a problem.
• Few pairs and clearly heavy-tailed differences, or one or two extreme changes you have no grounds to exclude: Wilcoxon signed-rank, reporting the Hodges-Lehmann estimate and its interval.
• Strongly asymmetric differences: be careful with both. The mean and the median of the differences are then different quantities; decide which one answers your question, and consider a transformation if the outcome is a ratio-type measurement.
• Three or more time points: neither. Use a repeated-measures model; see repeated-measures ANOVA or a mixed model.
• Two methods measuring the same quantity on each unit, where the question is whether they agree: a paired test checks only the average difference. See Bland-Altman agreement.

For independent groups rather than paired measurements, the corresponding comparison is t-test vs Mann-Whitney.

How Celeus handles this

When two columns look like the same measurement taken twice, such as before and after, Celeus proposes a paired t-test and checks normality on the paired differences, not on either column. If that check fails, it offers the Wilcoxon signed-rank test as a re-run, reported with the Hodges-Lehmann estimate and its confidence interval.

For the wider family of t-tests and how to choose among them, see t-tests explained.

# Results

- Test: Paired t-test
- n: 30
- statistic: 6.292
- p-value: < 0.001
- effect size: Cohen's d = 1.149 (95% CI 0.6794 to 1.606)
- 95% CI: [1.395, 2.738]
- seed: 61

## Assumption checks
- normality (Shapiro-Wilk): passed=TRUE (no evidence against normality)
- sample size: passed=TRUE (adequate n)

## Power (sensitivity, not observed power)
- minimum detectable effect at n=30: Cohen's d = 0.5292
- to detect a conventional medium effect: 34 pairs to detect d=0.50
- alpha = 0.05, target power = 0.8

# Celeus - reproduce.R  (run from this package folder; bundled data in data/)
# Test: Paired t-test  |  engine 0.49.0  |  catalog c40-dc08
# Research context: not provided for this run.
# Reproduces the recorded result deterministically.

set.seed(61)
.data_path <- "data/paired-t-test-vs-wilcoxon.csv"
if (!file.exists(.data_path)) stop("Run reproduce.R from the package folder (the bundled data is in data/).")
data <- readr::read_csv(.data_path, show_col_types = FALSE)  # needs the readr package
clean_cells <- function (df) 
{
    ctrl_collapse <- "[\r\n\t\v\f]+"
    zero_width_chars <- intToUtf8(c(8203L, 65279L), multiple = TRUE)
    uni_space_chars <- intToUtf8(c(160L, 8239L, 8199L), multiple = TRUE)
    clean_text <- function(x) {
        out <- x
        for (ch in zero_width_chars) out <- gsub(ch, "", out, 
            fixed = TRUE)
        out <- gsub(ctrl_collapse, " ", out)
        for (ch in uni_space_chars) out <- gsub(ch, " ", out, 
            fixed = TRUE)
        trimws(gsub(" +", " ", out))
    }
    changed_cols <- character(0)
    cells_cleaned <- 0L
    retyped <- character(0)
    for (j in seq_along(df)) {
        col <- df[[j]]
        if (!is.character(col)) 
            next
        cleaned <- clean_text(col)
        n_changed <- sum(!is.na(col) & !is.na(cleaned) & col != 
            cleaned, na.rm = TRUE)
        if (n_changed > 0L) {
            changed_cols <- c(changed_cols, names(df)[j])
            cells_cleaned <- cells_cleaned + n_changed
        }
        df[[j]] <- cleaned
        non_blank <- cleaned[!is.na(cleaned) & nzchar(cleaned)]
        if (length(non_blank) && all(grepl("^[+-]?(0|[1-9][0-9]*)([.][0-9]+)?$", 
            non_blank))) {
            parsed <- suppressWarnings(as.numeric(cleaned))
            was_blank <- is.na(cleaned) | !nzchar(cleaned)
            lost <- (is.na(parsed) | is.infinite(parsed)) & !was_blank
            if (!any(lost)) {
                finite <- parsed[!is.na(parsed)]
                all_whole <- all(finite == floor(finite))
                in_range <- all(abs(finite) <= .Machine$integer.max)
                df[[j]] <- if (all_whole && in_range) 
                  as.integer(parsed)
                else parsed
                retyped <- c(retyped, names(df)[j])
            }
        }
    }
    names(df) <- clean_text(names(df))
    if (length(changed_cols)) {
        attr(df, "ingest_clean_columns") <- changed_cols
        attr(df, "ingest_clean_cells_cleaned") <- cells_cleaned
    }
    if (length(retyped)) 
        attr(df, "ingest_clean_retyped") <- retyped
    df
}
data <- clean_cells(data)
# Complete-case analysis: drop rows missing any analysis value.
.acols <- c("after", "before")
data <- data[stats::complete.cases(data[, .acols, drop = FALSE]), , drop = FALSE]
result <- t.test(data[["after"]], data[["before"]], paired = TRUE)
print(result)

# Sensitivity: minimum detectable effect at n=30 (power=0.8, alpha=0.05). NOT observed power.
power <- pwr::pwr.t.test(n = 30, sig.level = 0.05, power = 0.8, type = "paired")
print(power)