Matched pairs design: when to match, how to analyse

A matched pairs design compares two conditions within pairs of similar units, or within the same unit measured under both, so that differences between units cancel out of the comparison. Analyse it on the within-pair differences, with a paired t-test or a Wilcoxon signed-rank test, rather than as two independent groups, unless an unmatched analysis was the plan before the data arrived.

The sections below cover when matching is worth the effort, which variables to match on, and how to analyse the result. The worked example runs one public dataset, R’s sleep data, through a deterministic engine three ways: the paired t-test the design calls for, the Wilcoxon signed-rank test on the same pairs, and a two-sample test that ignores the pairs, so the cost of the wrong analysis is measured rather than asserted. For where this design sits among the other common thesis designs, see the design-first guide to dissertation statistics.

What a matched pairs design is

A matched pairs design has two conditions and units arranged in pairs before either condition is applied. Within each pair, one member gets each condition. It comes in two forms.

Two different units, matched. Participants are grouped into pairs that are alike on something that predicts the outcome, such as prior grade, age or baseline score. Within each pair, a random draw decides who gets which condition. Twins are the version nature supplies.

The same unit under both conditions. Each person, animal or specimen receives both conditions, so each unit is its own match. Here the randomisation is of the order in which the conditions are given, because a time trend or an effect that carries over from the first condition can otherwise bias every within-person difference.

When the condition is randomised within each pair, both forms are, in effect, a randomised block design with blocks of two. The randomisation inside each pair is what keeps the comparison fair; matching only makes it more precise. Matching cases to controls in an observational study keeps the pairs in the analysis but not that guarantee, because no one assigned the conditions.

When matching helps, and when it does not

Matching helps in proportion to how well the matching variable predicts the outcome. When pair members resemble each other strongly on the outcome, the within-pair differences vary far less than the raw values, and the paired comparison is much more precise. The post on paired and unpaired designs sets out the variance algebra behind this.

When the matching variable barely predicts the outcome, the gain disappears and a cost remains. A paired t-test on n pairs has n - 1 degrees of freedom, against 2n - 2 for a two-sample test on the same units, so ineffective matching can leave you slightly worse off with few pairs. Matching also slows recruitment, excludes units with no suitable partner, and loses the whole pair when one member drops out.

Four rules keep matching useful:

• Match on variables that predict the outcome, measured before any condition is applied. A baseline measurement of the outcome itself is often the best one.
• Never match on anything the condition could change, and never on the outcome.
• Match on few variables. Each added variable makes partners harder to find and excludes more units.
• Fix the analysis before the data arrive. Diehr and colleagues (1995) studied when an unmatched analysis of a matched study with few pairs can be the better choice. Whatever rule you follow, write it into the analysis plan, and never pick the analysis by comparing the two p-values.

The design also fixes how the data must be stored: one identifier per pair, recorded at the time of matching. Without it the pairs cannot be rebuilt, and the analysis below is impossible.

Worked example: R’s sleep data

The sleep dataset ships with R and holds the data Student used as an example in his 1908 paper, the origin of the t-test. It records, for each of ten people, the extra hours of sleep compared with control under each of two drugs, originally reported by Cushny and Peebles in 1905. Every person took both drugs, so this is the self-paired form of the design, and the pairs are people.

The data are stored in long form, one row per person per drug, with a column called group for the drug. R’s own documentation warns that the group name may be misleading: the values are measurements on ten persons, not two groups. For the paired tests, the data were reshaped to one row per person, with columns extra_1 and extra_2, after checking that every person has exactly one value under each drug. The R script below the article reads that reshaped table. These are historical data used to illustrate a design, not evidence about either drug.

The paired t-test was chosen from the design before the run.

Under drug 2 these ten people slept 1.58 hours more on average than under drug 1. The 95% interval runs from 0.7001 to 2.46 hours, so the data are compatible with a mean difference anywhere in that range, and the interval excludes zero. The effect size is d_z, the mean difference divided by the standard deviation of the differences.

Checking the differences, and the Wilcoxon signed-rank test

The paired t-test assumes that the within-pair differences, not the raw values, are roughly normal. The engine ran Shapiro-Wilk on the ten differences: W = 0.830, p = 0.0333, evidence of non-normality. On that basis the run flagged the result and suggested the Wilcoxon signed-rank test as the alternative. With ten pairs a normality test has little power, so a pass would not have proved normality either.

The signed-rank test, from Wilcoxon’s 1945 paper, ranks the absolute differences and compares the ranks of positive and negative ones. On the same pairs it gave V = 54 and an exact two-sided p = 0.00391. Its estimate is the Hodges-Lehmann estimate (the median of all pairwise averages of the differences), 1.3 hours, with an interval from 0.9 to 2.7 hours. Because a rank test’s null distribution is discrete, the interval’s achieved level is 95.6% rather than exactly 95%, and the engine reports that level.

One person had an identical value under both drugs, a zero difference with no sign, which the engine records. Software differs in how it handles a zero difference, so another R version or another package can report a different V, p-value, Hodges-Lehmann estimate and interval; name the software and version. The rank-biserial correlation is 1, with an interval of 1 to 1: every non-zero difference points the same way, so no one slept less under drug 2. That interval is degenerate, not precise.

The two tests agree. Treat the Wilcoxon run as the sensitivity analysis it is, planned alongside the t-test, not as a second try at significance. The guide to the paired t-test or Wilcoxon signed-rank test explains what each one tests and how zeros and ties are handled.

What analysing the pairs as independent groups costs

The same twenty values were also run, unchanged and in long form, through Welch’s two-sample t-test, the usual test for comparing two independent groups. This is the analysis the misleading group column invites, and it is wrong for these data. The same cost appears when a test is picked from the column layout without the design; see how an AI picks a statistical test.

The estimate is identical, apart from the direction of subtraction, but the Welch interval is far wider and includes zero, and the p-value rises from 0.00283 to 0.0794. The data did not change; only the assumption about how they were collected did.

The reason is the shared between-person variation: people who gained a lot of sleep under one drug tended to gain a lot under the other, and Welch’s test leaves that in its standard error.

Welch’s non-significant p-value of 0.0794 is not evidence that the drugs perform the same, because it treats ten people as twenty unrelated ones; why an unpaired analysis of paired data is not the safe choice is set out in the post on paired and unpaired designs. The table’s two Cohen’s d values are not interchangeable either: d_z divides by the spread of the differences, the two-sample d by the spread of the raw values. Lakens (2013) sets out the options for paired designs; name the one you report.

Reporting a matched pairs analysis

Say how the pairs were formed, then give the estimate with its interval, the test, and the effect size with its denominator. For the example:

Extra sleep was compared between two drugs taken by the same 10 people. The mean difference (drug 2 minus drug 1) was 1.58 hours (95% CI 0.7001 to 2.46), paired t(9) = 4.062, p = 0.00283, Cohen’s d_z = 1.285 (95% CI 0.4146 to 2.118). The differences showed evidence of non-normality (Shapiro-Wilk p = 0.0333); a Wilcoxon signed-rank test, planned as a sensitivity analysis, agreed (V = 54, p = 0.00391; Hodges-Lehmann estimate 1.3 hours, 95.6% CI 0.9 to 2.7).

Add the number of pairs lost to a missing partner, and how the order of conditions was assigned in a self-paired design. The sleep data do not record the order, so a period effect cannot be checked here.

State what the design could detect, not the power observed after the fact. For the 10 sleep pairs, the engine’s sensitivity figure is d_z = 0.996 at 80% power and the 0.05 level (34 pairs for d_z = 0.50). For a matched design the useful point is that d_z is measured against the spread of the differences, so better matching lets a smaller raw difference reach the same d_z.

For a pass or fail outcome in matched pairs, use McNemar’s test. Three or more conditions per unit make the design repeated measures: see the guide to repeated-measures ANOVA or a mixed model. A before-and-after study with no control group is paired too, with its own threats, set out in the post on pretest-posttest designs.

Rerunning the example

Below the article are the engine’s R script, with its seed and data path, and its output. The script reads the reshaped data, one row per person with columns extra_1 and extra_2, and on the same package versions reproduces the paired t-test and the sensitivity power figures. It does not recompute d_z with its interval or the Shapiro-Wilk check, and it needs R with the readr and pwr packages.

The Wilcoxon and Welch results came from separate engine runs on the same data; in R they are wilcox.test(data$extra_2, data$extra_1, paired = TRUE, conf.int = TRUE) on the reshaped data (the engine used R 4.6.1; another R version can give a different V, p-value, estimate and interval because of the zero difference) and t.test(extra ~ group, data = sleep) on the data as shipped. The reference page for the paired t-test describes the method as the engine runs it, and the guide to reproducible statistical analysis explains what a reader needs to rerun a result.

# Results

- Test: Paired t-test
- n: 10
- statistic: 4.062
- p-value: 0.00283
- effect size: Cohen's d = 1.285 (95% CI 0.4146 to 2.118)
- 95% CI: [0.7001, 2.46]
- seed: 20261101

## Assumption checks
- normality (Shapiro-Wilk): passed=FALSE (evidence of non-normality)
- sample size: passed=TRUE (adequate n)

## Power (sensitivity, not observed power)
- minimum detectable effect at n=10: Cohen's d = 0.996 (severely underpowered)
- to detect a conventional medium effect: 34 pairs to detect d=0.50
- alpha = 0.05, target power = 0.8

# Celeus - reproduce.R  (run from this package folder; bundled data in data/)
# Test: Paired t-test  |  engine 0.41.7  |  catalog c40-dc08
# Research context: not provided for this run.
# Reproduces the recorded result deterministically.

set.seed(20261101)
.data_path <- "data/matched-pairs-design.csv"
if (!file.exists(.data_path)) stop("Run reproduce.R from the package folder (the bundled data is in data/).")
data <- readr::read_csv(.data_path, show_col_types = FALSE)  # needs the readr package
clean_cells <- function (df) 
{
    ctrl_collapse <- "[\r\n\t\v\f]+"
    zero_width_chars <- intToUtf8(c(8203L, 65279L), multiple = TRUE)
    uni_space_chars <- intToUtf8(c(160L, 8239L, 8199L), multiple = TRUE)
    clean_text <- function(x) {
        out <- x
        for (ch in zero_width_chars) out <- gsub(ch, "", out, 
            fixed = TRUE)
        out <- gsub(ctrl_collapse, " ", out)
        for (ch in uni_space_chars) out <- gsub(ch, " ", out, 
            fixed = TRUE)
        trimws(gsub(" +", " ", out))
    }
    changed_cols <- character(0)
    cells_cleaned <- 0L
    retyped <- character(0)
    for (j in seq_along(df)) {
        col <- df[[j]]
        if (!is.character(col)) 
            next
        cleaned <- clean_text(col)
        n_changed <- sum(!is.na(col) & !is.na(cleaned) & col != 
            cleaned, na.rm = TRUE)
        if (n_changed > 0L) {
            changed_cols <- c(changed_cols, names(df)[j])
            cells_cleaned <- cells_cleaned + n_changed
        }
        df[[j]] <- cleaned
        non_blank <- cleaned[!is.na(cleaned) & nzchar(cleaned)]
        if (length(non_blank) && all(grepl("^[+-]?(0|[1-9][0-9]*)([.][0-9]+)?$", 
            non_blank))) {
            parsed <- suppressWarnings(as.numeric(cleaned))
            was_blank <- is.na(cleaned) | !nzchar(cleaned)
            lost <- (is.na(parsed) | is.infinite(parsed)) & !was_blank
            if (!any(lost)) {
                finite <- parsed[!is.na(parsed)]
                all_whole <- all(finite == floor(finite))
                in_range <- all(abs(finite) <= .Machine$integer.max)
                df[[j]] <- if (all_whole && in_range) 
                  as.integer(parsed)
                else parsed
                retyped <- c(retyped, names(df)[j])
            }
        }
    }
    names(df) <- clean_text(names(df))
    if (length(changed_cols)) {
        attr(df, "ingest_clean_columns") <- changed_cols
        attr(df, "ingest_clean_cells_cleaned") <- cells_cleaned
    }
    if (length(retyped)) 
        attr(df, "ingest_clean_retyped") <- retyped
    df
}
data <- clean_cells(data)
# Complete-case analysis: drop rows missing any analysis value.
.acols <- c("extra_2", "extra_1")
data <- data[stats::complete.cases(data[, .acols, drop = FALSE]), , drop = FALSE]
result <- t.test(data[["extra_2"]], data[["extra_1"]], paired = TRUE)
print(result)

# Sensitivity: minimum detectable effect at n=10 (power=0.8, alpha=0.05). NOT observed power.
power <- pwr::pwr.t.test(n = 10, sig.level = 0.05, power = 0.8, type = "paired")
print(power)