Pretest posttest design: change scores or ANCOVA?

A pretest posttest design measures the same outcome on every participant before and after an intervention, usually in two groups: one that receives the intervention and one that does not. There are three common ways to analyse it, and they do not give the same answer. For two randomised groups with one pre-test and one post-test, the usual first choice is ANCOVA: analyse the post-test score with the pre-test score as a covariate. Change scores are a defensible but usually less precise alternative, and a mixed model is the better tool when follow-up scores are missing or there are more than two time points.

This post is for the decision itself: which analysis fits your design, why, and what to write in your methods section. It works one example in full, an ANCOVA run through a deterministic engine on simulated pre-test and post-test scores, and shows the R code for the other two analyses so you can run all three on the same data. The ANCOVA guide explains the method’s mechanics in more depth; this post sits one step earlier, at the choice.

First, which pretest posttest design do you have?

The analysis follows from the design, so answer three questions before choosing anything.

Is there a comparison group? With one group measured twice and no control, the only available comparison is post against pre within the same people. A paired t-test (or the Wilcoxon signed-rank test) estimates the mean change, and the paired t-test or Wilcoxon guide covers that choice. What it cannot do is attribute the change to the intervention, because anything else that happened between the two measurements appears in the same number. Everything below assumes two or more groups.

Were participants randomised to the groups? With randomisation, any baseline difference between the groups is chance, and every reasonable analysis estimates the same quantity; they differ in precision. Without randomisation, as with intact classes, two hospital wards or volunteers against non-volunteers, the analyses make different assumptions about what would have happened without the intervention, and they can disagree. That case has its own section below.

How many time points, and is anyone missing? One pre-test and one post-test with complete data is the classic case. More follow-up measurements, or participants who dropped out before the post-test, point towards a mixed model.

Why a control group matters. In a one-group design, the classic threats (Campbell and Stanley, 1963) of history (events outside the study), maturation (people changing over time), testing (a second sitting of the same test), instrumentation (the measure or rater changing) and regression to the mean all sit inside the pre-to-post change. A comparison group that is measured on the same schedule meets the same influences, so the difference between groups removes them, provided the groups are comparable. Dropout related to the outcome threatens both designs, and more so when it differs between groups. A control group measured only at post-test gives up the baseline, so you cannot check how alike the groups were at the start. The one-group design remains an option when a control group is impractical or unethical; report its change as descriptive.

If your study has a different shape altogether, such as several factors or a binary outcome, the guide to choosing a statistical test starts from the outcome type, and the thesis statistics guide maps the designs supervisors see most often.

Three ways to analyse two groups measured twice

Suppose each participant has a pre-test score, a post-test score and a group. Each of the three analyses below compares the groups, and each one treats the pre-test differently.

1. Post-test only. Compare the post-test means with a two-sample t-test and ignore the pre-test. This is unbiased under randomisation, but it throws away information: participants who started high tend to finish high, and that between-person spread stays in the error term.

2. Change scores. Compute post minus pre for each participant and compare the mean change between groups with a two-sample t-test. This removes the between-person spread, but it assumes that one point on the pre-test is worth exactly one point on the post-test. In regression terms, it fixes the slope of post-test on pre-test at 1.

3. ANCOVA. Fit a linear model with the post-test as the outcome, the group as a factor and the pre-test as a covariate. The model estimates the slope of post-test on pre-test from the data rather than assuming it, and compares the groups at the same pre-test score. The group effect is the difference between the two adjusted means.

A fourth analysis is often reported but is not really a fourth option. A repeated-measures ANOVA with group as a between-subjects factor and time (pre, post) as a within-subjects factor gives a group-by-time interaction. With two time points, that interaction tests exactly what a pooled-variance t-test on the change scores tests, and its F statistic is the square of that t. It is a change-score analysis with a different printout, and it inherits that analysis’s strengths and weaknesses. The repeated-measures ANOVA or mixed model guide covers it for designs with more time points.

Why ANCOVA is usually the most precise choice

The three analyses differ in how much noise they leave in the comparison, and that depends on one quantity: the correlation between pre-test and post-test scores within a group.

Assume the pre-test and post-test have similar spread, and call their within-group correlation r. Relative to the post-test-only analysis, the variance of the group difference is multiplied by 2(1 - r) for change scores and by (1 - r²) for ANCOVA. Frison and Pocock worked through these relations.

• When r is low, subtracting the pre-test adds noise rather than removing it, and change scores are less precise than simply comparing post-test scores. The break-even point is r = 0.5, where change scores and post-test only are equally precise.
• When r is high, change scores beat post-test only, which is the intuition behind them.
• In large samples, and when both arms share one pre-post slope and one residual variance, ANCOVA is at least as precise as the better of the two at every value of r, because it estimates the slope instead of fixing it at 0 (post-test only) or 1 (change scores). When r is close to 1 the three nearly coincide.

Vickers and Altman summarised the practical result for randomised trials: ANCOVA generally has greater statistical power to detect a treatment effect than either alternative. Van Breukelen reached the same conclusion: in randomised studies both ANCOVA and change scores are unbiased, but ANCOVA has more power.

The second reason is baseline imbalance. Randomisation balances groups on average, not in every trial, and in a thesis-sized sample the groups can easily start a few points apart by chance. Because pre-test scores are usually negatively correlated with change, a group that happens to start low will tend to show larger change even with no intervention, which is regression to the mean. Vickers and Altman point out that a follow-up-only analysis then underestimates the effect in a group that started worse, and a change-score analysis overestimates it. ANCOVA compares the groups at the same pre-test score, so it adjusts for the chance imbalance instead of being pushed around by it.

One caution: the precision argument is about the analysis plan, not your result. Choose the analysis in your plan before you see the data, and do not run all three and report whichever gives the smallest p-value.

Worked example: ANCOVA on simulated pre and post scores

The example uses synthetic data: 80 simulated participants randomised into a control arm and an intervention arm of 40 each, with a pre-test and a post-test score for every participant. The numbers describe no real study or person, and the example is about the method, not about any intervention. The research question is whether the mean post-test score differs between the arms after adjusting for the pre-test score.

Seeded and run in R; the method, ANCOVA with the post-test as the outcome, the arm as the group and the pre-test as the covariate, was chosen before the run.

The result

Quantity	Value
Participants analysed	80 (40 per arm, no missing values)
Adjusted mean post-test, control (95% CI)	51.26 (49.05 to 53.48)
Adjusted mean post-test, intervention (95% CI)	56.86 (54.65 to 59.07)
Group F test, adjusted for pre-test	F(1, 77) = 12.62, p < 0.001
Partial eta-squared for the group (95% CI)	0.1408 (0.02853 to 0.288)
Homogeneity of regression slopes	F = 1.015, p = 0.317
Normality of residuals (Shapiro-Wilk)	W = 0.9905, p = 0.8238

Reading it

Start with the estimate. The adjusted means are the predicted post-test scores for each arm at the overall mean pre-test score. The intervention arm’s adjusted mean is 56.86 and the control arm’s is 51.26, a difference of about 5.6 points (56.86 minus 51.26). That difference is the answer to the research question: how far apart the arms are at the same starting score.

Then the uncertainty. The engine does not print an interval for the difference itself, and per-group intervals are not a substitute for one. Add one line to the reproduce script to get it, pairs(emmeans::emmeans(.fit, "arm"), reverse = TRUE) with confint(), and report that interval next to the 5.6.

Then the test. The group F on 1 and 77 degrees of freedom is 12.62, with a p-value below 0.001 (0.00065 in the full output). With only two groups, this F is the square of the t statistic for the adjusted difference, so it tests the same thing. The data are hard to reconcile with no difference between the arms at the same pre-test score.

Then the effect size. Partial eta-squared is 0.1408: the group accounts for about 14% of the post-test variation that remains after the pre-test is taken into account. Because the pre-test has already removed much of the variation, it is larger than the same difference would give in a post-test-only ANOVA, so do not compare it directly with an unadjusted eta-squared from another study. Its 95% interval, 0.02853 to 0.288, is wide: the data are compatible with a small effect and with a large one. For a two-group design, the adjusted mean difference in the outcome’s own units, with its interval, is usually the more useful number for readers.

The assumption checks

Homogeneity of regression slopes: not detected as different. The engine added a pre-test-by-arm interaction and tested it: F = 1.015, p = 0.317. That means a difference in slopes between the arms was not detected, not that the slopes are equal; a test of an interaction has limited power at this sample size. A scatter plot of post against pre by arm is the better check. If the slopes clearly differ, a single adjusted difference averages over a gap that changes with the starting score, and the ANCOVA guide explains what to report instead.

Normality of residuals: no departure detected. Shapiro-Wilk on the model’s residuals gave W = 0.9905, p = 0.8238. As with any non-significant assumption test, that is an absence of detected departure, not evidence that the residuals are normal. Look at a residual plot as well; with 40 per arm and no extreme values, moderate departures would matter little for the group comparison.

Two assumptions no test can check. The pre-test must be measured before the intervention could affect it, which is a fact about the design. And the comparison is causal only because allocation was random. Both hold here by construction; in your study, state them.

The same data, three ways, in R

The script below the article reproduces the ANCOVA exactly. To see the other two analyses on the same data, add these lines to it after data is loaded. Only the ANCOVA was run through the engine, so run them and compare the estimates yourself.

# Change scores: post minus pre, compared between arms with Welch's t-test
data$change <- data$post - data$pre
t.test(change ~ arm, data = data)

# ANCOVA: the adjusted difference (intervention minus control) and its interval
fit <- lm(post ~ pre + arm, data = data)
confint(pairs(emmeans::emmeans(fit, "arm"), reverse = TRUE))

# Mixed model on long data: one row per participant per time point.
# The arm effect is allowed only at the post-test, so both arms share one
# baseline mean (the constrained form recommended for randomised designs).
data$id <- seq_len(nrow(data))
long <- data.frame(
  id    = rep(data$id, 2),
  arm   = rep(data$arm, 2),
  time  = rep(c("pre", "post"), each = nrow(data)),
  score = c(data$pre, data$post)
)
long$time <- factor(long$time, levels = c("pre", "post"))
long$arm_at_post <- as.numeric(long$time == "post" & long$arm == "intervention")
mm <- nlme::gls(score ~ time + arm_at_post, data = long,
                correlation = nlme::corSymm(form = ~ as.integer(time) | id),
                weights = nlme::varIdent(form = ~ 1 | time))
summary(mm)$tTable["arm_at_post", ]

With complete data, the constrained mixed model should land very close to ANCOVA. In the trial Coffman and colleagues analysed, the constrained mixed model and ANCOVA gave equivalent intervals on the complete data, and the mixed model could also use the participants with incomplete data, which ANCOVA cannot.

When a mixed model is the better tool

ANCOVA needs one row per participant with both scores present. A participant missing the post-test drops out entirely, and so does anyone missing the pre-test. A mixed model, fitted to the long data with one row per measurement, uses every observed score, and under the assumption that missingness depends only on what was observed, it gives valid estimates without discarding the incomplete cases.

Two forms are common, and the difference is the same as above.

• The constrained form (sometimes called constrained longitudinal data analysis) assumes both arms share one baseline mean, which randomisation justifies, and lets the arms differ only after the intervention. With complete data it behaves like ANCOVA. This is the form in the code above.
• The unconstrained form gives each arm its own baseline mean and tests the group-by-time interaction. With two time points that is the change-score analysis again, and Twisk and colleagues show that it does not adjust for baseline differences between the groups. Their recommendation for randomised trials is ANCOVA, or a repeated-measures model that includes the treatment-by-time interaction but no separate treatment term, which is the constrained form.

A mixed model is also the natural choice with several follow-up measurements, where the question becomes how the arms differ over time rather than at one post-test. The engine supports linear mixed models, and its mixed model reference page lists the specification it accepts.

When the groups were not randomised

Many thesis studies compare intact groups: two classes, two clinics, a programme’s participants against non-participants. Here the choice between ANCOVA and change scores is no longer a question of precision. Each one encodes a different assumption about how the groups would have developed without the intervention.

• Change scores assume that, without the intervention, both groups would have changed by the same amount from where they started.
• ANCOVA assumes that participants with the same pre-test score would have ended at the same post-test score, whichever group they were in.

When the groups differ at baseline, these assumptions cannot both hold, and the two analyses can give different answers, even in direction. This is Lord’s paradox, named after the 1967 paper that posed it. Van Breukelen showed when each analysis is biased: if allocation depended on the pre-test score itself, only ANCOVA is unbiased; with pre-existing groups that differ at baseline, change scores seem less biased than ANCOVA, though neither is guaranteed. Senn adds that baseline equality is not a strict requirement for ANCOVA to be unbiased, but that it is hard to find circumstances where ANCOVA is biased and change scores are not.

For a non-randomised thesis, the honest approach is to name the assumption you rely on, justify it from how the groups were formed, consider reporting both analyses as a sensitivity check, and describe the result as an association rather than the effect of the intervention.

How to report it

For the worked example, a results sentence would read:

Post-test scores were compared between arms with ANCOVA, adjusting for the pre-test score. Adjusted mean post-test scores were 56.86 (95% CI 54.65 to 59.07) in the intervention arm and 51.26 (95% CI 49.05 to 53.48) in the control arm, F(1, 77) = 12.62, p < 0.001, partial eta-squared = 0.14 (95% CI 0.03 to 0.29). A pre-test-by-arm interaction was not detected (p = 0.317).

In your own report, add the adjusted difference with its interval from the line shown above, name the covariate and say that it was chosen before the data were seen, and state how missing scores were handled.

Common mistakes

Separate paired t-tests in each group. Testing pre against post within the intervention group, then within the control group, and concluding that the intervention worked because only one test was significant is invalid. Bland and Altman show that this procedure is biased and that its false positive rate can reach 0.50 with two groups. Compare the groups directly.

Testing whether randomised groups differ at baseline, then choosing the analysis. In a randomised design, any baseline difference is chance by definition, so a significance test of it answers no useful question. Plan ANCOVA in advance and it handles the imbalance whatever its size.

Adjusting for a covariate measured after the intervention started. A mid-study measurement can be changed by the intervention itself, and adjusting for it removes part of the effect you are trying to estimate.

Percentage change as the outcome. In Vickers’s simulations of a randomised trial, percentage change had the lowest power of the four analyses compared and was highly sensitive to changes in variance; it also behaves badly when baseline scores are near zero. Use ANCOVA, and if readers want a percentage, convert its estimate using the mean baseline score.

Calling a non-significant result “no effect”. A p-value above 0.05 means the data are compatible with no difference, not that the difference is zero. The t-tests explained post shows how to read the interval and a sensitivity figure in that situation.

Rerunning the example

Below the article you will find the R script exactly as the engine generated it, with the seed and the bundled data path, and the output it printed. On the same package versions the script reproduces the ANCOVA table, the adjusted means and the partial eta-squared with its interval. It does not rerun the two assumption checks, which come from the engine; a pre:arm interaction model with anova() and shapiro.test(residuals(fit)) reproduce them. The guide to reproducible statistical analysis explains what a reviewer needs in order to rerun a result like this one, and the ANCOVA method reference lists what the engine accepts.

# Results

- Test: ANCOVA (analysis of covariance)
- n: 80
- statistic: 12.62
- p-value: < 0.001
- effect size: partial eta-squared = 0.1408 (95% CI 0.02853 to 0.288)
  (two-sided CI at the run's confidence level)
- seed: 20261021

## Assumption checks
- homogeneity of regression slopes: passed=TRUE (no evidence a covariate's effect differs by group; the common-slope ANCOVA model is appropriate)
- normality (Shapiro-Wilk): passed=TRUE (no evidence against normality)

## Power
- Power is not calculated for this test.

# Celeus - reproduce.R  (run from this package folder; bundled data in data/)
# Test: ANCOVA (analysis of covariance)  |  engine 0.30.0  |  catalog c39-d3cd
# Research context: not provided for this run.
# Reproduces the recorded result deterministically.

set.seed(20261021)
.data_path <- "data/pretest-posttest-design.csv"
if (!file.exists(.data_path)) stop("Run reproduce.R from the package folder (the bundled data is in data/).")
data <- readr::read_csv(.data_path, show_col_types = FALSE)  # needs the readr package
clean_cells <- function (df) 
{
    ctrl_collapse <- "[\r\n\t\v\f]+"
    zero_width_chars <- intToUtf8(c(8203L, 65279L), multiple = TRUE)
    uni_space_chars <- intToUtf8(c(160L, 8239L, 8199L), multiple = TRUE)
    clean_text <- function(x) {
        out <- x
        for (ch in zero_width_chars) out <- gsub(ch, "", out, 
            fixed = TRUE)
        out <- gsub(ctrl_collapse, " ", out)
        for (ch in uni_space_chars) out <- gsub(ch, " ", out, 
            fixed = TRUE)
        trimws(gsub(" +", " ", out))
    }
    changed_cols <- character(0)
    cells_cleaned <- 0L
    retyped <- character(0)
    for (j in seq_along(df)) {
        col <- df[[j]]
        if (!is.character(col)) 
            next
        cleaned <- clean_text(col)
        n_changed <- sum(!is.na(col) & !is.na(cleaned) & col != 
            cleaned, na.rm = TRUE)
        if (n_changed > 0L) {
            changed_cols <- c(changed_cols, names(df)[j])
            cells_cleaned <- cells_cleaned + n_changed
        }
        df[[j]] <- cleaned
        non_blank <- cleaned[!is.na(cleaned) & nzchar(cleaned)]
        if (length(non_blank) && all(grepl("^[+-]?(0|[1-9][0-9]*)([.][0-9]+)?$", 
            non_blank))) {
            parsed <- suppressWarnings(as.numeric(cleaned))
            was_blank <- is.na(cleaned) | !nzchar(cleaned)
            lost <- (is.na(parsed) | is.infinite(parsed)) & !was_blank
            if (!any(lost)) {
                finite <- parsed[!is.na(parsed)]
                all_whole <- all(finite == floor(finite))
                in_range <- all(abs(finite) <= .Machine$integer.max)
                df[[j]] <- if (all_whole && in_range) 
                  as.integer(parsed)
                else parsed
                retyped <- c(retyped, names(df)[j])
            }
        }
    }
    names(df) <- clean_text(names(df))
    if (length(changed_cols)) {
        attr(df, "ingest_clean_columns") <- changed_cols
        attr(df, "ingest_clean_cells_cleaned") <- cells_cleaned
    }
    if (length(retyped)) 
        attr(df, "ingest_clean_retyped") <- retyped
    df
}
data <- clean_cells(data)
# Complete-case analysis: drop rows missing any analysis value.
.acols <- c("post", "arm", "pre")
data <- data[stats::complete.cases(data[, .acols, drop = FALSE]), , drop = FALSE]
result <- { .d <- data; .d[["arm"]] <- as.factor(.d[["arm"]]); .fit <- stats::lm(`post` ~ `pre` + `arm`, data = .d); .an <- stats::anova(.fit); .em <- emmeans::emmeans(.fit, specs = "arm"); list(anova = .an, emmeans = summary(.em)) }
print(result)

# Effect size (partial_eta_sq) and its confidence interval.
effect <- { .es_d <- data; .es_d[["arm"]] <- as.factor(.es_d[["arm"]]); .es_an <- stats::anova(stats::lm(`post` ~ `pre` + `arm`, data = .es_d)); effectsize::F_to_eta2(.es_an[["F value"]][nrow(.es_an) - 1L], .es_an[["Df"]][nrow(.es_an) - 1L], .es_an[["Df"]][nrow(.es_an)], ci = 0.95, alternative = "two.sided") }
print(effect)