Multiple imputation vs complete-case analysis

Multiple imputation is not automatically better than complete-case analysis, and complete-case analysis is not automatically biased. Which one is right depends on why the values are missing and on what the imputation model knows that the regression does not. Here is the same regression fitted both ways on R’s airquality data, where the two answers turn out to be close, and an account of why.

A common rule of thumb is: complete-case analysis is acceptable only if data are missing completely at random, otherwise impute. That rule is too blunt. It sends people to imputation when it adds nothing, and it leaves them unable to say when dropping rows is actually safe. The data-cleaning checklist has a step on missing data that covers counting and diagnosing the gaps first; this post is about the choice that follows.

The two approaches in one paragraph each

Complete-case analysis, also called listwise deletion, keeps only the rows that have a value for every variable in the model. It is what R’s lm() does by default (its na.action is na.omit), and it is the engine’s default too. Its cost is sample size. Its risk is bias, when the rows that survive are not a fair sample of all the rows.

Multiple imputation fills each missing value several times. A model of the observed data supplies the plausible values, so the m completed datasets differ in the filled cells. The analysis is run on every completed dataset, and the m sets of estimates are combined by Rubin’s rules: the pooled estimate is the average of the m estimates, and its variance adds the average within-dataset variance to the spread between datasets, so uncertainty about the missing values is carried into the standard error. Sterne and colleagues (2009) give a practical account.

Three kinds of missing, and what each does to each method

Rubin (1976) separated three cases, and the whole decision rests on them.

The catch, as Sterne and colleagues stress, is that you cannot tell MAR from MNAR using the observed data alone. MAR is an assumption you defend from knowledge of how the data were collected, never something a test proves.

The exception that matters most for regression

The table says complete-case analysis “can be biased” under MAR. In regression there is a precise condition under which it is not. If the chance that a row is complete depends on the predictors, but not on the outcome once the predictors are accounted for, the complete rows still estimate the same regression coefficients (Little 1992; White and Carlin 2010). Deleting rows that are unusual in their predictors changes who is in the sample, not the relationship being estimated.

The predictors-only condition is worth checking before reaching for imputation. A related result by Arel-Bundock and Pelc (2018) argues the same direction: if you can control for what determines missingness in the model, multiple imputation offers little bias reduction over listwise deletion for regression estimates.

The condition fails when missingness depends on the outcome itself, for example when sicker participants skip the follow-up visit and the outcome is the thing measured at follow-up. Then the complete rows are selected on the outcome, and the coefficients shift.

The example: ozone on temperature and wind

The data are the airquality dataset shipped with R: 153 daily readings from New York, May to September 1973 (Chambers and colleagues, 1983). The question is how daily ozone, in parts per billion, changes with temperature in degrees Fahrenheit and wind speed in miles per hour. It is a linear regression of Ozone on Temp and Wind, and the reference page for the linear regression method lists what the engine reports.

Ozone is missing on 37 of the 153 days, about 24 percent. Temperature and wind are never missing. Solar radiation has 7 missing days, but it is not in the model, so those days are not dropped. Counted in base R from the same dataset, the missing ozone days are not spread evenly: 21 of the 37 fall in June, and the monthly counts are 5, 21, 5, 5 and 1 from May to September.

A check against the model’s own predictors is more informative than the calendar. Compare mean temperature and mean wind between days with and without an ozone value, with a Welch t-test for each, in base R:

miss <- is.na(airquality$Ozone)
tapply(airquality$Temp, miss, mean)
tapply(airquality$Wind, miss, mean)
t.test(Temp ~ miss, data = airquality)$p.value
t.test(Wind ~ miss, data = airquality)$p.value

   FALSE     TRUE 
77.87069 77.91892 
    FALSE      TRUE 
 9.862069 10.256757 
[1] 0.9786832
[1] 0.5446224

The means are close on both variables, and neither difference is distinguishable from chance: the two Welch p-values are the last two lines of that output. That fits, though it does not prove, a story in which ozone readings are missing for reasons unrelated to temperature and wind.

Fit one: complete cases

With no missing-data setting chosen, the engine drops the 37 incomplete days and fits the model to the remaining 116. The numbers below are from the engine’s own output.

In the complete-case fit, each extra degree goes with about 1.84 more ozone, and each extra mile per hour of wind with about 3.06 less, holding the other fixed. Together the two variables explain about 57 percent of the variation in ozone among the days used. The engine also flagged that the residuals look non-normal and that seven of the 116 days have a Cook’s distance above the conventional 4/n screening threshold. Those are worth a look before leaning on the p-values, but they are not about the missing data. Since the multiple-imputation run fits the same model, the residual and influence questions apply to it too.

Fit two: multiple imputation

The second run uses the same data, the same model and the same seed, with the engine’s multiple-imputation setting. It builds five completed datasets, with m = 5 the engine’s fixed setting, fits the regression to each and pools the five fits by Rubin’s rules. The engine’s imputation model contains only the outcome and the two predictors, with the imputation method left at the mice package default, which for a numeric column is predictive mean matching (van Buuren and Groothuis-Oudshoorn 2011). These numbers come from that second engine run; only the complete-case run is shown below the article.

Side by side

The two fits tell the same story. The temperature coefficients differ by about two hundredths. The wind coefficients differ by about 0.2, well under one standard error. Every interval overlaps the other almost entirely, and every p-value is small in both fits. Neither coefficient would change a conclusion.

Imputation did not shrink the standard errors by recovering 37 days. The wind standard error is essentially unchanged, and the temperature one is only slightly smaller. That is the honest result, and the reason is the next section.

Why imputation gained so little here

The missing values are all in the outcome, and the imputation model uses nothing except the outcome and the same two predictors that are in the regression. Every imputed ozone value is therefore a prediction from temperature and wind, drawn with some noise. It carries no information that the 116 observed days did not already give about how ozone relates to temperature and wind. Rubin’s rules then correctly widen the uncertainty for the guessing, which cancels the apparent gain in sample size. Little (1992) and von Hippel (2007) discuss this situation: with a missing outcome and no extra variables, imputing the outcome adds noise, not information.

When imputation does earn its place is when it can use something complete-case analysis cannot. That means variables that predict the missing values or the missingness but are not in the regression, such as a second measurement of ozone, or a site or season marker. Solar radiation and month are candidates in this dataset, since 21 of the missing ozone days are in June. This post did not add them, and This post did not run that version, so it makes no claim about their effect. Including such auxiliary variables is the standard way to make multiple imputation worth its extra work (Sterne and colleagues, 2009; White and Carlin, 2010).

A decision path

1. Count first. How many rows have every variable the model needs? Here, 116 of 153.
2. Ask what the missingness could depend on. The outcome, the predictors, something else you observed, or the missing value itself.
3. If it depends on the predictors only, or on nothing, complete-case analysis gives valid regression estimates. Report the number of rows used.
4. If it depends on observed variables outside the model, use multiple imputation and put those variables in the imputation model.
5. If it may depend on the missing outcome value itself, neither is enough. Run a sensitivity analysis that varies the assumption and say how far the answer moves.
6. Whichever you pick, show the other as a check. Agreement between the two, as here, is reassuring. Disagreement is a finding to explain, not a reason to choose the one you like.

What multiple imputation does not fix

Multiple imputation does not repair data that are missing not at random, and it does not make a small sample large. It also adds choices: the imputation model, the method, the number of imputations. White, Royston and Wood (2011) suggest, as a rule of thumb, that m should be about equal to the percentage of incomplete rows. With 24 percent incomplete here, that would mean about 25 imputations, far above the engine’s five. With five imputations the coefficients landed close to the complete-case ones, but a published analysis with this much missing data should use more.

A final caution about the example itself. The daily readings are consecutive days, and ozone on one day is related to ozone on the next. Neither fit models that serial dependence, so the standard errors in both columns are probably too small. This post compares two ways of handling missing values on the same model, not a final analysis of New York air quality.

Reporting it

State the number of rows used or imputed, how many values were missing in each variable, the assumed missing-data mechanism and why, and for multiple imputation the method, m, the variables in the imputation model and the software. For example: “Ozone was missing on 37 of 153 days (24%). We fitted the regression to the 116 complete days and, as a check, to five multiply imputed datasets pooled by Rubin’s rules; the temperature coefficient was 1.84 and 1.86 respectively.” For the wording of coefficients in general, see interpreting regression coefficients.

Reproducing this

Below the article, the engine’s R script for the complete-case fit is shown with its seed and printed output. It reads the airquality data that ships with R, so no data file is needed. The multiple-imputation fit is the same script with the missing-data setting changed from complete cases to multiple imputation. Both runs used the same seed, and a second run gave identical numbers. The post on linear regression assumptions explains the residual checks flagged above, and the guide to preparing data for statistics covers the cleaning that comes before this choice.

# Results

- Test: Linear regression
- n: 116
- statistic: 7.362
- p-value: < 0.001
- effect size: R-squared = 0.5687
- 95% CI (Temp): [1.345, 2.335]
- rows used: 116 of 153 (37 dropped)
- seed: 20261110

## Assumption checks
- normality (Shapiro-Wilk): passed=FALSE (evidence of non-normality)

## Power (sensitivity, not observed power)
- minimum detectable effect at n=116: Cohen's f2 = 0.0853
- to detect a conventional medium effect: 68 total to detect f2=0.15
- alpha = 0.05, target power = 0.8

# Celeus - reproduce.R  (run from this package folder; bundled data in data/)
# Test: Linear regression  |  engine 0.53.2  |  catalog c40-dc08
# Research context: not provided for this run.
# Reproduces the recorded result deterministically.

set.seed(20261110)
.data_path <- "data/multiple-imputation-vs-complete-case.csv"
if (!file.exists(.data_path)) stop("Run reproduce.R from the package folder (the bundled data is in data/).")
data <- readr::read_csv(.data_path, show_col_types = FALSE)  # needs the readr package
clean_cells <- function (df) 
{
    ctrl_collapse <- "[\r\n\t\v\f]+"
    zero_width_chars <- intToUtf8(c(8203L, 65279L), multiple = TRUE)
    uni_space_chars <- intToUtf8(c(160L, 8239L, 8199L), multiple = TRUE)
    clean_text <- function(x) {
        out <- x
        for (ch in zero_width_chars) out <- gsub(ch, "", out, 
            fixed = TRUE)
        out <- gsub(ctrl_collapse, " ", out)
        for (ch in uni_space_chars) out <- gsub(ch, " ", out, 
            fixed = TRUE)
        trimws(gsub(" +", " ", out))
    }
    changed_cols <- character(0)
    cells_cleaned <- 0L
    retyped <- character(0)
    for (j in seq_along(df)) {
        col <- df[[j]]
        if (!is.character(col)) 
            next
        cleaned <- clean_text(col)
        n_changed <- sum(!is.na(col) & !is.na(cleaned) & col != 
            cleaned, na.rm = TRUE)
        if (n_changed > 0L) {
            changed_cols <- c(changed_cols, names(df)[j])
            cells_cleaned <- cells_cleaned + n_changed
        }
        df[[j]] <- cleaned
        non_blank <- cleaned[!is.na(cleaned) & nzchar(cleaned)]
        if (length(non_blank) && all(grepl("^[+-]?(0|[1-9][0-9]*)([.][0-9]+)?$", 
            non_blank))) {
            parsed <- suppressWarnings(as.numeric(cleaned))
            was_blank <- is.na(cleaned) | !nzchar(cleaned)
            lost <- (is.na(parsed) | is.infinite(parsed)) & !was_blank
            if (!any(lost)) {
                finite <- parsed[!is.na(parsed)]
                all_whole <- all(finite == floor(finite))
                in_range <- all(abs(finite) <= .Machine$integer.max)
                df[[j]] <- if (all_whole && in_range) 
                  as.integer(parsed)
                else parsed
                retyped <- c(retyped, names(df)[j])
            }
        }
    }
    names(df) <- clean_text(names(df))
    if (length(changed_cols)) {
        attr(df, "ingest_clean_columns") <- changed_cols
        attr(df, "ingest_clean_cells_cleaned") <- cells_cleaned
    }
    if (length(retyped)) 
        attr(df, "ingest_clean_retyped") <- retyped
    df
}
data <- clean_cells(data)
# Complete-case analysis: drop rows missing any analysis value.
.acols <- c("Ozone", "Temp", "Wind")
data <- data[stats::complete.cases(data[, .acols, drop = FALSE]), , drop = FALSE]
result <- summary(lm(Ozone ~ Temp + Wind, data = data))
print(result)

# Sensitivity: minimum detectable effect at n=116 (power=0.8, alpha=0.05). NOT observed power.
power <- pwr::pwr.f2.test(u = 2, v = 113, sig.level = 0.05, power = 0.8)
print(power)