How to handle outliers before a statistical test

An outlier is a question, not a verdict. Before any test, ask why the value is extreme: a recording error is corrected, a unit outside the population you meant to study is excluded by a rule you wrote in advance, and a genuine extreme value stays in the data, with a sensitivity analysis showing how much the result depends on it.

This post turns that into a rule you can apply value by value, explains why “far from the others” is never enough reason to delete, and works one seeded synthetic example with three planted extreme values, one of each kind. The data cleaning checklist puts this step among the other pre-test checks; this post goes deeper on the outlier decision.

Detection is not a decision

Outlier detection methods answer one narrow question: which values deserve a look? They do not say what to do with them. The common rules measure distance from the centre:

• Standard deviations from the mean. Flag anything more than two or three standard deviations away. The mean and the standard deviation are pulled by the very values you are looking for, so a large error can partly hide itself.
• Tukey’s fences. Flag values more than one and a half interquartile ranges below the first quartile or above the third. Quartiles barely move when one value is extreme. The IQR rule post computes them step by step.
• The median absolute deviation. Distance from the median in units of the median absolute deviation. Leys and colleagues recommend it over the standard-deviation rule for univariate data, because both quantities resist the extremes.

Plot the values first, with a box plot or dot plot per group; the picture shows an impossible value or a second cluster faster than any cut-off. The standard-deviation rule is weakest in small samples: one extreme can hide another (masking), and with 10 or fewer observations a cut-off of three cannot fire at all, because no value can lie more than (n - 1)/√n sample standard deviations from the mean (Shiffler 1988). Formal tests assume approximately normal data: Grubbs’ test checks for one outlier, and Rosner’s (1983) generalized extreme studentized deviate test checks for up to a stated number, which copes better with masking. Local outlier factor, isolation forest and the noise points of the clustering method DBSCAN are built for many variables at once; for one outcome compared between groups, the three rules above are enough.

A value beyond the fence can be perfectly real, and an entry error can sit inside it: a score of 58 typed as 85 will never be flagged. So the rule below starts from the source of the value, not its distance. The engine’s outlier check works the same way: it reports values beyond Tukey’s fences as a note and removes nothing.

A decision rule in three questions

Aguinis, Gottfredson and Joo reviewed 46 methodological sources and found outliers defined, identified and handled inconsistently; they offer decision trees that treat error outliers differently from genuine but unusual ones. The version below asks three questions in order and stops at the first “no”.

1. Is the value possible, and was it recorded correctly?

Check the value against the scale and the source record. A score of 776 on a 0 to 100 scale, a negative age, pounds in a column of kilograms: these are errors.

• If the source record shows the true value, correct it and log the change.
• If the true value cannot be recovered, set it to missing and log why. Never replace it with a guess, the group mean or the fence value.

An error is fixed whether or not it is extreme.

2. Does the unit belong to the population you set out to study?

Some extreme values are real measurements from units that should never have been analysed: a participant who failed an inclusion criterion, a sample processed under a different protocol. The question is membership, not the value. Exclusion needs a criterion that could have been written before the outcomes were seen, applied to every unit, extreme or not, and you report how many units it removed.

3. Otherwise, it is a genuine extreme value: keep it

A possible, correctly recorded value from an eligible unit is data. Keeping it does not mean ignoring it. Choose one option before you see the result:

• Keep the planned analysis and add a pre-specified sensitivity analysis that limits the value’s pull, so a reader can see how much the conclusion depends on it.
• Use an analysis that limits the value’s pull as the primary analysis: a rank-based test, a trimmed-means test or a robust regression.
• Change the scale or the model for a substantive reason, for example a log scale for a strictly positive, right-skewed outcome. The reason has to come from the measurement, not from wanting the extreme value to look smaller.

What you found	Action	What to report
Impossible value or proven entry error	Correct from the source, or set to missing	Each change, with its reason
Unit outside the planned population	Exclude by the written criterion, applied to all units	The criterion and how many units it removed
Genuine extreme value	Keep it; robust primary or sensitivity analysis	The primary result and the sensitivity result side by side

Why “remove it because it is extreme” is not an option

Bakker and Wicherts simulated independent-samples t-tests on the kind of non-normal sum scores common in questionnaire research. Removing outliers with commonly used thresholds in standard deviations raised the false-positive rate far above the nominal 5%, past 20% in one of their settings, and the inflation was particularly severe when the threshold could be changed after seeing its effect on the result. They recommend keeping the values and using a Mann-Whitney or robust Yuen-Welch test instead, which held the nominal error rate in their simulations.

Deleting the values furthest from the centre shrinks the standard error, and doing it only when the first result disappoints adds another chance to cross the threshold. Simmons, Nelson and Simonsohn showed that such undisclosed flexibility can, in many cases, make false evidence of an effect more likely than correctly finding none.

Write the rule before you see the results

Leys and colleagues argue for pre-registering how outliers will be detected and managed, to remove the flexibility of deciding after the results are known. A usable plan names four things:

1. The detection rule and its threshold (for example, Tukey’s fences at one and a half interquartile ranges, within each group).
2. The checks every flagged value goes through: source record, scale limits, eligibility.
3. The action for each diagnosis, as in the table above.
4. The primary analysis and the sensitivity analysis that will be reported whatever the outcome.

If nothing was written down, write the rule before running the test and report it as set after data collection but before analysis.

Worked example: three planted values

The data are seeded synthetic scores, not a real study: 20 first-time students taught by a standard method and 20 taught by a new method, one test score each on a 0 to 100 scale. Each student is taught by one method only, so the groups are independent, and the question is whether the mean score differs between methods. The plan, written before the analysis, names Welch’s two-sample t-test as the primary analysis, Tukey’s fences as the detection rule, and a rank-based test as the sensitivity analysis.

Three values were planted, one of each kind:

• Student 7’s score of 77.6 was typed as 776.
• A repeat student, outside the “first-time students only” eligibility rule, scored 96.
• Student 40, eligible and correctly recorded, scored 31: a genuine extreme.

Each version was seeded and run in R. The main output and its script are below the article.

The data as received

Run on all 41 rows, the engine’s outlier note counted 3 values beyond Tukey’s fences in the score column: the 776, the 96 and the 31. The mean in the standard group came out at about 96 against 67.4 in the new group, so the estimated difference (new minus standard) was -28.6 points, with a 95% confidence interval from -103.8 to 46.6 and Cohen’s d of -0.25. One missing decimal point reversed the direction of the comparison and made the interval useless.

Shapiro-Wilk detected non-normality in both groups, but Levene’s test did not detect unequal variances even with 776 in one group. A non-significant check means a problem was not detected, and here one plainly existed: no test replaces looking at the values.

Applying the rule

• 776: impossible on a 0 to 100 scale. The source record reads 77.6, so the value is corrected and the change logged.
• 96, the repeat student: possible and correctly recorded, but the student fails the eligibility rule written before data collection. The student is excluded under that rule, which would apply just the same if the score had been 60.
• 31, student 40: possible, correctly recorded, eligible. It is a genuine extreme and stays.

That leaves 40 students, 20 per method, with one value still beyond the fence.

The decided analysis

On the decided data, Welch’s t-test gives t = 1.547 on 37.85 degrees of freedom, p = 0.13. The mean score was 65.94 in the new group and 61.045 in the standard group, a difference of 4.895 points with a 95% confidence interval from -1.511 to 11.3. Cohen’s d is 0.4893 (95% CI -0.1433 to 1.116); this d divides the difference by the pooled standard deviation of the two groups, and its interval assumes equal variances, an assumption Welch’s test itself does not make.

The data are compatible with no difference, with the new method scoring slightly lower, and with a gain of about 11 points. The result is inconclusive, not evidence that the methods are equivalent. The engine’s sensitivity figure shows why: with 20 students per group, only effects of d = 0.9092 or larger could be detected with 80% power; a medium effect would need 64 per group.

Shapiro-Wilk detected non-normality in the new group (W = 0.825, p = 0.0021) and not in the standard group (p = 0.824); Levene’s test did not detect unequal variances (p = 0.591). The engine therefore flagged a rank-based test, the sensitivity analysis the plan already names.

What the genuine extreme does

Two companion runs show how much the conclusion leans on student 40.

Rank-based sensitivity analysis, value kept. The Mann-Whitney test keeps all 40 students but uses ranks, so the score of 31 counts as the lowest rank rather than by its distance from the other scores. It gives p = 0.055 and a Hodges-Lehmann shift of 5.6 points in favour of the new method, with an interval from -0.1 to 11.5. That interval inverts the exact rank test, whose statistic moves in discrete steps, so its actual coverage is 95.17%, the nearest achievable level above 95%; like p = 0.055, it includes zero. The rank-biserial correlation is 0.355, with an interval from 0.0089 to 0.625 that just excludes zero; that interval uses a large-sample approximation, not the exact test, so the two can disagree near zero. That is a borderline result, and the engine flagged it as such. This test asks whether scores in one group tend to be higher, not whether the means differ.

What not to do: delete the genuine extreme. This run is shown only as the mistake to avoid. Deleting student 40 gives a difference of 6.73 points, a 95% interval from 1.41 to 12.06, p = 0.0147 and Cohen’s d of 0.82. Shapiro-Wilk no longer detects non-normality in either group.

Deleting one real, eligible student moves the result from p = 0.13 to below 0.05 and makes the assumption checks look cleaner; neither is a reason to delete. The honest summary is that any advantage of the new method is not established by these data and depends on how one genuine low score is weighted.

How to report it

A reporting paragraph for the example could read:

Outliers were screened with Tukey’s fences on the score column and checked against source records and the eligibility criteria, as specified before analysis. One score recorded as 776 was corrected to 77.6 from the source record; one repeat student was excluded under the first-time-students criterion; one genuine extreme score (31) was retained. In the primary analysis (Welch’s t-test, 20 students per method), the mean score was 4.9 points higher with the new method (95% CI -1.5 to 11.3; p = 0.13; Cohen’s d = 0.49, 95% CI -0.14 to 1.12). A rank-based sensitivity analysis gave a similar shift (Hodges-Lehmann 5.6, 95% CI -0.1 to 11.5; p = 0.055).

The run without student 40 is not a result to report.

Three cases that look like outliers but need something else

A value at a detection limit. A measurement recorded as “below the limit” is censored, not an outlier or an error, and deleting it or substituting a fraction of the limit biases the result. The guide to values below the detection limit covers methods that use censored values as they are.

A whole skewed distribution. When many values sit in a long tail, the issue is the shape of the outcome, not a few outliers: see the t-test or Mann-Whitney U test guide and the t-tests explainer.

A point that is extreme in a regression. A large residual, high leverage (unusual predictor values) and influence (removal would move the estimates, summarised by Cook’s distance) are different things. Refit without the point as a sensitivity analysis; the three-question rule still decides whether it may be changed or removed.

Common mistakes

• Deleting values because they are beyond the fence. The fence picks values to check. It is not a definition of error.
• Choosing the rule after seeing the results. Trying thresholds until the result looks right is a hidden multiple comparison.
• Applying an exclusion criterion only to extreme units. A real criterion applies to everyone it describes.
• Replacing an error with the mean or the fence value. That invents data.
• Reporting only the version you prefer. Report both the primary and the sensitivity analysis.

Rerunning the example

The script below the article reproduces the decided analysis only: Welch’s t-test and the sensitivity power calculation on the 40-row data file. It does not reproduce Cohen’s d, the assumption checks, the outlier note or the companion runs. By hand, effectsize::cohens_d() gives the effect size, shapiro.test() and car::leveneTest() the checks, boxplot.stats() the values beyond Tukey’s fences, and wilcox.test(score ~ method, conf.int = TRUE) the rank-based sensitivity analysis. The method reference pages for Welch’s t-test and the Mann-Whitney U test describe each test, and the guide to reproducible statistical analysis explains what a reviewer needs to rerun a result like this one.

# Results

- Test: Welch's t-test
- n: 40
- statistic: 1.547
- p-value: 0.13
- effect size: Cohen's d = 0.4893 (95% CI -0.1433 to 1.116)
- 95% CI: [-1.511, 11.3]
- seed: 20261022

## Assumption checks
- normality by group (Shapiro-Wilk): passed=FALSE
- variance homogeneity (Levene): passed=TRUE (no evidence of unequal variances)
- sample size: passed=TRUE (adequate n)

## Power (sensitivity, not observed power)
- minimum detectable effect at n=40: Cohen's d = 0.9092 (severely underpowered)
- to detect a conventional medium effect: 64 per group to detect d=0.50
- alpha = 0.05, target power = 0.8

# Celeus - reproduce.R  (run from this package folder; bundled data in data/)
# Test: Welch's t-test  |  engine 0.30.0  |  catalog c39-d3cd
# Research context: not provided for this run.
# Reproduces the recorded result deterministically.

set.seed(20261022)
.data_path <- "data/outliers-before-statistical-test.csv"
if (!file.exists(.data_path)) stop("Run reproduce.R from the package folder (the bundled data is in data/).")
data <- readr::read_csv(.data_path, show_col_types = FALSE)  # needs the readr package
clean_cells <- function (df) 
{
    ctrl_collapse <- "[\r\n\t\v\f]+"
    zero_width_chars <- intToUtf8(c(8203L, 65279L), multiple = TRUE)
    uni_space_chars <- intToUtf8(c(160L, 8239L, 8199L), multiple = TRUE)
    clean_text <- function(x) {
        out <- x
        for (ch in zero_width_chars) out <- gsub(ch, "", out, 
            fixed = TRUE)
        out <- gsub(ctrl_collapse, " ", out)
        for (ch in uni_space_chars) out <- gsub(ch, " ", out, 
            fixed = TRUE)
        trimws(gsub(" +", " ", out))
    }
    changed_cols <- character(0)
    cells_cleaned <- 0L
    retyped <- character(0)
    for (j in seq_along(df)) {
        col <- df[[j]]
        if (!is.character(col)) 
            next
        cleaned <- clean_text(col)
        n_changed <- sum(!is.na(col) & !is.na(cleaned) & col != 
            cleaned, na.rm = TRUE)
        if (n_changed > 0L) {
            changed_cols <- c(changed_cols, names(df)[j])
            cells_cleaned <- cells_cleaned + n_changed
        }
        df[[j]] <- cleaned
        non_blank <- cleaned[!is.na(cleaned) & nzchar(cleaned)]
        if (length(non_blank) && all(grepl("^[+-]?(0|[1-9][0-9]*)([.][0-9]+)?$", 
            non_blank))) {
            parsed <- suppressWarnings(as.numeric(cleaned))
            was_blank <- is.na(cleaned) | !nzchar(cleaned)
            lost <- (is.na(parsed) | is.infinite(parsed)) & !was_blank
            if (!any(lost)) {
                finite <- parsed[!is.na(parsed)]
                all_whole <- all(finite == floor(finite))
                in_range <- all(abs(finite) <= .Machine$integer.max)
                df[[j]] <- if (all_whole && in_range) 
                  as.integer(parsed)
                else parsed
                retyped <- c(retyped, names(df)[j])
            }
        }
    }
    names(df) <- clean_text(names(df))
    if (length(changed_cols)) {
        attr(df, "ingest_clean_columns") <- changed_cols
        attr(df, "ingest_clean_cells_cleaned") <- cells_cleaned
    }
    if (length(retyped)) 
        attr(df, "ingest_clean_retyped") <- retyped
    df
}
data <- clean_cells(data)
# Complete-case analysis: drop rows missing any analysis value.
.acols <- c("score", "method")
data <- data[stats::complete.cases(data[, .acols, drop = FALSE]), , drop = FALSE]
result <- t.test(`score` ~ `method`, data = data, var.equal = FALSE)
print(result)

# Sensitivity: minimum detectable effect at n=40 (power=0.8, alpha=0.05). NOT observed power.
power <- pwr::pwr.t2n.test(n1 = 20, n2 = 20, sig.level = 0.05, power = 0.8)
print(power)