The IQR rule: upper and lower fences for outliers

The upper and lower fences are the cut-offs of Tukey’s IQR rule: the lower fence is the first quartile minus 1.5 times the interquartile range, and the upper fence is the third quartile plus 1.5 times it. A value beyond a fence is a value to check, not a value to delete.

This post computes both fences by hand on a real public dataset, repeats the calculation with R’s boxplot.stats(), shows why the two disagree slightly, and then shows what a deterministic analysis engine does with the same values: it flags them, says so in its output, and leaves them in the data.

The rule in one line

John Tukey introduced the rule in his book on exploratory data analysis as a quick way to label values that sit unusually far from the bulk of the data. It needs three numbers:

• Q1, the first quartile: a quarter of the values lie below it.
• Q3, the third quartile: a quarter of the values lie above it.
• IQR = Q3 - Q1, the interquartile range: the width of the middle half of the data.

Then:

• Lower fence = Q1 - 1.5 x IQR
• Upper fence = Q3 + 1.5 x IQR

Tukey also described outer fences at 3 x IQR beyond the quartiles. A value between the inner and outer fence is “outside”; a value beyond the outer fence is “far out”. Both words are labels for a second look.

Why the IQR rather than the standard deviation? Because the quartiles barely move when a few extreme values are present. A rule built on the mean and standard deviation is pulled toward the very values it is meant to find. As Hoaglin, Iglewicz and Tukey put it, as long as the outliers are not too numerous, they do not affect the location of the cut-offs.

Worked by hand: ozone in New York, summer 1973

R’s built-in airquality data hold daily readings in New York from May to September 1973. The Ozone column is the mean ozone concentration in parts per billion between 1 pm and 3 pm at Roosevelt Island. Of 153 days, 37 have no ozone value, so the fences are computed on the 116 observed values. The missing days cannot be outliers or anything else: the rule has nothing to say about them.

Every quartile, fence and flagged value in this section and the next is computed by the base R code shown below, not taken from the engine’s output (the engine reports only how many values it flagged). The quartiles come from R’s quantile() at its default setting, which is type 7 of the nine definitions R offers. Type 7 places the p-th quantile at position 1 + (n - 1) x p in the sorted data and interpolates between neighbours:

• Q1 sits at position 29.75. The 29th and 30th sorted values are both 18, so Q1 = 18.
• Q3 sits at position 87.25. The 87th and 88th values are 63 and 64, so Q3 = 63 + 0.25 x 1 = 63.25.
• IQR = 63.25 - 18 = 45.25.

The fences follow:

• Lower fence = 18 - 1.5 x 45.25 = -49.875
• Upper fence = 63.25 + 1.5 x 45.25 = 131.125

Ozone cannot be negative, so nothing can fall below the lower fence. Two days lie above the upper fence, with readings of 135 and 168. The outer upper fence is 63.25 + 3 x 45.25 = 199, and no value reaches it, so both are “outside” but not “far out”.

x  <- airquality$Ozone[!is.na(airquality$Ozone)]  # 116 observed days
q  <- quantile(x, c(0.25, 0.75))                  # type 7: 18 and 63.25
iqr <- unname(q[2] - q[1])                        # 45.25
c(lower = q[[1]] - 1.5 * iqr, upper = q[[2]] + 1.5 * iqr)
x[x < q[[1]] - 1.5 * iqr | x > q[[2]] + 1.5 * iqr]  # 135 168

Notice the asymmetry. The middle half of the data runs from 18 to 63.25 around a median of 31.5, so the upper quarter is stretched much further than the lower one. Ozone is right-skewed, and on a skewed variable the rule tends to flag values in the long tail. That is a property of the distribution, not evidence that anything is wrong. Hubert and Vandervieren proposed an adjusted boxplot for skewed data, robustbase::adjbox() in R, which sets the whiskers using a robust skewness measure, the medcouple.

The same data with boxplot.stats()

R’s boxplot() draws its points from boxplot.stats(). Run it on the same 116 values:

b <- boxplot.stats(x)
b$stats  # 1, 18, 31.5, 63.5, 122: lower whisker, hinges, median, upper whisker
b$out    # 135 168

The box edges are 18 and 63.5, not 18 and 63.25. That is because boxplot.stats() does not call quantile(). It uses Tukey’s hinges from fivenum(): the median of the lower half and of the upper half of the sorted data. With 116 values each half has 58, and the upper hinge is the average of the 87th and 88th values, 63.5. R’s documentation says the hinges equal the quartiles for an odd number of values and can differ for an even number.

With hinges the box is 45.5 wide, so the fences are -50.25 and 131.75. The flagged values are the same two days, 135 and 168. The upper whisker stops at 122, the largest value inside the fence. Whiskers stop at the most extreme values inside the fences; anything beyond is drawn as a separate point.

So in this dataset both definitions agree on which days to flag. They need not: a reading of 131.5 would sit beyond the quantile() fence and inside the boxplot.stats() fence. Hyndman and Fan compared the quantile definitions used across statistical packages and noted that one package often uses one definition for an explicit quantile and another for a boxplot, which is exactly the situation here. Frigge, Hoaglin and Iglewicz found several boxplot constructions in common software, differing in how they compute the quartiles and the fences. When you report a fence, name the definition.

	Quartiles	Box width	Lower fence	Upper fence	Flagged
quantile(), type 7	18, 63.25	45.25	-49.875	131.125	135, 168
boxplot.stats(), hinges	18, 63.5	45.5	-50.25	131.75	135, 168

The same caution applies outside R. NumPy’s percentile() and quantile() default to linear interpolation, which the NumPy documentation lists as type 7 in Hyndman and Fan’s numbering, the same definition as R’s default. Excel’s QUARTILE.INC and QUARTILE.EXC use percentile values from 0 to 1 inclusive and exclusive respectively, so a spreadsheet fence depends on which one you call. Pick one definition, name it, and use it throughout.

What the engine does with the flagged values

The worked example below the article runs a Spearman rank correlation between the same Ozone column and maximum daily temperature (Temp, in degrees Fahrenheit at LaGuardia Airport) through Celeus’s deterministic engine. The correlation is there because a fence check runs on the columns an analysis uses. Before computing anything, the engine checks every numeric analysis column against Tukey’s 1.5 x IQR fence, with quartiles from quantile() at its default type 7, the same definition as the hand calculation above, not the hinges boxplot.stats() uses.

Its output carries this note:

Values beyond Tukey’s 1.5xIQR fence in: ‘Ozone’ (2). Descriptive only - nothing has been removed or changed; check whether these reflect genuine extreme cases or data-entry errors.

That is the whole intervention: two values flagged, none removed. Temp produced no flag. The engine also reports that it dropped 37 of 153 rows, the days with no ozone reading, and suggests considering imputation. Missing values and extreme values are separate problems, and the fence says nothing about the missing days.

The correlation itself: on n = 116 days, Spearman’s rho was 0.774 (p < 0.001), so days with higher maximum temperature tended to have higher ozone readings. That is an association in observational data. The engine reports rho without a confidence interval for this method, so none is quoted here. The engine’s only assumption check for this method is a minimum sample size, which passed; that is a floor, not a validity check. The p-value treats the 116 days as independent pairs; consecutive summer days are unlikely to be fully independent, so read it as a rough guide rather than an exact error rate.

The choice of a rank method matters for the flagged values. Spearman’s rho uses ranks, so the 168 reading enters as the highest of 116 ranks, the same rank it would have at 136. The two flagged days stay in the analysis, and their distance above the fence does not change the estimate. Pearson’s correlation, by contrast, uses the raw values, so the size of an extreme reading feeds straight into it. The t-test or Mann-Whitney guide discusses the same trade between means and ranks for two groups, and the Spearman method reference gives the engine’s specification.

A value beyond the fence is a value to check

The fence is a screening rule. It tells you where to look, not what you will find. A flagged value usually falls into one of three cases:

1. A recording or processing error: a typo, a unit mix-up, a decimal in the wrong place, a code such as 999 used for missing. Correct it from the source record if you can, or set it to missing and say so.
2. A genuine extreme: a real measurement from the same process as the rest of the data. Keep it. Deleting it makes the data look tidier than the world it came from and biases the estimates toward the middle.
3. A different process: a unit that does not belong to the population the question is about. Exclusion may be justified, but by that reason, stated in advance where possible, not by the fence.

Even data with nothing wrong in them produce values outside the fences. Hoaglin, Iglewicz and Tukey report that for Gaussian data the population outside rate is about 0.7% of observations. Their simulations show higher rates in small samples (between 1.7% and 8.6% of observations for samples of 5 to 20), and the rate rises again for heavier-tailed distributions. Flags on a long-tailed variable such as ozone are expected rather than alarming.

For the two ozone days, checking means looking at what else was recorded on those dates. The fence cannot settle that; the context can. The data cleaning checklist puts this check in order with the others to run before any test, and the post on outliers before a statistical test covers what to do once a flagged value has been checked.

Common mistakes

Deleting everything beyond the fence. The rule was designed to label, not to clean. If you exclude a value, write down why, and report the analysis with and without it so a reader can see whether the conclusion depends on it.

Re-applying the rule after deleting. Removing the flagged values shrinks the IQR and moves the fences inward, so a second pass flags new values. Repeated passes can strip away a real tail.

Applying one fence to groups that differ. When groups have different centres or spreads, a single fence on the pooled data flags the high group’s ordinary values. Check within each group.

Treating the fence as a test. There is no p-value behind 1.5. It is a convention, and a different multiplier would flag different points.

Comparing fences across software without naming the quartile definition. As the table above shows, two correct calculations on the same data can give different fences.

Applying it to a variable with a hard limit. Where values are censored at a detection limit, the extreme values are not measured at all, and a fence on the recorded numbers misleads. The guide to data below the detection limit covers that case.

Rerunning the example

Below the article are the R script the engine generated and the output it printed. On the same package versions, the script reproduces the Spearman statistic, p-value and rho. It does not reproduce the fence note, which comes from the engine’s own data check; the code blocks in this post reproduce the fences and flagged values with base R. The guide to reproducible statistical analysis explains what a reader needs in order to rerun a result like this one.

# Results

- Test: Spearman correlation
- n: 116
- statistic: 58780
- p-value: < 0.001
- effect size: Spearman's rho = 0.774
- rows used: 116 of 153 (37 dropped)
- seed: 20261022

## Assumption checks
- sample size: passed=TRUE (adequate n)

## Power
- Power is not calculated for this test. The matching parametric test gives a rough guide.

# Celeus - reproduce.R  (run from this package folder; bundled data in data/)
# Test: Spearman correlation  |  engine 0.30.0  |  catalog c39-d3cd
# Research context: not provided for this run.
# Reproduces the recorded result deterministically.

set.seed(20261022)
.data_path <- "data/iqr-rule-upper-lower-fence.csv"
if (!file.exists(.data_path)) stop("Run reproduce.R from the package folder (the bundled data is in data/).")
data <- readr::read_csv(.data_path, show_col_types = FALSE)  # needs the readr package
clean_cells <- function (df) 
{
    ctrl_collapse <- "[\r\n\t\v\f]+"
    zero_width_chars <- intToUtf8(c(8203L, 65279L), multiple = TRUE)
    uni_space_chars <- intToUtf8(c(160L, 8239L, 8199L), multiple = TRUE)
    clean_text <- function(x) {
        out <- x
        for (ch in zero_width_chars) out <- gsub(ch, "", out, 
            fixed = TRUE)
        out <- gsub(ctrl_collapse, " ", out)
        for (ch in uni_space_chars) out <- gsub(ch, " ", out, 
            fixed = TRUE)
        trimws(gsub(" +", " ", out))
    }
    changed_cols <- character(0)
    cells_cleaned <- 0L
    retyped <- character(0)
    for (j in seq_along(df)) {
        col <- df[[j]]
        if (!is.character(col)) 
            next
        cleaned <- clean_text(col)
        n_changed <- sum(!is.na(col) & !is.na(cleaned) & col != 
            cleaned, na.rm = TRUE)
        if (n_changed > 0L) {
            changed_cols <- c(changed_cols, names(df)[j])
            cells_cleaned <- cells_cleaned + n_changed
        }
        df[[j]] <- cleaned
        non_blank <- cleaned[!is.na(cleaned) & nzchar(cleaned)]
        if (length(non_blank) && all(grepl("^[+-]?(0|[1-9][0-9]*)([.][0-9]+)?$", 
            non_blank))) {
            parsed <- suppressWarnings(as.numeric(cleaned))
            was_blank <- is.na(cleaned) | !nzchar(cleaned)
            lost <- (is.na(parsed) | is.infinite(parsed)) & !was_blank
            if (!any(lost)) {
                finite <- parsed[!is.na(parsed)]
                all_whole <- all(finite == floor(finite))
                in_range <- all(abs(finite) <= .Machine$integer.max)
                df[[j]] <- if (all_whole && in_range) 
                  as.integer(parsed)
                else parsed
                retyped <- c(retyped, names(df)[j])
            }
        }
    }
    names(df) <- clean_text(names(df))
    if (length(changed_cols)) {
        attr(df, "ingest_clean_columns") <- changed_cols
        attr(df, "ingest_clean_cells_cleaned") <- cells_cleaned
    }
    if (length(retyped)) 
        attr(df, "ingest_clean_retyped") <- retyped
    df
}
data <- clean_cells(data)
# Complete-case analysis: drop rows missing any analysis value.
.acols <- c("Ozone", "Temp")
data <- data[stats::complete.cases(data[, .acols, drop = FALSE]), , drop = FALSE]
result <- cor.test(data[["Ozone"]], data[["Temp"]], method = "spearman")
print(result)