Parametric vs nonparametric tests, side by side

A parametric test and its nonparametric counterpart are two different instruments, not a strict test and a lenient one. Welch’s t-test asks whether two group means differ. The Mann-Whitney test asks whether a value from one group tends to be larger than a value from the other. This post runs both on the same 24 field plots and reads them next to each other.

The example is deliberately small and a little awkward: the two p-values land on opposite sides of 0.05. That makes it a useful place to see why the choice should come from the question, not from the p-value. For the full decision map, see how to choose a statistical test.

What “nonparametric” means

A parametric test models the data with a named distribution, usually the normal, and tests a parameter of it, usually a mean. A nonparametric test does not name the distribution. Most of the common ones replace each value with its rank in the combined sample, so the result depends on ordering rather than on the size of the gaps.

Ranking removes some assumptions and keeps others. Observations must still be independent, and the groups must still be comparable. The change also reaches the question: a rank-based test is not a test of means, and in general it is not a test of medians either.

The usual pairings, each covered by an existing guide, are:

The first row of that table is the one this post runs end to end.

The data and the design

The example uses R’s built-in InsectSprays data (Beall, 1942). Each of 72 field plots was treated with one of six sprays and its insects were counted once. We compare spray C with spray E, 12 plots each.

Each plot contributes one count and belongs to one spray, so the two groups are independent. That is the design both tests need. The outcome is a small whole-number count, with several tied values, which is the kind of data where a rank test and a test of means are both reasonable choices.

In the 24 plots, spray C had a mean count of 2.08 and spray E a mean of 3.50. The medians were one and a half and three. These group summaries were computed in R from the same rows the engine used.

The spray pair was picked after looking at the six groups, so treat the comparison as an illustration, not a pre-registered finding. Nothing here is a claim about which spray to buy.

Running both tests

The engine ran each test on the same filtered rows with a fixed seed, and the committed R script reproduces the numbers. The equivalent base-R code is short:

cs <- subset(InsectSprays, spray %in% c("C", "E"))
cs$spray <- droplevels(cs$spray)
t.test(count ~ spray, data = cs)
wilcox.test(count ~ spray, data = cs, conf.int = TRUE)

Its output, trimmed to the lines that matter, was:

	Welch Two Sample t-test
t = -1.868, df = 21.631, p-value = 0.07536
95 percent confidence interval:
 -2.9909888  0.1576555
mean in group C mean in group E 
       2.083333        3.500000 

	Wilcoxon rank sum exact test
W = 38.5, p-value = 0.04914
96.9 percent confidence interval:
 -3  0
difference in location 
                    -2 

Both calls ran in R 4.6.2, the version used by the engine image. The printed values match the engine output reported below.

Reading Welch’s t-test

Welch’s t-test compares the two group means without assuming equal variances. It gave t = -1.868 and p = 0.0754. The estimated difference in means (C minus E) is -1.42 counts, with a 95% confidence interval from -2.991 to 0.1577. The interval includes zero, which is the same message as the p-value: with 12 plots per spray, the data are compatible with no difference in mean counts and also with a fairly large one.

The standardised effect is Cohen’s d = -0.7626, with a 95% interval from -1.586 to 0.0763. By the usual rough labels that is a medium-to-large point estimate, but the interval is wide and reaches zero, so the sample is too small to pin it down.

The engine also reports a sensitivity figure: with 24 plots, the smallest effect this design could reliably detect at 80% power is d = 1.197, which it labels severely underpowered. A non-significant result here is inconclusive, not evidence of no difference.

On assumptions, the engine’s Levene check found no evidence of unequal variances. Its Shapiro-Wilk check, run per group, flagged evidence of non-normality in spray C and not in spray E. With only 12 plots per group and a count outcome, that flag is a prompt to think about the shape of the data, not an automatic switch to a rank test.

Reading the Mann-Whitney test

The Mann-Whitney test, also called the Wilcoxon rank-sum test, gave U = 38.5 and p = 0.0491. Its null hypothesis is that both groups come from the same distribution, and it has power against alternatives where one group tends to produce larger values.

The matching effect size is the rank-biserial correlation, r = -0.4653, with a 95% interval from -0.7511 to -0.03252. To read it, imagine all 144 pairs made of one C plot and one E plot. A recount in R gave 96 pairs where the E plot had the higher count, 19 ties and 29 pairs where the C plot was higher. Counting ties as half, the chance that a random E plot beats a random C plot is close to three in four. That is the same information as r = -0.4653 on a different scale, and the figure was recomputed from the pair counts above.

The accompanying Hodges-Lehmann estimate is a shift of -2 counts. The engine reports its interval as -3 to 0 at an achieved coverage of 96.88%, not the 95% requested. Rank-based exact intervals can reach only certain confidence levels, so the coverage moves in jumps. Report the level R gives you.

In this R version, the test used an exact method even though the counts contain ties, and printed “Wilcoxon rank sum exact test”. Other software, or older versions of R, may handle ties differently and give a different p-value. Here the difference matters: the normal approximation with a continuity correction (exact = FALSE), which is what older R versions fell back to when the data had ties, gives a p-value just above 0.05 on the same counts. Check what your software does before comparing numbers.

Why the p-values land on opposite sides of 0.05

A p-value of 0.0754 and a p-value of 0.0491 are not in conflict. They answer different questions. The t-test is about the difference in means and uses the sizes of the counts. The Mann-Whitney test is about ranks and uses only the ordering. Spray E’s counts are more often higher than spray C’s across pairs (the rank test’s focus), while the difference in averages carries a wide interval (the t-test’s focus). Both are fair summaries of the same 24 numbers.

What goes wrong is choosing between them after seeing which p-value is smaller. That turns a 0.05 threshold into a search over two tests, and the false-positive rate rises above the nominal level. The engine’s own report for this comparison states the safe habit: the pre-specified test stays the primary result, and the second is labelled a sensitivity analysis.

Both p-values sit close to 0.05, and both confidence intervals brush zero. The honest reading is a modest, uncertain difference in the direction of fewer insects on spray C, not a clear win for either test. A different spray pair, or a larger study, could tilt the picture either way.

What each test does and does not tell you

Welch’s t-test gives a difference in means with an interval, in the units of the outcome. It is the natural report when the mean is the quantity you care about, for example total counts across many plots. With moderate to large samples it holds up well for skewed data because it depends on the sampling distribution of the mean (Delacre et al., 2017).

The Mann-Whitney test gives a probability that one group’s value exceeds the other’s, and an effect size to match. It suits ordinal outcomes, or a question phrased as “which group tends to score higher”. It does not escape the unequal-spread problem: groups with different shapes or variances can produce a significant result on their own (Fay and Proschan, 2010).

Neither is, in general, a test of medians. The Mann-Whitney test compares medians only under a shift model, where the two distributions have the same shape and differ only in location (Divine et al., 2018). In this example the sample medians differ by one and a half, and the Hodges-Lehmann shift is -2, so the two numbers are close but are not the same quantity. For a direct comparison of medians, estimate them directly, for example with a bootstrap interval.

Assumptions, side by side

Both tests need independent observations. A plot counted twice, or two plots sharing a field in a way the analysis ignores, breaks either of them in the same way, and no rank transformation repairs that.

Welch’s t-test additionally relies on the sampling distribution of the group means being close to normal. With small samples and strongly skewed data that can fail, which is the situation where a rank test earns its place. It does not require equal variances, which is the point of the Welch correction (Welch, 1947).

The Mann-Whitney test does not require normality. It does require that the question be the one it answers. If the two groups differ in spread or shape as well as location, a significant result says the distributions differ in a way that makes one group’s values tend to be larger, and it does not isolate a shift.

Ties deserve a look with count data. Many equal values reduce the information that ranks carry, and software handles them in different ways, so state which method produced the p-value you report.

How to write each result up

For the t-test, state the direction in words, the difference in means, its interval and the test. For the rank test, state which group tended to have larger values, the effect size and its interval, and the test. Name which group came first, because a negative sign means nothing without it.

A fair write-up of this example reads: “In the primary analysis, mean insect counts were lower on spray C plots than on spray E plots (Welch’s t-test, p = 0.0754; difference in means -1.42, 95% CI -2.991 to 0.1577). In a Mann-Whitney sensitivity analysis, spray E plots tended to have higher counts (U = 38.5, p = 0.0491; rank-biserial correlation -0.4653, 95% CI -0.7511 to -0.03252). The comparison was chosen after inspecting six sprays and is exploratory.”

The wording above names the primary test, labels the second as a companion, gives the interval for each and flags how the pair was chosen. It does not say that one spray works or does not work.

A short decision guide

• You want a difference in means, with a confidence interval. Use Welch’s t-test. It is a safe default for two independent groups.
• The outcome is ordinal, or you want to know which group tends to be higher. Use Mann-Whitney, and report the rank-biserial correlation or the probability of superiority with its interval.
• The same units were measured twice. Neither test applies as written. Use the paired t-test or the Wilcoxon signed-rank test, as in the paired comparison guide.
• Three or more groups. Use one-way ANOVA with a Welch correction, or Kruskal-Wallis, as in the multi-group guide.
• The data are heavily skewed with a small sample. Ask whether the mean is the right summary at all before reaching for a different test.

The longer comparison of these two tests, including a case where equal medians still give a significant rank test, is in the t-test vs Mann-Whitney guide.

Common mistakes

• Choosing the test by the p-value. Pick from the question, before looking at results.
• Calling Mann-Whitney a test of medians. Outside a shift model it is not one.
• Treating “nonparametric” as “assumption-free”. Independence and a sensible comparison still apply.
• Switching tests only because a histogram looks skewed. A skewed raw distribution does not by itself make the t-test unusable, and the switch changes the question being asked.
• Reading a non-significant result as no effect. Look at the interval and at what the design could detect.
• Reporting only a p-value. Give the difference in means with its interval, or the rank-based effect size with its interval.

Reproducing the numbers

The engine example below carries the exact R script for the Mann-Whitney run, with the filter to sprays C and E. A companion run applies Welch’s t-test to the same rows with the same seed. For the effect size across other tests, see effect sizes for Mann-Whitney and Wilcoxon tests. Method details are on the reference pages for Welch’s t-test and the Mann-Whitney U test.

The frame here is research statistics, not clinical advice: the example compares insect counts and makes no claim about any patient or treatment.

# Results

- Test: Mann-Whitney U test
- n: 24
- statistic: 38.5
- p-value: 0.0491
- effect size: rank-biserial correlation = -0.4653 (95% CI -0.7511 to -0.03252)
- 96.8814% CI (Hodges-Lehmann difference in location): [-3, 0]
- seed: 20261111

## Assumption checks
- normality by group (Shapiro-Wilk): context only, passed=FALSE (evidence of non-normality - normality is not assumed by this rank-based test)
- sample size: passed=TRUE (adequate n)

## Power
- Power is not calculated for this test. The matching parametric test gives a rough guide.

# Celeus - reproduce.R  (run from this package folder; bundled data in data/)
# Test: Mann-Whitney U test  |  engine 0.53.2  |  catalog c40-dc08
# Research context: not provided for this run.
# Reproduces the recorded result deterministically.

set.seed(20261111)
.data_path <- "data/nonparametric-test-side-by-side.csv"
if (!file.exists(.data_path)) stop("Run reproduce.R from the package folder (the bundled data is in data/).")
data <- readr::read_csv(.data_path, show_col_types = FALSE)  # needs the readr package
clean_cells <- function (df) 
{
    ctrl_collapse <- "[\r\n\t\v\f]+"
    zero_width_chars <- intToUtf8(c(8203L, 65279L), multiple = TRUE)
    uni_space_chars <- intToUtf8(c(160L, 8239L, 8199L), multiple = TRUE)
    clean_text <- function(x) {
        out <- x
        for (ch in zero_width_chars) out <- gsub(ch, "", out, 
            fixed = TRUE)
        out <- gsub(ctrl_collapse, " ", out)
        for (ch in uni_space_chars) out <- gsub(ch, " ", out, 
            fixed = TRUE)
        trimws(gsub(" +", " ", out))
    }
    changed_cols <- character(0)
    cells_cleaned <- 0L
    retyped <- character(0)
    for (j in seq_along(df)) {
        col <- df[[j]]
        if (!is.character(col)) 
            next
        cleaned <- clean_text(col)
        n_changed <- sum(!is.na(col) & !is.na(cleaned) & col != 
            cleaned, na.rm = TRUE)
        if (n_changed > 0L) {
            changed_cols <- c(changed_cols, names(df)[j])
            cells_cleaned <- cells_cleaned + n_changed
        }
        df[[j]] <- cleaned
        non_blank <- cleaned[!is.na(cleaned) & nzchar(cleaned)]
        if (length(non_blank) && all(grepl("^[+-]?(0|[1-9][0-9]*)([.][0-9]+)?$", 
            non_blank))) {
            parsed <- suppressWarnings(as.numeric(cleaned))
            was_blank <- is.na(cleaned) | !nzchar(cleaned)
            lost <- (is.na(parsed) | is.infinite(parsed)) & !was_blank
            if (!any(lost)) {
                finite <- parsed[!is.na(parsed)]
                all_whole <- all(finite == floor(finite))
                in_range <- all(abs(finite) <= .Machine$integer.max)
                df[[j]] <- if (all_whole && in_range) 
                  as.integer(parsed)
                else parsed
                retyped <- c(retyped, names(df)[j])
            }
        }
    }
    names(df) <- clean_text(names(df))
    if (length(changed_cols)) {
        attr(df, "ingest_clean_columns") <- changed_cols
        attr(df, "ingest_clean_cells_cleaned") <- cells_cleaned
    }
    if (length(retyped)) 
        attr(df, "ingest_clean_retyped") <- retyped
    df
}
data <- clean_cells(data)

# Analysis population filter (recorded in the spec): spray in C, E
data <- data[which(data[["spray"]] %in% c("C", "E")), , drop = FALSE]
# Complete-case analysis: drop rows missing any analysis value.
.acols <- c("count", "spray")
data <- data[stats::complete.cases(data[, .acols, drop = FALSE]), , drop = FALSE]
result <- wilcox.test(`count` ~ `spray`, data = data, conf.int = TRUE, conf.level = 0.95)
print(result)