How AI picks a statistical test, and where it fails

A test recommendation, from a chatbot or anywhere else, can name a real, common test and still be the wrong one. Where it goes wrong is the step before the name: the test depends on facts about how the data were collected, and most of those facts are not in the columns. When an answer is built from column names and types alone, it can be fluent, confident and wrong.

The gap this post covers appears whether the answer comes from a person, a flowchart or a language model. It does not test any named AI product, and it makes no claim about how any of them behave. Instead it uses one seeded synthetic dataset, run twice through a deterministic engine, to show the failure that matters most, then gives a short checklist for judging any recommendation. The full decision framework is in the guide on choosing a statistical test.

What “an AI picks a test” can mean

Two quite different things get called “an AI picks a test”, and they fail in different ways.

A free-text answer is the first kind. You describe your data and your question in a chat box, and you get a recommendation in prose. The quality of that answer is capped by the quality of the description. If you leave out that each person was measured twice, the answer has no way to know.

Rule-based selection is the second kind. A program looks at a profile of the dataset (which columns are numeric, which are categories, how many levels each has) and applies fixed rules to propose a menu of suitable tests. Given the same profile it returns the same menu every time. Its weakness is the mirror image: it can only use what the profile and the stated roles contain.

Many test-selection tools mix the two, with rules narrowing the options and a model explaining or ranking them. In Celeus, for instance, the engine builds the candidate menu by fixed rules and runs the statistics deterministically, and no data rows are sent to the model. The point of this post is not that one approach is good and the other bad. It is that both need the design stated, and neither can read it off the spreadsheet.

What a spreadsheet can and cannot tell you

Some of the four facts that decide a test are visible in the data. The type of each column is: Stevens (1946) separated nominal, ordinal, interval and ratio measurements, and a program can usually tell a number from a label, and a two-level column from a six-level one. That is enough to narrow the choice from dozens of tests to a few.

Linkage and the target quantity are not visible in the data at all. Whether two columns are measurements on the same people is not a property of the numbers. Whether rows from the same school, clinic or litter are related to each other is not a property of the numbers either. And the quantity you want to report, a mean difference or an odds ratio or a survival time, is a decision about your question, not about your file.

An answer that treats the first two rows as the whole story will choose a test that is right for some layout of the data and wrong for the one you actually have.

One dataset, two designs

To make this concrete, the example uses seeded synthetic data, not a real study. A script generates 24 people, each with a score before a training session and a score after it. Every person has a stable baseline level drawn from a wide spread (mean 50, standard deviation 12), and the session adds a modest, fairly consistent change (mean 3, standard deviation 3). The numbers describe no real study.

The generated data can be laid out in two ways. Wide, there is one row per person with a before column and an after column. Stacked, there is one score column and a label saying before or after, 48 rows in all. The values are identical. A reader who sees only the stacked layout, a numeric score and a two-level label, would reasonably think of two groups and reach for a two-group t-test. A reader who knows each person appears in both groups would not.

The same seed and generator feed both runs, so the 48 values are the same. Only what the analysis is told about the design changes.

Version A: the test that fits the design

Each person contributes two scores, so the design is paired and the outcome is continuous. The test that follows from that is the paired t-test, which works on the 24 within-person changes and ignores the large differences between people. The reference page for the engine’s version is paired t-test.

Reading the paired result: in this sample the mean score is higher after the session than before, and the interval for the mean change excludes zero. The assumption check on the within-person changes passes, which is what the paired test needs. The engine also notes one value beyond Tukey’s IQR fence in each column, as a descriptive remark; nothing was removed.

Version B: the test a type-only answer would pick

Now the same 48 values, stacked, analysed as two independent groups of 24 with Welch’s t-test. This is the analysis that “numeric outcome, two-level label” points to when nothing else is known about the design. The reference page is Welch’s t-test.

The estimate of the mean difference is the same number as in Version A. The conclusion is not: the interval now runs from -4.433 to 9.524 and includes zero, with a p-value of 0.466.

Reading the two results side by side

Three things are worth taking from the comparison.

The data were the same, and the answer changed. The mean difference is 2.546 both times. What differs is how much noise it is compared against. The paired analysis compares the change to the spread of the within-person changes. The independent-groups analysis compares it to the spread of scores between people, which is far larger because people differ a lot in their baselines. Compare the interval ends above: the paired interval runs from 1.143 to 3.949, the Welch interval from -4.433 to 9.524, about five times wider. The two Cohen’s d values differ for the same reason and are not comparable: the paired d divides the mean change by the spread of the within-person changes, the Welch d by the spread of scores between people.

The wrong test passed every check it ran. Version B reports no evidence against normality and no evidence of unequal variances. Assumption checks on the numbers cannot reveal that the rows are linked, so a clean assumption panel is not evidence that the design is right. Rasch and colleagues (2011) found that choosing a two-sample test by pre-testing assumptions on the same data does not pay off, and the guide on choosing a statistical test makes the same case: the choice should rest on the design and the question, not on a pre-test.

The error runs in a direction you cannot assume. Here ignoring the pairing hid a change that was built into the data. With other data, treating independent groups as paired, or ignoring clustering, can make a result look stronger than it is. The fault is the mismatch between test and design, and its effect on the p-value can go either way.

Version B also carries the engine’s own note that a study of 48 (24 per group) could detect, at 80 percent power, only a very large effect (Cohen’s d of about 0.83 or more), so its non-significant result is inconclusive, not evidence that nothing happened. For a fuller treatment of why matching helps, see matched pairs design.

Where a free-text answer tends to go wrong

The example did not query a model, so the post cannot show what any particular answer would say. What it can do is name the failure modes you should look for in a free-text answer, each of which this example makes concrete.

No question about the design. The answer names a test without asking whether rows are linked. The fix is for you to state the design in the question: how many times each person, plot or animal appears, and what was measured on each occasion.

A code read as a number. A group coded 1, 2 and 3 is a category, even though it looks like a number. A tool that sees integers may offer a correlation where a group comparison is needed. State which columns are categories.

The wrong question answered. A recommendation that does not say which quantity it estimates (a mean difference, an odds ratio, a hazard ratio) may be right for a question you did not ask.

Arithmetic done in prose. If the numbers in an answer were typed rather than computed by a recorded program, you cannot tell whether they are right. Prefer a setup where the test is run by code and the result can be re-run.

No assumption check, effect size or interval. A bare p-value is a thin result. A p-value is not the probability the null hypothesis is true, and an interval is a statement about a procedure, not a 95 percent chance that this interval holds the truth (Greenland and colleagues, 2016).

A checklist for any test recommendation

Run these checks on a suggestion from a chat box, a colleague, or software, before you trust the result.

1. State the design first. Write one sentence saying how many times each unit appears, and what links the rows.
2. Name the quantity. Say what you want to report, such as a mean difference with an interval.
3. Check the test against the table above. Outcome type, number of groups and linkage should each point at the test. If the recommendation skips one, ask why.
4. Ask for the assumption checks on your data, not a list of assumptions in general.
5. Require an effect size with an interval, not just a p-value.
6. Ask whether the numbers can be re-run. The same input, the same seed and the same software versions should return identical output.
7. Keep the run record. The exact script, seed and package versions are what let someone else, or you in a year, check the result. The guide on reproducible statistical analysis covers what to record.

If the data are not yet in analysable shape, preparing data for statistics covers the cleaning and layout steps that come first.

What deterministic rules add, and what they do not

Fixed rules help in a specific way: the same inputs give the same menu and the same numbers, so a recommendation can be re-run and compared, and an answer that departs from the menu is visible as a departure. They do not remove the need for you to state the design. In the example, the paired test was run because the analysis was set up as paired, with each person’s two scores in one row, and the Welch test was run because the companion run was set up as two unrelated groups. The engine did what it was asked each time, and each run was internally clean.

The practical division of labour between you and the software is simple. You supply the facts the data cannot: the design and the question. The software supplies what it can do reliably: apply the rules, run the computation, report assumptions and effect sizes, and record everything. A statistician reviewing the plan is still worth having for anything high stakes, since nothing here replaces that review.

Reporting the paired result

For the paired analysis, a results sentence would read: the mean score was 2.546 points higher after the session than before (95% CI 1.143 to 3.949), paired t(23) = 3.753, p = 0.00104, Cohen’s d = 0.7661 (95% CI 0.303 to 1.216). State the design in the same sentence, since that is the fact the reader needs to judge the test. These are results from synthetic data built to show a method, and they describe no real study.

Reproducing the example

The example was run through a deterministic engine: seeded, on pinned package versions, run twice and checked to give identical output. The package includes the exact R script, reproduce.R, which recomputes the test statistic and p-value from the bundled data. The 24 baseline scores and the changes are produced by the generating script, so nothing here comes from a real person. The Welch run is a separate package built from the same seed, and it is shown only to measure what ignoring the pairing costs.

# Results

- Test: Paired t-test
- n: 24
- statistic: 3.753
- p-value: 0.00104
- effect size: Cohen's d = 0.7661 (95% CI 0.303 to 1.216)
- 95% CI: [1.143, 3.949]
- seed: 20261111

## Assumption checks
- normality (Shapiro-Wilk): passed=TRUE (no evidence against normality)
- sample size: passed=TRUE (adequate n)

## Power (sensitivity, not observed power)
- minimum detectable effect at n=24: Cohen's d = 0.5972
- to detect a conventional medium effect: 34 pairs to detect d=0.50
- alpha = 0.05, target power = 0.8

# Celeus - reproduce.R  (run from this package folder; bundled data in data/)
# Test: Paired t-test  |  engine 0.53.2  |  catalog c40-dc08
# Research context: not provided for this run.
# Reproduces the recorded result deterministically.

set.seed(20261111)
.data_path <- "data/ai-choose-statistical-test.csv"
if (!file.exists(.data_path)) stop("Run reproduce.R from the package folder (the bundled data is in data/).")
data <- readr::read_csv(.data_path, show_col_types = FALSE)  # needs the readr package
clean_cells <- function (df) 
{
    ctrl_collapse <- "[\r\n\t\v\f]+"
    zero_width_chars <- intToUtf8(c(8203L, 65279L), multiple = TRUE)
    uni_space_chars <- intToUtf8(c(160L, 8239L, 8199L), multiple = TRUE)
    clean_text <- function(x) {
        out <- x
        for (ch in zero_width_chars) out <- gsub(ch, "", out, 
            fixed = TRUE)
        out <- gsub(ctrl_collapse, " ", out)
        for (ch in uni_space_chars) out <- gsub(ch, " ", out, 
            fixed = TRUE)
        trimws(gsub(" +", " ", out))
    }
    changed_cols <- character(0)
    cells_cleaned <- 0L
    retyped <- character(0)
    for (j in seq_along(df)) {
        col <- df[[j]]
        if (!is.character(col)) 
            next
        cleaned <- clean_text(col)
        n_changed <- sum(!is.na(col) & !is.na(cleaned) & col != 
            cleaned, na.rm = TRUE)
        if (n_changed > 0L) {
            changed_cols <- c(changed_cols, names(df)[j])
            cells_cleaned <- cells_cleaned + n_changed
        }
        df[[j]] <- cleaned
        non_blank <- cleaned[!is.na(cleaned) & nzchar(cleaned)]
        if (length(non_blank) && all(grepl("^[+-]?(0|[1-9][0-9]*)([.][0-9]+)?$", 
            non_blank))) {
            parsed <- suppressWarnings(as.numeric(cleaned))
            was_blank <- is.na(cleaned) | !nzchar(cleaned)
            lost <- (is.na(parsed) | is.infinite(parsed)) & !was_blank
            if (!any(lost)) {
                finite <- parsed[!is.na(parsed)]
                all_whole <- all(finite == floor(finite))
                in_range <- all(abs(finite) <= .Machine$integer.max)
                df[[j]] <- if (all_whole && in_range) 
                  as.integer(parsed)
                else parsed
                retyped <- c(retyped, names(df)[j])
            }
        }
    }
    names(df) <- clean_text(names(df))
    if (length(changed_cols)) {
        attr(df, "ingest_clean_columns") <- changed_cols
        attr(df, "ingest_clean_cells_cleaned") <- cells_cleaned
    }
    if (length(retyped)) 
        attr(df, "ingest_clean_retyped") <- retyped
    df
}
data <- clean_cells(data)
# Complete-case analysis: drop rows missing any analysis value.
.acols <- c("after", "before")
data <- data[stats::complete.cases(data[, .acols, drop = FALSE]), , drop = FALSE]
result <- t.test(data[["after"]], data[["before"]], paired = TRUE)
print(result)

# Sensitivity: minimum detectable effect at n=24 (power=0.8, alpha=0.05). NOT observed power.
power <- pwr::pwr.t.test(n = 24, sig.level = 0.05, power = 0.8, type = "paired")
print(power)