Writing the statistical analysis methods section

The statistical analysis paragraph of a methods section tells a reader exactly what was done to the data, in enough detail that they could judge it and rerun it, and it contains no results. This guide gives a fill-in template, then builds one real paragraph from the record of a single R run so that every fact in it can be checked against a file.

The sentences you write after the analysis belong in the results section, covered in writing a results section and, for the general rules, reporting statistics in APA style.

What the paragraph has to contain

Reporting standards agree on the core. The APA’s journal article reporting standards for quantitative research (Appelbaum and colleagues, 2018) ask for the analytic strategy, the handling of missing data and the planned data diagnostics. The SAMPL guidelines (Lang and Altman, 2015) add the statistical software and the significance level. Put together, a reader needs these items:

1. The design and the outcome: what was compared, between or within subjects, and what was measured.
2. Each test, named exactly, with the model terms and, for sums-of-squares tests, the type.
3. The assumption checks, which check, on what quantity, and what you decided in advance if one failed.
4. The significance level and sidedness.
5. The effect size and its interval, not a p-value alone.
6. Multiplicity: which tests form one family, and the adjustment, or the statement that there is none.
7. Missing data and exclusions: how many rows, how they were handled.
8. The software, with versions, the packages that did the work and the random seed.

The first six are statistical decisions you make once and write down. The last two are facts about a run, and they are the ones people get wrong from memory.

A fill-in template

Replace each bracketed slot, and delete any sentence that does not apply to your study. Keep it in the past tense, because it describes what was done.

[Outcome] was compared across [factors and levels] with a [test name] including [terms], using [type] sums of squares. [Observations] were [independent / paired / clustered by X] because [design reason]. [Assumption] was checked with [test] on [the residuals / each group], and [assumption] with [test]; [plan if a check failed, written before the analysis]. Tests were two-sided at α = [level]. Effect sizes are [name] with [95%] confidence intervals. [Multiplicity statement]. [Missing-data statement]. Analyses were run in [software and version] using [packages], with random seed [seed].

Two sentences deserve a comment. The plan for a failed assumption has to exist before you see the data, otherwise it is a choice made after seeing which result you preferred. And the multiplicity sentence is the one most often missing, which is why the example below spends a section on it.

Worked example: a two-way ANOVA on ToothGrowth

The example uses ToothGrowth, a dataset that ships with R and comes from Crampton’s 1947 experiment. It records the length of the odontoblasts (the cells responsible for tooth growth) in 60 guinea pigs, 10 in each of six cells that cross delivery method (orange juice or ascorbic acid) with three doses (half a milligram, 1 and 2 mg a day). Each animal was measured once, so the design is a balanced, between-subjects 2 by 3 factorial.

The research question was whether the effect of delivery method on tooth length depends on the dose. The engine ran a two-way ANOVA with seed 20261101 and wrote a reproducible package: the data, the R script, the results and the settings. Every number below comes from that package, and the example is research framing on a public teaching dataset, not a claim about any treatment.

What the package recorded

The term tests, each with partial eta squared and its 95% interval, were:

The term tests are results, so none of them goes into the methods paragraph. What goes into the methods is everything that produced them, which is the first table.

Reading the record into sentences

Each sentence of the paragraph has one source in the package, and that is the useful discipline. A sentence with no source is a memory, and memories drift.

The script header names the engine version, and the script itself shows which R packages did the work: the ANOVA table comes from the car package, and the effect-size intervals from the effectsize package.

The assembled paragraph

Tooth length was compared across delivery method (orange juice, ascorbic acid) and dose (three levels) with a two-way between-subjects ANOVA including the interaction, using Type II sums of squares. Each of the 60 animals was measured once, in one of six cells of 10, so observations were treated as independent.

Normality of the residuals was checked with the Shapiro-Wilk test and equality of variances across the six cells with Levene’s test centred on the median (the Brown-Forsythe variant). Significance was set at α = .05. Effect sizes are partial eta squared with two-sided 95% confidence intervals. No observations were missing. Each of the three terms was tested at α = .05 without adjustment, because each addresses a separate question.

Analyses were run in R 4.6.2 using car 3.1-5 (ANOVA table), emmeans for the cell means and effectsize 1.0.2 (effect-size intervals), through the Celeus statistical engine, version 0.53.2, with random seed 20261101.

Three things are worth noticing. The paragraph says “the Shapiro-Wilk test”, where a paragraph written from memory often says only “normality was checked”. It names the quantity (the residuals for normality, the six cells for variances), because a normality check on the raw outcome across all six cells answers a different question. And it says what was done about multiplicity, even though the answer is “nothing”.

An F test has no direction to choose, so the paragraph states the significance level without calling the tests two-sided, and keeps “two-sided” for the intervals, which can be computed either way.

The multiplicity sentence has consequences

The sentence “each of the three terms was tested at α = .05 without adjustment” is a decision, and it changes how a reader treats the result. The interaction had p = 0.0219. If you instead treated the three terms as one family and applied a Bonferroni correction, the adjusted p-value would be about .066, which is above .05. That figure is the interaction’s p-value times three, not an engine output.

Unadjusted and Bonferroni-adjusted testing are both defensible. Treating each term of a factorial design as its own question is common practice, and the guide to multiple comparisons explains when a family is defined by the question and when by the study. The point for the methods paragraph is that the choice must be stated, so a reader can apply the other one if they prefer it.

What the paragraph does not say

The engine ran no post hoc comparisons for this model and calculated no power for the two-way ANOVA, so the paragraph is silent on both. A real study would add a sentence on how its sample size was chosen, and, if it followed up the interaction with pairwise comparisons, the named procedure and its family; the post hoc tests guide covers that sentence.

The paragraph also says nothing about what to do if an assumption check fails, because neither check did. In your own methods, write that rule before you look: for example, which robust or rank-based alternative you would use, decided in advance.

Sentence variants for the parts that change

The template’s slots hide the places where studies differ most. These variants are wording only, with no numbers, so adapt them to your own design.

Missing data

The ToothGrowth run had no missing values, so one short sentence was enough. When rows are missing, say how many, from which variables, and what you did about them.

Of [N] enrolled, [n] had a missing value in [variable] and were excluded from the model that used it (complete-case analysis); the amount of missing data was [description] and was not related to [variable] in [how you checked].

If you imputed, name the method, the number of imputed datasets and the variables in the imputation model. A reader cannot judge an imputation from the word “imputed”.

A fallback for a failed assumption

Write the plan before you analyse, and write it as a rule, not a promise to look.

If the residuals departed clearly from normality, or variances differed across groups, the primary analysis was to be repeated with [robust or rank-based alternative], and both results were to be reported.

If a check did fail and you switched, move the sentence to the past tense and add which check failed, so the switch is visible as data dependent.

Multiplicity, three ways

The [k] model terms were treated as separate questions, each tested at α = .05 with no adjustment.

The [k] pre-specified comparisons formed one family, and p-values were adjusted with the Holm procedure to control the family-wise error rate at .05.

Because the study was exploratory, no adjustment was made, and all p-values are descriptive; findings are interpreted as hypotheses.

The exploratory sentence is honest when it is true, and it should then be reflected in how the results are worded. The guide to multiple comparisons sets out the families and the procedures behind these three sentences.

What to record about the software

A version number is the cheapest part of the paragraph and the most often missing. Record four things.

1. The language or program and its version, for example R with its full version number.
2. The packages that did the work, because the same test can be implemented differently by two packages, and the same package can change between releases.
3. The tool that produced the analysis, if a program or service wrapped the code, with its version.
4. The seed, for any step that uses random numbers. If nothing random was involved, say nothing; an invented seed on a deterministic test misleads.

Do not copy the versions from the machine you are writing on. Take them from the session information the analysis saved when it ran, because that is the environment that produced the numbers. A package folder that carries its session information alongside the script makes this a copy-paste job instead of an act of memory.

A checklist before you submit

• Every test named in the paragraph appears in the results, and no test in the results is missing from the paragraph.
• The paragraph contains no numbers from the results, except the settings you chose (α, interval level, seed).
• The multiplicity sentence says what the family is, or says there is no adjustment.
• Each assumption check names the quantity it was run on.
• The software, its version and the seed match the run that produced the reported numbers, not a later rerun.
• The journal’s own author instructions have been checked, since many set a word limit or require a specific reporting guideline.

A paragraph typed from memory has three predictable faults. The version is the one on your laptop now, not the one used six months ago. The test is described as the one you planned, not the one you ran after the data forced a change. And the seed is missing, because nobody wrote it down. Two of the ten rules for reproducible computational research by Sandve and colleagues (2013) address the first and third directly: archive the exact versions of the external programs used, and note the random seeds of any analysis that involves randomness.

Copying the facts out of the run’s own record removes those faults. It does not remove the judgment calls: you still choose the multiplicity family, write the failure plan and decide what is worth a sentence. The guide to reproducible statistical analysis lists what a rerun needs beyond the paragraph.

Where each part goes, and in what tense

The statistical analysis paragraph is usually the last part of the methods section, after participants or materials and procedure. Keep three rules.

• Past tense. It describes what you did, such as “tooth length was compared”, not what the reader should do.
• No results. No means, no p-values, no intervals. The exception is a data-dependent decision, reported with a label saying so.
• Match the results section. Every test named in the methods appears in the results, and every test in the results was named in the methods. A reader who finds an unannounced test will ask why it was added.

The matching sentences for the results are in how to report ANOVA results, which uses the same ToothGrowth run to show the interaction-first write-up. For the effect size label, see the eta squared guide: a factorial design reports partial eta squared and says so.

Common mistakes

Naming the test but not the model. “A two-way ANOVA was used” does not say whether the interaction was included, or which type of sums of squares. With unequal cell sizes the types give different tests. The two-way ANOVA guide explains the difference, and the method reference for the two-way ANOVA lists what the engine fits.

Writing “assumptions were met”. It hides which assumptions, how they were checked and on what. A normality test has limited power to detect modest departures in a sample of this size, and a non-significant result is not evidence of normality; the post on parametric test assumptions discusses what to check and how.

Reporting only the p-value threshold. “Significant at p < .05” gives no effect size or interval. Lakens (2013) explains why effect sizes with intervals are needed alongside p-values, and Greenland and colleagues (2016) list the misreadings of a p-value that a bare threshold invites.

Leaving out the software version. “Analysed in R” is not enough. Name the version, the packages that did the test and the seed whenever randomness was involved.

Describing a plan you did not follow. If the analysis changed after looking at the data, the methods must say so. Silent changes are the ones reviewers find in the code and reasonably distrust.

Rerunning the example

The R script below the article reproduces the two-way ANOVA from the bundled data, with the seed set at the top, followed by the engine’s summary of the results. On the same package versions it recomputes the ANOVA table, the cell means and the partial eta squared values with their intervals, which you can compare with the recorded results below it. The script reproduces the ANOVA table and the effect sizes; the methods paragraph itself is written by hand from the record, and its sentences about design, independence and the multiplicity choice are the author’s decisions, not output.

# Results

- Test: Two-way ANOVA (factorial)
- n: 60
- statistic: 4.107
- p-value: 0.0219
- effect size: partial eta-squared = 0.132 (95% CI 0.001466 to 0.295)
  (two-sided CI at the run's confidence level)
- seed: 20261101

## Assumption checks
- normality (Shapiro-Wilk): passed=TRUE (no evidence against normality)
- variance homogeneity (Levene): passed=TRUE (no evidence of unequal variances)
- sample size: passed=TRUE (adequate n)

## Term tests
| Term | Test | Statistic | df | Residual df | p-value | Partial eta-squared (95% CI) |
|---|---|---|---|---|---|---|
| supp | F | 15.57 | 1 | 54 | < 0.001 | 0.2238 (0.05865 to 0.4021) |
| dose | F | 92 | 2 | 54 | < 0.001 | 0.7731 (0.6617 to 0.8382) |
| supp:dose | F | 4.107 | 2 | 54 | 0.0219 | 0.132 (0.001466 to 0.295) |

## Power
- Power is not calculated for this test.

# Celeus - reproduce.R  (run from this package folder; bundled data in data/)
# Test: Two-way ANOVA (factorial)  |  engine 0.53.2  |  catalog c40-dc08
# Research context: not provided for this run.
# Reproduces the recorded result deterministically.

set.seed(20261101)
.data_path <- "data/statistical-analysis-methods-section.csv"
if (!file.exists(.data_path)) stop("Run reproduce.R from the package folder (the bundled data is in data/).")
data <- readr::read_csv(.data_path, show_col_types = FALSE)  # needs the readr package
clean_cells <- function (df) 
{
    ctrl_collapse <- "[\r\n\t\v\f]+"
    zero_width_chars <- intToUtf8(c(8203L, 65279L), multiple = TRUE)
    uni_space_chars <- intToUtf8(c(160L, 8239L, 8199L), multiple = TRUE)
    clean_text <- function(x) {
        out <- x
        for (ch in zero_width_chars) out <- gsub(ch, "", out, 
            fixed = TRUE)
        out <- gsub(ctrl_collapse, " ", out)
        for (ch in uni_space_chars) out <- gsub(ch, " ", out, 
            fixed = TRUE)
        trimws(gsub(" +", " ", out))
    }
    changed_cols <- character(0)
    cells_cleaned <- 0L
    retyped <- character(0)
    for (j in seq_along(df)) {
        col <- df[[j]]
        if (!is.character(col)) 
            next
        cleaned <- clean_text(col)
        n_changed <- sum(!is.na(col) & !is.na(cleaned) & col != 
            cleaned, na.rm = TRUE)
        if (n_changed > 0L) {
            changed_cols <- c(changed_cols, names(df)[j])
            cells_cleaned <- cells_cleaned + n_changed
        }
        df[[j]] <- cleaned
        non_blank <- cleaned[!is.na(cleaned) & nzchar(cleaned)]
        if (length(non_blank) && all(grepl("^[+-]?(0|[1-9][0-9]*)([.][0-9]+)?$", 
            non_blank))) {
            parsed <- suppressWarnings(as.numeric(cleaned))
            was_blank <- is.na(cleaned) | !nzchar(cleaned)
            lost <- (is.na(parsed) | is.infinite(parsed)) & !was_blank
            if (!any(lost)) {
                finite <- parsed[!is.na(parsed)]
                all_whole <- all(finite == floor(finite))
                in_range <- all(abs(finite) <= .Machine$integer.max)
                df[[j]] <- if (all_whole && in_range) 
                  as.integer(parsed)
                else parsed
                retyped <- c(retyped, names(df)[j])
            }
        }
    }
    names(df) <- clean_text(names(df))
    if (length(changed_cols)) {
        attr(df, "ingest_clean_columns") <- changed_cols
        attr(df, "ingest_clean_cells_cleaned") <- cells_cleaned
    }
    if (length(retyped)) 
        attr(df, "ingest_clean_retyped") <- retyped
    df
}
data <- clean_cells(data)
# Complete-case analysis: drop rows missing any analysis value.
.acols <- c("len", "supp", "dose")
data <- data[stats::complete.cases(data[, .acols, drop = FALSE]), , drop = FALSE]
result <- { .d <- data; .d[["supp"]] <- as.factor(.d[["supp"]]); .d[["dose"]] <- as.factor(.d[["dose"]]); .fit <- stats::lm(`len` ~ `supp` * `dose`, data = .d); .an <- car::Anova(.fit, type = 2); .em <- emmeans::emmeans(.fit, specs = c("supp", "dose")); list(anova = .an, emmeans = summary(.em)) }
print(result)

# Effect size (partial_eta_sq) and its confidence interval.
effect <- { .es_d <- data; .es_d[["supp"]] <- as.factor(.es_d[["supp"]]); .es_d[["dose"]] <- as.factor(.es_d[["dose"]]); .es_an <- car::Anova(stats::lm(`len` ~ `supp` * `dose`, data = .es_d), type = 2); effectsize::F_to_eta2(.es_an[["F value"]][seq_len(nrow(.es_an) - 1L)], .es_an[["Df"]][seq_len(nrow(.es_an) - 1L)], .es_an[["Df"]][nrow(.es_an)], ci = 0.95, alternative = "two.sided") }
print(effect)