Reproducible research checklist for reviewers

A reviewer who is handed code and data has one job that the manuscript cannot do for them: find out whether the reported numbers come out of the files. This post is a checklist for that job. It has ten items, each with what to look for, what a pass looks like, and what a failure looks like, and every item that a worked example can show is checked on a real package from one small analysis, so you can see the evidence rather than take the description on trust.

The checklist is for the person doing the rerun. If you are the author, the guide to reproducible statistical analysis explains why each item exists and how to build the package. Here the question is narrower: what do I tick, and what do I look at to tick it?

What the checklist is checking

The National Academies (2019) define reproducibility as obtaining consistent results from the same data with the same code and methods. That is all this checklist tests. It does not test whether the study would replicate, and it does not test whether the model was the right one. Peng (2011) argues that reproducibility is the minimum standard when full replication is out of reach, and full replication is often out of reach. The practical consequence is simple: an analysis nobody can rerun cannot be checked by anyone.

The need is real. Stodden, Seiler and Ma (2018) sampled papers from a journal that required authors to share data and code on request. They obtained materials for 44% of the sampled papers and could reproduce the findings of 26%. Hardwicke and colleagues (2018) found that even after a journal made data sharing mandatory, in 13 of 35 checked articles at least one value could not be reproduced even with the authors’ help, and 11 of the 22 fully reproduced articles needed that help. A checklist cannot fix that, but it makes the gaps visible in the first hour instead of the third week.

The example package

Every item below is checked on one package. The analysis is a two-way ANOVA on R’s built-in ToothGrowth data from Crampton (1947): the length of the odontoblasts (the cells that form teeth) in 60 guinea pigs, by delivery method (orange juice or ascorbic acid) and vitamin C dose (half a milligram, 1 or 2 mg a day). Each animal was measured once, so the design is a balanced 2 by 3 between-subjects factorial with independent observations. The question is whether the effect of delivery method depends on the dose. The two-way ANOVA guide and the method reference cover the method; this post only uses it as something to check.

A small, fully analytic example is chosen on purpose. Nothing in it draws random numbers, so it makes one point clearly that a bootstrap example would blur: a seed can be correctly recorded and still change nothing, and the reviewer’s task is to confirm it is declared, not to expect it to matter. The package was written by the Celeus engine, and for this post the analysis was run twice in a sealed container and the two outputs compared before anything was published. That is the author’s check. The rest of this post is the reviewer’s.

The checklist, item by item

1. The analysis dataset is in the package

Look for: the exact rows the model saw, not the raw export and not “the cleaned file”. Pass: the data file is there and its row count matches the sample size in the results. Fail: the script reads a file that is not in the package, or the row count differs and nothing explains why.

On the example, the results record 60 rows and the missing-data record shows a complete-case strategy with 0 rows dropped, so the 60 guinea pigs in the dataset are the 60 animals analysed. A reviewer can open the data file, count the rows, and see that no exclusion happened between the data and the model. If rows had been dropped, that same record would say how many.

2. The data have a fingerprint

Look for: a checksum of each data file, recorded next to the results. Pass: you compute the checksum of the file you received and it matches. Fail: nothing is recorded, or the values differ.

The example package does not record a checksum, and that is worth saying plainly because a checklist that only reports passes is not much use. This item is partial: the reviewer computes tools::md5sum("data/reproducible-research-checklist.csv") on the received file, writes the value down, and asks the author to confirm it. Until someone has the author’s value, “the same data” is an assumption. Report it as supplied by you, not as checked.

3. One script, run from the package folder

Look for: a single script that reads the data with a relative path and runs top to bottom. Pass: it runs in a fresh session with nothing else loaded. Fail: it needs objects from an earlier session, an absolute path from the author’s machine, or console commands typed afterwards.

The example’s script reads data/ relative to the package folder and stops with an explicit message if it is run from anywhere else, which turns the most common failure into a clear error rather than a silent wrong answer. It then fits the model, prints the ANOVA table and the effect sizes, and nothing else is needed beyond the packages it calls (readr, car, emmeans and effectsize). Restart the session, run it from the folder, and the output appears. An error at this step is a finding; do not patch it quietly.

4. The seed is declared

Look for: a seed set before any random step, matching the seed recorded in the results. Pass: the script’s set.seed() value equals the recorded one. Fail: a random step with no seed, or a seed set after it.

The example sets set.seed(20261111) at the top, and the results file records the same value as the seed used. That is a pass on declaration. It is not a pass on sensitivity, because a two-way ANOVA with analytic confidence intervals draws no random numbers: change the seed and every number stays put. For an analysis with a bootstrap or permutation step, rerun with a different seed once and see which numbers move; the guide to reproducible analysis does exactly that on a Kruskal-Wallis example.

5. The software environment is recorded

Look for: session information listing the R version and every package version, and ideally a lockfile that can reinstall them. Pass: both exist and you can build the environment from them. Fail: “R 4” and a list of package names.

The files published with this post include a session information file with the R version and each package version used for the run. Read it before you run anything, so that if your numbers differ you already know which versions differ. Sandve and colleagues (2013) make the same point as one of their ten rules: archive the exact versions of the external programs you used. The full package a Celeus run produces also carries a lockfile, but the files here do not, so on this example item 5 is partial: you can see the versions, but you rebuild the environment yourself. Without a lockfile you can still rerun, but you cannot claim the environment was the same.

6. The analytic decisions are written down

Look for: what was chosen, and why, in a place a reviewer can read. Pass: each choice that could change the result is visible. Fail: the code shows what ran but nothing says why.

The package makes several choices visible even without a narrative.

The script fits len ~ supp * dose with both predictors converted to factors, and tests each term with Type II sums of squares. The results record a complete-case strategy and a two-sided 95% interval. One of those is worth a reviewer’s pause: dose is a quantity (half a milligram, 1 mg, 2 mg), and the script treats it as three categories. That is a legitimate choice for a three-level design, but it is a choice, and only the script lets you see it. The files here show what was chosen but not why, so this item is partial. The reason belongs in the methods section, and checking the paper against the files is item 10.

7. The assumption checks are in the output

Look for: the checks the author ran, with their results, in the recorded output. Pass: the checks are there and the text does not over-read them. Fail: “assumptions were met” with no numbers.

The example records a Shapiro-Wilk test of the residuals (p = 0.669) and a Levene test of equal variances (p = 0.148), both reporting no evidence against the assumption, plus a sample size check that passed. A reviewer should read these as absence of evidence at n = 60, not as proof the assumptions hold, and a wording like “the data were normal” would be an over-read. Whether the checks were the right ones for the design is the statistical review, which sits next to this checklist rather than inside it.

8. The results are saved in full precision

Look for: a machine-readable results file, not just a rounded table. Pass: you can compare every number at full precision. Fail: only a manuscript table, so “matches” means “matches after rounding”.

The example’s results file stores the interaction statistic as 4.1069910940225203 and its p-value as 0.021860268964790996, where the manuscript-style line says 4.107 and 0.0219. Both are right; they are the same number at two precisions. Rounding hides exactly the small discrepancies that signal a changed dataset or package, so compare the rerun to the long form first and the short form second.

9. The rerun matches the recorded results

Look for: the rerun’s statistic, p-value and effect size against the recorded ones. Pass: agreement to the recorded precision in the same environment. Fail: any difference in a digit that the manuscript reports.

The rerun comparison is the core of the test. On the example, the interaction between delivery method and dose gives F(2, 54) = 4.107, p = 0.0219, with partial eta squared 0.132 (95% CI 0.001466 to 0.295). The main effects are F(1, 54) = 15.57 for delivery method and F(2, 54) = 92.00 for dose, both p < 0.001. The author’s side of this check was automated for this post: the package’s own script was rerun and its statistic and p-value confirmed against the recorded ones. The reviewer repeats it by hand, number by number, and the interval is a good one to include because the lower bound is close to zero.

10. The manuscript matches the results file

Look for: every number in the paper, traced to the recorded output. Pass: each one is found, at the reported precision. Fail: a number typed by hand that differs from the output.

Tracing the manuscript is the step most people skip, because the rerun feels like the end. It is not: a rerun that matches the results file only shows the code is sound, while the manuscript can still carry a transposed digit or a figure from an earlier model. Read the paper’s results paragraph against the file line by line. Be especially careful with effect sizes and intervals, which are often copied from a draft.

Reading the result of the checklist

Tally the ten items as pass, partial or fail, and say which you checked yourself. The example can be scored on nine items, because item 10 needs a manuscript and a worked example has none. It passes six and is partial on three: the checksum is not recorded, so the data fingerprint is yours to supply; the versions are listed but there is no lockfile; and the choices are visible but their reasons are not. That is a typical honest outcome, and it is more useful to an editor than “reproducible”.

Two cautions. First, a full set of passes does not make the conclusion correct. The interaction’s confidence interval runs from 0.001 to 0.295, so the data are compatible with an interaction that accounts for almost none of the remaining variation or for nearly 30% of it, and reproducing the number says nothing about which. Second, a failure is not an accusation. Most failures are mundane, such as a missing file or a path from the author’s laptop, and the useful report says exactly which item failed and what you needed to proceed.

When the data cannot be shared

Some data cannot leave the institution that holds them. The checklist still applies, with items 1 and 2 changed: the reviewer cannot hold the file, so ask for the checksum, a description of the columns, the access route and a small synthetic stand-in with the same structure. The stand-in lets you run item 3 and confirm the script executes, but it cannot reproduce the published numbers, and the package should say so. Items 4 to 8 do not need the records at all, and they are often where the gaps are.

Where this checklist stops

The checklist covers the computation, which is the part a reviewer can settle from the files alone. Marwick, Boettiger and Mullen (2018) describe the fuller idea as a research compendium: data, code and documentation organised so that others can inspect, reproduce and extend the work. A compendium is a good target for authors, but a reviewer does not need one to start. Ten ticks are enough to say whether the numbers come out of the files.

What the checklist does not cover is the quality of the analysis. Whether a two-way ANOVA was the right choice for this design is a separate question, and the guide to choosing a statistical test covers how to answer it. Upstream of both, the data cleaning checklist covers the steps that should already be in the script before a reviewer sees it. For the same ToothGrowth data read from the effect-size side, see eta squared and partial eta squared, and for how the design itself works, factorial designs and blocking.

How the example package was produced

The example was run through the Celeus engine with a fixed seed. The engine wrote the script, the results and the run record. The whole analysis ran twice in a sealed container with no network, and the example was refused unless the two outputs were identical. The package’s own script was then rerun and its statistic and p-value checked against the recorded ones. The description of the reproducible package lists every file a full package contains; this post publishes the script, the data file, the results file and the session information. As the checklist above shows, the package does not record a checksum of the data file, so that is one item you compute yourself. Reproducible is not the same as correct, and nothing in a package changes that.

# Results

- Test: Two-way ANOVA (factorial)
- n: 60
- statistic: 4.107
- p-value: 0.0219
- effect size: partial eta-squared = 0.132 (95% CI 0.001466 to 0.295)
  (two-sided CI at the run's confidence level)
- seed: 20261111

## Assumption checks
- normality (Shapiro-Wilk): passed=TRUE (no evidence against normality)
- variance homogeneity (Levene): passed=TRUE (no evidence of unequal variances)
- sample size: passed=TRUE (adequate n)

## Term tests
| Term | Test | Statistic | df | Residual df | p-value | Partial eta-squared (95% CI) |
|---|---|---|---|---|---|---|
| supp | F | 15.57 | 1 | 54 | < 0.001 | 0.2238 (0.05865 to 0.4021) |
| dose | F | 92 | 2 | 54 | < 0.001 | 0.7731 (0.6617 to 0.8382) |
| supp:dose | F | 4.107 | 2 | 54 | 0.0219 | 0.132 (0.001466 to 0.295) |

## Power
- Power is not calculated for this test.

# Celeus - reproduce.R  (run from this package folder; bundled data in data/)
# Test: Two-way ANOVA (factorial)  |  engine 0.53.2  |  catalog c40-dc08
# Research context: not provided for this run.
# Reproduces the recorded result deterministically.

set.seed(20261111)
.data_path <- "data/reproducible-research-checklist.csv"
if (!file.exists(.data_path)) stop("Run reproduce.R from the package folder (the bundled data is in data/).")
data <- readr::read_csv(.data_path, show_col_types = FALSE)  # needs the readr package
clean_cells <- function (df) 
{
    ctrl_collapse <- "[\r\n\t\v\f]+"
    zero_width_chars <- intToUtf8(c(8203L, 65279L), multiple = TRUE)
    uni_space_chars <- intToUtf8(c(160L, 8239L, 8199L), multiple = TRUE)
    clean_text <- function(x) {
        out <- x
        for (ch in zero_width_chars) out <- gsub(ch, "", out, 
            fixed = TRUE)
        out <- gsub(ctrl_collapse, " ", out)
        for (ch in uni_space_chars) out <- gsub(ch, " ", out, 
            fixed = TRUE)
        trimws(gsub(" +", " ", out))
    }
    changed_cols <- character(0)
    cells_cleaned <- 0L
    retyped <- character(0)
    for (j in seq_along(df)) {
        col <- df[[j]]
        if (!is.character(col)) 
            next
        cleaned <- clean_text(col)
        n_changed <- sum(!is.na(col) & !is.na(cleaned) & col != 
            cleaned, na.rm = TRUE)
        if (n_changed > 0L) {
            changed_cols <- c(changed_cols, names(df)[j])
            cells_cleaned <- cells_cleaned + n_changed
        }
        df[[j]] <- cleaned
        non_blank <- cleaned[!is.na(cleaned) & nzchar(cleaned)]
        if (length(non_blank) && all(grepl("^[+-]?(0|[1-9][0-9]*)([.][0-9]+)?$", 
            non_blank))) {
            parsed <- suppressWarnings(as.numeric(cleaned))
            was_blank <- is.na(cleaned) | !nzchar(cleaned)
            lost <- (is.na(parsed) | is.infinite(parsed)) & !was_blank
            if (!any(lost)) {
                finite <- parsed[!is.na(parsed)]
                all_whole <- all(finite == floor(finite))
                in_range <- all(abs(finite) <= .Machine$integer.max)
                df[[j]] <- if (all_whole && in_range) 
                  as.integer(parsed)
                else parsed
                retyped <- c(retyped, names(df)[j])
            }
        }
    }
    names(df) <- clean_text(names(df))
    if (length(changed_cols)) {
        attr(df, "ingest_clean_columns") <- changed_cols
        attr(df, "ingest_clean_cells_cleaned") <- cells_cleaned
    }
    if (length(retyped)) 
        attr(df, "ingest_clean_retyped") <- retyped
    df
}
data <- clean_cells(data)
# Complete-case analysis: drop rows missing any analysis value.
.acols <- c("len", "supp", "dose")
data <- data[stats::complete.cases(data[, .acols, drop = FALSE]), , drop = FALSE]
result <- { .d <- data; .d[["supp"]] <- as.factor(.d[["supp"]]); .d[["dose"]] <- as.factor(.d[["dose"]]); .fit <- stats::lm(`len` ~ `supp` * `dose`, data = .d); .an <- car::Anova(.fit, type = 2); .em <- emmeans::emmeans(.fit, specs = c("supp", "dose")); list(anova = .an, emmeans = summary(.em)) }
print(result)

# Effect size (partial_eta_sq) and its confidence interval.
effect <- { .es_d <- data; .es_d[["supp"]] <- as.factor(.es_d[["supp"]]); .es_d[["dose"]] <- as.factor(.es_d[["dose"]]); .es_an <- car::Anova(stats::lm(`len` ~ `supp` * `dose`, data = .es_d), type = 2); effectsize::F_to_eta2(.es_an[["F value"]][seq_len(nrow(.es_an) - 1L)], .es_an[["Df"]][seq_len(nrow(.es_an) - 1L)], .es_an[["Df"]][nrow(.es_an)], ci = 0.95, alternative = "two.sided") }
print(effect)