Celeus engine validation report
category: validation audience: public status: current owner: TBD (owner)
This report is generated, not written by hand, and is byte-for-byte reproducible on the pinned engine (R 4.6.1 + a pinned dependency lockfile): it carries no timestamp, so re-running it and finding no diff is itself a demonstration of the determinism this product promises. Every dataset here is public and citable; no patient data is used anywhere.
What this is: evidence that the engine computes the right numbers and steers to a defendable test, on well-known public datasets, checked two independent ways. What this is NOT: a substitute for review by an independent statistician (the named next step, below), and NOT clinical decision support - Celeus is a research-statistics tool.
How to read the three tiers
Each result is checked at up to three levels of rigor:
- Canonical agreement (all cases). The engine’s number equals an INDEPENDENT computation in base R / survival / lme4 / pROC, to a tolerance of 1e-06. This catches the class of bug that actually occurs - spec routing, coercion, NA handling, serialization - the statistics themselves are R’s.
- Application-path agreement. The same equality through the full upload -> analyse -> report path, CI-gated in our automated test suite (not re-run here; asserted on every push).
- Literature anchor (a subset). The engine’s number also equals a value published in a citable source (a paper, textbook, or the dataset’s own package vignette) - a checksum a script cannot accidentally regenerate wrong. Only some cases have a single citable published number; the rest are honestly labelled canonical-only, which is a real check, just not a literature one.
Reproducibility stamp
| field | value |
|---|---|
| R version | R version 4.6.1 |
| pinned dependency lockfile md5 | dc02556bdb795e33269d37da113d908a |
| catalog version | c32-a6f7 |
| engine version | 0.1.0 |
Summary
- 40 cases across 32 catalog test types, 68 numeric comparisons total.
- 10 cases are additionally anchored to a published literature value; 30 are canonical-only (labelled per-case below).
- 0 unexplained divergences (a non-zero count here fails the generator).
- Test selection: the correct test family was proposed in 34 of 34 evaluated cases (directly, or via a documented post-analysis follow-up - see below); the grader recommended a sound alternative in 12 of 34; the test actually used was itself graded “Not appropriate” in 0 of 34. (6 of 40 total cases skipped this evaluation - external-downloaded, fixture not present; see “Dataset sources” above.)
- Reproducibility is separately CI-gated, two ways. Over a 34-case builtin corpus: each analysis re-runs its emitted
reproduce.Rstandalone, and re-analysing the package’s own bundled dataset must return every recorded number bit-for-bit. Across processes: the same analysis run in two independent operating-system processes must emit a byte-identical package - every file compared by hash, with nothing excluded. That proves reproducibility, not correctness - correctness is the tables below.
Dataset sources
Every case above draws on one of three source types:
| source type | cases |
|---|---|
| bundled package (a CRAN/base-R package, ships with the pinned dependency set - no network) | 32 |
| synthetic/illustrative (a documented worked example, NOT real patient data - honestly labelled) | 2 |
| externally downloaded (an opt-in network fetch, run weekly on a schedule - not part of this report’s own reproducibility check) | 6 |
External-sourced cases not run in THIS report
This report is generated network-free, to keep it byte-reproducible (see the reproducibility stamp above) - it never downloads external datasets. Those cases run on a separate weekly schedule instead:
| case | dataset | method |
|---|---|---|
| heart-logistic_regression | UCI Heart Disease (Cleveland) | logistic_regression |
| cervical-hinselmann-diagnostic | UCI Cervical Cancer (Hinselmann vs Biopsy) | diagnostic_accuracy |
| cervical-schiller-diagnostic | UCI Cervical Cancer (Schiller vs Biopsy) | diagnostic_accuracy |
| cervical-citology-diagnostic | UCI Cervical Cancer (Citology vs Biopsy) | diagnostic_accuracy |
| cervical-hinselmann-vs-schiller-mcnemar | UCI Cervical Cancer (Hinselmann vs Schiller, two index tests on the same patients) | mcnemar |
| cervical-schiller-vs-citology-mcnemar | UCI Cervical Cancer (Schiller vs Citology, two index tests on the same patients) | mcnemar |
Numeric agreement (all cases vs an independent computation)
| case | dataset (field) | Celeus | independent R | match |
|---|---|---|---|---|
| sleep-paired_t | datasets::sleep (p_value) | 0.00283289 | 0.00283289 | yes |
| sleep-paired_t | datasets::sleep (estimate) | -1.58 | -1.58 | yes |
| sleep-wilcoxon | datasets::sleep (p_value) | 0.00390625 | 0.00390625 | yes |
| ToothGrowth-welch_t | datasets::ToothGrowth (p_value) | 0.0606345 | 0.0606345 | yes |
| ToothGrowth-kruskal_wallis | datasets::ToothGrowth (p_value) | 1.47521e-09 | 1.47521e-09 | yes |
| iris-welch_anova | datasets::iris (p_value) | 1.50506e-28 | 1.50506e-28 | yes |
| mtcars-pearson | datasets::mtcars (p_value) | 1.29396e-10 | 1.29396e-10 | yes |
| mtcars-pearson | datasets::mtcars (estimate) | -0.867659 | -0.867659 | yes |
| mtcars-spearman | datasets::mtcars (p_value) | 1.48759e-11 | 1.48759e-11 | yes |
| mtcars-linear_regression | datasets::mtcars (estimate) | -3.87783 | -3.87783 | yes |
| mtcars-linear_regression | datasets::mtcars (p_value) | 1.11965e-06 | 1.11965e-06 | yes |
| mtcars-logistic_regression | datasets::mtcars (estimate) | -8.08348 | -8.08348 | yes |
| mtcars-logistic_regression | datasets::mtcars (p_value) | 0.00843381 | 0.00843381 | yes |
| mtcars-fisher_exact | datasets::mtcars (p_value) | 0.472697 | 0.472697 | yes |
| titanic-chi_square | carData::TitanicSurvival (p_value) | 4.58992e-81 | 4.58992e-81 | yes |
| birthwt-logistic_regression | MASS::birthwt (estimate) | -0.0389946 | -0.0389946 | yes |
| birthwt-logistic_regression | MASS::birthwt (p_value) | 0.23344 | 0.23344 | yes |
| sleepstudy-mixed_model | lme4::sleepstudy (estimate) | 10.4673 | 10.4673 | yes |
| airquality-mann_whitney-filtered | datasets::airquality (p_value) | 6.10874e-05 | 6.10874e-05 | yes |
| lung-kaplan_meier | survival::lung (status recoded 0/1) (p_value) | 0.00131116 | 0.00131116 | yes |
| lung-cox_ph | survival::lung (status recoded 0/1) (estimate) | 1.01719 | 1.01719 | yes |
| lung-cox_ph | survival::lung (status recoded 0/1) (p_value) | 0.064591 | 0.064591 | yes |
| lung-aft_weibull | survival::lung (status recoded 0/1) (estimate) | 0.987818 | 0.987818 | yes |
| lung-aft_weibull | survival::lung (status recoded 0/1) (p_value) | 0.0781189 | 0.0781189 | yes |
| mcnemar-disease-example | synthetic fixture from the McNemar’s test Wikipedia worked example (statistic) | 20.6722 | 20.6722 | yes |
| mcnemar-disease-example | synthetic fixture from the McNemar’s test Wikipedia worked example (p_value) | 5.45009e-06 | 5.45009e-06 | yes |
| respiratory-mcnemar | HSAUR::respiratory (baseline vs month-4 status) (statistic) | 1.93939 | 1.93939 | yes |
| respiratory-mcnemar | HSAUR::respiratory (baseline vs month-4 status) (p_value) | 0.163734 | 0.163734 | yes |
| anorexia-ancova | MASS::anorexia (statistic) | 7.86808 | 7.86808 | yes |
| anorexia-ancova | MASS::anorexia (p_value) | 0.00084384 | 0.00084384 | yes |
| warpbreaks-poisson_regression | datasets::warpbreaks (estimate) | -0.205988 | -0.205988 | yes |
| warpbreaks-poisson_regression | datasets::warpbreaks (p_value) | 6.48978e-05 | 6.48978e-05 | yes |
| quine-negbin_regression | MASS::quine (estimate) | -0.569372 | -0.569372 | yes |
| quine-negbin_regression | MASS::quine (p_value) | 0.00020458 | 0.00020458 | yes |
| housing-ordinal_logistic | MASS::housing (expanded one row per Freq) (estimate) | 0.566394 | 0.566394 | yes |
| housing-ordinal_logistic | MASS::housing (expanded one row per Freq) (p_value) | 6.22819e-08 | 6.22819e-08 | yes |
| housing-ordinal_logistic | MASS::housing (expanded one row per Freq) (residual_deviance) | 3479.15 | 3479.15 | yes |
| warpbreaks-two_way_anova | datasets::warpbreaks (statistic) | 4.18907 | 4.18907 | yes |
| warpbreaks-two_way_anova | datasets::warpbreaks (p_value) | 0.0210442 | 0.0210442 | yes |
| iris-multinomial_logistic | datasets::iris (estimate) | -9.74099 | -9.74099 | yes |
| iris-multinomial_logistic | datasets::iris (p_value) | 0.411011 | 0.411011 | yes |
| iris-multinomial_logistic | datasets::iris (residual_deviance) | 23.8612 | 23.8612 | yes |
| PlantGrowth-one_way_anova | datasets::PlantGrowth (statistic) | 4.84609 | 4.84609 | yes |
| PlantGrowth-one_way_anova | datasets::PlantGrowth (p_value) | 0.01591 | 0.01591 | yes |
| ToothGrowth-student_t | datasets::ToothGrowth (statistic) | 1.91527 | 1.91527 | yes |
| ToothGrowth-student_t | datasets::ToothGrowth (p_value) | 0.0603934 | 0.0603934 | yes |
| ToothGrowth-student_t | datasets::ToothGrowth (estimate) | 3.7 | 3.7 | yes |
| VerbAgg-mixed_logistic | lme4::VerbAgg (estimate) | matches refit (not frozen) | (independent refit) | yes |
| VerbAgg-mixed_logistic | lme4::VerbAgg (p_value) | matches refit (not frozen) | (independent refit) | yes |
| epil-mixed_poisson | MASS::epil (estimate) | matches refit (not frozen) | (independent refit) | yes |
| epil-mixed_poisson | MASS::epil (p_value) | matches refit (not frozen) | (independent refit) | yes |
| ergoStool-rm_anova | nlme::ergoStool (statistic) | 22.3556 | 22.3556 | yes |
| ergoStool-rm_anova | nlme::ergoStool (p_value) | 3.00556e-05 | 3.00556e-05 | yes |
| ergoStool-rm_anova | nlme::ergoStool (estimate) | 0.459375 | 0.459375 | yes |
| aSAH-roc_auc | pROC::aSAH (auc) | 0.731369 | 0.731369 | yes |
| aSAH-roc_compare | pROC::aSAH (s100b vs ndka) (auc1) | 0.731369 | 0.731369 | yes |
| aSAH-roc_compare | pROC::aSAH (s100b vs ndka) (auc2) | 0.611958 | 0.611958 | yes |
| aSAH-roc_compare | pROC::aSAH (s100b vs ndka) (auc_diff) | 0.119411 | 0.119411 | yes |
| aSAH-roc_compare | pROC::aSAH (s100b vs ndka) (roc_test_p) | 0.164295 | 0.164295 | yes |
| aSAH-roc_compare | pROC::aSAH (s100b vs ndka) (roc_test_stat) | 1.39077 | 1.39077 | yes |
| screening-diagnostic_accuracy | documented illustrative 2x2 screening fixture (sensitivity) | 0.95 | 0.95 | yes |
| screening-diagnostic_accuracy | documented illustrative 2x2 screening fixture (specificity) | 0.9 | 0.9 | yes |
| screening-diagnostic_accuracy | documented illustrative 2x2 screening fixture (ppv) | 0.513514 | 0.513514 | yes |
| screening-diagnostic_accuracy | documented illustrative 2x2 screening fixture (npv) | 0.993865 | 0.993865 | yes |
| screening-diagnostic_accuracy | documented illustrative 2x2 screening fixture (prevalence) | 0.1 | 0.1 | yes |
| davis-bland_altman | carData::Davis (measured vs self-reported weight) (bias) | 0.601093 | 0.601093 | yes |
| davis-bland_altman | carData::Davis (measured vs self-reported weight) (loa_lower) | -15.969 | -15.969 | yes |
| davis-bland_altman | carData::Davis (measured vs self-reported weight) (loa_upper) | 17.1711 | 17.1711 | yes |
“matches refit (not frozen)” marks iterative-optimizer fits (glmer mixed models) whose engine
value equals an independent refit on every run, but whose absolute value carries platform/optimizer
jitter below the agreement tolerance. The verdict is stable and reported; the raw number is not
printed because it is not byte-reproducible (which is exactly why these cases are canonical-only).
Literature-anchored cases
These cases additionally match a citable published value:
| case | field | Celeus | published value | citation | match |
|---|---|---|---|---|---|
| sleep-paired_t | p_value | 0.00283289 | 0.00283289 | Student (1908), Biometrika 6(1):1-25 | yes |
| mtcars-pearson | estimate | -0.867659 | -0.867659 | Henderson & Velleman (1981), Biometrics 37 | yes |
| lung-kaplan_meier | p_value | 0.00131116 | 0.00131116 | Loprinzi et al. (1994), J. Clinical Oncology 12(3) | yes |
| mcnemar-disease-example | statistic | 20.6722 | 20.6722 | Wikipedia contributors, “McNemar’s test” (worked example: n=314 patients, disease diagnosis before/after a drug), retrieved 2026-07-13 | yes |
| mcnemar-disease-example | p_value | 5.45009e-06 | 5.45009e-06 | Wikipedia contributors, “McNemar’s test” (worked example: n=314 patients, disease diagnosis before/after a drug), retrieved 2026-07-13 | yes |
| PlantGrowth-one_way_anova | statistic | 4.84609 | 4.84609 | Dobson (1983), An Introduction to Statistical Modelling - the textbook one-way ANOVA | yes |
| PlantGrowth-one_way_anova | p_value | 0.01591 | 0.01591 | Dobson (1983), An Introduction to Statistical Modelling - the textbook one-way ANOVA | yes |
| ToothGrowth-student_t | statistic | 1.91527 | 1.91527 | Crampton (1947), J. Nutrition 33(5) | yes |
| ToothGrowth-student_t | p_value | 0.0603934 | 0.0603934 | Crampton (1947), J. Nutrition 33(5) | yes |
| ToothGrowth-student_t | estimate | 3.7 | 3.7 | Crampton (1947), J. Nutrition 33(5) | yes |
| ergoStool-rm_anova | statistic | 22.3556 | 22.3556 | Pinheiro & Bates (2000), Mixed-Effects Models in S and S-PLUS | yes |
| ergoStool-rm_anova | p_value | 3.00556e-05 | 3.00556e-05 | Pinheiro & Bates (2000), Mixed-Effects Models in S and S-PLUS | yes |
| ergoStool-rm_anova | estimate | 0.459375 | 0.459375 | Pinheiro & Bates (2000), Mixed-Effects Models in S and S-PLUS | yes |
| aSAH-roc_auc | auc | 0.731369 | 0.7314 | Turck N et al. (2010), J. Proteome Research 9(11); the aSAH dataset shipped and validated by the pROC package (Robin et al. 2011, BMC Bioinformatics 12:77) | yes |
| aSAH-roc_compare | auc1 | 0.731369 | 0.7314 | Turck N et al. (2010), J. Proteome Research 9(11); the two-marker DeLong comparison is pROC’s own roc.test worked example (Robin et al. 2011, BMC Bioinformatics 12:77) | yes |
| aSAH-roc_compare | auc2 | 0.611958 | 0.612 | Turck N et al. (2010), J. Proteome Research 9(11); the two-marker DeLong comparison is pROC’s own roc.test worked example (Robin et al. 2011, BMC Bioinformatics 12:77) | yes |
| aSAH-roc_compare | roc_test_p | 0.164295 | 0.164295 | Turck N et al. (2010), J. Proteome Research 9(11); the two-marker DeLong comparison is pROC’s own roc.test worked example (Robin et al. 2011, BMC Bioinformatics 12:77) | yes |
| screening-diagnostic_accuracy | sensitivity | 0.95 | 0.95 | Illustrative screening-test 2x2 (TP=95, FP=90, FN=5, TN=810); textbook-style table, not real patient data | yes |
| screening-diagnostic_accuracy | specificity | 0.9 | 0.9 | Illustrative screening-test 2x2 (TP=95, FP=90, FN=5, TN=810); textbook-style table, not real patient data | yes |
| screening-diagnostic_accuracy | prevalence | 0.1 | 0.1 | Illustrative screening-test 2x2 (TP=95, FP=90, FN=5, TN=810); textbook-style table, not real patient data | yes |
Divergence detail
Per-comparison detail for every divergence counted above - case/field identify the
comparison, reference names what it was checked against (independent_r, anchor, or an
(engine_error)/(canonical_error) crash), and abs delta/tolerance are the exact numbers a
regression’s own PR needs, not just a bare match verdict. Empty in the healthy case, by
construction: the row list below is built from the SAME divergence events the summary count above
totals, so it can never show a count without matching rows or vice versa.
- (none)
Test selection (does the tool steer to the right test, and show alternatives?)
Correctness of the number matters only if the tool also proposes a defendable test. This tier uses the deterministic metadata advisor + suitability grader (no AI): it checks the correct test FAMILY is proposed for the data, and that the grader recommends a sound sibling and never calls the valid test ‘Not appropriate’. The alternatives offered column is the count of sibling tests the tool surfaces for the researcher to consider - the product deliberately shows the alternatives and lets the researcher (optionally with AI assistance) explore them, rather than dictating one answer.
| case | method | correct family proposed | grader recommends | alternatives offered |
|---|---|---|---|---|
| sleep-paired_t | paired_t | yes | - | 2 |
| sleep-wilcoxon | wilcoxon_signed_rank | yes | - | 2 |
| ToothGrowth-welch_t | welch_t | yes | mann_whitney | 3 |
| ToothGrowth-kruskal_wallis | kruskal_wallis | yes | welch_anova | 3 |
| iris-welch_anova | welch_anova | yes | welch_anova | 3 |
| mtcars-pearson | pearson | yes | pearson | 2 |
| mtcars-spearman | spearman | yes | pearson | 2 |
| mtcars-linear_regression | linear_regression | yes | - | 1 |
| mtcars-logistic_regression | logistic_regression | yes | - | 1 |
| mtcars-fisher_exact | fisher_exact | yes | chi_square | 2 |
| titanic-chi_square | chi_square | yes | chi_square | 2 |
| birthwt-logistic_regression | logistic_regression | yes | - | 1 |
| sleepstudy-mixed_model | mixed_model | yes | - | 1 |
| airquality-mann_whitney-filtered | mann_whitney | yes | mann_whitney | 3 |
| lung-kaplan_meier | kaplan_meier | yes | - | 1 |
| lung-cox_ph | cox_ph | yes | - | 2 |
| lung-aft_weibull | aft_weibull | yes | - | 2 |
| mcnemar-disease-example | mcnemar | yes | - | 1 |
| respiratory-mcnemar | mcnemar | yes | - | 1 |
| anorexia-ancova | ancova | yes | - | 1 |
| warpbreaks-poisson_regression | poisson_regression | yes | negbin_regression | 2 |
| quine-negbin_regression | negbin_regression | yes | negbin_regression | 2 |
| housing-ordinal_logistic | ordinal_logistic | yes | - | 1 |
| warpbreaks-two_way_anova | two_way_anova | yes | - | 1 |
| iris-multinomial_logistic | multinomial_logistic | yes | - | 1 |
| PlantGrowth-one_way_anova | one_way_anova | yes | welch_anova | 3 |
| ToothGrowth-student_t | student_t | yes | mann_whitney | 3 |
| VerbAgg-mixed_logistic | mixed_logistic | yes | - | 1 |
| epil-mixed_poisson | mixed_poisson | yes | - | 1 |
| ergoStool-rm_anova | rm_anova | yes | - | 1 |
| aSAH-roc_auc | roc_auc | yes | - | 1 |
| aSAH-roc_compare | roc_compare | yes | - | 1 |
| screening-diagnostic_accuracy | diagnostic_accuracy | yes | - | 1 |
| davis-bland_altman | bland_altman | via post-analysis follow-up | - | 1 |
What this does NOT validate (read this)
- 30 of 40 cases are canonical-only - they match an independent computation but no single citable published number. Presenting them as literature-validated would be dishonest; they are not.
- For several cases (paired t, Welch t, the regressions, Kaplan-Meier, Cox, the mixed models) the selection check is a ‘not wrong’ bar - it asserts the grader does not reject the correct test - rather than ‘graded strictly best’. Textbook-unambiguous cases assert the stronger ‘recommended’.
- One case’s test family is proposed only AFTER a related analysis actually runs, not before: a strongly-correlated, similar-scale pair run as a correlation gets a deterministic follow-up note recommending an agreement analysis instead - the advisor deliberately never guesses this pairing from column names alone (a false guess would steer a researcher to the wrong test), but once real data confirms the pattern, a conditional, honest suggestion is safe to make.
- The three
glmermixed-model cases are canonical-only ON PURPOSE: a frozen published coefficient would risk cross-platform optimizer jitter, so same-run engine-vs-refit equality is the guard. - This is a self-generated report. It is real evidence, but weaker than an independent statistician’s review, which is the explicit next credibility step and is not yet done.
- Celeus is research statistics, not clinical decision support: nothing here validates or implies patient-specific diagnosis or treatment.
Reproducibility guarantee
This report carries no timestamp: regenerating it against the same pinned environment (same code, same package versions, same corpus) produces a byte-identical file, character for character - so a clean regeneration finding zero diff is itself the determinism demonstration this artifact claims, not an assertion taken on faith.
Every individual analysis run on the platform emits its own seed, exact package versions, and a standalone reproduce script alongside its results - the same determinism guarantee this report demonstrates, applied per-run. This report’s own determinism rests on the pinned lockfile (md5 above) plus the fixed catalog version, not on a captured full session/locale snapshot (which would break byte-identity across otherwise-identical environments).
Next credibility step
Independent review by a qualified biostatistician of the method-selection logic and this corpus. A self-generated report cannot substitute for it; it is named here as the next step, deliberately out of scope for this artifact.