# Quantitative evidence audit

**Date:** 24 September 2026. **Result:** key numerical claims verified by local recomputation. This is an audit of an exploratory pilot on already exposed data, not independent scientific validation or evidence of a new mechanism.

The standalone [reproduction helper](reproduction/reproduce.py) passed all **50 declared artifact byte-hash checks**, all **31 registered metric replay checks**, and all **64 MFASS boundary-band metric checks**, using a 1e-9 absolute numerical tolerance. It also checked the top-100 membership claim, source labels and specialist predictions, training means and the constant-predictor calculations. [Aggregate results](reproduction/results.json) contain the complete numbers and source hashes. [Reproduction instructions](reproduction/README.md) distinguish public aggregate arithmetic from full recomputation requiring locally acquired source files. The private resolver was read in place and was not copied.

## MFASS: diagnostic recall reverses; database precision does not

The original test cohort contains **8,324 variants, including 315 positives**. Pangolin scores 8,301 variants and 314 positives; SpliceAI scores 8,194 variants and 308 positives. Every SpliceAI-scored variant is also scored by Pangolin in these saved predictions.

| Population | Pangolin hits / positives | Pangolin recall | SpliceAI hits / positives | SpliceAI recall |
| --- | ---: | ---: | ---: | ---: |
| Each model's scored rows | 65 / 314 | 20.7006369427% | 64 / 308 | 20.7792207792% |
| Common 8,194 rows | 65 / 308 | 21.1038961039% | 64 / 308 | 20.7792207792% |

The common-row filter removes 107 Pangolin-scored rows, including six positives. Each model retains **exactly its own original top-100 membership set**. This does not mean the two models select the same variants. None of the removed rows is in Pangolin's top 100. There is no tie spanning the cutoff in either whole-population ranking: Pangolin's 100th and 101st scores are 0.55 and 0.54; SpliceAI's are 0.67 and 0.66. Thus the reversal here comes entirely from the positive denominator, without a model, score, capacity or selected-membership change.

The metric is the research diagnostic `recall_at_capacity`, with capacity 100. The local database source `data/benchmark-runs/mfass-v2.json` reports `precision_at_100`, average precision and AUROC. It has **no recall field**. Pangolin's precision remains 65/100 = 0.65 and SpliceAI's remains 64/100 = 0.64. The article must not call this a reversal of the database leaderboard. The database source's revision and SHA-256 are recorded in [evaluator provenance](reproduction/evaluator-provenance.json).

Scored-population recall is conditional on each method returning a score. Common-population recall answers the paired-subset question. Neither is universally the correct end-to-end estimand. Both omit some eligible positives, and the common population excludes 130 of the 8,324 test variants. Coverage must therefore remain visible beside performance. Matching rows removes this particular population difference; it does not remove differences in model input sequence context or establish comparable deployment coverage. The common-set difference is one recovered positive, not a statistical or biological superiority claim.

## ProteinGym: inclusive percentile ties select every prediction

For the single `AMFR_HUMAN_Tsuboyama_2023_4G3O` assay, there are **2,972 rows** and **298** observations meeting the true 90th-percentile threshold, approximately 0.04206218069865769. A constant vector's 90th percentile equals its constant value; applying `prediction >= percentile(prediction, 90)` selects all 2,972 rows. It therefore receives **298/298 = 1.0 Top_recall** while providing no ranking information.

This follows the inclusive percentile comparison in the [pinned ProteinGym evaluator](https://github.com/OATML-Markslab/ProteinGym/blob/144fe22b07dfaeec2b366f2346203a9838a55b4c/proteingym/performance_DMS_benchmarks.py#L68-L75), retrieved 24 September 2026. It is not an exact top-10%-capacity selection. This audit makes no claim about full-suite performance, submitted real-model gaming, or the unexecuted single-versus-multiple-substitution comparison. The assay checksum was observed from downloaded bytes; it is not an independently published source checksum.

## FLIP2: high constant full-list NDCG

The Rhomax `by_wild_type` split has **184 test rows**. The training-mean control predicts 539.7328767123288 for every row, computed from **584 training targets**. Its NDCG is **0.9206667522227658**, versus composition's **0.954819941821582**; the difference is **0.03415318959881619**. Composition Spearman is approximately **0.418182**. Constant Spearman is undefined, not zero.

The calculation uses `ndcg_score(y - min(y), predictions)` with the full list, linear relevance gains, logarithmic discount and scikit-learn's default averaging over tied predicted scores. It matches the [pinned FLIP evaluator](https://github.com/J-SNACKKB/FLIP/blob/62cace8735f5610e2743cf06ce0f944b37fffaa6/baselines/aggregate.py#L131-L135), retrieved 24 September 2026. This is different from ProteinGym's top-decile NDCG implementation.

For a single all-tied predicted group, expected DCG is mean gain multiplied by the sum of discounts. The audit obtains **3219.780347860353**, with ideal DCG **3497.22670020052**; their ratio independently reproduces 0.9206667522227658. The minimum target subtracted is 436.0. The constant score quantifies this split and evaluator's no-information reference point; it does not establish that high absolute NDCG is uninformative in every task. An actual randomly permuted finite ranking can differ from this tie-averaged expectation.

## mRNABench: observed MSE reduction with unresolved subgroups

For the designed Sample MRL subset, both methods score the same **15,003 test rows**. Composition MSE is **1.914439715839851**; training-mean MSE is **2.3655294722554605**. The relative reduction is `(2.3655294722554605 - 1.914439715839851) / 2.3655294722554605 = 0.1906929343752835`, or **19.0693%**, rounded to **19.1%**.

The helper recomputes the mean from **70,011 training rows** as 5.7475703435024. The stored constant is 5.747570343502464; the approximately 6e-14 difference is ordinary summation-order precision and comfortably within tolerance. The control is fitted on training targets, not test targets. No composition model was refitted in this audit.

The frozen plan's prose assumes composition minus training mean, whereas the executed receipt explicitly reports training mean minus composition. The positive receipt difference, **0.45108975641560956**, is consistent with composition having lower MSE. The original plan remains unchanged. Length and GC subgroup comparisons were not executed; the executed duplicate category contains only `no`, so it provides no between-category contrast. These results neither locate the gain nor identify a molecular mechanism. The source dataset's reuse license was recorded as unreported, so no raw mRNA data is redistributed.

## Boundary-distance results and limits

The helper reconstructs boundary categories from source cohort fields using the minimum distance to either annotated exon boundary. All 64 subgroup metric values match their receipts. Whole-band outcome rates vary substantially:

| Boundary distance | Rows | Positives | Positive fraction |
| --- | ---: | ---: | ---: |
| 0–2 | 448 | 42 | 9.3750% |
| 3–10 | 1,690 | 81 | 4.7929% |
| 11–30 | 4,239 | 160 | 3.7745% |
| >30 | 1,947 | 32 | 1.6436% |

These are the full band populations; method-specific scored populations differ. For example, Pangolin AP is 0.645366 in 0–2 and 0.148675 in >30, while SpliceAI AP is 0.585731 and 0.073356. Such descriptive differences require prevalence and coverage context. Specialist subgroup top-100 scores **do have cutoff ties**, handled by the pilot's seeded label-independent permutation; the no-cutoff-tie statement applies only to the headline whole-population comparison. Subgroup capacities also select different fractions of the available variants. No replicate-agreement or gene/exon-concentration analysis establishes an explanation, and no biological mechanism follows from these subgroup metrics.

## Exact provenance and scope

The audit used Python **3.11.13**, NumPy **1.26.4**, SciPy **1.17.1**, and scikit-learn **1.9.1**, matching the recorded pilot-control dependency versions. The pilot runner source digest is `5c41ffd01ea787711cf00a8bc493aafc70e28b797b3a273da319b4954026c596`. [Evaluator provenance](reproduction/evaluator-provenance.json) records exact local metric/evaluator file hashes, upstream revisions, campaign-bundle hashes and the helper hash. The catalogue release is `2026-09-20-370b30415b09`; campaign `campaign-fd29c4413eaf4e24` remains exploratory and pending review.

| Research table | SHA-256 |
| --- | --- |
| MFASS | `c57f9cfe91d02ab54f6bc20934f5b8f678b37c3b13d8d20aa805fa7d73e62c4f` |
| ProteinGym AMFR | `4fac11aacc0f00a70e385ed7b8d73427aaa0716906a74e93bc3d48df5ff43f9f` |
| FLIP2 Rhomax | `8562d4891270a73026fc7852c05f41206ad52e8fcae1d706d83ec57572c7c949` |
| mRNABench | `d49ff1b4dacdfe0663c35bdad6a9ad91e79f5862fe96f36a627acb8407610bf3` |

Full recomputation authenticates the saved local artifacts and numerical transformations. It does not rerun all original model inference or reconstruct every upstream dataset-processing step. Aggregate-only reproduction authenticates neither source observations nor membership assertions. The supplement contains no raw assay rows, sequences, local resolver, model-session logs or credentials.