contamination.py reads the repositories named by the held-out pilot manifest and by every
benchmark report (accepted_repos.json and NOTES.md links under benchmarks/measurements), marks
each run directory clean or contaminated with the report digest and time, and refuses to count
an unchecked run toward a gain. ablation.analyse carries excluded_from_gain when its run is
contaminated. Implemented by Codex (gpt-6-astra), gated here; the benchmark import is lazy so
the package imports without the benchmarks tree.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
ablation.py registers the manifest (table digest, arms as masks, folds, metric, sample, budget,
rule, seed, program id) before any run and refuses a run or an analysis that does not match it.
Arms run over the same changes through runner.run, paired by change; effects on recall and on
suspicions per ordinary push are reported pooled and per family with the repository-clustered
bootstrap and exact McNemar, plus the recall-against-suspicions curve at every reported
confidence (never a tuned cut). The importance table and decisions read held-out effects only;
a neutral annotation survives only when a kept one needs it. 30 fixture tests with a scripted
model. Implemented by Codex (gpt-6-astra), gated here.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Six groups: table and producers; the hunter as a program with the search
metric and the run artifact; runner integration and a baseline on the 12
pilot positives; training set, folds, paired ablation and the importance
table; proposals and the acceptance rule; the pilot gain report.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
