feat(hunt): contamination check (4.5): runs that used a reported repository are marked and kept out of any gain

contamination.py reads the repositories named by the held-out pilot manifest and by every
benchmark report (accepted_repos.json and NOTES.md links under benchmarks/measurements), marks
each run directory clean or contaminated with the report digest and time, and refuses to count
an unchecked run toward a gain. ablation.analyse carries excluded_from_gain when its run is
contaminated. Implemented by Codex (gpt-6-astra), gated here; the benchmark import is lazy so
the package imports without the benchmarks tree.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-10-13 19:03:23 +02:00
co-authored by Claude Fable 5.1
parent 6e5d060de6
commit dbaaa86d5b
4 changed files with 319 additions and 2 deletions
+1 -1
View File
@@ -72,7 +72,7 @@
- For every annotation, its effect when removed from the full set and when added to the baseline, with intervals and counts; an annotation stays only on a held-out effect whose interval excludes zero or when a kept annotation needs it; one that raises suspicions without raising recall is dropped and recorded
- Done when the table is written for a fixture ablation and the rule's decisions match the fixture's expectations
- _Requirements: 4.4, 4.5, 4.6_
- [ ] 4.5 Contamination check
- [x] 4.5 Contamination check
- A training repository later found in the pilot or in a benchmark report marks every run that used it as contaminated and excludes those runs from any gain
- Done when a fixture run is marked contaminated after its repository is added to a report list
- _Requirements: 2.7_