feat(hunt): registered paired ablation, repository folds, leak audit, importance table and the keep/drop rule (4.2-4.4)

ablation.py registers the manifest (table digest, arms as masks, folds, metric, sample, budget,
rule, seed, program id) before any run and refuses a run or an analysis that does not match it.
Arms run over the same changes through runner.run, paired by change; effects on recall and on
suspicions per ordinary push are reported pooled and per family with the repository-clustered
bootstrap and exact McNemar, plus the recall-against-suspicions curve at every reported
confidence (never a tuned cut). The importance table and decisions read held-out effects only;
a neutral annotation survives only when a kept one needs it. 30 fixture tests with a scripted
model. Implemented by Codex (gpt-6-astra), gated here.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-10-13 18:49:58 +02:00
co-authored by Claude Fable 5.1
parent bbbb0f397e
commit 6e5d060de6
3 changed files with 816 additions and 3 deletions
+3 -3
View File
@@ -59,16 +59,16 @@
- About one hundred advisories through the fix-site catalog and verifier, pilot repositories excluded, at least 40 accepted positives across two languages, 20 ordinary pushes per positive from the same repositories; the sourcing run is recorded
- Done when the frozen training manifest lists its positives, negatives, languages and the pilot exclusion check
- _Requirements: 2.1, 2.2, 2.3_
- [ ] 4.2 Fold by repository and audit every annotation for leaks
- [x] 4.2 Fold by repository and audit every annotation for leaks
- Compile folds and held-out folds by repository group; the leak audit runs over every feature column and reports any that separates positives from negatives on its own
- Done when the folds are disjoint by repository and the audit report lists each column with its separation score
- _Requirements: 2.5, 2.6_
- [ ] 4.3 Register and run a paired ablation
- [x] 4.3 Register and run a paired ablation
- A manifest fixes the table digest, the masks, the folds, the metric, the sample size, the budget and the decision rule before anything runs; each arm runs over the same changes with the same program and sampling
- Results per arm carry the paired effect on search recall and suspicions per ordinary push, per family and pooled, with intervals from the repository-clustered bootstrap and a paired test
- Done when an ablation over a fixture manifest with the scripted model yields two arms, the paired statistics, and the full recall-against-suspicions curve
- _Requirements: 4.1, 4.2, 4.3, 4.7, 4.8_
- [ ] 4.4 Build the importance table and apply the keep/drop rule
- [x] 4.4 Build the importance table and apply the keep/drop rule
- For every annotation, its effect when removed from the full set and when added to the baseline, with intervals and counts; an annotation stays only on a held-out effect whose interval excludes zero or when a kept annotation needs it; one that raises suspicions without raising recall is dropped and recorded
- Done when the table is written for a fixture ablation and the rule's decisions match the fixture's expectations
- _Requirements: 4.4, 4.5, 4.6_