mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
feat(hunt): contamination check (4.5): runs that used a reported repository are marked and kept out of any gain
contamination.py reads the repositories named by the held-out pilot manifest and by every benchmark report (accepted_repos.json and NOTES.md links under benchmarks/measurements), marks each run directory clean or contaminated with the report digest and time, and refuses to count an unchecked run toward a gain. ablation.analyse carries excluded_from_gain when its run is contaminated. Implemented by Codex (gpt-6-astra), gated here; the benchmark import is lazy so the package imports without the benchmarks tree. Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
6e5d060de6
commit
dbaaa86d5b
@@ -72,7 +72,7 @@
|
||||
- For every annotation, its effect when removed from the full set and when added to the baseline, with intervals and counts; an annotation stays only on a held-out effect whose interval excludes zero or when a kept annotation needs it; one that raises suspicions without raising recall is dropped and recorded
|
||||
- Done when the table is written for a fixture ablation and the rule's decisions match the fixture's expectations
|
||||
- _Requirements: 4.4, 4.5, 4.6_
|
||||
- [ ] 4.5 Contamination check
|
||||
- [x] 4.5 Contamination check
|
||||
- A training repository later found in the pilot or in a benchmark report marks every run that used it as contaminated and excludes those runs from any gain
|
||||
- Done when a fixture run is marked contaminated after its repository is added to a report list
|
||||
- _Requirements: 2.7_
|
||||
|
||||
Reference in New Issue
Block a user