mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
feat(hunt): candidates scored on folds through the runner; the pilot gain, once per accepted program (6.1)
rounds.py runs a candidate over compile or held-out folds through runner.run, so a tool-using
program has its tools bound to each change's checkout, and turns paired records into the
FoldResults accept.decide reads (closes the gap noted in bacd7a1). report.py runs an accepted
program once over the held-out pilot: a ledger line reserved before the run refuses a second run
per program id, contaminated or unchecked training runs are refused, the instrument is proved per
change and more than 5% failures makes the run invalid with no numbers quoted, the pilot's own
scorer and metrics give the after numbers, the recorded hunter numbers give the before, baseline
repeats pair with the single run, the proof arm's workload per 100 pushes is counted, and the gate
distance is read off the confidence curve. 124 hunt and stats tests pass. Implemented by Codex
(gpt-6-astra), gated here.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
bacd7a1e3f
commit
6f9613329e
@@ -88,7 +88,7 @@
|
||||
- _Requirements: 5.6, 6.1, 6.2, 6.3, 6.4, 6.5, 6.7_
|
||||
|
||||
- [ ] 6. Validation: the pilot gain
|
||||
- [ ] 6.1 Run an accepted program once over the held-out pilot and report the gain
|
||||
- [x] 6.1 Run an accepted program once over the held-out pilot and report the gain
|
||||
- Same scorer and metrics as the pilot; before-and-after of search recall, suspicions per push, push-level recall, false alarms per 100 and precision, per family, with intervals; the proof arm's workload per 100 pushes; the gate distance read off the curve; the run recorded next to the pilot's measurement, at most once per accepted program
|
||||
- Done when the measurement record exists with the before-and-after table and the instrument proof for every change
|
||||
- _Requirements: 7.3, 7.4, 7.5, 8.1, 8.2, 8.3, 8.6_
|
||||
|
||||
Reference in New Issue
Block a user