mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-18 04:29:27 +00:00
Marius CiepluchandClaude Fable 5.1
9f7ca132ff
benchmarks/valid/hunter_arm.py runs the shipped deep-mode tool hunter (tool_hunter.run_tool_hunter via resolve_hunter_client) over each frozen change off-cluster: detached worktree at head with git hooks disabled, hotspots scoped to the changed files, instrument rule on every hotspot (bytes read printed, empty or unreadable fails the change loudly), model id from the config resolver, family from the taxonomy, never from the label. Records land in the exact shape score.py reads (push.admission.defects with locations), resumable per change. `combine` joins arm (a) replay records with arm (b) hunter records per change id, dedupes defects by (path, line, family), keeps (a)'s instrument, records which arms were present, and writes a file score.py loads directly. 9 offline tests (stub hunter, real scorer), 53 green with the benchmark tests. No network, no key handling, no live run. Co-Authored-By: Claude Fable 5.1 <[email protected]>
71 lines
5.0 KiB
Markdown
71 lines
5.0 KiB
Markdown
# Tasks: valid detection benchmark
|
|
|
|
Order is dependency order. Each task is small enough to review on its own. New code under
|
|
`benchmarks/valid/`. Implementation goes through Codex, gated here.
|
|
|
|
## Group 1 - positive catalog (fix-site, no SZZ)
|
|
- [x] 1.1 `benchmarks/valid/catalog.py`: given sourced advisories (reuse `source.py`/`draft.py` output),
|
|
build candidate positive cases by fix site. Wrap `extract.fix_sites()` + `pairs/materialize.py`; read
|
|
the pre-fix function at `parent` and the post-fix function at `commit`. Must not call `introducing` or
|
|
`last_touch`. Emit the `[[case]]` catalog in the design schema. (R1)
|
|
- [x] 1.2 Filter to Python and JS/TS only; attach `language` and `family`/`cwe`. (scope decision)
|
|
- [x] 1.3 Offline test: a fixture commit yields one case with the correct pre/post functions and
|
|
`method = "fix_site"`; assert SZZ methods are never invoked.
|
|
|
|
## Group 2 - LLM verification driver
|
|
- [x] 2.1 `benchmarks/valid/verify.py`: one DeepSeek call per case via the shipped provider (off-cluster).
|
|
Confirm vulnerable-pre-fix, fix-addresses-it, reachable-in-repo; name the entry point; return
|
|
confidence. Print bytes read per function before the call; reject a verdict on an empty read. (R3, instrument rule)
|
|
- [x] 2.2 Emit the verification record (design schema) to `verification.json`; `verdict = "accepted"`
|
|
only when all three checks pass. Reachability here is the app-shaped filter. (R2, R3)
|
|
- [x] 2.3 Human spot-check hook: a small CLI to record accept/reject + reviewer + date per case into the
|
|
same record. (R3)
|
|
- [x] 2.4 Offline test with a stub model: a case with all three true and a human accept becomes
|
|
`accepted`; any false, or a human reject, does not.
|
|
|
|
## Group 3 - negatives and assembly
|
|
- [x] 3.1 `benchmarks/valid/assemble.py`: draw negatives with `extract.draw_ordinary()` from the accepted
|
|
positives' repos, at least 20x, excluding the fix. (R4)
|
|
- [x] 3.2 Freeze accepted positives + negatives into a `pool.toml`-shaped manifest + `freeze-valid.json`
|
|
digest; run the held-out guard via `eligibility.normalize`. (R5, R8)
|
|
- [x] 3.3 Offline test: freeze is deterministic (same source -> same digest); held-out guard drops any
|
|
repo in the used set.
|
|
|
|
## Group 4 - metrics
|
|
- [x] 4.1 `benchmarks/valid/metrics.py`: precision (PPV) with Wilson 95% over positives+negatives. (R7)
|
|
- [x] 4.2 False alarms per 100 pushes as the presentation of `score.py`'s false-alarm rate. (R7)
|
|
- [x] 4.3 Exploit-verified rate: join flagged positives with a search-with-proof run by case `id`;
|
|
report "not yet proved" where absent. (R7)
|
|
- [x] 4.4 Build the headline string (recall + FP/100 + precision + exploit-verified). (R7)
|
|
- [x] 4.5 Offline test on a tiny hand-made record set: all four metrics and intervals match hand calcs.
|
|
|
|
## Group 5 - pilot run (research, local)
|
|
- [ ] 5.1 Source Python/JS advisories; run `catalog.py` to ~25 candidates. (deliberate, rate-aware)
|
|
- [ ] 5.2 Run `verify.py` + human spot-check to land ~12 accepted positives.
|
|
- [ ] 5.3 `assemble.py` freezes the pilot manifest + negatives (>=20x) + held-out guard.
|
|
- [ ] 5.4 `replay.py` on the Docker lane, arm (a): the push hook as shipped (quick + engine, 300 s).
|
|
Running overnight 2026-10-12; records under `~/ousast-results/valid/pilot/long_deadline`.
|
|
- [ ] 5.5 `score.py` + `metrics.py` on arm (a): recall, precision, FP/100, exploit-verified, intervals,
|
|
and the headline, labelled "push hook as shipped". Freeze the run under `benchmarks/measurements/`.
|
|
- [ ] 5.6 Write up the pilot: the four numbers for (a), what they mean, the engine-covered rate, and the
|
|
target set size from `required_n` for the full ~40-positive benchmark.
|
|
|
|
## Group 6 - hunter arm (the LLM tier required on pre-push)
|
|
- [x] 6.1 `benchmarks/valid/hunter_arm.py`: for each change in the frozen manifest, check out `head` in
|
|
the local clone (detached worktree), build hotspots for the changed files, run the shipped deep-mode
|
|
arm `tool_hunter.run_tool_hunter` with `resolve_hunter_client()` (the hunter that beat overlay 12/14;
|
|
wiring in `cli.py` `_attach_tool_hunter`), and emit a record `score.py` can read: findings go under
|
|
`push.admission.defects` with `family` and `locations = [[path, line]]` (the alerts stream), keyed by
|
|
change id. Runs off-cluster; the model key never enters a replay container. Honors the instrument rule
|
|
(bytes read per hotspot, fail loudly on an empty read).
|
|
- [x] 6.2 Combine arm (a) and arm (b) records per change id and rescore: this is the number the tool is
|
|
held to (R6). Report (a), (a)+(b), and the difference as the pre-push product gap.
|
|
- [x] 6.3 Offline test with a stub hunter: a flagged change joins the (a) record; a missing (b) record
|
|
leaves (a) unchanged; duplicate ids are rejected.
|
|
|
|
## Out of this spec
|
|
- Growing beyond the pilot to the full set (follow-up, once the pipeline is proven).
|
|
- Running on kube `ax` (blocked on gVisor; Docker lane is enough here).
|
|
- Any new detection capability, including wiring the hunter into the pre-push hook. That is product
|
|
work in `pre-push-safety-net`; this spec measures the push path with and without it.
|