From 98825db94ad6751f7eaf74e8bea387ed2f854bbc Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Mon, 12 Oct 2026 20:17:44 +0200 Subject: [PATCH] docs(benchmark): R6 holds the push path to the LLM tier; measure hook as shipped and hook + hunter The pre-push hook has no LLM detection tier today: push/assistance.py only selects a witness for already-admitted defects and cannot add a finding. The hunter lives in scan --mode deep. The maintainer's call is that the LLM tier is required on pre-push, so R6 now measures two arms over the same changes: (a) the hook as shipped (quick rules + engine), (b) the hook plus the hunter arm run off-cluster. (b) is the number the tool is held to; the gap is product work in pre-push-safety-net, not a benchmark artefact. Adds group 6 (hunter arm runner + combine + offline test) to tasks, aligns the brief and design, and labels the overnight replay as arm (a). Co-Authored-By: Claude Fable 5.1 --- .../specs/valid-detection-benchmark/brief.md | 6 +++-- .../specs/valid-detection-benchmark/design.md | 13 +++++++--- .../valid-detection-benchmark/requirements.md | 16 +++++++++--- .../specs/valid-detection-benchmark/tasks.md | 25 ++++++++++++++----- 4 files changed, 46 insertions(+), 14 deletions(-) diff --git a/.kiro/specs/valid-detection-benchmark/brief.md b/.kiro/specs/valid-detection-benchmark/brief.md index 2b50044..6decaaf 100644 --- a/.kiro/specs/valid-detection-benchmark/brief.md +++ b/.kiro/specs/valid-detection-benchmark/brief.md @@ -13,8 +13,10 @@ A pre-push security check must answer two questions: 1. **If I'm about to push a real vulnerability, does it stop me?** 2. **If my change is fine, does it leave me alone?** -The benchmark runs the **whole tool** as a developer uses it at push time. Quick rules and the engine are on, -as is the LLM-guided arm. The repositories are ones the tool has **never seen**. +The benchmark measures the **push path** as a developer uses it. The LLM detection tier (the hunter) is +required on pre-push. Today the shipped hook runs quick rules and the engine only, so the benchmark +reports the hook as shipped and the hook plus the hunter arm separately until the hunter is wired in +(see `requirements.md` R6). The repositories are ones the tool has **never seen**. ## What makes the data valid diff --git a/.kiro/specs/valid-detection-benchmark/design.md b/.kiro/specs/valid-detection-benchmark/design.md index b68aeba..1bbce1a 100644 --- a/.kiro/specs/valid-detection-benchmark/design.md +++ b/.kiro/specs/valid-detection-benchmark/design.md @@ -36,7 +36,9 @@ advisories (OSV/GHSA) -> human spot-check record decision, reviewer, date [manual, recorded in verification.json] -> draw_ordinary() negatives from the same repos [reuse] -> assemble.py freeze positives+negatives -> manifest + digest, held-out guard [new, wraps eligibility.normalize] - -> replay.py (docker lane) run the whole tool at push time [reuse] + -> replay.py (docker lane) arm (a): the push hook as shipped (quick + engine) [reuse] + -> hunter arm runner arm (b): the hunter over the same changes, off-cluster [new, group 6] + -> combine (a) and (b) joined per change id [new, group 6] -> score.py + metrics.py recall, precision, FP/100, exploit-verified, intervals [reuse + new] ``` @@ -108,8 +110,13 @@ Headline string, built here: 3. `verify.py` runs the LLM check; human spot-checks all of them (dozens, so full review is cheap). 4. `draw_ordinary()` draws >=20x negatives from the same repos. 5. `assemble.py` freezes the manifest + digest + held-out guard. -6. `replay.py` on the Docker lane runs the whole tool. -7. `score.py` + `metrics.py` produce the four metrics with intervals and the headline. +6. `replay.py` on the Docker lane runs the push hook as shipped (arm a). The hunter arm runner adds arm + (b) over the same changes, off-cluster, and the two are combined per change. +7. `score.py` + `metrics.py` produce the four metrics with intervals and the headline, for (a) and for + (a)+(b). (b) is the number the tool is held to; the difference is the product gap on pre-push. + +Wiring the hunter into the pre-push hook itself is product work in `pre-push-safety-net`. This spec +measures the push path with and without it; it does not change the hook. The pilot proves the pipeline and gives the first real numbers. Then grow positives to ~40 (set exactly by `required_n`) and, where the proof arm has run, fill in the exploit-verified rate. diff --git a/.kiro/specs/valid-detection-benchmark/requirements.md b/.kiro/specs/valid-detection-benchmark/requirements.md index 5b17c20..f9a746f 100644 --- a/.kiro/specs/valid-detection-benchmark/requirements.md +++ b/.kiro/specs/valid-detection-benchmark/requirements.md @@ -52,9 +52,19 @@ security metrics. - No repository in the benchmark was used to build, train, tune, mine, or harvest the tool. Enforce this with the existing used-set canonicalization (`eligibility.normalize`), the same guard the SZZ pool used. -### R6. Measure the whole tool on unseen repos -- Run the tool as a developer runs it at push time: quick rules on, engine on, LLM-guided arm on. -- Use the existing replay harness so the measured path is the shipped push hook. +### R6. Measure the push path on unseen repos, with the LLM tier +- The tool a developer runs at push time must include the LLM detection tier (the hunter). That is a + product requirement on the pre-push hook, owned by `pre-push-safety-net`. The benchmark measures the + push path, not a scan mode. +- Today the shipped hook runs quick rules and the engine only. Its model path (`push/assistance.py`) + selects witnesses for already-admitted defects and cannot add a finding. Until the hunter is wired into + the hook, the benchmark reports two arms over the same changes and labels them: + - (a) the push hook as shipped (quick rules + engine), through the replay harness; + - (b) the push hook plus the hunter arm, the hunter run off-cluster over the same changes and the + records combined per change. + (b) is the number the tool is held to. (a) is what ships today. The gap between them is product work, + not a benchmark artefact, and is reported as such. +- Use the existing replay harness for (a) so the measured path is the shipped push hook. ### R7. Standard metrics - Report recall (true-positive rate), precision (positive predictive value), false-positive rate as diff --git a/.kiro/specs/valid-detection-benchmark/tasks.md b/.kiro/specs/valid-detection-benchmark/tasks.md index d4b4af0..8b91600 100644 --- a/.kiro/specs/valid-detection-benchmark/tasks.md +++ b/.kiro/specs/valid-detection-benchmark/tasks.md @@ -43,13 +43,26 @@ Order is dependency order. Each task is small enough to review on its own. New c - [ ] 5.1 Source Python/JS advisories; run `catalog.py` to ~25 candidates. (deliberate, rate-aware) - [ ] 5.2 Run `verify.py` + human spot-check to land ~12 accepted positives. - [ ] 5.3 `assemble.py` freezes the pilot manifest + negatives (>=20x) + held-out guard. -- [ ] 5.4 `replay.py` on the Docker lane, whole tool on (quick + engine + LLM arm). -- [ ] 5.5 `score.py` + `metrics.py`: produce recall, precision, FP/100, exploit-verified, intervals, and - the headline. Freeze the run under `benchmarks/measurements/`. -- [ ] 5.6 Write up the pilot: the four numbers, what they mean, and the target set size from `required_n` - for the full ~40-positive benchmark. +- [ ] 5.4 `replay.py` on the Docker lane, arm (a): the push hook as shipped (quick + engine, 300 s). + Running overnight 2026-10-12; records under `~/ousast-results/valid/pilot/long_deadline`. +- [ ] 5.5 `score.py` + `metrics.py` on arm (a): recall, precision, FP/100, exploit-verified, intervals, + and the headline, labelled "push hook as shipped". Freeze the run under `benchmarks/measurements/`. +- [ ] 5.6 Write up the pilot: the four numbers for (a), what they mean, the engine-covered rate, and the + target set size from `required_n` for the full ~40-positive benchmark. + +## Group 6 - hunter arm (the LLM tier required on pre-push) +- [ ] 6.1 `benchmarks/valid/hunter_arm.py`: for each change in the frozen manifest, check out `head` in + the local clone, build `FileTarget`s scoped to the changed files, run `hunter.run_hunter_pool` + (`src/openultrasast/hunter.py`, wiring in `cli.py` `_attach_tool_hunter`), and emit a record `score.py` + can read (an alerts/advisory stream keyed by change id). Runs off-cluster; the model key never enters + a replay container. Honors the instrument rule (bytes read per target, fail loudly on an empty read). +- [ ] 6.2 Combine arm (a) and arm (b) records per change id and rescore: this is the number the tool is + held to (R6). Report (a), (a)+(b), and the difference as the pre-push product gap. +- [ ] 6.3 Offline test with a stub hunter: a flagged change joins the (a) record; a missing (b) record + leaves (a) unchanged; duplicate ids are rejected. ## Out of this spec - Growing beyond the pilot to the full set (follow-up, once the pipeline is proven). - Running on kube `ax` (blocked on gVisor; Docker lane is enough here). -- Any new detection capability. +- Any new detection capability, including wiring the hunter into the pre-push hook. That is product + work in `pre-push-safety-net`; this spec measures the push path with and without it.