docs(benchmark): R6 holds the push path to the LLM tier; measure hook as shipped and hook + hunter

The pre-push hook has no LLM detection tier today: push/assistance.py only
selects a witness for already-admitted defects and cannot add a finding. The
hunter lives in scan --mode deep. The maintainer's call is that the LLM tier is
required on pre-push, so R6 now measures two arms over the same changes:
(a) the hook as shipped (quick rules + engine), (b) the hook plus the hunter
arm run off-cluster. (b) is the number the tool is held to; the gap is product
work in pre-push-safety-net, not a benchmark artefact.

Adds group 6 (hunter arm runner + combine + offline test) to tasks, aligns the
brief and design, and labels the overnight replay as arm (a).

Co-Authored-By: Claude Fable 5.1 <[email protected]>

.kiro/specs/valid-detection-benchmark/brief.md

@@ -13,8 +13,10 @@ A pre-push security check must answer two questions:
 1. **If I'm about to push a real vulnerability, does it stop me?**
 2. **If my change is fine, does it leave me alone?**
 
-The benchmark runs the **whole tool** as a developer uses it at push time. Quick rules and the engine are on,
+The benchmark measures the **push path** as a developer uses it. The LLM detection tier (the hunter) is
-as is the LLM-guided arm. The repositories are ones the tool has **never seen**.
+required on pre-push. Today the shipped hook runs quick rules and the engine only, so the benchmark
+reports the hook as shipped and the hook plus the hunter arm separately until the hunter is wired in
+(see `requirements.md` R6). The repositories are ones the tool has **never seen**.
 
 ## What makes the data valid
 

.kiro/specs/valid-detection-benchmark/design.md

@@ -36,7 +36,9 @@ advisories (OSV/GHSA)
   -> human spot-check                record decision, reviewer, date       [manual, recorded in verification.json]
   -> draw_ordinary()                 negatives from the same repos          [reuse]
   -> assemble.py                     freeze positives+negatives -> manifest + digest, held-out guard  [new, wraps eligibility.normalize]
-  -> replay.py (docker lane)         run the whole tool at push time        [reuse]
+  -> replay.py (docker lane)         arm (a): the push hook as shipped (quick + engine)      [reuse]
+  -> hunter arm runner               arm (b): the hunter over the same changes, off-cluster  [new, group 6]
+  -> combine                         (a) and (b) joined per change id                        [new, group 6]
   -> score.py + metrics.py           recall, precision, FP/100, exploit-verified, intervals  [reuse + new]
 ```
 
@@ -108,8 +110,13 @@ Headline string, built here:
 3. `verify.py` runs the LLM check; human spot-checks all of them (dozens, so full review is cheap).
 4. `draw_ordinary()` draws >=20x negatives from the same repos.
 5. `assemble.py` freezes the manifest + digest + held-out guard.
-6. `replay.py` on the Docker lane runs the whole tool.
+6. `replay.py` on the Docker lane runs the push hook as shipped (arm a). The hunter arm runner adds arm
-7. `score.py` + `metrics.py` produce the four metrics with intervals and the headline.
+   (b) over the same changes, off-cluster, and the two are combined per change.
+7. `score.py` + `metrics.py` produce the four metrics with intervals and the headline, for (a) and for
+   (a)+(b). (b) is the number the tool is held to; the difference is the product gap on pre-push.
+
+Wiring the hunter into the pre-push hook itself is product work in `pre-push-safety-net`. This spec
+measures the push path with and without it; it does not change the hook.
 
 The pilot proves the pipeline and gives the first real numbers. Then grow positives to ~40 (set exactly
 by `required_n`) and, where the proof arm has run, fill in the exploit-verified rate.

.kiro/specs/valid-detection-benchmark/requirements.md

@@ -52,9 +52,19 @@ security metrics.
 - No repository in the benchmark was used to build, train, tune, mine, or harvest the tool. Enforce this
   with the existing used-set canonicalization (`eligibility.normalize`), the same guard the SZZ pool used.
 
-### R6. Measure the whole tool on unseen repos
+### R6. Measure the push path on unseen repos, with the LLM tier
-- Run the tool as a developer runs it at push time: quick rules on, engine on, LLM-guided arm on.
+- The tool a developer runs at push time must include the LLM detection tier (the hunter). That is a
-- Use the existing replay harness so the measured path is the shipped push hook.
+  product requirement on the pre-push hook, owned by `pre-push-safety-net`. The benchmark measures the
+  push path, not a scan mode.
+- Today the shipped hook runs quick rules and the engine only. Its model path (`push/assistance.py`)
+  selects witnesses for already-admitted defects and cannot add a finding. Until the hunter is wired into
+  the hook, the benchmark reports two arms over the same changes and labels them:
+  - (a) the push hook as shipped (quick rules + engine), through the replay harness;
+  - (b) the push hook plus the hunter arm, the hunter run off-cluster over the same changes and the
+    records combined per change.
+  (b) is the number the tool is held to. (a) is what ships today. The gap between them is product work,
+  not a benchmark artefact, and is reported as such.
+- Use the existing replay harness for (a) so the measured path is the shipped push hook.
 
 ### R7. Standard metrics
 - Report recall (true-positive rate), precision (positive predictive value), false-positive rate as

.kiro/specs/valid-detection-benchmark/tasks.md

@@ -43,13 +43,26 @@ Order is dependency order. Each task is small enough to review on its own. New c
 - [ ] 5.1 Source Python/JS advisories; run `catalog.py` to ~25 candidates. (deliberate, rate-aware)
 - [ ] 5.2 Run `verify.py` + human spot-check to land ~12 accepted positives.
 - [ ] 5.3 `assemble.py` freezes the pilot manifest + negatives (>=20x) + held-out guard.
-- [ ] 5.4 `replay.py` on the Docker lane, whole tool on (quick + engine + LLM arm).
+- [ ] 5.4 `replay.py` on the Docker lane, arm (a): the push hook as shipped (quick + engine, 300 s).
-- [ ] 5.5 `score.py` + `metrics.py`: produce recall, precision, FP/100, exploit-verified, intervals, and
+  Running overnight 2026-10-12; records under `~/ousast-results/valid/pilot/long_deadline`.
-  the headline. Freeze the run under `benchmarks/measurements/`.
+- [ ] 5.5 `score.py` + `metrics.py` on arm (a): recall, precision, FP/100, exploit-verified, intervals,
-- [ ] 5.6 Write up the pilot: the four numbers, what they mean, and the target set size from `required_n`
+  and the headline, labelled "push hook as shipped". Freeze the run under `benchmarks/measurements/`.
-  for the full ~40-positive benchmark.
+- [ ] 5.6 Write up the pilot: the four numbers for (a), what they mean, the engine-covered rate, and the
+  target set size from `required_n` for the full ~40-positive benchmark.
+
+## Group 6 - hunter arm (the LLM tier required on pre-push)
+- [ ] 6.1 `benchmarks/valid/hunter_arm.py`: for each change in the frozen manifest, check out `head` in
+  the local clone, build `FileTarget`s scoped to the changed files, run `hunter.run_hunter_pool`
+  (`src/openultrasast/hunter.py`, wiring in `cli.py` `_attach_tool_hunter`), and emit a record `score.py`
+  can read (an alerts/advisory stream keyed by change id). Runs off-cluster; the model key never enters
+  a replay container. Honors the instrument rule (bytes read per target, fail loudly on an empty read).
+- [ ] 6.2 Combine arm (a) and arm (b) records per change id and rescore: this is the number the tool is
+  held to (R6). Report (a), (a)+(b), and the difference as the pre-push product gap.
+- [ ] 6.3 Offline test with a stub hunter: a flagged change joins the (a) record; a missing (b) record
+  leaves (a) unchanged; duplicate ids are rejected.
 
 ## Out of this spec
 - Growing beyond the pilot to the full set (follow-up, once the pipeline is proven).
 - Running on kube `ax` (blocked on gVisor; Docker lane is enough here).
-- Any new detection capability.
+- Any new detection capability, including wiring the hunter into the pre-push hook. That is product
+  work in `pre-push-safety-net`; this spec measures the push path with and without it.