feat(benchmark): valid detection benchmark spec + offline modules (groups 1-4)

Builds the benchmark from .kiro/specs/valid-detection-benchmark/brief.md:
sound labels from the fix (no SZZ), app-shaped verified positives,
ordinary negatives from the same repos, whole tool on unseen repos,
standard metrics.

Spec: requirements.md, design.md, tasks.md. Reuses the existing sourcing,
fix-site labelling, negative drawing, replay, and Wilson scoring; drops
the SZZ layer.

New offline modules under benchmarks/valid/:
- catalog.py  positives labelled at the fix site (never introducing/last_touch)
- verify.py   one LLM call per case (vulnerable? fix addresses it? reachable
              in-repo?) + human spot-check; this is the app-shaped filter.
              Honors the instrument rule: bytes read per function, reject on
              an empty read.
- assemble.py >=20x ordinary negatives + a frozen, deterministic manifest with
              the held-out guard
- metrics.py  precision (PPV), false alarms per 100 pushes, exploit-verified
              rate, and the plain headline, layered on score.py

36 new offline tests, 495 regression tests green. No network, no harvest,
no model calls in tests. Group 5 (the live pilot) is not started.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

.kiro/specs/valid-detection-benchmark/tasks.md

@@ -0,0 +1,55 @@
+# Tasks: valid detection benchmark
+
+Order is dependency order. Each task is small enough to review on its own. New code under
+`benchmarks/valid/`. Implementation goes through Codex, gated here.
+
+## Group 1 - positive catalog (fix-site, no SZZ)
+- [x] 1.1 `benchmarks/valid/catalog.py`: given sourced advisories (reuse `source.py`/`draft.py` output),
+  build candidate positive cases by fix site. Wrap `extract.fix_sites()` + `pairs/materialize.py`; read
+  the pre-fix function at `parent` and the post-fix function at `commit`. Must not call `introducing` or
+  `last_touch`. Emit the `[[case]]` catalog in the design schema. (R1)
+- [x] 1.2 Filter to Python and JS/TS only; attach `language` and `family`/`cwe`. (scope decision)
+- [x] 1.3 Offline test: a fixture commit yields one case with the correct pre/post functions and
+  `method = "fix_site"`; assert SZZ methods are never invoked.
+
+## Group 2 - LLM verification driver
+- [x] 2.1 `benchmarks/valid/verify.py`: one DeepSeek call per case via the shipped provider (off-cluster).
+  Confirm vulnerable-pre-fix, fix-addresses-it, reachable-in-repo; name the entry point; return
+  confidence. Print bytes read per function before the call; reject a verdict on an empty read. (R3, instrument rule)
+- [x] 2.2 Emit the verification record (design schema) to `verification.json`; `verdict = "accepted"`
+  only when all three checks pass. Reachability here is the app-shaped filter. (R2, R3)
+- [x] 2.3 Human spot-check hook: a small CLI to record accept/reject + reviewer + date per case into the
+  same record. (R3)
+- [x] 2.4 Offline test with a stub model: a case with all three true and a human accept becomes
+  `accepted`; any false, or a human reject, does not.
+
+## Group 3 - negatives and assembly
+- [x] 3.1 `benchmarks/valid/assemble.py`: draw negatives with `extract.draw_ordinary()` from the accepted
+  positives' repos, at least 20x, excluding the fix. (R4)
+- [x] 3.2 Freeze accepted positives + negatives into a `pool.toml`-shaped manifest + `freeze-valid.json`
+  digest; run the held-out guard via `eligibility.normalize`. (R5, R8)
+- [x] 3.3 Offline test: freeze is deterministic (same source -> same digest); held-out guard drops any
+  repo in the used set.
+
+## Group 4 - metrics
+- [x] 4.1 `benchmarks/valid/metrics.py`: precision (PPV) with Wilson 95% over positives+negatives. (R7)
+- [x] 4.2 False alarms per 100 pushes as the presentation of `score.py`'s false-alarm rate. (R7)
+- [x] 4.3 Exploit-verified rate: join flagged positives with a search-with-proof run by case `id`;
+  report "not yet proved" where absent. (R7)
+- [x] 4.4 Build the headline string (recall + FP/100 + precision + exploit-verified). (R7)
+- [x] 4.5 Offline test on a tiny hand-made record set: all four metrics and intervals match hand calcs.
+
+## Group 5 - pilot run (research, local)
+- [ ] 5.1 Source Python/JS advisories; run `catalog.py` to ~25 candidates. (deliberate, rate-aware)
+- [ ] 5.2 Run `verify.py` + human spot-check to land ~12 accepted positives.
+- [ ] 5.3 `assemble.py` freezes the pilot manifest + negatives (>=20x) + held-out guard.
+- [ ] 5.4 `replay.py` on the Docker lane, whole tool on (quick + engine + LLM arm).
+- [ ] 5.5 `score.py` + `metrics.py`: produce recall, precision, FP/100, exploit-verified, intervals, and
+  the headline. Freeze the run under `benchmarks/measurements/`.
+- [ ] 5.6 Write up the pilot: the four numbers, what they mean, and the target set size from `required_n`
+  for the full ~40-positive benchmark.
+
+## Out of this spec
+- Growing beyond the pilot to the full set (follow-up, once the pipeline is proven).
+- Running on kube `ax` (blocked on gVisor; Docker lane is enough here).
+- Any new detection capability.