The pre-push hook has no LLM detection tier today: push/assistance.py only
selects a witness for already-admitted defects and cannot add a finding. The
hunter lives in scan --mode deep. The maintainer's call is that the LLM tier is
required on pre-push, so R6 now measures two arms over the same changes:
(a) the hook as shipped (quick rules + engine), (b) the hook plus the hunter
arm run off-cluster. (b) is the number the tool is held to; the gap is product
work in pre-push-safety-net, not a benchmark artefact.
Adds group 6 (hunter arm runner + combine + offline test) to tasks, aligns the
brief and design, and labels the overnight replay as arm (a).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Builds the benchmark from .kiro/specs/valid-detection-benchmark/brief.md:
sound labels from the fix (no SZZ), app-shaped verified positives,
ordinary negatives from the same repos, whole tool on unseen repos,
standard metrics.
Spec: requirements.md, design.md, tasks.md. Reuses the existing sourcing,
fix-site labelling, negative drawing, replay, and Wilson scoring; drops
the SZZ layer.
New offline modules under benchmarks/valid/:
- catalog.py positives labelled at the fix site (never introducing/last_touch)
- verify.py one LLM call per case (vulnerable? fix addresses it? reachable
in-repo?) + human spot-check; this is the app-shaped filter.
Honors the instrument rule: bytes read per function, reject on
an empty read.
- assemble.py >=20x ordinary negatives + a frozen, deterministic manifest with
the held-out guard
- metrics.py precision (PPV), false alarms per 100 pushes, exploit-verified
rate, and the plain headline, layered on score.py
36 new offline tests, 495 regression tests green. No network, no harvest,
no model calls in tests. Group 5 (the live pilot) is not started.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
