docs(benchmark): R6 holds the push path to the LLM tier; measure hook as shipped and hook + hunter

The pre-push hook has no LLM detection tier today: push/assistance.py only
selects a witness for already-admitted defects and cannot add a finding. The
hunter lives in scan --mode deep. The maintainer's call is that the LLM tier is
required on pre-push, so R6 now measures two arms over the same changes:
(a) the hook as shipped (quick rules + engine), (b) the hook plus the hunter
arm run off-cluster. (b) is the number the tool is held to; the gap is product
work in pre-push-safety-net, not a benchmark artefact.

Adds group 6 (hunter arm runner + combine + offline test) to tasks, aligns the
brief and design, and labels the overnight replay as arm (a).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-10-12 20:17:44 +02:00
co-authored by Claude Fable 5.1
parent 7a636d60fb
commit 98825db94a
4 changed files with 46 additions and 14 deletions
@@ -13,8 +13,10 @@ A pre-push security check must answer two questions:
1. **If I'm about to push a real vulnerability, does it stop me?**
2. **If my change is fine, does it leave me alone?**
The benchmark runs the **whole tool** as a developer uses it at push time. Quick rules and the engine are on,
as is the LLM-guided arm. The repositories are ones the tool has **never seen**.
The benchmark measures the **push path** as a developer uses it. The LLM detection tier (the hunter) is
required on pre-push. Today the shipped hook runs quick rules and the engine only, so the benchmark
reports the hook as shipped and the hook plus the hunter arm separately until the hunter is wired in
(see `requirements.md` R6). The repositories are ones the tool has **never seen**.
## What makes the data valid