mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-19 04:59:28 +00:00
docs(benchmark): R6 holds the push path to the LLM tier; measure hook as shipped and hook + hunter
The pre-push hook has no LLM detection tier today: push/assistance.py only selects a witness for already-admitted defects and cannot add a finding. The hunter lives in scan --mode deep. The maintainer's call is that the LLM tier is required on pre-push, so R6 now measures two arms over the same changes: (a) the hook as shipped (quick rules + engine), (b) the hook plus the hunter arm run off-cluster. (b) is the number the tool is held to; the gap is product work in pre-push-safety-net, not a benchmark artefact. Adds group 6 (hunter arm runner + combine + offline test) to tasks, aligns the brief and design, and labels the overnight replay as arm (a). Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
7a636d60fb
commit
98825db94a
@@ -13,8 +13,10 @@ A pre-push security check must answer two questions:
|
||||
1. **If I'm about to push a real vulnerability, does it stop me?**
|
||||
2. **If my change is fine, does it leave me alone?**
|
||||
|
||||
The benchmark runs the **whole tool** as a developer uses it at push time. Quick rules and the engine are on,
|
||||
as is the LLM-guided arm. The repositories are ones the tool has **never seen**.
|
||||
The benchmark measures the **push path** as a developer uses it. The LLM detection tier (the hunter) is
|
||||
required on pre-push. Today the shipped hook runs quick rules and the engine only, so the benchmark
|
||||
reports the hook as shipped and the hook plus the hunter arm separately until the hunter is wired in
|
||||
(see `requirements.md` R6). The repositories are ones the tool has **never seen**.
|
||||
|
||||
## What makes the data valid
|
||||
|
||||
|
||||
Reference in New Issue
Block a user