marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-19 13:10:03 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(benchmark): R6 holds the push path to the LLM tier; measure hook as shipped and hook + hunter

Browse Source 
The pre-push hook has no LLM detection tier today: push/assistance.py only
selects a witness for already-admitted defects and cannot add a finding. The
hunter lives in scan --mode deep. The maintainer's call is that the LLM tier is
required on pre-push, so R6 now measures two arms over the same changes:
(a) the hook as shipped (quick rules + engine), (b) the hook plus the hunter
arm run off-cluster. (b) is the number the tool is held to; the gap is product
work in pre-push-safety-net, not a benchmark artefact.

Adds group 6 (hunter arm runner + combine + offline test) to tasks, aligns the
brief and design, and labels the overnight replay as arm (a).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-10-12 20:17:44 +02:00
co-authored by Claude Fable 5.1
parent 7a636d60fb
commit 98825db94a
4 changed files with 46 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/specs/valid-detection-benchmark/requirements.md
+13 -3
View File
@@ -52,9 +52,19 @@ security metrics.
- No repository in the benchmark was used to build, train, tune, mine, or harvest the tool. Enforce this
with the existing used-set canonicalization (`eligibility.normalize`), the same guard the SZZ pool used.
### R6. Measure the whole tool on unseen repos
- Run the tool as a developer runs it at push time: quick rules on, engine on, LLM-guided arm on.
- Use the existing replay harness so the measured path is the shipped push hook.
### R6. Measure the push path on unseen repos, with the LLM tier
- The tool a developer runs at push time must include the LLM detection tier (the hunter). That is a
product requirement on the pre-push hook, owned by `pre-push-safety-net`. The benchmark measures the
push path, not a scan mode.
- Today the shipped hook runs quick rules and the engine only. Its model path (`push/assistance.py`)
selects witnesses for already-admitted defects and cannot add a finding. Until the hunter is wired into
the hook, the benchmark reports two arms over the same changes and labels them:
- (a) the push hook as shipped (quick rules + engine), through the replay harness;
- (b) the push hook plus the hunter arm, the hunter run off-cluster over the same changes and the
records combined per change.
(b) is the number the tool is held to. (a) is what ships today. The gap between them is product work,
not a benchmark artefact, and is reported as such.
- Use the existing replay harness for (a) so the measured path is the shipped push hook.
### R7. Standard metrics
- Report recall (true-positive rate), precision (positive predictive value), false-positive rate as