marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-19 04:59:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(benchmark): R6 holds the push path to the LLM tier; measure hook as shipped and hook + hunter

Browse Source 
The pre-push hook has no LLM detection tier today: push/assistance.py only
selects a witness for already-admitted defects and cannot add a finding. The
hunter lives in scan --mode deep. The maintainer's call is that the LLM tier is
required on pre-push, so R6 now measures two arms over the same changes:
(a) the hook as shipped (quick rules + engine), (b) the hook plus the hunter
arm run off-cluster. (b) is the number the tool is held to; the gap is product
work in pre-push-safety-net, not a benchmark artefact.

Adds group 6 (hunter arm runner + combine + offline test) to tasks, aligns the
brief and design, and labels the overnight replay as arm (a).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-10-12 20:17:44 +02:00
co-authored by Claude Fable 5.1
parent 7a636d60fb
commit 98825db94a
4 changed files with 46 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/specs/valid-detection-benchmark/design.md
+10 -3
View File
@@ -36,7 +36,9 @@ advisories (OSV/GHSA)
-> human spot-check record decision, reviewer, date [manual, recorded in verification.json]
-> draw_ordinary() negatives from the same repos [reuse]
-> assemble.py freeze positives+negatives -> manifest + digest, held-out guard [new, wraps eligibility.normalize]
-> replay.py (docker lane) run the whole tool at push time [reuse]
-> replay.py (docker lane) arm (a): the push hook as shipped (quick + engine) [reuse]
-> hunter arm runner arm (b): the hunter over the same changes, off-cluster [new, group 6]
-> combine (a) and (b) joined per change id [new, group 6]
-> score.py + metrics.py recall, precision, FP/100, exploit-verified, intervals [reuse + new]
```
@@ -108,8 +110,13 @@ Headline string, built here:
3. `verify.py` runs the LLM check; human spot-checks all of them (dozens, so full review is cheap).
4. `draw_ordinary()` draws >=20x negatives from the same repos.
5. `assemble.py` freezes the manifest + digest + held-out guard.
6. `replay.py` on the Docker lane runs the whole tool.
7. `score.py` + `metrics.py` produce the four metrics with intervals and the headline.
6. `replay.py` on the Docker lane runs the push hook as shipped (arm a). The hunter arm runner adds arm
(b) over the same changes, off-cluster, and the two are combined per change.
7. `score.py` + `metrics.py` produce the four metrics with intervals and the headline, for (a) and for
(a)+(b). (b) is the number the tool is held to; the difference is the product gap on pre-push.
Wiring the hunter into the pre-push hook itself is product work in `pre-push-safety-net`. This spec
measures the push path with and without it; it does not change the hook.
The pilot proves the pipeline and gives the first real numbers. Then grow positives to ~40 (set exactly
by `required_n`) and, where the proof arm has run, fill in the exploit-verified rate.