marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(benchmark): hunter arm runner and combiner (group 6)

Browse Source 
benchmarks/valid/hunter_arm.py runs the shipped deep-mode tool hunter
(tool_hunter.run_tool_hunter via resolve_hunter_client) over each frozen
change off-cluster: detached worktree at head with git hooks disabled,
hotspots scoped to the changed files, instrument rule on every hotspot
(bytes read printed, empty or unreadable fails the change loudly), model id
from the config resolver, family from the taxonomy, never from the label.
Records land in the exact shape score.py reads (push.admission.defects with
locations), resumable per change.

`combine` joins arm (a) replay records with arm (b) hunter records per change
id, dedupes defects by (path, line, family), keeps (a)'s instrument, records
which arms were present, and writes a file score.py loads directly.

9 offline tests (stub hunter, real scorer), 53 green with the benchmark
tests. No network, no key handling, no live run.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-10-12 20:29:03 +02:00
co-authored by Claude Fable 5.1
parent bd7b7db25d
commit 9f7ca132ff
3 changed files with 614 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/specs/valid-detection-benchmark/tasks.md
+9 -7
View File
@@ -51,14 +51,16 @@ Order is dependency order. Each task is small enough to review on its own. New c
target set size from `required_n` for the full ~40-positive benchmark.
## Group 6 - hunter arm (the LLM tier required on pre-push)
- [ ] 6.1 `benchmarks/valid/hunter_arm.py`: for each change in the frozen manifest, check out `head` in
the local clone, build `FileTarget`s scoped to the changed files, run `hunter.run_hunter_pool`
(`src/openultrasast/hunter.py`, wiring in `cli.py` `_attach_tool_hunter`), and emit a record `score.py`
can read (an alerts/advisory stream keyed by change id). Runs off-cluster; the model key never enters
a replay container. Honors the instrument rule (bytes read per target, fail loudly on an empty read).
- [ ] 6.2 Combine arm (a) and arm (b) records per change id and rescore: this is the number the tool is
- [x] 6.1 `benchmarks/valid/hunter_arm.py`: for each change in the frozen manifest, check out `head` in
the local clone (detached worktree), build hotspots for the changed files, run the shipped deep-mode
arm `tool_hunter.run_tool_hunter` with `resolve_hunter_client()` (the hunter that beat overlay 12/14;
wiring in `cli.py` `_attach_tool_hunter`), and emit a record `score.py` can read: findings go under
`push.admission.defects` with `family` and `locations = [[path, line]]` (the alerts stream), keyed by
change id. Runs off-cluster; the model key never enters a replay container. Honors the instrument rule
(bytes read per hotspot, fail loudly on an empty read).
- [x] 6.2 Combine arm (a) and arm (b) records per change id and rescore: this is the number the tool is
held to (R6). Report (a), (a)+(b), and the difference as the pre-push product gap.
- [ ] 6.3 Offline test with a stub hunter: a flagged change joins the (a) record; a missing (b) record
- [x] 6.3 Offline test with a stub hunter: a flagged change joins the (a) record; a missing (b) record
leaves (a) unchanged; duplicate ids are rejected.
## Out of this spec