marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-22 22:49:29 +00:00
Code Issues Packages Projects Releases Wiki Activity

measure(g0/g2): LLM corpus-label audit -- only 47% of public pool labels are sound

Browse Source 
Widened the banks label-mislocation finding across all 49 public pool vulns via DeepSeek
(advisory + head code + pool label -> sound/unsound), $0.038 total. Result: 23/49 (47%)
sound; 53% unsound (25 pre-exist in base, 16 sink outside the changed lines) -- a change-scoped
hook correctly stays quiet there. Systematic by family: injection 10/13 sound, but path 1/7,
output_encoding 1/8, config_secrets 1/4.

Implications: record-07's floor (catch 3/92) is scored against a corpus ~half unfair to a delta
hook; the honest catchable denominator is the sound subset (caveat added to record 07 NOTES).
Fair pilot re-score: only bedrock + graphiti (injection) of the 5 are sound and the engine
missed both = the real detection gaps for the LLM-guided hunter.

Also: GEPA-style reflective-retry (future spec) + GLM cyber model (future spec) noted; DeepSeek
is the working default. Artifact: benchmarks/measurements/2026-10-12-pool-p1-label-audit/.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-10-12 10:09:32 +02:00
co-authored by Claude Opus 4.8
parent 53b1d31d82
commit a43e6293f5
5 changed files with 170 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benchmarks/measurements/2026-10-11-unseen-07-slice1-baseline/NOTES.md
+6
View File
@@ -23,3 +23,9 @@ Ledger: benchmarks/unseen/usage-p1.jsonl — one `baseline` row (this is G0's ex
- **Sizing**: 100 vulns is ~±10% for one overall catch number; per-family (deser=4, config=10) is far too thin for
family-level decisions — needs slices 2+.
- Raw per-change records streamed to the RustFS store; local subdirs pruned-as-we-go (records + inputs.json kept).
- **CORPUS-LABEL CAVEAT (added 2026-10-12):** an LLM audit of the 49 public pool vulns found only **47% (23/49)
of labels are sound** (sink in the changed lines AND change-introduced); 53% pre-exist in base or are mislocated,
where a change-scoped hook *correctly* stays quiet. So this catch rate is scored against a corpus ~half of which
is unfair to a delta hook — the honest catchable denominator is the sound subset, and the floor should be
recomputed on it. By family the mislocation is systematic (path 1/7, output_encoding 1/8 sound; injection 10/13).
See `benchmarks/measurements/2026-10-12-pool-p1-label-audit/`.