mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
Marius CiepluchandClaude Opus 4.8
b08f372022
The frozen SZZ-labeled pool is not a valid benchmark. SZZ puts the vulnerability in the wrong place about half the time, and ~76% of entries are library-internal (not reachable in-repo), so its catch/false-alarm numbers (record 07's "3/92") are noise. Retired as a metric and a gate across the roadmap and briefs; no pool-scored number is quoted anywhere. Added .kiro/specs/valid-detection-benchmark/brief.md: positives labelled from the fix (not SZZ), kept only when app-shaped and reachable in-repo, each verified; ordinary negatives from the same repos; unseen repos; the whole tool measured; plain metrics (catch rate, false-alarms per 100 pushes, precision, proved-catch). Ran a plain-English pass (the humanizer rules) on the roadmap, README, and the two active briefs via Codex, then reviewed the diff: only these files changed, every number/path/hash/command is unchanged, and the retired-metric decision is intact. Co-Authored-By: Claude Opus 4.8 <[email protected]>