Lead with the names a security engineer already knows instead of the
coined ones:
- catch rate -> recall (true-positive rate)
- false-alarm rate -> false-positive rate (still reported as false
alarms per 100 pushes)
- proved-catch / proof rate -> exploit-verified findings (a working
proof of concept)
- precision stays precision (positive predictive value)
The plain-English question for each metric stays as the explanation.
Updated the valid-benchmark brief and the roadmap. No numbers changed.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The frozen SZZ-labeled pool is not a valid benchmark. SZZ puts the vulnerability in
the wrong place about half the time, and ~76% of entries are library-internal (not
reachable in-repo), so its catch/false-alarm numbers (record 07's "3/92") are noise.
Retired as a metric and a gate across the roadmap and briefs; no pool-scored number
is quoted anywhere.
Added .kiro/specs/valid-detection-benchmark/brief.md: positives labelled from the
fix (not SZZ), kept only when app-shaped and reachable in-repo, each verified;
ordinary negatives from the same repos; unseen repos; the whole tool measured; plain
metrics (catch rate, false-alarms per 100 pushes, precision, proved-catch).
Ran a plain-English pass (the humanizer rules) on the roadmap, README, and the two
active briefs via Codex, then reviewed the diff: only these files changed, every
number/path/hash/command is unchanged, and the retired-metric decision is intact.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
