Marius CiepluchandClaude Opus 4.8
b08f372022
The frozen SZZ-labeled pool is not a valid benchmark. SZZ puts the vulnerability in the wrong place about half the time, and ~76% of entries are library-internal (not reachable in-repo), so its catch/false-alarm numbers (record 07's "3/92") are noise. Retired as a metric and a gate across the roadmap and briefs; no pool-scored number is quoted anywhere. Added .kiro/specs/valid-detection-benchmark/brief.md: positives labelled from the fix (not SZZ), kept only when app-shaped and reachable in-repo, each verified; ordinary negatives from the same repos; unseen repos; the whole tool measured; plain metrics (catch rate, false-alarms per 100 pushes, precision, proved-catch). Ran a plain-English pass (the humanizer rules) on the roadmap, README, and the two active briefs via Codex, then reviewed the diff: only these files changed, every number/path/hash/command is unchanged, and the retired-metric decision is intact. Co-Authored-By: Claude Opus 4.8 <[email protected]>
4.9 KiB
Valid detection benchmark
Created 2026-10-12. This brief supersedes the frozen-SZZ-pool evaluation (unseen-repo-evaluation), which is retired.
SZZ guessed its labels and placed the vulnerability in the wrong location about half the time (audit: path 1/7,
output-encoding 1/8 sound). Also, ~76% of entries are library-internal (reachability not in-repo). Catch /
false-alarm numbers from that pool are not valid benchmark results. This brief defines a benchmark with
sound ground truth and metrics that engineers and managers can understand.
The two questions the benchmark answers
A pre-push security check must answer two questions:
- If I'm about to push a real vulnerability, does it stop me?
- If my change is fine, does it leave me alone?
The benchmark runs the whole tool as a developer uses it at push time. Quick rules and the engine are on, as is the LLM-guided arm. The repositories are ones the tool has never seen.
What makes the data valid
Positive cases — "should catch" (real, verified, reachable vulnerabilities)
- Take the label from the security fix, never from SZZ. Use the code just before the fix, at the file/function the fix changed. The fix establishes the vulnerability location without a blame walk-back or mislocation.
- Keep only app-shaped cases where the vulnerability is reachable from an entry point inside the repo. This can be a web route or a request handler; a CLI argument also qualifies. Drop library-internal vulnerabilities that depend on a caller outside the repo. A pre-push tool cannot see that trust boundary, so scoring it is unfair.
- Verify each case through LLM adjudication and a human spot-check. Confirm (a) that the pre-fix code is vulnerable and (b) that the fix addresses it. Also confirm (c) that it is reachable in-repo.
Negative cases — "should not alarm" (ordinary safe changes)
- Use everyday commits from the same application repositories that are not security fixes. They are plentiful and cheap to collect. Almost every real push is ordinary, so false alarms determine whether developers keep using the tool. The negative set must be much larger than the positive set.
Held out
- None of these repositories were used to build, train, tune, mine or harvest the tool.
A few dozen vulnerabilities with sound labels and in-repo reachability, each verified, are better than hundreds of SZZ guesses. SZZ is wrong about half the time. Curating this set takes manual work, supported by the LLM-audit tooling that found the SZZ pool's defects. Dozens of good cases are enough.
Metrics someone understands
| Metric | Plain-English question | Example |
|---|---|---|
| Catch rate (recall) | Of real vulnerabilities, how many did it flag? | "Caught 18 of 20 (90%)" |
| False-alarm rate | On safe changes, how often does it cry wolf? | "Flagged 3 of 100 ordinary pushes" |
| Precision | When it alarms, how often is it real? | "When it speaks up, it's right 8 of 10 times" |
| Proof rate | Of what it flags, how much can it demonstrate, not just suspect? | "12 of 18 catches came with a working demonstration" |
Report false alarms as "per 100 pushes", which reflects a developer's experience. Vulnerabilities are rare among real pushes. Precision depends on that ratio, so a 50/50 test set overstates it.
Report the first two metrics together:
"On unseen application repos: catches 90% of real vulnerabilities while false-alarming on 3% of ordinary pushes."
This wording is easier for a manager to understand than "3/92, Wilson95 1.1–9.2%".
The gold-standard metric — and the project's edge
Proved catches do not depend on labels. The search-with-proof / differential-exploit arm demonstrates the bad effect on the vulnerable code and its absence on the fixed code. The demonstration is the ground truth. The headline to aim for is:
"On unseen app repos: caught and proved N of M real vulnerabilities, with K false alarms per 100 ordinary pushes."
The architecture is built to produce this kind of benchmark.
Why the old approach was not feasible (recorded so it is not re-tried)
- SZZ ground truth is machine-guessed and ~half wrong → every catch/FP number off it is noise.
- Library-heavy corpus → most cases are unreachable in-repo, so a pre-push tool is scored on things it structurally cannot see.
- Scoring regex-only (engine and LLM off) measured a tier nobody would ship alone.
- Automated harvesting prioritized pool size over sound ground truth and in-repo reachability.
Open questions for the spec phase
- How many verified positives are enough for a credible catch-rate interval (target set size)?
- The negative:positive ratio that reflects real push traffic (for an honest false-alarms-per-100-pushes).
- Curation workflow: how LLM adjudication + human spot-check are recorded and frozen per case.
- Language/framework coverage of the positive set (be explicit about what is and isn't represented).