From 38559453118245e2b40f15c3c054be7d464549df Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Mon, 12 Oct 2026 18:24:13 +0200 Subject: [PATCH] docs: rename benchmark metrics to standard security terms Lead with the names a security engineer already knows instead of the coined ones: - catch rate -> recall (true-positive rate) - false-alarm rate -> false-positive rate (still reported as false alarms per 100 pushes) - proved-catch / proof rate -> exploit-verified findings (a working proof of concept) - precision stays precision (positive predictive value) The plain-English question for each metric stays as the explanation. Updated the valid-benchmark brief and the roadmap. No numbers changed. Co-Authored-By: Claude Opus 4.8 --- .../specs/valid-detection-benchmark/brief.md | 33 ++++++++++--------- .kiro/steering/roadmap.md | 19 ++++++----- 2 files changed, 28 insertions(+), 24 deletions(-) diff --git a/.kiro/specs/valid-detection-benchmark/brief.md b/.kiro/specs/valid-detection-benchmark/brief.md index 9524546..2b50044 100644 --- a/.kiro/specs/valid-detection-benchmark/brief.md +++ b/.kiro/specs/valid-detection-benchmark/brief.md @@ -2,7 +2,7 @@ Created 2026-10-12. This brief **supersedes the frozen-SZZ-pool evaluation** (`unseen-repo-evaluation`), which is retired. SZZ guessed its labels and placed the vulnerability in the wrong location about half the time (audit: path 1/7, -output-encoding 1/8 sound). Also, ~76% of entries are library-internal (reachability not in-repo). Catch / +output-encoding 1/8 sound). Also, ~76% of entries are library-internal (reachability not in-repo). Recall and false-alarm numbers from that pool are not valid benchmark results. This brief defines a benchmark with sound ground truth and metrics that engineers and managers can understand. @@ -41,15 +41,18 @@ tooling that found the SZZ pool's defects. Dozens of good cases are enough. ## Metrics someone understands +These are the standard detector metrics. Each row leads with the name a security engineer already knows and +gives the plain-English question it answers. + | Metric | Plain-English question | Example | |---|---|---| -| **Catch rate** (recall) | Of real vulnerabilities, how many did it flag? | "Caught 18 of 20 (90%)" | -| **False-alarm rate** | On safe changes, how often does it cry wolf? | "Flagged 3 of 100 ordinary pushes" | -| **Precision** | When it alarms, how often is it real? | "When it speaks up, it's right 8 of 10 times" | -| **Proof rate** | Of what it flags, how much can it *demonstrate*, not just suspect? | "12 of 18 catches came with a working demonstration" | +| **Recall** (true-positive rate) | Of real vulnerabilities, how many did it flag? | "Caught 18 of 20 (90%)" | +| **False-positive rate** | On safe changes, how often does it alarm? | "Flagged 3 of 100 ordinary pushes" | +| **Precision** (positive predictive value) | When it alarms, how often is it real? | "When it speaks up, it's right 8 of 10 times" | +| **Exploit-verified rate** | Of what it flags, how much can it demonstrate with a working proof of concept, not just suspect? | "12 of 18 catches came with a working proof of concept" | -Report false alarms as **"per 100 pushes"**, which reflects a developer's experience. Vulnerabilities are rare -among real pushes. Precision depends on that ratio, so a 50/50 test set overstates it. +Report the false-positive rate as **false alarms per 100 pushes**, which reflects a developer's experience. +Vulnerabilities are rare among real pushes. Precision depends on that ratio, so a 50/50 test set overstates it. Report the first two metrics together: @@ -58,20 +61,20 @@ Report the first two metrics together: This wording is easier for a manager to understand than "3/92, Wilson95 1.1–9.2%". -## The gold-standard metric — and the project's edge +## The gold-standard metric, and the project's edge -**Proved catches** do not depend on labels. The search-with-proof / differential-exploit arm demonstrates the -bad effect on the vulnerable code and its absence on the fixed code. The demonstration *is* the ground truth. -The headline to aim for is: +**Exploit-verified findings** do not depend on labels. The search-with-proof / differential-exploit arm +demonstrates the bad effect on the vulnerable code and its absence on the fixed code. A working proof of concept +is the ground truth. The headline to aim for is: -> "On unseen app repos: caught and *proved* N of M real vulnerabilities, with K false alarms per 100 ordinary -> pushes." +> "On unseen app repos: caught and exploit-verified N of M real vulnerabilities, with K false alarms per 100 +> ordinary pushes." The architecture is built to produce this kind of benchmark. ## Why the old approach was not feasible (recorded so it is not re-tried) -- SZZ ground truth is machine-guessed and ~half wrong → every catch/FP number off it is noise. +- SZZ ground truth is machine-guessed and ~half wrong → every recall or false-positive number off it is noise. - Library-heavy corpus → most cases are unreachable in-repo, so a pre-push tool is scored on things it structurally cannot see. - Scoring regex-only (engine and LLM off) measured a tier nobody would ship alone. @@ -79,7 +82,7 @@ The architecture is built to produce this kind of benchmark. ## Open questions for the spec phase -- How many verified positives are enough for a credible catch-rate interval (target set size)? +- How many verified positives are enough for a credible recall interval (target set size)? - The negative:positive ratio that reflects real push traffic (for an honest false-alarms-per-100-pushes). - Curation workflow: how LLM adjudication + human spot-check are recorded and frozen per case. - Language/framework coverage of the positive set (be explicit about what is and isn't represented). diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index 9e16605..2b220ba 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -22,8 +22,8 @@ The A/B experiments test whether the safety net works on repositories it has nev answer that question. Almost every push on an unseen repository is an ordinary change. Two numbers measure whether the safety net is useful, based on the hook's advisory output as a user sees it: -1. How often does it flag real vulnerability-introducing changes in unseen repositories (catch rate)? -2. How often does it flag anything on ordinary changes in the same unseen repositories (false-alarm rate)? +1. How often does it flag real vulnerability-introducing changes in unseen repositories (recall)? +2. How often does it flag anything on ordinary changes in the same unseen repositories (false-positive rate)? Within-pair AUC on known fix pairs answers neither. It stays a development signal, not the goal. @@ -71,7 +71,7 @@ machinery remains available. The engine/replay images are pinned. No catch/false Next, define and build the valid benchmark. Use sound, verified positives with in-repo reachability and ordinary negatives on unseen repos. Measure the whole tool with metrics an engineer understands. The design is in `.kiro/specs/valid-detection-benchmark/brief.md`. Only once that benchmark exists does any detection number -(catch rate, false-alarms-per-100-pushes, proved-catch) mean anything. The kube `ax` lane is still the place the +(recall, false alarms per 100 pushes, exploit-verified findings) mean anything. The kube `ax` lane is still the place the engine tier runs at scale, but it is infrastructure, not the gate. ### Restoring the kube `ax` lane (for G1, operator-owned) @@ -100,16 +100,17 @@ What remains before the G1 engine-arm replay: LLM adjudication and a human spot-check so the positives are sound, app-shaped and verified. 2. Ordinary negatives: everyday non-fix commits from the same application repos, a set much larger than the positives (false alarms discourage adoption). -3. Measure the **whole tool** (all tiers on) on **unseen** repos, and report metrics an engineer reads: catch - rate, **false-alarms per 100 pushes**, precision, and **proved-catch** (demonstrated, label-free). See the - brief for the full definition and why the SZZ pool was not feasible. +3. Measure the **whole tool** (all tiers on) on **unseen** repos, and report metrics an engineer reads: + **recall**, **false alarms per 100 pushes** (false-positive rate), **precision**, and **exploit-verified** + findings (demonstrated with a working proof of concept, label-free). See the brief for the full definition + and why the SZZ pool was not feasible. Exit: the benchmark exists (frozen, verified) and the whole tool has a first measured result on it. **G1. Score the arms on the valid benchmark** Quick rules, engine evidence, the LLM-guided arm, and their combination, each a registered decision rule, -scored by catch rate at a fixed false-alarms-per-100-pushes budget, with proved-catch where available. +scored by recall at a fixed false-alarms-per-100-pushes budget, with exploit-verified findings where available. Exit: one measured table of arms on unseen app-shaped code; keep only what generalises. (No number is quoted until it comes from this benchmark, never the SZZ pool.) @@ -138,7 +139,7 @@ against a realistic pre-push window (≤120 s target, 300 s hard): - **ENGINE-ON PILOT DONE (2026-10-12, clean host, 5 public repos), see brief "ENGINE-ON PILOT RESULTS".** **regex 1/5, engine 0/5.** The engine added no catches over the regex floor. The budget hypothesis is refuted: 3/5 repos finished their queue (deferred=0) and still emitted no finding, so change-scoping will not - improve catch rate. The engine ranked the vuln region in all 5 and found the right sink on 2 (socketio deser, + improve recall. The engine ranked the vuln region in all 5 and found the right sink on 2 (socketio deser, launch-editor, score 2.0), but withholds any finding it cannot anchor 1:1 to changed lines: all 5 hit `ambiguous_line_correspondence` → `witness_unresolved` → `coverage_status=incomplete`, suppressing even the scored sinks. This is not a shallow-clone artifact. Detection does not meet release requirements. @@ -168,7 +169,7 @@ against a realistic pre-push window (≤120 s target, 300 s hard): - The pilot lowers the priority of change-scoped engine analysis again. Branch `engine-change-scoped-evidence-pass` (unmerged). Fitting the budget does not produce catches (3/5 pilot repos finished the queue with zero findings), - so this is not required to improve catch rate. Keep only as a latency optimisation once detection works. + so this is not required to improve recall. Keep only as a latency optimisation once detection works. - **Line-correspondence (merged c39fbb8).** `ambiguous_line_correspondence` fires on 5/5 realistic diffs, but the pilot's corrected diagnosis shows it does not block detection here. The static tiers produce no vuln-family findings to anchor. A region-level witness fallback only matters once #1/#2 yield findings.