docs: retire the SZZ-pool metric, add the valid-benchmark brief, plain-English pass

The frozen SZZ-labeled pool is not a valid benchmark. SZZ puts the vulnerability in
the wrong place about half the time, and ~76% of entries are library-internal (not
reachable in-repo), so its catch/false-alarm numbers (record 07's "3/92") are noise.
Retired as a metric and a gate across the roadmap and briefs; no pool-scored number
is quoted anywhere.

Added .kiro/specs/valid-detection-benchmark/brief.md: positives labelled from the
fix (not SZZ), kept only when app-shaped and reachable in-repo, each verified;
ordinary negatives from the same repos; unseen repos; the whole tool measured; plain
metrics (catch rate, false-alarms per 100 pushes, precision, proved-catch).

Ran a plain-English pass (the humanizer rules) on the roadmap, README, and the two
active briefs via Codex, then reviewed the diff: only these files changed, every
number/path/hash/command is unchanged, and the retired-metric decision is intact.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

.kiro/specs/valid-detection-benchmark/brief.md

@@ -0,0 +1,85 @@
+# Valid detection benchmark
+
+Created 2026-10-12. This brief **supersedes the frozen-SZZ-pool evaluation** (`unseen-repo-evaluation`), which is retired.
+SZZ guessed its labels and placed the vulnerability in the wrong location about half the time (audit: path 1/7,
+output-encoding 1/8 sound). Also, ~76% of entries are library-internal (reachability not in-repo). Catch /
+false-alarm numbers from that pool are not valid benchmark results. This brief defines a benchmark with
+sound ground truth and metrics that engineers and managers can understand.
+
+## The two questions the benchmark answers
+
+A pre-push security check must answer two questions:
+
+1. **If I'm about to push a real vulnerability, does it stop me?**
+2. **If my change is fine, does it leave me alone?**
+
+The benchmark runs the **whole tool** as a developer uses it at push time. Quick rules and the engine are on,
+as is the LLM-guided arm. The repositories are ones the tool has **never seen**.
+
+## What makes the data valid
+
+### Positive cases — "should catch" (real, verified, reachable vulnerabilities)
+- Take the label from the security fix, never from SZZ. Use the code *just before* the fix, at the file/function
+  the fix changed. The fix establishes the vulnerability location without a blame walk-back or mislocation.
+- Keep only app-shaped cases where the vulnerability is reachable from an entry point *inside the repo*. This
+  can be a web route or a request handler; a CLI argument also qualifies. Drop library-internal vulnerabilities
+  that depend on a caller outside the repo. A pre-push tool cannot see that trust boundary, so scoring it is unfair.
+- Verify each case through LLM adjudication and a human spot-check. Confirm (a) that the pre-fix code is
+  vulnerable and (b) that the fix addresses it. Also confirm (c) that it is reachable in-repo.
+
+### Negative cases — "should not alarm" (ordinary safe changes)
+- Use everyday commits from the **same** application repositories that are not security fixes. They are plentiful
+  and cheap to collect. Almost every real push is ordinary, so false alarms determine whether developers keep
+  using the tool. The negative set must be much larger than the positive set.
+
+### Held out
+- None of these repositories were used to build, train, tune, mine or harvest the tool.
+
+A few dozen vulnerabilities with sound labels and in-repo reachability, each verified, are better than hundreds
+of SZZ guesses. SZZ is wrong about half the time. Curating this set takes manual work, supported by the LLM-audit
+tooling that found the SZZ pool's defects. Dozens of good cases are enough.
+
+## Metrics someone understands
+
+| Metric | Plain-English question | Example |
+|---|---|---|
+| **Catch rate** (recall) | Of real vulnerabilities, how many did it flag? | "Caught 18 of 20 (90%)" |
+| **False-alarm rate** | On safe changes, how often does it cry wolf? | "Flagged 3 of 100 ordinary pushes" |
+| **Precision** | When it alarms, how often is it real? | "When it speaks up, it's right 8 of 10 times" |
+| **Proof rate** | Of what it flags, how much can it *demonstrate*, not just suspect? | "12 of 18 catches came with a working demonstration" |
+
+Report false alarms as **"per 100 pushes"**, which reflects a developer's experience. Vulnerabilities are rare
+among real pushes. Precision depends on that ratio, so a 50/50 test set overstates it.
+
+Report the first two metrics together:
+
+> "On unseen application repos: catches 90% of real vulnerabilities while false-alarming on 3% of ordinary
+> pushes."
+
+This wording is easier for a manager to understand than "3/92, Wilson95 1.1–9.2%".
+
+## The gold-standard metric — and the project's edge
+
+**Proved catches** do not depend on labels. The search-with-proof / differential-exploit arm demonstrates the
+bad effect on the vulnerable code and its absence on the fixed code. The demonstration *is* the ground truth.
+The headline to aim for is:
+
+> "On unseen app repos: caught and *proved* N of M real vulnerabilities, with K false alarms per 100 ordinary
+> pushes."
+
+The architecture is built to produce this kind of benchmark.
+
+## Why the old approach was not feasible (recorded so it is not re-tried)
+
+- SZZ ground truth is machine-guessed and ~half wrong → every catch/FP number off it is noise.
+- Library-heavy corpus → most cases are unreachable in-repo, so a pre-push tool is scored on things it
+  structurally cannot see.
+- Scoring regex-only (engine and LLM off) measured a tier nobody would ship alone.
+- Automated harvesting prioritized pool size over sound ground truth and in-repo reachability.
+
+## Open questions for the spec phase
+
+- How many verified positives are enough for a credible catch-rate interval (target set size)?
+- The negative:positive ratio that reflects real push traffic (for an honest false-alarms-per-100-pushes).
+- Curation workflow: how LLM adjudication + human spot-check are recorded and frozen per case.
+- Language/framework coverage of the positive set (be explicit about what is and isn't represented).