feat(benchmark): valid detection benchmark spec + offline modules (groups 1-4)

Builds the benchmark from .kiro/specs/valid-detection-benchmark/brief.md: sound labels from the fix (no SZZ), app-shaped verified positives, ordinary negatives from the same repos, whole tool on unseen repos, standard metrics. Spec: requirements.md, design.md, tasks.md. Reuses the existing sourcing, fix-site labelling, negative drawing, replay, and Wilson scoring; drops the SZZ layer. New offline modules under benchmarks/valid/: - catalog.py positives labelled at the fix site (never introducing/last_touch) - verify.py one LLM call per case (vulnerable? fix addresses it? reachable in-repo?) + human spot-check; this is the app-shaped filter. Honors the instrument rule: bytes read per function, reject on an empty read. - assemble.py >=20x ordinary negatives + a frozen, deterministic manifest with the held-out guard - metrics.py precision (PPV), false alarms per 100 pushes, exploit-verified rate, and the plain headline, layered on score.py 36 new offline tests, 495 regression tests green. No network, no harvest, no model calls in tests. Group 5 (the live pilot) is not started. Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-16 03:49:26 +00:00 · 2026-10-12 18:55:42 +02:00
parent 3855945311
commit 43ef254134
9 changed files with 1296 additions and 0 deletions
@@ -0,0 +1,122 @@
+# Design: valid detection benchmark
+
+Builds the benchmark in R1-R8. New code lives under `benchmarks/valid/`. It reuses the sourcing,
+labelling, negative-drawing, replay, and scoring that already exist, and drops the SZZ layer.
+
+## What is reused and what is new
+
+Reused as-is:
+- Sourcing advisories to repos and fix commits: `benchmarks/unseen/source.py`, `draft.py`,
+  `eligibility.py` (the used-set held-out guard).
+- Fix-site labelling: `benchmarks/unseen/extract.py:fix_sites()` and the pair builder in
+  `benchmarks/pairs/advisory-fixes/` (`materialize.py`, `catalog.toml`, `verification.json`).
+- Function cutting: `openultrasast.learn.excerpt.function_bounds`.
+- Ordinary negatives: `extract.py:ordinary_history()` and `draw_ordinary()`.
+- Replay: `benchmarks/unseen/replay.py` and `replay_entry.py` (whole tool via the push hook).
+- Scoring base: `benchmarks/unseen/score.py` (catch, false alarm, Wilson, repo-clustered interval,
+  required_n, McNemar gate).
+
+New:
+1. `benchmarks/valid/catalog.py` - sourcing to a fix-site positive catalog, SZZ paths excluded.
+2. `benchmarks/valid/verify.py` - the committed LLM verification driver (replaces the scratchpad
+   `label_audit.py`). One call per case confirms vulnerable / fix-addresses / reachable-in-repo. This is
+   also the app-shaped filter: reachability is answered here, not by a separate classifier.
+3. `benchmarks/valid/metrics.py` - precision, false alarms per 100 pushes, and exploit-verified rate,
+   layered on `score.py` output.
+4. `benchmarks/valid/assemble.py` - freezes the verified positives plus the negative draw into a
+   replay manifest with a digest and the held-out guard.
+
+## Data flow
+
+```
+advisories (OSV/GHSA)
+  -> source.py / draft.py            resolve repo + fix commit + parent   [reuse]
+  -> catalog.py                      fix-site label (no SZZ); emit candidate cases  [new, wraps extract.fix_sites + pairs materialize]
+  -> verify.py                       LLM: vulnerable? fix addresses it? reachable in-repo? + confidence  [new]
+  -> human spot-check                record decision, reviewer, date       [manual, recorded in verification.json]
+  -> draw_ordinary()                 negatives from the same repos          [reuse]
+  -> assemble.py                     freeze positives+negatives -> manifest + digest, held-out guard  [new, wraps eligibility.normalize]
+  -> replay.py (docker lane)         run the whole tool at push time        [reuse]
+  -> score.py + metrics.py           recall, precision, FP/100, exploit-verified, intervals  [reuse + new]
+```
+
+## Schemas
+
+### Positive case (catalog.py output, extends `advisory-fixes/catalog.toml`)
+```
+[[case]]
+id            = "owner-repo-cve"
+repo          = "owner/repo"
+url           = "https://github.com/owner/repo"
+advisory      = "GHSA-xxxx / CVE-xxxx"
+cwe           = "CWE-89"
+family        = "sql_injection"
+language      = "python"
+commit        = "<fix sha>"      # post-fix (safe) side
+parent        = "<parent sha>"   # pre-fix (vulnerable) side
+path          = "app/db.py"
+function       = "get_member_by_code"
+function_lines = [[40, 58]]
+method        = "fix_site"       # never introducing/last_touch
+```
+
+### Verification record (verify.py output, extends `advisory-fixes/verification.json`)
+```
+{ "id": "...", "vulnerable_pre_fix": true, "fix_addresses_it": true,
+  "reachable_in_repo": true, "entry_point": "flask route /member/<code>",
+  "confidence": 0.86, "model": "deepseek-...", "advisory_cited": true,
+  "human_spot_check": {"decision": "accept", "reviewer": "<initials>", "date": "2026-10-13"},
+  "verdict": "accepted" }   # accepted only if all three true AND human accept
+```
+
+### Replay manifest (assemble.py output)
+A `pool.toml`-shaped file so `replay.py` consumes it unchanged: `[[repository]]` rows with `changes`,
+each change `kind = "vulnerability"` (positives, label from fix) or `kind = "ordinary"` (negatives),
+with `base`/`head`/`files`/`function`. A `freeze-valid.json` carries the freeze digest and the held-out
+guard result, matching the SZZ pool's freeze format so the ledger and replay flow are identical.
+
+## Metrics (metrics.py)
+
+Input is the scored records from `score.py`. New outputs:
+- **precision** = flagged-and-real / all-flagged, over positives and negatives together, with Wilson 95%.
+- **false alarms per 100 pushes** = false-positive rate x 100, the presentation of `score.py`'s
+  `false_alarm` rate. This is a presentation layer, not a new measurement.
+- **exploit-verified rate** = of flagged positives, the share the search-with-proof arm
+  (`src/openultrasast/search/`) demonstrates on the vulnerable side and not the fixed side. Joined by
+  case `id` from a proof-arm run; absent where the arm has not run (reported as "not yet proved", not as
+  a failure).
+
+Headline string, built here:
+> "On unseen app repos: recall X% (Wilson A-B), false alarms Y per 100 pushes, precision Z%; exploit-verified N of M."
+
+## Verification driver (verify.py) details
+
+- Model: DeepSeek via the shipped provider, off-cluster (cheap, ~$0.0002/call). The brain tier runs
+  locally in research; it does not need the kube engine.
+- Prompt inputs per case: the advisory text, the pre-fix function, the post-fix function, and the file
+  context around the function (for the entry-point question). No repo-wide context in the first version.
+- Output is the verification record above. Reachability uses the same entry-point and source concepts the
+  taint engine already models (`src/openultrasast/ruleset/frameworks.py`, framework routes/handlers), so
+  the LLM is asked a concrete question, not an open one.
+- Instrument rule: the driver prints bytes read for each side's function before it calls the model, and
+  fails loudly on an empty read. A verdict on an unread case is rejected. (AGENTS.md instrument rule.)
+
+## First increment (pilot)
+
+1. Source Python and JS advisories with a clean fix link, held out from the tool's training/mining sets.
+2. `catalog.py` builds candidate cases by fix site. Target ~25 candidates to land ~12 verified.
+3. `verify.py` runs the LLM check; human spot-checks all of them (dozens, so full review is cheap).
+4. `draw_ordinary()` draws >=20x negatives from the same repos.
+5. `assemble.py` freezes the manifest + digest + held-out guard.
+6. `replay.py` on the Docker lane runs the whole tool.
+7. `score.py` + `metrics.py` produce the four metrics with intervals and the headline.
+
+The pilot proves the pipeline and gives the first real numbers. Then grow positives to ~40 (set exactly
+by `required_n`) and, where the proof arm has run, fill in the exploit-verified rate.
+
+## Risks
+- Harvesting is rate-bound (GitHub 5k/hr) and infra is operator-owned; the sourcing step runs
+  deliberately, not in a tight loop, and reuses the banked resolver.
+- The LLM verifier can be wrong; the human spot-check is the backstop, and the proof arm is the
+  label-free check above it.
+- App-shaped is a judgement call; record the entry point the verifier names so a human can audit it.
@@ -0,0 +1,79 @@
+# Requirements: valid detection benchmark
+
+Turns `brief.md` into testable requirements. The brief supersedes the retired SZZ pool
+(`unseen-repo-evaluation`). This spec builds a benchmark with sound ground truth and standard
+security metrics.
+
+## Scope decisions (resolving the brief's open questions)
+
+- **First-slice size.** Build a pilot of 10 to 12 verified positives end to end first, to prove the
+  pipeline, then grow to a target of about 40 verified positives. 40 gives a recall interval of roughly
+  ±0.10 at a true recall near 0.85 to 0.90; `score.py:required_n(p, width)` computes the exact number,
+  so the target is set from that, not guessed.
+- **Negative-to-positive ratio.** At least 20 ordinary commits per positive, drawn from the same repos.
+  Negatives are free to collect, and false alarms decide adoption, so err large. Report false alarms as
+  a rate per 100 pushes.
+- **Languages first.** Python and JavaScript/TypeScript web applications (Flask, Django, FastAPI,
+  Express, Koa). These are where the tool is strongest and where in-repo entry points are clear. PHP and
+  others come later, named explicitly as not yet covered.
+- **Curation record.** Each case stores the advisory, repo, fix commit, parent, path, function, the
+  pre-fix and post-fix excerpts, the LLM verdict with confidence, and the human spot-check decision with
+  reviewer and date. The set is frozen with a digest, the same way `advisory-fixes/` freezes its pairs.
+- **Lane.** Curate and score locally on the Docker lane (research is local execution). The kube `ax`
+  lane stays the scale target but is not required here and is still blocked on the gVisor checkpoint path.
+
+## Requirements
+
+### R1. Positives labelled from the fix, never from SZZ
+- Take the vulnerability location from the security fix: the file and function the fix changed, read at
+  the parent commit (pre-fix).
+- Do not run SZZ blame walk-back. The sourcing must not use `extract.py`'s `introducing` or `last_touch`
+  methods. Use the fix site directly (`fix_sites()`) and first-parent history only.
+- Each positive is a pair: the vulnerable function at `parent`, and the same function at the fix commit.
+
+### R2. App-shaped and reachable in-repo
+- Keep a positive only if the vulnerability is reachable from an entry point inside the repo: a web
+  route, a request handler, or a CLI argument.
+- Drop library-internal cases whose trigger depends on a caller outside the repo. A pre-push tool cannot
+  see that trust boundary, so scoring it there is unfair.
+
+### R3. Every positive is verified
+- Verify each case with an LLM adjudication call and a human spot-check. The call must confirm:
+  (a) the pre-fix code is vulnerable, (b) the fix addresses it, (c) it is reachable from an in-repo entry
+  point.
+- Record the verdict, the confidence, and the human decision. A case that fails any of (a), (b), (c), or
+  the human spot-check does not enter the benchmark.
+
+### R4. Negatives are ordinary commits from the same repos
+- Draw non-fix commits from the first-parent history of the same application repos, excluding the fix.
+- The negative set is at least 20x the positive set.
+
+### R5. Held out
+- No repository in the benchmark was used to build, train, tune, mine, or harvest the tool. Enforce this
+  with the existing used-set canonicalization (`eligibility.normalize`), the same guard the SZZ pool used.
+
+### R6. Measure the whole tool on unseen repos
+- Run the tool as a developer runs it at push time: quick rules on, engine on, LLM-guided arm on.
+- Use the existing replay harness so the measured path is the shipped push hook.
+
+### R7. Standard metrics
+- Report recall (true-positive rate), precision (positive predictive value), false-positive rate as
+  false alarms per 100 pushes, and exploit-verified rate (the share of flagged positives the proof arm
+  demonstrates with a working proof of concept).
+- Report recall with a Wilson 95% interval and a repo-clustered interval, reusing `score.py`.
+- The headline pairs recall and false alarms per 100 pushes in one plain sentence.
+
+### R8. Reproducible and frozen
+- The positive catalog, the negative draw, and the held-out guarantee are frozen with digests so a run is
+  reproducible. Re-running the freeze on an unchanged source yields the same digest.
+
+## Non-goals
+- No SZZ, no frozen-pool catch/false-alarm numbers (retired).
+- No new detection capability in this spec. This measures the tool that exists.
+- Not required to run on kube `ax` to be valid; the Docker lane is enough for the first numbers.
+
+## Acceptance
+- A frozen pilot of at least 10 verified positives plus their negatives, on held-out Python/JS repos,
+  run through the whole tool, producing recall, precision, false alarms per 100 pushes, and
+  exploit-verified rate, with intervals. Every positive has an LLM verdict and a human spot-check on
+  record.
@@ -0,0 +1,55 @@
+# Tasks: valid detection benchmark
+
+Order is dependency order. Each task is small enough to review on its own. New code under
+`benchmarks/valid/`. Implementation goes through Codex, gated here.
+
+## Group 1 - positive catalog (fix-site, no SZZ)
+- [x] 1.1 `benchmarks/valid/catalog.py`: given sourced advisories (reuse `source.py`/`draft.py` output),
+  build candidate positive cases by fix site. Wrap `extract.fix_sites()` + `pairs/materialize.py`; read
+  the pre-fix function at `parent` and the post-fix function at `commit`. Must not call `introducing` or
+  `last_touch`. Emit the `[[case]]` catalog in the design schema. (R1)
+- [x] 1.2 Filter to Python and JS/TS only; attach `language` and `family`/`cwe`. (scope decision)
+- [x] 1.3 Offline test: a fixture commit yields one case with the correct pre/post functions and
+  `method = "fix_site"`; assert SZZ methods are never invoked.
+
+## Group 2 - LLM verification driver
+- [x] 2.1 `benchmarks/valid/verify.py`: one DeepSeek call per case via the shipped provider (off-cluster).
+  Confirm vulnerable-pre-fix, fix-addresses-it, reachable-in-repo; name the entry point; return
+  confidence. Print bytes read per function before the call; reject a verdict on an empty read. (R3, instrument rule)
+- [x] 2.2 Emit the verification record (design schema) to `verification.json`; `verdict = "accepted"`
+  only when all three checks pass. Reachability here is the app-shaped filter. (R2, R3)
+- [x] 2.3 Human spot-check hook: a small CLI to record accept/reject + reviewer + date per case into the
+  same record. (R3)
+- [x] 2.4 Offline test with a stub model: a case with all three true and a human accept becomes
+  `accepted`; any false, or a human reject, does not.
+
+## Group 3 - negatives and assembly
+- [x] 3.1 `benchmarks/valid/assemble.py`: draw negatives with `extract.draw_ordinary()` from the accepted
+  positives' repos, at least 20x, excluding the fix. (R4)
+- [x] 3.2 Freeze accepted positives + negatives into a `pool.toml`-shaped manifest + `freeze-valid.json`
+  digest; run the held-out guard via `eligibility.normalize`. (R5, R8)
+- [x] 3.3 Offline test: freeze is deterministic (same source -> same digest); held-out guard drops any
+  repo in the used set.
+
+## Group 4 - metrics
+- [x] 4.1 `benchmarks/valid/metrics.py`: precision (PPV) with Wilson 95% over positives+negatives. (R7)
+- [x] 4.2 False alarms per 100 pushes as the presentation of `score.py`'s false-alarm rate. (R7)
+- [x] 4.3 Exploit-verified rate: join flagged positives with a search-with-proof run by case `id`;
+  report "not yet proved" where absent. (R7)
+- [x] 4.4 Build the headline string (recall + FP/100 + precision + exploit-verified). (R7)
+- [x] 4.5 Offline test on a tiny hand-made record set: all four metrics and intervals match hand calcs.
+
+## Group 5 - pilot run (research, local)
+- [ ] 5.1 Source Python/JS advisories; run `catalog.py` to ~25 candidates. (deliberate, rate-aware)
+- [ ] 5.2 Run `verify.py` + human spot-check to land ~12 accepted positives.
+- [ ] 5.3 `assemble.py` freezes the pilot manifest + negatives (>=20x) + held-out guard.
+- [ ] 5.4 `replay.py` on the Docker lane, whole tool on (quick + engine + LLM arm).
+- [ ] 5.5 `score.py` + `metrics.py`: produce recall, precision, FP/100, exploit-verified, intervals, and
+  the headline. Freeze the run under `benchmarks/measurements/`.
+- [ ] 5.6 Write up the pilot: the four numbers, what they mean, and the target set size from `required_n`
+  for the full ~40-positive benchmark.
+
+## Out of this spec
+- Growing beyond the pilot to the full set (follow-up, once the pipeline is proven).
+- Running on kube `ax` (blocked on gVisor; Docker lane is enough here).
+- Any new detection capability.