Open_Ultra_SAST/.kiro/specs/valid-detection-benchmark/design.md

Design: valid detection benchmark

Builds the benchmark in R1-R8. New code lives under benchmarks/valid/. It reuses the sourcing, labelling, negative-drawing, replay, and scoring that already exist, and drops the SZZ layer.

What is reused and what is new

Reused as-is:

• Sourcing advisories to repos and fix commits: benchmarks/unseen/source.py, draft.py, eligibility.py (the used-set held-out guard).
• Fix-site labelling: benchmarks/unseen/extract.py:fix_sites() and the pair builder in benchmarks/pairs/advisory-fixes/ (materialize.py, catalog.toml, verification.json).
• Function cutting: openultrasast.learn.excerpt.function_bounds.
• Ordinary negatives: extract.py:ordinary_history() and draw_ordinary().
• Replay: benchmarks/unseen/replay.py and replay_entry.py (whole tool via the push hook).
• Scoring base: benchmarks/unseen/score.py (catch, false alarm, Wilson, repo-clustered interval, required_n, McNemar gate).

New:

1. benchmarks/valid/catalog.py - sourcing to a fix-site positive catalog, SZZ paths excluded.
2. benchmarks/valid/verify.py - the committed LLM verification driver (replaces the scratchpad label_audit.py). One call per case confirms vulnerable / fix-addresses / reachable-in-repo. This is also the app-shaped filter: reachability is answered here, not by a separate classifier.
3. benchmarks/valid/metrics.py - precision, false alarms per 100 pushes, and exploit-verified rate, layered on score.py output.
4. benchmarks/valid/assemble.py - freezes the verified positives plus the negative draw into a replay manifest with a digest and the held-out guard.

Data flow

advisories (OSV/GHSA)
  -> source.py / draft.py            resolve repo + fix commit + parent   [reuse]
  -> catalog.py                      fix-site label (no SZZ); emit candidate cases  [new, wraps extract.fix_sites + pairs materialize]
  -> verify.py                       LLM: vulnerable? fix addresses it? reachable in-repo? + confidence  [new]
  -> human spot-check                record decision, reviewer, date       [manual, recorded in verification.json]
  -> draw_ordinary()                 negatives from the same repos          [reuse]
  -> assemble.py                     freeze positives+negatives -> manifest + digest, held-out guard  [new, wraps eligibility.normalize]
  -> replay.py (docker lane)         run the whole tool at push time        [reuse]
  -> score.py + metrics.py           recall, precision, FP/100, exploit-verified, intervals  [reuse + new]

Schemas

Positive case (catalog.py output, extends advisory-fixes/catalog.toml)

[[case]]
id            = "owner-repo-cve"
repo          = "owner/repo"
url           = "https://github.com/owner/repo"
advisory      = "GHSA-xxxx / CVE-xxxx"
cwe           = "CWE-89"
family        = "sql_injection"
language      = "python"
commit        = "<fix sha>"      # post-fix (safe) side
parent        = "<parent sha>"   # pre-fix (vulnerable) side
path          = "app/db.py"
function       = "get_member_by_code"
function_lines = [[40, 58]]
method        = "fix_site"       # never introducing/last_touch

Verification record (verify.py output, extends advisory-fixes/verification.json)

{ "id": "...", "vulnerable_pre_fix": true, "fix_addresses_it": true,
  "reachable_in_repo": true, "entry_point": "flask route /member/<code>",
  "confidence": 0.86, "model": "deepseek-...", "advisory_cited": true,
  "human_spot_check": {"decision": "accept", "reviewer": "<initials>", "date": "2026-10-13"},
  "verdict": "accepted" }   # accepted only if all three true AND human accept

Replay manifest (assemble.py output)

A pool.toml-shaped file so replay.py consumes it unchanged: [[repository]] rows with changes, each change kind = "vulnerability" (positives, label from fix) or kind = "ordinary" (negatives), with base/head/files/function. A freeze-valid.json carries the freeze digest and the held-out guard result, matching the SZZ pool's freeze format so the ledger and replay flow are identical.

Metrics (metrics.py)

Input is the scored records from score.py. New outputs:

• precision = flagged-and-real / all-flagged, over positives and negatives together, with Wilson 95%.
• false alarms per 100 pushes = false-positive rate x 100, the presentation of score.py's false_alarm rate. This is a presentation layer, not a new measurement.
• exploit-verified rate = of flagged positives, the share the search-with-proof arm (src/openultrasast/search/) demonstrates on the vulnerable side and not the fixed side. Joined by case id from a proof-arm run; absent where the arm has not run (reported as "not yet proved", not as a failure).

Headline string, built here:

"On unseen app repos: recall X% (Wilson A-B), false alarms Y per 100 pushes, precision Z%; exploit-verified N of M."

Verification driver (verify.py) details

• Model: DeepSeek via the shipped provider, off-cluster (cheap, ~$0.0002/call). The brain tier runs locally in research; it does not need the kube engine.
• Prompt inputs per case: the advisory text, the pre-fix function, the post-fix function, and the file context around the function (for the entry-point question). No repo-wide context in the first version.
• Output is the verification record above. Reachability uses the same entry-point and source concepts the taint engine already models (src/openultrasast/ruleset/frameworks.py, framework routes/handlers), so the LLM is asked a concrete question, not an open one.
• Instrument rule: the driver prints bytes read for each side's function before it calls the model, and fails loudly on an empty read. A verdict on an unread case is rejected. (AGENTS.md instrument rule.)

First increment (pilot)

1. Source Python and JS advisories with a clean fix link, held out from the tool's training/mining sets.
2. catalog.py builds candidate cases by fix site. Target ~25 candidates to land ~12 verified.
3. verify.py runs the LLM check; human spot-checks all of them (dozens, so full review is cheap).
4. draw_ordinary() draws >=20x negatives from the same repos.
5. assemble.py freezes the manifest + digest + held-out guard.
6. replay.py on the Docker lane runs the whole tool.
7. score.py + metrics.py produce the four metrics with intervals and the headline.

The pilot proves the pipeline and gives the first real numbers. Then grow positives to ~40 (set exactly by required_n) and, where the proof arm has run, fill in the exploit-verified rate.

Risks

• Harvesting is rate-bound (GitHub 5k/hr) and infra is operator-owned; the sourcing step runs deliberately, not in a tight loop, and reuses the banked resolver.
• The LLM verifier can be wrong; the human spot-check is the backstop, and the proof arm is the label-free check above it.
• App-shaped is a judgement call; record the entry point the verifier names so a human can audit it.