Open_Ultra_SAST/.kiro/specs/valid-detection-benchmark/requirements.md

# Requirements: valid detection benchmark

Turns `brief.md` into testable requirements. The brief supersedes the retired SZZ pool
(`unseen-repo-evaluation`). This spec builds a benchmark with sound ground truth and standard
security metrics.

## Scope decisions (resolving the brief's open questions)

- **First-slice size.** Build a pilot of 10 to 12 verified positives end to end first, to prove the
  pipeline, then grow to a target of about 40 verified positives. 40 gives a recall interval of roughly
  ±0.10 at a true recall near 0.85 to 0.90; `score.py:required_n(p, width)` computes the exact number,
  so the target is set from that, not guessed.
- **Negative-to-positive ratio.** At least 20 ordinary commits per positive, drawn from the same repos.
  Negatives are free to collect, and false alarms decide adoption, so err large. Report false alarms as
  a rate per 100 pushes.
- **Languages first.** Python and JavaScript/TypeScript web applications (Flask, Django, FastAPI,
  Express, Koa). These are where the tool is strongest and where in-repo entry points are clear. PHP and
  others come later, named explicitly as not yet covered.
- **Curation record.** Each case stores the advisory, repo, fix commit, parent, path, function, the
  pre-fix and post-fix excerpts, the LLM verdict with confidence, and the human spot-check decision with
  reviewer and date. The set is frozen with a digest, the same way `advisory-fixes/` freezes its pairs.
- **Lane.** Curate and score locally on the Docker lane (research is local execution). The kube `ax`
  lane stays the scale target but is not required here and is still blocked on the gVisor checkpoint path.

## Requirements

### R1. Positives labelled from the fix, never from SZZ
- Take the vulnerability location from the security fix: the file and function the fix changed, read at
  the parent commit (pre-fix).
- Do not run SZZ blame walk-back. The sourcing must not use `extract.py`'s `introducing` or `last_touch`
  methods. Use the fix site directly (`fix_sites()`) and first-parent history only.
- Each positive is a pair: the vulnerable function at `parent`, and the same function at the fix commit.

### R2. App-shaped and reachable in-repo
- Keep a positive only if the vulnerability is reachable from an entry point inside the repo: a web
  route, a request handler, or a CLI argument.
- Drop library-internal cases whose trigger depends on a caller outside the repo. A pre-push tool cannot
  see that trust boundary, so scoring it there is unfair.

### R3. Every positive is verified
- Verify each case with an LLM adjudication call and a human spot-check. The call must confirm:
  (a) the pre-fix code is vulnerable, (b) the fix addresses it, (c) it is reachable from an in-repo entry
  point.
- Record the verdict, the confidence, and the human decision. A case that fails any of (a), (b), (c), or
  the human spot-check does not enter the benchmark.

### R4. Negatives are ordinary commits from the same repos
- Draw non-fix commits from the first-parent history of the same application repos, excluding the fix.
- The negative set is at least 20x the positive set.

### R5. Held out
- No repository in the benchmark was used to build, train, tune, mine, or harvest the tool. Enforce this
  with the existing used-set canonicalization (`eligibility.normalize`), the same guard the SZZ pool used.

### R6. Measure the push path on unseen repos, with the LLM tier
- The tool a developer runs at push time must include the LLM detection tier (the hunter). That is a
  product requirement on the pre-push hook, owned by `pre-push-safety-net`. The benchmark measures the
  push path, not a scan mode.
- Today the shipped hook runs quick rules and the engine only. Its model path (`push/assistance.py`)
  selects witnesses for already-admitted defects and cannot add a finding. Until the hunter is wired into
  the hook, the benchmark reports two arms over the same changes and labels them:
  - (a) the push hook as shipped (quick rules + engine), through the replay harness;
  - (b) the push hook plus the hunter arm, the hunter run off-cluster over the same changes and the
    records combined per change.
  (b) is the number the tool is held to. (a) is what ships today. The gap between them is product work,
  not a benchmark artefact, and is reported as such.
- Use the existing replay harness for (a) so the measured path is the shipped push hook.

### R7. Standard metrics
- Report recall (true-positive rate), precision (positive predictive value), false-positive rate as
  false alarms per 100 pushes, and exploit-verified rate (the share of flagged positives the proof arm
  demonstrates with a working proof of concept).
- Report recall with a Wilson 95% interval and a repo-clustered interval, reusing `score.py`.
- The headline pairs recall and false alarms per 100 pushes in one plain sentence.

### R8. Reproducible and frozen
- The positive catalog, the negative draw, and the held-out guarantee are frozen with digests so a run is
  reproducible. Re-running the freeze on an unchanged source yields the same digest.

## Non-goals
- No SZZ, no frozen-pool catch/false-alarm numbers (retired).
- No new detection capability in this spec. This measures the tool that exists.
- Not required to run on kube `ax` to be valid; the Docker lane is enough for the first numbers.

## Acceptance
- A frozen pilot of at least 10 verified positives plus their negatives, on held-out Python/JS repos,
  run through the whole tool, producing recall, precision, false alarms per 100 pushes, and
  exploit-verified rate, with intervals. Every positive has an LLM verdict and a human spot-check on
  record.