marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-18 04:29:27 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(benchmark): valid detection benchmark spec + offline modules (groups 1-4)

Browse Source 
Builds the benchmark from .kiro/specs/valid-detection-benchmark/brief.md:
sound labels from the fix (no SZZ), app-shaped verified positives,
ordinary negatives from the same repos, whole tool on unseen repos,
standard metrics.

Spec: requirements.md, design.md, tasks.md. Reuses the existing sourcing,
fix-site labelling, negative drawing, replay, and Wilson scoring; drops
the SZZ layer.

New offline modules under benchmarks/valid/:
- catalog.py  positives labelled at the fix site (never introducing/last_touch)
- verify.py   one LLM call per case (vulnerable? fix addresses it? reachable
              in-repo?) + human spot-check; this is the app-shaped filter.
              Honors the instrument rule: bytes read per function, reject on
              an empty read.
- assemble.py >=20x ordinary negatives + a frozen, deterministic manifest with
              the held-out guard
- metrics.py  precision (PPV), false alarms per 100 pushes, exploit-verified
              rate, and the plain headline, layered on score.py

36 new offline tests, 495 regression tests green. No network, no harvest,
no model calls in tests. Group 5 (the live pilot) is not started.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-10-12 18:55:42 +02:00
co-authored by Claude Opus 4.8
parent 3855945311
commit 43ef254134
9 changed files with 1296 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/specs/valid-detection-benchmark/design.md
+122
View File
@@ -0,0 +1,122 @@
# Design: valid detection benchmark
Builds the benchmark in R1-R8. New code lives under `benchmarks/valid/`. It reuses the sourcing,
labelling, negative-drawing, replay, and scoring that already exist, and drops the SZZ layer.
## What is reused and what is new
Reused as-is:
- Sourcing advisories to repos and fix commits: `benchmarks/unseen/source.py`, `draft.py`,
`eligibility.py` (the used-set held-out guard).
- Fix-site labelling: `benchmarks/unseen/extract.py:fix_sites()` and the pair builder in
`benchmarks/pairs/advisory-fixes/` (`materialize.py`, `catalog.toml`, `verification.json`).
- Function cutting: `openultrasast.learn.excerpt.function_bounds`.
- Ordinary negatives: `extract.py:ordinary_history()` and `draw_ordinary()`.
- Replay: `benchmarks/unseen/replay.py` and `replay_entry.py` (whole tool via the push hook).
- Scoring base: `benchmarks/unseen/score.py` (catch, false alarm, Wilson, repo-clustered interval,
required_n, McNemar gate).
New:
1. `benchmarks/valid/catalog.py` - sourcing to a fix-site positive catalog, SZZ paths excluded.
2. `benchmarks/valid/verify.py` - the committed LLM verification driver (replaces the scratchpad
`label_audit.py`). One call per case confirms vulnerable / fix-addresses / reachable-in-repo. This is
also the app-shaped filter: reachability is answered here, not by a separate classifier.
3. `benchmarks/valid/metrics.py` - precision, false alarms per 100 pushes, and exploit-verified rate,
layered on `score.py` output.
4. `benchmarks/valid/assemble.py` - freezes the verified positives plus the negative draw into a
replay manifest with a digest and the held-out guard.
## Data flow
```
advisories (OSV/GHSA)
-> source.py / draft.py resolve repo + fix commit + parent [reuse]
-> catalog.py fix-site label (no SZZ); emit candidate cases [new, wraps extract.fix_sites + pairs materialize]
-> verify.py LLM: vulnerable? fix addresses it? reachable in-repo? + confidence [new]
-> human spot-check record decision, reviewer, date [manual, recorded in verification.json]
-> draw_ordinary() negatives from the same repos [reuse]
-> assemble.py freeze positives+negatives -> manifest + digest, held-out guard [new, wraps eligibility.normalize]
-> replay.py (docker lane) run the whole tool at push time [reuse]
-> score.py + metrics.py recall, precision, FP/100, exploit-verified, intervals [reuse + new]
```
## Schemas
### Positive case (catalog.py output, extends `advisory-fixes/catalog.toml`)
```
[[case]]
id = "owner-repo-cve"
repo = "owner/repo"
url = "https://github.com/owner/repo"
advisory = "GHSA-xxxx / CVE-xxxx"
cwe = "CWE-89"
family = "sql_injection"
language = "python"
commit = "<fix sha>" # post-fix (safe) side
parent = "<parent sha>" # pre-fix (vulnerable) side
path = "app/db.py"
function = "get_member_by_code"
function_lines = [[40, 58]]
method = "fix_site" # never introducing/last_touch
```
### Verification record (verify.py output, extends `advisory-fixes/verification.json`)
```
{ "id": "...", "vulnerable_pre_fix": true, "fix_addresses_it": true,
"reachable_in_repo": true, "entry_point": "flask route /member/<code>",
"confidence": 0.86, "model": "deepseek-...", "advisory_cited": true,
"human_spot_check": {"decision": "accept", "reviewer": "<initials>", "date": "2026-10-13"},
"verdict": "accepted" } # accepted only if all three true AND human accept
```
### Replay manifest (assemble.py output)
A `pool.toml`-shaped file so `replay.py` consumes it unchanged: `[[repository]]` rows with `changes`,
each change `kind = "vulnerability"` (positives, label from fix) or `kind = "ordinary"` (negatives),
with `base`/`head`/`files`/`function`. A `freeze-valid.json` carries the freeze digest and the held-out
guard result, matching the SZZ pool's freeze format so the ledger and replay flow are identical.
## Metrics (metrics.py)
Input is the scored records from `score.py`. New outputs:
- **precision** = flagged-and-real / all-flagged, over positives and negatives together, with Wilson 95%.
- **false alarms per 100 pushes** = false-positive rate x 100, the presentation of `score.py`'s
`false_alarm` rate. This is a presentation layer, not a new measurement.
- **exploit-verified rate** = of flagged positives, the share the search-with-proof arm
(`src/openultrasast/search/`) demonstrates on the vulnerable side and not the fixed side. Joined by
case `id` from a proof-arm run; absent where the arm has not run (reported as "not yet proved", not as
a failure).
Headline string, built here:
> "On unseen app repos: recall X% (Wilson A-B), false alarms Y per 100 pushes, precision Z%; exploit-verified N of M."
## Verification driver (verify.py) details
- Model: DeepSeek via the shipped provider, off-cluster (cheap, ~$0.0002/call). The brain tier runs
locally in research; it does not need the kube engine.
- Prompt inputs per case: the advisory text, the pre-fix function, the post-fix function, and the file
context around the function (for the entry-point question). No repo-wide context in the first version.
- Output is the verification record above. Reachability uses the same entry-point and source concepts the
taint engine already models (`src/openultrasast/ruleset/frameworks.py`, framework routes/handlers), so
the LLM is asked a concrete question, not an open one.
- Instrument rule: the driver prints bytes read for each side's function before it calls the model, and
fails loudly on an empty read. A verdict on an unread case is rejected. (AGENTS.md instrument rule.)
## First increment (pilot)
1. Source Python and JS advisories with a clean fix link, held out from the tool's training/mining sets.
2. `catalog.py` builds candidate cases by fix site. Target ~25 candidates to land ~12 verified.
3. `verify.py` runs the LLM check; human spot-checks all of them (dozens, so full review is cheap).
4. `draw_ordinary()` draws >=20x negatives from the same repos.
5. `assemble.py` freezes the manifest + digest + held-out guard.
6. `replay.py` on the Docker lane runs the whole tool.
7. `score.py` + `metrics.py` produce the four metrics with intervals and the headline.
The pilot proves the pipeline and gives the first real numbers. Then grow positives to ~40 (set exactly
by `required_n`) and, where the proof arm has run, fill in the exploit-verified rate.
## Risks
- Harvesting is rate-bound (GitHub 5k/hr) and infra is operator-owned; the sourcing step runs
deliberately, not in a tight loop, and reuses the banked resolver.
- The LLM verifier can be wrong; the human spot-check is the backstop, and the proof arm is the
label-free check above it.
- App-shaped is a judgement call; record the entry point the verifier names so a human can audit it.