Open_Ultra_SAST/docs/evaluation.md

Evaluation

What has been measured on code the tool was not tuned on, and how. State as of 2026-10-02. Every number below is quoted from a committed record; the path is given with it.

Instruments

Instrument	Measures	Where
Detection gate	per-language recall/FP on the bundled cheat-sheet corpora (goal: >= 90% recall, < 10% FP)	python -m openultrasast.gate, benchmarks/manifests/
Pair corpus	fire on the vulnerable function, stay silent on its fix	ousast pairs, benchmarks/pairs/
Independent populations	real repositories at vulnerable, fixed and benign pins, scored under a pre-registered protocol	benchmarks/independent/

The detection gate and the local pairs are CI gates. They catch regressions; they do not estimate recall on real code, because the rules were written against them.

Pair corpora

benchmarks/pairs/catalog.toml and the slice directories (local, github, sast, vfc, vibe-py, vfc-js, agent-vfc, owasp) are runnable with ousast pairs --slice <name>; each pair carries a provenance profile and a train/holdout split (--profile, --split). Pointer pairs are not vendored: --pointers fetches them into ~/.cache/openultrasast/pairs/. benchmarks/pairs/README.md documents the schema and the review tiers.

advisory-fixes (benchmarks/pairs/advisory-fixes/, 2026-10-01) holds 97 pointer pairs, one per repository: a public CVE/GHSA fix commit whose advisory-named function is declared on both sides and changed by the fix. It was assembled to give the decision engine real negatives for families the corpus had few of (access control, deserialization, output encoding, path, untrusted destination). It is a label source of the decision engine (src/openultrasast/learn/sources.toml), not an ousast pairs slice; the research log and every rejection are in research-2026-10-01.md.

Independent populations and the M4 gates

The pre-push safety net's qualification milestone (M4) requires, together: at least 95% actionable precision, at least 90% supported recall, at least 95% supported-check completion, the runtime gates, and zero fixed-side or benign false alerts. A population is frozen before any of its cases is scanned, scored once under a pre-registered protocol, and then spent: it may inform changes, but it can no longer qualify anything.

Population	Status	Result	Record
v1 (11 cases)	spent	engine: recall 1/11, precision 1/12, 4 fixed-side and 1 benign false alert; every M4 gate failed. Eight of eleven cases completed no question at the vulnerable pin.	benchmarks/independent/results-v1.json
v2 (17 cases)	spent	engine: recall 0/17, 3 benign alerts, completion 13,206 of 108,374. Tool hunter: recall 1/17, precision 2/14, 1 fixed-side and 16 benign alerts, $8.88. No M4 gate met by either.	results-v2.json, results-v2-hunter.json
v2, model-driven pipeline	exploratory (v2 already spent)	stopped after 7 of 17 cases when the account emptied: recall 4/7; precision not adjudicated; no gate measurable	results-v2-model-pipeline.json
v3 (PHP, 15 cases; one SSRF slot left empty by the selection rule)	frozen, untouched	none: it is the one-time final check, run once under protocol-v3.md after a prediction is committed	population-v3-php.toml, freeze-v3-php.json

v3 is reserved: development and analysis work never reads its cases, and tests/test_independent_population.py fails when anything in the development tree (a recipe, catalog, manifest or measurement) names one of its repositories.

stateDiagram-v2
    [*] --> Frozen: cases selected and frozen before any scan
    Frozen --> Scored: scored once under the pre-registered protocol
    Scored --> Spent: result recorded
    Spent --> [*]: may inform changes, can no longer qualify anything
    note right of Frozen
        v3 (PHP, 15 cases) waits here as the one-time final check,
        run once after a prediction is committed
    end note
    note right of Spent
        v1 and v2 are spent
    end note

Folds: how learned decisions are kept out of their own training data

The decision engine is evaluated on repositories it was not shown (src/openultrasast/learn/folds.py). Every split is by repository group (the label builder's normalised owner/name, merged across URLs, forks and shared advisories), across all corpora, seeded and deterministic.

flowchart TD
    groups["All repository groups with labels"] --> c["Compile split: 25% of groups, never evaluated"]
    groups --> rest["The other 75%"]
    c --> boot["C_boot 60%: demonstrations"]
    c --> val["C_val 40%: instruction scoring"]
    rest --> outer["Outer grouped 5-fold: calibration cross-fitted, operating points nested"]
    rest --> loso["Leave-one-source-out: a source held out whole"]
    rest --> lofo["Leave-one-framework-out: frameworks with at least 10 groups"]

assert_disjoint runs before a compile: the compile split and the evaluation folds share no group. A framework with fewer than 10 groups gets insufficient data instead of a fold. As of 2026-10-02 the outer folds have been run for the six evaluable families; leave-one-source-out and leave-one-framework-out have not (decision-engine.md).

The plane increment

The first increment of the agentic plane ran the model-driven pipeline's validation set (46 candidates in 15 cases of v2, 20 declared vulnerable sites) on google/ax and compared it with the script pipeline on the same cases, candidates, pins and recorded triage. The pre-registered gate: at least 16 of 20 declared sites agreed, at a cost per candidate under $0.022.

• Increment 1 (2026-09-29, benchmarks/independent/plane-increment-1.json): 14 of 20 declared sites agreed against the reference's 16; $0.0194 per candidate with recorded triage. NOT MET on recall. Correction (2026-09-30, in the record): the first version read agree's declared_sites_matched (coverage by candidates, 19) as "found in either pass" and concluded detection was unchanged. Recounted from the agreed rows, the plane's passes flagged 17 of 20 declared sites in at least one pass against the reference's 19.
• Increment 2 (2026-09-30, benchmarks/independent/plane-increment-2.json): after the design revisit, a tie-break pass c on the 10 disputed candidates with 2-of-3 agreement. 16 of 20 declared sites agreed (reference 16) at $0.0205 per candidate over 46 ($0.02195 over the 43 after triage, against the reference's $0.022). MET, with two caveats the record carries: the tie-break was chosen after seeing the first measurement on the same validation set, so this compares plane with script and qualifies nothing on an independent population; and the plane's passes still flag fewer declared sites than the reference (17 vs 19). Precision is not part of this gate and remains the open problem.

The decision engine

Two slices of the learned decision engine have been measured out of repository: injection alone (2026-10-01) and the six evaluable families (2026-10-02: pooled AUC 0.77 to 0.96 per family, within-pair AUC 0.72 to 0.93, BLOCK offered for none). The numbers and their records are in decision-engine.md. The engine is not adopted.

Removal of the earlier agentic extra

The removal of the earlier agentic extra (2026-09-30) was checked for byte-identical deterministic outputs (pair replay, quick benchmark, standard scan, improve --dry-run): benchmarks/measurements/2026-09-30-harnessx-removal-baseline/ and -equality/.