marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: plain-language pass -- detection techniques, architecture, evaluation, decision engine

Browse Source 
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-10-02 21:03:24 +02:00
co-authored by Claude Opus 5.5
parent 50f4e69321
commit 352f8c07aa
4 changed files with 845 additions and 534 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/evaluation.md
+75 -52
View File
@@ -1,7 +1,8 @@
# Evaluation
What has been measured on code the tool was not tuned on, and how. State as of 2026-10-02.
Every number below is quoted from a committed record; the path is given with it.
This page shows what has been measured on code the tool was not tuned on, and how. State as of
2026-10-02. Every number below is quoted from a committed record, and the record's path is
given with it.
## Instruments
@@ -11,32 +12,47 @@ Every number below is quoted from a committed record; the path is given with it.
| Pair corpus | fire on the vulnerable function, stay silent on its fix | `ousast pairs`, `benchmarks/pairs/` |
| Independent populations | real repositories at vulnerable, fixed and benign pins, scored under a pre-registered protocol | `benchmarks/independent/` |
The detection gate and the local pairs are CI gates. They catch regressions; they do not
Recall means the share of known bugs found. FP means false positives.
The detection gate and the local pairs are CI gates. They catch regressions. They do not
estimate recall on real code, because the rules were written against them.
## Pair corpora
`benchmarks/pairs/catalog.toml` and the slice directories (`local`, `github`, `sast`, `vfc`,
`vibe-py`, `vfc-js`, `agent-vfc`, `owasp`) are runnable with `ousast pairs --slice <name>`; each
pair carries a provenance profile and a train/holdout split (`--profile`, `--split`). Pointer
pairs are not vendored: `--pointers` fetches them into `~/.cache/openultrasast/pairs/`.
`benchmarks/pairs/README.md` documents the schema and the review tiers.
A pair is a vulnerable function and its fixed version. You can run the catalog
`benchmarks/pairs/catalog.toml` and the slice directories with `ousast pairs --slice <name>`.
The slices are `local`, `github`, `sast`, `vfc`, `vibe-py`, `vfc-js`, `agent-vfc` and `owasp`.
- Each pair carries a provenance profile and a train/holdout split (`--profile`, `--split`).
- Pointer pairs are not vendored. `--pointers` fetches them into `~/.cache/openultrasast/pairs/`.
- `benchmarks/pairs/README.md` documents the schema and the review tiers.
**`advisory-fixes`** (`benchmarks/pairs/advisory-fixes/`, 2026-10-01) holds 97 pointer pairs,
one per repository: a public CVE/GHSA fix commit whose advisory-named function is declared on
both sides and changed by the fix. It was assembled to give the decision engine real negatives
for families the corpus had few of (access control, deserialization, output encoding, path,
untrusted destination). It is a label source of the decision engine
(`src/openultrasast/learn/sources.toml`), not an `ousast pairs` slice; the research log and
one per repository. Each is a public CVE/GHSA fix commit. The function the advisory names is
declared on both sides and changed by the fix.
It was assembled to give the decision engine real negatives (fixed, non-vulnerable code). It
targets families the corpus had few of: access control, deserialization, output encoding,
path, untrusted destination. It is a label source of the decision engine
(`src/openultrasast/learn/sources.toml`), not an `ousast pairs` slice. The research log and
every rejection are in `research-2026-10-01.md`.
## Independent populations and the M4 gates
The pre-push safety net's qualification milestone (M4) requires, together: at least 95%
actionable precision, at least 90% supported recall, at least 95% supported-check completion,
the runtime gates, and zero fixed-side or benign false alerts. A population is frozen before
any of its cases is scanned, scored once under a pre-registered protocol, and then **spent**:
it may inform changes, but it can no longer qualify anything.
M4 is the qualification milestone of the pre-push safety net. It requires all of these
together:
- at least 95% actionable precision;
- at least 90% supported recall;
- at least 95% supported-check completion;
- the runtime gates;
- zero fixed-side or benign false alerts.
A population is a set of real repositories. Its lifecycle has three steps:
1. It is frozen before any of its cases is scanned.
2. It is scored once under a pre-registered protocol.
3. It is then **spent**: it may inform changes, but it can no longer qualify anything.
| Population | Status | Result | Record |
| --- | --- | --- | --- |
@@ -45,9 +61,9 @@ it may inform changes, but it can no longer qualify anything.
| v2, model-driven pipeline | exploratory (v2 already spent) | stopped after 7 of 17 cases when the account emptied: recall 4/7; precision not adjudicated; no gate measurable | `results-v2-model-pipeline.json` |
| v3 (PHP, 15 cases; one SSRF slot left empty by the selection rule) | frozen, untouched | none: it is the one-time final check, run once under `protocol-v3.md` after a prediction is committed | `population-v3-php.toml`, `freeze-v3-php.json` |
v3 is reserved: development and analysis work never reads its cases, and
`tests/test_independent_population.py` fails when anything in the development tree (a recipe,
catalog, manifest or measurement) names one of its repositories.
v3 is reserved. Development and analysis work never reads its cases.
`tests/test_independent_population.py` fails when anything in the development tree names one
of its repositories. That covers a recipe, catalog, manifest or measurement.
```mermaid
stateDiagram-v2
@@ -66,9 +82,10 @@ stateDiagram-v2
## Folds: how learned decisions are kept out of their own training data
The decision engine is evaluated on repositories it was not shown (`src/openultrasast/learn/folds.py`).
Every split is by repository group (the label builder's normalised `owner/name`, merged across
URLs, forks and shared advisories), across all corpora, seeded and deterministic.
The decision engine is evaluated on repositories it was not shown
(`src/openultrasast/learn/folds.py`). Every split is by repository group. A group is the label
builder's normalised `owner/name`, merged across URLs, forks and shared advisories. Splits
cover all corpora and are seeded and deterministic.
```mermaid
flowchart TD
@@ -81,44 +98,50 @@ flowchart TD
rest --> lofo["Leave-one-framework-out: frameworks with at least 10 groups"]
```
`assert_disjoint` runs before a compile: the compile split and the evaluation folds share no
group. A framework with fewer than 10 groups gets `insufficient data` instead of a fold. As of
2026-10-02 the outer folds have been run for the six evaluable families; leave-one-source-out
and leave-one-framework-out have not
`assert_disjoint` runs before a compile. It checks that the compile split and the evaluation
folds share no group. A framework with fewer than 10 groups gets `insufficient data` instead of
a fold. As of 2026-10-02, the outer folds have been run for the six evaluable families.
Leave-one-source-out and leave-one-framework-out have not
([decision-engine.md](decision-engine.md#second-slice-six-families-2026-10-02)).
## The plane increment
The first increment of the agentic plane ran the model-driven pipeline's validation set (46
candidates in 15 cases of v2, 20 declared vulnerable sites) on google/ax and compared it with
the script pipeline on the same cases, candidates, pins and recorded triage. The pre-registered
gate: at least 16 of 20 declared sites agreed, at a cost per candidate under $0.022.
The first increment of the agentic plane ran on google/ax. It used the model-driven pipeline's
validation set: 46 candidates in 15 cases of v2, with 20 declared vulnerable sites. It was
compared with the script pipeline on the same cases, candidates, pins and recorded triage.
The pre-registered gate: at least 16 of 20 declared sites agreed, at a cost per candidate under
$0.022.
- **Increment 1** (2026-09-29, `benchmarks/independent/plane-increment-1.json`): 14 of 20
declared sites agreed against the reference's 16; $0.0194 per candidate with recorded triage.
**NOT MET** on recall.
*Correction (2026-09-30, in the record):* the first version read agree's
`declared_sites_matched` (coverage by candidates, 19) as "found in either pass" and concluded
detection was unchanged. Recounted from the agreed rows, the plane's passes flagged 17 of 20
declared sites in at least one pass against the reference's 19.
- **Increment 2** (2026-09-30, `benchmarks/independent/plane-increment-2.json`): after the design
revisit, a tie-break pass c on the 10 disputed candidates with 2-of-3 agreement. 16 of 20
declared sites agreed (reference 16) at $0.0205 per candidate over 46 ($0.02195 over the 43
after triage, against the reference's $0.022). **MET**, with two caveats the record carries:
the tie-break was chosen after seeing the first measurement on the same validation set, so
this compares plane with script and qualifies nothing on an independent population; and the
plane's passes still flag fewer declared sites than the reference (17 vs 19). Precision is
not part of this gate and remains the open problem.
declared sites agreed, against the reference's 16. Cost was $0.0194 per candidate with
recorded triage. **NOT MET** on recall.
*Correction (2026-09-30, in the record):* the first version misread agree's
`declared_sites_matched`. That field is coverage by candidates (19), but it was read as
"found in either pass". So the first version concluded detection was unchanged. Recounted
from the agreed rows, the plane's passes flagged 17 of 20 declared sites in at least one
pass. The reference flagged 19.
- **Increment 2** (2026-09-30, `benchmarks/independent/plane-increment-2.json`): came after the
design revisit. It added a tie-break pass c on the 10 disputed candidates, with 2-of-3
agreement. 16 of 20 declared sites agreed (reference 16). Cost was $0.0205 per candidate over
46, and $0.02195 over the 43 after triage (reference: $0.022). **MET**, with two
caveats the record carries:
1. The tie-break was chosen after seeing the first measurement on the same validation set.
So this compares plane with script and qualifies nothing on an independent population.
2. The plane's passes still flag fewer declared sites than the reference (17 vs 19).
Precision is not part of this gate and remains the open problem.
## The decision engine
Two slices have been measured out of repository (injection, 2026-10-01; six families,
2026-10-02). The engine is not adopted. The state against the gates, in one place with the plane
increment and the populations above, is on [Where we stand](where-we-stand.md#1-detection-capability-and-goals-no-go-measured);
the per-family tables are on [decision-engine.md](decision-engine.md).
Two slices have been measured out of repository: injection (2026-10-01) and six families
(2026-10-02). The engine is not adopted. [Where we
stand](where-we-stand.md#1-detection-capability-and-goals-no-go-measured) shows the state
against the gates in one place, with the plane increment and the populations above. The
per-family tables are on [decision-engine.md](decision-engine.md).
## Removal of the earlier agentic extra
The removal of the earlier agentic extra (2026-09-30) was checked for byte-identical
deterministic outputs (pair replay, quick benchmark, standard scan, `improve --dry-run`):
The earlier agentic extra was removed on 2026-09-30. The removal was checked for byte-identical
deterministic outputs: pair replay, quick benchmark, standard scan, `improve --dry-run`. Records:
`benchmarks/measurements/2026-09-30-harnessx-removal-baseline/` and `-equality/`.