marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: plain-language pass -- detection techniques, architecture, evaluation, decision engine

Browse Source 
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-10-02 21:03:24 +02:00
co-authored by Claude Opus 5.5
parent 50f4e69321
commit 352f8c07aa
4 changed files with 845 additions and 534 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture.md
+114 -67
View File
@@ -1,14 +1,22 @@
# Architecture # Architecture
How a scan works, how a finding earns its place, how severity and the project score are This page covers four things:
decided, and how the ruleset improves itself. State as of 2026-10-02; module paths are under
`src/openultrasast/`. - how a scan works;
- how a finding earns its place;
- how severity and the project score are decided;
- how the ruleset improves itself.
State as of 2026-10-02; module paths are under `src/openultrasast/`.
## The scan pipeline ## The scan pipeline
The CLI is thin; `HarnessRuntime` owns the lifecycle and writes an event trace for every stage The CLI is thin. `HarnessRuntime` owns the lifecycle. It writes an event trace for every stage
(`trace/events.jsonl`). The stages a mode requests are `MODE_STAGES` in `stages.py`: (`trace/events.jsonl`). `MODE_STAGES` in `stages.py` lists the stages each mode requests:
`quick` runs STATIC, `standard` adds MAP, `deep` adds REGRESS.
- `quick` runs STATIC;
- `standard` adds MAP;
- `deep` adds REGRESS.
``` ```
policy_load ─▶ ruleset_load ─▶ policy_check (fail loud on an ungoverned CWE) policy_load ─▶ ruleset_load ─▶ policy_check (fail loud on an ungoverned CWE)
@@ -28,14 +36,19 @@ policy_load ─▶ ruleset_load ─▶ policy_check (fail loud on an
``` ```
**The model layer** (`model/`, `cpg/`) is the core of `standard`. It builds one code property **The model layer** (`model/`, `cpg/`) is the core of `standard`. It builds one code property
graph per scan with Joern (`joern-parse`, then batched `joern --script` queries from graph (CPG: a graph of the program's syntax, control flow and data flow) per scan with Joern.
`cpg/queries/*.sc`) and dispatches each question to the arbiter of its spec type: taint It runs `joern-parse`, then batched `joern --script` queries from `cpg/queries/*.sc`. Each
(source to sink with no discharging sanitizer), dominance (does a guard govern the obligated question goes to the arbiter (a checker that decides one kind of question) for its spec type:
operation) and configuration (constant abstraction over a value). Taint facts ship for C,
Java, JavaScript, PHP and Python (`ruleset/semantic/*.toml`). Without Joern the layer records - taint (untrusted input reaching a dangerous call): source to sink with no discharging sanitizer;
`cpg_unavailable` and the scan is otherwise unchanged; the Docker image ships Joern and - dominance: does a guard govern the obligated operation;
`php-cli` (which the PHP frontend needs). The representations and techniques behind each stage - configuration: constant abstraction over a value.
are explained in [Detection techniques](detection-techniques.md).
Taint facts ship for C, Java, JavaScript, PHP and Python (`ruleset/semantic/*.toml`). Without
Joern, the layer records `cpg_unavailable` and the rest of the scan is unchanged. The Docker
image ships Joern and `php-cli`, which the PHP frontend needs. [Detection
techniques](detection-techniques.md) explains the representations and techniques behind each
stage.
The model layer reports on a rung ladder: The model layer reports on a rung ladder:
@@ -47,23 +60,29 @@ model_corroborated the graph found the shape; a judge agreed on the residual
suspicion proposed, nothing corroborated it suspicion proposed, nothing corroborated it
``` ```
The LLM (`model/pipeline.py`, `residual_question`) is asked only the residual question the graph The LLM answers only the residual question: the part the graph cannot settle
cannot settle, for a flow the graph already found; it can confirm or contradict, never create a (`model/pipeline.py`, `residual_question`). It is asked only about a flow the graph already
flow. When the graph resolved nothing, the enumerator's candidates (at most found. It can confirm or contradict that flow, but never create one. When the graph resolved
`MAX_JUDGED_CANDIDATES`, 8) are put to it, and anything it affirms is reported at `suspicion`. nothing, the LLM sees the enumerator's candidates (at most `MAX_JUDGED_CANDIDATES`, 8).
Without a provider key neither question is asked and only the graph's entailed findings are Anything it affirms is reported at `suspicion`. Without a provider key, neither question is
reported. asked. Only the graph's entailed findings are reported then.
**REGRESS** (`regress/`, `sandbox/`) runs only when `docker info` succeeds. Each promoted **REGRESS** (`regress/`, `sandbox/`) runs only when `docker info` succeeds. Each promoted
candidate gets a small language recipe (Python, JavaScript, C) that loads the candidate's candidate gets a small language recipe (Python, JavaScript, C). The recipe loads the
source inside a container with no network, a read-only `/workspace`, a non-root user and the candidate's source inside a container with:
`[sandbox]` limits. A structural deny-list rejects snippets mentioning sockets, curl, host
networking or writes under `/workspace` before anything runs. - no network;
- a read-only `/workspace`;
- a non-root user;
- the `[sandbox]` limits.
Before anything runs, a structural deny-list rejects snippets that mention sockets, curl, host
networking or writes under `/workspace`.
## Evidence ladder: how a false positive is eliminated ## Evidence ladder: how a false positive is eliminated
Evidence level is a state machine, not a label a model may assign. The verifier Evidence level is a state machine, not a label a model may assign. The verifier
(`verification.py`) runs on independent context and enforces: (`verification.py`) runs on independent context and enforces it:
``` ```
suspicion ─▶ static_corroboration ─▶ crash_reproduced ─▶ root_cause_explained suspicion ─▶ static_corroboration ─▶ crash_reproduced ─▶ root_cause_explained
@@ -75,31 +94,36 @@ suspicion ─▶ static_corroboration ─▶ crash_reproduced ─▶ root_cause_
call-graph, route, CLI, parser or dynamic evidence; call-graph, route, CLI, parser or dynamic evidence;
- `static_corroboration` with function-level reachability: `ACCEPTED`. - `static_corroboration` with function-level reachability: `ACCEPTED`.
A rejected finding becomes a **scoped false-positive learning** (`calibration.py`) with a A rejected finding becomes a **scoped false-positive learning** (`calibration.py`). It has a
reason from a fixed taxonomy (`unreachable_path`, `missing_attacker_control`, scope and a reason from a fixed taxonomy: `unreachable_path`, `missing_attacker_control`,
`sanitizer_disproved`, `static_rule_mismatch`, `incorrect_model_assumption`, `duplicate`, `sanitizer_disproved`, `static_rule_mismatch`, `incorrect_model_assumption`, `duplicate`,
`insufficient_impact`, `unsupported`, `contradicted`, `unverified`) and a scope. A rejection in `insufficient_impact`, `unsupported`, `contradicted`, `unverified`. A rejection in `auth/`
`auth/` demotes only `auth/...`, never the whole class demotes only `auth/...`, never the whole class
(`tests/test_calibration.py::test_scoped_false_positive_demotion_does_not_suppress_class_globally`). (`tests/test_calibration.py::test_scoped_false_positive_demotion_does_not_suppress_class_globally`).
## How a false negative is surfaced ## How a false negative is surfaced
Every expected vulnerability that no finding matched becomes a `BenchmarkCalibrationRecord` in An expected vulnerability that no finding matched becomes a `BenchmarkCalibrationRecord` in
`calibration_records.json`, with a `next_improvement_candidate` (a static rule, SARIF source, `calibration_records.json`. Each record names a `next_improvement_candidate`: a static rule,
entry-point mapping, retrieval package, hunter prompt, dynamic reproducer or skill route). SARIF source, entry-point mapping, retrieval package, hunter prompt, dynamic reproducer or
Once a miss is closed the detection gate keeps it closed in CI. skill route. Once a miss is closed, the detection gate keeps it closed in CI.
## Central CWE policy and the project score ## Central CWE policy and the project score
**Policy** (`policy/verycode.py`): a vendored `CWE_Score.tsv` (verycode-policies) is the single **Policy** (`policy/verycode.py`): a vendored `CWE_Score.tsv` (verycode-policies) is the single
source of truth. Each CWE carries a flaw category, a severity (0-5) and `static`/`dynamic` scope source of truth. Each CWE carries:
flags. A finding's severity comes only from policy, keyed on CWE. CWEs that are not `static`
resolve to 0 and are report-only. `assert_rules_resolve` runs as the `policy_check` stage: an - a flaw category;
enabled rule naming an ungoverned CWE aborts the scan. Verycode has no CWE-120, so the C/C++ - a severity (0-5);
- `static`/`dynamic` scope flags.
A finding's severity comes only from policy, keyed on CWE. CWEs that are not `static` resolve
to 0 and are report-only. `assert_rules_resolve` runs as the `policy_check` stage. An enabled
rule that names an ungoverned CWE aborts the scan. Verycode has no CWE-120, so the C/C++
buffer rules carry CWE-121. buffer rules carry CWE-121.
**Score** (`scoring/project_score.py`): each finding's penalty is **Score** (`scoring/project_score.py`): each finding's penalty is
`SEV_WEIGHT[severity] x REACH_MULT[reachability]`, and the score is `100 * e^(-total/k)` with `SEV_WEIGHT[severity] x REACH_MULT[reachability]`. The score is `100 * e^(-total/k)` with
`k = 60`. `k = 60`.
| Severity | Weight | | Reachability | Multiplier | | Severity | Weight | | Reachability | Multiplier |
@@ -109,10 +133,13 @@ buffer rules carry CWE-121.
| 3 | 10 | | `unknown` | 0.4 | | 3 | 10 | | `unknown` | 0.4 |
| 2 / 1 / 0 | 2 / 1 / 0 | | | | | 2 / 1 / 0 | 2 / 1 / 0 | | | |
A confirmed false positive lowers a finding's reachability multiplier instead of deleting the A confirmed false positive lowers a finding's reachability multiplier. The rule is not
rule. A severity-5 reachable finding always fails the score gate; `score < min_score` deleted. The score gate works like this:
(default 80) fails it only with `[score] blocking = true`. The `score` stage writes
`score.json` and merges it into `manifest.json`. - a severity-5 reachable finding always fails it;
- `score < min_score` (default 80) fails it only with `[score] blocking = true`.
The `score` stage writes `score.json` and merges it into `manifest.json`.
## Self-improvement ## Self-improvement
@@ -124,41 +151,61 @@ scan ─▶ verify ─▶ non-accepted findings ─▶ scoped learnings
next scan ─▶ rank ─▶ calibrate (demote rejected scopes) ─▶ findings next scan ─▶ rank ─▶ calibrate (demote rejected scopes) ─▶ findings
``` ```
`record_calibration` (after `verify`) merges this run's non-accepted outcomes into the ledger, 1. `record_calibration` runs after `verify`. It merges this run's non-accepted outcomes into
de-duplicated by finding ID; `calibrate` (after `rank`) demotes the priority of every scope the ledger, de-duplicated by finding ID.
that produced a rejection and writes `applied_calibrations.json`. Demotion is scoped and 2. `calibrate` runs after `rank`. It demotes the priority of every scope that produced a
reversible (`tests/test_pipeline_calibration.py`). rejection and writes `applied_calibrations.json`.
Demotion is scoped and reversible (`tests/test_pipeline_calibration.py`).
### Rule-level loop: `ousast improve` ### Rule-level loop: `ousast improve`
Each round (`improve/evolve.py`): benchmark with the current ledger, per-rule signals Each round (`improve/evolve.py`) runs these steps:
(`rule_signals.json`), bounded proposals, validation (`improve/validator.py`), replay smoke,
re-benchmark, and a hard gate. The levers are rule status (`enabled -> shadow` before 1. benchmark with the current ledger;
`disabled`) and score constants; the loop can never edit pattern text 2. per-rule signals (`rule_signals.json`);
or the authoritative CWE severity. A round is accepted only if recall >= 90%, FP < 10%, the 3. bounded proposals;
project score did not regress and no matched finding was lost; otherwise the ledger is not 4. validation (`improve/validator.py`);
written. With `--pair-catalog`, every round is also gated per provenance profile on the 5. replay smoke;
catalog's holdout pairs (`--no-pair-gate` skips this and is not for accepted ledgers). The same 6. re-benchmark;
detection gate runs in CI (`python -m openultrasast.gate`). 7. a hard gate.
The levers are rule status (`enabled -> shadow` before `disabled`) and score constants. The
loop can never edit pattern text or the authoritative CWE severity. A round is accepted only if
all of these hold:
- recall >= 90%;
- FP < 10%;
- the project score did not regress;
- no matched finding was lost.
Otherwise the ledger is not written. With `--pair-catalog`, each round is also gated per
provenance profile on the catalog's holdout pairs. `--no-pair-gate` skips this check; do not
use it for accepted ledgers. The same detection gate runs in CI (`python -m openultrasast.gate`).
### Proposals from plane memory ### Proposals from plane memory
With `--memory [STORE]`, `ousast improve` also reads the rows plane runs stored per repository With `--memory [STORE]`, `ousast improve` also reads the rows that plane runs stored per
and pin (`improve/memory.py`). Two deterministic rules turn them into rule-status proposals: a repository and pin (`improve/memory.py`). Two deterministic rules turn these rows into
rule whose alerts are repeatedly false across repositories is shadowed, and a shadow rule that rule-status proposals:
keeps hitting agreed, declared vulnerable sites is re-enabled. Rows from holdout pairs, from the
gated manifest's own cases and from any `--qualify-population` are dropped before any rule sees - a rule whose alerts are repeatedly false across repositories is shadowed;
them. Each proposal records the rows it came from (`memory_proposals.jsonl` next to the - a shadow rule that keeps hitting agreed, declared vulnerable sites is re-enabled.
journal) and goes through the same validator and gate.
Some rows are dropped before any rule sees them: rows from holdout pairs, from the gated
manifest's own cases and from any `--qualify-population`. Each proposal records the rows it
came from (`memory_proposals.jsonl` next to the journal). It then goes through the same
validator and gate.
### The loop as a plane Run ### The loop as a plane Run
`ousast plane workspaces <population> --validation-set ... --loop` appends the improvement loop `ousast plane workspaces <population> --validation-set ... --loop` appends the improvement loop
to a generated Run: `alerts` per case (the quick scan on the vulnerable and fixed pins), then to a generated Run. The Run holds `alerts` per case: the quick scan on the vulnerable and fixed
the steps `snapshot` (task `memory-snapshot`), `measure`, `propose` and `improve` (`plane/tasks/loop.py`), each model-free pins. Then come the steps `snapshot` (task `memory-snapshot`), `measure`, `propose` and
with a zero budget. The snapshot applies the same train-on-test guard before any row leaves the `improve` (`plane/tasks/loop.py`). Each step is model-free with a zero budget. The snapshot
store. A recorded live run: `benchmarks/measurements/2026-09-30-harnessx-removal-loop/`. applies the same train-on-test guard before any row leaves the store. A recorded live run:
`benchmarks/measurements/2026-09-30-harnessx-removal-loop/`.
For PHP, which quick mode covered with zero rules when that run was made, `ousast plane When that run was made, quick mode covered PHP with zero rules. So for PHP, `ousast plane
alerts-engine` produces a Run's `alerts` from the engine image on the host instead alerts-engine` produces a Run's `alerts` from the engine image on the host instead
(`benchmarks/measurements/2026-09-30-php-engine-alerts/`). (`benchmarks/measurements/2026-09-30-php-engine-alerts/`).
docs/decision-engine.md
+107 -67
View File
@@ -1,51 +1,77 @@
# The learned decision engine (in development) # The learned decision engine (in development)
**Status 2026-10-02: in development, not adopted.** No scan, pre-push check or report uses it; **Status 2026-10-02: in development, not adopted.** No scan, pre-push check or report uses it.
every user-facing path is today's deterministic one until a compiled program is adopted. The Every user-facing path stays on today's deterministic one until a compiled program is adopted.
design is the `learned-decision-engine` specification (maintainer tooling); the code is in The design is the `learned-decision-engine` specification (maintainer tooling). The code is in
`src/openultrasast/learn/`. `src/openultrasast/learn/`.
## Why ## Why
Every detection gain before the engine was a hand edit tuned on the cases that exposed it, and Before the engine, every detection gain was a hand edit. Each edit was tuned on the cases that
engine tuning measured on population v1 did not transfer to v2. The reasoning, and the step-by-step exposed the miss. Engine tuning measured on population v1 did not transfer to v2. [Where we
chain that replaced hand edits, are on [Where we stand](where-we-stand.md#2-our-approach-to-detection-step-by-step). stand](where-we-stand.md#2-our-approach-to-detection-step-by-step) gives the reasoning and the
step-by-step chain that replaced hand edits.
## Design ## Design
- **Signals, not verdicts.** Every instrument (quick rules, the Joern engine, model sink and - **Signals, not verdicts.** Every instrument contributes a feature record per candidate
role classification, the plane's verify passes and agreement, repository facts such as `(path, function, family)`. The instruments are quick rules, the Joern engine, model sink and
callers) contributes a feature record per candidate `(path, function, family)`, including an role classification, the plane's verify passes and agreement, and repository facts such as
explicit state when it did not run or failed. None decides alone. callers. The record includes an explicit state when an instrument did not run or failed. No
instrument decides alone.
- **Labels from ground truth.** `ousast learn labels` derives labels only from sources on a - **Labels from ground truth.** `ousast learn labels` derives labels only from sources on a
fail-closed allow-list (`learn/sources.toml`): the vulnerable and fixed sides of pairs, the fail-closed allow-list (`learn/sources.toml`). Fail-closed means a source not on the list is
declared sites and fix ranges of spent populations, benign deltas and recorded adjudications. never used. The sources are:
- the vulnerable and fixed sides of pairs;
- the declared sites and fix ranges of spent populations;
- benign deltas;
- recorded adjudications.
"Rejected by the plane" is never a negative. Reserved populations (v3) are not on the list. "Rejected by the plane" is never a negative. Reserved populations (v3) are not on the list.
- **An AI classifier with local memory, compiled in house in the DSPy style.** The program is an - **An AI classifier with local memory, compiled in house in the DSPy style.** The program has
instruction, a few demonstrations and up to six retrieved labelled examples, then the an instruction, a few demonstrations and up to six retrieved labelled examples. Then come the
candidate's code excerpt and signals; the model (`deepseek-flash`) answers candidate's code excerpt and signals. The model (`deepseek-flash`) answers vulnerable / not
vulnerable / not vulnerable / unsure with a rationale, sampled `k` times, giving a score `s`. vulnerable / unsure with a rationale. It is sampled `k` times, which gives a score `s`.
Compilation bootstraps demonstrations and searches instructions on a compile split, scored by Compilation bootstraps demonstrations and searches instructions on a compile split. A
a pre-registered balanced Brier score (`learn/compile.toml`). pre-registered balanced Brier score (`learn/compile.toml`) scores the candidates. The Brier score
- **Retrieval** from the memory store: nearest neighbours by Gower distance on the signal measures how far predicted probabilities are from the true labels.
profile, re-ranked by cosine similarity of code embeddings (OpenAI `text-embedding-3-small` - **Retrieval** from the memory store works in three steps:
through OpenRouter, cached by excerpt hash), under a label/family/repository balance rule. An 1. find nearest neighbours by Gower distance (a distance over mixed feature types) on the
evaluation-boundary filter keeps a candidate's own repository group out of what it is shown. signal profile;
2. re-rank them by cosine similarity of code embeddings (OpenAI `text-embedding-3-small`
through OpenRouter, cached by excerpt hash);
3. apply a label/family/repository balance rule.
An evaluation-boundary filter keeps a candidate's own repository group out of what it is
shown.
- **Repository-grouped evaluation.** The compile split (25% of repository groups) is never - **Repository-grouped evaluation.** The compile split (25% of repository groups) is never
evaluated. The rest is evaluated by outer grouped 5-fold cross-validation: the Platt evaluated. The rest is evaluated by outer grouped 5-fold cross-validation:
calibration map is cross-fitted on the other folds and the operating points are chosen on the - the Platt calibration map (which turns the score into a probability) is cross-fitted on
other folds (nested). One compile per profile, not one per fold. Intervals are 95% the other folds;
repository-cluster bootstrap. - the operating points are chosen on the other folds (nested).
- **Two operating points.** BLOCK is offered only when calibration holds (slope CI contains 1,
intercept CI contains 0, ECE <= 0.05) and a fold reaches the required precision; otherwise the There is one compile per profile, not one per fold. Intervals are 95% repository-cluster
engine is ADVISORY only. An `unsure` answer can never BLOCK. bootstrap.
- **Two operating points.** BLOCK is offered only when both of these hold:
- calibration holds: slope CI contains 1, intercept CI contains 0, ECE <= 0.05;
- a fold reaches the required precision.
Otherwise the engine is ADVISORY only. An `unsure` answer can never BLOCK. ECE (expected
calibration error) is the average gap between predicted and observed rates.
## First measured slice (2026-10-01) ## First measured slice (2026-10-01)
Record: `benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json` (counts Record: `benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json` (counts
only). Family **injection** only; memory of 724 static examples from all families; input profile only). The setup:
v1 with the engine signals and function length withheld (a leak audit found them separating the
sides of a pair by construction: `2026-10-01-decision-engine-leak-audit/`); framework priors off. - family **injection** only;
- memory of 724 static examples from all families;
- input profile v1, with the engine signals and function length withheld;
- framework priors off.
The signals were withheld because a leak audit found them separating the sides of a pair by
construction (`2026-10-01-decision-engine-leak-audit/`). AUC below is the chance the engine
ranks a real bug above a fixed one.
| Measure | Value (95% repository-cluster CI) | | Measure | Value (95% repository-cluster CI) |
| --- | --- | | --- | --- |
@@ -62,11 +88,17 @@ sides of a pair by construction: `2026-10-01-decision-engine-leak-audit/`); fram
## Second slice: six families (2026-10-02) ## Second slice: six families (2026-10-02)
Record: `benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json` (counts only). Record: `benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json` (counts only).
After the advisory-fix harvest the memory holds 904 static examples (724 before). One program was After the advisory-fix harvest, the memory holds 904 static examples (724 before).
compiled per evaluable family (injection recompiled, since the memory changed) and evaluated on
that family's own candidates, out of repository as above: compile split never evaluated, outer One program was compiled per evaluable family. Injection was recompiled, since the memory
grouped 5-fold, calibration cross-fitted, operating points nested, 95% repository-cluster changed. Each program was evaluated on its family's own candidates, out of repository as above:
bootstrap intervals (2,000 resamples); input profile v1, priors off, k = 5.
- compile split never evaluated;
- outer grouped 5-fold;
- calibration cross-fitted;
- operating points nested;
- 95% repository-cluster bootstrap intervals (2,000 resamples);
- input profile v1, priors off, k = 5.
| Family | Candidates (groups; positive) | Pooled AUC of `s` (95% CI) | Within-pair AUC (paired groups) | ADVISORY recall / precision | Canary agreement | | Family | Candidates (groups; positive) | Pooled AUC of `s` (95% CI) | Within-pair AUC (paired groups) | ADVISORY recall / precision | Canary agreement |
| --- | --- | --- | --- | --- | --- | | --- | --- | --- | --- | --- | --- |
@@ -77,45 +109,53 @@ bootstrap intervals (2,000 resamples); input profile v1, priors off, k = 5.
| path | 35 (21; 21) | 0.89 (0.79 - 0.97) | 0.83 (14) | 0.86 / 0.69 | 0.90 | | path | 35 (21; 21) | 0.89 (0.79 - 0.97) | 0.83 (14) | 0.86 / 0.69 | 0.90 |
| untrusted destination | 62 (37; 38) | 0.85 (0.76 - 0.94) | 0.72 (18) | 0.84 / 0.73 | 0.87 | | untrusted destination | 62 (37; 38) | 0.85 (0.76 - 0.94) | 0.72 (18) | 0.84 / 0.73 | 0.87 |
- **BLOCK is offered for no family.** Calibration holds only for output encoding (ECE 0.045), and - **BLOCK is offered for no family.** Calibration holds only for output encoding (ECE 0.045).
there no fold reaches the required precision (`precision_unreachable`: no fold reaches a Wilson There, no fold reaches the required precision (`precision_unreachable`: no fold reaches a
lower bound of 0.95). For the other five families calibration does not hold (ECE 0.071 to 0.119). Wilson lower bound of 0.95). For the other five families, calibration does not hold (ECE
- **The pooled AUC is the more favourable number.** Every family's pool includes candidates that 0.071 to 0.119).
carry only a positive label, with no fixed-side counterpart (7 to 19 per family; there are no - **The pooled AUC is the more favourable number.** Every family's pool includes candidates
single-label negatives), so the pooled AUC is inflated by positives that never had to be told that carry only a positive label, with no fixed-side counterpart (7 to 19 per family; there
apart from their own fix. The within-pair AUC over the paired groups, 0.72 to 0.93, is the are no single-label negatives). These positives never had to be told apart from their own
stricter reading (`paired_groups_check` in the record). fix, so they inflate the pooled AUC. The within-pair AUC over the paired groups, 0.72 to
0.93, is the stricter reading (`paired_groups_check` in the record).
- **Re-run agreement (canary, 30 candidates per family) is below 0.9 for three families:** - **Re-run agreement (canary, 30 candidates per family) is below 0.9 for three families:**
injection 0.77, access control 0.83, untrusted destination 0.87. injection 0.77, access control 0.83, untrusted destination 0.87.
- **One memory-size point for injection** (paired: the same program, candidates and folds, with a - **One memory-size point for injection.** This run is paired: the same program, candidates
retrieval memory of 724 vs 904 examples): AUC 0.76 vs 0.77, within the intervals. No injection and folds, with a retrieval memory of 724 vs 904 examples. AUC was 0.76 vs 0.77, within the
example was added, so the new examples reach an injection prompt only as contrast. intervals. No injection example was added. So the new examples reach an injection prompt
- **Spend:** $2.88 metered at list price for DeepSeek under an $8 ceiling, $0.0009 for only as contrast.
embeddings; $0.0027 to $0.0032 per evaluated candidate. The access-control evaluation replayed - **Spend:** $2.88 metered at list price for DeepSeek under an $8 ceiling, and $0.0009 for
from the response cache at $0 with identical predictions. embeddings. That is $0.0027 to $0.0032 per evaluated candidate. The access-control
- **An instrument fix is recorded:** compiled demonstrations ignored the near-duplicate rule evaluation replayed from the response cache at $0 with identical predictions.
(cosine >= 0.98) until the pre-call boundary assertion stopped the untrusted-destination - **An instrument fix is recorded.** Compiled demonstrations ignored the near-duplicate rule
evaluation; demonstrations are now swapped or dropped like retrieved examples, and the stopped (cosine >= 0.98). The pre-call boundary assertion caught this and stopped the
part was replayed after the fix. untrusted-destination evaluation. Demonstrations are now swapped or dropped like retrieved
examples. The stopped part was replayed after the fix.
**Not adopted.** What is still ahead is listed once, on **Not adopted.** [Where we stand](where-we-stand.md#5-open-items) lists, in one place, what is
[Where we stand](where-we-stand.md#5-open-items). still ahead.
## Experiments ## Experiments
Changes to prompts, models, pass counts, features or sampling are compared by controlled Changes to prompts, models, pass counts, features or sampling are compared by controlled
experiments from a pre-registered manifest (`plane/experiments/<id>.yaml`), so nothing is adopted experiments. Each experiment comes from a pre-registered manifest
on one run's number. Two are registered. exp-001 (known callers) has not run. exp-002 (a retrieval (`plane/experiments/<id>.yaml`). This way nothing is adopted on one run's number.
ensemble against today's temperature sampling) ran on 160 paired candidates in 56 repository
groups and was **rejected**: within-pair AUC 0.815 against 0.741 and $0.0021 against $0.0051 per Two experiments are registered:
candidate, with re-run agreement 0.875 against 0.906 and an interval spanning zero
(`benchmarks/experiments/exp-002-retrieval-ensemble/result.json`). The reading is on - exp-001 (known callers) has not run.
[Where we stand](where-we-stand.md#2-our-approach-to-detection-step-by-step). - exp-002 compared a retrieval ensemble against today's temperature sampling. It ran on 160
paired candidates in 56 repository groups and was **rejected**. Within-pair AUC was 0.815
against 0.741 and the cost was $0.0021 against $0.0051 per candidate. The record shows re-run agreement
0.875 against 0.906 and an interval spanning zero
(`benchmarks/experiments/exp-002-retrieval-ensemble/result.json`).
The reading is on [Where we stand](where-we-stand.md#2-our-approach-to-detection-step-by-step).
## Commands ## Commands
`ousast learn` is a maintainer command group over the engine's data. Paid steps take `ousast learn` is a maintainer command group over the engine's data. Paid steps take
`--budget-usd`; `--replay-only` answers every call from the response cache at no cost. `--budget-usd`. `--replay-only` answers every call from the response cache at no cost.
| Command | Does | | Command | Does |
| --- | --- | | --- | --- |
@@ -130,5 +170,5 @@ candidate, with re-run agreement 0.875 against 0.906 and an interval spanning ze
The memory store is the plane's (`OUSAST_MEMORY`; see [ops/ax/README.md](ops/ax/README.md)). The memory store is the plane's (`OUSAST_MEMORY`; see [ops/ax/README.md](ops/ax/README.md)).
The harvests that produced the plane signals ran as plane Runs generated by The harvests that produced the plane signals ran as plane Runs generated by
`ousast plane harvest` (`benchmarks/measurements/2026-10-01-decision-engine-harvest/record.json` `ousast plane harvest`. Records: `benchmarks/measurements/2026-10-01-decision-engine-harvest/record.json`
and, for the advisory-fix pairs, `2026-10-02-decision-engine-harvest-advisory/record.json`). and, for the advisory-fix pairs, `2026-10-02-decision-engine-harvest-advisory/record.json`.
docs/detection-techniques.md
+548 -347
View File
File diff suppressed because it is too large Load Diff
docs/evaluation.md
+75 -52
View File
@@ -1,7 +1,8 @@
# Evaluation # Evaluation
What has been measured on code the tool was not tuned on, and how. State as of 2026-10-02. This page shows what has been measured on code the tool was not tuned on, and how. State as of
Every number below is quoted from a committed record; the path is given with it. 2026-10-02. Every number below is quoted from a committed record, and the record's path is
given with it.
## Instruments ## Instruments
@@ -11,32 +12,47 @@ Every number below is quoted from a committed record; the path is given with it.
| Pair corpus | fire on the vulnerable function, stay silent on its fix | `ousast pairs`, `benchmarks/pairs/` | | Pair corpus | fire on the vulnerable function, stay silent on its fix | `ousast pairs`, `benchmarks/pairs/` |
| Independent populations | real repositories at vulnerable, fixed and benign pins, scored under a pre-registered protocol | `benchmarks/independent/` | | Independent populations | real repositories at vulnerable, fixed and benign pins, scored under a pre-registered protocol | `benchmarks/independent/` |
The detection gate and the local pairs are CI gates. They catch regressions; they do not Recall means the share of known bugs found. FP means false positives.
The detection gate and the local pairs are CI gates. They catch regressions. They do not
estimate recall on real code, because the rules were written against them. estimate recall on real code, because the rules were written against them.
## Pair corpora ## Pair corpora
`benchmarks/pairs/catalog.toml` and the slice directories (`local`, `github`, `sast`, `vfc`, A pair is a vulnerable function and its fixed version. You can run the catalog
`vibe-py`, `vfc-js`, `agent-vfc`, `owasp`) are runnable with `ousast pairs --slice <name>`; each `benchmarks/pairs/catalog.toml` and the slice directories with `ousast pairs --slice <name>`.
pair carries a provenance profile and a train/holdout split (`--profile`, `--split`). Pointer The slices are `local`, `github`, `sast`, `vfc`, `vibe-py`, `vfc-js`, `agent-vfc` and `owasp`.
pairs are not vendored: `--pointers` fetches them into `~/.cache/openultrasast/pairs/`.
`benchmarks/pairs/README.md` documents the schema and the review tiers. - Each pair carries a provenance profile and a train/holdout split (`--profile`, `--split`).
- Pointer pairs are not vendored. `--pointers` fetches them into `~/.cache/openultrasast/pairs/`.
- `benchmarks/pairs/README.md` documents the schema and the review tiers.
**`advisory-fixes`** (`benchmarks/pairs/advisory-fixes/`, 2026-10-01) holds 97 pointer pairs, **`advisory-fixes`** (`benchmarks/pairs/advisory-fixes/`, 2026-10-01) holds 97 pointer pairs,
one per repository: a public CVE/GHSA fix commit whose advisory-named function is declared on one per repository. Each is a public CVE/GHSA fix commit. The function the advisory names is
both sides and changed by the fix. It was assembled to give the decision engine real negatives declared on both sides and changed by the fix.
for families the corpus had few of (access control, deserialization, output encoding, path,
untrusted destination). It is a label source of the decision engine It was assembled to give the decision engine real negatives (fixed, non-vulnerable code). It
(`src/openultrasast/learn/sources.toml`), not an `ousast pairs` slice; the research log and targets families the corpus had few of: access control, deserialization, output encoding,
path, untrusted destination. It is a label source of the decision engine
(`src/openultrasast/learn/sources.toml`), not an `ousast pairs` slice. The research log and
every rejection are in `research-2026-10-01.md`. every rejection are in `research-2026-10-01.md`.
## Independent populations and the M4 gates ## Independent populations and the M4 gates
The pre-push safety net's qualification milestone (M4) requires, together: at least 95% M4 is the qualification milestone of the pre-push safety net. It requires all of these
actionable precision, at least 90% supported recall, at least 95% supported-check completion, together:
the runtime gates, and zero fixed-side or benign false alerts. A population is frozen before
any of its cases is scanned, scored once under a pre-registered protocol, and then **spent**: - at least 95% actionable precision;
it may inform changes, but it can no longer qualify anything. - at least 90% supported recall;
- at least 95% supported-check completion;
- the runtime gates;
- zero fixed-side or benign false alerts.
A population is a set of real repositories. Its lifecycle has three steps:
1. It is frozen before any of its cases is scanned.
2. It is scored once under a pre-registered protocol.
3. It is then **spent**: it may inform changes, but it can no longer qualify anything.
| Population | Status | Result | Record | | Population | Status | Result | Record |
| --- | --- | --- | --- | | --- | --- | --- | --- |
@@ -45,9 +61,9 @@ it may inform changes, but it can no longer qualify anything.
| v2, model-driven pipeline | exploratory (v2 already spent) | stopped after 7 of 17 cases when the account emptied: recall 4/7; precision not adjudicated; no gate measurable | `results-v2-model-pipeline.json` | | v2, model-driven pipeline | exploratory (v2 already spent) | stopped after 7 of 17 cases when the account emptied: recall 4/7; precision not adjudicated; no gate measurable | `results-v2-model-pipeline.json` |
| v3 (PHP, 15 cases; one SSRF slot left empty by the selection rule) | frozen, untouched | none: it is the one-time final check, run once under `protocol-v3.md` after a prediction is committed | `population-v3-php.toml`, `freeze-v3-php.json` | | v3 (PHP, 15 cases; one SSRF slot left empty by the selection rule) | frozen, untouched | none: it is the one-time final check, run once under `protocol-v3.md` after a prediction is committed | `population-v3-php.toml`, `freeze-v3-php.json` |
v3 is reserved: development and analysis work never reads its cases, and v3 is reserved. Development and analysis work never reads its cases.
`tests/test_independent_population.py` fails when anything in the development tree (a recipe, `tests/test_independent_population.py` fails when anything in the development tree names one
catalog, manifest or measurement) names one of its repositories. of its repositories. That covers a recipe, catalog, manifest or measurement.
```mermaid ```mermaid
stateDiagram-v2 stateDiagram-v2
@@ -66,9 +82,10 @@ stateDiagram-v2
## Folds: how learned decisions are kept out of their own training data ## Folds: how learned decisions are kept out of their own training data
The decision engine is evaluated on repositories it was not shown (`src/openultrasast/learn/folds.py`). The decision engine is evaluated on repositories it was not shown
Every split is by repository group (the label builder's normalised `owner/name`, merged across (`src/openultrasast/learn/folds.py`). Every split is by repository group. A group is the label
URLs, forks and shared advisories), across all corpora, seeded and deterministic. builder's normalised `owner/name`, merged across URLs, forks and shared advisories. Splits
cover all corpora and are seeded and deterministic.
```mermaid ```mermaid
flowchart TD flowchart TD
@@ -81,44 +98,50 @@ flowchart TD
rest --> lofo["Leave-one-framework-out: frameworks with at least 10 groups"] rest --> lofo["Leave-one-framework-out: frameworks with at least 10 groups"]
``` ```
`assert_disjoint` runs before a compile: the compile split and the evaluation folds share no `assert_disjoint` runs before a compile. It checks that the compile split and the evaluation
group. A framework with fewer than 10 groups gets `insufficient data` instead of a fold. As of folds share no group. A framework with fewer than 10 groups gets `insufficient data` instead of
2026-10-02 the outer folds have been run for the six evaluable families; leave-one-source-out a fold. As of 2026-10-02, the outer folds have been run for the six evaluable families.
and leave-one-framework-out have not Leave-one-source-out and leave-one-framework-out have not
([decision-engine.md](decision-engine.md#second-slice-six-families-2026-10-02)). ([decision-engine.md](decision-engine.md#second-slice-six-families-2026-10-02)).
## The plane increment ## The plane increment
The first increment of the agentic plane ran the model-driven pipeline's validation set (46 The first increment of the agentic plane ran on google/ax. It used the model-driven pipeline's
candidates in 15 cases of v2, 20 declared vulnerable sites) on google/ax and compared it with validation set: 46 candidates in 15 cases of v2, with 20 declared vulnerable sites. It was
the script pipeline on the same cases, candidates, pins and recorded triage. The pre-registered compared with the script pipeline on the same cases, candidates, pins and recorded triage.
gate: at least 16 of 20 declared sites agreed, at a cost per candidate under $0.022.
The pre-registered gate: at least 16 of 20 declared sites agreed, at a cost per candidate under
$0.022.
- **Increment 1** (2026-09-29, `benchmarks/independent/plane-increment-1.json`): 14 of 20 - **Increment 1** (2026-09-29, `benchmarks/independent/plane-increment-1.json`): 14 of 20
declared sites agreed against the reference's 16; $0.0194 per candidate with recorded triage. declared sites agreed, against the reference's 16. Cost was $0.0194 per candidate with
**NOT MET** on recall. recorded triage. **NOT MET** on recall.
*Correction (2026-09-30, in the record):* the first version read agree's *Correction (2026-09-30, in the record):* the first version misread agree's
`declared_sites_matched` (coverage by candidates, 19) as "found in either pass" and concluded `declared_sites_matched`. That field is coverage by candidates (19), but it was read as
detection was unchanged. Recounted from the agreed rows, the plane's passes flagged 17 of 20 "found in either pass". So the first version concluded detection was unchanged. Recounted
declared sites in at least one pass against the reference's 19. from the agreed rows, the plane's passes flagged 17 of 20 declared sites in at least one
- **Increment 2** (2026-09-30, `benchmarks/independent/plane-increment-2.json`): after the design pass. The reference flagged 19.
revisit, a tie-break pass c on the 10 disputed candidates with 2-of-3 agreement. 16 of 20 - **Increment 2** (2026-09-30, `benchmarks/independent/plane-increment-2.json`): came after the
declared sites agreed (reference 16) at $0.0205 per candidate over 46 ($0.02195 over the 43 design revisit. It added a tie-break pass c on the 10 disputed candidates, with 2-of-3
after triage, against the reference's $0.022). **MET**, with two caveats the record carries: agreement. 16 of 20 declared sites agreed (reference 16). Cost was $0.0205 per candidate over
the tie-break was chosen after seeing the first measurement on the same validation set, so 46, and $0.02195 over the 43 after triage (reference: $0.022). **MET**, with two
this compares plane with script and qualifies nothing on an independent population; and the caveats the record carries:
plane's passes still flag fewer declared sites than the reference (17 vs 19). Precision is 1. The tie-break was chosen after seeing the first measurement on the same validation set.
not part of this gate and remains the open problem. So this compares plane with script and qualifies nothing on an independent population.
2. The plane's passes still flag fewer declared sites than the reference (17 vs 19).
Precision is not part of this gate and remains the open problem.
## The decision engine ## The decision engine
Two slices have been measured out of repository (injection, 2026-10-01; six families, Two slices have been measured out of repository: injection (2026-10-01) and six families
2026-10-02). The engine is not adopted. The state against the gates, in one place with the plane (2026-10-02). The engine is not adopted. [Where we
increment and the populations above, is on [Where we stand](where-we-stand.md#1-detection-capability-and-goals-no-go-measured); stand](where-we-stand.md#1-detection-capability-and-goals-no-go-measured) shows the state
the per-family tables are on [decision-engine.md](decision-engine.md). against the gates in one place, with the plane increment and the populations above. The
per-family tables are on [decision-engine.md](decision-engine.md).
## Removal of the earlier agentic extra ## Removal of the earlier agentic extra
The removal of the earlier agentic extra (2026-09-30) was checked for byte-identical The earlier agentic extra was removed on 2026-09-30. The removal was checked for byte-identical
deterministic outputs (pair replay, quick benchmark, standard scan, `improve --dry-run`): deterministic outputs: pair replay, quick benchmark, standard scan, `improve --dry-run`. Records:
`benchmarks/measurements/2026-09-30-harnessx-removal-baseline/` and `-equality/`. `benchmarks/measurements/2026-09-30-harnessx-removal-baseline/` and `-equality/`.