diff --git a/docs/architecture.md b/docs/architecture.md index df16792..cee0b8f 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,14 +1,22 @@ # Architecture -How a scan works, how a finding earns its place, how severity and the project score are -decided, and how the ruleset improves itself. State as of 2026-10-02; module paths are under -`src/openultrasast/`. +This page covers four things: + +- how a scan works; +- how a finding earns its place; +- how severity and the project score are decided; +- how the ruleset improves itself. + +State as of 2026-10-02; module paths are under `src/openultrasast/`. ## The scan pipeline -The CLI is thin; `HarnessRuntime` owns the lifecycle and writes an event trace for every stage -(`trace/events.jsonl`). The stages a mode requests are `MODE_STAGES` in `stages.py`: -`quick` runs STATIC, `standard` adds MAP, `deep` adds REGRESS. +The CLI is thin. `HarnessRuntime` owns the lifecycle. It writes an event trace for every stage +(`trace/events.jsonl`). `MODE_STAGES` in `stages.py` lists the stages each mode requests: + +- `quick` runs STATIC; +- `standard` adds MAP; +- `deep` adds REGRESS. ``` policy_load ─▶ ruleset_load ─▶ policy_check (fail loud on an ungoverned CWE) @@ -28,14 +36,19 @@ policy_load ─▶ ruleset_load ─▶ policy_check (fail loud on an ``` **The model layer** (`model/`, `cpg/`) is the core of `standard`. It builds one code property -graph per scan with Joern (`joern-parse`, then batched `joern --script` queries from -`cpg/queries/*.sc`) and dispatches each question to the arbiter of its spec type: taint -(source to sink with no discharging sanitizer), dominance (does a guard govern the obligated -operation) and configuration (constant abstraction over a value). Taint facts ship for C, -Java, JavaScript, PHP and Python (`ruleset/semantic/*.toml`). Without Joern the layer records -`cpg_unavailable` and the scan is otherwise unchanged; the Docker image ships Joern and -`php-cli` (which the PHP frontend needs). The representations and techniques behind each stage -are explained in [Detection techniques](detection-techniques.md). +graph (CPG: a graph of the program's syntax, control flow and data flow) per scan with Joern. +It runs `joern-parse`, then batched `joern --script` queries from `cpg/queries/*.sc`. Each +question goes to the arbiter (a checker that decides one kind of question) for its spec type: + +- taint (untrusted input reaching a dangerous call): source to sink with no discharging sanitizer; +- dominance: does a guard govern the obligated operation; +- configuration: constant abstraction over a value. + +Taint facts ship for C, Java, JavaScript, PHP and Python (`ruleset/semantic/*.toml`). Without +Joern, the layer records `cpg_unavailable` and the rest of the scan is unchanged. The Docker +image ships Joern and `php-cli`, which the PHP frontend needs. [Detection +techniques](detection-techniques.md) explains the representations and techniques behind each +stage. The model layer reports on a rung ladder: @@ -47,23 +60,29 @@ model_corroborated the graph found the shape; a judge agreed on the residual suspicion proposed, nothing corroborated it ``` -The LLM (`model/pipeline.py`, `residual_question`) is asked only the residual question the graph -cannot settle, for a flow the graph already found; it can confirm or contradict, never create a -flow. When the graph resolved nothing, the enumerator's candidates (at most -`MAX_JUDGED_CANDIDATES`, 8) are put to it, and anything it affirms is reported at `suspicion`. -Without a provider key neither question is asked and only the graph's entailed findings are -reported. +The LLM answers only the residual question: the part the graph cannot settle +(`model/pipeline.py`, `residual_question`). It is asked only about a flow the graph already +found. It can confirm or contradict that flow, but never create one. When the graph resolved +nothing, the LLM sees the enumerator's candidates (at most `MAX_JUDGED_CANDIDATES`, 8). +Anything it affirms is reported at `suspicion`. Without a provider key, neither question is +asked. Only the graph's entailed findings are reported then. **REGRESS** (`regress/`, `sandbox/`) runs only when `docker info` succeeds. Each promoted -candidate gets a small language recipe (Python, JavaScript, C) that loads the candidate's -source inside a container with no network, a read-only `/workspace`, a non-root user and the -`[sandbox]` limits. A structural deny-list rejects snippets mentioning sockets, curl, host -networking or writes under `/workspace` before anything runs. +candidate gets a small language recipe (Python, JavaScript, C). The recipe loads the +candidate's source inside a container with: + +- no network; +- a read-only `/workspace`; +- a non-root user; +- the `[sandbox]` limits. + +Before anything runs, a structural deny-list rejects snippets that mention sockets, curl, host +networking or writes under `/workspace`. ## Evidence ladder: how a false positive is eliminated Evidence level is a state machine, not a label a model may assign. The verifier -(`verification.py`) runs on independent context and enforces: +(`verification.py`) runs on independent context and enforces it: ``` suspicion ─▶ static_corroboration ─▶ crash_reproduced ─▶ root_cause_explained @@ -75,31 +94,36 @@ suspicion ─▶ static_corroboration ─▶ crash_reproduced ─▶ root_cause_ call-graph, route, CLI, parser or dynamic evidence; - `static_corroboration` with function-level reachability: `ACCEPTED`. -A rejected finding becomes a **scoped false-positive learning** (`calibration.py`) with a -reason from a fixed taxonomy (`unreachable_path`, `missing_attacker_control`, +A rejected finding becomes a **scoped false-positive learning** (`calibration.py`). It has a +scope and a reason from a fixed taxonomy: `unreachable_path`, `missing_attacker_control`, `sanitizer_disproved`, `static_rule_mismatch`, `incorrect_model_assumption`, `duplicate`, -`insufficient_impact`, `unsupported`, `contradicted`, `unverified`) and a scope. A rejection in -`auth/` demotes only `auth/...`, never the whole class +`insufficient_impact`, `unsupported`, `contradicted`, `unverified`. A rejection in `auth/` +demotes only `auth/...`, never the whole class (`tests/test_calibration.py::test_scoped_false_positive_demotion_does_not_suppress_class_globally`). ## How a false negative is surfaced -Every expected vulnerability that no finding matched becomes a `BenchmarkCalibrationRecord` in -`calibration_records.json`, with a `next_improvement_candidate` (a static rule, SARIF source, -entry-point mapping, retrieval package, hunter prompt, dynamic reproducer or skill route). -Once a miss is closed the detection gate keeps it closed in CI. +An expected vulnerability that no finding matched becomes a `BenchmarkCalibrationRecord` in +`calibration_records.json`. Each record names a `next_improvement_candidate`: a static rule, +SARIF source, entry-point mapping, retrieval package, hunter prompt, dynamic reproducer or +skill route. Once a miss is closed, the detection gate keeps it closed in CI. ## Central CWE policy and the project score **Policy** (`policy/verycode.py`): a vendored `CWE_Score.tsv` (verycode-policies) is the single -source of truth. Each CWE carries a flaw category, a severity (0-5) and `static`/`dynamic` scope -flags. A finding's severity comes only from policy, keyed on CWE. CWEs that are not `static` -resolve to 0 and are report-only. `assert_rules_resolve` runs as the `policy_check` stage: an -enabled rule naming an ungoverned CWE aborts the scan. Verycode has no CWE-120, so the C/C++ +source of truth. Each CWE carries: + +- a flaw category; +- a severity (0-5); +- `static`/`dynamic` scope flags. + +A finding's severity comes only from policy, keyed on CWE. CWEs that are not `static` resolve +to 0 and are report-only. `assert_rules_resolve` runs as the `policy_check` stage. An enabled +rule that names an ungoverned CWE aborts the scan. Verycode has no CWE-120, so the C/C++ buffer rules carry CWE-121. **Score** (`scoring/project_score.py`): each finding's penalty is -`SEV_WEIGHT[severity] x REACH_MULT[reachability]`, and the score is `100 * e^(-total/k)` with +`SEV_WEIGHT[severity] x REACH_MULT[reachability]`. The score is `100 * e^(-total/k)` with `k = 60`. | Severity | Weight | | Reachability | Multiplier | @@ -109,10 +133,13 @@ buffer rules carry CWE-121. | 3 | 10 | | `unknown` | 0.4 | | 2 / 1 / 0 | 2 / 1 / 0 | | | | -A confirmed false positive lowers a finding's reachability multiplier instead of deleting the -rule. A severity-5 reachable finding always fails the score gate; `score < min_score` -(default 80) fails it only with `[score] blocking = true`. The `score` stage writes -`score.json` and merges it into `manifest.json`. +A confirmed false positive lowers a finding's reachability multiplier. The rule is not +deleted. The score gate works like this: + +- a severity-5 reachable finding always fails it; +- `score < min_score` (default 80) fails it only with `[score] blocking = true`. + +The `score` stage writes `score.json` and merges it into `manifest.json`. ## Self-improvement @@ -124,41 +151,61 @@ scan ─▶ verify ─▶ non-accepted findings ─▶ scoped learnings next scan ─▶ rank ─▶ calibrate (demote rejected scopes) ─▶ findings ``` -`record_calibration` (after `verify`) merges this run's non-accepted outcomes into the ledger, -de-duplicated by finding ID; `calibrate` (after `rank`) demotes the priority of every scope -that produced a rejection and writes `applied_calibrations.json`. Demotion is scoped and -reversible (`tests/test_pipeline_calibration.py`). +1. `record_calibration` runs after `verify`. It merges this run's non-accepted outcomes into + the ledger, de-duplicated by finding ID. +2. `calibrate` runs after `rank`. It demotes the priority of every scope that produced a + rejection and writes `applied_calibrations.json`. + +Demotion is scoped and reversible (`tests/test_pipeline_calibration.py`). ### Rule-level loop: `ousast improve` -Each round (`improve/evolve.py`): benchmark with the current ledger, per-rule signals -(`rule_signals.json`), bounded proposals, validation (`improve/validator.py`), replay smoke, -re-benchmark, and a hard gate. The levers are rule status (`enabled -> shadow` before -`disabled`) and score constants; the loop can never edit pattern text -or the authoritative CWE severity. A round is accepted only if recall >= 90%, FP < 10%, the -project score did not regress and no matched finding was lost; otherwise the ledger is not -written. With `--pair-catalog`, every round is also gated per provenance profile on the -catalog's holdout pairs (`--no-pair-gate` skips this and is not for accepted ledgers). The same -detection gate runs in CI (`python -m openultrasast.gate`). +Each round (`improve/evolve.py`) runs these steps: + +1. benchmark with the current ledger; +2. per-rule signals (`rule_signals.json`); +3. bounded proposals; +4. validation (`improve/validator.py`); +5. replay smoke; +6. re-benchmark; +7. a hard gate. + +The levers are rule status (`enabled -> shadow` before `disabled`) and score constants. The +loop can never edit pattern text or the authoritative CWE severity. A round is accepted only if +all of these hold: + +- recall >= 90%; +- FP < 10%; +- the project score did not regress; +- no matched finding was lost. + +Otherwise the ledger is not written. With `--pair-catalog`, each round is also gated per +provenance profile on the catalog's holdout pairs. `--no-pair-gate` skips this check; do not +use it for accepted ledgers. The same detection gate runs in CI (`python -m openultrasast.gate`). ### Proposals from plane memory -With `--memory [STORE]`, `ousast improve` also reads the rows plane runs stored per repository -and pin (`improve/memory.py`). Two deterministic rules turn them into rule-status proposals: a -rule whose alerts are repeatedly false across repositories is shadowed, and a shadow rule that -keeps hitting agreed, declared vulnerable sites is re-enabled. Rows from holdout pairs, from the -gated manifest's own cases and from any `--qualify-population` are dropped before any rule sees -them. Each proposal records the rows it came from (`memory_proposals.jsonl` next to the -journal) and goes through the same validator and gate. +With `--memory [STORE]`, `ousast improve` also reads the rows that plane runs stored per +repository and pin (`improve/memory.py`). Two deterministic rules turn these rows into +rule-status proposals: + +- a rule whose alerts are repeatedly false across repositories is shadowed; +- a shadow rule that keeps hitting agreed, declared vulnerable sites is re-enabled. + +Some rows are dropped before any rule sees them: rows from holdout pairs, from the gated +manifest's own cases and from any `--qualify-population`. Each proposal records the rows it +came from (`memory_proposals.jsonl` next to the journal). It then goes through the same +validator and gate. ### The loop as a plane Run `ousast plane workspaces --validation-set ... --loop` appends the improvement loop -to a generated Run: `alerts` per case (the quick scan on the vulnerable and fixed pins), then -the steps `snapshot` (task `memory-snapshot`), `measure`, `propose` and `improve` (`plane/tasks/loop.py`), each model-free -with a zero budget. The snapshot applies the same train-on-test guard before any row leaves the -store. A recorded live run: `benchmarks/measurements/2026-09-30-harnessx-removal-loop/`. +to a generated Run. The Run holds `alerts` per case: the quick scan on the vulnerable and fixed +pins. Then come the steps `snapshot` (task `memory-snapshot`), `measure`, `propose` and +`improve` (`plane/tasks/loop.py`). Each step is model-free with a zero budget. The snapshot +applies the same train-on-test guard before any row leaves the store. A recorded live run: +`benchmarks/measurements/2026-09-30-harnessx-removal-loop/`. -For PHP, which quick mode covered with zero rules when that run was made, `ousast plane +When that run was made, quick mode covered PHP with zero rules. So for PHP, `ousast plane alerts-engine` produces a Run's `alerts` from the engine image on the host instead (`benchmarks/measurements/2026-09-30-php-engine-alerts/`). diff --git a/docs/decision-engine.md b/docs/decision-engine.md index 83f4e72..c63a696 100644 --- a/docs/decision-engine.md +++ b/docs/decision-engine.md @@ -1,51 +1,77 @@ # The learned decision engine (in development) -**Status 2026-10-02: in development, not adopted.** No scan, pre-push check or report uses it; -every user-facing path is today's deterministic one until a compiled program is adopted. The -design is the `learned-decision-engine` specification (maintainer tooling); the code is in +**Status 2026-10-02: in development, not adopted.** No scan, pre-push check or report uses it. +Every user-facing path stays on today's deterministic one until a compiled program is adopted. +The design is the `learned-decision-engine` specification (maintainer tooling). The code is in `src/openultrasast/learn/`. ## Why -Every detection gain before the engine was a hand edit tuned on the cases that exposed it, and -engine tuning measured on population v1 did not transfer to v2. The reasoning, and the step-by-step -chain that replaced hand edits, are on [Where we stand](where-we-stand.md#2-our-approach-to-detection-step-by-step). +Before the engine, every detection gain was a hand edit. Each edit was tuned on the cases that +exposed the miss. Engine tuning measured on population v1 did not transfer to v2. [Where we +stand](where-we-stand.md#2-our-approach-to-detection-step-by-step) gives the reasoning and the +step-by-step chain that replaced hand edits. ## Design -- **Signals, not verdicts.** Every instrument (quick rules, the Joern engine, model sink and - role classification, the plane's verify passes and agreement, repository facts such as - callers) contributes a feature record per candidate `(path, function, family)`, including an - explicit state when it did not run or failed. None decides alone. +- **Signals, not verdicts.** Every instrument contributes a feature record per candidate + `(path, function, family)`. The instruments are quick rules, the Joern engine, model sink and + role classification, the plane's verify passes and agreement, and repository facts such as + callers. The record includes an explicit state when an instrument did not run or failed. No + instrument decides alone. - **Labels from ground truth.** `ousast learn labels` derives labels only from sources on a - fail-closed allow-list (`learn/sources.toml`): the vulnerable and fixed sides of pairs, the - declared sites and fix ranges of spent populations, benign deltas and recorded adjudications. + fail-closed allow-list (`learn/sources.toml`). Fail-closed means a source not on the list is + never used. The sources are: + - the vulnerable and fixed sides of pairs; + - the declared sites and fix ranges of spent populations; + - benign deltas; + - recorded adjudications. + "Rejected by the plane" is never a negative. Reserved populations (v3) are not on the list. -- **An AI classifier with local memory, compiled in house in the DSPy style.** The program is an - instruction, a few demonstrations and up to six retrieved labelled examples, then the - candidate's code excerpt and signals; the model (`deepseek-flash`) answers - vulnerable / not vulnerable / unsure with a rationale, sampled `k` times, giving a score `s`. - Compilation bootstraps demonstrations and searches instructions on a compile split, scored by - a pre-registered balanced Brier score (`learn/compile.toml`). -- **Retrieval** from the memory store: nearest neighbours by Gower distance on the signal - profile, re-ranked by cosine similarity of code embeddings (OpenAI `text-embedding-3-small` - through OpenRouter, cached by excerpt hash), under a label/family/repository balance rule. An - evaluation-boundary filter keeps a candidate's own repository group out of what it is shown. +- **An AI classifier with local memory, compiled in house in the DSPy style.** The program has + an instruction, a few demonstrations and up to six retrieved labelled examples. Then come the + candidate's code excerpt and signals. The model (`deepseek-flash`) answers vulnerable / not + vulnerable / unsure with a rationale. It is sampled `k` times, which gives a score `s`. + Compilation bootstraps demonstrations and searches instructions on a compile split. A + pre-registered balanced Brier score (`learn/compile.toml`) scores the candidates. The Brier score + measures how far predicted probabilities are from the true labels. +- **Retrieval** from the memory store works in three steps: + 1. find nearest neighbours by Gower distance (a distance over mixed feature types) on the + signal profile; + 2. re-rank them by cosine similarity of code embeddings (OpenAI `text-embedding-3-small` + through OpenRouter, cached by excerpt hash); + 3. apply a label/family/repository balance rule. + + An evaluation-boundary filter keeps a candidate's own repository group out of what it is + shown. - **Repository-grouped evaluation.** The compile split (25% of repository groups) is never - evaluated. The rest is evaluated by outer grouped 5-fold cross-validation: the Platt - calibration map is cross-fitted on the other folds and the operating points are chosen on the - other folds (nested). One compile per profile, not one per fold. Intervals are 95% - repository-cluster bootstrap. -- **Two operating points.** BLOCK is offered only when calibration holds (slope CI contains 1, - intercept CI contains 0, ECE <= 0.05) and a fold reaches the required precision; otherwise the - engine is ADVISORY only. An `unsure` answer can never BLOCK. + evaluated. The rest is evaluated by outer grouped 5-fold cross-validation: + - the Platt calibration map (which turns the score into a probability) is cross-fitted on + the other folds; + - the operating points are chosen on the other folds (nested). + + There is one compile per profile, not one per fold. Intervals are 95% repository-cluster + bootstrap. +- **Two operating points.** BLOCK is offered only when both of these hold: + - calibration holds: slope CI contains 1, intercept CI contains 0, ECE <= 0.05; + - a fold reaches the required precision. + + Otherwise the engine is ADVISORY only. An `unsure` answer can never BLOCK. ECE (expected + calibration error) is the average gap between predicted and observed rates. ## First measured slice (2026-10-01) Record: `benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json` (counts -only). Family **injection** only; memory of 724 static examples from all families; input profile -v1 with the engine signals and function length withheld (a leak audit found them separating the -sides of a pair by construction: `2026-10-01-decision-engine-leak-audit/`); framework priors off. +only). The setup: + +- family **injection** only; +- memory of 724 static examples from all families; +- input profile v1, with the engine signals and function length withheld; +- framework priors off. + +The signals were withheld because a leak audit found them separating the sides of a pair by +construction (`2026-10-01-decision-engine-leak-audit/`). AUC below is the chance the engine +ranks a real bug above a fixed one. | Measure | Value (95% repository-cluster CI) | | --- | --- | @@ -62,11 +88,17 @@ sides of a pair by construction: `2026-10-01-decision-engine-leak-audit/`); fram ## Second slice: six families (2026-10-02) Record: `benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json` (counts only). -After the advisory-fix harvest the memory holds 904 static examples (724 before). One program was -compiled per evaluable family (injection recompiled, since the memory changed) and evaluated on -that family's own candidates, out of repository as above: compile split never evaluated, outer -grouped 5-fold, calibration cross-fitted, operating points nested, 95% repository-cluster -bootstrap intervals (2,000 resamples); input profile v1, priors off, k = 5. +After the advisory-fix harvest, the memory holds 904 static examples (724 before). + +One program was compiled per evaluable family. Injection was recompiled, since the memory +changed. Each program was evaluated on its family's own candidates, out of repository as above: + +- compile split never evaluated; +- outer grouped 5-fold; +- calibration cross-fitted; +- operating points nested; +- 95% repository-cluster bootstrap intervals (2,000 resamples); +- input profile v1, priors off, k = 5. | Family | Candidates (groups; positive) | Pooled AUC of `s` (95% CI) | Within-pair AUC (paired groups) | ADVISORY recall / precision | Canary agreement | | --- | --- | --- | --- | --- | --- | @@ -77,45 +109,53 @@ bootstrap intervals (2,000 resamples); input profile v1, priors off, k = 5. | path | 35 (21; 21) | 0.89 (0.79 - 0.97) | 0.83 (14) | 0.86 / 0.69 | 0.90 | | untrusted destination | 62 (37; 38) | 0.85 (0.76 - 0.94) | 0.72 (18) | 0.84 / 0.73 | 0.87 | -- **BLOCK is offered for no family.** Calibration holds only for output encoding (ECE 0.045), and - there no fold reaches the required precision (`precision_unreachable`: no fold reaches a Wilson - lower bound of 0.95). For the other five families calibration does not hold (ECE 0.071 to 0.119). -- **The pooled AUC is the more favourable number.** Every family's pool includes candidates that - carry only a positive label, with no fixed-side counterpart (7 to 19 per family; there are no - single-label negatives), so the pooled AUC is inflated by positives that never had to be told - apart from their own fix. The within-pair AUC over the paired groups, 0.72 to 0.93, is the - stricter reading (`paired_groups_check` in the record). +- **BLOCK is offered for no family.** Calibration holds only for output encoding (ECE 0.045). + There, no fold reaches the required precision (`precision_unreachable`: no fold reaches a + Wilson lower bound of 0.95). For the other five families, calibration does not hold (ECE + 0.071 to 0.119). +- **The pooled AUC is the more favourable number.** Every family's pool includes candidates + that carry only a positive label, with no fixed-side counterpart (7 to 19 per family; there + are no single-label negatives). These positives never had to be told apart from their own + fix, so they inflate the pooled AUC. The within-pair AUC over the paired groups, 0.72 to + 0.93, is the stricter reading (`paired_groups_check` in the record). - **Re-run agreement (canary, 30 candidates per family) is below 0.9 for three families:** injection 0.77, access control 0.83, untrusted destination 0.87. -- **One memory-size point for injection** (paired: the same program, candidates and folds, with a - retrieval memory of 724 vs 904 examples): AUC 0.76 vs 0.77, within the intervals. No injection - example was added, so the new examples reach an injection prompt only as contrast. -- **Spend:** $2.88 metered at list price for DeepSeek under an $8 ceiling, $0.0009 for - embeddings; $0.0027 to $0.0032 per evaluated candidate. The access-control evaluation replayed - from the response cache at $0 with identical predictions. -- **An instrument fix is recorded:** compiled demonstrations ignored the near-duplicate rule - (cosine >= 0.98) until the pre-call boundary assertion stopped the untrusted-destination - evaluation; demonstrations are now swapped or dropped like retrieved examples, and the stopped - part was replayed after the fix. +- **One memory-size point for injection.** This run is paired: the same program, candidates + and folds, with a retrieval memory of 724 vs 904 examples. AUC was 0.76 vs 0.77, within the + intervals. No injection example was added. So the new examples reach an injection prompt + only as contrast. +- **Spend:** $2.88 metered at list price for DeepSeek under an $8 ceiling, and $0.0009 for + embeddings. That is $0.0027 to $0.0032 per evaluated candidate. The access-control + evaluation replayed from the response cache at $0 with identical predictions. +- **An instrument fix is recorded.** Compiled demonstrations ignored the near-duplicate rule + (cosine >= 0.98). The pre-call boundary assertion caught this and stopped the + untrusted-destination evaluation. Demonstrations are now swapped or dropped like retrieved + examples. The stopped part was replayed after the fix. -**Not adopted.** What is still ahead is listed once, on -[Where we stand](where-we-stand.md#5-open-items). +**Not adopted.** [Where we stand](where-we-stand.md#5-open-items) lists, in one place, what is +still ahead. ## Experiments Changes to prompts, models, pass counts, features or sampling are compared by controlled -experiments from a pre-registered manifest (`plane/experiments/.yaml`), so nothing is adopted -on one run's number. Two are registered. exp-001 (known callers) has not run. exp-002 (a retrieval -ensemble against today's temperature sampling) ran on 160 paired candidates in 56 repository -groups and was **rejected**: within-pair AUC 0.815 against 0.741 and $0.0021 against $0.0051 per -candidate, with re-run agreement 0.875 against 0.906 and an interval spanning zero -(`benchmarks/experiments/exp-002-retrieval-ensemble/result.json`). The reading is on -[Where we stand](where-we-stand.md#2-our-approach-to-detection-step-by-step). +experiments. Each experiment comes from a pre-registered manifest +(`plane/experiments/.yaml`). This way nothing is adopted on one run's number. + +Two experiments are registered: + +- exp-001 (known callers) has not run. +- exp-002 compared a retrieval ensemble against today's temperature sampling. It ran on 160 + paired candidates in 56 repository groups and was **rejected**. Within-pair AUC was 0.815 + against 0.741 and the cost was $0.0021 against $0.0051 per candidate. The record shows re-run agreement + 0.875 against 0.906 and an interval spanning zero + (`benchmarks/experiments/exp-002-retrieval-ensemble/result.json`). + +The reading is on [Where we stand](where-we-stand.md#2-our-approach-to-detection-step-by-step). ## Commands `ousast learn` is a maintainer command group over the engine's data. Paid steps take -`--budget-usd`; `--replay-only` answers every call from the response cache at no cost. +`--budget-usd`. `--replay-only` answers every call from the response cache at no cost. | Command | Does | | --- | --- | @@ -130,5 +170,5 @@ candidate, with re-run agreement 0.875 against 0.906 and an interval spanning ze The memory store is the plane's (`OUSAST_MEMORY`; see [ops/ax/README.md](ops/ax/README.md)). The harvests that produced the plane signals ran as plane Runs generated by -`ousast plane harvest` (`benchmarks/measurements/2026-10-01-decision-engine-harvest/record.json` -and, for the advisory-fix pairs, `2026-10-02-decision-engine-harvest-advisory/record.json`). +`ousast plane harvest`. Records: `benchmarks/measurements/2026-10-01-decision-engine-harvest/record.json` +and, for the advisory-fix pairs, `2026-10-02-decision-engine-harvest-advisory/record.json`. diff --git a/docs/detection-techniques.md b/docs/detection-techniques.md index ceb6163..099c441 100644 --- a/docs/detection-techniques.md +++ b/docs/detection-techniques.md @@ -1,17 +1,22 @@ # Detection techniques -This page explains the program analysis under a scan: the three representations of a program the -tool builds, what each one can and cannot see, and the techniques that run over them (taint, guard -dominance, constant abstraction, candidate enumeration, sandboxed reproduction). It is written for -readers who know static analysis. Every statement names the code it comes from -(`src/openultrasast/` paths, `file:line`) and every number names a committed record under +This page explains the program analysis under a scan. It covers: + +- the three representations of a program the tool builds; +- what each one can and cannot see; +- the techniques that run over them: taint, guard dominance, constant abstraction, candidate + enumeration and sandboxed reproduction. + +It is written for readers who know static analysis. Every statement names the code it comes +from (`src/openultrasast/` paths, `file:line`). Every number names a committed record under `benchmarks/measurements/` or `benchmarks/independent/`. The scan modes and commands are in -[Scanning](scanning.md); the stage order is in [Architecture](architecture.md). +[Scanning](scanning.md). The stage order is in [Architecture](architecture.md). ## 1. Three representations, three horizons -A finding is only as good as what its representation can see. The tool keeps three, from cheap to -expensive, and records which one produced each finding rather than averaging them. +A finding is only as good as what its representation can see. The tool keeps three +representations, from cheap to expensive. It records which one produced each finding instead +of averaging them. ```mermaid flowchart LR @@ -30,90 +35,125 @@ flowchart LR ### Text: quick rules Quick rules are Python regexes applied to the whole file text with `finditer` -(`findings.py:58-68`); a rule's line number is computed afterwards from the match offset -(`findings.py:103`). Rules live in `ruleset//rules.toml` as `[[rule]]` tables with -`rule_id`, `cwe`, `pattern`, `status` (`enabled`, `shadow`, `disabled`), `min_evidence_level`, -`precision_estimate` and optional `framework`/`library`/`fallback_for` tags (`ruleset/store.py:12, +(`findings.py:58-68`). A rule's line number is computed afterwards from the match offset +(`findings.py:103`). + +Rules live in `ruleset//rules.toml` as `[[rule]]` tables. Each has these fields: + +- `rule_id`, `cwe`, `pattern`; +- `status` (`enabled`, `shadow`, `disabled`); +- `min_evidence_level`, `precision_estimate`; +- optional `framework`/`library`/`fallback_for` tags. + +The fields are defined in `ruleset/store.py` (`ruleset/store.py:12, 19-41`). The self-improvement loop may change only `status`, `min_evidence_level` and -`precision_estimate`; the pattern is "human-PR-only" (`store.py:23-25`). There are 52 rules: C 6, -Groovy 1, Java 4, JavaScript 12, Python 11, PHP 18, of which 6 PHP rules are `shadow` (written to -`shadow_findings.json` and removed from the report, `cli.py:538-543`). +`precision_estimate`. The pattern is "human-PR-only" (`store.py:23-25`). There are 52 rules: +C 6, Groovy 1, Java 4, JavaScript 12, Python 11, PHP 18, of which 6 PHP rules are `shadow`. +Shadow rules are written to `shadow_findings.json` and removed from the report +(`cli.py:538-543`). -What a regex sees is one statement. The patterns say so: most are anchored with `[^\n]*`, and the -PHP header explains that each rule pairs a sink from the engine's vocabulary with "the shape that +A regex sees one statement. The patterns say so: most are anchored with `[^\n]*`. The PHP +header explains that each rule pairs a sink from the engine's vocabulary with "the shape that makes it dangerous WITHOUT flow -- a request superglobal on the sink's own line, or a query literal -composed with a variable" (`ruleset/php/rules.toml:1-15`). A few PHP rules reach further -(`php-sql-sprintf-unescaped` looks up to 2000 characters ahead for the closing `);`, -`php/rules.toml:102-116`), but none follows a value across statements. Comment suppression is also -line-local (`findings.py:186-205`). +composed with a variable" (`ruleset/php/rules.toml:1-15`). A few PHP rules reach further. +`php-sql-sprintf-unescaped` looks up to 2000 characters ahead for the closing `);` +(`php/rules.toml:102-116`). But no rule follows a value across statements. Comment suppression +is also line-local (`findings.py:186-205`). -The measured consequence is in `benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`: -on the PHP development corpus the rules hit the known site on the vulnerable pin in 4 of 5 cases -(pmpro, ultimate-member, phpmyfaq, yeswiki) and missed `wp-learnpress-sqli` on both pins -(`cases.wp-learnpress-sqli.vulnerable.at_known_site = []`). The record does not state the cause. -The case description (`benchmarks/independent/population-v1.toml:54-58`) places the sink at an -`implode` in `class-lp-db.php:607` with the fix on the source side in another file, and the PHP -fixture in the same record carries the general limit explicitly: its missed CWE-89 is "query built -from a request value on an earlier line (not visible to a line rule)" -(`fixtures.php-vulnerable.missed`). Precision on that corpus is a lower bound per rule -(`rules..precision_lower_bound`): 0.032 for `php-wpdb-sql-composition`, 0.034 for -`php-sql-unquoted-concat`, 0.109 for `php-sql-sprintf-unescaped`, 0.129 for -`php-sql-unquoted-interpolation`. Every number in that file is in-sample (`provenance.in_sample`). +The measured result is in `benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`. +On the PHP development corpus, the rules hit the known site on the vulnerable pin in 4 of 5 +cases (pmpro, ultimate-member, phpmyfaq, yeswiki). They missed `wp-learnpress-sqli` on both pins +(`cases.wp-learnpress-sqli.vulnerable.at_known_site = []`). + +The record does not state the cause of that miss. Two facts point at it: + +- The case description (`benchmarks/independent/population-v1.toml:54-58`) places the sink at + an `implode` in `class-lp-db.php:607`. The fix is on the source side, in another file. +- The PHP fixture in the same record states the general limit. Its missed CWE-89 is "query built + from a request value on an earlier line (not visible to a line rule)" + (`fixtures.php-vulnerable.missed`). + +Precision on that corpus is a lower bound per rule (`rules..precision_lower_bound`): + +- 0.032 for `php-wpdb-sql-composition`; +- 0.034 for `php-sql-unquoted-concat`; +- 0.109 for `php-sql-sprintf-unescaped`; +- 0.129 for `php-sql-unquoted-interpolation`. + +Every number in that file is in-sample (`provenance.in_sample`). ### Tree: the CST and the IR -The `semantic` extra installs tree-sitter grammars for Python, JavaScript, TypeScript and TSX, C, -C++ and Java (`pyproject.toml:25-34`; grammar table `semantic/extra.py:299-307`). PHP and Groovy -have no grammar: `ir.parse_file` returns `language_unsupported` for them (`semantic/ir.py:56-58`), -and PHP data flow is owned by the engine (`ruleset/php/rules.toml:3`). Without the extra, Python -falls back to the standard-library `ast` and every other language is `unadjudicated` -(`ir.py:50-67`, `semantic/overlay.py:20`). +A CST (concrete syntax tree) is the full parse tree of a file. An IR (intermediate +representation) is a smaller summary built from it. -The CST is reduced to a small IR per function (`semantic/ir.py:10-47`): `Bind(name, line, -value_text, is_constant, names, call_name)`, `CallSite(name, line, arg_texts, arg_is_constant, -arg_names, extra_arg_is_sequence)`, `FunctionIR(name, params, start_line, end_line, binds, calls)` -and `FileIR`. There is no control-flow graph, no guard record and no return record. A synthetic -`` function holds top-level statements (`semantic/cst.py:117-128`); nested functions are -skipped during a function's walk (`cst.py:146-165`). The literal set `_CONST` and the per-language +The `semantic` extra installs tree-sitter grammars for Python, JavaScript, TypeScript and TSX, +C, C++ and Java (`pyproject.toml:25-34`; grammar table `semantic/extra.py:299-307`). PHP and +Groovy have no grammar. For them, `ir.parse_file` returns `language_unsupported` +(`semantic/ir.py:56-58`), and the engine owns PHP data flow (`ruleset/php/rules.toml:3`). +Without the extra, Python falls back to the standard-library `ast`. Every other language is then +`unadjudicated` (`ir.py:50-67`, `semantic/overlay.py:20`). + +The CST is reduced to a small IR per function (`semantic/ir.py:10-47`): + +- `Bind(name, line, value_text, is_constant, names, call_name)`; +- `CallSite(name, line, arg_texts, arg_is_constant, arg_names, extra_arg_is_sequence)`; +- `FunctionIR(name, params, start_line, end_line, binds, calls)`; +- `FileIR`. + +There is no control-flow graph, no guard record and no return record. A synthetic `` +function holds top-level statements (`semantic/cst.py:117-128`). Nested functions are skipped +during a function's walk (`cst.py:146-165`). The literal set `_CONST` and the per-language `assign` and `function` node kinds are at `cst.py:32-84`. Three consumers read the IR: -- **The overlay** (`semantic/taint.py`, `semantic/overlay.py`) runs an intra-function, order- - insensitive fixpoint of at most eight rounds over the binds (`taint.py:86-96`), with parameters - taken as tainted (`taint.py:84-85`) and sources matched as substrings of a bind's text - (`taint.py:222-227`). A sink call with a tainted argument becomes a `TaintPath`; a sink call that - matches a sanitizer shape (`parameterized`, a constant `literal_format_arg`, or a sanitizer call) - or has only constant arguments becomes a `Demotion`; anything else is `flow_incomplete` - (`taint.py:123-177`). The overlay then re-labels quick findings as `promote`, `demote`, - `unadjudicated` or adds `coverage` records for sinks no rule inventoried (`overlay.py:19-20, - 94-122, 203-267`). Demotions are never proof: `prove_filter.py` keeps only promotions for the - REGRESS stage, and "unadjudicated is not negative proof" (`prove_filter.py:1-31`). +- **The overlay** (`semantic/taint.py`, `semantic/overlay.py`) runs a fixpoint over the binds. + The fixpoint is intra-function and order-insensitive, with at most eight rounds + (`taint.py:86-96`). Parameters are taken as tainted (`taint.py:84-85`). Sources are matched + as substrings of a bind's text (`taint.py:222-227`). Each sink call gets one outcome + (`taint.py:123-177`): + - a sink call with a tainted argument becomes a `TaintPath`; + - a sink call that matches a sanitizer shape (`parameterized`, a constant + `literal_format_arg`, or a sanitizer call) or has only constant arguments becomes a + `Demotion`; + - anything else is `flow_incomplete`. + + The overlay then re-labels quick findings as `promote`, `demote` or `unadjudicated`. It also + adds `coverage` records for sinks no rule inventoried (`overlay.py:19-20, + 94-122, 203-267`). Demotions are never proof: `prove_filter.py` keeps only promotions for the REGRESS stage, and + "unadjudicated is not negative proof" (`prove_filter.py:1-31`). - **Obligations** (`semantic/obligations/`), covered in section 4. - **The candidate enumerator** (`model/candidates.py`), covered in section 6. -The IR's limits were measured on pair corpora in September 2026 and drove the order of later work: -C `init_declarator` is not an assign kind, so `int n = x;` never binds; field stores clobber the -base name's taint; `weak_literals` and `format_arg` are loaded but not consulted by this path; and -without a guard record a NULL or bounds check cannot be a sanitizer here. Those gaps are why guard -reasoning lives in the graph (section 3), not the tree. +The IR's limits were measured on pair corpora in September 2026. They drove the order of later +work: + +- C `init_declarator` is not an assign kind, so `int n = x;` never binds; +- field stores clobber the base name's taint; +- `weak_literals` and `format_arg` are loaded but not consulted by this path; +- without a guard record, a NULL or bounds check cannot be a sanitizer here. + +Those gaps are why guard reasoning lives in the graph (section 3), not the tree. ### Graph: the Joern code property graph -The CPG is the only representation that follows a value across statements, calls and files. It is -built with Joern and queried in Scala; its use is the subject of section 2. +The CPG is the only representation that follows a value across statements, calls and files. It +is built with Joern and queried in Scala. Section 2 covers how it is used. ## 2. Joern: why a code property graph, and how this tool uses one -The engine exists because the first two representations cannot answer "does this request value -reach this query". A regex sees a statement; the IR sees a function with no control flow. Joern's -CPG unifies AST, control flow, dominance and data dependence into one graph, and its `dataflowOss` -overlay answers `reachableByFlows` queries interprocedurally (`cpg/backend.py:117`). Every taint, -dominance and configuration question in the `standard` and `deep` modes is a CPGQL script run over -that graph. +The engine exists because the first two representations cannot answer one question: "does this +request value reach this query". A regex sees a statement. The IR sees a function with no +control flow. Joern's CPG unifies AST, control flow, dominance and data dependence into one +graph. Its `dataflowOss` overlay answers `reachableByFlows` queries interprocedurally (across +function calls) (`cpg/backend.py:117`). Every taint, dominance and configuration question in +the `standard` and `deep` modes is a CPGQL script run over that graph. ### Frontends and their known defects +A frontend is the Joern parser that turns one language into a CPG. + | Language | Frontend (`cpg/backend.py:73-81`) | Notes | | --- | --- | --- | | PHP | `php2cpg` | Built directly, not through `joern-parse` (`backend.py:97, 607-628`). On one WordPress slice `joern-parse` produced 5 usable builds of 8 and reported success on empty graphs; `php2cpg` produced 11 of 12 and names the files it dropped (`backend.py:85-90`). Files with a `global` inside a closure are excluded for php2cpg before 4.0.625 (`backend.py:124-149`); joernio/joern#6281 is the graph bug fixed in 4.0.625 (`2026-09-20-php-runtime-after-graph-fix.json`, `.the_graph_bug_is_fixed`). The frontend needs a PHP interpreter; the image installs `php-cli` (`Dockerfile:17-21`). | @@ -123,8 +163,8 @@ that graph. | C, C++ | `c2cpg` | Adds `c_bounds_arithmetic_unmodeled`; the `memory` family models calls, not arithmetic (`ruleset/families.toml:91`). | Joern is pinned at `v4.0.625` in the image (`Dockerfile:14`, `docker-compose.yml:15`, -`ops/joern.py`), on `eclipse-temurin:21-jre-noble` (`Dockerfile:12`). The decision to leave the -pin there after the graph fix is recorded in `2026-09-20-php-runtime-after-graph-fix.json` +`ops/joern.py`), on `eclipse-temurin:21-jre-noble` (`Dockerfile:12`). The pin stayed there after +the graph fix. That decision is recorded in `2026-09-20-php-runtime-after-graph-fix.json` (`.on_the_engine_pin.decision`). ### Our approach @@ -144,66 +184,94 @@ flowchart TD run -- "timeout or crash" --> split["Split into 8, retry the unanswered remainder"] ``` -**One graph per language partition per scan.** `build_partitions` walks the tree, groups files by -extension (shebang fallback), excludes vendored paths and symlinks with the boundaries -`vendor_semantics_unresolved` and `symlink_context_unresolved`, copies each language into a scratch -directory and builds it into its own CPG (`model/partitions.py:168-305`). A request is routed to -the graph of its file's language (`partitions.py:110-165`). A repository with more than one -language records `cross_partition_semantics_unresolved` (`partitions.py:237-238`): a flow that -crosses languages "can hide a finding, never invent one" (`model/scan.py:706-715`). PHP may add a -second shard: files the frontend dropped after up to four retries are rebuilt separately into -`cpg-excluded.bin`, and the census then reports `cpg_sharded` (`backend.py:763-846`, -`scan.py:506-508`). +**One graph per language partition per scan.** `build_partitions` does the following +(`model/partitions.py:168-305`): -**Prove the tool read its input.** The project's repeated failure shape was a frontend that exited -0 and wrote a graph with no methods, which reads exactly like clean code. Three checks exist in -code. Before a PHP build, `php -r` must be able to open and size one sample file and the parser -itself, or the build is refused (`backend.py:719-761, 599-604`). After every build, `census.sc` -counts files, methods and calls; a census that cannot answer fails the build, and a graph holding -fewer files than expected sets `last_failure` (`backend.py:848-868, 809-819`; -`cpg/queries/census.sc:1-14`). On the host, `plane/engine_alerts.py:139-150` raises -`EngineAlertsError` when the engine log has no `read N files` line, N is 0, or no question was -asked, and `benchmarks/independent/evaluate.py:97-101` marks the same conditions -`instrument_error`. A JVM that returns in half a second did not run; the instrument check fails on -the census, not on elapsed time. +1. walks the tree; +2. groups files by extension (shebang fallback); +3. excludes vendored paths and symlinks, with the boundaries `vendor_semantics_unresolved` and + `symlink_context_unresolved`; +4. copies each language into a scratch directory; +5. builds each language into its own CPG. -**Questions are generated, not written.** One request per (region, family) is built from the fact -tables (`scan.py:1479-1505`, `model/specs.py:237-296`). A taint request carries `sources`, `sinks`, -`sanitizers`, `guards`, `quotedSanitizers`, `boundedSinks`, `fixedOrigin`, `originAnchors`, -`parameterSources`, `fieldParameterSources`, `callDepth` and the dispatch vocabulary -(`model/taint.py:21-67`). Requests are written to a JSON file, never the command line -(`backend.py:1214-1224`); shared fields are hoisted to one `--param` each (`backend.py:1681-1698`). +A request is routed to the graph of its file's language (`partitions.py:110-165`). A repository +with more than one language records `cross_partition_semantics_unresolved` +(`partitions.py:237-238`). A flow that crosses languages "can hide a finding, never invent one" +(`model/scan.py:706-715`). PHP may add a second shard. Files the frontend dropped after up to +four retries are rebuilt separately into `cpg-excluded.bin`. The census then reports +`cpg_sharded` (`backend.py:763-846`, `scan.py:506-508`). -**Batching follows a measured cost curve.** A fresh `joern --script` pays JVM start and graph load -each time. `2026-09-22-dataflow-batch-cost-curve.json` (`.measured`) shows 25 requests in 71.4 s, -100 in 155.4 s, 400 in 169.7 s: "roughly 70 s is fixed JVM start and graph load" (`.finding`). -Two fixes came from that. `overlay.sc` runs `importCpg` then `save` once after the build so the -dataflow layer is persisted: the saved graph reloads in 8 s where the raw one took 64 s -(`overlay.sc:11`), and without it each batch repaid 54 s on a 637-file plugin -(`backend.py:1001-1004`). Then `_PortionSizer` fits batches to `PORTION_TARGET_SECONDS = 120` -from a fixed-cost prior of 70 s (`scan.py:1121-1163`), with a hard cap of -`MAX_SINK_VISITS_PER_BATCH = 600` sink visits (`scan.py:926-957`). The evidence pass, which needs -no dataflow, sends `EVIDENCE_PORTION = 4000` requests per JVM (`scan.py:1249-1272`). A failed batch -is split into `_BATCH_SPLITS = 8` parts and only the unanswered remainder is retried -(`backend.py:55-57, 1115-1189`). An optional shared `joern --server` session exists behind -`OUSAST_ENGINE_SESSION=1` (`cpg/session.py`); its startup alone cost 9 to 13 s of a 30 s pre-push -budget and the profile recorded `NO-GO` (`2026-09-17-session-runtime-profile.json`, `.finding`, -`.runtime_verdict`). +**Prove the tool read its input.** The project's repeated failure was a frontend that exited 0 +and wrote a graph with no methods. That reads exactly like clean code. Three checks exist in +code: -**Deadlines.** Each query has a ceiling of `QUERY_TIMEOUT_SECONDS = 300` (`backend.py:54`) and each -build `BUILD_TIMEOUT_SECONDS = 900` (`backend.py:49`). The scan has one absolute monotonic -`ExecutionBudget` (`model/contracts.py:18-32`): 30 s for `pre-push` (`config.py:193`), 1800 s per -pin for `plane alerts-engine` (`engine_alerts.py:45`), 2400 s in the independent evaluation -(`evaluate.py:36`). A share of what remains is reserved for arbitration after querying -(`ARBITRATION_RESERVE_SHARE = 0.2`, floor 60 s, `scan.py:876-888`). A single query timeout does not -end the scan; a spent deadline does, and the whole process group is killed +1. Before a PHP build, `php -r` must be able to open and size one sample file and the parser + itself. Otherwise the build is refused (`backend.py:719-761, 599-604`). +2. After every build, `census.sc` counts files, methods and calls. A census that cannot answer + fails the build. A graph holding fewer files than expected sets `last_failure` + (`backend.py:848-868, 809-819`; `cpg/queries/census.sc:1-14`). +3. On the host, `plane/engine_alerts.py:139-150` raises `EngineAlertsError` in three cases: the + engine log has no `read N files` line, N is 0, or no question was asked. + `benchmarks/independent/evaluate.py:97-101` marks the same conditions `instrument_error`. + +A JVM that returns in half a second did not run. The instrument check fails on the census, not +on elapsed time. + +**Questions are generated, not written.** One request per (region, family) is built from the +fact tables (`scan.py:1479-1505`, `model/specs.py:237-296`). A taint request carries these +fields (`model/taint.py:21-67`): + +- `sources`, `sinks`, `sanitizers`, `guards`; +- `quotedSanitizers`, `boundedSinks`, `fixedOrigin`, `originAnchors`; +- `parameterSources`, `fieldParameterSources`, `callDepth`; +- the dispatch vocabulary. + +Requests are written to a JSON file, never the command line (`backend.py:1214-1224`). Shared +fields are hoisted to one `--param` each (`backend.py:1681-1698`). + +**Batching follows a measured cost curve.** A fresh `joern --script` pays JVM start and graph +load each time. `2026-09-22-dataflow-batch-cost-curve.json` (`.measured`) shows the cost: + +- 25 requests in 71.4 s; +- 100 in 155.4 s; +- 400 in 169.7 s. + +The record reads: "roughly 70 s is fixed JVM start and graph load" (`.finding`). Two fixes came +from that: + +1. `overlay.sc` runs `importCpg` then `save` once after the build, so the dataflow layer is + persisted. The saved graph reloads in 8 s where the raw one took 64 s (`overlay.sc:11`). + Without it, each batch repaid 54 s on a 637-file plugin (`backend.py:1001-1004`). +2. `_PortionSizer` fits batches to `PORTION_TARGET_SECONDS = 120` from a fixed-cost prior of + 70 s (`scan.py:1121-1163`). A hard cap of `MAX_SINK_VISITS_PER_BATCH = 600` sink visits + applies (`scan.py:926-957`). + +The evidence pass needs no dataflow. It sends `EVIDENCE_PORTION = 4000` requests per JVM +(`scan.py:1249-1272`). A failed batch is split into `_BATCH_SPLITS = 8` parts. Only the +unanswered remainder is retried (`backend.py:55-57, 1115-1189`). + +An optional shared `joern --server` session exists behind `OUSAST_ENGINE_SESSION=1` +(`cpg/session.py`). Its startup alone cost 9 to 13 s of a 30 s pre-push budget. The profile +recorded `NO-GO` (`2026-09-17-session-runtime-profile.json`, `.finding`, `.runtime_verdict`). + +**Deadlines.** Each query has a ceiling of `QUERY_TIMEOUT_SECONDS = 300` (`backend.py:54`). +Each build has `BUILD_TIMEOUT_SECONDS = 900` (`backend.py:49`). The scan has one absolute +monotonic `ExecutionBudget` (`model/contracts.py:18-32`): + +- 30 s for `pre-push` (`config.py:193`); +- 1800 s per pin for `plane alerts-engine` (`engine_alerts.py:45`); +- 2400 s in the independent evaluation (`evaluate.py:36`). + +A share of what remains is reserved for arbitration after querying +(`ARBITRATION_RESERVE_SHARE = 0.2`, floor 60 s, `scan.py:876-888`). A single query timeout does +not end the scan. A spent deadline does, and the whole process group is killed (`backend.py:1420-1440`). -**Results are rows, and silence is typed.** Taint scripts stream a `__census__` line first, then -one `{"id","rows","ms"}` line per request (`cpg/queries/taint.sc:1475-1527`); a killed batch keeps -its finished answers and is marked `__partial__` (`backend.py:196, 234-270`). `None` means the -engine could not answer and `{}` means a genuine empty answer (`backend.py:1200-1203`). The -degradations a scan can carry are: +**Results are rows, and silence is typed.** Taint scripts stream a `__census__` line first. Then +they stream one `{"id","rows","ms"}` line per request (`cpg/queries/taint.sc:1475-1527`). A +killed batch keeps its finished answers and is marked `__partial__` (`backend.py:196, 234-270`). +`None` means the engine could not answer. `{}` means a genuine empty answer +(`backend.py:1200-1203`). A scan can carry these degradations: | Degradation | Raised at | Condition | | --- | --- | --- | @@ -216,62 +284,79 @@ degradations a scan can carry are: | `query_failed` | `scan.py:356, 478-483, 512-523` | a kind or some ids were unanswered | | `cross_partition_semantics_unresolved` | `partitions.py:237-238` | more than one language partition | -`files_unparsed`, `cpg_empty` and `cpg_sharded` are "graph integrity gaps": completed answers in -the affected partition are demoted to `unresolved/graph_incomplete` rather than reported as clean -(`scan.py:716-729`). +`files_unparsed`, `cpg_empty` and `cpg_sharded` are "graph integrity gaps". Completed answers in +the affected partition are demoted to `unresolved/graph_incomplete`. They are not reported as +clean (`scan.py:716-729`). **Operating the engine.** The engine runs inside the `openultrasast:dev` image with `--network none --memory 3g`, one container at a time (`engine_alerts.py:43-44, 128-136`; -`evaluate.py:82-93`; `docker-compose.yml:30-31`: "two Joern JVMs at once is what triggered an -out-of-memory kill on a 7.7 GB machine"). The source mounted into the container is a frozen export -of a commit, never the live tree (`engine_alerts.py:102-110`), because a five-hour replay was once -confounded by edits landing mid-run. The heap is `CPG_HEAP_MB = 2048` by default, applied to both -the launcher and the forked worker JVM (`backend.py:63, 1466-1485`), and the census reports the -worker's `maxHeapMB` so a mismatch is visible (`backend.py:886-894`). Build, smoke and pre-push -commands are in [Engine and pre-push](ops/README.md). +`evaluate.py:82-93`). `docker-compose.yml:30-31` gives the reason: "two Joern JVMs at once is +what triggered an out-of-memory kill on a 7.7 GB machine". + +The source mounted into the container is a frozen export of a commit, never the live tree +(`engine_alerts.py:102-110`). This is because a five-hour replay was once confounded by edits +landing mid-run. + +The heap is `CPG_HEAP_MB = 2048` by default. It applies to both the launcher and the forked +worker JVM (`backend.py:63, 1466-1485`). The census reports the worker's `maxHeapMB`, so a +mismatch is visible (`backend.py:886-894`). Build, smoke and pre-push commands are in [Engine +and pre-push](ops/README.md). ### Where the engine sits in the plane -The engine is a signal producer, not a decider. On the host, `ousast plane alerts-engine` runs it -pin by pin for languages whose alerts of record come from the engine: `ENGINE_ALONGSIDE_QUICK = -{"php"}` keeps the engine running beside the PHP quick rules because the rules' measured precision -is low (`engine_alerts.py:47-50, 202-205`). Each finding becomes an alert row with -`rule_id = "engine:"` and `source = "engine"` (`engine_alerts.py:160-174`). In the -decision engine's feature records the engine contributes the `eng.*` block (`eng.findings`, -`eng.rung_max`, `eng.witness_steps_min`, `eng.sink_kind`, `eng.source_kind`, `eng.completion`, -`eng.degraded`, `eng.unstable`; `learn/schema.py:140-150`, `learn/features.py:199-224`). Input -profile `v1` withholds that block entirely: on the pair corpus the engine's "read the file, asked no -question" state tracks the label, because a fixed side loses its sink (`learn/schema.py:97-109`; -`2026-10-01-decision-engine-leak-audit/`). [Deployment](deployment.md) lists `alerts-engine` as -implemented and run on this host; its planned Kubernetes cluster says nothing about the engine. +The engine is a signal producer, not a decider. On the host, `ousast plane alerts-engine` runs +it pin by pin for languages whose alerts of record come from the engine. `ENGINE_ALONGSIDE_QUICK = +{"php"}` keeps the engine running beside the PHP quick rules, because the rules' measured +precision is low (`engine_alerts.py:47-50, 202-205`). Each finding becomes an alert row with +`rule_id = "engine:"` and `source = "engine"` (`engine_alerts.py:160-174`). + +In the decision engine's feature records, the engine contributes the `eng.*` block +(`learn/schema.py:140-150`, `learn/features.py:199-224`): + +- `eng.findings`, `eng.rung_max`, `eng.witness_steps_min`; +- `eng.sink_kind`, `eng.source_kind`; +- `eng.completion`, `eng.degraded`, `eng.unstable`. + +Input profile `v1` withholds that block entirely. On the pair corpus, the engine's "read the +file, asked no question" state tracks the label, because a fixed side loses its sink +(`learn/schema.py:97-109`; `2026-10-01-decision-engine-leak-audit/`). [Deployment](deployment.md) +lists `alerts-engine` as implemented and run on this host. Its planned Kubernetes cluster says +nothing about the engine. ### What the engine costs and misses, with records -- **33 minutes per pin on a mid-sized plugin.** On Paid Memberships Pro (637 files, 4.2 MB) the - engine took 1993.4 s on the vulnerable pin (382 questions, 125 completed, 10 findings, the CVE - site hit) and 2025.8 s on the fixed pin, with `process_timeout` and `query_budget_reserved` - among its degradations (`2026-09-30-php-quick-rules/measurement.json`, `engine.pmpro.*`). The - quick rules read the same tree in 5.3 s. -- **Per-question cost is one to two seconds of graph work** (`2026-09-20-php-detection-corrected.json`, - `.the_scale_limit.reading`): 25 regions produced 127 questions and 19 findings in 241 s of - query time; 500 regions produced 2,526 questions and completed none inside a 300 s ceiling, and - raising the ceiling to 1800 s "changed nothing except how long the failure took" - (`2026-09-20-php-runtime-after-graph-fix.json`, `.reading`). -- **Population v2: recall 0 of 17** (`benchmarks/independent/results-v2.json`, `.recall`), - completion 13,206 of 108,374 questions, 3 benign alerts, every M4 gate not met. Nine cases were - `unanswered`. The zero is not an instrument artefact: FUXA, GoPay, cct-studio, ReactPress, User - Registration and Directorist completed questions in the sink file itself and found nothing - (commit `21adc40`). -- **froxlor and GoPay on the host** (`2026-09-30-php-engine-alerts/`): froxlor completed 130 of 130 - questions per pin in about 390 s and emitted six `engine:injection` alerts, none at a declared - site and identical on both pins (`froxlor-*-alerts-summary.json`); GoPay completed 101 of 101 in - about 120 s with 0 alerts on both pins, under `vendor_semantics_unresolved`. Neither record states - the cause of the miss. -- **Population v1: recall 1 of 11**, with 8 of 11 cases completing no question at the vulnerable - pin (`results-v1.json`, `.summary`, `cases_with_zero_completed_questions_at_vulnerable_pin`). +- **33 minutes per pin on a mid-sized plugin.** Paid Memberships Pro has 637 files (4.2 MB). + - On the vulnerable pin, the engine took 1993.4 s: 382 questions, 125 completed, 10 findings, + the CVE site hit. + - On the fixed pin, it took 2025.8 s. + - Its degradations included `process_timeout` and `query_budget_reserved` + (`2026-09-30-php-quick-rules/measurement.json`, `engine.pmpro.*`). + - The quick rules read the same tree in 5.3 s. +- **Per-question cost is one to two seconds of graph work** + (`2026-09-20-php-detection-corrected.json`, `.the_scale_limit.reading`). + - 25 regions produced 127 questions and 19 findings in 241 s of query time. + - 500 regions produced 2,526 questions and completed none inside a 300 s ceiling. + - Raising the ceiling to 1800 s "changed nothing except how long the failure took" + (`2026-09-20-php-runtime-after-graph-fix.json`, `.reading`). +- **Population v2: recall 0 of 17** (`benchmarks/independent/results-v2.json`, `.recall`). + Completion was 13,206 of 108,374 questions, with 3 benign alerts. Every M4 gate was not met. + Nine cases were `unanswered`. The zero is not an instrument artefact. FUXA, GoPay, + cct-studio, ReactPress, User Registration and Directorist completed questions in the sink + file itself and found nothing (commit `21adc40`). +- **froxlor and GoPay on the host** (`2026-09-30-php-engine-alerts/`). + - froxlor completed 130 of 130 questions per pin in about 390 s. It emitted six + `engine:injection` alerts, none at a declared site and identical on both pins + (`froxlor-*-alerts-summary.json`). + - GoPay completed 101 of 101 in about 120 s, with 0 alerts on both pins, under + `vendor_semantics_unresolved`. + - Neither record states the cause of the miss. +- **Population v1: recall 1 of 11.** 8 of 11 cases completed no question at the vulnerable pin + (`results-v1.json`, `.summary`, `cases_with_zero_completed_questions_at_vulnerable_pin`). ## 3. Taint: fact tables to query to verdict +Taint analysis asks whether untrusted input reaches a dangerous call without being cleaned. + ```mermaid flowchart LR facts["ruleset/semantic/lang.toml: sources, sinks, sanitizers, dispatch"] --> specs["specs.py: TaintSpec per family"] @@ -286,37 +371,65 @@ flowchart LR rows --> verdict["ENTAILED if open, CORROBORATED if sanitized or bounded"] ``` -**Facts.** A language's facts are `[[source]]`, `[[sink]]`, `[[sanitizer]]`, `[[dispatch]]` and -`[[layout]]` tables in `ruleset/semantic/.toml`, loaded by `semantic/facts.py` -(`load_facts`, L412-427; schema L37-146). Counts per file: C 2/3/1, Java 2/9/4, JavaScript 2/17/6, -Python 5/18/4, PHP 3/11/11 (sources/sinks/sanitizers). A sink carries a CWE, call names, and -optional `weak_literals`, `format_arg`, `prefix_fixes_origin` and `origin_anchors` -(`facts.py:46-61`). A sanitizer is either a cleansing call or one of two typed shapes: `quoted_only` -(an escape, protects only inside a quoted literal) and `guard` (a check, "never a cleansing call on -a path") (`facts.py:65-79`). Families are not a fact field; `specs.py:269-279` groups sinks through -`families.family_of_cwe`, and `ruleset/families.toml` is the closed list of ten families with a -verifier kind each. `php.toml` is the worked example: sources `request` (`$_GET`, `$_POST`, -`$_REQUEST`, `$_COOKIE`, `$_FILES`), `server`, `raw_body`; sinks such as `wpdb` (CWE-89, the -qualified `$wpdb->get_var` form to avoid the generic-name trap, L40-59), `command` (CWE-78), -`unserialize` (CWE-502); sanitizers such as `sql_escape` with `quoted_only = true` (L176-186), -`allowlist` with `guard = true` (`in_array`, L200-207), `prepared_statement` with -`parameterized = true` (L234-237). `sanitize_text_field`, `esc_html` and `esc_attr` are -deliberately absent because they do not stop SQL injection (L195-199). Framework-tagged entries -can be switched off as a group through `priors` (`ruleset/frameworks.py:95-101`). +**Facts.** A language's facts live in `ruleset/semantic/.toml`. They are `[[source]]`, +`[[sink]]`, `[[sanitizer]]`, `[[dispatch]]` and `[[layout]]` tables. `semantic/facts.py` loads +them (`load_facts`, L412-427; schema L37-146). Counts per file (sources/sinks/sanitizers): -**The query.** `taint.sc` resolves the region's methods by name and file, scopes to lexical nesting -plus `callDepth` levels of callees (L192-278; `ENTRY_POINT_CALL_DEPTH = 3` for entry points, -`scan.py:70, 570-571`), builds source nodes from four pools (L1044), matches sink calls on name, -callee text or full name with word boundaries (L1098-1110), and asks -`sink.argument.argumentIndexGt(0).reachableByFlows(sourceNodes)` filtered by `plausible` (L1360). -Each row is one (sourceKind, source, sanitized) key with `sinkLine`, `length`, `paths`, `bounded` -and `inLabeledScope` (L1408-1442). The Python arbiter (`model/taint.py:129-173`) prefers flows from -a modelled source, orders them by a total `_rank`, and returns `ENTAILED` for a flow with no -sanitizer and no bound, `CORROBORATED` for a sanitized or bounded flow, and `None` when there are -no flows or the query failed, because "the engine could not decide" is not "there is no flow" +- C 2/3/1; +- Java 2/9/4; +- JavaScript 2/17/6; +- Python 5/18/4; +- PHP 3/11/11. + +A sink carries a CWE, call names, and optional `weak_literals`, `format_arg`, +`prefix_fixes_origin` and `origin_anchors` (`facts.py:46-61`). A sanitizer is a cleansing call +or one of two typed shapes (`facts.py:65-79`): + +- `quoted_only`: an escape that protects only inside a quoted literal; +- `guard`: a check, "never a cleansing call on a path". + +Families are not a fact field. `specs.py:269-279` groups sinks through +`families.family_of_cwe`. `ruleset/families.toml` is the closed list of ten families, each with a +verifier kind. + +`php.toml` is the worked example: + +- sources: `request` (`$_GET`, `$_POST`, `$_REQUEST`, `$_COOKIE`, `$_FILES`), `server`, + `raw_body`; +- sinks such as `wpdb` (CWE-89, the qualified `$wpdb->get_var` form to avoid the generic-name + trap, L40-59), `command` (CWE-78) and `unserialize` (CWE-502); +- sanitizers such as `sql_escape` with `quoted_only = true` (L176-186), `allowlist` with + `guard = true` (`in_array`, L200-207), and `prepared_statement` with `parameterized = true` + (L234-237). + +`sanitize_text_field`, `esc_html` and `esc_attr` are left out on purpose, because they do not +stop SQL injection (L195-199). Framework-tagged entries can be switched off as a group through +`priors` (`ruleset/frameworks.py:95-101`). + +**The query.** `taint.sc` works in these steps: + +1. It resolves the region's methods by name and file. +2. It scopes to lexical nesting plus `callDepth` levels of callees (L192-278). Entry points use + `ENTRY_POINT_CALL_DEPTH = 3` (`scan.py:70, 570-571`). +3. It builds source nodes from four pools (L1044). +4. It matches sink calls on name, callee text or full name, with word boundaries (L1098-1110). +5. It asks `sink.argument.argumentIndexGt(0).reachableByFlows(sourceNodes)`, filtered by + `plausible` (L1360). + +Each row is one (sourceKind, source, sanitized) key with `sinkLine`, `length`, `paths`, +`bounded` and `inLabeledScope` (L1408-1442). + +The Python arbiter (`model/taint.py:129-173`) prefers flows from a modelled source. It orders +them by a total `_rank`. It returns: + +- `ENTAILED` for a flow with no sanitizer and no bound; +- `CORROBORATED` for a sanitized or bounded flow; +- `None` when there are no flows or the query failed. + +`None` is used because "the engine could not decide" is not "there is no flow" (`taint.py:8-10`). -**Refinements, each with the miss or false positive that forced it.** Every one is a rule about +**Refinements, each with the miss or false positive that forced it.** Each one is a rule about what the graph may count, not a new source of flows. | Refinement | What it does (`cpg/queries/taint.sc`) | Record | @@ -329,20 +442,22 @@ what the graph may count, not a new source of flows. | Fixed-origin destinations | A sink with `prefix_fixes_origin` ignores a path that enters a concatenation anywhere but the leftmost operand when that operand already fixes the origin (`?`, `#`, `/x`, `scheme://host/`), or passes an anchor such as `reverse`; `"/" + x` and `"https://" + host` stay attacks (L463-542) | OpenCVE's vulnerable and fixed pins fell from 2 and 3 findings to 0 and 0; the SSRF itself is still missed (commit `562b670`) | | Bounded source tokens | A source pattern is a word-bounded token on the callee text or access, not a substring (L174-183); a package name is not a sink token | `axios` bounded inside `axios.get` (`2026-09-19-untrusted-destination-vocabulary.json`, `.design.bare_module_names`); YesWiki's 1,128 PHP questions all unresolved, and Flask's `request.get` claiming Django's `request.get_host()` (commit `3358631`); mongo-express's package name matching the SQL sink `query` (commit `ec3d145`) | -One refinement was reverted: treating a field as input only when a caller feeds it request data -cut findings 37 to 31 but removed the adjudicated true positive `discount-codes.php:154` and only -one of six object-field false positives (commits `6ebbc89`, `bdb5a86`). Known gaps are kept in the -records too: that true positive needs "a cross-file field join keyed by class" -(`2026-09-24-pmpro-sqli-adjudication-round3.json`, `.known_true_positive_not_reported`). +One refinement was reverted. It treated a field as input only when a caller feeds it request +data. It cut findings 37 to 31. But it removed the adjudicated true positive +`discount-codes.php:154` and only one of six object-field false positives (commits `6ebbc89`, +`bdb5a86`). Known gaps are kept in the records too. That true positive needs "a cross-file field +join keyed by class" (`2026-09-24-pmpro-sqli-adjudication-round3.json`, +`.known_true_positive_not_reported`). **Cost.** `2026-09-22-php-query-cost-identified.json` (`.the_answer`) found that a taint question -costs in proportion to the number of CALL nodes a family's sink names match, which is why PHP +costs in proportion to the number of CALL nodes a family's sink names match. This is why PHP `echo`/`print` sinks dominate output-encoding time. The budget controls are in section 5. ## 4. Absence bugs: obligations, dominance, configuration, census -An absence bug is an operation that should have been guarded and was not. There is no flow to find, -so it is modelled as an obligation and discharged by a witness. +An absence bug is an operation that should have been guarded and was not. There is no flow to +find. So it is modelled as an obligation (a duty the operation must meet) and discharged by a +witness (code that shows the duty is met). ```mermaid flowchart LR @@ -357,57 +472,85 @@ flowchart LR ``` **Obligation facts** live in `ruleset/obligations/{python,javascript,typescript}.toml` as -`[[operation]]` and `[[discharger]]` rows with closed vocabularies: operation kinds -`protected_read`, `protected_write`, `privileged_action`, `security_setting`; discharger kinds -`path_guard`, `identity_constraint`, `ownership_check`, `non_permissive_value`, `validated_input`; -provenance kinds `authenticated_context`, `request_input`, `constant`, `unknown` -(`semantic/obligations/facts.py`). Python has 5 operations and 6 dischargers, JavaScript and -TypeScript 7 and 6. Any other language records `obligations_language_unsupported` -(`obligations/check.py:76-78`). An example is `collection-read` (`javascript.toml:19-26`): `find` -with `query_document_arg = 1`, added only after a false-positive measurement showed 88 of 88 `find` -calls on one plugin's assets were selectors, matched as operations 0 times in the whole graph +`[[operation]]` and `[[discharger]]` rows. The vocabularies are closed +(`semantic/obligations/facts.py`): + +- operation kinds: `protected_read`, `protected_write`, `privileged_action`, `security_setting`; +- discharger kinds: `path_guard`, `identity_constraint`, `ownership_check`, + `non_permissive_value`, `validated_input`; +- provenance kinds: `authenticated_context`, `request_input`, `constant`, `unknown`. + +Python has 5 operations and 6 dischargers. JavaScript and TypeScript have 7 and 6. Any other +language records `obligations_language_unsupported` (`obligations/check.py:76-78`). + +An example is `collection-read` (`javascript.toml:19-26`): `find` with `query_document_arg = 1`. +It was added only after a false-positive measurement. On one plugin's assets, 88 of 88 `find` +calls were selectors, and they were matched as operations 0 times in the whole graph (`2026-09-20-collection-read-obligation.json`, `.false_positive_measurement`). -**Two dominance checks.** In the IR, `OrderDominance` accepts a decorator or router witness for the -whole handler, a statement witness that precedes the operation and either is a `path_guard` or is -followed by an early exit, or an earlier hop along a path record (`obligations/dominance.py:446-468`). -In the CPG, `dominance.sc` recognises five governing shapes: a discharger token in the operation's -own arguments, a control-structure condition in the same method, a non-operator call that -`op.dominatedBy` and mentions a discharger (the one place CFG dominance is used), a guard at the -carrier call when the obligation is carried across a module boundary through an external stub, and -identity reaching the arguments through at most three local assignments (`dominance.sc:300-363`). -The arbiter compares siblings in the file: an unguarded operation beside guarded ones is -`ENTAILED`; no guarded sibling, or an obligation carried across a module, is `CORROBORATED` -(`model/dominance.py:62-105`). The sibling scope was measured: file scope gave 6 entailed with 4 -true and 2 false, the narrower scope 2 entailed and 0 true, and file scope was kept -(`2026-09-20-sibling-scope-and-identity.json`, `.part_two`). On NodeGoat, 15 handlers were asked -and 9 carried an obligation across a module split, yielding 3 `model_corroborated` verdicts -(`2026-09-20-access-control-module-split.json`, `.measured_on_nodegoat`). Sibling consistency in -the IR path needs at least `min_siblings = 3` discharging siblings (`config.py:179`, -`obligations/siblings.py:470-504`). A project may declare routes public in -`.openultrasast/obligations.toml`, which waives `path_guard` (`obligations/policy.py`, +**Two dominance checks.** A guard dominates an operation when every path to the operation passes +through the guard. + +In the IR, `OrderDominance` accepts three kinds of witness (`obligations/dominance.py:446-468`): + +- a decorator or router witness for the whole handler; +- a statement witness that precedes the operation and either is a `path_guard` or is followed by + an early exit; +- an earlier hop along a path record. + +In the CPG, `dominance.sc` recognises five governing shapes (`dominance.sc:300-363`): + +1. a discharger token in the operation's own arguments; +2. a control-structure condition in the same method; +3. a non-operator call that `op.dominatedBy` and mentions a discharger (the one place CFG + dominance is used); +4. a guard at the carrier call, when the obligation is carried across a module boundary through + an external stub; +5. identity reaching the arguments through at most three local assignments. + +The arbiter compares siblings in the file (`model/dominance.py:62-105`): + +- an unguarded operation beside guarded ones is `ENTAILED`; +- no guarded sibling, or an obligation carried across a module, is `CORROBORATED`. + +The sibling scope was measured. File scope gave 6 entailed, with 4 true and 2 false. The +narrower scope gave 2 entailed and 0 true. File scope was kept +(`2026-09-20-sibling-scope-and-identity.json`, `.part_two`). On NodeGoat, 15 handlers were +asked. 9 carried an obligation across a module split, which yielded 3 `model_corroborated` +verdicts (`2026-09-20-access-control-module-split.json`, `.measured_on_nodegoat`). + +Sibling consistency in the IR path needs at least `min_siblings = 3` discharging siblings +(`config.py:179`, `obligations/siblings.py:470-504`). A project may declare routes public in +`.openultrasast/obligations.toml`. This waives `path_guard` (`obligations/policy.py`, `check.py:112`). -**Configuration: constant abstraction.** `config.sc` returns, for every call matching a setting -name, the argument texts and the literals found anywhere in each argument's AST subtree -(`config.sc:206-236`). The abstraction is in Python: literals are stripped of quotes, lower-cased -and compared to a closed permissive set from the obligation facts, with per-setting polarity -(`model/config_value.py:55-57, 106-118`). A permissive literal is `ENTAILED`; a computed argument is -`CORROBORATED`, not safe (`config_value.py:7-9, 75-86`). Weak-algorithm literals come from sink -facts' `weak_literals` (`specs.py:211-218`). Asking every permissive setting rather than one raised -NodeGoat's `config_secrets` findings from 1 to 2 (`2026-09-20-every-permissive-setting.json`). +**Configuration: constant abstraction.** `config.sc` looks at every call that matches a setting +name. For each, it returns the argument texts and the literals found anywhere in each +argument's AST subtree (`config.sc:206-236`). The abstraction is in Python +(`model/config_value.py:55-57, 106-118`): -**Census.** `census.sc` counts files, methods and calls, lists file names and overlays, and reports -the worker JVM's heap (`census.sc:17-31`). It is the instrument check of section 2, and it also -settles whether a frontend warning mattered: if the graph holds every expected file, the warning is -ignored (`backend.py:827-829`). `vocabulary.sc` is a related, dataflow-free query that lists every -call not covered by the fact tables as a candidate list, "never an ontology" -(`vocabulary.sc:72-74`); it feeds `benchmarks/push/ontology_gap.py`, not the scan. +1. literals are stripped of quotes and lower-cased; +2. they are compared to a closed permissive set from the obligation facts; +3. each setting has its own polarity. + +A permissive literal is `ENTAILED`. A computed argument is `CORROBORATED`, not safe +(`config_value.py:7-9, 75-86`). Weak-algorithm literals come from sink facts' `weak_literals` +(`specs.py:211-218`). Asking every permissive setting, not just one, raised NodeGoat's +`config_secrets` findings from 1 to 2 (`2026-09-20-every-permissive-setting.json`). + +**Census.** `census.sc` counts files, methods and calls. It lists file names and overlays and +reports the worker JVM's heap (`census.sc:17-31`). It is the instrument check of section 2, and it +settles whether a frontend warning mattered: if the graph holds every expected file, the +warning is ignored (`backend.py:827-829`). + +`vocabulary.sc` is a related query that needs no dataflow. It lists every call not covered by +the fact tables as a candidate list, "never an ontology" (`vocabulary.sc:72-74`). It feeds +`benchmarks/push/ontology_gap.py`, not the scan. ## 5. Regions, ranking and budgets -A repository is too large to ask the graph about everything, so the scan asks about regions in a -deterministic order under a budget and records what it did not ask. +A repository is too large to ask the graph about everything. So the scan asks about regions in a +deterministic order, under a budget, and records what it did not ask. ```mermaid flowchart TD @@ -424,44 +567,60 @@ flowchart TD ``` **Regions.** `regions_for` makes one `ScanRegion` per (path, function) from the mapper's entry -points, adds a `` region at rank 0.15 for files with only function-scoped regions and a -file fallback at rank 0.1 for uncovered files (`model/regions.py:56-171`). The rank is the entry's -access level: public 1.0, authenticated 0.8, role-restricted 0.6, contract-only 0.4, -review-required 0.3, local-only 0.2, unknown 0.5 (`regions.py:32-39, 125`). The sort key is -`(not shipped, -rank, path, function)`, "a total order so two runs of one repository agree" -(`regions.py:173-180`). Shipped status comes from the project's build files (`model/shipped.py`); -test paths are ordered after product code, not removed (`model/layout.py:67-73`). Mis-attributed -function-scoped regions once left 436 of NodeGoat's questions unresolved; region attribution -brought that to 0 with 299 completed on both pins (`2026-09-17-region-attribution.json`, -`.progression`). +points (`model/regions.py:56-171`). It also adds: -**What ties did to PHP.** Access rank has five distinct values, so a large plugin collapses into a -few tiers ordered by path. On Paid Memberships Pro the baseline shows 4,463 regions with -`largest_tier_share 0.838` (83.8% of regions at one rank, `distinct_ranks 5`) and CVE-2023-23488 -at position 390, so recall needed a budget of 391 regions; WP Statistics needed 504 -(`benchmarks/ranking/baselines/2026-09-10-access-rank.json`). The answer was an evidence pass that -needs no dataflow (`model/evidence.py`): each region's pairs get a tier (`TIER_EXCLUDE = 0` no -sink in reach, `TIER_NO_SOURCE`, `TIER_CLEANSED`, `TIER_OPEN`, `TIER_OPEN_PUBLIC = 4`) and a -within-tier score from published weights (`evidence.py:18-40, 89-127`). Two uses followed. Tier 0 -is never asked (`ScanBudget.prune_tier0 = True`, `scan.py:93`): at 500 regions the questions fell -from 2,526 to 406 and at 25 regions 19 findings stayed 19 (`2026-09-22-tier-zero-pruning.json`, -`.no_finding_lost`); commit `88ada8e` recorded 86% of 2,500 requests at tier 0 and a 7x taint -speed-up. Ordering regions by evidence (`order_by_evidence`) is implemented but **off by default** -"until the position benchmark says where the pinned CVEs land under it" (`scan.py:95-99`); when on, -ties fall back to the static order (`scan.py:1399-1422`). The host runs that produce alerts and -the independent evaluation pass `--max-regions 5000`, which on these repositories means no cut -(`engine_alerts.py:136`, `evaluate.py:82-93`). +- a `` region at rank 0.15 for files with only function-scoped regions; +- a file fallback at rank 0.1 for uncovered files. + +The rank is the entry's access level: public 1.0, authenticated 0.8, role-restricted 0.6, +contract-only 0.4, review-required 0.3, local-only 0.2, unknown 0.5 (`regions.py:32-39, 125`). + +The sort key is `(not shipped, -rank, path, function)`, "a total order so two runs of one +repository agree" (`regions.py:173-180`). Shipped status comes from the project's build files +(`model/shipped.py`). Test paths are ordered after product code, not removed +(`model/layout.py:67-73`). Mis-attributed function-scoped regions once left 436 of NodeGoat's +questions unresolved. Region attribution brought that to 0, with 299 completed on both pins +(`2026-09-17-region-attribution.json`, `.progression`). + +**What ties did to PHP.** Access rank has five distinct values. So a large plugin collapses into +a few tiers ordered by path. The baseline for Paid Memberships Pro shows: + +- 4,463 regions; +- `largest_tier_share 0.838` (83.8% of regions at one rank, `distinct_ranks 5`); +- CVE-2023-23488 at position 390, so recall needed a budget of 391 regions. + +WP Statistics needed 504 (`benchmarks/ranking/baselines/2026-09-10-access-rank.json`). + +The answer was an evidence pass that needs no dataflow (`model/evidence.py`). Each region's +pairs get a tier and a within-tier score from published weights (`evidence.py:18-40, 89-127`). +The tiers run from `TIER_EXCLUDE = 0` (no sink in reach) through `TIER_NO_SOURCE`, +`TIER_CLEANSED` and `TIER_OPEN` to `TIER_OPEN_PUBLIC = 4`. Two uses followed: + +1. Tier 0 is never asked (`ScanBudget.prune_tier0 = True`, `scan.py:93`). At 500 regions the + questions fell from 2,526 to 406. At 25 regions, 19 findings stayed 19 + (`2026-09-22-tier-zero-pruning.json`, `.no_finding_lost`). Commit `88ada8e` recorded 86% of + 2,500 requests at tier 0 and a 7x taint speed-up. +2. Ordering regions by evidence (`order_by_evidence`) is implemented but **off by default** + "until the position benchmark says where the pinned CVEs land under it" (`scan.py:95-99`). + When it is on, ties fall back to the static order (`scan.py:1399-1422`). + +The host runs that produce alerts, and the independent evaluation, pass `--max-regions 5000`. +On these repositories that means no cut (`engine_alerts.py:136`, `evaluate.py:82-93`). **Budgets.** `ScanBudget` defaults are `max_model_calls = 200` and `max_regions = 500` -(`scan.py:78-99`); the product configuration uses 100 and 300 (`config.py:164-172`, wired at -`cli.py:1183`). When regions exceed the cap, `regions_truncated` records `examined` and `total` -(`scan.py:321-335`; the comment cites PMPro: 4,463 regions, so 500 examines 11%). When model calls -run out, the judge is withheld, graph arbitration continues, and `budget_exhausted` with -`regions_unasked` is appended (`scan.py:546-589`). The time budget is section 2's -`ExecutionBudget`. A push can select far more questions than a census: 2,500 PHP questions is -roughly six hours of engine, which is why the question count is sized before any replay -(`benchmarks/push/python_overhead.py` runs the real pipeline against a stub backend in about a -minute). `2026-09-22-php-declared-target-in-scope.json` (`.result`) shows the shape: 500 regions, +(`scan.py:78-99`). The product configuration uses 100 and 300 (`config.py:164-172`, wired at +`cli.py:1183`). + +- When regions exceed the cap, `regions_truncated` records `examined` and `total` + (`scan.py:321-335`). The code comment cites PMPro: 4,463 regions, so 500 examines 11%. +- When model calls run out, the judge is withheld and graph arbitration continues. + `budget_exhausted` with `regions_unasked` is appended (`scan.py:546-589`). +- The time budget is section 2's `ExecutionBudget`. + +A push can select far more questions than a census. 2,500 PHP questions is roughly six hours of +engine. That is why the question count is sized before any replay. +`benchmarks/push/python_overhead.py` runs the real pipeline against a stub backend in about a +minute. `2026-09-22-php-declared-target-in-scope.json` (`.result`) shows the shape: 500 regions, 406 questions arbitrated, 271 completed, 2949.8 s, declared target found. ## 6. The model's role: the residual question only @@ -479,50 +638,74 @@ flowchart LR ``` The LLM never creates a flow. `model/pipeline.py:1` states the division: "the enumerator proposes, -the LLM answers, the model disposes", where "the model" is the graph and its rules. Three question -forms exist and each opens with "judging ONE ... Do not look for other issues" -(`pipeline.py:46-56, 183-222`). For a flow the graph found with a sanitizer on it, the residual -question is only "whether the sanitizer on that path actually makes it safe for the {family} -weakness class" (`pipeline.py:213-222`); the dominance form asks whether the operation is -legitimately public, the configuration form whether a value is permissive in a deployed -configuration. The answer is JSON `{"vulnerable": bool, "why": str}` with no tools and no retry; -anything else, or an endpoint failure, is `None`, "a degradation, never a verdict" -(`pipeline.py:225-244`). `scan_region` returns without a model call when any verdict is `ENTAILED` -(`pipeline.py:139-146`). +the LLM answers, the model disposes". Here "the model" is the graph and its rules. + +Three question forms exist. Each opens with "judging ONE ... Do not look for other issues" +(`pipeline.py:46-56, 183-222`): + +- **Taint form.** For a flow the graph found with a sanitizer on it, the residual question is + only "whether the sanitizer on that path actually makes it safe for the {family} weakness + class" (`pipeline.py:213-222`). +- **Dominance form.** It asks whether the operation is legitimately public. +- **Configuration form.** It asks whether a value is permissive in a deployed configuration. + +The answer is JSON `{"vulnerable": bool, "why": str}`, with no tools and no retry. Anything +else, or an endpoint failure, is `None`, "a degradation, never a verdict" +(`pipeline.py:225-244`). `scan_region` returns without a model call when any verdict is +`ENTAILED` (`pipeline.py:139-146`). When the graph resolved nothing, the enumerator's candidates are judged. `enumerate_candidates` -parses the region's file to IR, collects call sites, binds, operations and config sites per family -shape, sorts by `(path, line, name)` and caps at `MAX_CANDIDATES_PER_CALL = 12` -(`model/candidates.py:30-47, 97-140`); the pipeline then judges at most `MAX_JUDGED_CANDIDATES = 8` -(`pipeline.py:44, 164`). Each affirmed candidate is reported at `suspicion` with the contradiction -"the model resolved no flow here; its sink table may not cover this API" (`pipeline.py:171-179`). -Silence is not refutation: "23 of the 39 misses in the entailment ceiling were sinks simply absent -from the fact tables" (`pipeline.py:10-17`). The enumerator was measured before it was built: the -IR's call sites and bindings reach the labelled function in 96.6% of the corpus against 12.1% for -the pattern ruleset (`candidates.py:7-10`), which is why "the ruleset is never the generator". -Candidates are enumerated only when a judge client exists; enumerating them without one cost 737 s -on a 637-file plugin (`scan.py:551-558`). Without a provider key `client is None`, and both the -corroborated and the suspicion branches return nothing (`pipeline.py:152-153, 161-162`). +works like this (`model/candidates.py:30-47, 97-140`): + +1. it parses the region's file to IR; +2. it collects call sites, binds, operations and config sites per family shape; +3. it sorts by `(path, line, name)`; +4. it caps at `MAX_CANDIDATES_PER_CALL = 12`. + +The pipeline then judges at most `MAX_JUDGED_CANDIDATES = 8` (`pipeline.py:44, 164`). Each +affirmed candidate is reported at `suspicion` with the contradiction "the model resolved no flow +here; its sink table may not cover this API" (`pipeline.py:171-179`). + +Silence is not refutation: "23 of the 39 misses in the entailment ceiling were sinks simply +absent from the fact tables" (`pipeline.py:10-17`). The enumerator was measured before it was +built. The IR's call sites and bindings reach the labelled function in 96.6% of the corpus. The +pattern ruleset reaches it in 12.1% (`candidates.py:7-10`). This is why "the ruleset is never the +generator". + +Candidates are enumerated only when a judge client exists. Enumerating them without one cost +737 s on a 637-file plugin (`scan.py:551-558`). Without a provider key, `client is None`, and +both the corroborated and the suspicion branches return nothing +(`pipeline.py:152-153, 161-162`). Rungs are the ladder of `model/ladder.py:29-37`: `suspicion`, `model_corroborated`, `model_entailed`, `execution_confirmed`. `at_rung` is the only way a finding rises above -suspicion (`ladder.py:58-71`). Findings are ordered `(rung, -region rank, site)` and deduplicated -per (site, family) keeping the strongest rung (`scan.py:1588-1614`). +suspicion (`ladder.py:58-71`). Findings are ordered `(rung, -region rank, site)`. They are +deduplicated per (site, family), keeping the strongest rung (`scan.py:1588-1614`). -Two other model-driven paths are independent of the graph. The plane's `verify` task is a metered -tool hunt over enumerated candidates (`PER_HUNT = 6`, `MAX_STEPS = 6`, `plane/tasks/verify.py:32-33`), -run as passes a and b; `agree` makes no model calls and marks a candidate agreed only when both -passes flagged it, with pass c re-hunting only the disputes at 2 of 3 (`plane/tasks/agree.py:20-26, -73-82, 131-134`). The decision engine consumes all of the above as feature blocks per candidate: -`quick`, `engine`, `facts`, `source`, `entry_points`, `roles`, `delta`, `verify`, `agree`, -`model_sinks` (`learn/schema.py:49-51`), where an instrument that did not run records a state, -"never a zero" (`learn/features.py:10-11`). Per-repository roles are inferred deterministically over -the facts: a function whose parameter reaches a sink call becomes a derived sink, a sanitizer if it -also returns the cleaned value, a source if it returns a source's value, to `MAX_DEPTH = 3`, and -"a derived sink alone is never a finding" (`learn/roles.py:21-22, 48, 345-405`). How those signals -become a decision is in [Memory and detection](memory-and-detection.md); the measured gap between -a tool-equipped hunter and the overlay on a Python holdout (12/14 against 5/14 once scoring was -class-aware) is in `benchmarks/measurements/2026-09-06-vibe-py-holdout-hunter-*.json`. +Two other model-driven paths are independent of the graph: + +- **The plane's `verify` task** is a metered tool hunt over enumerated candidates + (`PER_HUNT = 6`, `MAX_STEPS = 6`, `plane/tasks/verify.py:32-33`). It runs as passes a and b. +- **`agree`** makes no model calls. It marks a candidate agreed only when both passes flagged + it. Pass c re-hunts only the disputes, at 2 of 3 (`plane/tasks/agree.py:20-26, 73-82, + 131-134`). + +The decision engine consumes all of the above as feature blocks per candidate: `quick`, +`engine`, `facts`, `source`, `entry_points`, `roles`, `delta`, `verify`, `agree`, `model_sinks` +(`learn/schema.py:49-51`). An instrument that did not run records a state, "never a zero" +(`learn/features.py:10-11`). + +Per-repository roles are inferred deterministically over the facts, to `MAX_DEPTH = 3` +(`learn/roles.py:21-22, 48, 345-405`): + +- a function whose parameter reaches a sink call becomes a derived sink; +- it is a sanitizer if it also returns the cleaned value; +- it is a source if it returns a source's value. + +"A derived sink alone is never a finding". [Memory and detection](memory-and-detection.md) +explains how those signals become a decision. A tool-equipped hunter was measured against the +overlay on a Python holdout: 12/14 against 5/14 once scoring was class-aware. That gap is in +`benchmarks/measurements/2026-09-06-vibe-py-holdout-hunter-*.json`. ## 7. REGRESS: sandboxed reproduction @@ -537,39 +720,56 @@ flowchart LR v --> worth["worth_fixing if triggerable and reachable"] ``` -REGRESS runs only in `deep` mode (`stages.py:17`) and only when `docker info` succeeds; otherwise -the stage records `sandbox_unavailable` (`sandbox/probe.py:24-36`, `cli.py:691-693`). The top -`max_candidates = 5` hotspots by score plus forced severity-5 reachable inventory are selected -(`regress/candidate.py:63-88`, `config.py:157-160`). A snippet is the one a finding proposed, or a -default that only loads the file: Python compiles it, JavaScript reads it, C is `int main(void) -{ return 0; }` (`candidate.py:153-174`). Recipes exist for Python, JavaScript and C on -`python:3.12-alpine`, `node:22-alpine` and `gcc:13` (`regress/recipes.py:16-52`, -`candidate.py:30-34`); any other language is `inconclusive` with `missing_recipe`. +REGRESS runs only in `deep` mode (`stages.py:17`) and only when `docker info` succeeds. +Otherwise the stage records `sandbox_unavailable` (`sandbox/probe.py:24-36`, `cli.py:691-693`). -The deny-list is a regex check before any container starts (`regress/safety.py:63-75`): the Docker -socket, the word `socket`, the word `curl`, a host-network request, and writes under `/workspace` -in Python, Node, C and shell forms (`safety.py:21-60`). The real boundary is the container -(`sandbox/runner.py:113-137`): `--network none --read-only --tmpfs /scratch:rw,exec --memory -m --pids-limit --cap-drop ALL --security-opt no-new-privileges --user 65534:65534` with -the repository bind-mounted read-only at `/workspace`; the runner also refuses an argv containing +It selects the top `max_candidates = 5` hotspots by score, plus forced severity-5 reachable +inventory (`regress/candidate.py:63-88`, `config.py:157-160`). A snippet is the one a finding +proposed. If there is none, a default only loads the file (`candidate.py:153-174`): + +- Python compiles it; +- JavaScript reads it; +- C is `int main(void) + { return 0; }` (`candidate.py:153-174`). + +Recipes exist for Python, JavaScript and C on `python:3.12-alpine`, `node:22-alpine` and +`gcc:13` (`regress/recipes.py:16-52`, `candidate.py:30-34`). Any other language is +`inconclusive` with `missing_recipe`. + +The deny-list is a regex check before any container starts (`regress/safety.py:63-75`). It +rejects (`safety.py:21-60`): + +- the Docker socket; +- the word `socket`; +- the word `curl`; +- a host-network request; +- writes under `/workspace` in Python, Node, C and shell forms. + +The real boundary is the container (`sandbox/runner.py:113-137`): `--network none --read-only --tmpfs /scratch:rw,exec --memory +m --pids-limit --cap-drop ALL --security-opt no-new-privileges --user 65534:65534`. The +repository is bind-mounted read-only at `/workspace`. The runner also refuses an argv containing `--privileged`, the Docker socket or host networking (`runner.py:182-192`). Defaults are 2048 MB, 300 s and 512 pids (`config.py:104-109`). -Verdicts (`regress/verdict.py:10-26, 64-110`): a safety rejection, missing recipe or sandbox, or -timeout is `inconclusive`; exit 0 under a covering test is `already_covered`; with an oracle, -`triggerable` on `oracle_fired` else `not_triggerable`; without one, a nonzero exit is -`triggerable` with reason `sanitizer_abort`, `assertion`, `crash` or `nonzero_exit`, and exit 0 is -`not_triggerable`. `is_worth_fixing` requires `triggerable` and reachability `reachable` or +Verdicts (`regress/verdict.py:10-26, 64-110`): + +- a safety rejection, missing recipe or sandbox, or timeout is `inconclusive`; +- exit 0 under a covering test is `already_covered`; +- with an oracle: `triggerable` on `oracle_fired`, else `not_triggerable`; +- without an oracle: a nonzero exit is `triggerable` with reason `sanitizer_abort`, `assertion`, + `crash` or `nonzero_exit`, and exit 0 is `not_triggerable`. + +`is_worth_fixing` requires `triggerable` and reachability `reachable` or `inferred-file-surface` (`verdict.py:28, 59-61`). Verdicts feed the complexity ledger and -`rank_obligations`, which caps every obligation finding strictly below the lowest proven finding -(`cli.py:724-727`, `rank.py:243-265`). +`rank_obligations`. `rank_obligations` caps every obligation finding strictly below the lowest +proven finding (`cli.py:724-727`, `rank.py:243-265`). **What `execution_confirmed` means today.** The rung is defined as "a sandbox reproduced it -(deferred tier, memory/C first)" (`ladder.py:10, 33`) and ordered above `model_entailed`. No code -path assigns it: REGRESS produces the four verdicts above, not a rung, and nothing maps +(deferred tier, memory/C first)" (`ladder.py:10, 33`). It is ordered above `model_entailed`. No +code path assigns it. REGRESS produces the four verdicts above, not a rung. Nothing maps `triggerable` onto `Rung.EXECUTION_CONFIRMED`. The default snippets prove that a file loads, not -that a vulnerability fires; a reproduction needs a finding-proposed snippet or an oracle. Read the -rung as reserved. +that a vulnerability fires. A reproduction needs a finding-proposed snippet or an oracle. Read +the rung as reserved. ## 8. What each technique misses @@ -584,6 +784,7 @@ rung as reserved. | The model | Anything the enumerator did not propose (a hard ceiling on recall); the residual question is bounded to 8 candidates per region; a public fix the model may have memorised is indistinguishable from reasoning, which is why labels come from ground truth and the engine block is withheld where it tracks the fix | `candidates.py:7-10`; `pipeline.py:44`; `learn/schema.py:97-109` | | REGRESS | Anything without a Python, JavaScript or C recipe; anything the default load-only snippet cannot trigger; no finding reaches `execution_confirmed` today | `recipes.py:16-52`; `candidate.py:153-174`; `ladder.py:33` | -The three representations are kept separate on purpose: a finding carries the rung and the -degradations of the representation that produced it, so a reader can tell a flow the graph entailed -from a statement a regex matched, and "did not look" from "looked and found nothing". +The three representations are kept separate on purpose. A finding carries the rung and the +degradations of the representation that produced it. So a reader can tell a flow the graph +entailed from a statement a regex matched. A reader can also tell "did not look" from "looked +and found nothing". diff --git a/docs/evaluation.md b/docs/evaluation.md index 8f26a35..5023d5e 100644 --- a/docs/evaluation.md +++ b/docs/evaluation.md @@ -1,7 +1,8 @@ # Evaluation -What has been measured on code the tool was not tuned on, and how. State as of 2026-10-02. -Every number below is quoted from a committed record; the path is given with it. +This page shows what has been measured on code the tool was not tuned on, and how. State as of +2026-10-02. Every number below is quoted from a committed record, and the record's path is +given with it. ## Instruments @@ -11,32 +12,47 @@ Every number below is quoted from a committed record; the path is given with it. | Pair corpus | fire on the vulnerable function, stay silent on its fix | `ousast pairs`, `benchmarks/pairs/` | | Independent populations | real repositories at vulnerable, fixed and benign pins, scored under a pre-registered protocol | `benchmarks/independent/` | -The detection gate and the local pairs are CI gates. They catch regressions; they do not +Recall means the share of known bugs found. FP means false positives. + +The detection gate and the local pairs are CI gates. They catch regressions. They do not estimate recall on real code, because the rules were written against them. ## Pair corpora -`benchmarks/pairs/catalog.toml` and the slice directories (`local`, `github`, `sast`, `vfc`, -`vibe-py`, `vfc-js`, `agent-vfc`, `owasp`) are runnable with `ousast pairs --slice `; each -pair carries a provenance profile and a train/holdout split (`--profile`, `--split`). Pointer -pairs are not vendored: `--pointers` fetches them into `~/.cache/openultrasast/pairs/`. -`benchmarks/pairs/README.md` documents the schema and the review tiers. +A pair is a vulnerable function and its fixed version. You can run the catalog +`benchmarks/pairs/catalog.toml` and the slice directories with `ousast pairs --slice `. +The slices are `local`, `github`, `sast`, `vfc`, `vibe-py`, `vfc-js`, `agent-vfc` and `owasp`. + +- Each pair carries a provenance profile and a train/holdout split (`--profile`, `--split`). +- Pointer pairs are not vendored. `--pointers` fetches them into `~/.cache/openultrasast/pairs/`. +- `benchmarks/pairs/README.md` documents the schema and the review tiers. **`advisory-fixes`** (`benchmarks/pairs/advisory-fixes/`, 2026-10-01) holds 97 pointer pairs, -one per repository: a public CVE/GHSA fix commit whose advisory-named function is declared on -both sides and changed by the fix. It was assembled to give the decision engine real negatives -for families the corpus had few of (access control, deserialization, output encoding, path, -untrusted destination). It is a label source of the decision engine -(`src/openultrasast/learn/sources.toml`), not an `ousast pairs` slice; the research log and +one per repository. Each is a public CVE/GHSA fix commit. The function the advisory names is +declared on both sides and changed by the fix. + +It was assembled to give the decision engine real negatives (fixed, non-vulnerable code). It +targets families the corpus had few of: access control, deserialization, output encoding, +path, untrusted destination. It is a label source of the decision engine +(`src/openultrasast/learn/sources.toml`), not an `ousast pairs` slice. The research log and every rejection are in `research-2026-10-01.md`. ## Independent populations and the M4 gates -The pre-push safety net's qualification milestone (M4) requires, together: at least 95% -actionable precision, at least 90% supported recall, at least 95% supported-check completion, -the runtime gates, and zero fixed-side or benign false alerts. A population is frozen before -any of its cases is scanned, scored once under a pre-registered protocol, and then **spent**: -it may inform changes, but it can no longer qualify anything. +M4 is the qualification milestone of the pre-push safety net. It requires all of these +together: + +- at least 95% actionable precision; +- at least 90% supported recall; +- at least 95% supported-check completion; +- the runtime gates; +- zero fixed-side or benign false alerts. + +A population is a set of real repositories. Its lifecycle has three steps: + +1. It is frozen before any of its cases is scanned. +2. It is scored once under a pre-registered protocol. +3. It is then **spent**: it may inform changes, but it can no longer qualify anything. | Population | Status | Result | Record | | --- | --- | --- | --- | @@ -45,9 +61,9 @@ it may inform changes, but it can no longer qualify anything. | v2, model-driven pipeline | exploratory (v2 already spent) | stopped after 7 of 17 cases when the account emptied: recall 4/7; precision not adjudicated; no gate measurable | `results-v2-model-pipeline.json` | | v3 (PHP, 15 cases; one SSRF slot left empty by the selection rule) | frozen, untouched | none: it is the one-time final check, run once under `protocol-v3.md` after a prediction is committed | `population-v3-php.toml`, `freeze-v3-php.json` | -v3 is reserved: development and analysis work never reads its cases, and -`tests/test_independent_population.py` fails when anything in the development tree (a recipe, -catalog, manifest or measurement) names one of its repositories. +v3 is reserved. Development and analysis work never reads its cases. +`tests/test_independent_population.py` fails when anything in the development tree names one +of its repositories. That covers a recipe, catalog, manifest or measurement. ```mermaid stateDiagram-v2 @@ -66,9 +82,10 @@ stateDiagram-v2 ## Folds: how learned decisions are kept out of their own training data -The decision engine is evaluated on repositories it was not shown (`src/openultrasast/learn/folds.py`). -Every split is by repository group (the label builder's normalised `owner/name`, merged across -URLs, forks and shared advisories), across all corpora, seeded and deterministic. +The decision engine is evaluated on repositories it was not shown +(`src/openultrasast/learn/folds.py`). Every split is by repository group. A group is the label +builder's normalised `owner/name`, merged across URLs, forks and shared advisories. Splits +cover all corpora and are seeded and deterministic. ```mermaid flowchart TD @@ -81,44 +98,50 @@ flowchart TD rest --> lofo["Leave-one-framework-out: frameworks with at least 10 groups"] ``` -`assert_disjoint` runs before a compile: the compile split and the evaluation folds share no -group. A framework with fewer than 10 groups gets `insufficient data` instead of a fold. As of -2026-10-02 the outer folds have been run for the six evaluable families; leave-one-source-out -and leave-one-framework-out have not +`assert_disjoint` runs before a compile. It checks that the compile split and the evaluation +folds share no group. A framework with fewer than 10 groups gets `insufficient data` instead of +a fold. As of 2026-10-02, the outer folds have been run for the six evaluable families. +Leave-one-source-out and leave-one-framework-out have not ([decision-engine.md](decision-engine.md#second-slice-six-families-2026-10-02)). ## The plane increment -The first increment of the agentic plane ran the model-driven pipeline's validation set (46 -candidates in 15 cases of v2, 20 declared vulnerable sites) on google/ax and compared it with -the script pipeline on the same cases, candidates, pins and recorded triage. The pre-registered -gate: at least 16 of 20 declared sites agreed, at a cost per candidate under $0.022. +The first increment of the agentic plane ran on google/ax. It used the model-driven pipeline's +validation set: 46 candidates in 15 cases of v2, with 20 declared vulnerable sites. It was +compared with the script pipeline on the same cases, candidates, pins and recorded triage. + +The pre-registered gate: at least 16 of 20 declared sites agreed, at a cost per candidate under +$0.022. - **Increment 1** (2026-09-29, `benchmarks/independent/plane-increment-1.json`): 14 of 20 - declared sites agreed against the reference's 16; $0.0194 per candidate with recorded triage. - **NOT MET** on recall. - *Correction (2026-09-30, in the record):* the first version read agree's - `declared_sites_matched` (coverage by candidates, 19) as "found in either pass" and concluded - detection was unchanged. Recounted from the agreed rows, the plane's passes flagged 17 of 20 - declared sites in at least one pass against the reference's 19. -- **Increment 2** (2026-09-30, `benchmarks/independent/plane-increment-2.json`): after the design - revisit, a tie-break pass c on the 10 disputed candidates with 2-of-3 agreement. 16 of 20 - declared sites agreed (reference 16) at $0.0205 per candidate over 46 ($0.02195 over the 43 - after triage, against the reference's $0.022). **MET**, with two caveats the record carries: - the tie-break was chosen after seeing the first measurement on the same validation set, so - this compares plane with script and qualifies nothing on an independent population; and the - plane's passes still flag fewer declared sites than the reference (17 vs 19). Precision is - not part of this gate and remains the open problem. + declared sites agreed, against the reference's 16. Cost was $0.0194 per candidate with + recorded triage. **NOT MET** on recall. + *Correction (2026-09-30, in the record):* the first version misread agree's + `declared_sites_matched`. That field is coverage by candidates (19), but it was read as + "found in either pass". So the first version concluded detection was unchanged. Recounted + from the agreed rows, the plane's passes flagged 17 of 20 declared sites in at least one + pass. The reference flagged 19. +- **Increment 2** (2026-09-30, `benchmarks/independent/plane-increment-2.json`): came after the + design revisit. It added a tie-break pass c on the 10 disputed candidates, with 2-of-3 + agreement. 16 of 20 declared sites agreed (reference 16). Cost was $0.0205 per candidate over + 46, and $0.02195 over the 43 after triage (reference: $0.022). **MET**, with two + caveats the record carries: + 1. The tie-break was chosen after seeing the first measurement on the same validation set. + So this compares plane with script and qualifies nothing on an independent population. + 2. The plane's passes still flag fewer declared sites than the reference (17 vs 19). + + Precision is not part of this gate and remains the open problem. ## The decision engine -Two slices have been measured out of repository (injection, 2026-10-01; six families, -2026-10-02). The engine is not adopted. The state against the gates, in one place with the plane -increment and the populations above, is on [Where we stand](where-we-stand.md#1-detection-capability-and-goals-no-go-measured); -the per-family tables are on [decision-engine.md](decision-engine.md). +Two slices have been measured out of repository: injection (2026-10-01) and six families +(2026-10-02). The engine is not adopted. [Where we +stand](where-we-stand.md#1-detection-capability-and-goals-no-go-measured) shows the state +against the gates in one place, with the plane increment and the populations above. The +per-family tables are on [decision-engine.md](decision-engine.md). ## Removal of the earlier agentic extra -The removal of the earlier agentic extra (2026-09-30) was checked for byte-identical -deterministic outputs (pair replay, quick benchmark, standard scan, `improve --dry-run`): +The earlier agentic extra was removed on 2026-09-30. The removal was checked for byte-identical +deterministic outputs: pair replay, quick benchmark, standard scan, `improve --dry-run`. Records: `benchmarks/measurements/2026-09-30-harnessx-removal-baseline/` and `-equality/`.