marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: plain-language pass -- detection techniques, architecture, evaluation, decision engine

Browse Source 
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-10-02 21:03:24 +02:00
co-authored by Claude Opus 5.5
parent 50f4e69321
commit 352f8c07aa
4 changed files with 845 additions and 534 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture.md
+114 -67
View File
@@ -1,14 +1,22 @@
# Architecture
How a scan works, how a finding earns its place, how severity and the project score are
decided, and how the ruleset improves itself. State as of 2026-10-02; module paths are under
`src/openultrasast/`.
This page covers four things:
- how a scan works;
- how a finding earns its place;
- how severity and the project score are decided;
- how the ruleset improves itself.
State as of 2026-10-02; module paths are under `src/openultrasast/`.
## The scan pipeline
The CLI is thin; `HarnessRuntime` owns the lifecycle and writes an event trace for every stage
(`trace/events.jsonl`). The stages a mode requests are `MODE_STAGES` in `stages.py`:
`quick` runs STATIC, `standard` adds MAP, `deep` adds REGRESS.
The CLI is thin. `HarnessRuntime` owns the lifecycle. It writes an event trace for every stage
(`trace/events.jsonl`). `MODE_STAGES` in `stages.py` lists the stages each mode requests:
- `quick` runs STATIC;
- `standard` adds MAP;
- `deep` adds REGRESS.
```
policy_load ─▶ ruleset_load ─▶ policy_check (fail loud on an ungoverned CWE)
@@ -28,14 +36,19 @@ policy_load ─▶ ruleset_load ─▶ policy_check (fail loud on an
```
**The model layer** (`model/`, `cpg/`) is the core of `standard`. It builds one code property
graph per scan with Joern (`joern-parse`, then batched `joern --script` queries from
`cpg/queries/*.sc`) and dispatches each question to the arbiter of its spec type: taint
(source to sink with no discharging sanitizer), dominance (does a guard govern the obligated
operation) and configuration (constant abstraction over a value). Taint facts ship for C,
Java, JavaScript, PHP and Python (`ruleset/semantic/*.toml`). Without Joern the layer records
`cpg_unavailable` and the scan is otherwise unchanged; the Docker image ships Joern and
`php-cli` (which the PHP frontend needs). The representations and techniques behind each stage
are explained in [Detection techniques](detection-techniques.md).
graph (CPG: a graph of the program's syntax, control flow and data flow) per scan with Joern.
It runs `joern-parse`, then batched `joern --script` queries from `cpg/queries/*.sc`. Each
question goes to the arbiter (a checker that decides one kind of question) for its spec type:
- taint (untrusted input reaching a dangerous call): source to sink with no discharging sanitizer;
- dominance: does a guard govern the obligated operation;
- configuration: constant abstraction over a value.
Taint facts ship for C, Java, JavaScript, PHP and Python (`ruleset/semantic/*.toml`). Without
Joern, the layer records `cpg_unavailable` and the rest of the scan is unchanged. The Docker
image ships Joern and `php-cli`, which the PHP frontend needs. [Detection
techniques](detection-techniques.md) explains the representations and techniques behind each
stage.
The model layer reports on a rung ladder:
@@ -47,23 +60,29 @@ model_corroborated the graph found the shape; a judge agreed on the residual
suspicion proposed, nothing corroborated it
```
The LLM (`model/pipeline.py`, `residual_question`) is asked only the residual question the graph
cannot settle, for a flow the graph already found; it can confirm or contradict, never create a
flow. When the graph resolved nothing, the enumerator's candidates (at most
`MAX_JUDGED_CANDIDATES`, 8) are put to it, and anything it affirms is reported at `suspicion`.
Without a provider key neither question is asked and only the graph's entailed findings are
reported.
The LLM answers only the residual question: the part the graph cannot settle
(`model/pipeline.py`, `residual_question`). It is asked only about a flow the graph already
found. It can confirm or contradict that flow, but never create one. When the graph resolved
nothing, the LLM sees the enumerator's candidates (at most `MAX_JUDGED_CANDIDATES`, 8).
Anything it affirms is reported at `suspicion`. Without a provider key, neither question is
asked. Only the graph's entailed findings are reported then.
**REGRESS** (`regress/`, `sandbox/`) runs only when `docker info` succeeds. Each promoted
candidate gets a small language recipe (Python, JavaScript, C) that loads the candidate's
source inside a container with no network, a read-only `/workspace`, a non-root user and the
`[sandbox]` limits. A structural deny-list rejects snippets mentioning sockets, curl, host
networking or writes under `/workspace` before anything runs.
candidate gets a small language recipe (Python, JavaScript, C). The recipe loads the
candidate's source inside a container with:
- no network;
- a read-only `/workspace`;
- a non-root user;
- the `[sandbox]` limits.
Before anything runs, a structural deny-list rejects snippets that mention sockets, curl, host
networking or writes under `/workspace`.
## Evidence ladder: how a false positive is eliminated
Evidence level is a state machine, not a label a model may assign. The verifier
(`verification.py`) runs on independent context and enforces:
(`verification.py`) runs on independent context and enforces it:
```
suspicion ─▶ static_corroboration ─▶ crash_reproduced ─▶ root_cause_explained
@@ -75,31 +94,36 @@ suspicion ─▶ static_corroboration ─▶ crash_reproduced ─▶ root_cause_
call-graph, route, CLI, parser or dynamic evidence;
- `static_corroboration` with function-level reachability: `ACCEPTED`.
A rejected finding becomes a **scoped false-positive learning** (`calibration.py`) with a
reason from a fixed taxonomy (`unreachable_path`, `missing_attacker_control`,
A rejected finding becomes a **scoped false-positive learning** (`calibration.py`). It has a
scope and a reason from a fixed taxonomy: `unreachable_path`, `missing_attacker_control`,
`sanitizer_disproved`, `static_rule_mismatch`, `incorrect_model_assumption`, `duplicate`,
`insufficient_impact`, `unsupported`, `contradicted`, `unverified`) and a scope. A rejection in
`auth/` demotes only `auth/...`, never the whole class
`insufficient_impact`, `unsupported`, `contradicted`, `unverified`. A rejection in `auth/`
demotes only `auth/...`, never the whole class
(`tests/test_calibration.py::test_scoped_false_positive_demotion_does_not_suppress_class_globally`).
## How a false negative is surfaced
Every expected vulnerability that no finding matched becomes a `BenchmarkCalibrationRecord` in
`calibration_records.json`, with a `next_improvement_candidate` (a static rule, SARIF source,
entry-point mapping, retrieval package, hunter prompt, dynamic reproducer or skill route).
Once a miss is closed the detection gate keeps it closed in CI.
An expected vulnerability that no finding matched becomes a `BenchmarkCalibrationRecord` in
`calibration_records.json`. Each record names a `next_improvement_candidate`: a static rule,
SARIF source, entry-point mapping, retrieval package, hunter prompt, dynamic reproducer or
skill route. Once a miss is closed, the detection gate keeps it closed in CI.
## Central CWE policy and the project score
**Policy** (`policy/verycode.py`): a vendored `CWE_Score.tsv` (verycode-policies) is the single
source of truth. Each CWE carries a flaw category, a severity (0-5) and `static`/`dynamic` scope
flags. A finding's severity comes only from policy, keyed on CWE. CWEs that are not `static`
resolve to 0 and are report-only. `assert_rules_resolve` runs as the `policy_check` stage: an
enabled rule naming an ungoverned CWE aborts the scan. Verycode has no CWE-120, so the C/C++
source of truth. Each CWE carries:
- a flaw category;
- a severity (0-5);
- `static`/`dynamic` scope flags.
A finding's severity comes only from policy, keyed on CWE. CWEs that are not `static` resolve
to 0 and are report-only. `assert_rules_resolve` runs as the `policy_check` stage. An enabled
rule that names an ungoverned CWE aborts the scan. Verycode has no CWE-120, so the C/C++
buffer rules carry CWE-121.
**Score** (`scoring/project_score.py`): each finding's penalty is
`SEV_WEIGHT[severity] x REACH_MULT[reachability]`, and the score is `100 * e^(-total/k)` with
`SEV_WEIGHT[severity] x REACH_MULT[reachability]`. The score is `100 * e^(-total/k)` with
`k = 60`.
| Severity | Weight | | Reachability | Multiplier |
@@ -109,10 +133,13 @@ buffer rules carry CWE-121.
| 3 | 10 | | `unknown` | 0.4 |
| 2 / 1 / 0 | 2 / 1 / 0 | | | |
A confirmed false positive lowers a finding's reachability multiplier instead of deleting the
rule. A severity-5 reachable finding always fails the score gate; `score < min_score`
(default 80) fails it only with `[score] blocking = true`. The `score` stage writes
`score.json` and merges it into `manifest.json`.
A confirmed false positive lowers a finding's reachability multiplier. The rule is not
deleted. The score gate works like this:
- a severity-5 reachable finding always fails it;
- `score < min_score` (default 80) fails it only with `[score] blocking = true`.
The `score` stage writes `score.json` and merges it into `manifest.json`.
## Self-improvement
@@ -124,41 +151,61 @@ scan ─▶ verify ─▶ non-accepted findings ─▶ scoped learnings
next scan ─▶ rank ─▶ calibrate (demote rejected scopes) ─▶ findings
```
`record_calibration` (after `verify`) merges this run's non-accepted outcomes into the ledger,
de-duplicated by finding ID; `calibrate` (after `rank`) demotes the priority of every scope
that produced a rejection and writes `applied_calibrations.json`. Demotion is scoped and
reversible (`tests/test_pipeline_calibration.py`).
1. `record_calibration` runs after `verify`. It merges this run's non-accepted outcomes into
the ledger, de-duplicated by finding ID.
2. `calibrate` runs after `rank`. It demotes the priority of every scope that produced a
rejection and writes `applied_calibrations.json`.
Demotion is scoped and reversible (`tests/test_pipeline_calibration.py`).
### Rule-level loop: `ousast improve`
Each round (`improve/evolve.py`): benchmark with the current ledger, per-rule signals
(`rule_signals.json`), bounded proposals, validation (`improve/validator.py`), replay smoke,
re-benchmark, and a hard gate. The levers are rule status (`enabled -> shadow` before
`disabled`) and score constants; the loop can never edit pattern text
or the authoritative CWE severity. A round is accepted only if recall >= 90%, FP < 10%, the
project score did not regress and no matched finding was lost; otherwise the ledger is not
written. With `--pair-catalog`, every round is also gated per provenance profile on the
catalog's holdout pairs (`--no-pair-gate` skips this and is not for accepted ledgers). The same
detection gate runs in CI (`python -m openultrasast.gate`).
Each round (`improve/evolve.py`) runs these steps:
1. benchmark with the current ledger;
2. per-rule signals (`rule_signals.json`);
3. bounded proposals;
4. validation (`improve/validator.py`);
5. replay smoke;
6. re-benchmark;
7. a hard gate.
The levers are rule status (`enabled -> shadow` before `disabled`) and score constants. The
loop can never edit pattern text or the authoritative CWE severity. A round is accepted only if
all of these hold:
- recall >= 90%;
- FP < 10%;
- the project score did not regress;
- no matched finding was lost.
Otherwise the ledger is not written. With `--pair-catalog`, each round is also gated per
provenance profile on the catalog's holdout pairs. `--no-pair-gate` skips this check; do not
use it for accepted ledgers. The same detection gate runs in CI (`python -m openultrasast.gate`).
### Proposals from plane memory
With `--memory [STORE]`, `ousast improve` also reads the rows plane runs stored per repository
and pin (`improve/memory.py`). Two deterministic rules turn them into rule-status proposals: a
rule whose alerts are repeatedly false across repositories is shadowed, and a shadow rule that
keeps hitting agreed, declared vulnerable sites is re-enabled. Rows from holdout pairs, from the
gated manifest's own cases and from any `--qualify-population` are dropped before any rule sees
them. Each proposal records the rows it came from (`memory_proposals.jsonl` next to the
journal) and goes through the same validator and gate.
With `--memory [STORE]`, `ousast improve` also reads the rows that plane runs stored per
repository and pin (`improve/memory.py`). Two deterministic rules turn these rows into
rule-status proposals:
- a rule whose alerts are repeatedly false across repositories is shadowed;
- a shadow rule that keeps hitting agreed, declared vulnerable sites is re-enabled.
Some rows are dropped before any rule sees them: rows from holdout pairs, from the gated
manifest's own cases and from any `--qualify-population`. Each proposal records the rows it
came from (`memory_proposals.jsonl` next to the journal). It then goes through the same
validator and gate.
### The loop as a plane Run
`ousast plane workspaces <population> --validation-set ... --loop` appends the improvement loop
to a generated Run: `alerts` per case (the quick scan on the vulnerable and fixed pins), then
the steps `snapshot` (task `memory-snapshot`), `measure`, `propose` and `improve` (`plane/tasks/loop.py`), each model-free
with a zero budget. The snapshot applies the same train-on-test guard before any row leaves the
store. A recorded live run: `benchmarks/measurements/2026-09-30-harnessx-removal-loop/`.
to a generated Run. The Run holds `alerts` per case: the quick scan on the vulnerable and fixed
pins. Then come the steps `snapshot` (task `memory-snapshot`), `measure`, `propose` and
`improve` (`plane/tasks/loop.py`). Each step is model-free with a zero budget. The snapshot
applies the same train-on-test guard before any row leaves the store. A recorded live run:
`benchmarks/measurements/2026-09-30-harnessx-removal-loop/`.
For PHP, which quick mode covered with zero rules when that run was made, `ousast plane
When that run was made, quick mode covered PHP with zero rules. So for PHP, `ousast plane
alerts-engine` produces a Run's `alerts` from the engine image on the host instead
(`benchmarks/measurements/2026-09-30-php-engine-alerts/`).