marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: plain-language pass -- README, index, where we stand, scanning, examples, threat model

Browse Source 
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-10-02 21:01:10 +02:00
co-authored by Claude Opus 5.5
parent 50f4e69321
commit 552671ca6a
6 changed files with 685 additions and 440 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+238 -144
View File
@@ -1,38 +1,52 @@
# OpenUltraSAST
OpenUltraSAST is an experimental static security analyser meant to work as a safety net on
repositories it has never seen: it scans a checkout, or the commits a `git push` would publish,
and separates what it can prove from what it merely suspects. A scan combines language-scoped
pattern rules, a tree-sitter overlay, a Joern code property graph and, optionally, a model that is
asked only the question the graph cannot settle; every finding carries the evidence that earned
its place. A scan that could not look at something records a degradation instead of reporting a
clean result. `ousast` is the interface; model work over whole repositories runs on a separate
agentic plane (google/ax), and a learned decision engine is in development. Neither is needed
for a scan.
OpenUltraSAST is an experimental static security analyser. It is meant to be a safety net on
repositories it has never seen. It scans a checkout, or the commits a `git push` would publish. It
keeps what it can prove apart from what it only suspects.
This file is the front door; each section links to its page of the documentation site (`docs/`,
A scan combines up to four parts:
- language-scoped pattern rules;
- a tree-sitter overlay (a syntax tree of each file, built by the tree-sitter parser);
- a Joern code property graph (a graph of the program's syntax, control flow and data flow);
- optionally, a model. It is asked only the question the graph cannot settle.
Every finding carries the evidence that earned its place. When a scan could not look at
something, it records a degradation. It does not report a clean result.
`ousast` is the interface. Model work over whole repositories runs on a separate agentic plane
(google/ax). A learned decision engine is in development. A scan needs neither of them.
This file is the front door. Each section links to its page of the documentation site (`docs/`,
`mkdocs.yml`; `uv sync --extra docs && uv run mkdocs serve`).
## Status
**v2.0.0, state as of 2026-10-02.** The pre-push safety net is **NO-GO** against every M4 gate:
no hook capability is enabled, and the two independent populations spent so far gave engine recall
1 of 11 and then 0 of 17. The learned decision engine ranks but may not block: BLOCK is offered for
no family. The project has stopped trying to reach the gates by hand; every detection change now
has to win a controlled measurement on held-out repositories. The one page that states all of this
with the record behind each figure is [docs/where-we-stand.md](docs/where-we-stand.md); the
release history is in [RELEASE_NOTES.md](RELEASE_NOTES.md).
**v2.0.0, state as of 2026-10-02.** The pre-push safety net is **NO-GO** against every M4 gate.
- No hook capability is enabled.
- Two independent populations (case sets frozen before any scan) have been spent so far. Engine
recall was 1 of 11 and then 0 of 17.
- The learned decision engine ranks but may not block. BLOCK is offered for no family.
The project has stopped trying to reach the gates by hand. Every detection change now has to win
a controlled measurement on held-out repositories. One page states all of this, with the record
behind each figure: [docs/where-we-stand.md](docs/where-we-stand.md). The release history is in
[RELEASE_NOTES.md](RELEASE_NOTES.md).
## Install and first scan
Which tier you need: a deterministic scan of a checkout or a CI gate is tier 1; the Joern engine
(`standard`, `deep`) is tier 2; running model work over whole repositories on the plane is tier 3,
for maintainers; tier 4 does not exist yet.
Pick the tier you need:
1. Tier 1: a deterministic scan of a checkout, or a CI gate.
2. Tier 2: the Joern engine (`standard`, `deep`).
3. Tier 3, for maintainers: model work over whole repositories on the plane.
4. Tier 4 does not exist yet.
### Tier 1: quick scan, no Docker
The core install depends only on PyYAML. From a checkout, `uv sync` (or `pip install .`) installs
the `ousast` entry point; `uv sync --group dev` adds the test and lint tools.
the `ousast` entry point. `uv sync --group dev` adds the test and lint tools.
```bash
uv sync
@@ -41,19 +55,26 @@ uv run ousast scan . --mode quick --fail-on verified # CI: exit 1 on any evide
```
This runs the language-scoped pattern rules, entry-point reachability hints, ranking,
verification and scoring: deterministic, reproducible, no network, no keys. What it cannot do:
there is no engine, so nothing follows a value across statements or files, and the PHP quick rules
are a low-precision, in-sample development set (per-rule precision lower bounds from 0.03 to 1.0 on
the real pins, `benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`). The optional
`semantic` extra (`uv sync --extra semantic`) installs the tree-sitter grammars for the overlay.
verification and scoring. It is deterministic and reproducible. It needs no network and no keys.
What it cannot do:
- There is no engine. Nothing follows a value across statements or files.
- The PHP quick rules are a low-precision, in-sample development set. Per-rule precision lower
bounds run from 0.03 to 1.0 on the real pins
(`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`).
The optional `semantic` extra (`uv sync --extra semantic`) installs the tree-sitter grammars for
the overlay.
### Tier 2: standard and deep scans (Docker)
`standard` and `deep` need Joern. The Docker image `openultrasast:dev` ships the tool, Joern
v4.0.625 and `php-cli` (needed by the PHP frontend) in one image; it is built locally from the
`Dockerfile` (no published image yet), and it is big: a JRE plus about 2 GB of Joern. The shell
wrapper builds it on first use and then runs `ousast` in the container with your directory mounted
read-only and the container network off:
`standard` and `deep` need Joern. The Docker image `openultrasast:dev` ships three things in one
image: the tool, Joern v4.0.625 and `php-cli` (the PHP frontend needs it). You build it locally
from the `Dockerfile`; there is no published image yet. It is big: a JRE plus about 2 GB of Joern.
The shell wrapper builds the image on first use. Then it runs `ousast` in the container. Your
directory is mounted read-only and the container network is off.
```bash
source /path/to/OpenUltraSAST/ops/shell/ousast.sh # PowerShell: ops/shell/ousast.ps1
@@ -62,90 +83,129 @@ ousast-with-judge scan . --mode standard # network on, the model is a
```
`docker compose build` and `TARGET=/path/to/repo docker compose run --rm ousast scan /target` do
the same without the wrapper. `deep` adds REGRESS, which runs each reproduction snippet with
`docker run` from wherever the scan runs; the engine container has no Docker socket, so run `deep`
from a host install with Joern on the PATH (`pyinfra @local ops/joern.py`, pinned and checksum-
verified) and a working `docker`, or accept the recorded `sandbox_unavailable` degradation.
the same without the wrapper.
`deep` adds REGRESS. REGRESS runs each reproduction snippet with `docker run` from wherever the
scan runs. The engine container has no Docker socket. So run `deep` in one of two ways:
- from a host install with Joern on the PATH (`pyinfra @local ops/joern.py`, pinned and checksum-
verified) and a working `docker`;
- or accept the recorded `sandbox_unavailable` degradation.
Details, including the host install, are in [ops/README.md](ops/README.md).
Keys and store settings live in `.env` in the working directory (gitignored; `.env.example` lists
the names). `DEEPSEEK_API_KEY` serves every LLM call; `OPENROUTER_API_KEY` is used for embeddings
only (`ousast learn memory embed`). `.env` never overrides a variable already exported in your
shell, so `unset` a stale export when a key change seems to have no effect. The plane's memory
store defaults to a local file store under `~/ousast-results/plane/memory`; an S3-compatible bucket
is optional (`uv sync --extra s3`, then `OUSAST_MEMORY=s3://<bucket>` with `S3_ENDPOINT` and the
AWS credentials in `.env`). Nothing in a scan needs it.
Keys and store settings live in `.env` in the working directory. The file is gitignored;
`.env.example` lists the names.
- `DEEPSEEK_API_KEY` serves every LLM call.
- `OPENROUTER_API_KEY` is used for embeddings only (`ousast learn memory embed`).
- `.env` never overrides a variable already exported in your shell. If a key change seems to have
no effect, `unset` the stale export.
- The plane's memory store defaults to a local file store under `~/ousast-results/plane/memory`.
- An S3-compatible bucket is optional: `uv sync --extra s3`, then `OUSAST_MEMORY=s3://<bucket>`
with `S3_ENDPOINT` and the AWS credentials in `.env`. Nothing in a scan needs it.
### Tier 3: the plane on ax (maintainers)
None of this is needed for tier 1 or 2. Today's profile is a single-node kind cluster on the
maintainer's host. `ops/ax/up.sh` needs `docker`, `go`, `kubectl`, `kind`, `ko` and `ax` on the
PATH (it refuses to start when one is missing) and builds Agent Substrate and ax from their
checkouts under `~/.cache/ousast/ax-src/`. Measured idle footprint with two workers: the kind node
takes 1.7 GiB of memory (27% of the 7.7 GiB host) and 4.9 GB of disk for images, with several GB
more of Go caches during the build; the worker pool is sized for that 7 GB host. The verify and
roles tasks need `DEEPSEEK_API_KEY`, and memory lives in the local file store or the S3 store
(RustFS is the tested server). Bring-up, doctor and the smoke Run: [ops/ax/README.md](ops/ax/README.md).
Tier 1 and 2 need none of this. Today's profile is a single-node kind cluster on the maintainer's
host.
- `ops/ax/up.sh` needs `docker`, `go`, `kubectl`, `kind`, `ko` and `ax` on the PATH. It refuses to
start when one is missing.
- It builds Agent Substrate and ax from their checkouts under `~/.cache/ousast/ax-src/`.
- Measured idle footprint with two workers: the kind node takes 1.7 GiB of memory (27% of the
7.7 GiB host) and 4.9 GB of disk for images. The build uses several GB more for Go caches.
- The worker pool is sized for that 7 GB host.
- The verify and roles tasks need `DEEPSEEK_API_KEY`.
- Memory lives in the local file store or the S3 store. RustFS is the tested server.
Bring-up, doctor and the smoke Run: [ops/ax/README.md](ops/ax/README.md).
### Tier 4: Kubernetes (k3s), planned
No deployment to any cluster other than that kind cluster has been made. The target is two
profiles of one code base, `kind` locally and `k3s` in production, where the same `ousast plane run`
submits a Run in a remote execution mode: it needs a kubeconfig context or the cluster's ingress
(and an `imagePullSecret` on the worker ServiceAccount for a private package), runner and engine
images published to GHCR and pinned by digest, and the S3 store as the artifact path, with no
receiver on a laptop. The plan, and the code changes it needs with file paths, are in
Nothing has been deployed to any cluster other than that kind cluster. The target is two profiles
of one code base: `kind` locally and `k3s` in production. The same `ousast plane run` would
submit a Run in a remote execution mode. It needs:
- a kubeconfig context or the cluster's ingress (and an `imagePullSecret` on the worker
ServiceAccount for a private package);
- runner and engine images published to GHCR and pinned by digest;
- the S3 store as the artifact path, with no receiver on a laptop.
The plan, and the code changes it needs with file paths, are in
[docs/deployment.md](docs/deployment.md#production-topology-ax-on-a-kubernetes-cluster).
## Scan modes
`quick` runs the pattern rules, hints, ranking, verification and score and needs nothing.
`standard` adds the MAP stage: complexity map, semantic overlay, authorization obligations and the
Joern model layer, which builds one code property graph per repository and decides taint,
guard-dominance and configuration questions on it; a model is asked only the residual question
(without a key it is skipped and only the graph's entailed findings are reported). `deep` adds
REGRESS: promoted candidates are loaded by a small snippet in a Docker sandbox and get a
`triggerable` or `not_triggerable` verdict. A missing engine is a recorded degradation
(`cpg_unavailable`, `sandbox_unavailable`), never a clean result.
There are three modes. Each adds to the one before.
- `quick` runs the pattern rules, hints, ranking, verification and score. It needs nothing.
- `standard` adds the MAP stage: complexity map, semantic overlay, authorization obligations and
the Joern model layer. The model layer builds one code property graph per repository. It decides
taint, guard-dominance and configuration questions on that graph. A model is asked only the
residual question. Without a key, that question is skipped and only the graph's entailed
findings are reported.
- `deep` adds REGRESS. Promoted candidates are loaded by a small snippet in a Docker sandbox. Each
gets a `triggerable` or `not_triggerable` verdict.
A missing engine is a recorded degradation (`cpg_unavailable`, `sandbox_unavailable`), never a
clean result.
`--fail-on never|findings|verified|worth-fixing` sets the exit code. `--config openultrasast.toml`
loads settings. Every run writes these files under `<target>/.openultrasast/runs/<scan-id>/`:
`manifest.json`, `findings.json`, `verification.json`, `score.json`, `report.md`, `report.sarif`
and `trace/events.jsonl`.
`ousast pre-push` (experimental) analyses the commits a push would publish against their base.
Only new or worsened defects become candidates. It is advisory by default. Its capability registry
is empty, so it emits no normal alert today.
`--fail-on never|findings|verified|worth-fixing` sets the exit code and `--config openultrasast.toml`
loads settings. Every run writes `manifest.json`, `findings.json`, `verification.json`,
`score.json`, `report.md`, `report.sarif` and `trace/events.jsonl` under
`<target>/.openultrasast/runs/<scan-id>/`. `ousast pre-push` (experimental) analyses the commits a
push would publish against their base so that only new or worsened defects are candidates; it is
advisory by default and its capability registry is empty, so no normal alert is emitted today.
Flows and the per-language rule table: [docs/scanning.md](docs/scanning.md).
## How detection works
Three representations, from cheap to expensive: text (regex quick rules, one statement at a time),
tree (the tree-sitter CST and an IR per function, for the overlay and the obligations) and graph
(the Joern CPG, the only one that follows a value across statements, calls and files). Taint runs
as fact tables, queries and verdicts on the graph; absence bugs (an operation that should have
been guarded) are found as obligations, dominance and configuration census; a model answers only
the residual question. [docs/detection-techniques.md](docs/detection-techniques.md) explains each
technique with the code it comes from and what it misses;
[docs/architecture.md](docs/architecture.md) gives the stage order, the evidence ladder (a state
machine, not a label a model may assign), the central CWE policy and the project score, and the
self-improving loops (`ousast improve`, with `--memory` for proposals from plane rows).
The tool uses three representations of code, from cheap to expensive:
Quick rules cover C/C++, Python, JavaScript/TypeScript, Java with Groovy templates, and PHP, and
run only against files of their language. The detection gate (`uv run python -m openultrasast.gate`)
enforces at least 90% recall and under 10% false positives per language on the bundled cheat-sheet
corpora; on this tree it reports 93.62% recall (44/47) and 0.0% false positives. Those corpora show
that a rule change did not break a known detection; they are not a recall estimate on real code,
and PHP is not in the gate. Framework knowledge (WordPress, Flask, Django, Express, Spring) is
tagged as an optional prior in `src/openultrasast/ruleset/frameworks.toml`, so its contribution can
be measured and switched off.
1. Text: regex quick rules, one statement at a time.
2. Tree: the tree-sitter CST (concrete syntax tree) and an IR (intermediate representation) per
function. The overlay and the obligations use it.
3. Graph: the Joern CPG. It is the only one that follows a value across statements, calls and
files.
Taint runs as fact tables, queries and verdicts on the graph. Absence bugs are operations that
should have been guarded. They are found as obligations, dominance and configuration census. A
model answers only the residual question.
- [docs/detection-techniques.md](docs/detection-techniques.md) explains each technique, with the
code it comes from and what it misses.
- [docs/architecture.md](docs/architecture.md) gives the stage order and the evidence ladder (a
state machine, not a label a model may assign). It also covers the central CWE policy, the
project score and the self-improving loops (`ousast improve`, with `--memory` for proposals from
plane rows).
Quick rules cover C/C++, Python, JavaScript/TypeScript, Java with Groovy templates, and PHP. Each
rule runs only against files of its language.
The detection gate (`uv run python -m openultrasast.gate`) enforces at least 90% recall and under
10% false positives per language on the bundled cheat-sheet corpora. On this tree it reports
93.62% recall (44/47) and 0.0% false positives. Those corpora show that a rule change did not
break a known detection. They are not a recall estimate on real code, and PHP is not in the gate.
Framework knowledge (WordPress, Flask, Django, Express, Spring) is tagged as an optional prior in
`src/openultrasast/ruleset/frameworks.toml`. So its contribution can be measured and switched off.
## The plane on ax
Model-driven work over whole repositories runs as a Run manifest, a DAG of ax Tasks, each in its
own gVisor actor on Agent Substrate. ax is the only agentic executor; there is no local
subprocess path. Each Task binds at most one Model and its own `usd`/`calls` budget, egress is
deny-by-default per Task (hostnames only), and the provider key travels only in the Task's start
request, never in a manifest or an image. Measured on the first increment: the plane agreed on 16
of 20 declared sites at $0.0205 per candidate.
Model-driven work over whole repositories runs as a Run manifest. A Run is a DAG of ax Tasks. Each
Task runs in its own gVisor actor (an isolated sandbox) on Agent Substrate. ax is the only agentic
executor; there is no local subprocess path.
- Each Task binds at most one Model and its own `usd`/`calls` budget.
- Egress is deny-by-default per Task, by hostname only.
- The provider key travels only in the Task's start request, never in a manifest or an image.
Measured on the first increment: the plane agreed on 16 of 20 declared sites at $0.0205 per
candidate.
```bash
uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image
@@ -155,42 +215,61 @@ uv run ousast plane remember validation-46 # ingest the run's rows
```
`workspaces`, `harvest`, `alerts-engine` and `memory-normalise` generate Runs, their inputs and
the store's rewrites. The flow and the task catalogue: [docs/plane.md](docs/plane.md); what runs
where today and what a separate cluster would take: [docs/deployment.md](docs/deployment.md).
the store's rewrites. The flow and the task catalogue: [docs/plane.md](docs/plane.md). What runs
where today, and what a separate cluster would take: [docs/deployment.md](docs/deployment.md).
HarnessX, the earlier agentic extra, was retired 2026-09-30: a `[harnessx]` section is ignored
with one warning, and `[models] verifier`, `[fusion] panel_model` and `[fusion] decider_model` fail
naming the plane replacement, as do config keys nothing ever read. Fusion is deterministic.
HarnessX, the earlier agentic extra, was retired 2026-09-30. A `[harnessx]` section is ignored
with one warning. `[models] verifier`, `[fusion] panel_model` and `[fusion] decider_model` fail and
name the plane replacement. Config keys that nothing ever read fail the same way. Fusion is
deterministic.
## Memory
One store, keyed by repository and pin, holds what plane runs and the decision engine learned:
rows (facts, verdicts, unit costs, alerts, coverage, feature records) and content-addressed blobs
(excerpts, embeddings, cached responses, compiled programs). `OUSAST_MEMORY` selects the backend:
a local file store, or `s3://<bucket>` on any S3-compatible server with versioning, lifecycle
rules, object tags and S3 Select; RustFS is the tested server. An admin sets the bucket up once;
the store verifies it at every start and refuses to run, naming each missing piece, rather than
fall back. Layout and the read and write paths: [docs/memory.md](docs/memory.md); the bucket
policy and admin steps: [docs/rustfs.md](docs/rustfs.md); how stored rows are meant to turn into
better decisions: [docs/memory-and-detection.md](docs/memory-and-detection.md).
One store holds what plane runs and the decision engine learned. It is keyed by repository and pin.
It holds two kinds of data:
- rows: facts, verdicts, unit costs, alerts, coverage and feature records;
- content-addressed blobs: excerpts, embeddings, cached responses and compiled programs.
`OUSAST_MEMORY` selects the backend. It is either a local file store or `s3://<bucket>` on any
S3-compatible server. The S3 server must support versioning, lifecycle rules, object tags and S3
Select. RustFS is the tested server.
An admin sets the bucket up once. The store verifies it at every start. If anything is missing, it
refuses to run and names each missing piece. It does not fall back.
- Layout and the read and write paths: [docs/memory.md](docs/memory.md).
- The bucket policy and admin steps: [docs/rustfs.md](docs/rustfs.md).
- How stored rows are meant to turn into better decisions:
[docs/memory-and-detection.md](docs/memory-and-detection.md).
## Decision engine and evaluation
A learned decision engine is being built to replace hand-tuned detection edits: every instrument
produces signals, never verdicts; labels come from ground truth only; a compiled AI classifier with
local memory turns retrieved similar cases and the candidate's signals into a calibrated
probability with BLOCK and ADVISORY operating points. It is **in development, not adopted**: no
scan, pre-push check or report uses it. Measured on six families, on repositories the compiled
program was not shown, the pooled AUC of its score runs from 0.77 (injection) to 0.96 (access
control); the stricter within-pair AUC is 0.72 to 0.93; BLOCK is offered for no family
(`benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`). Maintainer surface:
`ousast learn labels|memory|compile|evaluate|curve|experiment|audit-leaks`. Design, slices and
the A/B protocol: [docs/decision-engine.md](docs/decision-engine.md).
A learned decision engine is being built. It is meant to replace hand-tuned detection edits. It
follows three rules:
How anything is measured on code the tool was not tuned on, the pair corpora, the independent
populations and the M4 gates, and the fold discipline that keeps learned decisions out of their
own training data: [docs/evaluation.md](docs/evaluation.md). Population v3 (PHP, 15 cases) is
frozen as the one-time final check and nothing in the tree may read it.
- Every instrument produces signals, never verdicts.
- Labels come from ground truth only.
- A compiled AI classifier with local memory turns retrieved similar cases and the candidate's
signals into a calibrated probability. It has BLOCK and ADVISORY operating points.
It is **in development, not adopted**. No scan, pre-push check or report uses it.
It was measured on six families, on repositories the compiled program was not shown:
- The pooled AUC (area under the ROC curve) of its score runs from 0.77 (injection) to 0.96
(access control).
- The stricter within-pair AUC is 0.72 to 0.93 across the same families.
- BLOCK is offered for no family
(`benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`).
Maintainer surface: `ousast learn labels|memory|compile|evaluate|curve|experiment|audit-leaks`.
Design, slices and the A/B protocol: [docs/decision-engine.md](docs/decision-engine.md).
[docs/evaluation.md](docs/evaluation.md) explains how anything is measured on code the tool was not
tuned on. It covers the pair corpora, the independent populations and the M4 gates. It also covers
the fold discipline that keeps learned decisions out of their own training data. Population v3
(PHP, 15 cases) is frozen as the one-time final check. Nothing in the tree may read it.
```bash
uv run ousast benchmark benchmarks/manifests/python-vulnerable.toml --mode quick
@@ -201,19 +280,26 @@ uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dr
## Token economics
Model tokens are the one running cost. At runtime the plane routes them: one Model and one budget
per Task, a metered client that refuses the next call at the ceiling, repository facts computed
model-free and reused, blobs replayed at $0, triage before the hunt and a tie-break only on
disputed candidates. In development, 85.5% of the maintainers' coding-session tokens were tool
input and output rather than reasoning (down from 90%), and the remedies in `benchmarks/dev/`
target that. Mechanisms,
the measured costs and the before-and-after table: [docs/token-ergonomics.md](docs/token-ergonomics.md).
Model tokens are the one running cost. At runtime the plane routes them:
- one Model and one budget per Task;
- a metered client that refuses the next call at the ceiling;
- repository facts computed without a model and reused;
- blobs replayed at $0, with no new model call;
- triage before the hunt;
- a tie-break only on disputed candidates.
In development, 85.5% of the maintainers' coding-session tokens were tool input and output rather
than reasoning. That is down from 90%. The remedies in `benchmarks/dev/` target that share.
Mechanisms, the measured costs and the before-and-after table:
[docs/token-ergonomics.md](docs/token-ergonomics.md).
## Examples, threat model
End-to-end walkthroughs, from a quick scan to an MCP client: [docs/examples.md](docs/examples.md).
What the tool trusts and what it does not (scanned code is untrusted input, secret redaction,
per-task egress, budgets, the `deep` sandbox): [docs/threat-model.md](docs/threat-model.md).
- End-to-end walkthroughs, from a quick scan to an MCP client: [docs/examples.md](docs/examples.md).
- What the tool trusts and what it does not: [docs/threat-model.md](docs/threat-model.md). This
covers scanned code as untrusted input, secret redaction, per-task egress, budgets and the
`deep` sandbox.
## Contributing and maintainer tooling
@@ -228,25 +314,33 @@ uv run python dagger/ci.py # containerized CI pipeline
```
An agent that can run shell commands needs nothing but the CLI. For [OpenCode](https://opencode.ai)
there are two project skills in `.agents/skills/`: `openultrasast-scan` (modes and their
prerequisites, `--fail-on`, the evidence ladder, recorded degradations, run artifacts) and
`openultrasast-triage` (false-positive reasons, what an adjudication may feed, fixing and
re-scanning). `ousast mcp` runs a narrow MCP server over stdio with ten tools (`openultrasast.scan`,
`status`, `findings`, `get_finding`, `evidence`, `artifacts`, `benchmark`, `explain`,
`propose_patch`, `export_report`); no tool runs a shell, Docker or a free-form command:
there are two project skills in `.agents/skills/`:
- `openultrasast-scan`: modes and their prerequisites, `--fail-on`, the evidence ladder, recorded
degradations and run artifacts.
- `openultrasast-triage`: false-positive reasons, what an adjudication may feed, and fixing and
re-scanning.
`ousast mcp` runs a narrow MCP (Model Context Protocol) server over stdio. It has ten tools:
`openultrasast.scan`, `status`, `findings`, `get_finding`, `evidence`, `artifacts`, `benchmark`,
`explain`, `propose_patch` and `export_report`. No tool runs a shell, Docker or a free-form
command.
```jsonc
{ "command": "uv", "args": ["run", "ousast", "mcp"] }
```
The `kiro-*` skills, `AGENTS.md` and the Kiro-style specifications under `.kiro/` are the
maintainer's development tooling and can be ignored by users. Other maintainer commands:
`ousast repos` (pinned known-vulnerable checkouts) and `ousast model candidates` (what the
candidate enumerator can reach).
maintainer's development tooling. Users can ignore them. Other maintainer commands:
- `ousast repos`: pinned known-vulnerable checkouts.
- `ousast model candidates`: what the candidate enumerator can reach.
## Licence
OpenUltraSAST is licensed under the [Apache License 2.0](LICENSE). Third-party material keeps its
own terms: Joern (Apache-2.0) is downloaded at image build time, the tree-sitter wheels of the
`semantic` extra are MIT, and the published benchmark cases were licence-checked one by one (v1's
OpenCVE case is BUSL-1.1, see [RELEASE_NOTES.md](RELEASE_NOTES.md)).
own terms:
- Joern (Apache-2.0) is downloaded at image build time.
- The tree-sitter wheels of the `semantic` extra are MIT.
- The published benchmark cases were licence-checked one by one. v1's OpenCVE case is BUSL-1.1, see [RELEASE_NOTES.md](RELEASE_NOTES.md).
docs/examples.md
+28 -17
View File
@@ -1,7 +1,7 @@
# End-to-end examples
Each example is self-contained. Artifacts land under
`<target>/.openultrasast/runs/<scan-id>/` and `.openultrasast/` is gitignored.
Each example stands on its own. Artifacts land under `<target>/.openultrasast/runs/<scan-id>/`.
`.openultrasast/` is gitignored.
## 1. Quick scan a local repo (deterministic, no keys)
@@ -21,7 +21,7 @@ ousast scan . --mode standard # Joern model layer, container network off:
ousast-with-judge scan . --mode standard # network on, so the model is asked (needs DEEPSEEK_API_KEY)
```
Without a key the graph still decides: its entailed findings are reported, and the LLM's residual
Without a key, the graph still decides. Its entailed findings are reported. The LLM's residual
and suspicion questions go unasked.
## 3. CI gate
@@ -52,10 +52,11 @@ uv run ousast pairs --split holdout --profile human # filter by declared split
uv run python -m openultrasast.pair_gate # CI: local pairs must all pass
```
Each case is two isolated trees (vuln file vs patched file, same relative path).
`pair_correct` means the harness fired on the vuln side and stayed silent on the
fix. Misses and fix-side leaks become `miss`/`fp` signals for `ousast improve`.
The catalog and public dataset pointers live in `benchmarks/pairs/`.
Each case is two isolated trees: the vuln file and the patched file, at the same relative path.
- `pair_correct` means the harness fired on the vuln side and stayed silent on the fix.
- Misses become `miss` signals and fix-side leaks become `fp` signals for `ousast improve`.
- The catalog and public dataset pointers live in `benchmarks/pairs/`.
## 6. Self-improve the ruleset from benchmark feedback
@@ -68,10 +69,11 @@ uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml
## 7. Running the agentic plane
Model work over whole repositories runs as a Run manifest on ax, one isolated actor
per task (bring-up: [ops/ax/README.md](ops/ax/README.md)). The provider key is
read from the environment (or `.env`, which never overrides an exported variable) by
`ousast` and sent only in each task's start request.
Model work over whole repositories runs as a Run manifest on ax. Each task gets one isolated
actor. Bring-up is in [ops/ax/README.md](ops/ax/README.md).
`ousast` reads the provider key from the environment, or from `.env`. `.env` never overrides an
exported variable. The key is sent only in each task's start request.
```bash
uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image
@@ -83,16 +85,25 @@ uv run ousast plane remember validation-46 # re-ingest the run's m
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run --memory
```
Each task in the Run carries its own Model and `budget: {usd, calls}`; a task that
reaches its ceiling stops as `unfinished` and resumes on a rerun with a larger
budget. `ousast plane status` prints the token attribution table: calls, prompt,
cache-hit and output tokens and USD per task.
Each task in the Run carries its own Model and `budget: {usd, calls}`.
- A task that reaches its ceiling stops as `unfinished`. It resumes on a rerun with a larger
budget.
- `ousast plane status` prints the token attribution table. It shows calls, prompt, cache-hit and
output tokens, and USD per task.
## 8. Drive it from an MCP client (OpenCode / IDE)
MCP (Model Context Protocol) lets an editor or agent call the tool's functions.
```jsonc
{ "command": "uv", "args": ["run", "ousast", "mcp"] }
```
Then: `openultrasast.scan {path}` → `run_dir` → `openultrasast.findings {run_dir}` →
`openultrasast.explain {run_dir, finding_id}`. See the README section "Agent integrations".
Then call the tools in this order:
1. `openultrasast.scan {path}` returns a `run_dir`.
2. `openultrasast.findings {run_dir}` lists the findings.
3. `openultrasast.explain {run_dir, finding_id}` explains one.
See the README section "Agent integrations".
docs/index.md
+24 -18
View File
@@ -1,13 +1,13 @@
# OpenUltraSAST
OpenUltraSAST is an experimental static security analyser meant to work as a safety net on
repositories it has never seen: it scans a checkout, or the commits a `git push` would publish,
and separates what it can prove from what it merely suspects. A scan that could not look at
something records a degradation instead of reporting a clean result.
OpenUltraSAST is an experimental static security analyser. It is meant to be a safety net on
repositories it has never seen. It scans a checkout, or the commits a `git push` would publish. It
keeps what it can prove apart from what it only suspects. When a scan could not look at
something, it records a degradation. It does not report a clean result.
**State as of 2026-10-02 (v2.0.0):** the pre-push safety net is **NO-GO** for rollout.
No hook capability is enabled, and no independent population has passed the qualification gates
(see [Evaluation](evaluation.md)).
No hook capability is enabled. No independent population (a case set frozen before any scan) has
passed the qualification gates (see [Evaluation](evaluation.md)).
## Three layers
@@ -17,6 +17,9 @@ No hook capability is enabled, and no independent population has passed the qual
| **The agentic plane on ax** | Model-driven work over whole repositories, run as a DAG of tasks on google/ax, one isolated actor per task. What the runs learn is kept in a memory store. Maintainer surface: `ousast plane`. | [The plane on ax](plane.md), [Memory](memory.md) |
| **The decision engine** | A learned classifier that weighs every instrument's signals against similar labelled cases from memory. **In development, not adopted**: no scan, pre-push check or report uses it. Maintainer surface: `ousast learn`. | [Memory and detection](memory-and-detection.md), [Decision engine](decision-engine.md) |
A code property graph is a graph of the program's syntax, control flow and data flow. Joern builds
it.
## How they connect
```mermaid
@@ -33,31 +36,34 @@ flowchart LR
engine -. "intended, not adopted: BLOCK / ADVISORY" .-> report
```
- The **scanners** produce findings and, inside plane Runs, the `alerts` that become memory rows.
- The **plane** runs model work (verify passes, roles) and model-free bookkeeping (facts, agreement,
feature records) per case; its `remember` task turns a case's artifacts into rows of the
[memory store](memory.md).
- Memory feeds two consumers. `ousast improve --memory` turns stored rows into rule-status
proposals that pass through the same validator and gate as every other edit. The decision engine
retrieves labelled examples from the same store.
- The **scanners** produce findings. Inside plane Runs, they also produce the `alerts` that become
memory rows.
- The **plane** runs model work per case: verify passes and roles. It also runs model-free
bookkeeping per case: facts, agreement and feature records. Its `remember` task turns a case's
artifacts into rows of the [memory store](memory.md).
- Memory feeds two consumers:
1. `ousast improve --memory` turns stored rows into rule-status proposals. They pass through the
same validator and gate as every other edit.
2. The decision engine retrieves labelled examples from the same store.
- The decision engine's output (dotted line) is the intended path. Nothing user-facing reads it
today.
## Where to go next
- The current state in one page, every figure with its record: capability against the M4 gates, the
step-by-step approach, ax on the server, token economics before and after, and the open items:
[Where we stand](where-we-stand.md).
- The current state in one page, every figure with its record: [Where we stand](where-we-stand.md).
It covers capability against the M4 gates, the step-by-step approach, ax on the server, token
economics before and after, and the open items.
- Run a scan: [Examples](examples.md), and the commands in [Scanning](scanning.md).
- Understand a finding's evidence and score: [Architecture](architecture.md).
- The program analysis under a scan, what each representation can see and what it misses:
[Detection techniques](detection-techniques.md).
- Operate the plane and its store: [ax on this host](ops/ax/README.md) and
[RustFS setup](rustfs.md).
- Where the plane runs today and what a separate Kubernetes cluster would take (planned, not
- Where the plane runs today, and what a separate Kubernetes cluster would take (planned, not
done): [Deployment](deployment.md).
- Where the model spend goes, per Task at runtime and in the maintainers' own coding sessions,
and what is reused instead of paid again: [Token ergonomics](token-ergonomics.md).
- Trust boundaries and hardening: [Threat model](threat-model.md).
Every number in these pages is quoted from a committed record whose path is given next to it.
Every number in these pages is quoted from a committed record. The record's path is given next to
the number.
docs/scanning.md
+59 -33
View File
@@ -1,13 +1,21 @@
# Scanning
`ousast scan PATH --mode quick|standard|deep` scans a checkout; `ousast pre-push` checks only
what a push (or an explicit `--base`/`--head` pair) changes. The stage-by-stage pipeline,
evidence ladder and score are in [Architecture](architecture.md); this page shows the flows.
This page shows the scan flows.
- `ousast scan PATH --mode quick|standard|deep` scans a checkout.
- `ousast pre-push` checks only what a push changes, or what an explicit `--base`/`--head` pair
changes.
The stage-by-stage pipeline, the evidence ladder and the score are in
[Architecture](architecture.md).
## The three scan modes
The stages a mode requests are `MODE_STAGES` in `src/openultrasast/stages.py`: `quick` runs
STATIC, `standard` adds MAP, `deep` adds REGRESS.
The stages each mode requests are listed in `MODE_STAGES` in `src/openultrasast/stages.py`:
- `quick` runs STATIC.
- `standard` adds MAP.
- `deep` adds REGRESS.
```mermaid
flowchart TD
@@ -38,7 +46,7 @@ Sources: `README.md` (scan modes), `src/openultrasast/cli.py` (`run_stage` calls
## Quick rules per language
Rules run only against files of their language. The bundled ruleset lives in
Each rule runs only against files of its language. The bundled ruleset lives in
`src/openultrasast/ruleset/<language>/rules.toml`.
| Language | Classes (CWE) |
@@ -49,21 +57,31 @@ Rules run only against files of their language. The bundled ruleset lives in
| Java + Groovy templates | command injection (78), SQLi (89), weak hash (327), deserialization (502), unescaped template XSS (79) |
| PHP | SQLi (89), command injection (78), file inclusion (98), path (22), open redirect (601), SSRF (918); code eval (95), unserialize (502) and echo XSS (79) ship as `shadow` rules |
The detection gate (`python -m openultrasast.gate`) enforces at least 90% recall and under 10%
false positives per language on the bundled corpora in `benchmarks/manifests/`. Those corpora are
cheat-sheet fixtures: they catch regressions, they do not estimate recall on real code. **PHP is
not in the gate**; its numbers are in-sample development numbers
(`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`). See `README.md` for the
gate's per-language output.
The detection gate (`python -m openultrasast.gate`) enforces two limits per language on the
bundled corpora in `benchmarks/manifests/`: at least 90% recall and under 10% false positives.
- Those corpora are cheat-sheet fixtures. They catch regressions. They do not estimate recall on
real code.
- **PHP is not in the gate**. Its numbers are in-sample development numbers
(`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`).
See `README.md` for the gate's per-language output.
## The Joern engine
The model layer (`src/openultrasast/model/`, `src/openultrasast/cpg/`) is the core of `standard`.
It builds one code property graph per scan with Joern (`joern-parse`, then batched
`joern --script` queries from `cpg/queries/*.sc`) and sends each question to the arbiter of its
type: **taint** (source to sink with no discharging sanitizer), **dominance** (does a guard govern
the obligated operation) and **configuration** (constant abstraction over a value). Taint facts
ship for C, Java, JavaScript, PHP and Python (`src/openultrasast/ruleset/semantic/*.toml`).
It builds one code property graph per scan with Joern. A code property graph is a graph of the
program's syntax, control flow and data flow. The build runs `joern-parse`, then batched
`joern --script` queries from `cpg/queries/*.sc`.
The layer sends each question to the arbiter (the decider) of its type:
- **taint**: a flow from source to sink with no discharging sanitizer;
- **dominance**: does a guard govern the obligated operation;
- **configuration**: constant abstraction over a value.
Taint facts ship for C, Java, JavaScript, PHP and Python
(`src/openultrasast/ruleset/semantic/*.toml`).
```mermaid
flowchart LR
@@ -79,25 +97,33 @@ flowchart LR
rung -- "residual question only" --> judge["LLM judge: confirm or contradict, never create a flow"]
```
The Docker image ships Joern and `php-cli` (which the PHP frontend needs); without Joern the
layer records `cpg_unavailable` and the scan is otherwise unchanged. Engine installation and the
Joern features used are in [Engine and pre-push](ops/README.md).
How the graph is built, queried and refined, with the records behind each refinement, is in
[Detection techniques](detection-techniques.md).
The Docker image ships Joern and `php-cli`. The PHP frontend needs `php-cli`. Without Joern, the
layer records `cpg_unavailable`; the rest of the scan is unchanged.
- Engine installation and the Joern features used: [Engine and pre-push](ops/README.md).
- How the graph is built, queried and refined, with the records behind each refinement:
[Detection techniques](detection-techniques.md).
## Framework priors
Rule and taint-fact entries that know a framework or library carry a `framework` or `library` tag
naming a row of `src/openultrasast/ruleset/frameworks.toml` (for example `wordpress`, `flask`,
`django`, `express`, `spring`). The loaders take `priors=`: `"all"` (the default and today's
behaviour), `"off"` (language-level knowledge only) or a set of ids, so the contribution of
framework knowledge can be measured and switched off. The decision engine's first slice ran with
priors off ([Decision engine](decision-engine.md#first-measured-slice-2026-10-01)).
Some rule and taint-fact entries know a framework or library. They carry a `framework` or
`library` tag. The tag names a row of `src/openultrasast/ruleset/frameworks.toml` (for example
`wordpress`, `flask`, `django`, `express`, `spring`).
The loaders take `priors=` with one of three values:
- `"all"`: the default and today's behaviour;
- `"off"`: language-level knowledge only;
- a set of ids.
So the contribution of framework knowledge can be measured and switched off. The decision
engine's first slice ran with priors off
([Decision engine](decision-engine.md#first-measured-slice-2026-10-01)).
## Pre-push: the delta check (experimental)
`ousast pre-push` analyses the commits a `git push` would publish, or an explicit
`--base`/`--head` pair, and compares head with base so that only new or worsened defects are
`--base`/`--head` pair. It compares head with base. Only new or worsened defects become
candidates. Sources: `ousast pre-push --help`, `src/openultrasast/push/`,
[Engine and pre-push](ops/README.md#explicit-local-replay).
@@ -118,10 +144,10 @@ flowchart TD
exit -- "blocking, explicit opt-in" --> block["May block the push"]
```
- `--deadline` (30 s by default) bounds preparation, both revisions and reporting together;
`--incomplete-coverage block` opts into failing when coverage is incomplete.
- The default capability registry is empty pending independent qualification, so no normal alert
is emitted today; every candidate stays diagnostic.
- `--deadline` (30 s by default) bounds preparation, both revisions and reporting together.
- `--incomplete-coverage block` opts into failing when coverage is incomplete.
- The default capability registry is empty until independent qualification passes. So no normal
alert is emitted today. Every candidate stays diagnostic.
- A model is called only with an explicit `--model-config`.
- Hook installation (`ops/install-pre-push`) and the private cache (`--cache-dir`) are documented
in [Engine and pre-push](ops/README.md#experimental-pre-push-integration).
docs/threat-model.md
+74 -51
View File
@@ -1,78 +1,101 @@
# Threat model & hardening
OpenUltraSAST analyzes **untrusted code**. This document states what it trusts, what
it does not, and the controls that keep a scan safe to run in CI and safe to share.
OpenUltraSAST analyzes **untrusted code**. This page states what it trusts and what it does not.
It also lists the controls that keep a scan safe to run in CI and safe to share.
## Trust boundaries
- **Scanned code is untrusted input.** Quick and standard modes never import, build or run
the target: quick reads source as text, and standard additionally parses it into a code
property graph with Joern (`joern-parse`, a parser, not a build) and may send code excerpts
to the configured model. Only `deep` mode executes target-derived code: its REGRESS stage
loads promoted candidates inside the Docker sandbox below, and it runs only when `docker
- **Scanned code is untrusted input.** Quick and standard modes never import, build or run the
target.
- Quick mode reads source as text.
- Standard mode also parses it into a code property graph (a graph of the program's syntax,
control flow and data flow) with Joern. It uses `joern-parse`, a parser, not a build.
Standard mode may send code excerpts to the configured model.
- Only `deep` mode executes target-derived code. Its REGRESS stage loads promoted candidates
inside the Docker sandbox described below. It runs only when `docker
info` succeeds (otherwise the stage is skipped and recorded as `sandbox_unavailable`).
- **The ruleset and CWE policy are trusted, governed data.** They change only through
reviewed commits or the bounded self-improvement loop (`ousast improve`), which can
flip rule *status* and tune score constants but can never edit pattern text or the
- **The ruleset and CWE policy are trusted, governed data.** They change in only two ways:
through reviewed commits, or through the bounded self-improvement loop (`ousast improve`). The
loop can flip rule *status* and tune score constants. It can never edit pattern text or the
authoritative 0–5 severity. See [architecture.md](architecture.md#self-improvement).
- **Provider credentials are the operator's.** The core install (PyYAML only) makes no
network calls on its own. A local scan calls a model only in standard or deep mode, and
only when a key is available: the model layer asks the residual question the graph cannot
settle, and `[models] hunter` enables the tool hunter. LLM calls go to DeepSeek
(`DEEPSEEK_API_KEY`); OpenRouter (`OPENROUTER_API_KEY`) serves embeddings. Keys come from
the environment or a gitignored `.env`, which never overrides an exported variable. The
pre-push check calls a model only with an explicit `--model-config`. Agentic work runs on the plane (`ousast plane run`): `ousast` reads
the variable a task's Model names (`secretKey.key`) from the operator's environment or
`.env` and sends the key only in that task's start request, through Agent Substrate's
router to that task's actor. It is not baked into an image or task template, and it is
never logged or written to artifacts.
- **Provider credentials are the operator's.** The core install (PyYAML only) makes no network
calls on its own.
- A local scan calls a model only in standard or deep mode, and only when a key is available.
The model layer asks the residual question the graph cannot settle. `[models] hunter`
enables the tool hunter.
- LLM calls go to DeepSeek (`DEEPSEEK_API_KEY`). OpenRouter (`OPENROUTER_API_KEY`) serves
embeddings.
- Keys come from the environment or a gitignored `.env`. `.env` never overrides an exported
variable.
- The pre-push check calls a model only with an explicit `--model-config`.
- Agentic work runs on the plane (`ousast plane run`). `ousast` reads the variable a task's
Model names (`secretKey.key`) from the operator's environment or `.env`. It sends the key
only in that task's start request. The request goes through Agent Substrate's router to that
task's actor. The key is not baked into an image or task template. It is never logged or
written to artifacts.
## Controls
### Secret redaction
Scan artifacts can quote source that contains live credentials. `redaction.py` masks
recognizable secret shapes (provider API keys, AWS/GitHub/Slack/Google tokens, bearer
tokens, URL-embedded credentials, PEM private keys, and `key = value` assignments for
sensitive names) before traces (`trace/events.jsonl`) and the markdown report are
written. On by default; disable with `[hardening] redact_secrets = false`.
recognizable secret shapes before traces (`trace/events.jsonl`) and the markdown report are
written. It masks:
- provider API keys;
- AWS/GitHub/Slack/Google tokens;
- bearer tokens;
- URL-embedded credentials;
- PEM private keys;
- `key = value` assignments for sensitive names.
Redaction is on by default. Turn it off with `[hardening] redact_secrets = false`.
### Plane egress
Each plane task runs in its own actor behind Agent Substrate's egress gateway, which
denies by default. The task's EgressPolicy (`src/openultrasast/plane/egress.py`) allows exactly three
things: plain HTTP to the artifact receiver, and TLS passthrough to the Git hosts of the
task's bound Workspaces and to the hosts its bound Model declares
(`openultrasast.io/egress-hosts`). There are no wildcards, no IP addresses and no TLS
interception. The policy is deleted with the actor.
Each plane task runs in its own actor behind Agent Substrate's egress gateway. The gateway denies
by default. The task's EgressPolicy (`src/openultrasast/plane/egress.py`) allows exactly three
things:
1. plain HTTP to the artifact receiver;
2. TLS passthrough to the Git hosts of the task's bound Workspaces;
3. TLS passthrough to the hosts its bound Model declares (`openultrasast.io/egress-hosts`).
There are no wildcards, no IP addresses and no TLS interception. The policy is deleted with the
actor.
### Cost & CI budgets
- **Agentic spend** is bounded per plane task by the Run's `budget: {usd, calls}`. The
metered client refuses the next call once either ceiling is reached (the task is
`unfinished` and resumes on a rerun with a larger budget). A usd budget on a Model
without prices is refused rather than metered as zero, and an account or
authentication error fails the task and starts nothing further. The standard-mode tool
hunter is bounded by the per-tier hunter budgets.
- **Agentic spend** is bounded per plane task by the Run's `budget: {usd, calls}`.
- The metered client refuses the next call once either ceiling is reached. The task is then
`unfinished`. It resumes on a rerun with a larger budget.
- A usd budget on a Model without prices is refused. It is not metered as zero.
- An account or authentication error fails the task and starts nothing further.
- The standard-mode tool hunter is bounded by the per-tier hunter budgets.
- **Output size** is bounded by `[hardening] max_findings` (0 = unlimited). Truncation is
severity-ordered and disclosed as a `budget` degradation in the manifest.
severity-ordered. It is disclosed as a `budget` degradation in the manifest.
### Provider reliability
LLM/embedding calls retry transient failures (HTTP 429/5xx, connection errors, timeouts)
with exponential backoff; non-transient errors (e.g. 4xx, malformed JSON) fail fast.
LLM/embedding calls retry transient failures with exponential backoff. Transient failures are
HTTP 429/5xx, connection errors and timeouts. Non-transient errors fail fast, for example 4xx or
malformed JSON.
### Visible degradation & determinism
When an optional engine is missing (Joern: `cpg_unavailable`; Docker: `sandbox_unavailable`),
the scan falls back to its deterministic equivalent and records a `degradations` entry in the
manifest — it is never silently downgraded. Without a provider key the model layer asks no
question and reports only what the graph entailed. Runs are reproducible: fixed config,
artifact manifests, prompt hashes, and model identifiers are recorded.
A scan is never silently downgraded. When an optional engine is missing, the scan falls back to
its deterministic equivalent. It records a `degradations` entry in the manifest (Joern:
`cpg_unavailable`; Docker: `sandbox_unavailable`).
Without a provider key, the model layer asks no question. It reports only what the graph
entailed.
Runs are reproducible. The fixed config, artifact manifests, prompt hashes and model identifiers
are recorded.
### MCP surface
`ousast mcp` exposes only the ten narrow project tools. No tool runs arbitrary shell,
Docker, or internal hunter tools, and no tool accepts a free-form command argument.
`ousast mcp` exposes only the ten narrow project tools. No tool runs arbitrary shell, Docker or
internal hunter tools. No tool accepts a free-form command argument.
## Sandbox (deep mode)
The REGRESS stage (`regress/`, `sandbox/runner.py`) runs each snippet with `docker run`
and never mounts the host Docker socket:
The REGRESS stage (`regress/`, `sandbox/runner.py`) runs each snippet with `docker run`. It never
mounts the host Docker socket.
| Control | Setting |
|---|---|
@@ -84,5 +107,5 @@ and never mounts the host Docker socket:
| pids | `[sandbox] pids_limit`, default 512 |
| timeout | `[sandbox] timeout_seconds`, default 300 s |
Before anything runs, a structural deny-list (`regress/safety.py`) rejects a snippet that
mentions sockets, curl, host networking or writes under `/workspace`.
A structural deny-list (`regress/safety.py`) checks each snippet before anything runs. It rejects
a snippet that mentions sockets, curl, host networking or writes under `/workspace`.
docs/where-we-stand.md
+259 -174
View File
@@ -1,33 +1,50 @@
# Where we stand
State as of 2026-10-02 (v2.0.0). This page answers five questions in one place. How far is
detection from its gates. How does the project now try to close that gap. How does the model work
run on ax. What did the tokens cost before and after that move. What is still open. Every figure
is quoted from a committed record whose path stands next to it; detail tables stay on the pages
linked from each section.
State as of 2026-10-02 (v2.0.0). This page answers five questions in one place:
The goal has not changed: a safety net on repositories the tool has never seen, which says
BLOCK or ADVISORY about the code a push would change and stays silent on fixes and on benign
commits. Two independent populations have been spent against that goal, with engine recall 1 of
1. How far is detection from its gates?
2. How does the project now try to close that gap?
3. How does the model work run on ax?
4. What did the tokens cost before and after that move?
5. What is still open?
Every figure is quoted from a committed record. The record's path stands next to the figure.
Detail tables stay on the pages linked from each section.
The goal has not changed. The tool should be a safety net on repositories it has never seen. It
should say BLOCK or ADVISORY about the code a push would change. It should stay silent on fixes
and on benign commits.
Two independent populations have been spent against that goal. An independent population is a
set of cases frozen before any scan. Engine recall was 1 of
11 and then 0 of 17. The hand edits that closed misses on the first population did not carry to
the second. So the question is no longer which rule to add next. It is how a change to detection
can be shown to work on code it was not tuned on, at a cost per candidate that can be paid for
every push.
the second. So the question is no longer which rule to add next. The question is this: how can a
change to detection be shown to work on code it was not tuned on? And can that be done at a cost
per candidate that can be paid for every push?
The answer this page records: the pre-push safety net is NO-GO against every M4 gate, and the
project has stopped trying to reach the gates by hand. Every detection change now has to win a
controlled measurement on held-out repositories, run on an agentic plane whose cost per candidate
is known. The pieces of that method exist and have been measured. None of them has produced a
BLOCK decision yet.
The answer this page records:
- The pre-push safety net is NO-GO against every M4 gate.
- The project has stopped trying to reach the gates by hand.
- Every detection change now has to win a controlled measurement on held-out repositories. The
measurement runs on an agentic plane whose cost per candidate is known.
- The pieces of that method exist and have been measured. None of them has produced a BLOCK
decision yet.
## 1. Detection capability and goals: NO-GO, measured
No M4 gate is met on an independent population. The qualification milestone for the pre-push
safety net (M4) requires, together: at least 95% actionable precision, at least 90% supported
recall, at least 95% supported-check completion, the runtime gates, and zero fixed-side or benign
false alerts. A population is frozen before any
case is scanned, scored once under a pre-registered protocol, then spent; it can inform changes
but can qualify nothing afterwards ([Evaluation](evaluation.md#independent-populations-and-the-m4-gates)).
No M4 gate is met on an independent population. M4 is the qualification milestone for the
pre-push safety net. It requires all of these together:
- at least 95% actionable precision;
- at least 90% supported recall;
- at least 95% supported-check completion;
- the runtime gates;
- zero fixed-side or benign false alerts.
A population is frozen before any case is scanned. It is scored once under a pre-registered
protocol, then spent. After that it can inform changes but can qualify nothing
([Evaluation](evaluation.md#independent-populations-and-the-m4-gates)).
| Gate | Required | Latest independent result | Record |
| --- | --- | --- | --- |
@@ -36,156 +53,217 @@ but can qualify nothing afterwards ([Evaluation](evaluation.md#independent-popul
| Supported-check completion | >= 95% | 13,206 of 108,374 | `results-v2.json` |
| Fixed-side and benign alerts | zero | 0 fixed-side, which the record calls met trivially since nothing was detected; 3 benign alerts | `results-v2.json` |
Three later measurements narrow the picture without changing the verdict.
Three later measurements narrow the picture. They do not change the verdict.
- The plane reproduces the script pipeline's agreement at a known cost. On the model-driven
pipeline's validation set (46 candidates, 15 cases of v2, 20 declared vulnerable sites), the
plane's verify passes a and b plus a tie-break on the 10 disputed candidates agreed on 16 of 20
declared sites. The script reference agreed on 16 as well. The cost was $0.0205 per candidate
(`benchmarks/independent/plane-increment-2.json`). The record carries two caveats. The tie-break
was chosen after seeing the first measurement on the same set, so this compares plane with
script and qualifies nothing. And the passes flag fewer declared sites than the reference in
any pass, 17 against 19; that count is a correction in `plane-increment-1.json`, whose first
reading took agree's `declared_sites_matched` (19, coverage by candidates) for "found in either
pass".
- The learned decision engine ranks, but may not block. On six families, evaluated on
repositories the compiled program was not shown, the pooled AUC of its score runs from 0.77
(injection) to 0.96 (access control). The stricter reading is the within-pair AUC, which asks
whether the vulnerable side of a fix scores above its own fixed side: 0.72 to 0.93. BLOCK is
offered for no family; calibration holds only for output encoding, and there no fold reaches
the required precision (`benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`).
The per-family table is on [Decision engine](decision-engine.md#second-slice-six-families-2026-10-02).
- PHP coverage is in-sample. The PHP quick rules were written and tightened while looking at
the pins and fixtures they are scored on, so the record says so itself (`provenance.in_sample` in
`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`): a development check, not
a recall estimate. On real PHP code the Joern engine is the instrument.
- **The plane reproduces the script pipeline's agreement at a known cost.** The test used the
model-driven pipeline's validation set: 46 candidates, 15 cases of v2, 20 declared vulnerable
sites. The plane ran verify passes a and b, plus a tie-break on the 10 disputed candidates. It
agreed on 16 of 20 declared sites. The script reference agreed on 16 as well. The cost was
$0.0205 per candidate (`benchmarks/independent/plane-increment-2.json`). The record carries two
caveats:
1. The tie-break was chosen after seeing the first measurement on the same set. So this compares
plane with script and qualifies nothing.
2. The passes flag fewer declared sites than the reference in any pass: 17 against 19 (passes against reference). That
count is a correction in `plane-increment-1.json`. Its first reading took agree's
`declared_sites_matched` (19, coverage by candidates) to mean "found in either pass".
- **The learned decision engine ranks, but may not block.** It was evaluated on six families, on
repositories the compiled program was not shown.
- The pooled AUC (area under the ROC curve) of its score runs from 0.77 (injection) to 0.96
(access control).
- The stricter reading is the within-pair AUC. It asks whether the vulnerable side of a fix
scores above its own fixed side: 0.72 to 0.93.
- BLOCK is offered for no family. Calibration holds only for output encoding. There, no fold
reaches the required precision
(`benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`).
- The per-family table is on
[Decision engine](decision-engine.md#second-slice-six-families-2026-10-02).
- **PHP coverage is in-sample.** The PHP quick rules were written and tightened while looking at
the pins and fixtures they are scored on. The record says so itself (`provenance.in_sample` in
`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`). It is a development
check, not a recall estimate. On real PHP code, the Joern engine is the instrument.
What has not been touched: population v3, 15 PHP cases, frozen as the one-time final check. No
development or analysis work reads its cases, a test fails when anything in the tree names one
of its repositories, and it runs once, after a prediction is committed.
What has not been touched: population v3, 15 PHP cases, frozen as the one-time final check.
- No development or analysis work reads its cases.
- A test fails when anything in the tree names one of its repositories.
- It runs once, after a prediction is committed.
## 2. Our approach to detection, step by step
The path changed because hand edits did not transfer. Every gain before 2026-09-30 was a hand
edit after a miss: a rule, a taint vocabulary entry, a guard, a prompt line, a tie-break. Each was
tuned on the cases that exposed it, and the tuning measured on population v1 did not carry to v2
(1 of 11, then 0 of 17). The conclusions drawn: hand edits do not transfer. The data that could
teach a better decision cannot be produced by hand at the scale needed, so it has to be harvested
and labelled by machinery with a boundary around it. Framework knowledge (which function is a sink in which
framework) is useful as a prior but must stay optional, or the engine learns the vocabulary
instead of the code. A reader might object that a classifier is slower to show gains than the
next rule. It is. The difference is that its gains are measured where rules were not: on
repositories it never saw.
tuned on the cases that exposed it. The tuning measured on population v1 did not carry to v2
(1 of 11, then 0 of 17).
The chain, one job per step. The design is on [Memory and detection](memory-and-detection.md);
this is the order of operations.
The conclusions drawn:
1. Instruments produce signals, never verdicts. Quick rules, the Joern engine, model sink
classification, the plane's verify passes a and b with the tie-break c, repository facts
(callers, spans) and model source/sink/sanitizer roles each write one block of a candidate's
record. None decides alone.
2. Feature records are allow-listed, and missing is not zero. A candidate is `(path, function,
family)` of one repository and pin. An instrument that did not run, had no coverage or failed
records that state; "silent" and "did not look" stay distinguishable. Input profile v1 withholds
the engine block and the function length, which a leak audit found separate the two sides of a
pair by construction (`benchmarks/measurements/2026-10-01-decision-engine-leak-audit/`), and
since 2026-10-02 the plane's `verify.flag_b` and `ms.operation`, flagged the same way by the
audit of the plane profile (`2026-10-02-decision-engine-harvest-advisory/record.json`,
- Hand edits do not transfer.
- The data that could teach a better decision cannot be produced by hand at the scale needed. It
has to be harvested and labelled by machinery with a boundary around it.
- Framework knowledge (which function is a sink in which framework) is useful as a prior. But it
must stay optional. Otherwise the engine learns the vocabulary instead of the code.
A reader might object that a classifier is slower to show gains than the next rule. It is. The
difference is where its gains are measured: on repositories it never saw. Rules were not measured
there.
The chain has one job per step. The design is on [Memory and detection](memory-and-detection.md).
This list is the order of operations.
1. **Instruments produce signals, never verdicts.** Each instrument writes one block of a
candidate's record. None decides alone. The instruments are:
- quick rules;
- the Joern engine;
- model sink classification;
- the plane's verify passes a and b, with the tie-break c;
- repository facts (callers, spans);
- model source/sink/sanitizer roles.
2. **Feature records are allow-listed, and missing is not zero.** A candidate is `(path, function,
family)` of one repository and pin.
- An instrument that did not run, had no coverage or failed records that state. "Silent" and
"did not look" stay distinguishable.
- Input profile v1 withholds the engine block and the function length. A leak audit found that
they separate the two sides of a pair by construction
(`benchmarks/measurements/2026-10-01-decision-engine-leak-audit/`).
- Since 2026-10-02 it also withholds the plane's `verify.flag_b` and `ms.operation`. The audit
of the plane profile flagged them the same way
(`2026-10-02-decision-engine-harvest-advisory/record.json`,
`src/openultrasast/learn/schema.py`).
3. Labels come from ground truth only. A fail-closed allow-list (`src/openultrasast/learn/sources.toml`)
names the sources: the spent populations' declared sites and fix ranges, the vulnerable and
fixed sides of pairs, including the 97 real public fixes in `benchmarks/pairs/advisory-fixes/`,
benign controls and recorded adjudications. Assumed-benign commits are their own source,
weighted 0.5 and reported separately. A plane verdict is a feature, never a label: "rejected by
the plane" is not a negative, or the engine would learn to reproduce the plane's mistakes.
4. Retrieval finds similar labelled cases. A Gower distance over the signal profile, re-ranked
by cosine similarity of code embeddings (OpenAI `text-embedding-3-small` through OpenRouter,
cached by excerpt hash). The evaluation boundary is the only door into a prompt: never the
candidate's own repository group, never a group the fold evaluates.
5. A classifier compiled in house, DSPy-style. `retrieve -> classify(k)`: an instruction and
compiled demonstrations (a fixed prefix, so the provider's cache hits), the retrieved examples,
then the candidate's excerpt and signals. The model answers vulnerable, not vulnerable or
unsure with a rationale that cites a line; `k` samples give a score. One program is compiled
per profile and family on a compile split that is never evaluated.
6. Calibration on held-out repositories. Platt scaling of the score, cross-fitted by
repository group so no map is checked on the fold that fitted it. Calibration holds when the
slope interval contains 1, the intercept interval contains 0 and ECE <= 0.05.
7. Two operating points. BLOCK only where calibration holds and a fold reaches the required
precision; otherwise ADVISORY. An unsure majority can never BLOCK.
8. A/B experiments are the only way a change is adopted. Prompts, models, pass counts,
features and sampling are compared under a pre-registered manifest with frozen units, paired
arms, a repository bootstrap and two looks. exp-002 is the first to finish: a retrieval
ensemble (five samples over disjoint example sets at temperature 0) against today's
temperature sampling. It raised re-run agreement a little (0.906 against 0.875, interval
spanning zero) and lowered the within-pair AUC from 0.815 to 0.741 at 2.4 times the cost per
candidate ($0.0051 against $0.0021). Decision: REJECTED
(`benchmarks/experiments/exp-002-retrieval-ensemble/result.json`).
9. Extrapolation with intervals. Every number carries a 95% repository-cluster bootstrap
interval; the pooled and the within-pair readings are both reported, and the stricter one is
3. **Labels come from ground truth only.** A fail-closed allow-list
(`src/openultrasast/learn/sources.toml`) names the sources:
- the spent populations' declared sites and fix ranges;
- the vulnerable and fixed sides of pairs, including the 97 real public fixes in
`benchmarks/pairs/advisory-fixes/`;
- benign controls;
- recorded adjudications.
Assumed-benign commits are their own source. They are weighted 0.5 and reported separately. A
plane verdict is a feature, never a label. "Rejected by the plane" is not a negative.
Otherwise the engine would learn to reproduce the plane's mistakes.
4. **Retrieval finds similar labelled cases.** It uses a Gower distance (a distance for mixed
numeric and categorical data) over the signal profile. Results are re-ranked by cosine
similarity of code embeddings. The embeddings are OpenAI `text-embedding-3-small` through
OpenRouter, cached by excerpt hash. The evaluation boundary is the only door into a prompt. It
never admits the candidate's own repository group, or a group the fold evaluates.
5. **A classifier compiled in house, DSPy-style.** The program is `retrieve -> classify(k)`. The
prompt holds, in order:
- an instruction and compiled demonstrations (a fixed prefix, so the provider's cache hits);
- the retrieved examples;
- the candidate's excerpt and signals.
The model answers vulnerable, not vulnerable or unsure, with a rationale that cites a line. `k`
samples give a score. One program is compiled per profile and family, on a compile split that
is never evaluated.
6. **Calibration on held-out repositories.** Platt scaling (a logistic fit) maps the score to a
probability. It is cross-fitted by repository group, so no map is checked on the fold that
fitted it. Calibration holds when three conditions are met: the slope interval contains 1, the
intercept interval contains 0 and ECE <= 0.05.
7. **Two operating points.** BLOCK applies only where calibration holds and a fold reaches the
required precision. Otherwise the point is ADVISORY. An unsure majority can never BLOCK.
8. **A/B experiments are the only way a change is adopted.** Prompts, models, pass counts,
features and sampling are compared under a pre-registered manifest. The manifest fixes frozen
units, paired arms, a repository bootstrap and two looks. exp-002 is the first to finish:
- It compared a retrieval ensemble (five samples over disjoint example sets at temperature 0)
with today's temperature sampling.
- It raised re-run agreement a little (0.906 against 0.875, interval spanning zero).
- It lowered the within-pair AUC from 0.815 to 0.741 and cost 2.4 times as much per candidate
($0.0051 against $0.0021).
- Decision: REJECTED (`benchmarks/experiments/exp-002-retrieval-ensemble/result.json`).
9. **Extrapolation with intervals.** Every number carries a 95% repository-cluster bootstrap
interval. Both the pooled and the within-pair readings are reported. The stricter one is
named.
10. The one-time v3 check. When a program is adopted and a prediction is committed, population
v3 is scored once under `benchmarks/independent/protocol-v3.md`.
10. **The one-time v3 check.** It happens when a program is adopted and a prediction is
committed. Then population v3 is scored once under `benchmarks/independent/protocol-v3.md`.
## 3. AI integration: ax on the server
All model work runs as Tasks on google/ax over Agent Substrate, in a single-node kind cluster on
the maintainer's host, each Task in its own gVisor sandbox. There is no local subprocess path.
The reconciler (`ousast plane run`) is a host process: it applies a Run's Workspaces and Tasks,
resumes each Task, sends its start signal, collects its artifacts and records state. Flow and
contract: [The plane on ax](plane.md); bring-up and the host's lessons: [ax on this host](ops/ax/README.md).
All model work runs as Tasks on google/ax over Agent Substrate. They run in a single-node kind
cluster on the maintainer's host. Each Task runs in its own gVisor sandbox. There is no local
subprocess path.
The reconciler (`ousast plane run`) is a host process. It does five things:
1. applies a Run's Workspaces and Tasks;
2. resumes each Task;
3. sends its start signal;
4. collects its artifacts;
5. records state.
Flow and contract: [The plane on ax](plane.md). Bring-up and the host's lessons:
[ax on this host](ops/ax/README.md).
What each Run guarantees:
- One Model and one budget per Task. A Task binds at most one ax Model (`deepseek-flash` for
chat, `openrouter-embedding` for embeddings) and `budget: {usd, calls}`. The metered client
refuses the next call at the ceiling; the task ends unfinished and resumes on a rerun. Model-free
tasks run with a zero budget, so a stray call fails loudly.
- Credentials exist only in the start request. Agent Substrate boots each template once as a
golden actor, so the runner never starts by itself. It waits for `POST /ousast/v1/start`, which
the reconciler sends through `atenet-router` to that actor alone. The key is read from the
operator's environment or `.env`, kept in memory by the runner, passed to the command and
redacted from echoed stderr; no manifest, rendered Task or file carries it.
- Per-task egress. One EgressPolicy per actor through the `atenet-egress` gateway, deny by
default, hostnames only: plain HTTP to the artifact receiver, TLS passthrough to the Workspaces'
Git hosts and the Model's declared hosts. Deleted with the actor.
- Artifacts come back through the gateway. The runner posts its output directory as a tar to
the receiver, an HTTP server inside the reconciler process on the host, reached through a Service
with an EndpointSlice to the host. That delivery is the task's completion; ax has no Completed
phase.
- Memory on S3-compatible RustFS. `OUSAST_MEMORY=s3://sast-memory` on a RustFS server. The
store verifies the bucket at every start (versioning, lifecycle, tags, S3 Select) and refuses to
start when anything is missing; there is no local or fetch-and-filter fallback. Each row kind
has a fixed set of queryable fields, enforced at write time
- **One Model and one budget per Task.** A Task binds at most one ax Model and
`budget: {usd, calls}`. The Models are `deepseek-flash` for chat and `openrouter-embedding` for
embeddings. The metered client refuses the next call at the ceiling. The task ends unfinished
and resumes on a rerun. Model-free tasks run with a zero budget, so a stray call fails loudly.
- **Credentials exist only in the start request.** Agent Substrate boots each template once as a
golden actor. So the runner never starts by itself. It waits for `POST /ousast/v1/start`. The
reconciler sends that request through `atenet-router` to that actor alone.
- The key is read from the operator's environment or `.env`.
- The runner keeps it in memory and passes it to the command.
- It is redacted from echoed stderr.
- No manifest, rendered Task or file carries it.
- **Per-task egress.** Each actor gets one EgressPolicy through the `atenet-egress` gateway. It is
deny by default and uses hostnames only. It allows plain HTTP to the artifact receiver. It allows
TLS passthrough to the Workspaces' Git hosts and the Model's declared hosts. It is deleted with
the actor.
- **Artifacts come back through the gateway.** The runner posts its output directory as a tar to
the receiver. The receiver is an HTTP server inside the reconciler process on the host. It is
reached through a Service with an EndpointSlice to the host. That delivery is the task's
completion; ax has no Completed phase.
- **Memory on S3-compatible RustFS.** The setting is `OUSAST_MEMORY=s3://sast-memory` on a RustFS
server.
- The store verifies the bucket at every start: versioning, lifecycle, tags, S3 Select.
- It refuses to start when anything is missing. There is no local or fetch-and-filter fallback.
- Each row kind has a fixed set of queryable fields, enforced at write time
([Memory](memory.md#verify-at-startup-no-fallback), [RustFS setup](rustfs.md)).
Lessons the host taught, each from a failed live run (the full list is on
[ax on this host](ops/ax/README.md#what-this-host-taught-each-one-cost-a-failed-live-run)):
- Substrate installs no WorkerPool, and actor images must be pinned by digest.
- A release `ax` CLI skews from the server; build it from the deployed checkout.
- A release `ax` CLI skews from the server. Build it from the deployed checkout.
- ax has no Completed phase, so completion is our artifact delivery.
- The golden boot means the runner must wait for a start request.
- The egress gateway dials the address the actor connected to, so the receiver is a Service
- The egress gateway dials the address the actor connected to. So the receiver is a Service,
dialled by ClusterIP with its name as Host.
- ax's snapshot bucket must exist in your S3 server. When it did not, deleted golden actors piled
up in `DELETING` and filled the disk during a 600-task harvest on 2026-10-01.
up in `DELETING`. They filled the disk during a 600-task harvest on 2026-10-01.
Deployment status, in the two words [Deployment](deployment.md) uses strictly. Implemented:
ax as the only executor, per-task EgressPolicy, credentials only in the start request, per-task
budgets and attribution, the S3 store verified at startup, local bring-up, doctor and smoke Run,
re-pinning generated Tasks to another registry. Planned, not done: deployment to any cluster
other than this host's kind cluster, a configurable kube context (`kind-` is hard-coded in
`doctor.py`, `egress.py`, `router.py`), the receiver as an in-cluster Deployment, the
reconciler in-cluster with Secrets injected, presigned-URL delivery to the store.
Deployment status uses two words, as [Deployment](deployment.md) does, strictly.
Implemented:
- ax as the only executor;
- per-task EgressPolicy;
- credentials only in the start request;
- per-task budgets and attribution;
- the S3 store verified at startup;
- local bring-up, doctor and smoke Run;
- re-pinning generated Tasks to another registry.
Planned, not done:
- deployment to any cluster other than this host's kind cluster;
- a configurable kube context (`kind-` is hard-coded in `doctor.py`, `egress.py`, `router.py`);
- the receiver as an in-cluster Deployment;
- the reconciler in-cluster with Secrets injected;
- presigned-URL delivery to the store.
## 4. Token economics, before and after
Before the plane, the script pipeline cost $0.022 per candidate for two verify passes
(`benchmarks/independent/verifier-batched-check-2026-09-29.json`), and the maintainers' own
coding sessions spent about 90% of their tokens on tool input and output rather than reasoning.
After it, the per-candidate costs below are what the records show. The mechanisms (one budget
per Task, pay-once reuse, cheap-before-expensive ordering, replay at $0) are on
[Token ergonomics](token-ergonomics.md); this table is the before and after.
Before the plane, two costs stood out:
- The script pipeline cost $0.022 per candidate for two verify passes
(`benchmarks/independent/verifier-batched-check-2026-09-29.json`).
- The maintainers' own coding sessions spent about 90% of their tokens on tool input and output
rather than reasoning.
After it, the per-candidate costs below are what the records show. The mechanisms are on
[Token ergonomics](token-ergonomics.md): one budget per Task, pay-once reuse,
cheap-before-expensive ordering and replay at $0. This table is the before and after.
| Item | Before | After | Record |
| --- | --- | --- | --- |
@@ -195,63 +273,70 @@ per Task, pay-once reuse, cheap-before-expensive ordering, replay at $0) are on
| Harvest of plane signals for the engine | none existed | `verify` $2.5373 over 1,961 calls and 474 tasks; `roles` $1.4036 over 1,034 calls and 151 tasks | `2026-10-01-decision-engine-harvest/record.json` |
| Development transcript, tool I/O share | 90% | 85.5% over 2,701 tool calls (`dev_token_report`) | `plane-increment-1.json` |
Two notes from the records. The meter prices every call at the Model's list price
(`plane/models/deepseek-flash.yaml`); the DeepSeek account moved less, about one third of the
metered amount on the injection slice ($0.30 for $0.91 metered), and budgets and gates are set
against the meter, never against the balance. And the development share barely moved: 90 to 85.5
is a small gain, and the remedies (a read guard, a bulk reader, scripts in files) target file
contents because that is where the tokens go.
Two notes from the records:
1. The meter prices every call at the Model's list price (`plane/models/deepseek-flash.yaml`). The
DeepSeek account moved less: about one third of the metered amount on the injection slice
($0.30 for $0.91 metered). Budgets and gates are set against the meter, never against the
balance.
2. The development share barely moved: 90 to 85.5 is a small gain. The remedies are a read guard,
a bulk reader and scripts in files. They target file contents, because that is where the
tokens go.
## 5. Open items
Five groups, each item with its record: the engine's own gates, data, measurement debt,
infrastructure, and instrument gaps.
There are five groups. Each item has its record. The groups are the engine's own gates, data,
measurement debt, infrastructure, and instrument gaps.
The engine's gates.
- BLOCK is reachable for no family: calibration fails for five (ECE 0.071 to 0.119) and the
sixth reaches no fold with a Wilson lower bound of 0.95 on precision (slice-2 record).
- BLOCK is reachable for no family. Calibration fails for five (ECE 0.071 to 0.119). The sixth
reaches no fold with a Wilson lower bound of 0.95 on precision (slice-2 record).
- Re-run agreement is below 0.9 for three families: injection 0.77, access control 0.83,
untrusted destination 0.87. exp-002 tried the obvious variance fix and lost on the ranking.
Data.
- Positives-only groups: 98 (family, group) pairs in the 904-example memory hold positive examples
and no negative. 94 of them exist because the fixed side no longer has the function (the vibe-py
pairs). No advisory-fixes group is among them
(`benchmarks/measurements/2026-10-02-decision-engine-negatives-gap/record.json`). There are no
single-label negatives, so every family's pooled AUC is inflated by positives that never had to
be told apart from their own fix (slice-2 record, `paired_groups_check`).
- The model has seen public fixes. A verify pass named the advisory's CVE id for a public fix, an
observation the engine's design specification (maintainer tooling) records as visible
memorisation. Excerpts now redact advisory ids and security-worded comments
(`src/openultrasast/learn/excerpt.py`), but labels drawn from public advisories are partly in the
model's training data, and the within-pair numbers carry that.
- PHP is covered by the engine on real code and by in-sample quick rules; the population that
and no negative.
- 94 of them exist because the fixed side no longer has the function (the vibe-py pairs).
- No advisory-fixes group is among them
(`benchmarks/measurements/2026-10-02-decision-engine-negatives-gap/record.json`).
- There are no single-label negatives. So every family's pooled AUC is inflated by positives
that never had to be told apart from their own fix (slice-2 record, `paired_groups_check`).
- The model has seen public fixes. A verify pass named the advisory's CVE id for a public fix.
The engine's design specification (maintainer tooling) records this as visible memorisation.
Excerpts now redact advisory ids and security-worded comments
(`src/openultrasast/learn/excerpt.py`). But labels drawn from public advisories are partly in
the model's training data, and the within-pair numbers carry that.
- PHP is covered by the engine on real code and by in-sample quick rules. The population that
would measure it (v3) is reserved.
Measurement debt.
- exp-001 (known callers) is registered and has not run; the model-free candidate generation on
the 41 development repositories that its units need is not built
- exp-001 (known callers) is registered and has not run. Its units need model-free candidate
generation on the 41 development repositories. That generation is not built
(`benchmarks/experiments/exp-001-known-callers/registration.json`).
- Leave-one-source-out, leave-one-framework-out and the memory-size curve points (0, 25, 50%)
have not run; one paired point exists (724 against 904 examples, injection, AUC 0.76 against 0.77).
- The plane-profile leak audit flagged `verify.flag_b` and `ms.operation`; they are withheld from
profile v1, and whether they track the label on unseen code has to be decided before any
plane-profile program runs (`2026-10-02-decision-engine-harvest-advisory/record.json`).
have not run. One paired point exists (724 against 904 examples, injection, AUC 0.76 against 0.77).
- The plane-profile leak audit flagged `verify.flag_b` and `ms.operation`. They are withheld from
profile v1. Before any plane-profile program runs, someone must decide whether they track the
label on unseen code (`2026-10-02-decision-engine-harvest-advisory/record.json`).
- Population v3 waits for `benchmarks/independent/prediction-v3.json`, which does not exist. It is
written only after a program is adopted.
Infrastructure.
- Deployment to a separate Kubernetes cluster is planned, not done; the code changes it needs are
- Deployment to a separate Kubernetes cluster is planned, not done. The code changes it needs are
listed with file paths on [Deployment](deployment.md#what-is-implemented-and-what-is-planned).
Instrument gaps.
- A missing provider key is a quiet path in a scan. The plane refuses to start a Task whose Model
names an unset variable (`plane/router.py`, `StartError`). A scan without a key asks the model
nothing, reports only the graph's entailed findings, and records `learning_endpoint_unavailable`
in `manifest.json` (`cli.py`, after `resolve_chat_endpoint` returns nothing); `report.md` has no
line for that reason yet, so the report does not say the model layer was skipped.
- A missing provider key is a quiet path in a scan.
- The plane refuses to start a Task whose Model names an unset variable (`plane/router.py`,
`StartError`).
- A scan without a key asks the model nothing and reports only the graph's entailed findings.
It records `learning_endpoint_unavailable` in `manifest.json` (`cli.py`, after
`resolve_chat_endpoint` returns nothing).
- `report.md` has no line for that reason yet. So the report does not say the model layer was
skipped.