docs: plain-language pass -- README, index, where we stand, scanning, examples, threat model

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-10-02 21:01:10 +02:00
co-authored by Claude Opus 5.5
parent 50f4e69321
commit 552671ca6a
6 changed files with 685 additions and 440 deletions
+238 -144
View File
@@ -1,38 +1,52 @@
# OpenUltraSAST # OpenUltraSAST
OpenUltraSAST is an experimental static security analyser meant to work as a safety net on OpenUltraSAST is an experimental static security analyser. It is meant to be a safety net on
repositories it has never seen: it scans a checkout, or the commits a `git push` would publish, repositories it has never seen. It scans a checkout, or the commits a `git push` would publish. It
and separates what it can prove from what it merely suspects. A scan combines language-scoped keeps what it can prove apart from what it only suspects.
pattern rules, a tree-sitter overlay, a Joern code property graph and, optionally, a model that is
asked only the question the graph cannot settle; every finding carries the evidence that earned
its place. A scan that could not look at something records a degradation instead of reporting a
clean result. `ousast` is the interface; model work over whole repositories runs on a separate
agentic plane (google/ax), and a learned decision engine is in development. Neither is needed
for a scan.
This file is the front door; each section links to its page of the documentation site (`docs/`, A scan combines up to four parts:
- language-scoped pattern rules;
- a tree-sitter overlay (a syntax tree of each file, built by the tree-sitter parser);
- a Joern code property graph (a graph of the program's syntax, control flow and data flow);
- optionally, a model. It is asked only the question the graph cannot settle.
Every finding carries the evidence that earned its place. When a scan could not look at
something, it records a degradation. It does not report a clean result.
`ousast` is the interface. Model work over whole repositories runs on a separate agentic plane
(google/ax). A learned decision engine is in development. A scan needs neither of them.
This file is the front door. Each section links to its page of the documentation site (`docs/`,
`mkdocs.yml`; `uv sync --extra docs && uv run mkdocs serve`). `mkdocs.yml`; `uv sync --extra docs && uv run mkdocs serve`).
## Status ## Status
**v2.0.0, state as of 2026-10-02.** The pre-push safety net is **NO-GO** against every M4 gate: **v2.0.0, state as of 2026-10-02.** The pre-push safety net is **NO-GO** against every M4 gate.
no hook capability is enabled, and the two independent populations spent so far gave engine recall
1 of 11 and then 0 of 17. The learned decision engine ranks but may not block: BLOCK is offered for - No hook capability is enabled.
no family. The project has stopped trying to reach the gates by hand; every detection change now - Two independent populations (case sets frozen before any scan) have been spent so far. Engine
has to win a controlled measurement on held-out repositories. The one page that states all of this recall was 1 of 11 and then 0 of 17.
with the record behind each figure is [docs/where-we-stand.md](docs/where-we-stand.md); the - The learned decision engine ranks but may not block. BLOCK is offered for no family.
release history is in [RELEASE_NOTES.md](RELEASE_NOTES.md).
The project has stopped trying to reach the gates by hand. Every detection change now has to win
a controlled measurement on held-out repositories. One page states all of this, with the record
behind each figure: [docs/where-we-stand.md](docs/where-we-stand.md). The release history is in
[RELEASE_NOTES.md](RELEASE_NOTES.md).
## Install and first scan ## Install and first scan
Which tier you need: a deterministic scan of a checkout or a CI gate is tier 1; the Joern engine Pick the tier you need:
(`standard`, `deep`) is tier 2; running model work over whole repositories on the plane is tier 3,
for maintainers; tier 4 does not exist yet. 1. Tier 1: a deterministic scan of a checkout, or a CI gate.
2. Tier 2: the Joern engine (`standard`, `deep`).
3. Tier 3, for maintainers: model work over whole repositories on the plane.
4. Tier 4 does not exist yet.
### Tier 1: quick scan, no Docker ### Tier 1: quick scan, no Docker
The core install depends only on PyYAML. From a checkout, `uv sync` (or `pip install .`) installs The core install depends only on PyYAML. From a checkout, `uv sync` (or `pip install .`) installs
the `ousast` entry point; `uv sync --group dev` adds the test and lint tools. the `ousast` entry point. `uv sync --group dev` adds the test and lint tools.
```bash ```bash
uv sync uv sync
@@ -41,19 +55,26 @@ uv run ousast scan . --mode quick --fail-on verified # CI: exit 1 on any evide
``` ```
This runs the language-scoped pattern rules, entry-point reachability hints, ranking, This runs the language-scoped pattern rules, entry-point reachability hints, ranking,
verification and scoring: deterministic, reproducible, no network, no keys. What it cannot do: verification and scoring. It is deterministic and reproducible. It needs no network and no keys.
there is no engine, so nothing follows a value across statements or files, and the PHP quick rules
are a low-precision, in-sample development set (per-rule precision lower bounds from 0.03 to 1.0 on What it cannot do:
the real pins, `benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`). The optional
`semantic` extra (`uv sync --extra semantic`) installs the tree-sitter grammars for the overlay. - There is no engine. Nothing follows a value across statements or files.
- The PHP quick rules are a low-precision, in-sample development set. Per-rule precision lower
bounds run from 0.03 to 1.0 on the real pins
(`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`).
The optional `semantic` extra (`uv sync --extra semantic`) installs the tree-sitter grammars for
the overlay.
### Tier 2: standard and deep scans (Docker) ### Tier 2: standard and deep scans (Docker)
`standard` and `deep` need Joern. The Docker image `openultrasast:dev` ships the tool, Joern `standard` and `deep` need Joern. The Docker image `openultrasast:dev` ships three things in one
v4.0.625 and `php-cli` (needed by the PHP frontend) in one image; it is built locally from the image: the tool, Joern v4.0.625 and `php-cli` (the PHP frontend needs it). You build it locally
`Dockerfile` (no published image yet), and it is big: a JRE plus about 2 GB of Joern. The shell from the `Dockerfile`; there is no published image yet. It is big: a JRE plus about 2 GB of Joern.
wrapper builds it on first use and then runs `ousast` in the container with your directory mounted
read-only and the container network off: The shell wrapper builds the image on first use. Then it runs `ousast` in the container. Your
directory is mounted read-only and the container network is off.
```bash ```bash
source /path/to/OpenUltraSAST/ops/shell/ousast.sh # PowerShell: ops/shell/ousast.ps1 source /path/to/OpenUltraSAST/ops/shell/ousast.sh # PowerShell: ops/shell/ousast.ps1
@@ -62,90 +83,129 @@ ousast-with-judge scan . --mode standard # network on, the model is a
``` ```
`docker compose build` and `TARGET=/path/to/repo docker compose run --rm ousast scan /target` do `docker compose build` and `TARGET=/path/to/repo docker compose run --rm ousast scan /target` do
the same without the wrapper. `deep` adds REGRESS, which runs each reproduction snippet with the same without the wrapper.
`docker run` from wherever the scan runs; the engine container has no Docker socket, so run `deep`
from a host install with Joern on the PATH (`pyinfra @local ops/joern.py`, pinned and checksum- `deep` adds REGRESS. REGRESS runs each reproduction snippet with `docker run` from wherever the
verified) and a working `docker`, or accept the recorded `sandbox_unavailable` degradation. scan runs. The engine container has no Docker socket. So run `deep` in one of two ways:
- from a host install with Joern on the PATH (`pyinfra @local ops/joern.py`, pinned and checksum-
verified) and a working `docker`;
- or accept the recorded `sandbox_unavailable` degradation.
Details, including the host install, are in [ops/README.md](ops/README.md). Details, including the host install, are in [ops/README.md](ops/README.md).
Keys and store settings live in `.env` in the working directory (gitignored; `.env.example` lists Keys and store settings live in `.env` in the working directory. The file is gitignored;
the names). `DEEPSEEK_API_KEY` serves every LLM call; `OPENROUTER_API_KEY` is used for embeddings `.env.example` lists the names.
only (`ousast learn memory embed`). `.env` never overrides a variable already exported in your
shell, so `unset` a stale export when a key change seems to have no effect. The plane's memory - `DEEPSEEK_API_KEY` serves every LLM call.
store defaults to a local file store under `~/ousast-results/plane/memory`; an S3-compatible bucket - `OPENROUTER_API_KEY` is used for embeddings only (`ousast learn memory embed`).
is optional (`uv sync --extra s3`, then `OUSAST_MEMORY=s3://<bucket>` with `S3_ENDPOINT` and the - `.env` never overrides a variable already exported in your shell. If a key change seems to have
AWS credentials in `.env`). Nothing in a scan needs it. no effect, `unset` the stale export.
- The plane's memory store defaults to a local file store under `~/ousast-results/plane/memory`.
- An S3-compatible bucket is optional: `uv sync --extra s3`, then `OUSAST_MEMORY=s3://<bucket>`
with `S3_ENDPOINT` and the AWS credentials in `.env`. Nothing in a scan needs it.
### Tier 3: the plane on ax (maintainers) ### Tier 3: the plane on ax (maintainers)
None of this is needed for tier 1 or 2. Today's profile is a single-node kind cluster on the Tier 1 and 2 need none of this. Today's profile is a single-node kind cluster on the maintainer's
maintainer's host. `ops/ax/up.sh` needs `docker`, `go`, `kubectl`, `kind`, `ko` and `ax` on the host.
PATH (it refuses to start when one is missing) and builds Agent Substrate and ax from their
checkouts under `~/.cache/ousast/ax-src/`. Measured idle footprint with two workers: the kind node - `ops/ax/up.sh` needs `docker`, `go`, `kubectl`, `kind`, `ko` and `ax` on the PATH. It refuses to
takes 1.7 GiB of memory (27% of the 7.7 GiB host) and 4.9 GB of disk for images, with several GB start when one is missing.
more of Go caches during the build; the worker pool is sized for that 7 GB host. The verify and - It builds Agent Substrate and ax from their checkouts under `~/.cache/ousast/ax-src/`.
roles tasks need `DEEPSEEK_API_KEY`, and memory lives in the local file store or the S3 store - Measured idle footprint with two workers: the kind node takes 1.7 GiB of memory (27% of the
(RustFS is the tested server). Bring-up, doctor and the smoke Run: [ops/ax/README.md](ops/ax/README.md). 7.7 GiB host) and 4.9 GB of disk for images. The build uses several GB more for Go caches.
- The worker pool is sized for that 7 GB host.
- The verify and roles tasks need `DEEPSEEK_API_KEY`.
- Memory lives in the local file store or the S3 store. RustFS is the tested server.
Bring-up, doctor and the smoke Run: [ops/ax/README.md](ops/ax/README.md).
### Tier 4: Kubernetes (k3s), planned ### Tier 4: Kubernetes (k3s), planned
No deployment to any cluster other than that kind cluster has been made. The target is two Nothing has been deployed to any cluster other than that kind cluster. The target is two profiles
profiles of one code base, `kind` locally and `k3s` in production, where the same `ousast plane run` of one code base: `kind` locally and `k3s` in production. The same `ousast plane run` would
submits a Run in a remote execution mode: it needs a kubeconfig context or the cluster's ingress submit a Run in a remote execution mode. It needs:
(and an `imagePullSecret` on the worker ServiceAccount for a private package), runner and engine
images published to GHCR and pinned by digest, and the S3 store as the artifact path, with no - a kubeconfig context or the cluster's ingress (and an `imagePullSecret` on the worker
receiver on a laptop. The plan, and the code changes it needs with file paths, are in ServiceAccount for a private package);
- runner and engine images published to GHCR and pinned by digest;
- the S3 store as the artifact path, with no receiver on a laptop.
The plan, and the code changes it needs with file paths, are in
[docs/deployment.md](docs/deployment.md#production-topology-ax-on-a-kubernetes-cluster). [docs/deployment.md](docs/deployment.md#production-topology-ax-on-a-kubernetes-cluster).
## Scan modes ## Scan modes
`quick` runs the pattern rules, hints, ranking, verification and score and needs nothing. There are three modes. Each adds to the one before.
`standard` adds the MAP stage: complexity map, semantic overlay, authorization obligations and the
Joern model layer, which builds one code property graph per repository and decides taint, - `quick` runs the pattern rules, hints, ranking, verification and score. It needs nothing.
guard-dominance and configuration questions on it; a model is asked only the residual question - `standard` adds the MAP stage: complexity map, semantic overlay, authorization obligations and
(without a key it is skipped and only the graph's entailed findings are reported). `deep` adds the Joern model layer. The model layer builds one code property graph per repository. It decides
REGRESS: promoted candidates are loaded by a small snippet in a Docker sandbox and get a taint, guard-dominance and configuration questions on that graph. A model is asked only the
`triggerable` or `not_triggerable` verdict. A missing engine is a recorded degradation residual question. Without a key, that question is skipped and only the graph's entailed
(`cpg_unavailable`, `sandbox_unavailable`), never a clean result. findings are reported.
- `deep` adds REGRESS. Promoted candidates are loaded by a small snippet in a Docker sandbox. Each
gets a `triggerable` or `not_triggerable` verdict.
A missing engine is a recorded degradation (`cpg_unavailable`, `sandbox_unavailable`), never a
clean result.
`--fail-on never|findings|verified|worth-fixing` sets the exit code. `--config openultrasast.toml`
loads settings. Every run writes these files under `<target>/.openultrasast/runs/<scan-id>/`:
`manifest.json`, `findings.json`, `verification.json`, `score.json`, `report.md`, `report.sarif`
and `trace/events.jsonl`.
`ousast pre-push` (experimental) analyses the commits a push would publish against their base.
Only new or worsened defects become candidates. It is advisory by default. Its capability registry
is empty, so it emits no normal alert today.
`--fail-on never|findings|verified|worth-fixing` sets the exit code and `--config openultrasast.toml`
loads settings. Every run writes `manifest.json`, `findings.json`, `verification.json`,
`score.json`, `report.md`, `report.sarif` and `trace/events.jsonl` under
`<target>/.openultrasast/runs/<scan-id>/`. `ousast pre-push` (experimental) analyses the commits a
push would publish against their base so that only new or worsened defects are candidates; it is
advisory by default and its capability registry is empty, so no normal alert is emitted today.
Flows and the per-language rule table: [docs/scanning.md](docs/scanning.md). Flows and the per-language rule table: [docs/scanning.md](docs/scanning.md).
## How detection works ## How detection works
Three representations, from cheap to expensive: text (regex quick rules, one statement at a time), The tool uses three representations of code, from cheap to expensive:
tree (the tree-sitter CST and an IR per function, for the overlay and the obligations) and graph
(the Joern CPG, the only one that follows a value across statements, calls and files). Taint runs
as fact tables, queries and verdicts on the graph; absence bugs (an operation that should have
been guarded) are found as obligations, dominance and configuration census; a model answers only
the residual question. [docs/detection-techniques.md](docs/detection-techniques.md) explains each
technique with the code it comes from and what it misses;
[docs/architecture.md](docs/architecture.md) gives the stage order, the evidence ladder (a state
machine, not a label a model may assign), the central CWE policy and the project score, and the
self-improving loops (`ousast improve`, with `--memory` for proposals from plane rows).
Quick rules cover C/C++, Python, JavaScript/TypeScript, Java with Groovy templates, and PHP, and 1. Text: regex quick rules, one statement at a time.
run only against files of their language. The detection gate (`uv run python -m openultrasast.gate`) 2. Tree: the tree-sitter CST (concrete syntax tree) and an IR (intermediate representation) per
enforces at least 90% recall and under 10% false positives per language on the bundled cheat-sheet function. The overlay and the obligations use it.
corpora; on this tree it reports 93.62% recall (44/47) and 0.0% false positives. Those corpora show 3. Graph: the Joern CPG. It is the only one that follows a value across statements, calls and
that a rule change did not break a known detection; they are not a recall estimate on real code, files.
and PHP is not in the gate. Framework knowledge (WordPress, Flask, Django, Express, Spring) is
tagged as an optional prior in `src/openultrasast/ruleset/frameworks.toml`, so its contribution can Taint runs as fact tables, queries and verdicts on the graph. Absence bugs are operations that
be measured and switched off. should have been guarded. They are found as obligations, dominance and configuration census. A
model answers only the residual question.
- [docs/detection-techniques.md](docs/detection-techniques.md) explains each technique, with the
code it comes from and what it misses.
- [docs/architecture.md](docs/architecture.md) gives the stage order and the evidence ladder (a
state machine, not a label a model may assign). It also covers the central CWE policy, the
project score and the self-improving loops (`ousast improve`, with `--memory` for proposals from
plane rows).
Quick rules cover C/C++, Python, JavaScript/TypeScript, Java with Groovy templates, and PHP. Each
rule runs only against files of its language.
The detection gate (`uv run python -m openultrasast.gate`) enforces at least 90% recall and under
10% false positives per language on the bundled cheat-sheet corpora. On this tree it reports
93.62% recall (44/47) and 0.0% false positives. Those corpora show that a rule change did not
break a known detection. They are not a recall estimate on real code, and PHP is not in the gate.
Framework knowledge (WordPress, Flask, Django, Express, Spring) is tagged as an optional prior in
`src/openultrasast/ruleset/frameworks.toml`. So its contribution can be measured and switched off.
## The plane on ax ## The plane on ax
Model-driven work over whole repositories runs as a Run manifest, a DAG of ax Tasks, each in its Model-driven work over whole repositories runs as a Run manifest. A Run is a DAG of ax Tasks. Each
own gVisor actor on Agent Substrate. ax is the only agentic executor; there is no local Task runs in its own gVisor actor (an isolated sandbox) on Agent Substrate. ax is the only agentic
subprocess path. Each Task binds at most one Model and its own `usd`/`calls` budget, egress is executor; there is no local subprocess path.
deny-by-default per Task (hostnames only), and the provider key travels only in the Task's start
request, never in a manifest or an image. Measured on the first increment: the plane agreed on 16 - Each Task binds at most one Model and its own `usd`/`calls` budget.
of 20 declared sites at $0.0205 per candidate. - Egress is deny-by-default per Task, by hostname only.
- The provider key travels only in the Task's start request, never in a manifest or an image.
Measured on the first increment: the plane agreed on 16 of 20 declared sites at $0.0205 per
candidate.
```bash ```bash
uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image
@@ -155,42 +215,61 @@ uv run ousast plane remember validation-46 # ingest the run's rows
``` ```
`workspaces`, `harvest`, `alerts-engine` and `memory-normalise` generate Runs, their inputs and `workspaces`, `harvest`, `alerts-engine` and `memory-normalise` generate Runs, their inputs and
the store's rewrites. The flow and the task catalogue: [docs/plane.md](docs/plane.md); what runs the store's rewrites. The flow and the task catalogue: [docs/plane.md](docs/plane.md). What runs
where today and what a separate cluster would take: [docs/deployment.md](docs/deployment.md). where today, and what a separate cluster would take: [docs/deployment.md](docs/deployment.md).
HarnessX, the earlier agentic extra, was retired 2026-09-30: a `[harnessx]` section is ignored HarnessX, the earlier agentic extra, was retired 2026-09-30. A `[harnessx]` section is ignored
with one warning, and `[models] verifier`, `[fusion] panel_model` and `[fusion] decider_model` fail with one warning. `[models] verifier`, `[fusion] panel_model` and `[fusion] decider_model` fail and
naming the plane replacement, as do config keys nothing ever read. Fusion is deterministic. name the plane replacement. Config keys that nothing ever read fail the same way. Fusion is
deterministic.
## Memory ## Memory
One store, keyed by repository and pin, holds what plane runs and the decision engine learned: One store holds what plane runs and the decision engine learned. It is keyed by repository and pin.
rows (facts, verdicts, unit costs, alerts, coverage, feature records) and content-addressed blobs It holds two kinds of data:
(excerpts, embeddings, cached responses, compiled programs). `OUSAST_MEMORY` selects the backend:
a local file store, or `s3://<bucket>` on any S3-compatible server with versioning, lifecycle - rows: facts, verdicts, unit costs, alerts, coverage and feature records;
rules, object tags and S3 Select; RustFS is the tested server. An admin sets the bucket up once; - content-addressed blobs: excerpts, embeddings, cached responses and compiled programs.
the store verifies it at every start and refuses to run, naming each missing piece, rather than
fall back. Layout and the read and write paths: [docs/memory.md](docs/memory.md); the bucket `OUSAST_MEMORY` selects the backend. It is either a local file store or `s3://<bucket>` on any
policy and admin steps: [docs/rustfs.md](docs/rustfs.md); how stored rows are meant to turn into S3-compatible server. The S3 server must support versioning, lifecycle rules, object tags and S3
better decisions: [docs/memory-and-detection.md](docs/memory-and-detection.md). Select. RustFS is the tested server.
An admin sets the bucket up once. The store verifies it at every start. If anything is missing, it
refuses to run and names each missing piece. It does not fall back.
- Layout and the read and write paths: [docs/memory.md](docs/memory.md).
- The bucket policy and admin steps: [docs/rustfs.md](docs/rustfs.md).
- How stored rows are meant to turn into better decisions:
[docs/memory-and-detection.md](docs/memory-and-detection.md).
## Decision engine and evaluation ## Decision engine and evaluation
A learned decision engine is being built to replace hand-tuned detection edits: every instrument A learned decision engine is being built. It is meant to replace hand-tuned detection edits. It
produces signals, never verdicts; labels come from ground truth only; a compiled AI classifier with follows three rules:
local memory turns retrieved similar cases and the candidate's signals into a calibrated
probability with BLOCK and ADVISORY operating points. It is **in development, not adopted**: no
scan, pre-push check or report uses it. Measured on six families, on repositories the compiled
program was not shown, the pooled AUC of its score runs from 0.77 (injection) to 0.96 (access
control); the stricter within-pair AUC is 0.72 to 0.93; BLOCK is offered for no family
(`benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`). Maintainer surface:
`ousast learn labels|memory|compile|evaluate|curve|experiment|audit-leaks`. Design, slices and
the A/B protocol: [docs/decision-engine.md](docs/decision-engine.md).
How anything is measured on code the tool was not tuned on, the pair corpora, the independent - Every instrument produces signals, never verdicts.
populations and the M4 gates, and the fold discipline that keeps learned decisions out of their - Labels come from ground truth only.
own training data: [docs/evaluation.md](docs/evaluation.md). Population v3 (PHP, 15 cases) is - A compiled AI classifier with local memory turns retrieved similar cases and the candidate's
frozen as the one-time final check and nothing in the tree may read it. signals into a calibrated probability. It has BLOCK and ADVISORY operating points.
It is **in development, not adopted**. No scan, pre-push check or report uses it.
It was measured on six families, on repositories the compiled program was not shown:
- The pooled AUC (area under the ROC curve) of its score runs from 0.77 (injection) to 0.96
(access control).
- The stricter within-pair AUC is 0.72 to 0.93 across the same families.
- BLOCK is offered for no family
(`benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`).
Maintainer surface: `ousast learn labels|memory|compile|evaluate|curve|experiment|audit-leaks`.
Design, slices and the A/B protocol: [docs/decision-engine.md](docs/decision-engine.md).
[docs/evaluation.md](docs/evaluation.md) explains how anything is measured on code the tool was not
tuned on. It covers the pair corpora, the independent populations and the M4 gates. It also covers
the fold discipline that keeps learned decisions out of their own training data. Population v3
(PHP, 15 cases) is frozen as the one-time final check. Nothing in the tree may read it.
```bash ```bash
uv run ousast benchmark benchmarks/manifests/python-vulnerable.toml --mode quick uv run ousast benchmark benchmarks/manifests/python-vulnerable.toml --mode quick
@@ -201,19 +280,26 @@ uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dr
## Token economics ## Token economics
Model tokens are the one running cost. At runtime the plane routes them: one Model and one budget Model tokens are the one running cost. At runtime the plane routes them:
per Task, a metered client that refuses the next call at the ceiling, repository facts computed
model-free and reused, blobs replayed at $0, triage before the hunt and a tie-break only on - one Model and one budget per Task;
disputed candidates. In development, 85.5% of the maintainers' coding-session tokens were tool - a metered client that refuses the next call at the ceiling;
input and output rather than reasoning (down from 90%), and the remedies in `benchmarks/dev/` - repository facts computed without a model and reused;
target that. Mechanisms, - blobs replayed at $0, with no new model call;
the measured costs and the before-and-after table: [docs/token-ergonomics.md](docs/token-ergonomics.md). - triage before the hunt;
- a tie-break only on disputed candidates.
In development, 85.5% of the maintainers' coding-session tokens were tool input and output rather
than reasoning. That is down from 90%. The remedies in `benchmarks/dev/` target that share.
Mechanisms, the measured costs and the before-and-after table:
[docs/token-ergonomics.md](docs/token-ergonomics.md).
## Examples, threat model ## Examples, threat model
End-to-end walkthroughs, from a quick scan to an MCP client: [docs/examples.md](docs/examples.md). - End-to-end walkthroughs, from a quick scan to an MCP client: [docs/examples.md](docs/examples.md).
What the tool trusts and what it does not (scanned code is untrusted input, secret redaction, - What the tool trusts and what it does not: [docs/threat-model.md](docs/threat-model.md). This
per-task egress, budgets, the `deep` sandbox): [docs/threat-model.md](docs/threat-model.md). covers scanned code as untrusted input, secret redaction, per-task egress, budgets and the
`deep` sandbox.
## Contributing and maintainer tooling ## Contributing and maintainer tooling
@@ -228,25 +314,33 @@ uv run python dagger/ci.py # containerized CI pipeline
``` ```
An agent that can run shell commands needs nothing but the CLI. For [OpenCode](https://opencode.ai) An agent that can run shell commands needs nothing but the CLI. For [OpenCode](https://opencode.ai)
there are two project skills in `.agents/skills/`: `openultrasast-scan` (modes and their there are two project skills in `.agents/skills/`:
prerequisites, `--fail-on`, the evidence ladder, recorded degradations, run artifacts) and
`openultrasast-triage` (false-positive reasons, what an adjudication may feed, fixing and - `openultrasast-scan`: modes and their prerequisites, `--fail-on`, the evidence ladder, recorded
re-scanning). `ousast mcp` runs a narrow MCP server over stdio with ten tools (`openultrasast.scan`, degradations and run artifacts.
`status`, `findings`, `get_finding`, `evidence`, `artifacts`, `benchmark`, `explain`, - `openultrasast-triage`: false-positive reasons, what an adjudication may feed, and fixing and
`propose_patch`, `export_report`); no tool runs a shell, Docker or a free-form command: re-scanning.
`ousast mcp` runs a narrow MCP (Model Context Protocol) server over stdio. It has ten tools:
`openultrasast.scan`, `status`, `findings`, `get_finding`, `evidence`, `artifacts`, `benchmark`,
`explain`, `propose_patch` and `export_report`. No tool runs a shell, Docker or a free-form
command.
```jsonc ```jsonc
{ "command": "uv", "args": ["run", "ousast", "mcp"] } { "command": "uv", "args": ["run", "ousast", "mcp"] }
``` ```
The `kiro-*` skills, `AGENTS.md` and the Kiro-style specifications under `.kiro/` are the The `kiro-*` skills, `AGENTS.md` and the Kiro-style specifications under `.kiro/` are the
maintainer's development tooling and can be ignored by users. Other maintainer commands: maintainer's development tooling. Users can ignore them. Other maintainer commands:
`ousast repos` (pinned known-vulnerable checkouts) and `ousast model candidates` (what the
candidate enumerator can reach). - `ousast repos`: pinned known-vulnerable checkouts.
- `ousast model candidates`: what the candidate enumerator can reach.
## Licence ## Licence
OpenUltraSAST is licensed under the [Apache License 2.0](LICENSE). Third-party material keeps its OpenUltraSAST is licensed under the [Apache License 2.0](LICENSE). Third-party material keeps its
own terms: Joern (Apache-2.0) is downloaded at image build time, the tree-sitter wheels of the own terms:
`semantic` extra are MIT, and the published benchmark cases were licence-checked one by one (v1's
OpenCVE case is BUSL-1.1, see [RELEASE_NOTES.md](RELEASE_NOTES.md)). - Joern (Apache-2.0) is downloaded at image build time.
- The tree-sitter wheels of the `semantic` extra are MIT.
- The published benchmark cases were licence-checked one by one. v1's OpenCVE case is BUSL-1.1, see [RELEASE_NOTES.md](RELEASE_NOTES.md).
+28 -17
View File
@@ -1,7 +1,7 @@
# End-to-end examples # End-to-end examples
Each example is self-contained. Artifacts land under Each example stands on its own. Artifacts land under `<target>/.openultrasast/runs/<scan-id>/`.
`<target>/.openultrasast/runs/<scan-id>/` and `.openultrasast/` is gitignored. `.openultrasast/` is gitignored.
## 1. Quick scan a local repo (deterministic, no keys) ## 1. Quick scan a local repo (deterministic, no keys)
@@ -21,7 +21,7 @@ ousast scan . --mode standard # Joern model layer, container network off:
ousast-with-judge scan . --mode standard # network on, so the model is asked (needs DEEPSEEK_API_KEY) ousast-with-judge scan . --mode standard # network on, so the model is asked (needs DEEPSEEK_API_KEY)
``` ```
Without a key the graph still decides: its entailed findings are reported, and the LLM's residual Without a key, the graph still decides. Its entailed findings are reported. The LLM's residual
and suspicion questions go unasked. and suspicion questions go unasked.
## 3. CI gate ## 3. CI gate
@@ -52,10 +52,11 @@ uv run ousast pairs --split holdout --profile human # filter by declared split
uv run python -m openultrasast.pair_gate # CI: local pairs must all pass uv run python -m openultrasast.pair_gate # CI: local pairs must all pass
``` ```
Each case is two isolated trees (vuln file vs patched file, same relative path). Each case is two isolated trees: the vuln file and the patched file, at the same relative path.
`pair_correct` means the harness fired on the vuln side and stayed silent on the
fix. Misses and fix-side leaks become `miss`/`fp` signals for `ousast improve`. - `pair_correct` means the harness fired on the vuln side and stayed silent on the fix.
The catalog and public dataset pointers live in `benchmarks/pairs/`. - Misses become `miss` signals and fix-side leaks become `fp` signals for `ousast improve`.
- The catalog and public dataset pointers live in `benchmarks/pairs/`.
## 6. Self-improve the ruleset from benchmark feedback ## 6. Self-improve the ruleset from benchmark feedback
@@ -68,10 +69,11 @@ uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml
## 7. Running the agentic plane ## 7. Running the agentic plane
Model work over whole repositories runs as a Run manifest on ax, one isolated actor Model work over whole repositories runs as a Run manifest on ax. Each task gets one isolated
per task (bring-up: [ops/ax/README.md](ops/ax/README.md)). The provider key is actor. Bring-up is in [ops/ax/README.md](ops/ax/README.md).
read from the environment (or `.env`, which never overrides an exported variable) by
`ousast` and sent only in each task's start request. `ousast` reads the provider key from the environment, or from `.env`. `.env` never overrides an
exported variable. The key is sent only in each task's start request.
```bash ```bash
uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image
@@ -83,16 +85,25 @@ uv run ousast plane remember validation-46 # re-ingest the run's m
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run --memory uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run --memory
``` ```
Each task in the Run carries its own Model and `budget: {usd, calls}`; a task that Each task in the Run carries its own Model and `budget: {usd, calls}`.
reaches its ceiling stops as `unfinished` and resumes on a rerun with a larger
budget. `ousast plane status` prints the token attribution table: calls, prompt, - A task that reaches its ceiling stops as `unfinished`. It resumes on a rerun with a larger
cache-hit and output tokens and USD per task. budget.
- `ousast plane status` prints the token attribution table. It shows calls, prompt, cache-hit and
output tokens, and USD per task.
## 8. Drive it from an MCP client (OpenCode / IDE) ## 8. Drive it from an MCP client (OpenCode / IDE)
MCP (Model Context Protocol) lets an editor or agent call the tool's functions.
```jsonc ```jsonc
{ "command": "uv", "args": ["run", "ousast", "mcp"] } { "command": "uv", "args": ["run", "ousast", "mcp"] }
``` ```
Then: `openultrasast.scan {path}` → `run_dir` → `openultrasast.findings {run_dir}` → Then call the tools in this order:
`openultrasast.explain {run_dir, finding_id}`. See the README section "Agent integrations".
1. `openultrasast.scan {path}` returns a `run_dir`.
2. `openultrasast.findings {run_dir}` lists the findings.
3. `openultrasast.explain {run_dir, finding_id}` explains one.
See the README section "Agent integrations".
+24 -18
View File
@@ -1,13 +1,13 @@
# OpenUltraSAST # OpenUltraSAST
OpenUltraSAST is an experimental static security analyser meant to work as a safety net on OpenUltraSAST is an experimental static security analyser. It is meant to be a safety net on
repositories it has never seen: it scans a checkout, or the commits a `git push` would publish, repositories it has never seen. It scans a checkout, or the commits a `git push` would publish. It
and separates what it can prove from what it merely suspects. A scan that could not look at keeps what it can prove apart from what it only suspects. When a scan could not look at
something records a degradation instead of reporting a clean result. something, it records a degradation. It does not report a clean result.
**State as of 2026-10-02 (v2.0.0):** the pre-push safety net is **NO-GO** for rollout. **State as of 2026-10-02 (v2.0.0):** the pre-push safety net is **NO-GO** for rollout.
No hook capability is enabled, and no independent population has passed the qualification gates No hook capability is enabled. No independent population (a case set frozen before any scan) has
(see [Evaluation](evaluation.md)). passed the qualification gates (see [Evaluation](evaluation.md)).
## Three layers ## Three layers
@@ -17,6 +17,9 @@ No hook capability is enabled, and no independent population has passed the qual
| **The agentic plane on ax** | Model-driven work over whole repositories, run as a DAG of tasks on google/ax, one isolated actor per task. What the runs learn is kept in a memory store. Maintainer surface: `ousast plane`. | [The plane on ax](plane.md), [Memory](memory.md) | | **The agentic plane on ax** | Model-driven work over whole repositories, run as a DAG of tasks on google/ax, one isolated actor per task. What the runs learn is kept in a memory store. Maintainer surface: `ousast plane`. | [The plane on ax](plane.md), [Memory](memory.md) |
| **The decision engine** | A learned classifier that weighs every instrument's signals against similar labelled cases from memory. **In development, not adopted**: no scan, pre-push check or report uses it. Maintainer surface: `ousast learn`. | [Memory and detection](memory-and-detection.md), [Decision engine](decision-engine.md) | | **The decision engine** | A learned classifier that weighs every instrument's signals against similar labelled cases from memory. **In development, not adopted**: no scan, pre-push check or report uses it. Maintainer surface: `ousast learn`. | [Memory and detection](memory-and-detection.md), [Decision engine](decision-engine.md) |
A code property graph is a graph of the program's syntax, control flow and data flow. Joern builds
it.
## How they connect ## How they connect
```mermaid ```mermaid
@@ -33,31 +36,34 @@ flowchart LR
engine -. "intended, not adopted: BLOCK / ADVISORY" .-> report engine -. "intended, not adopted: BLOCK / ADVISORY" .-> report
``` ```
- The **scanners** produce findings and, inside plane Runs, the `alerts` that become memory rows. - The **scanners** produce findings. Inside plane Runs, they also produce the `alerts` that become
- The **plane** runs model work (verify passes, roles) and model-free bookkeeping (facts, agreement, memory rows.
feature records) per case; its `remember` task turns a case's artifacts into rows of the - The **plane** runs model work per case: verify passes and roles. It also runs model-free
[memory store](memory.md). bookkeeping per case: facts, agreement and feature records. Its `remember` task turns a case's
- Memory feeds two consumers. `ousast improve --memory` turns stored rows into rule-status artifacts into rows of the [memory store](memory.md).
proposals that pass through the same validator and gate as every other edit. The decision engine - Memory feeds two consumers:
retrieves labelled examples from the same store. 1. `ousast improve --memory` turns stored rows into rule-status proposals. They pass through the
same validator and gate as every other edit.
2. The decision engine retrieves labelled examples from the same store.
- The decision engine's output (dotted line) is the intended path. Nothing user-facing reads it - The decision engine's output (dotted line) is the intended path. Nothing user-facing reads it
today. today.
## Where to go next ## Where to go next
- The current state in one page, every figure with its record: capability against the M4 gates, the - The current state in one page, every figure with its record: [Where we stand](where-we-stand.md).
step-by-step approach, ax on the server, token economics before and after, and the open items: It covers capability against the M4 gates, the step-by-step approach, ax on the server, token
[Where we stand](where-we-stand.md). economics before and after, and the open items.
- Run a scan: [Examples](examples.md), and the commands in [Scanning](scanning.md). - Run a scan: [Examples](examples.md), and the commands in [Scanning](scanning.md).
- Understand a finding's evidence and score: [Architecture](architecture.md). - Understand a finding's evidence and score: [Architecture](architecture.md).
- The program analysis under a scan, what each representation can see and what it misses: - The program analysis under a scan, what each representation can see and what it misses:
[Detection techniques](detection-techniques.md). [Detection techniques](detection-techniques.md).
- Operate the plane and its store: [ax on this host](ops/ax/README.md) and - Operate the plane and its store: [ax on this host](ops/ax/README.md) and
[RustFS setup](rustfs.md). [RustFS setup](rustfs.md).
- Where the plane runs today and what a separate Kubernetes cluster would take (planned, not - Where the plane runs today, and what a separate Kubernetes cluster would take (planned, not
done): [Deployment](deployment.md). done): [Deployment](deployment.md).
- Where the model spend goes, per Task at runtime and in the maintainers' own coding sessions, - Where the model spend goes, per Task at runtime and in the maintainers' own coding sessions,
and what is reused instead of paid again: [Token ergonomics](token-ergonomics.md). and what is reused instead of paid again: [Token ergonomics](token-ergonomics.md).
- Trust boundaries and hardening: [Threat model](threat-model.md). - Trust boundaries and hardening: [Threat model](threat-model.md).
Every number in these pages is quoted from a committed record whose path is given next to it. Every number in these pages is quoted from a committed record. The record's path is given next to
the number.
+59 -33
View File
@@ -1,13 +1,21 @@
# Scanning # Scanning
`ousast scan PATH --mode quick|standard|deep` scans a checkout; `ousast pre-push` checks only This page shows the scan flows.
what a push (or an explicit `--base`/`--head` pair) changes. The stage-by-stage pipeline,
evidence ladder and score are in [Architecture](architecture.md); this page shows the flows. - `ousast scan PATH --mode quick|standard|deep` scans a checkout.
- `ousast pre-push` checks only what a push changes, or what an explicit `--base`/`--head` pair
changes.
The stage-by-stage pipeline, the evidence ladder and the score are in
[Architecture](architecture.md).
## The three scan modes ## The three scan modes
The stages a mode requests are `MODE_STAGES` in `src/openultrasast/stages.py`: `quick` runs The stages each mode requests are listed in `MODE_STAGES` in `src/openultrasast/stages.py`:
STATIC, `standard` adds MAP, `deep` adds REGRESS.
- `quick` runs STATIC.
- `standard` adds MAP.
- `deep` adds REGRESS.
```mermaid ```mermaid
flowchart TD flowchart TD
@@ -38,7 +46,7 @@ Sources: `README.md` (scan modes), `src/openultrasast/cli.py` (`run_stage` calls
## Quick rules per language ## Quick rules per language
Rules run only against files of their language. The bundled ruleset lives in Each rule runs only against files of its language. The bundled ruleset lives in
`src/openultrasast/ruleset/<language>/rules.toml`. `src/openultrasast/ruleset/<language>/rules.toml`.
| Language | Classes (CWE) | | Language | Classes (CWE) |
@@ -49,21 +57,31 @@ Rules run only against files of their language. The bundled ruleset lives in
| Java + Groovy templates | command injection (78), SQLi (89), weak hash (327), deserialization (502), unescaped template XSS (79) | | Java + Groovy templates | command injection (78), SQLi (89), weak hash (327), deserialization (502), unescaped template XSS (79) |
| PHP | SQLi (89), command injection (78), file inclusion (98), path (22), open redirect (601), SSRF (918); code eval (95), unserialize (502) and echo XSS (79) ship as `shadow` rules | | PHP | SQLi (89), command injection (78), file inclusion (98), path (22), open redirect (601), SSRF (918); code eval (95), unserialize (502) and echo XSS (79) ship as `shadow` rules |
The detection gate (`python -m openultrasast.gate`) enforces at least 90% recall and under 10% The detection gate (`python -m openultrasast.gate`) enforces two limits per language on the
false positives per language on the bundled corpora in `benchmarks/manifests/`. Those corpora are bundled corpora in `benchmarks/manifests/`: at least 90% recall and under 10% false positives.
cheat-sheet fixtures: they catch regressions, they do not estimate recall on real code. **PHP is
not in the gate**; its numbers are in-sample development numbers - Those corpora are cheat-sheet fixtures. They catch regressions. They do not estimate recall on
(`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`). See `README.md` for the real code.
gate's per-language output. - **PHP is not in the gate**. Its numbers are in-sample development numbers
(`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`).
See `README.md` for the gate's per-language output.
## The Joern engine ## The Joern engine
The model layer (`src/openultrasast/model/`, `src/openultrasast/cpg/`) is the core of `standard`. The model layer (`src/openultrasast/model/`, `src/openultrasast/cpg/`) is the core of `standard`.
It builds one code property graph per scan with Joern (`joern-parse`, then batched It builds one code property graph per scan with Joern. A code property graph is a graph of the
`joern --script` queries from `cpg/queries/*.sc`) and sends each question to the arbiter of its program's syntax, control flow and data flow. The build runs `joern-parse`, then batched
type: **taint** (source to sink with no discharging sanitizer), **dominance** (does a guard govern `joern --script` queries from `cpg/queries/*.sc`.
the obligated operation) and **configuration** (constant abstraction over a value). Taint facts
ship for C, Java, JavaScript, PHP and Python (`src/openultrasast/ruleset/semantic/*.toml`). The layer sends each question to the arbiter (the decider) of its type:
- **taint**: a flow from source to sink with no discharging sanitizer;
- **dominance**: does a guard govern the obligated operation;
- **configuration**: constant abstraction over a value.
Taint facts ship for C, Java, JavaScript, PHP and Python
(`src/openultrasast/ruleset/semantic/*.toml`).
```mermaid ```mermaid
flowchart LR flowchart LR
@@ -79,25 +97,33 @@ flowchart LR
rung -- "residual question only" --> judge["LLM judge: confirm or contradict, never create a flow"] rung -- "residual question only" --> judge["LLM judge: confirm or contradict, never create a flow"]
``` ```
The Docker image ships Joern and `php-cli` (which the PHP frontend needs); without Joern the The Docker image ships Joern and `php-cli`. The PHP frontend needs `php-cli`. Without Joern, the
layer records `cpg_unavailable` and the scan is otherwise unchanged. Engine installation and the layer records `cpg_unavailable`; the rest of the scan is unchanged.
Joern features used are in [Engine and pre-push](ops/README.md).
How the graph is built, queried and refined, with the records behind each refinement, is in - Engine installation and the Joern features used: [Engine and pre-push](ops/README.md).
[Detection techniques](detection-techniques.md). - How the graph is built, queried and refined, with the records behind each refinement:
[Detection techniques](detection-techniques.md).
## Framework priors ## Framework priors
Rule and taint-fact entries that know a framework or library carry a `framework` or `library` tag Some rule and taint-fact entries know a framework or library. They carry a `framework` or
naming a row of `src/openultrasast/ruleset/frameworks.toml` (for example `wordpress`, `flask`, `library` tag. The tag names a row of `src/openultrasast/ruleset/frameworks.toml` (for example
`django`, `express`, `spring`). The loaders take `priors=`: `"all"` (the default and today's `wordpress`, `flask`, `django`, `express`, `spring`).
behaviour), `"off"` (language-level knowledge only) or a set of ids, so the contribution of
framework knowledge can be measured and switched off. The decision engine's first slice ran with The loaders take `priors=` with one of three values:
priors off ([Decision engine](decision-engine.md#first-measured-slice-2026-10-01)).
- `"all"`: the default and today's behaviour;
- `"off"`: language-level knowledge only;
- a set of ids.
So the contribution of framework knowledge can be measured and switched off. The decision
engine's first slice ran with priors off
([Decision engine](decision-engine.md#first-measured-slice-2026-10-01)).
## Pre-push: the delta check (experimental) ## Pre-push: the delta check (experimental)
`ousast pre-push` analyses the commits a `git push` would publish, or an explicit `ousast pre-push` analyses the commits a `git push` would publish, or an explicit
`--base`/`--head` pair, and compares head with base so that only new or worsened defects are `--base`/`--head` pair. It compares head with base. Only new or worsened defects become
candidates. Sources: `ousast pre-push --help`, `src/openultrasast/push/`, candidates. Sources: `ousast pre-push --help`, `src/openultrasast/push/`,
[Engine and pre-push](ops/README.md#explicit-local-replay). [Engine and pre-push](ops/README.md#explicit-local-replay).
@@ -118,10 +144,10 @@ flowchart TD
exit -- "blocking, explicit opt-in" --> block["May block the push"] exit -- "blocking, explicit opt-in" --> block["May block the push"]
``` ```
- `--deadline` (30 s by default) bounds preparation, both revisions and reporting together; - `--deadline` (30 s by default) bounds preparation, both revisions and reporting together.
`--incomplete-coverage block` opts into failing when coverage is incomplete. - `--incomplete-coverage block` opts into failing when coverage is incomplete.
- The default capability registry is empty pending independent qualification, so no normal alert - The default capability registry is empty until independent qualification passes. So no normal
is emitted today; every candidate stays diagnostic. alert is emitted today. Every candidate stays diagnostic.
- A model is called only with an explicit `--model-config`. - A model is called only with an explicit `--model-config`.
- Hook installation (`ops/install-pre-push`) and the private cache (`--cache-dir`) are documented - Hook installation (`ops/install-pre-push`) and the private cache (`--cache-dir`) are documented
in [Engine and pre-push](ops/README.md#experimental-pre-push-integration). in [Engine and pre-push](ops/README.md#experimental-pre-push-integration).
+74 -51
View File
@@ -1,78 +1,101 @@
# Threat model & hardening # Threat model & hardening
OpenUltraSAST analyzes **untrusted code**. This document states what it trusts, what OpenUltraSAST analyzes **untrusted code**. This page states what it trusts and what it does not.
it does not, and the controls that keep a scan safe to run in CI and safe to share. It also lists the controls that keep a scan safe to run in CI and safe to share.
## Trust boundaries ## Trust boundaries
- **Scanned code is untrusted input.** Quick and standard modes never import, build or run - **Scanned code is untrusted input.** Quick and standard modes never import, build or run the
the target: quick reads source as text, and standard additionally parses it into a code target.
property graph with Joern (`joern-parse`, a parser, not a build) and may send code excerpts - Quick mode reads source as text.
to the configured model. Only `deep` mode executes target-derived code: its REGRESS stage - Standard mode also parses it into a code property graph (a graph of the program's syntax,
loads promoted candidates inside the Docker sandbox below, and it runs only when `docker control flow and data flow) with Joern. It uses `joern-parse`, a parser, not a build.
Standard mode may send code excerpts to the configured model.
- Only `deep` mode executes target-derived code. Its REGRESS stage loads promoted candidates
inside the Docker sandbox described below. It runs only when `docker
info` succeeds (otherwise the stage is skipped and recorded as `sandbox_unavailable`). info` succeeds (otherwise the stage is skipped and recorded as `sandbox_unavailable`).
- **The ruleset and CWE policy are trusted, governed data.** They change only through - **The ruleset and CWE policy are trusted, governed data.** They change in only two ways:
reviewed commits or the bounded self-improvement loop (`ousast improve`), which can through reviewed commits, or through the bounded self-improvement loop (`ousast improve`). The
flip rule *status* and tune score constants but can never edit pattern text or the loop can flip rule *status* and tune score constants. It can never edit pattern text or the
authoritative 0–5 severity. See [architecture.md](architecture.md#self-improvement). authoritative 0–5 severity. See [architecture.md](architecture.md#self-improvement).
- **Provider credentials are the operator's.** The core install (PyYAML only) makes no - **Provider credentials are the operator's.** The core install (PyYAML only) makes no network
network calls on its own. A local scan calls a model only in standard or deep mode, and calls on its own.
only when a key is available: the model layer asks the residual question the graph cannot - A local scan calls a model only in standard or deep mode, and only when a key is available.
settle, and `[models] hunter` enables the tool hunter. LLM calls go to DeepSeek The model layer asks the residual question the graph cannot settle. `[models] hunter`
(`DEEPSEEK_API_KEY`); OpenRouter (`OPENROUTER_API_KEY`) serves embeddings. Keys come from enables the tool hunter.
the environment or a gitignored `.env`, which never overrides an exported variable. The - LLM calls go to DeepSeek (`DEEPSEEK_API_KEY`). OpenRouter (`OPENROUTER_API_KEY`) serves
pre-push check calls a model only with an explicit `--model-config`. Agentic work runs on the plane (`ousast plane run`): `ousast` reads embeddings.
the variable a task's Model names (`secretKey.key`) from the operator's environment or - Keys come from the environment or a gitignored `.env`. `.env` never overrides an exported
`.env` and sends the key only in that task's start request, through Agent Substrate's variable.
router to that task's actor. It is not baked into an image or task template, and it is - The pre-push check calls a model only with an explicit `--model-config`.
never logged or written to artifacts. - Agentic work runs on the plane (`ousast plane run`). `ousast` reads the variable a task's
Model names (`secretKey.key`) from the operator's environment or `.env`. It sends the key
only in that task's start request. The request goes through Agent Substrate's router to that
task's actor. The key is not baked into an image or task template. It is never logged or
written to artifacts.
## Controls ## Controls
### Secret redaction ### Secret redaction
Scan artifacts can quote source that contains live credentials. `redaction.py` masks Scan artifacts can quote source that contains live credentials. `redaction.py` masks
recognizable secret shapes (provider API keys, AWS/GitHub/Slack/Google tokens, bearer recognizable secret shapes before traces (`trace/events.jsonl`) and the markdown report are
tokens, URL-embedded credentials, PEM private keys, and `key = value` assignments for written. It masks:
sensitive names) before traces (`trace/events.jsonl`) and the markdown report are
written. On by default; disable with `[hardening] redact_secrets = false`. - provider API keys;
- AWS/GitHub/Slack/Google tokens;
- bearer tokens;
- URL-embedded credentials;
- PEM private keys;
- `key = value` assignments for sensitive names.
Redaction is on by default. Turn it off with `[hardening] redact_secrets = false`.
### Plane egress ### Plane egress
Each plane task runs in its own actor behind Agent Substrate's egress gateway, which Each plane task runs in its own actor behind Agent Substrate's egress gateway. The gateway denies
denies by default. The task's EgressPolicy (`src/openultrasast/plane/egress.py`) allows exactly three by default. The task's EgressPolicy (`src/openultrasast/plane/egress.py`) allows exactly three
things: plain HTTP to the artifact receiver, and TLS passthrough to the Git hosts of the things:
task's bound Workspaces and to the hosts its bound Model declares
(`openultrasast.io/egress-hosts`). There are no wildcards, no IP addresses and no TLS 1. plain HTTP to the artifact receiver;
interception. The policy is deleted with the actor. 2. TLS passthrough to the Git hosts of the task's bound Workspaces;
3. TLS passthrough to the hosts its bound Model declares (`openultrasast.io/egress-hosts`).
There are no wildcards, no IP addresses and no TLS interception. The policy is deleted with the
actor.
### Cost & CI budgets ### Cost & CI budgets
- **Agentic spend** is bounded per plane task by the Run's `budget: {usd, calls}`. The - **Agentic spend** is bounded per plane task by the Run's `budget: {usd, calls}`.
metered client refuses the next call once either ceiling is reached (the task is - The metered client refuses the next call once either ceiling is reached. The task is then
`unfinished` and resumes on a rerun with a larger budget). A usd budget on a Model `unfinished`. It resumes on a rerun with a larger budget.
without prices is refused rather than metered as zero, and an account or - A usd budget on a Model without prices is refused. It is not metered as zero.
authentication error fails the task and starts nothing further. The standard-mode tool - An account or authentication error fails the task and starts nothing further.
hunter is bounded by the per-tier hunter budgets. - The standard-mode tool hunter is bounded by the per-tier hunter budgets.
- **Output size** is bounded by `[hardening] max_findings` (0 = unlimited). Truncation is - **Output size** is bounded by `[hardening] max_findings` (0 = unlimited). Truncation is
severity-ordered and disclosed as a `budget` degradation in the manifest. severity-ordered. It is disclosed as a `budget` degradation in the manifest.
### Provider reliability ### Provider reliability
LLM/embedding calls retry transient failures (HTTP 429/5xx, connection errors, timeouts) LLM/embedding calls retry transient failures with exponential backoff. Transient failures are
with exponential backoff; non-transient errors (e.g. 4xx, malformed JSON) fail fast. HTTP 429/5xx, connection errors and timeouts. Non-transient errors fail fast, for example 4xx or
malformed JSON.
### Visible degradation & determinism ### Visible degradation & determinism
When an optional engine is missing (Joern: `cpg_unavailable`; Docker: `sandbox_unavailable`), A scan is never silently downgraded. When an optional engine is missing, the scan falls back to
the scan falls back to its deterministic equivalent and records a `degradations` entry in the its deterministic equivalent. It records a `degradations` entry in the manifest (Joern:
manifest — it is never silently downgraded. Without a provider key the model layer asks no `cpg_unavailable`; Docker: `sandbox_unavailable`).
question and reports only what the graph entailed. Runs are reproducible: fixed config,
artifact manifests, prompt hashes, and model identifiers are recorded. Without a provider key, the model layer asks no question. It reports only what the graph
entailed.
Runs are reproducible. The fixed config, artifact manifests, prompt hashes and model identifiers
are recorded.
### MCP surface ### MCP surface
`ousast mcp` exposes only the ten narrow project tools. No tool runs arbitrary shell, `ousast mcp` exposes only the ten narrow project tools. No tool runs arbitrary shell, Docker or
Docker, or internal hunter tools, and no tool accepts a free-form command argument. internal hunter tools. No tool accepts a free-form command argument.
## Sandbox (deep mode) ## Sandbox (deep mode)
The REGRESS stage (`regress/`, `sandbox/runner.py`) runs each snippet with `docker run` The REGRESS stage (`regress/`, `sandbox/runner.py`) runs each snippet with `docker run`. It never
and never mounts the host Docker socket: mounts the host Docker socket.
| Control | Setting | | Control | Setting |
|---|---| |---|---|
@@ -84,5 +107,5 @@ and never mounts the host Docker socket:
| pids | `[sandbox] pids_limit`, default 512 | | pids | `[sandbox] pids_limit`, default 512 |
| timeout | `[sandbox] timeout_seconds`, default 300 s | | timeout | `[sandbox] timeout_seconds`, default 300 s |
Before anything runs, a structural deny-list (`regress/safety.py`) rejects a snippet that A structural deny-list (`regress/safety.py`) checks each snippet before anything runs. It rejects
mentions sockets, curl, host networking or writes under `/workspace`. a snippet that mentions sockets, curl, host networking or writes under `/workspace`.
+259 -174
View File
@@ -1,33 +1,50 @@
# Where we stand # Where we stand
State as of 2026-10-02 (v2.0.0). This page answers five questions in one place. How far is State as of 2026-10-02 (v2.0.0). This page answers five questions in one place:
detection from its gates. How does the project now try to close that gap. How does the model work
run on ax. What did the tokens cost before and after that move. What is still open. Every figure
is quoted from a committed record whose path stands next to it; detail tables stay on the pages
linked from each section.
The goal has not changed: a safety net on repositories the tool has never seen, which says 1. How far is detection from its gates?
BLOCK or ADVISORY about the code a push would change and stays silent on fixes and on benign 2. How does the project now try to close that gap?
commits. Two independent populations have been spent against that goal, with engine recall 1 of 3. How does the model work run on ax?
4. What did the tokens cost before and after that move?
5. What is still open?
Every figure is quoted from a committed record. The record's path stands next to the figure.
Detail tables stay on the pages linked from each section.
The goal has not changed. The tool should be a safety net on repositories it has never seen. It
should say BLOCK or ADVISORY about the code a push would change. It should stay silent on fixes
and on benign commits.
Two independent populations have been spent against that goal. An independent population is a
set of cases frozen before any scan. Engine recall was 1 of
11 and then 0 of 17. The hand edits that closed misses on the first population did not carry to 11 and then 0 of 17. The hand edits that closed misses on the first population did not carry to
the second. So the question is no longer which rule to add next. It is how a change to detection the second. So the question is no longer which rule to add next. The question is this: how can a
can be shown to work on code it was not tuned on, at a cost per candidate that can be paid for change to detection be shown to work on code it was not tuned on? And can that be done at a cost
every push. per candidate that can be paid for every push?
The answer this page records: the pre-push safety net is NO-GO against every M4 gate, and the The answer this page records:
project has stopped trying to reach the gates by hand. Every detection change now has to win a
controlled measurement on held-out repositories, run on an agentic plane whose cost per candidate - The pre-push safety net is NO-GO against every M4 gate.
is known. The pieces of that method exist and have been measured. None of them has produced a - The project has stopped trying to reach the gates by hand.
BLOCK decision yet. - Every detection change now has to win a controlled measurement on held-out repositories. The
measurement runs on an agentic plane whose cost per candidate is known.
- The pieces of that method exist and have been measured. None of them has produced a BLOCK
decision yet.
## 1. Detection capability and goals: NO-GO, measured ## 1. Detection capability and goals: NO-GO, measured
No M4 gate is met on an independent population. The qualification milestone for the pre-push No M4 gate is met on an independent population. M4 is the qualification milestone for the
safety net (M4) requires, together: at least 95% actionable precision, at least 90% supported pre-push safety net. It requires all of these together:
recall, at least 95% supported-check completion, the runtime gates, and zero fixed-side or benign
false alerts. A population is frozen before any - at least 95% actionable precision;
case is scanned, scored once under a pre-registered protocol, then spent; it can inform changes - at least 90% supported recall;
but can qualify nothing afterwards ([Evaluation](evaluation.md#independent-populations-and-the-m4-gates)). - at least 95% supported-check completion;
- the runtime gates;
- zero fixed-side or benign false alerts.
A population is frozen before any case is scanned. It is scored once under a pre-registered
protocol, then spent. After that it can inform changes but can qualify nothing
([Evaluation](evaluation.md#independent-populations-and-the-m4-gates)).
| Gate | Required | Latest independent result | Record | | Gate | Required | Latest independent result | Record |
| --- | --- | --- | --- | | --- | --- | --- | --- |
@@ -36,156 +53,217 @@ but can qualify nothing afterwards ([Evaluation](evaluation.md#independent-popul
| Supported-check completion | >= 95% | 13,206 of 108,374 | `results-v2.json` | | Supported-check completion | >= 95% | 13,206 of 108,374 | `results-v2.json` |
| Fixed-side and benign alerts | zero | 0 fixed-side, which the record calls met trivially since nothing was detected; 3 benign alerts | `results-v2.json` | | Fixed-side and benign alerts | zero | 0 fixed-side, which the record calls met trivially since nothing was detected; 3 benign alerts | `results-v2.json` |
Three later measurements narrow the picture without changing the verdict. Three later measurements narrow the picture. They do not change the verdict.
- The plane reproduces the script pipeline's agreement at a known cost. On the model-driven - **The plane reproduces the script pipeline's agreement at a known cost.** The test used the
pipeline's validation set (46 candidates, 15 cases of v2, 20 declared vulnerable sites), the model-driven pipeline's validation set: 46 candidates, 15 cases of v2, 20 declared vulnerable
plane's verify passes a and b plus a tie-break on the 10 disputed candidates agreed on 16 of 20 sites. The plane ran verify passes a and b, plus a tie-break on the 10 disputed candidates. It
declared sites. The script reference agreed on 16 as well. The cost was $0.0205 per candidate agreed on 16 of 20 declared sites. The script reference agreed on 16 as well. The cost was
(`benchmarks/independent/plane-increment-2.json`). The record carries two caveats. The tie-break $0.0205 per candidate (`benchmarks/independent/plane-increment-2.json`). The record carries two
was chosen after seeing the first measurement on the same set, so this compares plane with caveats:
script and qualifies nothing. And the passes flag fewer declared sites than the reference in 1. The tie-break was chosen after seeing the first measurement on the same set. So this compares
any pass, 17 against 19; that count is a correction in `plane-increment-1.json`, whose first plane with script and qualifies nothing.
reading took agree's `declared_sites_matched` (19, coverage by candidates) for "found in either 2. The passes flag fewer declared sites than the reference in any pass: 17 against 19 (passes against reference). That
pass". count is a correction in `plane-increment-1.json`. Its first reading took agree's
- The learned decision engine ranks, but may not block. On six families, evaluated on `declared_sites_matched` (19, coverage by candidates) to mean "found in either pass".
repositories the compiled program was not shown, the pooled AUC of its score runs from 0.77 - **The learned decision engine ranks, but may not block.** It was evaluated on six families, on
(injection) to 0.96 (access control). The stricter reading is the within-pair AUC, which asks repositories the compiled program was not shown.
whether the vulnerable side of a fix scores above its own fixed side: 0.72 to 0.93. BLOCK is - The pooled AUC (area under the ROC curve) of its score runs from 0.77 (injection) to 0.96
offered for no family; calibration holds only for output encoding, and there no fold reaches (access control).
the required precision (`benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`). - The stricter reading is the within-pair AUC. It asks whether the vulnerable side of a fix
The per-family table is on [Decision engine](decision-engine.md#second-slice-six-families-2026-10-02). scores above its own fixed side: 0.72 to 0.93.
- PHP coverage is in-sample. The PHP quick rules were written and tightened while looking at - BLOCK is offered for no family. Calibration holds only for output encoding. There, no fold
the pins and fixtures they are scored on, so the record says so itself (`provenance.in_sample` in reaches the required precision
`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`): a development check, not (`benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`).
a recall estimate. On real PHP code the Joern engine is the instrument. - The per-family table is on
[Decision engine](decision-engine.md#second-slice-six-families-2026-10-02).
- **PHP coverage is in-sample.** The PHP quick rules were written and tightened while looking at
the pins and fixtures they are scored on. The record says so itself (`provenance.in_sample` in
`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`). It is a development
check, not a recall estimate. On real PHP code, the Joern engine is the instrument.
What has not been touched: population v3, 15 PHP cases, frozen as the one-time final check. No What has not been touched: population v3, 15 PHP cases, frozen as the one-time final check.
development or analysis work reads its cases, a test fails when anything in the tree names one
of its repositories, and it runs once, after a prediction is committed. - No development or analysis work reads its cases.
- A test fails when anything in the tree names one of its repositories.
- It runs once, after a prediction is committed.
## 2. Our approach to detection, step by step ## 2. Our approach to detection, step by step
The path changed because hand edits did not transfer. Every gain before 2026-09-30 was a hand The path changed because hand edits did not transfer. Every gain before 2026-09-30 was a hand
edit after a miss: a rule, a taint vocabulary entry, a guard, a prompt line, a tie-break. Each was edit after a miss: a rule, a taint vocabulary entry, a guard, a prompt line, a tie-break. Each was
tuned on the cases that exposed it, and the tuning measured on population v1 did not carry to v2 tuned on the cases that exposed it. The tuning measured on population v1 did not carry to v2
(1 of 11, then 0 of 17). The conclusions drawn: hand edits do not transfer. The data that could (1 of 11, then 0 of 17).
teach a better decision cannot be produced by hand at the scale needed, so it has to be harvested
and labelled by machinery with a boundary around it. Framework knowledge (which function is a sink in which
framework) is useful as a prior but must stay optional, or the engine learns the vocabulary
instead of the code. A reader might object that a classifier is slower to show gains than the
next rule. It is. The difference is that its gains are measured where rules were not: on
repositories it never saw.
The chain, one job per step. The design is on [Memory and detection](memory-and-detection.md); The conclusions drawn:
this is the order of operations.
1. Instruments produce signals, never verdicts. Quick rules, the Joern engine, model sink - Hand edits do not transfer.
classification, the plane's verify passes a and b with the tie-break c, repository facts - The data that could teach a better decision cannot be produced by hand at the scale needed. It
(callers, spans) and model source/sink/sanitizer roles each write one block of a candidate's has to be harvested and labelled by machinery with a boundary around it.
record. None decides alone. - Framework knowledge (which function is a sink in which framework) is useful as a prior. But it
2. Feature records are allow-listed, and missing is not zero. A candidate is `(path, function, must stay optional. Otherwise the engine learns the vocabulary instead of the code.
family)` of one repository and pin. An instrument that did not run, had no coverage or failed
records that state; "silent" and "did not look" stay distinguishable. Input profile v1 withholds A reader might object that a classifier is slower to show gains than the next rule. It is. The
the engine block and the function length, which a leak audit found separate the two sides of a difference is where its gains are measured: on repositories it never saw. Rules were not measured
pair by construction (`benchmarks/measurements/2026-10-01-decision-engine-leak-audit/`), and there.
since 2026-10-02 the plane's `verify.flag_b` and `ms.operation`, flagged the same way by the
audit of the plane profile (`2026-10-02-decision-engine-harvest-advisory/record.json`, The chain has one job per step. The design is on [Memory and detection](memory-and-detection.md).
This list is the order of operations.
1. **Instruments produce signals, never verdicts.** Each instrument writes one block of a
candidate's record. None decides alone. The instruments are:
- quick rules;
- the Joern engine;
- model sink classification;
- the plane's verify passes a and b, with the tie-break c;
- repository facts (callers, spans);
- model source/sink/sanitizer roles.
2. **Feature records are allow-listed, and missing is not zero.** A candidate is `(path, function,
family)` of one repository and pin.
- An instrument that did not run, had no coverage or failed records that state. "Silent" and
"did not look" stay distinguishable.
- Input profile v1 withholds the engine block and the function length. A leak audit found that
they separate the two sides of a pair by construction
(`benchmarks/measurements/2026-10-01-decision-engine-leak-audit/`).
- Since 2026-10-02 it also withholds the plane's `verify.flag_b` and `ms.operation`. The audit
of the plane profile flagged them the same way
(`2026-10-02-decision-engine-harvest-advisory/record.json`,
`src/openultrasast/learn/schema.py`). `src/openultrasast/learn/schema.py`).
3. Labels come from ground truth only. A fail-closed allow-list (`src/openultrasast/learn/sources.toml`) 3. **Labels come from ground truth only.** A fail-closed allow-list
names the sources: the spent populations' declared sites and fix ranges, the vulnerable and (`src/openultrasast/learn/sources.toml`) names the sources:
fixed sides of pairs, including the 97 real public fixes in `benchmarks/pairs/advisory-fixes/`, - the spent populations' declared sites and fix ranges;
benign controls and recorded adjudications. Assumed-benign commits are their own source, - the vulnerable and fixed sides of pairs, including the 97 real public fixes in
weighted 0.5 and reported separately. A plane verdict is a feature, never a label: "rejected by `benchmarks/pairs/advisory-fixes/`;
the plane" is not a negative, or the engine would learn to reproduce the plane's mistakes. - benign controls;
4. Retrieval finds similar labelled cases. A Gower distance over the signal profile, re-ranked - recorded adjudications.
by cosine similarity of code embeddings (OpenAI `text-embedding-3-small` through OpenRouter,
cached by excerpt hash). The evaluation boundary is the only door into a prompt: never the Assumed-benign commits are their own source. They are weighted 0.5 and reported separately. A
candidate's own repository group, never a group the fold evaluates. plane verdict is a feature, never a label. "Rejected by the plane" is not a negative.
5. A classifier compiled in house, DSPy-style. `retrieve -> classify(k)`: an instruction and Otherwise the engine would learn to reproduce the plane's mistakes.
compiled demonstrations (a fixed prefix, so the provider's cache hits), the retrieved examples, 4. **Retrieval finds similar labelled cases.** It uses a Gower distance (a distance for mixed
then the candidate's excerpt and signals. The model answers vulnerable, not vulnerable or numeric and categorical data) over the signal profile. Results are re-ranked by cosine
unsure with a rationale that cites a line; `k` samples give a score. One program is compiled similarity of code embeddings. The embeddings are OpenAI `text-embedding-3-small` through
per profile and family on a compile split that is never evaluated. OpenRouter, cached by excerpt hash. The evaluation boundary is the only door into a prompt. It
6. Calibration on held-out repositories. Platt scaling of the score, cross-fitted by never admits the candidate's own repository group, or a group the fold evaluates.
repository group so no map is checked on the fold that fitted it. Calibration holds when the 5. **A classifier compiled in house, DSPy-style.** The program is `retrieve -> classify(k)`. The
slope interval contains 1, the intercept interval contains 0 and ECE <= 0.05. prompt holds, in order:
7. Two operating points. BLOCK only where calibration holds and a fold reaches the required - an instruction and compiled demonstrations (a fixed prefix, so the provider's cache hits);
precision; otherwise ADVISORY. An unsure majority can never BLOCK. - the retrieved examples;
8. A/B experiments are the only way a change is adopted. Prompts, models, pass counts, - the candidate's excerpt and signals.
features and sampling are compared under a pre-registered manifest with frozen units, paired
arms, a repository bootstrap and two looks. exp-002 is the first to finish: a retrieval The model answers vulnerable, not vulnerable or unsure, with a rationale that cites a line. `k`
ensemble (five samples over disjoint example sets at temperature 0) against today's samples give a score. One program is compiled per profile and family, on a compile split that
temperature sampling. It raised re-run agreement a little (0.906 against 0.875, interval is never evaluated.
spanning zero) and lowered the within-pair AUC from 0.815 to 0.741 at 2.4 times the cost per 6. **Calibration on held-out repositories.** Platt scaling (a logistic fit) maps the score to a
candidate ($0.0051 against $0.0021). Decision: REJECTED probability. It is cross-fitted by repository group, so no map is checked on the fold that
(`benchmarks/experiments/exp-002-retrieval-ensemble/result.json`). fitted it. Calibration holds when three conditions are met: the slope interval contains 1, the
9. Extrapolation with intervals. Every number carries a 95% repository-cluster bootstrap intercept interval contains 0 and ECE <= 0.05.
interval; the pooled and the within-pair readings are both reported, and the stricter one is 7. **Two operating points.** BLOCK applies only where calibration holds and a fold reaches the
required precision. Otherwise the point is ADVISORY. An unsure majority can never BLOCK.
8. **A/B experiments are the only way a change is adopted.** Prompts, models, pass counts,
features and sampling are compared under a pre-registered manifest. The manifest fixes frozen
units, paired arms, a repository bootstrap and two looks. exp-002 is the first to finish:
- It compared a retrieval ensemble (five samples over disjoint example sets at temperature 0)
with today's temperature sampling.
- It raised re-run agreement a little (0.906 against 0.875, interval spanning zero).
- It lowered the within-pair AUC from 0.815 to 0.741 and cost 2.4 times as much per candidate
($0.0051 against $0.0021).
- Decision: REJECTED (`benchmarks/experiments/exp-002-retrieval-ensemble/result.json`).
9. **Extrapolation with intervals.** Every number carries a 95% repository-cluster bootstrap
interval. Both the pooled and the within-pair readings are reported. The stricter one is
named. named.
10. The one-time v3 check. When a program is adopted and a prediction is committed, population 10. **The one-time v3 check.** It happens when a program is adopted and a prediction is
v3 is scored once under `benchmarks/independent/protocol-v3.md`. committed. Then population v3 is scored once under `benchmarks/independent/protocol-v3.md`.
## 3. AI integration: ax on the server ## 3. AI integration: ax on the server
All model work runs as Tasks on google/ax over Agent Substrate, in a single-node kind cluster on All model work runs as Tasks on google/ax over Agent Substrate. They run in a single-node kind
the maintainer's host, each Task in its own gVisor sandbox. There is no local subprocess path. cluster on the maintainer's host. Each Task runs in its own gVisor sandbox. There is no local
The reconciler (`ousast plane run`) is a host process: it applies a Run's Workspaces and Tasks, subprocess path.
resumes each Task, sends its start signal, collects its artifacts and records state. Flow and
contract: [The plane on ax](plane.md); bring-up and the host's lessons: [ax on this host](ops/ax/README.md). The reconciler (`ousast plane run`) is a host process. It does five things:
1. applies a Run's Workspaces and Tasks;
2. resumes each Task;
3. sends its start signal;
4. collects its artifacts;
5. records state.
Flow and contract: [The plane on ax](plane.md). Bring-up and the host's lessons:
[ax on this host](ops/ax/README.md).
What each Run guarantees: What each Run guarantees:
- One Model and one budget per Task. A Task binds at most one ax Model (`deepseek-flash` for - **One Model and one budget per Task.** A Task binds at most one ax Model and
chat, `openrouter-embedding` for embeddings) and `budget: {usd, calls}`. The metered client `budget: {usd, calls}`. The Models are `deepseek-flash` for chat and `openrouter-embedding` for
refuses the next call at the ceiling; the task ends unfinished and resumes on a rerun. Model-free embeddings. The metered client refuses the next call at the ceiling. The task ends unfinished
tasks run with a zero budget, so a stray call fails loudly. and resumes on a rerun. Model-free tasks run with a zero budget, so a stray call fails loudly.
- Credentials exist only in the start request. Agent Substrate boots each template once as a - **Credentials exist only in the start request.** Agent Substrate boots each template once as a
golden actor, so the runner never starts by itself. It waits for `POST /ousast/v1/start`, which golden actor. So the runner never starts by itself. It waits for `POST /ousast/v1/start`. The
the reconciler sends through `atenet-router` to that actor alone. The key is read from the reconciler sends that request through `atenet-router` to that actor alone.
operator's environment or `.env`, kept in memory by the runner, passed to the command and - The key is read from the operator's environment or `.env`.
redacted from echoed stderr; no manifest, rendered Task or file carries it. - The runner keeps it in memory and passes it to the command.
- Per-task egress. One EgressPolicy per actor through the `atenet-egress` gateway, deny by - It is redacted from echoed stderr.
default, hostnames only: plain HTTP to the artifact receiver, TLS passthrough to the Workspaces' - No manifest, rendered Task or file carries it.
Git hosts and the Model's declared hosts. Deleted with the actor. - **Per-task egress.** Each actor gets one EgressPolicy through the `atenet-egress` gateway. It is
- Artifacts come back through the gateway. The runner posts its output directory as a tar to deny by default and uses hostnames only. It allows plain HTTP to the artifact receiver. It allows
the receiver, an HTTP server inside the reconciler process on the host, reached through a Service TLS passthrough to the Workspaces' Git hosts and the Model's declared hosts. It is deleted with
with an EndpointSlice to the host. That delivery is the task's completion; ax has no Completed the actor.
phase. - **Artifacts come back through the gateway.** The runner posts its output directory as a tar to
- Memory on S3-compatible RustFS. `OUSAST_MEMORY=s3://sast-memory` on a RustFS server. The the receiver. The receiver is an HTTP server inside the reconciler process on the host. It is
store verifies the bucket at every start (versioning, lifecycle, tags, S3 Select) and refuses to reached through a Service with an EndpointSlice to the host. That delivery is the task's
start when anything is missing; there is no local or fetch-and-filter fallback. Each row kind completion; ax has no Completed phase.
has a fixed set of queryable fields, enforced at write time - **Memory on S3-compatible RustFS.** The setting is `OUSAST_MEMORY=s3://sast-memory` on a RustFS
server.
- The store verifies the bucket at every start: versioning, lifecycle, tags, S3 Select.
- It refuses to start when anything is missing. There is no local or fetch-and-filter fallback.
- Each row kind has a fixed set of queryable fields, enforced at write time
([Memory](memory.md#verify-at-startup-no-fallback), [RustFS setup](rustfs.md)). ([Memory](memory.md#verify-at-startup-no-fallback), [RustFS setup](rustfs.md)).
Lessons the host taught, each from a failed live run (the full list is on Lessons the host taught, each from a failed live run (the full list is on
[ax on this host](ops/ax/README.md#what-this-host-taught-each-one-cost-a-failed-live-run)): [ax on this host](ops/ax/README.md#what-this-host-taught-each-one-cost-a-failed-live-run)):
- Substrate installs no WorkerPool, and actor images must be pinned by digest. - Substrate installs no WorkerPool, and actor images must be pinned by digest.
- A release `ax` CLI skews from the server; build it from the deployed checkout. - A release `ax` CLI skews from the server. Build it from the deployed checkout.
- ax has no Completed phase, so completion is our artifact delivery. - ax has no Completed phase, so completion is our artifact delivery.
- The golden boot means the runner must wait for a start request. - The golden boot means the runner must wait for a start request.
- The egress gateway dials the address the actor connected to, so the receiver is a Service - The egress gateway dials the address the actor connected to. So the receiver is a Service,
dialled by ClusterIP with its name as Host. dialled by ClusterIP with its name as Host.
- ax's snapshot bucket must exist in your S3 server. When it did not, deleted golden actors piled - ax's snapshot bucket must exist in your S3 server. When it did not, deleted golden actors piled
up in `DELETING` and filled the disk during a 600-task harvest on 2026-10-01. up in `DELETING`. They filled the disk during a 600-task harvest on 2026-10-01.
Deployment status, in the two words [Deployment](deployment.md) uses strictly. Implemented: Deployment status uses two words, as [Deployment](deployment.md) does, strictly.
ax as the only executor, per-task EgressPolicy, credentials only in the start request, per-task
budgets and attribution, the S3 store verified at startup, local bring-up, doctor and smoke Run, Implemented:
re-pinning generated Tasks to another registry. Planned, not done: deployment to any cluster
other than this host's kind cluster, a configurable kube context (`kind-` is hard-coded in - ax as the only executor;
`doctor.py`, `egress.py`, `router.py`), the receiver as an in-cluster Deployment, the - per-task EgressPolicy;
reconciler in-cluster with Secrets injected, presigned-URL delivery to the store. - credentials only in the start request;
- per-task budgets and attribution;
- the S3 store verified at startup;
- local bring-up, doctor and smoke Run;
- re-pinning generated Tasks to another registry.
Planned, not done:
- deployment to any cluster other than this host's kind cluster;
- a configurable kube context (`kind-` is hard-coded in `doctor.py`, `egress.py`, `router.py`);
- the receiver as an in-cluster Deployment;
- the reconciler in-cluster with Secrets injected;
- presigned-URL delivery to the store.
## 4. Token economics, before and after ## 4. Token economics, before and after
Before the plane, the script pipeline cost $0.022 per candidate for two verify passes Before the plane, two costs stood out:
(`benchmarks/independent/verifier-batched-check-2026-09-29.json`), and the maintainers' own
coding sessions spent about 90% of their tokens on tool input and output rather than reasoning. - The script pipeline cost $0.022 per candidate for two verify passes
After it, the per-candidate costs below are what the records show. The mechanisms (one budget (`benchmarks/independent/verifier-batched-check-2026-09-29.json`).
per Task, pay-once reuse, cheap-before-expensive ordering, replay at $0) are on - The maintainers' own coding sessions spent about 90% of their tokens on tool input and output
[Token ergonomics](token-ergonomics.md); this table is the before and after. rather than reasoning.
After it, the per-candidate costs below are what the records show. The mechanisms are on
[Token ergonomics](token-ergonomics.md): one budget per Task, pay-once reuse,
cheap-before-expensive ordering and replay at $0. This table is the before and after.
| Item | Before | After | Record | | Item | Before | After | Record |
| --- | --- | --- | --- | | --- | --- | --- | --- |
@@ -195,63 +273,70 @@ per Task, pay-once reuse, cheap-before-expensive ordering, replay at $0) are on
| Harvest of plane signals for the engine | none existed | `verify` $2.5373 over 1,961 calls and 474 tasks; `roles` $1.4036 over 1,034 calls and 151 tasks | `2026-10-01-decision-engine-harvest/record.json` | | Harvest of plane signals for the engine | none existed | `verify` $2.5373 over 1,961 calls and 474 tasks; `roles` $1.4036 over 1,034 calls and 151 tasks | `2026-10-01-decision-engine-harvest/record.json` |
| Development transcript, tool I/O share | 90% | 85.5% over 2,701 tool calls (`dev_token_report`) | `plane-increment-1.json` | | Development transcript, tool I/O share | 90% | 85.5% over 2,701 tool calls (`dev_token_report`) | `plane-increment-1.json` |
Two notes from the records. The meter prices every call at the Model's list price Two notes from the records:
(`plane/models/deepseek-flash.yaml`); the DeepSeek account moved less, about one third of the
metered amount on the injection slice ($0.30 for $0.91 metered), and budgets and gates are set 1. The meter prices every call at the Model's list price (`plane/models/deepseek-flash.yaml`). The
against the meter, never against the balance. And the development share barely moved: 90 to 85.5 DeepSeek account moved less: about one third of the metered amount on the injection slice
is a small gain, and the remedies (a read guard, a bulk reader, scripts in files) target file ($0.30 for $0.91 metered). Budgets and gates are set against the meter, never against the
contents because that is where the tokens go. balance.
2. The development share barely moved: 90 to 85.5 is a small gain. The remedies are a read guard,
a bulk reader and scripts in files. They target file contents, because that is where the
tokens go.
## 5. Open items ## 5. Open items
Five groups, each item with its record: the engine's own gates, data, measurement debt, There are five groups. Each item has its record. The groups are the engine's own gates, data,
infrastructure, and instrument gaps. measurement debt, infrastructure, and instrument gaps.
The engine's gates. The engine's gates.
- BLOCK is reachable for no family: calibration fails for five (ECE 0.071 to 0.119) and the - BLOCK is reachable for no family. Calibration fails for five (ECE 0.071 to 0.119). The sixth
sixth reaches no fold with a Wilson lower bound of 0.95 on precision (slice-2 record). reaches no fold with a Wilson lower bound of 0.95 on precision (slice-2 record).
- Re-run agreement is below 0.9 for three families: injection 0.77, access control 0.83, - Re-run agreement is below 0.9 for three families: injection 0.77, access control 0.83,
untrusted destination 0.87. exp-002 tried the obvious variance fix and lost on the ranking. untrusted destination 0.87. exp-002 tried the obvious variance fix and lost on the ranking.
Data. Data.
- Positives-only groups: 98 (family, group) pairs in the 904-example memory hold positive examples - Positives-only groups: 98 (family, group) pairs in the 904-example memory hold positive examples
and no negative. 94 of them exist because the fixed side no longer has the function (the vibe-py and no negative.
pairs). No advisory-fixes group is among them - 94 of them exist because the fixed side no longer has the function (the vibe-py pairs).
(`benchmarks/measurements/2026-10-02-decision-engine-negatives-gap/record.json`). There are no - No advisory-fixes group is among them
single-label negatives, so every family's pooled AUC is inflated by positives that never had to (`benchmarks/measurements/2026-10-02-decision-engine-negatives-gap/record.json`).
be told apart from their own fix (slice-2 record, `paired_groups_check`). - There are no single-label negatives. So every family's pooled AUC is inflated by positives
- The model has seen public fixes. A verify pass named the advisory's CVE id for a public fix, an that never had to be told apart from their own fix (slice-2 record, `paired_groups_check`).
observation the engine's design specification (maintainer tooling) records as visible - The model has seen public fixes. A verify pass named the advisory's CVE id for a public fix.
memorisation. Excerpts now redact advisory ids and security-worded comments The engine's design specification (maintainer tooling) records this as visible memorisation.
(`src/openultrasast/learn/excerpt.py`), but labels drawn from public advisories are partly in the Excerpts now redact advisory ids and security-worded comments
model's training data, and the within-pair numbers carry that. (`src/openultrasast/learn/excerpt.py`). But labels drawn from public advisories are partly in
- PHP is covered by the engine on real code and by in-sample quick rules; the population that the model's training data, and the within-pair numbers carry that.
- PHP is covered by the engine on real code and by in-sample quick rules. The population that
would measure it (v3) is reserved. would measure it (v3) is reserved.
Measurement debt. Measurement debt.
- exp-001 (known callers) is registered and has not run; the model-free candidate generation on - exp-001 (known callers) is registered and has not run. Its units need model-free candidate
the 41 development repositories that its units need is not built generation on the 41 development repositories. That generation is not built
(`benchmarks/experiments/exp-001-known-callers/registration.json`). (`benchmarks/experiments/exp-001-known-callers/registration.json`).
- Leave-one-source-out, leave-one-framework-out and the memory-size curve points (0, 25, 50%) - Leave-one-source-out, leave-one-framework-out and the memory-size curve points (0, 25, 50%)
have not run; one paired point exists (724 against 904 examples, injection, AUC 0.76 against 0.77). have not run. One paired point exists (724 against 904 examples, injection, AUC 0.76 against 0.77).
- The plane-profile leak audit flagged `verify.flag_b` and `ms.operation`; they are withheld from - The plane-profile leak audit flagged `verify.flag_b` and `ms.operation`. They are withheld from
profile v1, and whether they track the label on unseen code has to be decided before any profile v1. Before any plane-profile program runs, someone must decide whether they track the
plane-profile program runs (`2026-10-02-decision-engine-harvest-advisory/record.json`). label on unseen code (`2026-10-02-decision-engine-harvest-advisory/record.json`).
- Population v3 waits for `benchmarks/independent/prediction-v3.json`, which does not exist. It is - Population v3 waits for `benchmarks/independent/prediction-v3.json`, which does not exist. It is
written only after a program is adopted. written only after a program is adopted.
Infrastructure. Infrastructure.
- Deployment to a separate Kubernetes cluster is planned, not done; the code changes it needs are - Deployment to a separate Kubernetes cluster is planned, not done. The code changes it needs are
listed with file paths on [Deployment](deployment.md#what-is-implemented-and-what-is-planned). listed with file paths on [Deployment](deployment.md#what-is-implemented-and-what-is-planned).
Instrument gaps. Instrument gaps.
- A missing provider key is a quiet path in a scan. The plane refuses to start a Task whose Model - A missing provider key is a quiet path in a scan.
names an unset variable (`plane/router.py`, `StartError`). A scan without a key asks the model - The plane refuses to start a Task whose Model names an unset variable (`plane/router.py`,
nothing, reports only the graph's entailed findings, and records `learning_endpoint_unavailable` `StartError`).
in `manifest.json` (`cli.py`, after `resolve_chat_endpoint` returns nothing); `report.md` has no - A scan without a key asks the model nothing and reports only the graph's entailed findings.
line for that reason yet, so the report does not say the model layer was skipped. It records `learning_endpoint_unavailable` in `manifest.json` (`cli.py`, after
`resolve_chat_endpoint` returns nothing).
- `report.md` has no line for that reason yet. So the report does not say the model layer was
skipped.