mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
docs: plain-language pass -- README, index, where we stand, scanning, examples, threat model
Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
50f4e69321
commit
552671ca6a
@@ -1,38 +1,52 @@
|
|||||||
# OpenUltraSAST
|
# OpenUltraSAST
|
||||||
|
|
||||||
OpenUltraSAST is an experimental static security analyser meant to work as a safety net on
|
OpenUltraSAST is an experimental static security analyser. It is meant to be a safety net on
|
||||||
repositories it has never seen: it scans a checkout, or the commits a `git push` would publish,
|
repositories it has never seen. It scans a checkout, or the commits a `git push` would publish. It
|
||||||
and separates what it can prove from what it merely suspects. A scan combines language-scoped
|
keeps what it can prove apart from what it only suspects.
|
||||||
pattern rules, a tree-sitter overlay, a Joern code property graph and, optionally, a model that is
|
|
||||||
asked only the question the graph cannot settle; every finding carries the evidence that earned
|
|
||||||
its place. A scan that could not look at something records a degradation instead of reporting a
|
|
||||||
clean result. `ousast` is the interface; model work over whole repositories runs on a separate
|
|
||||||
agentic plane (google/ax), and a learned decision engine is in development. Neither is needed
|
|
||||||
for a scan.
|
|
||||||
|
|
||||||
This file is the front door; each section links to its page of the documentation site (`docs/`,
|
A scan combines up to four parts:
|
||||||
|
|
||||||
|
- language-scoped pattern rules;
|
||||||
|
- a tree-sitter overlay (a syntax tree of each file, built by the tree-sitter parser);
|
||||||
|
- a Joern code property graph (a graph of the program's syntax, control flow and data flow);
|
||||||
|
- optionally, a model. It is asked only the question the graph cannot settle.
|
||||||
|
|
||||||
|
Every finding carries the evidence that earned its place. When a scan could not look at
|
||||||
|
something, it records a degradation. It does not report a clean result.
|
||||||
|
|
||||||
|
`ousast` is the interface. Model work over whole repositories runs on a separate agentic plane
|
||||||
|
(google/ax). A learned decision engine is in development. A scan needs neither of them.
|
||||||
|
|
||||||
|
This file is the front door. Each section links to its page of the documentation site (`docs/`,
|
||||||
`mkdocs.yml`; `uv sync --extra docs && uv run mkdocs serve`).
|
`mkdocs.yml`; `uv sync --extra docs && uv run mkdocs serve`).
|
||||||
|
|
||||||
## Status
|
## Status
|
||||||
|
|
||||||
**v2.0.0, state as of 2026-10-02.** The pre-push safety net is **NO-GO** against every M4 gate:
|
**v2.0.0, state as of 2026-10-02.** The pre-push safety net is **NO-GO** against every M4 gate.
|
||||||
no hook capability is enabled, and the two independent populations spent so far gave engine recall
|
|
||||||
1 of 11 and then 0 of 17. The learned decision engine ranks but may not block: BLOCK is offered for
|
- No hook capability is enabled.
|
||||||
no family. The project has stopped trying to reach the gates by hand; every detection change now
|
- Two independent populations (case sets frozen before any scan) have been spent so far. Engine
|
||||||
has to win a controlled measurement on held-out repositories. The one page that states all of this
|
recall was 1 of 11 and then 0 of 17.
|
||||||
with the record behind each figure is [docs/where-we-stand.md](docs/where-we-stand.md); the
|
- The learned decision engine ranks but may not block. BLOCK is offered for no family.
|
||||||
release history is in [RELEASE_NOTES.md](RELEASE_NOTES.md).
|
|
||||||
|
The project has stopped trying to reach the gates by hand. Every detection change now has to win
|
||||||
|
a controlled measurement on held-out repositories. One page states all of this, with the record
|
||||||
|
behind each figure: [docs/where-we-stand.md](docs/where-we-stand.md). The release history is in
|
||||||
|
[RELEASE_NOTES.md](RELEASE_NOTES.md).
|
||||||
|
|
||||||
## Install and first scan
|
## Install and first scan
|
||||||
|
|
||||||
Which tier you need: a deterministic scan of a checkout or a CI gate is tier 1; the Joern engine
|
Pick the tier you need:
|
||||||
(`standard`, `deep`) is tier 2; running model work over whole repositories on the plane is tier 3,
|
|
||||||
for maintainers; tier 4 does not exist yet.
|
1. Tier 1: a deterministic scan of a checkout, or a CI gate.
|
||||||
|
2. Tier 2: the Joern engine (`standard`, `deep`).
|
||||||
|
3. Tier 3, for maintainers: model work over whole repositories on the plane.
|
||||||
|
4. Tier 4 does not exist yet.
|
||||||
|
|
||||||
### Tier 1: quick scan, no Docker
|
### Tier 1: quick scan, no Docker
|
||||||
|
|
||||||
The core install depends only on PyYAML. From a checkout, `uv sync` (or `pip install .`) installs
|
The core install depends only on PyYAML. From a checkout, `uv sync` (or `pip install .`) installs
|
||||||
the `ousast` entry point; `uv sync --group dev` adds the test and lint tools.
|
the `ousast` entry point. `uv sync --group dev` adds the test and lint tools.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
uv sync
|
uv sync
|
||||||
@@ -41,19 +55,26 @@ uv run ousast scan . --mode quick --fail-on verified # CI: exit 1 on any evide
|
|||||||
```
|
```
|
||||||
|
|
||||||
This runs the language-scoped pattern rules, entry-point reachability hints, ranking,
|
This runs the language-scoped pattern rules, entry-point reachability hints, ranking,
|
||||||
verification and scoring: deterministic, reproducible, no network, no keys. What it cannot do:
|
verification and scoring. It is deterministic and reproducible. It needs no network and no keys.
|
||||||
there is no engine, so nothing follows a value across statements or files, and the PHP quick rules
|
|
||||||
are a low-precision, in-sample development set (per-rule precision lower bounds from 0.03 to 1.0 on
|
What it cannot do:
|
||||||
the real pins, `benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`). The optional
|
|
||||||
`semantic` extra (`uv sync --extra semantic`) installs the tree-sitter grammars for the overlay.
|
- There is no engine. Nothing follows a value across statements or files.
|
||||||
|
- The PHP quick rules are a low-precision, in-sample development set. Per-rule precision lower
|
||||||
|
bounds run from 0.03 to 1.0 on the real pins
|
||||||
|
(`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`).
|
||||||
|
|
||||||
|
The optional `semantic` extra (`uv sync --extra semantic`) installs the tree-sitter grammars for
|
||||||
|
the overlay.
|
||||||
|
|
||||||
### Tier 2: standard and deep scans (Docker)
|
### Tier 2: standard and deep scans (Docker)
|
||||||
|
|
||||||
`standard` and `deep` need Joern. The Docker image `openultrasast:dev` ships the tool, Joern
|
`standard` and `deep` need Joern. The Docker image `openultrasast:dev` ships three things in one
|
||||||
v4.0.625 and `php-cli` (needed by the PHP frontend) in one image; it is built locally from the
|
image: the tool, Joern v4.0.625 and `php-cli` (the PHP frontend needs it). You build it locally
|
||||||
`Dockerfile` (no published image yet), and it is big: a JRE plus about 2 GB of Joern. The shell
|
from the `Dockerfile`; there is no published image yet. It is big: a JRE plus about 2 GB of Joern.
|
||||||
wrapper builds it on first use and then runs `ousast` in the container with your directory mounted
|
|
||||||
read-only and the container network off:
|
The shell wrapper builds the image on first use. Then it runs `ousast` in the container. Your
|
||||||
|
directory is mounted read-only and the container network is off.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
source /path/to/OpenUltraSAST/ops/shell/ousast.sh # PowerShell: ops/shell/ousast.ps1
|
source /path/to/OpenUltraSAST/ops/shell/ousast.sh # PowerShell: ops/shell/ousast.ps1
|
||||||
@@ -62,90 +83,129 @@ ousast-with-judge scan . --mode standard # network on, the model is a
|
|||||||
```
|
```
|
||||||
|
|
||||||
`docker compose build` and `TARGET=/path/to/repo docker compose run --rm ousast scan /target` do
|
`docker compose build` and `TARGET=/path/to/repo docker compose run --rm ousast scan /target` do
|
||||||
the same without the wrapper. `deep` adds REGRESS, which runs each reproduction snippet with
|
the same without the wrapper.
|
||||||
`docker run` from wherever the scan runs; the engine container has no Docker socket, so run `deep`
|
|
||||||
from a host install with Joern on the PATH (`pyinfra @local ops/joern.py`, pinned and checksum-
|
`deep` adds REGRESS. REGRESS runs each reproduction snippet with `docker run` from wherever the
|
||||||
verified) and a working `docker`, or accept the recorded `sandbox_unavailable` degradation.
|
scan runs. The engine container has no Docker socket. So run `deep` in one of two ways:
|
||||||
|
|
||||||
|
- from a host install with Joern on the PATH (`pyinfra @local ops/joern.py`, pinned and checksum-
|
||||||
|
verified) and a working `docker`;
|
||||||
|
- or accept the recorded `sandbox_unavailable` degradation.
|
||||||
|
|
||||||
Details, including the host install, are in [ops/README.md](ops/README.md).
|
Details, including the host install, are in [ops/README.md](ops/README.md).
|
||||||
|
|
||||||
Keys and store settings live in `.env` in the working directory (gitignored; `.env.example` lists
|
Keys and store settings live in `.env` in the working directory. The file is gitignored;
|
||||||
the names). `DEEPSEEK_API_KEY` serves every LLM call; `OPENROUTER_API_KEY` is used for embeddings
|
`.env.example` lists the names.
|
||||||
only (`ousast learn memory embed`). `.env` never overrides a variable already exported in your
|
|
||||||
shell, so `unset` a stale export when a key change seems to have no effect. The plane's memory
|
- `DEEPSEEK_API_KEY` serves every LLM call.
|
||||||
store defaults to a local file store under `~/ousast-results/plane/memory`; an S3-compatible bucket
|
- `OPENROUTER_API_KEY` is used for embeddings only (`ousast learn memory embed`).
|
||||||
is optional (`uv sync --extra s3`, then `OUSAST_MEMORY=s3://<bucket>` with `S3_ENDPOINT` and the
|
- `.env` never overrides a variable already exported in your shell. If a key change seems to have
|
||||||
AWS credentials in `.env`). Nothing in a scan needs it.
|
no effect, `unset` the stale export.
|
||||||
|
- The plane's memory store defaults to a local file store under `~/ousast-results/plane/memory`.
|
||||||
|
- An S3-compatible bucket is optional: `uv sync --extra s3`, then `OUSAST_MEMORY=s3://<bucket>`
|
||||||
|
with `S3_ENDPOINT` and the AWS credentials in `.env`. Nothing in a scan needs it.
|
||||||
|
|
||||||
### Tier 3: the plane on ax (maintainers)
|
### Tier 3: the plane on ax (maintainers)
|
||||||
|
|
||||||
None of this is needed for tier 1 or 2. Today's profile is a single-node kind cluster on the
|
Tier 1 and 2 need none of this. Today's profile is a single-node kind cluster on the maintainer's
|
||||||
maintainer's host. `ops/ax/up.sh` needs `docker`, `go`, `kubectl`, `kind`, `ko` and `ax` on the
|
host.
|
||||||
PATH (it refuses to start when one is missing) and builds Agent Substrate and ax from their
|
|
||||||
checkouts under `~/.cache/ousast/ax-src/`. Measured idle footprint with two workers: the kind node
|
- `ops/ax/up.sh` needs `docker`, `go`, `kubectl`, `kind`, `ko` and `ax` on the PATH. It refuses to
|
||||||
takes 1.7 GiB of memory (27% of the 7.7 GiB host) and 4.9 GB of disk for images, with several GB
|
start when one is missing.
|
||||||
more of Go caches during the build; the worker pool is sized for that 7 GB host. The verify and
|
- It builds Agent Substrate and ax from their checkouts under `~/.cache/ousast/ax-src/`.
|
||||||
roles tasks need `DEEPSEEK_API_KEY`, and memory lives in the local file store or the S3 store
|
- Measured idle footprint with two workers: the kind node takes 1.7 GiB of memory (27% of the
|
||||||
(RustFS is the tested server). Bring-up, doctor and the smoke Run: [ops/ax/README.md](ops/ax/README.md).
|
7.7 GiB host) and 4.9 GB of disk for images. The build uses several GB more for Go caches.
|
||||||
|
- The worker pool is sized for that 7 GB host.
|
||||||
|
- The verify and roles tasks need `DEEPSEEK_API_KEY`.
|
||||||
|
- Memory lives in the local file store or the S3 store. RustFS is the tested server.
|
||||||
|
|
||||||
|
Bring-up, doctor and the smoke Run: [ops/ax/README.md](ops/ax/README.md).
|
||||||
|
|
||||||
### Tier 4: Kubernetes (k3s), planned
|
### Tier 4: Kubernetes (k3s), planned
|
||||||
|
|
||||||
No deployment to any cluster other than that kind cluster has been made. The target is two
|
Nothing has been deployed to any cluster other than that kind cluster. The target is two profiles
|
||||||
profiles of one code base, `kind` locally and `k3s` in production, where the same `ousast plane run`
|
of one code base: `kind` locally and `k3s` in production. The same `ousast plane run` would
|
||||||
submits a Run in a remote execution mode: it needs a kubeconfig context or the cluster's ingress
|
submit a Run in a remote execution mode. It needs:
|
||||||
(and an `imagePullSecret` on the worker ServiceAccount for a private package), runner and engine
|
|
||||||
images published to GHCR and pinned by digest, and the S3 store as the artifact path, with no
|
- a kubeconfig context or the cluster's ingress (and an `imagePullSecret` on the worker
|
||||||
receiver on a laptop. The plan, and the code changes it needs with file paths, are in
|
ServiceAccount for a private package);
|
||||||
|
- runner and engine images published to GHCR and pinned by digest;
|
||||||
|
- the S3 store as the artifact path, with no receiver on a laptop.
|
||||||
|
|
||||||
|
The plan, and the code changes it needs with file paths, are in
|
||||||
[docs/deployment.md](docs/deployment.md#production-topology-ax-on-a-kubernetes-cluster).
|
[docs/deployment.md](docs/deployment.md#production-topology-ax-on-a-kubernetes-cluster).
|
||||||
|
|
||||||
## Scan modes
|
## Scan modes
|
||||||
|
|
||||||
`quick` runs the pattern rules, hints, ranking, verification and score and needs nothing.
|
There are three modes. Each adds to the one before.
|
||||||
`standard` adds the MAP stage: complexity map, semantic overlay, authorization obligations and the
|
|
||||||
Joern model layer, which builds one code property graph per repository and decides taint,
|
- `quick` runs the pattern rules, hints, ranking, verification and score. It needs nothing.
|
||||||
guard-dominance and configuration questions on it; a model is asked only the residual question
|
- `standard` adds the MAP stage: complexity map, semantic overlay, authorization obligations and
|
||||||
(without a key it is skipped and only the graph's entailed findings are reported). `deep` adds
|
the Joern model layer. The model layer builds one code property graph per repository. It decides
|
||||||
REGRESS: promoted candidates are loaded by a small snippet in a Docker sandbox and get a
|
taint, guard-dominance and configuration questions on that graph. A model is asked only the
|
||||||
`triggerable` or `not_triggerable` verdict. A missing engine is a recorded degradation
|
residual question. Without a key, that question is skipped and only the graph's entailed
|
||||||
(`cpg_unavailable`, `sandbox_unavailable`), never a clean result.
|
findings are reported.
|
||||||
|
- `deep` adds REGRESS. Promoted candidates are loaded by a small snippet in a Docker sandbox. Each
|
||||||
|
gets a `triggerable` or `not_triggerable` verdict.
|
||||||
|
|
||||||
|
A missing engine is a recorded degradation (`cpg_unavailable`, `sandbox_unavailable`), never a
|
||||||
|
clean result.
|
||||||
|
|
||||||
|
`--fail-on never|findings|verified|worth-fixing` sets the exit code. `--config openultrasast.toml`
|
||||||
|
loads settings. Every run writes these files under `<target>/.openultrasast/runs/<scan-id>/`:
|
||||||
|
`manifest.json`, `findings.json`, `verification.json`, `score.json`, `report.md`, `report.sarif`
|
||||||
|
and `trace/events.jsonl`.
|
||||||
|
|
||||||
|
`ousast pre-push` (experimental) analyses the commits a push would publish against their base.
|
||||||
|
Only new or worsened defects become candidates. It is advisory by default. Its capability registry
|
||||||
|
is empty, so it emits no normal alert today.
|
||||||
|
|
||||||
`--fail-on never|findings|verified|worth-fixing` sets the exit code and `--config openultrasast.toml`
|
|
||||||
loads settings. Every run writes `manifest.json`, `findings.json`, `verification.json`,
|
|
||||||
`score.json`, `report.md`, `report.sarif` and `trace/events.jsonl` under
|
|
||||||
`<target>/.openultrasast/runs/<scan-id>/`. `ousast pre-push` (experimental) analyses the commits a
|
|
||||||
push would publish against their base so that only new or worsened defects are candidates; it is
|
|
||||||
advisory by default and its capability registry is empty, so no normal alert is emitted today.
|
|
||||||
Flows and the per-language rule table: [docs/scanning.md](docs/scanning.md).
|
Flows and the per-language rule table: [docs/scanning.md](docs/scanning.md).
|
||||||
|
|
||||||
## How detection works
|
## How detection works
|
||||||
|
|
||||||
Three representations, from cheap to expensive: text (regex quick rules, one statement at a time),
|
The tool uses three representations of code, from cheap to expensive:
|
||||||
tree (the tree-sitter CST and an IR per function, for the overlay and the obligations) and graph
|
|
||||||
(the Joern CPG, the only one that follows a value across statements, calls and files). Taint runs
|
|
||||||
as fact tables, queries and verdicts on the graph; absence bugs (an operation that should have
|
|
||||||
been guarded) are found as obligations, dominance and configuration census; a model answers only
|
|
||||||
the residual question. [docs/detection-techniques.md](docs/detection-techniques.md) explains each
|
|
||||||
technique with the code it comes from and what it misses;
|
|
||||||
[docs/architecture.md](docs/architecture.md) gives the stage order, the evidence ladder (a state
|
|
||||||
machine, not a label a model may assign), the central CWE policy and the project score, and the
|
|
||||||
self-improving loops (`ousast improve`, with `--memory` for proposals from plane rows).
|
|
||||||
|
|
||||||
Quick rules cover C/C++, Python, JavaScript/TypeScript, Java with Groovy templates, and PHP, and
|
1. Text: regex quick rules, one statement at a time.
|
||||||
run only against files of their language. The detection gate (`uv run python -m openultrasast.gate`)
|
2. Tree: the tree-sitter CST (concrete syntax tree) and an IR (intermediate representation) per
|
||||||
enforces at least 90% recall and under 10% false positives per language on the bundled cheat-sheet
|
function. The overlay and the obligations use it.
|
||||||
corpora; on this tree it reports 93.62% recall (44/47) and 0.0% false positives. Those corpora show
|
3. Graph: the Joern CPG. It is the only one that follows a value across statements, calls and
|
||||||
that a rule change did not break a known detection; they are not a recall estimate on real code,
|
files.
|
||||||
and PHP is not in the gate. Framework knowledge (WordPress, Flask, Django, Express, Spring) is
|
|
||||||
tagged as an optional prior in `src/openultrasast/ruleset/frameworks.toml`, so its contribution can
|
Taint runs as fact tables, queries and verdicts on the graph. Absence bugs are operations that
|
||||||
be measured and switched off.
|
should have been guarded. They are found as obligations, dominance and configuration census. A
|
||||||
|
model answers only the residual question.
|
||||||
|
|
||||||
|
- [docs/detection-techniques.md](docs/detection-techniques.md) explains each technique, with the
|
||||||
|
code it comes from and what it misses.
|
||||||
|
- [docs/architecture.md](docs/architecture.md) gives the stage order and the evidence ladder (a
|
||||||
|
state machine, not a label a model may assign). It also covers the central CWE policy, the
|
||||||
|
project score and the self-improving loops (`ousast improve`, with `--memory` for proposals from
|
||||||
|
plane rows).
|
||||||
|
|
||||||
|
Quick rules cover C/C++, Python, JavaScript/TypeScript, Java with Groovy templates, and PHP. Each
|
||||||
|
rule runs only against files of its language.
|
||||||
|
|
||||||
|
The detection gate (`uv run python -m openultrasast.gate`) enforces at least 90% recall and under
|
||||||
|
10% false positives per language on the bundled cheat-sheet corpora. On this tree it reports
|
||||||
|
93.62% recall (44/47) and 0.0% false positives. Those corpora show that a rule change did not
|
||||||
|
break a known detection. They are not a recall estimate on real code, and PHP is not in the gate.
|
||||||
|
|
||||||
|
Framework knowledge (WordPress, Flask, Django, Express, Spring) is tagged as an optional prior in
|
||||||
|
`src/openultrasast/ruleset/frameworks.toml`. So its contribution can be measured and switched off.
|
||||||
|
|
||||||
## The plane on ax
|
## The plane on ax
|
||||||
|
|
||||||
Model-driven work over whole repositories runs as a Run manifest, a DAG of ax Tasks, each in its
|
Model-driven work over whole repositories runs as a Run manifest. A Run is a DAG of ax Tasks. Each
|
||||||
own gVisor actor on Agent Substrate. ax is the only agentic executor; there is no local
|
Task runs in its own gVisor actor (an isolated sandbox) on Agent Substrate. ax is the only agentic
|
||||||
subprocess path. Each Task binds at most one Model and its own `usd`/`calls` budget, egress is
|
executor; there is no local subprocess path.
|
||||||
deny-by-default per Task (hostnames only), and the provider key travels only in the Task's start
|
|
||||||
request, never in a manifest or an image. Measured on the first increment: the plane agreed on 16
|
- Each Task binds at most one Model and its own `usd`/`calls` budget.
|
||||||
of 20 declared sites at $0.0205 per candidate.
|
- Egress is deny-by-default per Task, by hostname only.
|
||||||
|
- The provider key travels only in the Task's start request, never in a manifest or an image.
|
||||||
|
|
||||||
|
Measured on the first increment: the plane agreed on 16 of 20 declared sites at $0.0205 per
|
||||||
|
candidate.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image
|
uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image
|
||||||
@@ -155,42 +215,61 @@ uv run ousast plane remember validation-46 # ingest the run's rows
|
|||||||
```
|
```
|
||||||
|
|
||||||
`workspaces`, `harvest`, `alerts-engine` and `memory-normalise` generate Runs, their inputs and
|
`workspaces`, `harvest`, `alerts-engine` and `memory-normalise` generate Runs, their inputs and
|
||||||
the store's rewrites. The flow and the task catalogue: [docs/plane.md](docs/plane.md); what runs
|
the store's rewrites. The flow and the task catalogue: [docs/plane.md](docs/plane.md). What runs
|
||||||
where today and what a separate cluster would take: [docs/deployment.md](docs/deployment.md).
|
where today, and what a separate cluster would take: [docs/deployment.md](docs/deployment.md).
|
||||||
|
|
||||||
HarnessX, the earlier agentic extra, was retired 2026-09-30: a `[harnessx]` section is ignored
|
HarnessX, the earlier agentic extra, was retired 2026-09-30. A `[harnessx]` section is ignored
|
||||||
with one warning, and `[models] verifier`, `[fusion] panel_model` and `[fusion] decider_model` fail
|
with one warning. `[models] verifier`, `[fusion] panel_model` and `[fusion] decider_model` fail and
|
||||||
naming the plane replacement, as do config keys nothing ever read. Fusion is deterministic.
|
name the plane replacement. Config keys that nothing ever read fail the same way. Fusion is
|
||||||
|
deterministic.
|
||||||
|
|
||||||
## Memory
|
## Memory
|
||||||
|
|
||||||
One store, keyed by repository and pin, holds what plane runs and the decision engine learned:
|
One store holds what plane runs and the decision engine learned. It is keyed by repository and pin.
|
||||||
rows (facts, verdicts, unit costs, alerts, coverage, feature records) and content-addressed blobs
|
It holds two kinds of data:
|
||||||
(excerpts, embeddings, cached responses, compiled programs). `OUSAST_MEMORY` selects the backend:
|
|
||||||
a local file store, or `s3://<bucket>` on any S3-compatible server with versioning, lifecycle
|
- rows: facts, verdicts, unit costs, alerts, coverage and feature records;
|
||||||
rules, object tags and S3 Select; RustFS is the tested server. An admin sets the bucket up once;
|
- content-addressed blobs: excerpts, embeddings, cached responses and compiled programs.
|
||||||
the store verifies it at every start and refuses to run, naming each missing piece, rather than
|
|
||||||
fall back. Layout and the read and write paths: [docs/memory.md](docs/memory.md); the bucket
|
`OUSAST_MEMORY` selects the backend. It is either a local file store or `s3://<bucket>` on any
|
||||||
policy and admin steps: [docs/rustfs.md](docs/rustfs.md); how stored rows are meant to turn into
|
S3-compatible server. The S3 server must support versioning, lifecycle rules, object tags and S3
|
||||||
better decisions: [docs/memory-and-detection.md](docs/memory-and-detection.md).
|
Select. RustFS is the tested server.
|
||||||
|
|
||||||
|
An admin sets the bucket up once. The store verifies it at every start. If anything is missing, it
|
||||||
|
refuses to run and names each missing piece. It does not fall back.
|
||||||
|
|
||||||
|
- Layout and the read and write paths: [docs/memory.md](docs/memory.md).
|
||||||
|
- The bucket policy and admin steps: [docs/rustfs.md](docs/rustfs.md).
|
||||||
|
- How stored rows are meant to turn into better decisions:
|
||||||
|
[docs/memory-and-detection.md](docs/memory-and-detection.md).
|
||||||
|
|
||||||
## Decision engine and evaluation
|
## Decision engine and evaluation
|
||||||
|
|
||||||
A learned decision engine is being built to replace hand-tuned detection edits: every instrument
|
A learned decision engine is being built. It is meant to replace hand-tuned detection edits. It
|
||||||
produces signals, never verdicts; labels come from ground truth only; a compiled AI classifier with
|
follows three rules:
|
||||||
local memory turns retrieved similar cases and the candidate's signals into a calibrated
|
|
||||||
probability with BLOCK and ADVISORY operating points. It is **in development, not adopted**: no
|
|
||||||
scan, pre-push check or report uses it. Measured on six families, on repositories the compiled
|
|
||||||
program was not shown, the pooled AUC of its score runs from 0.77 (injection) to 0.96 (access
|
|
||||||
control); the stricter within-pair AUC is 0.72 to 0.93; BLOCK is offered for no family
|
|
||||||
(`benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`). Maintainer surface:
|
|
||||||
`ousast learn labels|memory|compile|evaluate|curve|experiment|audit-leaks`. Design, slices and
|
|
||||||
the A/B protocol: [docs/decision-engine.md](docs/decision-engine.md).
|
|
||||||
|
|
||||||
How anything is measured on code the tool was not tuned on, the pair corpora, the independent
|
- Every instrument produces signals, never verdicts.
|
||||||
populations and the M4 gates, and the fold discipline that keeps learned decisions out of their
|
- Labels come from ground truth only.
|
||||||
own training data: [docs/evaluation.md](docs/evaluation.md). Population v3 (PHP, 15 cases) is
|
- A compiled AI classifier with local memory turns retrieved similar cases and the candidate's
|
||||||
frozen as the one-time final check and nothing in the tree may read it.
|
signals into a calibrated probability. It has BLOCK and ADVISORY operating points.
|
||||||
|
|
||||||
|
It is **in development, not adopted**. No scan, pre-push check or report uses it.
|
||||||
|
|
||||||
|
It was measured on six families, on repositories the compiled program was not shown:
|
||||||
|
|
||||||
|
- The pooled AUC (area under the ROC curve) of its score runs from 0.77 (injection) to 0.96
|
||||||
|
(access control).
|
||||||
|
- The stricter within-pair AUC is 0.72 to 0.93 across the same families.
|
||||||
|
- BLOCK is offered for no family
|
||||||
|
(`benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`).
|
||||||
|
|
||||||
|
Maintainer surface: `ousast learn labels|memory|compile|evaluate|curve|experiment|audit-leaks`.
|
||||||
|
Design, slices and the A/B protocol: [docs/decision-engine.md](docs/decision-engine.md).
|
||||||
|
|
||||||
|
[docs/evaluation.md](docs/evaluation.md) explains how anything is measured on code the tool was not
|
||||||
|
tuned on. It covers the pair corpora, the independent populations and the M4 gates. It also covers
|
||||||
|
the fold discipline that keeps learned decisions out of their own training data. Population v3
|
||||||
|
(PHP, 15 cases) is frozen as the one-time final check. Nothing in the tree may read it.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
uv run ousast benchmark benchmarks/manifests/python-vulnerable.toml --mode quick
|
uv run ousast benchmark benchmarks/manifests/python-vulnerable.toml --mode quick
|
||||||
@@ -201,19 +280,26 @@ uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dr
|
|||||||
|
|
||||||
## Token economics
|
## Token economics
|
||||||
|
|
||||||
Model tokens are the one running cost. At runtime the plane routes them: one Model and one budget
|
Model tokens are the one running cost. At runtime the plane routes them:
|
||||||
per Task, a metered client that refuses the next call at the ceiling, repository facts computed
|
|
||||||
model-free and reused, blobs replayed at $0, triage before the hunt and a tie-break only on
|
- one Model and one budget per Task;
|
||||||
disputed candidates. In development, 85.5% of the maintainers' coding-session tokens were tool
|
- a metered client that refuses the next call at the ceiling;
|
||||||
input and output rather than reasoning (down from 90%), and the remedies in `benchmarks/dev/`
|
- repository facts computed without a model and reused;
|
||||||
target that. Mechanisms,
|
- blobs replayed at $0, with no new model call;
|
||||||
the measured costs and the before-and-after table: [docs/token-ergonomics.md](docs/token-ergonomics.md).
|
- triage before the hunt;
|
||||||
|
- a tie-break only on disputed candidates.
|
||||||
|
|
||||||
|
In development, 85.5% of the maintainers' coding-session tokens were tool input and output rather
|
||||||
|
than reasoning. That is down from 90%. The remedies in `benchmarks/dev/` target that share.
|
||||||
|
Mechanisms, the measured costs and the before-and-after table:
|
||||||
|
[docs/token-ergonomics.md](docs/token-ergonomics.md).
|
||||||
|
|
||||||
## Examples, threat model
|
## Examples, threat model
|
||||||
|
|
||||||
End-to-end walkthroughs, from a quick scan to an MCP client: [docs/examples.md](docs/examples.md).
|
- End-to-end walkthroughs, from a quick scan to an MCP client: [docs/examples.md](docs/examples.md).
|
||||||
What the tool trusts and what it does not (scanned code is untrusted input, secret redaction,
|
- What the tool trusts and what it does not: [docs/threat-model.md](docs/threat-model.md). This
|
||||||
per-task egress, budgets, the `deep` sandbox): [docs/threat-model.md](docs/threat-model.md).
|
covers scanned code as untrusted input, secret redaction, per-task egress, budgets and the
|
||||||
|
`deep` sandbox.
|
||||||
|
|
||||||
## Contributing and maintainer tooling
|
## Contributing and maintainer tooling
|
||||||
|
|
||||||
@@ -228,25 +314,33 @@ uv run python dagger/ci.py # containerized CI pipeline
|
|||||||
```
|
```
|
||||||
|
|
||||||
An agent that can run shell commands needs nothing but the CLI. For [OpenCode](https://opencode.ai)
|
An agent that can run shell commands needs nothing but the CLI. For [OpenCode](https://opencode.ai)
|
||||||
there are two project skills in `.agents/skills/`: `openultrasast-scan` (modes and their
|
there are two project skills in `.agents/skills/`:
|
||||||
prerequisites, `--fail-on`, the evidence ladder, recorded degradations, run artifacts) and
|
|
||||||
`openultrasast-triage` (false-positive reasons, what an adjudication may feed, fixing and
|
- `openultrasast-scan`: modes and their prerequisites, `--fail-on`, the evidence ladder, recorded
|
||||||
re-scanning). `ousast mcp` runs a narrow MCP server over stdio with ten tools (`openultrasast.scan`,
|
degradations and run artifacts.
|
||||||
`status`, `findings`, `get_finding`, `evidence`, `artifacts`, `benchmark`, `explain`,
|
- `openultrasast-triage`: false-positive reasons, what an adjudication may feed, and fixing and
|
||||||
`propose_patch`, `export_report`); no tool runs a shell, Docker or a free-form command:
|
re-scanning.
|
||||||
|
|
||||||
|
`ousast mcp` runs a narrow MCP (Model Context Protocol) server over stdio. It has ten tools:
|
||||||
|
`openultrasast.scan`, `status`, `findings`, `get_finding`, `evidence`, `artifacts`, `benchmark`,
|
||||||
|
`explain`, `propose_patch` and `export_report`. No tool runs a shell, Docker or a free-form
|
||||||
|
command.
|
||||||
|
|
||||||
```jsonc
|
```jsonc
|
||||||
{ "command": "uv", "args": ["run", "ousast", "mcp"] }
|
{ "command": "uv", "args": ["run", "ousast", "mcp"] }
|
||||||
```
|
```
|
||||||
|
|
||||||
The `kiro-*` skills, `AGENTS.md` and the Kiro-style specifications under `.kiro/` are the
|
The `kiro-*` skills, `AGENTS.md` and the Kiro-style specifications under `.kiro/` are the
|
||||||
maintainer's development tooling and can be ignored by users. Other maintainer commands:
|
maintainer's development tooling. Users can ignore them. Other maintainer commands:
|
||||||
`ousast repos` (pinned known-vulnerable checkouts) and `ousast model candidates` (what the
|
|
||||||
candidate enumerator can reach).
|
- `ousast repos`: pinned known-vulnerable checkouts.
|
||||||
|
- `ousast model candidates`: what the candidate enumerator can reach.
|
||||||
|
|
||||||
## Licence
|
## Licence
|
||||||
|
|
||||||
OpenUltraSAST is licensed under the [Apache License 2.0](LICENSE). Third-party material keeps its
|
OpenUltraSAST is licensed under the [Apache License 2.0](LICENSE). Third-party material keeps its
|
||||||
own terms: Joern (Apache-2.0) is downloaded at image build time, the tree-sitter wheels of the
|
own terms:
|
||||||
`semantic` extra are MIT, and the published benchmark cases were licence-checked one by one (v1's
|
|
||||||
OpenCVE case is BUSL-1.1, see [RELEASE_NOTES.md](RELEASE_NOTES.md)).
|
- Joern (Apache-2.0) is downloaded at image build time.
|
||||||
|
- The tree-sitter wheels of the `semantic` extra are MIT.
|
||||||
|
- The published benchmark cases were licence-checked one by one. v1's OpenCVE case is BUSL-1.1, see [RELEASE_NOTES.md](RELEASE_NOTES.md).
|
||||||
|
|||||||
+28
-17
@@ -1,7 +1,7 @@
|
|||||||
# End-to-end examples
|
# End-to-end examples
|
||||||
|
|
||||||
Each example is self-contained. Artifacts land under
|
Each example stands on its own. Artifacts land under `<target>/.openultrasast/runs/<scan-id>/`.
|
||||||
`<target>/.openultrasast/runs/<scan-id>/` and `.openultrasast/` is gitignored.
|
`.openultrasast/` is gitignored.
|
||||||
|
|
||||||
## 1. Quick scan a local repo (deterministic, no keys)
|
## 1. Quick scan a local repo (deterministic, no keys)
|
||||||
|
|
||||||
@@ -21,7 +21,7 @@ ousast scan . --mode standard # Joern model layer, container network off:
|
|||||||
ousast-with-judge scan . --mode standard # network on, so the model is asked (needs DEEPSEEK_API_KEY)
|
ousast-with-judge scan . --mode standard # network on, so the model is asked (needs DEEPSEEK_API_KEY)
|
||||||
```
|
```
|
||||||
|
|
||||||
Without a key the graph still decides: its entailed findings are reported, and the LLM's residual
|
Without a key, the graph still decides. Its entailed findings are reported. The LLM's residual
|
||||||
and suspicion questions go unasked.
|
and suspicion questions go unasked.
|
||||||
|
|
||||||
## 3. CI gate
|
## 3. CI gate
|
||||||
@@ -52,10 +52,11 @@ uv run ousast pairs --split holdout --profile human # filter by declared split
|
|||||||
uv run python -m openultrasast.pair_gate # CI: local pairs must all pass
|
uv run python -m openultrasast.pair_gate # CI: local pairs must all pass
|
||||||
```
|
```
|
||||||
|
|
||||||
Each case is two isolated trees (vuln file vs patched file, same relative path).
|
Each case is two isolated trees: the vuln file and the patched file, at the same relative path.
|
||||||
`pair_correct` means the harness fired on the vuln side and stayed silent on the
|
|
||||||
fix. Misses and fix-side leaks become `miss`/`fp` signals for `ousast improve`.
|
- `pair_correct` means the harness fired on the vuln side and stayed silent on the fix.
|
||||||
The catalog and public dataset pointers live in `benchmarks/pairs/`.
|
- Misses become `miss` signals and fix-side leaks become `fp` signals for `ousast improve`.
|
||||||
|
- The catalog and public dataset pointers live in `benchmarks/pairs/`.
|
||||||
|
|
||||||
## 6. Self-improve the ruleset from benchmark feedback
|
## 6. Self-improve the ruleset from benchmark feedback
|
||||||
|
|
||||||
@@ -68,10 +69,11 @@ uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml
|
|||||||
|
|
||||||
## 7. Running the agentic plane
|
## 7. Running the agentic plane
|
||||||
|
|
||||||
Model work over whole repositories runs as a Run manifest on ax, one isolated actor
|
Model work over whole repositories runs as a Run manifest on ax. Each task gets one isolated
|
||||||
per task (bring-up: [ops/ax/README.md](ops/ax/README.md)). The provider key is
|
actor. Bring-up is in [ops/ax/README.md](ops/ax/README.md).
|
||||||
read from the environment (or `.env`, which never overrides an exported variable) by
|
|
||||||
`ousast` and sent only in each task's start request.
|
`ousast` reads the provider key from the environment, or from `.env`. `.env` never overrides an
|
||||||
|
exported variable. The key is sent only in each task's start request.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image
|
uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image
|
||||||
@@ -83,16 +85,25 @@ uv run ousast plane remember validation-46 # re-ingest the run's m
|
|||||||
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run --memory
|
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run --memory
|
||||||
```
|
```
|
||||||
|
|
||||||
Each task in the Run carries its own Model and `budget: {usd, calls}`; a task that
|
Each task in the Run carries its own Model and `budget: {usd, calls}`.
|
||||||
reaches its ceiling stops as `unfinished` and resumes on a rerun with a larger
|
|
||||||
budget. `ousast plane status` prints the token attribution table: calls, prompt,
|
- A task that reaches its ceiling stops as `unfinished`. It resumes on a rerun with a larger
|
||||||
cache-hit and output tokens and USD per task.
|
budget.
|
||||||
|
- `ousast plane status` prints the token attribution table. It shows calls, prompt, cache-hit and
|
||||||
|
output tokens, and USD per task.
|
||||||
|
|
||||||
## 8. Drive it from an MCP client (OpenCode / IDE)
|
## 8. Drive it from an MCP client (OpenCode / IDE)
|
||||||
|
|
||||||
|
MCP (Model Context Protocol) lets an editor or agent call the tool's functions.
|
||||||
|
|
||||||
```jsonc
|
```jsonc
|
||||||
{ "command": "uv", "args": ["run", "ousast", "mcp"] }
|
{ "command": "uv", "args": ["run", "ousast", "mcp"] }
|
||||||
```
|
```
|
||||||
|
|
||||||
Then: `openultrasast.scan {path}` → `run_dir` → `openultrasast.findings {run_dir}` →
|
Then call the tools in this order:
|
||||||
`openultrasast.explain {run_dir, finding_id}`. See the README section "Agent integrations".
|
|
||||||
|
1. `openultrasast.scan {path}` returns a `run_dir`.
|
||||||
|
2. `openultrasast.findings {run_dir}` lists the findings.
|
||||||
|
3. `openultrasast.explain {run_dir, finding_id}` explains one.
|
||||||
|
|
||||||
|
See the README section "Agent integrations".
|
||||||
|
|||||||
+24
-18
@@ -1,13 +1,13 @@
|
|||||||
# OpenUltraSAST
|
# OpenUltraSAST
|
||||||
|
|
||||||
OpenUltraSAST is an experimental static security analyser meant to work as a safety net on
|
OpenUltraSAST is an experimental static security analyser. It is meant to be a safety net on
|
||||||
repositories it has never seen: it scans a checkout, or the commits a `git push` would publish,
|
repositories it has never seen. It scans a checkout, or the commits a `git push` would publish. It
|
||||||
and separates what it can prove from what it merely suspects. A scan that could not look at
|
keeps what it can prove apart from what it only suspects. When a scan could not look at
|
||||||
something records a degradation instead of reporting a clean result.
|
something, it records a degradation. It does not report a clean result.
|
||||||
|
|
||||||
**State as of 2026-10-02 (v2.0.0):** the pre-push safety net is **NO-GO** for rollout.
|
**State as of 2026-10-02 (v2.0.0):** the pre-push safety net is **NO-GO** for rollout.
|
||||||
No hook capability is enabled, and no independent population has passed the qualification gates
|
No hook capability is enabled. No independent population (a case set frozen before any scan) has
|
||||||
(see [Evaluation](evaluation.md)).
|
passed the qualification gates (see [Evaluation](evaluation.md)).
|
||||||
|
|
||||||
## Three layers
|
## Three layers
|
||||||
|
|
||||||
@@ -17,6 +17,9 @@ No hook capability is enabled, and no independent population has passed the qual
|
|||||||
| **The agentic plane on ax** | Model-driven work over whole repositories, run as a DAG of tasks on google/ax, one isolated actor per task. What the runs learn is kept in a memory store. Maintainer surface: `ousast plane`. | [The plane on ax](plane.md), [Memory](memory.md) |
|
| **The agentic plane on ax** | Model-driven work over whole repositories, run as a DAG of tasks on google/ax, one isolated actor per task. What the runs learn is kept in a memory store. Maintainer surface: `ousast plane`. | [The plane on ax](plane.md), [Memory](memory.md) |
|
||||||
| **The decision engine** | A learned classifier that weighs every instrument's signals against similar labelled cases from memory. **In development, not adopted**: no scan, pre-push check or report uses it. Maintainer surface: `ousast learn`. | [Memory and detection](memory-and-detection.md), [Decision engine](decision-engine.md) |
|
| **The decision engine** | A learned classifier that weighs every instrument's signals against similar labelled cases from memory. **In development, not adopted**: no scan, pre-push check or report uses it. Maintainer surface: `ousast learn`. | [Memory and detection](memory-and-detection.md), [Decision engine](decision-engine.md) |
|
||||||
|
|
||||||
|
A code property graph is a graph of the program's syntax, control flow and data flow. Joern builds
|
||||||
|
it.
|
||||||
|
|
||||||
## How they connect
|
## How they connect
|
||||||
|
|
||||||
```mermaid
|
```mermaid
|
||||||
@@ -33,31 +36,34 @@ flowchart LR
|
|||||||
engine -. "intended, not adopted: BLOCK / ADVISORY" .-> report
|
engine -. "intended, not adopted: BLOCK / ADVISORY" .-> report
|
||||||
```
|
```
|
||||||
|
|
||||||
- The **scanners** produce findings and, inside plane Runs, the `alerts` that become memory rows.
|
- The **scanners** produce findings. Inside plane Runs, they also produce the `alerts` that become
|
||||||
- The **plane** runs model work (verify passes, roles) and model-free bookkeeping (facts, agreement,
|
memory rows.
|
||||||
feature records) per case; its `remember` task turns a case's artifacts into rows of the
|
- The **plane** runs model work per case: verify passes and roles. It also runs model-free
|
||||||
[memory store](memory.md).
|
bookkeeping per case: facts, agreement and feature records. Its `remember` task turns a case's
|
||||||
- Memory feeds two consumers. `ousast improve --memory` turns stored rows into rule-status
|
artifacts into rows of the [memory store](memory.md).
|
||||||
proposals that pass through the same validator and gate as every other edit. The decision engine
|
- Memory feeds two consumers:
|
||||||
retrieves labelled examples from the same store.
|
1. `ousast improve --memory` turns stored rows into rule-status proposals. They pass through the
|
||||||
|
same validator and gate as every other edit.
|
||||||
|
2. The decision engine retrieves labelled examples from the same store.
|
||||||
- The decision engine's output (dotted line) is the intended path. Nothing user-facing reads it
|
- The decision engine's output (dotted line) is the intended path. Nothing user-facing reads it
|
||||||
today.
|
today.
|
||||||
|
|
||||||
## Where to go next
|
## Where to go next
|
||||||
|
|
||||||
- The current state in one page, every figure with its record: capability against the M4 gates, the
|
- The current state in one page, every figure with its record: [Where we stand](where-we-stand.md).
|
||||||
step-by-step approach, ax on the server, token economics before and after, and the open items:
|
It covers capability against the M4 gates, the step-by-step approach, ax on the server, token
|
||||||
[Where we stand](where-we-stand.md).
|
economics before and after, and the open items.
|
||||||
- Run a scan: [Examples](examples.md), and the commands in [Scanning](scanning.md).
|
- Run a scan: [Examples](examples.md), and the commands in [Scanning](scanning.md).
|
||||||
- Understand a finding's evidence and score: [Architecture](architecture.md).
|
- Understand a finding's evidence and score: [Architecture](architecture.md).
|
||||||
- The program analysis under a scan, what each representation can see and what it misses:
|
- The program analysis under a scan, what each representation can see and what it misses:
|
||||||
[Detection techniques](detection-techniques.md).
|
[Detection techniques](detection-techniques.md).
|
||||||
- Operate the plane and its store: [ax on this host](ops/ax/README.md) and
|
- Operate the plane and its store: [ax on this host](ops/ax/README.md) and
|
||||||
[RustFS setup](rustfs.md).
|
[RustFS setup](rustfs.md).
|
||||||
- Where the plane runs today and what a separate Kubernetes cluster would take (planned, not
|
- Where the plane runs today, and what a separate Kubernetes cluster would take (planned, not
|
||||||
done): [Deployment](deployment.md).
|
done): [Deployment](deployment.md).
|
||||||
- Where the model spend goes, per Task at runtime and in the maintainers' own coding sessions,
|
- Where the model spend goes, per Task at runtime and in the maintainers' own coding sessions,
|
||||||
and what is reused instead of paid again: [Token ergonomics](token-ergonomics.md).
|
and what is reused instead of paid again: [Token ergonomics](token-ergonomics.md).
|
||||||
- Trust boundaries and hardening: [Threat model](threat-model.md).
|
- Trust boundaries and hardening: [Threat model](threat-model.md).
|
||||||
|
|
||||||
Every number in these pages is quoted from a committed record whose path is given next to it.
|
Every number in these pages is quoted from a committed record. The record's path is given next to
|
||||||
|
the number.
|
||||||
|
|||||||
+59
-33
@@ -1,13 +1,21 @@
|
|||||||
# Scanning
|
# Scanning
|
||||||
|
|
||||||
`ousast scan PATH --mode quick|standard|deep` scans a checkout; `ousast pre-push` checks only
|
This page shows the scan flows.
|
||||||
what a push (or an explicit `--base`/`--head` pair) changes. The stage-by-stage pipeline,
|
|
||||||
evidence ladder and score are in [Architecture](architecture.md); this page shows the flows.
|
- `ousast scan PATH --mode quick|standard|deep` scans a checkout.
|
||||||
|
- `ousast pre-push` checks only what a push changes, or what an explicit `--base`/`--head` pair
|
||||||
|
changes.
|
||||||
|
|
||||||
|
The stage-by-stage pipeline, the evidence ladder and the score are in
|
||||||
|
[Architecture](architecture.md).
|
||||||
|
|
||||||
## The three scan modes
|
## The three scan modes
|
||||||
|
|
||||||
The stages a mode requests are `MODE_STAGES` in `src/openultrasast/stages.py`: `quick` runs
|
The stages each mode requests are listed in `MODE_STAGES` in `src/openultrasast/stages.py`:
|
||||||
STATIC, `standard` adds MAP, `deep` adds REGRESS.
|
|
||||||
|
- `quick` runs STATIC.
|
||||||
|
- `standard` adds MAP.
|
||||||
|
- `deep` adds REGRESS.
|
||||||
|
|
||||||
```mermaid
|
```mermaid
|
||||||
flowchart TD
|
flowchart TD
|
||||||
@@ -38,7 +46,7 @@ Sources: `README.md` (scan modes), `src/openultrasast/cli.py` (`run_stage` calls
|
|||||||
|
|
||||||
## Quick rules per language
|
## Quick rules per language
|
||||||
|
|
||||||
Rules run only against files of their language. The bundled ruleset lives in
|
Each rule runs only against files of its language. The bundled ruleset lives in
|
||||||
`src/openultrasast/ruleset/<language>/rules.toml`.
|
`src/openultrasast/ruleset/<language>/rules.toml`.
|
||||||
|
|
||||||
| Language | Classes (CWE) |
|
| Language | Classes (CWE) |
|
||||||
@@ -49,21 +57,31 @@ Rules run only against files of their language. The bundled ruleset lives in
|
|||||||
| Java + Groovy templates | command injection (78), SQLi (89), weak hash (327), deserialization (502), unescaped template XSS (79) |
|
| Java + Groovy templates | command injection (78), SQLi (89), weak hash (327), deserialization (502), unescaped template XSS (79) |
|
||||||
| PHP | SQLi (89), command injection (78), file inclusion (98), path (22), open redirect (601), SSRF (918); code eval (95), unserialize (502) and echo XSS (79) ship as `shadow` rules |
|
| PHP | SQLi (89), command injection (78), file inclusion (98), path (22), open redirect (601), SSRF (918); code eval (95), unserialize (502) and echo XSS (79) ship as `shadow` rules |
|
||||||
|
|
||||||
The detection gate (`python -m openultrasast.gate`) enforces at least 90% recall and under 10%
|
The detection gate (`python -m openultrasast.gate`) enforces two limits per language on the
|
||||||
false positives per language on the bundled corpora in `benchmarks/manifests/`. Those corpora are
|
bundled corpora in `benchmarks/manifests/`: at least 90% recall and under 10% false positives.
|
||||||
cheat-sheet fixtures: they catch regressions, they do not estimate recall on real code. **PHP is
|
|
||||||
not in the gate**; its numbers are in-sample development numbers
|
- Those corpora are cheat-sheet fixtures. They catch regressions. They do not estimate recall on
|
||||||
(`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`). See `README.md` for the
|
real code.
|
||||||
gate's per-language output.
|
- **PHP is not in the gate**. Its numbers are in-sample development numbers
|
||||||
|
(`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`).
|
||||||
|
|
||||||
|
See `README.md` for the gate's per-language output.
|
||||||
|
|
||||||
## The Joern engine
|
## The Joern engine
|
||||||
|
|
||||||
The model layer (`src/openultrasast/model/`, `src/openultrasast/cpg/`) is the core of `standard`.
|
The model layer (`src/openultrasast/model/`, `src/openultrasast/cpg/`) is the core of `standard`.
|
||||||
It builds one code property graph per scan with Joern (`joern-parse`, then batched
|
It builds one code property graph per scan with Joern. A code property graph is a graph of the
|
||||||
`joern --script` queries from `cpg/queries/*.sc`) and sends each question to the arbiter of its
|
program's syntax, control flow and data flow. The build runs `joern-parse`, then batched
|
||||||
type: **taint** (source to sink with no discharging sanitizer), **dominance** (does a guard govern
|
`joern --script` queries from `cpg/queries/*.sc`.
|
||||||
the obligated operation) and **configuration** (constant abstraction over a value). Taint facts
|
|
||||||
ship for C, Java, JavaScript, PHP and Python (`src/openultrasast/ruleset/semantic/*.toml`).
|
The layer sends each question to the arbiter (the decider) of its type:
|
||||||
|
|
||||||
|
- **taint**: a flow from source to sink with no discharging sanitizer;
|
||||||
|
- **dominance**: does a guard govern the obligated operation;
|
||||||
|
- **configuration**: constant abstraction over a value.
|
||||||
|
|
||||||
|
Taint facts ship for C, Java, JavaScript, PHP and Python
|
||||||
|
(`src/openultrasast/ruleset/semantic/*.toml`).
|
||||||
|
|
||||||
```mermaid
|
```mermaid
|
||||||
flowchart LR
|
flowchart LR
|
||||||
@@ -79,25 +97,33 @@ flowchart LR
|
|||||||
rung -- "residual question only" --> judge["LLM judge: confirm or contradict, never create a flow"]
|
rung -- "residual question only" --> judge["LLM judge: confirm or contradict, never create a flow"]
|
||||||
```
|
```
|
||||||
|
|
||||||
The Docker image ships Joern and `php-cli` (which the PHP frontend needs); without Joern the
|
The Docker image ships Joern and `php-cli`. The PHP frontend needs `php-cli`. Without Joern, the
|
||||||
layer records `cpg_unavailable` and the scan is otherwise unchanged. Engine installation and the
|
layer records `cpg_unavailable`; the rest of the scan is unchanged.
|
||||||
Joern features used are in [Engine and pre-push](ops/README.md).
|
|
||||||
How the graph is built, queried and refined, with the records behind each refinement, is in
|
- Engine installation and the Joern features used: [Engine and pre-push](ops/README.md).
|
||||||
[Detection techniques](detection-techniques.md).
|
- How the graph is built, queried and refined, with the records behind each refinement:
|
||||||
|
[Detection techniques](detection-techniques.md).
|
||||||
|
|
||||||
## Framework priors
|
## Framework priors
|
||||||
|
|
||||||
Rule and taint-fact entries that know a framework or library carry a `framework` or `library` tag
|
Some rule and taint-fact entries know a framework or library. They carry a `framework` or
|
||||||
naming a row of `src/openultrasast/ruleset/frameworks.toml` (for example `wordpress`, `flask`,
|
`library` tag. The tag names a row of `src/openultrasast/ruleset/frameworks.toml` (for example
|
||||||
`django`, `express`, `spring`). The loaders take `priors=`: `"all"` (the default and today's
|
`wordpress`, `flask`, `django`, `express`, `spring`).
|
||||||
behaviour), `"off"` (language-level knowledge only) or a set of ids, so the contribution of
|
|
||||||
framework knowledge can be measured and switched off. The decision engine's first slice ran with
|
The loaders take `priors=` with one of three values:
|
||||||
priors off ([Decision engine](decision-engine.md#first-measured-slice-2026-10-01)).
|
|
||||||
|
- `"all"`: the default and today's behaviour;
|
||||||
|
- `"off"`: language-level knowledge only;
|
||||||
|
- a set of ids.
|
||||||
|
|
||||||
|
So the contribution of framework knowledge can be measured and switched off. The decision
|
||||||
|
engine's first slice ran with priors off
|
||||||
|
([Decision engine](decision-engine.md#first-measured-slice-2026-10-01)).
|
||||||
|
|
||||||
## Pre-push: the delta check (experimental)
|
## Pre-push: the delta check (experimental)
|
||||||
|
|
||||||
`ousast pre-push` analyses the commits a `git push` would publish, or an explicit
|
`ousast pre-push` analyses the commits a `git push` would publish, or an explicit
|
||||||
`--base`/`--head` pair, and compares head with base so that only new or worsened defects are
|
`--base`/`--head` pair. It compares head with base. Only new or worsened defects become
|
||||||
candidates. Sources: `ousast pre-push --help`, `src/openultrasast/push/`,
|
candidates. Sources: `ousast pre-push --help`, `src/openultrasast/push/`,
|
||||||
[Engine and pre-push](ops/README.md#explicit-local-replay).
|
[Engine and pre-push](ops/README.md#explicit-local-replay).
|
||||||
|
|
||||||
@@ -118,10 +144,10 @@ flowchart TD
|
|||||||
exit -- "blocking, explicit opt-in" --> block["May block the push"]
|
exit -- "blocking, explicit opt-in" --> block["May block the push"]
|
||||||
```
|
```
|
||||||
|
|
||||||
- `--deadline` (30 s by default) bounds preparation, both revisions and reporting together;
|
- `--deadline` (30 s by default) bounds preparation, both revisions and reporting together.
|
||||||
`--incomplete-coverage block` opts into failing when coverage is incomplete.
|
- `--incomplete-coverage block` opts into failing when coverage is incomplete.
|
||||||
- The default capability registry is empty pending independent qualification, so no normal alert
|
- The default capability registry is empty until independent qualification passes. So no normal
|
||||||
is emitted today; every candidate stays diagnostic.
|
alert is emitted today. Every candidate stays diagnostic.
|
||||||
- A model is called only with an explicit `--model-config`.
|
- A model is called only with an explicit `--model-config`.
|
||||||
- Hook installation (`ops/install-pre-push`) and the private cache (`--cache-dir`) are documented
|
- Hook installation (`ops/install-pre-push`) and the private cache (`--cache-dir`) are documented
|
||||||
in [Engine and pre-push](ops/README.md#experimental-pre-push-integration).
|
in [Engine and pre-push](ops/README.md#experimental-pre-push-integration).
|
||||||
|
|||||||
+74
-51
@@ -1,78 +1,101 @@
|
|||||||
# Threat model & hardening
|
# Threat model & hardening
|
||||||
|
|
||||||
OpenUltraSAST analyzes **untrusted code**. This document states what it trusts, what
|
OpenUltraSAST analyzes **untrusted code**. This page states what it trusts and what it does not.
|
||||||
it does not, and the controls that keep a scan safe to run in CI and safe to share.
|
It also lists the controls that keep a scan safe to run in CI and safe to share.
|
||||||
|
|
||||||
## Trust boundaries
|
## Trust boundaries
|
||||||
|
|
||||||
- **Scanned code is untrusted input.** Quick and standard modes never import, build or run
|
- **Scanned code is untrusted input.** Quick and standard modes never import, build or run the
|
||||||
the target: quick reads source as text, and standard additionally parses it into a code
|
target.
|
||||||
property graph with Joern (`joern-parse`, a parser, not a build) and may send code excerpts
|
- Quick mode reads source as text.
|
||||||
to the configured model. Only `deep` mode executes target-derived code: its REGRESS stage
|
- Standard mode also parses it into a code property graph (a graph of the program's syntax,
|
||||||
loads promoted candidates inside the Docker sandbox below, and it runs only when `docker
|
control flow and data flow) with Joern. It uses `joern-parse`, a parser, not a build.
|
||||||
|
Standard mode may send code excerpts to the configured model.
|
||||||
|
- Only `deep` mode executes target-derived code. Its REGRESS stage loads promoted candidates
|
||||||
|
inside the Docker sandbox described below. It runs only when `docker
|
||||||
info` succeeds (otherwise the stage is skipped and recorded as `sandbox_unavailable`).
|
info` succeeds (otherwise the stage is skipped and recorded as `sandbox_unavailable`).
|
||||||
- **The ruleset and CWE policy are trusted, governed data.** They change only through
|
- **The ruleset and CWE policy are trusted, governed data.** They change in only two ways:
|
||||||
reviewed commits or the bounded self-improvement loop (`ousast improve`), which can
|
through reviewed commits, or through the bounded self-improvement loop (`ousast improve`). The
|
||||||
flip rule *status* and tune score constants but can never edit pattern text or the
|
loop can flip rule *status* and tune score constants. It can never edit pattern text or the
|
||||||
authoritative 0–5 severity. See [architecture.md](architecture.md#self-improvement).
|
authoritative 0–5 severity. See [architecture.md](architecture.md#self-improvement).
|
||||||
- **Provider credentials are the operator's.** The core install (PyYAML only) makes no
|
- **Provider credentials are the operator's.** The core install (PyYAML only) makes no network
|
||||||
network calls on its own. A local scan calls a model only in standard or deep mode, and
|
calls on its own.
|
||||||
only when a key is available: the model layer asks the residual question the graph cannot
|
- A local scan calls a model only in standard or deep mode, and only when a key is available.
|
||||||
settle, and `[models] hunter` enables the tool hunter. LLM calls go to DeepSeek
|
The model layer asks the residual question the graph cannot settle. `[models] hunter`
|
||||||
(`DEEPSEEK_API_KEY`); OpenRouter (`OPENROUTER_API_KEY`) serves embeddings. Keys come from
|
enables the tool hunter.
|
||||||
the environment or a gitignored `.env`, which never overrides an exported variable. The
|
- LLM calls go to DeepSeek (`DEEPSEEK_API_KEY`). OpenRouter (`OPENROUTER_API_KEY`) serves
|
||||||
pre-push check calls a model only with an explicit `--model-config`. Agentic work runs on the plane (`ousast plane run`): `ousast` reads
|
embeddings.
|
||||||
the variable a task's Model names (`secretKey.key`) from the operator's environment or
|
- Keys come from the environment or a gitignored `.env`. `.env` never overrides an exported
|
||||||
`.env` and sends the key only in that task's start request, through Agent Substrate's
|
variable.
|
||||||
router to that task's actor. It is not baked into an image or task template, and it is
|
- The pre-push check calls a model only with an explicit `--model-config`.
|
||||||
never logged or written to artifacts.
|
- Agentic work runs on the plane (`ousast plane run`). `ousast` reads the variable a task's
|
||||||
|
Model names (`secretKey.key`) from the operator's environment or `.env`. It sends the key
|
||||||
|
only in that task's start request. The request goes through Agent Substrate's router to that
|
||||||
|
task's actor. The key is not baked into an image or task template. It is never logged or
|
||||||
|
written to artifacts.
|
||||||
|
|
||||||
## Controls
|
## Controls
|
||||||
|
|
||||||
### Secret redaction
|
### Secret redaction
|
||||||
Scan artifacts can quote source that contains live credentials. `redaction.py` masks
|
Scan artifacts can quote source that contains live credentials. `redaction.py` masks
|
||||||
recognizable secret shapes (provider API keys, AWS/GitHub/Slack/Google tokens, bearer
|
recognizable secret shapes before traces (`trace/events.jsonl`) and the markdown report are
|
||||||
tokens, URL-embedded credentials, PEM private keys, and `key = value` assignments for
|
written. It masks:
|
||||||
sensitive names) before traces (`trace/events.jsonl`) and the markdown report are
|
|
||||||
written. On by default; disable with `[hardening] redact_secrets = false`.
|
- provider API keys;
|
||||||
|
- AWS/GitHub/Slack/Google tokens;
|
||||||
|
- bearer tokens;
|
||||||
|
- URL-embedded credentials;
|
||||||
|
- PEM private keys;
|
||||||
|
- `key = value` assignments for sensitive names.
|
||||||
|
|
||||||
|
Redaction is on by default. Turn it off with `[hardening] redact_secrets = false`.
|
||||||
|
|
||||||
### Plane egress
|
### Plane egress
|
||||||
Each plane task runs in its own actor behind Agent Substrate's egress gateway, which
|
Each plane task runs in its own actor behind Agent Substrate's egress gateway. The gateway denies
|
||||||
denies by default. The task's EgressPolicy (`src/openultrasast/plane/egress.py`) allows exactly three
|
by default. The task's EgressPolicy (`src/openultrasast/plane/egress.py`) allows exactly three
|
||||||
things: plain HTTP to the artifact receiver, and TLS passthrough to the Git hosts of the
|
things:
|
||||||
task's bound Workspaces and to the hosts its bound Model declares
|
|
||||||
(`openultrasast.io/egress-hosts`). There are no wildcards, no IP addresses and no TLS
|
1. plain HTTP to the artifact receiver;
|
||||||
interception. The policy is deleted with the actor.
|
2. TLS passthrough to the Git hosts of the task's bound Workspaces;
|
||||||
|
3. TLS passthrough to the hosts its bound Model declares (`openultrasast.io/egress-hosts`).
|
||||||
|
|
||||||
|
There are no wildcards, no IP addresses and no TLS interception. The policy is deleted with the
|
||||||
|
actor.
|
||||||
|
|
||||||
### Cost & CI budgets
|
### Cost & CI budgets
|
||||||
- **Agentic spend** is bounded per plane task by the Run's `budget: {usd, calls}`. The
|
- **Agentic spend** is bounded per plane task by the Run's `budget: {usd, calls}`.
|
||||||
metered client refuses the next call once either ceiling is reached (the task is
|
- The metered client refuses the next call once either ceiling is reached. The task is then
|
||||||
`unfinished` and resumes on a rerun with a larger budget). A usd budget on a Model
|
`unfinished`. It resumes on a rerun with a larger budget.
|
||||||
without prices is refused rather than metered as zero, and an account or
|
- A usd budget on a Model without prices is refused. It is not metered as zero.
|
||||||
authentication error fails the task and starts nothing further. The standard-mode tool
|
- An account or authentication error fails the task and starts nothing further.
|
||||||
hunter is bounded by the per-tier hunter budgets.
|
- The standard-mode tool hunter is bounded by the per-tier hunter budgets.
|
||||||
- **Output size** is bounded by `[hardening] max_findings` (0 = unlimited). Truncation is
|
- **Output size** is bounded by `[hardening] max_findings` (0 = unlimited). Truncation is
|
||||||
severity-ordered and disclosed as a `budget` degradation in the manifest.
|
severity-ordered. It is disclosed as a `budget` degradation in the manifest.
|
||||||
|
|
||||||
### Provider reliability
|
### Provider reliability
|
||||||
LLM/embedding calls retry transient failures (HTTP 429/5xx, connection errors, timeouts)
|
LLM/embedding calls retry transient failures with exponential backoff. Transient failures are
|
||||||
with exponential backoff; non-transient errors (e.g. 4xx, malformed JSON) fail fast.
|
HTTP 429/5xx, connection errors and timeouts. Non-transient errors fail fast, for example 4xx or
|
||||||
|
malformed JSON.
|
||||||
|
|
||||||
### Visible degradation & determinism
|
### Visible degradation & determinism
|
||||||
When an optional engine is missing (Joern: `cpg_unavailable`; Docker: `sandbox_unavailable`),
|
A scan is never silently downgraded. When an optional engine is missing, the scan falls back to
|
||||||
the scan falls back to its deterministic equivalent and records a `degradations` entry in the
|
its deterministic equivalent. It records a `degradations` entry in the manifest (Joern:
|
||||||
manifest — it is never silently downgraded. Without a provider key the model layer asks no
|
`cpg_unavailable`; Docker: `sandbox_unavailable`).
|
||||||
question and reports only what the graph entailed. Runs are reproducible: fixed config,
|
|
||||||
artifact manifests, prompt hashes, and model identifiers are recorded.
|
Without a provider key, the model layer asks no question. It reports only what the graph
|
||||||
|
entailed.
|
||||||
|
|
||||||
|
Runs are reproducible. The fixed config, artifact manifests, prompt hashes and model identifiers
|
||||||
|
are recorded.
|
||||||
|
|
||||||
### MCP surface
|
### MCP surface
|
||||||
`ousast mcp` exposes only the ten narrow project tools. No tool runs arbitrary shell,
|
`ousast mcp` exposes only the ten narrow project tools. No tool runs arbitrary shell, Docker or
|
||||||
Docker, or internal hunter tools, and no tool accepts a free-form command argument.
|
internal hunter tools. No tool accepts a free-form command argument.
|
||||||
|
|
||||||
## Sandbox (deep mode)
|
## Sandbox (deep mode)
|
||||||
|
|
||||||
The REGRESS stage (`regress/`, `sandbox/runner.py`) runs each snippet with `docker run`
|
The REGRESS stage (`regress/`, `sandbox/runner.py`) runs each snippet with `docker run`. It never
|
||||||
and never mounts the host Docker socket:
|
mounts the host Docker socket.
|
||||||
|
|
||||||
| Control | Setting |
|
| Control | Setting |
|
||||||
|---|---|
|
|---|---|
|
||||||
@@ -84,5 +107,5 @@ and never mounts the host Docker socket:
|
|||||||
| pids | `[sandbox] pids_limit`, default 512 |
|
| pids | `[sandbox] pids_limit`, default 512 |
|
||||||
| timeout | `[sandbox] timeout_seconds`, default 300 s |
|
| timeout | `[sandbox] timeout_seconds`, default 300 s |
|
||||||
|
|
||||||
Before anything runs, a structural deny-list (`regress/safety.py`) rejects a snippet that
|
A structural deny-list (`regress/safety.py`) checks each snippet before anything runs. It rejects
|
||||||
mentions sockets, curl, host networking or writes under `/workspace`.
|
a snippet that mentions sockets, curl, host networking or writes under `/workspace`.
|
||||||
|
|||||||
+259
-174
@@ -1,33 +1,50 @@
|
|||||||
# Where we stand
|
# Where we stand
|
||||||
|
|
||||||
State as of 2026-10-02 (v2.0.0). This page answers five questions in one place. How far is
|
State as of 2026-10-02 (v2.0.0). This page answers five questions in one place:
|
||||||
detection from its gates. How does the project now try to close that gap. How does the model work
|
|
||||||
run on ax. What did the tokens cost before and after that move. What is still open. Every figure
|
|
||||||
is quoted from a committed record whose path stands next to it; detail tables stay on the pages
|
|
||||||
linked from each section.
|
|
||||||
|
|
||||||
The goal has not changed: a safety net on repositories the tool has never seen, which says
|
1. How far is detection from its gates?
|
||||||
BLOCK or ADVISORY about the code a push would change and stays silent on fixes and on benign
|
2. How does the project now try to close that gap?
|
||||||
commits. Two independent populations have been spent against that goal, with engine recall 1 of
|
3. How does the model work run on ax?
|
||||||
|
4. What did the tokens cost before and after that move?
|
||||||
|
5. What is still open?
|
||||||
|
|
||||||
|
Every figure is quoted from a committed record. The record's path stands next to the figure.
|
||||||
|
Detail tables stay on the pages linked from each section.
|
||||||
|
|
||||||
|
The goal has not changed. The tool should be a safety net on repositories it has never seen. It
|
||||||
|
should say BLOCK or ADVISORY about the code a push would change. It should stay silent on fixes
|
||||||
|
and on benign commits.
|
||||||
|
|
||||||
|
Two independent populations have been spent against that goal. An independent population is a
|
||||||
|
set of cases frozen before any scan. Engine recall was 1 of
|
||||||
11 and then 0 of 17. The hand edits that closed misses on the first population did not carry to
|
11 and then 0 of 17. The hand edits that closed misses on the first population did not carry to
|
||||||
the second. So the question is no longer which rule to add next. It is how a change to detection
|
the second. So the question is no longer which rule to add next. The question is this: how can a
|
||||||
can be shown to work on code it was not tuned on, at a cost per candidate that can be paid for
|
change to detection be shown to work on code it was not tuned on? And can that be done at a cost
|
||||||
every push.
|
per candidate that can be paid for every push?
|
||||||
|
|
||||||
The answer this page records: the pre-push safety net is NO-GO against every M4 gate, and the
|
The answer this page records:
|
||||||
project has stopped trying to reach the gates by hand. Every detection change now has to win a
|
|
||||||
controlled measurement on held-out repositories, run on an agentic plane whose cost per candidate
|
- The pre-push safety net is NO-GO against every M4 gate.
|
||||||
is known. The pieces of that method exist and have been measured. None of them has produced a
|
- The project has stopped trying to reach the gates by hand.
|
||||||
BLOCK decision yet.
|
- Every detection change now has to win a controlled measurement on held-out repositories. The
|
||||||
|
measurement runs on an agentic plane whose cost per candidate is known.
|
||||||
|
- The pieces of that method exist and have been measured. None of them has produced a BLOCK
|
||||||
|
decision yet.
|
||||||
|
|
||||||
## 1. Detection capability and goals: NO-GO, measured
|
## 1. Detection capability and goals: NO-GO, measured
|
||||||
|
|
||||||
No M4 gate is met on an independent population. The qualification milestone for the pre-push
|
No M4 gate is met on an independent population. M4 is the qualification milestone for the
|
||||||
safety net (M4) requires, together: at least 95% actionable precision, at least 90% supported
|
pre-push safety net. It requires all of these together:
|
||||||
recall, at least 95% supported-check completion, the runtime gates, and zero fixed-side or benign
|
|
||||||
false alerts. A population is frozen before any
|
- at least 95% actionable precision;
|
||||||
case is scanned, scored once under a pre-registered protocol, then spent; it can inform changes
|
- at least 90% supported recall;
|
||||||
but can qualify nothing afterwards ([Evaluation](evaluation.md#independent-populations-and-the-m4-gates)).
|
- at least 95% supported-check completion;
|
||||||
|
- the runtime gates;
|
||||||
|
- zero fixed-side or benign false alerts.
|
||||||
|
|
||||||
|
A population is frozen before any case is scanned. It is scored once under a pre-registered
|
||||||
|
protocol, then spent. After that it can inform changes but can qualify nothing
|
||||||
|
([Evaluation](evaluation.md#independent-populations-and-the-m4-gates)).
|
||||||
|
|
||||||
| Gate | Required | Latest independent result | Record |
|
| Gate | Required | Latest independent result | Record |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
@@ -36,156 +53,217 @@ but can qualify nothing afterwards ([Evaluation](evaluation.md#independent-popul
|
|||||||
| Supported-check completion | >= 95% | 13,206 of 108,374 | `results-v2.json` |
|
| Supported-check completion | >= 95% | 13,206 of 108,374 | `results-v2.json` |
|
||||||
| Fixed-side and benign alerts | zero | 0 fixed-side, which the record calls met trivially since nothing was detected; 3 benign alerts | `results-v2.json` |
|
| Fixed-side and benign alerts | zero | 0 fixed-side, which the record calls met trivially since nothing was detected; 3 benign alerts | `results-v2.json` |
|
||||||
|
|
||||||
Three later measurements narrow the picture without changing the verdict.
|
Three later measurements narrow the picture. They do not change the verdict.
|
||||||
|
|
||||||
- The plane reproduces the script pipeline's agreement at a known cost. On the model-driven
|
- **The plane reproduces the script pipeline's agreement at a known cost.** The test used the
|
||||||
pipeline's validation set (46 candidates, 15 cases of v2, 20 declared vulnerable sites), the
|
model-driven pipeline's validation set: 46 candidates, 15 cases of v2, 20 declared vulnerable
|
||||||
plane's verify passes a and b plus a tie-break on the 10 disputed candidates agreed on 16 of 20
|
sites. The plane ran verify passes a and b, plus a tie-break on the 10 disputed candidates. It
|
||||||
declared sites. The script reference agreed on 16 as well. The cost was $0.0205 per candidate
|
agreed on 16 of 20 declared sites. The script reference agreed on 16 as well. The cost was
|
||||||
(`benchmarks/independent/plane-increment-2.json`). The record carries two caveats. The tie-break
|
$0.0205 per candidate (`benchmarks/independent/plane-increment-2.json`). The record carries two
|
||||||
was chosen after seeing the first measurement on the same set, so this compares plane with
|
caveats:
|
||||||
script and qualifies nothing. And the passes flag fewer declared sites than the reference in
|
1. The tie-break was chosen after seeing the first measurement on the same set. So this compares
|
||||||
any pass, 17 against 19; that count is a correction in `plane-increment-1.json`, whose first
|
plane with script and qualifies nothing.
|
||||||
reading took agree's `declared_sites_matched` (19, coverage by candidates) for "found in either
|
2. The passes flag fewer declared sites than the reference in any pass: 17 against 19 (passes against reference). That
|
||||||
pass".
|
count is a correction in `plane-increment-1.json`. Its first reading took agree's
|
||||||
- The learned decision engine ranks, but may not block. On six families, evaluated on
|
`declared_sites_matched` (19, coverage by candidates) to mean "found in either pass".
|
||||||
repositories the compiled program was not shown, the pooled AUC of its score runs from 0.77
|
- **The learned decision engine ranks, but may not block.** It was evaluated on six families, on
|
||||||
(injection) to 0.96 (access control). The stricter reading is the within-pair AUC, which asks
|
repositories the compiled program was not shown.
|
||||||
whether the vulnerable side of a fix scores above its own fixed side: 0.72 to 0.93. BLOCK is
|
- The pooled AUC (area under the ROC curve) of its score runs from 0.77 (injection) to 0.96
|
||||||
offered for no family; calibration holds only for output encoding, and there no fold reaches
|
(access control).
|
||||||
the required precision (`benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`).
|
- The stricter reading is the within-pair AUC. It asks whether the vulnerable side of a fix
|
||||||
The per-family table is on [Decision engine](decision-engine.md#second-slice-six-families-2026-10-02).
|
scores above its own fixed side: 0.72 to 0.93.
|
||||||
- PHP coverage is in-sample. The PHP quick rules were written and tightened while looking at
|
- BLOCK is offered for no family. Calibration holds only for output encoding. There, no fold
|
||||||
the pins and fixtures they are scored on, so the record says so itself (`provenance.in_sample` in
|
reaches the required precision
|
||||||
`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`): a development check, not
|
(`benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`).
|
||||||
a recall estimate. On real PHP code the Joern engine is the instrument.
|
- The per-family table is on
|
||||||
|
[Decision engine](decision-engine.md#second-slice-six-families-2026-10-02).
|
||||||
|
- **PHP coverage is in-sample.** The PHP quick rules were written and tightened while looking at
|
||||||
|
the pins and fixtures they are scored on. The record says so itself (`provenance.in_sample` in
|
||||||
|
`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`). It is a development
|
||||||
|
check, not a recall estimate. On real PHP code, the Joern engine is the instrument.
|
||||||
|
|
||||||
What has not been touched: population v3, 15 PHP cases, frozen as the one-time final check. No
|
What has not been touched: population v3, 15 PHP cases, frozen as the one-time final check.
|
||||||
development or analysis work reads its cases, a test fails when anything in the tree names one
|
|
||||||
of its repositories, and it runs once, after a prediction is committed.
|
- No development or analysis work reads its cases.
|
||||||
|
- A test fails when anything in the tree names one of its repositories.
|
||||||
|
- It runs once, after a prediction is committed.
|
||||||
|
|
||||||
## 2. Our approach to detection, step by step
|
## 2. Our approach to detection, step by step
|
||||||
|
|
||||||
The path changed because hand edits did not transfer. Every gain before 2026-09-30 was a hand
|
The path changed because hand edits did not transfer. Every gain before 2026-09-30 was a hand
|
||||||
edit after a miss: a rule, a taint vocabulary entry, a guard, a prompt line, a tie-break. Each was
|
edit after a miss: a rule, a taint vocabulary entry, a guard, a prompt line, a tie-break. Each was
|
||||||
tuned on the cases that exposed it, and the tuning measured on population v1 did not carry to v2
|
tuned on the cases that exposed it. The tuning measured on population v1 did not carry to v2
|
||||||
(1 of 11, then 0 of 17). The conclusions drawn: hand edits do not transfer. The data that could
|
(1 of 11, then 0 of 17).
|
||||||
teach a better decision cannot be produced by hand at the scale needed, so it has to be harvested
|
|
||||||
and labelled by machinery with a boundary around it. Framework knowledge (which function is a sink in which
|
|
||||||
framework) is useful as a prior but must stay optional, or the engine learns the vocabulary
|
|
||||||
instead of the code. A reader might object that a classifier is slower to show gains than the
|
|
||||||
next rule. It is. The difference is that its gains are measured where rules were not: on
|
|
||||||
repositories it never saw.
|
|
||||||
|
|
||||||
The chain, one job per step. The design is on [Memory and detection](memory-and-detection.md);
|
The conclusions drawn:
|
||||||
this is the order of operations.
|
|
||||||
|
|
||||||
1. Instruments produce signals, never verdicts. Quick rules, the Joern engine, model sink
|
- Hand edits do not transfer.
|
||||||
classification, the plane's verify passes a and b with the tie-break c, repository facts
|
- The data that could teach a better decision cannot be produced by hand at the scale needed. It
|
||||||
(callers, spans) and model source/sink/sanitizer roles each write one block of a candidate's
|
has to be harvested and labelled by machinery with a boundary around it.
|
||||||
record. None decides alone.
|
- Framework knowledge (which function is a sink in which framework) is useful as a prior. But it
|
||||||
2. Feature records are allow-listed, and missing is not zero. A candidate is `(path, function,
|
must stay optional. Otherwise the engine learns the vocabulary instead of the code.
|
||||||
family)` of one repository and pin. An instrument that did not run, had no coverage or failed
|
|
||||||
records that state; "silent" and "did not look" stay distinguishable. Input profile v1 withholds
|
A reader might object that a classifier is slower to show gains than the next rule. It is. The
|
||||||
the engine block and the function length, which a leak audit found separate the two sides of a
|
difference is where its gains are measured: on repositories it never saw. Rules were not measured
|
||||||
pair by construction (`benchmarks/measurements/2026-10-01-decision-engine-leak-audit/`), and
|
there.
|
||||||
since 2026-10-02 the plane's `verify.flag_b` and `ms.operation`, flagged the same way by the
|
|
||||||
audit of the plane profile (`2026-10-02-decision-engine-harvest-advisory/record.json`,
|
The chain has one job per step. The design is on [Memory and detection](memory-and-detection.md).
|
||||||
|
This list is the order of operations.
|
||||||
|
|
||||||
|
1. **Instruments produce signals, never verdicts.** Each instrument writes one block of a
|
||||||
|
candidate's record. None decides alone. The instruments are:
|
||||||
|
- quick rules;
|
||||||
|
- the Joern engine;
|
||||||
|
- model sink classification;
|
||||||
|
- the plane's verify passes a and b, with the tie-break c;
|
||||||
|
- repository facts (callers, spans);
|
||||||
|
- model source/sink/sanitizer roles.
|
||||||
|
2. **Feature records are allow-listed, and missing is not zero.** A candidate is `(path, function,
|
||||||
|
family)` of one repository and pin.
|
||||||
|
- An instrument that did not run, had no coverage or failed records that state. "Silent" and
|
||||||
|
"did not look" stay distinguishable.
|
||||||
|
- Input profile v1 withholds the engine block and the function length. A leak audit found that
|
||||||
|
they separate the two sides of a pair by construction
|
||||||
|
(`benchmarks/measurements/2026-10-01-decision-engine-leak-audit/`).
|
||||||
|
- Since 2026-10-02 it also withholds the plane's `verify.flag_b` and `ms.operation`. The audit
|
||||||
|
of the plane profile flagged them the same way
|
||||||
|
(`2026-10-02-decision-engine-harvest-advisory/record.json`,
|
||||||
`src/openultrasast/learn/schema.py`).
|
`src/openultrasast/learn/schema.py`).
|
||||||
3. Labels come from ground truth only. A fail-closed allow-list (`src/openultrasast/learn/sources.toml`)
|
3. **Labels come from ground truth only.** A fail-closed allow-list
|
||||||
names the sources: the spent populations' declared sites and fix ranges, the vulnerable and
|
(`src/openultrasast/learn/sources.toml`) names the sources:
|
||||||
fixed sides of pairs, including the 97 real public fixes in `benchmarks/pairs/advisory-fixes/`,
|
- the spent populations' declared sites and fix ranges;
|
||||||
benign controls and recorded adjudications. Assumed-benign commits are their own source,
|
- the vulnerable and fixed sides of pairs, including the 97 real public fixes in
|
||||||
weighted 0.5 and reported separately. A plane verdict is a feature, never a label: "rejected by
|
`benchmarks/pairs/advisory-fixes/`;
|
||||||
the plane" is not a negative, or the engine would learn to reproduce the plane's mistakes.
|
- benign controls;
|
||||||
4. Retrieval finds similar labelled cases. A Gower distance over the signal profile, re-ranked
|
- recorded adjudications.
|
||||||
by cosine similarity of code embeddings (OpenAI `text-embedding-3-small` through OpenRouter,
|
|
||||||
cached by excerpt hash). The evaluation boundary is the only door into a prompt: never the
|
Assumed-benign commits are their own source. They are weighted 0.5 and reported separately. A
|
||||||
candidate's own repository group, never a group the fold evaluates.
|
plane verdict is a feature, never a label. "Rejected by the plane" is not a negative.
|
||||||
5. A classifier compiled in house, DSPy-style. `retrieve -> classify(k)`: an instruction and
|
Otherwise the engine would learn to reproduce the plane's mistakes.
|
||||||
compiled demonstrations (a fixed prefix, so the provider's cache hits), the retrieved examples,
|
4. **Retrieval finds similar labelled cases.** It uses a Gower distance (a distance for mixed
|
||||||
then the candidate's excerpt and signals. The model answers vulnerable, not vulnerable or
|
numeric and categorical data) over the signal profile. Results are re-ranked by cosine
|
||||||
unsure with a rationale that cites a line; `k` samples give a score. One program is compiled
|
similarity of code embeddings. The embeddings are OpenAI `text-embedding-3-small` through
|
||||||
per profile and family on a compile split that is never evaluated.
|
OpenRouter, cached by excerpt hash. The evaluation boundary is the only door into a prompt. It
|
||||||
6. Calibration on held-out repositories. Platt scaling of the score, cross-fitted by
|
never admits the candidate's own repository group, or a group the fold evaluates.
|
||||||
repository group so no map is checked on the fold that fitted it. Calibration holds when the
|
5. **A classifier compiled in house, DSPy-style.** The program is `retrieve -> classify(k)`. The
|
||||||
slope interval contains 1, the intercept interval contains 0 and ECE <= 0.05.
|
prompt holds, in order:
|
||||||
7. Two operating points. BLOCK only where calibration holds and a fold reaches the required
|
- an instruction and compiled demonstrations (a fixed prefix, so the provider's cache hits);
|
||||||
precision; otherwise ADVISORY. An unsure majority can never BLOCK.
|
- the retrieved examples;
|
||||||
8. A/B experiments are the only way a change is adopted. Prompts, models, pass counts,
|
- the candidate's excerpt and signals.
|
||||||
features and sampling are compared under a pre-registered manifest with frozen units, paired
|
|
||||||
arms, a repository bootstrap and two looks. exp-002 is the first to finish: a retrieval
|
The model answers vulnerable, not vulnerable or unsure, with a rationale that cites a line. `k`
|
||||||
ensemble (five samples over disjoint example sets at temperature 0) against today's
|
samples give a score. One program is compiled per profile and family, on a compile split that
|
||||||
temperature sampling. It raised re-run agreement a little (0.906 against 0.875, interval
|
is never evaluated.
|
||||||
spanning zero) and lowered the within-pair AUC from 0.815 to 0.741 at 2.4 times the cost per
|
6. **Calibration on held-out repositories.** Platt scaling (a logistic fit) maps the score to a
|
||||||
candidate ($0.0051 against $0.0021). Decision: REJECTED
|
probability. It is cross-fitted by repository group, so no map is checked on the fold that
|
||||||
(`benchmarks/experiments/exp-002-retrieval-ensemble/result.json`).
|
fitted it. Calibration holds when three conditions are met: the slope interval contains 1, the
|
||||||
9. Extrapolation with intervals. Every number carries a 95% repository-cluster bootstrap
|
intercept interval contains 0 and ECE <= 0.05.
|
||||||
interval; the pooled and the within-pair readings are both reported, and the stricter one is
|
7. **Two operating points.** BLOCK applies only where calibration holds and a fold reaches the
|
||||||
|
required precision. Otherwise the point is ADVISORY. An unsure majority can never BLOCK.
|
||||||
|
8. **A/B experiments are the only way a change is adopted.** Prompts, models, pass counts,
|
||||||
|
features and sampling are compared under a pre-registered manifest. The manifest fixes frozen
|
||||||
|
units, paired arms, a repository bootstrap and two looks. exp-002 is the first to finish:
|
||||||
|
- It compared a retrieval ensemble (five samples over disjoint example sets at temperature 0)
|
||||||
|
with today's temperature sampling.
|
||||||
|
- It raised re-run agreement a little (0.906 against 0.875, interval spanning zero).
|
||||||
|
- It lowered the within-pair AUC from 0.815 to 0.741 and cost 2.4 times as much per candidate
|
||||||
|
($0.0051 against $0.0021).
|
||||||
|
- Decision: REJECTED (`benchmarks/experiments/exp-002-retrieval-ensemble/result.json`).
|
||||||
|
9. **Extrapolation with intervals.** Every number carries a 95% repository-cluster bootstrap
|
||||||
|
interval. Both the pooled and the within-pair readings are reported. The stricter one is
|
||||||
named.
|
named.
|
||||||
10. The one-time v3 check. When a program is adopted and a prediction is committed, population
|
10. **The one-time v3 check.** It happens when a program is adopted and a prediction is
|
||||||
v3 is scored once under `benchmarks/independent/protocol-v3.md`.
|
committed. Then population v3 is scored once under `benchmarks/independent/protocol-v3.md`.
|
||||||
|
|
||||||
## 3. AI integration: ax on the server
|
## 3. AI integration: ax on the server
|
||||||
|
|
||||||
All model work runs as Tasks on google/ax over Agent Substrate, in a single-node kind cluster on
|
All model work runs as Tasks on google/ax over Agent Substrate. They run in a single-node kind
|
||||||
the maintainer's host, each Task in its own gVisor sandbox. There is no local subprocess path.
|
cluster on the maintainer's host. Each Task runs in its own gVisor sandbox. There is no local
|
||||||
The reconciler (`ousast plane run`) is a host process: it applies a Run's Workspaces and Tasks,
|
subprocess path.
|
||||||
resumes each Task, sends its start signal, collects its artifacts and records state. Flow and
|
|
||||||
contract: [The plane on ax](plane.md); bring-up and the host's lessons: [ax on this host](ops/ax/README.md).
|
The reconciler (`ousast plane run`) is a host process. It does five things:
|
||||||
|
|
||||||
|
1. applies a Run's Workspaces and Tasks;
|
||||||
|
2. resumes each Task;
|
||||||
|
3. sends its start signal;
|
||||||
|
4. collects its artifacts;
|
||||||
|
5. records state.
|
||||||
|
|
||||||
|
Flow and contract: [The plane on ax](plane.md). Bring-up and the host's lessons:
|
||||||
|
[ax on this host](ops/ax/README.md).
|
||||||
|
|
||||||
What each Run guarantees:
|
What each Run guarantees:
|
||||||
|
|
||||||
- One Model and one budget per Task. A Task binds at most one ax Model (`deepseek-flash` for
|
- **One Model and one budget per Task.** A Task binds at most one ax Model and
|
||||||
chat, `openrouter-embedding` for embeddings) and `budget: {usd, calls}`. The metered client
|
`budget: {usd, calls}`. The Models are `deepseek-flash` for chat and `openrouter-embedding` for
|
||||||
refuses the next call at the ceiling; the task ends unfinished and resumes on a rerun. Model-free
|
embeddings. The metered client refuses the next call at the ceiling. The task ends unfinished
|
||||||
tasks run with a zero budget, so a stray call fails loudly.
|
and resumes on a rerun. Model-free tasks run with a zero budget, so a stray call fails loudly.
|
||||||
- Credentials exist only in the start request. Agent Substrate boots each template once as a
|
- **Credentials exist only in the start request.** Agent Substrate boots each template once as a
|
||||||
golden actor, so the runner never starts by itself. It waits for `POST /ousast/v1/start`, which
|
golden actor. So the runner never starts by itself. It waits for `POST /ousast/v1/start`. The
|
||||||
the reconciler sends through `atenet-router` to that actor alone. The key is read from the
|
reconciler sends that request through `atenet-router` to that actor alone.
|
||||||
operator's environment or `.env`, kept in memory by the runner, passed to the command and
|
- The key is read from the operator's environment or `.env`.
|
||||||
redacted from echoed stderr; no manifest, rendered Task or file carries it.
|
- The runner keeps it in memory and passes it to the command.
|
||||||
- Per-task egress. One EgressPolicy per actor through the `atenet-egress` gateway, deny by
|
- It is redacted from echoed stderr.
|
||||||
default, hostnames only: plain HTTP to the artifact receiver, TLS passthrough to the Workspaces'
|
- No manifest, rendered Task or file carries it.
|
||||||
Git hosts and the Model's declared hosts. Deleted with the actor.
|
- **Per-task egress.** Each actor gets one EgressPolicy through the `atenet-egress` gateway. It is
|
||||||
- Artifacts come back through the gateway. The runner posts its output directory as a tar to
|
deny by default and uses hostnames only. It allows plain HTTP to the artifact receiver. It allows
|
||||||
the receiver, an HTTP server inside the reconciler process on the host, reached through a Service
|
TLS passthrough to the Workspaces' Git hosts and the Model's declared hosts. It is deleted with
|
||||||
with an EndpointSlice to the host. That delivery is the task's completion; ax has no Completed
|
the actor.
|
||||||
phase.
|
- **Artifacts come back through the gateway.** The runner posts its output directory as a tar to
|
||||||
- Memory on S3-compatible RustFS. `OUSAST_MEMORY=s3://sast-memory` on a RustFS server. The
|
the receiver. The receiver is an HTTP server inside the reconciler process on the host. It is
|
||||||
store verifies the bucket at every start (versioning, lifecycle, tags, S3 Select) and refuses to
|
reached through a Service with an EndpointSlice to the host. That delivery is the task's
|
||||||
start when anything is missing; there is no local or fetch-and-filter fallback. Each row kind
|
completion; ax has no Completed phase.
|
||||||
has a fixed set of queryable fields, enforced at write time
|
- **Memory on S3-compatible RustFS.** The setting is `OUSAST_MEMORY=s3://sast-memory` on a RustFS
|
||||||
|
server.
|
||||||
|
- The store verifies the bucket at every start: versioning, lifecycle, tags, S3 Select.
|
||||||
|
- It refuses to start when anything is missing. There is no local or fetch-and-filter fallback.
|
||||||
|
- Each row kind has a fixed set of queryable fields, enforced at write time
|
||||||
([Memory](memory.md#verify-at-startup-no-fallback), [RustFS setup](rustfs.md)).
|
([Memory](memory.md#verify-at-startup-no-fallback), [RustFS setup](rustfs.md)).
|
||||||
|
|
||||||
Lessons the host taught, each from a failed live run (the full list is on
|
Lessons the host taught, each from a failed live run (the full list is on
|
||||||
[ax on this host](ops/ax/README.md#what-this-host-taught-each-one-cost-a-failed-live-run)):
|
[ax on this host](ops/ax/README.md#what-this-host-taught-each-one-cost-a-failed-live-run)):
|
||||||
|
|
||||||
- Substrate installs no WorkerPool, and actor images must be pinned by digest.
|
- Substrate installs no WorkerPool, and actor images must be pinned by digest.
|
||||||
- A release `ax` CLI skews from the server; build it from the deployed checkout.
|
- A release `ax` CLI skews from the server. Build it from the deployed checkout.
|
||||||
- ax has no Completed phase, so completion is our artifact delivery.
|
- ax has no Completed phase, so completion is our artifact delivery.
|
||||||
- The golden boot means the runner must wait for a start request.
|
- The golden boot means the runner must wait for a start request.
|
||||||
- The egress gateway dials the address the actor connected to, so the receiver is a Service
|
- The egress gateway dials the address the actor connected to. So the receiver is a Service,
|
||||||
dialled by ClusterIP with its name as Host.
|
dialled by ClusterIP with its name as Host.
|
||||||
- ax's snapshot bucket must exist in your S3 server. When it did not, deleted golden actors piled
|
- ax's snapshot bucket must exist in your S3 server. When it did not, deleted golden actors piled
|
||||||
up in `DELETING` and filled the disk during a 600-task harvest on 2026-10-01.
|
up in `DELETING`. They filled the disk during a 600-task harvest on 2026-10-01.
|
||||||
|
|
||||||
Deployment status, in the two words [Deployment](deployment.md) uses strictly. Implemented:
|
Deployment status uses two words, as [Deployment](deployment.md) does, strictly.
|
||||||
ax as the only executor, per-task EgressPolicy, credentials only in the start request, per-task
|
|
||||||
budgets and attribution, the S3 store verified at startup, local bring-up, doctor and smoke Run,
|
Implemented:
|
||||||
re-pinning generated Tasks to another registry. Planned, not done: deployment to any cluster
|
|
||||||
other than this host's kind cluster, a configurable kube context (`kind-` is hard-coded in
|
- ax as the only executor;
|
||||||
`doctor.py`, `egress.py`, `router.py`), the receiver as an in-cluster Deployment, the
|
- per-task EgressPolicy;
|
||||||
reconciler in-cluster with Secrets injected, presigned-URL delivery to the store.
|
- credentials only in the start request;
|
||||||
|
- per-task budgets and attribution;
|
||||||
|
- the S3 store verified at startup;
|
||||||
|
- local bring-up, doctor and smoke Run;
|
||||||
|
- re-pinning generated Tasks to another registry.
|
||||||
|
|
||||||
|
Planned, not done:
|
||||||
|
|
||||||
|
- deployment to any cluster other than this host's kind cluster;
|
||||||
|
- a configurable kube context (`kind-` is hard-coded in `doctor.py`, `egress.py`, `router.py`);
|
||||||
|
- the receiver as an in-cluster Deployment;
|
||||||
|
- the reconciler in-cluster with Secrets injected;
|
||||||
|
- presigned-URL delivery to the store.
|
||||||
|
|
||||||
## 4. Token economics, before and after
|
## 4. Token economics, before and after
|
||||||
|
|
||||||
Before the plane, the script pipeline cost $0.022 per candidate for two verify passes
|
Before the plane, two costs stood out:
|
||||||
(`benchmarks/independent/verifier-batched-check-2026-09-29.json`), and the maintainers' own
|
|
||||||
coding sessions spent about 90% of their tokens on tool input and output rather than reasoning.
|
- The script pipeline cost $0.022 per candidate for two verify passes
|
||||||
After it, the per-candidate costs below are what the records show. The mechanisms (one budget
|
(`benchmarks/independent/verifier-batched-check-2026-09-29.json`).
|
||||||
per Task, pay-once reuse, cheap-before-expensive ordering, replay at $0) are on
|
- The maintainers' own coding sessions spent about 90% of their tokens on tool input and output
|
||||||
[Token ergonomics](token-ergonomics.md); this table is the before and after.
|
rather than reasoning.
|
||||||
|
|
||||||
|
After it, the per-candidate costs below are what the records show. The mechanisms are on
|
||||||
|
[Token ergonomics](token-ergonomics.md): one budget per Task, pay-once reuse,
|
||||||
|
cheap-before-expensive ordering and replay at $0. This table is the before and after.
|
||||||
|
|
||||||
| Item | Before | After | Record |
|
| Item | Before | After | Record |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
@@ -195,63 +273,70 @@ per Task, pay-once reuse, cheap-before-expensive ordering, replay at $0) are on
|
|||||||
| Harvest of plane signals for the engine | none existed | `verify` $2.5373 over 1,961 calls and 474 tasks; `roles` $1.4036 over 1,034 calls and 151 tasks | `2026-10-01-decision-engine-harvest/record.json` |
|
| Harvest of plane signals for the engine | none existed | `verify` $2.5373 over 1,961 calls and 474 tasks; `roles` $1.4036 over 1,034 calls and 151 tasks | `2026-10-01-decision-engine-harvest/record.json` |
|
||||||
| Development transcript, tool I/O share | 90% | 85.5% over 2,701 tool calls (`dev_token_report`) | `plane-increment-1.json` |
|
| Development transcript, tool I/O share | 90% | 85.5% over 2,701 tool calls (`dev_token_report`) | `plane-increment-1.json` |
|
||||||
|
|
||||||
Two notes from the records. The meter prices every call at the Model's list price
|
Two notes from the records:
|
||||||
(`plane/models/deepseek-flash.yaml`); the DeepSeek account moved less, about one third of the
|
|
||||||
metered amount on the injection slice ($0.30 for $0.91 metered), and budgets and gates are set
|
1. The meter prices every call at the Model's list price (`plane/models/deepseek-flash.yaml`). The
|
||||||
against the meter, never against the balance. And the development share barely moved: 90 to 85.5
|
DeepSeek account moved less: about one third of the metered amount on the injection slice
|
||||||
is a small gain, and the remedies (a read guard, a bulk reader, scripts in files) target file
|
($0.30 for $0.91 metered). Budgets and gates are set against the meter, never against the
|
||||||
contents because that is where the tokens go.
|
balance.
|
||||||
|
2. The development share barely moved: 90 to 85.5 is a small gain. The remedies are a read guard,
|
||||||
|
a bulk reader and scripts in files. They target file contents, because that is where the
|
||||||
|
tokens go.
|
||||||
|
|
||||||
## 5. Open items
|
## 5. Open items
|
||||||
|
|
||||||
Five groups, each item with its record: the engine's own gates, data, measurement debt,
|
There are five groups. Each item has its record. The groups are the engine's own gates, data,
|
||||||
infrastructure, and instrument gaps.
|
measurement debt, infrastructure, and instrument gaps.
|
||||||
|
|
||||||
The engine's gates.
|
The engine's gates.
|
||||||
|
|
||||||
- BLOCK is reachable for no family: calibration fails for five (ECE 0.071 to 0.119) and the
|
- BLOCK is reachable for no family. Calibration fails for five (ECE 0.071 to 0.119). The sixth
|
||||||
sixth reaches no fold with a Wilson lower bound of 0.95 on precision (slice-2 record).
|
reaches no fold with a Wilson lower bound of 0.95 on precision (slice-2 record).
|
||||||
- Re-run agreement is below 0.9 for three families: injection 0.77, access control 0.83,
|
- Re-run agreement is below 0.9 for three families: injection 0.77, access control 0.83,
|
||||||
untrusted destination 0.87. exp-002 tried the obvious variance fix and lost on the ranking.
|
untrusted destination 0.87. exp-002 tried the obvious variance fix and lost on the ranking.
|
||||||
|
|
||||||
Data.
|
Data.
|
||||||
|
|
||||||
- Positives-only groups: 98 (family, group) pairs in the 904-example memory hold positive examples
|
- Positives-only groups: 98 (family, group) pairs in the 904-example memory hold positive examples
|
||||||
and no negative. 94 of them exist because the fixed side no longer has the function (the vibe-py
|
and no negative.
|
||||||
pairs). No advisory-fixes group is among them
|
- 94 of them exist because the fixed side no longer has the function (the vibe-py pairs).
|
||||||
(`benchmarks/measurements/2026-10-02-decision-engine-negatives-gap/record.json`). There are no
|
- No advisory-fixes group is among them
|
||||||
single-label negatives, so every family's pooled AUC is inflated by positives that never had to
|
(`benchmarks/measurements/2026-10-02-decision-engine-negatives-gap/record.json`).
|
||||||
be told apart from their own fix (slice-2 record, `paired_groups_check`).
|
- There are no single-label negatives. So every family's pooled AUC is inflated by positives
|
||||||
- The model has seen public fixes. A verify pass named the advisory's CVE id for a public fix, an
|
that never had to be told apart from their own fix (slice-2 record, `paired_groups_check`).
|
||||||
observation the engine's design specification (maintainer tooling) records as visible
|
- The model has seen public fixes. A verify pass named the advisory's CVE id for a public fix.
|
||||||
memorisation. Excerpts now redact advisory ids and security-worded comments
|
The engine's design specification (maintainer tooling) records this as visible memorisation.
|
||||||
(`src/openultrasast/learn/excerpt.py`), but labels drawn from public advisories are partly in the
|
Excerpts now redact advisory ids and security-worded comments
|
||||||
model's training data, and the within-pair numbers carry that.
|
(`src/openultrasast/learn/excerpt.py`). But labels drawn from public advisories are partly in
|
||||||
- PHP is covered by the engine on real code and by in-sample quick rules; the population that
|
the model's training data, and the within-pair numbers carry that.
|
||||||
|
- PHP is covered by the engine on real code and by in-sample quick rules. The population that
|
||||||
would measure it (v3) is reserved.
|
would measure it (v3) is reserved.
|
||||||
|
|
||||||
Measurement debt.
|
Measurement debt.
|
||||||
|
|
||||||
- exp-001 (known callers) is registered and has not run; the model-free candidate generation on
|
- exp-001 (known callers) is registered and has not run. Its units need model-free candidate
|
||||||
the 41 development repositories that its units need is not built
|
generation on the 41 development repositories. That generation is not built
|
||||||
(`benchmarks/experiments/exp-001-known-callers/registration.json`).
|
(`benchmarks/experiments/exp-001-known-callers/registration.json`).
|
||||||
- Leave-one-source-out, leave-one-framework-out and the memory-size curve points (0, 25, 50%)
|
- Leave-one-source-out, leave-one-framework-out and the memory-size curve points (0, 25, 50%)
|
||||||
have not run; one paired point exists (724 against 904 examples, injection, AUC 0.76 against 0.77).
|
have not run. One paired point exists (724 against 904 examples, injection, AUC 0.76 against 0.77).
|
||||||
- The plane-profile leak audit flagged `verify.flag_b` and `ms.operation`; they are withheld from
|
- The plane-profile leak audit flagged `verify.flag_b` and `ms.operation`. They are withheld from
|
||||||
profile v1, and whether they track the label on unseen code has to be decided before any
|
profile v1. Before any plane-profile program runs, someone must decide whether they track the
|
||||||
plane-profile program runs (`2026-10-02-decision-engine-harvest-advisory/record.json`).
|
label on unseen code (`2026-10-02-decision-engine-harvest-advisory/record.json`).
|
||||||
- Population v3 waits for `benchmarks/independent/prediction-v3.json`, which does not exist. It is
|
- Population v3 waits for `benchmarks/independent/prediction-v3.json`, which does not exist. It is
|
||||||
written only after a program is adopted.
|
written only after a program is adopted.
|
||||||
|
|
||||||
Infrastructure.
|
Infrastructure.
|
||||||
|
|
||||||
- Deployment to a separate Kubernetes cluster is planned, not done; the code changes it needs are
|
- Deployment to a separate Kubernetes cluster is planned, not done. The code changes it needs are
|
||||||
listed with file paths on [Deployment](deployment.md#what-is-implemented-and-what-is-planned).
|
listed with file paths on [Deployment](deployment.md#what-is-implemented-and-what-is-planned).
|
||||||
|
|
||||||
Instrument gaps.
|
Instrument gaps.
|
||||||
|
|
||||||
- A missing provider key is a quiet path in a scan. The plane refuses to start a Task whose Model
|
- A missing provider key is a quiet path in a scan.
|
||||||
names an unset variable (`plane/router.py`, `StartError`). A scan without a key asks the model
|
- The plane refuses to start a Task whose Model names an unset variable (`plane/router.py`,
|
||||||
nothing, reports only the graph's entailed findings, and records `learning_endpoint_unavailable`
|
`StartError`).
|
||||||
in `manifest.json` (`cli.py`, after `resolve_chat_endpoint` returns nothing); `report.md` has no
|
- A scan without a key asks the model nothing and reports only the graph's entailed findings.
|
||||||
line for that reason yet, so the report does not say the model layer was skipped.
|
It records `learning_endpoint_unavailable` in `manifest.json` (`cli.py`, after
|
||||||
|
`resolve_chat_endpoint` returns nothing).
|
||||||
|
- `report.md` has no line for that reason yet. So the report does not say the model layer was
|
||||||
|
skipped.
|
||||||
|
|||||||
Reference in New Issue
Block a user