marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: plain-language pass -- README, index, where we stand, scanning, examples, threat model

Browse Source 
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-10-02 21:01:10 +02:00
co-authored by Claude Opus 5.5
parent 50f4e69321
commit 552671ca6a
6 changed files with 685 additions and 440 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/examples.md
+28 -17
View File
@@ -1,7 +1,7 @@
# End-to-end examples
Each example is self-contained. Artifacts land under
`<target>/.openultrasast/runs/<scan-id>/` and `.openultrasast/` is gitignored.
Each example stands on its own. Artifacts land under `<target>/.openultrasast/runs/<scan-id>/`.
`.openultrasast/` is gitignored.
## 1. Quick scan a local repo (deterministic, no keys)
@@ -21,7 +21,7 @@ ousast scan . --mode standard # Joern model layer, container network off:
ousast-with-judge scan . --mode standard # network on, so the model is asked (needs DEEPSEEK_API_KEY)
```
Without a key the graph still decides: its entailed findings are reported, and the LLM's residual
Without a key, the graph still decides. Its entailed findings are reported. The LLM's residual
and suspicion questions go unasked.
## 3. CI gate
@@ -52,10 +52,11 @@ uv run ousast pairs --split holdout --profile human # filter by declared split
uv run python -m openultrasast.pair_gate # CI: local pairs must all pass
```
Each case is two isolated trees (vuln file vs patched file, same relative path).
`pair_correct` means the harness fired on the vuln side and stayed silent on the
fix. Misses and fix-side leaks become `miss`/`fp` signals for `ousast improve`.
The catalog and public dataset pointers live in `benchmarks/pairs/`.
Each case is two isolated trees: the vuln file and the patched file, at the same relative path.
- `pair_correct` means the harness fired on the vuln side and stayed silent on the fix.
- Misses become `miss` signals and fix-side leaks become `fp` signals for `ousast improve`.
- The catalog and public dataset pointers live in `benchmarks/pairs/`.
## 6. Self-improve the ruleset from benchmark feedback
@@ -68,10 +69,11 @@ uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml
## 7. Running the agentic plane
Model work over whole repositories runs as a Run manifest on ax, one isolated actor
per task (bring-up: [ops/ax/README.md](ops/ax/README.md)). The provider key is
read from the environment (or `.env`, which never overrides an exported variable) by
`ousast` and sent only in each task's start request.
Model work over whole repositories runs as a Run manifest on ax. Each task gets one isolated
actor. Bring-up is in [ops/ax/README.md](ops/ax/README.md).
`ousast` reads the provider key from the environment, or from `.env`. `.env` never overrides an
exported variable. The key is sent only in each task's start request.
```bash
uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image
@@ -83,16 +85,25 @@ uv run ousast plane remember validation-46 # re-ingest the run's m
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run --memory
```
Each task in the Run carries its own Model and `budget: {usd, calls}`; a task that
reaches its ceiling stops as `unfinished` and resumes on a rerun with a larger
budget. `ousast plane status` prints the token attribution table: calls, prompt,
cache-hit and output tokens and USD per task.
Each task in the Run carries its own Model and `budget: {usd, calls}`.
- A task that reaches its ceiling stops as `unfinished`. It resumes on a rerun with a larger
budget.
- `ousast plane status` prints the token attribution table. It shows calls, prompt, cache-hit and
output tokens, and USD per task.
## 8. Drive it from an MCP client (OpenCode / IDE)
MCP (Model Context Protocol) lets an editor or agent call the tool's functions.
```jsonc
{ "command": "uv", "args": ["run", "ousast", "mcp"] }
```
Then: `openultrasast.scan {path}` → `run_dir` → `openultrasast.findings {run_dir}` →
`openultrasast.explain {run_dir, finding_id}`. See the README section "Agent integrations".
Then call the tools in this order:
1. `openultrasast.scan {path}` returns a `run_dir`.
2. `openultrasast.findings {run_dir}` lists the findings.
3. `openultrasast.explain {run_dir, finding_id}` explains one.
See the README section "Agent integrations".
docs/index.md
+24 -18
View File
@@ -1,13 +1,13 @@
# OpenUltraSAST
OpenUltraSAST is an experimental static security analyser meant to work as a safety net on
repositories it has never seen: it scans a checkout, or the commits a `git push` would publish,
and separates what it can prove from what it merely suspects. A scan that could not look at
something records a degradation instead of reporting a clean result.
OpenUltraSAST is an experimental static security analyser. It is meant to be a safety net on
repositories it has never seen. It scans a checkout, or the commits a `git push` would publish. It
keeps what it can prove apart from what it only suspects. When a scan could not look at
something, it records a degradation. It does not report a clean result.
**State as of 2026-10-02 (v2.0.0):** the pre-push safety net is **NO-GO** for rollout.
No hook capability is enabled, and no independent population has passed the qualification gates
(see [Evaluation](evaluation.md)).
No hook capability is enabled. No independent population (a case set frozen before any scan) has
passed the qualification gates (see [Evaluation](evaluation.md)).
## Three layers
@@ -17,6 +17,9 @@ No hook capability is enabled, and no independent population has passed the qual
| **The agentic plane on ax** | Model-driven work over whole repositories, run as a DAG of tasks on google/ax, one isolated actor per task. What the runs learn is kept in a memory store. Maintainer surface: `ousast plane`. | [The plane on ax](plane.md), [Memory](memory.md) |
| **The decision engine** | A learned classifier that weighs every instrument's signals against similar labelled cases from memory. **In development, not adopted**: no scan, pre-push check or report uses it. Maintainer surface: `ousast learn`. | [Memory and detection](memory-and-detection.md), [Decision engine](decision-engine.md) |
A code property graph is a graph of the program's syntax, control flow and data flow. Joern builds
it.
## How they connect
```mermaid
@@ -33,31 +36,34 @@ flowchart LR
engine -. "intended, not adopted: BLOCK / ADVISORY" .-> report
```
- The **scanners** produce findings and, inside plane Runs, the `alerts` that become memory rows.
- The **plane** runs model work (verify passes, roles) and model-free bookkeeping (facts, agreement,
feature records) per case; its `remember` task turns a case's artifacts into rows of the
[memory store](memory.md).
- Memory feeds two consumers. `ousast improve --memory` turns stored rows into rule-status
proposals that pass through the same validator and gate as every other edit. The decision engine
retrieves labelled examples from the same store.
- The **scanners** produce findings. Inside plane Runs, they also produce the `alerts` that become
memory rows.
- The **plane** runs model work per case: verify passes and roles. It also runs model-free
bookkeeping per case: facts, agreement and feature records. Its `remember` task turns a case's
artifacts into rows of the [memory store](memory.md).
- Memory feeds two consumers:
1. `ousast improve --memory` turns stored rows into rule-status proposals. They pass through the
same validator and gate as every other edit.
2. The decision engine retrieves labelled examples from the same store.
- The decision engine's output (dotted line) is the intended path. Nothing user-facing reads it
today.
## Where to go next
- The current state in one page, every figure with its record: capability against the M4 gates, the
step-by-step approach, ax on the server, token economics before and after, and the open items:
[Where we stand](where-we-stand.md).
- The current state in one page, every figure with its record: [Where we stand](where-we-stand.md).
It covers capability against the M4 gates, the step-by-step approach, ax on the server, token
economics before and after, and the open items.
- Run a scan: [Examples](examples.md), and the commands in [Scanning](scanning.md).
- Understand a finding's evidence and score: [Architecture](architecture.md).
- The program analysis under a scan, what each representation can see and what it misses:
[Detection techniques](detection-techniques.md).
- Operate the plane and its store: [ax on this host](ops/ax/README.md) and
[RustFS setup](rustfs.md).
- Where the plane runs today and what a separate Kubernetes cluster would take (planned, not
- Where the plane runs today, and what a separate Kubernetes cluster would take (planned, not
done): [Deployment](deployment.md).
- Where the model spend goes, per Task at runtime and in the maintainers' own coding sessions,
and what is reused instead of paid again: [Token ergonomics](token-ergonomics.md).
- Trust boundaries and hardening: [Threat model](threat-model.md).
Every number in these pages is quoted from a committed record whose path is given next to it.
Every number in these pages is quoted from a committed record. The record's path is given next to
the number.
docs/scanning.md
+59 -33
View File
@@ -1,13 +1,21 @@
# Scanning
`ousast scan PATH --mode quick|standard|deep` scans a checkout; `ousast pre-push` checks only
what a push (or an explicit `--base`/`--head` pair) changes. The stage-by-stage pipeline,
evidence ladder and score are in [Architecture](architecture.md); this page shows the flows.
This page shows the scan flows.
- `ousast scan PATH --mode quick|standard|deep` scans a checkout.
- `ousast pre-push` checks only what a push changes, or what an explicit `--base`/`--head` pair
changes.
The stage-by-stage pipeline, the evidence ladder and the score are in
[Architecture](architecture.md).
## The three scan modes
The stages a mode requests are `MODE_STAGES` in `src/openultrasast/stages.py`: `quick` runs
STATIC, `standard` adds MAP, `deep` adds REGRESS.
The stages each mode requests are listed in `MODE_STAGES` in `src/openultrasast/stages.py`:
- `quick` runs STATIC.
- `standard` adds MAP.
- `deep` adds REGRESS.
```mermaid
flowchart TD
@@ -38,7 +46,7 @@ Sources: `README.md` (scan modes), `src/openultrasast/cli.py` (`run_stage` calls
## Quick rules per language
Rules run only against files of their language. The bundled ruleset lives in
Each rule runs only against files of its language. The bundled ruleset lives in
`src/openultrasast/ruleset/<language>/rules.toml`.
| Language | Classes (CWE) |
@@ -49,21 +57,31 @@ Rules run only against files of their language. The bundled ruleset lives in
| Java + Groovy templates | command injection (78), SQLi (89), weak hash (327), deserialization (502), unescaped template XSS (79) |
| PHP | SQLi (89), command injection (78), file inclusion (98), path (22), open redirect (601), SSRF (918); code eval (95), unserialize (502) and echo XSS (79) ship as `shadow` rules |
The detection gate (`python -m openultrasast.gate`) enforces at least 90% recall and under 10%
false positives per language on the bundled corpora in `benchmarks/manifests/`. Those corpora are
cheat-sheet fixtures: they catch regressions, they do not estimate recall on real code. **PHP is
not in the gate**; its numbers are in-sample development numbers
(`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`). See `README.md` for the
gate's per-language output.
The detection gate (`python -m openultrasast.gate`) enforces two limits per language on the
bundled corpora in `benchmarks/manifests/`: at least 90% recall and under 10% false positives.
- Those corpora are cheat-sheet fixtures. They catch regressions. They do not estimate recall on
real code.
- **PHP is not in the gate**. Its numbers are in-sample development numbers
(`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`).
See `README.md` for the gate's per-language output.
## The Joern engine
The model layer (`src/openultrasast/model/`, `src/openultrasast/cpg/`) is the core of `standard`.
It builds one code property graph per scan with Joern (`joern-parse`, then batched
`joern --script` queries from `cpg/queries/*.sc`) and sends each question to the arbiter of its
type: **taint** (source to sink with no discharging sanitizer), **dominance** (does a guard govern
the obligated operation) and **configuration** (constant abstraction over a value). Taint facts
ship for C, Java, JavaScript, PHP and Python (`src/openultrasast/ruleset/semantic/*.toml`).
It builds one code property graph per scan with Joern. A code property graph is a graph of the
program's syntax, control flow and data flow. The build runs `joern-parse`, then batched
`joern --script` queries from `cpg/queries/*.sc`.
The layer sends each question to the arbiter (the decider) of its type:
- **taint**: a flow from source to sink with no discharging sanitizer;
- **dominance**: does a guard govern the obligated operation;
- **configuration**: constant abstraction over a value.
Taint facts ship for C, Java, JavaScript, PHP and Python
(`src/openultrasast/ruleset/semantic/*.toml`).
```mermaid
flowchart LR
@@ -79,25 +97,33 @@ flowchart LR
rung -- "residual question only" --> judge["LLM judge: confirm or contradict, never create a flow"]
```
The Docker image ships Joern and `php-cli` (which the PHP frontend needs); without Joern the
layer records `cpg_unavailable` and the scan is otherwise unchanged. Engine installation and the
Joern features used are in [Engine and pre-push](ops/README.md).
How the graph is built, queried and refined, with the records behind each refinement, is in
[Detection techniques](detection-techniques.md).
The Docker image ships Joern and `php-cli`. The PHP frontend needs `php-cli`. Without Joern, the
layer records `cpg_unavailable`; the rest of the scan is unchanged.
- Engine installation and the Joern features used: [Engine and pre-push](ops/README.md).
- How the graph is built, queried and refined, with the records behind each refinement:
[Detection techniques](detection-techniques.md).
## Framework priors
Rule and taint-fact entries that know a framework or library carry a `framework` or `library` tag
naming a row of `src/openultrasast/ruleset/frameworks.toml` (for example `wordpress`, `flask`,
`django`, `express`, `spring`). The loaders take `priors=`: `"all"` (the default and today's
behaviour), `"off"` (language-level knowledge only) or a set of ids, so the contribution of
framework knowledge can be measured and switched off. The decision engine's first slice ran with
priors off ([Decision engine](decision-engine.md#first-measured-slice-2026-10-01)).
Some rule and taint-fact entries know a framework or library. They carry a `framework` or
`library` tag. The tag names a row of `src/openultrasast/ruleset/frameworks.toml` (for example
`wordpress`, `flask`, `django`, `express`, `spring`).
The loaders take `priors=` with one of three values:
- `"all"`: the default and today's behaviour;
- `"off"`: language-level knowledge only;
- a set of ids.
So the contribution of framework knowledge can be measured and switched off. The decision
engine's first slice ran with priors off
([Decision engine](decision-engine.md#first-measured-slice-2026-10-01)).
## Pre-push: the delta check (experimental)
`ousast pre-push` analyses the commits a `git push` would publish, or an explicit
`--base`/`--head` pair, and compares head with base so that only new or worsened defects are
`--base`/`--head` pair. It compares head with base. Only new or worsened defects become
candidates. Sources: `ousast pre-push --help`, `src/openultrasast/push/`,
[Engine and pre-push](ops/README.md#explicit-local-replay).
@@ -118,10 +144,10 @@ flowchart TD
exit -- "blocking, explicit opt-in" --> block["May block the push"]
```
- `--deadline` (30 s by default) bounds preparation, both revisions and reporting together;
`--incomplete-coverage block` opts into failing when coverage is incomplete.
- The default capability registry is empty pending independent qualification, so no normal alert
is emitted today; every candidate stays diagnostic.
- `--deadline` (30 s by default) bounds preparation, both revisions and reporting together.
- `--incomplete-coverage block` opts into failing when coverage is incomplete.
- The default capability registry is empty until independent qualification passes. So no normal
alert is emitted today. Every candidate stays diagnostic.
- A model is called only with an explicit `--model-config`.
- Hook installation (`ops/install-pre-push`) and the private cache (`--cache-dir`) are documented
in [Engine and pre-push](ops/README.md#experimental-pre-push-integration).
docs/threat-model.md
+75 -52
View File
@@ -1,78 +1,101 @@
# Threat model & hardening
OpenUltraSAST analyzes **untrusted code**. This document states what it trusts, what
it does not, and the controls that keep a scan safe to run in CI and safe to share.
OpenUltraSAST analyzes **untrusted code**. This page states what it trusts and what it does not.
It also lists the controls that keep a scan safe to run in CI and safe to share.
## Trust boundaries
- **Scanned code is untrusted input.** Quick and standard modes never import, build or run
the target: quick reads source as text, and standard additionally parses it into a code
property graph with Joern (`joern-parse`, a parser, not a build) and may send code excerpts
to the configured model. Only `deep` mode executes target-derived code: its REGRESS stage
loads promoted candidates inside the Docker sandbox below, and it runs only when `docker
info` succeeds (otherwise the stage is skipped and recorded as `sandbox_unavailable`).
- **The ruleset and CWE policy are trusted, governed data.** They change only through
reviewed commits or the bounded self-improvement loop (`ousast improve`), which can
flip rule *status* and tune score constants but can never edit pattern text or the
- **Scanned code is untrusted input.** Quick and standard modes never import, build or run the
target.
- Quick mode reads source as text.
- Standard mode also parses it into a code property graph (a graph of the program's syntax,
control flow and data flow) with Joern. It uses `joern-parse`, a parser, not a build.
Standard mode may send code excerpts to the configured model.
- Only `deep` mode executes target-derived code. Its REGRESS stage loads promoted candidates
inside the Docker sandbox described below. It runs only when `docker
info` succeeds (otherwise the stage is skipped and recorded as `sandbox_unavailable`).
- **The ruleset and CWE policy are trusted, governed data.** They change in only two ways:
through reviewed commits, or through the bounded self-improvement loop (`ousast improve`). The
loop can flip rule *status* and tune score constants. It can never edit pattern text or the
authoritative 0–5 severity. See [architecture.md](architecture.md#self-improvement).
- **Provider credentials are the operator's.** The core install (PyYAML only) makes no
network calls on its own. A local scan calls a model only in standard or deep mode, and
only when a key is available: the model layer asks the residual question the graph cannot
settle, and `[models] hunter` enables the tool hunter. LLM calls go to DeepSeek
(`DEEPSEEK_API_KEY`); OpenRouter (`OPENROUTER_API_KEY`) serves embeddings. Keys come from
the environment or a gitignored `.env`, which never overrides an exported variable. The
pre-push check calls a model only with an explicit `--model-config`. Agentic work runs on the plane (`ousast plane run`): `ousast` reads
the variable a task's Model names (`secretKey.key`) from the operator's environment or
`.env` and sends the key only in that task's start request, through Agent Substrate's
router to that task's actor. It is not baked into an image or task template, and it is
never logged or written to artifacts.
- **Provider credentials are the operator's.** The core install (PyYAML only) makes no network
calls on its own.
- A local scan calls a model only in standard or deep mode, and only when a key is available.
The model layer asks the residual question the graph cannot settle. `[models] hunter`
enables the tool hunter.
- LLM calls go to DeepSeek (`DEEPSEEK_API_KEY`). OpenRouter (`OPENROUTER_API_KEY`) serves
embeddings.
- Keys come from the environment or a gitignored `.env`. `.env` never overrides an exported
variable.
- The pre-push check calls a model only with an explicit `--model-config`.
- Agentic work runs on the plane (`ousast plane run`). `ousast` reads the variable a task's
Model names (`secretKey.key`) from the operator's environment or `.env`. It sends the key
only in that task's start request. The request goes through Agent Substrate's router to that
task's actor. The key is not baked into an image or task template. It is never logged or
written to artifacts.
## Controls
### Secret redaction
Scan artifacts can quote source that contains live credentials. `redaction.py` masks
recognizable secret shapes (provider API keys, AWS/GitHub/Slack/Google tokens, bearer
tokens, URL-embedded credentials, PEM private keys, and `key = value` assignments for
sensitive names) before traces (`trace/events.jsonl`) and the markdown report are
written. On by default; disable with `[hardening] redact_secrets = false`.
recognizable secret shapes before traces (`trace/events.jsonl`) and the markdown report are
written. It masks:
- provider API keys;
- AWS/GitHub/Slack/Google tokens;
- bearer tokens;
- URL-embedded credentials;
- PEM private keys;
- `key = value` assignments for sensitive names.
Redaction is on by default. Turn it off with `[hardening] redact_secrets = false`.
### Plane egress
Each plane task runs in its own actor behind Agent Substrate's egress gateway, which
denies by default. The task's EgressPolicy (`src/openultrasast/plane/egress.py`) allows exactly three
things: plain HTTP to the artifact receiver, and TLS passthrough to the Git hosts of the
task's bound Workspaces and to the hosts its bound Model declares
(`openultrasast.io/egress-hosts`). There are no wildcards, no IP addresses and no TLS
interception. The policy is deleted with the actor.
Each plane task runs in its own actor behind Agent Substrate's egress gateway. The gateway denies
by default. The task's EgressPolicy (`src/openultrasast/plane/egress.py`) allows exactly three
things:
1. plain HTTP to the artifact receiver;
2. TLS passthrough to the Git hosts of the task's bound Workspaces;
3. TLS passthrough to the hosts its bound Model declares (`openultrasast.io/egress-hosts`).
There are no wildcards, no IP addresses and no TLS interception. The policy is deleted with the
actor.
### Cost & CI budgets
- **Agentic spend** is bounded per plane task by the Run's `budget: {usd, calls}`. The
metered client refuses the next call once either ceiling is reached (the task is
`unfinished` and resumes on a rerun with a larger budget). A usd budget on a Model
without prices is refused rather than metered as zero, and an account or
authentication error fails the task and starts nothing further. The standard-mode tool
hunter is bounded by the per-tier hunter budgets.
- **Agentic spend** is bounded per plane task by the Run's `budget: {usd, calls}`.
- The metered client refuses the next call once either ceiling is reached. The task is then
`unfinished`. It resumes on a rerun with a larger budget.
- A usd budget on a Model without prices is refused. It is not metered as zero.
- An account or authentication error fails the task and starts nothing further.
- The standard-mode tool hunter is bounded by the per-tier hunter budgets.
- **Output size** is bounded by `[hardening] max_findings` (0 = unlimited). Truncation is
severity-ordered and disclosed as a `budget` degradation in the manifest.
severity-ordered. It is disclosed as a `budget` degradation in the manifest.
### Provider reliability
LLM/embedding calls retry transient failures (HTTP 429/5xx, connection errors, timeouts)
with exponential backoff; non-transient errors (e.g. 4xx, malformed JSON) fail fast.
LLM/embedding calls retry transient failures with exponential backoff. Transient failures are
HTTP 429/5xx, connection errors and timeouts. Non-transient errors fail fast, for example 4xx or
malformed JSON.
### Visible degradation & determinism
When an optional engine is missing (Joern: `cpg_unavailable`; Docker: `sandbox_unavailable`),
the scan falls back to its deterministic equivalent and records a `degradations` entry in the
manifest — it is never silently downgraded. Without a provider key the model layer asks no
question and reports only what the graph entailed. Runs are reproducible: fixed config,
artifact manifests, prompt hashes, and model identifiers are recorded.
A scan is never silently downgraded. When an optional engine is missing, the scan falls back to
its deterministic equivalent. It records a `degradations` entry in the manifest (Joern:
`cpg_unavailable`; Docker: `sandbox_unavailable`).
Without a provider key, the model layer asks no question. It reports only what the graph
entailed.
Runs are reproducible. The fixed config, artifact manifests, prompt hashes and model identifiers
are recorded.
### MCP surface
`ousast mcp` exposes only the ten narrow project tools. No tool runs arbitrary shell,
Docker, or internal hunter tools, and no tool accepts a free-form command argument.
`ousast mcp` exposes only the ten narrow project tools. No tool runs arbitrary shell, Docker or
internal hunter tools. No tool accepts a free-form command argument.
## Sandbox (deep mode)
The REGRESS stage (`regress/`, `sandbox/runner.py`) runs each snippet with `docker run`
and never mounts the host Docker socket:
The REGRESS stage (`regress/`, `sandbox/runner.py`) runs each snippet with `docker run`. It never
mounts the host Docker socket.
| Control | Setting |
|---|---|
@@ -84,5 +107,5 @@ and never mounts the host Docker socket:
| pids | `[sandbox] pids_limit`, default 512 |
| timeout | `[sandbox] timeout_seconds`, default 300 s |
Before anything runs, a structural deny-list (`regress/safety.py`) rejects a snippet that
mentions sockets, curl, host networking or writes under `/workspace`.
A structural deny-list (`regress/safety.py`) checks each snippet before anything runs. It rejects
a snippet that mentions sockets, curl, host networking or writes under `/workspace`.
docs/where-we-stand.md
+261 -176
View File
@@ -1,33 +1,50 @@
# Where we stand
State as of 2026-10-02 (v2.0.0). This page answers five questions in one place. How far is
detection from its gates. How does the project now try to close that gap. How does the model work
run on ax. What did the tokens cost before and after that move. What is still open. Every figure
is quoted from a committed record whose path stands next to it; detail tables stay on the pages
linked from each section.
State as of 2026-10-02 (v2.0.0). This page answers five questions in one place:
The goal has not changed: a safety net on repositories the tool has never seen, which says
BLOCK or ADVISORY about the code a push would change and stays silent on fixes and on benign
commits. Two independent populations have been spent against that goal, with engine recall 1 of
1. How far is detection from its gates?
2. How does the project now try to close that gap?
3. How does the model work run on ax?
4. What did the tokens cost before and after that move?
5. What is still open?
Every figure is quoted from a committed record. The record's path stands next to the figure.
Detail tables stay on the pages linked from each section.
The goal has not changed. The tool should be a safety net on repositories it has never seen. It
should say BLOCK or ADVISORY about the code a push would change. It should stay silent on fixes
and on benign commits.
Two independent populations have been spent against that goal. An independent population is a
set of cases frozen before any scan. Engine recall was 1 of
11 and then 0 of 17. The hand edits that closed misses on the first population did not carry to
the second. So the question is no longer which rule to add next. It is how a change to detection
can be shown to work on code it was not tuned on, at a cost per candidate that can be paid for
every push.
the second. So the question is no longer which rule to add next. The question is this: how can a
change to detection be shown to work on code it was not tuned on? And can that be done at a cost
per candidate that can be paid for every push?
The answer this page records: the pre-push safety net is NO-GO against every M4 gate, and the
project has stopped trying to reach the gates by hand. Every detection change now has to win a
controlled measurement on held-out repositories, run on an agentic plane whose cost per candidate
is known. The pieces of that method exist and have been measured. None of them has produced a
BLOCK decision yet.
The answer this page records:
- The pre-push safety net is NO-GO against every M4 gate.
- The project has stopped trying to reach the gates by hand.
- Every detection change now has to win a controlled measurement on held-out repositories. The
measurement runs on an agentic plane whose cost per candidate is known.
- The pieces of that method exist and have been measured. None of them has produced a BLOCK
decision yet.
## 1. Detection capability and goals: NO-GO, measured
No M4 gate is met on an independent population. The qualification milestone for the pre-push
safety net (M4) requires, together: at least 95% actionable precision, at least 90% supported
recall, at least 95% supported-check completion, the runtime gates, and zero fixed-side or benign
false alerts. A population is frozen before any
case is scanned, scored once under a pre-registered protocol, then spent; it can inform changes
but can qualify nothing afterwards ([Evaluation](evaluation.md#independent-populations-and-the-m4-gates)).
No M4 gate is met on an independent population. M4 is the qualification milestone for the
pre-push safety net. It requires all of these together:
- at least 95% actionable precision;
- at least 90% supported recall;
- at least 95% supported-check completion;
- the runtime gates;
- zero fixed-side or benign false alerts.
A population is frozen before any case is scanned. It is scored once under a pre-registered
protocol, then spent. After that it can inform changes but can qualify nothing
([Evaluation](evaluation.md#independent-populations-and-the-m4-gates)).
| Gate | Required | Latest independent result | Record |
| --- | --- | --- | --- |
@@ -36,156 +53,217 @@ but can qualify nothing afterwards ([Evaluation](evaluation.md#independent-popul
| Supported-check completion | >= 95% | 13,206 of 108,374 | `results-v2.json` |
| Fixed-side and benign alerts | zero | 0 fixed-side, which the record calls met trivially since nothing was detected; 3 benign alerts | `results-v2.json` |
Three later measurements narrow the picture without changing the verdict.
Three later measurements narrow the picture. They do not change the verdict.
- The plane reproduces the script pipeline's agreement at a known cost. On the model-driven
pipeline's validation set (46 candidates, 15 cases of v2, 20 declared vulnerable sites), the
plane's verify passes a and b plus a tie-break on the 10 disputed candidates agreed on 16 of 20
declared sites. The script reference agreed on 16 as well. The cost was $0.0205 per candidate
(`benchmarks/independent/plane-increment-2.json`). The record carries two caveats. The tie-break
was chosen after seeing the first measurement on the same set, so this compares plane with
script and qualifies nothing. And the passes flag fewer declared sites than the reference in
any pass, 17 against 19; that count is a correction in `plane-increment-1.json`, whose first
reading took agree's `declared_sites_matched` (19, coverage by candidates) for "found in either
pass".
- The learned decision engine ranks, but may not block. On six families, evaluated on
repositories the compiled program was not shown, the pooled AUC of its score runs from 0.77
(injection) to 0.96 (access control). The stricter reading is the within-pair AUC, which asks
whether the vulnerable side of a fix scores above its own fixed side: 0.72 to 0.93. BLOCK is
offered for no family; calibration holds only for output encoding, and there no fold reaches
the required precision (`benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`).
The per-family table is on [Decision engine](decision-engine.md#second-slice-six-families-2026-10-02).
- PHP coverage is in-sample. The PHP quick rules were written and tightened while looking at
the pins and fixtures they are scored on, so the record says so itself (`provenance.in_sample` in
`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`): a development check, not
a recall estimate. On real PHP code the Joern engine is the instrument.
- **The plane reproduces the script pipeline's agreement at a known cost.** The test used the
model-driven pipeline's validation set: 46 candidates, 15 cases of v2, 20 declared vulnerable
sites. The plane ran verify passes a and b, plus a tie-break on the 10 disputed candidates. It
agreed on 16 of 20 declared sites. The script reference agreed on 16 as well. The cost was
$0.0205 per candidate (`benchmarks/independent/plane-increment-2.json`). The record carries two
caveats:
1. The tie-break was chosen after seeing the first measurement on the same set. So this compares
plane with script and qualifies nothing.
2. The passes flag fewer declared sites than the reference in any pass: 17 against 19 (passes against reference). That
count is a correction in `plane-increment-1.json`. Its first reading took agree's
`declared_sites_matched` (19, coverage by candidates) to mean "found in either pass".
- **The learned decision engine ranks, but may not block.** It was evaluated on six families, on
repositories the compiled program was not shown.
- The pooled AUC (area under the ROC curve) of its score runs from 0.77 (injection) to 0.96
(access control).
- The stricter reading is the within-pair AUC. It asks whether the vulnerable side of a fix
scores above its own fixed side: 0.72 to 0.93.
- BLOCK is offered for no family. Calibration holds only for output encoding. There, no fold
reaches the required precision
(`benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`).
- The per-family table is on
[Decision engine](decision-engine.md#second-slice-six-families-2026-10-02).
- **PHP coverage is in-sample.** The PHP quick rules were written and tightened while looking at
the pins and fixtures they are scored on. The record says so itself (`provenance.in_sample` in
`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`). It is a development
check, not a recall estimate. On real PHP code, the Joern engine is the instrument.
What has not been touched: population v3, 15 PHP cases, frozen as the one-time final check. No
development or analysis work reads its cases, a test fails when anything in the tree names one
of its repositories, and it runs once, after a prediction is committed.
What has not been touched: population v3, 15 PHP cases, frozen as the one-time final check.
- No development or analysis work reads its cases.
- A test fails when anything in the tree names one of its repositories.
- It runs once, after a prediction is committed.
## 2. Our approach to detection, step by step
The path changed because hand edits did not transfer. Every gain before 2026-09-30 was a hand
edit after a miss: a rule, a taint vocabulary entry, a guard, a prompt line, a tie-break. Each was
tuned on the cases that exposed it, and the tuning measured on population v1 did not carry to v2
(1 of 11, then 0 of 17). The conclusions drawn: hand edits do not transfer. The data that could
teach a better decision cannot be produced by hand at the scale needed, so it has to be harvested
and labelled by machinery with a boundary around it. Framework knowledge (which function is a sink in which
framework) is useful as a prior but must stay optional, or the engine learns the vocabulary
instead of the code. A reader might object that a classifier is slower to show gains than the
next rule. It is. The difference is that its gains are measured where rules were not: on
repositories it never saw.
tuned on the cases that exposed it. The tuning measured on population v1 did not carry to v2
(1 of 11, then 0 of 17).
The chain, one job per step. The design is on [Memory and detection](memory-and-detection.md);
this is the order of operations.
The conclusions drawn:
1. Instruments produce signals, never verdicts. Quick rules, the Joern engine, model sink
classification, the plane's verify passes a and b with the tie-break c, repository facts
(callers, spans) and model source/sink/sanitizer roles each write one block of a candidate's
record. None decides alone.
2. Feature records are allow-listed, and missing is not zero. A candidate is `(path, function,
family)` of one repository and pin. An instrument that did not run, had no coverage or failed
records that state; "silent" and "did not look" stay distinguishable. Input profile v1 withholds
the engine block and the function length, which a leak audit found separate the two sides of a
pair by construction (`benchmarks/measurements/2026-10-01-decision-engine-leak-audit/`), and
since 2026-10-02 the plane's `verify.flag_b` and `ms.operation`, flagged the same way by the
audit of the plane profile (`2026-10-02-decision-engine-harvest-advisory/record.json`,
`src/openultrasast/learn/schema.py`).
3. Labels come from ground truth only. A fail-closed allow-list (`src/openultrasast/learn/sources.toml`)
names the sources: the spent populations' declared sites and fix ranges, the vulnerable and
fixed sides of pairs, including the 97 real public fixes in `benchmarks/pairs/advisory-fixes/`,
benign controls and recorded adjudications. Assumed-benign commits are their own source,
weighted 0.5 and reported separately. A plane verdict is a feature, never a label: "rejected by
the plane" is not a negative, or the engine would learn to reproduce the plane's mistakes.
4. Retrieval finds similar labelled cases. A Gower distance over the signal profile, re-ranked
by cosine similarity of code embeddings (OpenAI `text-embedding-3-small` through OpenRouter,
cached by excerpt hash). The evaluation boundary is the only door into a prompt: never the
candidate's own repository group, never a group the fold evaluates.
5. A classifier compiled in house, DSPy-style. `retrieve -> classify(k)`: an instruction and
compiled demonstrations (a fixed prefix, so the provider's cache hits), the retrieved examples,
then the candidate's excerpt and signals. The model answers vulnerable, not vulnerable or
unsure with a rationale that cites a line; `k` samples give a score. One program is compiled
per profile and family on a compile split that is never evaluated.
6. Calibration on held-out repositories. Platt scaling of the score, cross-fitted by
repository group so no map is checked on the fold that fitted it. Calibration holds when the
slope interval contains 1, the intercept interval contains 0 and ECE <= 0.05.
7. Two operating points. BLOCK only where calibration holds and a fold reaches the required
precision; otherwise ADVISORY. An unsure majority can never BLOCK.
8. A/B experiments are the only way a change is adopted. Prompts, models, pass counts,
features and sampling are compared under a pre-registered manifest with frozen units, paired
arms, a repository bootstrap and two looks. exp-002 is the first to finish: a retrieval
ensemble (five samples over disjoint example sets at temperature 0) against today's
temperature sampling. It raised re-run agreement a little (0.906 against 0.875, interval
spanning zero) and lowered the within-pair AUC from 0.815 to 0.741 at 2.4 times the cost per
candidate ($0.0051 against $0.0021). Decision: REJECTED
(`benchmarks/experiments/exp-002-retrieval-ensemble/result.json`).
9. Extrapolation with intervals. Every number carries a 95% repository-cluster bootstrap
interval; the pooled and the within-pair readings are both reported, and the stricter one is
- Hand edits do not transfer.
- The data that could teach a better decision cannot be produced by hand at the scale needed. It
has to be harvested and labelled by machinery with a boundary around it.
- Framework knowledge (which function is a sink in which framework) is useful as a prior. But it
must stay optional. Otherwise the engine learns the vocabulary instead of the code.
A reader might object that a classifier is slower to show gains than the next rule. It is. The
difference is where its gains are measured: on repositories it never saw. Rules were not measured
there.
The chain has one job per step. The design is on [Memory and detection](memory-and-detection.md).
This list is the order of operations.
1. **Instruments produce signals, never verdicts.** Each instrument writes one block of a
candidate's record. None decides alone. The instruments are:
- quick rules;
- the Joern engine;
- model sink classification;
- the plane's verify passes a and b, with the tie-break c;
- repository facts (callers, spans);
- model source/sink/sanitizer roles.
2. **Feature records are allow-listed, and missing is not zero.** A candidate is `(path, function,
family)` of one repository and pin.
- An instrument that did not run, had no coverage or failed records that state. "Silent" and
"did not look" stay distinguishable.
- Input profile v1 withholds the engine block and the function length. A leak audit found that
they separate the two sides of a pair by construction
(`benchmarks/measurements/2026-10-01-decision-engine-leak-audit/`).
- Since 2026-10-02 it also withholds the plane's `verify.flag_b` and `ms.operation`. The audit
of the plane profile flagged them the same way
(`2026-10-02-decision-engine-harvest-advisory/record.json`,
`src/openultrasast/learn/schema.py`).
3. **Labels come from ground truth only.** A fail-closed allow-list
(`src/openultrasast/learn/sources.toml`) names the sources:
- the spent populations' declared sites and fix ranges;
- the vulnerable and fixed sides of pairs, including the 97 real public fixes in
`benchmarks/pairs/advisory-fixes/`;
- benign controls;
- recorded adjudications.
Assumed-benign commits are their own source. They are weighted 0.5 and reported separately. A
plane verdict is a feature, never a label. "Rejected by the plane" is not a negative.
Otherwise the engine would learn to reproduce the plane's mistakes.
4. **Retrieval finds similar labelled cases.** It uses a Gower distance (a distance for mixed
numeric and categorical data) over the signal profile. Results are re-ranked by cosine
similarity of code embeddings. The embeddings are OpenAI `text-embedding-3-small` through
OpenRouter, cached by excerpt hash. The evaluation boundary is the only door into a prompt. It
never admits the candidate's own repository group, or a group the fold evaluates.
5. **A classifier compiled in house, DSPy-style.** The program is `retrieve -> classify(k)`. The
prompt holds, in order:
- an instruction and compiled demonstrations (a fixed prefix, so the provider's cache hits);
- the retrieved examples;
- the candidate's excerpt and signals.
The model answers vulnerable, not vulnerable or unsure, with a rationale that cites a line. `k`
samples give a score. One program is compiled per profile and family, on a compile split that
is never evaluated.
6. **Calibration on held-out repositories.** Platt scaling (a logistic fit) maps the score to a
probability. It is cross-fitted by repository group, so no map is checked on the fold that
fitted it. Calibration holds when three conditions are met: the slope interval contains 1, the
intercept interval contains 0 and ECE <= 0.05.
7. **Two operating points.** BLOCK applies only where calibration holds and a fold reaches the
required precision. Otherwise the point is ADVISORY. An unsure majority can never BLOCK.
8. **A/B experiments are the only way a change is adopted.** Prompts, models, pass counts,
features and sampling are compared under a pre-registered manifest. The manifest fixes frozen
units, paired arms, a repository bootstrap and two looks. exp-002 is the first to finish:
- It compared a retrieval ensemble (five samples over disjoint example sets at temperature 0)
with today's temperature sampling.
- It raised re-run agreement a little (0.906 against 0.875, interval spanning zero).
- It lowered the within-pair AUC from 0.815 to 0.741 and cost 2.4 times as much per candidate
($0.0051 against $0.0021).
- Decision: REJECTED (`benchmarks/experiments/exp-002-retrieval-ensemble/result.json`).
9. **Extrapolation with intervals.** Every number carries a 95% repository-cluster bootstrap
interval. Both the pooled and the within-pair readings are reported. The stricter one is
named.
10. The one-time v3 check. When a program is adopted and a prediction is committed, population
v3 is scored once under `benchmarks/independent/protocol-v3.md`.
10. **The one-time v3 check.** It happens when a program is adopted and a prediction is
committed. Then population v3 is scored once under `benchmarks/independent/protocol-v3.md`.
## 3. AI integration: ax on the server
All model work runs as Tasks on google/ax over Agent Substrate, in a single-node kind cluster on
the maintainer's host, each Task in its own gVisor sandbox. There is no local subprocess path.
The reconciler (`ousast plane run`) is a host process: it applies a Run's Workspaces and Tasks,
resumes each Task, sends its start signal, collects its artifacts and records state. Flow and
contract: [The plane on ax](plane.md); bring-up and the host's lessons: [ax on this host](ops/ax/README.md).
All model work runs as Tasks on google/ax over Agent Substrate. They run in a single-node kind
cluster on the maintainer's host. Each Task runs in its own gVisor sandbox. There is no local
subprocess path.
The reconciler (`ousast plane run`) is a host process. It does five things:
1. applies a Run's Workspaces and Tasks;
2. resumes each Task;
3. sends its start signal;
4. collects its artifacts;
5. records state.
Flow and contract: [The plane on ax](plane.md). Bring-up and the host's lessons:
[ax on this host](ops/ax/README.md).
What each Run guarantees:
- One Model and one budget per Task. A Task binds at most one ax Model (`deepseek-flash` for
chat, `openrouter-embedding` for embeddings) and `budget: {usd, calls}`. The metered client
refuses the next call at the ceiling; the task ends unfinished and resumes on a rerun. Model-free
tasks run with a zero budget, so a stray call fails loudly.
- Credentials exist only in the start request. Agent Substrate boots each template once as a
golden actor, so the runner never starts by itself. It waits for `POST /ousast/v1/start`, which
the reconciler sends through `atenet-router` to that actor alone. The key is read from the
operator's environment or `.env`, kept in memory by the runner, passed to the command and
redacted from echoed stderr; no manifest, rendered Task or file carries it.
- Per-task egress. One EgressPolicy per actor through the `atenet-egress` gateway, deny by
default, hostnames only: plain HTTP to the artifact receiver, TLS passthrough to the Workspaces'
Git hosts and the Model's declared hosts. Deleted with the actor.
- Artifacts come back through the gateway. The runner posts its output directory as a tar to
the receiver, an HTTP server inside the reconciler process on the host, reached through a Service
with an EndpointSlice to the host. That delivery is the task's completion; ax has no Completed
phase.
- Memory on S3-compatible RustFS. `OUSAST_MEMORY=s3://sast-memory` on a RustFS server. The
store verifies the bucket at every start (versioning, lifecycle, tags, S3 Select) and refuses to
start when anything is missing; there is no local or fetch-and-filter fallback. Each row kind
has a fixed set of queryable fields, enforced at write time
([Memory](memory.md#verify-at-startup-no-fallback), [RustFS setup](rustfs.md)).
- **One Model and one budget per Task.** A Task binds at most one ax Model and
`budget: {usd, calls}`. The Models are `deepseek-flash` for chat and `openrouter-embedding` for
embeddings. The metered client refuses the next call at the ceiling. The task ends unfinished
and resumes on a rerun. Model-free tasks run with a zero budget, so a stray call fails loudly.
- **Credentials exist only in the start request.** Agent Substrate boots each template once as a
golden actor. So the runner never starts by itself. It waits for `POST /ousast/v1/start`. The
reconciler sends that request through `atenet-router` to that actor alone.
- The key is read from the operator's environment or `.env`.
- The runner keeps it in memory and passes it to the command.
- It is redacted from echoed stderr.
- No manifest, rendered Task or file carries it.
- **Per-task egress.** Each actor gets one EgressPolicy through the `atenet-egress` gateway. It is
deny by default and uses hostnames only. It allows plain HTTP to the artifact receiver. It allows
TLS passthrough to the Workspaces' Git hosts and the Model's declared hosts. It is deleted with
the actor.
- **Artifacts come back through the gateway.** The runner posts its output directory as a tar to
the receiver. The receiver is an HTTP server inside the reconciler process on the host. It is
reached through a Service with an EndpointSlice to the host. That delivery is the task's
completion; ax has no Completed phase.
- **Memory on S3-compatible RustFS.** The setting is `OUSAST_MEMORY=s3://sast-memory` on a RustFS
server.
- The store verifies the bucket at every start: versioning, lifecycle, tags, S3 Select.
- It refuses to start when anything is missing. There is no local or fetch-and-filter fallback.
- Each row kind has a fixed set of queryable fields, enforced at write time
([Memory](memory.md#verify-at-startup-no-fallback), [RustFS setup](rustfs.md)).
Lessons the host taught, each from a failed live run (the full list is on
[ax on this host](ops/ax/README.md#what-this-host-taught-each-one-cost-a-failed-live-run)):
- Substrate installs no WorkerPool, and actor images must be pinned by digest.
- A release `ax` CLI skews from the server; build it from the deployed checkout.
- A release `ax` CLI skews from the server. Build it from the deployed checkout.
- ax has no Completed phase, so completion is our artifact delivery.
- The golden boot means the runner must wait for a start request.
- The egress gateway dials the address the actor connected to, so the receiver is a Service
- The egress gateway dials the address the actor connected to. So the receiver is a Service,
dialled by ClusterIP with its name as Host.
- ax's snapshot bucket must exist in your S3 server. When it did not, deleted golden actors piled
up in `DELETING` and filled the disk during a 600-task harvest on 2026-10-01.
up in `DELETING`. They filled the disk during a 600-task harvest on 2026-10-01.
Deployment status, in the two words [Deployment](deployment.md) uses strictly. Implemented:
ax as the only executor, per-task EgressPolicy, credentials only in the start request, per-task
budgets and attribution, the S3 store verified at startup, local bring-up, doctor and smoke Run,
re-pinning generated Tasks to another registry. Planned, not done: deployment to any cluster
other than this host's kind cluster, a configurable kube context (`kind-` is hard-coded in
`doctor.py`, `egress.py`, `router.py`), the receiver as an in-cluster Deployment, the
reconciler in-cluster with Secrets injected, presigned-URL delivery to the store.
Deployment status uses two words, as [Deployment](deployment.md) does, strictly.
Implemented:
- ax as the only executor;
- per-task EgressPolicy;
- credentials only in the start request;
- per-task budgets and attribution;
- the S3 store verified at startup;
- local bring-up, doctor and smoke Run;
- re-pinning generated Tasks to another registry.
Planned, not done:
- deployment to any cluster other than this host's kind cluster;
- a configurable kube context (`kind-` is hard-coded in `doctor.py`, `egress.py`, `router.py`);
- the receiver as an in-cluster Deployment;
- the reconciler in-cluster with Secrets injected;
- presigned-URL delivery to the store.
## 4. Token economics, before and after
Before the plane, the script pipeline cost $0.022 per candidate for two verify passes
(`benchmarks/independent/verifier-batched-check-2026-09-29.json`), and the maintainers' own
coding sessions spent about 90% of their tokens on tool input and output rather than reasoning.
After it, the per-candidate costs below are what the records show. The mechanisms (one budget
per Task, pay-once reuse, cheap-before-expensive ordering, replay at $0) are on
[Token ergonomics](token-ergonomics.md); this table is the before and after.
Before the plane, two costs stood out:
- The script pipeline cost $0.022 per candidate for two verify passes
(`benchmarks/independent/verifier-batched-check-2026-09-29.json`).
- The maintainers' own coding sessions spent about 90% of their tokens on tool input and output
rather than reasoning.
After it, the per-candidate costs below are what the records show. The mechanisms are on
[Token ergonomics](token-ergonomics.md): one budget per Task, pay-once reuse,
cheap-before-expensive ordering and replay at $0. This table is the before and after.
| Item | Before | After | Record |
| --- | --- | --- | --- |
@@ -195,63 +273,70 @@ per Task, pay-once reuse, cheap-before-expensive ordering, replay at $0) are on
| Harvest of plane signals for the engine | none existed | `verify` $2.5373 over 1,961 calls and 474 tasks; `roles` $1.4036 over 1,034 calls and 151 tasks | `2026-10-01-decision-engine-harvest/record.json` |
| Development transcript, tool I/O share | 90% | 85.5% over 2,701 tool calls (`dev_token_report`) | `plane-increment-1.json` |
Two notes from the records. The meter prices every call at the Model's list price
(`plane/models/deepseek-flash.yaml`); the DeepSeek account moved less, about one third of the
metered amount on the injection slice ($0.30 for $0.91 metered), and budgets and gates are set
against the meter, never against the balance. And the development share barely moved: 90 to 85.5
is a small gain, and the remedies (a read guard, a bulk reader, scripts in files) target file
contents because that is where the tokens go.
Two notes from the records:
1. The meter prices every call at the Model's list price (`plane/models/deepseek-flash.yaml`). The
DeepSeek account moved less: about one third of the metered amount on the injection slice
($0.30 for $0.91 metered). Budgets and gates are set against the meter, never against the
balance.
2. The development share barely moved: 90 to 85.5 is a small gain. The remedies are a read guard,
a bulk reader and scripts in files. They target file contents, because that is where the
tokens go.
## 5. Open items
Five groups, each item with its record: the engine's own gates, data, measurement debt,
infrastructure, and instrument gaps.
There are five groups. Each item has its record. The groups are the engine's own gates, data,
measurement debt, infrastructure, and instrument gaps.
The engine's gates.
- BLOCK is reachable for no family: calibration fails for five (ECE 0.071 to 0.119) and the
sixth reaches no fold with a Wilson lower bound of 0.95 on precision (slice-2 record).
- BLOCK is reachable for no family. Calibration fails for five (ECE 0.071 to 0.119). The sixth
reaches no fold with a Wilson lower bound of 0.95 on precision (slice-2 record).
- Re-run agreement is below 0.9 for three families: injection 0.77, access control 0.83,
untrusted destination 0.87. exp-002 tried the obvious variance fix and lost on the ranking.
Data.
- Positives-only groups: 98 (family, group) pairs in the 904-example memory hold positive examples
and no negative. 94 of them exist because the fixed side no longer has the function (the vibe-py
pairs). No advisory-fixes group is among them
(`benchmarks/measurements/2026-10-02-decision-engine-negatives-gap/record.json`). There are no
single-label negatives, so every family's pooled AUC is inflated by positives that never had to
be told apart from their own fix (slice-2 record, `paired_groups_check`).
- The model has seen public fixes. A verify pass named the advisory's CVE id for a public fix, an
observation the engine's design specification (maintainer tooling) records as visible
memorisation. Excerpts now redact advisory ids and security-worded comments
(`src/openultrasast/learn/excerpt.py`), but labels drawn from public advisories are partly in the
model's training data, and the within-pair numbers carry that.
- PHP is covered by the engine on real code and by in-sample quick rules; the population that
and no negative.
- 94 of them exist because the fixed side no longer has the function (the vibe-py pairs).
- No advisory-fixes group is among them
(`benchmarks/measurements/2026-10-02-decision-engine-negatives-gap/record.json`).
- There are no single-label negatives. So every family's pooled AUC is inflated by positives
that never had to be told apart from their own fix (slice-2 record, `paired_groups_check`).
- The model has seen public fixes. A verify pass named the advisory's CVE id for a public fix.
The engine's design specification (maintainer tooling) records this as visible memorisation.
Excerpts now redact advisory ids and security-worded comments
(`src/openultrasast/learn/excerpt.py`). But labels drawn from public advisories are partly in
the model's training data, and the within-pair numbers carry that.
- PHP is covered by the engine on real code and by in-sample quick rules. The population that
would measure it (v3) is reserved.
Measurement debt.
- exp-001 (known callers) is registered and has not run; the model-free candidate generation on
the 41 development repositories that its units need is not built
- exp-001 (known callers) is registered and has not run. Its units need model-free candidate
generation on the 41 development repositories. That generation is not built
(`benchmarks/experiments/exp-001-known-callers/registration.json`).
- Leave-one-source-out, leave-one-framework-out and the memory-size curve points (0, 25, 50%)
have not run; one paired point exists (724 against 904 examples, injection, AUC 0.76 against 0.77).
- The plane-profile leak audit flagged `verify.flag_b` and `ms.operation`; they are withheld from
profile v1, and whether they track the label on unseen code has to be decided before any
plane-profile program runs (`2026-10-02-decision-engine-harvest-advisory/record.json`).
have not run. One paired point exists (724 against 904 examples, injection, AUC 0.76 against 0.77).
- The plane-profile leak audit flagged `verify.flag_b` and `ms.operation`. They are withheld from
profile v1. Before any plane-profile program runs, someone must decide whether they track the
label on unseen code (`2026-10-02-decision-engine-harvest-advisory/record.json`).
- Population v3 waits for `benchmarks/independent/prediction-v3.json`, which does not exist. It is
written only after a program is adopted.
Infrastructure.
- Deployment to a separate Kubernetes cluster is planned, not done; the code changes it needs are
- Deployment to a separate Kubernetes cluster is planned, not done. The code changes it needs are
listed with file paths on [Deployment](deployment.md#what-is-implemented-and-what-is-planned).
Instrument gaps.
- A missing provider key is a quiet path in a scan. The plane refuses to start a Task whose Model
names an unset variable (`plane/router.py`, `StartError`). A scan without a key asks the model
nothing, reports only the graph's entailed findings, and records `learning_endpoint_unavailable`
in `manifest.json` (`cli.py`, after `resolve_chat_endpoint` returns nothing); `report.md` has no
line for that reason yet, so the report does not say the model layer was skipped.
- A missing provider key is a quiet path in a scan.
- The plane refuses to start a Task whose Model names an unset variable (`plane/router.py`,
`StartError`).
- A scan without a key asks the model nothing and reports only the graph's entailed findings.
It records `learning_endpoint_unavailable` in `manifest.json` (`cli.py`, after
`resolve_chat_endpoint` returns nothing).
- `report.md` has no line for that reason yet. So the report does not say the model layer was
skipped.