From 552671ca6a7ee2bc8b5d8799c0ca6b3ea3a4835d Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Fri, 2 Oct 2026 21:01:10 +0200 Subject: [PATCH] docs: plain-language pass -- README, index, where we stand, scanning, examples, threat model Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo --- README.md | 382 +++++++++++++++++++++-------------- docs/examples.md | 45 +++-- docs/index.md | 42 ++-- docs/scanning.md | 92 +++++---- docs/threat-model.md | 127 +++++++----- docs/where-we-stand.md | 437 ++++++++++++++++++++++++----------------- 6 files changed, 685 insertions(+), 440 deletions(-) diff --git a/README.md b/README.md index 24834bf..022c82d 100644 --- a/README.md +++ b/README.md @@ -1,38 +1,52 @@ # OpenUltraSAST -OpenUltraSAST is an experimental static security analyser meant to work as a safety net on -repositories it has never seen: it scans a checkout, or the commits a `git push` would publish, -and separates what it can prove from what it merely suspects. A scan combines language-scoped -pattern rules, a tree-sitter overlay, a Joern code property graph and, optionally, a model that is -asked only the question the graph cannot settle; every finding carries the evidence that earned -its place. A scan that could not look at something records a degradation instead of reporting a -clean result. `ousast` is the interface; model work over whole repositories runs on a separate -agentic plane (google/ax), and a learned decision engine is in development. Neither is needed -for a scan. +OpenUltraSAST is an experimental static security analyser. It is meant to be a safety net on +repositories it has never seen. It scans a checkout, or the commits a `git push` would publish. It +keeps what it can prove apart from what it only suspects. -This file is the front door; each section links to its page of the documentation site (`docs/`, +A scan combines up to four parts: + +- language-scoped pattern rules; +- a tree-sitter overlay (a syntax tree of each file, built by the tree-sitter parser); +- a Joern code property graph (a graph of the program's syntax, control flow and data flow); +- optionally, a model. It is asked only the question the graph cannot settle. + +Every finding carries the evidence that earned its place. When a scan could not look at +something, it records a degradation. It does not report a clean result. + +`ousast` is the interface. Model work over whole repositories runs on a separate agentic plane +(google/ax). A learned decision engine is in development. A scan needs neither of them. + +This file is the front door. Each section links to its page of the documentation site (`docs/`, `mkdocs.yml`; `uv sync --extra docs && uv run mkdocs serve`). ## Status -**v2.0.0, state as of 2026-10-02.** The pre-push safety net is **NO-GO** against every M4 gate: -no hook capability is enabled, and the two independent populations spent so far gave engine recall -1 of 11 and then 0 of 17. The learned decision engine ranks but may not block: BLOCK is offered for -no family. The project has stopped trying to reach the gates by hand; every detection change now -has to win a controlled measurement on held-out repositories. The one page that states all of this -with the record behind each figure is [docs/where-we-stand.md](docs/where-we-stand.md); the -release history is in [RELEASE_NOTES.md](RELEASE_NOTES.md). +**v2.0.0, state as of 2026-10-02.** The pre-push safety net is **NO-GO** against every M4 gate. + +- No hook capability is enabled. +- Two independent populations (case sets frozen before any scan) have been spent so far. Engine + recall was 1 of 11 and then 0 of 17. +- The learned decision engine ranks but may not block. BLOCK is offered for no family. + +The project has stopped trying to reach the gates by hand. Every detection change now has to win +a controlled measurement on held-out repositories. One page states all of this, with the record +behind each figure: [docs/where-we-stand.md](docs/where-we-stand.md). The release history is in +[RELEASE_NOTES.md](RELEASE_NOTES.md). ## Install and first scan -Which tier you need: a deterministic scan of a checkout or a CI gate is tier 1; the Joern engine -(`standard`, `deep`) is tier 2; running model work over whole repositories on the plane is tier 3, -for maintainers; tier 4 does not exist yet. +Pick the tier you need: + +1. Tier 1: a deterministic scan of a checkout, or a CI gate. +2. Tier 2: the Joern engine (`standard`, `deep`). +3. Tier 3, for maintainers: model work over whole repositories on the plane. +4. Tier 4 does not exist yet. ### Tier 1: quick scan, no Docker The core install depends only on PyYAML. From a checkout, `uv sync` (or `pip install .`) installs -the `ousast` entry point; `uv sync --group dev` adds the test and lint tools. +the `ousast` entry point. `uv sync --group dev` adds the test and lint tools. ```bash uv sync @@ -41,19 +55,26 @@ uv run ousast scan . --mode quick --fail-on verified # CI: exit 1 on any evide ``` This runs the language-scoped pattern rules, entry-point reachability hints, ranking, -verification and scoring: deterministic, reproducible, no network, no keys. What it cannot do: -there is no engine, so nothing follows a value across statements or files, and the PHP quick rules -are a low-precision, in-sample development set (per-rule precision lower bounds from 0.03 to 1.0 on -the real pins, `benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`). The optional -`semantic` extra (`uv sync --extra semantic`) installs the tree-sitter grammars for the overlay. +verification and scoring. It is deterministic and reproducible. It needs no network and no keys. + +What it cannot do: + +- There is no engine. Nothing follows a value across statements or files. +- The PHP quick rules are a low-precision, in-sample development set. Per-rule precision lower + bounds run from 0.03 to 1.0 on the real pins + (`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`). + +The optional `semantic` extra (`uv sync --extra semantic`) installs the tree-sitter grammars for +the overlay. ### Tier 2: standard and deep scans (Docker) -`standard` and `deep` need Joern. The Docker image `openultrasast:dev` ships the tool, Joern -v4.0.625 and `php-cli` (needed by the PHP frontend) in one image; it is built locally from the -`Dockerfile` (no published image yet), and it is big: a JRE plus about 2 GB of Joern. The shell -wrapper builds it on first use and then runs `ousast` in the container with your directory mounted -read-only and the container network off: +`standard` and `deep` need Joern. The Docker image `openultrasast:dev` ships three things in one +image: the tool, Joern v4.0.625 and `php-cli` (the PHP frontend needs it). You build it locally +from the `Dockerfile`; there is no published image yet. It is big: a JRE plus about 2 GB of Joern. + +The shell wrapper builds the image on first use. Then it runs `ousast` in the container. Your +directory is mounted read-only and the container network is off. ```bash source /path/to/OpenUltraSAST/ops/shell/ousast.sh # PowerShell: ops/shell/ousast.ps1 @@ -62,90 +83,129 @@ ousast-with-judge scan . --mode standard # network on, the model is a ``` `docker compose build` and `TARGET=/path/to/repo docker compose run --rm ousast scan /target` do -the same without the wrapper. `deep` adds REGRESS, which runs each reproduction snippet with -`docker run` from wherever the scan runs; the engine container has no Docker socket, so run `deep` -from a host install with Joern on the PATH (`pyinfra @local ops/joern.py`, pinned and checksum- -verified) and a working `docker`, or accept the recorded `sandbox_unavailable` degradation. +the same without the wrapper. + +`deep` adds REGRESS. REGRESS runs each reproduction snippet with `docker run` from wherever the +scan runs. The engine container has no Docker socket. So run `deep` in one of two ways: + +- from a host install with Joern on the PATH (`pyinfra @local ops/joern.py`, pinned and checksum- + verified) and a working `docker`; +- or accept the recorded `sandbox_unavailable` degradation. + Details, including the host install, are in [ops/README.md](ops/README.md). -Keys and store settings live in `.env` in the working directory (gitignored; `.env.example` lists -the names). `DEEPSEEK_API_KEY` serves every LLM call; `OPENROUTER_API_KEY` is used for embeddings -only (`ousast learn memory embed`). `.env` never overrides a variable already exported in your -shell, so `unset` a stale export when a key change seems to have no effect. The plane's memory -store defaults to a local file store under `~/ousast-results/plane/memory`; an S3-compatible bucket -is optional (`uv sync --extra s3`, then `OUSAST_MEMORY=s3://` with `S3_ENDPOINT` and the -AWS credentials in `.env`). Nothing in a scan needs it. +Keys and store settings live in `.env` in the working directory. The file is gitignored; +`.env.example` lists the names. + +- `DEEPSEEK_API_KEY` serves every LLM call. +- `OPENROUTER_API_KEY` is used for embeddings only (`ousast learn memory embed`). +- `.env` never overrides a variable already exported in your shell. If a key change seems to have + no effect, `unset` the stale export. +- The plane's memory store defaults to a local file store under `~/ousast-results/plane/memory`. +- An S3-compatible bucket is optional: `uv sync --extra s3`, then `OUSAST_MEMORY=s3://` + with `S3_ENDPOINT` and the AWS credentials in `.env`. Nothing in a scan needs it. ### Tier 3: the plane on ax (maintainers) -None of this is needed for tier 1 or 2. Today's profile is a single-node kind cluster on the -maintainer's host. `ops/ax/up.sh` needs `docker`, `go`, `kubectl`, `kind`, `ko` and `ax` on the -PATH (it refuses to start when one is missing) and builds Agent Substrate and ax from their -checkouts under `~/.cache/ousast/ax-src/`. Measured idle footprint with two workers: the kind node -takes 1.7 GiB of memory (27% of the 7.7 GiB host) and 4.9 GB of disk for images, with several GB -more of Go caches during the build; the worker pool is sized for that 7 GB host. The verify and -roles tasks need `DEEPSEEK_API_KEY`, and memory lives in the local file store or the S3 store -(RustFS is the tested server). Bring-up, doctor and the smoke Run: [ops/ax/README.md](ops/ax/README.md). +Tier 1 and 2 need none of this. Today's profile is a single-node kind cluster on the maintainer's +host. + +- `ops/ax/up.sh` needs `docker`, `go`, `kubectl`, `kind`, `ko` and `ax` on the PATH. It refuses to + start when one is missing. +- It builds Agent Substrate and ax from their checkouts under `~/.cache/ousast/ax-src/`. +- Measured idle footprint with two workers: the kind node takes 1.7 GiB of memory (27% of the + 7.7 GiB host) and 4.9 GB of disk for images. The build uses several GB more for Go caches. +- The worker pool is sized for that 7 GB host. +- The verify and roles tasks need `DEEPSEEK_API_KEY`. +- Memory lives in the local file store or the S3 store. RustFS is the tested server. + +Bring-up, doctor and the smoke Run: [ops/ax/README.md](ops/ax/README.md). ### Tier 4: Kubernetes (k3s), planned -No deployment to any cluster other than that kind cluster has been made. The target is two -profiles of one code base, `kind` locally and `k3s` in production, where the same `ousast plane run` -submits a Run in a remote execution mode: it needs a kubeconfig context or the cluster's ingress -(and an `imagePullSecret` on the worker ServiceAccount for a private package), runner and engine -images published to GHCR and pinned by digest, and the S3 store as the artifact path, with no -receiver on a laptop. The plan, and the code changes it needs with file paths, are in +Nothing has been deployed to any cluster other than that kind cluster. The target is two profiles +of one code base: `kind` locally and `k3s` in production. The same `ousast plane run` would +submit a Run in a remote execution mode. It needs: + +- a kubeconfig context or the cluster's ingress (and an `imagePullSecret` on the worker + ServiceAccount for a private package); +- runner and engine images published to GHCR and pinned by digest; +- the S3 store as the artifact path, with no receiver on a laptop. + +The plan, and the code changes it needs with file paths, are in [docs/deployment.md](docs/deployment.md#production-topology-ax-on-a-kubernetes-cluster). ## Scan modes -`quick` runs the pattern rules, hints, ranking, verification and score and needs nothing. -`standard` adds the MAP stage: complexity map, semantic overlay, authorization obligations and the -Joern model layer, which builds one code property graph per repository and decides taint, -guard-dominance and configuration questions on it; a model is asked only the residual question -(without a key it is skipped and only the graph's entailed findings are reported). `deep` adds -REGRESS: promoted candidates are loaded by a small snippet in a Docker sandbox and get a -`triggerable` or `not_triggerable` verdict. A missing engine is a recorded degradation -(`cpg_unavailable`, `sandbox_unavailable`), never a clean result. +There are three modes. Each adds to the one before. + +- `quick` runs the pattern rules, hints, ranking, verification and score. It needs nothing. +- `standard` adds the MAP stage: complexity map, semantic overlay, authorization obligations and + the Joern model layer. The model layer builds one code property graph per repository. It decides + taint, guard-dominance and configuration questions on that graph. A model is asked only the + residual question. Without a key, that question is skipped and only the graph's entailed + findings are reported. +- `deep` adds REGRESS. Promoted candidates are loaded by a small snippet in a Docker sandbox. Each + gets a `triggerable` or `not_triggerable` verdict. + +A missing engine is a recorded degradation (`cpg_unavailable`, `sandbox_unavailable`), never a +clean result. + +`--fail-on never|findings|verified|worth-fixing` sets the exit code. `--config openultrasast.toml` +loads settings. Every run writes these files under `/.openultrasast/runs//`: +`manifest.json`, `findings.json`, `verification.json`, `score.json`, `report.md`, `report.sarif` +and `trace/events.jsonl`. + +`ousast pre-push` (experimental) analyses the commits a push would publish against their base. +Only new or worsened defects become candidates. It is advisory by default. Its capability registry +is empty, so it emits no normal alert today. -`--fail-on never|findings|verified|worth-fixing` sets the exit code and `--config openultrasast.toml` -loads settings. Every run writes `manifest.json`, `findings.json`, `verification.json`, -`score.json`, `report.md`, `report.sarif` and `trace/events.jsonl` under -`/.openultrasast/runs//`. `ousast pre-push` (experimental) analyses the commits a -push would publish against their base so that only new or worsened defects are candidates; it is -advisory by default and its capability registry is empty, so no normal alert is emitted today. Flows and the per-language rule table: [docs/scanning.md](docs/scanning.md). ## How detection works -Three representations, from cheap to expensive: text (regex quick rules, one statement at a time), -tree (the tree-sitter CST and an IR per function, for the overlay and the obligations) and graph -(the Joern CPG, the only one that follows a value across statements, calls and files). Taint runs -as fact tables, queries and verdicts on the graph; absence bugs (an operation that should have -been guarded) are found as obligations, dominance and configuration census; a model answers only -the residual question. [docs/detection-techniques.md](docs/detection-techniques.md) explains each -technique with the code it comes from and what it misses; -[docs/architecture.md](docs/architecture.md) gives the stage order, the evidence ladder (a state -machine, not a label a model may assign), the central CWE policy and the project score, and the -self-improving loops (`ousast improve`, with `--memory` for proposals from plane rows). +The tool uses three representations of code, from cheap to expensive: -Quick rules cover C/C++, Python, JavaScript/TypeScript, Java with Groovy templates, and PHP, and -run only against files of their language. The detection gate (`uv run python -m openultrasast.gate`) -enforces at least 90% recall and under 10% false positives per language on the bundled cheat-sheet -corpora; on this tree it reports 93.62% recall (44/47) and 0.0% false positives. Those corpora show -that a rule change did not break a known detection; they are not a recall estimate on real code, -and PHP is not in the gate. Framework knowledge (WordPress, Flask, Django, Express, Spring) is -tagged as an optional prior in `src/openultrasast/ruleset/frameworks.toml`, so its contribution can -be measured and switched off. +1. Text: regex quick rules, one statement at a time. +2. Tree: the tree-sitter CST (concrete syntax tree) and an IR (intermediate representation) per + function. The overlay and the obligations use it. +3. Graph: the Joern CPG. It is the only one that follows a value across statements, calls and + files. + +Taint runs as fact tables, queries and verdicts on the graph. Absence bugs are operations that +should have been guarded. They are found as obligations, dominance and configuration census. A +model answers only the residual question. + +- [docs/detection-techniques.md](docs/detection-techniques.md) explains each technique, with the + code it comes from and what it misses. +- [docs/architecture.md](docs/architecture.md) gives the stage order and the evidence ladder (a + state machine, not a label a model may assign). It also covers the central CWE policy, the + project score and the self-improving loops (`ousast improve`, with `--memory` for proposals from + plane rows). + +Quick rules cover C/C++, Python, JavaScript/TypeScript, Java with Groovy templates, and PHP. Each +rule runs only against files of its language. + +The detection gate (`uv run python -m openultrasast.gate`) enforces at least 90% recall and under +10% false positives per language on the bundled cheat-sheet corpora. On this tree it reports +93.62% recall (44/47) and 0.0% false positives. Those corpora show that a rule change did not +break a known detection. They are not a recall estimate on real code, and PHP is not in the gate. + +Framework knowledge (WordPress, Flask, Django, Express, Spring) is tagged as an optional prior in +`src/openultrasast/ruleset/frameworks.toml`. So its contribution can be measured and switched off. ## The plane on ax -Model-driven work over whole repositories runs as a Run manifest, a DAG of ax Tasks, each in its -own gVisor actor on Agent Substrate. ax is the only agentic executor; there is no local -subprocess path. Each Task binds at most one Model and its own `usd`/`calls` budget, egress is -deny-by-default per Task (hostnames only), and the provider key travels only in the Task's start -request, never in a manifest or an image. Measured on the first increment: the plane agreed on 16 -of 20 declared sites at $0.0205 per candidate. +Model-driven work over whole repositories runs as a Run manifest. A Run is a DAG of ax Tasks. Each +Task runs in its own gVisor actor (an isolated sandbox) on Agent Substrate. ax is the only agentic +executor; there is no local subprocess path. + +- Each Task binds at most one Model and its own `usd`/`calls` budget. +- Egress is deny-by-default per Task, by hostname only. +- The provider key travels only in the Task's start request, never in a manifest or an image. + +Measured on the first increment: the plane agreed on 16 of 20 declared sites at $0.0205 per +candidate. ```bash uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image @@ -155,42 +215,61 @@ uv run ousast plane remember validation-46 # ingest the run's rows ``` `workspaces`, `harvest`, `alerts-engine` and `memory-normalise` generate Runs, their inputs and -the store's rewrites. The flow and the task catalogue: [docs/plane.md](docs/plane.md); what runs -where today and what a separate cluster would take: [docs/deployment.md](docs/deployment.md). +the store's rewrites. The flow and the task catalogue: [docs/plane.md](docs/plane.md). What runs +where today, and what a separate cluster would take: [docs/deployment.md](docs/deployment.md). -HarnessX, the earlier agentic extra, was retired 2026-09-30: a `[harnessx]` section is ignored -with one warning, and `[models] verifier`, `[fusion] panel_model` and `[fusion] decider_model` fail -naming the plane replacement, as do config keys nothing ever read. Fusion is deterministic. +HarnessX, the earlier agentic extra, was retired 2026-09-30. A `[harnessx]` section is ignored +with one warning. `[models] verifier`, `[fusion] panel_model` and `[fusion] decider_model` fail and +name the plane replacement. Config keys that nothing ever read fail the same way. Fusion is +deterministic. ## Memory -One store, keyed by repository and pin, holds what plane runs and the decision engine learned: -rows (facts, verdicts, unit costs, alerts, coverage, feature records) and content-addressed blobs -(excerpts, embeddings, cached responses, compiled programs). `OUSAST_MEMORY` selects the backend: -a local file store, or `s3://` on any S3-compatible server with versioning, lifecycle -rules, object tags and S3 Select; RustFS is the tested server. An admin sets the bucket up once; -the store verifies it at every start and refuses to run, naming each missing piece, rather than -fall back. Layout and the read and write paths: [docs/memory.md](docs/memory.md); the bucket -policy and admin steps: [docs/rustfs.md](docs/rustfs.md); how stored rows are meant to turn into -better decisions: [docs/memory-and-detection.md](docs/memory-and-detection.md). +One store holds what plane runs and the decision engine learned. It is keyed by repository and pin. +It holds two kinds of data: + +- rows: facts, verdicts, unit costs, alerts, coverage and feature records; +- content-addressed blobs: excerpts, embeddings, cached responses and compiled programs. + +`OUSAST_MEMORY` selects the backend. It is either a local file store or `s3://` on any +S3-compatible server. The S3 server must support versioning, lifecycle rules, object tags and S3 +Select. RustFS is the tested server. + +An admin sets the bucket up once. The store verifies it at every start. If anything is missing, it +refuses to run and names each missing piece. It does not fall back. + +- Layout and the read and write paths: [docs/memory.md](docs/memory.md). +- The bucket policy and admin steps: [docs/rustfs.md](docs/rustfs.md). +- How stored rows are meant to turn into better decisions: + [docs/memory-and-detection.md](docs/memory-and-detection.md). ## Decision engine and evaluation -A learned decision engine is being built to replace hand-tuned detection edits: every instrument -produces signals, never verdicts; labels come from ground truth only; a compiled AI classifier with -local memory turns retrieved similar cases and the candidate's signals into a calibrated -probability with BLOCK and ADVISORY operating points. It is **in development, not adopted**: no -scan, pre-push check or report uses it. Measured on six families, on repositories the compiled -program was not shown, the pooled AUC of its score runs from 0.77 (injection) to 0.96 (access -control); the stricter within-pair AUC is 0.72 to 0.93; BLOCK is offered for no family -(`benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`). Maintainer surface: -`ousast learn labels|memory|compile|evaluate|curve|experiment|audit-leaks`. Design, slices and -the A/B protocol: [docs/decision-engine.md](docs/decision-engine.md). +A learned decision engine is being built. It is meant to replace hand-tuned detection edits. It +follows three rules: -How anything is measured on code the tool was not tuned on, the pair corpora, the independent -populations and the M4 gates, and the fold discipline that keeps learned decisions out of their -own training data: [docs/evaluation.md](docs/evaluation.md). Population v3 (PHP, 15 cases) is -frozen as the one-time final check and nothing in the tree may read it. +- Every instrument produces signals, never verdicts. +- Labels come from ground truth only. +- A compiled AI classifier with local memory turns retrieved similar cases and the candidate's + signals into a calibrated probability. It has BLOCK and ADVISORY operating points. + +It is **in development, not adopted**. No scan, pre-push check or report uses it. + +It was measured on six families, on repositories the compiled program was not shown: + +- The pooled AUC (area under the ROC curve) of its score runs from 0.77 (injection) to 0.96 + (access control). +- The stricter within-pair AUC is 0.72 to 0.93 across the same families. +- BLOCK is offered for no family + (`benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`). + +Maintainer surface: `ousast learn labels|memory|compile|evaluate|curve|experiment|audit-leaks`. +Design, slices and the A/B protocol: [docs/decision-engine.md](docs/decision-engine.md). + +[docs/evaluation.md](docs/evaluation.md) explains how anything is measured on code the tool was not +tuned on. It covers the pair corpora, the independent populations and the M4 gates. It also covers +the fold discipline that keeps learned decisions out of their own training data. Population v3 +(PHP, 15 cases) is frozen as the one-time final check. Nothing in the tree may read it. ```bash uv run ousast benchmark benchmarks/manifests/python-vulnerable.toml --mode quick @@ -201,19 +280,26 @@ uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dr ## Token economics -Model tokens are the one running cost. At runtime the plane routes them: one Model and one budget -per Task, a metered client that refuses the next call at the ceiling, repository facts computed -model-free and reused, blobs replayed at $0, triage before the hunt and a tie-break only on -disputed candidates. In development, 85.5% of the maintainers' coding-session tokens were tool -input and output rather than reasoning (down from 90%), and the remedies in `benchmarks/dev/` -target that. Mechanisms, -the measured costs and the before-and-after table: [docs/token-ergonomics.md](docs/token-ergonomics.md). +Model tokens are the one running cost. At runtime the plane routes them: + +- one Model and one budget per Task; +- a metered client that refuses the next call at the ceiling; +- repository facts computed without a model and reused; +- blobs replayed at $0, with no new model call; +- triage before the hunt; +- a tie-break only on disputed candidates. + +In development, 85.5% of the maintainers' coding-session tokens were tool input and output rather +than reasoning. That is down from 90%. The remedies in `benchmarks/dev/` target that share. +Mechanisms, the measured costs and the before-and-after table: +[docs/token-ergonomics.md](docs/token-ergonomics.md). ## Examples, threat model -End-to-end walkthroughs, from a quick scan to an MCP client: [docs/examples.md](docs/examples.md). -What the tool trusts and what it does not (scanned code is untrusted input, secret redaction, -per-task egress, budgets, the `deep` sandbox): [docs/threat-model.md](docs/threat-model.md). +- End-to-end walkthroughs, from a quick scan to an MCP client: [docs/examples.md](docs/examples.md). +- What the tool trusts and what it does not: [docs/threat-model.md](docs/threat-model.md). This + covers scanned code as untrusted input, secret redaction, per-task egress, budgets and the + `deep` sandbox. ## Contributing and maintainer tooling @@ -228,25 +314,33 @@ uv run python dagger/ci.py # containerized CI pipeline ``` An agent that can run shell commands needs nothing but the CLI. For [OpenCode](https://opencode.ai) -there are two project skills in `.agents/skills/`: `openultrasast-scan` (modes and their -prerequisites, `--fail-on`, the evidence ladder, recorded degradations, run artifacts) and -`openultrasast-triage` (false-positive reasons, what an adjudication may feed, fixing and -re-scanning). `ousast mcp` runs a narrow MCP server over stdio with ten tools (`openultrasast.scan`, -`status`, `findings`, `get_finding`, `evidence`, `artifacts`, `benchmark`, `explain`, -`propose_patch`, `export_report`); no tool runs a shell, Docker or a free-form command: +there are two project skills in `.agents/skills/`: + +- `openultrasast-scan`: modes and their prerequisites, `--fail-on`, the evidence ladder, recorded + degradations and run artifacts. +- `openultrasast-triage`: false-positive reasons, what an adjudication may feed, and fixing and + re-scanning. + +`ousast mcp` runs a narrow MCP (Model Context Protocol) server over stdio. It has ten tools: +`openultrasast.scan`, `status`, `findings`, `get_finding`, `evidence`, `artifacts`, `benchmark`, +`explain`, `propose_patch` and `export_report`. No tool runs a shell, Docker or a free-form +command. ```jsonc { "command": "uv", "args": ["run", "ousast", "mcp"] } ``` The `kiro-*` skills, `AGENTS.md` and the Kiro-style specifications under `.kiro/` are the -maintainer's development tooling and can be ignored by users. Other maintainer commands: -`ousast repos` (pinned known-vulnerable checkouts) and `ousast model candidates` (what the -candidate enumerator can reach). +maintainer's development tooling. Users can ignore them. Other maintainer commands: + +- `ousast repos`: pinned known-vulnerable checkouts. +- `ousast model candidates`: what the candidate enumerator can reach. ## Licence OpenUltraSAST is licensed under the [Apache License 2.0](LICENSE). Third-party material keeps its -own terms: Joern (Apache-2.0) is downloaded at image build time, the tree-sitter wheels of the -`semantic` extra are MIT, and the published benchmark cases were licence-checked one by one (v1's -OpenCVE case is BUSL-1.1, see [RELEASE_NOTES.md](RELEASE_NOTES.md)). +own terms: + +- Joern (Apache-2.0) is downloaded at image build time. +- The tree-sitter wheels of the `semantic` extra are MIT. +- The published benchmark cases were licence-checked one by one. v1's OpenCVE case is BUSL-1.1, see [RELEASE_NOTES.md](RELEASE_NOTES.md). diff --git a/docs/examples.md b/docs/examples.md index 6922646..ef41097 100644 --- a/docs/examples.md +++ b/docs/examples.md @@ -1,7 +1,7 @@ # End-to-end examples -Each example is self-contained. Artifacts land under -`/.openultrasast/runs//` and `.openultrasast/` is gitignored. +Each example stands on its own. Artifacts land under `/.openultrasast/runs//`. +`.openultrasast/` is gitignored. ## 1. Quick scan a local repo (deterministic, no keys) @@ -21,7 +21,7 @@ ousast scan . --mode standard # Joern model layer, container network off: ousast-with-judge scan . --mode standard # network on, so the model is asked (needs DEEPSEEK_API_KEY) ``` -Without a key the graph still decides: its entailed findings are reported, and the LLM's residual +Without a key, the graph still decides. Its entailed findings are reported. The LLM's residual and suspicion questions go unasked. ## 3. CI gate @@ -52,10 +52,11 @@ uv run ousast pairs --split holdout --profile human # filter by declared split uv run python -m openultrasast.pair_gate # CI: local pairs must all pass ``` -Each case is two isolated trees (vuln file vs patched file, same relative path). -`pair_correct` means the harness fired on the vuln side and stayed silent on the -fix. Misses and fix-side leaks become `miss`/`fp` signals for `ousast improve`. -The catalog and public dataset pointers live in `benchmarks/pairs/`. +Each case is two isolated trees: the vuln file and the patched file, at the same relative path. + +- `pair_correct` means the harness fired on the vuln side and stayed silent on the fix. +- Misses become `miss` signals and fix-side leaks become `fp` signals for `ousast improve`. +- The catalog and public dataset pointers live in `benchmarks/pairs/`. ## 6. Self-improve the ruleset from benchmark feedback @@ -68,10 +69,11 @@ uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml ## 7. Running the agentic plane -Model work over whole repositories runs as a Run manifest on ax, one isolated actor -per task (bring-up: [ops/ax/README.md](ops/ax/README.md)). The provider key is -read from the environment (or `.env`, which never overrides an exported variable) by -`ousast` and sent only in each task's start request. +Model work over whole repositories runs as a Run manifest on ax. Each task gets one isolated +actor. Bring-up is in [ops/ax/README.md](ops/ax/README.md). + +`ousast` reads the provider key from the environment, or from `.env`. `.env` never overrides an +exported variable. The key is sent only in each task's start request. ```bash uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image @@ -83,16 +85,25 @@ uv run ousast plane remember validation-46 # re-ingest the run's m uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run --memory ``` -Each task in the Run carries its own Model and `budget: {usd, calls}`; a task that -reaches its ceiling stops as `unfinished` and resumes on a rerun with a larger -budget. `ousast plane status` prints the token attribution table: calls, prompt, -cache-hit and output tokens and USD per task. +Each task in the Run carries its own Model and `budget: {usd, calls}`. + +- A task that reaches its ceiling stops as `unfinished`. It resumes on a rerun with a larger + budget. +- `ousast plane status` prints the token attribution table. It shows calls, prompt, cache-hit and + output tokens, and USD per task. ## 8. Drive it from an MCP client (OpenCode / IDE) +MCP (Model Context Protocol) lets an editor or agent call the tool's functions. + ```jsonc { "command": "uv", "args": ["run", "ousast", "mcp"] } ``` -Then: `openultrasast.scan {path}` → `run_dir` → `openultrasast.findings {run_dir}` → -`openultrasast.explain {run_dir, finding_id}`. See the README section "Agent integrations". +Then call the tools in this order: + +1. `openultrasast.scan {path}` returns a `run_dir`. +2. `openultrasast.findings {run_dir}` lists the findings. +3. `openultrasast.explain {run_dir, finding_id}` explains one. + +See the README section "Agent integrations". diff --git a/docs/index.md b/docs/index.md index 57433e6..b6511dc 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,13 +1,13 @@ # OpenUltraSAST -OpenUltraSAST is an experimental static security analyser meant to work as a safety net on -repositories it has never seen: it scans a checkout, or the commits a `git push` would publish, -and separates what it can prove from what it merely suspects. A scan that could not look at -something records a degradation instead of reporting a clean result. +OpenUltraSAST is an experimental static security analyser. It is meant to be a safety net on +repositories it has never seen. It scans a checkout, or the commits a `git push` would publish. It +keeps what it can prove apart from what it only suspects. When a scan could not look at +something, it records a degradation. It does not report a clean result. **State as of 2026-10-02 (v2.0.0):** the pre-push safety net is **NO-GO** for rollout. -No hook capability is enabled, and no independent population has passed the qualification gates -(see [Evaluation](evaluation.md)). +No hook capability is enabled. No independent population (a case set frozen before any scan) has +passed the qualification gates (see [Evaluation](evaluation.md)). ## Three layers @@ -17,6 +17,9 @@ No hook capability is enabled, and no independent population has passed the qual | **The agentic plane on ax** | Model-driven work over whole repositories, run as a DAG of tasks on google/ax, one isolated actor per task. What the runs learn is kept in a memory store. Maintainer surface: `ousast plane`. | [The plane on ax](plane.md), [Memory](memory.md) | | **The decision engine** | A learned classifier that weighs every instrument's signals against similar labelled cases from memory. **In development, not adopted**: no scan, pre-push check or report uses it. Maintainer surface: `ousast learn`. | [Memory and detection](memory-and-detection.md), [Decision engine](decision-engine.md) | +A code property graph is a graph of the program's syntax, control flow and data flow. Joern builds +it. + ## How they connect ```mermaid @@ -33,31 +36,34 @@ flowchart LR engine -. "intended, not adopted: BLOCK / ADVISORY" .-> report ``` -- The **scanners** produce findings and, inside plane Runs, the `alerts` that become memory rows. -- The **plane** runs model work (verify passes, roles) and model-free bookkeeping (facts, agreement, - feature records) per case; its `remember` task turns a case's artifacts into rows of the - [memory store](memory.md). -- Memory feeds two consumers. `ousast improve --memory` turns stored rows into rule-status - proposals that pass through the same validator and gate as every other edit. The decision engine - retrieves labelled examples from the same store. +- The **scanners** produce findings. Inside plane Runs, they also produce the `alerts` that become + memory rows. +- The **plane** runs model work per case: verify passes and roles. It also runs model-free + bookkeeping per case: facts, agreement and feature records. Its `remember` task turns a case's + artifacts into rows of the [memory store](memory.md). +- Memory feeds two consumers: + 1. `ousast improve --memory` turns stored rows into rule-status proposals. They pass through the + same validator and gate as every other edit. + 2. The decision engine retrieves labelled examples from the same store. - The decision engine's output (dotted line) is the intended path. Nothing user-facing reads it today. ## Where to go next -- The current state in one page, every figure with its record: capability against the M4 gates, the - step-by-step approach, ax on the server, token economics before and after, and the open items: - [Where we stand](where-we-stand.md). +- The current state in one page, every figure with its record: [Where we stand](where-we-stand.md). + It covers capability against the M4 gates, the step-by-step approach, ax on the server, token + economics before and after, and the open items. - Run a scan: [Examples](examples.md), and the commands in [Scanning](scanning.md). - Understand a finding's evidence and score: [Architecture](architecture.md). - The program analysis under a scan, what each representation can see and what it misses: [Detection techniques](detection-techniques.md). - Operate the plane and its store: [ax on this host](ops/ax/README.md) and [RustFS setup](rustfs.md). -- Where the plane runs today and what a separate Kubernetes cluster would take (planned, not +- Where the plane runs today, and what a separate Kubernetes cluster would take (planned, not done): [Deployment](deployment.md). - Where the model spend goes, per Task at runtime and in the maintainers' own coding sessions, and what is reused instead of paid again: [Token ergonomics](token-ergonomics.md). - Trust boundaries and hardening: [Threat model](threat-model.md). -Every number in these pages is quoted from a committed record whose path is given next to it. +Every number in these pages is quoted from a committed record. The record's path is given next to +the number. diff --git a/docs/scanning.md b/docs/scanning.md index 66cd588..2dcc14c 100644 --- a/docs/scanning.md +++ b/docs/scanning.md @@ -1,13 +1,21 @@ # Scanning -`ousast scan PATH --mode quick|standard|deep` scans a checkout; `ousast pre-push` checks only -what a push (or an explicit `--base`/`--head` pair) changes. The stage-by-stage pipeline, -evidence ladder and score are in [Architecture](architecture.md); this page shows the flows. +This page shows the scan flows. + +- `ousast scan PATH --mode quick|standard|deep` scans a checkout. +- `ousast pre-push` checks only what a push changes, or what an explicit `--base`/`--head` pair + changes. + +The stage-by-stage pipeline, the evidence ladder and the score are in +[Architecture](architecture.md). ## The three scan modes -The stages a mode requests are `MODE_STAGES` in `src/openultrasast/stages.py`: `quick` runs -STATIC, `standard` adds MAP, `deep` adds REGRESS. +The stages each mode requests are listed in `MODE_STAGES` in `src/openultrasast/stages.py`: + +- `quick` runs STATIC. +- `standard` adds MAP. +- `deep` adds REGRESS. ```mermaid flowchart TD @@ -38,7 +46,7 @@ Sources: `README.md` (scan modes), `src/openultrasast/cli.py` (`run_stage` calls ## Quick rules per language -Rules run only against files of their language. The bundled ruleset lives in +Each rule runs only against files of its language. The bundled ruleset lives in `src/openultrasast/ruleset//rules.toml`. | Language | Classes (CWE) | @@ -49,21 +57,31 @@ Rules run only against files of their language. The bundled ruleset lives in | Java + Groovy templates | command injection (78), SQLi (89), weak hash (327), deserialization (502), unescaped template XSS (79) | | PHP | SQLi (89), command injection (78), file inclusion (98), path (22), open redirect (601), SSRF (918); code eval (95), unserialize (502) and echo XSS (79) ship as `shadow` rules | -The detection gate (`python -m openultrasast.gate`) enforces at least 90% recall and under 10% -false positives per language on the bundled corpora in `benchmarks/manifests/`. Those corpora are -cheat-sheet fixtures: they catch regressions, they do not estimate recall on real code. **PHP is -not in the gate**; its numbers are in-sample development numbers -(`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`). See `README.md` for the -gate's per-language output. +The detection gate (`python -m openultrasast.gate`) enforces two limits per language on the +bundled corpora in `benchmarks/manifests/`: at least 90% recall and under 10% false positives. + +- Those corpora are cheat-sheet fixtures. They catch regressions. They do not estimate recall on + real code. +- **PHP is not in the gate**. Its numbers are in-sample development numbers + (`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`). + +See `README.md` for the gate's per-language output. ## The Joern engine The model layer (`src/openultrasast/model/`, `src/openultrasast/cpg/`) is the core of `standard`. -It builds one code property graph per scan with Joern (`joern-parse`, then batched -`joern --script` queries from `cpg/queries/*.sc`) and sends each question to the arbiter of its -type: **taint** (source to sink with no discharging sanitizer), **dominance** (does a guard govern -the obligated operation) and **configuration** (constant abstraction over a value). Taint facts -ship for C, Java, JavaScript, PHP and Python (`src/openultrasast/ruleset/semantic/*.toml`). +It builds one code property graph per scan with Joern. A code property graph is a graph of the +program's syntax, control flow and data flow. The build runs `joern-parse`, then batched +`joern --script` queries from `cpg/queries/*.sc`. + +The layer sends each question to the arbiter (the decider) of its type: + +- **taint**: a flow from source to sink with no discharging sanitizer; +- **dominance**: does a guard govern the obligated operation; +- **configuration**: constant abstraction over a value. + +Taint facts ship for C, Java, JavaScript, PHP and Python +(`src/openultrasast/ruleset/semantic/*.toml`). ```mermaid flowchart LR @@ -79,25 +97,33 @@ flowchart LR rung -- "residual question only" --> judge["LLM judge: confirm or contradict, never create a flow"] ``` -The Docker image ships Joern and `php-cli` (which the PHP frontend needs); without Joern the -layer records `cpg_unavailable` and the scan is otherwise unchanged. Engine installation and the -Joern features used are in [Engine and pre-push](ops/README.md). -How the graph is built, queried and refined, with the records behind each refinement, is in -[Detection techniques](detection-techniques.md). +The Docker image ships Joern and `php-cli`. The PHP frontend needs `php-cli`. Without Joern, the +layer records `cpg_unavailable`; the rest of the scan is unchanged. + +- Engine installation and the Joern features used: [Engine and pre-push](ops/README.md). +- How the graph is built, queried and refined, with the records behind each refinement: + [Detection techniques](detection-techniques.md). ## Framework priors -Rule and taint-fact entries that know a framework or library carry a `framework` or `library` tag -naming a row of `src/openultrasast/ruleset/frameworks.toml` (for example `wordpress`, `flask`, -`django`, `express`, `spring`). The loaders take `priors=`: `"all"` (the default and today's -behaviour), `"off"` (language-level knowledge only) or a set of ids, so the contribution of -framework knowledge can be measured and switched off. The decision engine's first slice ran with -priors off ([Decision engine](decision-engine.md#first-measured-slice-2026-10-01)). +Some rule and taint-fact entries know a framework or library. They carry a `framework` or +`library` tag. The tag names a row of `src/openultrasast/ruleset/frameworks.toml` (for example +`wordpress`, `flask`, `django`, `express`, `spring`). + +The loaders take `priors=` with one of three values: + +- `"all"`: the default and today's behaviour; +- `"off"`: language-level knowledge only; +- a set of ids. + +So the contribution of framework knowledge can be measured and switched off. The decision +engine's first slice ran with priors off +([Decision engine](decision-engine.md#first-measured-slice-2026-10-01)). ## Pre-push: the delta check (experimental) `ousast pre-push` analyses the commits a `git push` would publish, or an explicit -`--base`/`--head` pair, and compares head with base so that only new or worsened defects are +`--base`/`--head` pair. It compares head with base. Only new or worsened defects become candidates. Sources: `ousast pre-push --help`, `src/openultrasast/push/`, [Engine and pre-push](ops/README.md#explicit-local-replay). @@ -118,10 +144,10 @@ flowchart TD exit -- "blocking, explicit opt-in" --> block["May block the push"] ``` -- `--deadline` (30 s by default) bounds preparation, both revisions and reporting together; - `--incomplete-coverage block` opts into failing when coverage is incomplete. -- The default capability registry is empty pending independent qualification, so no normal alert - is emitted today; every candidate stays diagnostic. +- `--deadline` (30 s by default) bounds preparation, both revisions and reporting together. +- `--incomplete-coverage block` opts into failing when coverage is incomplete. +- The default capability registry is empty until independent qualification passes. So no normal + alert is emitted today. Every candidate stays diagnostic. - A model is called only with an explicit `--model-config`. - Hook installation (`ops/install-pre-push`) and the private cache (`--cache-dir`) are documented in [Engine and pre-push](ops/README.md#experimental-pre-push-integration). diff --git a/docs/threat-model.md b/docs/threat-model.md index 7261399..0088e77 100644 --- a/docs/threat-model.md +++ b/docs/threat-model.md @@ -1,78 +1,101 @@ # Threat model & hardening -OpenUltraSAST analyzes **untrusted code**. This document states what it trusts, what -it does not, and the controls that keep a scan safe to run in CI and safe to share. +OpenUltraSAST analyzes **untrusted code**. This page states what it trusts and what it does not. +It also lists the controls that keep a scan safe to run in CI and safe to share. ## Trust boundaries -- **Scanned code is untrusted input.** Quick and standard modes never import, build or run - the target: quick reads source as text, and standard additionally parses it into a code - property graph with Joern (`joern-parse`, a parser, not a build) and may send code excerpts - to the configured model. Only `deep` mode executes target-derived code: its REGRESS stage - loads promoted candidates inside the Docker sandbox below, and it runs only when `docker - info` succeeds (otherwise the stage is skipped and recorded as `sandbox_unavailable`). -- **The ruleset and CWE policy are trusted, governed data.** They change only through - reviewed commits or the bounded self-improvement loop (`ousast improve`), which can - flip rule *status* and tune score constants but can never edit pattern text or the +- **Scanned code is untrusted input.** Quick and standard modes never import, build or run the + target. + - Quick mode reads source as text. + - Standard mode also parses it into a code property graph (a graph of the program's syntax, + control flow and data flow) with Joern. It uses `joern-parse`, a parser, not a build. + Standard mode may send code excerpts to the configured model. + - Only `deep` mode executes target-derived code. Its REGRESS stage loads promoted candidates + inside the Docker sandbox described below. It runs only when `docker + info` succeeds (otherwise the stage is skipped and recorded as `sandbox_unavailable`). +- **The ruleset and CWE policy are trusted, governed data.** They change in only two ways: + through reviewed commits, or through the bounded self-improvement loop (`ousast improve`). The + loop can flip rule *status* and tune score constants. It can never edit pattern text or the authoritative 0–5 severity. See [architecture.md](architecture.md#self-improvement). -- **Provider credentials are the operator's.** The core install (PyYAML only) makes no - network calls on its own. A local scan calls a model only in standard or deep mode, and - only when a key is available: the model layer asks the residual question the graph cannot - settle, and `[models] hunter` enables the tool hunter. LLM calls go to DeepSeek - (`DEEPSEEK_API_KEY`); OpenRouter (`OPENROUTER_API_KEY`) serves embeddings. Keys come from - the environment or a gitignored `.env`, which never overrides an exported variable. The - pre-push check calls a model only with an explicit `--model-config`. Agentic work runs on the plane (`ousast plane run`): `ousast` reads - the variable a task's Model names (`secretKey.key`) from the operator's environment or - `.env` and sends the key only in that task's start request, through Agent Substrate's - router to that task's actor. It is not baked into an image or task template, and it is - never logged or written to artifacts. +- **Provider credentials are the operator's.** The core install (PyYAML only) makes no network + calls on its own. + - A local scan calls a model only in standard or deep mode, and only when a key is available. + The model layer asks the residual question the graph cannot settle. `[models] hunter` + enables the tool hunter. + - LLM calls go to DeepSeek (`DEEPSEEK_API_KEY`). OpenRouter (`OPENROUTER_API_KEY`) serves + embeddings. + - Keys come from the environment or a gitignored `.env`. `.env` never overrides an exported + variable. + - The pre-push check calls a model only with an explicit `--model-config`. + - Agentic work runs on the plane (`ousast plane run`). `ousast` reads the variable a task's + Model names (`secretKey.key`) from the operator's environment or `.env`. It sends the key + only in that task's start request. The request goes through Agent Substrate's router to that + task's actor. The key is not baked into an image or task template. It is never logged or + written to artifacts. ## Controls ### Secret redaction Scan artifacts can quote source that contains live credentials. `redaction.py` masks -recognizable secret shapes (provider API keys, AWS/GitHub/Slack/Google tokens, bearer -tokens, URL-embedded credentials, PEM private keys, and `key = value` assignments for -sensitive names) before traces (`trace/events.jsonl`) and the markdown report are -written. On by default; disable with `[hardening] redact_secrets = false`. +recognizable secret shapes before traces (`trace/events.jsonl`) and the markdown report are +written. It masks: + +- provider API keys; +- AWS/GitHub/Slack/Google tokens; +- bearer tokens; +- URL-embedded credentials; +- PEM private keys; +- `key = value` assignments for sensitive names. + +Redaction is on by default. Turn it off with `[hardening] redact_secrets = false`. ### Plane egress -Each plane task runs in its own actor behind Agent Substrate's egress gateway, which -denies by default. The task's EgressPolicy (`src/openultrasast/plane/egress.py`) allows exactly three -things: plain HTTP to the artifact receiver, and TLS passthrough to the Git hosts of the -task's bound Workspaces and to the hosts its bound Model declares -(`openultrasast.io/egress-hosts`). There are no wildcards, no IP addresses and no TLS -interception. The policy is deleted with the actor. +Each plane task runs in its own actor behind Agent Substrate's egress gateway. The gateway denies +by default. The task's EgressPolicy (`src/openultrasast/plane/egress.py`) allows exactly three +things: + +1. plain HTTP to the artifact receiver; +2. TLS passthrough to the Git hosts of the task's bound Workspaces; +3. TLS passthrough to the hosts its bound Model declares (`openultrasast.io/egress-hosts`). + +There are no wildcards, no IP addresses and no TLS interception. The policy is deleted with the +actor. ### Cost & CI budgets -- **Agentic spend** is bounded per plane task by the Run's `budget: {usd, calls}`. The - metered client refuses the next call once either ceiling is reached (the task is - `unfinished` and resumes on a rerun with a larger budget). A usd budget on a Model - without prices is refused rather than metered as zero, and an account or - authentication error fails the task and starts nothing further. The standard-mode tool - hunter is bounded by the per-tier hunter budgets. +- **Agentic spend** is bounded per plane task by the Run's `budget: {usd, calls}`. + - The metered client refuses the next call once either ceiling is reached. The task is then + `unfinished`. It resumes on a rerun with a larger budget. + - A usd budget on a Model without prices is refused. It is not metered as zero. + - An account or authentication error fails the task and starts nothing further. + - The standard-mode tool hunter is bounded by the per-tier hunter budgets. - **Output size** is bounded by `[hardening] max_findings` (0 = unlimited). Truncation is - severity-ordered and disclosed as a `budget` degradation in the manifest. + severity-ordered. It is disclosed as a `budget` degradation in the manifest. ### Provider reliability -LLM/embedding calls retry transient failures (HTTP 429/5xx, connection errors, timeouts) -with exponential backoff; non-transient errors (e.g. 4xx, malformed JSON) fail fast. +LLM/embedding calls retry transient failures with exponential backoff. Transient failures are +HTTP 429/5xx, connection errors and timeouts. Non-transient errors fail fast, for example 4xx or +malformed JSON. ### Visible degradation & determinism -When an optional engine is missing (Joern: `cpg_unavailable`; Docker: `sandbox_unavailable`), -the scan falls back to its deterministic equivalent and records a `degradations` entry in the -manifest — it is never silently downgraded. Without a provider key the model layer asks no -question and reports only what the graph entailed. Runs are reproducible: fixed config, -artifact manifests, prompt hashes, and model identifiers are recorded. +A scan is never silently downgraded. When an optional engine is missing, the scan falls back to +its deterministic equivalent. It records a `degradations` entry in the manifest (Joern: +`cpg_unavailable`; Docker: `sandbox_unavailable`). + +Without a provider key, the model layer asks no question. It reports only what the graph +entailed. + +Runs are reproducible. The fixed config, artifact manifests, prompt hashes and model identifiers +are recorded. ### MCP surface -`ousast mcp` exposes only the ten narrow project tools. No tool runs arbitrary shell, -Docker, or internal hunter tools, and no tool accepts a free-form command argument. +`ousast mcp` exposes only the ten narrow project tools. No tool runs arbitrary shell, Docker or +internal hunter tools. No tool accepts a free-form command argument. ## Sandbox (deep mode) -The REGRESS stage (`regress/`, `sandbox/runner.py`) runs each snippet with `docker run` -and never mounts the host Docker socket: +The REGRESS stage (`regress/`, `sandbox/runner.py`) runs each snippet with `docker run`. It never +mounts the host Docker socket. | Control | Setting | |---|---| @@ -84,5 +107,5 @@ and never mounts the host Docker socket: | pids | `[sandbox] pids_limit`, default 512 | | timeout | `[sandbox] timeout_seconds`, default 300 s | -Before anything runs, a structural deny-list (`regress/safety.py`) rejects a snippet that -mentions sockets, curl, host networking or writes under `/workspace`. +A structural deny-list (`regress/safety.py`) checks each snippet before anything runs. It rejects +a snippet that mentions sockets, curl, host networking or writes under `/workspace`. diff --git a/docs/where-we-stand.md b/docs/where-we-stand.md index 9b98332..7a7a36d 100644 --- a/docs/where-we-stand.md +++ b/docs/where-we-stand.md @@ -1,33 +1,50 @@ # Where we stand -State as of 2026-10-02 (v2.0.0). This page answers five questions in one place. How far is -detection from its gates. How does the project now try to close that gap. How does the model work -run on ax. What did the tokens cost before and after that move. What is still open. Every figure -is quoted from a committed record whose path stands next to it; detail tables stay on the pages -linked from each section. +State as of 2026-10-02 (v2.0.0). This page answers five questions in one place: -The goal has not changed: a safety net on repositories the tool has never seen, which says -BLOCK or ADVISORY about the code a push would change and stays silent on fixes and on benign -commits. Two independent populations have been spent against that goal, with engine recall 1 of +1. How far is detection from its gates? +2. How does the project now try to close that gap? +3. How does the model work run on ax? +4. What did the tokens cost before and after that move? +5. What is still open? + +Every figure is quoted from a committed record. The record's path stands next to the figure. +Detail tables stay on the pages linked from each section. + +The goal has not changed. The tool should be a safety net on repositories it has never seen. It +should say BLOCK or ADVISORY about the code a push would change. It should stay silent on fixes +and on benign commits. + +Two independent populations have been spent against that goal. An independent population is a +set of cases frozen before any scan. Engine recall was 1 of 11 and then 0 of 17. The hand edits that closed misses on the first population did not carry to -the second. So the question is no longer which rule to add next. It is how a change to detection -can be shown to work on code it was not tuned on, at a cost per candidate that can be paid for -every push. +the second. So the question is no longer which rule to add next. The question is this: how can a +change to detection be shown to work on code it was not tuned on? And can that be done at a cost +per candidate that can be paid for every push? -The answer this page records: the pre-push safety net is NO-GO against every M4 gate, and the -project has stopped trying to reach the gates by hand. Every detection change now has to win a -controlled measurement on held-out repositories, run on an agentic plane whose cost per candidate -is known. The pieces of that method exist and have been measured. None of them has produced a -BLOCK decision yet. +The answer this page records: + +- The pre-push safety net is NO-GO against every M4 gate. +- The project has stopped trying to reach the gates by hand. +- Every detection change now has to win a controlled measurement on held-out repositories. The + measurement runs on an agentic plane whose cost per candidate is known. +- The pieces of that method exist and have been measured. None of them has produced a BLOCK + decision yet. ## 1. Detection capability and goals: NO-GO, measured -No M4 gate is met on an independent population. The qualification milestone for the pre-push -safety net (M4) requires, together: at least 95% actionable precision, at least 90% supported -recall, at least 95% supported-check completion, the runtime gates, and zero fixed-side or benign -false alerts. A population is frozen before any -case is scanned, scored once under a pre-registered protocol, then spent; it can inform changes -but can qualify nothing afterwards ([Evaluation](evaluation.md#independent-populations-and-the-m4-gates)). +No M4 gate is met on an independent population. M4 is the qualification milestone for the +pre-push safety net. It requires all of these together: + +- at least 95% actionable precision; +- at least 90% supported recall; +- at least 95% supported-check completion; +- the runtime gates; +- zero fixed-side or benign false alerts. + +A population is frozen before any case is scanned. It is scored once under a pre-registered +protocol, then spent. After that it can inform changes but can qualify nothing +([Evaluation](evaluation.md#independent-populations-and-the-m4-gates)). | Gate | Required | Latest independent result | Record | | --- | --- | --- | --- | @@ -36,156 +53,217 @@ but can qualify nothing afterwards ([Evaluation](evaluation.md#independent-popul | Supported-check completion | >= 95% | 13,206 of 108,374 | `results-v2.json` | | Fixed-side and benign alerts | zero | 0 fixed-side, which the record calls met trivially since nothing was detected; 3 benign alerts | `results-v2.json` | -Three later measurements narrow the picture without changing the verdict. +Three later measurements narrow the picture. They do not change the verdict. -- The plane reproduces the script pipeline's agreement at a known cost. On the model-driven - pipeline's validation set (46 candidates, 15 cases of v2, 20 declared vulnerable sites), the - plane's verify passes a and b plus a tie-break on the 10 disputed candidates agreed on 16 of 20 - declared sites. The script reference agreed on 16 as well. The cost was $0.0205 per candidate - (`benchmarks/independent/plane-increment-2.json`). The record carries two caveats. The tie-break - was chosen after seeing the first measurement on the same set, so this compares plane with - script and qualifies nothing. And the passes flag fewer declared sites than the reference in - any pass, 17 against 19; that count is a correction in `plane-increment-1.json`, whose first - reading took agree's `declared_sites_matched` (19, coverage by candidates) for "found in either - pass". -- The learned decision engine ranks, but may not block. On six families, evaluated on - repositories the compiled program was not shown, the pooled AUC of its score runs from 0.77 - (injection) to 0.96 (access control). The stricter reading is the within-pair AUC, which asks - whether the vulnerable side of a fix scores above its own fixed side: 0.72 to 0.93. BLOCK is - offered for no family; calibration holds only for output encoding, and there no fold reaches - the required precision (`benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`). - The per-family table is on [Decision engine](decision-engine.md#second-slice-six-families-2026-10-02). -- PHP coverage is in-sample. The PHP quick rules were written and tightened while looking at - the pins and fixtures they are scored on, so the record says so itself (`provenance.in_sample` in - `benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`): a development check, not - a recall estimate. On real PHP code the Joern engine is the instrument. +- **The plane reproduces the script pipeline's agreement at a known cost.** The test used the + model-driven pipeline's validation set: 46 candidates, 15 cases of v2, 20 declared vulnerable + sites. The plane ran verify passes a and b, plus a tie-break on the 10 disputed candidates. It + agreed on 16 of 20 declared sites. The script reference agreed on 16 as well. The cost was + $0.0205 per candidate (`benchmarks/independent/plane-increment-2.json`). The record carries two + caveats: + 1. The tie-break was chosen after seeing the first measurement on the same set. So this compares + plane with script and qualifies nothing. + 2. The passes flag fewer declared sites than the reference in any pass: 17 against 19 (passes against reference). That + count is a correction in `plane-increment-1.json`. Its first reading took agree's + `declared_sites_matched` (19, coverage by candidates) to mean "found in either pass". +- **The learned decision engine ranks, but may not block.** It was evaluated on six families, on + repositories the compiled program was not shown. + - The pooled AUC (area under the ROC curve) of its score runs from 0.77 (injection) to 0.96 + (access control). + - The stricter reading is the within-pair AUC. It asks whether the vulnerable side of a fix + scores above its own fixed side: 0.72 to 0.93. + - BLOCK is offered for no family. Calibration holds only for output encoding. There, no fold + reaches the required precision + (`benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`). + - The per-family table is on + [Decision engine](decision-engine.md#second-slice-six-families-2026-10-02). +- **PHP coverage is in-sample.** The PHP quick rules were written and tightened while looking at + the pins and fixtures they are scored on. The record says so itself (`provenance.in_sample` in + `benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`). It is a development + check, not a recall estimate. On real PHP code, the Joern engine is the instrument. -What has not been touched: population v3, 15 PHP cases, frozen as the one-time final check. No -development or analysis work reads its cases, a test fails when anything in the tree names one -of its repositories, and it runs once, after a prediction is committed. +What has not been touched: population v3, 15 PHP cases, frozen as the one-time final check. + +- No development or analysis work reads its cases. +- A test fails when anything in the tree names one of its repositories. +- It runs once, after a prediction is committed. ## 2. Our approach to detection, step by step The path changed because hand edits did not transfer. Every gain before 2026-09-30 was a hand edit after a miss: a rule, a taint vocabulary entry, a guard, a prompt line, a tie-break. Each was -tuned on the cases that exposed it, and the tuning measured on population v1 did not carry to v2 -(1 of 11, then 0 of 17). The conclusions drawn: hand edits do not transfer. The data that could -teach a better decision cannot be produced by hand at the scale needed, so it has to be harvested -and labelled by machinery with a boundary around it. Framework knowledge (which function is a sink in which -framework) is useful as a prior but must stay optional, or the engine learns the vocabulary -instead of the code. A reader might object that a classifier is slower to show gains than the -next rule. It is. The difference is that its gains are measured where rules were not: on -repositories it never saw. +tuned on the cases that exposed it. The tuning measured on population v1 did not carry to v2 +(1 of 11, then 0 of 17). -The chain, one job per step. The design is on [Memory and detection](memory-and-detection.md); -this is the order of operations. +The conclusions drawn: -1. Instruments produce signals, never verdicts. Quick rules, the Joern engine, model sink - classification, the plane's verify passes a and b with the tie-break c, repository facts - (callers, spans) and model source/sink/sanitizer roles each write one block of a candidate's - record. None decides alone. -2. Feature records are allow-listed, and missing is not zero. A candidate is `(path, function, - family)` of one repository and pin. An instrument that did not run, had no coverage or failed - records that state; "silent" and "did not look" stay distinguishable. Input profile v1 withholds - the engine block and the function length, which a leak audit found separate the two sides of a - pair by construction (`benchmarks/measurements/2026-10-01-decision-engine-leak-audit/`), and - since 2026-10-02 the plane's `verify.flag_b` and `ms.operation`, flagged the same way by the - audit of the plane profile (`2026-10-02-decision-engine-harvest-advisory/record.json`, - `src/openultrasast/learn/schema.py`). -3. Labels come from ground truth only. A fail-closed allow-list (`src/openultrasast/learn/sources.toml`) - names the sources: the spent populations' declared sites and fix ranges, the vulnerable and - fixed sides of pairs, including the 97 real public fixes in `benchmarks/pairs/advisory-fixes/`, - benign controls and recorded adjudications. Assumed-benign commits are their own source, - weighted 0.5 and reported separately. A plane verdict is a feature, never a label: "rejected by - the plane" is not a negative, or the engine would learn to reproduce the plane's mistakes. -4. Retrieval finds similar labelled cases. A Gower distance over the signal profile, re-ranked - by cosine similarity of code embeddings (OpenAI `text-embedding-3-small` through OpenRouter, - cached by excerpt hash). The evaluation boundary is the only door into a prompt: never the - candidate's own repository group, never a group the fold evaluates. -5. A classifier compiled in house, DSPy-style. `retrieve -> classify(k)`: an instruction and - compiled demonstrations (a fixed prefix, so the provider's cache hits), the retrieved examples, - then the candidate's excerpt and signals. The model answers vulnerable, not vulnerable or - unsure with a rationale that cites a line; `k` samples give a score. One program is compiled - per profile and family on a compile split that is never evaluated. -6. Calibration on held-out repositories. Platt scaling of the score, cross-fitted by - repository group so no map is checked on the fold that fitted it. Calibration holds when the - slope interval contains 1, the intercept interval contains 0 and ECE <= 0.05. -7. Two operating points. BLOCK only where calibration holds and a fold reaches the required - precision; otherwise ADVISORY. An unsure majority can never BLOCK. -8. A/B experiments are the only way a change is adopted. Prompts, models, pass counts, - features and sampling are compared under a pre-registered manifest with frozen units, paired - arms, a repository bootstrap and two looks. exp-002 is the first to finish: a retrieval - ensemble (five samples over disjoint example sets at temperature 0) against today's - temperature sampling. It raised re-run agreement a little (0.906 against 0.875, interval - spanning zero) and lowered the within-pair AUC from 0.815 to 0.741 at 2.4 times the cost per - candidate ($0.0051 against $0.0021). Decision: REJECTED - (`benchmarks/experiments/exp-002-retrieval-ensemble/result.json`). -9. Extrapolation with intervals. Every number carries a 95% repository-cluster bootstrap - interval; the pooled and the within-pair readings are both reported, and the stricter one is +- Hand edits do not transfer. +- The data that could teach a better decision cannot be produced by hand at the scale needed. It + has to be harvested and labelled by machinery with a boundary around it. +- Framework knowledge (which function is a sink in which framework) is useful as a prior. But it + must stay optional. Otherwise the engine learns the vocabulary instead of the code. + +A reader might object that a classifier is slower to show gains than the next rule. It is. The +difference is where its gains are measured: on repositories it never saw. Rules were not measured +there. + +The chain has one job per step. The design is on [Memory and detection](memory-and-detection.md). +This list is the order of operations. + +1. **Instruments produce signals, never verdicts.** Each instrument writes one block of a + candidate's record. None decides alone. The instruments are: + - quick rules; + - the Joern engine; + - model sink classification; + - the plane's verify passes a and b, with the tie-break c; + - repository facts (callers, spans); + - model source/sink/sanitizer roles. +2. **Feature records are allow-listed, and missing is not zero.** A candidate is `(path, function, + family)` of one repository and pin. + - An instrument that did not run, had no coverage or failed records that state. "Silent" and + "did not look" stay distinguishable. + - Input profile v1 withholds the engine block and the function length. A leak audit found that + they separate the two sides of a pair by construction + (`benchmarks/measurements/2026-10-01-decision-engine-leak-audit/`). + - Since 2026-10-02 it also withholds the plane's `verify.flag_b` and `ms.operation`. The audit + of the plane profile flagged them the same way + (`2026-10-02-decision-engine-harvest-advisory/record.json`, + `src/openultrasast/learn/schema.py`). +3. **Labels come from ground truth only.** A fail-closed allow-list + (`src/openultrasast/learn/sources.toml`) names the sources: + - the spent populations' declared sites and fix ranges; + - the vulnerable and fixed sides of pairs, including the 97 real public fixes in + `benchmarks/pairs/advisory-fixes/`; + - benign controls; + - recorded adjudications. + + Assumed-benign commits are their own source. They are weighted 0.5 and reported separately. A + plane verdict is a feature, never a label. "Rejected by the plane" is not a negative. + Otherwise the engine would learn to reproduce the plane's mistakes. +4. **Retrieval finds similar labelled cases.** It uses a Gower distance (a distance for mixed + numeric and categorical data) over the signal profile. Results are re-ranked by cosine + similarity of code embeddings. The embeddings are OpenAI `text-embedding-3-small` through + OpenRouter, cached by excerpt hash. The evaluation boundary is the only door into a prompt. It + never admits the candidate's own repository group, or a group the fold evaluates. +5. **A classifier compiled in house, DSPy-style.** The program is `retrieve -> classify(k)`. The + prompt holds, in order: + - an instruction and compiled demonstrations (a fixed prefix, so the provider's cache hits); + - the retrieved examples; + - the candidate's excerpt and signals. + + The model answers vulnerable, not vulnerable or unsure, with a rationale that cites a line. `k` + samples give a score. One program is compiled per profile and family, on a compile split that + is never evaluated. +6. **Calibration on held-out repositories.** Platt scaling (a logistic fit) maps the score to a + probability. It is cross-fitted by repository group, so no map is checked on the fold that + fitted it. Calibration holds when three conditions are met: the slope interval contains 1, the + intercept interval contains 0 and ECE <= 0.05. +7. **Two operating points.** BLOCK applies only where calibration holds and a fold reaches the + required precision. Otherwise the point is ADVISORY. An unsure majority can never BLOCK. +8. **A/B experiments are the only way a change is adopted.** Prompts, models, pass counts, + features and sampling are compared under a pre-registered manifest. The manifest fixes frozen + units, paired arms, a repository bootstrap and two looks. exp-002 is the first to finish: + - It compared a retrieval ensemble (five samples over disjoint example sets at temperature 0) + with today's temperature sampling. + - It raised re-run agreement a little (0.906 against 0.875, interval spanning zero). + - It lowered the within-pair AUC from 0.815 to 0.741 and cost 2.4 times as much per candidate + ($0.0051 against $0.0021). + - Decision: REJECTED (`benchmarks/experiments/exp-002-retrieval-ensemble/result.json`). +9. **Extrapolation with intervals.** Every number carries a 95% repository-cluster bootstrap + interval. Both the pooled and the within-pair readings are reported. The stricter one is named. -10. The one-time v3 check. When a program is adopted and a prediction is committed, population - v3 is scored once under `benchmarks/independent/protocol-v3.md`. +10. **The one-time v3 check.** It happens when a program is adopted and a prediction is + committed. Then population v3 is scored once under `benchmarks/independent/protocol-v3.md`. ## 3. AI integration: ax on the server -All model work runs as Tasks on google/ax over Agent Substrate, in a single-node kind cluster on -the maintainer's host, each Task in its own gVisor sandbox. There is no local subprocess path. -The reconciler (`ousast plane run`) is a host process: it applies a Run's Workspaces and Tasks, -resumes each Task, sends its start signal, collects its artifacts and records state. Flow and -contract: [The plane on ax](plane.md); bring-up and the host's lessons: [ax on this host](ops/ax/README.md). +All model work runs as Tasks on google/ax over Agent Substrate. They run in a single-node kind +cluster on the maintainer's host. Each Task runs in its own gVisor sandbox. There is no local +subprocess path. + +The reconciler (`ousast plane run`) is a host process. It does five things: + +1. applies a Run's Workspaces and Tasks; +2. resumes each Task; +3. sends its start signal; +4. collects its artifacts; +5. records state. + +Flow and contract: [The plane on ax](plane.md). Bring-up and the host's lessons: +[ax on this host](ops/ax/README.md). What each Run guarantees: -- One Model and one budget per Task. A Task binds at most one ax Model (`deepseek-flash` for - chat, `openrouter-embedding` for embeddings) and `budget: {usd, calls}`. The metered client - refuses the next call at the ceiling; the task ends unfinished and resumes on a rerun. Model-free - tasks run with a zero budget, so a stray call fails loudly. -- Credentials exist only in the start request. Agent Substrate boots each template once as a - golden actor, so the runner never starts by itself. It waits for `POST /ousast/v1/start`, which - the reconciler sends through `atenet-router` to that actor alone. The key is read from the - operator's environment or `.env`, kept in memory by the runner, passed to the command and - redacted from echoed stderr; no manifest, rendered Task or file carries it. -- Per-task egress. One EgressPolicy per actor through the `atenet-egress` gateway, deny by - default, hostnames only: plain HTTP to the artifact receiver, TLS passthrough to the Workspaces' - Git hosts and the Model's declared hosts. Deleted with the actor. -- Artifacts come back through the gateway. The runner posts its output directory as a tar to - the receiver, an HTTP server inside the reconciler process on the host, reached through a Service - with an EndpointSlice to the host. That delivery is the task's completion; ax has no Completed - phase. -- Memory on S3-compatible RustFS. `OUSAST_MEMORY=s3://sast-memory` on a RustFS server. The - store verifies the bucket at every start (versioning, lifecycle, tags, S3 Select) and refuses to - start when anything is missing; there is no local or fetch-and-filter fallback. Each row kind - has a fixed set of queryable fields, enforced at write time - ([Memory](memory.md#verify-at-startup-no-fallback), [RustFS setup](rustfs.md)). +- **One Model and one budget per Task.** A Task binds at most one ax Model and + `budget: {usd, calls}`. The Models are `deepseek-flash` for chat and `openrouter-embedding` for + embeddings. The metered client refuses the next call at the ceiling. The task ends unfinished + and resumes on a rerun. Model-free tasks run with a zero budget, so a stray call fails loudly. +- **Credentials exist only in the start request.** Agent Substrate boots each template once as a + golden actor. So the runner never starts by itself. It waits for `POST /ousast/v1/start`. The + reconciler sends that request through `atenet-router` to that actor alone. + - The key is read from the operator's environment or `.env`. + - The runner keeps it in memory and passes it to the command. + - It is redacted from echoed stderr. + - No manifest, rendered Task or file carries it. +- **Per-task egress.** Each actor gets one EgressPolicy through the `atenet-egress` gateway. It is + deny by default and uses hostnames only. It allows plain HTTP to the artifact receiver. It allows + TLS passthrough to the Workspaces' Git hosts and the Model's declared hosts. It is deleted with + the actor. +- **Artifacts come back through the gateway.** The runner posts its output directory as a tar to + the receiver. The receiver is an HTTP server inside the reconciler process on the host. It is + reached through a Service with an EndpointSlice to the host. That delivery is the task's + completion; ax has no Completed phase. +- **Memory on S3-compatible RustFS.** The setting is `OUSAST_MEMORY=s3://sast-memory` on a RustFS + server. + - The store verifies the bucket at every start: versioning, lifecycle, tags, S3 Select. + - It refuses to start when anything is missing. There is no local or fetch-and-filter fallback. + - Each row kind has a fixed set of queryable fields, enforced at write time + ([Memory](memory.md#verify-at-startup-no-fallback), [RustFS setup](rustfs.md)). Lessons the host taught, each from a failed live run (the full list is on [ax on this host](ops/ax/README.md#what-this-host-taught-each-one-cost-a-failed-live-run)): - Substrate installs no WorkerPool, and actor images must be pinned by digest. -- A release `ax` CLI skews from the server; build it from the deployed checkout. +- A release `ax` CLI skews from the server. Build it from the deployed checkout. - ax has no Completed phase, so completion is our artifact delivery. - The golden boot means the runner must wait for a start request. -- The egress gateway dials the address the actor connected to, so the receiver is a Service +- The egress gateway dials the address the actor connected to. So the receiver is a Service, dialled by ClusterIP with its name as Host. - ax's snapshot bucket must exist in your S3 server. When it did not, deleted golden actors piled - up in `DELETING` and filled the disk during a 600-task harvest on 2026-10-01. + up in `DELETING`. They filled the disk during a 600-task harvest on 2026-10-01. -Deployment status, in the two words [Deployment](deployment.md) uses strictly. Implemented: -ax as the only executor, per-task EgressPolicy, credentials only in the start request, per-task -budgets and attribution, the S3 store verified at startup, local bring-up, doctor and smoke Run, -re-pinning generated Tasks to another registry. Planned, not done: deployment to any cluster -other than this host's kind cluster, a configurable kube context (`kind-` is hard-coded in -`doctor.py`, `egress.py`, `router.py`), the receiver as an in-cluster Deployment, the -reconciler in-cluster with Secrets injected, presigned-URL delivery to the store. +Deployment status uses two words, as [Deployment](deployment.md) does, strictly. + +Implemented: + +- ax as the only executor; +- per-task EgressPolicy; +- credentials only in the start request; +- per-task budgets and attribution; +- the S3 store verified at startup; +- local bring-up, doctor and smoke Run; +- re-pinning generated Tasks to another registry. + +Planned, not done: + +- deployment to any cluster other than this host's kind cluster; +- a configurable kube context (`kind-` is hard-coded in `doctor.py`, `egress.py`, `router.py`); +- the receiver as an in-cluster Deployment; +- the reconciler in-cluster with Secrets injected; +- presigned-URL delivery to the store. ## 4. Token economics, before and after -Before the plane, the script pipeline cost $0.022 per candidate for two verify passes -(`benchmarks/independent/verifier-batched-check-2026-09-29.json`), and the maintainers' own -coding sessions spent about 90% of their tokens on tool input and output rather than reasoning. -After it, the per-candidate costs below are what the records show. The mechanisms (one budget -per Task, pay-once reuse, cheap-before-expensive ordering, replay at $0) are on -[Token ergonomics](token-ergonomics.md); this table is the before and after. +Before the plane, two costs stood out: + +- The script pipeline cost $0.022 per candidate for two verify passes + (`benchmarks/independent/verifier-batched-check-2026-09-29.json`). +- The maintainers' own coding sessions spent about 90% of their tokens on tool input and output + rather than reasoning. + +After it, the per-candidate costs below are what the records show. The mechanisms are on +[Token ergonomics](token-ergonomics.md): one budget per Task, pay-once reuse, +cheap-before-expensive ordering and replay at $0. This table is the before and after. | Item | Before | After | Record | | --- | --- | --- | --- | @@ -195,63 +273,70 @@ per Task, pay-once reuse, cheap-before-expensive ordering, replay at $0) are on | Harvest of plane signals for the engine | none existed | `verify` $2.5373 over 1,961 calls and 474 tasks; `roles` $1.4036 over 1,034 calls and 151 tasks | `2026-10-01-decision-engine-harvest/record.json` | | Development transcript, tool I/O share | 90% | 85.5% over 2,701 tool calls (`dev_token_report`) | `plane-increment-1.json` | -Two notes from the records. The meter prices every call at the Model's list price -(`plane/models/deepseek-flash.yaml`); the DeepSeek account moved less, about one third of the -metered amount on the injection slice ($0.30 for $0.91 metered), and budgets and gates are set -against the meter, never against the balance. And the development share barely moved: 90 to 85.5 -is a small gain, and the remedies (a read guard, a bulk reader, scripts in files) target file -contents because that is where the tokens go. +Two notes from the records: + +1. The meter prices every call at the Model's list price (`plane/models/deepseek-flash.yaml`). The + DeepSeek account moved less: about one third of the metered amount on the injection slice + ($0.30 for $0.91 metered). Budgets and gates are set against the meter, never against the + balance. +2. The development share barely moved: 90 to 85.5 is a small gain. The remedies are a read guard, + a bulk reader and scripts in files. They target file contents, because that is where the + tokens go. ## 5. Open items -Five groups, each item with its record: the engine's own gates, data, measurement debt, -infrastructure, and instrument gaps. +There are five groups. Each item has its record. The groups are the engine's own gates, data, +measurement debt, infrastructure, and instrument gaps. The engine's gates. -- BLOCK is reachable for no family: calibration fails for five (ECE 0.071 to 0.119) and the - sixth reaches no fold with a Wilson lower bound of 0.95 on precision (slice-2 record). +- BLOCK is reachable for no family. Calibration fails for five (ECE 0.071 to 0.119). The sixth + reaches no fold with a Wilson lower bound of 0.95 on precision (slice-2 record). - Re-run agreement is below 0.9 for three families: injection 0.77, access control 0.83, untrusted destination 0.87. exp-002 tried the obvious variance fix and lost on the ranking. Data. - Positives-only groups: 98 (family, group) pairs in the 904-example memory hold positive examples - and no negative. 94 of them exist because the fixed side no longer has the function (the vibe-py - pairs). No advisory-fixes group is among them - (`benchmarks/measurements/2026-10-02-decision-engine-negatives-gap/record.json`). There are no - single-label negatives, so every family's pooled AUC is inflated by positives that never had to - be told apart from their own fix (slice-2 record, `paired_groups_check`). -- The model has seen public fixes. A verify pass named the advisory's CVE id for a public fix, an - observation the engine's design specification (maintainer tooling) records as visible - memorisation. Excerpts now redact advisory ids and security-worded comments - (`src/openultrasast/learn/excerpt.py`), but labels drawn from public advisories are partly in the - model's training data, and the within-pair numbers carry that. -- PHP is covered by the engine on real code and by in-sample quick rules; the population that + and no negative. + - 94 of them exist because the fixed side no longer has the function (the vibe-py pairs). + - No advisory-fixes group is among them + (`benchmarks/measurements/2026-10-02-decision-engine-negatives-gap/record.json`). + - There are no single-label negatives. So every family's pooled AUC is inflated by positives + that never had to be told apart from their own fix (slice-2 record, `paired_groups_check`). +- The model has seen public fixes. A verify pass named the advisory's CVE id for a public fix. + The engine's design specification (maintainer tooling) records this as visible memorisation. + Excerpts now redact advisory ids and security-worded comments + (`src/openultrasast/learn/excerpt.py`). But labels drawn from public advisories are partly in + the model's training data, and the within-pair numbers carry that. +- PHP is covered by the engine on real code and by in-sample quick rules. The population that would measure it (v3) is reserved. Measurement debt. -- exp-001 (known callers) is registered and has not run; the model-free candidate generation on - the 41 development repositories that its units need is not built +- exp-001 (known callers) is registered and has not run. Its units need model-free candidate + generation on the 41 development repositories. That generation is not built (`benchmarks/experiments/exp-001-known-callers/registration.json`). - Leave-one-source-out, leave-one-framework-out and the memory-size curve points (0, 25, 50%) - have not run; one paired point exists (724 against 904 examples, injection, AUC 0.76 against 0.77). -- The plane-profile leak audit flagged `verify.flag_b` and `ms.operation`; they are withheld from - profile v1, and whether they track the label on unseen code has to be decided before any - plane-profile program runs (`2026-10-02-decision-engine-harvest-advisory/record.json`). + have not run. One paired point exists (724 against 904 examples, injection, AUC 0.76 against 0.77). +- The plane-profile leak audit flagged `verify.flag_b` and `ms.operation`. They are withheld from + profile v1. Before any plane-profile program runs, someone must decide whether they track the + label on unseen code (`2026-10-02-decision-engine-harvest-advisory/record.json`). - Population v3 waits for `benchmarks/independent/prediction-v3.json`, which does not exist. It is written only after a program is adopted. Infrastructure. -- Deployment to a separate Kubernetes cluster is planned, not done; the code changes it needs are +- Deployment to a separate Kubernetes cluster is planned, not done. The code changes it needs are listed with file paths on [Deployment](deployment.md#what-is-implemented-and-what-is-planned). Instrument gaps. -- A missing provider key is a quiet path in a scan. The plane refuses to start a Task whose Model - names an unset variable (`plane/router.py`, `StartError`). A scan without a key asks the model - nothing, reports only the graph's entailed findings, and records `learning_endpoint_unavailable` - in `manifest.json` (`cli.py`, after `resolve_chat_endpoint` returns nothing); `report.md` has no - line for that reason yet, so the report does not say the model layer was skipped. +- A missing provider key is a quiet path in a scan. + - The plane refuses to start a Task whose Model names an unset variable (`plane/router.py`, + `StartError`). + - A scan without a key asks the model nothing and reports only the graph's entailed findings. + It records `learning_endpoint_unavailable` in `manifest.json` (`cli.py`, after + `resolve_chat_endpoint` returns nothing). + - `report.md` has no line for that reason yet. So the report does not say the model layer was + skipped.