marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: MkDocs site with flow diagrams; RustFS permissions

Browse Source 
Material for MkDocs site (mkdocs.yml, optional extra `docs`) with Mermaid via
the superfences custom fence. New pages: overview, scanning (modes, quick
rules, Joern engine, priors, pre-push delta), the plane on ax (one task as a
sequence, golden snapshot, Run DAGs), memory (row kinds, write/read paths,
FileStore vs S3, verify at startup), memory and detection, and RustFS setup
(admin steps, complete agent policy, .env, refusal messages, Select limit).
The ops READMEs are included at build time; evaluation gains the population
lifecycle and the fold layout. tests/test_docs_site.py runs the strict build
when mkdocs is importable and checks Mermaid blocks and the RustFS page.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-10-02 08:17:40 +02:00
co-authored by Claude Opus 5.5
parent 30cde99e2d
commit 95a252b89b
16 changed files with 1080 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/decision-engine.md
+1 -1
View File
@@ -80,6 +80,6 @@ v3 check are still ahead.
| `ousast learn curve --program ID` | the metric over memory size on a fixed candidate subset |
| `ousast learn audit-leaks` | how well each signal alone separates the sides of a pair (no model) |
The memory store is the plane's (`OUSAST_MEMORY`; see [ops/ax/README.md](../ops/ax/README.md)).
The memory store is the plane's (`OUSAST_MEMORY`; see [ops/ax/README.md](ops/ax/README.md)).
The harvest that produced the plane signals runs as plane Runs generated by
`ousast plane harvest` (`benchmarks/measurements/2026-10-01-decision-engine-harvest/record.json`).
docs/evaluation.md
+37
View File
@@ -49,6 +49,43 @@ v3 is reserved: development and analysis work never reads its cases, and
`tests/test_independent_population.py` fails when anything in the development tree (a recipe,
catalog, manifest or measurement) names one of its repositories.
```mermaid
stateDiagram-v2
[*] --> Frozen: cases selected and frozen before any scan
Frozen --> Scored: scored once under the pre-registered protocol
Scored --> Spent: result recorded
Spent --> [*]: may inform changes, can no longer qualify anything
note right of Frozen
v3 (PHP, 15 cases) waits here as the one-time final check,
run once after a prediction is committed
end note
note right of Spent
v1 and v2 are spent
end note
```
## Folds: how learned decisions are kept out of their own training data
The decision engine is evaluated on repositories it was not shown (`src/openultrasast/learn/folds.py`).
Every split is by repository group (the label builder's normalised `owner/name`, merged across
URLs, forks and shared advisories), across all corpora, seeded and deterministic.
```mermaid
flowchart TD
groups["All repository groups with labels"] --> c["Compile split: 25% of groups, never evaluated"]
groups --> rest["The other 75%"]
c --> boot["C_boot 60%: demonstrations"]
c --> val["C_val 40%: instruction scoring"]
rest --> outer["Outer grouped 5-fold: calibration cross-fitted, operating points nested"]
rest --> loso["Leave-one-source-out: a source held out whole"]
rest --> lofo["Leave-one-framework-out: frameworks with at least 10 groups"]
```
`assert_disjoint` runs before a compile: the compile split and the evaluation folds share no
group. A framework with fewer than 10 groups gets `insufficient data` instead of a fold. As of
2026-10-02 only the outer folds of the injection family have been run
([decision-engine.md](decision-engine.md#first-measured-slice-2026-10-01)).
## The plane increment
The first increment of the agentic plane ran the model-driven pipeline's validation set (46
docs/examples.md
+1 -1
View File
@@ -69,7 +69,7 @@ uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml
## 7. Running the agentic plane
Model work over whole repositories runs as a Run manifest on ax, one isolated actor
per task (bring-up: [ops/ax/README.md](../ops/ax/README.md)). The provider key is
per task (bring-up: [ops/ax/README.md](ops/ax/README.md)). The provider key is
read from the environment (or `.env`, which never overrides an exported variable) by
`ousast` and sent only in each task's start request.
docs/index.md
+54
View File
@@ -0,0 +1,54 @@
# OpenUltraSAST
OpenUltraSAST is an experimental static security analyser meant to work as a safety net on
repositories it has never seen: it scans a checkout, or the commits a `git push` would publish,
and separates what it can prove from what it merely suspects. A scan that could not look at
something records a degradation instead of reporting a clean result.
**State as of 2026-10-02 (v1.2.0-alpha.1):** the pre-push safety net is **NO-GO** for rollout.
No hook capability is enabled, and no independent population has passed the qualification gates
(see [Evaluation](evaluation.md)).
## Three layers
| Layer | What it is | Where |
| --- | --- | --- |
| **Scanners** | `ousast scan` (`quick`, `standard`, `deep`) and `ousast pre-push`: language-scoped pattern rules, a Joern code property graph engine, verification and scoring. Deterministic in `quick`; no model is needed for a scan. | [Scanning](scanning.md), [Architecture](architecture.md) |
| **The agentic plane on ax** | Model-driven work over whole repositories, run as a DAG of tasks on google/ax, one isolated actor per task. What the runs learn is kept in a memory store. Maintainer surface: `ousast plane`. | [The plane on ax](plane.md), [Memory](memory.md) |
| **The decision engine** | A learned classifier that weighs every instrument's signals against similar labelled cases from memory. **In development, not adopted**: no scan, pre-push check or report uses it. Maintainer surface: `ousast learn`. | [Memory and detection](memory-and-detection.md), [Decision engine](decision-engine.md) |
## How they connect
```mermaid
flowchart LR
repo["Repository or push delta"] --> scan["Scanners: quick rules, Joern engine, verify, score"]
scan --> report["Report: findings with evidence, SARIF, score"]
scan -- "alerts on vulnerable and fixed pins" --> plane["Plane on ax: repo-facts, verify a/b/c, agree, features"]
plane -- "remember" --> mem[("Memory store: rows, facts, examples, blobs")]
mem -- "improve --memory: rule-status proposals" --> gate{"Validator and detection gate"}
gate -- "accepted round" --> ledger["Ruleset ledger: rule status"]
ledger --> scan
labels["Labels from ground truth"] --> mem
mem -- "retrieval of similar labelled cases" --> engine["Decision engine: compiled AI classifier"]
engine -. "intended, not adopted: BLOCK / ADVISORY" .-> report
```
- The **scanners** produce findings and, inside plane Runs, the `alerts` that become memory rows.
- The **plane** runs model work (verify passes, roles) and model-free bookkeeping (facts, agreement,
feature records) per case; its `remember` task turns a case's artifacts into rows of the
[memory store](memory.md).
- Memory feeds two consumers. `ousast improve --memory` turns stored rows into rule-status
proposals that pass through the same validator and gate as every other edit. The decision engine
retrieves labelled examples from the same store.
- The decision engine's output (dotted line) is the intended path. Nothing user-facing reads it
today.
## Where to go next
- Run a scan: [Examples](examples.md), and the commands in [Scanning](scanning.md).
- Understand a finding's evidence and score: [Architecture](architecture.md).
- Operate the plane and its store: [ax on this host](ops/ax/README.md) and
[RustFS setup](rustfs.md).
- Trust boundaries and hardening: [Threat model](threat-model.md).
Every number in these pages is quoted from a committed record whose path is given next to it.
docs/memory-and-detection.md
+156
View File
@@ -0,0 +1,156 @@
# Memory and detection
How what the system remembers is meant to turn into better decisions on code it has not seen.
This is the design of the learned decision engine (`src/openultrasast/learn/`), which is **in
development and not adopted**: no scan, pre-push check or report uses it today. Every user-facing
decision is still the deterministic one described in [Scanning](scanning.md) and
[Architecture](architecture.md). The engine's commands and status are on
[Decision engine](decision-engine.md); the store itself is on [Memory](memory.md).
## From instruments to a decision
```mermaid
flowchart TD
inst["Instruments: quick rules, Joern engine, verify passes and agree, model roles, repo facts"] --> feat["Feature record per candidate: path, function, family"]
feat --> prof["Input profile v1: engine and function length withheld"]
lab["Labels from ground truth: pairs, populations v1/v2, advisory fixes, adjudications"] --> ex["Example rows in memory: label + features + excerpt"]
prof --> ret["Retrieval: signal profile, then code embeddings"]
ex --> ret
ret --> bnd{"Evaluation boundary"}
bnd --> prog["Compiled AI classifier: instruction, demonstrations, up to 6 examples, candidate"]
prog --> score["Score s: mean over k samples"]
score --> cal["Platt calibration, cross-fitted on held-out repositories"]
cal --> op{"Operating point"}
op -- "calibration holds and precision reachable" --> block["BLOCK"]
op -- "otherwise" --> adv["ADVISORY"]
```
The intended place for these decisions is changed code: `learn/features.py` builds feature records
for a whole-repository scan or for a base..head delta (`build_for_scan` with `Delta`), the same
delta `ousast pre-push` checks. Integration into scan and pre-push is still ahead.
## 1. Instruments produce signals, not verdicts
Each instrument contributes one block to a candidate's feature record (`learn/schema.py`,
`INSTRUMENTS`): `language`, `quick` (quick rules), `engine` (the Joern model layer), `facts` (repo
facts, such as callers and function span), `source`, `entry_points`, `roles` (model source, sink
and sanitizer roles), `delta`, `verify` and `agree` (the plane's verify passes and their
agreement), and `model_sinks`. A candidate is `(path, function, family)` of one repository and pin
where at least one instrument emitted a signal of that family (`learn/features.py`).
An instrument that did not run, had no coverage or failed records that state (`none`, `failed`,
`not_applicable`), never a zero, so "silent" and "did not look" stay distinguishable. None of the
instruments decides alone.
**Why profile v1 withholds the engine and function length.** A leak audit
(`ousast learn audit-leaks`, `benchmarks/measurements/2026-10-01-decision-engine-leak-audit/`)
found that both separate the vulnerable and fixed sides of a pair by construction: a fix removes
a sink, and a guard lengthens the function. That holds on the pair corpus, not on unseen code, so
`INPUT_PROFILES["v1"]` drops the `engine` block and `facts.function_lines` from prompts and from
the retrieval distance.
## 2. Labels come from ground truth only
`ousast learn labels` (`learn/labels.py`) reads only the sources on a fail-closed allow-list
(`learn/sources.toml`): `population-v1`, `population-v2`, `dev-php` (reviewed recipe sites),
`pairs`, `advisory-fixes`, `adjudications` and `assumed_benign`. A file the list does not name is
refused before it is opened; the reserved population v3 is excluded unread.
- **Positives** are declared sites and the vulnerable side of a fix; **negatives** are the same
function on the fixed side where the fix changed it, and recorded `False`/`FP` adjudications.
A vulnerable-pin candidate matching no site is unlabelled, never negative.
- **Assumed benign** (maintainer decision 2026-09-30): ordinary non-security commits of the training
repositories, at least 90 days from any fix, with no later security commit touching the same
files. They are their own source with weight 0.5, conditional on a signal at the tip and none at
the base, and reported separately, never merged with verified negatives.
- **Plane verdicts are features, not labels.** "Agreed", "rejected" or "disputed" is the output of
the verify instrument; it enters the record as the `verify` and `agree` blocks. Using it as a
label would teach the engine to reproduce the plane, mistakes included, so "rejected by the
plane" is never a negative (`learn/labels.py`: "A plane verdict ... is never a label").
`ousast learn memory build` joins each label with the candidate's feature record and an excerpt of
its function and stores one `example` row (no rationale) plus the excerpt blob.
## 3. Retrieval of similar labelled cases
`learn/retrieve.py`, no model:
1. **Signal profile.** A Gower distance over the allow-listed features of the active profile,
per instrument block, with missing values explicit (a different state counts as distance 1).
The nearest 40 examples of the candidate's family and the nearest 40 of other families go on.
2. **Code embeddings.** A cosine re-rank of the excerpts' embeddings (OpenAI
`text-embedding-3-small` through OpenRouter, 1,536 dimensions, cached by excerpt hash):
`r = 0.5 (1 - D) + 0.5 cos`. Without the candidate's vector the retrieval is signals only.
3. **Balance.** Six examples: at most half per label, at most 2 per repository group, at most 2
contrast examples from other families.
**The evaluation boundary.** `eligible()` is the only way an example enters a prompt, for retrieved
examples and compiled demonstrations alike: never the candidate's own repository group, never a
group the fold evaluates, never the held-out source or framework, and (outside deployment) never
a near-duplicate of the candidate (cosine >= 0.98). The program repeats the check on the ids it
actually rendered before any call.
## 4. The AI classifier, compiled
`learn/program.py` and `learn/compile.py`, DSPy-style and in-house. The prompt is the instruction
plus the compiled demonstrations (a prefix identical for every candidate, so the provider's prefix
cache hits), then the retrieved examples, then the candidate's excerpt, signals, roles and family.
The model (`deepseek-flash`) answers `vulnerable`, `not_vulnerable` or `unsure` with a confidence
and a rationale that must cite a line of the code. It is sampled `k` times; the score `s` is the
mean over samples, and a majority `unsure` can never BLOCK. Compilation bootstraps demonstrations
and searches instructions on a compile split of repository groups, scored by a pre-registered
balanced Brier score (`learn/compile.toml`); the program is stored as `programs/<id>.json`.
## 5. Calibration and the two operating points
`learn/calibrate.py`: Platt scaling of `s`, cross-fitted so that no map is checked on the fold it
was fitted on. Calibration **holds** when the slope interval contains 1, the intercept interval
contains 0 and ECE <= 0.05. **BLOCK** is offered only when calibration holds and a fold reaches the
required precision; otherwise the engine offers **ADVISORY** only.
## The loop: memory improves decisions without retraining
```mermaid
flowchart LR
new["New labels and adjudications"] --> labels["ousast learn labels"]
labels --> build["ousast learn memory build, embed"]
build --> mem[("Memory store: example rows")]
mem --> retr["Retrieved per candidate at decision time"]
retr --> prog["Same compiled program"]
prog --> dec["Decision"]
runs["Plane runs: alert, verdict, coverage rows"] --> mem2[("Memory store: run rows")]
mem2 --> imp["ousast improve --memory: M1, M2 proposals"]
imp --> gate{"Unchanged validator and gate"}
gate -- "accepted round" --> ledger["Rule status in the ledger"]
```
- **Decisions.** The compiled part of the program is its instruction and demonstrations; the
examples are retrieved from memory for each candidate. A new `example` row is therefore eligible
for retrieval in the next decision without recompiling the program, subject to the same
evaluation boundary. `ousast learn curve` measures the metric over memory size on a fixed
candidate subset.
- **Rules.** `ousast improve --memory` reads the run rows (alerts, verdicts, coverage) and proposes
rule-status edits only (shadow a rule that is repeatedly false across repositories; re-enable a
shadow rule that keeps hitting agreed declared sites). Rows from holdout pairs, the gated
manifest's own cases and any `--qualify-population` are dropped first, and every proposal goes
through the same validator and gate as any other edit (`improve/memory.py`).
## What the first slice measured
Record: `benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json` (counts
only). Family **injection**; memory of 724 static examples from all families; input profile v1;
framework priors off; out-of-repository evaluation (compile split never evaluated, outer grouped
5-fold, calibration cross-fitted, operating points nested); 95% repository-cluster bootstrap
intervals.
| Measure | Value | What it means |
| --- | --- | --- |
| Candidates | 110 in 29 repository groups, 58 positive | the positive share is 58/110 = 0.53 |
| AUC of the score `s` | 0.74 (0.65 - 0.85) | the score ranks a vulnerable candidate above a non-vulnerable one about three times in four: there is ranking signal |
| ADVISORY | recall 0.90 (0.82 - 0.97), precision 0.54 (0.44 - 0.65) | precision is at the positive share (0.53), so at this threshold ADVISORY is **not discriminating**: it flags nearly as indiscriminately as flagging everything would |
| BLOCK | not offered | calibration does not hold (ECE 0.064 > 0.05), and every outer fold reported `precision_unreachable` |
**The engine is not adopted.** These numbers come from one family and one slice; the other
families, the leave-one-source-out and leave-one-framework-out evaluations, integration into scan
and pre-push, and the one-time check on population v3 are still ahead
([Decision engine](decision-engine.md#first-measured-slice-2026-10-01)).
docs/memory.md
+131
View File
@@ -0,0 +1,131 @@
# Memory
One store, keyed by repository and pin, holds what plane runs and the decision engine learned.
The code is `src/openultrasast/plane/memory.py`; the same layout serves both backends.
How the stored rows feed detection is on [Memory and detection](memory-and-detection.md).
## What is stored
```
index.jsonl one row per ingested (run, task): sha256 of its rows, row count
facts/<sha256>.json repository facts by content (identical facts stored once)
repos/<host>__<owner>__<name>/<pin>.jsonl the rows of one repository + 40-hex pin, sorted by id
excerpts/<sha256>.txt code excerpts, by content
embeddings/<model>/<excerpt sha>.json an excerpt's embedding vector, per embedding model
responses/<request sha>.json cached model responses
programs/<program id>.json compiled decision-engine programs
```
**Rows.** Every row carries `id`, `kind`, `repo`, `pin`, `run`, `task`, `population`, `split`
and `image` (`ROW_FIELDS`); `id` is the sha256 of (kind, run, task, subject), so a repeated row
replaces itself. The kinds (`KINDS`) and who writes them:
| Kind | Written by | Holds |
| --- | --- | --- |
| `facts` | `remember` task | the facts entry of a case (points at `facts/<sha256>.json`; reused by memory key) |
| `verdict` | `remember` task | per candidate: each verify pass's answer, the final outcome, whether a tie-break ran, cost |
| `unit_cost` | `remember` task | per verify pass and file: usd, calls and token usage |
| `alert` | `remember` task | a quick-scan or engine alert on the vulnerable or fixed pin (loop Runs) |
| `coverage` | `remember` task | per language and pin: whether any rule could have fired there |
| `features` | `remember` task, from the `features` task | one feature record per candidate and family |
| `proposal_outcome` | `ousast improve --memory` (not with `--dry-run`), and the loop's `improve` step | what happened to each memory proposal in a round |
| `example` | `ousast learn memory build` | a labelled case: label + feature record + excerpt sha (no rationale) |
| `label`, `decision`, `experiment`, `arm_outcome`, `experiment_result` | reserved for later decision-engine tasks | declared in `KINDS`; not written by the code on main yet |
**Blobs** (`put_blob`) are content-addressed: a 64-hex name under a fixed prefix. Excerpts are
written by `learn memory build` (`learn/examples.py`), embeddings by `learn memory embed`
(`learn/embeddings.py`, keyed by excerpt sha so an excerpt is embedded once), responses by the
program's caller (`learn/program.py`, keyed by the sha256 of model, parameters digest, messages,
temperature, sample and format, so a rerun replays at no cost), and compiled programs by
`learn compile` (`learn/compile.py`).
## Write path
```mermaid
flowchart LR
tasks["Plane tasks: facts, verify a/b/c, agree, alerts, features"] --> rem["remember task: memory.jsonl"]
rem --> recv["Delivered to the host with the run's artifacts"]
recv --> ingest["ingest after ousast plane run, or ousast plane remember RUN"]
loop["Loop improve step: proposal_outcome"] --> ingest
improve["ousast improve --memory: proposal_outcome"] --> put
build["ousast learn memory build: labels + features + excerpts"] --> put["put_rows, put_blob"]
embed["ousast learn memory embed"] --> put
compile["ousast learn compile, evaluate"] --> put
ingest --> idx{"index.jsonl: same run, task and sha256?"}
idx -- "yes" --> skip["Skipped"]
idx -- "no" --> put
put --> store[("Memory store")]
```
- `ingest` validates every row first: a malformed file fails before anything is written.
- Rows are merged into their repository + pin object by `id`. Every object is written whole and the
index last, so an interrupted ingest leaves nothing a repeat would not repair.
- Each row object carries tags (`TAG_FIELDS`: `repo`, `pin`, `kind`, `family`, `run`,
`population`, `split`) listing the distinct values of its rows (`*` when they exceed 256
characters).
- The plane's store failures never change a run's result: an ingest can be repeated by hand
(`cli.py`, `_plane_memory`).
## Read path
```mermaid
flowchart LR
store[("Memory store")] --> seed["seed before a Run"]
seed --> reuse["Facts reused by memory key: repo, pin, candidates, runner image"]
seed --> snap["memory-snapshot: rows after the train-on-test guard"]
store --> examples["learn: load example rows, excerpts, embeddings"]
examples --> retr["Retrieval of similar labelled cases"]
store --> responses["Response cache: replay at no cost"]
store --> imp["ousast improve --memory: alert, verdict, coverage rows"]
imp --> guard["Train-on-test guard"]
guard --> rules["Rules M1, M2 to rule-status proposals"]
rules --> gate{"Unchanged validator and gate"}
```
- **Seed** (`memory.seed`): a Run task whose `openultrasast.io/memory-key` matches stored facts gets
them and is marked done; a changed pin, candidate set or runner image finds nothing and
recomputes. The loop's `memory-snapshot` is seeded the same way, because a sandboxed task cannot
read the store.
- **Retrieval** (`learn/retrieve.py`): see
[Memory and detection](memory-and-detection.md#3-retrieval-of-similar-labelled-cases).
- **`improve --memory`** (`improve/memory.py`): rows from holdout pairs, from the gated manifest's
own cases and from any `--qualify-population` are dropped before any rule sees them. See
[Architecture](architecture.md#proposals-from-plane-memory).
## Backends: FileStore and the S3 store
`OUSAST_MEMORY` selects the backend (`open_store`):
| | `FileStore` | `MinioStore` (S3: RustFS or MinIO) |
| --- | --- | --- |
| Selected by | `file:///path`, or nothing (default `$OUSAST_RESULTS/plane/memory`) | `minio://<bucket>[/<prefix>]` |
| Needs | a local disk; refuses to write below 1 GiB free | the `minio` extra, `MINIO_*` settings from `.env` or the environment |
| Filtered reads | read and filter locally | S3 Select pushdown (`SelectObjectContent` over JSON Lines); kind filter by object tag |
| Object version cited by provenance | the file's content hash | the bucket's object version id |
| Bucket setup | none | done once by an admin, verified at every start ([RustFS setup](rustfs.md)) |
### Verify at startup, no fallback
The S3 store never configures its bucket. Each time a `MinioStore` is opened for a bucket and
prefix (once per process), `verify_bucket()` checks, using reads plus one small probe object at
`<prefix>/_probe/select.jsonl`:
1. versioning is `Enabled` (provenance cites object versions);
2. an enabled lifecycle rule expires `<prefix>/runs/` after a number of days, as a plain prefix
filter, and no enabled rule expires the whole store;
3. S3 Select answers a probe query over JSON Lines;
4. the probe's object tags are readable.
Anything missing is named, with the admin command that fixes it, in one `MemoryStoreError`, and
the store does not start. There is no local or fetch-and-filter fallback: a filtered read that
S3 Select cannot answer raises a `MemoryStoreError` instead of returning "no rows". The exact
messages are listed on [RustFS setup](rustfs.md#what-the-store-verifies-at-startup).
### Queryable fields per kind (in progress)
RustFS's Select infers an object's JSON schema from its leading rows, so a `where` field missing
there fails with `EvaluatorBindingDoesNotExist` even when a later row has it (measured
2026-10-02; `memory.py`, `MinioStore._select`). On main today the store turns that failure into a
`MemoryStoreError` naming the field, never into an empty answer. A fixed set of queryable fields
per row kind, so that every filtered field is present in every row of its kind, is in progress
and not on main as of this page.
docs/ops/README.md
+3
View File
@@ -0,0 +1,3 @@
--8<-- "ops/README.md"
*Source: `ops/README.md` in the repository; this page includes it when the docs site is built.*
docs/ops/ax/README.md
+10
View File
@@ -0,0 +1,10 @@
# ax on this host
The agentic plane runs every task on google/ax over Agent Substrate in a single-node kind cluster. ax is the
only executor; `ousast plane run` submits, resumes, starts and collects. The flow is drawn in
[The plane on ax](../../plane.md); the bucket setup for the S3 memory store is in
[RustFS setup](../../rustfs.md).
--8<-- "ops/ax/README.md:5"
*Source: `ops/ax/README.md` in the repository, from its line 5 on; this page includes it when the docs site is built.*
docs/plane.md
+144
View File
@@ -0,0 +1,144 @@
# The plane on ax
Model-driven work over whole repositories runs on a separate agentic plane: a **Run**
(`plane/runs/*.yaml`, kind `openultrasast.io/v1alpha1 Run`) is a DAG of steps, and each step is
an ax **Task** (`plane/tasks/`) that binds its Workspaces (`plane/workspaces/`), at most one ax
**Model** (`plane/models/`: `deepseek-flash` for chat, `openrouter-embedding` for embeddings) and
its own `budget: {usd, calls}`. ax (google/ax over Agent Substrate, in a single-node kind cluster
on the maintainer's host) is the only executor. Bring-up and the host's lessons are in
[ax on this host](ops/ax/README.md); this page draws the flow.
Sources: `src/openultrasast/plane/reconciler.py`, `router.py`, `egress.py`, `runner.py`,
`generate.py`, `memory.py`, and `src/openultrasast/plane/tasks/`.
## One task, end to end
`ousast plane run RUN.yaml` (the reconciler) does this for every ready task; `--workers` bounds
the tasks in flight, and a rerun skips tasks already done.
```mermaid
sequenceDiagram
autonumber
participant R as Reconciler (ousast plane run)
participant AX as ax control plane
participant EG as Egress gateway (atenet-egress)
participant RT as Substrate router (atenet-router)
participant RN as Runner (actor in gVisor)
participant RC as Artifact receiver (host)
R->>R: render Model, Workspaces, Task to task.yaml, read the Model's key from env or .env
R->>AX: ax apply -f task.yaml (creates the actor, egress denied)
R->>EG: EgressPolicy for this actor only
R->>AX: ax resume task (retried while the golden snapshot builds)
loop until phase Running
R->>AX: ax get task
end
R->>RT: POST /ousast/v1/start {run, task, credentials}
RT->>RN: forwarded to this task's actor only
RN->>RN: materialise Workspaces (git at pinned commits), after the start
RN->>RC: GET /inputs/producer/artifact (through the gateway)
RN->>RN: python -m openultrasast.plane.tasks.MODULE (key in its env only, metered budget)
RN->>RC: POST tar of OUSAST_OUTPUT_DIR (through the gateway) = completion
RN->>RN: write the completion marker
R->>AX: ax delete task, then delete the EgressPolicy
R->>R: summary.json into state.json, plane status writes attribution.json
```
What each step guarantees:
- **Credentials** (`router.py`, `runner.py`). The reconciler reads the variable the Model's
`secretKey.key` names from the operator's environment or `.env` (which never overrides an
exported variable). The key exists only in the start request; the runner keeps it in memory,
passes it to the command's environment, redacts it from echoed stderr and never writes it.
- **Egress** (`egress.py`). One EgressPolicy per actor, deny by default, hostnames only: plain
HTTP to the artifact receiver, TLS passthrough to the source hosts of the bound Workspaces and
to the hosts the bound Model declares (`openultrasast.io/egress-hosts`). It is written between
`ax apply` and the resume, and deleted with the actor.
- **Inputs** (`reconciler.py`). Artifacts other tasks produced are never rendered into the
manifest; the runner fetches them from the receiver after the start, and the receiver serves a
running task its declared inputs only.
- **Completion** (`runner.py`). ax has no Completed phase: the runner's delivery of the output tar
is the task's completion. A completion marker stops a resumed actor from repeating billed work;
when the marker says the delivery failed, a later boot retries the delivery alone.
- **Budgets** (`plane/budget.py`). The metered client refuses the next call once a task's `usd` or
`calls` ceiling is reached; the task ends `unfinished` and resumes on a rerun with a larger
budget. Model-free tasks run with `{usd: 0, calls: 0}`, so a stray call fails loudly.
- **Attribution.** `ousast plane status RUN` prints per task its status, `calls`, `prompt`,
`cache_hit` and `output` tokens and `usd` from the tasks' `summary.json`, and writes
`attribution.json` in the run directory (`$OUSAST_RESULTS/plane/RUN/`, default
`~/ousast-results/plane/`).
!!! note "The golden snapshot"
Agent Substrate boots every new actor template once as a *golden* actor to snapshot it, with
the same image, Task and environment, and inside the sandbox nothing tells that boot from the
real one. So the runner's command never starts by itself: it waits for the start request,
which the reconciler sends through `atenet-router` to the task's actor, never to the golden
one. Workspaces are prepared only after the start, so the golden boot touches no network and a
started actor already runs under its egress policy (a clone at boot failed on the live cluster
while the actor was being restored, 2026-09-29; `runner.py`, `_run_once`). The first resume of
a new image can time out while the snapshot is built, which is why the reconciler retries it
for up to `OUSAST_RESUME_TIMEOUT` seconds (900).
## The Runs: validation and loop
`ousast plane workspaces POPULATION --validation-set ...` (`generate.py`) writes one chain per
case; `--loop` adds the `alerts` step per case and the loop's singleton steps. Every step except
the verify passes (and `roles` in harvest Runs) is model-free.
### Per case
```mermaid
flowchart LR
facts["facts: repo-facts"] --> va["va: verify pass a"]
facts --> vb["vb: verify pass b"]
va --> agree["agree: a vs b"]
vb --> agree
agree -- "disputed only" --> vc["vc: verify pass c"]
va --> final["final: 2-of-3"]
vb --> final
vc --> final
final --> features["features: one record per candidate"]
alerts["alerts: quick scan on vulnerable and fixed pins (--loop)"] -.-> features
features --> remember["remember: memory.jsonl"]
alerts -.-> remember
final --> remember
```
### The loop (with `--loop`)
```mermaid
flowchart LR
store[("Memory store")] -- "seeded before the Run, after the train-on-test guard" --> snap["memory-snapshot"]
rem["remember of every case"] --> measure["loop-measure"]
snap --> measure
measure --> propose["loop-propose: rules M1 and M2, M3 advisory"]
snap -- "index.json" --> propose
propose --> improve["loop-improve: one evolve round, unchanged validator and gate"]
improve --> outcome["gate.json, journal.json, and rule_policy.json only if accepted"]
```
- **Before the Run** `ousast plane run` seeds from the store (`memory.seed`): a task whose
`openultrasast.io/memory-key` (repository, pin, candidates digest, runner image digest) has
stored facts gets them and is marked done, and the sandboxed `memory-snapshot` task gets the
store's rows after the guard. **After the Run** it ingests every delivered `memory.jsonl`
(`ousast plane remember RUN` repeats this by hand). See [Memory](memory.md).
- The loop's rules and its guard are described in
[Architecture](architecture.md#proposals-from-plane-memory). Adopting an accepted ledger stays a
maintainer commit (`src/openultrasast/plane/tasks/loop.py`).
- `ousast plane harvest` writes the decision engine's harvest Runs (verify a/b with agree, and
model `roles`); `ousast plane alerts-engine` produces a Run's `alerts` from the Joern engine
image on the host for PHP and other languages quick mode does not cover.
## Tasks
| Task | Model | Does |
| --- | --- | --- |
| `repo-facts` | none | functions per product file, and each candidate's call sites in other files |
| `verify` (passes a, b, c) | `deepseek-flash` | batched tool hunt per file over the candidates, with their known callers |
| `agree` | none | a and b agreed or disputed; after pass c on the disputed, 2-of-3 (the step `final`) |
| `features` | none | one feature record per candidate for the decision engine |
| `remember` | none | the case's artifacts as memory rows |
| `alerts` | none | the quick scan on the vulnerable and fixed pins (loop Runs) |
| `loop` | none | the steps `snapshot`, `measure`, `propose`, `improve` |
| `roles` | `deepseek-flash` | per-repository source, sink and sanitizer roles without a vocabulary |
Source: [ax on this host](ops/ax/README.md#tasks).
docs/rustfs.md
+184
View File
@@ -0,0 +1,184 @@
# RustFS setup for the memory store
The plane's memory store can live in an S3-compatible bucket: `OUSAST_MEMORY=minio://sast-memory`
selects `MinioStore` (`src/openultrasast/plane/memory.py`), which works with RustFS and MinIO. The
maintainer's server is RustFS at `files.because-security.com`, bucket `sast-memory`, store at the
bucket root. What the store keeps there is described on [Memory](memory.md).
**The store never configures its bucket.** An admin sets the bucket up once with admin
credentials. The agent account the store runs as can read the bucket's settings but not change
them, and reads and writes objects freely. Every time the store opens, it checks the setup and
refuses to start if anything is missing.
```mermaid
flowchart LR
admin["Admin, once: versioning on, runs/ expiry rule"] --> bucket[("Bucket sast-memory")]
agent["Agent account: read settings, read and write objects and tags"] --> verify{"verify_bucket at startup"}
bucket --> verify
verify -- "all present" --> store["MinioStore serves reads and writes"]
verify -- "anything missing" --> refuse["MemoryStoreError naming each missing piece and its admin command"]
```
## One-time admin steps
Run these with **admin** credentials (not the agent's). Both use the AWS CLI against the RustFS
endpoint.
**1. Turn versioning on.** Provenance cites object version ids.
```bash
aws s3api put-bucket-versioning \
--endpoint-url https://files.because-security.com \
--bucket sast-memory \
--versioning-configuration Status=Enabled
```
**2. Expire raw run outputs under `runs/` after 30 days.** Everything else (`repos/`, `facts/`,
`index.jsonl` and the blobs) is kept, so the rule must be a plain prefix filter on `runs/`.
```bash
aws s3api put-bucket-lifecycle-configuration \
--endpoint-url https://files.because-security.com \
--bucket sast-memory \
--lifecycle-configuration '{"Rules":[{"ID":"ousast-runs-expiry","Status":"Enabled","Filter":{"Prefix":"runs/"},"Expiration":{"Days":30}}]}'
```
`put-bucket-lifecycle-configuration` **replaces every rule on the bucket**. If the bucket already
has rules, merge this one into them first. The rule id `ousast-runs-expiry` and the 30 days are the
store's defaults (`RUNS_RULE_ID`, `RUNS_EXPIRE_DAYS`); the store accepts any enabled, untagged
prefix rule on `runs/` with a number of days.
To read the result back:
```bash
aws s3api get-bucket-versioning --endpoint-url https://files.because-security.com --bucket sast-memory
aws s3api get-bucket-lifecycle-configuration --endpoint-url https://files.because-security.com --bucket sast-memory
```
## The agent policy
Attach this policy to the agent account (the account whose keys go into `.env`). It is the
complete policy: nothing else is needed, and it grants no `Put*` on versioning or lifecycle.
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "BucketSettingsReadOnly",
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:ListBucketVersions",
"s3:GetBucketLocation",
"s3:GetBucketVersioning",
"s3:GetLifecycleConfiguration"
],
"Resource": ["arn:aws:s3:::sast-memory"]
},
{
"Sid": "ObjectsReadWrite",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:GetObjectVersion",
"s3:PutObject",
"s3:DeleteObject",
"s3:DeleteObjectVersion",
"s3:GetObjectTagging",
"s3:PutObjectTagging",
"s3:GetObjectVersionTagging"
],
"Resource": ["arn:aws:s3:::sast-memory/*"]
}
]
}
```
| Statement | Resource | In one line |
| --- | --- | --- |
| `BucketSettingsReadOnly` | `arn:aws:s3:::sast-memory` | The agent can list the bucket and its versions and *read* its location, versioning and lifecycle settings, which is what the startup check needs; it cannot change them. |
| `ObjectsReadWrite` | `arn:aws:s3:::sast-memory/*` | The agent reads, writes and deletes objects and their versions and reads and writes object tags; this covers row objects, blobs, the index, the Select probe, and the `kind` tag the store filters by. |
So the agent writes objects freely but cannot change bucket settings: turning versioning off or
removing the expiry rule needs the admin.
## The `.env` settings
The store reads its endpoint and credentials from `.env` in the working directory or from the
environment (`minio_settings`), never from a manifest, and never prints them.
| Variable | Value for this server | Notes |
| --- | --- | --- |
| `OUSAST_MEMORY` | `minio://sast-memory` | `minio://<bucket>[/<prefix>]`; unset means the local `FileStore` |
| `MINIO_ENDPOINT` | `files.because-security.com` | host[:port] without a scheme (the `minio` SDK's endpoint); required |
| `MINIO_ACCESS_KEY` | the agent account's access key | required |
| `MINIO_SECRET_KEY` | the agent account's secret key | required |
| `MINIO_SECURE` | `true` | TLS; `0`, `false`, `no` or `off` turn it off; default `true` |
| `MINIO_REGION` | optional | spares a GetBucketLocation call |
- **Never commit `.env`.** It is gitignored (`.gitignore`: `.env`, `.env.*`, except
`.env.example`). `.env.example` lists these names with empty values.
- **`.env` never overrides a variable already exported in your shell** (`config.load_dotenv`).
A stale `export MINIO_SECRET_KEY=...` in your profile silently wins over the file; `unset` it
when a change in `.env` seems to have no effect.
- The store needs the `minio` extra (`uv sync --extra minio`); without it opening the store fails
with `OUSAST_MEMORY=minio://... needs the minio SDK: install openultrasast[minio]`. A missing
variable fails with `OUSAST_MEMORY=minio://... needs MINIO_ENDPOINT, MINIO_ACCESS_KEY,
MINIO_SECRET_KEY in .env or the environment` (naming only the missing ones).
## What the store verifies at startup
`MinioStore.verify_bucket()` runs once per process for each bucket and prefix. It reads the
versioning status and the lifecycle rules, writes a two-line probe object at
`_probe/select.jsonl` (under the prefix, if any), queries it with S3 Select and reads its tags.
Every missing piece is collected into one error:
```text
memory minio://sast-memory: the bucket is not set up for the store. The store never configures its bucket (an admin sets it once, the store verifies); missing:
- <one line per missing piece>
```
The lines, exactly as the code writes them for this bucket at the root (`<failure>` is the
server's error code and the first 200 characters of its message):
| Check | Refusal line |
| --- | --- |
| versioning unreadable | `versioning cannot be read (<failure>): the agent needs s3:GetBucketVersioning on arn:aws:s3:::sast-memory, and versioning must be Enabled` |
| versioning off | `versioning is Off, it must be Enabled (provenance cites object versions). Admin: aws s3api put-bucket-versioning --endpoint-url "$MINIO_ENDPOINT" --bucket sast-memory --versioning-configuration Status=Enabled` (the actual status, e.g. `Suspended`, replaces `Off` when set) |
| lifecycle unreadable | `the lifecycle configuration cannot be read (<failure>): the agent needs s3:GetLifecycleConfiguration on arn:aws:s3:::sast-memory, and a rule must expire runs/` |
| a rule expires everything | `lifecycle rule(s) <rule ids> expire the whole store (repos/, facts/ and the index would be deleted); limit them to runs/` |
| no `runs/` rule | `no enabled lifecycle rule expires runs/ after a number of days. Admin (merged with any rules the bucket already has): aws s3api put-bucket-lifecycle-configuration --endpoint-url "$MINIO_ENDPOINT" --bucket sast-memory --lifecycle-configuration '{"Rules":[{"ID":"ousast-runs-expiry","Status":"Enabled","Filter":{"Prefix":"runs/"},"Expiration":{"Days":30}}]}'` |
| tags unreadable | `object tags cannot be read on _probe/select.jsonl (<failure>): the store filters row objects by their kind tag, so the agent needs s3:GetObjectTagging on arn:aws:s3:::sast-memory/*` |
| S3 Select missing | `S3 Select (SelectObjectContent over JSON Lines) does not answer on _probe/select.jsonl (<failure>). The store requires it and has no local fallback: use a server with S3 Select (RustFS has it; some MinIO releases removed it) and let the agent s3:PutObject and s3:GetObject` |
The admin commands inside the messages use `--endpoint-url "$MINIO_ENDPOINT"`; with the
host-only `MINIO_ENDPOINT` above, pass the full URL (`https://files.because-security.com`) as in
the steps on this page.
There is **no fallback**: the store does not start in a degraded mode, does not drop to a local
store, and does not fetch and filter objects itself.
## The RustFS Select limit and the fixed-schema rule
RustFS's S3 Select infers an object's JSON schema from its leading rows. A `where` field that is
absent from those rows is unbound, even if a later row has it: the query fails with
`EvaluatorBindingDoesNotExist` instead of matching (measured 2026-10-02 with the field first
present at row 5000). The store does not read that as "no rows", since that answer could silently
drop matches; it raises:
```text
memory minio://sast-memory: S3 Select on <key> failed (EvaluatorBindingDoesNotExist: ...) (a field of [<fields>] is absent from the object's leading rows, which the server reads as its schema); the store has no local fallback
```
The rule that follows: a field the store filters on must be present in every row of its kind
from the first row on, i.e. each row kind has a fixed set of queryable fields. Enforcing that set
per kind in the store is in progress and not on main as of this page (see
[Memory](memory.md#queryable-fields-per-kind-in-progress)).
## Checks against the real server
The contract tests that talk to a real server run only with `OUSAST_MEMORY_TEST_MINIO=1` (bucket
`OUSAST_MEMORY_TEST_BUCKET`, else `MINIO_BUCKET`). They verify the bucket at its root and write
only under a fresh `contract-<id>/` prefix; they never configure the bucket. More on the host
setup: [ax on this host](ops/ax/README.md).
docs/scanning.md
+125
View File
@@ -0,0 +1,125 @@
# Scanning
`ousast scan PATH --mode quick|standard|deep` scans a checkout; `ousast pre-push` checks only
what a push (or an explicit `--base`/`--head` pair) changes. The stage-by-stage pipeline,
evidence ladder and score are in [Architecture](architecture.md); this page shows the flows.
## The three scan modes
The stages a mode requests are `MODE_STAGES` in `src/openultrasast/stages.py`: `quick` runs
STATIC, `standard` adds MAP, `deep` adds REGRESS.
```mermaid
flowchart TD
start["ousast scan PATH --mode M"] --> policy["policy_load, ruleset_load, policy_check"]
policy --> prep["static_mapping, preprocess, entry_point_mapping, rank, calibrate"]
prep --> which{"mode"}
which -- "quick, deep" --> qf["quick_findings: language-scoped pattern rules"]
which -- "standard" --> hp["hunter_pool: the same rules per ranked target"]
qf --> verify["verify, record_calibration, score"]
hp --> verify
verify --> mapq{"MAP requested?"}
mapq -- "quick: no" --> out["report.md, report.sarif, manifest.json"]
mapq -- "standard, deep" --> map["MAP: complexity map, semantic overlay, obligations, Joern model layer"]
map --> regq{"REGRESS requested?"}
regq -- "standard: no" --> out
regq -- "deep" --> reg["REGRESS: snippets in a Docker sandbox, verdicts.json"]
reg --> out
```
| Mode | Adds | Needs | Without it |
| --- | --- | --- | --- |
| `quick` | pattern rules, entry-point hints, ranking, verification, score | nothing | - |
| `standard` | MAP: complexity map, semantic overlay, authorization obligations, the Joern model layer; `fusion` (deterministic) | Joern; a provider key for the residual LLM question | `cpg_unavailable` degradation; the LLM parts are skipped and recorded |
| `deep` | REGRESS: promoted candidates loaded in a sandbox with no network, read-only source, non-root | a working `docker` | `sandbox_unavailable` degradation |
Sources: `README.md` (scan modes), `src/openultrasast/cli.py` (`run_stage` calls),
[Architecture](architecture.md#the-scan-pipeline).
## Quick rules per language
Rules run only against files of their language. The bundled ruleset lives in
`src/openultrasast/ruleset/<language>/rules.toml`.
| Language | Classes (CWE) |
| --- | --- |
| C / C++ | stack buffer overflow (121), format string (134), command injection (78) |
| Python | command injection (78), code injection (95), SSTI (94), deserialization (502), SQLi (89), path traversal (22), SSRF (918), weak hash (327), insecure default (489) |
| JavaScript / TS | command injection (78), code injection (95), XSS (79), SQLi (89), path traversal (22), SSRF (918), weak hash (327), deserialization (502) |
| Java + Groovy templates | command injection (78), SQLi (89), weak hash (327), deserialization (502), unescaped template XSS (79) |
| PHP | SQLi (89), command injection (78), file inclusion (98), path (22), open redirect (601), SSRF (918); code eval (95), unserialize (502) and echo XSS (79) ship as `shadow` rules |
The detection gate (`python -m openultrasast.gate`) enforces at least 90% recall and under 10%
false positives per language on the bundled corpora in `benchmarks/manifests/`. Those corpora are
cheat-sheet fixtures: they catch regressions, they do not estimate recall on real code. **PHP is
not in the gate**; its numbers are in-sample development numbers
(`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`). See `README.md` for the
gate's per-language output.
## The Joern engine
The model layer (`src/openultrasast/model/`, `src/openultrasast/cpg/`) is the core of `standard`.
It builds one code property graph per scan with Joern (`joern-parse`, then batched
`joern --script` queries from `cpg/queries/*.sc`) and sends each question to the arbiter of its
type: **taint** (source to sink with no discharging sanitizer), **dominance** (does a guard govern
the obligated operation) and **configuration** (constant abstraction over a value). Taint facts
ship for C, Java, JavaScript, PHP and Python (`src/openultrasast/ruleset/semantic/*.toml`).
```mermaid
flowchart LR
src["Checkout"] --> parse["joern-parse: one CPG per repository"]
parse --> q["Batched joern --script queries"]
q --> arb{"Arbiter by question type"}
arb --> taint["Taint: flow with no discharging sanitizer"]
arb --> dom["Dominance: obligation with no guard"]
arb --> conf["Configuration: permissive value"]
taint --> rung["Rung: model_entailed or suspicion"]
dom --> rung
conf --> rung
rung -- "residual question only" --> judge["LLM judge: confirm or contradict, never create a flow"]
```
The Docker image ships Joern and `php-cli` (which the PHP frontend needs); without Joern the
layer records `cpg_unavailable` and the scan is otherwise unchanged. Engine installation and the
Joern features used are in [Engine and pre-push](ops/README.md).
## Framework priors
Rule and taint-fact entries that know a framework or library carry a `framework` or `library` tag
naming a row of `src/openultrasast/ruleset/frameworks.toml` (for example `wordpress`, `flask`,
`django`, `express`, `spring`). The loaders take `priors=`: `"all"` (the default and today's
behaviour), `"off"` (language-level knowledge only) or a set of ids, so the contribution of
framework knowledge can be measured and switched off. The decision engine's first slice ran with
priors off ([Decision engine](decision-engine.md#first-measured-slice-2026-10-01)).
## Pre-push: the delta check (experimental)
`ousast pre-push` analyses the commits a `git push` would publish, or an explicit
`--base`/`--head` pair, and compares head with base so that only new or worsened defects are
candidates. Sources: `ousast pre-push --help`, `src/openultrasast/push/`,
[Engine and pre-push](ops/README.md#explicit-local-replay).
```mermaid
flowchart TD
input["git push refs on stdin, or --base / --head"] --> resolve["Resolve both commit identities"]
resolve --> snap["Materialise tracked bytes in private scratch storage"]
snap --> regions["Discover changed regions, up to --max-regions"]
regions --> engine["Evidence ranker and arbiter on head"]
engine --> cmp["Targeted base comparison: new or worsened only"]
cmp --> admit{"Admission policy: capability registry"}
admit -- "registry empty today" --> diag["Diagnostic candidates in the artifact"]
admit -- "an enabled capability" --> alert["Normal alert"]
diag --> art["JSON artifact outside the repository"]
alert --> art
art --> exit{"--mode"}
exit -- "advisory, the default" --> allow["Exit 0: allow"]
exit -- "blocking, explicit opt-in" --> block["May block the push"]
```
- `--deadline` (30 s by default) bounds preparation, both revisions and reporting together;
`--incomplete-coverage block` opts into failing when coverage is incomplete.
- The default capability registry is empty pending independent qualification, so no normal alert
is emitted today; every candidate stays diagnostic.
- A model is called only with an explicit `--model-config`.
- Hook installation (`ops/install-pre-push`) and the private cache (`--cache-dir`) are documented
in [Engine and pre-push](ops/README.md#experimental-pre-push-integration).