diff --git a/.gitignore b/.gitignore index e24654d..9dd8031 100644 --- a/.gitignore +++ b/.gitignore @@ -33,3 +33,6 @@ vector-store/ # own scratch dir so it never lands here, but a maintainer querying a CPG by hand from the repo root # will create one. workspace/ + +# MkDocs build output (mkdocs build) +site/ diff --git a/README.md b/README.md index 81d5f6e..6dbb282 100644 --- a/README.md +++ b/README.md @@ -221,6 +221,8 @@ shell, Docker or a free-form command. ## Further reading +Documentation site with flow diagrams (`docs/`, `mkdocs.yml`): `uv sync --extra docs && uv run mkdocs serve`. + - [docs/architecture.md](docs/architecture.md): the scan pipeline, evidence ladder, CWE policy and project score, and the self-improving loops. - [docs/evaluation.md](docs/evaluation.md): independent populations, the qualification gates, diff --git a/docs/decision-engine.md b/docs/decision-engine.md index a92209b..401986a 100644 --- a/docs/decision-engine.md +++ b/docs/decision-engine.md @@ -80,6 +80,6 @@ v3 check are still ahead. | `ousast learn curve --program ID` | the metric over memory size on a fixed candidate subset | | `ousast learn audit-leaks` | how well each signal alone separates the sides of a pair (no model) | -The memory store is the plane's (`OUSAST_MEMORY`; see [ops/ax/README.md](../ops/ax/README.md)). +The memory store is the plane's (`OUSAST_MEMORY`; see [ops/ax/README.md](ops/ax/README.md)). The harvest that produced the plane signals runs as plane Runs generated by `ousast plane harvest` (`benchmarks/measurements/2026-10-01-decision-engine-harvest/record.json`). diff --git a/docs/evaluation.md b/docs/evaluation.md index ad76b4c..63b86d4 100644 --- a/docs/evaluation.md +++ b/docs/evaluation.md @@ -49,6 +49,43 @@ v3 is reserved: development and analysis work never reads its cases, and `tests/test_independent_population.py` fails when anything in the development tree (a recipe, catalog, manifest or measurement) names one of its repositories. +```mermaid +stateDiagram-v2 + [*] --> Frozen: cases selected and frozen before any scan + Frozen --> Scored: scored once under the pre-registered protocol + Scored --> Spent: result recorded + Spent --> [*]: may inform changes, can no longer qualify anything + note right of Frozen + v3 (PHP, 15 cases) waits here as the one-time final check, + run once after a prediction is committed + end note + note right of Spent + v1 and v2 are spent + end note +``` + +## Folds: how learned decisions are kept out of their own training data + +The decision engine is evaluated on repositories it was not shown (`src/openultrasast/learn/folds.py`). +Every split is by repository group (the label builder's normalised `owner/name`, merged across +URLs, forks and shared advisories), across all corpora, seeded and deterministic. + +```mermaid +flowchart TD + groups["All repository groups with labels"] --> c["Compile split: 25% of groups, never evaluated"] + groups --> rest["The other 75%"] + c --> boot["C_boot 60%: demonstrations"] + c --> val["C_val 40%: instruction scoring"] + rest --> outer["Outer grouped 5-fold: calibration cross-fitted, operating points nested"] + rest --> loso["Leave-one-source-out: a source held out whole"] + rest --> lofo["Leave-one-framework-out: frameworks with at least 10 groups"] +``` + +`assert_disjoint` runs before a compile: the compile split and the evaluation folds share no +group. A framework with fewer than 10 groups gets `insufficient data` instead of a fold. As of +2026-10-02 only the outer folds of the injection family have been run +([decision-engine.md](decision-engine.md#first-measured-slice-2026-10-01)). + ## The plane increment The first increment of the agentic plane ran the model-driven pipeline's validation set (46 diff --git a/docs/examples.md b/docs/examples.md index 926b8ed..0d642b1 100644 --- a/docs/examples.md +++ b/docs/examples.md @@ -69,7 +69,7 @@ uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml ## 7. Running the agentic plane Model work over whole repositories runs as a Run manifest on ax, one isolated actor -per task (bring-up: [ops/ax/README.md](../ops/ax/README.md)). The provider key is +per task (bring-up: [ops/ax/README.md](ops/ax/README.md)). The provider key is read from the environment (or `.env`, which never overrides an exported variable) by `ousast` and sent only in each task's start request. diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..d6f7bce --- /dev/null +++ b/docs/index.md @@ -0,0 +1,54 @@ +# OpenUltraSAST + +OpenUltraSAST is an experimental static security analyser meant to work as a safety net on +repositories it has never seen: it scans a checkout, or the commits a `git push` would publish, +and separates what it can prove from what it merely suspects. A scan that could not look at +something records a degradation instead of reporting a clean result. + +**State as of 2026-10-02 (v1.2.0-alpha.1):** the pre-push safety net is **NO-GO** for rollout. +No hook capability is enabled, and no independent population has passed the qualification gates +(see [Evaluation](evaluation.md)). + +## Three layers + +| Layer | What it is | Where | +| --- | --- | --- | +| **Scanners** | `ousast scan` (`quick`, `standard`, `deep`) and `ousast pre-push`: language-scoped pattern rules, a Joern code property graph engine, verification and scoring. Deterministic in `quick`; no model is needed for a scan. | [Scanning](scanning.md), [Architecture](architecture.md) | +| **The agentic plane on ax** | Model-driven work over whole repositories, run as a DAG of tasks on google/ax, one isolated actor per task. What the runs learn is kept in a memory store. Maintainer surface: `ousast plane`. | [The plane on ax](plane.md), [Memory](memory.md) | +| **The decision engine** | A learned classifier that weighs every instrument's signals against similar labelled cases from memory. **In development, not adopted**: no scan, pre-push check or report uses it. Maintainer surface: `ousast learn`. | [Memory and detection](memory-and-detection.md), [Decision engine](decision-engine.md) | + +## How they connect + +```mermaid +flowchart LR + repo["Repository or push delta"] --> scan["Scanners: quick rules, Joern engine, verify, score"] + scan --> report["Report: findings with evidence, SARIF, score"] + scan -- "alerts on vulnerable and fixed pins" --> plane["Plane on ax: repo-facts, verify a/b/c, agree, features"] + plane -- "remember" --> mem[("Memory store: rows, facts, examples, blobs")] + mem -- "improve --memory: rule-status proposals" --> gate{"Validator and detection gate"} + gate -- "accepted round" --> ledger["Ruleset ledger: rule status"] + ledger --> scan + labels["Labels from ground truth"] --> mem + mem -- "retrieval of similar labelled cases" --> engine["Decision engine: compiled AI classifier"] + engine -. "intended, not adopted: BLOCK / ADVISORY" .-> report +``` + +- The **scanners** produce findings and, inside plane Runs, the `alerts` that become memory rows. +- The **plane** runs model work (verify passes, roles) and model-free bookkeeping (facts, agreement, + feature records) per case; its `remember` task turns a case's artifacts into rows of the + [memory store](memory.md). +- Memory feeds two consumers. `ousast improve --memory` turns stored rows into rule-status + proposals that pass through the same validator and gate as every other edit. The decision engine + retrieves labelled examples from the same store. +- The decision engine's output (dotted line) is the intended path. Nothing user-facing reads it + today. + +## Where to go next + +- Run a scan: [Examples](examples.md), and the commands in [Scanning](scanning.md). +- Understand a finding's evidence and score: [Architecture](architecture.md). +- Operate the plane and its store: [ax on this host](ops/ax/README.md) and + [RustFS setup](rustfs.md). +- Trust boundaries and hardening: [Threat model](threat-model.md). + +Every number in these pages is quoted from a committed record whose path is given next to it. diff --git a/docs/memory-and-detection.md b/docs/memory-and-detection.md new file mode 100644 index 0000000..77ee9d7 --- /dev/null +++ b/docs/memory-and-detection.md @@ -0,0 +1,156 @@ +# Memory and detection + +How what the system remembers is meant to turn into better decisions on code it has not seen. +This is the design of the learned decision engine (`src/openultrasast/learn/`), which is **in +development and not adopted**: no scan, pre-push check or report uses it today. Every user-facing +decision is still the deterministic one described in [Scanning](scanning.md) and +[Architecture](architecture.md). The engine's commands and status are on +[Decision engine](decision-engine.md); the store itself is on [Memory](memory.md). + +## From instruments to a decision + +```mermaid +flowchart TD + inst["Instruments: quick rules, Joern engine, verify passes and agree, model roles, repo facts"] --> feat["Feature record per candidate: path, function, family"] + feat --> prof["Input profile v1: engine and function length withheld"] + lab["Labels from ground truth: pairs, populations v1/v2, advisory fixes, adjudications"] --> ex["Example rows in memory: label + features + excerpt"] + prof --> ret["Retrieval: signal profile, then code embeddings"] + ex --> ret + ret --> bnd{"Evaluation boundary"} + bnd --> prog["Compiled AI classifier: instruction, demonstrations, up to 6 examples, candidate"] + prog --> score["Score s: mean over k samples"] + score --> cal["Platt calibration, cross-fitted on held-out repositories"] + cal --> op{"Operating point"} + op -- "calibration holds and precision reachable" --> block["BLOCK"] + op -- "otherwise" --> adv["ADVISORY"] +``` + +The intended place for these decisions is changed code: `learn/features.py` builds feature records +for a whole-repository scan or for a base..head delta (`build_for_scan` with `Delta`), the same +delta `ousast pre-push` checks. Integration into scan and pre-push is still ahead. + +## 1. Instruments produce signals, not verdicts + +Each instrument contributes one block to a candidate's feature record (`learn/schema.py`, +`INSTRUMENTS`): `language`, `quick` (quick rules), `engine` (the Joern model layer), `facts` (repo +facts, such as callers and function span), `source`, `entry_points`, `roles` (model source, sink +and sanitizer roles), `delta`, `verify` and `agree` (the plane's verify passes and their +agreement), and `model_sinks`. A candidate is `(path, function, family)` of one repository and pin +where at least one instrument emitted a signal of that family (`learn/features.py`). + +An instrument that did not run, had no coverage or failed records that state (`none`, `failed`, +`not_applicable`), never a zero, so "silent" and "did not look" stay distinguishable. None of the +instruments decides alone. + +**Why profile v1 withholds the engine and function length.** A leak audit +(`ousast learn audit-leaks`, `benchmarks/measurements/2026-10-01-decision-engine-leak-audit/`) +found that both separate the vulnerable and fixed sides of a pair by construction: a fix removes +a sink, and a guard lengthens the function. That holds on the pair corpus, not on unseen code, so +`INPUT_PROFILES["v1"]` drops the `engine` block and `facts.function_lines` from prompts and from +the retrieval distance. + +## 2. Labels come from ground truth only + +`ousast learn labels` (`learn/labels.py`) reads only the sources on a fail-closed allow-list +(`learn/sources.toml`): `population-v1`, `population-v2`, `dev-php` (reviewed recipe sites), +`pairs`, `advisory-fixes`, `adjudications` and `assumed_benign`. A file the list does not name is +refused before it is opened; the reserved population v3 is excluded unread. + +- **Positives** are declared sites and the vulnerable side of a fix; **negatives** are the same + function on the fixed side where the fix changed it, and recorded `False`/`FP` adjudications. + A vulnerable-pin candidate matching no site is unlabelled, never negative. +- **Assumed benign** (maintainer decision 2026-09-30): ordinary non-security commits of the training + repositories, at least 90 days from any fix, with no later security commit touching the same + files. They are their own source with weight 0.5, conditional on a signal at the tip and none at + the base, and reported separately, never merged with verified negatives. +- **Plane verdicts are features, not labels.** "Agreed", "rejected" or "disputed" is the output of + the verify instrument; it enters the record as the `verify` and `agree` blocks. Using it as a + label would teach the engine to reproduce the plane, mistakes included, so "rejected by the + plane" is never a negative (`learn/labels.py`: "A plane verdict ... is never a label"). + +`ousast learn memory build` joins each label with the candidate's feature record and an excerpt of +its function and stores one `example` row (no rationale) plus the excerpt blob. + +## 3. Retrieval of similar labelled cases + +`learn/retrieve.py`, no model: + +1. **Signal profile.** A Gower distance over the allow-listed features of the active profile, + per instrument block, with missing values explicit (a different state counts as distance 1). + The nearest 40 examples of the candidate's family and the nearest 40 of other families go on. +2. **Code embeddings.** A cosine re-rank of the excerpts' embeddings (OpenAI + `text-embedding-3-small` through OpenRouter, 1,536 dimensions, cached by excerpt hash): + `r = 0.5 (1 - D) + 0.5 cos`. Without the candidate's vector the retrieval is signals only. +3. **Balance.** Six examples: at most half per label, at most 2 per repository group, at most 2 + contrast examples from other families. + +**The evaluation boundary.** `eligible()` is the only way an example enters a prompt, for retrieved +examples and compiled demonstrations alike: never the candidate's own repository group, never a +group the fold evaluates, never the held-out source or framework, and (outside deployment) never +a near-duplicate of the candidate (cosine >= 0.98). The program repeats the check on the ids it +actually rendered before any call. + +## 4. The AI classifier, compiled + +`learn/program.py` and `learn/compile.py`, DSPy-style and in-house. The prompt is the instruction +plus the compiled demonstrations (a prefix identical for every candidate, so the provider's prefix +cache hits), then the retrieved examples, then the candidate's excerpt, signals, roles and family. +The model (`deepseek-flash`) answers `vulnerable`, `not_vulnerable` or `unsure` with a confidence +and a rationale that must cite a line of the code. It is sampled `k` times; the score `s` is the +mean over samples, and a majority `unsure` can never BLOCK. Compilation bootstraps demonstrations +and searches instructions on a compile split of repository groups, scored by a pre-registered +balanced Brier score (`learn/compile.toml`); the program is stored as `programs/.json`. + +## 5. Calibration and the two operating points + +`learn/calibrate.py`: Platt scaling of `s`, cross-fitted so that no map is checked on the fold it +was fitted on. Calibration **holds** when the slope interval contains 1, the intercept interval +contains 0 and ECE <= 0.05. **BLOCK** is offered only when calibration holds and a fold reaches the +required precision; otherwise the engine offers **ADVISORY** only. + +## The loop: memory improves decisions without retraining + +```mermaid +flowchart LR + new["New labels and adjudications"] --> labels["ousast learn labels"] + labels --> build["ousast learn memory build, embed"] + build --> mem[("Memory store: example rows")] + mem --> retr["Retrieved per candidate at decision time"] + retr --> prog["Same compiled program"] + prog --> dec["Decision"] + runs["Plane runs: alert, verdict, coverage rows"] --> mem2[("Memory store: run rows")] + mem2 --> imp["ousast improve --memory: M1, M2 proposals"] + imp --> gate{"Unchanged validator and gate"} + gate -- "accepted round" --> ledger["Rule status in the ledger"] +``` + +- **Decisions.** The compiled part of the program is its instruction and demonstrations; the + examples are retrieved from memory for each candidate. A new `example` row is therefore eligible + for retrieval in the next decision without recompiling the program, subject to the same + evaluation boundary. `ousast learn curve` measures the metric over memory size on a fixed + candidate subset. +- **Rules.** `ousast improve --memory` reads the run rows (alerts, verdicts, coverage) and proposes + rule-status edits only (shadow a rule that is repeatedly false across repositories; re-enable a + shadow rule that keeps hitting agreed declared sites). Rows from holdout pairs, the gated + manifest's own cases and any `--qualify-population` are dropped first, and every proposal goes + through the same validator and gate as any other edit (`improve/memory.py`). + +## What the first slice measured + +Record: `benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json` (counts +only). Family **injection**; memory of 724 static examples from all families; input profile v1; +framework priors off; out-of-repository evaluation (compile split never evaluated, outer grouped +5-fold, calibration cross-fitted, operating points nested); 95% repository-cluster bootstrap +intervals. + +| Measure | Value | What it means | +| --- | --- | --- | +| Candidates | 110 in 29 repository groups, 58 positive | the positive share is 58/110 = 0.53 | +| AUC of the score `s` | 0.74 (0.65 - 0.85) | the score ranks a vulnerable candidate above a non-vulnerable one about three times in four: there is ranking signal | +| ADVISORY | recall 0.90 (0.82 - 0.97), precision 0.54 (0.44 - 0.65) | precision is at the positive share (0.53), so at this threshold ADVISORY is **not discriminating**: it flags nearly as indiscriminately as flagging everything would | +| BLOCK | not offered | calibration does not hold (ECE 0.064 > 0.05), and every outer fold reported `precision_unreachable` | + +**The engine is not adopted.** These numbers come from one family and one slice; the other +families, the leave-one-source-out and leave-one-framework-out evaluations, integration into scan +and pre-push, and the one-time check on population v3 are still ahead +([Decision engine](decision-engine.md#first-measured-slice-2026-10-01)). diff --git a/docs/memory.md b/docs/memory.md new file mode 100644 index 0000000..e9f4737 --- /dev/null +++ b/docs/memory.md @@ -0,0 +1,131 @@ +# Memory + +One store, keyed by repository and pin, holds what plane runs and the decision engine learned. +The code is `src/openultrasast/plane/memory.py`; the same layout serves both backends. +How the stored rows feed detection is on [Memory and detection](memory-and-detection.md). + +## What is stored + +``` +index.jsonl one row per ingested (run, task): sha256 of its rows, row count +facts/.json repository facts by content (identical facts stored once) +repos/____/.jsonl the rows of one repository + 40-hex pin, sorted by id +excerpts/.txt code excerpts, by content +embeddings//.json an excerpt's embedding vector, per embedding model +responses/.json cached model responses +programs/.json compiled decision-engine programs +``` + +**Rows.** Every row carries `id`, `kind`, `repo`, `pin`, `run`, `task`, `population`, `split` +and `image` (`ROW_FIELDS`); `id` is the sha256 of (kind, run, task, subject), so a repeated row +replaces itself. The kinds (`KINDS`) and who writes them: + +| Kind | Written by | Holds | +| --- | --- | --- | +| `facts` | `remember` task | the facts entry of a case (points at `facts/.json`; reused by memory key) | +| `verdict` | `remember` task | per candidate: each verify pass's answer, the final outcome, whether a tie-break ran, cost | +| `unit_cost` | `remember` task | per verify pass and file: usd, calls and token usage | +| `alert` | `remember` task | a quick-scan or engine alert on the vulnerable or fixed pin (loop Runs) | +| `coverage` | `remember` task | per language and pin: whether any rule could have fired there | +| `features` | `remember` task, from the `features` task | one feature record per candidate and family | +| `proposal_outcome` | `ousast improve --memory` (not with `--dry-run`), and the loop's `improve` step | what happened to each memory proposal in a round | +| `example` | `ousast learn memory build` | a labelled case: label + feature record + excerpt sha (no rationale) | +| `label`, `decision`, `experiment`, `arm_outcome`, `experiment_result` | reserved for later decision-engine tasks | declared in `KINDS`; not written by the code on main yet | + +**Blobs** (`put_blob`) are content-addressed: a 64-hex name under a fixed prefix. Excerpts are +written by `learn memory build` (`learn/examples.py`), embeddings by `learn memory embed` +(`learn/embeddings.py`, keyed by excerpt sha so an excerpt is embedded once), responses by the +program's caller (`learn/program.py`, keyed by the sha256 of model, parameters digest, messages, +temperature, sample and format, so a rerun replays at no cost), and compiled programs by +`learn compile` (`learn/compile.py`). + +## Write path + +```mermaid +flowchart LR + tasks["Plane tasks: facts, verify a/b/c, agree, alerts, features"] --> rem["remember task: memory.jsonl"] + rem --> recv["Delivered to the host with the run's artifacts"] + recv --> ingest["ingest after ousast plane run, or ousast plane remember RUN"] + loop["Loop improve step: proposal_outcome"] --> ingest + improve["ousast improve --memory: proposal_outcome"] --> put + build["ousast learn memory build: labels + features + excerpts"] --> put["put_rows, put_blob"] + embed["ousast learn memory embed"] --> put + compile["ousast learn compile, evaluate"] --> put + ingest --> idx{"index.jsonl: same run, task and sha256?"} + idx -- "yes" --> skip["Skipped"] + idx -- "no" --> put + put --> store[("Memory store")] +``` + +- `ingest` validates every row first: a malformed file fails before anything is written. +- Rows are merged into their repository + pin object by `id`. Every object is written whole and the + index last, so an interrupted ingest leaves nothing a repeat would not repair. +- Each row object carries tags (`TAG_FIELDS`: `repo`, `pin`, `kind`, `family`, `run`, + `population`, `split`) listing the distinct values of its rows (`*` when they exceed 256 + characters). +- The plane's store failures never change a run's result: an ingest can be repeated by hand + (`cli.py`, `_plane_memory`). + +## Read path + +```mermaid +flowchart LR + store[("Memory store")] --> seed["seed before a Run"] + seed --> reuse["Facts reused by memory key: repo, pin, candidates, runner image"] + seed --> snap["memory-snapshot: rows after the train-on-test guard"] + store --> examples["learn: load example rows, excerpts, embeddings"] + examples --> retr["Retrieval of similar labelled cases"] + store --> responses["Response cache: replay at no cost"] + store --> imp["ousast improve --memory: alert, verdict, coverage rows"] + imp --> guard["Train-on-test guard"] + guard --> rules["Rules M1, M2 to rule-status proposals"] + rules --> gate{"Unchanged validator and gate"} +``` + +- **Seed** (`memory.seed`): a Run task whose `openultrasast.io/memory-key` matches stored facts gets + them and is marked done; a changed pin, candidate set or runner image finds nothing and + recomputes. The loop's `memory-snapshot` is seeded the same way, because a sandboxed task cannot + read the store. +- **Retrieval** (`learn/retrieve.py`): see + [Memory and detection](memory-and-detection.md#3-retrieval-of-similar-labelled-cases). +- **`improve --memory`** (`improve/memory.py`): rows from holdout pairs, from the gated manifest's + own cases and from any `--qualify-population` are dropped before any rule sees them. See + [Architecture](architecture.md#proposals-from-plane-memory). + +## Backends: FileStore and the S3 store + +`OUSAST_MEMORY` selects the backend (`open_store`): + +| | `FileStore` | `MinioStore` (S3: RustFS or MinIO) | +| --- | --- | --- | +| Selected by | `file:///path`, or nothing (default `$OUSAST_RESULTS/plane/memory`) | `minio://[/]` | +| Needs | a local disk; refuses to write below 1 GiB free | the `minio` extra, `MINIO_*` settings from `.env` or the environment | +| Filtered reads | read and filter locally | S3 Select pushdown (`SelectObjectContent` over JSON Lines); kind filter by object tag | +| Object version cited by provenance | the file's content hash | the bucket's object version id | +| Bucket setup | none | done once by an admin, verified at every start ([RustFS setup](rustfs.md)) | + +### Verify at startup, no fallback + +The S3 store never configures its bucket. Each time a `MinioStore` is opened for a bucket and +prefix (once per process), `verify_bucket()` checks, using reads plus one small probe object at +`/_probe/select.jsonl`: + +1. versioning is `Enabled` (provenance cites object versions); +2. an enabled lifecycle rule expires `/runs/` after a number of days, as a plain prefix + filter, and no enabled rule expires the whole store; +3. S3 Select answers a probe query over JSON Lines; +4. the probe's object tags are readable. + +Anything missing is named, with the admin command that fixes it, in one `MemoryStoreError`, and +the store does not start. There is no local or fetch-and-filter fallback: a filtered read that +S3 Select cannot answer raises a `MemoryStoreError` instead of returning "no rows". The exact +messages are listed on [RustFS setup](rustfs.md#what-the-store-verifies-at-startup). + +### Queryable fields per kind (in progress) + +RustFS's Select infers an object's JSON schema from its leading rows, so a `where` field missing +there fails with `EvaluatorBindingDoesNotExist` even when a later row has it (measured +2026-10-02; `memory.py`, `MinioStore._select`). On main today the store turns that failure into a +`MemoryStoreError` naming the field, never into an empty answer. A fixed set of queryable fields +per row kind, so that every filtered field is present in every row of its kind, is in progress +and not on main as of this page. diff --git a/docs/ops/README.md b/docs/ops/README.md new file mode 100644 index 0000000..ac79737 --- /dev/null +++ b/docs/ops/README.md @@ -0,0 +1,3 @@ +--8<-- "ops/README.md" + +*Source: `ops/README.md` in the repository; this page includes it when the docs site is built.* diff --git a/docs/ops/ax/README.md b/docs/ops/ax/README.md new file mode 100644 index 0000000..d6c8bc4 --- /dev/null +++ b/docs/ops/ax/README.md @@ -0,0 +1,10 @@ +# ax on this host + +The agentic plane runs every task on google/ax over Agent Substrate in a single-node kind cluster. ax is the +only executor; `ousast plane run` submits, resumes, starts and collects. The flow is drawn in +[The plane on ax](../../plane.md); the bucket setup for the S3 memory store is in +[RustFS setup](../../rustfs.md). + +--8<-- "ops/ax/README.md:5" + +*Source: `ops/ax/README.md` in the repository, from its line 5 on; this page includes it when the docs site is built.* diff --git a/docs/plane.md b/docs/plane.md new file mode 100644 index 0000000..8da1b26 --- /dev/null +++ b/docs/plane.md @@ -0,0 +1,144 @@ +# The plane on ax + +Model-driven work over whole repositories runs on a separate agentic plane: a **Run** +(`plane/runs/*.yaml`, kind `openultrasast.io/v1alpha1 Run`) is a DAG of steps, and each step is +an ax **Task** (`plane/tasks/`) that binds its Workspaces (`plane/workspaces/`), at most one ax +**Model** (`plane/models/`: `deepseek-flash` for chat, `openrouter-embedding` for embeddings) and +its own `budget: {usd, calls}`. ax (google/ax over Agent Substrate, in a single-node kind cluster +on the maintainer's host) is the only executor. Bring-up and the host's lessons are in +[ax on this host](ops/ax/README.md); this page draws the flow. + +Sources: `src/openultrasast/plane/reconciler.py`, `router.py`, `egress.py`, `runner.py`, +`generate.py`, `memory.py`, and `src/openultrasast/plane/tasks/`. + +## One task, end to end + +`ousast plane run RUN.yaml` (the reconciler) does this for every ready task; `--workers` bounds +the tasks in flight, and a rerun skips tasks already done. + +```mermaid +sequenceDiagram + autonumber + participant R as Reconciler (ousast plane run) + participant AX as ax control plane + participant EG as Egress gateway (atenet-egress) + participant RT as Substrate router (atenet-router) + participant RN as Runner (actor in gVisor) + participant RC as Artifact receiver (host) + R->>R: render Model, Workspaces, Task to task.yaml, read the Model's key from env or .env + R->>AX: ax apply -f task.yaml (creates the actor, egress denied) + R->>EG: EgressPolicy for this actor only + R->>AX: ax resume task (retried while the golden snapshot builds) + loop until phase Running + R->>AX: ax get task + end + R->>RT: POST /ousast/v1/start {run, task, credentials} + RT->>RN: forwarded to this task's actor only + RN->>RN: materialise Workspaces (git at pinned commits), after the start + RN->>RC: GET /inputs/producer/artifact (through the gateway) + RN->>RN: python -m openultrasast.plane.tasks.MODULE (key in its env only, metered budget) + RN->>RC: POST tar of OUSAST_OUTPUT_DIR (through the gateway) = completion + RN->>RN: write the completion marker + R->>AX: ax delete task, then delete the EgressPolicy + R->>R: summary.json into state.json, plane status writes attribution.json +``` + +What each step guarantees: + +- **Credentials** (`router.py`, `runner.py`). The reconciler reads the variable the Model's + `secretKey.key` names from the operator's environment or `.env` (which never overrides an + exported variable). The key exists only in the start request; the runner keeps it in memory, + passes it to the command's environment, redacts it from echoed stderr and never writes it. +- **Egress** (`egress.py`). One EgressPolicy per actor, deny by default, hostnames only: plain + HTTP to the artifact receiver, TLS passthrough to the source hosts of the bound Workspaces and + to the hosts the bound Model declares (`openultrasast.io/egress-hosts`). It is written between + `ax apply` and the resume, and deleted with the actor. +- **Inputs** (`reconciler.py`). Artifacts other tasks produced are never rendered into the + manifest; the runner fetches them from the receiver after the start, and the receiver serves a + running task its declared inputs only. +- **Completion** (`runner.py`). ax has no Completed phase: the runner's delivery of the output tar + is the task's completion. A completion marker stops a resumed actor from repeating billed work; + when the marker says the delivery failed, a later boot retries the delivery alone. +- **Budgets** (`plane/budget.py`). The metered client refuses the next call once a task's `usd` or + `calls` ceiling is reached; the task ends `unfinished` and resumes on a rerun with a larger + budget. Model-free tasks run with `{usd: 0, calls: 0}`, so a stray call fails loudly. +- **Attribution.** `ousast plane status RUN` prints per task its status, `calls`, `prompt`, + `cache_hit` and `output` tokens and `usd` from the tasks' `summary.json`, and writes + `attribution.json` in the run directory (`$OUSAST_RESULTS/plane/RUN/`, default + `~/ousast-results/plane/`). + +!!! note "The golden snapshot" + Agent Substrate boots every new actor template once as a *golden* actor to snapshot it, with + the same image, Task and environment, and inside the sandbox nothing tells that boot from the + real one. So the runner's command never starts by itself: it waits for the start request, + which the reconciler sends through `atenet-router` to the task's actor, never to the golden + one. Workspaces are prepared only after the start, so the golden boot touches no network and a + started actor already runs under its egress policy (a clone at boot failed on the live cluster + while the actor was being restored, 2026-09-29; `runner.py`, `_run_once`). The first resume of + a new image can time out while the snapshot is built, which is why the reconciler retries it + for up to `OUSAST_RESUME_TIMEOUT` seconds (900). + +## The Runs: validation and loop + +`ousast plane workspaces POPULATION --validation-set ...` (`generate.py`) writes one chain per +case; `--loop` adds the `alerts` step per case and the loop's singleton steps. Every step except +the verify passes (and `roles` in harvest Runs) is model-free. + +### Per case + +```mermaid +flowchart LR + facts["facts: repo-facts"] --> va["va: verify pass a"] + facts --> vb["vb: verify pass b"] + va --> agree["agree: a vs b"] + vb --> agree + agree -- "disputed only" --> vc["vc: verify pass c"] + va --> final["final: 2-of-3"] + vb --> final + vc --> final + final --> features["features: one record per candidate"] + alerts["alerts: quick scan on vulnerable and fixed pins (--loop)"] -.-> features + features --> remember["remember: memory.jsonl"] + alerts -.-> remember + final --> remember +``` + +### The loop (with `--loop`) + +```mermaid +flowchart LR + store[("Memory store")] -- "seeded before the Run, after the train-on-test guard" --> snap["memory-snapshot"] + rem["remember of every case"] --> measure["loop-measure"] + snap --> measure + measure --> propose["loop-propose: rules M1 and M2, M3 advisory"] + snap -- "index.json" --> propose + propose --> improve["loop-improve: one evolve round, unchanged validator and gate"] + improve --> outcome["gate.json, journal.json, and rule_policy.json only if accepted"] +``` + +- **Before the Run** `ousast plane run` seeds from the store (`memory.seed`): a task whose + `openultrasast.io/memory-key` (repository, pin, candidates digest, runner image digest) has + stored facts gets them and is marked done, and the sandboxed `memory-snapshot` task gets the + store's rows after the guard. **After the Run** it ingests every delivered `memory.jsonl` + (`ousast plane remember RUN` repeats this by hand). See [Memory](memory.md). +- The loop's rules and its guard are described in + [Architecture](architecture.md#proposals-from-plane-memory). Adopting an accepted ledger stays a + maintainer commit (`src/openultrasast/plane/tasks/loop.py`). +- `ousast plane harvest` writes the decision engine's harvest Runs (verify a/b with agree, and + model `roles`); `ousast plane alerts-engine` produces a Run's `alerts` from the Joern engine + image on the host for PHP and other languages quick mode does not cover. + +## Tasks + +| Task | Model | Does | +| --- | --- | --- | +| `repo-facts` | none | functions per product file, and each candidate's call sites in other files | +| `verify` (passes a, b, c) | `deepseek-flash` | batched tool hunt per file over the candidates, with their known callers | +| `agree` | none | a and b agreed or disputed; after pass c on the disputed, 2-of-3 (the step `final`) | +| `features` | none | one feature record per candidate for the decision engine | +| `remember` | none | the case's artifacts as memory rows | +| `alerts` | none | the quick scan on the vulnerable and fixed pins (loop Runs) | +| `loop` | none | the steps `snapshot`, `measure`, `propose`, `improve` | +| `roles` | `deepseek-flash` | per-repository source, sink and sanitizer roles without a vocabulary | + +Source: [ax on this host](ops/ax/README.md#tasks). diff --git a/docs/rustfs.md b/docs/rustfs.md new file mode 100644 index 0000000..7909d6b --- /dev/null +++ b/docs/rustfs.md @@ -0,0 +1,184 @@ +# RustFS setup for the memory store + +The plane's memory store can live in an S3-compatible bucket: `OUSAST_MEMORY=minio://sast-memory` +selects `MinioStore` (`src/openultrasast/plane/memory.py`), which works with RustFS and MinIO. The +maintainer's server is RustFS at `files.because-security.com`, bucket `sast-memory`, store at the +bucket root. What the store keeps there is described on [Memory](memory.md). + +**The store never configures its bucket.** An admin sets the bucket up once with admin +credentials. The agent account the store runs as can read the bucket's settings but not change +them, and reads and writes objects freely. Every time the store opens, it checks the setup and +refuses to start if anything is missing. + +```mermaid +flowchart LR + admin["Admin, once: versioning on, runs/ expiry rule"] --> bucket[("Bucket sast-memory")] + agent["Agent account: read settings, read and write objects and tags"] --> verify{"verify_bucket at startup"} + bucket --> verify + verify -- "all present" --> store["MinioStore serves reads and writes"] + verify -- "anything missing" --> refuse["MemoryStoreError naming each missing piece and its admin command"] +``` + +## One-time admin steps + +Run these with **admin** credentials (not the agent's). Both use the AWS CLI against the RustFS +endpoint. + +**1. Turn versioning on.** Provenance cites object version ids. + +```bash +aws s3api put-bucket-versioning \ + --endpoint-url https://files.because-security.com \ + --bucket sast-memory \ + --versioning-configuration Status=Enabled +``` + +**2. Expire raw run outputs under `runs/` after 30 days.** Everything else (`repos/`, `facts/`, +`index.jsonl` and the blobs) is kept, so the rule must be a plain prefix filter on `runs/`. + +```bash +aws s3api put-bucket-lifecycle-configuration \ + --endpoint-url https://files.because-security.com \ + --bucket sast-memory \ + --lifecycle-configuration '{"Rules":[{"ID":"ousast-runs-expiry","Status":"Enabled","Filter":{"Prefix":"runs/"},"Expiration":{"Days":30}}]}' +``` + +`put-bucket-lifecycle-configuration` **replaces every rule on the bucket**. If the bucket already +has rules, merge this one into them first. The rule id `ousast-runs-expiry` and the 30 days are the +store's defaults (`RUNS_RULE_ID`, `RUNS_EXPIRE_DAYS`); the store accepts any enabled, untagged +prefix rule on `runs/` with a number of days. + +To read the result back: + +```bash +aws s3api get-bucket-versioning --endpoint-url https://files.because-security.com --bucket sast-memory +aws s3api get-bucket-lifecycle-configuration --endpoint-url https://files.because-security.com --bucket sast-memory +``` + +## The agent policy + +Attach this policy to the agent account (the account whose keys go into `.env`). It is the +complete policy: nothing else is needed, and it grants no `Put*` on versioning or lifecycle. + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "BucketSettingsReadOnly", + "Effect": "Allow", + "Action": [ + "s3:ListBucket", + "s3:ListBucketVersions", + "s3:GetBucketLocation", + "s3:GetBucketVersioning", + "s3:GetLifecycleConfiguration" + ], + "Resource": ["arn:aws:s3:::sast-memory"] + }, + { + "Sid": "ObjectsReadWrite", + "Effect": "Allow", + "Action": [ + "s3:GetObject", + "s3:GetObjectVersion", + "s3:PutObject", + "s3:DeleteObject", + "s3:DeleteObjectVersion", + "s3:GetObjectTagging", + "s3:PutObjectTagging", + "s3:GetObjectVersionTagging" + ], + "Resource": ["arn:aws:s3:::sast-memory/*"] + } + ] +} +``` + +| Statement | Resource | In one line | +| --- | --- | --- | +| `BucketSettingsReadOnly` | `arn:aws:s3:::sast-memory` | The agent can list the bucket and its versions and *read* its location, versioning and lifecycle settings, which is what the startup check needs; it cannot change them. | +| `ObjectsReadWrite` | `arn:aws:s3:::sast-memory/*` | The agent reads, writes and deletes objects and their versions and reads and writes object tags; this covers row objects, blobs, the index, the Select probe, and the `kind` tag the store filters by. | + +So the agent writes objects freely but cannot change bucket settings: turning versioning off or +removing the expiry rule needs the admin. + +## The `.env` settings + +The store reads its endpoint and credentials from `.env` in the working directory or from the +environment (`minio_settings`), never from a manifest, and never prints them. + +| Variable | Value for this server | Notes | +| --- | --- | --- | +| `OUSAST_MEMORY` | `minio://sast-memory` | `minio://[/]`; unset means the local `FileStore` | +| `MINIO_ENDPOINT` | `files.because-security.com` | host[:port] without a scheme (the `minio` SDK's endpoint); required | +| `MINIO_ACCESS_KEY` | the agent account's access key | required | +| `MINIO_SECRET_KEY` | the agent account's secret key | required | +| `MINIO_SECURE` | `true` | TLS; `0`, `false`, `no` or `off` turn it off; default `true` | +| `MINIO_REGION` | optional | spares a GetBucketLocation call | + +- **Never commit `.env`.** It is gitignored (`.gitignore`: `.env`, `.env.*`, except + `.env.example`). `.env.example` lists these names with empty values. +- **`.env` never overrides a variable already exported in your shell** (`config.load_dotenv`). + A stale `export MINIO_SECRET_KEY=...` in your profile silently wins over the file; `unset` it + when a change in `.env` seems to have no effect. +- The store needs the `minio` extra (`uv sync --extra minio`); without it opening the store fails + with `OUSAST_MEMORY=minio://... needs the minio SDK: install openultrasast[minio]`. A missing + variable fails with `OUSAST_MEMORY=minio://... needs MINIO_ENDPOINT, MINIO_ACCESS_KEY, + MINIO_SECRET_KEY in .env or the environment` (naming only the missing ones). + +## What the store verifies at startup + +`MinioStore.verify_bucket()` runs once per process for each bucket and prefix. It reads the +versioning status and the lifecycle rules, writes a two-line probe object at +`_probe/select.jsonl` (under the prefix, if any), queries it with S3 Select and reads its tags. +Every missing piece is collected into one error: + +```text +memory minio://sast-memory: the bucket is not set up for the store. The store never configures its bucket (an admin sets it once, the store verifies); missing: + - +``` + +The lines, exactly as the code writes them for this bucket at the root (`` is the +server's error code and the first 200 characters of its message): + +| Check | Refusal line | +| --- | --- | +| versioning unreadable | `versioning cannot be read (): the agent needs s3:GetBucketVersioning on arn:aws:s3:::sast-memory, and versioning must be Enabled` | +| versioning off | `versioning is Off, it must be Enabled (provenance cites object versions). Admin: aws s3api put-bucket-versioning --endpoint-url "$MINIO_ENDPOINT" --bucket sast-memory --versioning-configuration Status=Enabled` (the actual status, e.g. `Suspended`, replaces `Off` when set) | +| lifecycle unreadable | `the lifecycle configuration cannot be read (): the agent needs s3:GetLifecycleConfiguration on arn:aws:s3:::sast-memory, and a rule must expire runs/` | +| a rule expires everything | `lifecycle rule(s) expire the whole store (repos/, facts/ and the index would be deleted); limit them to runs/` | +| no `runs/` rule | `no enabled lifecycle rule expires runs/ after a number of days. Admin (merged with any rules the bucket already has): aws s3api put-bucket-lifecycle-configuration --endpoint-url "$MINIO_ENDPOINT" --bucket sast-memory --lifecycle-configuration '{"Rules":[{"ID":"ousast-runs-expiry","Status":"Enabled","Filter":{"Prefix":"runs/"},"Expiration":{"Days":30}}]}'` | +| tags unreadable | `object tags cannot be read on _probe/select.jsonl (): the store filters row objects by their kind tag, so the agent needs s3:GetObjectTagging on arn:aws:s3:::sast-memory/*` | +| S3 Select missing | `S3 Select (SelectObjectContent over JSON Lines) does not answer on _probe/select.jsonl (). The store requires it and has no local fallback: use a server with S3 Select (RustFS has it; some MinIO releases removed it) and let the agent s3:PutObject and s3:GetObject` | + +The admin commands inside the messages use `--endpoint-url "$MINIO_ENDPOINT"`; with the +host-only `MINIO_ENDPOINT` above, pass the full URL (`https://files.because-security.com`) as in +the steps on this page. + +There is **no fallback**: the store does not start in a degraded mode, does not drop to a local +store, and does not fetch and filter objects itself. + +## The RustFS Select limit and the fixed-schema rule + +RustFS's S3 Select infers an object's JSON schema from its leading rows. A `where` field that is +absent from those rows is unbound, even if a later row has it: the query fails with +`EvaluatorBindingDoesNotExist` instead of matching (measured 2026-10-02 with the field first +present at row 5000). The store does not read that as "no rows", since that answer could silently +drop matches; it raises: + +```text +memory minio://sast-memory: S3 Select on failed (EvaluatorBindingDoesNotExist: ...) (a field of [] is absent from the object's leading rows, which the server reads as its schema); the store has no local fallback +``` + +The rule that follows: a field the store filters on must be present in every row of its kind +from the first row on, i.e. each row kind has a fixed set of queryable fields. Enforcing that set +per kind in the store is in progress and not on main as of this page (see +[Memory](memory.md#queryable-fields-per-kind-in-progress)). + +## Checks against the real server + +The contract tests that talk to a real server run only with `OUSAST_MEMORY_TEST_MINIO=1` (bucket +`OUSAST_MEMORY_TEST_BUCKET`, else `MINIO_BUCKET`). They verify the bucket at its root and write +only under a fresh `contract-/` prefix; they never configure the bucket. More on the host +setup: [ax on this host](ops/ax/README.md). diff --git a/docs/scanning.md b/docs/scanning.md new file mode 100644 index 0000000..fb6882d --- /dev/null +++ b/docs/scanning.md @@ -0,0 +1,125 @@ +# Scanning + +`ousast scan PATH --mode quick|standard|deep` scans a checkout; `ousast pre-push` checks only +what a push (or an explicit `--base`/`--head` pair) changes. The stage-by-stage pipeline, +evidence ladder and score are in [Architecture](architecture.md); this page shows the flows. + +## The three scan modes + +The stages a mode requests are `MODE_STAGES` in `src/openultrasast/stages.py`: `quick` runs +STATIC, `standard` adds MAP, `deep` adds REGRESS. + +```mermaid +flowchart TD + start["ousast scan PATH --mode M"] --> policy["policy_load, ruleset_load, policy_check"] + policy --> prep["static_mapping, preprocess, entry_point_mapping, rank, calibrate"] + prep --> which{"mode"} + which -- "quick, deep" --> qf["quick_findings: language-scoped pattern rules"] + which -- "standard" --> hp["hunter_pool: the same rules per ranked target"] + qf --> verify["verify, record_calibration, score"] + hp --> verify + verify --> mapq{"MAP requested?"} + mapq -- "quick: no" --> out["report.md, report.sarif, manifest.json"] + mapq -- "standard, deep" --> map["MAP: complexity map, semantic overlay, obligations, Joern model layer"] + map --> regq{"REGRESS requested?"} + regq -- "standard: no" --> out + regq -- "deep" --> reg["REGRESS: snippets in a Docker sandbox, verdicts.json"] + reg --> out +``` + +| Mode | Adds | Needs | Without it | +| --- | --- | --- | --- | +| `quick` | pattern rules, entry-point hints, ranking, verification, score | nothing | - | +| `standard` | MAP: complexity map, semantic overlay, authorization obligations, the Joern model layer; `fusion` (deterministic) | Joern; a provider key for the residual LLM question | `cpg_unavailable` degradation; the LLM parts are skipped and recorded | +| `deep` | REGRESS: promoted candidates loaded in a sandbox with no network, read-only source, non-root | a working `docker` | `sandbox_unavailable` degradation | + +Sources: `README.md` (scan modes), `src/openultrasast/cli.py` (`run_stage` calls), +[Architecture](architecture.md#the-scan-pipeline). + +## Quick rules per language + +Rules run only against files of their language. The bundled ruleset lives in +`src/openultrasast/ruleset//rules.toml`. + +| Language | Classes (CWE) | +| --- | --- | +| C / C++ | stack buffer overflow (121), format string (134), command injection (78) | +| Python | command injection (78), code injection (95), SSTI (94), deserialization (502), SQLi (89), path traversal (22), SSRF (918), weak hash (327), insecure default (489) | +| JavaScript / TS | command injection (78), code injection (95), XSS (79), SQLi (89), path traversal (22), SSRF (918), weak hash (327), deserialization (502) | +| Java + Groovy templates | command injection (78), SQLi (89), weak hash (327), deserialization (502), unescaped template XSS (79) | +| PHP | SQLi (89), command injection (78), file inclusion (98), path (22), open redirect (601), SSRF (918); code eval (95), unserialize (502) and echo XSS (79) ship as `shadow` rules | + +The detection gate (`python -m openultrasast.gate`) enforces at least 90% recall and under 10% +false positives per language on the bundled corpora in `benchmarks/manifests/`. Those corpora are +cheat-sheet fixtures: they catch regressions, they do not estimate recall on real code. **PHP is +not in the gate**; its numbers are in-sample development numbers +(`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`). See `README.md` for the +gate's per-language output. + +## The Joern engine + +The model layer (`src/openultrasast/model/`, `src/openultrasast/cpg/`) is the core of `standard`. +It builds one code property graph per scan with Joern (`joern-parse`, then batched +`joern --script` queries from `cpg/queries/*.sc`) and sends each question to the arbiter of its +type: **taint** (source to sink with no discharging sanitizer), **dominance** (does a guard govern +the obligated operation) and **configuration** (constant abstraction over a value). Taint facts +ship for C, Java, JavaScript, PHP and Python (`src/openultrasast/ruleset/semantic/*.toml`). + +```mermaid +flowchart LR + src["Checkout"] --> parse["joern-parse: one CPG per repository"] + parse --> q["Batched joern --script queries"] + q --> arb{"Arbiter by question type"} + arb --> taint["Taint: flow with no discharging sanitizer"] + arb --> dom["Dominance: obligation with no guard"] + arb --> conf["Configuration: permissive value"] + taint --> rung["Rung: model_entailed or suspicion"] + dom --> rung + conf --> rung + rung -- "residual question only" --> judge["LLM judge: confirm or contradict, never create a flow"] +``` + +The Docker image ships Joern and `php-cli` (which the PHP frontend needs); without Joern the +layer records `cpg_unavailable` and the scan is otherwise unchanged. Engine installation and the +Joern features used are in [Engine and pre-push](ops/README.md). + +## Framework priors + +Rule and taint-fact entries that know a framework or library carry a `framework` or `library` tag +naming a row of `src/openultrasast/ruleset/frameworks.toml` (for example `wordpress`, `flask`, +`django`, `express`, `spring`). The loaders take `priors=`: `"all"` (the default and today's +behaviour), `"off"` (language-level knowledge only) or a set of ids, so the contribution of +framework knowledge can be measured and switched off. The decision engine's first slice ran with +priors off ([Decision engine](decision-engine.md#first-measured-slice-2026-10-01)). + +## Pre-push: the delta check (experimental) + +`ousast pre-push` analyses the commits a `git push` would publish, or an explicit +`--base`/`--head` pair, and compares head with base so that only new or worsened defects are +candidates. Sources: `ousast pre-push --help`, `src/openultrasast/push/`, +[Engine and pre-push](ops/README.md#explicit-local-replay). + +```mermaid +flowchart TD + input["git push refs on stdin, or --base / --head"] --> resolve["Resolve both commit identities"] + resolve --> snap["Materialise tracked bytes in private scratch storage"] + snap --> regions["Discover changed regions, up to --max-regions"] + regions --> engine["Evidence ranker and arbiter on head"] + engine --> cmp["Targeted base comparison: new or worsened only"] + cmp --> admit{"Admission policy: capability registry"} + admit -- "registry empty today" --> diag["Diagnostic candidates in the artifact"] + admit -- "an enabled capability" --> alert["Normal alert"] + diag --> art["JSON artifact outside the repository"] + alert --> art + art --> exit{"--mode"} + exit -- "advisory, the default" --> allow["Exit 0: allow"] + exit -- "blocking, explicit opt-in" --> block["May block the push"] +``` + +- `--deadline` (30 s by default) bounds preparation, both revisions and reporting together; + `--incomplete-coverage block` opts into failing when coverage is incomplete. +- The default capability registry is empty pending independent qualification, so no normal alert + is emitted today; every candidate stays diagnostic. +- A model is called only with an explicit `--model-config`. +- Hook installation (`ops/install-pre-push`) and the private cache (`--cache-dir`) are documented + in [Engine and pre-push](ops/README.md#experimental-pre-push-integration). diff --git a/mkdocs.yml b/mkdocs.yml new file mode 100644 index 0000000..43f55dc --- /dev/null +++ b/mkdocs.yml @@ -0,0 +1,66 @@ +site_name: OpenUltraSAST +site_description: An experimental static security analyser and repository safety net, with an agentic plane and a learned decision engine in development. +repo_url: https://github.com/norandom/OpenUltraSAST +edit_uri: edit/main/docs/ +docs_dir: docs +site_dir: site +strict: true + +theme: + name: material + features: + - navigation.sections + - navigation.top + - content.code.copy + - search.highlight + palette: + - media: "(prefers-color-scheme: light)" + scheme: default + toggle: + icon: material/brightness-4 + name: Dark mode + - media: "(prefers-color-scheme: dark)" + scheme: slate + toggle: + icon: material/brightness-7 + name: Light mode + +validation: + omitted_files: warn + absolute_links: warn + unrecognized_links: warn + anchors: warn + +markdown_extensions: + - admonition + - attr_list + - tables + - toc: + permalink: true + - pymdownx.details + - pymdownx.snippets: + base_path: ["."] + check_paths: true + - pymdownx.superfences: + custom_fences: + - name: mermaid + class: mermaid + format: !!python/name:pymdownx.superfences.fence_code_format + +nav: + - Overview: index.md + - How it works: + - Scanning: scanning.md + - Architecture: architecture.md + - The plane on ax: plane.md + - Memory: memory.md + - Memory and detection: memory-and-detection.md + - Decision engine: decision-engine.md + - Evaluation: evaluation.md + - Using it: + - Examples: examples.md + - Threat model: threat-model.md + - Operations (maintainers): + - Engine and pre-push: ops/README.md + - ax on this host: ops/ax/README.md + - RustFS setup for the memory store: rustfs.md diff --git a/pyproject.toml b/pyproject.toml index 102a8a2..e22909c 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -11,6 +11,11 @@ requires-python = ">=3.11" dependencies = ["pyyaml>=6.0"] [project.optional-dependencies] +# The documentation site (mkdocs.yml, docs/). Mermaid renders client-side through Material's superfences fence. +docs = [ + "mkdocs>=1.6,<2", + "mkdocs-material>=9.5,<10", +] # The plane's memory store on MinIO (OUSAST_MEMORY=minio://, plane/memory.py). Imported lazily; the # core install and the file store need nothing beyond PyYAML. minio = [ diff --git a/tests/test_docs_site.py b/tests/test_docs_site.py new file mode 100644 index 0000000..49f033b --- /dev/null +++ b/tests/test_docs_site.py @@ -0,0 +1,158 @@ +"""The documentation site builds strictly, its Mermaid diagrams are well formed, and the RustFS page is complete. + +The strict build runs only where ``mkdocs`` is importable (the ``docs`` extra); the other checks read the Markdown +and always run. The Mermaid check is syntactic (fences, a known diagram type, balanced brackets and quotes, the node +budget): the site renders diagrams in the browser, so nothing here can run the real parser. +""" + +from __future__ import annotations + +import importlib.util +import re +import subprocess +import sys +from pathlib import Path + +import pytest + +ROOT = Path(__file__).resolve().parents[1] +DOCS = ROOT / "docs" +PAGES = sorted(DOCS.rglob("*.md")) +DIAGRAM_TYPES = ("flowchart", "graph", "sequenceDiagram", "stateDiagram-v2", "stateDiagram", "classDiagram", "erDiagram") +MAX_NODES = 15 +FENCE = re.compile(r"^(\s*)(`{3,}|~{3,})(.*)$") + +BUCKET_ACTIONS = ( + "s3:ListBucket", "s3:ListBucketVersions", "s3:GetBucketLocation", "s3:GetBucketVersioning", "s3:GetLifecycleConfiguration", +) # fmt: skip +OBJECT_ACTIONS = ( + "s3:GetObject", "s3:GetObjectVersion", "s3:PutObject", "s3:DeleteObject", "s3:DeleteObjectVersion", + "s3:GetObjectTagging", "s3:PutObjectTagging", "s3:GetObjectVersionTagging", +) # fmt: skip +SECRET_SHAPES = ( + re.compile(r"\b(?:AKIA|ASIA)[0-9A-Z]{16}\b"), # AWS-style access key id + re.compile(r"-----BEGIN [A-Z ]*PRIVATE KEY-----"), + re.compile(r"\b(?:sk|pk)-[A-Za-z0-9_-]{16,}"), # provider API keys + re.compile(r"(?:SECRET|ACCESS)_KEY\s*=\s*[^\s<`.]"), # an assignment with a value + re.compile(r"(? list[str]: + """The bodies of the ```mermaid fences. Every fence must close with the same marker (balanced fences).""" + blocks: list[str] = [] + opener: str | None = None # the open fence's marker, None outside a fence + language, body = "", [] + for line in text.splitlines(): + match = FENCE.match(line) + if opener is None: + if match: + opener, language, body = match.group(2), match.group(3).strip(), [] + continue + if match and match.group(2).startswith(opener) and not match.group(3).strip(): + if language == "mermaid": + blocks.append("\n".join(body)) + opener = None + continue + body.append(line) + assert opener is None, f"{where}: a {opener} fence is never closed" + return blocks + + +def _balanced(line: str) -> bool: + outside = re.sub(r'"[^"]*"', '""', line) + if outside.count('"') % 2: + return False + stack: list[str] = [] + for char in outside: + if char in "[({": + stack.append(char) + elif char in "])}" and (not stack or "[({"["])}".index(char)] != stack.pop()): + return False + return not stack + + +def _nodes(block: str) -> set[str]: + """Node ids of a flowchart: the leading identifier of every edge endpoint, labels and shapes removed.""" + nodes: set[str] = set() + for raw in block.splitlines()[1:]: + line = re.sub(r'"[^"]*"', "", raw).strip() + if not line or line.startswith(("%%", "subgraph", "end", "classDef", "class ", "style ", "direction")): + continue + line = re.sub(r"\[[^\]]*\]|\([^)]*\)|\{[^}]*\}", "", line) + for part in re.split(r"-\.->|-->|==>|---|--|&", line): + match = re.match(r"\s*([A-Za-z_]\w*)", part) + if match: + nodes.add(match.group(1)) + return nodes + + +def test_every_page_has_well_formed_mermaid() -> None: + assert PAGES, "docs/ has no pages" + total = 0 + for page in PAGES: + for block in mermaid_blocks(page.read_text(encoding="utf-8"), page.name): + total += 1 + lines = [ln for ln in block.splitlines() if ln.strip() and not ln.strip().startswith("%%")] + assert lines, f"{page.name}: empty mermaid block" + kind = lines[0].split()[0] + assert kind in DIAGRAM_TYPES, f"{page.name}: unknown diagram type {kind!r}" + for line in lines[1:]: + assert _balanced(line), f"{page.name}: unbalanced brackets or quotes in {line.strip()!r}" + if kind == "sequenceDiagram": + assert ";" not in line, f"{page.name}: ';' ends a sequence-diagram statement: {line.strip()!r}" + if kind in ("flowchart", "graph"): + found = _nodes("\n".join(lines)) + assert len(found) <= MAX_NODES, f"{page.name}: {len(found)} nodes > {MAX_NODES}: split the diagram" + assert total >= 10, f"only {total} mermaid diagrams in docs/" + + +def test_mermaid_fence_is_configured() -> None: + config = (ROOT / "mkdocs.yml").read_text(encoding="utf-8") + assert "pymdownx.superfences" in config and "name: mermaid" in config and "fence_code_format" in config + + +def test_rustfs_page_lists_the_complete_policy_and_no_secret() -> None: + text = (DOCS / "rustfs.md").read_text(encoding="utf-8") + for action in BUCKET_ACTIONS + OBJECT_ACTIONS: + assert f'"{action}"' in text, f"rustfs.md: the policy lacks {action}" + for needle in ( + '"Sid": "BucketSettingsReadOnly"', '"Sid": "ObjectsReadWrite"', '"arn:aws:s3:::sast-memory"', '"arn:aws:s3:::sast-memory/*"', + "put-bucket-versioning", "put-bucket-lifecycle-configuration", "--endpoint-url https://files.because-security.com", + "--bucket sast-memory", '"Prefix":"runs/"', '"Days":30', "OUSAST_MEMORY", "minio://sast-memory", "MINIO_ENDPOINT", + "MINIO_ACCESS_KEY", "MINIO_SECRET_KEY", "MINIO_SECURE", "MINIO_REGION", "EvaluatorBindingDoesNotExist", + ): # fmt: skip + assert needle in text, f"rustfs.md lacks {needle!r}" + for shape in SECRET_SHAPES: + hit = shape.search(text) + assert hit is None, f"rustfs.md has a secret-looking string near {hit.group(0)[:6]!r}..." + + +def test_rustfs_page_quotes_the_refusals_verify_bucket_writes() -> None: + """Each refusal line the page quotes still appears, as a template fragment, in the store's code.""" + code = (ROOT / "src" / "openultrasast" / "plane" / "memory.py").read_text(encoding="utf-8") + for fragment in ( + "the bucket is not set up for the store. The store never configures its ", + "it must be Enabled (provenance cites object versions)", + "expire the whole store (repos/, facts/ and the index would be ", + "no enabled lifecycle rule expires", + "the store filters row objects by ", + "The store requires it and has no local fallback", + "is absent from the object's leading rows, which the server reads as its schema", + ): + assert fragment in code, f"memory.py no longer writes {fragment!r}: update docs/rustfs.md" + + +@pytest.mark.skipif(importlib.util.find_spec("mkdocs") is None, reason="mkdocs is not installed (the docs extra)") +def test_site_builds_strictly(tmp_path: Path) -> None: + proc = subprocess.run( + [sys.executable, "-m", "mkdocs", "build", "--strict", "-f", str(ROOT / "mkdocs.yml"), "-d", str(tmp_path / "site")], + cwd=ROOT, capture_output=True, text=True, timeout=300, + ) # fmt: skip + assert proc.returncode == 0, f"mkdocs build --strict exited {proc.returncode}:\n{proc.stderr[-3000:]}" + pages = list((tmp_path / "site").rglob("*.html")) + assert len(pages) >= 12, f"the build wrote only {len(pages)} pages" # proves the build read the docs + ax = (tmp_path / "site" / "ops" / "ax" / "index.html").read_text(encoding="utf-8") + assert "Bring-up" in ax, "the ax operations page did not include ops/ax/README.md" + leaked = [p.name for p in pages if ".kiro" in p.read_text(encoding="utf-8") or "kiro-" in p.read_text(encoding="utf-8")] + assert not leaked, f"maintainer Kiro tooling reached the site: {leaked}"