From 5aac7eac365412006efa5927d22c07be223e2963 Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Fri, 2 Oct 2026 16:03:46 +0200 Subject: [PATCH] docs: audit against v2.0.0; deployment guide (local ax and a separate Kubernetes cluster) Every README and docs page checked against ousast --help, the code and the committed records. New docs/deployment.md (in the nav): what runs where today, this host's setup (up.sh, doctor, smoke Run, footprint), step-by-step guidance for a separate Kubernetes cluster with implemented vs planned stated per item (no cluster deployment has been made; kube context and doctor's registry check are hard-coded in doctor.py, egress.py, router.py), and what the plane/ manifests need changed. Decision-engine pages state slice 2 (six families, pooled and within-pair AUC, BLOCK offered nowhere, canary below 0.9 for three families, positives-only inflation) and that the engine is not adopted; exp-002 registered, not run, no result quoted. Fixed: model/judge.py reference (now model/pipeline.py), missing-key behaviour, .env reference (OPENROUTER_EMBEDDING_MODEL, store variables), docs extra, plane memory-normalise and ousast index listed, ax README's cluster table made precise. RELEASE_NOTES gets an Unreleased entry. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo --- README.md | 69 ++++++--- RELEASE_NOTES.md | 13 ++ docs/architecture.md | 9 +- docs/decision-engine.md | 57 ++++++- docs/deployment.md | 283 +++++++++++++++++++++++++++++++++++ docs/evaluation.md | 11 +- docs/examples.md | 4 +- docs/index.md | 2 + docs/memory-and-detection.md | 10 +- docs/plane.md | 7 +- docs/scanning.md | 2 +- docs/threat-model.md | 7 +- docs/token-ergonomics.md | 1 + mkdocs.yml | 1 + ops/README.md | 5 +- ops/ax/README.md | 20 ++- 16 files changed, 448 insertions(+), 53 deletions(-) create mode 100644 docs/deployment.md diff --git a/README.md b/README.md index f645812..a985c4a 100644 --- a/README.md +++ b/README.md @@ -27,8 +27,10 @@ uv run ousast scan /path/to/repo --mode quick uv run ousast scan . --mode quick --fail-on verified ``` -The core install depends only on PyYAML. Optional extras: `semantic` (tree-sitter grammars -for the overlay) and `s3` (the plane's memory store on an S3-compatible bucket, boto3). +The core install depends only on PyYAML. Optional extras (`pyproject.toml`): `semantic` +(tree-sitter grammars for the overlay), `s3` (boto3, for the plane's memory store on an +S3-compatible bucket) and `docs` (MkDocs with the Material theme, for the documentation site); +install one with `uv sync --extra s3`. For `standard` and `deep` scans, use the Docker image, which ships Joern (with `php-cli` for the PHP frontend) next to the tool. Source the shell wrapper once and `ousast` runs in the @@ -50,7 +52,7 @@ Every run writes auditable artifacts under `/.openultrasast/runs/ # or s3:///; needs the s3 extra +S3_ENDPOINT=https://host[:port] # the full URL +S3_REGION=us-east-1 # optional; spares a GetBucketLocation call +S3_BUCKET= # the bucket `OUSAST_MEMORY=s3://` alone names +AWS_ACCESS_KEY_ID=... +AWS_SECRET_ACCESS_KEY=... ``` The project uses DeepSeek (`deepseek-flash`) for LLM calls and OpenRouter only for embeddings. -Set `DEEPSEEK_API_KEY`: without it, chat calls fall back to OpenRouter when that key is set. +Set `DEEPSEEK_API_KEY`: without it, chat calls fall back to OpenRouter when that key is set +(`tool_hunter.resolve_hunter_client`). The S3 bucket is set up once by an admin and verified +by the store at every start ([docs/rustfs.md](docs/rustfs.md)). `ousast pre-push` does not read +`.env`; it takes credentials from the Git process environment. **`.env` never overrides a variable already exported in your shell.** A stale `export DEEPSEEK_API_KEY=...` in your profile wins over the `.env` file silently; `unset` it @@ -171,20 +186,30 @@ any `--qualify-population` are dropped first. Details: ## The agentic plane (maintainers) Model work over whole repositories runs as a Run manifest on [google/ax](ops/ax/README.md) -over Agent Substrate (a kind cluster on the maintainer's host), one isolated actor per task. -Each task binds its own Model and its own `usd`/`calls` budget, egress is deny-by-default per -task, and the provider key travels only in the task's start request. +over Agent Substrate, one isolated actor per task. ax is the only agentic executor: there is +no local subprocess path, and a scan needs none of it. Each task binds its own Model and its +own `usd`/`calls` budget; egress is deny-by-default per task (one EgressPolicy per actor, +hostnames only: the artifact receiver, the Workspaces' Git hosts, the Model's declared host); +and the provider key travels only in the task's start request, sent through Substrate's +`atenet-router` to that task's actor, never in a manifest or an image. + +What runs where today: the reconciler (`ousast plane run`) and the artifact receiver run on +the operator's host; the tasks run in gVisor sandboxes on Agent Substrate in a single-node kind +cluster on the maintainer's host; the memory store is a local file store or an S3-compatible +bucket (`OUSAST_MEMORY=s3://`; RustFS is the tested server). ```bash uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image -uv run ousast plane run plane/runs/validation-46.yaml # a rerun skips tasks already done -uv run ousast plane status validation-46 # per-task status and token attribution +uv run ousast plane run plane/runs/validation-46.yaml # a rerun skips tasks already done; --workers bounds tasks in flight +uv run ousast plane status validation-46 # per-task status and the token attribution table (--units per unit) uv run ousast plane remember validation-46 # ingest the run's rows into the memory store +uv run ousast plane memory-normalise --dry-run # count the stored rows a rewrite to the fixed schema would touch ``` -`ousast plane workspaces`, `alerts-engine` and `harvest` generate Runs and inputs. Bring-up, -the task catalogue, the memory store (`OUSAST_MEMORY`) and the checklist for moving the plane -to another Kubernetes cluster are in [ops/ax/README.md](ops/ax/README.md). +`ousast plane workspaces`, `harvest` and `alerts-engine` generate Runs and their inputs. +Bring-up, the task catalogue, the memory store (`OUSAST_MEMORY`) and the host's lessons are in +[ops/ax/README.md](ops/ax/README.md); what runs where, this host's setup and the guidance for a +separate Kubernetes cluster (planned, not done) are in [docs/deployment.md](docs/deployment.md). > HarnessX, the earlier optional agentic extra, was retired 2026-09-30, and a leftover > `[harnessx]` section (retired 2026-09-30) is ignored with one warning. `[models] verifier`, @@ -198,8 +223,11 @@ A learned decision engine is being built to replace hand-tuned detection edits: instrument (quick rules, the Joern engine, model sink classification, verify passes, repository facts) produces signals, never verdicts, and a compiled AI classifier with local memory turns them into a calibrated probability per candidate with BLOCK and ADVISORY operating points. -No scan, pre-push or report uses it yet. Status and the first measured slice: -[docs/decision-engine.md](docs/decision-engine.md). +No scan, pre-push or report uses it yet, and it is not adopted. Measured so far out of +repository (slice 2, 2026-10-02, six families): pooled AUC 0.77 to 0.96 per family, within-pair +AUC 0.72 to 0.93, BLOCK offered for no family, re-run agreement below 0.9 for three of the six +(`benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`). Status and the +numbers: [docs/decision-engine.md](docs/decision-engine.md). ## Agent integrations @@ -227,6 +255,8 @@ Documentation site with flow diagrams (`docs/`, `mkdocs.yml`): `uv sync --extra and project score, and the self-improving loops. - [docs/evaluation.md](docs/evaluation.md): independent populations, the qualification gates, pair corpora and the plane increment results. +- [docs/deployment.md](docs/deployment.md): the plane on this host, and the guidance for a + separate Kubernetes cluster (planned) with what is implemented and what is not. - [docs/decision-engine.md](docs/decision-engine.md): the learned decision engine. - [docs/threat-model.md](docs/threat-model.md): trust boundaries, sandbox and hardening. - [docs/token-ergonomics.md](docs/token-ergonomics.md): model spend at runtime (budgets, reuse, @@ -246,5 +276,6 @@ uv run python dagger/ci.py # containerized CI pipeline ``` Maintainer commands not covered above: `ousast repos` (pinned known-vulnerable checkouts), -`ousast model candidates` (what the candidate enumerator can reach) and `ousast learn` -(the decision engine's data; see [docs/decision-engine.md](docs/decision-engine.md)). +`ousast index` (chunk a repository for embedding-index construction), `ousast model candidates` +(what the candidate enumerator can reach) and `ousast learn` (the decision engine's data; see +[docs/decision-engine.md](docs/decision-engine.md)). diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index b3d91df..338ef76 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -1,3 +1,16 @@ +# Unreleased + +- **Documentation audited against v2.0.0** (no behaviour change). New `docs/deployment.md`: where the plane runs + today, this host's setup, and step-by-step guidance for a separate Kubernetes cluster with what is implemented + and what is planned; no cluster deployment has been made, and the kube context (`kind-$KIND_CLUSTER_NAME`) and + `doctor`'s registry check (`localhost:5001`) are still hard-coded. The decision-engine pages state slice 2 + (six families, `benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`: pooled AUC 0.77-0.96, + within-pair 0.72-0.93, BLOCK offered nowhere, canary below 0.9 for three families) and that the engine is not + adopted. The model layer's residual question is documented at `model/pipeline.py` (`model/judge.py` was + deleted); a missing provider key is described as it behaves (no question asked, entailed findings reported), + not as a recorded degradation. The `.env` reference lists `OPENROUTER_EMBEDDING_MODEL` and the memory-store + variables; `ousast plane memory-normalise` and `ousast index` are listed. + # v2.0.0 (2026-10-02) A new major release: the agentic plane runs on google/ax, HarnessX is gone, detection decisions move diff --git a/docs/architecture.md b/docs/architecture.md index 399a36a..a11eb6b 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -46,9 +46,12 @@ model_corroborated the graph found the shape; a judge agreed on the residual suspicion proposed, nothing corroborated it ``` -The LLM (`model/judge.py`) is asked only the residual question the graph cannot settle, for a -flow the graph already found; it can confirm or contradict, never create a flow. Without a -provider key only the `suspicion` band goes unasked. +The LLM (`model/pipeline.py`, `residual_question`) is asked only the residual question the graph +cannot settle, for a flow the graph already found; it can confirm or contradict, never create a +flow. When the graph resolved nothing, the enumerator's candidates (at most +`MAX_JUDGED_CANDIDATES`, 8) are put to it, and anything it affirms is reported at `suspicion`. +Without a provider key neither question is asked and only the graph's entailed findings are +reported. **REGRESS** (`regress/`, `sandbox/`) runs only when `docker info` succeeds. Each promoted candidate gets a small language recipe (Python, JavaScript, C) that loads the candidate's diff --git a/docs/decision-engine.md b/docs/decision-engine.md index 401986a..2fefb44 100644 --- a/docs/decision-engine.md +++ b/docs/decision-engine.md @@ -61,9 +61,55 @@ sides of a pair by construction: `2026-10-01-decision-engine-leak-audit/`); fram | Canary agreement on re-run | 0.83 (30 candidates) | | Spend | $0.91 metered at list price for DeepSeek, $0.0040 for embeddings; $0.0032 per evaluated candidate | -The other evaluable families are not evaluated yet. Leave-one-source-out, leave-one-framework-out -and learning curves, experiments, integration into scan and pre-push, adoption, and the one-time -v3 check are still ahead. +## Second slice: six families (2026-10-02) + +Record: `benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json` (counts only). +After the advisory-fix harvest the memory holds 904 static examples (724 before). One program was +compiled per evaluable family (injection recompiled, since the memory changed) and evaluated on +that family's own candidates, out of repository as above: compile split never evaluated, outer +grouped 5-fold, calibration cross-fitted, operating points nested, 95% repository-cluster +bootstrap intervals (2,000 resamples); input profile v1, priors off, k = 5. + +| Family | Candidates (groups; positive) | Pooled AUC of `s` (95% CI) | Within-pair AUC (paired groups) | ADVISORY recall / precision | Canary agreement | +| --- | --- | --- | --- | --- | --- | +| access control | 37 (22; 23) | 0.96 (0.90 - 1.0) | 0.93 (14) | 0.91 / 0.91 | 0.83 | +| deserialization | 46 (30; 30) | 0.90 (0.82 - 0.97) | 0.81 (16) | 0.83 / 0.76 | 0.93 | +| injection | 110 (29; 58) | 0.77 (0.67 - 0.88) | 0.72 (22) | 0.88 / 0.57 | 0.77 | +| output encoding | 44 (27; 29) | 0.91 (0.82 - 0.97) | 0.82 (15) | 0.86 / 0.81 | 0.90 | +| path | 35 (21; 21) | 0.89 (0.79 - 0.97) | 0.83 (14) | 0.86 / 0.69 | 0.90 | +| untrusted destination | 62 (37; 38) | 0.85 (0.76 - 0.94) | 0.72 (18) | 0.84 / 0.73 | 0.87 | + +- **BLOCK is offered for no family.** Calibration holds only for output encoding (ECE 0.045), and + there no fold reaches the required precision (`precision_unreachable`: no fold reaches a Wilson + lower bound of 0.95). For the other five families calibration does not hold (ECE 0.071 to 0.119). +- **The pooled AUC is the more favourable number.** Every family's pool includes candidates that + carry only a positive label, with no fixed-side counterpart (7 to 19 per family; there are no + single-label negatives), so the pooled AUC is inflated by positives that never had to be told + apart from their own fix. The within-pair AUC over the paired groups, 0.72 to 0.93, is the + stricter reading (`paired_groups_check` in the record). +- **Re-run agreement (canary, 30 candidates per family) is below 0.9 for three families:** + injection 0.77, access control 0.83, untrusted destination 0.87. +- **One memory-size point for injection** (paired: the same program, candidates and folds, with a + retrieval memory of 724 vs 904 examples): AUC 0.76 vs 0.77, within the intervals. No injection + example was added, so the new examples reach an injection prompt only as contrast. +- **Spend:** $2.88 metered at list price for DeepSeek under an $8 ceiling, $0.0009 for + embeddings; $0.0027 to $0.0032 per evaluated candidate. The access-control evaluation replayed + from the response cache at $0 with identical predictions. +- **An instrument fix is recorded:** compiled demonstrations ignored the near-duplicate rule + (cosine >= 0.98) until the pre-call boundary assertion stopped the untrusted-destination + evaluation; demonstrations are now swapped or dropped like retrieved examples, and the stopped + part was replayed after the fix. + +**Not adopted.** Still ahead: leave-one-source-out, leave-one-framework-out, the remaining +learning-curve points (memory size 0/25/50%, number of sources), integration into scan and +pre-push, adoption, and the one-time v3 check. + +## Experiments + +Changes to prompts, models, pass counts or features are compared by controlled experiments, so +nothing is adopted on one run's number. One is registered and has not run: exp-002 (variance +reduction of the score). No result exists; this page carries one only once a record under +`benchmarks/measurements/` does. ## Commands @@ -81,5 +127,6 @@ v3 check are still ahead. | `ousast learn audit-leaks` | how well each signal alone separates the sides of a pair (no model) | The memory store is the plane's (`OUSAST_MEMORY`; see [ops/ax/README.md](ops/ax/README.md)). -The harvest that produced the plane signals runs as plane Runs generated by -`ousast plane harvest` (`benchmarks/measurements/2026-10-01-decision-engine-harvest/record.json`). +The harvests that produced the plane signals ran as plane Runs generated by +`ousast plane harvest` (`benchmarks/measurements/2026-10-01-decision-engine-harvest/record.json` +and, for the advisory-fix pairs, `2026-10-02-decision-engine-harvest-advisory/record.json`). diff --git a/docs/deployment.md b/docs/deployment.md new file mode 100644 index 0000000..631957a --- /dev/null +++ b/docs/deployment.md @@ -0,0 +1,283 @@ +# Deployment + +How the agentic plane is deployed today, and what moving it to a separate Kubernetes cluster +would take. State as of 2026-10-02 (v2.0.0). A scan needs none of this: `ousast scan` and +`ousast pre-push` run from the Python package (and the Docker image for Joern, see +[Engine and pre-push](ops/README.md)). The plane is the maintainers' surface for model work over +whole repositories, driven by `ousast plane`. + +Two words are used strictly on this page. **Implemented** means the code on `main` does it and it +has run on the maintainer's host. **Planned** means it has not been done; the guidance is derived +from the code and the host's lessons, and the code changes it needs are named with their file paths. + +**No deployment of the plane to a cluster other than the maintainer's single-node kind cluster has +been made.** Section B is guidance, not a record. + +## What runs where today (implemented) + +```mermaid +flowchart LR + subgraph host ["Operator's host"] + rec["Reconciler: ousast plane run"] + rcv["Artifact receiver: in-process HTTP server, port 18090"] + env[".env or exported variables: provider keys, S3 settings"] + end + subgraph cluster ["kind cluster on the same host"] + ax["ax control plane (ax-system)"] + sub["Agent Substrate (ate-system): atenet-router, atenet-egress"] + act["Task actors in gVisor workers (WorkerPool ousast-pool)"] + svc["Service ousast-receiver + EndpointSlice to the host"] + reg["Registry localhost:5001: runner image by digest"] + end + mem[("Memory store: file store on the host, or an S3 bucket (RustFS tested)")] + rec -- "ax apply, resume, get, delete" --> ax + rec -- "EgressPolicy per actor; start request via port-forward" --> sub + sub --> act + act -- "HTTP by Service name through the gateway" --> svc + svc --> rcv + rec --> mem + reg -. "pull by digest" .-> act +``` + +| Part | Where it runs | Code or manifest | +| --- | --- | --- | +| Reconciler (`ousast plane run`, `status`, `remember`) | the operator's host | `src/openultrasast/plane/reconciler.py` | +| Artifact receiver | an HTTP server inside the reconciler process, on the host, port `OUSAST_ARTIFACT_PORT` (18090) | `src/openultrasast/plane/egress.py` (`Receiver`, `RECEIVER_PORT`) | +| How actors reach the receiver | the Service `ousast-receiver.ax-system` (port 80) with a selector-less EndpointSlice whose endpoint is the kind network's gateway address on the host; the runner dials the ClusterIP with the Service name as `Host`, which the egress gateway allows by hostname | `ops/ax/receiver-service.yaml.tmpl`, rendered by `up.sh` | +| ax control plane, Agent Substrate, egress gateway, router | the kind cluster (`ax-system`, `ate-system`) | installed by `ops/ax/up.sh` from the upstream checkouts under `~/.cache/ousast/ax-src/` | +| Tasks | one actor per Task in a gVisor worker; two warm workers of 1 CPU / 1.5 GiB | `ops/ax/workerpool.yaml.tmpl` | +| Runner image | the kind-local registry `localhost:5001`, pinned by digest (`~/.cache/ousast/ax-src/runner-image`) | `plane/Dockerfile.runner`; the `image` of every template under `plane/tasks/` | +| Provider keys | read from the operator's environment or `.env` (which never overrides an exported variable), sent only in the start request through a `kubectl port-forward` to `svc/atenet-router` | `src/openultrasast/plane/router.py` | +| Memory store | `OUSAST_MEMORY`: a file store under `~/ousast-results/plane/memory` by default, or `s3://[/]` on any reachable S3-compatible server (RustFS is the tested one) | `src/openultrasast/plane/memory.py`, [Memory](memory.md), [RustFS setup](rustfs.md) | +| Run state and artifacts | `$OUSAST_RESULTS/plane//` on the host (default `~/ousast-results/plane/`) | `reconciler.py` (`run_dir`) | + +Only the reconciler talks to the memory store: a sandboxed task cannot read it, so the loop's +`memory-snapshot` is seeded by the reconciler before the Run ([The plane on ax](plane.md)). + +## A. This host: the local setup (implemented) + +What `ops/ax/up.sh` needs on the PATH: `docker`, `go`, `kubectl`, `kind`, `ko` and `ax` (it +refuses to start when one is missing). Tools live in `~/go/bin` and `~/.local/bin`; sources and +rendered files in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`). + +1. **Bring up the cluster and the plane.** + + ```bash + ops/ax/up.sh + ``` + + Idempotent: each step is skipped when already done. It creates the kind cluster + (`KIND_CLUSTER_NAME`, default `ousast`) with its registry (`KO_DOCKER_REPO`, default + `localhost:5001`), installs Agent Substrate, applies the egress gateway (the agentgateway + variant, one prebuilt image) when the install left it out, deploys the ax control plane, applies + the receiver Service with the host's kind-gateway address as its endpoint, applies the gVisor + worker pool, builds and pushes the runner image and writes its digest pin to + `~/.cache/ousast/ax-src/runner-image`, installs the `ax` CLI from the deployed checkout (a release + CLI skews from the server), and runs one smoke Task. It ends by printing the host's memory and the + kind containers' usage. + +2. **Check it.** + + ```bash + uv run ousast plane doctor + ``` + + Four checks: the kind context is reachable, every pod in `ate-system` and in `ax-system` is ready, + and the registry at `localhost:5001` holds `ousast-runner`. Each failing line names + `ops/ax/up.sh` as the fix. `doctor` is informational: `ousast plane run` does not call it. + +3. **Run the end-to-end smoke Run.** + + ```bash + ops/ax/smoke-run.sh + ``` + + It renders `ops/ax/smoke-run.yaml.tmpl` with the current runner digest and submits the Run + `ax-e2e-smoke` through the reconciler: one model-free `repo-facts` Task (`budget: {usd: 0, + calls: 0}`) over a files-only Workspace, resumed, started through the router, its `facts.json` and + `summary.json` delivered to the host, then `ousast plane status ax-e2e-smoke` prints the attribution + table. The script calls the repository's `.venv/bin/ousast`, so `uv sync` first. + +4. **Keys and the store.** Put `DEEPSEEK_API_KEY` (chat) and `OPENROUTER_API_KEY` (embeddings) in + `.env` or export them; a Task's Model names the variable (`secretKey.key`), and the key travels + only in that Task's start request. Set `OUSAST_MEMORY=s3://` with `S3_ENDPOINT`, + `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` (optional `S3_REGION`, `S3_BUCKET`) for the S3 store; + the bucket must be set up once by an admin as [RustFS setup](rustfs.md) describes, and the store + verifies it at every start. + +5. **Run a real Run**, as in [Examples](examples.md#7-running-the-agentic-plane): `ousast plane run + plane/runs/validation-46.yaml`, then `status` and `remember`. + +6. **Tear down** with `ops/ax/down.sh`, which deletes the cluster and its registry. + +**Footprint** (measured 2026-09-29, idle, two workers; [ax on this host](ops/ax/README.md#measured-footprint-2026-09-29-idle-two-workers)): + +| Item | Value | +| --- | --- | +| kind node container memory | 1.7 GiB (27% of the 7.7 GiB host) | +| registry container memory | 34 MiB | +| node volume (images, containerd) | 4.9 GB on disk | +| worker pool | 2 gVisor workers, 1 CPU / 1.5 GiB limit each | + +Disk is the binding constraint: the Substrate install and the `ko` builds need several GB of Go +caches (`go clean -cache -modcache` afterwards), and every Task is its own actor template with a +golden actor (about 24 MB each under `/var/lib/ate/actors` on the node). The failures the host +produced, and what each one taught, are listed in +[ax on this host](ops/ax/README.md#what-this-host-taught-each-one-cost-a-failed-live-run). + +## B. A separate Kubernetes cluster (planned) + +The target: a cluster that runs Agent Substrate with its egress gateway and the ax control plane, +pulls the runner image from a registry of yours, receives task artifacts without a developer +laptop in the path, and keeps its memory in your S3 bucket. The provider key already travels only +in the start request through `atenet-router`, which works the same through a port-forward to any +cluster. Everything else below is either a step you take or a code change that is not made yet. + +### B.1 Cluster with Agent Substrate, its egress gateway and ax + +- Install Agent Substrate on the cluster following its own instructions. `ops/ax/up.sh` uses + Substrate's kind helpers (`hack/create-kind-cluster.sh`, `hack/install-ate-kind.sh`), which do + not apply to another cluster. +- Apply the egress gateway. On this host it is the agentgateway variant + (`manifests/ate-install/agentgateway-egress` in the Substrate checkout, applied with + `kubectl kustomize --load-restrictor=LoadRestrictionsNone`), which needs no Rust build; the + Envoy variant does. Without the gateway an actor has no network at all, and the per-task + EgressPolicies the reconciler writes (`src/openultrasast/plane/egress.py`) work unchanged + against the Substrate-installed gateway: deny by default, hostnames only, plain HTTP to the + receiver, TLS passthrough to the Workspaces' Git hosts and the Model's declared hosts. +- Deploy ax (`make deploy AX_IMAGE_REPO=` in the ax checkout) **after** B.2. +- Apply a WorkerPool (B.6). Substrate installs none by itself ("no free workers"). + +### B.2 ax's snapshot bucket: `AX_SNAPSHOTS_BUCKET` + +ax's deploy manifest (`deploy/ax-server.yaml` in the ax checkout) sets `AX_SNAPSHOTS_BUCKET` to +the ax authors' own bucket (`gs://dberkov-gke-dev3/ate-env/` in the checkout used here). Point it +at **your own bucket before deploying ax**. On this host the bucket named there did not exist in +the S3 server, deleted golden actors piled up in `DELETING` and filled the disk during a 600-task +harvest (2026-10-01); creating the bucket let deletes complete, and even then a deleted actor's +directory stays on the node, so long Runs need a janitor that removes directories no live actor +owns. + +### B.3 A registry the workers can pull from, the runner image pinned by digest + +- Build the runner from the repository root and push it to your registry: + + ```bash + docker build -f plane/Dockerfile.runner -t /ousast-runner:dev . + docker push /ousast-runner:dev + docker inspect --format '{{range .RepoDigests}}{{println .}}{{end}}' /ousast-runner:dev + ``` + +- Substrate rejects tags: every Task's `image` must be the `/ousast-runner@sha256:...` + reference. Write it into a file (on this host, `~/.cache/ousast/ax-src/runner-image`) and pass + `--runner-image FILE` to `ousast plane workspaces` and `ousast plane harvest`, which re-pin the + Tasks they generate. The committed templates under `plane/tasks/` and the generated + `plane/tasks/validation-46.yaml` carry `localhost:5001/ousast-runner@sha256:...` and need the same + re-pin (regenerate, or replace the reference) before they run elsewhere. +- Workers must be able to pull from the registry (credentials, if any, are the cluster's concern; + ax's default runner image on `gcr.io` needs credentials and is not used here). +- **Not yet configurable:** `ousast plane doctor` checks the registry at `localhost:5001` by name + (`src/openultrasast/plane/doctor.py:45`), so its fourth check fails on another cluster even when + the image is in place. `doctor` is informational, `plane run` does not depend on it. + +### B.4 One configurable kube context + +The reconciler, the egress client and the router port-forward all use the context +`kind-$KIND_CLUSTER_NAME` (`kind-ousast` by default). **Not yet configurable**: the prefix `kind-` +is hard-coded in `src/openultrasast/plane/doctor.py:39`, `src/openultrasast/plane/egress.py:115` +and `:148`, and `src/openultrasast/plane/router.py:145`; `ops/ax/up.sh:24` builds the same name. +The planned change is one variable (for example `OUSAST_KUBE_CONTEXT`) read in those three +modules. Until then the only way to drive another cluster from the unchanged code is to give its +kubeconfig context a name of that form (`kubectl config rename-context kind-` and +`KIND_CLUSTER_NAME=`), which is a workaround, not a supported configuration. The egress +client also needs `kubectl-ate` (or the executable `OUSAST_KUBECTL_ATE` names) able to use that +context. + +### B.5 The receiver: in-cluster instead of on the host + +Today the receiver is a thread inside the reconciler process (`egress.py`, `Receiver`), and the +reconciler learns of a task's completion by watching that object's `delivered` set in memory +(`reconciler.py`). Actors reach it through the Service `ousast-receiver.ax-system` whose +EndpointSlice points at the host. Three paths, in order of effort: + +1. **Interim, works with the unchanged code:** keep the host receiver and render + `ops/ax/receiver-service.yaml.tmpl` with an endpoint address the egress gateway pod can reach + (`KIND_GATEWAY` is just an IPv4 address in the template, `OUSAST_ARTIFACT_PORT` the port). The + reconciler's host must be reachable from the cluster (a VPN or a routed address), and the + gateway dials the address the actor connected to, so the Service's ClusterIP must still map to it. +2. **Planned: the receiver as an in-cluster Deployment** behind the same Service name, with the + reconciler reading deliveries and serving inputs from it instead of from its own thread. This + needs code: the receiver and its `delivered`/`inputs` state are in-process today, and the + runner's `GET /inputs//` is served by that same process. +3. **Follow-on: presigned-URL delivery to the S3 store.** The runner would `PUT` its output tar to a + presigned URL on the memory store's bucket and fetch inputs the same way, and the reconciler + would read from the bucket. Nothing of this exists; the EgressPolicy would then have to allow + the store's hostname for the task. + +### B.6 Worker pool sizing + +`ops/ax/workerpool.yaml.tmpl` declares two warm gVisor workers of 1 CPU / 1.5 GiB (requests 250m / +1.5 GiB) for the 7 GB host, selected onto nodes by `ate.dev/substrate-version`. An actor occupies a +whole worker, so the pool size is the number of tasks that can run at once; set `--workers` on +`ousast plane run` to match. Size the workers above the largest Task's limits (`verify`: requests +250m / 384Mi, limits 1 CPU / 1 GiB; `roles` and the others are in `plane/tasks/`), and keep the +golden-actor storage on the node in mind (B.2). + +### B.7 The S3 memory store reachable from where the reconciler runs + +The `s3://` store already works against any reachable S3-compatible server with versioning, +lifecycle rules, object tags and S3 Select (RustFS is the tested one; the bucket setup and the +agent policy are on [RustFS setup](rustfs.md)). Only the reconciler reads and writes it, so today +it must be reachable from the operator's host; if the reconciler moves into the cluster, from its +pod. No Task talks to the store directly; the day one does (B.5, path 3), that Task's EgressPolicy +must allow the store's hostname (the policies allow hostnames only, never addresses), the way a +Model declares its hosts today through `openultrasast.io/egress-hosts`. + +### B.8 Secrets in Kubernetes Secrets, injected into the reconciler's environment + +The reconciler reads the variable a Model's `secretKey.key` names (`DEEPSEEK_API_KEY`, +`OPENROUTER_API_KEY`) and the store's `S3_ENDPOINT`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` +(optional `AWS_SESSION_TOKEN`, `S3_REGION`, `S3_BUCKET`) from its own environment or `.env`. When +the reconciler runs in the cluster, keep them in Kubernetes Secrets and inject them into **the +reconciler's** pod environment (`envFrom` with `secretRef`, or `valueFrom.secretKeyRef` per +variable). Never put a value into a manifest: the Model manifests name a variable, the rendered +Task carries no key, and the key reaches an actor only in the start request, where the runner keeps +it in memory and redacts it from echoed stderr. The `secretKey.name` values in `plane/models/` +(`deepseek`, `openrouter`) are the names such Secrets would carry. Running the reconciler +in-cluster is itself planned: today it is a host process, and its state directory +(`OUSAST_RESULTS`) would need a persistent volume. + +### What is implemented and what is planned + +| Item | Status | +| --- | --- | +| ax as the only executor; `plane run|status|doctor|remember|memory-normalise|workspaces|harvest|alerts-engine` | implemented, run on this host | +| Per-task EgressPolicy, credentials only in the start request, per-task budgets and attribution | implemented | +| S3 memory store against any reachable S3-compatible server, verified at startup | implemented (RustFS tested) | +| Local bring-up, doctor, smoke Run (`ops/ax/`) | implemented | +| Re-pinning generated Tasks to another registry (`--runner-image`) | implemented; the committed templates still name `localhost:5001` | +| Deployment to a cluster other than this host's kind cluster | **not done** | +| One configurable kube context | **planned**; `kind-` hard-coded in `doctor.py`, `egress.py`, `router.py` | +| `doctor` registry check against a configurable registry | **planned**; `localhost:5001` hard-coded in `doctor.py:45` | +| Receiver as an in-cluster Deployment | **planned** (code change) | +| Presigned-URL delivery to the S3 store | **follow-on**, not started | +| Reconciler running in-cluster with Secrets injected into its environment | **planned** | +| `AX_SNAPSHOTS_BUCKET` on your own bucket, worker pool sized to the cluster | your deployment's setting; documented, nothing to change in this repository | + +## C. The manifests under `plane/` + +| Path | Moves unchanged? | What changes | +| --- | --- | --- | +| `plane/models/*.yaml` | yes | nothing: provider, prices, `secretKey` variable names and `openultrasast.io/egress-hosts` are cluster-independent | +| `plane/tasks/*.yaml` (templates) | no | only the `image` reference: `/ousast-runner@sha256:` (B.3); resources stay unless the pool is sized differently | +| `plane/tasks/validation-46.yaml`, `plane/runs/*.yaml` | regenerate | generated by `ousast plane workspaces --validation-set ... --runner-image FILE`; the Run spec itself (steps, inputs, outputs, budgets) is cluster-independent | +| `plane/workspaces/*.yaml` | yes | nothing: Git sources at pinned commits; their hosts enter each task's EgressPolicy automatically; inline `files` stay under the ~20 KB limit the actor template tolerates | +| `plane/Dockerfile.runner` | yes | nothing; build from the repository root and push to your registry | +| `ops/ax/receiver-service.yaml.tmpl` | no | the endpoint address (B.5, path 1), or replaced by an in-cluster receiver (path 2) | +| `ops/ax/workerpool.yaml.tmpl` | no | replicas and limits (B.6) | +| `ops/ax/smoke-task.yaml.tmpl`, `smoke-run.yaml.tmpl` | yes | rendered with `RUNNER_IMAGE` from your pin file | +| `ops/ax/up.sh`, `down.sh` | no | kind-specific; they do not apply to another cluster | + +The Run and Task contract itself (`plane/runner.py` as PID 1 of the runner image, `AX_TASK_YAML` +and `AX_WORKSPACES_YAML`, `/healthz` and `/readyz`, the start request, the tar delivery as +completion) does not depend on where the cluster is: [The plane on ax](plane.md). diff --git a/docs/evaluation.md b/docs/evaluation.md index 63b86d4..ee303a3 100644 --- a/docs/evaluation.md +++ b/docs/evaluation.md @@ -83,8 +83,9 @@ flowchart TD `assert_disjoint` runs before a compile: the compile split and the evaluation folds share no group. A framework with fewer than 10 groups gets `insufficient data` instead of a fold. As of -2026-10-02 only the outer folds of the injection family have been run -([decision-engine.md](decision-engine.md#first-measured-slice-2026-10-01)). +2026-10-02 the outer folds have been run for the six evaluable families; leave-one-source-out +and leave-one-framework-out have not +([decision-engine.md](decision-engine.md#second-slice-six-families-2026-10-02)). ## The plane increment @@ -111,8 +112,10 @@ gate: at least 16 of 20 declared sites agreed, at a cost per candidate under $0. ## The decision engine -The first measured slice of the learned decision engine (injection family, out-of-repository -evaluation) is described with its numbers in [decision-engine.md](decision-engine.md). +Two slices of the learned decision engine have been measured out of repository: injection alone +(2026-10-01) and the six evaluable families (2026-10-02: pooled AUC 0.77 to 0.96 per family, +within-pair AUC 0.72 to 0.93, BLOCK offered for none). The numbers and their records are in +[decision-engine.md](decision-engine.md). The engine is not adopted. ## Removal of the earlier agentic extra diff --git a/docs/examples.md b/docs/examples.md index 0d642b1..6922646 100644 --- a/docs/examples.md +++ b/docs/examples.md @@ -21,8 +21,8 @@ ousast scan . --mode standard # Joern model layer, container network off: ousast-with-judge scan . --mode standard # network on, so the model is asked (needs DEEPSEEK_API_KEY) ``` -Without a key the graph still decides; only the `suspicion` band goes unasked, and the manifest -records the degradation. +Without a key the graph still decides: its entailed findings are reported, and the LLM's residual +and suspicion questions go unasked. ## 3. CI gate diff --git a/docs/index.md b/docs/index.md index 2e98c2a..c8d98d3 100644 --- a/docs/index.md +++ b/docs/index.md @@ -49,6 +49,8 @@ flowchart LR - Understand a finding's evidence and score: [Architecture](architecture.md). - Operate the plane and its store: [ax on this host](ops/ax/README.md) and [RustFS setup](rustfs.md). +- Where the plane runs today and what a separate Kubernetes cluster would take (planned, not + done): [Deployment](deployment.md). - Where the model spend goes, per Task at runtime and in the maintainers' own coding sessions, and what is reused instead of paid again: [Token ergonomics](token-ergonomics.md). - Trust boundaries and hardening: [Threat model](threat-model.md). diff --git a/docs/memory-and-detection.md b/docs/memory-and-detection.md index fcc4cdd..edb2725 100644 --- a/docs/memory-and-detection.md +++ b/docs/memory-and-detection.md @@ -151,7 +151,9 @@ $0) is on [Token ergonomics](token-ergonomics.md#measured-costs). | ADVISORY | recall 0.90 (0.82 - 0.97), precision 0.54 (0.44 - 0.65) | precision is at the positive share (0.53), so at this threshold ADVISORY is **not discriminating**: it flags nearly as indiscriminately as flagging everything would | | BLOCK | not offered | calibration does not hold (ECE 0.064 > 0.05), and every outer fold reported `precision_unreachable` | -**The engine is not adopted.** These numbers come from one family and one slice; the other -families, the leave-one-source-out and leave-one-framework-out evaluations, integration into scan -and pre-push, and the one-time check on population v3 are still ahead -([Decision engine](decision-engine.md#first-measured-slice-2026-10-01)). +**The engine is not adopted.** A second slice (2026-10-02) evaluated the six evaluable families +the same way: pooled AUC 0.77 to 0.96 per family, within-pair AUC 0.72 to 0.93, BLOCK offered for +none, re-run agreement below 0.9 for three families +([Decision engine](decision-engine.md#second-slice-six-families-2026-10-02)). Leave-one-source-out +and leave-one-framework-out, integration into scan and pre-push, and the one-time check on +population v3 are still ahead. diff --git a/docs/plane.md b/docs/plane.md index 8da1b26..50d275a 100644 --- a/docs/plane.md +++ b/docs/plane.md @@ -5,8 +5,11 @@ Model-driven work over whole repositories runs on a separate agentic plane: a ** an ax **Task** (`plane/tasks/`) that binds its Workspaces (`plane/workspaces/`), at most one ax **Model** (`plane/models/`: `deepseek-flash` for chat, `openrouter-embedding` for embeddings) and its own `budget: {usd, calls}`. ax (google/ax over Agent Substrate, in a single-node kind cluster -on the maintainer's host) is the only executor. Bring-up and the host's lessons are in -[ax on this host](ops/ax/README.md); this page draws the flow. +on the maintainer's host) is the only executor; there is no local subprocess path. Bring-up and +the host's lessons are in [ax on this host](ops/ax/README.md); where each part runs today (the +reconciler and the receiver on the host, the tasks in gVisor on Agent Substrate) and what a +separate Kubernetes cluster would take (planned) are on [Deployment](deployment.md); this page +draws the flow. Sources: `src/openultrasast/plane/reconciler.py`, `router.py`, `egress.py`, `runner.py`, `generate.py`, `memory.py`, and `src/openultrasast/plane/tasks/`. diff --git a/docs/scanning.md b/docs/scanning.md index fb6882d..1057a36 100644 --- a/docs/scanning.md +++ b/docs/scanning.md @@ -30,7 +30,7 @@ flowchart TD | Mode | Adds | Needs | Without it | | --- | --- | --- | --- | | `quick` | pattern rules, entry-point hints, ranking, verification, score | nothing | - | -| `standard` | MAP: complexity map, semantic overlay, authorization obligations, the Joern model layer; `fusion` (deterministic) | Joern; a provider key for the residual LLM question | `cpg_unavailable` degradation; the LLM parts are skipped and recorded | +| `standard` | MAP: complexity map, semantic overlay, authorization obligations, the Joern model layer; `fusion` (deterministic) | Joern; a provider key for the residual LLM question | `cpg_unavailable` degradation; the LLM's questions are skipped and only the graph's entailed findings are reported | | `deep` | REGRESS: promoted candidates loaded in a sandbox with no network, read-only source, non-root | a working `docker` | `sandbox_unavailable` degradation | Sources: `README.md` (scan modes), `src/openultrasast/cli.py` (`run_stage` calls), diff --git a/docs/threat-model.md b/docs/threat-model.md index 5a90711..7261399 100644 --- a/docs/threat-model.md +++ b/docs/threat-model.md @@ -59,9 +59,10 @@ LLM/embedding calls retry transient failures (HTTP 429/5xx, connection errors, t with exponential backoff; non-transient errors (e.g. 4xx, malformed JSON) fail fast. ### Visible degradation & determinism -When an optional capability is missing (Joern, Docker, a model, a provider key), the scan -falls back to its deterministic equivalent and records a `degradations` entry in the -manifest — it is never silently downgraded. Runs are reproducible: fixed config, +When an optional engine is missing (Joern: `cpg_unavailable`; Docker: `sandbox_unavailable`), +the scan falls back to its deterministic equivalent and records a `degradations` entry in the +manifest — it is never silently downgraded. Without a provider key the model layer asks no +question and reports only what the graph entailed. Runs are reproducible: fixed config, artifact manifests, prompt hashes, and model identifiers are recorded. ### MCP surface diff --git a/docs/token-ergonomics.md b/docs/token-ergonomics.md index fbf2990..e56b965 100644 --- a/docs/token-ergonomics.md +++ b/docs/token-ergonomics.md @@ -108,6 +108,7 @@ or a file. | Plane increment 2: verify a, b and the tie-break on the 46-candidate validation set | $0.0205 per candidate over the 46 including the recorded triage, against the $0.022 reference (`gate.cost_per_candidate_with_recorded_triage_usd`); totals: plane $0.8567, recorded triage $0.0872, with triage $0.9439, reference $1.0123 (`detail.usd`); the tie-break asked 10 disputed candidates and flipped 5 to agreed | `benchmarks/independent/plane-increment-2.json` | | Plane increment 1: verify a and b alone | 357 calls, 54 hunts at 5.37 turns per hunt, $0.7471 plus $0.0872 recorded triage (`detail`) | `benchmarks/independent/plane-increment-1.json` | | Decision engine, injection slice: evaluation at k = 5 | $0.003215 per candidate (`spend.cost_per_candidate_evaluate_usd`); metered at list price: canary $0.010158, compile $0.429194, a first compile attempt that failed with HTTP 400 $0.049047, evaluate $0.353626, smoke $0.070812, in all $0.912838, plus $0.003976 of embeddings; 8,601,840 of 9,936,337 prompt tokens were cache hits (`spend`) | `benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json` | +| Decision engine, slice 2 (six families): compile, canary and evaluation per family | $2.8832 metered at list price for DeepSeek under an $8 ceiling (`spend.metered_usd_deepseek_total`), $0.000868 of embeddings; $0.0027 to $0.0032 per evaluated candidate (`families..spend.cost_per_candidate_usd`); 25,633,349 of 29,779,297 prompt tokens were cache hits (`spend.deepseek_tokens`); balance $17.73 to $16.73 | `benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json` | | Decision engine harvest: `verify` part | $2.5373 over 1,961 calls, 474 tasks done, 10,921,167 prompt tokens of which 6,275,060 cache hits, 306,941 output tokens, task budgets summing to $8.624 under a $10 ceiling (`verify`) | `benchmarks/measurements/2026-10-01-decision-engine-harvest/record.json` | | Decision engine harvest: `roles` part | $1.4036 over 1,034 calls, 151 tasks done, 2,398,669 prompt tokens of which 329,600 cache hits, 370,155 output tokens, task budgets summing to $4.917 under a $10 ceiling (`roles`) | same record | diff --git a/mkdocs.yml b/mkdocs.yml index 3b6d312..add431b 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -64,4 +64,5 @@ nav: - Operations (maintainers): - Engine and pre-push: ops/README.md - ax on this host: ops/ax/README.md + - Deployment: deployment.md - RustFS setup for the memory store: rustfs.md diff --git a/ops/README.md b/ops/README.md index f04bc8d..002da85 100644 --- a/ops/README.md +++ b/ops/README.md @@ -47,7 +47,7 @@ TARGET=/path/to/repo docker compose run --rm ousast scan /target The target is mounted read-only; `/work` is a tmpfs so CPG scratch never outlives the run. The LLM endpoint is optional and passed through the environment, never baked into a layer — without it the model layer still -entails and only the `suspicion` band goes unasked. `network_mode` defaults to `none`; set it to `bridge` +entails, and its residual and suspicion questions go unasked. `network_mode` defaults to `none`; set it to `bridge` (`OUSAST_NETWORK=bridge`, which `ousast-with-judge` does) only when the model layer should reach the model endpoint. Compose passes `DEEPSEEK_API_KEY` and `OPENROUTER_API_KEY` through from the environment. @@ -377,4 +377,5 @@ read under the same analysis deadline. Model-driven work over whole repositories runs as tasks on google/ax over Agent Substrate. Bring-up, the task catalogue, the memory store, the measured footprint, the lessons from the live cluster and the checklist for a separate Kubernetes cluster are in [`ops/ax/README.md`](ax/README.md); -`ousast plane run|status|doctor|workspaces|remember|alerts-engine|harvest` drives it. +`ousast plane run|status|doctor|remember|memory-normalise|alerts-engine|harvest|workspaces` drives it. Where each part +runs today and the guidance for a separate Kubernetes cluster (planned, not done) are in `docs/deployment.md`. diff --git a/ops/ax/README.md b/ops/ax/README.md index c4d35d7..6e20d0a 100644 --- a/ops/ax/README.md +++ b/ops/ax/README.md @@ -183,16 +183,18 @@ The real-server contract tests (`OUSAST_MEMORY_TEST_S3=1`, bucket `OUSAST_MEMORY `S3_BUCKET`) verify the bucket at its root and write only under a fresh `contract-/` prefix. They never configure the bucket. -## Moving to a separate Kubernetes cluster (planned) +## Moving to a separate Kubernetes cluster (planned, not done) -The maintainer will deploy the plane to its own Kubernetes environment. The manifests under `plane/` move as they -are; these parts assume this laptop and must change first (checked 2026-09-30): +No deployment outside this host's kind cluster has been made. The step-by-step guidance, with what is implemented +and what is planned, is `docs/deployment.md` (the site page "Deployment"). The Model and Workspace manifests under +`plane/` move as they are and the Task templates need their image reference re-pinned to your registry; these parts +assume this laptop and must change first (checked 2026-10-02): | Assumption | Where | Needed in a real cluster | |---|---|---| | Artifact receiver runs on the developer host, reached through the `ousast-receiver` Service with an EndpointSlice to the kind gateway `172.19.0.1:18090` | `egress.py`, `receiver-service.yaml.tmpl` | the receiver as an in-cluster Deployment (or object storage, e.g. Substrate's S3-compatible store), with the reconciler reading from it | -| Registry `localhost:5001`, rewritten by Substrate for kind | `up.sh`, `doctor.py:45`, runner digest pin | a registry the workers can pull from; keep digest pins (Substrate rejects tags) | -| kubectl context `kind-$KIND_CLUSTER_NAME` | `doctor.py:39`, `egress.py`, `router.py` | one configurable context (for example `OUSAST_KUBE_CONTEXT`) | +| Registry `localhost:5001`, rewritten by Substrate for kind | `up.sh` (`KO_DOCKER_REPO`), `doctor.py:45` (hard-coded check), the runner digest pin, the `image` of every template under `plane/tasks/` | a registry the workers can pull from; keep digest pins (Substrate rejects tags); `--runner-image FILE` re-pins generated Tasks | +| kubectl context `kind-$KIND_CLUSTER_NAME` | `doctor.py:39`, `egress.py:115` and `:148`, `router.py:145`, `up.sh:24` | one configurable context (for example `OUSAST_KUBE_CONTEXT`); not yet configurable | | ax's snapshot bucket: `AX_SNAPSHOTS_BUCKET` in ax's `deploy/ax-server.yaml` points at the ax authors' GCS bucket | ax deploy manifest | your own bucket, set before deploying ax | | Egress gateway applied by hand (agentgateway variant, no Rust build) | this README | the Substrate-installed gateway; per-task EgressPolicies work unchanged | | Worker pool of 2 x 1 CPU / 1.5 GiB for the 7 GB host | `workerpool.yaml.tmpl` | sized to the cluster; `--workers` to match | @@ -203,6 +205,8 @@ a port-forward to any cluster. ## What a larger ax deployment needs -A Kubernetes cluster with Agent Substrate (and its egress gateway), the ax control plane, a registry the workers -can pull from, the runner image pinned by digest, the receiver reachable as a Service, and the provider host -allowed per task. The manifests under `plane/` move unchanged; only `OUSAST_ARTIFACT_*` and the context change. +A Kubernetes cluster with Agent Substrate (and its egress gateway), the ax control plane with `AX_SNAPSHOTS_BUCKET` +pointing at your own bucket, a registry the workers can pull from, the runner image pinned by digest, the receiver +reachable as a Service, and the provider host allowed per task (which each task's EgressPolicy already does). The +Model and Workspace manifests under `plane/` move unchanged; the Task templates get a new image reference; the kube +context and `doctor`'s registry check are not configurable yet (the table above). Step by step: `docs/deployment.md`.