marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: plain-language pass -- deployment, plane, memory, RustFS, token economics

Browse Source 
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-10-02 21:00:37 +02:00
co-authored by Claude Opus 5.5
parent 50f4e69321
commit 7e23379e96
6 changed files with 916 additions and 591 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/deployment.md
+370 -231
View File
@@ -1,17 +1,21 @@
# Deployment
How the agentic plane is deployed today, and what moving it to a separate Kubernetes cluster
would take. State as of 2026-10-02 (v2.0.0). A scan needs none of this: `ousast scan` and
`ousast pre-push` run from the Python package (and the Docker image for Joern, see
[Engine and pre-push](ops/README.md)). The plane is the maintainers' surface for model work over
whole repositories, driven by `ousast plane`.
This page covers two things. First, how the agentic plane is deployed today. Second, what moving it to a
separate Kubernetes cluster would take. It describes the state as of 2026-10-02 (v2.0.0).
Two words are used strictly on this page. **Implemented** means the code on `main` does it and it
has run on the maintainer's host. **Planned** means it has not been done; the guidance is derived
from the code and the host's lessons, and the code changes it needs are named with their file paths.
A scan needs none of this. `ousast scan` and `ousast pre-push` run from the Python package. Joern runs
from the Docker image (see [Engine and pre-push](ops/README.md)). The plane is the maintainers' tool for
model work over whole repositories. `ousast plane` drives it.
This page uses two words strictly:
- **Implemented** means the code on `main` does it, and it has run on the maintainer's host.
- **Planned** means it has not been done. The guidance comes from the code and from what the host taught
us. Each code change it needs is named with its file path.
**No deployment of the plane to a cluster other than the maintainer's single-node kind cluster has
been made.** Section B is guidance, not a record.
been made.** Section B is guidance, not a record. (kind runs a Kubernetes cluster inside Docker
containers on one machine.)
## What runs where today (implemented)
@@ -51,14 +55,21 @@ flowchart LR
| Memory store | `OUSAST_MEMORY`: a file store under `~/ousast-results/plane/memory` by default, or `s3://<bucket>[/<prefix>]` on any reachable S3-compatible server (RustFS is the tested one) | `src/openultrasast/plane/memory.py`, [Memory](memory.md), [RustFS setup](rustfs.md) |
| Run state and artifacts | `$OUSAST_RESULTS/plane/<run>/` on the host (default `~/ousast-results/plane/`) | `reconciler.py` (`run_dir`) |
Only the reconciler talks to the memory store: a sandboxed task cannot read it, so the loop's
`memory-snapshot` is seeded by the reconciler before the Run ([The plane on ax](plane.md)).
Terms in the table:
- **ax** is Google's controller that runs agent tasks on Kubernetes.
- **Agent Substrate** is the layer below ax that owns the machines and runs sandboxes.
- **gVisor** is a sandbox runtime that isolates each actor from the node's kernel.
- An **actor** is one running Task inside a sandbox.
Only the reconciler talks to the memory store. A sandboxed task cannot read it. So the reconciler seeds
the loop's `memory-snapshot` before the Run starts ([The plane on ax](plane.md)).
## A. This host: the local setup (implemented)
What `ops/ax/up.sh` needs on the PATH: `docker`, `go`, `kubectl`, `kind`, `ko` and `ax` (it
refuses to start when one is missing). Tools live in `~/go/bin` and `~/.local/bin`; sources and
rendered files in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`).
`ops/ax/up.sh` needs these tools on the PATH: `docker`, `go`, `kubectl`, `kind`, `ko` and `ax`. It
refuses to start when one is missing. Tools live in `~/go/bin` and `~/.local/bin`. Sources and rendered
files live in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`).
1. **Bring up the cluster and the plane.**
@@ -66,15 +77,20 @@ rendered files in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`).
ops/ax/up.sh
```
Idempotent: each step is skipped when already done. It creates the kind cluster
(`KIND_CLUSTER_NAME`, default `ousast`) with its registry (`KO_DOCKER_REPO`, default
`localhost:5001`), installs Agent Substrate, applies the egress gateway (the agentgateway
variant, one prebuilt image) when the install left it out, deploys the ax control plane, applies
the receiver Service with the host's kind-gateway address as its endpoint, applies the gVisor
worker pool, builds and pushes the runner image and writes its digest pin to
`~/.cache/ousast/ax-src/runner-image`, installs the `ax` CLI from the deployed checkout (a release
CLI skews from the server), and runs one smoke Task. It ends by printing the host's memory and the
kind containers' usage.
The script is idempotent: it skips each step that is already done. In order, it:
- creates the kind cluster (`KIND_CLUSTER_NAME`, default `ousast`) with its registry
(`KO_DOCKER_REPO`, default `localhost:5001`);
- installs Agent Substrate;
- applies the egress gateway if the install left it out (the agentgateway variant, one prebuilt image);
- deploys the ax control plane;
- applies the receiver Service, with the host's kind-gateway address as its endpoint;
- applies the gVisor worker pool;
- builds and pushes the runner image, and writes its digest pin to `~/.cache/ousast/ax-src/runner-image`;
- installs the `ax` CLI from the deployed checkout (a release CLI does not match the server version);
- runs one smoke Task.
At the end it prints the host's memory and the kind containers' usage.
2. **Check it.**
@@ -82,9 +98,15 @@ rendered files in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`).
uv run ousast plane doctor
```
Four checks: the kind context is reachable, every pod in `ate-system` and in `ax-system` is ready,
and the registry at `localhost:5001` holds `ousast-runner`. Each failing line names
`ops/ax/up.sh` as the fix. `doctor` is informational: `ousast plane run` does not call it.
It runs four checks:
- the kind context is reachable;
- every pod in `ate-system` is ready;
- every pod in `ax-system` is ready;
- the registry at `localhost:5001` holds `ousast-runner`.
Each failing line names `ops/ax/up.sh` as the fix. `doctor` is informational only.
`ousast plane run` does not call it.
3. **Run the end-to-end smoke Run.**
@@ -92,23 +114,27 @@ rendered files in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`).
ops/ax/smoke-run.sh
```
It renders `ops/ax/smoke-run.yaml.tmpl` with the current runner digest and submits the Run
`ax-e2e-smoke` through the reconciler: one model-free `repo-facts` Task (`budget: {usd: 0,
calls: 0}`) over a files-only Workspace, resumed, started through the router, its `facts.json` and
`summary.json` delivered to the host, then `ousast plane status ax-e2e-smoke` prints the attribution
table. The script calls the repository's `.venv/bin/ousast`, so `uv sync` first.
It renders `ops/ax/smoke-run.yaml.tmpl` with the current runner digest. It then submits the Run
`ax-e2e-smoke` through the reconciler. The Run has one model-free `repo-facts` Task (`budget: {usd: 0,
calls: 0}`) over a files-only Workspace, resumed, started through the router, its `facts.json`
and `summary.json` delivered to the host. Then `ousast plane status ax-e2e-smoke`
prints the attribution table. The script calls the repository's `.venv/bin/ousast`, so run `uv sync`
first.
4. **Keys and the store.** Put `DEEPSEEK_API_KEY` (chat) and `OPENROUTER_API_KEY` (embeddings) in
`.env` or export them; a Task's Model names the variable (`secretKey.key`), and the key travels
only in that Task's start request. Set `OUSAST_MEMORY=s3://<bucket>` with `S3_ENDPOINT`,
`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` (optional `S3_REGION`, `S3_BUCKET`) for the S3 store;
the bucket must be set up once by an admin as [RustFS setup](rustfs.md) describes, and the store
verifies it at every start.
4. **Keys and the store.**
- Put `DEEPSEEK_API_KEY` (chat) and `OPENROUTER_API_KEY` (embeddings) in `.env`, or export them.
- A Task's Model names the variable (`secretKey.key`). The key travels only in that Task's start
request.
- For the S3 store, set `OUSAST_MEMORY=s3://<bucket>` with `S3_ENDPOINT`, `AWS_ACCESS_KEY_ID` and
`AWS_SECRET_ACCESS_KEY`. `S3_REGION` and `S3_BUCKET` are optional.
- An admin must set up the bucket once, as [RustFS setup](rustfs.md) describes. The store verifies
it at every start.
5. **Run a real Run**, as in [Examples](examples.md#7-running-the-agentic-plane): `ousast plane run
plane/runs/validation-46.yaml`, then `status` and `remember`.
6. **Tear down** with `ops/ax/down.sh`, which deletes the cluster and its registry.
6. **Tear down** with `ops/ax/down.sh`. It deletes the cluster and its registry.
**Footprint** (measured 2026-09-29, idle, two workers; [ax on this host](ops/ax/README.md#measured-footprint-2026-09-29-idle-two-workers)):
@@ -119,44 +145,62 @@ rendered files in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`).
| node volume (images, containerd) | 4.9 GB on disk |
| worker pool | 2 gVisor workers, 1 CPU / 1.5 GiB limit each |
Disk is the binding constraint: the Substrate install and the `ko` builds need several GB of Go
caches (`go clean -cache -modcache` afterwards), and every Task is its own actor template with a
golden actor (about 24 MB each under `/var/lib/ate/actors` on the node). The failures the host
produced, and what each one taught, are listed in
[ax on this host](ops/ax/README.md#what-this-host-taught-each-one-cost-a-failed-live-run).
Disk is the tightest limit, for two reasons:
- The Substrate install and the `ko` builds need several GB of Go caches. Run `go clean -cache -modcache`
afterwards.
- Every Task is its own actor template with a golden actor (a saved snapshot the actor boots from). Each
takes about 24 MB under `/var/lib/ate/actors` on the node.
[ax on this host](ops/ax/README.md#what-this-host-taught-each-one-cost-a-failed-live-run) lists the
failures the host produced and what each one taught.
## B. A separate Kubernetes cluster (planned)
The target: a cluster that runs Agent Substrate with its egress gateway and the ax control plane,
pulls the runner image from a registry of yours, receives task artifacts without a developer
laptop in the path, and keeps its memory in your S3 bucket. The provider key already travels only
in the start request through `atenet-router`, which works the same through a port-forward to any
cluster. Everything else below is either a step you take or a code change that is not made yet.
The target cluster would do four things:
- run Agent Substrate with its egress gateway and the ax control plane;
- pull the runner image from a registry of yours;
- receive task artifacts without a developer laptop in the path;
- keep its memory in your S3 bucket.
The provider key already travels only in the start request through `atenet-router`. That works the same
through a port-forward to any cluster. Everything else below is either a step you take or a code change
that is not made yet.
### B.1 Cluster with Agent Substrate, its egress gateway and ax
- Install Agent Substrate on the cluster following its own instructions. `ops/ax/up.sh` uses
Substrate's kind helpers (`hack/create-kind-cluster.sh`, `hack/install-ate-kind.sh`), which do
not apply to another cluster.
- Install Agent Substrate on the cluster with its own instructions. `ops/ax/up.sh` uses Substrate's kind
helpers (`hack/create-kind-cluster.sh`, `hack/install-ate-kind.sh`). These do not apply to another
cluster.
- Apply the egress gateway. On this host it is the agentgateway variant
(`manifests/ate-install/agentgateway-egress` in the Substrate checkout, applied with
`kubectl kustomize --load-restrictor=LoadRestrictionsNone`), which needs no Rust build; the
Envoy variant does. Without the gateway an actor has no network at all, and the per-task
EgressPolicies the reconciler writes (`src/openultrasast/plane/egress.py`) work unchanged
against the Substrate-installed gateway: deny by default, hostnames only, plain HTTP to the
receiver, TLS passthrough to the Workspaces' Git hosts and the Model's declared hosts.
(`manifests/ate-install/agentgateway-egress` in the Substrate checkout). It is applied with
`kubectl kustomize --load-restrictor=LoadRestrictionsNone`. It needs no Rust build; the Envoy variant
does.
- Without the gateway an actor has no network at all.
- The reconciler writes one EgressPolicy per task (`src/openultrasast/plane/egress.py`). These work
unchanged against the gateway that Substrate installs. The rules are:
- deny by default;
- hostnames only;
- plain HTTP to the receiver;
- TLS passthrough to the Workspaces' Git hosts and the Model's declared hosts.
- Deploy ax (`make deploy AX_IMAGE_REPO=<your registry>` in the ax checkout) **after** B.2.
- Apply a WorkerPool (B.6). Substrate installs none by itself ("no free workers").
### B.2 ax's snapshot bucket: `AX_SNAPSHOTS_BUCKET`
ax's deploy manifest (`deploy/ax-server.yaml` in the ax checkout) sets `AX_SNAPSHOTS_BUCKET` to
the ax authors' own bucket (`gs://dberkov-gke-dev3/ate-env/` in the checkout used here). Point it
at **your own bucket before deploying ax**. On this host the bucket named there did not exist in
the S3 server, deleted golden actors piled up in `DELETING` and filled the disk during a 600-task
harvest (2026-10-01); creating the bucket let deletes complete, and even then a deleted actor's
directory stays on the node, so long Runs need a janitor that removes directories no live actor
owns.
Point `AX_SNAPSHOTS_BUCKET` at **your own bucket before deploying ax**. ax's deploy manifest
(`deploy/ax-server.yaml` in the ax checkout) sets it to the ax authors' own bucket. In the checkout used
here that is `gs://dberkov-gke-dev3/ate-env/`.
This host shows what goes wrong otherwise (2026-10-01):
1. The bucket named there did not exist in the S3 server.
2. Deleted golden actors piled up in `DELETING`.
3. They filled the disk during a 600-task harvest.
Creating the bucket let deletes complete. Even then, a deleted actor's directory stays on the node. So
long Runs need a janitor that removes directories no live actor owns.
### B.3 A registry the workers can pull from, the runner image pinned by digest
@@ -168,83 +212,117 @@ owns.
docker inspect --format '{{range .RepoDigests}}{{println .}}{{end}}' <registry>/ousast-runner:dev
```
- Substrate rejects tags: every Task's `image` must be the `<registry>/ousast-runner@sha256:...`
reference. Write it into a file (on this host, `~/.cache/ousast/ax-src/runner-image`) and pass
`--runner-image FILE` to `ousast plane workspaces` and `ousast plane harvest`, which re-pin the
Tasks they generate. The committed templates under `plane/tasks/` and the generated
`plane/tasks/validation-46.yaml` carry `localhost:5001/ousast-runner@sha256:...` and need the same
re-pin (regenerate, or replace the reference) before they run elsewhere.
- Workers must be able to pull from the registry (credentials, if any, are the cluster's concern;
ax's default runner image on `gcr.io` needs credentials and is not used here).
- Substrate rejects tags. Every Task's `image` must be the `<registry>/ousast-runner@sha256:...`
reference.
- Write that reference into a file. On this host the file is `~/.cache/ousast/ax-src/runner-image`.
- Pass `--runner-image FILE` to `ousast plane workspaces` and `ousast plane harvest`. They re-pin the
Tasks they generate.
- The committed templates under `plane/tasks/` and the generated `plane/tasks/validation-46.yaml` carry
`localhost:5001/ousast-runner@sha256:...`. They need the same re-pin before they run elsewhere.
Regenerate them, or replace the reference.
- Workers must be able to pull from the registry. Credentials, if any, are the cluster's concern. ax's
default runner image on `gcr.io` needs credentials and is not used here.
- **Not yet configurable:** `ousast plane doctor` checks the registry at `localhost:5001` by name
(`src/openultrasast/plane/doctor.py:45`), so its fourth check fails on another cluster even when
the image is in place. `doctor` is informational, `plane run` does not depend on it.
(`src/openultrasast/plane/doctor.py:45`). So its fourth check fails on another cluster, even when the
image is in place. `doctor` is informational; `plane run` does not depend on it.
### B.4 One configurable kube context
The reconciler, the egress client and the router port-forward all use the context
`kind-$KIND_CLUSTER_NAME` (`kind-ousast` by default). **Not yet configurable**: the prefix `kind-`
is hard-coded in `src/openultrasast/plane/doctor.py:39`, `src/openultrasast/plane/egress.py:115`
and `:148`, and `src/openultrasast/plane/router.py:145`; `ops/ax/up.sh:24` builds the same name.
The planned change is one variable (for example `OUSAST_KUBE_CONTEXT`) read in those three
modules. Until then the only way to drive another cluster from the unchanged code is to give its
kubeconfig context a name of that form (`kubectl config rename-context <ctx> kind-<name>` and
`KIND_CLUSTER_NAME=<name>`), which is a workaround, not a supported configuration. The egress
client also needs `kubectl-ate` (or the executable `OUSAST_KUBECTL_ATE` names) able to use that
context.
The reconciler, the egress client and the router port-forward all use one kube context. Its name is
`kind-$KIND_CLUSTER_NAME` (`kind-ousast` by default).
**Not yet configurable**: the prefix `kind-` is hard-coded in these places:
- `src/openultrasast/plane/doctor.py:39`;
- `src/openultrasast/plane/egress.py:115` and `:148`;
- `src/openultrasast/plane/router.py:145`;
- `ops/ax/up.sh:24` builds the same name.
The planned change is one variable read in those three modules, for example `OUSAST_KUBE_CONTEXT`.
Until then, there is one workaround to drive another cluster with the unchanged code. Rename its
kubeconfig context to that form: `kubectl config rename-context <ctx> kind-<name>` and
`KIND_CLUSTER_NAME=<name>`. This is a workaround, not a supported configuration.
The egress client also needs `kubectl-ate` (or the executable `OUSAST_KUBECTL_ATE` names). It must be
able to use that context.
### B.5 The receiver: in-cluster instead of on the host
Today the receiver is a thread inside the reconciler process (`egress.py`, `Receiver`), and the
reconciler learns of a task's completion by watching that object's `delivered` set in memory
(`reconciler.py`). Actors reach it through the Service `ousast-receiver.ax-system` whose
EndpointSlice points at the host. Three paths, in order of effort:
Today the receiver is a thread inside the reconciler process (`egress.py`, `Receiver`). The reconciler
learns that a task finished by watching that object's `delivered` set in memory (`reconciler.py`).
Actors reach it through the Service `ousast-receiver.ax-system`. That Service's EndpointSlice points at
the host.
1. **Interim, works with the unchanged code:** keep the host receiver and render
`ops/ax/receiver-service.yaml.tmpl` with an endpoint address the egress gateway pod can reach
(`KIND_GATEWAY` is just an IPv4 address in the template, `OUSAST_ARTIFACT_PORT` the port). The
reconciler's host must be reachable from the cluster (a VPN or a routed address), and the
gateway dials the address the actor connected to, so the Service's ClusterIP must still map to it.
2. **Planned: the receiver as an in-cluster Deployment** behind the same Service name, with the
reconciler reading deliveries and serving inputs from it instead of from its own thread. This
needs code: the receiver and its `delivered`/`inputs` state are in-process today, and the
runner's `GET /inputs/<producer>/<artifact>` is served by that same process.
3. **Follow-on: presigned-URL delivery to the S3 store.** The runner would `PUT` its output tar to a
presigned URL on the memory store's bucket and fetch inputs the same way, and the reconciler
would read from the bucket. Nothing of this exists; the EgressPolicy would then have to allow
the store's hostname for the task.
There are three paths, from least to most effort:
1. **Interim, works with the unchanged code:** keep the host receiver.
Render `ops/ax/receiver-service.yaml.tmpl` with an endpoint address the egress gateway pod can reach.
In the template, `KIND_GATEWAY` is just an IPv4 address and `OUSAST_ARTIFACT_PORT` is the port.
The cluster must be able to reach the reconciler's host, through a VPN or a routed address.
The gateway dials the address the actor connected to. So the Service's ClusterIP must still map to it.
2. **Planned: the receiver as an in-cluster Deployment** behind the same Service name.
The reconciler would read deliveries from it and serve inputs from it, instead of from its own thread.
This needs code. The receiver and its `delivered`/`inputs` state are in-process today.
The same process serves the runner's `GET /inputs/<producer>/<artifact>`.
3. **Follow-on: presigned-URL delivery to the S3 store.** (A presigned URL grants one upload or download
without credentials.)
The runner would `PUT` its output tar to a presigned URL on the memory store's bucket.
It would fetch inputs the same way, and the reconciler would read from the bucket.
None of this exists. The EgressPolicy would then have to allow the store's hostname for the task.
### B.6 Worker pool sizing
`ops/ax/workerpool.yaml.tmpl` declares two warm gVisor workers of 1 CPU / 1.5 GiB (requests 250m /
1.5 GiB) for the 7 GB host, selected onto nodes by `ate.dev/substrate-version`. An actor occupies a
whole worker, so the pool size is the number of tasks that can run at once; set `--workers` on
`ousast plane run` to match. Size the workers above the largest Task's limits (`verify`: requests
250m / 384Mi, limits 1 CPU / 1 GiB; `roles` and the others are in `plane/tasks/`), and keep the
golden-actor storage on the node in mind (B.2).
`ops/ax/workerpool.yaml.tmpl` declares two warm gVisor workers for the 7 GB host. Each has 1 CPU /
1.5 GiB (requests 250m / 1.5 GiB). They are placed on nodes by `ate.dev/substrate-version`.
An actor occupies a whole worker. So the pool size is the number of tasks that can run at once. Set
`--workers` on `ousast plane run` to match.
Size the workers above the largest Task's limits. For `verify` these are requests 250m / 384Mi and
limits 1 CPU / 1 GiB. `roles` and the others are in `plane/tasks/`. Also keep the golden-actor storage
on the node in mind (B.2).
### B.7 The S3 memory store reachable from where the reconciler runs
The `s3://` store already works against any reachable S3-compatible server with versioning,
lifecycle rules, object tags and S3 Select (RustFS is the tested one; the bucket setup and the
agent policy are on [RustFS setup](rustfs.md)). Only the reconciler reads and writes it, so today
it must be reachable from the operator's host; if the reconciler moves into the cluster, from its
pod. No Task talks to the store directly; the day one does (B.5, path 3), that Task's EgressPolicy
must allow the store's hostname (the policies allow hostnames only, never addresses), the way a
Model declares its hosts today through `openultrasast.io/egress-hosts`.
The `s3://` store already works against any reachable S3-compatible server. The server must support:
- versioning;
- lifecycle rules;
- object tags;
- S3 Select (a server-side query over JSON objects).
RustFS is the tested server. The bucket setup and the agent policy are on [RustFS setup](rustfs.md).
Only the reconciler reads and writes the store. So today the operator's host must reach it. If the
reconciler moves into the cluster, its pod must reach it.
No Task talks to the store directly. If one ever does (B.5, path 3), its EgressPolicy must allow the
store's hostname. The policies allow hostnames only, never addresses. A Model already declares its hosts
this way today, through `openultrasast.io/egress-hosts`.
### B.8 Secrets in Kubernetes Secrets, injected into the reconciler's environment
The reconciler reads the variable a Model's `secretKey.key` names (`DEEPSEEK_API_KEY`,
`OPENROUTER_API_KEY`) and the store's `S3_ENDPOINT`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`
(optional `AWS_SESSION_TOKEN`, `S3_REGION`, `S3_BUCKET`) from its own environment or `.env`. When
the reconciler runs in the cluster, keep them in Kubernetes Secrets and inject them into **the
reconciler's** pod environment (`envFrom` with `secretRef`, or `valueFrom.secretKeyRef` per
variable). Never put a value into a manifest: the Model manifests name a variable, the rendered
Task carries no key, and the key reaches an actor only in the start request, where the runner keeps
it in memory and redacts it from echoed stderr. The `secretKey.name` values in `plane/models/`
(`deepseek`, `openrouter`) are the names such Secrets would carry. Running the reconciler
in-cluster is itself planned: today it is a host process, and its state directory
The reconciler reads these variables from its own environment or `.env`:
- the variable a Model's `secretKey.key` names (`DEEPSEEK_API_KEY`, `OPENROUTER_API_KEY`);
- the store's `S3_ENDPOINT`, `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`;
- optionally `AWS_SESSION_TOKEN`, `S3_REGION` and `S3_BUCKET`.
When the reconciler runs in the cluster, keep these in Kubernetes Secrets. Inject them into **the
reconciler's** pod environment. Use `envFrom` with `secretRef`, or `valueFrom.secretKeyRef` per variable.
Never put a value into a manifest. The key stays out of every manifest:
- the Model manifests name a variable;
- the rendered Task carries no key;
- the key reaches an actor only in the start request;
- the runner keeps it in memory and redacts it from echoed stderr.
The `secretKey.name` values in `plane/models/` (`deepseek`, `openrouter`) are the names such Secrets
would carry.
Running the reconciler in-cluster is itself planned. Today it is a host process. Its state directory
(`OUSAST_RESULTS`) would need a persistent volume.
### What is implemented and what is planned
@@ -266,12 +344,20 @@ in-cluster is itself planned: today it is a host process, and its state director
## Production topology: ax on a Kubernetes cluster
The maintainer's questions, answered from the code and the upstream checkouts under
`~/.cache/ousast/ax-src/`: how is it hosted in ax, is there one container, how does ax relate to
Kubernetes, and what scaling the pre-push hook would take. The decisions of 2026-10-02 set the
target: **two profiles of one code base**, `kind` (this laptop, kept as a first-class local
execution mode, where every change is proven first) and `k3s` (production). Everything marked
**planned** is not built; the rest is what runs on this host today.
This section answers the maintainer's questions from the code and the upstream checkouts under
`~/.cache/ousast/ax-src/`:
- How is it hosted in ax?
- Is there one container?
- How does ax relate to Kubernetes?
- What would scaling the pre-push hook take?
The decisions of 2026-10-02 set the target: **two profiles of one code base**.
- `kind` is this laptop. It stays a first-class local execution mode. Every change is proven here first.
- `k3s` is production. (k3s is a small Kubernetes distribution.)
Everything marked **planned** is not built. The rest is what runs on this host today.
```mermaid
flowchart LR
@@ -308,61 +394,92 @@ flowchart LR
cli -- "polls: status, attribution" --> s3
```
No inbound path exists in this picture: the CLI reaches ax-server and the router from outside, the
tasks push their artifacts to the store through the egress gateway, and the CLI reads the store.
Today's EndpointSlice back to a laptop (`ops/ax/receiver-service.yaml.tmpl`) is the kind profile's
device and does not appear in production.
This picture has no inbound path:
- the CLI reaches ax-server and the router from outside;
- the tasks push their artifacts to the store through the egress gateway;
- the CLI reads the store.
Today's EndpointSlice back to a laptop (`ops/ax/receiver-service.yaml.tmpl`) belongs to the kind
profile only. It does not appear in production.
### ax, Kubernetes and Agent Substrate in three sentences
**ax is a Kubernetes application**: a Deployment `ax-server` with a Redis beside it in the
namespace `ax-system` (`ax/deploy/ax-server.yaml` in the ax checkout; its controller runs with
`--template=default-template --template-atespace=ax-system`), and the `ax` CLI reaches it through a
`kubectl port-forward` to `svc/ax-server` (`ax/internal/tunnel/tunnel.go`), so `ousast plane run`
needs a kube context or an ingress, not a public endpoint. **ax does not run containers itself**:
it turns every Task into an ActorTemplate on Agent Substrate (`ax/internal/substrate/client.go`,
`CreateActorTemplate`), built from the Task's image and environment, and the image must be pinned
by digest (Substrate's `SandboxConfig` validation rejects tags: "All images must include a
digest"). **Agent Substrate is the layer that owns the machines**: in the namespace `ate-system`
it runs `ate-api-server`, `ate-controller`, the `atelet` DaemonSet, `atenet-router`, the
`atenet-egress` gateway, a Postgres and a RustFS (`substrate/manifests/ate-install/`), and three
CRDs under `ate.dev/v1alpha1`, `WorkerPool`, `SandboxConfig` and `CSIDriverConfig`; a
`WorkerPool` keeps warm worker pods (`ko://.../cmd/ateom-gvisor`), and an actor is one gVisor
sandbox inside one of those workers, booted from the golden snapshot that
[The plane on ax](plane.md) describes, suspended and resumed by the controller. Names are limited
to 63 bytes (`MaxNameLength` in `ax/pkg/apis/v1alpha1/types.go`), which is why the generated Task
names are short.
**ax is a Kubernetes application.**
**What k3s changes** (planned; nothing has been applied to a k3s cluster): k3s runs containerd
with no Docker socket, so the host-Docker path of `ousast plane alerts-engine` does not exist
there and the engine must become a task (below); Substrate's gVisor workers need `runsc` and a
`RuntimeClass` on the nodes they are scheduled to; k3s ships Traefik (the ingress the CLI can use
instead of a kubeconfig) and local-path storage. Images are built by GitHub Actions from tagged
commits, published to GitHub Container Registry and digest-pinned in the manifests; a private
package needs an `imagePullSecret` on the worker ServiceAccount. Substrate's and ax's own images
are built with `ko` from their checkouts and pushed to the same registry. The kind profile keeps
`localhost:5001` and `ops/ax/up.sh`.
- It is a Deployment `ax-server` with a Redis beside it, in the namespace `ax-system`
(`ax/deploy/ax-server.yaml` in the ax checkout).
- Its controller runs with `--template=default-template --template-atespace=ax-system`.
- The `ax` CLI reaches it through a `kubectl port-forward` to `svc/ax-server`
(`ax/internal/tunnel/tunnel.go`).
- So `ousast plane run` needs a kube context or an ingress, not a public endpoint.
**ax does not run containers itself.**
- It turns every Task into an ActorTemplate on Agent Substrate (`ax/internal/substrate/client.go`,
`CreateActorTemplate`).
- The ActorTemplate is built from the Task's image and environment.
- The image must be pinned by digest. Substrate's `SandboxConfig` validation rejects tags: "All images
must include a digest".
**Agent Substrate is the layer that owns the machines.**
- In the namespace `ate-system` it runs `ate-api-server`, `ate-controller`, the `atelet` DaemonSet,
`atenet-router`, the `atenet-egress` gateway, a Postgres and a RustFS
(`substrate/manifests/ate-install/`).
- It defines three CRDs (custom Kubernetes resource types) under `ate.dev/v1alpha1`: `WorkerPool`,
`SandboxConfig` and `CSIDriverConfig`.
- A `WorkerPool` keeps warm worker pods (`ko://.../cmd/ateom-gvisor`).
- An actor is one gVisor sandbox inside one of those workers.
- The actor boots from the golden snapshot that [The plane on ax](plane.md) describes. The controller
suspends and resumes it.
- Names are limited to 63 bytes (`MaxNameLength` in `ax/pkg/apis/v1alpha1/types.go`). That is why the
generated Task names are short.
**What k3s changes** (planned; nothing has been applied to a k3s cluster):
- k3s runs containerd with no Docker socket. So the host-Docker path of `ousast plane alerts-engine`
does not exist there. The engine must become a task (below).
- Substrate's gVisor workers need `runsc` and a `RuntimeClass` on the nodes they run on.
- k3s ships Traefik and local-path storage. Traefik is an ingress the CLI can use instead of a kubeconfig.
- GitHub Actions builds the images from tagged commits. They are published to GitHub Container Registry
and digest-pinned in the manifests.
- A private package needs an `imagePullSecret` on the worker ServiceAccount.
- Substrate's and ax's own images are built with `ko` from their checkouts. They are pushed to the same
registry.
- The kind profile keeps `localhost:5001` and `ops/ax/up.sh`.
### One image, and what concurrency equals
There is **one container image** for every Task: `plane/Dockerfile.runner` (`python:3.12-slim`
plus this package, PID 1 `/usr/local/bin/ax-task-runner`, which is
`openultrasast.plane.runner`). Each Task manifest under `plane/tasks/` names the same image by
digest (the pin is kept in `~/.cache/ousast/ax-src/runner-image`) and selects its work with
`spec.command`: the runner starts `python -m openultrasast.plane.tasks.<command[0]>`
(`runner.py`, `task_module`). `repo-facts`, `verify`, `agree`, `features`, `remember`, `alerts`,
`loop` and `roles` are modules in that image, not images of their own; the engine would be the
second image (planned). So the production answer to "how many containers" is: one image, one
actor per Task, and **an actor occupies a whole worker** (its limits are the ActorTemplate's, the
worker holds one actor at a time). Concurrency is therefore the number of workers in the
`WorkerPool`, nothing else: `ops/ax/workerpool.yaml.tmpl` declares `replicas: 2` of 1 CPU /
1.5 GiB, so at most two Tasks run at once on this host, and `ousast plane run --workers` must not
exceed the pool. Substrate ships an autoscaled pool
(`substrate/demos/autoscaled-workerpool/README.md`: an HPA on the external metric
`ate_workerpool_workers{state=at_capacity}` through prometheus-adapter, writing
`WorkerPool.spec.replicas`; its README says the demo is kind-only today) and request parking
(`substrate/demos/parking/`: the router holds a resume until a worker frees up). Neither is
applied by `ops/ax/up.sh`; the production profile's pool manifests and autoscaling rule are
There is **one container image** for every Task: `plane/Dockerfile.runner`. It is `python:3.12-slim`
plus this package. PID 1 is `/usr/local/bin/ax-task-runner`, which is `openultrasast.plane.runner`.
Each Task manifest under `plane/tasks/` names the same image by digest. The pin is kept in
`~/.cache/ousast/ax-src/runner-image`. The manifest picks its work with `spec.command`. The runner
starts `python -m openultrasast.plane.tasks.<command[0]>` (`runner.py`, `task_module`).
`repo-facts`, `verify`, `agree`, `features`, `remember`, `alerts`, `loop` and `roles` are modules in
that image. They are not images of their own. The engine would be the second image (planned).
So the production answer to "how many containers" is:
- one image;
- one actor per Task;
- **an actor occupies a whole worker**. Its limits are the ActorTemplate's, and the worker holds one
actor at a time.
Concurrency is therefore the number of workers in the `WorkerPool`, nothing else.
`ops/ax/workerpool.yaml.tmpl` declares `replicas: 2` of 1 CPU / 1.5 GiB. So at most two Tasks run at
once on this host. `ousast plane run --workers` must not exceed the pool.
Substrate ships two scaling aids:
- **An autoscaled pool** (`substrate/demos/autoscaled-workerpool/README.md`). An HPA (Horizontal Pod
Autoscaler) reads the external metric `ate_workerpool_workers{state=at_capacity}` through
prometheus-adapter. It writes `WorkerPool.spec.replicas`. Its README says the demo is kind-only today.
- **Request parking** (`substrate/demos/parking/`). The router holds a resume until a worker frees up.
`ops/ax/up.sh` applies neither. The production profile's pool manifests and autoscaling rule are
**planned**.
### The three components that are bound to the host today
@@ -373,59 +490,75 @@ applied by `ops/ax/up.sh`; the production profile's pool manifests and autoscali
| Artifact receiver (`plane/egress.py`, the `Receiver` thread inside the reconciler) | reached through the Service `ousast-receiver.ax-system` whose EndpointSlice points at the kind gateway address on the host (`ops/ax/receiver-service.yaml.tmpl`) | **no receiver in remote mode.** Tasks deliver their artifacts and fetch their inputs through presigned URLs on the S3 memory store (B.5, path 3; the store's hostname enters each Task's EgressPolicy, B.7), and the CLI reads completion from the store. The kind profile keeps the receiver thread; an in-cluster receiver Deployment (B.5, path 2) is not the chosen direction |
| Joern engine (`plane/engine_alerts.py`, the host's `openultrasast:dev` container, run with `--memory 3g`) | `ousast plane alerts-engine` runs it on the host, one container at a time, and marks the Run's `alerts` tasks done | **an `engine` task on a second, larger `WorkerPool`** (the engine does not fit a 1.5 GiB worker, and k3s has no Docker socket), from an engine image digest-pinned in GHCR, emitting the same `alerts.jsonl` rows and coverage; `alerts-engine` on the host stays as the development path |
The S3 memory store (RustFS tested) is already external; in remote mode it is also the artifact
path, so it must be reachable from the CLI's host and, through the egress gateway, from the
workers.
The S3 memory store (RustFS tested) is already external. In remote mode it is also the artifact path.
So the CLI's host must reach it, and the workers must reach it through the egress gateway.
### Scaling the pre-push hook in production
**Today** `ousast pre-push` is a local, deterministic check: it materialises base and head,
discovers the changed regions, runs the evidence ranker and arbiter on head and compares with base,
all within `--deadline` (30 s by default) and in `--mode advisory` unless blocking is opted into
([Pre-push: the delta check](scanning.md#pre-push-the-delta-check-experimental),
`src/openultrasast/push/`, [Engine and pre-push](ops/README.md#experimental-pre-push-integration)).
**The plane is not in that path**: no hook submits a Run, no model is called without an explicit
`--model-config`, and nothing leaves the developer's machine.
**Today** `ousast pre-push` is a local, deterministic check. It does four things:
1. materialises base and head;
2. finds the changed regions;
3. runs the evidence ranker and arbiter on head;
4. compares the result with base.
All of this runs within `--deadline` (30 s by default). It runs in `--mode advisory` unless you opt
into blocking. See [Pre-push: the delta check](scanning.md#pre-push-the-delta-check-experimental),
`src/openultrasast/push/` and [Engine and pre-push](ops/README.md#experimental-pre-push-integration).
**The plane is not in that path.**
- No hook submits a Run.
- No model is called without an explicit `--model-config`.
- Nothing leaves the developer's machine.
**The production shape** (planned, none of it built) splits the work by latency:
1. **Blocking, inside the hook's deadline**: what runs today, the quick rules and the delta
engine on the changed regions. It stays local and deterministic, so the push is never held for
a model or a cluster.
2. **Asynchronous, on the plane**: the hook or the CI job calls the CLI in remote mode with a
*delta Run* whose candidates are the functions the push changed; the Run is the per-case chain
of [The plane on ax](plane.md#per-case) (`facts`, `verify` a and b, `agree`, `verify` c on the
disputed only, `final`). The caller either waits, with its own deadline, for the CLI to print
the status and attribution table, or returns at once and a later `ousast plane status <run>`
reports from the store; a commit status check or a review comment would be built on that. The
worker pool is sized to the push rate or autoscaled on `at_capacity` workers. An unattended
in-cluster reconciler that accepts submissions without a CLI session is the later option named
above.
1. **Blocking, inside the hook's deadline**: what runs today, the quick rules and the delta engine on
the changed regions. It stays local and deterministic. So the push never waits for a model or a
cluster.
2. **Asynchronous, on the plane**: the hook or the CI job calls the CLI in remote mode with a *delta
Run*.
- The Run's candidates are the functions the push changed.
- The Run is the per-case chain of [The plane on ax](plane.md#per-case): `facts`, `verify` a and b,
`agree`, `verify` c on the disputed only, `final`.
- The caller can wait, with its own deadline, for the CLI to print the status and attribution table.
- Or the caller returns at once, and a later `ousast plane status <run>` reports from the store.
- A commit status check or a review comment would be built on that.
- The worker pool is sized to the push rate, or autoscaled on `at_capacity` workers.
- An unattended in-cluster reconciler that accepts submissions without a CLI session is the later
option named above.
**Cost per push, estimated from measured rates.** Two measured per-candidate rates exist for one
model verdict: `$0.003215` (`spend.cost_per_candidate_evaluate_usd` in
`benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`) and `$0.0021`
(exp-002 arm A, `overall.usd_per_candidate.A` in
`benchmarks/measurements/2026-10-02-exp-002/record.json`). A push that changes five functions needs
two verify passes per candidate: 5 x 2 x $0.0021 = $0.021, or 5 x 2 x $0.003215 = $0.032 at the
higher rate, plus a third pass on the disputed candidates only. That is where "about two cents per
push" comes from; it is arithmetic over the decision engine's measured rates, not a measurement of
a push Run, and `ousast plane status` would print the real figure per Task.
**Cost per push, estimated from measured rates.** Two measured per-candidate rates exist for one model
verdict:
- `$0.003215` (`spend.cost_per_candidate_evaluate_usd` in
`benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`);
- `$0.0021` (exp-002 arm A, `overall.usd_per_candidate.A` in
`benchmarks/measurements/2026-10-02-exp-002/record.json`).
A push that changes five functions needs two verify passes per candidate. That gives 5 x 2 x $0.0021 =
$0.021, or 5 x 2 x $0.003215 = $0.032 at the higher rate. A third pass runs on the disputed candidates
only. That is where "about two cents per push" comes from. It is arithmetic over the decision engine's
measured rates, not a measurement of a push Run. `ousast plane status` would print the real figure per
Task.
**What is not built** (each is a named gap, not a configuration):
- The delta-Run command. `ousast plane scan <repo> --base <commit> --head <commit>` **does not
exist**; `ousast plane` has `run`, `status`, `doctor`, `remember`, `memory-normalise`,
exist**. `ousast plane` has `run`, `status`, `doctor`, `remember`, `memory-normalise`,
`workspaces`, `harvest` and `alerts-engine` (`src/openultrasast/cli.py`).
- A candidates task for an arbitrary repository. `repo-facts` takes its candidates from a
population's validation set; a step that derives them from a diff is not written.
population's validation set. No step derives them from a diff yet.
- Workspace generation from a URL and a commit outside a population. `plane/generate.py` is
population-driven (`read_cases` reads a validation set and a population TOML), so a Run for a
population-driven: `read_cases` reads a validation set and a population TOML. So a Run for a
repository that is not in a population cannot be generated today.
- The remote execution mode itself: the `local`/`remote` switch, configurable cluster context,
ax-server and router addresses (the `kind-` prefix is hard-coded, B.4), presigned delivery from
the runner to the store, and the CLI reading completion and `summary.json` from the store
instead of its receiver thread.
- The remote execution mode itself:
- the `local`/`remote` switch;
- a configurable cluster context and ax-server and router addresses (the `kind-` prefix is
hard-coded, B.4);
- presigned delivery from the runner to the store;
- the CLI reading completion and `summary.json` from the store instead of its receiver thread.
- The engine task and its WorkerPool, the GHCR build, the production pool and autoscaling
manifests, and the k3s runbook and checklist.
- The check or comment that carries the asynchronous verdict back to the push.
@@ -433,8 +566,7 @@ a push Run, and `ousast plane status` would print the real figure per Task.
### Sizing
The worker is the unit. Every Task's `limits` fit the 1 CPU / 1.5 GiB worker of
`ops/ax/workerpool.yaml.tmpl`; the requests are what the Task declares, the worker is what it
occupies.
`ops/ax/workerpool.yaml.tmpl`. The requests are what the Task declares. The worker is what it occupies.
| Task (`plane/tasks/*.yaml`) | Requests | Limits | Per push |
| --- | --- | --- | --- |
@@ -446,17 +578,18 @@ occupies.
| `roles` | 250m / 384Mi | 1 CPU / 1Gi | not in a push Run (harvest Runs only) |
| `engine` (planned) | from a measured run | above 3 GiB (the host run uses `--memory 3g`) | its own pool, not the default one |
- **Actors per push**: 4 (`facts`, `va`, `vb`, `agree`), 6 with a tie-break (`vc`, `final`);
- **Actors per push**: 4 (`facts`, `va`, `vb`, `agree`), or 6 with a tie-break (`vc`, `final`).
`features` and `remember` add two more when the verdict is to be kept.
- **Critical path**: `facts` then `va` and `vb` side by side, then `agree`, then (sometimes) `vc`
and `final`: three to five Task boots in series, each a golden-snapshot restore plus the model
calls of the verify passes.
- **One actor per worker**: `agree` asks for 256Mi and still holds a 1.5 GiB worker while it runs,
because the worker's capacity, not the Task's request, is the unit. A pool of `W` workers runs
at most `W/2` pushes through their verify phase at once; on this host (`W = 2`) that is one
push at a time, and every further push waits (the reconciler's `--workers` bound, or Substrate's
request parking if the pool is saturated). Autoscaling the pool on `at_capacity` workers is the
upstream answer and is **planned** here.
- **Critical path**: `facts`, then `va` and `vb` side by side, then `agree`, then sometimes `vc`
and `final`. That is three to five Task boots in series. Each boot is a golden-snapshot restore plus
the model calls of the verify passes.
- **One actor per worker**: `agree` asks for 256Mi but still holds a 1.5 GiB worker while it runs. The
worker's capacity, not the Task's request, is the unit.
- A pool of `W` workers runs at most `W/2` pushes through their verify phase at once.
- On this host (`W = 2`) that is one push at a time.
- Every further push waits, on the reconciler's `--workers` bound or on Substrate's request parking
if the pool is saturated.
- Autoscaling the pool on `at_capacity` workers is the upstream answer. It is **planned** here.
## C. The manifests under `plane/`
@@ -472,6 +605,12 @@ occupies.
| `ops/ax/smoke-task.yaml.tmpl`, `smoke-run.yaml.tmpl` | yes | rendered with `RUNNER_IMAGE` from your pin file |
| `ops/ax/up.sh`, `down.sh` | no | kind-specific; they do not apply to another cluster |
The Run and Task contract itself (`plane/runner.py` as PID 1 of the runner image, `AX_TASK_YAML`
and `AX_WORKSPACES_YAML`, `/healthz` and `/readyz`, the start request, the tar delivery as
completion) does not depend on where the cluster is: [The plane on ax](plane.md).
The Run and Task contract does not depend on where the cluster is. That contract covers:
- `plane/runner.py` as PID 1 of the runner image;
- `AX_TASK_YAML` and `AX_WORKSPACES_YAML`;
- `/healthz` and `/readyz`;
- the start request;
- the tar delivery as completion.
See [The plane on ax](plane.md).