docs: plain-language pass -- deployment, plane, memory, RustFS, token economics

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-10-02 21:00:37 +02:00
co-authored by Claude Opus 5.5
parent 50f4e69321
commit 7e23379e96
6 changed files with 916 additions and 591 deletions
+370 -231
View File
@@ -1,17 +1,21 @@
# Deployment # Deployment
How the agentic plane is deployed today, and what moving it to a separate Kubernetes cluster This page covers two things. First, how the agentic plane is deployed today. Second, what moving it to a
would take. State as of 2026-10-02 (v2.0.0). A scan needs none of this: `ousast scan` and separate Kubernetes cluster would take. It describes the state as of 2026-10-02 (v2.0.0).
`ousast pre-push` run from the Python package (and the Docker image for Joern, see
[Engine and pre-push](ops/README.md)). The plane is the maintainers' surface for model work over
whole repositories, driven by `ousast plane`.
Two words are used strictly on this page. **Implemented** means the code on `main` does it and it A scan needs none of this. `ousast scan` and `ousast pre-push` run from the Python package. Joern runs
has run on the maintainer's host. **Planned** means it has not been done; the guidance is derived from the Docker image (see [Engine and pre-push](ops/README.md)). The plane is the maintainers' tool for
from the code and the host's lessons, and the code changes it needs are named with their file paths. model work over whole repositories. `ousast plane` drives it.
This page uses two words strictly:
- **Implemented** means the code on `main` does it, and it has run on the maintainer's host.
- **Planned** means it has not been done. The guidance comes from the code and from what the host taught
us. Each code change it needs is named with its file path.
**No deployment of the plane to a cluster other than the maintainer's single-node kind cluster has **No deployment of the plane to a cluster other than the maintainer's single-node kind cluster has
been made.** Section B is guidance, not a record. been made.** Section B is guidance, not a record. (kind runs a Kubernetes cluster inside Docker
containers on one machine.)
## What runs where today (implemented) ## What runs where today (implemented)
@@ -51,14 +55,21 @@ flowchart LR
| Memory store | `OUSAST_MEMORY`: a file store under `~/ousast-results/plane/memory` by default, or `s3://<bucket>[/<prefix>]` on any reachable S3-compatible server (RustFS is the tested one) | `src/openultrasast/plane/memory.py`, [Memory](memory.md), [RustFS setup](rustfs.md) | | Memory store | `OUSAST_MEMORY`: a file store under `~/ousast-results/plane/memory` by default, or `s3://<bucket>[/<prefix>]` on any reachable S3-compatible server (RustFS is the tested one) | `src/openultrasast/plane/memory.py`, [Memory](memory.md), [RustFS setup](rustfs.md) |
| Run state and artifacts | `$OUSAST_RESULTS/plane/<run>/` on the host (default `~/ousast-results/plane/`) | `reconciler.py` (`run_dir`) | | Run state and artifacts | `$OUSAST_RESULTS/plane/<run>/` on the host (default `~/ousast-results/plane/`) | `reconciler.py` (`run_dir`) |
Only the reconciler talks to the memory store: a sandboxed task cannot read it, so the loop's Terms in the table:
`memory-snapshot` is seeded by the reconciler before the Run ([The plane on ax](plane.md)).
- **ax** is Google's controller that runs agent tasks on Kubernetes.
- **Agent Substrate** is the layer below ax that owns the machines and runs sandboxes.
- **gVisor** is a sandbox runtime that isolates each actor from the node's kernel.
- An **actor** is one running Task inside a sandbox.
Only the reconciler talks to the memory store. A sandboxed task cannot read it. So the reconciler seeds
the loop's `memory-snapshot` before the Run starts ([The plane on ax](plane.md)).
## A. This host: the local setup (implemented) ## A. This host: the local setup (implemented)
What `ops/ax/up.sh` needs on the PATH: `docker`, `go`, `kubectl`, `kind`, `ko` and `ax` (it `ops/ax/up.sh` needs these tools on the PATH: `docker`, `go`, `kubectl`, `kind`, `ko` and `ax`. It
refuses to start when one is missing). Tools live in `~/go/bin` and `~/.local/bin`; sources and refuses to start when one is missing. Tools live in `~/go/bin` and `~/.local/bin`. Sources and rendered
rendered files in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`). files live in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`).
1. **Bring up the cluster and the plane.** 1. **Bring up the cluster and the plane.**
@@ -66,15 +77,20 @@ rendered files in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`).
ops/ax/up.sh ops/ax/up.sh
``` ```
Idempotent: each step is skipped when already done. It creates the kind cluster The script is idempotent: it skips each step that is already done. In order, it:
(`KIND_CLUSTER_NAME`, default `ousast`) with its registry (`KO_DOCKER_REPO`, default
`localhost:5001`), installs Agent Substrate, applies the egress gateway (the agentgateway - creates the kind cluster (`KIND_CLUSTER_NAME`, default `ousast`) with its registry
variant, one prebuilt image) when the install left it out, deploys the ax control plane, applies (`KO_DOCKER_REPO`, default `localhost:5001`);
the receiver Service with the host's kind-gateway address as its endpoint, applies the gVisor - installs Agent Substrate;
worker pool, builds and pushes the runner image and writes its digest pin to - applies the egress gateway if the install left it out (the agentgateway variant, one prebuilt image);
`~/.cache/ousast/ax-src/runner-image`, installs the `ax` CLI from the deployed checkout (a release - deploys the ax control plane;
CLI skews from the server), and runs one smoke Task. It ends by printing the host's memory and the - applies the receiver Service, with the host's kind-gateway address as its endpoint;
kind containers' usage. - applies the gVisor worker pool;
- builds and pushes the runner image, and writes its digest pin to `~/.cache/ousast/ax-src/runner-image`;
- installs the `ax` CLI from the deployed checkout (a release CLI does not match the server version);
- runs one smoke Task.
At the end it prints the host's memory and the kind containers' usage.
2. **Check it.** 2. **Check it.**
@@ -82,9 +98,15 @@ rendered files in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`).
uv run ousast plane doctor uv run ousast plane doctor
``` ```
Four checks: the kind context is reachable, every pod in `ate-system` and in `ax-system` is ready, It runs four checks:
and the registry at `localhost:5001` holds `ousast-runner`. Each failing line names
`ops/ax/up.sh` as the fix. `doctor` is informational: `ousast plane run` does not call it. - the kind context is reachable;
- every pod in `ate-system` is ready;
- every pod in `ax-system` is ready;
- the registry at `localhost:5001` holds `ousast-runner`.
Each failing line names `ops/ax/up.sh` as the fix. `doctor` is informational only.
`ousast plane run` does not call it.
3. **Run the end-to-end smoke Run.** 3. **Run the end-to-end smoke Run.**
@@ -92,23 +114,27 @@ rendered files in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`).
ops/ax/smoke-run.sh ops/ax/smoke-run.sh
``` ```
It renders `ops/ax/smoke-run.yaml.tmpl` with the current runner digest and submits the Run It renders `ops/ax/smoke-run.yaml.tmpl` with the current runner digest. It then submits the Run
`ax-e2e-smoke` through the reconciler: one model-free `repo-facts` Task (`budget: {usd: 0, `ax-e2e-smoke` through the reconciler. The Run has one model-free `repo-facts` Task (`budget: {usd: 0,
calls: 0}`) over a files-only Workspace, resumed, started through the router, its `facts.json` and calls: 0}`) over a files-only Workspace, resumed, started through the router, its `facts.json`
`summary.json` delivered to the host, then `ousast plane status ax-e2e-smoke` prints the attribution and `summary.json` delivered to the host. Then `ousast plane status ax-e2e-smoke`
table. The script calls the repository's `.venv/bin/ousast`, so `uv sync` first. prints the attribution table. The script calls the repository's `.venv/bin/ousast`, so run `uv sync`
first.
4. **Keys and the store.** Put `DEEPSEEK_API_KEY` (chat) and `OPENROUTER_API_KEY` (embeddings) in 4. **Keys and the store.**
`.env` or export them; a Task's Model names the variable (`secretKey.key`), and the key travels
only in that Task's start request. Set `OUSAST_MEMORY=s3://<bucket>` with `S3_ENDPOINT`, - Put `DEEPSEEK_API_KEY` (chat) and `OPENROUTER_API_KEY` (embeddings) in `.env`, or export them.
`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` (optional `S3_REGION`, `S3_BUCKET`) for the S3 store; - A Task's Model names the variable (`secretKey.key`). The key travels only in that Task's start
the bucket must be set up once by an admin as [RustFS setup](rustfs.md) describes, and the store request.
verifies it at every start. - For the S3 store, set `OUSAST_MEMORY=s3://<bucket>` with `S3_ENDPOINT`, `AWS_ACCESS_KEY_ID` and
`AWS_SECRET_ACCESS_KEY`. `S3_REGION` and `S3_BUCKET` are optional.
- An admin must set up the bucket once, as [RustFS setup](rustfs.md) describes. The store verifies
it at every start.
5. **Run a real Run**, as in [Examples](examples.md#7-running-the-agentic-plane): `ousast plane run 5. **Run a real Run**, as in [Examples](examples.md#7-running-the-agentic-plane): `ousast plane run
plane/runs/validation-46.yaml`, then `status` and `remember`. plane/runs/validation-46.yaml`, then `status` and `remember`.
6. **Tear down** with `ops/ax/down.sh`, which deletes the cluster and its registry. 6. **Tear down** with `ops/ax/down.sh`. It deletes the cluster and its registry.
**Footprint** (measured 2026-09-29, idle, two workers; [ax on this host](ops/ax/README.md#measured-footprint-2026-09-29-idle-two-workers)): **Footprint** (measured 2026-09-29, idle, two workers; [ax on this host](ops/ax/README.md#measured-footprint-2026-09-29-idle-two-workers)):
@@ -119,44 +145,62 @@ rendered files in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`).
| node volume (images, containerd) | 4.9 GB on disk | | node volume (images, containerd) | 4.9 GB on disk |
| worker pool | 2 gVisor workers, 1 CPU / 1.5 GiB limit each | | worker pool | 2 gVisor workers, 1 CPU / 1.5 GiB limit each |
Disk is the binding constraint: the Substrate install and the `ko` builds need several GB of Go Disk is the tightest limit, for two reasons:
caches (`go clean -cache -modcache` afterwards), and every Task is its own actor template with a
golden actor (about 24 MB each under `/var/lib/ate/actors` on the node). The failures the host - The Substrate install and the `ko` builds need several GB of Go caches. Run `go clean -cache -modcache`
produced, and what each one taught, are listed in afterwards.
[ax on this host](ops/ax/README.md#what-this-host-taught-each-one-cost-a-failed-live-run). - Every Task is its own actor template with a golden actor (a saved snapshot the actor boots from). Each
takes about 24 MB under `/var/lib/ate/actors` on the node.
[ax on this host](ops/ax/README.md#what-this-host-taught-each-one-cost-a-failed-live-run) lists the
failures the host produced and what each one taught.
## B. A separate Kubernetes cluster (planned) ## B. A separate Kubernetes cluster (planned)
The target: a cluster that runs Agent Substrate with its egress gateway and the ax control plane, The target cluster would do four things:
pulls the runner image from a registry of yours, receives task artifacts without a developer
laptop in the path, and keeps its memory in your S3 bucket. The provider key already travels only - run Agent Substrate with its egress gateway and the ax control plane;
in the start request through `atenet-router`, which works the same through a port-forward to any - pull the runner image from a registry of yours;
cluster. Everything else below is either a step you take or a code change that is not made yet. - receive task artifacts without a developer laptop in the path;
- keep its memory in your S3 bucket.
The provider key already travels only in the start request through `atenet-router`. That works the same
through a port-forward to any cluster. Everything else below is either a step you take or a code change
that is not made yet.
### B.1 Cluster with Agent Substrate, its egress gateway and ax ### B.1 Cluster with Agent Substrate, its egress gateway and ax
- Install Agent Substrate on the cluster following its own instructions. `ops/ax/up.sh` uses - Install Agent Substrate on the cluster with its own instructions. `ops/ax/up.sh` uses Substrate's kind
Substrate's kind helpers (`hack/create-kind-cluster.sh`, `hack/install-ate-kind.sh`), which do helpers (`hack/create-kind-cluster.sh`, `hack/install-ate-kind.sh`). These do not apply to another
not apply to another cluster. cluster.
- Apply the egress gateway. On this host it is the agentgateway variant - Apply the egress gateway. On this host it is the agentgateway variant
(`manifests/ate-install/agentgateway-egress` in the Substrate checkout, applied with (`manifests/ate-install/agentgateway-egress` in the Substrate checkout). It is applied with
`kubectl kustomize --load-restrictor=LoadRestrictionsNone`), which needs no Rust build; the `kubectl kustomize --load-restrictor=LoadRestrictionsNone`. It needs no Rust build; the Envoy variant
Envoy variant does. Without the gateway an actor has no network at all, and the per-task does.
EgressPolicies the reconciler writes (`src/openultrasast/plane/egress.py`) work unchanged - Without the gateway an actor has no network at all.
against the Substrate-installed gateway: deny by default, hostnames only, plain HTTP to the - The reconciler writes one EgressPolicy per task (`src/openultrasast/plane/egress.py`). These work
receiver, TLS passthrough to the Workspaces' Git hosts and the Model's declared hosts. unchanged against the gateway that Substrate installs. The rules are:
- deny by default;
- hostnames only;
- plain HTTP to the receiver;
- TLS passthrough to the Workspaces' Git hosts and the Model's declared hosts.
- Deploy ax (`make deploy AX_IMAGE_REPO=<your registry>` in the ax checkout) **after** B.2. - Deploy ax (`make deploy AX_IMAGE_REPO=<your registry>` in the ax checkout) **after** B.2.
- Apply a WorkerPool (B.6). Substrate installs none by itself ("no free workers"). - Apply a WorkerPool (B.6). Substrate installs none by itself ("no free workers").
### B.2 ax's snapshot bucket: `AX_SNAPSHOTS_BUCKET` ### B.2 ax's snapshot bucket: `AX_SNAPSHOTS_BUCKET`
ax's deploy manifest (`deploy/ax-server.yaml` in the ax checkout) sets `AX_SNAPSHOTS_BUCKET` to Point `AX_SNAPSHOTS_BUCKET` at **your own bucket before deploying ax**. ax's deploy manifest
the ax authors' own bucket (`gs://dberkov-gke-dev3/ate-env/` in the checkout used here). Point it (`deploy/ax-server.yaml` in the ax checkout) sets it to the ax authors' own bucket. In the checkout used
at **your own bucket before deploying ax**. On this host the bucket named there did not exist in here that is `gs://dberkov-gke-dev3/ate-env/`.
the S3 server, deleted golden actors piled up in `DELETING` and filled the disk during a 600-task
harvest (2026-10-01); creating the bucket let deletes complete, and even then a deleted actor's This host shows what goes wrong otherwise (2026-10-01):
directory stays on the node, so long Runs need a janitor that removes directories no live actor
owns. 1. The bucket named there did not exist in the S3 server.
2. Deleted golden actors piled up in `DELETING`.
3. They filled the disk during a 600-task harvest.
Creating the bucket let deletes complete. Even then, a deleted actor's directory stays on the node. So
long Runs need a janitor that removes directories no live actor owns.
### B.3 A registry the workers can pull from, the runner image pinned by digest ### B.3 A registry the workers can pull from, the runner image pinned by digest
@@ -168,83 +212,117 @@ owns.
docker inspect --format '{{range .RepoDigests}}{{println .}}{{end}}' <registry>/ousast-runner:dev docker inspect --format '{{range .RepoDigests}}{{println .}}{{end}}' <registry>/ousast-runner:dev
``` ```
- Substrate rejects tags: every Task's `image` must be the `<registry>/ousast-runner@sha256:...` - Substrate rejects tags. Every Task's `image` must be the `<registry>/ousast-runner@sha256:...`
reference. Write it into a file (on this host, `~/.cache/ousast/ax-src/runner-image`) and pass reference.
`--runner-image FILE` to `ousast plane workspaces` and `ousast plane harvest`, which re-pin the - Write that reference into a file. On this host the file is `~/.cache/ousast/ax-src/runner-image`.
Tasks they generate. The committed templates under `plane/tasks/` and the generated - Pass `--runner-image FILE` to `ousast plane workspaces` and `ousast plane harvest`. They re-pin the
`plane/tasks/validation-46.yaml` carry `localhost:5001/ousast-runner@sha256:...` and need the same Tasks they generate.
re-pin (regenerate, or replace the reference) before they run elsewhere. - The committed templates under `plane/tasks/` and the generated `plane/tasks/validation-46.yaml` carry
- Workers must be able to pull from the registry (credentials, if any, are the cluster's concern; `localhost:5001/ousast-runner@sha256:...`. They need the same re-pin before they run elsewhere.
ax's default runner image on `gcr.io` needs credentials and is not used here). Regenerate them, or replace the reference.
- Workers must be able to pull from the registry. Credentials, if any, are the cluster's concern. ax's
default runner image on `gcr.io` needs credentials and is not used here.
- **Not yet configurable:** `ousast plane doctor` checks the registry at `localhost:5001` by name - **Not yet configurable:** `ousast plane doctor` checks the registry at `localhost:5001` by name
(`src/openultrasast/plane/doctor.py:45`), so its fourth check fails on another cluster even when (`src/openultrasast/plane/doctor.py:45`). So its fourth check fails on another cluster, even when the
the image is in place. `doctor` is informational, `plane run` does not depend on it. image is in place. `doctor` is informational; `plane run` does not depend on it.
### B.4 One configurable kube context ### B.4 One configurable kube context
The reconciler, the egress client and the router port-forward all use the context The reconciler, the egress client and the router port-forward all use one kube context. Its name is
`kind-$KIND_CLUSTER_NAME` (`kind-ousast` by default). **Not yet configurable**: the prefix `kind-` `kind-$KIND_CLUSTER_NAME` (`kind-ousast` by default).
is hard-coded in `src/openultrasast/plane/doctor.py:39`, `src/openultrasast/plane/egress.py:115`
and `:148`, and `src/openultrasast/plane/router.py:145`; `ops/ax/up.sh:24` builds the same name. **Not yet configurable**: the prefix `kind-` is hard-coded in these places:
The planned change is one variable (for example `OUSAST_KUBE_CONTEXT`) read in those three
modules. Until then the only way to drive another cluster from the unchanged code is to give its - `src/openultrasast/plane/doctor.py:39`;
kubeconfig context a name of that form (`kubectl config rename-context <ctx> kind-<name>` and - `src/openultrasast/plane/egress.py:115` and `:148`;
`KIND_CLUSTER_NAME=<name>`), which is a workaround, not a supported configuration. The egress - `src/openultrasast/plane/router.py:145`;
client also needs `kubectl-ate` (or the executable `OUSAST_KUBECTL_ATE` names) able to use that - `ops/ax/up.sh:24` builds the same name.
context.
The planned change is one variable read in those three modules, for example `OUSAST_KUBE_CONTEXT`.
Until then, there is one workaround to drive another cluster with the unchanged code. Rename its
kubeconfig context to that form: `kubectl config rename-context <ctx> kind-<name>` and
`KIND_CLUSTER_NAME=<name>`. This is a workaround, not a supported configuration.
The egress client also needs `kubectl-ate` (or the executable `OUSAST_KUBECTL_ATE` names). It must be
able to use that context.
### B.5 The receiver: in-cluster instead of on the host ### B.5 The receiver: in-cluster instead of on the host
Today the receiver is a thread inside the reconciler process (`egress.py`, `Receiver`), and the Today the receiver is a thread inside the reconciler process (`egress.py`, `Receiver`). The reconciler
reconciler learns of a task's completion by watching that object's `delivered` set in memory learns that a task finished by watching that object's `delivered` set in memory (`reconciler.py`).
(`reconciler.py`). Actors reach it through the Service `ousast-receiver.ax-system` whose Actors reach it through the Service `ousast-receiver.ax-system`. That Service's EndpointSlice points at
EndpointSlice points at the host. Three paths, in order of effort: the host.
1. **Interim, works with the unchanged code:** keep the host receiver and render There are three paths, from least to most effort:
`ops/ax/receiver-service.yaml.tmpl` with an endpoint address the egress gateway pod can reach
(`KIND_GATEWAY` is just an IPv4 address in the template, `OUSAST_ARTIFACT_PORT` the port). The 1. **Interim, works with the unchanged code:** keep the host receiver.
reconciler's host must be reachable from the cluster (a VPN or a routed address), and the Render `ops/ax/receiver-service.yaml.tmpl` with an endpoint address the egress gateway pod can reach.
gateway dials the address the actor connected to, so the Service's ClusterIP must still map to it. In the template, `KIND_GATEWAY` is just an IPv4 address and `OUSAST_ARTIFACT_PORT` is the port.
2. **Planned: the receiver as an in-cluster Deployment** behind the same Service name, with the The cluster must be able to reach the reconciler's host, through a VPN or a routed address.
reconciler reading deliveries and serving inputs from it instead of from its own thread. This The gateway dials the address the actor connected to. So the Service's ClusterIP must still map to it.
needs code: the receiver and its `delivered`/`inputs` state are in-process today, and the 2. **Planned: the receiver as an in-cluster Deployment** behind the same Service name.
runner's `GET /inputs/<producer>/<artifact>` is served by that same process. The reconciler would read deliveries from it and serve inputs from it, instead of from its own thread.
3. **Follow-on: presigned-URL delivery to the S3 store.** The runner would `PUT` its output tar to a This needs code. The receiver and its `delivered`/`inputs` state are in-process today.
presigned URL on the memory store's bucket and fetch inputs the same way, and the reconciler The same process serves the runner's `GET /inputs/<producer>/<artifact>`.
would read from the bucket. Nothing of this exists; the EgressPolicy would then have to allow 3. **Follow-on: presigned-URL delivery to the S3 store.** (A presigned URL grants one upload or download
the store's hostname for the task. without credentials.)
The runner would `PUT` its output tar to a presigned URL on the memory store's bucket.
It would fetch inputs the same way, and the reconciler would read from the bucket.
None of this exists. The EgressPolicy would then have to allow the store's hostname for the task.
### B.6 Worker pool sizing ### B.6 Worker pool sizing
`ops/ax/workerpool.yaml.tmpl` declares two warm gVisor workers of 1 CPU / 1.5 GiB (requests 250m / `ops/ax/workerpool.yaml.tmpl` declares two warm gVisor workers for the 7 GB host. Each has 1 CPU /
1.5 GiB) for the 7 GB host, selected onto nodes by `ate.dev/substrate-version`. An actor occupies a 1.5 GiB (requests 250m / 1.5 GiB). They are placed on nodes by `ate.dev/substrate-version`.
whole worker, so the pool size is the number of tasks that can run at once; set `--workers` on
`ousast plane run` to match. Size the workers above the largest Task's limits (`verify`: requests An actor occupies a whole worker. So the pool size is the number of tasks that can run at once. Set
250m / 384Mi, limits 1 CPU / 1 GiB; `roles` and the others are in `plane/tasks/`), and keep the `--workers` on `ousast plane run` to match.
golden-actor storage on the node in mind (B.2).
Size the workers above the largest Task's limits. For `verify` these are requests 250m / 384Mi and
limits 1 CPU / 1 GiB. `roles` and the others are in `plane/tasks/`. Also keep the golden-actor storage
on the node in mind (B.2).
### B.7 The S3 memory store reachable from where the reconciler runs ### B.7 The S3 memory store reachable from where the reconciler runs
The `s3://` store already works against any reachable S3-compatible server with versioning, The `s3://` store already works against any reachable S3-compatible server. The server must support:
lifecycle rules, object tags and S3 Select (RustFS is the tested one; the bucket setup and the
agent policy are on [RustFS setup](rustfs.md)). Only the reconciler reads and writes it, so today - versioning;
it must be reachable from the operator's host; if the reconciler moves into the cluster, from its - lifecycle rules;
pod. No Task talks to the store directly; the day one does (B.5, path 3), that Task's EgressPolicy - object tags;
must allow the store's hostname (the policies allow hostnames only, never addresses), the way a - S3 Select (a server-side query over JSON objects).
Model declares its hosts today through `openultrasast.io/egress-hosts`.
RustFS is the tested server. The bucket setup and the agent policy are on [RustFS setup](rustfs.md).
Only the reconciler reads and writes the store. So today the operator's host must reach it. If the
reconciler moves into the cluster, its pod must reach it.
No Task talks to the store directly. If one ever does (B.5, path 3), its EgressPolicy must allow the
store's hostname. The policies allow hostnames only, never addresses. A Model already declares its hosts
this way today, through `openultrasast.io/egress-hosts`.
### B.8 Secrets in Kubernetes Secrets, injected into the reconciler's environment ### B.8 Secrets in Kubernetes Secrets, injected into the reconciler's environment
The reconciler reads the variable a Model's `secretKey.key` names (`DEEPSEEK_API_KEY`, The reconciler reads these variables from its own environment or `.env`:
`OPENROUTER_API_KEY`) and the store's `S3_ENDPOINT`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`
(optional `AWS_SESSION_TOKEN`, `S3_REGION`, `S3_BUCKET`) from its own environment or `.env`. When - the variable a Model's `secretKey.key` names (`DEEPSEEK_API_KEY`, `OPENROUTER_API_KEY`);
the reconciler runs in the cluster, keep them in Kubernetes Secrets and inject them into **the - the store's `S3_ENDPOINT`, `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`;
reconciler's** pod environment (`envFrom` with `secretRef`, or `valueFrom.secretKeyRef` per - optionally `AWS_SESSION_TOKEN`, `S3_REGION` and `S3_BUCKET`.
variable). Never put a value into a manifest: the Model manifests name a variable, the rendered
Task carries no key, and the key reaches an actor only in the start request, where the runner keeps When the reconciler runs in the cluster, keep these in Kubernetes Secrets. Inject them into **the
it in memory and redacts it from echoed stderr. The `secretKey.name` values in `plane/models/` reconciler's** pod environment. Use `envFrom` with `secretRef`, or `valueFrom.secretKeyRef` per variable.
(`deepseek`, `openrouter`) are the names such Secrets would carry. Running the reconciler
in-cluster is itself planned: today it is a host process, and its state directory Never put a value into a manifest. The key stays out of every manifest:
- the Model manifests name a variable;
- the rendered Task carries no key;
- the key reaches an actor only in the start request;
- the runner keeps it in memory and redacts it from echoed stderr.
The `secretKey.name` values in `plane/models/` (`deepseek`, `openrouter`) are the names such Secrets
would carry.
Running the reconciler in-cluster is itself planned. Today it is a host process. Its state directory
(`OUSAST_RESULTS`) would need a persistent volume. (`OUSAST_RESULTS`) would need a persistent volume.
### What is implemented and what is planned ### What is implemented and what is planned
@@ -266,12 +344,20 @@ in-cluster is itself planned: today it is a host process, and its state director
## Production topology: ax on a Kubernetes cluster ## Production topology: ax on a Kubernetes cluster
The maintainer's questions, answered from the code and the upstream checkouts under This section answers the maintainer's questions from the code and the upstream checkouts under
`~/.cache/ousast/ax-src/`: how is it hosted in ax, is there one container, how does ax relate to `~/.cache/ousast/ax-src/`:
Kubernetes, and what scaling the pre-push hook would take. The decisions of 2026-10-02 set the
target: **two profiles of one code base**, `kind` (this laptop, kept as a first-class local - How is it hosted in ax?
execution mode, where every change is proven first) and `k3s` (production). Everything marked - Is there one container?
**planned** is not built; the rest is what runs on this host today. - How does ax relate to Kubernetes?
- What would scaling the pre-push hook take?
The decisions of 2026-10-02 set the target: **two profiles of one code base**.
- `kind` is this laptop. It stays a first-class local execution mode. Every change is proven here first.
- `k3s` is production. (k3s is a small Kubernetes distribution.)
Everything marked **planned** is not built. The rest is what runs on this host today.
```mermaid ```mermaid
flowchart LR flowchart LR
@@ -308,61 +394,92 @@ flowchart LR
cli -- "polls: status, attribution" --> s3 cli -- "polls: status, attribution" --> s3
``` ```
No inbound path exists in this picture: the CLI reaches ax-server and the router from outside, the This picture has no inbound path:
tasks push their artifacts to the store through the egress gateway, and the CLI reads the store.
Today's EndpointSlice back to a laptop (`ops/ax/receiver-service.yaml.tmpl`) is the kind profile's - the CLI reaches ax-server and the router from outside;
device and does not appear in production. - the tasks push their artifacts to the store through the egress gateway;
- the CLI reads the store.
Today's EndpointSlice back to a laptop (`ops/ax/receiver-service.yaml.tmpl`) belongs to the kind
profile only. It does not appear in production.
### ax, Kubernetes and Agent Substrate in three sentences ### ax, Kubernetes and Agent Substrate in three sentences
**ax is a Kubernetes application**: a Deployment `ax-server` with a Redis beside it in the **ax is a Kubernetes application.**
namespace `ax-system` (`ax/deploy/ax-server.yaml` in the ax checkout; its controller runs with
`--template=default-template --template-atespace=ax-system`), and the `ax` CLI reaches it through a
`kubectl port-forward` to `svc/ax-server` (`ax/internal/tunnel/tunnel.go`), so `ousast plane run`
needs a kube context or an ingress, not a public endpoint. **ax does not run containers itself**:
it turns every Task into an ActorTemplate on Agent Substrate (`ax/internal/substrate/client.go`,
`CreateActorTemplate`), built from the Task's image and environment, and the image must be pinned
by digest (Substrate's `SandboxConfig` validation rejects tags: "All images must include a
digest"). **Agent Substrate is the layer that owns the machines**: in the namespace `ate-system`
it runs `ate-api-server`, `ate-controller`, the `atelet` DaemonSet, `atenet-router`, the
`atenet-egress` gateway, a Postgres and a RustFS (`substrate/manifests/ate-install/`), and three
CRDs under `ate.dev/v1alpha1`, `WorkerPool`, `SandboxConfig` and `CSIDriverConfig`; a
`WorkerPool` keeps warm worker pods (`ko://.../cmd/ateom-gvisor`), and an actor is one gVisor
sandbox inside one of those workers, booted from the golden snapshot that
[The plane on ax](plane.md) describes, suspended and resumed by the controller. Names are limited
to 63 bytes (`MaxNameLength` in `ax/pkg/apis/v1alpha1/types.go`), which is why the generated Task
names are short.
**What k3s changes** (planned; nothing has been applied to a k3s cluster): k3s runs containerd - It is a Deployment `ax-server` with a Redis beside it, in the namespace `ax-system`
with no Docker socket, so the host-Docker path of `ousast plane alerts-engine` does not exist (`ax/deploy/ax-server.yaml` in the ax checkout).
there and the engine must become a task (below); Substrate's gVisor workers need `runsc` and a - Its controller runs with `--template=default-template --template-atespace=ax-system`.
`RuntimeClass` on the nodes they are scheduled to; k3s ships Traefik (the ingress the CLI can use - The `ax` CLI reaches it through a `kubectl port-forward` to `svc/ax-server`
instead of a kubeconfig) and local-path storage. Images are built by GitHub Actions from tagged (`ax/internal/tunnel/tunnel.go`).
commits, published to GitHub Container Registry and digest-pinned in the manifests; a private - So `ousast plane run` needs a kube context or an ingress, not a public endpoint.
package needs an `imagePullSecret` on the worker ServiceAccount. Substrate's and ax's own images
are built with `ko` from their checkouts and pushed to the same registry. The kind profile keeps **ax does not run containers itself.**
`localhost:5001` and `ops/ax/up.sh`.
- It turns every Task into an ActorTemplate on Agent Substrate (`ax/internal/substrate/client.go`,
`CreateActorTemplate`).
- The ActorTemplate is built from the Task's image and environment.
- The image must be pinned by digest. Substrate's `SandboxConfig` validation rejects tags: "All images
must include a digest".
**Agent Substrate is the layer that owns the machines.**
- In the namespace `ate-system` it runs `ate-api-server`, `ate-controller`, the `atelet` DaemonSet,
`atenet-router`, the `atenet-egress` gateway, a Postgres and a RustFS
(`substrate/manifests/ate-install/`).
- It defines three CRDs (custom Kubernetes resource types) under `ate.dev/v1alpha1`: `WorkerPool`,
`SandboxConfig` and `CSIDriverConfig`.
- A `WorkerPool` keeps warm worker pods (`ko://.../cmd/ateom-gvisor`).
- An actor is one gVisor sandbox inside one of those workers.
- The actor boots from the golden snapshot that [The plane on ax](plane.md) describes. The controller
suspends and resumes it.
- Names are limited to 63 bytes (`MaxNameLength` in `ax/pkg/apis/v1alpha1/types.go`). That is why the
generated Task names are short.
**What k3s changes** (planned; nothing has been applied to a k3s cluster):
- k3s runs containerd with no Docker socket. So the host-Docker path of `ousast plane alerts-engine`
does not exist there. The engine must become a task (below).
- Substrate's gVisor workers need `runsc` and a `RuntimeClass` on the nodes they run on.
- k3s ships Traefik and local-path storage. Traefik is an ingress the CLI can use instead of a kubeconfig.
- GitHub Actions builds the images from tagged commits. They are published to GitHub Container Registry
and digest-pinned in the manifests.
- A private package needs an `imagePullSecret` on the worker ServiceAccount.
- Substrate's and ax's own images are built with `ko` from their checkouts. They are pushed to the same
registry.
- The kind profile keeps `localhost:5001` and `ops/ax/up.sh`.
### One image, and what concurrency equals ### One image, and what concurrency equals
There is **one container image** for every Task: `plane/Dockerfile.runner` (`python:3.12-slim` There is **one container image** for every Task: `plane/Dockerfile.runner`. It is `python:3.12-slim`
plus this package, PID 1 `/usr/local/bin/ax-task-runner`, which is plus this package. PID 1 is `/usr/local/bin/ax-task-runner`, which is `openultrasast.plane.runner`.
`openultrasast.plane.runner`). Each Task manifest under `plane/tasks/` names the same image by
digest (the pin is kept in `~/.cache/ousast/ax-src/runner-image`) and selects its work with Each Task manifest under `plane/tasks/` names the same image by digest. The pin is kept in
`spec.command`: the runner starts `python -m openultrasast.plane.tasks.<command[0]>` `~/.cache/ousast/ax-src/runner-image`. The manifest picks its work with `spec.command`. The runner
(`runner.py`, `task_module`). `repo-facts`, `verify`, `agree`, `features`, `remember`, `alerts`, starts `python -m openultrasast.plane.tasks.<command[0]>` (`runner.py`, `task_module`).
`loop` and `roles` are modules in that image, not images of their own; the engine would be the
second image (planned). So the production answer to "how many containers" is: one image, one `repo-facts`, `verify`, `agree`, `features`, `remember`, `alerts`, `loop` and `roles` are modules in
actor per Task, and **an actor occupies a whole worker** (its limits are the ActorTemplate's, the that image. They are not images of their own. The engine would be the second image (planned).
worker holds one actor at a time). Concurrency is therefore the number of workers in the
`WorkerPool`, nothing else: `ops/ax/workerpool.yaml.tmpl` declares `replicas: 2` of 1 CPU / So the production answer to "how many containers" is:
1.5 GiB, so at most two Tasks run at once on this host, and `ousast plane run --workers` must not
exceed the pool. Substrate ships an autoscaled pool - one image;
(`substrate/demos/autoscaled-workerpool/README.md`: an HPA on the external metric - one actor per Task;
`ate_workerpool_workers{state=at_capacity}` through prometheus-adapter, writing - **an actor occupies a whole worker**. Its limits are the ActorTemplate's, and the worker holds one
`WorkerPool.spec.replicas`; its README says the demo is kind-only today) and request parking actor at a time.
(`substrate/demos/parking/`: the router holds a resume until a worker frees up). Neither is
applied by `ops/ax/up.sh`; the production profile's pool manifests and autoscaling rule are Concurrency is therefore the number of workers in the `WorkerPool`, nothing else.
`ops/ax/workerpool.yaml.tmpl` declares `replicas: 2` of 1 CPU / 1.5 GiB. So at most two Tasks run at
once on this host. `ousast plane run --workers` must not exceed the pool.
Substrate ships two scaling aids:
- **An autoscaled pool** (`substrate/demos/autoscaled-workerpool/README.md`). An HPA (Horizontal Pod
Autoscaler) reads the external metric `ate_workerpool_workers{state=at_capacity}` through
prometheus-adapter. It writes `WorkerPool.spec.replicas`. Its README says the demo is kind-only today.
- **Request parking** (`substrate/demos/parking/`). The router holds a resume until a worker frees up.
`ops/ax/up.sh` applies neither. The production profile's pool manifests and autoscaling rule are
**planned**. **planned**.
### The three components that are bound to the host today ### The three components that are bound to the host today
@@ -373,59 +490,75 @@ applied by `ops/ax/up.sh`; the production profile's pool manifests and autoscali
| Artifact receiver (`plane/egress.py`, the `Receiver` thread inside the reconciler) | reached through the Service `ousast-receiver.ax-system` whose EndpointSlice points at the kind gateway address on the host (`ops/ax/receiver-service.yaml.tmpl`) | **no receiver in remote mode.** Tasks deliver their artifacts and fetch their inputs through presigned URLs on the S3 memory store (B.5, path 3; the store's hostname enters each Task's EgressPolicy, B.7), and the CLI reads completion from the store. The kind profile keeps the receiver thread; an in-cluster receiver Deployment (B.5, path 2) is not the chosen direction | | Artifact receiver (`plane/egress.py`, the `Receiver` thread inside the reconciler) | reached through the Service `ousast-receiver.ax-system` whose EndpointSlice points at the kind gateway address on the host (`ops/ax/receiver-service.yaml.tmpl`) | **no receiver in remote mode.** Tasks deliver their artifacts and fetch their inputs through presigned URLs on the S3 memory store (B.5, path 3; the store's hostname enters each Task's EgressPolicy, B.7), and the CLI reads completion from the store. The kind profile keeps the receiver thread; an in-cluster receiver Deployment (B.5, path 2) is not the chosen direction |
| Joern engine (`plane/engine_alerts.py`, the host's `openultrasast:dev` container, run with `--memory 3g`) | `ousast plane alerts-engine` runs it on the host, one container at a time, and marks the Run's `alerts` tasks done | **an `engine` task on a second, larger `WorkerPool`** (the engine does not fit a 1.5 GiB worker, and k3s has no Docker socket), from an engine image digest-pinned in GHCR, emitting the same `alerts.jsonl` rows and coverage; `alerts-engine` on the host stays as the development path | | Joern engine (`plane/engine_alerts.py`, the host's `openultrasast:dev` container, run with `--memory 3g`) | `ousast plane alerts-engine` runs it on the host, one container at a time, and marks the Run's `alerts` tasks done | **an `engine` task on a second, larger `WorkerPool`** (the engine does not fit a 1.5 GiB worker, and k3s has no Docker socket), from an engine image digest-pinned in GHCR, emitting the same `alerts.jsonl` rows and coverage; `alerts-engine` on the host stays as the development path |
The S3 memory store (RustFS tested) is already external; in remote mode it is also the artifact The S3 memory store (RustFS tested) is already external. In remote mode it is also the artifact path.
path, so it must be reachable from the CLI's host and, through the egress gateway, from the So the CLI's host must reach it, and the workers must reach it through the egress gateway.
workers.
### Scaling the pre-push hook in production ### Scaling the pre-push hook in production
**Today** `ousast pre-push` is a local, deterministic check: it materialises base and head, **Today** `ousast pre-push` is a local, deterministic check. It does four things:
discovers the changed regions, runs the evidence ranker and arbiter on head and compares with base,
all within `--deadline` (30 s by default) and in `--mode advisory` unless blocking is opted into 1. materialises base and head;
([Pre-push: the delta check](scanning.md#pre-push-the-delta-check-experimental), 2. finds the changed regions;
`src/openultrasast/push/`, [Engine and pre-push](ops/README.md#experimental-pre-push-integration)). 3. runs the evidence ranker and arbiter on head;
**The plane is not in that path**: no hook submits a Run, no model is called without an explicit 4. compares the result with base.
`--model-config`, and nothing leaves the developer's machine.
All of this runs within `--deadline` (30 s by default). It runs in `--mode advisory` unless you opt
into blocking. See [Pre-push: the delta check](scanning.md#pre-push-the-delta-check-experimental),
`src/openultrasast/push/` and [Engine and pre-push](ops/README.md#experimental-pre-push-integration).
**The plane is not in that path.**
- No hook submits a Run.
- No model is called without an explicit `--model-config`.
- Nothing leaves the developer's machine.
**The production shape** (planned, none of it built) splits the work by latency: **The production shape** (planned, none of it built) splits the work by latency:
1. **Blocking, inside the hook's deadline**: what runs today, the quick rules and the delta 1. **Blocking, inside the hook's deadline**: what runs today, the quick rules and the delta engine on
engine on the changed regions. It stays local and deterministic, so the push is never held for the changed regions. It stays local and deterministic. So the push never waits for a model or a
a model or a cluster. cluster.
2. **Asynchronous, on the plane**: the hook or the CI job calls the CLI in remote mode with a 2. **Asynchronous, on the plane**: the hook or the CI job calls the CLI in remote mode with a *delta
*delta Run* whose candidates are the functions the push changed; the Run is the per-case chain Run*.
of [The plane on ax](plane.md#per-case) (`facts`, `verify` a and b, `agree`, `verify` c on the - The Run's candidates are the functions the push changed.
disputed only, `final`). The caller either waits, with its own deadline, for the CLI to print - The Run is the per-case chain of [The plane on ax](plane.md#per-case): `facts`, `verify` a and b,
the status and attribution table, or returns at once and a later `ousast plane status <run>` `agree`, `verify` c on the disputed only, `final`.
reports from the store; a commit status check or a review comment would be built on that. The - The caller can wait, with its own deadline, for the CLI to print the status and attribution table.
worker pool is sized to the push rate or autoscaled on `at_capacity` workers. An unattended - Or the caller returns at once, and a later `ousast plane status <run>` reports from the store.
in-cluster reconciler that accepts submissions without a CLI session is the later option named - A commit status check or a review comment would be built on that.
above. - The worker pool is sized to the push rate, or autoscaled on `at_capacity` workers.
- An unattended in-cluster reconciler that accepts submissions without a CLI session is the later
option named above.
**Cost per push, estimated from measured rates.** Two measured per-candidate rates exist for one **Cost per push, estimated from measured rates.** Two measured per-candidate rates exist for one model
model verdict: `$0.003215` (`spend.cost_per_candidate_evaluate_usd` in verdict:
`benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`) and `$0.0021`
(exp-002 arm A, `overall.usd_per_candidate.A` in - `$0.003215` (`spend.cost_per_candidate_evaluate_usd` in
`benchmarks/measurements/2026-10-02-exp-002/record.json`). A push that changes five functions needs `benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`);
two verify passes per candidate: 5 x 2 x $0.0021 = $0.021, or 5 x 2 x $0.003215 = $0.032 at the - `$0.0021` (exp-002 arm A, `overall.usd_per_candidate.A` in
higher rate, plus a third pass on the disputed candidates only. That is where "about two cents per `benchmarks/measurements/2026-10-02-exp-002/record.json`).
push" comes from; it is arithmetic over the decision engine's measured rates, not a measurement of
a push Run, and `ousast plane status` would print the real figure per Task. A push that changes five functions needs two verify passes per candidate. That gives 5 x 2 x $0.0021 =
$0.021, or 5 x 2 x $0.003215 = $0.032 at the higher rate. A third pass runs on the disputed candidates
only. That is where "about two cents per push" comes from. It is arithmetic over the decision engine's
measured rates, not a measurement of a push Run. `ousast plane status` would print the real figure per
Task.
**What is not built** (each is a named gap, not a configuration): **What is not built** (each is a named gap, not a configuration):
- The delta-Run command. `ousast plane scan <repo> --base <commit> --head <commit>` **does not - The delta-Run command. `ousast plane scan <repo> --base <commit> --head <commit>` **does not
exist**; `ousast plane` has `run`, `status`, `doctor`, `remember`, `memory-normalise`, exist**. `ousast plane` has `run`, `status`, `doctor`, `remember`, `memory-normalise`,
`workspaces`, `harvest` and `alerts-engine` (`src/openultrasast/cli.py`). `workspaces`, `harvest` and `alerts-engine` (`src/openultrasast/cli.py`).
- A candidates task for an arbitrary repository. `repo-facts` takes its candidates from a - A candidates task for an arbitrary repository. `repo-facts` takes its candidates from a
population's validation set; a step that derives them from a diff is not written. population's validation set. No step derives them from a diff yet.
- Workspace generation from a URL and a commit outside a population. `plane/generate.py` is - Workspace generation from a URL and a commit outside a population. `plane/generate.py` is
population-driven (`read_cases` reads a validation set and a population TOML), so a Run for a population-driven: `read_cases` reads a validation set and a population TOML. So a Run for a
repository that is not in a population cannot be generated today. repository that is not in a population cannot be generated today.
- The remote execution mode itself: the `local`/`remote` switch, configurable cluster context, - The remote execution mode itself:
ax-server and router addresses (the `kind-` prefix is hard-coded, B.4), presigned delivery from - the `local`/`remote` switch;
the runner to the store, and the CLI reading completion and `summary.json` from the store - a configurable cluster context and ax-server and router addresses (the `kind-` prefix is
instead of its receiver thread. hard-coded, B.4);
- presigned delivery from the runner to the store;
- the CLI reading completion and `summary.json` from the store instead of its receiver thread.
- The engine task and its WorkerPool, the GHCR build, the production pool and autoscaling - The engine task and its WorkerPool, the GHCR build, the production pool and autoscaling
manifests, and the k3s runbook and checklist. manifests, and the k3s runbook and checklist.
- The check or comment that carries the asynchronous verdict back to the push. - The check or comment that carries the asynchronous verdict back to the push.
@@ -433,8 +566,7 @@ a push Run, and `ousast plane status` would print the real figure per Task.
### Sizing ### Sizing
The worker is the unit. Every Task's `limits` fit the 1 CPU / 1.5 GiB worker of The worker is the unit. Every Task's `limits` fit the 1 CPU / 1.5 GiB worker of
`ops/ax/workerpool.yaml.tmpl`; the requests are what the Task declares, the worker is what it `ops/ax/workerpool.yaml.tmpl`. The requests are what the Task declares. The worker is what it occupies.
occupies.
| Task (`plane/tasks/*.yaml`) | Requests | Limits | Per push | | Task (`plane/tasks/*.yaml`) | Requests | Limits | Per push |
| --- | --- | --- | --- | | --- | --- | --- | --- |
@@ -446,17 +578,18 @@ occupies.
| `roles` | 250m / 384Mi | 1 CPU / 1Gi | not in a push Run (harvest Runs only) | | `roles` | 250m / 384Mi | 1 CPU / 1Gi | not in a push Run (harvest Runs only) |
| `engine` (planned) | from a measured run | above 3 GiB (the host run uses `--memory 3g`) | its own pool, not the default one | | `engine` (planned) | from a measured run | above 3 GiB (the host run uses `--memory 3g`) | its own pool, not the default one |
- **Actors per push**: 4 (`facts`, `va`, `vb`, `agree`), 6 with a tie-break (`vc`, `final`); - **Actors per push**: 4 (`facts`, `va`, `vb`, `agree`), or 6 with a tie-break (`vc`, `final`).
`features` and `remember` add two more when the verdict is to be kept. `features` and `remember` add two more when the verdict is to be kept.
- **Critical path**: `facts` then `va` and `vb` side by side, then `agree`, then (sometimes) `vc` - **Critical path**: `facts`, then `va` and `vb` side by side, then `agree`, then sometimes `vc`
and `final`: three to five Task boots in series, each a golden-snapshot restore plus the model and `final`. That is three to five Task boots in series. Each boot is a golden-snapshot restore plus
calls of the verify passes. the model calls of the verify passes.
- **One actor per worker**: `agree` asks for 256Mi and still holds a 1.5 GiB worker while it runs, - **One actor per worker**: `agree` asks for 256Mi but still holds a 1.5 GiB worker while it runs. The
because the worker's capacity, not the Task's request, is the unit. A pool of `W` workers runs worker's capacity, not the Task's request, is the unit.
at most `W/2` pushes through their verify phase at once; on this host (`W = 2`) that is one - A pool of `W` workers runs at most `W/2` pushes through their verify phase at once.
push at a time, and every further push waits (the reconciler's `--workers` bound, or Substrate's - On this host (`W = 2`) that is one push at a time.
request parking if the pool is saturated). Autoscaling the pool on `at_capacity` workers is the - Every further push waits, on the reconciler's `--workers` bound or on Substrate's request parking
upstream answer and is **planned** here. if the pool is saturated.
- Autoscaling the pool on `at_capacity` workers is the upstream answer. It is **planned** here.
## C. The manifests under `plane/` ## C. The manifests under `plane/`
@@ -472,6 +605,12 @@ occupies.
| `ops/ax/smoke-task.yaml.tmpl`, `smoke-run.yaml.tmpl` | yes | rendered with `RUNNER_IMAGE` from your pin file | | `ops/ax/smoke-task.yaml.tmpl`, `smoke-run.yaml.tmpl` | yes | rendered with `RUNNER_IMAGE` from your pin file |
| `ops/ax/up.sh`, `down.sh` | no | kind-specific; they do not apply to another cluster | | `ops/ax/up.sh`, `down.sh` | no | kind-specific; they do not apply to another cluster |
The Run and Task contract itself (`plane/runner.py` as PID 1 of the runner image, `AX_TASK_YAML` The Run and Task contract does not depend on where the cluster is. That contract covers:
and `AX_WORKSPACES_YAML`, `/healthz` and `/readyz`, the start request, the tar delivery as
completion) does not depend on where the cluster is: [The plane on ax](plane.md). - `plane/runner.py` as PID 1 of the runner image;
- `AX_TASK_YAML` and `AX_WORKSPACES_YAML`;
- `/healthz` and `/readyz`;
- the start request;
- the tar delivery as completion.
See [The plane on ax](plane.md).
+139 -85
View File
@@ -1,11 +1,12 @@
# Memory and detection # Memory and detection
How what the system remembers is meant to turn into better decisions on code it has not seen. This page shows how the system's memory is meant to improve decisions on code it has not seen. It
This is the design of the learned decision engine (`src/openultrasast/learn/`), which is **in describes the design of the learned decision engine (`src/openultrasast/learn/`).
development and not adopted**: no scan, pre-push check or report uses it today. Every user-facing
decision is still the deterministic one described in [Scanning](scanning.md) and The engine is **in development and not adopted**. No scan, pre-push check or report uses it today.
Every user-facing decision is still the deterministic one described in [Scanning](scanning.md) and
[Architecture](architecture.md). The engine's commands and status are on [Architecture](architecture.md). The engine's commands and status are on
[Decision engine](decision-engine.md); the store itself is on [Memory](memory.md). [Decision engine](decision-engine.md). The store itself is on [Memory](memory.md).
## From instruments to a decision ## From instruments to a decision
@@ -25,89 +26,128 @@ flowchart TD
op -- "otherwise" --> adv["ADVISORY"] op -- "otherwise" --> adv["ADVISORY"]
``` ```
The intended place for these decisions is changed code: `learn/features.py` builds feature records These decisions are meant for changed code. `learn/features.py` builds feature records in two ways:
for a whole-repository scan or for a base..head delta (`build_for_scan` with `Delta`), the same
delta `ousast pre-push` checks. Integration into scan and pre-push is still ahead. - for a whole-repository scan;
- for a base..head delta (`build_for_scan` with `Delta`). This is the same delta `ousast pre-push` checks.
Integration into scan and pre-push is still ahead.
## 1. Instruments produce signals, not verdicts ## 1. Instruments produce signals, not verdicts
Each instrument contributes one block to a candidate's feature record (`learn/schema.py`, Each instrument contributes one block to a candidate's feature record (`learn/schema.py`,
`INSTRUMENTS`): `language`, `quick` (quick rules), `engine` (the Joern model layer), `facts` (repo `INSTRUMENTS`). The blocks are:
facts, such as callers and function span), `source`, `entry_points`, `roles` (model source, sink
and sanitizer roles), `delta`, `verify` and `agree` (the plane's verify passes and their
agreement), and `model_sinks` (the archived stage-1 classifier, `benchmarks/archive/model_sinks.py`, whose
logic now lives in `plane/tasks/roles.py`). A candidate is `(path, function, family)` of one repository and pin
where at least one instrument emitted a signal of that family (`learn/features.py`).
An instrument that did not run, had no coverage or failed records that state (`none`, `failed`, - `language`;
`not_applicable`), never a zero, so "silent" and "did not look" stay distinguishable. None of the - `quick` (quick rules);
instruments decides alone. - `engine` (the Joern model layer);
- `facts` (repo facts, such as callers and function span);
- `source`;
- `entry_points`;
- `roles` (model source, sink and sanitizer roles);
- `delta`;
- `verify` and `agree` (the plane's verify passes and their agreement);
- `model_sinks` (the archived stage-1 classifier, `benchmarks/archive/model_sinks.py`). Its logic now
lives in `plane/tasks/roles.py`.
**Why profile v1 withholds the engine and function length.** A leak audit A candidate is `(path, function, family)` of one repository and pin. At least one instrument must have
(`ousast learn audit-leaks`, `benchmarks/measurements/2026-10-01-decision-engine-leak-audit/`) emitted a signal of that family there (`learn/features.py`).
found that both separate the vulnerable and fixed sides of a pair by construction: a fix removes
a sink, and a guard lengthens the function. That holds on the pair corpus, not on unseen code, so An instrument may not run, may have no coverage, or may fail. It then records that state (`none`,
`INPUT_PROFILES["v1"]` drops the `engine` block and `facts.function_lines` from prompts and from `failed`, `not_applicable`), never a zero. So "silent" and "did not look" stay distinguishable. No
the retrieval distance. instrument decides alone.
**Why profile v1 withholds the engine and function length.** A leak audit found that both separate the
vulnerable and fixed sides of a pair by construction (`ousast learn audit-leaks`,
`benchmarks/measurements/2026-10-01-decision-engine-leak-audit/`):
- a fix removes a sink;
- a guard lengthens the function.
That holds on the pair corpus, not on unseen code. So `INPUT_PROFILES["v1"]` drops the `engine` block
and `facts.function_lines` from prompts and from the retrieval distance.
## 2. Labels come from ground truth only ## 2. Labels come from ground truth only
`ousast learn labels` (`learn/labels.py`) reads only the sources on a fail-closed allow-list `ousast learn labels` (`learn/labels.py`) reads only the sources on a fail-closed allow-list
(`learn/sources.toml`): `population-v1`, `population-v2`, `dev-php` (reviewed recipe sites), (`learn/sources.toml`). The sources are `population-v1`, `population-v2`, `dev-php` (reviewed recipe
`pairs`, `advisory-fixes`, `adjudications` and `assumed_benign`. A file the list does not name is sites), `pairs`, `advisory-fixes`, `adjudications` and `assumed_benign`. A file the list does not name
refused before it is opened; the reserved population v3 is excluded unread. is refused before it is opened. The reserved population v3 is excluded unread.
- **Positives** are declared sites and the vulnerable side of a fix; **negatives** are the same - **Positives** are declared sites and the vulnerable side of a fix.
function on the fixed side where the fix changed it, and recorded `False`/`FP` adjudications. - **Negatives** are two kinds:
A vulnerable-pin candidate matching no site is unlabelled, never negative. - the same function on the fixed side, where the fix changed it;
- **Assumed benign** (maintainer decision 2026-09-30): ordinary non-security commits of the training - recorded `False`/`FP` adjudications.
repositories, at least 90 days from any fix, with no later security commit touching the same - A vulnerable-pin candidate matching no site is unlabelled, never negative.
files. They are their own source with weight 0.5, conditional on a signal at the tip and none at - **Assumed benign** (maintainer decision 2026-09-30) covers ordinary non-security commits of the
the base, and reported separately, never merged with verified negatives. training repositories. Each is at least 90 days from any fix. No later security commit touches the
- **Plane verdicts are features, not labels.** "Agreed", "rejected" or "disputed" is the output of same files.
the verify instrument; it enters the record as the `verify` and `agree` blocks. Using it as a - They are their own source with weight 0.5, conditional on a signal at the tip and none at the base.
label would teach the engine to reproduce the plane, mistakes included, so "rejected by the - They are reported separately, never merged with verified negatives.
plane" is never a negative (`learn/labels.py`: "A plane verdict ... is never a label"). - **Plane verdicts are features, not labels.** "Agreed", "rejected" or "disputed" is the output of the
verify instrument. It enters the record as the `verify` and `agree` blocks. Using it as a label would
teach the engine to reproduce the plane, mistakes included. So "rejected by the plane" is never a
negative (`learn/labels.py`: "A plane verdict ... is never a label").
`ousast learn memory build` joins each label with the candidate's feature record and an excerpt of `ousast learn memory build` joins each label with the candidate's feature record and an excerpt of
its function and stores one `example` row (no rationale) plus the excerpt blob. its function. It stores one `example` row (no rationale) plus the excerpt blob.
## 3. Retrieval of similar labelled cases ## 3. Retrieval of similar labelled cases
`learn/retrieve.py`, no model: `learn/retrieve.py` uses no model. It works in three steps:
1. **Signal profile.** A Gower distance over the allow-listed features of the active profile, 1. **Signal profile.** It computes a Gower distance (a distance for mixed-type features) over the
per instrument block, with missing values explicit (a different state counts as distance 1). allow-listed features of the active profile, per instrument block. Missing values are explicit: a
The nearest 40 examples of the candidate's family and the nearest 40 of other families go on. different state counts as distance 1. The nearest 40 examples of the candidate's family go on. So do
2. **Code embeddings.** A cosine re-rank of the excerpts' embeddings (OpenAI the nearest 40 of other families.
`text-embedding-3-small` through OpenRouter, 1,536 dimensions, cached by excerpt hash): 2. **Code embeddings.** It re-ranks by cosine over the excerpts' embeddings. The embeddings come from
`r = 0.5 (1 - D) + 0.5 cos`. Without the candidate's vector the retrieval is signals only. OpenAI `text-embedding-3-small` through OpenRouter (1,536 dimensions, cached by excerpt hash). The
3. **Balance.** Six examples: at most half per label, at most 2 per repository group, at most 2 formula is `r = 0.5 (1 - D) + 0.5 cos`. Without the candidate's vector, the retrieval uses signals only.
contrast examples from other families. 3. **Balance.** It picks six examples:
- at most half per label;
- at most 2 per repository group;
- at most 2 contrast examples from other families.
**The evaluation boundary.** `eligible()` is the only way an example enters a prompt, for retrieved **The evaluation boundary.** `eligible()` is the only way an example enters a prompt. This holds for
examples and compiled demonstrations alike: never the candidate's own repository group, never a retrieved examples and compiled demonstrations alike. An eligible example is never:
group the fold evaluates, never the held-out source or framework, and (outside deployment) never
a near-duplicate of the candidate (cosine >= 0.98). The program repeats the check on the ids it - from the candidate's own repository group;
actually rendered before any call. - from a group the fold evaluates;
- from the held-out source or framework;
- a near-duplicate of the candidate (cosine >= 0.98), outside deployment.
The program repeats the check on the ids it actually rendered, before any call.
## 4. The AI classifier, compiled ## 4. The AI classifier, compiled
`learn/program.py` and `learn/compile.py`, DSPy-style and in-house. The prompt is the instruction The classifier lives in `learn/program.py` and `learn/compile.py`. It is DSPy-style (DSPy is a
plus the compiled demonstrations (a prefix identical for every candidate, so the provider's prefix framework that compiles prompts from examples) and built in-house.
cache hits), then the retrieved examples, then the candidate's excerpt, signals, roles and family.
The model (`deepseek-flash`) answers `vulnerable`, `not_vulnerable` or `unsure` with a confidence The prompt has this order:
and a rationale that must cite a line of the code. It is sampled `k` times; the score `s` is the
mean over samples, and a majority `unsure` can never BLOCK. Compilation bootstraps demonstrations 1. the instruction plus the compiled demonstrations. This prefix is identical for every candidate, so
and searches instructions on a compile split of repository groups, scored by a pre-registered the provider's prefix cache hits;
balanced Brier score (`learn/compile.toml`); the program is stored as `programs/<id>.json`. 2. the retrieved examples;
3. the candidate's excerpt, signals, roles and family.
The model (`deepseek-flash`) answers `vulnerable`, `not_vulnerable` or `unsure`. It adds a confidence
and a rationale. The rationale must cite a line of the code. The model is sampled `k` times. The score
`s` is the mean over samples. A majority `unsure` can never BLOCK.
Compilation bootstraps demonstrations and searches instructions on a compile split of repository groups.
A pre-registered balanced Brier score scores it (`learn/compile.toml`). The program is stored as
`programs/<id>.json`.
## 5. Calibration and the two operating points ## 5. Calibration and the two operating points
`learn/calibrate.py`: Platt scaling of `s`, cross-fitted so that no map is checked on the fold it `learn/calibrate.py` applies Platt scaling (a logistic map from score to probability) to `s`. It is
was fitted on. Calibration **holds** when the slope interval contains 1, the intercept interval cross-fitted, so no map is checked on the fold it was fitted on.
contains 0 and ECE <= 0.05. **BLOCK** is offered only when calibration holds and a fold reaches the
required precision; otherwise the engine offers **ADVISORY** only. Calibration **holds** when the slope interval contains 1, the intercept interval contains 0 and ECE
(expected calibration error) <= 0.05.
**BLOCK** is offered only when calibration holds and a fold reaches the required precision. Otherwise
the engine offers **ADVISORY** only.
## The loop: memory improves decisions without retraining ## The loop: memory improves decisions without retraining
@@ -125,25 +165,36 @@ flowchart LR
gate -- "accepted round" --> ledger["Rule status in the ledger"] gate -- "accepted round" --> ledger["Rule status in the ledger"]
``` ```
- **Decisions.** The compiled part of the program is its instruction and demonstrations; the - **Decisions.**
examples are retrieved from memory for each candidate. A new `example` row is therefore eligible - The compiled part of the program is its instruction and demonstrations.
for retrieval in the next decision without recompiling the program, subject to the same - The examples are retrieved from memory for each candidate.
evaluation boundary. `ousast learn curve` measures the metric over memory size on a fixed - So a new `example` row is eligible for retrieval in the next decision without recompiling. The same
candidate subset. evaluation boundary applies.
- **Rules.** `ousast improve --memory` reads the run rows (alerts, verdicts, coverage) and proposes - `ousast learn curve` measures the metric over memory size on a fixed candidate subset.
rule-status edits only (shadow a rule that is repeatedly false across repositories; re-enable a - **Rules.** `ousast improve --memory` reads the run rows (alerts, verdicts, coverage). It proposes
shadow rule that keeps hitting agreed declared sites). Rows from holdout pairs, the gated rule-status edits only:
manifest's own cases and any `--qualify-population` are dropped first, and every proposal goes - shadow a rule that is repeatedly false across repositories;
through the same validator and gate as any other edit (`improve/memory.py`). - re-enable a shadow rule that keeps hitting agreed declared sites.
Some rows are dropped first: rows from holdout pairs, the gated manifest's own cases and any
`--qualify-population`. Every proposal goes through the same validator and gate as any other edit
(`improve/memory.py`).
## What the first slice measured ## What the first slice measured
Record: `benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json` (counts Record: `benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json` (counts
only). Family **injection**; memory of 724 static examples from all families; input profile v1; only). The setup:
framework priors off; out-of-repository evaluation (compile split never evaluated, outer grouped
5-fold, calibration cross-fitted, operating points nested); 95% repository-cluster bootstrap - family **injection**;
intervals. The slice's spend ($0.003215 per candidate at evaluation, k = 5, and its replay at - memory of 724 static examples from all families;
$0) is on [Token ergonomics](token-ergonomics.md#measured-costs). - input profile v1;
- framework priors off;
- out-of-repository evaluation: compile split never evaluated, outer grouped 5-fold, calibration
cross-fitted, operating points nested;
- 95% repository-cluster bootstrap intervals.
The slice's spend is on [Token ergonomics](token-ergonomics.md#measured-costs). It was $0.003215 per
candidate at evaluation, k = 5, and its replay cost $0.
| Measure | Value | What it means | | Measure | Value | What it means |
| --- | --- | --- | | --- | --- | --- |
@@ -152,9 +203,12 @@ $0) is on [Token ergonomics](token-ergonomics.md#measured-costs).
| ADVISORY | recall 0.90 (0.82 - 0.97), precision 0.54 (0.44 - 0.65) | precision is at the positive share (0.53), so at this threshold ADVISORY is **not discriminating**: it flags nearly as indiscriminately as flagging everything would | | ADVISORY | recall 0.90 (0.82 - 0.97), precision 0.54 (0.44 - 0.65) | precision is at the positive share (0.53), so at this threshold ADVISORY is **not discriminating**: it flags nearly as indiscriminately as flagging everything would |
| BLOCK | not offered | calibration does not hold (ECE 0.064 > 0.05), and every outer fold reported `precision_unreachable` | | BLOCK | not offered | calibration does not hold (ECE 0.064 > 0.05), and every outer fold reported `precision_unreachable` |
**The engine is not adopted.** A second slice (2026-10-02) evaluated the six evaluable families **The engine is not adopted.** A second slice (2026-10-02) evaluated the six evaluable families the
the same way: pooled AUC 0.77 to 0.96 per family, within-pair AUC 0.72 to 0.93, BLOCK offered for same way ([Decision engine](decision-engine.md#second-slice-six-families-2026-10-02)). The results: pooled AUC 0.77 to 0.96 per family, within-pair AUC 0.72 to 0.93, BLOCK offered for
none, re-run agreement below 0.9 for three families none. Re-run agreement was below 0.9 for three families.
([Decision engine](decision-engine.md#second-slice-six-families-2026-10-02)). Leave-one-source-out
and leave-one-framework-out, integration into scan and pre-push, and the one-time check on Still ahead:
population v3 are still ahead.
- leave-one-source-out and leave-one-framework-out;
- integration into scan and pre-push;
- the one-time check on population v3.
+50 -40
View File
@@ -1,8 +1,8 @@
# Memory # Memory
One store, keyed by repository and pin, holds what plane runs and the decision engine learned. One store holds what plane runs and the decision engine learned. It is keyed by repository and pin
The code is `src/openultrasast/plane/memory.py`; the same layout serves both backends. (a fixed commit). The code is `src/openultrasast/plane/memory.py`. The same layout serves both backends.
How the stored rows feed detection is on [Memory and detection](memory-and-detection.md). [Memory and detection](memory-and-detection.md) shows how the stored rows feed detection.
## What is stored ## What is stored
@@ -17,7 +17,7 @@ programs/<program id>.json compiled decision-engine programs
``` ```
**Rows.** Every row carries `id`, `kind`, `repo`, `pin`, `run`, `task`, `population`, `split` **Rows.** Every row carries `id`, `kind`, `repo`, `pin`, `run`, `task`, `population`, `split`
and `image` (`ROW_FIELDS`); `id` is the sha256 of (kind, run, task, subject), so a repeated row and `image` (`ROW_FIELDS`). `id` is the sha256 of (kind, run, task, subject). So a repeated row
replaces itself. The kinds (`KINDS`) and who writes them: replaces itself. The kinds (`KINDS`) and who writes them:
| Kind | Written by | Holds | | Kind | Written by | Holds |
@@ -35,12 +35,15 @@ replaces itself. The kinds (`KINDS`) and who writes them:
| `experiment_result` | `ousast learn experiment analyse --record` | the paired estimates, the looks and the adoption verdict (counts and intervals only) | | `experiment_result` | `ousast learn experiment analyse --record` | the paired estimates, the looks and the adoption verdict (counts and intervals only) |
| `label`, `decision` | reserved for later decision-engine tasks | declared in `KINDS`; not written by the code on main yet | | `label`, `decision` | reserved for later decision-engine tasks | declared in `KINDS`; not written by the code on main yet |
**Blobs** (`put_blob`) are content-addressed: a 64-hex name under a fixed prefix. Excerpts are **Blobs** (`put_blob`) are content-addressed: each has a 64-hex name under a fixed prefix.
written by `learn memory build` (`learn/examples.py`), embeddings by `learn memory embed` Content-addressed means the name is the hash of the content. Who writes each blob:
(`learn/embeddings.py`, keyed by excerpt sha so an excerpt is embedded once), responses by the
program's caller (`learn/program.py`, keyed by the sha256 of model, parameters digest, messages, - Excerpts: `learn memory build` (`learn/examples.py`).
temperature, sample and format, so a rerun replays at no cost), and compiled programs by - Embeddings: `learn memory embed` (`learn/embeddings.py`). They are keyed by excerpt sha, so an
`learn compile` (`learn/compile.py`). excerpt is embedded once.
- Responses: the program's caller (`learn/program.py`). The key is the sha256 of model, parameters
digest, messages, temperature, sample and format. So a rerun replays at no cost.
- Compiled programs: `learn compile` (`learn/compile.py`).
## Write path ## Write path
@@ -60,13 +63,14 @@ flowchart LR
put --> store[("Memory store")] put --> store[("Memory store")]
``` ```
- `ingest` validates every row first: a malformed file fails before anything is written. - `ingest` validates every row first. A malformed file fails before anything is written.
- Rows are merged into their repository + pin object by `id`. Every object is written whole and the - Rows are merged by `id` into their repository + pin object.
index last, so an interrupted ingest leaves nothing a repeat would not repair. - Every object is written whole, and the index last. So a repeat repairs anything an interrupted
ingest left behind.
- Each row object carries tags (`TAG_FIELDS`: `repo`, `pin`, `kind`, `family`, `run`, - Each row object carries tags (`TAG_FIELDS`: `repo`, `pin`, `kind`, `family`, `run`,
`population`, `split`) listing the distinct values of its rows (`*` when they exceed 256 `population`, `split`). The tags list the distinct values of its rows. A tag is `*` when the
characters). values exceed 256 characters.
- The plane's store failures never change a run's result: an ingest can be repeated by hand - The plane's store failures never change a run's result. An ingest can be repeated by hand
(`cli.py`, `_plane_memory`). (`cli.py`, `_plane_memory`).
## Read path ## Read path
@@ -85,14 +89,14 @@ flowchart LR
rules --> gate{"Unchanged validator and gate"} rules --> gate{"Unchanged validator and gate"}
``` ```
- **Seed** (`memory.seed`): a Run task whose `openultrasast.io/memory-key` matches stored facts gets - **Seed** (`memory.seed`).
them and is marked done; a changed pin, candidate set or runner image finds nothing and - A Run task gets stored facts when its `openultrasast.io/memory-key` matches them. It is then marked done.
recomputes. The loop's `memory-snapshot` is seeded the same way, because a sandboxed task cannot - A changed pin, candidate set or runner image finds nothing and recomputes.
read the store. - The loop's `memory-snapshot` is seeded the same way. A sandboxed task cannot read the store.
- **Retrieval** (`learn/retrieve.py`): see - **Retrieval** (`learn/retrieve.py`): see
[Memory and detection](memory-and-detection.md#3-retrieval-of-similar-labelled-cases). [Memory and detection](memory-and-detection.md#3-retrieval-of-similar-labelled-cases).
- **`improve --memory`** (`improve/memory.py`): rows from holdout pairs, from the gated manifest's - **`improve --memory`** (`improve/memory.py`). Some rows are dropped before any rule sees them:
own cases and from any `--qualify-population` are dropped before any rule sees them. See rows from holdout pairs, from the gated manifest's own cases, and from any `--qualify-population`. See
[Architecture](architecture.md#proposals-from-plane-memory). [Architecture](architecture.md#proposals-from-plane-memory).
## Backends: FileStore and the S3 store ## Backends: FileStore and the S3 store
@@ -109,26 +113,32 @@ flowchart LR
### Verify at startup, no fallback ### Verify at startup, no fallback
The S3 store never configures its bucket. Each time an `S3Store` is opened for a bucket and The S3 store never configures its bucket. It only checks it. An `S3Store` is opened for a bucket and
prefix (once per process), `verify_bucket()` checks, using reads plus one small probe object at prefix once per process. Each time, `verify_bucket()` runs these checks. It uses reads plus one small
`<prefix>/_probe/select.jsonl`: probe object at `<prefix>/_probe/select.jsonl`.
1. versioning is `Enabled` (provenance cites object versions); 1. Versioning is `Enabled`. Provenance cites object versions.
2. an enabled lifecycle rule expires `<prefix>/runs/` after a number of days, as a plain prefix 2. An enabled lifecycle rule expires `<prefix>/runs/` after a number of days. It uses a plain prefix
filter, and no enabled rule expires the whole store; filter. No enabled rule expires the whole store.
3. S3 Select answers a probe query over JSON Lines; 3. S3 Select (a server-side query over JSON objects) answers a probe query over JSON Lines.
4. the probe's object tags are readable. 4. The probe's object tags are readable.
Anything missing is named, with the admin command that fixes it, in one `MemoryStoreError`, and If anything is missing, the store does not start. One `MemoryStoreError` names each problem and the
the store does not start. There is no local or fetch-and-filter fallback: a filtered read that admin command that fixes it.
S3 Select cannot answer raises a `MemoryStoreError` instead of returning "no rows". The exact
messages are listed on [RustFS setup](rustfs.md#what-the-store-verifies-at-startup). There is no local or fetch-and-filter fallback. A filtered read that S3 Select cannot answer raises a
`MemoryStoreError`. It does not return "no rows". [RustFS setup](rustfs.md#what-the-store-verifies-at-startup)
lists the exact messages.
### Queryable fields per kind ### Queryable fields per kind
RustFS's Select infers an object's JSON schema from its leading rows, so a `where` field missing RustFS's Select infers an object's JSON schema from its leading rows. This causes two failures
there fails with `EvaluatorBindingDoesNotExist` even when a later row has it, and a column that is (measured 2026-10-02; `memory.py`, `S3Store._select`):
null in the first 1000 rows breaks Select on the whole object (measured 2026-10-02; `memory.py`,
`S3Store._select`). The store therefore writes every row with its kind's full set of queryable - A `where` field missing from the leading rows fails with `EvaluatorBindingDoesNotExist`. This
fields (`QUERY_FIELDS`), using `""` where a field does not apply, and refuses a `where` on an happens even when a later row has the field.
undeclared field. Existing rows are rewritten once with `ousast plane memory-normalise`. - A column that is null in the first 1000 rows breaks Select on the whole object.
So the store writes every row with its kind's full set of queryable fields (`QUERY_FIELDS`). It uses
`""` where a field does not apply. It refuses a `where` on an undeclared field. Existing rows are
rewritten once with `ousast plane memory-normalise`.
+82 -56
View File
@@ -1,23 +1,34 @@
# The plane on ax # The plane on ax
Model-driven work over whole repositories runs on a separate agentic plane: a **Run** Model-driven work over whole repositories runs on a separate agentic plane. The plane has these parts:
(`plane/runs/*.yaml`, kind `openultrasast.io/v1alpha1 Run`) is a DAG of steps, and each step is
an ax **Task** (`plane/tasks/`) that binds its Workspaces (`plane/workspaces/`), at most one ax - A **Run** (`plane/runs/*.yaml`, kind `openultrasast.io/v1alpha1 Run`) is a DAG (a graph of steps
**Model** (`plane/models/`: `deepseek-flash` for chat, `openrouter-embedding` for embeddings) and with no cycles).
its own `budget: {usd, calls}`. ax (google/ax over Agent Substrate, in a single-node kind cluster - Each step is an ax **Task** (`plane/tasks/`).
on the maintainer's host) is the only executor; there is no local subprocess path. Bring-up and - A Task binds its Workspaces (`plane/workspaces/`).
the host's lessons are in [ax on this host](ops/ax/README.md); where each part runs today (the - A Task binds at most one ax **Model** (`plane/models/`). `deepseek-flash` serves chat.
reconciler and the receiver on the host, the tasks in gVisor on Agent Substrate) and what a `openrouter-embedding` serves embeddings.
separate Kubernetes cluster would take (planned) are on [Deployment](deployment.md); this page - A Task carries its own `budget: {usd, calls}`.
draws the flow.
ax is the only executor. It is google/ax, Google's controller that runs agent tasks, over Agent Substrate (its
sandboxed actor runtime). It runs in a single-node kind cluster (Kubernetes in Docker) on the maintainer's host.
There is no local subprocess path.
Where to read more:
- Bring-up and the host's lessons are in [ax on this host](ops/ax/README.md).
- [Deployment](deployment.md) shows where each part runs today. The reconciler and the receiver run on the host.
The tasks run in gVisor (a sandboxing kernel) on Agent Substrate. It also covers what a separate Kubernetes
cluster would take (planned).
- This page draws the flow.
Sources: `src/openultrasast/plane/reconciler.py`, `router.py`, `egress.py`, `runner.py`, Sources: `src/openultrasast/plane/reconciler.py`, `router.py`, `egress.py`, `runner.py`,
`generate.py`, `memory.py`, and `src/openultrasast/plane/tasks/`. `generate.py`, `memory.py`, and `src/openultrasast/plane/tasks/`.
## One task, end to end ## One task, end to end
`ousast plane run RUN.yaml` (the reconciler) does this for every ready task; `--workers` bounds `ousast plane run RUN.yaml` (the reconciler) does this for every ready task. `--workers` bounds the
the tasks in flight, and a rerun skips tasks already done. tasks in flight. A rerun skips tasks already done.
```mermaid ```mermaid
sequenceDiagram sequenceDiagram
@@ -48,44 +59,55 @@ sequenceDiagram
What each step guarantees: What each step guarantees:
- **Credentials** (`router.py`, `runner.py`). The reconciler reads the variable the Model's - **Credentials** (`router.py`, `runner.py`).
`secretKey.key` names from the operator's environment or `.env` (which never overrides an - The reconciler reads the variable that the Model's `secretKey.key` names.
exported variable). The key exists only in the start request; the runner keeps it in memory, - It reads it from the operator's environment or `.env`. `.env` never overrides an exported variable.
passes it to the command's environment, redacts it from echoed stderr and never writes it. - The key exists only in the start request.
- **Egress** (`egress.py`). One EgressPolicy per actor, deny by default, hostnames only: plain - The runner keeps it in memory and passes it to the command's environment.
HTTP to the artifact receiver, TLS passthrough to the source hosts of the bound Workspaces and - The runner redacts it from echoed stderr and never writes it.
to the hosts the bound Model declares (`openultrasast.io/egress-hosts`). It is written between - **Egress** (`egress.py`). Egress means outbound network traffic.
`ax apply` and the resume, and deleted with the actor. - Each actor gets one EgressPolicy. It denies by default and names hostnames only.
- **Inputs** (`reconciler.py`). Artifacts other tasks produced are never rendered into the - It allows plain HTTP to the artifact receiver.
manifest; the runner fetches them from the receiver after the start, and the receiver serves a - It allows TLS passthrough to the source hosts of the bound Workspaces.
running task its declared inputs only. - It allows TLS passthrough to the hosts the bound Model declares (`openultrasast.io/egress-hosts`).
- **Completion** (`runner.py`). ax has no Completed phase: the runner's delivery of the output tar - The reconciler writes it between `ax apply` and the resume. It is deleted with the actor.
is the task's completion. A completion marker stops a resumed actor from repeating billed work; - **Inputs** (`reconciler.py`).
when the marker says the delivery failed, a later boot retries the delivery alone. - Artifacts that other tasks produced are never rendered into the manifest.
- **Budgets** (`plane/budget.py`). The metered client refuses the next call once a task's `usd` or - The runner fetches them from the receiver after the start.
`calls` ceiling is reached; the task ends `unfinished` and resumes on a rerun with a larger - The receiver serves a running task its declared inputs only.
budget. Model-free tasks run with `{usd: 0, calls: 0}`, so a stray call fails loudly. - **Completion** (`runner.py`).
- **Attribution.** `ousast plane status RUN` prints per task its status, `calls`, `prompt`, - ax has no Completed phase. The runner's delivery of the output tar is the task's completion.
`cache_hit` and `output` tokens and `usd` from the tasks' `summary.json`, and writes - A completion marker stops a resumed actor from repeating billed work.
`attribution.json` in the run directory (`$OUSAST_RESULTS/plane/RUN/`, default - If the marker says the delivery failed, a later boot retries only the delivery.
- **Budgets** (`plane/budget.py`).
- The metered client refuses the next call once a task reaches its `usd` or `calls` ceiling.
- The task then ends `unfinished`. It resumes on a rerun with a larger budget.
- Model-free tasks run with `{usd: 0, calls: 0}`, so a stray call fails loudly.
- **Attribution.**
- `ousast plane status RUN` prints these per task: status, `calls`, `prompt`, `cache_hit` and
`output` tokens, and `usd`. It reads them from the tasks' `summary.json`.
- It writes `attribution.json` in the run directory (`$OUSAST_RESULTS/plane/RUN/`, default
`~/ousast-results/plane/`). `~/ousast-results/plane/`).
!!! note "The golden snapshot" !!! note "The golden snapshot"
Agent Substrate boots every new actor template once as a *golden* actor to snapshot it, with Agent Substrate boots every new actor template once as a *golden* actor and snapshots it. That boot
the same image, Task and environment, and inside the sandbox nothing tells that boot from the uses the same image, Task and environment. Inside the sandbox, nothing tells that boot from the real one.
real one. So the runner's command never starts by itself: it waits for the start request,
which the reconciler sends through `atenet-router` to the task's actor, never to the golden So the runner's command never starts by itself. It waits for the start request. The reconciler sends
one. Workspaces are prepared only after the start, so the golden boot touches no network and a that request through `atenet-router` to the task's actor, never to the golden one.
started actor already runs under its egress policy (a clone at boot failed on the live cluster
while the actor was being restored, 2026-09-29; `runner.py`, `_run_once`). The first resume of Workspaces are prepared only after the start. So the golden boot touches no network. A started actor
a new image can time out while the snapshot is built, which is why the reconciler retries it already runs under its egress policy. A clone at boot once failed on the live cluster while the actor
was being restored (2026-09-29; `runner.py`, `_run_once`).
The first resume of a new image can time out while the snapshot is built. So the reconciler retries it
for up to `OUSAST_RESUME_TIMEOUT` seconds (900). for up to `OUSAST_RESUME_TIMEOUT` seconds (900).
## The Runs: validation and loop ## The Runs: validation and loop
`ousast plane workspaces POPULATION --validation-set ...` (`generate.py`) writes one chain per `ousast plane workspaces POPULATION --validation-set ...` (`generate.py`) writes one chain per case.
case; `--loop` adds the `alerts` step per case and the loop's singleton steps. Every step except `--loop` adds the `alerts` step per case and the loop's singleton steps. Every step is model-free
the verify passes (and `roles` in harvest Runs) is model-free. except the verify passes (and `roles` in harvest Runs).
### Per case ### Per case
@@ -119,23 +141,27 @@ flowchart LR
improve --> outcome["gate.json, journal.json, and rule_policy.json only if accepted"] improve --> outcome["gate.json, journal.json, and rule_policy.json only if accepted"]
``` ```
- **Before the Run** `ousast plane run` seeds from the store (`memory.seed`): a task whose - **Before the Run**, `ousast plane run` seeds from the store (`memory.seed`).
`openultrasast.io/memory-key` (repository, pin, candidates digest, runner image digest) has - Each task has an `openultrasast.io/memory-key`: repository, pin, candidates digest and runner image digest.
stored facts gets them and is marked done, and the sandboxed `memory-snapshot` task gets the - If the store has facts for that key, the task gets them and is marked done.
store's rows after the guard. **After the Run** it ingests every delivered `memory.jsonl` - The sandboxed `memory-snapshot` task gets the store's rows after the guard.
(`ousast plane remember RUN` repeats this by hand). See [Memory](memory.md). - **After the Run**, it ingests every delivered `memory.jsonl`. `ousast plane remember RUN` repeats
- The loop's rules and its guard are described in this by hand. See [Memory](memory.md).
[Architecture](architecture.md#proposals-from-plane-memory). Adopting an accepted ledger stays a - [Architecture](architecture.md#proposals-from-plane-memory) describes the loop's rules and its guard.
maintainer commit (`src/openultrasast/plane/tasks/loop.py`). Adopting an accepted ledger stays a maintainer commit (`src/openultrasast/plane/tasks/loop.py`).
- `ousast plane harvest` writes the decision engine's harvest Runs (verify a/b with agree, and - `ousast plane harvest` writes the decision engine's harvest Runs (verify a/b with agree, and
model `roles`); `ousast plane alerts-engine` produces a Run's `alerts` from the Joern engine model `roles`).
image on the host for PHP and other languages quick mode does not cover. - `ousast plane alerts-engine` produces a Run's `alerts` from the Joern engine image on the host. It
covers PHP and other languages that quick mode does not cover.
## Where this runs in production ## Where this runs in production
ax is a Kubernetes application over Agent Substrate, every Task runs the one runner image as an actor that ax is a Kubernetes application over Agent Substrate. Every Task runs the one runner image as an actor. Each actor
occupies a whole worker, and two profiles of this code base are the target: `kind` on a laptop (today) and `k3s` occupies a whole worker. This code base targets two profiles:
in production (planned), where the same CLI submits Runs remotely and the tasks deliver to the S3 store:
- `kind` on a laptop (today).
- `k3s` in production (planned). There the same CLI submits Runs remotely, and the tasks deliver to the S3 store.
[Production topology: ax on a Kubernetes cluster](deployment.md#production-topology-ax-on-a-kubernetes-cluster) [Production topology: ax on a Kubernetes cluster](deployment.md#production-topology-ax-on-a-kubernetes-cluster)
draws it and sizes the pre-push hook at scale. draws it and sizes the pre-push hook at scale.
+74 -50
View File
@@ -1,16 +1,22 @@
# RustFS setup for the memory store # RustFS setup for the memory store
The plane's memory store can live in an S3-compatible bucket: `OUSAST_MEMORY=s3://sast-memory` The plane's memory store can live in an S3-compatible bucket. RustFS is an S3-compatible object
selects `S3Store` (`src/openultrasast/plane/memory.py`), which talks to the server through boto3. storage server.
Any S3-compatible store with bucket versioning, lifecycle rules, object tags and S3 Select works;
RustFS is the server the contract tests run against. The maintainer's server is RustFS at
`files.because-security.com`, bucket `sast-memory`, store at the bucket root. What the store keeps
there is described on [Memory](memory.md).
**The store never configures its bucket.** An admin sets the bucket up once with admin - `OUSAST_MEMORY=s3://sast-memory` selects `S3Store` (`src/openultrasast/plane/memory.py`).
credentials. The agent account the store runs as can read the bucket's settings but not change - `S3Store` talks to the server through boto3 (the AWS SDK for Python).
them, and reads and writes objects freely. Every time the store opens, it checks the setup and - Any S3-compatible store works if it has bucket versioning, lifecycle rules, object tags and S3 Select
refuses to start if anything is missing. (a server-side query over JSON objects).
- RustFS is the server the contract tests run against.
- The maintainer's server is RustFS at `files.because-security.com`. The bucket is `sast-memory`. The
store sits at the bucket root.
[Memory](memory.md) describes what the store keeps there.
**The store never configures its bucket.** An admin sets the bucket up once with admin credentials.
The store runs as an agent account. That account can read the bucket's settings but not change them.
It reads and writes objects freely. Every time the store opens, it checks the setup. It refuses to start
if anything is missing.
```mermaid ```mermaid
flowchart LR flowchart LR
@@ -35,8 +41,8 @@ aws s3api put-bucket-versioning \
--versioning-configuration Status=Enabled --versioning-configuration Status=Enabled
``` ```
**2. Expire raw run outputs under `runs/` after 30 days.** Everything else (`repos/`, `facts/`, **2. Expire raw run outputs under `runs/` after 30 days.** Everything else is kept: `repos/`, `facts/`,
`index.jsonl` and the blobs) is kept, so the rule must be a plain prefix filter on `runs/`. `index.jsonl` and the blobs. So the rule must be a plain prefix filter on `runs/`.
```bash ```bash
aws s3api put-bucket-lifecycle-configuration \ aws s3api put-bucket-lifecycle-configuration \
@@ -46,9 +52,10 @@ aws s3api put-bucket-lifecycle-configuration \
``` ```
`put-bucket-lifecycle-configuration` **replaces every rule on the bucket**. If the bucket already `put-bucket-lifecycle-configuration` **replaces every rule on the bucket**. If the bucket already
has rules, merge this one into them first. The rule id `ousast-runs-expiry` and the 30 days are the has rules, merge this one into them first.
store's defaults (`RUNS_RULE_ID`, `RUNS_EXPIRE_DAYS`); the store accepts any enabled, untagged
prefix rule on `runs/` with a number of days. The rule id `ousast-runs-expiry` and the 30 days are the store's defaults (`RUNS_RULE_ID`,
`RUNS_EXPIRE_DAYS`). The store accepts any enabled, untagged prefix rule on `runs/` with a number of days.
To read the result back: To read the result back:
@@ -59,8 +66,8 @@ aws s3api get-bucket-lifecycle-configuration --endpoint-url https://files.becaus
## The agent policy ## The agent policy
Attach this policy to the agent account (the account whose keys go into `.env`). It is the Attach this policy to the agent account. That is the account whose keys go into `.env`. It is the
complete policy: nothing else is needed, and it grants no `Put*` on versioning or lifecycle. complete policy, and nothing else is needed. It grants no `Put*` on versioning or lifecycle.
```json ```json
{ {
@@ -102,14 +109,14 @@ complete policy: nothing else is needed, and it grants no `Put*` on versioning o
| `BucketSettingsReadOnly` | `arn:aws:s3:::sast-memory` | The agent can list the bucket and its versions and *read* its location, versioning and lifecycle settings, which is what the startup check needs; it cannot change them. | | `BucketSettingsReadOnly` | `arn:aws:s3:::sast-memory` | The agent can list the bucket and its versions and *read* its location, versioning and lifecycle settings, which is what the startup check needs; it cannot change them. |
| `ObjectsReadWrite` | `arn:aws:s3:::sast-memory/*` | The agent reads, writes and deletes objects and their versions and reads and writes object tags; this covers row objects, blobs, the index, the Select probe, and the `kind` tag the store filters by. | | `ObjectsReadWrite` | `arn:aws:s3:::sast-memory/*` | The agent reads, writes and deletes objects and their versions and reads and writes object tags; this covers row objects, blobs, the index, the Select probe, and the `kind` tag the store filters by. |
So the agent writes objects freely but cannot change bucket settings: turning versioning off or So the agent writes objects freely but cannot change bucket settings. Turning versioning off or
removing the expiry rule needs the admin. removing the expiry rule needs the admin.
## The `.env` settings ## The `.env` settings
The store reads its endpoint and credentials from `.env` in the working directory or from the The store reads its endpoint and credentials from `.env` in the working directory or from the
environment (`s3_settings`), never from a manifest, and never prints them. The names are the ones environment (`s3_settings`). It never reads them from a manifest. It never prints them. The names are
the AWS SDKs and CLI use for credentials, so one `.env` serves both the store and `aws s3api`. the ones the AWS SDKs and CLI use for credentials. So one `.env` serves both the store and `aws s3api`.
| Variable | Value for this server | Notes | | Variable | Value for this server | Notes |
| --- | --- | --- | | --- | --- | --- |
@@ -121,26 +128,33 @@ the AWS SDKs and CLI use for credentials, so one `.env` serves both the store an
| `S3_REGION` | optional (`us-east-1` when unset) | the SigV4 signing region; spares a GetBucketLocation call | | `S3_REGION` | optional (`us-east-1` when unset) | the SigV4 signing region; spares a GetBucketLocation call |
| `S3_BUCKET` | `sast-memory` | the bucket `s3://` names when the URL has none; also the real-server tests' default bucket | | `S3_BUCKET` | `sast-memory` | the bucket `s3://` names when the URL has none; also the real-server tests' default bucket |
The client signs with SigV4 and uses path-style addressing (`https://host/bucket/key`), which The client signs with SigV4 (AWS's request-signing scheme). It uses path-style addressing
RustFS needs. The SDK's own checksum trailers are off, because not every S3-compatible server (`https://host/bucket/key`), which RustFS needs. The SDK's own checksum trailers are off. Not every
accepts them. S3-compatible server accepts them.
- **Never commit `.env`.** It is gitignored (`.gitignore`: `.env`, `.env.*`, except - **Never commit `.env`.** It is gitignored (`.gitignore`: `.env`, `.env.*`, except
`.env.example`). `.env.example` lists these names with empty values. `.env.example`). `.env.example` lists these names with empty values.
- **`.env` never overrides a variable already exported in your shell** (`config.load_dotenv`). - **`.env` never overrides a variable already exported in your shell** (`config.load_dotenv`).
A stale `export AWS_SECRET_ACCESS_KEY=...` in your profile silently wins over the file; `unset` A stale `export AWS_SECRET_ACCESS_KEY=...` in your profile silently wins over the file. `unset`
it when a change in `.env` seems to have no effect. it when a change in `.env` seems to have no effect.
- The store needs the `s3` extra (`uv sync --extra s3`, which installs boto3); without it opening - The store needs the `s3` extra (`uv sync --extra s3`, which installs boto3). These are the
the store fails with `OUSAST_MEMORY=s3://... needs boto3: install openultrasast[s3]`. A missing startup errors:
variable fails with `OUSAST_MEMORY=s3://... needs S3_ENDPOINT, AWS_ACCESS_KEY_ID, - Without the extra, opening the store fails with
AWS_SECRET_ACCESS_KEY in .env or the environment` (naming only the missing ones); an endpoint `OUSAST_MEMORY=s3://... needs boto3: install openultrasast[s3]`.
without a scheme fails with `S3_ENDPOINT must be a URL with a scheme (https://host[:port])`. - A missing variable fails with `OUSAST_MEMORY=s3://... needs S3_ENDPOINT, AWS_ACCESS_KEY_ID,
AWS_SECRET_ACCESS_KEY in .env or the environment`. The message names only the missing ones.
- An endpoint without a scheme fails with
`S3_ENDPOINT must be a URL with a scheme (https://host[:port])`.
## What the store verifies at startup ## What the store verifies at startup
`S3Store.verify_bucket()` runs once per process for each bucket and prefix. It reads the `S3Store.verify_bucket()` runs once per process for each bucket and prefix. It does four things:
versioning status and the lifecycle rules, writes a two-line probe object at
`_probe/select.jsonl` (under the prefix, if any), queries it with S3 Select and reads its tags. 1. It reads the versioning status.
2. It reads the lifecycle rules.
3. It writes a two-line probe object at `_probe/select.jsonl` (under the prefix, if any).
4. It queries the probe with S3 Select and reads its tags.
Every missing piece is collected into one error: Every missing piece is collected into one error:
```text ```text
@@ -148,8 +162,8 @@ memory s3://sast-memory: the bucket is not set up for the store. The store never
- <one line per missing piece> - <one line per missing piece>
``` ```
The lines, exactly as the code writes them for this bucket at the root (`<failure>` is the The table lists the lines exactly as the code writes them for this bucket at the root. `<failure>` is
server's error code and the first 200 characters of its message): the server's error code and the first 200 characters of its message.
| Check | Refusal line | | Check | Refusal line |
| --- | --- | | --- | --- |
@@ -161,32 +175,42 @@ server's error code and the first 200 characters of its message):
| tags unreadable | `object tags cannot be read on _probe/select.jsonl (<failure>): the store filters row objects by their kind tag, so the agent needs s3:GetObjectTagging on arn:aws:s3:::sast-memory/*` | | tags unreadable | `object tags cannot be read on _probe/select.jsonl (<failure>): the store filters row objects by their kind tag, so the agent needs s3:GetObjectTagging on arn:aws:s3:::sast-memory/*` |
| S3 Select missing | `S3 Select (SelectObjectContent over JSON Lines) does not answer on _probe/select.jsonl (<failure>). The store requires it and has no local fallback: use a server with S3 Select (RustFS has it; not every S3-compatible server does) and let the agent s3:PutObject and s3:GetObject` | | S3 Select missing | `S3 Select (SelectObjectContent over JSON Lines) does not answer on _probe/select.jsonl (<failure>). The store requires it and has no local fallback: use a server with S3 Select (RustFS has it; not every S3-compatible server does) and let the agent s3:PutObject and s3:GetObject` |
The admin commands inside the messages use `--endpoint-url "$S3_ENDPOINT"`, which is the full The admin commands inside the messages use `--endpoint-url "$S3_ENDPOINT"`. That is the full URL from
URL from `.env`, so they run as written once `.env` is exported. `.env`. So they run as written once `.env` is exported.
There is **no fallback**: the store does not start in a degraded mode, does not drop to a local There is **no fallback**. The store does not:
store, and does not fetch and filter objects itself.
- start in a degraded mode;
- drop to a local store;
- fetch and filter objects itself.
## The RustFS Select limit and the fixed-schema rule ## The RustFS Select limit and the fixed-schema rule
RustFS's S3 Select infers an object's JSON schema from its leading rows. A `where` field that is RustFS's S3 Select infers an object's JSON schema from its leading rows. A `where` field absent from
absent from those rows is unbound, even if a later row has it: the query fails with those rows is unbound, even if a later row has it. The query then fails with
`EvaluatorBindingDoesNotExist` instead of matching (measured 2026-10-02 with the field first `EvaluatorBindingDoesNotExist` instead of matching (measured 2026-10-02 with the field first
present at row 5000). The store does not read that as "no rows", since that answer could silently present at row 5000).
drop matches; it raises:
The store does not read that failure as "no rows". That answer could silently drop matches. It raises:
```text ```text
memory s3://sast-memory: S3 Select on <key> failed (EvaluatorBindingDoesNotExist: ...) (a field of [<fields>] is absent from the object's leading rows, which the server reads as its schema); the store has no local fallback memory s3://sast-memory: S3 Select on <key> failed (EvaluatorBindingDoesNotExist: ...) (a field of [<fields>] is absent from the object's leading rows, which the server reads as its schema); the store has no local fallback
``` ```
The rule that follows: a field the store filters on is present in every row of its kind from The rule that follows has two parts (see [Memory](memory.md#queryable-fields-per-kind)):
the first row on, as `""` where it does not apply; each row kind has a fixed set of queryable
fields, enforced at write time (see [Memory](memory.md#queryable-fields-per-kind)). - A field the store filters on is present in every row of its kind from the first row on. It is `""`
where it does not apply.
- Each row kind has a fixed set of queryable fields. The store enforces it at write time.
## Checks against the real server ## Checks against the real server
The contract tests that talk to a real server run only with `OUSAST_MEMORY_TEST_S3=1` (bucket The contract tests that talk to a real server run only with `OUSAST_MEMORY_TEST_S3=1`. They use the
`OUSAST_MEMORY_TEST_BUCKET`, else `S3_BUCKET`). They verify the bucket at its root, write only bucket `OUSAST_MEMORY_TEST_BUCKET`, else `S3_BUCKET`. The tests:
under a fresh `contract-<id>/` prefix, and remove every version and delete marker under it
afterwards; they never configure the bucket. More on the host setup: - verify the bucket at its root;
[ax on this host](ops/ax/README.md). - write only under a fresh `contract-<id>/` prefix;
- remove every version and delete marker under it afterwards;
- never configure the bucket.
More on the host setup: [ax on this host](ops/ax/README.md).
+200 -128
View File
@@ -1,18 +1,26 @@
# Token ergonomics # Token ergonomics
Model tokens are the one running cost of this project, and they are spent in two places: by the Model tokens are the one running cost of this project. They are spent in two places:
**plane** when a Run verifies candidates or compiles the decision engine, and by the maintainers'
own **agentic coding sessions**. The maintainer's framing (2026-09-29): "the token saving strategy
is a routing concern. ax can handle that and structure our services in small components." So the
runtime half is routing, one Model per Task with its own budget, and reuse of what was already
paid for; the development half is tooling that keeps bulk file contents out of the session.
Every number on this page comes from a committed record, named next to it. Records are counts - by the **plane** (the system that runs scans as small tasks) when a Run verifies candidates
only; none holds code, prompts or responses. or compiles the decision engine;
- by the maintainers' own **agentic coding sessions**.
The before-and-after summary (script pipeline against plane, and the development share) is on The maintainer's framing (2026-09-29): "the token saving strategy is a routing concern. ax can
[Where we stand](where-we-stand.md#4-token-economics-before-and-after); this page holds the handle that and structure our services in small components." Here ax is Google's controller
mechanisms and the detail table. that runs agent tasks on Kubernetes. So the two halves look like this:
- **Runtime** is routing: one Model per Task with its own budget, and reuse of what was
already paid for.
- **Development** is tooling that keeps bulk file contents out of the session.
Every number on this page comes from a committed record, named next to it. Records hold counts
only. None holds code, prompts or responses.
The before-and-after summary is on
[Where we stand](where-we-stand.md#4-token-economics-before-and-after). It compares the script
pipeline with the plane and gives the development share. This page holds the mechanisms and the
detail table.
## A. Runtime: the plane's token economy ## A. Runtime: the plane's token economy
@@ -37,46 +45,67 @@ flowchart LR
### One Model per Task, one budget per Task ### One Model per Task, one budget per Task
A Run is a DAG of ax Tasks; each Task binds at most one Model and its own `budget: {usd, calls}` A Run is a DAG (a graph of steps with no cycles) of ax Tasks. Each Task binds at most one Model.
([The plane on ax](plane.md)). The budget is enforced by the metered client Each Task has its own `budget: {usd, calls}` ([The plane on ax](plane.md)).
(`src/openultrasast/plane/budget.py`, `MeteredClient`): `check()` runs before every call and
raises `BudgetExhausted` once the spent `usd` or the call count reaches the ceiling, so the call
that crosses the ceiling completes and the next one is refused. The task then ends `unfinished`
and resumes from its `units.jsonl` on a rerun with a larger budget; an HTTP 401/402 or
"Insufficient Balance" is an `AccountError`, which fails the task and starts nothing further.
Usage is attributed per call: the wrapped client appends one usage row per call
(`prompt_tokens`, `prompt_cache_hit_tokens`, `completion_tokens`), rows are absorbed even when
the call raises, and they are priced by the Model's price list. An unpriced Model cannot run
under a `usd` budget, and `summary()` reports `usd: None` for it, never 0. Model-free tasks run
with `{usd: 0, calls: 0}`, so a stray call fails loudly.
`ousast plane status RUN` is the attribution table: one row per task with `calls`, `prompt`, The metered client enforces the budget (`src/openultrasast/plane/budget.py`, `MeteredClient`):
`cache_hit`, `output` and `usd` from each task's `summary.json`, a `subtotal` row per Model and a
`total` (a missing value makes a sum `n/a`, never a smaller number); `--units` adds the per-unit 1. `check()` runs before every call.
rows. The same table is written as `attribution.json` in the run directory 2. It raises `BudgetExhausted` once the spent `usd` or the call count reaches the ceiling.
(`src/openultrasast/plane/reconciler.py`, `attribution()`/`status()`). Provider credentials 3. So the call that crosses the ceiling completes, and the next one is refused.
reach a task only in the start request: `router.py` reads the variable the Model's 4. The task then ends `unfinished`. A rerun with a larger budget resumes from its `units.jsonl`.
`secretKey.key` names, the runner keeps the value in memory, passes it to the command's
environment and redacts it from echoed stderr; it never appears in a manifest, a rendered Task An HTTP 401/402 or "Insufficient Balance" is an `AccountError`. It fails the task and starts
or a file. nothing further.
Usage is attributed per call:
- The wrapped client appends one usage row per call (`prompt_tokens`,
`prompt_cache_hit_tokens`, `completion_tokens`).
- Rows are absorbed even when the call raises.
- Rows are priced by the Model's price list.
- An unpriced Model cannot run under a `usd` budget. `summary()` reports `usd: None` for it,
never 0.
- Model-free tasks run with `{usd: 0, calls: 0}`, so a stray call fails loudly.
`ousast plane status RUN` prints the attribution table. It reads each task's `summary.json`
and shows:
- one row per task with `calls`, `prompt`, `cache_hit`, `output` and `usd`;
- a `subtotal` row per Model;
- a `total`. A missing value makes a sum `n/a`, never a smaller number.
`--units` adds the per-unit rows. The same table is written as `attribution.json` in the run
directory (`src/openultrasast/plane/reconciler.py`, `attribution()`/`status()`).
Provider credentials reach a task only in the start request:
- `router.py` reads the variable that the Model's `secretKey.key` names.
- The runner keeps the value in memory and passes it to the command's environment.
- The runner redacts it from echoed stderr.
- It never appears in a manifest, a rendered Task or a file.
### Pay once, reuse ### Pay once, reuse
- **Repo facts are computed model-free and reused.** `repo-facts` reads source text only (no - **Repo facts are computed model-free and reused.**
model, no git) and its `facts.json` is byte-identical across runs over the same tree. The - `repo-facts` reads source text only (no model, no git).
memory store keeps a facts entry under the key `(repo, pin, candidates digest, runner image - Its `facts.json` is byte-identical across runs over the same tree.
digest)` (`src/openultrasast/plane/memory.py`, `memory_key()`); when a later Run's task - The memory store keeps a facts entry under the key `(repo, pin, candidates digest, runner image
carries the same key, `seed()` writes the stored `facts.json`, a `done` summary with digest)` (`src/openultrasast/plane/memory.py`, `memory_key()`).
`calls: 0` and no `usd`, and marks the task done without starting an actor. A changed pin, - A later Run's task may carry the same key. Then `seed()` writes the stored `facts.json` and a
candidate set or image finds nothing and recomputes. `done` summary with `calls: 0` and no `usd`. It marks the task done without starting an
- **Content-addressed memory** (`src/openultrasast/learn/`). Excerpts are stored as actor.
`excerpts/<sha256 of the normalised text>.txt`; an embedding lives at - A changed pin, candidate set or image finds nothing and recomputes.
`embeddings/<model>/<excerpt sha>.json` and "an excerpt is embedded once; a rerun makes no - **Content-addressed memory** (`src/openultrasast/learn/`). Each item is stored under a hash
call" (`embeddings.py`). Model responses are cached under the sha256 of of its content.
`{model, messages, params, temperature, sample}` (`program.py`, `request_key()`), and a - Excerpts are stored as `excerpts/<sha256 of the normalised text>.txt`.
compiled program is `programs/<sha256 of its artifact>.json` with the digests of the memory - An embedding lives at `embeddings/<model>/<excerpt sha>.json`. The source says: "an excerpt
snapshot and of the response cache it was built from (`compile.py`). is embedded once; a rerun makes no call" (`embeddings.py`).
- **Replay is free.** With no client, the response cache replays at $0 and a miss raises - Model responses are cached under the sha256 of `{model, messages, params, temperature, sample}`
(`program.py`, `request_key()`).
- A compiled program is `programs/<sha256 of its artifact>.json`. It records the digests of
the memory snapshot and of the response cache it was built from (`compile.py`).
- **Replay is free.** With no client, the response cache replays at $0. A miss raises
`ReplayMiss` instead of calling. The injection-slice record states it: "evaluate re-run `ReplayMiss` instead of calling. The injection-slice record states it: "evaluate re-run
replay-only on the committed code: 550 responses replayed, $0, identical overall and replay-only on the committed code: 550 responses replayed, $0, identical overall and
calibration" (`benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`, calibration" (`benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`,
@@ -84,26 +113,35 @@ or a file.
### Cheap before expensive ### Cheap before expensive
- **Triage first.** Verification runs only over the candidates the recorded triage kept; the - **Triage first.** Verification runs only over the candidates the recorded triage kept.
plane applies that recording without a model call and prices it as a separate line - The plane applies that recording without a model call.
(`recorded_triage_usd`), so every per-candidate cost below is reported both with and without - It prices the recording as a separate line (`recorded_triage_usd`).
it (`src/openultrasast/plane/tasks/agree.py`). - So every per-candidate cost below is reported both with and without it
(`src/openultrasast/plane/tasks/agree.py`).
- **Hunts are batched per file.** `verify` runs one tool hunt per file for up to `PER_HUNT = 6` - **Hunts are batched per file.** `verify` runs one tool hunt per file for up to `PER_HUNT = 6`
candidates, with the candidates' known callers (capped at 8) in the prompt candidates. The prompt includes the candidates' known callers, capped at 8
(`src/openultrasast/plane/tasks/verify.py`, `units_of()`). (`src/openultrasast/plane/tasks/verify.py`, `units_of()`).
- **The tie-break pass runs only on disputes.** `agree` compares passes a and b; pass c is - **The tie-break pass runs only on disputes.**
started with `inputs.only` set to the case's `disputed.json`, an empty list ends it `done` - `agree` compares passes a and b.
with no model call, and the final verdict is 2-of-3 ("pass c alone never decides", - Pass c starts with `inputs.only` set to the case's `disputed.json`.
`agree.py`). - An empty list ends it `done` with no model call.
- **Cache-friendly prompt order.** The decision engine's prompt is the compiled prefix - The final verdict is 2-of-3 ("pass c alone never decides", `agree.py`).
(instruction and fixed demonstrations, identical for every candidate so the provider's - **Cache-friendly prompt order.** The decision engine's prompt has three parts, in this order
prefix cache hits), then the retrieved examples, then the candidate (`learn/program.py`). (`learn/program.py`):
Its self-consistency `k` is tied to that cache: "k = 5 when samples 2..k hit the prefix cache 1. The compiled prefix: instruction and fixed demonstrations. It is identical for every
for >= 0.8 of their prompt tokens"; the measured repeat-hit share was 0.96 (injection-slice candidate, so the provider's prefix cache hits.
record, `smoke.k_rule`, `smoke.k5_repeat_hit_share_measured`). The verify hunts share a prefix 2. The retrieved examples.
too: in the first plane increment 2,385,152 of 3,820,395 prompt tokens were cache hits 3. The candidate.
(`benchmarks/independent/plane-increment-1.json`, `detail.cache_hit_tokens`,
`detail.prompt_tokens`; its reading: "Cost fell 17% (62% of prompt tokens were cache hits)"). The self-consistency `k` (the number of samples per candidate) is tied to that cache: "k = 5
when samples 2..k hit the prefix cache for >= 0.8 of their prompt tokens". The measured
repeat-hit share was 0.96 (injection-slice record, `smoke.k_rule`,
`smoke.k5_repeat_hit_share_measured`).
The verify hunts share a prefix too. In the first plane increment, 2,385,152 of 3,820,395
prompt tokens were cache hits (`benchmarks/independent/plane-increment-1.json`,
`detail.cache_hit_tokens`, `detail.prompt_tokens`). The record's reading: "Cost fell 17% (62%
of prompt tokens were cache hits)".
### Measured costs ### Measured costs
@@ -116,37 +154,41 @@ or a file.
| Decision engine harvest: `verify` part | $2.5373 over 1,961 calls, 474 tasks done, 10,921,167 prompt tokens of which 6,275,060 cache hits, 306,941 output tokens, task budgets summing to $8.624 under a $10 ceiling (`verify`) | `benchmarks/measurements/2026-10-01-decision-engine-harvest/record.json` | | Decision engine harvest: `verify` part | $2.5373 over 1,961 calls, 474 tasks done, 10,921,167 prompt tokens of which 6,275,060 cache hits, 306,941 output tokens, task budgets summing to $8.624 under a $10 ceiling (`verify`) | `benchmarks/measurements/2026-10-01-decision-engine-harvest/record.json` |
| Decision engine harvest: `roles` part | $1.4036 over 1,034 calls, 151 tasks done, 2,398,669 prompt tokens of which 329,600 cache hits, 370,155 output tokens, task budgets summing to $4.917 under a $10 ceiling (`roles`) | same record | | Decision engine harvest: `roles` part | $1.4036 over 1,034 calls, 151 tasks done, 2,398,669 prompt tokens of which 329,600 cache hits, 370,155 output tokens, task budgets summing to $4.917 under a $10 ceiling (`roles`) | same record |
The increment-2 record's note on the two denominators: "the reference divides by the 46 The increment-2 record notes that the two denominators differ: "the reference divides by the 46
candidates before triage; on that basis the plane is 7% cheaper, on the 43 after triage it is candidates before triage; on that basis the plane is 7% cheaper, on the 43 after triage it is
0.2% under". 0.2% under".
!!! note "The meter prices at list; the account moved less" !!! note "The meter prices at list; the account moved less"
The attribution table prices every call at the Model's list price The attribution table prices every call at the Model's list price
(`plane/models/deepseek-flash.yaml`). The harvest record: "Spend is the plane's attribution (`plane/models/deepseek-flash.yaml`). The harvest record says: "Spend is the plane's
table (`ousast plane status`, task meters priced at plane/models/deepseek-flash.yaml); the attribution table (`ousast plane status`, task meters priced at
DeepSeek account moved less (balance below)", with the balance at $19.63 before, $18.75 plane/models/deepseek-flash.yaml); the DeepSeek account moved less (balance below)".
after `verify` and $18.23 after `roles`. The injection-slice record names the ratio: "the The balance was $19.63 before, $18.75 after `verify` and $18.23 after `roles`.
provider charged ~1/3 of the list-price meter (balance moved $0.30 for $0.91 metered): The injection-slice record names the ratio: "the provider charged ~1/3 of the list-price
DeepSeek's off-peak discount window, presumably; the meter prices at list" meter (balance moved $0.30 for $0.91 metered): DeepSeek's off-peak discount window,
(`spend.balance_note`, balance $18.2 to $17.9). Budgets and gates are set against the presumably; the meter prices at list" (`spend.balance_note`, balance $18.2 to $17.9).
meter, never against the balance. Budgets and gates are set against the meter, never against the balance.
### Not done yet ### Not done yet
- **No model-side token budget in ax itself.** The plane's requirements state what ax does - **No model-side token budget in ax itself.**
not yet carry: "task dependencies, artifacts, token budgets (on ax's roadmap), a DeepSeek - The plane's requirements state what ax does not yet carry: "task dependencies, artifacts,
provider"; the thin `Run` layer carries them and dissolves into ax as ax gains them. Until token budgets (on ax's roadmap), a DeepSeek provider".
then the ceiling lives in the metered client, not in the executor. - The thin `Run` layer carries them. It dissolves into ax as ax gains them.
- **Triage is applied, not run.** The plane has no triage task; it filters by the recorded - Until then the ceiling lives in the metered client, not in the executor.
triage and adds that recording's cost as its own line. A triage task is planned as a later - **Triage is applied, not run.**
specification. - The plane has no triage task.
- **`k` and the contrast examples are not knobs.** `k` is a compiled setting in {1, 3, 5} - It filters by the recorded triage and adds that recording's cost as its own line.
(`learn/compile.toml`, "k and lam remain A/B variants"), changed only through the decision - A triage task is planned as a later specification.
engine's A/B experiments, its requirement that "changes to prompts, models, pass counts, rule - **`k` and the contrast examples are not knobs.**
sets and features [are] compared by controlled experiments, so that nothing is adopted on one - `k` is a compiled setting in {1, 3, 5} (`learn/compile.toml`, "k and lam remain A/B
run's number" (Requirement 4 of its specification). The retrieval cap of at most two variants").
contrast examples from other families is a module constant (`learn/retrieve.py`, - It changes only through the decision engine's A/B experiments. Requirement 4 of its
`CONTRAST = 2`), a maintainer decision, not an experiment variant yet. specification asks that "changes to prompts, models, pass counts, rule sets and features
[are] compared by controlled experiments, so that nothing is adopted on one run's number".
- Retrieval adds at most two contrast examples from other families. This cap is a module
constant (`learn/retrieve.py`, `CONTRAST = 2`).
- The cap is a maintainer decision, not an experiment variant yet.
## B. Development: the maintainers' own sessions ## B. Development: the maintainers' own sessions
@@ -165,28 +207,47 @@ flowchart LR
### The measurement: tool I/O dominates ### The measurement: tool I/O dominates
`benchmarks/dev/token_report.py` reads a Claude Code session transcript (`.jsonl`; without an `benchmarks/dev/token_report.py` reads a Claude Code session transcript (`.jsonl`). Without an
argument, the newest under `~/.claude/projects/`) and reports, per content kind, characters and argument it reads the newest one under `~/.claude/projects/`.
share: `tool_use_input` (the tool call's input as JSON), `tool_result`, `thinking`,
`assistant:text`, `user:text`, plus the tool-call count and the top five tools. Tokens are It reports characters and share per content kind:
approximated at four characters each. It prints to stdout and writes nothing; what gets
committed is the counts-only summary. The one in `benchmarks/independent/plane-increment-1.json` - `tool_use_input` (the tool call's input as JSON);
(`dev_token_report`, for "the whole development session (c09e30b2), not the increment alone"): - `tool_result`;
7.3M unique characters (~1.81M tokens), `tool_use_input_pct` 45.4, `tool_result_pct` 40.1, - `thinking`;
`tool_io_pct` 85.5 against `previous_tool_io_pct` 90, over 2,701 tool calls. The reasoning the - `assistant:text`;
session is paid for (assistant text and thinking) is the small remainder. That is why the - `user:text`.
remedies below target file contents, not prose.
It also reports the tool-call count and the top five tools. Tokens are approximated at four
characters each. It prints to stdout and writes nothing. Only the counts-only summary is
committed.
The committed summary is in `benchmarks/independent/plane-increment-1.json`
(`dev_token_report`). It covers "the whole development session (c09e30b2), not the increment
alone". It counts 7.3M unique characters (~1.81M tokens). The shares
are `tool_use_input_pct` 45.4, `tool_result_pct` 40.1, and `tool_io_pct` 85.5 against
`previous_tool_io_pct` 90, over 2,701 tool calls.
The reasoning the session is paid for (assistant text and thinking) is the small remainder.
That is why the remedies below target file contents, not prose.
### The remedies in `benchmarks/dev/` ### The remedies in `benchmarks/dev/`
**`read_guard.py`**, a `PreToolUse` hook for agentic coding sessions. It reads the hook event on **`read_guard.py`** is a `PreToolUse` hook (a script that runs before each tool call) for
stdin and, for the `Read` tool only, denies a whole-file read of a file longer than 350 lines agentic coding sessions.
(`LIMIT = 350`); a read with `offset` or `limit`, a missing path, an unreadable file or a file
within the limit passes. It always exits 0 ("never fails the tool"). A denial is one JSON line - It reads the hook event on stdin.
whose `permissionDecisionReason` tells the agent to read a range or to ask the bulk reader. The - For the `Read` tool only, it denies a whole-file read of a file longer than 350 lines
script's docstring records that it is wired in `.claude/settings.local.json` (a local, untracked (`LIMIT = 350`).
file) as `PreToolUse` with matcher `Read`; the setting has this shape, with the command - These pass: a read with `offset` or `limit`, a missing path, an unreadable file, or a file
repo-relative because hooks run from the project directory: within the limit.
- It always exits 0 ("never fails the tool").
- A denial is one JSON line. Its `permissionDecisionReason` tells the agent to read a range or
to ask the bulk reader.
The script's docstring records how it is wired. It sits in `.claude/settings.local.json` (a
local, untracked file) as `PreToolUse` with matcher `Read`. The setting has this shape. The
command is repo-relative because hooks run from the project directory:
```json ```json
{ {
@@ -203,37 +264,48 @@ repo-relative because hooks run from the project directory:
} }
``` ```
**`bulk_read.py`**, the portal pattern: a cheap model reads the files and answers one question, **`bulk_read.py`** follows the portal pattern. A cheap model reads the files and answers one
citing `path:line` for every claim; it reports what the files say and decides nothing. question. It cites `path:line` for every claim. It reports what the files say and decides
nothing.
```text ```text
.venv/bin/python benchmarks/dev/bulk_read.py "<question>" FILE [FILE ...] [--max-usd 0.05] .venv/bin/python benchmarks/dev/bulk_read.py "<question>" FILE [FILE ...] [--max-usd 0.05]
``` ```
Files go to the model in 400-line chunks with a `--- path (lines a-b)` header and numbered How it works:
lines; the client is the project's detector client (`deepseek-flash` through `DEEPSEEK_API_KEY`,
otherwise OpenRouter), each call has a 90 s timeout and no tools. `--max-usd` (default 0.05) is - Files go to the model in 400-line chunks, with a `--- path (lines a-b)` header and numbered
checked before every chunk; at the ceiling the answer ends with lines.
`[stopped at the $X ceiling before: <chunk header>]`. The answer goes to stdout, the summary - The client is the project's detector client: `deepseek-flash` through `DEEPSEEK_API_KEY`,
`[bulk_read: N chunk(s), $X.XXXX]` to stderr. Without a chat client it exits 2. otherwise OpenRouter.
- Each call has a 90 s timeout and no tools.
- `--max-usd` (default 0.05) is checked before every chunk. At the ceiling the answer ends with
`[stopped at the $X ceiling before: <chunk header>]`.
- The answer goes to stdout. The summary `[bulk_read: N chunk(s), $X.XXXX]` goes to stderr.
- Without a chat client it exits 2.
**`token_report.py`**: `python benchmarks/dev/token_report.py [TRANSCRIPT.jsonl]`, described **`token_report.py`**: `python benchmarks/dev/token_report.py [TRANSCRIPT.jsonl]`, described
above. above.
### Practices ### Practices
- **Scripts into files, not heredocs.** A heredoc is `tool_use_input` and its output - **Scripts into files, not heredocs.** A heredoc (a script typed inline in a shell command) is
`tool_result`; both are paid again at every later turn of the session. A script on disk is `tool_use_input`, and its output is `tool_result`. Both are paid again at every later turn of
read once, by the interpreter. the session. A script on disk is read once, by the interpreter.
- **Terse outputs.** Commands print counts, exit codes and the lines that matter; a harness - **Terse outputs.** Commands print counts, exit codes and the lines that matter. A harness
fails loudly with the exit code rather than printing a table that has to be read back. fails loudly with the exit code. It does not print a table that has to be read back.
- **Subagents with worktrees** for work whose output need not enter the main context: a - **Subagents with worktrees** for work whose output need not enter the main context.
subagent's reads and command output stay in its own context and only its conclusion comes - A subagent's reads and command output stay in its own context.
back; a git worktree keeps its edits off the main checkout. - Only its conclusion comes back.
- A git worktree (a separate checkout of the same repository) keeps its edits off the main
checkout.
- **Counts-only measurement records.** A record under `benchmarks/measurements/` or - **Counts-only measurement records.** A record under `benchmarks/measurements/` or
`benchmarks/independent/` holds numbers and their keys, never code, prompts or responses, so `benchmarks/independent/` holds numbers and their keys. It never holds code, prompts or
it can be quoted on a page like this one and reread by a model at a few hundred tokens. responses. So it can be quoted on a page like this one. A model can reread it at a few
hundred tokens.
Related: [The plane on ax](plane.md) for the Run and Task contract, [Memory](memory.md) for Related:
the store the reuse keys live in, [Evaluation](evaluation.md) for the gates these costs are
measured against. - [The plane on ax](plane.md) for the Run and Task contract;
- [Memory](memory.md) for the store the reuse keys live in;
- [Evaluation](evaluation.md) for the gates these costs are measured against.