Merge branch 'worktree-agent-af2e4a098d650d0a2'

This commit is contained in:
Marius Ciepluch
2026-10-02 21:03:41 +02:00
6 changed files with 916 additions and 591 deletions
+370 -231
View File
@@ -1,17 +1,21 @@
# Deployment
How the agentic plane is deployed today, and what moving it to a separate Kubernetes cluster
would take. State as of 2026-10-02 (v2.0.0). A scan needs none of this: `ousast scan` and
`ousast pre-push` run from the Python package (and the Docker image for Joern, see
[Engine and pre-push](ops/README.md)). The plane is the maintainers' surface for model work over
whole repositories, driven by `ousast plane`.
This page covers two things. First, how the agentic plane is deployed today. Second, what moving it to a
separate Kubernetes cluster would take. It describes the state as of 2026-10-02 (v2.0.0).
Two words are used strictly on this page. **Implemented** means the code on `main` does it and it
has run on the maintainer's host. **Planned** means it has not been done; the guidance is derived
from the code and the host's lessons, and the code changes it needs are named with their file paths.
A scan needs none of this. `ousast scan` and `ousast pre-push` run from the Python package. Joern runs
from the Docker image (see [Engine and pre-push](ops/README.md)). The plane is the maintainers' tool for
model work over whole repositories. `ousast plane` drives it.
This page uses two words strictly:
- **Implemented** means the code on `main` does it, and it has run on the maintainer's host.
- **Planned** means it has not been done. The guidance comes from the code and from what the host taught
us. Each code change it needs is named with its file path.
**No deployment of the plane to a cluster other than the maintainer's single-node kind cluster has
been made.** Section B is guidance, not a record.
been made.** Section B is guidance, not a record. (kind runs a Kubernetes cluster inside Docker
containers on one machine.)
## What runs where today (implemented)
@@ -51,14 +55,21 @@ flowchart LR
| Memory store | `OUSAST_MEMORY`: a file store under `~/ousast-results/plane/memory` by default, or `s3://<bucket>[/<prefix>]` on any reachable S3-compatible server (RustFS is the tested one) | `src/openultrasast/plane/memory.py`, [Memory](memory.md), [RustFS setup](rustfs.md) |
| Run state and artifacts | `$OUSAST_RESULTS/plane/<run>/` on the host (default `~/ousast-results/plane/`) | `reconciler.py` (`run_dir`) |
Only the reconciler talks to the memory store: a sandboxed task cannot read it, so the loop's
`memory-snapshot` is seeded by the reconciler before the Run ([The plane on ax](plane.md)).
Terms in the table:
- **ax** is Google's controller that runs agent tasks on Kubernetes.
- **Agent Substrate** is the layer below ax that owns the machines and runs sandboxes.
- **gVisor** is a sandbox runtime that isolates each actor from the node's kernel.
- An **actor** is one running Task inside a sandbox.
Only the reconciler talks to the memory store. A sandboxed task cannot read it. So the reconciler seeds
the loop's `memory-snapshot` before the Run starts ([The plane on ax](plane.md)).
## A. This host: the local setup (implemented)
What `ops/ax/up.sh` needs on the PATH: `docker`, `go`, `kubectl`, `kind`, `ko` and `ax` (it
refuses to start when one is missing). Tools live in `~/go/bin` and `~/.local/bin`; sources and
rendered files in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`).
`ops/ax/up.sh` needs these tools on the PATH: `docker`, `go`, `kubectl`, `kind`, `ko` and `ax`. It
refuses to start when one is missing. Tools live in `~/go/bin` and `~/.local/bin`. Sources and rendered
files live in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`).
1. **Bring up the cluster and the plane.**
@@ -66,15 +77,20 @@ rendered files in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`).
ops/ax/up.sh
```
Idempotent: each step is skipped when already done. It creates the kind cluster
(`KIND_CLUSTER_NAME`, default `ousast`) with its registry (`KO_DOCKER_REPO`, default
`localhost:5001`), installs Agent Substrate, applies the egress gateway (the agentgateway
variant, one prebuilt image) when the install left it out, deploys the ax control plane, applies
the receiver Service with the host's kind-gateway address as its endpoint, applies the gVisor
worker pool, builds and pushes the runner image and writes its digest pin to
`~/.cache/ousast/ax-src/runner-image`, installs the `ax` CLI from the deployed checkout (a release
CLI skews from the server), and runs one smoke Task. It ends by printing the host's memory and the
kind containers' usage.
The script is idempotent: it skips each step that is already done. In order, it:
- creates the kind cluster (`KIND_CLUSTER_NAME`, default `ousast`) with its registry
(`KO_DOCKER_REPO`, default `localhost:5001`);
- installs Agent Substrate;
- applies the egress gateway if the install left it out (the agentgateway variant, one prebuilt image);
- deploys the ax control plane;
- applies the receiver Service, with the host's kind-gateway address as its endpoint;
- applies the gVisor worker pool;
- builds and pushes the runner image, and writes its digest pin to `~/.cache/ousast/ax-src/runner-image`;
- installs the `ax` CLI from the deployed checkout (a release CLI does not match the server version);
- runs one smoke Task.
At the end it prints the host's memory and the kind containers' usage.
2. **Check it.**
@@ -82,9 +98,15 @@ rendered files in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`).
uv run ousast plane doctor
```
Four checks: the kind context is reachable, every pod in `ate-system` and in `ax-system` is ready,
and the registry at `localhost:5001` holds `ousast-runner`. Each failing line names
`ops/ax/up.sh` as the fix. `doctor` is informational: `ousast plane run` does not call it.
It runs four checks:
- the kind context is reachable;
- every pod in `ate-system` is ready;
- every pod in `ax-system` is ready;
- the registry at `localhost:5001` holds `ousast-runner`.
Each failing line names `ops/ax/up.sh` as the fix. `doctor` is informational only.
`ousast plane run` does not call it.
3. **Run the end-to-end smoke Run.**
@@ -92,23 +114,27 @@ rendered files in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`).
ops/ax/smoke-run.sh
```
It renders `ops/ax/smoke-run.yaml.tmpl` with the current runner digest and submits the Run
`ax-e2e-smoke` through the reconciler: one model-free `repo-facts` Task (`budget: {usd: 0,
calls: 0}`) over a files-only Workspace, resumed, started through the router, its `facts.json` and
`summary.json` delivered to the host, then `ousast plane status ax-e2e-smoke` prints the attribution
table. The script calls the repository's `.venv/bin/ousast`, so `uv sync` first.
It renders `ops/ax/smoke-run.yaml.tmpl` with the current runner digest. It then submits the Run
`ax-e2e-smoke` through the reconciler. The Run has one model-free `repo-facts` Task (`budget: {usd: 0,
calls: 0}`) over a files-only Workspace, resumed, started through the router, its `facts.json`
and `summary.json` delivered to the host. Then `ousast plane status ax-e2e-smoke`
prints the attribution table. The script calls the repository's `.venv/bin/ousast`, so run `uv sync`
first.
4. **Keys and the store.** Put `DEEPSEEK_API_KEY` (chat) and `OPENROUTER_API_KEY` (embeddings) in
`.env` or export them; a Task's Model names the variable (`secretKey.key`), and the key travels
only in that Task's start request. Set `OUSAST_MEMORY=s3://<bucket>` with `S3_ENDPOINT`,
`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` (optional `S3_REGION`, `S3_BUCKET`) for the S3 store;
the bucket must be set up once by an admin as [RustFS setup](rustfs.md) describes, and the store
verifies it at every start.
4. **Keys and the store.**
- Put `DEEPSEEK_API_KEY` (chat) and `OPENROUTER_API_KEY` (embeddings) in `.env`, or export them.
- A Task's Model names the variable (`secretKey.key`). The key travels only in that Task's start
request.
- For the S3 store, set `OUSAST_MEMORY=s3://<bucket>` with `S3_ENDPOINT`, `AWS_ACCESS_KEY_ID` and
`AWS_SECRET_ACCESS_KEY`. `S3_REGION` and `S3_BUCKET` are optional.
- An admin must set up the bucket once, as [RustFS setup](rustfs.md) describes. The store verifies
it at every start.
5. **Run a real Run**, as in [Examples](examples.md#7-running-the-agentic-plane): `ousast plane run
plane/runs/validation-46.yaml`, then `status` and `remember`.
6. **Tear down** with `ops/ax/down.sh`, which deletes the cluster and its registry.
6. **Tear down** with `ops/ax/down.sh`. It deletes the cluster and its registry.
**Footprint** (measured 2026-09-29, idle, two workers; [ax on this host](ops/ax/README.md#measured-footprint-2026-09-29-idle-two-workers)):
@@ -119,44 +145,62 @@ rendered files in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`).
| node volume (images, containerd) | 4.9 GB on disk |
| worker pool | 2 gVisor workers, 1 CPU / 1.5 GiB limit each |
Disk is the binding constraint: the Substrate install and the `ko` builds need several GB of Go
caches (`go clean -cache -modcache` afterwards), and every Task is its own actor template with a
golden actor (about 24 MB each under `/var/lib/ate/actors` on the node). The failures the host
produced, and what each one taught, are listed in
[ax on this host](ops/ax/README.md#what-this-host-taught-each-one-cost-a-failed-live-run).
Disk is the tightest limit, for two reasons:
- The Substrate install and the `ko` builds need several GB of Go caches. Run `go clean -cache -modcache`
afterwards.
- Every Task is its own actor template with a golden actor (a saved snapshot the actor boots from). Each
takes about 24 MB under `/var/lib/ate/actors` on the node.
[ax on this host](ops/ax/README.md#what-this-host-taught-each-one-cost-a-failed-live-run) lists the
failures the host produced and what each one taught.
## B. A separate Kubernetes cluster (planned)
The target: a cluster that runs Agent Substrate with its egress gateway and the ax control plane,
pulls the runner image from a registry of yours, receives task artifacts without a developer
laptop in the path, and keeps its memory in your S3 bucket. The provider key already travels only
in the start request through `atenet-router`, which works the same through a port-forward to any
cluster. Everything else below is either a step you take or a code change that is not made yet.
The target cluster would do four things:
- run Agent Substrate with its egress gateway and the ax control plane;
- pull the runner image from a registry of yours;
- receive task artifacts without a developer laptop in the path;
- keep its memory in your S3 bucket.
The provider key already travels only in the start request through `atenet-router`. That works the same
through a port-forward to any cluster. Everything else below is either a step you take or a code change
that is not made yet.
### B.1 Cluster with Agent Substrate, its egress gateway and ax
- Install Agent Substrate on the cluster following its own instructions. `ops/ax/up.sh` uses
Substrate's kind helpers (`hack/create-kind-cluster.sh`, `hack/install-ate-kind.sh`), which do
not apply to another cluster.
- Install Agent Substrate on the cluster with its own instructions. `ops/ax/up.sh` uses Substrate's kind
helpers (`hack/create-kind-cluster.sh`, `hack/install-ate-kind.sh`). These do not apply to another
cluster.
- Apply the egress gateway. On this host it is the agentgateway variant
(`manifests/ate-install/agentgateway-egress` in the Substrate checkout, applied with
`kubectl kustomize --load-restrictor=LoadRestrictionsNone`), which needs no Rust build; the
Envoy variant does. Without the gateway an actor has no network at all, and the per-task
EgressPolicies the reconciler writes (`src/openultrasast/plane/egress.py`) work unchanged
against the Substrate-installed gateway: deny by default, hostnames only, plain HTTP to the
receiver, TLS passthrough to the Workspaces' Git hosts and the Model's declared hosts.
(`manifests/ate-install/agentgateway-egress` in the Substrate checkout). It is applied with
`kubectl kustomize --load-restrictor=LoadRestrictionsNone`. It needs no Rust build; the Envoy variant
does.
- Without the gateway an actor has no network at all.
- The reconciler writes one EgressPolicy per task (`src/openultrasast/plane/egress.py`). These work
unchanged against the gateway that Substrate installs. The rules are:
- deny by default;
- hostnames only;
- plain HTTP to the receiver;
- TLS passthrough to the Workspaces' Git hosts and the Model's declared hosts.
- Deploy ax (`make deploy AX_IMAGE_REPO=<your registry>` in the ax checkout) **after** B.2.
- Apply a WorkerPool (B.6). Substrate installs none by itself ("no free workers").
### B.2 ax's snapshot bucket: `AX_SNAPSHOTS_BUCKET`
ax's deploy manifest (`deploy/ax-server.yaml` in the ax checkout) sets `AX_SNAPSHOTS_BUCKET` to
the ax authors' own bucket (`gs://dberkov-gke-dev3/ate-env/` in the checkout used here). Point it
at **your own bucket before deploying ax**. On this host the bucket named there did not exist in
the S3 server, deleted golden actors piled up in `DELETING` and filled the disk during a 600-task
harvest (2026-10-01); creating the bucket let deletes complete, and even then a deleted actor's
directory stays on the node, so long Runs need a janitor that removes directories no live actor
owns.
Point `AX_SNAPSHOTS_BUCKET` at **your own bucket before deploying ax**. ax's deploy manifest
(`deploy/ax-server.yaml` in the ax checkout) sets it to the ax authors' own bucket. In the checkout used
here that is `gs://dberkov-gke-dev3/ate-env/`.
This host shows what goes wrong otherwise (2026-10-01):
1. The bucket named there did not exist in the S3 server.
2. Deleted golden actors piled up in `DELETING`.
3. They filled the disk during a 600-task harvest.
Creating the bucket let deletes complete. Even then, a deleted actor's directory stays on the node. So
long Runs need a janitor that removes directories no live actor owns.
### B.3 A registry the workers can pull from, the runner image pinned by digest
@@ -168,83 +212,117 @@ owns.
docker inspect --format '{{range .RepoDigests}}{{println .}}{{end}}' <registry>/ousast-runner:dev
```
- Substrate rejects tags: every Task's `image` must be the `<registry>/ousast-runner@sha256:...`
reference. Write it into a file (on this host, `~/.cache/ousast/ax-src/runner-image`) and pass
`--runner-image FILE` to `ousast plane workspaces` and `ousast plane harvest`, which re-pin the
Tasks they generate. The committed templates under `plane/tasks/` and the generated
`plane/tasks/validation-46.yaml` carry `localhost:5001/ousast-runner@sha256:...` and need the same
re-pin (regenerate, or replace the reference) before they run elsewhere.
- Workers must be able to pull from the registry (credentials, if any, are the cluster's concern;
ax's default runner image on `gcr.io` needs credentials and is not used here).
- Substrate rejects tags. Every Task's `image` must be the `<registry>/ousast-runner@sha256:...`
reference.
- Write that reference into a file. On this host the file is `~/.cache/ousast/ax-src/runner-image`.
- Pass `--runner-image FILE` to `ousast plane workspaces` and `ousast plane harvest`. They re-pin the
Tasks they generate.
- The committed templates under `plane/tasks/` and the generated `plane/tasks/validation-46.yaml` carry
`localhost:5001/ousast-runner@sha256:...`. They need the same re-pin before they run elsewhere.
Regenerate them, or replace the reference.
- Workers must be able to pull from the registry. Credentials, if any, are the cluster's concern. ax's
default runner image on `gcr.io` needs credentials and is not used here.
- **Not yet configurable:** `ousast plane doctor` checks the registry at `localhost:5001` by name
(`src/openultrasast/plane/doctor.py:45`), so its fourth check fails on another cluster even when
the image is in place. `doctor` is informational, `plane run` does not depend on it.
(`src/openultrasast/plane/doctor.py:45`). So its fourth check fails on another cluster, even when the
image is in place. `doctor` is informational; `plane run` does not depend on it.
### B.4 One configurable kube context
The reconciler, the egress client and the router port-forward all use the context
`kind-$KIND_CLUSTER_NAME` (`kind-ousast` by default). **Not yet configurable**: the prefix `kind-`
is hard-coded in `src/openultrasast/plane/doctor.py:39`, `src/openultrasast/plane/egress.py:115`
and `:148`, and `src/openultrasast/plane/router.py:145`; `ops/ax/up.sh:24` builds the same name.
The planned change is one variable (for example `OUSAST_KUBE_CONTEXT`) read in those three
modules. Until then the only way to drive another cluster from the unchanged code is to give its
kubeconfig context a name of that form (`kubectl config rename-context <ctx> kind-<name>` and
`KIND_CLUSTER_NAME=<name>`), which is a workaround, not a supported configuration. The egress
client also needs `kubectl-ate` (or the executable `OUSAST_KUBECTL_ATE` names) able to use that
context.
The reconciler, the egress client and the router port-forward all use one kube context. Its name is
`kind-$KIND_CLUSTER_NAME` (`kind-ousast` by default).
**Not yet configurable**: the prefix `kind-` is hard-coded in these places:
- `src/openultrasast/plane/doctor.py:39`;
- `src/openultrasast/plane/egress.py:115` and `:148`;
- `src/openultrasast/plane/router.py:145`;
- `ops/ax/up.sh:24` builds the same name.
The planned change is one variable read in those three modules, for example `OUSAST_KUBE_CONTEXT`.
Until then, there is one workaround to drive another cluster with the unchanged code. Rename its
kubeconfig context to that form: `kubectl config rename-context <ctx> kind-<name>` and
`KIND_CLUSTER_NAME=<name>`. This is a workaround, not a supported configuration.
The egress client also needs `kubectl-ate` (or the executable `OUSAST_KUBECTL_ATE` names). It must be
able to use that context.
### B.5 The receiver: in-cluster instead of on the host
Today the receiver is a thread inside the reconciler process (`egress.py`, `Receiver`), and the
reconciler learns of a task's completion by watching that object's `delivered` set in memory
(`reconciler.py`). Actors reach it through the Service `ousast-receiver.ax-system` whose
EndpointSlice points at the host. Three paths, in order of effort:
Today the receiver is a thread inside the reconciler process (`egress.py`, `Receiver`). The reconciler
learns that a task finished by watching that object's `delivered` set in memory (`reconciler.py`).
Actors reach it through the Service `ousast-receiver.ax-system`. That Service's EndpointSlice points at
the host.
1. **Interim, works with the unchanged code:** keep the host receiver and render
`ops/ax/receiver-service.yaml.tmpl` with an endpoint address the egress gateway pod can reach
(`KIND_GATEWAY` is just an IPv4 address in the template, `OUSAST_ARTIFACT_PORT` the port). The
reconciler's host must be reachable from the cluster (a VPN or a routed address), and the
gateway dials the address the actor connected to, so the Service's ClusterIP must still map to it.
2. **Planned: the receiver as an in-cluster Deployment** behind the same Service name, with the
reconciler reading deliveries and serving inputs from it instead of from its own thread. This
needs code: the receiver and its `delivered`/`inputs` state are in-process today, and the
runner's `GET /inputs/<producer>/<artifact>` is served by that same process.
3. **Follow-on: presigned-URL delivery to the S3 store.** The runner would `PUT` its output tar to a
presigned URL on the memory store's bucket and fetch inputs the same way, and the reconciler
would read from the bucket. Nothing of this exists; the EgressPolicy would then have to allow
the store's hostname for the task.
There are three paths, from least to most effort:
1. **Interim, works with the unchanged code:** keep the host receiver.
Render `ops/ax/receiver-service.yaml.tmpl` with an endpoint address the egress gateway pod can reach.
In the template, `KIND_GATEWAY` is just an IPv4 address and `OUSAST_ARTIFACT_PORT` is the port.
The cluster must be able to reach the reconciler's host, through a VPN or a routed address.
The gateway dials the address the actor connected to. So the Service's ClusterIP must still map to it.
2. **Planned: the receiver as an in-cluster Deployment** behind the same Service name.
The reconciler would read deliveries from it and serve inputs from it, instead of from its own thread.
This needs code. The receiver and its `delivered`/`inputs` state are in-process today.
The same process serves the runner's `GET /inputs/<producer>/<artifact>`.
3. **Follow-on: presigned-URL delivery to the S3 store.** (A presigned URL grants one upload or download
without credentials.)
The runner would `PUT` its output tar to a presigned URL on the memory store's bucket.
It would fetch inputs the same way, and the reconciler would read from the bucket.
None of this exists. The EgressPolicy would then have to allow the store's hostname for the task.
### B.6 Worker pool sizing
`ops/ax/workerpool.yaml.tmpl` declares two warm gVisor workers of 1 CPU / 1.5 GiB (requests 250m /
1.5 GiB) for the 7 GB host, selected onto nodes by `ate.dev/substrate-version`. An actor occupies a
whole worker, so the pool size is the number of tasks that can run at once; set `--workers` on
`ousast plane run` to match. Size the workers above the largest Task's limits (`verify`: requests
250m / 384Mi, limits 1 CPU / 1 GiB; `roles` and the others are in `plane/tasks/`), and keep the
golden-actor storage on the node in mind (B.2).
`ops/ax/workerpool.yaml.tmpl` declares two warm gVisor workers for the 7 GB host. Each has 1 CPU /
1.5 GiB (requests 250m / 1.5 GiB). They are placed on nodes by `ate.dev/substrate-version`.
An actor occupies a whole worker. So the pool size is the number of tasks that can run at once. Set
`--workers` on `ousast plane run` to match.
Size the workers above the largest Task's limits. For `verify` these are requests 250m / 384Mi and
limits 1 CPU / 1 GiB. `roles` and the others are in `plane/tasks/`. Also keep the golden-actor storage
on the node in mind (B.2).
### B.7 The S3 memory store reachable from where the reconciler runs
The `s3://` store already works against any reachable S3-compatible server with versioning,
lifecycle rules, object tags and S3 Select (RustFS is the tested one; the bucket setup and the
agent policy are on [RustFS setup](rustfs.md)). Only the reconciler reads and writes it, so today
it must be reachable from the operator's host; if the reconciler moves into the cluster, from its
pod. No Task talks to the store directly; the day one does (B.5, path 3), that Task's EgressPolicy
must allow the store's hostname (the policies allow hostnames only, never addresses), the way a
Model declares its hosts today through `openultrasast.io/egress-hosts`.
The `s3://` store already works against any reachable S3-compatible server. The server must support:
- versioning;
- lifecycle rules;
- object tags;
- S3 Select (a server-side query over JSON objects).
RustFS is the tested server. The bucket setup and the agent policy are on [RustFS setup](rustfs.md).
Only the reconciler reads and writes the store. So today the operator's host must reach it. If the
reconciler moves into the cluster, its pod must reach it.
No Task talks to the store directly. If one ever does (B.5, path 3), its EgressPolicy must allow the
store's hostname. The policies allow hostnames only, never addresses. A Model already declares its hosts
this way today, through `openultrasast.io/egress-hosts`.
### B.8 Secrets in Kubernetes Secrets, injected into the reconciler's environment
The reconciler reads the variable a Model's `secretKey.key` names (`DEEPSEEK_API_KEY`,
`OPENROUTER_API_KEY`) and the store's `S3_ENDPOINT`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`
(optional `AWS_SESSION_TOKEN`, `S3_REGION`, `S3_BUCKET`) from its own environment or `.env`. When
the reconciler runs in the cluster, keep them in Kubernetes Secrets and inject them into **the
reconciler's** pod environment (`envFrom` with `secretRef`, or `valueFrom.secretKeyRef` per
variable). Never put a value into a manifest: the Model manifests name a variable, the rendered
Task carries no key, and the key reaches an actor only in the start request, where the runner keeps
it in memory and redacts it from echoed stderr. The `secretKey.name` values in `plane/models/`
(`deepseek`, `openrouter`) are the names such Secrets would carry. Running the reconciler
in-cluster is itself planned: today it is a host process, and its state directory
The reconciler reads these variables from its own environment or `.env`:
- the variable a Model's `secretKey.key` names (`DEEPSEEK_API_KEY`, `OPENROUTER_API_KEY`);
- the store's `S3_ENDPOINT`, `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`;
- optionally `AWS_SESSION_TOKEN`, `S3_REGION` and `S3_BUCKET`.
When the reconciler runs in the cluster, keep these in Kubernetes Secrets. Inject them into **the
reconciler's** pod environment. Use `envFrom` with `secretRef`, or `valueFrom.secretKeyRef` per variable.
Never put a value into a manifest. The key stays out of every manifest:
- the Model manifests name a variable;
- the rendered Task carries no key;
- the key reaches an actor only in the start request;
- the runner keeps it in memory and redacts it from echoed stderr.
The `secretKey.name` values in `plane/models/` (`deepseek`, `openrouter`) are the names such Secrets
would carry.
Running the reconciler in-cluster is itself planned. Today it is a host process. Its state directory
(`OUSAST_RESULTS`) would need a persistent volume.
### What is implemented and what is planned
@@ -266,12 +344,20 @@ in-cluster is itself planned: today it is a host process, and its state director
## Production topology: ax on a Kubernetes cluster
The maintainer's questions, answered from the code and the upstream checkouts under
`~/.cache/ousast/ax-src/`: how is it hosted in ax, is there one container, how does ax relate to
Kubernetes, and what scaling the pre-push hook would take. The decisions of 2026-10-02 set the
target: **two profiles of one code base**, `kind` (this laptop, kept as a first-class local
execution mode, where every change is proven first) and `k3s` (production). Everything marked
**planned** is not built; the rest is what runs on this host today.
This section answers the maintainer's questions from the code and the upstream checkouts under
`~/.cache/ousast/ax-src/`:
- How is it hosted in ax?
- Is there one container?
- How does ax relate to Kubernetes?
- What would scaling the pre-push hook take?
The decisions of 2026-10-02 set the target: **two profiles of one code base**.
- `kind` is this laptop. It stays a first-class local execution mode. Every change is proven here first.
- `k3s` is production. (k3s is a small Kubernetes distribution.)
Everything marked **planned** is not built. The rest is what runs on this host today.
```mermaid
flowchart LR
@@ -308,61 +394,92 @@ flowchart LR
cli -- "polls: status, attribution" --> s3
```
No inbound path exists in this picture: the CLI reaches ax-server and the router from outside, the
tasks push their artifacts to the store through the egress gateway, and the CLI reads the store.
Today's EndpointSlice back to a laptop (`ops/ax/receiver-service.yaml.tmpl`) is the kind profile's
device and does not appear in production.
This picture has no inbound path:
- the CLI reaches ax-server and the router from outside;
- the tasks push their artifacts to the store through the egress gateway;
- the CLI reads the store.
Today's EndpointSlice back to a laptop (`ops/ax/receiver-service.yaml.tmpl`) belongs to the kind
profile only. It does not appear in production.
### ax, Kubernetes and Agent Substrate in three sentences
**ax is a Kubernetes application**: a Deployment `ax-server` with a Redis beside it in the
namespace `ax-system` (`ax/deploy/ax-server.yaml` in the ax checkout; its controller runs with
`--template=default-template --template-atespace=ax-system`), and the `ax` CLI reaches it through a
`kubectl port-forward` to `svc/ax-server` (`ax/internal/tunnel/tunnel.go`), so `ousast plane run`
needs a kube context or an ingress, not a public endpoint. **ax does not run containers itself**:
it turns every Task into an ActorTemplate on Agent Substrate (`ax/internal/substrate/client.go`,
`CreateActorTemplate`), built from the Task's image and environment, and the image must be pinned
by digest (Substrate's `SandboxConfig` validation rejects tags: "All images must include a
digest"). **Agent Substrate is the layer that owns the machines**: in the namespace `ate-system`
it runs `ate-api-server`, `ate-controller`, the `atelet` DaemonSet, `atenet-router`, the
`atenet-egress` gateway, a Postgres and a RustFS (`substrate/manifests/ate-install/`), and three
CRDs under `ate.dev/v1alpha1`, `WorkerPool`, `SandboxConfig` and `CSIDriverConfig`; a
`WorkerPool` keeps warm worker pods (`ko://.../cmd/ateom-gvisor`), and an actor is one gVisor
sandbox inside one of those workers, booted from the golden snapshot that
[The plane on ax](plane.md) describes, suspended and resumed by the controller. Names are limited
to 63 bytes (`MaxNameLength` in `ax/pkg/apis/v1alpha1/types.go`), which is why the generated Task
names are short.
**ax is a Kubernetes application.**
**What k3s changes** (planned; nothing has been applied to a k3s cluster): k3s runs containerd
with no Docker socket, so the host-Docker path of `ousast plane alerts-engine` does not exist
there and the engine must become a task (below); Substrate's gVisor workers need `runsc` and a
`RuntimeClass` on the nodes they are scheduled to; k3s ships Traefik (the ingress the CLI can use
instead of a kubeconfig) and local-path storage. Images are built by GitHub Actions from tagged
commits, published to GitHub Container Registry and digest-pinned in the manifests; a private
package needs an `imagePullSecret` on the worker ServiceAccount. Substrate's and ax's own images
are built with `ko` from their checkouts and pushed to the same registry. The kind profile keeps
`localhost:5001` and `ops/ax/up.sh`.
- It is a Deployment `ax-server` with a Redis beside it, in the namespace `ax-system`
(`ax/deploy/ax-server.yaml` in the ax checkout).
- Its controller runs with `--template=default-template --template-atespace=ax-system`.
- The `ax` CLI reaches it through a `kubectl port-forward` to `svc/ax-server`
(`ax/internal/tunnel/tunnel.go`).
- So `ousast plane run` needs a kube context or an ingress, not a public endpoint.
**ax does not run containers itself.**
- It turns every Task into an ActorTemplate on Agent Substrate (`ax/internal/substrate/client.go`,
`CreateActorTemplate`).
- The ActorTemplate is built from the Task's image and environment.
- The image must be pinned by digest. Substrate's `SandboxConfig` validation rejects tags: "All images
must include a digest".
**Agent Substrate is the layer that owns the machines.**
- In the namespace `ate-system` it runs `ate-api-server`, `ate-controller`, the `atelet` DaemonSet,
`atenet-router`, the `atenet-egress` gateway, a Postgres and a RustFS
(`substrate/manifests/ate-install/`).
- It defines three CRDs (custom Kubernetes resource types) under `ate.dev/v1alpha1`: `WorkerPool`,
`SandboxConfig` and `CSIDriverConfig`.
- A `WorkerPool` keeps warm worker pods (`ko://.../cmd/ateom-gvisor`).
- An actor is one gVisor sandbox inside one of those workers.
- The actor boots from the golden snapshot that [The plane on ax](plane.md) describes. The controller
suspends and resumes it.
- Names are limited to 63 bytes (`MaxNameLength` in `ax/pkg/apis/v1alpha1/types.go`). That is why the
generated Task names are short.
**What k3s changes** (planned; nothing has been applied to a k3s cluster):
- k3s runs containerd with no Docker socket. So the host-Docker path of `ousast plane alerts-engine`
does not exist there. The engine must become a task (below).
- Substrate's gVisor workers need `runsc` and a `RuntimeClass` on the nodes they run on.
- k3s ships Traefik and local-path storage. Traefik is an ingress the CLI can use instead of a kubeconfig.
- GitHub Actions builds the images from tagged commits. They are published to GitHub Container Registry
and digest-pinned in the manifests.
- A private package needs an `imagePullSecret` on the worker ServiceAccount.
- Substrate's and ax's own images are built with `ko` from their checkouts. They are pushed to the same
registry.
- The kind profile keeps `localhost:5001` and `ops/ax/up.sh`.
### One image, and what concurrency equals
There is **one container image** for every Task: `plane/Dockerfile.runner` (`python:3.12-slim`
plus this package, PID 1 `/usr/local/bin/ax-task-runner`, which is
`openultrasast.plane.runner`). Each Task manifest under `plane/tasks/` names the same image by
digest (the pin is kept in `~/.cache/ousast/ax-src/runner-image`) and selects its work with
`spec.command`: the runner starts `python -m openultrasast.plane.tasks.<command[0]>`
(`runner.py`, `task_module`). `repo-facts`, `verify`, `agree`, `features`, `remember`, `alerts`,
`loop` and `roles` are modules in that image, not images of their own; the engine would be the
second image (planned). So the production answer to "how many containers" is: one image, one
actor per Task, and **an actor occupies a whole worker** (its limits are the ActorTemplate's, the
worker holds one actor at a time). Concurrency is therefore the number of workers in the
`WorkerPool`, nothing else: `ops/ax/workerpool.yaml.tmpl` declares `replicas: 2` of 1 CPU /
1.5 GiB, so at most two Tasks run at once on this host, and `ousast plane run --workers` must not
exceed the pool. Substrate ships an autoscaled pool
(`substrate/demos/autoscaled-workerpool/README.md`: an HPA on the external metric
`ate_workerpool_workers{state=at_capacity}` through prometheus-adapter, writing
`WorkerPool.spec.replicas`; its README says the demo is kind-only today) and request parking
(`substrate/demos/parking/`: the router holds a resume until a worker frees up). Neither is
applied by `ops/ax/up.sh`; the production profile's pool manifests and autoscaling rule are
There is **one container image** for every Task: `plane/Dockerfile.runner`. It is `python:3.12-slim`
plus this package. PID 1 is `/usr/local/bin/ax-task-runner`, which is `openultrasast.plane.runner`.
Each Task manifest under `plane/tasks/` names the same image by digest. The pin is kept in
`~/.cache/ousast/ax-src/runner-image`. The manifest picks its work with `spec.command`. The runner
starts `python -m openultrasast.plane.tasks.<command[0]>` (`runner.py`, `task_module`).
`repo-facts`, `verify`, `agree`, `features`, `remember`, `alerts`, `loop` and `roles` are modules in
that image. They are not images of their own. The engine would be the second image (planned).
So the production answer to "how many containers" is:
- one image;
- one actor per Task;
- **an actor occupies a whole worker**. Its limits are the ActorTemplate's, and the worker holds one
actor at a time.
Concurrency is therefore the number of workers in the `WorkerPool`, nothing else.
`ops/ax/workerpool.yaml.tmpl` declares `replicas: 2` of 1 CPU / 1.5 GiB. So at most two Tasks run at
once on this host. `ousast plane run --workers` must not exceed the pool.
Substrate ships two scaling aids:
- **An autoscaled pool** (`substrate/demos/autoscaled-workerpool/README.md`). An HPA (Horizontal Pod
Autoscaler) reads the external metric `ate_workerpool_workers{state=at_capacity}` through
prometheus-adapter. It writes `WorkerPool.spec.replicas`. Its README says the demo is kind-only today.
- **Request parking** (`substrate/demos/parking/`). The router holds a resume until a worker frees up.
`ops/ax/up.sh` applies neither. The production profile's pool manifests and autoscaling rule are
**planned**.
### The three components that are bound to the host today
@@ -373,59 +490,75 @@ applied by `ops/ax/up.sh`; the production profile's pool manifests and autoscali
| Artifact receiver (`plane/egress.py`, the `Receiver` thread inside the reconciler) | reached through the Service `ousast-receiver.ax-system` whose EndpointSlice points at the kind gateway address on the host (`ops/ax/receiver-service.yaml.tmpl`) | **no receiver in remote mode.** Tasks deliver their artifacts and fetch their inputs through presigned URLs on the S3 memory store (B.5, path 3; the store's hostname enters each Task's EgressPolicy, B.7), and the CLI reads completion from the store. The kind profile keeps the receiver thread; an in-cluster receiver Deployment (B.5, path 2) is not the chosen direction |
| Joern engine (`plane/engine_alerts.py`, the host's `openultrasast:dev` container, run with `--memory 3g`) | `ousast plane alerts-engine` runs it on the host, one container at a time, and marks the Run's `alerts` tasks done | **an `engine` task on a second, larger `WorkerPool`** (the engine does not fit a 1.5 GiB worker, and k3s has no Docker socket), from an engine image digest-pinned in GHCR, emitting the same `alerts.jsonl` rows and coverage; `alerts-engine` on the host stays as the development path |
The S3 memory store (RustFS tested) is already external; in remote mode it is also the artifact
path, so it must be reachable from the CLI's host and, through the egress gateway, from the
workers.
The S3 memory store (RustFS tested) is already external. In remote mode it is also the artifact path.
So the CLI's host must reach it, and the workers must reach it through the egress gateway.
### Scaling the pre-push hook in production
**Today** `ousast pre-push` is a local, deterministic check: it materialises base and head,
discovers the changed regions, runs the evidence ranker and arbiter on head and compares with base,
all within `--deadline` (30 s by default) and in `--mode advisory` unless blocking is opted into
([Pre-push: the delta check](scanning.md#pre-push-the-delta-check-experimental),
`src/openultrasast/push/`, [Engine and pre-push](ops/README.md#experimental-pre-push-integration)).
**The plane is not in that path**: no hook submits a Run, no model is called without an explicit
`--model-config`, and nothing leaves the developer's machine.
**Today** `ousast pre-push` is a local, deterministic check. It does four things:
1. materialises base and head;
2. finds the changed regions;
3. runs the evidence ranker and arbiter on head;
4. compares the result with base.
All of this runs within `--deadline` (30 s by default). It runs in `--mode advisory` unless you opt
into blocking. See [Pre-push: the delta check](scanning.md#pre-push-the-delta-check-experimental),
`src/openultrasast/push/` and [Engine and pre-push](ops/README.md#experimental-pre-push-integration).
**The plane is not in that path.**
- No hook submits a Run.
- No model is called without an explicit `--model-config`.
- Nothing leaves the developer's machine.
**The production shape** (planned, none of it built) splits the work by latency:
1. **Blocking, inside the hook's deadline**: what runs today, the quick rules and the delta
engine on the changed regions. It stays local and deterministic, so the push is never held for
a model or a cluster.
2. **Asynchronous, on the plane**: the hook or the CI job calls the CLI in remote mode with a
*delta Run* whose candidates are the functions the push changed; the Run is the per-case chain
of [The plane on ax](plane.md#per-case) (`facts`, `verify` a and b, `agree`, `verify` c on the
disputed only, `final`). The caller either waits, with its own deadline, for the CLI to print
the status and attribution table, or returns at once and a later `ousast plane status <run>`
reports from the store; a commit status check or a review comment would be built on that. The
worker pool is sized to the push rate or autoscaled on `at_capacity` workers. An unattended
in-cluster reconciler that accepts submissions without a CLI session is the later option named
above.
1. **Blocking, inside the hook's deadline**: what runs today, the quick rules and the delta engine on
the changed regions. It stays local and deterministic. So the push never waits for a model or a
cluster.
2. **Asynchronous, on the plane**: the hook or the CI job calls the CLI in remote mode with a *delta
Run*.
- The Run's candidates are the functions the push changed.
- The Run is the per-case chain of [The plane on ax](plane.md#per-case): `facts`, `verify` a and b,
`agree`, `verify` c on the disputed only, `final`.
- The caller can wait, with its own deadline, for the CLI to print the status and attribution table.
- Or the caller returns at once, and a later `ousast plane status <run>` reports from the store.
- A commit status check or a review comment would be built on that.
- The worker pool is sized to the push rate, or autoscaled on `at_capacity` workers.
- An unattended in-cluster reconciler that accepts submissions without a CLI session is the later
option named above.
**Cost per push, estimated from measured rates.** Two measured per-candidate rates exist for one
model verdict: `$0.003215` (`spend.cost_per_candidate_evaluate_usd` in
`benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`) and `$0.0021`
(exp-002 arm A, `overall.usd_per_candidate.A` in
`benchmarks/measurements/2026-10-02-exp-002/record.json`). A push that changes five functions needs
two verify passes per candidate: 5 x 2 x $0.0021 = $0.021, or 5 x 2 x $0.003215 = $0.032 at the
higher rate, plus a third pass on the disputed candidates only. That is where "about two cents per
push" comes from; it is arithmetic over the decision engine's measured rates, not a measurement of
a push Run, and `ousast plane status` would print the real figure per Task.
**Cost per push, estimated from measured rates.** Two measured per-candidate rates exist for one model
verdict:
- `$0.003215` (`spend.cost_per_candidate_evaluate_usd` in
`benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`);
- `$0.0021` (exp-002 arm A, `overall.usd_per_candidate.A` in
`benchmarks/measurements/2026-10-02-exp-002/record.json`).
A push that changes five functions needs two verify passes per candidate. That gives 5 x 2 x $0.0021 =
$0.021, or 5 x 2 x $0.003215 = $0.032 at the higher rate. A third pass runs on the disputed candidates
only. That is where "about two cents per push" comes from. It is arithmetic over the decision engine's
measured rates, not a measurement of a push Run. `ousast plane status` would print the real figure per
Task.
**What is not built** (each is a named gap, not a configuration):
- The delta-Run command. `ousast plane scan <repo> --base <commit> --head <commit>` **does not
exist**; `ousast plane` has `run`, `status`, `doctor`, `remember`, `memory-normalise`,
exist**. `ousast plane` has `run`, `status`, `doctor`, `remember`, `memory-normalise`,
`workspaces`, `harvest` and `alerts-engine` (`src/openultrasast/cli.py`).
- A candidates task for an arbitrary repository. `repo-facts` takes its candidates from a
population's validation set; a step that derives them from a diff is not written.
population's validation set. No step derives them from a diff yet.
- Workspace generation from a URL and a commit outside a population. `plane/generate.py` is
population-driven (`read_cases` reads a validation set and a population TOML), so a Run for a
population-driven: `read_cases` reads a validation set and a population TOML. So a Run for a
repository that is not in a population cannot be generated today.
- The remote execution mode itself: the `local`/`remote` switch, configurable cluster context,
ax-server and router addresses (the `kind-` prefix is hard-coded, B.4), presigned delivery from
the runner to the store, and the CLI reading completion and `summary.json` from the store
instead of its receiver thread.
- The remote execution mode itself:
- the `local`/`remote` switch;
- a configurable cluster context and ax-server and router addresses (the `kind-` prefix is
hard-coded, B.4);
- presigned delivery from the runner to the store;
- the CLI reading completion and `summary.json` from the store instead of its receiver thread.
- The engine task and its WorkerPool, the GHCR build, the production pool and autoscaling
manifests, and the k3s runbook and checklist.
- The check or comment that carries the asynchronous verdict back to the push.
@@ -433,8 +566,7 @@ a push Run, and `ousast plane status` would print the real figure per Task.
### Sizing
The worker is the unit. Every Task's `limits` fit the 1 CPU / 1.5 GiB worker of
`ops/ax/workerpool.yaml.tmpl`; the requests are what the Task declares, the worker is what it
occupies.
`ops/ax/workerpool.yaml.tmpl`. The requests are what the Task declares. The worker is what it occupies.
| Task (`plane/tasks/*.yaml`) | Requests | Limits | Per push |
| --- | --- | --- | --- |
@@ -446,17 +578,18 @@ occupies.
| `roles` | 250m / 384Mi | 1 CPU / 1Gi | not in a push Run (harvest Runs only) |
| `engine` (planned) | from a measured run | above 3 GiB (the host run uses `--memory 3g`) | its own pool, not the default one |
- **Actors per push**: 4 (`facts`, `va`, `vb`, `agree`), 6 with a tie-break (`vc`, `final`);
- **Actors per push**: 4 (`facts`, `va`, `vb`, `agree`), or 6 with a tie-break (`vc`, `final`).
`features` and `remember` add two more when the verdict is to be kept.
- **Critical path**: `facts` then `va` and `vb` side by side, then `agree`, then (sometimes) `vc`
and `final`: three to five Task boots in series, each a golden-snapshot restore plus the model
calls of the verify passes.
- **One actor per worker**: `agree` asks for 256Mi and still holds a 1.5 GiB worker while it runs,
because the worker's capacity, not the Task's request, is the unit. A pool of `W` workers runs
at most `W/2` pushes through their verify phase at once; on this host (`W = 2`) that is one
push at a time, and every further push waits (the reconciler's `--workers` bound, or Substrate's
request parking if the pool is saturated). Autoscaling the pool on `at_capacity` workers is the
upstream answer and is **planned** here.
- **Critical path**: `facts`, then `va` and `vb` side by side, then `agree`, then sometimes `vc`
and `final`. That is three to five Task boots in series. Each boot is a golden-snapshot restore plus
the model calls of the verify passes.
- **One actor per worker**: `agree` asks for 256Mi but still holds a 1.5 GiB worker while it runs. The
worker's capacity, not the Task's request, is the unit.
- A pool of `W` workers runs at most `W/2` pushes through their verify phase at once.
- On this host (`W = 2`) that is one push at a time.
- Every further push waits, on the reconciler's `--workers` bound or on Substrate's request parking
if the pool is saturated.
- Autoscaling the pool on `at_capacity` workers is the upstream answer. It is **planned** here.
## C. The manifests under `plane/`
@@ -472,6 +605,12 @@ occupies.
| `ops/ax/smoke-task.yaml.tmpl`, `smoke-run.yaml.tmpl` | yes | rendered with `RUNNER_IMAGE` from your pin file |
| `ops/ax/up.sh`, `down.sh` | no | kind-specific; they do not apply to another cluster |
The Run and Task contract itself (`plane/runner.py` as PID 1 of the runner image, `AX_TASK_YAML`
and `AX_WORKSPACES_YAML`, `/healthz` and `/readyz`, the start request, the tar delivery as
completion) does not depend on where the cluster is: [The plane on ax](plane.md).
The Run and Task contract does not depend on where the cluster is. That contract covers:
- `plane/runner.py` as PID 1 of the runner image;
- `AX_TASK_YAML` and `AX_WORKSPACES_YAML`;
- `/healthz` and `/readyz`;
- the start request;
- the tar delivery as completion.
See [The plane on ax](plane.md).
+139 -85
View File
@@ -1,11 +1,12 @@
# Memory and detection
How what the system remembers is meant to turn into better decisions on code it has not seen.
This is the design of the learned decision engine (`src/openultrasast/learn/`), which is **in
development and not adopted**: no scan, pre-push check or report uses it today. Every user-facing
decision is still the deterministic one described in [Scanning](scanning.md) and
This page shows how the system's memory is meant to improve decisions on code it has not seen. It
describes the design of the learned decision engine (`src/openultrasast/learn/`).
The engine is **in development and not adopted**. No scan, pre-push check or report uses it today.
Every user-facing decision is still the deterministic one described in [Scanning](scanning.md) and
[Architecture](architecture.md). The engine's commands and status are on
[Decision engine](decision-engine.md); the store itself is on [Memory](memory.md).
[Decision engine](decision-engine.md). The store itself is on [Memory](memory.md).
## From instruments to a decision
@@ -25,89 +26,128 @@ flowchart TD
op -- "otherwise" --> adv["ADVISORY"]
```
The intended place for these decisions is changed code: `learn/features.py` builds feature records
for a whole-repository scan or for a base..head delta (`build_for_scan` with `Delta`), the same
delta `ousast pre-push` checks. Integration into scan and pre-push is still ahead.
These decisions are meant for changed code. `learn/features.py` builds feature records in two ways:
- for a whole-repository scan;
- for a base..head delta (`build_for_scan` with `Delta`). This is the same delta `ousast pre-push` checks.
Integration into scan and pre-push is still ahead.
## 1. Instruments produce signals, not verdicts
Each instrument contributes one block to a candidate's feature record (`learn/schema.py`,
`INSTRUMENTS`): `language`, `quick` (quick rules), `engine` (the Joern model layer), `facts` (repo
facts, such as callers and function span), `source`, `entry_points`, `roles` (model source, sink
and sanitizer roles), `delta`, `verify` and `agree` (the plane's verify passes and their
agreement), and `model_sinks` (the archived stage-1 classifier, `benchmarks/archive/model_sinks.py`, whose
logic now lives in `plane/tasks/roles.py`). A candidate is `(path, function, family)` of one repository and pin
where at least one instrument emitted a signal of that family (`learn/features.py`).
`INSTRUMENTS`). The blocks are:
An instrument that did not run, had no coverage or failed records that state (`none`, `failed`,
`not_applicable`), never a zero, so "silent" and "did not look" stay distinguishable. None of the
instruments decides alone.
- `language`;
- `quick` (quick rules);
- `engine` (the Joern model layer);
- `facts` (repo facts, such as callers and function span);
- `source`;
- `entry_points`;
- `roles` (model source, sink and sanitizer roles);
- `delta`;
- `verify` and `agree` (the plane's verify passes and their agreement);
- `model_sinks` (the archived stage-1 classifier, `benchmarks/archive/model_sinks.py`). Its logic now
lives in `plane/tasks/roles.py`.
**Why profile v1 withholds the engine and function length.** A leak audit
(`ousast learn audit-leaks`, `benchmarks/measurements/2026-10-01-decision-engine-leak-audit/`)
found that both separate the vulnerable and fixed sides of a pair by construction: a fix removes
a sink, and a guard lengthens the function. That holds on the pair corpus, not on unseen code, so
`INPUT_PROFILES["v1"]` drops the `engine` block and `facts.function_lines` from prompts and from
the retrieval distance.
A candidate is `(path, function, family)` of one repository and pin. At least one instrument must have
emitted a signal of that family there (`learn/features.py`).
An instrument may not run, may have no coverage, or may fail. It then records that state (`none`,
`failed`, `not_applicable`), never a zero. So "silent" and "did not look" stay distinguishable. No
instrument decides alone.
**Why profile v1 withholds the engine and function length.** A leak audit found that both separate the
vulnerable and fixed sides of a pair by construction (`ousast learn audit-leaks`,
`benchmarks/measurements/2026-10-01-decision-engine-leak-audit/`):
- a fix removes a sink;
- a guard lengthens the function.
That holds on the pair corpus, not on unseen code. So `INPUT_PROFILES["v1"]` drops the `engine` block
and `facts.function_lines` from prompts and from the retrieval distance.
## 2. Labels come from ground truth only
`ousast learn labels` (`learn/labels.py`) reads only the sources on a fail-closed allow-list
(`learn/sources.toml`): `population-v1`, `population-v2`, `dev-php` (reviewed recipe sites),
`pairs`, `advisory-fixes`, `adjudications` and `assumed_benign`. A file the list does not name is
refused before it is opened; the reserved population v3 is excluded unread.
(`learn/sources.toml`). The sources are `population-v1`, `population-v2`, `dev-php` (reviewed recipe
sites), `pairs`, `advisory-fixes`, `adjudications` and `assumed_benign`. A file the list does not name
is refused before it is opened. The reserved population v3 is excluded unread.
- **Positives** are declared sites and the vulnerable side of a fix; **negatives** are the same
function on the fixed side where the fix changed it, and recorded `False`/`FP` adjudications.
A vulnerable-pin candidate matching no site is unlabelled, never negative.
- **Assumed benign** (maintainer decision 2026-09-30): ordinary non-security commits of the training
repositories, at least 90 days from any fix, with no later security commit touching the same
files. They are their own source with weight 0.5, conditional on a signal at the tip and none at
the base, and reported separately, never merged with verified negatives.
- **Plane verdicts are features, not labels.** "Agreed", "rejected" or "disputed" is the output of
the verify instrument; it enters the record as the `verify` and `agree` blocks. Using it as a
label would teach the engine to reproduce the plane, mistakes included, so "rejected by the
plane" is never a negative (`learn/labels.py`: "A plane verdict ... is never a label").
- **Positives** are declared sites and the vulnerable side of a fix.
- **Negatives** are two kinds:
- the same function on the fixed side, where the fix changed it;
- recorded `False`/`FP` adjudications.
- A vulnerable-pin candidate matching no site is unlabelled, never negative.
- **Assumed benign** (maintainer decision 2026-09-30) covers ordinary non-security commits of the
training repositories. Each is at least 90 days from any fix. No later security commit touches the
same files.
- They are their own source with weight 0.5, conditional on a signal at the tip and none at the base.
- They are reported separately, never merged with verified negatives.
- **Plane verdicts are features, not labels.** "Agreed", "rejected" or "disputed" is the output of the
verify instrument. It enters the record as the `verify` and `agree` blocks. Using it as a label would
teach the engine to reproduce the plane, mistakes included. So "rejected by the plane" is never a
negative (`learn/labels.py`: "A plane verdict ... is never a label").
`ousast learn memory build` joins each label with the candidate's feature record and an excerpt of
its function and stores one `example` row (no rationale) plus the excerpt blob.
its function. It stores one `example` row (no rationale) plus the excerpt blob.
## 3. Retrieval of similar labelled cases
`learn/retrieve.py`, no model:
`learn/retrieve.py` uses no model. It works in three steps:
1. **Signal profile.** A Gower distance over the allow-listed features of the active profile,
per instrument block, with missing values explicit (a different state counts as distance 1).
The nearest 40 examples of the candidate's family and the nearest 40 of other families go on.
2. **Code embeddings.** A cosine re-rank of the excerpts' embeddings (OpenAI
`text-embedding-3-small` through OpenRouter, 1,536 dimensions, cached by excerpt hash):
`r = 0.5 (1 - D) + 0.5 cos`. Without the candidate's vector the retrieval is signals only.
3. **Balance.** Six examples: at most half per label, at most 2 per repository group, at most 2
contrast examples from other families.
1. **Signal profile.** It computes a Gower distance (a distance for mixed-type features) over the
allow-listed features of the active profile, per instrument block. Missing values are explicit: a
different state counts as distance 1. The nearest 40 examples of the candidate's family go on. So do
the nearest 40 of other families.
2. **Code embeddings.** It re-ranks by cosine over the excerpts' embeddings. The embeddings come from
OpenAI `text-embedding-3-small` through OpenRouter (1,536 dimensions, cached by excerpt hash). The
formula is `r = 0.5 (1 - D) + 0.5 cos`. Without the candidate's vector, the retrieval uses signals only.
3. **Balance.** It picks six examples:
- at most half per label;
- at most 2 per repository group;
- at most 2 contrast examples from other families.
**The evaluation boundary.** `eligible()` is the only way an example enters a prompt, for retrieved
examples and compiled demonstrations alike: never the candidate's own repository group, never a
group the fold evaluates, never the held-out source or framework, and (outside deployment) never
a near-duplicate of the candidate (cosine >= 0.98). The program repeats the check on the ids it
actually rendered before any call.
**The evaluation boundary.** `eligible()` is the only way an example enters a prompt. This holds for
retrieved examples and compiled demonstrations alike. An eligible example is never:
- from the candidate's own repository group;
- from a group the fold evaluates;
- from the held-out source or framework;
- a near-duplicate of the candidate (cosine >= 0.98), outside deployment.
The program repeats the check on the ids it actually rendered, before any call.
## 4. The AI classifier, compiled
`learn/program.py` and `learn/compile.py`, DSPy-style and in-house. The prompt is the instruction
plus the compiled demonstrations (a prefix identical for every candidate, so the provider's prefix
cache hits), then the retrieved examples, then the candidate's excerpt, signals, roles and family.
The model (`deepseek-flash`) answers `vulnerable`, `not_vulnerable` or `unsure` with a confidence
and a rationale that must cite a line of the code. It is sampled `k` times; the score `s` is the
mean over samples, and a majority `unsure` can never BLOCK. Compilation bootstraps demonstrations
and searches instructions on a compile split of repository groups, scored by a pre-registered
balanced Brier score (`learn/compile.toml`); the program is stored as `programs/<id>.json`.
The classifier lives in `learn/program.py` and `learn/compile.py`. It is DSPy-style (DSPy is a
framework that compiles prompts from examples) and built in-house.
The prompt has this order:
1. the instruction plus the compiled demonstrations. This prefix is identical for every candidate, so
the provider's prefix cache hits;
2. the retrieved examples;
3. the candidate's excerpt, signals, roles and family.
The model (`deepseek-flash`) answers `vulnerable`, `not_vulnerable` or `unsure`. It adds a confidence
and a rationale. The rationale must cite a line of the code. The model is sampled `k` times. The score
`s` is the mean over samples. A majority `unsure` can never BLOCK.
Compilation bootstraps demonstrations and searches instructions on a compile split of repository groups.
A pre-registered balanced Brier score scores it (`learn/compile.toml`). The program is stored as
`programs/<id>.json`.
## 5. Calibration and the two operating points
`learn/calibrate.py`: Platt scaling of `s`, cross-fitted so that no map is checked on the fold it
was fitted on. Calibration **holds** when the slope interval contains 1, the intercept interval
contains 0 and ECE <= 0.05. **BLOCK** is offered only when calibration holds and a fold reaches the
required precision; otherwise the engine offers **ADVISORY** only.
`learn/calibrate.py` applies Platt scaling (a logistic map from score to probability) to `s`. It is
cross-fitted, so no map is checked on the fold it was fitted on.
Calibration **holds** when the slope interval contains 1, the intercept interval contains 0 and ECE
(expected calibration error) <= 0.05.
**BLOCK** is offered only when calibration holds and a fold reaches the required precision. Otherwise
the engine offers **ADVISORY** only.
## The loop: memory improves decisions without retraining
@@ -125,25 +165,36 @@ flowchart LR
gate -- "accepted round" --> ledger["Rule status in the ledger"]
```
- **Decisions.** The compiled part of the program is its instruction and demonstrations; the
examples are retrieved from memory for each candidate. A new `example` row is therefore eligible
for retrieval in the next decision without recompiling the program, subject to the same
evaluation boundary. `ousast learn curve` measures the metric over memory size on a fixed
candidate subset.
- **Rules.** `ousast improve --memory` reads the run rows (alerts, verdicts, coverage) and proposes
rule-status edits only (shadow a rule that is repeatedly false across repositories; re-enable a
shadow rule that keeps hitting agreed declared sites). Rows from holdout pairs, the gated
manifest's own cases and any `--qualify-population` are dropped first, and every proposal goes
through the same validator and gate as any other edit (`improve/memory.py`).
- **Decisions.**
- The compiled part of the program is its instruction and demonstrations.
- The examples are retrieved from memory for each candidate.
- So a new `example` row is eligible for retrieval in the next decision without recompiling. The same
evaluation boundary applies.
- `ousast learn curve` measures the metric over memory size on a fixed candidate subset.
- **Rules.** `ousast improve --memory` reads the run rows (alerts, verdicts, coverage). It proposes
rule-status edits only:
- shadow a rule that is repeatedly false across repositories;
- re-enable a shadow rule that keeps hitting agreed declared sites.
Some rows are dropped first: rows from holdout pairs, the gated manifest's own cases and any
`--qualify-population`. Every proposal goes through the same validator and gate as any other edit
(`improve/memory.py`).
## What the first slice measured
Record: `benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json` (counts
only). Family **injection**; memory of 724 static examples from all families; input profile v1;
framework priors off; out-of-repository evaluation (compile split never evaluated, outer grouped
5-fold, calibration cross-fitted, operating points nested); 95% repository-cluster bootstrap
intervals. The slice's spend ($0.003215 per candidate at evaluation, k = 5, and its replay at
$0) is on [Token ergonomics](token-ergonomics.md#measured-costs).
only). The setup:
- family **injection**;
- memory of 724 static examples from all families;
- input profile v1;
- framework priors off;
- out-of-repository evaluation: compile split never evaluated, outer grouped 5-fold, calibration
cross-fitted, operating points nested;
- 95% repository-cluster bootstrap intervals.
The slice's spend is on [Token ergonomics](token-ergonomics.md#measured-costs). It was $0.003215 per
candidate at evaluation, k = 5, and its replay cost $0.
| Measure | Value | What it means |
| --- | --- | --- |
@@ -152,9 +203,12 @@ $0) is on [Token ergonomics](token-ergonomics.md#measured-costs).
| ADVISORY | recall 0.90 (0.82 - 0.97), precision 0.54 (0.44 - 0.65) | precision is at the positive share (0.53), so at this threshold ADVISORY is **not discriminating**: it flags nearly as indiscriminately as flagging everything would |
| BLOCK | not offered | calibration does not hold (ECE 0.064 > 0.05), and every outer fold reported `precision_unreachable` |
**The engine is not adopted.** A second slice (2026-10-02) evaluated the six evaluable families
the same way: pooled AUC 0.77 to 0.96 per family, within-pair AUC 0.72 to 0.93, BLOCK offered for
none, re-run agreement below 0.9 for three families
([Decision engine](decision-engine.md#second-slice-six-families-2026-10-02)). Leave-one-source-out
and leave-one-framework-out, integration into scan and pre-push, and the one-time check on
population v3 are still ahead.
**The engine is not adopted.** A second slice (2026-10-02) evaluated the six evaluable families the
same way ([Decision engine](decision-engine.md#second-slice-six-families-2026-10-02)). The results: pooled AUC 0.77 to 0.96 per family, within-pair AUC 0.72 to 0.93, BLOCK offered for
none. Re-run agreement was below 0.9 for three families.
Still ahead:
- leave-one-source-out and leave-one-framework-out;
- integration into scan and pre-push;
- the one-time check on population v3.
+50 -40
View File
@@ -1,8 +1,8 @@
# Memory
One store, keyed by repository and pin, holds what plane runs and the decision engine learned.
The code is `src/openultrasast/plane/memory.py`; the same layout serves both backends.
How the stored rows feed detection is on [Memory and detection](memory-and-detection.md).
One store holds what plane runs and the decision engine learned. It is keyed by repository and pin
(a fixed commit). The code is `src/openultrasast/plane/memory.py`. The same layout serves both backends.
[Memory and detection](memory-and-detection.md) shows how the stored rows feed detection.
## What is stored
@@ -17,7 +17,7 @@ programs/<program id>.json compiled decision-engine programs
```
**Rows.** Every row carries `id`, `kind`, `repo`, `pin`, `run`, `task`, `population`, `split`
and `image` (`ROW_FIELDS`); `id` is the sha256 of (kind, run, task, subject), so a repeated row
and `image` (`ROW_FIELDS`). `id` is the sha256 of (kind, run, task, subject). So a repeated row
replaces itself. The kinds (`KINDS`) and who writes them:
| Kind | Written by | Holds |
@@ -35,12 +35,15 @@ replaces itself. The kinds (`KINDS`) and who writes them:
| `experiment_result` | `ousast learn experiment analyse --record` | the paired estimates, the looks and the adoption verdict (counts and intervals only) |
| `label`, `decision` | reserved for later decision-engine tasks | declared in `KINDS`; not written by the code on main yet |
**Blobs** (`put_blob`) are content-addressed: a 64-hex name under a fixed prefix. Excerpts are
written by `learn memory build` (`learn/examples.py`), embeddings by `learn memory embed`
(`learn/embeddings.py`, keyed by excerpt sha so an excerpt is embedded once), responses by the
program's caller (`learn/program.py`, keyed by the sha256 of model, parameters digest, messages,
temperature, sample and format, so a rerun replays at no cost), and compiled programs by
`learn compile` (`learn/compile.py`).
**Blobs** (`put_blob`) are content-addressed: each has a 64-hex name under a fixed prefix.
Content-addressed means the name is the hash of the content. Who writes each blob:
- Excerpts: `learn memory build` (`learn/examples.py`).
- Embeddings: `learn memory embed` (`learn/embeddings.py`). They are keyed by excerpt sha, so an
excerpt is embedded once.
- Responses: the program's caller (`learn/program.py`). The key is the sha256 of model, parameters
digest, messages, temperature, sample and format. So a rerun replays at no cost.
- Compiled programs: `learn compile` (`learn/compile.py`).
## Write path
@@ -60,13 +63,14 @@ flowchart LR
put --> store[("Memory store")]
```
- `ingest` validates every row first: a malformed file fails before anything is written.
- Rows are merged into their repository + pin object by `id`. Every object is written whole and the
index last, so an interrupted ingest leaves nothing a repeat would not repair.
- `ingest` validates every row first. A malformed file fails before anything is written.
- Rows are merged by `id` into their repository + pin object.
- Every object is written whole, and the index last. So a repeat repairs anything an interrupted
ingest left behind.
- Each row object carries tags (`TAG_FIELDS`: `repo`, `pin`, `kind`, `family`, `run`,
`population`, `split`) listing the distinct values of its rows (`*` when they exceed 256
characters).
- The plane's store failures never change a run's result: an ingest can be repeated by hand
`population`, `split`). The tags list the distinct values of its rows. A tag is `*` when the
values exceed 256 characters.
- The plane's store failures never change a run's result. An ingest can be repeated by hand
(`cli.py`, `_plane_memory`).
## Read path
@@ -85,14 +89,14 @@ flowchart LR
rules --> gate{"Unchanged validator and gate"}
```
- **Seed** (`memory.seed`): a Run task whose `openultrasast.io/memory-key` matches stored facts gets
them and is marked done; a changed pin, candidate set or runner image finds nothing and
recomputes. The loop's `memory-snapshot` is seeded the same way, because a sandboxed task cannot
read the store.
- **Seed** (`memory.seed`).
- A Run task gets stored facts when its `openultrasast.io/memory-key` matches them. It is then marked done.
- A changed pin, candidate set or runner image finds nothing and recomputes.
- The loop's `memory-snapshot` is seeded the same way. A sandboxed task cannot read the store.
- **Retrieval** (`learn/retrieve.py`): see
[Memory and detection](memory-and-detection.md#3-retrieval-of-similar-labelled-cases).
- **`improve --memory`** (`improve/memory.py`): rows from holdout pairs, from the gated manifest's
own cases and from any `--qualify-population` are dropped before any rule sees them. See
- **`improve --memory`** (`improve/memory.py`). Some rows are dropped before any rule sees them:
rows from holdout pairs, from the gated manifest's own cases, and from any `--qualify-population`. See
[Architecture](architecture.md#proposals-from-plane-memory).
## Backends: FileStore and the S3 store
@@ -109,26 +113,32 @@ flowchart LR
### Verify at startup, no fallback
The S3 store never configures its bucket. Each time an `S3Store` is opened for a bucket and
prefix (once per process), `verify_bucket()` checks, using reads plus one small probe object at
`<prefix>/_probe/select.jsonl`:
The S3 store never configures its bucket. It only checks it. An `S3Store` is opened for a bucket and
prefix once per process. Each time, `verify_bucket()` runs these checks. It uses reads plus one small
probe object at `<prefix>/_probe/select.jsonl`.
1. versioning is `Enabled` (provenance cites object versions);
2. an enabled lifecycle rule expires `<prefix>/runs/` after a number of days, as a plain prefix
filter, and no enabled rule expires the whole store;
3. S3 Select answers a probe query over JSON Lines;
4. the probe's object tags are readable.
1. Versioning is `Enabled`. Provenance cites object versions.
2. An enabled lifecycle rule expires `<prefix>/runs/` after a number of days. It uses a plain prefix
filter. No enabled rule expires the whole store.
3. S3 Select (a server-side query over JSON objects) answers a probe query over JSON Lines.
4. The probe's object tags are readable.
Anything missing is named, with the admin command that fixes it, in one `MemoryStoreError`, and
the store does not start. There is no local or fetch-and-filter fallback: a filtered read that
S3 Select cannot answer raises a `MemoryStoreError` instead of returning "no rows". The exact
messages are listed on [RustFS setup](rustfs.md#what-the-store-verifies-at-startup).
If anything is missing, the store does not start. One `MemoryStoreError` names each problem and the
admin command that fixes it.
There is no local or fetch-and-filter fallback. A filtered read that S3 Select cannot answer raises a
`MemoryStoreError`. It does not return "no rows". [RustFS setup](rustfs.md#what-the-store-verifies-at-startup)
lists the exact messages.
### Queryable fields per kind
RustFS's Select infers an object's JSON schema from its leading rows, so a `where` field missing
there fails with `EvaluatorBindingDoesNotExist` even when a later row has it, and a column that is
null in the first 1000 rows breaks Select on the whole object (measured 2026-10-02; `memory.py`,
`S3Store._select`). The store therefore writes every row with its kind's full set of queryable
fields (`QUERY_FIELDS`), using `""` where a field does not apply, and refuses a `where` on an
undeclared field. Existing rows are rewritten once with `ousast plane memory-normalise`.
RustFS's Select infers an object's JSON schema from its leading rows. This causes two failures
(measured 2026-10-02; `memory.py`, `S3Store._select`):
- A `where` field missing from the leading rows fails with `EvaluatorBindingDoesNotExist`. This
happens even when a later row has the field.
- A column that is null in the first 1000 rows breaks Select on the whole object.
So the store writes every row with its kind's full set of queryable fields (`QUERY_FIELDS`). It uses
`""` where a field does not apply. It refuses a `where` on an undeclared field. Existing rows are
rewritten once with `ousast plane memory-normalise`.
+83 -57
View File
@@ -1,23 +1,34 @@
# The plane on ax
Model-driven work over whole repositories runs on a separate agentic plane: a **Run**
(`plane/runs/*.yaml`, kind `openultrasast.io/v1alpha1 Run`) is a DAG of steps, and each step is
an ax **Task** (`plane/tasks/`) that binds its Workspaces (`plane/workspaces/`), at most one ax
**Model** (`plane/models/`: `deepseek-flash` for chat, `openrouter-embedding` for embeddings) and
its own `budget: {usd, calls}`. ax (google/ax over Agent Substrate, in a single-node kind cluster
on the maintainer's host) is the only executor; there is no local subprocess path. Bring-up and
the host's lessons are in [ax on this host](ops/ax/README.md); where each part runs today (the
reconciler and the receiver on the host, the tasks in gVisor on Agent Substrate) and what a
separate Kubernetes cluster would take (planned) are on [Deployment](deployment.md); this page
draws the flow.
Model-driven work over whole repositories runs on a separate agentic plane. The plane has these parts:
- A **Run** (`plane/runs/*.yaml`, kind `openultrasast.io/v1alpha1 Run`) is a DAG (a graph of steps
with no cycles).
- Each step is an ax **Task** (`plane/tasks/`).
- A Task binds its Workspaces (`plane/workspaces/`).
- A Task binds at most one ax **Model** (`plane/models/`). `deepseek-flash` serves chat.
`openrouter-embedding` serves embeddings.
- A Task carries its own `budget: {usd, calls}`.
ax is the only executor. It is google/ax, Google's controller that runs agent tasks, over Agent Substrate (its
sandboxed actor runtime). It runs in a single-node kind cluster (Kubernetes in Docker) on the maintainer's host.
There is no local subprocess path.
Where to read more:
- Bring-up and the host's lessons are in [ax on this host](ops/ax/README.md).
- [Deployment](deployment.md) shows where each part runs today. The reconciler and the receiver run on the host.
The tasks run in gVisor (a sandboxing kernel) on Agent Substrate. It also covers what a separate Kubernetes
cluster would take (planned).
- This page draws the flow.
Sources: `src/openultrasast/plane/reconciler.py`, `router.py`, `egress.py`, `runner.py`,
`generate.py`, `memory.py`, and `src/openultrasast/plane/tasks/`.
## One task, end to end
`ousast plane run RUN.yaml` (the reconciler) does this for every ready task; `--workers` bounds
the tasks in flight, and a rerun skips tasks already done.
`ousast plane run RUN.yaml` (the reconciler) does this for every ready task. `--workers` bounds the
tasks in flight. A rerun skips tasks already done.
```mermaid
sequenceDiagram
@@ -48,44 +59,55 @@ sequenceDiagram
What each step guarantees:
- **Credentials** (`router.py`, `runner.py`). The reconciler reads the variable the Model's
`secretKey.key` names from the operator's environment or `.env` (which never overrides an
exported variable). The key exists only in the start request; the runner keeps it in memory,
passes it to the command's environment, redacts it from echoed stderr and never writes it.
- **Egress** (`egress.py`). One EgressPolicy per actor, deny by default, hostnames only: plain
HTTP to the artifact receiver, TLS passthrough to the source hosts of the bound Workspaces and
to the hosts the bound Model declares (`openultrasast.io/egress-hosts`). It is written between
`ax apply` and the resume, and deleted with the actor.
- **Inputs** (`reconciler.py`). Artifacts other tasks produced are never rendered into the
manifest; the runner fetches them from the receiver after the start, and the receiver serves a
running task its declared inputs only.
- **Completion** (`runner.py`). ax has no Completed phase: the runner's delivery of the output tar
is the task's completion. A completion marker stops a resumed actor from repeating billed work;
when the marker says the delivery failed, a later boot retries the delivery alone.
- **Budgets** (`plane/budget.py`). The metered client refuses the next call once a task's `usd` or
`calls` ceiling is reached; the task ends `unfinished` and resumes on a rerun with a larger
budget. Model-free tasks run with `{usd: 0, calls: 0}`, so a stray call fails loudly.
- **Attribution.** `ousast plane status RUN` prints per task its status, `calls`, `prompt`,
`cache_hit` and `output` tokens and `usd` from the tasks' `summary.json`, and writes
`attribution.json` in the run directory (`$OUSAST_RESULTS/plane/RUN/`, default
`~/ousast-results/plane/`).
- **Credentials** (`router.py`, `runner.py`).
- The reconciler reads the variable that the Model's `secretKey.key` names.
- It reads it from the operator's environment or `.env`. `.env` never overrides an exported variable.
- The key exists only in the start request.
- The runner keeps it in memory and passes it to the command's environment.
- The runner redacts it from echoed stderr and never writes it.
- **Egress** (`egress.py`). Egress means outbound network traffic.
- Each actor gets one EgressPolicy. It denies by default and names hostnames only.
- It allows plain HTTP to the artifact receiver.
- It allows TLS passthrough to the source hosts of the bound Workspaces.
- It allows TLS passthrough to the hosts the bound Model declares (`openultrasast.io/egress-hosts`).
- The reconciler writes it between `ax apply` and the resume. It is deleted with the actor.
- **Inputs** (`reconciler.py`).
- Artifacts that other tasks produced are never rendered into the manifest.
- The runner fetches them from the receiver after the start.
- The receiver serves a running task its declared inputs only.
- **Completion** (`runner.py`).
- ax has no Completed phase. The runner's delivery of the output tar is the task's completion.
- A completion marker stops a resumed actor from repeating billed work.
- If the marker says the delivery failed, a later boot retries only the delivery.
- **Budgets** (`plane/budget.py`).
- The metered client refuses the next call once a task reaches its `usd` or `calls` ceiling.
- The task then ends `unfinished`. It resumes on a rerun with a larger budget.
- Model-free tasks run with `{usd: 0, calls: 0}`, so a stray call fails loudly.
- **Attribution.**
- `ousast plane status RUN` prints these per task: status, `calls`, `prompt`, `cache_hit` and
`output` tokens, and `usd`. It reads them from the tasks' `summary.json`.
- It writes `attribution.json` in the run directory (`$OUSAST_RESULTS/plane/RUN/`, default
`~/ousast-results/plane/`).
!!! note "The golden snapshot"
Agent Substrate boots every new actor template once as a *golden* actor to snapshot it, with
the same image, Task and environment, and inside the sandbox nothing tells that boot from the
real one. So the runner's command never starts by itself: it waits for the start request,
which the reconciler sends through `atenet-router` to the task's actor, never to the golden
one. Workspaces are prepared only after the start, so the golden boot touches no network and a
started actor already runs under its egress policy (a clone at boot failed on the live cluster
while the actor was being restored, 2026-09-29; `runner.py`, `_run_once`). The first resume of
a new image can time out while the snapshot is built, which is why the reconciler retries it
Agent Substrate boots every new actor template once as a *golden* actor and snapshots it. That boot
uses the same image, Task and environment. Inside the sandbox, nothing tells that boot from the real one.
So the runner's command never starts by itself. It waits for the start request. The reconciler sends
that request through `atenet-router` to the task's actor, never to the golden one.
Workspaces are prepared only after the start. So the golden boot touches no network. A started actor
already runs under its egress policy. A clone at boot once failed on the live cluster while the actor
was being restored (2026-09-29; `runner.py`, `_run_once`).
The first resume of a new image can time out while the snapshot is built. So the reconciler retries it
for up to `OUSAST_RESUME_TIMEOUT` seconds (900).
## The Runs: validation and loop
`ousast plane workspaces POPULATION --validation-set ...` (`generate.py`) writes one chain per
case; `--loop` adds the `alerts` step per case and the loop's singleton steps. Every step except
the verify passes (and `roles` in harvest Runs) is model-free.
`ousast plane workspaces POPULATION --validation-set ...` (`generate.py`) writes one chain per case.
`--loop` adds the `alerts` step per case and the loop's singleton steps. Every step is model-free
except the verify passes (and `roles` in harvest Runs).
### Per case
@@ -119,23 +141,27 @@ flowchart LR
improve --> outcome["gate.json, journal.json, and rule_policy.json only if accepted"]
```
- **Before the Run** `ousast plane run` seeds from the store (`memory.seed`): a task whose
`openultrasast.io/memory-key` (repository, pin, candidates digest, runner image digest) has
stored facts gets them and is marked done, and the sandboxed `memory-snapshot` task gets the
store's rows after the guard. **After the Run** it ingests every delivered `memory.jsonl`
(`ousast plane remember RUN` repeats this by hand). See [Memory](memory.md).
- The loop's rules and its guard are described in
[Architecture](architecture.md#proposals-from-plane-memory). Adopting an accepted ledger stays a
maintainer commit (`src/openultrasast/plane/tasks/loop.py`).
- **Before the Run**, `ousast plane run` seeds from the store (`memory.seed`).
- Each task has an `openultrasast.io/memory-key`: repository, pin, candidates digest and runner image digest.
- If the store has facts for that key, the task gets them and is marked done.
- The sandboxed `memory-snapshot` task gets the store's rows after the guard.
- **After the Run**, it ingests every delivered `memory.jsonl`. `ousast plane remember RUN` repeats
this by hand. See [Memory](memory.md).
- [Architecture](architecture.md#proposals-from-plane-memory) describes the loop's rules and its guard.
Adopting an accepted ledger stays a maintainer commit (`src/openultrasast/plane/tasks/loop.py`).
- `ousast plane harvest` writes the decision engine's harvest Runs (verify a/b with agree, and
model `roles`); `ousast plane alerts-engine` produces a Run's `alerts` from the Joern engine
image on the host for PHP and other languages quick mode does not cover.
model `roles`).
- `ousast plane alerts-engine` produces a Run's `alerts` from the Joern engine image on the host. It
covers PHP and other languages that quick mode does not cover.
## Where this runs in production
ax is a Kubernetes application over Agent Substrate, every Task runs the one runner image as an actor that
occupies a whole worker, and two profiles of this code base are the target: `kind` on a laptop (today) and `k3s`
in production (planned), where the same CLI submits Runs remotely and the tasks deliver to the S3 store:
ax is a Kubernetes application over Agent Substrate. Every Task runs the one runner image as an actor. Each actor
occupies a whole worker. This code base targets two profiles:
- `kind` on a laptop (today).
- `k3s` in production (planned). There the same CLI submits Runs remotely, and the tasks deliver to the S3 store.
[Production topology: ax on a Kubernetes cluster](deployment.md#production-topology-ax-on-a-kubernetes-cluster)
draws it and sizes the pre-push hook at scale.
+74 -50
View File
@@ -1,16 +1,22 @@
# RustFS setup for the memory store
The plane's memory store can live in an S3-compatible bucket: `OUSAST_MEMORY=s3://sast-memory`
selects `S3Store` (`src/openultrasast/plane/memory.py`), which talks to the server through boto3.
Any S3-compatible store with bucket versioning, lifecycle rules, object tags and S3 Select works;
RustFS is the server the contract tests run against. The maintainer's server is RustFS at
`files.because-security.com`, bucket `sast-memory`, store at the bucket root. What the store keeps
there is described on [Memory](memory.md).
The plane's memory store can live in an S3-compatible bucket. RustFS is an S3-compatible object
storage server.
**The store never configures its bucket.** An admin sets the bucket up once with admin
credentials. The agent account the store runs as can read the bucket's settings but not change
them, and reads and writes objects freely. Every time the store opens, it checks the setup and
refuses to start if anything is missing.
- `OUSAST_MEMORY=s3://sast-memory` selects `S3Store` (`src/openultrasast/plane/memory.py`).
- `S3Store` talks to the server through boto3 (the AWS SDK for Python).
- Any S3-compatible store works if it has bucket versioning, lifecycle rules, object tags and S3 Select
(a server-side query over JSON objects).
- RustFS is the server the contract tests run against.
- The maintainer's server is RustFS at `files.because-security.com`. The bucket is `sast-memory`. The
store sits at the bucket root.
[Memory](memory.md) describes what the store keeps there.
**The store never configures its bucket.** An admin sets the bucket up once with admin credentials.
The store runs as an agent account. That account can read the bucket's settings but not change them.
It reads and writes objects freely. Every time the store opens, it checks the setup. It refuses to start
if anything is missing.
```mermaid
flowchart LR
@@ -35,8 +41,8 @@ aws s3api put-bucket-versioning \
--versioning-configuration Status=Enabled
```
**2. Expire raw run outputs under `runs/` after 30 days.** Everything else (`repos/`, `facts/`,
`index.jsonl` and the blobs) is kept, so the rule must be a plain prefix filter on `runs/`.
**2. Expire raw run outputs under `runs/` after 30 days.** Everything else is kept: `repos/`, `facts/`,
`index.jsonl` and the blobs. So the rule must be a plain prefix filter on `runs/`.
```bash
aws s3api put-bucket-lifecycle-configuration \
@@ -46,9 +52,10 @@ aws s3api put-bucket-lifecycle-configuration \
```
`put-bucket-lifecycle-configuration` **replaces every rule on the bucket**. If the bucket already
has rules, merge this one into them first. The rule id `ousast-runs-expiry` and the 30 days are the
store's defaults (`RUNS_RULE_ID`, `RUNS_EXPIRE_DAYS`); the store accepts any enabled, untagged
prefix rule on `runs/` with a number of days.
has rules, merge this one into them first.
The rule id `ousast-runs-expiry` and the 30 days are the store's defaults (`RUNS_RULE_ID`,
`RUNS_EXPIRE_DAYS`). The store accepts any enabled, untagged prefix rule on `runs/` with a number of days.
To read the result back:
@@ -59,8 +66,8 @@ aws s3api get-bucket-lifecycle-configuration --endpoint-url https://files.becaus
## The agent policy
Attach this policy to the agent account (the account whose keys go into `.env`). It is the
complete policy: nothing else is needed, and it grants no `Put*` on versioning or lifecycle.
Attach this policy to the agent account. That is the account whose keys go into `.env`. It is the
complete policy, and nothing else is needed. It grants no `Put*` on versioning or lifecycle.
```json
{
@@ -102,14 +109,14 @@ complete policy: nothing else is needed, and it grants no `Put*` on versioning o
| `BucketSettingsReadOnly` | `arn:aws:s3:::sast-memory` | The agent can list the bucket and its versions and *read* its location, versioning and lifecycle settings, which is what the startup check needs; it cannot change them. |
| `ObjectsReadWrite` | `arn:aws:s3:::sast-memory/*` | The agent reads, writes and deletes objects and their versions and reads and writes object tags; this covers row objects, blobs, the index, the Select probe, and the `kind` tag the store filters by. |
So the agent writes objects freely but cannot change bucket settings: turning versioning off or
So the agent writes objects freely but cannot change bucket settings. Turning versioning off or
removing the expiry rule needs the admin.
## The `.env` settings
The store reads its endpoint and credentials from `.env` in the working directory or from the
environment (`s3_settings`), never from a manifest, and never prints them. The names are the ones
the AWS SDKs and CLI use for credentials, so one `.env` serves both the store and `aws s3api`.
environment (`s3_settings`). It never reads them from a manifest. It never prints them. The names are
the ones the AWS SDKs and CLI use for credentials. So one `.env` serves both the store and `aws s3api`.
| Variable | Value for this server | Notes |
| --- | --- | --- |
@@ -121,26 +128,33 @@ the AWS SDKs and CLI use for credentials, so one `.env` serves both the store an
| `S3_REGION` | optional (`us-east-1` when unset) | the SigV4 signing region; spares a GetBucketLocation call |
| `S3_BUCKET` | `sast-memory` | the bucket `s3://` names when the URL has none; also the real-server tests' default bucket |
The client signs with SigV4 and uses path-style addressing (`https://host/bucket/key`), which
RustFS needs. The SDK's own checksum trailers are off, because not every S3-compatible server
accepts them.
The client signs with SigV4 (AWS's request-signing scheme). It uses path-style addressing
(`https://host/bucket/key`), which RustFS needs. The SDK's own checksum trailers are off. Not every
S3-compatible server accepts them.
- **Never commit `.env`.** It is gitignored (`.gitignore`: `.env`, `.env.*`, except
`.env.example`). `.env.example` lists these names with empty values.
- **`.env` never overrides a variable already exported in your shell** (`config.load_dotenv`).
A stale `export AWS_SECRET_ACCESS_KEY=...` in your profile silently wins over the file; `unset`
A stale `export AWS_SECRET_ACCESS_KEY=...` in your profile silently wins over the file. `unset`
it when a change in `.env` seems to have no effect.
- The store needs the `s3` extra (`uv sync --extra s3`, which installs boto3); without it opening
the store fails with `OUSAST_MEMORY=s3://... needs boto3: install openultrasast[s3]`. A missing
variable fails with `OUSAST_MEMORY=s3://... needs S3_ENDPOINT, AWS_ACCESS_KEY_ID,
AWS_SECRET_ACCESS_KEY in .env or the environment` (naming only the missing ones); an endpoint
without a scheme fails with `S3_ENDPOINT must be a URL with a scheme (https://host[:port])`.
- The store needs the `s3` extra (`uv sync --extra s3`, which installs boto3). These are the
startup errors:
- Without the extra, opening the store fails with
`OUSAST_MEMORY=s3://... needs boto3: install openultrasast[s3]`.
- A missing variable fails with `OUSAST_MEMORY=s3://... needs S3_ENDPOINT, AWS_ACCESS_KEY_ID,
AWS_SECRET_ACCESS_KEY in .env or the environment`. The message names only the missing ones.
- An endpoint without a scheme fails with
`S3_ENDPOINT must be a URL with a scheme (https://host[:port])`.
## What the store verifies at startup
`S3Store.verify_bucket()` runs once per process for each bucket and prefix. It reads the
versioning status and the lifecycle rules, writes a two-line probe object at
`_probe/select.jsonl` (under the prefix, if any), queries it with S3 Select and reads its tags.
`S3Store.verify_bucket()` runs once per process for each bucket and prefix. It does four things:
1. It reads the versioning status.
2. It reads the lifecycle rules.
3. It writes a two-line probe object at `_probe/select.jsonl` (under the prefix, if any).
4. It queries the probe with S3 Select and reads its tags.
Every missing piece is collected into one error:
```text
@@ -148,8 +162,8 @@ memory s3://sast-memory: the bucket is not set up for the store. The store never
- <one line per missing piece>
```
The lines, exactly as the code writes them for this bucket at the root (`<failure>` is the
server's error code and the first 200 characters of its message):
The table lists the lines exactly as the code writes them for this bucket at the root. `<failure>` is
the server's error code and the first 200 characters of its message.
| Check | Refusal line |
| --- | --- |
@@ -161,32 +175,42 @@ server's error code and the first 200 characters of its message):
| tags unreadable | `object tags cannot be read on _probe/select.jsonl (<failure>): the store filters row objects by their kind tag, so the agent needs s3:GetObjectTagging on arn:aws:s3:::sast-memory/*` |
| S3 Select missing | `S3 Select (SelectObjectContent over JSON Lines) does not answer on _probe/select.jsonl (<failure>). The store requires it and has no local fallback: use a server with S3 Select (RustFS has it; not every S3-compatible server does) and let the agent s3:PutObject and s3:GetObject` |
The admin commands inside the messages use `--endpoint-url "$S3_ENDPOINT"`, which is the full
URL from `.env`, so they run as written once `.env` is exported.
The admin commands inside the messages use `--endpoint-url "$S3_ENDPOINT"`. That is the full URL from
`.env`. So they run as written once `.env` is exported.
There is **no fallback**: the store does not start in a degraded mode, does not drop to a local
store, and does not fetch and filter objects itself.
There is **no fallback**. The store does not:
- start in a degraded mode;
- drop to a local store;
- fetch and filter objects itself.
## The RustFS Select limit and the fixed-schema rule
RustFS's S3 Select infers an object's JSON schema from its leading rows. A `where` field that is
absent from those rows is unbound, even if a later row has it: the query fails with
RustFS's S3 Select infers an object's JSON schema from its leading rows. A `where` field absent from
those rows is unbound, even if a later row has it. The query then fails with
`EvaluatorBindingDoesNotExist` instead of matching (measured 2026-10-02 with the field first
present at row 5000). The store does not read that as "no rows", since that answer could silently
drop matches; it raises:
present at row 5000).
The store does not read that failure as "no rows". That answer could silently drop matches. It raises:
```text
memory s3://sast-memory: S3 Select on <key> failed (EvaluatorBindingDoesNotExist: ...) (a field of [<fields>] is absent from the object's leading rows, which the server reads as its schema); the store has no local fallback
```
The rule that follows: a field the store filters on is present in every row of its kind from
the first row on, as `""` where it does not apply; each row kind has a fixed set of queryable
fields, enforced at write time (see [Memory](memory.md#queryable-fields-per-kind)).
The rule that follows has two parts (see [Memory](memory.md#queryable-fields-per-kind)):
- A field the store filters on is present in every row of its kind from the first row on. It is `""`
where it does not apply.
- Each row kind has a fixed set of queryable fields. The store enforces it at write time.
## Checks against the real server
The contract tests that talk to a real server run only with `OUSAST_MEMORY_TEST_S3=1` (bucket
`OUSAST_MEMORY_TEST_BUCKET`, else `S3_BUCKET`). They verify the bucket at its root, write only
under a fresh `contract-<id>/` prefix, and remove every version and delete marker under it
afterwards; they never configure the bucket. More on the host setup:
[ax on this host](ops/ax/README.md).
The contract tests that talk to a real server run only with `OUSAST_MEMORY_TEST_S3=1`. They use the
bucket `OUSAST_MEMORY_TEST_BUCKET`, else `S3_BUCKET`. The tests:
- verify the bucket at its root;
- write only under a fresh `contract-<id>/` prefix;
- remove every version and delete marker under it afterwards;
- never configure the bucket.
More on the host setup: [ax on this host](ops/ax/README.md).
+200 -128
View File
@@ -1,18 +1,26 @@
# Token ergonomics
Model tokens are the one running cost of this project, and they are spent in two places: by the
**plane** when a Run verifies candidates or compiles the decision engine, and by the maintainers'
own **agentic coding sessions**. The maintainer's framing (2026-09-29): "the token saving strategy
is a routing concern. ax can handle that and structure our services in small components." So the
runtime half is routing, one Model per Task with its own budget, and reuse of what was already
paid for; the development half is tooling that keeps bulk file contents out of the session.
Model tokens are the one running cost of this project. They are spent in two places:
Every number on this page comes from a committed record, named next to it. Records are counts
only; none holds code, prompts or responses.
- by the **plane** (the system that runs scans as small tasks) when a Run verifies candidates
or compiles the decision engine;
- by the maintainers' own **agentic coding sessions**.
The before-and-after summary (script pipeline against plane, and the development share) is on
[Where we stand](where-we-stand.md#4-token-economics-before-and-after); this page holds the
mechanisms and the detail table.
The maintainer's framing (2026-09-29): "the token saving strategy is a routing concern. ax can
handle that and structure our services in small components." Here ax is Google's controller
that runs agent tasks on Kubernetes. So the two halves look like this:
- **Runtime** is routing: one Model per Task with its own budget, and reuse of what was
already paid for.
- **Development** is tooling that keeps bulk file contents out of the session.
Every number on this page comes from a committed record, named next to it. Records hold counts
only. None holds code, prompts or responses.
The before-and-after summary is on
[Where we stand](where-we-stand.md#4-token-economics-before-and-after). It compares the script
pipeline with the plane and gives the development share. This page holds the mechanisms and the
detail table.
## A. Runtime: the plane's token economy
@@ -37,46 +45,67 @@ flowchart LR
### One Model per Task, one budget per Task
A Run is a DAG of ax Tasks; each Task binds at most one Model and its own `budget: {usd, calls}`
([The plane on ax](plane.md)). The budget is enforced by the metered client
(`src/openultrasast/plane/budget.py`, `MeteredClient`): `check()` runs before every call and
raises `BudgetExhausted` once the spent `usd` or the call count reaches the ceiling, so the call
that crosses the ceiling completes and the next one is refused. The task then ends `unfinished`
and resumes from its `units.jsonl` on a rerun with a larger budget; an HTTP 401/402 or
"Insufficient Balance" is an `AccountError`, which fails the task and starts nothing further.
Usage is attributed per call: the wrapped client appends one usage row per call
(`prompt_tokens`, `prompt_cache_hit_tokens`, `completion_tokens`), rows are absorbed even when
the call raises, and they are priced by the Model's price list. An unpriced Model cannot run
under a `usd` budget, and `summary()` reports `usd: None` for it, never 0. Model-free tasks run
with `{usd: 0, calls: 0}`, so a stray call fails loudly.
A Run is a DAG (a graph of steps with no cycles) of ax Tasks. Each Task binds at most one Model.
Each Task has its own `budget: {usd, calls}` ([The plane on ax](plane.md)).
`ousast plane status RUN` is the attribution table: one row per task with `calls`, `prompt`,
`cache_hit`, `output` and `usd` from each task's `summary.json`, a `subtotal` row per Model and a
`total` (a missing value makes a sum `n/a`, never a smaller number); `--units` adds the per-unit
rows. The same table is written as `attribution.json` in the run directory
(`src/openultrasast/plane/reconciler.py`, `attribution()`/`status()`). Provider credentials
reach a task only in the start request: `router.py` reads the variable the Model's
`secretKey.key` names, the runner keeps the value in memory, passes it to the command's
environment and redacts it from echoed stderr; it never appears in a manifest, a rendered Task
or a file.
The metered client enforces the budget (`src/openultrasast/plane/budget.py`, `MeteredClient`):
1. `check()` runs before every call.
2. It raises `BudgetExhausted` once the spent `usd` or the call count reaches the ceiling.
3. So the call that crosses the ceiling completes, and the next one is refused.
4. The task then ends `unfinished`. A rerun with a larger budget resumes from its `units.jsonl`.
An HTTP 401/402 or "Insufficient Balance" is an `AccountError`. It fails the task and starts
nothing further.
Usage is attributed per call:
- The wrapped client appends one usage row per call (`prompt_tokens`,
`prompt_cache_hit_tokens`, `completion_tokens`).
- Rows are absorbed even when the call raises.
- Rows are priced by the Model's price list.
- An unpriced Model cannot run under a `usd` budget. `summary()` reports `usd: None` for it,
never 0.
- Model-free tasks run with `{usd: 0, calls: 0}`, so a stray call fails loudly.
`ousast plane status RUN` prints the attribution table. It reads each task's `summary.json`
and shows:
- one row per task with `calls`, `prompt`, `cache_hit`, `output` and `usd`;
- a `subtotal` row per Model;
- a `total`. A missing value makes a sum `n/a`, never a smaller number.
`--units` adds the per-unit rows. The same table is written as `attribution.json` in the run
directory (`src/openultrasast/plane/reconciler.py`, `attribution()`/`status()`).
Provider credentials reach a task only in the start request:
- `router.py` reads the variable that the Model's `secretKey.key` names.
- The runner keeps the value in memory and passes it to the command's environment.
- The runner redacts it from echoed stderr.
- It never appears in a manifest, a rendered Task or a file.
### Pay once, reuse
- **Repo facts are computed model-free and reused.** `repo-facts` reads source text only (no
model, no git) and its `facts.json` is byte-identical across runs over the same tree. The
memory store keeps a facts entry under the key `(repo, pin, candidates digest, runner image
digest)` (`src/openultrasast/plane/memory.py`, `memory_key()`); when a later Run's task
carries the same key, `seed()` writes the stored `facts.json`, a `done` summary with
`calls: 0` and no `usd`, and marks the task done without starting an actor. A changed pin,
candidate set or image finds nothing and recomputes.
- **Content-addressed memory** (`src/openultrasast/learn/`). Excerpts are stored as
`excerpts/<sha256 of the normalised text>.txt`; an embedding lives at
`embeddings/<model>/<excerpt sha>.json` and "an excerpt is embedded once; a rerun makes no
call" (`embeddings.py`). Model responses are cached under the sha256 of
`{model, messages, params, temperature, sample}` (`program.py`, `request_key()`), and a
compiled program is `programs/<sha256 of its artifact>.json` with the digests of the memory
snapshot and of the response cache it was built from (`compile.py`).
- **Replay is free.** With no client, the response cache replays at $0 and a miss raises
- **Repo facts are computed model-free and reused.**
- `repo-facts` reads source text only (no model, no git).
- Its `facts.json` is byte-identical across runs over the same tree.
- The memory store keeps a facts entry under the key `(repo, pin, candidates digest, runner image
digest)` (`src/openultrasast/plane/memory.py`, `memory_key()`).
- A later Run's task may carry the same key. Then `seed()` writes the stored `facts.json` and a
`done` summary with `calls: 0` and no `usd`. It marks the task done without starting an
actor.
- A changed pin, candidate set or image finds nothing and recomputes.
- **Content-addressed memory** (`src/openultrasast/learn/`). Each item is stored under a hash
of its content.
- Excerpts are stored as `excerpts/<sha256 of the normalised text>.txt`.
- An embedding lives at `embeddings/<model>/<excerpt sha>.json`. The source says: "an excerpt
is embedded once; a rerun makes no call" (`embeddings.py`).
- Model responses are cached under the sha256 of `{model, messages, params, temperature, sample}`
(`program.py`, `request_key()`).
- A compiled program is `programs/<sha256 of its artifact>.json`. It records the digests of
the memory snapshot and of the response cache it was built from (`compile.py`).
- **Replay is free.** With no client, the response cache replays at $0. A miss raises
`ReplayMiss` instead of calling. The injection-slice record states it: "evaluate re-run
replay-only on the committed code: 550 responses replayed, $0, identical overall and
calibration" (`benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`,
@@ -84,26 +113,35 @@ or a file.
### Cheap before expensive
- **Triage first.** Verification runs only over the candidates the recorded triage kept; the
plane applies that recording without a model call and prices it as a separate line
(`recorded_triage_usd`), so every per-candidate cost below is reported both with and without
it (`src/openultrasast/plane/tasks/agree.py`).
- **Triage first.** Verification runs only over the candidates the recorded triage kept.
- The plane applies that recording without a model call.
- It prices the recording as a separate line (`recorded_triage_usd`).
- So every per-candidate cost below is reported both with and without it
(`src/openultrasast/plane/tasks/agree.py`).
- **Hunts are batched per file.** `verify` runs one tool hunt per file for up to `PER_HUNT = 6`
candidates, with the candidates' known callers (capped at 8) in the prompt
candidates. The prompt includes the candidates' known callers, capped at 8
(`src/openultrasast/plane/tasks/verify.py`, `units_of()`).
- **The tie-break pass runs only on disputes.** `agree` compares passes a and b; pass c is
started with `inputs.only` set to the case's `disputed.json`, an empty list ends it `done`
with no model call, and the final verdict is 2-of-3 ("pass c alone never decides",
`agree.py`).
- **Cache-friendly prompt order.** The decision engine's prompt is the compiled prefix
(instruction and fixed demonstrations, identical for every candidate so the provider's
prefix cache hits), then the retrieved examples, then the candidate (`learn/program.py`).
Its self-consistency `k` is tied to that cache: "k = 5 when samples 2..k hit the prefix cache
for >= 0.8 of their prompt tokens"; the measured repeat-hit share was 0.96 (injection-slice
record, `smoke.k_rule`, `smoke.k5_repeat_hit_share_measured`). The verify hunts share a prefix
too: in the first plane increment 2,385,152 of 3,820,395 prompt tokens were cache hits
(`benchmarks/independent/plane-increment-1.json`, `detail.cache_hit_tokens`,
`detail.prompt_tokens`; its reading: "Cost fell 17% (62% of prompt tokens were cache hits)").
- **The tie-break pass runs only on disputes.**
- `agree` compares passes a and b.
- Pass c starts with `inputs.only` set to the case's `disputed.json`.
- An empty list ends it `done` with no model call.
- The final verdict is 2-of-3 ("pass c alone never decides", `agree.py`).
- **Cache-friendly prompt order.** The decision engine's prompt has three parts, in this order
(`learn/program.py`):
1. The compiled prefix: instruction and fixed demonstrations. It is identical for every
candidate, so the provider's prefix cache hits.
2. The retrieved examples.
3. The candidate.
The self-consistency `k` (the number of samples per candidate) is tied to that cache: "k = 5
when samples 2..k hit the prefix cache for >= 0.8 of their prompt tokens". The measured
repeat-hit share was 0.96 (injection-slice record, `smoke.k_rule`,
`smoke.k5_repeat_hit_share_measured`).
The verify hunts share a prefix too. In the first plane increment, 2,385,152 of 3,820,395
prompt tokens were cache hits (`benchmarks/independent/plane-increment-1.json`,
`detail.cache_hit_tokens`, `detail.prompt_tokens`). The record's reading: "Cost fell 17% (62%
of prompt tokens were cache hits)".
### Measured costs
@@ -116,37 +154,41 @@ or a file.
| Decision engine harvest: `verify` part | $2.5373 over 1,961 calls, 474 tasks done, 10,921,167 prompt tokens of which 6,275,060 cache hits, 306,941 output tokens, task budgets summing to $8.624 under a $10 ceiling (`verify`) | `benchmarks/measurements/2026-10-01-decision-engine-harvest/record.json` |
| Decision engine harvest: `roles` part | $1.4036 over 1,034 calls, 151 tasks done, 2,398,669 prompt tokens of which 329,600 cache hits, 370,155 output tokens, task budgets summing to $4.917 under a $10 ceiling (`roles`) | same record |
The increment-2 record's note on the two denominators: "the reference divides by the 46
The increment-2 record notes that the two denominators differ: "the reference divides by the 46
candidates before triage; on that basis the plane is 7% cheaper, on the 43 after triage it is
0.2% under".
!!! note "The meter prices at list; the account moved less"
The attribution table prices every call at the Model's list price
(`plane/models/deepseek-flash.yaml`). The harvest record: "Spend is the plane's attribution
table (`ousast plane status`, task meters priced at plane/models/deepseek-flash.yaml); the
DeepSeek account moved less (balance below)", with the balance at $19.63 before, $18.75
after `verify` and $18.23 after `roles`. The injection-slice record names the ratio: "the
provider charged ~1/3 of the list-price meter (balance moved $0.30 for $0.91 metered):
DeepSeek's off-peak discount window, presumably; the meter prices at list"
(`spend.balance_note`, balance $18.2 to $17.9). Budgets and gates are set against the
meter, never against the balance.
(`plane/models/deepseek-flash.yaml`). The harvest record says: "Spend is the plane's
attribution table (`ousast plane status`, task meters priced at
plane/models/deepseek-flash.yaml); the DeepSeek account moved less (balance below)".
The balance was $19.63 before, $18.75 after `verify` and $18.23 after `roles`.
The injection-slice record names the ratio: "the provider charged ~1/3 of the list-price
meter (balance moved $0.30 for $0.91 metered): DeepSeek's off-peak discount window,
presumably; the meter prices at list" (`spend.balance_note`, balance $18.2 to $17.9).
Budgets and gates are set against the meter, never against the balance.
### Not done yet
- **No model-side token budget in ax itself.** The plane's requirements state what ax does
not yet carry: "task dependencies, artifacts, token budgets (on ax's roadmap), a DeepSeek
provider"; the thin `Run` layer carries them and dissolves into ax as ax gains them. Until
then the ceiling lives in the metered client, not in the executor.
- **Triage is applied, not run.** The plane has no triage task; it filters by the recorded
triage and adds that recording's cost as its own line. A triage task is planned as a later
specification.
- **`k` and the contrast examples are not knobs.** `k` is a compiled setting in {1, 3, 5}
(`learn/compile.toml`, "k and lam remain A/B variants"), changed only through the decision
engine's A/B experiments, its requirement that "changes to prompts, models, pass counts, rule
sets and features [are] compared by controlled experiments, so that nothing is adopted on one
run's number" (Requirement 4 of its specification). The retrieval cap of at most two
contrast examples from other families is a module constant (`learn/retrieve.py`,
`CONTRAST = 2`), a maintainer decision, not an experiment variant yet.
- **No model-side token budget in ax itself.**
- The plane's requirements state what ax does not yet carry: "task dependencies, artifacts,
token budgets (on ax's roadmap), a DeepSeek provider".
- The thin `Run` layer carries them. It dissolves into ax as ax gains them.
- Until then the ceiling lives in the metered client, not in the executor.
- **Triage is applied, not run.**
- The plane has no triage task.
- It filters by the recorded triage and adds that recording's cost as its own line.
- A triage task is planned as a later specification.
- **`k` and the contrast examples are not knobs.**
- `k` is a compiled setting in {1, 3, 5} (`learn/compile.toml`, "k and lam remain A/B
variants").
- It changes only through the decision engine's A/B experiments. Requirement 4 of its
specification asks that "changes to prompts, models, pass counts, rule sets and features
[are] compared by controlled experiments, so that nothing is adopted on one run's number".
- Retrieval adds at most two contrast examples from other families. This cap is a module
constant (`learn/retrieve.py`, `CONTRAST = 2`).
- The cap is a maintainer decision, not an experiment variant yet.
## B. Development: the maintainers' own sessions
@@ -165,28 +207,47 @@ flowchart LR
### The measurement: tool I/O dominates
`benchmarks/dev/token_report.py` reads a Claude Code session transcript (`.jsonl`; without an
argument, the newest under `~/.claude/projects/`) and reports, per content kind, characters and
share: `tool_use_input` (the tool call's input as JSON), `tool_result`, `thinking`,
`assistant:text`, `user:text`, plus the tool-call count and the top five tools. Tokens are
approximated at four characters each. It prints to stdout and writes nothing; what gets
committed is the counts-only summary. The one in `benchmarks/independent/plane-increment-1.json`
(`dev_token_report`, for "the whole development session (c09e30b2), not the increment alone"):
7.3M unique characters (~1.81M tokens), `tool_use_input_pct` 45.4, `tool_result_pct` 40.1,
`tool_io_pct` 85.5 against `previous_tool_io_pct` 90, over 2,701 tool calls. The reasoning the
session is paid for (assistant text and thinking) is the small remainder. That is why the
remedies below target file contents, not prose.
`benchmarks/dev/token_report.py` reads a Claude Code session transcript (`.jsonl`). Without an
argument it reads the newest one under `~/.claude/projects/`.
It reports characters and share per content kind:
- `tool_use_input` (the tool call's input as JSON);
- `tool_result`;
- `thinking`;
- `assistant:text`;
- `user:text`.
It also reports the tool-call count and the top five tools. Tokens are approximated at four
characters each. It prints to stdout and writes nothing. Only the counts-only summary is
committed.
The committed summary is in `benchmarks/independent/plane-increment-1.json`
(`dev_token_report`). It covers "the whole development session (c09e30b2), not the increment
alone". It counts 7.3M unique characters (~1.81M tokens). The shares
are `tool_use_input_pct` 45.4, `tool_result_pct` 40.1, and `tool_io_pct` 85.5 against
`previous_tool_io_pct` 90, over 2,701 tool calls.
The reasoning the session is paid for (assistant text and thinking) is the small remainder.
That is why the remedies below target file contents, not prose.
### The remedies in `benchmarks/dev/`
**`read_guard.py`**, a `PreToolUse` hook for agentic coding sessions. It reads the hook event on
stdin and, for the `Read` tool only, denies a whole-file read of a file longer than 350 lines
(`LIMIT = 350`); a read with `offset` or `limit`, a missing path, an unreadable file or a file
within the limit passes. It always exits 0 ("never fails the tool"). A denial is one JSON line
whose `permissionDecisionReason` tells the agent to read a range or to ask the bulk reader. The
script's docstring records that it is wired in `.claude/settings.local.json` (a local, untracked
file) as `PreToolUse` with matcher `Read`; the setting has this shape, with the command
repo-relative because hooks run from the project directory:
**`read_guard.py`** is a `PreToolUse` hook (a script that runs before each tool call) for
agentic coding sessions.
- It reads the hook event on stdin.
- For the `Read` tool only, it denies a whole-file read of a file longer than 350 lines
(`LIMIT = 350`).
- These pass: a read with `offset` or `limit`, a missing path, an unreadable file, or a file
within the limit.
- It always exits 0 ("never fails the tool").
- A denial is one JSON line. Its `permissionDecisionReason` tells the agent to read a range or
to ask the bulk reader.
The script's docstring records how it is wired. It sits in `.claude/settings.local.json` (a
local, untracked file) as `PreToolUse` with matcher `Read`. The setting has this shape. The
command is repo-relative because hooks run from the project directory:
```json
{
@@ -203,37 +264,48 @@ repo-relative because hooks run from the project directory:
}
```
**`bulk_read.py`**, the portal pattern: a cheap model reads the files and answers one question,
citing `path:line` for every claim; it reports what the files say and decides nothing.
**`bulk_read.py`** follows the portal pattern. A cheap model reads the files and answers one
question. It cites `path:line` for every claim. It reports what the files say and decides
nothing.
```text
.venv/bin/python benchmarks/dev/bulk_read.py "<question>" FILE [FILE ...] [--max-usd 0.05]
```
Files go to the model in 400-line chunks with a `--- path (lines a-b)` header and numbered
lines; the client is the project's detector client (`deepseek-flash` through `DEEPSEEK_API_KEY`,
otherwise OpenRouter), each call has a 90 s timeout and no tools. `--max-usd` (default 0.05) is
checked before every chunk; at the ceiling the answer ends with
`[stopped at the $X ceiling before: <chunk header>]`. The answer goes to stdout, the summary
`[bulk_read: N chunk(s), $X.XXXX]` to stderr. Without a chat client it exits 2.
How it works:
- Files go to the model in 400-line chunks, with a `--- path (lines a-b)` header and numbered
lines.
- The client is the project's detector client: `deepseek-flash` through `DEEPSEEK_API_KEY`,
otherwise OpenRouter.
- Each call has a 90 s timeout and no tools.
- `--max-usd` (default 0.05) is checked before every chunk. At the ceiling the answer ends with
`[stopped at the $X ceiling before: <chunk header>]`.
- The answer goes to stdout. The summary `[bulk_read: N chunk(s), $X.XXXX]` goes to stderr.
- Without a chat client it exits 2.
**`token_report.py`**: `python benchmarks/dev/token_report.py [TRANSCRIPT.jsonl]`, described
above.
### Practices
- **Scripts into files, not heredocs.** A heredoc is `tool_use_input` and its output
`tool_result`; both are paid again at every later turn of the session. A script on disk is
read once, by the interpreter.
- **Terse outputs.** Commands print counts, exit codes and the lines that matter; a harness
fails loudly with the exit code rather than printing a table that has to be read back.
- **Subagents with worktrees** for work whose output need not enter the main context: a
subagent's reads and command output stay in its own context and only its conclusion comes
back; a git worktree keeps its edits off the main checkout.
- **Scripts into files, not heredocs.** A heredoc (a script typed inline in a shell command) is
`tool_use_input`, and its output is `tool_result`. Both are paid again at every later turn of
the session. A script on disk is read once, by the interpreter.
- **Terse outputs.** Commands print counts, exit codes and the lines that matter. A harness
fails loudly with the exit code. It does not print a table that has to be read back.
- **Subagents with worktrees** for work whose output need not enter the main context.
- A subagent's reads and command output stay in its own context.
- Only its conclusion comes back.
- A git worktree (a separate checkout of the same repository) keeps its edits off the main
checkout.
- **Counts-only measurement records.** A record under `benchmarks/measurements/` or
`benchmarks/independent/` holds numbers and their keys, never code, prompts or responses, so
it can be quoted on a page like this one and reread by a model at a few hundred tokens.
`benchmarks/independent/` holds numbers and their keys. It never holds code, prompts or
responses. So it can be quoted on a page like this one. A model can reread it at a few
hundred tokens.
Related: [The plane on ax](plane.md) for the Run and Task contract, [Memory](memory.md) for
the store the reuse keys live in, [Evaluation](evaluation.md) for the gates these costs are
measured against.
Related:
- [The plane on ax](plane.md) for the Run and Task contract;
- [Memory](memory.md) for the store the reuse keys live in;
- [Evaluation](evaluation.md) for the gates these costs are measured against.