From 7e23379e9697e057d1875e8b3cddf3dd7559ec58 Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Fri, 2 Oct 2026 21:00:37 +0200 Subject: [PATCH] docs: plain-language pass -- deployment, plane, memory, RustFS, token economics Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo --- docs/deployment.md | 601 +++++++++++++++++++++-------------- docs/memory-and-detection.md | 224 ++++++++----- docs/memory.md | 90 +++--- docs/plane.md | 140 ++++---- docs/rustfs.md | 124 +++++--- docs/token-ergonomics.md | 328 +++++++++++-------- 6 files changed, 916 insertions(+), 591 deletions(-) diff --git a/docs/deployment.md b/docs/deployment.md index c82de46..2ac3de6 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -1,17 +1,21 @@ # Deployment -How the agentic plane is deployed today, and what moving it to a separate Kubernetes cluster -would take. State as of 2026-10-02 (v2.0.0). A scan needs none of this: `ousast scan` and -`ousast pre-push` run from the Python package (and the Docker image for Joern, see -[Engine and pre-push](ops/README.md)). The plane is the maintainers' surface for model work over -whole repositories, driven by `ousast plane`. +This page covers two things. First, how the agentic plane is deployed today. Second, what moving it to a +separate Kubernetes cluster would take. It describes the state as of 2026-10-02 (v2.0.0). -Two words are used strictly on this page. **Implemented** means the code on `main` does it and it -has run on the maintainer's host. **Planned** means it has not been done; the guidance is derived -from the code and the host's lessons, and the code changes it needs are named with their file paths. +A scan needs none of this. `ousast scan` and `ousast pre-push` run from the Python package. Joern runs +from the Docker image (see [Engine and pre-push](ops/README.md)). The plane is the maintainers' tool for +model work over whole repositories. `ousast plane` drives it. + +This page uses two words strictly: + +- **Implemented** means the code on `main` does it, and it has run on the maintainer's host. +- **Planned** means it has not been done. The guidance comes from the code and from what the host taught + us. Each code change it needs is named with its file path. **No deployment of the plane to a cluster other than the maintainer's single-node kind cluster has -been made.** Section B is guidance, not a record. +been made.** Section B is guidance, not a record. (kind runs a Kubernetes cluster inside Docker +containers on one machine.) ## What runs where today (implemented) @@ -51,14 +55,21 @@ flowchart LR | Memory store | `OUSAST_MEMORY`: a file store under `~/ousast-results/plane/memory` by default, or `s3://[/]` on any reachable S3-compatible server (RustFS is the tested one) | `src/openultrasast/plane/memory.py`, [Memory](memory.md), [RustFS setup](rustfs.md) | | Run state and artifacts | `$OUSAST_RESULTS/plane//` on the host (default `~/ousast-results/plane/`) | `reconciler.py` (`run_dir`) | -Only the reconciler talks to the memory store: a sandboxed task cannot read it, so the loop's -`memory-snapshot` is seeded by the reconciler before the Run ([The plane on ax](plane.md)). +Terms in the table: + +- **ax** is Google's controller that runs agent tasks on Kubernetes. +- **Agent Substrate** is the layer below ax that owns the machines and runs sandboxes. +- **gVisor** is a sandbox runtime that isolates each actor from the node's kernel. +- An **actor** is one running Task inside a sandbox. + +Only the reconciler talks to the memory store. A sandboxed task cannot read it. So the reconciler seeds +the loop's `memory-snapshot` before the Run starts ([The plane on ax](plane.md)). ## A. This host: the local setup (implemented) -What `ops/ax/up.sh` needs on the PATH: `docker`, `go`, `kubectl`, `kind`, `ko` and `ax` (it -refuses to start when one is missing). Tools live in `~/go/bin` and `~/.local/bin`; sources and -rendered files in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`). +`ops/ax/up.sh` needs these tools on the PATH: `docker`, `go`, `kubectl`, `kind`, `ko` and `ax`. It +refuses to start when one is missing. Tools live in `~/go/bin` and `~/.local/bin`. Sources and rendered +files live in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`). 1. **Bring up the cluster and the plane.** @@ -66,15 +77,20 @@ rendered files in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`). ops/ax/up.sh ``` - Idempotent: each step is skipped when already done. It creates the kind cluster - (`KIND_CLUSTER_NAME`, default `ousast`) with its registry (`KO_DOCKER_REPO`, default - `localhost:5001`), installs Agent Substrate, applies the egress gateway (the agentgateway - variant, one prebuilt image) when the install left it out, deploys the ax control plane, applies - the receiver Service with the host's kind-gateway address as its endpoint, applies the gVisor - worker pool, builds and pushes the runner image and writes its digest pin to - `~/.cache/ousast/ax-src/runner-image`, installs the `ax` CLI from the deployed checkout (a release - CLI skews from the server), and runs one smoke Task. It ends by printing the host's memory and the - kind containers' usage. + The script is idempotent: it skips each step that is already done. In order, it: + + - creates the kind cluster (`KIND_CLUSTER_NAME`, default `ousast`) with its registry + (`KO_DOCKER_REPO`, default `localhost:5001`); + - installs Agent Substrate; + - applies the egress gateway if the install left it out (the agentgateway variant, one prebuilt image); + - deploys the ax control plane; + - applies the receiver Service, with the host's kind-gateway address as its endpoint; + - applies the gVisor worker pool; + - builds and pushes the runner image, and writes its digest pin to `~/.cache/ousast/ax-src/runner-image`; + - installs the `ax` CLI from the deployed checkout (a release CLI does not match the server version); + - runs one smoke Task. + + At the end it prints the host's memory and the kind containers' usage. 2. **Check it.** @@ -82,9 +98,15 @@ rendered files in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`). uv run ousast plane doctor ``` - Four checks: the kind context is reachable, every pod in `ate-system` and in `ax-system` is ready, - and the registry at `localhost:5001` holds `ousast-runner`. Each failing line names - `ops/ax/up.sh` as the fix. `doctor` is informational: `ousast plane run` does not call it. + It runs four checks: + + - the kind context is reachable; + - every pod in `ate-system` is ready; + - every pod in `ax-system` is ready; + - the registry at `localhost:5001` holds `ousast-runner`. + + Each failing line names `ops/ax/up.sh` as the fix. `doctor` is informational only. + `ousast plane run` does not call it. 3. **Run the end-to-end smoke Run.** @@ -92,23 +114,27 @@ rendered files in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`). ops/ax/smoke-run.sh ``` - It renders `ops/ax/smoke-run.yaml.tmpl` with the current runner digest and submits the Run - `ax-e2e-smoke` through the reconciler: one model-free `repo-facts` Task (`budget: {usd: 0, - calls: 0}`) over a files-only Workspace, resumed, started through the router, its `facts.json` and - `summary.json` delivered to the host, then `ousast plane status ax-e2e-smoke` prints the attribution - table. The script calls the repository's `.venv/bin/ousast`, so `uv sync` first. + It renders `ops/ax/smoke-run.yaml.tmpl` with the current runner digest. It then submits the Run + `ax-e2e-smoke` through the reconciler. The Run has one model-free `repo-facts` Task (`budget: {usd: 0, + calls: 0}`) over a files-only Workspace, resumed, started through the router, its `facts.json` + and `summary.json` delivered to the host. Then `ousast plane status ax-e2e-smoke` + prints the attribution table. The script calls the repository's `.venv/bin/ousast`, so run `uv sync` + first. -4. **Keys and the store.** Put `DEEPSEEK_API_KEY` (chat) and `OPENROUTER_API_KEY` (embeddings) in - `.env` or export them; a Task's Model names the variable (`secretKey.key`), and the key travels - only in that Task's start request. Set `OUSAST_MEMORY=s3://` with `S3_ENDPOINT`, - `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` (optional `S3_REGION`, `S3_BUCKET`) for the S3 store; - the bucket must be set up once by an admin as [RustFS setup](rustfs.md) describes, and the store - verifies it at every start. +4. **Keys and the store.** + + - Put `DEEPSEEK_API_KEY` (chat) and `OPENROUTER_API_KEY` (embeddings) in `.env`, or export them. + - A Task's Model names the variable (`secretKey.key`). The key travels only in that Task's start + request. + - For the S3 store, set `OUSAST_MEMORY=s3://` with `S3_ENDPOINT`, `AWS_ACCESS_KEY_ID` and + `AWS_SECRET_ACCESS_KEY`. `S3_REGION` and `S3_BUCKET` are optional. + - An admin must set up the bucket once, as [RustFS setup](rustfs.md) describes. The store verifies + it at every start. 5. **Run a real Run**, as in [Examples](examples.md#7-running-the-agentic-plane): `ousast plane run plane/runs/validation-46.yaml`, then `status` and `remember`. -6. **Tear down** with `ops/ax/down.sh`, which deletes the cluster and its registry. +6. **Tear down** with `ops/ax/down.sh`. It deletes the cluster and its registry. **Footprint** (measured 2026-09-29, idle, two workers; [ax on this host](ops/ax/README.md#measured-footprint-2026-09-29-idle-two-workers)): @@ -119,44 +145,62 @@ rendered files in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`). | node volume (images, containerd) | 4.9 GB on disk | | worker pool | 2 gVisor workers, 1 CPU / 1.5 GiB limit each | -Disk is the binding constraint: the Substrate install and the `ko` builds need several GB of Go -caches (`go clean -cache -modcache` afterwards), and every Task is its own actor template with a -golden actor (about 24 MB each under `/var/lib/ate/actors` on the node). The failures the host -produced, and what each one taught, are listed in -[ax on this host](ops/ax/README.md#what-this-host-taught-each-one-cost-a-failed-live-run). +Disk is the tightest limit, for two reasons: + +- The Substrate install and the `ko` builds need several GB of Go caches. Run `go clean -cache -modcache` + afterwards. +- Every Task is its own actor template with a golden actor (a saved snapshot the actor boots from). Each + takes about 24 MB under `/var/lib/ate/actors` on the node. + +[ax on this host](ops/ax/README.md#what-this-host-taught-each-one-cost-a-failed-live-run) lists the +failures the host produced and what each one taught. ## B. A separate Kubernetes cluster (planned) -The target: a cluster that runs Agent Substrate with its egress gateway and the ax control plane, -pulls the runner image from a registry of yours, receives task artifacts without a developer -laptop in the path, and keeps its memory in your S3 bucket. The provider key already travels only -in the start request through `atenet-router`, which works the same through a port-forward to any -cluster. Everything else below is either a step you take or a code change that is not made yet. +The target cluster would do four things: + +- run Agent Substrate with its egress gateway and the ax control plane; +- pull the runner image from a registry of yours; +- receive task artifacts without a developer laptop in the path; +- keep its memory in your S3 bucket. + +The provider key already travels only in the start request through `atenet-router`. That works the same +through a port-forward to any cluster. Everything else below is either a step you take or a code change +that is not made yet. ### B.1 Cluster with Agent Substrate, its egress gateway and ax -- Install Agent Substrate on the cluster following its own instructions. `ops/ax/up.sh` uses - Substrate's kind helpers (`hack/create-kind-cluster.sh`, `hack/install-ate-kind.sh`), which do - not apply to another cluster. +- Install Agent Substrate on the cluster with its own instructions. `ops/ax/up.sh` uses Substrate's kind + helpers (`hack/create-kind-cluster.sh`, `hack/install-ate-kind.sh`). These do not apply to another + cluster. - Apply the egress gateway. On this host it is the agentgateway variant - (`manifests/ate-install/agentgateway-egress` in the Substrate checkout, applied with - `kubectl kustomize --load-restrictor=LoadRestrictionsNone`), which needs no Rust build; the - Envoy variant does. Without the gateway an actor has no network at all, and the per-task - EgressPolicies the reconciler writes (`src/openultrasast/plane/egress.py`) work unchanged - against the Substrate-installed gateway: deny by default, hostnames only, plain HTTP to the - receiver, TLS passthrough to the Workspaces' Git hosts and the Model's declared hosts. + (`manifests/ate-install/agentgateway-egress` in the Substrate checkout). It is applied with + `kubectl kustomize --load-restrictor=LoadRestrictionsNone`. It needs no Rust build; the Envoy variant + does. +- Without the gateway an actor has no network at all. +- The reconciler writes one EgressPolicy per task (`src/openultrasast/plane/egress.py`). These work + unchanged against the gateway that Substrate installs. The rules are: + - deny by default; + - hostnames only; + - plain HTTP to the receiver; + - TLS passthrough to the Workspaces' Git hosts and the Model's declared hosts. - Deploy ax (`make deploy AX_IMAGE_REPO=` in the ax checkout) **after** B.2. - Apply a WorkerPool (B.6). Substrate installs none by itself ("no free workers"). ### B.2 ax's snapshot bucket: `AX_SNAPSHOTS_BUCKET` -ax's deploy manifest (`deploy/ax-server.yaml` in the ax checkout) sets `AX_SNAPSHOTS_BUCKET` to -the ax authors' own bucket (`gs://dberkov-gke-dev3/ate-env/` in the checkout used here). Point it -at **your own bucket before deploying ax**. On this host the bucket named there did not exist in -the S3 server, deleted golden actors piled up in `DELETING` and filled the disk during a 600-task -harvest (2026-10-01); creating the bucket let deletes complete, and even then a deleted actor's -directory stays on the node, so long Runs need a janitor that removes directories no live actor -owns. +Point `AX_SNAPSHOTS_BUCKET` at **your own bucket before deploying ax**. ax's deploy manifest +(`deploy/ax-server.yaml` in the ax checkout) sets it to the ax authors' own bucket. In the checkout used +here that is `gs://dberkov-gke-dev3/ate-env/`. + +This host shows what goes wrong otherwise (2026-10-01): + +1. The bucket named there did not exist in the S3 server. +2. Deleted golden actors piled up in `DELETING`. +3. They filled the disk during a 600-task harvest. + +Creating the bucket let deletes complete. Even then, a deleted actor's directory stays on the node. So +long Runs need a janitor that removes directories no live actor owns. ### B.3 A registry the workers can pull from, the runner image pinned by digest @@ -168,83 +212,117 @@ owns. docker inspect --format '{{range .RepoDigests}}{{println .}}{{end}}' /ousast-runner:dev ``` -- Substrate rejects tags: every Task's `image` must be the `/ousast-runner@sha256:...` - reference. Write it into a file (on this host, `~/.cache/ousast/ax-src/runner-image`) and pass - `--runner-image FILE` to `ousast plane workspaces` and `ousast plane harvest`, which re-pin the - Tasks they generate. The committed templates under `plane/tasks/` and the generated - `plane/tasks/validation-46.yaml` carry `localhost:5001/ousast-runner@sha256:...` and need the same - re-pin (regenerate, or replace the reference) before they run elsewhere. -- Workers must be able to pull from the registry (credentials, if any, are the cluster's concern; - ax's default runner image on `gcr.io` needs credentials and is not used here). +- Substrate rejects tags. Every Task's `image` must be the `/ousast-runner@sha256:...` + reference. +- Write that reference into a file. On this host the file is `~/.cache/ousast/ax-src/runner-image`. +- Pass `--runner-image FILE` to `ousast plane workspaces` and `ousast plane harvest`. They re-pin the + Tasks they generate. +- The committed templates under `plane/tasks/` and the generated `plane/tasks/validation-46.yaml` carry + `localhost:5001/ousast-runner@sha256:...`. They need the same re-pin before they run elsewhere. + Regenerate them, or replace the reference. +- Workers must be able to pull from the registry. Credentials, if any, are the cluster's concern. ax's + default runner image on `gcr.io` needs credentials and is not used here. - **Not yet configurable:** `ousast plane doctor` checks the registry at `localhost:5001` by name - (`src/openultrasast/plane/doctor.py:45`), so its fourth check fails on another cluster even when - the image is in place. `doctor` is informational, `plane run` does not depend on it. + (`src/openultrasast/plane/doctor.py:45`). So its fourth check fails on another cluster, even when the + image is in place. `doctor` is informational; `plane run` does not depend on it. ### B.4 One configurable kube context -The reconciler, the egress client and the router port-forward all use the context -`kind-$KIND_CLUSTER_NAME` (`kind-ousast` by default). **Not yet configurable**: the prefix `kind-` -is hard-coded in `src/openultrasast/plane/doctor.py:39`, `src/openultrasast/plane/egress.py:115` -and `:148`, and `src/openultrasast/plane/router.py:145`; `ops/ax/up.sh:24` builds the same name. -The planned change is one variable (for example `OUSAST_KUBE_CONTEXT`) read in those three -modules. Until then the only way to drive another cluster from the unchanged code is to give its -kubeconfig context a name of that form (`kubectl config rename-context kind-` and -`KIND_CLUSTER_NAME=`), which is a workaround, not a supported configuration. The egress -client also needs `kubectl-ate` (or the executable `OUSAST_KUBECTL_ATE` names) able to use that -context. +The reconciler, the egress client and the router port-forward all use one kube context. Its name is +`kind-$KIND_CLUSTER_NAME` (`kind-ousast` by default). + +**Not yet configurable**: the prefix `kind-` is hard-coded in these places: + +- `src/openultrasast/plane/doctor.py:39`; +- `src/openultrasast/plane/egress.py:115` and `:148`; +- `src/openultrasast/plane/router.py:145`; +- `ops/ax/up.sh:24` builds the same name. + +The planned change is one variable read in those three modules, for example `OUSAST_KUBE_CONTEXT`. + +Until then, there is one workaround to drive another cluster with the unchanged code. Rename its +kubeconfig context to that form: `kubectl config rename-context kind-` and +`KIND_CLUSTER_NAME=`. This is a workaround, not a supported configuration. + +The egress client also needs `kubectl-ate` (or the executable `OUSAST_KUBECTL_ATE` names). It must be +able to use that context. ### B.5 The receiver: in-cluster instead of on the host -Today the receiver is a thread inside the reconciler process (`egress.py`, `Receiver`), and the -reconciler learns of a task's completion by watching that object's `delivered` set in memory -(`reconciler.py`). Actors reach it through the Service `ousast-receiver.ax-system` whose -EndpointSlice points at the host. Three paths, in order of effort: +Today the receiver is a thread inside the reconciler process (`egress.py`, `Receiver`). The reconciler +learns that a task finished by watching that object's `delivered` set in memory (`reconciler.py`). +Actors reach it through the Service `ousast-receiver.ax-system`. That Service's EndpointSlice points at +the host. -1. **Interim, works with the unchanged code:** keep the host receiver and render - `ops/ax/receiver-service.yaml.tmpl` with an endpoint address the egress gateway pod can reach - (`KIND_GATEWAY` is just an IPv4 address in the template, `OUSAST_ARTIFACT_PORT` the port). The - reconciler's host must be reachable from the cluster (a VPN or a routed address), and the - gateway dials the address the actor connected to, so the Service's ClusterIP must still map to it. -2. **Planned: the receiver as an in-cluster Deployment** behind the same Service name, with the - reconciler reading deliveries and serving inputs from it instead of from its own thread. This - needs code: the receiver and its `delivered`/`inputs` state are in-process today, and the - runner's `GET /inputs//` is served by that same process. -3. **Follow-on: presigned-URL delivery to the S3 store.** The runner would `PUT` its output tar to a - presigned URL on the memory store's bucket and fetch inputs the same way, and the reconciler - would read from the bucket. Nothing of this exists; the EgressPolicy would then have to allow - the store's hostname for the task. +There are three paths, from least to most effort: + +1. **Interim, works with the unchanged code:** keep the host receiver. + Render `ops/ax/receiver-service.yaml.tmpl` with an endpoint address the egress gateway pod can reach. + In the template, `KIND_GATEWAY` is just an IPv4 address and `OUSAST_ARTIFACT_PORT` is the port. + The cluster must be able to reach the reconciler's host, through a VPN or a routed address. + The gateway dials the address the actor connected to. So the Service's ClusterIP must still map to it. +2. **Planned: the receiver as an in-cluster Deployment** behind the same Service name. + The reconciler would read deliveries from it and serve inputs from it, instead of from its own thread. + This needs code. The receiver and its `delivered`/`inputs` state are in-process today. + The same process serves the runner's `GET /inputs//`. +3. **Follow-on: presigned-URL delivery to the S3 store.** (A presigned URL grants one upload or download + without credentials.) + The runner would `PUT` its output tar to a presigned URL on the memory store's bucket. + It would fetch inputs the same way, and the reconciler would read from the bucket. + None of this exists. The EgressPolicy would then have to allow the store's hostname for the task. ### B.6 Worker pool sizing -`ops/ax/workerpool.yaml.tmpl` declares two warm gVisor workers of 1 CPU / 1.5 GiB (requests 250m / -1.5 GiB) for the 7 GB host, selected onto nodes by `ate.dev/substrate-version`. An actor occupies a -whole worker, so the pool size is the number of tasks that can run at once; set `--workers` on -`ousast plane run` to match. Size the workers above the largest Task's limits (`verify`: requests -250m / 384Mi, limits 1 CPU / 1 GiB; `roles` and the others are in `plane/tasks/`), and keep the -golden-actor storage on the node in mind (B.2). +`ops/ax/workerpool.yaml.tmpl` declares two warm gVisor workers for the 7 GB host. Each has 1 CPU / +1.5 GiB (requests 250m / 1.5 GiB). They are placed on nodes by `ate.dev/substrate-version`. + +An actor occupies a whole worker. So the pool size is the number of tasks that can run at once. Set +`--workers` on `ousast plane run` to match. + +Size the workers above the largest Task's limits. For `verify` these are requests 250m / 384Mi and +limits 1 CPU / 1 GiB. `roles` and the others are in `plane/tasks/`. Also keep the golden-actor storage +on the node in mind (B.2). ### B.7 The S3 memory store reachable from where the reconciler runs -The `s3://` store already works against any reachable S3-compatible server with versioning, -lifecycle rules, object tags and S3 Select (RustFS is the tested one; the bucket setup and the -agent policy are on [RustFS setup](rustfs.md)). Only the reconciler reads and writes it, so today -it must be reachable from the operator's host; if the reconciler moves into the cluster, from its -pod. No Task talks to the store directly; the day one does (B.5, path 3), that Task's EgressPolicy -must allow the store's hostname (the policies allow hostnames only, never addresses), the way a -Model declares its hosts today through `openultrasast.io/egress-hosts`. +The `s3://` store already works against any reachable S3-compatible server. The server must support: + +- versioning; +- lifecycle rules; +- object tags; +- S3 Select (a server-side query over JSON objects). + +RustFS is the tested server. The bucket setup and the agent policy are on [RustFS setup](rustfs.md). + +Only the reconciler reads and writes the store. So today the operator's host must reach it. If the +reconciler moves into the cluster, its pod must reach it. + +No Task talks to the store directly. If one ever does (B.5, path 3), its EgressPolicy must allow the +store's hostname. The policies allow hostnames only, never addresses. A Model already declares its hosts +this way today, through `openultrasast.io/egress-hosts`. ### B.8 Secrets in Kubernetes Secrets, injected into the reconciler's environment -The reconciler reads the variable a Model's `secretKey.key` names (`DEEPSEEK_API_KEY`, -`OPENROUTER_API_KEY`) and the store's `S3_ENDPOINT`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` -(optional `AWS_SESSION_TOKEN`, `S3_REGION`, `S3_BUCKET`) from its own environment or `.env`. When -the reconciler runs in the cluster, keep them in Kubernetes Secrets and inject them into **the -reconciler's** pod environment (`envFrom` with `secretRef`, or `valueFrom.secretKeyRef` per -variable). Never put a value into a manifest: the Model manifests name a variable, the rendered -Task carries no key, and the key reaches an actor only in the start request, where the runner keeps -it in memory and redacts it from echoed stderr. The `secretKey.name` values in `plane/models/` -(`deepseek`, `openrouter`) are the names such Secrets would carry. Running the reconciler -in-cluster is itself planned: today it is a host process, and its state directory +The reconciler reads these variables from its own environment or `.env`: + +- the variable a Model's `secretKey.key` names (`DEEPSEEK_API_KEY`, `OPENROUTER_API_KEY`); +- the store's `S3_ENDPOINT`, `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`; +- optionally `AWS_SESSION_TOKEN`, `S3_REGION` and `S3_BUCKET`. + +When the reconciler runs in the cluster, keep these in Kubernetes Secrets. Inject them into **the +reconciler's** pod environment. Use `envFrom` with `secretRef`, or `valueFrom.secretKeyRef` per variable. + +Never put a value into a manifest. The key stays out of every manifest: + +- the Model manifests name a variable; +- the rendered Task carries no key; +- the key reaches an actor only in the start request; +- the runner keeps it in memory and redacts it from echoed stderr. + +The `secretKey.name` values in `plane/models/` (`deepseek`, `openrouter`) are the names such Secrets +would carry. + +Running the reconciler in-cluster is itself planned. Today it is a host process. Its state directory (`OUSAST_RESULTS`) would need a persistent volume. ### What is implemented and what is planned @@ -266,12 +344,20 @@ in-cluster is itself planned: today it is a host process, and its state director ## Production topology: ax on a Kubernetes cluster -The maintainer's questions, answered from the code and the upstream checkouts under -`~/.cache/ousast/ax-src/`: how is it hosted in ax, is there one container, how does ax relate to -Kubernetes, and what scaling the pre-push hook would take. The decisions of 2026-10-02 set the -target: **two profiles of one code base**, `kind` (this laptop, kept as a first-class local -execution mode, where every change is proven first) and `k3s` (production). Everything marked -**planned** is not built; the rest is what runs on this host today. +This section answers the maintainer's questions from the code and the upstream checkouts under +`~/.cache/ousast/ax-src/`: + +- How is it hosted in ax? +- Is there one container? +- How does ax relate to Kubernetes? +- What would scaling the pre-push hook take? + +The decisions of 2026-10-02 set the target: **two profiles of one code base**. + +- `kind` is this laptop. It stays a first-class local execution mode. Every change is proven here first. +- `k3s` is production. (k3s is a small Kubernetes distribution.) + +Everything marked **planned** is not built. The rest is what runs on this host today. ```mermaid flowchart LR @@ -308,61 +394,92 @@ flowchart LR cli -- "polls: status, attribution" --> s3 ``` -No inbound path exists in this picture: the CLI reaches ax-server and the router from outside, the -tasks push their artifacts to the store through the egress gateway, and the CLI reads the store. -Today's EndpointSlice back to a laptop (`ops/ax/receiver-service.yaml.tmpl`) is the kind profile's -device and does not appear in production. +This picture has no inbound path: + +- the CLI reaches ax-server and the router from outside; +- the tasks push their artifacts to the store through the egress gateway; +- the CLI reads the store. + +Today's EndpointSlice back to a laptop (`ops/ax/receiver-service.yaml.tmpl`) belongs to the kind +profile only. It does not appear in production. ### ax, Kubernetes and Agent Substrate in three sentences -**ax is a Kubernetes application**: a Deployment `ax-server` with a Redis beside it in the -namespace `ax-system` (`ax/deploy/ax-server.yaml` in the ax checkout; its controller runs with -`--template=default-template --template-atespace=ax-system`), and the `ax` CLI reaches it through a -`kubectl port-forward` to `svc/ax-server` (`ax/internal/tunnel/tunnel.go`), so `ousast plane run` -needs a kube context or an ingress, not a public endpoint. **ax does not run containers itself**: -it turns every Task into an ActorTemplate on Agent Substrate (`ax/internal/substrate/client.go`, -`CreateActorTemplate`), built from the Task's image and environment, and the image must be pinned -by digest (Substrate's `SandboxConfig` validation rejects tags: "All images must include a -digest"). **Agent Substrate is the layer that owns the machines**: in the namespace `ate-system` -it runs `ate-api-server`, `ate-controller`, the `atelet` DaemonSet, `atenet-router`, the -`atenet-egress` gateway, a Postgres and a RustFS (`substrate/manifests/ate-install/`), and three -CRDs under `ate.dev/v1alpha1`, `WorkerPool`, `SandboxConfig` and `CSIDriverConfig`; a -`WorkerPool` keeps warm worker pods (`ko://.../cmd/ateom-gvisor`), and an actor is one gVisor -sandbox inside one of those workers, booted from the golden snapshot that -[The plane on ax](plane.md) describes, suspended and resumed by the controller. Names are limited -to 63 bytes (`MaxNameLength` in `ax/pkg/apis/v1alpha1/types.go`), which is why the generated Task -names are short. +**ax is a Kubernetes application.** -**What k3s changes** (planned; nothing has been applied to a k3s cluster): k3s runs containerd -with no Docker socket, so the host-Docker path of `ousast plane alerts-engine` does not exist -there and the engine must become a task (below); Substrate's gVisor workers need `runsc` and a -`RuntimeClass` on the nodes they are scheduled to; k3s ships Traefik (the ingress the CLI can use -instead of a kubeconfig) and local-path storage. Images are built by GitHub Actions from tagged -commits, published to GitHub Container Registry and digest-pinned in the manifests; a private -package needs an `imagePullSecret` on the worker ServiceAccount. Substrate's and ax's own images -are built with `ko` from their checkouts and pushed to the same registry. The kind profile keeps -`localhost:5001` and `ops/ax/up.sh`. +- It is a Deployment `ax-server` with a Redis beside it, in the namespace `ax-system` + (`ax/deploy/ax-server.yaml` in the ax checkout). +- Its controller runs with `--template=default-template --template-atespace=ax-system`. +- The `ax` CLI reaches it through a `kubectl port-forward` to `svc/ax-server` + (`ax/internal/tunnel/tunnel.go`). +- So `ousast plane run` needs a kube context or an ingress, not a public endpoint. + +**ax does not run containers itself.** + +- It turns every Task into an ActorTemplate on Agent Substrate (`ax/internal/substrate/client.go`, + `CreateActorTemplate`). +- The ActorTemplate is built from the Task's image and environment. +- The image must be pinned by digest. Substrate's `SandboxConfig` validation rejects tags: "All images + must include a digest". + +**Agent Substrate is the layer that owns the machines.** + +- In the namespace `ate-system` it runs `ate-api-server`, `ate-controller`, the `atelet` DaemonSet, + `atenet-router`, the `atenet-egress` gateway, a Postgres and a RustFS + (`substrate/manifests/ate-install/`). +- It defines three CRDs (custom Kubernetes resource types) under `ate.dev/v1alpha1`: `WorkerPool`, + `SandboxConfig` and `CSIDriverConfig`. +- A `WorkerPool` keeps warm worker pods (`ko://.../cmd/ateom-gvisor`). +- An actor is one gVisor sandbox inside one of those workers. +- The actor boots from the golden snapshot that [The plane on ax](plane.md) describes. The controller + suspends and resumes it. +- Names are limited to 63 bytes (`MaxNameLength` in `ax/pkg/apis/v1alpha1/types.go`). That is why the + generated Task names are short. + +**What k3s changes** (planned; nothing has been applied to a k3s cluster): + +- k3s runs containerd with no Docker socket. So the host-Docker path of `ousast plane alerts-engine` + does not exist there. The engine must become a task (below). +- Substrate's gVisor workers need `runsc` and a `RuntimeClass` on the nodes they run on. +- k3s ships Traefik and local-path storage. Traefik is an ingress the CLI can use instead of a kubeconfig. +- GitHub Actions builds the images from tagged commits. They are published to GitHub Container Registry + and digest-pinned in the manifests. +- A private package needs an `imagePullSecret` on the worker ServiceAccount. +- Substrate's and ax's own images are built with `ko` from their checkouts. They are pushed to the same + registry. +- The kind profile keeps `localhost:5001` and `ops/ax/up.sh`. ### One image, and what concurrency equals -There is **one container image** for every Task: `plane/Dockerfile.runner` (`python:3.12-slim` -plus this package, PID 1 `/usr/local/bin/ax-task-runner`, which is -`openultrasast.plane.runner`). Each Task manifest under `plane/tasks/` names the same image by -digest (the pin is kept in `~/.cache/ousast/ax-src/runner-image`) and selects its work with -`spec.command`: the runner starts `python -m openultrasast.plane.tasks.` -(`runner.py`, `task_module`). `repo-facts`, `verify`, `agree`, `features`, `remember`, `alerts`, -`loop` and `roles` are modules in that image, not images of their own; the engine would be the -second image (planned). So the production answer to "how many containers" is: one image, one -actor per Task, and **an actor occupies a whole worker** (its limits are the ActorTemplate's, the -worker holds one actor at a time). Concurrency is therefore the number of workers in the -`WorkerPool`, nothing else: `ops/ax/workerpool.yaml.tmpl` declares `replicas: 2` of 1 CPU / -1.5 GiB, so at most two Tasks run at once on this host, and `ousast plane run --workers` must not -exceed the pool. Substrate ships an autoscaled pool -(`substrate/demos/autoscaled-workerpool/README.md`: an HPA on the external metric -`ate_workerpool_workers{state=at_capacity}` through prometheus-adapter, writing -`WorkerPool.spec.replicas`; its README says the demo is kind-only today) and request parking -(`substrate/demos/parking/`: the router holds a resume until a worker frees up). Neither is -applied by `ops/ax/up.sh`; the production profile's pool manifests and autoscaling rule are +There is **one container image** for every Task: `plane/Dockerfile.runner`. It is `python:3.12-slim` +plus this package. PID 1 is `/usr/local/bin/ax-task-runner`, which is `openultrasast.plane.runner`. + +Each Task manifest under `plane/tasks/` names the same image by digest. The pin is kept in +`~/.cache/ousast/ax-src/runner-image`. The manifest picks its work with `spec.command`. The runner +starts `python -m openultrasast.plane.tasks.` (`runner.py`, `task_module`). + +`repo-facts`, `verify`, `agree`, `features`, `remember`, `alerts`, `loop` and `roles` are modules in +that image. They are not images of their own. The engine would be the second image (planned). + +So the production answer to "how many containers" is: + +- one image; +- one actor per Task; +- **an actor occupies a whole worker**. Its limits are the ActorTemplate's, and the worker holds one + actor at a time. + +Concurrency is therefore the number of workers in the `WorkerPool`, nothing else. +`ops/ax/workerpool.yaml.tmpl` declares `replicas: 2` of 1 CPU / 1.5 GiB. So at most two Tasks run at +once on this host. `ousast plane run --workers` must not exceed the pool. + +Substrate ships two scaling aids: + +- **An autoscaled pool** (`substrate/demos/autoscaled-workerpool/README.md`). An HPA (Horizontal Pod + Autoscaler) reads the external metric `ate_workerpool_workers{state=at_capacity}` through + prometheus-adapter. It writes `WorkerPool.spec.replicas`. Its README says the demo is kind-only today. +- **Request parking** (`substrate/demos/parking/`). The router holds a resume until a worker frees up. + +`ops/ax/up.sh` applies neither. The production profile's pool manifests and autoscaling rule are **planned**. ### The three components that are bound to the host today @@ -373,59 +490,75 @@ applied by `ops/ax/up.sh`; the production profile's pool manifests and autoscali | Artifact receiver (`plane/egress.py`, the `Receiver` thread inside the reconciler) | reached through the Service `ousast-receiver.ax-system` whose EndpointSlice points at the kind gateway address on the host (`ops/ax/receiver-service.yaml.tmpl`) | **no receiver in remote mode.** Tasks deliver their artifacts and fetch their inputs through presigned URLs on the S3 memory store (B.5, path 3; the store's hostname enters each Task's EgressPolicy, B.7), and the CLI reads completion from the store. The kind profile keeps the receiver thread; an in-cluster receiver Deployment (B.5, path 2) is not the chosen direction | | Joern engine (`plane/engine_alerts.py`, the host's `openultrasast:dev` container, run with `--memory 3g`) | `ousast plane alerts-engine` runs it on the host, one container at a time, and marks the Run's `alerts` tasks done | **an `engine` task on a second, larger `WorkerPool`** (the engine does not fit a 1.5 GiB worker, and k3s has no Docker socket), from an engine image digest-pinned in GHCR, emitting the same `alerts.jsonl` rows and coverage; `alerts-engine` on the host stays as the development path | -The S3 memory store (RustFS tested) is already external; in remote mode it is also the artifact -path, so it must be reachable from the CLI's host and, through the egress gateway, from the -workers. +The S3 memory store (RustFS tested) is already external. In remote mode it is also the artifact path. +So the CLI's host must reach it, and the workers must reach it through the egress gateway. ### Scaling the pre-push hook in production -**Today** `ousast pre-push` is a local, deterministic check: it materialises base and head, -discovers the changed regions, runs the evidence ranker and arbiter on head and compares with base, -all within `--deadline` (30 s by default) and in `--mode advisory` unless blocking is opted into -([Pre-push: the delta check](scanning.md#pre-push-the-delta-check-experimental), -`src/openultrasast/push/`, [Engine and pre-push](ops/README.md#experimental-pre-push-integration)). -**The plane is not in that path**: no hook submits a Run, no model is called without an explicit -`--model-config`, and nothing leaves the developer's machine. +**Today** `ousast pre-push` is a local, deterministic check. It does four things: + +1. materialises base and head; +2. finds the changed regions; +3. runs the evidence ranker and arbiter on head; +4. compares the result with base. + +All of this runs within `--deadline` (30 s by default). It runs in `--mode advisory` unless you opt +into blocking. See [Pre-push: the delta check](scanning.md#pre-push-the-delta-check-experimental), +`src/openultrasast/push/` and [Engine and pre-push](ops/README.md#experimental-pre-push-integration). + +**The plane is not in that path.** + +- No hook submits a Run. +- No model is called without an explicit `--model-config`. +- Nothing leaves the developer's machine. **The production shape** (planned, none of it built) splits the work by latency: -1. **Blocking, inside the hook's deadline**: what runs today, the quick rules and the delta - engine on the changed regions. It stays local and deterministic, so the push is never held for - a model or a cluster. -2. **Asynchronous, on the plane**: the hook or the CI job calls the CLI in remote mode with a - *delta Run* whose candidates are the functions the push changed; the Run is the per-case chain - of [The plane on ax](plane.md#per-case) (`facts`, `verify` a and b, `agree`, `verify` c on the - disputed only, `final`). The caller either waits, with its own deadline, for the CLI to print - the status and attribution table, or returns at once and a later `ousast plane status ` - reports from the store; a commit status check or a review comment would be built on that. The - worker pool is sized to the push rate or autoscaled on `at_capacity` workers. An unattended - in-cluster reconciler that accepts submissions without a CLI session is the later option named - above. +1. **Blocking, inside the hook's deadline**: what runs today, the quick rules and the delta engine on + the changed regions. It stays local and deterministic. So the push never waits for a model or a + cluster. +2. **Asynchronous, on the plane**: the hook or the CI job calls the CLI in remote mode with a *delta + Run*. + - The Run's candidates are the functions the push changed. + - The Run is the per-case chain of [The plane on ax](plane.md#per-case): `facts`, `verify` a and b, + `agree`, `verify` c on the disputed only, `final`. + - The caller can wait, with its own deadline, for the CLI to print the status and attribution table. + - Or the caller returns at once, and a later `ousast plane status ` reports from the store. + - A commit status check or a review comment would be built on that. + - The worker pool is sized to the push rate, or autoscaled on `at_capacity` workers. + - An unattended in-cluster reconciler that accepts submissions without a CLI session is the later + option named above. -**Cost per push, estimated from measured rates.** Two measured per-candidate rates exist for one -model verdict: `$0.003215` (`spend.cost_per_candidate_evaluate_usd` in -`benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`) and `$0.0021` -(exp-002 arm A, `overall.usd_per_candidate.A` in -`benchmarks/measurements/2026-10-02-exp-002/record.json`). A push that changes five functions needs -two verify passes per candidate: 5 x 2 x $0.0021 = $0.021, or 5 x 2 x $0.003215 = $0.032 at the -higher rate, plus a third pass on the disputed candidates only. That is where "about two cents per -push" comes from; it is arithmetic over the decision engine's measured rates, not a measurement of -a push Run, and `ousast plane status` would print the real figure per Task. +**Cost per push, estimated from measured rates.** Two measured per-candidate rates exist for one model +verdict: + +- `$0.003215` (`spend.cost_per_candidate_evaluate_usd` in + `benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`); +- `$0.0021` (exp-002 arm A, `overall.usd_per_candidate.A` in + `benchmarks/measurements/2026-10-02-exp-002/record.json`). + +A push that changes five functions needs two verify passes per candidate. That gives 5 x 2 x $0.0021 = +$0.021, or 5 x 2 x $0.003215 = $0.032 at the higher rate. A third pass runs on the disputed candidates +only. That is where "about two cents per push" comes from. It is arithmetic over the decision engine's +measured rates, not a measurement of a push Run. `ousast plane status` would print the real figure per +Task. **What is not built** (each is a named gap, not a configuration): - The delta-Run command. `ousast plane scan --base --head ` **does not - exist**; `ousast plane` has `run`, `status`, `doctor`, `remember`, `memory-normalise`, + exist**. `ousast plane` has `run`, `status`, `doctor`, `remember`, `memory-normalise`, `workspaces`, `harvest` and `alerts-engine` (`src/openultrasast/cli.py`). - A candidates task for an arbitrary repository. `repo-facts` takes its candidates from a - population's validation set; a step that derives them from a diff is not written. + population's validation set. No step derives them from a diff yet. - Workspace generation from a URL and a commit outside a population. `plane/generate.py` is - population-driven (`read_cases` reads a validation set and a population TOML), so a Run for a + population-driven: `read_cases` reads a validation set and a population TOML. So a Run for a repository that is not in a population cannot be generated today. -- The remote execution mode itself: the `local`/`remote` switch, configurable cluster context, - ax-server and router addresses (the `kind-` prefix is hard-coded, B.4), presigned delivery from - the runner to the store, and the CLI reading completion and `summary.json` from the store - instead of its receiver thread. +- The remote execution mode itself: + - the `local`/`remote` switch; + - a configurable cluster context and ax-server and router addresses (the `kind-` prefix is + hard-coded, B.4); + - presigned delivery from the runner to the store; + - the CLI reading completion and `summary.json` from the store instead of its receiver thread. - The engine task and its WorkerPool, the GHCR build, the production pool and autoscaling manifests, and the k3s runbook and checklist. - The check or comment that carries the asynchronous verdict back to the push. @@ -433,8 +566,7 @@ a push Run, and `ousast plane status` would print the real figure per Task. ### Sizing The worker is the unit. Every Task's `limits` fit the 1 CPU / 1.5 GiB worker of -`ops/ax/workerpool.yaml.tmpl`; the requests are what the Task declares, the worker is what it -occupies. +`ops/ax/workerpool.yaml.tmpl`. The requests are what the Task declares. The worker is what it occupies. | Task (`plane/tasks/*.yaml`) | Requests | Limits | Per push | | --- | --- | --- | --- | @@ -446,17 +578,18 @@ occupies. | `roles` | 250m / 384Mi | 1 CPU / 1Gi | not in a push Run (harvest Runs only) | | `engine` (planned) | from a measured run | above 3 GiB (the host run uses `--memory 3g`) | its own pool, not the default one | -- **Actors per push**: 4 (`facts`, `va`, `vb`, `agree`), 6 with a tie-break (`vc`, `final`); +- **Actors per push**: 4 (`facts`, `va`, `vb`, `agree`), or 6 with a tie-break (`vc`, `final`). `features` and `remember` add two more when the verdict is to be kept. -- **Critical path**: `facts` then `va` and `vb` side by side, then `agree`, then (sometimes) `vc` - and `final`: three to five Task boots in series, each a golden-snapshot restore plus the model - calls of the verify passes. -- **One actor per worker**: `agree` asks for 256Mi and still holds a 1.5 GiB worker while it runs, - because the worker's capacity, not the Task's request, is the unit. A pool of `W` workers runs - at most `W/2` pushes through their verify phase at once; on this host (`W = 2`) that is one - push at a time, and every further push waits (the reconciler's `--workers` bound, or Substrate's - request parking if the pool is saturated). Autoscaling the pool on `at_capacity` workers is the - upstream answer and is **planned** here. +- **Critical path**: `facts`, then `va` and `vb` side by side, then `agree`, then sometimes `vc` + and `final`. That is three to five Task boots in series. Each boot is a golden-snapshot restore plus + the model calls of the verify passes. +- **One actor per worker**: `agree` asks for 256Mi but still holds a 1.5 GiB worker while it runs. The + worker's capacity, not the Task's request, is the unit. + - A pool of `W` workers runs at most `W/2` pushes through their verify phase at once. + - On this host (`W = 2`) that is one push at a time. + - Every further push waits, on the reconciler's `--workers` bound or on Substrate's request parking + if the pool is saturated. + - Autoscaling the pool on `at_capacity` workers is the upstream answer. It is **planned** here. ## C. The manifests under `plane/` @@ -472,6 +605,12 @@ occupies. | `ops/ax/smoke-task.yaml.tmpl`, `smoke-run.yaml.tmpl` | yes | rendered with `RUNNER_IMAGE` from your pin file | | `ops/ax/up.sh`, `down.sh` | no | kind-specific; they do not apply to another cluster | -The Run and Task contract itself (`plane/runner.py` as PID 1 of the runner image, `AX_TASK_YAML` -and `AX_WORKSPACES_YAML`, `/healthz` and `/readyz`, the start request, the tar delivery as -completion) does not depend on where the cluster is: [The plane on ax](plane.md). +The Run and Task contract does not depend on where the cluster is. That contract covers: + +- `plane/runner.py` as PID 1 of the runner image; +- `AX_TASK_YAML` and `AX_WORKSPACES_YAML`; +- `/healthz` and `/readyz`; +- the start request; +- the tar delivery as completion. + +See [The plane on ax](plane.md). diff --git a/docs/memory-and-detection.md b/docs/memory-and-detection.md index 3c0a612..4fc01fe 100644 --- a/docs/memory-and-detection.md +++ b/docs/memory-and-detection.md @@ -1,11 +1,12 @@ # Memory and detection -How what the system remembers is meant to turn into better decisions on code it has not seen. -This is the design of the learned decision engine (`src/openultrasast/learn/`), which is **in -development and not adopted**: no scan, pre-push check or report uses it today. Every user-facing -decision is still the deterministic one described in [Scanning](scanning.md) and +This page shows how the system's memory is meant to improve decisions on code it has not seen. It +describes the design of the learned decision engine (`src/openultrasast/learn/`). + +The engine is **in development and not adopted**. No scan, pre-push check or report uses it today. +Every user-facing decision is still the deterministic one described in [Scanning](scanning.md) and [Architecture](architecture.md). The engine's commands and status are on -[Decision engine](decision-engine.md); the store itself is on [Memory](memory.md). +[Decision engine](decision-engine.md). The store itself is on [Memory](memory.md). ## From instruments to a decision @@ -25,89 +26,128 @@ flowchart TD op -- "otherwise" --> adv["ADVISORY"] ``` -The intended place for these decisions is changed code: `learn/features.py` builds feature records -for a whole-repository scan or for a base..head delta (`build_for_scan` with `Delta`), the same -delta `ousast pre-push` checks. Integration into scan and pre-push is still ahead. +These decisions are meant for changed code. `learn/features.py` builds feature records in two ways: + +- for a whole-repository scan; +- for a base..head delta (`build_for_scan` with `Delta`). This is the same delta `ousast pre-push` checks. + +Integration into scan and pre-push is still ahead. ## 1. Instruments produce signals, not verdicts Each instrument contributes one block to a candidate's feature record (`learn/schema.py`, -`INSTRUMENTS`): `language`, `quick` (quick rules), `engine` (the Joern model layer), `facts` (repo -facts, such as callers and function span), `source`, `entry_points`, `roles` (model source, sink -and sanitizer roles), `delta`, `verify` and `agree` (the plane's verify passes and their -agreement), and `model_sinks` (the archived stage-1 classifier, `benchmarks/archive/model_sinks.py`, whose -logic now lives in `plane/tasks/roles.py`). A candidate is `(path, function, family)` of one repository and pin -where at least one instrument emitted a signal of that family (`learn/features.py`). +`INSTRUMENTS`). The blocks are: -An instrument that did not run, had no coverage or failed records that state (`none`, `failed`, -`not_applicable`), never a zero, so "silent" and "did not look" stay distinguishable. None of the -instruments decides alone. +- `language`; +- `quick` (quick rules); +- `engine` (the Joern model layer); +- `facts` (repo facts, such as callers and function span); +- `source`; +- `entry_points`; +- `roles` (model source, sink and sanitizer roles); +- `delta`; +- `verify` and `agree` (the plane's verify passes and their agreement); +- `model_sinks` (the archived stage-1 classifier, `benchmarks/archive/model_sinks.py`). Its logic now + lives in `plane/tasks/roles.py`. -**Why profile v1 withholds the engine and function length.** A leak audit -(`ousast learn audit-leaks`, `benchmarks/measurements/2026-10-01-decision-engine-leak-audit/`) -found that both separate the vulnerable and fixed sides of a pair by construction: a fix removes -a sink, and a guard lengthens the function. That holds on the pair corpus, not on unseen code, so -`INPUT_PROFILES["v1"]` drops the `engine` block and `facts.function_lines` from prompts and from -the retrieval distance. +A candidate is `(path, function, family)` of one repository and pin. At least one instrument must have +emitted a signal of that family there (`learn/features.py`). + +An instrument may not run, may have no coverage, or may fail. It then records that state (`none`, +`failed`, `not_applicable`), never a zero. So "silent" and "did not look" stay distinguishable. No +instrument decides alone. + +**Why profile v1 withholds the engine and function length.** A leak audit found that both separate the +vulnerable and fixed sides of a pair by construction (`ousast learn audit-leaks`, +`benchmarks/measurements/2026-10-01-decision-engine-leak-audit/`): + +- a fix removes a sink; +- a guard lengthens the function. + +That holds on the pair corpus, not on unseen code. So `INPUT_PROFILES["v1"]` drops the `engine` block +and `facts.function_lines` from prompts and from the retrieval distance. ## 2. Labels come from ground truth only `ousast learn labels` (`learn/labels.py`) reads only the sources on a fail-closed allow-list -(`learn/sources.toml`): `population-v1`, `population-v2`, `dev-php` (reviewed recipe sites), -`pairs`, `advisory-fixes`, `adjudications` and `assumed_benign`. A file the list does not name is -refused before it is opened; the reserved population v3 is excluded unread. +(`learn/sources.toml`). The sources are `population-v1`, `population-v2`, `dev-php` (reviewed recipe +sites), `pairs`, `advisory-fixes`, `adjudications` and `assumed_benign`. A file the list does not name +is refused before it is opened. The reserved population v3 is excluded unread. -- **Positives** are declared sites and the vulnerable side of a fix; **negatives** are the same - function on the fixed side where the fix changed it, and recorded `False`/`FP` adjudications. - A vulnerable-pin candidate matching no site is unlabelled, never negative. -- **Assumed benign** (maintainer decision 2026-09-30): ordinary non-security commits of the training - repositories, at least 90 days from any fix, with no later security commit touching the same - files. They are their own source with weight 0.5, conditional on a signal at the tip and none at - the base, and reported separately, never merged with verified negatives. -- **Plane verdicts are features, not labels.** "Agreed", "rejected" or "disputed" is the output of - the verify instrument; it enters the record as the `verify` and `agree` blocks. Using it as a - label would teach the engine to reproduce the plane, mistakes included, so "rejected by the - plane" is never a negative (`learn/labels.py`: "A plane verdict ... is never a label"). +- **Positives** are declared sites and the vulnerable side of a fix. +- **Negatives** are two kinds: + - the same function on the fixed side, where the fix changed it; + - recorded `False`/`FP` adjudications. +- A vulnerable-pin candidate matching no site is unlabelled, never negative. +- **Assumed benign** (maintainer decision 2026-09-30) covers ordinary non-security commits of the + training repositories. Each is at least 90 days from any fix. No later security commit touches the + same files. + - They are their own source with weight 0.5, conditional on a signal at the tip and none at the base. + - They are reported separately, never merged with verified negatives. +- **Plane verdicts are features, not labels.** "Agreed", "rejected" or "disputed" is the output of the + verify instrument. It enters the record as the `verify` and `agree` blocks. Using it as a label would + teach the engine to reproduce the plane, mistakes included. So "rejected by the plane" is never a + negative (`learn/labels.py`: "A plane verdict ... is never a label"). `ousast learn memory build` joins each label with the candidate's feature record and an excerpt of -its function and stores one `example` row (no rationale) plus the excerpt blob. +its function. It stores one `example` row (no rationale) plus the excerpt blob. ## 3. Retrieval of similar labelled cases -`learn/retrieve.py`, no model: +`learn/retrieve.py` uses no model. It works in three steps: -1. **Signal profile.** A Gower distance over the allow-listed features of the active profile, - per instrument block, with missing values explicit (a different state counts as distance 1). - The nearest 40 examples of the candidate's family and the nearest 40 of other families go on. -2. **Code embeddings.** A cosine re-rank of the excerpts' embeddings (OpenAI - `text-embedding-3-small` through OpenRouter, 1,536 dimensions, cached by excerpt hash): - `r = 0.5 (1 - D) + 0.5 cos`. Without the candidate's vector the retrieval is signals only. -3. **Balance.** Six examples: at most half per label, at most 2 per repository group, at most 2 - contrast examples from other families. +1. **Signal profile.** It computes a Gower distance (a distance for mixed-type features) over the + allow-listed features of the active profile, per instrument block. Missing values are explicit: a + different state counts as distance 1. The nearest 40 examples of the candidate's family go on. So do + the nearest 40 of other families. +2. **Code embeddings.** It re-ranks by cosine over the excerpts' embeddings. The embeddings come from + OpenAI `text-embedding-3-small` through OpenRouter (1,536 dimensions, cached by excerpt hash). The + formula is `r = 0.5 (1 - D) + 0.5 cos`. Without the candidate's vector, the retrieval uses signals only. +3. **Balance.** It picks six examples: + - at most half per label; + - at most 2 per repository group; + - at most 2 contrast examples from other families. -**The evaluation boundary.** `eligible()` is the only way an example enters a prompt, for retrieved -examples and compiled demonstrations alike: never the candidate's own repository group, never a -group the fold evaluates, never the held-out source or framework, and (outside deployment) never -a near-duplicate of the candidate (cosine >= 0.98). The program repeats the check on the ids it -actually rendered before any call. +**The evaluation boundary.** `eligible()` is the only way an example enters a prompt. This holds for +retrieved examples and compiled demonstrations alike. An eligible example is never: + +- from the candidate's own repository group; +- from a group the fold evaluates; +- from the held-out source or framework; +- a near-duplicate of the candidate (cosine >= 0.98), outside deployment. + +The program repeats the check on the ids it actually rendered, before any call. ## 4. The AI classifier, compiled -`learn/program.py` and `learn/compile.py`, DSPy-style and in-house. The prompt is the instruction -plus the compiled demonstrations (a prefix identical for every candidate, so the provider's prefix -cache hits), then the retrieved examples, then the candidate's excerpt, signals, roles and family. -The model (`deepseek-flash`) answers `vulnerable`, `not_vulnerable` or `unsure` with a confidence -and a rationale that must cite a line of the code. It is sampled `k` times; the score `s` is the -mean over samples, and a majority `unsure` can never BLOCK. Compilation bootstraps demonstrations -and searches instructions on a compile split of repository groups, scored by a pre-registered -balanced Brier score (`learn/compile.toml`); the program is stored as `programs/.json`. +The classifier lives in `learn/program.py` and `learn/compile.py`. It is DSPy-style (DSPy is a +framework that compiles prompts from examples) and built in-house. + +The prompt has this order: + +1. the instruction plus the compiled demonstrations. This prefix is identical for every candidate, so + the provider's prefix cache hits; +2. the retrieved examples; +3. the candidate's excerpt, signals, roles and family. + +The model (`deepseek-flash`) answers `vulnerable`, `not_vulnerable` or `unsure`. It adds a confidence +and a rationale. The rationale must cite a line of the code. The model is sampled `k` times. The score +`s` is the mean over samples. A majority `unsure` can never BLOCK. + +Compilation bootstraps demonstrations and searches instructions on a compile split of repository groups. +A pre-registered balanced Brier score scores it (`learn/compile.toml`). The program is stored as +`programs/.json`. ## 5. Calibration and the two operating points -`learn/calibrate.py`: Platt scaling of `s`, cross-fitted so that no map is checked on the fold it -was fitted on. Calibration **holds** when the slope interval contains 1, the intercept interval -contains 0 and ECE <= 0.05. **BLOCK** is offered only when calibration holds and a fold reaches the -required precision; otherwise the engine offers **ADVISORY** only. +`learn/calibrate.py` applies Platt scaling (a logistic map from score to probability) to `s`. It is +cross-fitted, so no map is checked on the fold it was fitted on. + +Calibration **holds** when the slope interval contains 1, the intercept interval contains 0 and ECE +(expected calibration error) <= 0.05. + +**BLOCK** is offered only when calibration holds and a fold reaches the required precision. Otherwise +the engine offers **ADVISORY** only. ## The loop: memory improves decisions without retraining @@ -125,25 +165,36 @@ flowchart LR gate -- "accepted round" --> ledger["Rule status in the ledger"] ``` -- **Decisions.** The compiled part of the program is its instruction and demonstrations; the - examples are retrieved from memory for each candidate. A new `example` row is therefore eligible - for retrieval in the next decision without recompiling the program, subject to the same - evaluation boundary. `ousast learn curve` measures the metric over memory size on a fixed - candidate subset. -- **Rules.** `ousast improve --memory` reads the run rows (alerts, verdicts, coverage) and proposes - rule-status edits only (shadow a rule that is repeatedly false across repositories; re-enable a - shadow rule that keeps hitting agreed declared sites). Rows from holdout pairs, the gated - manifest's own cases and any `--qualify-population` are dropped first, and every proposal goes - through the same validator and gate as any other edit (`improve/memory.py`). +- **Decisions.** + - The compiled part of the program is its instruction and demonstrations. + - The examples are retrieved from memory for each candidate. + - So a new `example` row is eligible for retrieval in the next decision without recompiling. The same + evaluation boundary applies. + - `ousast learn curve` measures the metric over memory size on a fixed candidate subset. +- **Rules.** `ousast improve --memory` reads the run rows (alerts, verdicts, coverage). It proposes + rule-status edits only: + - shadow a rule that is repeatedly false across repositories; + - re-enable a shadow rule that keeps hitting agreed declared sites. + + Some rows are dropped first: rows from holdout pairs, the gated manifest's own cases and any + `--qualify-population`. Every proposal goes through the same validator and gate as any other edit + (`improve/memory.py`). ## What the first slice measured Record: `benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json` (counts -only). Family **injection**; memory of 724 static examples from all families; input profile v1; -framework priors off; out-of-repository evaluation (compile split never evaluated, outer grouped -5-fold, calibration cross-fitted, operating points nested); 95% repository-cluster bootstrap -intervals. The slice's spend ($0.003215 per candidate at evaluation, k = 5, and its replay at -$0) is on [Token ergonomics](token-ergonomics.md#measured-costs). +only). The setup: + +- family **injection**; +- memory of 724 static examples from all families; +- input profile v1; +- framework priors off; +- out-of-repository evaluation: compile split never evaluated, outer grouped 5-fold, calibration + cross-fitted, operating points nested; +- 95% repository-cluster bootstrap intervals. + +The slice's spend is on [Token ergonomics](token-ergonomics.md#measured-costs). It was $0.003215 per +candidate at evaluation, k = 5, and its replay cost $0. | Measure | Value | What it means | | --- | --- | --- | @@ -152,9 +203,12 @@ $0) is on [Token ergonomics](token-ergonomics.md#measured-costs). | ADVISORY | recall 0.90 (0.82 - 0.97), precision 0.54 (0.44 - 0.65) | precision is at the positive share (0.53), so at this threshold ADVISORY is **not discriminating**: it flags nearly as indiscriminately as flagging everything would | | BLOCK | not offered | calibration does not hold (ECE 0.064 > 0.05), and every outer fold reported `precision_unreachable` | -**The engine is not adopted.** A second slice (2026-10-02) evaluated the six evaluable families -the same way: pooled AUC 0.77 to 0.96 per family, within-pair AUC 0.72 to 0.93, BLOCK offered for -none, re-run agreement below 0.9 for three families -([Decision engine](decision-engine.md#second-slice-six-families-2026-10-02)). Leave-one-source-out -and leave-one-framework-out, integration into scan and pre-push, and the one-time check on -population v3 are still ahead. +**The engine is not adopted.** A second slice (2026-10-02) evaluated the six evaluable families the +same way ([Decision engine](decision-engine.md#second-slice-six-families-2026-10-02)). The results: pooled AUC 0.77 to 0.96 per family, within-pair AUC 0.72 to 0.93, BLOCK offered for +none. Re-run agreement was below 0.9 for three families. + +Still ahead: + +- leave-one-source-out and leave-one-framework-out; +- integration into scan and pre-push; +- the one-time check on population v3. diff --git a/docs/memory.md b/docs/memory.md index 692c59e..421e964 100644 --- a/docs/memory.md +++ b/docs/memory.md @@ -1,8 +1,8 @@ # Memory -One store, keyed by repository and pin, holds what plane runs and the decision engine learned. -The code is `src/openultrasast/plane/memory.py`; the same layout serves both backends. -How the stored rows feed detection is on [Memory and detection](memory-and-detection.md). +One store holds what plane runs and the decision engine learned. It is keyed by repository and pin +(a fixed commit). The code is `src/openultrasast/plane/memory.py`. The same layout serves both backends. +[Memory and detection](memory-and-detection.md) shows how the stored rows feed detection. ## What is stored @@ -17,7 +17,7 @@ programs/.json compiled decision-engine programs ``` **Rows.** Every row carries `id`, `kind`, `repo`, `pin`, `run`, `task`, `population`, `split` -and `image` (`ROW_FIELDS`); `id` is the sha256 of (kind, run, task, subject), so a repeated row +and `image` (`ROW_FIELDS`). `id` is the sha256 of (kind, run, task, subject). So a repeated row replaces itself. The kinds (`KINDS`) and who writes them: | Kind | Written by | Holds | @@ -35,12 +35,15 @@ replaces itself. The kinds (`KINDS`) and who writes them: | `experiment_result` | `ousast learn experiment analyse --record` | the paired estimates, the looks and the adoption verdict (counts and intervals only) | | `label`, `decision` | reserved for later decision-engine tasks | declared in `KINDS`; not written by the code on main yet | -**Blobs** (`put_blob`) are content-addressed: a 64-hex name under a fixed prefix. Excerpts are -written by `learn memory build` (`learn/examples.py`), embeddings by `learn memory embed` -(`learn/embeddings.py`, keyed by excerpt sha so an excerpt is embedded once), responses by the -program's caller (`learn/program.py`, keyed by the sha256 of model, parameters digest, messages, -temperature, sample and format, so a rerun replays at no cost), and compiled programs by -`learn compile` (`learn/compile.py`). +**Blobs** (`put_blob`) are content-addressed: each has a 64-hex name under a fixed prefix. +Content-addressed means the name is the hash of the content. Who writes each blob: + +- Excerpts: `learn memory build` (`learn/examples.py`). +- Embeddings: `learn memory embed` (`learn/embeddings.py`). They are keyed by excerpt sha, so an + excerpt is embedded once. +- Responses: the program's caller (`learn/program.py`). The key is the sha256 of model, parameters + digest, messages, temperature, sample and format. So a rerun replays at no cost. +- Compiled programs: `learn compile` (`learn/compile.py`). ## Write path @@ -60,13 +63,14 @@ flowchart LR put --> store[("Memory store")] ``` -- `ingest` validates every row first: a malformed file fails before anything is written. -- Rows are merged into their repository + pin object by `id`. Every object is written whole and the - index last, so an interrupted ingest leaves nothing a repeat would not repair. +- `ingest` validates every row first. A malformed file fails before anything is written. +- Rows are merged by `id` into their repository + pin object. +- Every object is written whole, and the index last. So a repeat repairs anything an interrupted + ingest left behind. - Each row object carries tags (`TAG_FIELDS`: `repo`, `pin`, `kind`, `family`, `run`, - `population`, `split`) listing the distinct values of its rows (`*` when they exceed 256 - characters). -- The plane's store failures never change a run's result: an ingest can be repeated by hand + `population`, `split`). The tags list the distinct values of its rows. A tag is `*` when the + values exceed 256 characters. +- The plane's store failures never change a run's result. An ingest can be repeated by hand (`cli.py`, `_plane_memory`). ## Read path @@ -85,14 +89,14 @@ flowchart LR rules --> gate{"Unchanged validator and gate"} ``` -- **Seed** (`memory.seed`): a Run task whose `openultrasast.io/memory-key` matches stored facts gets - them and is marked done; a changed pin, candidate set or runner image finds nothing and - recomputes. The loop's `memory-snapshot` is seeded the same way, because a sandboxed task cannot - read the store. +- **Seed** (`memory.seed`). + - A Run task gets stored facts when its `openultrasast.io/memory-key` matches them. It is then marked done. + - A changed pin, candidate set or runner image finds nothing and recomputes. + - The loop's `memory-snapshot` is seeded the same way. A sandboxed task cannot read the store. - **Retrieval** (`learn/retrieve.py`): see [Memory and detection](memory-and-detection.md#3-retrieval-of-similar-labelled-cases). -- **`improve --memory`** (`improve/memory.py`): rows from holdout pairs, from the gated manifest's - own cases and from any `--qualify-population` are dropped before any rule sees them. See +- **`improve --memory`** (`improve/memory.py`). Some rows are dropped before any rule sees them: + rows from holdout pairs, from the gated manifest's own cases, and from any `--qualify-population`. See [Architecture](architecture.md#proposals-from-plane-memory). ## Backends: FileStore and the S3 store @@ -109,26 +113,32 @@ flowchart LR ### Verify at startup, no fallback -The S3 store never configures its bucket. Each time an `S3Store` is opened for a bucket and -prefix (once per process), `verify_bucket()` checks, using reads plus one small probe object at -`/_probe/select.jsonl`: +The S3 store never configures its bucket. It only checks it. An `S3Store` is opened for a bucket and +prefix once per process. Each time, `verify_bucket()` runs these checks. It uses reads plus one small +probe object at `/_probe/select.jsonl`. -1. versioning is `Enabled` (provenance cites object versions); -2. an enabled lifecycle rule expires `/runs/` after a number of days, as a plain prefix - filter, and no enabled rule expires the whole store; -3. S3 Select answers a probe query over JSON Lines; -4. the probe's object tags are readable. +1. Versioning is `Enabled`. Provenance cites object versions. +2. An enabled lifecycle rule expires `/runs/` after a number of days. It uses a plain prefix + filter. No enabled rule expires the whole store. +3. S3 Select (a server-side query over JSON objects) answers a probe query over JSON Lines. +4. The probe's object tags are readable. -Anything missing is named, with the admin command that fixes it, in one `MemoryStoreError`, and -the store does not start. There is no local or fetch-and-filter fallback: a filtered read that -S3 Select cannot answer raises a `MemoryStoreError` instead of returning "no rows". The exact -messages are listed on [RustFS setup](rustfs.md#what-the-store-verifies-at-startup). +If anything is missing, the store does not start. One `MemoryStoreError` names each problem and the +admin command that fixes it. + +There is no local or fetch-and-filter fallback. A filtered read that S3 Select cannot answer raises a +`MemoryStoreError`. It does not return "no rows". [RustFS setup](rustfs.md#what-the-store-verifies-at-startup) +lists the exact messages. ### Queryable fields per kind -RustFS's Select infers an object's JSON schema from its leading rows, so a `where` field missing -there fails with `EvaluatorBindingDoesNotExist` even when a later row has it, and a column that is -null in the first 1000 rows breaks Select on the whole object (measured 2026-10-02; `memory.py`, -`S3Store._select`). The store therefore writes every row with its kind's full set of queryable -fields (`QUERY_FIELDS`), using `""` where a field does not apply, and refuses a `where` on an -undeclared field. Existing rows are rewritten once with `ousast plane memory-normalise`. +RustFS's Select infers an object's JSON schema from its leading rows. This causes two failures +(measured 2026-10-02; `memory.py`, `S3Store._select`): + +- A `where` field missing from the leading rows fails with `EvaluatorBindingDoesNotExist`. This + happens even when a later row has the field. +- A column that is null in the first 1000 rows breaks Select on the whole object. + +So the store writes every row with its kind's full set of queryable fields (`QUERY_FIELDS`). It uses +`""` where a field does not apply. It refuses a `where` on an undeclared field. Existing rows are +rewritten once with `ousast plane memory-normalise`. diff --git a/docs/plane.md b/docs/plane.md index 11276f4..45b15c1 100644 --- a/docs/plane.md +++ b/docs/plane.md @@ -1,23 +1,34 @@ # The plane on ax -Model-driven work over whole repositories runs on a separate agentic plane: a **Run** -(`plane/runs/*.yaml`, kind `openultrasast.io/v1alpha1 Run`) is a DAG of steps, and each step is -an ax **Task** (`plane/tasks/`) that binds its Workspaces (`plane/workspaces/`), at most one ax -**Model** (`plane/models/`: `deepseek-flash` for chat, `openrouter-embedding` for embeddings) and -its own `budget: {usd, calls}`. ax (google/ax over Agent Substrate, in a single-node kind cluster -on the maintainer's host) is the only executor; there is no local subprocess path. Bring-up and -the host's lessons are in [ax on this host](ops/ax/README.md); where each part runs today (the -reconciler and the receiver on the host, the tasks in gVisor on Agent Substrate) and what a -separate Kubernetes cluster would take (planned) are on [Deployment](deployment.md); this page -draws the flow. +Model-driven work over whole repositories runs on a separate agentic plane. The plane has these parts: + +- A **Run** (`plane/runs/*.yaml`, kind `openultrasast.io/v1alpha1 Run`) is a DAG (a graph of steps + with no cycles). +- Each step is an ax **Task** (`plane/tasks/`). +- A Task binds its Workspaces (`plane/workspaces/`). +- A Task binds at most one ax **Model** (`plane/models/`). `deepseek-flash` serves chat. + `openrouter-embedding` serves embeddings. +- A Task carries its own `budget: {usd, calls}`. + +ax is the only executor. It is google/ax, Google's controller that runs agent tasks, over Agent Substrate (its +sandboxed actor runtime). It runs in a single-node kind cluster (Kubernetes in Docker) on the maintainer's host. +There is no local subprocess path. + +Where to read more: + +- Bring-up and the host's lessons are in [ax on this host](ops/ax/README.md). +- [Deployment](deployment.md) shows where each part runs today. The reconciler and the receiver run on the host. + The tasks run in gVisor (a sandboxing kernel) on Agent Substrate. It also covers what a separate Kubernetes + cluster would take (planned). +- This page draws the flow. Sources: `src/openultrasast/plane/reconciler.py`, `router.py`, `egress.py`, `runner.py`, `generate.py`, `memory.py`, and `src/openultrasast/plane/tasks/`. ## One task, end to end -`ousast plane run RUN.yaml` (the reconciler) does this for every ready task; `--workers` bounds -the tasks in flight, and a rerun skips tasks already done. +`ousast plane run RUN.yaml` (the reconciler) does this for every ready task. `--workers` bounds the +tasks in flight. A rerun skips tasks already done. ```mermaid sequenceDiagram @@ -48,44 +59,55 @@ sequenceDiagram What each step guarantees: -- **Credentials** (`router.py`, `runner.py`). The reconciler reads the variable the Model's - `secretKey.key` names from the operator's environment or `.env` (which never overrides an - exported variable). The key exists only in the start request; the runner keeps it in memory, - passes it to the command's environment, redacts it from echoed stderr and never writes it. -- **Egress** (`egress.py`). One EgressPolicy per actor, deny by default, hostnames only: plain - HTTP to the artifact receiver, TLS passthrough to the source hosts of the bound Workspaces and - to the hosts the bound Model declares (`openultrasast.io/egress-hosts`). It is written between - `ax apply` and the resume, and deleted with the actor. -- **Inputs** (`reconciler.py`). Artifacts other tasks produced are never rendered into the - manifest; the runner fetches them from the receiver after the start, and the receiver serves a - running task its declared inputs only. -- **Completion** (`runner.py`). ax has no Completed phase: the runner's delivery of the output tar - is the task's completion. A completion marker stops a resumed actor from repeating billed work; - when the marker says the delivery failed, a later boot retries the delivery alone. -- **Budgets** (`plane/budget.py`). The metered client refuses the next call once a task's `usd` or - `calls` ceiling is reached; the task ends `unfinished` and resumes on a rerun with a larger - budget. Model-free tasks run with `{usd: 0, calls: 0}`, so a stray call fails loudly. -- **Attribution.** `ousast plane status RUN` prints per task its status, `calls`, `prompt`, - `cache_hit` and `output` tokens and `usd` from the tasks' `summary.json`, and writes - `attribution.json` in the run directory (`$OUSAST_RESULTS/plane/RUN/`, default - `~/ousast-results/plane/`). +- **Credentials** (`router.py`, `runner.py`). + - The reconciler reads the variable that the Model's `secretKey.key` names. + - It reads it from the operator's environment or `.env`. `.env` never overrides an exported variable. + - The key exists only in the start request. + - The runner keeps it in memory and passes it to the command's environment. + - The runner redacts it from echoed stderr and never writes it. +- **Egress** (`egress.py`). Egress means outbound network traffic. + - Each actor gets one EgressPolicy. It denies by default and names hostnames only. + - It allows plain HTTP to the artifact receiver. + - It allows TLS passthrough to the source hosts of the bound Workspaces. + - It allows TLS passthrough to the hosts the bound Model declares (`openultrasast.io/egress-hosts`). + - The reconciler writes it between `ax apply` and the resume. It is deleted with the actor. +- **Inputs** (`reconciler.py`). + - Artifacts that other tasks produced are never rendered into the manifest. + - The runner fetches them from the receiver after the start. + - The receiver serves a running task its declared inputs only. +- **Completion** (`runner.py`). + - ax has no Completed phase. The runner's delivery of the output tar is the task's completion. + - A completion marker stops a resumed actor from repeating billed work. + - If the marker says the delivery failed, a later boot retries only the delivery. +- **Budgets** (`plane/budget.py`). + - The metered client refuses the next call once a task reaches its `usd` or `calls` ceiling. + - The task then ends `unfinished`. It resumes on a rerun with a larger budget. + - Model-free tasks run with `{usd: 0, calls: 0}`, so a stray call fails loudly. +- **Attribution.** + - `ousast plane status RUN` prints these per task: status, `calls`, `prompt`, `cache_hit` and + `output` tokens, and `usd`. It reads them from the tasks' `summary.json`. + - It writes `attribution.json` in the run directory (`$OUSAST_RESULTS/plane/RUN/`, default + `~/ousast-results/plane/`). !!! note "The golden snapshot" - Agent Substrate boots every new actor template once as a *golden* actor to snapshot it, with - the same image, Task and environment, and inside the sandbox nothing tells that boot from the - real one. So the runner's command never starts by itself: it waits for the start request, - which the reconciler sends through `atenet-router` to the task's actor, never to the golden - one. Workspaces are prepared only after the start, so the golden boot touches no network and a - started actor already runs under its egress policy (a clone at boot failed on the live cluster - while the actor was being restored, 2026-09-29; `runner.py`, `_run_once`). The first resume of - a new image can time out while the snapshot is built, which is why the reconciler retries it + Agent Substrate boots every new actor template once as a *golden* actor and snapshots it. That boot + uses the same image, Task and environment. Inside the sandbox, nothing tells that boot from the real one. + + So the runner's command never starts by itself. It waits for the start request. The reconciler sends + that request through `atenet-router` to the task's actor, never to the golden one. + + Workspaces are prepared only after the start. So the golden boot touches no network. A started actor + already runs under its egress policy. A clone at boot once failed on the live cluster while the actor + was being restored (2026-09-29; `runner.py`, `_run_once`). + + The first resume of a new image can time out while the snapshot is built. So the reconciler retries it for up to `OUSAST_RESUME_TIMEOUT` seconds (900). ## The Runs: validation and loop -`ousast plane workspaces POPULATION --validation-set ...` (`generate.py`) writes one chain per -case; `--loop` adds the `alerts` step per case and the loop's singleton steps. Every step except -the verify passes (and `roles` in harvest Runs) is model-free. +`ousast plane workspaces POPULATION --validation-set ...` (`generate.py`) writes one chain per case. +`--loop` adds the `alerts` step per case and the loop's singleton steps. Every step is model-free +except the verify passes (and `roles` in harvest Runs). ### Per case @@ -119,23 +141,27 @@ flowchart LR improve --> outcome["gate.json, journal.json, and rule_policy.json only if accepted"] ``` -- **Before the Run** `ousast plane run` seeds from the store (`memory.seed`): a task whose - `openultrasast.io/memory-key` (repository, pin, candidates digest, runner image digest) has - stored facts gets them and is marked done, and the sandboxed `memory-snapshot` task gets the - store's rows after the guard. **After the Run** it ingests every delivered `memory.jsonl` - (`ousast plane remember RUN` repeats this by hand). See [Memory](memory.md). -- The loop's rules and its guard are described in - [Architecture](architecture.md#proposals-from-plane-memory). Adopting an accepted ledger stays a - maintainer commit (`src/openultrasast/plane/tasks/loop.py`). +- **Before the Run**, `ousast plane run` seeds from the store (`memory.seed`). + - Each task has an `openultrasast.io/memory-key`: repository, pin, candidates digest and runner image digest. + - If the store has facts for that key, the task gets them and is marked done. + - The sandboxed `memory-snapshot` task gets the store's rows after the guard. +- **After the Run**, it ingests every delivered `memory.jsonl`. `ousast plane remember RUN` repeats + this by hand. See [Memory](memory.md). +- [Architecture](architecture.md#proposals-from-plane-memory) describes the loop's rules and its guard. + Adopting an accepted ledger stays a maintainer commit (`src/openultrasast/plane/tasks/loop.py`). - `ousast plane harvest` writes the decision engine's harvest Runs (verify a/b with agree, and - model `roles`); `ousast plane alerts-engine` produces a Run's `alerts` from the Joern engine - image on the host for PHP and other languages quick mode does not cover. + model `roles`). +- `ousast plane alerts-engine` produces a Run's `alerts` from the Joern engine image on the host. It + covers PHP and other languages that quick mode does not cover. ## Where this runs in production -ax is a Kubernetes application over Agent Substrate, every Task runs the one runner image as an actor that -occupies a whole worker, and two profiles of this code base are the target: `kind` on a laptop (today) and `k3s` -in production (planned), where the same CLI submits Runs remotely and the tasks deliver to the S3 store: +ax is a Kubernetes application over Agent Substrate. Every Task runs the one runner image as an actor. Each actor +occupies a whole worker. This code base targets two profiles: + +- `kind` on a laptop (today). +- `k3s` in production (planned). There the same CLI submits Runs remotely, and the tasks deliver to the S3 store. + [Production topology: ax on a Kubernetes cluster](deployment.md#production-topology-ax-on-a-kubernetes-cluster) draws it and sizes the pre-push hook at scale. diff --git a/docs/rustfs.md b/docs/rustfs.md index 1c1d6d8..6fb7d8b 100644 --- a/docs/rustfs.md +++ b/docs/rustfs.md @@ -1,16 +1,22 @@ # RustFS setup for the memory store -The plane's memory store can live in an S3-compatible bucket: `OUSAST_MEMORY=s3://sast-memory` -selects `S3Store` (`src/openultrasast/plane/memory.py`), which talks to the server through boto3. -Any S3-compatible store with bucket versioning, lifecycle rules, object tags and S3 Select works; -RustFS is the server the contract tests run against. The maintainer's server is RustFS at -`files.because-security.com`, bucket `sast-memory`, store at the bucket root. What the store keeps -there is described on [Memory](memory.md). +The plane's memory store can live in an S3-compatible bucket. RustFS is an S3-compatible object +storage server. -**The store never configures its bucket.** An admin sets the bucket up once with admin -credentials. The agent account the store runs as can read the bucket's settings but not change -them, and reads and writes objects freely. Every time the store opens, it checks the setup and -refuses to start if anything is missing. +- `OUSAST_MEMORY=s3://sast-memory` selects `S3Store` (`src/openultrasast/plane/memory.py`). +- `S3Store` talks to the server through boto3 (the AWS SDK for Python). +- Any S3-compatible store works if it has bucket versioning, lifecycle rules, object tags and S3 Select + (a server-side query over JSON objects). +- RustFS is the server the contract tests run against. +- The maintainer's server is RustFS at `files.because-security.com`. The bucket is `sast-memory`. The + store sits at the bucket root. + +[Memory](memory.md) describes what the store keeps there. + +**The store never configures its bucket.** An admin sets the bucket up once with admin credentials. +The store runs as an agent account. That account can read the bucket's settings but not change them. +It reads and writes objects freely. Every time the store opens, it checks the setup. It refuses to start +if anything is missing. ```mermaid flowchart LR @@ -35,8 +41,8 @@ aws s3api put-bucket-versioning \ --versioning-configuration Status=Enabled ``` -**2. Expire raw run outputs under `runs/` after 30 days.** Everything else (`repos/`, `facts/`, -`index.jsonl` and the blobs) is kept, so the rule must be a plain prefix filter on `runs/`. +**2. Expire raw run outputs under `runs/` after 30 days.** Everything else is kept: `repos/`, `facts/`, +`index.jsonl` and the blobs. So the rule must be a plain prefix filter on `runs/`. ```bash aws s3api put-bucket-lifecycle-configuration \ @@ -46,9 +52,10 @@ aws s3api put-bucket-lifecycle-configuration \ ``` `put-bucket-lifecycle-configuration` **replaces every rule on the bucket**. If the bucket already -has rules, merge this one into them first. The rule id `ousast-runs-expiry` and the 30 days are the -store's defaults (`RUNS_RULE_ID`, `RUNS_EXPIRE_DAYS`); the store accepts any enabled, untagged -prefix rule on `runs/` with a number of days. +has rules, merge this one into them first. + +The rule id `ousast-runs-expiry` and the 30 days are the store's defaults (`RUNS_RULE_ID`, +`RUNS_EXPIRE_DAYS`). The store accepts any enabled, untagged prefix rule on `runs/` with a number of days. To read the result back: @@ -59,8 +66,8 @@ aws s3api get-bucket-lifecycle-configuration --endpoint-url https://files.becaus ## The agent policy -Attach this policy to the agent account (the account whose keys go into `.env`). It is the -complete policy: nothing else is needed, and it grants no `Put*` on versioning or lifecycle. +Attach this policy to the agent account. That is the account whose keys go into `.env`. It is the +complete policy, and nothing else is needed. It grants no `Put*` on versioning or lifecycle. ```json { @@ -102,14 +109,14 @@ complete policy: nothing else is needed, and it grants no `Put*` on versioning o | `BucketSettingsReadOnly` | `arn:aws:s3:::sast-memory` | The agent can list the bucket and its versions and *read* its location, versioning and lifecycle settings, which is what the startup check needs; it cannot change them. | | `ObjectsReadWrite` | `arn:aws:s3:::sast-memory/*` | The agent reads, writes and deletes objects and their versions and reads and writes object tags; this covers row objects, blobs, the index, the Select probe, and the `kind` tag the store filters by. | -So the agent writes objects freely but cannot change bucket settings: turning versioning off or +So the agent writes objects freely but cannot change bucket settings. Turning versioning off or removing the expiry rule needs the admin. ## The `.env` settings The store reads its endpoint and credentials from `.env` in the working directory or from the -environment (`s3_settings`), never from a manifest, and never prints them. The names are the ones -the AWS SDKs and CLI use for credentials, so one `.env` serves both the store and `aws s3api`. +environment (`s3_settings`). It never reads them from a manifest. It never prints them. The names are +the ones the AWS SDKs and CLI use for credentials. So one `.env` serves both the store and `aws s3api`. | Variable | Value for this server | Notes | | --- | --- | --- | @@ -121,26 +128,33 @@ the AWS SDKs and CLI use for credentials, so one `.env` serves both the store an | `S3_REGION` | optional (`us-east-1` when unset) | the SigV4 signing region; spares a GetBucketLocation call | | `S3_BUCKET` | `sast-memory` | the bucket `s3://` names when the URL has none; also the real-server tests' default bucket | -The client signs with SigV4 and uses path-style addressing (`https://host/bucket/key`), which -RustFS needs. The SDK's own checksum trailers are off, because not every S3-compatible server -accepts them. +The client signs with SigV4 (AWS's request-signing scheme). It uses path-style addressing +(`https://host/bucket/key`), which RustFS needs. The SDK's own checksum trailers are off. Not every +S3-compatible server accepts them. - **Never commit `.env`.** It is gitignored (`.gitignore`: `.env`, `.env.*`, except `.env.example`). `.env.example` lists these names with empty values. - **`.env` never overrides a variable already exported in your shell** (`config.load_dotenv`). - A stale `export AWS_SECRET_ACCESS_KEY=...` in your profile silently wins over the file; `unset` + A stale `export AWS_SECRET_ACCESS_KEY=...` in your profile silently wins over the file. `unset` it when a change in `.env` seems to have no effect. -- The store needs the `s3` extra (`uv sync --extra s3`, which installs boto3); without it opening - the store fails with `OUSAST_MEMORY=s3://... needs boto3: install openultrasast[s3]`. A missing - variable fails with `OUSAST_MEMORY=s3://... needs S3_ENDPOINT, AWS_ACCESS_KEY_ID, - AWS_SECRET_ACCESS_KEY in .env or the environment` (naming only the missing ones); an endpoint - without a scheme fails with `S3_ENDPOINT must be a URL with a scheme (https://host[:port])`. +- The store needs the `s3` extra (`uv sync --extra s3`, which installs boto3). These are the + startup errors: + - Without the extra, opening the store fails with + `OUSAST_MEMORY=s3://... needs boto3: install openultrasast[s3]`. + - A missing variable fails with `OUSAST_MEMORY=s3://... needs S3_ENDPOINT, AWS_ACCESS_KEY_ID, + AWS_SECRET_ACCESS_KEY in .env or the environment`. The message names only the missing ones. + - An endpoint without a scheme fails with + `S3_ENDPOINT must be a URL with a scheme (https://host[:port])`. ## What the store verifies at startup -`S3Store.verify_bucket()` runs once per process for each bucket and prefix. It reads the -versioning status and the lifecycle rules, writes a two-line probe object at -`_probe/select.jsonl` (under the prefix, if any), queries it with S3 Select and reads its tags. +`S3Store.verify_bucket()` runs once per process for each bucket and prefix. It does four things: + +1. It reads the versioning status. +2. It reads the lifecycle rules. +3. It writes a two-line probe object at `_probe/select.jsonl` (under the prefix, if any). +4. It queries the probe with S3 Select and reads its tags. + Every missing piece is collected into one error: ```text @@ -148,8 +162,8 @@ memory s3://sast-memory: the bucket is not set up for the store. The store never - ``` -The lines, exactly as the code writes them for this bucket at the root (`` is the -server's error code and the first 200 characters of its message): +The table lists the lines exactly as the code writes them for this bucket at the root. `` is +the server's error code and the first 200 characters of its message. | Check | Refusal line | | --- | --- | @@ -161,32 +175,42 @@ server's error code and the first 200 characters of its message): | tags unreadable | `object tags cannot be read on _probe/select.jsonl (): the store filters row objects by their kind tag, so the agent needs s3:GetObjectTagging on arn:aws:s3:::sast-memory/*` | | S3 Select missing | `S3 Select (SelectObjectContent over JSON Lines) does not answer on _probe/select.jsonl (). The store requires it and has no local fallback: use a server with S3 Select (RustFS has it; not every S3-compatible server does) and let the agent s3:PutObject and s3:GetObject` | -The admin commands inside the messages use `--endpoint-url "$S3_ENDPOINT"`, which is the full -URL from `.env`, so they run as written once `.env` is exported. +The admin commands inside the messages use `--endpoint-url "$S3_ENDPOINT"`. That is the full URL from +`.env`. So they run as written once `.env` is exported. -There is **no fallback**: the store does not start in a degraded mode, does not drop to a local -store, and does not fetch and filter objects itself. +There is **no fallback**. The store does not: + +- start in a degraded mode; +- drop to a local store; +- fetch and filter objects itself. ## The RustFS Select limit and the fixed-schema rule -RustFS's S3 Select infers an object's JSON schema from its leading rows. A `where` field that is -absent from those rows is unbound, even if a later row has it: the query fails with +RustFS's S3 Select infers an object's JSON schema from its leading rows. A `where` field absent from +those rows is unbound, even if a later row has it. The query then fails with `EvaluatorBindingDoesNotExist` instead of matching (measured 2026-10-02 with the field first -present at row 5000). The store does not read that as "no rows", since that answer could silently -drop matches; it raises: +present at row 5000). + +The store does not read that failure as "no rows". That answer could silently drop matches. It raises: ```text memory s3://sast-memory: S3 Select on failed (EvaluatorBindingDoesNotExist: ...) (a field of [] is absent from the object's leading rows, which the server reads as its schema); the store has no local fallback ``` -The rule that follows: a field the store filters on is present in every row of its kind from -the first row on, as `""` where it does not apply; each row kind has a fixed set of queryable -fields, enforced at write time (see [Memory](memory.md#queryable-fields-per-kind)). +The rule that follows has two parts (see [Memory](memory.md#queryable-fields-per-kind)): + +- A field the store filters on is present in every row of its kind from the first row on. It is `""` + where it does not apply. +- Each row kind has a fixed set of queryable fields. The store enforces it at write time. ## Checks against the real server -The contract tests that talk to a real server run only with `OUSAST_MEMORY_TEST_S3=1` (bucket -`OUSAST_MEMORY_TEST_BUCKET`, else `S3_BUCKET`). They verify the bucket at its root, write only -under a fresh `contract-/` prefix, and remove every version and delete marker under it -afterwards; they never configure the bucket. More on the host setup: -[ax on this host](ops/ax/README.md). +The contract tests that talk to a real server run only with `OUSAST_MEMORY_TEST_S3=1`. They use the +bucket `OUSAST_MEMORY_TEST_BUCKET`, else `S3_BUCKET`. The tests: + +- verify the bucket at its root; +- write only under a fresh `contract-/` prefix; +- remove every version and delete marker under it afterwards; +- never configure the bucket. + +More on the host setup: [ax on this host](ops/ax/README.md). diff --git a/docs/token-ergonomics.md b/docs/token-ergonomics.md index 9ea9e5d..1f60dd8 100644 --- a/docs/token-ergonomics.md +++ b/docs/token-ergonomics.md @@ -1,18 +1,26 @@ # Token ergonomics -Model tokens are the one running cost of this project, and they are spent in two places: by the -**plane** when a Run verifies candidates or compiles the decision engine, and by the maintainers' -own **agentic coding sessions**. The maintainer's framing (2026-09-29): "the token saving strategy -is a routing concern. ax can handle that and structure our services in small components." So the -runtime half is routing, one Model per Task with its own budget, and reuse of what was already -paid for; the development half is tooling that keeps bulk file contents out of the session. +Model tokens are the one running cost of this project. They are spent in two places: -Every number on this page comes from a committed record, named next to it. Records are counts -only; none holds code, prompts or responses. +- by the **plane** (the system that runs scans as small tasks) when a Run verifies candidates + or compiles the decision engine; +- by the maintainers' own **agentic coding sessions**. -The before-and-after summary (script pipeline against plane, and the development share) is on -[Where we stand](where-we-stand.md#4-token-economics-before-and-after); this page holds the -mechanisms and the detail table. +The maintainer's framing (2026-09-29): "the token saving strategy is a routing concern. ax can +handle that and structure our services in small components." Here ax is Google's controller +that runs agent tasks on Kubernetes. So the two halves look like this: + +- **Runtime** is routing: one Model per Task with its own budget, and reuse of what was + already paid for. +- **Development** is tooling that keeps bulk file contents out of the session. + +Every number on this page comes from a committed record, named next to it. Records hold counts +only. None holds code, prompts or responses. + +The before-and-after summary is on +[Where we stand](where-we-stand.md#4-token-economics-before-and-after). It compares the script +pipeline with the plane and gives the development share. This page holds the mechanisms and the +detail table. ## A. Runtime: the plane's token economy @@ -37,46 +45,67 @@ flowchart LR ### One Model per Task, one budget per Task -A Run is a DAG of ax Tasks; each Task binds at most one Model and its own `budget: {usd, calls}` -([The plane on ax](plane.md)). The budget is enforced by the metered client -(`src/openultrasast/plane/budget.py`, `MeteredClient`): `check()` runs before every call and -raises `BudgetExhausted` once the spent `usd` or the call count reaches the ceiling, so the call -that crosses the ceiling completes and the next one is refused. The task then ends `unfinished` -and resumes from its `units.jsonl` on a rerun with a larger budget; an HTTP 401/402 or -"Insufficient Balance" is an `AccountError`, which fails the task and starts nothing further. -Usage is attributed per call: the wrapped client appends one usage row per call -(`prompt_tokens`, `prompt_cache_hit_tokens`, `completion_tokens`), rows are absorbed even when -the call raises, and they are priced by the Model's price list. An unpriced Model cannot run -under a `usd` budget, and `summary()` reports `usd: None` for it, never 0. Model-free tasks run -with `{usd: 0, calls: 0}`, so a stray call fails loudly. +A Run is a DAG (a graph of steps with no cycles) of ax Tasks. Each Task binds at most one Model. +Each Task has its own `budget: {usd, calls}` ([The plane on ax](plane.md)). -`ousast plane status RUN` is the attribution table: one row per task with `calls`, `prompt`, -`cache_hit`, `output` and `usd` from each task's `summary.json`, a `subtotal` row per Model and a -`total` (a missing value makes a sum `n/a`, never a smaller number); `--units` adds the per-unit -rows. The same table is written as `attribution.json` in the run directory -(`src/openultrasast/plane/reconciler.py`, `attribution()`/`status()`). Provider credentials -reach a task only in the start request: `router.py` reads the variable the Model's -`secretKey.key` names, the runner keeps the value in memory, passes it to the command's -environment and redacts it from echoed stderr; it never appears in a manifest, a rendered Task -or a file. +The metered client enforces the budget (`src/openultrasast/plane/budget.py`, `MeteredClient`): + +1. `check()` runs before every call. +2. It raises `BudgetExhausted` once the spent `usd` or the call count reaches the ceiling. +3. So the call that crosses the ceiling completes, and the next one is refused. +4. The task then ends `unfinished`. A rerun with a larger budget resumes from its `units.jsonl`. + +An HTTP 401/402 or "Insufficient Balance" is an `AccountError`. It fails the task and starts +nothing further. + +Usage is attributed per call: + +- The wrapped client appends one usage row per call (`prompt_tokens`, + `prompt_cache_hit_tokens`, `completion_tokens`). +- Rows are absorbed even when the call raises. +- Rows are priced by the Model's price list. +- An unpriced Model cannot run under a `usd` budget. `summary()` reports `usd: None` for it, + never 0. +- Model-free tasks run with `{usd: 0, calls: 0}`, so a stray call fails loudly. + +`ousast plane status RUN` prints the attribution table. It reads each task's `summary.json` +and shows: + +- one row per task with `calls`, `prompt`, `cache_hit`, `output` and `usd`; +- a `subtotal` row per Model; +- a `total`. A missing value makes a sum `n/a`, never a smaller number. + +`--units` adds the per-unit rows. The same table is written as `attribution.json` in the run +directory (`src/openultrasast/plane/reconciler.py`, `attribution()`/`status()`). + +Provider credentials reach a task only in the start request: + +- `router.py` reads the variable that the Model's `secretKey.key` names. +- The runner keeps the value in memory and passes it to the command's environment. +- The runner redacts it from echoed stderr. +- It never appears in a manifest, a rendered Task or a file. ### Pay once, reuse -- **Repo facts are computed model-free and reused.** `repo-facts` reads source text only (no - model, no git) and its `facts.json` is byte-identical across runs over the same tree. The - memory store keeps a facts entry under the key `(repo, pin, candidates digest, runner image - digest)` (`src/openultrasast/plane/memory.py`, `memory_key()`); when a later Run's task - carries the same key, `seed()` writes the stored `facts.json`, a `done` summary with - `calls: 0` and no `usd`, and marks the task done without starting an actor. A changed pin, - candidate set or image finds nothing and recomputes. -- **Content-addressed memory** (`src/openultrasast/learn/`). Excerpts are stored as - `excerpts/.txt`; an embedding lives at - `embeddings//.json` and "an excerpt is embedded once; a rerun makes no - call" (`embeddings.py`). Model responses are cached under the sha256 of - `{model, messages, params, temperature, sample}` (`program.py`, `request_key()`), and a - compiled program is `programs/.json` with the digests of the memory - snapshot and of the response cache it was built from (`compile.py`). -- **Replay is free.** With no client, the response cache replays at $0 and a miss raises +- **Repo facts are computed model-free and reused.** + - `repo-facts` reads source text only (no model, no git). + - Its `facts.json` is byte-identical across runs over the same tree. + - The memory store keeps a facts entry under the key `(repo, pin, candidates digest, runner image + digest)` (`src/openultrasast/plane/memory.py`, `memory_key()`). + - A later Run's task may carry the same key. Then `seed()` writes the stored `facts.json` and a + `done` summary with `calls: 0` and no `usd`. It marks the task done without starting an + actor. + - A changed pin, candidate set or image finds nothing and recomputes. +- **Content-addressed memory** (`src/openultrasast/learn/`). Each item is stored under a hash + of its content. + - Excerpts are stored as `excerpts/.txt`. + - An embedding lives at `embeddings//.json`. The source says: "an excerpt + is embedded once; a rerun makes no call" (`embeddings.py`). + - Model responses are cached under the sha256 of `{model, messages, params, temperature, sample}` + (`program.py`, `request_key()`). + - A compiled program is `programs/.json`. It records the digests of + the memory snapshot and of the response cache it was built from (`compile.py`). +- **Replay is free.** With no client, the response cache replays at $0. A miss raises `ReplayMiss` instead of calling. The injection-slice record states it: "evaluate re-run replay-only on the committed code: 550 responses replayed, $0, identical overall and calibration" (`benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`, @@ -84,26 +113,35 @@ or a file. ### Cheap before expensive -- **Triage first.** Verification runs only over the candidates the recorded triage kept; the - plane applies that recording without a model call and prices it as a separate line - (`recorded_triage_usd`), so every per-candidate cost below is reported both with and without - it (`src/openultrasast/plane/tasks/agree.py`). +- **Triage first.** Verification runs only over the candidates the recorded triage kept. + - The plane applies that recording without a model call. + - It prices the recording as a separate line (`recorded_triage_usd`). + - So every per-candidate cost below is reported both with and without it + (`src/openultrasast/plane/tasks/agree.py`). - **Hunts are batched per file.** `verify` runs one tool hunt per file for up to `PER_HUNT = 6` - candidates, with the candidates' known callers (capped at 8) in the prompt + candidates. The prompt includes the candidates' known callers, capped at 8 (`src/openultrasast/plane/tasks/verify.py`, `units_of()`). -- **The tie-break pass runs only on disputes.** `agree` compares passes a and b; pass c is - started with `inputs.only` set to the case's `disputed.json`, an empty list ends it `done` - with no model call, and the final verdict is 2-of-3 ("pass c alone never decides", - `agree.py`). -- **Cache-friendly prompt order.** The decision engine's prompt is the compiled prefix - (instruction and fixed demonstrations, identical for every candidate so the provider's - prefix cache hits), then the retrieved examples, then the candidate (`learn/program.py`). - Its self-consistency `k` is tied to that cache: "k = 5 when samples 2..k hit the prefix cache - for >= 0.8 of their prompt tokens"; the measured repeat-hit share was 0.96 (injection-slice - record, `smoke.k_rule`, `smoke.k5_repeat_hit_share_measured`). The verify hunts share a prefix - too: in the first plane increment 2,385,152 of 3,820,395 prompt tokens were cache hits - (`benchmarks/independent/plane-increment-1.json`, `detail.cache_hit_tokens`, - `detail.prompt_tokens`; its reading: "Cost fell 17% (62% of prompt tokens were cache hits)"). +- **The tie-break pass runs only on disputes.** + - `agree` compares passes a and b. + - Pass c starts with `inputs.only` set to the case's `disputed.json`. + - An empty list ends it `done` with no model call. + - The final verdict is 2-of-3 ("pass c alone never decides", `agree.py`). +- **Cache-friendly prompt order.** The decision engine's prompt has three parts, in this order + (`learn/program.py`): + 1. The compiled prefix: instruction and fixed demonstrations. It is identical for every + candidate, so the provider's prefix cache hits. + 2. The retrieved examples. + 3. The candidate. + + The self-consistency `k` (the number of samples per candidate) is tied to that cache: "k = 5 + when samples 2..k hit the prefix cache for >= 0.8 of their prompt tokens". The measured + repeat-hit share was 0.96 (injection-slice record, `smoke.k_rule`, + `smoke.k5_repeat_hit_share_measured`). + + The verify hunts share a prefix too. In the first plane increment, 2,385,152 of 3,820,395 + prompt tokens were cache hits (`benchmarks/independent/plane-increment-1.json`, + `detail.cache_hit_tokens`, `detail.prompt_tokens`). The record's reading: "Cost fell 17% (62% + of prompt tokens were cache hits)". ### Measured costs @@ -116,37 +154,41 @@ or a file. | Decision engine harvest: `verify` part | $2.5373 over 1,961 calls, 474 tasks done, 10,921,167 prompt tokens of which 6,275,060 cache hits, 306,941 output tokens, task budgets summing to $8.624 under a $10 ceiling (`verify`) | `benchmarks/measurements/2026-10-01-decision-engine-harvest/record.json` | | Decision engine harvest: `roles` part | $1.4036 over 1,034 calls, 151 tasks done, 2,398,669 prompt tokens of which 329,600 cache hits, 370,155 output tokens, task budgets summing to $4.917 under a $10 ceiling (`roles`) | same record | -The increment-2 record's note on the two denominators: "the reference divides by the 46 +The increment-2 record notes that the two denominators differ: "the reference divides by the 46 candidates before triage; on that basis the plane is 7% cheaper, on the 43 after triage it is 0.2% under". !!! note "The meter prices at list; the account moved less" The attribution table prices every call at the Model's list price - (`plane/models/deepseek-flash.yaml`). The harvest record: "Spend is the plane's attribution - table (`ousast plane status`, task meters priced at plane/models/deepseek-flash.yaml); the - DeepSeek account moved less (balance below)", with the balance at $19.63 before, $18.75 - after `verify` and $18.23 after `roles`. The injection-slice record names the ratio: "the - provider charged ~1/3 of the list-price meter (balance moved $0.30 for $0.91 metered): - DeepSeek's off-peak discount window, presumably; the meter prices at list" - (`spend.balance_note`, balance $18.2 to $17.9). Budgets and gates are set against the - meter, never against the balance. + (`plane/models/deepseek-flash.yaml`). The harvest record says: "Spend is the plane's + attribution table (`ousast plane status`, task meters priced at + plane/models/deepseek-flash.yaml); the DeepSeek account moved less (balance below)". + The balance was $19.63 before, $18.75 after `verify` and $18.23 after `roles`. + The injection-slice record names the ratio: "the provider charged ~1/3 of the list-price + meter (balance moved $0.30 for $0.91 metered): DeepSeek's off-peak discount window, + presumably; the meter prices at list" (`spend.balance_note`, balance $18.2 to $17.9). + Budgets and gates are set against the meter, never against the balance. ### Not done yet -- **No model-side token budget in ax itself.** The plane's requirements state what ax does - not yet carry: "task dependencies, artifacts, token budgets (on ax's roadmap), a DeepSeek - provider"; the thin `Run` layer carries them and dissolves into ax as ax gains them. Until - then the ceiling lives in the metered client, not in the executor. -- **Triage is applied, not run.** The plane has no triage task; it filters by the recorded - triage and adds that recording's cost as its own line. A triage task is planned as a later - specification. -- **`k` and the contrast examples are not knobs.** `k` is a compiled setting in {1, 3, 5} - (`learn/compile.toml`, "k and lam remain A/B variants"), changed only through the decision - engine's A/B experiments, its requirement that "changes to prompts, models, pass counts, rule - sets and features [are] compared by controlled experiments, so that nothing is adopted on one - run's number" (Requirement 4 of its specification). The retrieval cap of at most two - contrast examples from other families is a module constant (`learn/retrieve.py`, - `CONTRAST = 2`), a maintainer decision, not an experiment variant yet. +- **No model-side token budget in ax itself.** + - The plane's requirements state what ax does not yet carry: "task dependencies, artifacts, + token budgets (on ax's roadmap), a DeepSeek provider". + - The thin `Run` layer carries them. It dissolves into ax as ax gains them. + - Until then the ceiling lives in the metered client, not in the executor. +- **Triage is applied, not run.** + - The plane has no triage task. + - It filters by the recorded triage and adds that recording's cost as its own line. + - A triage task is planned as a later specification. +- **`k` and the contrast examples are not knobs.** + - `k` is a compiled setting in {1, 3, 5} (`learn/compile.toml`, "k and lam remain A/B + variants"). + - It changes only through the decision engine's A/B experiments. Requirement 4 of its + specification asks that "changes to prompts, models, pass counts, rule sets and features + [are] compared by controlled experiments, so that nothing is adopted on one run's number". + - Retrieval adds at most two contrast examples from other families. This cap is a module + constant (`learn/retrieve.py`, `CONTRAST = 2`). + - The cap is a maintainer decision, not an experiment variant yet. ## B. Development: the maintainers' own sessions @@ -165,28 +207,47 @@ flowchart LR ### The measurement: tool I/O dominates -`benchmarks/dev/token_report.py` reads a Claude Code session transcript (`.jsonl`; without an -argument, the newest under `~/.claude/projects/`) and reports, per content kind, characters and -share: `tool_use_input` (the tool call's input as JSON), `tool_result`, `thinking`, -`assistant:text`, `user:text`, plus the tool-call count and the top five tools. Tokens are -approximated at four characters each. It prints to stdout and writes nothing; what gets -committed is the counts-only summary. The one in `benchmarks/independent/plane-increment-1.json` -(`dev_token_report`, for "the whole development session (c09e30b2), not the increment alone"): -7.3M unique characters (~1.81M tokens), `tool_use_input_pct` 45.4, `tool_result_pct` 40.1, -`tool_io_pct` 85.5 against `previous_tool_io_pct` 90, over 2,701 tool calls. The reasoning the -session is paid for (assistant text and thinking) is the small remainder. That is why the -remedies below target file contents, not prose. +`benchmarks/dev/token_report.py` reads a Claude Code session transcript (`.jsonl`). Without an +argument it reads the newest one under `~/.claude/projects/`. + +It reports characters and share per content kind: + +- `tool_use_input` (the tool call's input as JSON); +- `tool_result`; +- `thinking`; +- `assistant:text`; +- `user:text`. + +It also reports the tool-call count and the top five tools. Tokens are approximated at four +characters each. It prints to stdout and writes nothing. Only the counts-only summary is +committed. + +The committed summary is in `benchmarks/independent/plane-increment-1.json` +(`dev_token_report`). It covers "the whole development session (c09e30b2), not the increment +alone". It counts 7.3M unique characters (~1.81M tokens). The shares +are `tool_use_input_pct` 45.4, `tool_result_pct` 40.1, and `tool_io_pct` 85.5 against +`previous_tool_io_pct` 90, over 2,701 tool calls. + +The reasoning the session is paid for (assistant text and thinking) is the small remainder. +That is why the remedies below target file contents, not prose. ### The remedies in `benchmarks/dev/` -**`read_guard.py`**, a `PreToolUse` hook for agentic coding sessions. It reads the hook event on -stdin and, for the `Read` tool only, denies a whole-file read of a file longer than 350 lines -(`LIMIT = 350`); a read with `offset` or `limit`, a missing path, an unreadable file or a file -within the limit passes. It always exits 0 ("never fails the tool"). A denial is one JSON line -whose `permissionDecisionReason` tells the agent to read a range or to ask the bulk reader. The -script's docstring records that it is wired in `.claude/settings.local.json` (a local, untracked -file) as `PreToolUse` with matcher `Read`; the setting has this shape, with the command -repo-relative because hooks run from the project directory: +**`read_guard.py`** is a `PreToolUse` hook (a script that runs before each tool call) for +agentic coding sessions. + +- It reads the hook event on stdin. +- For the `Read` tool only, it denies a whole-file read of a file longer than 350 lines + (`LIMIT = 350`). +- These pass: a read with `offset` or `limit`, a missing path, an unreadable file, or a file + within the limit. +- It always exits 0 ("never fails the tool"). +- A denial is one JSON line. Its `permissionDecisionReason` tells the agent to read a range or + to ask the bulk reader. + +The script's docstring records how it is wired. It sits in `.claude/settings.local.json` (a +local, untracked file) as `PreToolUse` with matcher `Read`. The setting has this shape. The +command is repo-relative because hooks run from the project directory: ```json { @@ -203,37 +264,48 @@ repo-relative because hooks run from the project directory: } ``` -**`bulk_read.py`**, the portal pattern: a cheap model reads the files and answers one question, -citing `path:line` for every claim; it reports what the files say and decides nothing. +**`bulk_read.py`** follows the portal pattern. A cheap model reads the files and answers one +question. It cites `path:line` for every claim. It reports what the files say and decides +nothing. ```text .venv/bin/python benchmarks/dev/bulk_read.py "" FILE [FILE ...] [--max-usd 0.05] ``` -Files go to the model in 400-line chunks with a `--- path (lines a-b)` header and numbered -lines; the client is the project's detector client (`deepseek-flash` through `DEEPSEEK_API_KEY`, -otherwise OpenRouter), each call has a 90 s timeout and no tools. `--max-usd` (default 0.05) is -checked before every chunk; at the ceiling the answer ends with -`[stopped at the $X ceiling before: ]`. The answer goes to stdout, the summary -`[bulk_read: N chunk(s), $X.XXXX]` to stderr. Without a chat client it exits 2. +How it works: + +- Files go to the model in 400-line chunks, with a `--- path (lines a-b)` header and numbered + lines. +- The client is the project's detector client: `deepseek-flash` through `DEEPSEEK_API_KEY`, + otherwise OpenRouter. +- Each call has a 90 s timeout and no tools. +- `--max-usd` (default 0.05) is checked before every chunk. At the ceiling the answer ends with + `[stopped at the $X ceiling before: ]`. +- The answer goes to stdout. The summary `[bulk_read: N chunk(s), $X.XXXX]` goes to stderr. +- Without a chat client it exits 2. **`token_report.py`**: `python benchmarks/dev/token_report.py [TRANSCRIPT.jsonl]`, described above. ### Practices -- **Scripts into files, not heredocs.** A heredoc is `tool_use_input` and its output - `tool_result`; both are paid again at every later turn of the session. A script on disk is - read once, by the interpreter. -- **Terse outputs.** Commands print counts, exit codes and the lines that matter; a harness - fails loudly with the exit code rather than printing a table that has to be read back. -- **Subagents with worktrees** for work whose output need not enter the main context: a - subagent's reads and command output stay in its own context and only its conclusion comes - back; a git worktree keeps its edits off the main checkout. +- **Scripts into files, not heredocs.** A heredoc (a script typed inline in a shell command) is + `tool_use_input`, and its output is `tool_result`. Both are paid again at every later turn of + the session. A script on disk is read once, by the interpreter. +- **Terse outputs.** Commands print counts, exit codes and the lines that matter. A harness + fails loudly with the exit code. It does not print a table that has to be read back. +- **Subagents with worktrees** for work whose output need not enter the main context. + - A subagent's reads and command output stay in its own context. + - Only its conclusion comes back. + - A git worktree (a separate checkout of the same repository) keeps its edits off the main + checkout. - **Counts-only measurement records.** A record under `benchmarks/measurements/` or - `benchmarks/independent/` holds numbers and their keys, never code, prompts or responses, so - it can be quoted on a page like this one and reread by a model at a few hundred tokens. + `benchmarks/independent/` holds numbers and their keys. It never holds code, prompts or + responses. So it can be quoted on a page like this one. A model can reread it at a few + hundred tokens. -Related: [The plane on ax](plane.md) for the Run and Task contract, [Memory](memory.md) for -the store the reuse keys live in, [Evaluation](evaluation.md) for the gates these costs are -measured against. +Related: + +- [The plane on ax](plane.md) for the Run and Task contract; +- [Memory](memory.md) for the store the reuse keys live in; +- [Evaluation](evaluation.md) for the gates these costs are measured against.