mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
docs: plain-language pass -- deployment, plane, memory, RustFS, token economics
Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
50f4e69321
commit
7e23379e96
+370
-231
@@ -1,17 +1,21 @@
|
|||||||
# Deployment
|
# Deployment
|
||||||
|
|
||||||
How the agentic plane is deployed today, and what moving it to a separate Kubernetes cluster
|
This page covers two things. First, how the agentic plane is deployed today. Second, what moving it to a
|
||||||
would take. State as of 2026-10-02 (v2.0.0). A scan needs none of this: `ousast scan` and
|
separate Kubernetes cluster would take. It describes the state as of 2026-10-02 (v2.0.0).
|
||||||
`ousast pre-push` run from the Python package (and the Docker image for Joern, see
|
|
||||||
[Engine and pre-push](ops/README.md)). The plane is the maintainers' surface for model work over
|
|
||||||
whole repositories, driven by `ousast plane`.
|
|
||||||
|
|
||||||
Two words are used strictly on this page. **Implemented** means the code on `main` does it and it
|
A scan needs none of this. `ousast scan` and `ousast pre-push` run from the Python package. Joern runs
|
||||||
has run on the maintainer's host. **Planned** means it has not been done; the guidance is derived
|
from the Docker image (see [Engine and pre-push](ops/README.md)). The plane is the maintainers' tool for
|
||||||
from the code and the host's lessons, and the code changes it needs are named with their file paths.
|
model work over whole repositories. `ousast plane` drives it.
|
||||||
|
|
||||||
|
This page uses two words strictly:
|
||||||
|
|
||||||
|
- **Implemented** means the code on `main` does it, and it has run on the maintainer's host.
|
||||||
|
- **Planned** means it has not been done. The guidance comes from the code and from what the host taught
|
||||||
|
us. Each code change it needs is named with its file path.
|
||||||
|
|
||||||
**No deployment of the plane to a cluster other than the maintainer's single-node kind cluster has
|
**No deployment of the plane to a cluster other than the maintainer's single-node kind cluster has
|
||||||
been made.** Section B is guidance, not a record.
|
been made.** Section B is guidance, not a record. (kind runs a Kubernetes cluster inside Docker
|
||||||
|
containers on one machine.)
|
||||||
|
|
||||||
## What runs where today (implemented)
|
## What runs where today (implemented)
|
||||||
|
|
||||||
@@ -51,14 +55,21 @@ flowchart LR
|
|||||||
| Memory store | `OUSAST_MEMORY`: a file store under `~/ousast-results/plane/memory` by default, or `s3://<bucket>[/<prefix>]` on any reachable S3-compatible server (RustFS is the tested one) | `src/openultrasast/plane/memory.py`, [Memory](memory.md), [RustFS setup](rustfs.md) |
|
| Memory store | `OUSAST_MEMORY`: a file store under `~/ousast-results/plane/memory` by default, or `s3://<bucket>[/<prefix>]` on any reachable S3-compatible server (RustFS is the tested one) | `src/openultrasast/plane/memory.py`, [Memory](memory.md), [RustFS setup](rustfs.md) |
|
||||||
| Run state and artifacts | `$OUSAST_RESULTS/plane/<run>/` on the host (default `~/ousast-results/plane/`) | `reconciler.py` (`run_dir`) |
|
| Run state and artifacts | `$OUSAST_RESULTS/plane/<run>/` on the host (default `~/ousast-results/plane/`) | `reconciler.py` (`run_dir`) |
|
||||||
|
|
||||||
Only the reconciler talks to the memory store: a sandboxed task cannot read it, so the loop's
|
Terms in the table:
|
||||||
`memory-snapshot` is seeded by the reconciler before the Run ([The plane on ax](plane.md)).
|
|
||||||
|
- **ax** is Google's controller that runs agent tasks on Kubernetes.
|
||||||
|
- **Agent Substrate** is the layer below ax that owns the machines and runs sandboxes.
|
||||||
|
- **gVisor** is a sandbox runtime that isolates each actor from the node's kernel.
|
||||||
|
- An **actor** is one running Task inside a sandbox.
|
||||||
|
|
||||||
|
Only the reconciler talks to the memory store. A sandboxed task cannot read it. So the reconciler seeds
|
||||||
|
the loop's `memory-snapshot` before the Run starts ([The plane on ax](plane.md)).
|
||||||
|
|
||||||
## A. This host: the local setup (implemented)
|
## A. This host: the local setup (implemented)
|
||||||
|
|
||||||
What `ops/ax/up.sh` needs on the PATH: `docker`, `go`, `kubectl`, `kind`, `ko` and `ax` (it
|
`ops/ax/up.sh` needs these tools on the PATH: `docker`, `go`, `kubectl`, `kind`, `ko` and `ax`. It
|
||||||
refuses to start when one is missing). Tools live in `~/go/bin` and `~/.local/bin`; sources and
|
refuses to start when one is missing. Tools live in `~/go/bin` and `~/.local/bin`. Sources and rendered
|
||||||
rendered files in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`).
|
files live in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`).
|
||||||
|
|
||||||
1. **Bring up the cluster and the plane.**
|
1. **Bring up the cluster and the plane.**
|
||||||
|
|
||||||
@@ -66,15 +77,20 @@ rendered files in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`).
|
|||||||
ops/ax/up.sh
|
ops/ax/up.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
Idempotent: each step is skipped when already done. It creates the kind cluster
|
The script is idempotent: it skips each step that is already done. In order, it:
|
||||||
(`KIND_CLUSTER_NAME`, default `ousast`) with its registry (`KO_DOCKER_REPO`, default
|
|
||||||
`localhost:5001`), installs Agent Substrate, applies the egress gateway (the agentgateway
|
- creates the kind cluster (`KIND_CLUSTER_NAME`, default `ousast`) with its registry
|
||||||
variant, one prebuilt image) when the install left it out, deploys the ax control plane, applies
|
(`KO_DOCKER_REPO`, default `localhost:5001`);
|
||||||
the receiver Service with the host's kind-gateway address as its endpoint, applies the gVisor
|
- installs Agent Substrate;
|
||||||
worker pool, builds and pushes the runner image and writes its digest pin to
|
- applies the egress gateway if the install left it out (the agentgateway variant, one prebuilt image);
|
||||||
`~/.cache/ousast/ax-src/runner-image`, installs the `ax` CLI from the deployed checkout (a release
|
- deploys the ax control plane;
|
||||||
CLI skews from the server), and runs one smoke Task. It ends by printing the host's memory and the
|
- applies the receiver Service, with the host's kind-gateway address as its endpoint;
|
||||||
kind containers' usage.
|
- applies the gVisor worker pool;
|
||||||
|
- builds and pushes the runner image, and writes its digest pin to `~/.cache/ousast/ax-src/runner-image`;
|
||||||
|
- installs the `ax` CLI from the deployed checkout (a release CLI does not match the server version);
|
||||||
|
- runs one smoke Task.
|
||||||
|
|
||||||
|
At the end it prints the host's memory and the kind containers' usage.
|
||||||
|
|
||||||
2. **Check it.**
|
2. **Check it.**
|
||||||
|
|
||||||
@@ -82,9 +98,15 @@ rendered files in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`).
|
|||||||
uv run ousast plane doctor
|
uv run ousast plane doctor
|
||||||
```
|
```
|
||||||
|
|
||||||
Four checks: the kind context is reachable, every pod in `ate-system` and in `ax-system` is ready,
|
It runs four checks:
|
||||||
and the registry at `localhost:5001` holds `ousast-runner`. Each failing line names
|
|
||||||
`ops/ax/up.sh` as the fix. `doctor` is informational: `ousast plane run` does not call it.
|
- the kind context is reachable;
|
||||||
|
- every pod in `ate-system` is ready;
|
||||||
|
- every pod in `ax-system` is ready;
|
||||||
|
- the registry at `localhost:5001` holds `ousast-runner`.
|
||||||
|
|
||||||
|
Each failing line names `ops/ax/up.sh` as the fix. `doctor` is informational only.
|
||||||
|
`ousast plane run` does not call it.
|
||||||
|
|
||||||
3. **Run the end-to-end smoke Run.**
|
3. **Run the end-to-end smoke Run.**
|
||||||
|
|
||||||
@@ -92,23 +114,27 @@ rendered files in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`).
|
|||||||
ops/ax/smoke-run.sh
|
ops/ax/smoke-run.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
It renders `ops/ax/smoke-run.yaml.tmpl` with the current runner digest and submits the Run
|
It renders `ops/ax/smoke-run.yaml.tmpl` with the current runner digest. It then submits the Run
|
||||||
`ax-e2e-smoke` through the reconciler: one model-free `repo-facts` Task (`budget: {usd: 0,
|
`ax-e2e-smoke` through the reconciler. The Run has one model-free `repo-facts` Task (`budget: {usd: 0,
|
||||||
calls: 0}`) over a files-only Workspace, resumed, started through the router, its `facts.json` and
|
calls: 0}`) over a files-only Workspace, resumed, started through the router, its `facts.json`
|
||||||
`summary.json` delivered to the host, then `ousast plane status ax-e2e-smoke` prints the attribution
|
and `summary.json` delivered to the host. Then `ousast plane status ax-e2e-smoke`
|
||||||
table. The script calls the repository's `.venv/bin/ousast`, so `uv sync` first.
|
prints the attribution table. The script calls the repository's `.venv/bin/ousast`, so run `uv sync`
|
||||||
|
first.
|
||||||
|
|
||||||
4. **Keys and the store.** Put `DEEPSEEK_API_KEY` (chat) and `OPENROUTER_API_KEY` (embeddings) in
|
4. **Keys and the store.**
|
||||||
`.env` or export them; a Task's Model names the variable (`secretKey.key`), and the key travels
|
|
||||||
only in that Task's start request. Set `OUSAST_MEMORY=s3://<bucket>` with `S3_ENDPOINT`,
|
- Put `DEEPSEEK_API_KEY` (chat) and `OPENROUTER_API_KEY` (embeddings) in `.env`, or export them.
|
||||||
`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` (optional `S3_REGION`, `S3_BUCKET`) for the S3 store;
|
- A Task's Model names the variable (`secretKey.key`). The key travels only in that Task's start
|
||||||
the bucket must be set up once by an admin as [RustFS setup](rustfs.md) describes, and the store
|
request.
|
||||||
verifies it at every start.
|
- For the S3 store, set `OUSAST_MEMORY=s3://<bucket>` with `S3_ENDPOINT`, `AWS_ACCESS_KEY_ID` and
|
||||||
|
`AWS_SECRET_ACCESS_KEY`. `S3_REGION` and `S3_BUCKET` are optional.
|
||||||
|
- An admin must set up the bucket once, as [RustFS setup](rustfs.md) describes. The store verifies
|
||||||
|
it at every start.
|
||||||
|
|
||||||
5. **Run a real Run**, as in [Examples](examples.md#7-running-the-agentic-plane): `ousast plane run
|
5. **Run a real Run**, as in [Examples](examples.md#7-running-the-agentic-plane): `ousast plane run
|
||||||
plane/runs/validation-46.yaml`, then `status` and `remember`.
|
plane/runs/validation-46.yaml`, then `status` and `remember`.
|
||||||
|
|
||||||
6. **Tear down** with `ops/ax/down.sh`, which deletes the cluster and its registry.
|
6. **Tear down** with `ops/ax/down.sh`. It deletes the cluster and its registry.
|
||||||
|
|
||||||
**Footprint** (measured 2026-09-29, idle, two workers; [ax on this host](ops/ax/README.md#measured-footprint-2026-09-29-idle-two-workers)):
|
**Footprint** (measured 2026-09-29, idle, two workers; [ax on this host](ops/ax/README.md#measured-footprint-2026-09-29-idle-two-workers)):
|
||||||
|
|
||||||
@@ -119,44 +145,62 @@ rendered files in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`).
|
|||||||
| node volume (images, containerd) | 4.9 GB on disk |
|
| node volume (images, containerd) | 4.9 GB on disk |
|
||||||
| worker pool | 2 gVisor workers, 1 CPU / 1.5 GiB limit each |
|
| worker pool | 2 gVisor workers, 1 CPU / 1.5 GiB limit each |
|
||||||
|
|
||||||
Disk is the binding constraint: the Substrate install and the `ko` builds need several GB of Go
|
Disk is the tightest limit, for two reasons:
|
||||||
caches (`go clean -cache -modcache` afterwards), and every Task is its own actor template with a
|
|
||||||
golden actor (about 24 MB each under `/var/lib/ate/actors` on the node). The failures the host
|
- The Substrate install and the `ko` builds need several GB of Go caches. Run `go clean -cache -modcache`
|
||||||
produced, and what each one taught, are listed in
|
afterwards.
|
||||||
[ax on this host](ops/ax/README.md#what-this-host-taught-each-one-cost-a-failed-live-run).
|
- Every Task is its own actor template with a golden actor (a saved snapshot the actor boots from). Each
|
||||||
|
takes about 24 MB under `/var/lib/ate/actors` on the node.
|
||||||
|
|
||||||
|
[ax on this host](ops/ax/README.md#what-this-host-taught-each-one-cost-a-failed-live-run) lists the
|
||||||
|
failures the host produced and what each one taught.
|
||||||
|
|
||||||
## B. A separate Kubernetes cluster (planned)
|
## B. A separate Kubernetes cluster (planned)
|
||||||
|
|
||||||
The target: a cluster that runs Agent Substrate with its egress gateway and the ax control plane,
|
The target cluster would do four things:
|
||||||
pulls the runner image from a registry of yours, receives task artifacts without a developer
|
|
||||||
laptop in the path, and keeps its memory in your S3 bucket. The provider key already travels only
|
- run Agent Substrate with its egress gateway and the ax control plane;
|
||||||
in the start request through `atenet-router`, which works the same through a port-forward to any
|
- pull the runner image from a registry of yours;
|
||||||
cluster. Everything else below is either a step you take or a code change that is not made yet.
|
- receive task artifacts without a developer laptop in the path;
|
||||||
|
- keep its memory in your S3 bucket.
|
||||||
|
|
||||||
|
The provider key already travels only in the start request through `atenet-router`. That works the same
|
||||||
|
through a port-forward to any cluster. Everything else below is either a step you take or a code change
|
||||||
|
that is not made yet.
|
||||||
|
|
||||||
### B.1 Cluster with Agent Substrate, its egress gateway and ax
|
### B.1 Cluster with Agent Substrate, its egress gateway and ax
|
||||||
|
|
||||||
- Install Agent Substrate on the cluster following its own instructions. `ops/ax/up.sh` uses
|
- Install Agent Substrate on the cluster with its own instructions. `ops/ax/up.sh` uses Substrate's kind
|
||||||
Substrate's kind helpers (`hack/create-kind-cluster.sh`, `hack/install-ate-kind.sh`), which do
|
helpers (`hack/create-kind-cluster.sh`, `hack/install-ate-kind.sh`). These do not apply to another
|
||||||
not apply to another cluster.
|
cluster.
|
||||||
- Apply the egress gateway. On this host it is the agentgateway variant
|
- Apply the egress gateway. On this host it is the agentgateway variant
|
||||||
(`manifests/ate-install/agentgateway-egress` in the Substrate checkout, applied with
|
(`manifests/ate-install/agentgateway-egress` in the Substrate checkout). It is applied with
|
||||||
`kubectl kustomize --load-restrictor=LoadRestrictionsNone`), which needs no Rust build; the
|
`kubectl kustomize --load-restrictor=LoadRestrictionsNone`. It needs no Rust build; the Envoy variant
|
||||||
Envoy variant does. Without the gateway an actor has no network at all, and the per-task
|
does.
|
||||||
EgressPolicies the reconciler writes (`src/openultrasast/plane/egress.py`) work unchanged
|
- Without the gateway an actor has no network at all.
|
||||||
against the Substrate-installed gateway: deny by default, hostnames only, plain HTTP to the
|
- The reconciler writes one EgressPolicy per task (`src/openultrasast/plane/egress.py`). These work
|
||||||
receiver, TLS passthrough to the Workspaces' Git hosts and the Model's declared hosts.
|
unchanged against the gateway that Substrate installs. The rules are:
|
||||||
|
- deny by default;
|
||||||
|
- hostnames only;
|
||||||
|
- plain HTTP to the receiver;
|
||||||
|
- TLS passthrough to the Workspaces' Git hosts and the Model's declared hosts.
|
||||||
- Deploy ax (`make deploy AX_IMAGE_REPO=<your registry>` in the ax checkout) **after** B.2.
|
- Deploy ax (`make deploy AX_IMAGE_REPO=<your registry>` in the ax checkout) **after** B.2.
|
||||||
- Apply a WorkerPool (B.6). Substrate installs none by itself ("no free workers").
|
- Apply a WorkerPool (B.6). Substrate installs none by itself ("no free workers").
|
||||||
|
|
||||||
### B.2 ax's snapshot bucket: `AX_SNAPSHOTS_BUCKET`
|
### B.2 ax's snapshot bucket: `AX_SNAPSHOTS_BUCKET`
|
||||||
|
|
||||||
ax's deploy manifest (`deploy/ax-server.yaml` in the ax checkout) sets `AX_SNAPSHOTS_BUCKET` to
|
Point `AX_SNAPSHOTS_BUCKET` at **your own bucket before deploying ax**. ax's deploy manifest
|
||||||
the ax authors' own bucket (`gs://dberkov-gke-dev3/ate-env/` in the checkout used here). Point it
|
(`deploy/ax-server.yaml` in the ax checkout) sets it to the ax authors' own bucket. In the checkout used
|
||||||
at **your own bucket before deploying ax**. On this host the bucket named there did not exist in
|
here that is `gs://dberkov-gke-dev3/ate-env/`.
|
||||||
the S3 server, deleted golden actors piled up in `DELETING` and filled the disk during a 600-task
|
|
||||||
harvest (2026-10-01); creating the bucket let deletes complete, and even then a deleted actor's
|
This host shows what goes wrong otherwise (2026-10-01):
|
||||||
directory stays on the node, so long Runs need a janitor that removes directories no live actor
|
|
||||||
owns.
|
1. The bucket named there did not exist in the S3 server.
|
||||||
|
2. Deleted golden actors piled up in `DELETING`.
|
||||||
|
3. They filled the disk during a 600-task harvest.
|
||||||
|
|
||||||
|
Creating the bucket let deletes complete. Even then, a deleted actor's directory stays on the node. So
|
||||||
|
long Runs need a janitor that removes directories no live actor owns.
|
||||||
|
|
||||||
### B.3 A registry the workers can pull from, the runner image pinned by digest
|
### B.3 A registry the workers can pull from, the runner image pinned by digest
|
||||||
|
|
||||||
@@ -168,83 +212,117 @@ owns.
|
|||||||
docker inspect --format '{{range .RepoDigests}}{{println .}}{{end}}' <registry>/ousast-runner:dev
|
docker inspect --format '{{range .RepoDigests}}{{println .}}{{end}}' <registry>/ousast-runner:dev
|
||||||
```
|
```
|
||||||
|
|
||||||
- Substrate rejects tags: every Task's `image` must be the `<registry>/ousast-runner@sha256:...`
|
- Substrate rejects tags. Every Task's `image` must be the `<registry>/ousast-runner@sha256:...`
|
||||||
reference. Write it into a file (on this host, `~/.cache/ousast/ax-src/runner-image`) and pass
|
reference.
|
||||||
`--runner-image FILE` to `ousast plane workspaces` and `ousast plane harvest`, which re-pin the
|
- Write that reference into a file. On this host the file is `~/.cache/ousast/ax-src/runner-image`.
|
||||||
Tasks they generate. The committed templates under `plane/tasks/` and the generated
|
- Pass `--runner-image FILE` to `ousast plane workspaces` and `ousast plane harvest`. They re-pin the
|
||||||
`plane/tasks/validation-46.yaml` carry `localhost:5001/ousast-runner@sha256:...` and need the same
|
Tasks they generate.
|
||||||
re-pin (regenerate, or replace the reference) before they run elsewhere.
|
- The committed templates under `plane/tasks/` and the generated `plane/tasks/validation-46.yaml` carry
|
||||||
- Workers must be able to pull from the registry (credentials, if any, are the cluster's concern;
|
`localhost:5001/ousast-runner@sha256:...`. They need the same re-pin before they run elsewhere.
|
||||||
ax's default runner image on `gcr.io` needs credentials and is not used here).
|
Regenerate them, or replace the reference.
|
||||||
|
- Workers must be able to pull from the registry. Credentials, if any, are the cluster's concern. ax's
|
||||||
|
default runner image on `gcr.io` needs credentials and is not used here.
|
||||||
- **Not yet configurable:** `ousast plane doctor` checks the registry at `localhost:5001` by name
|
- **Not yet configurable:** `ousast plane doctor` checks the registry at `localhost:5001` by name
|
||||||
(`src/openultrasast/plane/doctor.py:45`), so its fourth check fails on another cluster even when
|
(`src/openultrasast/plane/doctor.py:45`). So its fourth check fails on another cluster, even when the
|
||||||
the image is in place. `doctor` is informational, `plane run` does not depend on it.
|
image is in place. `doctor` is informational; `plane run` does not depend on it.
|
||||||
|
|
||||||
### B.4 One configurable kube context
|
### B.4 One configurable kube context
|
||||||
|
|
||||||
The reconciler, the egress client and the router port-forward all use the context
|
The reconciler, the egress client and the router port-forward all use one kube context. Its name is
|
||||||
`kind-$KIND_CLUSTER_NAME` (`kind-ousast` by default). **Not yet configurable**: the prefix `kind-`
|
`kind-$KIND_CLUSTER_NAME` (`kind-ousast` by default).
|
||||||
is hard-coded in `src/openultrasast/plane/doctor.py:39`, `src/openultrasast/plane/egress.py:115`
|
|
||||||
and `:148`, and `src/openultrasast/plane/router.py:145`; `ops/ax/up.sh:24` builds the same name.
|
**Not yet configurable**: the prefix `kind-` is hard-coded in these places:
|
||||||
The planned change is one variable (for example `OUSAST_KUBE_CONTEXT`) read in those three
|
|
||||||
modules. Until then the only way to drive another cluster from the unchanged code is to give its
|
- `src/openultrasast/plane/doctor.py:39`;
|
||||||
kubeconfig context a name of that form (`kubectl config rename-context <ctx> kind-<name>` and
|
- `src/openultrasast/plane/egress.py:115` and `:148`;
|
||||||
`KIND_CLUSTER_NAME=<name>`), which is a workaround, not a supported configuration. The egress
|
- `src/openultrasast/plane/router.py:145`;
|
||||||
client also needs `kubectl-ate` (or the executable `OUSAST_KUBECTL_ATE` names) able to use that
|
- `ops/ax/up.sh:24` builds the same name.
|
||||||
context.
|
|
||||||
|
The planned change is one variable read in those three modules, for example `OUSAST_KUBE_CONTEXT`.
|
||||||
|
|
||||||
|
Until then, there is one workaround to drive another cluster with the unchanged code. Rename its
|
||||||
|
kubeconfig context to that form: `kubectl config rename-context <ctx> kind-<name>` and
|
||||||
|
`KIND_CLUSTER_NAME=<name>`. This is a workaround, not a supported configuration.
|
||||||
|
|
||||||
|
The egress client also needs `kubectl-ate` (or the executable `OUSAST_KUBECTL_ATE` names). It must be
|
||||||
|
able to use that context.
|
||||||
|
|
||||||
### B.5 The receiver: in-cluster instead of on the host
|
### B.5 The receiver: in-cluster instead of on the host
|
||||||
|
|
||||||
Today the receiver is a thread inside the reconciler process (`egress.py`, `Receiver`), and the
|
Today the receiver is a thread inside the reconciler process (`egress.py`, `Receiver`). The reconciler
|
||||||
reconciler learns of a task's completion by watching that object's `delivered` set in memory
|
learns that a task finished by watching that object's `delivered` set in memory (`reconciler.py`).
|
||||||
(`reconciler.py`). Actors reach it through the Service `ousast-receiver.ax-system` whose
|
Actors reach it through the Service `ousast-receiver.ax-system`. That Service's EndpointSlice points at
|
||||||
EndpointSlice points at the host. Three paths, in order of effort:
|
the host.
|
||||||
|
|
||||||
1. **Interim, works with the unchanged code:** keep the host receiver and render
|
There are three paths, from least to most effort:
|
||||||
`ops/ax/receiver-service.yaml.tmpl` with an endpoint address the egress gateway pod can reach
|
|
||||||
(`KIND_GATEWAY` is just an IPv4 address in the template, `OUSAST_ARTIFACT_PORT` the port). The
|
1. **Interim, works with the unchanged code:** keep the host receiver.
|
||||||
reconciler's host must be reachable from the cluster (a VPN or a routed address), and the
|
Render `ops/ax/receiver-service.yaml.tmpl` with an endpoint address the egress gateway pod can reach.
|
||||||
gateway dials the address the actor connected to, so the Service's ClusterIP must still map to it.
|
In the template, `KIND_GATEWAY` is just an IPv4 address and `OUSAST_ARTIFACT_PORT` is the port.
|
||||||
2. **Planned: the receiver as an in-cluster Deployment** behind the same Service name, with the
|
The cluster must be able to reach the reconciler's host, through a VPN or a routed address.
|
||||||
reconciler reading deliveries and serving inputs from it instead of from its own thread. This
|
The gateway dials the address the actor connected to. So the Service's ClusterIP must still map to it.
|
||||||
needs code: the receiver and its `delivered`/`inputs` state are in-process today, and the
|
2. **Planned: the receiver as an in-cluster Deployment** behind the same Service name.
|
||||||
runner's `GET /inputs/<producer>/<artifact>` is served by that same process.
|
The reconciler would read deliveries from it and serve inputs from it, instead of from its own thread.
|
||||||
3. **Follow-on: presigned-URL delivery to the S3 store.** The runner would `PUT` its output tar to a
|
This needs code. The receiver and its `delivered`/`inputs` state are in-process today.
|
||||||
presigned URL on the memory store's bucket and fetch inputs the same way, and the reconciler
|
The same process serves the runner's `GET /inputs/<producer>/<artifact>`.
|
||||||
would read from the bucket. Nothing of this exists; the EgressPolicy would then have to allow
|
3. **Follow-on: presigned-URL delivery to the S3 store.** (A presigned URL grants one upload or download
|
||||||
the store's hostname for the task.
|
without credentials.)
|
||||||
|
The runner would `PUT` its output tar to a presigned URL on the memory store's bucket.
|
||||||
|
It would fetch inputs the same way, and the reconciler would read from the bucket.
|
||||||
|
None of this exists. The EgressPolicy would then have to allow the store's hostname for the task.
|
||||||
|
|
||||||
### B.6 Worker pool sizing
|
### B.6 Worker pool sizing
|
||||||
|
|
||||||
`ops/ax/workerpool.yaml.tmpl` declares two warm gVisor workers of 1 CPU / 1.5 GiB (requests 250m /
|
`ops/ax/workerpool.yaml.tmpl` declares two warm gVisor workers for the 7 GB host. Each has 1 CPU /
|
||||||
1.5 GiB) for the 7 GB host, selected onto nodes by `ate.dev/substrate-version`. An actor occupies a
|
1.5 GiB (requests 250m / 1.5 GiB). They are placed on nodes by `ate.dev/substrate-version`.
|
||||||
whole worker, so the pool size is the number of tasks that can run at once; set `--workers` on
|
|
||||||
`ousast plane run` to match. Size the workers above the largest Task's limits (`verify`: requests
|
An actor occupies a whole worker. So the pool size is the number of tasks that can run at once. Set
|
||||||
250m / 384Mi, limits 1 CPU / 1 GiB; `roles` and the others are in `plane/tasks/`), and keep the
|
`--workers` on `ousast plane run` to match.
|
||||||
golden-actor storage on the node in mind (B.2).
|
|
||||||
|
Size the workers above the largest Task's limits. For `verify` these are requests 250m / 384Mi and
|
||||||
|
limits 1 CPU / 1 GiB. `roles` and the others are in `plane/tasks/`. Also keep the golden-actor storage
|
||||||
|
on the node in mind (B.2).
|
||||||
|
|
||||||
### B.7 The S3 memory store reachable from where the reconciler runs
|
### B.7 The S3 memory store reachable from where the reconciler runs
|
||||||
|
|
||||||
The `s3://` store already works against any reachable S3-compatible server with versioning,
|
The `s3://` store already works against any reachable S3-compatible server. The server must support:
|
||||||
lifecycle rules, object tags and S3 Select (RustFS is the tested one; the bucket setup and the
|
|
||||||
agent policy are on [RustFS setup](rustfs.md)). Only the reconciler reads and writes it, so today
|
- versioning;
|
||||||
it must be reachable from the operator's host; if the reconciler moves into the cluster, from its
|
- lifecycle rules;
|
||||||
pod. No Task talks to the store directly; the day one does (B.5, path 3), that Task's EgressPolicy
|
- object tags;
|
||||||
must allow the store's hostname (the policies allow hostnames only, never addresses), the way a
|
- S3 Select (a server-side query over JSON objects).
|
||||||
Model declares its hosts today through `openultrasast.io/egress-hosts`.
|
|
||||||
|
RustFS is the tested server. The bucket setup and the agent policy are on [RustFS setup](rustfs.md).
|
||||||
|
|
||||||
|
Only the reconciler reads and writes the store. So today the operator's host must reach it. If the
|
||||||
|
reconciler moves into the cluster, its pod must reach it.
|
||||||
|
|
||||||
|
No Task talks to the store directly. If one ever does (B.5, path 3), its EgressPolicy must allow the
|
||||||
|
store's hostname. The policies allow hostnames only, never addresses. A Model already declares its hosts
|
||||||
|
this way today, through `openultrasast.io/egress-hosts`.
|
||||||
|
|
||||||
### B.8 Secrets in Kubernetes Secrets, injected into the reconciler's environment
|
### B.8 Secrets in Kubernetes Secrets, injected into the reconciler's environment
|
||||||
|
|
||||||
The reconciler reads the variable a Model's `secretKey.key` names (`DEEPSEEK_API_KEY`,
|
The reconciler reads these variables from its own environment or `.env`:
|
||||||
`OPENROUTER_API_KEY`) and the store's `S3_ENDPOINT`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`
|
|
||||||
(optional `AWS_SESSION_TOKEN`, `S3_REGION`, `S3_BUCKET`) from its own environment or `.env`. When
|
- the variable a Model's `secretKey.key` names (`DEEPSEEK_API_KEY`, `OPENROUTER_API_KEY`);
|
||||||
the reconciler runs in the cluster, keep them in Kubernetes Secrets and inject them into **the
|
- the store's `S3_ENDPOINT`, `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`;
|
||||||
reconciler's** pod environment (`envFrom` with `secretRef`, or `valueFrom.secretKeyRef` per
|
- optionally `AWS_SESSION_TOKEN`, `S3_REGION` and `S3_BUCKET`.
|
||||||
variable). Never put a value into a manifest: the Model manifests name a variable, the rendered
|
|
||||||
Task carries no key, and the key reaches an actor only in the start request, where the runner keeps
|
When the reconciler runs in the cluster, keep these in Kubernetes Secrets. Inject them into **the
|
||||||
it in memory and redacts it from echoed stderr. The `secretKey.name` values in `plane/models/`
|
reconciler's** pod environment. Use `envFrom` with `secretRef`, or `valueFrom.secretKeyRef` per variable.
|
||||||
(`deepseek`, `openrouter`) are the names such Secrets would carry. Running the reconciler
|
|
||||||
in-cluster is itself planned: today it is a host process, and its state directory
|
Never put a value into a manifest. The key stays out of every manifest:
|
||||||
|
|
||||||
|
- the Model manifests name a variable;
|
||||||
|
- the rendered Task carries no key;
|
||||||
|
- the key reaches an actor only in the start request;
|
||||||
|
- the runner keeps it in memory and redacts it from echoed stderr.
|
||||||
|
|
||||||
|
The `secretKey.name` values in `plane/models/` (`deepseek`, `openrouter`) are the names such Secrets
|
||||||
|
would carry.
|
||||||
|
|
||||||
|
Running the reconciler in-cluster is itself planned. Today it is a host process. Its state directory
|
||||||
(`OUSAST_RESULTS`) would need a persistent volume.
|
(`OUSAST_RESULTS`) would need a persistent volume.
|
||||||
|
|
||||||
### What is implemented and what is planned
|
### What is implemented and what is planned
|
||||||
@@ -266,12 +344,20 @@ in-cluster is itself planned: today it is a host process, and its state director
|
|||||||
|
|
||||||
## Production topology: ax on a Kubernetes cluster
|
## Production topology: ax on a Kubernetes cluster
|
||||||
|
|
||||||
The maintainer's questions, answered from the code and the upstream checkouts under
|
This section answers the maintainer's questions from the code and the upstream checkouts under
|
||||||
`~/.cache/ousast/ax-src/`: how is it hosted in ax, is there one container, how does ax relate to
|
`~/.cache/ousast/ax-src/`:
|
||||||
Kubernetes, and what scaling the pre-push hook would take. The decisions of 2026-10-02 set the
|
|
||||||
target: **two profiles of one code base**, `kind` (this laptop, kept as a first-class local
|
- How is it hosted in ax?
|
||||||
execution mode, where every change is proven first) and `k3s` (production). Everything marked
|
- Is there one container?
|
||||||
**planned** is not built; the rest is what runs on this host today.
|
- How does ax relate to Kubernetes?
|
||||||
|
- What would scaling the pre-push hook take?
|
||||||
|
|
||||||
|
The decisions of 2026-10-02 set the target: **two profiles of one code base**.
|
||||||
|
|
||||||
|
- `kind` is this laptop. It stays a first-class local execution mode. Every change is proven here first.
|
||||||
|
- `k3s` is production. (k3s is a small Kubernetes distribution.)
|
||||||
|
|
||||||
|
Everything marked **planned** is not built. The rest is what runs on this host today.
|
||||||
|
|
||||||
```mermaid
|
```mermaid
|
||||||
flowchart LR
|
flowchart LR
|
||||||
@@ -308,61 +394,92 @@ flowchart LR
|
|||||||
cli -- "polls: status, attribution" --> s3
|
cli -- "polls: status, attribution" --> s3
|
||||||
```
|
```
|
||||||
|
|
||||||
No inbound path exists in this picture: the CLI reaches ax-server and the router from outside, the
|
This picture has no inbound path:
|
||||||
tasks push their artifacts to the store through the egress gateway, and the CLI reads the store.
|
|
||||||
Today's EndpointSlice back to a laptop (`ops/ax/receiver-service.yaml.tmpl`) is the kind profile's
|
- the CLI reaches ax-server and the router from outside;
|
||||||
device and does not appear in production.
|
- the tasks push their artifacts to the store through the egress gateway;
|
||||||
|
- the CLI reads the store.
|
||||||
|
|
||||||
|
Today's EndpointSlice back to a laptop (`ops/ax/receiver-service.yaml.tmpl`) belongs to the kind
|
||||||
|
profile only. It does not appear in production.
|
||||||
|
|
||||||
### ax, Kubernetes and Agent Substrate in three sentences
|
### ax, Kubernetes and Agent Substrate in three sentences
|
||||||
|
|
||||||
**ax is a Kubernetes application**: a Deployment `ax-server` with a Redis beside it in the
|
**ax is a Kubernetes application.**
|
||||||
namespace `ax-system` (`ax/deploy/ax-server.yaml` in the ax checkout; its controller runs with
|
|
||||||
`--template=default-template --template-atespace=ax-system`), and the `ax` CLI reaches it through a
|
|
||||||
`kubectl port-forward` to `svc/ax-server` (`ax/internal/tunnel/tunnel.go`), so `ousast plane run`
|
|
||||||
needs a kube context or an ingress, not a public endpoint. **ax does not run containers itself**:
|
|
||||||
it turns every Task into an ActorTemplate on Agent Substrate (`ax/internal/substrate/client.go`,
|
|
||||||
`CreateActorTemplate`), built from the Task's image and environment, and the image must be pinned
|
|
||||||
by digest (Substrate's `SandboxConfig` validation rejects tags: "All images must include a
|
|
||||||
digest"). **Agent Substrate is the layer that owns the machines**: in the namespace `ate-system`
|
|
||||||
it runs `ate-api-server`, `ate-controller`, the `atelet` DaemonSet, `atenet-router`, the
|
|
||||||
`atenet-egress` gateway, a Postgres and a RustFS (`substrate/manifests/ate-install/`), and three
|
|
||||||
CRDs under `ate.dev/v1alpha1`, `WorkerPool`, `SandboxConfig` and `CSIDriverConfig`; a
|
|
||||||
`WorkerPool` keeps warm worker pods (`ko://.../cmd/ateom-gvisor`), and an actor is one gVisor
|
|
||||||
sandbox inside one of those workers, booted from the golden snapshot that
|
|
||||||
[The plane on ax](plane.md) describes, suspended and resumed by the controller. Names are limited
|
|
||||||
to 63 bytes (`MaxNameLength` in `ax/pkg/apis/v1alpha1/types.go`), which is why the generated Task
|
|
||||||
names are short.
|
|
||||||
|
|
||||||
**What k3s changes** (planned; nothing has been applied to a k3s cluster): k3s runs containerd
|
- It is a Deployment `ax-server` with a Redis beside it, in the namespace `ax-system`
|
||||||
with no Docker socket, so the host-Docker path of `ousast plane alerts-engine` does not exist
|
(`ax/deploy/ax-server.yaml` in the ax checkout).
|
||||||
there and the engine must become a task (below); Substrate's gVisor workers need `runsc` and a
|
- Its controller runs with `--template=default-template --template-atespace=ax-system`.
|
||||||
`RuntimeClass` on the nodes they are scheduled to; k3s ships Traefik (the ingress the CLI can use
|
- The `ax` CLI reaches it through a `kubectl port-forward` to `svc/ax-server`
|
||||||
instead of a kubeconfig) and local-path storage. Images are built by GitHub Actions from tagged
|
(`ax/internal/tunnel/tunnel.go`).
|
||||||
commits, published to GitHub Container Registry and digest-pinned in the manifests; a private
|
- So `ousast plane run` needs a kube context or an ingress, not a public endpoint.
|
||||||
package needs an `imagePullSecret` on the worker ServiceAccount. Substrate's and ax's own images
|
|
||||||
are built with `ko` from their checkouts and pushed to the same registry. The kind profile keeps
|
**ax does not run containers itself.**
|
||||||
`localhost:5001` and `ops/ax/up.sh`.
|
|
||||||
|
- It turns every Task into an ActorTemplate on Agent Substrate (`ax/internal/substrate/client.go`,
|
||||||
|
`CreateActorTemplate`).
|
||||||
|
- The ActorTemplate is built from the Task's image and environment.
|
||||||
|
- The image must be pinned by digest. Substrate's `SandboxConfig` validation rejects tags: "All images
|
||||||
|
must include a digest".
|
||||||
|
|
||||||
|
**Agent Substrate is the layer that owns the machines.**
|
||||||
|
|
||||||
|
- In the namespace `ate-system` it runs `ate-api-server`, `ate-controller`, the `atelet` DaemonSet,
|
||||||
|
`atenet-router`, the `atenet-egress` gateway, a Postgres and a RustFS
|
||||||
|
(`substrate/manifests/ate-install/`).
|
||||||
|
- It defines three CRDs (custom Kubernetes resource types) under `ate.dev/v1alpha1`: `WorkerPool`,
|
||||||
|
`SandboxConfig` and `CSIDriverConfig`.
|
||||||
|
- A `WorkerPool` keeps warm worker pods (`ko://.../cmd/ateom-gvisor`).
|
||||||
|
- An actor is one gVisor sandbox inside one of those workers.
|
||||||
|
- The actor boots from the golden snapshot that [The plane on ax](plane.md) describes. The controller
|
||||||
|
suspends and resumes it.
|
||||||
|
- Names are limited to 63 bytes (`MaxNameLength` in `ax/pkg/apis/v1alpha1/types.go`). That is why the
|
||||||
|
generated Task names are short.
|
||||||
|
|
||||||
|
**What k3s changes** (planned; nothing has been applied to a k3s cluster):
|
||||||
|
|
||||||
|
- k3s runs containerd with no Docker socket. So the host-Docker path of `ousast plane alerts-engine`
|
||||||
|
does not exist there. The engine must become a task (below).
|
||||||
|
- Substrate's gVisor workers need `runsc` and a `RuntimeClass` on the nodes they run on.
|
||||||
|
- k3s ships Traefik and local-path storage. Traefik is an ingress the CLI can use instead of a kubeconfig.
|
||||||
|
- GitHub Actions builds the images from tagged commits. They are published to GitHub Container Registry
|
||||||
|
and digest-pinned in the manifests.
|
||||||
|
- A private package needs an `imagePullSecret` on the worker ServiceAccount.
|
||||||
|
- Substrate's and ax's own images are built with `ko` from their checkouts. They are pushed to the same
|
||||||
|
registry.
|
||||||
|
- The kind profile keeps `localhost:5001` and `ops/ax/up.sh`.
|
||||||
|
|
||||||
### One image, and what concurrency equals
|
### One image, and what concurrency equals
|
||||||
|
|
||||||
There is **one container image** for every Task: `plane/Dockerfile.runner` (`python:3.12-slim`
|
There is **one container image** for every Task: `plane/Dockerfile.runner`. It is `python:3.12-slim`
|
||||||
plus this package, PID 1 `/usr/local/bin/ax-task-runner`, which is
|
plus this package. PID 1 is `/usr/local/bin/ax-task-runner`, which is `openultrasast.plane.runner`.
|
||||||
`openultrasast.plane.runner`). Each Task manifest under `plane/tasks/` names the same image by
|
|
||||||
digest (the pin is kept in `~/.cache/ousast/ax-src/runner-image`) and selects its work with
|
Each Task manifest under `plane/tasks/` names the same image by digest. The pin is kept in
|
||||||
`spec.command`: the runner starts `python -m openultrasast.plane.tasks.<command[0]>`
|
`~/.cache/ousast/ax-src/runner-image`. The manifest picks its work with `spec.command`. The runner
|
||||||
(`runner.py`, `task_module`). `repo-facts`, `verify`, `agree`, `features`, `remember`, `alerts`,
|
starts `python -m openultrasast.plane.tasks.<command[0]>` (`runner.py`, `task_module`).
|
||||||
`loop` and `roles` are modules in that image, not images of their own; the engine would be the
|
|
||||||
second image (planned). So the production answer to "how many containers" is: one image, one
|
`repo-facts`, `verify`, `agree`, `features`, `remember`, `alerts`, `loop` and `roles` are modules in
|
||||||
actor per Task, and **an actor occupies a whole worker** (its limits are the ActorTemplate's, the
|
that image. They are not images of their own. The engine would be the second image (planned).
|
||||||
worker holds one actor at a time). Concurrency is therefore the number of workers in the
|
|
||||||
`WorkerPool`, nothing else: `ops/ax/workerpool.yaml.tmpl` declares `replicas: 2` of 1 CPU /
|
So the production answer to "how many containers" is:
|
||||||
1.5 GiB, so at most two Tasks run at once on this host, and `ousast plane run --workers` must not
|
|
||||||
exceed the pool. Substrate ships an autoscaled pool
|
- one image;
|
||||||
(`substrate/demos/autoscaled-workerpool/README.md`: an HPA on the external metric
|
- one actor per Task;
|
||||||
`ate_workerpool_workers{state=at_capacity}` through prometheus-adapter, writing
|
- **an actor occupies a whole worker**. Its limits are the ActorTemplate's, and the worker holds one
|
||||||
`WorkerPool.spec.replicas`; its README says the demo is kind-only today) and request parking
|
actor at a time.
|
||||||
(`substrate/demos/parking/`: the router holds a resume until a worker frees up). Neither is
|
|
||||||
applied by `ops/ax/up.sh`; the production profile's pool manifests and autoscaling rule are
|
Concurrency is therefore the number of workers in the `WorkerPool`, nothing else.
|
||||||
|
`ops/ax/workerpool.yaml.tmpl` declares `replicas: 2` of 1 CPU / 1.5 GiB. So at most two Tasks run at
|
||||||
|
once on this host. `ousast plane run --workers` must not exceed the pool.
|
||||||
|
|
||||||
|
Substrate ships two scaling aids:
|
||||||
|
|
||||||
|
- **An autoscaled pool** (`substrate/demos/autoscaled-workerpool/README.md`). An HPA (Horizontal Pod
|
||||||
|
Autoscaler) reads the external metric `ate_workerpool_workers{state=at_capacity}` through
|
||||||
|
prometheus-adapter. It writes `WorkerPool.spec.replicas`. Its README says the demo is kind-only today.
|
||||||
|
- **Request parking** (`substrate/demos/parking/`). The router holds a resume until a worker frees up.
|
||||||
|
|
||||||
|
`ops/ax/up.sh` applies neither. The production profile's pool manifests and autoscaling rule are
|
||||||
**planned**.
|
**planned**.
|
||||||
|
|
||||||
### The three components that are bound to the host today
|
### The three components that are bound to the host today
|
||||||
@@ -373,59 +490,75 @@ applied by `ops/ax/up.sh`; the production profile's pool manifests and autoscali
|
|||||||
| Artifact receiver (`plane/egress.py`, the `Receiver` thread inside the reconciler) | reached through the Service `ousast-receiver.ax-system` whose EndpointSlice points at the kind gateway address on the host (`ops/ax/receiver-service.yaml.tmpl`) | **no receiver in remote mode.** Tasks deliver their artifacts and fetch their inputs through presigned URLs on the S3 memory store (B.5, path 3; the store's hostname enters each Task's EgressPolicy, B.7), and the CLI reads completion from the store. The kind profile keeps the receiver thread; an in-cluster receiver Deployment (B.5, path 2) is not the chosen direction |
|
| Artifact receiver (`plane/egress.py`, the `Receiver` thread inside the reconciler) | reached through the Service `ousast-receiver.ax-system` whose EndpointSlice points at the kind gateway address on the host (`ops/ax/receiver-service.yaml.tmpl`) | **no receiver in remote mode.** Tasks deliver their artifacts and fetch their inputs through presigned URLs on the S3 memory store (B.5, path 3; the store's hostname enters each Task's EgressPolicy, B.7), and the CLI reads completion from the store. The kind profile keeps the receiver thread; an in-cluster receiver Deployment (B.5, path 2) is not the chosen direction |
|
||||||
| Joern engine (`plane/engine_alerts.py`, the host's `openultrasast:dev` container, run with `--memory 3g`) | `ousast plane alerts-engine` runs it on the host, one container at a time, and marks the Run's `alerts` tasks done | **an `engine` task on a second, larger `WorkerPool`** (the engine does not fit a 1.5 GiB worker, and k3s has no Docker socket), from an engine image digest-pinned in GHCR, emitting the same `alerts.jsonl` rows and coverage; `alerts-engine` on the host stays as the development path |
|
| Joern engine (`plane/engine_alerts.py`, the host's `openultrasast:dev` container, run with `--memory 3g`) | `ousast plane alerts-engine` runs it on the host, one container at a time, and marks the Run's `alerts` tasks done | **an `engine` task on a second, larger `WorkerPool`** (the engine does not fit a 1.5 GiB worker, and k3s has no Docker socket), from an engine image digest-pinned in GHCR, emitting the same `alerts.jsonl` rows and coverage; `alerts-engine` on the host stays as the development path |
|
||||||
|
|
||||||
The S3 memory store (RustFS tested) is already external; in remote mode it is also the artifact
|
The S3 memory store (RustFS tested) is already external. In remote mode it is also the artifact path.
|
||||||
path, so it must be reachable from the CLI's host and, through the egress gateway, from the
|
So the CLI's host must reach it, and the workers must reach it through the egress gateway.
|
||||||
workers.
|
|
||||||
|
|
||||||
### Scaling the pre-push hook in production
|
### Scaling the pre-push hook in production
|
||||||
|
|
||||||
**Today** `ousast pre-push` is a local, deterministic check: it materialises base and head,
|
**Today** `ousast pre-push` is a local, deterministic check. It does four things:
|
||||||
discovers the changed regions, runs the evidence ranker and arbiter on head and compares with base,
|
|
||||||
all within `--deadline` (30 s by default) and in `--mode advisory` unless blocking is opted into
|
1. materialises base and head;
|
||||||
([Pre-push: the delta check](scanning.md#pre-push-the-delta-check-experimental),
|
2. finds the changed regions;
|
||||||
`src/openultrasast/push/`, [Engine and pre-push](ops/README.md#experimental-pre-push-integration)).
|
3. runs the evidence ranker and arbiter on head;
|
||||||
**The plane is not in that path**: no hook submits a Run, no model is called without an explicit
|
4. compares the result with base.
|
||||||
`--model-config`, and nothing leaves the developer's machine.
|
|
||||||
|
All of this runs within `--deadline` (30 s by default). It runs in `--mode advisory` unless you opt
|
||||||
|
into blocking. See [Pre-push: the delta check](scanning.md#pre-push-the-delta-check-experimental),
|
||||||
|
`src/openultrasast/push/` and [Engine and pre-push](ops/README.md#experimental-pre-push-integration).
|
||||||
|
|
||||||
|
**The plane is not in that path.**
|
||||||
|
|
||||||
|
- No hook submits a Run.
|
||||||
|
- No model is called without an explicit `--model-config`.
|
||||||
|
- Nothing leaves the developer's machine.
|
||||||
|
|
||||||
**The production shape** (planned, none of it built) splits the work by latency:
|
**The production shape** (planned, none of it built) splits the work by latency:
|
||||||
|
|
||||||
1. **Blocking, inside the hook's deadline**: what runs today, the quick rules and the delta
|
1. **Blocking, inside the hook's deadline**: what runs today, the quick rules and the delta engine on
|
||||||
engine on the changed regions. It stays local and deterministic, so the push is never held for
|
the changed regions. It stays local and deterministic. So the push never waits for a model or a
|
||||||
a model or a cluster.
|
cluster.
|
||||||
2. **Asynchronous, on the plane**: the hook or the CI job calls the CLI in remote mode with a
|
2. **Asynchronous, on the plane**: the hook or the CI job calls the CLI in remote mode with a *delta
|
||||||
*delta Run* whose candidates are the functions the push changed; the Run is the per-case chain
|
Run*.
|
||||||
of [The plane on ax](plane.md#per-case) (`facts`, `verify` a and b, `agree`, `verify` c on the
|
- The Run's candidates are the functions the push changed.
|
||||||
disputed only, `final`). The caller either waits, with its own deadline, for the CLI to print
|
- The Run is the per-case chain of [The plane on ax](plane.md#per-case): `facts`, `verify` a and b,
|
||||||
the status and attribution table, or returns at once and a later `ousast plane status <run>`
|
`agree`, `verify` c on the disputed only, `final`.
|
||||||
reports from the store; a commit status check or a review comment would be built on that. The
|
- The caller can wait, with its own deadline, for the CLI to print the status and attribution table.
|
||||||
worker pool is sized to the push rate or autoscaled on `at_capacity` workers. An unattended
|
- Or the caller returns at once, and a later `ousast plane status <run>` reports from the store.
|
||||||
in-cluster reconciler that accepts submissions without a CLI session is the later option named
|
- A commit status check or a review comment would be built on that.
|
||||||
above.
|
- The worker pool is sized to the push rate, or autoscaled on `at_capacity` workers.
|
||||||
|
- An unattended in-cluster reconciler that accepts submissions without a CLI session is the later
|
||||||
|
option named above.
|
||||||
|
|
||||||
**Cost per push, estimated from measured rates.** Two measured per-candidate rates exist for one
|
**Cost per push, estimated from measured rates.** Two measured per-candidate rates exist for one model
|
||||||
model verdict: `$0.003215` (`spend.cost_per_candidate_evaluate_usd` in
|
verdict:
|
||||||
`benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`) and `$0.0021`
|
|
||||||
(exp-002 arm A, `overall.usd_per_candidate.A` in
|
- `$0.003215` (`spend.cost_per_candidate_evaluate_usd` in
|
||||||
`benchmarks/measurements/2026-10-02-exp-002/record.json`). A push that changes five functions needs
|
`benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`);
|
||||||
two verify passes per candidate: 5 x 2 x $0.0021 = $0.021, or 5 x 2 x $0.003215 = $0.032 at the
|
- `$0.0021` (exp-002 arm A, `overall.usd_per_candidate.A` in
|
||||||
higher rate, plus a third pass on the disputed candidates only. That is where "about two cents per
|
`benchmarks/measurements/2026-10-02-exp-002/record.json`).
|
||||||
push" comes from; it is arithmetic over the decision engine's measured rates, not a measurement of
|
|
||||||
a push Run, and `ousast plane status` would print the real figure per Task.
|
A push that changes five functions needs two verify passes per candidate. That gives 5 x 2 x $0.0021 =
|
||||||
|
$0.021, or 5 x 2 x $0.003215 = $0.032 at the higher rate. A third pass runs on the disputed candidates
|
||||||
|
only. That is where "about two cents per push" comes from. It is arithmetic over the decision engine's
|
||||||
|
measured rates, not a measurement of a push Run. `ousast plane status` would print the real figure per
|
||||||
|
Task.
|
||||||
|
|
||||||
**What is not built** (each is a named gap, not a configuration):
|
**What is not built** (each is a named gap, not a configuration):
|
||||||
|
|
||||||
- The delta-Run command. `ousast plane scan <repo> --base <commit> --head <commit>` **does not
|
- The delta-Run command. `ousast plane scan <repo> --base <commit> --head <commit>` **does not
|
||||||
exist**; `ousast plane` has `run`, `status`, `doctor`, `remember`, `memory-normalise`,
|
exist**. `ousast plane` has `run`, `status`, `doctor`, `remember`, `memory-normalise`,
|
||||||
`workspaces`, `harvest` and `alerts-engine` (`src/openultrasast/cli.py`).
|
`workspaces`, `harvest` and `alerts-engine` (`src/openultrasast/cli.py`).
|
||||||
- A candidates task for an arbitrary repository. `repo-facts` takes its candidates from a
|
- A candidates task for an arbitrary repository. `repo-facts` takes its candidates from a
|
||||||
population's validation set; a step that derives them from a diff is not written.
|
population's validation set. No step derives them from a diff yet.
|
||||||
- Workspace generation from a URL and a commit outside a population. `plane/generate.py` is
|
- Workspace generation from a URL and a commit outside a population. `plane/generate.py` is
|
||||||
population-driven (`read_cases` reads a validation set and a population TOML), so a Run for a
|
population-driven: `read_cases` reads a validation set and a population TOML. So a Run for a
|
||||||
repository that is not in a population cannot be generated today.
|
repository that is not in a population cannot be generated today.
|
||||||
- The remote execution mode itself: the `local`/`remote` switch, configurable cluster context,
|
- The remote execution mode itself:
|
||||||
ax-server and router addresses (the `kind-` prefix is hard-coded, B.4), presigned delivery from
|
- the `local`/`remote` switch;
|
||||||
the runner to the store, and the CLI reading completion and `summary.json` from the store
|
- a configurable cluster context and ax-server and router addresses (the `kind-` prefix is
|
||||||
instead of its receiver thread.
|
hard-coded, B.4);
|
||||||
|
- presigned delivery from the runner to the store;
|
||||||
|
- the CLI reading completion and `summary.json` from the store instead of its receiver thread.
|
||||||
- The engine task and its WorkerPool, the GHCR build, the production pool and autoscaling
|
- The engine task and its WorkerPool, the GHCR build, the production pool and autoscaling
|
||||||
manifests, and the k3s runbook and checklist.
|
manifests, and the k3s runbook and checklist.
|
||||||
- The check or comment that carries the asynchronous verdict back to the push.
|
- The check or comment that carries the asynchronous verdict back to the push.
|
||||||
@@ -433,8 +566,7 @@ a push Run, and `ousast plane status` would print the real figure per Task.
|
|||||||
### Sizing
|
### Sizing
|
||||||
|
|
||||||
The worker is the unit. Every Task's `limits` fit the 1 CPU / 1.5 GiB worker of
|
The worker is the unit. Every Task's `limits` fit the 1 CPU / 1.5 GiB worker of
|
||||||
`ops/ax/workerpool.yaml.tmpl`; the requests are what the Task declares, the worker is what it
|
`ops/ax/workerpool.yaml.tmpl`. The requests are what the Task declares. The worker is what it occupies.
|
||||||
occupies.
|
|
||||||
|
|
||||||
| Task (`plane/tasks/*.yaml`) | Requests | Limits | Per push |
|
| Task (`plane/tasks/*.yaml`) | Requests | Limits | Per push |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
@@ -446,17 +578,18 @@ occupies.
|
|||||||
| `roles` | 250m / 384Mi | 1 CPU / 1Gi | not in a push Run (harvest Runs only) |
|
| `roles` | 250m / 384Mi | 1 CPU / 1Gi | not in a push Run (harvest Runs only) |
|
||||||
| `engine` (planned) | from a measured run | above 3 GiB (the host run uses `--memory 3g`) | its own pool, not the default one |
|
| `engine` (planned) | from a measured run | above 3 GiB (the host run uses `--memory 3g`) | its own pool, not the default one |
|
||||||
|
|
||||||
- **Actors per push**: 4 (`facts`, `va`, `vb`, `agree`), 6 with a tie-break (`vc`, `final`);
|
- **Actors per push**: 4 (`facts`, `va`, `vb`, `agree`), or 6 with a tie-break (`vc`, `final`).
|
||||||
`features` and `remember` add two more when the verdict is to be kept.
|
`features` and `remember` add two more when the verdict is to be kept.
|
||||||
- **Critical path**: `facts` then `va` and `vb` side by side, then `agree`, then (sometimes) `vc`
|
- **Critical path**: `facts`, then `va` and `vb` side by side, then `agree`, then sometimes `vc`
|
||||||
and `final`: three to five Task boots in series, each a golden-snapshot restore plus the model
|
and `final`. That is three to five Task boots in series. Each boot is a golden-snapshot restore plus
|
||||||
calls of the verify passes.
|
the model calls of the verify passes.
|
||||||
- **One actor per worker**: `agree` asks for 256Mi and still holds a 1.5 GiB worker while it runs,
|
- **One actor per worker**: `agree` asks for 256Mi but still holds a 1.5 GiB worker while it runs. The
|
||||||
because the worker's capacity, not the Task's request, is the unit. A pool of `W` workers runs
|
worker's capacity, not the Task's request, is the unit.
|
||||||
at most `W/2` pushes through their verify phase at once; on this host (`W = 2`) that is one
|
- A pool of `W` workers runs at most `W/2` pushes through their verify phase at once.
|
||||||
push at a time, and every further push waits (the reconciler's `--workers` bound, or Substrate's
|
- On this host (`W = 2`) that is one push at a time.
|
||||||
request parking if the pool is saturated). Autoscaling the pool on `at_capacity` workers is the
|
- Every further push waits, on the reconciler's `--workers` bound or on Substrate's request parking
|
||||||
upstream answer and is **planned** here.
|
if the pool is saturated.
|
||||||
|
- Autoscaling the pool on `at_capacity` workers is the upstream answer. It is **planned** here.
|
||||||
|
|
||||||
## C. The manifests under `plane/`
|
## C. The manifests under `plane/`
|
||||||
|
|
||||||
@@ -472,6 +605,12 @@ occupies.
|
|||||||
| `ops/ax/smoke-task.yaml.tmpl`, `smoke-run.yaml.tmpl` | yes | rendered with `RUNNER_IMAGE` from your pin file |
|
| `ops/ax/smoke-task.yaml.tmpl`, `smoke-run.yaml.tmpl` | yes | rendered with `RUNNER_IMAGE` from your pin file |
|
||||||
| `ops/ax/up.sh`, `down.sh` | no | kind-specific; they do not apply to another cluster |
|
| `ops/ax/up.sh`, `down.sh` | no | kind-specific; they do not apply to another cluster |
|
||||||
|
|
||||||
The Run and Task contract itself (`plane/runner.py` as PID 1 of the runner image, `AX_TASK_YAML`
|
The Run and Task contract does not depend on where the cluster is. That contract covers:
|
||||||
and `AX_WORKSPACES_YAML`, `/healthz` and `/readyz`, the start request, the tar delivery as
|
|
||||||
completion) does not depend on where the cluster is: [The plane on ax](plane.md).
|
- `plane/runner.py` as PID 1 of the runner image;
|
||||||
|
- `AX_TASK_YAML` and `AX_WORKSPACES_YAML`;
|
||||||
|
- `/healthz` and `/readyz`;
|
||||||
|
- the start request;
|
||||||
|
- the tar delivery as completion.
|
||||||
|
|
||||||
|
See [The plane on ax](plane.md).
|
||||||
|
|||||||
+139
-85
@@ -1,11 +1,12 @@
|
|||||||
# Memory and detection
|
# Memory and detection
|
||||||
|
|
||||||
How what the system remembers is meant to turn into better decisions on code it has not seen.
|
This page shows how the system's memory is meant to improve decisions on code it has not seen. It
|
||||||
This is the design of the learned decision engine (`src/openultrasast/learn/`), which is **in
|
describes the design of the learned decision engine (`src/openultrasast/learn/`).
|
||||||
development and not adopted**: no scan, pre-push check or report uses it today. Every user-facing
|
|
||||||
decision is still the deterministic one described in [Scanning](scanning.md) and
|
The engine is **in development and not adopted**. No scan, pre-push check or report uses it today.
|
||||||
|
Every user-facing decision is still the deterministic one described in [Scanning](scanning.md) and
|
||||||
[Architecture](architecture.md). The engine's commands and status are on
|
[Architecture](architecture.md). The engine's commands and status are on
|
||||||
[Decision engine](decision-engine.md); the store itself is on [Memory](memory.md).
|
[Decision engine](decision-engine.md). The store itself is on [Memory](memory.md).
|
||||||
|
|
||||||
## From instruments to a decision
|
## From instruments to a decision
|
||||||
|
|
||||||
@@ -25,89 +26,128 @@ flowchart TD
|
|||||||
op -- "otherwise" --> adv["ADVISORY"]
|
op -- "otherwise" --> adv["ADVISORY"]
|
||||||
```
|
```
|
||||||
|
|
||||||
The intended place for these decisions is changed code: `learn/features.py` builds feature records
|
These decisions are meant for changed code. `learn/features.py` builds feature records in two ways:
|
||||||
for a whole-repository scan or for a base..head delta (`build_for_scan` with `Delta`), the same
|
|
||||||
delta `ousast pre-push` checks. Integration into scan and pre-push is still ahead.
|
- for a whole-repository scan;
|
||||||
|
- for a base..head delta (`build_for_scan` with `Delta`). This is the same delta `ousast pre-push` checks.
|
||||||
|
|
||||||
|
Integration into scan and pre-push is still ahead.
|
||||||
|
|
||||||
## 1. Instruments produce signals, not verdicts
|
## 1. Instruments produce signals, not verdicts
|
||||||
|
|
||||||
Each instrument contributes one block to a candidate's feature record (`learn/schema.py`,
|
Each instrument contributes one block to a candidate's feature record (`learn/schema.py`,
|
||||||
`INSTRUMENTS`): `language`, `quick` (quick rules), `engine` (the Joern model layer), `facts` (repo
|
`INSTRUMENTS`). The blocks are:
|
||||||
facts, such as callers and function span), `source`, `entry_points`, `roles` (model source, sink
|
|
||||||
and sanitizer roles), `delta`, `verify` and `agree` (the plane's verify passes and their
|
|
||||||
agreement), and `model_sinks` (the archived stage-1 classifier, `benchmarks/archive/model_sinks.py`, whose
|
|
||||||
logic now lives in `plane/tasks/roles.py`). A candidate is `(path, function, family)` of one repository and pin
|
|
||||||
where at least one instrument emitted a signal of that family (`learn/features.py`).
|
|
||||||
|
|
||||||
An instrument that did not run, had no coverage or failed records that state (`none`, `failed`,
|
- `language`;
|
||||||
`not_applicable`), never a zero, so "silent" and "did not look" stay distinguishable. None of the
|
- `quick` (quick rules);
|
||||||
instruments decides alone.
|
- `engine` (the Joern model layer);
|
||||||
|
- `facts` (repo facts, such as callers and function span);
|
||||||
|
- `source`;
|
||||||
|
- `entry_points`;
|
||||||
|
- `roles` (model source, sink and sanitizer roles);
|
||||||
|
- `delta`;
|
||||||
|
- `verify` and `agree` (the plane's verify passes and their agreement);
|
||||||
|
- `model_sinks` (the archived stage-1 classifier, `benchmarks/archive/model_sinks.py`). Its logic now
|
||||||
|
lives in `plane/tasks/roles.py`.
|
||||||
|
|
||||||
**Why profile v1 withholds the engine and function length.** A leak audit
|
A candidate is `(path, function, family)` of one repository and pin. At least one instrument must have
|
||||||
(`ousast learn audit-leaks`, `benchmarks/measurements/2026-10-01-decision-engine-leak-audit/`)
|
emitted a signal of that family there (`learn/features.py`).
|
||||||
found that both separate the vulnerable and fixed sides of a pair by construction: a fix removes
|
|
||||||
a sink, and a guard lengthens the function. That holds on the pair corpus, not on unseen code, so
|
An instrument may not run, may have no coverage, or may fail. It then records that state (`none`,
|
||||||
`INPUT_PROFILES["v1"]` drops the `engine` block and `facts.function_lines` from prompts and from
|
`failed`, `not_applicable`), never a zero. So "silent" and "did not look" stay distinguishable. No
|
||||||
the retrieval distance.
|
instrument decides alone.
|
||||||
|
|
||||||
|
**Why profile v1 withholds the engine and function length.** A leak audit found that both separate the
|
||||||
|
vulnerable and fixed sides of a pair by construction (`ousast learn audit-leaks`,
|
||||||
|
`benchmarks/measurements/2026-10-01-decision-engine-leak-audit/`):
|
||||||
|
|
||||||
|
- a fix removes a sink;
|
||||||
|
- a guard lengthens the function.
|
||||||
|
|
||||||
|
That holds on the pair corpus, not on unseen code. So `INPUT_PROFILES["v1"]` drops the `engine` block
|
||||||
|
and `facts.function_lines` from prompts and from the retrieval distance.
|
||||||
|
|
||||||
## 2. Labels come from ground truth only
|
## 2. Labels come from ground truth only
|
||||||
|
|
||||||
`ousast learn labels` (`learn/labels.py`) reads only the sources on a fail-closed allow-list
|
`ousast learn labels` (`learn/labels.py`) reads only the sources on a fail-closed allow-list
|
||||||
(`learn/sources.toml`): `population-v1`, `population-v2`, `dev-php` (reviewed recipe sites),
|
(`learn/sources.toml`). The sources are `population-v1`, `population-v2`, `dev-php` (reviewed recipe
|
||||||
`pairs`, `advisory-fixes`, `adjudications` and `assumed_benign`. A file the list does not name is
|
sites), `pairs`, `advisory-fixes`, `adjudications` and `assumed_benign`. A file the list does not name
|
||||||
refused before it is opened; the reserved population v3 is excluded unread.
|
is refused before it is opened. The reserved population v3 is excluded unread.
|
||||||
|
|
||||||
- **Positives** are declared sites and the vulnerable side of a fix; **negatives** are the same
|
- **Positives** are declared sites and the vulnerable side of a fix.
|
||||||
function on the fixed side where the fix changed it, and recorded `False`/`FP` adjudications.
|
- **Negatives** are two kinds:
|
||||||
A vulnerable-pin candidate matching no site is unlabelled, never negative.
|
- the same function on the fixed side, where the fix changed it;
|
||||||
- **Assumed benign** (maintainer decision 2026-09-30): ordinary non-security commits of the training
|
- recorded `False`/`FP` adjudications.
|
||||||
repositories, at least 90 days from any fix, with no later security commit touching the same
|
- A vulnerable-pin candidate matching no site is unlabelled, never negative.
|
||||||
files. They are their own source with weight 0.5, conditional on a signal at the tip and none at
|
- **Assumed benign** (maintainer decision 2026-09-30) covers ordinary non-security commits of the
|
||||||
the base, and reported separately, never merged with verified negatives.
|
training repositories. Each is at least 90 days from any fix. No later security commit touches the
|
||||||
- **Plane verdicts are features, not labels.** "Agreed", "rejected" or "disputed" is the output of
|
same files.
|
||||||
the verify instrument; it enters the record as the `verify` and `agree` blocks. Using it as a
|
- They are their own source with weight 0.5, conditional on a signal at the tip and none at the base.
|
||||||
label would teach the engine to reproduce the plane, mistakes included, so "rejected by the
|
- They are reported separately, never merged with verified negatives.
|
||||||
plane" is never a negative (`learn/labels.py`: "A plane verdict ... is never a label").
|
- **Plane verdicts are features, not labels.** "Agreed", "rejected" or "disputed" is the output of the
|
||||||
|
verify instrument. It enters the record as the `verify` and `agree` blocks. Using it as a label would
|
||||||
|
teach the engine to reproduce the plane, mistakes included. So "rejected by the plane" is never a
|
||||||
|
negative (`learn/labels.py`: "A plane verdict ... is never a label").
|
||||||
|
|
||||||
`ousast learn memory build` joins each label with the candidate's feature record and an excerpt of
|
`ousast learn memory build` joins each label with the candidate's feature record and an excerpt of
|
||||||
its function and stores one `example` row (no rationale) plus the excerpt blob.
|
its function. It stores one `example` row (no rationale) plus the excerpt blob.
|
||||||
|
|
||||||
## 3. Retrieval of similar labelled cases
|
## 3. Retrieval of similar labelled cases
|
||||||
|
|
||||||
`learn/retrieve.py`, no model:
|
`learn/retrieve.py` uses no model. It works in three steps:
|
||||||
|
|
||||||
1. **Signal profile.** A Gower distance over the allow-listed features of the active profile,
|
1. **Signal profile.** It computes a Gower distance (a distance for mixed-type features) over the
|
||||||
per instrument block, with missing values explicit (a different state counts as distance 1).
|
allow-listed features of the active profile, per instrument block. Missing values are explicit: a
|
||||||
The nearest 40 examples of the candidate's family and the nearest 40 of other families go on.
|
different state counts as distance 1. The nearest 40 examples of the candidate's family go on. So do
|
||||||
2. **Code embeddings.** A cosine re-rank of the excerpts' embeddings (OpenAI
|
the nearest 40 of other families.
|
||||||
`text-embedding-3-small` through OpenRouter, 1,536 dimensions, cached by excerpt hash):
|
2. **Code embeddings.** It re-ranks by cosine over the excerpts' embeddings. The embeddings come from
|
||||||
`r = 0.5 (1 - D) + 0.5 cos`. Without the candidate's vector the retrieval is signals only.
|
OpenAI `text-embedding-3-small` through OpenRouter (1,536 dimensions, cached by excerpt hash). The
|
||||||
3. **Balance.** Six examples: at most half per label, at most 2 per repository group, at most 2
|
formula is `r = 0.5 (1 - D) + 0.5 cos`. Without the candidate's vector, the retrieval uses signals only.
|
||||||
contrast examples from other families.
|
3. **Balance.** It picks six examples:
|
||||||
|
- at most half per label;
|
||||||
|
- at most 2 per repository group;
|
||||||
|
- at most 2 contrast examples from other families.
|
||||||
|
|
||||||
**The evaluation boundary.** `eligible()` is the only way an example enters a prompt, for retrieved
|
**The evaluation boundary.** `eligible()` is the only way an example enters a prompt. This holds for
|
||||||
examples and compiled demonstrations alike: never the candidate's own repository group, never a
|
retrieved examples and compiled demonstrations alike. An eligible example is never:
|
||||||
group the fold evaluates, never the held-out source or framework, and (outside deployment) never
|
|
||||||
a near-duplicate of the candidate (cosine >= 0.98). The program repeats the check on the ids it
|
- from the candidate's own repository group;
|
||||||
actually rendered before any call.
|
- from a group the fold evaluates;
|
||||||
|
- from the held-out source or framework;
|
||||||
|
- a near-duplicate of the candidate (cosine >= 0.98), outside deployment.
|
||||||
|
|
||||||
|
The program repeats the check on the ids it actually rendered, before any call.
|
||||||
|
|
||||||
## 4. The AI classifier, compiled
|
## 4. The AI classifier, compiled
|
||||||
|
|
||||||
`learn/program.py` and `learn/compile.py`, DSPy-style and in-house. The prompt is the instruction
|
The classifier lives in `learn/program.py` and `learn/compile.py`. It is DSPy-style (DSPy is a
|
||||||
plus the compiled demonstrations (a prefix identical for every candidate, so the provider's prefix
|
framework that compiles prompts from examples) and built in-house.
|
||||||
cache hits), then the retrieved examples, then the candidate's excerpt, signals, roles and family.
|
|
||||||
The model (`deepseek-flash`) answers `vulnerable`, `not_vulnerable` or `unsure` with a confidence
|
The prompt has this order:
|
||||||
and a rationale that must cite a line of the code. It is sampled `k` times; the score `s` is the
|
|
||||||
mean over samples, and a majority `unsure` can never BLOCK. Compilation bootstraps demonstrations
|
1. the instruction plus the compiled demonstrations. This prefix is identical for every candidate, so
|
||||||
and searches instructions on a compile split of repository groups, scored by a pre-registered
|
the provider's prefix cache hits;
|
||||||
balanced Brier score (`learn/compile.toml`); the program is stored as `programs/<id>.json`.
|
2. the retrieved examples;
|
||||||
|
3. the candidate's excerpt, signals, roles and family.
|
||||||
|
|
||||||
|
The model (`deepseek-flash`) answers `vulnerable`, `not_vulnerable` or `unsure`. It adds a confidence
|
||||||
|
and a rationale. The rationale must cite a line of the code. The model is sampled `k` times. The score
|
||||||
|
`s` is the mean over samples. A majority `unsure` can never BLOCK.
|
||||||
|
|
||||||
|
Compilation bootstraps demonstrations and searches instructions on a compile split of repository groups.
|
||||||
|
A pre-registered balanced Brier score scores it (`learn/compile.toml`). The program is stored as
|
||||||
|
`programs/<id>.json`.
|
||||||
|
|
||||||
## 5. Calibration and the two operating points
|
## 5. Calibration and the two operating points
|
||||||
|
|
||||||
`learn/calibrate.py`: Platt scaling of `s`, cross-fitted so that no map is checked on the fold it
|
`learn/calibrate.py` applies Platt scaling (a logistic map from score to probability) to `s`. It is
|
||||||
was fitted on. Calibration **holds** when the slope interval contains 1, the intercept interval
|
cross-fitted, so no map is checked on the fold it was fitted on.
|
||||||
contains 0 and ECE <= 0.05. **BLOCK** is offered only when calibration holds and a fold reaches the
|
|
||||||
required precision; otherwise the engine offers **ADVISORY** only.
|
Calibration **holds** when the slope interval contains 1, the intercept interval contains 0 and ECE
|
||||||
|
(expected calibration error) <= 0.05.
|
||||||
|
|
||||||
|
**BLOCK** is offered only when calibration holds and a fold reaches the required precision. Otherwise
|
||||||
|
the engine offers **ADVISORY** only.
|
||||||
|
|
||||||
## The loop: memory improves decisions without retraining
|
## The loop: memory improves decisions without retraining
|
||||||
|
|
||||||
@@ -125,25 +165,36 @@ flowchart LR
|
|||||||
gate -- "accepted round" --> ledger["Rule status in the ledger"]
|
gate -- "accepted round" --> ledger["Rule status in the ledger"]
|
||||||
```
|
```
|
||||||
|
|
||||||
- **Decisions.** The compiled part of the program is its instruction and demonstrations; the
|
- **Decisions.**
|
||||||
examples are retrieved from memory for each candidate. A new `example` row is therefore eligible
|
- The compiled part of the program is its instruction and demonstrations.
|
||||||
for retrieval in the next decision without recompiling the program, subject to the same
|
- The examples are retrieved from memory for each candidate.
|
||||||
evaluation boundary. `ousast learn curve` measures the metric over memory size on a fixed
|
- So a new `example` row is eligible for retrieval in the next decision without recompiling. The same
|
||||||
candidate subset.
|
evaluation boundary applies.
|
||||||
- **Rules.** `ousast improve --memory` reads the run rows (alerts, verdicts, coverage) and proposes
|
- `ousast learn curve` measures the metric over memory size on a fixed candidate subset.
|
||||||
rule-status edits only (shadow a rule that is repeatedly false across repositories; re-enable a
|
- **Rules.** `ousast improve --memory` reads the run rows (alerts, verdicts, coverage). It proposes
|
||||||
shadow rule that keeps hitting agreed declared sites). Rows from holdout pairs, the gated
|
rule-status edits only:
|
||||||
manifest's own cases and any `--qualify-population` are dropped first, and every proposal goes
|
- shadow a rule that is repeatedly false across repositories;
|
||||||
through the same validator and gate as any other edit (`improve/memory.py`).
|
- re-enable a shadow rule that keeps hitting agreed declared sites.
|
||||||
|
|
||||||
|
Some rows are dropped first: rows from holdout pairs, the gated manifest's own cases and any
|
||||||
|
`--qualify-population`. Every proposal goes through the same validator and gate as any other edit
|
||||||
|
(`improve/memory.py`).
|
||||||
|
|
||||||
## What the first slice measured
|
## What the first slice measured
|
||||||
|
|
||||||
Record: `benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json` (counts
|
Record: `benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json` (counts
|
||||||
only). Family **injection**; memory of 724 static examples from all families; input profile v1;
|
only). The setup:
|
||||||
framework priors off; out-of-repository evaluation (compile split never evaluated, outer grouped
|
|
||||||
5-fold, calibration cross-fitted, operating points nested); 95% repository-cluster bootstrap
|
- family **injection**;
|
||||||
intervals. The slice's spend ($0.003215 per candidate at evaluation, k = 5, and its replay at
|
- memory of 724 static examples from all families;
|
||||||
$0) is on [Token ergonomics](token-ergonomics.md#measured-costs).
|
- input profile v1;
|
||||||
|
- framework priors off;
|
||||||
|
- out-of-repository evaluation: compile split never evaluated, outer grouped 5-fold, calibration
|
||||||
|
cross-fitted, operating points nested;
|
||||||
|
- 95% repository-cluster bootstrap intervals.
|
||||||
|
|
||||||
|
The slice's spend is on [Token ergonomics](token-ergonomics.md#measured-costs). It was $0.003215 per
|
||||||
|
candidate at evaluation, k = 5, and its replay cost $0.
|
||||||
|
|
||||||
| Measure | Value | What it means |
|
| Measure | Value | What it means |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
@@ -152,9 +203,12 @@ $0) is on [Token ergonomics](token-ergonomics.md#measured-costs).
|
|||||||
| ADVISORY | recall 0.90 (0.82 - 0.97), precision 0.54 (0.44 - 0.65) | precision is at the positive share (0.53), so at this threshold ADVISORY is **not discriminating**: it flags nearly as indiscriminately as flagging everything would |
|
| ADVISORY | recall 0.90 (0.82 - 0.97), precision 0.54 (0.44 - 0.65) | precision is at the positive share (0.53), so at this threshold ADVISORY is **not discriminating**: it flags nearly as indiscriminately as flagging everything would |
|
||||||
| BLOCK | not offered | calibration does not hold (ECE 0.064 > 0.05), and every outer fold reported `precision_unreachable` |
|
| BLOCK | not offered | calibration does not hold (ECE 0.064 > 0.05), and every outer fold reported `precision_unreachable` |
|
||||||
|
|
||||||
**The engine is not adopted.** A second slice (2026-10-02) evaluated the six evaluable families
|
**The engine is not adopted.** A second slice (2026-10-02) evaluated the six evaluable families the
|
||||||
the same way: pooled AUC 0.77 to 0.96 per family, within-pair AUC 0.72 to 0.93, BLOCK offered for
|
same way ([Decision engine](decision-engine.md#second-slice-six-families-2026-10-02)). The results: pooled AUC 0.77 to 0.96 per family, within-pair AUC 0.72 to 0.93, BLOCK offered for
|
||||||
none, re-run agreement below 0.9 for three families
|
none. Re-run agreement was below 0.9 for three families.
|
||||||
([Decision engine](decision-engine.md#second-slice-six-families-2026-10-02)). Leave-one-source-out
|
|
||||||
and leave-one-framework-out, integration into scan and pre-push, and the one-time check on
|
Still ahead:
|
||||||
population v3 are still ahead.
|
|
||||||
|
- leave-one-source-out and leave-one-framework-out;
|
||||||
|
- integration into scan and pre-push;
|
||||||
|
- the one-time check on population v3.
|
||||||
|
|||||||
+50
-40
@@ -1,8 +1,8 @@
|
|||||||
# Memory
|
# Memory
|
||||||
|
|
||||||
One store, keyed by repository and pin, holds what plane runs and the decision engine learned.
|
One store holds what plane runs and the decision engine learned. It is keyed by repository and pin
|
||||||
The code is `src/openultrasast/plane/memory.py`; the same layout serves both backends.
|
(a fixed commit). The code is `src/openultrasast/plane/memory.py`. The same layout serves both backends.
|
||||||
How the stored rows feed detection is on [Memory and detection](memory-and-detection.md).
|
[Memory and detection](memory-and-detection.md) shows how the stored rows feed detection.
|
||||||
|
|
||||||
## What is stored
|
## What is stored
|
||||||
|
|
||||||
@@ -17,7 +17,7 @@ programs/<program id>.json compiled decision-engine programs
|
|||||||
```
|
```
|
||||||
|
|
||||||
**Rows.** Every row carries `id`, `kind`, `repo`, `pin`, `run`, `task`, `population`, `split`
|
**Rows.** Every row carries `id`, `kind`, `repo`, `pin`, `run`, `task`, `population`, `split`
|
||||||
and `image` (`ROW_FIELDS`); `id` is the sha256 of (kind, run, task, subject), so a repeated row
|
and `image` (`ROW_FIELDS`). `id` is the sha256 of (kind, run, task, subject). So a repeated row
|
||||||
replaces itself. The kinds (`KINDS`) and who writes them:
|
replaces itself. The kinds (`KINDS`) and who writes them:
|
||||||
|
|
||||||
| Kind | Written by | Holds |
|
| Kind | Written by | Holds |
|
||||||
@@ -35,12 +35,15 @@ replaces itself. The kinds (`KINDS`) and who writes them:
|
|||||||
| `experiment_result` | `ousast learn experiment analyse --record` | the paired estimates, the looks and the adoption verdict (counts and intervals only) |
|
| `experiment_result` | `ousast learn experiment analyse --record` | the paired estimates, the looks and the adoption verdict (counts and intervals only) |
|
||||||
| `label`, `decision` | reserved for later decision-engine tasks | declared in `KINDS`; not written by the code on main yet |
|
| `label`, `decision` | reserved for later decision-engine tasks | declared in `KINDS`; not written by the code on main yet |
|
||||||
|
|
||||||
**Blobs** (`put_blob`) are content-addressed: a 64-hex name under a fixed prefix. Excerpts are
|
**Blobs** (`put_blob`) are content-addressed: each has a 64-hex name under a fixed prefix.
|
||||||
written by `learn memory build` (`learn/examples.py`), embeddings by `learn memory embed`
|
Content-addressed means the name is the hash of the content. Who writes each blob:
|
||||||
(`learn/embeddings.py`, keyed by excerpt sha so an excerpt is embedded once), responses by the
|
|
||||||
program's caller (`learn/program.py`, keyed by the sha256 of model, parameters digest, messages,
|
- Excerpts: `learn memory build` (`learn/examples.py`).
|
||||||
temperature, sample and format, so a rerun replays at no cost), and compiled programs by
|
- Embeddings: `learn memory embed` (`learn/embeddings.py`). They are keyed by excerpt sha, so an
|
||||||
`learn compile` (`learn/compile.py`).
|
excerpt is embedded once.
|
||||||
|
- Responses: the program's caller (`learn/program.py`). The key is the sha256 of model, parameters
|
||||||
|
digest, messages, temperature, sample and format. So a rerun replays at no cost.
|
||||||
|
- Compiled programs: `learn compile` (`learn/compile.py`).
|
||||||
|
|
||||||
## Write path
|
## Write path
|
||||||
|
|
||||||
@@ -60,13 +63,14 @@ flowchart LR
|
|||||||
put --> store[("Memory store")]
|
put --> store[("Memory store")]
|
||||||
```
|
```
|
||||||
|
|
||||||
- `ingest` validates every row first: a malformed file fails before anything is written.
|
- `ingest` validates every row first. A malformed file fails before anything is written.
|
||||||
- Rows are merged into their repository + pin object by `id`. Every object is written whole and the
|
- Rows are merged by `id` into their repository + pin object.
|
||||||
index last, so an interrupted ingest leaves nothing a repeat would not repair.
|
- Every object is written whole, and the index last. So a repeat repairs anything an interrupted
|
||||||
|
ingest left behind.
|
||||||
- Each row object carries tags (`TAG_FIELDS`: `repo`, `pin`, `kind`, `family`, `run`,
|
- Each row object carries tags (`TAG_FIELDS`: `repo`, `pin`, `kind`, `family`, `run`,
|
||||||
`population`, `split`) listing the distinct values of its rows (`*` when they exceed 256
|
`population`, `split`). The tags list the distinct values of its rows. A tag is `*` when the
|
||||||
characters).
|
values exceed 256 characters.
|
||||||
- The plane's store failures never change a run's result: an ingest can be repeated by hand
|
- The plane's store failures never change a run's result. An ingest can be repeated by hand
|
||||||
(`cli.py`, `_plane_memory`).
|
(`cli.py`, `_plane_memory`).
|
||||||
|
|
||||||
## Read path
|
## Read path
|
||||||
@@ -85,14 +89,14 @@ flowchart LR
|
|||||||
rules --> gate{"Unchanged validator and gate"}
|
rules --> gate{"Unchanged validator and gate"}
|
||||||
```
|
```
|
||||||
|
|
||||||
- **Seed** (`memory.seed`): a Run task whose `openultrasast.io/memory-key` matches stored facts gets
|
- **Seed** (`memory.seed`).
|
||||||
them and is marked done; a changed pin, candidate set or runner image finds nothing and
|
- A Run task gets stored facts when its `openultrasast.io/memory-key` matches them. It is then marked done.
|
||||||
recomputes. The loop's `memory-snapshot` is seeded the same way, because a sandboxed task cannot
|
- A changed pin, candidate set or runner image finds nothing and recomputes.
|
||||||
read the store.
|
- The loop's `memory-snapshot` is seeded the same way. A sandboxed task cannot read the store.
|
||||||
- **Retrieval** (`learn/retrieve.py`): see
|
- **Retrieval** (`learn/retrieve.py`): see
|
||||||
[Memory and detection](memory-and-detection.md#3-retrieval-of-similar-labelled-cases).
|
[Memory and detection](memory-and-detection.md#3-retrieval-of-similar-labelled-cases).
|
||||||
- **`improve --memory`** (`improve/memory.py`): rows from holdout pairs, from the gated manifest's
|
- **`improve --memory`** (`improve/memory.py`). Some rows are dropped before any rule sees them:
|
||||||
own cases and from any `--qualify-population` are dropped before any rule sees them. See
|
rows from holdout pairs, from the gated manifest's own cases, and from any `--qualify-population`. See
|
||||||
[Architecture](architecture.md#proposals-from-plane-memory).
|
[Architecture](architecture.md#proposals-from-plane-memory).
|
||||||
|
|
||||||
## Backends: FileStore and the S3 store
|
## Backends: FileStore and the S3 store
|
||||||
@@ -109,26 +113,32 @@ flowchart LR
|
|||||||
|
|
||||||
### Verify at startup, no fallback
|
### Verify at startup, no fallback
|
||||||
|
|
||||||
The S3 store never configures its bucket. Each time an `S3Store` is opened for a bucket and
|
The S3 store never configures its bucket. It only checks it. An `S3Store` is opened for a bucket and
|
||||||
prefix (once per process), `verify_bucket()` checks, using reads plus one small probe object at
|
prefix once per process. Each time, `verify_bucket()` runs these checks. It uses reads plus one small
|
||||||
`<prefix>/_probe/select.jsonl`:
|
probe object at `<prefix>/_probe/select.jsonl`.
|
||||||
|
|
||||||
1. versioning is `Enabled` (provenance cites object versions);
|
1. Versioning is `Enabled`. Provenance cites object versions.
|
||||||
2. an enabled lifecycle rule expires `<prefix>/runs/` after a number of days, as a plain prefix
|
2. An enabled lifecycle rule expires `<prefix>/runs/` after a number of days. It uses a plain prefix
|
||||||
filter, and no enabled rule expires the whole store;
|
filter. No enabled rule expires the whole store.
|
||||||
3. S3 Select answers a probe query over JSON Lines;
|
3. S3 Select (a server-side query over JSON objects) answers a probe query over JSON Lines.
|
||||||
4. the probe's object tags are readable.
|
4. The probe's object tags are readable.
|
||||||
|
|
||||||
Anything missing is named, with the admin command that fixes it, in one `MemoryStoreError`, and
|
If anything is missing, the store does not start. One `MemoryStoreError` names each problem and the
|
||||||
the store does not start. There is no local or fetch-and-filter fallback: a filtered read that
|
admin command that fixes it.
|
||||||
S3 Select cannot answer raises a `MemoryStoreError` instead of returning "no rows". The exact
|
|
||||||
messages are listed on [RustFS setup](rustfs.md#what-the-store-verifies-at-startup).
|
There is no local or fetch-and-filter fallback. A filtered read that S3 Select cannot answer raises a
|
||||||
|
`MemoryStoreError`. It does not return "no rows". [RustFS setup](rustfs.md#what-the-store-verifies-at-startup)
|
||||||
|
lists the exact messages.
|
||||||
|
|
||||||
### Queryable fields per kind
|
### Queryable fields per kind
|
||||||
|
|
||||||
RustFS's Select infers an object's JSON schema from its leading rows, so a `where` field missing
|
RustFS's Select infers an object's JSON schema from its leading rows. This causes two failures
|
||||||
there fails with `EvaluatorBindingDoesNotExist` even when a later row has it, and a column that is
|
(measured 2026-10-02; `memory.py`, `S3Store._select`):
|
||||||
null in the first 1000 rows breaks Select on the whole object (measured 2026-10-02; `memory.py`,
|
|
||||||
`S3Store._select`). The store therefore writes every row with its kind's full set of queryable
|
- A `where` field missing from the leading rows fails with `EvaluatorBindingDoesNotExist`. This
|
||||||
fields (`QUERY_FIELDS`), using `""` where a field does not apply, and refuses a `where` on an
|
happens even when a later row has the field.
|
||||||
undeclared field. Existing rows are rewritten once with `ousast plane memory-normalise`.
|
- A column that is null in the first 1000 rows breaks Select on the whole object.
|
||||||
|
|
||||||
|
So the store writes every row with its kind's full set of queryable fields (`QUERY_FIELDS`). It uses
|
||||||
|
`""` where a field does not apply. It refuses a `where` on an undeclared field. Existing rows are
|
||||||
|
rewritten once with `ousast plane memory-normalise`.
|
||||||
|
|||||||
+83
-57
@@ -1,23 +1,34 @@
|
|||||||
# The plane on ax
|
# The plane on ax
|
||||||
|
|
||||||
Model-driven work over whole repositories runs on a separate agentic plane: a **Run**
|
Model-driven work over whole repositories runs on a separate agentic plane. The plane has these parts:
|
||||||
(`plane/runs/*.yaml`, kind `openultrasast.io/v1alpha1 Run`) is a DAG of steps, and each step is
|
|
||||||
an ax **Task** (`plane/tasks/`) that binds its Workspaces (`plane/workspaces/`), at most one ax
|
- A **Run** (`plane/runs/*.yaml`, kind `openultrasast.io/v1alpha1 Run`) is a DAG (a graph of steps
|
||||||
**Model** (`plane/models/`: `deepseek-flash` for chat, `openrouter-embedding` for embeddings) and
|
with no cycles).
|
||||||
its own `budget: {usd, calls}`. ax (google/ax over Agent Substrate, in a single-node kind cluster
|
- Each step is an ax **Task** (`plane/tasks/`).
|
||||||
on the maintainer's host) is the only executor; there is no local subprocess path. Bring-up and
|
- A Task binds its Workspaces (`plane/workspaces/`).
|
||||||
the host's lessons are in [ax on this host](ops/ax/README.md); where each part runs today (the
|
- A Task binds at most one ax **Model** (`plane/models/`). `deepseek-flash` serves chat.
|
||||||
reconciler and the receiver on the host, the tasks in gVisor on Agent Substrate) and what a
|
`openrouter-embedding` serves embeddings.
|
||||||
separate Kubernetes cluster would take (planned) are on [Deployment](deployment.md); this page
|
- A Task carries its own `budget: {usd, calls}`.
|
||||||
draws the flow.
|
|
||||||
|
ax is the only executor. It is google/ax, Google's controller that runs agent tasks, over Agent Substrate (its
|
||||||
|
sandboxed actor runtime). It runs in a single-node kind cluster (Kubernetes in Docker) on the maintainer's host.
|
||||||
|
There is no local subprocess path.
|
||||||
|
|
||||||
|
Where to read more:
|
||||||
|
|
||||||
|
- Bring-up and the host's lessons are in [ax on this host](ops/ax/README.md).
|
||||||
|
- [Deployment](deployment.md) shows where each part runs today. The reconciler and the receiver run on the host.
|
||||||
|
The tasks run in gVisor (a sandboxing kernel) on Agent Substrate. It also covers what a separate Kubernetes
|
||||||
|
cluster would take (planned).
|
||||||
|
- This page draws the flow.
|
||||||
|
|
||||||
Sources: `src/openultrasast/plane/reconciler.py`, `router.py`, `egress.py`, `runner.py`,
|
Sources: `src/openultrasast/plane/reconciler.py`, `router.py`, `egress.py`, `runner.py`,
|
||||||
`generate.py`, `memory.py`, and `src/openultrasast/plane/tasks/`.
|
`generate.py`, `memory.py`, and `src/openultrasast/plane/tasks/`.
|
||||||
|
|
||||||
## One task, end to end
|
## One task, end to end
|
||||||
|
|
||||||
`ousast plane run RUN.yaml` (the reconciler) does this for every ready task; `--workers` bounds
|
`ousast plane run RUN.yaml` (the reconciler) does this for every ready task. `--workers` bounds the
|
||||||
the tasks in flight, and a rerun skips tasks already done.
|
tasks in flight. A rerun skips tasks already done.
|
||||||
|
|
||||||
```mermaid
|
```mermaid
|
||||||
sequenceDiagram
|
sequenceDiagram
|
||||||
@@ -48,44 +59,55 @@ sequenceDiagram
|
|||||||
|
|
||||||
What each step guarantees:
|
What each step guarantees:
|
||||||
|
|
||||||
- **Credentials** (`router.py`, `runner.py`). The reconciler reads the variable the Model's
|
- **Credentials** (`router.py`, `runner.py`).
|
||||||
`secretKey.key` names from the operator's environment or `.env` (which never overrides an
|
- The reconciler reads the variable that the Model's `secretKey.key` names.
|
||||||
exported variable). The key exists only in the start request; the runner keeps it in memory,
|
- It reads it from the operator's environment or `.env`. `.env` never overrides an exported variable.
|
||||||
passes it to the command's environment, redacts it from echoed stderr and never writes it.
|
- The key exists only in the start request.
|
||||||
- **Egress** (`egress.py`). One EgressPolicy per actor, deny by default, hostnames only: plain
|
- The runner keeps it in memory and passes it to the command's environment.
|
||||||
HTTP to the artifact receiver, TLS passthrough to the source hosts of the bound Workspaces and
|
- The runner redacts it from echoed stderr and never writes it.
|
||||||
to the hosts the bound Model declares (`openultrasast.io/egress-hosts`). It is written between
|
- **Egress** (`egress.py`). Egress means outbound network traffic.
|
||||||
`ax apply` and the resume, and deleted with the actor.
|
- Each actor gets one EgressPolicy. It denies by default and names hostnames only.
|
||||||
- **Inputs** (`reconciler.py`). Artifacts other tasks produced are never rendered into the
|
- It allows plain HTTP to the artifact receiver.
|
||||||
manifest; the runner fetches them from the receiver after the start, and the receiver serves a
|
- It allows TLS passthrough to the source hosts of the bound Workspaces.
|
||||||
running task its declared inputs only.
|
- It allows TLS passthrough to the hosts the bound Model declares (`openultrasast.io/egress-hosts`).
|
||||||
- **Completion** (`runner.py`). ax has no Completed phase: the runner's delivery of the output tar
|
- The reconciler writes it between `ax apply` and the resume. It is deleted with the actor.
|
||||||
is the task's completion. A completion marker stops a resumed actor from repeating billed work;
|
- **Inputs** (`reconciler.py`).
|
||||||
when the marker says the delivery failed, a later boot retries the delivery alone.
|
- Artifacts that other tasks produced are never rendered into the manifest.
|
||||||
- **Budgets** (`plane/budget.py`). The metered client refuses the next call once a task's `usd` or
|
- The runner fetches them from the receiver after the start.
|
||||||
`calls` ceiling is reached; the task ends `unfinished` and resumes on a rerun with a larger
|
- The receiver serves a running task its declared inputs only.
|
||||||
budget. Model-free tasks run with `{usd: 0, calls: 0}`, so a stray call fails loudly.
|
- **Completion** (`runner.py`).
|
||||||
- **Attribution.** `ousast plane status RUN` prints per task its status, `calls`, `prompt`,
|
- ax has no Completed phase. The runner's delivery of the output tar is the task's completion.
|
||||||
`cache_hit` and `output` tokens and `usd` from the tasks' `summary.json`, and writes
|
- A completion marker stops a resumed actor from repeating billed work.
|
||||||
`attribution.json` in the run directory (`$OUSAST_RESULTS/plane/RUN/`, default
|
- If the marker says the delivery failed, a later boot retries only the delivery.
|
||||||
`~/ousast-results/plane/`).
|
- **Budgets** (`plane/budget.py`).
|
||||||
|
- The metered client refuses the next call once a task reaches its `usd` or `calls` ceiling.
|
||||||
|
- The task then ends `unfinished`. It resumes on a rerun with a larger budget.
|
||||||
|
- Model-free tasks run with `{usd: 0, calls: 0}`, so a stray call fails loudly.
|
||||||
|
- **Attribution.**
|
||||||
|
- `ousast plane status RUN` prints these per task: status, `calls`, `prompt`, `cache_hit` and
|
||||||
|
`output` tokens, and `usd`. It reads them from the tasks' `summary.json`.
|
||||||
|
- It writes `attribution.json` in the run directory (`$OUSAST_RESULTS/plane/RUN/`, default
|
||||||
|
`~/ousast-results/plane/`).
|
||||||
|
|
||||||
!!! note "The golden snapshot"
|
!!! note "The golden snapshot"
|
||||||
Agent Substrate boots every new actor template once as a *golden* actor to snapshot it, with
|
Agent Substrate boots every new actor template once as a *golden* actor and snapshots it. That boot
|
||||||
the same image, Task and environment, and inside the sandbox nothing tells that boot from the
|
uses the same image, Task and environment. Inside the sandbox, nothing tells that boot from the real one.
|
||||||
real one. So the runner's command never starts by itself: it waits for the start request,
|
|
||||||
which the reconciler sends through `atenet-router` to the task's actor, never to the golden
|
So the runner's command never starts by itself. It waits for the start request. The reconciler sends
|
||||||
one. Workspaces are prepared only after the start, so the golden boot touches no network and a
|
that request through `atenet-router` to the task's actor, never to the golden one.
|
||||||
started actor already runs under its egress policy (a clone at boot failed on the live cluster
|
|
||||||
while the actor was being restored, 2026-09-29; `runner.py`, `_run_once`). The first resume of
|
Workspaces are prepared only after the start. So the golden boot touches no network. A started actor
|
||||||
a new image can time out while the snapshot is built, which is why the reconciler retries it
|
already runs under its egress policy. A clone at boot once failed on the live cluster while the actor
|
||||||
|
was being restored (2026-09-29; `runner.py`, `_run_once`).
|
||||||
|
|
||||||
|
The first resume of a new image can time out while the snapshot is built. So the reconciler retries it
|
||||||
for up to `OUSAST_RESUME_TIMEOUT` seconds (900).
|
for up to `OUSAST_RESUME_TIMEOUT` seconds (900).
|
||||||
|
|
||||||
## The Runs: validation and loop
|
## The Runs: validation and loop
|
||||||
|
|
||||||
`ousast plane workspaces POPULATION --validation-set ...` (`generate.py`) writes one chain per
|
`ousast plane workspaces POPULATION --validation-set ...` (`generate.py`) writes one chain per case.
|
||||||
case; `--loop` adds the `alerts` step per case and the loop's singleton steps. Every step except
|
`--loop` adds the `alerts` step per case and the loop's singleton steps. Every step is model-free
|
||||||
the verify passes (and `roles` in harvest Runs) is model-free.
|
except the verify passes (and `roles` in harvest Runs).
|
||||||
|
|
||||||
### Per case
|
### Per case
|
||||||
|
|
||||||
@@ -119,23 +141,27 @@ flowchart LR
|
|||||||
improve --> outcome["gate.json, journal.json, and rule_policy.json only if accepted"]
|
improve --> outcome["gate.json, journal.json, and rule_policy.json only if accepted"]
|
||||||
```
|
```
|
||||||
|
|
||||||
- **Before the Run** `ousast plane run` seeds from the store (`memory.seed`): a task whose
|
- **Before the Run**, `ousast plane run` seeds from the store (`memory.seed`).
|
||||||
`openultrasast.io/memory-key` (repository, pin, candidates digest, runner image digest) has
|
- Each task has an `openultrasast.io/memory-key`: repository, pin, candidates digest and runner image digest.
|
||||||
stored facts gets them and is marked done, and the sandboxed `memory-snapshot` task gets the
|
- If the store has facts for that key, the task gets them and is marked done.
|
||||||
store's rows after the guard. **After the Run** it ingests every delivered `memory.jsonl`
|
- The sandboxed `memory-snapshot` task gets the store's rows after the guard.
|
||||||
(`ousast plane remember RUN` repeats this by hand). See [Memory](memory.md).
|
- **After the Run**, it ingests every delivered `memory.jsonl`. `ousast plane remember RUN` repeats
|
||||||
- The loop's rules and its guard are described in
|
this by hand. See [Memory](memory.md).
|
||||||
[Architecture](architecture.md#proposals-from-plane-memory). Adopting an accepted ledger stays a
|
- [Architecture](architecture.md#proposals-from-plane-memory) describes the loop's rules and its guard.
|
||||||
maintainer commit (`src/openultrasast/plane/tasks/loop.py`).
|
Adopting an accepted ledger stays a maintainer commit (`src/openultrasast/plane/tasks/loop.py`).
|
||||||
- `ousast plane harvest` writes the decision engine's harvest Runs (verify a/b with agree, and
|
- `ousast plane harvest` writes the decision engine's harvest Runs (verify a/b with agree, and
|
||||||
model `roles`); `ousast plane alerts-engine` produces a Run's `alerts` from the Joern engine
|
model `roles`).
|
||||||
image on the host for PHP and other languages quick mode does not cover.
|
- `ousast plane alerts-engine` produces a Run's `alerts` from the Joern engine image on the host. It
|
||||||
|
covers PHP and other languages that quick mode does not cover.
|
||||||
|
|
||||||
## Where this runs in production
|
## Where this runs in production
|
||||||
|
|
||||||
ax is a Kubernetes application over Agent Substrate, every Task runs the one runner image as an actor that
|
ax is a Kubernetes application over Agent Substrate. Every Task runs the one runner image as an actor. Each actor
|
||||||
occupies a whole worker, and two profiles of this code base are the target: `kind` on a laptop (today) and `k3s`
|
occupies a whole worker. This code base targets two profiles:
|
||||||
in production (planned), where the same CLI submits Runs remotely and the tasks deliver to the S3 store:
|
|
||||||
|
- `kind` on a laptop (today).
|
||||||
|
- `k3s` in production (planned). There the same CLI submits Runs remotely, and the tasks deliver to the S3 store.
|
||||||
|
|
||||||
[Production topology: ax on a Kubernetes cluster](deployment.md#production-topology-ax-on-a-kubernetes-cluster)
|
[Production topology: ax on a Kubernetes cluster](deployment.md#production-topology-ax-on-a-kubernetes-cluster)
|
||||||
draws it and sizes the pre-push hook at scale.
|
draws it and sizes the pre-push hook at scale.
|
||||||
|
|
||||||
|
|||||||
+74
-50
@@ -1,16 +1,22 @@
|
|||||||
# RustFS setup for the memory store
|
# RustFS setup for the memory store
|
||||||
|
|
||||||
The plane's memory store can live in an S3-compatible bucket: `OUSAST_MEMORY=s3://sast-memory`
|
The plane's memory store can live in an S3-compatible bucket. RustFS is an S3-compatible object
|
||||||
selects `S3Store` (`src/openultrasast/plane/memory.py`), which talks to the server through boto3.
|
storage server.
|
||||||
Any S3-compatible store with bucket versioning, lifecycle rules, object tags and S3 Select works;
|
|
||||||
RustFS is the server the contract tests run against. The maintainer's server is RustFS at
|
|
||||||
`files.because-security.com`, bucket `sast-memory`, store at the bucket root. What the store keeps
|
|
||||||
there is described on [Memory](memory.md).
|
|
||||||
|
|
||||||
**The store never configures its bucket.** An admin sets the bucket up once with admin
|
- `OUSAST_MEMORY=s3://sast-memory` selects `S3Store` (`src/openultrasast/plane/memory.py`).
|
||||||
credentials. The agent account the store runs as can read the bucket's settings but not change
|
- `S3Store` talks to the server through boto3 (the AWS SDK for Python).
|
||||||
them, and reads and writes objects freely. Every time the store opens, it checks the setup and
|
- Any S3-compatible store works if it has bucket versioning, lifecycle rules, object tags and S3 Select
|
||||||
refuses to start if anything is missing.
|
(a server-side query over JSON objects).
|
||||||
|
- RustFS is the server the contract tests run against.
|
||||||
|
- The maintainer's server is RustFS at `files.because-security.com`. The bucket is `sast-memory`. The
|
||||||
|
store sits at the bucket root.
|
||||||
|
|
||||||
|
[Memory](memory.md) describes what the store keeps there.
|
||||||
|
|
||||||
|
**The store never configures its bucket.** An admin sets the bucket up once with admin credentials.
|
||||||
|
The store runs as an agent account. That account can read the bucket's settings but not change them.
|
||||||
|
It reads and writes objects freely. Every time the store opens, it checks the setup. It refuses to start
|
||||||
|
if anything is missing.
|
||||||
|
|
||||||
```mermaid
|
```mermaid
|
||||||
flowchart LR
|
flowchart LR
|
||||||
@@ -35,8 +41,8 @@ aws s3api put-bucket-versioning \
|
|||||||
--versioning-configuration Status=Enabled
|
--versioning-configuration Status=Enabled
|
||||||
```
|
```
|
||||||
|
|
||||||
**2. Expire raw run outputs under `runs/` after 30 days.** Everything else (`repos/`, `facts/`,
|
**2. Expire raw run outputs under `runs/` after 30 days.** Everything else is kept: `repos/`, `facts/`,
|
||||||
`index.jsonl` and the blobs) is kept, so the rule must be a plain prefix filter on `runs/`.
|
`index.jsonl` and the blobs. So the rule must be a plain prefix filter on `runs/`.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
aws s3api put-bucket-lifecycle-configuration \
|
aws s3api put-bucket-lifecycle-configuration \
|
||||||
@@ -46,9 +52,10 @@ aws s3api put-bucket-lifecycle-configuration \
|
|||||||
```
|
```
|
||||||
|
|
||||||
`put-bucket-lifecycle-configuration` **replaces every rule on the bucket**. If the bucket already
|
`put-bucket-lifecycle-configuration` **replaces every rule on the bucket**. If the bucket already
|
||||||
has rules, merge this one into them first. The rule id `ousast-runs-expiry` and the 30 days are the
|
has rules, merge this one into them first.
|
||||||
store's defaults (`RUNS_RULE_ID`, `RUNS_EXPIRE_DAYS`); the store accepts any enabled, untagged
|
|
||||||
prefix rule on `runs/` with a number of days.
|
The rule id `ousast-runs-expiry` and the 30 days are the store's defaults (`RUNS_RULE_ID`,
|
||||||
|
`RUNS_EXPIRE_DAYS`). The store accepts any enabled, untagged prefix rule on `runs/` with a number of days.
|
||||||
|
|
||||||
To read the result back:
|
To read the result back:
|
||||||
|
|
||||||
@@ -59,8 +66,8 @@ aws s3api get-bucket-lifecycle-configuration --endpoint-url https://files.becaus
|
|||||||
|
|
||||||
## The agent policy
|
## The agent policy
|
||||||
|
|
||||||
Attach this policy to the agent account (the account whose keys go into `.env`). It is the
|
Attach this policy to the agent account. That is the account whose keys go into `.env`. It is the
|
||||||
complete policy: nothing else is needed, and it grants no `Put*` on versioning or lifecycle.
|
complete policy, and nothing else is needed. It grants no `Put*` on versioning or lifecycle.
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
@@ -102,14 +109,14 @@ complete policy: nothing else is needed, and it grants no `Put*` on versioning o
|
|||||||
| `BucketSettingsReadOnly` | `arn:aws:s3:::sast-memory` | The agent can list the bucket and its versions and *read* its location, versioning and lifecycle settings, which is what the startup check needs; it cannot change them. |
|
| `BucketSettingsReadOnly` | `arn:aws:s3:::sast-memory` | The agent can list the bucket and its versions and *read* its location, versioning and lifecycle settings, which is what the startup check needs; it cannot change them. |
|
||||||
| `ObjectsReadWrite` | `arn:aws:s3:::sast-memory/*` | The agent reads, writes and deletes objects and their versions and reads and writes object tags; this covers row objects, blobs, the index, the Select probe, and the `kind` tag the store filters by. |
|
| `ObjectsReadWrite` | `arn:aws:s3:::sast-memory/*` | The agent reads, writes and deletes objects and their versions and reads and writes object tags; this covers row objects, blobs, the index, the Select probe, and the `kind` tag the store filters by. |
|
||||||
|
|
||||||
So the agent writes objects freely but cannot change bucket settings: turning versioning off or
|
So the agent writes objects freely but cannot change bucket settings. Turning versioning off or
|
||||||
removing the expiry rule needs the admin.
|
removing the expiry rule needs the admin.
|
||||||
|
|
||||||
## The `.env` settings
|
## The `.env` settings
|
||||||
|
|
||||||
The store reads its endpoint and credentials from `.env` in the working directory or from the
|
The store reads its endpoint and credentials from `.env` in the working directory or from the
|
||||||
environment (`s3_settings`), never from a manifest, and never prints them. The names are the ones
|
environment (`s3_settings`). It never reads them from a manifest. It never prints them. The names are
|
||||||
the AWS SDKs and CLI use for credentials, so one `.env` serves both the store and `aws s3api`.
|
the ones the AWS SDKs and CLI use for credentials. So one `.env` serves both the store and `aws s3api`.
|
||||||
|
|
||||||
| Variable | Value for this server | Notes |
|
| Variable | Value for this server | Notes |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
@@ -121,26 +128,33 @@ the AWS SDKs and CLI use for credentials, so one `.env` serves both the store an
|
|||||||
| `S3_REGION` | optional (`us-east-1` when unset) | the SigV4 signing region; spares a GetBucketLocation call |
|
| `S3_REGION` | optional (`us-east-1` when unset) | the SigV4 signing region; spares a GetBucketLocation call |
|
||||||
| `S3_BUCKET` | `sast-memory` | the bucket `s3://` names when the URL has none; also the real-server tests' default bucket |
|
| `S3_BUCKET` | `sast-memory` | the bucket `s3://` names when the URL has none; also the real-server tests' default bucket |
|
||||||
|
|
||||||
The client signs with SigV4 and uses path-style addressing (`https://host/bucket/key`), which
|
The client signs with SigV4 (AWS's request-signing scheme). It uses path-style addressing
|
||||||
RustFS needs. The SDK's own checksum trailers are off, because not every S3-compatible server
|
(`https://host/bucket/key`), which RustFS needs. The SDK's own checksum trailers are off. Not every
|
||||||
accepts them.
|
S3-compatible server accepts them.
|
||||||
|
|
||||||
- **Never commit `.env`.** It is gitignored (`.gitignore`: `.env`, `.env.*`, except
|
- **Never commit `.env`.** It is gitignored (`.gitignore`: `.env`, `.env.*`, except
|
||||||
`.env.example`). `.env.example` lists these names with empty values.
|
`.env.example`). `.env.example` lists these names with empty values.
|
||||||
- **`.env` never overrides a variable already exported in your shell** (`config.load_dotenv`).
|
- **`.env` never overrides a variable already exported in your shell** (`config.load_dotenv`).
|
||||||
A stale `export AWS_SECRET_ACCESS_KEY=...` in your profile silently wins over the file; `unset`
|
A stale `export AWS_SECRET_ACCESS_KEY=...` in your profile silently wins over the file. `unset`
|
||||||
it when a change in `.env` seems to have no effect.
|
it when a change in `.env` seems to have no effect.
|
||||||
- The store needs the `s3` extra (`uv sync --extra s3`, which installs boto3); without it opening
|
- The store needs the `s3` extra (`uv sync --extra s3`, which installs boto3). These are the
|
||||||
the store fails with `OUSAST_MEMORY=s3://... needs boto3: install openultrasast[s3]`. A missing
|
startup errors:
|
||||||
variable fails with `OUSAST_MEMORY=s3://... needs S3_ENDPOINT, AWS_ACCESS_KEY_ID,
|
- Without the extra, opening the store fails with
|
||||||
AWS_SECRET_ACCESS_KEY in .env or the environment` (naming only the missing ones); an endpoint
|
`OUSAST_MEMORY=s3://... needs boto3: install openultrasast[s3]`.
|
||||||
without a scheme fails with `S3_ENDPOINT must be a URL with a scheme (https://host[:port])`.
|
- A missing variable fails with `OUSAST_MEMORY=s3://... needs S3_ENDPOINT, AWS_ACCESS_KEY_ID,
|
||||||
|
AWS_SECRET_ACCESS_KEY in .env or the environment`. The message names only the missing ones.
|
||||||
|
- An endpoint without a scheme fails with
|
||||||
|
`S3_ENDPOINT must be a URL with a scheme (https://host[:port])`.
|
||||||
|
|
||||||
## What the store verifies at startup
|
## What the store verifies at startup
|
||||||
|
|
||||||
`S3Store.verify_bucket()` runs once per process for each bucket and prefix. It reads the
|
`S3Store.verify_bucket()` runs once per process for each bucket and prefix. It does four things:
|
||||||
versioning status and the lifecycle rules, writes a two-line probe object at
|
|
||||||
`_probe/select.jsonl` (under the prefix, if any), queries it with S3 Select and reads its tags.
|
1. It reads the versioning status.
|
||||||
|
2. It reads the lifecycle rules.
|
||||||
|
3. It writes a two-line probe object at `_probe/select.jsonl` (under the prefix, if any).
|
||||||
|
4. It queries the probe with S3 Select and reads its tags.
|
||||||
|
|
||||||
Every missing piece is collected into one error:
|
Every missing piece is collected into one error:
|
||||||
|
|
||||||
```text
|
```text
|
||||||
@@ -148,8 +162,8 @@ memory s3://sast-memory: the bucket is not set up for the store. The store never
|
|||||||
- <one line per missing piece>
|
- <one line per missing piece>
|
||||||
```
|
```
|
||||||
|
|
||||||
The lines, exactly as the code writes them for this bucket at the root (`<failure>` is the
|
The table lists the lines exactly as the code writes them for this bucket at the root. `<failure>` is
|
||||||
server's error code and the first 200 characters of its message):
|
the server's error code and the first 200 characters of its message.
|
||||||
|
|
||||||
| Check | Refusal line |
|
| Check | Refusal line |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
@@ -161,32 +175,42 @@ server's error code and the first 200 characters of its message):
|
|||||||
| tags unreadable | `object tags cannot be read on _probe/select.jsonl (<failure>): the store filters row objects by their kind tag, so the agent needs s3:GetObjectTagging on arn:aws:s3:::sast-memory/*` |
|
| tags unreadable | `object tags cannot be read on _probe/select.jsonl (<failure>): the store filters row objects by their kind tag, so the agent needs s3:GetObjectTagging on arn:aws:s3:::sast-memory/*` |
|
||||||
| S3 Select missing | `S3 Select (SelectObjectContent over JSON Lines) does not answer on _probe/select.jsonl (<failure>). The store requires it and has no local fallback: use a server with S3 Select (RustFS has it; not every S3-compatible server does) and let the agent s3:PutObject and s3:GetObject` |
|
| S3 Select missing | `S3 Select (SelectObjectContent over JSON Lines) does not answer on _probe/select.jsonl (<failure>). The store requires it and has no local fallback: use a server with S3 Select (RustFS has it; not every S3-compatible server does) and let the agent s3:PutObject and s3:GetObject` |
|
||||||
|
|
||||||
The admin commands inside the messages use `--endpoint-url "$S3_ENDPOINT"`, which is the full
|
The admin commands inside the messages use `--endpoint-url "$S3_ENDPOINT"`. That is the full URL from
|
||||||
URL from `.env`, so they run as written once `.env` is exported.
|
`.env`. So they run as written once `.env` is exported.
|
||||||
|
|
||||||
There is **no fallback**: the store does not start in a degraded mode, does not drop to a local
|
There is **no fallback**. The store does not:
|
||||||
store, and does not fetch and filter objects itself.
|
|
||||||
|
- start in a degraded mode;
|
||||||
|
- drop to a local store;
|
||||||
|
- fetch and filter objects itself.
|
||||||
|
|
||||||
## The RustFS Select limit and the fixed-schema rule
|
## The RustFS Select limit and the fixed-schema rule
|
||||||
|
|
||||||
RustFS's S3 Select infers an object's JSON schema from its leading rows. A `where` field that is
|
RustFS's S3 Select infers an object's JSON schema from its leading rows. A `where` field absent from
|
||||||
absent from those rows is unbound, even if a later row has it: the query fails with
|
those rows is unbound, even if a later row has it. The query then fails with
|
||||||
`EvaluatorBindingDoesNotExist` instead of matching (measured 2026-10-02 with the field first
|
`EvaluatorBindingDoesNotExist` instead of matching (measured 2026-10-02 with the field first
|
||||||
present at row 5000). The store does not read that as "no rows", since that answer could silently
|
present at row 5000).
|
||||||
drop matches; it raises:
|
|
||||||
|
The store does not read that failure as "no rows". That answer could silently drop matches. It raises:
|
||||||
|
|
||||||
```text
|
```text
|
||||||
memory s3://sast-memory: S3 Select on <key> failed (EvaluatorBindingDoesNotExist: ...) (a field of [<fields>] is absent from the object's leading rows, which the server reads as its schema); the store has no local fallback
|
memory s3://sast-memory: S3 Select on <key> failed (EvaluatorBindingDoesNotExist: ...) (a field of [<fields>] is absent from the object's leading rows, which the server reads as its schema); the store has no local fallback
|
||||||
```
|
```
|
||||||
|
|
||||||
The rule that follows: a field the store filters on is present in every row of its kind from
|
The rule that follows has two parts (see [Memory](memory.md#queryable-fields-per-kind)):
|
||||||
the first row on, as `""` where it does not apply; each row kind has a fixed set of queryable
|
|
||||||
fields, enforced at write time (see [Memory](memory.md#queryable-fields-per-kind)).
|
- A field the store filters on is present in every row of its kind from the first row on. It is `""`
|
||||||
|
where it does not apply.
|
||||||
|
- Each row kind has a fixed set of queryable fields. The store enforces it at write time.
|
||||||
|
|
||||||
## Checks against the real server
|
## Checks against the real server
|
||||||
|
|
||||||
The contract tests that talk to a real server run only with `OUSAST_MEMORY_TEST_S3=1` (bucket
|
The contract tests that talk to a real server run only with `OUSAST_MEMORY_TEST_S3=1`. They use the
|
||||||
`OUSAST_MEMORY_TEST_BUCKET`, else `S3_BUCKET`). They verify the bucket at its root, write only
|
bucket `OUSAST_MEMORY_TEST_BUCKET`, else `S3_BUCKET`. The tests:
|
||||||
under a fresh `contract-<id>/` prefix, and remove every version and delete marker under it
|
|
||||||
afterwards; they never configure the bucket. More on the host setup:
|
- verify the bucket at its root;
|
||||||
[ax on this host](ops/ax/README.md).
|
- write only under a fresh `contract-<id>/` prefix;
|
||||||
|
- remove every version and delete marker under it afterwards;
|
||||||
|
- never configure the bucket.
|
||||||
|
|
||||||
|
More on the host setup: [ax on this host](ops/ax/README.md).
|
||||||
|
|||||||
+200
-128
@@ -1,18 +1,26 @@
|
|||||||
# Token ergonomics
|
# Token ergonomics
|
||||||
|
|
||||||
Model tokens are the one running cost of this project, and they are spent in two places: by the
|
Model tokens are the one running cost of this project. They are spent in two places:
|
||||||
**plane** when a Run verifies candidates or compiles the decision engine, and by the maintainers'
|
|
||||||
own **agentic coding sessions**. The maintainer's framing (2026-09-29): "the token saving strategy
|
|
||||||
is a routing concern. ax can handle that and structure our services in small components." So the
|
|
||||||
runtime half is routing, one Model per Task with its own budget, and reuse of what was already
|
|
||||||
paid for; the development half is tooling that keeps bulk file contents out of the session.
|
|
||||||
|
|
||||||
Every number on this page comes from a committed record, named next to it. Records are counts
|
- by the **plane** (the system that runs scans as small tasks) when a Run verifies candidates
|
||||||
only; none holds code, prompts or responses.
|
or compiles the decision engine;
|
||||||
|
- by the maintainers' own **agentic coding sessions**.
|
||||||
|
|
||||||
The before-and-after summary (script pipeline against plane, and the development share) is on
|
The maintainer's framing (2026-09-29): "the token saving strategy is a routing concern. ax can
|
||||||
[Where we stand](where-we-stand.md#4-token-economics-before-and-after); this page holds the
|
handle that and structure our services in small components." Here ax is Google's controller
|
||||||
mechanisms and the detail table.
|
that runs agent tasks on Kubernetes. So the two halves look like this:
|
||||||
|
|
||||||
|
- **Runtime** is routing: one Model per Task with its own budget, and reuse of what was
|
||||||
|
already paid for.
|
||||||
|
- **Development** is tooling that keeps bulk file contents out of the session.
|
||||||
|
|
||||||
|
Every number on this page comes from a committed record, named next to it. Records hold counts
|
||||||
|
only. None holds code, prompts or responses.
|
||||||
|
|
||||||
|
The before-and-after summary is on
|
||||||
|
[Where we stand](where-we-stand.md#4-token-economics-before-and-after). It compares the script
|
||||||
|
pipeline with the plane and gives the development share. This page holds the mechanisms and the
|
||||||
|
detail table.
|
||||||
|
|
||||||
## A. Runtime: the plane's token economy
|
## A. Runtime: the plane's token economy
|
||||||
|
|
||||||
@@ -37,46 +45,67 @@ flowchart LR
|
|||||||
|
|
||||||
### One Model per Task, one budget per Task
|
### One Model per Task, one budget per Task
|
||||||
|
|
||||||
A Run is a DAG of ax Tasks; each Task binds at most one Model and its own `budget: {usd, calls}`
|
A Run is a DAG (a graph of steps with no cycles) of ax Tasks. Each Task binds at most one Model.
|
||||||
([The plane on ax](plane.md)). The budget is enforced by the metered client
|
Each Task has its own `budget: {usd, calls}` ([The plane on ax](plane.md)).
|
||||||
(`src/openultrasast/plane/budget.py`, `MeteredClient`): `check()` runs before every call and
|
|
||||||
raises `BudgetExhausted` once the spent `usd` or the call count reaches the ceiling, so the call
|
|
||||||
that crosses the ceiling completes and the next one is refused. The task then ends `unfinished`
|
|
||||||
and resumes from its `units.jsonl` on a rerun with a larger budget; an HTTP 401/402 or
|
|
||||||
"Insufficient Balance" is an `AccountError`, which fails the task and starts nothing further.
|
|
||||||
Usage is attributed per call: the wrapped client appends one usage row per call
|
|
||||||
(`prompt_tokens`, `prompt_cache_hit_tokens`, `completion_tokens`), rows are absorbed even when
|
|
||||||
the call raises, and they are priced by the Model's price list. An unpriced Model cannot run
|
|
||||||
under a `usd` budget, and `summary()` reports `usd: None` for it, never 0. Model-free tasks run
|
|
||||||
with `{usd: 0, calls: 0}`, so a stray call fails loudly.
|
|
||||||
|
|
||||||
`ousast plane status RUN` is the attribution table: one row per task with `calls`, `prompt`,
|
The metered client enforces the budget (`src/openultrasast/plane/budget.py`, `MeteredClient`):
|
||||||
`cache_hit`, `output` and `usd` from each task's `summary.json`, a `subtotal` row per Model and a
|
|
||||||
`total` (a missing value makes a sum `n/a`, never a smaller number); `--units` adds the per-unit
|
1. `check()` runs before every call.
|
||||||
rows. The same table is written as `attribution.json` in the run directory
|
2. It raises `BudgetExhausted` once the spent `usd` or the call count reaches the ceiling.
|
||||||
(`src/openultrasast/plane/reconciler.py`, `attribution()`/`status()`). Provider credentials
|
3. So the call that crosses the ceiling completes, and the next one is refused.
|
||||||
reach a task only in the start request: `router.py` reads the variable the Model's
|
4. The task then ends `unfinished`. A rerun with a larger budget resumes from its `units.jsonl`.
|
||||||
`secretKey.key` names, the runner keeps the value in memory, passes it to the command's
|
|
||||||
environment and redacts it from echoed stderr; it never appears in a manifest, a rendered Task
|
An HTTP 401/402 or "Insufficient Balance" is an `AccountError`. It fails the task and starts
|
||||||
or a file.
|
nothing further.
|
||||||
|
|
||||||
|
Usage is attributed per call:
|
||||||
|
|
||||||
|
- The wrapped client appends one usage row per call (`prompt_tokens`,
|
||||||
|
`prompt_cache_hit_tokens`, `completion_tokens`).
|
||||||
|
- Rows are absorbed even when the call raises.
|
||||||
|
- Rows are priced by the Model's price list.
|
||||||
|
- An unpriced Model cannot run under a `usd` budget. `summary()` reports `usd: None` for it,
|
||||||
|
never 0.
|
||||||
|
- Model-free tasks run with `{usd: 0, calls: 0}`, so a stray call fails loudly.
|
||||||
|
|
||||||
|
`ousast plane status RUN` prints the attribution table. It reads each task's `summary.json`
|
||||||
|
and shows:
|
||||||
|
|
||||||
|
- one row per task with `calls`, `prompt`, `cache_hit`, `output` and `usd`;
|
||||||
|
- a `subtotal` row per Model;
|
||||||
|
- a `total`. A missing value makes a sum `n/a`, never a smaller number.
|
||||||
|
|
||||||
|
`--units` adds the per-unit rows. The same table is written as `attribution.json` in the run
|
||||||
|
directory (`src/openultrasast/plane/reconciler.py`, `attribution()`/`status()`).
|
||||||
|
|
||||||
|
Provider credentials reach a task only in the start request:
|
||||||
|
|
||||||
|
- `router.py` reads the variable that the Model's `secretKey.key` names.
|
||||||
|
- The runner keeps the value in memory and passes it to the command's environment.
|
||||||
|
- The runner redacts it from echoed stderr.
|
||||||
|
- It never appears in a manifest, a rendered Task or a file.
|
||||||
|
|
||||||
### Pay once, reuse
|
### Pay once, reuse
|
||||||
|
|
||||||
- **Repo facts are computed model-free and reused.** `repo-facts` reads source text only (no
|
- **Repo facts are computed model-free and reused.**
|
||||||
model, no git) and its `facts.json` is byte-identical across runs over the same tree. The
|
- `repo-facts` reads source text only (no model, no git).
|
||||||
memory store keeps a facts entry under the key `(repo, pin, candidates digest, runner image
|
- Its `facts.json` is byte-identical across runs over the same tree.
|
||||||
digest)` (`src/openultrasast/plane/memory.py`, `memory_key()`); when a later Run's task
|
- The memory store keeps a facts entry under the key `(repo, pin, candidates digest, runner image
|
||||||
carries the same key, `seed()` writes the stored `facts.json`, a `done` summary with
|
digest)` (`src/openultrasast/plane/memory.py`, `memory_key()`).
|
||||||
`calls: 0` and no `usd`, and marks the task done without starting an actor. A changed pin,
|
- A later Run's task may carry the same key. Then `seed()` writes the stored `facts.json` and a
|
||||||
candidate set or image finds nothing and recomputes.
|
`done` summary with `calls: 0` and no `usd`. It marks the task done without starting an
|
||||||
- **Content-addressed memory** (`src/openultrasast/learn/`). Excerpts are stored as
|
actor.
|
||||||
`excerpts/<sha256 of the normalised text>.txt`; an embedding lives at
|
- A changed pin, candidate set or image finds nothing and recomputes.
|
||||||
`embeddings/<model>/<excerpt sha>.json` and "an excerpt is embedded once; a rerun makes no
|
- **Content-addressed memory** (`src/openultrasast/learn/`). Each item is stored under a hash
|
||||||
call" (`embeddings.py`). Model responses are cached under the sha256 of
|
of its content.
|
||||||
`{model, messages, params, temperature, sample}` (`program.py`, `request_key()`), and a
|
- Excerpts are stored as `excerpts/<sha256 of the normalised text>.txt`.
|
||||||
compiled program is `programs/<sha256 of its artifact>.json` with the digests of the memory
|
- An embedding lives at `embeddings/<model>/<excerpt sha>.json`. The source says: "an excerpt
|
||||||
snapshot and of the response cache it was built from (`compile.py`).
|
is embedded once; a rerun makes no call" (`embeddings.py`).
|
||||||
- **Replay is free.** With no client, the response cache replays at $0 and a miss raises
|
- Model responses are cached under the sha256 of `{model, messages, params, temperature, sample}`
|
||||||
|
(`program.py`, `request_key()`).
|
||||||
|
- A compiled program is `programs/<sha256 of its artifact>.json`. It records the digests of
|
||||||
|
the memory snapshot and of the response cache it was built from (`compile.py`).
|
||||||
|
- **Replay is free.** With no client, the response cache replays at $0. A miss raises
|
||||||
`ReplayMiss` instead of calling. The injection-slice record states it: "evaluate re-run
|
`ReplayMiss` instead of calling. The injection-slice record states it: "evaluate re-run
|
||||||
replay-only on the committed code: 550 responses replayed, $0, identical overall and
|
replay-only on the committed code: 550 responses replayed, $0, identical overall and
|
||||||
calibration" (`benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`,
|
calibration" (`benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`,
|
||||||
@@ -84,26 +113,35 @@ or a file.
|
|||||||
|
|
||||||
### Cheap before expensive
|
### Cheap before expensive
|
||||||
|
|
||||||
- **Triage first.** Verification runs only over the candidates the recorded triage kept; the
|
- **Triage first.** Verification runs only over the candidates the recorded triage kept.
|
||||||
plane applies that recording without a model call and prices it as a separate line
|
- The plane applies that recording without a model call.
|
||||||
(`recorded_triage_usd`), so every per-candidate cost below is reported both with and without
|
- It prices the recording as a separate line (`recorded_triage_usd`).
|
||||||
it (`src/openultrasast/plane/tasks/agree.py`).
|
- So every per-candidate cost below is reported both with and without it
|
||||||
|
(`src/openultrasast/plane/tasks/agree.py`).
|
||||||
- **Hunts are batched per file.** `verify` runs one tool hunt per file for up to `PER_HUNT = 6`
|
- **Hunts are batched per file.** `verify` runs one tool hunt per file for up to `PER_HUNT = 6`
|
||||||
candidates, with the candidates' known callers (capped at 8) in the prompt
|
candidates. The prompt includes the candidates' known callers, capped at 8
|
||||||
(`src/openultrasast/plane/tasks/verify.py`, `units_of()`).
|
(`src/openultrasast/plane/tasks/verify.py`, `units_of()`).
|
||||||
- **The tie-break pass runs only on disputes.** `agree` compares passes a and b; pass c is
|
- **The tie-break pass runs only on disputes.**
|
||||||
started with `inputs.only` set to the case's `disputed.json`, an empty list ends it `done`
|
- `agree` compares passes a and b.
|
||||||
with no model call, and the final verdict is 2-of-3 ("pass c alone never decides",
|
- Pass c starts with `inputs.only` set to the case's `disputed.json`.
|
||||||
`agree.py`).
|
- An empty list ends it `done` with no model call.
|
||||||
- **Cache-friendly prompt order.** The decision engine's prompt is the compiled prefix
|
- The final verdict is 2-of-3 ("pass c alone never decides", `agree.py`).
|
||||||
(instruction and fixed demonstrations, identical for every candidate so the provider's
|
- **Cache-friendly prompt order.** The decision engine's prompt has three parts, in this order
|
||||||
prefix cache hits), then the retrieved examples, then the candidate (`learn/program.py`).
|
(`learn/program.py`):
|
||||||
Its self-consistency `k` is tied to that cache: "k = 5 when samples 2..k hit the prefix cache
|
1. The compiled prefix: instruction and fixed demonstrations. It is identical for every
|
||||||
for >= 0.8 of their prompt tokens"; the measured repeat-hit share was 0.96 (injection-slice
|
candidate, so the provider's prefix cache hits.
|
||||||
record, `smoke.k_rule`, `smoke.k5_repeat_hit_share_measured`). The verify hunts share a prefix
|
2. The retrieved examples.
|
||||||
too: in the first plane increment 2,385,152 of 3,820,395 prompt tokens were cache hits
|
3. The candidate.
|
||||||
(`benchmarks/independent/plane-increment-1.json`, `detail.cache_hit_tokens`,
|
|
||||||
`detail.prompt_tokens`; its reading: "Cost fell 17% (62% of prompt tokens were cache hits)").
|
The self-consistency `k` (the number of samples per candidate) is tied to that cache: "k = 5
|
||||||
|
when samples 2..k hit the prefix cache for >= 0.8 of their prompt tokens". The measured
|
||||||
|
repeat-hit share was 0.96 (injection-slice record, `smoke.k_rule`,
|
||||||
|
`smoke.k5_repeat_hit_share_measured`).
|
||||||
|
|
||||||
|
The verify hunts share a prefix too. In the first plane increment, 2,385,152 of 3,820,395
|
||||||
|
prompt tokens were cache hits (`benchmarks/independent/plane-increment-1.json`,
|
||||||
|
`detail.cache_hit_tokens`, `detail.prompt_tokens`). The record's reading: "Cost fell 17% (62%
|
||||||
|
of prompt tokens were cache hits)".
|
||||||
|
|
||||||
### Measured costs
|
### Measured costs
|
||||||
|
|
||||||
@@ -116,37 +154,41 @@ or a file.
|
|||||||
| Decision engine harvest: `verify` part | $2.5373 over 1,961 calls, 474 tasks done, 10,921,167 prompt tokens of which 6,275,060 cache hits, 306,941 output tokens, task budgets summing to $8.624 under a $10 ceiling (`verify`) | `benchmarks/measurements/2026-10-01-decision-engine-harvest/record.json` |
|
| Decision engine harvest: `verify` part | $2.5373 over 1,961 calls, 474 tasks done, 10,921,167 prompt tokens of which 6,275,060 cache hits, 306,941 output tokens, task budgets summing to $8.624 under a $10 ceiling (`verify`) | `benchmarks/measurements/2026-10-01-decision-engine-harvest/record.json` |
|
||||||
| Decision engine harvest: `roles` part | $1.4036 over 1,034 calls, 151 tasks done, 2,398,669 prompt tokens of which 329,600 cache hits, 370,155 output tokens, task budgets summing to $4.917 under a $10 ceiling (`roles`) | same record |
|
| Decision engine harvest: `roles` part | $1.4036 over 1,034 calls, 151 tasks done, 2,398,669 prompt tokens of which 329,600 cache hits, 370,155 output tokens, task budgets summing to $4.917 under a $10 ceiling (`roles`) | same record |
|
||||||
|
|
||||||
The increment-2 record's note on the two denominators: "the reference divides by the 46
|
The increment-2 record notes that the two denominators differ: "the reference divides by the 46
|
||||||
candidates before triage; on that basis the plane is 7% cheaper, on the 43 after triage it is
|
candidates before triage; on that basis the plane is 7% cheaper, on the 43 after triage it is
|
||||||
0.2% under".
|
0.2% under".
|
||||||
|
|
||||||
!!! note "The meter prices at list; the account moved less"
|
!!! note "The meter prices at list; the account moved less"
|
||||||
The attribution table prices every call at the Model's list price
|
The attribution table prices every call at the Model's list price
|
||||||
(`plane/models/deepseek-flash.yaml`). The harvest record: "Spend is the plane's attribution
|
(`plane/models/deepseek-flash.yaml`). The harvest record says: "Spend is the plane's
|
||||||
table (`ousast plane status`, task meters priced at plane/models/deepseek-flash.yaml); the
|
attribution table (`ousast plane status`, task meters priced at
|
||||||
DeepSeek account moved less (balance below)", with the balance at $19.63 before, $18.75
|
plane/models/deepseek-flash.yaml); the DeepSeek account moved less (balance below)".
|
||||||
after `verify` and $18.23 after `roles`. The injection-slice record names the ratio: "the
|
The balance was $19.63 before, $18.75 after `verify` and $18.23 after `roles`.
|
||||||
provider charged ~1/3 of the list-price meter (balance moved $0.30 for $0.91 metered):
|
The injection-slice record names the ratio: "the provider charged ~1/3 of the list-price
|
||||||
DeepSeek's off-peak discount window, presumably; the meter prices at list"
|
meter (balance moved $0.30 for $0.91 metered): DeepSeek's off-peak discount window,
|
||||||
(`spend.balance_note`, balance $18.2 to $17.9). Budgets and gates are set against the
|
presumably; the meter prices at list" (`spend.balance_note`, balance $18.2 to $17.9).
|
||||||
meter, never against the balance.
|
Budgets and gates are set against the meter, never against the balance.
|
||||||
|
|
||||||
### Not done yet
|
### Not done yet
|
||||||
|
|
||||||
- **No model-side token budget in ax itself.** The plane's requirements state what ax does
|
- **No model-side token budget in ax itself.**
|
||||||
not yet carry: "task dependencies, artifacts, token budgets (on ax's roadmap), a DeepSeek
|
- The plane's requirements state what ax does not yet carry: "task dependencies, artifacts,
|
||||||
provider"; the thin `Run` layer carries them and dissolves into ax as ax gains them. Until
|
token budgets (on ax's roadmap), a DeepSeek provider".
|
||||||
then the ceiling lives in the metered client, not in the executor.
|
- The thin `Run` layer carries them. It dissolves into ax as ax gains them.
|
||||||
- **Triage is applied, not run.** The plane has no triage task; it filters by the recorded
|
- Until then the ceiling lives in the metered client, not in the executor.
|
||||||
triage and adds that recording's cost as its own line. A triage task is planned as a later
|
- **Triage is applied, not run.**
|
||||||
specification.
|
- The plane has no triage task.
|
||||||
- **`k` and the contrast examples are not knobs.** `k` is a compiled setting in {1, 3, 5}
|
- It filters by the recorded triage and adds that recording's cost as its own line.
|
||||||
(`learn/compile.toml`, "k and lam remain A/B variants"), changed only through the decision
|
- A triage task is planned as a later specification.
|
||||||
engine's A/B experiments, its requirement that "changes to prompts, models, pass counts, rule
|
- **`k` and the contrast examples are not knobs.**
|
||||||
sets and features [are] compared by controlled experiments, so that nothing is adopted on one
|
- `k` is a compiled setting in {1, 3, 5} (`learn/compile.toml`, "k and lam remain A/B
|
||||||
run's number" (Requirement 4 of its specification). The retrieval cap of at most two
|
variants").
|
||||||
contrast examples from other families is a module constant (`learn/retrieve.py`,
|
- It changes only through the decision engine's A/B experiments. Requirement 4 of its
|
||||||
`CONTRAST = 2`), a maintainer decision, not an experiment variant yet.
|
specification asks that "changes to prompts, models, pass counts, rule sets and features
|
||||||
|
[are] compared by controlled experiments, so that nothing is adopted on one run's number".
|
||||||
|
- Retrieval adds at most two contrast examples from other families. This cap is a module
|
||||||
|
constant (`learn/retrieve.py`, `CONTRAST = 2`).
|
||||||
|
- The cap is a maintainer decision, not an experiment variant yet.
|
||||||
|
|
||||||
## B. Development: the maintainers' own sessions
|
## B. Development: the maintainers' own sessions
|
||||||
|
|
||||||
@@ -165,28 +207,47 @@ flowchart LR
|
|||||||
|
|
||||||
### The measurement: tool I/O dominates
|
### The measurement: tool I/O dominates
|
||||||
|
|
||||||
`benchmarks/dev/token_report.py` reads a Claude Code session transcript (`.jsonl`; without an
|
`benchmarks/dev/token_report.py` reads a Claude Code session transcript (`.jsonl`). Without an
|
||||||
argument, the newest under `~/.claude/projects/`) and reports, per content kind, characters and
|
argument it reads the newest one under `~/.claude/projects/`.
|
||||||
share: `tool_use_input` (the tool call's input as JSON), `tool_result`, `thinking`,
|
|
||||||
`assistant:text`, `user:text`, plus the tool-call count and the top five tools. Tokens are
|
It reports characters and share per content kind:
|
||||||
approximated at four characters each. It prints to stdout and writes nothing; what gets
|
|
||||||
committed is the counts-only summary. The one in `benchmarks/independent/plane-increment-1.json`
|
- `tool_use_input` (the tool call's input as JSON);
|
||||||
(`dev_token_report`, for "the whole development session (c09e30b2), not the increment alone"):
|
- `tool_result`;
|
||||||
7.3M unique characters (~1.81M tokens), `tool_use_input_pct` 45.4, `tool_result_pct` 40.1,
|
- `thinking`;
|
||||||
`tool_io_pct` 85.5 against `previous_tool_io_pct` 90, over 2,701 tool calls. The reasoning the
|
- `assistant:text`;
|
||||||
session is paid for (assistant text and thinking) is the small remainder. That is why the
|
- `user:text`.
|
||||||
remedies below target file contents, not prose.
|
|
||||||
|
It also reports the tool-call count and the top five tools. Tokens are approximated at four
|
||||||
|
characters each. It prints to stdout and writes nothing. Only the counts-only summary is
|
||||||
|
committed.
|
||||||
|
|
||||||
|
The committed summary is in `benchmarks/independent/plane-increment-1.json`
|
||||||
|
(`dev_token_report`). It covers "the whole development session (c09e30b2), not the increment
|
||||||
|
alone". It counts 7.3M unique characters (~1.81M tokens). The shares
|
||||||
|
are `tool_use_input_pct` 45.4, `tool_result_pct` 40.1, and `tool_io_pct` 85.5 against
|
||||||
|
`previous_tool_io_pct` 90, over 2,701 tool calls.
|
||||||
|
|
||||||
|
The reasoning the session is paid for (assistant text and thinking) is the small remainder.
|
||||||
|
That is why the remedies below target file contents, not prose.
|
||||||
|
|
||||||
### The remedies in `benchmarks/dev/`
|
### The remedies in `benchmarks/dev/`
|
||||||
|
|
||||||
**`read_guard.py`**, a `PreToolUse` hook for agentic coding sessions. It reads the hook event on
|
**`read_guard.py`** is a `PreToolUse` hook (a script that runs before each tool call) for
|
||||||
stdin and, for the `Read` tool only, denies a whole-file read of a file longer than 350 lines
|
agentic coding sessions.
|
||||||
(`LIMIT = 350`); a read with `offset` or `limit`, a missing path, an unreadable file or a file
|
|
||||||
within the limit passes. It always exits 0 ("never fails the tool"). A denial is one JSON line
|
- It reads the hook event on stdin.
|
||||||
whose `permissionDecisionReason` tells the agent to read a range or to ask the bulk reader. The
|
- For the `Read` tool only, it denies a whole-file read of a file longer than 350 lines
|
||||||
script's docstring records that it is wired in `.claude/settings.local.json` (a local, untracked
|
(`LIMIT = 350`).
|
||||||
file) as `PreToolUse` with matcher `Read`; the setting has this shape, with the command
|
- These pass: a read with `offset` or `limit`, a missing path, an unreadable file, or a file
|
||||||
repo-relative because hooks run from the project directory:
|
within the limit.
|
||||||
|
- It always exits 0 ("never fails the tool").
|
||||||
|
- A denial is one JSON line. Its `permissionDecisionReason` tells the agent to read a range or
|
||||||
|
to ask the bulk reader.
|
||||||
|
|
||||||
|
The script's docstring records how it is wired. It sits in `.claude/settings.local.json` (a
|
||||||
|
local, untracked file) as `PreToolUse` with matcher `Read`. The setting has this shape. The
|
||||||
|
command is repo-relative because hooks run from the project directory:
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
@@ -203,37 +264,48 @@ repo-relative because hooks run from the project directory:
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
**`bulk_read.py`**, the portal pattern: a cheap model reads the files and answers one question,
|
**`bulk_read.py`** follows the portal pattern. A cheap model reads the files and answers one
|
||||||
citing `path:line` for every claim; it reports what the files say and decides nothing.
|
question. It cites `path:line` for every claim. It reports what the files say and decides
|
||||||
|
nothing.
|
||||||
|
|
||||||
```text
|
```text
|
||||||
.venv/bin/python benchmarks/dev/bulk_read.py "<question>" FILE [FILE ...] [--max-usd 0.05]
|
.venv/bin/python benchmarks/dev/bulk_read.py "<question>" FILE [FILE ...] [--max-usd 0.05]
|
||||||
```
|
```
|
||||||
|
|
||||||
Files go to the model in 400-line chunks with a `--- path (lines a-b)` header and numbered
|
How it works:
|
||||||
lines; the client is the project's detector client (`deepseek-flash` through `DEEPSEEK_API_KEY`,
|
|
||||||
otherwise OpenRouter), each call has a 90 s timeout and no tools. `--max-usd` (default 0.05) is
|
- Files go to the model in 400-line chunks, with a `--- path (lines a-b)` header and numbered
|
||||||
checked before every chunk; at the ceiling the answer ends with
|
lines.
|
||||||
`[stopped at the $X ceiling before: <chunk header>]`. The answer goes to stdout, the summary
|
- The client is the project's detector client: `deepseek-flash` through `DEEPSEEK_API_KEY`,
|
||||||
`[bulk_read: N chunk(s), $X.XXXX]` to stderr. Without a chat client it exits 2.
|
otherwise OpenRouter.
|
||||||
|
- Each call has a 90 s timeout and no tools.
|
||||||
|
- `--max-usd` (default 0.05) is checked before every chunk. At the ceiling the answer ends with
|
||||||
|
`[stopped at the $X ceiling before: <chunk header>]`.
|
||||||
|
- The answer goes to stdout. The summary `[bulk_read: N chunk(s), $X.XXXX]` goes to stderr.
|
||||||
|
- Without a chat client it exits 2.
|
||||||
|
|
||||||
**`token_report.py`**: `python benchmarks/dev/token_report.py [TRANSCRIPT.jsonl]`, described
|
**`token_report.py`**: `python benchmarks/dev/token_report.py [TRANSCRIPT.jsonl]`, described
|
||||||
above.
|
above.
|
||||||
|
|
||||||
### Practices
|
### Practices
|
||||||
|
|
||||||
- **Scripts into files, not heredocs.** A heredoc is `tool_use_input` and its output
|
- **Scripts into files, not heredocs.** A heredoc (a script typed inline in a shell command) is
|
||||||
`tool_result`; both are paid again at every later turn of the session. A script on disk is
|
`tool_use_input`, and its output is `tool_result`. Both are paid again at every later turn of
|
||||||
read once, by the interpreter.
|
the session. A script on disk is read once, by the interpreter.
|
||||||
- **Terse outputs.** Commands print counts, exit codes and the lines that matter; a harness
|
- **Terse outputs.** Commands print counts, exit codes and the lines that matter. A harness
|
||||||
fails loudly with the exit code rather than printing a table that has to be read back.
|
fails loudly with the exit code. It does not print a table that has to be read back.
|
||||||
- **Subagents with worktrees** for work whose output need not enter the main context: a
|
- **Subagents with worktrees** for work whose output need not enter the main context.
|
||||||
subagent's reads and command output stay in its own context and only its conclusion comes
|
- A subagent's reads and command output stay in its own context.
|
||||||
back; a git worktree keeps its edits off the main checkout.
|
- Only its conclusion comes back.
|
||||||
|
- A git worktree (a separate checkout of the same repository) keeps its edits off the main
|
||||||
|
checkout.
|
||||||
- **Counts-only measurement records.** A record under `benchmarks/measurements/` or
|
- **Counts-only measurement records.** A record under `benchmarks/measurements/` or
|
||||||
`benchmarks/independent/` holds numbers and their keys, never code, prompts or responses, so
|
`benchmarks/independent/` holds numbers and their keys. It never holds code, prompts or
|
||||||
it can be quoted on a page like this one and reread by a model at a few hundred tokens.
|
responses. So it can be quoted on a page like this one. A model can reread it at a few
|
||||||
|
hundred tokens.
|
||||||
|
|
||||||
Related: [The plane on ax](plane.md) for the Run and Task contract, [Memory](memory.md) for
|
Related:
|
||||||
the store the reuse keys live in, [Evaluation](evaluation.md) for the gates these costs are
|
|
||||||
measured against.
|
- [The plane on ax](plane.md) for the Run and Task contract;
|
||||||
|
- [Memory](memory.md) for the store the reuse keys live in;
|
||||||
|
- [Evaluation](evaluation.md) for the gates these costs are measured against.
|
||||||
|
|||||||
Reference in New Issue
Block a user