spec(ai-service-plane): switch to ax completely; routing and token attribution requirements

Maintainer decisions 2026-09-29: ax (kind + Agent Substrate + ax controller on
this host) is the only executor; the Run layer carries dependencies, artifacts,
budgets and attribution until ax has them. Requirement 7 declares routing per
Task (one Model per Task, no runtime switching) and a token attribution table
from provider usage fields. Task 0 (bring-up, doctor), runner artifact delivery,
ax-backed reconciler with a fake-ax test seam. harnessx-removal brief records
the HarnessX inventory; removal is gated on the increment's measurement.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-09-29 17:42:39 +02:00
co-authored by Claude Fable 5.1
parent 152728c052
commit 6884bc8c2c
5 changed files with 141 additions and 44 deletions
+50 -12
View File
@@ -7,20 +7,28 @@ one long conversation, each with its own resume, budget and pass logic. Measured
tokens are tool I/O around those scripts, and verification re-derives the same per-repository facts in every
hunt and every pass. This feature restructures the pipeline as a Kubernetes-style service plane at the model
level, in the shape of google/ax: declarative `Task`, `Workspace` and `Model` manifests, a reconciler that
executes them in isolation with resume, and a runner contract. ax's software (Kubernetes, Agent Substrate) is a
later deployment target; its design is adopted now (`brief.md`).
executes them in isolation with resume, and a runner contract. The plane runs on ax itself: a `kind` cluster
with Agent Substrate and the ax controller on this host (maintainer decision 2026-09-29: "switch to ax
completely now"). What ax does not yet carry -- task dependencies, artifacts, token budgets (on ax's roadmap),
a DeepSeek provider -- is carried by our thin `Run` layer and dissolves into ax as ax gains it.
## Boundary Context
- **In scope**: the manifest kinds and a `Run` kind for dependencies, artifacts and budgets; a local reconciler
for this host (one engine container at a time); the runner contract; the first two tasks, `repo-facts` and
- **In scope**: the manifest kinds and a `Run` kind for dependencies, artifacts and budgets; bring-up of ax on
this host (kind + Agent Substrate + ax controller) with a doctor command; an ax-backed reconciler that
submits Tasks to ax (one engine-bearing task at a time) and is the only executor; the runner contract and a
runner image; the first two tasks, `repo-facts` and
`verify`, with the existing scripts as reference behaviour; measurement on the 46-candidate validation set.
- **Out of scope**: running on ax or Kubernetes; changing what the model is asked (prompts, families, pass rules)
- **Out of scope**: a remote or multi-node cluster; contributing providers or budgets to ax upstream (a
follow-on once the increment is measured); changing what the model is asked (prompts, families, pass rules)
except as required to consume `repo-facts`; the judge stage, the config-family question and population v3
(they become tasks later, under their own specs); any change to `evaluate.py`'s protocol scoring.
- **Adjacent expectations**: the independent-population protocols and records stay as they are; `finding_dump`
and the engine path are untouched; `vulnerabilities-over-plumbing` applies -- this is plumbing that must pay
for itself in measured tokens and unchanged or better recall.
- **Target state (maintainer, 2026-09-29)**: this plane replaces HarnessX as the agentic plane; HarnessX is
removed by its own spec (`harnessx-removal`) once Requirement 6's measurement shows value; the `Run` layer
shrinks as ax gains dependencies, budgets and artifacts. No new HarnessX feature is added meanwhile.
## Requirements
@@ -55,19 +63,25 @@ without ever recording partial work as a result.
4. A task's spend is metered from the provider's usage fields and stops the task at its budget; a task with no
priced model reports "unpriced", not zero.
### Requirement 3: A local reconciler executes a Run on this host
### Requirement 3: An ax-backed reconciler executes a Run on this host's ax
**User Story:** As the maintainer, I want to run a whole pipeline with one command on the laptop, with the same
manifests a cluster would take.
**User Story:** As the maintainer, I want to run a whole pipeline with one command on the laptop, with every
task executed by ax on the local cluster, so that the same manifests move to a bigger cluster unchanged.
#### Acceptance Criteria
1. The reconciler executes tasks in dependency order, at most one engine-bearing task at a time, and passes
artifacts by path.
1. The reconciler submits each task to ax as a `Task` (with its bound `Workspace`s and `Model`) in dependency
order, at most one engine-bearing task at a time, reads task outcome from ax's status plus the task's
delivered `summary.json`, and passes artifacts between tasks by placing a producer's outputs where the
consumer's manifest expects them. It never executes a task as a local subprocess; ax is the only executor.
2. Task status (`pending`, `running`, `suspended`, `done`, `failed`, `unfinished`) is persisted and shown by
one status command.
3. A suspended or interrupted run resumes from persisted status (Requirement 2.2) with the same command.
4. The reconciler is under 500 lines and holds no pipeline logic: no prompts, no scoring, no model calls.
5. `ousast plane doctor` verifies the host's ax in one command: kind cluster reachable, Agent Substrate control
API up, ax controller ready, runner image loaded; each check names what is missing and how to bring it up.
6. Bring-up on this host (kind + Agent Substrate + ax) is scripted, documented with its measured memory and
CPU footprint, and smoke-tested with one trivial Task before any pipeline task runs.
### Requirement 4: The runner contract is ax's
@@ -78,8 +92,12 @@ later.
1. An entrypoint receives its task and workspace manifests as YAML in `AX_TASK_YAML` and `AX_WORKSPACES_YAML`
and takes nothing else from the command line except what the manifest's `command` states.
2. The default image serves `/healthz` and `/readyz` when run as a container; locally these may be no-ops.
2. The runner image serves `/healthz` and `/readyz` on port 80, `/readyz` returning 503 until workspaces are
materialised, as ax's runner contract requires; the image contains this package and is loaded into kind.
3. One fixture test per task type runs the entrypoint against a small workspace and checks its artifacts.
4. Because ax has no artifact channel, the runner delivers the task's output directory (including
`summary.json` and `units.jsonl`) to the reconciler's artifact endpoint named in `OUSAST_ARTIFACT_URL`
before it reports completion; a delivery failure is reported as `failed`, never as done.
### Requirement 5: `repo-facts` is the memory layer
@@ -101,9 +119,29 @@ before the remaining concerns are converted.
#### Acceptance Criteria
1. `repo-facts` + `verify` run on the 46-candidate validation set through the reconciler, two passes, and
1. `repo-facts` + `verify` run on the 46-candidate validation set through the reconciler on ax, two passes, and
report cost per candidate, tool turns per hunt and declared sites agreed.
2. Acceptance: declared sites agreed at least 16 of 20 (the current value) and cost per candidate below the
current $0.022 for two passes; otherwise the increment is reported as not met and the design revisited.
3. The development transcript measure (`benchmarks/dev/token_report.py`) is run at the end of the increment and
recorded next to the result.
### Requirement 7: Routing is a declared concern of the plane
**User Story:** As the maintainer, I want which model serves which concern to be a manifest decision with a
budget, not a prompt trick inside one conversation, so that token spend is structured by component.
#### Acceptance Criteria
1. Every `Task` in a `Run` binds exactly one `Model`; a Run may bind different Models to different tasks
(a cheap model for reads and triage, a stronger one for verification and judgement), and the reconciler
passes the bound Model's identity and prices to the task.
2. A task never chooses or switches its model at runtime; the Model it was bound to is recorded in its
`summary.json`.
3. `ousast plane status <run>` prints a token attribution table: one row per task and Model binding with
calls, input tokens, cache-hit tokens, output tokens and USD, a total row, and a per-Model subtotal; every
number comes from the provider's usage fields recorded in the tasks' `summary.json`, never estimated. A
`--units` flag breaks a task down per unit (file or candidate) from `units.jsonl`. The same table is
written to `attribution.json` in the run directory so runs can be compared.
4. A task that needs a second model for a sub-step (a judge over a verifier's output) is a separate task with
its own Model binding, never a second client inside one task.