feat(plane): router-addressed start signal; credentials only in the start request

Agent Substrate's golden-snapshot boot runs the same image with the same task env,
so the runner no longer starts its command by itself: after the workspaces are
ready it waits for POST /ousast/v1/start (run/task checked, once per boot, refused
after a delivered run). The reconciler sends it through svc/atenet-router with
ate-target-actor, which never addresses the golden actor (plane/router.py), and the
bound Model's secretKey variable travels only in that body, into the child's env.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-09-29 19:41:35 +02:00
co-authored by Claude Opus 5.5
parent ac296ee3a4
commit 14ea48b1ea
10 changed files with 919 additions and 95 deletions
+33
View File
@@ -76,6 +76,27 @@ with a failed delivery retries the delivery only. SIGTERM (ax's stop and suspend
process group, followed by SIGKILL after `OUSAST_TERM_GRACE` (10 s), and the runner exits 0.
`AX_RUNNER_EXIT_AFTER_COMMAND=1` is a unit-test escape that exits with the task's code after delivery.
The command never starts by itself (Requirements 4.5, 4.6). Once the workspaces are ready the runner serves
`POST /ousast/v1/start` on port 80 with the body `{"run", "task", "credentials"}`: 503 before the workspaces are
ready (the reconciler retries), 409 when `run`/`task` differ from `OUSAST_RUN` and the Task's name, 409 for a
second request on the same boot, 409 with the reason when the completion marker records a delivered run, 400 for
a malformed body or a credential name outside `^[A-Z][A-Z0-9_]*$` (or under `AX_`/`OUSAST_`), and 202 once.
Only then does the command run -- or, when the marker records a failed delivery, only the delivery is retried; a
runner failure (bad manifests, a workspace that cannot be materialised) is likewise delivered only after a start,
or refused with its reason. The credentials stay in the runner's memory and are merged into the child's
environment only; they are never logged or written, and their values are redacted from the child's echoed
stderr and the stderr tail a crash summary keeps. `AX_RUNNER_AUTOSTART=1` (unit tests only; required with
`AX_RUNNER_HTTP=0`) starts without a request.
**Golden snapshot.** Agent Substrate boots every new ActorTemplate once as a temporary "golden" actor (atespace
`ate-golden`, a UUID name) to capture a snapshot. ax bakes `AX_TASK_YAML`, `AX_WORKSPACES_YAML` and `spec.env`
into the template, and inside the sandbox the hostname is always `actor` with no identity variable, so the golden
boot and the real one are indistinguishable from within (observed on the live cluster, 2026-09-29: the command
ran in both). The distinction exists only outside: Substrate's router (`svc/atenet-router` in `ate-system`)
routes on `ate-target-actor: <atespace>/<name>` to the task's actor, resuming it if suspended, and never to the
golden one. Hence the start request, and hence credentials only in it: a template, a rendered Task or an ax
manifest is visible to the golden boot and to anyone who can read the cluster's objects.
### Reconciler (`reconciler.py`, ax-backed)
`ousast plane run <Run.yaml>`, `ousast plane status <run>` and `ousast plane doctor`. State: `state.json` with
@@ -98,6 +119,18 @@ the token attribution table (Requirement 7.3) from the delivered summaries and w
the others re-executed; per-unit resume is the task's own duty (Requirement 2.2). No prompts, scoring or model
calls anywhere in this module; a test counts its lines.
Start (Requirements 4.5, 4.6; `router.py`, kept out of the reconciler's line budget): for the whole run the
reconciler holds one `kubectl --context kind-ousast -n ate-system port-forward svc/atenet-router :80` (random local
port read from kubectl's `Forwarding from 127.0.0.1:NNNN`; the child is terminated when the run ends;
`OUSAST_ROUTER_URL` replaces it, as in the tests). When ax reports `Running` after an accepted resume -- again
after a re-resume of a suspended task, where an "already started on this boot" answer means the command
survived and still runs -- it posts the start body with `ate-target-actor: <atespace>/<task>`,
retrying 502/503/504 and connection errors with backoff up to `OUSAST_START_TIMEOUT` (300 s); a 409 or the timeout
fails the task with the response text unless it already delivered. Credentials come from the Model binding: the
bound Model's `spec.secretKey.key` names the variable (`DEEPSEEK_API_KEY` for `deepseek-flash`), read from the
reconciler's own environment (`ousast` loads `.env` at startup); a Model-bound task whose variable is unset fails
before `ax apply`, and a task without a Model gets none. The value is never rendered, logged or written.
### Budget (`budget.py`)
Wraps a chat client: sums the provider's usage fields per call (`prompt_tokens`, `prompt_cache_hit_tokens`,
+23 -6
View File
@@ -41,12 +41,20 @@ mypy are clean, and the commit is gated on pytest's own exit code. Script-level
write `/workspace/.ousast-state/<task>.done.json`, and keep serving until SIGTERM (ax's contract for PID 1);
a boot with the marker never reruns the command and retries only a failed delivery; SIGTERM goes to the
command's group, SIGKILL after `OUSAST_TERM_GRACE`, exit 0. `AX_RUNNER_EXIT_AFTER_COMMAND=1` (tests only)
exits after delivery with 0 done, 2 failed, 3 unfinished. `plane/Dockerfile.runner`: this package on Python 3.12, installed to
exits after delivery with 0 done, 2 failed, 3 unfinished. The command never starts by itself: after the
workspaces are ready the runner waits for `POST /ousast/v1/start` (`{run, task, credentials}`; 503 before
ready, 409 on a run/task mismatch, a second request, or a marker recording a delivered run; 202 once),
merges the credentials into the child's environment only and redacts them from its echoed stderr, so
Agent Substrate's golden-snapshot boot runs nothing (Req 4.5, 4.6). `AX_RUNNER_AUTOSTART=1` (tests only)
starts without a request. `plane/Dockerfile.runner`: this package on Python 3.12, installed to
`/usr/local/bin/ax-task-runner`, loaded into kind by `ops/ax/up.sh`.
- `tests/test_plane_runner.py`: the env contract on a fixture workspace; readyz 503 then 200; delivery to a
local receiver; delivery failure yields `failed`; the child's cwd and `AX_METADATA_URL`; the runner stays
up after the command; SIGTERM kills a long-running command's group; the marker prevents a rerun.
- _Requirements: 4.1, 4.2, 4.3, 4.4_
up after the command; SIGTERM kills a long-running command's group; the marker prevents a rerun; no command
before a start request, 409 on a wrong run/task and on a second start, the credential reaches the child
(its hash in the artifact) and no file or log record, a crash printing it is redacted, and of two runners
with identical env only the addressed one runs (the golden scenario).
- _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 4.6_
- [ ] 4. ax-backed reconciler and CLI
- `src/openultrasast/plane/reconciler.py` (under 500 lines, no pipeline logic, no local subprocess
@@ -56,7 +64,12 @@ mypy are clean, and the commit is gated on pytest's own exit code. Script-level
none for a finished command); `Failed` or a vanished task before delivery, or no delivery within
`OUSAST_TASK_TIMEOUT`, fails the task with ax's condition message; the artifact HTTP receiver, `serialize` labels never overlapping, `--workers`, budgets and
artifact env passed in the rendered Task manifest, outcome read from the delivered `summary.json`, skip
`done` on rerun, PID lock per run, `ax delete` of finished Tasks.
`done` on rerun, PID lock per run, `ax delete` of finished Tasks. Once ax reports `Running` after a
resume, one start request per resume through Agent Substrate's router (`src/openultrasast/plane/router.py`:
`kubectl port-forward svc/atenet-router` for the run, or `OUSAST_ROUTER_URL`; header
`ate-target-actor: <atespace>/<task>`; 502/503/504 and connection errors retried up to
`OUSAST_START_TIMEOUT`, a 409 final), carrying the bound Model's `secretKey.key` variable from the
reconciler's environment; a Model-bound task whose variable is unset fails before `ax apply`.
- `ousast plane run <Run.yaml>`, `ousast plane status <run>` (token attribution table per task and Model
from the tasks' `summary.json` usage; `--units` from `units.jsonl`; also written to `attribution.json`),
`ousast plane workspaces <population.toml>` (generates Workspace manifests per case pin).
@@ -64,9 +77,13 @@ mypy are clean, and the commit is gated on pytest's own exit code. Script-level
failing DeadlineExceeded, Running without a Completed phase) drives a DAG: order, non-overlap,
skip-on-rerun, `unfinished` and `failed` propagation, resume retry and timeout, `Failed` with a condition
message, the task timeout without delivery, the lock, the attribution table, and the
line-count assertion. `tests/test_plane_ax_live.py` (marker `ax`, skipped without a live cluster): one
line-count assertion; with a fake router: one start per task after the accepted resume and before the
delivery, the credential in the verify start only and in no file under the run or tmp dirs, a missing
secret failing the task, a refused start failing it. `tests/test_plane_router.py`: port parsing from a
fake kubectl and its termination on exit, retry on 503/502 then 202, 409 raised with redacted text, the
start timeout. `tests/test_plane_ax_live.py` (marker `ax`, skipped without a live cluster): one
trivial Task end to end.
- _Requirements: 3.1, 3.2, 3.3, 3.4, 2.2, 7.1, 7.3_
- _Requirements: 3.1, 3.2, 3.3, 3.4, 2.2, 4.5, 4.6, 7.1, 7.3_
- [ ] 5. `repo-facts` task
- `src/openultrasast/plane/tasks/repo_facts.py`: product files, functions per file (per-language