mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-21 14:09:27 +00:00
Marius CiepluchandClaude Opus 5.5
bb5c534fb2
The first live run on the local ax cluster and ax's source (docs/runner.md,
docs/sandbox.md, internal/controller/reconciler.go, internal/server/server.go)
showed a lifecycle the task-3/4 code did not model:
- `ax apply` creates a Task Suspended (Ready=False, TaskSuspended); it runs only
after `ax resume task`. The first resume of a new image can fail with
DeadlineExceeded while Agent Substrate builds the golden snapshot, and ax's
server then stores phase Failed; a later resume succeeds.
- ax has no Completed phase and never reads a command's exit back.
- The runner is PID 1 and must keep serving after the command exits; stop and
suspend deliver SIGTERM, to be forwarded to the command's process group.
Our runner imported the task in-process and exited, which stopped the
container ("Container guest is stopped") and made every resume time out.
- ax's controller probes /readyz?check=workspace; the query string made our
handler answer 404.
Runner (Req 4.1-4.4): spec.command runs as a child
`python -m openultrasast.plane.tasks.<name>` in its own process group, cwd the
first workspace, AX_METADATA_URL set; ax's metadata paths are served. After the
child exits the runner reads summary.json (a crash without one becomes failed
with the exit code and stderr tail), delivers the output directory, writes
/workspace/.ousast-state/<task>.done.json and serves until SIGTERM (forwarded
to the group, SIGKILL after OUSAST_TERM_GRACE, exit 0). A boot with the marker
never reruns the command; a failed delivery is retried alone. Workspaces are
prepared on the first boot only. AX_RUNNER_EXIT_AFTER_COMMAND=1 is the
unit-test escape.
Reconciler (Req 2.2, 2.3, 3.1-3.3): `ax resume task` after apply, retrying
DeadlineExceeded/Unavailable with backoff up to OUSAST_RESUME_TIMEOUT (900 s);
completion is the artifact delivery; Failed after a resume went through, a
vanished task, or no delivery within OUSAST_TASK_TIMEOUT (7200 s) fail the
task with ax's condition message; `ax delete task` follows. Ax gains resume()
and phase() returns the condition message. doctor moves to plane/doctor.py
(tasks.md task 0 allows the sibling) to keep reconciler.py under 500 lines.
Tests: the fake ax follows the real lifecycle (Suspended, a failing first
resume that leaves Failed, Running forever, delivery only after a resume went
through), plus Failed-with-message, vanish, resume-timeout and task-timeout
scenarios; runner tests cover the child's cwd/group/metadata URL, stay-up and
SIGTERM in a real subprocess, SIGKILL of a SIGTERM-ignoring group, and the
marker. The opt-in live test now checks resume, Running, suspend and a second
resume instead of a Completed phase. design.md and tasks.md state the
lifecycle; the module audit manifest counts plane.doctor.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
112 lines
8.3 KiB
Markdown
112 lines
8.3 KiB
Markdown
# Implementation Plan
|
|
|
|
Order follows the dependency chain: ax bring-up (parallel with 1-2) -> manifests -> budget -> runner ->
|
|
ax-backed reconciler -> tasks -> increment run on ax -> measurement. Every task ships with its tests; a task is done when its tests pass in the host suite, ruff and
|
|
mypy are clean, and the commit is gated on pytest's own exit code. Script-level pipeline work stays paused.
|
|
|
|
- [ ] 0. ax bring-up on this host
|
|
- Install `kind`, `kubectl`, the `ax` CLI and `ko` (user-local). `ops/ax/up.sh`: create the kind cluster and
|
|
install Agent Substrate with its `hack/create-kind-cluster.sh` and `hack/install-ate-kind.sh
|
|
--deploy-ate-system`, deploy the ax control plane with `ko` to the kind registry, apply one trivial Task
|
|
and wait for it to finish; `ops/ax/down.sh` tears it down. Record the measured idle footprint (RSS, CPU)
|
|
in `ops/ax/README.md` next to the 7 GB host constraint.
|
|
- `ousast plane doctor` (in `reconciler.py` or a sibling `doctor.py`): the four checks of Requirement 3.5.
|
|
- Stop rule: if the cluster does not come up within one working session, report the exact failure and stop;
|
|
the decision to work around it is the maintainer's.
|
|
- _Requirements: 3.5, 3.6_
|
|
|
|
- [ ] 1. Manifest schemas and validation
|
|
- Add `pyyaml` as a declared dependency in `pyproject.toml`.
|
|
- `src/openultrasast/plane/manifests.py`: dataclasses and loaders for `Task`, `Workspace`, `Model`
|
|
(`ax.io/v1alpha1`, ax's fields only) and `Run` (`openultrasast.io/v1alpha1`: tasks, dependsOn, inputs,
|
|
outputs, budget, serialize); reject unknown fields, a provider outside ax's two without the
|
|
`openultrasast.io/provider-extension` annotation, and a Run with a cycle or an input naming no task.
|
|
- `tests/test_plane_manifests.py`: valid manifests load; each rejection names the field.
|
|
- _Requirements: 1.1, 1.2, 1.3, 1.4_
|
|
|
|
- [ ] 2. Budget metering
|
|
- `src/openultrasast/plane/budget.py`: a client wrapper that sums provider usage per call, prices from the
|
|
Model's parameters, raises `BudgetExhausted` at the ceiling and `AccountError` on 401/402 or an
|
|
"insufficient balance" body; an unpriced model reports `unpriced` and refuses a `usd` budget.
|
|
- Tests with a scripted client: ceiling stop, account error, unpriced refusal, cache-hit pricing.
|
|
- _Requirements: 2.3, 2.4_
|
|
|
|
- [ ] 3. Runner contract
|
|
- `src/openultrasast/plane/runner.py`: read `AX_TASK_YAML` / `AX_WORKSPACES_YAML`; materialise `git`
|
|
entries at the pinned commit from the local case cache and write `spec.files` (first boot only); run
|
|
`spec.command` as a child `python -m openultrasast.plane.tasks.<name>` in its own process group, cwd the
|
|
first workspace, `AX_METADATA_URL` set; `/healthz` `/readyz` and ax's metadata paths on port 80 (503 until
|
|
workspaces are ready; disabled only under `AX_RUNNER_HTTP=0` for unit tests); after the command exits,
|
|
POST the output directory as a tar stream to `OUSAST_ARTIFACT_URL` (report `failed` if delivery fails),
|
|
write `/workspace/.ousast-state/<task>.done.json`, and keep serving until SIGTERM (ax's contract for PID 1);
|
|
a boot with the marker never reruns the command and retries only a failed delivery; SIGTERM goes to the
|
|
command's group, SIGKILL after `OUSAST_TERM_GRACE`, exit 0. `AX_RUNNER_EXIT_AFTER_COMMAND=1` (tests only)
|
|
exits after delivery with 0 done, 2 failed, 3 unfinished. `plane/Dockerfile.runner`: this package on Python 3.12, installed to
|
|
`/usr/local/bin/ax-task-runner`, loaded into kind by `ops/ax/up.sh`.
|
|
- `tests/test_plane_runner.py`: the env contract on a fixture workspace; readyz 503 then 200; delivery to a
|
|
local receiver; delivery failure yields `failed`; the child's cwd and `AX_METADATA_URL`; the runner stays
|
|
up after the command; SIGTERM kills a long-running command's group; the marker prevents a rerun.
|
|
- _Requirements: 4.1, 4.2, 4.3, 4.4_
|
|
|
|
- [ ] 4. ax-backed reconciler and CLI
|
|
- `src/openultrasast/plane/reconciler.py` (under 500 lines, no pipeline logic, no local subprocess
|
|
executor): `state.json`, statuses, topological submission through the `ax` CLI (path injectable),
|
|
`ax resume task` after `ax apply` (a created Task is `Suspended`), retrying DeadlineExceeded/Unavailable
|
|
with backoff up to `OUSAST_RESUME_TIMEOUT`; completion is the artifact delivery, never an ax phase (ax has
|
|
none for a finished command); `Failed` or a vanished task before delivery, or no delivery within
|
|
`OUSAST_TASK_TIMEOUT`, fails the task with ax's condition message; the artifact HTTP receiver, `serialize` labels never overlapping, `--workers`, budgets and
|
|
artifact env passed in the rendered Task manifest, outcome read from the delivered `summary.json`, skip
|
|
`done` on rerun, PID lock per run, `ax delete` of finished Tasks.
|
|
- `ousast plane run <Run.yaml>`, `ousast plane status <run>` (token attribution table per task and Model
|
|
from the tasks' `summary.json` usage; `--units` from `units.jsonl`; also written to `attribution.json`),
|
|
`ousast plane workspaces <population.toml>` (generates Workspace manifests per case pin).
|
|
- `tests/test_plane_reconciler.py`: a fake `ax` CLI with ax's real lifecycle (Suspended, a first resume
|
|
failing DeadlineExceeded, Running without a Completed phase) drives a DAG: order, non-overlap,
|
|
skip-on-rerun, `unfinished` and `failed` propagation, resume retry and timeout, `Failed` with a condition
|
|
message, the task timeout without delivery, the lock, the attribution table, and the
|
|
line-count assertion. `tests/test_plane_ax_live.py` (marker `ax`, skipped without a live cluster): one
|
|
trivial Task end to end.
|
|
- _Requirements: 3.1, 3.2, 3.3, 3.4, 2.2, 7.1, 7.3_
|
|
|
|
- [ ] 5. `repo-facts` task
|
|
- `src/openultrasast/plane/tasks/repo_facts.py`: product files, functions per file (per-language
|
|
declaration patterns), call sites per candidate function across product files (path, line, enclosing
|
|
function, cap 40), sorted and deterministic; `summary.json` per the task contract.
|
|
- `tests/test_plane_repo_facts.py`: three-file tree; cross-file callers found, tests and vendor excluded,
|
|
identical output on a second run.
|
|
- _Requirements: 5.1, 5.2, 2.1_
|
|
|
|
- [ ] 6. `verify` and `agree` tasks
|
|
- `src/openultrasast/plane/tasks/verify.py`: the batched hunt moved from `verify_batched.py` (one hunt per
|
|
file, up to 6 candidates, 6 steps, candidate-anchored sites) with known callers from `facts.json` in the
|
|
prompt, tool turns and usage per hunt in `units.jsonl`, per-unit resume, `pass` from env, budget wrapper
|
|
from task 2; `summary.json` never `done` unless every unit finished, and records the bound Model and
|
|
the summed usage fields (prompt, cache-hit, output tokens, calls, usd).
|
|
- `src/openultrasast/plane/tasks/agree.py`: agreement across pass outputs, per-candidate cost and turns,
|
|
declared-site matching via `evaluate.matches_v2`.
|
|
- `tests/test_plane_verify.py`: scripted client; prompt carries the callers; resume asks only the
|
|
unfinished file; a scripted 402 yields `failed`; agree on two scripted passes.
|
|
- _Requirements: 5.3, 2.1, 2.2, 2.3, 7.2_
|
|
|
|
- [ ] 7. Manifests for the increment
|
|
- `plane/models/deepseek-flash.yaml` (annotated extension, prices in parameters);
|
|
`plane/tasks/repo-facts.yaml`, `plane/tasks/verify.yaml`, `plane/tasks/agree.yaml`;
|
|
`plane/runs/validation-46.yaml`: repo-facts -> verify pass a, verify pass b -> agree, budgets per task,
|
|
inputs from the recorded candidate and triage artifacts.
|
|
- Workspaces generated for the 15 validation cases' vulnerable pins; the runner image rebuilt and loaded.
|
|
- _Requirements: 1.1, 1.2, 6.1_
|
|
|
|
- [ ] 8. Measurement of the first increment
|
|
- Run `validation-46` through the reconciler on ax (needs DeepSeek credit); record cost per candidate, tool turns
|
|
per hunt, declared sites agreed, against `verifier-batched-check-2026-09-29.json`.
|
|
- Run `benchmarks/dev/token_report.py` on the increment's development session.
|
|
- Record `benchmarks/independent/plane-increment-1.json` with the gate result (met or not met) and commit;
|
|
if not met, revise the design before task 9.
|
|
- _Requirements: 6.1, 6.2, 6.3_
|
|
|
|
- [ ] 9. Handover
|
|
- `ops/README.md`: how to run a Run locally and what an ax deployment would need (Kubernetes, Agent
|
|
Substrate, the runner image, the provider extension); memory note updated; scripts that the tasks replaced
|
|
marked as reference-only in their docstrings.
|
|
- _Requirements: 4.2_
|