marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-20 13:39:27 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
bb5c534fb2464285b7cd6c605e1695d6bcb26f37
Open_Ultra_SAST/.kiro/specs/ai-service-plane/tasks.md
T
Marius CiepluchandClaude Opus 5.5 bb5c534fb2 fix(plane): ax task lifecycle -- resume, stay-up runner, delivery as completion 
The first live run on the local ax cluster and ax's source (docs/runner.md,
docs/sandbox.md, internal/controller/reconciler.go, internal/server/server.go)
showed a lifecycle the task-3/4 code did not model:

- `ax apply` creates a Task Suspended (Ready=False, TaskSuspended); it runs only
  after `ax resume task`. The first resume of a new image can fail with
  DeadlineExceeded while Agent Substrate builds the golden snapshot, and ax's
  server then stores phase Failed; a later resume succeeds.
- ax has no Completed phase and never reads a command's exit back.
- The runner is PID 1 and must keep serving after the command exits; stop and
  suspend deliver SIGTERM, to be forwarded to the command's process group.
  Our runner imported the task in-process and exited, which stopped the
  container ("Container guest is stopped") and made every resume time out.
- ax's controller probes /readyz?check=workspace; the query string made our
  handler answer 404.

Runner (Req 4.1-4.4): spec.command runs as a child
`python -m openultrasast.plane.tasks.<name>` in its own process group, cwd the
first workspace, AX_METADATA_URL set; ax's metadata paths are served. After the
child exits the runner reads summary.json (a crash without one becomes failed
with the exit code and stderr tail), delivers the output directory, writes
/workspace/.ousast-state/<task>.done.json and serves until SIGTERM (forwarded
to the group, SIGKILL after OUSAST_TERM_GRACE, exit 0). A boot with the marker
never reruns the command; a failed delivery is retried alone. Workspaces are
prepared on the first boot only. AX_RUNNER_EXIT_AFTER_COMMAND=1 is the
unit-test escape.

Reconciler (Req 2.2, 2.3, 3.1-3.3): `ax resume task` after apply, retrying
DeadlineExceeded/Unavailable with backoff up to OUSAST_RESUME_TIMEOUT (900 s);
completion is the artifact delivery; Failed after a resume went through, a
vanished task, or no delivery within OUSAST_TASK_TIMEOUT (7200 s) fail the
task with ax's condition message; `ax delete task` follows. Ax gains resume()
and phase() returns the condition message. doctor moves to plane/doctor.py
(tasks.md task 0 allows the sibling) to keep reconciler.py under 500 lines.

Tests: the fake ax follows the real lifecycle (Suspended, a failing first
resume that leaves Failed, Running forever, delivery only after a resume went
through), plus Failed-with-message, vanish, resume-timeout and task-timeout
scenarios; runner tests cover the child's cwd/group/metadata URL, stay-up and
SIGTERM in a real subprocess, SIGKILL of a SIGTERM-ignoring group, and the
marker. The opt-in live test now checks resume, Running, suspend and a second
resume instead of a Completed phase. design.md and tasks.md state the
lifecycle; the module audit manifest counts plane.doctor.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-09-29 19:04:52 +02:00

8.3 KiB
Raw Blame History

Implementation Plan

Order follows the dependency chain: ax bring-up (parallel with 1-2) -> manifests -> budget -> runner -> ax-backed reconciler -> tasks -> increment run on ax -> measurement. Every task ships with its tests; a task is done when its tests pass in the host suite, ruff and mypy are clean, and the commit is gated on pytest's own exit code. Script-level pipeline work stays paused.

  • 0. ax bring-up on this host

    • Install kind, kubectl, the ax CLI and ko (user-local). ops/ax/up.sh: create the kind cluster and install Agent Substrate with its hack/create-kind-cluster.sh and hack/install-ate-kind.sh --deploy-ate-system, deploy the ax control plane with ko to the kind registry, apply one trivial Task and wait for it to finish; ops/ax/down.sh tears it down. Record the measured idle footprint (RSS, CPU) in ops/ax/README.md next to the 7 GB host constraint.
    • ousast plane doctor (in reconciler.py or a sibling doctor.py): the four checks of Requirement 3.5.
    • Stop rule: if the cluster does not come up within one working session, report the exact failure and stop; the decision to work around it is the maintainer's.
    • Requirements: 3.5, 3.6

  • 1. Manifest schemas and validation

    • Add pyyaml as a declared dependency in pyproject.toml.
    • src/openultrasast/plane/manifests.py: dataclasses and loaders for Task, Workspace, Model (ax.io/v1alpha1, ax's fields only) and Run (openultrasast.io/v1alpha1: tasks, dependsOn, inputs, outputs, budget, serialize); reject unknown fields, a provider outside ax's two without the openultrasast.io/provider-extension annotation, and a Run with a cycle or an input naming no task.
    • tests/test_plane_manifests.py: valid manifests load; each rejection names the field.
    • Requirements: 1.1, 1.2, 1.3, 1.4

  • 2. Budget metering

    • src/openultrasast/plane/budget.py: a client wrapper that sums provider usage per call, prices from the Model's parameters, raises BudgetExhausted at the ceiling and AccountError on 401/402 or an "insufficient balance" body; an unpriced model reports unpriced and refuses a usd budget.
    • Tests with a scripted client: ceiling stop, account error, unpriced refusal, cache-hit pricing.
    • Requirements: 2.3, 2.4

  • 3. Runner contract

    • src/openultrasast/plane/runner.py: read AX_TASK_YAML / AX_WORKSPACES_YAML; materialise git entries at the pinned commit from the local case cache and write spec.files (first boot only); run spec.command as a child python -m openultrasast.plane.tasks.<name> in its own process group, cwd the first workspace, AX_METADATA_URL set; /healthz /readyz and ax's metadata paths on port 80 (503 until workspaces are ready; disabled only under AX_RUNNER_HTTP=0 for unit tests); after the command exits, POST the output directory as a tar stream to OUSAST_ARTIFACT_URL (report failed if delivery fails), write /workspace/.ousast-state/<task>.done.json, and keep serving until SIGTERM (ax's contract for PID 1); a boot with the marker never reruns the command and retries only a failed delivery; SIGTERM goes to the command's group, SIGKILL after OUSAST_TERM_GRACE, exit 0. AX_RUNNER_EXIT_AFTER_COMMAND=1 (tests only) exits after delivery with 0 done, 2 failed, 3 unfinished. plane/Dockerfile.runner: this package on Python 3.12, installed to /usr/local/bin/ax-task-runner, loaded into kind by ops/ax/up.sh.
    • tests/test_plane_runner.py: the env contract on a fixture workspace; readyz 503 then 200; delivery to a local receiver; delivery failure yields failed; the child's cwd and AX_METADATA_URL; the runner stays up after the command; SIGTERM kills a long-running command's group; the marker prevents a rerun.
    • Requirements: 4.1, 4.2, 4.3, 4.4

  • 4. ax-backed reconciler and CLI

    • src/openultrasast/plane/reconciler.py (under 500 lines, no pipeline logic, no local subprocess executor): state.json, statuses, topological submission through the ax CLI (path injectable), ax resume task after ax apply (a created Task is Suspended), retrying DeadlineExceeded/Unavailable with backoff up to OUSAST_RESUME_TIMEOUT; completion is the artifact delivery, never an ax phase (ax has none for a finished command); Failed or a vanished task before delivery, or no delivery within OUSAST_TASK_TIMEOUT, fails the task with ax's condition message; the artifact HTTP receiver, serialize labels never overlapping, --workers, budgets and artifact env passed in the rendered Task manifest, outcome read from the delivered summary.json, skip done on rerun, PID lock per run, ax delete of finished Tasks.
    • ousast plane run <Run.yaml>, ousast plane status <run> (token attribution table per task and Model from the tasks' summary.json usage; --units from units.jsonl; also written to attribution.json), ousast plane workspaces <population.toml> (generates Workspace manifests per case pin).
    • tests/test_plane_reconciler.py: a fake ax CLI with ax's real lifecycle (Suspended, a first resume failing DeadlineExceeded, Running without a Completed phase) drives a DAG: order, non-overlap, skip-on-rerun, unfinished and failed propagation, resume retry and timeout, Failed with a condition message, the task timeout without delivery, the lock, the attribution table, and the line-count assertion. tests/test_plane_ax_live.py (marker ax, skipped without a live cluster): one trivial Task end to end.
    • Requirements: 3.1, 3.2, 3.3, 3.4, 2.2, 7.1, 7.3

  • 5. repo-facts task

    • src/openultrasast/plane/tasks/repo_facts.py: product files, functions per file (per-language declaration patterns), call sites per candidate function across product files (path, line, enclosing function, cap 40), sorted and deterministic; summary.json per the task contract.
    • tests/test_plane_repo_facts.py: three-file tree; cross-file callers found, tests and vendor excluded, identical output on a second run.
    • Requirements: 5.1, 5.2, 2.1

  • 6. verify and agree tasks

    • src/openultrasast/plane/tasks/verify.py: the batched hunt moved from verify_batched.py (one hunt per file, up to 6 candidates, 6 steps, candidate-anchored sites) with known callers from facts.json in the prompt, tool turns and usage per hunt in units.jsonl, per-unit resume, pass from env, budget wrapper from task 2; summary.json never done unless every unit finished, and records the bound Model and the summed usage fields (prompt, cache-hit, output tokens, calls, usd).
    • src/openultrasast/plane/tasks/agree.py: agreement across pass outputs, per-candidate cost and turns, declared-site matching via evaluate.matches_v2.
    • tests/test_plane_verify.py: scripted client; prompt carries the callers; resume asks only the unfinished file; a scripted 402 yields failed; agree on two scripted passes.
    • Requirements: 5.3, 2.1, 2.2, 2.3, 7.2

  • 7. Manifests for the increment

    • plane/models/deepseek-flash.yaml (annotated extension, prices in parameters); plane/tasks/repo-facts.yaml, plane/tasks/verify.yaml, plane/tasks/agree.yaml; plane/runs/validation-46.yaml: repo-facts -> verify pass a, verify pass b -> agree, budgets per task, inputs from the recorded candidate and triage artifacts.
    • Workspaces generated for the 15 validation cases' vulnerable pins; the runner image rebuilt and loaded.
    • Requirements: 1.1, 1.2, 6.1

  • 8. Measurement of the first increment

    • Run validation-46 through the reconciler on ax (needs DeepSeek credit); record cost per candidate, tool turns per hunt, declared sites agreed, against verifier-batched-check-2026-09-29.json.
    • Run benchmarks/dev/token_report.py on the increment's development session.
    • Record benchmarks/independent/plane-increment-1.json with the gate result (met or not met) and commit; if not met, revise the design before task 9.
    • Requirements: 6.1, 6.2, 6.3

  • 9. Handover

    • ops/README.md: how to run a Run locally and what an ax deployment would need (Kubernetes, Agent Substrate, the runner image, the provider extension); memory note updated; scripts that the tasks replaced marked as reference-only in their docstrings.
    • Requirements: 4.2
Reference in New Issue View Git Blame Copy Permalink