Open_Ultra_SAST/.kiro/specs/ai-service-plane/design.md

Design Document

Overview

A thin service plane under src/openultrasast/plane/: manifest schemas in ax's shape plus a Run kind, an ax-backed reconciler that submits a Run's tasks to ax on this host's kind cluster in dependency order with persisted status and resume, a runner that honours ax's contract and delivers artifacts, and task entrypoints. ax (with Agent Substrate) is the only executor; there is no local subprocess path. The pipeline's logic lives only in the task entrypoints; the reconciler and runner know nothing about prompts, models or scoring (Requirement 3.4). The first increment moves repo-facts and verify (plus a model-free agree step) and measures them on the 46-candidate set (Requirement 6); the other concerns stay as scripts until that measurement is recorded.

Architecture

plane/                         manifests (YAML), checked in
  models/deepseek-flash.yaml     Model
  workspaces/<case>-<pin>.yaml   Workspace, generated from population-v2.toml by `ousast plane workspaces`
  tasks/repo-facts.yaml          Task
  tasks/verify.yaml              Task
  runs/validation-46.yaml        Run: DAG, artifacts, budgets
src/openultrasast/plane/
  manifests.py                   load + validate Task / Workspace / Model / Run (PyYAML, declared dependency)
  reconciler.py                  Run execution, status, resume  (<= 500 lines, test-enforced)
  runner.py                      ax runner contract: AX_TASK_YAML / AX_WORKSPACES_YAML -> entrypoint
  doctor.py                      `ousast plane doctor`: the four checks of Requirement 3.5
  budget.py                      spend and call metering from provider usage fields
  tasks/repo_facts.py            source-only facts artifact
  tasks/verify.py                one hunt per file with facts in the prompt; per-unit resume
  tasks/agree.py                 pass agreement, no model
~/ousast-results/plane/<run>/    state.json + one directory per task with summary.json and artifacts

Locally the reconciler runs a task as a subprocess of the same interpreter with the ax environment set; in a container the same module is PID 1 (ax-task-runner), built as a target of the root Dockerfile that extends openultrasast:dev. Nothing in a task depends on which of the two started it.

Components and Interfaces

Manifests (manifests.py)

• Task, Workspace, Model: apiVersion: ax.io/v1alpha1, fields exactly as ax documents; unknown fields are rejected with the path named. Model.spec.provider accepts ax's google and anthropic, and deepseek only when metadata.annotations["openultrasast.io/provider-extension"] == "true" (Requirement 1.4): the deviation is declared, not hidden in another field.
• Field names are ax's verbatim (ax.proto; its API server decodes with protojson, so an unknown field fails ax apply): metadata is name and atespace (not namespace), and a Workspace git entry is name, repo, branch, dir, depth (no commit; a commit there is rejected with a pointer to the annotation). metadata.annotations is this project's only extension, and nothing under it reaches ax: the reconciler strips it and carries it to the task as env. The Model binding (openultrasast.io/model) becomes OUSAST_MODEL and OUSAST_MODEL_PARAMS. Commit pins (openultrasast.io/git-commits: "<git name>=<40-hex sha>,..." on a Workspace) become OUSAST_GIT_PINS = {"<workspace>/<git name>": "<sha>"}, which the runner checks out after fetching the way ax's internal/workspace/setup.go does, with the same dir and depth rules.
• Run: apiVersion: openultrasast.io/v1alpha1. spec.tasks[]: name, task (a Task manifest name), dependsOn[], inputs{name: "<task>/<artifact>"}, outputs[], budget{usd, calls}, serialize (a label for tasks that must not overlap; engine-bearing tasks carry engine). The reconciler passes budgets and artifact paths to a task as env entries (OUSAST_BUDGET_USD, OUSAST_BUDGET_CALLS, OUSAST_INPUT_<NAME>, OUSAST_OUTPUT_DIR), which is ax's own mechanism for task configuration, so a Run never adds a field to a Task. Inputs another task produced are never rendered into a Workspace (ax puts every bound Workspace into one env value, AX_WORKSPACES_YAML, and Agent Substrate rejects values over 32768 characters): the Task carries OUSAST_INPUTS = {"<NAME>": "<producer>/<artifact>"}, and after the start request the runner fetches each from the receiver's GET /inputs/<producer>/<artifact> (same headers, dial and retries as delivery; only a running task's declared inputs are served) to its OUSAST_INPUT_<NAME> path; a failed fetch delivers a failed summary naming the input and the command never runs.

Runner (runner.py)

Reads AX_TASK_YAML and AX_WORKSPACES_YAML, materialises each workspace's git entries at the pinned commit into the bound path (locally a git archive from the case cache, as evaluate.export does today), writes spec.files, then runs spec.command as ax's runner contract requires (ax docs/runner.md, docs/sandbox.md): a child process python -m openultrasast.plane.tasks.<name> <args> in its own process group, with the first workspace as working directory and AX_METADATA_URL pointing at the runner's own server. It serves /healthz, /readyz (503 until workspaces are ready; ax probes /readyz?check=workspace) and ax's two metadata paths on port 80 (Requirement 4.2). Workspaces are prepared on the first boot only; /workspace survives suspend and resume. After the child exits the runner reads summary.json (a crash without one becomes failed with the exit code and stderr tail), delivers the output directory (Requirement 4.4), writes the completion marker /workspace/.ousast-state/<task>.done.json (exit, status, delivered) and then keeps running as PID 1 until SIGTERM: an exiting runner stops the container, which the first live run showed as every resume timing out. A boot that finds the marker does not rerun the command (a resumed actor must not repeat billed work); a marker with a failed delivery retries the delivery only. SIGTERM (ax's stop and suspend) is forwarded to the command's process group, followed by SIGKILL after OUSAST_TERM_GRACE (10 s), and the runner exits 0. AX_RUNNER_EXIT_AFTER_COMMAND=1 is a unit-test escape that exits with the task's code after delivery.

The command never starts by itself (Requirements 4.5, 4.6). Once the workspaces are ready the runner serves POST /ousast/v1/start on port 80 with the body {"run", "task", "credentials"}: 503 before the workspaces are ready (the reconciler retries), 409 when run/task differ from OUSAST_RUN and the Task's name, 409 for a second request on the same boot, 409 with the reason when the completion marker records a delivered run, 400 for a malformed body or a credential name outside ^[A-Z][A-Z0-9_]*$ (or under AX_/OUSAST_), and 202 once. Only then does the command run -- or, when the marker records a failed delivery, only the delivery is retried; a runner failure (bad manifests, a workspace that cannot be materialised) is likewise delivered only after a start, or refused with its reason. The credentials stay in the runner's memory and are merged into the child's environment only; they are never logged or written, and their values are redacted from the child's echoed stderr and the stderr tail a crash summary keeps. AX_RUNNER_AUTOSTART=1 (unit tests only; required with AX_RUNNER_HTTP=0) starts without a request.

Golden snapshot. Agent Substrate boots every new ActorTemplate once as a temporary "golden" actor (atespace ate-golden, a UUID name) to capture a snapshot. ax bakes AX_TASK_YAML, AX_WORKSPACES_YAML and spec.env into the template, and inside the sandbox the hostname is always actor with no identity variable, so the golden boot and the real one are indistinguishable from within (observed on the live cluster, 2026-09-29: the command ran in both). The distinction exists only outside: Substrate's router (svc/atenet-router in ate-system) routes on ate-target-actor: <atespace>/<name> to the task's actor, resuming it if suspended, and never to the golden one. Hence the start request, and hence credentials only in it: a template, a rendered Task or an ax manifest is visible to the golden boot and to anyone who can read the cluster's objects.

Reconciler (reconciler.py, ax-backed)

ousast plane run <Run.yaml>, ousast plane status <run> and ousast plane doctor. State: state.json with per-task status in pending | running | suspended | done | failed | unfinished, timestamps, spend. Execution: topological order; for each ready task the reconciler renders the Task manifest with the run's env (OUSAST_BUDGET_*, OUSAST_INPUT_*, OUSAST_OUTPUT_DIR, OUSAST_ARTIFACT_URL), applies it and its Workspaces through the ax CLI (ax apply, ax resume, ax get, ax delete; the CLI path is injectable so tests use a fake). ax creates a Task Suspended; it runs only after ax resume task, and the first resume of a new image can fail with DeadlineExceeded while Agent Substrate builds its golden snapshot (ax then reports Failed), so the reconciler retries a DeadlineExceeded/Unavailable resume with backoff for up to OUSAST_RESUME_TIMEOUT (900 s). ax has no Completed phase and never reports that a command exited: completion is the artifact delivery. Failed after a resume went through, or the task disappearing, before delivery fails the task with ax's condition message; no delivery within OUSAST_TASK_TIMEOUT (7200 s) fails it too; ax delete task follows either way. Artifacts: the reconciler serves a small HTTP receiver on the host (reachable from actors by name through the egress gateway, see Egress below); the runner posts the output directory there after its command exits. Tasks sharing a serialize label never overlap; everything else may run in parallel up to --workers. A task's outcome is read from its delivered summary.json: done only when the task says every unit finished; unfinished on a budget stop; failed on an account or authentication error, a crash, or a missing delivery. status prints the token attribution table (Requirement 7.3) from the delivered summaries and writes attribution.json. On rerun, done tasks are skipped and the others re-executed; per-unit resume is the task's own duty (Requirement 2.2). No prompts, scoring or model calls anywhere in this module; a test counts its lines.

Start (Requirements 4.5, 4.6; router.py, kept out of the reconciler's line budget): for the whole run the reconciler holds one kubectl --context kind-ousast -n ate-system port-forward svc/atenet-router :80 (random local port read from kubectl's Forwarding from 127.0.0.1:NNNN; the child is terminated when the run ends; OUSAST_ROUTER_URL replaces it, as in the tests). When ax reports Running after an accepted resume -- again after a re-resume of a suspended task, where an "already started on this boot" answer means the command survived and still runs -- it posts the start body with ate-target-actor: <atespace>/<task>, retrying 502/503/504 and connection errors with backoff up to OUSAST_START_TIMEOUT (300 s); a 409 or the timeout fails the task with the response text unless it already delivered. Credentials come from the Model binding: the bound Model's spec.secretKey.key names the variable (DEEPSEEK_API_KEY for deepseek-flash), read from the reconciler's own environment (ousast loads .env at startup); a Model-bound task whose variable is unset fails before ax apply, and a task without a Model gets none. The value is never rendered, logged or written.

Egress (egress.py)

Every actor connection except DNS goes through Agent Substrate's egress gateway (atenet-egress), which denies an actor without an EgressPolicy. Three pieces make the plane work under that rule:

• Receiver by name. ops/ax/receiver-service.yaml.tmpl (applied by up.sh) is a selector-less Service ousast-receiver in ax-system, port 80, with an EndpointSlice on the kind gateway at OUSAST_ARTIFACT_PORT (fixed, 18090), where the reconciler binds its receiver. Tasks get OUSAST_ARTIFACT_URL=http://ousast-receiver.ax-system.svc.cluster.local/ plus OUSAST_ARTIFACT_DIAL=<the Service's ClusterIP>:80 (looked up with kubectl unless set): the actor cannot resolve cluster names, so the runner connects to the dial address and sends the URL's host as Host. The gateway decides the policy on Host but connects to the address the actor dialled (measured: dialling the kind gateway on port 80 was allowed and then refused upstream), so the dial address is the ClusterIP, which kube-proxy maps to the host. OUSAST_ARTIFACT_HOST set keeps the old direct URL (tests).
• Per-task policy. policy_for grants http on port 80 to the receiver name, and tls_passthrough on 443 to the Git hosts of the bound Workspaces' https repos and to the bound Model's openultrasast.io/egress-hosts (api.deepseek.com for deepseek-flash, validated as a plain DNS name). Nothing else: no IP address or wildcard, and passthrough only -- an https rule intercepts TLS and the actor would need the gateway CA, which ax templates do not mount.
• Order. ax apply creates the actor; the reconciler writes the policy with kubectl ate create egress-policy (update when one exists), reads it back, waits OUSAST_EGRESS_SETTLE (11 s: the gateway caches a missing policy as deny for 10 s), then resumes. kubectl-ate has no delete verb; Substrate's store deletes the policy with its actor, so after ax delete task the reconciler confirms it is gone.

Budget (budget.py)

Wraps a chat client: sums the provider's usage fields per call (prompt_tokens, prompt_cache_hit_tokens, completion_tokens), prices them from the Model's parameters, raises BudgetExhausted at the ceiling and AccountError on 401/402 or an "insufficient balance" body. A model without prices makes the meter report unpriced and refuse to run under a usd budget (Requirement 2.4).

Task contract (all tasks)

Inputs from OUSAST_INPUT_* paths and the bound workspace only; outputs under OUSAST_OUTPUT_DIR; progress in a units.jsonl appended per finished unit and read back on start; summary.json written last with status, units_done, units_total, usd, calls, usage. A task never writes summary.json with status: done unless units_done == units_total.

repo-facts (tasks/repo_facts.py)

Source only, no model. For every product file (the enumerate_source_files + product filter of model_sinks.py): the functions it defines (the per-language declaration patterns of sink_candidates.py), and for each function name in the candidate list (input candidates.json) the call sites in other product files: path, line, the enclosing function, capped at 40 per name. Output facts.json, keyed by file, sorted, deterministic for a workspace (Requirement 5.2).

verify (tasks/verify.py)

The hunt of verify_batched.py (system prompt, one hunt per file, up to 6 candidates, 6 steps, candidate- anchored sites) moved into the package, with two changes: the prompt lists each candidate's known call sites from facts.json ("Known callers: a.py:12 in handler(), ..."), and each hunt records its tool turns and usage in units.jsonl. pass is an env value from the Run, so two passes are two task instances with distinct output directories. Candidates and triage come from the existing independent-v2-modelsinks and batched-check artifacts for the increment; the classify and triage tasks are later specs.

agree (tasks/agree.py)

Reads the pass outputs, emits agreed.json and disputed.json by the same rule as verify_batched.agreement, plus the per-candidate cost, turns and the declared-site match (evaluate.matches_v2) for the measurement. The rule is reproduced in the package (agree.matches_declared, parity-tested against matches_v2 on the recorded reference outputs) and the generator writes the fix ranges it matches against into each case's case.json, so the runner image carries no benchmark tree.

Measurement. Requirement 6.2's cost is judged on cost_per_candidate_with_recorded_triage: the plane's two verify passes plus the recorded triage cost of the case, over the candidates before triage, summed across the run. The reference's $0.022 (verifier-batched-check-2026-09-29.json) paid for per-file triage and the hunts while the plane applies the recorded triage without a model call, so cost_per_candidate (plane spend alone) is reported but not judged; the triage share is derived per file from the recorded spend minus the priced hunt usage (about $0.087 of the $1.01) and labelled as derived in case.json. Sites are judged as declared_sites_matched_agreed against declared_sites_in_set, the reference's 16 of 20. The tie-break (Requirement 6.4) appends per case <case>-vc, verify pass c restricted by OUSAST_INPUT_ONLY to the agree task's disputed.json (an empty list is done with no model call), and <case>-final, agree over passes a, b and c, which decides 2-of-3 where pass c asked and keeps the a/b rule elsewhere, so pass c alone never decides and metrics.tiebreak counts the flips; the first 60 entries stay byte-identical, so a rerun keeps their done state, executes only the 30 new tasks, and pass c's cost adds to usd_total for the 6.2 judgement.

Data Models

• facts.json: { "<path>": { "functions": ["name", ...], "callers": { "<fn>": [ {"path", "line", "in"} ] } } }
• units.jsonl (verify): one row per file hunt: {"path", "candidates", "flagged": [...], "turns", "usage", "usd"}; rows with "error" are re-asked on resume.
• summary.json: {"status", "units_done", "units_total", "usd", "calls", "usage", "reason"?}
• state.json (run): {"run", "started", "tasks": {"<name>": {"status", "started", "finished", "usd"}}}

Error Handling

• Manifest errors stop before any task starts, naming the manifest, field and rule.
• AccountError inside a task: the task writes summary.json with status: failed and the provider's message, exits 2; the reconciler marks the run failed and starts nothing further (Requirement 2.3).
• BudgetExhausted: status: unfinished, exit 3; dependants are not started; a rerun with a larger budget continues from units.jsonl.
• A crash: failed, traceback in summary.json; units.jsonl is still valid for resume.
• Two runs of the same Run name never share a directory: the reconciler refuses to start a second run while one is running (a lock file with the PID), which also keeps engine tasks to one at a time on this host.

Testing Strategy

• tests/test_plane_manifests.py: valid ax manifests load; an unknown field, a bad provider without the annotation, and a Run with a cycle are rejected with the field named.
• tests/test_plane_reconciler.py: a fake ax CLI with ax's real lifecycle (apply -> Suspended, a first resume failing DeadlineExceeded, then Running forever, and a scripted summary.json delivery only after a resume went through) drives a DAG: order, serialize non-overlap, skip-on-rerun, unfinished and failed propagation, resume retry and timeout, Failed with a condition message, a vanished task, the task timeout without delivery, the lock, the attribution table; a test asserts the module is under 500 lines. tests/test_plane_ax_live.py (marker ax, skipped unless ousast plane doctor passes) applies one trivial Task on the real cluster.
• tests/test_plane_runner.py: the env contract on a fixture workspace; /readyz 503 then 200; the command as a child in its own group in the first workspace; the runner serving on after the command; SIGTERM reaching the command's group; the completion marker preventing a rerun and retrying a failed delivery alone.
• tests/test_plane_repo_facts.py: a three-file tree; callers found across files, none from tests or vendor; identical output on a second run.
• tests/test_plane_verify.py: a scripted client; the prompt carries the callers; units.jsonl resume asks only the unfinished file; a scripted 402 yields failed.
• Measurement (Requirement 6): plane/runs/validation-46.yaml executed twice (passes a and b); the result compared with verifier-batched-check-2026-09-29.json and recorded in benchmarks/independent/, with the transcript measure of the increment's development session next to it.