Open_Ultra_SAST/.kiro/specs/ai-service-plane/design.md

Design Document

Overview

A thin service plane under src/openultrasast/plane/: manifest schemas in ax's shape plus a Run kind, a local reconciler that executes a Run's tasks in dependency order with persisted status and resume, a runner that honours ax's contract, and task entrypoints. The pipeline's logic lives only in the task entrypoints; the reconciler and runner know nothing about prompts, models or scoring (Requirement 3.4). The first increment moves repo-facts and verify (plus a model-free agree step) and measures them on the 46-candidate set (Requirement 6); the other concerns stay as scripts until that measurement is recorded.

Architecture

plane/                         manifests (YAML), checked in
  models/deepseek-flash.yaml     Model
  workspaces/<case>-<pin>.yaml   Workspace, generated from population-v2.toml by `ousast plane workspaces`
  tasks/repo-facts.yaml          Task
  tasks/verify.yaml              Task
  runs/validation-46.yaml        Run: DAG, artifacts, budgets
src/openultrasast/plane/
  manifests.py                   load + validate Task / Workspace / Model / Run (PyYAML, declared dependency)
  reconciler.py                  Run execution, status, resume  (<= 500 lines, test-enforced)
  runner.py                      ax runner contract: AX_TASK_YAML / AX_WORKSPACES_YAML -> entrypoint
  budget.py                      spend and call metering from provider usage fields
  tasks/repo_facts.py            source-only facts artifact
  tasks/verify.py                one hunt per file with facts in the prompt; per-unit resume
  tasks/agree.py                 pass agreement, no model
~/ousast-results/plane/<run>/    state.json + one directory per task with summary.json and artifacts

Locally the reconciler runs a task as a subprocess of the same interpreter with the ax environment set; in a container the same module is PID 1 (ax-task-runner), built as a target of the root Dockerfile that extends openultrasast:dev. Nothing in a task depends on which of the two started it.

Components and Interfaces

Manifests (manifests.py)

• Task, Workspace, Model: apiVersion: ax.io/v1alpha1, fields exactly as ax documents; unknown fields are rejected with the path named. Model.spec.provider accepts ax's google and anthropic, and deepseek only when metadata.annotations["openultrasast.io/provider-extension"] == "true" (Requirement 1.4): the deviation is declared, not hidden in another field.
• Run: apiVersion: openultrasast.io/v1alpha1. spec.tasks[]: name, task (a Task manifest name), dependsOn[], inputs{name: "<task>/<artifact>"}, outputs[], budget{usd, calls}, serialize (a label for tasks that must not overlap; engine-bearing tasks carry engine). The reconciler passes budgets and artifact paths to a task as env entries (OUSAST_BUDGET_USD, OUSAST_BUDGET_CALLS, OUSAST_INPUT_<NAME>, OUSAST_OUTPUT_DIR), which is ax's own mechanism for task configuration, so a Run never adds a field to a Task.

Runner (runner.py)

Reads AX_TASK_YAML and AX_WORKSPACES_YAML, materialises each workspace's git entries at the pinned commit into the bound path (locally a git archive from the case cache, as evaluate.export does today), writes spec.files, then executes spec.command, whose first element names a task module (openultrasast.plane.tasks.<name>). With AX_RUNNER_HTTP=1 it serves /healthz and /readyz (503 until workspaces are ready) on port 80 in a thread; locally the flag is unset (Requirement 4.2).

Reconciler (reconciler.py)

ousast plane run <Run.yaml> and ousast plane status <run>. State: state.json with per-task status in pending | running | suspended | done | failed | unfinished, timestamps, spend. Execution: topological order; tasks sharing a serialize label never overlap; everything else may run in parallel up to --workers. A task's outcome is read from its summary.json: done only when the task says every unit finished; unfinished on a budget stop; failed on an account or authentication error or a crash. On rerun, done tasks are skipped and the others re-executed; per-unit resume is the task's own duty (Requirement 2.2). No prompts, scoring or model calls anywhere in this module; a test counts its lines.

Budget (budget.py)

Wraps a chat client: sums the provider's usage fields per call (prompt_tokens, prompt_cache_hit_tokens, completion_tokens), prices them from the Model's parameters, raises BudgetExhausted at the ceiling and AccountError on 401/402 or an "insufficient balance" body. A model without prices makes the meter report unpriced and refuse to run under a usd budget (Requirement 2.4).

Task contract (all tasks)

Inputs from OUSAST_INPUT_* paths and the bound workspace only; outputs under OUSAST_OUTPUT_DIR; progress in a units.jsonl appended per finished unit and read back on start; summary.json written last with status, units_done, units_total, usd, calls, usage. A task never writes summary.json with status: done unless units_done == units_total.

repo-facts (tasks/repo_facts.py)

Source only, no model. For every product file (the enumerate_source_files + product filter of model_sinks.py): the functions it defines (the per-language declaration patterns of sink_candidates.py), and for each function name in the candidate list (input candidates.json) the call sites in other product files: path, line, the enclosing function, capped at 40 per name. Output facts.json, keyed by file, sorted, deterministic for a workspace (Requirement 5.2).

verify (tasks/verify.py)

The hunt of verify_batched.py (system prompt, one hunt per file, up to 6 candidates, 6 steps, candidate- anchored sites) moved into the package, with two changes: the prompt lists each candidate's known call sites from facts.json ("Known callers: a.py:12 in handler(), ..."), and each hunt records its tool turns and usage in units.jsonl. pass is an env value from the Run, so two passes are two task instances with distinct output directories. Candidates and triage come from the existing independent-v2-modelsinks and batched-check artifacts for the increment; the classify and triage tasks are later specs.

agree (tasks/agree.py)

Reads the pass outputs, emits agreed.json and disputed.json by the same rule as verify_batched.agreement, plus the per-candidate cost, turns and the declared-site match (evaluate.matches_v2) for the measurement.

Data Models

• facts.json: { "<path>": { "functions": ["name", ...], "callers": { "<fn>": [ {"path", "line", "in"} ] } } }
• units.jsonl (verify): one row per file hunt: {"path", "candidates", "flagged": [...], "turns", "usage", "usd"}; rows with "error" are re-asked on resume.
• summary.json: {"status", "units_done", "units_total", "usd", "calls", "usage", "reason"?}
• state.json (run): {"run", "started", "tasks": {"<name>": {"status", "started", "finished", "usd"}}}

Error Handling

• Manifest errors stop before any task starts, naming the manifest, field and rule.
• AccountError inside a task: the task writes summary.json with status: failed and the provider's message, exits 2; the reconciler marks the run failed and starts nothing further (Requirement 2.3).
• BudgetExhausted: status: unfinished, exit 3; dependants are not started; a rerun with a larger budget continues from units.jsonl.
• A crash: failed, traceback in summary.json; units.jsonl is still valid for resume.
• Two runs of the same Run name never share a directory: the reconciler refuses to start a second run while one is running (a lock file with the PID), which also keeps engine tasks to one at a time on this host.

Testing Strategy

• tests/test_plane_manifests.py: valid ax manifests load; an unknown field, a bad provider without the annotation, and a Run with a cycle are rejected with the field named.
• tests/test_plane_reconciler.py: stub tasks (shell commands writing summary.json) in a DAG: order, serialize non-overlap, skip-on-rerun, unfinished and failed propagation, the lock; a test asserts the module is under 500 lines.
• tests/test_plane_runner.py: the env contract on a fixture workspace; /readyz 503 then 200 with the flag.
• tests/test_plane_repo_facts.py: a three-file tree; callers found across files, none from tests or vendor; identical output on a second run.
• tests/test_plane_verify.py: a scripted client; the prompt carries the callers; units.jsonl resume asks only the unfinished file; a scripted 402 yields failed.
• Measurement (Requirement 6): plane/runs/validation-46.yaml executed twice (passes a and b); the result compared with verifier-batched-check-2026-09-29.json and recorded in benchmarks/independent/, with the transcript measure of the increment's development session next to it.