diff --git a/.kiro/specs/ai-service-plane/design.md b/.kiro/specs/ai-service-plane/design.md new file mode 100644 index 0000000..6eafd5e --- /dev/null +++ b/.kiro/specs/ai-service-plane/design.md @@ -0,0 +1,138 @@ +# Design Document + +## Overview + +A thin service plane under `src/openultrasast/plane/`: manifest schemas in ax's shape plus a `Run` kind, a local +reconciler that executes a Run's tasks in dependency order with persisted status and resume, a runner that +honours ax's contract, and task entrypoints. The pipeline's logic lives only in the task entrypoints; the +reconciler and runner know nothing about prompts, models or scoring (Requirement 3.4). The first increment moves +`repo-facts` and `verify` (plus a model-free `agree` step) and measures them on the 46-candidate set +(Requirement 6); the other concerns stay as scripts until that measurement is recorded. + +## Architecture + +``` +plane/ manifests (YAML), checked in + models/deepseek-flash.yaml Model + workspaces/-.yaml Workspace, generated from population-v2.toml by `ousast plane workspaces` + tasks/repo-facts.yaml Task + tasks/verify.yaml Task + runs/validation-46.yaml Run: DAG, artifacts, budgets +src/openultrasast/plane/ + manifests.py load + validate Task / Workspace / Model / Run (PyYAML, declared dependency) + reconciler.py Run execution, status, resume (<= 500 lines, test-enforced) + runner.py ax runner contract: AX_TASK_YAML / AX_WORKSPACES_YAML -> entrypoint + budget.py spend and call metering from provider usage fields + tasks/repo_facts.py source-only facts artifact + tasks/verify.py one hunt per file with facts in the prompt; per-unit resume + tasks/agree.py pass agreement, no model +~/ousast-results/plane// state.json + one directory per task with summary.json and artifacts +``` + +Locally the reconciler runs a task as a subprocess of the same interpreter with the ax environment set; in a +container the same module is PID 1 (`ax-task-runner`), built as a target of the root `Dockerfile` that extends +`openultrasast:dev`. Nothing in a task depends on which of the two started it. + +## Components and Interfaces + +### Manifests (`manifests.py`) + +- `Task`, `Workspace`, `Model`: `apiVersion: ax.io/v1alpha1`, fields exactly as ax documents; unknown fields + are rejected with the path named. `Model.spec.provider` accepts ax's `google` and `anthropic`, and `deepseek` + only when `metadata.annotations["openultrasast.io/provider-extension"] == "true"` (Requirement 1.4): the + deviation is declared, not hidden in another field. +- `Run`: `apiVersion: openultrasast.io/v1alpha1`. `spec.tasks[]`: `name`, `task` (a Task manifest name), + `dependsOn[]`, `inputs{name: "/"}`, `outputs[]`, `budget{usd, calls}`, `serialize` (a label + for tasks that must not overlap; engine-bearing tasks carry `engine`). The reconciler passes budgets and + artifact paths to a task as `env` entries (`OUSAST_BUDGET_USD`, `OUSAST_BUDGET_CALLS`, `OUSAST_INPUT_`, + `OUSAST_OUTPUT_DIR`), which is ax's own mechanism for task configuration, so a Run never adds a field to a + Task. + +### Runner (`runner.py`) + +Reads `AX_TASK_YAML` and `AX_WORKSPACES_YAML`, materialises each workspace's `git` entries at the pinned commit +into the bound path (locally a `git archive` from the case cache, as `evaluate.export` does today), writes +`spec.files`, then executes `spec.command`, whose first element names a task module +(`openultrasast.plane.tasks.`). With `AX_RUNNER_HTTP=1` it serves `/healthz` and `/readyz` (503 until +workspaces are ready) on port 80 in a thread; locally the flag is unset (Requirement 4.2). + +### Reconciler (`reconciler.py`) + +`ousast plane run ` and `ousast plane status `. State: `state.json` with per-task status in +`pending | running | suspended | done | failed | unfinished`, timestamps, spend. Execution: topological order; +tasks sharing a `serialize` label never overlap; everything else may run in parallel up to `--workers`. A task's +outcome is read from its `summary.json`: `done` only when the task says every unit finished; `unfinished` on a +budget stop; `failed` on an account or authentication error or a crash. On rerun, `done` tasks are skipped and +the others re-executed; per-unit resume is the task's own duty (Requirement 2.2). No prompts, scoring or model +calls anywhere in this module; a test counts its lines. + +### Budget (`budget.py`) + +Wraps a chat client: sums the provider's usage fields per call (`prompt_tokens`, `prompt_cache_hit_tokens`, +`completion_tokens`), prices them from the Model's parameters, raises `BudgetExhausted` at the ceiling and +`AccountError` on 401/402 or an "insufficient balance" body. A model without prices makes the meter report +`unpriced` and refuse to run under a `usd` budget (Requirement 2.4). + +### Task contract (all tasks) + +Inputs from `OUSAST_INPUT_*` paths and the bound workspace only; outputs under `OUSAST_OUTPUT_DIR`; progress in a +`units.jsonl` appended per finished unit and read back on start; `summary.json` written last with +`status`, `units_done`, `units_total`, `usd`, `calls`, `usage`. A task never writes `summary.json` with +`status: done` unless `units_done == units_total`. + +### `repo-facts` (`tasks/repo_facts.py`) + +Source only, no model. For every product file (the `enumerate_source_files` + product filter of +`model_sinks.py`): the functions it defines (the per-language declaration patterns of `sink_candidates.py`), +and for each function name in the candidate list (input `candidates.json`) the call sites in other product +files: `path`, `line`, the enclosing function, capped at 40 per name. Output `facts.json`, keyed by file, +sorted, deterministic for a workspace (Requirement 5.2). + +### `verify` (`tasks/verify.py`) + +The hunt of `verify_batched.py` (system prompt, one hunt per file, up to 6 candidates, 6 steps, candidate- +anchored sites) moved into the package, with two changes: the prompt lists each candidate's known call sites +from `facts.json` ("Known callers: a.py:12 in handler(), ..."), and each hunt records its tool turns and usage +in `units.jsonl`. `pass` is an env value from the Run, so two passes are two task instances with distinct output +directories. Candidates and triage come from the existing `independent-v2-modelsinks` and batched-check +artifacts for the increment; the `classify` and `triage` tasks are later specs. + +### `agree` (`tasks/agree.py`) + +Reads the pass outputs, emits `agreed.json` and `disputed.json` by the same rule as `verify_batched.agreement`, +plus the per-candidate cost, turns and the declared-site match (`evaluate.matches_v2`) for the measurement. + +## Data Models + +- `facts.json`: `{ "": { "functions": ["name", ...], "callers": { "": [ {"path", "line", "in"} ] } } }` +- `units.jsonl` (verify): one row per file hunt: `{"path", "candidates", "flagged": [...], "turns", "usage", + "usd"}`; rows with `"error"` are re-asked on resume. +- `summary.json`: `{"status", "units_done", "units_total", "usd", "calls", "usage", "reason"?}` +- `state.json` (run): `{"run", "started", "tasks": {"": {"status", "started", "finished", "usd"}}}` + +## Error Handling + +- Manifest errors stop before any task starts, naming the manifest, field and rule. +- `AccountError` inside a task: the task writes `summary.json` with `status: failed` and the provider's + message, exits 2; the reconciler marks the run `failed` and starts nothing further (Requirement 2.3). +- `BudgetExhausted`: `status: unfinished`, exit 3; dependants are not started; a rerun with a larger budget + continues from `units.jsonl`. +- A crash: `failed`, traceback in `summary.json`; `units.jsonl` is still valid for resume. +- Two runs of the same Run name never share a directory: the reconciler refuses to start a second run while one + is `running` (a lock file with the PID), which also keeps engine tasks to one at a time on this host. + +## Testing Strategy + +- `tests/test_plane_manifests.py`: valid ax manifests load; an unknown field, a bad provider without the + annotation, and a Run with a cycle are rejected with the field named. +- `tests/test_plane_reconciler.py`: stub tasks (shell commands writing `summary.json`) in a DAG: order, + `serialize` non-overlap, skip-on-rerun, `unfinished` and `failed` propagation, the lock; a test asserts the + module is under 500 lines. +- `tests/test_plane_runner.py`: the env contract on a fixture workspace; `/readyz` 503 then 200 with the flag. +- `tests/test_plane_repo_facts.py`: a three-file tree; callers found across files, none from tests or vendor; + identical output on a second run. +- `tests/test_plane_verify.py`: a scripted client; the prompt carries the callers; `units.jsonl` resume asks + only the unfinished file; a scripted 402 yields `failed`. +- Measurement (Requirement 6): `plane/runs/validation-46.yaml` executed twice (passes a and b); the result + compared with `verifier-batched-check-2026-09-29.json` and recorded in `benchmarks/independent/`, with the + transcript measure of the increment's development session next to it. diff --git a/.kiro/specs/ai-service-plane/spec.json b/.kiro/specs/ai-service-plane/spec.json index cb09cb0..bbf23b6 100644 --- a/.kiro/specs/ai-service-plane/spec.json +++ b/.kiro/specs/ai-service-plane/spec.json @@ -1,16 +1,16 @@ { "feature_name": "ai-service-plane", "created_at": "2026-09-29T15:40:00Z", - "updated_at": "2026-09-29T15:40:00Z", + "updated_at": "2026-09-29T16:05:00Z", "language": "en", - "phase": "requirements", + "phase": "design", "approvals": { "requirements": { "generated": true, - "approved": false + "approved": true }, "design": { - "generated": false, + "generated": true, "approved": false }, "tasks": {