From 6884bc8c2cc5a4bebc15828f9ab728da560d0db8 Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Tue, 29 Sep 2026 17:42:39 +0200 Subject: [PATCH] spec(ai-service-plane): switch to ax completely; routing and token attribution requirements Maintainer decisions 2026-09-29: ax (kind + Agent Substrate + ax controller on this host) is the only executor; the Run layer carries dependencies, artifacts, budgets and attribution until ax has them. Requirement 7 declares routing per Task (one Model per Task, no runtime switching) and a token attribution table from provider usage fields. Task 0 (bring-up, doctor), runner artifact delivery, ax-backed reconciler with a fake-ax test seam. harnessx-removal brief records the HarnessX inventory; removal is gated on the increment's measurement. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo --- .kiro/specs/ai-service-plane/design.md | 33 +++++++---- .kiro/specs/ai-service-plane/requirements.md | 62 ++++++++++++++++---- .kiro/specs/ai-service-plane/spec.json | 2 +- .kiro/specs/ai-service-plane/tasks.md | 59 +++++++++++++------ .kiro/specs/harnessx-removal/brief.md | 29 +++++++++ 5 files changed, 141 insertions(+), 44 deletions(-) create mode 100644 .kiro/specs/harnessx-removal/brief.md diff --git a/.kiro/specs/ai-service-plane/design.md b/.kiro/specs/ai-service-plane/design.md index 6eafd5e..63e4b8b 100644 --- a/.kiro/specs/ai-service-plane/design.md +++ b/.kiro/specs/ai-service-plane/design.md @@ -2,9 +2,10 @@ ## Overview -A thin service plane under `src/openultrasast/plane/`: manifest schemas in ax's shape plus a `Run` kind, a local -reconciler that executes a Run's tasks in dependency order with persisted status and resume, a runner that -honours ax's contract, and task entrypoints. The pipeline's logic lives only in the task entrypoints; the +A thin service plane under `src/openultrasast/plane/`: manifest schemas in ax's shape plus a `Run` kind, an +ax-backed reconciler that submits a Run's tasks to ax on this host's kind cluster in dependency order with +persisted status and resume, a runner that honours ax's contract and delivers artifacts, and task entrypoints. +ax (with Agent Substrate) is the only executor; there is no local subprocess path. The pipeline's logic lives only in the task entrypoints; the reconciler and runner know nothing about prompts, models or scoring (Requirement 3.4). The first increment moves `repo-facts` and `verify` (plus a model-free `agree` step) and measures them on the 46-candidate set (Requirement 6); the other concerns stay as scripts until that measurement is recorded. @@ -56,13 +57,19 @@ into the bound path (locally a `git archive` from the case cache, as `evaluate.e (`openultrasast.plane.tasks.`). With `AX_RUNNER_HTTP=1` it serves `/healthz` and `/readyz` (503 until workspaces are ready) on port 80 in a thread; locally the flag is unset (Requirement 4.2). -### Reconciler (`reconciler.py`) +### Reconciler (`reconciler.py`, ax-backed) -`ousast plane run ` and `ousast plane status `. State: `state.json` with per-task status in -`pending | running | suspended | done | failed | unfinished`, timestamps, spend. Execution: topological order; -tasks sharing a `serialize` label never overlap; everything else may run in parallel up to `--workers`. A task's -outcome is read from its `summary.json`: `done` only when the task says every unit finished; `unfinished` on a -budget stop; `failed` on an account or authentication error or a crash. On rerun, `done` tasks are skipped and +`ousast plane run `, `ousast plane status ` and `ousast plane doctor`. State: `state.json` with +per-task status in `pending | running | suspended | done | failed | unfinished`, timestamps, spend. Execution: +topological order; for each ready task the reconciler renders the Task manifest with the run's env +(`OUSAST_BUDGET_*`, `OUSAST_INPUT_*`, `OUSAST_OUTPUT_DIR`, `OUSAST_ARTIFACT_URL`), applies it and its +Workspaces through the `ax` CLI (`ax apply`, `ax get`, `ax delete`; the CLI path is injectable so tests use a +fake), and polls ax's status. Artifacts: the reconciler serves a small HTTP receiver on the host (reachable from +kind pods via the node's host gateway); the runner posts the output directory there at exit. Tasks sharing a +`serialize` label never overlap; everything else may run in parallel up to `--workers`. A task's outcome is +read from its delivered `summary.json`: `done` only when the task says every unit finished; `unfinished` on a +budget stop; `failed` on an account or authentication error, a crash, or a missing delivery. `status` prints +the token attribution table (Requirement 7.3) from the delivered summaries and writes `attribution.json`. On rerun, `done` tasks are skipped and the others re-executed; per-unit resume is the task's own duty (Requirement 2.2). No prompts, scoring or model calls anywhere in this module; a test counts its lines. @@ -125,9 +132,11 @@ plus the per-candidate cost, turns and the declared-site match (`evaluate.matche - `tests/test_plane_manifests.py`: valid ax manifests load; an unknown field, a bad provider without the annotation, and a Run with a cycle are rejected with the field named. -- `tests/test_plane_reconciler.py`: stub tasks (shell commands writing `summary.json`) in a DAG: order, - `serialize` non-overlap, skip-on-rerun, `unfinished` and `failed` propagation, the lock; a test asserts the - module is under 500 lines. +- `tests/test_plane_reconciler.py`: a fake `ax` CLI (a script recording `apply`/`get`/`delete` calls and + posting scripted `summary.json` deliveries to the receiver) drives a DAG: order, `serialize` non-overlap, + skip-on-rerun, `unfinished` and `failed` propagation, missing delivery, the lock, the attribution table; a + test asserts the module is under 500 lines. `tests/test_plane_ax_live.py` (marker `ax`, skipped unless + `ousast plane doctor` passes) applies one trivial Task on the real cluster. - `tests/test_plane_runner.py`: the env contract on a fixture workspace; `/readyz` 503 then 200 with the flag. - `tests/test_plane_repo_facts.py`: a three-file tree; callers found across files, none from tests or vendor; identical output on a second run. diff --git a/.kiro/specs/ai-service-plane/requirements.md b/.kiro/specs/ai-service-plane/requirements.md index e807244..579842b 100644 --- a/.kiro/specs/ai-service-plane/requirements.md +++ b/.kiro/specs/ai-service-plane/requirements.md @@ -7,20 +7,28 @@ one long conversation, each with its own resume, budget and pass logic. Measured tokens are tool I/O around those scripts, and verification re-derives the same per-repository facts in every hunt and every pass. This feature restructures the pipeline as a Kubernetes-style service plane at the model level, in the shape of google/ax: declarative `Task`, `Workspace` and `Model` manifests, a reconciler that -executes them in isolation with resume, and a runner contract. ax's software (Kubernetes, Agent Substrate) is a -later deployment target; its design is adopted now (`brief.md`). +executes them in isolation with resume, and a runner contract. The plane runs on ax itself: a `kind` cluster +with Agent Substrate and the ax controller on this host (maintainer decision 2026-09-29: "switch to ax +completely now"). What ax does not yet carry -- task dependencies, artifacts, token budgets (on ax's roadmap), +a DeepSeek provider -- is carried by our thin `Run` layer and dissolves into ax as ax gains it. ## Boundary Context -- **In scope**: the manifest kinds and a `Run` kind for dependencies, artifacts and budgets; a local reconciler - for this host (one engine container at a time); the runner contract; the first two tasks, `repo-facts` and +- **In scope**: the manifest kinds and a `Run` kind for dependencies, artifacts and budgets; bring-up of ax on + this host (kind + Agent Substrate + ax controller) with a doctor command; an ax-backed reconciler that + submits Tasks to ax (one engine-bearing task at a time) and is the only executor; the runner contract and a + runner image; the first two tasks, `repo-facts` and `verify`, with the existing scripts as reference behaviour; measurement on the 46-candidate validation set. -- **Out of scope**: running on ax or Kubernetes; changing what the model is asked (prompts, families, pass rules) +- **Out of scope**: a remote or multi-node cluster; contributing providers or budgets to ax upstream (a + follow-on once the increment is measured); changing what the model is asked (prompts, families, pass rules) except as required to consume `repo-facts`; the judge stage, the config-family question and population v3 (they become tasks later, under their own specs); any change to `evaluate.py`'s protocol scoring. - **Adjacent expectations**: the independent-population protocols and records stay as they are; `finding_dump` and the engine path are untouched; `vulnerabilities-over-plumbing` applies -- this is plumbing that must pay for itself in measured tokens and unchanged or better recall. +- **Target state (maintainer, 2026-09-29)**: this plane replaces HarnessX as the agentic plane; HarnessX is + removed by its own spec (`harnessx-removal`) once Requirement 6's measurement shows value; the `Run` layer + shrinks as ax gains dependencies, budgets and artifacts. No new HarnessX feature is added meanwhile. ## Requirements @@ -55,19 +63,25 @@ without ever recording partial work as a result. 4. A task's spend is metered from the provider's usage fields and stops the task at its budget; a task with no priced model reports "unpriced", not zero. -### Requirement 3: A local reconciler executes a Run on this host +### Requirement 3: An ax-backed reconciler executes a Run on this host's ax -**User Story:** As the maintainer, I want to run a whole pipeline with one command on the laptop, with the same -manifests a cluster would take. +**User Story:** As the maintainer, I want to run a whole pipeline with one command on the laptop, with every +task executed by ax on the local cluster, so that the same manifests move to a bigger cluster unchanged. #### Acceptance Criteria -1. The reconciler executes tasks in dependency order, at most one engine-bearing task at a time, and passes - artifacts by path. +1. The reconciler submits each task to ax as a `Task` (with its bound `Workspace`s and `Model`) in dependency + order, at most one engine-bearing task at a time, reads task outcome from ax's status plus the task's + delivered `summary.json`, and passes artifacts between tasks by placing a producer's outputs where the + consumer's manifest expects them. It never executes a task as a local subprocess; ax is the only executor. 2. Task status (`pending`, `running`, `suspended`, `done`, `failed`, `unfinished`) is persisted and shown by one status command. 3. A suspended or interrupted run resumes from persisted status (Requirement 2.2) with the same command. 4. The reconciler is under 500 lines and holds no pipeline logic: no prompts, no scoring, no model calls. +5. `ousast plane doctor` verifies the host's ax in one command: kind cluster reachable, Agent Substrate control + API up, ax controller ready, runner image loaded; each check names what is missing and how to bring it up. +6. Bring-up on this host (kind + Agent Substrate + ax) is scripted, documented with its measured memory and + CPU footprint, and smoke-tested with one trivial Task before any pipeline task runs. ### Requirement 4: The runner contract is ax's @@ -78,8 +92,12 @@ later. 1. An entrypoint receives its task and workspace manifests as YAML in `AX_TASK_YAML` and `AX_WORKSPACES_YAML` and takes nothing else from the command line except what the manifest's `command` states. -2. The default image serves `/healthz` and `/readyz` when run as a container; locally these may be no-ops. +2. The runner image serves `/healthz` and `/readyz` on port 80, `/readyz` returning 503 until workspaces are + materialised, as ax's runner contract requires; the image contains this package and is loaded into kind. 3. One fixture test per task type runs the entrypoint against a small workspace and checks its artifacts. +4. Because ax has no artifact channel, the runner delivers the task's output directory (including + `summary.json` and `units.jsonl`) to the reconciler's artifact endpoint named in `OUSAST_ARTIFACT_URL` + before it reports completion; a delivery failure is reported as `failed`, never as done. ### Requirement 5: `repo-facts` is the memory layer @@ -101,9 +119,29 @@ before the remaining concerns are converted. #### Acceptance Criteria -1. `repo-facts` + `verify` run on the 46-candidate validation set through the reconciler, two passes, and +1. `repo-facts` + `verify` run on the 46-candidate validation set through the reconciler on ax, two passes, and report cost per candidate, tool turns per hunt and declared sites agreed. 2. Acceptance: declared sites agreed at least 16 of 20 (the current value) and cost per candidate below the current $0.022 for two passes; otherwise the increment is reported as not met and the design revisited. 3. The development transcript measure (`benchmarks/dev/token_report.py`) is run at the end of the increment and recorded next to the result. + +### Requirement 7: Routing is a declared concern of the plane + +**User Story:** As the maintainer, I want which model serves which concern to be a manifest decision with a +budget, not a prompt trick inside one conversation, so that token spend is structured by component. + +#### Acceptance Criteria + +1. Every `Task` in a `Run` binds exactly one `Model`; a Run may bind different Models to different tasks + (a cheap model for reads and triage, a stronger one for verification and judgement), and the reconciler + passes the bound Model's identity and prices to the task. +2. A task never chooses or switches its model at runtime; the Model it was bound to is recorded in its + `summary.json`. +3. `ousast plane status ` prints a token attribution table: one row per task and Model binding with + calls, input tokens, cache-hit tokens, output tokens and USD, a total row, and a per-Model subtotal; every + number comes from the provider's usage fields recorded in the tasks' `summary.json`, never estimated. A + `--units` flag breaks a task down per unit (file or candidate) from `units.jsonl`. The same table is + written to `attribution.json` in the run directory so runs can be compared. +4. A task that needs a second model for a sub-step (a judge over a verifier's output) is a separate task with + its own Model binding, never a second client inside one task. diff --git a/.kiro/specs/ai-service-plane/spec.json b/.kiro/specs/ai-service-plane/spec.json index 52f0f5a..affcf20 100644 --- a/.kiro/specs/ai-service-plane/spec.json +++ b/.kiro/specs/ai-service-plane/spec.json @@ -1,7 +1,7 @@ { "feature_name": "ai-service-plane", "created_at": "2026-09-29T15:40:00Z", - "updated_at": "2026-09-29T16:40:00Z", + "updated_at": "2026-09-29T17:55:00Z", "language": "en", "phase": "implementation", "approvals": { diff --git a/.kiro/specs/ai-service-plane/tasks.md b/.kiro/specs/ai-service-plane/tasks.md index 08d343b..6d29d70 100644 --- a/.kiro/specs/ai-service-plane/tasks.md +++ b/.kiro/specs/ai-service-plane/tasks.md @@ -1,9 +1,20 @@ # Implementation Plan -Order follows the dependency chain: manifests -> budget -> runner -> reconciler -> tasks -> increment run -> -measurement. Every task ships with its tests; a task is done when its tests pass in the host suite, ruff and +Order follows the dependency chain: ax bring-up (parallel with 1-2) -> manifests -> budget -> runner -> +ax-backed reconciler -> tasks -> increment run on ax -> measurement. Every task ships with its tests; a task is done when its tests pass in the host suite, ruff and mypy are clean, and the commit is gated on pytest's own exit code. Script-level pipeline work stays paused. +- [ ] 0. ax bring-up on this host + - Install `kind`, `kubectl`, the `ax` CLI and `ko` (user-local). `ops/ax/up.sh`: create the kind cluster and + install Agent Substrate with its `hack/create-kind-cluster.sh` and `hack/install-ate-kind.sh + --deploy-ate-system`, deploy the ax control plane with `ko` to the kind registry, apply one trivial Task + and wait for it to finish; `ops/ax/down.sh` tears it down. Record the measured idle footprint (RSS, CPU) + in `ops/ax/README.md` next to the 7 GB host constraint. + - `ousast plane doctor` (in `reconciler.py` or a sibling `doctor.py`): the four checks of Requirement 3.5. + - Stop rule: if the cluster does not come up within one working session, report the exact failure and stop; + the decision to work around it is the maintainer's. + - _Requirements: 3.5, 3.6_ + - [ ] 1. Manifest schemas and validation - Add `pyyaml` as a declared dependency in `pyproject.toml`. - `src/openultrasast/plane/manifests.py`: dataclasses and loaders for `Task`, `Workspace`, `Model` @@ -23,20 +34,29 @@ mypy are clean, and the commit is gated on pytest's own exit code. Script-level - [ ] 3. Runner contract - `src/openultrasast/plane/runner.py`: read `AX_TASK_YAML` / `AX_WORKSPACES_YAML`; materialise `git` entries at the pinned commit from the local case cache and write `spec.files`; execute `spec.command` - by importing `openultrasast.plane.tasks.`; optional `/healthz` `/readyz` server under - `AX_RUNNER_HTTP=1`; exit codes 0 done, 2 failed, 3 unfinished. - - `tests/test_plane_runner.py`: the env contract on a fixture workspace; readyz 503 then 200. - - _Requirements: 4.1, 4.2, 4.3_ + by importing `openultrasast.plane.tasks.`; `/healthz` `/readyz` server on port 80 (503 until + workspaces are ready; disabled only under `AX_RUNNER_HTTP=0` for unit tests); at exit, POST the output + directory as a tar stream to `OUSAST_ARTIFACT_URL` and report `failed` if delivery fails; exit codes + 0 done, 2 failed, 3 unfinished. `plane/Dockerfile.runner`: this package on Python 3.12, installed to + `/usr/local/bin/ax-task-runner`, loaded into kind by `ops/ax/up.sh`. + - `tests/test_plane_runner.py`: the env contract on a fixture workspace; readyz 503 then 200; delivery to a + local receiver; delivery failure yields `failed`. + - _Requirements: 4.1, 4.2, 4.3, 4.4_ -- [ ] 4. Local reconciler and CLI - - `src/openultrasast/plane/reconciler.py` (under 500 lines, no pipeline logic): `state.json`, statuses, - topological execution, `serialize` labels never overlapping, `--workers`, budgets and artifact paths - passed as task env, outcome read from `summary.json`, skip `done` on rerun, PID lock per run. - - `ousast plane run `, `ousast plane status `, `ousast plane workspaces ` - (generates Workspace manifests per case pin). - - `tests/test_plane_reconciler.py`: stub tasks in a DAG: order, non-overlap, skip-on-rerun, `unfinished` - and `failed` propagation, the lock, and the line-count assertion. - - _Requirements: 3.1, 3.2, 3.3, 3.4, 2.2_ +- [ ] 4. ax-backed reconciler and CLI + - `src/openultrasast/plane/reconciler.py` (under 500 lines, no pipeline logic, no local subprocess + executor): `state.json`, statuses, topological submission through the `ax` CLI (path injectable), status + polling, the artifact HTTP receiver, `serialize` labels never overlapping, `--workers`, budgets and + artifact env passed in the rendered Task manifest, outcome read from the delivered `summary.json`, skip + `done` on rerun, PID lock per run, `ax delete` of finished Tasks. + - `ousast plane run `, `ousast plane status ` (token attribution table per task and Model + from the tasks' `summary.json` usage; `--units` from `units.jsonl`; also written to `attribution.json`), + `ousast plane workspaces ` (generates Workspace manifests per case pin). + - `tests/test_plane_reconciler.py`: a fake `ax` CLI drives a DAG: order, non-overlap, skip-on-rerun, + `unfinished` and `failed` propagation, missing delivery, the lock, the attribution table, and the + line-count assertion. `tests/test_plane_ax_live.py` (marker `ax`, skipped without a live cluster): one + trivial Task end to end. + - _Requirements: 3.1, 3.2, 3.3, 3.4, 2.2, 7.1, 7.3_ - [ ] 5. `repo-facts` task - `src/openultrasast/plane/tasks/repo_facts.py`: product files, functions per file (per-language @@ -50,23 +70,24 @@ mypy are clean, and the commit is gated on pytest's own exit code. Script-level - `src/openultrasast/plane/tasks/verify.py`: the batched hunt moved from `verify_batched.py` (one hunt per file, up to 6 candidates, 6 steps, candidate-anchored sites) with known callers from `facts.json` in the prompt, tool turns and usage per hunt in `units.jsonl`, per-unit resume, `pass` from env, budget wrapper - from task 2; `summary.json` never `done` unless every unit finished. + from task 2; `summary.json` never `done` unless every unit finished, and records the bound Model and + the summed usage fields (prompt, cache-hit, output tokens, calls, usd). - `src/openultrasast/plane/tasks/agree.py`: agreement across pass outputs, per-candidate cost and turns, declared-site matching via `evaluate.matches_v2`. - `tests/test_plane_verify.py`: scripted client; prompt carries the callers; resume asks only the unfinished file; a scripted 402 yields `failed`; agree on two scripted passes. - - _Requirements: 5.3, 2.1, 2.2, 2.3_ + - _Requirements: 5.3, 2.1, 2.2, 2.3, 7.2_ - [ ] 7. Manifests for the increment - `plane/models/deepseek-flash.yaml` (annotated extension, prices in parameters); `plane/tasks/repo-facts.yaml`, `plane/tasks/verify.yaml`, `plane/tasks/agree.yaml`; `plane/runs/validation-46.yaml`: repo-facts -> verify pass a, verify pass b -> agree, budgets per task, inputs from the recorded candidate and triage artifacts. - - Workspaces generated for the 15 validation cases' vulnerable pins. + - Workspaces generated for the 15 validation cases' vulnerable pins; the runner image rebuilt and loaded. - _Requirements: 1.1, 1.2, 6.1_ - [ ] 8. Measurement of the first increment - - Run `validation-46` through the reconciler (needs DeepSeek credit); record cost per candidate, tool turns + - Run `validation-46` through the reconciler on ax (needs DeepSeek credit); record cost per candidate, tool turns per hunt, declared sites agreed, against `verifier-batched-check-2026-09-29.json`. - Run `benchmarks/dev/token_report.py` on the increment's development session. - Record `benchmarks/independent/plane-increment-1.json` with the gate result (met or not met) and commit; diff --git a/.kiro/specs/harnessx-removal/brief.md b/.kiro/specs/harnessx-removal/brief.md new file mode 100644 index 0000000..59acc6e --- /dev/null +++ b/.kiro/specs/harnessx-removal/brief.md @@ -0,0 +1,29 @@ +# Brief: remove HarnessX after the plane proves value + +Status: brief only, gated. Not started until `ai-service-plane` Requirement 6 records a met gate. + +## Inventory (read-only, 2026-09-29) + +- Code: `src/openultrasast/harness_ext.py` (97 lines) and `hunter_harness.py` (197) exist only for HarnessX; + guarded branches in `cli.py` (372-512), `fusion.py` (LLM panels, ~100 lines), `verify_judge.py` (judge path, + ~90), `stage_processors.py` (`host_under_harnessx`, ~40), `config.py` (`HarnessxConfig`). About 575 of + 30,243 lines (1.9%). Every path falls back deterministically when the extra is absent and records a + degradation (`harnessx_extra_unavailable`; fallbacks `run_hunter_pool`, `structural_verifier`, + `deterministic_panels`). `evolve`, `regress`, `benchmark`, `gate` and quick mode do not depend on it. +- Packaging: `pyproject.toml` extra `harnessx` pinned to a git commit; mypy override; `uv.lock` entry. +- Tests: 14 files touch it; dedicated `test_harness_ext.py`, `test_hunter_harness.py`, `test_cli_hx_dispatch.py`, + `test_verify_judge.py`; `tests/conftest.py` fixture `assert_cold_of_harnessx`. +- Docs: README sections "Configuring the HarnessX agentic plane" and "The HarnessX self-improving cycle"; + `docs/examples.md` section 6; `docs/threat-model.md` egress and spend caps. +- Specs: `harnessx-self-improving-rulesets` (approved, 43/43 done, Requirement 1 is direct adoption); + `learning-harness` (30/33); `three-stage-scan` framed as its follow-up; `openrouter-sast-harness` (no + spec.json); mentions in four more; steering `overview.md:175`. + +## Shape of the removal + +1. Replace the three guarded paths with plane tasks (hunter pool -> `verify`; judge -> `judge`; panels -> + `judge`/`agree`) so their fallbacks become the only path, then delete the two modules, the config block, + the extra, the override and the fixture. +2. Retire the four specs' HarnessX requirements with a dated note, not a rewrite; update README and docs. +3. Gate: the plane increment met (recall and cost), and the replaced features measured no worse on the + development pins before deletion.