# Implementation Plan > **Status 2026-10-04.** Groups 1-2 are done on kind (records `benchmarks/measurements/2026-10-02-k8s-01-profile/` > and `2026-10-02-k8s-02-store-delivery/`). Task 2.6 is paused. The production target changed. The operator runs a > kube-ax cluster (k3s, Agent Substrate v0.3.0, AX v0.3.1, 2 nodes). There we may only submit AX Tasks: no k8s > object creation, no secrets, no access to `ate-system`. The engine now runs there as AX Tasks through > `benchmarks/learn/engine_trace_ax.py`. The engine-task image (`plane/Dockerfile.engine-task`) is built FROM > `ghcr.io/norandom/ax-task-runner:v0.3.1`. Tasks read and write through presigned HTTPS links, are named > `ousast-engine-*`, and run a 1.7 GB heap with `ActiveProcessorCount=2` and one resident Joern session per pin. A > pin whose sandbox dies is retried once under a new name, then handed to the VM lane (2.5 GB heap). Egress is > operator-managed: HTTPS 443 to public hosts is open; LAN and cluster-internal addresses are blocked. Archive > names (`.tar`, `.gz`, `.zip`, `.bin`) failed through the store host's Cloudflare cache until the operator added a > cache-bypass rule; presigned objects are staged as `.dat` as a safeguard. Groups 3-8 are **to re-plan** against > that cluster under roadmap G4 (`.kiro/steering/roadmap.md`, section of 2026-10-04). Their text below is kept > as written. G4 (detection on push for any repository: `ousast plane scan --base --head`, engine pins on kube-ax, > model calls on the VM or CI) is the new target for group 7, sized by what G1-G3 keep. One task group per increment of `design.md` section 7, in that order; a later group never starts before the earlier group's kind proof is recorded. Every task ships with its tests; a task is done when its tests pass in the host suite, ruff and mypy are clean, and nothing in `src/`, `plane/`, `ops/` (minus `ops/ax/up.sh` and `ops/k8s/profiles/`) or `docs/` names `kind-ousast`, `localhost:5001`, `172.19.` or a port-forward address (task 1.3 makes that a test). Nothing is deleted before its replacement passes its proof. Each group ends with its kind proof and the record it writes under `benchmarks/measurements/-k8s-NN-/record.json` (commit, images, the instrument checks, the numbers). Steps that call a model are marked **paid: needs a budget go-ahead** with a ceiling; steps that need a hand on a cluster that does not exist yet are marked **manual**. - [x] 1. Profile and literals (design section 1; open-question checks 2, 3, 7) - [x] 1.1 `PlaneProfile` - `src/openultrasast/plane/profile.py`: frozen dataclass with the fields of design section 1; `load_profile()` reads `ops/k8s/profiles/.toml` (a name or a path; default `kind`), then applies `OUSAST_` overrides; rejects an unknown key, a secret-looking key (`*_KEY`, `*_SECRET`), an `exec` outside `local|remote`, an `images` file without digest pins. `ops/k8s/profiles/kind.toml` and `k3s.toml` committed with the design's values; `ops/k8s/profiles/kind-images.json` from `~/.cache/ousast/ax-src/runner-image`. - `tests/test_plane_profile.py`: defaults; file then env precedence; each rejection names the field; both committed profiles load; no field holds a value matching a credential pattern. - _Requirements: 1.1_ - [x] 1.2 Literals out of `doctor.py`, `egress.py`, `router.py`, `generate.py` - `doctor(profile)`: context from `profile.kube_context`; the registry check resolves the `images` file's runner reference (`docker manifest inspect` for a remote registry, `/v2/_catalog` for a local one); `open_store` verifies `profile.memory`; prints every configured address (Req 1.2). `Egress(context=...)`, `open_router(context=...)` and `receiver_cluster_ip(context)` take the profile's context; `generate.ATESPACE` becomes `profile.atespace`. `cli.py`: `ousast plane doctor|run|workspaces|harvest` accept `--profile`. - `tests/test_plane_reconciler.py`, `test_plane_router.py`, `test_plane_egress.py`: the fakes receive the profile's context, never a built one. - Evidence: `grep -rn "kind-" src/openultrasast/plane` matches only comments citing the kind profile. - _Requirements: 1.1, 1.2_ - [x] 1.3 Host-literal scan test - `tests/test_plane_literals.py`: pattern built at runtime (as `tests/test_removed_plane_references.py` does) over `src/`, `plane/`, `ops/` minus `ops/ax/up.sh` and `ops/k8s/profiles/`, and `docs/`, for `kind-ousast`, `localhost:5001`, `172.19.`, `127.0.0.1:` and `port-forward` outside `router.py` and lines tagged `ax-tunnel`; fails listing `path:line`. - Evidence: the test passes on this commit and fails when a literal is reinserted (shown once in the record). - _Requirements: 1.3_ - [x] 1.4 `up.sh` by profile - `ops/ax/up.sh`: `KO_DOCKER_REPO` and the context from `kind.toml` (read with `python -m openultrasast.plane.profile --print registry|kube_context`); writes `kind-images.json` instead of `runner-image`; `generate.repin_templates` accepts that file (runner key only until task 4.3). - Evidence: `ops/ax/up.sh` reruns idempotently on kind; `ops/ax/smoke-run.sh` passes. - _Requirements: 1.1, 2.4_ - [x] 1.5 Placement probe (open question 2, decided before the engine work) -- result: `placement: any` - `ops/ax/probe-placement.sh`: applies a second WorkerPool `probe-pool` (1 replica, 3 Gi) from the design's engine template, then one model-free `repo-facts` Task with `resources.requests.memory: 2560Mi`, and records with `kubectl ate get workers` which worker took the actor; then the same Task with 256Mi. Removed after. - Evidence: `record.json` says whether Substrate places by resource fit (`placement: fit|any`). `any` selects the design's fallback for group 5 and is written into `design.md` section 4 before group 5 starts. - _Requirements: 4.1_ - [x] 1.6 Reconciler bound (open question 7) -- 489 lines after 1.2; projected 514 during group 2: the bound moves to 550 in 2.4 - Record `wc -l src/openultrasast/plane/reconciler.py` (477 at 17ac7ad) and the lines task 2.4 may add (receiver wiring removed: `-12`, `Delivery` calls: `+8` estimated). If the estimate crosses 500, the bound in `tests/test_plane_reconciler.py:803` moves to 550 in task 2.4 with the reason in the assertion message. - Evidence: the numbers in `record.json`. - _Requirements: 2.1_ - [x] 1.7 gvisor tarball reachability (open question 3) **manual** -- documented as planned in `docs/deployment.md` B.1; the live check waits for k3s - `docs/deployment.md` (B.1, planned): the k3s node must fetch `gs://gvisor/releases/nightly/2026-09-02//gvisor.tar.zstd` (`substrate/manifests/ate-install/sandboxconfig-gvisor.yaml`) or a mirrored copy named in a copied SandboxConfig; the check is `kubectl -n ate-system logs` of the worker pod's atelet fetch line, run by hand on the day k3s exists. Labelled planned until then. - _Requirements: 7.2_ - [x] 1.8 Kind proof and record 01 - `ousast plane doctor --profile kind` passes every line; `ousast plane doctor --profile k3s` fails only on the unreachable context (the expected line); `test_plane_literals` 0 offenders; the probe of 1.5; the bound of 1.6. - `benchmarks/measurements/-k8s-01-profile/record.json`. - _Requirements: 1.1, 1.2, 1.3, 7.1_ - [ ] 2. Store delivery, receiver-less egress (design section 3) - [x] 2.1 Presigned URLs and the store run directory - `src/openultrasast/plane/delivery.py`: `Delivery(store, run)` mints presigned PUT for `runs///output.tar` and GET for `runs///` (boto3 `generate_presigned_url` on `S3Client.sdk`, expiry `OUSAST_TASK_TIMEOUT + 600`); `delivered(task)` HEADs the tar; `collect(task)` downloads, extracts under the run dir with the member checks `Receiver.accept` has, re-puts declared `outputs[]` as single objects, writes `done.json`; `put_input(name, path)` for inputs no task produces; `state.json` mirrored to `runs//state.json` and seeded from it on a rerun. - `tests/test_plane_delivery.py`: against a local S3-shaped fake (PUT/GET/HEAD by key); URLs never appear in logs; expiry; member rejection; declared-only re-put; state seeding. - _Requirements: 3.1, 2.1_ - [x] 2.2 Runner: `delivery` in the start body - `src/openultrasast/plane/runner.py`: `StartGate.offer` validates an optional `delivery {put, inputs}` (https, hostname, no echo); `_deliver_output` PUTs the tar to `delivery.put` (same retry and `DeliveryError` semantics); `fetch_inputs` GETs from `delivery.inputs[name]`; `OUSAST_ARTIFACT_URL` and `OUSAST_ARTIFACT_DIAL` stay honoured until task 2.6 so both paths run in the same build. - `tests/test_plane_runner.py`: accepted body with `delivery`; 400 on an `http` or IP URL; PUT to a local server; inputs fetched from presigned-shaped URLs; the marker and retry-only boot unchanged. - _Requirements: 3.1_ - [x] 2.3 Egress without the receiver - `src/openultrasast/plane/egress.py`: `policy_for(task, workspaces, model, store_host)` emits one `tlsPassthrough` rule on 443 over the sorted union of store host, Git hosts, Model hosts; no `http` rule; an IP store host raises naming `S3_ENDPOINT`. - `tests/test_plane_egress.py`: for every committed template under `plane/tasks/` the rendered policy is the expected set; the IP refusal. - _Requirements: 3.2_ - [x] 2.4 Reconciler on `Delivery` - `src/openultrasast/plane/reconciler.py`: `run()` opens the store first (no fallback), builds `Delivery`, passes `delivery` to `router.starter`/`start_task` (body extended in `router.py`), `_await` on `Delivery.delivered`, outcome from the collected `summary.json`; the receiver thread is kept behind `OUSAST_DELIVERY=receiver` for this group only. Line bound per task 1.6. - `tests/test_plane_reconciler.py`: the ax fake run completes through the delivery fake; a rerun from an empty results root resumes from the store's `state.json`. - _Requirements: 2.1, 3.1_ - [x] 2.5 Kind proof and record 02 - `ops/ax/smoke-run.sh` with `OUSAST_DELIVERY=store`: `facts.json` and `summary.json` arrive through the store; the actor's policy read back has one TLS rule naming the store host; `ousast plane status ax-e2e-smoke` prints the table. Instrument: the tar's byte size from the store equals the extracted files' total. - `benchmarks/measurements/-k8s-02-store-delivery/record.json`: bytes, PUT-to-poll seconds, rules. - _Requirements: 3.1, 3.2, 7.1_ - [ ] 2.6 Remove the receiver (last; after 2.5 is recorded) -- paused 2026-10-04 - Delete `Receiver`, `_Handler`, `receiver_address`, `receiver_cluster_ip`, `RECEIVER_HOST`, `RECEIVER_PORT` from `egress.py`; `OUSAST_ARTIFACT_URL/DIAL/HOST/PORT` from `runner.py` and `reconciler.py`; the `OUSAST_DELIVERY` switch; `ops/ax/receiver-service.yaml.tmpl` and the `up.sh` receiver step; the EndpointSlice from kind (`kubectl delete`). Docs: `docs/plane.md` sequence diagram, `docs/deployment.md`, `ops/ax/README.md`. - Tests using `OUSAST_ARTIFACT_HOST` move to the delivery fake; `test_plane_literals` covers `172.19.`. - Evidence: `ops/ax/smoke-run.sh` passes again on the build without the receiver; recorded in 02's record as `receiver_removed: `. - _Requirements: 3.1, 1.3_ - [ ] 3. `--exec remote` from a CI-shaped principal (design section 2; open-question check 1) -- to re-plan (kube-ax, roadmap G4) - [ ] 3.1 Execution mode and RBAC manifests - `cli.py`: `ousast plane run --exec local|remote` (default `profile.exec`); `reconciler.Ax` passes the profile's context to `ax` (its context flag, read from `ax --help` and pinned in a test); `remote` refuses a `file://` store. `ops/k8s/base/rbac-ci.yaml`: SA `ousast-ci` in `ax-system`; Roles and RoleBindings in `ax-system` and `ate-system` with `pods` get/list, `services` get, `pods/portforward` create, nothing else. - `tests/test_plane_rbac.py`: fake `kubectl`, `ax`, `kubectl-ate` record argv over a full fake Run; the verbs implied (`port-forward` -> `pods/portforward` create + `pods` get + `services` get) equal the rendered Role's rules exactly. - _Requirements: 2.1, 2.2_ - [ ] 3.2 Substrate's authorization of the SA token (open question 1) - `ops/ax/probe-ci-token.sh`: `kubectl create token ousast-ci --duration=2h` into a kubeconfig; `kubectl ate --token-file - get workers` and `create egress-policy` on a throwaway actor with that kubeconfig. If refused, read `substrate/docs/api-guide.md` for the grant (atespace role or token audience) and add the manifest to `ops/k8s/base/rbac-ci.yaml`; if no grant exists for a non-admin token, record it and the design's remote mode uses the maintainer's admin kubeconfig for the proof, with the gap listed in the runbook. - Evidence: `record.json` names the grant used (`grant: ` or `admin-kubeconfig`). - _Requirements: 2.1_ - [ ] 3.3 CI-shaped container - `ops/k8s/ci-shell.Dockerfile`: `python:3.12-slim` + this package + `kubectl`, `kubectl-ate`, `ax` (built from the checkouts); no `.env`; runs with `-e KUBECONFIG=/run/kc -e DEEPSEEK_API_KEY -e S3_*` only. `.github/workflows/plane-remote.yml` (dispatch only, no schedule): the same image running `ousast plane run --exec remote --profile k3s` with repository secrets; disabled until k3s exists (labelled in the file). - _Requirements: 2.2, 2.3_ - [ ] 3.4 Kind proof and record 03 **paid: needs a budget go-ahead, ceiling 0.50 USD** - From the container of 3.3 against kind (`kind.toml` with `exec = "remote"`): the smoke Run (model-free), then one `verify` pass over one case of `plane/runs/validation-46.yaml` (budget `{usd: 0.5, calls: 60}`) so the credential path is exercised remotely. Kube audit log (kind apiserver `--audit-policy` of the SA's verbs) lists only the Role's verbs. - `benchmarks/measurements/-k8s-03-remote-exec/record.json`: audit verbs vs Role, grant of 3.2, run status, usd spent. - _Requirements: 2.1, 2.2, 2.3, 2.4, 7.1_ - [ ] 4. Images on GHCR (design section 8) -- to re-plan (kube-ax, roadmap G4) - [ ] 4.1 `images.yml` - `.github/workflows/images.yml` on tags `v*`: `docker/build-push-action` for `plane/Dockerfile.runner` -> `ghcr.io/norandom/ousast-runner:` and the root `Dockerfile` -> `ghcr.io/norandom/ousast-engine:`; digests as job outputs and as the release asset `images.json` (`{"runner": "...@sha256:...", "engine": ...}`); `GITHUB_TOKEN` with `packages: write`. - Evidence: a dry run on a `v*-rc` tag publishes both and the asset (the engine build is ~2.8 GB: record the build minutes). - _Requirements: 1.1, 4.2_ - [ ] 4.2 Engine image under the runner contract - Root `Dockerfile`: the `ax-task-runner` shim and `/workspace` as `plane/Dockerfile.runner:25-29`; the `s3` extra is not needed in either image (the runner PUTs by URL). - `tests/test_plane_images.py`: both Dockerfiles declare the shim and entrypoint (text check). - _Requirements: 4.2_ - [ ] 4.3 `repin` for two images - `generate.repin(plane, images)`: `engine.yaml` (added in 5.1; until then absent is fine) takes `images["engine"]`, every other template `images["runner"]`; `ousast plane repin images.json`; `--runner-image FILE` keeps the single-image file working. `ops/k8s/profiles/k3s.toml` names `images.json` downloaded from the release. - `tests/test_plane_increment.py`: repin with a two-key file; a template with two image lines is refused. - _Requirements: 1.1, 4.2_ - [ ] 4.4 Kind proof and record 04 - Pools' pod template takes `imagePullSecret` from the profile (`ops/k8s/base/workerpool-*.tmpl`, task 6.1 brings the full template; here the field only); kind pulls `ghcr.io/norandom/ousast-runner@sha256:...` with `ghcr-pull` if the package is private; the smoke Run passes on the GHCR image. - `benchmarks/measurements/-k8s-04-ghcr-images/record.json`: image sizes, pull seconds, visibility. - _Requirements: 1.1, 4.2, 7.1_ - [ ] 5. The engine as a plane task (design section 4) -- to re-plan (kube-ax, roadmap G4) - [ ] 5.1 `cpg/dump.py` and `tasks/engine.py` - Lift `benchmarks/push/finding_dump.py`'s scan into `src/openultrasast/cpg/dump.py` (the script becomes a thin caller; `ops/README.md:78` command unchanged). `src/openultrasast/plane/tasks/engine.py`: env `OUSAST_WORKSPACE_DIR`, `OUSAST_FIXED_DIR`, `OUSAST_INPUT_CASE`, pins, `OUSAST_FAMILIES`, `OUSAST_DEADLINE`; per pin the quick scan (`tasks.alerts.scan`) and the engine in-process; rows via `engine_alerts.engine_rows`, the instrument checks via `read_record`'s rules; `summary.json` as `engine_alerts.produce` writes it with `host: false`. `plane/tasks/engine.yaml`: `command: ["engine"]`, requests 3Gi/1, limits 4Gi/2, image from `images["engine"]`. `generate.TEMPLATES` gains `engine`; `--loop` emits `engine` instead of `alerts` for a case needing the engine when the images file has an engine key. - `tests/test_plane_engine_task.py`: a fixture record -> rows and summary equal to `produce`'s shape; the unread-input and zero-question failures; the generator's choice. - _Requirements: 4.1_ - [ ] 5.2 `engine-pool` - `ops/k8s/base/workerpool-engine.yaml.tmpl` rendered by `plane/k8s.py` (task 6.1 generalises; here the engine values): kind replicas 1, 4Gi/2; if 1.5 recorded `any`, apply the design's fallback and write it here. - Evidence: `kubectl -n ax-system get workerpool engine-pool` ready on kind with `ousast-pool` at 1 replica. - _Requirements: 4.1, 5.1_ - [ ] 5.3 `alerts-engine` marked development-only - `cli.py` help, `docs/plane.md` task table, `ops/README.md`: "development only (needs Docker on the host); the plane task is `engine`". No code removed. - _Requirements: 4.3_ - [ ] 5.4 Kind proof and record 05 - The froxlor case of `validation-46` as an `engine` task on kind (model-free, deadline 1800 s per pin): rows equal `benchmarks/measurements/2026-09-30-php-engine-alerts/` (6/6 per pin), placement on `engine-pool` shown by `kubectl ate get workers`, peak memory from `kubectl top`. - `benchmarks/measurements/-k8s-05-engine-task/record.json`: rows diff, seconds per pin, peak memory, placement, image size and pull seconds. - _Requirements: 4.1, 4.2, 7.1_ - [ ] 6. Pools, autoscaling, node checklist (design section 5) -- to re-plan (kube-ax, roadmap G4) - [ ] 6.1 `plane/k8s.py` and the manifests - `src/openultrasast/plane/k8s.py`: `render(profile, out)` writes `workerpool-default.yaml`, `workerpool-engine.yaml`, `ax-server-snapshots.yaml` (patch with `profile.snapshots_bucket`), `rbac-ci.yaml`; for `k3s` also `imagepullsecret.yaml`, `hpa-default.yaml`, `hpa-engine.yaml`, `prometheus-adapter.yaml` (copied from `substrate/demos/autoscaled-workerpool`, namespace `ax-system`, metric `ate_workerpool_workers{state=at_capacity}`, `averageValue 0.7`). `ousast plane manifests --profile P --out DIR`; `ops/k8s/render.sh`. `ops/ax/workerpool.yaml.tmpl` deleted once `up.sh` uses the renderer. - `tests/test_plane_k8s.py`: both profiles render; kind has no HPA and no pull secret; k3s has both; engine pool limits >= `engine.yaml` limits; every image digest-pinned; no host literal in the output. - _Requirements: 5.1_ - [ ] 6.2 Node checklist and footprint - `docs/deployment.md`: the gVisor correction (no RuntimeClass; `ateom-gvisor` runs `runsc` from the SandboxConfig tarball), PodSecurity for the worker pod's securityContext (recorded from kind), local-path for `/var/lib/ate`, the measured footprint per task type (`verify`, `repo-facts`, `engine` from 05) and the queueing consequence of one actor per worker with `--workers`. - _Requirements: 5.2, 7.2_ - [ ] 6.3 Kind proof and record 06 (HPA: render only; live is k3s-only, **manual** later) - `render.sh kind` applied on kind replaces the hand template with identical pools (`kubectl diff` empty but for labels); worker pod `securityContext` captured; `render.sh k3s` validates with `kubectl apply --dry-run=client` against kind's API (CRDs present). - `benchmarks/measurements/-k8s-06-pools/record.json`: footprint table, securityContext, dry-run result. - _Requirements: 5.1, 5.2, 7.1_ - [ ] 7. `ousast plane scan` (design section 6) -- to re-plan (kube-ax, roadmap G4) - [ ] 7.1 Workspaces and candidates from URL + commits - `generate.scan_cases(url, base, head, cache, *, all=False)`: shallow clone into the case cache, two pinned Workspaces, head-side hunks via `fix_ranges(side="new")`, functions via `repo_facts.enclosing` over `product_files`, quick alerts via `tasks.alerts.scan` on the changed files, `candidates.json` and `functions.json` as store inputs (`Delivery.put_input`); `--all` takes every product function. - `tests/test_plane_scan.py`: a two-commit fixture repository yields exactly the changed functions; a rename and a deleted function are handled; `--all` count. - _Requirements: 6.1_ - [ ] 7.2 Run composition and the command - `src/openultrasast/plane/scan.py` + `cli.py`: `ousast plane scan URL --base C --head C [--all] [--ceiling USD] [--exec] [--wait S | --no-wait]`: per set `facts -> va, vb -> agree -> vc -> final -> features -> remember`, `engine` when a changed file's language needs it, `roles` once per repository when memory has no `roles` rows for `repo_key(url)`; budgets from `verify_budget`; Run annotated `openultrasast.io/population: scan`, `split: `; after the Run, `memory.ingest`, then the attribution table and the plane's agreement (`final`), the decision engine's verdict only when adopted. `--no-wait` prints the Run name and exits 0. - `tests/test_plane_scan.py`: the rendered Run for the fixture; budgets sum under the ceiling; `roles` included only when the store lacks rows; the report's verdict line. - _Requirements: 6.1, 6.2_ - [ ] 7.3 Hook opt-in - `ops/pre-push`: with `OUSAST_PLANE_SCAN=1`, after the blocking check, `ousast plane scan --base --head --exec remote --no-wait` and the Run name in the report; the hook's exit code never depends on it. `ops/README.md` and `docs/deployment.md`: the latency split. - `tests/test_push_hook.py`: the wrapper calls the scan only with the variable set and ignores its failure. - _Requirements: 6.3_ - [ ] 7.4 Kind proof and record 07 **paid: needs a budget go-ahead, ceiling 3.00 USD** - A PHP pair from the development corpus (pmpro's CVE commit and its parent, never a v2 or v3 case) scanned on kind: candidates, the Run through `final`, `engine` for the PHP files, rows under `repos/____/.jsonl` on the store, the report; then `ousast plane status`. - `benchmarks/measurements/-k8s-07-plane-scan/record.json`: candidates, usd, calls, agreement, rows, seconds from submit to verdict, whether the known CVE site is among the agreed. - _Requirements: 6.1, 6.2, 6.3, 7.1_ - [ ] 8. Runbook (design section 7, Req 7) -- to re-plan (kube-ax, roadmap G4) - [ ] 8.1 `docs/deployment.md` as the runbook - Sections: prerequisites for k3s (Substrate, ax with the snapshots patch, GHCR and the pull secret, buckets, the CI SA and its grant from 3.2, the node checklist of 6.2), the profile switch, `--exec remote` from a laptop and from CI, and the checklist whose every item names its requirement and its record 01-07; items without a recorded run (HPA live, 1.7's fetch) labelled planned. `ops/ax/README.md` and `ops/README.md` point to it; the "moving to a separate cluster" tables are replaced by the checklist. - _Requirements: 7.2_ - [ ] 8.2 Spec closure - `design.md` open questions updated with the outcomes of 1.5, 1.6, 3.2 (1.7 stays open until k3s); the `kind-tooling` memory and `.kiro/steering/overview.md` current-state line updated; `/kiro-spec-status`. - Evidence: every record 01-07 linked from the runbook's checklist. - _Requirements: 7.1, 7.2_