plane(1.5): placement probe -- Substrate places by any free worker, not by fit

ops/ax/probe-placement.sh scales ousast-pool to one worker, applies probe-pool
(1 x 2 CPU / 3Gi, the already-built ateom-gvisor image, no ko build), runs one
model-free repo-facts Task requesting 2560Mi (limit 3Gi) and one requesting
256Mi, records `kubectl ate get workers -o json` after each resume, and removes
everything it created. Live on kind: both actors landed on the 1536Mi
ousast-pool worker (status.allocated.actors=1 there, 0 on probe-pool) ->
placement: any. design.md section 4 records the outcome and the fallback group
5 must take (an engine atespace with engine-pool as its only pool if Substrate
scopes pools by atespace; else a plain Kubernetes Job on k3s). Evidence under
benchmarks/measurements/2026-10-02-k8s-01-profile/probe/.

Gates (tree identical to 1.4 in src/ and tests/; ops/ gains the probe script,
which the literal scan covers): pytest 2010 passed, 42 skipped, exit 0; ruff
check 0; ruff format 0; mypy 0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo

.kiro/specs/plane-on-kubernetes/design.md

@@ -254,6 +254,13 @@ containerd; `Dockerfile:1-7` already argues against it).
   a 3 Gi actor lands only on `engine-pool`. If Substrate does not place by resource fit, the fallback is a
   separate atespace with its own pool if Substrate scopes pools that way; failing that, the engine stays
   `alerts-engine` on a Docker host and the k3s engine becomes a plain Kubernetes Job outside ax, recorded as such.
+  **Probe outcome (task 1.5, 2026-10-02, `benchmarks/measurements/2026-10-02-k8s-01-profile/`): `placement: any`.**
+  With `ousast-pool` at one 1536Mi worker and a `probe-pool` of one 3Gi worker both ACTIVE, a `repo-facts` actor
+  requesting 2560Mi (limit 3Gi) was placed on the 1536Mi `ousast-pool` worker (`kubectl ate get workers`: ACTORS
+  1/1000 there, 0/1000 on `probe-pool`), and so was the 256Mi actor. Substrate does not place by resource fit, so
+  group 5 takes the fallback: the engine pool must be the only pool its actors can land on. Before 5.2, check whether
+  Substrate scopes pools by atespace (an `engine` atespace with `engine-pool` as its only pool); if it does not, the
+  k3s engine runs as a plain Kubernetes Job outside ax and `alerts-engine` stays the kind path.
 - **`alerts-engine` stays** for development on a host with Docker (kind profile), its help text and
   `docs/plane.md` row marked "development only; the plane task is `engine`" (Req 4.3). Generated Runs choose
   `engine` when the profile's `images` file has an engine image, else leave `alerts` for `alerts-engine`.

.kiro/specs/plane-on-kubernetes/tasks.md

@@ -42,7 +42,7 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
     `runner-image`; `generate.repin_templates` accepts that file (runner key only until task 4.3).
   - Evidence: `ops/ax/up.sh` reruns idempotently on kind; `ops/ax/smoke-run.sh` passes.
   - _Requirements: 1.1, 2.4_
-- [ ] 1.5 Placement probe (open question 2, decided before the engine work)
+- [x] 1.5 Placement probe (open question 2, decided before the engine work) -- result: `placement: any`
   - `ops/ax/probe-placement.sh`: applies a second WorkerPool `probe-pool` (1 replica, 3 Gi) from the design's
     engine template, then one model-free `repo-facts` Task with `resources.requests.memory: 2560Mi`, and records
     with `kubectl ate get workers` which worker took the actor; then the same Task with 256Mi. Removed after.

benchmarks/measurements/2026-10-02-k8s-01-profile/probe/ousast-probe-big.yaml

@@ -0,0 +1,16 @@
+apiVersion: ax.io/v1alpha1
+kind: Workspace
+metadata: {name: ousast-probe-big-ws, atespace: default}
+spec:
+  files:
+    - {path: "hello.py", content: "def hello():\n    return greet()\n"}
+---
+apiVersion: ax.io/v1alpha1
+kind: Task
+metadata: {name: ousast-probe-big, atespace: default}
+spec:
+  image: "localhost:5001/ousast-runner@sha256:412db0a1504af8f1b23a395af8d1dcbd152c1bb6d16594f2b733067163a31884"
+  command: ["repo-facts"]
+  env: [{name: OUSAST_OUTPUT_DIR, value: "/workspace/.ousast-out/probe"}]
+  resources: {requests: {cpu: "250m", memory: "2560Mi"}, limits: {cpu: "1", memory: "3Gi"}}
+  workspaces: [{name: ousast-probe-big-ws, path: "/workspace"}]

benchmarks/measurements/2026-10-02-k8s-01-profile/probe/ousast-probe-small.yaml

@@ -0,0 +1,16 @@
+apiVersion: ax.io/v1alpha1
+kind: Workspace
+metadata: {name: ousast-probe-small-ws, atespace: default}
+spec:
+  files:
+    - {path: "hello.py", content: "def hello():\n    return greet()\n"}
+---
+apiVersion: ax.io/v1alpha1
+kind: Task
+metadata: {name: ousast-probe-small, atespace: default}
+spec:
+  image: "localhost:5001/ousast-runner@sha256:412db0a1504af8f1b23a395af8d1dcbd152c1bb6d16594f2b733067163a31884"
+  command: ["repo-facts"]
+  env: [{name: OUSAST_OUTPUT_DIR, value: "/workspace/.ousast-out/probe"}]
+  resources: {requests: {cpu: "250m", memory: "256Mi"}, limits: {cpu: "1", memory: "512Mi"}}
+  workspaces: [{name: ousast-probe-small-ws, path: "/workspace"}]

benchmarks/measurements/2026-10-02-k8s-01-profile/probe/probe-pool.yaml

@@ -0,0 +1,26 @@
+# Warm gVisor workers for ax tasks (Agent Substrate WorkerPool), rendered by ops/ax/up.sh (and by
+# ops/ax/probe-placement.sh for the placement probe) from the profile's pool: POOL_NAME, POOL_REPLICAS, POOL_CPU,
+# POOL_MEMORY (ops/k8s/profiles/<profile>.toml, [pools.default]) and WORKER_IMAGE (ko:// for a build, or a
+# digest-pinned reference of a built worker image). ax's per-task ActorTemplates set no workerSelector, so every
+# pool of the matching sandbox class is eligible. An actor occupies a whole worker. Replaced by the renderer of
+# plane-on-kubernetes task 6.1 (ops/k8s/base/workerpool-*.yaml.tmpl).
+apiVersion: ate.dev/v1alpha1
+kind: WorkerPool
+metadata:
+  name: probe-pool
+  namespace: ax-system
+  labels:
+    workload: ousast
+spec:
+  replicas: 1
+  workerImage: localhost:5001/ateom-gvisor-715889664656de67e44382a8d6ab981d@sha256:220f1cf13ea485b73a2e4270fa544901a18ae826d8f22bd54d42761944a41c5c
+  template:
+    nodeSelector:
+      ate.dev/substrate-version: "4f9c293"
+    resources:
+      limits:
+        cpu: "2"
+        memory: 3Gi
+      requests:
+        cpu: 250m
+        memory: 3Gi

benchmarks/measurements/2026-10-02-k8s-01-profile/probe/workers-before.json

@@ -0,0 +1,78 @@
+{
+  "workers": [
+    {
+      "ip": "10.244.0.22",
+      "labels": {
+        "workload": "ousast"
+      },
+      "metadata": {
+        "createTime": "2026-09-29T16:32:08.594663703Z",
+        "name": "769f7d8b-142a-4ed5-b896-08d6a04bedb6",
+        "uid": "152cf7c0-3f4d-4634-88c5-9fe501bb403e",
+        "updateTime": "2026-10-02T05:24:59.069304031Z",
+        "version": "1758"
+      },
+      "nodeName": "ousast-control-plane",
+      "sandboxClass": "gvisor",
+      "status": {
+        "capacity": {
+          "actors": 1000,
+          "resources": {
+            "limits": [
+              {
+                "name": "cpu",
+                "quantity": "1"
+              },
+              {
+                "name": "memory",
+                "quantity": "1536Mi"
+              }
+            ]
+          }
+        },
+        "state": "WORKER_STATE_ACTIVE"
+      },
+      "workerNamespace": "ax-system",
+      "workerPod": "ousast-pool-6656b55f5b-9wbrw",
+      "workerPodUid": "769f7d8b-142a-4ed5-b896-08d6a04bedb6",
+      "workerPool": "ousast-pool"
+    },
+    {
+      "ip": "10.244.0.28",
+      "labels": {
+        "workload": "ousast"
+      },
+      "metadata": {
+        "createTime": "2026-10-02T18:29:30.486026311Z",
+        "name": "1407a421-7dcf-4ad7-b999-70a4624bf890",
+        "uid": "2d496d7a-4f32-459d-9b92-256a2c5b4800",
+        "updateTime": "2026-10-02T18:29:30.551668391Z",
+        "version": "2"
+      },
+      "nodeName": "ousast-control-plane",
+      "sandboxClass": "gvisor",
+      "status": {
+        "capacity": {
+          "actors": 1000,
+          "resources": {
+            "limits": [
+              {
+                "name": "cpu",
+                "quantity": "2"
+              },
+              {
+                "name": "memory",
+                "quantity": "3Gi"
+              }
+            ]
+          }
+        },
+        "state": "WORKER_STATE_ACTIVE"
+      },
+      "workerNamespace": "ax-system",
+      "workerPod": "probe-pool-5dffd74bdc-dqlww",
+      "workerPodUid": "1407a421-7dcf-4ad7-b999-70a4624bf890",
+      "workerPool": "probe-pool"
+    }
+  ]
+}

benchmarks/measurements/2026-10-02-k8s-01-profile/probe/workers-ousast-probe-big.json

@@ -0,0 +1,81 @@
+{
+  "workers": [
+    {
+      "ip": "10.244.0.22",
+      "labels": {
+        "workload": "ousast"
+      },
+      "metadata": {
+        "createTime": "2026-09-29T16:32:08.594663703Z",
+        "name": "769f7d8b-142a-4ed5-b896-08d6a04bedb6",
+        "uid": "152cf7c0-3f4d-4634-88c5-9fe501bb403e",
+        "updateTime": "2026-10-02T18:29:45.281958347Z",
+        "version": "1761"
+      },
+      "nodeName": "ousast-control-plane",
+      "sandboxClass": "gvisor",
+      "status": {
+        "allocated": {
+          "actors": 1
+        },
+        "capacity": {
+          "actors": 1000,
+          "resources": {
+            "limits": [
+              {
+                "name": "cpu",
+                "quantity": "1"
+              },
+              {
+                "name": "memory",
+                "quantity": "1536Mi"
+              }
+            ]
+          }
+        },
+        "state": "WORKER_STATE_ACTIVE"
+      },
+      "workerNamespace": "ax-system",
+      "workerPod": "ousast-pool-6656b55f5b-9wbrw",
+      "workerPodUid": "769f7d8b-142a-4ed5-b896-08d6a04bedb6",
+      "workerPool": "ousast-pool"
+    },
+    {
+      "ip": "10.244.0.28",
+      "labels": {
+        "workload": "ousast"
+      },
+      "metadata": {
+        "createTime": "2026-10-02T18:29:30.486026311Z",
+        "name": "1407a421-7dcf-4ad7-b999-70a4624bf890",
+        "uid": "2d496d7a-4f32-459d-9b92-256a2c5b4800",
+        "updateTime": "2026-10-02T18:29:30.551668391Z",
+        "version": "2"
+      },
+      "nodeName": "ousast-control-plane",
+      "sandboxClass": "gvisor",
+      "status": {
+        "capacity": {
+          "actors": 1000,
+          "resources": {
+            "limits": [
+              {
+                "name": "cpu",
+                "quantity": "2"
+              },
+              {
+                "name": "memory",
+                "quantity": "3Gi"
+              }
+            ]
+          }
+        },
+        "state": "WORKER_STATE_ACTIVE"
+      },
+      "workerNamespace": "ax-system",
+      "workerPod": "probe-pool-5dffd74bdc-dqlww",
+      "workerPodUid": "1407a421-7dcf-4ad7-b999-70a4624bf890",
+      "workerPool": "probe-pool"
+    }
+  ]
+}

benchmarks/measurements/2026-10-02-k8s-01-profile/probe/workers-ousast-probe-small.json

@@ -0,0 +1,81 @@
+{
+  "workers": [
+    {
+      "ip": "10.244.0.22",
+      "labels": {
+        "workload": "ousast"
+      },
+      "metadata": {
+        "createTime": "2026-09-29T16:32:08.594663703Z",
+        "name": "769f7d8b-142a-4ed5-b896-08d6a04bedb6",
+        "uid": "152cf7c0-3f4d-4634-88c5-9fe501bb403e",
+        "updateTime": "2026-10-02T18:29:53.001457323Z",
+        "version": "1763"
+      },
+      "nodeName": "ousast-control-plane",
+      "sandboxClass": "gvisor",
+      "status": {
+        "allocated": {
+          "actors": 1
+        },
+        "capacity": {
+          "actors": 1000,
+          "resources": {
+            "limits": [
+              {
+                "name": "cpu",
+                "quantity": "1"
+              },
+              {
+                "name": "memory",
+                "quantity": "1536Mi"
+              }
+            ]
+          }
+        },
+        "state": "WORKER_STATE_ACTIVE"
+      },
+      "workerNamespace": "ax-system",
+      "workerPod": "ousast-pool-6656b55f5b-9wbrw",
+      "workerPodUid": "769f7d8b-142a-4ed5-b896-08d6a04bedb6",
+      "workerPool": "ousast-pool"
+    },
+    {
+      "ip": "10.244.0.28",
+      "labels": {
+        "workload": "ousast"
+      },
+      "metadata": {
+        "createTime": "2026-10-02T18:29:30.486026311Z",
+        "name": "1407a421-7dcf-4ad7-b999-70a4624bf890",
+        "uid": "2d496d7a-4f32-459d-9b92-256a2c5b4800",
+        "updateTime": "2026-10-02T18:30:04.825058843Z",
+        "version": "4"
+      },
+      "nodeName": "ousast-control-plane",
+      "sandboxClass": "gvisor",
+      "status": {
+        "capacity": {
+          "actors": 1000,
+          "resources": {
+            "limits": [
+              {
+                "name": "cpu",
+                "quantity": "2"
+              },
+              {
+                "name": "memory",
+                "quantity": "3Gi"
+              }
+            ]
+          }
+        },
+        "state": "WORKER_STATE_ACTIVE"
+      },
+      "workerNamespace": "ax-system",
+      "workerPod": "probe-pool-5dffd74bdc-dqlww",
+      "workerPodUid": "1407a421-7dcf-4ad7-b999-70a4624bf890",
+      "workerPool": "probe-pool"
+    }
+  ]
+}

ops/ax/probe-placement.sh

@@ -0,0 +1,81 @@
+#!/usr/bin/env bash
+# Placement probe (plane-on-kubernetes task 1.5, design open question 2): does Agent Substrate place an actor on a
+# worker that fits its memory request, or on any free worker of the sandbox class?
+#
+# Applies a second WorkerPool `probe-pool` (1 replica, 2 CPU / 3Gi) from ops/ax/workerpool.yaml.tmpl with the worker
+# image the existing pool already built (no ko build), scales the default pool to one worker so the 7 GB host holds
+# both, then runs one model-free repo-facts Task asking 2560Mi (limit 3Gi) and one asking 256Mi, recording after
+# each resume which worker holds the actor (`kubectl ate get workers -o json`). Everything it created is removed
+# on exit and the default pool is scaled back. Usage: ops/ax/probe-placement.sh OUT_DIR  (the JSON evidence lands there).
+set -euo pipefail
+export PATH="$HOME/go/bin:$HOME/.local/bin:$PATH"
+OUT="${1:?usage: probe-placement.sh OUT_DIR}"
+mkdir -p "$OUT"
+REPO_ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
+PY="${OUSAST_PYTHON:-$REPO_ROOT/.venv/bin/python}"
+profile() { PYTHONPATH="$REPO_ROOT/src" "$PY" -m openultrasast.plane.profile --profile "${OUSAST_PLANE_PROFILE:-kind}" --print "$@"; }
+CTX="$(profile kube_context)"
+RUNNER_IMAGE="$(profile images.runner)"
+DEFAULT_REPLICAS="$(profile pools.default.replicas)"
+K() { kubectl --context "$CTX" -n ax-system "$@"; }
+log() { printf '%s %s\n' "$(date +%H:%M:%S)" "$*" | tee -a "$OUT/probe.log"; }
+
+cleanup() {
+  rc=$?
+  log "cleanup (exit $rc): delete the probe tasks and probe-pool, scale ousast-pool back to $DEFAULT_REPLICAS"
+  for t in ousast-probe-big ousast-probe-small; do ax delete task "$t" >/dev/null 2>&1 || true; done
+  K delete workerpool probe-pool --ignore-not-found >/dev/null 2>&1 || true
+  K patch workerpool ousast-pool --type merge -p "{\"spec\":{\"replicas\":$DEFAULT_REPLICAS}}" >/dev/null 2>&1 || true
+  echo "probe exit $rc"
+}
+trap cleanup EXIT
+
+workers() { kubectl ate --context "$CTX" get workers -o json > "$OUT/$1.json"; kubectl ate --context "$CTX" get workers | tee -a "$OUT/probe.log"; }
+
+export POOL_NAME=probe-pool POOL_REPLICAS=1 POOL_CPU=2 POOL_MEMORY=3Gi
+WORKER_IMAGE="$(K get workerpool ousast-pool -o jsonpath='{.spec.workerImage}')"   # already built and pulled by the node
+SUBSTRATE_VERSION="$(kubectl --context "$CTX" get nodes -o jsonpath='{.items[0].metadata.labels.ate\.dev/substrate-version}')"
+export WORKER_IMAGE SUBSTRATE_VERSION
+log "default pool to 1 replica (room for a 3Gi worker on this host)"
+K patch workerpool ousast-pool --type merge -p '{"spec":{"replicas":1}}'
+for i in $(seq 1 60); do [ "$(K get pods -l ate.dev/worker-pool=ousast-pool --no-headers 2>/dev/null | wc -l)" = 1 ] && break; sleep 2; done
+log "apply probe-pool: $POOL_REPLICAS x $POOL_CPU CPU / $POOL_MEMORY, image $WORKER_IMAGE"
+envsubst < "$(dirname "$0")/workerpool.yaml.tmpl" | tee "$OUT/probe-pool.yaml" | K apply -f -
+K wait --for=condition=Ready pod -l ate.dev/worker-pool=probe-pool --timeout=600s
+workers workers-before
+
+task() {  # name, request memory, limit memory
+  cat > "$OUT/$1.yaml" <<EOF
+apiVersion: ax.io/v1alpha1
+kind: Workspace
+metadata: {name: $1-ws, atespace: default}
+spec:
+  files:
+    - {path: "hello.py", content: "def hello():\n    return greet()\n"}
+---
+apiVersion: ax.io/v1alpha1
+kind: Task
+metadata: {name: $1, atespace: default}
+spec:
+  image: "$RUNNER_IMAGE"
+  command: ["repo-facts"]
+  env: [{name: OUSAST_OUTPUT_DIR, value: "/workspace/.ousast-out/probe"}]
+  resources: {requests: {cpu: "250m", memory: "$2"}, limits: {cpu: "1", memory: "$3"}}
+  workspaces: [{name: $1-ws, path: "/workspace"}]
+EOF
+  ax delete task "$1" >/dev/null 2>&1 || true
+  log "apply $1 (requests $2, limits $3)"
+  ax apply -f "$OUT/$1.yaml"
+  for i in 1 2 3 4 5 6 7 8 9; do
+    if ax resume task "$1" 2>&1 | tee -a "$OUT/probe.log"; then break; fi   # the first resume may wait for the golden snapshot
+    sleep 20
+  done
+  sleep 15
+  ax get task "$1" | tee "$OUT/$1.status.yaml" | grep -E "phase|message" || true
+  workers "workers-$1"
+  ax delete task "$1" >/dev/null 2>&1 || true
+  for i in $(seq 1 30); do kubectl ate --context "$CTX" get workers | grep -q " 1/1000 " || break; sleep 2; done
+}
+task ousast-probe-big 2560Mi 3Gi
+task ousast-probe-small 256Mi 512Mi
+log "done; evidence under $OUT"