diff --git a/.kiro/specs/plane-on-kubernetes/design.md b/.kiro/specs/plane-on-kubernetes/design.md index d5e7277..f34fab5 100644 --- a/.kiro/specs/plane-on-kubernetes/design.md +++ b/.kiro/specs/plane-on-kubernetes/design.md @@ -254,6 +254,13 @@ containerd; `Dockerfile:1-7` already argues against it). a 3 Gi actor lands only on `engine-pool`. If Substrate does not place by resource fit, the fallback is a separate atespace with its own pool if Substrate scopes pools that way; failing that, the engine stays `alerts-engine` on a Docker host and the k3s engine becomes a plain Kubernetes Job outside ax, recorded as such. + **Probe outcome (task 1.5, 2026-10-02, `benchmarks/measurements/2026-10-02-k8s-01-profile/`): `placement: any`.** + With `ousast-pool` at one 1536Mi worker and a `probe-pool` of one 3Gi worker both ACTIVE, a `repo-facts` actor + requesting 2560Mi (limit 3Gi) was placed on the 1536Mi `ousast-pool` worker (`kubectl ate get workers`: ACTORS + 1/1000 there, 0/1000 on `probe-pool`), and so was the 256Mi actor. Substrate does not place by resource fit, so + group 5 takes the fallback: the engine pool must be the only pool its actors can land on. Before 5.2, check whether + Substrate scopes pools by atespace (an `engine` atespace with `engine-pool` as its only pool); if it does not, the + k3s engine runs as a plain Kubernetes Job outside ax and `alerts-engine` stays the kind path. - **`alerts-engine` stays** for development on a host with Docker (kind profile), its help text and `docs/plane.md` row marked "development only; the plane task is `engine`" (Req 4.3). Generated Runs choose `engine` when the profile's `images` file has an engine image, else leave `alerts` for `alerts-engine`. diff --git a/.kiro/specs/plane-on-kubernetes/tasks.md b/.kiro/specs/plane-on-kubernetes/tasks.md index fbec09c..321fd58 100644 --- a/.kiro/specs/plane-on-kubernetes/tasks.md +++ b/.kiro/specs/plane-on-kubernetes/tasks.md @@ -42,7 +42,7 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi `runner-image`; `generate.repin_templates` accepts that file (runner key only until task 4.3). - Evidence: `ops/ax/up.sh` reruns idempotently on kind; `ops/ax/smoke-run.sh` passes. - _Requirements: 1.1, 2.4_ -- [ ] 1.5 Placement probe (open question 2, decided before the engine work) +- [x] 1.5 Placement probe (open question 2, decided before the engine work) -- result: `placement: any` - `ops/ax/probe-placement.sh`: applies a second WorkerPool `probe-pool` (1 replica, 3 Gi) from the design's engine template, then one model-free `repo-facts` Task with `resources.requests.memory: 2560Mi`, and records with `kubectl ate get workers` which worker took the actor; then the same Task with 256Mi. Removed after. diff --git a/benchmarks/measurements/2026-10-02-k8s-01-profile/probe/ousast-probe-big.yaml b/benchmarks/measurements/2026-10-02-k8s-01-profile/probe/ousast-probe-big.yaml new file mode 100644 index 0000000..3693ca4 --- /dev/null +++ b/benchmarks/measurements/2026-10-02-k8s-01-profile/probe/ousast-probe-big.yaml @@ -0,0 +1,16 @@ +apiVersion: ax.io/v1alpha1 +kind: Workspace +metadata: {name: ousast-probe-big-ws, atespace: default} +spec: + files: + - {path: "hello.py", content: "def hello():\n return greet()\n"} +--- +apiVersion: ax.io/v1alpha1 +kind: Task +metadata: {name: ousast-probe-big, atespace: default} +spec: + image: "localhost:5001/ousast-runner@sha256:412db0a1504af8f1b23a395af8d1dcbd152c1bb6d16594f2b733067163a31884" + command: ["repo-facts"] + env: [{name: OUSAST_OUTPUT_DIR, value: "/workspace/.ousast-out/probe"}] + resources: {requests: {cpu: "250m", memory: "2560Mi"}, limits: {cpu: "1", memory: "3Gi"}} + workspaces: [{name: ousast-probe-big-ws, path: "/workspace"}] diff --git a/benchmarks/measurements/2026-10-02-k8s-01-profile/probe/ousast-probe-small.yaml b/benchmarks/measurements/2026-10-02-k8s-01-profile/probe/ousast-probe-small.yaml new file mode 100644 index 0000000..3b86c99 --- /dev/null +++ b/benchmarks/measurements/2026-10-02-k8s-01-profile/probe/ousast-probe-small.yaml @@ -0,0 +1,16 @@ +apiVersion: ax.io/v1alpha1 +kind: Workspace +metadata: {name: ousast-probe-small-ws, atespace: default} +spec: + files: + - {path: "hello.py", content: "def hello():\n return greet()\n"} +--- +apiVersion: ax.io/v1alpha1 +kind: Task +metadata: {name: ousast-probe-small, atespace: default} +spec: + image: "localhost:5001/ousast-runner@sha256:412db0a1504af8f1b23a395af8d1dcbd152c1bb6d16594f2b733067163a31884" + command: ["repo-facts"] + env: [{name: OUSAST_OUTPUT_DIR, value: "/workspace/.ousast-out/probe"}] + resources: {requests: {cpu: "250m", memory: "256Mi"}, limits: {cpu: "1", memory: "512Mi"}} + workspaces: [{name: ousast-probe-small-ws, path: "/workspace"}] diff --git a/benchmarks/measurements/2026-10-02-k8s-01-profile/probe/probe-pool.yaml b/benchmarks/measurements/2026-10-02-k8s-01-profile/probe/probe-pool.yaml new file mode 100644 index 0000000..157f6cb --- /dev/null +++ b/benchmarks/measurements/2026-10-02-k8s-01-profile/probe/probe-pool.yaml @@ -0,0 +1,26 @@ +# Warm gVisor workers for ax tasks (Agent Substrate WorkerPool), rendered by ops/ax/up.sh (and by +# ops/ax/probe-placement.sh for the placement probe) from the profile's pool: POOL_NAME, POOL_REPLICAS, POOL_CPU, +# POOL_MEMORY (ops/k8s/profiles/.toml, [pools.default]) and WORKER_IMAGE (ko:// for a build, or a +# digest-pinned reference of a built worker image). ax's per-task ActorTemplates set no workerSelector, so every +# pool of the matching sandbox class is eligible. An actor occupies a whole worker. Replaced by the renderer of +# plane-on-kubernetes task 6.1 (ops/k8s/base/workerpool-*.yaml.tmpl). +apiVersion: ate.dev/v1alpha1 +kind: WorkerPool +metadata: + name: probe-pool + namespace: ax-system + labels: + workload: ousast +spec: + replicas: 1 + workerImage: localhost:5001/ateom-gvisor-715889664656de67e44382a8d6ab981d@sha256:220f1cf13ea485b73a2e4270fa544901a18ae826d8f22bd54d42761944a41c5c + template: + nodeSelector: + ate.dev/substrate-version: "4f9c293" + resources: + limits: + cpu: "2" + memory: 3Gi + requests: + cpu: 250m + memory: 3Gi diff --git a/benchmarks/measurements/2026-10-02-k8s-01-profile/probe/workers-before.json b/benchmarks/measurements/2026-10-02-k8s-01-profile/probe/workers-before.json new file mode 100644 index 0000000..79d9783 --- /dev/null +++ b/benchmarks/measurements/2026-10-02-k8s-01-profile/probe/workers-before.json @@ -0,0 +1,78 @@ +{ + "workers": [ + { + "ip": "10.244.0.22", + "labels": { + "workload": "ousast" + }, + "metadata": { + "createTime": "2026-09-29T16:32:08.594663703Z", + "name": "769f7d8b-142a-4ed5-b896-08d6a04bedb6", + "uid": "152cf7c0-3f4d-4634-88c5-9fe501bb403e", + "updateTime": "2026-10-02T05:24:59.069304031Z", + "version": "1758" + }, + "nodeName": "ousast-control-plane", + "sandboxClass": "gvisor", + "status": { + "capacity": { + "actors": 1000, + "resources": { + "limits": [ + { + "name": "cpu", + "quantity": "1" + }, + { + "name": "memory", + "quantity": "1536Mi" + } + ] + } + }, + "state": "WORKER_STATE_ACTIVE" + }, + "workerNamespace": "ax-system", + "workerPod": "ousast-pool-6656b55f5b-9wbrw", + "workerPodUid": "769f7d8b-142a-4ed5-b896-08d6a04bedb6", + "workerPool": "ousast-pool" + }, + { + "ip": "10.244.0.28", + "labels": { + "workload": "ousast" + }, + "metadata": { + "createTime": "2026-10-02T18:29:30.486026311Z", + "name": "1407a421-7dcf-4ad7-b999-70a4624bf890", + "uid": "2d496d7a-4f32-459d-9b92-256a2c5b4800", + "updateTime": "2026-10-02T18:29:30.551668391Z", + "version": "2" + }, + "nodeName": "ousast-control-plane", + "sandboxClass": "gvisor", + "status": { + "capacity": { + "actors": 1000, + "resources": { + "limits": [ + { + "name": "cpu", + "quantity": "2" + }, + { + "name": "memory", + "quantity": "3Gi" + } + ] + } + }, + "state": "WORKER_STATE_ACTIVE" + }, + "workerNamespace": "ax-system", + "workerPod": "probe-pool-5dffd74bdc-dqlww", + "workerPodUid": "1407a421-7dcf-4ad7-b999-70a4624bf890", + "workerPool": "probe-pool" + } + ] +} diff --git a/benchmarks/measurements/2026-10-02-k8s-01-profile/probe/workers-ousast-probe-big.json b/benchmarks/measurements/2026-10-02-k8s-01-profile/probe/workers-ousast-probe-big.json new file mode 100644 index 0000000..cd90d21 --- /dev/null +++ b/benchmarks/measurements/2026-10-02-k8s-01-profile/probe/workers-ousast-probe-big.json @@ -0,0 +1,81 @@ +{ + "workers": [ + { + "ip": "10.244.0.22", + "labels": { + "workload": "ousast" + }, + "metadata": { + "createTime": "2026-09-29T16:32:08.594663703Z", + "name": "769f7d8b-142a-4ed5-b896-08d6a04bedb6", + "uid": "152cf7c0-3f4d-4634-88c5-9fe501bb403e", + "updateTime": "2026-10-02T18:29:45.281958347Z", + "version": "1761" + }, + "nodeName": "ousast-control-plane", + "sandboxClass": "gvisor", + "status": { + "allocated": { + "actors": 1 + }, + "capacity": { + "actors": 1000, + "resources": { + "limits": [ + { + "name": "cpu", + "quantity": "1" + }, + { + "name": "memory", + "quantity": "1536Mi" + } + ] + } + }, + "state": "WORKER_STATE_ACTIVE" + }, + "workerNamespace": "ax-system", + "workerPod": "ousast-pool-6656b55f5b-9wbrw", + "workerPodUid": "769f7d8b-142a-4ed5-b896-08d6a04bedb6", + "workerPool": "ousast-pool" + }, + { + "ip": "10.244.0.28", + "labels": { + "workload": "ousast" + }, + "metadata": { + "createTime": "2026-10-02T18:29:30.486026311Z", + "name": "1407a421-7dcf-4ad7-b999-70a4624bf890", + "uid": "2d496d7a-4f32-459d-9b92-256a2c5b4800", + "updateTime": "2026-10-02T18:29:30.551668391Z", + "version": "2" + }, + "nodeName": "ousast-control-plane", + "sandboxClass": "gvisor", + "status": { + "capacity": { + "actors": 1000, + "resources": { + "limits": [ + { + "name": "cpu", + "quantity": "2" + }, + { + "name": "memory", + "quantity": "3Gi" + } + ] + } + }, + "state": "WORKER_STATE_ACTIVE" + }, + "workerNamespace": "ax-system", + "workerPod": "probe-pool-5dffd74bdc-dqlww", + "workerPodUid": "1407a421-7dcf-4ad7-b999-70a4624bf890", + "workerPool": "probe-pool" + } + ] +} diff --git a/benchmarks/measurements/2026-10-02-k8s-01-profile/probe/workers-ousast-probe-small.json b/benchmarks/measurements/2026-10-02-k8s-01-profile/probe/workers-ousast-probe-small.json new file mode 100644 index 0000000..566d053 --- /dev/null +++ b/benchmarks/measurements/2026-10-02-k8s-01-profile/probe/workers-ousast-probe-small.json @@ -0,0 +1,81 @@ +{ + "workers": [ + { + "ip": "10.244.0.22", + "labels": { + "workload": "ousast" + }, + "metadata": { + "createTime": "2026-09-29T16:32:08.594663703Z", + "name": "769f7d8b-142a-4ed5-b896-08d6a04bedb6", + "uid": "152cf7c0-3f4d-4634-88c5-9fe501bb403e", + "updateTime": "2026-10-02T18:29:53.001457323Z", + "version": "1763" + }, + "nodeName": "ousast-control-plane", + "sandboxClass": "gvisor", + "status": { + "allocated": { + "actors": 1 + }, + "capacity": { + "actors": 1000, + "resources": { + "limits": [ + { + "name": "cpu", + "quantity": "1" + }, + { + "name": "memory", + "quantity": "1536Mi" + } + ] + } + }, + "state": "WORKER_STATE_ACTIVE" + }, + "workerNamespace": "ax-system", + "workerPod": "ousast-pool-6656b55f5b-9wbrw", + "workerPodUid": "769f7d8b-142a-4ed5-b896-08d6a04bedb6", + "workerPool": "ousast-pool" + }, + { + "ip": "10.244.0.28", + "labels": { + "workload": "ousast" + }, + "metadata": { + "createTime": "2026-10-02T18:29:30.486026311Z", + "name": "1407a421-7dcf-4ad7-b999-70a4624bf890", + "uid": "2d496d7a-4f32-459d-9b92-256a2c5b4800", + "updateTime": "2026-10-02T18:30:04.825058843Z", + "version": "4" + }, + "nodeName": "ousast-control-plane", + "sandboxClass": "gvisor", + "status": { + "capacity": { + "actors": 1000, + "resources": { + "limits": [ + { + "name": "cpu", + "quantity": "2" + }, + { + "name": "memory", + "quantity": "3Gi" + } + ] + } + }, + "state": "WORKER_STATE_ACTIVE" + }, + "workerNamespace": "ax-system", + "workerPod": "probe-pool-5dffd74bdc-dqlww", + "workerPodUid": "1407a421-7dcf-4ad7-b999-70a4624bf890", + "workerPool": "probe-pool" + } + ] +} diff --git a/ops/ax/probe-placement.sh b/ops/ax/probe-placement.sh new file mode 100755 index 0000000..660a7ef --- /dev/null +++ b/ops/ax/probe-placement.sh @@ -0,0 +1,81 @@ +#!/usr/bin/env bash +# Placement probe (plane-on-kubernetes task 1.5, design open question 2): does Agent Substrate place an actor on a +# worker that fits its memory request, or on any free worker of the sandbox class? +# +# Applies a second WorkerPool `probe-pool` (1 replica, 2 CPU / 3Gi) from ops/ax/workerpool.yaml.tmpl with the worker +# image the existing pool already built (no ko build), scales the default pool to one worker so the 7 GB host holds +# both, then runs one model-free repo-facts Task asking 2560Mi (limit 3Gi) and one asking 256Mi, recording after +# each resume which worker holds the actor (`kubectl ate get workers -o json`). Everything it created is removed +# on exit and the default pool is scaled back. Usage: ops/ax/probe-placement.sh OUT_DIR (the JSON evidence lands there). +set -euo pipefail +export PATH="$HOME/go/bin:$HOME/.local/bin:$PATH" +OUT="${1:?usage: probe-placement.sh OUT_DIR}" +mkdir -p "$OUT" +REPO_ROOT="$(cd "$(dirname "$0")/../.." && pwd)" +PY="${OUSAST_PYTHON:-$REPO_ROOT/.venv/bin/python}" +profile() { PYTHONPATH="$REPO_ROOT/src" "$PY" -m openultrasast.plane.profile --profile "${OUSAST_PLANE_PROFILE:-kind}" --print "$@"; } +CTX="$(profile kube_context)" +RUNNER_IMAGE="$(profile images.runner)" +DEFAULT_REPLICAS="$(profile pools.default.replicas)" +K() { kubectl --context "$CTX" -n ax-system "$@"; } +log() { printf '%s %s\n' "$(date +%H:%M:%S)" "$*" | tee -a "$OUT/probe.log"; } + +cleanup() { + rc=$? + log "cleanup (exit $rc): delete the probe tasks and probe-pool, scale ousast-pool back to $DEFAULT_REPLICAS" + for t in ousast-probe-big ousast-probe-small; do ax delete task "$t" >/dev/null 2>&1 || true; done + K delete workerpool probe-pool --ignore-not-found >/dev/null 2>&1 || true + K patch workerpool ousast-pool --type merge -p "{\"spec\":{\"replicas\":$DEFAULT_REPLICAS}}" >/dev/null 2>&1 || true + echo "probe exit $rc" +} +trap cleanup EXIT + +workers() { kubectl ate --context "$CTX" get workers -o json > "$OUT/$1.json"; kubectl ate --context "$CTX" get workers | tee -a "$OUT/probe.log"; } + +export POOL_NAME=probe-pool POOL_REPLICAS=1 POOL_CPU=2 POOL_MEMORY=3Gi +WORKER_IMAGE="$(K get workerpool ousast-pool -o jsonpath='{.spec.workerImage}')" # already built and pulled by the node +SUBSTRATE_VERSION="$(kubectl --context "$CTX" get nodes -o jsonpath='{.items[0].metadata.labels.ate\.dev/substrate-version}')" +export WORKER_IMAGE SUBSTRATE_VERSION +log "default pool to 1 replica (room for a 3Gi worker on this host)" +K patch workerpool ousast-pool --type merge -p '{"spec":{"replicas":1}}' +for i in $(seq 1 60); do [ "$(K get pods -l ate.dev/worker-pool=ousast-pool --no-headers 2>/dev/null | wc -l)" = 1 ] && break; sleep 2; done +log "apply probe-pool: $POOL_REPLICAS x $POOL_CPU CPU / $POOL_MEMORY, image $WORKER_IMAGE" +envsubst < "$(dirname "$0")/workerpool.yaml.tmpl" | tee "$OUT/probe-pool.yaml" | K apply -f - +K wait --for=condition=Ready pod -l ate.dev/worker-pool=probe-pool --timeout=600s +workers workers-before + +task() { # name, request memory, limit memory + cat > "$OUT/$1.yaml" </dev/null 2>&1 || true + log "apply $1 (requests $2, limits $3)" + ax apply -f "$OUT/$1.yaml" + for i in 1 2 3 4 5 6 7 8 9; do + if ax resume task "$1" 2>&1 | tee -a "$OUT/probe.log"; then break; fi # the first resume may wait for the golden snapshot + sleep 20 + done + sleep 15 + ax get task "$1" | tee "$OUT/$1.status.yaml" | grep -E "phase|message" || true + workers "workers-$1" + ax delete task "$1" >/dev/null 2>&1 || true + for i in $(seq 1 30); do kubectl ate --context "$CTX" get workers | grep -q " 1/1000 " || break; sleep 2; done +} +task ousast-probe-big 2560Mi 3Gi +task ousast-probe-small 256Mi 512Mi +log "done; evidence under $OUT"