engine: worker Deployment from the image (S3 queue with leases, dispatcher --executor queue, engine-image workflow, scoped RustFS policy); merge main

Implemented with Codex. Cluster deploy pending the k3s kubeconfig and the scoped account.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

ops/ax/README.md

@@ -17,6 +17,21 @@ in `~/.cache/ousast/ax-src/`. The cluster's context, registry and pool sizes com
 (`ops/k8s/profiles/kind-images.json`), which the templates are re-pinned from and which is committed with them.
 `up.sh` and `smoke-run.sh` run the checkout's code with `OUSAST_PYTHON` (default `.venv/bin/python`).
 
+The kind profile runs on a 7.7 GB server VM. Observability is off by default
+(`kind_observability = false` in `ops/k8s/profiles/kind.toml`). This saves the measured
+930 MiB used by `otel-system`, about 40% of the cluster's 2.3 GB. The smoke Run still
+completes with those deployments stopped. To enable the stack, run
+`OUSAST_KIND_OBSERVABILITY=1 ops/ax/up.sh` or set `kind_observability = true` in the profile.
+
+`up.sh` stages copied Substrate manifests in a temporary checkout view and installs through
+`hack/install-ate-kind.sh`. The local `ops/ax/substrate-kind-lean/` overlay removes the collector,
+Prometheus, Jaeger and their supporting resources before installation. This preserves the
+installer's secret, CRD and image setup. The upstream checkout is unchanged. The shared
+`ate-otel-config` disables trace, metric and log export and the OTEL SDK; it removes the old
+collector endpoint and metric timing keys. On an existing cluster, `up.sh` scales the three
+observability deployments to zero, replaces the ConfigMap data and rolls its consumers when
+the configuration changes. Opting in restores the upstream ConfigMap and observability manifests.
+
 ## How a Run executes
 
 A Run (`plane/runs/*.yaml`, kind `openultrasast.io/v1alpha1 Run`) is a DAG of steps; each step names an ax Task
@@ -191,7 +206,7 @@ configure the bucket.
 No deployment outside this host's kind cluster has been made. The step-by-step guidance, with what is implemented
 and what is planned, is `docs/deployment.md` (the site page "Deployment"). The Model and Workspace manifests under
 `plane/` move as they are and the Task templates need their image reference re-pinned to your registry; these parts
-assume this laptop and must change first (checked 2026-10-02):
+assume this server VM and must change first (checked 2026-10-02):
 
 | Assumption | Where | Needed in a real cluster |
 |---|---|---|

ops/ax/prepare-substrate.py

@@ -0,0 +1,43 @@
+#!/usr/bin/env python3
+"""Stage an installer checkout with lean kind manifests; never modify the source checkout.
+
+Usage: python ops/ax/prepare-substrate.py SOURCE EMPTY_DESTINATION
+The destination lives only for an install/render. Source/build inputs are symlinked;
+manifests are copied because the installer uses fixed paths (including the OTEL ConfigMap).
+"""
+
+from __future__ import annotations
+
+import shutil
+import sys
+from pathlib import Path
+
+import yaml
+
+
+def prepare(source: Path, destination: Path) -> None:
+    source = source.resolve()
+    destination.mkdir(parents=True, exist_ok=True)
+    if any(destination.iterdir()):
+        raise ValueError(f"staging directory must be empty: {destination}")
+    for entry in source.iterdir():
+        if entry.name != "manifests":
+            (destination / entry.name).symlink_to(entry, target_is_directory=entry.is_dir())
+    shutil.copytree(source / "manifests", destination / "manifests")
+    install = destination / "manifests/ate-install"
+    shutil.copytree(install / "kind", install / "kind-upstream")
+    overlay = Path(__file__).resolve().parent / "substrate-kind-lean"
+    shutil.copytree(overlay, install / "kind", dirs_exist_ok=True)
+    # The installer applies this file directly before it renders the bundle.
+    # Keep the strategic-merge directive only in the overlay's patch file.
+    config = yaml.safe_load((overlay / "ate-otel-config.yaml").read_text())
+    config["data"].pop("$patch")
+    (install / "kind/disabled-otel-config.yaml").write_text(yaml.safe_dump(config))
+    (install / "kind/ate-otel-config.yaml").rename(install / "kind/otel-config-patch.yaml")
+    (install / "kind/disabled-otel-config.yaml").rename(install / "kind/ate-otel-config.yaml")
+    kustomization = install / "kind/kustomization.yaml"
+    kustomization.write_text(kustomization.read_text().replace("path: ate-otel-config.yaml", "path: otel-config-patch.yaml"))
+
+
+if __name__ == "__main__":
+    prepare(Path(sys.argv[1]), Path(sys.argv[2]))

ops/ax/substrate-kind-lean/ate-otel-config.yaml

@@ -0,0 +1,11 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: ate-otel-config
+  namespace: ate-system
+data:
+  $patch: replace
+  OTEL_TRACES_EXPORTER: none
+  OTEL_METRICS_EXPORTER: none
+  OTEL_LOGS_EXPORTER: none
+  OTEL_SDK_DISABLED: "true"

ops/ax/substrate-kind-lean/kustomization.yaml

@@ -0,0 +1,44 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+# prepare-substrate.py stages the upstream kind overlay beside this directory.
+resources:
+  - ../kind-upstream
+patches:
+  - target:
+      namespace: otel-system
+    patch: |-
+      $patch: delete
+      apiVersion: v1
+      kind: ConfigMap
+      metadata:
+        name: unused
+  - target:
+      kind: Namespace
+      name: otel-system
+    patch: |-
+      $patch: delete
+      apiVersion: v1
+      kind: Namespace
+      metadata:
+        name: otel-system
+  - target:
+      kind: ClusterRole
+      name: ate-prometheus
+    patch: |-
+      $patch: delete
+      apiVersion: rbac.authorization.k8s.io/v1
+      kind: ClusterRole
+      metadata:
+        name: ate-prometheus
+  - target:
+      kind: RoleBinding
+      name: ate-prometheus
+      namespace: ate-system
+    patch: |-
+      $patch: delete
+      apiVersion: rbac.authorization.k8s.io/v1
+      kind: RoleBinding
+      metadata:
+        name: ate-prometheus
+        namespace: ate-system
+  - path: ate-otel-config.yaml

ops/ax/up.sh

@@ -15,6 +15,7 @@ profile() { PYTHONPATH="$REPO_ROOT/src" "$PY" -m openultrasast.plane.profile --p
 CTX="$(profile kube_context)"                       # kind names its context kind-<cluster>
 export KIND_CLUSTER_NAME="${CTX#kind-}"
 export KO_DOCKER_REPO="${KO_DOCKER_REPO:-$(profile registry)}"
+OBSERVABILITY="$(profile kind_observability)"
 IMAGES_FILE="$(profile images)"                     # the digest pins the reconciler and the templates read
 SRC="${OUSAST_AX_SRC:-$HOME/.cache/ousast/ax-src}"
 
@@ -37,9 +38,68 @@ else
 fi
 
 step "agent substrate (ate-system)"
+# The upstream installer has fixed manifest paths and also creates secrets/CRDs.
+# Give it a temporary checkout view so it never installs the observability resources.
+install_substrate() (
+  if [ "$OBSERVABILITY" = true ]; then
+    cd "$SRC/substrate"
+  else
+    # Symlinked build inputs must not make the staged view a different binary version.
+    export VERSION="${VERSION:-$(git -C "$SRC/substrate" describe --tags --always --dirty)}"
+    staged="$(mktemp -d)"
+    trap 'rm -rf "$staged"' EXIT
+    "$PY" "$REPO_ROOT/ops/ax/prepare-substrate.py" "$SRC/substrate" "$staged"
+    cd "$staged"
+  fi
+  KUBECTL_CONTEXT="$CTX" hack/install-ate-kind.sh --deploy-ate-system
+)
 if ready ate-system; then echo "substrate ready; kept"; else
-  (cd "$SRC/substrate" && hack/install-ate-kind.sh --deploy-ate-system)
+  install_substrate
 fi
+# Reconcile the choice even when the existing control plane is already ready.
+if [ "$OBSERVABILITY" = true ]; then
+  for manifest in otel-collector prometheus; do
+    kubectl --context "$CTX" apply -f "$SRC/substrate/manifests/ate-install/kind/$manifest.yaml"
+  done
+  otel_config="$SRC/substrate/manifests/ate-install/kind/ate-otel-config.yaml"
+else
+  otel_config="$REPO_ROOT/ops/ax/substrate-kind-lean/ate-otel-config.yaml"
+  for deployment in opentelemetry-collector prometheus jaeger; do
+    existing="$(kubectl --context "$CTX" -n otel-system get deployment "$deployment" --ignore-not-found -o name)"
+    if [ -n "$existing" ]; then
+      kubectl --context "$CTX" -n otel-system scale "$existing" --replicas=0
+    fi
+  done
+  echo "kind observability off: existing otel-system deployments scaled to 0; opt in with OUSAST_KIND_OBSERVABILITY=1 ops/ax/up.sh"
+fi
+# Replace data to remove stale exporter/endpoint keys in either direction. ConfigMap
+# envFrom is read at pod start; a content annotation triggers only changed rollouts.
+otel_data="$("$PY" - "$otel_config" <<'PYCONFIG'
+import json
+import sys
+import yaml
+config = yaml.safe_load(open(sys.argv[1]))
+config["data"].pop("$patch", None)
+print(json.dumps(config["data"], sort_keys=True))
+PYCONFIG
+)"
+kubectl --context "$CTX" -n ate-system patch configmap ate-otel-config --type=json \
+  -p "[{\"op\":\"replace\",\"path\":\"/data\",\"value\":$otel_data}]"
+otel_hash="$(printf '%s' "$otel_data" | sha256sum | cut -d ' ' -f1)"
+otel_workloads="$(kubectl --context "$CTX" -n ate-system get deployments,daemonsets -o json | "$PY" -c '
+import json, sys
+for item in json.load(sys.stdin)["items"]:
+    spec = item["spec"]["template"]["spec"]
+    containers = spec.get("containers", []) + spec.get("initContainers", [])
+    if any(env.get("configMapRef", {}).get("name") == "ate-otel-config"
+           for container in containers for env in container.get("envFrom", [])):
+        print(item["kind"].lower() + "/" + item["metadata"]["name"])
+')"
+for workload in $otel_workloads; do
+  kubectl --context "$CTX" -n ate-system patch "$workload" --type=merge \
+    -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"ousast.io/otel-config\":\"$otel_hash\"}}}}}"
+  kubectl --context "$CTX" -n ate-system rollout status "$workload" --timeout=300s
+done
 kubectl --context "$CTX" -n ate-system get pods
 
 step "egress gateway (atenet-egress, agentgateway variant: one prebuilt image, nothing built)"

ops/k8s/engine/README.md

@@ -0,0 +1,85 @@
+# Engine workers on k3s
+
+Build the new **Engine image** workflow on main (or dispatch it), then copy its
+`ghcr.io/<owner>/openultrasast@sha256:...` job-summary value. The old September
+image does not include boto3 or this service. The release workflow still builds
+version tags. The Dockerfile copies the worker and parser under `/app/benchmarks/learn`
+and installs `.[semantic,s3]`; no analyzer tarball is sent by the dispatcher.
+
+Use two schedulable nodes with at least 3 GiB available each. Required hostname
+anti-affinity places one worker on each node. Updates use Recreate so a third
+worker does not wait forever for a third node. Local scratch uses the container's
+filesystem, with no host mount or Kubernetes API access.
+
+Create a **new scoped account** using the policy in `docs/rustfs.md`. Export its
+`S3_ENDPOINT`, `S3_BUCKET=sast-memory`, `AWS_ACCESS_KEY_ID`, and
+`AWS_SECRET_ACCESS_KEY` in the coordinator shell. Do not use the memory-store
+account. The following commands are for the coordinator; they were not run by the
+implementation agent.
+
+```sh
+export ENGINE_IMAGE='ghcr.io/<owner>/openultrasast@sha256:<digest-from-workflow>'
+# From the repository root, edit the digest parameter without changing the template:
+export ENGINE_OVERLAY="$(mktemp -d /tmp/ousast-engine.XXXXXX)"
+cp ops/k8s/engine/{namespace,deployment,kustomization}.yaml "$ENGINE_OVERLAY/"
+python - <<'PY'
+import os
+from pathlib import Path
+image, digest = os.environ['ENGINE_IMAGE'].split('@', 1)
+p = Path(os.environ['ENGINE_OVERLAY']) / 'kustomization.yaml'
+p.write_text(p.read_text().replace('ghcr.io/OWNER/openultrasast', image)
+             .replace('sha256:REPLACE_WITH_ENGINE_IMAGE_DIGEST', digest))
+PY
+kubectl apply -f ops/k8s/engine/namespace.yaml
+kubectl -n ousast-engine create secret generic ousast-engine-s3 \
+  --from-literal=S3_ENDPOINT="$S3_ENDPOINT" \
+  --from-literal=S3_BUCKET="$S3_BUCKET" \
+  --from-literal=AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" \
+  --from-literal=AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY"
+kubectl apply -k "$ENGINE_OVERLAY"
+kubectl -n ousast-engine rollout status deployment/ousast-engine-worker
+kubectl -n ousast-engine get pods -o wide
+
+export PATH=/home/mc/Source/OpenUltraSAST/.venv/bin:$PATH
+export PYTHONPATH="$PWD/src"
+python benchmarks/learn/engine_trace.py --queue-status
+python benchmarks/learn/engine_trace.py --executor queue \
+  --image "$ENGINE_IMAGE" --parallel 2 --deadline 900 --queue-timeout 3600 \
+  --units plane/experiments/exp-005-graph-slice.units.jsonl \
+  --out "$HOME/ousast-results/plane/engine-trace-queue"
+```
+
+The dispatcher uses the existing local memory examples by default. Add
+`--examples /path/to/examples.jsonl` for a frozen example snapshot. Materialization
+uses locally cached source pins, just as the Docker executor does. `--executor
+k8s-jobs` preserves the old Job lane (`k8s` is its legacy alias); `queue` requires
+no kubectl access. Resume skips completed local units and retries failed or timed
+out units. `progress.json`, `STOP`, filtering, and `--summary` retain their existing
+meaning. Each invocation uses a new run ID, so an old remote result cannot satisfy
+a new retry. The dispatcher records worker and pod identity in each pin JSON.
+
+Tasks contain the pin, units, prepared questions, source object key, and deadlines.
+Results live at `engine-results/<run>/<pin-id>.json`; inputs are removed after
+publication. The Docker lane retains JSON and a log, not a result tar, so the queue
+also returns JSON. Queue timeout records timeout locally and removes pending;
+an already claimed task finishes remotely and cleans its input. Orphaned inputs
+from a failed submission can be removed under `engine-queue/inputs/` after checking
+that no pending task or claim references them.
+
+Claims use per-worker choosing markers followed by ordered tickets. A worker waits
+for choosing peers and every lower ticket before removing pending and starting.
+This requires strongly consistent GET/PUT/LIST from RustFS, including paginated
+LIST, and synchronized node clocks with skew below 30 seconds. Half of the 60-second lease margin is reserved for clock
+skew; the remaining execution budget is checked again after claiming.
+It does not require conditional writes, object tags, S3 Select, or bucket-admin
+permissions. A dead worker's task is recovered from its expired claim. Execution
+runs in a child process. Linux subreaping and a frozen descendant walk kill its
+entire tree, including Joern's separate sessions, at the task deadline before the
+lease expires. As with any unfenced lease, a node suspended beyond its lease must be
+terminated before it rejoins; the protocol cannot guarantee exclusivity against
+arbitrary process suspension or an inconsistent object store.
+
+SIGTERM stops new claims and lets the current task finish. The 900-second
+termination grace matches the maximum task deadline. Keep these values aligned
+if changing the deployment. `/healthz` serves both probes and becomes unready on
+shutdown. Errors omit SDK exception text to avoid exposing credentials.

ops/k8s/engine/deployment.yaml

@@ -0,0 +1,63 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: ousast-engine-worker
+spec:
+  replicas: 2
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app: ousast-engine-worker
+  template:
+    metadata:
+      labels:
+        app: ousast-engine-worker
+    spec:
+      automountServiceAccountToken: false
+      terminationGracePeriodSeconds: 900
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        runAsGroup: 1000
+      affinity:
+        podAntiAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+            - labelSelector:
+                matchLabels:
+                  app: ousast-engine-worker
+              topologyKey: kubernetes.io/hostname
+      containers:
+        - name: worker
+          image: engine-image
+          command: [python, /app/benchmarks/learn/engine_worker_service.py]
+          envFrom:
+            - secretRef:
+                name: ousast-engine-s3
+          env:
+            - name: ENGINE_IMAGE
+              value: engine-image
+          resources:
+            requests:
+              memory: 3Gi
+              cpu: '1'
+            limits:
+              memory: 3Gi
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: [ALL]
+          ports:
+            - name: health
+              containerPort: 8080
+          readinessProbe:
+            httpGet:
+              path: /healthz
+              port: health
+            periodSeconds: 5
+          livenessProbe:
+            httpGet:
+              path: /healthz
+              port: health
+            initialDelaySeconds: 15
+            periodSeconds: 10

ops/k8s/engine/kustomization.yaml

@@ -0,0 +1,22 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: ousast-engine
+resources:
+  - namespace.yaml
+  - deployment.yaml
+# Deliberately non-pullable until the coordinator supplies the workflow's digest.
+images:
+  - name: engine-image
+    newName: ghcr.io/OWNER/openultrasast
+    digest: sha256:REPLACE_WITH_ENGINE_IMAGE_DIGEST
+replacements:
+  - source:
+      kind: Deployment
+      name: ousast-engine-worker
+      fieldPath: spec.template.spec.containers.0.image
+    targets:
+      - select:
+          kind: Deployment
+          name: ousast-engine-worker
+        fieldPaths:
+          - spec.template.spec.containers.0.env.0.value

ops/k8s/engine/namespace.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: ousast-engine

ops/k8s/engine/secret.example.yaml

@@ -0,0 +1,9 @@
+# Reference only; excluded from kustomization. Create using the README command.
+apiVersion: v1
+kind: Secret
+metadata:
+  name: ousast-engine-s3
+  namespace: ousast-engine
+type: Opaque
+stringData: {}
+# Required keys: S3_ENDPOINT, S3_BUCKET, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY

ops/k8s/profiles/kind-images.json

@@ -1,3 +1,3 @@
 {
-  "runner": "localhost:5001/ousast-runner@sha256:d2a55293999cc34581ef1f02a1ffb0c2dd8fc0cbe001ac952df81600e868f019"
+  "runner": "localhost:5001/ousast-runner@sha256:784bc6afc869c679c4966d04d3f9653a8d379726b899809590cc83327f4038e3"
 }

ops/k8s/profiles/kind.toml

@@ -1,8 +1,9 @@
-# The kind profile: the plane on this laptop's single-node kind cluster, which ops/ax/up.sh creates and which is
+# The kind profile: the plane on this server VM's single-node kind cluster, which ops/ax/up.sh creates and which is
 # the development default (plane-on-kubernetes design section 1). Values only; a credential never goes here
 # (keys stay in the environment or .env). Every field is overridable by OUSAST_<FIELD> in the environment, a pool
 # by OUSAST_POOL_<NAME>_<FIELD>. This file may name the kind cluster; the code must not (tests/test_plane_literals.py).
 
+kind_observability = false      # opt in with OUSAST_KIND_OBSERVABILITY=1
 exec = "local"                   # the reconciler runs on this host against the cluster below
 kube_context = "kind-ousast"     # kind's context for KIND_CLUSTER_NAME=ousast
 registry = "localhost:5001"      # the kind-local registry up.sh creates (Substrate rewrites it for the node)