engine: worker Deployment from the image (S3 queue with leases, dispatcher --executor queue, engine-image workflow, scoped RustFS policy); merge main

Implemented with Codex. Cluster deploy pending the k3s kubeconfig and the scoped account.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

ops/k8s/engine/README.md

@@ -0,0 +1,85 @@
+# Engine workers on k3s
+
+Build the new **Engine image** workflow on main (or dispatch it), then copy its
+`ghcr.io/<owner>/openultrasast@sha256:...` job-summary value. The old September
+image does not include boto3 or this service. The release workflow still builds
+version tags. The Dockerfile copies the worker and parser under `/app/benchmarks/learn`
+and installs `.[semantic,s3]`; no analyzer tarball is sent by the dispatcher.
+
+Use two schedulable nodes with at least 3 GiB available each. Required hostname
+anti-affinity places one worker on each node. Updates use Recreate so a third
+worker does not wait forever for a third node. Local scratch uses the container's
+filesystem, with no host mount or Kubernetes API access.
+
+Create a **new scoped account** using the policy in `docs/rustfs.md`. Export its
+`S3_ENDPOINT`, `S3_BUCKET=sast-memory`, `AWS_ACCESS_KEY_ID`, and
+`AWS_SECRET_ACCESS_KEY` in the coordinator shell. Do not use the memory-store
+account. The following commands are for the coordinator; they were not run by the
+implementation agent.
+
+```sh
+export ENGINE_IMAGE='ghcr.io/<owner>/openultrasast@sha256:<digest-from-workflow>'
+# From the repository root, edit the digest parameter without changing the template:
+export ENGINE_OVERLAY="$(mktemp -d /tmp/ousast-engine.XXXXXX)"
+cp ops/k8s/engine/{namespace,deployment,kustomization}.yaml "$ENGINE_OVERLAY/"
+python - <<'PY'
+import os
+from pathlib import Path
+image, digest = os.environ['ENGINE_IMAGE'].split('@', 1)
+p = Path(os.environ['ENGINE_OVERLAY']) / 'kustomization.yaml'
+p.write_text(p.read_text().replace('ghcr.io/OWNER/openultrasast', image)
+             .replace('sha256:REPLACE_WITH_ENGINE_IMAGE_DIGEST', digest))
+PY
+kubectl apply -f ops/k8s/engine/namespace.yaml
+kubectl -n ousast-engine create secret generic ousast-engine-s3 \
+  --from-literal=S3_ENDPOINT="$S3_ENDPOINT" \
+  --from-literal=S3_BUCKET="$S3_BUCKET" \
+  --from-literal=AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" \
+  --from-literal=AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY"
+kubectl apply -k "$ENGINE_OVERLAY"
+kubectl -n ousast-engine rollout status deployment/ousast-engine-worker
+kubectl -n ousast-engine get pods -o wide
+
+export PATH=/home/mc/Source/OpenUltraSAST/.venv/bin:$PATH
+export PYTHONPATH="$PWD/src"
+python benchmarks/learn/engine_trace.py --queue-status
+python benchmarks/learn/engine_trace.py --executor queue \
+  --image "$ENGINE_IMAGE" --parallel 2 --deadline 900 --queue-timeout 3600 \
+  --units plane/experiments/exp-005-graph-slice.units.jsonl \
+  --out "$HOME/ousast-results/plane/engine-trace-queue"
+```
+
+The dispatcher uses the existing local memory examples by default. Add
+`--examples /path/to/examples.jsonl` for a frozen example snapshot. Materialization
+uses locally cached source pins, just as the Docker executor does. `--executor
+k8s-jobs` preserves the old Job lane (`k8s` is its legacy alias); `queue` requires
+no kubectl access. Resume skips completed local units and retries failed or timed
+out units. `progress.json`, `STOP`, filtering, and `--summary` retain their existing
+meaning. Each invocation uses a new run ID, so an old remote result cannot satisfy
+a new retry. The dispatcher records worker and pod identity in each pin JSON.
+
+Tasks contain the pin, units, prepared questions, source object key, and deadlines.
+Results live at `engine-results/<run>/<pin-id>.json`; inputs are removed after
+publication. The Docker lane retains JSON and a log, not a result tar, so the queue
+also returns JSON. Queue timeout records timeout locally and removes pending;
+an already claimed task finishes remotely and cleans its input. Orphaned inputs
+from a failed submission can be removed under `engine-queue/inputs/` after checking
+that no pending task or claim references them.
+
+Claims use per-worker choosing markers followed by ordered tickets. A worker waits
+for choosing peers and every lower ticket before removing pending and starting.
+This requires strongly consistent GET/PUT/LIST from RustFS, including paginated
+LIST, and synchronized node clocks with skew below 30 seconds. Half of the 60-second lease margin is reserved for clock
+skew; the remaining execution budget is checked again after claiming.
+It does not require conditional writes, object tags, S3 Select, or bucket-admin
+permissions. A dead worker's task is recovered from its expired claim. Execution
+runs in a child process. Linux subreaping and a frozen descendant walk kill its
+entire tree, including Joern's separate sessions, at the task deadline before the
+lease expires. As with any unfenced lease, a node suspended beyond its lease must be
+terminated before it rejoins; the protocol cannot guarantee exclusivity against
+arbitrary process suspension or an inconsistent object store.
+
+SIGTERM stops new claims and lets the current task finish. The 900-second
+termination grace matches the maximum task deadline. Keep these values aligned
+if changing the deployment. `/healthz` serves both probes and becomes unready on
+shutdown. Errors omit SDK exception text to avoid exposing credentials.

ops/k8s/engine/deployment.yaml

@@ -0,0 +1,63 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: ousast-engine-worker
+spec:
+  replicas: 2
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app: ousast-engine-worker
+  template:
+    metadata:
+      labels:
+        app: ousast-engine-worker
+    spec:
+      automountServiceAccountToken: false
+      terminationGracePeriodSeconds: 900
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        runAsGroup: 1000
+      affinity:
+        podAntiAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+            - labelSelector:
+                matchLabels:
+                  app: ousast-engine-worker
+              topologyKey: kubernetes.io/hostname
+      containers:
+        - name: worker
+          image: engine-image
+          command: [python, /app/benchmarks/learn/engine_worker_service.py]
+          envFrom:
+            - secretRef:
+                name: ousast-engine-s3
+          env:
+            - name: ENGINE_IMAGE
+              value: engine-image
+          resources:
+            requests:
+              memory: 3Gi
+              cpu: '1'
+            limits:
+              memory: 3Gi
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: [ALL]
+          ports:
+            - name: health
+              containerPort: 8080
+          readinessProbe:
+            httpGet:
+              path: /healthz
+              port: health
+            periodSeconds: 5
+          livenessProbe:
+            httpGet:
+              path: /healthz
+              port: health
+            initialDelaySeconds: 15
+            periodSeconds: 10

ops/k8s/engine/kustomization.yaml

@@ -0,0 +1,22 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: ousast-engine
+resources:
+  - namespace.yaml
+  - deployment.yaml
+# Deliberately non-pullable until the coordinator supplies the workflow's digest.
+images:
+  - name: engine-image
+    newName: ghcr.io/OWNER/openultrasast
+    digest: sha256:REPLACE_WITH_ENGINE_IMAGE_DIGEST
+replacements:
+  - source:
+      kind: Deployment
+      name: ousast-engine-worker
+      fieldPath: spec.template.spec.containers.0.image
+    targets:
+      - select:
+          kind: Deployment
+          name: ousast-engine-worker
+        fieldPaths:
+          - spec.template.spec.containers.0.env.0.value

ops/k8s/engine/namespace.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: ousast-engine

ops/k8s/engine/secret.example.yaml

@@ -0,0 +1,9 @@
+# Reference only; excluded from kustomization. Create using the README command.
+apiVersion: v1
+kind: Secret
+metadata:
+  name: ousast-engine-s3
+  namespace: ousast-engine
+type: Opaque
+stringData: {}
+# Required keys: S3_ENDPOINT, S3_BUCKET, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY