ops(kind): skip Substrate's observability stack by default (930 MiB of 2.3 GB); opt in with OUSAST_KIND_OBSERVABILITY=1

A lean overlay omits otel-collector, prometheus and jaeger and disables the exporters; on an existing cluster up.sh scales them to 0. Verified live: render has no otel-system resources, up.sh idempotent (54 s second run), smoke Run done, kind node 2.66 -> 1.82 GiB. Docs say server VM. Implemented with Codex. Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-16 03:49:26 +00:00 · 2026-10-04 12:03:19 +02:00
parent e555ba68dc
commit 0d78d33fd2
12 changed files with 363 additions and 10 deletions
@@ -17,6 +17,21 @@ in `~/.cache/ousast/ax-src/`. The cluster's context, registry and pool sizes com
 (`ops/k8s/profiles/kind-images.json`), which the templates are re-pinned from and which is committed with them.
 `up.sh` and `smoke-run.sh` run the checkout's code with `OUSAST_PYTHON` (default `.venv/bin/python`).

+The kind profile runs on a 7.7 GB server VM. Observability is off by default
+(`kind_observability = false` in `ops/k8s/profiles/kind.toml`). This saves the measured
+930 MiB used by `otel-system`, about 40% of the cluster's 2.3 GB. The smoke Run still
+completes with those deployments stopped. To enable the stack, run
+`OUSAST_KIND_OBSERVABILITY=1 ops/ax/up.sh` or set `kind_observability = true` in the profile.
+
+`up.sh` stages copied Substrate manifests in a temporary checkout view and installs through
+`hack/install-ate-kind.sh`. The local `ops/ax/substrate-kind-lean/` overlay removes the collector,
+Prometheus, Jaeger and their supporting resources before installation. This preserves the
+installer's secret, CRD and image setup. The upstream checkout is unchanged. The shared
+`ate-otel-config` disables trace, metric and log export and the OTEL SDK; it removes the old
+collector endpoint and metric timing keys. On an existing cluster, `up.sh` scales the three
+observability deployments to zero, replaces the ConfigMap data and rolls its consumers when
+the configuration changes. Opting in restores the upstream ConfigMap and observability manifests.
+
 ## How a Run executes

 A Run (`plane/runs/*.yaml`, kind `openultrasast.io/v1alpha1 Run`) is a DAG of steps; each step names an ax Task
@@ -191,7 +206,7 @@ configure the bucket.
 No deployment outside this host's kind cluster has been made. The step-by-step guidance, with what is implemented
 and what is planned, is `docs/deployment.md` (the site page "Deployment"). The Model and Workspace manifests under
 `plane/` move as they are and the Task templates need their image reference re-pinned to your registry; these parts
-assume this laptop and must change first (checked 2026-10-02):
+assume this server VM and must change first (checked 2026-10-02):

 | Assumption | Where | Needed in a real cluster |
 |---|---|---|
@@ -0,0 +1,43 @@
+#!/usr/bin/env python3
+"""Stage an installer checkout with lean kind manifests; never modify the source checkout.
+
+Usage: python ops/ax/prepare-substrate.py SOURCE EMPTY_DESTINATION
+The destination lives only for an install/render. Source/build inputs are symlinked;
+manifests are copied because the installer uses fixed paths (including the OTEL ConfigMap).
+"""
+
+from __future__ import annotations
+
+import shutil
+import sys
+from pathlib import Path
+
+import yaml
+
+
+def prepare(source: Path, destination: Path) -> None:
+    source = source.resolve()
+    destination.mkdir(parents=True, exist_ok=True)
+    if any(destination.iterdir()):
+        raise ValueError(f"staging directory must be empty: {destination}")
+    for entry in source.iterdir():
+        if entry.name != "manifests":
+            (destination / entry.name).symlink_to(entry, target_is_directory=entry.is_dir())
+    shutil.copytree(source / "manifests", destination / "manifests")
+    install = destination / "manifests/ate-install"
+    shutil.copytree(install / "kind", install / "kind-upstream")
+    overlay = Path(__file__).resolve().parent / "substrate-kind-lean"
+    shutil.copytree(overlay, install / "kind", dirs_exist_ok=True)
+    # The installer applies this file directly before it renders the bundle.
+    # Keep the strategic-merge directive only in the overlay's patch file.
+    config = yaml.safe_load((overlay / "ate-otel-config.yaml").read_text())
+    config["data"].pop("$patch")
+    (install / "kind/disabled-otel-config.yaml").write_text(yaml.safe_dump(config))
+    (install / "kind/ate-otel-config.yaml").rename(install / "kind/otel-config-patch.yaml")
+    (install / "kind/disabled-otel-config.yaml").rename(install / "kind/ate-otel-config.yaml")
+    kustomization = install / "kind/kustomization.yaml"
+    kustomization.write_text(kustomization.read_text().replace("path: ate-otel-config.yaml", "path: otel-config-patch.yaml"))
+
+
+if __name__ == "__main__":
+    prepare(Path(sys.argv[1]), Path(sys.argv[2]))
@@ -0,0 +1,11 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: ate-otel-config
+  namespace: ate-system
+data:
+  $patch: replace
+  OTEL_TRACES_EXPORTER: none
+  OTEL_METRICS_EXPORTER: none
+  OTEL_LOGS_EXPORTER: none
+  OTEL_SDK_DISABLED: "true"
@@ -0,0 +1,44 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+# prepare-substrate.py stages the upstream kind overlay beside this directory.
+resources:
+  - ../kind-upstream
+patches:
+  - target:
+      namespace: otel-system
+    patch: |-
+      $patch: delete
+      apiVersion: v1
+      kind: ConfigMap
+      metadata:
+        name: unused
+  - target:
+      kind: Namespace
+      name: otel-system
+    patch: |-
+      $patch: delete
+      apiVersion: v1
+      kind: Namespace
+      metadata:
+        name: otel-system
+  - target:
+      kind: ClusterRole
+      name: ate-prometheus
+    patch: |-
+      $patch: delete
+      apiVersion: rbac.authorization.k8s.io/v1
+      kind: ClusterRole
+      metadata:
+        name: ate-prometheus
+  - target:
+      kind: RoleBinding
+      name: ate-prometheus
+      namespace: ate-system
+    patch: |-
+      $patch: delete
+      apiVersion: rbac.authorization.k8s.io/v1
+      kind: RoleBinding
+      metadata:
+        name: ate-prometheus
+        namespace: ate-system
+  - path: ate-otel-config.yaml
@@ -15,6 +15,7 @@ profile() { PYTHONPATH="$REPO_ROOT/src" "$PY" -m openultrasast.plane.profile --p
 CTX="$(profile kube_context)"                       # kind names its context kind-<cluster>
 export KIND_CLUSTER_NAME="${CTX#kind-}"
 export KO_DOCKER_REPO="${KO_DOCKER_REPO:-$(profile registry)}"
+OBSERVABILITY="$(profile kind_observability)"
 IMAGES_FILE="$(profile images)"                     # the digest pins the reconciler and the templates read
 SRC="${OUSAST_AX_SRC:-$HOME/.cache/ousast/ax-src}"

@@ -37,9 +38,68 @@ else
 fi

 step "agent substrate (ate-system)"
+# The upstream installer has fixed manifest paths and also creates secrets/CRDs.
+# Give it a temporary checkout view so it never installs the observability resources.
+install_substrate() (
+  if [ "$OBSERVABILITY" = true ]; then
+    cd "$SRC/substrate"
+  else
+    # Symlinked build inputs must not make the staged view a different binary version.
+    export VERSION="${VERSION:-$(git -C "$SRC/substrate" describe --tags --always --dirty)}"
+    staged="$(mktemp -d)"
+    trap 'rm -rf "$staged"' EXIT
+    "$PY" "$REPO_ROOT/ops/ax/prepare-substrate.py" "$SRC/substrate" "$staged"
+    cd "$staged"
+  fi
+  KUBECTL_CONTEXT="$CTX" hack/install-ate-kind.sh --deploy-ate-system
+)
 if ready ate-system; then echo "substrate ready; kept"; else
-  (cd "$SRC/substrate" && hack/install-ate-kind.sh --deploy-ate-system)
+  install_substrate
 fi
+# Reconcile the choice even when the existing control plane is already ready.
+if [ "$OBSERVABILITY" = true ]; then
+  for manifest in otel-collector prometheus; do
+    kubectl --context "$CTX" apply -f "$SRC/substrate/manifests/ate-install/kind/$manifest.yaml"
+  done
+  otel_config="$SRC/substrate/manifests/ate-install/kind/ate-otel-config.yaml"
+else
+  otel_config="$REPO_ROOT/ops/ax/substrate-kind-lean/ate-otel-config.yaml"
+  for deployment in opentelemetry-collector prometheus jaeger; do
+    existing="$(kubectl --context "$CTX" -n otel-system get deployment "$deployment" --ignore-not-found -o name)"
+    if [ -n "$existing" ]; then
+      kubectl --context "$CTX" -n otel-system scale "$existing" --replicas=0
+    fi
+  done
+  echo "kind observability off: existing otel-system deployments scaled to 0; opt in with OUSAST_KIND_OBSERVABILITY=1 ops/ax/up.sh"
+fi
+# Replace data to remove stale exporter/endpoint keys in either direction. ConfigMap
+# envFrom is read at pod start; a content annotation triggers only changed rollouts.
+otel_data="$("$PY" - "$otel_config" <<'PYCONFIG'
+import json
+import sys
+import yaml
+config = yaml.safe_load(open(sys.argv[1]))
+config["data"].pop("$patch", None)
+print(json.dumps(config["data"], sort_keys=True))
+PYCONFIG
+)"
+kubectl --context "$CTX" -n ate-system patch configmap ate-otel-config --type=json \
+  -p "[{\"op\":\"replace\",\"path\":\"/data\",\"value\":$otel_data}]"
+otel_hash="$(printf '%s' "$otel_data" | sha256sum | cut -d ' ' -f1)"
+otel_workloads="$(kubectl --context "$CTX" -n ate-system get deployments,daemonsets -o json | "$PY" -c '
+import json, sys
+for item in json.load(sys.stdin)["items"]:
+    spec = item["spec"]["template"]["spec"]
+    containers = spec.get("containers", []) + spec.get("initContainers", [])
+    if any(env.get("configMapRef", {}).get("name") == "ate-otel-config"
+           for container in containers for env in container.get("envFrom", [])):
+        print(item["kind"].lower() + "/" + item["metadata"]["name"])
+')"
+for workload in $otel_workloads; do
+  kubectl --context "$CTX" -n ate-system patch "$workload" --type=merge \
+    -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"ousast.io/otel-config\":\"$otel_hash\"}}}}}"
+  kubectl --context "$CTX" -n ate-system rollout status "$workload" --timeout=300s
+done
 kubectl --context "$CTX" -n ate-system get pods

 step "egress gateway (atenet-egress, agentgateway variant: one prebuilt image, nothing built)"