mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
merge(plane-on-kubernetes): groups 1-2 -- cluster profile, no host literals, delivery through the S3 store
Kind proofs: records 01 and 02. Task 2.6 (receiver removal) is paused while the safety net and the decision engine come first. deployment.md conflicts resolved in the plain-language style; corrects the RuntimeClass claim (Substrate runs runsc inside its worker pod). Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
+44
-39
@@ -31,11 +31,11 @@ flowchart LR
|
||||
sub["Agent Substrate (ate-system): atenet-router, atenet-egress"]
|
||||
act["Task actors in gVisor workers (WorkerPool ousast-pool)"]
|
||||
svc["Service ousast-receiver + EndpointSlice to the host"]
|
||||
reg["Registry localhost:5001: runner image by digest"]
|
||||
reg["Registry named by the profile: runner image by digest"]
|
||||
end
|
||||
mem[("Memory store: file store on the host, or an S3 bucket (RustFS tested)")]
|
||||
rec -- "ax apply, resume, get, delete" --> ax
|
||||
rec -- "EgressPolicy per actor; start request via port-forward" --> sub
|
||||
rec -- "EgressPolicy per actor; start request through the kube-context tunnel" --> sub
|
||||
sub --> act
|
||||
act -- "HTTP by Service name through the gateway" --> svc
|
||||
svc --> rcv
|
||||
@@ -50,8 +50,8 @@ flowchart LR
|
||||
| How actors reach the receiver | the Service `ousast-receiver.ax-system` (port 80) with a selector-less EndpointSlice whose endpoint is the kind network's gateway address on the host; the runner dials the ClusterIP with the Service name as `Host`, which the egress gateway allows by hostname | `ops/ax/receiver-service.yaml.tmpl`, rendered by `up.sh` |
|
||||
| ax control plane, Agent Substrate, egress gateway, router | the kind cluster (`ax-system`, `ate-system`) | installed by `ops/ax/up.sh` from the upstream checkouts under `~/.cache/ousast/ax-src/` |
|
||||
| Tasks | one actor per Task in a gVisor worker; two warm workers of 1 CPU / 1.5 GiB | `ops/ax/workerpool.yaml.tmpl` |
|
||||
| Runner image | the kind-local registry `localhost:5001`, pinned by digest (`~/.cache/ousast/ax-src/runner-image`) | `plane/Dockerfile.runner`; the `image` of every template under `plane/tasks/` |
|
||||
| Provider keys | read from the operator's environment or `.env` (which never overrides an exported variable), sent only in the start request through a `kubectl port-forward` to `svc/atenet-router` | `src/openultrasast/plane/router.py` |
|
||||
| Runner image | the registry the profile names (`registry` in `ops/k8s/profiles/kind.toml`: the kind-local one), pinned by digest in the profile's images file (`ops/k8s/profiles/kind-images.json`) | `plane/Dockerfile.runner`; the `image` of every template under `plane/tasks/` |
|
||||
| Provider keys | read from the operator's environment or `.env` (which never overrides an exported variable), sent only in the start request through the tunnel `router.py` opens to `svc/atenet-router` in the profile's kube context | `src/openultrasast/plane/router.py` |
|
||||
| Memory store | `OUSAST_MEMORY`: a file store under `~/ousast-results/plane/memory` by default, or `s3://<bucket>[/<prefix>]` on any reachable S3-compatible server (RustFS is the tested one) | `src/openultrasast/plane/memory.py`, [Memory](memory.md), [RustFS setup](rustfs.md) |
|
||||
| Run state and artifacts | `$OUSAST_RESULTS/plane/<run>/` on the host (default `~/ousast-results/plane/`) | `reconciler.py` (`run_dir`) |
|
||||
|
||||
@@ -80,7 +80,7 @@ files live in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`).
|
||||
The script is idempotent: it skips each step that is already done. In order, it:
|
||||
|
||||
- creates the kind cluster (`KIND_CLUSTER_NAME`, default `ousast`) with its registry
|
||||
(`KO_DOCKER_REPO`, default `localhost:5001`);
|
||||
(`KO_DOCKER_REPO`, default: the `registry` of `ops/k8s/profiles/kind.toml`);
|
||||
- installs Agent Substrate;
|
||||
- applies the egress gateway if the install left it out (the agentgateway variant, one prebuilt image);
|
||||
- deploys the ax control plane;
|
||||
@@ -95,17 +95,20 @@ files live in `~/.cache/ousast/ax-src/` (`OUSAST_AX_SRC`).
|
||||
2. **Check it.**
|
||||
|
||||
```bash
|
||||
uv run ousast plane doctor
|
||||
uv run ousast plane doctor --profile kind
|
||||
```
|
||||
|
||||
It runs four checks:
|
||||
It checks the profile you pass (`--profile`, default `kind`; the profiles are in `ops/k8s/profiles/`):
|
||||
|
||||
- the kind context is reachable;
|
||||
- its configured addresses, printed on the first line;
|
||||
- the kube context is reachable;
|
||||
- every pod in `ate-system` is ready;
|
||||
- every pod in `ax-system` is ready;
|
||||
- the registry at `localhost:5001` holds `ousast-runner`.
|
||||
- the runner image of the profile's images file is in its registry (a local registry over its v2
|
||||
API, a remote one through `docker manifest inspect`);
|
||||
- the memory store opens (an S3 bucket is verified).
|
||||
|
||||
Each failing line names `ops/ax/up.sh` as the fix. `doctor` is informational only.
|
||||
Each failing line names the fix (`ops/ax/up.sh` for the kind profile). `doctor` is informational:
|
||||
`ousast plane run` does not call it.
|
||||
|
||||
3. **Run the end-to-end smoke Run.**
|
||||
@@ -165,8 +168,8 @@ The target cluster would do four things:
|
||||
- keep its memory in your S3 bucket.
|
||||
|
||||
The provider key already travels only in the start request through `atenet-router`. That works the same
|
||||
through a port-forward to any cluster. Everything else below is either a step you take or a code change
|
||||
that is not made yet.
|
||||
through the kube-context tunnel to any cluster. Everything else below is either a step you take or a code
|
||||
change that is not made yet.
|
||||
|
||||
### B.1 Cluster with Agent Substrate, its egress gateway and ax
|
||||
|
||||
@@ -186,6 +189,14 @@ that is not made yet.
|
||||
- TLS passthrough to the Workspaces' Git hosts and the Model's declared hosts.
|
||||
- Deploy ax (`make deploy AX_IMAGE_REPO=<your registry>` in the ax checkout) **after** B.2.
|
||||
- Apply a WorkerPool (B.6). Substrate installs none by itself ("no free workers").
|
||||
- **gVisor on the nodes (planned, checked by hand on the day k3s exists).** Substrate does not use a Kubernetes
|
||||
`RuntimeClass`: the `ateom-gvisor` worker runs `runsc` itself, from the gVisor release tarball named in
|
||||
`manifests/ate-install/sandboxconfig-gvisor.yaml` of the Substrate checkout
|
||||
(`gs://gvisor/releases/nightly/2026-09-02/<arch>/gvisor.tar.zstd`, sha256-pinned per architecture), which
|
||||
the atelet fetches on the worker node. The node must reach that URL, or a copied `SandboxConfig` must name a
|
||||
mirrored copy with the same sha256 and the ActorTemplates must name that config. The check is the atelet's
|
||||
fetch line in `kubectl -n ate-system logs` of the worker pod; it is labelled planned until that log has been
|
||||
read on a k3s node (plane-on-kubernetes task 1.7, design open question 3).
|
||||
|
||||
### B.2 ax's snapshot bucket: `AX_SNAPSHOTS_BUCKET`
|
||||
|
||||
@@ -218,31 +229,23 @@ long Runs need a janitor that removes directories no live actor owns.
|
||||
- Pass `--runner-image FILE` to `ousast plane workspaces` and `ousast plane harvest`. They re-pin the
|
||||
Tasks they generate.
|
||||
- The committed templates under `plane/tasks/` and the generated `plane/tasks/validation-46.yaml` carry
|
||||
`localhost:5001/ousast-runner@sha256:...`. They need the same re-pin before they run elsewhere.
|
||||
the kind profile's registry and digest. They need the same re-pin before they run elsewhere.
|
||||
Regenerate them, or replace the reference.
|
||||
- Workers must be able to pull from the registry. Credentials, if any, are the cluster's concern. ax's
|
||||
default runner image on `gcr.io` needs credentials and is not used here.
|
||||
- **Not yet configurable:** `ousast plane doctor` checks the registry at `localhost:5001` by name
|
||||
(`src/openultrasast/plane/doctor.py:45`). So its fourth check fails on another cluster, even when the
|
||||
image is in place. `doctor` is informational; `plane run` does not depend on it.
|
||||
- `ousast plane doctor --profile <name>` looks up the runner reference of the profile's images file in
|
||||
the profile's `registry` (implemented 2026-10-02, plane-on-kubernetes task 1.2). `doctor` is
|
||||
informational; `plane run` does not depend on it.
|
||||
|
||||
### B.4 One configurable kube context
|
||||
### B.4 One configurable kube context (implemented 2026-10-02)
|
||||
|
||||
The reconciler, the egress client and the router port-forward all use one kube context. Its name is
|
||||
`kind-$KIND_CLUSTER_NAME` (`kind-ousast` by default).
|
||||
The reconciler, the egress client and the router tunnel take the kube context from the profile.
|
||||
|
||||
**Not yet configurable**: the prefix `kind-` is hard-coded in these places:
|
||||
|
||||
- `src/openultrasast/plane/doctor.py:39`;
|
||||
- `src/openultrasast/plane/egress.py:115` and `:148`;
|
||||
- `src/openultrasast/plane/router.py:145`;
|
||||
- `ops/ax/up.sh:24` builds the same name.
|
||||
|
||||
The planned change is one variable read in those three modules, for example `OUSAST_KUBE_CONTEXT`.
|
||||
|
||||
Until then, there is one workaround to drive another cluster with the unchanged code. Rename its
|
||||
kubeconfig context to that form: `kubectl config rename-context <ctx> kind-<name>` and
|
||||
`KIND_CLUSTER_NAME=<name>`. This is a workaround, not a supported configuration.
|
||||
- The setting is `kube_context` in `ops/k8s/profiles/<name>.toml`; `OUSAST_KUBE_CONTEXT` overrides it
|
||||
(plane-on-kubernetes task 1.2).
|
||||
- `doctor.py`, `egress.py` and `router.py` are handed the context. They build none themselves.
|
||||
- `tests/test_plane_literals.py` fails on any module, manifest or page that names this host's cluster,
|
||||
registry or addresses.
|
||||
|
||||
The egress client also needs `kubectl-ate` (or the executable `OUSAST_KUBECTL_ATE` names). It must be
|
||||
able to use that context.
|
||||
@@ -333,10 +336,10 @@ Running the reconciler in-cluster is itself planned. Today it is a host process.
|
||||
| Per-task EgressPolicy, credentials only in the start request, per-task budgets and attribution | implemented |
|
||||
| S3 memory store against any reachable S3-compatible server, verified at startup | implemented (RustFS tested) |
|
||||
| Local bring-up, doctor, smoke Run (`ops/ax/`) | implemented |
|
||||
| Re-pinning generated Tasks to another registry (`--runner-image`) | implemented; the committed templates still name `localhost:5001` |
|
||||
| Re-pinning generated Tasks to another registry (`--runner-image`) | implemented; the committed templates name the kind profile's registry (`tests/test_plane_literals.py` checks it) |
|
||||
| Deployment to a cluster other than this host's kind cluster | **not done** |
|
||||
| One configurable kube context | **planned**; `kind-` hard-coded in `doctor.py`, `egress.py`, `router.py` |
|
||||
| `doctor` registry check against a configurable registry | **planned**; `localhost:5001` hard-coded in `doctor.py:45` |
|
||||
| One configurable kube context | implemented (2026-10-02): `kube_context` in the profile, `OUSAST_KUBE_CONTEXT` overrides |
|
||||
| `doctor` registry check against a configurable registry | implemented (2026-10-02): the profile's `registry` and images file |
|
||||
| Receiver as an in-cluster Deployment | **planned** (code change) |
|
||||
| Presigned-URL delivery to the S3 store | **follow-on**, not started |
|
||||
| Reconciler running in-cluster with Secrets injected into its environment | **planned** |
|
||||
@@ -373,7 +376,7 @@ flowchart LR
|
||||
ctl["ate-api-server, ate-controller, atelet"]
|
||||
rt["atenet-router"]
|
||||
eg["atenet-egress gateway"]
|
||||
wp["WorkerPool: gVisor workers, runsc RuntimeClass, HPA (planned)"]
|
||||
wp["WorkerPool: gVisor workers, runsc in the worker, HPA (planned)"]
|
||||
ewp["Engine WorkerPool: Joern as a task (planned)"]
|
||||
end
|
||||
end
|
||||
@@ -410,7 +413,7 @@ profile only. It does not appear in production.
|
||||
- It is a Deployment `ax-server` with a Redis beside it, in the namespace `ax-system`
|
||||
(`ax/deploy/ax-server.yaml` in the ax checkout).
|
||||
- Its controller runs with `--template=default-template --template-atespace=ax-system`.
|
||||
- The `ax` CLI reaches it through a `kubectl port-forward` to `svc/ax-server`
|
||||
- The `ax` CLI reaches it through a tunnel to `svc/ax-server` over the kube context
|
||||
(`ax/internal/tunnel/tunnel.go`).
|
||||
- So `ousast plane run` needs a kube context or an ingress, not a public endpoint.
|
||||
|
||||
@@ -440,14 +443,16 @@ profile only. It does not appear in production.
|
||||
|
||||
- k3s runs containerd with no Docker socket. So the host-Docker path of `ousast plane alerts-engine`
|
||||
does not exist there. The engine must become a task (below).
|
||||
- Substrate's gVisor workers need `runsc` and a `RuntimeClass` on the nodes they run on.
|
||||
- Substrate's gVisor workers need no Kubernetes `RuntimeClass`: the worker fetches a pinned gVisor
|
||||
tarball and runs `runsc` inside its own pod. The node checklist is pod security, tarball
|
||||
reachability and local-path storage (B.1).
|
||||
- k3s ships Traefik and local-path storage. Traefik is an ingress the CLI can use instead of a kubeconfig.
|
||||
- GitHub Actions builds the images from tagged commits. They are published to GitHub Container Registry
|
||||
and digest-pinned in the manifests.
|
||||
- A private package needs an `imagePullSecret` on the worker ServiceAccount.
|
||||
- Substrate's and ax's own images are built with `ko` from their checkouts. They are pushed to the same
|
||||
registry.
|
||||
- The kind profile keeps `localhost:5001` and `ops/ax/up.sh`.
|
||||
- The kind profile keeps its local registry (`ops/k8s/profiles/kind.toml`) and `ops/ax/up.sh`.
|
||||
|
||||
### One image, and what concurrency equals
|
||||
|
||||
@@ -486,7 +491,7 @@ Substrate ships two scaling aids:
|
||||
|
||||
| Component | Today (the kind profile) | Production (the k3s profile, planned) |
|
||||
| --- | --- | --- |
|
||||
| Reconciler (`ousast plane run`, `src/openultrasast/plane/reconciler.py`) | a process on the operator's host; calls `ax` and `kubectl` against `kind-ousast`, holds the Run's state under `OUSAST_RESULTS` | **the reconciler stays the CLI.** An execution mode `local` or `remote`: in `remote`, the same `ousast plane run` from a laptop or a CI job submits the Run to ax-server in k3s (over the cluster's ingress or a kubeconfig), sends the start requests, then polls the store and prints the same `status` and attribution table. Credentials come from the caller's environment as today (CI secrets in a job; B.8's rule, no value in a manifest). An unattended in-cluster reconciler (a Deployment with a submit API, or a Job per Run) is a **later option**, not the first step |
|
||||
| Reconciler (`ousast plane run`, `src/openultrasast/plane/reconciler.py`) | a process on the operator's host; calls `ax` and `kubectl` against the profile's `kube_context`, holds the Run's state under `OUSAST_RESULTS` | **the reconciler stays the CLI.** An execution mode `local` or `remote`: in `remote`, the same `ousast plane run` from a laptop or a CI job submits the Run to ax-server in k3s (over the cluster's ingress or a kubeconfig), sends the start requests, then polls the store and prints the same `status` and attribution table. Credentials come from the caller's environment as today (CI secrets in a job; B.8's rule, no value in a manifest). An unattended in-cluster reconciler (a Deployment with a submit API, or a Job per Run) is a **later option**, not the first step |
|
||||
| Artifact receiver (`plane/egress.py`, the `Receiver` thread inside the reconciler) | reached through the Service `ousast-receiver.ax-system` whose EndpointSlice points at the kind gateway address on the host (`ops/ax/receiver-service.yaml.tmpl`) | **no receiver in remote mode.** Tasks deliver their artifacts and fetch their inputs through presigned URLs on the S3 memory store (B.5, path 3; the store's hostname enters each Task's EgressPolicy, B.7), and the CLI reads completion from the store. The kind profile keeps the receiver thread; an in-cluster receiver Deployment (B.5, path 2) is not the chosen direction |
|
||||
| Joern engine (`plane/engine_alerts.py`, the host's `openultrasast:dev` container, run with `--memory 3g`) | `ousast plane alerts-engine` runs it on the host, one container at a time, and marks the Run's `alerts` tasks done | **an `engine` task on a second, larger `WorkerPool`** (the engine does not fit a 1.5 GiB worker, and k3s has no Docker socket), from an engine image digest-pinned in GHCR, emitting the same `alerts.jsonl` rows and coverage; `alerts-engine` on the host stays as the development path |
|
||||
|
||||
|
||||
Reference in New Issue
Block a user