feat(plane): per-task egress policy; receiver by name through the gateway

Agent Substrate's egress gateway denies an actor without an EgressPolicy, so
no task could deliver its artifacts or reach a model. egress.py grants each
task exactly: http/80 to ousast-receiver.ax-system.svc.cluster.local, and
TLS passthrough/443 to the bound Workspaces' https Git hosts and the bound
Model's openultrasast.io/egress-hosts (api.deepseek.com for deepseek-flash).
No IPs, no wildcards, no https (MITM) rules.

The reconciler writes the policy with kubectl-ate between ax apply and
resume, reads it back, waits OUSAST_EGRESS_SETTLE (11 s, the gateway's
cached deny), and after ax delete task confirms the policy went with its
actor (kubectl-ate has no delete verb; the store cascades).

The receiver is a selector-less Service (ops/ax/receiver-service.yaml.tmpl,
applied by up.sh together with the no-build agentgateway egress gateway)
on port 18090 of the host. Measured live: the gateway checks the policy on
Host but connects to the address the actor dialled, so the runner dials the
Service's ClusterIP (OUSAST_ARTIFACT_DIAL) with the receiver's name as Host.
The runner now labels its delivery with the Run's task name (OUSAST_TASK),
which is what the receiver accepts; the ax Task name was refused with 403.
The Receiver moved to egress.py to keep reconciler.py at 460 lines.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-09-29 20:51:21 +02:00
co-authored by Claude Opus 5.5
parent 0744a157bf
commit 137c6c8ed5
15 changed files with 703 additions and 82 deletions
+26 -2
View File
@@ -110,8 +110,8 @@ new image can fail with `DeadlineExceeded` while Agent Substrate builds its gold
`OUSAST_RESUME_TIMEOUT` (900 s). ax has no Completed phase and never reports that a command exited: completion is `OUSAST_RESUME_TIMEOUT` (900 s). ax has no Completed phase and never reports that a command exited: completion is
the artifact delivery. `Failed` after a resume went through, or the task disappearing, before delivery fails the the artifact delivery. `Failed` after a resume went through, or the task disappearing, before delivery fails the
task with ax's condition message; no delivery within `OUSAST_TASK_TIMEOUT` (7200 s) fails it too; `ax delete task` task with ax's condition message; no delivery within `OUSAST_TASK_TIMEOUT` (7200 s) fails it too; `ax delete task`
follows either way. Artifacts: the reconciler serves a small HTTP receiver on the host (reachable from kind pods follows either way. Artifacts: the reconciler serves a small HTTP receiver on the host (reachable from actors
via the docker bridge); the runner posts the output directory there after its command exits. Tasks sharing a by name through the egress gateway, see Egress below); the runner posts the output directory there after its command exits. Tasks sharing a
`serialize` label never overlap; everything else may run in parallel up to `--workers`. A task's outcome is `serialize` label never overlap; everything else may run in parallel up to `--workers`. A task's outcome is
read from its delivered `summary.json`: `done` only when the task says every unit finished; `unfinished` on a read from its delivered `summary.json`: `done` only when the task says every unit finished; `unfinished` on a
budget stop; `failed` on an account or authentication error, a crash, or a missing delivery. `status` prints budget stop; `failed` on an account or authentication error, a crash, or a missing delivery. `status` prints
@@ -131,6 +131,30 @@ bound Model's `spec.secretKey.key` names the variable (`DEEPSEEK_API_KEY` for `d
reconciler's own environment (`ousast` loads `.env` at startup); a Model-bound task whose variable is unset fails reconciler's own environment (`ousast` loads `.env` at startup); a Model-bound task whose variable is unset fails
before `ax apply`, and a task without a Model gets none. The value is never rendered, logged or written. before `ax apply`, and a task without a Model gets none. The value is never rendered, logged or written.
#### Egress (`egress.py`)
Every actor connection except DNS goes through Agent Substrate's egress gateway (`atenet-egress`), which denies
an actor without an `EgressPolicy`. Three pieces make the plane work under that rule:
- **Receiver by name.** `ops/ax/receiver-service.yaml.tmpl` (applied by `up.sh`) is a selector-less Service
`ousast-receiver` in `ax-system`, port 80, with an EndpointSlice on the kind gateway at `OUSAST_ARTIFACT_PORT`
(fixed, 18090), where the reconciler binds its receiver. Tasks get
`OUSAST_ARTIFACT_URL=http://ousast-receiver.ax-system.svc.cluster.local/` plus `OUSAST_ARTIFACT_DIAL=<the
Service's ClusterIP>:80` (looked up with kubectl unless set): the actor cannot resolve cluster names, so the
runner connects to the dial address and sends the URL's host as `Host`. The gateway decides the policy on
`Host` but connects to the address the actor dialled (measured: dialling the kind gateway on port 80 was allowed
and then refused upstream), so the dial address is the ClusterIP, which kube-proxy maps to the host.
`OUSAST_ARTIFACT_HOST` set keeps the old direct URL (tests).
- **Per-task policy.** `policy_for` grants `http` on port 80 to the receiver name, and `tls_passthrough` on 443
to the Git hosts of the bound Workspaces' `https` repos and to the bound Model's
`openultrasast.io/egress-hosts` (`api.deepseek.com` for `deepseek-flash`, validated as a plain DNS name).
Nothing else: no IP address or wildcard, and passthrough only -- an `https` rule intercepts TLS and the actor
would need the gateway CA, which ax templates do not mount.
- **Order.** `ax apply` creates the actor; the reconciler writes the policy with `kubectl ate create
egress-policy` (update when one exists), reads it back, waits `OUSAST_EGRESS_SETTLE` (11 s: the gateway caches a
missing policy as deny for 10 s), then resumes. kubectl-ate has no delete verb; Substrate's store deletes the
policy with its actor, so after `ax delete task` the reconciler confirms it is gone.
### Budget (`budget.py`) ### Budget (`budget.py`)
Wraps a chat client: sums the provider's usage fields per call (`prompt_tokens`, `prompt_cache_hit_tokens`, Wraps a chat client: sums the provider's usage fields per call (`prompt_tokens`, `prompt_cache_hit_tokens`,
+6
View File
@@ -11,6 +11,12 @@ mypy are clean, and the commit is gated on pytest's own exit code. Script-level
and wait for it to finish; `ops/ax/down.sh` tears it down. Record the measured idle footprint (RSS, CPU) and wait for it to finish; `ops/ax/down.sh` tears it down. Record the measured idle footprint (RSS, CPU)
in `ops/ax/README.md` next to the 7 GB host constraint. in `ops/ax/README.md` next to the 7 GB host constraint.
- `ousast plane doctor` (in `reconciler.py` or a sibling `doctor.py`): the four checks of Requirement 3.5. - `ousast plane doctor` (in `reconciler.py` or a sibling `doctor.py`): the four checks of Requirement 3.5.
- Egress: `up.sh` applies Substrate's agentgateway egress gateway (`atenet-egress`, prebuilt image, rendered
with `kubectl kustomize`) when absent, and the receiver Service `ousast-receiver.ax-system`
(`receiver-service.yaml.tmpl`, endpoint the kind gateway at port 18090). The reconciler writes each task's
egress policy (`egress.py`: receiver over HTTP, Git and Model hosts as TLS passthrough) between `ax apply` and
resume; the runner delivers to `OUSAST_ARTIFACT_DIAL` (the Service's ClusterIP, port 80) with the receiver's name as
`Host`.
- Stop rule: if the cluster does not come up within one working session, report the exact failure and stop; - Stop rule: if the cluster does not come up within one working session, report the exact failure and stop;
the decision to work around it is the maintainer's. the decision to work around it is the maintainer's.
- _Requirements: 3.5, 3.6_ - _Requirements: 3.5, 3.6_
@@ -1,7 +1,7 @@
{ {
"modules": 131, "modules": 132,
"counts": { "counts": {
"load_bearing": 122, "load_bearing": 123,
"standalone_capability": 9 "standalone_capability": 9
}, },
"orphaned": [], "orphaned": [],
@@ -661,6 +661,7 @@
"reason": "ai-service-plane Req 1-4: the service plane package; runner and reconciler integration in tasks 3-4", "reason": "ai-service-plane Req 1-4: the service plane package; runner and reconciler integration in tasks 3-4",
"importers": [ "importers": [
"cli", "cli",
"plane.egress",
"plane.generate", "plane.generate",
"plane.reconciler", "plane.reconciler",
"plane.router", "plane.router",
@@ -686,6 +687,14 @@
"plane.reconciler" "plane.reconciler"
] ]
}, },
{
"module": "plane.egress",
"classification": "load_bearing",
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
"importers": [
"plane.reconciler"
]
},
{ {
"module": "plane.generate", "module": "plane.generate",
"classification": "load_bearing", "classification": "load_bearing",
@@ -699,6 +708,7 @@
"classification": "load_bearing", "classification": "load_bearing",
"reason": "ai-service-plane Req 1.1-1.4: ax-shaped Task/Workspace/Model and the Run kind, validated before anything runs", "reason": "ai-service-plane Req 1.1-1.4: ax-shaped Task/Workspace/Model and the Run kind, validated before anything runs",
"importers": [ "importers": [
"plane.egress",
"plane.generate", "plane.generate",
"plane.reconciler", "plane.reconciler",
"plane.router", "plane.router",
+31
View File
@@ -0,0 +1,31 @@
# The reconciler's artifact receiver by name (ai-service-plane task 0). Actors reach nothing but through Agent
# Substrate's egress gateway, and the gateway allows HTTP by hostname only (never an IP address): the runner dials
# this Service's ClusterIP on port 80 with Host `ousast-receiver.ax-system.svc.cluster.local`; the gateway allows
# it on that Host and connects to the ClusterIP, which kube-proxy maps to the endpoint below, the host's receiver
# on ${OUSAST_ARTIFACT_PORT} at the kind gateway. Rendered by up.sh.
apiVersion: v1
kind: Service
metadata:
name: ousast-receiver
namespace: ax-system
spec:
ports:
- name: http
port: 80
targetPort: ${OUSAST_ARTIFACT_PORT}
protocol: TCP
---
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
name: ousast-receiver-host
namespace: ax-system
labels:
kubernetes.io/service-name: ousast-receiver
addressType: IPv4
ports:
- name: http
port: ${OUSAST_ARTIFACT_PORT}
protocol: TCP
endpoints:
- addresses: ["${KIND_GATEWAY}"]
+4 -2
View File
@@ -4,8 +4,10 @@ set -euo pipefail
export PATH="$HOME/go/bin:$HOME/.local/bin:$PATH" export PATH="$HOME/go/bin:$HOME/.local/bin:$PATH"
SRC="${OUSAST_AX_SRC:-$HOME/.cache/ousast/ax-src}" SRC="${OUSAST_AX_SRC:-$HOME/.cache/ousast/ax-src}"
RUNNER_IMAGE="$(cat "$SRC/runner-image")"; export RUNNER_IMAGE RUNNER_IMAGE="$(cat "$SRC/runner-image")"; export RUNNER_IMAGE
KIND_GW="$(docker network inspect kind --format '{{range .IPAM.Config}}{{if .Gateway}}{{.Gateway}} {{end}}{{end}}' | tr ' ' '\n' | grep -m1 '\.')" # Actors reach the receiver by its Service name through the egress gateway (ops/ax/receiver-service.yaml.tmpl,
export OUSAST_ARTIFACT_HOST="${OUSAST_ARTIFACT_HOST:-$KIND_GW}" # sandboxes reach the host via the kind gateway # applied by up.sh, endpoint this host at OUSAST_ARTIFACT_PORT): the runner dials the Service's ClusterIP on port
# 80 with that name as Host; the reconciler looks the ClusterIP up unless OUSAST_ARTIFACT_DIAL is set.
export OUSAST_ARTIFACT_PORT="${OUSAST_ARTIFACT_PORT:-18090}"
envsubst < "$(dirname "$0")/smoke-run.yaml.tmpl" > "$SRC/smoke-run.yaml" envsubst < "$(dirname "$0")/smoke-run.yaml.tmpl" > "$SRC/smoke-run.yaml"
REPO="$(cd "$(dirname "$0")/../.." && pwd)" REPO="$(cd "$(dirname "$0")/../.." && pwd)"
"$REPO/.venv/bin/ousast" plane run "$SRC/smoke-run.yaml" "$@" "$REPO/.venv/bin/ousast" plane run "$SRC/smoke-run.yaml" "$@"
+14
View File
@@ -35,6 +35,15 @@ if ready ate-system; then echo "substrate ready; kept"; else
fi fi
kubectl --context "$CTX" -n ate-system get pods kubectl --context "$CTX" -n ate-system get pods
step "egress gateway (atenet-egress, agentgateway variant: one prebuilt image, nothing built)"
# Every actor connection goes through this gateway; without it an actor has no network at all. The Substrate
# install can die before deploying it (e.g. a full disk), so it is applied on its own when absent.
if kubectl --context "$CTX" -n ate-system get deploy atenet-egress >/dev/null 2>&1; then echo "gateway present; kept"; else
kubectl kustomize --load-restrictor=LoadRestrictionsNone "$SRC/substrate/manifests/ate-install/agentgateway-egress" \
| kubectl --context "$CTX" apply -f -
kubectl --context "$CTX" -n ate-system rollout status deploy/atenet-egress --timeout=300s
fi
step "ax control plane (ax-system)" step "ax control plane (ax-system)"
if ready ax-system; then echo "ax ready; kept"; else if ready ax-system; then echo "ax ready; kept"; else
(cd "$SRC/ax" && make deploy AX_IMAGE_REPO="$KO_DOCKER_REPO") (cd "$SRC/ax" && make deploy AX_IMAGE_REPO="$KO_DOCKER_REPO")
@@ -42,6 +51,11 @@ if ready ax-system; then echo "ax ready; kept"; else
fi fi
kubectl --context "$CTX" -n ax-system get pods kubectl --context "$CTX" -n ax-system get pods
step "artifact receiver Service (ousast-receiver.ax-system -> this host)"
KIND_GATEWAY="$(docker network inspect kind --format '{{range .IPAM.Config}}{{if .Gateway}}{{.Gateway}} {{end}}{{end}}' | tr ' ' '\n' | grep -m1 '\.')"
export KIND_GATEWAY OUSAST_ARTIFACT_PORT="${OUSAST_ARTIFACT_PORT:-18090}" # the reconciler's fixed receiver port
envsubst < "$(dirname "$0")/receiver-service.yaml.tmpl" | kubectl --context "$CTX" apply -f -
step "worker pool (gVisor)" step "worker pool (gVisor)"
SUBSTRATE_VERSION="$(kubectl --context "$CTX" get nodes -o jsonpath='{.items[0].metadata.labels.ate\.dev/substrate-version}')" SUBSTRATE_VERSION="$(kubectl --context "$CTX" get nodes -o jsonpath='{.items[0].metadata.labels.ate\.dev/substrate-version}')"
export SUBSTRATE_VERSION export SUBSTRATE_VERSION
+1
View File
@@ -10,6 +10,7 @@ metadata:
atespace: default atespace: default
annotations: annotations:
openultrasast.io/provider-extension: "true" openultrasast.io/provider-extension: "true"
openultrasast.io/egress-hosts: "api.deepseek.com" # model.endpoint.DEEPSEEK_BASE_URL; the task's only model egress
spec: spec:
provider: deepseek provider: deepseek
model: deepseek-flash model: deepseek-flash
+252
View File
@@ -0,0 +1,252 @@
"""Per-task network egress on Agent Substrate (ai-service-plane, task 0: gateway + receiver Service).
Every TCP connection an actor opens (DNS aside) is redirected to Substrate's egress gateway
(``atenet-egress.ate-system``), which denies by default: an actor without an ``EgressPolicy`` gets no tunnel, and
a refused connection is accepted and then closed. :func:`policy_for` grants a task exactly what it needs:
* ``http`` on port 80 to the artifact receiver's cluster name (``ousast-receiver.ax-system.svc.cluster.local``, a
selector-less Service whose endpoint is this host's receiver, ``ops/ax/receiver-service.yaml.tmpl``). The actor
cannot resolve cluster names, so the runner dials ``OUSAST_ARTIFACT_DIAL`` (the Service's ClusterIP, port 80)
with the receiver's name as ``Host``. The gateway decides the policy on that ``Host`` but connects to the
address the actor dialled (measured live: agentgateway's upstream is the CONNECT authority, not the Host), so
the dial address must be one the gateway pod reaches: the ClusterIP, which kube-proxy maps to the host.
* ``tls_passthrough`` on 443 for the Git hosts of the bound Workspaces' ``https`` repos (smart HTTP talks to the
repo's host only) and for the bound Model's ``openultrasast.io/egress-hosts`` (``api.deepseek.com``).
Nothing else: no wildcard, no IP address (the gateway rejects them), no ``https`` rule (that one intercepts TLS and
the actor would have to trust the gateway CA, which ax templates do not mount). :class:`Egress` writes the policy
with ``kubectl ate create egress-policy`` after ``ax apply`` created the actor, reads it back, and before resume
waits ``OUSAST_EGRESS_SETTLE`` s (11) because the gateway caches a missing policy as deny for 10 s. The kubectl-ate
CLI has no delete verb: Substrate's store deletes the policy with its actor (``ON DELETE CASCADE``), so
:meth:`Egress.delete` runs after ``ax delete task`` and confirms the policy is gone.
"""
from __future__ import annotations
import io
import ipaddress
import json
import os
import subprocess
import tarfile
import threading
import time
from collections.abc import Iterable, Mapping
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
from typing import Any
from urllib.parse import urlsplit
from .manifests import Model, Task, Workspace
__all__ = [
"RECEIVER_HOST",
"RECEIVER_PORT",
"Egress",
"Receiver",
"git_hosts",
"receiver_cluster_ip",
"policy_for",
"receiver_address",
"rules_of",
]
RECEIVER_HOST = "ousast-receiver.ax-system.svc.cluster.local"
RECEIVER_PORT = 18090 # the receiver Service's targetPort; ops/ax/up.sh renders the Service with the same number
_NOT_FOUND = ("not found", "notfound")
def _is_ip(host: str) -> bool:
try:
ipaddress.ip_address(host.strip("[]"))
except ValueError:
return False
return True
def git_hosts(workspaces: Iterable[Workspace]) -> tuple[list[str], list[str]]:
"""(https hosts, http hosts) of the Workspaces' ``spec.git[].repo`` URLs; ssh and IP-address repos add nothing."""
tls, plain = set[str](), set[str]()
for ws in workspaces:
for source in ws.git:
parts = urlsplit(source.repo)
host = (parts.hostname or "").lower()
if (
host
and not _is_ip(host)
and parts.scheme in ("https", "http")
and parts.port in (None, 443 if parts.scheme == "https" else 80)
):
(tls if parts.scheme == "https" else plain).add(host)
return sorted(tls), sorted(plain)
def policy_for(task: Task, workspaces: Iterable[Workspace], model: Model | None, receiver_host: str | None) -> dict[str, Any]:
"""The task's EgressPolicy body: the receiver over HTTP/80, Git and Model hosts as TLS passthrough on 443."""
tls, plain = git_hosts(workspaces)
tls = sorted(set(tls) | set(model.egress_hosts if model else ()))
if receiver_host and not _is_ip(receiver_host): # an address cannot be allowed; only the direct test mode has one
plain = sorted({*plain, receiver_host.lower()})
rules: list[dict[str, Any]] = []
if plain:
rules.append({"http": {"hostnames": plain, "ports": {"numbers": [80]}}})
if tls:
rules.append({"tlsPassthrough": {"hostnames": tls, "ports": {"numbers": [443]}}})
return {"rules": rules}
def rules_of(policy: Mapping[str, Any]) -> list[dict[str, Any]]:
"""The rules of a policy as ``kubectl ate get -o json`` prints them, normalised for comparison."""
out = []
for rule in policy.get("rules") or []:
for kind, body in rule.items():
key = "tlsPassthrough" if kind in ("tls_passthrough", "tlsPassthrough") else kind
ports = body.get("ports") or {}
norm_ports: dict[str, Any] = {"all": {}} if "all" in ports else {"numbers": sorted(int(n) for n in ports.get("numbers") or [])}
out.append({key: {"hostnames": sorted(body.get("hostnames") or []), "ports": norm_ports}})
return sorted(out, key=lambda r: json.dumps(r, sort_keys=True))
def receiver_cluster_ip(context: str | None = None) -> str | None:
"""The ClusterIP of the ``ousast-receiver`` Service (``kubectl get svc``); None when kubectl cannot say."""
context = context or "kind-" + (os.environ.get("KIND_CLUSTER_NAME") or "ousast")
argv = ["kubectl", "--context", context, "-n", "ax-system", "get", "svc", "ousast-receiver", "-o", "jsonpath={.spec.clusterIP}"]
try:
proc = subprocess.run(argv, capture_output=True, text=True, check=False, timeout=20)
except (OSError, subprocess.TimeoutExpired):
return None
address = proc.stdout.strip()
return address if proc.returncode == 0 and _is_ip(address) else None
def receiver_address(port: int) -> tuple[str, str | None]:
"""(``OUSAST_ARTIFACT_URL``, ``OUSAST_ARTIFACT_DIAL``) for a receiver bound on ``port``.
``OUSAST_ARTIFACT_HOST`` set: the old direct URL ``http://<host>:<port>/`` and no dial (tests, a host the actor
reaches without the gateway). Otherwise the receiver Service's name, dialled at ``OUSAST_ARTIFACT_DIAL`` or the
Service's ClusterIP on port 80: the gateway checks the policy against ``Host`` but connects to the address the
actor dialled (its CONNECT authority), so that address must reach the receiver from the gateway pod.
"""
host = os.environ.get("OUSAST_ARTIFACT_HOST")
if host:
return f"http://{host}:{port}/", None
dial = os.environ.get("OUSAST_ARTIFACT_DIAL")
if not dial:
address = receiver_cluster_ip()
dial = f"{address}:80" if address else None
return f"http://{RECEIVER_HOST}/", dial
class Egress:
"""``kubectl ate`` for one actor's egress policy; the executable is injectable so tests use a fake."""
def __init__(self, executable: str | None = None, context: str | None = None) -> None:
self.executable = executable or os.environ.get("OUSAST_KUBECTL_ATE") or "kubectl-ate"
self.context = context or "kind-" + (os.environ.get("KIND_CLUSTER_NAME") or "ousast")
def _run(self, *args: str, stdin: str | None = None) -> subprocess.CompletedProcess[str]:
argv = [self.executable, "--context", self.context, *args]
return subprocess.run(argv, input=stdin, capture_output=True, text=True, check=False)
@staticmethod
def _error(proc: subprocess.CompletedProcess[str]) -> str:
return (proc.stderr.strip() or proc.stdout.strip() or f"exit {proc.returncode}")[-400:]
def get(self, actor: str, atespace: str) -> dict[str, Any] | None:
"""The stored policy, None when there is none (or no actor); RuntimeError when kubectl-ate fails otherwise."""
proc = self._run("get", "egress-policy", actor, "-a", atespace, "-o", "json")
if proc.returncode != 0:
if "notfound" in self._error(proc).lower().replace(" ", ""):
return None
raise RuntimeError(f"kubectl ate get egress-policy {atespace}/{actor} failed: {self._error(proc)}")
if not proc.stdout.strip(): # an actor without a policy: exit 0, "has no egress policy" on stderr only
return None
try:
doc = json.loads(proc.stdout)
except ValueError as exc:
raise RuntimeError(f"kubectl ate get egress-policy {atespace}/{actor}: unreadable output ({exc})") from None
return doc if isinstance(doc, dict) else None
def apply(self, actor: str, atespace: str, policy: Mapping[str, Any], *, timeout: float | None = None) -> None:
"""Create (or replace) the policy, retrying while the actor does not exist yet, then read it back."""
limit = float(os.environ.get("OUSAST_EGRESS_TIMEOUT") or 60) if timeout is None else timeout
deadline, backoff = time.monotonic() + limit, 0.5
while True:
existing = self.get(actor, atespace)
if existing is None:
proc = self._run("create", "egress-policy", actor, "-a", atespace, "-f", "-", stdin=json.dumps(dict(policy)))
else:
meta = {k: existing.get("metadata", {}).get(k) for k in ("name", "atespace", "uid", "version")}
body = json.dumps({**dict(policy), "metadata": meta})
proc = self._run("update", "egress-policy", actor, "-a", atespace, "-f", "-", stdin=body)
if proc.returncode == 0:
break
error = self._error(proc)
retry = any(word in error.lower() for word in (*_NOT_FOUND, "already exists", "conflict", "unavailable"))
if not retry or time.monotonic() + backoff > deadline:
raise RuntimeError(f"egress policy for {atespace}/{actor} not written: {error}")
time.sleep(backoff)
backoff = min(backoff * 2, 5.0)
stored = self.get(actor, atespace)
if stored is None or rules_of(stored) != rules_of(policy):
raise RuntimeError(f"egress policy for {atespace}/{actor} reads back {stored!r}, not {dict(policy)!r}")
def delete(self, actor: str, atespace: str) -> str | None:
"""After ``ax delete task``: None once the policy is gone with its actor, else why it is still there."""
try:
stored = self.get(actor, atespace)
except RuntimeError as exc:
return str(exc)
return None if stored is None else f"egress policy for {atespace}/{actor} outlived its actor"
@staticmethod
def settle() -> None:
"""Wait out the gateway's cached deny (10 s) before the actor's first connection: ``OUSAST_EGRESS_SETTLE``."""
time.sleep(float(os.environ.get("OUSAST_EGRESS_SETTLE") or 11))
class Receiver(ThreadingHTTPServer):
"""Accepts one tar per running task and extracts it under ``<run dir>/<task>/``."""
daemon_threads = True
def __init__(self, run_name: str, base: Path, port: int) -> None:
super().__init__(("0.0.0.0", port), _Handler)
self.run_name, self.base, self.running, self.delivered = run_name, base, set[str](), set[str]()
self.lock = threading.Lock()
self.url, self.dial = receiver_address(self.server_address[1])
def accept(self, headers: Any, body: bytes) -> tuple[int, str]:
task = headers.get("X-Ousast-Task", "")
if headers.get("X-Ousast-Run") != self.run_name or task not in self.running:
return 403, "unknown run or task"
if not str(headers.get("Content-Type", "")).startswith("application/x-tar"):
return 415, "expected application/x-tar"
dest = (self.base / task).resolve()
try:
with tarfile.open(fileobj=io.BytesIO(body), mode="r:") as tar:
members = tar.getmembers()
for m in members:
if m.issym() or m.islnk() or not (dest / m.name).resolve().is_relative_to(dest):
return 400, f"rejected member {m.name}"
dest.mkdir(parents=True, exist_ok=True)
tar.extractall(dest, members) # every member checked above
except tarfile.TarError as exc:
return 400, f"bad tar: {exc}"
with self.lock:
self.delivered.add(task)
return 200, "ok"
class _Handler(BaseHTTPRequestHandler):
def do_POST(self) -> None:
server: Receiver = self.server # type: ignore[assignment]
body = self.rfile.read(int(self.headers.get("Content-Length") or 0))
code, text = server.accept(self.headers, body)
self.send_response(code)
self.send_header("Content-Type", "text/plain")
self.end_headers()
self.wfile.write(text.encode())
def log_message(self, format: str, *args: object) -> None:
return
+16
View File
@@ -24,6 +24,7 @@ import yaml
__all__ = [ __all__ = [
"AX_API_VERSION", "AX_API_VERSION",
"AX_PROVIDERS", "AX_PROVIDERS",
"EGRESS_HOSTS_ANNOTATION",
"EXTENSION_PROVIDERS", "EXTENSION_PROVIDERS",
"GIT_COMMITS_ANNOTATION", "GIT_COMMITS_ANNOTATION",
"PROVIDER_EXTENSION_ANNOTATION", "PROVIDER_EXTENSION_ANNOTATION",
@@ -55,6 +56,8 @@ AX_API_VERSION = "ax.io/v1alpha1"
RUN_API_VERSION = "openultrasast.io/v1alpha1" RUN_API_VERSION = "openultrasast.io/v1alpha1"
PROVIDER_EXTENSION_ANNOTATION = "openultrasast.io/provider-extension" PROVIDER_EXTENSION_ANNOTATION = "openultrasast.io/provider-extension"
GIT_COMMITS_ANNOTATION = "openultrasast.io/git-commits" GIT_COMMITS_ANNOTATION = "openultrasast.io/git-commits"
EGRESS_HOSTS_ANNOTATION = "openultrasast.io/egress-hosts" # the Model API hosts a bound task may reach
_DNS_NAME = re.compile(r"^(?=.{1,253}$)[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?(\.[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?)*$")
AX_PROVIDERS = frozenset({"google", "anthropic"}) AX_PROVIDERS = frozenset({"google", "anthropic"})
EXTENSION_PROVIDERS = frozenset({"deepseek"}) EXTENSION_PROVIDERS = frozenset({"deepseek"})
@@ -168,6 +171,15 @@ class Model:
secret_key: SecretKey | None = None secret_key: SecretKey | None = None
parameters: Mapping[str, Any] = field(default_factory=dict) parameters: Mapping[str, Any] = field(default_factory=dict)
@property
def egress_hosts(self) -> tuple[str, ...]:
"""DNS names from ``metadata.annotations["openultrasast.io/egress-hosts"]`` (comma-separated)."""
return _split_hosts(self.metadata.annotations.get(EGRESS_HOSTS_ANNOTATION, ""))
def _split_hosts(text: str) -> tuple[str, ...]:
return tuple(h.strip() for h in text.split(",") if h.strip())
@dataclass(frozen=True) @dataclass(frozen=True)
class Budget: class Budget:
@@ -426,6 +438,10 @@ def _parse_model(check: _Check, metadata: Metadata, spec: Mapping[str, Any]) ->
) )
elif provider not in AX_PROVIDERS: elif provider not in AX_PROVIDERS:
raise check.fail("spec.provider", f'"{provider}" is not one of {", ".join(sorted(AX_PROVIDERS | EXTENSION_PROVIDERS))}') raise check.fail("spec.provider", f'"{provider}" is not one of {", ".join(sorted(AX_PROVIDERS | EXTENSION_PROVIDERS))}')
for host in _split_hosts(metadata.annotations.get(EGRESS_HOSTS_ANNOTATION, "")):
if not _DNS_NAME.match(host) or host.replace(".", "").isdigit():
path = f'metadata.annotations["{EGRESS_HOSTS_ANNOTATION}"]'
raise check.fail(path, f"{host!r} is not a lowercase DNS name (no wildcard, IP address, port or scheme)")
secret = check.optional_mapping(spec, "spec", "secretKey", ("name", "key")) secret = check.optional_mapping(spec, "spec", "secretKey", ("name", "key"))
parameters = spec.get("parameters") or {} parameters = spec.get("parameters") or {}
if not isinstance(parameters, dict): if not isinstance(parameters, dict):
+30 -68
View File
@@ -12,40 +12,39 @@ vanishing, a refused start or ``OUSAST_TASK_TIMEOUT`` s (7200) without delivery
ax rejects unknown fields, so rendered documents carry only ax's fields; ``openultrasast.io/model`` reaches the ax rejects unknown fields, so rendered documents carry only ax's fields; ``openultrasast.io/model`` reaches the
task as ``OUSAST_MODEL`` plus ``OUSAST_MODEL_PARAMS`` (Req 7.1), each Workspace's ``openultrasast.io/git-commits`` task as ``OUSAST_MODEL`` plus ``OUSAST_MODEL_PARAMS`` (Req 7.1), each Workspace's ``openultrasast.io/git-commits``
as ``OUSAST_GIT_PINS``; ``OUSAST_INPUT_<NAME>`` points into ``/workspace/.ousast-in/<task>``. The receiver takes as ``OUSAST_GIT_PINS``; ``OUSAST_INPUT_<NAME>`` points into ``/workspace/.ousast-in/<task>``. The receiver takes
the runner's tar (``X-Ousast-Run``, ``X-Ousast-Task``) on ``OUSAST_ARTIFACT_PORT``, advertised as the runner's tar (``X-Ousast-Run``, ``X-Ousast-Task``) on ``OUSAST_ARTIFACT_PORT`` (18090), reached by its Service
``OUSAST_ARTIFACT_HOST`` (default ``172.17.0.1``, the docker bridge). State: ``~/ousast-results/plane/<run>/`` name through the egress gateway; between ``ax apply`` and resume the task's egress policy is written (``egress.py``).
(``OUSAST_RESULTS``), polled every ``OUSAST_POLL_SECONDS`` (3). State: ``~/ousast-results/plane/<run>/`` (``OUSAST_RESULTS``), polled every ``OUSAST_POLL_SECONDS`` (3).
""" """
from __future__ import annotations from __future__ import annotations
import contextlib import contextlib
import io
import json import json
import os import os
import queue import queue
import re import re
import subprocess import subprocess
import tarfile import sys
import threading import threading
import time import time
from collections.abc import Callable, Iterable, Mapping from collections.abc import Callable, Iterable, Mapping
from dataclasses import dataclass, field from dataclasses import dataclass, field
from datetime import UTC, datetime from datetime import UTC, datetime
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path from pathlib import Path
from typing import Any from typing import Any
from urllib.parse import urlsplit
import yaml import yaml
from .doctor import doctor from .doctor import doctor
from .egress import RECEIVER_PORT, Egress, Receiver, policy_for
from .manifests import AX_API_VERSION, Manifests, Model, Run, RunTask, Task, Workspace, load_manifests from .manifests import AX_API_VERSION, Manifests, Model, Run, RunTask, Task, Workspace, load_manifests
from .router import Router, credentials, open_router from .router import Router, credentials, open_router
MODEL_ANNOTATION = "openultrasast.io/model" MODEL_ANNOTATION = "openultrasast.io/model"
OUTPUT_ROOT = "/workspace/.ousast-out" OUTPUT_ROOT = "/workspace/.ousast-out"
INPUTS_ROOT = "/workspace/.ousast-in" INPUTS_ROOT = "/workspace/.ousast-in"
DEFAULT_ARTIFACT_HOST = "172.17.0.1"
TRANSIENT = re.compile(r"deadline ?exceeded|unavailable", re.IGNORECASE) # the golden-snapshot build outlives a resume TRANSIENT = re.compile(r"deadline ?exceeded|unavailable", re.IGNORECASE) # the golden-snapshot build outlives a resume
_ROW = "{:<28} {:<20} {:>7} {:>10} {:>10} {:>9} {:>10}" _ROW = "{:<28} {:<20} {:>7} {:>10} {:>10} {:>9} {:>10}"
_COLUMNS = ("calls", "prompt", "cache_hit", "output", "usd") _COLUMNS = ("calls", "prompt", "cache_hit", "output", "usd")
@@ -134,13 +133,15 @@ def _input_files(entry: RunTask, base: Path) -> list[dict[str, str]]:
return files return files
def render_task(run: Run, entry: RunTask, manifests: Manifests, base: Path, artifact_url: str) -> list[dict[str, Any]]: def render_task(
run: Run, entry: RunTask, manifests: Manifests, base: Path, artifact_url: str, dial: str | None = None
) -> list[dict[str, Any]]:
"""The documents ``ax apply`` receives for one Run task: Task, its Model, its Workspaces, the inputs Workspace.""" """The documents ``ax apply`` receives for one Run task: Task, its Model, its Workspaces, the inputs Workspace."""
task: Task = manifests.tasks[entry.task] task: Task = manifests.tasks[entry.task]
ax_name = _ax_name(run.metadata.name, entry.name) ax_name = _ax_name(run.metadata.name, entry.name)
env = {e.name: e.value for e in task.env} env = {e.name: e.value for e in task.env}
env.update(OUSAST_RUN=run.metadata.name, OUSAST_TASK=entry.name, OUSAST_OUTPUT_DIR=f"{OUTPUT_ROOT}/{entry.name}") env.update(OUSAST_RUN=run.metadata.name, OUSAST_TASK=entry.name, OUSAST_OUTPUT_DIR=f"{OUTPUT_ROOT}/{entry.name}")
env["OUSAST_ARTIFACT_URL"] = artifact_url env.update(OUSAST_ARTIFACT_URL=artifact_url, **({"OUSAST_ARTIFACT_DIAL": dial} if dial else {}))
budget = _fields(entry.budget) if entry.budget else {} budget = _fields(entry.budget) if entry.budget else {}
env.update({f"OUSAST_BUDGET_{k.upper()}": str(v) for k, v in budget.items()}) env.update({f"OUSAST_BUDGET_{k.upper()}": str(v) for k, v in budget.items()})
for input_name, ref in entry.inputs.items(): for input_name, ref in entry.inputs.items():
@@ -214,55 +215,6 @@ class Ax:
self._run("delete", "task", name) self._run("delete", "task", name)
class Receiver(ThreadingHTTPServer):
"""Accepts one tar per running task and extracts it under ``<run dir>/<task>/``."""
daemon_threads = True
def __init__(self, run_name: str, base: Path, port: int) -> None:
super().__init__(("0.0.0.0", port), _Handler)
self.run_name, self.base, self.running, self.delivered = run_name, base, set[str](), set[str]()
self.lock = threading.Lock()
def url(self) -> str:
return f"http://{os.environ.get('OUSAST_ARTIFACT_HOST') or DEFAULT_ARTIFACT_HOST}:{self.server_address[1]}/"
def accept(self, headers: Any, body: bytes) -> tuple[int, str]:
task = headers.get("X-Ousast-Task", "")
if headers.get("X-Ousast-Run") != self.run_name or task not in self.running:
return 403, "unknown run or task"
if not str(headers.get("Content-Type", "")).startswith("application/x-tar"):
return 415, "expected application/x-tar"
dest = (self.base / task).resolve()
try:
with tarfile.open(fileobj=io.BytesIO(body), mode="r:") as tar:
members = tar.getmembers()
for m in members:
if m.issym() or m.islnk() or not (dest / m.name).resolve().is_relative_to(dest):
return 400, f"rejected member {m.name}"
dest.mkdir(parents=True, exist_ok=True)
tar.extractall(dest, members) # every member checked above
except tarfile.TarError as exc:
return 400, f"bad tar: {exc}"
with self.lock:
self.delivered.add(task)
return 200, "ok"
class _Handler(BaseHTTPRequestHandler):
def do_POST(self) -> None:
server: Receiver = self.server # type: ignore[assignment]
body = self.rfile.read(int(self.headers.get("Content-Length") or 0))
code, text = server.accept(self.headers, body)
self.send_response(code)
self.send_header("Content-Type", "text/plain")
self.end_headers()
self.wfile.write(text.encode())
def log_message(self, format: str, *args: object) -> None:
return
# --- the run ------------------------------------------------------------------------------------------------------- # --- the run -------------------------------------------------------------------------------------------------------
@@ -347,35 +299,45 @@ def _await(ax: Ax, name: str, delivered: Callable[[], bool], start: Callable[[],
return None return None
def _execute(ax: Ax, run: Run, entry: RunTask, manifests: Manifests, base: Path, receiver: Receiver, router: Router) -> tuple[str, str]: def _execute(
ax: Ax, egress: Egress, run: Run, entry: RunTask, manifests: Manifests, base: Path, receiver: Receiver, router: Router
) -> tuple[str, str]:
ax_name, task = _ax_name(run.metadata.name, entry.name), manifests.tasks[entry.task] ax_name, task = _ax_name(run.metadata.name, entry.name), manifests.tasks[entry.task]
manifest = base / entry.name / "task.yaml" atespace, manifest = task.metadata.atespace or "default", base / entry.name / "task.yaml"
manifest.parent.mkdir(parents=True, exist_ok=True) manifest.parent.mkdir(parents=True, exist_ok=True)
docs = render_task(run, entry, manifests, base, receiver.url()) docs = render_task(run, entry, manifests, base, receiver.url, receiver.dial)
manifest.write_text(yaml.safe_dump_all(docs, sort_keys=False), encoding="utf-8") manifest.write_text(yaml.safe_dump_all(docs, sort_keys=False), encoding="utf-8")
model = manifests.models.get(task.metadata.annotations.get(MODEL_ANNOTATION) or "")
policy = policy_for(task, [manifests.workspaces[b.name] for b in task.workspaces], model, urlsplit(receiver.url).hostname)
try: # Req 4.6: the bound Model's credential is read here and only ever sent in the start request try: # Req 4.6: the bound Model's credential is read here and only ever sent in the start request
secrets = credentials(manifests.models.get(task.metadata.annotations.get(MODEL_ANNOTATION) or "")) start = router.starter(atespace, ax_name, run.metadata.name, credentials(model))
start = router.starter(task.metadata.atespace or "default", ax_name, run.metadata.name, secrets) ax.apply(manifest) # creates the Substrate actor; its egress is denied until the policy exists
ax.apply(manifest) egress.apply(ax_name, atespace, policy)
egress.settle()
except RuntimeError as exc: # StartError included: a Model-bound task whose secret is unset except RuntimeError as exc: # StartError included: a Model-bound task whose secret is unset
ax.delete(ax_name)
return "failed", str(exc) return "failed", str(exc)
try: try:
failure = _await(ax, ax_name, lambda: entry.name in receiver.delivered, start) failure = _await(ax, ax_name, lambda: entry.name in receiver.delivered, start)
finally: finally:
ax.delete(ax_name) ax.delete(ax_name)
leftover = egress.delete(ax_name, atespace) # the policy goes with its actor; a survivor is reported
if leftover:
print(leftover, file=sys.stderr)
return ("failed", failure) if failure else outcome_of(_read_json(base / entry.name / "summary.json")) return ("failed", failure) if failure else outcome_of(_read_json(base / entry.name / "summary.json"))
def run(run_manifest: Path, *, workers: int = 1, ax: str = "ax", results_root: Path | None = None) -> str: def run(run_manifest: Path, *, workers: int = 1, ax: str = "ax", results_root: Path | None = None, kubectl_ate: str | None = None) -> str:
"""Execute a Run on ax and return its status: ``done``, ``unfinished`` or ``failed`` (Req 3.1-3.3, 2.2).""" """Execute a Run on ax and return its status: ``done``, ``unfinished`` or ``failed`` (Req 3.1-3.3, 2.2)."""
run_spec, manifests = load_run(Path(run_manifest)) run_spec, manifests = load_run(Path(run_manifest))
base = run_dir(run_spec.metadata.name, results_root) base = run_dir(run_spec.metadata.name, results_root)
base.mkdir(parents=True, exist_ok=True) base.mkdir(parents=True, exist_ok=True)
lock = _acquire_lock(base) lock = _acquire_lock(base)
state = _State(base / "state.json", _read_json(base / "state.json") or {"run": run_spec.metadata.name, "started": _now(), "tasks": {}}) state = _State(base / "state.json", _read_json(base / "state.json") or {"run": run_spec.metadata.name, "started": _now(), "tasks": {}})
receiver = Receiver(run_spec.metadata.name, base, int(os.environ.get("OUSAST_ARTIFACT_PORT") or 0)) port = os.environ.get("OUSAST_ARTIFACT_PORT") or (0 if os.environ.get("OUSAST_ARTIFACT_HOST") else RECEIVER_PORT)
receiver = Receiver(run_spec.metadata.name, base, int(port)) # the receiver Service targets RECEIVER_PORT
threading.Thread(target=receiver.serve_forever, daemon=True).start() threading.Thread(target=receiver.serve_forever, daemon=True).start()
cli, finished = Ax(ax), queue.Queue[tuple[str, str, str]]() cli, egress, finished = Ax(ax), Egress(kubectl_ate), queue.Queue[tuple[str, str, str]]()
pending = [t for t in run_spec.tasks if state.status(t.name) != "done"] pending = [t for t in run_spec.tasks if state.status(t.name) != "done"]
for entry in run_spec.tasks: for entry in run_spec.tasks:
state.set(entry.name, **({"status": "pending"} if entry in pending else {})) state.set(entry.name, **({"status": "pending"} if entry in pending else {}))
@@ -386,7 +348,7 @@ def run(run_manifest: Path, *, workers: int = 1, ax: str = "ax", results_root: P
if state.status(entry.name) in ("running", "suspended"): if state.status(entry.name) in ("running", "suspended"):
cli.delete(_ax_name(run_spec.metadata.name, entry.name)) # an interrupted attempt; ax must not keep it cli.delete(_ax_name(run_spec.metadata.name, entry.name)) # an interrupted attempt; ax must not keep it
state.set(entry.name, status="running", started=_now(), finished=None) state.set(entry.name, status="running", started=_now(), finished=None)
finished.put((entry.name, *_execute(cli, run_spec, entry, manifests, base, receiver, router))) finished.put((entry.name, *_execute(cli, egress, run_spec, entry, manifests, base, receiver, router)))
try: try:
with open_router() as router: # one port-forward to Substrate's router for the whole run with open_router() as router: # one port-forward to Substrate's router for the whole run
+25 -6
View File
@@ -43,6 +43,8 @@ Environment the runner reads, all optional except the two ax variables:
OUSAST_CASE_CACHE a directory of git checkouts (``benchmarks/independent`` cache layout) OUSAST_CASE_CACHE a directory of git checkouts (``benchmarks/independent`` cache layout)
OUSAST_GIT_PINS JSON ``{"<workspace>/<git name>": "<40-hex commit>"}`` from the reconciler OUSAST_GIT_PINS JSON ``{"<workspace>/<git name>": "<40-hex commit>"}`` from the reconciler
OUSAST_ARTIFACT_URL, OUSAST_RUN where to deliver, and the run name carried in ``X-Ousast-Run`` OUSAST_ARTIFACT_URL, OUSAST_RUN where to deliver, and the run name carried in ``X-Ousast-Run``
OUSAST_ARTIFACT_DIAL ``host:port`` to connect to instead of the URL's host, which is sent as
``Host`` (the egress gateway decides on and resolves that name)
OUSAST_DELIVERY_BACKOFF seconds before the second delivery attempt, doubled per retry OUSAST_DELIVERY_BACKOFF seconds before the second delivery attempt, doubled per retry
The runner exports ``OUSAST_WORKSPACE_DIR`` (the first bound workspace) when the Task's env leaves it unset, The runner exports ``OUSAST_WORKSPACE_DIR`` (the first bound workspace) when the Task's env leaves it unset,
@@ -72,6 +74,7 @@ from dataclasses import dataclass
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path from pathlib import Path
from typing import Any from typing import Any
from urllib.parse import urlsplit
import yaml import yaml
@@ -632,11 +635,26 @@ def build_tar(output_dir: Path) -> bytes:
return buffer.getvalue() return buffer.getvalue()
def deliver(url: str, payload: bytes, headers: Mapping[str, str], *, attempts: int = DELIVERY_ATTEMPTS, backoff: float = 1.0) -> int: def deliver(
"""POST ``payload`` to ``url`` with ``headers``; retry with doubling backoff; return the 2xx status.""" url: str,
last = "no attempt made" payload: bytes,
headers: Mapping[str, str],
*,
attempts: int = DELIVERY_ATTEMPTS,
backoff: float = 1.0,
dial: str | None = None,
) -> int:
"""POST ``payload`` to ``url`` with ``headers``; retry with doubling backoff; return the 2xx status.
``dial`` (``host:port``): connect there instead and send the URL's host as ``Host``, as the actor must -- it
cannot resolve the receiver's cluster name, and the egress gateway decides on ``Host`` and resolves it itself.
"""
last, target, headers = "no attempt made", url, dict(headers)
if dial:
parts = urlsplit(url)
target, headers["Host"] = parts._replace(netloc=dial).geturl(), parts.netloc
for attempt in range(1, attempts + 1): for attempt in range(1, attempts + 1):
request = urllib.request.Request(url, data=payload, method="POST", headers=dict(headers)) request = urllib.request.Request(target, data=payload, method="POST", headers=headers)
try: try:
with urllib.request.urlopen(request, timeout=60) as response: # noqa: S310 - the reconciler's URL with urllib.request.urlopen(request, timeout=60) as response: # noqa: S310 - the reconciler's URL
status = int(response.status) status = int(response.status)
@@ -658,12 +676,13 @@ def _deliver_output(task: Task, output_dir: Path, environ: Mapping[str, str]) ->
url = environ.get("OUSAST_ARTIFACT_URL") url = environ.get("OUSAST_ARTIFACT_URL")
if not url: if not url:
return False return False
headers = {"Content-Type": "application/x-tar", "X-Ousast-Task": task.metadata.name} # the Run's task name (``OUSAST_TASK``) is what the receiver knows; the ax Task is named ``<run>-<task>``
headers = {"Content-Type": "application/x-tar", "X-Ousast-Task": environ.get("OUSAST_TASK") or task.metadata.name}
run = environ.get("OUSAST_RUN") run = environ.get("OUSAST_RUN")
if run: if run:
headers["X-Ousast-Run"] = run headers["X-Ousast-Run"] = run
backoff = float(environ.get("OUSAST_DELIVERY_BACKOFF", "1.0")) backoff = float(environ.get("OUSAST_DELIVERY_BACKOFF", "1.0"))
deliver(url, build_tar(output_dir), headers, backoff=backoff) deliver(url, build_tar(output_dir), headers, backoff=backoff, dial=environ.get("OUSAST_ARTIFACT_DIAL") or None)
return True return True
+48
View File
@@ -62,3 +62,51 @@ def _scratch_under_tmp_path(monkeypatch: pytest.MonkeyPatch, tmp_path_factory: p
scratch = tmp_path_factory.mktemp("scratch") scratch = tmp_path_factory.mktemp("scratch")
monkeypatch.setattr(tempfile, "tempdir", str(scratch)) monkeypatch.setattr(tempfile, "tempdir", str(scratch))
FAKE_KUBECTL_ATE = r'''#!{python}
"""Fake kubectl-ate for egress policies: an actor exists while ``applied/<actor>.json`` exists and is not gone
(the fake ax's record); its policy goes with it, as Substrate's store cascades. Every call is logged to ax.log."""
import json, sys, time
from pathlib import Path
HERE = Path(__file__).resolve().parent
argv = sys.argv[1:]
if argv[:1] == ["--context"]:
argv = argv[2:]
verb, actor, space = argv[0], argv[2], argv[argv.index("-a") + 1]
with (HERE / "ax.log").open("a") as log:
log.write(f"{time.monotonic():.4f} egress {verb} {actor}\n")
record = HERE / "applied" / f"{actor}.json"
alive = record.exists() and not json.loads(record.read_text()).get("gone")
store = HERE / "policies" / f"{space}_{actor}.json"
if not alive:
store.unlink(missing_ok=True)
sys.exit(print(f'Error: actor "{actor}" in atespace "{space}" not found', file=sys.stderr) or 1)
if verb == "get":
if not store.exists(): # kubectl-ate: no policy is a valid state, exit 0 with nothing on stdout
sys.exit(print(f'actor "{actor}" in atespace "{space}" has no egress policy', file=sys.stderr) or 0)
print(store.read_text())
elif verb in ("create", "update"):
body = json.loads(sys.stdin.read())
old = json.loads(store.read_text()) if store.exists() else None
if verb == "create" and old is not None:
sys.exit(print("Error: already exists", file=sys.stderr) or 1)
version = 1 if old is None else old["metadata"]["version"] + 1
body["metadata"] = {"name": "default", "atespace": space, "uid": "uid-1", "version": version}
store.parent.mkdir(exist_ok=True)
store.write_text(json.dumps(body))
(HERE / "policies" / f"{space}_{actor}.written.json").write_text(json.dumps(body)) # survives the cascade
print(f"egress policy {verb}d")
'''
@pytest.fixture
def kubectl_ate(tmp_path, monkeypatch: pytest.MonkeyPatch):
"""A fake ``kubectl-ate`` in ``tmp_path`` (shared with the fake ax there); no settle wait."""
path = tmp_path / "kubectl-ate"
path.write_text(FAKE_KUBECTL_ATE.replace("{python}", sys.executable), encoding="utf-8")
path.chmod(0o755)
monkeypatch.setenv("OUSAST_KUBECTL_ATE", str(path))
monkeypatch.setenv("OUSAST_EGRESS_SETTLE", "0")
return path
+199
View File
@@ -0,0 +1,199 @@
"""ai-service-plane task 0: the per-task egress policy, its kubectl-ate calls, and the runner's dialled delivery.
Agent Substrate's egress gateway denies by default; :func:`egress.policy_for` must grant exactly the receiver (HTTP
by name), the bound Workspaces' Git hosts and the bound Model's API host (TLS passthrough), and nothing a gateway
would reject or that would intercept TLS: no IP address, no wildcard, no ``https`` rule.
"""
from __future__ import annotations
import ipaddress
import json
import threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
import pytest
from openultrasast.plane import egress, runner
from openultrasast.plane.manifests import ManifestError, load_manifests, parse_manifest
MANIFESTS = """
apiVersion: ax.io/v1alpha1
kind: Model
metadata:
name: deepseek-flash
annotations: {openultrasast.io/provider-extension: "true", openultrasast.io/egress-hosts: api.deepseek.com}
spec: {provider: deepseek, model: deepseek-flash, secretKey: {name: deepseek, key: DEEPSEEK_API_KEY}}
---
apiVersion: ax.io/v1alpha1
kind: Workspace
metadata: {name: cases}
spec:
git:
- {name: a, repo: https://github.com/example-org/alpha}
- {name: b, repo: https://github.com/example-org/beta.git}
- {name: c, repo: "[email protected]:group/repo.git"}
- {name: d, repo: https://10.0.0.7/mirror/repo.git}
---
apiVersion: ax.io/v1alpha1
kind: Task
metadata: {name: verify, annotations: {openultrasast.io/model: deepseek-flash}}
spec: {command: [verify], workspaces: [{name: cases, path: /workspace}]}
---
apiVersion: ax.io/v1alpha1
kind: Task
metadata: {name: facts}
spec: {command: [repo-facts]}
"""
@pytest.fixture
def manifests(tmp_path: Path):
path = tmp_path / "m.yaml"
path.write_text(MANIFESTS, encoding="utf-8")
return load_manifests([path])
def hostnames(policy: dict) -> list[str]:
return [h for rule in policy["rules"] for body in rule.values() for h in body["hostnames"]]
def assert_tight(policy: dict) -> None:
for rule in policy["rules"]:
assert set(rule) <= {"http", "tlsPassthrough"}, f"only http and passthrough rules, got {rule}"
body = next(iter(rule.values()))
assert "all" not in body["ports"], "never every port"
for host in hostnames(policy):
assert "*" not in host, f"no wildcard: {host}"
with pytest.raises(ValueError):
ipaddress.ip_address(host)
def test_model_free_task_gets_only_the_receiver(manifests) -> None:
policy = egress.policy_for(manifests.tasks["facts"], [], None, egress.RECEIVER_HOST)
assert policy == {"rules": [{"http": {"hostnames": [egress.RECEIVER_HOST], "ports": {"numbers": [80]}}}]}
assert_tight(policy)
def test_model_bound_task_with_git_workspaces(manifests) -> None:
task = manifests.tasks["verify"]
policy = egress.policy_for(task, [manifests.workspaces["cases"]], manifests.models["deepseek-flash"], egress.RECEIVER_HOST)
assert policy["rules"] == [
{"http": {"hostnames": [egress.RECEIVER_HOST], "ports": {"numbers": [80]}}},
{"tlsPassthrough": {"hostnames": ["api.deepseek.com", "github.com"], "ports": {"numbers": [443]}}},
], "one github.com for two repos; the ssh and IP-address repos add nothing"
assert_tight(policy)
def test_an_ip_receiver_is_not_granted(manifests) -> None:
assert egress.policy_for(manifests.tasks["facts"], [], None, "127.0.0.1") == {"rules": []}
def test_deepseek_model_manifest_names_the_endpoint_host() -> None:
from openultrasast.model.endpoint import DEEPSEEK_BASE_URL
model = load_manifests([Path("plane/models/deepseek-flash.yaml")]).models["deepseek-flash"]
assert model.egress_hosts == (DEEPSEEK_BASE_URL.removeprefix("https://"),)
@pytest.mark.parametrize("hosts", ["*.deepseek.com", "10.1.2.3", "https://api.deepseek.com", "api.deepseek.com:443", "API.deepseek.com"])
def test_egress_hosts_annotation_is_validated(hosts: str) -> None:
doc = {
"apiVersion": "ax.io/v1alpha1",
"kind": "Model",
"metadata": {"name": "m", "annotations": {"openultrasast.io/provider-extension": "true", "openultrasast.io/egress-hosts": hosts}},
"spec": {"provider": "deepseek", "model": "x"},
}
with pytest.raises(ManifestError, match="egress-hosts"):
parse_manifest(doc)
# --- kubectl-ate ---------------------------------------------------------------------------------------------------
def actor(tmp_path: Path, name: str, gone: bool = False) -> None:
(tmp_path / "applied").mkdir(exist_ok=True)
(tmp_path / "applied" / f"{name}.json").write_text(json.dumps({"gone": gone}), encoding="utf-8")
def calls(tmp_path: Path) -> list[str]:
return [" ".join(line.split()[2:4]) for line in (tmp_path / "ax.log").read_text().splitlines()]
POLICY = {"rules": [{"http": {"hostnames": [egress.RECEIVER_HOST], "ports": {"numbers": [80]}}}]}
def test_apply_creates_reads_back_and_delete_confirms_the_cascade(tmp_path: Path, kubectl_ate: Path) -> None:
cli = egress.Egress(str(kubectl_ate), context="kind-test")
actor(tmp_path, "t1")
cli.apply("t1", "default", POLICY)
stored = cli.get("t1", "default")
assert stored is not None and egress.rules_of(stored) == egress.rules_of(POLICY)
assert cli.delete("t1", "default") == "egress policy for default/t1 outlived its actor"
actor(tmp_path, "t1", gone=True) # ax delete task: the actor, and with it the policy, is gone
assert cli.delete("t1", "default") is None
assert calls(tmp_path) == ["get t1", "create t1", "get t1", "get t1", "get t1", "get t1"]
def test_apply_replaces_an_existing_policy(tmp_path: Path, kubectl_ate: Path) -> None:
cli = egress.Egress(str(kubectl_ate))
actor(tmp_path, "t2")
cli.apply("t2", "default", {"rules": []})
cli.apply("t2", "default", POLICY)
assert calls(tmp_path) == ["get t2", "create t2", "get t2", "get t2", "update t2", "get t2"]
assert json.loads((tmp_path / "policies" / "default_t2.json").read_text())["metadata"]["version"] == 2
def test_apply_waits_for_the_actor_then_gives_up(tmp_path: Path, kubectl_ate: Path) -> None:
cli = egress.Egress(str(kubectl_ate))
with pytest.raises(RuntimeError, match="not found"):
cli.apply("missing", "default", POLICY, timeout=0.2)
timer = threading.Timer(0.3, actor, args=(tmp_path, "late"))
timer.start()
cli.apply("late", "default", POLICY, timeout=10)
timer.join()
def test_receiver_address_modes(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("OUSAST_ARTIFACT_HOST", "127.0.0.1")
assert egress.receiver_address(4242) == ("http://127.0.0.1:4242/", None)
monkeypatch.delenv("OUSAST_ARTIFACT_HOST")
monkeypatch.setenv("OUSAST_ARTIFACT_DIAL", "172.19.0.1:80")
assert egress.receiver_address(18090) == (f"http://{egress.RECEIVER_HOST}/", "172.19.0.1:80")
# --- the runner dials the gateway address with the receiver's name as Host ----------------------------------------
class _Seen(ThreadingHTTPServer):
daemon_threads = True
seen: list[tuple[str, str | None, bytes]]
class _Handler(BaseHTTPRequestHandler):
server: _Seen
def do_POST(self) -> None: # noqa: N802
body = self.rfile.read(int(self.headers.get("Content-Length") or 0))
self.server.seen.append((self.path, self.headers.get("Host"), body))
self.send_response(200)
self.send_header("Content-Length", "0")
self.end_headers()
def log_message(self, format: str, *args: object) -> None: # noqa: A002
pass
def test_delivery_dials_the_override_with_the_url_host() -> None:
server = _Seen(("127.0.0.1", 0), _Handler)
server.seen = []
threading.Thread(target=server.serve_forever, daemon=True).start()
try:
dial = f"127.0.0.1:{server.server_address[1]}"
url = f"http://{egress.RECEIVER_HOST}/in"
assert runner.deliver(url, b"tar", {"Content-Type": "application/x-tar"}, backoff=0, dial=dial) == 200
assert server.seen == [("/in", egress.RECEIVER_HOST, b"tar")]
finally:
server.shutdown()
server.server_close()
+35 -2
View File
@@ -156,7 +156,7 @@ apiVersion: ax.io/v1alpha1
kind: Model kind: Model
metadata: metadata:
name: deepseek-flash name: deepseek-flash
annotations: {openultrasast.io/provider-extension: "true"} annotations: {openultrasast.io/provider-extension: "true", openultrasast.io/egress-hosts: api.deepseek.com}
spec: spec:
provider: deepseek provider: deepseek
model: deepseek-chat model: deepseek-chat
@@ -269,6 +269,8 @@ class Fake:
out.append((stamp, Path(words[2]).parent.name)) out.append((stamp, Path(words[2]).parent.name))
elif verb in ("delete", "resume") and words[:2] == [verb, "task"]: elif verb in ("delete", "resume") and words[:2] == [verb, "task"]:
out.append((stamp, words[2])) out.append((stamp, words[2]))
elif verb == "egress" and words[0] == verb:
out.append((stamp, f"{words[1]} {words[2]}"))
elif verb in ("delivered", "rejected") and words[0] == verb: elif verb in ("delivered", "rejected") and words[0] == verb:
out.append((stamp, words[1])) out.append((stamp, words[1]))
return out return out
@@ -313,7 +315,7 @@ class _RouterHandler(BaseHTTPRequestHandler):
@pytest.fixture @pytest.fixture
def fake(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Iterator[Fake]: def fake(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, kubectl_ate: Path) -> Iterator[Fake]:
monkeypatch.setenv("OUSAST_RESULTS", str(tmp_path / "results")) monkeypatch.setenv("OUSAST_RESULTS", str(tmp_path / "results"))
monkeypatch.setenv("OUSAST_ARTIFACT_HOST", "127.0.0.1") monkeypatch.setenv("OUSAST_ARTIFACT_HOST", "127.0.0.1")
monkeypatch.setenv("OUSAST_POLL_SECONDS", "0.02") monkeypatch.setenv("OUSAST_POLL_SECONDS", "0.02")
@@ -747,3 +749,34 @@ def test_doctor_treats_an_empty_namespace_as_not_ready(monkeypatch: pytest.Monke
assert doctor._pods_ready("ns", "ctx") == (True, "2 pods ready") assert doctor._pods_ready("ns", "ctx") == (True, "2 pods ready")
monkeypatch.setattr(doctor, "_sh", lambda *a: (True, "a-1 0/1 Running 0 1m")) monkeypatch.setattr(doctor, "_sh", lambda *a: (True, "a-1 0/1 Running 0 1m"))
assert doctor._pods_ready("ns", "ctx")[0] is False assert doctor._pods_ready("ns", "ctx")[0] is False
# --- egress (task 0: the per-task policy) --------------------------------------------------------------------------
def test_egress_policy_between_apply_and_resume_and_checked_after_delete(fake: Fake, tmp_path: Path) -> None:
"""apply -> policy -> resume -> start -> delivery -> delete task -> the policy is gone with its actor."""
assert reconciler.run(write_run(tmp_path, "eg"), ax=str(fake.ax)) == "done"
lines = [" ".join(words) for _, words in fake.log()]
def first(prefix: str, after: int = 0) -> int:
return next(i for i, line in enumerate(lines) if i >= after and line.startswith(prefix))
order = [first("apply -f " + str(reconciler.run_dir("eg") / "verify"))]
steps = ("egress create eg-verify", "resume task eg-verify", "started eg-verify 202", "delivered verify", "delete task eg-verify")
for prefix in (*steps, "egress get eg-verify"): # the last one: the check after the delete
order.append(first(prefix, order[-1] + 1))
assert order == sorted(order)
assert not (tmp_path / "policies" / "default_eg-verify.json").exists(), "the policy went with its actor"
written = json.loads((tmp_path / "policies" / "default_eg-verify.written.json").read_text())
tls = {"hostnames": ["api.deepseek.com", "example.invalid"], "ports": {"numbers": [443]}}
assert written["rules"] == [{"tlsPassthrough": tls}], "the Model host and the Git host, nothing else"
facts = json.loads((tmp_path / "policies" / "default_eg-repo-facts.written.json").read_text())
assert facts["rules"] == [{"tlsPassthrough": {"hostnames": ["example.invalid"], "ports": {"numbers": [443]}}}]
def test_a_refused_egress_policy_fails_the_task_before_resume(fake: Fake, tmp_path: Path, kubectl_ate: Path) -> None:
kubectl_ate.write_text("#!/bin/sh\necho 'Error: permission denied' >&2\nexit 1\n", encoding="utf-8")
assert reconciler.run(write_run(tmp_path, "egfail"), ax=str(fake.ax)) == "failed"
assert "permission denied" in state_of("egfail")["tasks"]["repo-facts"]["reason"]
assert fake.events("resume") == [] and [n for _, n in fake.events("delete")] == ["egfail-repo-facts"]
+4
View File
@@ -244,6 +244,10 @@ def test_output_dir_is_delivered_as_a_tar_with_the_headers(
assert headers["Content-Type"] == "application/x-tar" assert headers["Content-Type"] == "application/x-tar"
assert headers["X-Ousast-Task"] == "stub" assert headers["X-Ousast-Task"] == "stub"
assert headers["X-Ousast-Run"] == "validation-46" assert headers["X-Ousast-Run"] == "validation-46"
receiver.posts.clear()
again = {"OUSAST_ARTIFACT_URL": receiver.url, "OUSAST_TASK": "facts", "OUSAST_STATE_DIR": str(tmp_path / "again-state")}
code, _ = run_main(monkeypatch, tmp_path / "again", files_workspace_yaml(), **again)
assert code == 0 and receiver.posts[0][0]["X-Ousast-Task"] == "facts", "the Run's task name, not the ax Task's"
with tarfile.open(fileobj=io.BytesIO(body), mode="r:") as archive: with tarfile.open(fileobj=io.BytesIO(body), mode="r:") as archive:
names = sorted(archive.getnames()) names = sorted(archive.getnames())
summary = archive.extractfile("summary.json") summary = archive.extractfile("summary.json")