fix(plane): ax field names verbatim; pins and bindings as stripped annotations

Requirement 1.1 (Task/Workspace/Model use only fields ax documents) and
Requirement 1.4 (no ax field repurposed) were violated: manifests accepted
metadata.namespace and a Workspace git `commit`, and the reconciler dumped
every git field into what `ax apply` receives.

Facts from ax's source (pkg/apis/v1alpha1/ax.proto, internal/workspace/setup.go):
- ObjectMeta is `name` and `atespace` (plus server-set creation_timestamp);
  no `namespace`, no `annotations`.
- GitRepo is `name`, `repo`, `branch`, `dir`, `depth`; no `commit`.
- The API server decodes with protojson.UnmarshalOptions{}: an unknown field
  is an error at `ax apply` (confirmed on the live cluster: a rendered
  Workspace+Model applied cleanly; the same Workspace with `commit` was
  refused with `unknown field "commit"`).
- ValidateModel checks only metadata; any spec.provider string is accepted.

Changes:
- manifests: metadata takes `name`, `atespace`, `annotations` (our extension,
  never sent to ax); git entries take exactly ax's five fields (depth int);
  `commit` in spec.git is rejected with a pointer to the new Workspace
  annotation `openultrasast.io/git-commits: "<git name>=<40-hex>,..."`,
  validated for unknown git names and non-hex shas.
- reconciler: rendered docs carry only ax fields; each bound Workspace's pins
  reach the Task as env OUSAST_GIT_PINS = {"<workspace>/<git name>": "<sha>"}.
  A test walks every rendered doc against ax's field lists.
- runner: pins come from OUSAST_GIT_PINS; git is fetched as setup.go does
  (init/remote/fetch with --depth when depth > 0/checkout FETCH_HEAD, default
  branch main, cloneDestination's dir and placeholder-name rules), then the
  pin is checked out. A pinned entry without a branch fetches the commit.
- workspaces: emits the annotation, `dir: repo` and `depth: 1` instead of
  `commit`, so generated case workspaces keep <path>/repo at depth 1.
- design.md: ax field names verbatim; pins and model binding as stripped
  openultrasast.io/* annotations carried to the task by env.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-09-29 18:36:29 +02:00
co-authored by Claude Opus 5.5
parent e6d5124ffc
commit 18b67cf942
9 changed files with 450 additions and 81 deletions
+8
View File
@@ -42,6 +42,14 @@ container the same module is PID 1 (`ax-task-runner`), built as a target of the
are rejected with the path named. `Model.spec.provider` accepts ax's `google` and `anthropic`, and `deepseek` are rejected with the path named. `Model.spec.provider` accepts ax's `google` and `anthropic`, and `deepseek`
only when `metadata.annotations["openultrasast.io/provider-extension"] == "true"` (Requirement 1.4): the only when `metadata.annotations["openultrasast.io/provider-extension"] == "true"` (Requirement 1.4): the
deviation is declared, not hidden in another field. deviation is declared, not hidden in another field.
- Field names are ax's verbatim (`ax.proto`; its API server decodes with `protojson`, so an unknown field fails
`ax apply`): `metadata` is `name` and `atespace` (not `namespace`), and a Workspace `git` entry is `name`,
`repo`, `branch`, `dir`, `depth` (no `commit`; a `commit` there is rejected with a pointer to the annotation).
`metadata.annotations` is this project's only extension, and nothing under it reaches ax: the reconciler strips
it and carries it to the task as env. The Model binding (`openultrasast.io/model`) becomes `OUSAST_MODEL` and
`OUSAST_MODEL_PARAMS`. Commit pins (`openultrasast.io/git-commits: "<git name>=<40-hex sha>,..."` on a
Workspace) become `OUSAST_GIT_PINS` = `{"<workspace>/<git name>": "<sha>"}`, which the runner checks out after
fetching the way ax's `internal/workspace/setup.go` does, with the same `dir` and `depth` rules.
- `Run`: `apiVersion: openultrasast.io/v1alpha1`. `spec.tasks[]`: `name`, `task` (a Task manifest name), - `Run`: `apiVersion: openultrasast.io/v1alpha1`. `spec.tasks[]`: `name`, `task` (a Task manifest name),
`dependsOn[]`, `inputs{name: "<task>/<artifact>"}`, `outputs[]`, `budget{usd, calls}`, `serialize` (a label `dependsOn[]`, `inputs{name: "<task>/<artifact>"}`, `outputs[]`, `budget{usd, calls}`, `serialize` (a label
for tasks that must not overlap; engine-bearing tasks carry `engine`). The reconciler passes budgets and for tasks that must not overlap; engine-bearing tasks carry `engine`). The reconciler passes budgets and
+74 -14
View File
@@ -1,7 +1,11 @@
"""Manifests of the model-level service plane. """Manifests of the model-level service plane.
``Task``, ``Workspace`` and ``Model`` follow google/ax (``apiVersion: ax.io/v1alpha1``) and carry only the ``Task``, ``Workspace`` and ``Model`` follow google/ax (``apiVersion: ax.io/v1alpha1``) and carry only the
fields ax documents. ``Run`` (``apiVersion: openultrasast.io/v1alpha1``) is this project's kind for the task fields ax documents, under ax's own names (``metadata.atespace``, not ``namespace``). ``metadata.annotations``
is this project's extension: ax's ObjectMeta has no such field and its API server rejects unknown fields, so the
reconciler strips annotations before anything reaches ``ax apply``. They declare the provider extension and carry
the Model binding and the Workspace commit pins (``openultrasast.io/git-commits: "<git name>=<sha>,..."``), which
reach the task as env. ``Run`` (``apiVersion: openultrasast.io/v1alpha1``) is this project's kind for the task
graph, artifacts and budgets; it holds nothing a task needs to execute. Every schema violation raises graph, artifacts and budgets; it holds nothing a task needs to execute. Every schema violation raises
:class:`ManifestError`, whose message names the kind, the ``metadata.name`` and the offending field path :class:`ManifestError`, whose message names the kind, the ``metadata.name`` and the offending field path
(``Task/verify: spec.foo is not a field``), before anything runs. (``Task/verify: spec.foo is not a field``), before anything runs.
@@ -9,6 +13,7 @@ graph, artifacts and budgets; it holds nothing a task needs to execute. Every sc
from __future__ import annotations from __future__ import annotations
import re
from collections.abc import Iterable, Mapping from collections.abc import Iterable, Mapping
from dataclasses import dataclass, field from dataclasses import dataclass, field
from pathlib import Path from pathlib import Path
@@ -20,6 +25,7 @@ __all__ = [
"AX_API_VERSION", "AX_API_VERSION",
"AX_PROVIDERS", "AX_PROVIDERS",
"EXTENSION_PROVIDERS", "EXTENSION_PROVIDERS",
"GIT_COMMITS_ANNOTATION",
"PROVIDER_EXTENSION_ANNOTATION", "PROVIDER_EXTENSION_ANNOTATION",
"RUN_API_VERSION", "RUN_API_VERSION",
"Budget", "Budget",
@@ -48,6 +54,7 @@ __all__ = [
AX_API_VERSION = "ax.io/v1alpha1" AX_API_VERSION = "ax.io/v1alpha1"
RUN_API_VERSION = "openultrasast.io/v1alpha1" RUN_API_VERSION = "openultrasast.io/v1alpha1"
PROVIDER_EXTENSION_ANNOTATION = "openultrasast.io/provider-extension" PROVIDER_EXTENSION_ANNOTATION = "openultrasast.io/provider-extension"
GIT_COMMITS_ANNOTATION = "openultrasast.io/git-commits"
AX_PROVIDERS = frozenset({"google", "anthropic"}) AX_PROVIDERS = frozenset({"google", "anthropic"})
EXTENSION_PROVIDERS = frozenset({"deepseek"}) EXTENSION_PROVIDERS = frozenset({"deepseek"})
@@ -64,7 +71,7 @@ class ManifestError(ValueError):
@dataclass(frozen=True) @dataclass(frozen=True)
class Metadata: class Metadata:
name: str name: str
namespace: str | None = None atespace: str | None = None
annotations: Mapping[str, str] = field(default_factory=dict) annotations: Mapping[str, str] = field(default_factory=dict)
@@ -106,10 +113,13 @@ class Task:
@dataclass(frozen=True) @dataclass(frozen=True)
class GitSource: class GitSource:
"""ax's ``GitRepo``: exactly ``name``, ``repo``, ``branch``, ``dir``, ``depth`` (a commit pin is an annotation)."""
name: str name: str
repo: str repo: str
branch: str | None = None branch: str | None = None
commit: str | None = None dir: str | None = None
depth: int | None = None
@dataclass(frozen=True) @dataclass(frozen=True)
@@ -138,6 +148,11 @@ class Workspace:
mcp: McpSpec | None = None mcp: McpSpec | None = None
skills: SkillsSpec | None = None skills: SkillsSpec | None = None
@property
def pins(self) -> dict[str, str]:
"""Git entry name -> 40-hex commit, from ``metadata.annotations["openultrasast.io/git-commits"]``."""
return _split_pins(self.metadata.annotations.get(GIT_COMMITS_ANNOTATION, ""))
@dataclass(frozen=True) @dataclass(frozen=True)
class SecretKey: class SecretKey:
@@ -275,6 +290,12 @@ class _Check:
raise self.fail(_join(_join(path, key), str(item_key)), "must map a string to a non-empty string") raise self.fail(_join(_join(path, key), str(item_key)), "must map a string to a non-empty string")
return value return value
def optional_int(self, parent: Mapping[str, Any], path: str, key: str) -> int | None:
value = parent.get(key)
if value is not None and (isinstance(value, bool) or not isinstance(value, int) or value < 0):
raise self.fail(_join(path, key), "must be a non-negative integer")
return value
def boolean(self, parent: Mapping[str, Any], path: str, key: str) -> bool: def boolean(self, parent: Mapping[str, Any], path: str, key: str) -> bool:
value = parent.get(key, False) value = parent.get(key, False)
if not isinstance(value, bool): if not isinstance(value, bool):
@@ -286,10 +307,10 @@ class _Check:
def _parse_metadata(check: _Check, value: object) -> Metadata: def _parse_metadata(check: _Check, value: object) -> Metadata:
meta = check.mapping(value, "metadata", ("name", "namespace", "annotations")) meta = check.mapping(value, "metadata", ("name", "atespace", "annotations"))
return Metadata( return Metadata(
name=check.string(meta, "metadata", "name"), name=check.string(meta, "metadata", "name"),
namespace=check.optional_string(meta, "metadata", "namespace"), atespace=check.optional_string(meta, "metadata", "atespace"),
annotations=check.string_map(meta, "metadata", "annotations"), annotations=check.string_map(meta, "metadata", "annotations"),
) )
@@ -329,20 +350,59 @@ def _parse_task(check: _Check, metadata: Metadata, spec: Mapping[str, Any]) -> T
) )
_SHA = re.compile(r"[0-9a-f]{40}")
def _pin_items(value: str) -> list[str]:
return [item for item in (part.strip() for part in value.split(",")) if item]
def _split_pins(value: str) -> dict[str, str]:
"""``name=sha,name=sha`` -> ``{name: sha}`` (validated at parse time by :func:`_check_pins`)."""
pairs = (item.partition("=") for item in _pin_items(value))
return {name.strip(): sha.strip() for name, _, sha in pairs}
def _check_pins(check: _Check, metadata: Metadata, git: tuple[GitSource, ...]) -> None:
path = f'metadata.annotations["{GIT_COMMITS_ANNOTATION}"]'
names = {g.name for g in git}
for item in _pin_items(metadata.annotations.get(GIT_COMMITS_ANNOTATION, "")):
name, eq, sha = (part.strip() for part in item.partition("="))
if not eq or not name:
raise check.fail(path, f'item "{item}" must be "<git name>=<40-hex commit>"')
if name not in names:
raise check.fail(path, f'"{name}" names no spec.git entry')
if not _SHA.fullmatch(sha):
raise check.fail(path, f'"{name}={sha}" is not a 40-hex commit')
def _parse_git(check: _Check, spec: Mapping[str, Any]) -> tuple[GitSource, ...]:
for index, entry in enumerate(check.sequence(spec, "spec", "git")):
if isinstance(entry, dict) and "commit" in entry:
raise check.fail(
f"spec.git[{index}].commit",
f'is not an ax field; pin the commit in metadata.annotations["{GIT_COMMITS_ANNOTATION}"] as "<git name>=<sha>"',
)
return tuple(
GitSource(
check.string(g, p, "name"),
check.string(g, p, "repo"),
check.optional_string(g, p, "branch"),
check.optional_string(g, p, "dir"),
check.optional_int(g, p, "depth"),
)
for p, g in check.mappings(spec, "spec", "git", ("name", "repo", "branch", "dir", "depth"))
)
def _parse_workspace(check: _Check, metadata: Metadata, spec: Mapping[str, Any]) -> Workspace: def _parse_workspace(check: _Check, metadata: Metadata, spec: Mapping[str, Any]) -> Workspace:
mcp = check.optional_mapping(spec, "spec", "mcp", ("registries", "servers")) mcp = check.optional_mapping(spec, "spec", "mcp", ("registries", "servers"))
skills = check.optional_mapping(spec, "spec", "skills", ("registries", "path")) skills = check.optional_mapping(spec, "spec", "skills", ("registries", "path"))
git = _parse_git(check, spec)
_check_pins(check, metadata, git)
return Workspace( return Workspace(
metadata=metadata, metadata=metadata,
git=tuple( git=git,
GitSource(
check.string(g, p, "name"),
check.string(g, p, "repo"),
check.optional_string(g, p, "branch"),
check.optional_string(g, p, "commit"),
)
for p, g in check.mappings(spec, "spec", "git", ("name", "repo", "branch", "commit"))
),
files=tuple( files=tuple(
FileEntry(check.string(f, p, "path"), check.string(f, p, "content")) FileEntry(check.string(f, p, "path"), check.string(f, p, "content"))
for p, f in check.mappings(spec, "spec", "files", ("path", "content")) for p, f in check.mappings(spec, "spec", "files", ("path", "content"))
+13 -9
View File
@@ -6,9 +6,11 @@ file (the Task with its env extended, its Workspaces, one generated Workspace ca
``files`` entries, and the bound Model), runs ``ax apply -f``, polls ``ax get task`` until a terminal phase and ``files`` entries, and the bound Model), runs ``ax apply -f``, polls ``ax get task`` until a terminal phase and
``ax delete task`` afterwards. The outcome is read from the delivered ``summary.json``, never from the phase alone. ``ax delete task`` afterwards. The outcome is read from the delivered ``summary.json``, never from the phase alone.
Model binding: a Task names its Model in ``metadata.annotations["openultrasast.io/model"]`` (ax's ObjectMeta has no ax's ObjectMeta is ``name``/``atespace`` and its API server rejects unknown fields, so rendered documents carry only
annotations, so the rendered Task carries only ``name``/``atespace``); the task receives ``OUSAST_MODEL`` and the ax's fields; ``openultrasast.io/*`` annotations are stripped and reach the task as env. Model binding: a Task names
Model's ``spec.parameters`` as ``OUSAST_MODEL_PARAMS`` JSON (Req 7.1). Inputs: ``OUSAST_INPUT_<NAME>`` points into a its Model in ``openultrasast.io/model``; the task receives ``OUSAST_MODEL`` and the Model's ``spec.parameters`` as
``OUSAST_MODEL_PARAMS`` JSON (Req 7.1). Commit pins: each bound Workspace's ``openultrasast.io/git-commits`` becomes
``OUSAST_GIT_PINS`` = ``{"<workspace>/<git name>": "<sha>"}``. Inputs: ``OUSAST_INPUT_<NAME>`` points into a
generated Workspace bound at ``/workspace/.ousast-in/<task>`` (ax mounts every workspace under ``/workspace``). generated Workspace bound at ``/workspace/.ousast-in/<task>`` (ax mounts every workspace under ``/workspace``).
Artifacts: the runner POSTs an uncompressed tar of its output directory to ``OUSAST_ARTIFACT_URL`` with Artifacts: the runner POSTs an uncompressed tar of its output directory to ``OUSAST_ARTIFACT_URL`` with
``X-Ousast-Run`` and ``X-Ousast-Task`` headers; the receiver listens on ``0.0.0.0`` at ``OUSAST_ARTIFACT_PORT`` ``X-Ousast-Run`` and ``X-Ousast-Task`` headers; the receiver listens on ``0.0.0.0`` at ``OUSAST_ARTIFACT_PORT``
@@ -97,8 +99,8 @@ def load_run(run_manifest: Path) -> tuple[Run, Manifests]:
return next(iter(manifests.runs.values())), manifests return next(iter(manifests.runs.values())), manifests
def _doc(kind: str, name: str, spec: dict[str, Any], namespace: str | None = None) -> dict[str, Any]: def _doc(kind: str, name: str, spec: dict[str, Any], atespace: str | None = None) -> dict[str, Any]:
meta = {"name": name, **({"atespace": namespace} if namespace else {})} meta = {"name": name, **({"atespace": atespace} if atespace else {})}
return {"apiVersion": AX_API_VERSION, "kind": kind, "metadata": meta, "spec": spec} return {"apiVersion": AX_API_VERSION, "kind": kind, "metadata": meta, "spec": spec}
@@ -116,7 +118,7 @@ def _workspace_doc(ws: Workspace) -> dict[str, Any]:
spec["mcp"] = {"registries": list(ws.mcp.registries), "servers": list(ws.mcp.servers)} spec["mcp"] = {"registries": list(ws.mcp.registries), "servers": list(ws.mcp.servers)}
if ws.skills: if ws.skills:
spec["skills"] = {"registries": list(ws.skills.registries), **({"path": ws.skills.path} if ws.skills.path else {})} spec["skills"] = {"registries": list(ws.skills.registries), **({"path": ws.skills.path} if ws.skills.path else {})}
return _doc("Workspace", ws.metadata.name, spec, ws.metadata.namespace) return _doc("Workspace", ws.metadata.name, spec, ws.metadata.atespace)
def _model_doc(model: Model) -> dict[str, Any]: def _model_doc(model: Model) -> dict[str, Any]:
@@ -125,7 +127,7 @@ def _model_doc(model: Model) -> dict[str, Any]:
spec["secretKey"] = _fields(model.secret_key) spec["secretKey"] = _fields(model.secret_key)
if model.parameters: if model.parameters:
spec["parameters"] = dict(model.parameters) spec["parameters"] = dict(model.parameters)
return _doc("Model", model.metadata.name, spec, model.metadata.namespace) return _doc("Model", model.metadata.name, spec, model.metadata.atespace)
def _input_files(entry: RunTask, base: Path) -> list[dict[str, str]]: def _input_files(entry: RunTask, base: Path) -> list[dict[str, str]]:
@@ -157,8 +159,10 @@ def render_task(run: Run, entry: RunTask, manifests: Manifests, base: Path, arti
docs.append(_model_doc(model)) docs.append(_model_doc(model))
bindings = [_fields(b) for b in task.workspaces] bindings = [_fields(b) for b in task.workspaces]
docs.extend(_workspace_doc(manifests.workspaces[b.name]) for b in task.workspaces) docs.extend(_workspace_doc(manifests.workspaces[b.name]) for b in task.workspaces)
pins = {f"{b.name}/{git}": sha for b in task.workspaces for git, sha in manifests.workspaces[b.name].pins.items()}
env.update({"OUSAST_GIT_PINS": json.dumps(pins, sort_keys=True)} if pins else {})
if entry.inputs: if entry.inputs:
docs.append(_doc("Workspace", f"{ax_name}-inputs", {"files": _input_files(entry, base)}, task.metadata.namespace)) docs.append(_doc("Workspace", f"{ax_name}-inputs", {"files": _input_files(entry, base)}, task.metadata.atespace))
bindings.append({"name": f"{ax_name}-inputs", "path": f"{INPUTS_ROOT}/{entry.name}"}) bindings.append({"name": f"{ax_name}-inputs", "path": f"{INPUTS_ROOT}/{entry.name}"})
spec: dict[str, Any] = {"command": list(task.command), "env": [{"name": k, "value": v} for k, v in env.items()]} spec: dict[str, Any] = {"command": list(task.command), "env": [{"name": k, "value": v} for k, v in env.items()]}
if task.image: if task.image:
@@ -169,7 +173,7 @@ def render_task(run: Run, entry: RunTask, manifests: Manifests, base: Path, arti
spec["workspaces"] = bindings spec["workspaces"] = bindings
if task.debug: if task.debug:
spec["debug"] = True spec["debug"] = True
return [_doc("Task", ax_name, spec, task.metadata.namespace), *docs] return [_doc("Task", ax_name, spec, task.metadata.atespace), *docs]
class Ax: class Ax:
+85 -33
View File
@@ -17,6 +17,7 @@ Environment the runner reads, all optional except the two ax variables:
AX_RUNNER_HTTP "0" disables the health server (unit tests only) AX_RUNNER_HTTP "0" disables the health server (unit tests only)
AX_RUNNER_PORT port of the health server, default 80; "0" picks a free port AX_RUNNER_PORT port of the health server, default 80; "0" picks a free port
OUSAST_CASE_CACHE a directory of git checkouts (``benchmarks/independent`` cache layout) OUSAST_CASE_CACHE a directory of git checkouts (``benchmarks/independent`` cache layout)
OUSAST_GIT_PINS JSON ``{"<workspace>/<git name>": "<40-hex commit>"}`` from the reconciler
OUSAST_ARTIFACT_URL, OUSAST_RUN where to deliver, and the run name carried in ``X-Ousast-Run`` OUSAST_ARTIFACT_URL, OUSAST_RUN where to deliver, and the run name carried in ``X-Ousast-Run``
OUSAST_DELIVERY_BACKOFF seconds before the second delivery attempt, doubled per retry OUSAST_DELIVERY_BACKOFF seconds before the second delivery attempt, doubled per retry
@@ -32,7 +33,7 @@ import io
import json import json
import logging import logging
import os import os
import shutil import re
import subprocess import subprocess
import sys import sys
import tarfile import tarfile
@@ -61,7 +62,9 @@ __all__ = [
"RunnerError", "RunnerError",
"build_tar", "build_tar",
"deliver", "deliver",
"clone_destination",
"find_cached_checkout", "find_cached_checkout",
"load_pins",
"load_task", "load_task",
"load_workspaces", "load_workspaces",
"main", "main",
@@ -77,6 +80,9 @@ DELIVERY_ATTEMPTS = 3
TASK_PACKAGE = "openultrasast.plane.tasks" TASK_PACKAGE = "openultrasast.plane.tasks"
EXIT_BY_STATUS = {"done": 0, "failed": 2, "unfinished": 3} EXIT_BY_STATUS = {"done": 0, "failed": 2, "unfinished": 3}
SUMMARY = "summary.json" SUMMARY = "summary.json"
DEFAULT_BRANCH = "main" # ax internal/workspace/setup.go: defaultBranch
_PLACEHOLDER_NAMES = ("", "repo", "origin") # setup.go: names that do not name a checkout directory
_SHA = re.compile(r"[0-9a-f]{40}")
log = logging.getLogger("ousast.plane.runner") log = logging.getLogger("ousast.plane.runner")
@@ -221,13 +227,45 @@ def find_cached_checkout(cache: Path, repo: str) -> Path | None:
return None return None
def _pinned_ref(source: GitSource) -> str: def load_pins(text: str) -> dict[str, str]:
return source.commit or source.branch or "HEAD" """Parse ``OUSAST_GIT_PINS``: the reconciler's rendering of the Workspaces' ``openultrasast.io/git-commits``."""
if not text.strip():
return {}
try:
pins = json.loads(text)
except ValueError as exc:
raise RunnerError(f"OUSAST_GIT_PINS is not JSON: {exc}") from exc
if not isinstance(pins, dict) or not all(
isinstance(k, str) and "/" in k and isinstance(v, str) and _SHA.fullmatch(v) for k, v in pins.items()
):
raise RunnerError(f'OUSAST_GIT_PINS must map "<workspace>/<git name>" to a 40-hex commit, got {text[:200]!r}')
return pins
def _export_from_cache(checkout: Path, source: GitSource, dest: Path) -> None: def repo_dir_name(url: str) -> str:
"""setup.go ``RepoDirName``: ``https://github.com/chalk/chalk.git`` -> ``chalk``."""
trimmed = url.strip().removesuffix("/").removesuffix(".git")
return trimmed[max(trimmed.rfind("/"), trimmed.rfind(":")) + 1 :]
def clone_destination(source: GitSource, root: Path) -> Path:
"""setup.go ``cloneDestination``: ``dir`` wins (``.`` the workspace root, absolute as-is, else under the root);
otherwise the entry's ``name``, unless that is a placeholder (``repo``, ``origin``), then the URL's last segment."""
if source.dir:
return root if source.dir == "." else root / source.dir # an absolute dir replaces root in the join
name = source.name
if name in _PLACEHOLDER_NAMES:
name = repo_dir_name(source.repo) or name or "repo"
return root / name
def _pinned_ref(source: GitSource, commit: str | None) -> str:
return commit or source.branch or DEFAULT_BRANCH
def _export_from_cache(checkout: Path, source: GitSource, commit: str | None, dest: Path) -> None:
"""``git archive`` the pinned ref out of the cache into ``dest`` (no ``.git``), as ``evaluate.export`` does.""" """``git archive`` the pinned ref out of the cache into ``dest`` (no ``.git``), as ``evaluate.export`` does."""
ref = source.commit or (f"origin/{source.branch}" if source.branch else "HEAD") ref = commit or (f"origin/{source.branch}" if source.branch else "HEAD")
done = subprocess.run(["git", "-C", str(checkout), "archive", ref], capture_output=True, check=False) done = subprocess.run(["git", "-C", str(checkout), "archive", ref], capture_output=True, check=False)
if done.returncode != 0: if done.returncode != 0:
raise RunnerError(f"git archive {ref} in {checkout} failed: {done.stderr.decode(errors='replace').strip()[:400]}") raise RunnerError(f"git archive {ref} in {checkout} failed: {done.stderr.decode(errors='replace').strip()[:400]}")
@@ -241,47 +279,60 @@ def _export_from_cache(checkout: Path, source: GitSource, dest: Path) -> None:
raise RunnerError(f"{source.repo}@{ref}: the export from {checkout} is empty") raise RunnerError(f"{source.repo}@{ref}: the export from {checkout} is empty")
def _shallow_clone(source: GitSource, dest: Path) -> None: def _fetch(source: GitSource, commit: str | None, dest: Path) -> None:
"""Shallow clone at the branch, then make sure the pinned commit is checked out.""" """setup.go ``fetchRepo`` (init, remote, fetch ``--depth`` when ``depth`` > 0, checkout FETCH_HEAD), then the pin.
dest.parent.mkdir(parents=True, exist_ok=True)
args = ["clone", "--quiet", "--depth", "1"] A pinned entry without a branch fetches the commit itself rather than ax's default ``main``.
if source.branch: """
args += ["--branch", source.branch] dest.mkdir(parents=True, exist_ok=True)
_git(*args, source.repo, str(dest)) depth = [f"--depth={source.depth}"] if source.depth else []
if source.commit: _git("init", "--quiet", cwd=dest)
present = subprocess.run(["git", "cat-file", "-e", f"{source.commit}^{{commit}}"], cwd=dest, capture_output=True, check=False) try:
_git("remote", "add", "origin", source.repo, cwd=dest)
except RunnerError:
_git("remote", "set-url", "origin", source.repo, cwd=dest)
_git("fetch", "--quiet", *depth, "origin", source.branch or commit or DEFAULT_BRANCH, cwd=dest)
_git("checkout", "--quiet", "-f", "FETCH_HEAD", cwd=dest)
if commit:
present = subprocess.run(["git", "cat-file", "-e", f"{commit}^{{commit}}"], cwd=dest, capture_output=True, check=False)
if present.returncode != 0: if present.returncode != 0:
_git("fetch", "--quiet", "--depth", "1", "origin", source.commit, cwd=dest) _git("fetch", "--quiet", *depth, "origin", commit, cwd=dest)
_git("checkout", "--quiet", "--detach", source.commit, cwd=dest) _git("checkout", "--quiet", "-f", "--detach", commit, cwd=dest)
def _materialise_git(source: GitSource, root: Path, cache: Path | None) -> Path: def _materialise_git(source: GitSource, root: Path, cache: Path | None, commit: str | None) -> Path:
dest = root / source.name dest = clone_destination(source, root)
if dest.is_dir() and any(dest.iterdir()): resumed = (dest / ".git").exists() if dest == root else dest.is_dir() and any(dest.iterdir())
if resumed:
log.info("workspace repo %s already present at %s (resumed volume); keeping it", source.repo, dest) log.info("workspace repo %s already present at %s (resumed volume); keeping it", source.repo, dest)
return dest return dest
checkout = find_cached_checkout(cache, source.repo) if cache is not None else None checkout = find_cached_checkout(cache, source.repo) if cache is not None else None
if checkout is not None: if checkout is not None:
log.info("exporting %s@%s from cache %s to %s", source.repo, _pinned_ref(source), checkout, dest) log.info("exporting %s@%s from cache %s to %s", source.repo, _pinned_ref(source, commit), checkout, dest)
_export_from_cache(checkout, source, dest) _export_from_cache(checkout, source, commit, dest)
else: else:
log.info("cloning %s@%s to %s", source.repo, _pinned_ref(source), dest) log.info("fetching %s@%s to %s (depth %s)", source.repo, _pinned_ref(source, commit), dest, source.depth or "full")
if dest.exists(): _fetch(source, commit, dest)
shutil.rmtree(dest)
_shallow_clone(source, dest)
return dest return dest
def materialise(task: Task, workspaces: Mapping[str, Workspace], cache: Path | None) -> list[Path]: def materialise(task: Task, workspaces: Mapping[str, Workspace], cache: Path | None, pins: Mapping[str, str] | None = None) -> list[Path]:
"""Write every bound workspace at its path (``spec.files`` verbatim, ``spec.git`` at ``<path>/<name>``). """Write every bound workspace at its path (``spec.files`` verbatim, ``spec.git`` where ax's setup puts it).
Returns the bound paths in binding order; raises :class:`RunnerError` for a binding without a Workspace. ``pins`` (from ``OUSAST_GIT_PINS``) names the commit of ``<workspace>/<git name>``; a pin naming no bound git
entry, or a binding without a Workspace, raises :class:`RunnerError` before anything is written.
""" """
pins = dict(pins or {})
for binding in task.workspaces:
if binding.name not in workspaces:
raise RunnerError(f"Task/{task.metadata.name}: workspace {binding.name!r} is bound but not in AX_WORKSPACES_YAML")
known = {f"{b.name}/{g.name}" for b in task.workspaces for g in workspaces[b.name].git}
unknown = sorted(set(pins) - known)
if unknown:
raise RunnerError(f"OUSAST_GIT_PINS names no bound git entry: {', '.join(unknown)}")
paths: list[Path] = [] paths: list[Path] = []
for binding in task.workspaces: for binding in task.workspaces:
workspace = workspaces.get(binding.name) workspace = workspaces[binding.name]
if workspace is None:
raise RunnerError(f"Task/{task.metadata.name}: workspace {binding.name!r} is bound but not in AX_WORKSPACES_YAML")
root = Path(binding.path) root = Path(binding.path)
root.mkdir(parents=True, exist_ok=True) root.mkdir(parents=True, exist_ok=True)
for entry in workspace.files: for entry in workspace.files:
@@ -289,7 +340,7 @@ def materialise(task: Task, workspaces: Mapping[str, Workspace], cache: Path | N
target.parent.mkdir(parents=True, exist_ok=True) target.parent.mkdir(parents=True, exist_ok=True)
target.write_text(entry.content, encoding="utf-8") target.write_text(entry.content, encoding="utf-8")
for source in workspace.git: for source in workspace.git:
_materialise_git(source, root, cache) _materialise_git(source, root, cache, pins.get(f"{binding.name}/{source.name}"))
paths.append(root) paths.append(root)
return paths return paths
@@ -449,7 +500,8 @@ def main(argv: Sequence[str] | None = None, environ: MutableMapping[str, str] |
try: try:
session = _prepare(env) session = _prepare(env)
cache_dir = env.get("OUSAST_CASE_CACHE") cache_dir = env.get("OUSAST_CASE_CACHE")
materialise(session.task, session.workspaces, Path(cache_dir) if cache_dir else None) pins = load_pins(env.get("OUSAST_GIT_PINS", ""))
materialise(session.task, session.workspaces, Path(cache_dir) if cache_dir else None, pins)
ready.set() ready.set()
log.info("workspaces ready; running task %s: %s", session.task.metadata.name, list(session.task.command)) log.info("workspaces ready; running task %s: %s", session.task.metadata.name, list(session.task.command))
summary = run_task_module(session.task.command, session.output_dir) summary = run_task_module(session.task.command, session.output_dir)
+6 -3
View File
@@ -1,8 +1,9 @@
"""Workspace manifests for the cases of an independent population file (ai-service-plane task 4). """Workspace manifests for the cases of an independent population file (ai-service-plane task 4).
``ousast plane workspaces <population.toml> [--out DIR]`` writes one ax ``Workspace`` per case pin (vulnerable, ``ousast plane workspaces <population.toml> [--out DIR]`` writes one ax ``Workspace`` per case pin (vulnerable,
fixed, benign base, benign tip) with the case's repository URL and the pinned commit, so a Run binds a case at a fixed, benign base, benign tip) with the case's repository URL (checked out at ``<path>/repo``, depth 1) and the
pin by name. The layout of a case is that of ``benchmarks/independent/evaluate.py``'s ``pins()``. pinned commit in the ``openultrasast.io/git-commits`` annotation (ax's ``GitRepo`` has no commit field), so a Run
binds a case at a pin by name. The layout of a case is that of ``benchmarks/independent/evaluate.py``'s ``pins()``.
""" """
from __future__ import annotations from __future__ import annotations
@@ -12,6 +13,7 @@ from pathlib import Path
import yaml import yaml
from .manifests import GIT_COMMITS_ANNOTATION
from .reconciler import _ax_name, _doc from .reconciler import _ax_name, _doc
__all__ = ["workspaces"] __all__ = ["workspaces"]
@@ -26,7 +28,8 @@ def workspaces(population: Path, out: Path) -> list[Path]:
pins = {"vulnerable": case["vulnerable"], "fixed": case["fixed"], "benign-base": benign["base"], "benign-tip": benign["tip"]} pins = {"vulnerable": case["vulnerable"], "fixed": case["fixed"], "benign-base": benign["base"], "benign-tip": benign["tip"]}
for label, commit in pins.items(): for label, commit in pins.items():
name = _ax_name(case["id"], label) name = _ax_name(case["id"], label)
doc = _doc("Workspace", name, {"git": [{"name": "repo", "repo": case["repo"], "commit": commit}]}) doc = _doc("Workspace", name, {"git": [{"name": "repo", "repo": case["repo"], "dir": "repo", "depth": 1}]})
doc["metadata"]["annotations"] = {GIT_COMMITS_ANNOTATION: f"repo={commit}"}
text = f"# generated by `ousast plane workspaces` from {population.name}\n" + yaml.safe_dump(doc, sort_keys=False) text = f"# generated by `ousast plane workspaces` from {population.name}\n" + yaml.safe_dump(doc, sort_keys=False)
(out / f"{name}.yaml").write_text(text, encoding="utf-8") (out / f"{name}.yaml").write_text(text, encoding="utf-8")
written.append(out / f"{name}.yaml") written.append(out / f"{name}.yaml")
+29
View File
@@ -0,0 +1,29 @@
# Minimal Task for the bring-up smoke test: default runner image, a files-only workspace, no MCP or skills.
apiVersion: ax.io/v1alpha1
kind: Task
metadata:
name: ousast-smoke
atespace: default
spec:
image: "gcr.io/ax-substrate/ate-images/ax-task-runner@sha256:3a0dea6ad8b55278685db58aca6e37dc4ba04056831d45bef3aaeafdca43cac6"
command: ["sh", "-c", "cat /workspace/hello.txt && echo smoke-ok"]
resources:
requests:
cpu: "100m"
memory: "256Mi"
limits:
cpu: "1"
memory: "1Gi"
workspaces:
- name: ousast-smoke-ws
path: "/workspace"
---
apiVersion: ax.io/v1alpha1
kind: Workspace
metadata:
name: ousast-smoke-ws
atespace: default
spec:
files:
- path: "hello.txt"
content: "hello from openultrasast\n"
+63 -6
View File
@@ -25,7 +25,7 @@ apiVersion: ax.io/v1alpha1
kind: Task kind: Task
metadata: metadata:
name: verify name: verify
namespace: sast atespace: sast
spec: spec:
image: openultrasast:dev image: openultrasast:dev
command: ["verify", "--pass", "a"] command: ["verify", "--pass", "a"]
@@ -47,12 +47,15 @@ apiVersion: ax.io/v1alpha1
kind: Workspace kind: Workspace
metadata: metadata:
name: flask-abc123 name: flask-abc123
annotations:
openultrasast.io/git-commits: "repo=0123456789abcdef0123456789abcdef01234567"
spec: spec:
git: git:
- name: repo - name: repo
repo: https://github.com/pallets/flask.git repo: https://github.com/pallets/flask.git
branch: main branch: main
commit: abc123 dir: flask
depth: 1
files: files:
- path: candidates.json - path: candidates.json
content: '{"sinks": []}' content: '{"sinks": []}'
@@ -143,7 +146,7 @@ def test_task_loads_every_ax_field() -> None:
task = parse_manifest(doc(TASK_YAML)) task = parse_manifest(doc(TASK_YAML))
assert isinstance(task, Task) assert isinstance(task, Task)
assert task.metadata.name == "verify" assert task.metadata.name == "verify"
assert task.metadata.namespace == "sast" assert task.metadata.atespace == "sast"
assert task.image == "openultrasast:dev" assert task.image == "openultrasast:dev"
assert task.command == ("verify", "--pass", "a") assert task.command == ("verify", "--pass", "a")
assert task.env[0].name == "OUSAST_PASS" and task.env[0].value == "a" assert task.env[0].name == "OUSAST_PASS" and task.env[0].value == "a"
@@ -162,12 +165,66 @@ def test_task_minimal_defaults() -> None:
def test_workspace_loads() -> None: def test_workspace_loads() -> None:
workspace = parse_manifest(doc(WORKSPACE_YAML)) workspace = parse_manifest(doc(WORKSPACE_YAML))
assert isinstance(workspace, Workspace) assert isinstance(workspace, Workspace)
assert workspace.git[0].repo.endswith("flask.git") and workspace.git[0].commit == "abc123" assert workspace.git[0].repo.endswith("flask.git")
assert workspace.git[0].dir == "flask" and workspace.git[0].depth == 1
assert workspace.pins == {"repo": "0123456789abcdef0123456789abcdef01234567"}
assert workspace.files[0].path == "candidates.json" assert workspace.files[0].path == "candidates.json"
assert workspace.mcp is not None and len(workspace.mcp.registries) == 1 and len(workspace.mcp.servers) == 1 assert workspace.mcp is not None and len(workspace.mcp.registries) == 1 and len(workspace.mcp.servers) == 1
assert workspace.skills is not None and workspace.skills.path == "/skills" assert workspace.skills is not None and workspace.skills.path == "/skills"
SMOKE = Path(__file__).parent / "fixtures" / "plane" / "smoke-task.yaml"
def test_the_bring_up_smoke_manifest_loads() -> None:
"""``ops/ax/smoke-task.yaml`` (copied as a fixture) is what ``ax apply`` accepted; it must load here too."""
manifests = load_manifests([SMOKE])
assert manifests.tasks["ousast-smoke"].metadata.atespace == "default"
assert manifests.workspaces["ousast-smoke-ws"].metadata.atespace == "default"
def test_the_shipped_smoke_manifest_matches_the_fixture() -> None:
shipped = Path(__file__).parent.parent / "ops" / "ax" / "smoke-task.yaml"
if not shipped.is_file():
pytest.skip("ops/ax/smoke-task.yaml is not in this checkout")
assert load_manifests([shipped]).tasks.keys() == {"ousast-smoke"}
@pytest.mark.parametrize("key", ["namespace", "labels", "creation_timestamp"])
def test_metadata_takes_ax_names_only(key: str) -> None:
task = doc(TASK_YAML)
task["metadata"][key] = "sast"
rejected(task, f"Task/verify: metadata.{key} is not a field")
def test_git_commit_is_rejected_with_a_pointer_to_the_annotation() -> None:
workspace = doc(WORKSPACE_YAML)
workspace["spec"]["git"][0]["commit"] = "abc123"
message = rejected(workspace, "Workspace/flask-abc123: spec.git[0].commit is not an ax field")
assert "openultrasast.io/git-commits" in message
@pytest.mark.parametrize(
("value", "rule"),
[
("other=" + "a" * 40, '"other" names no spec.git entry'),
("repo=abc123", '"repo=abc123" is not a 40-hex commit'),
("repo", 'item "repo" must be'),
],
)
def test_git_commits_annotation_is_validated(value: str, rule: str) -> None:
workspace = doc(WORKSPACE_YAML)
workspace["metadata"]["annotations"]["openultrasast.io/git-commits"] = value
message = rejected(workspace, 'metadata.annotations["openultrasast.io/git-commits"]')
assert rule in message, message
def test_git_depth_must_be_an_integer() -> None:
workspace = doc(WORKSPACE_YAML)
workspace["spec"]["git"][0]["depth"] = "1"
rejected(workspace, "Workspace/flask-abc123: spec.git[0].depth must be a non-negative integer")
def test_model_loads_with_declared_extension() -> None: def test_model_loads_with_declared_extension() -> None:
model = parse_manifest(doc(MODEL_YAML)) model = parse_manifest(doc(MODEL_YAML))
assert isinstance(model, Model) assert isinstance(model, Model)
@@ -235,8 +292,8 @@ def test_unknown_nested_fields_are_named_with_index() -> None:
task["spec"]["resources"]["requests"]["gpu"] = "1" task["spec"]["resources"]["requests"]["gpu"] = "1"
rejected(task, "spec.resources.requests.gpu is not a field") rejected(task, "spec.resources.requests.gpu is not a field")
workspace = doc(WORKSPACE_YAML) workspace = doc(WORKSPACE_YAML)
workspace["spec"]["git"][0]["depth"] = 1 workspace["spec"]["git"][0]["submodules"] = True
rejected(workspace, "Workspace/flask-abc123: spec.git[0].depth is not a field") rejected(workspace, "Workspace/flask-abc123: spec.git[0].submodules is not a field")
model = doc(MODEL_YAML) model = doc(MODEL_YAML)
model["spec"]["secretKey"]["namespace"] = "x" model["spec"]["secretKey"]["namespace"] = "x"
rejected(model, "Model/deepseek-flash: spec.secretKey.namespace is not a field") rejected(model, "Model/deepseek-flash: spec.secretKey.namespace is not a field")
+63 -6
View File
@@ -24,6 +24,40 @@ from openultrasast.plane import reconciler
from openultrasast.plane.workspaces import workspaces from openultrasast.plane.workspaces import workspaces
RECONCILER = Path("src/openultrasast/plane/reconciler.py") RECONCILER = Path("src/openultrasast/plane/reconciler.py")
PIN = "0123456789abcdef0123456789abcdef01234567"
# The fields ax's API server decodes (pkg/apis/v1alpha1/ax.proto, protojson: an unknown field is an error at
# ``ax apply``). ``None`` marks a value ax types as free-form (a Struct) or that this test does not descend into.
_RESOURCES = {"cpu": None, "memory": None}
AX_FIELDS: dict[str, dict] = {
"Task": {
"image": None,
"command": None,
"env": {"name": None, "value": None},
"resources": {"requests": _RESOURCES, "limits": _RESOURCES},
"workspaces": {"name": None, "path": None, "goal": None},
"debug": None,
},
"Workspace": {
"git": {"name": None, "repo": None, "branch": None, "dir": None, "depth": None},
"files": {"path": None, "content": None},
"mcp": {"registries": None, "servers": None},
"skills": {"registries": None, "path": None},
},
"Model": {"provider": None, "model": None, "secretKey": {"name": None, "key": None}, "parameters": None},
}
def non_ax_fields(value: object, schema: dict | None, path: str) -> list[str]:
"""Every key under ``value`` that ``schema`` does not name, as dotted paths."""
if schema is None:
return []
if isinstance(value, list):
return [bad for i, item in enumerate(value) for bad in non_ax_fields(item, schema, f"{path}[{i}]")]
assert isinstance(value, dict), f"{path} must be a mapping"
bad = [f"{path}.{k}" for k in value if k not in schema]
return bad + [b for k, v in value.items() if k in schema for b in non_ax_fields(v, schema[k], f"{path}.{k}")]
FAKE_AX = '''#!{python} FAKE_AX = '''#!{python}
"""Fake ax CLI for the reconciler tests: argv log, scripted phases, scripted deliveries.""" """Fake ax CLI for the reconciler tests: argv log, scripted phases, scripted deliveries."""
@@ -90,8 +124,10 @@ apiVersion: ax.io/v1alpha1
kind: Workspace kind: Workspace
metadata: metadata:
name: case-pin name: case-pin
atespace: default
annotations: {openultrasast.io/git-commits: "repo=0123456789abcdef0123456789abcdef01234567"}
spec: spec:
git: [{name: repo, repo: https://example.invalid/repo.git, commit: abc123}] git: [{name: repo, repo: https://example.invalid/repo.git, dir: src, depth: 1}]
--- ---
apiVersion: ax.io/v1alpha1 apiVersion: ax.io/v1alpha1
kind: Task kind: Task
@@ -242,6 +278,10 @@ def test_chain_runs_in_order_with_inputs_env_and_model(fake: Fake, tmp_path: Pat
assert inputs_ws["spec"]["files"] == [{"path": "repo-facts/facts.json", "content": json.dumps({"a.py": {"functions": ["f"]}})}] assert inputs_ws["spec"]["files"] == [{"path": "repo-facts/facts.json", "content": json.dumps({"a.py": {"functions": ["f"]}})}]
assert verify["docs"][0]["spec"]["workspaces"][-1] == {"name": "chain-verify-inputs", "path": "/workspace/.ousast-in/verify"} assert verify["docs"][0]["spec"]["workspaces"][-1] == {"name": "chain-verify-inputs", "path": "/workspace/.ousast-in/verify"}
assert "annotations" not in verify["docs"][0]["metadata"], "ax's ObjectMeta has no annotations" assert "annotations" not in verify["docs"][0]["metadata"], "ax's ObjectMeta has no annotations"
assert json.loads(env["OUSAST_GIT_PINS"]) == {"case-pin/repo": PIN}, "the Workspace annotation reaches the task as env"
assert verify["docs"][2]["spec"]["git"] == [{"name": "repo", "repo": "https://example.invalid/repo.git", "dir": "src", "depth": 1}]
assert verify["docs"][2]["metadata"] == {"name": "case-pin", "atespace": "default"}
assert "OUSAST_GIT_PINS" not in fake.applied("chain", "agree")["env"], "no bound workspace, no pins"
facts = fake.applied("chain", "repo-facts") facts = fake.applied("chain", "repo-facts")
assert "OUSAST_MODEL" not in facts["env"] and "OUSAST_BUDGET_USD" not in facts["env"] assert "OUSAST_MODEL" not in facts["env"] and "OUSAST_BUDGET_USD" not in facts["env"]
assert facts["docs"][0]["spec"]["image"] == "ousast-runner:dev" assert facts["docs"][0]["spec"]["image"] == "ousast-runner:dev"
@@ -440,8 +480,8 @@ def test_doctor_names_the_bring_up_script_when_nothing_is_reachable(monkeypatch:
def test_workspaces_one_manifest_per_case_pin(tmp_path: Path) -> None: def test_workspaces_one_manifest_per_case_pin(tmp_path: Path) -> None:
population = tmp_path / "population.toml" population = tmp_path / "population.toml"
population.write_text( population.write_text(
'[[case]]\nid = "demo-sqli"\nrepo = "https://github.com/x/y"\nvulnerable = "aaa"\nfixed = "bbb"\n' '[[case]]\nid = "demo-sqli"\nrepo = "https://github.com/x/y"\nvulnerable = "' + "a" * 40 + '"\nfixed = "' + "b" * 40 + '"\n'
'benign = { base = "ccc", tip = "ddd" }\n', 'benign = { base = "' + "c" * 40 + '", tip = "' + "d" * 40 + '" }\n',
encoding="utf-8", encoding="utf-8",
) )
written = workspaces(population, tmp_path / "out") written = workspaces(population, tmp_path / "out")
@@ -451,12 +491,29 @@ def test_workspaces_one_manifest_per_case_pin(tmp_path: Path) -> None:
assert doc == { assert doc == {
"apiVersion": "ax.io/v1alpha1", "apiVersion": "ax.io/v1alpha1",
"kind": "Workspace", "kind": "Workspace",
"metadata": {"name": "demo-sqli-vulnerable"}, "metadata": {"name": "demo-sqli-vulnerable", "annotations": {"openultrasast.io/git-commits": "repo=" + "a" * 40}},
"spec": {"git": [{"name": "repo", "repo": "https://github.com/x/y", "commit": "aaa"}]}, "spec": {"git": [{"name": "repo", "repo": "https://github.com/x/y", "dir": "repo", "depth": 1}]},
} }
from openultrasast.plane.manifests import load_manifests from openultrasast.plane.manifests import load_manifests
assert set(load_manifests(written).workspaces) == {p.stem for p in written} loaded = load_manifests(written).workspaces
assert set(loaded) == {p.stem for p in written}
assert loaded["demo-sqli-benign-tip"].pins == {"repo": "d" * 40}
def test_rendered_documents_carry_only_ax_fields(tmp_path: Path) -> None:
"""Req 1.1/1.4: whatever ``ax apply`` receives names only fields ax documents; annotations never leave."""
(tmp_path / "repo-facts").mkdir()
(tmp_path / "repo-facts" / "facts.json").write_text("{}", encoding="utf-8")
run, manifests = reconciler.load_run(write_run(tmp_path, "fields"))
docs = [d for entry in run.tasks for d in reconciler.render_task(run, entry, manifests, tmp_path, "http://h:1/")]
assert {d["kind"] for d in docs} == {"Task", "Workspace", "Model"}
for rendered in docs:
where = f"{rendered['kind']}/{rendered['metadata']['name']}"
assert set(rendered) == {"apiVersion", "kind", "metadata", "spec"}, where
assert set(rendered["metadata"]) <= {"name", "atespace"}, where
assert non_ax_fields(rendered["spec"], AX_FIELDS[rendered["kind"]], f"{where} spec") == []
assert "openultrasast.io/" not in yaml.safe_dump(rendered["metadata"]), where
def test_reconciler_is_under_500_lines_and_holds_no_pipeline_logic() -> None: def test_reconciler_is_under_500_lines_and_holds_no_pipeline_logic() -> None:
+109 -10
View File
@@ -19,7 +19,15 @@ from pathlib import Path
import pytest import pytest
from openultrasast.plane import runner from openultrasast.plane import runner
from openultrasast.plane.runner import EXIT_BY_STATUS, DeliveryError, deliver, find_cached_checkout, start_health_server from openultrasast.plane.manifests import GitSource
from openultrasast.plane.runner import (
EXIT_BY_STATUS,
DeliveryError,
clone_destination,
deliver,
find_cached_checkout,
start_health_server,
)
STUB = Path(__file__).parent / "plane_stub_task.py" STUB = Path(__file__).parent / "plane_stub_task.py"
STUB_MODULE = f"{runner.TASK_PACKAGE}.stub_task" STUB_MODULE = f"{runner.TASK_PACKAGE}.stub_task"
@@ -313,14 +321,19 @@ def upstream(tmp_path: Path) -> tuple[Path, str]:
return repo, pin return repo, pin
def git_workspace_yaml(repo_url: str, pin: str) -> str: def git_workspace_yaml(repo_url: str, extra: str = "") -> str:
"""ax's Workspace as ax hands it over: no annotations, no commit (the pin travels in ``OUSAST_GIT_PINS``)."""
return ( return (
"apiVersion: ax.io/v1alpha1\nkind: Workspace\nmetadata:\n name: case\nspec:\n git:\n" "apiVersion: ax.io/v1alpha1\nkind: Workspace\nmetadata:\n name: case\nspec:\n git:\n"
f" - name: repo\n repo: {repo_url}\n branch: main\n commit: {pin}\n" f" - name: code\n repo: {repo_url}\n branch: main\n{extra}"
" files:\n - path: candidates.json\n content: '{\"sinks\": []}'\n" " files:\n - path: candidates.json\n content: '{\"sinks\": []}'\n"
) )
def pins(pin: str, key: str = "case/code") -> str:
return json.dumps({key: pin})
def test_git_entry_is_exported_from_the_case_cache_at_the_pin( def test_git_entry_is_exported_from_the_case_cache_at_the_pin(
env: dict[str, str], monkeypatch: pytest.MonkeyPatch, tmp_path: Path, upstream: tuple[Path, str] env: dict[str, str], monkeypatch: pytest.MonkeyPatch, tmp_path: Path, upstream: tuple[Path, str]
) -> None: ) -> None:
@@ -332,22 +345,24 @@ def test_git_entry_is_exported_from_the_case_cache_at_the_pin(
git("remote", "set-url", "origin", url, cwd=case) git("remote", "set-url", "origin", url, cwd=case)
assert find_cached_checkout(cache, "https://example.invalid/acme/repo") == case assert find_cached_checkout(cache, "https://example.invalid/acme/repo") == case
assert find_cached_checkout(cache, "https://example.invalid/other/repo.git") is None assert find_cached_checkout(cache, "https://example.invalid/other/repo.git") is None
code, output = run_main(monkeypatch, tmp_path, git_workspace_yaml(url, pin), OUSAST_CASE_CACHE=str(cache)) code, output = run_main(monkeypatch, tmp_path, git_workspace_yaml(url), OUSAST_CASE_CACHE=str(cache), OUSAST_GIT_PINS=pins(pin))
assert code == 0, (output / "summary.json").read_text() assert code == 0, (output / "summary.json").read_text()
checkout = tmp_path / "ws" / "repo" checkout = tmp_path / "ws" / "code"
assert (checkout / "hello.py").read_text() == "print('v1')\n", "the pinned commit, not the branch tip" assert (checkout / "hello.py").read_text() == "print('v1')\n", "the pinned commit, not the branch tip"
assert not (checkout / ".git").exists(), "an export, as evaluate.export makes" assert not (checkout / ".git").exists(), "an export, as evaluate.export makes"
facts = json.loads((output / "facts.json").read_text()) facts = json.loads((output / "facts.json").read_text())
assert facts["workspace_files"] == ["candidates.json", "repo/hello.py"] assert facts["workspace_files"] == ["candidates.json", "code/hello.py"]
def test_git_entry_without_a_cache_is_shallow_cloned_at_the_pin( def test_git_entry_without_a_cache_is_shallow_cloned_at_the_pin(
env: dict[str, str], monkeypatch: pytest.MonkeyPatch, tmp_path: Path, upstream: tuple[Path, str] env: dict[str, str], monkeypatch: pytest.MonkeyPatch, tmp_path: Path, upstream: tuple[Path, str]
) -> None: ) -> None:
repo, pin = upstream repo, pin = upstream
code, output = run_main(monkeypatch, tmp_path, git_workspace_yaml(repo.as_uri(), pin), OUSAST_CASE_CACHE=str(tmp_path / "no-cache")) code, output = run_main(
monkeypatch, tmp_path, git_workspace_yaml(repo.as_uri()), OUSAST_CASE_CACHE=str(tmp_path / "no-cache"), OUSAST_GIT_PINS=pins(pin)
)
assert code == 0, (output / "summary.json").read_text() assert code == 0, (output / "summary.json").read_text()
checkout = tmp_path / "ws" / "repo" checkout = tmp_path / "ws" / "code"
assert (checkout / "hello.py").read_text() == "print('v1')\n" assert (checkout / "hello.py").read_text() == "print('v1')\n"
assert git("rev-parse", "HEAD", cwd=checkout) == pin assert git("rev-parse", "HEAD", cwd=checkout) == pin
@@ -356,9 +371,93 @@ def test_a_materialised_repo_survives_a_resume(
env: dict[str, str], monkeypatch: pytest.MonkeyPatch, tmp_path: Path, upstream: tuple[Path, str] env: dict[str, str], monkeypatch: pytest.MonkeyPatch, tmp_path: Path, upstream: tuple[Path, str]
) -> None: ) -> None:
repo, pin = upstream repo, pin = upstream
checkout = tmp_path / "ws" / "repo" checkout = tmp_path / "ws" / "code"
checkout.mkdir(parents=True) checkout.mkdir(parents=True)
(checkout / "hello.py").write_text("kept\n") (checkout / "hello.py").write_text("kept\n")
code, _ = run_main(monkeypatch, tmp_path, git_workspace_yaml(repo.as_uri(), pin)) code, _ = run_main(monkeypatch, tmp_path, git_workspace_yaml(repo.as_uri()), OUSAST_GIT_PINS=pins(pin))
assert code == 0 assert code == 0
assert (checkout / "hello.py").read_text() == "kept\n", "/workspace persists across suspend and resume; no re-clone" assert (checkout / "hello.py").read_text() == "kept\n", "/workspace persists across suspend and resume; no re-clone"
# --- ax's GitRepo fields: dir, depth; the commit pin from OUSAST_GIT_PINS ------------------------------------------------
@pytest.mark.parametrize(
("name", "repo", "dir_", "expected"),
[
("code", "https://github.com/chalk/chalk.git", None, "/ws/code"),
("repo", "https://github.com/chalk/chalk.git", None, "/ws/chalk"),
("origin", "[email protected]:chalk/chalk", None, "/ws/chalk"),
("code", "https://github.com/chalk/chalk.git", "src", "/ws/src"),
("code", "https://github.com/chalk/chalk.git", ".", "/ws"),
("code", "https://github.com/chalk/chalk.git", "/elsewhere", "/elsewhere"),
],
)
def test_clone_destination_follows_ax_setup(name: str, repo: str, dir_: str | None, expected: str) -> None:
"""ax internal/workspace/setup.go cloneDestination: dir wins; placeholder names derive from the URL."""
assert clone_destination(GitSource(name, repo, dir=dir_), Path("/ws")) == Path(expected)
def test_git_dir_and_depth_are_honoured(
env: dict[str, str], monkeypatch: pytest.MonkeyPatch, tmp_path: Path, upstream: tuple[Path, str]
) -> None:
repo, _ = upstream
code, output = run_main(monkeypatch, tmp_path, git_workspace_yaml(repo.as_uri(), " dir: src\n depth: 1\n"))
assert code == 0, (output / "summary.json").read_text()
checkout = tmp_path / "ws" / "src"
assert (checkout / "hello.py").read_text() == "print('v2')\n", "no pin: the branch tip"
assert git("rev-parse", "--is-shallow-repository", cwd=checkout) == "true", "depth 1 is a shallow fetch"
assert not (tmp_path / "ws" / "code").exists()
def test_a_pin_naming_no_bound_git_entry_fails_the_task(
env: dict[str, str], monkeypatch: pytest.MonkeyPatch, tmp_path: Path, upstream: tuple[Path, str]
) -> None:
repo, pin = upstream
code, output = run_main(monkeypatch, tmp_path, git_workspace_yaml(repo.as_uri()), OUSAST_GIT_PINS=pins(pin, "case/other"))
assert code == EXIT_BY_STATUS["failed"]
assert "case/other" in json.loads((output / "summary.json").read_text())["reason"]
def test_unreadable_pins_fail_the_task(env: dict[str, str], monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
code, output = run_main(monkeypatch, tmp_path, files_workspace_yaml(), OUSAST_GIT_PINS='{"case/code": "abc"}')
assert code == EXIT_BY_STATUS["failed"]
assert "OUSAST_GIT_PINS" in json.loads((output / "summary.json").read_text())["reason"]
def test_git_commits_annotation_round_trips_to_the_checkout(
env: dict[str, str], monkeypatch: pytest.MonkeyPatch, tmp_path: Path, upstream: tuple[Path, str]
) -> None:
"""Workspace annotation -> reconciler -> ``OUSAST_GIT_PINS`` in the Task env -> the runner checks out the pin."""
import yaml
from openultrasast.plane import reconciler
repo, pin = upstream
output, workspace = tmp_path / "out", tmp_path / "ws"
manifest = tmp_path / "run.yaml"
manifest.write_text(
"apiVersion: ax.io/v1alpha1\nkind: Workspace\nmetadata:\n name: case\n"
f" annotations: {{openultrasast.io/git-commits: 'code={pin}'}}\n"
f"spec:\n git: [{{name: code, repo: '{repo.as_uri()}', branch: main}}]\n---\n"
"apiVersion: ax.io/v1alpha1\nkind: Task\nmetadata: {name: stub}\n"
f"spec: {{command: [stub-task], workspaces: [{{name: case, path: '{workspace}'}}]}}\n---\n"
"apiVersion: openultrasast.io/v1alpha1\nkind: Run\nmetadata: {name: rt}\nspec: {tasks: [{name: stub, task: stub}]}\n"
)
run, manifests = reconciler.load_run(manifest)
task_doc, *rest = reconciler.render_task(run, run.tasks[0], manifests, tmp_path, "unused")
assert rest[0]["metadata"] == {"name": "case"}, "annotations are stripped before ax"
assert rest[0]["spec"]["git"] == [{"name": "code", "repo": repo.as_uri(), "branch": "main"}], "no commit for ax"
overrides = {"OUSAST_OUTPUT_DIR": str(output)}
task_doc["spec"]["env"] = [
{"name": e["name"], "value": overrides.get(e["name"], e["value"])}
for e in task_doc["spec"]["env"]
if e["name"] != "OUSAST_ARTIFACT_URL"
]
assert json.loads({e["name"]: e["value"] for e in task_doc["spec"]["env"]}["OUSAST_GIT_PINS"]) == {"case/code": pin}
monkeypatch.setenv("AX_TASK_YAML", yaml.safe_dump(task_doc))
monkeypatch.setenv("AX_WORKSPACES_YAML", yaml.safe_dump_all([d for d in rest if d["kind"] == "Workspace"]))
assert runner.main() == 0, (output / "summary.json").read_text()
checkout = workspace / "code"
assert git("rev-parse", "HEAD", cwd=checkout) == pin, "the pinned commit, not the branch tip"
assert (checkout / "hello.py").read_text() == "print('v1')\n"