mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 19:59:26 +00:00
Marius CiepluchandClaude Opus 5.5
bb5c534fb2
The first live run on the local ax cluster and ax's source (docs/runner.md,
docs/sandbox.md, internal/controller/reconciler.go, internal/server/server.go)
showed a lifecycle the task-3/4 code did not model:
- `ax apply` creates a Task Suspended (Ready=False, TaskSuspended); it runs only
after `ax resume task`. The first resume of a new image can fail with
DeadlineExceeded while Agent Substrate builds the golden snapshot, and ax's
server then stores phase Failed; a later resume succeeds.
- ax has no Completed phase and never reads a command's exit back.
- The runner is PID 1 and must keep serving after the command exits; stop and
suspend deliver SIGTERM, to be forwarded to the command's process group.
Our runner imported the task in-process and exited, which stopped the
container ("Container guest is stopped") and made every resume time out.
- ax's controller probes /readyz?check=workspace; the query string made our
handler answer 404.
Runner (Req 4.1-4.4): spec.command runs as a child
`python -m openultrasast.plane.tasks.<name>` in its own process group, cwd the
first workspace, AX_METADATA_URL set; ax's metadata paths are served. After the
child exits the runner reads summary.json (a crash without one becomes failed
with the exit code and stderr tail), delivers the output directory, writes
/workspace/.ousast-state/<task>.done.json and serves until SIGTERM (forwarded
to the group, SIGKILL after OUSAST_TERM_GRACE, exit 0). A boot with the marker
never reruns the command; a failed delivery is retried alone. Workspaces are
prepared on the first boot only. AX_RUNNER_EXIT_AFTER_COMMAND=1 is the
unit-test escape.
Reconciler (Req 2.2, 2.3, 3.1-3.3): `ax resume task` after apply, retrying
DeadlineExceeded/Unavailable with backoff up to OUSAST_RESUME_TIMEOUT (900 s);
completion is the artifact delivery; Failed after a resume went through, a
vanished task, or no delivery within OUSAST_TASK_TIMEOUT (7200 s) fail the
task with ax's condition message; `ax delete task` follows. Ax gains resume()
and phase() returns the condition message. doctor moves to plane/doctor.py
(tasks.md task 0 allows the sibling) to keep reconciler.py under 500 lines.
Tests: the fake ax follows the real lifecycle (Suspended, a failing first
resume that leaves Failed, Running forever, delivery only after a resume went
through), plus Failed-with-message, vanish, resume-timeout and task-timeout
scenarios; runner tests cover the child's cwd/group/metadata URL, stay-up and
SIGTERM in a real subprocess, SIGKILL of a SIGTERM-ignoring group, and the
marker. The opt-in live test now checks resume, Running, suspend and a second
resume instead of a Completed phase. design.md and tasks.md state the
lifecycle; the module audit manifest counts plane.doctor.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
67 lines
2.8 KiB
Python
67 lines
2.8 KiB
Python
"""ai-service-plane Req 3.5/3.6: one trivial Task end to end on the real ax cluster of ``ops/ax/up.sh``.
|
|
|
|
Marker ``ax``; opt-in only (``OUSAST_AX_LIVE=1``) and skipped unless every check of ``ousast plane doctor``
|
|
passes, so the host suite never depends on cluster state. It applies the digest-pinned smoke manifest that
|
|
``ops/ax/up.sh`` renders. Run it on purpose: ``OUSAST_AX_LIVE=1 pytest -m ax tests/test_plane_ax_live.py``.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import os
|
|
import subprocess
|
|
import time
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
from openultrasast.plane.reconciler import TRANSIENT, Ax, doctor
|
|
|
|
pytestmark = pytest.mark.ax
|
|
SMOKE = Path.home() / ".cache/ousast/ax-src/smoke-task.yaml"
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def live_ax() -> Ax:
|
|
if os.environ.get("OUSAST_AX_LIVE") != "1":
|
|
pytest.skip("live ax test is opt-in: set OUSAST_AX_LIVE=1")
|
|
checks = doctor()
|
|
failing = [f"{name}: {text}" for name, ok, text in checks if not ok]
|
|
if failing:
|
|
pytest.skip("no live ax cluster: " + "; ".join(failing))
|
|
if not SMOKE.is_file():
|
|
pytest.skip(f"{SMOKE} is not in this checkout")
|
|
return Ax()
|
|
|
|
|
|
def resume_until_accepted(ax: Ax, name: str, deadline: float) -> None:
|
|
"""ax creates a Task Suspended; the first resume of a new image may time out while its snapshot is built."""
|
|
error: str | None = "not tried"
|
|
while error is not None and time.monotonic() < deadline:
|
|
error = ax.resume(name)
|
|
if error is not None and not TRANSIENT.search(error):
|
|
raise AssertionError(f"ax resume task {name} failed: {error}")
|
|
if error is not None:
|
|
time.sleep(10)
|
|
assert error is None, f"ax resume task {name} never went through: {error}"
|
|
|
|
|
|
def test_smoke_task_runs_and_its_runner_survives_suspend_and_resume(live_ax: Ax) -> None:
|
|
"""ax has no Completed phase: the smoke task must reach Running after resume, and its runner must stay up after
|
|
the command so that a suspend and a second resume still go through (an exiting runner made resumes time out)."""
|
|
name = "ousast-smoke"
|
|
live_ax.delete(name)
|
|
live_ax.apply(SMOKE)
|
|
deadline = time.monotonic() + 900
|
|
try:
|
|
resume_until_accepted(live_ax, name, deadline)
|
|
phase, message = live_ax.phase(name)
|
|
assert phase == "Running", f"after resume: {phase} ({message})"
|
|
time.sleep(30) # the model-free repo-facts command is finished by now
|
|
suspended = subprocess.run(["ax", "suspend", "task", name], capture_output=True, text=True, check=False)
|
|
assert suspended.returncode == 0, suspended.stderr
|
|
resume_until_accepted(live_ax, name, deadline)
|
|
phase, message = live_ax.phase(name)
|
|
assert phase == "Running", f"after the second resume: {phase} ({message})"
|
|
finally:
|
|
live_ax.delete(name)
|