fix(plane): ax task lifecycle -- resume, stay-up runner, delivery as completion

The first live run on the local ax cluster and ax's source (docs/runner.md,
docs/sandbox.md, internal/controller/reconciler.go, internal/server/server.go)
showed a lifecycle the task-3/4 code did not model:

- `ax apply` creates a Task Suspended (Ready=False, TaskSuspended); it runs only
  after `ax resume task`. The first resume of a new image can fail with
  DeadlineExceeded while Agent Substrate builds the golden snapshot, and ax's
  server then stores phase Failed; a later resume succeeds.
- ax has no Completed phase and never reads a command's exit back.
- The runner is PID 1 and must keep serving after the command exits; stop and
  suspend deliver SIGTERM, to be forwarded to the command's process group.
  Our runner imported the task in-process and exited, which stopped the
  container ("Container guest is stopped") and made every resume time out.
- ax's controller probes /readyz?check=workspace; the query string made our
  handler answer 404.

Runner (Req 4.1-4.4): spec.command runs as a child
`python -m openultrasast.plane.tasks.<name>` in its own process group, cwd the
first workspace, AX_METADATA_URL set; ax's metadata paths are served. After the
child exits the runner reads summary.json (a crash without one becomes failed
with the exit code and stderr tail), delivers the output directory, writes
/workspace/.ousast-state/<task>.done.json and serves until SIGTERM (forwarded
to the group, SIGKILL after OUSAST_TERM_GRACE, exit 0). A boot with the marker
never reruns the command; a failed delivery is retried alone. Workspaces are
prepared on the first boot only. AX_RUNNER_EXIT_AFTER_COMMAND=1 is the
unit-test escape.

Reconciler (Req 2.2, 2.3, 3.1-3.3): `ax resume task` after apply, retrying
DeadlineExceeded/Unavailable with backoff up to OUSAST_RESUME_TIMEOUT (900 s);
completion is the artifact delivery; Failed after a resume went through, a
vanished task, or no delivery within OUSAST_TASK_TIMEOUT (7200 s) fail the
task with ax's condition message; `ax delete task` follows. Ax gains resume()
and phase() returns the condition message. doctor moves to plane/doctor.py
(tasks.md task 0 allows the sibling) to keep reconciler.py under 500 lines.

Tests: the fake ax follows the real lifecycle (Suspended, a failing first
resume that leaves Failed, Running forever, delivery only after a resume went
through), plus Failed-with-message, vanish, resume-timeout and task-timeout
scenarios; runner tests cover the child's cwd/group/metadata URL, stay-up and
SIGTERM in a real subprocess, SIGKILL of a SIGTERM-ignoring group, and the
marker. The opt-in live test now checks resume, Running, suspend and a second
resume instead of a Completed phase. design.md and tasks.md state the
lifecycle; the module audit manifest counts plane.doctor.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo

tests/test_plane_ax_live.py

@@ -8,12 +8,13 @@ passes, so the host suite never depends on cluster state. It applies the digest-
 from __future__ import annotations
 
 import os
+import subprocess
 import time
 from pathlib import Path
 
 import pytest
 
-from openultrasast.plane.reconciler import Ax, doctor
+from openultrasast.plane.reconciler import TRANSIENT, Ax, doctor
 
 pytestmark = pytest.mark.ax
 SMOKE = Path.home() / ".cache/ousast/ax-src/smoke-task.yaml"
@@ -32,16 +33,34 @@ def live_ax() -> Ax:
     return Ax()
 
 
-def test_smoke_task_completes_on_the_cluster(live_ax: Ax) -> None:
+def resume_until_accepted(ax: Ax, name: str, deadline: float) -> None:
+    """ax creates a Task Suspended; the first resume of a new image may time out while its snapshot is built."""
+    error: str | None = "not tried"
+    while error is not None and time.monotonic() < deadline:
+        error = ax.resume(name)
+        if error is not None and not TRANSIENT.search(error):
+            raise AssertionError(f"ax resume task {name} failed: {error}")
+        if error is not None:
+            time.sleep(10)
+    assert error is None, f"ax resume task {name} never went through: {error}"
+
+
+def test_smoke_task_runs_and_its_runner_survives_suspend_and_resume(live_ax: Ax) -> None:
+    """ax has no Completed phase: the smoke task must reach Running after resume, and its runner must stay up after
+    the command so that a suspend and a second resume still go through (an exiting runner made resumes time out)."""
+    name = "ousast-smoke"
+    live_ax.delete(name)
     live_ax.apply(SMOKE)
-    deadline = time.monotonic() + 600
-    phase = "Pending"
+    deadline = time.monotonic() + 900
     try:
-        while time.monotonic() < deadline:
-            phase = live_ax.phase("ousast-smoke")
-            if any(word in phase.lower() for word in ("completed", "succeeded", "failed", "error")):
-                break
-            time.sleep(5)
+        resume_until_accepted(live_ax, name, deadline)
+        phase, message = live_ax.phase(name)
+        assert phase == "Running", f"after resume: {phase} ({message})"
+        time.sleep(30)  # the model-free repo-facts command is finished by now
+        suspended = subprocess.run(["ax", "suspend", "task", name], capture_output=True, text=True, check=False)
+        assert suspended.returncode == 0, suspended.stderr
+        resume_until_accepted(live_ax, name, deadline)
+        phase, message = live_ax.phase(name)
+        assert phase == "Running", f"after the second resume: {phase} ({message})"
     finally:
-        live_ax.delete("ousast-smoke")
-    assert phase.lower() in ("completed", "succeeded"), f"smoke task ended in phase {phase!r}"
+        live_ax.delete(name)