plane(2.4): reconciler on Delivery -- the start body carries the URLs, the store completes the task

_execute builds the task's `delivery {put, inputs}` from Delivery, passes it to
router.starter, grants the store's host in the egress policy, awaits
Delivery.delivered (the tar's exact key in the store), then collect()s:
download, extract with the member checks, re-put the declared artifacts, write
done.json; the outcome is the collected summary.json. Three findings from the
tests: a rerun of an unfinished task completed at once on the previous
attempt's tar, so Delivery.reset(task) deletes output.tar and done.json before
every attempt; a worker thread that raised left the run blocked on its queue,
so the worker now turns any exception into a failed task; a consumer may
declare a producer's summary.json without the producer listing it (the loop
Run does), so declared(run) publishes outputs[] plus every artifact a consumer
names. The receiver thread survives behind OUSAST_DELIVERY=receiver for this
group only (no presigned URLs, the http rule, OUSAST_ARTIFACT_URL in the env);
the default path renders no receiver address into any Task. The fake ax PUTs
to the start body's URL and GETs its inputs from theirs; the reconciler tests
run through the S3-shaped fake store and assert the objects, done.json, the
seeded rerun and the receiver mode. reconciler.py: 549 lines (bound 550).

Gates: pytest 2031 passed, 42 skipped, exit 0; ruff check 0; ruff format 0;
mypy 0.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-10-02 21:49:58 +02:00
co-authored by Claude Fable 5.1
parent c49eae4f30
commit 605b7af5ba
5 changed files with 203 additions and 54 deletions
+20 -1
View File
@@ -15,7 +15,7 @@ from pathlib import Path
import pytest
from plane_fake_store import HttpObjectStore
from openultrasast.plane.delivery import GRACE_SECONDS, Delivery, DeliveryError, expiry, input_var
from openultrasast.plane.delivery import GRACE_SECONDS, Delivery, DeliveryError, declared, expiry, input_var
@pytest.fixture
@@ -92,6 +92,9 @@ def test_delivered_polls_the_store_and_collect_extracts_republishes_and_marks_do
with urllib.request.urlopen(urls["FACTS"], timeout=10) as response: # noqa: S310
assert response.read() == b'{"a": 1}'
assert "X-Amz" not in caplog.text and "fake-bucket" not in caplog.text, "URLs never appear in logs"
delivery.reset("facts") # the next attempt must not complete on this attempt's tar
assert delivery.delivered("facts") is False and "runs/r/facts/done.json" not in store.objects
assert "runs/r/facts/facts.json" in store.objects, "a re-put output stays for its consumers until it is re-published"
@pytest.mark.parametrize("bad", ["../escape.json", "/abs.json"])
@@ -136,6 +139,22 @@ def test_put_input_stores_a_file_no_task_produces(store: HttpObjectStore, tmp_pa
assert response.read() == b"[1, 2, 3]"
def test_declared_artifacts_are_outputs_plus_what_consumers_name(tmp_path: Path) -> None:
"""A consumer may declare a producer's summary.json (every task writes one) without the producer listing it."""
from openultrasast.plane.manifests import load_manifests
text = (
"apiVersion: ax.io/v1alpha1\nkind: Task\nmetadata: {name: t}\nspec: {command: [repo-facts]}\n---\n"
"apiVersion: openultrasast.io/v1alpha1\nkind: Run\nmetadata: {name: r}\nspec:\n tasks:\n"
" - {name: facts, task: t, outputs: [facts.json]}\n"
" - {name: verify, task: t, inputs: {facts: facts/facts.json, facts_summary: facts/summary.json}, outputs: [units.jsonl]}\n"
" - {name: agree, task: t, inputs: {pass: verify/units.jsonl, pass_summary: verify/summary.json}}\n"
)
(tmp_path / "run.yaml").write_text(text, encoding="utf-8")
run = next(iter(load_manifests([tmp_path / "run.yaml"]).runs.values()))
assert declared(run) == {"facts": ["facts.json", "summary.json"], "verify": ["units.jsonl", "summary.json"], "agree": []}
def test_publish_reputs_only_declared_files_under_the_task_directory(store: HttpObjectStore, tmp_path: Path) -> None:
delivery = Delivery(store, "r", tmp_path / "r")
(tmp_path / "r" / "seeded").mkdir(parents=True)