search-with-proof: cluster-unavailable detection, kept messages, run-level abort

Zero running workers behind "no free workers available" now raises
ClusterUnavailable instead of waiting out the capacity deadline; CapacityDeadline
carries its message; cleanup failures no longer mask the original exception; the
pilot driver aborts the run on infrastructure failure before any model call.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-10-05 20:26:37 +02:00
co-authored by Claude Fable 5.1
parent e39a4882b0
commit 4be2063d40
7 changed files with 284 additions and 16 deletions
+9 -1
View File
@@ -116,4 +116,12 @@ def diagnostic(value: object, *, maximum: int = 300, private: tuple[str, ...] =
def exception_reason(exc: BaseException, *, private: tuple[str, ...] = ()) -> str:
prefix = type(exc).__name__ + ": "
return prefix + diagnostic(exc, maximum=max(0, 300 - len(prefix)), private=private)
maximum = max(0, 300 - len(prefix))
cleanup = getattr(exc, "executor_cleanup_reason", None)
if cleanup:
original = str(exc).removesuffix("; " + cleanup)
# Reserve space for both causes: a long teardown must not erase the
# original failure from the bounded public record.
half = max(0, (maximum - 2) // 2)
return prefix + diagnostic(original, maximum=half, private=private) + "; " + diagnostic(cleanup, maximum=half, private=private)
return prefix + diagnostic(exc, maximum=maximum, private=private)