Open_Ultra_SAST/.kiro/specs/search-with-proof/operator-constraints-validation.md

Host launcher repair and kube-ax operator constraints

2026-10-05, branch search-with-proof. Implemented under the maintainer's explicit instructions. No commits or network operations. Existing uncommitted work was preserved. This supersedes the build/transport and disk-bound statements in implementation-trust-domains.md where they differ.

Root cause and repair

The HTTP driver calls subprocess.Popen(..., stdin=DEVNULL, stdout=DEVNULL, stderr=DEVNULL) inside bubblewrap. Its filesystem had no /dev/null; Python failed in _get_handles before starting the app, binding port 18765, or applying captures. Other probe apps do not use this nested HTTP launcher path.

A socket-free before/after reproduction read a 19-byte instrument input. Removing --dev /dev reproduced exit 1 and the reported subprocess.py:992 → _get_handles stack, ending with FileNotFoundError: [Errno 2] No such file or directory: '/dev/null'. With the private device mount, the identical launcher exited 0 with empty stderr. The original HTTP capture test's fixed port, token exchange, and differential assertion remain unchanged. Reasons now retain the last 2,000 stderr characters; executor result truncation retains stderr tails too.

Operator contract

• Shared two-slot AX reservations cover workload lane instances and long-lived search executors. Capacity rejection retries the identical manifest with 15–30 seconds of jitter until the deadline. Exhaustion is capacity_timeout, never sandbox_failure, and does not trigger another execution attempt or VM fallback. Uncertain task cleanup retains the reservation and halts dispatch.
• Task entrypoints configure temporary files and the specified package caches under /workspace. The aggregate guard defaults to 2 GiB, configurable through OUSAST_SCRATCH_BYTES. It checks workspace files, caches, and unlinked output logs during child execution and after exit. Exceeding it returns could_not_build with scratch limit; unreadable scratch is not interpreted as zero usage.
• The default AX side dispatcher first builds/packages each side in an executor, retrieves its digest-checked archive, and deletes the executor before verification. One archive contains the checkout, products, and spec. Three fresh verifier tasks consume that archive through VERIFY_INPUT_URL; results use RESULT_URL. Verifiers reject dependency builds. Git metadata is omitted from source archives; safe internal generated links are materialized before export.
• Search tasks receive source archives. Executor tool guidance and project/operator documentation require HTTPS GitHub git clone or codeload, never api.github.com. First-download retries tolerate delayed egress activation for up to 60 seconds.

Files changed in this follow-up

Production:

• src/openultrasast/search/_sandbox.py
• src/openultrasast/search/verify.py
• src/openultrasast/search/executor.py
• src/openultrasast/search/verify_task.py
• src/openultrasast/search/task_storage.py (new)
• src/openultrasast/search/worker.py
• benchmarks/ax/batch.py
• benchmarks/ax/search_verify.py

Tests:

• tests/test_search_isolation.py
• tests/test_search_verify.py
• tests/test_search_verify_transport.py
• tests/test_search_task_storage.py (new)
• tests/test_ax_batch.py

Documentation/evidence:

• .kiro/steering/kube-ax.md (new)
• ops/ax/README.md
• benchmarks/measurements/2026-09-08-module-audit.json
• This report (new).

Other files already modified/untracked at session start remain uncommitted.

Fresh verification

• Requested suite, with -rs for skip reasons: /home/mc/Source/OpenUltraSAST/.venv/bin/pytest -q tests/test_search_*.py tests/test_ax_batch.py tests/test_plane_literals.py -rs Result: 181 passed, 3 skipped, exit 0, 24.94 seconds.
• Final executor/storage regression and module audit tests after tightening stderr truncation: 34 passed, exit 0, including all 6 module audit tests.
• Shared dispatcher consumer tests/test_unseen_calibrate.py: 10 passed, exit 0.
• Module audit regenerated from the worktree using PYTHONPATH=src, with an explicit source read/byte count: 172 modules, 166 load-bearing, 6 standalone capabilities, 0 orphaned.
• Ruff passes across search, AX, and changed tests; strict mypy passes for the six changed typed implementation modules. git diff --check passes.

Gaps

• Host policy prohibits Chromium network/isolation setup, SSRF localhost sockets, and the full HTTP capture test. These are the three skips. The exact missing-device failure is reproduced and repaired without sockets; full HTTP host confirmation remains necessary.
• No live AX tasks, image builds/pulls, registry installs, or external downloads were run. Mocked transport plus local real child processes establish code contracts, not deployed cluster readiness. Verifier egress to files.because-security.com:443 is requested from the operator and still pending.
• Scratch enforcement is a polling guard, not an instantaneous filesystem quota. Programs that hardcode RAM-backed paths can still consume the worker memory limit. Archive bounds remain 64 MiB compressed / 256 MiB file contents / 20,000 entries; larger built projects fail explicitly and require a separately qualified envelope.
• Multi-ecosystem dependency build/runtime qualification remains open. The offline preparation tests cover no-build projects and controlled Python build products.