search-with-proof: declarative demos, brain/executor split, kube-ax constraints

Demos are data the verifier executes (fixed build recipes, runtime+path start,
HTTP/CLI steps with captures; the canary is never referable). The agent loop with
the key never runs repository code: executor tasks without a key exchange commands
and results through presigned objects. AX capacity errors are back-pressure with at
most 2 tasks in flight; caches and builds under /workspace with a 2 GiB guard;
verifiers take one presigned archive per side. Launcher fix: private /dev in bwrap.
190 tests on the host; probe 5/5 demonstrated.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

.kiro/specs/search-with-proof/implementation-trust-domains.md

@@ -0,0 +1,80 @@
+# Declarative verification and separate execution domains
+
+Implemented on `search-with-proof`, 2026-10-05, under the maintainer's explicit instructions.
+`git merge -q main` failed because the shared Git metadata is read-only. The amended requirements
+were read from the main checkout, including Requirements 3.1 and 5.2. No commits or network operations.
+
+## Changes
+
+- Strict, bounded JSON demonstration schema, also supplied to the model's `write_demo` tool.
+  Verifier-owned recipes cover pip, npm, composer, Maven, Gradle and no build. Runtime entrypoints,
+  CLI/HTTP sequences and earlier-response captures are validated; private fixtures, environment
+  substitutions, canary captures and script artefacts are rejected. Injection/traversal strings remain
+  legitimate request payload data, never executable verifier instructions.
+- The verifier owns build, readiness and request execution. Local runs retain Bubblewrap (`userns`
+  on this host). HTTP requests require an acknowledged readiness gate; incomplete observations fail
+  closed. Build-generated internal symlinks are supported without accepting external or cyclic links.
+- A supplied AX side dispatcher selects `task-boundary` and creates three separate fresh tasks per
+  side. Failure of local isolation without a dispatcher fails closed.
+- The model-holding worker delegates list/read/grep/run to an executor interface. Production uses
+  presigned command/result objects; the in-process adapter is for tests. Commands have sequence
+  numbers, identical-retry caching, deadlines, stop and bounded results. Uncertain outcomes prevent
+  later commands from reusing that executor session.
+- AX executor tasks use the `ousast-engine-search-exec-` prefix. Task entrypoints reject inherited
+  credentials before reading checkout input, then use an explicit environment allowlist.
+
+## Changed files
+
+Production:
+- `src/openultrasast/search/demo.py`
+- `src/openultrasast/search/executor.py`
+- `src/openultrasast/search/worker.py`
+- `src/openultrasast/search/verify.py`
+- `src/openultrasast/search/verify_task.py`
+- `src/openultrasast/search/_sandbox.py`
+- `src/openultrasast/search/oracles.py`
+- `src/openultrasast/search/probe.py`
+- `src/openultrasast/search/probe_apps/app.py.txt`
+- `benchmarks/ax/batch.py`
+- `benchmarks/ax/search_verify.py`
+- `plane/Dockerfile.search-task`
+
+Tests and audit:
+- `tests/test_search_demo.py`
+- `tests/test_search_executor.py`
+- `tests/test_search_verify_transport.py`
+- `tests/test_search_verify.py`
+- `tests/test_search_worker.py`
+- `tests/test_search_isolation.py`
+- `tests/test_search_probe.py`
+- `tests/test_ax_batch.py`
+- `benchmarks/measurements/2026-09-08-module-audit.json`
+
+This report is the additional documentation file.
+
+## Fresh validation
+
+`/home/mc/Source/OpenUltraSAST/.venv/bin/pytest -q tests/test_search_*.py tests/test_ax_batch.py tests/test_plane_literals.py tests/test_module_audit.py -rs`
+
+Result: **171 passed, 3 skipped, exit 0**, in 35.46 seconds.
+The three skips are Chromium isolation/network setup, SSRF localhost sockets and HTTP localhost sockets,
+all prohibited by this host's policy. Host verification mode was explicitly checked: `userns`.
+
+Ruff passes for search sources, AX changes and affected tests. Targeted strict mypy passes for the seven
+changed Python search implementation modules checked. Independent review approved the implementation,
+including follow-up timeout, readiness, capture, credential and generated-link regressions.
+
+Module audit: **171 modules**, **165 load-bearing**, **6 standalone capabilities**, **0 orphaned**.
+The audit read the worktree source explicitly with `PYTHONPATH=src`.
+
+## Remaining validation gaps
+
+- No live AX/gVisor/kind dispatch, image build/pull/publish, or paid model calls were performed.
+- Actual ecosystem dependency installations remain unqualified. Python dependency loading was tested
+  with a controlled offline build output and real isolated execution.
+- The three socket/browser integrations need a host or task where those operations are permitted.
+- Hard aggregate disk isolation in `task-boundary` mode requires an outer task storage quota. The
+  executor records this requirement and additionally bounds individual files and monitors checkout
+  growth; it cannot enforce a quota across every writable task path without that outer resource limit.
+- AX tasks configured to inject a model credential will fail closed; executor and verifier task classes
+  must be provisioned without that injection.

.kiro/specs/search-with-proof/operator-constraints-validation.md

@@ -0,0 +1,101 @@
+# Host launcher repair and kube-ax operator constraints
+
+2026-10-05, branch `search-with-proof`. Implemented under the maintainer's explicit
+instructions. No commits or network operations. Existing uncommitted work was preserved.
+This supersedes the build/transport and disk-bound statements in
+`implementation-trust-domains.md` where they differ.
+
+## Root cause and repair
+
+The HTTP driver calls `subprocess.Popen(..., stdin=DEVNULL, stdout=DEVNULL,
+stderr=DEVNULL)` inside bubblewrap. Its filesystem had no `/dev/null`; Python failed
+in `_get_handles` before starting the app, binding port 18765, or applying captures.
+Other probe apps do not use this nested HTTP launcher path.
+
+A socket-free before/after reproduction read a 19-byte instrument input. Removing
+`--dev /dev` reproduced exit 1 and the reported `subprocess.py:992 → _get_handles`
+stack, ending with `FileNotFoundError: [Errno 2] No such file or directory: '/dev/null'`.
+With the private device mount, the identical launcher exited 0 with empty stderr.
+The original HTTP capture test's fixed port, token exchange, and differential
+assertion remain unchanged. Reasons now retain the last 2,000 stderr characters;
+executor result truncation retains stderr tails too.
+
+## Operator contract
+
+- Shared two-slot AX reservations cover workload lane instances and long-lived
+  search executors. Capacity rejection retries the identical manifest with 15–30
+  seconds of jitter until the deadline. Exhaustion is `capacity_timeout`, never
+  `sandbox_failure`, and does not trigger another execution attempt or VM fallback.
+  Uncertain task cleanup retains the reservation and halts dispatch.
+- Task entrypoints configure temporary files and the specified package caches under
+  `/workspace`. The aggregate guard defaults to 2 GiB, configurable through
+  `OUSAST_SCRATCH_BYTES`. It checks workspace files, caches, and unlinked output logs
+  during child execution and after exit. Exceeding it returns `could_not_build` with
+  `scratch limit`; unreadable scratch is not interpreted as zero usage.
+- The default AX side dispatcher first builds/packages each side in an executor,
+  retrieves its digest-checked archive, and deletes the executor before verification.
+  One archive contains the checkout, products, and spec. Three fresh verifier tasks
+  consume that archive through `VERIFY_INPUT_URL`; results use `RESULT_URL`.
+  Verifiers reject dependency builds. Git metadata is omitted from source archives;
+  safe internal generated links are materialized before export.
+- Search tasks receive source archives. Executor tool guidance and project/operator
+  documentation require HTTPS GitHub git clone or codeload, never `api.github.com`.
+  First-download retries tolerate delayed egress activation for up to 60 seconds.
+
+## Files changed in this follow-up
+
+Production:
+- `src/openultrasast/search/_sandbox.py`
+- `src/openultrasast/search/verify.py`
+- `src/openultrasast/search/executor.py`
+- `src/openultrasast/search/verify_task.py`
+- `src/openultrasast/search/task_storage.py` (new)
+- `src/openultrasast/search/worker.py`
+- `benchmarks/ax/batch.py`
+- `benchmarks/ax/search_verify.py`
+
+Tests:
+- `tests/test_search_isolation.py`
+- `tests/test_search_verify.py`
+- `tests/test_search_verify_transport.py`
+- `tests/test_search_task_storage.py` (new)
+- `tests/test_ax_batch.py`
+
+Documentation/evidence:
+- `.kiro/steering/kube-ax.md` (new)
+- `ops/ax/README.md`
+- `benchmarks/measurements/2026-09-08-module-audit.json`
+- This report (new).
+
+Other files already modified/untracked at session start remain uncommitted.
+
+## Fresh verification
+
+- Requested suite, with `-rs` for skip reasons:
+  `/home/mc/Source/OpenUltraSAST/.venv/bin/pytest -q tests/test_search_*.py tests/test_ax_batch.py tests/test_plane_literals.py -rs`
+  — **181 passed, 3 skipped, exit 0**, 24.94 seconds.
+- Final executor/storage regression and module audit tests after tightening stderr
+  truncation: **34 passed, exit 0**, including all **6 module audit tests**.
+- Shared dispatcher consumer `tests/test_unseen_calibrate.py`: **10 passed, exit 0**.
+- Module audit regenerated from the worktree using `PYTHONPATH=src`, with an explicit
+  source read/byte count: **172 modules, 166 load-bearing, 6 standalone capabilities,
+  0 orphaned**.
+- Ruff passes across search, AX, and changed tests; strict mypy passes for the six
+  changed typed implementation modules. `git diff --check` passes.
+
+## Gaps
+
+- Host policy prohibits Chromium network/isolation setup, SSRF localhost sockets,
+  and the full HTTP capture test. These are the three skips. The exact missing-device
+  failure is reproduced and repaired without sockets; full HTTP host confirmation
+  remains necessary.
+- No live AX tasks, image builds/pulls, registry installs, or external downloads were
+  run. Mocked transport plus local real child processes establish code contracts,
+  not deployed cluster readiness. Verifier egress to
+  `files.because-security.com:443` is requested from the operator and still pending.
+- Scratch enforcement is a polling guard, not an instantaneous filesystem quota.
+  Programs that hardcode RAM-backed paths can still consume the worker memory limit.
+  Archive bounds remain 64 MiB compressed / 256 MiB file contents / 20,000 entries;
+  larger built projects fail explicitly and require a separately qualified envelope.
+- Multi-ecosystem dependency build/runtime qualification remains open. The offline
+  preparation tests cover no-build projects and controlled Python build products.