Commit Graph
819 Commits
Author SHA1 Message Date
Marius CiepluchandClaude Fable 5.1 a40122d1b5 merge(search-with-proof): oracle mapping, build memory, tolerant tool calls [skip ci]
Image rebuild dispatched by hand (code in the last layer, ~10 MB).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-05 21:09:08 +02:00
Marius CiepluchandClaude Fable 5.1 bf243e6680 search-with-proof: family-to-oracle mapping, build memory, tolerant tool calls
Demos declare their oracle within the family allowed set (injection: sql or
command; output_encoding: xss; untrusted_destination: ssrf; path: path);
builds and app starts use the search memory budget instead of the 256 MB tool
limit (npm died with SIGTRAP); malformed tool-call JSON is fed back once.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-05 21:07:58 +02:00
Marius CiepluchandClaude Fable 5.1 0750c6fdd7 merge(search-with-proof): named exhaustion limits and partial evidence [skip ci]
Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-05 21:01:31 +02:00
Marius Ciepluch c190d34f60 Merge branch 'main' into search-with-proof
# Conflicts:
#	tests/test_search_task_storage.py
2026-10-05 20:58:10 +02:00
Marius CiepluchandClaude Fable 5.1 c33b053b15 search-with-proof: name the exhausted limit, keep partial evidence, tighter reservation
budget_spent carries end_detail (spend, task_spend, reason_rounds, ...); a task
whose allowance cannot cover the next call finishes with its facts; per-task
metrics in pilot records; input reservation bytes/2 + 1024 per message (measured
3-4x over-reservation before).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-05 20:57:56 +02:00
Marius CiepluchandClaude Fable 5.1 35f942f4c8 images: pin task base b696c08 (base and browser) by digest
Triggers the one intended rebuild of engine, replay and search on the shared base.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-05 20:50:16 +02:00
Marius Ciepluch b696c08446 merge(image-layering): shared task base and code-last layering [skip ci]
Co-Authored-By: Claude Fable 5.1 <[email protected]>

# Conflicts:
#	tests/test_search_task_storage.py
2026-10-05 20:44:02 +02:00
Marius CiepluchandClaude Fable 5.1 94d6a7dd11 search executor: NPROC bound counts threads, as the kernel does [skip ci]
Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-05 20:44:02 +02:00
Marius CiepluchandClaude Fable 5.1 1b9d1ca85c images: shared task base, code-last layers, no per-push rebuilds [skip ci]
One base (Temurin 21 JDK, node/npm/tsc, php/composer, maven, sqlite, bwrap)
built only when its Dockerfile or dependencies change, plus a browser variant;
engine/search/VM images FROM base@sha256 with the code in the last layer; the
image workflow runs on dispatch, tags, Dockerfile or pin changes, with a build
cache. Executor NPROC bound relative to the current process count.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-05 20:40:13 +02:00
Marius CiepluchandClaude Fable 5.1 509c6562fe search budget: recalibrated caps after the first measured searches [skip ci]
Settled ~$0.003 per model call; worst-case reservations exhausted the $0.05 task
allowance after 2-4 calls. Now $0.40 per task, $2 per search (maintainer decision).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-05 20:35:27 +02:00
Marius CiepluchandClaude Fable 5.1 b44fb56d8d ax batch: workload-kind prefixes in task names (pass-, replay-) [skip ci]
The operator keys per-host egress rules on the task-name prefix; engine passes and
replays had none. Names stay within 49 bytes.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-05 20:33:35 +02:00
Marius Ciepluch daf95b16a1 merge(search-with-proof): cluster-unavailable handling [skip ci]
No image rebuild: VM-side changes only (operator asked to stop per-merge rebuilds).

Co-Authored-By: Claude Fable 5.1 <[email protected]>

# Conflicts:
#	benchmarks/ax/batch.py
2026-10-05 20:27:01 +02:00
Marius CiepluchandClaude Fable 5.1 4be2063d40 search-with-proof: cluster-unavailable detection, kept messages, run-level abort
Zero running workers behind "no free workers available" now raises
ClusterUnavailable instead of waiting out the capacity deadline; CapacityDeadline
carries its message; cleanup failures no longer mask the original exception; the
pilot driver aborts the run on infrastructure failure before any model call.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-05 20:26:37 +02:00
Marius CiepluchandClaude Opus 5.5 7fb061fb69 merge(search-with-proof): search contracts
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 19:18:12 +02:00
Marius CiepluchandClaude Opus 5.5 e39a4882b0 search-with-proof: bootstrap intent, real reason/explore contracts, visible refusals
First live run: the reason step got ~190 tokens and returned {} -> silent
no_evidence. Now: a bootstrap intent reads the candidate and its entry point; the
reason prompt states the goal, family oracles, demo schema and must propose
intents when none are open; empty replies retry once then end as reason_refused;
failure records keep messages.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 19:18:12 +02:00
Marius CiepluchandClaude Opus 5.5 0fb098c478 merge(search-with-proof): pilot driver
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 18:58:24 +02:00
Marius CiepluchandClaude Opus 5.5 87b82564b4 search-with-proof: pilot driver (6.2)
One side per run over selected pilot pairs: board, brain on the VM with the
project DeepSeek client, executor tasks on kube-ax, verification on the VM lane
(AX verify lane switchable). Hard run ceiling with reservation before every call,
per-search and step caps; fixed side never sees the vulnerable revision; records
carry digests only. Dry run with the stub model reaches demonstrated.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 18:58:24 +02:00
Marius CiepluchandClaude Opus 5.5 89516d4000 ax batch: fast capacity retries in the first minute (golden-snapshot window)
Operator 2026-10-05: Substrate boots a ~1 s golden-snapshot actor per new task on the
other worker; retry apply after 2-5 s for the first minute, then 15-30 s.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 18:07:47 +02:00
Marius CiepluchandClaude Opus 5.5 941fbb5fb0 search-with-proof: diagnosable failures, pip install race, no duplicate tasks
Reasons carry type plus a sanitised message tail; command failures carry phase,
exit code and bounded stderr. Scratch walk tolerates directories pip removes
mid-walk; writable /workspace/home and runtime paths. Capacity retries delete a
possibly created task before resubmitting. Docker smoke checks the image first.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 17:58:27 +02:00
Marius CiepluchandClaude Opus 5.5 7c1af9f518 merge(search-with-proof): diagnosable failures and dispatch fixes
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 17:58:27 +02:00
Marius CiepluchandClaude Opus 5.5 cf10f38589 search-task image: clear the base image GPG_KEY fingerprint
The official Python base sets GPG_KEY (public CPython signing fingerprint); the
executor rightly refused it as a credential-like name on kube-ax.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 17:42:03 +02:00
Marius CiepluchandClaude Opus 5.5 75f2134eb4 search executor: name the credential-like variables it refuses (names only)
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 17:36:12 +02:00
Marius CiepluchandClaude Opus 5.5 1d67412e2b search-with-proof: executor smoke driver and pilot selection (6.1)
20 development pairs, one per repository, 4 per language, 5 per family; identities
in a gitignored manifest, counts and digest committed. Executor smoke drives one
executor task end to end without a model and fails loudly on an unread clone.
kube-ax operator constraints as steering.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 17:26:06 +02:00
Marius CiepluchandClaude Opus 5.5 1bfb729422 merge(search-with-proof): executor smoke and pilot selection
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 17:26:06 +02:00
Marius CiepluchandClaude Opus 5.5 85f4f0b3d4 search-with-proof: declarative demos, brain/executor split, kube-ax constraints
Demos are data the verifier executes (fixed build recipes, runtime+path start,
HTTP/CLI steps with captures; the canary is never referable). The agent loop with
the key never runs repository code: executor tasks without a key exchange commands
and results through presigned objects. AX capacity errors are back-pressure with at
most 2 tasks in flight; caches and builds under /workspace with a 2 GiB guard;
verifiers take one presigned archive per side. Launcher fix: private /dev in bwrap.
190 tests on the host; probe 5/5 demonstrated.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 17:12:46 +02:00
Marius CiepluchandClaude Opus 5.5 ccd64757c8 merge(search-with-proof): trust-domain split and operator constraints
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 17:12:46 +02:00
Marius CiepluchandClaude Opus 5.5 721c4b0997 spec(search-with-proof): separate trust domains (brain with key, executor with repo code)
Replaces the accepted key-exposure exception: the key never shares a sandbox with
repository code; executor AX tasks exchange commands and results via presigned
objects. Firecracker executors only if the pilot build-failure rate requires it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 16:26:19 +02:00
Marius CiepluchandClaude Opus 5.5 5e03bdf07f spec(search-with-proof): gVisor is the cluster boundary; declarative demos; capped rotated key
kube-ax probe: AX tasks have no capabilities (no namespaces, no setuid), so no
in-task isolation. Demos become data the verifier executes; the explore key risk is
accepted and bounded by cap and rotation.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 16:20:26 +02:00
Marius CiepluchandClaude Opus 5.5 e5ce630edc merge(search-with-proof): gVisor isolation mode
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 15:00:25 +02:00
Marius CiepluchandClaude Opus 5.5 03857c9bec search-with-proof: root-no-userns isolation mode for gVisor
The kube-ax probe showed bwrap cannot map uids in gVisor as an unprivileged user.
As root (gVisor is the outer boundary) the sandbox uses pid/ipc/uts/net/mount
namespaces without a user namespace and drops the artefact to its own uid via
setpriv; the mode is recorded. The probe reports each namespace kind separately.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 15:00:25 +02:00
Marius CiepluchandClaude Opus 5.5 e6871fef9a merge(search-with-proof): probe harness and agent worker
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 14:45:48 +02:00
Marius CiepluchandClaude Opus 5.5 0f74767db9 search-with-proof: probe harness (3.2) and agent worker (group 5)
Probe runs each oracle toy pair with a real demo and two gaming demos, records
image facts, RSS and disk, fails loudly if isolation is unavailable; dispatchable
as an AX task. Worker: reason step without tools, explore tool loop whose run tool
goes through the sandbox with a scrubbed env (model key absent), spend reserved
before each call. Host probe: 5/5 real demos demonstrated, 10/10 gaming demos
inconclusive. 89 tests.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 14:45:48 +02:00
Marius CiepluchandClaude Opus 5.5 a17a6afaf8 merge(search-with-proof): group 4 and the search-task image
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 14:26:43 +02:00
Marius CiepluchandClaude Opus 5.5 4b38239f68 search-with-proof group 4 + image: board, spend reservation, coordinator, search-task image
Board as immutable versioned blobs (coordinator sole writer); budget reserve/settle
before model calls; per-task and run caps; coordinator loop over an injected
executor; batch.py accepts a fixed allow-list of search variables; XSS browser
oracle; ousast-search-task image (Python, Node/TypeScript, PHP, Java 17, sqlite,
bwrap, chromium) built by CI. 103 tests on the host.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 14:26:43 +02:00
Marius CiepluchandClaude Opus 5.5 6394ee473d merge(search-with-proof): group 2 verifier and oracles
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 14:09:48 +02:00
Marius CiepluchandClaude Opus 5.5 742469d671 search-with-proof group 2: verifier with owned oracles and differential
The verifier, not the artefact, observes the effect: SQL, path, command and SSRF
oracles owned by the verifier (XSS needs a browser executor, else no_oracle);
artefact exit code and output ignored; immutable checkout and dependencies;
3 runs per side; one-sided build or readiness failure is inconclusive. An
isolation failure raises IsolationUnavailable instead of reading as
could_not_build (the NPROC limit now applies inside the user namespace).
35 tests, including gaming attempts and 300 host processes.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 14:09:48 +02:00
Marius CiepluchandClaude Opus 5.5 c74b5eaeb0 spec(learned-decision-engine): engine becomes triage where a demonstration oracle exists
Req 6.5 (maintainer decision 2026-10-05, with search-with-proof): BLOCK comes from
verified demonstrations; the engine is judged by top-K recall per push and advisory
false alarms; the pooled block gate governs only families without an oracle.
Task 17 adds the metric and re-scores existing arms at no cost.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 13:45:10 +02:00
Marius CiepluchandClaude Opus 5.5 9d70ef9257 spec(search-with-proof): requirements and design approved; tasks for probe and pilot
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 13:35:55 +02:00
Marius CiepluchandClaude Opus 5.5 c3820e184c spec(search-with-proof): requirements amended after two Codex reviews; design v2
False alarms count every visible finding; the verifier owns per-family oracles;
inconclusive on one-sided build failure; feasibility pilot separated from a
73-negative BLOCK qualification; coverage ceilings; spend reserved before calls;
key isolated from executed code; cluster model calls supersede the pool VM-only
rule for this arm only. Requirements approval reopened.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 13:35:18 +02:00
Marius CiepluchandClaude Opus 5.5 8bbea86cf8 spec(search-with-proof): requirements approved
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 12:01:39 +02:00
Marius CiepluchandClaude Opus 5.5 45e641f25c spec(search-with-proof): brief and requirements for review
After reviewing Cairn: detection so far classified excerpts and hit a precision
wall; this adds directed search with an objective goal (a re-verified
demonstration), a fact/intent board, and reason/explore steps as AX Tasks on
kube-ax. Design only learns from Cairn (AGPL); no code copied.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 12:01:07 +02:00
Marius CiepluchandClaude Opus 5.5 6d27c9bfcf unseen-eval: calibrated replay deadline + native agreement check, docker-lane diagnostics, proof 4.4 record; engine runner cleans up fetched sources
gVisor replays exhausted the hook's 30 s deadline in the snapshot step on a
1,725-file repository; cluster replays now use 30 s x a measured slowdown
factor, verified on a native sample. The engine runner deletes each pin's fetched
source (1.8 GB had accumulated), caps the cache at 500 MB and refuses below
1.5 GB free. Proof 4.4: two engine pins via the generic dispatcher match the VM
baseline exactly. Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-05 08:14:57 +02:00
Marius CiepluchandClaude Opus 5.5 5dc86c2f3f merge(unseen-eval): groups 1-2 and 4-6 -- used set, sourcing with resume, AX batch dispatcher, replay task, scorer
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 20:28:15 +02:00
Marius CiepluchandClaude Opus 5.5 c6e87a8517 unseen-eval groups 4-6: generic AX batch dispatcher (engine_trace on it), replay task and image stage, scorer with adoption gate
Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 20:19:14 +02:00
Marius CiepluchandClaude Opus 5.5 48ae918dec unseen-eval group 2 (2.1-2.4): advisory sourcing, vulnerability-introducing ranges, ordinary-commit draw, draft manifest and slices; journal with resume
Implemented with Codex. Sourcing run (2.5) next, by the coordinator.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 19:51:30 +02:00
Marius CiepluchandClaude Opus 5.5 8213d3baad unseen-eval group 1: used set (1,029 repositories, every source read), extended guard, usage ledger; record 01
Implemented with Codex; task 1.5 run by the coordinator against the real
sources (S3 memory read-only).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 19:17:35 +02:00
Marius CiepluchandClaude Opus 5.5 60512fff88 spec(unseen-repo-evaluation): design and tasks approved; run through the baseline
Decisions: long-deadline arm (300 s), adoption needs no unseen regression
(Req 5.4), permissive repositories by name only, two AX lanes. Also adds the
trained-classifier and joint-optimisation arms to learned-decision-engine.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 18:48:13 +02:00
Marius CiepluchandClaude Opus 5.5 d4467f4236 steering: roadmap re-centred on generalisation to unseen repositories (G0-G5); specs updated; unseen-repo-evaluation brief and requirements (approved)
Maintainer 2026-10-04: the safety net is for unseen code and the A/B tests exist
to prove the approach generalises; infrastructure work had drifted from that.
Runs are designed for AX on kube-ax, kind is the fallback.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 17:34:57 +02:00
Marius CiepluchandClaude Opus 5.5 ee422b63e0 spec(learned-decision-engine): fix-mechanisms plan approved -- discharge facts first, >=2 proposing repos, per language
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 17:25:35 +02:00
Marius CiepluchandClaude Opus 5.5 02b2cbf99c engine: retry a pin once when its AX sandbox dies, then hand it to the VM lane
Two engine actors died at gVisor level during a 12-pin run (AX phase Failed, no
result). Implemented with Codex.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-04 17:14:20 +02:00