The resilient draw only covered extraction. A per-candidate transport
failure in the decide/eligibility phase -- get_commit on an unresolvable
fix ref raising ValueError("http_failure") (observed: ljharb/shell-quote
commit 1518179) -- still aborted the whole draft, because decide only
caught source.Rejected and eligibility.RepositoryNotFound.
decide now catches that http_failure and returns a "decide_http_failure"
skip, and the run loop feeds decide + extract per-candidate instrument
failures into one systemic guard (25 with zero successes, or >200), so a
single bad commit lookup skips while a broken instrument (e.g. a network
outage failing every lookup) still aborts loudly.
Full unseen draft suite passes (48), ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Individual repositories in the advisory population raise per-candidate
extract.InstrumentFailure during extraction (git_exit_128 on an
unfetchable fix commit, clone_size_cap, and others), and each one
aborted the whole 300-repo draft. The draw is healthy; one bad repo
should not reset it.
extract_repository now also catches extract.InstrumentFailure and
OSError and records them as "extract_<reason>" rejections, so the draw
skips that repo and continues (the reason, e.g. extract_git_exit_128,
shows in the rejections counter). ExtractionTimeout keeps its existing
extraction_timeout skip; any other exception still propagates as a real
abort, so genuine code bugs are not masked.
A systemic guard preserves the instrument rule: run() raises
InstrumentFailure("systemic_extraction_failure") if 25 extraction
attempts fail with zero successes (e.g. git missing, auth broken) or if
extraction failures exceed 200 overall. A handful of bad repos with many
successes never trips it.
190 unseen tests pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A pathologically slow repository (observed: tensorzero, 183s to clone,
then `git blame -C` past the 180s command limit) raised
subprocess.TimeoutExpired, which Git.run mislabeled as a fatal
InstrumentFailure("git_launch_failed") and aborted the whole 300-repo
draft. The same slowness stalled throughput (a 54-minute no-progress
gap).
Give each clone a per-repo wall-clock budget (--extract-deadline,
default 240s, plumbed Clones -> Git, counting the initial clone). A
timeout -- a command exceeding the remaining budget, or the budget
elapsing -- raises ExtractionTimeout(Rejected), which the existing
rejection handler turns into an "extraction_timeout" skip so the draw
moves to the next candidate. Genuine launch failures and non-zero git
exits stay fatal; timeouts and launch failures are now distinct. The
deadline is recorded on the report only, not the journal inputs, so
changing it does not refuse resume.
181 unseen tests pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
disk_bytes() summed the clone tree with rglob + stat() as a size cap
before every git command. On a blob:none clone, `git show` triggers an
on-demand blob fetch that writes and deletes transient
.git/objects/pack/*.rev files, so a path rglob had just listed could be
gone by the stat() call -> FileNotFoundError aborted the whole draft as
an instrument_failure. The race can hit any candidate, so it was a
non-deterministic draft-killer (observed extracting a real npm repo).
Skip a file that raises OSError mid-walk; the size cap is unaffected.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The GitHub /advisories endpoint ignores the `page` query parameter: it
paginates with the Link header's rel="next" cursor. The page-number
pager re-fetched the same first 100 rows forever and never terminated,
so enumeration never completed and no draft converged (a run fetched
5137 "pip pages" holding only 100 unique advisories).
Surface the successful response headers from JSONTransport, and walk the
rel="next" cursor per ecosystem until it is absent. The ecosystem /
published>=2019 / bare-number cwes filters are preserved on the first
request. Journal keys stay ecosystem:n (now cursor-walk positions) with
the saved next URL for resume; journal_version 4 refuses older journals.
Live smoke via the production pager: 300 unique advisories pulled in 3
API calls, advancing past the first page. Offline: 170 unseen tests
pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
enumerate_advisories paged the entire global GitHub reviewed-advisory
database (every ecosystem, every year, every CWE) before extraction, so
a 300-repo draft never converged -- a live run reached 21,962 advisories
in 4.5h with extraction never started.
Push the existing candidate() rejections into the query: page per
ecosystem, published>=2019-01-01, and cwes restricted to the seven
families. The cwes filter uses bare CWE numbers (89), which is what the
GitHub advisories API matches -- "CWE-89" silently returns zero. The
whole cwes filter is omitted if the taxonomy ever holds a non-exact
token, so a selectable advisory is never dropped. Journal version 3
(ecosystem:page keys) refuses incompatible older journals.
Live page-1 smoke (pip, npm): 100/100 rows carry a family CWE, none
published before 2019. Offline: 166 unseen tests pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A live G0 sourcing run failed on both: a transient connection/decode error aborted
the whole enumeration (the bare except raised http_failure on the first blip), and
the advisory journal stored full raw advisory JSON and reached 1.77 GB on a near-full
host. Fix: the JSON transport now retries URLError/timeout/ConnectionError/IncompleteRead/
JSONDecodeError through the same attempt-4 backoff as HTTP 5xx/429 (fatal only after the
cap or on a genuinely non-retryable error); enumerate_advisories journals only the fields
resolution/extraction consume via source.reduce_advisory (ghsa id, published/withdrawn,
type, CWE ids, package ecosystem/name for OSV, fix-commit links), with journal_version
bumped to 2 so an old full-format journal is rejected and resume-skip stays intact.
Offline: test_unseen_source/draft/extract 133 passed; ruff + format clean; mypy unchanged
(pre-existing verify.py lambda only).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A lockless npm project (e.g. glance, 2015) resolved an unpinned transitive range to the
latest major ([email protected] -> mime@4, which dropped .lookup()), so the built app crashed
serving files -- a build-reproducibility defect, not an app or agent fault. The executor
now, for the npm-install (no-lock) case only and only when .git is present, resolves
dependencies as of the checkout's HEAD commit date via `npm install --before <date>`,
giving era-appropriate, compatible versions. npm ci (lockfile present) is untouched, and
any git/date failure falls back to plain install. demo.py also recognizes
npm-shrinkwrap.json as a lock for the ci-vs-install choice.
Tests: executor + smoke 75 passed, demo 107 passed; ruff clean; mypy clean on executor.py.
Executor-side change -> image rebuild required.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
When a path demo does not serve ${served_root}, the keystone control fails
(SideRecord.control_observed False) but the search only reported a generic
inconclusive, so the agent kept making the same wiring mistake. The pilot now
surfaces the affected side's control_observed in the verify result, and the
coordinator appends one advisory, deduplicated "served-root" hint (visible in the
board snapshot the agent reads) telling it to set the served-directory config to the
${served_root} placeholder. Advisory only: verdicts, _end, the oracle, the differential
and the demo validator are unchanged; the hint fires only when control_observed is False.
Tests: coordinator + pilot_run suites 78 passed; ruff clean; mypy unchanged (pre-existing
benchmarks errors only). Host-side change, no image rebuild.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
--ceiling-usd was one shared SpendBudget passed to every search, so an expensive
search drained the pool and starved later ones: in the last 2-pair/$2 run the first
search spent $1.60 and the second hit the ceiling at $0.34 after finishing its analysis.
The driver now allocates a dynamic fair share per search (remaining_ceiling /
searches_not_yet_started), rolls unused settled budget forward, caps each search at its
share, and keeps total settled <= ceiling. A search now gets room to author and verify
after it finds the path.
Driver-only change (benchmarks/search/pilot_run.py) plus tests. tests/test_search_pilot_run.py
39 passed; ruff clean; mypy unchanged (pre-existing benchmarks errors only).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The HTTP verifier could not distinguish a mis-wired or non-serving app from a
genuine no-effect: both yielded `inconclusive`. This adds a positive control for
the path oracle.
- PathOracle plants a benign in-root marker file (random token from os.urandom,
distinct from the secret canary) and exposes it via a new Oracle.control()
hook (default None; only the path oracle implements it today).
- On the first run of an HTTP side, _side runs a benign GET for that marker as
its own fully-guarded app start. The marker returning proves the interface
serves the fixture; its absence — or any control failure — is recorded in the
new SideRecord.control_observed field but never changes the verdict, since a
valid app may answer only its own routes. A crash is still caught by the
existing could_not_run path.
A 2xx-based "nothing served" block was rejected: the verifier deliberately
treats any HTTP status as the app being alive (a 503-only app is a valid
observed side), so status-based blocking would false-fail valid apps. Soundly
converting a mis-wired-but-responding app into could_not_run requires a
demo-declared control, which is a separate follow-up.
Tests: new tests/test_search_control.py covers marker observed, absent marker
not blocking, control failure not blocking a working attack, and crash still
could_not_run. Full search-verify/probe/worker suites pass; ruff clean; mypy
unchanged (only the pre-existing file_bindings lambda note).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Replays a stored demo through the executor task and the verifier lane for one
side or both, with per-phase timings, so verifier changes are checked on the
exact image before any paid search.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The pip recipe ran under the system Python 3.13 while apps started under the
venv 3.12, so compiled extensions never loaded. Both now use sys.executable.
Package-style starts see the checkout root; NODE_PATH lists the project
node_modules under the checkout before the products mount.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Demos may reference ${served_root}, ${database_path}, ${marker_dir} or
${callback_url}; the verifier substitutes them per side, so apps that take a
directory or database argument can be wired to the oracle. Python apps start
under the verifier interpreter (the one pip built the products for): the image
carries a second system Python whose ABI tag broke compiled extensions.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Archives are gzipped and uploaded in parts of at most 90 MB with per-part
digests (edge request limit); pip installs ignore the task venv so transitive
dependencies land in products; the verifier sets NODE_PATH and the Java
classpath from products when starting the app.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
HTTP start waits up to 90 s (CLI 30 s) with TCP then HTTP probing; app stdout
and stderr tails are recorded; tool-call turns get 4096 output tokens with one
retry on truncation; tasks get 40 steps with reminders.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Built projects spool to /workspace without package caches (cap 1 GiB, both
sides); a plain-text model reply is fed back once; mailbox PUT/GET retry for up
to 90 s with idempotent re-send by seq; executor tasks are deleted on every exit
path.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Demos may set a bounded environment (loader variables denied) and up to 8 small
data files under .demo/; the reason step sees conclusions and may not re-propose
them; a task spend limit concludes its intent, not the search; finish wins over
stray tool calls.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The agent passes a manifest file or project directory; pip uses -r for
requirements files and the directory for pyproject/setup, npm ci/install runs in
the directory, composer/maven/gradle likewise. Build and verify failures carry
phase, exit code and a sanitised stderr tail.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Demos declare their oracle within the family allowed set (injection: sql or
command; output_encoding: xss; untrusted_destination: ssrf; path: path);
builds and app starts use the search memory budget instead of the 256 MB tool
limit (npm died with SIGTRAP); malformed tool-call JSON is fed back once.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
budget_spent carries end_detail (spend, task_spend, reason_rounds, ...); a task
whose allowance cannot cover the next call finishes with its facts; per-task
metrics in pilot records; input reservation bytes/2 + 1024 per message (measured
3-4x over-reservation before).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
One base (Temurin 21 JDK, node/npm/tsc, php/composer, maven, sqlite, bwrap)
built only when its Dockerfile or dependencies change, plus a browser variant;
engine/search/VM images FROM base@sha256 with the code in the last layer; the
image workflow runs on dispatch, tags, Dockerfile or pin changes, with a build
cache. Executor NPROC bound relative to the current process count.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Settled ~$0.003 per model call; worst-case reservations exhausted the $0.05 task
allowance after 2-4 calls. Now $0.40 per task, $2 per search (maintainer decision).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Zero running workers behind "no free workers available" now raises
ClusterUnavailable instead of waiting out the capacity deadline; CapacityDeadline
carries its message; cleanup failures no longer mask the original exception; the
pilot driver aborts the run on infrastructure failure before any model call.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
First live run: the reason step got ~190 tokens and returned {} -> silent
no_evidence. Now: a bootstrap intent reads the candidate and its entry point; the
reason prompt states the goal, family oracles, demo schema and must propose
intents when none are open; empty replies retry once then end as reason_refused;
failure records keep messages.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
One side per run over selected pilot pairs: board, brain on the VM with the
project DeepSeek client, executor tasks on kube-ax, verification on the VM lane
(AX verify lane switchable). Hard run ceiling with reservation before every call,
per-search and step caps; fixed side never sees the vulnerable revision; records
carry digests only. Dry run with the stub model reaches demonstrated.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Reasons carry type plus a sanitised message tail; command failures carry phase,
exit code and bounded stderr. Scratch walk tolerates directories pip removes
mid-walk; writable /workspace/home and runtime paths. Capacity retries delete a
possibly created task before resubmitting. Docker smoke checks the image first.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
20 development pairs, one per repository, 4 per language, 5 per family; identities
in a gitignored manifest, counts and digest committed. Executor smoke drives one
executor task end to end without a model and fails loudly on an unread clone.
kube-ax operator constraints as steering.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Demos are data the verifier executes (fixed build recipes, runtime+path start,
HTTP/CLI steps with captures; the canary is never referable). The agent loop with
the key never runs repository code: executor tasks without a key exchange commands
and results through presigned objects. AX capacity errors are back-pressure with at
most 2 tasks in flight; caches and builds under /workspace with a 2 GiB guard;
verifiers take one presigned archive per side. Launcher fix: private /dev in bwrap.
190 tests on the host; probe 5/5 demonstrated.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The kube-ax probe showed bwrap cannot map uids in gVisor as an unprivileged user.
As root (gVisor is the outer boundary) the sandbox uses pid/ipc/uts/net/mount
namespaces without a user namespace and drops the artefact to its own uid via
setpriv; the mode is recorded. The probe reports each namespace kind separately.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Probe runs each oracle toy pair with a real demo and two gaming demos, records
image facts, RSS and disk, fails loudly if isolation is unavailable; dispatchable
as an AX task. Worker: reason step without tools, explore tool loop whose run tool
goes through the sandbox with a scrubbed env (model key absent), spend reserved
before each call. Host probe: 5/5 real demos demonstrated, 10/10 gaming demos
inconclusive. 89 tests.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Board as immutable versioned blobs (coordinator sole writer); budget reserve/settle
before model calls; per-task and run caps; coordinator loop over an injected
executor; batch.py accepts a fixed allow-list of search variables; XSS browser
oracle; ousast-search-task image (Python, Node/TypeScript, PHP, Java 17, sqlite,
bwrap, chromium) built by CI. 103 tests on the host.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The verifier, not the artefact, observes the effect: SQL, path, command and SSRF
oracles owned by the verifier (XSS needs a browser executor, else no_oracle);
artefact exit code and output ignored; immutable checkout and dependencies;
3 runs per side; one-sided build or readiness failure is inconclusive. An
isolation failure raises IsolationUnavailable instead of reading as
could_not_build (the NPROC limit now applies inside the user namespace).
35 tests, including gaming attempts and 300 host processes.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
gVisor replays exhausted the hook's 30 s deadline in the snapshot step on a
1,725-file repository; cluster replays now use 30 s x a measured slowdown
factor, verified on a native sample. The engine runner deletes each pin's fetched
source (1.8 GB had accumulated), caps the cache at 500 MB and refuses below
1.5 GB free. Proof 4.4: two engine pins via the generic dispatcher match the VM
baseline exactly. Implemented with Codex.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Implemented with Codex; task 1.5 run by the coordinator against the real
sources (S3 memory read-only).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Two engine actors died at gVisor level during a 12-pin run (AX phase Failed, no
result). Implemented with Codex.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Through Cloudflare, signed requests for keys ending .tar/.gz/.zip/.bin reached
RustFS altered and failed with 403 (cf-cache-status BYPASS); the same signed
request straight to RustFS on the LAN returned 200. A Cloudflare cache-bypass
rule on files.because-security.com fixes it (all extensions 200, DYNAMIC); the
engine executors keep .dat as a safeguard, the plane keeps output.tar.
Implemented with Codex.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The Joern leak-audit record named spent population v1/v2 repositories, which
the reserved-repository guard forbids outside benchmarks/independent/ (no v3
repository was involved: 0 matches). Records now carry counts and SHA-256
digests. With Codex.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
A lean overlay omits otel-collector, prometheus and jaeger and disables the
exporters; on an existing cluster up.sh scales them to 0. Verified live: render
has no otel-system resources, up.sh idempotent (54 s second run), smoke Run
done, kind node 2.66 -> 1.82 GiB. Docs say server VM. Implemented with Codex.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Batch 3 of real (vulnerable function, fixed function) pointer pairs, mined at
zero model cost from reviewed GHSA advisories (pip, npm, maven, composer,
published since 2023, newest first). Same rules as batch 2, widened three
ways: a merged PR's single commit when the advisory links no commit, commits
up to 6 files / 240 lines, and two-family advisories with the family read
from the diff. Every candidate's function diff was read: 412 reviewed, 255
kept, 3 one-per-repository, 2 withheld by the reserved-population guard
(bisected with output discarded). Permissive licences only; no repository
named elsewhere under benchmarks/; no WordPress plugin.
Kept 251: injection 56, untrusted_destination 26, path 71, output_encoding
69, access_control 20, deserialization 9; python 83, typescript 48, php 46,
java 37, javascript 37. Only 12 post-cutoff: batch 2 had used up the
post-2026-06-01 pool. Memory 1133 -> 1635 static examples; complete pairs in
the whole memory 459 -> 710. Embeddings: 15 calls, $0.0030. No chat-model
call, no Docker, no Joern. Record:
benchmarks/measurements/2026-10-04-decision-engine-harvest-3.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
