When a path demo does not serve ${served_root}, the keystone control fails
(SideRecord.control_observed False) but the search only reported a generic
inconclusive, so the agent kept making the same wiring mistake. The pilot now
surfaces the affected side's control_observed in the verify result, and the
coordinator appends one advisory, deduplicated "served-root" hint (visible in the
board snapshot the agent reads) telling it to set the served-directory config to the
${served_root} placeholder. Advisory only: verdicts, _end, the oracle, the differential
and the demo validator are unchanged; the hint fires only when control_observed is False.
Tests: coordinator + pilot_run suites 78 passed; ruff clean; mypy unchanged (pre-existing
benchmarks errors only). Host-side change, no image rebuild.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Oracle broadening works in the agent: demo now targets the prefix-sibling vector. Still
inconclusive (0 false proofs, $0.29): the agent set the served-dir env var to its own
value, not ${served_root}, so the sibling canary was outside the served tree. Served-root
wiring is nondeterministic (a prior run wired it). Last-mile agent-reliability gap.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Plant a sibling canary so prefix-bypass path traversals (startswith without separator)
are demonstrable; parent canary + differential unchanged; glance still demonstrates.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The path oracle only modelled classic parent traversal (a `canary` file in the parent of
${served_root}). A common real path-traversal class is the prefix-check bypass:
startswith(base) without a trailing separator, which the fix closes with base + os.sep;
it reaches a SIBLING whose path string-extends the served directory, not the parent.
PathOracle now also plants a sibling canary `public_leak` beside the served `public`
(same nonce, 0600), so a prefix-bypass demo leaks it on the vulnerable side and the fix
blocks it. ORACLE_GUIDE names the second target and tells the agent to pick the vector
its code shows. One shared nonce stays correct: a canary leaks only when the demo's own
request reaches it, and each demo targets one vector whose fix blocks it on the fixed side.
observe()/capture()/the differential/SideRecord are UNCHANGED. Local glance ceiling proof
still demonstrates (parent canary, 3/3 vuln, 0/3 fixed, control on both sides). Gate:
worker+control+verify suites 106 passed / 4 socket-skips; ruff clean; mypy clean on oracles.py.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Served-root guidance validated: authored demo now references ${served_root}. Still
inconclusive (0 false proofs); agent spread budget across vectors and hit run_ceiling
(164 calls, $1.47/$1.50) without converging, and the committed traversal did not
differential. Remaining = per-app exploit-authoring + convergence (the layer the $0 local
glance proof did by hand).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Agent guidance: point the app's served directory at the ${served_root} placeholder via
arg/env/config so the planted observable is reachable. Prompt strings only; 37 worker tests pass.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Path searches authored and verified many demos but all came back with no differential:
the demo never pointed the app's served directory at ${served_root}, so the observable
(which lives relative to ${served_root}) was outside the served tree and unreachable. The
agent had set an app env var to a value of its own; the guide only showed the --dir form.
ORACLE_GUIDE now states the agent MUST set the app's served/root to the ${served_root}
placeholder via whichever mechanism the app exposes (command-line argument, environment
variable, or config written to start.files), not only --dir, and that serving elsewhere
makes the observable unreachable. Verify-time-only clarification, observable names, and all
prohibitions are kept. Prompt strings only.
Tests: tests/test_search_worker.py 37 passed; ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Fair-share budget validated: the Python agent completed the full loop and verified 18
demos within $0.74 of $1.50 (0 verify tasks before). Still inconclusive, 0 false proofs.
Root cause of no-differential: the authored demo never references ${served_root}; the
agent pointed the app's served dir at its own value, so the observable was outside the
served tree. Next: guidance to wire the served dir to ${served_root} for the path oracle.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Each pilot search gets a fair share of --ceiling-usd with settled rollover, so one
expensive search can no longer starve the others; total stays within the ceiling.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
--ceiling-usd was one shared SpendBudget passed to every search, so an expensive
search drained the pool and starved later ones: in the last 2-pair/$2 run the first
search spent $1.60 and the second hit the ceiling at $0.34 after finishing its analysis.
The driver now allocates a dynamic fair share per search (remaining_ceiling /
searches_not_yet_started), rolls unused settled budget forward, caps each search at its
share, and keeps total settled <= ceiling. A search now gets room to author and verify
after it finds the path.
Driver-only change (benchmarks/search/pilot_run.py) plus tests. tests/test_search_pilot_run.py
39 passed; ruff clean; mypy unchanged (pre-existing benchmarks errors only).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2 path searches (ts/py), vulnerable, cluster, $1.95 of $2. The trap fix is validated
behaviourally: ts pair 9 now authors a demo (verify_tasks 0->2; fails could_not_build on
entrypoint), py pair 13 drops 118->36 calls and its board shows a complete route-to-sink
+ entrypoint trace but hits the per-run budget ceiling before authoring/verifying. Still
0 demonstrations, 0 false proofs. Payload-free aggregate; raw stays gitignored. Next
levers: per-run budget/step allocation for demo-write+verify after analysis; demo
entrypoint/build (start.path) for compiled/served apps.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Agent guidance so path searches stop trying to confirm the oracle observable during
exploration (where it is absent) and instead author a demo for the differential verifier.
Prompt strings only; validators untouched; 37 worker tests pass.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The parity fix named each oracle's observable, but path searches then spent their
whole budget without authoring a demo: the agent tried to confirm reachability during
exploration, where the oracle fixture does not exist (it is injected only in the
verifier's sandbox at verify time), concluded "unreachable", and never committed a demo.
ORACLE_GUIDE now states the fixture/observable is injected at VERIFY time only and is
absent during exploration; its explore-time absence never means unreachable or justifies
ending the search. The agent confirms the mechanism (sink reachability, access to data the
app should not expose) from app code and its own sandbox test data, targets the observable's
known verify-time location in the demo, and relies on differential verification as the only
proof. All prior prohibitions (no out-of-band/off-disk access, no input capture, no revision
sniffing) and the parity observable names are kept. Adjacent wording shortened to fit.
Tests: tests/test_search_worker.py 37 passed; ruff clean; mypy unchanged (pre-existing
verify.py lambda note only). Prompt strings only; validators untouched.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
3 path searches (php/ts/py), vulnerable side, cluster verify lane, $2.95 of $3.
Payload-free aggregate; raw boards/demos stay gitignored. Finding: the parity fix
produced directed, target-aware search (the agent reasons about reaching the named
observable and inspects the app's serving guard), but no demonstration landed -- all
three spent their per-pair allowance before a differential, and two wrote zero verify
demos (budget consumed in reason/explore). Next levers: allowance/budget split toward
demo-writing, entrypoint discovery for php, and the explore-vs-verify path layout the
py agent flagged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The path family's agent guidance named no target, while command (${marker_dir}/marker)
and ssrf (${callback_url}) disclose theirs, so path searches could not author a demo
that surfaces the observable through the app. ORACLE_GUIDE now names it (canary in the
parent of ${served_root}, matching PathOracle.prepare). The reason/explore constraints
are refined to forbid only out-of-band access (direct disk reads, non-app channels),
input capture, and revision sniffing, while allowing the app to disclose it through the
vulnerability under test. Prompt strings only; the demo validator and its _PRIVATE guard
are untouched, so direct references to verifier paths remain blocked.
Tests: tests/test_search_worker.py 37 passed; ruff clean; mypy unchanged (pre-existing
verify.py lambda note only).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Outcome-only record of the first end-to-end demonstrated verdict and the two
masking defects (regenerated npm lock pinning an incompatible mime; agent demo
targeting the wrong file). Payloads stay in the gitignored private area.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Honest serve signal for the HTTP verifier: SideRecord.control_observed records
whether a benign verifier-planted in-root marker came back through the app's
interface, as its own guarded app start that never changes the verdict. A
2xx-based block was rejected (any HTTP status counts as alive). Gate green;
ceiling re-proof on glance still demonstrated with control_observed on both
sides. The search task image needs a rebuild before the next cluster pilot.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The HTTP verifier could not distinguish a mis-wired or non-serving app from a
genuine no-effect: both yielded `inconclusive`. This adds a positive control for
the path oracle.
- PathOracle plants a benign in-root marker file (random token from os.urandom,
distinct from the secret canary) and exposes it via a new Oracle.control()
hook (default None; only the path oracle implements it today).
- On the first run of an HTTP side, _side runs a benign GET for that marker as
its own fully-guarded app start. The marker returning proves the interface
serves the fixture; its absence — or any control failure — is recorded in the
new SideRecord.control_observed field but never changes the verdict, since a
valid app may answer only its own routes. A crash is still caught by the
existing could_not_run path.
A 2xx-based "nothing served" block was rejected: the verifier deliberately
treats any HTTP status as the app being alive (a 503-only app is a valid
observed side), so status-based blocking would false-fail valid apps. Soundly
converting a mis-wired-but-responding app into could_not_run requires a
demo-declared control, which is a separate follow-up.
Tests: new tests/test_search_control.py covers marker observed, absent marker
not blocking, control failure not blocking a working attack, and crash still
could_not_run. Full search-verify/probe/worker suites pass; ruff clean; mypy
unchanged (only the pre-existing file_bindings lambda note).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Replays a stored demo through the executor task and the verifier lane for one
side or both, with per-phase timings, so verifier changes are checked on the
exact image before any paid search.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The pip recipe ran under the system Python 3.13 while apps started under the
venv 3.12, so compiled extensions never loaded. Both now use sys.executable.
Package-style starts see the checkout root; NODE_PATH lists the project
node_modules under the checkout before the products mount.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Demos may reference ${served_root}, ${database_path}, ${marker_dir} or
${callback_url}; the verifier substitutes them per side, so apps that take a
directory or database argument can be wired to the oracle. Python apps start
under the verifier interpreter (the one pip built the products for): the image
carries a second system Python whose ABI tag broke compiled extensions.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Archives are gzipped and uploaded in parts of at most 90 MB with per-part
digests (edge request limit); pip installs ignore the task venv so transitive
dependencies land in products; the verifier sets NODE_PATH and the Java
classpath from products when starting the app.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
HTTP start waits up to 90 s (CLI 30 s) with TCP then HTTP probing; app stdout
and stderr tails are recorded; tool-call turns get 4096 output tokens with one
retry on truncation; tasks get 40 steps with reminders.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The kind default "pin" had produced ousast-engine-pin-search-verify-* names,
which the operator per-host egress rules do not match.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Built projects spool to /workspace without package caches (cap 1 GiB, both
sides); a plain-text model reply is fed back once; mailbox PUT/GET retry for up
to 90 s with idempotent re-send by seq; executor tasks are deleted on every exit
path.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Demos may set a bounded environment (loader variables denied) and up to 8 small
data files under .demo/; the reason step sees conclusions and may not re-propose
them; a task spend limit concludes its intent, not the search; finish wins over
stray tool calls.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
All five oracles demonstrated, gaming demos inconclusive, 38 s, ~500 MB RSS.
The probe dispatcher accepts a skipped isolation check in task-boundary mode.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The agent passes a manifest file or project directory; pip uses -r for
requirements files and the directory for pyproject/setup, npm ci/install runs in
the directory, composer/maven/gradle likewise. Build and verify failures carry
phase, exit code and a sanitised stderr tail.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
On kube-ax the probe detected task-boundary and then failed the bwrap preflight it
no longer needs; now it skips the preflight, runs every oracle pair through the
verifier path a task uses, and reports ok only if every real demo is
demonstrated and no gaming demo is. Fixes a double fixture-path translation.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Demos declare their oracle within the family allowed set (injection: sql or
command; output_encoding: xss; untrusted_destination: ssrf; path: path);
builds and app starts use the search memory budget instead of the 256 MB tool
limit (npm died with SIGTRAP); malformed tool-call JSON is fed back once.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
budget_spent carries end_detail (spend, task_spend, reason_rounds, ...); a task
whose allowance cannot cover the next call finishes with its facts; per-task
metrics in pilot records; input reservation bytes/2 + 1024 per message (measured
3-4x over-reservation before).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
