From d1fe78987b5eb01b9fefb3b3dd6d6eb89c654f62 Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Wed, 7 Oct 2026 11:52:19 +0200 Subject: [PATCH] measurements: xiaomusic vuln search -- oracle fix landed (agent targets prefix-sibling) but served-dir wiring missed this run [skip ci] Oracle broadening works in the agent: demo now targets the prefix-sibling vector. Still inconclusive (0 false proofs, $0.29): the agent set the served-dir env var to its own value, not ${served_root}, so the sibling canary was outside the served tree. Served-root wiring is nondeterministic (a prior run wired it). Last-mile agent-reliability gap. Co-Authored-By: Claude Opus 4.8 --- .../record.json | 12 ++++++++++++ 1 file changed, 12 insertions(+) create mode 100644 benchmarks/measurements/2026-10-07-search-pilot-xiaomusic-vuln/record.json diff --git a/benchmarks/measurements/2026-10-07-search-pilot-xiaomusic-vuln/record.json b/benchmarks/measurements/2026-10-07-search-pilot-xiaomusic-vuln/record.json new file mode 100644 index 0000000..ebbdf1d --- /dev/null +++ b/benchmarks/measurements/2026-10-07-search-pilot-xiaomusic-vuln/record.json @@ -0,0 +1,12 @@ +{ + "record": "2026-10-07 xiaomusic vulnerable-side search on the oracle-broadened image (prefix-bypass canary present)", + "image": "ghcr.io/norandom/ousast-search-task@sha256:e2c8b460...", + "ceiling_usd": 1.5, + "outcome": "inconclusive", + "verify_tasks": 6, + "model_calls": 49, + "settled_usd": 0.2922, + "false_proofs_fixed_side": 0, + "finding": "Oracle broadening landed in the agent: the authored demo now TARGETS the new prefix-sibling vector (references the sibling leak file) instead of the parent-only canary. But it did not demonstrate: the agent set the app's served-directory env var to a value of its OWN instead of the ${served_root} placeholder, so the app served the wrong tree and the sibling canary was outside it (unreachable). This is the served-root-wiring behaviour recurring NONDETERMINISTICALLY (a prior run did wire ${served_root}; this one did not). Cheap ($0.29), ended naturally at 49 calls, not on budget. Remaining gap = agent RELIABILITY on one demo detail: the served-directory config VALUE must be the literal ${served_root}, not the agent's explore-time test path.", + "progression": "oracle now covers the vuln class and the agent targets it; last-mile blocker is reliably setting the served-dir config to ${served_root}." +}