From e387ca16a266801e5ac23bbfa0ed0c10aa0e6635 Mon Sep 17 00:00:00 2001
From: Marius Ciepluch <11855163+norandom@users.noreply.github.com>
Date: Wed, 7 Oct 2026 09:23:33 +0200
Subject: [PATCH] measurements: py re-validation -- budget fix works (18 verify
 tasks, 0 before); demo didn't wire served_root [skip ci]

Fair-share budget validated: the Python agent completed the full loop and verified 18
demos within $0.74 of $1.50 (0 verify tasks before). Still inconclusive, 0 false proofs.
Root cause of no-differential: the authored demo never references ${served_root}; the
agent pointed the app's served dir at its own value, so the observable was outside the
served tree. Next: guidance to wire the served dir to ${served_root} for the path oracle.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---
 .../record.json                                    | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
 create mode 100644 benchmarks/measurements/2026-10-07-search-pilot-py-revalidate/record.json

diff --git a/benchmarks/measurements/2026-10-07-search-pilot-py-revalidate/record.json b/benchmarks/measurements/2026-10-07-search-pilot-py-revalidate/record.json
new file mode 100644
index 0000000..f048cdd
--- /dev/null
+++ b/benchmarks/measurements/2026-10-07-search-pilot-py-revalidate/record.json
@@ -0,0 +1,14 @@
+{
+  "record": "2026-10-07 search-with-proof: Python pair re-validation of the per-search fair-share budget fix",
+  "ceiling_usd": 1.5,
+  "pair": {"index": 13, "language": "python", "family": "path"},
+  "outcome": "inconclusive",
+  "end_detail": "demonstrations",
+  "verify_tasks": 18,
+  "model_calls": 98,
+  "settled_usd": 0.7411,
+  "wall_seconds": 2722,
+  "false_proofs_fixed_side": 0,
+  "finding": "Budget fix VALIDATED: with a fair per-search share the Python agent completed the full loop and ran 18 verify tasks (0 before), authoring and verifying demos within budget. Still inconclusive ('no consistent differential'), zero false proofs. Structural root cause confirmed from the authored demo: it does NOT reference ${served_root} anywhere; the agent set the app's served-directory config (an app env var) to a value of its own rather than to the oracle placeholder ${served_root}, so the app served the wrong tree and the planted observable (which lives relative to ${served_root}) was never reachable. Next guidance lever: require the agent to point the application's served/root directory at ${served_root} (via arg, env, or config) for the path oracle, so the observable is in the served tree.",
+  "progression": "parity fix -> agent targets observable; verify-time-only fix -> agent stops the explore reachability trap; budget fair-share -> agent authors+verifies within budget; remaining -> wire served dir to ${served_root}."
+}