marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

measurements: py re-validation -- budget fix works (18 verify tasks, 0 before); demo didn't wire served_root [skip ci]

Browse Source 
Fair-share budget validated: the Python agent completed the full loop and verified 18
demos within $0.74 of $1.50 (0 verify tasks before). Still inconclusive, 0 false proofs.
Root cause of no-differential: the authored demo never references ${served_root}; the
agent pointed the app's served dir at its own value, so the observable was outside the
served tree. Next: guidance to wire the served dir to ${served_root} for the path oracle.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-10-07 09:23:33 +02:00
co-authored by Claude Opus 4.8
parent f3ce5be8c4
commit e387ca16a2
1 changed files with 14 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benchmarks/measurements/2026-10-07-search-pilot-py-revalidate/record.json
+14
View File
@@ -0,0 +1,14 @@
{
"record": "2026-10-07 search-with-proof: Python pair re-validation of the per-search fair-share budget fix",
"ceiling_usd": 1.5,
"pair": {"index": 13, "language": "python", "family": "path"},
"outcome": "inconclusive",
"end_detail": "demonstrations",
"verify_tasks": 18,
"model_calls": 98,
"settled_usd": 0.7411,
"wall_seconds": 2722,
"false_proofs_fixed_side": 0,
"finding": "Budget fix VALIDATED: with a fair per-search share the Python agent completed the full loop and ran 18 verify tasks (0 before), authoring and verifying demos within budget. Still inconclusive ('no consistent differential'), zero false proofs. Structural root cause confirmed from the authored demo: it does NOT reference ${served_root} anywhere; the agent set the app's served-directory config (an app env var) to a value of its own rather than to the oracle placeholder ${served_root}, so the app served the wrong tree and the planted observable (which lives relative to ${served_root}) was never reachable. Next guidance lever: require the agent to point the application's served/root directory at ${served_root} (via arg, env, or config) for the path oracle, so the observable is in the served tree.",
"progression": "parity fix -> agent targets observable; verify-time-only fix -> agent stops the explore reachability trap; budget fair-share -> agent authors+verifies within budget; remaining -> wire served dir to ${served_root}."
}