Files
Open_Ultra_SAST/benchmarks/independent
Marius CiepluchandClaude Fable 5.1 072a102b20 measure(independent): triage + per-file batching + two-pass agreement -- 16/20 sites, flicker cut 26 -> 9, 34% cheaper
verify_batched.py replaces the per-candidate verifier's design:
- triage per file (no tools): where does the operation's data originate;
  internal/constant candidates are not verified;
- one 6-step hunt per file for up to 6 candidates, a verdict per operation;
- two independent passes; only agreement is reported, the rest is "disputed".

On the 46-candidate validation set ($1.01):
- declared sites: 16 of 20 agreed, 12 of 15 cases (the per-candidate design
  confirmed 11). Budibase and Froxlor are new; Unleash dropped to one pass of two.
  winml is triaged out as "constant": a CORS literal, which the reachability
  question does not fit.
- the 26 pgAdmin candidates that flickered between single runs: 2 triaged out,
  9 agreed in both passes, 4 disputed, 13 quiet.
- $0.022 per candidate for two passes vs $0.033 under the old design; on a file
  with several candidates, 2.3x cheaper per pass.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-09-29 10:22:11 +02:00
..