Files
Open_Ultra_SAST/benchmarks/measurements
Marius CiepluchandClaude Fable 5.1 dee23c23f5 measure: both arms over the access-control training pool (2 of 5 pushes, 4 of 48 locations each); runner books real cost
DSPy baseline: 2/5 pushes, 4/48 locations, 14 of 90 ordinary pushes flagged, 25 suspicions (3 true), 11 invalid-request
failures in shopper and trigger.dev, $6.13. Pilot hunter: 2/5 pushes, 4/48 locations, 99 of 100 ordinary pushes
flagged, 594 findings (3 true). Different pushes caught; nothing in trigger.dev.

The runner booked usd 0.0 per change so --budget-usd never stopped a run; it now prices every LM call of the change
at the registered prices and writes usd into the record.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-14 01:14:46 +02:00
..