mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 11:49:28 +00:00
Marius CiepluchandClaude Fable 5.1
3259d0469f
Measured 2026-09-06 on the vibe-py holdout: the existing tool hunter, with no learning, scores 12/14 pair-correct (Youden +0.857, no leaks) once detection and silence are keyed on the labeled class and function and the byte-identical twins are excluded; the overlay scores 5/14 on the same rows. The published hunter number (0/17, Youden -0.588) was a scoring artifact: the scorer wanted the literal CWE id in the finding text and the prompt never asked for it. The brief proposes an auto-classifier as the routing key, one evolved hunter configuration per class, a class-aware scorer with unscorable denominators, the split enforced by construction, and the HarnessX evolve loop with per-class attribution as the round primitive. Raw measurements are committed under reports/measurements. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG