mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
Marius CiepluchandClaude Opus 5
95ffd901e4
Task 2.4. Per family, and per slice when asked: outcome counts, recall, silence, Youden, a directional-bias index and recall at a fixed false-positive ceiling. Rows that cannot be scored are listed by reason and leave every denominator, and a family with nothing scorable reports no rate rather than a perfect one. Two numbers exist because Youden cannot tell apart a detector that flags everything from one that says everything is safe: the directional-bias index separates them, and recall at a fixed ceiling is forfeited entirely once a family leaks more than the budget allows. Change against a baseline is not a difference of means. Flips are counted per pair in both directions and a two-sided exact sign test decides whether the change is bigger than the noise, so one flip in ten reads as noise and six reads as a real regression. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG