mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
Marius CiepluchandClaude Opus 5.5
97afa1d220
Decision-engine task 6.7 (partial). After the advisory-fix harvest (memory 724 -> 904 static examples), one compile per evaluable family and an outer grouped 5-fold evaluation of that family's own candidates, input profile v1, priors off, k = 5, cross-fitted Platt, nested operating points (repository-cluster bootstrap 95%): family n AUC(s) ADVISORY R / P 'vulnerable' P / R ECE canary injection 110 0.77 [0.67, 0.88] 0.88 / 0.57 0.84 / 0.36 0.093 0.77 untrusted_dest. 62 0.85 [0.76, 0.94] 0.84 / 0.73 0.96 / 0.66 0.071 0.87 path 35 0.89 [0.79, 0.97] 0.86 / 0.69 0.89 / 0.76 0.119 0.90 output_encoding 44 0.91 [0.82, 0.97] 0.86 / 0.81 0.94 / 0.55 0.045 0.90 deserialization 46 0.90 [0.82, 0.97] 0.83 / 0.76 0.93 / 0.83 0.095 0.93 access_control 37 0.96 [0.90, 1.00] 0.91 / 0.91 1.00 / 0.61 0.076 0.83 BLOCK offered for none: calibration fails everywhere except output_encoding, where no fold reaches a Wilson lower bound of 0.95. Within paired groups (a function and its own fix) AUC is lower, 0.72-0.93: every single-label group in the new families holds positives only. Injection learning-curve point, paired (same program, candidates, folds): memory 724 -> 904, AUC 0.76 -> 0.77 (no injection example was added). $2.88 metered DeepSeek (ceiling $8; balance 17.73 -> 16.73), $0.0009 embeddings. Instrument fix: compiled demonstrations now follow the near-duplicate rule (cos >= 0.98) that retrieval and the pre-call assertion apply; the assertion had stopped the untrusted_destination evaluation. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo