marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-21 14:09:27 +00:00
Code Issues Packages Projects Releases Wiki Activity

spec(unseen-repo-evaluation): design and tasks approved; run through the baseline

Browse Source 
Decisions: long-deadline arm (300 s), adoption needs no unseen regression
(Req 5.4), permissive repositories by name only, two AX lanes. Also adds the
trained-classifier and joint-optimisation arms to learned-decision-engine.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-10-04 18:48:13 +02:00
co-authored by Claude Opus 5.5
parent d4467f4236
commit 60512fff88
5 changed files with 780 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/specs/unseen-repo-evaluation/requirements.md
+4
View File
@@ -101,3 +101,7 @@ worn out by tuning.
slice, recorded with its freeze digest.
3. Every learned-decision-engine experiment from now on reports its unseen-repo score from this pool next to its
held-out-fold score.
4. An arm is adopted only if, on the untouched slice that qualifies it, it shows no significant regression against
the current production arm: catch at the 5% false-alarm budget not significantly lower, and for an arm with one
operating point the false-alarm rate not significantly higher (exact paired tests, two-sided, alpha 0.05).
Reporting the unseen score alone is not enough to adopt. *(amended 2026-10-04, maintainer decision)*