mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 19:59:26 +00:00
feat(learning-harness): aggregation, noise handling and honest denominators
Task 2.4. Per family, and per slice when asked: outcome counts, recall, silence, Youden, a directional-bias index and recall at a fixed false-positive ceiling. Rows that cannot be scored are listed by reason and leave every denominator, and a family with nothing scorable reports no rate rather than a perfect one. Two numbers exist because Youden cannot tell apart a detector that flags everything from one that says everything is safe: the directional-bias index separates them, and recall at a fixed ceiling is forfeited entirely once a family leaks more than the budget allows. Change against a baseline is not a difference of means. Flips are counted per pair in both directions and a two-sided exact sign test decides whether the change is bigger than the noise, so one flip in ten reads as noise and six reads as a real regression. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
co-authored by
Claude Opus 5
parent
241af31a9a
commit
95ffd901e4
@@ -1,9 +1,9 @@
|
||||
{
|
||||
"feature_name": "learning-harness",
|
||||
"created_at": "2026-09-06T06:32:41Z",
|
||||
"updated_at": "2026-09-06T06:57:33Z",
|
||||
"updated_at": "2026-09-06T07:00:52Z",
|
||||
"language": "en",
|
||||
"phase": "tasks-generated",
|
||||
"phase": "implementation",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
@@ -15,10 +15,10 @@
|
||||
},
|
||||
"tasks": {
|
||||
"generated": true,
|
||||
"approved": false
|
||||
"approved": true
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": false,
|
||||
"ready_for_implementation": true,
|
||||
"depends_on": [
|
||||
"pair-corpus-honesty",
|
||||
"corpus-seeded-mechanisms",
|
||||
@@ -26,5 +26,5 @@
|
||||
"tree-sitter-overlay-extra",
|
||||
"harnessx-self-improving-rulesets"
|
||||
],
|
||||
"approval_note": "2026-09-06 requirements approved by the maintainer in session ('approve the requirements and run the design phase'); 2026-09-06 design approved by the maintainer in session ('approve the design and generate the tasks')"
|
||||
"approval_note": "2026-09-06 requirements approved by the maintainer in session ('approve the requirements and run the design phase'); 2026-09-06 design approved by the maintainer in session ('approve the design and generate the tasks'); 2026-09-06 tasks approved by the maintainer in session ('approve the tasks and go on with the implementation')"
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user