feat(learning-harness): aggregation, noise handling and honest denominators

Task 2.4. Per family, and per slice when asked: outcome counts, recall, silence,
Youden, a directional-bias index and recall at a fixed false-positive ceiling.
Rows that cannot be scored are listed by reason and leave every denominator, and
a family with nothing scorable reports no rate rather than a perfect one.

Two numbers exist because Youden cannot tell apart a detector that flags
everything from one that says everything is safe: the directional-bias index
separates them, and recall at a fixed ceiling is forfeited entirely once a
family leaks more than the budget allows.

Change against a baseline is not a difference of means. Flips are counted per
pair in both directions and a two-sided exact sign test decides whether the
change is bigger than the noise, so one flip in ten reads as noise and six reads
as a real regression.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-06 09:33:15 +02:00
co-authored by Claude Opus 5
parent 241af31a9a
commit 95ffd901e4
3 changed files with 230 additions and 9 deletions
+5 -5
View File
@@ -1,9 +1,9 @@
{
"feature_name": "learning-harness",
"created_at": "2026-09-06T06:32:41Z",
"updated_at": "2026-09-06T06:57:33Z",
"updated_at": "2026-09-06T07:00:52Z",
"language": "en",
"phase": "tasks-generated",
"phase": "implementation",
"approvals": {
"requirements": {
"generated": true,
@@ -15,10 +15,10 @@
},
"tasks": {
"generated": true,
"approved": false
"approved": true
}
},
"ready_for_implementation": false,
"ready_for_implementation": true,
"depends_on": [
"pair-corpus-honesty",
"corpus-seeded-mechanisms",
@@ -26,5 +26,5 @@
"tree-sitter-overlay-extra",
"harnessx-self-improving-rulesets"
],
"approval_note": "2026-09-06 requirements approved by the maintainer in session ('approve the requirements and run the design phase'); 2026-09-06 design approved by the maintainer in session ('approve the design and generate the tasks')"
"approval_note": "2026-09-06 requirements approved by the maintainer in session ('approve the requirements and run the design phase'); 2026-09-06 design approved by the maintainer in session ('approve the design and generate the tasks'); 2026-09-06 tasks approved by the maintainer in session ('approve the tasks and go on with the implementation')"
}