From 1b291e81b75141b075335c6bcd5a4af5b8be1650 Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Wed, 30 Sep 2026 18:01:06 +0200 Subject: [PATCH] spec(learned-decision-engine): Req 7 -- judged on unseen repositories as a repo safety net Maintainer: the tool will never run against the test cases it is trained with; its purpose is a repo safety net. Repository-grouped folds, repository-agnostic features, BLOCK/ADVISORY operating points for changed code, out-of-repo calibration, adaptation kept separate. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo --- .../learned-decision-engine/requirements.md | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/.kiro/specs/learned-decision-engine/requirements.md b/.kiro/specs/learned-decision-engine/requirements.md index 9a48fef..ba38b86 100644 --- a/.kiro/specs/learned-decision-engine/requirements.md +++ b/.kiro/specs/learned-decision-engine/requirements.md @@ -110,3 +110,23 @@ rule hits. 2. Without a trained model available the tools fall back to today's behaviour and say so in the report. 3. Hand-written rules remain as signal producers; changing their enabled/shadow status no longer changes what is reported unless the engine's decision changes. + +### Requirement 7: Built for unseen repositories -- a repo safety net + +**User Story:** As the maintainer, I want the engine judged by how it behaves on repositories it never saw, because +"the tool will never run against the test cases it's being trained with. it's purpose is a repo safety net." +(maintainer, 2026-09-30). + +#### Acceptance Criteria + +1. Every reported metric comes from folds grouped by repository across all corpora: a repository in an evaluation + fold contributes no row to training. +2. Features are repository-agnostic by an allow-listed schema (no repository, path, file, function or commit + identities), enforced by a test. +3. Two operating points serve the safety net: BLOCK, chosen for >= 95% precision on unseen repositories, and + ADVISORY, trading precision for recall; both reported with out-of-repository intervals. The decision unit is a + candidate in changed code (`pre-push` delta), with whole-repository scans as a secondary mode. +4. Calibration is checked on held-out repositories; where it does not hold, only ADVISORY is offered for that + family. +5. Any adaptation to a user's own repository (facts reuse, their dismissals) is opt-in, separate from the + generalisation model, and never contributes to its reported numbers.