marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

spec(learned-decision-engine): Req 7 -- judged on unseen repositories as a repo safety net

Browse Source 
Maintainer: the tool will never run against the test cases it is trained with;
its purpose is a repo safety net. Repository-grouped folds, repository-agnostic
features, BLOCK/ADVISORY operating points for changed code, out-of-repo
calibration, adaptation kept separate.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-09-30 18:01:06 +02:00
co-authored by Claude Opus 5.5
parent 0fad93fcc5
commit 1b291e81b7
1 changed files with 20 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/specs/learned-decision-engine/requirements.md
+20
View File
@@ -110,3 +110,23 @@ rule hits.
2. Without a trained model available the tools fall back to today's behaviour and say so in the report.
3. Hand-written rules remain as signal producers; changing their enabled/shadow status no longer changes what is
reported unless the engine's decision changes.
### Requirement 7: Built for unseen repositories -- a repo safety net
**User Story:** As the maintainer, I want the engine judged by how it behaves on repositories it never saw, because
"the tool will never run against the test cases it's being trained with. it's purpose is a repo safety net."
(maintainer, 2026-09-30).
#### Acceptance Criteria
1. Every reported metric comes from folds grouped by repository across all corpora: a repository in an evaluation
fold contributes no row to training.
2. Features are repository-agnostic by an allow-listed schema (no repository, path, file, function or commit
identities), enforced by a test.
3. Two operating points serve the safety net: BLOCK, chosen for >= 95% precision on unseen repositories, and
ADVISORY, trading precision for recall; both reported with out-of-repository intervals. The decision unit is a
candidate in changed code (`pre-push` delta), with whole-repository scans as a secondary mode.
4. Calibration is checked on held-out repositories; where it does not hold, only ADVISORY is offered for that
family.
5. Any adaptation to a user's own repository (facts reuse, their dismissals) is opt-in, separate from the
generalisation model, and never contributes to its reported numbers.