diff --git a/.kiro/specs/learned-decision-engine/requirements.md b/.kiro/specs/learned-decision-engine/requirements.md index 345a655..4875c85 100644 --- a/.kiro/specs/learned-decision-engine/requirements.md +++ b/.kiro/specs/learned-decision-engine/requirements.md @@ -132,6 +132,13 @@ rule hits. - **t_quiet**: per family, the highest threshold keeping >= 90% of the paired positives at or above it (chosen on training folds, its held-out recall reported); findings below it are silent. - No held-out label moves a threshold or a normalisation map; candidates whose verdict is `unsure` never block. +5. *(Amended 2026-10-05, maintainer decision, with `search-with-proof`.)* Where a family has an owned demonstration + oracle, the engine is **triage, not the gate**: BLOCK comes only from a verified demonstration (`search-with-proof` + Req 3, 6.3), and the engine's job is to put the true vulnerable function among the top K candidates of a push. + For those families the engine is judged by (a) top-K recall per push (K = 3 to start) on the unseen-repo pool and + (b) its advisory false-alarm rate there; the pooled block gate of 6.4 then governs only families without an + oracle (access control, deserialisation, configuration and secrets), where the engine remains the only signal. + Every arm reports top-K recall next to its catch and false-alarm rates. ### Requirement 7: Built for unseen repositories -- a repo safety net diff --git a/.kiro/specs/learned-decision-engine/tasks.md b/.kiro/specs/learned-decision-engine/tasks.md index c980443..091345d 100644 --- a/.kiro/specs/learned-decision-engine/tasks.md +++ b/.kiro/specs/learned-decision-engine/tasks.md @@ -254,3 +254,13 @@ rate per push. Within-pair AUC stays a development signal. Tasks 11-14 below fol - Search input fields, retrieval parameters, k, instruction and demonstrations together (DSPy MIPRO-style) on training folds; no hand-set input profile. Budget go-ahead before paid search. - _Requirements: 3, 6, 7_ + +- [ ] 17. Triage objective (Req 6.5, amended 2026-10-05 with `search-with-proof`) + - [ ] 17.1 Metric: top-K recall per push (K = 1, 3, 5) of the true vulnerable function among changed functions, + on held-out folds and on the unseen-repo pool; added to the pool scorer next to catch and false alarm. + - [ ] 17.2 Score the existing arms on it ($0 replay): trained logistic (cheapest per candidate, likely default), + LLM programs (exp-005 A), engine-evidence ordering, and an averaged ensemble. + - [ ] 17.3 Re-scope 14 (pooled block gate) to families without a demonstration oracle; 15.4 and 16 report top-K + recall as their primary unseen-pool number for families with an oracle. + - _Requirements: 6.5, 7_ +