spec(learned-decision-engine): brief and requirements for review

Maintainer direction: A/B testing to extrapolate detection instead of hard-coding
it, and a trained decision engine. Instruments become signal producers; labels
from ground truth with provenance; a simple calibrated engine with population
folds; experiments as plane Runs; extrapolation with intervals, checked once on
frozen v3; the engine is what scan/pre-push/plane report.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo

.kiro/specs/learned-decision-engine/brief.md

@@ -0,0 +1,47 @@
+# Brief: a trained decision engine and A/B experiments instead of hand-coded detection
+
+Maintainer, 2026-09-30: "you need to go to a different approach with a b testing to extrapolate detection rather
+than hardcoding it and train a decision engine".
+
+## Why
+
+Every detection gain so far was a hand edit made after a miss: regex rules, taint vocabulary, guards, prompt
+lines, a tie-break rule. Each edit was tuned on the cases that exposed it and had to be re-qualified on a fresh
+population; v1 -> v2 showed engine tuning does not transfer (0/17). The plane changed what is possible: every task
+records per-candidate outcomes, costs and repository facts into one memory store, and runs are cheap to repeat.
+
+## Direction
+
+- Instruments become signal producers: quick rules, the Joern engine, model sink classification, verify passes,
+  tie-break, repo facts (callers, reachability), triage. None decides alone.
+- A decision engine, trained on labelled outcomes, turns the signals for a candidate into a calibrated probability
+  that it is a real vulnerability of its family; operating points (block, report, drop) are thresholds on it.
+- Changes to the pipeline (prompts, models, pass counts, rule sets, features) are variants compared by A/B
+  experiments run as plane Runs with pre-registered metrics and confidence intervals, never adopted on a single
+  run's point estimate.
+- Detection on unseen code is extrapolated from measured learning curves and leave-one-population-out results,
+  reported with uncertainty, and checked once on the frozen v3.
+
+## Data on hand (2026-09-30)
+
+- Pair corpus: 317 vulnerable/fixed pairs, 342 labelled findings (`ousast pairs`), with train/holdout splits.
+- Populations: v1 (spent), v2 validation set (46 candidates, 20 declared sites; plane runs recorded), PHP
+  development corpus (pmpro, v1 PHP, MediaWiki, WP Statistics). v3 (PHP, 16 cases) is being frozen now and stays
+  untouched until the final check.
+- Memory store rows: facts, per-pass verdicts, disputes, tie-breaks, costs, alerts, coverage.
+
+## Risks
+
+- Small labelled data: the engine must be simple and calibrated (regularised linear or shallow trees), with
+  population-level folds; a deep model would memorise.
+- Leakage: labels and features must never come from the population a number is reported on (the train-on-test
+  rule; see the closed-loop leak).
+- Label quality: "rejected by the plane" is not "safe"; negatives come from fixed and benign pins and adjudicated
+  findings, not from absence of an alert.
+
+## Relation to earlier specs
+
+Replaces the hand-edit loop of `improve` for detection decisions (the memory proposer's M1/M2 status edits become
+one variant family among others). `learning-harness` and `harnessx-self-improving-rulesets` are retired with
+HarnessX. `finding-feedback-loop`, `constrained-detector` and `model-grounded-detection` are inputs to read in
+design, not dependencies.

.kiro/specs/learned-decision-engine/requirements.md

@@ -0,0 +1,112 @@
+# Requirements Document
+
+## Introduction
+
+Detection decisions stop being hand-coded. Every instrument becomes a producer of per-candidate signals, a trained
+decision engine turns signals into a calibrated probability per candidate and family, pipeline changes are chosen
+by A/B experiments run on the plane, and expected detection on unseen code is extrapolated with stated uncertainty.
+Maintainer direction and context: `brief.md`.
+
+## Boundary Context
+
+- **In scope**: a feature record per candidate built from every instrument and the memory store; a labelled
+  dataset with provenance; training, calibration and versioning of the decision engine; A/B experiments as plane
+  Runs; learning curves and leave-one-population-out estimates; the engine's use in `ousast scan`, `ousast
+  pre-push` and plane Runs; a final one-time check on frozen population v3.
+- **Out of scope**: new instruments (they plug in as signal producers later); deep or large models; any change to
+  population v3 or its protocol; publishing results (licences still gate that).
+- **Adjacent expectations**: instruments keep working standalone; the memory store (harnessx-removal Req 6) is
+  the system of record; the plane attributes every experiment's token spend per task.
+
+## Requirements
+
+### Requirement 1: Instruments emit signals, not verdicts
+
+**User Story:** As the maintainer, I want every detector's output to become evidence for one decision, so that no
+single hand-written rule or prompt decides whether code is vulnerable.
+
+#### Acceptance Criteria
+
+1. For each candidate (repository, pin, file, function, family) the system assembles one feature record from every
+   instrument that ran: quick-rule hits (rule ids, counts), engine findings and questions completed, model sink
+   classification, triage, verify pass verdicts and turns, tie-break, repository facts (callers, entry-point
+   distance), coverage, language and family.
+2. A missing instrument is recorded as missing, never as a negative signal (coverage `none` stays distinct from
+   "ran and found nothing").
+3. Feature records are stored in the memory store with the instrument versions and image digests that produced
+   them, so any record can be rebuilt.
+
+### Requirement 2: Labels come from ground truth with provenance
+
+**User Story:** As the maintainer, I want the engine to learn only from outcomes we can defend.
+
+#### Acceptance Criteria
+
+1. Positive labels: candidates matching a declared site or fix range on a vulnerable pin (populations, pair
+   corpus). Negative labels: the same functions on fixed pins, benign-control pins, and findings adjudicated false
+   by the maintainer or a judge task with recorded evidence.
+2. A plane verdict alone (agreed, rejected, disputed) is a feature, never a label.
+3. Every label records its source (population, pair, adjudication), split, and the date it was created; labels from
+   a reserved or frozen population are unusable for training until that population's evaluation is recorded.
+
+### Requirement 3: A trained, calibrated, versioned decision engine
+
+**User Story:** As the maintainer, I want one engine that decides from the signals, retrained as data grows, so
+that detection improves without hand edits.
+
+#### Acceptance Criteria
+
+1. The engine is a simple, inspectable model (regularised logistic regression or shallow gradient-boosted trees),
+   trained per family where data allows and pooled otherwise; its output is a calibrated probability.
+2. Training is reproducible from the memory store and a recorded split: the same inputs give the same model, and
+   every model version records its data snapshot, feature set, hyperparameters and metrics.
+3. Evaluation uses population-level folds (leave-one-population-out, and pairs grouped by repository): no
+   repository appears in both training and evaluation of a reported number.
+4. Operating points (block, report, drop) are thresholds chosen on training folds for stated precision targets and
+   reported with the recall they give; the M4 gates (>= 95% precision, >= 90% recall) are the targets, not assumed.
+5. The engine explains each decision by its top contributing features, stored with the finding.
+
+### Requirement 4: Pipeline changes are chosen by A/B experiments
+
+**User Story:** As the maintainer, I want changes to prompts, models, pass counts, rule sets and features compared
+by controlled experiments, so that nothing is adopted on one run's number.
+
+#### Acceptance Criteria
+
+1. An experiment is declared before it runs: arms (variants as plane Run configurations), unit of randomisation
+   (candidate, file or repository), metric (detection, precision, cost per candidate), minimum detectable effect,
+   sample size or sequential stopping rule, and budget.
+2. Arms run on the same units where the design allows (paired), with assignment recorded; results and spend go to
+   the memory store per arm.
+3. A variant is adopted only when the pre-registered test shows a difference with its confidence interval
+   excluding zero, or declared equivalent within a margin for cost-only changes; otherwise the result is recorded
+   as inconclusive.
+4. The first experiment is the known open question: the known-callers prompt line (17 vs 19 sites flagged in any
+   pass), on the development corpus.
+
+### Requirement 5: Detection is extrapolated, with uncertainty
+
+**User Story:** As the maintainer, I want to know how well the system will detect on code nobody has seen, before
+and without tuning on it.
+
+#### Acceptance Criteria
+
+1. Learning curves: engine metrics as a function of training data size and number of populations, per family,
+   with confidence bands.
+2. Expected recall and precision on unseen code are reported per family and language as intervals from the
+   population-level folds; a family with too few labels is reported as "insufficient data", never as a number.
+3. The extrapolation is checked exactly once against frozen population v3 under protocol v3, after the engine and
+   its operating points are fixed; the comparison of predicted and measured intervals is recorded.
+
+### Requirement 6: The engine is what users get
+
+**User Story:** As a user scanning a fresh codebase, I want the decision engine's judgement, not a raw list of
+rule hits.
+
+#### Acceptance Criteria
+
+1. `ousast scan`, `ousast pre-push` and plane Runs report findings with the engine's probability, the operating
+   point they passed and the top contributing features.
+2. Without a trained model available the tools fall back to today's behaviour and say so in the report.
+3. Hand-written rules remain as signal producers; changing their enabled/shadow status no longer changes what is
+   reported unless the engine's decision changes.

.kiro/specs/learned-decision-engine/spec.json

@@ -0,0 +1,13 @@
+{
+  "feature_name": "learned-decision-engine",
+  "created_at": "2026-09-30T14:00:00Z",
+  "updated_at": "2026-09-30T14:00:00Z",
+  "language": "en",
+  "phase": "requirements-generated",
+  "approvals": {
+    "requirements": {"generated": true, "approved": false},
+    "design": {"generated": false, "approved": false},
+    "tasks": {"generated": false, "approved": false}
+  },
+  "ready_for_implementation": false
+}