Open_Ultra_SAST/.kiro/specs/learned-decision-engine/requirements.md

# Requirements Document

## Introduction

Detection decisions stop being hand-coded. Every instrument becomes a producer of per-candidate signals, a trained
decision engine turns signals into a calibrated probability per candidate and family, pipeline changes are chosen
by A/B experiments run on the plane, and expected detection on unseen code is extrapolated with stated uncertainty.
Maintainer direction and context: `brief.md`.

## Boundary Context

- **In scope**: a feature record per candidate built from every instrument and the memory store; a labelled
  dataset with provenance; training, calibration and versioning of the decision engine; A/B experiments as plane
  Runs; learning curves and leave-one-population-out estimates; the engine's use in `ousast scan`, `ousast
  pre-push` and plane Runs; a final one-time check on frozen population v3.
- **Out of scope**: new instruments (they plug in as signal producers later); deep or large models; any change to
  population v3 or its protocol; publishing results (licences still gate that).
- **Adjacent expectations**: instruments keep working standalone; the memory store (harnessx-removal Req 6) is
  the system of record; the plane attributes every experiment's token spend per task.

## Requirements

### Requirement 1: Instruments emit signals, not verdicts

**User Story:** As the maintainer, I want every detector's output to become evidence for one decision, so that no
single hand-written rule or prompt decides whether code is vulnerable.

#### Acceptance Criteria

1. For each candidate (repository, pin, file, function, family) the system assembles one feature record from every
   instrument that ran: quick-rule hits (rule ids, counts), engine findings and questions completed, model sink
   classification, triage, verify pass verdicts and turns, tie-break, repository facts (callers, entry-point
   distance), coverage, language and family.
2. A missing instrument is recorded as missing, never as a negative signal (coverage `none` stays distinct from
   "ran and found nothing").
3. Feature records are stored in the memory store with the instrument versions and image digests that produced
   them, so any record can be rebuilt.

### Requirement 2: Labels come from ground truth with provenance

**User Story:** As the maintainer, I want the engine to learn only from outcomes we can defend.

#### Acceptance Criteria

1. Positive labels: candidates matching a declared site or fix range on a vulnerable pin (populations, pair
   corpus). Negative labels: the same functions on fixed pins, benign-control pins, and findings adjudicated false
   by the maintainer or a judge task with recorded evidence.
2. A plane verdict alone (agreed, rejected, disputed) is a feature, never a label.
3. Every label records its source (population, pair, adjudication), split, and the date it was created; labels from
   a reserved or frozen population are unusable for training until that population's evaluation is recorded.

### Requirement 3: A trained, calibrated, versioned decision engine

**User Story:** As the maintainer, I want one engine that decides from the signals, retrained as data grows, so
that detection improves without hand edits.

#### Acceptance Criteria

1. The engine is a simple, inspectable model (regularised logistic regression or shallow gradient-boosted trees),
   trained per family where data allows and pooled otherwise; its output is a calibrated probability.
2. Training is reproducible from the memory store and a recorded split: the same inputs give the same model, and
   every model version records its data snapshot, feature set, hyperparameters and metrics.
3. Evaluation uses population-level folds (leave-one-population-out, and pairs grouped by repository): no
   repository appears in both training and evaluation of a reported number.
4. Operating points (block, report, drop) are thresholds chosen on training folds for stated precision targets and
   reported with the recall they give; the M4 gates (>= 95% precision, >= 90% recall) are the targets, not assumed.
5. The engine explains each decision by its top contributing features, stored with the finding.

### Requirement 4: Pipeline changes are chosen by A/B experiments

**User Story:** As the maintainer, I want changes to prompts, models, pass counts, rule sets and features compared
by controlled experiments, so that nothing is adopted on one run's number.

#### Acceptance Criteria

1. An experiment is declared before it runs: arms (variants as plane Run configurations), unit of randomisation
   (candidate, file or repository), metric (detection, precision, cost per candidate), minimum detectable effect,
   sample size or sequential stopping rule, and budget.
2. Arms run on the same units where the design allows (paired), with assignment recorded; results and spend go to
   the memory store per arm.
3. A variant is adopted only when the pre-registered test shows a difference with its confidence interval
   excluding zero, or declared equivalent within a margin for cost-only changes; otherwise the result is recorded
   as inconclusive.
4. The first experiment is the known open question: the known-callers prompt line (17 vs 19 sites flagged in any
   pass), on the development corpus.

### Requirement 5: Detection is extrapolated, with uncertainty

**User Story:** As the maintainer, I want to know how well the system will detect on code nobody has seen, before
and without tuning on it.

#### Acceptance Criteria

1. Learning curves: engine metrics as a function of training data size and number of populations, per family,
   with confidence bands.
2. Expected recall and precision on unseen code are reported per family and language as intervals from the
   population-level folds; a family with too few labels is reported as "insufficient data", never as a number.
3. The extrapolation is checked exactly once against frozen population v3 under protocol v3, after the engine and
   its operating points are fixed; the comparison of predicted and measured intervals is recorded.

### Requirement 6: The engine is what users get

**User Story:** As a user scanning a fresh codebase, I want the decision engine's judgement, not a raw list of
rule hits.

#### Acceptance Criteria

1. `ousast scan`, `ousast pre-push` and plane Runs report findings with the engine's probability, the operating
   point they passed and the top contributing features.
2. Without a trained model available the tools fall back to today's behaviour and say so in the report.
3. Hand-written rules remain as signal producers; changing their enabled/shadow status no longer changes what is
   reported unless the engine's decision changes.