Open_Ultra_SAST/.kiro/specs/learned-decision-engine/requirements.md

# Requirements Document

## Introduction

Detection decisions stop being hand-coded. Every instrument becomes a producer of per-candidate signals, a trained
decision engine turns signals into a calibrated probability per candidate and family, pipeline changes are chosen
by A/B experiments run on the plane, and expected detection on unseen code is extrapolated with stated uncertainty.
Maintainer direction and context: `brief.md`.

## Boundary Context

- In scope: a feature record per candidate built from every instrument and the memory store; a labelled
  dataset with provenance; training, calibration and versioning of the decision engine; A/B experiments as plane
  Runs; learning curves and leave-one-population-out estimates; the engine's use in `ousast scan`, `ousast
  pre-push` and plane Runs; a final one-time check on frozen population v3.
- Out of scope: new instruments (they plug in as signal producers later); deep or large models; any change to
  population v3 or its protocol; publishing results (licences still gate that).
- Adjacent expectations: instruments keep working standalone; the memory store (harnessx-removal Req 6) is
  the system of record; the plane attributes every experiment's token spend per task.

## Requirements

### Requirement 1: Instruments emit signals, not verdicts

**User Story:** As the maintainer, I want every detector's output to become evidence for one decision, so that no
single hand-written rule or prompt decides whether code is vulnerable.

#### Acceptance Criteria

1. For each candidate (repository, pin, file, function, family) the system assembles one feature record from every
   instrument that ran: quick-rule hits (rule ids, counts), engine findings and questions completed, model sink
   classification, triage, verify pass verdicts and turns, tie-break, repository facts (callers, entry-point
   distance), coverage, language and family.
2. A missing instrument is recorded as missing, never as a negative signal (coverage `none` stays distinct from
   "ran and found nothing").
3. Feature records are stored in the memory store with the instrument versions and image digests that produced
   them, so any record can be rebuilt.

### Requirement 2: Labels come from ground truth with provenance

**User Story:** As the maintainer, I want the engine to learn only from outcomes we can defend.

#### Acceptance Criteria

1. Positive labels: candidates matching a declared site or fix range on a vulnerable pin (populations, pair
   corpus). Negative labels: the same functions on fixed pins, benign-control pins, and findings adjudicated false
   by the maintainer or a judge task with recorded evidence.
2. A plane verdict alone (agreed, rejected, disputed) is a feature, never a label.
3. Every label records its source (population, pair, adjudication), split, and the date it was created; labels from
   a reserved or frozen population are unusable for training until that population's evaluation is recorded.

### Requirement 3: An AI classifier with local memory, compiled DSPy-style

**User Story:** As the maintainer, I want the decision made by an AI classifier that learns from a small, growing
memory of labelled cases, because "We will not be able to scale up data" and "we need to have the decision
engine on AI classifier not ML. with local memory, dspy style" (maintainer, 2026-10-01).

#### Acceptance Criteria

1. The decision engine is a declared language-model program (DSPy-style signature: candidate code, instrument
   signals, inferred repository roles in; verdict, family, confidence and rationale out), built in-house on the
   plane's metered client so every call stays within per-task budgets, attribution and Model routing; no
   statistical model is trained.
2. For each candidate the program retrieves similar labelled cases from the local memory store as in-context
   examples: nearest neighbours by the repository-agnostic signal profile, re-ranked by code embeddings
   (OpenRouter, embeddings only); identities are never used for retrieval.
3. The program is compiled, not fitted: its instructions and example-selection policy are optimised against a
   pre-registered metric on training folds (bootstrap few-shot and instruction search); each compiled program is
   versioned in the memory store with its data snapshot, model, prompts, retrieval settings and metrics, and the
   same inputs reproduce it.
4. Retrieval never crosses the evaluation boundary: when a number is reported for a repository, examples come
   only from other repositories (and other frameworks in leave-one-framework-out folds); a test proves it.
5. The confidence the program reports is calibrated on held-out repositories before operating points are set;
   where it does not calibrate, only ADVISORY is offered (Req 7.4).
6. New labels (adjudications, dismissals in the opt-in adaptation layer) improve decisions by entering memory,
   without retraining; their effect is measured by A/B experiments like any other change.

### Requirement 4: Pipeline changes are chosen by A/B experiments

**User Story:** As the maintainer, I want changes to prompts, models, pass counts, rule sets and features compared
by controlled experiments, so that nothing is adopted on one run's number.

#### Acceptance Criteria

1. An experiment is declared before it runs: arms (variants as plane Run configurations), unit of randomisation
   (candidate, file or repository), metric (detection, precision, cost per candidate), minimum detectable effect,
   sample size or sequential stopping rule, and budget.
2. Arms run on the same units where the design allows (paired), with assignment recorded; results and spend go to
   the memory store per arm.
3. A variant is adopted only when the pre-registered test shows a difference with its confidence interval
   excluding zero, or declared equivalent within a margin for cost-only changes; otherwise the result is recorded
   as inconclusive.
4. The first experiment is the known open question: the known-callers prompt line (17 vs 19 sites flagged in any
   pass), on the development corpus.

### Requirement 5: Detection is extrapolated, with uncertainty

**User Story:** As the maintainer, I want to know how well the system will detect on code nobody has seen, before
and without tuning on it.

#### Acceptance Criteria

1. Learning curves: engine metrics as a function of memory size (labelled cases available for retrieval) and number of populations, per family,
   with confidence bands.
2. Expected recall and precision on unseen code are reported per family and language as intervals from the
   population-level folds; a family with too few labels is reported as "insufficient data", never as a number.
3. The extrapolation is checked exactly once against frozen population v3 under protocol v3, after the engine and
   its operating points are fixed; the comparison of predicted and measured intervals is recorded.

### Requirement 6: The engine is what users get

**User Story:** As a user scanning a fresh codebase, I want the decision engine's judgement, not a raw list of
rule hits.

#### Acceptance Criteria

1. `ousast scan`, `ousast pre-push` and plane Runs report findings with the engine's probability, the operating
   point they passed and the top contributing features.
2. Without a trained model available the tools fall back to today's behaviour and say so in the report.
3. Hand-written rules remain as signal producers; changing their enabled/shadow status no longer changes what is
   reported unless the engine's decision changes.
4. *(Amended 2026-10-03, maintainer decision; refines the BLOCK rule of 7.3 and 7.4. Tested by exp-004.)* BLOCK
   precision is demonstrated pooled across families, not per family:
   - Normalisation, declared before any run: each family's raw score is put on one scale by a per-family Platt
     map (`learn.calibrate.fit_map` with its floors) fitted on training folds only. A family the training folds
     cannot map does not block on that fold.
   - **t_block**: one threshold on the normalised score for all families, chosen on training folds as the lowest
     one whose Wilson 95% lower bound on precision, over the training flags of all families together, is >= 0.95.
     It is reported by the Wilson 95% lower bound over all held-out flags of all families, before the floor.
   - **Family floor**: a family may block only if its own held-out flags have zero errors and number at least F
     (F = 10, reasoned in the exp-004 registration); otherwise that family stays ADVISORY.
   - **t_quiet**: per family, the highest threshold keeping >= 90% of the paired positives at or above it (chosen
     on training folds, its held-out recall reported); findings below it are silent.
   - No held-out label moves a threshold or a normalisation map; candidates whose verdict is `unsure` never block.
5. *(Amended 2026-10-05, maintainer decision, with `search-with-proof`.)* Where a family has an owned demonstration
   oracle, the engine is **triage, not the gate**: BLOCK comes only from a verified demonstration (`search-with-proof`
   Req 3, 6.3), and the engine's job is to put the true vulnerable function among the top K candidates of a push.
   For those families the engine is judged by (a) top-K recall per push (K = 3 to start) on the unseen-repo pool and
   (b) its advisory false-alarm rate there; the pooled block gate of 6.4 then governs only families without an
   oracle (access control, deserialisation, configuration and secrets), where the engine remains the only signal.
   Every arm reports top-K recall next to its catch and false-alarm rates.

### Requirement 7: Built for unseen repositories -- a repo safety net

**User Story:** As the maintainer, I want the engine judged by how it behaves on repositories it never saw, because
"the tool will never run against the test cases it's being trained with. it's purpose is a repo safety net."
(maintainer, 2026-09-30).

#### Acceptance Criteria

1. Every reported metric comes from folds grouped by repository across all corpora: a repository in an evaluation
   fold contributes no row to training.
2. Features are repository-agnostic by an allow-listed schema (no repository, path, file, function or commit
   identities), enforced by a test.
3. Two operating points serve the safety net: BLOCK, chosen for >= 95% precision on unseen repositories, and
   ADVISORY, trading precision for recall; both reported with out-of-repository intervals. The decision unit is a
   candidate in changed code (`pre-push` delta), with whole-repository scans as a secondary mode.
4. Calibration is checked on held-out repositories; where it does not hold, only ADVISORY is offered for that
   family. *(Amended 2026-10-03: once exp-004 adopts the pooled gate of 6.4, BLOCK precision is shown by its held-out
   bound and family floor, and calibration is reported, not required.)*
5. Any adaptation to a user's own repository (facts reuse, their dismissals) is opt-in, separate from the
   generalisation model, and never contributes to its reported numbers.

### Requirement 8: No framework or application knowledge as a prerequisite

**User Story:** As the maintainer, I want the safety net to work on any application in a language, because "if
there is hardcoded application specification needed to build trees this is problematic. wordpress isn't the only
php application. same for python, js etc" (maintainer, 2026-09-30).

#### Acceptance Criteria

1. Hand-written knowledge is limited to the language level (built-ins, superglobals, process execution, raw
   database drivers, deserialisers). Roles specific to a framework or application (its request objects, query
   builders, hooks, sanitisers, guards) are inferred per repository at scan time from its own source and its
   dependencies, not required from a hand-written table.
2. Existing framework-specific entries (WordPress, Django, Flask, Express, Spring and others in `ruleset/semantic/`
   and `ruleset/<lang>/`) are tagged with their framework and enter the engine only as optional prior features; a
   prior is kept only if an A/B experiment on held-out repositories shows it helps, and the engine must still work
   with every prior switched off.
3. Evaluation is stratified by framework and includes leave-one-framework-out folds (for example: train without any
   WordPress repository, evaluate on WordPress), so a number never depends on having seen the target's framework;
   results are reported per framework and for "framework unseen in training".
4. The share of each reported result that depends on framework priors is stated (with priors vs without).