diff --git a/.kiro/specs/learned-decision-engine/brief.md b/.kiro/specs/learned-decision-engine/brief.md new file mode 100644 index 0000000..e10192d --- /dev/null +++ b/.kiro/specs/learned-decision-engine/brief.md @@ -0,0 +1,47 @@ +# Brief: a trained decision engine and A/B experiments instead of hand-coded detection + +Maintainer, 2026-09-30: "you need to go to a different approach with a b testing to extrapolate detection rather +than hardcoding it and train a decision engine". + +## Why + +Every detection gain so far was a hand edit made after a miss: regex rules, taint vocabulary, guards, prompt +lines, a tie-break rule. Each edit was tuned on the cases that exposed it and had to be re-qualified on a fresh +population; v1 -> v2 showed engine tuning does not transfer (0/17). The plane changed what is possible: every task +records per-candidate outcomes, costs and repository facts into one memory store, and runs are cheap to repeat. + +## Direction + +- Instruments become signal producers: quick rules, the Joern engine, model sink classification, verify passes, + tie-break, repo facts (callers, reachability), triage. None decides alone. +- A decision engine, trained on labelled outcomes, turns the signals for a candidate into a calibrated probability + that it is a real vulnerability of its family; operating points (block, report, drop) are thresholds on it. +- Changes to the pipeline (prompts, models, pass counts, rule sets, features) are variants compared by A/B + experiments run as plane Runs with pre-registered metrics and confidence intervals, never adopted on a single + run's point estimate. +- Detection on unseen code is extrapolated from measured learning curves and leave-one-population-out results, + reported with uncertainty, and checked once on the frozen v3. + +## Data on hand (2026-09-30) + +- Pair corpus: 317 vulnerable/fixed pairs, 342 labelled findings (`ousast pairs`), with train/holdout splits. +- Populations: v1 (spent), v2 validation set (46 candidates, 20 declared sites; plane runs recorded), PHP + development corpus (pmpro, v1 PHP, MediaWiki, WP Statistics). v3 (PHP, 16 cases) is being frozen now and stays + untouched until the final check. +- Memory store rows: facts, per-pass verdicts, disputes, tie-breaks, costs, alerts, coverage. + +## Risks + +- Small labelled data: the engine must be simple and calibrated (regularised linear or shallow trees), with + population-level folds; a deep model would memorise. +- Leakage: labels and features must never come from the population a number is reported on (the train-on-test + rule; see the closed-loop leak). +- Label quality: "rejected by the plane" is not "safe"; negatives come from fixed and benign pins and adjudicated + findings, not from absence of an alert. + +## Relation to earlier specs + +Replaces the hand-edit loop of `improve` for detection decisions (the memory proposer's M1/M2 status edits become +one variant family among others). `learning-harness` and `harnessx-self-improving-rulesets` are retired with +HarnessX. `finding-feedback-loop`, `constrained-detector` and `model-grounded-detection` are inputs to read in +design, not dependencies. diff --git a/.kiro/specs/learned-decision-engine/requirements.md b/.kiro/specs/learned-decision-engine/requirements.md new file mode 100644 index 0000000..9a48fef --- /dev/null +++ b/.kiro/specs/learned-decision-engine/requirements.md @@ -0,0 +1,112 @@ +# Requirements Document + +## Introduction + +Detection decisions stop being hand-coded. Every instrument becomes a producer of per-candidate signals, a trained +decision engine turns signals into a calibrated probability per candidate and family, pipeline changes are chosen +by A/B experiments run on the plane, and expected detection on unseen code is extrapolated with stated uncertainty. +Maintainer direction and context: `brief.md`. + +## Boundary Context + +- **In scope**: a feature record per candidate built from every instrument and the memory store; a labelled + dataset with provenance; training, calibration and versioning of the decision engine; A/B experiments as plane + Runs; learning curves and leave-one-population-out estimates; the engine's use in `ousast scan`, `ousast + pre-push` and plane Runs; a final one-time check on frozen population v3. +- **Out of scope**: new instruments (they plug in as signal producers later); deep or large models; any change to + population v3 or its protocol; publishing results (licences still gate that). +- **Adjacent expectations**: instruments keep working standalone; the memory store (harnessx-removal Req 6) is + the system of record; the plane attributes every experiment's token spend per task. + +## Requirements + +### Requirement 1: Instruments emit signals, not verdicts + +**User Story:** As the maintainer, I want every detector's output to become evidence for one decision, so that no +single hand-written rule or prompt decides whether code is vulnerable. + +#### Acceptance Criteria + +1. For each candidate (repository, pin, file, function, family) the system assembles one feature record from every + instrument that ran: quick-rule hits (rule ids, counts), engine findings and questions completed, model sink + classification, triage, verify pass verdicts and turns, tie-break, repository facts (callers, entry-point + distance), coverage, language and family. +2. A missing instrument is recorded as missing, never as a negative signal (coverage `none` stays distinct from + "ran and found nothing"). +3. Feature records are stored in the memory store with the instrument versions and image digests that produced + them, so any record can be rebuilt. + +### Requirement 2: Labels come from ground truth with provenance + +**User Story:** As the maintainer, I want the engine to learn only from outcomes we can defend. + +#### Acceptance Criteria + +1. Positive labels: candidates matching a declared site or fix range on a vulnerable pin (populations, pair + corpus). Negative labels: the same functions on fixed pins, benign-control pins, and findings adjudicated false + by the maintainer or a judge task with recorded evidence. +2. A plane verdict alone (agreed, rejected, disputed) is a feature, never a label. +3. Every label records its source (population, pair, adjudication), split, and the date it was created; labels from + a reserved or frozen population are unusable for training until that population's evaluation is recorded. + +### Requirement 3: A trained, calibrated, versioned decision engine + +**User Story:** As the maintainer, I want one engine that decides from the signals, retrained as data grows, so +that detection improves without hand edits. + +#### Acceptance Criteria + +1. The engine is a simple, inspectable model (regularised logistic regression or shallow gradient-boosted trees), + trained per family where data allows and pooled otherwise; its output is a calibrated probability. +2. Training is reproducible from the memory store and a recorded split: the same inputs give the same model, and + every model version records its data snapshot, feature set, hyperparameters and metrics. +3. Evaluation uses population-level folds (leave-one-population-out, and pairs grouped by repository): no + repository appears in both training and evaluation of a reported number. +4. Operating points (block, report, drop) are thresholds chosen on training folds for stated precision targets and + reported with the recall they give; the M4 gates (>= 95% precision, >= 90% recall) are the targets, not assumed. +5. The engine explains each decision by its top contributing features, stored with the finding. + +### Requirement 4: Pipeline changes are chosen by A/B experiments + +**User Story:** As the maintainer, I want changes to prompts, models, pass counts, rule sets and features compared +by controlled experiments, so that nothing is adopted on one run's number. + +#### Acceptance Criteria + +1. An experiment is declared before it runs: arms (variants as plane Run configurations), unit of randomisation + (candidate, file or repository), metric (detection, precision, cost per candidate), minimum detectable effect, + sample size or sequential stopping rule, and budget. +2. Arms run on the same units where the design allows (paired), with assignment recorded; results and spend go to + the memory store per arm. +3. A variant is adopted only when the pre-registered test shows a difference with its confidence interval + excluding zero, or declared equivalent within a margin for cost-only changes; otherwise the result is recorded + as inconclusive. +4. The first experiment is the known open question: the known-callers prompt line (17 vs 19 sites flagged in any + pass), on the development corpus. + +### Requirement 5: Detection is extrapolated, with uncertainty + +**User Story:** As the maintainer, I want to know how well the system will detect on code nobody has seen, before +and without tuning on it. + +#### Acceptance Criteria + +1. Learning curves: engine metrics as a function of training data size and number of populations, per family, + with confidence bands. +2. Expected recall and precision on unseen code are reported per family and language as intervals from the + population-level folds; a family with too few labels is reported as "insufficient data", never as a number. +3. The extrapolation is checked exactly once against frozen population v3 under protocol v3, after the engine and + its operating points are fixed; the comparison of predicted and measured intervals is recorded. + +### Requirement 6: The engine is what users get + +**User Story:** As a user scanning a fresh codebase, I want the decision engine's judgement, not a raw list of +rule hits. + +#### Acceptance Criteria + +1. `ousast scan`, `ousast pre-push` and plane Runs report findings with the engine's probability, the operating + point they passed and the top contributing features. +2. Without a trained model available the tools fall back to today's behaviour and say so in the report. +3. Hand-written rules remain as signal producers; changing their enabled/shadow status no longer changes what is + reported unless the engine's decision changes. diff --git a/.kiro/specs/learned-decision-engine/spec.json b/.kiro/specs/learned-decision-engine/spec.json new file mode 100644 index 0000000..daed8a9 --- /dev/null +++ b/.kiro/specs/learned-decision-engine/spec.json @@ -0,0 +1,13 @@ +{ + "feature_name": "learned-decision-engine", + "created_at": "2026-09-30T14:00:00Z", + "updated_at": "2026-09-30T14:00:00Z", + "language": "en", + "phase": "requirements-generated", + "approvals": { + "requirements": {"generated": true, "approved": false}, + "design": {"generated": false, "approved": false}, + "tasks": {"generated": false, "approved": false} + }, + "ready_for_implementation": false +}