Open_Ultra_SAST/docs/decision-engine.md

# The learned decision engine (in development)

**Status 2026-10-02: in development, not adopted.** No scan, pre-push check or report uses it;
every user-facing path is today's deterministic one until a compiled program is adopted. The
design is the `learned-decision-engine` specification (maintainer tooling); the code is in
`src/openultrasast/learn/`.

## Why

Every detection gain so far was a hand edit made after a miss: a rule, a taint vocabulary entry,
a guard, a prompt line, a tie-break. Each was tuned on the cases that exposed it, and engine
tuning measured on population v1 did not transfer to v2 (see [evaluation.md](evaluation.md)).
The engine replaces that with a decision learned from labelled outcomes and checked out of
repository.

## Design

- **Signals, not verdicts.** Every instrument (quick rules, the Joern engine, model sink and
  role classification, the plane's verify passes and agreement, repository facts such as
  callers) contributes a feature record per candidate `(path, function, family)`, including an
  explicit state when it did not run or failed. None decides alone.
- **Labels from ground truth.** `ousast learn labels` derives labels only from sources on a
  fail-closed allow-list (`learn/sources.toml`): the vulnerable and fixed sides of pairs, the
  declared sites and fix ranges of spent populations, benign deltas and recorded adjudications.
  "Rejected by the plane" is never a negative. Reserved populations (v3) are not on the list.
- **An AI classifier with local memory, compiled in house in the DSPy style.** The program is an
  instruction, a few demonstrations and up to six retrieved labelled examples, then the
  candidate's code excerpt and signals; the model (`deepseek-flash`) answers
  vulnerable / not vulnerable / unsure with a rationale, sampled `k` times, giving a score `s`.
  Compilation bootstraps demonstrations and searches instructions on a compile split, scored by
  a pre-registered balanced Brier score (`learn/compile.toml`).
- **Retrieval** from the memory store: nearest neighbours by Gower distance on the signal
  profile, re-ranked by cosine similarity of code embeddings (OpenAI `text-embedding-3-small`
  through OpenRouter, cached by excerpt hash), under a label/family/repository balance rule. An
  evaluation-boundary filter keeps a candidate's own repository group out of what it is shown.
- **Repository-grouped evaluation.** The compile split (25% of repository groups) is never
  evaluated. The rest is evaluated by outer grouped 5-fold cross-validation: the Platt
  calibration map is cross-fitted on the other folds and the operating points are chosen on the
  other folds (nested). One compile per profile, not one per fold. Intervals are 95%
  repository-cluster bootstrap.
- **Two operating points.** BLOCK is offered only when calibration holds (slope CI contains 1,
  intercept CI contains 0, ECE <= 0.05) and a fold reaches the required precision; otherwise the
  engine is ADVISORY only. An `unsure` answer can never BLOCK.

## First measured slice (2026-10-01)

Record: `benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json` (counts
only). Family **injection** only; memory of 724 static examples from all families; input profile
v1 with the engine signals and function length withheld (a leak audit found them separating the
sides of a pair by construction: `2026-10-01-decision-engine-leak-audit/`); framework priors off.

| Measure | Value (95% repository-cluster CI) |
| --- | --- |
| Candidates evaluated | 110 in 29 repository groups, 58 positive |
| ADVISORY recall | 0.90 (0.82 - 0.97) |
| ADVISORY precision | 0.54 (0.44 - 0.65) |
| AUC of the score `s` | 0.74 (0.65 - 0.85) |
| Majority "vulnerable" verdict | precision 0.86 (0.72 - 1.0), recall 0.43 (0.29 - 0.62) |
| BLOCK | not offered: ECE 0.064 > 0.05, and every fold reported `precision_unreachable` |
| Compiled program | balanced Brier 0.128 on the validation split vs 0.199 for the baseline instruction |
| Canary agreement on re-run | 0.83 (30 candidates) |
| Spend | $0.91 metered at list price for DeepSeek, $0.0040 for embeddings; $0.0032 per evaluated candidate |

The other evaluable families are not evaluated yet. Leave-one-source-out, leave-one-framework-out
and learning curves, experiments, integration into scan and pre-push, adoption, and the one-time
v3 check are still ahead.

## Commands

`ousast learn` is a maintainer command group over the engine's data. Paid steps take
`--budget-usd`; `--replay-only` answers every call from the response cache at no cost.

| Command | Does |
| --- | --- |
| `ousast learn labels` | labels from the allow-listed sources (host only); `--snapshot` writes the counts-only record |
| `ousast learn memory build` | label + feature record + excerpt -> `example` rows (no model) |
| `ousast learn memory embed` | embed the stored excerpts (OpenRouter, metered, cached) |
| `ousast learn compile` | compile the program on the compile split |
| `ousast learn evaluate --program ID` | outer-fold evaluation, cross-fitted calibration, nested operating points |
| `ousast learn curve --program ID` | the metric over memory size on a fixed candidate subset |
| `ousast learn audit-leaks` | how well each signal alone separates the sides of a pair (no model) |
| `ousast learn experiment register\|units\|run\|analyse MANIFEST` | A/B experiments from a pre-registered manifest (`plane/experiments/<id>.yaml`): record its digest, freeze the units (no model), run the arms paired (arm A may replay from the cache at $0), analyse (repository bootstrap, exact McNemar, two looks, the adoption verdict) |

The memory store is the plane's (`OUSAST_MEMORY`; see [ops/ax/README.md](ops/ax/README.md)).
The harvest that produced the plane signals runs as plane Runs generated by
`ousast plane harvest` (`benchmarks/measurements/2026-10-01-decision-engine-harvest/record.json`).