Marius CiepluchandClaude Fable 5.1
fbc3220bbe
learn/experiments.py: YAML manifests under plane/experiments/<id>.yaml, digest-checked at register, run and analyse time; `experiment`, `arm_outcome` and `experiment_result` rows; paired arms in a seeded repository order with a per-unit coin; arm A replay-only at $0; canary replicates under a fresh cache salt; repository-cluster paired bootstrap, exact McNemar, two O'Brien-Fleming looks, adopt/reject/equivalent/ inconclusive. `ousast learn experiment register|units|run|analyse`. Program: `ProgramSpec.sampling = retrieval_ensemble` (k samples at temperature 0 over k disjoint retrieved example sets; default unchanged); `Caller.salt`; `retrieve(exclude=)`. verify: OUSAST_VERIFY_CALLERS=off. Manifests exp-001-known-callers (plane arms, units pending) and exp-002-retrieval-ensemble (slice-2 incumbents, ceiling $4) -- not run. 18 scripted tests; module audit regenerated (151 modules). Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
5.8 KiB
The learned decision engine (in development)
Status 2026-10-02: in development, not adopted. No scan, pre-push check or report uses it;
every user-facing path is today's deterministic one until a compiled program is adopted. The
design is the learned-decision-engine specification (maintainer tooling); the code is in
src/openultrasast/learn/.
Why
Every detection gain so far was a hand edit made after a miss: a rule, a taint vocabulary entry, a guard, a prompt line, a tie-break. Each was tuned on the cases that exposed it, and engine tuning measured on population v1 did not transfer to v2 (see evaluation.md). The engine replaces that with a decision learned from labelled outcomes and checked out of repository.
Design
- Signals, not verdicts. Every instrument (quick rules, the Joern engine, model sink and
role classification, the plane's verify passes and agreement, repository facts such as
callers) contributes a feature record per candidate
(path, function, family), including an explicit state when it did not run or failed. None decides alone. - Labels from ground truth.
ousast learn labelsderives labels only from sources on a fail-closed allow-list (learn/sources.toml): the vulnerable and fixed sides of pairs, the declared sites and fix ranges of spent populations, benign deltas and recorded adjudications. "Rejected by the plane" is never a negative. Reserved populations (v3) are not on the list. - An AI classifier with local memory, compiled in house in the DSPy style. The program is an
instruction, a few demonstrations and up to six retrieved labelled examples, then the
candidate's code excerpt and signals; the model (
deepseek-flash) answers vulnerable / not vulnerable / unsure with a rationale, sampledktimes, giving a scores. Compilation bootstraps demonstrations and searches instructions on a compile split, scored by a pre-registered balanced Brier score (learn/compile.toml). - Retrieval from the memory store: nearest neighbours by Gower distance on the signal
profile, re-ranked by cosine similarity of code embeddings (OpenAI
text-embedding-3-smallthrough OpenRouter, cached by excerpt hash), under a label/family/repository balance rule. An evaluation-boundary filter keeps a candidate's own repository group out of what it is shown. - Repository-grouped evaluation. The compile split (25% of repository groups) is never evaluated. The rest is evaluated by outer grouped 5-fold cross-validation: the Platt calibration map is cross-fitted on the other folds and the operating points are chosen on the other folds (nested). One compile per profile, not one per fold. Intervals are 95% repository-cluster bootstrap.
- Two operating points. BLOCK is offered only when calibration holds (slope CI contains 1,
intercept CI contains 0, ECE <= 0.05) and a fold reaches the required precision; otherwise the
engine is ADVISORY only. An
unsureanswer can never BLOCK.
First measured slice (2026-10-01)
Record: benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json (counts
only). Family injection only; memory of 724 static examples from all families; input profile
v1 with the engine signals and function length withheld (a leak audit found them separating the
sides of a pair by construction: 2026-10-01-decision-engine-leak-audit/); framework priors off.
| Measure | Value (95% repository-cluster CI) |
|---|---|
| Candidates evaluated | 110 in 29 repository groups, 58 positive |
| ADVISORY recall | 0.90 (0.82 - 0.97) |
| ADVISORY precision | 0.54 (0.44 - 0.65) |
AUC of the score s |
0.74 (0.65 - 0.85) |
| Majority "vulnerable" verdict | precision 0.86 (0.72 - 1.0), recall 0.43 (0.29 - 0.62) |
| BLOCK | not offered: ECE 0.064 > 0.05, and every fold reported precision_unreachable |
| Compiled program | balanced Brier 0.128 on the validation split vs 0.199 for the baseline instruction |
| Canary agreement on re-run | 0.83 (30 candidates) |
| Spend | $0.91 metered at list price for DeepSeek, $0.0040 for embeddings; $0.0032 per evaluated candidate |
The other evaluable families are not evaluated yet. Leave-one-source-out, leave-one-framework-out and learning curves, experiments, integration into scan and pre-push, adoption, and the one-time v3 check are still ahead.
Commands
ousast learn is a maintainer command group over the engine's data. Paid steps take
--budget-usd; --replay-only answers every call from the response cache at no cost.
| Command | Does |
|---|---|
ousast learn labels |
labels from the allow-listed sources (host only); --snapshot writes the counts-only record |
ousast learn memory build |
label + feature record + excerpt -> example rows (no model) |
ousast learn memory embed |
embed the stored excerpts (OpenRouter, metered, cached) |
ousast learn compile |
compile the program on the compile split |
ousast learn evaluate --program ID |
outer-fold evaluation, cross-fitted calibration, nested operating points |
ousast learn curve --program ID |
the metric over memory size on a fixed candidate subset |
ousast learn audit-leaks |
how well each signal alone separates the sides of a pair (no model) |
ousast learn experiment register|units|run|analyse MANIFEST |
A/B experiments from a pre-registered manifest (plane/experiments/<id>.yaml): record its digest, freeze the units (no model), run the arms paired (arm A may replay from the cache at $0), analyse (repository bootstrap, exact McNemar, two looks, the adoption verdict) |
The memory store is the plane's (OUSAST_MEMORY; see ops/ax/README.md).
The harvest that produced the plane signals runs as plane Runs generated by
ousast plane harvest (benchmarks/measurements/2026-10-01-decision-engine-harvest/record.json).