Open_Ultra_SAST/.kiro/specs/learned-decision-engine/fix-mechanisms-plan.md

Plan: learn fix mechanisms into the engine (draft for review, 2026-10-04)

Status: approved 2026-10-04 by the maintainer. Increment 1 = discharge facts only; coverage additions (Req 7) come later. Decisions: minimum 2 proposing repositories before a candidate is tried; admission per language (not pooled by callee name).

Why

The Joern trace run over exp-005's 501 units shows the engine cannot tell a vulnerable function from its fix. In the 105 pairs where both sides were analysed, a taint path appears on both sides in 12, on the vulnerable side only in 2, on the fixed side only in 0, and on neither in 91 (benchmarks/measurements/2026-10-04-joern-slice-leak-audit/record.json).

So the fix is invisible to the engine. Either the flow survives and the engine does not know the fix's mechanism (an allowed_classes argument, a path normalisation check, an escaping call), or the engine's sources and sinks miss the function on both sides. A classifier fed this evidence cannot improve on it.

The corpus now holds 710 complete fix pairs (2026-10-04-decision-engine-harvest-3/record.json). Each fix diff shows what was added. That is the material the engine lacks.

Goal

Grow the engine's discharge knowledge (sanitizers, guards, safe-argument forms) and, second, its source and sink coverage, from the fix diffs. Admit a fact only when it improves held-out separation. Spend no model money.

Requirements (proposed)

1. Mining. From each fix pair, extract what the fixed function adds on the vulnerable flow: new calls, new conditions guarding the sink, new or changed sink arguments. Record each as a candidate fact with its language, family, form (sanitizer call, guard predicate, safe argument, sink restriction) and the pairs that propose it. Mining is deterministic (tree-sitter diff of the two function bodies). No model calls.
2. Generalisation is explicit. A candidate is a pattern, not a repository name: a callee name, an argument key, a predicate shape. Candidates naming project-specific identifiers are rejected (Req 8: no application-specific trees).
3. Admission by held-out separation. Folds are by repository, as in the decision engine. A candidate is admitted for a fold only if, on that fold's held-out pairs, adding it to the fact tables:
   • increases pairs where the vulnerable side keeps a path and the fixed side loses it or becomes sanitized;
   • and does not remove any held-out vulnerable-side path (no recall loss). A candidate proposed by fewer than 2 distinct repositories in the training folds is not tried.
4. No train-on-test. Mining and admission never read held-out pairs; the closed-loop leak recorded in memory (closed-loop-train-on-test-leak) is the failure this rule exists to prevent. Population v3 is never read.
5. Measurement by the trace runner. Each admission round re-runs the engine on the affected pins only (the runner's resume and selection already support this; on k3s the worker Deployment runs them in parallel). The outcome is the pair-asymmetry table per family, before and after, on held-out folds.
6. Gate before production. An admitted fact enters src/openultrasast/ruleset/semantic/*.toml only after the detection gates (gate, map_gate, pair_gate) show no regression, and only as a registered experiment whose decision rule was fixed before the run.
7. Coverage second. After discharge facts, the same loop proposes source and sink additions from pairs where neither side has a path (91 of 105 today), admitted under the same held-out rule.

How it relates to what exists

• corpus-seeded-mechanisms already has a closed loop that admits and retracts mechanisms by leave-one-out. This plan reuses its lever and store, and points it at discharge facts mined from real fix diffs instead of variant search.
• benchmarks/pairs/mechanisms.toml (23 mechanisms) labels what kind of bug a pair is. The new candidates describe what a fix does. They live in the fact tables, not in the mechanism vocabulary.

Measures of success

• Pair asymmetry on held-out folds: vulnerable-only paths rise from 2 of 105 toward a substantial share, with zero fixed-only paths.
• No held-out recall loss on vulnerable sides.
• Then the decision engine is re-run with the new engine evidence (exp-005 design), and the pooled block gate (exp-004 design) is re-measured.

Cost and time

• Model cost: $0 (mining and admission are deterministic; the classifier re-run afterwards is a separate, approved spend).
• Engine time: one pass over affected pins per admission round; about 2 hours per full pass on k3s plus the VM, about 6 hours on the VM alone.

Open questions for the maintainer

1. Should coverage additions (Req 7) be in the same increment or a later one?
2. Minimum proposing repositories: 2 is the proposal. Higher is safer, lower finds more.
3. Admission per family and language, or pooled across languages for the same callee name?