From 8d6921d2dfa10280b3c9019573b105f14753881d Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Sun, 4 Oct 2026 14:11:41 +0200 Subject: [PATCH] spec(learned-decision-engine): fix-mechanisms plan for review -- learn discharge facts from 710 fix diffs, admitted by held-out pair separation Co-Authored-By: Claude Opus 5.5 --- .../fix-mechanisms-plan.md | 79 +++++++++++++++++++ 1 file changed, 79 insertions(+) create mode 100644 .kiro/specs/learned-decision-engine/fix-mechanisms-plan.md diff --git a/.kiro/specs/learned-decision-engine/fix-mechanisms-plan.md b/.kiro/specs/learned-decision-engine/fix-mechanisms-plan.md new file mode 100644 index 0000000..d64c554 --- /dev/null +++ b/.kiro/specs/learned-decision-engine/fix-mechanisms-plan.md @@ -0,0 +1,79 @@ +# Plan: learn fix mechanisms into the engine (draft for review, 2026-10-04) + +Status: proposed. Not approved. Amends the learned-decision-engine spec once the maintainer approves it. + +## Why + +The Joern trace run over exp-005's 501 units shows the engine cannot tell a vulnerable function from its +fix. In the 105 pairs where both sides were analysed, a taint path appears on both sides in 12, on the +vulnerable side only in 2, on the fixed side only in 0, and on neither in 91 +(`benchmarks/measurements/2026-10-04-joern-slice-leak-audit/record.json`). + +So the fix is invisible to the engine. Either the flow survives and the engine does not know the fix's +mechanism (an `allowed_classes` argument, a path normalisation check, an escaping call), or the engine's +sources and sinks miss the function on both sides. A classifier fed this evidence cannot improve on it. + +The corpus now holds 710 complete fix pairs (`2026-10-04-decision-engine-harvest-3/record.json`). Each fix +diff shows what was added. That is the material the engine lacks. + +## Goal + +Grow the engine's discharge knowledge (sanitizers, guards, safe-argument forms) and, second, its source and +sink coverage, from the fix diffs. Admit a fact only when it improves held-out separation. Spend no model +money. + +## Requirements (proposed) + +1. **Mining.** From each fix pair, extract what the fixed function adds on the vulnerable flow: new calls, + new conditions guarding the sink, new or changed sink arguments. Record each as a candidate fact with its + language, family, form (sanitizer call, guard predicate, safe argument, sink restriction) and the pairs + that propose it. Mining is deterministic (tree-sitter diff of the two function bodies). No model calls. +2. **Generalisation is explicit.** A candidate is a pattern, not a repository name: a callee name, an + argument key, a predicate shape. Candidates naming project-specific identifiers are rejected (Req 8: no + application-specific trees). +3. **Admission by held-out separation.** Folds are by repository, as in the decision engine. A candidate is + admitted for a fold only if, on that fold's held-out pairs, adding it to the fact tables: + - increases pairs where the vulnerable side keeps a path and the fixed side loses it or becomes + sanitized; + - and does not remove any held-out vulnerable-side path (no recall loss). + A candidate proposed by fewer than 2 distinct repositories in the training folds is not tried. +4. **No train-on-test.** Mining and admission never read held-out pairs; the closed-loop leak recorded in + memory (`closed-loop-train-on-test-leak`) is the failure this rule exists to prevent. Population v3 is + never read. +5. **Measurement by the trace runner.** Each admission round re-runs the engine on the affected pins only + (the runner's resume and selection already support this; on k3s the worker Deployment runs them in + parallel). The outcome is the pair-asymmetry table per family, before and after, on held-out folds. +6. **Gate before production.** An admitted fact enters `src/openultrasast/ruleset/semantic/*.toml` only after + the detection gates (`gate`, `map_gate`, `pair_gate`) show no regression, and only as a registered + experiment whose decision rule was fixed before the run. +7. **Coverage second.** After discharge facts, the same loop proposes source and sink additions from pairs + where neither side has a path (91 of 105 today), admitted under the same held-out rule. + +## How it relates to what exists + +- `corpus-seeded-mechanisms` already has a closed loop that admits and retracts mechanisms by leave-one-out. + This plan reuses its lever and store, and points it at discharge facts mined from real fix diffs instead + of variant search. +- `benchmarks/pairs/mechanisms.toml` (23 mechanisms) labels what kind of bug a pair is. The new candidates + describe what a fix does. They live in the fact tables, not in the mechanism vocabulary. + +## Measures of success + +- Pair asymmetry on held-out folds: vulnerable-only paths rise from 2 of 105 toward a substantial share, + with zero fixed-only paths. +- No held-out recall loss on vulnerable sides. +- Then the decision engine is re-run with the new engine evidence (exp-005 design), and the pooled block gate + (exp-004 design) is re-measured. + +## Cost and time + +- Model cost: $0 (mining and admission are deterministic; the classifier re-run afterwards is a separate, + approved spend). +- Engine time: one pass over affected pins per admission round; about 2 hours per full pass on k3s plus the + VM, about 6 hours on the VM alone. + +## Open questions for the maintainer + +1. Should coverage additions (Req 7) be in the same increment or a later one? +2. Minimum proposing repositories: 2 is the proposal. Higher is safer, lower finds more. +3. Admission per family and language, or pooled across languages for the same callee name?