Open_Ultra_SAST/.kiro/steering/overview.md

Overview — what this is, and what is actually true of it

Written 2026-09-09. Fourteen specs have accumulated and the roadmap still described a pre-Joern world, so this is the synthesis above them: what the system does, what has been measured, and what is not true yet. Every number here comes from a committed artifact in benchmarks/measurements/.

What the tool is

A static analyser that separates what it can prove from what it merely suspects, and says which is which.

The thesis is not the dataflow engine — CodeQL and Semgrep have one and developers do not turn it on. It is the rung: a claim you can act on without writing a query, and an explicit admission when the engine could not decide.

execution_confirmed   a test or sandbox reproduced it                (built, no production caller)
model_entailed        the graph decided: a flow with no discharge,
                      an obligation with no guard, a permissive value  ← the product
model_corroborated    the graph found the shape; a judge with
                      coverage agreed on the residual question
suspicion             an enumerator or a pattern proposed it and
                      nothing corroborated it                          ← everything below the line is noise
                                                                          until proven otherwise

Silence is not a clean bill of health. If no fact table covers a site, the model is silent, not negative — 23 of the 39 misses in the original entailment ceiling were sinks simply absent from the tables.

How it works, in one pass

preprocess ─▶ entry-point map ─▶ MAP stage ─┬─ static patterns
                                            ├─ semantic overlay
                                            ├─ obligations (absence bugs)
                                            ├─ tool hunter (LLM)
                                            └─ MODEL LAYER ──▶ one CPG per repository (Joern, subprocess)
                                                               ├─ taint      reachability + sanitizer lattice
                                                               ├─ dominance  guard governs obligated operation
                                                               └─ config     constant abstraction over a value

Three abstractions, one arbiter each, dispatched on spec type so a family without an arbiter cannot be silently routed to the wrong one. The CPG is built once per scan and every question is batched into one invocation per query kind.

What has been measured

	measured	source
entailment, flat IR → CPG	2.2% → 41.3% of the corpus	2026-09-08-model-entailment-ceiling*.json
vibe-py pairs (30)	90.0% pair-correct, 6.7% leak, $0.0068	2026-09-08-pipeline-vibe-py.json
— against the LLM hunter	76.7% / 12.3% at 5× the calls, ~100× the cost	same
vampi (python, 520 lines)	3 of 3 known vulnerabilities at model_entailed	2026-09-09-repo-vampi-authz-precision.json
vampi access-control precision	28.6% → 50.0%	same
access-control leak, pairs nobody tuned against	42.9% → 0.0%, pair-correct unchanged	same
libpng envelope (89k lines of C)	build 16s, query 250s, scan 446s, 911 MB	2026-09-09-repo-libpng-envelope.json
libpng valid findings	zero	same
report proportionality	libpng 198 sections → 39	2026-09-09-report-proportionality.json
CVE-2023-23488, production plugin code	found at class.memberorder.php:936, the exact line	2026-09-09-pmpro-cve-found.json
— precision on that file	16 of 26 unprepared $wpdb queries; 0 of 26 use ->prepare	same

What is not true yet

• The C story does not work. c/memory has produced zero true positives and three distinct classes of false one. It models string functions; libpng overflows through arithmetic, and its library code never calls a sink we model. This is structural, not tuning.
• Two languages are demonstrated. Python end to end, and PHP on a real plugin slice where it found a real CVE. Ruby and Go still have zero families — Joern parses them and the engine says nothing.
• Not at repository scale for PHP. The 637-file checkout does not load; the 2-file slice does. That boundary is unmeasured (task 5.10).
• The suspicion band is still too loud. vampi emits 61 report sections for a 520-line application, 38 of them a candidate the graph could not decide with a judge's opinion attached.
• The evidence base is two repositories. One has never produced a true positive.
• Control dependence is unavailable. 0 of 13,184 calls in the libpng CPG and 0 of 1,152 in vampi's have a controlling control structure, so every guard question runs on CFG dominance instead.
• execution_confirmed has no production caller. The rung exists and nothing reaches it.

What scoping is for, which is not what it looks like

The first real CVE was found by cutting, not by scanning more. Two of Paid Memberships Pro's 637 files — the ones holding both ends of the bug — load in 9 seconds, query in 111, and report CVE-2023-23488 at the exact line. The full checkout does not load at all.

That two-file slice is the contributor case. A developer editing class.memberorder.php gets precisely that scan. So scoping is not a workaround for a scale problem; it is the thesis, and the scale problem is what happens when the thesis is ignored.

The distinction that matters is not whether to cut but who decides and whether the decision is recorded:

	decided by	recoverable afterwards
scoping — a build declaration, a diff, a subtree	the project or the developer	yes, and it is in the artifact
triage — "this looks like boilerplate"	a model	no; if it drops the file with the bug, the scan reports clean

Both give the same answer when the model is right. Only one of them tells you when it was wrong, and every failure found on this checkout was of exactly that shape.

Why a corpus of benchmarks is not enough on its own

This is the finding worth teaching, and it is not an opinion — it is what the gates say.

Twelve engine defects were found in one session by pointing the tool at three real repositories. The three gates stayed byte-identical through every single fix. A gate that does not move is a corpus that could not see the bug.

defect	what the corpus saw
resolve matched inside resolveUrl (sanitizers)	nothing
user matched inside users (dischargers)	nothing
gets matched inside fgets (sinks) — 26 identical false positives	nothing
the model layer had zero production callers	nothing
query_batch implemented, tested, unreachable	nothing
a paid budget halting unpaid arbitration	nothing
a region with no function claiming the whole repository's answer	nothing
a guard discharging an obligation it is not a guard of	nothing
joern-parse failing where the frontend succeeds	nothing
a 1.49MB batch over Linux's 128KB argv cap	nothing
an array callable not recognised as a handler	nothing
one defect reported as twenty-six findings	nothing

The reason is structural rather than a gap in the corpus's coverage. A pair is a labelled function with a declared family and a known answer. It cannot exercise: which region to look at, what to spend, whether the capability is even reachable, whether two entry points share a sink, whether the question fits in an argv slot, or whether a flow crosses a module boundary. Every one of those is a property of a repository, and none of them is a property of a function.

Three of the twelve are worse than invisible — they are quiet failures, where the tool produced a clean report and every honesty counter agreed with it. The rung ladder labels a verdict; when no verdict is produced there is nothing to label. That is the failure mode a benchmark score cannot express at all, because the score is computed from the verdicts that exist.

So the corpus is necessary and not sufficient: it is the holdout that stops per-repository tuning, and it is the wrong instrument for finding out whether the tool works. Both instruments, or neither number means anything.

The lesson that shaped the current specs

Nine engine defects were found in one session by pointing the tool at two real repositories. Three were the same bug in three matchers — a spec token matched as a substring: resolve in resolveUrl, user in users, gets in fgets. Two were a capability built and unreachable: the model layer itself, then the query batcher. One was a paid budget stopping unpaid work.

The three gates stayed byte-identical through every one of those fixes. That is the finding: a corpus of function-level pairs could not see any of them, and a repository showed all nine within minutes.

Where the specs stand

spec	phase	state
contributor-scan	implementation	14 done, 15 open — the active one
finding-feedback-loop	design	requirements approved, design awaiting approval
model-grounded-detection	complete	16/16; built the arbiter this all rests on
authorization-obligations	tasks approved	24 open — superseded in practice by the dominance arbiter
constrained-detector	tasks	4 of 20, tasks unapproved
reachability-flow-model	tasks generated	23 open, nothing approved — largely delivered by adopting Joern
learning-harness	implementation	30 done, 3 open
six others	complete	pair-corpus-honesty, corpus-seeded-mechanisms, propose-adjudicate-prove, three-stage-scan, real-world-vfc-slice, harnessx-self-improving-rulesets, tree-sitter-overlay-extra

97 source modules, 90 load-bearing and 7 standalone capabilities; 814 tests.