Open_Ultra_SAST/.kiro/steering/overview.md

Overview — what this is, and what is actually true of it

Current state, 2026-10-13

Everything below this section is dated history; where it disagrees with this section, this section wins.

• Goal, unchanged. A pre-push safety net that works on repositories it has never seen, judged by recall on real vulnerability changes and false alarms per 100 ordinary pushes, measured on a valid benchmark.
• The valid benchmark exists and ran (valid-detection-benchmark, benchmarks/valid/, pilot frozen at benchmarks/valid/pilot/; numbers in benchmarks/measurements/2026-10-12-valid-benchmark-pilot/NOTES.md). 12 verified fix-site positives, 240 ordinary pushes, 12 unseen app repos. Push hook as shipped: 0 of 12, 0.7 false alarms per 100. LLM hunter: 5 to 6 of 12, 54 false alarms per 100 after the classifier. Neither meets the gate. The SZZ pool (pool-p1) and its spec are retired; no number from it is quoted.
• What the numbers say. The hook has no LLM detection tier and catches nothing on unseen code. The hunter finds about half but searches whole files, so it re-reports old code on every push (250 of 343 "vulnerable" suspicions on ordinary pushes sit more than 10 lines from the diff). Thresholds on that output are not a fix.
• Active work, in order. (1) hunter-learning-loop: an annotation model for the hunter (diff hunks, enclosing function and callers, entry points, roles, framework facts, closed families) chosen by statistical ablation under the learn harness; requirements written 2026-10-13, design next. (2) pre-push-safety-net: the LLM tier on pre-push once the search is good. (3) search-with-proof / proof-arm-opencode: proof on what survives the search, as the block gate. (4) engine-change-scoped-analysis: the deterministic facts that feed the hunter's annotations.
• Everything else is history or parked. Retired spec directories were removed from the tree on 2026-10-13 (git show <commit>:.kiro/specs/<name>/ still has them): harnessx-removal (51d064e), harnessx-self-improving-rulesets and openrouter-sast-harness (c8d2fef), learning-harness, constrained-detector, three-stage-scan, real-world-vfc-slice (39a1873), unseen-repo-evaluation (abb2058). The learn harness code (src/openultrasast/learn/) stays; hunter-learning-loop builds on it.
• Rules that stay in force. Plain English in every document, commit and comment. Standard metrics only. Prove the instrument before believing a zero. Train on one set, report on another. Research runs locally; deployed behaviour is observed, not re-measured by hand.

Current state, 2026-10-04 (superseded by the section above)

Everything below this section is dated history; where it disagrees with this section, this section wins. The direction lives in roadmap.md, section of 2026-10-04 (phases G0-G5).

• Goal. A pre-push safety net that works on repositories it has never seen. Two numbers per push decide it: catch rate on real vulnerability-introducing changes, and false-alarm rate on ordinary changes. Neither has been measured yet; the unseen-repository pool (G0, spec unseen-repo-evaluation) comes first.
• Correction, 2026-10-04. Infrastructure and engine work had drifted from that goal. It is now done only when it removes a bottleneck for those measurements.
• Engine. Joern runs on the operator's kube-ax cluster as AX Tasks, with a VM fallback lane; kind stays the development profile. It is almost blind to fixes: paths differ between sides in 2 of 105 both-asked pairs.
• Decision engine. Ranks but cannot block (exp-003 to exp-005 inconclusive; 710 fix pairs in memory). Fix mechanisms is one G2 arm, paused after mining until the pool exists.
• Hook. ousast pre-push is operational and advisory on any repository; rollout stays NO-GO.

State, 2026-10-02 (superseded by the section above)

User-facing documentation: README.md, docs/architecture.md, docs/evaluation.md, docs/decision-engine.md, ops/ax/README.md.

• Scanning. ousast scan --mode quick|standard|deep. Quick rules for C/C++, Python, JavaScript/TypeScript, Java/Groovy and PHP (PHP numbers are in-sample only: benchmarks/measurements/2026-09-30-php-quick-rules/). Standard adds the MAP stage with the Joern model layer (shipped in the Docker image); deep adds sandboxed regression snippets when Docker is available. Framework knowledge is tagged as optional priors (ruleset/frameworks.toml, priors=). ousast pre-push compares head against base; the capability registry is empty, rollout NO-GO.
• Agentic plane. google/ax over Agent Substrate on kind is the only executor of model work over whole repositories (ousast plane run|status|doctor|workspaces|remember|alerts-engine|harvest). Credentials travel only in the start request through atenet-router; egress is per task; budgets are per task; rows go to the memory store (OUSAST_MEMORY: FileStore, or S3Store on an S3-compatible bucket; RustFS is the tested server).
• Removed. HarnessX (retired 2026-09-30): its config keys fail, a [harnessx] section warns. The LLM judge of the verifier and the LLM fusion panels are gone; fusion is deterministic.
• Improvement. ousast improve --memory proposes rule-status edits from plane memory through the unchanged validator and gate; the loop also runs as a plane Run.
• Decision engine (learned-decision-engine, implementation, not adopted). First slice, injection only: ADVISORY recall 0.90, precision 0.54, BLOCK not offered (benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json).
• Evaluation. Populations v1 and v2 are spent and failed every M4 gate (benchmarks/independent/results-v1.json, results-v2.json); v3 (PHP) is frozen and untouched for the one-time final check. The plane increment met its gate on the second measurement with a tie-break chosen after the first (benchmarks/independent/plane-increment-2.json).
• Providers. DeepSeek for LLM calls, OpenRouter only for OpenAI embeddings; keys in .env, which never overrides an exported shell variable.

History

Direction, 2026-09-12: an actionable pre-push security safety net for AI-accelerated development across supported ecosystems. See safety-net.md and the approved pre-push-safety-net spec. The ranker remains the scope mechanism; third-party vendor code stays outside targets and graphs. Advisory must meet the same actionability bar as blocking.

Historical scoreboard below: it was written September 9. Later commits record a completing whole-PMPro scan and Joern 4.0.625 ranking measurements (82/91/54 on the three plugins), while default installation pins are now aligned to 4.0.625 with verified PHP/JavaScript image smoke. The pre-push workflow, changed-code latency and actionable precision are not demonstrated by those results. In particular, the manually selected two-file slice below is a useful experiment, not evidence that an automatic push scope would select the same context. Current scope/status corrections live in flow-aware-ranking/brief.md and the roadmap.

The new scoring instrument retains missing targets and unanswered queries. Historical rank positions do not establish detection recall; VAmPI's old budget of 16 omitted its unmatched SQLI target. The corrected positional run keeps that target unresolved. Labeled PMPro/Node inputs and a frozen diagnostic profile are committed; independent capability evaluation is still required.

Written 2026-09-09. Fourteen specs have accumulated and the roadmap still described a pre-Joern world, so this is the synthesis above them: what the system does, what has been measured, and what is not true yet. Every number here comes from a committed artifact in benchmarks/measurements/.

What the tool is

A static analyser that separates what it can prove from what it merely suspects, and says which is which.

The thesis is not the dataflow engine — CodeQL and Semgrep have one and developers do not turn it on. It is the rung: a claim you can act on without writing a query, and an explicit admission when the engine could not decide.

execution_confirmed   a test or sandbox reproduced it                (built, no production caller)
model_entailed        the graph decided: a flow with no discharge,
                      an obligation with no guard, a permissive value  ← the product
model_corroborated    the graph found the shape; a judge with
                      coverage agreed on the residual question
suspicion             an enumerator or a pattern proposed it and
                      nothing corroborated it                          ← everything below the line is noise
                                                                          until proven otherwise

Silence is not a clean bill of health. If no fact table covers a site, the model is silent, not negative — 23 of the 39 misses in the original entailment ceiling were sinks simply absent from the tables.

How it works, in one pass

preprocess ─▶ entry-point map ─▶ MAP stage ─┬─ static patterns
                                            ├─ semantic overlay
                                            ├─ obligations (absence bugs)
                                            ├─ tool hunter (LLM)
                                            └─ MODEL LAYER ──▶ one CPG per repository (Joern, subprocess)
                                                               ├─ taint      reachability + sanitizer lattice
                                                               ├─ dominance  guard governs obligated operation
                                                               └─ config     constant abstraction over a value

Three abstractions, one arbiter each, dispatched on spec type so a family without an arbiter cannot be silently routed to the wrong one. The CPG is built once per scan and every question is batched into one invocation per query kind.

What has been measured

	measured	source
entailment, flat IR → CPG	2.2% → 41.3% of the corpus	2026-09-08-model-entailment-ceiling*.json
vibe-py pairs (30)	90.0% pair-correct, 6.7% leak, $0.0068	2026-09-08-pipeline-vibe-py.json
— against the LLM hunter	76.7% / 12.3% at 5× the calls, ~100× the cost	same
vampi (python, 520 lines)	3 of 3 known vulnerabilities at model_entailed	2026-09-09-repo-vampi-authz-precision.json
vampi access-control precision	28.6% → 50.0%	same
access-control leak, pairs nobody tuned against	42.9% → 0.0%, pair-correct unchanged	same
libpng envelope (89k lines of C)	build 16s, query 250s, scan 446s, 911 MB	2026-09-09-repo-libpng-envelope.json
libpng valid findings	zero	same
report proportionality	libpng 198 sections → 39	2026-09-09-report-proportionality.json
CVE-2023-23488, production plugin code	found at class.memberorder.php:936, the exact line	2026-09-09-pmpro-cve-found.json
— precision on that file	16 of 26 unprepared $wpdb queries; 0 of 26 use ->prepare	same

What is not true yet

• The C story does not work. c/memory has produced zero true positives and three distinct classes of false one. It models string functions; libpng overflows through arithmetic, and its library code never calls a sink we model. This is structural, not tuning.
• Two languages are demonstrated. Python end to end, and PHP on a real plugin slice where it found a real CVE. Ruby and Go still have zero families — Joern parses them and the engine says nothing.
• Not at repository scale for PHP. The 637-file checkout does not load; the 2-file slice does. That boundary is unmeasured (task 5.10).
• The suspicion band is still too loud. vampi emits 61 report sections for a 520-line application, 38 of them a candidate the graph could not decide with a judge's opinion attached.
• The evidence base is two repositories. One has never produced a true positive.
• Control dependence is unavailable. 0 of 13,184 calls in the libpng CPG and 0 of 1,152 in vampi's have a controlling control structure, so every guard question runs on CFG dominance instead.
• execution_confirmed has no production caller. The rung exists and nothing reaches it.

What scoping is for, which is not what it looks like

The first real CVE was found by cutting, not by scanning more. Two of Paid Memberships Pro's 637 files — the ones holding both ends of the bug — load in 9 seconds, query in 111, and report CVE-2023-23488 at the exact line. The full checkout does not load at all.

That two-file slice is the contributor case. A developer editing class.memberorder.php gets precisely that scan. So scoping is not a workaround for a scale problem; it is the thesis, and the scale problem is what happens when the thesis is ignored.

The distinction that matters is not whether to cut but who decides and whether the decision is recorded:

	decided by	recoverable afterwards
scoping — a build declaration, a diff, a subtree	the project or the developer	yes, and it is in the artifact
triage — "this looks like boilerplate"	a model	no; if it drops the file with the bug, the scan reports clean

Both give the same answer when the model is right. Only one of them tells you when it was wrong, and every failure found on this checkout was of exactly that shape.

Why a corpus of benchmarks is not enough on its own

This is the finding worth teaching, and it is not an opinion — it is what the gates say.

Twelve engine defects were found in one session by pointing the tool at three real repositories. The three gates stayed byte-identical through every single fix. A gate that does not move is a corpus that could not see the bug.

defect	what the corpus saw
resolve matched inside resolveUrl (sanitizers)	nothing
user matched inside users (dischargers)	nothing
gets matched inside fgets (sinks) — 26 identical false positives	nothing
the model layer had zero production callers	nothing
query_batch implemented, tested, unreachable	nothing
a paid budget halting unpaid arbitration	nothing
a region with no function claiming the whole repository's answer	nothing
a guard discharging an obligation it is not a guard of	nothing
joern-parse failing where the frontend succeeds	nothing
a 1.49MB batch over Linux's 128KB argv cap	nothing
an array callable not recognised as a handler	nothing
one defect reported as twenty-six findings	nothing

The reason is structural rather than a gap in the corpus's coverage. A pair is a labelled function with a declared family and a known answer. It cannot exercise: which region to look at, what to spend, whether the capability is even reachable, whether two entry points share a sink, whether the question fits in an argv slot, or whether a flow crosses a module boundary. Every one of those is a property of a repository, and none of them is a property of a function.

Three of the twelve are worse than invisible — they are quiet failures, where the tool produced a clean report and every honesty counter agreed with it. The rung ladder labels a verdict; when no verdict is produced there is nothing to label. That is the failure mode a benchmark score cannot express at all, because the score is computed from the verdicts that exist.

So the corpus is necessary and not sufficient: it is the holdout that stops per-repository tuning, and it is the wrong instrument for finding out whether the tool works. Both instruments, or neither number means anything.

The lesson that shaped the current specs

Nine engine defects were found in one session by pointing the tool at two real repositories. Three were the same bug in three matchers — a spec token matched as a substring: resolve in resolveUrl, user in users, gets in fgets. Two were a capability built and unreachable: the model layer itself, then the query batcher. One was a paid budget stopping unpaid work.

The three gates stayed byte-identical through every one of those fixes. That is the finding: a corpus of function-level pairs could not see any of them, and a repository showed all nine within minutes.

Where the specs stand

Counted 2026-10-02 from each spec's tasks.md checkboxes and spec.json.

spec	phase	state
learned-decision-engine	implementation	11 of 17 tasks done
pre-push-safety-net	implementation	67 of 79 tasks done
contributor-scan	paused	27 of 51 tasks done; paused since 2026-09-23 (v1.2.0-alpha.1 NO-GO), waits on pre-push M2/M3
ai-service-plane	complete	11 of 11; the ax plane is the agentic path
harnessx-removal	complete, directory removed 2026-10-13	8 of 8; HarnessX removed 2026-09-30
finding-feedback-loop	dormant	requirements approved, design unapproved since 2026-09-09; depends on contributor-scan
model-grounded-detection	complete	16 of 16; built the arbiter this all rests on
authorization-obligations	dormant	tasks approved, 0 of 24 started; superseded in practice by the dominance arbiter
reachability-flow-model	dormant	23 tasks generated, nothing approved; largely delivered by adopting Joern
flow-aware-ranking, zero-with-a-reason, ir-consolidation	dormant	brief only, unapproved
four others	complete	pair-corpus-honesty, corpus-seeded-mechanisms, propose-adjudicate-prove, tree-sitter-overlay-extra
six others	retired or superseded, directories removed 2026-10-13	openrouter-sast-harness, three-stage-scan, harnessx-self-improving-rulesets, learning-harness (retired with HarnessX), constrained-detector, real-world-vfc-slice (superseded)

Retired 2026-09-30: HarnessX was removed by harnessx-removal; the agentic plane is ax (ai-service-plane), and evolutionary proposals come from the plane memory. The HarnessX specs in the table above are history.

151 source modules, 146 load-bearing and 5 standalone capabilities, 0 orphaned (benchmarks/measurements/2026-09-08-module-audit.json, regenerated 2026-10-02); 2,023 tests collected.