Open_Ultra_SAST/.kiro/steering/roadmap.md

Roadmap

Roadmap to detection with the decision engine on kube-ax (2026-10-04)

This section supersedes the ordering below it. The older sections stay as history.

Destination

On every push, from a hook or CI, OpenUltraSAST answers for the changed code. The answer is advisory for every family, and BLOCK only for families whose precision has been proven on repositories the system never saw. The work is split by what each place is good at:

• kube-ax cluster (operator k3s, Agent Substrate v0.3.0, AX v0.3.1, 2 nodes): the Joern engine, as AX Tasks. No API keys and no store credentials in the cluster; tasks read and write through short-lived presigned links.
• The VM or CI runner: candidate selection, the decision engine's model calls (DeepSeek), the memory store, and the fallback engine lane (2.5 GB heap) for pins that do not fit the cluster's 1.7 GB heap.
• kind stays the local development profile, with the same code and manifests.

The question everything answers

Does the safety net work on repositories it has never seen? That is the purpose of the A/B experiments, and every phase below is judged by it. On an unseen repository almost every push is an ordinary change, so two numbers decide whether the safety net is useful, measured on the hook's advisory output exactly as a user would see it:

1. Catch rate: on real vulnerability-introducing changes in unseen repositories, how often it flags the change.
2. False-alarm rate: on ordinary changes in the same unseen repositories, how often it flags anything.

Within-pair AUC on known fix pairs answers neither. It stays a development signal, not the goal.

Where we are (2026-10-04)

Area	State	Evidence
Generalisation measured so far	The classifier ranks above chance on repository-held-out folds (within-pair AUC about 0.76) but cannot reach the block bar; the engine-based scan failed fresh populations (v1 1/11, v2 0/17); push-level catch and false-alarm rates on unseen repositories have never been measured	exp-003/004/005, results-v1.json, results-v2.json
Pre-push hook	Operational and advisory on any repository	2026-10-02-pre-push-operational-1/record.json
Labelled data	710 complete real fix pairs	2026-10-04-decision-engine-harvest-3/record.json
Engine	Almost blind to fixes (paths differ between sides in 2 of 105 both-asked pairs)	2026-10-04-joern-slice-leak-audit/record.json
Infrastructure	Engine runs on kube-ax as AX Tasks with a VM fallback; kind for development	memory kube-ax-cluster

Correction recorded 2026-10-04: the previous two days went mostly into infrastructure and engine internals that did not move a generalisation number. From here, infrastructure and engine work is done only when it removes a bottleneck for the measurements below.

Where we are (2026-10-11)

The G0 measurement is now runnable: the unseen pool is frozen and the whole pipeline is hardened and bucket-first. The baseline replay is the one remaining step, to run on kube.

Area	State	Evidence
Unseen pool (G0.1)	Frozen. pool-p1.toml = 100 repos (slice 1); freeze-p1.json guard=0, dropped=0; deterministic (manifest 6bd2fa5d). License 49 permissive / 51 private; methods 81 introducing / 19 last_touch.	benchmarks/unseen/pool-p1.toml; archived ousast-corpus/unseen/20260601/pool/
Labels	100 machine-derived (SZZ), unchecked. Record 07 is a machine-label baseline (caveat stated); a human label pass is a later refinement, before any published claim.	freeze-p1.json label_check_counts
Harvest	Rebuilt to converge. Pure GitHub-API harvesting is a dead end for this benchmark (it needs the SZZ introducing commit + 12 stratified history negatives, not one commit). The fix: a full-objects clone (drop --filter=blob:none, which caused ~600 per-blob lazy fetches) + bucket-first durability + a get_repo cache that paid the 4,499-name used-set canonicalization once. Yield ~2.2%; GitHub-rate-limit bound, not CPU.	95bed70, 857263d, 21aebe8, 7b711a6
Persistence	Bucket-first, not local disk. ousast-corpus is the source of truth: enumeration, resolution, get_repo, and accept/reject decisions all stream to and resume from it. The host disk (~9-13 GB free) is the scarce resource; RustFS/S3 has the room, so run artifacts and replay outputs go to RustFS too, not ~/ousast-results on local disk.	storage-streaming.md, bank.py
Replay image	Provenance pinned. CI engine-image.yml builds ousast-replay-task (harness with /app/benchmarks/unseen/replay_entry.py) + ousast-engine-task (Joern hook), publicly pullable by digest. The old ousast-runner:dev (248 MB ax-runner, no /app/benchmarks, no Joern) is the wrong image for replay.	CI run 38116281676
Baseline (G0.2)	Not yet. Replay reads the frozen pool cleanly (dry-run = 1,300 changes, freeze_digest verified). Remaining: mirror CI's passing replay job wiring, and stand up the kube lane (ax binary + a public-HTTPS presign-capable store endpoint; GHCR pull already public).	CI replay job (passed)

Lesson, recorded so it is not re-paid: sourcing and engine-image work cost real time. The durable takeaway now lives in the pipeline and these docs — bucket-first persistence, pinned image provenance, a measured yield — so the next pool or arm is cheap, not a rediscovery.

Readiness plan (to G0 exit = record 07)

1. Kube replay lane (infra, operator-owned): install the ax binary; provide the store endpoint the AX lane needs. The only thing settled in-repo is a code fact: benchmarks/ax/batch.py currently expects a public-HTTPS DNS, presign-capable S3 endpoint. How that endpoint and the cluster's access to the store are provided is an operator/AX decision, not an assumption to make here. GHCR pull of the images is verified public.
2. Unblock the replay by mirroring CI's passing replay job (replay-task + engine-task wired, store + network).
3. Run slice-1 baseline on kube: --lane ax --arm default, 1,300 changes, no model cost, resumable; outputs to RustFS.
4. Adjudicate (7.3) -> record 07 (7.4) = the measured machine-label unseen baseline = G0 exit.
5. Then G1 arms (DeepSeek decision engine) on the frozen pool.

Phases

G0. The unseen-repository pool and the push-level metric (new spec unseen-repo-evaluation)

1. A frozen pool of repositories never used for training, tuning, mining or harvesting, and not in any population. Two kinds of change per repository: real vulnerability-introducing commits (the parent of a fix, post-cutoff where possible) and ordinary commits from the same history.
2. The metric: replay each change through ousast pre-push (and the decision engine where an arm uses it) and record catch rate and false-alarm rate, with confidence intervals, per family and overall.
3. Rotation: the pool is spent per decision, like the populations; a new slice is drawn for the next one.

Exit: the pool is frozen and the current production hook has a measured baseline on it.

G1. Re-score the arms we have on the pool

Current hook (quick rules + engine), decision-engine advisory (current programs), combiner, engine evidence. Each is an arm with a registered decision rule, scored by catch rate at a fixed false-alarm budget.

Exit: one measured table of arms on unseen code; keep only what generalises.

G2. Improve what moves the unseen-repo numbers

Candidates, each admitted only if it improves the pool metric on a fresh slice: fix mechanisms in the engine (approved plan, now one arm, not the spine), engine coverage and TypeScript, decision-engine prompt and retrieval changes, more labelled pairs from more distinct repositories.

Active G2 workstreams (2026-10-11, opened while driving G0; progress tracked in their briefs):

• Change-scoped engine analysis (.kiro/specs/engine-change-scoped-analysis/brief.md). Live diagnosis on the frozen pool showed the engine's "no finding" on unseen changes is usually analysis-never-runs: the engine ranks repo-wide candidates and deferred: deadline_exhausted before analyzing anything (even at 300 s), and the ranked sites are unrelated to the pushed change. Lever: seed analysis from the change's declarations + dataflow neighbourhood (rank.py/mapping.py) so it runs within budget. Increment 0 (confirm CPG-build-vs-analysis cost split) gates the rest. Scale measurement needs the kube lane.
• Line-correspondence (done, necessary-not-sufficient). snapshot.py discarded base↔head anchors on non-globally-unique lines → comparison_unknown → suppressed findings; fixed to run-aware anchoring (commit c39fbb8, 163 tests). Removes one cause; coverage still needs change-scoped analysis + the capability registry (Task 8.3).

G3. Decide BLOCK per family on the pool's false-alarm rate and the pooled precision bound (Req 6.4).

G4. Detection on push for any repository (plane-on-kubernetes P4 work: ousast plane scan --base --head, engine pins on kube-ax, model calls on the VM or CI), sized by what G1-G3 kept.

G5. Qualification and rollout: adopt, commit prediction-v3.json, spend population v3 once, advisory rollout, BLOCK only for families that passed.

Cross-cutting rules

• No hand-tuning per miss. Every detection change is a registered experiment or an admission by held-out separation.
• Records carry counts, digests and pattern names, never repository names: the reserved-repository guard covers every population.
• Instrument first: a zero from an unread input is a failed run (AGENTS.md).
• Cluster stability: sandbox deaths are retried once, then sent to the VM lane; the operator owns workers, egress and Substrate.

Spec map

Spec	Role in this roadmap	State
unseen-repo-evaluation (new)	G0, G1: the pool and the push-level metric	To specify
learned-decision-engine	G1-G3, G5; fix-mechanisms plan is one G2 arm	Implementation; experiments 3-5 recorded
plane-on-kubernetes	G4; the engine executors for kube-ax	Groups 1-2 done (kind); production path re-scoped to the operator's kube-ax (AX Tasks), groups 3-8 to re-plan
pre-push-safety-net	the hook replayed in G0-G1; M4-M5 in G5	Operational increment 1 done; M4-M5 open
ai-service-plane	The plane the above runs on	Complete

Status, 2026-10-02

The milestone table below dates from 2026-09-14. Since then (.kiro/specs/pre-push-safety-net/release-milestones.md): M1a and M1b are met, M2 is measured NO-GO, and M3 is next, with M4 population preparation in parallel; none of M1-M5 is verified and rollout remains NO-GO. Populations v1 and v2 were evaluated and failed every M4 gate (benchmarks/independent/results-v1.json, results-v2.json); v3 (PHP) is frozen for the one-time final check.

Work since then moved to: the ax plane as the only agentic executor (ai-service-plane, HarnessX retired 2026-09-30 by harnessx-removal), and the learned decision engine (learned-decision-engine, task 6 in progress, not adopted). See overview.md for the current-state summary.

Release path as accepted 2026-09-14

Build an actionable pre-push security safety net for AI-accelerated development. The maintainer accepted the following order: prove one useful detection, make it fast, prove cross-language transfer, qualify independently, then enable narrow advisory rollout. The executable outcome contract lives in the existing pre-push-safety-net release milestone spec.

We are at M1b. M1a is met; no other release milestone is verified yet. The published v1.2.0-alpha.1 and 28/29 completed implementation tasks describe delivered machinery, not a demonstrated useful safety net. Task 8.3 and rollout remain NO-GO.

Order	Milestone	Observable exit	Current state
M1a	Show the NodeGoat detection with every admission veto recorded, not applied	Raw finding with witness on the vulnerable revision; analyzed fixed twin and unchanged control quiet; per-finding veto list. One-week time box	MET 2026-09-16
M1b	Repair only the vetoes M1a showed to matter	Supported evaluation finding through production rules, exact change/base evidence and repair direction; fixed twin and unchanged control stay quiet	IN PROGRESS: comparison half verified on all three revisions; the consequence and repair rendering (10.10) remains
M2	Make that same analysis fast	Equivalent evidence; representative warm changed-code p95 <=30s, cancellation/report <=2s, >=95% supported completion	Pending M1
M3	Prove the shared design transfers	PMPro/PHP and VAmPI/Python vulnerable/fixed evidence; unchanged core rules; Node replay after any PMPro core change	Pending M2
M4	Qualify on independent cases	Reviewed untouched vulnerable/fixed/benign populations pass joint quality, coverage and latency gates per capability	Pending M3
M5	Enable a narrow advisory rollout	Exact passing eligibility and packaged acceptance; only qualified capabilities enabled	Pending M4

Immediate next action

Finish M1b by rendering the experimental evaluation finding with its consequence and repair direction (10.10), through an explicitly unreviewed declaration and with eligibility still disabled. The comparison half is verified: the production rules classify the vulnerable, fixed and unchanged NodeGoat revisions correctly, and admission rejects the vulnerable one on the eligibility gate alone. What is missing is the explanation, which no declaration currently supports, and inventing it is exactly what admission exists to prevent. Then M2 runtime. Do not start another broad speed campaign before this outcome exists. See the M1a evidence and the owner diagnosis.

Approach and boundaries

The existing ranker owns scope. Vendor code stays out of targets and graphs. Framework facts and frontends supply evidence to a language-agnostic core. Unknown dependencies remain explicit; removing global completeness vetoes requires question-owned evidence, not dropped safety checks. Advisory and blocking share the same actionability bar.

Graph persistence now works: identical-tip replays take 18.5–18.9s. All 18 other measured transactions still time out and 0/24 target checks complete. A local Joern session census probe is promising, but full-query equivalence and hook performance are unverified. These findings inform M2; they do not complete M1. No second IR/engine, new heuristic scope selector, differential graph mutation or unrelated backlog is part of this path.

Existing Spec Updates

• pre-push-safety-net — Execute the accepted M1–M5 release milestone amendment. Owns immutable replay, comparison, admission, evaluation and reporting. Preserve the existing completed tasks; task 8.3 remains blocked until qualification passes.
• contributor-scan — Supply the required query/operation/context fixes for M1, followed by equivalent bounded session execution for M2 and shared transfer evidence. Frontend retention 2.18 and named census repair 2.19 are already verified.
• flow-aware-ranking — Preserve ranker-owned scope and normalized cross-language behavior. New ranking research is not a prerequisite for the first useful comparison.
• zero-with-a-reason — Adjacent diagnostics work; finishing its whole backlog is not required before M1.

Specs (dependency order)

• pre-push-safety-net — Existing approved implementation, dependent on contributor-scan. Groups 1–7 and tasks 8.1, 8.2, 8.4 are verified; 26/27 executable tasks complete. Remaining release work follows M1–M5. This amendment creates no duplicate implementation spec and resets no prior approvals.

Progress reporting

Lead with the current milestone, the demonstrated developer outcome, the blocker and the next action. Keep implementation counts separate from release readiness. Preserve prior evidence and review limits in the implementation records. The earlier PHP-first and runtime-first narrative below is historical and does not supersede this accepted sequence. Blocking deployment follows a later explicit decision.

Historical roadmap narrative — September 9–11

The sections below preserve prior plans and measurements. Statements about the current bottleneck, PHP coverage, rollout order or feature readiness are superseded by the current section above.

Rewritten 2026-09-09. The previous version (kept as .kiro/archive/steering/roadmap-2026-09-04.md, outside steering) described a pre-Joern world — "adjudication is still Python-ast plus a tree-sitter CLI stub", "Joern-as-required deferred". Joern was adopted, the model layer was built, wired and measured, and two repositories have been scanned. See overview.md for what is true today; this is what happens next and in what order.

The target: a first release for small projects

"a first release that is ready for smaller projects like libpng or wordpress"

Those two are, today, exactly the cases that do not work — and they fail for different reasons, which is what makes them a good pair of targets rather than one.

target	why it fails now	what a release needs
WordPress (php)	taint_specs(language="php") returns zero families. Joern's php2cpg parses it; we say nothing.	Fact tables and entry-point mapping. The engine is right for this: PHP web bugs are the classic taint families, which is what the taint arbiter already decides well.
libpng (c)	c/memory models string functions; libpng overflows through arithmetic, and its library never calls a sink we model.	Either a second abstraction (size arithmetic — a constraint problem) or an honest, stated limit.

So "ready for" has to mean two different things, and the release criteria say which applies where:

• For a project whose bug class a family can decide (WordPress, vibe-code): find the known vulnerability.
• For one it cannot (libpng today): run inside its envelope, report nothing false, and state the limit rather than returning a quiet clean bill of health.

The second is not a consolation. A tool that says "I do not model this class" is usable; one that silently finds nothing is the thing that teaches developers to distrust SAST.

Release criteria (v0.1)

1. Runs unattended on a pinned checkout of each target within a stated time and memory envelope.
2. Zero known-false entailed findings on every pinned checkout.
3. Finds the known vulnerability wherever a family can decide it in principle (Req 4.6).
4. States what each family cannot decide, in the report (Req 5.6).
5. The report is proportionate — no rule repeating itself past a stated count (Req 5.5).
6. Installs and runs without the user having Joern, a JVM, or a PHP interpreter (docker compose + wrapper).
7. A committed regression baseline per target, so the next change is measurable rather than argued about.

What that reorders

PHP moves up: it is the shortest path from here to a target the maintainer named, and it is the case the existing arbiter is best suited to. The C decision moves up with it, because criterion 4 needs c/memory either earning its place or being scoped — and scoping it is cheap.

The one test everything is ordered by

"it must find the bugs, otherwise it's SAST noise. most devs know codeql / semgrep have dataflow capabilities but never use it"

So the ordering principle is: make the output worth reading, then make it find more. A better arbiter behind an unreadable report changes nothing, and a second language before the report is fixed just doubles the noise.

Now — make a repository scan finish at all

The output work this section used to hold is done, and PHP went considerably further than it planned: three CVEs on real WordPress plugins, entailed by the graph with no model in the loop, each dropping on its fixed side. The detector is not the problem any more. Finishing is.

A 637-file plugin builds in 76 seconds and then the taint query does not complete. Everything below is ordered by one question: what is the shortest path to a scan of a real plugin that reports its CVE?

1. 5.14 — stop asking questions that cannot have an answer. 53% of taint requests are put to a family whose sinks do not appear in scope; every one of PMPro's 487 deserialization requests is asked of a repository that never calls unserialize. Arithmetic, not judgement, and it cannot lose a finding. 4.6 hours → 2.2. Do this first because it makes every later measurement cheaper to run.
2. 5.12 — rank, because 84% of regions currently share one rank ordered alphabetically. Two thirds of the budget is spent arbitrarily. Fifty well-chosen regions beat five hundred; combined with (1) that is the difference between hours and minutes. The ranker is an LLM job, and it is safe as one because ranking cannot manufacture a finding -- it decides what is looked at, never what is reported. The loop is closed and that is a design requirement, not a caveat: a region the ranker excludes is never arbitrated, so it never produces a label, so the ranker never learns it was wrong. An exploration slice and a small fully-labelled corpus are what make the signal unbiased.
3. 5.13 — the 6.63 s/request itself, if (1) and (2) are not enough. callDepth=3 over 7,061 methods is the term that makes a request cost seconds; the two-stage joins are NOT the cost (measured: 22%, and they found two of the three CVEs).

Only after a scan finishes is there any point tuning what it finds.

Next — the families that cannot be asked

4. 5.9 — PHP obligation facts. dominance_specs(language="php") is empty, so missing-capability checks are not asked on any WordPress scan. Not a miss; nothing was measured.
5. 5.6 — a sanitizer cleanses the family it cleanses. The list is flat, so esc_sql reads as sufficient for XSS and esc_html as insufficient for SQL. Wrong in both directions, so no output_encoding verdict currently means anything.
6. File the php2cpg defect upstream. A global inside a closure yields a CPG Joern's own dataflow overlay rejects, bisected to four lines. Worked around here (0.2% of files, detected and excluded and reported), but it costs the WHOLE graph rather than the file, and other users will hit it.

Then — close the deferred engine questions

8. 2.14 — control dependence. Absent from both CPGs, so every guard question runs on CFG dominance. Find out whether Joern's CDG pass can be applied from the query script and what it costs. If it cannot, that is a real bound on what this engine can reason about and should be recorded as one.
9. 2.13 — a destination allocated to size. Deliberately blocked on 6, so the rule can be measured for what it silences as well as what it clears.
10. Group 3–4 — ship it. Contributor output, the Docker image, the memory-family decision, retiring evidence_level now that the rung carries the meaning.

Alongside — keep the codebase from growing into the problem

"we should make sure our code base here doesn't explode. for example we have a custom IR."

.kiro/specs/ir-consolidation/ (brief, unapproved) separates two questions with opposite answers:

• The source IR is already carried, not about to be added. semantic/ is 3,218 lines — 16% of the codebase — holding a hand-written IR and flat taint engine measured at 2.2% entailment against the CPG's 41.3%, still running as a proposer. Vine and Ghidra P-Code are binary IRs and do not help here; the work is subtraction. A candidate ~1,000-line deletion with a number attached.
• The missing abstraction is C arithmetic, and there an existing IR is exactly right. GhidraPAL's three-valued-logic abstract interpreter over P-Code decides "can this index exceed this allocation", which is the class c/memory cannot see and every libpng overflow is. Vine is dormant since 2009 and is not the candidate.

The ordering constraint is the point: question 1 lands before question 2 starts, or adopting an IR makes the growth problem worse rather than better.

After — the feedback loop

11. finding-feedback-loop (design generated, awaiting approval). Its ledger and dismissals are useful immediately; its evolution should not be trusted until step 5 or 6 lands a second true-positive checkout. The boundary is fixed: an optimiser may evolve the question, never the facts.

Not now, and why

• authorization-obligations (24 open tasks, approved) — the dominance arbiter delivered its substance by another route. It should be reconciled or retired rather than worked, and carrying it as "approved, not started" misrepresents both.
• reachability-flow-model (23 open, nothing approved) — largely delivered by adopting Joern. Same treatment.
• constrained-detector (4 of 20, tasks unapproved) — parked.
• More corpus volume. 198 of 238 OWASP pairs were left unvendored deliberately: volume without readability makes every number worse, and the corpus is not the binding constraint. Reachability is.
• A second CPG engine, a server mode, an execution rung. Server mode was measured and declined (6% of a scan). The other two have no evidence calling for them yet.

How to know it is working

The scoreboard in overview.md, and one question: can a developer run this on a repository they did not write and get something they act on? Today that is true for one Python application and false for one C library, and the specs above are ordered by what changes that.

A release is ready when that question is answered yes for WordPress and honestly for libpng — found for the one whose class we decide, and a stated limit for the one we do not. Both are release criteria; only one of them is a detection claim.