Open_Ultra_SAST/.kiro/steering/overview.md

# Overview — what this is, and what is actually true of it

## Current state, 2026-10-13

Everything below this section is dated history; where it disagrees with this section, this section wins.

- **Goal, unchanged.** A pre-push safety net that works on repositories it has never seen, judged by recall on
  real vulnerability changes and false alarms per 100 ordinary pushes, measured on a valid benchmark.
- **The valid benchmark exists and ran** (`valid-detection-benchmark`, `benchmarks/valid/`, pilot frozen at
  `benchmarks/valid/pilot/`; numbers in `benchmarks/measurements/2026-10-12-valid-benchmark-pilot/NOTES.md`).
  12 verified fix-site positives, 240 ordinary pushes, 12 unseen app repos. Push hook as shipped: 0 of 12,
  0.7 false alarms per 100. LLM hunter: 5 to 6 of 12, 54 false alarms per 100 after the classifier. Neither
  meets the gate. The SZZ pool (`pool-p1`) and its spec are retired; no number from it is quoted.
- **What the numbers say.** The hook has no LLM detection tier and catches nothing on unseen code. The hunter
  finds about half but searches whole files, so it re-reports old code on every push (250 of 343 "vulnerable"
  suspicions on ordinary pushes sit more than 10 lines from the diff). Thresholds on that output are not a fix.
- **Active work, in order.** (1) `hunter-learning-loop`: an annotation model for the hunter (diff hunks,
  enclosing function and callers, entry points, roles, framework facts, closed families) chosen by statistical
  ablation under the learn harness; requirements written 2026-10-13, design next. (2) `pre-push-safety-net`:
  the LLM tier on pre-push once the search is good. (3) `search-with-proof` / `proof-arm-opencode`: proof on
  what survives the search, as the block gate. (4) `engine-change-scoped-analysis`: the deterministic facts
  that feed the hunter's annotations.
- **Everything else is history or parked.** Retired spec directories were removed from the tree on 2026-10-13
  (`git show <commit>:.kiro/specs/<name>/` still has them): `harnessx-removal` (51d064e),
  `harnessx-self-improving-rulesets` and `openrouter-sast-harness` (c8d2fef), `learning-harness`,
  `constrained-detector`, `three-stage-scan`, `real-world-vfc-slice` (39a1873), `unseen-repo-evaluation`
  (abb2058). The learn harness code (`src/openultrasast/learn/`) stays; `hunter-learning-loop` builds on it.
- **Rules that stay in force.** Plain English in every document, commit and comment. Standard metrics only.
  Prove the instrument before believing a zero. Train on one set, report on another. Research runs locally;
  deployed behaviour is observed, not re-measured by hand.

## Current state, 2026-10-04 (superseded by the section above)

Everything below this section is dated history; where it disagrees with this section, this section wins. The
direction lives in `roadmap.md`, section of 2026-10-04 (phases G0-G5).

- **Goal.** A pre-push safety net that works on repositories it has never seen. Two numbers per push decide it:
  catch rate on real vulnerability-introducing changes, and false-alarm rate on ordinary changes. Neither has been
  measured yet; the unseen-repository pool (G0, spec `unseen-repo-evaluation`) comes first.
- **Correction, 2026-10-04.** Infrastructure and engine work had drifted from that goal. It is now done only when
  it removes a bottleneck for those measurements.
- **Engine.** Joern runs on the operator's kube-ax cluster as AX Tasks, with a VM fallback lane; kind stays the
  development profile. It is almost blind to fixes: paths differ between sides in 2 of 105 both-asked pairs.
- **Decision engine.** Ranks but cannot block (exp-003 to exp-005 inconclusive; 710 fix pairs in memory). Fix
  mechanisms is one G2 arm, paused after mining until the pool exists.
- **Hook.** `ousast pre-push` is operational and advisory on any repository; rollout stays NO-GO.

## State, 2026-10-02 (superseded by the section above)

User-facing documentation: `README.md`, `docs/architecture.md`, `docs/evaluation.md`, `docs/decision-engine.md`,
`ops/ax/README.md`.

- **Scanning.** `ousast scan --mode quick|standard|deep`. Quick rules for C/C++, Python, JavaScript/TypeScript,
  Java/Groovy and PHP (PHP numbers are in-sample only: `benchmarks/measurements/2026-09-30-php-quick-rules/`).
  Standard adds the MAP stage with the Joern model layer (shipped in the Docker image); deep adds sandboxed
  regression snippets when Docker is available. Framework knowledge is tagged as optional priors
  (`ruleset/frameworks.toml`, `priors=`). `ousast pre-push` compares head against base; the capability registry
  is empty, rollout NO-GO.
- **Agentic plane.** google/ax over Agent Substrate on kind is the only executor of model work over whole
  repositories (`ousast plane run|status|doctor|workspaces|remember|alerts-engine|harvest`). Credentials travel
  only in the start request through `atenet-router`; egress is per task; budgets are per task; rows go to the
  memory store (`OUSAST_MEMORY`: FileStore, or S3Store on an S3-compatible bucket; RustFS is the tested server).
- **Removed.** HarnessX (retired 2026-09-30): its config keys fail, a `[harnessx]` section warns. The LLM judge
  of the verifier and the LLM fusion panels are gone; fusion is deterministic.
- **Improvement.** `ousast improve --memory` proposes rule-status edits from plane memory through the unchanged
  validator and gate; the loop also runs as a plane Run.
- **Decision engine** (`learned-decision-engine`, implementation, not adopted). First slice, injection only:
  ADVISORY recall 0.90, precision 0.54, BLOCK not offered
  (`benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`).
- **Evaluation.** Populations v1 and v2 are spent and failed every M4 gate (`benchmarks/independent/results-v1.json`,
  `results-v2.json`); v3 (PHP) is frozen and untouched for the one-time final check. The plane increment met its
  gate on the second measurement with a tie-break chosen after the first (`benchmarks/independent/plane-increment-2.json`).
- **Providers.** DeepSeek for LLM calls, OpenRouter only for OpenAI embeddings; keys in `.env`, which never
  overrides an exported shell variable.

## History

**Direction, 2026-09-12:** an actionable pre-push security safety net for AI-accelerated
development across supported ecosystems. See `safety-net.md` and the approved `pre-push-safety-net`
spec. The ranker remains the scope mechanism; third-party vendor code stays outside targets and
graphs. Advisory must meet the same actionability bar as blocking.

**Historical scoreboard below:** it was written September 9. Later commits record a completing
whole-PMPro scan and Joern 4.0.625 ranking measurements (82/91/54 on the three plugins), while
default installation pins are now aligned to 4.0.625 with verified PHP/JavaScript image smoke.
The pre-push workflow, changed-code latency and actionable
precision are not demonstrated by those results. In particular, the manually selected two-file
slice below is a useful experiment, not evidence that an automatic push scope would select the
same context. Current scope/status corrections live in `flow-aware-ranking/brief.md` and the roadmap.

The new scoring instrument retains missing targets and unanswered queries. Historical rank positions
do not establish detection recall; VAmPI's old budget of 16 omitted its unmatched SQLI target. The
corrected positional run keeps that target unresolved. Labeled PMPro/Node inputs and a frozen
diagnostic profile are committed; independent capability evaluation is still required.

**Written 2026-09-09.** Fourteen specs have accumulated and the roadmap still described a pre-Joern world, so
this is the synthesis above them: what the system does, what has been measured, and what is not true yet.
Every number here comes from a committed artifact in `benchmarks/measurements/`.

## What the tool is

A static analyser that separates **what it can prove from what it merely suspects**, and says which is which.

The thesis is not the dataflow engine — CodeQL and Semgrep have one and developers do not turn it on. It is
the **rung**: a claim you can act on without writing a query, and an explicit admission when the engine could
not decide.

```
execution_confirmed   a test or sandbox reproduced it                (built, no production caller)
model_entailed        the graph decided: a flow with no discharge,
                      an obligation with no guard, a permissive value  ← the product
model_corroborated    the graph found the shape; a judge with
                      coverage agreed on the residual question
suspicion             an enumerator or a pattern proposed it and
                      nothing corroborated it                          ← everything below the line is noise
                                                                          until proven otherwise
```

Silence is not a clean bill of health. If no fact table covers a site, the model is **silent, not negative** —
23 of the 39 misses in the original entailment ceiling were sinks simply absent from the tables.

## How it works, in one pass

```
preprocess ─▶ entry-point map ─▶ MAP stage ─┬─ static patterns
                                            ├─ semantic overlay
                                            ├─ obligations (absence bugs)
                                            ├─ tool hunter (LLM)
                                            └─ MODEL LAYER ──▶ one CPG per repository (Joern, subprocess)
                                                               ├─ taint      reachability + sanitizer lattice
                                                               ├─ dominance  guard governs obligated operation
                                                               └─ config     constant abstraction over a value
```

Three abstractions, one arbiter each, dispatched on spec type so a family without an arbiter cannot be
silently routed to the wrong one. The CPG is built **once per scan** and every question is batched into one
invocation per query kind.

## What has been measured

| | measured | source |
|---|---|---|
| entailment, flat IR → CPG | 2.2% → **41.3%** of the corpus | `2026-09-08-model-entailment-ceiling*.json` |
| vibe-py pairs (30) | **90.0% pair-correct**, 6.7% leak, $0.0068 | `2026-09-08-pipeline-vibe-py.json` |
| — against the LLM hunter | 76.7% / 12.3% at 5× the calls, ~100× the cost | same |
| vampi (python, 520 lines) | **3 of 3** known vulnerabilities at `model_entailed` | `2026-09-09-repo-vampi-authz-precision.json` |
| vampi access-control precision | 28.6% → **50.0%** | same |
| access-control leak, pairs nobody tuned against | 42.9% → **0.0%**, pair-correct unchanged | same |
| libpng envelope (89k lines of C) | build 16s, query 250s, scan 446s, **911 MB** | `2026-09-09-repo-libpng-envelope.json` |
| libpng valid findings | **zero** | same |
| report proportionality | libpng 198 sections → **39** | `2026-09-09-report-proportionality.json` |
| **CVE-2023-23488, production plugin code** | **found at `class.memberorder.php:936`**, the exact line | `2026-09-09-pmpro-cve-found.json` |
| — precision on that file | 16 of 26 unprepared `$wpdb` queries; **0 of 26 use `->prepare`** | same |

## What is not true yet

- **The C story does not work.** `c/memory` has produced zero true positives and three distinct classes of
  false one. It models string functions; libpng overflows through arithmetic, and its library code never
  calls a sink we model. This is structural, not tuning.
- **Two languages are demonstrated.** Python end to end, and PHP on a real plugin slice where it found a
  real CVE. Ruby and Go still have **zero families** — Joern parses them and the engine says nothing.
- **Not at repository scale for PHP.** The 637-file checkout does not load; the 2-file slice does. That
  boundary is unmeasured (task 5.10).
- **The suspicion band is still too loud.** vampi emits 61 report sections for a 520-line application, 38 of
  them a candidate the graph could not decide with a judge's opinion attached.
- **The evidence base is two repositories.** One has never produced a true positive.
- **Control dependence is unavailable.** 0 of 13,184 calls in the libpng CPG and 0 of 1,152 in vampi's have a
  controlling control structure, so every guard question runs on CFG dominance instead.
- **`execution_confirmed` has no production caller.** The rung exists and nothing reaches it.

## What scoping is for, which is not what it looks like

The first real CVE was found by **cutting**, not by scanning more. Two of Paid Memberships Pro's 637 files —
the ones holding both ends of the bug — load in 9 seconds, query in 111, and report CVE-2023-23488 at the
exact line. The full checkout does not load at all.

That two-file slice **is the contributor case**. A developer editing `class.memberorder.php` gets precisely
that scan. So scoping is not a workaround for a scale problem; it is the thesis, and the scale problem is
what happens when the thesis is ignored.

The distinction that matters is not whether to cut but **who decides and whether the decision is recorded**:

| | decided by | recoverable afterwards |
| --- | --- | --- |
| **scoping** — a build declaration, a diff, a subtree | the project or the developer | yes, and it is in the artifact |
| **triage** — "this looks like boilerplate" | a model | no; if it drops the file with the bug, the scan reports clean |

Both give the same answer when the model is right. Only one of them tells you when it was wrong, and every
failure found on this checkout was of exactly that shape.

## Why a corpus of benchmarks is not enough on its own

This is the finding worth teaching, and it is not an opinion — it is what the gates say.

**Twelve engine defects were found in one session by pointing the tool at three real repositories. The three
gates stayed byte-identical through every single fix.** A gate that does not move is a corpus that could not
see the bug.

| defect | what the corpus saw |
|---|---|
| `resolve` matched inside `resolveUrl` (sanitizers) | nothing |
| `user` matched inside `users` (dischargers) | nothing |
| `gets` matched inside `fgets` (sinks) — 26 identical false positives | nothing |
| the model layer had zero production callers | nothing |
| `query_batch` implemented, tested, unreachable | nothing |
| a paid budget halting unpaid arbitration | nothing |
| a region with no function claiming the whole repository's answer | nothing |
| a guard discharging an obligation it is not a guard of | nothing |
| `joern-parse` failing where the frontend succeeds | nothing |
| a 1.49MB batch over Linux's 128KB argv cap | nothing |
| an array callable not recognised as a handler | nothing |
| one defect reported as twenty-six findings | nothing |

The reason is structural rather than a gap in the corpus's coverage. **A pair is a labelled function with a
declared family and a known answer.** It cannot exercise: which region to look at, what to spend, whether the
capability is even reachable, whether two entry points share a sink, whether the question fits in an argv
slot, or whether a flow crosses a module boundary. Every one of those is a property of a *repository*, and
none of them is a property of a function.

Three of the twelve are worse than invisible — they are **quiet failures**, where the tool produced a clean
report and every honesty counter agreed with it. The rung ladder labels a *verdict*; when no verdict is
produced there is nothing to label. That is the failure mode a benchmark score cannot express at all, because
the score is computed from the verdicts that exist.

So the corpus is necessary and not sufficient: it is the **holdout** that stops per-repository tuning, and it
is the wrong instrument for finding out whether the tool works. Both instruments, or neither number means
anything.

## The lesson that shaped the current specs

Nine engine defects were found in one session by pointing the tool at two real repositories. **Three were the
same bug in three matchers** — a spec token matched as a substring: `resolve` in `resolveUrl`, `user` in
`users`, `gets` in `fgets`. Two were a capability built and unreachable: the model layer itself, then the
query batcher. One was a paid budget stopping unpaid work.

The three gates stayed **byte-identical through every one of those fixes**. That is the finding: a corpus of
function-level pairs could not see any of them, and a repository showed all nine within minutes.

## Where the specs stand

Counted 2026-10-02 from each spec's `tasks.md` checkboxes and `spec.json`.

| spec | phase | state |
|---|---|---|
| `learned-decision-engine` | implementation | 11 of 17 tasks done |
| `pre-push-safety-net` | implementation | 67 of 79 tasks done |
| `contributor-scan` | paused | 27 of 51 tasks done; paused since 2026-09-23 (`v1.2.0-alpha.1` NO-GO), waits on pre-push M2/M3 |
| `ai-service-plane` | complete | 11 of 11; the ax plane is the agentic path |
| `harnessx-removal` | complete, directory removed 2026-10-13 | 8 of 8; HarnessX removed 2026-09-30 |
| `finding-feedback-loop` | dormant | requirements approved, design unapproved since 2026-09-09; depends on `contributor-scan` |
| `model-grounded-detection` | complete | 16 of 16; built the arbiter this all rests on |
| `authorization-obligations` | dormant | tasks approved, 0 of 24 started; superseded in practice by the dominance arbiter |
| `reachability-flow-model` | dormant | 23 tasks generated, nothing approved; largely delivered by adopting Joern |
| `flow-aware-ranking`, `zero-with-a-reason`, `ir-consolidation` | dormant | brief only, unapproved |
| four others | complete | `pair-corpus-honesty`, `corpus-seeded-mechanisms`, `propose-adjudicate-prove`, `tree-sitter-overlay-extra` |
| six others | retired or superseded, directories removed 2026-10-13 | `openrouter-sast-harness`, `three-stage-scan`, `harnessx-self-improving-rulesets`, `learning-harness` (retired with HarnessX), `constrained-detector`, `real-world-vfc-slice` (superseded) |

> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax (`ai-service-plane`),
> and evolutionary proposals come from the plane memory. The HarnessX specs in the table above are history.

151 source modules, 146 load-bearing and 5 standalone capabilities, 0 orphaned
(`benchmarks/measurements/2026-09-08-module-audit.json`, regenerated 2026-10-02); 2,023 tests collected.