Open_Ultra_SAST/.kiro/steering/overview.md

# Overview — what this is, and what is actually true of it

## Current state, 2026-10-02

Everything below this section is dated history; where it disagrees with this section, this section wins.
User-facing documentation: `README.md`, `docs/architecture.md`, `docs/evaluation.md`, `docs/decision-engine.md`,
`ops/ax/README.md`.

- **Scanning.** `ousast scan --mode quick|standard|deep`. Quick rules for C/C++, Python, JavaScript/TypeScript,
  Java/Groovy and PHP (PHP numbers are in-sample only: `benchmarks/measurements/2026-09-30-php-quick-rules/`).
  Standard adds the MAP stage with the Joern model layer (shipped in the Docker image); deep adds sandboxed
  regression snippets when Docker is available. Framework knowledge is tagged as optional priors
  (`ruleset/frameworks.toml`, `priors=`). `ousast pre-push` compares head against base; the capability registry
  is empty, rollout NO-GO.
- **Agentic plane.** google/ax over Agent Substrate on kind is the only executor of model work over whole
  repositories (`ousast plane run|status|doctor|workspaces|remember|alerts-engine|harvest`). Credentials travel
  only in the start request through `atenet-router`; egress is per task; budgets are per task; rows go to the
  memory store (`OUSAST_MEMORY`: FileStore, or S3Store on an S3-compatible bucket; RustFS is the tested server).
- **Removed.** HarnessX (retired 2026-09-30): its config keys fail, a `[harnessx]` section warns. The LLM judge
  of the verifier and the LLM fusion panels are gone; fusion is deterministic.
- **Improvement.** `ousast improve --memory` proposes rule-status edits from plane memory through the unchanged
  validator and gate; the loop also runs as a plane Run.
- **Decision engine** (`learned-decision-engine`, implementation, not adopted). First slice, injection only:
  ADVISORY recall 0.90, precision 0.54, BLOCK not offered
  (`benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`).
- **Evaluation.** Populations v1 and v2 are spent and failed every M4 gate (`benchmarks/independent/results-v1.json`,
  `results-v2.json`); v3 (PHP) is frozen and untouched for the one-time final check. The plane increment met its
  gate on the second measurement with a tie-break chosen after the first (`benchmarks/independent/plane-increment-2.json`).
- **Providers.** DeepSeek for LLM calls, OpenRouter only for OpenAI embeddings; keys in `.env`, which never
  overrides an exported shell variable.

## History

**Direction, 2026-09-12:** an actionable pre-push security safety net for AI-accelerated
development across supported ecosystems. See `safety-net.md` and the approved `pre-push-safety-net`
spec. The ranker remains the scope mechanism; third-party vendor code stays outside targets and
graphs. Advisory must meet the same actionability bar as blocking.

**Historical scoreboard below:** it was written September 9. Later commits record a completing
whole-PMPro scan and Joern 4.0.625 ranking measurements (82/91/54 on the three plugins), while
default installation pins are now aligned to 4.0.625 with verified PHP/JavaScript image smoke.
The pre-push workflow, changed-code latency and actionable
precision are not demonstrated by those results. In particular, the manually selected two-file
slice below is a useful experiment, not evidence that an automatic push scope would select the
same context. Current scope/status corrections live in `flow-aware-ranking/brief.md` and the roadmap.

The new scoring instrument retains missing targets and unanswered queries. Historical rank positions
do not establish detection recall; VAmPI's old budget of 16 omitted its unmatched SQLI target. The
corrected positional run keeps that target unresolved. Labeled PMPro/Node inputs and a frozen
diagnostic profile are committed; independent capability evaluation is still required.

**Written 2026-09-09.** Fourteen specs have accumulated and the roadmap still described a pre-Joern world, so
this is the synthesis above them: what the system does, what has been measured, and what is not true yet.
Every number here comes from a committed artifact in `benchmarks/measurements/`.

## What the tool is

A static analyser that separates **what it can prove from what it merely suspects**, and says which is which.

The thesis is not the dataflow engine — CodeQL and Semgrep have one and developers do not turn it on. It is
the **rung**: a claim you can act on without writing a query, and an explicit admission when the engine could
not decide.

```
execution_confirmed   a test or sandbox reproduced it                (built, no production caller)
model_entailed        the graph decided: a flow with no discharge,
                      an obligation with no guard, a permissive value  ← the product
model_corroborated    the graph found the shape; a judge with
                      coverage agreed on the residual question
suspicion             an enumerator or a pattern proposed it and
                      nothing corroborated it                          ← everything below the line is noise
                                                                          until proven otherwise
```

Silence is not a clean bill of health. If no fact table covers a site, the model is **silent, not negative** —
23 of the 39 misses in the original entailment ceiling were sinks simply absent from the tables.

## How it works, in one pass

```
preprocess ─▶ entry-point map ─▶ MAP stage ─┬─ static patterns
                                            ├─ semantic overlay
                                            ├─ obligations (absence bugs)
                                            ├─ tool hunter (LLM)
                                            └─ MODEL LAYER ──▶ one CPG per repository (Joern, subprocess)
                                                               ├─ taint      reachability + sanitizer lattice
                                                               ├─ dominance  guard governs obligated operation
                                                               └─ config     constant abstraction over a value
```

Three abstractions, one arbiter each, dispatched on spec type so a family without an arbiter cannot be
silently routed to the wrong one. The CPG is built **once per scan** and every question is batched into one
invocation per query kind.

## What has been measured

| | measured | source |
|---|---|---|
| entailment, flat IR → CPG | 2.2% → **41.3%** of the corpus | `2026-09-08-model-entailment-ceiling*.json` |
| vibe-py pairs (30) | **90.0% pair-correct**, 6.7% leak, $0.0068 | `2026-09-08-pipeline-vibe-py.json` |
| — against the LLM hunter | 76.7% / 12.3% at 5× the calls, ~100× the cost | same |
| vampi (python, 520 lines) | **3 of 3** known vulnerabilities at `model_entailed` | `2026-09-09-repo-vampi-authz-precision.json` |
| vampi access-control precision | 28.6% → **50.0%** | same |
| access-control leak, pairs nobody tuned against | 42.9% → **0.0%**, pair-correct unchanged | same |
| libpng envelope (89k lines of C) | build 16s, query 250s, scan 446s, **911 MB** | `2026-09-09-repo-libpng-envelope.json` |
| libpng valid findings | **zero** | same |
| report proportionality | libpng 198 sections → **39** | `2026-09-09-report-proportionality.json` |
| **CVE-2023-23488, production plugin code** | **found at `class.memberorder.php:936`**, the exact line | `2026-09-09-pmpro-cve-found.json` |
| — precision on that file | 16 of 26 unprepared `$wpdb` queries; **0 of 26 use `->prepare`** | same |

## What is not true yet

- **The C story does not work.** `c/memory` has produced zero true positives and three distinct classes of
  false one. It models string functions; libpng overflows through arithmetic, and its library code never
  calls a sink we model. This is structural, not tuning.
- **Two languages are demonstrated.** Python end to end, and PHP on a real plugin slice where it found a
  real CVE. Ruby and Go still have **zero families** — Joern parses them and the engine says nothing.
- **Not at repository scale for PHP.** The 637-file checkout does not load; the 2-file slice does. That
  boundary is unmeasured (task 5.10).
- **The suspicion band is still too loud.** vampi emits 61 report sections for a 520-line application, 38 of
  them a candidate the graph could not decide with a judge's opinion attached.
- **The evidence base is two repositories.** One has never produced a true positive.
- **Control dependence is unavailable.** 0 of 13,184 calls in the libpng CPG and 0 of 1,152 in vampi's have a
  controlling control structure, so every guard question runs on CFG dominance instead.
- **`execution_confirmed` has no production caller.** The rung exists and nothing reaches it.

## What scoping is for, which is not what it looks like

The first real CVE was found by **cutting**, not by scanning more. Two of Paid Memberships Pro's 637 files —
the ones holding both ends of the bug — load in 9 seconds, query in 111, and report CVE-2023-23488 at the
exact line. The full checkout does not load at all.

That two-file slice **is the contributor case**. A developer editing `class.memberorder.php` gets precisely
that scan. So scoping is not a workaround for a scale problem; it is the thesis, and the scale problem is
what happens when the thesis is ignored.

The distinction that matters is not whether to cut but **who decides and whether the decision is recorded**:

| | decided by | recoverable afterwards |
| --- | --- | --- |
| **scoping** — a build declaration, a diff, a subtree | the project or the developer | yes, and it is in the artifact |
| **triage** — "this looks like boilerplate" | a model | no; if it drops the file with the bug, the scan reports clean |

Both give the same answer when the model is right. Only one of them tells you when it was wrong, and every
failure found on this checkout was of exactly that shape.

## Why a corpus of benchmarks is not enough on its own

This is the finding worth teaching, and it is not an opinion — it is what the gates say.

**Twelve engine defects were found in one session by pointing the tool at three real repositories. The three
gates stayed byte-identical through every single fix.** A gate that does not move is a corpus that could not
see the bug.

| defect | what the corpus saw |
|---|---|
| `resolve` matched inside `resolveUrl` (sanitizers) | nothing |
| `user` matched inside `users` (dischargers) | nothing |
| `gets` matched inside `fgets` (sinks) — 26 identical false positives | nothing |
| the model layer had zero production callers | nothing |
| `query_batch` implemented, tested, unreachable | nothing |
| a paid budget halting unpaid arbitration | nothing |
| a region with no function claiming the whole repository's answer | nothing |
| a guard discharging an obligation it is not a guard of | nothing |
| `joern-parse` failing where the frontend succeeds | nothing |
| a 1.49MB batch over Linux's 128KB argv cap | nothing |
| an array callable not recognised as a handler | nothing |
| one defect reported as twenty-six findings | nothing |

The reason is structural rather than a gap in the corpus's coverage. **A pair is a labelled function with a
declared family and a known answer.** It cannot exercise: which region to look at, what to spend, whether the
capability is even reachable, whether two entry points share a sink, whether the question fits in an argv
slot, or whether a flow crosses a module boundary. Every one of those is a property of a *repository*, and
none of them is a property of a function.

Three of the twelve are worse than invisible — they are **quiet failures**, where the tool produced a clean
report and every honesty counter agreed with it. The rung ladder labels a *verdict*; when no verdict is
produced there is nothing to label. That is the failure mode a benchmark score cannot express at all, because
the score is computed from the verdicts that exist.

So the corpus is necessary and not sufficient: it is the **holdout** that stops per-repository tuning, and it
is the wrong instrument for finding out whether the tool works. Both instruments, or neither number means
anything.

## The lesson that shaped the current specs

Nine engine defects were found in one session by pointing the tool at two real repositories. **Three were the
same bug in three matchers** — a spec token matched as a substring: `resolve` in `resolveUrl`, `user` in
`users`, `gets` in `fgets`. Two were a capability built and unreachable: the model layer itself, then the
query batcher. One was a paid budget stopping unpaid work.

The three gates stayed **byte-identical through every one of those fixes**. That is the finding: a corpus of
function-level pairs could not see any of them, and a repository showed all nine within minutes.

## Where the specs stand

Counted 2026-10-02 from each spec's `tasks.md` checkboxes and `spec.json`.

| spec | phase | state |
|---|---|---|
| `learned-decision-engine` | implementation | 11 of 17 tasks done |
| `pre-push-safety-net` | implementation | 67 of 79 tasks done |
| `contributor-scan` | implementation | 27 of 51 tasks done |
| `ai-service-plane` | implementation | 11 of 11 tasks done; the ax plane is the agentic path |
| `harnessx-removal` | complete | 8 of 8; HarnessX removed 2026-09-30 |
| `finding-feedback-loop` | design | requirements approved, design awaiting approval |
| `model-grounded-detection` | complete | 16 of 16; built the arbiter this all rests on |
| `authorization-obligations` | dormant | tasks approved, 0 of 24 started; superseded in practice by the dominance arbiter |
| `reachability-flow-model` | dormant | 23 tasks generated, nothing approved; largely delivered by adopting Joern |
| `flow-aware-ranking`, `zero-with-a-reason`, `ir-consolidation` | dormant | brief only, unapproved |
| four others | complete | `pair-corpus-honesty`, `corpus-seeded-mechanisms`, `propose-adjudicate-prove`, `tree-sitter-overlay-extra` |
| six others | retired or superseded | `openrouter-sast-harness`, `three-stage-scan`, `harnessx-self-improving-rulesets`, `learning-harness` (retired with HarnessX), `constrained-detector`, `real-world-vfc-slice` (superseded) |

> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax (`ai-service-plane`),
> and evolutionary proposals come from the plane memory. The HarnessX specs in the table above are history.

As of 2026-09-09: 97 source modules, 90 load-bearing and 7 standalone capabilities; 814 tests.
151 source modules, 146 load-bearing and 5 standalone capabilities, 0 orphaned
(`benchmarks/measurements/2026-09-08-module-audit.json`, regenerated 2026-10-02); 2,009 tests collected.