marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(steering): an overview of what is true, and a roadmap ordered by one test

Browse Source 
Fourteen specs have accumulated and the roadmap still described a
pre-Joern world -- "adjudication is still Python-ast plus a tree-sitter CLI
stub", "Joern-as-required deferred" -- which is stale enough to mislead
someone reading it to decide what to do next. It is kept as
roadmap-2026-09-04.md rather than deleted.

**overview.md** is the synthesis above the specs: what the tool is, how one
pass works, what has been measured, and what is not true yet. Every number
cites a committed artifact. The thesis is stated plainly -- the
differentiator is not the dataflow engine, since CodeQL and Semgrep have
one and developers do not turn it on, but the rung: a claim you can act on
without writing a query, and an explicit admission when the engine could
not decide.

It says what does not work as clearly as what does. The C story is
structurally broken, not undertuned. One language is demonstrated. PHP,
Ruby and Go have zero families, so Joern would parse them and the engine
would say nothing. The evidence base is two repositories and one has never
produced a true positive. Control dependence is unavailable in both CPGs.
execution_confirmed has no production caller.

And it records the lesson that shaped the current specs: nine defects found
by pointing the tool at two real repositories, three of them the same bug
in three matchers, with the three gates byte-identical through every fix --
so a corpus of function-level pairs could not have caught any of them.

**roadmap.md** is ordered by the standing test, that the tool must find
bugs or it is SAST noise. Which gives the ordering principle: make the
output worth reading, then make it find more. A better arbiter behind an
unreadable report changes nothing, and a second language before the report
is fixed doubles the noise.

It also says what is NOT being worked and why -- authorization-obligations
and reachability-flow-model were largely delivered by other routes and
should be reconciled or retired rather than carried as approved-not-started,
which misrepresents both.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-09 13:50:46 +02:00
co-authored by Claude Opus 5
parent 290ecff9de
commit 9a65c998b4
3 changed files with 279 additions and 108 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/steering/overview.md
+97
View File
@@ -0,0 +1,97 @@
# Overview — what this is, and what is actually true of it
**Written 2026-09-09.** Fourteen specs have accumulated and the roadmap still described a pre-Joern world, so
this is the synthesis above them: what the system does, what has been measured, and what is not true yet.
Every number here comes from a committed artifact in `benchmarks/measurements/`.
## What the tool is
A static analyser that separates **what it can prove from what it merely suspects**, and says which is which.
The thesis is not the dataflow engine — CodeQL and Semgrep have one and developers do not turn it on. It is
the **rung**: a claim you can act on without writing a query, and an explicit admission when the engine could
not decide.
```
execution_confirmed a test or sandbox reproduced it (built, no production caller)
model_entailed the graph decided: a flow with no discharge,
an obligation with no guard, a permissive value ← the product
model_corroborated the graph found the shape; a judge with
coverage agreed on the residual question
suspicion an enumerator or a pattern proposed it and
nothing corroborated it ← everything below the line is noise
until proven otherwise
```
Silence is not a clean bill of health. If no fact table covers a site, the model is **silent, not negative** —
23 of the 39 misses in the original entailment ceiling were sinks simply absent from the tables.
## How it works, in one pass
```
preprocess ─▶ entry-point map ─▶ MAP stage ─┬─ static patterns
├─ semantic overlay
├─ obligations (absence bugs)
├─ tool hunter (LLM)
└─ MODEL LAYER ──▶ one CPG per repository (Joern, subprocess)
├─ taint reachability + sanitizer lattice
├─ dominance guard governs obligated operation
└─ config constant abstraction over a value
```
Three abstractions, one arbiter each, dispatched on spec type so a family without an arbiter cannot be
silently routed to the wrong one. The CPG is built **once per scan** and every question is batched into one
invocation per query kind.
## What has been measured
| | measured | source |
|---|---|---|
| entailment, flat IR → CPG | 2.2% → **41.3%** of the corpus | `2026-09-08-model-entailment-ceiling*.json` |
| vibe-py pairs (30) | **90.0% pair-correct**, 6.7% leak, $0.0068 | `2026-09-08-pipeline-vibe-py.json` |
| — against the LLM hunter | 76.7% / 12.3% at 5× the calls, ~100× the cost | same |
| vampi (python, 520 lines) | **3 of 3** known vulnerabilities at `model_entailed` | `2026-09-09-repo-vampi-authz-precision.json` |
| vampi access-control precision | 28.6% → **50.0%** | same |
| access-control leak, pairs nobody tuned against | 42.9% → **0.0%**, pair-correct unchanged | same |
| libpng envelope (89k lines of C) | build 16s, query 250s, scan 446s, **911 MB** | `2026-09-09-repo-libpng-envelope.json` |
| libpng valid findings | **zero** | same |
| report proportionality | libpng 198 sections → **39** | `2026-09-09-report-proportionality.json` |
## What is not true yet
- **The C story does not work.** `c/memory` has produced zero true positives and three distinct classes of
false one. It models string functions; libpng overflows through arithmetic, and its library code never
calls a sink we model. This is structural, not tuning.
- **One language is demonstrated.** Python. JavaScript has fact tables and pair coverage but no repository.
PHP, Ruby and Go have **zero families** — Joern would parse them and the engine would say nothing.
- **The suspicion band is still too loud.** vampi emits 61 report sections for a 520-line application, 38 of
them a candidate the graph could not decide with a judge's opinion attached.
- **The evidence base is two repositories.** One has never produced a true positive.
- **Control dependence is unavailable.** 0 of 13,184 calls in the libpng CPG and 0 of 1,152 in vampi's have a
controlling control structure, so every guard question runs on CFG dominance instead.
- **`execution_confirmed` has no production caller.** The rung exists and nothing reaches it.
## The lesson that shaped the current specs
Nine engine defects were found in one session by pointing the tool at two real repositories. **Three were the
same bug in three matchers** — a spec token matched as a substring: `resolve` in `resolveUrl`, `user` in
`users`, `gets` in `fgets`. Two were a capability built and unreachable: the model layer itself, then the
query batcher. One was a paid budget stopping unpaid work.
The three gates stayed **byte-identical through every one of those fixes**. That is the finding: a corpus of
function-level pairs could not see any of them, and a repository showed all nine within minutes.
## Where the specs stand
| spec | phase | state |
|---|---|---|
| `contributor-scan` | implementation | 14 done, 15 open — **the active one** |
| `finding-feedback-loop` | design | requirements approved, design awaiting approval |
| `model-grounded-detection` | complete | 16/16; built the arbiter this all rests on |
| `authorization-obligations` | tasks approved | 24 open — superseded in practice by the dominance arbiter |
| `constrained-detector` | tasks | 4 of 20, tasks unapproved |
| `reachability-flow-model` | tasks generated | 23 open, nothing approved — largely delivered by adopting Joern |
| `learning-harness` | implementation | 30 done, 3 open |
| six others | complete | `pair-corpus-honesty`, `corpus-seeded-mechanisms`, `propose-adjudicate-prove`, `three-stage-scan`, `real-world-vfc-slice`, `harnessx-self-improving-rulesets`, `tree-sitter-overlay-extra` |
97 source modules, 90 load-bearing and 7 standalone capabilities; 814 tests.
.kiro/steering/roadmap-2026-09-04.md
+126
View File
@@ -0,0 +1,126 @@
# Roadmap
## Overview
OpenUltraSAST can propose sinks with regex and prove a few of them in a sandbox, but adjudication is still Python-`ast` plus a tree-sitter CLI stub. Textbook SAST pairs (OWASP/Juliet) did not improve after the overlay: Youden went from **+9.09% inventory to 0% overlay** on the same 11-pair slice. Real operator trees and browser/C/C++ history are not in the catalog. The next work is to put a real tree-sitter extra under the existing overlay, then grow an offline function-level vuln-vs-fix slice from public VFC datasets (including Firefox/gecko-dev where commits exist) without vendoring whole megarepos.
## Direction (2026-09-04)
Detection logic is ~16% of the source; harness plumbing, governance, and ranking are the rest. Recall on real code comes from the model with tools plus proof; precision and silence come from static checks. Shift: grow the corpus past the license and review walls (pointer slices, review tiers), turn every reviewed pair into a searchable mechanism (Clearwing's variant loop), then build the path model (call graph, summaries, fix points), put the hunter on paths, let model answers become gated facts, and stop adding harness plumbing until the detector exists. Recall first under a false-positive ceiling, per profile. Youden on non-local slices stays off the merge gate.
## Approach Decision
- **Chosen**: Optional tree-sitter extra first, then a harvested VFC slice scored like `github` (honesty dashboard, not smoke gate). Isolated functions only. Firefox is one project filter among Chromium, OpenSSL, curl — not a special-case full mozilla-central clone.
- **Why**: The overlay already has dispositions, facts, and taint. JS/C/Java stay `language_unsupported` because we refused a parser library. Textbook Youden is the wrong north star; large-project VFCs are the right one. Adopting tree-sitter deletes the need to hand-write language parsers.
- **Rejected alternatives**:
- Whole Firefox tree in CI — multi-GB, memory-safety heavy, security-bug commit messages are deliberately scrubbed.
- FixFox dataset — embargoed until 2030.
- Joern-as-required or inter-file taint in this phase — deferred by `propose-adjudicate-prove`.
- Chasing OWASP Youden with more regex — leaks on goodG2B/false files.
## Scope
- **In**: tree-sitter + grammars as `openultrasast[semantic]`; CST walker into existing FileIR/taint; harvested function-level pairs from CVEfixes/MoreFixes/CWE-Bench-Java/SVEN with provenance; overlay scoring on that slice; keep `quick` zero-dep.
- **Out**: Cloning mozilla-central/Chromium into the repo; making overlay Youden a merge gate; buffer-size/CWE-121 semantics; inter-file taint; auto-facts from improve; LLM as parser.
## Measured 2026-09-04 (extra on, before pair-corpus-honesty)
| Slice | Vuln recall | Silent on fix | Pair-correct | Youden |
|-------|-------------|---------------|--------------|--------|
| sast (11) | 45.5% | 81.8% | 4/11 | +27.3% |
| vfc (176) | 3.4% | 82.4% | 0/176 | −14.2% |
Overlay column equals inventory on vfc (no C memory sink fact). 25 curl pairs parse_failed (harvest anchored on a comment mention).
## Measured 2026-09-04 (after pair-corpus-honesty scorer: coverage counted, function-scoped matching)
| Slice | Vuln recall | Silent on fix | Pair-correct | Youden | Achievable | Loss |
|-------|-------------|---------------|--------------|--------|------------|------|
| sast (11) | 54.5% | 72.7% | 4/11 | +27.3% | 4/10 (java-hash known_limit) | 4 unadjudicated |
| vfc (176) | 0.0% | 81.8% | 0/176 | −18.2% | 0/176 | 50 parse_failed files, 128 unadjudicated |
New slices (pair-corpus-honesty, 2026-09-05, after the debug round: shared declarator naming, containment matching, sink-location labels): vibe-py (35, human, Real-Vuln licensed repos) 13/35 pair-correct, Youden +37.1% overlay vs 7/35 inventory; vfc-js (17) 5/17, Youden +23.5%; agent-vfc (29, agent, tier `title`) 0/29, Youden 0.0: no excerpt carries a rule-known sink and the overlay has no TypeScript grammar (18 rows); the unreviewed-tier baseline, never a gate. vfc parse-failed files 50 → 36 after the harvest fix.
Leave-one-out (corpus-seeded-mechanisms, 2026-09-05; the corpus as a teacher, never a gate). Each pair is held out once, a store is seeded from the other `seeded`/`reviewed` pairs, both excerpts are searched for structural variants:
| slice | pairs | detected | silent | Youden | teaching pairs | note |
|---|---|---|---|---|---|---|
| vibe-py (vendored) | 35 | 11 | 21 | −0.086 | 16 | `execute/1 parameter` shape taught by 7 pairs; 14 leaks on trap twins |
| local | 3 | 0 | 3 | 0.000 | 0 | reviewed, but labels name no sink call site |
| sast | 10 | 0 | 10 | 0.000 | 0 | advisory tier never teaches (known_limit pair skipped) |
| github | 6 | 0 | 6 | 0.000 | 0 | advisory tier |
| vfc-js | 17 | 0 | 17 | 0.000 | 0 | advisory tier |
The loop is closed: `ousast mechanisms export` writes candidates, `ousast improve --pair-catalog` admits the ones that recover a holdout pair without leaking (per-profile clause over variant search; `mechanism_leak`/`mechanism_no_gain` revert byte for byte), the scan searches the admitted store and reports name the mechanism and its known fix. Measured 2026-09-05 on the `python-vulnerable` benchmark target with the vibe-py candidates: 4 records admitted in round 1 (`execute/1 parameter` taught by 7 pairs among them), superseded by the learning-harness section below: that number was measured with holdout pairs teaching (see `benchmarks/measurements/2026-09-06-mechanism-lever-split.json`).
Measured 2026-09-05 after review tiers (Req 9) and pointer slices (Req 10): vibe-py 115 = 35 vendored human (13/35, Youden +37.1%) + 80 `agent` pointer pairs over the 40 LLM-generated Real-Vuln repositories (0/80, Youden 0.0; the LLM-repo baseline, scored by the nightly `pairs --slice vibe-py --pointers` only, never in tests or gates); agent-vfc 59 = 29 vendored + 30 pointer, tier `title`, 1/59, Youden 0.0, 2 unresolved labels; sast, vfc-js, vfc unchanged; detection/map/local-pair gate outputs byte-identical to the pre-spec commit. Only `seeded`/`reviewed` pairs gate the improve loop. Carry-forwards: `language_unsupported` loss counter and multi-line C/C++ declarator rescue (overlay-ir-completeness), `profiles_ungated` reporting and an offline test for the agent-vfc pointer builder (corpus-seeded-mechanisms).
sast: java-sqli moved MISS→LEAK (coverage counted on both sides; the fixed twin leaks via parameter-as-source). vfc: recall fell to zero because every label now names a function and no C detection lands inside it; that is the honest baseline the engine specs measure against.
## Constraints
- Core `dependencies = []`. Parser wheels live only in an extra.
- Pair fixtures stay out of `LANGUAGE_MANIFESTS`.
- Do not vendor 16 GB VFC dumps. Pointers in `datasets.toml`; a small reviewed catalog of isolated functions is what CI runs.
- DiverseVul-style auto labels (~60% accurate) are not ground truth without review.
## Boundary Strategy
- **Why this split**: Engine (how we parse) is independent of corpus (what we score). The extra can ship without any new pairs; the corpus can score inventory-only until the extra lands, then overlay.
- **Shared seams to watch**: `FileIR` / `parse_file`; pair catalog `slice` values; overlay vs inventory scoring in `pairs.py`.
## Existing Spec Updates
- [x] propose-adjudicate-prove -- overlay roles, facts, prove-budget, sast overlay scoring (landed). Residual: tree-sitter path returns no IR.
- [ ] propose-adjudicate-prove -- do not reopen for parsers or corpora; those are the new specs below.
## Direct Implementation Candidates
- [x] Record inventory vs overlay Youden side-by-side in `ousast pairs --slice sast` JSON -- folded into pair-corpus-honesty task 2.3. Done (`scorers` block, 2026-09-05).
- [x] Add Firefox/gecko-dev, Chromium, OpenSSL pointers to `benchmarks/pairs/datasets.toml`.
## Specs (dependency order)
- [x] tree-sitter-overlay-extra -- Optional extra: tree-sitter + grammars, one CST walker, `quick` stays zero-dep. Dependencies: none
- [x] real-world-vfc-slice -- Harvest reviewed function-level vuln/fix pairs from large projects (Firefox/gecko-dev, Chromium, OpenSSL, …) as an honesty slice. Seed is testable (`ousast pairs --slice vfc`). Dependencies: none. **Closed 2026-09-04, superseded by pair-corpus-honesty on the corpus/scorer plane.**
- [x] pair-corpus-honesty -- Scorer counts coverage and matches by function; labels carry mechanism/provenance/split/known_limit; harvest anchors at the declarator with line-range and hunk modes; sast hygiene; new slices `vibe-py` (Real-Vuln-Benchmark), `vfc-js` (SecBench.js fix commits), `agent-vfc` (agent-trailer fixes); deterministic provenance fingerprint; per-profile improve acceptance; review tiers and pointer slices (amendment). Dependencies: none (supersedes real-world-vfc-slice). **Completed 2026-09-05 (11 review rounds).**
- [x] corpus-seeded-mechanisms -- Every seeded or reviewed pair becomes a mechanism record (source shape, sink shape, guard the fix added); structural variant search over the scanned tree proposes matches as `variant` findings that the overlay and sandbox verify; leave-one-out recall on the pair corpus is the corpus's own detection rate; the improve loop gains a recall-raising `mechanisms` lever under the per-profile holdout gate. Principle: recall first under a false-positive ceiling, per profile. Dependencies: pair-corpus-honesty (tiers, pointer slices). **Completed 2026-09-05 (6 review rounds).**
- [ ] reachability-flow-model -- Name-resolved call graph over `FunctionIR`; per-function taint summaries composed along the graph into path records; graph reachability drives rank, map, worth-fixing; hunter gets `path_context` and hunter findings with a locatable sink become sandbox candidates; closed-kind fact oracle with candidate cache; `facts` improve lever under the per-profile gate; fix points ranked by reachable risk removed. Dependencies: pair-corpus-honesty **Critical path: two consumers wait on it (guard-dominance-regime and authorization-obligations).**
- [ ] overlay-ir-completeness -- Shared CST walker completeness (C `init_declarator`, lvalue paths for field stores, per-grammar literal kinds, local ERROR-node tolerance, multi-line C/C++ declarators the walker names `<anon>` and `functions.py`'s one-line rescue cannot name: `chromium-cve-2022-2162`, `openssl-cve-2025-69421`; a Scorer-side multi-line-aware rescue is the cheaper alternative) plus Python parity with stdlib ast (f-string constness, augmented assignment, kwargs); injection-regime facts: container/transformer taint, `weak_literals` consulted, parameter sources off for injection. Measured on sast and vibe-py. Dependencies: pair-corpus-honesty
- [ ] authorization-obligations -- Absence bugs as obligations, not flows: obligated operations (protected read/write, privileged action, security setting) and discharger kinds (path guard, identity constraint from authenticated context, ownership check, non-permissive value) as closed facts; obligation shapes learned from trusted pairs; the application's policy recovered from sibling-handler consistency and an optional declared, versioned policy file; violations evaluated on the path (function-local degraded mode without it) with `consistency_violation` / `declared_policy_violation` rungs below proof; `obligations` improve lever under the holdout gate; the hunter adjudicates intent only. Measured leave-one-out on the vibe-py and agent-vfc absence pairs (today 0 detected; 15 of 29 agent-vfc rows). Dependencies: reachability-flow-model, guard-dominance-regime, corpus-seeded-mechanisms. **Requirements drafted 2026-09-05, unapproved.**
- [ ] guard-dominance-regime -- Guard and Use IR records; guard dominance by nesting or early exit; memory sinks (memcpy/malloc/deref) with operand roles, nullable callees, out-param transformers, fixed-array capacity; same regime applied to web absence bugs (route auth guard, identity from auth context). Consumes path records. Measured on vfc, vfc-js, agent-vfc. Dependencies: reachability-flow-model, overlay-ir-completeness
<!-- learning-harness:begin -->
## Classified detector numbers
Generated by `uv run ousast learning publish` from the artifacts committed beside these numbers.
| model | slice | family | scorable | unscorable | recall | silence | Youden | at fixed FPR | noise floor | cost per correct pair |
|---|---|---|---|---|---|---|---|---|---|---|
| deepseek-v4-flash | agent-vfc | access_control | 4 (taxonomy 1) | known_limit:client_side_caller: 1, unresolved_label: 1 | 0.000 | 1.000 | 0.000 | 0.000 | budget 4 | - |
| deepseek-v4-flash | agent-vfc | config_secrets | 6 (taxonomy 1) | none | 0.333 | 0.833 | 0.167 | 0.000 | budget 2 | 0.95 |
| deepseek-v4-flash | agent-vfc | injection | 2 (taxonomy 1) | none | 0.500 | 0.500 | 0.000 | 0.000 | budget 1 | - |
| deepseek-v4-flash | agent-vfc | path | 3 (taxonomy 1) | none | 1.000 | 0.667 | 0.667 | 0.000 | budget 3 | 0.48 |
| deepseek-v4-flash | agent-vfc | prototype | 7 (taxonomy 1) | none | 0.429 | 1.000 | 0.429 | 0.429 | budget 5 | 0.32 |
| deepseek-v4-flash | agent-vfc | untrusted_destination | 5 (taxonomy 1) | none | 0.600 | 0.600 | 0.200 | 0.000 | budget 3 | 0.95 |
| deepseek-v4-flash | vibe-py | access_control | 3 (taxonomy 1) | identical_twin: 1 | 1.000 | 1.000 | 1.000 | 1.000 | budget 3 | 0.23 |
| deepseek-v4-flash | vibe-py | config_secrets | 3 (taxonomy 1) | none | 1.000 | 1.000 | 1.000 | 1.000 | budget 1 | 0.23 |
| deepseek-v4-flash | vibe-py | deserialization | 2 (taxonomy 1) | identical_twin: 1 | 1.000 | 1.000 | 1.000 | 1.000 | budget 0 | 0.34 |
| deepseek-v4-flash | vibe-py | injection | 20 (taxonomy 1) | identical_twin: 2 | 0.750 | 0.900 | 0.650 | 0.750 | budget 11 | 0.05 |
| deepseek-v4-flash | vibe-py | output_encoding | 1 (taxonomy 1) | identical_twin: 1 | 1.000 | 1.000 | 1.000 | 1.000 | budget 1 | 0.69 |
| deepseek-v4-flash | vibe-py | path | 1 (taxonomy 1) | none | 1.000 | 1.000 | 1.000 | 1.000 | budget 1 | 0.69 |
| deepseek-v4-pro | vibe-py | access_control | 3 (taxonomy 1) | identical_twin: 1 | 0.000 | 1.000 | 0.000 | 0.000 | budget 2 | - |
| deepseek-v4-pro | vibe-py | config_secrets | 3 (taxonomy 1) | none | 1.000 | 1.000 | 1.000 | 1.000 | budget 2 | 0.16 |
| deepseek-v4-pro | vibe-py | deserialization | 2 (taxonomy 1) | identical_twin: 1 | 1.000 | 1.000 | 1.000 | 1.000 | budget 1 | 0.24 |
| deepseek-v4-pro | vibe-py | injection | 20 (taxonomy 1) | identical_twin: 2 | 0.350 | 0.900 | 0.250 | 0.350 | budget 14 | 0.09 |
| deepseek-v4-pro | vibe-py | output_encoding | 1 (taxonomy 1) | identical_twin: 1 | 1.000 | 1.000 | 1.000 | 1.000 | budget 0 | 0.47 |
| deepseek-v4-pro | vibe-py | path | 1 (taxonomy 1) | none | 1.000 | 1.000 | 1.000 | 1.000 | budget 0 | 0.47 |
No family in this taxonomy declares a parent, so hierarchical partial credit could not apply to any figure above.
No detector cutoff is configured, so no post-cutoff slice is reported; every figure above is over the whole corpus.
Mechanism lever on the vibe-py holdout: detected 5 of 17 at Youden -0.0588 once only train-split pairs teach, against detected 7 at Youden 0.0588 when every pair taught. The difference is what holdout pairs stopped teaching.
<!-- learning-harness:end -->
.kiro/steering/roadmap.md
+56 -108
View File
@@ -1,126 +1,74 @@
# Roadmap
## Overview
**Rewritten 2026-09-09.** The previous version (kept as `roadmap-2026-09-04.md`) described a pre-Joern world —
"adjudication is still Python-`ast` plus a tree-sitter CLI stub", "Joern-as-required deferred". Joern was
adopted, the model layer was built, wired and measured, and two repositories have been scanned. See
`overview.md` for what is true today; this is what happens next and in what order.
OpenUltraSAST can propose sinks with regex and prove a few of them in a sandbox, but adjudication is still Python-`ast` plus a tree-sitter CLI stub. Textbook SAST pairs (OWASP/Juliet) did not improve after the overlay: Youden went from **+9.09% inventory to 0% overlay** on the same 11-pair slice. Real operator trees and browser/C/C++ history are not in the catalog. The next work is to put a real tree-sitter extra under the existing overlay, then grow an offline function-level vuln-vs-fix slice from public VFC datasets (including Firefox/gecko-dev where commits exist) without vendoring whole megarepos.
## The one test everything is ordered by
## Direction (2026-09-04)
> "it must find the bugs, otherwise it's SAST noise. most devs know codeql / semgrep have dataflow
> capabilities but never use it"
Detection logic is ~16% of the source; harness plumbing, governance, and ranking are the rest. Recall on real code comes from the model with tools plus proof; precision and silence come from static checks. Shift: grow the corpus past the license and review walls (pointer slices, review tiers), turn every reviewed pair into a searchable mechanism (Clearwing's variant loop), then build the path model (call graph, summaries, fix points), put the hunter on paths, let model answers become gated facts, and stop adding harness plumbing until the detector exists. Recall first under a false-positive ceiling, per profile. Youden on non-local slices stays off the merge gate.
So the ordering principle is: **make the output worth reading, then make it find more.** A better arbiter
behind an unreadable report changes nothing, and a second language before the report is fixed just doubles the
noise.
## Approach Decision
## Now — make the output worth reading
- **Chosen**: Optional tree-sitter extra first, then a harvested VFC slice scored like `github` (honesty dashboard, not smoke gate). Isolated functions only. Firefox is one project filter among Chromium, OpenSSL, curl — not a special-case full mozilla-central clone.
- **Why**: The overlay already has dispositions, facts, and taint. JS/C/Java stay `language_unsupported` because we refused a parser library. Textbook Youden is the wrong north star; large-project VFCs are the right one. Adopting tree-sitter deletes the need to hand-write language parsers.
- **Rejected alternatives**:
- Whole Firefox tree in CI — multi-GB, memory-safety heavy, security-bug commit messages are deliberately scrubbed.
- FixFox dataset — embargoed until 2030.
- Joern-as-required or inter-file taint in this phase — deferred by `propose-adjudicate-prove`.
- Chasing OWASP Youden with more regex — leaks on goodG2B/false files.
Everything here is in `contributor-scan`, group 2, and all of it is measurable on the two checkouts that exist.
## Scope
1. **2.17 — the model's own suspicion band.** vampi emits 61 sections for 520 lines; 38 are a candidate the
graph could not decide with a judge's opinion attached. Expected: 61 → ~23. Measure what it hides.
2. **2.9 — one defect is one finding.** 26 identical rows for one site. Dedupe on the site, keep the strongest
rung, record `reached_from`.
3. **2.16 — say what a family cannot decide.** Req 5.6. This is the anti-quiet-failure fix, and it matters
most for the teaching goal: an LLM handed a silent C scan will fluently explain why the code is fine.
4. **2.8 — a declared-public endpoint carries no obligation.** Removes both remaining vampi false positives.
Needs the access level's *provenance*, because for decorator frameworks "public" means "no decorator
found", which is the bug itself.
- **In**: tree-sitter + grammars as `openultrasast[semantic]`; CST walker into existing FileIR/taint; harvested function-level pairs from CVEfixes/MoreFixes/CWE-Bench-Java/SVEN with provenance; overlay scoring on that slice; keep `quick` zero-dep.
- **Out**: Cloning mozilla-central/Chromium into the repo; making overlay Youden a merge gate; buffer-size/CWE-121 semantics; inter-file taint; auto-facts from improve; LLM as parser.
## Next — make it find more, in the cheapest order
## Measured 2026-09-04 (extra on, before pair-corpus-honesty)
5. **5.2 — a pinned vibe-code repository.** The cheapest route to a *second* checkout that can produce a true
positive, because Python and JavaScript are the two languages whose fact tables are complete. Everything
downstream is tuned on two repositories until this lands.
6. **2.15 — a C checkout whose CVE goes through a sink we model.** Decides whether `c/memory` earns its place
or gets scoped (2.16). libpng cannot answer this; no version of it can.
7. **5.1 — PHP fact tables, then WordPress.** `php2cpg` exists; our PHP families are empty. Tables first,
corpus second — the other order produces a confident silence.
| Slice | Vuln recall | Silent on fix | Pair-correct | Youden |
|-------|-------------|---------------|--------------|--------|
| sast (11) | 45.5% | 81.8% | 4/11 | +27.3% |
| vfc (176) | 3.4% | 82.4% | 0/176 | −14.2% |
## Then — close the deferred engine questions
Overlay column equals inventory on vfc (no C memory sink fact). 25 curl pairs parse_failed (harvest anchored on a comment mention).
8. **2.14 — control dependence.** Absent from both CPGs, so every guard question runs on CFG dominance. Find
out whether Joern's CDG pass can be applied from the query script and what it costs. If it cannot, that is
a real bound on what this engine can reason about and should be recorded as one.
9. **2.13 — a destination allocated to size.** Deliberately blocked on 6, so the rule can be measured for what
it *silences* as well as what it clears.
10. **Group 3–4 — ship it.** Contributor output, the Docker image, the memory-family decision, retiring
`evidence_level` now that the rung carries the meaning.
## Measured 2026-09-04 (after pair-corpus-honesty scorer: coverage counted, function-scoped matching)
## After — the feedback loop
| Slice | Vuln recall | Silent on fix | Pair-correct | Youden | Achievable | Loss |
|-------|-------------|---------------|--------------|--------|------------|------|
| sast (11) | 54.5% | 72.7% | 4/11 | +27.3% | 4/10 (java-hash known_limit) | 4 unadjudicated |
| vfc (176) | 0.0% | 81.8% | 0/176 | −18.2% | 0/176 | 50 parse_failed files, 128 unadjudicated |
11. **`finding-feedback-loop`** (design generated, awaiting approval). Its ledger and dismissals are useful
immediately; its *evolution* should not be trusted until step 5 or 6 lands a second true-positive
checkout. The boundary is fixed: an optimiser may evolve the **question**, never the **facts**.
New slices (pair-corpus-honesty, 2026-09-05, after the debug round: shared declarator naming, containment matching, sink-location labels): vibe-py (35, human, Real-Vuln licensed repos) 13/35 pair-correct, Youden +37.1% overlay vs 7/35 inventory; vfc-js (17) 5/17, Youden +23.5%; agent-vfc (29, agent, tier `title`) 0/29, Youden 0.0: no excerpt carries a rule-known sink and the overlay has no TypeScript grammar (18 rows); the unreviewed-tier baseline, never a gate. vfc parse-failed files 50 → 36 after the harvest fix.
## Not now, and why
Leave-one-out (corpus-seeded-mechanisms, 2026-09-05; the corpus as a teacher, never a gate). Each pair is held out once, a store is seeded from the other `seeded`/`reviewed` pairs, both excerpts are searched for structural variants:
- **`authorization-obligations`** (24 open tasks, approved) — the dominance arbiter delivered its substance by
another route. It should be reconciled or retired rather than worked, and carrying it as "approved, not
started" misrepresents both.
- **`reachability-flow-model`** (23 open, nothing approved) — largely delivered by adopting Joern. Same
treatment.
- **`constrained-detector`** (4 of 20, tasks unapproved) — parked.
- **More corpus volume.** 198 of 238 OWASP pairs were left unvendored deliberately: volume without
readability makes every number worse, and the corpus is not the binding constraint. Reachability is.
- **A second CPG engine, a server mode, an execution rung.** Server mode was measured and declined (6% of a
scan). The other two have no evidence calling for them yet.
| slice | pairs | detected | silent | Youden | teaching pairs | note |
|---|---|---|---|---|---|---|
| vibe-py (vendored) | 35 | 11 | 21 | −0.086 | 16 | `execute/1 parameter` shape taught by 7 pairs; 14 leaks on trap twins |
| local | 3 | 0 | 3 | 0.000 | 0 | reviewed, but labels name no sink call site |
| sast | 10 | 0 | 10 | 0.000 | 0 | advisory tier never teaches (known_limit pair skipped) |
| github | 6 | 0 | 6 | 0.000 | 0 | advisory tier |
| vfc-js | 17 | 0 | 17 | 0.000 | 0 | advisory tier |
## How to know it is working
The loop is closed: `ousast mechanisms export` writes candidates, `ousast improve --pair-catalog` admits the ones that recover a holdout pair without leaking (per-profile clause over variant search; `mechanism_leak`/`mechanism_no_gain` revert byte for byte), the scan searches the admitted store and reports name the mechanism and its known fix. Measured 2026-09-05 on the `python-vulnerable` benchmark target with the vibe-py candidates: 4 records admitted in round 1 (`execute/1 parameter` taught by 7 pairs among them), superseded by the learning-harness section below: that number was measured with holdout pairs teaching (see `benchmarks/measurements/2026-09-06-mechanism-lever-split.json`).
Measured 2026-09-05 after review tiers (Req 9) and pointer slices (Req 10): vibe-py 115 = 35 vendored human (13/35, Youden +37.1%) + 80 `agent` pointer pairs over the 40 LLM-generated Real-Vuln repositories (0/80, Youden 0.0; the LLM-repo baseline, scored by the nightly `pairs --slice vibe-py --pointers` only, never in tests or gates); agent-vfc 59 = 29 vendored + 30 pointer, tier `title`, 1/59, Youden 0.0, 2 unresolved labels; sast, vfc-js, vfc unchanged; detection/map/local-pair gate outputs byte-identical to the pre-spec commit. Only `seeded`/`reviewed` pairs gate the improve loop. Carry-forwards: `language_unsupported` loss counter and multi-line C/C++ declarator rescue (overlay-ir-completeness), `profiles_ungated` reporting and an offline test for the agent-vfc pointer builder (corpus-seeded-mechanisms).
sast: java-sqli moved MISS→LEAK (coverage counted on both sides; the fixed twin leaks via parameter-as-source). vfc: recall fell to zero because every label now names a function and no C detection lands inside it; that is the honest baseline the engine specs measure against.
## Constraints
- Core `dependencies = []`. Parser wheels live only in an extra.
- Pair fixtures stay out of `LANGUAGE_MANIFESTS`.
- Do not vendor 16 GB VFC dumps. Pointers in `datasets.toml`; a small reviewed catalog of isolated functions is what CI runs.
- DiverseVul-style auto labels (~60% accurate) are not ground truth without review.
## Boundary Strategy
- **Why this split**: Engine (how we parse) is independent of corpus (what we score). The extra can ship without any new pairs; the corpus can score inventory-only until the extra lands, then overlay.
- **Shared seams to watch**: `FileIR` / `parse_file`; pair catalog `slice` values; overlay vs inventory scoring in `pairs.py`.
## Existing Spec Updates
- [x] propose-adjudicate-prove -- overlay roles, facts, prove-budget, sast overlay scoring (landed). Residual: tree-sitter path returns no IR.
- [ ] propose-adjudicate-prove -- do not reopen for parsers or corpora; those are the new specs below.
## Direct Implementation Candidates
- [x] Record inventory vs overlay Youden side-by-side in `ousast pairs --slice sast` JSON -- folded into pair-corpus-honesty task 2.3. Done (`scorers` block, 2026-09-05).
- [x] Add Firefox/gecko-dev, Chromium, OpenSSL pointers to `benchmarks/pairs/datasets.toml`.
## Specs (dependency order)
- [x] tree-sitter-overlay-extra -- Optional extra: tree-sitter + grammars, one CST walker, `quick` stays zero-dep. Dependencies: none
- [x] real-world-vfc-slice -- Harvest reviewed function-level vuln/fix pairs from large projects (Firefox/gecko-dev, Chromium, OpenSSL, …) as an honesty slice. Seed is testable (`ousast pairs --slice vfc`). Dependencies: none. **Closed 2026-09-04, superseded by pair-corpus-honesty on the corpus/scorer plane.**
- [x] pair-corpus-honesty -- Scorer counts coverage and matches by function; labels carry mechanism/provenance/split/known_limit; harvest anchors at the declarator with line-range and hunk modes; sast hygiene; new slices `vibe-py` (Real-Vuln-Benchmark), `vfc-js` (SecBench.js fix commits), `agent-vfc` (agent-trailer fixes); deterministic provenance fingerprint; per-profile improve acceptance; review tiers and pointer slices (amendment). Dependencies: none (supersedes real-world-vfc-slice). **Completed 2026-09-05 (11 review rounds).**
- [x] corpus-seeded-mechanisms -- Every seeded or reviewed pair becomes a mechanism record (source shape, sink shape, guard the fix added); structural variant search over the scanned tree proposes matches as `variant` findings that the overlay and sandbox verify; leave-one-out recall on the pair corpus is the corpus's own detection rate; the improve loop gains a recall-raising `mechanisms` lever under the per-profile holdout gate. Principle: recall first under a false-positive ceiling, per profile. Dependencies: pair-corpus-honesty (tiers, pointer slices). **Completed 2026-09-05 (6 review rounds).**
- [ ] reachability-flow-model -- Name-resolved call graph over `FunctionIR`; per-function taint summaries composed along the graph into path records; graph reachability drives rank, map, worth-fixing; hunter gets `path_context` and hunter findings with a locatable sink become sandbox candidates; closed-kind fact oracle with candidate cache; `facts` improve lever under the per-profile gate; fix points ranked by reachable risk removed. Dependencies: pair-corpus-honesty **Critical path: two consumers wait on it (guard-dominance-regime and authorization-obligations).**
- [ ] overlay-ir-completeness -- Shared CST walker completeness (C `init_declarator`, lvalue paths for field stores, per-grammar literal kinds, local ERROR-node tolerance, multi-line C/C++ declarators the walker names `<anon>` and `functions.py`'s one-line rescue cannot name: `chromium-cve-2022-2162`, `openssl-cve-2025-69421`; a Scorer-side multi-line-aware rescue is the cheaper alternative) plus Python parity with stdlib ast (f-string constness, augmented assignment, kwargs); injection-regime facts: container/transformer taint, `weak_literals` consulted, parameter sources off for injection. Measured on sast and vibe-py. Dependencies: pair-corpus-honesty
- [ ] authorization-obligations -- Absence bugs as obligations, not flows: obligated operations (protected read/write, privileged action, security setting) and discharger kinds (path guard, identity constraint from authenticated context, ownership check, non-permissive value) as closed facts; obligation shapes learned from trusted pairs; the application's policy recovered from sibling-handler consistency and an optional declared, versioned policy file; violations evaluated on the path (function-local degraded mode without it) with `consistency_violation` / `declared_policy_violation` rungs below proof; `obligations` improve lever under the holdout gate; the hunter adjudicates intent only. Measured leave-one-out on the vibe-py and agent-vfc absence pairs (today 0 detected; 15 of 29 agent-vfc rows). Dependencies: reachability-flow-model, guard-dominance-regime, corpus-seeded-mechanisms. **Requirements drafted 2026-09-05, unapproved.**
- [ ] guard-dominance-regime -- Guard and Use IR records; guard dominance by nesting or early exit; memory sinks (memcpy/malloc/deref) with operand roles, nullable callees, out-param transformers, fixed-array capacity; same regime applied to web absence bugs (route auth guard, identity from auth context). Consumes path records. Measured on vfc, vfc-js, agent-vfc. Dependencies: reachability-flow-model, overlay-ir-completeness
<!-- learning-harness:begin -->
## Classified detector numbers
Generated by `uv run ousast learning publish` from the artifacts committed beside these numbers.
| model | slice | family | scorable | unscorable | recall | silence | Youden | at fixed FPR | noise floor | cost per correct pair |
|---|---|---|---|---|---|---|---|---|---|---|
| deepseek-v4-flash | agent-vfc | access_control | 4 (taxonomy 1) | known_limit:client_side_caller: 1, unresolved_label: 1 | 0.000 | 1.000 | 0.000 | 0.000 | budget 4 | - |
| deepseek-v4-flash | agent-vfc | config_secrets | 6 (taxonomy 1) | none | 0.333 | 0.833 | 0.167 | 0.000 | budget 2 | 0.95 |
| deepseek-v4-flash | agent-vfc | injection | 2 (taxonomy 1) | none | 0.500 | 0.500 | 0.000 | 0.000 | budget 1 | - |
| deepseek-v4-flash | agent-vfc | path | 3 (taxonomy 1) | none | 1.000 | 0.667 | 0.667 | 0.000 | budget 3 | 0.48 |
| deepseek-v4-flash | agent-vfc | prototype | 7 (taxonomy 1) | none | 0.429 | 1.000 | 0.429 | 0.429 | budget 5 | 0.32 |
| deepseek-v4-flash | agent-vfc | untrusted_destination | 5 (taxonomy 1) | none | 0.600 | 0.600 | 0.200 | 0.000 | budget 3 | 0.95 |
| deepseek-v4-flash | vibe-py | access_control | 3 (taxonomy 1) | identical_twin: 1 | 1.000 | 1.000 | 1.000 | 1.000 | budget 3 | 0.23 |
| deepseek-v4-flash | vibe-py | config_secrets | 3 (taxonomy 1) | none | 1.000 | 1.000 | 1.000 | 1.000 | budget 1 | 0.23 |
| deepseek-v4-flash | vibe-py | deserialization | 2 (taxonomy 1) | identical_twin: 1 | 1.000 | 1.000 | 1.000 | 1.000 | budget 0 | 0.34 |
| deepseek-v4-flash | vibe-py | injection | 20 (taxonomy 1) | identical_twin: 2 | 0.750 | 0.900 | 0.650 | 0.750 | budget 11 | 0.05 |
| deepseek-v4-flash | vibe-py | output_encoding | 1 (taxonomy 1) | identical_twin: 1 | 1.000 | 1.000 | 1.000 | 1.000 | budget 1 | 0.69 |
| deepseek-v4-flash | vibe-py | path | 1 (taxonomy 1) | none | 1.000 | 1.000 | 1.000 | 1.000 | budget 1 | 0.69 |
| deepseek-v4-pro | vibe-py | access_control | 3 (taxonomy 1) | identical_twin: 1 | 0.000 | 1.000 | 0.000 | 0.000 | budget 2 | - |
| deepseek-v4-pro | vibe-py | config_secrets | 3 (taxonomy 1) | none | 1.000 | 1.000 | 1.000 | 1.000 | budget 2 | 0.16 |
| deepseek-v4-pro | vibe-py | deserialization | 2 (taxonomy 1) | identical_twin: 1 | 1.000 | 1.000 | 1.000 | 1.000 | budget 1 | 0.24 |
| deepseek-v4-pro | vibe-py | injection | 20 (taxonomy 1) | identical_twin: 2 | 0.350 | 0.900 | 0.250 | 0.350 | budget 14 | 0.09 |
| deepseek-v4-pro | vibe-py | output_encoding | 1 (taxonomy 1) | identical_twin: 1 | 1.000 | 1.000 | 1.000 | 1.000 | budget 0 | 0.47 |
| deepseek-v4-pro | vibe-py | path | 1 (taxonomy 1) | none | 1.000 | 1.000 | 1.000 | 1.000 | budget 0 | 0.47 |
No family in this taxonomy declares a parent, so hierarchical partial credit could not apply to any figure above.
No detector cutoff is configured, so no post-cutoff slice is reported; every figure above is over the whole corpus.
Mechanism lever on the vibe-py holdout: detected 5 of 17 at Youden -0.0588 once only train-split pairs teach, against detected 7 at Youden 0.0588 when every pair taught. The difference is what holdout pairs stopped teaching.
<!-- learning-harness:end -->
The scoreboard in `overview.md`, and one question: **can a developer run this on a repository they did not
write and get something they act on?** Today that is true for one Python application and false for one C
library, and the specs above are ordered by what changes that.