Marius CiepluchandClaude Opus 5
9a65c998b4
Fourteen specs have accumulated and the roadmap still described a pre-Joern world -- "adjudication is still Python-ast plus a tree-sitter CLI stub", "Joern-as-required deferred" -- which is stale enough to mislead someone reading it to decide what to do next. It is kept as roadmap-2026-09-04.md rather than deleted. **overview.md** is the synthesis above the specs: what the tool is, how one pass works, what has been measured, and what is not true yet. Every number cites a committed artifact. The thesis is stated plainly -- the differentiator is not the dataflow engine, since CodeQL and Semgrep have one and developers do not turn it on, but the rung: a claim you can act on without writing a query, and an explicit admission when the engine could not decide. It says what does not work as clearly as what does. The C story is structurally broken, not undertuned. One language is demonstrated. PHP, Ruby and Go have zero families, so Joern would parse them and the engine would say nothing. The evidence base is two repositories and one has never produced a true positive. Control dependence is unavailable in both CPGs. execution_confirmed has no production caller. And it records the lesson that shaped the current specs: nine defects found by pointing the tool at two real repositories, three of them the same bug in three matchers, with the three gates byte-identical through every fix -- so a corpus of function-level pairs could not have caught any of them. **roadmap.md** is ordered by the standing test, that the tool must find bugs or it is SAST noise. Which gives the ordering principle: make the output worth reading, then make it find more. A better arbiter behind an unreadable report changes nothing, and a second language before the report is fixed doubles the noise. It also says what is NOT being worked and why -- authorization-obligations and reachability-flow-model were largely delivered by other routes and should be reconciled or retired rather than carried as approved-not-started, which misrepresents both. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
15 KiB
Roadmap
Overview
OpenUltraSAST can propose sinks with regex and prove a few of them in a sandbox, but adjudication is still Python-ast plus a tree-sitter CLI stub. Textbook SAST pairs (OWASP/Juliet) did not improve after the overlay: Youden went from +9.09% inventory to 0% overlay on the same 11-pair slice. Real operator trees and browser/C/C++ history are not in the catalog. The next work is to put a real tree-sitter extra under the existing overlay, then grow an offline function-level vuln-vs-fix slice from public VFC datasets (including Firefox/gecko-dev where commits exist) without vendoring whole megarepos.
Direction (2026-09-04)
Detection logic is ~16% of the source; harness plumbing, governance, and ranking are the rest. Recall on real code comes from the model with tools plus proof; precision and silence come from static checks. Shift: grow the corpus past the license and review walls (pointer slices, review tiers), turn every reviewed pair into a searchable mechanism (Clearwing's variant loop), then build the path model (call graph, summaries, fix points), put the hunter on paths, let model answers become gated facts, and stop adding harness plumbing until the detector exists. Recall first under a false-positive ceiling, per profile. Youden on non-local slices stays off the merge gate.
Approach Decision
- Chosen: Optional tree-sitter extra first, then a harvested VFC slice scored like
github(honesty dashboard, not smoke gate). Isolated functions only. Firefox is one project filter among Chromium, OpenSSL, curl — not a special-case full mozilla-central clone. - Why: The overlay already has dispositions, facts, and taint. JS/C/Java stay
language_unsupportedbecause we refused a parser library. Textbook Youden is the wrong north star; large-project VFCs are the right one. Adopting tree-sitter deletes the need to hand-write language parsers. - Rejected alternatives:
- Whole Firefox tree in CI — multi-GB, memory-safety heavy, security-bug commit messages are deliberately scrubbed.
- FixFox dataset — embargoed until 2030.
- Joern-as-required or inter-file taint in this phase — deferred by
propose-adjudicate-prove. - Chasing OWASP Youden with more regex — leaks on goodG2B/false files.
Scope
- In: tree-sitter + grammars as
openultrasast[semantic]; CST walker into existing FileIR/taint; harvested function-level pairs from CVEfixes/MoreFixes/CWE-Bench-Java/SVEN with provenance; overlay scoring on that slice; keepquickzero-dep. - Out: Cloning mozilla-central/Chromium into the repo; making overlay Youden a merge gate; buffer-size/CWE-121 semantics; inter-file taint; auto-facts from improve; LLM as parser.
Measured 2026-09-04 (extra on, before pair-corpus-honesty)
| Slice | Vuln recall | Silent on fix | Pair-correct | Youden |
|---|---|---|---|---|
| sast (11) | 45.5% | 81.8% | 4/11 | +27.3% |
| vfc (176) | 3.4% | 82.4% | 0/176 | −14.2% |
Overlay column equals inventory on vfc (no C memory sink fact). 25 curl pairs parse_failed (harvest anchored on a comment mention).
Measured 2026-09-04 (after pair-corpus-honesty scorer: coverage counted, function-scoped matching)
| Slice | Vuln recall | Silent on fix | Pair-correct | Youden | Achievable | Loss |
|---|---|---|---|---|---|---|
| sast (11) | 54.5% | 72.7% | 4/11 | +27.3% | 4/10 (java-hash known_limit) | 4 unadjudicated |
| vfc (176) | 0.0% | 81.8% | 0/176 | −18.2% | 0/176 | 50 parse_failed files, 128 unadjudicated |
New slices (pair-corpus-honesty, 2026-09-05, after the debug round: shared declarator naming, containment matching, sink-location labels): vibe-py (35, human, Real-Vuln licensed repos) 13/35 pair-correct, Youden +37.1% overlay vs 7/35 inventory; vfc-js (17) 5/17, Youden +23.5%; agent-vfc (29, agent, tier title) 0/29, Youden 0.0: no excerpt carries a rule-known sink and the overlay has no TypeScript grammar (18 rows); the unreviewed-tier baseline, never a gate. vfc parse-failed files 50 → 36 after the harvest fix.
Leave-one-out (corpus-seeded-mechanisms, 2026-09-05; the corpus as a teacher, never a gate). Each pair is held out once, a store is seeded from the other seeded/reviewed pairs, both excerpts are searched for structural variants:
| slice | pairs | detected | silent | Youden | teaching pairs | note |
|---|---|---|---|---|---|---|
| vibe-py (vendored) | 35 | 11 | 21 | −0.086 | 16 | execute/1 parameter shape taught by 7 pairs; 14 leaks on trap twins |
| local | 3 | 0 | 3 | 0.000 | 0 | reviewed, but labels name no sink call site |
| sast | 10 | 0 | 10 | 0.000 | 0 | advisory tier never teaches (known_limit pair skipped) |
| github | 6 | 0 | 6 | 0.000 | 0 | advisory tier |
| vfc-js | 17 | 0 | 17 | 0.000 | 0 | advisory tier |
The loop is closed: ousast mechanisms export writes candidates, ousast improve --pair-catalog admits the ones that recover a holdout pair without leaking (per-profile clause over variant search; mechanism_leak/mechanism_no_gain revert byte for byte), the scan searches the admitted store and reports name the mechanism and its known fix. Measured 2026-09-05 on the python-vulnerable benchmark target with the vibe-py candidates: 4 records admitted in round 1 (execute/1 parameter taught by 7 pairs among them), superseded by the learning-harness section below: that number was measured with holdout pairs teaching (see benchmarks/measurements/2026-09-06-mechanism-lever-split.json).
Measured 2026-09-05 after review tiers (Req 9) and pointer slices (Req 10): vibe-py 115 = 35 vendored human (13/35, Youden +37.1%) + 80 agent pointer pairs over the 40 LLM-generated Real-Vuln repositories (0/80, Youden 0.0; the LLM-repo baseline, scored by the nightly pairs --slice vibe-py --pointers only, never in tests or gates); agent-vfc 59 = 29 vendored + 30 pointer, tier title, 1/59, Youden 0.0, 2 unresolved labels; sast, vfc-js, vfc unchanged; detection/map/local-pair gate outputs byte-identical to the pre-spec commit. Only seeded/reviewed pairs gate the improve loop. Carry-forwards: language_unsupported loss counter and multi-line C/C++ declarator rescue (overlay-ir-completeness), profiles_ungated reporting and an offline test for the agent-vfc pointer builder (corpus-seeded-mechanisms).
sast: java-sqli moved MISS→LEAK (coverage counted on both sides; the fixed twin leaks via parameter-as-source). vfc: recall fell to zero because every label now names a function and no C detection lands inside it; that is the honest baseline the engine specs measure against.
Constraints
- Core
dependencies = []. Parser wheels live only in an extra. - Pair fixtures stay out of
LANGUAGE_MANIFESTS. - Do not vendor 16 GB VFC dumps. Pointers in
datasets.toml; a small reviewed catalog of isolated functions is what CI runs. - DiverseVul-style auto labels (~60% accurate) are not ground truth without review.
Boundary Strategy
- Why this split: Engine (how we parse) is independent of corpus (what we score). The extra can ship without any new pairs; the corpus can score inventory-only until the extra lands, then overlay.
- Shared seams to watch:
FileIR/parse_file; pair catalogslicevalues; overlay vs inventory scoring inpairs.py.
Existing Spec Updates
- propose-adjudicate-prove -- overlay roles, facts, prove-budget, sast overlay scoring (landed). Residual: tree-sitter path returns no IR.
- propose-adjudicate-prove -- do not reopen for parsers or corpora; those are the new specs below.
Direct Implementation Candidates
- Record inventory vs overlay Youden side-by-side in
ousast pairs --slice sastJSON -- folded into pair-corpus-honesty task 2.3. Done (scorersblock, 2026-09-05). - Add Firefox/gecko-dev, Chromium, OpenSSL pointers to
benchmarks/pairs/datasets.toml.
Specs (dependency order)
- tree-sitter-overlay-extra -- Optional extra: tree-sitter + grammars, one CST walker,
quickstays zero-dep. Dependencies: none - real-world-vfc-slice -- Harvest reviewed function-level vuln/fix pairs from large projects (Firefox/gecko-dev, Chromium, OpenSSL, …) as an honesty slice. Seed is testable (
ousast pairs --slice vfc). Dependencies: none. Closed 2026-09-04, superseded by pair-corpus-honesty on the corpus/scorer plane. - pair-corpus-honesty -- Scorer counts coverage and matches by function; labels carry mechanism/provenance/split/known_limit; harvest anchors at the declarator with line-range and hunk modes; sast hygiene; new slices
vibe-py(Real-Vuln-Benchmark),vfc-js(SecBench.js fix commits),agent-vfc(agent-trailer fixes); deterministic provenance fingerprint; per-profile improve acceptance; review tiers and pointer slices (amendment). Dependencies: none (supersedes real-world-vfc-slice). Completed 2026-09-05 (11 review rounds). - corpus-seeded-mechanisms -- Every seeded or reviewed pair becomes a mechanism record (source shape, sink shape, guard the fix added); structural variant search over the scanned tree proposes matches as
variantfindings that the overlay and sandbox verify; leave-one-out recall on the pair corpus is the corpus's own detection rate; the improve loop gains a recall-raisingmechanismslever under the per-profile holdout gate. Principle: recall first under a false-positive ceiling, per profile. Dependencies: pair-corpus-honesty (tiers, pointer slices). Completed 2026-09-05 (6 review rounds). - reachability-flow-model -- Name-resolved call graph over
FunctionIR; per-function taint summaries composed along the graph into path records; graph reachability drives rank, map, worth-fixing; hunter getspath_contextand hunter findings with a locatable sink become sandbox candidates; closed-kind fact oracle with candidate cache;factsimprove lever under the per-profile gate; fix points ranked by reachable risk removed. Dependencies: pair-corpus-honesty Critical path: two consumers wait on it (guard-dominance-regime and authorization-obligations). - overlay-ir-completeness -- Shared CST walker completeness (C
init_declarator, lvalue paths for field stores, per-grammar literal kinds, local ERROR-node tolerance, multi-line C/C++ declarators the walker names<anon>andfunctions.py's one-line rescue cannot name:chromium-cve-2022-2162,openssl-cve-2025-69421; a Scorer-side multi-line-aware rescue is the cheaper alternative) plus Python parity with stdlib ast (f-string constness, augmented assignment, kwargs); injection-regime facts: container/transformer taint,weak_literalsconsulted, parameter sources off for injection. Measured on sast and vibe-py. Dependencies: pair-corpus-honesty - authorization-obligations -- Absence bugs as obligations, not flows: obligated operations (protected read/write, privileged action, security setting) and discharger kinds (path guard, identity constraint from authenticated context, ownership check, non-permissive value) as closed facts; obligation shapes learned from trusted pairs; the application's policy recovered from sibling-handler consistency and an optional declared, versioned policy file; violations evaluated on the path (function-local degraded mode without it) with
consistency_violation/declared_policy_violationrungs below proof;obligationsimprove lever under the holdout gate; the hunter adjudicates intent only. Measured leave-one-out on the vibe-py and agent-vfc absence pairs (today 0 detected; 15 of 29 agent-vfc rows). Dependencies: reachability-flow-model, guard-dominance-regime, corpus-seeded-mechanisms. Requirements drafted 2026-09-05, unapproved. - guard-dominance-regime -- Guard and Use IR records; guard dominance by nesting or early exit; memory sinks (memcpy/malloc/deref) with operand roles, nullable callees, out-param transformers, fixed-array capacity; same regime applied to web absence bugs (route auth guard, identity from auth context). Consumes path records. Measured on vfc, vfc-js, agent-vfc. Dependencies: reachability-flow-model, overlay-ir-completeness
Classified detector numbers
Generated by uv run ousast learning publish from the artifacts committed beside these numbers.
| model | slice | family | scorable | unscorable | recall | silence | Youden | at fixed FPR | noise floor | cost per correct pair |
|---|---|---|---|---|---|---|---|---|---|---|
| deepseek-v4-flash | agent-vfc | access_control | 4 (taxonomy 1) | known_limit:client_side_caller: 1, unresolved_label: 1 | 0.000 | 1.000 | 0.000 | 0.000 | budget 4 | - |
| deepseek-v4-flash | agent-vfc | config_secrets | 6 (taxonomy 1) | none | 0.333 | 0.833 | 0.167 | 0.000 | budget 2 | 0.95 |
| deepseek-v4-flash | agent-vfc | injection | 2 (taxonomy 1) | none | 0.500 | 0.500 | 0.000 | 0.000 | budget 1 | - |
| deepseek-v4-flash | agent-vfc | path | 3 (taxonomy 1) | none | 1.000 | 0.667 | 0.667 | 0.000 | budget 3 | 0.48 |
| deepseek-v4-flash | agent-vfc | prototype | 7 (taxonomy 1) | none | 0.429 | 1.000 | 0.429 | 0.429 | budget 5 | 0.32 |
| deepseek-v4-flash | agent-vfc | untrusted_destination | 5 (taxonomy 1) | none | 0.600 | 0.600 | 0.200 | 0.000 | budget 3 | 0.95 |
| deepseek-v4-flash | vibe-py | access_control | 3 (taxonomy 1) | identical_twin: 1 | 1.000 | 1.000 | 1.000 | 1.000 | budget 3 | 0.23 |
| deepseek-v4-flash | vibe-py | config_secrets | 3 (taxonomy 1) | none | 1.000 | 1.000 | 1.000 | 1.000 | budget 1 | 0.23 |
| deepseek-v4-flash | vibe-py | deserialization | 2 (taxonomy 1) | identical_twin: 1 | 1.000 | 1.000 | 1.000 | 1.000 | budget 0 | 0.34 |
| deepseek-v4-flash | vibe-py | injection | 20 (taxonomy 1) | identical_twin: 2 | 0.750 | 0.900 | 0.650 | 0.750 | budget 11 | 0.05 |
| deepseek-v4-flash | vibe-py | output_encoding | 1 (taxonomy 1) | identical_twin: 1 | 1.000 | 1.000 | 1.000 | 1.000 | budget 1 | 0.69 |
| deepseek-v4-flash | vibe-py | path | 1 (taxonomy 1) | none | 1.000 | 1.000 | 1.000 | 1.000 | budget 1 | 0.69 |
| deepseek-v4-pro | vibe-py | access_control | 3 (taxonomy 1) | identical_twin: 1 | 0.000 | 1.000 | 0.000 | 0.000 | budget 2 | - |
| deepseek-v4-pro | vibe-py | config_secrets | 3 (taxonomy 1) | none | 1.000 | 1.000 | 1.000 | 1.000 | budget 2 | 0.16 |
| deepseek-v4-pro | vibe-py | deserialization | 2 (taxonomy 1) | identical_twin: 1 | 1.000 | 1.000 | 1.000 | 1.000 | budget 1 | 0.24 |
| deepseek-v4-pro | vibe-py | injection | 20 (taxonomy 1) | identical_twin: 2 | 0.350 | 0.900 | 0.250 | 0.350 | budget 14 | 0.09 |
| deepseek-v4-pro | vibe-py | output_encoding | 1 (taxonomy 1) | identical_twin: 1 | 1.000 | 1.000 | 1.000 | 1.000 | budget 0 | 0.47 |
| deepseek-v4-pro | vibe-py | path | 1 (taxonomy 1) | none | 1.000 | 1.000 | 1.000 | 1.000 | budget 0 | 0.47 |
No family in this taxonomy declares a parent, so hierarchical partial credit could not apply to any figure above.
No detector cutoff is configured, so no post-cutoff slice is reported; every figure above is over the whole corpus.
Mechanism lever on the vibe-py holdout: detected 5 of 17 at Youden -0.0588 once only train-split pairs teach, against detected 7 at Youden 0.0588 when every pair taught. The difference is what holdout pairs stopped teaching.