mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-18 20:49:28 +00:00

Files

T

Marius CiepluchandClaude Fable 5.1 3eba6e10a0 docs(pair-corpus-honesty): spec, review record, roadmap; close real-world-vfc-slice as superseded

Task 7.2 and the spec itself (brief, requirements incl. Req 9/10
amendment, research, design, tasks with eight review rounds recorded).

- real-world-vfc-slice: spec.json closed, superseded_by
  pair-corpus-honesty; brief carries the closure note.
- roadmap: Direction section, per-slice baselines (sast 4/11, vibe-py
  13/35, vfc-js 5/17, vfc 0/176, agent-vfc 0/29 title tier), spec order
  pair-corpus-honesty -> corpus-seeded-mechanisms -> reachability-flow-model
  -> overlay-ir-completeness -> guard-dominance-regime.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG

2026-09-05 10:45:43 +02:00

4.0 KiB

Raw Blame History

Brief: real-world-vfc-slice

Problem

Operators care whether the harness fires on a real buggy snapshot and stays quiet after the patch — Firefox, Chromium, OpenSSL — not whether OWASP BenchmarkTest00074 is a TP. The labeled SAST slice is 11 textbook files. Overlay Youden there is worse than inventory (+9.09% → 0%). We cannot tell if the tool got better on messy product code.

Current State

Slices today: local (CI 100%), github (6 vendored functions: SVEN, hutool, node-serialize), sast (OWASP/Juliet). datasets.toml already points at CVEfixes, MoreFixes, CWE-Bench-Java, Vul4J. Nothing is harvested from gecko-dev / Firefox. Whole mozilla-central is not in-tree (and should not be). FixFox (Zenodo) is embargoed until 2030. Firefox security commit messages are often scrubbed of “security” words, so grep-the-log is a bad miner.

Measured on the current 11-pair sast slice (2026-09-04):

Scorer	pair_correct	vuln recall	silent-on-fix	Youden
Inventory (regex)	2/11 (18%)	54.5%	54.5%	+9.09%
Overlay (current)	2/11 (18%)	45.5%	54.5%	0%

The only overlay change vs inventory on that slice: owasp-python-codeinj went from LEAK (regex hit both sides) to MISS (ConfigParser heap → flow_incomplete, no promote). C/Java Juliet goodG2B still leak because those languages are unadjudicated and scoring falls back to inventory.

Desired Outcome

A new pair slice (e.g. vfc) of reviewed, isolated function pairs from large projects, including Firefox/gecko-dev when a public fix commit exists in CVEfixes/MoreFixes. Offline CI. Honesty dashboard like github, not a 90% smoke gate. Overlay scoring once tree-sitter can parse C/JS/Java; inventory scoring until then. Provenance on every file (repo, commit, CVE, license).

Approach

Do not clone Firefox. Filter public VFC databases for gecko-dev / mozilla-firefox/firefox, chromium, openssl, curl, plus existing CWE-Bench-Java. Extract the changed function before/after. Human-review labels (DiverseVul ~60% is not ground truth). Cap the runnable catalog (tens of pairs, not thousands). Prefer injection/XSS/path CWEs the overlay can talk about; keep UAF/buffer overflows as known inventory-only leaks until a later memory-safety spec.

Scope

In: harvest script or documented procedure; catalog slice; provenance headers; overlay-vs-inventory scoreboard on that slice; dataset.toml pointers for Firefox/Chromium.
Out: Vendoring mozilla-central; live Bugzilla scraping; making Youden a merge gate; buffer-size semantics; tree-sitter packaging (owned by tree-sitter-overlay-extra).

Boundary Candidates

Catalog schema / slice name vs pair eval scoring
Harvest tooling vs vendored excerpts

Out of Boundary

Parser extra, Joern, improve-loop auto-facts, LANGUAGE_MANIFESTS

Upstream / Downstream

Upstream: pairs.py catalog format; propose-adjudicate-prove overlay scoring
Downstream: improve-loop signals from real misses; later memory-safety spec can reuse the same slice

Existing Spec Touchpoints

Extends: pair catalog only (new slice). Does not change smoke gate.
Adjacent: tree-sitter-overlay-extra (C/Java overlay on this slice once the extra exists)

Constraints

Isolated functions, same relpath vuln/fixed. License in the header. No 16 GB dumps in git. Firefox pairs only when a public git commit pair exists — MFSA text without a commit is not a pair.

Closure (2026-09-04)

Closed. All tasks done; tests/test_pair_corpus.py and tests/test_vfc_harvest.py pass (17). Seeds from OpenSSL, Firefox, Chromium, and curl landed (176 pairs). Residuals found in review are corpus quality and scorer fidelity, not seed coverage: 25 curl pairs start mid-comment because harvest anchors on a comment mention; 129 expected rows have sink = "unknown"; the overlay column equals inventory because C facts have no memory sink. These are owned by pair-corpus-honesty, which supersedes this spec on the corpus and scorer plane. Do not reopen.

4.0 KiB Raw Blame History