mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
Marius CiepluchandClaude Fable 5.1
071dea4712
Tasks 3.1, 3.2 (harvest library) and the Scorer-owned naming helper. - semantic/functions.py: one source of function names for scorer, overlay and catalog generator (declarator wins over parser name; JS/TS bare-param arrows and undeclared methods are <anon>; C++ Class::method resolves to its unqualified tail; Perl sub; statements never name). - benchmarks/pairs/harvest.py: shared library with name / line_range / hunk / enclosing modes, comment-blind masking (triple-quote aware), upstream_start headers, code-safe quoted-only redaction. - benchmarks/pairs/catalog_gen.py: derive function labels and sinks from the declarator at the mapped sink location, CWE-family sink tokens, --prune with refusal when nothing derived, fails loud without the package; vfc/harvest.py and vfc/generate_catalog.py become shims. - tests/test_pair_harvest.py replaces test_vfc_harvest.py. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Vuln vs fixed pair corpus
This corpus measures harness efficiency, not cheat-sheet recall.
The stage-1 smoke gate (python -m openultrasast.gate) scans one tree that
already contains the planted sinks. A pair eval instead scans two isolated
trees:
- vuln — only the vulnerable file. The harness should fire (true positive).
- fixed — only the patched file, same relative path. The harness should stay silent on that CWE (true negative).
pair_correct = detected on vuln and silent on the fix. That is the signal
the improvement loop consumes (miss on the vuln side, fp on the fix side).
Layout
| Path | Role |
|---|---|
catalog.toml |
Runnable offline pairs (local fixtures + vendored GitHub functions) |
datasets.toml |
Pointers to public VFC datasets — do not vendor the dumps |
datasets/cwe-bench-java-project_info.csv |
120 vetted Java CVE buggy/fix commits |
github/ |
Isolated function excerpts with provenance headers |
Pairs in this directory must not join LANGUAGE_MANIFESTS.
Slices
- local — in-tree fixture counterparts (
app.pyvssafe.py, …). CI gate: every local pair must pass (python -m openultrasast.pair_gate). - github — real commit/function pairs (SVEN, CWE-Bench-Java, NVD patterns). Measured, not a 90% smoke gate. Misses here are the honesty / evolve-the-harness dashboard.
- sast — OWASP Benchmark Java/Python true-vs-false files and Juliet
bad-vs-good sinks (
benchmarks/pairs/sast/). Score is Youden (TPR − FPR). Honesty dashboard, not a CI fail. - vfc — reviewed isolated functions from OpenSSL, Firefox, Chromium, and
curl (
benchmarks/pairs/vfc/). Overlay vs inventory scoreboard. Honesty dashboard, not a CI fail. Training labels (vuln vs fixed) arevfc/training/manifest.jsonl(vuln vs fixed, one snapshot per line). Grow with a bloblessgit log --grep=CVE-202mine plusvfc/harvest.py, thenvfc/generate_catalog.py(maintainer, not CI).
Commands
uv run ousast pairs
uv run ousast pairs --slice github --json
uv run ousast pairs --slice vfc --json
uv run python -m openultrasast.pair_gate
Growing the corpus
Prefer function-level before/after from a reviewed source (SVEN jsonl, a
CWE-Bench-Java fix_info.csv row, a GHSA with a tiny patch). Isolate the
changed function into vuln / fixed files with the same relpath. Leave
whole-repo clones to --fetch later; CI stays offline.