mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
Marius CiepluchandClaude Fable 5.1
c3e841003c
Tasks 3.3 and 4.1. - vfc curl and openssl-cve-2021-3449 excerpts re-harvested through the shared library: whole functions with upstream_start headers, the vendored key redacted code-safely, no truncated bodies. - vfc/recipes.toml carries mode, fix_path and function anchors; training/manifest.jsonl regenerated. - Juliet CWE-121 strcpy pair gets its buffer declarations back so the vulnerable side is a real overflow again. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Vuln vs fixed pair corpus
This corpus measures harness efficiency, not cheat-sheet recall.
The stage-1 smoke gate (python -m openultrasast.gate) scans one tree that
already contains the planted sinks. A pair eval instead scans two isolated
trees:
- vuln — only the vulnerable file. The harness should fire (true positive).
- fixed — only the patched file, same relative path. The harness should stay silent on that CWE (true negative).
pair_correct = detected on vuln and silent on the fix. That is the signal
the improvement loop consumes (miss on the vuln side, fp on the fix side).
Layout
| Path | Role |
|---|---|
catalog.toml |
Runnable offline pairs (local fixtures + vendored GitHub functions) |
datasets.toml |
Pointers to public VFC datasets — do not vendor the dumps |
datasets/cwe-bench-java-project_info.csv |
120 vetted Java CVE buggy/fix commits |
github/ |
Isolated function excerpts with provenance headers |
Pairs in this directory must not join LANGUAGE_MANIFESTS.
Slices
- local — in-tree fixture counterparts (
app.pyvssafe.py, …). CI gate: every local pair must pass (python -m openultrasast.pair_gate). - github — real commit/function pairs (SVEN, CWE-Bench-Java, NVD patterns). Measured, not a 90% smoke gate. Misses here are the honesty / evolve-the-harness dashboard.
- sast — OWASP Benchmark Java/Python true-vs-false files and Juliet
bad-vs-good sinks (
benchmarks/pairs/sast/). Score is Youden (TPR − FPR). Honesty dashboard, not a CI fail. - vfc — reviewed isolated functions from OpenSSL, Firefox, Chromium, and
curl (
benchmarks/pairs/vfc/). Overlay vs inventory scoreboard. Honesty dashboard, not a CI fail. Training labels (vuln vs fixed) arevfc/training/manifest.jsonl(vuln vs fixed, one snapshot per line). Grow with a bloblessgit log --grep=CVE-202mine plusvfc/harvest.py, thenvfc/generate_catalog.py(maintainer, not CI).
Commands
uv run ousast pairs
uv run ousast pairs --slice github --json
uv run ousast pairs --slice vfc --json
uv run python -m openultrasast.pair_gate
Growing the corpus
Prefer function-level before/after from a reviewed source (SVEN jsonl, a
CWE-Bench-Java fix_info.csv row, a GHSA with a tiny patch). Isolate the
changed function into vuln / fixed files with the same relpath. Leave
whole-repo clones to --fetch later; CI stays offline.