mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 11:49:28 +00:00
Marius CiepluchandClaude Fable 5.1
7a636d60fb
First live run of the valid detection benchmark. Labels come from the fix site (fix^1), never SZZ. - 32 held-out Python/JS advisories sourced from the public pool-p1 set, CWE + text from OSV. 133 candidate locations -> 125 after dropping tests/minified. - 125 DeepSeek verification calls: 31 locations passed all three checks (vulnerable pre-fix, fix addresses it, reachable in-repo) across 12 repos. - Maintainer spot-check accepted all 12. - 20 ordinary negatives per positive change from the same repos, 240 total. - Held-out guard: 0 dropped against the training-only used set. - Frozen manifest at benchmarks/valid/pilot/ (252 changes), provenance and NOTES under benchmarks/measurements/2026-10-12-valid-benchmark-pilot/. Three fixes found by the live run: - catalog.py: skip a site the fix renamed or removed instead of aborting. - assemble.py: negative ratio counts positive changes, not accepted functions (malla would have demanded 260 negatives). Test updated. - held-out guard: run against training sources only; build_used() also scans measurements/populations, which flagged the 12 repos via my own SZZ-pool analysis outputs (circular). Replay and metrics not yet run; see NOTES. Co-Authored-By: Claude Fable 5.1 <[email protected]>