mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-18 20:49:28 +00:00
Marius CiepluchandClaude Fable 5.1
7a636d60fb
First live run of the valid detection benchmark. Labels come from the fix site (fix^1), never SZZ. - 32 held-out Python/JS advisories sourced from the public pool-p1 set, CWE + text from OSV. 133 candidate locations -> 125 after dropping tests/minified. - 125 DeepSeek verification calls: 31 locations passed all three checks (vulnerable pre-fix, fix addresses it, reachable in-repo) across 12 repos. - Maintainer spot-check accepted all 12. - 20 ordinary negatives per positive change from the same repos, 240 total. - Held-out guard: 0 dropped against the training-only used set. - Frozen manifest at benchmarks/valid/pilot/ (252 changes), provenance and NOTES under benchmarks/measurements/2026-10-12-valid-benchmark-pilot/. Three fixes found by the live run: - catalog.py: skip a site the fix renamed or removed instead of aborting. - assemble.py: negative ratio counts positive changes, not accepted functions (malla would have demanded 260 negatives). Test updated. - held-out guard: run against training sources only; build_used() also scans measurements/populations, which flagged the 12 repos via my own SZZ-pool analysis outputs (circular). Replay and metrics not yet run; see NOTES. Co-Authored-By: Claude Fable 5.1 <[email protected]>
1 line
329 B
JSON
1 line
329 B
JSON
["bytebase/dbhub", "contentful/contentful-mcp-server", "fastapi-users/fastapi-users", "i18next/i18next-http-middleware", "imzbf/md-editor-v3", "jupyter-lsp/jupyterlab-lsp", "matrix-org/matrix-appservice-bridge", "nvbn/thefuck", "ondata/ckan-mcp-server", "webpack/webpack-bundle-analyzer", "zenitram/malla", "zereight/gitlab-mcp"] |