marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(benchmark): freeze the valid-benchmark pilot, 12 verified positives + 240 negatives

Browse Source 
First live run of the valid detection benchmark. Labels come from the fix
site (fix^1), never SZZ.

- 32 held-out Python/JS advisories sourced from the public pool-p1 set,
  CWE + text from OSV. 133 candidate locations -> 125 after dropping
  tests/minified.
- 125 DeepSeek verification calls: 31 locations passed all three checks
  (vulnerable pre-fix, fix addresses it, reachable in-repo) across 12 repos.
- Maintainer spot-check accepted all 12.
- 20 ordinary negatives per positive change from the same repos, 240 total.
- Held-out guard: 0 dropped against the training-only used set.
- Frozen manifest at benchmarks/valid/pilot/ (252 changes), provenance and
  NOTES under benchmarks/measurements/2026-10-12-valid-benchmark-pilot/.

Three fixes found by the live run:
- catalog.py: skip a site the fix renamed or removed instead of aborting.
- assemble.py: negative ratio counts positive changes, not accepted
  functions (malla would have demanded 260 negatives). Test updated.
- held-out guard: run against training sources only; build_used() also
  scans measurements/populations, which flagged the 12 repos via my own
  SZZ-pool analysis outputs (circular).

Replay and metrics not yet run; see NOTES.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-10-12 19:42:45 +02:00
co-authored by Claude Fable 5.1
parent 43ef254134
commit 7a636d60fb
10 changed files with 4606 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/test_valid_benchmark.py
+3 -1
View File
@@ -438,7 +438,9 @@ def test_multi_function_fix_is_one_push_with_all_case_ids(pair, tmp_path, monkey
tmp_path / "multi", cases=cases, verifications=[accepted(case) for case in cases], repos={original["repo"]: git}, used=set()
)
loaded = replay.load_slice(tmp_path / "multi/pool-valid.toml", 1)
assert len(loaded) == 41 and len({row["id"] for row in loaded}) == 41
# One fix commit is one positive change, so the 20x negative ratio is 20, not 20 per
# accepted function. Two accepted functions still collapse to a single vulnerability push.
assert len(loaded) == 21 and len({row["id"] for row in loaded}) == 21
(positive,) = [row for row in loaded if row["kind"] == "vulnerability"]
assert set(positive["case_ids"]) == {case["id"] for case in cases}
target = positive["targets"][1]