mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
Marius CiepluchandClaude Opus 5
3e71e9e88b
The catalog generator fills `fix_date` from the recipe at the precision the recipe has -- a full ISO day when one was recorded, otherwise the year. Every harvested recipe carries a year and almost none carries a day, and writing `-01-01` would turn a year into a claim about a month. A row counts as post-cutoff only when its earliest possible date is after the cutoff, so a coarse date can under-claim the clean slice but never over-claim it. Each family now reports the same numbers over the rows fixed after the cutoff, beside the whole number and never instead of it, with the undated count on its own line. Without a configured cutoff every scoreboard says so rather than inventing one: guessing a model's training cutoff and then publishing a contamination number from the guess would be worse than reporting none. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
VFC training catalog
Each JSONL row is one isolated snapshot: vuln (positive) or fixed (negative).
python3 benchmarks/pairs/vfc/generate_catalog.py
manifest.jsonl fields: id, pair, label (vuln|fixed), cve, year, cwe, project, file, relpath, license, commit, commit_url.
The JSONL is the training index. Pair eval (ousast pairs --slice vfc) is the honesty dashboard, not a merge gate.
Filter 2020–2026:
import json
from pathlib import Path
rows = [json.loads(line) for line in Path("benchmarks/pairs/vfc/training/manifest.jsonl").read_text().splitlines()]
recent = [row for row in rows if 2020 <= row["year"] <= 2026]
Do not treat this slice as a merge gate. Memory-safety pairs often LEAK after the fix; UAF/type-confusion often MISS inventory.