Marius CiepluchandClaude Opus 5
3e71e9e88b
The catalog generator fills `fix_date` from the recipe at the precision the recipe has -- a full ISO day when one was recorded, otherwise the year. Every harvested recipe carries a year and almost none carries a day, and writing `-01-01` would turn a year into a claim about a month. A row counts as post-cutoff only when its earliest possible date is after the cutoff, so a coarse date can under-claim the clean slice but never over-claim it. Each family now reports the same numbers over the rows fixed after the cutoff, beside the whole number and never instead of it, with the undated count on its own line. Without a configured cutoff every scoreboard says so rather than inventing one: guessing a model's training cutoff and then publishing a contamination number from the guess would be worse than reporting none. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
vibe-py slice
Isolated Python web functions from Real-Vuln-Benchmark (Apache-2.0 benchmark; each target repository keeps its own license).
-
Vuln side: the function enclosing a labeled vulnerable finding (
is_vulnerable = true). -
Fixed twin: a false-positive trap function (
is_vulnerable = false) from the same repository. The scanner must stay silent there, so silent-on-fix here measures precision on traps. -
Provenance:
humanfor the 26 human-authored apps,agentfor the 40 LLM-generated apps. The LLM-generated repositories (kolega-ai-dev/vc-*) carry no license file, so they are never vendored (Req 5.4):build_recipes.py --pointersappends them asvendored = falserows (80 pairs over the 40 repositories,agent,review_tier = "seeded") thatousast pairs --slice vibe-py --pointersharvests into the local cache and scores (Req 10). The default run skips them withpointer_pair_skipped. -
Caveat: the human corpus is educational and CTF style (DVPWA, VAmPI, vulpy, ...), not production code. Say so when reading numbers.
-
Matching: every row names the function; the upstream scorer tolerates ±10 lines, ours requires the detection inside the function.
-
Excerpt shape: line-range excerpts of methods keep their class indentation, so 27 of them fail stdlib
ast.parse("unexpected indent") while tree-sitter parses them with correct function ranges.parse_failed 0rests on tree-sitter; stdlib-ast parity would need dedenting.
Rebuild: see the docstring of build_recipes.py (add --pointers for the LLM repositories). Regenerate the catalog with
python benchmarks/pairs/catalog_gen.py --slice vibe-py. Maintainer only; CI never fetches.