Marius CiepluchandClaude Fable 5.1
8ab4207618
Tasks 1.1, 5.1, 5.2, 5.4 and the data half of 5.3. - mechanisms.toml: closed vocabulary v1 (20 ids) every expected row cites. - vibe-py: 35 Real-Vuln-Benchmark pairs (Apache-2.0), human provenance only because the LLM-generated repositories carry no license (escalated). - vfc-js: 17 Node fix-commit pairs harvested upstream (MIT/Apache-2.0), SecBench.js itself is not vendored (no license); sinks derived on 13/17. - agent-vfc: 29 agent-trailer fixes as catalog-candidates.toml plus REVIEW.md; the loaded catalog stays empty until a human reviews them. - datasets.toml and README document every slice and its review status. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
vibe-py slice
Isolated Python web functions from Real-Vuln-Benchmark (Apache-2.0 benchmark; each target repository keeps its own license).
-
Vuln side: the function enclosing a labeled vulnerable finding (
is_vulnerable = true). -
Fixed twin: a false-positive trap function (
is_vulnerable = false) from the same repository. The scanner must stay silent there, so silent-on-fix here measures precision on traps. -
Provenance:
humanfor the 26 human-authored apps,agentfor the 40 LLM-generated apps. The LLM-generated repositories (kolega-ai-dev/vc-*) currently carry no license file, so they are skipped bybuild_recipes.pyand not vendored (Req 5.4). Re-run with--allow-unlicensedonly to inspect them locally; do not commit those excerpts. -
Caveat: the human corpus is educational and CTF style (DVPWA, VAmPI, vulpy, ...), not production code. Say so when reading numbers.
-
Matching: every row names the function; the upstream scorer tolerates ±10 lines, ours requires the detection inside the function.
-
Excerpt shape: line-range excerpts of methods keep their class indentation, so 27 of them fail stdlib
ast.parse("unexpected indent") while tree-sitter parses them with correct function ranges.parse_failed 0rests on tree-sitter; stdlib-ast parity would need dedenting.
Rebuild: see the docstring of build_recipes.py. Regenerate the catalog with
python benchmarks/pairs/catalog_gen.py --slice vibe-py. Maintainer only; CI never fetches.