Marius CiepluchandClaude Opus 5
c2d9c4ad01
Eight of the ten vendored access-control rows had no named handler at all, so every one of them would have scored `missing_context` forever rather than as a detector miss. After the re-harvest nine are named and the tenth states its reason. The handler-context mode now carries the class its framework registers, multi-line decorators, and commented-out guards -- `#@token_required` above a ThreatByte handler is the entire bug. Where the registration lives in another document (connexion's `operationId`, an express router), a recipe names `context = [...]`, the harvester writes that stanza per side, and the pair materializes both files at their repository-relative paths so the entry-point mapper sees the layout the project has. `harvest_recipe` fetches every blob before the first write and the driver takes `--report`, so a failed fetch leaves a reviewed excerpt exactly as it was. Two defects the re-harvest exposed: `.mjs` was not a known suffix, so the one pair vendored as an ES module had no file target at all; and `_python_route_access` matched auth words inside decorator prose, reading `@ns.doc(description='... without proper authorization checks')` as a guard. Recorded in the hygiene artifact: nine of the ten duplicate groups sit on both sides of the split, which the repository-level split cannot see. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
vibe-py slice
Isolated Python web functions from Real-Vuln-Benchmark (Apache-2.0 benchmark; each target repository keeps its own license).
-
Vuln side: the function enclosing a labeled vulnerable finding (
is_vulnerable = true). -
Fixed twin: a false-positive trap function (
is_vulnerable = false) from the same repository. The scanner must stay silent there, so silent-on-fix here measures precision on traps. -
Provenance:
humanfor the 26 human-authored apps,agentfor the 40 LLM-generated apps. The LLM-generated repositories (kolega-ai-dev/vc-*) carry no license file, so they are never vendored (Req 5.4):build_recipes.py --pointersappends them asvendored = falserows (80 pairs over the 40 repositories,agent,review_tier = "seeded") thatousast pairs --slice vibe-py --pointersharvests into the local cache and scores (Req 10). The default run skips them withpointer_pair_skipped. -
Caveat: the human corpus is educational and CTF style (DVPWA, VAmPI, vulpy, ...), not production code. Say so when reading numbers.
-
Matching: every row names the function; the upstream scorer tolerates ±10 lines, ours requires the detection inside the function.
-
Excerpt shape: line-range excerpts of methods keep their class indentation, so 27 of them fail stdlib
ast.parse("unexpected indent") while tree-sitter parses them with correct function ranges.parse_failed 0rests on tree-sitter; stdlib-ast parity would need dedenting.
Rebuild: see the docstring of build_recipes.py (add --pointers for the LLM repositories). Regenerate the catalog with
python benchmarks/pairs/catalog_gen.py --slice vibe-py. Maintainer only; CI never fetches.