mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-18 20:49:28 +00:00
Marius CiepluchandClaude Fable 5.1
8ab4207618
Tasks 1.1, 5.1, 5.2, 5.4 and the data half of 5.3. - mechanisms.toml: closed vocabulary v1 (20 ids) every expected row cites. - vibe-py: 35 Real-Vuln-Benchmark pairs (Apache-2.0), human provenance only because the LLM-generated repositories carry no license (escalated). - vfc-js: 17 Node fix-commit pairs harvested upstream (MIT/Apache-2.0), SecBench.js itself is not vendored (no license); sinks derived on 13/17. - agent-vfc: 29 agent-trailer fixes as catalog-candidates.toml plus REVIEW.md; the loaded catalog stays empty until a human reviews them. - datasets.toml and README document every slice and its review status. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
24 lines
1.5 KiB
Markdown
24 lines
1.5 KiB
Markdown
# vibe-py slice
|
|
|
|
Isolated Python web functions from [Real-Vuln-Benchmark](https://github.com/kolega-ai/Real-Vuln-Benchmark)
|
|
(Apache-2.0 benchmark; each target repository keeps its own license).
|
|
|
|
- **Vuln side**: the function enclosing a labeled vulnerable finding (`is_vulnerable = true`).
|
|
- **Fixed twin**: a false-positive trap function (`is_vulnerable = false`) from the same repository.
|
|
The scanner must stay silent there, so silent-on-fix here measures precision on traps.
|
|
- **Provenance**: `human` for the 26 human-authored apps, `agent` for the 40 LLM-generated apps.
|
|
The LLM-generated repositories (`kolega-ai-dev/vc-*`) currently carry **no license file**,
|
|
so they are skipped by `build_recipes.py` and not vendored (Req 5.4). Re-run with
|
|
`--allow-unlicensed` only to inspect them locally; do not commit those excerpts.
|
|
- **Caveat**: the human corpus is educational and CTF style (DVPWA, VAmPI, vulpy, ...), not
|
|
production code. Say so when reading numbers.
|
|
- **Matching**: every row names the function; the upstream scorer tolerates ±10 lines, ours
|
|
requires the detection inside the function.
|
|
|
|
- **Excerpt shape**: line-range excerpts of methods keep their class indentation, so 27 of them
|
|
fail stdlib `ast.parse` ("unexpected indent") while tree-sitter parses them with correct
|
|
function ranges. `parse_failed 0` rests on tree-sitter; stdlib-ast parity would need dedenting.
|
|
|
|
Rebuild: see the docstring of `build_recipes.py`. Regenerate the catalog with
|
|
`python benchmarks/pairs/catalog_gen.py --slice vibe-py`. Maintainer only; CI never fetches.
|