mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 19:59:26 +00:00
Marius CiepluchandClaude Fable 5.1
3c36f13e03
Task 8.4 (Req 10.6, 5.1). - harvest.merge_recipes appends recipes by name and never rewrites existing rows. - vibe-py/build_recipes.py --pointers: the 40 LLM-generated repositories without a license become vendored = false rows (80 pairs, agent, seeded) appended to recipes.toml; unlicensed human repos stay skipped. - agent-vfc/build_recipes.py --from ... --pointers [--cache]: unlicensed agent-authored fixes become title-tier pointer rows whose function label is derived from the cached excerpt (30 added from a fresh 662-candidate search); TOML booleans rendered correctly. - catalogs regenerated: vibe-py 115 rows (35 vendored + 80 pointer), agent-vfc 59 (29 + 30); no excerpt is vendored for any pointer row. - `pairs --slice vibe-py --pointers` scores both provenances (human 35, agent 80); the default run skips 80 pairs with pointer_pair_skipped; suite and gates unchanged. README/docs name the nightly command. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
26 lines
1.7 KiB
Markdown
26 lines
1.7 KiB
Markdown
# vibe-py slice
|
|
|
|
Isolated Python web functions from [Real-Vuln-Benchmark](https://github.com/kolega-ai/Real-Vuln-Benchmark)
|
|
(Apache-2.0 benchmark; each target repository keeps its own license).
|
|
|
|
- **Vuln side**: the function enclosing a labeled vulnerable finding (`is_vulnerable = true`).
|
|
- **Fixed twin**: a false-positive trap function (`is_vulnerable = false`) from the same repository.
|
|
The scanner must stay silent there, so silent-on-fix here measures precision on traps.
|
|
- **Provenance**: `human` for the 26 human-authored apps, `agent` for the 40 LLM-generated apps.
|
|
The LLM-generated repositories (`kolega-ai-dev/vc-*`) carry **no license file**, so they
|
|
are never vendored (Req 5.4): `build_recipes.py --pointers` appends them as `vendored = false`
|
|
rows (80 pairs over the 40 repositories, `agent`, `review_tier = "seeded"`) that
|
|
`ousast pairs --slice vibe-py --pointers` harvests into the local cache and scores (Req 10).
|
|
The default run skips them with `pointer_pair_skipped`.
|
|
- **Caveat**: the human corpus is educational and CTF style (DVPWA, VAmPI, vulpy, ...), not
|
|
production code. Say so when reading numbers.
|
|
- **Matching**: every row names the function; the upstream scorer tolerates ±10 lines, ours
|
|
requires the detection inside the function.
|
|
|
|
- **Excerpt shape**: line-range excerpts of methods keep their class indentation, so 27 of them
|
|
fail stdlib `ast.parse` ("unexpected indent") while tree-sitter parses them with correct
|
|
function ranges. `parse_failed 0` rests on tree-sitter; stdlib-ast parity would need dedenting.
|
|
|
|
Rebuild: see the docstring of `build_recipes.py` (add `--pointers` for the LLM repositories). Regenerate the catalog with
|
|
`python benchmarks/pairs/catalog_gen.py --slice vibe-py`. Maintainer only; CI never fetches.
|