marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-19 21:19:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(pair-corpus-honesty): LLM-generated Real-Vuln repos and unlicensed agent fixes as pointer pairs

Browse Source 
Task 8.4 (Req 10.6, 5.1).

- harvest.merge_recipes appends recipes by name and never rewrites
  existing rows.
- vibe-py/build_recipes.py --pointers: the 40 LLM-generated repositories
  without a license become vendored = false rows (80 pairs, agent,
  seeded) appended to recipes.toml; unlicensed human repos stay skipped.
- agent-vfc/build_recipes.py --from ... --pointers [--cache]: unlicensed
  agent-authored fixes become title-tier pointer rows whose function
  label is derived from the cached excerpt (30 added from a fresh
  662-candidate search); TOML booleans rendered correctly.
- catalogs regenerated: vibe-py 115 rows (35 vendored + 80 pointer),
  agent-vfc 59 (29 + 30); no excerpt is vendored for any pointer row.
- `pairs --slice vibe-py --pointers` scores both provenances (human 35,
  agent 80); the default run skips 80 pairs with pointer_pair_skipped;
  suite and gates unchanged. README/docs name the nightly command.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-05 10:45:43 +02:00
co-authored by Claude Fable 5.1
parent 4102404442
commit 3c36f13e03
15 changed files with 7380 additions and 42 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benchmarks/pairs/vibe-py/README.md
+6 -4
View File
@@ -7,9 +7,11 @@ Isolated Python web functions from [Real-Vuln-Benchmark](https://github.com/kole
- **Fixed twin**: a false-positive trap function (`is_vulnerable = false`) from the same repository.
The scanner must stay silent there, so silent-on-fix here measures precision on traps.
- **Provenance**: `human` for the 26 human-authored apps, `agent` for the 40 LLM-generated apps.
The LLM-generated repositories (`kolega-ai-dev/vc-*`) currently carry **no license file**,
so they are skipped by `build_recipes.py` and not vendored (Req 5.4). Re-run with
`--allow-unlicensed` only to inspect them locally; do not commit those excerpts.
The LLM-generated repositories (`kolega-ai-dev/vc-*`) carry **no license file**, so they
are never vendored (Req 5.4): `build_recipes.py --pointers` appends them as `vendored = false`
rows (80 pairs over the 40 repositories, `agent`, `review_tier = "seeded"`) that
`ousast pairs --slice vibe-py --pointers` harvests into the local cache and scores (Req 10).
The default run skips them with `pointer_pair_skipped`.
- **Caveat**: the human corpus is educational and CTF style (DVPWA, VAmPI, vulpy, ...), not
production code. Say so when reading numbers.
- **Matching**: every row names the function; the upstream scorer tolerates ±10 lines, ours
@@ -19,5 +21,5 @@ Isolated Python web functions from [Real-Vuln-Benchmark](https://github.com/kole
fail stdlib `ast.parse` ("unexpected indent") while tree-sitter parses them with correct
function ranges. `parse_failed 0` rests on tree-sitter; stdlib-ast parity would need dedenting.
Rebuild: see the docstring of `build_recipes.py`. Regenerate the catalog with
Rebuild: see the docstring of `build_recipes.py` (add `--pointers` for the LLM repositories). Regenerate the catalog with
`python benchmarks/pairs/catalog_gen.py --slice vibe-py`. Maintainer only; CI never fetches.