feat(pair-corpus-honesty): LLM-generated Real-Vuln repos and unlicensed agent fixes as pointer pairs

Task 8.4 (Req 10.6, 5.1).

- harvest.merge_recipes appends recipes by name and never rewrites
  existing rows.
- vibe-py/build_recipes.py --pointers: the 40 LLM-generated repositories
  without a license become vendored = false rows (80 pairs, agent,
  seeded) appended to recipes.toml; unlicensed human repos stay skipped.
- agent-vfc/build_recipes.py --from ... --pointers [--cache]: unlicensed
  agent-authored fixes become title-tier pointer rows whose function
  label is derived from the cached excerpt (30 added from a fresh
  662-candidate search); TOML booleans rendered correctly.
- catalogs regenerated: vibe-py 115 rows (35 vendored + 80 pointer),
  agent-vfc 59 (29 + 30); no excerpt is vendored for any pointer row.
- `pairs --slice vibe-py --pointers` scores both provenances (human 35,
  agent 80); the default run skips 80 pairs with pointer_pair_skipped;
  suite and gates unchanged. README/docs name the nightly command.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-05 10:45:43 +02:00
co-authored by Claude Fable 5.1
parent 4102404442
commit 3c36f13e03
15 changed files with 7380 additions and 42 deletions
+1
View File
@@ -199,6 +199,7 @@
- Task 8.3 (2026-09-05, RED first: 6 failing tests captured — `pointer_recipe`/`select_vendored` missing, loader `KeyError: 'vuln'` on a `vendored = false` row (x2), `pairs --pointers` unknown flag (`SystemExit: 2`), `validate_recipe(require_license=)` unknown, `catalog_gen` skipped pointer recipes as missing excerpts; then green): `PairCase.vendored`/`recipe`, `pair_cache_dir` (`OPENULTRASAST_PAIR_CACHE`, default `~/.cache/openultrasast/pairs`), `network_allowed` (`--pointers` per run or `OPENULTRASAST_PAIRS_NETWORK=1`), `select_vendored`, `_pointer_ready` (cached -> score; network -> `harvest.materialize_pointer` via the shared library; else `pointer_pair_skipped`; a failed fetch is `pointer_pair_fetch_failed`), `evaluate_catalog(pointers=)`; `harvest.validate_recipe(require_license=False)` and `materialize_pointer` (cache only, redaction applies); `catalog_gen` emits `vendored = false` rows with recipe fields and no excerpts, training manifest skips them; `pair_gate` scores vendored local pairs only; `.gitignore` covers the cache path. Smoke with real network on two Real-Vuln LLM-repo recipes into a scratchpad cache: network off -> 0 outcomes, 2 x `pointer_pair_skipped`; network on -> both harvested to `<cache>/vibe-py/<name>/{vuln,fixed}.py` and scored (silent fix on both, vuln side undetected, unresolved 0); nothing written under the repository. Suite 479 passed / 4 skipped, extra-free 466 / 17, ruff/mypy clean, three gates PASS. - Task 8.3 (2026-09-05, RED first: 6 failing tests captured — `pointer_recipe`/`select_vendored` missing, loader `KeyError: 'vuln'` on a `vendored = false` row (x2), `pairs --pointers` unknown flag (`SystemExit: 2`), `validate_recipe(require_license=)` unknown, `catalog_gen` skipped pointer recipes as missing excerpts; then green): `PairCase.vendored`/`recipe`, `pair_cache_dir` (`OPENULTRASAST_PAIR_CACHE`, default `~/.cache/openultrasast/pairs`), `network_allowed` (`--pointers` per run or `OPENULTRASAST_PAIRS_NETWORK=1`), `select_vendored`, `_pointer_ready` (cached -> score; network -> `harvest.materialize_pointer` via the shared library; else `pointer_pair_skipped`; a failed fetch is `pointer_pair_fetch_failed`), `evaluate_catalog(pointers=)`; `harvest.validate_recipe(require_license=False)` and `materialize_pointer` (cache only, redaction applies); `catalog_gen` emits `vendored = false` rows with recipe fields and no excerpts, training manifest skips them; `pair_gate` scores vendored local pairs only; `.gitignore` covers the cache path. Smoke with real network on two Real-Vuln LLM-repo recipes into a scratchpad cache: network off -> 0 outcomes, 2 x `pointer_pair_skipped`; network on -> both harvested to `<cache>/vibe-py/<name>/{vuln,fixed}.py` and scored (silent fix on both, vuln side undetected, unresolved 0); nothing written under the repository. Suite 479 passed / 4 skipped, extra-free 466 / 17, ruff/mypy clean, three gates PASS.
- Review round 9 (2026-09-05) REJECTED 8.3 on two findings, both fixed RED-first (2 failing tests captured: warm cache + network off scored the pointer pair; `evaluate_profiles` kept an `agent` pointer pair): `_pointer_ready` consults `network_allowed` before the cache, so a warm cache only spares the fetch under `--pointers`/env and never turns a default run into a pointer run; `evolve.evaluate_profiles` applies `select_vendored` before the tier filter (design amendment: "tests and gates call `select_vendored`"; ProfileGate is the gate the design names, so the one-line filter is in boundary). Suggestions applied: fetch failures carry a `detail`; the harvest module load is memoized; pointer excerpt headers get `function`/`mechanism` from the expected row; `tests/conftest.py` autouse fixture clears `OPENULTRASAST_PAIRS_NETWORK` and points the cache at a temp dir for every test; full-catalog on-disk assertions use `select_vendored`. - Review round 9 (2026-09-05) REJECTED 8.3 on two findings, both fixed RED-first (2 failing tests captured: warm cache + network off scored the pointer pair; `evaluate_profiles` kept an `agent` pointer pair): `_pointer_ready` consults `network_allowed` before the cache, so a warm cache only spares the fetch under `--pointers`/env and never turns a default run into a pointer run; `evolve.evaluate_profiles` applies `select_vendored` before the tier filter (design amendment: "tests and gates call `select_vendored`"; ProfileGate is the gate the design names, so the one-line filter is in boundary). Suggestions applied: fetch failures carry a `detail`; the harvest module load is memoized; pointer excerpt headers get `function`/`mechanism` from the expected row; `tests/conftest.py` autouse fixture clears `OPENULTRASAST_PAIRS_NETWORK` and points the cache at a temp dir for every test; full-catalog on-disk assertions use `select_vendored`.
- Review round 10 (2026-09-05) APPROVED 8.3 (verify-completion: `pytest -q` 481 passed / 4 skipped; extra-free 468 / 17 inside a no-network namespace; ruff/format/mypy clean; three gates PASS; reviewer reproduced both RED tests by mutation, the warm-cache/network-off skip on the real cache, and that the default run creates no cache and touches no network). Non-blocking: top-level README/docs mention `--pointers` only in benchmarks/pairs/README.md (ride with 8.4); the cwd-relative harvest library path follows the DEFAULT_CATALOG convention. - Review round 10 (2026-09-05) APPROVED 8.3 (verify-completion: `pytest -q` 481 passed / 4 skipped; extra-free 468 / 17 inside a no-network namespace; ruff/format/mypy clean; three gates PASS; reviewer reproduced both RED tests by mutation, the warm-cache/network-off skip on the real cache, and that the default run creates no cache and touches no network). Non-blocking: top-level README/docs mention `--pointers` only in benchmarks/pairs/README.md (ride with 8.4); the cwd-relative harvest library path follows the DEFAULT_CATALOG convention.
- Task 8.4 (2026-09-05, RED first: 4 failing tests captured — `build(pointers=)` unknown in the vibe-py builder, `merge_recipes` missing from the harvest library, the agent-vfc writer rendered booleans as strings, no pointer rows in the default catalog; then green): `harvest.merge_recipes` appends new recipes by name and never rewrites existing rows; `vibe-py/build_recipes.py --pointers` keeps the LLM-generated unlicensed repositories as `vendored = false`, `agent`, `review_tier = "seeded"` rows and appends them to recipes.toml (80 pairs over the 40 repositories; the 36 pre-existing recipes are byte-for-byte identical after reload; unlicensed human repositories stay skipped); `agent-vfc/build_recipes.py --from ... --pointers [--cache]` inverts the license rule, harvests each unlicensed candidate into the cache to derive its function label with `catalog_gen.derive_function`, dedupes against known commits/repos, and appends `title` pointer rows (30 added from a fresh 662-candidate search; 2 candidates failed to harvest, 10 had no named function; the 39 pre-existing recipes are identical). Catalogs regenerated: vibe-py 115 rows (35 vendored + 80 pointer), agent-vfc 59 rows (29 vendored + 30 pointer); training manifests unchanged (pointer rows carry no vendored files). Evidence for the observable, cache in the scratchpad: `pairs --slice vibe-py --pointers --json` scored 115 pairs, per_profile human 35 (13 correct, Youden +0.371) and agent 80 (0 correct, Youden 0.000; the LLM-repo baseline), all 80 pointer pairs fetched, no fetch failure; `pairs --slice agent-vfc --pointers --json` 1/59, Youden 0.000, unresolved 2; the default `pairs --slice vibe-py --json` scores 35 pairs and records 80 `pointer_pair_skipped`; `~/.cache/openultrasast` was never created. README/docs name the nightly `--pointers` command. Suite and gates unchanged (see round 11).
- ESCALATION (task 5.1 observable): vibe-py has `human` provenance only because the 40 LLM-generated Real-Vuln repos have no license (Req 5.4 wins). Spec owner decision needed: accept human-only vibe-py, or obtain a license statement upstream. - ESCALATION (task 5.1 observable): vibe-py has `human` provenance only because the 40 LLM-generated Real-Vuln repos have no license (Req 5.4 wins). Spec owner decision needed: accept human-only vibe-py, or obtain a license statement upstream.
- ESCALATION (Req 5.3): agent-vfc rows are unreviewed. They are NOT in the loaded catalog; `benchmarks/pairs/agent-vfc/catalog-candidates.toml` and `REVIEW.md` list them for a human reviewer. Task 5.3 stays open until reviewed. - ESCALATION (Req 5.3): agent-vfc rows are unreviewed. They are NOT in the loaded catalog; `benchmarks/pairs/agent-vfc/catalog-candidates.toml` and `REVIEW.md` list them for a human reviewer. Task 5.3 stays open until reviewed.
+1
View File
@@ -33,6 +33,7 @@ uv run ousast benchmark benchmarks/manifests/python-vulnerable.toml --mode quick
# Differential pair eval: fire on vuln, stay silent on the fix (local + real GitHub VFCs) # Differential pair eval: fire on vuln, stay silent on the fix (local + real GitHub VFCs)
uv run ousast pairs uv run ousast pairs
uv run ousast pairs --slice sast # OWASP Benchmark Java/Python + Juliet uv run ousast pairs --slice sast # OWASP Benchmark Java/Python + Juliet
uv run ousast pairs --slice vibe-py --pointers # nightly: also harvest non-vendored pointer pairs into ~/.cache
uv run python -m openultrasast.pair_gate uv run python -m openultrasast.pair_gate
# Run the bounded self-improvement loop over the ruleset (benchmark-driven) # Run the bounded self-improvement loop over the ruleset (benchmark-driven)
+3
View File
@@ -69,6 +69,9 @@ and `reviewed` pairs, so `advisory` and `title` pairs never gate the improve loo
## Pointer pairs (not vendored) ## Pointer pairs (not vendored)
Current pointer rows: vibe-py 80 (the 40 LLM-generated Real-Vuln repositories, `agent`,
`seeded`) and agent-vfc (unlicensed agent-authored fixes, `title`).
A row with `vendored = false` carries its harvest recipe (`repo`, `parent`, `commit`, A row with `vendored = false` carries its harvest recipe (`repo`, `parent`, `commit`,
`path`, `mode`, `line`/`fix_path`/`fix_line`, ...) and no excerpt files (Req 10). The `path`, `mode`, `line`/`fix_path`/`fix_line`, ...) and no excerpt files (Req 10). The
repository never redistributes that code: `pairs --pointers` (or repository never redistributes that code: `pairs --pointers` (or
+3 -1
View File
@@ -2,7 +2,9 @@
Security-fix commits carrying an AI coding agent co-author trailer or generation marker Security-fix commits carrying an AI coding agent co-author trailer or generation marker
(Claude Code, Codex, Cursor, Copilot, Devin), in JavaScript, TypeScript, or Python. (Claude Code, Codex, Cursor, Copilot, Devin), in JavaScript, TypeScript, or Python.
`provenance = "agent"`. Only repositories with a stated license are vendored. `provenance = "agent"`. Only repositories with a stated license are vendored; fixes in
unlicensed repositories are `vendored = false` pointer rows (`build_recipes.py --from ... --pointers`,
function label derived from the cached excerpt) scored by `ousast pairs --slice agent-vfc --pointers`.
## Review checklist (Req 5.3) ## Review checklist (Req 5.3)
+100 -33
View File
@@ -131,11 +131,41 @@ def classify(title: str) -> tuple[str, str, str] | None:
return None return None
def build(source: Path, *, limit: int, max_lines: int) -> tuple[list[dict[str, object]], dict[str, int]]: def _pair_tools() -> tuple[object, object]:
"""The shared harvest library and catalog generator (pointer recipes need a cached excerpt to derive their function)."""
import importlib.util
modules = []
for name, path in (("pair_harvest_lib", ROOT.parent / "harvest.py"), ("catalog_gen_lib", ROOT.parent / "catalog_gen.py")):
spec = importlib.util.spec_from_file_location(name, path)
assert spec is not None and spec.loader is not None
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
modules.append(module)
return modules[0], modules[1]
def build(
source: Path, *, limit: int, max_lines: int, pointers: bool = False, cache: Path | None = None, known: set[str] | None = None
) -> tuple[list[dict[str, object]], dict[str, int]]:
"""``pointers=True`` inverts the license rule: only unlicensed repositories are kept, as ``vendored = false`` rows whose
excerpts live in ``cache`` (fetched here to derive the function label; Req 10.6). ``known`` commits/repos are skipped."""
recipes: list[dict[str, object]] = [] recipes: list[dict[str, object]] = []
skipped = {"noise_title": 0, "unclassified": 0, "unlicensed": 0, "no_code_file": 0, "too_big": 0, "api": 0, "duplicate_repo": 0} skipped = {
"noise_title": 0,
"unclassified": 0,
"unlicensed": 0,
"licensed": 0,
"no_code_file": 0,
"too_big": 0,
"api": 0,
"duplicate_repo": 0,
"harvest": 0,
"no_function": 0,
}
per_repo: dict[str, int] = {} per_repo: dict[str, int] = {}
seen_commits: set[str] = set() # the same fix commit shows up in every fork seen_commits: set[str] = set(known or ()) # the same fix commit shows up in every fork
harvest, catalog_gen = _pair_tools() if pointers else (None, None)
rows = [json.loads(line) for line in source.read_text().splitlines() if line.strip()] rows = [json.loads(line) for line in source.read_text().splitlines() if line.strip()]
for row in sorted(rows, key=lambda r: (r["repo"], r["date"])): for row in sorted(rows, key=lambda r: (r["repo"], r["date"])):
if len(recipes) >= limit: if len(recipes) >= limit:
@@ -148,7 +178,7 @@ def build(source: Path, *, limit: int, max_lines: int) -> tuple[list[dict[str, o
if classified is None: if classified is None:
skipped["unclassified"] += 1 skipped["unclassified"] += 1
continue continue
if per_repo.get(row["repo"], 0) >= 2 or row["sha"] in seen_commits: if per_repo.get(row["repo"], 0) >= (1 if pointers else 2) or row["sha"] in seen_commits or row["repo"] in seen_commits:
skipped["duplicate_repo"] += 1 skipped["duplicate_repo"] += 1
continue continue
seen_commits.add(row["sha"]) seen_commits.add(row["sha"])
@@ -161,7 +191,10 @@ def build(source: Path, *, limit: int, max_lines: int) -> tuple[list[dict[str, o
continue continue
assert isinstance(info, dict) and isinstance(commit, dict) assert isinstance(info, dict) and isinstance(commit, dict)
license_id = (info.get("license") or {}).get("spdx_id") or "" license_id = (info.get("license") or {}).get("spdx_id") or ""
if license_id not in ACCEPTED_LICENSES: if pointers and license_id in ACCEPTED_LICENSES:
skipped["licensed"] += 1 # licensed fixes are vendored by the default mode, not pointed at
continue
if not pointers and license_id not in ACCEPTED_LICENSES:
skipped["unlicensed"] += 1 skipped["unlicensed"] += 1
continue continue
files = [f for f in commit.get("files", []) if _CODE.search(f["filename"]) and not _SKIP.search(f["filename"]) and f.get("patch")] files = [f for f in commit.get("files", []) if _CODE.search(f["filename"]) and not _SKIP.search(f["filename"]) and f.get("patch")]
@@ -183,33 +216,47 @@ def build(source: Path, *, limit: int, max_lines: int) -> tuple[list[dict[str, o
language = {".py": "python", ".ts": "typescript", ".tsx": "typescript"}.get(suffix, "javascript") language = {".py": "python", ".ts": "typescript", ".tsx": "typescript"}.get(suffix, "javascript")
base = re.sub(r"[^a-z0-9]+", "-", f"{row['repo'].split('/')[-1]}-{Path(target['filename']).stem}".lower()).strip("-")[:56] base = re.sub(r"[^a-z0-9]+", "-", f"{row['repo'].split('/')[-1]}-{Path(target['filename']).stem}".lower()).strip("-")[:56]
name = f"{base}-{row['sha'][:6]}" name = f"{base}-{row['sha'][:6]}"
recipe: dict[str, object] = {
"name": name,
"host": "github",
"repo": row["repo"],
"parent": parents[0]["sha"],
"commit": commit["sha"],
"path": target["filename"],
"mode": "hunk",
"language": language,
"relpath": target["filename"],
"license": license_id,
"cve": "",
"year": int(str(commit["commit"]["author"]["date"])[:4]),
"cwe": cwe,
"class": cls,
"mechanism": mechanism,
"provenance": "agent",
"origin": "github-agent-trailer",
"project": row["repo"].split("/")[-1],
"trailer": trailer[:120],
"reviewer": "pending",
"evidence": title[:200],
"commit_url": f"https://github.com/{row['repo']}/commit/{commit['sha']}",
}
if pointers:
assert harvest is not None and catalog_gen is not None and cache is not None
recipe["license"] = "unlicensed"
recipe["vendored"] = False
try:
harvest.materialize_pointer(recipe, cache, slice_name="agent-vfc") # type: ignore[attr-defined]
except Exception as exc: # noqa: BLE001 - a candidate that cannot be fetched is skipped, not fatal
skipped["harvest"] += 1
print(f"skip {name}: {str(exc)[:100]}")
continue
function = catalog_gen.derive_function(cache / "agent-vfc", recipe) # type: ignore[attr-defined]
if not function:
skipped["no_function"] += 1
continue
recipe["function"] = function
per_repo[row["repo"]] = per_repo.get(row["repo"], 0) + 1 per_repo[row["repo"]] = per_repo.get(row["repo"], 0) + 1
recipes.append( recipes.append(recipe)
{
"name": name,
"host": "github",
"repo": row["repo"],
"parent": parents[0]["sha"],
"commit": commit["sha"],
"path": target["filename"],
"mode": "hunk",
"language": language,
"relpath": target["filename"],
"license": license_id,
"cve": "",
"year": int(str(commit["commit"]["author"]["date"])[:4]),
"cwe": cwe,
"class": cls,
"mechanism": mechanism,
"provenance": "agent",
"origin": "github-agent-trailer",
"project": row["repo"].split("/")[-1],
"trailer": trailer[:120],
"reviewer": "pending",
"evidence": title[:200],
"commit_url": f"https://github.com/{row['repo']}/commit/{commit['sha']}",
}
)
for index, recipe in enumerate(sorted(recipes, key=lambda r: str(r["name"]))): for index, recipe in enumerate(sorted(recipes, key=lambda r: str(r["name"]))):
recipe["split"] = "holdout" if index % 2 else "train" recipe["split"] = "holdout" if index % 2 else "train"
return recipes, skipped return recipes, skipped
@@ -224,7 +271,9 @@ def to_toml(recipes: list[dict[str, object]]) -> str:
for recipe in sorted(recipes, key=lambda r: str(r["name"])): for recipe in sorted(recipes, key=lambda r: str(r["name"])):
lines.append("[[recipe]]") lines.append("[[recipe]]")
for key, value in recipe.items(): for key, value in recipe.items():
if isinstance(value, int) and not isinstance(value, bool): if isinstance(value, bool):
lines.append(f"{key} = {'true' if value else 'false'}")
elif isinstance(value, int):
lines.append(f"{key} = {value}") lines.append(f"{key} = {value}")
else: else:
lines.append(f'{key} = "{str(value).replace(chr(92), chr(92) * 2).replace(chr(34), chr(92) + chr(34))}"') lines.append(f'{key} = "{str(value).replace(chr(92), chr(92) * 2).replace(chr(34), chr(92) + chr(34))}"')
@@ -239,11 +288,29 @@ def main(argv: list[str] | None = None) -> int:
parser.add_argument("--limit", type=int, default=40) parser.add_argument("--limit", type=int, default=40)
parser.add_argument("--max-lines", type=int, default=80) parser.add_argument("--max-lines", type=int, default=80)
parser.add_argument("--out", type=Path, default=ROOT / "recipes.toml") parser.add_argument("--out", type=Path, default=ROOT / "recipes.toml")
parser.add_argument(
"--pointers", action="store_true", help="unlicensed repos only, as vendored = false rows appended to --out (Req 10.6)"
)
parser.add_argument(
"--cache", type=Path, default=None, help="pointer cache root (default: OPENULTRASAST_PAIR_CACHE or ~/.cache/openultrasast/pairs)"
)
args = parser.parse_args(argv) args = parser.parse_args(argv)
if args.search: if args.search:
print(f"searched {search(args.search)} candidates -> {args.search}") print(f"searched {search(args.search)} candidates -> {args.search}")
if args.source: if args.source:
recipes, skipped = build(args.source, limit=args.limit, max_lines=args.max_lines) import os
import tomllib
existing = [dict(r) for r in tomllib.loads(args.out.read_text()).get("recipe", [])] if args.out.is_file() else []
if args.pointers:
cache = args.cache or Path(os.environ.get("OPENULTRASAST_PAIR_CACHE") or Path.home() / ".cache" / "openultrasast" / "pairs")
known = {str(r["commit"]) for r in existing} | {str(r["repo"]) for r in existing}
new, skipped = build(args.source, limit=args.limit, max_lines=args.max_lines, pointers=True, cache=cache, known=known)
harvest, _ = _pair_tools()
recipes, added = harvest.merge_recipes(existing, new) # type: ignore[attr-defined]
print(f"appended {len(added)} pointer recipes")
else:
recipes, skipped = build(args.source, limit=args.limit, max_lines=args.max_lines)
args.out.write_text(to_toml(recipes), encoding="utf-8") args.out.write_text(to_toml(recipes), encoding="utf-8")
print(f"wrote {args.out} ({len(recipes)} recipes); skipped {skipped}") print(f"wrote {args.out} ({len(recipes)} recipes); skipped {skipped}")
return 0 return 0
File diff suppressed because it is too large Load Diff
+816
View File
@@ -26,6 +26,60 @@ evidence = "fix(hub): allow DSH token validation CORS"
commit_url = "https://github.com/1173591564/academic-based-qoder/commit/a79ddb90def54e91f7507cbf6b1548b224d642aa" commit_url = "https://github.com/1173591564/academic-based-qoder/commit/a79ddb90def54e91f7507cbf6b1548b224d642aa"
split = "train" split = "train"
[[recipe]]
name = "ah-05-07-main-0f68d7"
host = "github"
repo = "AI-HealthCare-05/AH_05_07"
parent = "350f67510bce9de606fa3ccca2a90bcd79789d7c"
commit = "0f68d71093ec0b64a4374f50a7e832e00a39170e"
path = "app/main.py"
mode = "hunk"
language = "python"
relpath = "app/main.py"
license = "unlicensed"
cve = ""
year = 2026
cwe = "CWE-16"
class = "permissive default"
mechanism = "permissive_default"
provenance = "agent"
origin = "github-agent-trailer"
project = "AH_05_07"
trailer = "Co-authored-by: Codex <[email protected]>"
reviewer = "pending"
evidence = "fix: allow browser updates through CORS (#115)"
commit_url = "https://github.com/AI-HealthCare-05/AH_05_07/commit/0f68d71093ec0b64a4374f50a7e832e00a39170e"
vendored = false
function = "request_validation_error_handler"
split = "train"
[[recipe]]
name = "ai-file-manager-searcher-2c8f65"
host = "github"
repo = "Rem763/ai-file-manager"
parent = "96416d160231278e898135a1d193c2b9d26ec341"
commit = "2c8f6556131405338fd63f989556ff7481e8a706"
path = "ai-file-manager/core/searcher.py"
mode = "hunk"
language = "python"
relpath = "ai-file-manager/core/searcher.py"
license = "unlicensed"
cve = ""
year = 2026
cwe = "CWE-89"
class = "sql injection"
mechanism = "source_reaches_sink"
provenance = "agent"
origin = "github-agent-trailer"
project = "ai-file-manager"
trailer = "Co-Authored-By: Claude Opus 4.7 <[email protected]>"
reviewer = "pending"
evidence = "fix: SQL injection hardening and API cleanup for searcher"
commit_url = "https://github.com/Rem763/ai-file-manager/commit/2c8f6556131405338fd63f989556ff7481e8a706"
vendored = false
function = "search"
split = "holdout"
[[recipe]] [[recipe]]
name = "anastasios-pw-session-14ec1a" name = "anastasios-pw-session-14ec1a"
host = "github" host = "github"
@@ -101,6 +155,115 @@ evidence = "fix(browser): harden SSRF redirect guard against non-navigation docu
commit_url = "https://github.com/P1R4N351/astroclaw/commit/1553a8adf123913cf7f1f1c75be4c4e5cd6c796b" commit_url = "https://github.com/P1R4N351/astroclaw/commit/1553a8adf123913cf7f1f1c75be4c4e5cd6c796b"
split = "holdout" split = "holdout"
[[recipe]]
name = "audienceos-command-center-route-7b2b31"
host = "github"
repo = "Diiiploy/audienceos-command-center"
parent = "f79cc1bc6288709bc33af5168fbaf5970122f3e8"
commit = "7b2b31e33cd5f582bc5f06e39b139154d08c5ad6"
path = "app/auth/callback/route.ts"
mode = "hunk"
language = "typescript"
relpath = "app/auth/callback/route.ts"
license = "unlicensed"
cve = ""
year = 2026
cwe = "CWE-601"
class = "open redirect"
mechanism = "source_reaches_sink"
provenance = "agent"
origin = "github-agent-trailer"
project = "audienceos-command-center"
trailer = "Co-Authored-By: Claude Opus 4.5 <[email protected]>"
reviewer = "pending"
evidence = "security(auth): harden OAuth callback against open redirect and error leakage"
commit_url = "https://github.com/Diiiploy/audienceos-command-center/commit/7b2b31e33cd5f582bc5f06e39b139154d08c5ad6"
vendored = false
function = "GET"
split = "train"
[[recipe]]
name = "backend-gym-main-37aec0"
host = "github"
repo = "JhonatanAbrigo11/backend-gym"
parent = "42a704179bb0c053de4dc8508f692fc25f828953"
commit = "37aec05262214c538a7543cf92921ff478e2d621"
path = "backend/src/main.ts"
mode = "hunk"
language = "typescript"
relpath = "backend/src/main.ts"
license = "unlicensed"
cve = ""
year = 2026
cwe = "CWE-16"
class = "permissive default"
mechanism = "permissive_default"
provenance = "agent"
origin = "github-agent-trailer"
project = "backend-gym"
trailer = "Co-authored-by: Cursor <[email protected]>"
reviewer = "pending"
evidence = "fix: allow Vercel frontend origin in CORS"
commit_url = "https://github.com/JhonatanAbrigo11/backend-gym/commit/37aec05262214c538a7543cf92921ff478e2d621"
vendored = false
sink = "path"
function = "bootstrap"
split = "holdout"
[[recipe]]
name = "bricks-document-viewer-client-8daa7c"
host = "github"
repo = "Lamisi/Bricks"
parent = "12501c96609d8b6de2117bed9cc42829085c4f7b"
commit = "8daa7c83d71475f8d23c0d9c6a4ee21813281c8d"
path = "components/document-viewer-client.tsx"
mode = "hunk"
language = "typescript"
relpath = "components/document-viewer-client.tsx"
license = "unlicensed"
cve = ""
year = 2026
cwe = "CWE-862"
class = "missing authorization"
mechanism = "missing_auth_guard"
provenance = "agent"
origin = "github-agent-trailer"
project = "Bricks"
trailer = "Co-Authored-By: Claude Sonnet 4.6 <[email protected]>"
reviewer = "pending"
evidence = "Fix GCM decryption missing authentication tag length (#98) (#102)"
commit_url = "https://github.com/Lamisi/Bricks/commit/8daa7c83d71475f8d23c0d9c6a4ee21813281c8d"
vendored = false
function = "openPdfExport"
split = "train"
[[recipe]]
name = "bushiri-index-bd1c99"
host = "github"
repo = "Handoni/BUSHIRI"
parent = "5da2a544ec6328f9fe39d6c897af7d5f196471f4"
commit = "bd1c99277aa6808b440a9f3c9267d477a07c8d48"
path = "apps/worker/src/index.ts"
mode = "hunk"
language = "typescript"
relpath = "apps/worker/src/index.ts"
license = "unlicensed"
cve = ""
year = 2026
cwe = "CWE-16"
class = "permissive default"
mechanism = "permissive_default"
provenance = "agent"
origin = "github-agent-trailer"
project = "BUSHIRI"
trailer = "Co-authored-by: Codex <[email protected]>"
reviewer = "pending"
evidence = "fix: Worker CORS 허용 Origin 추가"
commit_url = "https://github.com/Handoni/BUSHIRI/commit/bd1c99277aa6808b440a9f3c9267d477a07c8d48"
vendored = false
function = "resolveCorsOrigin"
split = "holdout"
[[recipe]] [[recipe]]
name = "capgo-app-apptabs-7d73fd" name = "capgo-app-apptabs-7d73fd"
host = "github" host = "github"
@@ -151,6 +314,87 @@ evidence = "fix(api): CORS for Capacitor localhost WebView (#3105)"
commit_url = "https://github.com/Cap-go/capgo.app/commit/4dadd2580d02213d227a331eee0232aa59859b59" commit_url = "https://github.com/Cap-go/capgo.app/commit/4dadd2580d02213d227a331eee0232aa59859b59"
split = "holdout" split = "holdout"
[[recipe]]
name = "career-copilot-matches-eaa75e"
host = "github"
repo = "ahmednashatnoaman-svg/career-copilot"
parent = "6dacd24d8c300eab5eb69b605ffafad68a60c008"
commit = "eaa75e6b04449a69fcae82461eca3bd67931f8c9"
path = "app/api/matches.py"
mode = "hunk"
language = "python"
relpath = "app/api/matches.py"
license = "unlicensed"
cve = ""
year = 2026
cwe = "CWE-639"
class = "insecure direct object reference"
mechanism = "identity_from_request_body"
provenance = "agent"
origin = "github-agent-trailer"
project = "career-copilot"
trailer = "Co-Authored-By: Claude <[email protected]>"
reviewer = "pending"
evidence = "security: fix IDOR in matches API — use request.state.user_id instead of query param"
commit_url = "https://github.com/ahmednashatnoaman-svg/career-copilot/commit/eaa75e6b04449a69fcae82461eca3bd67931f8c9"
vendored = false
function = "list_matches"
split = "train"
[[recipe]]
name = "chatpro-index-99dcd3"
host = "github"
repo = "anggiadiputra/chatpro"
parent = "237ccac69ce0986b8d0ef7f5f295a0aaff8979cb"
commit = "99dcd3c09a3648cb56e2335047ffffff500959fc"
path = "apps/backend/src/index.ts"
mode = "hunk"
language = "typescript"
relpath = "apps/backend/src/index.ts"
license = "unlicensed"
cve = ""
year = 2026
cwe = "CWE-16"
class = "permissive default"
mechanism = "permissive_default"
provenance = "agent"
origin = "github-agent-trailer"
project = "chatpro"
trailer = "Co-authored-by: Copilot <[email protected]>"
reviewer = "pending"
evidence = "fix(auth): allow Turnstile token in CORS preflight"
commit_url = "https://github.com/anggiadiputra/chatpro/commit/99dcd3c09a3648cb56e2335047ffffff500959fc"
vendored = false
function = "origin"
split = "holdout"
[[recipe]]
name = "comp-6000-capstone2-index-388195"
host = "github"
repo = "SpaceRobot-268369/COMP-6000-Capstone2"
parent = "a4c42810ee353ad87e6dc8f7f751c0dc8375a42d"
commit = "388195676bb0fec13a4d1163f6cb5c7155198e97"
path = "backend/src/index.js"
mode = "hunk"
language = "javascript"
relpath = "backend/src/index.js"
license = "unlicensed"
cve = ""
year = 2026
cwe = "CWE-16"
class = "permissive default"
mechanism = "permissive_default"
provenance = "agent"
origin = "github-agent-trailer"
project = "COMP-6000-Capstone2"
trailer = "Co-authored-by: Codex <[email protected]>"
reviewer = "pending"
evidence = "Fix backend CORS origin reflection"
commit_url = "https://github.com/SpaceRobot-268369/COMP-6000-Capstone2/commit/388195676bb0fec13a4d1163f6cb5c7155198e97"
vendored = false
function = "origin"
split = "train"
[[recipe]] [[recipe]]
name = "dailyforge-server-ed1470" name = "dailyforge-server-ed1470"
host = "github" host = "github"
@@ -201,6 +445,33 @@ evidence = "fix(config): prevent path traversal in repository names"
commit_url = "https://github.com/jfut/dnf-plugin-anyrepo/commit/26d19aeb8884f244e34ccfdc4f49f07055adc4d0" commit_url = "https://github.com/jfut/dnf-plugin-anyrepo/commit/26d19aeb8884f244e34ccfdc4f49f07055adc4d0"
split = "holdout" split = "holdout"
[[recipe]]
name = "easy-q-api-a31c36"
host = "github"
repo = "LucRoDZ/EASY-.Q"
parent = "f9b8fac326e9c45522bf8008dfedf6875b2449f6"
commit = "a31c36230f3902a3c2a4aa191de2528dd5993620"
path = "frontend/src/api.js"
mode = "hunk"
language = "javascript"
relpath = "frontend/src/api.js"
license = "unlicensed"
cve = ""
year = 2026
cwe = "CWE-862"
class = "missing authorization"
mechanism = "missing_auth_guard"
provenance = "agent"
origin = "github-agent-trailer"
project = "EASY-.Q"
trailer = "Co-Authored-By: Claude Sonnet 4.6 <[email protected]>"
reviewer = "pending"
evidence = "fix(auth): add Bearer token to all missing authenticated API calls"
commit_url = "https://github.com/LucRoDZ/EASY-.Q/commit/a31c36230f3902a3c2a4aa191de2528dd5993620"
vendored = false
function = "listTables"
split = "holdout"
[[recipe]] [[recipe]]
name = "edutrace-page-ee7ba2" name = "edutrace-page-ee7ba2"
host = "github" host = "github"
@@ -226,6 +497,61 @@ evidence = "fix(front): impedir prototype pollution nos handlers de formulário
commit_url = "https://github.com/jardimdesoftware/edutrace/commit/ee7ba25fb23ea89cc1b67549cb5e32c8ead92739" commit_url = "https://github.com/jardimdesoftware/edutrace/commit/ee7ba25fb23ea89cc1b67549cb5e32c8ead92739"
split = "train" split = "train"
[[recipe]]
name = "ems-contract-analyzer-app-00884c"
host = "github"
repo = "chgreer1070/ems-contract-analyzer"
parent = "440c38fd0a48dc70f183f2bd972258a1dc17c45e"
commit = "00884c98d1c43db4b647077e7f30d0ec84eaccb9"
path = "app.py"
mode = "hunk"
language = "python"
relpath = "app.py"
license = "unlicensed"
cve = ""
year = 2026
cwe = "CWE-22"
class = "path traversal"
mechanism = "path_join_user_input"
provenance = "agent"
origin = "github-agent-trailer"
project = "ems-contract-analyzer"
trailer = "Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>"
reviewer = "pending"
evidence = "Fix security issues: path traversal, debug mode, migrate PyPDF2 to pypdf"
commit_url = "https://github.com/chgreer1070/ems-contract-analyzer/commit/00884c98d1c43db4b647077e7f30d0ec84eaccb9"
vendored = false
sink = "path"
function = "analyze"
split = "train"
[[recipe]]
name = "enterprise-ai-lakehouse-platform-main-11e36d"
host = "github"
repo = "KIRAN4056/enterprise-ai-lakehouse-platform"
parent = "5f6a17f4fb49752633c2b33c123609c1118ab883"
commit = "11e36da584c79ee622faa0786a44a31d6fecf685"
path = "processing/query-engine/python-engine/main.py"
mode = "hunk"
language = "python"
relpath = "processing/query-engine/python-engine/main.py"
license = "unlicensed"
cve = ""
year = 2026
cwe = "CWE-89"
class = "sql injection"
mechanism = "source_reaches_sink"
provenance = "agent"
origin = "github-agent-trailer"
project = "enterprise-ai-lakehouse-platform"
trailer = "Co-Authored-By: Claude Sonnet 5 <[email protected]>"
reviewer = "pending"
evidence = "fix: Address SQL injection vulnerabilities in query engine and MLflow services"
commit_url = "https://github.com/KIRAN4056/enterprise-ai-lakehouse-platform/commit/11e36da584c79ee622faa0786a44a31d6fecf685"
vendored = false
function = "execute_query"
split = "holdout"
[[recipe]] [[recipe]]
name = "etoj-index-605274" name = "etoj-index-605274"
host = "github" host = "github"
@@ -301,6 +627,114 @@ evidence = "fix: make prototype pollution guard recognizable by CodeQL"
commit_url = "https://github.com/dayflower/etude-lexical-markdown/commit/db4f54b1ab80ad596de6425e84f90aa781c49999" commit_url = "https://github.com/dayflower/etude-lexical-markdown/commit/db4f54b1ab80ad596de6425e84f90aa781c49999"
split = "holdout" split = "holdout"
[[recipe]]
name = "findconstructionstaffing-route-24edf4"
host = "github"
repo = "boweiliu/findconstructionstaffing"
parent = "ff1338c0f2d4bb27a2499f82a16b2908635ec66f"
commit = "24edf4ad9c506fcb7df7878f7c25827307863fb6"
path = "app/auth/callback/route.ts"
mode = "hunk"
language = "typescript"
relpath = "app/auth/callback/route.ts"
license = "unlicensed"
cve = ""
year = 2026
cwe = "CWE-601"
class = "open redirect"
mechanism = "source_reaches_sink"
provenance = "agent"
origin = "github-agent-trailer"
project = "findconstructionstaffing"
trailer = "Co-Authored-By: Claude Opus 4.5 <[email protected]>"
reviewer = "pending"
evidence = "fix(auth): add open redirect protection and improve test types"
commit_url = "https://github.com/boweiliu/findconstructionstaffing/commit/24edf4ad9c506fcb7df7878f7c25827307863fb6"
vendored = false
function = "GET"
split = "train"
[[recipe]]
name = "fitnesstracker-communityfoods-115204"
host = "github"
repo = "JimmyMc1213/fitnesstracker"
parent = "a9bc50f9b5fecaa3c8ddb739f327fdbd74972e52"
commit = "1152045961b896dca319f2dabc0467444a72d38c"
path = "src/fitness/communityFoods.ts"
mode = "hunk"
language = "typescript"
relpath = "src/fitness/communityFoods.ts"
license = "unlicensed"
cve = ""
year = 2026
cwe = "CWE-16"
class = "permissive default"
mechanism = "permissive_default"
provenance = "agent"
origin = "github-agent-trailer"
project = "fitnesstracker"
trailer = "Co-authored-by: Cursor <[email protected]>"
reviewer = "pending"
evidence = "Fix barcode scan saves to community_foods by handling upsert errors and RLS."
commit_url = "https://github.com/JimmyMc1213/fitnesstracker/commit/1152045961b896dca319f2dabc0467444a72d38c"
vendored = false
function = "upsertCommunityFood"
split = "holdout"
[[recipe]]
name = "flytte24-views-8a5f7a"
host = "github"
repo = "bogdantro/flytte24"
parent = "63f71e32ec916f6a4cde737cbde262bd61185d56"
commit = "8a5f7afbd901af696514842e6e174318802f1219"
path = "apps/dashboard/views.py"
mode = "hunk"
language = "python"
relpath = "apps/dashboard/views.py"
license = "unlicensed"
cve = ""
year = 2026
cwe = "CWE-601"
class = "open redirect"
mechanism = "source_reaches_sink"
provenance = "agent"
origin = "github-agent-trailer"
project = "flytte24"
trailer = "Co-Authored-By: Claude Sonnet 5 <[email protected]>"
reviewer = "pending"
evidence = "fix: address final review findings (open redirect, missing confirm dialog, stylesheet path, seeded illustrations, rating validation)"
commit_url = "https://github.com/bogdantro/flytte24/commit/8a5f7afbd901af696514842e6e174318802f1219"
vendored = false
function = "business_toggle_active"
split = "train"
[[recipe]]
name = "fusmle-index-47f075"
host = "github"
repo = "am-gs/FUSMLE"
parent = "b216e440fef42b38f33dbdcd61bee98d6407aa51"
commit = "47f07502c588a2aaf9cf8d75aa918d51fd114bb1"
path = "vercel/uworld-api-deploy/index.py"
mode = "hunk"
language = "python"
relpath = "vercel/uworld-api-deploy/index.py"
license = "unlicensed"
cve = ""
year = 2026
cwe = "CWE-798"
class = "hardcoded credentials"
mechanism = "secret_in_client"
provenance = "agent"
origin = "github-agent-trailer"
project = "FUSMLE"
trailer = "Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>"
reviewer = "pending"
evidence = "security: fix vulnerable deps, restrict CORS, drop hardcoded secret, validate input"
commit_url = "https://github.com/am-gs/FUSMLE/commit/47f07502c588a2aaf9cf8d75aa918d51fd114bb1"
vendored = false
function = "qbank_generate_test"
split = "holdout"
[[recipe]] [[recipe]]
name = "gitim-git-cors-proxy-e4c3a4" name = "gitim-git-cors-proxy-e4c3a4"
host = "github" host = "github"
@@ -326,6 +760,143 @@ evidence = "fix(frontend): use gitim cors proxy by default"
commit_url = "https://github.com/CiferaTeam/GitIM/commit/e4c3a4794dbe9fbcce679f7df1ecd2b09b40f9d6" commit_url = "https://github.com/CiferaTeam/GitIM/commit/e4c3a4794dbe9fbcce679f7df1ecd2b09b40f9d6"
split = "train" split = "train"
[[recipe]]
name = "green-mobile-offline-backend-example-482952"
host = "github"
repo = "aurelien-sema/GREEN-mobile-offline"
parent = "75af5411ff56de45f979d8abd60b69d915ed2198"
commit = "482952a41b9abb23bfcb726a97c6cca72c5165a1"
path = "backend_example.py"
mode = "hunk"
language = "python"
relpath = "backend_example.py"
license = "unlicensed"
cve = ""
year = 2026
cwe = "CWE-22"
class = "path traversal"
mechanism = "path_join_user_input"
provenance = "agent"
origin = "github-agent-trailer"
project = "GREEN-mobile-offline"
trailer = "Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>"
reviewer = "pending"
evidence = "security: fix hardcoded signing password, path traversal, cleartext traffic, weak PBKDF2 & CORS"
commit_url = "https://github.com/aurelien-sema/GREEN-mobile-offline/commit/482952a41b9abb23bfcb726a97c6cca72c5165a1"
vendored = false
sink = "open"
function = "detect_disease"
split = "train"
[[recipe]]
name = "iterviz-orchestrator-c59620"
host = "github"
repo = "Ezimakyu/IterViz"
parent = "62341c4b967b8361080b918a62891e0bdb6b570d"
commit = "c59620161a32a1e3c1ab332fd5e487fa1a101e16"
path = "backend/app/orchestrator.py"
mode = "hunk"
language = "python"
relpath = "backend/app/orchestrator.py"
license = "unlicensed"
cve = ""
year = 2026
cwe = "CWE-22"
class = "path traversal"
mechanism = "path_join_user_input"
provenance = "agent"
origin = "github-agent-trailer"
project = "IterViz"
trailer = "Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>"
reviewer = "pending"
evidence = "fix(m5): address Devin Review issues (path traversal, idempotency, ws race)"
commit_url = "https://github.com/Ezimakyu/IterViz/commit/c59620161a32a1e3c1ab332fd5e487fa1a101e16"
vendored = false
function = "create_assignments"
split = "holdout"
[[recipe]]
name = "jogo-da-velha-rotas-326bac"
host = "github"
repo = "Moon15810/jogo-da-velha"
parent = "351095cda75f65fce82218b792136bea6d863d4c"
commit = "326bac6435718a958d9ef41ec5339077c89ff64a"
path = "plataforma/auth/rotas.py"
mode = "hunk"
language = "python"
relpath = "plataforma/auth/rotas.py"
license = "unlicensed"
cve = ""
year = 2026
cwe = "CWE-601"
class = "open redirect"
mechanism = "source_reaches_sink"
provenance = "agent"
origin = "github-agent-trailer"
project = "jogo-da-velha"
trailer = "Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>"
reviewer = "pending"
evidence = "fix: valida o parametro next no login (evita open redirect)"
commit_url = "https://github.com/Moon15810/jogo-da-velha/commit/326bac6435718a958d9ef41ec5339077c89ff64a"
vendored = false
function = "login"
split = "train"
[[recipe]]
name = "joidy-main-a2fa2c"
host = "github"
repo = "Axel-DaMage/joidy"
parent = "c26b1055eae2d74ed7d51c860adc6864c75f19a6"
commit = "a2fa2c6f6c1ffd15394309952ebfcc3e7715fcc7"
path = "ai-service/main.py"
mode = "hunk"
language = "python"
relpath = "ai-service/main.py"
license = "unlicensed"
cve = ""
year = 2026
cwe = "CWE-89"
class = "sql injection"
mechanism = "source_reaches_sink"
provenance = "agent"
origin = "github-agent-trailer"
project = "joidy"
trailer = "Generated with [Devin](https://devin.ai)"
reviewer = "pending"
evidence = "fix(ai-service): fix connection leak and SQL injection in /cluster endpoint (#614)"
commit_url = "https://github.com/Axel-DaMage/joidy/commit/a2fa2c6f6c1ffd15394309952ebfcc3e7715fcc7"
vendored = false
sink = "execute"
function = "cluster_notes"
split = "holdout"
[[recipe]]
name = "learnly-page-a4c6b2"
host = "github"
repo = "ParanyuLion/Learnly"
parent = "30b667e4dfbaac7b6f380ea3a53fafbc41a65795"
commit = "a4c6b2259340151fbc0411ad1c778d3caa2aea76"
path = "app/login/page.tsx"
mode = "hunk"
language = "typescript"
relpath = "app/login/page.tsx"
license = "unlicensed"
cve = ""
year = 2026
cwe = "CWE-601"
class = "open redirect"
mechanism = "source_reaches_sink"
provenance = "agent"
origin = "github-agent-trailer"
project = "Learnly"
trailer = "Co-Authored-By: Claude Sonnet 5 <[email protected]>"
reviewer = "pending"
evidence = "fix: validate login redirect target to prevent open redirect (final review fix)"
commit_url = "https://github.com/ParanyuLion/Learnly/commit/a4c6b2259340151fbc0411ad1c778d3caa2aea76"
vendored = false
function = "submit"
split = "train"
[[recipe]] [[recipe]]
name = "lensops-kubernetesmanagement-fe4910" name = "lensops-kubernetesmanagement-fe4910"
host = "github" host = "github"
@@ -476,6 +1047,33 @@ evidence = "fix(core): add prototype pollution guard to normalizeInput"
commit_url = "https://github.com/DavideCarvalho/nestjs-filter/commit/aa0e600d05997ea3d77538ea1c2db427c439f2e8" commit_url = "https://github.com/DavideCarvalho/nestjs-filter/commit/aa0e600d05997ea3d77538ea1c2db427c439f2e8"
split = "train" split = "train"
[[recipe]]
name = "neutral-server-033e5f"
host = "github"
repo = "El-Ninjo1965/Neutral"
parent = "9eb92fb1b843f4e558c72549264a9156930bf72a"
commit = "033e5f0f6f325e0a52fc13fe24f8e3b7ae2c30fd"
path = "server/bootstrap/server.js"
mode = "hunk"
language = "javascript"
relpath = "server/bootstrap/server.js"
license = "unlicensed"
cve = ""
year = 2026
cwe = "CWE-16"
class = "permissive default"
mechanism = "permissive_default"
provenance = "agent"
origin = "github-agent-trailer"
project = "Neutral"
trailer = "Co-authored-by: Copilot <[email protected]>"
reviewer = "pending"
evidence = "Fix PHP setup save and CORS bootstrap"
commit_url = "https://github.com/El-Ninjo1965/Neutral/commit/033e5f0f6f325e0a52fc13fe24f8e3b7ae2c30fd"
vendored = false
function = "sendJson"
split = "holdout"
[[recipe]] [[recipe]]
name = "new-erp-final-app-6e9644" name = "new-erp-final-app-6e9644"
host = "github" host = "github"
@@ -526,6 +1124,61 @@ evidence = "fix: resolve command injection and SSRF/Open Redirect security issue
commit_url = "https://github.com/Irnhakim/nimestream/commit/d17f535bf3012041b056ff3aa991ff6cfa198e43" commit_url = "https://github.com/Irnhakim/nimestream/commit/d17f535bf3012041b056ff3aa991ff6cfa198e43"
split = "train" split = "train"
[[recipe]]
name = "niners-initializecollections-b90f9c"
host = "github"
repo = "Ja9vill/Niners"
parent = "3e828f4f08ea0d09181226857c71e6e6b725bdff"
commit = "b90f9c325cf63cc6fe39dd046606301c2af9d119"
path = "scripts/initializeCollections.js"
mode = "hunk"
language = "javascript"
relpath = "scripts/initializeCollections.js"
license = "unlicensed"
cve = ""
year = 2026
cwe = "CWE-798"
class = "hardcoded credentials"
mechanism = "secret_in_client"
provenance = "agent"
origin = "github-agent-trailer"
project = "Niners"
trailer = "Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>"
reviewer = "pending"
evidence = "fix(security): harden auth, remove hardcoded secrets, protect endpoints"
commit_url = "https://github.com/Ja9vill/Niners/commit/b90f9c325cf63cc6fe39dd046606301c2af9d119"
vendored = false
function = "runInitialization"
split = "train"
[[recipe]]
name = "nuvanx-siteground-validate-page-templates-7ccb5b"
host = "github"
repo = "NUVANX-Medicina-Estetica-Laser/nuvanx-siteground"
parent = "0dc8e7b6cf3f3d3586b136c96bca56658872e4a8"
commit = "7ccb5b352d9ceab76dbad6bebb717a0bb2a71cf7"
path = "scripts/validate-page-templates.mjs"
mode = "hunk"
language = "javascript"
relpath = "scripts/validate-page-templates.mjs"
license = "unlicensed"
cve = ""
year = 2026
cwe = "CWE-22"
class = "path traversal"
mechanism = "path_join_user_input"
provenance = "agent"
origin = "github-agent-trailer"
project = "nuvanx-siteground"
trailer = "Generated with [Devin](https://devin.ai)"
reviewer = "pending"
evidence = "fix(security): prevent path traversal in template validation"
commit_url = "https://github.com/NUVANX-Medicina-Estetica-Laser/nuvanx-siteground/commit/7ccb5b352d9ceab76dbad6bebb717a0bb2a71cf7"
vendored = false
sink = "path"
function = "templateExists"
split = "holdout"
[[recipe]] [[recipe]]
name = "onlook-helper-57ce1c" name = "onlook-helper-57ce1c"
host = "github" host = "github"
@@ -726,6 +1379,88 @@ evidence = "Guard against ENABLE ROW LEVEL SECURITY on tables with no policies (
commit_url = "https://github.com/DelfinaCare/rls/commit/d2b6af47347fb285b7a3820a7f2d5dded20907ec" commit_url = "https://github.com/DelfinaCare/rls/commit/d2b6af47347fb285b7a3820a7f2d5dded20907ec"
split = "train" split = "train"
[[recipe]]
name = "shift-left-or-die-database-68a00a"
host = "github"
repo = "Krystian-key/shift-left-or-die"
parent = "353a8b384eb24faf34ae11fdcaf2d02b6f74f3bc"
commit = "68a00a547a97b8b9c78cdd1b91d4ad61a7a748bb"
path = "app/database.py"
mode = "hunk"
language = "python"
relpath = "app/database.py"
license = "unlicensed"
cve = ""
year = 2026
cwe = "CWE-89"
class = "sql injection"
mechanism = "source_reaches_sink"
provenance = "agent"
origin = "github-agent-trailer"
project = "shift-left-or-die"
trailer = "Co-Authored-By: Claude Haiku 4.5 <[email protected]>"
reviewer = "pending"
evidence = "Fix SQL Injection: Replace f-string SQL with parameterized SQLAlchemy ORM query"
commit_url = "https://github.com/Krystian-key/shift-left-or-die/commit/68a00a547a97b8b9c78cdd1b91d4ad61a7a748bb"
vendored = false
sink = "execute"
function = "search_scans_by_query"
split = "train"
[[recipe]]
name = "signaldigest-route-8ca292"
host = "github"
repo = "bwaide/SignalDigest"
parent = "8f7eb45c5c7cd8ffa71fa3d53730688ab147cf51"
commit = "8ca29233dd9f4603891f626216be6dfd352b0045"
path = "app/auth/callback/route.ts"
mode = "hunk"
language = "typescript"
relpath = "app/auth/callback/route.ts"
license = "unlicensed"
cve = ""
year = 2025
cwe = "CWE-601"
class = "open redirect"
mechanism = "source_reaches_sink"
provenance = "agent"
origin = "github-agent-trailer"
project = "SignalDigest"
trailer = "Co-Authored-By: Claude Sonnet 4.5 <[email protected]>"
reviewer = "pending"
evidence = "fix: prevent open redirect vulnerability in auth callback"
commit_url = "https://github.com/bwaide/SignalDigest/commit/8ca29233dd9f4603891f626216be6dfd352b0045"
vendored = false
function = "GET"
split = "holdout"
[[recipe]]
name = "soulclaw-pw-session-1346a0"
host = "github"
repo = "clawsouls/soulclaw"
parent = "60712ab1b2f57d180c77d3f25f41837d047309c1"
commit = "1346a0a62342e86cd083f5373a21cfe0678424ee"
path = "extensions/browser/src/browser/pw-session.ts"
mode = "hunk"
language = "typescript"
relpath = "extensions/browser/src/browser/pw-session.ts"
license = "unlicensed"
cve = ""
year = 2026
cwe = "CWE-918"
class = "server-side request forgery"
mechanism = "source_reaches_sink"
provenance = "agent"
origin = "github-agent-trailer"
project = "soulclaw"
trailer = "Co-authored-by: Devin Robison <[email protected]>"
reviewer = "pending"
evidence = "fix(browser): harden SSRF redirect guard against non-navigation document hops [AI] (#62355)"
commit_url = "https://github.com/clawsouls/soulclaw/commit/1346a0a62342e86cd083f5373a21cfe0678424ee"
vendored = false
function = "isTopLevelNavigationRequest"
split = "train"
[[recipe]] [[recipe]]
name = "supplieradvisor-mvp-route-387bcf" name = "supplieradvisor-mvp-route-387bcf"
host = "github" host = "github"
@@ -776,6 +1511,33 @@ evidence = "Brief 60: Fix IDOR on opportunities GET/PATCH/DELETE (missing compan
commit_url = "https://github.com/bigfivefoods/supplieradvisor-mvp/commit/a69e8f27c0f4d402573d16f0612b51155bbc46e1" commit_url = "https://github.com/bigfivefoods/supplieradvisor-mvp/commit/a69e8f27c0f4d402573d16f0612b51155bbc46e1"
split = "train" split = "train"
[[recipe]]
name = "tabletap-safe-next-33ff1b"
host = "github"
repo = "chitkid/tabletap"
parent = "3871de5c3523283a65132250a38aa8650a865f49"
commit = "33ff1b0ef0301320e7fa3984bee4e5146ae1f8d5"
path = "apps/web/lib/safe-next.ts"
mode = "hunk"
language = "typescript"
relpath = "apps/web/lib/safe-next.ts"
license = "unlicensed"
cve = ""
year = 2026
cwe = "CWE-601"
class = "open redirect"
mechanism = "source_reaches_sink"
provenance = "agent"
origin = "github-agent-trailer"
project = "tabletap"
trailer = "Co-Authored-By: Claude Fable 5.1 <[email protected]>"
reviewer = "pending"
evidence = "fix(web): close the backslash open redirect in safeNext"
commit_url = "https://github.com/chitkid/tabletap/commit/33ff1b0ef0301320e7fa3984bee4e5146ae1f8d5"
vendored = false
function = "safeNext"
split = "holdout"
[[recipe]] [[recipe]]
name = "tools-cli-tools-a96bc0" name = "tools-cli-tools-a96bc0"
host = "github" host = "github"
@@ -826,6 +1588,33 @@ evidence = "fix(security): validate mesh filenames against path traversal in _do
commit_url = "https://github.com/D-sorganization/Tools/commit/e18686aaa174a8a4843c07b4727771c92e0f179e" commit_url = "https://github.com/D-sorganization/Tools/commit/e18686aaa174a8a4843c07b4727771c92e0f179e"
split = "train" split = "train"
[[recipe]]
name = "tracechantier-web-route-cbdafd"
host = "github"
repo = "TraceChantier/tracechantier-web"
parent = "bd44165738fef8c37679eaf18eefe52eb05bb018"
commit = "cbdafd847aeea247f42482e169267558b201d434"
path = "app/api/modifier-chantier/route.ts"
mode = "hunk"
language = "typescript"
relpath = "app/api/modifier-chantier/route.ts"
license = "unlicensed"
cve = ""
year = 2026
cwe = "CWE-639"
class = "insecure direct object reference"
mechanism = "identity_from_request_body"
provenance = "agent"
origin = "github-agent-trailer"
project = "tracechantier-web"
trailer = "Co-Authored-By: claude-flow <[email protected]>"
reviewer = "pending"
evidence = "Update video copy + fix IDOR on zone deletion"
commit_url = "https://github.com/TraceChantier/tracechantier-web/commit/cbdafd847aeea247f42482e169267558b201d434"
vendored = false
function = "POST"
split = "train"
[[recipe]] [[recipe]]
name = "trackly-cli-path-diagnostics-1c1aed" name = "trackly-cli-path-diagnostics-1c1aed"
host = "github" host = "github"
@@ -851,6 +1640,33 @@ evidence = "fix(apply): report blocked path traversal"
commit_url = "https://github.com/trackly-app/trackly-cli/commit/1c1aed89b07edb04051830c8f086e02b65e961c6" commit_url = "https://github.com/trackly-app/trackly-cli/commit/1c1aed89b07edb04051830c8f086e02b65e961c6"
split = "holdout" split = "holdout"
[[recipe]]
name = "unimart-backend-auth-controller-3ee79a"
host = "github"
repo = "Richmond-24/unimart-backend"
parent = "58e26326cc1f5c01bef96461136ebcb9388fba5d"
commit = "3ee79a6123eaba498d40a5f4ba5cc0a6be5fe5c8"
path = "src/controllers/auth.controller.js"
mode = "hunk"
language = "javascript"
relpath = "src/controllers/auth.controller.js"
license = "unlicensed"
cve = ""
year = 2026
cwe = "CWE-16"
class = "permissive default"
mechanism = "permissive_default"
provenance = "agent"
origin = "github-agent-trailer"
project = "unimart-backend"
trailer = "Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>"
reviewer = "pending"
evidence = "Fix signup hanging and CORS 'can't connect to server' errors"
commit_url = "https://github.com/Richmond-24/unimart-backend/commit/3ee79a6123eaba498d40a5f4ba5cc0a6be5fe5c8"
vendored = false
function = "register"
split = "holdout"
[[recipe]] [[recipe]]
name = "upstreamdrift-local-server-c3a078" name = "upstreamdrift-local-server-c3a078"
host = "github" host = "github"
+7
View File
@@ -553,6 +553,13 @@ def fetch_url(url: str, timeout: int = 60) -> str:
return response.read().decode("utf-8", "replace") return response.read().decode("utf-8", "replace")
def merge_recipes(existing: list[dict[str, Any]], new: list[dict[str, Any]]) -> tuple[list[dict[str, Any]], list[str]]:
"""Append ``new`` recipes whose name is not already present; existing rows are never rewritten (vendored labels stay put)."""
names = {str(r["name"]) for r in existing}
added = [r for r in new if str(r["name"]) not in names]
return [*existing, *added], [str(r["name"]) for r in added]
def load_recipes(path: Path) -> tuple[dict[str, Any], ...]: def load_recipes(path: Path) -> tuple[dict[str, Any], ...]:
payload = tomllib.loads(path.read_text()) payload = tomllib.loads(path.read_text())
return tuple(dict(item) for item in payload.get("recipe", [])) return tuple(dict(item) for item in payload.get("recipe", []))
+6 -4
View File
@@ -7,9 +7,11 @@ Isolated Python web functions from [Real-Vuln-Benchmark](https://github.com/kole
- **Fixed twin**: a false-positive trap function (`is_vulnerable = false`) from the same repository. - **Fixed twin**: a false-positive trap function (`is_vulnerable = false`) from the same repository.
The scanner must stay silent there, so silent-on-fix here measures precision on traps. The scanner must stay silent there, so silent-on-fix here measures precision on traps.
- **Provenance**: `human` for the 26 human-authored apps, `agent` for the 40 LLM-generated apps. - **Provenance**: `human` for the 26 human-authored apps, `agent` for the 40 LLM-generated apps.
The LLM-generated repositories (`kolega-ai-dev/vc-*`) currently carry **no license file**, The LLM-generated repositories (`kolega-ai-dev/vc-*`) carry **no license file**, so they
so they are skipped by `build_recipes.py` and not vendored (Req 5.4). Re-run with are never vendored (Req 5.4): `build_recipes.py --pointers` appends them as `vendored = false`
`--allow-unlicensed` only to inspect them locally; do not commit those excerpts. rows (80 pairs over the 40 repositories, `agent`, `review_tier = "seeded"`) that
`ousast pairs --slice vibe-py --pointers` harvests into the local cache and scores (Req 10).
The default run skips them with `pointer_pair_skipped`.
- **Caveat**: the human corpus is educational and CTF style (DVPWA, VAmPI, vulpy, ...), not - **Caveat**: the human corpus is educational and CTF style (DVPWA, VAmPI, vulpy, ...), not
production code. Say so when reading numbers. production code. Say so when reading numbers.
- **Matching**: every row names the function; the upstream scorer tolerates ±10 lines, ours - **Matching**: every row names the function; the upstream scorer tolerates ±10 lines, ours
@@ -19,5 +21,5 @@ Isolated Python web functions from [Real-Vuln-Benchmark](https://github.com/kole
fail stdlib `ast.parse` ("unexpected indent") while tree-sitter parses them with correct fail stdlib `ast.parse` ("unexpected indent") while tree-sitter parses them with correct
function ranges. `parse_failed 0` rests on tree-sitter; stdlib-ast parity would need dedenting. function ranges. `parse_failed 0` rests on tree-sitter; stdlib-ast parity would need dedenting.
Rebuild: see the docstring of `build_recipes.py`. Regenerate the catalog with Rebuild: see the docstring of `build_recipes.py` (add `--pointers` for the LLM repositories). Regenerate the catalog with
`python benchmarks/pairs/catalog_gen.py --slice vibe-py`. Maintainer only; CI never fetches. `python benchmarks/pairs/catalog_gen.py --slice vibe-py`. Maintainer only; CI never fetches.
+24 -2
View File
@@ -87,8 +87,10 @@ def _slug(text: str) -> str:
def build( def build(
ground_truth: Path, licenses: Path, *, per_repo: int, classes: set[str], allow_unlicensed: bool ground_truth: Path, licenses: Path, *, per_repo: int, classes: set[str], allow_unlicensed: bool, pointers: bool = False
) -> tuple[list[dict[str, object]], dict[str, int]]: ) -> tuple[list[dict[str, object]], dict[str, int]]:
"""``pointers=True`` keeps the LLM-generated repositories without a license as ``vendored = false`` rows (Req 10.6):
scored through the cache with ``pairs --pointers``, never vendored. Unlicensed human repositories stay skipped."""
manifest = json.loads((ground_truth / "manifest.json").read_text()) manifest = json.loads((ground_truth / "manifest.json").read_text())
license_by_repo = json.loads(licenses.read_text()) if licenses.is_file() else {} license_by_repo = json.loads(licenses.read_text()) if licenses.is_file() else {}
recipes: list[dict[str, object]] = [] recipes: list[dict[str, object]] = []
@@ -100,7 +102,9 @@ def build(
gt = json.loads(gt_path.read_text()) gt = json.loads(gt_path.read_text())
info = license_by_repo.get(repo_id, {}) info = license_by_repo.get(repo_id, {})
license_id = info.get("license") or "" license_id = info.get("license") or ""
if license_id not in ACCEPTED_LICENSES and not allow_unlicensed: licensed = license_id in ACCEPTED_LICENSES
pointer = pointers and not licensed and meta["authorship"] == "llm_generated"
if not licensed and not allow_unlicensed and not pointer:
skipped["unlicensed"] += 1 skipped["unlicensed"] += 1
continue continue
full = info.get("full") or "/".join(meta["repo_url"].rstrip("/").split("/")[-2:]) full = info.get("full") or "/".join(meta["repo_url"].rstrip("/").split("/")[-2:])
@@ -171,6 +175,9 @@ def build(
"commit_url": f"https://github.com/{full}/blob/{commit}/{file}#L{int(location['start_line'])}", "commit_url": f"https://github.com/{full}/blob/{commit}/{file}#L{int(location['start_line'])}",
} }
) )
if pointer:
recipes[-1]["vendored"] = False
recipes[-1]["review_tier"] = "seeded" # ground truth reviewed upstream; the code is never redistributed
taken += 1 taken += 1
# deterministic holdout: every other recipe by sorted name # deterministic holdout: every other recipe by sorted name
for index, recipe in enumerate(sorted(recipes, key=lambda r: str(r["name"]))): for index, recipe in enumerate(sorted(recipes, key=lambda r: str(r["name"]))):
@@ -208,6 +215,9 @@ def main(argv: list[str] | None = None) -> int:
parser.add_argument("--per-repo", type=int, default=2) parser.add_argument("--per-repo", type=int, default=2)
parser.add_argument("--classes", default=",".join(DEFAULT_CLASSES)) parser.add_argument("--classes", default=",".join(DEFAULT_CLASSES))
parser.add_argument("--allow-unlicensed", action="store_true", help="include repositories without a stated license (NOT for vendoring)") parser.add_argument("--allow-unlicensed", action="store_true", help="include repositories without a stated license (NOT for vendoring)")
parser.add_argument(
"--pointers", action="store_true", help="append the LLM-generated unlicensed repos as vendored = false rows to --out (Req 10.6)"
)
parser.add_argument("--out", type=Path, default=ROOT / "recipes.toml") parser.add_argument("--out", type=Path, default=ROOT / "recipes.toml")
args = parser.parse_args(argv) args = parser.parse_args(argv)
recipes, skipped = build( recipes, skipped = build(
@@ -216,7 +226,19 @@ def main(argv: list[str] | None = None) -> int:
per_repo=args.per_repo, per_repo=args.per_repo,
classes=set(args.classes.split(",")), classes=set(args.classes.split(",")),
allow_unlicensed=args.allow_unlicensed, allow_unlicensed=args.allow_unlicensed,
pointers=args.pointers,
) )
if args.pointers and args.out.is_file():
import importlib.util
import tomllib
spec = importlib.util.spec_from_file_location("pair_harvest_lib", ROOT.parent / "harvest.py")
assert spec is not None and spec.loader is not None
harvest = importlib.util.module_from_spec(spec)
spec.loader.exec_module(harvest)
existing = [dict(r) for r in tomllib.loads(args.out.read_text()).get("recipe", [])]
recipes, added = harvest.merge_recipes(existing, [r for r in recipes if r.get("vendored") is False])
print(f"appended {len(added)} pointer recipes to {args.out}")
args.out.write_text(to_toml(recipes), encoding="utf-8") args.out.write_text(to_toml(recipes), encoding="utf-8")
print(f"wrote {args.out} ({len(recipes)} recipes); skipped {skipped}") print(f"wrote {args.out} ({len(recipes)} recipes); skipped {skipped}")
return 0 return 0
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+1
View File
@@ -36,6 +36,7 @@ uv run ousast benchmark benchmarks/manifests/python-vulnerable.toml --mode quick
uv run ousast pairs # local fixtures + vendored GitHub VFCs uv run ousast pairs # local fixtures + vendored GitHub VFCs
uv run ousast pairs --slice github --json # honesty dashboard as JSON uv run ousast pairs --slice github --json # honesty dashboard as JSON
uv run ousast pairs --slice sast # OWASP Benchmark + Juliet Youden (TPR-FPR) uv run ousast pairs --slice sast # OWASP Benchmark + Juliet Youden (TPR-FPR)
uv run ousast pairs --slice vibe-py --pointers # nightly: pointer pairs via the local cache (network)
uv run python -m openultrasast.pair_gate # CI: local pairs must all pass uv run python -m openultrasast.pair_gate # CI: local pairs must all pass
``` ```
+24 -2
View File
@@ -464,9 +464,10 @@ def test_new_slices_load_offline_and_score_with_profiles() -> None:
assert all(case.license for case in select_vendored(cases)), slice_name # pointer rows may be unlicensed (Req 10) assert all(case.license for case in select_vendored(cases)), slice_name # pointer rows may be unlicensed (Req 10)
assert all(case.provenance == expected_provenance for case in select_vendored(cases)), slice_name assert all(case.provenance == expected_provenance for case in select_vendored(cases)), slice_name
assert {case.split for case in cases} <= {"train", "holdout"} assert {case.split for case in cases} <= {"train", "holdout"}
header = cases[0].vuln_file.read_text()[:600] vendored = select_vendored(cases) # pointer rows have no excerpt in the repository (Req 10)
header = vendored[0].vuln_file.read_text()[:600]
assert "Provenance:" in header and "license:" in header assert "Provenance:" in header and "license:" in header
result = evaluate_catalog(cases[:2]) result = evaluate_catalog(vendored[:2])
assert result.per_slice[slice_name].pairs == 2 assert result.per_slice[slice_name].pairs == 2
assert set(result.per_profile) <= PROVENANCES and expected_provenance in result.per_profile assert set(result.per_profile) <= PROVENANCES and expected_provenance in result.per_profile
assert result.per_mechanism and slice_name in result.loss assert result.per_mechanism and slice_name in result.loss
@@ -963,3 +964,24 @@ def test_warm_cache_never_scores_pointer_pairs_when_network_is_off(tmp_path: Pat
assert result.outcomes == () assert result.outcomes == ()
assert {"stage": "pairs", "reason": "pointer_pair_skipped", "pair": "ptr"} in result.degradations assert {"stage": "pairs", "reason": "pointer_pair_skipped", "pair": "ptr"} in result.degradations
assert evaluate_catalog([case], pointers=True).outcomes[0].pair_correct # warm cache, network on: no fetch, scored assert evaluate_catalog([case], pointers=True).outcomes[0].pair_correct # warm cache, network on: no fetch, scored
def test_vibe_py_and_agent_vfc_carry_pointer_rows_and_the_default_run_skips_them() -> None:
"""Req 10.6: LLM-generated Real-Vuln repos are agent/seeded pointer rows; unlicensed agent fixes are title pointer rows."""
import io
from contextlib import redirect_stdout
catalog = load_pair_catalog()
vibe = select_slice(catalog, "vibe-py")
pointers = [case for case in vibe if not case.vendored]
assert len(pointers) >= 40 and len({case.repo for case in pointers}) >= 40
assert all(case.provenance == "agent" and case.review_tier == "seeded" and not case.vuln_file.exists() for case in pointers)
assert len(select_vendored(vibe)) == 35 and all(case.provenance == "human" for case in select_vendored(vibe))
agent_pointers = [case for case in select_slice(catalog, "agent-vfc") if not case.vendored]
assert len(agent_pointers) >= 20 and all(case.review_tier == "title" and case.license == "unlicensed" for case in agent_pointers)
buf = io.StringIO()
with redirect_stdout(buf):
assert main(["pairs", "--slice", "vibe-py", "--json"]) == 0
payload = json.loads(buf.getvalue())
skipped = [item for item in payload["degradations"] if item["reason"] == "pointer_pair_skipped"]
assert len(skipped) == len(pointers) and payload["overall"]["pairs"] == 35
+98
View File
@@ -424,3 +424,101 @@ def test_catalog_gen_emits_pointer_rows_without_excerpts(tmp_path: Path, monkeyp
(case,) = _load_catalog_file(catalog) (case,) = _load_catalog_file(catalog)
assert case.vendored is False and case.provenance == "agent" assert case.vendored is False and case.provenance == "agent"
assert not (slice_root / "training" / "manifest.jsonl").read_text().strip() # pointer rows have no vendored files to train on assert not (slice_root / "training" / "manifest.jsonl").read_text().strip() # pointer rows have no vendored files to train on
# --- task 8.4: pointer recipes from the builders (Req 10.6) ---------------------
def _ground_truth(tmp_path: Path) -> Path:
import json
gt = tmp_path / "gt"
gt.mkdir()
repos = {
"realvuln-vc-llm-app": {
"repo_url": "https://github.com/k/vc-llm-app",
"commit_sha": "c1",
"authorship": "llm_generated",
"framework": "flask",
},
"realvuln-human-nolic": {
"repo_url": "https://github.com/h/nolic",
"commit_sha": "c2",
"authorship": "human_authored",
"framework": "flask",
},
"realvuln-human-mit": {
"repo_url": "https://github.com/h/mit",
"commit_sha": "c3",
"authorship": "human_authored",
"framework": "flask",
},
}
(gt / "manifest.json").write_text(json.dumps({"repos": repos}))
findings = [
{
"id": "v1",
"is_vulnerable": True,
"file": "app.py",
"location": {"function": "run", "start_line": 3},
"vulnerability_class": "command_injection",
"primary_cwe": "CWE-78",
"evidence": {"description": "x"},
},
{
"id": "t1",
"is_vulnerable": False,
"file": "app.py",
"location": {"function": "safe", "start_line": 9},
"vulnerability_class": "command_injection",
"primary_cwe": "CWE-78",
},
]
for repo_id in repos:
(gt / f"{repo_id}.json").write_text(json.dumps({"findings": findings}))
(gt / "licenses.json").write_text(
json.dumps(
{
"realvuln-vc-llm-app": {"full": "k/vc-llm-app", "license": ""},
"realvuln-human-nolic": {"full": "h/nolic", "license": ""},
"realvuln-human-mit": {"full": "h/mit", "license": "MIT"},
}
)
)
return gt
def test_vibe_py_builder_pointers_mode_emits_llm_repos_as_unvendored_agent_rows(tmp_path: Path) -> None:
b = runpy.run_path(str(Path("benchmarks/pairs/vibe-py/build_recipes.py")), run_name="vibe_builder")
gt = _ground_truth(tmp_path)
recipes, skipped = b["build"](
gt, gt / "licenses.json", per_repo=2, classes={"command_injection"}, allow_unlicensed=False, pointers=True
)
by_repo = {r["repo"]: r for r in recipes}
assert set(by_repo) == {"k/vc-llm-app", "h/mit"} # unlicensed human repos are neither vendored nor pointed at
llm = by_repo["k/vc-llm-app"]
assert llm["vendored"] is False and llm["provenance"] == "agent" and llm["review_tier"] == "seeded" and llm["license"] == "unlicensed"
assert llm["function"] == "run" and llm["fix_path"] == "app.py" and llm["fix_line"] == 9
assert "vendored" not in by_repo["h/mit"] and by_repo["h/mit"]["provenance"] == "human"
assert skipped["unlicensed"] == 1
text = b["to_toml"](recipes)
import tomllib
rows = tomllib.loads(text)["recipe"]
assert any(row.get("vendored") is False for row in rows)
def test_merge_recipes_keeps_existing_rows_and_appends_new_by_name() -> None:
lib = _lib()
existing = [{"name": "a", "repo": "x", "license": "MIT"}, {"name": "b", "repo": "y", "license": "MIT"}]
new = [{"name": "b", "repo": "changed"}, {"name": "c", "repo": "z", "vendored": False}]
merged, added = lib["merge_recipes"](existing, new)
assert [r["name"] for r in merged] == ["a", "b", "c"] and merged[1]["repo"] == "y" and added == ["c"]
def test_agent_vfc_builder_writes_toml_booleans() -> None:
import tomllib
b = runpy.run_path(str(Path("benchmarks/pairs/agent-vfc/build_recipes.py")), run_name="agent_builder")
(row,) = tomllib.loads(b["to_toml"]([{"name": "p", "vendored": False, "line": 3, "reviewer": "pending"}]))["recipe"]
assert row["vendored"] is False and row["line"] == 3