diff --git a/.gitignore b/.gitignore index dc67fa4..c0baa61 100644 --- a/.gitignore +++ b/.gitignore @@ -19,6 +19,8 @@ build/ *.egg-info/ **/.openultrasast/ +.cache/openultrasast/ +**/.cache/openultrasast/ artifacts/ reports/ vector-store/ diff --git a/.kiro/specs/pair-corpus-honesty/tasks.md b/.kiro/specs/pair-corpus-honesty/tasks.md index 3f4a897..afd1273 100644 --- a/.kiro/specs/pair-corpus-honesty/tasks.md +++ b/.kiro/specs/pair-corpus-honesty/tasks.md @@ -169,7 +169,7 @@ - _Boundary: ProfileGate_ - _Depends: 8.1_ -- [ ] 8.3 Score non-vendored pointer pairs through a local cache +- [x] 8.3 Score non-vendored pointer pairs through a local cache - Catalog rows with `vendored = false` carry the recipe; evaluation harvests both sides into the cache when `OPENULTRASAST_PAIRS_NETWORK=1` (or `pairs --pointers`), otherwise skips with a degradation; the cache is git-ignored; tests and gates select vendored pairs only. - Observable: a pointer row evaluates to the same outcome shape as a vendored row when network is on, is skipped with `pointer_pair_skipped` when off, and no cache file appears under the repository. - _Requirements: 10.1, 10.2, 10.3, 10.4, 10.5_ @@ -196,6 +196,9 @@ - Review round 7 (2026-09-05) APPROVED tasks 7.1 and 7.2; REJECTED 8.1 on documentation only (REVIEW.md still described the candidates file; READMEs claimed the tier gate filter before 8.2 existed; TS zeros misattributed). Remediated: REVIEW.md/READMEs rewritten to tier semantics, note corrected, `catalog_gen._review_tier` precedence aligned with the loader (explicit tier wins), `training/manifest.jsonl` rows carry `review_tier`, design §LabelSchema candidates sentence superseded, roadmap agent-vfc line updated. Deferred to the engine specs: a `language_unsupported` loss counter (reviewer suggestion). - Task 8.2 (2026-09-05, RED first: `test_profile_gate_ignores_title_and_advisory_tiers` failed with `reason='profile_regression:agent'` on five title-tier pairs; then green): `evolve.evaluate_profiles` filters holdout pairs to `GATING_TIERS` via `select_tier` before scoring, so `advisory`/`title` pairs never gate and a regression on `reviewed` pairs still rejects the round. - Review round 8 (2026-09-05) APPROVED 8.1 and 8.2 (verify-completion: `pytest -q` 473 passed / 4 skipped, extra-free 460 / 17, ruff/mypy clean, three gates PASS; reviewer reproduced RED by removing the tier filter and checked the default holdout gate now covers `human` seeded pairs only, with the 16 title-tier agent rows excluded). Task 5.3 closed with 8.1 as re-scoped. Applied suggestion: the agent-vfc tier test accepts `reviewed` rows so promoting a row does not break it. Follow-up for corpus-seeded-mechanisms: report holdout profiles excluded by tier (`profiles_ungated`) so an operator sees why `agent`/`synthetic` do not gate. +- Task 8.3 (2026-09-05, RED first: 6 failing tests captured — `pointer_recipe`/`select_vendored` missing, loader `KeyError: 'vuln'` on a `vendored = false` row (x2), `pairs --pointers` unknown flag (`SystemExit: 2`), `validate_recipe(require_license=)` unknown, `catalog_gen` skipped pointer recipes as missing excerpts; then green): `PairCase.vendored`/`recipe`, `pair_cache_dir` (`OPENULTRASAST_PAIR_CACHE`, default `~/.cache/openultrasast/pairs`), `network_allowed` (`--pointers` per run or `OPENULTRASAST_PAIRS_NETWORK=1`), `select_vendored`, `_pointer_ready` (cached -> score; network -> `harvest.materialize_pointer` via the shared library; else `pointer_pair_skipped`; a failed fetch is `pointer_pair_fetch_failed`), `evaluate_catalog(pointers=)`; `harvest.validate_recipe(require_license=False)` and `materialize_pointer` (cache only, redaction applies); `catalog_gen` emits `vendored = false` rows with recipe fields and no excerpts, training manifest skips them; `pair_gate` scores vendored local pairs only; `.gitignore` covers the cache path. Smoke with real network on two Real-Vuln LLM-repo recipes into a scratchpad cache: network off -> 0 outcomes, 2 x `pointer_pair_skipped`; network on -> both harvested to `/vibe-py//{vuln,fixed}.py` and scored (silent fix on both, vuln side undetected, unresolved 0); nothing written under the repository. Suite 479 passed / 4 skipped, extra-free 466 / 17, ruff/mypy clean, three gates PASS. +- Review round 9 (2026-09-05) REJECTED 8.3 on two findings, both fixed RED-first (2 failing tests captured: warm cache + network off scored the pointer pair; `evaluate_profiles` kept an `agent` pointer pair): `_pointer_ready` consults `network_allowed` before the cache, so a warm cache only spares the fetch under `--pointers`/env and never turns a default run into a pointer run; `evolve.evaluate_profiles` applies `select_vendored` before the tier filter (design amendment: "tests and gates call `select_vendored`"; ProfileGate is the gate the design names, so the one-line filter is in boundary). Suggestions applied: fetch failures carry a `detail`; the harvest module load is memoized; pointer excerpt headers get `function`/`mechanism` from the expected row; `tests/conftest.py` autouse fixture clears `OPENULTRASAST_PAIRS_NETWORK` and points the cache at a temp dir for every test; full-catalog on-disk assertions use `select_vendored`. +- Review round 10 (2026-09-05) APPROVED 8.3 (verify-completion: `pytest -q` 481 passed / 4 skipped; extra-free 468 / 17 inside a no-network namespace; ruff/format/mypy clean; three gates PASS; reviewer reproduced both RED tests by mutation, the warm-cache/network-off skip on the real cache, and that the default run creates no cache and touches no network). Non-blocking: top-level README/docs mention `--pointers` only in benchmarks/pairs/README.md (ride with 8.4); the cwd-relative harvest library path follows the DEFAULT_CATALOG convention. - ESCALATION (task 5.1 observable): vibe-py has `human` provenance only because the 40 LLM-generated Real-Vuln repos have no license (Req 5.4 wins). Spec owner decision needed: accept human-only vibe-py, or obtain a license statement upstream. - ESCALATION (Req 5.3): agent-vfc rows are unreviewed. They are NOT in the loaded catalog; `benchmarks/pairs/agent-vfc/catalog-candidates.toml` and `REVIEW.md` list them for a human reviewer. Task 5.3 stays open until reviewed. diff --git a/benchmarks/pairs/README.md b/benchmarks/pairs/README.md index aacea87..81ad01c 100644 --- a/benchmarks/pairs/README.md +++ b/benchmarks/pairs/README.md @@ -67,6 +67,18 @@ Defaults per slice: local `reviewed`, vibe-py `seeded`, sast/vfc/vfc-js/github `advisory`, agent-vfc `title`. `pairs --json` reports `per_tier`; `evolve.evaluate_profiles` scores only `seeded` and `reviewed` pairs, so `advisory` and `title` pairs never gate the improve loop (task 8.2). +## Pointer pairs (not vendored) + +A row with `vendored = false` carries its harvest recipe (`repo`, `parent`, `commit`, +`path`, `mode`, `line`/`fix_path`/`fix_line`, ...) and no excerpt files (Req 10). The +repository never redistributes that code: `pairs --pointers` (or +`OPENULTRASAST_PAIRS_NETWORK=1`) harvests both sides into +`OPENULTRASAST_PAIR_CACHE` (default `~/.cache/openultrasast/pairs///`, +git-ignored) and scores them exactly like a vendored pair; without network the pair is +skipped with a `pointer_pair_skipped` degradation and a failed fetch is +`pointer_pair_fetch_failed`, never an error. The default `pytest` run and every gate +score vendored pairs only (`select_vendored`); the pointer run is the nightly command. + ## Labels, profiles, and mechanisms Every `[[pair.expected]]` row names a `mechanism` from the closed vocabulary in diff --git a/benchmarks/pairs/catalog_gen.py b/benchmarks/pairs/catalog_gen.py index b600a6d..7caf4c4 100644 --- a/benchmarks/pairs/catalog_gen.py +++ b/benchmarks/pairs/catalog_gen.py @@ -52,6 +52,7 @@ def catalog_text(slice_name: str, recipes: list[dict[str, object]]) -> str: for recipe in recipes: vuln, fixed = excerpt_rel(recipe) lang = LANG_TO_PAIR.get(str(recipe.get("language", "c")), "c_cpp") + pointer = recipe.get("vendored") is False lines += [ "[[pair]]", f'name = "{_q(recipe["name"])}"', @@ -65,8 +66,13 @@ def catalog_text(slice_name: str, recipes: list[dict[str, object]]) -> str: f'commit = "{_q(recipe.get("commit", ""))}"', f'parent = "{_q(recipe.get("parent", ""))}"', f'commit_url = "{_q(recipe.get("commit_url", ""))}"', - f'vuln = "{vuln}"', - f'fixed = "{fixed}"', + ] + if pointer: + lines.append("vendored = false") # Req 10.1: the recipe travels in the row, no excerpt in the repository + lines += _pointer_lines(recipe) + else: + lines += [f'vuln = "{vuln}"', f'fixed = "{fixed}"'] + lines += [ f'relpath = "{_q(recipe.get("relpath", recipe.get("path", "")))}"', "min_recall = 1.0", 'fix_policy = "silent"', @@ -114,9 +120,21 @@ def _review_tier(slice_name: str, recipe: dict[str, object]) -> str: return _DEFAULT_TIER.get(slice_name, "advisory") +_POINTER_STR = ("host", "path", "fix_repo", "fix_path", "mode") +_POINTER_INT = ("line", "fix_line", "line_start", "line_end") + + +def _pointer_lines(recipe: dict[str, object]) -> list[str]: + out = [f'{key} = "{_q(recipe[key])}"' for key in _POINTER_STR if recipe.get(key)] + out += [f"{key} = {int(recipe[key])}" for key in _POINTER_INT if isinstance(recipe.get(key), int)] + return out + + def training_rows(slice_name: str, recipes: list[dict[str, object]]) -> list[dict[str, object]]: rows: list[dict[str, object]] = [] for recipe in recipes: + if recipe.get("vendored") is False: + continue # nothing vendored to train on vuln, fixed = excerpt_rel(recipe) base = { "pair": recipe["name"], @@ -353,7 +371,12 @@ def generate(slice_name: str, *, prune: bool = False) -> tuple[Path, int]: recipes: list[dict[str, object]] = [] for recipe in loaded: vuln, fixed = excerpt_rel(recipe) - if (slice_root / vuln).is_file() and (slice_root / fixed).is_file(): + if recipe.get("vendored") is False: + if not recipe.get("function"): + print(f"skip {recipe['name']}: pointer recipe without a function label (nothing to derive from)") + continue + recipes.append(recipe) + elif (slice_root / vuln).is_file() and (slice_root / fixed).is_file(): if not recipe.get("function"): derived = derive_function(slice_root, recipe) if not derived: diff --git a/benchmarks/pairs/harvest.py b/benchmarks/pairs/harvest.py index f875073..37e3bf9 100644 --- a/benchmarks/pairs/harvest.py +++ b/benchmarks/pairs/harvest.py @@ -373,7 +373,7 @@ def provenance_header(recipe: dict[str, Any], *, side: str, sha: str, upstream_s return "/* " + "\n * ".join(lines) + "\n */\n" -def validate_recipe(recipe: dict[str, Any]) -> None: +def validate_recipe(recipe: dict[str, Any], *, require_license: bool = True) -> None: name = str(recipe.get("name", "")) if any(token in str(recipe).lower() for token in BLOCKED_TOKENS): raise RecipeError(f"{name}: embargoed or blocked dataset") @@ -390,7 +390,7 @@ def validate_recipe(recipe: dict[str, Any]) -> None: raise RecipeError(f"{name}: line_range mode needs integer line_start and line_end") if mode == "enclosing" and not isinstance(recipe.get("line"), int): raise RecipeError(f"{name}: enclosing mode needs an integer line") - if not recipe.get("license"): + if require_license and not recipe.get("license"): raise RecipeError(f"{name}: license required") @@ -514,8 +514,8 @@ def extract_pair(recipe: dict[str, Any], parent_src: str, fix_src: str) -> tuple return extract_function(parent_src, function, language), extract_function(fix_src, function, language) -def harvest_recipe(recipe: dict[str, Any], root: Path) -> tuple[Path, Path]: - validate_recipe(recipe) +def harvest_recipe(recipe: dict[str, Any], root: Path, *, require_license: bool = True) -> tuple[Path, Path]: + validate_recipe(recipe, require_license=require_license) if str(recipe.get("host", "github")) != "github": raise RecipeError(f"{recipe.get('name')}: live fetch currently implemented for host=github only") parent_src = fetch_url(github_raw_url(str(recipe["repo"]), str(recipe["parent"]), str(recipe["path"]))) @@ -531,6 +531,19 @@ def harvest_recipe(recipe: dict[str, Any], root: Path) -> tuple[Path, Path]: return vuln_path, fixed_path +def materialize_pointer(recipe: dict[str, Any], cache_root: Path, *, slice_name: str) -> tuple[Path, Path]: + """Harvest a non-vendored pair into ``cache_root///`` (Req 10.2). + + The cache lives outside the repository; a missing upstream license is allowed here because nothing is redistributed, + the excerpts stay on the operator's machine. Redaction applies as for vendored excerpts. + """ + root = Path(cache_root) / slice_name + vuln_path, fixed_path = excerpt_paths(recipe, root) + if vuln_path.is_file() and fixed_path.is_file(): + return vuln_path, fixed_path + return harvest_recipe(recipe, root, require_license=False) + + def fetch_url(url: str, timeout: int = 60) -> str: host = urlparse(url).hostname or "" if host not in ALLOWED_HOSTS: diff --git a/src/openultrasast/cli.py b/src/openultrasast/cli.py index 1e0bbb2..7428e28 100644 --- a/src/openultrasast/cli.py +++ b/src/openultrasast/cli.py @@ -156,6 +156,7 @@ def main(argv: list[str] | None = None) -> int: pairs.add_argument("--split", choices=("all", "train", "holdout"), default="all", help="filter pairs by declared split") pairs.add_argument("--hunter", action="store_true", help="also score the LLM tool hunter on overlay slices (needs a hunter model)") pairs.add_argument("--hunter-model", default=None, help="model id for --hunter (default: [models].hunter from openultrasast.toml)") + pairs.add_argument("--pointers", action="store_true", help="allow network for non-vendored pointer pairs this run (nightly; CI never)") pairs.add_argument("--json", action="store_true", help="print the pair scoreboard as JSON") subparsers.add_parser("mcp", help="run the narrow MCP server over stdio for OpenCode integration") @@ -190,6 +191,7 @@ def main(argv: list[str] | None = None) -> int: split=args.split, hunter=args.hunter, hunter_model=args.hunter_model, + pointers=args.pointers, ) if args.command == "mcp": from .mcp import serve # lazy: keeps the import cycle (mcp -> cli) one-directional @@ -787,6 +789,7 @@ def _pairs( split: str = "all", hunter: bool = False, hunter_model: str | None = None, + pointers: bool = False, ) -> int: if not catalog.exists() or not catalog.is_file(): raise SystemExit(f"pair catalog is not a file: {catalog}") @@ -799,7 +802,7 @@ def _pairs( client = tool_hunter.resolve_hunter_client() if model and client is not None: scan = make_hunter_scan(client, model) - result = evaluate_catalog(cases, hunter=scan) + result = evaluate_catalog(cases, hunter=scan, pointers=True if pointers else None) if json_out: print(json.dumps(result_payload(result), indent=2, sort_keys=True)) return 0 diff --git a/src/openultrasast/improve/evolve.py b/src/openultrasast/improve/evolve.py index 3ba651f..989d7be 100644 --- a/src/openultrasast/improve/evolve.py +++ b/src/openultrasast/improve/evolve.py @@ -61,9 +61,10 @@ def evaluate_profiles( Only ``seeded`` and ``reviewed`` pairs gate (Req 9.3); ``advisory`` and ``title`` pairs are scored by ``pairs``, never here. """ - from ..pairs import GATING_TIERS, PairCase, evaluate_catalog, select_tier + from ..pairs import GATING_TIERS, PairCase, evaluate_catalog, select_tier, select_vendored - cases = list(select_tier([case for case in pair_cases if isinstance(case, PairCase)], GATING_TIERS)) + # Gates score vendored pairs only (Req 10.4); pointer pairs are for `pairs --pointers`, never for the improve gate. + cases = list(select_tier(select_vendored([case for case in pair_cases if isinstance(case, PairCase)]), GATING_TIERS)) if not cases: return {} result = evaluate_catalog(cases, ruleset=rules, inventory_sidecar=False) diff --git a/src/openultrasast/pair_gate.py b/src/openultrasast/pair_gate.py index 5be88db..7a96e16 100644 --- a/src/openultrasast/pair_gate.py +++ b/src/openultrasast/pair_gate.py @@ -21,6 +21,7 @@ from .pairs import ( evaluate_catalog, load_pair_catalog, select_slice, + select_vendored, ) @@ -33,7 +34,7 @@ class PairGateVerdict: def pair_gate(cases: Sequence[PairCase] | None = None) -> PairGateVerdict: catalog = tuple(cases) if cases is not None else load_pair_catalog(DEFAULT_CATALOG) - local = select_slice(catalog, "local") + local = select_vendored(select_slice(catalog, "local")) # Req 10.4: gates score vendored pairs only if not local: return PairGateVerdict(passed=False, reasons=("no local pairs in catalog",), result=evaluate_catalog(())) result = evaluate_catalog(local) diff --git a/src/openultrasast/pairs.py b/src/openultrasast/pairs.py index 6aa7297..a216187 100644 --- a/src/openultrasast/pairs.py +++ b/src/openultrasast/pairs.py @@ -13,6 +13,8 @@ evaluated but excluded from the achievable numbers. from __future__ import annotations +import importlib.util +import os import shutil import tempfile import tomllib @@ -58,6 +60,29 @@ SPLITS = frozenset({"train", "holdout"}) # Req 9: how a label was established. Only seeded and reviewed pairs steer the improve loop. REVIEW_TIERS = frozenset({"seeded", "advisory", "title", "reviewed"}) GATING_TIERS = frozenset({"seeded", "reviewed"}) +# Req 10: pointer pairs are harvested into a cache outside the repository, never vendored. +PAIR_CACHE_ENV = "OPENULTRASAST_PAIR_CACHE" +PAIRS_NETWORK_ENV = "OPENULTRASAST_PAIRS_NETWORK" +DEFAULT_PAIR_CACHE = Path.home() / ".cache" / "openultrasast" / "pairs" +POINTER_REQUIRED = ("repo", "parent", "commit", "path", "mode") +POINTER_FIELDS = POINTER_REQUIRED + ( + "host", + "fix_repo", + "fix_path", + "line", + "fix_line", + "line_start", + "line_end", + "function", + "language", + "license", + "relpath", + "provenance", + "mechanism", + "cve", + "commit_url", +) +HARVEST_LIBRARY = Path("benchmarks/pairs/harvest.py") _DEFAULT_TIER = { "local": "reviewed", "github": "advisory", @@ -99,6 +124,8 @@ class PairCase: known_limit: str | None = None review_tier: str = "advisory" reviewer: str = "" + vendored: bool = True + recipe: tuple[tuple[str, object], ...] = () # pointer pairs carry their harvest recipe instead of excerpts @dataclass(frozen=True) @@ -222,14 +249,20 @@ def _load_catalog_file(path: Path) -> tuple[PairCase, ...]: raise CatalogError(f"pair {name}: unknown split {split!r}; expected train or holdout") known_limit = item.get("known_limit") review_tier, reviewer = _review_tier(item, slice_name, name) + vendored = item.get("vendored", True) is not False + recipe: tuple[tuple[str, object], ...] = () + if vendored: + vuln_file, fixed_file = _resolve(root, str(item["vuln"])), _resolve(root, str(item["fixed"])) + else: + vuln_file, fixed_file, recipe = _pointer_files(item, slice_name, name) cases.append( PairCase( name=name, slice=slice_name, language=str(item.get("language", "")), origin=str(item.get("origin", "")), - vuln_file=_resolve(root, str(item["vuln"])), - fixed_file=_resolve(root, str(item["fixed"])), + vuln_file=vuln_file, + fixed_file=fixed_file, relpath=str(item["relpath"]), expected=expected, min_recall=float(item.get("min_recall", 1.0)), @@ -244,11 +277,90 @@ def _load_catalog_file(path: Path) -> tuple[PairCase, ...]: known_limit=str(known_limit) if known_limit else None, review_tier=review_tier, reviewer=reviewer, + vendored=vendored, + recipe=recipe, ) ) return tuple(cases) +def pair_cache_dir() -> Path: + """Cache root for pointer pairs (Req 10.2); outside the repository unless the operator points it elsewhere.""" + return Path(os.environ.get(PAIR_CACHE_ENV) or DEFAULT_PAIR_CACHE).expanduser() + + +def network_allowed(pointers: bool | None = None) -> bool: + """``pairs --pointers`` for one run, or OPENULTRASAST_PAIRS_NETWORK=1; CI never sets either (Req 10.3, 10.4).""" + if pointers is not None: + return pointers + return os.environ.get(PAIRS_NETWORK_ENV, "") == "1" + + +def pointer_recipe(case: PairCase) -> dict[str, object]: + return dict(case.recipe) + + +def select_vendored(cases: Sequence[PairCase]) -> tuple[PairCase, ...]: + """Tests and gates score vendored pairs only (Req 10.4).""" + return tuple(case for case in cases if case.vendored) + + +def _pointer_files(item: dict[str, object], slice_name: str, name: str) -> tuple[Path, Path, tuple[tuple[str, object], ...]]: + missing = [key for key in POINTER_REQUIRED if not item.get(key)] + if missing: + raise CatalogError(f"pair {name}: vendored = false requires {', '.join(POINTER_REQUIRED)} (missing {', '.join(missing)})") + recipe = {key: item[key] for key in POINTER_FIELDS if key in item} + recipe["name"] = name + recipe["slice"] = slice_name + expected = item.get("expected") + if isinstance(expected, list) and expected and isinstance(expected[0], dict): # header fields live on the expected row + for key in ("function", "mechanism"): + recipe.setdefault(key, expected[0].get(key, "")) + ext = Path(str(item["path"])).suffix or ".c" + folder = (pair_cache_dir() / slice_name / name).resolve() + return folder / f"vuln{ext}", folder / f"fixed{ext}", tuple(sorted(recipe.items())) + + +@lru_cache(maxsize=1) +def _harvest_library() -> object: + spec = importlib.util.spec_from_file_location("openultrasast_pair_harvest", HARVEST_LIBRARY.resolve()) + if spec is None or spec.loader is None: + raise OSError(f"harvest library not found: {HARVEST_LIBRARY}") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def _materialize_pointer(recipe: dict[str, object], cache_root: Path) -> tuple[Path, Path]: + """Harvest both sides of a pointer pair into ``cache_root`` through the shared harvest library (network).""" + module = _harvest_library() + vuln, fixed = module.materialize_pointer( # type: ignore[attr-defined] + recipe, cache_root, slice_name=str(recipe["slice"]) + ) + return Path(vuln), Path(fixed) + + +def _pointer_ready(case: PairCase, *, pointers: bool | None) -> dict[str, object] | None: + """None when the pair can be scored; otherwise the degradation entry (skipped or fetch failed). + + The network switch is consulted first: a warm cache only spares the fetch under ``--pointers``/env, it never turns a + default run into a pointer run (Req 10.3). + """ + if case.vendored: + return None + if not network_allowed(pointers): + return {"stage": "pairs", "reason": "pointer_pair_skipped", "pair": case.name} + if case.vuln_file.is_file() and case.fixed_file.is_file(): + return None + try: + _materialize_pointer(pointer_recipe(case), pair_cache_dir()) + except Exception as exc: # noqa: BLE001 - a failed fetch is a recorded loss, never a crash of the whole run + return {"stage": "pairs", "reason": "pointer_pair_fetch_failed", "pair": case.name, "detail": f"{type(exc).__name__}: {exc}"[:200]} + if case.vuln_file.is_file() and case.fixed_file.is_file(): + return None + return {"stage": "pairs", "reason": "pointer_pair_fetch_failed", "pair": case.name, "detail": "harvest wrote no excerpts"} + + def _review_tier(item: dict[str, object], slice_name: str, name: str) -> tuple[str, str]: """Tier from the row, else from ``reviewer`` (pending -> title, a name -> reviewed), else the slice default.""" reviewer = str(item.get("reviewer", "") or "") @@ -286,11 +398,24 @@ def evaluate_catalog( hunter: HunterScan | None = None, ruleset: tuple[PatternRule, ...] | None = None, inventory_sidecar: bool = True, + pointers: bool | None = None, ) -> PairEvalResult: - """Score every case. ``ruleset`` lets the improve loop score a candidate ledger; ``inventory_sidecar=False`` skips the second scan.""" + """Score every case. ``ruleset`` lets the improve loop score a candidate ledger; ``inventory_sidecar=False`` skips the second scan. + + Pointer pairs (``vendored = false``) are harvested into the cache when ``pointers`` (or OPENULTRASAST_PAIRS_NETWORK=1) allows the + network, otherwise skipped with a ``pointer_pair_skipped`` degradation (Req 10.2, 10.3). + """ + degradations: list[dict[str, object]] = [] + ready: list[PairCase] = [] + for case in cases: + degradation = _pointer_ready(case, pointers=pointers) + if degradation is None: + ready.append(case) + else: + degradations.append(degradation) + cases = tuple(ready) outcomes = tuple(evaluate_pair(case, ruleset=ruleset) for case in cases) scorers: dict[str, dict[str, PairCorpusMetrics]] = {} - degradations: list[dict[str, object]] = [] for slice_name in sorted({case.slice for case in cases if case.slice in OVERLAY_SLICES}): slice_cases = tuple(case for case in cases if case.slice == slice_name) scorers[slice_name] = {"overlay": _metrics(tuple(item for item in outcomes if item.slice == slice_name))} diff --git a/tests/conftest.py b/tests/conftest.py index 7881e94..6b1aab4 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -12,6 +12,13 @@ _HARNESSX_GUARD = ( ) +@pytest.fixture(autouse=True) +def _pointer_pairs_offline(monkeypatch: pytest.MonkeyPatch, tmp_path_factory: pytest.TempPathFactory) -> None: + """Req 10.4: the default test run never touches the network for pointer pairs, whatever the operator's shell exports.""" + monkeypatch.delenv("OPENULTRASAST_PAIRS_NETWORK", raising=False) + monkeypatch.setenv("OPENULTRASAST_PAIR_CACHE", str(tmp_path_factory.mktemp("pair-cache"))) + + @pytest.fixture def assert_cold_of_harnessx(): # type: ignore[no-untyped-def] """Run ``code`` in a fresh interpreter and assert it imported no ``harnessx`` module. diff --git a/tests/test_improve.py b/tests/test_improve.py index 18f38a0..6839ecb 100644 --- a/tests/test_improve.py +++ b/tests/test_improve.py @@ -293,3 +293,32 @@ def test_profile_gate_ignores_title_and_advisory_tiers(tmp_path: Path) -> None: ) assert not outcome.accepted and outcome.reason == "profile_regression:agent" assert outcome.per_profile_before["agent"]["pairs"] == 5.0 # the five title pairs are scored elsewhere, never here + + +def test_profile_gate_scores_vendored_pairs_only(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """Req 10.4 / design: gates call select_vendored; a pointer pair never reaches the improve gate, cached or not.""" + from openultrasast.improve.evolve import evaluate_profiles + from openultrasast.pairs import PairCase + from openultrasast.ruleset import load_ruleset + + monkeypatch.setenv("OPENULTRASAST_PAIRS_NETWORK", "1") + vendored = _pair( + tmp_path, "v", "def h(y):\n return eval(y)\n", "def h(y):\n return y\n", "python-unsafe-eval", "eval", provenance="human" + ) + pointer_src = tmp_path / "cache" / "github" / "p" + pointer_src.mkdir(parents=True) + (pointer_src / "vuln.py").write_text("def h(y):\n return eval(y)\n") + (pointer_src / "fixed.py").write_text("def h(y):\n return y\n") + pointer = PairCase( + **{ + **vendored.__dict__, + "name": "p", + "provenance": "agent", + "vendored": False, + "vuln_file": pointer_src / "vuln.py", + "fixed_file": pointer_src / "fixed.py", + "recipe": (("repo", "o/r"), ("parent", "a"), ("commit", "b"), ("path", "app.py"), ("mode", "name")), + } + ) + profiles = evaluate_profiles([vendored, pointer], load_ruleset()) + assert set(profiles) == {"human"} # the agent pointer pair is scored by `pairs --pointers`, never by the gate diff --git a/tests/test_pair_corpus.py b/tests/test_pair_corpus.py index e527333..b8e5517 100644 --- a/tests/test_pair_corpus.py +++ b/tests/test_pair_corpus.py @@ -20,6 +20,7 @@ from openultrasast.pairs import ( load_datasets, load_pair_catalog, select_slice, + select_vendored, ) @@ -36,7 +37,7 @@ def test_catalog_loads_local_and_github_slices() -> None: assert "openssl-cve-2014-0160" in names assert "firefox-cve-2020-15667" in names assert "chromium-cve-2019-5786" in names - assert all(case.vuln_file.is_file() and case.fixed_file.is_file() for case in cases) + assert all(case.vuln_file.is_file() and case.fixed_file.is_file() for case in select_vendored(cases)) def test_pair_catalog_is_outside_stage1_smoke_gate() -> None: @@ -459,9 +460,9 @@ def test_new_slices_load_offline_and_score_with_profiles() -> None: for slice_name, expected_provenance in (("vibe-py", "human"), ("vfc-js", "human")): cases = select_slice(catalog, slice_name) assert cases, slice_name - assert all(case.vuln_file.is_file() and case.fixed_file.is_file() for case in cases) - assert all(case.license for case in cases), slice_name - assert all(case.provenance == expected_provenance for case in cases), slice_name + assert all(case.vuln_file.is_file() and case.fixed_file.is_file() for case in select_vendored(cases)) + assert all(case.license for case in select_vendored(cases)), slice_name # pointer rows may be unlicensed (Req 10) + assert all(case.provenance == expected_provenance for case in select_vendored(cases)), slice_name assert {case.split for case in cases} <= {"train", "holdout"} header = cases[0].vuln_file.read_text()[:600] assert "Provenance:" in header and "license:" in header @@ -678,7 +679,7 @@ def test_payload_reports_metrics_per_review_tier(tmp_path: Path) -> None: def test_default_catalog_has_the_new_slices_on_disk_and_off_the_stage1_list() -> None: catalog = load_pair_catalog() assert {"vibe-py", "vfc-js", "agent-vfc"} <= {case.slice for case in catalog} # Req 8.1 - assert all(case.vuln_file.is_file() and case.fixed_file.is_file() for case in catalog) + assert all(case.vuln_file.is_file() and case.fixed_file.is_file() for case in select_vendored(catalog)) stage1 = {name for names in LANGUAGE_MANIFESTS.values() for name in names} assert not ({case.name for case in catalog} & stage1) @@ -859,3 +860,106 @@ def test_cpp_qualified_method_range_is_found_by_its_unqualified_label() -> None: assert ranges is not None assert spans_named(ranges, "m") == ((1, 3),) assert function_at(ranges, 2) == "m" + + +# --- task 8.3: pointer pairs (Req 10) ----------------------------------------- + + +_POINTER_ROW = ( + '[[pair]]\nname = "ptr"\nslice = "vibe-py"\nlanguage = "python"\nvendored = false\nrepo = "o/r"\nparent = "aaa"\ncommit = "bbb"\n' + 'path = "app.py"\nmode = "enclosing"\nline = 3\nrelpath = "app.py"\nprovenance = "agent"\nreview_tier = "seeded"\nlicense = ""\n\n' + '[[pair.expected]]\ncwe = "CWE-95"\nclass = "code injection"\npath = "app.py"\nfunction = "f"\n' + 'sink = "eval"\nmechanism = "source_reaches_sink"\n' +) + + +def test_pointer_rows_load_without_excerpts_and_point_into_the_cache(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + from openultrasast.pairs import CatalogError, _load_catalog_file, pointer_recipe, select_vendored + + monkeypatch.setenv("OPENULTRASAST_PAIR_CACHE", str(tmp_path / "cache")) + (tmp_path / "c.toml").write_text(_POINTER_ROW) + (case,) = _load_catalog_file(tmp_path / "c.toml") + assert case.vendored is False + assert case.vuln_file == (tmp_path / "cache" / "vibe-py" / "ptr" / "vuln.py").resolve() + assert pointer_recipe(case)["parent"] == "aaa" and pointer_recipe(case)["mode"] == "enclosing" + assert select_vendored([case]) == () + (tmp_path / "bad.toml").write_text(_POINTER_ROW.replace('parent = "aaa"\n', "")) + with pytest.raises(CatalogError, match="parent"): + _load_catalog_file(tmp_path / "bad.toml") + + +def test_pointer_pairs_skip_with_a_degradation_when_network_is_off(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + from openultrasast.pairs import _load_catalog_file, evaluate_catalog + + monkeypatch.setenv("OPENULTRASAST_PAIR_CACHE", str(tmp_path / "cache")) + monkeypatch.delenv("OPENULTRASAST_PAIRS_NETWORK", raising=False) + (tmp_path / "c.toml").write_text(_POINTER_ROW) + result = evaluate_catalog(_load_catalog_file(tmp_path / "c.toml")) + assert result.outcomes == () + assert {"stage": "pairs", "reason": "pointer_pair_skipped", "pair": "ptr"} in result.degradations + assert not (tmp_path / "cache").exists() + + +def test_pointer_pairs_materialize_into_the_cache_and_score_like_vendored_pairs(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + from openultrasast import pairs as pairs_mod + + monkeypatch.setenv("OPENULTRASAST_PAIR_CACHE", str(tmp_path / "cache")) + monkeypatch.setenv("OPENULTRASAST_PAIRS_NETWORK", "1") + (tmp_path / "c.toml").write_text(_POINTER_ROW) + (case,) = pairs_mod._load_catalog_file(tmp_path / "c.toml") + + def fake_materialize(recipe: dict[str, object], cache_root: Path) -> tuple[Path, Path]: + folder = cache_root / "vibe-py" / str(recipe["name"]) + folder.mkdir(parents=True) + (folder / "vuln.py").write_text("from flask import request\n\ndef f():\n return eval(request.args.get('x'))\n") + (folder / "fixed.py").write_text("def f():\n return 1\n") + return folder / "vuln.py", folder / "fixed.py" + + monkeypatch.setattr(pairs_mod, "_materialize_pointer", fake_materialize) + result = pairs_mod.evaluate_catalog([case]) + assert len(result.outcomes) == 1 and result.outcomes[0].name == "ptr" and result.outcomes[0].detected_vuln + assert not any(item.get("reason", "").startswith("pointer_pair") for item in result.degradations) + # a second evaluation reuses the cache: no fetch + monkeypatch.setattr(pairs_mod, "_materialize_pointer", lambda *_: pytest.fail("cache hit expected")) + assert pairs_mod.evaluate_catalog([case]).outcomes[0].pair_correct + # a failing fetch is a recorded loss, never an exception + (tmp_path / "c2.toml").write_text(_POINTER_ROW.replace('name = "ptr"', 'name = "ptr2"')) + (case2,) = pairs_mod._load_catalog_file(tmp_path / "c2.toml") + + def boom(recipe: dict[str, object], cache_root: Path) -> tuple[Path, Path]: + raise OSError("offline") + + monkeypatch.setattr(pairs_mod, "_materialize_pointer", boom) + failed = pairs_mod.evaluate_catalog([case2]) + assert failed.outcomes == () and any( + item["reason"] == "pointer_pair_fetch_failed" and item["pair"] == "ptr2" for item in failed.degradations + ) + + +def test_cli_pairs_pointers_flag_turns_network_on_for_one_run(monkeypatch: pytest.MonkeyPatch) -> None: + import io + from contextlib import redirect_stdout + + monkeypatch.delenv("OPENULTRASAST_PAIRS_NETWORK", raising=False) + buf = io.StringIO() + with redirect_stdout(buf): + assert main(["pairs", "--slice", "local", "--pointers", "--json"]) == 0 + assert json.loads(buf.getvalue())["overall"]["pairs"] == 3 + assert "OPENULTRASAST_PAIRS_NETWORK" not in __import__("os").environ # the flag is per run, not a persistent env change + + +def test_warm_cache_never_scores_pointer_pairs_when_network_is_off(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """Req 10.3: the cache only spares the fetch under --pointers; it never turns a default run into a pointer run.""" + from openultrasast.pairs import _load_catalog_file, evaluate_catalog + + monkeypatch.setenv("OPENULTRASAST_PAIR_CACHE", str(tmp_path / "cache")) + monkeypatch.delenv("OPENULTRASAST_PAIRS_NETWORK", raising=False) + (tmp_path / "c.toml").write_text(_POINTER_ROW) + (case,) = _load_catalog_file(tmp_path / "c.toml") + case.vuln_file.parent.mkdir(parents=True) + case.vuln_file.write_text("from flask import request\n\ndef f():\n return eval(request.args.get('x'))\n") + case.fixed_file.write_text("def f():\n return 1\n") + result = evaluate_catalog([case]) + assert result.outcomes == () + assert {"stage": "pairs", "reason": "pointer_pair_skipped", "pair": "ptr"} in result.degradations + assert evaluate_catalog([case], pointers=True).outcomes[0].pair_correct # warm cache, network on: no fetch, scored diff --git a/tests/test_pair_harvest.py b/tests/test_pair_harvest.py index 361f67e..978fe40 100644 --- a/tests/test_pair_harvest.py +++ b/tests/test_pair_harvest.py @@ -363,3 +363,64 @@ def test_catalog_gen_fails_loud_without_the_package_and_refuses_to_prune_an_empt monkeypatch.setitem(sys.modules, "openultrasast.semantic.functions", None) with pytest.raises((SystemExit, ImportError)): runpy.run_path(str(Path("benchmarks/pairs/catalog_gen.py")), run_name="catalog_gen_lib_blocked") + + +# --- task 8.3: pointer pairs (Req 10) ----------------------------------------- + + +def test_materialize_pointer_writes_only_into_the_cache_root_and_needs_no_license(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + lib = _lib() + recipe = { + "name": "ptr", + "repo": "o/r", + "parent": "aaa", + "commit": "bbb", + "path": "app.py", + "mode": "enclosing", + "line": 3, + "license": "", + } + with pytest.raises(lib["RecipeError"], match="license"): + lib["validate_recipe"](recipe) + lib["validate_recipe"](recipe, require_license=False) + sources = {"aaa": "import os\n\ndef f(x):\n return eval(x)\n", "bbb": "import os\n\ndef f(x):\n return int(x)\n"} + fetched: list[str] = [] + + def fake_fetch(url: str, timeout: int = 60) -> str: + fetched.append(url) + return sources[url.split("/")[5]] + + lib["fetch_url"] = fake_fetch + lib["harvest_recipe"].__globals__["fetch_url"] = fake_fetch + vuln, fixed = lib["materialize_pointer"](recipe, tmp_path / "cache", slice_name="vibe-py") + assert vuln == tmp_path / "cache" / "vibe-py" / "ptr" / "vuln.py" and fixed.is_file() + assert "def f(x):" in vuln.read_text() and "upstream_start" in vuln.read_text() + assert len(fetched) == 2 and not list(Path("benchmarks/pairs/vibe-py").glob("ptr*")) + + +def test_catalog_gen_emits_pointer_rows_without_excerpts(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + import tomllib + + gen = runpy.run_path(str(Path("benchmarks/pairs/catalog_gen.py")), run_name="catalog_gen_lib") + slice_root = tmp_path / "vibe-py" + slice_root.mkdir() + (slice_root / "recipes.toml").write_text( + '[[recipe]]\nname = "ptr"\nrepo = "o/r"\nparent = "aaa"\ncommit = "bbb"\npath = "app.py"\nfix_path = "trap.py"\n' + 'mode = "enclosing"\n' + 'line = 3\nfix_line = 9\nfunction = "f"\nlanguage = "python"\nrelpath = "app.py"\nlicense = "unlicensed"\ncwe = "CWE-95"\n' + 'class = "code injection"\nmechanism = "source_reaches_sink"\nprovenance = "agent"\nvendored = false\nsplit = "holdout"\n' + ) + gen["ROOT"] = tmp_path + gen["generate"].__globals__["ROOT"] = tmp_path + catalog, count = gen["generate"]("vibe-py") + assert count == 1 + (row,) = tomllib.loads(catalog.read_text())["pair"] + assert row["vendored"] is False and "vuln" not in row and "fixed" not in row + assert row["parent"] == "aaa" and row["fix_path"] == "trap.py" and row["fix_line"] == 9 and row["mode"] == "enclosing" + assert row["expected"][0]["function"] == "f" + from openultrasast.pairs import _load_catalog_file + + monkeypatch.setenv("OPENULTRASAST_PAIR_CACHE", str(tmp_path / "cache")) + (case,) = _load_catalog_file(catalog) + assert case.vendored is False and case.provenance == "agent" + assert not (slice_root / "training" / "manifest.jsonl").read_text().strip() # pointer rows have no vendored files to train on