diff --git a/.kiro/specs/unseen-repo-evaluation/tasks.md b/.kiro/specs/unseen-repo-evaluation/tasks.md index 0ad2cb4..9195df9 100644 --- a/.kiro/specs/unseen-repo-evaluation/tasks.md +++ b/.kiro/specs/unseen-repo-evaluation/tasks.md @@ -77,7 +77,7 @@ Execution rules: - _Requirements: 1.1, 1.3_ - [ ] 2. Sourcing and extraction (design 1.1, 1.3, 1.4, 2, 3; increment 2) -- [ ] 2.1 Advisory candidates and repository filters +- [x] 2.1 Advisory candidates and repository filters - `benchmarks/unseen/source.py`: from GHSA advisory JSON (pip, npm, maven, composer; published 2019 or later) keep those whose CWE maps to a family through `src/openultrasast/model/taxonomy.py` and that link one fix commit with one parent in one repository; attach the OSV first affected version; flag `post_cutoff` when @@ -87,7 +87,7 @@ Execution rules: multi-repository fixes dropped, cutoff flag, licence classing (GPL, AGPL, LGPL, none and source-available go private). - _Requirements: 1.4, 2.1_ -- [ ] 2.2 Vulnerability-introducing range +- [x] 2.2 Vulnerability-introducing range - `benchmarks/unseen/extract.py`: the `introducing` method of design 2.1 (blame of the sink line, or the declaration for an absence bug; absence at V's first parent; no merge or root; at most 50 files and 2,000 lines; OSV version bracket when present; walk-back of at most five steps) and the `last_touch` fallback. Each @@ -96,7 +96,7 @@ Execution rules: - `tests/test_unseen_extract.py` on a synthetic repository built in a temp dir: accepted V; a reformat walked back; a merge rejected; size cap; bracket rejection; fallback chosen when no V passes. - _Requirements: 2.1_ -- [ ] 2.3 Ordinary-commit draw +- [x] 2.3 Ordinary-commit draw - `extract.py`: first-parent, non-merge commits touching an analysed language; the exclusions of design 2.2 (security words of the narrow pattern, reverts, bot and bump commits, manifest- or lock-only, generated or vendored, advisory fix commits, commits touching a known vulnerable function, more than 1,000 source lines, @@ -105,7 +105,7 @@ Execution rules: - Tests: each exclusion; the narrow pattern keeps "query", "token", "session"; allocation and the seed reproduce the same draw; the 40-commit floor. - _Requirements: 2.2, 2.3_ -- [ ] 2.4 Draft manifest and slices +- [x] 2.4 Draft manifest and slices - `benchmarks/unseen/draft.py`: writes `benchmarks/unseen/draft-p1.toml` (permissive entries by name) and `benchmarks/unseen/private/draft-p1.toml` (the rest); the public file holds an opaque id and sha256(URL, commits) per private entry. Seeded shuffle into three slices of 100 repositories, stratified by @@ -334,3 +334,23 @@ Execution rules: Select capability probe. Default store behavior is unchanged; inventory uses LIST/GET only. - The freeze writer must match the canonical TOML digest encoding documented in the pool README and exercised by `test_independent_population.py`; final pool discovery does not trust mutable status. + +### Group 2 implementation evidence (2026-10-04) + +Tasks 2.1–2.4 are implemented offline and independently reviewed. Task 2.5 is +`python -m benchmarks.unseen.draft --root "$PWD" --s3-store s3:// --seed 20260601` +with the documented PATH/PYTHONPATH and GH_TOKEN environment. No networked draw, +label check, group-2 measurement record, guard bisection, or commit was performed. +The group header remains pending its coordinator exits 2.5/2.6. + +The command rebuilds the group-1 used set, resolves renamed/fork repositories, +uses disposable external blobless clones, re-reads licences at extracted heads, +and emits counts/digests only. The README documents injected clients, method bias, +the hard 300-repository floor, and an explicitly unmeasured resource estimate. +Independent review caught and fixed ambiguous parent-function absence and stale +candidate-404 handling; regression tests cover both. + +Validation: requested pytest selection 103 passed, 1 skipped, exit 0 (population-test +output discarded); `ruff check .` exit 0; `ruff format --check .` exit 0 (411 files); +`mypy src/openultrasast` exit 0 (160 source files). The CLI `--help` smoke exited 0. +Source-read instrumentation printed 20,101 bytes for the extraction implementation. diff --git a/benchmarks/unseen/README.md b/benchmarks/unseen/README.md index fdc1ff8..9274f76 100644 --- a/benchmarks/unseen/README.md +++ b/benchmarks/unseen/README.md @@ -60,3 +60,140 @@ The ledger exposes `append(path, row)` and `read(path)`. Rows contain `slice`, ` changed freeze, and reject qualification of an informed slice for the same decision. Keep the published final digest to authenticate history: the chain detects row edits, but cannot independently prove that an entire history was not replaced or truncated. + +## Sourcing and extraction (tasks 2.1–2.4) + +The coordinator's networked task **2.5** is now: + +```sh +export PATH=/home/mc/Source/OpenUltraSAST/.venv/bin:$PATH +export PYTHONPATH=$PWD/src +python -m benchmarks.unseen.draft --root "$PWD" --s3-store s3:// --seed 20260601 \ + --cache /home/mc/ousast-results/plane/unseen-draft --max-hours 8 +``` + +Export `GH_TOKEN` first, or supply it through the root `.env` (exported values win). +The same read-only local/S3 stores, `--pointer-manifest`, and explicit `--known-empty` +options as task 1.5 apply. Missing or unreadable sources fail. The used set is rebuilt +and canonicalized before candidate selection, including redirects and fork networks. +No population v3 file is opened and no guard/bisection is run by this command; that is +reserved for group 3. An existing draft is never overwritten. + +The command writes `--cache/journal/draft-p1.toml` and +`--cache/journal/private-draft-p1.toml` only after all sourcing stages complete. +Both stay outside the repository, with mode 0600. The public-format file contains permissive entries +and, for every private entry, only an opaque ID, slice, private marker, and SHA-256 +of canonical JSON containing its URL and ordered `[base, head]` pairs. Private entries +carry the complete records, including licences, in the private-format file. +Licensing is re-read through GitHub's licence endpoint **at the actual extracted +vulnerability replay head**; the candidate's earlier fix-parent classification is +provisional. An explicit SPDX declaration in the pinned file takes precedence over +the API's classification. Missing/unknown/compound licences remain private. + +Three seeded slices each contain 100 repositories, one vulnerability range and twelve +ordinary changes per repository. Each ecosystem/post-cutoff stratum differs from its +proportional per-slice target by less than one repository. Post-cutoff candidates are +attempted first within each ecosystem. Below 300 eligible repositories the command +exits 1 with counts and does not write a partial draft or relax any selection bar. +The group-2 measured exit and label confirmation remain coordinator tasks 2.5/2.6; +passing implementation tests does not assert that 300 eligible repositories exist. + +Clones and checkpoints default to `results_root()/unseen-draft` (normally +`~/ousast-results/plane/unseen-draft`), overridable by `--cache /outside/path`. +The cache cannot lie inside `--root`. One opaque, disposable clone exists at a time; +it is removed on success, rejection, or failure. `--filter=blob:none --no-checkout` +keeps the commit graph needed for temporal thirds, blame, and release-tag ancestry, +and fetches source blobs on demand. This is a **partial clone**, not a shallow history. +The 500 MiB disk cap is checked after Git commands, including lazy blob fetches; +one in-flight command can transiently exceed it before rejection and cleanup. +Git output is captured, never forwarded. Each repository must read a nonempty source +blob before extraction. Summary instrumentation includes source read operations and +actual bytes, clone count, cumulative/peak disk bytes, wall time, and logical API calls +and HTTP attempts. Zero source bytes or a failed Git read is an instrument failure, +not an eligibility drop. Stdout is a JSON summary of counts/digests only; no names or +URLs enter stdout or a measurement record. Raw per-change data stays in the external journal and manifests. + +Extraction uses Git and syntax boundaries, with no scanner, engine, or model. Python +uses stdlib AST; Java/PHP/JS/TS use conservative named-function brace matching. Ambiguous +or unparseable functions can reject a candidate. The first removed code line per +fix-touched function is the SZZ sink proxy; pure-addition fixes use its declaration. +Normalized text absence, non-root/non-merge ancestry, file/line limits, and the OSV +release bracket are checked before accepting `introducing`. Matching tags are `VERSION` +or `vVERSION`; an absent/ambiguous tag or missing prior release cannot certify the +bracket. After at most five walk-backs, `last_touch` uses `git log -L` with the resolved +numeric function extent (also works where Git lacks a language-specific function driver). +Method, bias, fallback reasons, head function/range, files/hunks, family, advisory, and +cutoff flag are recorded. Task 2.6 must review every third `introducing` label. + +All reviewed advisories are enumerated to exclude **all** known repository fixes from +ordinary changes, including old advisories and advisories outside the four candidate +ecosystems. Fix URLs are canonicalized before indexing so renames cannot hide them. +The ordinary filter uses only the design's narrowed security pattern. It counts +rejections (including oversized commits and repositories failing the 40-commit floor), +checks both sides for touches to known vulnerable functions, and inspects generated +headers and inherited `.gitattributes`. Sampling allocates twelve draws proportionally +across three history-time thirds and four source-size buckets, with one per third. + +For offline tests, inject GitHub/OSV clients, the clone context, and `UsedSet` into +`draft.run`; HTTP transports and Git subprocess runners also accept fakes. Tests use +sanitized REST-shaped data and a temporary synthetic local Git history. No implementation +test contacts GitHub, OSV, a model, or a cluster. HTTP retries are bounded to five attempts; +rate-limit reset/Retry-After waits and transient 5xx backoff are tested with fake clocks +and sleepers. A stale candidate 404 is counted; other service failures fail the run. + +### Checkpoint and resume + +The command above uses journal directory +`/home/mc/ousast-results/plane/unseen-draft/journal`. Run the same command to resume. +`--max-hours` is a per-invocation budget; changing it does not invalidate checkpoints. +Create `journal/STOP` to stop between units, then remove it before resuming. A page, +repository resolution, candidate decision, or repository extraction already running +finishes first, so the time budget is not a hard timeout on a large repository. +Clean stops exit 0 with `status: stopped`; failures and shortfalls exit 1. Completion +exits 0 with `status: draft`. No partial manifest is published. + +`inputs.json` binds the seed, source-labelled used-set digest, known-empty declarations, +clone cap, and design fingerprints (sourcing/extraction/eligibility/publication code and +family taxonomy). A mismatch refuses resume and prints only the changed parameter +labels. Use a new external cache for a deliberately changed design or population. +The used set is rebuilt each invocation; repository resolutions, including 404s, are +reused from the journal. Live GitHub metadata for unfinished units can still change; +completed pages and decisions are the durable sourcing snapshot. + +Each completed unit is appended to its stage JSONL and fsynced before work advances: +`advisories.jsonl` (pages), `resolution.jsonl` (alias/fork metadata), +`eligibility.jsonl` (candidate decisions), `extraction.jsonl` (repository/advisory +results, vulnerable range, ordinary draws, pinned licence, method and counts), and +`completion.jsonl` (final manifest digest). A lock permits only one writer. +`progress.json` is atomically rewritten after every unit with stage counts, last unit, +last observed GitHub rate-limit headers, and cumulative active elapsed seconds. +These files may contain identities and must stay outside version control. Stdout and +repository records contain only counts, digests and fixed labels. + +Replaying completed units makes no GitHub/OSV or clone calls. Final manifests are +reconstructed deterministically from journal results; an interrupted publication can +be retried, but a different existing manifest is refused. Completed JSONL rows are +never rewritten. Malformed or torn records fail closed instead of being interpreted +as completed work. Tests interrupt each stage with fakes, then compare resumed and +uninterrupted manifests byte for byte. + +### Planning estimate for task 2.5 (not measured) + +Let A be all reviewed advisories, R distinct fix-linked repository identities, U used +identities not already cached, C candidate advisories passing early filters, and K +repositories surviving extraction. GitHub requests are approximately +`ceil(A/100) + R + U + 2*C + K` (enumeration, canonical repositories, fix metadata, +provisional licence, final licence), plus retries. OSV requests are at most C. Git +fetches and lazy blob transfers are additional network operations, not GitHub API calls. + +For a planning scenario of A=40,000, R=15,000, U≈1,000, C=600–1,500, K=300: +about **18,000–20,000 GitHub calls** and **600–1,500 OSV calls**. The all-advisory index +makes this substantially more than 300 repository lookups. At an assumed 5,000-request +hourly allowance, allow several rate-limit windows. Actual token limits can differ. +For 300–1,000 attempted clones averaging 20–100 MB after source hydration, plan roughly +**6–100 GB transferred**, with one clone occupying about 500 MiB at most between commands +plus the retained journal and manifests after completion. Transfer volume is an estimate; disk bytes +are only a proxy. A working planning range is **8–48 hours**, potentially longer on +large histories: every first-parent commit is inspected and lazy blobs can dominate. +These are assumptions, not evidence from a network run. The command reports actual +counts and wall time; task 2.5 establishes the measured estimate. diff --git a/benchmarks/unseen/draft.py b/benchmarks/unseen/draft.py new file mode 100644 index 0000000..6597699 --- /dev/null +++ b/benchmarks/unseen/draft.py @@ -0,0 +1,472 @@ +"""Coordinator-only sourcing command: python -m benchmarks.unseen.draft --help. + +All network and git operations are injected at the run() boundary. Only counts, +digests and fixed failure codes go to stdout; identity-bearing output stays in the +private journal outside the repository. No population guard is run here. +""" + +from __future__ import annotations + +import argparse +import copy +import hashlib +import json +import os +import random +import subprocess +import tempfile +import time +from collections import Counter, defaultdict +from contextlib import contextmanager +from dataclasses import asdict +from pathlib import Path + +from . import eligibility, extract, source +from .extract import Git, InstrumentFailure +from .journal import Journal, ResolvedGitHub, ResumeMismatch, Stopped, atomic, digest, external + + +def canonical(value) -> bytes: + return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode() + + +def private_digest(row: dict) -> str: + return hashlib.sha256(canonical({"url": row["url"], "commits": [[c["base"], c["head"]] for c in row["changes"]]})).hexdigest() + + +def assign_slices(rows: list[dict], *, seed: int) -> list[dict]: + if len(rows) != 300 or len({r["repository"] for r in rows}) != 300: + raise ValueError("pool_size") + for row in rows: + kinds = Counter(c["kind"] for c in row["changes"]) + if kinds != {"vulnerability": 1, "ordinary": 12}: + raise ValueError("change_count") + if row["license_class"] not in {"permissive", "private"}: + raise ValueError("license_class") + if row["license_class"] == "permissive" and row["license_spdx"] not in source.PERMISSIVE: + raise ValueError("license_class") + rng = random.Random(seed) + groups = defaultdict(list) + for row in sorted(rows, key=lambda r: r["repository"]): + groups[row["ecosystem"], row["post_cutoff"]].append(row) + sizes = Counter({1: 0, 2: 0, 3: 0}) + assigned = [] + for key in sorted(groups): + group = groups[key] + rng.shuffle(group) + quotient, remainder = divmod(len(group), 3) + # Give each stratum its floor on all slices, then fill globally least-full slices. + quotas = {i: quotient for i in sizes} + order = list(sizes) + rng.shuffle(order) + order.sort(key=lambda i: sizes[i]) + for i in order[:remainder]: + quotas[i] += 1 + offset = 0 + for slice_id in sorted(quotas): + for row in group[offset : offset + quotas[slice_id]]: + digest = private_digest(row) + assigned.append({**row, "id": "r-" + digest[:24], "slice": slice_id}) + sizes[slice_id] += quotas[slice_id] + offset += quotas[slice_id] + if set(sizes.values()) != {100}: + raise ValueError("slice_size") + return sorted(assigned, key=lambda r: (r["slice"], r["id"])) + + +def toml(value) -> str: + """Inline TOML values keep nested change records structured without a new dependency.""" + if isinstance(value, bool): + return "true" if value else "false" + if isinstance(value, str): + return json.dumps(value, ensure_ascii=False) + if isinstance(value, int): + return str(value) + if isinstance(value, list): + return "[" + ", ".join(toml(v) for v in value) + "]" + if isinstance(value, dict): + return "{" + ", ".join(json.dumps(k) + " = " + toml(v) for k, v in sorted(value.items()) if v is not None) + "}" + raise ValueError("unsupported_manifest_type") + + +def manifest(rows: list[dict], seed: int) -> str: + lines = ['status = "draft"', 'pool = "p1"', f"seed = {seed}"] + for row in rows: + lines.append("\n[[repository]]") + lines.extend(f"{key} = {toml(value)}" for key, value in sorted(row.items()) if value is not None) + return "\n".join(lines) + "\n" + + +def write_draft(root: Path, assigned: list[dict], *, seed: int) -> tuple[Path, Path]: + public_path = root / "benchmarks/unseen/draft-p1.toml" + private_path = root / "benchmarks/unseen/private/draft-p1.toml" + if public_path.exists() or private_path.exists(): + raise FileExistsError("draft_exists") + public, private = [], [] + for row in assigned: + if row["license_class"] == "permissive": + public.append(row) + else: + private.append(row) + public.append({"id": row["id"], "sha256": private_digest(row), "slice": row["slice"], "private": True}) + private_path.parent.mkdir(parents=True, exist_ok=True, mode=0o700) + # Exclusive creates protect drafts awaiting human label review. Private first: never + # publish a pointer whose private manifest failed to write. + with private_path.open("x") as handle: + os.chmod(private_path, 0o600) + handle.write(manifest(private, seed)) + with public_path.open("x") as handle: + handle.write(manifest(public, seed)) + return public_path, private_path + + +class Clones: + """One disposable blobless clone at a time; no name in cache paths or subprocess output.""" + + def __init__(self, root: Path, cache: Path, *, runner=None, limit_bytes=500 * 1024 * 1024): + self.cache = external(root, cache) + self.runner = runner or subprocess.run + self.limit_bytes = limit_bytes + self.counts = Counter() + + @contextmanager + def open(self, url: str): + self.cache.mkdir(parents=True, exist_ok=True) + started = time.monotonic() + with tempfile.TemporaryDirectory(prefix="clone-", dir=self.cache) as directory: + path = Path(directory) + git = Git(path, runner=self.runner, limit_bytes=self.limit_bytes) + try: + # Full commit graph is needed for temporal thirds and OSV brackets; blobs + # are fetched lazily. This is a partial clone, not a truncated history. + done = self.runner( + ["git", "-c", "credential.helper=", "clone", "--quiet", "--filter=blob:none", "--no-checkout", url, str(path)], + stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + check=False, + timeout=180, + env={**os.environ, "GIT_TERMINAL_PROMPT": "0"}, + ) + if done.returncode: + raise InstrumentFailure("clone_failed") + self.counts["clones"] += 1 + if git.disk_bytes() > self.limit_bytes: + raise InstrumentFailure("clone_size_cap") + yield git + finally: + size = git.disk_bytes() + self.counts["clone_bytes"] += size + self.counts["peak_clone_bytes"] = max(self.counts["peak_clone_bytes"], size) + self.counts["source_bytes_read"] += git.bytes_read + self.counts["source_files_read"] += git.files_read + self.counts["clone_wall_seconds"] += round(time.monotonic() - started, 3) + + +def prove_source(git, revision: str) -> dict: + files = git.run("ls-tree", "-r", "--name-only", revision).splitlines() + for path in files: + if Path(path).suffix in extract.LANGUAGES and not extract.generated_path(path): + text = git.blob(revision, path) + if text: + return {"files": 1, "bytes": getattr(git, "last_blob_bytes", len(text.encode()))} + raise InstrumentFailure("zero_source_bytes") + + +def advisory_index(rows: list[dict], github) -> dict[str, set[str]]: + """Include ALL enumerated fixes, even old, unmapped, multi-fix and other ecosystems. + + Canonicalize link names as well, so an old advisory URL survives repository renames. + A 404 is not treated as proof that an advisory is unrelated to a selected repository. + """ + index = defaultdict(set) + for row in rows: + for name, sha in source.fix_links(row): + try: + resolved = eligibility.resolve(name, github) + except eligibility.RepositoryNotFound: + index[name].add(sha) + continue + index[resolved.full_name].add(sha) + return dict(index) + + +def repository_changes(git, candidate, fixes: set[str], *, seed: int): + proof = prove_source(git, candidate.parent) + change = extract.introducing(git, candidate) + known = set() + expanded = set() + for fix in sorted(fixes | {candidate.fix}): + # Explicitly fetch detached advisory commits unavailable on default-branch history. + # All git including this possible network call still goes through the injected client. + git.run("fetch", "--quiet", "--filter=blob:none", "origin", fix) + full = git.run("rev-parse", fix).strip() + expanded.add(full) + parents = git.parents(full) + for parent in parents: + sites = extract.fix_sites(git, full, parent, global_ok=True) + known.update((site["path"], site["function"]) for site in sites) + rows = extract.ordinary_history(git, "HEAD") + ordinary, exclusions = extract.draw_ordinary(rows, expanded, known, seed=seed) + return [change, *ordinary], exclusions, proof + + +def inputs(used, seed, known_empty, clones): + from openultrasast.model.taxonomy import DEFAULT_FAMILIES_PATH + + parameters = { + "seed": seed, + "used_set_digest": digest({label: sorted(names) for label, names in used.sources.items()}), + "known_empty": sorted(known_empty), + "clone_limit_bytes": getattr(clones, "limit_bytes", 500 * 1024 * 1024), + "journal_version": 1, + } + # Hardcoded design thresholds and algorithms are covered as well as taxonomy data. + for name in ("draft", "source", "extract", "eligibility", "journal"): + parameters["design_" + name] = hashlib.sha256(Path(__file__).with_name(name + ".py").read_bytes()).hexdigest() + parameters["design_taxonomy"] = hashlib.sha256(DEFAULT_FAMILIES_PATH.read_bytes()).hexdigest() + return parameters + + +def enumerate_advisories(github, journal): + page, records = 1, {} + while True: + key = str(page) + if key not in journal.rows["advisories"]: + journal.boundary() + result = github.advisory_page(page) + journal.put("advisories", key, result) + result = journal.rows["advisories"][key] + records.update((row["ghsa_id"], row) for row in result["rows"]) + if result["done"]: + break + page += 1 + if not records: + raise InstrumentFailure("empty_advisories") + return list(records.values()) + + +def decide(row, github, osv, used, selected_networks): + try: + links = source.fix_links(row) + if len(links) != 1: + raise source.Rejected("fix_links") + name, _ = next(iter(links)) + resolved = eligibility.resolve(name, github) + rejected_by = eligibility.rejection(resolved, used, set(selected_networks)) + if rejected_by: + raise source.Rejected("used_" + rejected_by) + candidate = source.candidate(row, github, osv) + return {"candidate": asdict(candidate), "network": resolved.network} + except source.Rejected as exc: + return {"rejected": str(exc)} + except eligibility.RepositoryNotFound: + return {"rejected": "repository_or_commit_not_found"} + + +def extract_repository(candidate, github, clones, fixes, seed): + before = dict(clones.counts) + try: + with clones.open(candidate.url) as git: + git.run("fetch", "--quiet", "--filter=blob:none", "origin", candidate.fix) + changes, exclusions, proof = repository_changes(git, candidate, fixes, seed=seed) + pinned = changes[0]["head"] + license_record = github.get_license(candidate.repository, pinned) + spdx, license_class = source.license_info(license_record) + entry = { + **asdict(candidate), + "license_spdx": spdx, + "license_class": license_class, + "license_ref": pinned, + "license_path": license_record.get("path", ""), + "changes": changes, + "instrument": proof, + } + result = {"entry": entry, "ordinary_exclusions": exclusions} + except source.Rejected as exc: + result = {"rejected": str(exc), "ordinary_exclusions": getattr(exc, "counts", {})} + except eligibility.RepositoryNotFound: + result = {"rejected": "repository_or_commit_not_found", "ordinary_exclusions": {}} + result["counts"] = {key: value - before.get(key, 0) for key, value in clones.counts.items()} + return result + + +def publish(journal, assigned, seed): + """Idempotent publication from completed journal data, never inside the checkout.""" + public, private = [], [] + for row in assigned: + if row["license_class"] == "permissive": + public.append(row) + else: + private.append(row) + public.append({"id": row["id"], "sha256": private_digest(row), "slice": row["slice"], "private": True}) + for name, rows in (("private-draft-p1.toml", private), ("draft-p1.toml", public)): + path = journal.path / name + data = manifest(rows, seed).encode() + if path.exists(): + if path.read_bytes() != data: + raise ValueError("draft_exists") + else: + atomic(path, data) + + +def run( + root: Path, + *, + github, + osv, + clones, + used: eligibility.UsedSet, + seed: int, + known_empty=None, + journal_path=None, + max_hours=None, + clock=None, +) -> dict: + started = time.monotonic() + report = {"status": "sourcing", "seed": seed, "candidates": {}, "rejections": {}, "ordinary_exclusions": {}} + counts, rejected, ordinary_counts = Counter(), Counter(), Counter() + checkpoint = None + try: + known_empty = set(known_empty or ()) + used.report(known_empty) + cache = getattr(clones, "cache", root.parent / (root.name + "-cache")) + path = external(root, journal_path or cache / "journal") + checkpoint = Journal( + path, max_hours=max_hours, clock=clock, rate_limit=lambda: getattr(getattr(github, "transport", None), "rate_limit", {}) + ) + with checkpoint.open(inputs(used, seed, known_empty, clones)): + checkpoint.boundary() + api = ResolvedGitHub(github, checkpoint) + used = copy.deepcopy(used) # canonicalization must not mutate the resume input + eligibility.canonicalize_used(used, api) + report["used_set"] = used.report(known_empty) + rows = enumerate_advisories(api, checkpoint) + report["advisories"] = len(rows) + fixes_by_repo = advisory_index(rows, api) + selected_networks, entries = set(), [] + for row in source.ordered_advisories(rows, seed): + checkpoint.boundary() + key = row["ghsa_id"] + counts[source.ecosystem(row)] += 1 + if key not in checkpoint.rows["eligibility"]: + decision = decide(row, api, osv, used, selected_networks) + checkpoint.put("eligibility", key, decision) + decision = checkpoint.rows["eligibility"][key] + if "rejected" in decision: + rejected[decision["rejected"]] += 1 + continue + candidate = source.Candidate(**decision["candidate"]) + # Multiple advisories for one repository can have different fix/range inputs. + extraction_key = digest({"repository": candidate.repository, "advisory": key, "fix": candidate.fix}) + if extraction_key not in checkpoint.rows["extraction"]: + checkpoint.boundary() + result = extract_repository(candidate, api, clones, fixes_by_repo.get(candidate.repository, set()), seed) + checkpoint.put("extraction", extraction_key, result) + result = checkpoint.rows["extraction"][extraction_key] + ordinary_counts.update(result["ordinary_exclusions"]) + if "rejected" in result: + rejected[result["rejected"]] += 1 + continue + selected_networks.add(decision["network"]) + entries.append(result["entry"]) + if len(entries) == 300: + break + report.update( + { + "repositories": len(entries), + "candidates": dict(counts), + "rejections": dict(rejected), + "ordinary_exclusions": dict(ordinary_counts), + } + ) + checkpoint.boundary() + if len(entries) != 300: + report["status"] = "insufficient_repositories" + checkpoint.progress(report["status"]) + return report + assigned = assign_slices(entries, seed=seed) + completion = {"manifest_sha256": digest(assigned), "repositories": len(entries)} + if "all" not in checkpoint.rows["completion"]: + checkpoint.put("completion", "all", completion) + elif checkpoint.rows["completion"]["all"] != completion: + raise ValueError("journal_completion_mismatch") + checkpoint.boundary() + publish(checkpoint, assigned, seed) + report.update( + { + "status": "draft", + "changes": 3900, + "slices": 3, + "license_split": dict(Counter(e["license_class"] for e in entries)), + "methods": dict(Counter(e["changes"][0]["method"] for e in entries)), + "post_cutoff": sum(e["post_cutoff"] for e in entries), + **completion, + } + ) + checkpoint.progress("draft") + return report + except ResumeMismatch as exc: + report.update({"status": "resume_refused", "changed_parameters": exc.parameters}) + return report + except Stopped as exc: + report.update({"status": "stopped", "reason": str(exc)}) + return report + except Exception as exc: + report.update({"status": "instrument_failure", "error_type": type(exc).__name__}) + return report + finally: + report.update({"candidates": dict(counts), "rejections": dict(rejected), "ordinary_exclusions": dict(ordinary_counts)}) + report["instrument"] = dict(clones.counts) + report["wall_seconds"] = round(time.monotonic() - started, 3) + report["github_calls"] = github.calls + report["osv_calls"] = osv.calls + report["github_http_attempts"] = getattr(getattr(github, "transport", None), "attempts", github.calls) + report["osv_http_attempts"] = getattr(getattr(osv, "transport", None), "attempts", osv.calls) + + +def main(argv: list[str] | None = None) -> int: + from openultrasast import config + from openultrasast.plane import memory + from openultrasast.plane.reconciler import results_root + + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--root", type=Path, default=Path.cwd()) + parser.add_argument("--cache", type=Path) + parser.add_argument("--max-hours", type=float, help="Stop between units after this many hours per invocation") + parser.add_argument("--seed", type=int, default=20260601) + parser.add_argument("--local-store") + parser.add_argument("--s3-store", default="s3://") + parser.add_argument("--pointer-manifest", type=Path, action="append", default=[]) + parser.add_argument("--known-empty", choices=(*eligibility.SOURCES, "memory_local", "memory_s3"), action="append", default=[]) + args = parser.parse_args(argv) + try: + config.load_dotenv(args.root / ".env") + # Refuse an existing draft before doing any network work. + if any((args.root / p).exists() for p in ("benchmarks/unseen/draft-p1.toml", "benchmarks/unseen/private/draft-p1.toml")): + raise FileExistsError("draft_exists") + clones = Clones(args.root, args.cache or results_root() / "unseen-draft") + stores = { + "local": memory.open_store(args.local_store or "file://" + str(results_root() / "memory"), read_only=True), + "s3": memory.open_store(args.s3_store, read_only=True), + } + used = eligibility.build_used(args.root, stores=stores, pointer_manifests=args.pointer_manifest) + report = run( + args.root, + github=source.GitHub(os.environ.get("GH_TOKEN", "")), + osv=source.OSV(), + clones=clones, + used=used, + seed=args.seed, + known_empty=set(args.known_empty), + journal_path=clones.cache / "journal", + max_hours=args.max_hours, + ) + except Exception as exc: + report = {"status": "instrument_failure", "error_type": type(exc).__name__} + print(json.dumps(report, sort_keys=True)) + return 0 if report["status"] in {"draft", "stopped"} else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/benchmarks/unseen/extract.py b/benchmarks/unseen/extract.py new file mode 100644 index 0000000..0e7a5a5 --- /dev/null +++ b/benchmarks/unseen/extract.py @@ -0,0 +1,474 @@ +"""Git-only change extraction; clients and subprocess execution are injectable. + +No scanner, engine or model runs here. Function boundaries are syntax, not a security +judgement: the first removed code line of a fix is the SZZ sink proxy. Ambiguous sites +are dropped, and the recorded SZZ bias remains subject to task 2.6's label check. +""" + +from __future__ import annotations + +import ast +import json +import re +import subprocess +from pathlib import Path + +from .source import Candidate, Rejected + +LANGUAGES = {".py", ".js", ".jsx", ".mjs", ".cjs", ".ts", ".tsx", ".php", ".java"} +HUNK = re.compile(r"^@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@", re.M) +SHA = re.compile(r"[a-f0-9]{40}") +BIAS = { + "introducing": "SZZ last-writer proxy: refactors, moves and reformats can misidentify the introduction in either direction.", + "last_touch": "Pessimistic lower bound: the flaw may already exist in base, so a delta hook can correctly stay quiet.", +} + + +class InstrumentFailure(ValueError): + """A failed read or command must never masquerade as an empty history.""" + + +class Git: + def __init__(self, path: Path, *, runner=None, limit_bytes: int | None = None): + self.path = path + self.runner = runner or subprocess.run + self.bytes_read = 0 + self.files_read = 0 + self.last_blob_bytes = 0 + self.limit_bytes = limit_bytes + + def run(self, *args: str, allowed=(0,)) -> str: + try: + done = self.runner( + ["git", "-c", "core.quotePath=false", "-C", str(self.path), *args], + stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + timeout=180, + check=False, + ) + except Exception: + raise InstrumentFailure("git_launch_failed") from None + if done.returncode not in allowed: + raise InstrumentFailure("git_exit_" + str(done.returncode)) + if self.limit_bytes is not None and self.disk_bytes() > self.limit_bytes: + raise InstrumentFailure("clone_size_cap") + self.last_blob_bytes = len(done.stdout) + return done.stdout.decode("utf-8", errors="replace") + + def disk_bytes(self) -> int: + return sum(p.stat().st_size for p in self.path.rglob("*") if p.is_file() and not p.is_symlink()) + + def parents(self, sha: str) -> list[str]: + return self.run("rev-list", "--parents", "-n", "1", sha).split()[1:] + + def blob(self, sha: str, path: str) -> str | None: + # Missing is established by a successful tree read, never by swallowing a show error. + if not self.run("ls-tree", sha, "--", path).strip(): + return None + text = self.run("show", f"{sha}:{path}") + if Path(path).suffix in LANGUAGES: + self.files_read += 1 + self.bytes_read += self.last_blob_bytes + return text + + def stats(self, base: str, head: str) -> list[tuple[str, int, int]]: + rows = [] + for line in self.run("diff", "--no-renames", "--numstat", base, head, "--").splitlines(): + added, removed, path = line.split("\t", 2) + rows.append((path, int(added) if added.isdigit() else 0, int(removed) if removed.isdigit() else 0)) + return rows + + def hunks(self, base: str, head: str, path: str) -> list[tuple[int, int, int, int]]: + diff = self.run("diff", "--no-ext-diff", "--no-renames", "-U0", base, head, "--", path) + return [(int(m[1]), int(m[2] or 1), int(m[3]), int(m[4] or 1)) for m in HUNK.finditer(diff)] + + def ancestor(self, older: str, newer: str) -> bool: + # rev-list avoids interpreting a command failure as a negative ancestry result. + return not self.run("rev-list", "--max-count=1", older, "--not", newer).strip() + + def blame(self, revision: str, path: str, line: int) -> tuple[str, str, int]: + text = self.run("blame", "-w", "-M", "-C", "--porcelain", f"-L{line},{line}", revision, "--", path) + header = text.splitlines()[0].split() + if len(header) < 3 or not SHA.fullmatch(header[0]): + raise InstrumentFailure("invalid_blame") + filename = next((row[9:] for row in text.splitlines() if row.startswith("filename ")), path) + if filename.startswith('"'): + filename = json.loads(filename) + return header[0], filename, int(header[1]) + + +def functions(path: str, text: str) -> list[tuple[str, int, int]]: + """Conservative syntax boundaries. No scanner import; unresolved sites are rejected.""" + if Path(path).suffix == ".py": + try: + tree = ast.parse(text) + except SyntaxError: + raise Rejected("function_parse") from None + return [ + (n.name, n.lineno, n.end_lineno or n.lineno) for n in ast.walk(tree) if isinstance(n, ast.FunctionDef | ast.AsyncFunctionDef) + ] + # Blank comments and quoted strings before brace matching. Keep newlines/offsets. + masked = re.sub( + r'/\*.*?\*/|//[^\n]*|\#[^\n]*|"(?:\\.|[^"\\])*"|\'(?:\\.|[^\'\\])*\'|`(?:\\.|[^`\\])*`', + lambda m: "".join("\n" if c == "\n" else " " for c in m[0]), + text, + flags=re.S, + ) + declaration = re.compile( + r"\bfunction\s*&?\s*(\w+)\s*\([^)]*\)\s*(?::[^\n{]+)?\s*\{|" + r"\b(?:const|let|var)\s+(\w+)\s*=\s*(?:async\s*)?(?:\([^)]*\)|\w+)\s*=>\s*\{|" + r"\b([\w$]+)\s*\([^;{}]*\)\s*(?:throws\s+[\w., ]+)?\s*\{" + ) + found = [] + for match in declaration.finditer(masked): + name = next(g for g in match.groups() if g) + if name in {"if", "while", "for", "switch", "catch", "with", "synchronized"}: + continue + depth = 1 + end = match.end() + while end < len(masked) and depth: + depth += (masked[end] == "{") - (masked[end] == "}") + end += 1 + if depth == 0: + found.append((name, masked.count("\n", 0, match.start()) + 1, masked.count("\n", 0, end) + 1)) + return found + + +def enclosing(path: str, text: str, line: int, *, global_ok=False) -> tuple[str, int, int] | None: + spans = [s for s in functions(path, text) if s[1] <= line <= s[2]] + if spans: + return min(spans, key=lambda s: s[2] - s[1]) + return ("", 1, max(1, len(text.splitlines()))) if global_ok else None + + +def normalized(text: str) -> str: + return re.sub(r"\s+", "", text) + + +def site_at(git, revision: str, path: str, name: str): + text = git.blob(revision, path) + if text is None: + return None + spans = [("", 1, max(1, len(text.splitlines())))] if name == "" else [s for s in functions(path, text) if s[0] == name] + if len(spans) != 1: + return None + return text, spans[0] + + +def fix_sites(git, fix: str, parent: str, *, global_ok=False) -> list[dict]: + sites = [] + for path, _, _ in git.stats(parent, fix): + if Path(path).suffix not in LANGUAGES: + continue + text = git.blob(parent, path) + if text is None: + continue + lines = text.splitlines() + for start, count, _, _ in git.hunks(parent, fix, path): + for line in range(max(1, start), max(1, start) + max(1, count)): + if line > len(lines): + continue + span = enclosing(path, text, line, global_ok=global_ok) + if span is None: + continue + sink_line = line if count else span[1] + sink = lines[sink_line - 1] + if not sink.strip() or sink.lstrip().startswith(("#", "//", "/*", "*")): + continue + key = (path, span[0]) + if any((s["path"], s["function"]) == key for s in sites): + continue + sites.append({"path": path, "function": span[0], "line": sink_line, "text": sink, "absence": count == 0, "span": span}) + return sites + + +def version_bracket(git, candidate: Candidate, revision: str) -> bool: + if not candidate.first_affected: + return True + tags = git.run("tag", "--sort=version:refname").splitlines() + matching = [t for t in tags if t.removeprefix("v") == candidate.first_affected] + if len(matching) != 1: + return False + affected = matching[0] + earlier = [t for t in tags[: tags.index(affected)] if git.ancestor(t, affected)] + return bool(earlier) and git.ancestor(revision, affected) and not git.ancestor(revision, earlier[-1]) + + +def change_files(git, base: str, head: str) -> list[dict]: + return [ + {"path": path, "head_lines": [[max(1, new), max(1, new + max(count, 1) - 1)] for _, _, new, count in git.hunks(base, head, path)]} + for path, _, _ in git.stats(base, head) + ] + + +def introducing(git, candidate: Candidate) -> dict: + from collections import Counter + + if git.parents(candidate.fix) != [candidate.parent]: + raise Rejected("fix_parents") + sites = fix_sites(git, candidate.fix, candidate.parent, global_ok=candidate.family == "config_secrets") + reasons = Counter() + + def record(method, base, head, path, name): + resolved = site_at(git, head, path, name) + if resolved is None: + return None + return { + "kind": "vulnerability", + "base": base, + "head": head, + "method": method, + "family": candidate.family, + "function": name, + "path": path, + "function_lines": list(resolved[1][1:]), + "files": change_files(git, base, head), + "post_cutoff": candidate.post_cutoff, + "advisory": candidate.advisory, + "bias": BIAS[method], + "fallback_reasons": dict(reasons), + } + + for site in sites: + revision, path, line = candidate.parent, site["path"], site["line"] + for _ in range(6): # initial blame plus at most five walk-back steps + blamed, blamed_path, blamed_line = git.blame(revision, path, line) + parents = git.parents(blamed) + if len(parents) != 1: + reasons["merge_or_root"] += 1 + break + base = parents[0] + head_site = site_at(git, blamed, blamed_path, site["function"]) + if head_site is None: + reasons["unresolved_function"] += 1 + break + parent_text = git.blob(base, blamed_path) + if parent_text is not None and site["function"] != "": + matching = [span for span in functions(blamed_path, parent_text) if span[0] == site["function"]] + if len(matching) > 1: + reasons["ambiguous_parent"] += 1 + break + old_site = site_at(git, base, blamed_path, site["function"]) + occurrences = [] + if old_site: + text, (_, start, end) = old_site + occurrences = [ + i for i, value in enumerate(text.splitlines(), 1) if start <= i <= end and normalized(value) == normalized(site["text"]) + ] + if occurrences: + reasons["text_present"] += 1 + revision, path, line = base, blamed_path, occurrences[0] + continue + stats = git.stats(base, blamed) + if len(stats) > 50 or sum(a + d for _, a, d in stats) > 2000: + reasons["size_cap"] += 1 + break + if not version_bracket(git, candidate, blamed): + reasons["version_bracket"] += 1 + break + result = record("introducing", base, blamed, blamed_path, site["function"]) + if result: + return result + else: + reasons["walkback_limit"] += 1 + for site in sites: + # Numeric function extent implements log -L for all five languages, including + # languages for which Git has no built-in :function: driver. + _, start, end = site["span"] + history = git.run("log", "--format=%H", "--no-patch", f"-L{start},{end}:{site['path']}", candidate.parent).splitlines() + if not history: + continue + latest = history[0] + parents = git.parents(latest) + if len(parents) != 1: + continue + result = record("last_touch", parents[0], candidate.parent, site["path"], site["function"]) + if result: + return result + raise Rejected("no_range") + + +SECURITY = re.compile(r"secur|cve|ghsa|vuln|inject|xss|csrf|ssrf|exploit|sanitiz|escap", re.I) +DEPENDENCY = re.compile(r"\bbump\b|\bupgrade .* from\b", re.I) +MANIFESTS = { + "package.json", + "package-lock.json", + "yarn.lock", + "pnpm-lock.yaml", + "composer.json", + "composer.lock", + "pom.xml", + "build.gradle", + "build.gradle.kts", + "gradle.lockfile", + "pyproject.toml", + "setup.py", + "setup.cfg", + "Pipfile", + "Pipfile.lock", + "poetry.lock", + "uv.lock", +} + + +def generated_path(path: str) -> bool: + return bool({"vendor", "node_modules", "dist"} & set(Path(path).parts)) or path.endswith(".min.js") + + +def manifest(path: str) -> bool: + name = Path(path).name + return name in MANIFESTS or name.endswith(".lock") or bool(re.fullmatch(r"requirements[^/]*\.txt", name)) + + +def ordinary_rejection(row: dict, fixes: set[str], known_functions: set[tuple[str, str]]) -> str | None: + if len(row["parents"]) != 1: + return "merge_or_root" + if row["head"] in fixes: + return "advisory_fix" + if SECURITY.search(row["message"]): + return "security" + if re.search(r"^Revert|This reverts commit", row["message"], re.I | re.M): + return "revert" + if re.search(r"dependabot|renovate|\[bot\]", row["author"], re.I) or DEPENDENCY.search(row["message"]): + return "dependency" + files = row["files"] + if files and all(manifest(f["path"]) for f in files): + return "manifest_only" + source = [f for f in files if Path(f["path"]).suffix in LANGUAGES and not manifest(f["path"])] + if not source: + return "no_source" + if all(f["generated"] or generated_path(f["path"]) for f in source): + return "generated" + if any((f["path"], name) in known_functions for f in files for name in f["functions"]): + return "vulnerable_function" + if any((f["path"], "") in known_functions for f in files): + return "vulnerable_function" + if sum(f["added"] + f["removed"] for f in source) > 1000: + return "size_cap" + return None + + +def generated(git, revision: str, path: str, text: str) -> bool: + import fnmatch + + if generated_path(path): + return True + header = "\n".join(text.splitlines()[:20]) + if re.search(r"(?im)^\s*(?:#|//|/\*|\*|