mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
unseen-eval group 2 (2.1-2.4): advisory sourcing, vulnerability-introducing ranges, ordinary-commit draw, draft manifest and slices; journal with resume
Implemented with Codex. Sourcing run (2.5) next, by the coordinator. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Marius Ciepluch
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
8213d3baad
commit
48ae918dec
@@ -77,7 +77,7 @@ Execution rules:
|
||||
- _Requirements: 1.1, 1.3_
|
||||
|
||||
- [ ] 2. Sourcing and extraction (design 1.1, 1.3, 1.4, 2, 3; increment 2)
|
||||
- [ ] 2.1 Advisory candidates and repository filters
|
||||
- [x] 2.1 Advisory candidates and repository filters
|
||||
- `benchmarks/unseen/source.py`: from GHSA advisory JSON (pip, npm, maven, composer; published 2019 or later)
|
||||
keep those whose CWE maps to a family through `src/openultrasast/model/taxonomy.py` and that link one fix
|
||||
commit with one parent in one repository; attach the OSV first affected version; flag `post_cutoff` when
|
||||
@@ -87,7 +87,7 @@ Execution rules:
|
||||
multi-repository fixes dropped, cutoff flag, licence classing (GPL, AGPL, LGPL, none and source-available go
|
||||
private).
|
||||
- _Requirements: 1.4, 2.1_
|
||||
- [ ] 2.2 Vulnerability-introducing range
|
||||
- [x] 2.2 Vulnerability-introducing range
|
||||
- `benchmarks/unseen/extract.py`: the `introducing` method of design 2.1 (blame of the sink line, or the
|
||||
declaration for an absence bug; absence at V's first parent; no merge or root; at most 50 files and 2,000
|
||||
lines; OSV version bracket when present; walk-back of at most five steps) and the `last_touch` fallback. Each
|
||||
@@ -96,7 +96,7 @@ Execution rules:
|
||||
- `tests/test_unseen_extract.py` on a synthetic repository built in a temp dir: accepted V; a reformat walked
|
||||
back; a merge rejected; size cap; bracket rejection; fallback chosen when no V passes.
|
||||
- _Requirements: 2.1_
|
||||
- [ ] 2.3 Ordinary-commit draw
|
||||
- [x] 2.3 Ordinary-commit draw
|
||||
- `extract.py`: first-parent, non-merge commits touching an analysed language; the exclusions of design 2.2
|
||||
(security words of the narrow pattern, reverts, bot and bump commits, manifest- or lock-only, generated or
|
||||
vendored, advisory fix commits, commits touching a known vulnerable function, more than 1,000 source lines,
|
||||
@@ -105,7 +105,7 @@ Execution rules:
|
||||
- Tests: each exclusion; the narrow pattern keeps "query", "token", "session"; allocation and the seed
|
||||
reproduce the same draw; the 40-commit floor.
|
||||
- _Requirements: 2.2, 2.3_
|
||||
- [ ] 2.4 Draft manifest and slices
|
||||
- [x] 2.4 Draft manifest and slices
|
||||
- `benchmarks/unseen/draft.py`: writes `benchmarks/unseen/draft-p1.toml` (permissive entries by name) and
|
||||
`benchmarks/unseen/private/draft-p1.toml` (the rest); the public file holds an opaque id and
|
||||
sha256(URL, commits) per private entry. Seeded shuffle into three slices of 100 repositories, stratified by
|
||||
@@ -334,3 +334,23 @@ Execution rules:
|
||||
Select capability probe. Default store behavior is unchanged; inventory uses LIST/GET only.
|
||||
- The freeze writer must match the canonical TOML digest encoding documented in the pool README and
|
||||
exercised by `test_independent_population.py`; final pool discovery does not trust mutable status.
|
||||
|
||||
### Group 2 implementation evidence (2026-10-04)
|
||||
|
||||
Tasks 2.1–2.4 are implemented offline and independently reviewed. Task 2.5 is
|
||||
`python -m benchmarks.unseen.draft --root "$PWD" --s3-store s3:// --seed 20260601`
|
||||
with the documented PATH/PYTHONPATH and GH_TOKEN environment. No networked draw,
|
||||
label check, group-2 measurement record, guard bisection, or commit was performed.
|
||||
The group header remains pending its coordinator exits 2.5/2.6.
|
||||
|
||||
The command rebuilds the group-1 used set, resolves renamed/fork repositories,
|
||||
uses disposable external blobless clones, re-reads licences at extracted heads,
|
||||
and emits counts/digests only. The README documents injected clients, method bias,
|
||||
the hard 300-repository floor, and an explicitly unmeasured resource estimate.
|
||||
Independent review caught and fixed ambiguous parent-function absence and stale
|
||||
candidate-404 handling; regression tests cover both.
|
||||
|
||||
Validation: requested pytest selection 103 passed, 1 skipped, exit 0 (population-test
|
||||
output discarded); `ruff check .` exit 0; `ruff format --check .` exit 0 (411 files);
|
||||
`mypy src/openultrasast` exit 0 (160 source files). The CLI `--help` smoke exited 0.
|
||||
Source-read instrumentation printed 20,101 bytes for the extraction implementation.
|
||||
|
||||
@@ -60,3 +60,140 @@ The ledger exposes `append(path, row)` and `read(path)`. Rows contain `slice`, `
|
||||
changed freeze, and reject qualification of an informed slice for the same decision.
|
||||
Keep the published final digest to authenticate history: the chain detects row edits,
|
||||
but cannot independently prove that an entire history was not replaced or truncated.
|
||||
|
||||
## Sourcing and extraction (tasks 2.1–2.4)
|
||||
|
||||
The coordinator's networked task **2.5** is now:
|
||||
|
||||
```sh
|
||||
export PATH=/home/mc/Source/OpenUltraSAST/.venv/bin:$PATH
|
||||
export PYTHONPATH=$PWD/src
|
||||
python -m benchmarks.unseen.draft --root "$PWD" --s3-store s3:// --seed 20260601 \
|
||||
--cache /home/mc/ousast-results/plane/unseen-draft --max-hours 8
|
||||
```
|
||||
|
||||
Export `GH_TOKEN` first, or supply it through the root `.env` (exported values win).
|
||||
The same read-only local/S3 stores, `--pointer-manifest`, and explicit `--known-empty`
|
||||
options as task 1.5 apply. Missing or unreadable sources fail. The used set is rebuilt
|
||||
and canonicalized before candidate selection, including redirects and fork networks.
|
||||
No population v3 file is opened and no guard/bisection is run by this command; that is
|
||||
reserved for group 3. An existing draft is never overwritten.
|
||||
|
||||
The command writes `--cache/journal/draft-p1.toml` and
|
||||
`--cache/journal/private-draft-p1.toml` only after all sourcing stages complete.
|
||||
Both stay outside the repository, with mode 0600. The public-format file contains permissive entries
|
||||
and, for every private entry, only an opaque ID, slice, private marker, and SHA-256
|
||||
of canonical JSON containing its URL and ordered `[base, head]` pairs. Private entries
|
||||
carry the complete records, including licences, in the private-format file.
|
||||
Licensing is re-read through GitHub's licence endpoint **at the actual extracted
|
||||
vulnerability replay head**; the candidate's earlier fix-parent classification is
|
||||
provisional. An explicit SPDX declaration in the pinned file takes precedence over
|
||||
the API's classification. Missing/unknown/compound licences remain private.
|
||||
|
||||
Three seeded slices each contain 100 repositories, one vulnerability range and twelve
|
||||
ordinary changes per repository. Each ecosystem/post-cutoff stratum differs from its
|
||||
proportional per-slice target by less than one repository. Post-cutoff candidates are
|
||||
attempted first within each ecosystem. Below 300 eligible repositories the command
|
||||
exits 1 with counts and does not write a partial draft or relax any selection bar.
|
||||
The group-2 measured exit and label confirmation remain coordinator tasks 2.5/2.6;
|
||||
passing implementation tests does not assert that 300 eligible repositories exist.
|
||||
|
||||
Clones and checkpoints default to `results_root()/unseen-draft` (normally
|
||||
`~/ousast-results/plane/unseen-draft`), overridable by `--cache /outside/path`.
|
||||
The cache cannot lie inside `--root`. One opaque, disposable clone exists at a time;
|
||||
it is removed on success, rejection, or failure. `--filter=blob:none --no-checkout`
|
||||
keeps the commit graph needed for temporal thirds, blame, and release-tag ancestry,
|
||||
and fetches source blobs on demand. This is a **partial clone**, not a shallow history.
|
||||
The 500 MiB disk cap is checked after Git commands, including lazy blob fetches;
|
||||
one in-flight command can transiently exceed it before rejection and cleanup.
|
||||
Git output is captured, never forwarded. Each repository must read a nonempty source
|
||||
blob before extraction. Summary instrumentation includes source read operations and
|
||||
actual bytes, clone count, cumulative/peak disk bytes, wall time, and logical API calls
|
||||
and HTTP attempts. Zero source bytes or a failed Git read is an instrument failure,
|
||||
not an eligibility drop. Stdout is a JSON summary of counts/digests only; no names or
|
||||
URLs enter stdout or a measurement record. Raw per-change data stays in the external journal and manifests.
|
||||
|
||||
Extraction uses Git and syntax boundaries, with no scanner, engine, or model. Python
|
||||
uses stdlib AST; Java/PHP/JS/TS use conservative named-function brace matching. Ambiguous
|
||||
or unparseable functions can reject a candidate. The first removed code line per
|
||||
fix-touched function is the SZZ sink proxy; pure-addition fixes use its declaration.
|
||||
Normalized text absence, non-root/non-merge ancestry, file/line limits, and the OSV
|
||||
release bracket are checked before accepting `introducing`. Matching tags are `VERSION`
|
||||
or `vVERSION`; an absent/ambiguous tag or missing prior release cannot certify the
|
||||
bracket. After at most five walk-backs, `last_touch` uses `git log -L` with the resolved
|
||||
numeric function extent (also works where Git lacks a language-specific function driver).
|
||||
Method, bias, fallback reasons, head function/range, files/hunks, family, advisory, and
|
||||
cutoff flag are recorded. Task 2.6 must review every third `introducing` label.
|
||||
|
||||
All reviewed advisories are enumerated to exclude **all** known repository fixes from
|
||||
ordinary changes, including old advisories and advisories outside the four candidate
|
||||
ecosystems. Fix URLs are canonicalized before indexing so renames cannot hide them.
|
||||
The ordinary filter uses only the design's narrowed security pattern. It counts
|
||||
rejections (including oversized commits and repositories failing the 40-commit floor),
|
||||
checks both sides for touches to known vulnerable functions, and inspects generated
|
||||
headers and inherited `.gitattributes`. Sampling allocates twelve draws proportionally
|
||||
across three history-time thirds and four source-size buckets, with one per third.
|
||||
|
||||
For offline tests, inject GitHub/OSV clients, the clone context, and `UsedSet` into
|
||||
`draft.run`; HTTP transports and Git subprocess runners also accept fakes. Tests use
|
||||
sanitized REST-shaped data and a temporary synthetic local Git history. No implementation
|
||||
test contacts GitHub, OSV, a model, or a cluster. HTTP retries are bounded to five attempts;
|
||||
rate-limit reset/Retry-After waits and transient 5xx backoff are tested with fake clocks
|
||||
and sleepers. A stale candidate 404 is counted; other service failures fail the run.
|
||||
|
||||
### Checkpoint and resume
|
||||
|
||||
The command above uses journal directory
|
||||
`/home/mc/ousast-results/plane/unseen-draft/journal`. Run the same command to resume.
|
||||
`--max-hours` is a per-invocation budget; changing it does not invalidate checkpoints.
|
||||
Create `journal/STOP` to stop between units, then remove it before resuming. A page,
|
||||
repository resolution, candidate decision, or repository extraction already running
|
||||
finishes first, so the time budget is not a hard timeout on a large repository.
|
||||
Clean stops exit 0 with `status: stopped`; failures and shortfalls exit 1. Completion
|
||||
exits 0 with `status: draft`. No partial manifest is published.
|
||||
|
||||
`inputs.json` binds the seed, source-labelled used-set digest, known-empty declarations,
|
||||
clone cap, and design fingerprints (sourcing/extraction/eligibility/publication code and
|
||||
family taxonomy). A mismatch refuses resume and prints only the changed parameter
|
||||
labels. Use a new external cache for a deliberately changed design or population.
|
||||
The used set is rebuilt each invocation; repository resolutions, including 404s, are
|
||||
reused from the journal. Live GitHub metadata for unfinished units can still change;
|
||||
completed pages and decisions are the durable sourcing snapshot.
|
||||
|
||||
Each completed unit is appended to its stage JSONL and fsynced before work advances:
|
||||
`advisories.jsonl` (pages), `resolution.jsonl` (alias/fork metadata),
|
||||
`eligibility.jsonl` (candidate decisions), `extraction.jsonl` (repository/advisory
|
||||
results, vulnerable range, ordinary draws, pinned licence, method and counts), and
|
||||
`completion.jsonl` (final manifest digest). A lock permits only one writer.
|
||||
`progress.json` is atomically rewritten after every unit with stage counts, last unit,
|
||||
last observed GitHub rate-limit headers, and cumulative active elapsed seconds.
|
||||
These files may contain identities and must stay outside version control. Stdout and
|
||||
repository records contain only counts, digests and fixed labels.
|
||||
|
||||
Replaying completed units makes no GitHub/OSV or clone calls. Final manifests are
|
||||
reconstructed deterministically from journal results; an interrupted publication can
|
||||
be retried, but a different existing manifest is refused. Completed JSONL rows are
|
||||
never rewritten. Malformed or torn records fail closed instead of being interpreted
|
||||
as completed work. Tests interrupt each stage with fakes, then compare resumed and
|
||||
uninterrupted manifests byte for byte.
|
||||
|
||||
### Planning estimate for task 2.5 (not measured)
|
||||
|
||||
Let A be all reviewed advisories, R distinct fix-linked repository identities, U used
|
||||
identities not already cached, C candidate advisories passing early filters, and K
|
||||
repositories surviving extraction. GitHub requests are approximately
|
||||
`ceil(A/100) + R + U + 2*C + K` (enumeration, canonical repositories, fix metadata,
|
||||
provisional licence, final licence), plus retries. OSV requests are at most C. Git
|
||||
fetches and lazy blob transfers are additional network operations, not GitHub API calls.
|
||||
|
||||
For a planning scenario of A=40,000, R=15,000, U≈1,000, C=600–1,500, K=300:
|
||||
about **18,000–20,000 GitHub calls** and **600–1,500 OSV calls**. The all-advisory index
|
||||
makes this substantially more than 300 repository lookups. At an assumed 5,000-request
|
||||
hourly allowance, allow several rate-limit windows. Actual token limits can differ.
|
||||
For 300–1,000 attempted clones averaging 20–100 MB after source hydration, plan roughly
|
||||
**6–100 GB transferred**, with one clone occupying about 500 MiB at most between commands
|
||||
plus the retained journal and manifests after completion. Transfer volume is an estimate; disk bytes
|
||||
are only a proxy. A working planning range is **8–48 hours**, potentially longer on
|
||||
large histories: every first-parent commit is inspected and lazy blobs can dominate.
|
||||
These are assumptions, not evidence from a network run. The command reports actual
|
||||
counts and wall time; task 2.5 establishes the measured estimate.
|
||||
|
||||
@@ -0,0 +1,472 @@
|
||||
"""Coordinator-only sourcing command: python -m benchmarks.unseen.draft --help.
|
||||
|
||||
All network and git operations are injected at the run() boundary. Only counts,
|
||||
digests and fixed failure codes go to stdout; identity-bearing output stays in the
|
||||
private journal outside the repository. No population guard is run here.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import copy
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import random
|
||||
import subprocess
|
||||
import tempfile
|
||||
import time
|
||||
from collections import Counter, defaultdict
|
||||
from contextlib import contextmanager
|
||||
from dataclasses import asdict
|
||||
from pathlib import Path
|
||||
|
||||
from . import eligibility, extract, source
|
||||
from .extract import Git, InstrumentFailure
|
||||
from .journal import Journal, ResolvedGitHub, ResumeMismatch, Stopped, atomic, digest, external
|
||||
|
||||
|
||||
def canonical(value) -> bytes:
|
||||
return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode()
|
||||
|
||||
|
||||
def private_digest(row: dict) -> str:
|
||||
return hashlib.sha256(canonical({"url": row["url"], "commits": [[c["base"], c["head"]] for c in row["changes"]]})).hexdigest()
|
||||
|
||||
|
||||
def assign_slices(rows: list[dict], *, seed: int) -> list[dict]:
|
||||
if len(rows) != 300 or len({r["repository"] for r in rows}) != 300:
|
||||
raise ValueError("pool_size")
|
||||
for row in rows:
|
||||
kinds = Counter(c["kind"] for c in row["changes"])
|
||||
if kinds != {"vulnerability": 1, "ordinary": 12}:
|
||||
raise ValueError("change_count")
|
||||
if row["license_class"] not in {"permissive", "private"}:
|
||||
raise ValueError("license_class")
|
||||
if row["license_class"] == "permissive" and row["license_spdx"] not in source.PERMISSIVE:
|
||||
raise ValueError("license_class")
|
||||
rng = random.Random(seed)
|
||||
groups = defaultdict(list)
|
||||
for row in sorted(rows, key=lambda r: r["repository"]):
|
||||
groups[row["ecosystem"], row["post_cutoff"]].append(row)
|
||||
sizes = Counter({1: 0, 2: 0, 3: 0})
|
||||
assigned = []
|
||||
for key in sorted(groups):
|
||||
group = groups[key]
|
||||
rng.shuffle(group)
|
||||
quotient, remainder = divmod(len(group), 3)
|
||||
# Give each stratum its floor on all slices, then fill globally least-full slices.
|
||||
quotas = {i: quotient for i in sizes}
|
||||
order = list(sizes)
|
||||
rng.shuffle(order)
|
||||
order.sort(key=lambda i: sizes[i])
|
||||
for i in order[:remainder]:
|
||||
quotas[i] += 1
|
||||
offset = 0
|
||||
for slice_id in sorted(quotas):
|
||||
for row in group[offset : offset + quotas[slice_id]]:
|
||||
digest = private_digest(row)
|
||||
assigned.append({**row, "id": "r-" + digest[:24], "slice": slice_id})
|
||||
sizes[slice_id] += quotas[slice_id]
|
||||
offset += quotas[slice_id]
|
||||
if set(sizes.values()) != {100}:
|
||||
raise ValueError("slice_size")
|
||||
return sorted(assigned, key=lambda r: (r["slice"], r["id"]))
|
||||
|
||||
|
||||
def toml(value) -> str:
|
||||
"""Inline TOML values keep nested change records structured without a new dependency."""
|
||||
if isinstance(value, bool):
|
||||
return "true" if value else "false"
|
||||
if isinstance(value, str):
|
||||
return json.dumps(value, ensure_ascii=False)
|
||||
if isinstance(value, int):
|
||||
return str(value)
|
||||
if isinstance(value, list):
|
||||
return "[" + ", ".join(toml(v) for v in value) + "]"
|
||||
if isinstance(value, dict):
|
||||
return "{" + ", ".join(json.dumps(k) + " = " + toml(v) for k, v in sorted(value.items()) if v is not None) + "}"
|
||||
raise ValueError("unsupported_manifest_type")
|
||||
|
||||
|
||||
def manifest(rows: list[dict], seed: int) -> str:
|
||||
lines = ['status = "draft"', 'pool = "p1"', f"seed = {seed}"]
|
||||
for row in rows:
|
||||
lines.append("\n[[repository]]")
|
||||
lines.extend(f"{key} = {toml(value)}" for key, value in sorted(row.items()) if value is not None)
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def write_draft(root: Path, assigned: list[dict], *, seed: int) -> tuple[Path, Path]:
|
||||
public_path = root / "benchmarks/unseen/draft-p1.toml"
|
||||
private_path = root / "benchmarks/unseen/private/draft-p1.toml"
|
||||
if public_path.exists() or private_path.exists():
|
||||
raise FileExistsError("draft_exists")
|
||||
public, private = [], []
|
||||
for row in assigned:
|
||||
if row["license_class"] == "permissive":
|
||||
public.append(row)
|
||||
else:
|
||||
private.append(row)
|
||||
public.append({"id": row["id"], "sha256": private_digest(row), "slice": row["slice"], "private": True})
|
||||
private_path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
# Exclusive creates protect drafts awaiting human label review. Private first: never
|
||||
# publish a pointer whose private manifest failed to write.
|
||||
with private_path.open("x") as handle:
|
||||
os.chmod(private_path, 0o600)
|
||||
handle.write(manifest(private, seed))
|
||||
with public_path.open("x") as handle:
|
||||
handle.write(manifest(public, seed))
|
||||
return public_path, private_path
|
||||
|
||||
|
||||
class Clones:
|
||||
"""One disposable blobless clone at a time; no name in cache paths or subprocess output."""
|
||||
|
||||
def __init__(self, root: Path, cache: Path, *, runner=None, limit_bytes=500 * 1024 * 1024):
|
||||
self.cache = external(root, cache)
|
||||
self.runner = runner or subprocess.run
|
||||
self.limit_bytes = limit_bytes
|
||||
self.counts = Counter()
|
||||
|
||||
@contextmanager
|
||||
def open(self, url: str):
|
||||
self.cache.mkdir(parents=True, exist_ok=True)
|
||||
started = time.monotonic()
|
||||
with tempfile.TemporaryDirectory(prefix="clone-", dir=self.cache) as directory:
|
||||
path = Path(directory)
|
||||
git = Git(path, runner=self.runner, limit_bytes=self.limit_bytes)
|
||||
try:
|
||||
# Full commit graph is needed for temporal thirds and OSV brackets; blobs
|
||||
# are fetched lazily. This is a partial clone, not a truncated history.
|
||||
done = self.runner(
|
||||
["git", "-c", "credential.helper=", "clone", "--quiet", "--filter=blob:none", "--no-checkout", url, str(path)],
|
||||
stdin=subprocess.DEVNULL,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
check=False,
|
||||
timeout=180,
|
||||
env={**os.environ, "GIT_TERMINAL_PROMPT": "0"},
|
||||
)
|
||||
if done.returncode:
|
||||
raise InstrumentFailure("clone_failed")
|
||||
self.counts["clones"] += 1
|
||||
if git.disk_bytes() > self.limit_bytes:
|
||||
raise InstrumentFailure("clone_size_cap")
|
||||
yield git
|
||||
finally:
|
||||
size = git.disk_bytes()
|
||||
self.counts["clone_bytes"] += size
|
||||
self.counts["peak_clone_bytes"] = max(self.counts["peak_clone_bytes"], size)
|
||||
self.counts["source_bytes_read"] += git.bytes_read
|
||||
self.counts["source_files_read"] += git.files_read
|
||||
self.counts["clone_wall_seconds"] += round(time.monotonic() - started, 3)
|
||||
|
||||
|
||||
def prove_source(git, revision: str) -> dict:
|
||||
files = git.run("ls-tree", "-r", "--name-only", revision).splitlines()
|
||||
for path in files:
|
||||
if Path(path).suffix in extract.LANGUAGES and not extract.generated_path(path):
|
||||
text = git.blob(revision, path)
|
||||
if text:
|
||||
return {"files": 1, "bytes": getattr(git, "last_blob_bytes", len(text.encode()))}
|
||||
raise InstrumentFailure("zero_source_bytes")
|
||||
|
||||
|
||||
def advisory_index(rows: list[dict], github) -> dict[str, set[str]]:
|
||||
"""Include ALL enumerated fixes, even old, unmapped, multi-fix and other ecosystems.
|
||||
|
||||
Canonicalize link names as well, so an old advisory URL survives repository renames.
|
||||
A 404 is not treated as proof that an advisory is unrelated to a selected repository.
|
||||
"""
|
||||
index = defaultdict(set)
|
||||
for row in rows:
|
||||
for name, sha in source.fix_links(row):
|
||||
try:
|
||||
resolved = eligibility.resolve(name, github)
|
||||
except eligibility.RepositoryNotFound:
|
||||
index[name].add(sha)
|
||||
continue
|
||||
index[resolved.full_name].add(sha)
|
||||
return dict(index)
|
||||
|
||||
|
||||
def repository_changes(git, candidate, fixes: set[str], *, seed: int):
|
||||
proof = prove_source(git, candidate.parent)
|
||||
change = extract.introducing(git, candidate)
|
||||
known = set()
|
||||
expanded = set()
|
||||
for fix in sorted(fixes | {candidate.fix}):
|
||||
# Explicitly fetch detached advisory commits unavailable on default-branch history.
|
||||
# All git including this possible network call still goes through the injected client.
|
||||
git.run("fetch", "--quiet", "--filter=blob:none", "origin", fix)
|
||||
full = git.run("rev-parse", fix).strip()
|
||||
expanded.add(full)
|
||||
parents = git.parents(full)
|
||||
for parent in parents:
|
||||
sites = extract.fix_sites(git, full, parent, global_ok=True)
|
||||
known.update((site["path"], site["function"]) for site in sites)
|
||||
rows = extract.ordinary_history(git, "HEAD")
|
||||
ordinary, exclusions = extract.draw_ordinary(rows, expanded, known, seed=seed)
|
||||
return [change, *ordinary], exclusions, proof
|
||||
|
||||
|
||||
def inputs(used, seed, known_empty, clones):
|
||||
from openultrasast.model.taxonomy import DEFAULT_FAMILIES_PATH
|
||||
|
||||
parameters = {
|
||||
"seed": seed,
|
||||
"used_set_digest": digest({label: sorted(names) for label, names in used.sources.items()}),
|
||||
"known_empty": sorted(known_empty),
|
||||
"clone_limit_bytes": getattr(clones, "limit_bytes", 500 * 1024 * 1024),
|
||||
"journal_version": 1,
|
||||
}
|
||||
# Hardcoded design thresholds and algorithms are covered as well as taxonomy data.
|
||||
for name in ("draft", "source", "extract", "eligibility", "journal"):
|
||||
parameters["design_" + name] = hashlib.sha256(Path(__file__).with_name(name + ".py").read_bytes()).hexdigest()
|
||||
parameters["design_taxonomy"] = hashlib.sha256(DEFAULT_FAMILIES_PATH.read_bytes()).hexdigest()
|
||||
return parameters
|
||||
|
||||
|
||||
def enumerate_advisories(github, journal):
|
||||
page, records = 1, {}
|
||||
while True:
|
||||
key = str(page)
|
||||
if key not in journal.rows["advisories"]:
|
||||
journal.boundary()
|
||||
result = github.advisory_page(page)
|
||||
journal.put("advisories", key, result)
|
||||
result = journal.rows["advisories"][key]
|
||||
records.update((row["ghsa_id"], row) for row in result["rows"])
|
||||
if result["done"]:
|
||||
break
|
||||
page += 1
|
||||
if not records:
|
||||
raise InstrumentFailure("empty_advisories")
|
||||
return list(records.values())
|
||||
|
||||
|
||||
def decide(row, github, osv, used, selected_networks):
|
||||
try:
|
||||
links = source.fix_links(row)
|
||||
if len(links) != 1:
|
||||
raise source.Rejected("fix_links")
|
||||
name, _ = next(iter(links))
|
||||
resolved = eligibility.resolve(name, github)
|
||||
rejected_by = eligibility.rejection(resolved, used, set(selected_networks))
|
||||
if rejected_by:
|
||||
raise source.Rejected("used_" + rejected_by)
|
||||
candidate = source.candidate(row, github, osv)
|
||||
return {"candidate": asdict(candidate), "network": resolved.network}
|
||||
except source.Rejected as exc:
|
||||
return {"rejected": str(exc)}
|
||||
except eligibility.RepositoryNotFound:
|
||||
return {"rejected": "repository_or_commit_not_found"}
|
||||
|
||||
|
||||
def extract_repository(candidate, github, clones, fixes, seed):
|
||||
before = dict(clones.counts)
|
||||
try:
|
||||
with clones.open(candidate.url) as git:
|
||||
git.run("fetch", "--quiet", "--filter=blob:none", "origin", candidate.fix)
|
||||
changes, exclusions, proof = repository_changes(git, candidate, fixes, seed=seed)
|
||||
pinned = changes[0]["head"]
|
||||
license_record = github.get_license(candidate.repository, pinned)
|
||||
spdx, license_class = source.license_info(license_record)
|
||||
entry = {
|
||||
**asdict(candidate),
|
||||
"license_spdx": spdx,
|
||||
"license_class": license_class,
|
||||
"license_ref": pinned,
|
||||
"license_path": license_record.get("path", ""),
|
||||
"changes": changes,
|
||||
"instrument": proof,
|
||||
}
|
||||
result = {"entry": entry, "ordinary_exclusions": exclusions}
|
||||
except source.Rejected as exc:
|
||||
result = {"rejected": str(exc), "ordinary_exclusions": getattr(exc, "counts", {})}
|
||||
except eligibility.RepositoryNotFound:
|
||||
result = {"rejected": "repository_or_commit_not_found", "ordinary_exclusions": {}}
|
||||
result["counts"] = {key: value - before.get(key, 0) for key, value in clones.counts.items()}
|
||||
return result
|
||||
|
||||
|
||||
def publish(journal, assigned, seed):
|
||||
"""Idempotent publication from completed journal data, never inside the checkout."""
|
||||
public, private = [], []
|
||||
for row in assigned:
|
||||
if row["license_class"] == "permissive":
|
||||
public.append(row)
|
||||
else:
|
||||
private.append(row)
|
||||
public.append({"id": row["id"], "sha256": private_digest(row), "slice": row["slice"], "private": True})
|
||||
for name, rows in (("private-draft-p1.toml", private), ("draft-p1.toml", public)):
|
||||
path = journal.path / name
|
||||
data = manifest(rows, seed).encode()
|
||||
if path.exists():
|
||||
if path.read_bytes() != data:
|
||||
raise ValueError("draft_exists")
|
||||
else:
|
||||
atomic(path, data)
|
||||
|
||||
|
||||
def run(
|
||||
root: Path,
|
||||
*,
|
||||
github,
|
||||
osv,
|
||||
clones,
|
||||
used: eligibility.UsedSet,
|
||||
seed: int,
|
||||
known_empty=None,
|
||||
journal_path=None,
|
||||
max_hours=None,
|
||||
clock=None,
|
||||
) -> dict:
|
||||
started = time.monotonic()
|
||||
report = {"status": "sourcing", "seed": seed, "candidates": {}, "rejections": {}, "ordinary_exclusions": {}}
|
||||
counts, rejected, ordinary_counts = Counter(), Counter(), Counter()
|
||||
checkpoint = None
|
||||
try:
|
||||
known_empty = set(known_empty or ())
|
||||
used.report(known_empty)
|
||||
cache = getattr(clones, "cache", root.parent / (root.name + "-cache"))
|
||||
path = external(root, journal_path or cache / "journal")
|
||||
checkpoint = Journal(
|
||||
path, max_hours=max_hours, clock=clock, rate_limit=lambda: getattr(getattr(github, "transport", None), "rate_limit", {})
|
||||
)
|
||||
with checkpoint.open(inputs(used, seed, known_empty, clones)):
|
||||
checkpoint.boundary()
|
||||
api = ResolvedGitHub(github, checkpoint)
|
||||
used = copy.deepcopy(used) # canonicalization must not mutate the resume input
|
||||
eligibility.canonicalize_used(used, api)
|
||||
report["used_set"] = used.report(known_empty)
|
||||
rows = enumerate_advisories(api, checkpoint)
|
||||
report["advisories"] = len(rows)
|
||||
fixes_by_repo = advisory_index(rows, api)
|
||||
selected_networks, entries = set(), []
|
||||
for row in source.ordered_advisories(rows, seed):
|
||||
checkpoint.boundary()
|
||||
key = row["ghsa_id"]
|
||||
counts[source.ecosystem(row)] += 1
|
||||
if key not in checkpoint.rows["eligibility"]:
|
||||
decision = decide(row, api, osv, used, selected_networks)
|
||||
checkpoint.put("eligibility", key, decision)
|
||||
decision = checkpoint.rows["eligibility"][key]
|
||||
if "rejected" in decision:
|
||||
rejected[decision["rejected"]] += 1
|
||||
continue
|
||||
candidate = source.Candidate(**decision["candidate"])
|
||||
# Multiple advisories for one repository can have different fix/range inputs.
|
||||
extraction_key = digest({"repository": candidate.repository, "advisory": key, "fix": candidate.fix})
|
||||
if extraction_key not in checkpoint.rows["extraction"]:
|
||||
checkpoint.boundary()
|
||||
result = extract_repository(candidate, api, clones, fixes_by_repo.get(candidate.repository, set()), seed)
|
||||
checkpoint.put("extraction", extraction_key, result)
|
||||
result = checkpoint.rows["extraction"][extraction_key]
|
||||
ordinary_counts.update(result["ordinary_exclusions"])
|
||||
if "rejected" in result:
|
||||
rejected[result["rejected"]] += 1
|
||||
continue
|
||||
selected_networks.add(decision["network"])
|
||||
entries.append(result["entry"])
|
||||
if len(entries) == 300:
|
||||
break
|
||||
report.update(
|
||||
{
|
||||
"repositories": len(entries),
|
||||
"candidates": dict(counts),
|
||||
"rejections": dict(rejected),
|
||||
"ordinary_exclusions": dict(ordinary_counts),
|
||||
}
|
||||
)
|
||||
checkpoint.boundary()
|
||||
if len(entries) != 300:
|
||||
report["status"] = "insufficient_repositories"
|
||||
checkpoint.progress(report["status"])
|
||||
return report
|
||||
assigned = assign_slices(entries, seed=seed)
|
||||
completion = {"manifest_sha256": digest(assigned), "repositories": len(entries)}
|
||||
if "all" not in checkpoint.rows["completion"]:
|
||||
checkpoint.put("completion", "all", completion)
|
||||
elif checkpoint.rows["completion"]["all"] != completion:
|
||||
raise ValueError("journal_completion_mismatch")
|
||||
checkpoint.boundary()
|
||||
publish(checkpoint, assigned, seed)
|
||||
report.update(
|
||||
{
|
||||
"status": "draft",
|
||||
"changes": 3900,
|
||||
"slices": 3,
|
||||
"license_split": dict(Counter(e["license_class"] for e in entries)),
|
||||
"methods": dict(Counter(e["changes"][0]["method"] for e in entries)),
|
||||
"post_cutoff": sum(e["post_cutoff"] for e in entries),
|
||||
**completion,
|
||||
}
|
||||
)
|
||||
checkpoint.progress("draft")
|
||||
return report
|
||||
except ResumeMismatch as exc:
|
||||
report.update({"status": "resume_refused", "changed_parameters": exc.parameters})
|
||||
return report
|
||||
except Stopped as exc:
|
||||
report.update({"status": "stopped", "reason": str(exc)})
|
||||
return report
|
||||
except Exception as exc:
|
||||
report.update({"status": "instrument_failure", "error_type": type(exc).__name__})
|
||||
return report
|
||||
finally:
|
||||
report.update({"candidates": dict(counts), "rejections": dict(rejected), "ordinary_exclusions": dict(ordinary_counts)})
|
||||
report["instrument"] = dict(clones.counts)
|
||||
report["wall_seconds"] = round(time.monotonic() - started, 3)
|
||||
report["github_calls"] = github.calls
|
||||
report["osv_calls"] = osv.calls
|
||||
report["github_http_attempts"] = getattr(getattr(github, "transport", None), "attempts", github.calls)
|
||||
report["osv_http_attempts"] = getattr(getattr(osv, "transport", None), "attempts", osv.calls)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
from openultrasast import config
|
||||
from openultrasast.plane import memory
|
||||
from openultrasast.plane.reconciler import results_root
|
||||
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--root", type=Path, default=Path.cwd())
|
||||
parser.add_argument("--cache", type=Path)
|
||||
parser.add_argument("--max-hours", type=float, help="Stop between units after this many hours per invocation")
|
||||
parser.add_argument("--seed", type=int, default=20260601)
|
||||
parser.add_argument("--local-store")
|
||||
parser.add_argument("--s3-store", default="s3://")
|
||||
parser.add_argument("--pointer-manifest", type=Path, action="append", default=[])
|
||||
parser.add_argument("--known-empty", choices=(*eligibility.SOURCES, "memory_local", "memory_s3"), action="append", default=[])
|
||||
args = parser.parse_args(argv)
|
||||
try:
|
||||
config.load_dotenv(args.root / ".env")
|
||||
# Refuse an existing draft before doing any network work.
|
||||
if any((args.root / p).exists() for p in ("benchmarks/unseen/draft-p1.toml", "benchmarks/unseen/private/draft-p1.toml")):
|
||||
raise FileExistsError("draft_exists")
|
||||
clones = Clones(args.root, args.cache or results_root() / "unseen-draft")
|
||||
stores = {
|
||||
"local": memory.open_store(args.local_store or "file://" + str(results_root() / "memory"), read_only=True),
|
||||
"s3": memory.open_store(args.s3_store, read_only=True),
|
||||
}
|
||||
used = eligibility.build_used(args.root, stores=stores, pointer_manifests=args.pointer_manifest)
|
||||
report = run(
|
||||
args.root,
|
||||
github=source.GitHub(os.environ.get("GH_TOKEN", "")),
|
||||
osv=source.OSV(),
|
||||
clones=clones,
|
||||
used=used,
|
||||
seed=args.seed,
|
||||
known_empty=set(args.known_empty),
|
||||
journal_path=clones.cache / "journal",
|
||||
max_hours=args.max_hours,
|
||||
)
|
||||
except Exception as exc:
|
||||
report = {"status": "instrument_failure", "error_type": type(exc).__name__}
|
||||
print(json.dumps(report, sort_keys=True))
|
||||
return 0 if report["status"] in {"draft", "stopped"} else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,474 @@
|
||||
"""Git-only change extraction; clients and subprocess execution are injectable.
|
||||
|
||||
No scanner, engine or model runs here. Function boundaries are syntax, not a security
|
||||
judgement: the first removed code line of a fix is the SZZ sink proxy. Ambiguous sites
|
||||
are dropped, and the recorded SZZ bias remains subject to task 2.6's label check.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import ast
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
from .source import Candidate, Rejected
|
||||
|
||||
LANGUAGES = {".py", ".js", ".jsx", ".mjs", ".cjs", ".ts", ".tsx", ".php", ".java"}
|
||||
HUNK = re.compile(r"^@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@", re.M)
|
||||
SHA = re.compile(r"[a-f0-9]{40}")
|
||||
BIAS = {
|
||||
"introducing": "SZZ last-writer proxy: refactors, moves and reformats can misidentify the introduction in either direction.",
|
||||
"last_touch": "Pessimistic lower bound: the flaw may already exist in base, so a delta hook can correctly stay quiet.",
|
||||
}
|
||||
|
||||
|
||||
class InstrumentFailure(ValueError):
|
||||
"""A failed read or command must never masquerade as an empty history."""
|
||||
|
||||
|
||||
class Git:
|
||||
def __init__(self, path: Path, *, runner=None, limit_bytes: int | None = None):
|
||||
self.path = path
|
||||
self.runner = runner or subprocess.run
|
||||
self.bytes_read = 0
|
||||
self.files_read = 0
|
||||
self.last_blob_bytes = 0
|
||||
self.limit_bytes = limit_bytes
|
||||
|
||||
def run(self, *args: str, allowed=(0,)) -> str:
|
||||
try:
|
||||
done = self.runner(
|
||||
["git", "-c", "core.quotePath=false", "-C", str(self.path), *args],
|
||||
stdin=subprocess.DEVNULL,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
timeout=180,
|
||||
check=False,
|
||||
)
|
||||
except Exception:
|
||||
raise InstrumentFailure("git_launch_failed") from None
|
||||
if done.returncode not in allowed:
|
||||
raise InstrumentFailure("git_exit_" + str(done.returncode))
|
||||
if self.limit_bytes is not None and self.disk_bytes() > self.limit_bytes:
|
||||
raise InstrumentFailure("clone_size_cap")
|
||||
self.last_blob_bytes = len(done.stdout)
|
||||
return done.stdout.decode("utf-8", errors="replace")
|
||||
|
||||
def disk_bytes(self) -> int:
|
||||
return sum(p.stat().st_size for p in self.path.rglob("*") if p.is_file() and not p.is_symlink())
|
||||
|
||||
def parents(self, sha: str) -> list[str]:
|
||||
return self.run("rev-list", "--parents", "-n", "1", sha).split()[1:]
|
||||
|
||||
def blob(self, sha: str, path: str) -> str | None:
|
||||
# Missing is established by a successful tree read, never by swallowing a show error.
|
||||
if not self.run("ls-tree", sha, "--", path).strip():
|
||||
return None
|
||||
text = self.run("show", f"{sha}:{path}")
|
||||
if Path(path).suffix in LANGUAGES:
|
||||
self.files_read += 1
|
||||
self.bytes_read += self.last_blob_bytes
|
||||
return text
|
||||
|
||||
def stats(self, base: str, head: str) -> list[tuple[str, int, int]]:
|
||||
rows = []
|
||||
for line in self.run("diff", "--no-renames", "--numstat", base, head, "--").splitlines():
|
||||
added, removed, path = line.split("\t", 2)
|
||||
rows.append((path, int(added) if added.isdigit() else 0, int(removed) if removed.isdigit() else 0))
|
||||
return rows
|
||||
|
||||
def hunks(self, base: str, head: str, path: str) -> list[tuple[int, int, int, int]]:
|
||||
diff = self.run("diff", "--no-ext-diff", "--no-renames", "-U0", base, head, "--", path)
|
||||
return [(int(m[1]), int(m[2] or 1), int(m[3]), int(m[4] or 1)) for m in HUNK.finditer(diff)]
|
||||
|
||||
def ancestor(self, older: str, newer: str) -> bool:
|
||||
# rev-list avoids interpreting a command failure as a negative ancestry result.
|
||||
return not self.run("rev-list", "--max-count=1", older, "--not", newer).strip()
|
||||
|
||||
def blame(self, revision: str, path: str, line: int) -> tuple[str, str, int]:
|
||||
text = self.run("blame", "-w", "-M", "-C", "--porcelain", f"-L{line},{line}", revision, "--", path)
|
||||
header = text.splitlines()[0].split()
|
||||
if len(header) < 3 or not SHA.fullmatch(header[0]):
|
||||
raise InstrumentFailure("invalid_blame")
|
||||
filename = next((row[9:] for row in text.splitlines() if row.startswith("filename ")), path)
|
||||
if filename.startswith('"'):
|
||||
filename = json.loads(filename)
|
||||
return header[0], filename, int(header[1])
|
||||
|
||||
|
||||
def functions(path: str, text: str) -> list[tuple[str, int, int]]:
|
||||
"""Conservative syntax boundaries. No scanner import; unresolved sites are rejected."""
|
||||
if Path(path).suffix == ".py":
|
||||
try:
|
||||
tree = ast.parse(text)
|
||||
except SyntaxError:
|
||||
raise Rejected("function_parse") from None
|
||||
return [
|
||||
(n.name, n.lineno, n.end_lineno or n.lineno) for n in ast.walk(tree) if isinstance(n, ast.FunctionDef | ast.AsyncFunctionDef)
|
||||
]
|
||||
# Blank comments and quoted strings before brace matching. Keep newlines/offsets.
|
||||
masked = re.sub(
|
||||
r'/\*.*?\*/|//[^\n]*|\#[^\n]*|"(?:\\.|[^"\\])*"|\'(?:\\.|[^\'\\])*\'|`(?:\\.|[^`\\])*`',
|
||||
lambda m: "".join("\n" if c == "\n" else " " for c in m[0]),
|
||||
text,
|
||||
flags=re.S,
|
||||
)
|
||||
declaration = re.compile(
|
||||
r"\bfunction\s*&?\s*(\w+)\s*\([^)]*\)\s*(?::[^\n{]+)?\s*\{|"
|
||||
r"\b(?:const|let|var)\s+(\w+)\s*=\s*(?:async\s*)?(?:\([^)]*\)|\w+)\s*=>\s*\{|"
|
||||
r"\b([\w$]+)\s*\([^;{}]*\)\s*(?:throws\s+[\w., ]+)?\s*\{"
|
||||
)
|
||||
found = []
|
||||
for match in declaration.finditer(masked):
|
||||
name = next(g for g in match.groups() if g)
|
||||
if name in {"if", "while", "for", "switch", "catch", "with", "synchronized"}:
|
||||
continue
|
||||
depth = 1
|
||||
end = match.end()
|
||||
while end < len(masked) and depth:
|
||||
depth += (masked[end] == "{") - (masked[end] == "}")
|
||||
end += 1
|
||||
if depth == 0:
|
||||
found.append((name, masked.count("\n", 0, match.start()) + 1, masked.count("\n", 0, end) + 1))
|
||||
return found
|
||||
|
||||
|
||||
def enclosing(path: str, text: str, line: int, *, global_ok=False) -> tuple[str, int, int] | None:
|
||||
spans = [s for s in functions(path, text) if s[1] <= line <= s[2]]
|
||||
if spans:
|
||||
return min(spans, key=lambda s: s[2] - s[1])
|
||||
return ("<global>", 1, max(1, len(text.splitlines()))) if global_ok else None
|
||||
|
||||
|
||||
def normalized(text: str) -> str:
|
||||
return re.sub(r"\s+", "", text)
|
||||
|
||||
|
||||
def site_at(git, revision: str, path: str, name: str):
|
||||
text = git.blob(revision, path)
|
||||
if text is None:
|
||||
return None
|
||||
spans = [("<global>", 1, max(1, len(text.splitlines())))] if name == "<global>" else [s for s in functions(path, text) if s[0] == name]
|
||||
if len(spans) != 1:
|
||||
return None
|
||||
return text, spans[0]
|
||||
|
||||
|
||||
def fix_sites(git, fix: str, parent: str, *, global_ok=False) -> list[dict]:
|
||||
sites = []
|
||||
for path, _, _ in git.stats(parent, fix):
|
||||
if Path(path).suffix not in LANGUAGES:
|
||||
continue
|
||||
text = git.blob(parent, path)
|
||||
if text is None:
|
||||
continue
|
||||
lines = text.splitlines()
|
||||
for start, count, _, _ in git.hunks(parent, fix, path):
|
||||
for line in range(max(1, start), max(1, start) + max(1, count)):
|
||||
if line > len(lines):
|
||||
continue
|
||||
span = enclosing(path, text, line, global_ok=global_ok)
|
||||
if span is None:
|
||||
continue
|
||||
sink_line = line if count else span[1]
|
||||
sink = lines[sink_line - 1]
|
||||
if not sink.strip() or sink.lstrip().startswith(("#", "//", "/*", "*")):
|
||||
continue
|
||||
key = (path, span[0])
|
||||
if any((s["path"], s["function"]) == key for s in sites):
|
||||
continue
|
||||
sites.append({"path": path, "function": span[0], "line": sink_line, "text": sink, "absence": count == 0, "span": span})
|
||||
return sites
|
||||
|
||||
|
||||
def version_bracket(git, candidate: Candidate, revision: str) -> bool:
|
||||
if not candidate.first_affected:
|
||||
return True
|
||||
tags = git.run("tag", "--sort=version:refname").splitlines()
|
||||
matching = [t for t in tags if t.removeprefix("v") == candidate.first_affected]
|
||||
if len(matching) != 1:
|
||||
return False
|
||||
affected = matching[0]
|
||||
earlier = [t for t in tags[: tags.index(affected)] if git.ancestor(t, affected)]
|
||||
return bool(earlier) and git.ancestor(revision, affected) and not git.ancestor(revision, earlier[-1])
|
||||
|
||||
|
||||
def change_files(git, base: str, head: str) -> list[dict]:
|
||||
return [
|
||||
{"path": path, "head_lines": [[max(1, new), max(1, new + max(count, 1) - 1)] for _, _, new, count in git.hunks(base, head, path)]}
|
||||
for path, _, _ in git.stats(base, head)
|
||||
]
|
||||
|
||||
|
||||
def introducing(git, candidate: Candidate) -> dict:
|
||||
from collections import Counter
|
||||
|
||||
if git.parents(candidate.fix) != [candidate.parent]:
|
||||
raise Rejected("fix_parents")
|
||||
sites = fix_sites(git, candidate.fix, candidate.parent, global_ok=candidate.family == "config_secrets")
|
||||
reasons = Counter()
|
||||
|
||||
def record(method, base, head, path, name):
|
||||
resolved = site_at(git, head, path, name)
|
||||
if resolved is None:
|
||||
return None
|
||||
return {
|
||||
"kind": "vulnerability",
|
||||
"base": base,
|
||||
"head": head,
|
||||
"method": method,
|
||||
"family": candidate.family,
|
||||
"function": name,
|
||||
"path": path,
|
||||
"function_lines": list(resolved[1][1:]),
|
||||
"files": change_files(git, base, head),
|
||||
"post_cutoff": candidate.post_cutoff,
|
||||
"advisory": candidate.advisory,
|
||||
"bias": BIAS[method],
|
||||
"fallback_reasons": dict(reasons),
|
||||
}
|
||||
|
||||
for site in sites:
|
||||
revision, path, line = candidate.parent, site["path"], site["line"]
|
||||
for _ in range(6): # initial blame plus at most five walk-back steps
|
||||
blamed, blamed_path, blamed_line = git.blame(revision, path, line)
|
||||
parents = git.parents(blamed)
|
||||
if len(parents) != 1:
|
||||
reasons["merge_or_root"] += 1
|
||||
break
|
||||
base = parents[0]
|
||||
head_site = site_at(git, blamed, blamed_path, site["function"])
|
||||
if head_site is None:
|
||||
reasons["unresolved_function"] += 1
|
||||
break
|
||||
parent_text = git.blob(base, blamed_path)
|
||||
if parent_text is not None and site["function"] != "<global>":
|
||||
matching = [span for span in functions(blamed_path, parent_text) if span[0] == site["function"]]
|
||||
if len(matching) > 1:
|
||||
reasons["ambiguous_parent"] += 1
|
||||
break
|
||||
old_site = site_at(git, base, blamed_path, site["function"])
|
||||
occurrences = []
|
||||
if old_site:
|
||||
text, (_, start, end) = old_site
|
||||
occurrences = [
|
||||
i for i, value in enumerate(text.splitlines(), 1) if start <= i <= end and normalized(value) == normalized(site["text"])
|
||||
]
|
||||
if occurrences:
|
||||
reasons["text_present"] += 1
|
||||
revision, path, line = base, blamed_path, occurrences[0]
|
||||
continue
|
||||
stats = git.stats(base, blamed)
|
||||
if len(stats) > 50 or sum(a + d for _, a, d in stats) > 2000:
|
||||
reasons["size_cap"] += 1
|
||||
break
|
||||
if not version_bracket(git, candidate, blamed):
|
||||
reasons["version_bracket"] += 1
|
||||
break
|
||||
result = record("introducing", base, blamed, blamed_path, site["function"])
|
||||
if result:
|
||||
return result
|
||||
else:
|
||||
reasons["walkback_limit"] += 1
|
||||
for site in sites:
|
||||
# Numeric function extent implements log -L for all five languages, including
|
||||
# languages for which Git has no built-in :function: driver.
|
||||
_, start, end = site["span"]
|
||||
history = git.run("log", "--format=%H", "--no-patch", f"-L{start},{end}:{site['path']}", candidate.parent).splitlines()
|
||||
if not history:
|
||||
continue
|
||||
latest = history[0]
|
||||
parents = git.parents(latest)
|
||||
if len(parents) != 1:
|
||||
continue
|
||||
result = record("last_touch", parents[0], candidate.parent, site["path"], site["function"])
|
||||
if result:
|
||||
return result
|
||||
raise Rejected("no_range")
|
||||
|
||||
|
||||
SECURITY = re.compile(r"secur|cve|ghsa|vuln|inject|xss|csrf|ssrf|exploit|sanitiz|escap", re.I)
|
||||
DEPENDENCY = re.compile(r"\bbump\b|\bupgrade .* from\b", re.I)
|
||||
MANIFESTS = {
|
||||
"package.json",
|
||||
"package-lock.json",
|
||||
"yarn.lock",
|
||||
"pnpm-lock.yaml",
|
||||
"composer.json",
|
||||
"composer.lock",
|
||||
"pom.xml",
|
||||
"build.gradle",
|
||||
"build.gradle.kts",
|
||||
"gradle.lockfile",
|
||||
"pyproject.toml",
|
||||
"setup.py",
|
||||
"setup.cfg",
|
||||
"Pipfile",
|
||||
"Pipfile.lock",
|
||||
"poetry.lock",
|
||||
"uv.lock",
|
||||
}
|
||||
|
||||
|
||||
def generated_path(path: str) -> bool:
|
||||
return bool({"vendor", "node_modules", "dist"} & set(Path(path).parts)) or path.endswith(".min.js")
|
||||
|
||||
|
||||
def manifest(path: str) -> bool:
|
||||
name = Path(path).name
|
||||
return name in MANIFESTS or name.endswith(".lock") or bool(re.fullmatch(r"requirements[^/]*\.txt", name))
|
||||
|
||||
|
||||
def ordinary_rejection(row: dict, fixes: set[str], known_functions: set[tuple[str, str]]) -> str | None:
|
||||
if len(row["parents"]) != 1:
|
||||
return "merge_or_root"
|
||||
if row["head"] in fixes:
|
||||
return "advisory_fix"
|
||||
if SECURITY.search(row["message"]):
|
||||
return "security"
|
||||
if re.search(r"^Revert|This reverts commit", row["message"], re.I | re.M):
|
||||
return "revert"
|
||||
if re.search(r"dependabot|renovate|\[bot\]", row["author"], re.I) or DEPENDENCY.search(row["message"]):
|
||||
return "dependency"
|
||||
files = row["files"]
|
||||
if files and all(manifest(f["path"]) for f in files):
|
||||
return "manifest_only"
|
||||
source = [f for f in files if Path(f["path"]).suffix in LANGUAGES and not manifest(f["path"])]
|
||||
if not source:
|
||||
return "no_source"
|
||||
if all(f["generated"] or generated_path(f["path"]) for f in source):
|
||||
return "generated"
|
||||
if any((f["path"], name) in known_functions for f in files for name in f["functions"]):
|
||||
return "vulnerable_function"
|
||||
if any((f["path"], "<global>") in known_functions for f in files):
|
||||
return "vulnerable_function"
|
||||
if sum(f["added"] + f["removed"] for f in source) > 1000:
|
||||
return "size_cap"
|
||||
return None
|
||||
|
||||
|
||||
def generated(git, revision: str, path: str, text: str) -> bool:
|
||||
import fnmatch
|
||||
|
||||
if generated_path(path):
|
||||
return True
|
||||
header = "\n".join(text.splitlines()[:20])
|
||||
if re.search(r"(?im)^\s*(?:#|//|/\*|\*|<!--).*?(?:@generated|auto.?generated|generated\s+(?:by|file|code)|do not edit)", header):
|
||||
return True
|
||||
result = False
|
||||
parts = Path(path).parts
|
||||
for index in range(len(parts)):
|
||||
directory = "/".join(parts[:index])
|
||||
attributes = git.blob(revision, (directory + "/" if directory else "") + ".gitattributes") or ""
|
||||
relative = "/".join(parts[index:])
|
||||
for line in attributes.splitlines():
|
||||
tokens = line.split()
|
||||
if not tokens or tokens[0].startswith("#"):
|
||||
continue
|
||||
pattern = tokens[0].lstrip("/")
|
||||
target = relative if "/" in pattern else Path(relative).name
|
||||
if fnmatch.fnmatchcase(target, pattern):
|
||||
for token in tokens[1:]:
|
||||
if token in {"linguist-generated", "linguist-generated=true"}:
|
||||
result = True
|
||||
elif token in {"-linguist-generated", "!linguist-generated", "linguist-generated=false"}:
|
||||
result = False
|
||||
return result
|
||||
|
||||
|
||||
def ordinary_history(git, head: str) -> list[dict]:
|
||||
raw = git.run("log", "--first-parent", "--format=%H%x00%P%x00%ct%x00%an%x00%B%x1e", head)
|
||||
rows = []
|
||||
for entry in raw.split("\x1e"):
|
||||
if not entry.strip():
|
||||
continue
|
||||
fields = entry.strip().split("\x00")
|
||||
if len(fields) != 5 or not SHA.fullmatch(fields[0]):
|
||||
raise InstrumentFailure("invalid_history")
|
||||
sha, parents, timestamp, author, message = fields
|
||||
row = {"head": sha, "parents": parents.split(), "timestamp": int(timestamp), "author": author, "message": message, "files": []}
|
||||
rows.append(row)
|
||||
if len(row["parents"]) != 1:
|
||||
continue
|
||||
base = row["parents"][0]
|
||||
for path, added, removed in git.stats(base, sha):
|
||||
hunks = git.hunks(base, sha, path)
|
||||
info = {
|
||||
"path": path,
|
||||
"added": added,
|
||||
"removed": removed,
|
||||
"head_lines": [[max(1, n), max(1, n + max(c, 1) - 1)] for _, _, n, c in hunks],
|
||||
"functions": [],
|
||||
"generated": False,
|
||||
}
|
||||
if Path(path).suffix in LANGUAGES:
|
||||
# Inspect both sides: deleting a vulnerable function is still touching it.
|
||||
for revision, side in ((base, 0), (sha, 2)):
|
||||
text = git.blob(revision, path)
|
||||
if text is None:
|
||||
continue
|
||||
spans = functions(path, text)
|
||||
for hunk in hunks:
|
||||
start, count = hunk[side : side + 2]
|
||||
for name, lo, hi in spans:
|
||||
if max(1, start) <= hi and start + max(count, 1) - 1 >= lo:
|
||||
info["functions"].append(name)
|
||||
info["generated"] |= generated(git, revision, path, text)
|
||||
row["files"].append(info)
|
||||
return rows
|
||||
|
||||
|
||||
def draw_ordinary(rows: list[dict], fixes: set[str], known_functions: set[tuple[str, str]], *, seed: int) -> tuple[list[dict], dict]:
|
||||
import random
|
||||
from collections import Counter, defaultdict
|
||||
|
||||
counts = Counter()
|
||||
eligible = []
|
||||
for row in rows:
|
||||
reason = ordinary_rejection(row, fixes, known_functions)
|
||||
counts[reason or "eligible"] += 1
|
||||
if reason is None:
|
||||
eligible.append(row)
|
||||
if len(eligible) < 40:
|
||||
failure = Rejected("ordinary_floor")
|
||||
failure.counts = dict(counts)
|
||||
raise failure
|
||||
earliest, latest = min(r["timestamp"] for r in rows), max(r["timestamp"] for r in rows)
|
||||
cells = defaultdict(list)
|
||||
for row in eligible:
|
||||
third = min(2, 3 * (row["timestamp"] - earliest) // max(1, latest - earliest + 1))
|
||||
size = sum(f["added"] + f["removed"] for f in row["files"] if Path(f["path"]).suffix in LANGUAGES and not manifest(f["path"]))
|
||||
bucket = 0 if size <= 10 else 1 if size <= 50 else 2 if size <= 200 else 3
|
||||
cells[third, bucket].append(row)
|
||||
if {t for t, _ in cells} != {0, 1, 2}:
|
||||
failure = Rejected("time_thirds")
|
||||
failure.counts = dict(counts)
|
||||
raise failure
|
||||
# Largest-deficit rounding with the mandatory minimum of one per time third.
|
||||
# No cell can receive more draws than it has eligible commits.
|
||||
allocation = Counter()
|
||||
targets = {cell: 12 * len(items) / len(eligible) for cell, items in cells.items()}
|
||||
for third in range(3):
|
||||
cell = max(sorted(c for c in cells if c[0] == third), key=lambda c: targets[c])
|
||||
allocation[cell] = 1
|
||||
while sum(allocation.values()) < 12:
|
||||
cell = max(sorted(c for c in cells if allocation[c] < len(cells[c])), key=lambda c: targets[c] - allocation[c])
|
||||
allocation[cell] += 1
|
||||
rng = random.Random(seed)
|
||||
drawn = []
|
||||
for cell in sorted(cells):
|
||||
for row in rng.sample(sorted(cells[cell], key=lambda r: r["head"]), allocation[cell]):
|
||||
drawn.append(
|
||||
{
|
||||
"kind": "ordinary",
|
||||
"base": row["parents"][0],
|
||||
"head": row["head"],
|
||||
"method": "first_parent",
|
||||
"time_third": cell[0],
|
||||
"size_bucket": cell[1],
|
||||
"files": [{"path": f["path"], "head_lines": f["head_lines"]} for f in row["files"]],
|
||||
}
|
||||
)
|
||||
return drawn, dict(counts)
|
||||
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"ghsa_id": "GHSA-test-test-test",
|
||||
"published_at": "2026-06-02T00:00:00Z",
|
||||
"type": "reviewed",
|
||||
"withdrawn_at": null,
|
||||
"cwes": [{"cwe_id": "CWE-89"}],
|
||||
"vulnerabilities": [{"package": {"ecosystem": "pip", "name": "fixture"}}],
|
||||
"references": ["https://github.com/fixture/project/commit/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"]
|
||||
}
|
||||
@@ -0,0 +1,184 @@
|
||||
"""Private, append-only sourcing checkpoints. No identity-bearing data leaves this directory."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import fcntl
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import time
|
||||
from contextlib import contextmanager
|
||||
from pathlib import Path
|
||||
|
||||
STAGES = ("advisories", "resolution", "eligibility", "extraction", "completion")
|
||||
|
||||
|
||||
def encoded(value) -> bytes:
|
||||
return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode()
|
||||
|
||||
|
||||
def digest(value) -> str:
|
||||
return hashlib.sha256(encoded(value)).hexdigest()
|
||||
|
||||
|
||||
class ResumeMismatch(ValueError):
|
||||
def __init__(self, parameters):
|
||||
self.parameters = parameters
|
||||
super().__init__("resume_inputs_changed")
|
||||
|
||||
|
||||
class Stopped(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def external(root: Path, path: Path) -> Path:
|
||||
path = path.expanduser().resolve()
|
||||
# Also reject the actual checkout when a different --root is supplied.
|
||||
if any(path.is_relative_to(p.resolve()) for p in (root, Path(__file__).resolve().parents[2])):
|
||||
raise ValueError("cache_inside_repository")
|
||||
return path
|
||||
|
||||
|
||||
def atomic(path: Path, payload: bytes) -> None:
|
||||
temporary = path.with_suffix(path.suffix + ".tmp")
|
||||
with temporary.open("wb") as handle:
|
||||
os.chmod(temporary, 0o600)
|
||||
handle.write(payload)
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
temporary.replace(path)
|
||||
sync_directory(path.parent)
|
||||
|
||||
|
||||
def sync_directory(path: Path) -> None:
|
||||
fd = os.open(path, os.O_RDONLY | os.O_DIRECTORY)
|
||||
try:
|
||||
os.fsync(fd)
|
||||
finally:
|
||||
os.close(fd)
|
||||
|
||||
|
||||
class Journal:
|
||||
def __init__(self, path: Path, *, max_hours=None, clock=None, rate_limit=None):
|
||||
if max_hours is not None and (not math.isfinite(max_hours) or max_hours < 0):
|
||||
raise ValueError("invalid_max_hours")
|
||||
self.path = path
|
||||
self.clock = clock or time.monotonic
|
||||
self.started = self.clock()
|
||||
self.max_seconds = None if max_hours is None else max_hours * 3600
|
||||
self.rate_limit = rate_limit or (lambda: {})
|
||||
self.rows = {stage: {} for stage in STAGES}
|
||||
self.elapsed_before = 0.0
|
||||
self.last = None
|
||||
self.previous_rate = {}
|
||||
|
||||
@contextmanager
|
||||
def open(self, parameters):
|
||||
self.path.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
with (self.path / "LOCK").open("a") as lock:
|
||||
try:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except BlockingIOError:
|
||||
raise ValueError("journal_busy") from None
|
||||
metadata = self.path / "inputs.json"
|
||||
if metadata.exists():
|
||||
old = json.loads(metadata.read_bytes())
|
||||
# Only current, fixed parameter labels can enter stdout.
|
||||
changed = [key for key in parameters if old.get(key) != parameters[key]]
|
||||
if old.keys() != parameters.keys():
|
||||
changed.append("parameter_schema")
|
||||
if changed:
|
||||
raise ResumeMismatch(changed)
|
||||
else:
|
||||
if any(self.path.glob("*.jsonl")):
|
||||
raise ValueError("journal_missing_inputs")
|
||||
atomic(metadata, encoded(parameters))
|
||||
for stage in STAGES:
|
||||
path = self.path / (stage + ".jsonl")
|
||||
if not path.exists():
|
||||
continue
|
||||
with path.open("rb") as handle:
|
||||
for line in handle:
|
||||
# Fail closed on torn/corrupt writes; never silently lose a completed unit.
|
||||
if not line.endswith(b"\n"):
|
||||
raise ValueError("journal_incomplete_record")
|
||||
row = json.loads(line)
|
||||
if row["id"] in self.rows[stage]:
|
||||
raise ValueError("journal_duplicate_unit")
|
||||
self.rows[stage][row["id"]] = row["value"]
|
||||
if row["elapsed_seconds"] >= self.elapsed_before:
|
||||
self.elapsed_before = row["elapsed_seconds"]
|
||||
self.last = {"stage": stage, "id": row["id"]}
|
||||
self.previous_rate = row["rate_limit"]
|
||||
self.progress("sourcing")
|
||||
yield self
|
||||
|
||||
def boundary(self):
|
||||
reason = None
|
||||
if (self.path / "STOP").exists():
|
||||
reason = "STOP"
|
||||
elif self.max_seconds is not None and self.clock() - self.started >= self.max_seconds:
|
||||
reason = "max_hours"
|
||||
if reason:
|
||||
self.progress("stopped", reason=reason)
|
||||
raise Stopped(reason)
|
||||
|
||||
def elapsed(self):
|
||||
return self.elapsed_before + self.clock() - self.started
|
||||
|
||||
def progress(self, status, **extra):
|
||||
atomic(
|
||||
self.path / "progress.json",
|
||||
encoded(
|
||||
{
|
||||
"status": status,
|
||||
"counts": {stage: len(rows) for stage, rows in self.rows.items()},
|
||||
"last_unit": self.last,
|
||||
"rate_limit": self.rate_limit() or self.previous_rate,
|
||||
"elapsed_seconds": self.elapsed(),
|
||||
**extra,
|
||||
}
|
||||
),
|
||||
)
|
||||
|
||||
def put(self, stage, key, value):
|
||||
if key in self.rows[stage]:
|
||||
raise ValueError("journal_duplicate_unit")
|
||||
row = {"id": key, "value": value, "elapsed_seconds": self.elapsed(), "rate_limit": self.rate_limit() or self.previous_rate}
|
||||
path = self.path / (stage + ".jsonl")
|
||||
with path.open("ab") as handle:
|
||||
os.chmod(path, 0o600)
|
||||
handle.write(encoded(row) + b"\n")
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
sync_directory(self.path)
|
||||
self.rows[stage][key] = value
|
||||
self.last = {"stage": stage, "id": key}
|
||||
self.progress("sourcing")
|
||||
|
||||
|
||||
class ResolvedGitHub:
|
||||
"""Persist successful repository lookups and 404s used by both eligibility and the fix index."""
|
||||
|
||||
def __init__(self, github, journal):
|
||||
self.github, self.journal = github, journal
|
||||
|
||||
def __getattr__(self, name):
|
||||
return getattr(self.github, name)
|
||||
|
||||
def get_repo(self, name):
|
||||
from .eligibility import RepositoryNotFound, normalize
|
||||
|
||||
key = normalize(name)
|
||||
if key not in self.journal.rows["resolution"]:
|
||||
self.journal.boundary()
|
||||
try:
|
||||
value = self.github.get_repo(key)
|
||||
except RepositoryNotFound:
|
||||
value = None
|
||||
self.journal.put("resolution", key, value)
|
||||
value = self.journal.rows["resolution"][key]
|
||||
if value is None:
|
||||
raise RepositoryNotFound("not_found")
|
||||
return value
|
||||
@@ -0,0 +1,280 @@
|
||||
"""Advisory selection. Network adapters are constructed only by the coordinator CLI."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
import random
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from datetime import date
|
||||
from urllib.parse import urlencode, urlsplit
|
||||
|
||||
from openultrasast.model.taxonomy import load_families
|
||||
|
||||
from .eligibility import RepositoryNotFound, normalize
|
||||
|
||||
ECOSYSTEMS = ("pip", "npm", "maven", "composer")
|
||||
OSV_ECOSYSTEMS = dict(zip(ECOSYSTEMS, ("PyPI", "npm", "Maven", "Packagist"), strict=True))
|
||||
FAMILIES = {"injection", "path", "deserialization", "access_control", "output_encoding", "untrusted_destination", "config_secrets"}
|
||||
PERMISSIVE = {"MIT", "Apache-2.0", "BSD-2-Clause", "BSD-3-Clause", "ISC", "0BSD", "Unlicense", "Zlib", "PostgreSQL", "BSL-1.0"}
|
||||
FIX = re.compile(r"https://github\.com/([\w.-]+/[\w.-]+)/commit/([a-fA-F0-9]{7,40})(?:[/?#].*)?$", re.I)
|
||||
|
||||
|
||||
class Rejected(ValueError):
|
||||
"""Expected eligibility rejection; messages are fixed reason codes, never names."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Candidate:
|
||||
repository: str
|
||||
url: str
|
||||
fix: str
|
||||
parent: str
|
||||
advisory: str
|
||||
ecosystem: str
|
||||
family: str
|
||||
post_cutoff: bool
|
||||
first_affected: str | None
|
||||
license_spdx: str
|
||||
license_class: str
|
||||
license_ref: str
|
||||
license_path: str
|
||||
|
||||
|
||||
def post_cutoff(row: dict) -> bool:
|
||||
return date.fromisoformat(row["published_at"][:10]) > date(2026, 6, 1)
|
||||
|
||||
|
||||
def fix_links(row: dict) -> set[tuple[str, str]]:
|
||||
links = set()
|
||||
for reference in row.get("references", []):
|
||||
url = reference if isinstance(reference, str) else reference.get("url", "")
|
||||
match = FIX.fullmatch(url)
|
||||
if match:
|
||||
links.add((normalize(match[1]), match[2].lower()))
|
||||
return links
|
||||
|
||||
|
||||
def ecosystem(row: dict) -> str | None:
|
||||
values = {v["package"]["ecosystem"].lower() for v in row.get("vulnerabilities", [])}
|
||||
return next((e for e in ECOSYSTEMS if e in values), None)
|
||||
|
||||
|
||||
def ordered_advisories(rows: list[dict], seed: int) -> list[dict]:
|
||||
"""Stable seeded ordering, post-cutoff first in each ecosystem; round-robin strata."""
|
||||
rng = random.Random(seed)
|
||||
groups = []
|
||||
for eco in ECOSYSTEMS:
|
||||
group = sorted((r for r in rows if ecosystem(r) == eco), key=lambda r: r["ghsa_id"])
|
||||
rng.shuffle(group)
|
||||
group.sort(key=lambda r: not post_cutoff(r))
|
||||
groups.append(group)
|
||||
return [group[i] for i in range(max(map(len, groups), default=0)) for group in groups if i < len(group)]
|
||||
|
||||
|
||||
def first_affected(record: dict, row: dict, eco: str) -> str | None:
|
||||
packages = {v["package"]["name"] for v in row["vulnerabilities"] if v["package"]["ecosystem"].lower() == eco}
|
||||
versions = set()
|
||||
for affected in record.get("affected", []):
|
||||
package = affected.get("package", {})
|
||||
if package.get("ecosystem") != OSV_ECOSYSTEMS[eco] or package.get("name") not in packages:
|
||||
continue
|
||||
for span in affected.get("ranges", []):
|
||||
if span.get("type") not in {"ECOSYSTEM", "SEMVER"}:
|
||||
continue
|
||||
for event in span.get("events", []):
|
||||
if event.get("introduced") not in {None, "0"}:
|
||||
versions.add(event["introduced"])
|
||||
if len(versions) > 1:
|
||||
raise Rejected("ambiguous_version_bracket")
|
||||
return next(iter(versions), None)
|
||||
|
||||
|
||||
def license_info(record: dict) -> tuple[str, str]:
|
||||
# Explicit pinned-file SPDX wins. For GitHub-recognized full texts its license endpoint
|
||||
# provides the detected SPDX. Unknown/custom/compound declarations stay private.
|
||||
declared = re.search(r"SPDX-License-Identifier:\s*([^\r\n]+)", record.get("text", ""))
|
||||
spdx = declared[1].strip().removesuffix("*/").strip() if declared else record.get("spdx_id") or "NONE"
|
||||
return spdx, "permissive" if spdx in PERMISSIVE else "private"
|
||||
|
||||
|
||||
def candidate(row: dict, github, osv) -> Candidate:
|
||||
if date.fromisoformat(row["published_at"][:10]) < date(2019, 1, 1):
|
||||
raise Rejected("date")
|
||||
if row.get("type", "reviewed") != "reviewed" or row.get("withdrawn_at"):
|
||||
raise Rejected("advisory_status")
|
||||
eco = ecosystem(row)
|
||||
if eco is None:
|
||||
raise Rejected("ecosystem")
|
||||
taxonomy = load_families()
|
||||
families = {f.id for c in row.get("cwes", []) if (f := taxonomy.family_of_cwe(c["cwe_id"])) and f.id in FAMILIES}
|
||||
if len(families) != 1:
|
||||
raise Rejected("family")
|
||||
links = fix_links(row)
|
||||
if len(links) != 1:
|
||||
raise Rejected("fix_links")
|
||||
name, fix = next(iter(links))
|
||||
repo = github.get_repo(name)
|
||||
url = urlsplit(repo.get("clone_url", ""))
|
||||
if (
|
||||
repo.get("private", True)
|
||||
or not 0 <= repo.get("size", -1) <= 500 * 1024
|
||||
or url.scheme != "https"
|
||||
or url.hostname != "github.com"
|
||||
or url.username
|
||||
or url.password
|
||||
or url.query
|
||||
or url.fragment
|
||||
or normalize(repo["clone_url"]) != normalize(repo["full_name"])
|
||||
):
|
||||
raise Rejected("repository")
|
||||
commit = github.get_commit(repo["full_name"], fix)
|
||||
if len(commit["parents"]) != 1:
|
||||
raise Rejected("parents")
|
||||
parent = commit["parents"][0]["sha"]
|
||||
license_record = github.get_license(repo["full_name"], parent)
|
||||
spdx, license_class = license_info(license_record)
|
||||
return Candidate(
|
||||
normalize(repo["full_name"]),
|
||||
repo["clone_url"],
|
||||
commit["sha"],
|
||||
parent,
|
||||
row["ghsa_id"],
|
||||
eco,
|
||||
next(iter(families)),
|
||||
post_cutoff(row),
|
||||
first_affected(osv.get(row["ghsa_id"]), row, eco),
|
||||
spdx,
|
||||
license_class,
|
||||
parent,
|
||||
license_record.get("path", ""),
|
||||
)
|
||||
|
||||
|
||||
class JSONTransport:
|
||||
"""Coordinator GET transport with bounded retries; all I/O and waits are injectable."""
|
||||
|
||||
def __init__(self, *, opener=None, sleeper=None, clock=None):
|
||||
import time
|
||||
from urllib.request import urlopen
|
||||
|
||||
self.opener = opener or urlopen
|
||||
self.sleeper = sleeper or time.sleep
|
||||
self.clock = clock or time.time
|
||||
self.attempts = 0
|
||||
self.rate_limit = {}
|
||||
|
||||
def get(self, url: str, headers: dict) -> object:
|
||||
from urllib.error import HTTPError
|
||||
from urllib.request import Request
|
||||
|
||||
for attempt in range(5):
|
||||
self.attempts += 1
|
||||
try:
|
||||
with self.opener(Request(url, headers=headers), timeout=60) as response:
|
||||
self.record_rate(getattr(response, "headers", {}))
|
||||
return json.load(response)
|
||||
except HTTPError as exc:
|
||||
self.record_rate(exc.headers)
|
||||
if exc.code == 404:
|
||||
raise RepositoryNotFound("not_found") from None
|
||||
limited = exc.code == 429 or (
|
||||
exc.code == 403 and (exc.headers.get("X-RateLimit-Remaining") == "0" or exc.headers.get("Retry-After"))
|
||||
)
|
||||
transient = exc.code in {500, 502, 503, 504}
|
||||
if attempt == 4 or not (limited or transient):
|
||||
raise ValueError("http_failure") from None
|
||||
try:
|
||||
delay = float(exc.headers.get("Retry-After", "0"))
|
||||
if limited and not delay:
|
||||
delay = max(60, float(exc.headers.get("X-RateLimit-Reset", "0")) - self.clock() + 1)
|
||||
delay = max(delay, 2**attempt)
|
||||
if delay > 3605:
|
||||
raise ValueError("retry_window")
|
||||
except (TypeError, ValueError):
|
||||
raise ValueError("http_retry_failed") from None
|
||||
self.sleeper(delay)
|
||||
except Exception:
|
||||
raise ValueError("http_failure") from None
|
||||
raise ValueError("http_failure")
|
||||
|
||||
def record_rate(self, headers):
|
||||
self.rate_limit = {
|
||||
key: headers.get(header)
|
||||
for key, header in (
|
||||
("remaining", "X-RateLimit-Remaining"),
|
||||
("limit", "X-RateLimit-Limit"),
|
||||
("reset", "X-RateLimit-Reset"),
|
||||
("retry_after", "Retry-After"),
|
||||
)
|
||||
if headers.get(header) is not None
|
||||
}
|
||||
|
||||
|
||||
class GitHub:
|
||||
def __init__(self, token: str, transport=None):
|
||||
if not token:
|
||||
raise ValueError("missing_gh_token")
|
||||
self.transport = transport or JSONTransport()
|
||||
self.headers = {"Authorization": "Bearer " + token, "Accept": "application/vnd.github+json"}
|
||||
self.repos: dict[str, dict] = {}
|
||||
self.calls = 0
|
||||
|
||||
def get(self, path: str):
|
||||
self.calls += 1
|
||||
return self.transport.get("https://api.github.com" + path, self.headers)
|
||||
|
||||
def get_repo(self, name: str) -> dict:
|
||||
name = normalize(name)
|
||||
if name not in self.repos:
|
||||
self.repos[name] = self.get("/repos/" + name)
|
||||
return self.repos[name]
|
||||
|
||||
def get_commit(self, name: str, sha: str) -> dict:
|
||||
return self.get("/repos/" + normalize(name) + "/commits/" + sha)
|
||||
|
||||
def get_license(self, name: str, ref: str) -> dict:
|
||||
try:
|
||||
row = self.get("/repos/" + normalize(name) + "/license?" + urlencode({"ref": ref}))
|
||||
except RepositoryNotFound:
|
||||
return {"spdx_id": "NONE", "text": "", "path": ""}
|
||||
return {
|
||||
"spdx_id": (row.get("license") or {}).get("spdx_id"),
|
||||
"path": row.get("path", ""),
|
||||
"text": base64.b64decode(row.get("content", "")).decode("utf-8", errors="replace"),
|
||||
}
|
||||
|
||||
def advisory_page(self, page: int) -> dict:
|
||||
# Oldest first: newly published advisories do not shift already persisted pages.
|
||||
rows = self.get(
|
||||
"/advisories?" + urlencode({"type": "reviewed", "per_page": 100, "page": page, "sort": "published", "direction": "asc"})
|
||||
)
|
||||
if not isinstance(rows, list):
|
||||
raise ValueError("invalid_advisory_page")
|
||||
return {"rows": rows, "done": len(rows) < 100}
|
||||
|
||||
def advisories(self):
|
||||
# Enumerate every reviewed advisory, including non-candidate fixes.
|
||||
page = 1
|
||||
while True:
|
||||
result = self.advisory_page(page)
|
||||
yield from result["rows"]
|
||||
if result["done"]:
|
||||
break
|
||||
page += 1
|
||||
|
||||
|
||||
class OSV:
|
||||
def __init__(self, transport=None):
|
||||
self.transport = transport or JSONTransport()
|
||||
self.calls = 0
|
||||
|
||||
def get(self, identifier: str) -> dict:
|
||||
if not re.fullmatch(r"GHSA-[\w-]+", identifier):
|
||||
raise ValueError("invalid_advisory_id")
|
||||
self.calls += 1
|
||||
try:
|
||||
return self.transport.get("https://api.osv.dev/v1/vulns/" + identifier, {})
|
||||
except RepositoryNotFound:
|
||||
return {} # No OSV bracket is distinct from a failed service call.
|
||||
@@ -0,0 +1,527 @@
|
||||
"""Draft and coordinator tests contain synthetic identities only and use injected clients."""
|
||||
|
||||
import copy
|
||||
import importlib
|
||||
import sys
|
||||
import tomllib
|
||||
from collections import Counter
|
||||
from contextlib import contextmanager
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
d = importlib.import_module("benchmarks.unseen.draft")
|
||||
s = importlib.import_module("benchmarks.unseen.source")
|
||||
e = importlib.import_module("benchmarks.unseen.eligibility")
|
||||
|
||||
|
||||
def entries():
|
||||
rows = []
|
||||
for i in range(300):
|
||||
changes = [{"kind": "vulnerability", "method": "introducing", "base": "a" * 40, "head": "b" * 40}]
|
||||
changes += [{"kind": "ordinary", "method": "first_parent", "base": f"{j:040x}", "head": f"{j + 20:040x}"} for j in range(12)]
|
||||
rows.append(
|
||||
{
|
||||
"repository": f"fixture/project{i:03d}",
|
||||
"url": f"https://github.com/fixture/project{i:03d}",
|
||||
"ecosystem": s.ECOSYSTEMS[i % 4],
|
||||
"post_cutoff": i % 7 == 0,
|
||||
"license_spdx": "MIT" if i % 2 else "GPL-3.0",
|
||||
"license_class": "permissive" if i % 2 else "private",
|
||||
"license_ref": "b" * 40,
|
||||
"changes": changes,
|
||||
}
|
||||
)
|
||||
return rows
|
||||
|
||||
|
||||
def test_draft_split_digest_and_balanced_slices(tmp_path):
|
||||
rows = entries()
|
||||
assigned = d.assign_slices(rows, seed=19)
|
||||
assert Counter(r["slice"] for r in assigned) == {1: 100, 2: 100, 3: 100}
|
||||
assert assigned == d.assign_slices(list(reversed(rows)), seed=19)
|
||||
strata = Counter((r["ecosystem"], r["post_cutoff"]) for r in assigned)
|
||||
for slice_id in (1, 2, 3):
|
||||
counts = Counter((r["ecosystem"], r["post_cutoff"]) for r in assigned if r["slice"] == slice_id)
|
||||
assert all(abs(counts[key] - total / 3) < 1 for key, total in strata.items())
|
||||
public, private = d.write_draft(tmp_path, assigned, seed=19)
|
||||
public_text = public.read_text()
|
||||
shared = tomllib.loads(public_text)
|
||||
hidden = tomllib.loads(private.read_text())
|
||||
assert len(shared["repository"]) == 300 and len(hidden["repository"]) == 150
|
||||
assert sum(len(r["changes"]) for r in shared["repository"] if "changes" in r) == 1950
|
||||
for row in hidden["repository"]:
|
||||
assert row["url"] not in public_text and row["repository"] not in public_text
|
||||
pointer = next(p for p in shared["repository"] if p["id"] == row["id"])
|
||||
assert set(pointer) == {"id", "sha256", "slice", "private"}
|
||||
assert pointer["sha256"] == d.private_digest(row)
|
||||
changed = copy.deepcopy(row)
|
||||
changed["changes"][0]["head"] = "c" * 40
|
||||
assert d.private_digest(changed) != pointer["sha256"]
|
||||
assert private.is_relative_to(tmp_path / "benchmarks/unseen/private")
|
||||
with pytest.raises(FileExistsError):
|
||||
d.write_draft(tmp_path, assigned, seed=19)
|
||||
|
||||
|
||||
def test_draft_requires_full_size_and_valid_changes():
|
||||
with pytest.raises(ValueError, match="pool_size"):
|
||||
d.assign_slices(entries()[:299], seed=1)
|
||||
rows = entries()
|
||||
rows[0]["changes"].pop()
|
||||
with pytest.raises(ValueError, match="change_count"):
|
||||
d.assign_slices(rows, seed=1)
|
||||
|
||||
|
||||
def test_cache_is_outside_repository_deleted_and_partial(tmp_path):
|
||||
import subprocess
|
||||
|
||||
root, cache = tmp_path / "repo", tmp_path / "cache"
|
||||
root.mkdir()
|
||||
with pytest.raises(ValueError, match="cache_inside_repository"):
|
||||
d.Clones(root, root / "cache")
|
||||
calls = []
|
||||
|
||||
def runner(argv, **kwargs):
|
||||
calls.append(argv)
|
||||
if "clone" in argv:
|
||||
target = Path(argv[-1])
|
||||
(target / ".git").mkdir()
|
||||
(target / ".git/pack").write_bytes(b"123")
|
||||
return subprocess.CompletedProcess(argv, 0, b"", b"")
|
||||
|
||||
clones = d.Clones(root, cache, runner=runner)
|
||||
with clones.open("https://github.com/fixture/project") as git:
|
||||
assert git.path.exists()
|
||||
clone_path = git.path
|
||||
assert not clone_path.exists()
|
||||
assert "--filter=blob:none" in calls[0] and "--no-checkout" in calls[0]
|
||||
assert clones.counts["clones"] == 1 and clones.counts["clone_bytes"] == 3
|
||||
|
||||
|
||||
def test_clone_failure_sanitized_and_cleaned(tmp_path, capsys):
|
||||
import subprocess
|
||||
|
||||
def runner(argv, **kwargs):
|
||||
return subprocess.CompletedProcess(argv, 128, b"", b"private identity")
|
||||
|
||||
clones = d.Clones(tmp_path / "repo", tmp_path / "cache", runner=runner)
|
||||
with pytest.raises(d.InstrumentFailure), clones.open("https://github.com/fixture/project"):
|
||||
pass
|
||||
assert list((tmp_path / "cache").iterdir()) == []
|
||||
assert not capsys.readouterr().out
|
||||
|
||||
|
||||
class FakeAPI:
|
||||
calls = 0
|
||||
|
||||
def __init__(self, count=300):
|
||||
self.count = count
|
||||
self.licenses = []
|
||||
|
||||
def get_repo(self, name):
|
||||
self.calls += 1
|
||||
if name == "fixture/old":
|
||||
name = "fixture/project000"
|
||||
repo = {"full_name": name, "private": False, "size": 2, "clone_url": "https://github.com/" + name}
|
||||
if name == "fixture/fork":
|
||||
repo["source"] = {"full_name": "fixture/project001"}
|
||||
return repo
|
||||
|
||||
def get_commit(self, name, sha):
|
||||
self.calls += 1
|
||||
return {"sha": sha, "parents": [{"sha": "b" * 40}]}
|
||||
|
||||
def get_license(self, name, ref):
|
||||
self.licenses.append((name, ref))
|
||||
self.calls += 1
|
||||
# P is MIT but actual introducing head is GPL: public output must use V.
|
||||
return {"text": "", "spdx_id": "MIT" if ref == "b" * 40 else "GPL-3.0", "path": "LICENSE"}
|
||||
|
||||
def advisory_page(self, page):
|
||||
self.calls += 1
|
||||
rows = list(self.advisories())[(page - 1) * 100 : page * 100]
|
||||
return {"rows": rows, "done": len(rows) < 100}
|
||||
|
||||
def advisories(self):
|
||||
for i in range(self.count):
|
||||
yield {
|
||||
"ghsa_id": f"GHSA-test-test-{i:04d}",
|
||||
"published_at": "2026-06-02T00:00:00Z",
|
||||
"cwes": [{"cwe_id": "CWE-89"}],
|
||||
"vulnerabilities": [{"package": {"ecosystem": "pip", "name": "fixture"}}],
|
||||
"references": [f"https://github.com/fixture/project{i:03d}/commit/" + "a" * 40],
|
||||
}
|
||||
|
||||
|
||||
class FakeOSV:
|
||||
calls = 0
|
||||
|
||||
def get(self, identifier):
|
||||
self.calls += 1
|
||||
return {}
|
||||
|
||||
|
||||
class FakeClones:
|
||||
def __init__(self):
|
||||
self.counts = Counter()
|
||||
self.commands = []
|
||||
|
||||
@contextmanager
|
||||
def open(self, url):
|
||||
self.counts["clones"] += 1
|
||||
yield self
|
||||
|
||||
def run(self, *args):
|
||||
self.commands.append(args)
|
||||
return ""
|
||||
|
||||
|
||||
def test_coordinator_end_to_end_fakes_pinned_license_and_no_names(tmp_path, monkeypatch, capsys):
|
||||
rows = entries()
|
||||
changes = copy.deepcopy(rows[0]["changes"])
|
||||
changes[0]["head"] = "c" * 40
|
||||
monkeypatch.setattr(d, "repository_changes", lambda *a, **kw: (changes, {"security": 4, "eligible": 40}, {"bytes": 12, "files": 1}))
|
||||
used, api, clones = e.UsedSet(), FakeAPI(), FakeClones()
|
||||
report = d.run(tmp_path, github=api, osv=FakeOSV(), clones=clones, used=used, seed=7, known_empty=set(used.sources))
|
||||
assert report["status"] == "draft" and report["changes"] == 3900
|
||||
assert report["license_split"] == {"private": 300}
|
||||
assert report["ordinary_exclusions"]["security"] == 1200
|
||||
assert len([ref for _, ref in api.licenses if ref == "c" * 40]) == 300
|
||||
public = (tmp_path.parent / (tmp_path.name + "-cache") / "journal/draft-p1.toml").read_text()
|
||||
assert "fixture/" not in public
|
||||
import json
|
||||
|
||||
assert "fixture/" not in json.dumps(report)
|
||||
assert capsys.readouterr().out == ""
|
||||
|
||||
|
||||
def test_used_rename_and_fork_resolution_and_shortfall(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(d, "repository_changes", lambda *a, **kw: (entries()[0]["changes"], {}, {"bytes": 1, "files": 1}))
|
||||
used, api, clones = e.UsedSet(), FakeAPI(2), FakeClones()
|
||||
used.add("pairs", {"fixture/old"}, 10)
|
||||
# project000 resolves from used alias; project001 is reached via used fork's source.
|
||||
used.add("pairs", {"fixture/fork"}, 10)
|
||||
report = d.run(tmp_path, github=api, osv=FakeOSV(), clones=clones, used=used, seed=7, known_empty=set(used.sources) - {"pairs"})
|
||||
assert report["status"] == "insufficient_repositories"
|
||||
assert report["rejections"] == {"used_pairs": 2}
|
||||
assert report["repositories"] == 0 and not clones.counts
|
||||
assert not (tmp_path / "benchmarks/unseen/draft-p1.toml").exists()
|
||||
|
||||
|
||||
def test_zero_source_is_instrument_failure_not_rejection(tmp_path, monkeypatch):
|
||||
def unread(*args, **kwargs):
|
||||
raise d.InstrumentFailure("zero_source_bytes")
|
||||
|
||||
monkeypatch.setattr(d, "repository_changes", unread)
|
||||
used = e.UsedSet()
|
||||
report = d.run(tmp_path, github=FakeAPI(1), osv=FakeOSV(), clones=FakeClones(), used=used, seed=1, known_empty=set(used.sources))
|
||||
assert report["status"] == "instrument_failure" and not report["rejections"]
|
||||
|
||||
|
||||
def test_cli_uses_environment_and_injected_stores(tmp_path, monkeypatch, capsys):
|
||||
from openultrasast import config
|
||||
from openultrasast.plane import memory
|
||||
|
||||
seen = []
|
||||
monkeypatch.setenv("GH_TOKEN", "fake-secret")
|
||||
monkeypatch.setattr(config, "load_dotenv", lambda path: None)
|
||||
monkeypatch.setattr(memory, "open_store", lambda value, **kw: seen.append((value, kw)) or object())
|
||||
monkeypatch.setattr(d.eligibility, "build_used", lambda *a, **kw: e.UsedSet())
|
||||
monkeypatch.setattr(d.source, "GitHub", lambda token: seen.append(token) or FakeAPI(0))
|
||||
monkeypatch.setattr(d.source, "OSV", FakeOSV)
|
||||
args = ["--root", str(tmp_path), "--cache", str(tmp_path.parent / "clone-cache")]
|
||||
for label in e.SOURCES:
|
||||
args += ["--known-empty", label]
|
||||
assert d.main(args) == 1
|
||||
output = capsys.readouterr().out
|
||||
assert "instrument_failure" in output and "fake-secret" not in output and "fixture/" not in output
|
||||
assert seen[-1] == "fake-secret"
|
||||
assert all(kwargs == {"read_only": True} for _, kwargs in seen[:-1])
|
||||
|
||||
|
||||
def test_repository_changes_excludes_all_advisory_sites(monkeypatch):
|
||||
calls = []
|
||||
|
||||
class FakeGit:
|
||||
def run(self, *args):
|
||||
calls.append(args)
|
||||
if args[0] == "ls-tree":
|
||||
return "app.py\n"
|
||||
return args[-1] if args[0] == "rev-parse" else ""
|
||||
|
||||
def blob(self, revision, path):
|
||||
return "def handler():\n return 1\n"
|
||||
|
||||
def parents(self, fix):
|
||||
return ["b" * 40]
|
||||
|
||||
candidate = s.Candidate(
|
||||
"fixture/project",
|
||||
"https://github.com/fixture/project",
|
||||
"a" * 40,
|
||||
"b" * 40,
|
||||
"GHSA-test-test-test",
|
||||
"pip",
|
||||
"injection",
|
||||
True,
|
||||
None,
|
||||
"MIT",
|
||||
"permissive",
|
||||
"b" * 40,
|
||||
"LICENSE",
|
||||
)
|
||||
monkeypatch.setattr(d.extract, "introducing", lambda *a: {"head": "c" * 40})
|
||||
monkeypatch.setattr(d.extract, "fix_sites", lambda git, fix, parent, **kw: [{"path": "app.py", "function": fix}])
|
||||
monkeypatch.setattr(d.extract, "ordinary_history", lambda *a: [])
|
||||
|
||||
def draw(rows, fixes, known, **kwargs):
|
||||
assert fixes == {"a" * 40, "d" * 40}
|
||||
assert known == {("app.py", "a" * 40), ("app.py", "d" * 40)}
|
||||
return [], {}
|
||||
|
||||
monkeypatch.setattr(d.extract, "draw_ordinary", draw)
|
||||
_, _, proof = d.repository_changes(FakeGit(), candidate, {"d" * 40}, seed=1)
|
||||
assert proof["bytes"] > 0
|
||||
assert len([c for c in calls if c[0] == "fetch"]) == 2
|
||||
|
||||
|
||||
def test_candidate_404_is_counted_but_service_failure_aborts(tmp_path):
|
||||
class Missing(FakeAPI):
|
||||
def get_commit(self, name, sha):
|
||||
raise e.RepositoryNotFound("not_found")
|
||||
|
||||
class Broken(FakeAPI):
|
||||
def get_commit(self, name, sha):
|
||||
raise ValueError("private URL must never escape")
|
||||
|
||||
for api, status in [(Missing(1), "insufficient_repositories"), (Broken(1), "instrument_failure")]:
|
||||
used = e.UsedSet()
|
||||
report = d.run(
|
||||
tmp_path,
|
||||
github=api,
|
||||
osv=FakeOSV(),
|
||||
clones=FakeClones(),
|
||||
used=used,
|
||||
seed=1,
|
||||
known_empty=set(used.sources),
|
||||
journal_path=tmp_path.parent / (tmp_path.name + type(api).__name__) / "journal",
|
||||
)
|
||||
assert report["status"] == status
|
||||
if status == "insufficient_repositories":
|
||||
assert report["rejections"] == {"repository_or_commit_not_found": 1}
|
||||
assert "private URL" not in str(report)
|
||||
|
||||
|
||||
def test_shortfall_keeps_ordinary_exclusion_counts(tmp_path, monkeypatch):
|
||||
def too_few(*args, **kwargs):
|
||||
exc = s.Rejected("ordinary_floor")
|
||||
exc.counts = {"eligible": 39, "size_cap": 10}
|
||||
raise exc
|
||||
|
||||
monkeypatch.setattr(d, "repository_changes", too_few)
|
||||
used = e.UsedSet()
|
||||
report = d.run(tmp_path, github=FakeAPI(1), osv=FakeOSV(), clones=FakeClones(), used=used, seed=1, known_empty=set(used.sources))
|
||||
assert report["ordinary_exclusions"] == {"eligible": 39, "size_cap": 10}
|
||||
|
||||
|
||||
def test_clone_disk_cap_cleans_up(tmp_path):
|
||||
import subprocess
|
||||
|
||||
def runner(argv, **kwargs):
|
||||
(Path(argv[-1]) / "big").write_bytes(b"12345")
|
||||
return subprocess.CompletedProcess(argv, 0, b"", b"")
|
||||
|
||||
clones = d.Clones(tmp_path / "repo", tmp_path / "cache", runner=runner, limit_bytes=4)
|
||||
with pytest.raises(d.InstrumentFailure, match="clone_size_cap"), clones.open("https://github.com/fixture/project"):
|
||||
pass
|
||||
assert list((tmp_path / "cache").iterdir()) == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize("stage,n", [("advisories", 2), ("resolution", 103), ("eligibility", 17), ("extraction", 23), ("completion", 1)])
|
||||
def test_checkpoint_interrupt_resume_matches_uninterrupted(tmp_path, monkeypatch, stage, n):
|
||||
import json
|
||||
|
||||
journal_module = importlib.import_module("benchmarks.unseen.journal")
|
||||
# Large fake populations exercise replay; durability is checked separately below.
|
||||
monkeypatch.setattr(journal_module.os, "fsync", lambda fd: None)
|
||||
root = tmp_path / "repo"
|
||||
full, interrupted = tmp_path / "full/journal", tmp_path / "interrupted/journal"
|
||||
changes = entries()[0]["changes"]
|
||||
monkeypatch.setattr(d, "repository_changes", lambda *a, **kw: (changes, {"eligible": 40}, {"bytes": 12, "files": 1}))
|
||||
used = e.UsedSet()
|
||||
|
||||
def run(path, api=None, clones=None):
|
||||
return d.run(
|
||||
root,
|
||||
github=api or FakeAPI(),
|
||||
osv=FakeOSV(),
|
||||
clones=clones or FakeClones(),
|
||||
used=used,
|
||||
seed=7,
|
||||
known_empty=set(used.sources),
|
||||
journal_path=path,
|
||||
)
|
||||
|
||||
assert run(full)["status"] == "draft"
|
||||
original = d.Journal.put
|
||||
|
||||
def interrupt(self, current, key, value):
|
||||
original(self, current, key, value)
|
||||
if current == stage and len(self.rows[current]) == n:
|
||||
raise KeyboardInterrupt
|
||||
|
||||
monkeypatch.setattr(d.Journal, "put", interrupt)
|
||||
with pytest.raises(KeyboardInterrupt):
|
||||
run(interrupted)
|
||||
assert not (interrupted / "draft-p1.toml").exists()
|
||||
before = {p.name: p.read_bytes() for p in interrupted.glob("*.jsonl")}
|
||||
progress = json.loads((interrupted / "progress.json").read_text())
|
||||
assert progress["counts"][stage] == n
|
||||
assert progress["last_unit"]["stage"] == stage
|
||||
assert progress["elapsed_seconds"] >= 0
|
||||
assert "rate_limit" in progress
|
||||
monkeypatch.setattr(d.Journal, "put", original)
|
||||
clones = FakeClones()
|
||||
assert run(interrupted, clones=clones)["status"] == "draft"
|
||||
assert clones.counts["clones"] == 300 - progress["counts"]["extraction"]
|
||||
for name in ("draft-p1.toml", "private-draft-p1.toml"):
|
||||
assert (full / name).read_bytes() == (interrupted / name).read_bytes()
|
||||
assert all((interrupted / name).read_bytes().startswith(data) for name, data in before.items())
|
||||
assert not root.exists() # no names, manifests, or other records in the checkout
|
||||
api, clones = FakeAPI(), FakeClones()
|
||||
assert run(interrupted, api, clones)["status"] == "draft"
|
||||
assert api.calls == 0 and clones.counts == {}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("parameter", ["seed", "used_set_digest", "clone_limit_bytes", "design_extract"])
|
||||
def test_resume_refuses_changed_inputs_before_network(tmp_path, monkeypatch, parameter):
|
||||
path, root = tmp_path / "cache/journal", tmp_path / "repo"
|
||||
used = e.UsedSet()
|
||||
kwargs = dict(
|
||||
root=root,
|
||||
github=FakeAPI(),
|
||||
osv=FakeOSV(),
|
||||
clones=FakeClones(),
|
||||
used=used,
|
||||
seed=7,
|
||||
known_empty=set(used.sources),
|
||||
journal_path=path,
|
||||
max_hours=0,
|
||||
)
|
||||
assert d.run(**kwargs)["status"] == "stopped"
|
||||
if parameter == "seed":
|
||||
kwargs["seed"] = 8
|
||||
elif parameter == "used_set_digest":
|
||||
used.add("pairs", {"fixture/new"}, 12)
|
||||
elif parameter == "clone_limit_bytes":
|
||||
kwargs["clones"].limit_bytes = 12
|
||||
else:
|
||||
original = d.inputs
|
||||
monkeypatch.setattr(d, "inputs", lambda *a: {**original(*a), "design_extract": "changed"})
|
||||
report = d.run(**kwargs)
|
||||
assert report["status"] == "resume_refused"
|
||||
assert report["changed_parameters"] == [parameter]
|
||||
assert kwargs["github"].calls == 0
|
||||
|
||||
|
||||
def test_stop_between_units_and_progress_rate_state(tmp_path, monkeypatch):
|
||||
import json
|
||||
from types import SimpleNamespace
|
||||
|
||||
path, root = tmp_path / "cache/journal", tmp_path / "repo"
|
||||
api = FakeAPI(1)
|
||||
api.transport = SimpleNamespace(rate_limit={"remaining": "27", "reset": "1234"}, attempts=0)
|
||||
used = e.UsedSet()
|
||||
original = d.Journal.put
|
||||
|
||||
def stop(self, stage, key, value):
|
||||
original(self, stage, key, value)
|
||||
if stage == "eligibility":
|
||||
(self.path / "STOP").touch()
|
||||
|
||||
monkeypatch.setattr(d.Journal, "put", stop)
|
||||
clones = FakeClones()
|
||||
report = d.run(root, github=api, osv=FakeOSV(), clones=clones, used=used, seed=1, known_empty=set(used.sources), journal_path=path)
|
||||
assert report["status"] == "stopped" and report["reason"] == "STOP"
|
||||
assert clones.counts == {}
|
||||
progress = json.loads((path / "progress.json").read_text())
|
||||
assert progress["status"] == "stopped" and progress["counts"]["eligibility"] == 1
|
||||
assert progress["rate_limit"] == api.transport.rate_limit
|
||||
assert not (path / "draft-p1.toml").exists()
|
||||
|
||||
|
||||
def test_max_hours_stops_after_completed_page(tmp_path):
|
||||
import json
|
||||
|
||||
now = [0.0]
|
||||
|
||||
class TimedAPI(FakeAPI):
|
||||
def advisory_page(self, page):
|
||||
result = super().advisory_page(page)
|
||||
now[0] += 3601
|
||||
return result
|
||||
|
||||
path = tmp_path / "cache/journal"
|
||||
api, used = TimedAPI(), e.UsedSet()
|
||||
report = d.run(
|
||||
tmp_path / "repo",
|
||||
github=api,
|
||||
osv=FakeOSV(),
|
||||
clones=FakeClones(),
|
||||
used=used,
|
||||
seed=1,
|
||||
known_empty=set(used.sources),
|
||||
journal_path=path,
|
||||
max_hours=1,
|
||||
clock=lambda: now[0],
|
||||
)
|
||||
assert report["status"] == "stopped" and report["reason"] == "max_hours"
|
||||
assert api.calls == 1
|
||||
progress = json.loads((path / "progress.json").read_text())
|
||||
assert progress["counts"]["advisories"] == 1
|
||||
assert progress["elapsed_seconds"] == 3601
|
||||
|
||||
|
||||
def test_journal_cannot_be_inside_repository(tmp_path):
|
||||
used = e.UsedSet()
|
||||
api = FakeAPI()
|
||||
report = d.run(
|
||||
tmp_path,
|
||||
github=api,
|
||||
osv=FakeOSV(),
|
||||
clones=FakeClones(),
|
||||
used=used,
|
||||
seed=1,
|
||||
known_empty=set(used.sources),
|
||||
journal_path=tmp_path / "journal",
|
||||
)
|
||||
assert report["status"] == "instrument_failure" and api.calls == 0
|
||||
assert not (tmp_path / "journal").exists()
|
||||
|
||||
|
||||
def test_journal_fsync_lock_and_corruption(tmp_path, monkeypatch):
|
||||
import json
|
||||
|
||||
j = importlib.import_module("benchmarks.unseen.journal")
|
||||
original = j.os.fsync
|
||||
synced = []
|
||||
|
||||
def fsync(fd):
|
||||
synced.append(fd)
|
||||
original(fd)
|
||||
|
||||
monkeypatch.setattr(j.os, "fsync", fsync)
|
||||
path = tmp_path / "journal"
|
||||
with j.Journal(path).open({"seed": 1}) as journal:
|
||||
before = len(synced)
|
||||
journal.put("advisories", "1", {"rows": [], "done": True})
|
||||
# Appended JSONL, directory, atomic progress file, directory.
|
||||
assert len(synced) - before == 4
|
||||
assert json.loads((path / "progress.json").read_text())["counts"]["advisories"] == 1
|
||||
with pytest.raises(ValueError, match="journal_busy"), j.Journal(path).open({"seed": 1}):
|
||||
pass
|
||||
with (path / "advisories.jsonl").open("ab") as handle:
|
||||
handle.write(b'{"id":')
|
||||
with pytest.raises(ValueError, match="journal_incomplete_record"), j.Journal(path).open({"seed": 1}):
|
||||
pass
|
||||
@@ -0,0 +1,314 @@
|
||||
"""Offline extraction: an injected local Git client over a synthetic temporary history."""
|
||||
|
||||
import importlib
|
||||
import subprocess
|
||||
import sys
|
||||
from dataclasses import replace
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
e = importlib.import_module("benchmarks.unseen.extract")
|
||||
s = importlib.import_module("benchmarks.unseen.source")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def history(tmp_path):
|
||||
calls = []
|
||||
|
||||
def runner(argv, **kwargs):
|
||||
assert not {"clone", "fetch", "push", "pull"} & set(argv)
|
||||
calls.append(argv)
|
||||
return subprocess.run(argv, **kwargs)
|
||||
|
||||
git = e.Git(tmp_path, runner=runner)
|
||||
git.run("init", "-q")
|
||||
git.run("config", "user.name", "Fixture")
|
||||
git.run("config", "user.email", "[email protected]")
|
||||
|
||||
def commit(text, message="ordinary", extra=None):
|
||||
(tmp_path / "app.py").write_text(text)
|
||||
for path, content in (extra or {}).items():
|
||||
(tmp_path / path).write_text(content)
|
||||
git.run("add", ".")
|
||||
git.run("commit", "-q", "-m", message)
|
||||
return git.run("rev-parse", "HEAD").strip()
|
||||
|
||||
root = commit("def handler(value):\n return value\n")
|
||||
git.run("tag", "v1.0")
|
||||
intro = commit("def handler(value):\n return eval(value)\n")
|
||||
git.run("tag", "v1.1")
|
||||
parent = commit("def handler(value):\n return eval( value )\n", "format")
|
||||
fix = commit("def handler(value):\n return int(value)\n", "security fix")
|
||||
candidate = s.Candidate(
|
||||
"fixture/project",
|
||||
"https://github.com/fixture/project",
|
||||
fix,
|
||||
parent,
|
||||
"GHSA-test-test-test",
|
||||
"pip",
|
||||
"injection",
|
||||
True,
|
||||
"1.1",
|
||||
"MIT",
|
||||
"permissive",
|
||||
parent,
|
||||
"LICENSE",
|
||||
)
|
||||
return git, commit, candidate, root, intro, parent, calls
|
||||
|
||||
|
||||
def test_introducing_with_reformat_walkback(history):
|
||||
git, _, candidate, root, intro, _, calls = history
|
||||
change = e.introducing(git, candidate)
|
||||
assert change["method"] == "introducing"
|
||||
assert (change["base"], change["head"]) == (root, intro)
|
||||
assert change["function"] == "handler"
|
||||
assert change["family"] == "injection" and change["post_cutoff"]
|
||||
assert change["files"][0]["head_lines"] == [[2, 2]]
|
||||
assert change["bias"] and change["advisory"] == candidate.advisory
|
||||
assert any("-w" in args and "-M" in args and "-C" in args for args in calls)
|
||||
assert git.bytes_read > 0
|
||||
|
||||
|
||||
def test_bracket_rejection_uses_fallback(history):
|
||||
git, _, candidate, _, _, parent, _ = history
|
||||
change = e.introducing(git, replace(candidate, first_affected="1.0"))
|
||||
assert change["method"] == "last_touch" and change["head"] == parent
|
||||
assert change["fallback_reasons"]["version_bracket"] >= 1
|
||||
|
||||
|
||||
def test_size_cap_and_merge_rejection(history):
|
||||
git, _, candidate, _, _, _, _ = history
|
||||
|
||||
class Oversize:
|
||||
def __getattr__(self, name):
|
||||
return getattr(git, name)
|
||||
|
||||
def stats(self, base, head):
|
||||
return [("app.py", 2001, 0)]
|
||||
|
||||
assert e.introducing(Oversize(), candidate)["method"] == "last_touch"
|
||||
|
||||
class Merge(Oversize):
|
||||
def parents(self, sha):
|
||||
parents = git.parents(sha)
|
||||
return parents + ["f" * 40] if sha != candidate.fix else parents
|
||||
|
||||
with pytest.raises(s.Rejected, match="no_range"):
|
||||
e.introducing(Merge(), candidate)
|
||||
|
||||
|
||||
def test_pure_addition_blames_function_declaration(history):
|
||||
git, commit, candidate, _, _, _, _ = history
|
||||
git.run("checkout", "-q", candidate.parent)
|
||||
parent = commit("def handler(value):\n return eval(value)\n\ndef added(value):\n return value\n")
|
||||
fix = commit("def handler(value):\n return eval(value)\n\ndef added(value):\n assert value\n return value\n")
|
||||
change = e.introducing(git, replace(candidate, fix=fix, parent=parent, first_affected=None, family="access_control"))
|
||||
assert change["head"] == parent and change["method"] == "introducing"
|
||||
assert change["function"] == "added"
|
||||
|
||||
|
||||
def test_git_failures_are_sanitized_and_not_empty(tmp_path):
|
||||
def failed(argv, **kwargs):
|
||||
return subprocess.CompletedProcess(argv, 128, b"", b"private repository secret")
|
||||
|
||||
with pytest.raises(e.InstrumentFailure, match="git_exit_128"):
|
||||
e.Git(tmp_path, runner=failed).run("log")
|
||||
|
||||
|
||||
def test_walkback_is_bounded_and_file_cap_falls_back(history):
|
||||
git, _, candidate, _, _, parent, _ = history
|
||||
|
||||
class FileCap:
|
||||
def __getattr__(self, name):
|
||||
return getattr(git, name)
|
||||
|
||||
def stats(self, base, head):
|
||||
return [("app.py", 1, 0)] * 51
|
||||
|
||||
result = e.introducing(FileCap(), candidate)
|
||||
assert result["method"] == "last_touch"
|
||||
assert result["fallback_reasons"]["size_cap"]
|
||||
|
||||
class Walk:
|
||||
calls = 0
|
||||
|
||||
def __getattr__(self, name):
|
||||
return getattr(git, name)
|
||||
|
||||
def blame(self, revision, path, line):
|
||||
self.calls += 1
|
||||
return parent, path, 2
|
||||
|
||||
def parents(self, revision):
|
||||
return [parent] if revision != candidate.fix else [candidate.parent]
|
||||
|
||||
walk = Walk()
|
||||
result = e.introducing(walk, candidate)
|
||||
assert walk.calls == 6
|
||||
assert result["fallback_reasons"]["walkback_limit"] == 1
|
||||
|
||||
|
||||
def test_root_rejected_and_head_function_resolved(history):
|
||||
git, _, candidate, root, _, _, _ = history
|
||||
|
||||
class Root:
|
||||
def __getattr__(self, name):
|
||||
return getattr(git, name)
|
||||
|
||||
def blame(self, revision, path, line):
|
||||
return root, path, 1
|
||||
|
||||
result = e.introducing(Root(), candidate)
|
||||
assert result["method"] == "last_touch"
|
||||
assert result["fallback_reasons"]["merge_or_root"] == 1
|
||||
assert result["function_lines"] == [1, 2]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"path,text",
|
||||
[
|
||||
("a.php", "<?php\nfunction handle($x) {\n return $x;\n}\n"),
|
||||
("a.js", "function handle(x) {\n return x;\n}\n"),
|
||||
("a.ts", "const handle = (x) => {\n return x;\n}\n"),
|
||||
("A.java", "class A {\n public String handle(String x) {\n return x;\n }\n}\n"),
|
||||
],
|
||||
)
|
||||
def test_supported_function_boundaries(path, text):
|
||||
ranges = e.functions(path, text)
|
||||
assert len(ranges) == 1 and ranges[0][0] == "handle"
|
||||
assert ranges[0][2] < len(text.splitlines()) + 1
|
||||
|
||||
|
||||
def test_ambiguous_parent_is_not_proof_of_absence(history):
|
||||
git, commit, candidate, _, _, _, _ = history
|
||||
duplicate = (
|
||||
"class A:\n def handler(self, value):\n return eval(value)\n\n"
|
||||
"class B:\n def handler(self, value):\n return value\n"
|
||||
)
|
||||
commit(duplicate)
|
||||
parent = commit("class A:\n def handler(self, value):\n return eval( value )\n")
|
||||
fix = commit("class A:\n def handler(self, value):\n return int(value)\n")
|
||||
selected = replace(candidate, fix=fix, parent=parent, first_affected=None)
|
||||
|
||||
class BlameLast:
|
||||
def __getattr__(self, name):
|
||||
return getattr(git, name)
|
||||
|
||||
def blame(self, revision, path, line):
|
||||
return parent, path, 3
|
||||
|
||||
result = e.introducing(BlameLast(), selected)
|
||||
assert result["method"] == "last_touch"
|
||||
assert result["fallback_reasons"]["ambiguous_parent"] == 1
|
||||
|
||||
|
||||
def ordinary(index, **overrides):
|
||||
row = {
|
||||
"head": f"{index:040x}",
|
||||
"parents": ["f" * 40],
|
||||
"timestamp": index * 100,
|
||||
"message": "update query token session valid",
|
||||
"author": "Human",
|
||||
"files": [{"path": "app.py", "added": 5, "removed": 0, "head_lines": [[2, 6]], "functions": ["ordinary"], "generated": False}],
|
||||
}
|
||||
row.update(overrides)
|
||||
return row
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"message",
|
||||
[
|
||||
"security fix",
|
||||
"CVE-2026",
|
||||
"GHSA-test",
|
||||
"vulnerability",
|
||||
"injection fix",
|
||||
"XSS",
|
||||
"CSRF",
|
||||
"SSRF",
|
||||
"exploit",
|
||||
"sanitize",
|
||||
"escaping",
|
||||
"Revert changes",
|
||||
"This reverts commit abc",
|
||||
"bump package",
|
||||
"upgrade package from 1 to 2",
|
||||
],
|
||||
)
|
||||
def test_ordinary_message_exclusions(message):
|
||||
assert e.ordinary_rejection(ordinary(1, message=message), set(), set()) is not None
|
||||
|
||||
|
||||
def test_narrow_filter_and_every_other_exclusion():
|
||||
assert e.ordinary_rejection(ordinary(1), set(), set()) is None
|
||||
assert e.ordinary_rejection(ordinary(1, parents=["a", "b"]), set(), set()) == "merge_or_root"
|
||||
assert e.ordinary_rejection(ordinary(1, author="dependabot[bot]"), set(), set()) == "dependency"
|
||||
assert e.ordinary_rejection(ordinary(1, author="renovate"), set(), set()) == "dependency"
|
||||
assert e.ordinary_rejection(ordinary(1), {f"{1:040x}"}, set()) == "advisory_fix"
|
||||
assert e.ordinary_rejection(ordinary(1), set(), {("app.py", "ordinary")}) == "vulnerable_function"
|
||||
row = ordinary(1)
|
||||
row["files"][0]["added"] = 1001
|
||||
assert e.ordinary_rejection(row, set(), set()) == "size_cap"
|
||||
for path, reason in [
|
||||
("README.md", "no_source"),
|
||||
("package.json", "manifest_only"),
|
||||
("package-lock.json", "manifest_only"),
|
||||
("vendor/a.py", "generated"),
|
||||
("node_modules/a.js", "generated"),
|
||||
("dist/a.js", "generated"),
|
||||
("app.min.js", "generated"),
|
||||
]:
|
||||
row = ordinary(1)
|
||||
row["files"][0]["path"] = path
|
||||
assert e.ordinary_rejection(row, set(), set()) == reason
|
||||
row = ordinary(1)
|
||||
row["files"][0]["generated"] = True
|
||||
assert e.ordinary_rejection(row, set(), set()) == "generated"
|
||||
|
||||
|
||||
def test_draw_seed_proportional_cells_floor_and_thirds():
|
||||
rows = [ordinary(i) for i in range(120)]
|
||||
for i, row in enumerate(rows):
|
||||
row["files"][0]["added"] = [5, 20, 100, 500][i % 4]
|
||||
drawn, counts = e.draw_ordinary(rows, set(), set(), seed=19)
|
||||
assert len(drawn) == 12 and len({r["head"] for r in drawn}) == 12
|
||||
assert {r["time_third"] for r in drawn} == {0, 1, 2}
|
||||
assert len({(r["time_third"], r["size_bucket"]) for r in drawn}) == 12
|
||||
assert drawn == e.draw_ordinary(list(reversed(rows)), set(), set(), seed=19)[0]
|
||||
assert counts["eligible"] == 120
|
||||
with pytest.raises(s.Rejected, match="ordinary_floor"):
|
||||
e.draw_ordinary(rows[:39], set(), set(), seed=19)
|
||||
# Thirds come from the history's time span, not rank among eligible changes.
|
||||
with pytest.raises(s.Rejected, match="time_thirds"):
|
||||
e.draw_ordinary([ordinary(i) for i in range(40)] + [ordinary(10000, message="security")], set(), set(), seed=1)
|
||||
|
||||
|
||||
def test_git_ordinary_collection_generated_attributes_and_headers(history):
|
||||
git, commit, _, _, _, _, _ = history
|
||||
commit("# Generated by fixture\nx = 2\n", extra={".gitattributes": "other.py linguist-generated=true\n", "other.py": "x = 1\n"})
|
||||
rows = e.ordinary_history(git, "HEAD")
|
||||
assert rows and rows[0]["files"]
|
||||
files = {f["path"]: f for f in rows[0]["files"]}
|
||||
assert files["app.py"]["generated"] and files["other.py"]["generated"]
|
||||
assert files["app.py"]["head_lines"]
|
||||
|
||||
|
||||
def test_attribute_unset_and_mixed_generated_source(history):
|
||||
git, commit, _, _, _, _, _ = history
|
||||
sha = commit("x = 1\n", extra={".gitattributes": "*.py linguist-generated\napp.py -linguist-generated\n"})
|
||||
assert not e.generated(git, sha, "app.py", "x = 1\n")
|
||||
row = ordinary(1)
|
||||
row["files"].append({**row["files"][0], "path": "dist/b.js", "generated": True})
|
||||
assert e.ordinary_rejection(row, set(), set()) is None
|
||||
|
||||
|
||||
def test_unbalanced_cells_remain_proportional_and_seed_changes_draw():
|
||||
rows = [ordinary(i) for i in range(120)]
|
||||
for i, row in enumerate(rows):
|
||||
row["files"][0]["added"] = 5 if i % 4 else 100
|
||||
draw, _ = e.draw_ordinary(rows, set(), set(), seed=1)
|
||||
assert len([r for r in draw if r["size_bucket"] == 0]) == 9
|
||||
assert draw != e.draw_ordinary(rows, set(), set(), seed=2)[0]
|
||||
@@ -0,0 +1,182 @@
|
||||
"""Sanitized REST-shaped advisory fixtures; every external client is a fake."""
|
||||
|
||||
import copy
|
||||
import importlib
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
s = importlib.import_module("benchmarks.unseen.source")
|
||||
|
||||
|
||||
def advisory():
|
||||
path = Path(__file__).resolve().parents[1] / "benchmarks/unseen/fixtures/advisory.json"
|
||||
return json.loads(path.read_bytes())
|
||||
|
||||
|
||||
class API:
|
||||
def __init__(self):
|
||||
self.parents = ["b" * 40]
|
||||
self.spdx = "MIT"
|
||||
self.license_text = "SPDX-License-Identifier: MIT"
|
||||
self.calls = []
|
||||
self.repo = {"full_name": "fixture/project", "private": False, "size": 100, "clone_url": "https://github.com/fixture/project.git"}
|
||||
|
||||
def get_repo(self, name):
|
||||
return copy.deepcopy(self.repo)
|
||||
|
||||
def get_commit(self, name, sha):
|
||||
return {"sha": sha, "parents": [{"sha": p} for p in self.parents]}
|
||||
|
||||
def get_license(self, name, ref):
|
||||
self.calls.append((name, ref))
|
||||
return {"text": self.license_text, "spdx_id": self.spdx, "path": "LICENSE"}
|
||||
|
||||
|
||||
class OSV:
|
||||
def get(self, identifier):
|
||||
return {
|
||||
"affected": [
|
||||
{
|
||||
"package": {"ecosystem": "PyPI", "name": "fixture"},
|
||||
"ranges": [{"type": "ECOSYSTEM", "events": [{"introduced": "1.2"}, {"fixed": "1.3"}]}],
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
def test_candidate_cwe_cutoff_bracket_and_pinned_license():
|
||||
api = API()
|
||||
candidate = s.candidate(advisory(), api, OSV())
|
||||
assert candidate.family == "injection" and candidate.post_cutoff
|
||||
assert candidate.first_affected == "1.2"
|
||||
assert candidate.license_class == "permissive"
|
||||
assert api.calls == [("fixture/project", "b" * 40)]
|
||||
raw = advisory()
|
||||
raw["published_at"] = "2026-06-01T12:00:00Z"
|
||||
assert not s.candidate(raw, api, OSV()).post_cutoff
|
||||
raw["published_at"] = "2018-12-31T00:00:00Z"
|
||||
with pytest.raises(s.Rejected, match="date"):
|
||||
s.candidate(raw, api, OSV())
|
||||
|
||||
|
||||
@pytest.mark.parametrize("spdx", ["GPL-3.0", "AGPL-3.0-only", "LGPL-2.1", "NOASSERTION", "", "BUSL-1.1", "MIT OR GPL-3.0"])
|
||||
def test_private_licenses(spdx):
|
||||
api = API()
|
||||
api.spdx = spdx
|
||||
api.license_text = ""
|
||||
assert s.candidate(advisory(), api, OSV()).license_class == "private"
|
||||
|
||||
|
||||
def test_license_file_overrides_api_and_missing_file_is_private():
|
||||
api = API()
|
||||
api.license_text = "SPDX-License-Identifier: GPL-3.0-only"
|
||||
assert s.candidate(advisory(), api, OSV()).license_class == "private"
|
||||
api.license_text = ""
|
||||
api.spdx = "Apache-2.0"
|
||||
assert s.candidate(advisory(), api, OSV()).license_spdx == "Apache-2.0"
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"mutation,reason",
|
||||
[
|
||||
(lambda a, api: api.parents.append("c" * 40), "parents"),
|
||||
(lambda a, api: a["references"].append("https://github.com/fixture/other/commit/" + "d" * 40), "fix_links"),
|
||||
(lambda a, api: a.update(cwes=[{"cwe_id": "CWE-99999"}]), "family"),
|
||||
(lambda a, api: api.repo.update(size=512001), "repository"),
|
||||
(lambda a, api: api.repo.update(private=True), "repository"),
|
||||
(lambda a, api: api.repo.update(clone_url="ssh://fixture/project"), "repository"),
|
||||
],
|
||||
)
|
||||
def test_rejections(mutation, reason):
|
||||
api, raw = API(), advisory()
|
||||
mutation(raw, api)
|
||||
with pytest.raises(s.Rejected, match=reason):
|
||||
s.candidate(raw, api, OSV())
|
||||
|
||||
|
||||
def test_post_cutoff_first_within_ecosystem_is_seeded():
|
||||
rows = [advisory() for _ in range(12)]
|
||||
for i, row in enumerate(rows):
|
||||
row["ghsa_id"] += str(i)
|
||||
if i % 2:
|
||||
row["published_at"] = "2025-01-01T00:00:00Z"
|
||||
ordered = s.ordered_advisories(rows, 17)
|
||||
assert ordered == s.ordered_advisories(list(reversed(rows)), 17)
|
||||
assert all(s.post_cutoff(row) for row in ordered[:6])
|
||||
|
||||
|
||||
def test_http_adapters_with_fake_transport():
|
||||
import base64
|
||||
from urllib.parse import parse_qs, urlsplit
|
||||
|
||||
class Transport:
|
||||
def __init__(self):
|
||||
self.calls = []
|
||||
|
||||
def get(self, url, headers):
|
||||
self.calls.append((url, headers))
|
||||
parsed = urlsplit(url)
|
||||
if parsed.path == "/advisories":
|
||||
page = int(parse_qs(parsed.query)["page"][0])
|
||||
return [advisory()] * (100 if page == 1 else 1)
|
||||
if "/license" in parsed.path:
|
||||
assert parse_qs(parsed.query)["ref"] == ["b" * 40]
|
||||
return {
|
||||
"license": {"spdx_id": "MIT"},
|
||||
"path": "LICENSE",
|
||||
"content": base64.b64encode(b"SPDX-License-Identifier: MIT").decode(),
|
||||
}
|
||||
if "/commits/" in parsed.path:
|
||||
return {"sha": "a" * 40, "parents": [{"sha": "b" * 40}]}
|
||||
if parsed.hostname == "api.osv.dev":
|
||||
return OSV().get("ignored")
|
||||
return API().repo
|
||||
|
||||
transport = Transport()
|
||||
api = s.GitHub("fake-token", transport)
|
||||
assert len(list(api.advisories())) == 101
|
||||
assert s.candidate(advisory(), api, s.OSV(transport)).first_affected == "1.2"
|
||||
assert all(headers["Authorization"] == "Bearer fake-token" for url, headers in transport.calls if "api.github.com" in url)
|
||||
count = api.calls
|
||||
api.get_repo("fixture/project")
|
||||
assert api.calls == count
|
||||
|
||||
|
||||
def test_http_failures_are_not_empty_results():
|
||||
class Transport:
|
||||
def get(self, url, headers):
|
||||
raise ValueError("http_failure")
|
||||
|
||||
with pytest.raises(ValueError, match="http_failure"):
|
||||
list(s.GitHub("fake", Transport()).advisories())
|
||||
with pytest.raises(ValueError, match="http_failure"):
|
||||
s.OSV(Transport()).get("GHSA-test-test-test")
|
||||
|
||||
|
||||
def test_transport_rate_limit_retry_and_sanitized_failure():
|
||||
import io
|
||||
from urllib.error import HTTPError
|
||||
|
||||
waits, attempts = [], []
|
||||
|
||||
def opener(request, timeout):
|
||||
attempts.append(request)
|
||||
if len(attempts) == 1:
|
||||
raise HTTPError(request.full_url, 403, "private message", {"X-RateLimit-Remaining": "0", "X-RateLimit-Reset": "120"}, None)
|
||||
return io.BytesIO(b'{"ok": true}')
|
||||
|
||||
transport = s.JSONTransport(opener=opener, sleeper=waits.append, clock=lambda: 100)
|
||||
assert transport.get("https://example.invalid", {}) == {"ok": True}
|
||||
assert waits == [60] and transport.attempts == 2
|
||||
|
||||
def down(request, timeout):
|
||||
raise HTTPError(request.full_url, 503, "private name", {}, None)
|
||||
|
||||
transport = s.JSONTransport(opener=down, sleeper=waits.append)
|
||||
with pytest.raises(ValueError, match="^http_failure$"):
|
||||
transport.get("https://example.invalid", {})
|
||||
assert transport.attempts == 5
|
||||
Reference in New Issue
Block a user