marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

merge(unseen-eval): groups 1-2 and 4-6 -- used set, sourcing with resume, AX batch dispatcher, replay task, scorer

Browse Source 
Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-10-04 20:28:15 +02:00
co-authored by Claude Opus 5.5
parent 60512fff88 c6e87a8517
commit 5dc86c2f3f
35 changed files with 6689 additions and 328 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/engine-image.yml
+6
View File
@@ -7,6 +7,7 @@ on:
- 'src/**'
- 'benchmarks/learn/engine_trace*.py'
- 'benchmarks/learn/engine_task_entry.py'
- 'benchmarks/unseen/replay_entry.py'
- 'benchmarks/learn/engine_worker_service.py'
- 'Dockerfile'
- 'plane/Dockerfile.engine-task'
@@ -30,6 +31,10 @@ jobs:
dockerfile: Dockerfile
- package: ousast-engine-task
dockerfile: plane/Dockerfile.engine-task
target: engine
- package: ousast-replay-task
dockerfile: plane/Dockerfile.engine-task
target: replay
steps:
- uses: actions/checkout@v5
- uses: docker/setup-buildx-action@v3
@@ -50,6 +55,7 @@ jobs:
with:
context: .
file: ${{ matrix.dockerfile }}
target: ${{ matrix.target }}
platforms: linux/amd64
push: true
tags: |
.gitignore
+3
View File
@@ -36,3 +36,6 @@ workspace/
# MkDocs build output (mkdocs build)
site/
# Unseen evaluation pointers for non-permissive repositories.
benchmarks/unseen/private/
.kiro/specs/unseen-repo-evaluation/tasks.md
+84 -21
View File
@@ -29,8 +29,8 @@ Execution rules:
- **Instrument first** (AGENTS.md). A zero, an empty draft or a fast run is treated as an instrument failure until
bytes read and wall time prove otherwise.
- [ ] 1. Used set and guard (design 1.2, 6; increment 1)
- [ ] 1.1 Used-set builder
- [x] 1. Used set and guard (design 1.2, 6; increment 1)
- [x] 1.1 Used-set builder
- `benchmarks/unseen/eligibility.py`: normalises names (lower case, no `.git`, no trailing slash) and builds the
used set from every source of design table 1.2: pair catalogs, recipes and datasets under `benchmarks/pairs/`
plus the local pointer-pair manifests as `benchmarks/pairs/catalog_gen.py` reads them; `population-v1.toml`
@@ -43,14 +43,14 @@ Execution rules:
and trailing-slash variants collapse; stdout and the output JSON contain no name; a v3-shaped file planted in
the synthetic tree is not opened (open is patched to fail on it).
- _Requirements: 1.1_
- [ ] 1.2 Canonical resolution, renames and forks
- [x] 1.2 Canonical resolution, renames and forks
- `eligibility.py`: `resolve(candidate, api)` with an injected GitHub client returns the canonical `full_name`,
former names (redirects) and the fork network (`parent`, `source`). A candidate is ineligible when any of them
is in the used set; one candidate per fork network is kept.
- Tests with a fake client: a planted used repository, a renamed one and a fork of a used one are each
rejected; two forks of one network keep one; the rejection reason is a source label, not a name.
- _Requirements: 1.1_
- [ ] 1.3 Guard extended to pools
- [x] 1.3 Guard extended to pools
- `tests/test_independent_population.py`: the sweep becomes a list of (reservation directory, names):
`benchmarks/independent/` for populations, `benchmarks/unseen/` for pools (public manifests, plus
`benchmarks/unseen/private/*.toml` when present). The sweep also covers `plane/`. New tests: pool and
@@ -62,13 +62,13 @@ Execution rules:
- Evidence: the guard is green with an empty `benchmarks/unseen/`, and red when a test fixture names a pool
repository in `benchmarks/measurements/` (shown once in the record).
- _Requirements: 1.2, 1.3, 1.4_
- [ ] 1.4 Usage ledger
- [x] 1.4 Usage ledger
- `benchmarks/unseen/ledger.py`: append-only `usage-pN.jsonl`, one row per use (slice, decision or experiment
id, purpose `baseline | exploratory | informed | qualifies`, date, result digest, freeze digest). It refuses a
`qualifies` row for a change whose slice already has an `informed` row for it, and refuses to rewrite rows.
- `tests/test_unseen_ledger.py`: append, the refusal, a rewritten row detected by a digest chain.
- _Requirements: 5.2_
- [ ] 1.5 Used-set measurement and record 01 **(coordinator, network)**
- [x] 1.5 Used-set measurement and record 01 **(coordinator, network)**
- Run `eligibility.py` against the real sources, the S3 memory store included (read only).
- Instrument: each source reports files read and bytes; a source with zero rows fails the run unless it is
known empty.
@@ -77,7 +77,7 @@ Execution rules:
- _Requirements: 1.1, 1.3_
- [ ] 2. Sourcing and extraction (design 1.1, 1.3, 1.4, 2, 3; increment 2)
- [ ] 2.1 Advisory candidates and repository filters
- [x] 2.1 Advisory candidates and repository filters
- `benchmarks/unseen/source.py`: from GHSA advisory JSON (pip, npm, maven, composer; published 2019 or later)
keep those whose CWE maps to a family through `src/openultrasast/model/taxonomy.py` and that link one fix
commit with one parent in one repository; attach the OSV first affected version; flag `post_cutoff` when
@@ -87,7 +87,7 @@ Execution rules:
multi-repository fixes dropped, cutoff flag, licence classing (GPL, AGPL, LGPL, none and source-available go
private).
- _Requirements: 1.4, 2.1_
- [ ] 2.2 Vulnerability-introducing range
- [x] 2.2 Vulnerability-introducing range
- `benchmarks/unseen/extract.py`: the `introducing` method of design 2.1 (blame of the sink line, or the
declaration for an absence bug; absence at V's first parent; no merge or root; at most 50 files and 2,000
lines; OSV version bracket when present; walk-back of at most five steps) and the `last_touch` fallback. Each
@@ -96,7 +96,7 @@ Execution rules:
- `tests/test_unseen_extract.py` on a synthetic repository built in a temp dir: accepted V; a reformat walked
back; a merge rejected; size cap; bracket rejection; fallback chosen when no V passes.
- _Requirements: 2.1_
- [ ] 2.3 Ordinary-commit draw
- [x] 2.3 Ordinary-commit draw
- `extract.py`: first-parent, non-merge commits touching an analysed language; the exclusions of design 2.2
(security words of the narrow pattern, reverts, bot and bump commits, manifest- or lock-only, generated or
vendored, advisory fix commits, commits touching a known vulnerable function, more than 1,000 source lines,
@@ -105,7 +105,7 @@ Execution rules:
- Tests: each exclusion; the narrow pattern keeps "query", "token", "session"; allocation and the seed
reproduce the same draw; the 40-commit floor.
- _Requirements: 2.2, 2.3_
- [ ] 2.4 Draft manifest and slices
- [x] 2.4 Draft manifest and slices
- `benchmarks/unseen/draft.py`: writes `benchmarks/unseen/draft-p1.toml` (permissive entries by name) and
`benchmarks/unseen/private/draft-p1.toml` (the rest); the public file holds an opaque id and
sha256(URL, commits) per private entry. Seeded shuffle into three slices of 100 repositories, stratified by
@@ -158,7 +158,7 @@ Execution rules:
- _Requirements: 1.1, 1.2, 1.3_
- [ ] 4. Dispatcher (design 5.3; increment 4)
- [ ] 4.1 Items, workloads and the AX lane
- [x] 4.1 Items, workloads and the AX lane
- `benchmarks/ax/batch.py`: `Item` (id, input objects, result object name, command, extra env, input digest),
`Workload` (image digest, allowed URL env names, deadline, result validator) and the `ax` lane, moved from
`benchmarks/learn/engine_trace_ax.py`: manifest render, egress readiness, presigned GET and PUT as `.dat`,
@@ -167,7 +167,7 @@ Execution rules:
- `tests/test_ax_batch.py` with a fake `ax` CLI and a fake store: manifest holds only the allowed URL env
names and no credential; names carry the `ousast-engine-` prefix; cleanup on success and failure.
- _Requirements: 4.1, 4.2, 4.5_
- [ ] 4.2 Scheduler, fallback and resume
- [x] 4.2 Scheduler, fallback and resume
- `batch.py`: two AX lanes and one VM `docker` lane by default (one container at a time); sandbox death retried
once under a new name, a second death or a JVM OOM goes to the VM fallback queue ahead of overflow;
`progress.json` written by the scheduler only; a `STOP` file halts launches; resume skips items whose stored
@@ -175,7 +175,7 @@ Execution rules:
- Tests with fake lanes: retry then fallback; OOM straight to fallback; resume skips; `STOP`; cleanup failure
halts; never more than two AX tasks in flight.
- _Requirements: 4.3_
- [ ] 4.3 Engine pass on the dispatcher
- [x] 4.3 Engine pass on the dispatcher
- `benchmarks/learn/engine_trace_ax.py` and `engine_trace.run_plan` become an engine `Workload` over `batch.py`;
no behaviour change.
- Observable: `tests/test_ax_executor.py` and `tests/test_engine_trace.py` pass with their assertions unchanged
@@ -188,14 +188,14 @@ Execution rules:
- _Requirements: 4.3_
- [ ] 5. Replay task (design 5.1, 5.2; increment 5)
- [ ] 5.1 Image stage
- [x] 5.1 Image stage
- `plane/Dockerfile.engine-task`: second stage `FROM engine AS replay` adding `git`, the package with its
runtime dependencies and `benchmarks/unseen/replay_entry.py`; smoke `ousast pre-push --help`. The root image
stays `ghcr.io/norandom/ax-task-runner:v0.3.1`; the engine stage is unchanged.
- `tests/test_unseen_image.py`: parses the Dockerfile; the `replay` stage builds from `engine`; the root image
pin; the engine stage's text is byte-identical to before.
- _Requirements: 4.1_
- [ ] 5.2 Task entry
- [x] 5.2 Task entry
- `benchmarks/unseen/replay_entry.py`, standard library only, reusing `engine_task_entry.transfer`: GET the spec;
`git init` and depth-1 fetches of head and base over HTTPS (180 s bound), verified with `git cat-file -e`; run
`ousast pre-push` with the spec's hook flags (defaults, or `--deadline 300` for the long-deadline arm), heap
@@ -206,7 +206,7 @@ Execution rules:
fields; a missing commit, a failed fetch and a hook crash each give a failure record; zero bytes read and a
replay faster than JVM start are flagged `suspect_fast`.
- _Requirements: 3.1, 4.1, 4.2_
- [ ] 5.3 Replay workload
- [x] 5.3 Replay workload
- `benchmarks/unseen/replay.py`: builds `Item`s from a frozen slice and an arm (`default` or `long_deadline`);
the spec object carries the repository URL so names stay out of AX status and telemetry; task names
`ousast-engine-replay-<slug>-<digest>` with an opaque slug; the validator rejects a record without
@@ -223,28 +223,28 @@ Execution rules:
exit codes, the quick-tier comparison.
- _Requirements: 4.1, 4.2, 4.5_
- [ ] 6. Scorer (design 3, 4, 7.1, 7.3; increment 6)
- [ ] 6.1 Coverage states
- [x] 6. Scorer (design 3, 4, 7.1, 7.3; increment 6)
- [x] 6.1 Coverage states
- `benchmarks/unseen/score.py`: each record gets one state (`instrument_failure`, `unanalysable`, `quick_only`,
`engine_covered`) from `instrument.json` and the artifact; `suspect_fast` goes to failure; a slice above 5%
`instrument_failure` after re-runs is refused.
- `tests/test_unseen_score.py` on synthetic records: every state; the refusal; a failure never counts as quiet
or missed.
- _Requirements: 3.3_
- [ ] 6.2 Catch and false-alarm rules
- [x] 6.2 Catch and false-alarm rules
- `score.py`: family match (engine family, quick-rule CWE through `taxonomy.py`), the location table of design
4.1, all three streams with the stream named per catch; false alarm on any finding of any stream; the
alerts-only rate.
- Tests: right family wrong place misses; `access_control` on the handler; `config_secrets` file scope; a
quick-rule-only catch; one advisory finding on an ordinary change is a false alarm.
- _Requirements: 3.2_
- [ ] 6.3 Rates, intervals and breakdowns
- [x] 6.3 Rates, intervals and breakdowns
- `score.py`: Wilson 95% intervals; repository-cluster bootstrap for false alarms; ICC; breakdowns per family,
method, `post_cutoff`, stream, coverage state and the confirmed subset.
- Tests: the n values of design section 3 (315, 483, 756, 62, 88, 97) reproduced; bootstrap seeded; ICC on a
synthetic clustered set.
- _Requirements: 2.3, 3.3_
- [ ] 6.4 Budgets, paired tests, the gate and record 06
- [x] 6.4 Budgets, paired tests, the gate and record 06
- `score.py`: per-budget thresholds (1%, 5%, 10%), labelled `exploratory` in-slice and `fixed` when supplied;
a binary arm's single operating point with the budgets it meets; exact two-sided McNemar for catch and for
false alarm; `gate(A, B)` returning `pass | regression | not_run` per design 7.3.
@@ -319,3 +319,66 @@ Execution rules:
false-alarm rates, coverage per state, spend; `exploratory` rows in `usage-p1.jsonl`.
- Observable: one committed table of arms on slice 1; G1's table exists.
- _Requirements: 3.4, 5.2, 5.3_
## Implementation Notes
- 2026-10-04: Tasks 1.1–1.4 implemented offline and independently reviewed. Task 1.5 remains for the
coordinator; no networked measurement or commit was performed. Command and API contracts:
`benchmarks/unseen/README.md`. Offline evidence:
`benchmarks/measurements/2026-10-04-unseen-01-offline/record.json`.
- Population guard scope is preserved under the maintainer's hard rule: benchmarks/src/tests for
populations, plus plane for pools. Each directory exemption is reservation-specific, including private
pool manifests. Extending population scope to plane exposed 30 existing references; no historical
records were changed or exempted inside the existing population scope.
- S3 inventory opens through `open_store(read_only=True)` because ordinary store construction writes a
Select capability probe. Default store behavior is unchanged; inventory uses LIST/GET only.
- The freeze writer must match the canonical TOML digest encoding documented in the pool README and
exercised by `test_independent_population.py`; final pool discovery does not trust mutable status.
### Group 2 implementation evidence (2026-10-04)
Tasks 2.1–2.4 are implemented offline and independently reviewed. Task 2.5 is
`python -m benchmarks.unseen.draft --root "$PWD" --s3-store s3:// --seed 20260601`
with the documented PATH/PYTHONPATH and GH_TOKEN environment. No networked draw,
label check, group-2 measurement record, guard bisection, or commit was performed.
The group header remains pending its coordinator exits 2.5/2.6.
The command rebuilds the group-1 used set, resolves renamed/fork repositories,
uses disposable external blobless clones, re-reads licences at extracted heads,
and emits counts/digests only. The README documents injected clients, method bias,
the hard 300-repository floor, and an explicitly unmeasured resource estimate.
Independent review caught and fixed ambiguous parent-function absence and stale
candidate-404 handling; regression tests cover both.
Validation: requested pytest selection 103 passed, 1 skipped, exit 0 (population-test
output discarded); `ruff check .` exit 0; `ruff format --check .` exit 0 (411 files);
`mypy src/openultrasast` exit 0 (160 source files). The CLI `--help` smoke exited 0.
Source-read instrumentation printed 20,101 bytes for the extraction implementation.
### Groups 4–6 offline implementation (2026-10-04)
Tasks 4.1–4.3, 5.1–5.3 and 6.1–6.4 are implemented and independently reviewed.
Tasks 4.4 and 5.4 remain coordinator-owned network proofs; group headers 4 and 5 stay
pending those exits. No commit, network request, Docker, AX or kubectl command was run.
The sourcing job's draft.py, source.py, extract.py and journal.py are unchanged.
The shared dispatcher retains the engine adapter's historical questions.json object
name to preserve its unchanged assertions; replay uses spec.dat and result.dat.
The engine Dockerfile body is unchanged except its required AS engine alias; CI builds
engine and replay separately. S3Store's read_only class default preserves existing
constructor-bypassing transport fixtures; initialized read-only stores still refuse writes.
Exact coordinator commands and input contracts are in benchmarks/unseen/README.md,
including frozen TOML/private pointer verification, scoring joins, own-history proof
preparation and anonymous measurement-record generation. JVM wall time is the hook's
aggregate scan build/query timing and is labelled as such. Unsupported-only partitions
with explicit language_not_covered evidence do not pretend to have launched a JVM.
Record 06 contains the committed synthetic fixture and scorer-output digests. The
reserved-repository guard was run as a black box with both output streams discarded.
Independent review: APPROVED; all reported integration and coverage defects resolved.
Final verification: requested pytest command 279 passed, 8 skipped, exit 0; separate
`tests/test_ax_batch.py` 5 passed, exit 0; `ruff check .`, `ruff format --check .`,
`mypy src/openultrasast`, and the output-discarded reserved-repository guard all exit 0.
The own-history proof preparer and replay dry run read five items and exit 0.
benchmarks/ax/batch.py
+434
View File
@@ -0,0 +1,434 @@
"""Host-only AX Task transport; credentials never enter task manifests."""
from __future__ import annotations
import hashlib
import ipaddress
import json
import os
import queue
import re
import shutil
import subprocess
import threading
import time
import uuid
from collections import Counter, deque
from concurrent.futures import FIRST_COMPLETED, ThreadPoolExecutor, wait
from dataclasses import dataclass, field
from datetime import timedelta
from pathlib import Path
from urllib.parse import urlsplit
import yaml
from openultrasast.config import load_dotenv
from openultrasast.plane.memory import PRESIGNED_ARCHIVE_SUFFIX, S3Store, open_store
class EgressError(ValueError):
"""Safe diagnostic without presigned URL contents."""
class SandboxFailure(Exception):
"""A terminal AX task without an authoritative result object."""
def public_endpoint(url):
"""Reject endpoints the actor's hostname-based HTTPS egress policy cannot reach."""
parts = urlsplit(url)
host = parts.hostname or ""
try:
ipaddress.ip_address(host)
is_ip = True
except ValueError:
is_ip = False
if (
parts.scheme != "https"
or not host
or is_ip
or "." not in host
or host.endswith((".localhost", ".local", ".internal"))
or parts.port not in (None, 443)
or parts.username
or parts.password
):
raise EgressError(
"AX requires a public HTTPS DNS hostname in S3_ENDPOINT and presigned URLs: "
"the egress policy allows the store hostname on port 443 only (no IP or plain HTTP)"
)
return host.lower()
def task_state(text, name):
"""Accept AX YAML/JSON output and its human-readable tasks table."""
try:
doc = yaml.safe_load(text)
except yaml.YAMLError:
doc = None
rows = doc.get("items", doc.get("tasks", [doc])) if isinstance(doc, dict) else doc if isinstance(doc, list) else []
for row in rows:
if not isinstance(row, dict) or row.get("metadata", {}).get("name", row.get("name")) != name:
continue
status = row.get("status", {})
ip = next(
(status.get(key) or row.get(key) for key in ("workerIP", "workerIp", "worker_ip", "ip") if status.get(key) or row.get(key)),
None,
)
worker = status.get("worker")
if not ip and isinstance(worker, dict):
ip = worker.get("ip") or worker.get("address")
return str(status.get("phase", row.get("phase", ""))), ip
lines = [line.split() for line in text.splitlines() if line.strip()]
if lines:
headers = [re.sub(r"[^a-z]", "", cell.lower()) for cell in lines[0]]
for cells in lines[1:]:
if name not in cells:
continue
row = dict(zip(headers, cells, strict=False))
phase = row.get("phase", row.get("status", ""))
ip = row.get("workerip") or row.get("ip")
if not ip:
for cell in cells:
try:
ipaddress.ip_address(cell)
ip = cell
break
except ValueError:
pass
return phase, None if ip in {"<none>", "-", ""} else ip
listed = isinstance(doc, (dict, list)) or bool(lines and "name" in headers and {"phase", "status"}.intersection(headers))
return "Missing" if listed else "", None
def bounded_read(read, timeout):
"""Do not let SDK retries postpone task teardown beyond the pin deadline.
Only the read runs in a daemon thread; it cannot publish a late result to disk.
No executor shutdown waits for an SDK call whose socket has stopped responding.
"""
if timeout <= 0:
raise TimeoutError("AX result deadline")
replies = queue.Queue(maxsize=1)
def fetch():
try:
replies.put((True, read()))
except Exception as exc:
replies.put((False, exc))
threading.Thread(target=fetch, daemon=True, name="engine-ax-result-read").start()
try:
ok, result = replies.get(timeout=timeout)
except queue.Empty:
raise TimeoutError("AX result deadline") from None
if not ok:
raise result
return result
class AXLane:
def __init__(self, args, workload, store=None, pause=time.sleep, clock=time.monotonic):
self.workload = workload
self.args = args
self.pause, self.clock = pause, clock
self.run = uuid.uuid4().hex
load_dotenv()
self.host = public_endpoint(os.environ.get("S3_ENDPOINT", ""))
# Validate the image before opening a store, whose constructor probes the bucket.
validate_image(workload.image)
self.store = store if store is not None else open_store()
if store is None and not isinstance(self.store, S3Store):
raise ValueError("AX requires OUSAST_MEMORY=s3://bucket[/prefix]")
def __call__(self, item, output, attempt):
attempts = []
try:
done, worker = self._execute_attempt(item, output, attempt, attempts)
except SandboxFailure:
return {"status": "sandbox_failure", "ax_attempts": attempts}
if done is None or done.returncode:
return {"status": "instrument_failure", "ax_attempts": attempts}
record = json.loads((output / "result.json").read_text())
return {**record, "worker_ip": worker, "ax_attempts": attempts}
def ax(self, *args, manifest=None, timeout=30):
env = dict(os.environ)
if self.args.kubeconfig:
env["KUBECONFIG"] = str(Path(self.args.kubeconfig).expanduser())
done = subprocess.run(
[str(Path(self.args.ax_bin).expanduser()), *args, *(["--atespace", self.args.atespace] if args[0] != "apply" else [])],
input=json.dumps(manifest) if manifest is not None else None,
env=env,
capture_output=True,
text=True,
timeout=max(0.01, timeout),
)
if done.returncode:
# AX errors may echo manifest URLs; report only verb and exit code.
raise ValueError(f"ax {args[0]} failed (exit {done.returncode})")
return done
def _execute_attempt(self, item, output, attempt, attempts):
pin_id = item.id
digest = hashlib.sha256(f"{self.run}/{pin_id}/attempt-{attempt}".encode()).hexdigest()[:12]
slug = re.sub(r"[^a-z0-9]+", "-", str(pin_id).lower()).strip("-")[:22]
name = f"ousast-engine-{slug or 'pin'}-{digest}"
key_id = hashlib.sha256(str(pin_id).encode()).hexdigest()
prefix = f"engine-queue/{self.run}/{key_id}/attempt-{attempt}/"
keys = [prefix + item.object_names.get(key, key.lower() + PRESIGNED_ARCHIVE_SUFFIX) for key in item.inputs]
keys.append(prefix + item.result_object)
expires = timedelta(seconds=self.workload.deadline + 600)
worker_ip = None
submitted = False
started = self.clock()
phase = ""
try:
for key, data in zip(keys, item.inputs.values(), strict=False):
self.store._put(key, data)
urls = {env: self.store.presign_get(key, expires) for env, key in zip(item.inputs, keys, strict=False)}
urls["RESULT_URL"] = self.store.presign_put(keys[-1], expires)
if any(public_endpoint(url) != self.host for url in urls.values()):
raise EgressError("AX presign hostname must match S3_ENDPOINT for the store egress policy")
document = manifest(name, self.workload, item, urls, self.args.atespace)
end = self.clock() + self.workload.deadline
# An apply timeout can still have created the task; cleanup must cover it.
submitted = True
self.ax("apply", "-f", "-", manifest=document, timeout=min(30, self.workload.deadline))
while self.clock() < end:
result = bounded_read(lambda: self.store._get(keys[-1]), end - self.clock())
if self.clock() >= end:
return None, worker_ip
phase = ""
try:
state = self.ax("get", "tasks", timeout=min(30, end - self.clock())).stdout
phase, ip = task_state(state, name)
worker_ip = ip or worker_ip
except (ValueError, OSError, subprocess.SubprocessError):
# The object is authoritative even when AX status is temporarily unavailable.
pass
if self.clock() >= end:
return None, worker_ip
if result is not None:
self.workload.validate(result[0], output)
return subprocess.CompletedProcess([], 0, "", ""), worker_ip
if phase.lower() in {"failed", "missing"}:
# The worker may have uploaded while we were reading AX status.
result = bounded_read(lambda: self.store._get(keys[-1]), end - self.clock())
if self.clock() >= end:
return None, worker_ip
if result is not None:
self.workload.validate(result[0], output)
return subprocess.CompletedProcess([], 0, "", ""), worker_ip
raise SandboxFailure
self.pause(min(2, max(0, end - self.clock())))
return None, worker_ip
except (subprocess.TimeoutExpired, TimeoutError):
return None, worker_ip
except SandboxFailure:
raise
except Exception as exc:
# Never expose exception strings which may contain object-scoped bearer URLs.
detail = str(exc) if isinstance(exc, EgressError) else type(exc).__name__
return subprocess.CompletedProcess([], 1, "", "AX transport failed: " + detail), worker_ip
finally:
attempts.append({"name": name, "phase": phase, "seconds": self.clock() - started})
errors = []
if submitted:
try:
self.ax("delete", "task", name)
except Exception as exc:
# A successful list already confirmed a disappeared task is gone.
if phase != "Missing":
errors.append("task: " + type(exc).__name__)
for key in keys:
try:
self.store._delete(key)
except Exception as exc:
errors.append("object: " + type(exc).__name__)
if errors:
raise RuntimeError("AX cleanup failed; stop dispatching: " + ", ".join(errors))
@dataclass
class Item:
id: str
inputs: dict[str, bytes]
result_object: str
command: tuple[str, ...]
extra_env: dict[str, str]
input_digest: str
object_names: dict[str, str] = field(default_factory=dict)
@dataclass(frozen=True)
class Workload:
image: str
url_env: frozenset[str]
deadline: float
validate: object
def validate_image(image):
if not re.fullmatch(r"[^\s@]+@sha256:[0-9a-f]{64}", image):
raise ValueError("AX --image must be pinned by sha256 digest")
def manifest(name, workload, item, urls, atespace="default", *, validate_name=True):
validate_image(workload.image)
if validate_name and (not re.fullmatch(r"ousast-engine-[a-z0-9-]+", name) or len(name.encode()) > 49):
raise ValueError("invalid AX task name")
if set(urls) != workload.url_env:
raise ValueError("unexpected URL environment names")
# Extra env is deliberately a small numeric execution contract, never inherited credentials.
if set(item.extra_env) - {"DEADLINE", "QUESTION_DEADLINE"} or any(
not re.fullmatch(r"[0-9]+(?:\.[0-9]+)?", str(v)) for v in item.extra_env.values()
):
raise ValueError("unexpected task environment")
for url in urls.values():
public_endpoint(url)
env = [{"name": key, "value": str(value)} for key, value in {**urls, **item.extra_env}.items()]
if len(json.dumps(env).encode()) >= 32768:
raise ValueError("AX Task env exceeds the 32 KB limit")
return {
"apiVersion": "ax.io/v1alpha1",
"kind": "Task",
"metadata": {"name": name, "atespace": atespace},
"spec": {"image": workload.image, "env": env, "command": list(item.command)},
}
def write_json(path, record):
path.parent.mkdir(parents=True, exist_ok=True)
temporary = path.with_suffix(".tmp")
temporary.write_text(json.dumps(record, sort_keys=True) + "\n")
temporary.replace(path)
def schedule(pending, lanes, out, execute, complete, *, fallbacks=(), overflow=True, progress=None):
"""The sole progress writer; callbacks carry workload-specific records, never transport."""
if sum(lane in {"ax", "kind"} for lane in lanes) > 2 or lanes.count("docker") > 1:
raise ValueError("at most two AX lanes and one VM lane")
pending, fallbacks = deque(pending), deque(fallbacks)
progress = progress if progress is not None else {"pins_done": 0, "pins_total": len(pending) + len(fallbacks)}
progress["lanes"] = dict(Counter(lanes))
active = {}
failures = False
def persist():
progress.update(pins_in_flight=len(active), active_lanes=dict(Counter(lane for _, lane in active.values())))
if any(lane in {"ax", "kind"} for lane in lanes):
progress["fallbacks_pending"] = len(fallbacks)
write_json(out / "progress.json", progress)
with ThreadPoolExecutor(max_workers=len(lanes)) as pool:
def launch(lane):
if (out / "STOP").exists():
return
if lane == "docker" and fallbacks:
item, previous = fallbacks.popleft()
elif pending and (lane != "docker" or overflow):
item, previous = pending.popleft(), None
else:
return
active[pool.submit(execute, item, lane, previous)] = item, lane
for lane in lanes:
launch(lane)
persist()
try:
while active:
finished, _ = wait(active, return_when=FIRST_COMPLETED)
# Resolve ALL completed futures before launching; cleanup exceptions stop launches.
results = [(active.pop(future), future.result()) for future in finished]
for (item, lane), result in results:
failed, fallback = complete(item, lane, result, progress)
if fallback is not None:
fallbacks.append((item, fallback))
else:
failures |= failed
progress["pins_done"] += 1
for lane, count in (Counter(lanes) - Counter(lane for _, lane in active.values())).items():
for _ in range(count):
launch(lane)
persist()
finally:
persist()
return failures
def dispatch(items, workload, out, executors, *, lanes=("ax", "ax", "docker"), overflow=True):
"""Digest checkpoints, one instrument rerun, sandbox retry, priority VM fallback."""
out.mkdir(parents=True, exist_ok=True)
pending, fallbacks = [], []
def path(item):
return out / (hashlib.sha256(item.id.encode()).hexdigest() + ".json")
for item in items:
previous = json.loads(path(item).read_text()) if path(item).exists() else {}
if previous.get("input_digest") == item.input_digest:
if previous.get("fallback_pending"):
fallbacks.append((item, previous))
continue
if previous.get("done"):
continue
pending.append(item)
def execute(item, lane, previous):
output = out / hashlib.sha256(item.id.encode()).hexdigest()
output.mkdir(exist_ok=True)
attempts = []
for attempt in range(2):
attempt_output = output / ("attempt-" + uuid.uuid4().hex)
attempt_output.mkdir()
record = executors[lane](item, attempt_output, attempt)
for artifact in attempt_output.iterdir():
if artifact.is_file():
shutil.copyfile(artifact, output / artifact.name)
attempts.append({"lane": lane, "status": record.get("status"), "attempt": attempt, "transport": record.get("ax_attempts", [])})
if record.get("status") not in {"sandbox_failure", "instrument_failure"}:
break
return {**record, "attempts": (previous or {}).get("attempts", []) + attempts, "input_digest": item.input_digest, "done": True}
def complete(item, lane, record, progress):
fallback = lane in {"ax", "kind"} and record.get("status") in {"sandbox_failure", "oom"} and "docker" in lanes
record["fallback_pending"] = fallback
write_json(path(item), record)
progress["last_item"] = hashlib.sha256(item.id.encode()).hexdigest()
return record.get("status") not in {"ok", "unanalysable", "quick_only", "engine_covered"}, record if fallback else None
return schedule(pending, lanes, out, execute, complete, fallbacks=fallbacks, overflow=overflow)
class DockerLane(AXLane):
"""VM lane uses the same presigned-object protocol and pinned replay image."""
def _execute_attempt(self, item, output, attempt, attempts):
# Reuse all object/validation/cleanup logic, replacing only the task runner.
import copy
lane = copy.copy(self)
lane.ax = lane._docker_command
lane._docker_name = None
return AXLane._execute_attempt(lane, item, output, attempt, attempts)
def _docker_command(self, *args, manifest=None, timeout=30):
if args[0] == "apply":
self._docker_name = manifest["metadata"]["name"]
command = ["docker", "run", "-d", "--name", self._docker_name, "--entrypoint", manifest["spec"]["command"][0]]
for env in manifest["spec"]["env"]:
command += ["-e", env["name"] + "=" + env["value"]]
command += [manifest["spec"]["image"], *manifest["spec"]["command"][1:], "--heap-profile", "vm"]
elif args[0] == "get":
command = ["docker", "inspect", "--format", "{{.State.Running}} {{.State.ExitCode}}", self._docker_name]
else:
command = ["docker", "rm", "-f", self._docker_name or args[2]]
done = subprocess.run(command, capture_output=True, text=True, timeout=timeout)
if done.returncode:
raise ValueError("docker command failed")
if args[0] == "get":
phase = "Running" if done.stdout.startswith("true") else "Failed"
done.stdout = json.dumps({"metadata": {"name": self._docker_name}, "status": {"phase": phase}})
return done
benchmarks/learn/engine_trace.py
+21 -55
View File
@@ -16,13 +16,13 @@ import os
import re
import shutil
import subprocess
import sys
import tempfile
import textwrap
import threading
import time
import tomllib
from collections import Counter, defaultdict, deque
from concurrent.futures import FIRST_COMPLETED, ThreadPoolExecutor, wait
from collections import Counter, defaultdict
from pathlib import Path
from engine_trace_k8s import IMAGE, Kubernetes
@@ -39,6 +39,9 @@ from openultrasast.plane.memory import FileStore
from openultrasast.plane.tasks.alerts import engine_languages
from openultrasast.preprocess import detect_language
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
from benchmarks.ax.batch import schedule
ROOT = Path(__file__).resolve().parents[2]
DEFAULT_OUT = Path.home() / "ousast-results/plane/engine-trace"
SNAPSHOT_FILES = (
@@ -47,6 +50,7 @@ SNAPSHOT_FILES = (
"benchmarks/learn/engine_trace_k8s.py",
"benchmarks/learn/engine_trace_queue.py",
"benchmarks/learn/engine_trace_ax.py",
"benchmarks/ax/batch.py",
"benchmarks/learn/engine_task_entry.py",
"benchmarks/learn/engine_worker_service.py",
"benchmarks/learn/engine_trace_parse.py",
@@ -791,8 +795,8 @@ def run_plan(plan, source, provenance, inputs, args):
"active_lanes": {},
}
counts = Counter()
pending = deque(plan)
fallbacks = deque()
pending = list(plan)
fallbacks = []
if args.executor in {"ax", "mixed-ax"}:
pending.clear()
for pin in plan:
@@ -803,12 +807,6 @@ def run_plan(plan, source, provenance, inputs, args):
fallbacks.append((pin, old))
else:
pending.append(pin)
active = {}
failures = False
def persist():
progress.update(pins_in_flight=len(active), active_lanes=dict(Counter(lane for _, lane in active.values())))
write_json(args.out / "progress.json", progress)
def execute(pin, lane, attempt=None):
if not any(u["supported"] for u in pin["units"]):
@@ -825,52 +823,20 @@ def run_plan(plan, source, provenance, inputs, args):
options.image = getattr(args, "docker_image", args.image)
return run_pin(pin, source, provenance, inputs, args.out, options)
with ThreadPoolExecutor(max_workers=len(lanes)) as pool:
def complete(pin, lane, failed, progress):
result = json.loads((args.out / pin_name(pin)).read_text())
selected = {u["unit"] for u in pin["units"]}
reasons = [u.get("reason", "") for u in result["units"] if u["unit"] in selected]
sandbox_failed = args.executor == "mixed-ax" and result.get("container_exit") == 1 and "AX sandbox failed twice" in reasons
if lane == "ax" and (sandbox_failed or any("JVM out of memory" in reason for reason in reasons)):
result.update(fallback="vm-sandbox" if sandbox_failed else "vm-oom", fallback_pending=True)
write_json(args.out / pin_name(pin), result)
return False, result
counts.update(u["status"] for u in result["units"] if u["unit"] in selected)
progress.update(last_pin={"repo": pin["repo"], "pin": pin["pin"]}, statuses=dict(counts))
return failed, None
def launch(lane):
if (args.out / "STOP").exists():
return
attempt = None
if lane == "docker" and fallbacks:
pin, attempt = fallbacks.popleft()
elif pending and not (args.executor == "ax" and lane == "docker"):
pin = pending.popleft()
else:
return
active[pool.submit(execute, pin, lane, attempt)] = (pin, lane)
for lane in lanes:
launch(lane)
persist()
while active:
completed, _ = wait(active, return_when=FIRST_COMPLETED)
for future in completed:
pin, lane = active.pop(future)
failed = future.result()
result = json.loads((args.out / pin_name(pin)).read_text())
selected = {u["unit"] for u in pin["units"]}
reasons = [u.get("reason", "") for u in result["units"] if u["unit"] in selected]
sandbox_failed = args.executor == "mixed-ax" and result.get("container_exit") == 1 and "AX sandbox failed twice" in reasons
if lane == "ax" and (sandbox_failed or any("JVM out of memory" in reason for reason in reasons)):
result.update(fallback="vm-sandbox" if sandbox_failed else "vm-oom", fallback_pending=True)
write_json(args.out / pin_name(pin), result)
fallbacks.append((pin, result))
continue
failures |= failed
selected = {u["unit"] for u in pin["units"]}
counts.update(u["status"] for u in result["units"] if u["unit"] in selected)
progress.update(
pins_done=progress["pins_done"] + 1, last_pin={"repo": pin["repo"], "pin": pin["pin"]}, statuses=dict(counts)
)
idle = Counter(lanes) - Counter(lane for _, lane in active.values())
# Give a waiting fallback priority over overflow on the single VM lane.
for lane, count in idle.items():
for _ in range(count):
launch(lane)
if args.executor in {"ax", "mixed-ax"}:
progress["fallbacks_pending"] = len(fallbacks)
persist()
return failures
return schedule(pending, lanes, args.out, execute, complete, fallbacks=fallbacks, overflow=args.executor != "ax", progress=progress)
def main():
benchmarks/learn/engine_trace_ax.py
+27 -238
View File
@@ -1,184 +1,50 @@
"""Host-only AX Task transport; credentials never enter task manifests."""
"""Engine workload adapter for the shared AX transport."""
from __future__ import annotations
import hashlib
import io
import ipaddress
import json
import os
import queue
import re
import shutil
import subprocess
import sys
import tarfile
import tempfile
import threading
import time
import uuid
from datetime import timedelta
from pathlib import Path
from urllib.parse import urlsplit
import yaml
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
from engine_trace_k8s import pack
from openultrasast.config import load_dotenv
from openultrasast.plane.memory import PRESIGNED_ARCHIVE_SUFFIX, S3Store, open_store
class EgressError(ValueError):
"""Safe diagnostic without presigned URL contents."""
class SandboxFailure(Exception):
"""A terminal AX task without an authoritative result object."""
def public_endpoint(url):
"""Reject endpoints the actor's hostname-based HTTPS egress policy cannot reach."""
parts = urlsplit(url)
host = parts.hostname or ""
try:
ipaddress.ip_address(host)
is_ip = True
except ValueError:
is_ip = False
if (
parts.scheme != "https"
or not host
or is_ip
or "." not in host
or host.endswith((".localhost", ".local", ".internal"))
or parts.port not in (None, 443)
or parts.username
or parts.password
):
raise EgressError(
"AX requires a public HTTPS DNS hostname in S3_ENDPOINT and presigned URLs: "
"the egress policy allows the store hostname on port 443 only (no IP or plain HTTP)"
)
return host.lower()
from benchmarks.ax.batch import AXLane, Item, SandboxFailure, Workload, manifest
from benchmarks.ax.batch import task_state as task_state
from openultrasast.plane.memory import open_store as open_store
def task_manifest(name, args, urls):
if not re.fullmatch(r"[^\s@]+@sha256:[0-9a-f]{64}", args.image):
raise ValueError("AX --image must be pinned by sha256 digest")
if set(urls) != {"SOURCE_URL", "QUESTIONS_URL", "RESULT_URL"}:
raise ValueError("AX Task accepts only source, questions and result presigned URLs")
for url in urls.values():
public_endpoint(url)
env = [{"name": key, "value": value} for key, value in urls.items()]
env.extend(
{"name": key, "value": str(value)}
for key, value in {"DEADLINE": args.deadline, "QUESTION_DEADLINE": args.question_deadline}.items()
return manifest(
name, engine_workload(args, lambda *_: None), engine_item({}, "pin", b"", args), urls, args.atespace, validate_name=False
)
if len(json.dumps(env).encode()) >= 32768:
raise ValueError("AX Task env exceeds the 32 KB limit")
return {
"apiVersion": "ax.io/v1alpha1",
"kind": "Task",
"metadata": {"name": name, "atespace": args.atespace},
"spec": {"image": args.image, "env": env, "command": ["python3", "/app/benchmarks/learn/engine_task_entry.py"]},
}
def task_state(text, name):
"""Accept AX YAML/JSON output and its human-readable tasks table."""
try:
doc = yaml.safe_load(text)
except yaml.YAMLError:
doc = None
rows = doc.get("items", doc.get("tasks", [doc])) if isinstance(doc, dict) else doc if isinstance(doc, list) else []
for row in rows:
if not isinstance(row, dict) or row.get("metadata", {}).get("name", row.get("name")) != name:
continue
status = row.get("status", {})
ip = next(
(status.get(key) or row.get(key) for key in ("workerIP", "workerIp", "worker_ip", "ip") if status.get(key) or row.get(key)),
None,
)
worker = status.get("worker")
if not ip and isinstance(worker, dict):
ip = worker.get("ip") or worker.get("address")
return str(status.get("phase", row.get("phase", ""))), ip
lines = [line.split() for line in text.splitlines() if line.strip()]
if lines:
headers = [re.sub(r"[^a-z]", "", cell.lower()) for cell in lines[0]]
for cells in lines[1:]:
if name not in cells:
continue
row = dict(zip(headers, cells, strict=False))
phase = row.get("phase", row.get("status", ""))
ip = row.get("workerip") or row.get("ip")
if not ip:
for cell in cells:
try:
ipaddress.ip_address(cell)
ip = cell
break
except ValueError:
pass
return phase, None if ip in {"<none>", "-", ""} else ip
listed = isinstance(doc, (dict, list)) or bool(lines and "name" in headers and {"phase", "status"}.intersection(headers))
return "Missing" if listed else "", None
def engine_workload(args, validate):
return Workload(args.image, frozenset({"SOURCE_URL", "QUESTIONS_URL", "RESULT_URL"}), args.deadline, validate)
def bounded_read(read, timeout):
"""Do not let SDK retries postpone task teardown beyond the pin deadline.
Only the read runs in a daemon thread; it cannot publish a late result to disk.
No executor shutdown waits for an SDK call whose socket has stopped responding.
"""
if timeout <= 0:
raise TimeoutError("AX result deadline")
replies = queue.Queue(maxsize=1)
def fetch():
try:
replies.put((True, read()))
except Exception as exc:
replies.put((False, exc))
threading.Thread(target=fetch, daemon=True, name="engine-ax-result-read").start()
try:
ok, result = replies.get(timeout=timeout)
except queue.Empty:
raise TimeoutError("AX result deadline") from None
if not ok:
raise result
return result
def engine_item(pin, pin_id, source, args):
return Item(
pin_id,
{"SOURCE_URL": source, "QUESTIONS_URL": json.dumps(pin).encode()},
"result.dat",
("python3", "/app/benchmarks/learn/engine_task_entry.py"),
{"DEADLINE": str(args.deadline), "QUESTION_DEADLINE": str(args.question_deadline)},
"",
{"SOURCE_URL": "source.dat", "QUESTIONS_URL": "questions.json"},
)
class AX:
class AX(AXLane):
def __init__(self, args, store=None, pause=time.sleep, clock=time.monotonic):
self.args = args
self.pause, self.clock = pause, clock
self.run = uuid.uuid4().hex
load_dotenv()
self.host = public_endpoint(os.environ.get("S3_ENDPOINT", ""))
# Validate the image before opening a store, whose constructor probes the bucket.
task_manifest("validate", args, {key: f"https://{self.host}/" for key in ("SOURCE_URL", "QUESTIONS_URL", "RESULT_URL")})
self.store = store if store is not None else open_store()
if store is None and not isinstance(self.store, S3Store):
raise ValueError("AX requires OUSAST_MEMORY=s3://bucket[/prefix]")
def ax(self, *args, manifest=None, timeout=30):
env = dict(os.environ)
if self.args.kubeconfig:
env["KUBECONFIG"] = str(Path(self.args.kubeconfig).expanduser())
done = subprocess.run(
[str(Path(self.args.ax_bin).expanduser()), *args, *(["--atespace", self.args.atespace] if args[0] != "apply" else [])],
input=json.dumps(manifest) if manifest is not None else None,
env=env,
capture_output=True,
text=True,
timeout=max(0.01, timeout),
)
if done.returncode:
# AX errors may echo manifest URLs; report only verb and exit code.
raise ValueError(f"ax {args[0]} failed (exit {done.returncode})")
return done
super().__init__(args, engine_workload(args, lambda *_: None), store, pause, clock)
def receive(self, data, output, pin):
# Validate in isolation: malformed results must never become usable checkpoints.
@@ -220,11 +86,15 @@ class AX:
shutil.copytree(root, output, dirs_exist_ok=True)
def execute(self, checkout, output, pin, pin_id):
import copy
self = copy.copy(self)
self.workload = engine_workload(self.args, lambda data, target: self.receive(data, target, pin))
attempts = []
try:
for attempt in range(2):
try:
return self._execute_attempt(checkout, output, pin, pin_id, attempt, attempts)
return self._execute_attempt(engine_item(pin, pin_id, pack(checkout), self.args), output, attempt, attempts)
except SandboxFailure:
if attempt == 0:
self.pause(10)
@@ -236,84 +106,3 @@ class AX:
record = json.loads(result_path.read_text())
record["ax_attempts"] = attempts
result_path.write_text(json.dumps(record))
def _execute_attempt(self, checkout, output, pin, pin_id, attempt, attempts):
digest = hashlib.sha256(f"{self.run}/{pin_id}/attempt-{attempt}".encode()).hexdigest()[:12]
slug = re.sub(r"[^a-z0-9]+", "-", str(pin_id).lower()).strip("-")[:22]
name = f"ousast-engine-{slug or 'pin'}-{digest}"
key_id = hashlib.sha256(str(pin_id).encode()).hexdigest()
prefix = f"engine-queue/{self.run}/{key_id}/attempt-{attempt}/"
keys = [prefix + suffix for suffix in (f"source{PRESIGNED_ARCHIVE_SUFFIX}", "questions.json", f"result{PRESIGNED_ARCHIVE_SUFFIX}")]
expires = timedelta(seconds=self.args.deadline + 600)
worker_ip = None
submitted = False
started = self.clock()
phase = ""
try:
self.store._put(keys[0], pack(checkout))
self.store._put(keys[1], json.dumps(pin).encode())
urls = {
"SOURCE_URL": self.store.presign_get(keys[0], expires),
"QUESTIONS_URL": self.store.presign_get(keys[1], expires),
"RESULT_URL": self.store.presign_put(keys[2], expires),
}
if any(public_endpoint(url) != self.host for url in urls.values()):
raise EgressError("AX presign hostname must match S3_ENDPOINT for the store egress policy")
manifest = task_manifest(name, self.args, urls)
end = self.clock() + self.args.deadline
# An apply timeout can still have created the task; cleanup must cover it.
submitted = True
self.ax("apply", "-f", "-", manifest=manifest, timeout=min(30, self.args.deadline))
while self.clock() < end:
result = bounded_read(lambda: self.store._get(keys[2]), end - self.clock())
if self.clock() >= end:
return None, worker_ip
phase = ""
try:
state = self.ax("get", "tasks", timeout=min(30, end - self.clock())).stdout
phase, ip = task_state(state, name)
worker_ip = ip or worker_ip
except (ValueError, OSError, subprocess.SubprocessError):
# The object is authoritative even when AX status is temporarily unavailable.
pass
if self.clock() >= end:
return None, worker_ip
if result is not None:
self.receive(result[0], output, pin)
return subprocess.CompletedProcess([], 0, "", ""), worker_ip
if phase.lower() in {"failed", "missing"}:
# The worker may have uploaded while we were reading AX status.
result = bounded_read(lambda: self.store._get(keys[2]), end - self.clock())
if self.clock() >= end:
return None, worker_ip
if result is not None:
self.receive(result[0], output, pin)
return subprocess.CompletedProcess([], 0, "", ""), worker_ip
raise SandboxFailure
self.pause(min(2, max(0, end - self.clock())))
return None, worker_ip
except (subprocess.TimeoutExpired, TimeoutError):
return None, worker_ip
except SandboxFailure:
raise
except Exception as exc:
# Never expose exception strings which may contain object-scoped bearer URLs.
detail = str(exc) if isinstance(exc, EgressError) else type(exc).__name__
return subprocess.CompletedProcess([], 1, "", "AX transport failed: " + detail), worker_ip
finally:
attempts.append({"name": name, "phase": phase, "seconds": self.clock() - started})
errors = []
if submitted:
try:
self.ax("delete", "task", name)
except Exception as exc:
# A successful list already confirmed a disappeared task is gone.
if phase != "Missing":
errors.append("task: " + type(exc).__name__)
for key in keys:
try:
self.store._delete(key)
except Exception as exc:
errors.append("object: " + type(exc).__name__)
if errors:
raise RuntimeError("AX cleanup failed; stop dispatching: " + ", ".join(errors))
benchmarks/measurements/2026-10-04-unseen-01-used-set/record.json
+86
View File
@@ -0,0 +1,86 @@
{
"count": 1029,
"guard": "green",
"sha256": "5a644a1e93c68a0a35c794e653286d0532b7eb8d757123ccce5f08b11bf9a4a2",
"sources": {
"harvest": {
"bytes": 501031,
"count": 467,
"files": 3,
"known_empty": false,
"rows": 446,
"skipped_values": 446
},
"measurements": {
"bytes": 21169903,
"count": 278,
"files": 221,
"known_empty": false,
"rows": 376,
"skipped_values": 681
},
"memory_local": {
"bytes": 8084523,
"count": 192,
"files": 776,
"known_empty": false,
"rows": 759,
"skipped_values": 2
},
"memory_s3": {
"bytes": 15928676,
"count": 572,
"files": 1499,
"known_empty": false,
"rows": 1490,
"skipped_values": 2
},
"pairs": {
"bytes": 1680480,
"count": 728,
"files": 28,
"known_empty": false,
"rows": 900,
"skipped_values": 1260
},
"pointers": {
"bytes": 892448,
"count": 537,
"files": 7,
"known_empty": false,
"rows": 586,
"skipped_values": 666
},
"populations": {
"bytes": 32431,
"count": 28,
"files": 2,
"known_empty": false,
"rows": 28,
"skipped_values": 0
},
"sweep": {
"bytes": 34006439,
"count": 1029,
"files": 1619,
"known_empty": false,
"rows": 2155,
"skipped_values": 0
},
"units": {
"bytes": 319641,
"count": 229,
"files": 9,
"known_empty": false,
"rows": 642,
"skipped_values": 0
}
},
"test_exit_code": 0,
"tests": {
"errors": 0,
"failures": 0,
"skipped": 1,
"tests": 41
}
}
benchmarks/measurements/2026-10-04-unseen-06-scorer/record.json
+41
View File
@@ -0,0 +1,41 @@
{
"commit": "48ae918dec25119142ae781607ba99306ae5ec17",
"fixture_bytes": 15374,
"fixture_sha256": "dd3c774a47d553105ed85b9cea8810736c233f870e5a015559a1524d5db5a14d",
"implementation_sha256": {
"benchmarks/ax/batch.py": "37aecac93beb25c02826e81eacaf064fbebb447a998fe139f6d94a07035b8685",
"benchmarks/unseen/replay.py": "32efb3e0b9e9d247f2780b5f3ceeb01b4c2cbda3656c75a741f9925621cfdfbc",
"benchmarks/unseen/replay_entry.py": "994d8e239eb505dc2c5bad4cdb8b5d45967a8a2aa9ef48fb4c2f20ba45736280",
"benchmarks/unseen/score.py": "56e0a7b34482b5d84df39824b8ee447c283e35bd932f38cfe692cf84cb52cef7"
},
"network_proofs": {
"4.4": "coordinator_pending",
"5.4": "coordinator_pending"
},
"scope": "offline synthetic scorer only",
"score_command_exit": 0,
"scorer_output_sha256": "77f154eedb6a4acd098ab63a0b17434291f0d226dabe8ac66f2e0b362154d002",
"scorer_test_exit": 0,
"scorer_tests": 14,
"synthetic_records": 20,
"uncommitted": true,
"validation": {
"generic_dispatcher_tests": {
"exit_code": 0,
"passed": 5
},
"mypy": 0,
"proof_and_scorer_tests": {
"exit_code": 0,
"passed": 16
},
"requested_pytest": {
"exit_code": 0,
"passed": 279,
"skipped": 8
},
"reserved_repository_guard": 0,
"ruff_check": 0,
"ruff_format_check": 0
}
}
benchmarks/unseen/README.md
+291
View File
@@ -0,0 +1,291 @@
# Unseen evaluation, group 1
The coordinator runs task 1.5 from the repository root:
```sh
export PATH=/home/mc/Source/OpenUltraSAST/.venv/bin:$PATH
export PYTHONPATH=$PWD/src
python -m benchmarks.unseen.eligibility --s3-store s3:// --verify-tests --output benchmarks/measurements/2026-10-04-unseen-01-used-set/record.json
```
This is the networked command; implementation tests use fakes. It loads `.env` with
`openultrasast.config.load_dotenv` (exported values win), opens the local FileStore and
S3 MemoryStore through `openultrasast.plane.memory.open_store(..., read_only=True)`,
and uses `GH_TOKEN` for GitHub GET resolution. `s3://` selects `S3_BUCKET`; pass
`--s3-store s3://BUCKET/PREFIX` for a prefixed store and `--local-store file:///PATH`
for a different local store. Read-only S3 opening skips the normal write/Select probe;
collection uses LIST/GET only. Existing normal store opening is unchanged.
Pair catalogs/recipes include local pointer rows (`vendored = false`, including
`fix_repo`). Additional external manifests can be supplied with repeatable
`--pointer-manifest PATH`. Registered units in memory and experiment YAML are followed;
missing units with a registered digest fail. Plans without frozen units are not used data.
Experiment units identify repositories in `group`. Binary measurement artifacts are
read and counted; structured JSON, TOML and multi-document YAML are parsed strictly.
Each source records files/objects read, bytes read, repository-bearing rows (distinct
names per input), and distinct repository counts. The digest is SHA-256 of sorted,
normalized names joined with newlines. Source counts overlap. An empty source fails;
only after establishing it is empty may the coordinator pass `--known-empty LABEL`.
The output records that declaration. Missing/unreadable inputs still fail.
GitHub redirects add the requested and canonical names, plus parent/source fork
identities. Used names are also resolved so an old used name excludes a renamed
candidate. GitHub cannot enumerate all historical names: injected clients may supply
`former_names`; actual historical aliases come from used references and redirects.
A 404 for a historical/synthetic used identity retains its literal identity; other
API failures stop collection. Candidate resolution always fails closed on a 404.
`resolve(candidate, api)` and `rejection(resolution, used, selected_networks)` are
available to the later sourcing task. Names stay in memory; output contains only
counts, labels and SHA-256 digests. Output files are created exclusively, never overwritten.
`--verify-tests` runs the three group-1 test modules with both output streams discarded,
including the existing population guard. Only aggregate test counts and the exit code
enter the record; a failed test prevents the record. V3 paths are excluded before
eligibility reads; only the existing guard handles v3 inside its black-box test process.
Population reservations retain their existing sweep over `benchmarks/`, `src/`, and
`tests/`. Pool reservations also cover `plane/`. Each reservation exempts only its own
directory; population checks include all of `benchmarks/unseen/private/`. This preserves
the existing population behavior while adding pools (the hard rule for this increment).
All `pool-pN.toml` files require a matching `freeze-pN.json`; drafts use `draft-pN.toml`.
The freeze digest is SHA-256 of parsed public TOML encoded as UTF-8 JSON with sorted
keys, compact separators and `ensure_ascii=False`. Private entries must be represented
by opaque digests in that public manifest, as task 3.2 requires.
The ledger exposes `append(path, row)` and `read(path)`. Rows contain `slice`, `decision`
(an opaque decision or experiment label), `purpose`, ISO `date`, `result_digest`, and
`freeze_digest`. Appends lock and validate the existing chain, reject duplicates or a
changed freeze, and reject qualification of an informed slice for the same decision.
Keep the published final digest to authenticate history: the chain detects row edits,
but cannot independently prove that an entire history was not replaced or truncated.
## Sourcing and extraction (tasks 2.1–2.4)
The coordinator's networked task **2.5** is now:
```sh
export PATH=/home/mc/Source/OpenUltraSAST/.venv/bin:$PATH
export PYTHONPATH=$PWD/src
python -m benchmarks.unseen.draft --root "$PWD" --s3-store s3:// --seed 20260601 \
--cache /home/mc/ousast-results/plane/unseen-draft --max-hours 8
```
Export `GH_TOKEN` first, or supply it through the root `.env` (exported values win).
The same read-only local/S3 stores, `--pointer-manifest`, and explicit `--known-empty`
options as task 1.5 apply. Missing or unreadable sources fail. The used set is rebuilt
and canonicalized before candidate selection, including redirects and fork networks.
No population v3 file is opened and no guard/bisection is run by this command; that is
reserved for group 3. An existing draft is never overwritten.
The command writes `--cache/journal/draft-p1.toml` and
`--cache/journal/private-draft-p1.toml` only after all sourcing stages complete.
Both stay outside the repository, with mode 0600. The public-format file contains permissive entries
and, for every private entry, only an opaque ID, slice, private marker, and SHA-256
of canonical JSON containing its URL and ordered `[base, head]` pairs. Private entries
carry the complete records, including licences, in the private-format file.
Licensing is re-read through GitHub's licence endpoint **at the actual extracted
vulnerability replay head**; the candidate's earlier fix-parent classification is
provisional. An explicit SPDX declaration in the pinned file takes precedence over
the API's classification. Missing/unknown/compound licences remain private.
Three seeded slices each contain 100 repositories, one vulnerability range and twelve
ordinary changes per repository. Each ecosystem/post-cutoff stratum differs from its
proportional per-slice target by less than one repository. Post-cutoff candidates are
attempted first within each ecosystem. Below 300 eligible repositories the command
exits 1 with counts and does not write a partial draft or relax any selection bar.
The group-2 measured exit and label confirmation remain coordinator tasks 2.5/2.6;
passing implementation tests does not assert that 300 eligible repositories exist.
Clones and checkpoints default to `results_root()/unseen-draft` (normally
`~/ousast-results/plane/unseen-draft`), overridable by `--cache /outside/path`.
The cache cannot lie inside `--root`. One opaque, disposable clone exists at a time;
it is removed on success, rejection, or failure. `--filter=blob:none --no-checkout`
keeps the commit graph needed for temporal thirds, blame, and release-tag ancestry,
and fetches source blobs on demand. This is a **partial clone**, not a shallow history.
The 500 MiB disk cap is checked after Git commands, including lazy blob fetches;
one in-flight command can transiently exceed it before rejection and cleanup.
Git output is captured, never forwarded. Each repository must read a nonempty source
blob before extraction. Summary instrumentation includes source read operations and
actual bytes, clone count, cumulative/peak disk bytes, wall time, and logical API calls
and HTTP attempts. Zero source bytes or a failed Git read is an instrument failure,
not an eligibility drop. Stdout is a JSON summary of counts/digests only; no names or
URLs enter stdout or a measurement record. Raw per-change data stays in the external journal and manifests.
Extraction uses Git and syntax boundaries, with no scanner, engine, or model. Python
uses stdlib AST; Java/PHP/JS/TS use conservative named-function brace matching. Ambiguous
or unparseable functions can reject a candidate. The first removed code line per
fix-touched function is the SZZ sink proxy; pure-addition fixes use its declaration.
Normalized text absence, non-root/non-merge ancestry, file/line limits, and the OSV
release bracket are checked before accepting `introducing`. Matching tags are `VERSION`
or `vVERSION`; an absent/ambiguous tag or missing prior release cannot certify the
bracket. After at most five walk-backs, `last_touch` uses `git log -L` with the resolved
numeric function extent (also works where Git lacks a language-specific function driver).
Method, bias, fallback reasons, head function/range, files/hunks, family, advisory, and
cutoff flag are recorded. Task 2.6 must review every third `introducing` label.
All reviewed advisories are enumerated to exclude **all** known repository fixes from
ordinary changes, including old advisories and advisories outside the four candidate
ecosystems. Fix URLs are canonicalized before indexing so renames cannot hide them.
The ordinary filter uses only the design's narrowed security pattern. It counts
rejections (including oversized commits and repositories failing the 40-commit floor),
checks both sides for touches to known vulnerable functions, and inspects generated
headers and inherited `.gitattributes`. Sampling allocates twelve draws proportionally
across three history-time thirds and four source-size buckets, with one per third.
For offline tests, inject GitHub/OSV clients, the clone context, and `UsedSet` into
`draft.run`; HTTP transports and Git subprocess runners also accept fakes. Tests use
sanitized REST-shaped data and a temporary synthetic local Git history. No implementation
test contacts GitHub, OSV, a model, or a cluster. HTTP retries are bounded to five attempts;
rate-limit reset/Retry-After waits and transient 5xx backoff are tested with fake clocks
and sleepers. A stale candidate 404 is counted; other service failures fail the run.
### Checkpoint and resume
The command above uses journal directory
`/home/mc/ousast-results/plane/unseen-draft/journal`. Run the same command to resume.
`--max-hours` is a per-invocation budget; changing it does not invalidate checkpoints.
Create `journal/STOP` to stop between units, then remove it before resuming. A page,
repository resolution, candidate decision, or repository extraction already running
finishes first, so the time budget is not a hard timeout on a large repository.
Clean stops exit 0 with `status: stopped`; failures and shortfalls exit 1. Completion
exits 0 with `status: draft`. No partial manifest is published.
`inputs.json` binds the seed, source-labelled used-set digest, known-empty declarations,
clone cap, and design fingerprints (sourcing/extraction/eligibility/publication code and
family taxonomy). A mismatch refuses resume and prints only the changed parameter
labels. Use a new external cache for a deliberately changed design or population.
The used set is rebuilt each invocation; repository resolutions, including 404s, are
reused from the journal. Live GitHub metadata for unfinished units can still change;
completed pages and decisions are the durable sourcing snapshot.
Each completed unit is appended to its stage JSONL and fsynced before work advances:
`advisories.jsonl` (pages), `resolution.jsonl` (alias/fork metadata),
`eligibility.jsonl` (candidate decisions), `extraction.jsonl` (repository/advisory
results, vulnerable range, ordinary draws, pinned licence, method and counts), and
`completion.jsonl` (final manifest digest). A lock permits only one writer.
`progress.json` is atomically rewritten after every unit with stage counts, last unit,
last observed GitHub rate-limit headers, and cumulative active elapsed seconds.
These files may contain identities and must stay outside version control. Stdout and
repository records contain only counts, digests and fixed labels.
Replaying completed units makes no GitHub/OSV or clone calls. Final manifests are
reconstructed deterministically from journal results; an interrupted publication can
be retried, but a different existing manifest is refused. Completed JSONL rows are
never rewritten. Malformed or torn records fail closed instead of being interpreted
as completed work. Tests interrupt each stage with fakes, then compare resumed and
uninterrupted manifests byte for byte.
### Planning estimate for task 2.5 (not measured)
Let A be all reviewed advisories, R distinct fix-linked repository identities, U used
identities not already cached, C candidate advisories passing early filters, and K
repositories surviving extraction. GitHub requests are approximately
`ceil(A/100) + R + U + 2*C + K` (enumeration, canonical repositories, fix metadata,
provisional licence, final licence), plus retries. OSV requests are at most C. Git
fetches and lazy blob transfers are additional network operations, not GitHub API calls.
For a planning scenario of A=40,000, R=15,000, U≈1,000, C=600–1,500, K=300:
about **18,000–20,000 GitHub calls** and **600–1,500 OSV calls**. The all-advisory index
makes this substantially more than 300 repository lookups. At an assumed 5,000-request
hourly allowance, allow several rate-limit windows. Actual token limits can differ.
For 300–1,000 attempted clones averaging 20–100 MB after source hydration, plan roughly
**6–100 GB transferred**, with one clone occupying about 500 MiB at most between commands
plus the retained journal and manifests after completion. Transfer volume is an estimate; disk bytes
are only a proxy. A working planning range is **8–48 hours**, potentially longer on
large histories: every first-parent commit is inspected and lazy blobs can dominate.
These are assumptions, not evidence from a network run. The command reports actual
counts and wall time; task 2.5 establishes the measured estimate.
## Dispatcher, replay and scoring (groups 4–6)
`benchmarks.ax.batch` owns Task transport and the shared scheduler. New workloads supply
`Item` input bytes/commands and a `Workload` image, URL allowlist, deadline and result
validator. Two AX slots and one VM slot are the default; sandbox death gets one fresh
Task before VM fallback, OOM goes straight to VM, and instrument failures get one rerun.
Cleanup failure prevents further launches. Create `STOP` in the output directory to stop
launching; remove it and repeat the command to resume matching input digests. The legacy
engine adapter retains its `questions.json` object name for compatibility; replay inputs
and outputs use `.dat`. Both use the same object polling and cleanup implementation.
Replay reads `pool-pN.toml`, verifies its adjacent `freeze-pN.json` canonical digest,
selects `--slice`, and resolves private pointers through `private/pool-pN.toml` (or
`--private-manifest`). It never reads a draft. `--dry-run` prints only counts and digests.
The five-change proof uses a separate frozen JSON manifest from this project's own
source-changing history. Raw output must be outside the repository.
Each output root contains `inputs.json` and digest-keyed records and artifact directories.
The scorer accepts that root directly, joins change labels, and includes missing or stale
results as instrument failures. It refuses a slice above 5% failure after reruns. Example:
```sh
python -m benchmarks.unseen.score "$HOME/ousast-results/unseen/p1/1/default" \
--out "$HOME/ousast-results/unseen/p1/1/default-score.json"
```
For paired binary arms add `--against BASELINE_ROOT --slice-id p1/2`; `--informed` forces
`unseen.gate=not_run`. The later experiment integration owns ledger verification; callers
of `gate` must pass that ledger decision. Score-bearing arms use `budgets` with fixed
thresholds and `gate(..., binary=False, threshold=..., threshold_fixed=True)`. In-slice
thresholds are exploratory and cannot qualify adoption. JVM wall time is explicitly the
sum of the hook artifact's scan build/query times, not a separately observed process timer.
### Coordinator task 4.4
Run from the merged repository root. Set `ENGINE_IMAGE` to the **engine** digest printed
by Engine image CI, `ENGINE_UNITS` and `ENGINE_EXAMPLES` to the frozen input files for the
two stored pins, `ENGINE_BASELINE` to their stored record directory, and `AX_KUBECONFIG`
to the operator's kube-ax kubeconfig. Host store credentials stay on the VM; use the public
HTTPS `S3_ENDPOINT=https://files.because-security.com` and the operator's `OUSAST_MEMORY`.
Use a fresh output directory, so the smoke executes instead of resuming prior records.
```sh
export PATH=/home/mc/Source/OpenUltraSAST/.venv/bin:$PATH
export PYTHONPATH=$PWD/src
export S3_ENDPOINT=https://files.because-security.com
python benchmarks/learn/engine_trace.py \
--units "$ENGINE_UNITS" --examples "$ENGINE_EXAMPLES" \
--executor ax --parallel 2 --limit 2 --image "$ENGINE_IMAGE" \
--kubeconfig "$AX_KUBECONFIG" --out "$HOME/ousast-results/unseen/proof-04"
python -m benchmarks.unseen.proof engine --image "$ENGINE_IMAGE" \
--records "$HOME/ousast-results/unseen/proof-04" --baseline "$ENGINE_BASELINE" \
--out "benchmarks/measurements/$(date +%F)-unseen-04-dispatcher/record.json"
```
### Coordinator task 5.4
Merge before running this proof. `.github/workflows/engine-image.yml` builds the `engine`
and `replay` targets on push to main; it publishes `ousast-engine-task` and
`ousast-replay-task` separately and prints their digests. Set `REPLAY_IMAGE` to the latter
full `ghcr.io/...@sha256:...` reference. No manual image build is needed. Set
`KIND_KUBECONFIG` and `AX_KUBECONFIG` to the corresponding AX kubeconfigs; the kind lane
uses the identical Task API/manifest, never a Kubernetes Job.
```sh
export PATH=/home/mc/Source/OpenUltraSAST/.venv/bin:$PATH
export PYTHONPATH=$PWD/src
export S3_ENDPOINT=https://files.because-security.com
python -m benchmarks.unseen.replay --proof-history \
--manifest "$HOME/ousast-results/unseen/proof-05-inputs.json"
python -m benchmarks.unseen.replay --manifest "$HOME/ousast-results/unseen/proof-05-inputs.json" \
--image "$REPLAY_IMAGE" --lane kind --kubeconfig "$KIND_KUBECONFIG" \
--out "$HOME/ousast-results/unseen/proof-05-kind"
python -m benchmarks.unseen.replay --manifest "$HOME/ousast-results/unseen/proof-05-inputs.json" \
--image "$REPLAY_IMAGE" --lane ax --kubeconfig "$AX_KUBECONFIG" \
--out "$HOME/ousast-results/unseen/proof-05-ax"
python -m benchmarks.unseen.replay --manifest "$HOME/ousast-results/unseen/proof-05-inputs.json" \
--image "$REPLAY_IMAGE" --lane docker \
--out "$HOME/ousast-results/unseen/proof-05-vm"
python -m benchmarks.unseen.proof replay --image "$REPLAY_IMAGE" \
--kind-records "$HOME/ousast-results/unseen/proof-05-kind" \
--ax-records "$HOME/ousast-results/unseen/proof-05-ax" \
--vm-records "$HOME/ousast-results/unseen/proof-05-vm" \
--out "benchmarks/measurements/$(date +%F)-unseen-05-replay-task/record.json"
```
The record command fails if fewer than five results exist, identities differ, bytes/JVM
proof is missing, or quick-tier findings differ. It does not convert a quick-only run
without a JVM into an engine proof. Pool replays use `--lane mixed` (default) for the two
AX slots and single VM overflow/fallback lane; `--arm long_deadline` changes only the
hook deadline to 300 seconds and is a separate input digest.
benchmarks/unseen/draft.py
+472
View File
@@ -0,0 +1,472 @@
"""Coordinator-only sourcing command: python -m benchmarks.unseen.draft --help.
All network and git operations are injected at the run() boundary. Only counts,
digests and fixed failure codes go to stdout; identity-bearing output stays in the
private journal outside the repository. No population guard is run here.
"""
from __future__ import annotations
import argparse
import copy
import hashlib
import json
import os
import random
import subprocess
import tempfile
import time
from collections import Counter, defaultdict
from contextlib import contextmanager
from dataclasses import asdict
from pathlib import Path
from . import eligibility, extract, source
from .extract import Git, InstrumentFailure
from .journal import Journal, ResolvedGitHub, ResumeMismatch, Stopped, atomic, digest, external
def canonical(value) -> bytes:
return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode()
def private_digest(row: dict) -> str:
return hashlib.sha256(canonical({"url": row["url"], "commits": [[c["base"], c["head"]] for c in row["changes"]]})).hexdigest()
def assign_slices(rows: list[dict], *, seed: int) -> list[dict]:
if len(rows) != 300 or len({r["repository"] for r in rows}) != 300:
raise ValueError("pool_size")
for row in rows:
kinds = Counter(c["kind"] for c in row["changes"])
if kinds != {"vulnerability": 1, "ordinary": 12}:
raise ValueError("change_count")
if row["license_class"] not in {"permissive", "private"}:
raise ValueError("license_class")
if row["license_class"] == "permissive" and row["license_spdx"] not in source.PERMISSIVE:
raise ValueError("license_class")
rng = random.Random(seed)
groups = defaultdict(list)
for row in sorted(rows, key=lambda r: r["repository"]):
groups[row["ecosystem"], row["post_cutoff"]].append(row)
sizes = Counter({1: 0, 2: 0, 3: 0})
assigned = []
for key in sorted(groups):
group = groups[key]
rng.shuffle(group)
quotient, remainder = divmod(len(group), 3)
# Give each stratum its floor on all slices, then fill globally least-full slices.
quotas = {i: quotient for i in sizes}
order = list(sizes)
rng.shuffle(order)
order.sort(key=lambda i: sizes[i])
for i in order[:remainder]:
quotas[i] += 1
offset = 0
for slice_id in sorted(quotas):
for row in group[offset : offset + quotas[slice_id]]:
digest = private_digest(row)
assigned.append({**row, "id": "r-" + digest[:24], "slice": slice_id})
sizes[slice_id] += quotas[slice_id]
offset += quotas[slice_id]
if set(sizes.values()) != {100}:
raise ValueError("slice_size")
return sorted(assigned, key=lambda r: (r["slice"], r["id"]))
def toml(value) -> str:
"""Inline TOML values keep nested change records structured without a new dependency."""
if isinstance(value, bool):
return "true" if value else "false"
if isinstance(value, str):
return json.dumps(value, ensure_ascii=False)
if isinstance(value, int):
return str(value)
if isinstance(value, list):
return "[" + ", ".join(toml(v) for v in value) + "]"
if isinstance(value, dict):
return "{" + ", ".join(json.dumps(k) + " = " + toml(v) for k, v in sorted(value.items()) if v is not None) + "}"
raise ValueError("unsupported_manifest_type")
def manifest(rows: list[dict], seed: int) -> str:
lines = ['status = "draft"', 'pool = "p1"', f"seed = {seed}"]
for row in rows:
lines.append("\n[[repository]]")
lines.extend(f"{key} = {toml(value)}" for key, value in sorted(row.items()) if value is not None)
return "\n".join(lines) + "\n"
def write_draft(root: Path, assigned: list[dict], *, seed: int) -> tuple[Path, Path]:
public_path = root / "benchmarks/unseen/draft-p1.toml"
private_path = root / "benchmarks/unseen/private/draft-p1.toml"
if public_path.exists() or private_path.exists():
raise FileExistsError("draft_exists")
public, private = [], []
for row in assigned:
if row["license_class"] == "permissive":
public.append(row)
else:
private.append(row)
public.append({"id": row["id"], "sha256": private_digest(row), "slice": row["slice"], "private": True})
private_path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
# Exclusive creates protect drafts awaiting human label review. Private first: never
# publish a pointer whose private manifest failed to write.
with private_path.open("x") as handle:
os.chmod(private_path, 0o600)
handle.write(manifest(private, seed))
with public_path.open("x") as handle:
handle.write(manifest(public, seed))
return public_path, private_path
class Clones:
"""One disposable blobless clone at a time; no name in cache paths or subprocess output."""
def __init__(self, root: Path, cache: Path, *, runner=None, limit_bytes=500 * 1024 * 1024):
self.cache = external(root, cache)
self.runner = runner or subprocess.run
self.limit_bytes = limit_bytes
self.counts = Counter()
@contextmanager
def open(self, url: str):
self.cache.mkdir(parents=True, exist_ok=True)
started = time.monotonic()
with tempfile.TemporaryDirectory(prefix="clone-", dir=self.cache) as directory:
path = Path(directory)
git = Git(path, runner=self.runner, limit_bytes=self.limit_bytes)
try:
# Full commit graph is needed for temporal thirds and OSV brackets; blobs
# are fetched lazily. This is a partial clone, not a truncated history.
done = self.runner(
["git", "-c", "credential.helper=", "clone", "--quiet", "--filter=blob:none", "--no-checkout", url, str(path)],
stdin=subprocess.DEVNULL,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
check=False,
timeout=180,
env={**os.environ, "GIT_TERMINAL_PROMPT": "0"},
)
if done.returncode:
raise InstrumentFailure("clone_failed")
self.counts["clones"] += 1
if git.disk_bytes() > self.limit_bytes:
raise InstrumentFailure("clone_size_cap")
yield git
finally:
size = git.disk_bytes()
self.counts["clone_bytes"] += size
self.counts["peak_clone_bytes"] = max(self.counts["peak_clone_bytes"], size)
self.counts["source_bytes_read"] += git.bytes_read
self.counts["source_files_read"] += git.files_read
self.counts["clone_wall_seconds"] += round(time.monotonic() - started, 3)
def prove_source(git, revision: str) -> dict:
files = git.run("ls-tree", "-r", "--name-only", revision).splitlines()
for path in files:
if Path(path).suffix in extract.LANGUAGES and not extract.generated_path(path):
text = git.blob(revision, path)
if text:
return {"files": 1, "bytes": getattr(git, "last_blob_bytes", len(text.encode()))}
raise InstrumentFailure("zero_source_bytes")
def advisory_index(rows: list[dict], github) -> dict[str, set[str]]:
"""Include ALL enumerated fixes, even old, unmapped, multi-fix and other ecosystems.
Canonicalize link names as well, so an old advisory URL survives repository renames.
A 404 is not treated as proof that an advisory is unrelated to a selected repository.
"""
index = defaultdict(set)
for row in rows:
for name, sha in source.fix_links(row):
try:
resolved = eligibility.resolve(name, github)
except eligibility.RepositoryNotFound:
index[name].add(sha)
continue
index[resolved.full_name].add(sha)
return dict(index)
def repository_changes(git, candidate, fixes: set[str], *, seed: int):
proof = prove_source(git, candidate.parent)
change = extract.introducing(git, candidate)
known = set()
expanded = set()
for fix in sorted(fixes | {candidate.fix}):
# Explicitly fetch detached advisory commits unavailable on default-branch history.
# All git including this possible network call still goes through the injected client.
git.run("fetch", "--quiet", "--filter=blob:none", "origin", fix)
full = git.run("rev-parse", fix).strip()
expanded.add(full)
parents = git.parents(full)
for parent in parents:
sites = extract.fix_sites(git, full, parent, global_ok=True)
known.update((site["path"], site["function"]) for site in sites)
rows = extract.ordinary_history(git, "HEAD")
ordinary, exclusions = extract.draw_ordinary(rows, expanded, known, seed=seed)
return [change, *ordinary], exclusions, proof
def inputs(used, seed, known_empty, clones):
from openultrasast.model.taxonomy import DEFAULT_FAMILIES_PATH
parameters = {
"seed": seed,
"used_set_digest": digest({label: sorted(names) for label, names in used.sources.items()}),
"known_empty": sorted(known_empty),
"clone_limit_bytes": getattr(clones, "limit_bytes", 500 * 1024 * 1024),
"journal_version": 1,
}
# Hardcoded design thresholds and algorithms are covered as well as taxonomy data.
for name in ("draft", "source", "extract", "eligibility", "journal"):
parameters["design_" + name] = hashlib.sha256(Path(__file__).with_name(name + ".py").read_bytes()).hexdigest()
parameters["design_taxonomy"] = hashlib.sha256(DEFAULT_FAMILIES_PATH.read_bytes()).hexdigest()
return parameters
def enumerate_advisories(github, journal):
page, records = 1, {}
while True:
key = str(page)
if key not in journal.rows["advisories"]:
journal.boundary()
result = github.advisory_page(page)
journal.put("advisories", key, result)
result = journal.rows["advisories"][key]
records.update((row["ghsa_id"], row) for row in result["rows"])
if result["done"]:
break
page += 1
if not records:
raise InstrumentFailure("empty_advisories")
return list(records.values())
def decide(row, github, osv, used, selected_networks):
try:
links = source.fix_links(row)
if len(links) != 1:
raise source.Rejected("fix_links")
name, _ = next(iter(links))
resolved = eligibility.resolve(name, github)
rejected_by = eligibility.rejection(resolved, used, set(selected_networks))
if rejected_by:
raise source.Rejected("used_" + rejected_by)
candidate = source.candidate(row, github, osv)
return {"candidate": asdict(candidate), "network": resolved.network}
except source.Rejected as exc:
return {"rejected": str(exc)}
except eligibility.RepositoryNotFound:
return {"rejected": "repository_or_commit_not_found"}
def extract_repository(candidate, github, clones, fixes, seed):
before = dict(clones.counts)
try:
with clones.open(candidate.url) as git:
git.run("fetch", "--quiet", "--filter=blob:none", "origin", candidate.fix)
changes, exclusions, proof = repository_changes(git, candidate, fixes, seed=seed)
pinned = changes[0]["head"]
license_record = github.get_license(candidate.repository, pinned)
spdx, license_class = source.license_info(license_record)
entry = {
**asdict(candidate),
"license_spdx": spdx,
"license_class": license_class,
"license_ref": pinned,
"license_path": license_record.get("path", ""),
"changes": changes,
"instrument": proof,
}
result = {"entry": entry, "ordinary_exclusions": exclusions}
except source.Rejected as exc:
result = {"rejected": str(exc), "ordinary_exclusions": getattr(exc, "counts", {})}
except eligibility.RepositoryNotFound:
result = {"rejected": "repository_or_commit_not_found", "ordinary_exclusions": {}}
result["counts"] = {key: value - before.get(key, 0) for key, value in clones.counts.items()}
return result
def publish(journal, assigned, seed):
"""Idempotent publication from completed journal data, never inside the checkout."""
public, private = [], []
for row in assigned:
if row["license_class"] == "permissive":
public.append(row)
else:
private.append(row)
public.append({"id": row["id"], "sha256": private_digest(row), "slice": row["slice"], "private": True})
for name, rows in (("private-draft-p1.toml", private), ("draft-p1.toml", public)):
path = journal.path / name
data = manifest(rows, seed).encode()
if path.exists():
if path.read_bytes() != data:
raise ValueError("draft_exists")
else:
atomic(path, data)
def run(
root: Path,
*,
github,
osv,
clones,
used: eligibility.UsedSet,
seed: int,
known_empty=None,
journal_path=None,
max_hours=None,
clock=None,
) -> dict:
started = time.monotonic()
report = {"status": "sourcing", "seed": seed, "candidates": {}, "rejections": {}, "ordinary_exclusions": {}}
counts, rejected, ordinary_counts = Counter(), Counter(), Counter()
checkpoint = None
try:
known_empty = set(known_empty or ())
used.report(known_empty)
cache = getattr(clones, "cache", root.parent / (root.name + "-cache"))
path = external(root, journal_path or cache / "journal")
checkpoint = Journal(
path, max_hours=max_hours, clock=clock, rate_limit=lambda: getattr(getattr(github, "transport", None), "rate_limit", {})
)
with checkpoint.open(inputs(used, seed, known_empty, clones)):
checkpoint.boundary()
api = ResolvedGitHub(github, checkpoint)
used = copy.deepcopy(used) # canonicalization must not mutate the resume input
eligibility.canonicalize_used(used, api)
report["used_set"] = used.report(known_empty)
rows = enumerate_advisories(api, checkpoint)
report["advisories"] = len(rows)
fixes_by_repo = advisory_index(rows, api)
selected_networks, entries = set(), []
for row in source.ordered_advisories(rows, seed):
checkpoint.boundary()
key = row["ghsa_id"]
counts[source.ecosystem(row)] += 1
if key not in checkpoint.rows["eligibility"]:
decision = decide(row, api, osv, used, selected_networks)
checkpoint.put("eligibility", key, decision)
decision = checkpoint.rows["eligibility"][key]
if "rejected" in decision:
rejected[decision["rejected"]] += 1
continue
candidate = source.Candidate(**decision["candidate"])
# Multiple advisories for one repository can have different fix/range inputs.
extraction_key = digest({"repository": candidate.repository, "advisory": key, "fix": candidate.fix})
if extraction_key not in checkpoint.rows["extraction"]:
checkpoint.boundary()
result = extract_repository(candidate, api, clones, fixes_by_repo.get(candidate.repository, set()), seed)
checkpoint.put("extraction", extraction_key, result)
result = checkpoint.rows["extraction"][extraction_key]
ordinary_counts.update(result["ordinary_exclusions"])
if "rejected" in result:
rejected[result["rejected"]] += 1
continue
selected_networks.add(decision["network"])
entries.append(result["entry"])
if len(entries) == 300:
break
report.update(
{
"repositories": len(entries),
"candidates": dict(counts),
"rejections": dict(rejected),
"ordinary_exclusions": dict(ordinary_counts),
}
)
checkpoint.boundary()
if len(entries) != 300:
report["status"] = "insufficient_repositories"
checkpoint.progress(report["status"])
return report
assigned = assign_slices(entries, seed=seed)
completion = {"manifest_sha256": digest(assigned), "repositories": len(entries)}
if "all" not in checkpoint.rows["completion"]:
checkpoint.put("completion", "all", completion)
elif checkpoint.rows["completion"]["all"] != completion:
raise ValueError("journal_completion_mismatch")
checkpoint.boundary()
publish(checkpoint, assigned, seed)
report.update(
{
"status": "draft",
"changes": 3900,
"slices": 3,
"license_split": dict(Counter(e["license_class"] for e in entries)),
"methods": dict(Counter(e["changes"][0]["method"] for e in entries)),
"post_cutoff": sum(e["post_cutoff"] for e in entries),
**completion,
}
)
checkpoint.progress("draft")
return report
except ResumeMismatch as exc:
report.update({"status": "resume_refused", "changed_parameters": exc.parameters})
return report
except Stopped as exc:
report.update({"status": "stopped", "reason": str(exc)})
return report
except Exception as exc:
report.update({"status": "instrument_failure", "error_type": type(exc).__name__})
return report
finally:
report.update({"candidates": dict(counts), "rejections": dict(rejected), "ordinary_exclusions": dict(ordinary_counts)})
report["instrument"] = dict(clones.counts)
report["wall_seconds"] = round(time.monotonic() - started, 3)
report["github_calls"] = github.calls
report["osv_calls"] = osv.calls
report["github_http_attempts"] = getattr(getattr(github, "transport", None), "attempts", github.calls)
report["osv_http_attempts"] = getattr(getattr(osv, "transport", None), "attempts", osv.calls)
def main(argv: list[str] | None = None) -> int:
from openultrasast import config
from openultrasast.plane import memory
from openultrasast.plane.reconciler import results_root
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--root", type=Path, default=Path.cwd())
parser.add_argument("--cache", type=Path)
parser.add_argument("--max-hours", type=float, help="Stop between units after this many hours per invocation")
parser.add_argument("--seed", type=int, default=20260601)
parser.add_argument("--local-store")
parser.add_argument("--s3-store", default="s3://")
parser.add_argument("--pointer-manifest", type=Path, action="append", default=[])
parser.add_argument("--known-empty", choices=(*eligibility.SOURCES, "memory_local", "memory_s3"), action="append", default=[])
args = parser.parse_args(argv)
try:
config.load_dotenv(args.root / ".env")
# Refuse an existing draft before doing any network work.
if any((args.root / p).exists() for p in ("benchmarks/unseen/draft-p1.toml", "benchmarks/unseen/private/draft-p1.toml")):
raise FileExistsError("draft_exists")
clones = Clones(args.root, args.cache or results_root() / "unseen-draft")
stores = {
"local": memory.open_store(args.local_store or "file://" + str(results_root() / "memory"), read_only=True),
"s3": memory.open_store(args.s3_store, read_only=True),
}
used = eligibility.build_used(args.root, stores=stores, pointer_manifests=args.pointer_manifest)
report = run(
args.root,
github=source.GitHub(os.environ.get("GH_TOKEN", "")),
osv=source.OSV(),
clones=clones,
used=used,
seed=args.seed,
known_empty=set(args.known_empty),
journal_path=clones.cache / "journal",
max_hours=args.max_hours,
)
except Exception as exc:
report = {"status": "instrument_failure", "error_type": type(exc).__name__}
print(json.dumps(report, sort_keys=True))
return 0 if report["status"] in {"draft", "stopped"} else 1
if __name__ == "__main__":
raise SystemExit(main())
benchmarks/unseen/eligibility.py
+379
View File
@@ -0,0 +1,379 @@
"""Read-only used-set collection. Only counts, source labels and digests leave this module's CLI.
Run from the repository root: python -m benchmarks.unseen.eligibility --help.
Population v3 is exclusively the existing guard's responsibility.
"""
from __future__ import annotations
import argparse
import hashlib
import json
import re
import tomllib
from collections.abc import Iterable, Mapping
from dataclasses import dataclass, field
from pathlib import Path
from urllib.parse import urlsplit
import yaml
from openultrasast.plane.memory import MemoryStore
URL = re.compile(r"github\.com/([\w.-]+/[\w.-]+)", re.I)
NAME = re.compile(r"[\w.-]+/[\w.-]+")
SOURCES = ("pairs", "pointers", "populations", "harvest", "units", "measurements", "sweep")
def normalize(value: str) -> str:
text = value.strip().lower().rstrip("/")
if NAME.fullmatch(text):
name = text.removesuffix(".git")
else:
# Structured fields must be identities, not prose containing a URL.
if any(char.isspace() for char in text):
raise ValueError("invalid_repository")
url = urlsplit(text if "://" in text else "https://" + text)
if url.scheme not in {"http", "https", "ssh", "git"} or not url.hostname or "." not in url.hostname:
raise ValueError("invalid_repository")
parts = url.path.strip("/").split("/")
if len(parts) < 2:
raise ValueError("invalid_repository")
name = "/".join(parts[:2]).removesuffix(".git")
if not NAME.fullmatch(name):
raise ValueError("invalid_repository")
if url.hostname != "github.com":
return url.hostname + "/" + name
if not NAME.fullmatch(name) or any(part in {".", ".."} for part in name.split("/")):
raise ValueError("invalid_repository")
return name
def excluded(path: Path) -> bool:
return any("v3" in part.lower() or part in {".git", "__pycache__", ".venv"} for part in path.parts)
def repositories(value: object, counts: dict[str, int] | None = None) -> set[str]:
"""Structured repo fields include bare owner/name pointers; URLs also occur in prose."""
found = set()
if isinstance(value, dict):
for key, item in value.items():
if key in {"repo", "fix_repo", "repository", "full_name", "origin"} or (key == "group" and "unit" in value):
try:
if not isinstance(item, str):
raise ValueError("invalid_repository")
found.add(normalize(item))
except ValueError:
if counts is not None:
counts["skipped_values"] = counts.get("skipped_values", 0) + 1
else:
found.update(repositories(item, counts))
elif isinstance(value, list):
for item in value:
found.update(repositories(item, counts))
elif isinstance(value, str):
found.update(normalize(m[1]) for m in URL.finditer(value))
return found
def pointer_repositories(value: object, counts: dict[str, int] | None = None) -> set[str]:
if isinstance(value, dict):
if value.get("vendored") is False:
return repositories(value, counts)
return set().union(*(pointer_repositories(v, counts) for v in value.values()))
if isinstance(value, list):
return set().union(*(pointer_repositories(v, counts) for v in value))
return set()
def parse(path: Path, raw: bytes) -> object:
structured = path.suffix in {".toml", ".json", ".jsonl", ".yaml", ".yml"}
text = raw.decode("utf-8", errors="strict" if structured else "ignore")
if path.suffix == ".toml":
return tomllib.loads(text)
if path.suffix == ".json":
return json.loads(text)
if path.suffix == ".jsonl":
return [json.loads(line) for line in text.splitlines() if line.strip()]
if path.suffix in {".yaml", ".yml"}:
documents = list(yaml.safe_load_all(text))
return documents[0] if len(documents) == 1 else documents
return text
@dataclass
class UsedSet:
sources: dict[str, set[str]] = field(default_factory=lambda: {label: set() for label in SOURCES})
instrument: dict[str, dict[str, int]] = field(default_factory=dict)
def add(self, label: str, names: set[str], size: int, files: int = 1, skipped_values: int = 0) -> None:
self.sources.setdefault(label, set()).update(names)
counts = self.instrument.setdefault(label, {"files": 0, "bytes": 0, "rows": 0, "skipped_values": 0})
counts["files"] += files
counts["bytes"] += size
counts["rows"] += len(names)
counts["skipped_values"] += skipped_values
def report(self, known_empty: set[str] | None = None) -> dict:
known_empty = known_empty or set()
if known_empty - self.sources.keys():
raise ValueError("unknown_source_label")
records = {}
for label, names in self.sources.items():
counts = self.instrument.get(label, {"files": 0, "bytes": 0, "rows": 0, "skipped_values": 0})
if not counts["rows"] and label not in known_empty:
raise ValueError("empty_source:" + label)
records[label] = {**counts, "count": len(names), "known_empty": label in known_empty}
names = set().union(*self.sources.values())
return {"sources": records, "count": len(names), "sha256": hashlib.sha256("\n".join(sorted(names)).encode()).hexdigest()}
def build_used(root: Path, *, stores: Mapping[str, MemoryStore] | None = None, pointer_manifests: Iterable[Path] = ()) -> UsedSet:
used = UsedSet()
def read(label: str, paths: Iterable[Path], structured: bool = True) -> list[object]:
values = []
for path in sorted(set(paths)):
if excluded(path):
continue
raw = path.read_bytes() # failure is an instrument failure, never a quiet empty source
value = parse(path, raw) if structured else raw.decode("utf-8", errors="ignore")
counts: dict[str, int] = {}
names = repositories(value, counts)
used.add(label, names, len(raw), skipped_values=counts.get("skipped_values", 0))
if label == "pairs":
pointer_counts: dict[str, int] = {}
pointers = pointer_repositories(value, pointer_counts)
if pointers or pointer_counts:
used.add("pointers", pointers, len(raw), skipped_values=pointer_counts.get("skipped_values", 0))
values.append(value)
return values
pairs = root / "benchmarks/pairs"
read("pairs", (p for p in pairs.rglob("*") if p.is_file() and p.suffix in {".toml", ".json", ".jsonl", ".csv", ".yaml", ".yml"}))
read("pointers", pointer_manifests)
read("populations", (p for p in (root / "benchmarks/independent").glob("population-v[12].toml")))
read("harvest", (p for directory in pairs.glob("advisory-fixes*") for p in directory.rglob("*.toml")))
units = set((root / "plane/experiments").glob("*.units.jsonl"))
for path in (root / "plane/experiments").glob("*.y*ml"):
if excluded(path):
continue
raw = path.read_bytes()
manifest = parse(path, raw)
counts = {}
names = repositories(manifest, counts)
used.add("units", names, len(raw), skipped_values=counts.get("skipped_values", 0))
if isinstance(manifest, dict) and manifest.get("units_file"):
declared = root / Path(manifest["units_file"]).expanduser()
if declared.exists(): # an unregistered plan may not have frozen units yet
units.add(declared)
for label, store in (stores or {}).items():
if label not in {"local", "s3"}:
raise ValueError("unknown_store_label")
source = "memory_" + label
used.add(source, set(), 0, 0)
# Read actual objects, not a serialized row size masquerading as bytes read.
for key in store._keys("repos/"):
if not key.endswith(".jsonl") or excluded(Path(key)):
continue
result = store._get(key)
if result is None:
raise ValueError("missing_memory_object")
raw = result[0]
rows = parse(Path("rows.jsonl"), raw)
names = set()
counts = {}
for row in rows:
if row.get("kind") == "example" and "repo" in row:
names.update(repositories({"repo": row["repo"]}, counts))
if row.get("kind") == "experiment" and row.get("units_file"):
registered = root / Path(row["units_file"]).expanduser()
if row.get("units_digest") or registered.exists():
units.add(registered)
used.add(source, names, len(raw), skipped_values=counts.get("skipped_values", 0))
read("units", units)
read("measurements", (p for top in ("measurements", "experiments") for p in (root / "benchmarks" / top).rglob("*") if p.is_file()))
read(
"sweep",
(
p
for top in ("benchmarks", "src", "tests", "plane", ".kiro")
for p in (root / top).rglob("*")
if p.is_file() and not p.is_relative_to(root / "benchmarks/unseen") and not excluded(p)
),
structured=False,
)
return used
@dataclass(frozen=True)
class Resolution:
full_name: str
names: frozenset[str]
network: str
def resolve(candidate: str, api) -> Resolution:
name = normalize(candidate)
if name.count("/") > 1:
return Resolution(name, frozenset({name}), name)
data = api.get_repo(name)
canonical = normalize(data["full_name"])
names = {name, canonical, *(normalize(n) for n in data.get("former_names", []))}
for key in ("parent", "source"):
if data.get(key):
names.add(normalize(data[key]["full_name"]))
network = normalize((data.get("source") or data.get("parent") or data)["full_name"])
return Resolution(canonical, frozenset(names), network)
def canonicalize_used(used: UsedSet, api) -> None:
"""Resolve old used names too: GitHub exposes redirects, not a complete rename history."""
cache = {}
for names in used.sources.values():
for name in sorted(names):
if name not in cache:
try:
cache[name] = resolve(name, api)
except RepositoryNotFound:
cache[name] = Resolution(name, frozenset({name}), name)
names.update(cache[name].names)
def rejection(candidate: Resolution, used: UsedSet, selected: set[str]) -> str | None:
for label, names in used.sources.items():
if candidate.names & names:
return label
if candidate.network in selected:
return "fork_network"
selected.add(candidate.network)
return None
class RepositoryNotFound(ValueError):
"""A historical or synthetic used identity no longer resolves; retain it verbatim."""
class GitHub:
"""Coordinator-only GET client. Exceptions never disclose request URLs or tokens."""
def __init__(self, token: str):
if not token:
raise ValueError("missing_gh_token")
self.token = token
def get_repo(self, name: str) -> dict:
from urllib.error import HTTPError
from urllib.request import Request, urlopen
request = Request(
"https://api.github.com/repos/" + normalize(name),
headers={
"Authorization": "Bearer " + self.token,
"Accept": "application/vnd.github+json",
},
)
try:
with urlopen(request, timeout=30) as response:
return json.load(response)
except HTTPError as exc:
if exc.code == 404:
raise RepositoryNotFound("github_not_found") from None
raise ValueError("github_resolution_failed") from None
except Exception:
raise ValueError("github_resolution_failed") from None
def verify_tests(root: Path) -> dict:
"""The coordinator runs the guard as a black box; only exit codes and counts escape."""
import subprocess
import sys
import tempfile
from xml.etree import ElementTree
with tempfile.TemporaryDirectory() as directory:
report = Path(directory) / "tests.xml"
done = subprocess.run(
[
sys.executable,
"-m",
"pytest",
"-q",
"-p",
"no:cacheprovider",
"tests/test_unseen_eligibility.py",
"tests/test_unseen_ledger.py",
"tests/test_independent_population.py",
"--junitxml=" + str(report),
],
cwd=root,
stdin=subprocess.DEVNULL,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
check=False,
)
if done.returncode != 0:
raise ValueError("guard_or_tests_failed")
suites = ElementTree.parse(report).getroot().iter("testsuite")
counts = {key: 0 for key in ("tests", "failures", "errors", "skipped")}
for suite in suites:
for key in counts:
counts[key] += int(suite.attrib[key])
if not counts["tests"]:
raise ValueError("empty_test_run")
return {"guard": "green", "test_exit_code": done.returncode, "tests": counts}
def main(argv: list[str] | None = None) -> int:
import os
from openultrasast import config
from openultrasast.plane import memory
from openultrasast.plane.reconciler import results_root
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--root", type=Path, default=Path.cwd())
parser.add_argument("--local-store", default=None)
parser.add_argument("--s3-store", default="s3://")
parser.add_argument("--pointer-manifest", type=Path, action="append", default=[])
parser.add_argument("--known-empty", choices=(*SOURCES, "memory_local", "memory_s3"), action="append", default=[])
parser.add_argument("--output", type=Path)
parser.add_argument("--verify-tests", action="store_true")
args = parser.parse_args(argv)
step = "open_stores"
try:
config.load_dotenv(args.root / ".env")
stores = {
"local": memory.open_store(args.local_store or "file://" + str(results_root() / "memory"), read_only=True),
"s3": memory.open_store(args.s3_store, read_only=True),
}
step = "build_used"
used = build_used(args.root, stores=stores, pointer_manifests=args.pointer_manifest)
step = "report"
used.report(set(args.known_empty)) # prove inputs before API work
step = "canonicalize"
canonicalize_used(used, GitHub(os.environ.get("GH_TOKEN", "")))
step = "report"
report = used.report(set(args.known_empty))
if args.verify_tests:
step = "verify_tests"
report.update(verify_tests(args.root))
step = "report"
text = json.dumps(report, sort_keys=True, indent=2) + "\n"
if args.output:
args.output.parent.mkdir(parents=True, exist_ok=True)
with args.output.open("x") as handle:
handle.write(text)
print(text, end="")
return 0
except Exception as exc:
# Source paths, store exceptions and JSON parser errors can contain repository names.
report = {"status": "instrument_failure", "step": step, "type": type(exc).__name__}
if str(exc) in {"empty_source:" + label for label in (*SOURCES, "memory_local", "memory_s3")}:
report["source"] = str(exc).partition(":")[2]
print(json.dumps(report))
return 1
if __name__ == "__main__":
raise SystemExit(main())
benchmarks/unseen/extract.py
+474
View File
@@ -0,0 +1,474 @@
"""Git-only change extraction; clients and subprocess execution are injectable.
No scanner, engine or model runs here. Function boundaries are syntax, not a security
judgement: the first removed code line of a fix is the SZZ sink proxy. Ambiguous sites
are dropped, and the recorded SZZ bias remains subject to task 2.6's label check.
"""
from __future__ import annotations
import ast
import json
import re
import subprocess
from pathlib import Path
from .source import Candidate, Rejected
LANGUAGES = {".py", ".js", ".jsx", ".mjs", ".cjs", ".ts", ".tsx", ".php", ".java"}
HUNK = re.compile(r"^@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@", re.M)
SHA = re.compile(r"[a-f0-9]{40}")
BIAS = {
"introducing": "SZZ last-writer proxy: refactors, moves and reformats can misidentify the introduction in either direction.",
"last_touch": "Pessimistic lower bound: the flaw may already exist in base, so a delta hook can correctly stay quiet.",
}
class InstrumentFailure(ValueError):
"""A failed read or command must never masquerade as an empty history."""
class Git:
def __init__(self, path: Path, *, runner=None, limit_bytes: int | None = None):
self.path = path
self.runner = runner or subprocess.run
self.bytes_read = 0
self.files_read = 0
self.last_blob_bytes = 0
self.limit_bytes = limit_bytes
def run(self, *args: str, allowed=(0,)) -> str:
try:
done = self.runner(
["git", "-c", "core.quotePath=false", "-C", str(self.path), *args],
stdin=subprocess.DEVNULL,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
timeout=180,
check=False,
)
except Exception:
raise InstrumentFailure("git_launch_failed") from None
if done.returncode not in allowed:
raise InstrumentFailure("git_exit_" + str(done.returncode))
if self.limit_bytes is not None and self.disk_bytes() > self.limit_bytes:
raise InstrumentFailure("clone_size_cap")
self.last_blob_bytes = len(done.stdout)
return done.stdout.decode("utf-8", errors="replace")
def disk_bytes(self) -> int:
return sum(p.stat().st_size for p in self.path.rglob("*") if p.is_file() and not p.is_symlink())
def parents(self, sha: str) -> list[str]:
return self.run("rev-list", "--parents", "-n", "1", sha).split()[1:]
def blob(self, sha: str, path: str) -> str | None:
# Missing is established by a successful tree read, never by swallowing a show error.
if not self.run("ls-tree", sha, "--", path).strip():
return None
text = self.run("show", f"{sha}:{path}")
if Path(path).suffix in LANGUAGES:
self.files_read += 1
self.bytes_read += self.last_blob_bytes
return text
def stats(self, base: str, head: str) -> list[tuple[str, int, int]]:
rows = []
for line in self.run("diff", "--no-renames", "--numstat", base, head, "--").splitlines():
added, removed, path = line.split("\t", 2)
rows.append((path, int(added) if added.isdigit() else 0, int(removed) if removed.isdigit() else 0))
return rows
def hunks(self, base: str, head: str, path: str) -> list[tuple[int, int, int, int]]:
diff = self.run("diff", "--no-ext-diff", "--no-renames", "-U0", base, head, "--", path)
return [(int(m[1]), int(m[2] or 1), int(m[3]), int(m[4] or 1)) for m in HUNK.finditer(diff)]
def ancestor(self, older: str, newer: str) -> bool:
# rev-list avoids interpreting a command failure as a negative ancestry result.
return not self.run("rev-list", "--max-count=1", older, "--not", newer).strip()
def blame(self, revision: str, path: str, line: int) -> tuple[str, str, int]:
text = self.run("blame", "-w", "-M", "-C", "--porcelain", f"-L{line},{line}", revision, "--", path)
header = text.splitlines()[0].split()
if len(header) < 3 or not SHA.fullmatch(header[0]):
raise InstrumentFailure("invalid_blame")
filename = next((row[9:] for row in text.splitlines() if row.startswith("filename ")), path)
if filename.startswith('"'):
filename = json.loads(filename)
return header[0], filename, int(header[1])
def functions(path: str, text: str) -> list[tuple[str, int, int]]:
"""Conservative syntax boundaries. No scanner import; unresolved sites are rejected."""
if Path(path).suffix == ".py":
try:
tree = ast.parse(text)
except SyntaxError:
raise Rejected("function_parse") from None
return [
(n.name, n.lineno, n.end_lineno or n.lineno) for n in ast.walk(tree) if isinstance(n, ast.FunctionDef | ast.AsyncFunctionDef)
]
# Blank comments and quoted strings before brace matching. Keep newlines/offsets.
masked = re.sub(
r'/\*.*?\*/|//[^\n]*|\#[^\n]*|"(?:\\.|[^"\\])*"|\'(?:\\.|[^\'\\])*\'|`(?:\\.|[^`\\])*`',
lambda m: "".join("\n" if c == "\n" else " " for c in m[0]),
text,
flags=re.S,
)
declaration = re.compile(
r"\bfunction\s*&?\s*(\w+)\s*\([^)]*\)\s*(?::[^\n{]+)?\s*\{|"
r"\b(?:const|let|var)\s+(\w+)\s*=\s*(?:async\s*)?(?:\([^)]*\)|\w+)\s*=>\s*\{|"
r"\b([\w$]+)\s*\([^;{}]*\)\s*(?:throws\s+[\w., ]+)?\s*\{"
)
found = []
for match in declaration.finditer(masked):
name = next(g for g in match.groups() if g)
if name in {"if", "while", "for", "switch", "catch", "with", "synchronized"}:
continue
depth = 1
end = match.end()
while end < len(masked) and depth:
depth += (masked[end] == "{") - (masked[end] == "}")
end += 1
if depth == 0:
found.append((name, masked.count("\n", 0, match.start()) + 1, masked.count("\n", 0, end) + 1))
return found
def enclosing(path: str, text: str, line: int, *, global_ok=False) -> tuple[str, int, int] | None:
spans = [s for s in functions(path, text) if s[1] <= line <= s[2]]
if spans:
return min(spans, key=lambda s: s[2] - s[1])
return ("<global>", 1, max(1, len(text.splitlines()))) if global_ok else None
def normalized(text: str) -> str:
return re.sub(r"\s+", "", text)
def site_at(git, revision: str, path: str, name: str):
text = git.blob(revision, path)
if text is None:
return None
spans = [("<global>", 1, max(1, len(text.splitlines())))] if name == "<global>" else [s for s in functions(path, text) if s[0] == name]
if len(spans) != 1:
return None
return text, spans[0]
def fix_sites(git, fix: str, parent: str, *, global_ok=False) -> list[dict]:
sites = []
for path, _, _ in git.stats(parent, fix):
if Path(path).suffix not in LANGUAGES:
continue
text = git.blob(parent, path)
if text is None:
continue
lines = text.splitlines()
for start, count, _, _ in git.hunks(parent, fix, path):
for line in range(max(1, start), max(1, start) + max(1, count)):
if line > len(lines):
continue
span = enclosing(path, text, line, global_ok=global_ok)
if span is None:
continue
sink_line = line if count else span[1]
sink = lines[sink_line - 1]
if not sink.strip() or sink.lstrip().startswith(("#", "//", "/*", "*")):
continue
key = (path, span[0])
if any((s["path"], s["function"]) == key for s in sites):
continue
sites.append({"path": path, "function": span[0], "line": sink_line, "text": sink, "absence": count == 0, "span": span})
return sites
def version_bracket(git, candidate: Candidate, revision: str) -> bool:
if not candidate.first_affected:
return True
tags = git.run("tag", "--sort=version:refname").splitlines()
matching = [t for t in tags if t.removeprefix("v") == candidate.first_affected]
if len(matching) != 1:
return False
affected = matching[0]
earlier = [t for t in tags[: tags.index(affected)] if git.ancestor(t, affected)]
return bool(earlier) and git.ancestor(revision, affected) and not git.ancestor(revision, earlier[-1])
def change_files(git, base: str, head: str) -> list[dict]:
return [
{"path": path, "head_lines": [[max(1, new), max(1, new + max(count, 1) - 1)] for _, _, new, count in git.hunks(base, head, path)]}
for path, _, _ in git.stats(base, head)
]
def introducing(git, candidate: Candidate) -> dict:
from collections import Counter
if git.parents(candidate.fix) != [candidate.parent]:
raise Rejected("fix_parents")
sites = fix_sites(git, candidate.fix, candidate.parent, global_ok=candidate.family == "config_secrets")
reasons = Counter()
def record(method, base, head, path, name):
resolved = site_at(git, head, path, name)
if resolved is None:
return None
return {
"kind": "vulnerability",
"base": base,
"head": head,
"method": method,
"family": candidate.family,
"function": name,
"path": path,
"function_lines": list(resolved[1][1:]),
"files": change_files(git, base, head),
"post_cutoff": candidate.post_cutoff,
"advisory": candidate.advisory,
"bias": BIAS[method],
"fallback_reasons": dict(reasons),
}
for site in sites:
revision, path, line = candidate.parent, site["path"], site["line"]
for _ in range(6): # initial blame plus at most five walk-back steps
blamed, blamed_path, blamed_line = git.blame(revision, path, line)
parents = git.parents(blamed)
if len(parents) != 1:
reasons["merge_or_root"] += 1
break
base = parents[0]
head_site = site_at(git, blamed, blamed_path, site["function"])
if head_site is None:
reasons["unresolved_function"] += 1
break
parent_text = git.blob(base, blamed_path)
if parent_text is not None and site["function"] != "<global>":
matching = [span for span in functions(blamed_path, parent_text) if span[0] == site["function"]]
if len(matching) > 1:
reasons["ambiguous_parent"] += 1
break
old_site = site_at(git, base, blamed_path, site["function"])
occurrences = []
if old_site:
text, (_, start, end) = old_site
occurrences = [
i for i, value in enumerate(text.splitlines(), 1) if start <= i <= end and normalized(value) == normalized(site["text"])
]
if occurrences:
reasons["text_present"] += 1
revision, path, line = base, blamed_path, occurrences[0]
continue
stats = git.stats(base, blamed)
if len(stats) > 50 or sum(a + d for _, a, d in stats) > 2000:
reasons["size_cap"] += 1
break
if not version_bracket(git, candidate, blamed):
reasons["version_bracket"] += 1
break
result = record("introducing", base, blamed, blamed_path, site["function"])
if result:
return result
else:
reasons["walkback_limit"] += 1
for site in sites:
# Numeric function extent implements log -L for all five languages, including
# languages for which Git has no built-in :function: driver.
_, start, end = site["span"]
history = git.run("log", "--format=%H", "--no-patch", f"-L{start},{end}:{site['path']}", candidate.parent).splitlines()
if not history:
continue
latest = history[0]
parents = git.parents(latest)
if len(parents) != 1:
continue
result = record("last_touch", parents[0], candidate.parent, site["path"], site["function"])
if result:
return result
raise Rejected("no_range")
SECURITY = re.compile(r"secur|cve|ghsa|vuln|inject|xss|csrf|ssrf|exploit|sanitiz|escap", re.I)
DEPENDENCY = re.compile(r"\bbump\b|\bupgrade .* from\b", re.I)
MANIFESTS = {
"package.json",
"package-lock.json",
"yarn.lock",
"pnpm-lock.yaml",
"composer.json",
"composer.lock",
"pom.xml",
"build.gradle",
"build.gradle.kts",
"gradle.lockfile",
"pyproject.toml",
"setup.py",
"setup.cfg",
"Pipfile",
"Pipfile.lock",
"poetry.lock",
"uv.lock",
}
def generated_path(path: str) -> bool:
return bool({"vendor", "node_modules", "dist"} & set(Path(path).parts)) or path.endswith(".min.js")
def manifest(path: str) -> bool:
name = Path(path).name
return name in MANIFESTS or name.endswith(".lock") or bool(re.fullmatch(r"requirements[^/]*\.txt", name))
def ordinary_rejection(row: dict, fixes: set[str], known_functions: set[tuple[str, str]]) -> str | None:
if len(row["parents"]) != 1:
return "merge_or_root"
if row["head"] in fixes:
return "advisory_fix"
if SECURITY.search(row["message"]):
return "security"
if re.search(r"^Revert|This reverts commit", row["message"], re.I | re.M):
return "revert"
if re.search(r"dependabot|renovate|\[bot\]", row["author"], re.I) or DEPENDENCY.search(row["message"]):
return "dependency"
files = row["files"]
if files and all(manifest(f["path"]) for f in files):
return "manifest_only"
source = [f for f in files if Path(f["path"]).suffix in LANGUAGES and not manifest(f["path"])]
if not source:
return "no_source"
if all(f["generated"] or generated_path(f["path"]) for f in source):
return "generated"
if any((f["path"], name) in known_functions for f in files for name in f["functions"]):
return "vulnerable_function"
if any((f["path"], "<global>") in known_functions for f in files):
return "vulnerable_function"
if sum(f["added"] + f["removed"] for f in source) > 1000:
return "size_cap"
return None
def generated(git, revision: str, path: str, text: str) -> bool:
import fnmatch
if generated_path(path):
return True
header = "\n".join(text.splitlines()[:20])
if re.search(r"(?im)^\s*(?:#|//|/\*|\*|<!--).*?(?:@generated|auto.?generated|generated\s+(?:by|file|code)|do not edit)", header):
return True
result = False
parts = Path(path).parts
for index in range(len(parts)):
directory = "/".join(parts[:index])
attributes = git.blob(revision, (directory + "/" if directory else "") + ".gitattributes") or ""
relative = "/".join(parts[index:])
for line in attributes.splitlines():
tokens = line.split()
if not tokens or tokens[0].startswith("#"):
continue
pattern = tokens[0].lstrip("/")
target = relative if "/" in pattern else Path(relative).name
if fnmatch.fnmatchcase(target, pattern):
for token in tokens[1:]:
if token in {"linguist-generated", "linguist-generated=true"}:
result = True
elif token in {"-linguist-generated", "!linguist-generated", "linguist-generated=false"}:
result = False
return result
def ordinary_history(git, head: str) -> list[dict]:
raw = git.run("log", "--first-parent", "--format=%H%x00%P%x00%ct%x00%an%x00%B%x1e", head)
rows = []
for entry in raw.split("\x1e"):
if not entry.strip():
continue
fields = entry.strip().split("\x00")
if len(fields) != 5 or not SHA.fullmatch(fields[0]):
raise InstrumentFailure("invalid_history")
sha, parents, timestamp, author, message = fields
row = {"head": sha, "parents": parents.split(), "timestamp": int(timestamp), "author": author, "message": message, "files": []}
rows.append(row)
if len(row["parents"]) != 1:
continue
base = row["parents"][0]
for path, added, removed in git.stats(base, sha):
hunks = git.hunks(base, sha, path)
info = {
"path": path,
"added": added,
"removed": removed,
"head_lines": [[max(1, n), max(1, n + max(c, 1) - 1)] for _, _, n, c in hunks],
"functions": [],
"generated": False,
}
if Path(path).suffix in LANGUAGES:
# Inspect both sides: deleting a vulnerable function is still touching it.
for revision, side in ((base, 0), (sha, 2)):
text = git.blob(revision, path)
if text is None:
continue
spans = functions(path, text)
for hunk in hunks:
start, count = hunk[side : side + 2]
for name, lo, hi in spans:
if max(1, start) <= hi and start + max(count, 1) - 1 >= lo:
info["functions"].append(name)
info["generated"] |= generated(git, revision, path, text)
row["files"].append(info)
return rows
def draw_ordinary(rows: list[dict], fixes: set[str], known_functions: set[tuple[str, str]], *, seed: int) -> tuple[list[dict], dict]:
import random
from collections import Counter, defaultdict
counts = Counter()
eligible = []
for row in rows:
reason = ordinary_rejection(row, fixes, known_functions)
counts[reason or "eligible"] += 1
if reason is None:
eligible.append(row)
if len(eligible) < 40:
failure = Rejected("ordinary_floor")
failure.counts = dict(counts)
raise failure
earliest, latest = min(r["timestamp"] for r in rows), max(r["timestamp"] for r in rows)
cells = defaultdict(list)
for row in eligible:
third = min(2, 3 * (row["timestamp"] - earliest) // max(1, latest - earliest + 1))
size = sum(f["added"] + f["removed"] for f in row["files"] if Path(f["path"]).suffix in LANGUAGES and not manifest(f["path"]))
bucket = 0 if size <= 10 else 1 if size <= 50 else 2 if size <= 200 else 3
cells[third, bucket].append(row)
if {t for t, _ in cells} != {0, 1, 2}:
failure = Rejected("time_thirds")
failure.counts = dict(counts)
raise failure
# Largest-deficit rounding with the mandatory minimum of one per time third.
# No cell can receive more draws than it has eligible commits.
allocation = Counter()
targets = {cell: 12 * len(items) / len(eligible) for cell, items in cells.items()}
for third in range(3):
cell = max(sorted(c for c in cells if c[0] == third), key=lambda c: targets[c])
allocation[cell] = 1
while sum(allocation.values()) < 12:
cell = max(sorted(c for c in cells if allocation[c] < len(cells[c])), key=lambda c: targets[c] - allocation[c])
allocation[cell] += 1
rng = random.Random(seed)
drawn = []
for cell in sorted(cells):
for row in rng.sample(sorted(cells[cell], key=lambda r: r["head"]), allocation[cell]):
drawn.append(
{
"kind": "ordinary",
"base": row["parents"][0],
"head": row["head"],
"method": "first_parent",
"time_third": cell[0],
"size_bucket": cell[1],
"files": [{"path": f["path"], "head_lines": f["head_lines"]} for f in row["files"]],
}
)
return drawn, dict(counts)
benchmarks/unseen/fixtures/advisory.json
+9
View File
@@ -0,0 +1,9 @@
{
"ghsa_id": "GHSA-test-test-test",
"published_at": "2026-06-02T00:00:00Z",
"type": "reviewed",
"withdrawn_at": null,
"cwes": [{"cwe_id": "CWE-89"}],
"vulnerabilities": [{"package": {"ecosystem": "pip", "name": "fixture"}}],
"references": ["https://github.com/fixture/project/commit/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"]
}
benchmarks/unseen/fixtures/scorer.json
+860
View File
@@ -0,0 +1,860 @@
[
{
"id": "0",
"repo": "0",
"kind": "ordinary",
"family": "injection",
"method": "introducing",
"post_cutoff": true,
"label_check": "confirmed",
"path": "app.py",
"function": "handler",
"files": [
{
"path": "app.py",
"head_lines": [
[
10,
12
]
]
}
],
"instrument": {
"head_verified": true,
"base_verified": true,
"changed_bytes": {
"app.py": 20
},
"hook_bytes_read": 20,
"hook_exit": 0
},
"push": {
"quick_tier": [
{
"files": [
"app.py"
],
"bytes_read": 20,
"findings": [
{
"cwe": "CWE-89",
"path": "app.py",
"line": 10
}
]
}
]
}
},
{
"id": "1",
"repo": "0",
"kind": "ordinary",
"family": "injection",
"method": "introducing",
"post_cutoff": true,
"label_check": "confirmed",
"path": "app.py",
"function": "handler",
"files": [
{
"path": "app.py",
"head_lines": [
[
10,
12
]
]
}
],
"instrument": {
"head_verified": true,
"base_verified": true,
"changed_bytes": {
"app.py": 20
},
"hook_bytes_read": 20,
"hook_exit": 0
},
"push": {
"quick_tier": [
{
"files": [
"app.py"
],
"bytes_read": 20,
"findings": []
}
]
}
},
{
"id": "2",
"repo": "1",
"kind": "ordinary",
"family": "injection",
"method": "introducing",
"post_cutoff": true,
"label_check": "confirmed",
"path": "app.py",
"function": "handler",
"files": [
{
"path": "app.py",
"head_lines": [
[
10,
12
]
]
}
],
"instrument": {
"head_verified": true,
"base_verified": true,
"changed_bytes": {
"app.py": 20
},
"hook_bytes_read": 20,
"hook_exit": 0
},
"push": {
"quick_tier": [
{
"files": [
"app.py"
],
"bytes_read": 20,
"findings": []
}
]
}
},
{
"id": "3",
"repo": "1",
"kind": "ordinary",
"family": "injection",
"method": "introducing",
"post_cutoff": true,
"label_check": "confirmed",
"path": "app.py",
"function": "handler",
"files": [
{
"path": "app.py",
"head_lines": [
[
10,
12
]
]
}
],
"instrument": {
"head_verified": true,
"base_verified": true,
"changed_bytes": {
"app.py": 20
},
"hook_bytes_read": 20,
"hook_exit": 0
},
"push": {
"quick_tier": [
{
"files": [
"app.py"
],
"bytes_read": 20,
"findings": []
}
]
}
},
{
"id": "4",
"repo": "2",
"kind": "ordinary",
"family": "injection",
"method": "introducing",
"post_cutoff": true,
"label_check": "confirmed",
"path": "app.py",
"function": "handler",
"files": [
{
"path": "app.py",
"head_lines": [
[
10,
12
]
]
}
],
"instrument": {
"head_verified": true,
"base_verified": true,
"changed_bytes": {
"app.py": 20
},
"hook_bytes_read": 20,
"hook_exit": 0
},
"push": {
"quick_tier": [
{
"files": [
"app.py"
],
"bytes_read": 20,
"findings": []
}
]
}
},
{
"id": "5",
"repo": "2",
"kind": "ordinary",
"family": "injection",
"method": "introducing",
"post_cutoff": true,
"label_check": "confirmed",
"path": "app.py",
"function": "handler",
"files": [
{
"path": "app.py",
"head_lines": [
[
10,
12
]
]
}
],
"instrument": {
"head_verified": true,
"base_verified": true,
"changed_bytes": {
"app.py": 20
},
"hook_bytes_read": 20,
"hook_exit": 0
},
"push": {
"quick_tier": [
{
"files": [
"app.py"
],
"bytes_read": 20,
"findings": []
}
]
}
},
{
"id": "6",
"repo": "3",
"kind": "ordinary",
"family": "injection",
"method": "introducing",
"post_cutoff": true,
"label_check": "confirmed",
"path": "app.py",
"function": "handler",
"files": [
{
"path": "app.py",
"head_lines": [
[
10,
12
]
]
}
],
"instrument": {
"head_verified": true,
"base_verified": true,
"changed_bytes": {
"app.py": 20
},
"hook_bytes_read": 20,
"hook_exit": 0
},
"push": {
"quick_tier": [
{
"files": [
"app.py"
],
"bytes_read": 20,
"findings": []
}
]
}
},
{
"id": "7",
"repo": "3",
"kind": "ordinary",
"family": "injection",
"method": "introducing",
"post_cutoff": true,
"label_check": "confirmed",
"path": "app.py",
"function": "handler",
"files": [
{
"path": "app.py",
"head_lines": [
[
10,
12
]
]
}
],
"instrument": {
"head_verified": true,
"base_verified": true,
"changed_bytes": {
"app.py": 20
},
"hook_bytes_read": 20,
"hook_exit": 0
},
"push": {
"quick_tier": [
{
"files": [
"app.py"
],
"bytes_read": 20,
"findings": []
}
]
}
},
{
"id": "8",
"repo": "4",
"kind": "ordinary",
"family": "injection",
"method": "introducing",
"post_cutoff": true,
"label_check": "confirmed",
"path": "app.py",
"function": "handler",
"files": [
{
"path": "app.py",
"head_lines": [
[
10,
12
]
]
}
],
"instrument": {
"head_verified": true,
"base_verified": true,
"changed_bytes": {
"app.py": 20
},
"hook_bytes_read": 20,
"hook_exit": 0
},
"push": {
"quick_tier": [
{
"files": [
"app.py"
],
"bytes_read": 20,
"findings": []
}
]
}
},
{
"id": "9",
"repo": "4",
"kind": "ordinary",
"family": "injection",
"method": "introducing",
"post_cutoff": true,
"label_check": "confirmed",
"path": "app.py",
"function": "handler",
"files": [
{
"path": "app.py",
"head_lines": [
[
10,
12
]
]
}
],
"instrument": {
"head_verified": true,
"base_verified": true,
"changed_bytes": {
"app.py": 20
},
"hook_bytes_read": 20,
"hook_exit": 0
},
"push": {
"quick_tier": [
{
"files": [
"app.py"
],
"bytes_read": 20,
"findings": []
}
]
}
},
{
"id": "10",
"repo": "5",
"kind": "ordinary",
"family": "injection",
"method": "introducing",
"post_cutoff": true,
"label_check": "confirmed",
"path": "app.py",
"function": "handler",
"files": [
{
"path": "app.py",
"head_lines": [
[
10,
12
]
]
}
],
"instrument": {
"head_verified": true,
"base_verified": true,
"changed_bytes": {
"app.py": 20
},
"hook_bytes_read": 20,
"hook_exit": 0
},
"push": {
"quick_tier": [
{
"files": [
"app.py"
],
"bytes_read": 20,
"findings": []
}
]
}
},
{
"id": "11",
"repo": "5",
"kind": "ordinary",
"family": "injection",
"method": "introducing",
"post_cutoff": true,
"label_check": "confirmed",
"path": "app.py",
"function": "handler",
"files": [
{
"path": "app.py",
"head_lines": [
[
10,
12
]
]
}
],
"instrument": {
"head_verified": true,
"base_verified": true,
"changed_bytes": {
"app.py": 20
},
"hook_bytes_read": 20,
"hook_exit": 0
},
"push": {
"quick_tier": [
{
"files": [
"app.py"
],
"bytes_read": 20,
"findings": []
}
]
}
},
{
"id": "12",
"repo": "6",
"kind": "ordinary",
"family": "injection",
"method": "introducing",
"post_cutoff": true,
"label_check": "confirmed",
"path": "app.py",
"function": "handler",
"files": [
{
"path": "app.py",
"head_lines": [
[
10,
12
]
]
}
],
"instrument": {
"head_verified": true,
"base_verified": true,
"changed_bytes": {
"app.py": 20
},
"hook_bytes_read": 20,
"hook_exit": 0
},
"push": {
"quick_tier": [
{
"files": [
"app.py"
],
"bytes_read": 20,
"findings": []
}
]
}
},
{
"id": "13",
"repo": "6",
"kind": "ordinary",
"family": "injection",
"method": "introducing",
"post_cutoff": true,
"label_check": "confirmed",
"path": "app.py",
"function": "handler",
"files": [
{
"path": "app.py",
"head_lines": [
[
10,
12
]
]
}
],
"instrument": {
"head_verified": true,
"base_verified": true,
"changed_bytes": {
"app.py": 20
},
"hook_bytes_read": 20,
"hook_exit": 0
},
"push": {
"quick_tier": [
{
"files": [
"app.py"
],
"bytes_read": 20,
"findings": []
}
]
}
},
{
"id": "14",
"repo": "7",
"kind": "ordinary",
"family": "injection",
"method": "introducing",
"post_cutoff": true,
"label_check": "confirmed",
"path": "app.py",
"function": "handler",
"files": [
{
"path": "app.py",
"head_lines": [
[
10,
12
]
]
}
],
"instrument": {
"head_verified": true,
"base_verified": true,
"changed_bytes": {
"app.py": 20
},
"hook_bytes_read": 20,
"hook_exit": 0
},
"push": {
"quick_tier": [
{
"files": [
"app.py"
],
"bytes_read": 20,
"findings": []
}
]
}
},
{
"id": "15",
"repo": "7",
"kind": "ordinary",
"family": "injection",
"method": "introducing",
"post_cutoff": true,
"label_check": "confirmed",
"path": "app.py",
"function": "handler",
"files": [
{
"path": "app.py",
"head_lines": [
[
10,
12
]
]
}
],
"instrument": {
"head_verified": true,
"base_verified": true,
"changed_bytes": {
"app.py": 20
},
"hook_bytes_read": 20,
"hook_exit": 0
},
"push": {
"quick_tier": [
{
"files": [
"app.py"
],
"bytes_read": 20,
"findings": []
}
]
}
},
{
"id": "16",
"repo": "8",
"kind": "vulnerability",
"family": "injection",
"method": "introducing",
"post_cutoff": true,
"label_check": "confirmed",
"path": "app.py",
"function": "handler",
"files": [
{
"path": "app.py",
"head_lines": [
[
10,
12
]
]
}
],
"instrument": {
"head_verified": true,
"base_verified": true,
"changed_bytes": {
"app.py": 20
},
"hook_bytes_read": 20,
"hook_exit": 0
},
"push": {
"quick_tier": [
{
"files": [
"app.py"
],
"bytes_read": 20,
"findings": [
{
"cwe": "CWE-89",
"path": "app.py",
"line": 10
}
]
}
]
}
},
{
"id": "17",
"repo": "8",
"kind": "vulnerability",
"family": "injection",
"method": "introducing",
"post_cutoff": true,
"label_check": "confirmed",
"path": "app.py",
"function": "handler",
"files": [
{
"path": "app.py",
"head_lines": [
[
10,
12
]
]
}
],
"instrument": {
"head_verified": true,
"base_verified": true,
"changed_bytes": {
"app.py": 20
},
"hook_bytes_read": 20,
"hook_exit": 0
},
"push": {
"quick_tier": [
{
"files": [
"app.py"
],
"bytes_read": 20,
"findings": [
{
"cwe": "CWE-89",
"path": "app.py",
"line": 10
}
]
}
]
}
},
{
"id": "18",
"repo": "9",
"kind": "vulnerability",
"family": "injection",
"method": "introducing",
"post_cutoff": true,
"label_check": "confirmed",
"path": "app.py",
"function": "handler",
"files": [
{
"path": "app.py",
"head_lines": [
[
10,
12
]
]
}
],
"instrument": {
"head_verified": true,
"base_verified": true,
"changed_bytes": {
"app.py": 20
},
"hook_bytes_read": 20,
"hook_exit": 0
},
"push": {
"quick_tier": [
{
"files": [
"app.py"
],
"bytes_read": 20,
"findings": []
}
]
}
},
{
"id": "19",
"repo": "9",
"kind": "vulnerability",
"family": "injection",
"method": "introducing",
"post_cutoff": true,
"label_check": "confirmed",
"path": "app.py",
"function": "handler",
"files": [
{
"path": "app.py",
"head_lines": [
[
10,
12
]
]
}
],
"instrument": {
"head_verified": true,
"base_verified": true,
"changed_bytes": {
"app.py": 20
},
"hook_bytes_read": 20,
"hook_exit": 0
},
"push": {
"quick_tier": [
{
"files": [
"app.py"
],
"bytes_read": 20,
"findings": []
}
]
}
}
]
benchmarks/unseen/journal.py
+184
View File
@@ -0,0 +1,184 @@
"""Private, append-only sourcing checkpoints. No identity-bearing data leaves this directory."""
from __future__ import annotations
import fcntl
import hashlib
import json
import math
import os
import time
from contextlib import contextmanager
from pathlib import Path
STAGES = ("advisories", "resolution", "eligibility", "extraction", "completion")
def encoded(value) -> bytes:
return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode()
def digest(value) -> str:
return hashlib.sha256(encoded(value)).hexdigest()
class ResumeMismatch(ValueError):
def __init__(self, parameters):
self.parameters = parameters
super().__init__("resume_inputs_changed")
class Stopped(Exception):
pass
def external(root: Path, path: Path) -> Path:
path = path.expanduser().resolve()
# Also reject the actual checkout when a different --root is supplied.
if any(path.is_relative_to(p.resolve()) for p in (root, Path(__file__).resolve().parents[2])):
raise ValueError("cache_inside_repository")
return path
def atomic(path: Path, payload: bytes) -> None:
temporary = path.with_suffix(path.suffix + ".tmp")
with temporary.open("wb") as handle:
os.chmod(temporary, 0o600)
handle.write(payload)
handle.flush()
os.fsync(handle.fileno())
temporary.replace(path)
sync_directory(path.parent)
def sync_directory(path: Path) -> None:
fd = os.open(path, os.O_RDONLY | os.O_DIRECTORY)
try:
os.fsync(fd)
finally:
os.close(fd)
class Journal:
def __init__(self, path: Path, *, max_hours=None, clock=None, rate_limit=None):
if max_hours is not None and (not math.isfinite(max_hours) or max_hours < 0):
raise ValueError("invalid_max_hours")
self.path = path
self.clock = clock or time.monotonic
self.started = self.clock()
self.max_seconds = None if max_hours is None else max_hours * 3600
self.rate_limit = rate_limit or (lambda: {})
self.rows = {stage: {} for stage in STAGES}
self.elapsed_before = 0.0
self.last = None
self.previous_rate = {}
@contextmanager
def open(self, parameters):
self.path.mkdir(parents=True, exist_ok=True, mode=0o700)
with (self.path / "LOCK").open("a") as lock:
try:
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
except BlockingIOError:
raise ValueError("journal_busy") from None
metadata = self.path / "inputs.json"
if metadata.exists():
old = json.loads(metadata.read_bytes())
# Only current, fixed parameter labels can enter stdout.
changed = [key for key in parameters if old.get(key) != parameters[key]]
if old.keys() != parameters.keys():
changed.append("parameter_schema")
if changed:
raise ResumeMismatch(changed)
else:
if any(self.path.glob("*.jsonl")):
raise ValueError("journal_missing_inputs")
atomic(metadata, encoded(parameters))
for stage in STAGES:
path = self.path / (stage + ".jsonl")
if not path.exists():
continue
with path.open("rb") as handle:
for line in handle:
# Fail closed on torn/corrupt writes; never silently lose a completed unit.
if not line.endswith(b"\n"):
raise ValueError("journal_incomplete_record")
row = json.loads(line)
if row["id"] in self.rows[stage]:
raise ValueError("journal_duplicate_unit")
self.rows[stage][row["id"]] = row["value"]
if row["elapsed_seconds"] >= self.elapsed_before:
self.elapsed_before = row["elapsed_seconds"]
self.last = {"stage": stage, "id": row["id"]}
self.previous_rate = row["rate_limit"]
self.progress("sourcing")
yield self
def boundary(self):
reason = None
if (self.path / "STOP").exists():
reason = "STOP"
elif self.max_seconds is not None and self.clock() - self.started >= self.max_seconds:
reason = "max_hours"
if reason:
self.progress("stopped", reason=reason)
raise Stopped(reason)
def elapsed(self):
return self.elapsed_before + self.clock() - self.started
def progress(self, status, **extra):
atomic(
self.path / "progress.json",
encoded(
{
"status": status,
"counts": {stage: len(rows) for stage, rows in self.rows.items()},
"last_unit": self.last,
"rate_limit": self.rate_limit() or self.previous_rate,
"elapsed_seconds": self.elapsed(),
**extra,
}
),
)
def put(self, stage, key, value):
if key in self.rows[stage]:
raise ValueError("journal_duplicate_unit")
row = {"id": key, "value": value, "elapsed_seconds": self.elapsed(), "rate_limit": self.rate_limit() or self.previous_rate}
path = self.path / (stage + ".jsonl")
with path.open("ab") as handle:
os.chmod(path, 0o600)
handle.write(encoded(row) + b"\n")
handle.flush()
os.fsync(handle.fileno())
sync_directory(self.path)
self.rows[stage][key] = value
self.last = {"stage": stage, "id": key}
self.progress("sourcing")
class ResolvedGitHub:
"""Persist successful repository lookups and 404s used by both eligibility and the fix index."""
def __init__(self, github, journal):
self.github, self.journal = github, journal
def __getattr__(self, name):
return getattr(self.github, name)
def get_repo(self, name):
from .eligibility import RepositoryNotFound, normalize
key = normalize(name)
if key not in self.journal.rows["resolution"]:
self.journal.boundary()
try:
value = self.github.get_repo(key)
except RepositoryNotFound:
value = None
self.journal.put("resolution", key, value)
value = self.journal.rows["resolution"][key]
if value is None:
raise RepositoryNotFound("not_found")
return value
benchmarks/unseen/ledger.py
+88
View File
@@ -0,0 +1,88 @@
"""Append-only slice usage; a hash chain detects rewritten rows, not a rebuilt entire history.
Keep the published final digest to authenticate a history against truncation or wholesale replacement.
Only opaque decision/experiment labels belong here; repository identities are never accepted.
"""
from __future__ import annotations
import fcntl
import hashlib
import json
import re
from datetime import date
from pathlib import Path
FIELDS = {"slice", "decision", "purpose", "date", "result_digest", "freeze_digest"}
PURPOSES = {"baseline", "exploratory", "informed", "qualifies"}
ZERO = "0" * 64
def digest(row: dict) -> str:
return hashlib.sha256(json.dumps(row, sort_keys=True, separators=(",", ":")).encode()).hexdigest()
def validate(row: dict) -> None:
if set(row) != FIELDS or row["purpose"] not in PURPOSES:
raise ValueError("invalid_usage_fields")
if type(row["slice"]) is not int or row["slice"] < 1:
raise ValueError("invalid_slice")
if not isinstance(row["decision"], str) or not re.fullmatch(r"[A-Za-z0-9_-]+", row["decision"]):
raise ValueError("invalid_decision_label")
for key in ("result_digest", "freeze_digest"):
if not isinstance(row[key], str) or not re.fullmatch(r"[0-9a-f]{64}", row[key]):
raise ValueError("invalid_digest")
try:
if date.fromisoformat(row["date"]).isoformat() != row["date"]:
raise ValueError
except (ValueError, TypeError):
raise ValueError("invalid_date") from None
def _read(text: str) -> list[dict]:
previous = ZERO
rows = []
if text and not text.endswith("\n"):
raise ValueError("broken_digest_chain")
for line in text.splitlines():
try:
row = json.loads(line)
stored = row.pop("digest")
if row["previous"] != previous or digest(row) != stored:
raise ValueError
validate({k: v for k, v in row.items() if k != "previous"})
except (ValueError, KeyError, TypeError, AttributeError):
raise ValueError("broken_digest_chain") from None
rows.append({**row, "digest": stored})
previous = stored
return rows
def read(path: Path) -> list[dict]:
if not path.exists():
return []
with path.open() as handle:
fcntl.flock(handle, fcntl.LOCK_SH)
return _read(handle.read())
def append(path: Path, row: dict) -> str:
validate(row)
path.parent.mkdir(parents=True, exist_ok=True)
with path.open("a+") as handle:
fcntl.flock(handle, fcntl.LOCK_EX)
handle.seek(0)
rows = _read(handle.read())
if any(all(old[key] == row[key] for key in FIELDS) for old in rows):
raise ValueError("duplicate_usage")
if rows and rows[0]["freeze_digest"] != row["freeze_digest"]:
raise ValueError("freeze_digest_changed")
if row["purpose"] == "qualifies" and any(
old["slice"] == row["slice"] and old["decision"] == row["decision"] and old["purpose"] == "informed" for old in rows
):
raise ValueError("slice_informed_decision")
entry = {**row, "previous": rows[-1]["digest"] if rows else ZERO}
sha = digest(entry)
handle.write(json.dumps({**entry, "digest": sha}, sort_keys=True) + "\n")
handle.flush()
return sha
benchmarks/unseen/proof.py
+128
View File
@@ -0,0 +1,128 @@
"""Build anonymous coordinator proof records from completed dispatcher outputs."""
from __future__ import annotations
import argparse
import hashlib
import json
import subprocess
from pathlib import Path
def digest(value):
return hashlib.sha256(json.dumps(value, sort_keys=True).encode()).hexdigest()
def records(root):
return [json.loads(path.read_text()) for path in sorted(root.glob("*.json")) if path.name not in {"progress.json", "inputs.json"}]
def replay_proof(roots):
lanes = {lane: records(root) for lane, root in roots.items()}
if any(len(rows) != 5 for rows in lanes.values()):
raise ValueError("proof requires exactly five completed changes per lane")
keys = [{row["input_digest"] for row in rows} for rows in lanes.values()]
if not all(key == keys[0] for key in keys):
raise ValueError("lane input identities differ")
quick = {}
evidence = {}
for lane, rows in lanes.items():
quick[lane] = {}
evidence[lane] = []
for row in rows:
inst = row.get("instrument", {})
if (
row.get("status") not in {"quick_only", "engine_covered"}
or not inst.get("hook_bytes_read")
or inst.get("suspect_fast")
or inst.get("jvm_wall_seconds", 0) < 5
):
raise ValueError("proof instrument has unread input or implausible JVM time")
quick[lane][row["input_digest"]] = digest([t.get("findings", []) for t in row["push"].get("quick_tier", [])])
evidence[lane].append(
{
"input_digest": row["input_digest"],
"seconds": inst["seconds"],
"jvm_seconds": inst["jvm_wall_seconds"],
"bytes": inst["hook_bytes_read"],
"exit_code": inst["hook_exit"],
}
)
equal = all(value == next(iter(quick.values())) for value in quick.values())
if not equal:
raise ValueError("quick-tier findings differ across lanes")
return {"lanes": evidence, "quick_tier_equal": True, "quick_digests": quick, "cleanup": "completed-before-checkpoint"}
def engine_proof(root, baseline):
current = records(root)
old = {digest([r["repo"], r["pin"]]): r for r in records(baseline) if "repo" in r and "pin" in r}
current = [r for r in current if "repo" in r and "pin" in r]
if len(current) != 2:
raise ValueError("engine proof requires exactly two pins")
evidence = []
def stable(record):
return [
{key: u.get(key) for key in ("unit", "status", "questions_asked", "questions_completed", "witness_rows", "traces")}
for u in record["units"]
]
for row in current:
key = digest([row["repo"], row["pin"]])
previous = old[key]
instruments = [u.get("instrument", {}) for u in row["units"]]
if (
not row.get("done")
or not any(i.get("bytes", 0) > 0 for i in instruments)
or not any(stage.get("seconds", 0) >= 5 for i in instruments for stage in i.get("jvm", []))
):
raise ValueError("engine proof lacks bytes or realistic JVM time")
evidence.append(
{
"pin_digest": key,
"seconds": row.get("seconds"),
"record_digest": digest(stable(row)),
"baseline_digest": digest(stable(previous)),
"equal": stable(row) == stable(previous),
"instrument": [
{
"bytes": u.get("instrument", {}).get("bytes", 0),
"cpg_bytes": u.get("instrument", {}).get("cpg_bytes", 0),
"jvm": [
{key: stage.get(key) for key in ("seconds", "exit_code", "success", "heap_mb")}
for stage in u.get("instrument", {}).get("jvm", [])
],
}
for u in row["units"]
],
}
)
return {"pins": evidence, "differences": sum(not e["equal"] for e in evidence), "cleanup": "completed-before-checkpoint"}
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("kind", choices=("engine", "replay"))
parser.add_argument("--image", required=True)
parser.add_argument("--out", type=Path, required=True)
parser.add_argument("--records", type=Path)
parser.add_argument("--baseline", type=Path)
parser.add_argument("--kind-records", type=Path)
parser.add_argument("--ax-records", type=Path)
parser.add_argument("--vm-records", type=Path)
args = parser.parse_args()
result = (
engine_proof(args.records, args.baseline)
if args.kind == "engine"
else replay_proof({"kind": args.kind_records, "ax": args.ax_records, "vm": args.vm_records})
)
result.update(
image=args.image, commit=subprocess.run(["git", "rev-parse", "HEAD"], capture_output=True, text=True, check=True).stdout.strip()
)
args.out.parent.mkdir(parents=True, exist_ok=True)
args.out.write_text(json.dumps(result, sort_keys=True, indent=2) + "\n")
if __name__ == "__main__":
main()
benchmarks/unseen/replay.py
+193
View File
@@ -0,0 +1,193 @@
"""Dispatch frozen unseen changes or a five-change project-history proof."""
from __future__ import annotations
import argparse
import hashlib
import io
import json
import subprocess
import tarfile
import tomllib
from pathlib import Path
from benchmarks.ax.batch import AXLane, DockerLane, Item, Workload, dispatch, write_json
from benchmarks.unseen.score import coverage
def make_item(change, arm, image):
if arm not in {"default", "long_deadline"}:
raise ValueError("unknown replay arm")
identity = hashlib.sha256(str(change["id"]).encode()).hexdigest()[:14]
spec = {**change, "id": identity, "hook_flags": [] if arm == "default" else ["--deadline", "300"], "image": image}
data = json.dumps(spec, sort_keys=True).encode()
digest = hashlib.sha256(data).hexdigest()
return Item(
"replay-" + identity,
{"SPEC_URL": data},
"result.dat",
("python3", "/app/benchmarks/unseen/replay_entry.py"),
{},
digest,
{"SPEC_URL": "spec.dat"},
)
def validate_result(data, output):
with tarfile.open(fileobj=io.BytesIO(data)) as archive:
members = archive.getmembers()
required = {"push.json", "push.txt", "engine.json", "instrument.json"}
if {m.name for m in members} != required or any(not m.isfile() for m in members):
raise ValueError("replay result requires push, engine and instrument.json")
payload = {m.name: archive.extractfile(m).read() for m in members}
instrument = json.loads(payload["instrument.json"])
push = json.loads(payload["push.json"])
record = {"instrument": instrument, "push": push}
record["status"] = coverage(record)
# Transport/scheduler needs an explicit OOM signal, not a quiet failed replay.
if "OutOfMemoryError" in json.dumps(push) or "JVM out of memory" in json.dumps(push):
record["status"] = "oom"
output.mkdir(parents=True, exist_ok=True)
for name, body in payload.items():
(output / name).write_bytes(body)
(output / "result.dat").write_bytes(data)
write_json(output / "result.json", record)
return record
def proof_changes(root):
"""Only the operator's project history; no pool data or fetches."""
def git(*args):
return subprocess.run(["git", "-C", str(root), *args], capture_output=True, text=True, check=True).stdout.strip()
changes = []
for head in git("rev-list", "--no-merges", "--max-count=100", "HEAD", "--", "src").splitlines():
base = git("rev-parse", head + "^")
changed = git("diff", "--name-only", base, head).splitlines()
if not any(name.startswith("src/") and name.endswith(".py") for name in changed):
continue
changes.append(
{
"id": hashlib.sha256(head.encode()).hexdigest(),
"repository_url": "https://github.com/norandom/OpenUltraSAST.git",
"repo": "project-proof",
"kind": "ordinary",
"base": base,
"head": head,
}
)
if len(changes) == 5:
break
if len(changes) != 5:
raise ValueError("history lacks five source changes")
return changes
def canonical(value):
return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode()
def load_slice(path, slice_id, private_path=None):
if path.suffix != ".toml":
frozen = json.loads(path.read_text())
changes = frozen["changes"]
if frozen["sha256"] != hashlib.sha256(json.dumps(changes, sort_keys=True).encode()).hexdigest():
raise ValueError("frozen changes digest mismatch")
return changes
if not path.name.startswith("pool-"):
raise ValueError("replay requires a frozen pool, never a draft")
pool = tomllib.loads(path.read_text())
freeze = json.loads(path.with_name("freeze-" + path.stem.removeprefix("pool-") + ".json").read_text())
if hashlib.sha256(canonical(pool)).hexdigest() != freeze["freeze_digest"]:
raise ValueError("frozen pool digest mismatch")
private_path = private_path or path.parent / "private" / path.name
private = {}
if private_path.exists():
private = {row["id"]: row for row in tomllib.loads(private_path.read_text()).get("repository", [])}
changes = []
for public in pool.get("repository", []):
if str(public["slice"]) != str(slice_id):
continue
row = public
if public.get("private"):
row = private.get(public["id"])
if row is None:
raise ValueError("missing private pool entry")
payload = {"url": row["url"], "commits": [[c["base"], c["head"]] for c in row["changes"]]}
if hashlib.sha256(canonical(payload)).hexdigest() != public["sha256"]:
raise ValueError("private pool digest mismatch")
for change in row["changes"]:
identity = hashlib.sha256(canonical([public["id"], change["kind"], change["base"], change["head"]])).hexdigest()
changes.append(
{
**change,
"id": identity,
"repo": public["id"],
"repository_url": row["url"],
"freeze_digest": freeze["freeze_digest"],
"slice": slice_id,
}
)
if not changes:
raise ValueError("frozen slice has no changes")
return changes
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--manifest", type=Path, required=True, help="frozen JSON: changes array plus sha256 of canonical changes JSON")
parser.add_argument("--proof-history", action="store_true", help="write a five-change own-history manifest, then exit")
parser.add_argument("--image")
parser.add_argument("--private-manifest", type=Path)
parser.add_argument("--dry-run", action="store_true")
parser.add_argument("--arm", choices=("default", "long_deadline"), default="default")
parser.add_argument("--pool", default="p1")
parser.add_argument("--slice", default="1")
parser.add_argument("--out", type=Path)
parser.add_argument("--lane", choices=("ax", "kind", "docker", "mixed"), default="mixed")
parser.add_argument("--ax-bin", default="ax")
parser.add_argument("--kubeconfig", type=Path)
parser.add_argument("--atespace", default="default")
args = parser.parse_args()
if args.proof_history:
changes = proof_changes(Path.cwd())
write_json(args.manifest, {"changes": changes, "sha256": hashlib.sha256(json.dumps(changes, sort_keys=True).encode()).hexdigest()})
return 0
if not args.image:
parser.error("--image digest is required")
changes = load_slice(args.manifest, args.slice, args.private_manifest)
items = [make_item(change, args.arm, args.image) for change in changes]
if len({i.id for i in items}) != len(items):
parser.error("duplicate change identity")
args.out = args.out or Path.home() / "ousast-results/unseen" / args.pool / args.slice / args.arm
if args.dry_run:
print(json.dumps({"items": len(items), "input_digest": hashlib.sha256(canonical([i.input_digest for i in items])).hexdigest()}))
return 0
if args.out.resolve().is_relative_to(Path(__file__).resolve().parents[2]):
parser.error("raw replay output must be outside the repository")
write_json(
args.out / "inputs.json",
[{"item_id": item.id, "change": change, "input_digest": item.input_digest} for item, change in zip(items, changes, strict=True)],
)
workload = Workload(args.image, frozenset({"SPEC_URL", "RESULT_URL"}), 600 if args.arm == "default" else 900, validate_result)
lanes = ("ax", "ax", "docker") if args.lane == "mixed" else (args.lane,) * (2 if args.lane in {"ax", "kind"} else 1)
executors = {}
for lane in set(lanes):
transport = (DockerLane if lane == "docker" else AXLane)(args, workload)
def execute(item, output, attempt, transport=transport):
record = {**json.loads(item.inputs["SPEC_URL"]), **transport(item, output, attempt)}
if (output / "result.dat").exists():
# Durable archive lives separately from attempt staging objects, which are deleted.
key = f"unseen/{item.input_digest}/result.dat"
transport.store._put(key, (output / "result.dat").read_bytes())
record["result_object"] = key
return record
executors[lane] = execute
return 2 if dispatch(items, workload, args.out, executors, lanes=lanes) else 0
if __name__ == "__main__":
raise SystemExit(main())
benchmarks/unseen/replay_entry.py
+219
View File
@@ -0,0 +1,219 @@
"""Credential-free replay actor. Standard library only; PUT is completion."""
from __future__ import annotations
import argparse
import json
import os
import re
import subprocess
import sys
import tarfile
import time
import urllib.error
import urllib.request
from pathlib import Path
from urllib.parse import urlsplit
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "learn"))
from engine_task_entry import transfer
CLUSTER_FLAGS = (
"-Xms1700m -Xmx1700m -XX:MaxMetaspaceSize=256m -XX:ReservedCodeCacheSize=128m "
"-XX:MaxDirectMemorySize=256m -Xss512k -XX:ActiveProcessorCount=2 -XX:+UseG1GC"
)
class ReplayFailure(Exception):
"""Locally controlled diagnostic only."""
def validate_spec(spec):
parts = urlsplit(spec["repository_url"])
if (
parts.scheme != "https"
or not parts.hostname
or "." not in parts.hostname
or parts.username
or parts.password
or parts.query
or parts.fragment
):
raise ReplayFailure("repository must be public HTTPS")
if parts.port not in (None, 443) or parts.hostname.endswith((".local", ".internal", ".localhost")):
raise ReplayFailure("repository must be public HTTPS")
import ipaddress
try:
ipaddress.ip_address(parts.hostname)
except ValueError:
pass
else:
raise ReplayFailure("repository must use public DNS")
if any(not re.fullmatch("[0-9a-f]{40,64}", spec[key]) for key in ("base", "head")):
raise ReplayFailure("invalid revision")
if spec.get("hook_flags", []) not in ([], ["--deadline", "300"]):
raise ReplayFailure("unsupported hook flags")
def main(*, work_root=Path("/tmp"), runner=subprocess.run, heap_profile="cluster", clock=time.monotonic):
output = work_root / "out"
output.mkdir(parents=True, exist_ok=True)
instrument = {
"status": "instrument_failure",
"head_verified": False,
"base_verified": False,
"changed_bytes": {},
"checkout_files": 0,
"checkout_bytes": 0,
"hook_bytes_read": 0,
"jvm_wall_seconds": 0,
"jvm_time_basis": "scan build_seconds + query_seconds",
"hook_exit": None,
"timings": {},
"heap_profile": heap_profile,
}
spec = {}
step = "GET spec"
started = clock()
def run(command, *, timeout=180, env=None):
before = clock()
result = runner(command, capture_output=True, text=True, timeout=timeout, env=env)
instrument["timings"][step] = instrument["timings"].get(step, 0) + clock() - before
if result.returncode:
raise ReplayFailure(f"exit {result.returncode}")
return result
try:
spec_path = work_root / "spec.dat"
transfer(os.environ["SPEC_URL"], spec_path, readiness_budget=60)
step = "validate spec"
spec = json.loads(spec_path.read_text())
instrument["image"] = spec.get("image")
validate_spec(spec)
case = work_root / "case"
case.mkdir()
git = ["git", "-c", "core.hooksPath=/dev/null", "-c", "protocol.file.allow=never", "-C", str(case)]
step = "git init"
run([*git, "init", "-q"])
run([*git, "remote", "add", "origin", spec["repository_url"]])
# One total fetch budget, independent of the hook deadline.
fetch_end = clock() + 180
for key in ("head", "base"):
step = "fetch " + key
remaining = fetch_end - clock()
if remaining <= 0:
raise ReplayFailure("fetch deadline")
run([*git, "fetch", "--no-tags", "--depth", "1", "origin", spec[key]], timeout=remaining)
step = "verify " + key
run([*git, "cat-file", "-e", spec[key] + "^{commit}"])
instrument[key + "_verified"] = True
step = "checkout"
run([*git, "checkout", "--detach", spec["head"]])
paths = run([*git, "ls-files", "-z"]).stdout.split("\0")
for name in filter(None, paths):
path = case / name
if path.is_file() and not path.is_symlink():
data = path.read_bytes()
instrument["checkout_files"] += 1
instrument["checkout_bytes"] += len(data)
if instrument["checkout_bytes"] > 500 * 1024 * 1024:
raise ReplayFailure("checkout size limit")
step = "changed files"
changed = run([*git, "diff", "--name-only", "--diff-filter=ACMRT", "-z", spec["base"], spec["head"]]).stdout.split("\0")
for name in filter(None, changed):
path = case / name
if path.is_file() and not path.is_symlink() and path.resolve().is_relative_to(case.resolve()):
instrument["changed_bytes"][name] = len(path.read_bytes())
if not instrument["checkout_bytes"] or not sum(instrument["changed_bytes"].values()):
instrument["suspect_fast"] = True
raise ReplayFailure("no changed bytes read")
step = "pre-push"
env = {
key: value for key, value in os.environ.items() if key not in {"SPEC_URL", "RESULT_URL", "_JAVA_OPTIONS", "JDK_JAVA_OPTIONS"}
}
env["OPENULTRASAST_CPG_HEAP_MB"] = "1700" if heap_profile == "cluster" else "2560"
env["JAVA_TOOL_OPTIONS"] = CLUSTER_FLAGS if heap_profile == "cluster" else "-Xmx2560m"
command = [
"ousast",
"pre-push",
str(case),
"--base",
spec["base"],
"--head",
spec["head"],
"--artifact",
str(output / "push.json"),
*spec.get("hook_flags", []),
]
before = clock()
completed = runner(command, capture_output=True, text=True, timeout=330 if spec.get("hook_flags") else 60, env=env)
instrument["timings"][step] = clock() - before
instrument["hook_exit"] = completed.returncode
(output / "push.txt").write_text(completed.stdout)
if completed.returncode not in (0, 1):
raise ReplayFailure(f"hook exit {completed.returncode}")
step = "read push artifact"
push = json.loads((output / "push.json").read_text())
instrument["hook_bytes_read"] = sum(t.get("bytes_read", 0) for t in push.get("quick_tier", []))
scans = push.get("scans", [])
instrument["jvm_wall_seconds"] = sum(s.get("scan", {}).get(k, 0) for s in scans for k in ("build_seconds", "query_seconds"))
uncovered = "language_not_covered" in json.dumps(push.get("admission", {}).get("coverage_reasons", []))
partitions = [part for scan in scans for part in scan.get("scan", {}).get("partitions", [])]
unsupported_only = (
uncovered
and instrument["hook_bytes_read"] == 0
and bool(partitions)
and all(part.get("status") == "unsupported" for part in partitions)
)
instrument["suspect_fast"] = bool(scans) and not unsupported_only and instrument["jvm_wall_seconds"] < 5
if instrument["suspect_fast"] or (instrument["hook_bytes_read"] <= 0 and not uncovered):
instrument["suspect_fast"] = True
raise ReplayFailure("suspect_fast or unread hook input")
engine = {
"repository": spec["repository_url"],
"revisions": {key: spec[key] for key in ("base", "head")},
"scans": scans,
"admission": push.get("admission", {}),
}
(output / "engine.json").write_text(json.dumps(engine))
instrument["status"] = "ok"
except Exception as exc:
detail = (
f"HTTP {exc.code}"
if isinstance(exc, urllib.error.HTTPError)
else str(exc)
if isinstance(exc, ReplayFailure)
else type(exc).__name__
)
instrument["failure"] = {"step": step, "status": detail}
# A crash artifact must never be mistaken for a completed scan.
if not (output / "push.json").exists():
(output / "push.json").write_text("{}")
if not (output / "push.txt").exists():
(output / "push.txt").write_text("")
if not (output / "engine.json").exists():
(output / "engine.json").write_text("{}")
instrument["seconds"] = clock() - started
(output / "instrument.json").write_text(json.dumps(instrument))
try:
archive = work_root / "result.dat"
with tarfile.open(archive, "w") as tar:
for name in ("push.json", "push.txt", "engine.json", "instrument.json"):
tar.add(output / name, arcname=name)
request = urllib.request.Request(
os.environ["RESULT_URL"], data=archive.read_bytes(), method="PUT", headers={"Content-Type": "application/x-tar"}
)
transfer(request, readiness_budget=60)
except Exception as exc:
detail = f"HTTP {exc.code}" if isinstance(exc, urllib.error.HTTPError) else type(exc).__name__
print("PUT result: " + detail, file=sys.stderr)
return 1
return int(instrument["status"] != "ok")
if __name__ == "__main__":
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--heap-profile", choices=("cluster", "vm"), default="cluster")
raise SystemExit(main(heap_profile=parser.parse_args().heap_profile))
benchmarks/unseen/score.py
+308
View File
@@ -0,0 +1,308 @@
"""Push-level unseen evaluation, explicit coverage and paired adoption evidence."""
from __future__ import annotations
import argparse
import hashlib
import json
import math
import random
from collections import Counter, defaultdict
from pathlib import Path
from openultrasast.model.taxonomy import load_families
from openultrasast.push.report import CAPABILITY_VETOES
STATES = ("instrument_failure", "unanalysable", "quick_only", "engine_covered")
def coverage(record):
instrument = record.get("instrument", {})
push = record.get("push")
if (
not isinstance(push, dict)
or not instrument.get("head_verified")
or not instrument.get("base_verified")
or instrument.get("suspect_fast")
or instrument.get("failure")
or instrument.get("hook_exit") not in (0, 1)
or sum(instrument.get("changed_bytes", {}).values()) <= 0
):
return "instrument_failure"
if "language_not_covered" in json.dumps([push.get("skipped", []), push.get("admission", {}).get("coverage_reasons", [])]) and not any(
tier.get("bytes_read", 0) for tier in push.get("quick_tier", [])
):
return "unanalysable"
if instrument.get("hook_bytes_read", 0) <= 0:
return "instrument_failure"
changed = set(instrument["changed_bytes"])
if any(
outcome.get("status") == "completed" and outcome.get("identity", {}).get("path") in changed
for scan in push.get("scans", [])
if scan.get("side") == "head"
for outcome in scan.get("scan", {}).get("question_outcomes", [])
):
return "engine_covered"
return "quick_only"
def findings(record):
push = record.get("push", {})
admission = push.get("admission", {})
for defect in admission.get("defects", []):
for path, line in defect.get("locations", []):
yield "alerts", {**defect, "path": path, "line": line}
seen = set()
for disposition in admission.get("dispositions", []):
delta = disposition.get("candidate", {}).get("delta", {})
operation = delta.get("head_operation")
if (
disposition.get("admitted")
or delta.get("novelty") not in {"new", "worsened"}
or not set(disposition.get("reasons", [])) <= CAPABILITY_VETOES
or operation is None
or delta.get("witness") is None
or delta.get("defect_id") in seen
):
continue
seen.add(delta.get("defect_id"))
yield "advisory", {**operation, "family": delta.get("family")}
taxonomy = load_families()
for tier in push.get("quick_tier", []):
for finding in tier.get("findings", []):
family = taxonomy.family_of_cwe(finding.get("cwe") or "")
yield "quick", {**finding, "family": family.id if family else None}
def location_match(record, finding):
for file in record.get("files", []):
if finding.get("path") != file["path"]:
continue
line = finding.get("line", 0)
if any(start - 10 <= line <= end + 10 for start, end in file.get("head_lines", [])):
return True
if record.get("family") != "config_secrets" and file["path"] == record.get("path", file["path"]):
function = record.get("function")
if function and function != "<global>" and finding.get("function") == function:
return True
span = record.get("function_lines") or file.get("function_lines")
if span and span[0] <= line <= span[1]:
return True
return False
def catches(record):
return sorted(
{
stream
for stream, finding in findings(record)
if finding.get("family") == record.get("family") and location_match(record, finding)
}
)
def false_alarm(record, *, alerts_only=False):
return any(not alerts_only or stream == "alerts" for stream, _ in findings(record))
def wilson(successes, n):
if n == 0:
return [None, None]
p, z = successes / n, 1.96
centre = (p + z * z / (2 * n)) / (1 + z * z / n)
half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
return [max(0, centre - half), min(1, centre + half)]
def required_n(p, width):
for n in range(1, 1000000):
low, high = wilson(p * n, n)
if max(p - low, high - p) <= width:
return n
raise ValueError("requested precision exceeds sizing range")
def rate(values):
n, k = len(values), sum(values)
return {"n": n, "count": k, "rate": k / n if n else None, "wilson95": wilson(k, n)}
def cluster_interval(rows, *, seed=0, draws=2000):
clusters = defaultdict(list)
for repo, value in rows:
clusters[repo].append(value)
if not clusters:
return [None, None]
rng = random.Random(seed)
groups = list(clusters.values())
samples = []
for _ in range(draws):
values = [value for group in rng.choices(groups, k=len(groups)) for value in group]
samples.append(sum(values) / len(values))
samples.sort()
return [samples[int(0.025 * (draws - 1))], samples[int(0.975 * (draws - 1))]]
def icc(rows):
clusters = defaultdict(list)
for repo, value in rows:
clusters[repo].append(value)
n, k = len(rows), len(clusters)
if k < 2 or n <= k:
return None
mean = sum(v for _, v in rows) / n
between = sum(len(v) * (sum(v) / len(v) - mean) ** 2 for v in clusters.values()) / (k - 1)
within = sum(sum((x - sum(v) / len(v)) ** 2 for x in v) for v in clusters.values()) / (n - k)
size = (n - sum(len(v) ** 2 for v in clusters.values()) / n) / (k - 1)
denominator = between + (size - 1) * within
return (between - within) / denominator if denominator else 0.0
def mcnemar(a, b):
pairs = list(zip(a, b, strict=True))
losses = sum(x and not y for x, y in pairs)
gains = sum(y and not x for x, y in pairs)
n = losses + gains
return min(1.0, 2 * sum(math.comb(n, i) for i in range(min(losses, gains) + 1)) / 2**n) if n else 1.0
def score(records, *, seed=0):
counts = Counter(coverage(r) for r in records)
if not records or counts["instrument_failure"] / len(records) > 0.05:
raise ValueError("slice exceeds 5% instrument_failure (or is empty)")
usable = [r for r in records if coverage(r) != "instrument_failure"]
def metrics(rows):
ordinary = [r for r in rows if r["kind"] == "ordinary"]
vulnerable = [r for r in rows if r["kind"] != "ordinary"]
clustered = [(r["repo"], int(false_alarm(r))) for r in ordinary]
return {
"catch": rate([bool(catches(r)) for r in vulnerable]),
"false_alarm": {
**rate([false_alarm(r) for r in ordinary]),
"cluster95": cluster_interval(clustered, seed=seed),
"icc": icc(clustered),
},
"alerts_only": rate([false_alarm(r, alerts_only=True) for r in ordinary]),
}
breakdown = {}
for key in ("family", "method", "post_cutoff"):
groups = defaultdict(list)
for r in usable:
groups[str(r.get(key, "unknown"))].append(r)
breakdown[key] = {key: metrics(rows) for key, rows in sorted(groups.items())}
breakdown["coverage"] = {state: metrics([r for r in usable if coverage(r) == state]) for state in STATES[1:]}
breakdown["stream"] = {
stream: rate([stream in catches(r) for r in usable if r["kind"] != "ordinary"]) for stream in ("alerts", "advisory", "quick")
}
return {
**metrics(usable),
"total": len(records),
"coverage": {state: counts[state] for state in STATES},
"breakdowns": breakdown,
"confirmed": metrics([r for r in usable if r.get("label_check", r.get("confirmed")) in ("confirmed", True)]),
}
def budgets(rows, *, thresholds=None, binary=False):
ordinary = [r for r in rows if r["kind"] == "ordinary"]
vulnerable = [r for r in rows if r["kind"] != "ordinary"]
if binary:
fa = rate([r["prediction"] for r in ordinary])
return {
"catch": rate([r["prediction"] for r in vulnerable]),
"false_alarm": fa,
"budgets_met": [f"{budget}%" for budget in (1, 5, 10) if fa["rate"] is not None and fa["rate"] <= budget / 100],
}
output = {}
for budget in (1, 5, 10):
key = f"{budget}%"
if thresholds is not None and key not in thresholds:
continue
if not ordinary:
output[key] = {"label": "not_run"}
continue
if thresholds is not None:
threshold = thresholds[key]
else:
candidates = sorted({r["score"] for r in rows})
candidates.append(math.nextafter(candidates[-1], math.inf))
threshold = next(t for t in candidates if sum(r["score"] >= t for r in ordinary) / len(ordinary) <= budget / 100)
output[key] = {
"threshold": threshold,
"label": "fixed" if thresholds is not None else "exploratory",
"catch": rate([r["score"] >= threshold for r in vulnerable]),
"false_alarm": rate([r["score"] >= threshold for r in ordinary]),
}
return output
def gate(a, b, *, slice_id=None, informed=False, binary=True, threshold=None, threshold_fixed=False, seed=0):
not_run = {"gate": "not_run"}
if not slice_id or informed or not a or not b or (not binary and (not threshold_fixed or threshold is None)):
return not_run
try:
score(a, seed=seed)
score(b, seed=seed)
except ValueError:
return not_run
aa, bb = {r["id"]: r for r in a}, {r["id"]: r for r in b}
if len(aa) != len(a) or len(bb) != len(b) or aa.keys() != bb.keys():
return not_run
pairs = [
(aa[key], bb[key]) for key in sorted(aa) if coverage(aa[key]) != "instrument_failure" and coverage(bb[key]) != "instrument_failure"
]
if any(x["kind"] != y["kind"] or x["repo"] != y["repo"] for x, y in pairs):
return not_run
vulnerable = [(x, y) for x, y in pairs if x["kind"] != "ordinary"]
ordinary = [(x, y) for x, y in pairs if x["kind"] == "ordinary"]
if not vulnerable or not ordinary:
return not_run
ca = [bool(catches(x)) for x, _ in vulnerable]
cb = [bool(catches(y)) if binary else y["score"] >= threshold for _, y in vulnerable]
fa = [false_alarm(x) for x, _ in ordinary]
fb = [false_alarm(y) if binary else y["score"] >= threshold for _, y in ordinary]
catch_p, fa_p = mcnemar(ca, cb), mcnemar(fa, fb)
regression = (sum(cb) < sum(ca) and catch_p < 0.05) or (binary and sum(fb) > sum(fa) and fa_p < 0.05)
return {
"gate": "regression" if regression else "pass",
"slice": slice_id,
"catch": {"difference": sum(cb) / len(cb) - sum(ca) / len(ca), "mcnemar_p": catch_p},
"false_alarm": {
"difference": sum(fb) / len(fb) - sum(fa) / len(fa),
"mcnemar_p": fa_p,
"cluster95": cluster_interval([(x["repo"], int(v) - int(u)) for (x, _), u, v in zip(ordinary, fa, fb, strict=True)], seed=seed),
},
}
def load_records(path):
if path.is_file():
return json.loads(path.read_text())
rows = []
for entry in json.loads((path / "inputs.json").read_text()):
result = path / (hashlib.sha256(entry["item_id"].encode()).hexdigest() + ".json")
record = json.loads(result.read_text()) if result.exists() else {}
if record.get("input_digest") != entry["input_digest"]:
record = {}
rows.append({**entry["change"], **record, "id": entry["change"]["id"]})
return rows
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("records", type=Path, help="JSON array of joined change/instrument/push records")
parser.add_argument("--out", type=Path, required=True)
parser.add_argument("--against", type=Path, help="production-arm records for a paired adoption gate")
parser.add_argument("--slice-id")
parser.add_argument("--informed", action="store_true")
args = parser.parse_args()
current = load_records(args.records)
result = score(current)
result["unseen"] = gate(load_records(args.against) if args.against else [], current, slice_id=args.slice_id, informed=args.informed)
args.out.write_text(json.dumps(result, sort_keys=True, indent=2) + "\n")
if __name__ == "__main__":
main()
benchmarks/unseen/source.py
+280
View File
@@ -0,0 +1,280 @@
"""Advisory selection. Network adapters are constructed only by the coordinator CLI."""
from __future__ import annotations
import base64
import json
import random
import re
from dataclasses import dataclass
from datetime import date
from urllib.parse import urlencode, urlsplit
from openultrasast.model.taxonomy import load_families
from .eligibility import RepositoryNotFound, normalize
ECOSYSTEMS = ("pip", "npm", "maven", "composer")
OSV_ECOSYSTEMS = dict(zip(ECOSYSTEMS, ("PyPI", "npm", "Maven", "Packagist"), strict=True))
FAMILIES = {"injection", "path", "deserialization", "access_control", "output_encoding", "untrusted_destination", "config_secrets"}
PERMISSIVE = {"MIT", "Apache-2.0", "BSD-2-Clause", "BSD-3-Clause", "ISC", "0BSD", "Unlicense", "Zlib", "PostgreSQL", "BSL-1.0"}
FIX = re.compile(r"https://github\.com/([\w.-]+/[\w.-]+)/commit/([a-fA-F0-9]{7,40})(?:[/?#].*)?$", re.I)
class Rejected(ValueError):
"""Expected eligibility rejection; messages are fixed reason codes, never names."""
@dataclass(frozen=True)
class Candidate:
repository: str
url: str
fix: str
parent: str
advisory: str
ecosystem: str
family: str
post_cutoff: bool
first_affected: str | None
license_spdx: str
license_class: str
license_ref: str
license_path: str
def post_cutoff(row: dict) -> bool:
return date.fromisoformat(row["published_at"][:10]) > date(2026, 6, 1)
def fix_links(row: dict) -> set[tuple[str, str]]:
links = set()
for reference in row.get("references", []):
url = reference if isinstance(reference, str) else reference.get("url", "")
match = FIX.fullmatch(url)
if match:
links.add((normalize(match[1]), match[2].lower()))
return links
def ecosystem(row: dict) -> str | None:
values = {v["package"]["ecosystem"].lower() for v in row.get("vulnerabilities", [])}
return next((e for e in ECOSYSTEMS if e in values), None)
def ordered_advisories(rows: list[dict], seed: int) -> list[dict]:
"""Stable seeded ordering, post-cutoff first in each ecosystem; round-robin strata."""
rng = random.Random(seed)
groups = []
for eco in ECOSYSTEMS:
group = sorted((r for r in rows if ecosystem(r) == eco), key=lambda r: r["ghsa_id"])
rng.shuffle(group)
group.sort(key=lambda r: not post_cutoff(r))
groups.append(group)
return [group[i] for i in range(max(map(len, groups), default=0)) for group in groups if i < len(group)]
def first_affected(record: dict, row: dict, eco: str) -> str | None:
packages = {v["package"]["name"] for v in row["vulnerabilities"] if v["package"]["ecosystem"].lower() == eco}
versions = set()
for affected in record.get("affected", []):
package = affected.get("package", {})
if package.get("ecosystem") != OSV_ECOSYSTEMS[eco] or package.get("name") not in packages:
continue
for span in affected.get("ranges", []):
if span.get("type") not in {"ECOSYSTEM", "SEMVER"}:
continue
for event in span.get("events", []):
if event.get("introduced") not in {None, "0"}:
versions.add(event["introduced"])
if len(versions) > 1:
raise Rejected("ambiguous_version_bracket")
return next(iter(versions), None)
def license_info(record: dict) -> tuple[str, str]:
# Explicit pinned-file SPDX wins. For GitHub-recognized full texts its license endpoint
# provides the detected SPDX. Unknown/custom/compound declarations stay private.
declared = re.search(r"SPDX-License-Identifier:\s*([^\r\n]+)", record.get("text", ""))
spdx = declared[1].strip().removesuffix("*/").strip() if declared else record.get("spdx_id") or "NONE"
return spdx, "permissive" if spdx in PERMISSIVE else "private"
def candidate(row: dict, github, osv) -> Candidate:
if date.fromisoformat(row["published_at"][:10]) < date(2019, 1, 1):
raise Rejected("date")
if row.get("type", "reviewed") != "reviewed" or row.get("withdrawn_at"):
raise Rejected("advisory_status")
eco = ecosystem(row)
if eco is None:
raise Rejected("ecosystem")
taxonomy = load_families()
families = {f.id for c in row.get("cwes", []) if (f := taxonomy.family_of_cwe(c["cwe_id"])) and f.id in FAMILIES}
if len(families) != 1:
raise Rejected("family")
links = fix_links(row)
if len(links) != 1:
raise Rejected("fix_links")
name, fix = next(iter(links))
repo = github.get_repo(name)
url = urlsplit(repo.get("clone_url", ""))
if (
repo.get("private", True)
or not 0 <= repo.get("size", -1) <= 500 * 1024
or url.scheme != "https"
or url.hostname != "github.com"
or url.username
or url.password
or url.query
or url.fragment
or normalize(repo["clone_url"]) != normalize(repo["full_name"])
):
raise Rejected("repository")
commit = github.get_commit(repo["full_name"], fix)
if len(commit["parents"]) != 1:
raise Rejected("parents")
parent = commit["parents"][0]["sha"]
license_record = github.get_license(repo["full_name"], parent)
spdx, license_class = license_info(license_record)
return Candidate(
normalize(repo["full_name"]),
repo["clone_url"],
commit["sha"],
parent,
row["ghsa_id"],
eco,
next(iter(families)),
post_cutoff(row),
first_affected(osv.get(row["ghsa_id"]), row, eco),
spdx,
license_class,
parent,
license_record.get("path", ""),
)
class JSONTransport:
"""Coordinator GET transport with bounded retries; all I/O and waits are injectable."""
def __init__(self, *, opener=None, sleeper=None, clock=None):
import time
from urllib.request import urlopen
self.opener = opener or urlopen
self.sleeper = sleeper or time.sleep
self.clock = clock or time.time
self.attempts = 0
self.rate_limit = {}
def get(self, url: str, headers: dict) -> object:
from urllib.error import HTTPError
from urllib.request import Request
for attempt in range(5):
self.attempts += 1
try:
with self.opener(Request(url, headers=headers), timeout=60) as response:
self.record_rate(getattr(response, "headers", {}))
return json.load(response)
except HTTPError as exc:
self.record_rate(exc.headers)
if exc.code == 404:
raise RepositoryNotFound("not_found") from None
limited = exc.code == 429 or (
exc.code == 403 and (exc.headers.get("X-RateLimit-Remaining") == "0" or exc.headers.get("Retry-After"))
)
transient = exc.code in {500, 502, 503, 504}
if attempt == 4 or not (limited or transient):
raise ValueError("http_failure") from None
try:
delay = float(exc.headers.get("Retry-After", "0"))
if limited and not delay:
delay = max(60, float(exc.headers.get("X-RateLimit-Reset", "0")) - self.clock() + 1)
delay = max(delay, 2**attempt)
if delay > 3605:
raise ValueError("retry_window")
except (TypeError, ValueError):
raise ValueError("http_retry_failed") from None
self.sleeper(delay)
except Exception:
raise ValueError("http_failure") from None
raise ValueError("http_failure")
def record_rate(self, headers):
self.rate_limit = {
key: headers.get(header)
for key, header in (
("remaining", "X-RateLimit-Remaining"),
("limit", "X-RateLimit-Limit"),
("reset", "X-RateLimit-Reset"),
("retry_after", "Retry-After"),
)
if headers.get(header) is not None
}
class GitHub:
def __init__(self, token: str, transport=None):
if not token:
raise ValueError("missing_gh_token")
self.transport = transport or JSONTransport()
self.headers = {"Authorization": "Bearer " + token, "Accept": "application/vnd.github+json"}
self.repos: dict[str, dict] = {}
self.calls = 0
def get(self, path: str):
self.calls += 1
return self.transport.get("https://api.github.com" + path, self.headers)
def get_repo(self, name: str) -> dict:
name = normalize(name)
if name not in self.repos:
self.repos[name] = self.get("/repos/" + name)
return self.repos[name]
def get_commit(self, name: str, sha: str) -> dict:
return self.get("/repos/" + normalize(name) + "/commits/" + sha)
def get_license(self, name: str, ref: str) -> dict:
try:
row = self.get("/repos/" + normalize(name) + "/license?" + urlencode({"ref": ref}))
except RepositoryNotFound:
return {"spdx_id": "NONE", "text": "", "path": ""}
return {
"spdx_id": (row.get("license") or {}).get("spdx_id"),
"path": row.get("path", ""),
"text": base64.b64decode(row.get("content", "")).decode("utf-8", errors="replace"),
}
def advisory_page(self, page: int) -> dict:
# Oldest first: newly published advisories do not shift already persisted pages.
rows = self.get(
"/advisories?" + urlencode({"type": "reviewed", "per_page": 100, "page": page, "sort": "published", "direction": "asc"})
)
if not isinstance(rows, list):
raise ValueError("invalid_advisory_page")
return {"rows": rows, "done": len(rows) < 100}
def advisories(self):
# Enumerate every reviewed advisory, including non-candidate fixes.
page = 1
while True:
result = self.advisory_page(page)
yield from result["rows"]
if result["done"]:
break
page += 1
class OSV:
def __init__(self, transport=None):
self.transport = transport or JSONTransport()
self.calls = 0
def get(self, identifier: str) -> dict:
if not re.fullmatch(r"GHSA-[\w-]+", identifier):
raise ValueError("invalid_advisory_id")
self.calls += 1
try:
return self.transport.get("https://api.osv.dev/v1/vulns/" + identifier, {})
except RepositoryNotFound:
return {} # No OSV bracket is distinct from a failed service call.
plane/Dockerfile.engine-task
+9 -1
View File
@@ -2,7 +2,7 @@
# preserve its runner as PID 1 and put the engine command in the Task manifest.
FROM eclipse-temurin:21-jre-jammy AS jre
FROM ghcr.io/norandom/ax-task-runner:v0.3.1
FROM ghcr.io/norandom/ax-task-runner:v0.3.1 AS engine
LABEL org.opencontainers.image.licenses="Apache-2.0" \
org.opencontainers.image.source="https://github.com/norandom/OpenUltraSAST"
@@ -50,3 +50,11 @@ RUN python3 -m venv /venv \
&& /venv/bin/python -I -c 'import pathlib, runpy, sys; sys.modules.update(yaml=None, boto3=None); sys.path.insert(0, "/app/benchmarks/learn"); p = pathlib.Path("/app/benchmarks/learn/engine_trace_worker.py"); print("Worker import smoke bytes:", len(p.read_bytes())); runpy.run_path(str(p), run_name="engine_worker_import_smoke")'
# Do not set ENTRYPOINT or CMD: both belong to the upstream AX runner.
FROM engine AS replay
RUN apt-get update \
&& apt-get install -y --no-install-recommends git \
&& rm -rf /var/lib/apt/lists/*
COPY benchmarks/unseen/replay_entry.py ./benchmarks/unseen/replay_entry.py
RUN /venv/bin/pip install --no-cache-dir . \
&& /venv/bin/ousast pre-push --help > /dev/null
src/openultrasast/plane/memory.py
+18 -6
View File
@@ -615,9 +615,12 @@ class S3Store(MemoryStore):
presigned URLs. An admin configures the bucket once; opening a store verifies it (:meth:`verify_bucket`) and
refuses a bucket without versioning, the ``runs/`` expiry rule or S3 Select. There is no local fallback."""
def __init__(self, client: ObjectClient, bucket: str, prefix: str = "") -> None:
read_only: bool = False
def __init__(self, client: ObjectClient, bucket: str, prefix: str = "", *, read_only: bool = False) -> None:
self.client, self.bucket, self.prefix = client, bucket, prefix.strip("/")
if self.describe() not in _VERIFIED:
self.read_only = read_only
if not read_only and self.describe() not in _VERIFIED:
self.verify_bucket()
_VERIFIED.add(self.describe())
@@ -631,9 +634,11 @@ class S3Store(MemoryStore):
return self.client.get(self._k(key))
def _put(self, key: str, data: bytes, labels: Mapping[str, str] | None = None) -> None:
self._writable()
self.client.put(self._k(key), data, dict(labels or {}))
def _delete(self, key: str) -> None:
self._writable()
self.client.delete(self._k(key))
def _keys(self, prefix: str) -> list[str]:
@@ -647,11 +652,16 @@ class S3Store(MemoryStore):
tagged = self.client.tags(self._k(key)).get("kind")
return tagged is None or tagged == "*" or kind in tagged.split()
def _writable(self) -> None:
if self.read_only:
raise MemoryStoreError("read-only store")
def verify_bucket(self) -> None:
"""Read-only check that the admin's one-time setup is in place: versioning ``Enabled``, an enabled
"""Check the admin's one-time setup, writing only the capability probe: versioning ``Enabled``, an enabled
lifecycle rule expiring ``<prefix>/runs/`` and none expiring the whole store, S3 Select answering a probe
over JSON Lines, and the probe's object tags readable. Every missing piece is named with its admin action
in one :class:`MemoryStoreError`."""
self._writable()
runs = self._k(RUNS_PREFIX)
endpoint = '--endpoint-url "$S3_ENDPOINT"'
missing: list[str] = []
@@ -739,6 +749,7 @@ class S3Store(MemoryStore):
) from exc
def presign_put(self, key: str, expires: timedelta = timedelta(hours=1)) -> str:
self._writable()
return self.client.presign("PUT", self._k(key), expires)
def presign_get(self, key: str, expires: timedelta = timedelta(hours=1)) -> str:
@@ -915,9 +926,10 @@ def s3_settings(environ: Mapping[str, str] | None = None) -> dict[str, Any]:
return settings
def open_store(spec: str | None = None, environ: Mapping[str, str] | None = None) -> MemoryStore:
def open_store(spec: str | None = None, environ: Mapping[str, str] | None = None, *, read_only: bool = False) -> MemoryStore:
"""The store ``spec`` (else ``OUSAST_MEMORY``, else ``file://<results_root()>/memory``) names. ``s3://`` with no
bucket uses ``S3_BUCKET``."""
bucket uses ``S3_BUCKET``. ``read_only`` skips the S3 write/Select capability probe and refuses S3 mutations;
it is intended for GET/list inventory readers, which do not need Select. Default opening is unchanged."""
from .reconciler import results_root
env = environ if environ is not None else os.environ
@@ -932,7 +944,7 @@ def open_store(spec: str | None = None, environ: Mapping[str, str] | None = None
if not bucket:
raise MemoryStoreError("OUSAST_MEMORY=s3:// names no bucket and S3_BUCKET is unset: set one of them")
settings = s3_settings(environ)
return S3Store(S3Client(bucket=bucket, **settings), bucket, parts.path)
return S3Store(S3Client(bucket=bucket, **settings), bucket, parts.path, read_only=read_only)
raise MemoryStoreError(f"OUSAST_MEMORY must be file:///<path> or s3://<bucket>[/<prefix>], got {text!r}")
tests/test_ax_batch.py
+159
View File
@@ -0,0 +1,159 @@
"""Offline shared dispatcher contracts."""
import json
import sys
from pathlib import Path
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
from benchmarks.ax.batch import Item, Workload, dispatch, manifest
IMAGE = "registry/task@sha256:" + "a" * 64
def item(identity="one"):
return Item(identity, {"SPEC_URL": b"{}"}, "result.dat", ("python3", "/entry.py"), {}, identity)
def workload():
return Workload(IMAGE, frozenset({"SPEC_URL", "RESULT_URL"}), 900, lambda data, output: {})
def test_manifest_contract():
doc = manifest(
"ousast-engine-replay-abc",
workload(),
item(),
{"SPEC_URL": "https://files.example/spec.dat", "RESULT_URL": "https://files.example/result.dat"},
)
assert {v["name"] for v in doc["spec"]["env"]} == {"SPEC_URL", "RESULT_URL"}
with pytest.raises(ValueError):
manifest("bad", workload(), item(), {})
bad = item()
bad.extra_env["AWS_SECRET_ACCESS_KEY"] = "secret"
with pytest.raises(ValueError):
manifest(
"ousast-engine-test",
workload(),
bad,
{"SPEC_URL": "https://files.example/spec.dat", "RESULT_URL": "https://files.example/result.dat"},
)
def test_dispatch_retry_fallback_resume_stop(tmp_path):
calls = []
def lane(it, output, attempt):
calls.append(("ax", attempt))
return {"status": "sandbox_failure"}
def vm(it, output, attempt):
calls.append(("docker", attempt))
return {"status": "ok"}
assert not dispatch([item()], workload(), tmp_path, {"ax": lane, "docker": vm}, lanes=("ax", "ax", "docker"), overflow=False)
assert [c[0] for c in calls] == ["ax", "ax", "docker"]
assert not dispatch([item()], workload(), tmp_path, {"ax": lane, "docker": vm})
assert len(calls) == 3
(tmp_path / "STOP").touch()
dispatch([item("two")], workload(), tmp_path, {"ax": lane, "docker": vm})
assert len(calls) == 3
assert json.loads((tmp_path / "progress.json").read_text())["pins_in_flight"] == 0
def test_cleanup_failure_halts(tmp_path):
calls = []
def broken(it, output, attempt):
calls.append(it.id)
raise RuntimeError("cleanup failed")
with pytest.raises(RuntimeError, match="cleanup"):
dispatch([item("one"), item("two")], workload(), tmp_path, {"ax": broken}, lanes=("ax",))
assert calls == ["one"]
def test_ax_lane_fake_cli_and_store(tmp_path, monkeypatch):
import subprocess
from types import SimpleNamespace
from benchmarks.ax.batch import AXLane
calls = []
class Store:
objects = {}
def _put(self, key, data):
self.objects[key] = data
def _get(self, key):
return (b"completed", None)
def _delete(self, key):
self.objects.pop(key, None)
def presign_get(self, key, expiry):
assert key.endswith(".dat")
return "https://files.example/" + key + "?signature=secret"
presign_put = presign_get
def run(command, **kwargs):
calls.append((command, kwargs))
if command[1] == "get":
name = json.loads(calls[0][1]["input"])["metadata"]["name"]
return subprocess.CompletedProcess(command, 0, json.dumps({"metadata": {"name": name}, "status": {"phase": "Running"}}), "")
return subprocess.CompletedProcess(command, 0, "", "")
monkeypatch.setattr(subprocess, "run", run)
monkeypatch.setenv("S3_ENDPOINT", "https://files.example")
def validate(data, output):
assert data == b"completed"
(output / "result.json").write_text('{"status":"ok"}')
work = Workload(IMAGE, frozenset({"SPEC_URL", "RESULT_URL"}), 10, validate)
lane = AXLane(SimpleNamespace(ax_bin="fake-ax", kubeconfig=None, atespace="default"), work, Store())
assert lane(item(), tmp_path, 0)["status"] == "ok"
doc = json.loads(calls[0][1]["input"])
assert doc["metadata"]["name"].startswith("ousast-engine-") and len(doc["metadata"]["name"]) <= 49
assert {e["name"] for e in doc["spec"]["env"]} == {"SPEC_URL", "RESULT_URL"}
assert calls[-1][0][1:3] == ["delete", "task"]
assert not lane.store.objects
def test_bounded_lanes_oom_and_instrument_retry(tmp_path):
import threading
import time
from collections import Counter
active, peak, calls = Counter(), Counter(), []
lock = threading.Lock()
def execute(lane):
def run(it, output, attempt):
with lock:
calls.append((it.id, lane, attempt))
active[lane] += 1
peak[lane] = max(peak[lane], active[lane])
time.sleep(0.01)
with lock:
active[lane] -= 1
return {"status": "oom" if lane == "ax" else "ok"}
return run
assert not dispatch([item(str(i)) for i in range(9)], workload(), tmp_path, {"ax": execute("ax"), "docker": execute("docker")})
assert peak["ax"] == 2 and peak["docker"] == 1
assert all(attempt == 0 for _, _, attempt in calls)
attempts = []
def failed(it, output, attempt):
attempts.append(attempt)
return {"status": "instrument_failure"}
assert dispatch([item("fail")], workload(), tmp_path, {"ax": failed}, lanes=("ax",))
assert attempts == [0, 1]
tests/test_independent_population.py
+123 -7
View File
@@ -8,6 +8,8 @@ anywhere else, which is how an evaluation set quietly becomes a development one.
from __future__ import annotations
import hashlib
import json
import re
import tomllib
from pathlib import Path
@@ -64,17 +66,131 @@ def test_a_multi_language_repository_is_reserved(path: Path) -> None:
assert members and all(by_id[m]["language"] == "php" and "javascript" in by_id[m].get("also", []) for m in members)
def test_no_reserved_repository_is_referenced_by_the_development_tree() -> None:
def _pool_manifests() -> list[Path]:
directory = ROOT / "benchmarks/unseen"
return sorted([*directory.glob("*.toml"), *directory.glob("private/*.toml")])
def _pool_names() -> set[str]:
names = set()
for path in _pool_manifests():
text = path.read_text().lower()
names.update(m.rstrip("/").removesuffix(".git") for m in re.findall(r"github\.com/([\w.-]+/[\w.-]+)", text))
return names
def _population_names() -> set[str]:
reserved = {case["repo"].rstrip("/").lower() for path in POPULATIONS for case in _population(path)["case"]}
names = {url.rsplit("/", 2)[-2] + "/" + url.rsplit("/", 1)[-1] for url in reserved}
offenders = []
for top in ("benchmarks", "src", "tests"):
return {url.rsplit("/", 2)[-2] + "/" + url.rsplit("/", 1)[-1] for url in reserved}
def test_no_reserved_repository_is_referenced_by_the_development_tree() -> None:
reservations = [(ROOT / "benchmarks/independent", _population_names()), (ROOT / "benchmarks/unseen", _pool_names())]
offenders = 0
counts = {}
for top in ("benchmarks", "src", "tests", "plane"):
for path in (ROOT / top).rglob("*"):
if not path.is_file() or POPULATION.parent in path.parents or path == Path(__file__) or path.suffix in {".pyc", ".bin", ".gz"}:
if not path.is_file() or path == Path(__file__) or path.suffix in {".pyc", ".bin", ".gz"}:
continue
# Preserve the population guard's existing scope; plane is newly reserved for pools.
active = [
(directory, names)
for directory, names in reservations
if directory not in path.parents and names and not (top == "plane" and directory == ROOT / "benchmarks/independent")
]
if not active:
continue
try:
text = path.read_text(errors="ignore").lower()
except OSError:
continue
offenders += [f"{path.relative_to(ROOT)}: {name}" for name in names if f"github.com/{name}" in text]
assert not offenders, "a reserved repository is referenced outside the reservation: " + "; ".join(offenders[:10])
count = sum(1 for _, names in active for name in names if f"github.com/{name}" in text)
offenders += count
counts[top] = counts.get(top, 0) + count
assert not offenders, f"reserved repository referenced outside its reservation: count={offenders}, sources={counts}"
def test_pool_and_population_names_are_disjoint() -> None:
count = len(_population_names() & _pool_names())
assert not count, f"overlapping reservations: count={count}"
def manifest_digest(path: Path) -> str:
"""Canonical parsed TOML, UTF-8 JSON with sorted keys and compact separators.
Private entries are covered by their opaque pointers' digests in the public manifest.
The freeze writer must use the same encoding (task 3.2).
"""
raw = json.dumps(tomllib.loads(path.read_text()), sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode()
return hashlib.sha256(raw).hexdigest()
def check_pool_digest(path: Path) -> None:
record = path.with_name("freeze-" + path.stem.removeprefix("pool-") + ".json")
expected = json.loads(record.read_text())["freeze_digest"]
assert manifest_digest(path) == expected, "frozen pool digest mismatch"
def test_frozen_pool_manifest_digest() -> None:
pools = sorted((ROOT / "benchmarks/unseen").glob("pool-p*.toml"))
if not pools:
pytest.skip("no pool frozen")
for path in pools:
check_pool_digest(path)
def test_pool_guard_controls(tmp_path, monkeypatch) -> None:
"""A synthetic reservation proves green, then red outside its own directory."""
population = tmp_path / "benchmarks/independent/population-v1.toml"
pool = tmp_path / "benchmarks/unseen/pool-p1.toml"
population.parent.mkdir(parents=True)
pool.parent.mkdir(parents=True)
prefix = "https://" + "github.com/"
population.write_text('[[case]]\nrepo = "' + prefix + 'synthetic/population"\n')
pool.write_text('[[repository]]\nrepo = "' + prefix + 'synthetic/pool"\n')
monkeypatch.setattr(__import__(__name__), "ROOT", tmp_path)
monkeypatch.setattr(__import__(__name__), "POPULATIONS", [population])
test_no_reserved_repository_is_referenced_by_the_development_tree()
leak = tmp_path / "benchmarks/measurements/fixture.txt"
leak.parent.mkdir(parents=True)
leak.write_text(prefix + "synthetic/pool")
with pytest.raises(AssertionError, match="count=1"):
test_no_reserved_repository_is_referenced_by_the_development_tree()
leak.unlink()
plane = tmp_path / "plane/fixture.txt"
plane.parent.mkdir()
plane.write_text(prefix + "synthetic/pool")
with pytest.raises(AssertionError, match="count=1"):
test_no_reserved_repository_is_referenced_by_the_development_tree()
plane.write_text(prefix + "synthetic/population")
test_no_reserved_repository_is_referenced_by_the_development_tree()
plane.unlink()
private = pool.parent / "private/draft-p1.toml"
private.parent.mkdir()
private.write_text('[[repository]]\nrepo = "' + prefix + 'synthetic/population"\n')
with pytest.raises(AssertionError, match="count=2"):
test_no_reserved_repository_is_referenced_by_the_development_tree()
def test_pool_digest_control(tmp_path) -> None:
pool = tmp_path / "pool-p1.toml"
pool.write_text('status = "frozen"\n[[repository]]\nid = "opaque"\n')
record = tmp_path / "freeze-p1.json"
record.write_text(json.dumps({"freeze_digest": manifest_digest(pool)}))
check_pool_digest(pool)
pool.write_text(pool.read_text() + "slice = 2\n")
with pytest.raises(AssertionError, match="digest"):
check_pool_digest(pool)
def test_frozen_pool_discovery_cannot_be_disabled(tmp_path, monkeypatch) -> None:
pool = tmp_path / "benchmarks/unseen/pool-p1.toml"
pool.parent.mkdir(parents=True)
pool.write_text('status = "frozen"\n')
record = pool.with_name("freeze-p1.json")
record.write_text(json.dumps({"freeze_digest": manifest_digest(pool)}))
monkeypatch.setattr(__import__(__name__), "ROOT", tmp_path)
test_frozen_pool_manifest_digest()
pool.write_text('status = "draft"\n')
with pytest.raises(AssertionError, match="digest"):
test_frozen_pool_manifest_digest()
tests/test_unseen_draft.py
+527
View File
@@ -0,0 +1,527 @@
"""Draft and coordinator tests contain synthetic identities only and use injected clients."""
import copy
import importlib
import sys
import tomllib
from collections import Counter
from contextlib import contextmanager
from pathlib import Path
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
d = importlib.import_module("benchmarks.unseen.draft")
s = importlib.import_module("benchmarks.unseen.source")
e = importlib.import_module("benchmarks.unseen.eligibility")
def entries():
rows = []
for i in range(300):
changes = [{"kind": "vulnerability", "method": "introducing", "base": "a" * 40, "head": "b" * 40}]
changes += [{"kind": "ordinary", "method": "first_parent", "base": f"{j:040x}", "head": f"{j + 20:040x}"} for j in range(12)]
rows.append(
{
"repository": f"fixture/project{i:03d}",
"url": f"https://github.com/fixture/project{i:03d}",
"ecosystem": s.ECOSYSTEMS[i % 4],
"post_cutoff": i % 7 == 0,
"license_spdx": "MIT" if i % 2 else "GPL-3.0",
"license_class": "permissive" if i % 2 else "private",
"license_ref": "b" * 40,
"changes": changes,
}
)
return rows
def test_draft_split_digest_and_balanced_slices(tmp_path):
rows = entries()
assigned = d.assign_slices(rows, seed=19)
assert Counter(r["slice"] for r in assigned) == {1: 100, 2: 100, 3: 100}
assert assigned == d.assign_slices(list(reversed(rows)), seed=19)
strata = Counter((r["ecosystem"], r["post_cutoff"]) for r in assigned)
for slice_id in (1, 2, 3):
counts = Counter((r["ecosystem"], r["post_cutoff"]) for r in assigned if r["slice"] == slice_id)
assert all(abs(counts[key] - total / 3) < 1 for key, total in strata.items())
public, private = d.write_draft(tmp_path, assigned, seed=19)
public_text = public.read_text()
shared = tomllib.loads(public_text)
hidden = tomllib.loads(private.read_text())
assert len(shared["repository"]) == 300 and len(hidden["repository"]) == 150
assert sum(len(r["changes"]) for r in shared["repository"] if "changes" in r) == 1950
for row in hidden["repository"]:
assert row["url"] not in public_text and row["repository"] not in public_text
pointer = next(p for p in shared["repository"] if p["id"] == row["id"])
assert set(pointer) == {"id", "sha256", "slice", "private"}
assert pointer["sha256"] == d.private_digest(row)
changed = copy.deepcopy(row)
changed["changes"][0]["head"] = "c" * 40
assert d.private_digest(changed) != pointer["sha256"]
assert private.is_relative_to(tmp_path / "benchmarks/unseen/private")
with pytest.raises(FileExistsError):
d.write_draft(tmp_path, assigned, seed=19)
def test_draft_requires_full_size_and_valid_changes():
with pytest.raises(ValueError, match="pool_size"):
d.assign_slices(entries()[:299], seed=1)
rows = entries()
rows[0]["changes"].pop()
with pytest.raises(ValueError, match="change_count"):
d.assign_slices(rows, seed=1)
def test_cache_is_outside_repository_deleted_and_partial(tmp_path):
import subprocess
root, cache = tmp_path / "repo", tmp_path / "cache"
root.mkdir()
with pytest.raises(ValueError, match="cache_inside_repository"):
d.Clones(root, root / "cache")
calls = []
def runner(argv, **kwargs):
calls.append(argv)
if "clone" in argv:
target = Path(argv[-1])
(target / ".git").mkdir()
(target / ".git/pack").write_bytes(b"123")
return subprocess.CompletedProcess(argv, 0, b"", b"")
clones = d.Clones(root, cache, runner=runner)
with clones.open("https://github.com/fixture/project") as git:
assert git.path.exists()
clone_path = git.path
assert not clone_path.exists()
assert "--filter=blob:none" in calls[0] and "--no-checkout" in calls[0]
assert clones.counts["clones"] == 1 and clones.counts["clone_bytes"] == 3
def test_clone_failure_sanitized_and_cleaned(tmp_path, capsys):
import subprocess
def runner(argv, **kwargs):
return subprocess.CompletedProcess(argv, 128, b"", b"private identity")
clones = d.Clones(tmp_path / "repo", tmp_path / "cache", runner=runner)
with pytest.raises(d.InstrumentFailure), clones.open("https://github.com/fixture/project"):
pass
assert list((tmp_path / "cache").iterdir()) == []
assert not capsys.readouterr().out
class FakeAPI:
calls = 0
def __init__(self, count=300):
self.count = count
self.licenses = []
def get_repo(self, name):
self.calls += 1
if name == "fixture/old":
name = "fixture/project000"
repo = {"full_name": name, "private": False, "size": 2, "clone_url": "https://github.com/" + name}
if name == "fixture/fork":
repo["source"] = {"full_name": "fixture/project001"}
return repo
def get_commit(self, name, sha):
self.calls += 1
return {"sha": sha, "parents": [{"sha": "b" * 40}]}
def get_license(self, name, ref):
self.licenses.append((name, ref))
self.calls += 1
# P is MIT but actual introducing head is GPL: public output must use V.
return {"text": "", "spdx_id": "MIT" if ref == "b" * 40 else "GPL-3.0", "path": "LICENSE"}
def advisory_page(self, page):
self.calls += 1
rows = list(self.advisories())[(page - 1) * 100 : page * 100]
return {"rows": rows, "done": len(rows) < 100}
def advisories(self):
for i in range(self.count):
yield {
"ghsa_id": f"GHSA-test-test-{i:04d}",
"published_at": "2026-06-02T00:00:00Z",
"cwes": [{"cwe_id": "CWE-89"}],
"vulnerabilities": [{"package": {"ecosystem": "pip", "name": "fixture"}}],
"references": [f"https://github.com/fixture/project{i:03d}/commit/" + "a" * 40],
}
class FakeOSV:
calls = 0
def get(self, identifier):
self.calls += 1
return {}
class FakeClones:
def __init__(self):
self.counts = Counter()
self.commands = []
@contextmanager
def open(self, url):
self.counts["clones"] += 1
yield self
def run(self, *args):
self.commands.append(args)
return ""
def test_coordinator_end_to_end_fakes_pinned_license_and_no_names(tmp_path, monkeypatch, capsys):
rows = entries()
changes = copy.deepcopy(rows[0]["changes"])
changes[0]["head"] = "c" * 40
monkeypatch.setattr(d, "repository_changes", lambda *a, **kw: (changes, {"security": 4, "eligible": 40}, {"bytes": 12, "files": 1}))
used, api, clones = e.UsedSet(), FakeAPI(), FakeClones()
report = d.run(tmp_path, github=api, osv=FakeOSV(), clones=clones, used=used, seed=7, known_empty=set(used.sources))
assert report["status"] == "draft" and report["changes"] == 3900
assert report["license_split"] == {"private": 300}
assert report["ordinary_exclusions"]["security"] == 1200
assert len([ref for _, ref in api.licenses if ref == "c" * 40]) == 300
public = (tmp_path.parent / (tmp_path.name + "-cache") / "journal/draft-p1.toml").read_text()
assert "fixture/" not in public
import json
assert "fixture/" not in json.dumps(report)
assert capsys.readouterr().out == ""
def test_used_rename_and_fork_resolution_and_shortfall(tmp_path, monkeypatch):
monkeypatch.setattr(d, "repository_changes", lambda *a, **kw: (entries()[0]["changes"], {}, {"bytes": 1, "files": 1}))
used, api, clones = e.UsedSet(), FakeAPI(2), FakeClones()
used.add("pairs", {"fixture/old"}, 10)
# project000 resolves from used alias; project001 is reached via used fork's source.
used.add("pairs", {"fixture/fork"}, 10)
report = d.run(tmp_path, github=api, osv=FakeOSV(), clones=clones, used=used, seed=7, known_empty=set(used.sources) - {"pairs"})
assert report["status"] == "insufficient_repositories"
assert report["rejections"] == {"used_pairs": 2}
assert report["repositories"] == 0 and not clones.counts
assert not (tmp_path / "benchmarks/unseen/draft-p1.toml").exists()
def test_zero_source_is_instrument_failure_not_rejection(tmp_path, monkeypatch):
def unread(*args, **kwargs):
raise d.InstrumentFailure("zero_source_bytes")
monkeypatch.setattr(d, "repository_changes", unread)
used = e.UsedSet()
report = d.run(tmp_path, github=FakeAPI(1), osv=FakeOSV(), clones=FakeClones(), used=used, seed=1, known_empty=set(used.sources))
assert report["status"] == "instrument_failure" and not report["rejections"]
def test_cli_uses_environment_and_injected_stores(tmp_path, monkeypatch, capsys):
from openultrasast import config
from openultrasast.plane import memory
seen = []
monkeypatch.setenv("GH_TOKEN", "fake-secret")
monkeypatch.setattr(config, "load_dotenv", lambda path: None)
monkeypatch.setattr(memory, "open_store", lambda value, **kw: seen.append((value, kw)) or object())
monkeypatch.setattr(d.eligibility, "build_used", lambda *a, **kw: e.UsedSet())
monkeypatch.setattr(d.source, "GitHub", lambda token: seen.append(token) or FakeAPI(0))
monkeypatch.setattr(d.source, "OSV", FakeOSV)
args = ["--root", str(tmp_path), "--cache", str(tmp_path.parent / "clone-cache")]
for label in e.SOURCES:
args += ["--known-empty", label]
assert d.main(args) == 1
output = capsys.readouterr().out
assert "instrument_failure" in output and "fake-secret" not in output and "fixture/" not in output
assert seen[-1] == "fake-secret"
assert all(kwargs == {"read_only": True} for _, kwargs in seen[:-1])
def test_repository_changes_excludes_all_advisory_sites(monkeypatch):
calls = []
class FakeGit:
def run(self, *args):
calls.append(args)
if args[0] == "ls-tree":
return "app.py\n"
return args[-1] if args[0] == "rev-parse" else ""
def blob(self, revision, path):
return "def handler():\n return 1\n"
def parents(self, fix):
return ["b" * 40]
candidate = s.Candidate(
"fixture/project",
"https://github.com/fixture/project",
"a" * 40,
"b" * 40,
"GHSA-test-test-test",
"pip",
"injection",
True,
None,
"MIT",
"permissive",
"b" * 40,
"LICENSE",
)
monkeypatch.setattr(d.extract, "introducing", lambda *a: {"head": "c" * 40})
monkeypatch.setattr(d.extract, "fix_sites", lambda git, fix, parent, **kw: [{"path": "app.py", "function": fix}])
monkeypatch.setattr(d.extract, "ordinary_history", lambda *a: [])
def draw(rows, fixes, known, **kwargs):
assert fixes == {"a" * 40, "d" * 40}
assert known == {("app.py", "a" * 40), ("app.py", "d" * 40)}
return [], {}
monkeypatch.setattr(d.extract, "draw_ordinary", draw)
_, _, proof = d.repository_changes(FakeGit(), candidate, {"d" * 40}, seed=1)
assert proof["bytes"] > 0
assert len([c for c in calls if c[0] == "fetch"]) == 2
def test_candidate_404_is_counted_but_service_failure_aborts(tmp_path):
class Missing(FakeAPI):
def get_commit(self, name, sha):
raise e.RepositoryNotFound("not_found")
class Broken(FakeAPI):
def get_commit(self, name, sha):
raise ValueError("private URL must never escape")
for api, status in [(Missing(1), "insufficient_repositories"), (Broken(1), "instrument_failure")]:
used = e.UsedSet()
report = d.run(
tmp_path,
github=api,
osv=FakeOSV(),
clones=FakeClones(),
used=used,
seed=1,
known_empty=set(used.sources),
journal_path=tmp_path.parent / (tmp_path.name + type(api).__name__) / "journal",
)
assert report["status"] == status
if status == "insufficient_repositories":
assert report["rejections"] == {"repository_or_commit_not_found": 1}
assert "private URL" not in str(report)
def test_shortfall_keeps_ordinary_exclusion_counts(tmp_path, monkeypatch):
def too_few(*args, **kwargs):
exc = s.Rejected("ordinary_floor")
exc.counts = {"eligible": 39, "size_cap": 10}
raise exc
monkeypatch.setattr(d, "repository_changes", too_few)
used = e.UsedSet()
report = d.run(tmp_path, github=FakeAPI(1), osv=FakeOSV(), clones=FakeClones(), used=used, seed=1, known_empty=set(used.sources))
assert report["ordinary_exclusions"] == {"eligible": 39, "size_cap": 10}
def test_clone_disk_cap_cleans_up(tmp_path):
import subprocess
def runner(argv, **kwargs):
(Path(argv[-1]) / "big").write_bytes(b"12345")
return subprocess.CompletedProcess(argv, 0, b"", b"")
clones = d.Clones(tmp_path / "repo", tmp_path / "cache", runner=runner, limit_bytes=4)
with pytest.raises(d.InstrumentFailure, match="clone_size_cap"), clones.open("https://github.com/fixture/project"):
pass
assert list((tmp_path / "cache").iterdir()) == []
@pytest.mark.parametrize("stage,n", [("advisories", 2), ("resolution", 103), ("eligibility", 17), ("extraction", 23), ("completion", 1)])
def test_checkpoint_interrupt_resume_matches_uninterrupted(tmp_path, monkeypatch, stage, n):
import json
journal_module = importlib.import_module("benchmarks.unseen.journal")
# Large fake populations exercise replay; durability is checked separately below.
monkeypatch.setattr(journal_module.os, "fsync", lambda fd: None)
root = tmp_path / "repo"
full, interrupted = tmp_path / "full/journal", tmp_path / "interrupted/journal"
changes = entries()[0]["changes"]
monkeypatch.setattr(d, "repository_changes", lambda *a, **kw: (changes, {"eligible": 40}, {"bytes": 12, "files": 1}))
used = e.UsedSet()
def run(path, api=None, clones=None):
return d.run(
root,
github=api or FakeAPI(),
osv=FakeOSV(),
clones=clones or FakeClones(),
used=used,
seed=7,
known_empty=set(used.sources),
journal_path=path,
)
assert run(full)["status"] == "draft"
original = d.Journal.put
def interrupt(self, current, key, value):
original(self, current, key, value)
if current == stage and len(self.rows[current]) == n:
raise KeyboardInterrupt
monkeypatch.setattr(d.Journal, "put", interrupt)
with pytest.raises(KeyboardInterrupt):
run(interrupted)
assert not (interrupted / "draft-p1.toml").exists()
before = {p.name: p.read_bytes() for p in interrupted.glob("*.jsonl")}
progress = json.loads((interrupted / "progress.json").read_text())
assert progress["counts"][stage] == n
assert progress["last_unit"]["stage"] == stage
assert progress["elapsed_seconds"] >= 0
assert "rate_limit" in progress
monkeypatch.setattr(d.Journal, "put", original)
clones = FakeClones()
assert run(interrupted, clones=clones)["status"] == "draft"
assert clones.counts["clones"] == 300 - progress["counts"]["extraction"]
for name in ("draft-p1.toml", "private-draft-p1.toml"):
assert (full / name).read_bytes() == (interrupted / name).read_bytes()
assert all((interrupted / name).read_bytes().startswith(data) for name, data in before.items())
assert not root.exists() # no names, manifests, or other records in the checkout
api, clones = FakeAPI(), FakeClones()
assert run(interrupted, api, clones)["status"] == "draft"
assert api.calls == 0 and clones.counts == {}
@pytest.mark.parametrize("parameter", ["seed", "used_set_digest", "clone_limit_bytes", "design_extract"])
def test_resume_refuses_changed_inputs_before_network(tmp_path, monkeypatch, parameter):
path, root = tmp_path / "cache/journal", tmp_path / "repo"
used = e.UsedSet()
kwargs = dict(
root=root,
github=FakeAPI(),
osv=FakeOSV(),
clones=FakeClones(),
used=used,
seed=7,
known_empty=set(used.sources),
journal_path=path,
max_hours=0,
)
assert d.run(**kwargs)["status"] == "stopped"
if parameter == "seed":
kwargs["seed"] = 8
elif parameter == "used_set_digest":
used.add("pairs", {"fixture/new"}, 12)
elif parameter == "clone_limit_bytes":
kwargs["clones"].limit_bytes = 12
else:
original = d.inputs
monkeypatch.setattr(d, "inputs", lambda *a: {**original(*a), "design_extract": "changed"})
report = d.run(**kwargs)
assert report["status"] == "resume_refused"
assert report["changed_parameters"] == [parameter]
assert kwargs["github"].calls == 0
def test_stop_between_units_and_progress_rate_state(tmp_path, monkeypatch):
import json
from types import SimpleNamespace
path, root = tmp_path / "cache/journal", tmp_path / "repo"
api = FakeAPI(1)
api.transport = SimpleNamespace(rate_limit={"remaining": "27", "reset": "1234"}, attempts=0)
used = e.UsedSet()
original = d.Journal.put
def stop(self, stage, key, value):
original(self, stage, key, value)
if stage == "eligibility":
(self.path / "STOP").touch()
monkeypatch.setattr(d.Journal, "put", stop)
clones = FakeClones()
report = d.run(root, github=api, osv=FakeOSV(), clones=clones, used=used, seed=1, known_empty=set(used.sources), journal_path=path)
assert report["status"] == "stopped" and report["reason"] == "STOP"
assert clones.counts == {}
progress = json.loads((path / "progress.json").read_text())
assert progress["status"] == "stopped" and progress["counts"]["eligibility"] == 1
assert progress["rate_limit"] == api.transport.rate_limit
assert not (path / "draft-p1.toml").exists()
def test_max_hours_stops_after_completed_page(tmp_path):
import json
now = [0.0]
class TimedAPI(FakeAPI):
def advisory_page(self, page):
result = super().advisory_page(page)
now[0] += 3601
return result
path = tmp_path / "cache/journal"
api, used = TimedAPI(), e.UsedSet()
report = d.run(
tmp_path / "repo",
github=api,
osv=FakeOSV(),
clones=FakeClones(),
used=used,
seed=1,
known_empty=set(used.sources),
journal_path=path,
max_hours=1,
clock=lambda: now[0],
)
assert report["status"] == "stopped" and report["reason"] == "max_hours"
assert api.calls == 1
progress = json.loads((path / "progress.json").read_text())
assert progress["counts"]["advisories"] == 1
assert progress["elapsed_seconds"] == 3601
def test_journal_cannot_be_inside_repository(tmp_path):
used = e.UsedSet()
api = FakeAPI()
report = d.run(
tmp_path,
github=api,
osv=FakeOSV(),
clones=FakeClones(),
used=used,
seed=1,
known_empty=set(used.sources),
journal_path=tmp_path / "journal",
)
assert report["status"] == "instrument_failure" and api.calls == 0
assert not (tmp_path / "journal").exists()
def test_journal_fsync_lock_and_corruption(tmp_path, monkeypatch):
import json
j = importlib.import_module("benchmarks.unseen.journal")
original = j.os.fsync
synced = []
def fsync(fd):
synced.append(fd)
original(fd)
monkeypatch.setattr(j.os, "fsync", fsync)
path = tmp_path / "journal"
with j.Journal(path).open({"seed": 1}) as journal:
before = len(synced)
journal.put("advisories", "1", {"rows": [], "done": True})
# Appended JSONL, directory, atomic progress file, directory.
assert len(synced) - before == 4
assert json.loads((path / "progress.json").read_text())["counts"]["advisories"] == 1
with pytest.raises(ValueError, match="journal_busy"), j.Journal(path).open({"seed": 1}):
pass
with (path / "advisories.jsonl").open("ab") as handle:
handle.write(b'{"id":')
with pytest.raises(ValueError, match="journal_incomplete_record"), j.Journal(path).open({"seed": 1}):
pass
tests/test_unseen_eligibility.py
+293
View File
@@ -0,0 +1,293 @@
"""Offline eligibility fixtures; no real repository identities are needed."""
import io
import json
from pathlib import Path
import pytest
from openultrasast.plane.memory import FileStore
with pytest.MonkeyPatch.context() as patch:
patch.syspath_prepend(str(Path(__file__).resolve().parents[1] / "benchmarks/unseen"))
import eligibility as e
def plant(root, path, text):
target = root / path
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text(text)
return target
def test_sources_and_reserved_input_never_opened(tmp_path, monkeypatch, capsys):
sources = {
"benchmarks/pairs/a/catalog.toml": ("pairs", "pair"),
"benchmarks/pairs/a/recipes.toml": ("pairs", "recipe"),
"benchmarks/pairs/a/dataset.json": ("pairs", "dataset"),
"benchmarks/pairs/advisory-fixes-b/catalog.toml": ("harvest", "harvest"),
"benchmarks/independent/population-v1.toml": ("populations", "first"),
"benchmarks/independent/population-v2.toml": ("populations", "second"),
"plane/experiments/a.units.jsonl": ("units", "unit"),
"benchmarks/measurements/a/record.json": ("measurements", "measurement"),
"benchmarks/experiments/a.json": ("measurements", "experiment"),
"src/example.txt": ("sweep", "sweep"),
}
for path, (_, name) in sources.items():
text = f'repo = "https://github.com/fixture/{name}"' if path.endswith(".toml") else json.dumps({"repo": f"fixture/{name}"})
if name == "sweep":
text = "https://github.com/fixture/sweep"
plant(tmp_path, path, text)
pointer = plant(tmp_path, "local/pointers.toml", '[[pair]]\nrepo = "fixture/pointer"\nfix_repo = "fixture/fix"\nvendored = false')
plant(tmp_path, "benchmarks/independent/population-v3.toml", "do not read")
plant(tmp_path, "benchmarks/independent/results-v3.json", "do not read")
original = io.open
def guarded(file, *args, **kwargs):
if "v3" in str(file):
pytest.fail("reserved input opened")
return original(file, *args, **kwargs)
monkeypatch.setattr(io, "open", guarded)
store = FileStore(tmp_path / "memory")
plant(tmp_path, "memory/repos/opaque/pin.jsonl", json.dumps({"id": "1", "kind": "example", "repo": "fixture/memory"}) + "\n")
used = e.build_used(tmp_path, stores={"local": store}, pointer_manifests=[pointer])
for label, name in sources.values():
assert f"fixture/{name}" in used.sources[label]
assert {"fixture/pointer", "fixture/fix"} <= used.sources["pointers"]
assert "fixture/memory" in used.sources["memory_local"]
report = used.report()
assert report["sources"]["memory_local"]["bytes"] > 0
assert all(item["files"] > 0 and item["bytes"] > 0 for item in report["sources"].values())
assert "fixture/" not in json.dumps(report)
assert capsys.readouterr().out == ""
assert e.normalize("HTTPS://GitHub.com/Fixture/Pair.GIT/") == "fixture/pair"
assert len({e.normalize(s) for s in ["fixture/pair", "Fixture/Pair.git/", "github.com/fixture/pair"]}) == 1
def test_empty_sources_fail_unless_declared(tmp_path):
used = e.build_used(tmp_path)
with pytest.raises(ValueError, match="empty_source"):
used.report()
assert used.report(known_empty=set(used.sources))["count"] == 0
class FakeAPI:
def __init__(self, records):
self.records = records
def get_repo(self, name):
return self.records[name]
def test_resolution_rejects_used_renames_and_forks():
used = e.UsedSet()
used.add("pairs", {"fixture/used"}, 10)
api = FakeAPI(
{
"fixture/used": {"full_name": "fixture/used"},
"fixture/old": {"full_name": "fixture/new", "former_names": ["fixture/used"]},
"fixture/fork": {"full_name": "fixture/fork", "parent": {"full_name": "fixture/used"}, "source": {"full_name": "fixture/used"}},
"fixture/a": {"full_name": "fixture/a", "source": {"full_name": "fixture/root"}},
"fixture/b": {"full_name": "fixture/b", "source": {"full_name": "fixture/root"}},
}
)
selected = set()
for name in ("used", "old", "fork"):
resolved = e.resolve("fixture/" + name, api)
assert e.rejection(resolved, used, selected) == "pairs"
assert e.rejection(e.resolve("fixture/a", api), used, selected) is None
assert e.rejection(e.resolve("fixture/b", api), used, selected) == "fork_network"
def test_registered_units_and_read_only_store(tmp_path):
units = plant(tmp_path, "local/units.jsonl", json.dumps({"repo": "fixture/registered"}))
store = FileStore(tmp_path / "memory")
plant(tmp_path, "memory/repos/opaque/pin.jsonl", json.dumps({"kind": "experiment", "units_file": str(units)}))
used = e.build_used(tmp_path, stores={"s3": store})
assert "fixture/registered" in used.sources["units"]
def test_cli_injected_store_and_credentials(tmp_path, monkeypatch, capsys):
from openultrasast import config
from openultrasast.plane import memory
calls = []
monkeypatch.setattr(config, "load_dotenv", lambda path: calls.append("dotenv"))
monkeypatch.setenv("GH_TOKEN", "test-token")
store = FileStore(tmp_path / "store")
monkeypatch.setattr(memory, "open_store", lambda spec, **kwargs: calls.append(spec) or store)
monkeypatch.setattr(e, "GitHub", lambda token: calls.append(token) or FakeAPI({}))
args = ["--root", str(tmp_path), "--s3-store", "s3://", "--local-store", "file:///unused"]
for label in (*e.SOURCES, "memory_local", "memory_s3"):
args += ["--known-empty", label]
assert e.main(args) == 0
assert calls == ["dotenv", "file:///unused", "s3://", "test-token"]
output = capsys.readouterr().out
assert json.loads(output)["count"] == 0
assert "test-token" not in output
def test_s3_open_is_read_only(tmp_path, monkeypatch):
from openultrasast.plane import memory
class Objects:
def keys(self, prefix):
return ["repos/opaque/pin.jsonl"]
def get(self, key):
return json.dumps({"kind": "example", "repo": "fixture/stored"}).encode(), "1"
def __getattr__(self, name):
raise AssertionError("unexpected_store_operation")
monkeypatch.setattr(memory, "S3Client", lambda **kwargs: Objects())
monkeypatch.setattr(memory, "s3_settings", lambda env: {})
store = memory.open_store("s3://fixture-bucket", read_only=True)
used = e.build_used(tmp_path, stores={"s3": store})
assert used.sources["memory_s3"] == {"fixture/stored"}
with pytest.raises(memory.MemoryStoreError, match="read.only"):
store._put("key", b"value")
with pytest.raises(memory.MemoryStoreError, match="read.only"):
store._delete("key")
def test_non_repository_provenance_and_missing_used_identity(tmp_path):
plant(tmp_path, "benchmarks/pairs/recipes.toml", 'repo = "provenance-label"')
assert not e.build_used(tmp_path).sources["pairs"]
used = e.UsedSet()
used.add("pairs", {"fixture/old", "fixture/nonexistent"}, 1)
class API:
def get_repo(self, name):
if name.endswith("nonexistent"):
raise e.RepositoryNotFound
return {"full_name": "fixture/new"}
e.canonicalize_used(used, API())
assert "fixture/nonexistent" in used.sources["pairs"]
assert e.rejection(e.resolve("fixture/new", API()), used, set()) == "pairs"
def test_binary_measurement_and_local_pointer_discovery(tmp_path):
target = tmp_path / "benchmarks/measurements/a/image.bin"
target.parent.mkdir(parents=True)
target.write_bytes(b"\xff\x00")
plant(tmp_path, "benchmarks/pairs/a/recipes.toml", '[[pair]]\nrepo = "fixture/pointer"\nvendored = false')
used = e.build_used(tmp_path)
assert used.instrument["measurements"]["bytes"] == 2
assert used.sources["pointers"] == {"fixture/pointer"}
def test_coordinator_verification_is_black_box(tmp_path, monkeypatch):
import subprocess
from types import SimpleNamespace
def run(command, **kwargs):
assert kwargs["stdout"] == kwargs["stderr"] == kwargs["stdin"] == subprocess.DEVNULL
path = Path(next(part.split("=", 1)[1] for part in command if part.startswith("--junitxml=")))
path.write_text('<testsuites><testsuite tests="25" failures="0" errors="0" skipped="1"/></testsuites>')
return SimpleNamespace(returncode=0)
monkeypatch.setattr(subprocess, "run", run)
assert e.verify_tests(tmp_path)["tests"]["tests"] == 25
monkeypatch.setattr(subprocess, "run", lambda *args, **kwargs: SimpleNamespace(returncode=2))
with pytest.raises(ValueError, match="guard_or_tests_failed"):
e.verify_tests(tmp_path)
def test_registered_unit_group_is_repository(tmp_path):
plant(tmp_path, "plane/experiments/a.units.jsonl", json.dumps({"unit": "opaque", "group": "fixture/unit", "label": 1}))
assert "fixture/unit" in e.build_used(tmp_path).sources["units"]
def test_multidocument_measurement_and_missing_registered_units(tmp_path):
plant(tmp_path, "benchmarks/measurements/a/run.yaml", "repo: fixture/first\n---\nrepo: fixture/second\n")
assert e.build_used(tmp_path).sources["measurements"] == {"fixture/first", "fixture/second"}
store = FileStore(tmp_path / "memory")
plant(
tmp_path,
"memory/repos/opaque/pin.jsonl",
json.dumps(
{
"kind": "experiment",
"units_file": "absent.units.jsonl",
"units_digest": "a" * 64,
}
),
)
with pytest.raises(FileNotFoundError):
e.build_used(tmp_path, stores={"local": store})
def test_catalog_skips_fixture_and_preserves_host_identity(tmp_path, capsys):
plant(
tmp_path,
"benchmarks/pairs/catalog.toml",
'[[pair]]\norigin = "in-tree-fixture"\nvendored = false\n[[pair]]\nrepo = "https://GitLab.com/Fixture/Private.git"\n',
)
used = e.build_used(tmp_path)
assert used.sources["pairs"] == {"gitlab.com/fixture/private"}
assert used.instrument["pairs"]["skipped_values"] == 1
assert used.instrument["pointers"]["skipped_values"] == 1
assert used.sources["sweep"] == set()
e.canonicalize_used(used, FakeAPI({})) # Non-GitHub identities never reach this API.
assert e.rejection(e.resolve("https://gitlab.com/fixture/private", FakeAPI({})), used, set()) == "pairs"
report = used.report(set(e.SOURCES) - {"pairs"})
assert report["sources"]["pairs"]["skipped_values"] == 1
assert "fixture" not in json.dumps(report)
assert capsys.readouterr() == ("", "")
@pytest.mark.parametrize(
("value", "expected"),
[
("https://bitbucket.org/Fixture/Repo.git", "bitbucket.org/fixture/repo"),
("https://git.example.org/Fixture/Repo", "git.example.org/fixture/repo"),
("https://github.com/Fixture/Repo/tree/main", "fixture/repo"),
("Fixture/Repo.git/", "fixture/repo"),
],
)
def test_repository_hosts(value, expected):
assert e.normalize(value) == expected
assert e.normalize(expected) == expected
def test_memory_values_are_skipped_and_counted(tmp_path, capsys):
rows = [
{"kind": "example", "repo": value}
for value in ["local-slice", "free text https://github.com/fixture/prose", None, "https://bitbucket.org/fixture/repo"]
]
plant(tmp_path, "memory/repos/opaque/pin.jsonl", "\n".join(json.dumps(row) for row in rows))
used = e.build_used(tmp_path, stores={"local": FileStore(tmp_path / "memory")})
assert used.sources["memory_local"] == {"bitbucket.org/fixture/repo"}
assert used.instrument["memory_local"]["skipped_values"] == 3
assert capsys.readouterr() == ("", "")
@pytest.mark.parametrize("step", ["open_stores", "build_used", "report", "canonicalize", "verify_tests"])
def test_cli_failure_reports_only_step_and_exception_type(tmp_path, monkeypatch, capsys, step):
from openultrasast import config
from openultrasast.plane import memory
def fail(*args, **kwargs):
raise RuntimeError("sensitive/identity")
monkeypatch.setattr(config, "load_dotenv", lambda path: None)
monkeypatch.setattr(memory, "open_store", lambda *args, **kwargs: object())
monkeypatch.setattr(e, "build_used", lambda *args, **kwargs: e.UsedSet())
monkeypatch.setattr(e.UsedSet, "report", lambda *args, **kwargs: {})
monkeypatch.setattr(e, "GitHub", lambda token: object())
monkeypatch.setattr(e, "canonicalize_used", lambda *args: None)
monkeypatch.setattr(e, "verify_tests", lambda *args: {})
if step == "open_stores":
monkeypatch.setattr(memory, "open_store", fail)
elif step == "report":
monkeypatch.setattr(e.UsedSet, "report", fail)
else:
monkeypatch.setattr(e, "canonicalize_used" if step == "canonicalize" else step, fail)
assert e.main(["--root", str(tmp_path), "--verify-tests"]) == 1
output = capsys.readouterr()
assert output.err == ""
assert json.loads(output.out) == {"status": "instrument_failure", "step": step, "type": "RuntimeError"}
assert "sensitive" not in output.out
tests/test_unseen_extract.py
+314
View File
@@ -0,0 +1,314 @@
"""Offline extraction: an injected local Git client over a synthetic temporary history."""
import importlib
import subprocess
import sys
from dataclasses import replace
from pathlib import Path
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
e = importlib.import_module("benchmarks.unseen.extract")
s = importlib.import_module("benchmarks.unseen.source")
@pytest.fixture
def history(tmp_path):
calls = []
def runner(argv, **kwargs):
assert not {"clone", "fetch", "push", "pull"} & set(argv)
calls.append(argv)
return subprocess.run(argv, **kwargs)
git = e.Git(tmp_path, runner=runner)
git.run("init", "-q")
git.run("config", "user.name", "Fixture")
git.run("config", "user.email", "[email protected]")
def commit(text, message="ordinary", extra=None):
(tmp_path / "app.py").write_text(text)
for path, content in (extra or {}).items():
(tmp_path / path).write_text(content)
git.run("add", ".")
git.run("commit", "-q", "-m", message)
return git.run("rev-parse", "HEAD").strip()
root = commit("def handler(value):\n return value\n")
git.run("tag", "v1.0")
intro = commit("def handler(value):\n return eval(value)\n")
git.run("tag", "v1.1")
parent = commit("def handler(value):\n return eval( value )\n", "format")
fix = commit("def handler(value):\n return int(value)\n", "security fix")
candidate = s.Candidate(
"fixture/project",
"https://github.com/fixture/project",
fix,
parent,
"GHSA-test-test-test",
"pip",
"injection",
True,
"1.1",
"MIT",
"permissive",
parent,
"LICENSE",
)
return git, commit, candidate, root, intro, parent, calls
def test_introducing_with_reformat_walkback(history):
git, _, candidate, root, intro, _, calls = history
change = e.introducing(git, candidate)
assert change["method"] == "introducing"
assert (change["base"], change["head"]) == (root, intro)
assert change["function"] == "handler"
assert change["family"] == "injection" and change["post_cutoff"]
assert change["files"][0]["head_lines"] == [[2, 2]]
assert change["bias"] and change["advisory"] == candidate.advisory
assert any("-w" in args and "-M" in args and "-C" in args for args in calls)
assert git.bytes_read > 0
def test_bracket_rejection_uses_fallback(history):
git, _, candidate, _, _, parent, _ = history
change = e.introducing(git, replace(candidate, first_affected="1.0"))
assert change["method"] == "last_touch" and change["head"] == parent
assert change["fallback_reasons"]["version_bracket"] >= 1
def test_size_cap_and_merge_rejection(history):
git, _, candidate, _, _, _, _ = history
class Oversize:
def __getattr__(self, name):
return getattr(git, name)
def stats(self, base, head):
return [("app.py", 2001, 0)]
assert e.introducing(Oversize(), candidate)["method"] == "last_touch"
class Merge(Oversize):
def parents(self, sha):
parents = git.parents(sha)
return parents + ["f" * 40] if sha != candidate.fix else parents
with pytest.raises(s.Rejected, match="no_range"):
e.introducing(Merge(), candidate)
def test_pure_addition_blames_function_declaration(history):
git, commit, candidate, _, _, _, _ = history
git.run("checkout", "-q", candidate.parent)
parent = commit("def handler(value):\n return eval(value)\n\ndef added(value):\n return value\n")
fix = commit("def handler(value):\n return eval(value)\n\ndef added(value):\n assert value\n return value\n")
change = e.introducing(git, replace(candidate, fix=fix, parent=parent, first_affected=None, family="access_control"))
assert change["head"] == parent and change["method"] == "introducing"
assert change["function"] == "added"
def test_git_failures_are_sanitized_and_not_empty(tmp_path):
def failed(argv, **kwargs):
return subprocess.CompletedProcess(argv, 128, b"", b"private repository secret")
with pytest.raises(e.InstrumentFailure, match="git_exit_128"):
e.Git(tmp_path, runner=failed).run("log")
def test_walkback_is_bounded_and_file_cap_falls_back(history):
git, _, candidate, _, _, parent, _ = history
class FileCap:
def __getattr__(self, name):
return getattr(git, name)
def stats(self, base, head):
return [("app.py", 1, 0)] * 51
result = e.introducing(FileCap(), candidate)
assert result["method"] == "last_touch"
assert result["fallback_reasons"]["size_cap"]
class Walk:
calls = 0
def __getattr__(self, name):
return getattr(git, name)
def blame(self, revision, path, line):
self.calls += 1
return parent, path, 2
def parents(self, revision):
return [parent] if revision != candidate.fix else [candidate.parent]
walk = Walk()
result = e.introducing(walk, candidate)
assert walk.calls == 6
assert result["fallback_reasons"]["walkback_limit"] == 1
def test_root_rejected_and_head_function_resolved(history):
git, _, candidate, root, _, _, _ = history
class Root:
def __getattr__(self, name):
return getattr(git, name)
def blame(self, revision, path, line):
return root, path, 1
result = e.introducing(Root(), candidate)
assert result["method"] == "last_touch"
assert result["fallback_reasons"]["merge_or_root"] == 1
assert result["function_lines"] == [1, 2]
@pytest.mark.parametrize(
"path,text",
[
("a.php", "<?php\nfunction handle($x) {\n return $x;\n}\n"),
("a.js", "function handle(x) {\n return x;\n}\n"),
("a.ts", "const handle = (x) => {\n return x;\n}\n"),
("A.java", "class A {\n public String handle(String x) {\n return x;\n }\n}\n"),
],
)
def test_supported_function_boundaries(path, text):
ranges = e.functions(path, text)
assert len(ranges) == 1 and ranges[0][0] == "handle"
assert ranges[0][2] < len(text.splitlines()) + 1
def test_ambiguous_parent_is_not_proof_of_absence(history):
git, commit, candidate, _, _, _, _ = history
duplicate = (
"class A:\n def handler(self, value):\n return eval(value)\n\n"
"class B:\n def handler(self, value):\n return value\n"
)
commit(duplicate)
parent = commit("class A:\n def handler(self, value):\n return eval( value )\n")
fix = commit("class A:\n def handler(self, value):\n return int(value)\n")
selected = replace(candidate, fix=fix, parent=parent, first_affected=None)
class BlameLast:
def __getattr__(self, name):
return getattr(git, name)
def blame(self, revision, path, line):
return parent, path, 3
result = e.introducing(BlameLast(), selected)
assert result["method"] == "last_touch"
assert result["fallback_reasons"]["ambiguous_parent"] == 1
def ordinary(index, **overrides):
row = {
"head": f"{index:040x}",
"parents": ["f" * 40],
"timestamp": index * 100,
"message": "update query token session valid",
"author": "Human",
"files": [{"path": "app.py", "added": 5, "removed": 0, "head_lines": [[2, 6]], "functions": ["ordinary"], "generated": False}],
}
row.update(overrides)
return row
@pytest.mark.parametrize(
"message",
[
"security fix",
"CVE-2026",
"GHSA-test",
"vulnerability",
"injection fix",
"XSS",
"CSRF",
"SSRF",
"exploit",
"sanitize",
"escaping",
"Revert changes",
"This reverts commit abc",
"bump package",
"upgrade package from 1 to 2",
],
)
def test_ordinary_message_exclusions(message):
assert e.ordinary_rejection(ordinary(1, message=message), set(), set()) is not None
def test_narrow_filter_and_every_other_exclusion():
assert e.ordinary_rejection(ordinary(1), set(), set()) is None
assert e.ordinary_rejection(ordinary(1, parents=["a", "b"]), set(), set()) == "merge_or_root"
assert e.ordinary_rejection(ordinary(1, author="dependabot[bot]"), set(), set()) == "dependency"
assert e.ordinary_rejection(ordinary(1, author="renovate"), set(), set()) == "dependency"
assert e.ordinary_rejection(ordinary(1), {f"{1:040x}"}, set()) == "advisory_fix"
assert e.ordinary_rejection(ordinary(1), set(), {("app.py", "ordinary")}) == "vulnerable_function"
row = ordinary(1)
row["files"][0]["added"] = 1001
assert e.ordinary_rejection(row, set(), set()) == "size_cap"
for path, reason in [
("README.md", "no_source"),
("package.json", "manifest_only"),
("package-lock.json", "manifest_only"),
("vendor/a.py", "generated"),
("node_modules/a.js", "generated"),
("dist/a.js", "generated"),
("app.min.js", "generated"),
]:
row = ordinary(1)
row["files"][0]["path"] = path
assert e.ordinary_rejection(row, set(), set()) == reason
row = ordinary(1)
row["files"][0]["generated"] = True
assert e.ordinary_rejection(row, set(), set()) == "generated"
def test_draw_seed_proportional_cells_floor_and_thirds():
rows = [ordinary(i) for i in range(120)]
for i, row in enumerate(rows):
row["files"][0]["added"] = [5, 20, 100, 500][i % 4]
drawn, counts = e.draw_ordinary(rows, set(), set(), seed=19)
assert len(drawn) == 12 and len({r["head"] for r in drawn}) == 12
assert {r["time_third"] for r in drawn} == {0, 1, 2}
assert len({(r["time_third"], r["size_bucket"]) for r in drawn}) == 12
assert drawn == e.draw_ordinary(list(reversed(rows)), set(), set(), seed=19)[0]
assert counts["eligible"] == 120
with pytest.raises(s.Rejected, match="ordinary_floor"):
e.draw_ordinary(rows[:39], set(), set(), seed=19)
# Thirds come from the history's time span, not rank among eligible changes.
with pytest.raises(s.Rejected, match="time_thirds"):
e.draw_ordinary([ordinary(i) for i in range(40)] + [ordinary(10000, message="security")], set(), set(), seed=1)
def test_git_ordinary_collection_generated_attributes_and_headers(history):
git, commit, _, _, _, _, _ = history
commit("# Generated by fixture\nx = 2\n", extra={".gitattributes": "other.py linguist-generated=true\n", "other.py": "x = 1\n"})
rows = e.ordinary_history(git, "HEAD")
assert rows and rows[0]["files"]
files = {f["path"]: f for f in rows[0]["files"]}
assert files["app.py"]["generated"] and files["other.py"]["generated"]
assert files["app.py"]["head_lines"]
def test_attribute_unset_and_mixed_generated_source(history):
git, commit, _, _, _, _, _ = history
sha = commit("x = 1\n", extra={".gitattributes": "*.py linguist-generated\napp.py -linguist-generated\n"})
assert not e.generated(git, sha, "app.py", "x = 1\n")
row = ordinary(1)
row["files"].append({**row["files"][0], "path": "dist/b.js", "generated": True})
assert e.ordinary_rejection(row, set(), set()) is None
def test_unbalanced_cells_remain_proportional_and_seed_changes_draw():
rows = [ordinary(i) for i in range(120)]
for i, row in enumerate(rows):
row["files"][0]["added"] = 5 if i % 4 else 100
draw, _ = e.draw_ordinary(rows, set(), set(), seed=1)
assert len([r for r in draw if r["size_bucket"] == 0]) == 9
assert draw != e.draw_ordinary(rows, set(), set(), seed=2)[0]
tests/test_unseen_image.py
+16
View File
@@ -0,0 +1,16 @@
"""Static image contract; no Docker invocation."""
import hashlib
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
ENGINE_SHA = "06ec67d411cd7abbfcd0735b1c2a70e4f3eb65ddcfeef89975709c664f59ad42"
def test_image_stages():
text = (ROOT / "plane/Dockerfile.engine-task").read_text()
engine, replay = text.split("FROM engine AS replay", 1)
original = engine.replace("v0.3.1 AS engine", "v0.3.1").rstrip() + "\n"
assert hashlib.sha256(original.encode()).hexdigest() == ENGINE_SHA
assert "FROM ghcr.io/norandom/ax-task-runner:v0.3.1 AS engine" in engine
assert "git" in replay and "replay_entry.py" in replay and "ousast pre-push --help" in replay
tests/test_unseen_ledger.py
+63
View File
@@ -0,0 +1,63 @@
"""Append-only use and tamper detection, with opaque fixture identifiers."""
import json
from pathlib import Path
import pytest
with pytest.MonkeyPatch.context() as patch:
patch.syspath_prepend(str(Path(__file__).resolve().parents[1] / "benchmarks/unseen"))
import ledger
def row(**extra):
return {
"slice": 1,
"decision": "decision-001",
"purpose": "baseline",
"date": "2026-10-04",
"result_digest": "a" * 64,
"freeze_digest": "b" * 64,
**extra,
}
def test_append_and_refuse_rewrite(tmp_path):
path = tmp_path / "usage-p1.jsonl"
ledger.append(path, row())
original = path.read_bytes()
ledger.append(path, row(purpose="exploratory"))
assert path.read_bytes().startswith(original)
assert len(ledger.read(path)) == 2
with pytest.raises(ValueError, match="duplicate"):
ledger.append(path, row())
def test_informed_slice_cannot_qualify_same_decision(tmp_path):
path = tmp_path / "usage-p1.jsonl"
ledger.append(path, row(purpose="informed"))
before = path.read_bytes()
with pytest.raises(ValueError, match="informed"):
ledger.append(path, row(purpose="qualifies"))
assert path.read_bytes() == before
ledger.append(path, row(purpose="qualifies", slice=2))
ledger.append(path, row(purpose="qualifies", decision="decision-002"))
def test_rewritten_row_breaks_digest_chain(tmp_path):
path = tmp_path / "usage-p1.jsonl"
ledger.append(path, row())
ledger.append(path, row(purpose="exploratory"))
rows = [json.loads(line) for line in path.read_text().splitlines()]
rows[0]["purpose"] = "informed"
path.write_text("".join(json.dumps(r) + "\n" for r in rows))
with pytest.raises(ValueError, match="chain"):
ledger.read(path)
with pytest.raises(ValueError, match="chain"):
ledger.append(path, row(slice=2))
@pytest.mark.parametrize("extra", [{"decision": "owner/name"}, {"purpose": "other"}, {"freeze_digest": "bad"}, {"slice": 0}])
def test_invalid_rows_refused(tmp_path, extra):
with pytest.raises(ValueError):
ledger.append(tmp_path / "usage-p1.jsonl", row(**extra))
tests/test_unseen_proof.py
+56
View File
@@ -0,0 +1,56 @@
"""Proof records fail closed using synthetic local results."""
import importlib
import json
import sys
from pathlib import Path
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
p = importlib.import_module("benchmarks.unseen.proof")
def test_replay_proof(tmp_path):
roots = {}
for lane in ("kind", "ax", "vm"):
root = tmp_path / lane
root.mkdir()
roots[lane] = root
(root / "inputs.json").write_text("[]")
for i in range(5):
(root / f"{i}.json").write_text(
json.dumps(
{
"input_digest": str(i),
"status": "engine_covered",
"instrument": {"hook_bytes_read": 20, "jvm_wall_seconds": 8, "seconds": 30, "hook_exit": 0},
"push": {"quick_tier": [{"findings": []}]},
}
)
)
assert p.replay_proof(roots)["quick_tier_equal"]
path = roots["vm"] / "0.json"
record = json.loads(path.read_text())
record["instrument"]["jvm_wall_seconds"] = 0
path.write_text(json.dumps(record))
with pytest.raises(ValueError, match="instrument"):
p.replay_proof(roots)
def test_engine_proof_counts_differences_without_names(tmp_path):
current, baseline = tmp_path / "current", tmp_path / "baseline"
current.mkdir()
baseline.mkdir()
for i in range(2):
record = {
"repo": "synthetic",
"pin": str(i),
"done": True,
"seconds": 20,
"units": [{"unit": str(i), "status": "path", "instrument": {"bytes": 20, "jvm": [{"seconds": 8, "success": True}]}}],
}
(current / f"{i}.json").write_text(json.dumps(record))
(baseline / f"{i}.json").write_text(json.dumps(record))
proof = p.engine_proof(current, baseline)
assert proof["differences"] == 0 and "synthetic" not in json.dumps(proof)
tests/test_unseen_replay.py
+79
View File
@@ -0,0 +1,79 @@
"""Replay workload privacy, validation and input identity."""
import importlib
import io
import json
import sys
import tarfile
from pathlib import Path
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
r = importlib.import_module("benchmarks.unseen.replay")
def test_items_private_and_arm_identity():
change = {
"id": "repo-name-secret",
"repo": "owner/private-name",
"repository_url": "https://public.example/private-name",
"base": "a" * 40,
"head": "b" * 40,
"files": [{"path": "app.py", "head_lines": [[1, 2]]}],
}
image = "registry/replay@sha256:" + "a" * 64
a = r.make_item(change, "default", image)
b = r.make_item(change, "long_deadline", image)
assert a.input_digest != b.input_digest
assert "private-name" not in a.id and "repo-name" not in a.id
assert set(a.inputs) == {"SPEC_URL"} and not a.extra_env
spec = json.loads(a.inputs["SPEC_URL"])
assert spec["repository_url"] == change["repository_url"]
assert json.loads(b.inputs["SPEC_URL"])["hook_flags"] == ["--deadline", "300"]
def test_validator_requires_instrument(tmp_path):
buffer = io.BytesIO()
with tarfile.open(fileobj=buffer, mode="w") as tar:
data = b"{}"
info = tarfile.TarInfo("push.json")
info.size = len(data)
tar.addfile(info, io.BytesIO(data))
with pytest.raises(ValueError, match="instrument"):
r.validate_result(buffer.getvalue(), tmp_path)
def test_frozen_toml_slice_and_scoring_join(tmp_path):
import hashlib
from benchmarks.unseen.score import load_records
pool = tmp_path / "pool-p1.toml"
pool.write_text("""pool="p1"
[[repository]]
id="opaque-a"
slice=1
url="https://public.example/repo-a"
changes=[{kind="ordinary",base="aaa",head="bbb",files=[]}]
[[repository]]
id="opaque-b"
slice=2
url="https://public.example/repo-b"
changes=[{kind="ordinary",base="ccc",head="ddd",files=[]}]
""")
import tomllib
freeze = hashlib.sha256(r.canonical(tomllib.loads(pool.read_text()))).hexdigest()
(tmp_path / "freeze-p1.json").write_text(json.dumps({"freeze_digest": freeze}))
changes = r.load_slice(pool, 2)
assert len(changes) == 1 and changes[0]["repo"] == "opaque-b"
(tmp_path / "inputs.json").write_text(json.dumps([{"item_id": "replay-opaque", "change": changes[0], "input_digest": "digest"}]))
assert load_records(tmp_path)[0]["kind"] == "ordinary"
assert "instrument" not in load_records(tmp_path)[0] # Missing result remains a measured failure.
result = tmp_path / (hashlib.sha256(b"replay-opaque").hexdigest() + ".json")
result.write_text(json.dumps({"input_digest": "digest", "instrument": {"head_verified": True}}))
assert load_records(tmp_path)[0]["instrument"]["head_verified"]
pool.write_text(pool.read_text() + "changed=true\n")
with pytest.raises(ValueError, match="digest"):
r.load_slice(pool, 2)
tests/test_unseen_replay_entry.py
+136
View File
@@ -0,0 +1,136 @@
"""Replay entry uses real local git and an injected transfer, never a network."""
import importlib
import io
import json
import subprocess
import sys
import tarfile
from pathlib import Path
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
e = importlib.import_module("benchmarks.unseen.replay_entry")
@pytest.fixture
def local_repo(tmp_path):
repo = tmp_path / "repo"
repo.mkdir()
def git(*args):
return subprocess.run(["git", "-C", str(repo), *args], capture_output=True, text=True, check=True).stdout.strip()
git("init", "-q")
git("config", "user.email", "[email protected]")
git("config", "user.name", "Fixture")
(repo / "app.py").write_text("print(1)\n")
git("add", "app.py")
git("commit", "-qm", "base")
base = git("rev-parse", "HEAD")
(repo / "app.py").write_text("print(2)\n")
git("commit", "-qam", "head")
return repo, base, git("rev-parse", "HEAD")
@pytest.mark.parametrize("failure", [None, "missing", "fetch", "crash", "fast", "unread", "unsupported"])
def test_entry(local_repo, tmp_path, monkeypatch, failure):
repo, base, head = local_repo
spec = {
"repository_url": "https://public.example/project.git",
"base": base,
"head": head,
"id": "opaque",
"image": "digest",
"hook_flags": [],
}
if failure == "missing":
spec["head"] = "a" * 40
received = {}
def transfer(request, destination=None, **kwargs):
assert kwargs["readiness_budget"] == 60
if destination:
destination.write_text(json.dumps(spec))
else:
with tarfile.open(fileobj=io.BytesIO(request.data)) as archive:
received.update({m.name: archive.extractfile(m).read() for m in archive.getmembers()})
monkeypatch.setenv("SPEC_URL", "https://files.example/spec.dat?secret")
monkeypatch.setenv("RESULT_URL", "https://files.example/result.dat?secret")
monkeypatch.setattr(e, "transfer", transfer)
real_run = subprocess.run
def run(command, **kwargs):
if command[0] == "ousast":
assert command[command.index("--base") + 1] == base
assert kwargs["env"]["OPENULTRASAST_CPG_HEAP_MB"] == "1700"
if failure == "crash":
return subprocess.CompletedProcess(command, 2, "", "crash https://secret")
target = Path(command[command.index("--artifact") + 1])
target.write_text(
json.dumps(
{
"quick_tier": [{"files": ["app.py"], "bytes_read": 0 if failure == "unread" else 9, "findings": []}],
"scans": [
{
"side": "head",
"scan": {"build_seconds": 0.1 if failure == "fast" else 8, "query_seconds": 2, "question_outcomes": []},
}
],
}
)
)
if failure == "unsupported":
target.write_text(
json.dumps(
{
"quick_tier": [{"files": [], "bytes_read": 0, "findings": []}],
"admission": {"coverage_reasons": ["language_not_covered:rust:1:1"]},
"scans": [
{
"side": "head",
"scan": {
"build_seconds": 0.01,
"query_seconds": 0,
"partitions": [
{
"language": "rust",
"frontend": None,
"paths": ["app.rs"],
"source_bytes": 0,
"status": "unsupported",
}
],
},
}
],
}
)
)
return subprocess.CompletedProcess(command, 0, "Skipped: deadline\n", "")
command = [
str(repo) if arg == spec["repository_url"] else "protocol.file.allow=always" if arg == "protocol.file.allow=never" else arg
for arg in command
]
if failure == "fetch" and "fetch" in command:
return subprocess.CompletedProcess(command, 1, "", "fetch secret")
return real_run(command, **kwargs)
result = e.main(work_root=tmp_path / "work", runner=run)
instrument = json.loads(received["instrument.json"])
assert instrument["image"] == "digest"
assert b"secret" not in received["instrument.json"]
if failure == "unsupported":
from benchmarks.unseen.score import coverage
assert result == 0 and not instrument["suspect_fast"]
assert coverage({"instrument": instrument, "push": json.loads(received["push.json"])}) == "unanalysable"
elif failure:
assert result == 1 and instrument["status"] == "instrument_failure"
else:
assert result == 0 and instrument["changed_bytes"] == {"app.py": 9}
assert instrument["head_verified"] and instrument["base_verified"]
assert instrument["jvm_wall_seconds"] == 10
assert b"Skipped:" in received["push.txt"]
tests/test_unseen_score.py
+127
View File
@@ -0,0 +1,127 @@
"""Synthetic push-level statistical controls; no corpus reads."""
import importlib
import sys
from pathlib import Path
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
s = importlib.import_module("benchmarks.unseen.score")
def record(kind="introducing", **updates):
return {
"id": "a",
"repo": "opaque",
"kind": kind,
"family": "injection",
"function": "handler",
"files": [{"path": "app.py", "head_lines": [[20, 22]]}],
"instrument": {
"head_verified": True,
"base_verified": True,
"changed_bytes": {"app.py": 20},
"hook_exit": 0,
"hook_bytes_read": 20,
},
"push": {"quick_tier": [{"bytes_read": 20, "findings": []}]},
**updates,
}
def test_coverage_and_failure_denominator():
r = record()
assert s.coverage(r) == "quick_only"
r["push"]["scans"] = [{"side": "head", "scan": {"question_outcomes": [{"status": "completed", "identity": {"path": "app.py"}}]}}]
assert s.coverage(r) == "engine_covered"
r["instrument"]["suspect_fast"] = True
assert s.coverage(r) == "instrument_failure"
assert s.coverage(record(push={"skipped": ["language_not_covered:other:1:1"]})) == "unanalysable"
assert s.coverage({}) == "instrument_failure"
with pytest.raises(ValueError, match="5%"):
s.score([record(), r])
@pytest.mark.parametrize(
"p,width,n", [(0.05, 0.03, 315), (0.1, 0.03, 483), (0.2, 0.03, 756), (0.1, 0.1, 62), (0.25, 0.1, 88), (0.4, 0.1, 97)]
)
def test_sample_sizes(p, width, n):
assert s.required_n(p, width) == n
def test_catch_and_all_stream_false_alarm():
r = record()
r["push"]["quick_tier"][0]["findings"] = [{"cwe": "CWE-89", "path": "app.py", "line": 20}]
assert s.catches(r) == ["quick"]
r["push"]["quick_tier"][0]["findings"][0]["line"] = 100
assert not s.catches(r)
r["kind"] = "ordinary"
assert s.false_alarm(r)
assert not s.false_alarm(r, alerts_only=True)
def test_location_rules():
assert s.location_match(record(family="access_control"), {"path": "app.py", "function": "handler", "line": 200})
assert not s.location_match(record(family="config_secrets"), {"path": "app.py", "function": "handler", "line": 200})
assert s.location_match(record(family="config_secrets"), {"path": "app.py", "line": 30})
assert not s.location_match(record(), {"path": "else.py", "function": "handler", "line": 20})
def test_bootstrap_icc_mcnemar():
rows = [("a", 1), ("a", 1), ("b", 0), ("b", 0)]
assert s.cluster_interval(rows, seed=7) == s.cluster_interval(rows, seed=7)
assert s.icc(rows) == pytest.approx(1)
assert s.mcnemar([True] * 6, [False] * 6) == pytest.approx(0.03125)
assert s.mcnemar([True], [True]) == 1
def test_budgets():
rows = [{"kind": "ordinary", "score": x} for x in range(100)] + [{"kind": "introducing", "score": 99}]
result = s.budgets(rows)
assert result["5%"]["threshold"] == 95
assert result["5%"]["label"] == "exploratory"
assert s.budgets(rows, thresholds={"5%": 96})["5%"]["label"] == "fixed"
binary = s.budgets([{"kind": "ordinary", "prediction": False}, {"kind": "introducing", "prediction": True}], binary=True)
assert binary["budgets_met"] == ["1%", "5%", "10%"]
def test_gate():
a = [record(kind="ordinary", id=str(i), repo=str(i)) for i in range(10)] + [record(id="v")]
b = [
record(kind="ordinary", id=str(i), repo=str(i), push={"quick_tier": [{"findings": [{"cwe": "CWE-89", "path": "x", "line": 1}]}]})
for i in range(10)
] + [record(id="v")]
assert s.gate(a, b, slice_id="p1/2")["gate"] == "regression"
assert s.gate(a, b)["gate"] == "not_run"
assert s.gate(a, b, slice_id="p1/2", informed=True)["gate"] == "not_run"
assert s.gate(a, [{}], slice_id="p1/2")["gate"] == "not_run"
def test_real_artifact_coverage_and_location_binding():
r = record(path="app.py", function_lines=[1, 100], files=[{"path": "other.py", "head_lines": [[200, 210]]}])
assert not s.location_match(r, {"path": "other.py", "line": 40})
assert (
s.coverage(
record(
instrument={"head_verified": True, "base_verified": True, "changed_bytes": {"x": 20}, "hook_exit": 0, "hook_bytes_read": 0},
push={"admission": {"coverage_reasons": ["language_not_covered:unknown:1:1"]}},
)
)
== "unanalysable"
)
r["push"]["quick_tier"][0]["findings"] = [{"cwe": None, "path": "other.py", "line": 200}]
assert not s.catches(r)
assert s.false_alarm(r)
def test_committed_synthetic_fixture():
import json
fixture = Path(__file__).resolve().parents[1] / "benchmarks/unseen/fixtures/scorer.json"
payload = fixture.read_bytes()
assert len(payload) > 0
report = s.score(json.loads(payload))
assert report["catch"]["n"] == 4 and report["catch"]["count"] == 2
assert report["false_alarm"]["n"] == 16 and report["false_alarm"]["count"] == 1
assert report["coverage"]["quick_only"] == 20
tests/test_unseen_source.py
+182
View File
@@ -0,0 +1,182 @@
"""Sanitized REST-shaped advisory fixtures; every external client is a fake."""
import copy
import importlib
import json
import sys
from pathlib import Path
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
s = importlib.import_module("benchmarks.unseen.source")
def advisory():
path = Path(__file__).resolve().parents[1] / "benchmarks/unseen/fixtures/advisory.json"
return json.loads(path.read_bytes())
class API:
def __init__(self):
self.parents = ["b" * 40]
self.spdx = "MIT"
self.license_text = "SPDX-License-Identifier: MIT"
self.calls = []
self.repo = {"full_name": "fixture/project", "private": False, "size": 100, "clone_url": "https://github.com/fixture/project.git"}
def get_repo(self, name):
return copy.deepcopy(self.repo)
def get_commit(self, name, sha):
return {"sha": sha, "parents": [{"sha": p} for p in self.parents]}
def get_license(self, name, ref):
self.calls.append((name, ref))
return {"text": self.license_text, "spdx_id": self.spdx, "path": "LICENSE"}
class OSV:
def get(self, identifier):
return {
"affected": [
{
"package": {"ecosystem": "PyPI", "name": "fixture"},
"ranges": [{"type": "ECOSYSTEM", "events": [{"introduced": "1.2"}, {"fixed": "1.3"}]}],
}
]
}
def test_candidate_cwe_cutoff_bracket_and_pinned_license():
api = API()
candidate = s.candidate(advisory(), api, OSV())
assert candidate.family == "injection" and candidate.post_cutoff
assert candidate.first_affected == "1.2"
assert candidate.license_class == "permissive"
assert api.calls == [("fixture/project", "b" * 40)]
raw = advisory()
raw["published_at"] = "2026-06-01T12:00:00Z"
assert not s.candidate(raw, api, OSV()).post_cutoff
raw["published_at"] = "2018-12-31T00:00:00Z"
with pytest.raises(s.Rejected, match="date"):
s.candidate(raw, api, OSV())
@pytest.mark.parametrize("spdx", ["GPL-3.0", "AGPL-3.0-only", "LGPL-2.1", "NOASSERTION", "", "BUSL-1.1", "MIT OR GPL-3.0"])
def test_private_licenses(spdx):
api = API()
api.spdx = spdx
api.license_text = ""
assert s.candidate(advisory(), api, OSV()).license_class == "private"
def test_license_file_overrides_api_and_missing_file_is_private():
api = API()
api.license_text = "SPDX-License-Identifier: GPL-3.0-only"
assert s.candidate(advisory(), api, OSV()).license_class == "private"
api.license_text = ""
api.spdx = "Apache-2.0"
assert s.candidate(advisory(), api, OSV()).license_spdx == "Apache-2.0"
@pytest.mark.parametrize(
"mutation,reason",
[
(lambda a, api: api.parents.append("c" * 40), "parents"),
(lambda a, api: a["references"].append("https://github.com/fixture/other/commit/" + "d" * 40), "fix_links"),
(lambda a, api: a.update(cwes=[{"cwe_id": "CWE-99999"}]), "family"),
(lambda a, api: api.repo.update(size=512001), "repository"),
(lambda a, api: api.repo.update(private=True), "repository"),
(lambda a, api: api.repo.update(clone_url="ssh://fixture/project"), "repository"),
],
)
def test_rejections(mutation, reason):
api, raw = API(), advisory()
mutation(raw, api)
with pytest.raises(s.Rejected, match=reason):
s.candidate(raw, api, OSV())
def test_post_cutoff_first_within_ecosystem_is_seeded():
rows = [advisory() for _ in range(12)]
for i, row in enumerate(rows):
row["ghsa_id"] += str(i)
if i % 2:
row["published_at"] = "2025-01-01T00:00:00Z"
ordered = s.ordered_advisories(rows, 17)
assert ordered == s.ordered_advisories(list(reversed(rows)), 17)
assert all(s.post_cutoff(row) for row in ordered[:6])
def test_http_adapters_with_fake_transport():
import base64
from urllib.parse import parse_qs, urlsplit
class Transport:
def __init__(self):
self.calls = []
def get(self, url, headers):
self.calls.append((url, headers))
parsed = urlsplit(url)
if parsed.path == "/advisories":
page = int(parse_qs(parsed.query)["page"][0])
return [advisory()] * (100 if page == 1 else 1)
if "/license" in parsed.path:
assert parse_qs(parsed.query)["ref"] == ["b" * 40]
return {
"license": {"spdx_id": "MIT"},
"path": "LICENSE",
"content": base64.b64encode(b"SPDX-License-Identifier: MIT").decode(),
}
if "/commits/" in parsed.path:
return {"sha": "a" * 40, "parents": [{"sha": "b" * 40}]}
if parsed.hostname == "api.osv.dev":
return OSV().get("ignored")
return API().repo
transport = Transport()
api = s.GitHub("fake-token", transport)
assert len(list(api.advisories())) == 101
assert s.candidate(advisory(), api, s.OSV(transport)).first_affected == "1.2"
assert all(headers["Authorization"] == "Bearer fake-token" for url, headers in transport.calls if "api.github.com" in url)
count = api.calls
api.get_repo("fixture/project")
assert api.calls == count
def test_http_failures_are_not_empty_results():
class Transport:
def get(self, url, headers):
raise ValueError("http_failure")
with pytest.raises(ValueError, match="http_failure"):
list(s.GitHub("fake", Transport()).advisories())
with pytest.raises(ValueError, match="http_failure"):
s.OSV(Transport()).get("GHSA-test-test-test")
def test_transport_rate_limit_retry_and_sanitized_failure():
import io
from urllib.error import HTTPError
waits, attempts = [], []
def opener(request, timeout):
attempts.append(request)
if len(attempts) == 1:
raise HTTPError(request.full_url, 403, "private message", {"X-RateLimit-Remaining": "0", "X-RateLimit-Reset": "120"}, None)
return io.BytesIO(b'{"ok": true}')
transport = s.JSONTransport(opener=opener, sleeper=waits.append, clock=lambda: 100)
assert transport.get("https://example.invalid", {}) == {"ok": True}
assert waits == [60] and transport.attempts == 2
def down(request, timeout):
raise HTTPError(request.full_url, 503, "private name", {}, None)
transport = s.JSONTransport(opener=down, sleeper=waits.append)
with pytest.raises(ValueError, match="^http_failure$"):
transport.get("https://example.invalid", {})
assert transport.attempts == 5