mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 19:59:26 +00:00
Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Marius CiepluchandClaude Opus 4.8
|
485aacca02 |
@@ -7,7 +7,6 @@ on:
|
||||
- 'src/**'
|
||||
- 'benchmarks/learn/engine_trace*.py'
|
||||
- 'benchmarks/learn/engine_task_entry.py'
|
||||
- 'benchmarks/unseen/replay_entry.py'
|
||||
- 'benchmarks/learn/engine_worker_service.py'
|
||||
- 'Dockerfile'
|
||||
- 'plane/Dockerfile.engine-task'
|
||||
@@ -31,10 +30,6 @@ jobs:
|
||||
dockerfile: Dockerfile
|
||||
- package: ousast-engine-task
|
||||
dockerfile: plane/Dockerfile.engine-task
|
||||
target: engine
|
||||
- package: ousast-replay-task
|
||||
dockerfile: plane/Dockerfile.engine-task
|
||||
target: replay
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
- uses: docker/setup-buildx-action@v3
|
||||
@@ -55,7 +50,6 @@ jobs:
|
||||
with:
|
||||
context: .
|
||||
file: ${{ matrix.dockerfile }}
|
||||
target: ${{ matrix.target }}
|
||||
platforms: linux/amd64
|
||||
push: true
|
||||
tags: |
|
||||
|
||||
@@ -36,6 +36,3 @@ workspace/
|
||||
|
||||
# MkDocs build output (mkdocs build)
|
||||
site/
|
||||
|
||||
# Unseen evaluation pointers for non-permissive repositories.
|
||||
benchmarks/unseen/private/
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"feature_name": "learned-decision-engine",
|
||||
"created_at": "2026-09-30T14:00:00Z",
|
||||
"updated_at": "2026-10-04T18:00:00Z",
|
||||
"updated_at": "2026-10-01T10:20:00Z",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"approvals": {
|
||||
|
||||
@@ -8,10 +8,6 @@ memory, compiled DSPy-style in-house); tasks 1-5 are unaffected, tasks 6-10 are
|
||||
against the provider balance read just before it, and none starts with a ceiling above that balance. No test makes
|
||||
a model or embedding call.
|
||||
|
||||
Direction 2026-10-04: `.kiro/steering/roadmap.md` (section of 2026-10-04, phases G0-G5) is the authority. Every
|
||||
change is judged on the unseen-repository pool (G0, spec `unseen-repo-evaluation`) by catch rate and false-alarm
|
||||
rate per push. Within-pair AUC stays a development signal. Tasks 11-14 below follow that roadmap.
|
||||
|
||||
- [x] 1. Schema and features
|
||||
- `learn/schema.py` (closed, typed allow-list; no identities; missing != zero), `learn/features.py` (host
|
||||
builder), new memory `KINDS`, `plane/tasks/features.py` and the `remember` input; tests enforcing the
|
||||
@@ -49,11 +45,6 @@ rate per push. Within-pair AUC stays a development signal. Tasks 11-14 below fol
|
||||
- _Requirements: 1.1_
|
||||
|
||||
- [ ] 6. Program, local memory and compilation (design sections 4-5)
|
||||
- Status 2026-10-04: 6.1-6.6 are done. 6.7 is partial: leave-one-source-out, the 0/25/50% memory-size points and
|
||||
the source-count curve are not run. Around this task the experiment machinery grew: rule arms
|
||||
(`precision_bound_block`, `pooled_block_gate` with a family floor) in `learn/experiments.py`, the response-cache
|
||||
fill with a hard ceiling and a dry-run estimate in `learn/fill.py`, and the in-function slice in `learn/slice.py`.
|
||||
These ran exp-003, exp-004 and exp-005 (7.5-7.7). Memory now holds 710 complete fix pairs (7.9).
|
||||
- [x] 6.1 Excerpts and the example store (no model)
|
||||
- `learn/excerpt.py` (function span at the label's pin, bounds 80/4,000 for candidates and 40/2,000 for
|
||||
examples, numbered lines, no path/repository/commit, advisory ids and security wording in comments
|
||||
@@ -160,28 +151,6 @@ rate per push. Within-pair AUC stays a development signal. Tasks 11-14 below fol
|
||||
rows' null plane-only features (1f525ff), the 788 embedding vectors absent from RustFS (copied from the file
|
||||
store), the result row's column-type collision under S3 Select (780caaa). Further program variants (`k = 5` vs
|
||||
`k = 1`, the confidence source) as new manifests.
|
||||
- [x] 7.5 exp-003 run and recorded 2026-10-03 ($0, replay only): **inconclusive**. Selective blocking (a per-family
|
||||
raw-score threshold chosen on training folds by its Wilson lower bound) offers no BLOCK on held-out paired units
|
||||
in any family; the incumbent stays. 211 of 353 frozen units evaluated (142 replay misses). Proving 0.95 needs at
|
||||
least 73 clean flags. Result: `benchmarks/experiments/exp-003-precision-bound-blocking/result.json`.
|
||||
- [x] 7.6 exp-004 run and recorded 2026-10-03: **inconclusive**. One pooled BLOCK threshold over all families with a
|
||||
family floor (Req 6.4). No outer fold found a threshold, pooled or per family; B offers no BLOCK. 501 units,
|
||||
replay at $0 after a cache fill of $1.24 (ceiling $2; the dry-run estimate over-priced 4.5x). Result and record:
|
||||
`benchmarks/experiments/exp-004-pooled-block-gate/result.json`, `record.json`.
|
||||
- [x] 7.7 exp-005 run and recorded 2026-10-03: **inconclusive**. The in-function def-use slice in the prompt of the
|
||||
six slice-2 programs. Within-pair AUC 0.762 -> 0.746 (CI [-0.063, 0.032], spans 0); unsure rate 0.29 -> 0.24
|
||||
(favours B); usd per candidate slightly higher. A $0 leak audit came first
|
||||
(`benchmarks/measurements/2026-10-03-graph-slice-leak-audit/record.json`). Result:
|
||||
`benchmarks/experiments/exp-005-graph-slice/result.json`.
|
||||
- [x] 7.8 Negatives harvest 2026-10-03 ($0.0014 embeddings, no chat-model call): 98 real fix pairs whose function
|
||||
survives the fix, from reviewed advisories, as pointer-pair corpus `advisory-fixes-2`; memory 937 -> 1133
|
||||
examples. Record: `benchmarks/measurements/2026-10-03-decision-engine-negatives-harvest/record.json`.
|
||||
- [x] 7.9 Harvest 3, 2026-10-04 ($0.003 embeddings, no chat-model call): 251 pairs as `advisory-fixes-3`; complete
|
||||
pairs in memory 459 -> 710. Record: `benchmarks/measurements/2026-10-04-decision-engine-harvest-3/record.json`.
|
||||
- [x] 7.10 Joern trace run over exp-005's 501 units and its leak audit, 2026-10-04 ($0): no leak flagged. In the 105
|
||||
pairs where both sides were asked, a path appears on both sides in 12, on the vulnerable side only in 2, on the
|
||||
fixed side only in 0, and on neither in 91. The engine cannot see what a fix changed. Record:
|
||||
`benchmarks/measurements/2026-10-04-joern-slice-leak-audit/record.json`.
|
||||
- _Requirements: 3.6, 4.1-4.4_
|
||||
|
||||
- [ ] 8. Integration
|
||||
@@ -207,50 +176,3 @@ rate per push. Within-pair AUC stays a development signal. Tasks 11-14 below fol
|
||||
`result-v3-engine.json` is committed. Nothing between the two commits may touch the program, memory snapshot,
|
||||
features, priors or thresholds. This spec's code never opens v3 files.
|
||||
- _Requirements: 5.3_
|
||||
|
||||
- [ ] 11. Fix mechanisms (plan approved 2026-10-04; see `fix-mechanisms-plan.md`)
|
||||
- Roadmap G2 arm (`.kiro/steering/roadmap.md`, section of 2026-10-04): one candidate improvement, not the spine.
|
||||
It is judged on the unseen-repository pool (G0, spec `unseen-repo-evaluation`) by catch rate and false-alarm
|
||||
rate per push. **Paused after 11.1 until the G0 pool exists.**
|
||||
- [ ] 11.1 Mine candidate discharge facts from the 710 fix diffs (sanitizer calls, guarding conditions, safe sink
|
||||
arguments, safe replacements); deterministic, $0. In progress.
|
||||
- [ ] 11.2 Admission by held-out separation per outer fold: a candidate turns "path on both sides" into "path on the
|
||||
vulnerable side only" on folds it was not mined from, removes no held-out vulnerable path, has at least 2
|
||||
proposing repositories, and is admitted per language.
|
||||
- [ ] 11.3 Engine re-run on the affected pins (kube-ax lanes plus the VM lane).
|
||||
- [ ] 11.4 Registered experiment, with the detection gates green, before `ruleset/semantic/*.toml` changes.
|
||||
- [ ] 11.5 Coverage increment: source and sink additions for pairs where neither side has a path (later).
|
||||
- [ ] 11.6 TypeScript admission in the engine (branch `engine-typescript`).
|
||||
- _Requirements: 4.1-4.4, 7.1, 8.1_
|
||||
|
||||
- [ ] 12. exp-006: Joern witness slice in the prompt **(budget go-ahead, about $2-4)**
|
||||
- Planned, roadmap G1/G2. Against the current prompt, after a $0 leak audit, on re-frozen units. Judged on the
|
||||
unseen-repository pool, not on within-pair AUC alone.
|
||||
- _Requirements: 4.1-4.4, 7.1_
|
||||
|
||||
- [ ] 13. exp-007: the combiner ($0 replay)
|
||||
- Planned, roadmap G1/G2. Three or four weights (score, engine entailment, neighbour agreement) over cached
|
||||
responses. Judged on the unseen-repository pool.
|
||||
- _Requirements: 4.1-4.4, 7.1_
|
||||
|
||||
- [ ] 14. Re-run the pooled block gate on re-frozen units
|
||||
- Planned, roadmap G3. The exp-004 design with the family floor (Req 6.4), together with the pool's false-alarm
|
||||
rate per family.
|
||||
- _Requirements: 6.4, 7.3_
|
||||
|
||||
- [ ] 15. Trained classifier arm (maintainer decision 2026-10-04: train a model, not hand-set profiles; judged on the
|
||||
unseen-repo pool, roadmap G1)
|
||||
- [ ] 15.1 Features: code embedding of the excerpt (stored vectors) + every allow-listed signal; no hand-picked
|
||||
profile. Automated screening: a signal that separates pair sides by itself on training folds above the leak
|
||||
threshold is dropped for that fold (the leak audit as a filter, fitted per fold).
|
||||
- [ ] 15.2 Models: regularised logistic regression and gradient-boosted trees, fitted per repository-held-out outer
|
||||
fold with inner-fold hyper-parameter search; per-family and pooled variants.
|
||||
- [ ] 15.3 Scores on held-out folds: within-pair AUC, pooled AUC, top-k precision, Wilson bound at the pooled
|
||||
threshold; compared with the current LLM programs on the same units.
|
||||
- [ ] 15.4 Scored on the unseen-repo pool when G0 exists: catch rate at fixed false-alarm budgets.
|
||||
- _Requirements: 3 (revised 2026-10-04 by maintainer decision), 6, 7_
|
||||
|
||||
- [ ] 16. Jointly optimised LLM program arm (judged on the unseen-repo pool)
|
||||
- Search input fields, retrieval parameters, k, instruction and demonstrations together (DSPy MIPRO-style) on
|
||||
training folds; no hand-set input profile. Budget go-ahead before paid search.
|
||||
- _Requirements: 3, 6, 7_
|
||||
|
||||
@@ -1,20 +1,5 @@
|
||||
# Design Document
|
||||
|
||||
> **Status 2026-10-04.** Groups 1-2 are done on kind (records `benchmarks/measurements/2026-10-02-k8s-01-profile/`
|
||||
> and `2026-10-02-k8s-02-store-delivery/`). Task 2.6 is paused. The production target changed. The operator runs a
|
||||
> kube-ax cluster (k3s, Agent Substrate v0.3.0, AX v0.3.1, 2 nodes). There we may only submit AX Tasks: no k8s
|
||||
> object creation, no secrets, no access to `ate-system`. The engine now runs there as AX Tasks through
|
||||
> `benchmarks/learn/engine_trace_ax.py`. The engine-task image (`plane/Dockerfile.engine-task`) is built FROM
|
||||
> `ghcr.io/norandom/ax-task-runner:v0.3.1`. Tasks read and write through presigned HTTPS links, are named
|
||||
> `ousast-engine-*`, and run a 1.7 GB heap with `ActiveProcessorCount=2` and one resident Joern session per pin. A
|
||||
> pin whose sandbox dies is retried once under a new name, then handed to the VM lane (2.5 GB heap). Egress is
|
||||
> operator-managed: HTTPS 443 to public hosts is open; LAN and cluster-internal addresses are blocked. Archive
|
||||
> names (`.tar`, `.gz`, `.zip`, `.bin`) failed through the store host's Cloudflare cache until the operator added a
|
||||
> cache-bypass rule; presigned objects are staged as `.dat` as a safeguard. Groups 3-8 are **to re-plan** against
|
||||
> that cluster under roadmap G4 (`.kiro/steering/roadmap.md`, section of 2026-10-04). Their text below is kept
|
||||
> as written. G4 (detection on push for any repository: `ousast plane scan --base --head`, engine pins on kube-ax,
|
||||
> model calls on the VM or CI) is the new target for group 7, sized by what G1-G3 keep.
|
||||
|
||||
## Overview
|
||||
|
||||
The plane keeps its shape: `ousast plane run` is the reconciler, ax is the only executor, every task is one
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"feature_name": "plane-on-kubernetes",
|
||||
"created_at": "2026-10-02T20:00:00Z",
|
||||
"updated_at": "2026-10-04T18:00:00Z",
|
||||
"updated_at": "2026-10-02T22:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"approvals": {
|
||||
|
||||
@@ -1,20 +1,5 @@
|
||||
# Implementation Plan
|
||||
|
||||
> **Status 2026-10-04.** Groups 1-2 are done on kind (records `benchmarks/measurements/2026-10-02-k8s-01-profile/`
|
||||
> and `2026-10-02-k8s-02-store-delivery/`). Task 2.6 is paused. The production target changed. The operator runs a
|
||||
> kube-ax cluster (k3s, Agent Substrate v0.3.0, AX v0.3.1, 2 nodes). There we may only submit AX Tasks: no k8s
|
||||
> object creation, no secrets, no access to `ate-system`. The engine now runs there as AX Tasks through
|
||||
> `benchmarks/learn/engine_trace_ax.py`. The engine-task image (`plane/Dockerfile.engine-task`) is built FROM
|
||||
> `ghcr.io/norandom/ax-task-runner:v0.3.1`. Tasks read and write through presigned HTTPS links, are named
|
||||
> `ousast-engine-*`, and run a 1.7 GB heap with `ActiveProcessorCount=2` and one resident Joern session per pin. A
|
||||
> pin whose sandbox dies is retried once under a new name, then handed to the VM lane (2.5 GB heap). Egress is
|
||||
> operator-managed: HTTPS 443 to public hosts is open; LAN and cluster-internal addresses are blocked. Archive
|
||||
> names (`.tar`, `.gz`, `.zip`, `.bin`) failed through the store host's Cloudflare cache until the operator added a
|
||||
> cache-bypass rule; presigned objects are staged as `.dat` as a safeguard. Groups 3-8 are **to re-plan** against
|
||||
> that cluster under roadmap G4 (`.kiro/steering/roadmap.md`, section of 2026-10-04). Their text below is kept
|
||||
> as written. G4 (detection on push for any repository: `ousast plane scan --base --head`, engine pins on kube-ax,
|
||||
> model calls on the VM or CI) is the new target for group 7, sized by what G1-G3 keep.
|
||||
|
||||
One task group per increment of `design.md` section 7, in that order; a later group never starts before the
|
||||
earlier group's kind proof is recorded. Every task ships with its tests; a task is done when its tests pass in
|
||||
the host suite, ruff and mypy are clean, and nothing in `src/`, `plane/`, `ops/` (minus `ops/ax/up.sh` and
|
||||
@@ -24,7 +9,7 @@ its kind proof and the record it writes under `benchmarks/measurements/<date>-k8
|
||||
(commit, images, the instrument checks, the numbers). Steps that call a model are marked **paid: needs a budget
|
||||
go-ahead** with a ceiling; steps that need a hand on a cluster that does not exist yet are marked **manual**.
|
||||
|
||||
- [x] 1. Profile and literals (design section 1; open-question checks 2, 3, 7)
|
||||
- [ ] 1. Profile and literals (design section 1; open-question checks 2, 3, 7)
|
||||
- [x] 1.1 `PlaneProfile`
|
||||
- `src/openultrasast/plane/profile.py`: frozen dataclass with the fields of design section 1; `load_profile()`
|
||||
reads `ops/k8s/profiles/<OUSAST_PLANE_PROFILE>.toml` (a name or a path; default `kind`), then applies
|
||||
@@ -122,7 +107,7 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
|
||||
prints the table. Instrument: the tar's byte size from the store equals the extracted files' total.
|
||||
- `benchmarks/measurements/<date>-k8s-02-store-delivery/record.json`: bytes, PUT-to-poll seconds, rules.
|
||||
- _Requirements: 3.1, 3.2, 7.1_
|
||||
- [ ] 2.6 Remove the receiver (last; after 2.5 is recorded) -- paused 2026-10-04
|
||||
- [ ] 2.6 Remove the receiver (last; after 2.5 is recorded)
|
||||
- Delete `Receiver`, `_Handler`, `receiver_address`, `receiver_cluster_ip`, `RECEIVER_HOST`, `RECEIVER_PORT`
|
||||
from `egress.py`; `OUSAST_ARTIFACT_URL/DIAL/HOST/PORT` from `runner.py` and `reconciler.py`; the
|
||||
`OUSAST_DELIVERY` switch; `ops/ax/receiver-service.yaml.tmpl` and the `up.sh` receiver step; the EndpointSlice
|
||||
@@ -132,7 +117,7 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
|
||||
`receiver_removed: <commit>`.
|
||||
- _Requirements: 3.1, 1.3_
|
||||
|
||||
- [ ] 3. `--exec remote` from a CI-shaped principal (design section 2; open-question check 1) -- to re-plan (kube-ax, roadmap G4)
|
||||
- [ ] 3. `--exec remote` from a CI-shaped principal (design section 2; open-question check 1)
|
||||
- [ ] 3.1 Execution mode and RBAC manifests
|
||||
- `cli.py`: `ousast plane run --exec local|remote` (default `profile.exec`); `reconciler.Ax` passes the
|
||||
profile's context to `ax` (its context flag, read from `ax --help` and pinned in a test); `remote` refuses a
|
||||
@@ -165,7 +150,7 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
|
||||
status, usd spent.
|
||||
- _Requirements: 2.1, 2.2, 2.3, 2.4, 7.1_
|
||||
|
||||
- [ ] 4. Images on GHCR (design section 8) -- to re-plan (kube-ax, roadmap G4)
|
||||
- [ ] 4. Images on GHCR (design section 8)
|
||||
- [ ] 4.1 `images.yml`
|
||||
- `.github/workflows/images.yml` on tags `v*`: `docker/build-push-action` for `plane/Dockerfile.runner` ->
|
||||
`ghcr.io/norandom/ousast-runner:<tag>` and the root `Dockerfile` -> `ghcr.io/norandom/ousast-engine:<tag>`;
|
||||
@@ -193,7 +178,7 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
|
||||
- `benchmarks/measurements/<date>-k8s-04-ghcr-images/record.json`: image sizes, pull seconds, visibility.
|
||||
- _Requirements: 1.1, 4.2, 7.1_
|
||||
|
||||
- [ ] 5. The engine as a plane task (design section 4) -- to re-plan (kube-ax, roadmap G4)
|
||||
- [ ] 5. The engine as a plane task (design section 4)
|
||||
- [ ] 5.1 `cpg/dump.py` and `tasks/engine.py`
|
||||
- Lift `benchmarks/push/finding_dump.py`'s scan into `src/openultrasast/cpg/dump.py` (the script becomes a
|
||||
thin caller; `ops/README.md:78` command unchanged). `src/openultrasast/plane/tasks/engine.py`: env
|
||||
@@ -223,7 +208,7 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
|
||||
placement, image size and pull seconds.
|
||||
- _Requirements: 4.1, 4.2, 7.1_
|
||||
|
||||
- [ ] 6. Pools, autoscaling, node checklist (design section 5) -- to re-plan (kube-ax, roadmap G4)
|
||||
- [ ] 6. Pools, autoscaling, node checklist (design section 5)
|
||||
- [ ] 6.1 `plane/k8s.py` and the manifests
|
||||
- `src/openultrasast/plane/k8s.py`: `render(profile, out)` writes `workerpool-default.yaml`,
|
||||
`workerpool-engine.yaml`, `ax-server-snapshots.yaml` (patch with `profile.snapshots_bucket`), `rbac-ci.yaml`;
|
||||
@@ -247,7 +232,7 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
|
||||
- `benchmarks/measurements/<date>-k8s-06-pools/record.json`: footprint table, securityContext, dry-run result.
|
||||
- _Requirements: 5.1, 5.2, 7.1_
|
||||
|
||||
- [ ] 7. `ousast plane scan` (design section 6) -- to re-plan (kube-ax, roadmap G4)
|
||||
- [ ] 7. `ousast plane scan` (design section 6)
|
||||
- [ ] 7.1 Workspaces and candidates from URL + commits
|
||||
- `generate.scan_cases(url, base, head, cache, *, all=False)`: shallow clone into the case cache, two pinned
|
||||
Workspaces, head-side hunks via `fix_ranges(side="new")`, functions via `repo_facts.enclosing` over
|
||||
@@ -280,7 +265,7 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
|
||||
seconds from submit to verdict, whether the known CVE site is among the agreed.
|
||||
- _Requirements: 6.1, 6.2, 6.3, 7.1_
|
||||
|
||||
- [ ] 8. Runbook (design section 7, Req 7) -- to re-plan (kube-ax, roadmap G4)
|
||||
- [ ] 8. Runbook (design section 7, Req 7)
|
||||
- [ ] 8.1 `docs/deployment.md` as the runbook
|
||||
- Sections: prerequisites for k3s (Substrate, ax with the snapshots patch, GHCR and the pull secret, buckets,
|
||||
the CI SA and its grant from 3.2, the node checklist of 6.2), the profile switch, `--exec remote` from a laptop
|
||||
|
||||
@@ -623,7 +623,6 @@ Preserve all completed task evidence below.
|
||||
- [x] 17. Operational readiness (2026-10-02)
|
||||
- **Named by the 2026-10-02 operational-readiness audit**, maintainer-approved as the first increment. The maintainer's definition: a user installs the hook on ANY repository and gets honest, advisory-only results; nothing ever blocks a push. The audit found the plumbing sound (snapshots, resolution, deadline accounting, advisory exit 0, the JSON artifact) and five things in the way: the hook can block a push, quick rules are not part of pre-push, the engine cannot finish a 26-file repository inside 30 s, the terminal hides every skip reason, and there is no supported install path without the source checkout. Before numbers for every item are the audit's live runs on records (Python), serve-static (JavaScript) and unrolled/secure (Go), each with one planted commit.
|
||||
- This group is plumbing and honesty. It enables no capability, edits no rule for a planted sample, and leaves graph reuse (the engine speed work decided in 11.6) as a later task.
|
||||
- Record: `benchmarks/measurements/2026-10-02-pre-push-operational-1/record.json` (see 17.7).
|
||||
- _Done 2026-10-02:_ all seven subtasks done. The engine still does not fit the 30 s hook deadline (graph reuse, the 11.6 direction, stays a later task), and plane verdicts after the push were not in this increment.
|
||||
- _Requirements: 9.1–9.6_
|
||||
- [x] 17.1 Never block a push the user did not choose to block
|
||||
|
||||
@@ -1,32 +0,0 @@
|
||||
# Brief: does the safety net work on repositories it has never seen?
|
||||
|
||||
Maintainer, 2026-10-04: "we want to spread the safety net across repos in general, unseen code. that was the purpose
|
||||
of the AB tests to see if our approach is generalizeable." And: "we should run our tests / runs in kube. now we can
|
||||
speed it up and design the workloads for ax directly. kind is only a fallback."
|
||||
|
||||
## Why this spec exists
|
||||
|
||||
The A/B experiments so far (exp-003, exp-004, exp-005) score the decision engine on repository-held-out folds of known
|
||||
fix pairs. That measures whether the vulnerable side of a pair scores above its fixed side. It does not measure what a
|
||||
safety net on an arbitrary repository lives on:
|
||||
|
||||
- on an ordinary push, does it stay quiet? (false-alarm rate)
|
||||
- on a push that introduces a real vulnerability, does it say so? (catch rate)
|
||||
|
||||
Populations v1 and v2 measured the engine scan on fresh cases once (1 of 11, 0 of 17) and are spent. Nothing since has
|
||||
measured push-level behaviour on unseen repositories.
|
||||
|
||||
## What this spec delivers
|
||||
|
||||
1. A frozen pool of repositories never used anywhere in development, with two kinds of change each: real
|
||||
vulnerability-introducing commits and ordinary commits.
|
||||
2. A push-level metric: each change replayed through the hook exactly as a user would run it, and through other
|
||||
arms (decision engine, engine evidence) where an experiment asks.
|
||||
3. Execution designed for the kube-ax cluster from the start: one push replay is one AX Task; kind and the VM are
|
||||
fallbacks.
|
||||
4. A baseline for today's hook, then a table of arms scored on the same pool.
|
||||
|
||||
## Not in scope
|
||||
|
||||
Changing detection. This spec only builds the measurement. Improvements are separate, registered experiments judged
|
||||
by it.
|
||||
@@ -1,431 +0,0 @@
|
||||
# Design Document
|
||||
|
||||
## Overview
|
||||
|
||||
This spec builds the instrument for roadmap G0: a frozen pool of repositories that no development step has seen,
|
||||
and a push-level score on it. The score is what a user of the hook sees. On a vulnerability-introducing push, did
|
||||
the hook say so (catch rate)? On an ordinary push, did it say anything (false-alarm rate)? Replays run as AX Tasks
|
||||
on kube-ax. kind and the VM run the same tasks as fallbacks.
|
||||
|
||||
Four facts from the code shape the design:
|
||||
|
||||
- **Populations already have a freeze-and-guard pattern.** `benchmarks/independent/freeze.py` verifies cases by
|
||||
reading git only, `evaluate.py` scores with fixed rules and Wilson intervals, and
|
||||
`tests/test_independent_population.py` fails when a reserved repository is named anywhere in `benchmarks/`,
|
||||
`src/` or `tests/` outside its directory. Harvest batch 3 used that guard as a black box: candidates written,
|
||||
guard run with output discarded, subsets bisected on red. That is how v3 stays unread.
|
||||
- **The hook can already replay a change.** `ousast pre-push PATH --base B --head H --artifact F` resolves two
|
||||
local commits (`snapshot.py`, `resolve_replay`: no fetch, no merge-base), runs quick rules and the engine inside
|
||||
one deadline (default 30 s, engine inline) and writes a JSON artifact. The terminal text has one `Skipped:` line
|
||||
per skipped check. The artifact holds three finding streams: admitted defects, advisory engine findings
|
||||
(new or worsened, vetoed only by capability) and quick-rule matches with their CWE.
|
||||
- **The AX pattern works but lives in one script.** `engine_trace_ax.py` (manifest, egress checks, presigned
|
||||
objects, retry once under a new name, cleanup) and the lane scheduler in `engine_trace.py` (`run_plan`: fixed
|
||||
lanes, fallback queue, `progress.json`, `STOP`, resume by input digest) are engine-specific.
|
||||
- **Placement on kube-ax is random.** One worker per node, shared by all actors. Two heavy tasks fit per worker,
|
||||
but only two concurrent tasks in all guarantee at most two on one worker.
|
||||
|
||||
### Goals
|
||||
|
||||
- A pool frozen by code against every used source (Req 1), with two kinds of change per repository (Req 2).
|
||||
- A scorer whose numbers mean what the hook shows, with silent zeros visible (Req 3).
|
||||
- One replay is one AX Task via a dispatcher shared with the engine (Req 4); a baseline and fresh slices (Req 5).
|
||||
|
||||
### Non-Goals
|
||||
|
||||
Detection changes, population v3, model calls in the cluster, languages the hook does not analyse.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
M[pool manifest slice] --> D[batch dispatcher on VM]
|
||||
D --> P[presign spec GET and record PUT]
|
||||
P --> T[AX Task ousast-engine-replay]
|
||||
T --> G[shallow fetch base and head from GitHub]
|
||||
G --> H[ousast pre-push --base --head]
|
||||
H --> R[record tar: artifact, text, instrument]
|
||||
R --> S[(store via presigned PUT)]
|
||||
D -->|sandbox died twice or OOM| V[VM docker lane]
|
||||
D -->|cluster down| K[kind lane, same tasks]
|
||||
V --> S
|
||||
K --> S
|
||||
S --> C[scorer: catch, false alarm, coverage]
|
||||
S --> A[model arms on VM, budgeted]
|
||||
A --> C
|
||||
C --> O[record: counts and digests]
|
||||
```
|
||||
|
||||
## Components and Interfaces
|
||||
|
||||
### 1. Pool sourcing and eligibility (Req 1)
|
||||
|
||||
**Decision 1.1: sources and ecosystems.** Candidates come from GitHub reviewed advisories (`gh api /advisories`,
|
||||
the enumeration harvest-3 used) for pip, npm, maven and composer, published 2019 or later, whose CWE maps to one of
|
||||
the seven families through the project's family CWE table. Each needs a linked fix commit with one parent in one
|
||||
repository. OSV records for the same advisory give the first affected version. That version is a cross-check for
|
||||
the introducing commit (section 2). Reason: these four ecosystems are the five languages the hook analyses
|
||||
(Python, JavaScript, TypeScript, PHP, Java). GHSA links commits and CWEs; OSV adds version brackets. Rejected:
|
||||
drawing from every ecosystem (Go, Ruby, C#). That would mostly measure missing frontends, which is already known.
|
||||
Coverage still reports uncovered files inside the drawn repositories. Rejected: NVD and published fix datasets.
|
||||
NVD links commits weakly, and the datasets are mined corpora that development or model training has likely seen.
|
||||
|
||||
**Decision 1.2: eligibility by code against every used source.** `benchmarks/unseen/eligibility.py` builds the
|
||||
used set before the draw. It never prints names; it records counts per source and a digest of the set.
|
||||
|
||||
| Source (Req 1.1) | Where it is read |
|
||||
| --- | --- |
|
||||
| Pair corpora | every catalog, recipe and dataset file under `benchmarks/pairs/`, plus the local pointer-pair manifests |
|
||||
| Populations v1, v2 | `population-v1.toml`, `population-v2.toml`, read directly |
|
||||
| Population v3 | never opened; the existing guard test only (below) |
|
||||
| Harvest records | the batch catalogs under `benchmarks/pairs/advisory-fixes*` (the records hold counts only) |
|
||||
| Memory examples | `example` rows of the local FileStore and the S3 store (`repo` field) |
|
||||
| Experiment units | `plane/experiments/*.units.jsonl` and registered units files |
|
||||
| Measurement records | everything under `benchmarks/measurements/` and `benchmarks/experiments/` |
|
||||
| Anything else | a text sweep for `github.com/<owner>/<name>` over `benchmarks/`, `src/`, `tests/`, `plane/`, `.kiro/` |
|
||||
|
||||
Names are normalised (lower case, no `.git`, no trailing slash). Each candidate is resolved through the GitHub API
|
||||
to its canonical `full_name`, its former names (redirects) and its fork network (`parent`, `source`). A candidate
|
||||
is out if any of these is in the used set. A fork shares code with what it was forked from.
|
||||
|
||||
v3 is checked like harvest-3 did. The draft manifest is written into `benchmarks/unseen/`. Then
|
||||
`pytest -q tests/test_independent_population.py -k referenced` runs with its output discarded. The guard already
|
||||
sweeps `benchmarks/` outside the population directory. On red, the candidate list is bisected by rewriting the
|
||||
draft until green, and the dropped count is recorded. No v3 name is ever printed or read by a person.
|
||||
|
||||
**Decision 1.3: post-cutoff preference.** An advisory published after 2026-06-01 (harvest-3's cutoff for the
|
||||
decision engine's model) is flagged `post_cutoff`. Within each ecosystem stratum post-cutoff advisories are drawn
|
||||
first. Results are reported for both groups. Reason: a post-cutoff advisory cannot be in the model's training
|
||||
data as a known vulnerability. Rejected: post-cutoff only. Harvest-3 kept 12 post-cutoff pairs out of 251, and
|
||||
three slices of 100 repositories cannot be filled from that window.
|
||||
|
||||
**Decision 1.4: licences.** The licence is read at the pinned head commit (LICENSE file, then the API's SPDX id).
|
||||
Only permissive licences are published by name: MIT, Apache-2.0, BSD-2-Clause, BSD-3-Clause, ISC, 0BSD, Unlicense,
|
||||
Zlib, PostgreSQL, BSL-1.0. Everything else (copyleft, source-available, none) goes to the gitignored
|
||||
`benchmarks/unseen/private/pool-p1.toml`. The committed manifest then holds an opaque id and the sha256 of
|
||||
(URL, commits) for that entry, so the freeze digest still covers it. Reason: Req 1.4 read literally. Rejected:
|
||||
"OSI-approved" as the bar, as population v2 used. It is wider than the requirement says (open question 1,
|
||||
resolved 2026-10-04: copyleft stays private).
|
||||
|
||||
Also required: anonymous HTTPS, at most 500 MB, at least 40 eligible ordinary commits, one per fork network.
|
||||
|
||||
### 2. Change extraction (Req 2.1, 2.2)
|
||||
|
||||
Extraction runs on the VM over a full `--filter=blob:none` clone in a scratch cache. It uses git only, like
|
||||
`freeze.py`. No scanner, engine or model touches a pool repository before the freeze.
|
||||
|
||||
**Decision 2.1: the vulnerability-introducing range.** Let F be the fix and P its parent. The vulnerable lines are
|
||||
the old-side lines of F's hunks in the vulnerable function. For a pure-addition fix (an absence bug such as a
|
||||
missing guard), they are the vulnerable function's own lines.
|
||||
|
||||
1. **`introducing` (preferred).** `git blame -w -M -C --porcelain P -L <lines> -- <file>`. The candidate V is the
|
||||
commit that last wrote the sink line (for absence bugs, the function's declaration). V is accepted if all hold:
|
||||
at V's first parent the sink line's normalised text is absent from the file's vulnerable function; V is not a
|
||||
merge or root commit; V changes at most 50 files and 2,000 lines; when OSV names a first affected version, V is
|
||||
reachable from that tag and not from the previous release tag. If the text is present at V's parent, blame
|
||||
walks back from there, at most five steps. The replay is `base = V^`, `head = V`.
|
||||
2. **`last_touch` (fallback).** When no V passes, `git log -L :<function>:<file> P` gives the newest commit L
|
||||
before F that changed the vulnerable function. The replay is `base = L^`, `head = P`. The vulnerable code is in
|
||||
head, and the range contains a change to the vulnerable function.
|
||||
|
||||
Each change records its method, family, vulnerable function (resolved in head), head-side changed lines,
|
||||
`post_cutoff` and the advisory id.
|
||||
|
||||
Bias of each, stated in the record. `introducing` inherits SZZ noise: blame finds the last writer, not the author
|
||||
of the flaw. A refactor, a move the heuristics miss or a reformat can stand in for the real introduction. The
|
||||
absence check, the walk-back and the version bracket cut this down, but do not remove it. The bias runs both
|
||||
ways: a wrong V may introduce nothing (a miss is not the hook's fault) or may move a flaw (the delta-based hook is
|
||||
right to call it existing). `last_touch` is pessimistic for a delta hook. The flaw is often already in base, so
|
||||
the hook's novelty rule correctly stays quiet and the change counts as a miss. Catch is therefore reported per
|
||||
method. The headline is `introducing`; `last_touch` is a lower bound. Rejected: fix-parent against a fixed old
|
||||
base (for example 50 commits back). The range often misses the vulnerable function entirely and grows without
|
||||
bound on busy repositories.
|
||||
|
||||
Label check: every third `introducing` change is read by an agent before the freeze, with no model call, as
|
||||
harvest-3 reviewed diffs. It is marked `confirmed`, `refactor` or `unclear`. The confirmed share is the label
|
||||
noise estimate. The scorer also reports catch on the confirmed subset.
|
||||
|
||||
**Decision 2.2: ordinary commits.** The candidates are non-merge commits on the first-parent line of the default
|
||||
branch that change at least one source file of an analysed language. Excluded:
|
||||
|
||||
- explicit security signals in the message: `secur|cve|ghsa|vuln|inject|xss|csrf|ssrf|exploit|sanitiz|escap`;
|
||||
- reverts (`^Revert`, `This reverts commit`);
|
||||
- dependency bumps (bot authors such as dependabot and renovate, `bump|upgrade .* from`, or only manifest and lock
|
||||
files changed);
|
||||
- commits whose source changes are all generated or vendored (`vendor/`, `node_modules/`, `dist/`, `*.min.js`,
|
||||
lock files, `linguist-generated` in `.gitattributes`, "generated" headers);
|
||||
- any fix commit of any advisory of the repository, and any commit touching a known vulnerable function of it;
|
||||
- commits over 1,000 changed source lines (counted, reported).
|
||||
|
||||
Twelve are drawn per repository with a seeded random draw. Strata are three time thirds of the repository's
|
||||
history crossed with four size buckets of changed source lines (1-10, 11-50, 51-200, 201-1,000). Allocation is
|
||||
proportional to the repository's own commits per cell, with at least one per time third. Each change records base
|
||||
(its first parent), head, changed files and line ranges. Reason: proportional allocation keeps the false-alarm
|
||||
rate a per-push rate as a user meets it. Rejected: `freeze.py`'s broad `SECURITY_WORDS`. It also drops commits
|
||||
saying "query", "token", "session" or "valid", which removes the ordinary commits where false alarms are most
|
||||
likely, and the rate would look better than it is. Rejected: docs-only commits. They are trivially quiet and
|
||||
would dilute the rate.
|
||||
|
||||
Label noise: an ordinary commit may itself introduce an unknown vulnerability, so a flag counted as a false alarm
|
||||
may be true. The false-alarm rate stays as Req 3.2 defines it. Flagged ordinary changes are adjudicated beside it
|
||||
(all if 30 or fewer, else every third) against protocol-v2's standard, and the adjudicated share is reported.
|
||||
|
||||
### 3. Sizing and slices (Req 2.3, 5.2)
|
||||
|
||||
Wilson 95% interval, z = 1.96: centre (p + z²/2n)/(1 + z²/n), half width z·sqrt(p(1-p)/n + z²/4n²)/(1 + z²/n).
|
||||
Taking the larger distance from p as the width, the smallest n that meets the bar, computed exactly:
|
||||
|
||||
| Rate | Assumed p | Bar | n needed |
|
||||
| --- | --- | --- | --- |
|
||||
| false alarm | 0.05 | +/- 3 points | 315 |
|
||||
| false alarm | 0.10 | +/- 3 points | 483 |
|
||||
| false alarm | 0.20 | +/- 3 points | 756 |
|
||||
| catch | 0.10 | +/- 10 points | 62 |
|
||||
| catch | 0.25 | +/- 10 points | 88 |
|
||||
| catch | 0.40 | +/- 10 points | 97 |
|
||||
|
||||
Catch: 100 vulnerability-introducing changes per slice, one per repository. At p = 0.4, Wilson gives
|
||||
[0.309, 0.498], within 10 points. One per repository keeps them independent.
|
||||
|
||||
False alarm: ordinary changes of one repository are correlated (noisy code flags often). Wilson assumes
|
||||
independence, so the count is inflated by a design effect 1 + (m - 1)·ICC. With m = 12 per repository and an
|
||||
assumed ICC of 0.05, that is 1.55. 100 repositories × 12 = 1,200 changes, effective n 774 ≥ 756. The plain Wilson
|
||||
interval at p = 0.2, n = 1,200 is about +/- 2.3 points. A repository-cluster bootstrap interval is reported beside
|
||||
it, and the ICC measured on slice 1 is recorded. If it exceeds 0.05, the next pool sizes up. A frozen slice is
|
||||
never changed.
|
||||
|
||||
**Decision 3.1: slices.** Pool p1 has three slices of 100 repositories each (100 + 1,200 changes per slice; 300
|
||||
repositories and 3,900 changes in all). Repositories are assigned by a seeded shuffle, stratified by ecosystem and
|
||||
`post_cutoff`. Slice 1 serves the baseline and the G1 arms table. Slices 2 and 3 serve the first two adoption
|
||||
decisions. When they are spent, pool p2 is drawn by the same code, and it excludes p1. Reason: each slice alone
|
||||
meets the Req 2.3 widths, so any decision can stand on one slice. Rejected: one large pool scored whole. Every
|
||||
decision would then see every repository, which Req 5.2 forbids. Rejected: ten small slices. None would meet the
|
||||
widths. Per family, 100 changes split over seven families give intervals of about +/- 20 to 25 points. They are
|
||||
reported, not sized (open question 3).
|
||||
|
||||
Use ledger: `benchmarks/unseen/usage-p1.jsonl` is append-only, one row per use (slice, decision or experiment id,
|
||||
date, result digest). A slice with a row whose purpose is `informed` cannot appear again as `qualifies` for that
|
||||
change. A test checks this.
|
||||
|
||||
### 4. Scoring rules (Req 3.1, 3.2, 3.3)
|
||||
|
||||
Each replay uses the hook's defaults: `--deadline 30`, `--engine inline`, `--max-regions 500`, `--mode advisory`.
|
||||
That is what a user's push gets. The text output is kept with every `Skipped:` line, and so is the artifact.
|
||||
|
||||
**Decision 4.0: a registered long-deadline arm** (maintainer decision 2026-10-04, open question 5). Beside the
|
||||
default replay, slice 1 is replayed once more with `--deadline 300` and every other flag unchanged. The arm is
|
||||
registered (manifest committed, `unseen: {pool: p1, slice: 1}`) before its first replay. G1 then reports both
|
||||
what the user sees (30 s) and what the engine could catch given time (300 s), so a miss caused by the deadline is
|
||||
told apart from a miss by the engine. It is a measurement arm, not a proposed default: adopting a longer hook
|
||||
deadline would be a separate decision under 7.2.
|
||||
|
||||
**Decision 4.1: catch, per family.** A finding counts if its family is the change's family. Engine streams carry a
|
||||
family; a quick-rule match maps its CWE through the family CWE table. Its head location must also be in the
|
||||
vulnerable function or within 10 lines of the change's head-side hunks (protocol v1's window).
|
||||
|
||||
| Family | Location rule |
|
||||
| --- | --- |
|
||||
| injection, path, deserialization, output_encoding, untrusted_destination | sink line in the vulnerable function, or the changed lines +/- 10 |
|
||||
| access_control | the vulnerable function itself (the handler that lacks the check), or the changed lines +/- 10 |
|
||||
| config_secrets | the changed file and lines +/- 10 (file scope; function `<global>`) |
|
||||
|
||||
All three streams count: admitted defects, advisory engine findings and quick-rule matches. Each catch also names
|
||||
the stream that made it. Reason: the hook shows all three to the user. Rejected: admitted defects only. That
|
||||
scores the blocking path, which G3 decides separately, and would hide what users read.
|
||||
|
||||
**Decision 4.2: false alarm.** An ordinary change is a false alarm when any of the three streams holds at least one
|
||||
finding, of any family, anywhere. The alerts-only rate (admitted defects) is reported beside it.
|
||||
|
||||
**Decision 4.3: coverage, so zeros cannot hide.** Every change gets exactly one state:
|
||||
|
||||
- `instrument_failure`: no record, a failed fetch, the head commit not verified, or zero bytes of the changed files
|
||||
read. It is re-run once, then reported. It never counts as quiet or missed.
|
||||
- `unanalysable`: no changed file is in a language the hook covers (the hook's `language_not_covered` notice).
|
||||
- `quick_only`: the quick tier read the changed files, but no engine question completed on them.
|
||||
- `engine_covered`: at least one engine question completed on a changed file.
|
||||
|
||||
The task entry proves its input as AGENTS.md requires. It records files and bytes in the checkout, bytes of each
|
||||
changed file, the JVM's wall time and the hook's exit code. A replay faster than a JVM can start is flagged
|
||||
`suspect_fast` and goes to the failure path. Catch and false alarm are reported over all changes and over each
|
||||
coverage state.
|
||||
|
||||
### 5. The AX workload (Req 4)
|
||||
|
||||
**Decision 5.1: the image.** `plane/Dockerfile.engine-task` gains a second stage, `FROM engine AS replay`, built
|
||||
with `--target replay` and pinned by digest. It adds `git`, the package with its runtime dependencies and
|
||||
`benchmarks/unseen/replay_entry.py`. Joern, php-cli, Java 21 and the frontend-retention patch come from the engine
|
||||
stage, so both lanes run the same Joern. The import smoke runs `ousast pre-push --help`. The root image stays
|
||||
`ghcr.io/norandom/ax-task-runner:v0.3.1`. Rejected: a separate Dockerfile. It would duplicate the Joern install,
|
||||
and two Joern copies are two instruments that drift. Rejected: one stage for both. The engine image's isolated
|
||||
venv without yaml and boto3 is proven, and replay dependencies would change its digest on every bump.
|
||||
|
||||
**Decision 5.2: the task entry.** `replay_entry.py` is standard library only, like `engine_task_entry.py`, and
|
||||
reuses its `transfer` (egress readiness wait, three attempts, no URL text in errors).
|
||||
|
||||
1. GET `SPEC_URL` (presigned, `.json`): repository URL, base, head, change id, hook flags.
|
||||
2. `git init`, then `git fetch --depth 1 origin <head>` and `git fetch --depth 1 origin <base>` over HTTPS
|
||||
(GitHub serves commits by SHA). Verify both with `git cat-file -e`. The fetch is bounded at 180 s; the 500 MB
|
||||
repository cap bounds its size.
|
||||
3. Run `ousast pre-push <checkout> --base <base> --head <head> --artifact /tmp/out/push.json` with defaults, the
|
||||
backend heap at 1700 MB and the cluster JVM flags of `engine_trace_worker.py`. Capture stdout.
|
||||
4. Pack `push.json`, `push.txt`, `engine.json` (the head and base scan records and witness rows from the artifact,
|
||||
keyed by repository and revision, for engine-evidence arms) and `instrument.json` (section 4.3, image digest,
|
||||
step timings). PUT to `RESULT_URL` as `.dat`. The PUT is completion.
|
||||
|
||||
On any error it PUTs a failure record naming the step and HTTP status, as the engine entry does. Task names are
|
||||
`ousast-engine-replay-<slug>-<digest>`, the prefix the operator's egress watcher matches. No key or store
|
||||
credential enters the manifest. Only the two presigned URLs do, and the spec object keeps repository names out of
|
||||
AX status and cluster telemetry.
|
||||
|
||||
**Decision 5.3: the generic dispatcher.** `benchmarks/ax/batch.py` takes over from `engine_trace_ax.AX` and
|
||||
`engine_trace.run_plan`:
|
||||
|
||||
- `Item`: id, input objects (name to bytes), the result object name, command, extra env, input digest.
|
||||
- `Workload`: image digest, allowed URL env names, deadline, how to validate a received result.
|
||||
- Lanes: `ax` (manifest, apply, poll the result object and AX phase, delete, clean objects), `kind` (the same
|
||||
manifest against the kind AX), `docker` (the VM image, one container at a time).
|
||||
- Scheduler: two AX lanes plus one VM lane by default. A sandbox death is retried once under a new name; a second
|
||||
death or a JVM OOM goes to the VM fallback queue, which beats overflow on the VM lane. `progress.json` is written
|
||||
only by the scheduler. A `STOP` file halts new launches. Resume skips items whose stored record has the same
|
||||
input digest. Cleanup failure stops dispatching.
|
||||
|
||||
The engine pass becomes a workload over it, with the same tests. Reason: the egress, secrecy and cleanup logic took
|
||||
several fixes and must exist once. Rejected: a replay-specific copy of `engine_trace_ax.py`. Rejected: putting it in
|
||||
`src/` now. G4's `ousast plane scan` may need it there, and it moves when G4 is planned.
|
||||
|
||||
Concurrency is two AX tasks. Placement is random, and only two guarantees two per worker at most. Four would need
|
||||
the operator to add placement (open question 2, resolved 2026-10-04: stays at two, no placement change
|
||||
requested). Estimated wall time per slice: about 75 s per replay (fetch, 30 s hook, setup), 1,300 replays over
|
||||
three lanes, about 9 hours. The long-deadline arm (4.0) is bounded at about 345 s per replay; at that bound 1,300
|
||||
replays over three lanes take about 42 hours, less where the hook finishes early. It runs with `STOP` and resume
|
||||
across sessions.
|
||||
|
||||
**Decision 5.4: model arms on the VM.** Arms that need DeepSeek (the decision engine, the combiner) read the stored
|
||||
records on the VM. They build candidates from the head side's changed functions, score them, and take the change
|
||||
score as the maximum. Each runs under its experiment manifest's `budget_usd` through the metered client, dry run
|
||||
first, with responses cached. Nothing in the cluster calls a model.
|
||||
|
||||
**kind (Req 4.5)** runs the identical manifest and image through the `kind` lane. Every new task is proven there
|
||||
before kube-ax, as `plane-on-kubernetes` requires.
|
||||
|
||||
### 6. Records and the guard (Req 1.2, 1.3, 5.1)
|
||||
|
||||
`benchmarks/unseen/` holds `pool-p1.toml` (repositories, licences, slices, changes with base, head, method,
|
||||
family, function, lines), `freeze-p1.json` (used-set digests and counts per source, guard result, label-check
|
||||
counts, freeze digest = sha256 of the canonical manifest), `protocol-p1.md` (the scoring rules of section 4,
|
||||
committed before the first replay) and `usage-p1.jsonl`. The `private/` directory is gitignored.
|
||||
|
||||
The guard grows by one parametrised reservation. The existing sweep becomes a list of (reservation directory,
|
||||
names): `benchmarks/independent/` for populations and `benchmarks/unseen/` for pools, including the private
|
||||
manifest when present. It also sweeps `plane/`. A second test asserts that pool and population names are disjoint,
|
||||
and prints only a count on failure. A third asserts that a frozen pool's digest matches `freeze-p1.json`, so the
|
||||
pool is never edited. Raw per-change records stay outside the repository (`~/ousast-results/unseen/` and the
|
||||
store). Committed result records under `benchmarks/measurements/` carry counts, rates, intervals and digests,
|
||||
never names.
|
||||
|
||||
### 7. Metrics and reporting (Req 3.3, 3.4, 5.3)
|
||||
|
||||
`benchmarks/unseen/score.py` reports per family and overall: catch and false-alarm rate with Wilson 95%
|
||||
intervals, the cluster bootstrap interval for false alarms, coverage counts per state, catch per method,
|
||||
post-cutoff and per stream, and the alerts-only rate.
|
||||
|
||||
**Decision 7.1: arms at fixed budgets.** An arm with a score is thresholded per budget (1%, 5%, 10%). The
|
||||
threshold is the lowest score whose false-alarm rate on the slice's ordinary changes stays within the budget.
|
||||
Catch at that threshold is reported with its Wilson interval and the realised false-alarm rate. That in-slice
|
||||
threshold is labelled exploratory. For a qualification, the threshold is fixed beforehand on an earlier slice or
|
||||
on held-out folds, and it is applied unchanged. A binary arm (today's hook) has one operating point, reported with
|
||||
the budgets it meets. Rejected: AUC over changes (it averages operating points no user runs).
|
||||
|
||||
**Decision 7.2: next to held-out folds.** Experiment manifests registered after this spec lands carry
|
||||
`unseen: {pool, slice}`. `analyse` writes an `unseen` block into `result.json` next to the fold estimates: catch at
|
||||
each budget for each arm, and the paired difference B - A with an exact McNemar test on the 100 changes. The
|
||||
fold rule decides as today. The slice is recorded in the usage ledger.
|
||||
|
||||
**Decision 7.3: adoption gate on the unseen slice** (amended 2026-10-04, maintainer decision; Req 5.4; open
|
||||
question 4 resolved). Reporting is not enough. An arm B is adopted over the production arm A only if, on the
|
||||
untouched slice that qualifies it, both hold:
|
||||
|
||||
- catch at the 5% budget, with B's threshold fixed beforehand (7.1), is not significantly lower than A's: exact
|
||||
two-sided McNemar on the paired vulnerability-introducing changes, alpha 0.05;
|
||||
- for an arm with one operating point (a binary arm such as a hook setting), its false-alarm rate is not
|
||||
significantly higher than A's: exact two-sided McNemar on the paired ordinary changes, alpha 0.05, with the
|
||||
repository-cluster bootstrap interval of the difference reported beside it, since those changes are clustered.
|
||||
|
||||
`analyse` writes `unseen.gate: pass | regression | not_run` into `result.json`. `not_run` (no slice, slice
|
||||
already `informed` for this change, or more than 5% `instrument_failure`) blocks adoption like `regression`. The
|
||||
fold rule and this gate must both pass.
|
||||
|
||||
### 8. Increments
|
||||
|
||||
1. **Used set and guard.** `eligibility.py`, the extended guard and the usage-ledger test. Exit: guard green on an
|
||||
empty pool; a planted used repository, a renamed one and a fork are each rejected in tests.
|
||||
2. **Sourcing and extraction.** Candidates, both change kinds and the label check, on the VM. Exit: a draft for 300
|
||||
repositories with method counts, the confirmed share and exclusion counts.
|
||||
3. **Freeze p1.** Guard bisection for v3, then manifest, freeze record and protocol committed. Exit: freeze digest
|
||||
committed and guard green.
|
||||
4. **Dispatcher.** `benchmarks/ax/batch.py`, with the engine pass moved onto it. Exit: existing tests green and a
|
||||
two-pin engine smoke on kube-ax matching its stored records.
|
||||
5. **Replay task.** Image stage and entry, proven on kind, then kube-ax on five changes from this repository's own
|
||||
history (never pool changes). Exit: instrument fields show bytes read and realistic JVM time, and quick-tier
|
||||
output matches the VM lane.
|
||||
6. **Scorer.** `score.py` on synthetic records, including every coverage state and an instrument failure. Exit:
|
||||
unit tests green.
|
||||
7. **Baseline.** Today's hook, from a frozen export of HEAD, on slice 1 (1,300 replays). Exit: the record committed
|
||||
(counts and digests only) and a `baseline` row in the usage ledger. That is G0's exit.
|
||||
8. **Arms on slice 1 (G1)** (added 2026-10-04 with decisions 4.0 and 7.3). The registered long-deadline arm,
|
||||
the engine-evidence arm from the stored records, the model arms on the VM (5.4) under their budgets, and the
|
||||
`unseen` block with the adoption gate in `analyse`. Exit: one committed table of arms on slice 1 at the 1/5/10%
|
||||
budgets (counts, rates, intervals, digests), and `exploratory` rows in the usage ledger.
|
||||
|
||||
## Error Handling
|
||||
|
||||
A failed fetch, a missing commit or a hook crash gives a failure record, never an empty success. A dead sandbox is
|
||||
retried once, then the VM runs it. A store or AX cleanup error stops the dispatcher. A slice with more than 5%
|
||||
`instrument_failure` after re-runs is not scored; the cause is fixed first.
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
Unit tests: used-set normalisation (case, `.git`, renames, forks), blame acceptance on a synthetic repository,
|
||||
ordinary-commit exclusions, Wilson values against section 3, budget thresholds, coverage states, the task manifest
|
||||
and dispatcher resume and fallback with a fake lane. Guard tests run in CI; cluster runs stay manual.
|
||||
|
||||
## Requirement Traceability
|
||||
|
||||
| Requirement | Design |
|
||||
| --- | --- |
|
||||
| 1.1 eligibility against every source, v3 via the guard | 1.2 |
|
||||
| 1.2 frozen manifest, never edited | 6, increment 3 |
|
||||
| 1.3 pool directory, guard extended | 6 |
|
||||
| 1.4 licences, private manifest | 1.4 |
|
||||
| 2.1 introducing changes, family, function, post-cutoff | 1.1, 1.3, 2.1 |
|
||||
| 2.2 ordinary changes across time and size | 2.2 |
|
||||
| 2.3 sizes for interval widths | 3 |
|
||||
| 3.1 default replay, skipped checks kept | 4, 5.2 |
|
||||
| 3.2 catch and false-alarm rules | 4.1, 4.2 |
|
||||
| 3.3 per family, Wilson, coverage | 4.3, 7 |
|
||||
| 3.4 arms at 1/5/10% budgets | 4.0, 7.1, increment 8 |
|
||||
| 4.1 one change one Task, prefix, image by digest | 5.1, 5.2 |
|
||||
| 4.2 HTTPS fetch, presigned record, no keys | 5.2 |
|
||||
| 4.3 generic dispatcher, concurrency, retry, fallback, resume | 5.3 |
|
||||
| 4.4 model arms on the VM within budgets | 5.4 |
|
||||
| 4.5 kind runs the same tasks | 5.3, 5 (kind paragraph) |
|
||||
| 5.1 baseline on slice 1, counts and digests | 6, increment 7 |
|
||||
| 5.2 fresh slice per decision | 3.1, usage ledger |
|
||||
| 5.3 experiments report unseen next to folds | 7.2 |
|
||||
| 5.4 adoption needs no significant unseen regression | 7.3 |
|
||||
|
||||
## Open Questions
|
||||
|
||||
Maintainer decisions of 2026-10-04 resolve questions 1, 2, 4 and 5; question 3 stays open for G3.
|
||||
|
||||
1. **Resolved 2026-10-04.** Licences: only permissively licensed repositories are published by name. Copyleft and
|
||||
all other licences go to the gitignored private manifest, covered in the committed manifest by an opaque id and
|
||||
a digest (Req 1.4 as written; decision 1.4). Population v2's OSI-approved bar is not used.
|
||||
2. **Resolved 2026-10-04.** Concurrency stays at two AX lanes plus the VM lane. No operator placement change is
|
||||
requested (5.3).
|
||||
3. Per-family BLOCK (G3) needs about 100 changes per family. Pool slices together, or draw a family-enriched pool
|
||||
for G3?
|
||||
4. **Resolved 2026-10-04.** An adoption requires no significant regression on the unseen slice, not only a report
|
||||
(Req 5.4, amended; decision 7.3).
|
||||
5. **Resolved 2026-10-04.** G1 adds a registered long-deadline arm (`--deadline 300`) next to the default 30 s, so
|
||||
the table shows both what the user sees and what the engine could catch (decision 4.0, increment 8).
|
||||
@@ -1,107 +0,0 @@
|
||||
# Requirements Document
|
||||
|
||||
## Introduction
|
||||
|
||||
The safety net is meant for repositories it has never seen. This spec builds the instrument that measures that: a
|
||||
frozen pool of unseen repositories, a push-level metric (catch rate and false-alarm rate as the hook reports them),
|
||||
and an execution path that runs the replays as AX Tasks on the kube-ax cluster. Context: `brief.md`, the roadmap's
|
||||
G0-G1 (`.kiro/steering/roadmap.md`).
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: pool selection and freezing; change extraction (vulnerability-introducing and ordinary commits);
|
||||
replay of each change through `ousast pre-push` and through registered experiment arms; the metric and its
|
||||
confidence intervals; AX Task execution with kind/VM fallback; the baseline for today's hook; rotation of pool
|
||||
slices.
|
||||
- **Out of scope**: any detection change; population v3 (stays reserved and unread); model calls inside the cluster.
|
||||
- **Adjacent expectations**: `learned-decision-engine` experiments use this pool as their primary unseen-repo score;
|
||||
`pre-push-safety-net` owns the hook being replayed; `plane-on-kubernetes` and the AX executor
|
||||
(`benchmarks/learn/engine_trace_ax.py`) supply the execution pattern; the reserved-repository guard covers every
|
||||
population and this pool.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement 1: A pool no development step has touched
|
||||
|
||||
**User Story:** As the maintainer, I want the evaluation repositories to be ones nothing in development has seen, so a
|
||||
good number means the approach generalises.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. A repository is eligible only if it appears in no pair corpus, no population (v1, v2, v3), no harvest record, no
|
||||
memory example, no experiment unit and no measurement record. Eligibility is checked by code against every source
|
||||
before the pool is frozen; v3 is checked through the existing guard without being opened.
|
||||
2. The pool is frozen (a committed manifest with repository URLs, commit SHAs, licences and the freeze digest) before
|
||||
any change in it is replayed. A frozen pool is never edited; a correction is a new pool.
|
||||
3. Pool repositories are referenced only inside the pool's own directory (like `benchmarks/independent/`), and the
|
||||
reserved-repository guard is extended to them.
|
||||
4. Only permissively licensed repositories are published by name; others may be used with their names kept out of
|
||||
the repository (pointers in a local, gitignored manifest), recorded as such.
|
||||
|
||||
### Requirement 2: Two kinds of change per repository
|
||||
|
||||
**User Story:** As the maintainer, I want both vulnerable and ordinary pushes from the same repositories, so the
|
||||
false-alarm rate is measured on the code the catch rate is measured on.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. Vulnerability-introducing changes: from public advisories with a linked fix, the commit range that introduced the
|
||||
vulnerable code where it can be determined, else the fix's parent compared with an earlier base; each labelled
|
||||
with family and the vulnerable function. Post-model-cutoff advisories are preferred and flagged.
|
||||
2. Ordinary changes: commits from the same repositories' histories with no security fix, revert or advisory link,
|
||||
sampled across time and size; each recorded with its changed files and lines.
|
||||
3. Per repository, at least one vulnerability-introducing change and several ordinary changes; the pool's totals are
|
||||
sized so the false-alarm rate's 95% interval is no wider than +/- 3 points and the catch rate's no wider than
|
||||
+/- 10 points (sizes computed in the design).
|
||||
|
||||
### Requirement 3: The push-level metric
|
||||
|
||||
**User Story:** As the maintainer, I want the score to be what a user would see, so the number means something.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. Each change is replayed as `ousast pre-push --base <base> --head <head>` with the hook's default settings, in a
|
||||
clean checkout, with the record of every skipped check kept.
|
||||
2. Catch: a vulnerability-introducing change counts as caught when the hook (or the arm) reports a finding of the
|
||||
right family on the vulnerable function or its changed lines. False alarm: an ordinary change counts as a false
|
||||
alarm when the hook reports any advisory finding.
|
||||
3. Reported per family and overall: catch rate, false-alarm rate, and both with 95% Wilson intervals; plus coverage
|
||||
(changes where the hook could analyse the changed language) so a silent zero is visible.
|
||||
4. Arms (decision engine, combiner, engine evidence, fix mechanisms) are scored on the same replays: catch rate at
|
||||
fixed false-alarm budgets (1%, 5%, 10%), so arms are compared at equal noise.
|
||||
|
||||
### Requirement 4: Execution on kube-ax, designed for AX
|
||||
|
||||
**User Story:** As the maintainer, I want the replays to run on the cluster, so measurements are fast and the VM is
|
||||
free.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. One replayed change is one AX Task, named with the `ousast-engine-` prefix the operator's egress watcher matches,
|
||||
using an image built `FROM ghcr.io/norandom/ax-task-runner:v0.3.1` with this package, Joern and php-cli, pinned by
|
||||
digest.
|
||||
2. The task fetches the repository at the needed commits itself over HTTPS (public hosts are reachable from actors)
|
||||
and writes its record through a presigned link; no API keys or store credentials enter the cluster.
|
||||
3. A generic AX batch dispatcher serves both this replay and the existing engine passes: items to Tasks, at most the
|
||||
cluster's safe concurrency (two heavy tasks per worker), retry once on sandbox death, then a VM or kind fallback
|
||||
lane; progress and resume as in the engine runner.
|
||||
4. Arms that need model calls take the cluster's records and make their calls on the VM, within registered budgets.
|
||||
5. kind runs the same tasks unchanged as the local fallback.
|
||||
|
||||
### Requirement 5: Baseline, rotation and use
|
||||
|
||||
**User Story:** As the maintainer, I want a baseline now and a fresh slice for each decision, so the pool is not
|
||||
worn out by tuning.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The first use is a baseline of today's hook on the first slice; its record is committed (counts and digests only,
|
||||
no repository names outside the pool directory).
|
||||
2. A slice that informed a change can no longer qualify that change; each adoption decision draws a new, untouched
|
||||
slice, recorded with its freeze digest.
|
||||
3. Every learned-decision-engine experiment from now on reports its unseen-repo score from this pool next to its
|
||||
held-out-fold score.
|
||||
4. An arm is adopted only if, on the untouched slice that qualifies it, it shows no significant regression against
|
||||
the current production arm: catch at the 5% false-alarm budget not significantly lower, and for an arm with one
|
||||
operating point the false-alarm rate not significantly higher (exact paired tests, two-sided, alpha 0.05).
|
||||
Reporting the unseen score alone is not enough to adopt. *(amended 2026-10-04, maintainer decision)*
|
||||
@@ -1,22 +0,0 @@
|
||||
{
|
||||
"feature_name": "unseen-repo-evaluation",
|
||||
"created_at": "2026-10-04T18:00:00Z",
|
||||
"updated_at": "2026-10-04T20:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"design": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"tasks": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": true
|
||||
}
|
||||
@@ -1,384 +0,0 @@
|
||||
# Implementation Plan
|
||||
|
||||
One task group per increment of `design.md` section 8, in that order. A group starts only after the previous
|
||||
group's exit is recorded. Maintainer decisions of 2026-10-04 are folded in: a registered long-deadline arm
|
||||
(`--deadline 300`) beside the hook's default 30 s (design 4.0, group 8); adoption needs no significant regression
|
||||
on the unseen slice (Req 5.4 as amended, design 7.3, tasks 6.4 and 8.2); only permissively licensed repositories
|
||||
are published by name, everything else goes to the gitignored private manifest under a digest (Req 1.4, design
|
||||
1.4); concurrency stays at two AX lanes plus the VM lane (design 5.3).
|
||||
|
||||
Execution rules:
|
||||
|
||||
- **Who does what.** Codex implements every coding task offline: no network, no model, no store, no cluster. The
|
||||
coordinator runs every networked step (GitHub and OSV API calls, git fetches and clones, the S3 store, image
|
||||
pushes, kind and kube-ax). Those tasks are marked **(coordinator, network)**. Steps that need the coordinator but
|
||||
no network (agent label reads, adjudication, the v3 guard bisection, commits) are marked **(coordinator)**.
|
||||
Steps that spend model money are marked **paid: needs a budget go-ahead**. Each starts only after the maintainer
|
||||
confirms a ceiling against the provider balance read just before it.
|
||||
- **Done means.** A coding task is done when its tests pass in the host suite, `ruff check`, `ruff format --check`
|
||||
and `mypy` are clean, and the reserved-repository guard is green. No test makes a network, model or cluster call.
|
||||
- **Names stay in the pool directory.** Pool repository names and URLs appear only under `benchmarks/unseen/`
|
||||
(copyleft and other non-permissive ones only under the gitignored `benchmarks/unseen/private/`). Raw per-change
|
||||
records live in `~/ousast-results/unseen/` and the store, never in the repository.
|
||||
- **Population v3 is never opened.** It is checked only through the guard test, run as a black box with its
|
||||
output discarded.
|
||||
- **Every group ends with a measured exit** and a record at
|
||||
`benchmarks/measurements/<date>-unseen-NN-<slug>/record.json`: commit, image digests where used, the instrument
|
||||
checks (bytes read, JVM time, exit codes), counts and digests. No repository names. The guard sweeps these
|
||||
records.
|
||||
- **Instrument first** (AGENTS.md). A zero, an empty draft or a fast run is treated as an instrument failure until
|
||||
bytes read and wall time prove otherwise.
|
||||
|
||||
- [x] 1. Used set and guard (design 1.2, 6; increment 1)
|
||||
- [x] 1.1 Used-set builder
|
||||
- `benchmarks/unseen/eligibility.py`: normalises names (lower case, no `.git`, no trailing slash) and builds the
|
||||
used set from every source of design table 1.2: pair catalogs, recipes and datasets under `benchmarks/pairs/`
|
||||
plus the local pointer-pair manifests as `benchmarks/pairs/catalog_gen.py` reads them; `population-v1.toml`
|
||||
and `population-v2.toml`; the advisory-fix batch catalogs; memory `example` rows (`repo` field) through an
|
||||
injected `MemoryStore` (`FileStore` locally, the S3 store when the coordinator passes one); experiment units
|
||||
under `plane/experiments/`; `benchmarks/measurements/` and `benchmarks/experiments/`; a text sweep for
|
||||
`github.com/<owner>/<name>` over `benchmarks/`, `src/`, `tests/`, `plane/` and `.kiro/`. It never reads
|
||||
population v3 and never prints a name. Output: counts per source and the sha256 of the sorted set.
|
||||
- `tests/test_unseen_eligibility.py`: a synthetic tree with one repository per source, each found; case, `.git`
|
||||
and trailing-slash variants collapse; stdout and the output JSON contain no name; a v3-shaped file planted in
|
||||
the synthetic tree is not opened (open is patched to fail on it).
|
||||
- _Requirements: 1.1_
|
||||
- [x] 1.2 Canonical resolution, renames and forks
|
||||
- `eligibility.py`: `resolve(candidate, api)` with an injected GitHub client returns the canonical `full_name`,
|
||||
former names (redirects) and the fork network (`parent`, `source`). A candidate is ineligible when any of them
|
||||
is in the used set; one candidate per fork network is kept.
|
||||
- Tests with a fake client: a planted used repository, a renamed one and a fork of a used one are each
|
||||
rejected; two forks of one network keep one; the rejection reason is a source label, not a name.
|
||||
- _Requirements: 1.1_
|
||||
- [x] 1.3 Guard extended to pools
|
||||
- `tests/test_independent_population.py`: the sweep becomes a list of (reservation directory, names):
|
||||
`benchmarks/independent/` for populations, `benchmarks/unseen/` for pools (public manifests, plus
|
||||
`benchmarks/unseen/private/*.toml` when present). The sweep also covers `plane/`. New tests: pool and
|
||||
population names are disjoint (failure prints a count only); a frozen pool's manifest digest equals its
|
||||
`freeze-pN.json` (skipped while no pool is frozen). The existing v3 handling is unchanged.
|
||||
- Each reservation is exempt only inside its own directory: population names (v3 included) are still swept in
|
||||
`benchmarks/unseen/`, the gitignored `private/` files included. Task 3.1 depends on this.
|
||||
- `.gitignore`: `benchmarks/unseen/private/`.
|
||||
- Evidence: the guard is green with an empty `benchmarks/unseen/`, and red when a test fixture names a pool
|
||||
repository in `benchmarks/measurements/` (shown once in the record).
|
||||
- _Requirements: 1.2, 1.3, 1.4_
|
||||
- [x] 1.4 Usage ledger
|
||||
- `benchmarks/unseen/ledger.py`: append-only `usage-pN.jsonl`, one row per use (slice, decision or experiment
|
||||
id, purpose `baseline | exploratory | informed | qualifies`, date, result digest, freeze digest). It refuses a
|
||||
`qualifies` row for a change whose slice already has an `informed` row for it, and refuses to rewrite rows.
|
||||
- `tests/test_unseen_ledger.py`: append, the refusal, a rewritten row detected by a digest chain.
|
||||
- _Requirements: 5.2_
|
||||
- [x] 1.5 Used-set measurement and record 01 **(coordinator, network)**
|
||||
- Run `eligibility.py` against the real sources, the S3 memory store included (read only).
|
||||
- Instrument: each source reports files read and bytes; a source with zero rows fails the run unless it is
|
||||
known empty.
|
||||
- `benchmarks/measurements/<date>-unseen-01-used-set/record.json`: counts per source, used-set digest, guard
|
||||
result (green on the empty pool), test counts of 1.1-1.4.
|
||||
- _Requirements: 1.1, 1.3_
|
||||
|
||||
- [ ] 2. Sourcing and extraction (design 1.1, 1.3, 1.4, 2, 3; increment 2)
|
||||
- [x] 2.1 Advisory candidates and repository filters
|
||||
- `benchmarks/unseen/source.py`: from GHSA advisory JSON (pip, npm, maven, composer; published 2019 or later)
|
||||
keep those whose CWE maps to a family through `src/openultrasast/model/taxonomy.py` and that link one fix
|
||||
commit with one parent in one repository; attach the OSV first affected version; flag `post_cutoff` when
|
||||
published after 2026-06-01. Repository filters: anonymous HTTPS, at most 500 MB, licence at the pinned head
|
||||
(LICENSE file, then the API's SPDX id) classed `permissive` (the design 1.4 list) or `private`.
|
||||
- `tests/test_unseen_source.py` on recorded fixture JSON (no network): CWE mapping, multi-parent and
|
||||
multi-repository fixes dropped, cutoff flag, licence classing (GPL, AGPL, LGPL, none and source-available go
|
||||
private).
|
||||
- _Requirements: 1.4, 2.1_
|
||||
- [x] 2.2 Vulnerability-introducing range
|
||||
- `benchmarks/unseen/extract.py`: the `introducing` method of design 2.1 (blame of the sink line, or the
|
||||
declaration for an absence bug; absence at V's first parent; no merge or root; at most 50 files and 2,000
|
||||
lines; OSV version bracket when present; walk-back of at most five steps) and the `last_touch` fallback. Each
|
||||
change records method, family, vulnerable function resolved in head, head-side lines, `post_cutoff`, advisory
|
||||
id. Git only; no scanner or engine.
|
||||
- `tests/test_unseen_extract.py` on a synthetic repository built in a temp dir: accepted V; a reformat walked
|
||||
back; a merge rejected; size cap; bracket rejection; fallback chosen when no V passes.
|
||||
- _Requirements: 2.1_
|
||||
- [x] 2.3 Ordinary-commit draw
|
||||
- `extract.py`: first-parent, non-merge commits touching an analysed language; the exclusions of design 2.2
|
||||
(security words of the narrow pattern, reverts, bot and bump commits, manifest- or lock-only, generated or
|
||||
vendored, advisory fix commits, commits touching a known vulnerable function, more than 1,000 source lines,
|
||||
counted). Seeded draw of 12 per repository over three time thirds by four size buckets, proportional, at least
|
||||
one per third; a repository with fewer than 40 eligible commits is dropped.
|
||||
- Tests: each exclusion; the narrow pattern keeps "query", "token", "session"; allocation and the seed
|
||||
reproduce the same draw; the 40-commit floor.
|
||||
- _Requirements: 2.2, 2.3_
|
||||
- [x] 2.4 Draft manifest and slices
|
||||
- `benchmarks/unseen/draft.py`: writes `benchmarks/unseen/draft-p1.toml` (permissive entries by name) and
|
||||
`benchmarks/unseen/private/draft-p1.toml` (the rest); the public file holds an opaque id and
|
||||
sha256(URL, commits) per private entry. Seeded shuffle into three slices of 100 repositories, stratified by
|
||||
ecosystem and `post_cutoff`; each repository has one introducing change and 12 ordinary changes.
|
||||
- Tests: the digest covers private entries; no private URL in the public file; the same seed gives the same
|
||||
slices; per-slice strata within one repository of proportional.
|
||||
- _Requirements: 1.2, 1.4, 2.3, 5.2_
|
||||
- [ ] 2.5 Sourcing run **(coordinator, network)**
|
||||
- `gh api /advisories` enumeration, OSV lookups, canonical resolution against the used set (1.2), blobless
|
||||
clones into a scratch cache, extraction (2.2, 2.3), draft (2.4).
|
||||
- Instrument: clones counted with bytes on disk; a repository whose clone reads zero source files is an
|
||||
instrument failure, not a drop. Fewer than 300 eligible repositories stops the run with counts; the
|
||||
eligibility bars are not relaxed to fill it.
|
||||
- Observable: a draft of 300 repositories (3,900 changes) on disk.
|
||||
- _Requirements: 1.1, 1.4, 2.1, 2.2, 2.3_
|
||||
- [ ] 2.6 Label check and record 02 **(coordinator)**
|
||||
- Every third `introducing` change read by an agent, no model call, marked `confirmed | refactor | unclear`
|
||||
in the draft.
|
||||
- `benchmarks/measurements/<date>-unseen-02-draft/record.json`: candidates per ecosystem, rejections per used
|
||||
source, licence split (public, private), method counts, post-cutoff counts, confirmed share, ordinary-commit
|
||||
exclusion counts. No names.
|
||||
- _Requirements: 2.1, 2.2_
|
||||
|
||||
- [ ] 3. Freeze p1 (design 1.2, 6; increment 3)
|
||||
- [ ] 3.1 Guard bisection tool
|
||||
- `benchmarks/unseen/bisect_guard.py`: rewrites the draft with a candidate subset, runs
|
||||
`.venv/bin/pytest -q -p no:cacheprovider tests/test_independent_population.py -k referenced` with stdout and
|
||||
stderr discarded, reads only the exit code, and bisects until green. Reports the dropped count only.
|
||||
- `tests/test_unseen_bisect.py` with an injected fake guard (a hidden set): finds and drops exactly the hidden
|
||||
members; never logs a candidate; an exit code other than 0 or 1 aborts (instrument failure).
|
||||
- _Requirements: 1.1_
|
||||
- [ ] 3.2 Freeze writer
|
||||
- `benchmarks/unseen/freeze.py`: canonical `pool-p1.toml` (and the private counterpart) from the bisected draft;
|
||||
`freeze-p1.json` with used-set digest and per-source counts, guard result, dropped count, label-check counts
|
||||
and the freeze digest (sha256 of the canonical manifest). Refuses to write over a frozen pool; a correction
|
||||
is `p2`.
|
||||
- Tests: digest stable under key order; overwrite refused; the guard's digest test (1.3) passes on a fixture
|
||||
pool.
|
||||
- _Requirements: 1.2_
|
||||
- [ ] 3.3 Protocol p1
|
||||
- `benchmarks/unseen/protocol-p1.md`: the scoring rules of design section 4 (default flags, decision 4.0's
|
||||
long-deadline arm, catch and location rules, false alarm, coverage states), the statistics of section 3 and
|
||||
7, and the adoption gate of 7.3, stated before any replay.
|
||||
- Observable: the file exists and its sha256 is listed in `freeze-p1.json`.
|
||||
- _Requirements: 3.1, 3.2, 3.3, 3.4, 5.4_
|
||||
- [ ] 3.4 Freeze run and record 03 **(coordinator)**
|
||||
- Run the bisection (3.1), then the freeze (3.2). Guard green, digest test active.
|
||||
- `benchmarks/measurements/<date>-unseen-03-freeze/record.json`: freeze digest, protocol digest, dropped
|
||||
count, final counts per slice and kind.
|
||||
- _Requirements: 1.1, 1.2, 1.3_
|
||||
|
||||
- [ ] 4. Dispatcher (design 5.3; increment 4)
|
||||
- [x] 4.1 Items, workloads and the AX lane
|
||||
- `benchmarks/ax/batch.py`: `Item` (id, input objects, result object name, command, extra env, input digest),
|
||||
`Workload` (image digest, allowed URL env names, deadline, result validator) and the `ax` lane, moved from
|
||||
`benchmarks/learn/engine_trace_ax.py`: manifest render, egress readiness, presigned GET and PUT as `.dat`,
|
||||
apply, poll result object and AX phase, delete, object cleanup. The `kind` lane is the same lane with the
|
||||
kind context.
|
||||
- `tests/test_ax_batch.py` with a fake `ax` CLI and a fake store: manifest holds only the allowed URL env
|
||||
names and no credential; names carry the `ousast-engine-` prefix; cleanup on success and failure.
|
||||
- _Requirements: 4.1, 4.2, 4.5_
|
||||
- [x] 4.2 Scheduler, fallback and resume
|
||||
- `batch.py`: two AX lanes and one VM `docker` lane by default (one container at a time); sandbox death retried
|
||||
once under a new name, a second death or a JVM OOM goes to the VM fallback queue ahead of overflow;
|
||||
`progress.json` written by the scheduler only; a `STOP` file halts launches; resume skips items whose stored
|
||||
record has the same input digest; a cleanup failure stops dispatching.
|
||||
- Tests with fake lanes: retry then fallback; OOM straight to fallback; resume skips; `STOP`; cleanup failure
|
||||
halts; never more than two AX tasks in flight.
|
||||
- _Requirements: 4.3_
|
||||
- [x] 4.3 Engine pass on the dispatcher
|
||||
- `benchmarks/learn/engine_trace_ax.py` and `engine_trace.run_plan` become an engine `Workload` over `batch.py`;
|
||||
no behaviour change.
|
||||
- Observable: `tests/test_ax_executor.py` and `tests/test_engine_trace.py` pass with their assertions unchanged
|
||||
(moved imports only).
|
||||
- _Requirements: 4.3_
|
||||
- [ ] 4.4 Engine smoke on kube-ax and record 04 **(coordinator, network)**
|
||||
- Two pins through the new dispatcher on kube-ax, compared with their stored records.
|
||||
- `benchmarks/measurements/<date>-unseen-04-dispatcher/record.json`: image digest, per-pin wall time and JVM
|
||||
time, record digests equal to the stored ones (or the diff counted), cleanup result.
|
||||
- _Requirements: 4.3_
|
||||
|
||||
- [ ] 5. Replay task (design 5.1, 5.2; increment 5)
|
||||
- [x] 5.1 Image stage
|
||||
- `plane/Dockerfile.engine-task`: second stage `FROM engine AS replay` adding `git`, the package with its
|
||||
runtime dependencies and `benchmarks/unseen/replay_entry.py`; smoke `ousast pre-push --help`. The root image
|
||||
stays `ghcr.io/norandom/ax-task-runner:v0.3.1`; the engine stage is unchanged.
|
||||
- `tests/test_unseen_image.py`: parses the Dockerfile; the `replay` stage builds from `engine`; the root image
|
||||
pin; the engine stage's text is byte-identical to before.
|
||||
- _Requirements: 4.1_
|
||||
- [x] 5.2 Task entry
|
||||
- `benchmarks/unseen/replay_entry.py`, standard library only, reusing `engine_task_entry.transfer`: GET the spec;
|
||||
`git init` and depth-1 fetches of head and base over HTTPS (180 s bound), verified with `git cat-file -e`; run
|
||||
`ousast pre-push` with the spec's hook flags (defaults, or `--deadline 300` for the long-deadline arm), heap
|
||||
1700 MB and the cluster JVM flags; pack `push.json`, `push.txt`, `engine.json`, `instrument.json` (files and
|
||||
bytes in the checkout, bytes of each changed file, JVM wall time, hook exit code, step timings, image
|
||||
digest); PUT as `.dat`. Any error PUTs a failure record naming the step and status, never URL text.
|
||||
- `tests/test_unseen_replay_entry.py`: a local bare repository and a localhost presign stub; success record
|
||||
fields; a missing commit, a failed fetch and a hook crash each give a failure record; zero bytes read and a
|
||||
replay faster than JVM start are flagged `suspect_fast`.
|
||||
- _Requirements: 3.1, 4.1, 4.2_
|
||||
- [x] 5.3 Replay workload
|
||||
- `benchmarks/unseen/replay.py`: builds `Item`s from a frozen slice and an arm (`default` or `long_deadline`);
|
||||
the spec object carries the repository URL so names stay out of AX status and telemetry; task names
|
||||
`ousast-engine-replay-<slug>-<digest>` with an opaque slug; the validator rejects a record without
|
||||
`instrument.json`; records go to `~/ousast-results/unseen/<pool>/<slice>/<arm>/` and the store.
|
||||
- Tests: no repository name in any task name or manifest; only the two presigned URLs in the env; input
|
||||
digest changes with the arm's flags; `instrument_failure` items queued for one re-run.
|
||||
- _Requirements: 3.1, 4.1, 4.2, 4.3_
|
||||
- [ ] 5.4 Proof on kind, then kube-ax, and record 05 **(coordinator, network)**
|
||||
- Build and push the `replay` target, pinned by digest. Five changes from this repository's own history (never
|
||||
pool changes) on kind, then the same five on kube-ax, then on the VM lane.
|
||||
- Exit: instrument fields show bytes read and realistic JVM time on every lane; quick-tier findings equal
|
||||
across lanes.
|
||||
- `benchmarks/measurements/<date>-unseen-05-replay-task/record.json`: image digest, per-lane timings, bytes,
|
||||
exit codes, the quick-tier comparison.
|
||||
- _Requirements: 4.1, 4.2, 4.5_
|
||||
|
||||
- [x] 6. Scorer (design 3, 4, 7.1, 7.3; increment 6)
|
||||
- [x] 6.1 Coverage states
|
||||
- `benchmarks/unseen/score.py`: each record gets one state (`instrument_failure`, `unanalysable`, `quick_only`,
|
||||
`engine_covered`) from `instrument.json` and the artifact; `suspect_fast` goes to failure; a slice above 5%
|
||||
`instrument_failure` after re-runs is refused.
|
||||
- `tests/test_unseen_score.py` on synthetic records: every state; the refusal; a failure never counts as quiet
|
||||
or missed.
|
||||
- _Requirements: 3.3_
|
||||
- [x] 6.2 Catch and false-alarm rules
|
||||
- `score.py`: family match (engine family, quick-rule CWE through `taxonomy.py`), the location table of design
|
||||
4.1, all three streams with the stream named per catch; false alarm on any finding of any stream; the
|
||||
alerts-only rate.
|
||||
- Tests: right family wrong place misses; `access_control` on the handler; `config_secrets` file scope; a
|
||||
quick-rule-only catch; one advisory finding on an ordinary change is a false alarm.
|
||||
- _Requirements: 3.2_
|
||||
- [x] 6.3 Rates, intervals and breakdowns
|
||||
- `score.py`: Wilson 95% intervals; repository-cluster bootstrap for false alarms; ICC; breakdowns per family,
|
||||
method, `post_cutoff`, stream, coverage state and the confirmed subset.
|
||||
- Tests: the n values of design section 3 (315, 483, 756, 62, 88, 97) reproduced; bootstrap seeded; ICC on a
|
||||
synthetic clustered set.
|
||||
- _Requirements: 2.3, 3.3_
|
||||
- [x] 6.4 Budgets, paired tests, the gate and record 06
|
||||
- `score.py`: per-budget thresholds (1%, 5%, 10%), labelled `exploratory` in-slice and `fixed` when supplied;
|
||||
a binary arm's single operating point with the budgets it meets; exact two-sided McNemar for catch and for
|
||||
false alarm; `gate(A, B)` returning `pass | regression | not_run` per design 7.3.
|
||||
- Tests: thresholds on a synthetic score set; McNemar exact values; a binary arm with a significant
|
||||
false-alarm increase gives `regression`; missing slice or excess failures give `not_run`.
|
||||
- `benchmarks/measurements/<date>-unseen-06-scorer/record.json`: test counts and the digest of the scorer's
|
||||
output on the committed synthetic fixture.
|
||||
- _Requirements: 3.4, 5.4_
|
||||
|
||||
- [ ] 7. Baseline (design 4, 6; increment 7; G0 exit)
|
||||
- [ ] 7.1 Frozen source and dry run **(coordinator, network)**
|
||||
- Export HEAD to a frozen tree; build and push the `replay` image from it, pinned by digest.
|
||||
- `benchmarks/push/python_overhead.py` run with the stub engine to size per-replay Python cost; dispatcher dry
|
||||
run on slice 1 lists 1,300 items without launching.
|
||||
- _Requirements: 5.1, 4.1_
|
||||
- [ ] 7.2 Baseline replay of slice 1 **(coordinator, network)**
|
||||
- Default arm, 1,300 replays over two AX lanes and the VM lane (about 9 hours); `instrument_failure` items
|
||||
re-run once; the run stops for a cause above 5% failures.
|
||||
- Observable: 1,300 records in `~/ousast-results/unseen/p1/1/default/` and the store, each with
|
||||
`instrument.json`.
|
||||
- _Requirements: 3.1, 4.3, 5.1_
|
||||
- [ ] 7.3 Adjudication **(coordinator)**
|
||||
- Flagged ordinary changes adjudicated against protocol-v2's standard (all if 30 or fewer, else every third);
|
||||
the adjudicated share kept beside the false-alarm rate. ICC measured; above 0.05 noted as "next pool sizes
|
||||
up".
|
||||
- _Requirements: 3.2, 2.3_
|
||||
- [ ] 7.4 Record 07 and the ledger **(coordinator)**
|
||||
- `benchmarks/measurements/<date>-unseen-07-baseline/record.json`: `score.py` output (counts, rates, Wilson and
|
||||
bootstrap intervals, coverage per state, per family, method, stream), freeze digest, image digest, commit.
|
||||
`usage-p1.jsonl` row `baseline`.
|
||||
- Observable: guard green over the record; G0's exit met.
|
||||
- _Requirements: 5.1, 5.2, 3.3_
|
||||
|
||||
- [ ] 8. Arms on slice 1 (design 4.0, 5.4, 7.1-7.3; increment 8; G1)
|
||||
- [ ] 8.1 Registered long-deadline arm
|
||||
- `plane/experiments/` manifest for the long-deadline arm: hook flags with `--deadline 300`, budget 0,
|
||||
`unseen: {pool: p1, slice: 1}`, committed before its first replay. `src/openultrasast/learn/experiments.py`
|
||||
accepts `unseen` and a hook-flag arm.
|
||||
- `tests/test_learn_experiments.py`: the manifest loads; an `unseen` block without pool or slice is refused.
|
||||
- _Requirements: 3.4, 5.3_
|
||||
- [ ] 8.2 `unseen` block and adoption gate in `analyse`
|
||||
- `experiments.py` `analyse`: writes `unseen` into `result.json` (catch at each budget per arm, paired B - A,
|
||||
McNemar, `gate`) from `score.py`; checks the usage ledger; adoption requires the fold rule and
|
||||
`gate == pass`.
|
||||
- Tests: `regression` and `not_run` each block adoption; an `informed` slice cannot qualify; result JSON shape.
|
||||
- _Requirements: 5.2, 5.3, 5.4_
|
||||
- [ ] 8.3 Engine-evidence arm
|
||||
- `benchmarks/unseen/arms.py`: a score per change from `engine.json` (witness rows on head-side changed
|
||||
functions), no model; plugs into the budget thresholds of 6.4.
|
||||
- Tests on synthetic records: score monotone in witness strength; a change without engine coverage gets no
|
||||
score and is reported, not counted quiet.
|
||||
- _Requirements: 3.4_
|
||||
- [ ] 8.4 Model-arm adapter
|
||||
- `arms.py`: candidates from head-side changed functions of stored records, scored by the current decision
|
||||
engine program (and the combiner when its program exists; else its row reads `not_built`); change score is the
|
||||
maximum; metered client, dry run first, response cache, the manifest's `budget_usd` as the hard ceiling.
|
||||
- Tests with a scripted client: dry-run estimate, ceiling stop, cache replay; no model call.
|
||||
- _Requirements: 4.4, 3.4_
|
||||
- [ ] 8.5 Long-deadline replay of slice 1 **(coordinator, network)**
|
||||
- 1,300 replays of the `long_deadline` arm, bounded at about 345 s each (up to about 42 hours over three
|
||||
lanes), with `STOP` and resume across sessions; failures re-run once.
|
||||
- Observable: 1,300 records under `~/ousast-results/unseen/p1/1/long_deadline/`, JVM time in each.
|
||||
- _Requirements: 3.1, 3.4, 4.3_
|
||||
- [ ] 8.6 Model arms on slice 1 **(coordinator, network; paid: needs a budget go-ahead)**
|
||||
- Dry run gives the candidate count; estimate at the injection slice's measured cost (about $0.008 per
|
||||
candidate at k = 5); the maintainer sets the ceiling against the balance read just before. Then the decision
|
||||
engine arm (and the combiner if built) on the VM.
|
||||
- _Requirements: 4.4, 3.4_
|
||||
- [ ] 8.7 Record 08: the arms table **(coordinator)**
|
||||
- `benchmarks/measurements/<date>-unseen-08-arms/record.json`: per arm (hook 30 s, hook 300 s, engine evidence,
|
||||
decision engine, combiner or `not_built`) catch at 1/5/10% budgets with Wilson intervals and realised
|
||||
false-alarm rates, coverage per state, spend; `exploratory` rows in `usage-p1.jsonl`.
|
||||
- Observable: one committed table of arms on slice 1; G1's table exists.
|
||||
- _Requirements: 3.4, 5.2, 5.3_
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- 2026-10-04: Tasks 1.1–1.4 implemented offline and independently reviewed. Task 1.5 remains for the
|
||||
coordinator; no networked measurement or commit was performed. Command and API contracts:
|
||||
`benchmarks/unseen/README.md`. Offline evidence:
|
||||
`benchmarks/measurements/2026-10-04-unseen-01-offline/record.json`.
|
||||
- Population guard scope is preserved under the maintainer's hard rule: benchmarks/src/tests for
|
||||
populations, plus plane for pools. Each directory exemption is reservation-specific, including private
|
||||
pool manifests. Extending population scope to plane exposed 30 existing references; no historical
|
||||
records were changed or exempted inside the existing population scope.
|
||||
- S3 inventory opens through `open_store(read_only=True)` because ordinary store construction writes a
|
||||
Select capability probe. Default store behavior is unchanged; inventory uses LIST/GET only.
|
||||
- The freeze writer must match the canonical TOML digest encoding documented in the pool README and
|
||||
exercised by `test_independent_population.py`; final pool discovery does not trust mutable status.
|
||||
|
||||
### Group 2 implementation evidence (2026-10-04)
|
||||
|
||||
Tasks 2.1–2.4 are implemented offline and independently reviewed. Task 2.5 is
|
||||
`python -m benchmarks.unseen.draft --root "$PWD" --s3-store s3:// --seed 20260601`
|
||||
with the documented PATH/PYTHONPATH and GH_TOKEN environment. No networked draw,
|
||||
label check, group-2 measurement record, guard bisection, or commit was performed.
|
||||
The group header remains pending its coordinator exits 2.5/2.6.
|
||||
|
||||
The command rebuilds the group-1 used set, resolves renamed/fork repositories,
|
||||
uses disposable external blobless clones, re-reads licences at extracted heads,
|
||||
and emits counts/digests only. The README documents injected clients, method bias,
|
||||
the hard 300-repository floor, and an explicitly unmeasured resource estimate.
|
||||
Independent review caught and fixed ambiguous parent-function absence and stale
|
||||
candidate-404 handling; regression tests cover both.
|
||||
|
||||
Validation: requested pytest selection 103 passed, 1 skipped, exit 0 (population-test
|
||||
output discarded); `ruff check .` exit 0; `ruff format --check .` exit 0 (411 files);
|
||||
`mypy src/openultrasast` exit 0 (160 source files). The CLI `--help` smoke exited 0.
|
||||
Source-read instrumentation printed 20,101 bytes for the extraction implementation.
|
||||
|
||||
### Groups 4–6 offline implementation (2026-10-04)
|
||||
|
||||
Tasks 4.1–4.3, 5.1–5.3 and 6.1–6.4 are implemented and independently reviewed.
|
||||
Tasks 4.4 and 5.4 remain coordinator-owned network proofs; group headers 4 and 5 stay
|
||||
pending those exits. No commit, network request, Docker, AX or kubectl command was run.
|
||||
The sourcing job's draft.py, source.py, extract.py and journal.py are unchanged.
|
||||
|
||||
The shared dispatcher retains the engine adapter's historical questions.json object
|
||||
name to preserve its unchanged assertions; replay uses spec.dat and result.dat.
|
||||
The engine Dockerfile body is unchanged except its required AS engine alias; CI builds
|
||||
engine and replay separately. S3Store's read_only class default preserves existing
|
||||
constructor-bypassing transport fixtures; initialized read-only stores still refuse writes.
|
||||
|
||||
Exact coordinator commands and input contracts are in benchmarks/unseen/README.md,
|
||||
including frozen TOML/private pointer verification, scoring joins, own-history proof
|
||||
preparation and anonymous measurement-record generation. JVM wall time is the hook's
|
||||
aggregate scan build/query timing and is labelled as such. Unsupported-only partitions
|
||||
with explicit language_not_covered evidence do not pretend to have launched a JVM.
|
||||
|
||||
Record 06 contains the committed synthetic fixture and scorer-output digests. The
|
||||
reserved-repository guard was run as a black box with both output streams discarded.
|
||||
Independent review: APPROVED; all reported integration and coverage defects resolved.
|
||||
|
||||
Final verification: requested pytest command 279 passed, 8 skipped, exit 0; separate
|
||||
`tests/test_ax_batch.py` 5 passed, exit 0; `ruff check .`, `ruff format --check .`,
|
||||
`mypy src/openultrasast`, and the output-discarded reserved-repository guard all exit 0.
|
||||
The own-history proof preparer and replay dry run read five items and exit 0.
|
||||
@@ -1,23 +1,8 @@
|
||||
# Overview — what this is, and what is actually true of it
|
||||
|
||||
## Current state, 2026-10-04
|
||||
|
||||
Everything below this section is dated history; where it disagrees with this section, this section wins. The
|
||||
direction lives in `roadmap.md`, section of 2026-10-04 (phases G0-G5).
|
||||
|
||||
- **Goal.** A pre-push safety net that works on repositories it has never seen. Two numbers per push decide it:
|
||||
catch rate on real vulnerability-introducing changes, and false-alarm rate on ordinary changes. Neither has been
|
||||
measured yet; the unseen-repository pool (G0, spec `unseen-repo-evaluation`) comes first.
|
||||
- **Correction, 2026-10-04.** Infrastructure and engine work had drifted from that goal. It is now done only when
|
||||
it removes a bottleneck for those measurements.
|
||||
- **Engine.** Joern runs on the operator's kube-ax cluster as AX Tasks, with a VM fallback lane; kind stays the
|
||||
development profile. It is almost blind to fixes: paths differ between sides in 2 of 105 both-asked pairs.
|
||||
- **Decision engine.** Ranks but cannot block (exp-003 to exp-005 inconclusive; 710 fix pairs in memory). Fix
|
||||
mechanisms is one G2 arm, paused after mining until the pool exists.
|
||||
- **Hook.** `ousast pre-push` is operational and advisory on any repository; rollout stays NO-GO.
|
||||
|
||||
## State, 2026-10-02 (superseded by the section above)
|
||||
## Current state, 2026-10-02
|
||||
|
||||
Everything below this section is dated history; where it disagrees with this section, this section wins.
|
||||
User-facing documentation: `README.md`, `docs/architecture.md`, `docs/evaluation.md`, `docs/decision-engine.md`,
|
||||
`ops/ax/README.md`.
|
||||
|
||||
|
||||
@@ -1,100 +1,5 @@
|
||||
# Roadmap
|
||||
|
||||
## Roadmap to detection with the decision engine on kube-ax (2026-10-04)
|
||||
|
||||
This section supersedes the ordering below it. The older sections stay as history.
|
||||
|
||||
### Destination
|
||||
|
||||
On every push, from a hook or CI, OpenUltraSAST answers for the changed code. The answer is advisory for every
|
||||
family, and BLOCK only for families whose precision has been proven on repositories the system never saw. The work
|
||||
is split by what each place is good at:
|
||||
|
||||
- **kube-ax cluster** (operator k3s, Agent Substrate v0.3.0, AX v0.3.1, 2 nodes): the Joern engine, as AX Tasks.
|
||||
No API keys and no store credentials in the cluster; tasks read and write through short-lived presigned links.
|
||||
- **The VM or CI runner**: candidate selection, the decision engine's model calls (DeepSeek), the memory store,
|
||||
and the fallback engine lane (2.5 GB heap) for pins that do not fit the cluster's 1.7 GB heap.
|
||||
- **kind** stays the local development profile, with the same code and manifests.
|
||||
|
||||
### The question everything answers
|
||||
|
||||
Does the safety net work on repositories it has never seen? That is the purpose of the A/B experiments, and every
|
||||
phase below is judged by it. On an unseen repository almost every push is an ordinary change, so two numbers decide
|
||||
whether the safety net is useful, measured on the hook's advisory output exactly as a user would see it:
|
||||
|
||||
1. **Catch rate:** on real vulnerability-introducing changes in unseen repositories, how often it flags the change.
|
||||
2. **False-alarm rate:** on ordinary changes in the same unseen repositories, how often it flags anything.
|
||||
|
||||
Within-pair AUC on known fix pairs answers neither. It stays a development signal, not the goal.
|
||||
|
||||
### Where we are (2026-10-04)
|
||||
|
||||
| Area | State | Evidence |
|
||||
| --- | --- | --- |
|
||||
| Generalisation measured so far | The classifier ranks above chance on repository-held-out folds (within-pair AUC about 0.76) but cannot reach the block bar; the engine-based scan failed fresh populations (v1 1/11, v2 0/17); push-level catch and false-alarm rates on unseen repositories have never been measured | exp-003/004/005, `results-v1.json`, `results-v2.json` |
|
||||
| Pre-push hook | Operational and advisory on any repository | `2026-10-02-pre-push-operational-1/record.json` |
|
||||
| Labelled data | 710 complete real fix pairs | `2026-10-04-decision-engine-harvest-3/record.json` |
|
||||
| Engine | Almost blind to fixes (paths differ between sides in 2 of 105 both-asked pairs) | `2026-10-04-joern-slice-leak-audit/record.json` |
|
||||
| Infrastructure | Engine runs on kube-ax as AX Tasks with a VM fallback; kind for development | memory `kube-ax-cluster` |
|
||||
|
||||
Correction recorded 2026-10-04: the previous two days went mostly into infrastructure and engine internals that did
|
||||
not move a generalisation number. From here, infrastructure and engine work is done only when it removes a
|
||||
bottleneck for the measurements below.
|
||||
|
||||
### Phases
|
||||
|
||||
**G0. The unseen-repository pool and the push-level metric** (new spec `unseen-repo-evaluation`)
|
||||
|
||||
1. A frozen pool of repositories never used for training, tuning, mining or harvesting, and not in any population.
|
||||
Two kinds of change per repository: real vulnerability-introducing commits (the parent of a fix, post-cutoff where
|
||||
possible) and ordinary commits from the same history.
|
||||
2. The metric: replay each change through `ousast pre-push` (and the decision engine where an arm uses it) and
|
||||
record catch rate and false-alarm rate, with confidence intervals, per family and overall.
|
||||
3. Rotation: the pool is spent per decision, like the populations; a new slice is drawn for the next one.
|
||||
|
||||
Exit: the pool is frozen and the current production hook has a measured baseline on it.
|
||||
|
||||
**G1. Re-score the arms we have on the pool**
|
||||
|
||||
Current hook (quick rules + engine), decision-engine advisory (current programs), combiner, engine evidence. Each is
|
||||
an arm with a registered decision rule, scored by catch rate at a fixed false-alarm budget.
|
||||
|
||||
Exit: one measured table of arms on unseen code; keep only what generalises.
|
||||
|
||||
**G2. Improve what moves the unseen-repo numbers**
|
||||
|
||||
Candidates, each admitted only if it improves the pool metric on a fresh slice: fix mechanisms in the engine
|
||||
(approved plan, now one arm, not the spine), engine coverage and TypeScript, decision-engine prompt and retrieval
|
||||
changes, more labelled pairs from more distinct repositories.
|
||||
|
||||
**G3. Decide BLOCK per family** on the pool's false-alarm rate and the pooled precision bound (Req 6.4).
|
||||
|
||||
**G4. Detection on push for any repository** (`plane-on-kubernetes` P4 work: `ousast plane scan --base --head`,
|
||||
engine pins on kube-ax, model calls on the VM or CI), sized by what G1-G3 kept.
|
||||
|
||||
**G5. Qualification and rollout:** adopt, commit `prediction-v3.json`, spend population v3 once, advisory rollout,
|
||||
BLOCK only for families that passed.
|
||||
|
||||
### Cross-cutting rules
|
||||
|
||||
- No hand-tuning per miss. Every detection change is a registered experiment or an admission by held-out
|
||||
separation.
|
||||
- Records carry counts, digests and pattern names, never repository names: the reserved-repository guard covers
|
||||
every population.
|
||||
- Instrument first: a zero from an unread input is a failed run (AGENTS.md).
|
||||
- Cluster stability: sandbox deaths are retried once, then sent to the VM lane; the operator owns workers, egress
|
||||
and Substrate.
|
||||
|
||||
### Spec map
|
||||
|
||||
| Spec | Role in this roadmap | State |
|
||||
| --- | --- | --- |
|
||||
| `unseen-repo-evaluation` (new) | G0, G1: the pool and the push-level metric | To specify |
|
||||
| `learned-decision-engine` | G1-G3, G5; fix-mechanisms plan is one G2 arm | Implementation; experiments 3-5 recorded |
|
||||
| `plane-on-kubernetes` | G4; the engine executors for kube-ax | Groups 1-2 done (kind); production path re-scoped to the operator's kube-ax (AX Tasks), groups 3-8 to re-plan |
|
||||
| `pre-push-safety-net` | the hook replayed in G0-G1; M4-M5 in G5 | Operational increment 1 done; M4-M5 open |
|
||||
| `ai-service-plane` | The plane the above runs on | Complete |
|
||||
|
||||
## Status, 2026-10-02
|
||||
|
||||
The milestone table below dates from 2026-09-14. Since then (`.kiro/specs/pre-push-safety-net/release-milestones.md`):
|
||||
|
||||
@@ -1,434 +0,0 @@
|
||||
"""Host-only AX Task transport; credentials never enter task manifests."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import ipaddress
|
||||
import json
|
||||
import os
|
||||
import queue
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
from collections import Counter, deque
|
||||
from concurrent.futures import FIRST_COMPLETED, ThreadPoolExecutor, wait
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import timedelta
|
||||
from pathlib import Path
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
import yaml
|
||||
|
||||
from openultrasast.config import load_dotenv
|
||||
from openultrasast.plane.memory import PRESIGNED_ARCHIVE_SUFFIX, S3Store, open_store
|
||||
|
||||
|
||||
class EgressError(ValueError):
|
||||
"""Safe diagnostic without presigned URL contents."""
|
||||
|
||||
|
||||
class SandboxFailure(Exception):
|
||||
"""A terminal AX task without an authoritative result object."""
|
||||
|
||||
|
||||
def public_endpoint(url):
|
||||
"""Reject endpoints the actor's hostname-based HTTPS egress policy cannot reach."""
|
||||
parts = urlsplit(url)
|
||||
host = parts.hostname or ""
|
||||
try:
|
||||
ipaddress.ip_address(host)
|
||||
is_ip = True
|
||||
except ValueError:
|
||||
is_ip = False
|
||||
if (
|
||||
parts.scheme != "https"
|
||||
or not host
|
||||
or is_ip
|
||||
or "." not in host
|
||||
or host.endswith((".localhost", ".local", ".internal"))
|
||||
or parts.port not in (None, 443)
|
||||
or parts.username
|
||||
or parts.password
|
||||
):
|
||||
raise EgressError(
|
||||
"AX requires a public HTTPS DNS hostname in S3_ENDPOINT and presigned URLs: "
|
||||
"the egress policy allows the store hostname on port 443 only (no IP or plain HTTP)"
|
||||
)
|
||||
return host.lower()
|
||||
|
||||
|
||||
def task_state(text, name):
|
||||
"""Accept AX YAML/JSON output and its human-readable tasks table."""
|
||||
try:
|
||||
doc = yaml.safe_load(text)
|
||||
except yaml.YAMLError:
|
||||
doc = None
|
||||
rows = doc.get("items", doc.get("tasks", [doc])) if isinstance(doc, dict) else doc if isinstance(doc, list) else []
|
||||
for row in rows:
|
||||
if not isinstance(row, dict) or row.get("metadata", {}).get("name", row.get("name")) != name:
|
||||
continue
|
||||
status = row.get("status", {})
|
||||
ip = next(
|
||||
(status.get(key) or row.get(key) for key in ("workerIP", "workerIp", "worker_ip", "ip") if status.get(key) or row.get(key)),
|
||||
None,
|
||||
)
|
||||
worker = status.get("worker")
|
||||
if not ip and isinstance(worker, dict):
|
||||
ip = worker.get("ip") or worker.get("address")
|
||||
return str(status.get("phase", row.get("phase", ""))), ip
|
||||
lines = [line.split() for line in text.splitlines() if line.strip()]
|
||||
if lines:
|
||||
headers = [re.sub(r"[^a-z]", "", cell.lower()) for cell in lines[0]]
|
||||
for cells in lines[1:]:
|
||||
if name not in cells:
|
||||
continue
|
||||
row = dict(zip(headers, cells, strict=False))
|
||||
phase = row.get("phase", row.get("status", ""))
|
||||
ip = row.get("workerip") or row.get("ip")
|
||||
if not ip:
|
||||
for cell in cells:
|
||||
try:
|
||||
ipaddress.ip_address(cell)
|
||||
ip = cell
|
||||
break
|
||||
except ValueError:
|
||||
pass
|
||||
return phase, None if ip in {"<none>", "-", ""} else ip
|
||||
listed = isinstance(doc, (dict, list)) or bool(lines and "name" in headers and {"phase", "status"}.intersection(headers))
|
||||
return "Missing" if listed else "", None
|
||||
|
||||
|
||||
def bounded_read(read, timeout):
|
||||
"""Do not let SDK retries postpone task teardown beyond the pin deadline.
|
||||
|
||||
Only the read runs in a daemon thread; it cannot publish a late result to disk.
|
||||
No executor shutdown waits for an SDK call whose socket has stopped responding.
|
||||
"""
|
||||
if timeout <= 0:
|
||||
raise TimeoutError("AX result deadline")
|
||||
replies = queue.Queue(maxsize=1)
|
||||
|
||||
def fetch():
|
||||
try:
|
||||
replies.put((True, read()))
|
||||
except Exception as exc:
|
||||
replies.put((False, exc))
|
||||
|
||||
threading.Thread(target=fetch, daemon=True, name="engine-ax-result-read").start()
|
||||
try:
|
||||
ok, result = replies.get(timeout=timeout)
|
||||
except queue.Empty:
|
||||
raise TimeoutError("AX result deadline") from None
|
||||
if not ok:
|
||||
raise result
|
||||
return result
|
||||
|
||||
|
||||
class AXLane:
|
||||
def __init__(self, args, workload, store=None, pause=time.sleep, clock=time.monotonic):
|
||||
self.workload = workload
|
||||
self.args = args
|
||||
self.pause, self.clock = pause, clock
|
||||
self.run = uuid.uuid4().hex
|
||||
load_dotenv()
|
||||
self.host = public_endpoint(os.environ.get("S3_ENDPOINT", ""))
|
||||
# Validate the image before opening a store, whose constructor probes the bucket.
|
||||
validate_image(workload.image)
|
||||
self.store = store if store is not None else open_store()
|
||||
if store is None and not isinstance(self.store, S3Store):
|
||||
raise ValueError("AX requires OUSAST_MEMORY=s3://bucket[/prefix]")
|
||||
|
||||
def __call__(self, item, output, attempt):
|
||||
attempts = []
|
||||
try:
|
||||
done, worker = self._execute_attempt(item, output, attempt, attempts)
|
||||
except SandboxFailure:
|
||||
return {"status": "sandbox_failure", "ax_attempts": attempts}
|
||||
if done is None or done.returncode:
|
||||
return {"status": "instrument_failure", "ax_attempts": attempts}
|
||||
record = json.loads((output / "result.json").read_text())
|
||||
return {**record, "worker_ip": worker, "ax_attempts": attempts}
|
||||
|
||||
def ax(self, *args, manifest=None, timeout=30):
|
||||
env = dict(os.environ)
|
||||
if self.args.kubeconfig:
|
||||
env["KUBECONFIG"] = str(Path(self.args.kubeconfig).expanduser())
|
||||
done = subprocess.run(
|
||||
[str(Path(self.args.ax_bin).expanduser()), *args, *(["--atespace", self.args.atespace] if args[0] != "apply" else [])],
|
||||
input=json.dumps(manifest) if manifest is not None else None,
|
||||
env=env,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=max(0.01, timeout),
|
||||
)
|
||||
if done.returncode:
|
||||
# AX errors may echo manifest URLs; report only verb and exit code.
|
||||
raise ValueError(f"ax {args[0]} failed (exit {done.returncode})")
|
||||
return done
|
||||
|
||||
def _execute_attempt(self, item, output, attempt, attempts):
|
||||
pin_id = item.id
|
||||
digest = hashlib.sha256(f"{self.run}/{pin_id}/attempt-{attempt}".encode()).hexdigest()[:12]
|
||||
slug = re.sub(r"[^a-z0-9]+", "-", str(pin_id).lower()).strip("-")[:22]
|
||||
name = f"ousast-engine-{slug or 'pin'}-{digest}"
|
||||
key_id = hashlib.sha256(str(pin_id).encode()).hexdigest()
|
||||
prefix = f"engine-queue/{self.run}/{key_id}/attempt-{attempt}/"
|
||||
keys = [prefix + item.object_names.get(key, key.lower() + PRESIGNED_ARCHIVE_SUFFIX) for key in item.inputs]
|
||||
keys.append(prefix + item.result_object)
|
||||
expires = timedelta(seconds=self.workload.deadline + 600)
|
||||
worker_ip = None
|
||||
submitted = False
|
||||
started = self.clock()
|
||||
phase = ""
|
||||
try:
|
||||
for key, data in zip(keys, item.inputs.values(), strict=False):
|
||||
self.store._put(key, data)
|
||||
urls = {env: self.store.presign_get(key, expires) for env, key in zip(item.inputs, keys, strict=False)}
|
||||
urls["RESULT_URL"] = self.store.presign_put(keys[-1], expires)
|
||||
if any(public_endpoint(url) != self.host for url in urls.values()):
|
||||
raise EgressError("AX presign hostname must match S3_ENDPOINT for the store egress policy")
|
||||
document = manifest(name, self.workload, item, urls, self.args.atespace)
|
||||
end = self.clock() + self.workload.deadline
|
||||
# An apply timeout can still have created the task; cleanup must cover it.
|
||||
submitted = True
|
||||
self.ax("apply", "-f", "-", manifest=document, timeout=min(30, self.workload.deadline))
|
||||
while self.clock() < end:
|
||||
result = bounded_read(lambda: self.store._get(keys[-1]), end - self.clock())
|
||||
if self.clock() >= end:
|
||||
return None, worker_ip
|
||||
phase = ""
|
||||
try:
|
||||
state = self.ax("get", "tasks", timeout=min(30, end - self.clock())).stdout
|
||||
phase, ip = task_state(state, name)
|
||||
worker_ip = ip or worker_ip
|
||||
except (ValueError, OSError, subprocess.SubprocessError):
|
||||
# The object is authoritative even when AX status is temporarily unavailable.
|
||||
pass
|
||||
if self.clock() >= end:
|
||||
return None, worker_ip
|
||||
if result is not None:
|
||||
self.workload.validate(result[0], output)
|
||||
return subprocess.CompletedProcess([], 0, "", ""), worker_ip
|
||||
if phase.lower() in {"failed", "missing"}:
|
||||
# The worker may have uploaded while we were reading AX status.
|
||||
result = bounded_read(lambda: self.store._get(keys[-1]), end - self.clock())
|
||||
if self.clock() >= end:
|
||||
return None, worker_ip
|
||||
if result is not None:
|
||||
self.workload.validate(result[0], output)
|
||||
return subprocess.CompletedProcess([], 0, "", ""), worker_ip
|
||||
raise SandboxFailure
|
||||
self.pause(min(2, max(0, end - self.clock())))
|
||||
return None, worker_ip
|
||||
except (subprocess.TimeoutExpired, TimeoutError):
|
||||
return None, worker_ip
|
||||
except SandboxFailure:
|
||||
raise
|
||||
except Exception as exc:
|
||||
# Never expose exception strings which may contain object-scoped bearer URLs.
|
||||
detail = str(exc) if isinstance(exc, EgressError) else type(exc).__name__
|
||||
return subprocess.CompletedProcess([], 1, "", "AX transport failed: " + detail), worker_ip
|
||||
finally:
|
||||
attempts.append({"name": name, "phase": phase, "seconds": self.clock() - started})
|
||||
errors = []
|
||||
if submitted:
|
||||
try:
|
||||
self.ax("delete", "task", name)
|
||||
except Exception as exc:
|
||||
# A successful list already confirmed a disappeared task is gone.
|
||||
if phase != "Missing":
|
||||
errors.append("task: " + type(exc).__name__)
|
||||
for key in keys:
|
||||
try:
|
||||
self.store._delete(key)
|
||||
except Exception as exc:
|
||||
errors.append("object: " + type(exc).__name__)
|
||||
if errors:
|
||||
raise RuntimeError("AX cleanup failed; stop dispatching: " + ", ".join(errors))
|
||||
|
||||
|
||||
@dataclass
|
||||
class Item:
|
||||
id: str
|
||||
inputs: dict[str, bytes]
|
||||
result_object: str
|
||||
command: tuple[str, ...]
|
||||
extra_env: dict[str, str]
|
||||
input_digest: str
|
||||
object_names: dict[str, str] = field(default_factory=dict)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Workload:
|
||||
image: str
|
||||
url_env: frozenset[str]
|
||||
deadline: float
|
||||
validate: object
|
||||
|
||||
|
||||
def validate_image(image):
|
||||
if not re.fullmatch(r"[^\s@]+@sha256:[0-9a-f]{64}", image):
|
||||
raise ValueError("AX --image must be pinned by sha256 digest")
|
||||
|
||||
|
||||
def manifest(name, workload, item, urls, atespace="default", *, validate_name=True):
|
||||
validate_image(workload.image)
|
||||
if validate_name and (not re.fullmatch(r"ousast-engine-[a-z0-9-]+", name) or len(name.encode()) > 49):
|
||||
raise ValueError("invalid AX task name")
|
||||
if set(urls) != workload.url_env:
|
||||
raise ValueError("unexpected URL environment names")
|
||||
# Extra env is deliberately a small numeric execution contract, never inherited credentials.
|
||||
if set(item.extra_env) - {"DEADLINE", "QUESTION_DEADLINE"} or any(
|
||||
not re.fullmatch(r"[0-9]+(?:\.[0-9]+)?", str(v)) for v in item.extra_env.values()
|
||||
):
|
||||
raise ValueError("unexpected task environment")
|
||||
for url in urls.values():
|
||||
public_endpoint(url)
|
||||
env = [{"name": key, "value": str(value)} for key, value in {**urls, **item.extra_env}.items()]
|
||||
if len(json.dumps(env).encode()) >= 32768:
|
||||
raise ValueError("AX Task env exceeds the 32 KB limit")
|
||||
return {
|
||||
"apiVersion": "ax.io/v1alpha1",
|
||||
"kind": "Task",
|
||||
"metadata": {"name": name, "atespace": atespace},
|
||||
"spec": {"image": workload.image, "env": env, "command": list(item.command)},
|
||||
}
|
||||
|
||||
|
||||
def write_json(path, record):
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
temporary = path.with_suffix(".tmp")
|
||||
temporary.write_text(json.dumps(record, sort_keys=True) + "\n")
|
||||
temporary.replace(path)
|
||||
|
||||
|
||||
def schedule(pending, lanes, out, execute, complete, *, fallbacks=(), overflow=True, progress=None):
|
||||
"""The sole progress writer; callbacks carry workload-specific records, never transport."""
|
||||
if sum(lane in {"ax", "kind"} for lane in lanes) > 2 or lanes.count("docker") > 1:
|
||||
raise ValueError("at most two AX lanes and one VM lane")
|
||||
pending, fallbacks = deque(pending), deque(fallbacks)
|
||||
progress = progress if progress is not None else {"pins_done": 0, "pins_total": len(pending) + len(fallbacks)}
|
||||
progress["lanes"] = dict(Counter(lanes))
|
||||
active = {}
|
||||
failures = False
|
||||
|
||||
def persist():
|
||||
progress.update(pins_in_flight=len(active), active_lanes=dict(Counter(lane for _, lane in active.values())))
|
||||
if any(lane in {"ax", "kind"} for lane in lanes):
|
||||
progress["fallbacks_pending"] = len(fallbacks)
|
||||
write_json(out / "progress.json", progress)
|
||||
|
||||
with ThreadPoolExecutor(max_workers=len(lanes)) as pool:
|
||||
|
||||
def launch(lane):
|
||||
if (out / "STOP").exists():
|
||||
return
|
||||
if lane == "docker" and fallbacks:
|
||||
item, previous = fallbacks.popleft()
|
||||
elif pending and (lane != "docker" or overflow):
|
||||
item, previous = pending.popleft(), None
|
||||
else:
|
||||
return
|
||||
active[pool.submit(execute, item, lane, previous)] = item, lane
|
||||
|
||||
for lane in lanes:
|
||||
launch(lane)
|
||||
persist()
|
||||
try:
|
||||
while active:
|
||||
finished, _ = wait(active, return_when=FIRST_COMPLETED)
|
||||
# Resolve ALL completed futures before launching; cleanup exceptions stop launches.
|
||||
results = [(active.pop(future), future.result()) for future in finished]
|
||||
for (item, lane), result in results:
|
||||
failed, fallback = complete(item, lane, result, progress)
|
||||
if fallback is not None:
|
||||
fallbacks.append((item, fallback))
|
||||
else:
|
||||
failures |= failed
|
||||
progress["pins_done"] += 1
|
||||
for lane, count in (Counter(lanes) - Counter(lane for _, lane in active.values())).items():
|
||||
for _ in range(count):
|
||||
launch(lane)
|
||||
persist()
|
||||
finally:
|
||||
persist()
|
||||
return failures
|
||||
|
||||
|
||||
def dispatch(items, workload, out, executors, *, lanes=("ax", "ax", "docker"), overflow=True):
|
||||
"""Digest checkpoints, one instrument rerun, sandbox retry, priority VM fallback."""
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
pending, fallbacks = [], []
|
||||
|
||||
def path(item):
|
||||
return out / (hashlib.sha256(item.id.encode()).hexdigest() + ".json")
|
||||
|
||||
for item in items:
|
||||
previous = json.loads(path(item).read_text()) if path(item).exists() else {}
|
||||
if previous.get("input_digest") == item.input_digest:
|
||||
if previous.get("fallback_pending"):
|
||||
fallbacks.append((item, previous))
|
||||
continue
|
||||
if previous.get("done"):
|
||||
continue
|
||||
pending.append(item)
|
||||
|
||||
def execute(item, lane, previous):
|
||||
output = out / hashlib.sha256(item.id.encode()).hexdigest()
|
||||
output.mkdir(exist_ok=True)
|
||||
attempts = []
|
||||
for attempt in range(2):
|
||||
attempt_output = output / ("attempt-" + uuid.uuid4().hex)
|
||||
attempt_output.mkdir()
|
||||
record = executors[lane](item, attempt_output, attempt)
|
||||
for artifact in attempt_output.iterdir():
|
||||
if artifact.is_file():
|
||||
shutil.copyfile(artifact, output / artifact.name)
|
||||
attempts.append({"lane": lane, "status": record.get("status"), "attempt": attempt, "transport": record.get("ax_attempts", [])})
|
||||
if record.get("status") not in {"sandbox_failure", "instrument_failure"}:
|
||||
break
|
||||
return {**record, "attempts": (previous or {}).get("attempts", []) + attempts, "input_digest": item.input_digest, "done": True}
|
||||
|
||||
def complete(item, lane, record, progress):
|
||||
fallback = lane in {"ax", "kind"} and record.get("status") in {"sandbox_failure", "oom"} and "docker" in lanes
|
||||
record["fallback_pending"] = fallback
|
||||
write_json(path(item), record)
|
||||
progress["last_item"] = hashlib.sha256(item.id.encode()).hexdigest()
|
||||
return record.get("status") not in {"ok", "unanalysable", "quick_only", "engine_covered"}, record if fallback else None
|
||||
|
||||
return schedule(pending, lanes, out, execute, complete, fallbacks=fallbacks, overflow=overflow)
|
||||
|
||||
|
||||
class DockerLane(AXLane):
|
||||
"""VM lane uses the same presigned-object protocol and pinned replay image."""
|
||||
|
||||
def _execute_attempt(self, item, output, attempt, attempts):
|
||||
# Reuse all object/validation/cleanup logic, replacing only the task runner.
|
||||
import copy
|
||||
|
||||
lane = copy.copy(self)
|
||||
lane.ax = lane._docker_command
|
||||
lane._docker_name = None
|
||||
return AXLane._execute_attempt(lane, item, output, attempt, attempts)
|
||||
|
||||
def _docker_command(self, *args, manifest=None, timeout=30):
|
||||
if args[0] == "apply":
|
||||
self._docker_name = manifest["metadata"]["name"]
|
||||
command = ["docker", "run", "-d", "--name", self._docker_name, "--entrypoint", manifest["spec"]["command"][0]]
|
||||
for env in manifest["spec"]["env"]:
|
||||
command += ["-e", env["name"] + "=" + env["value"]]
|
||||
command += [manifest["spec"]["image"], *manifest["spec"]["command"][1:], "--heap-profile", "vm"]
|
||||
elif args[0] == "get":
|
||||
command = ["docker", "inspect", "--format", "{{.State.Running}} {{.State.ExitCode}}", self._docker_name]
|
||||
else:
|
||||
command = ["docker", "rm", "-f", self._docker_name or args[2]]
|
||||
done = subprocess.run(command, capture_output=True, text=True, timeout=timeout)
|
||||
if done.returncode:
|
||||
raise ValueError("docker command failed")
|
||||
if args[0] == "get":
|
||||
phase = "Running" if done.stdout.startswith("true") else "Failed"
|
||||
done.stdout = json.dumps({"metadata": {"name": self._docker_name}, "status": {"phase": phase}})
|
||||
return done
|
||||
@@ -16,13 +16,13 @@ import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import textwrap
|
||||
import threading
|
||||
import time
|
||||
import tomllib
|
||||
from collections import Counter, defaultdict
|
||||
from collections import Counter, defaultdict, deque
|
||||
from concurrent.futures import FIRST_COMPLETED, ThreadPoolExecutor, wait
|
||||
from pathlib import Path
|
||||
|
||||
from engine_trace_k8s import IMAGE, Kubernetes
|
||||
@@ -39,9 +39,6 @@ from openultrasast.plane.memory import FileStore
|
||||
from openultrasast.plane.tasks.alerts import engine_languages
|
||||
from openultrasast.preprocess import detect_language
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
from benchmarks.ax.batch import schedule
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
DEFAULT_OUT = Path.home() / "ousast-results/plane/engine-trace"
|
||||
SNAPSHOT_FILES = (
|
||||
@@ -50,7 +47,6 @@ SNAPSHOT_FILES = (
|
||||
"benchmarks/learn/engine_trace_k8s.py",
|
||||
"benchmarks/learn/engine_trace_queue.py",
|
||||
"benchmarks/learn/engine_trace_ax.py",
|
||||
"benchmarks/ax/batch.py",
|
||||
"benchmarks/learn/engine_task_entry.py",
|
||||
"benchmarks/learn/engine_worker_service.py",
|
||||
"benchmarks/learn/engine_trace_parse.py",
|
||||
@@ -795,8 +791,8 @@ def run_plan(plan, source, provenance, inputs, args):
|
||||
"active_lanes": {},
|
||||
}
|
||||
counts = Counter()
|
||||
pending = list(plan)
|
||||
fallbacks = []
|
||||
pending = deque(plan)
|
||||
fallbacks = deque()
|
||||
if args.executor in {"ax", "mixed-ax"}:
|
||||
pending.clear()
|
||||
for pin in plan:
|
||||
@@ -807,6 +803,12 @@ def run_plan(plan, source, provenance, inputs, args):
|
||||
fallbacks.append((pin, old))
|
||||
else:
|
||||
pending.append(pin)
|
||||
active = {}
|
||||
failures = False
|
||||
|
||||
def persist():
|
||||
progress.update(pins_in_flight=len(active), active_lanes=dict(Counter(lane for _, lane in active.values())))
|
||||
write_json(args.out / "progress.json", progress)
|
||||
|
||||
def execute(pin, lane, attempt=None):
|
||||
if not any(u["supported"] for u in pin["units"]):
|
||||
@@ -823,20 +825,52 @@ def run_plan(plan, source, provenance, inputs, args):
|
||||
options.image = getattr(args, "docker_image", args.image)
|
||||
return run_pin(pin, source, provenance, inputs, args.out, options)
|
||||
|
||||
def complete(pin, lane, failed, progress):
|
||||
result = json.loads((args.out / pin_name(pin)).read_text())
|
||||
selected = {u["unit"] for u in pin["units"]}
|
||||
reasons = [u.get("reason", "") for u in result["units"] if u["unit"] in selected]
|
||||
sandbox_failed = args.executor == "mixed-ax" and result.get("container_exit") == 1 and "AX sandbox failed twice" in reasons
|
||||
if lane == "ax" and (sandbox_failed or any("JVM out of memory" in reason for reason in reasons)):
|
||||
result.update(fallback="vm-sandbox" if sandbox_failed else "vm-oom", fallback_pending=True)
|
||||
write_json(args.out / pin_name(pin), result)
|
||||
return False, result
|
||||
counts.update(u["status"] for u in result["units"] if u["unit"] in selected)
|
||||
progress.update(last_pin={"repo": pin["repo"], "pin": pin["pin"]}, statuses=dict(counts))
|
||||
return failed, None
|
||||
with ThreadPoolExecutor(max_workers=len(lanes)) as pool:
|
||||
|
||||
return schedule(pending, lanes, args.out, execute, complete, fallbacks=fallbacks, overflow=args.executor != "ax", progress=progress)
|
||||
def launch(lane):
|
||||
if (args.out / "STOP").exists():
|
||||
return
|
||||
attempt = None
|
||||
if lane == "docker" and fallbacks:
|
||||
pin, attempt = fallbacks.popleft()
|
||||
elif pending and not (args.executor == "ax" and lane == "docker"):
|
||||
pin = pending.popleft()
|
||||
else:
|
||||
return
|
||||
active[pool.submit(execute, pin, lane, attempt)] = (pin, lane)
|
||||
|
||||
for lane in lanes:
|
||||
launch(lane)
|
||||
persist()
|
||||
while active:
|
||||
completed, _ = wait(active, return_when=FIRST_COMPLETED)
|
||||
for future in completed:
|
||||
pin, lane = active.pop(future)
|
||||
failed = future.result()
|
||||
result = json.loads((args.out / pin_name(pin)).read_text())
|
||||
selected = {u["unit"] for u in pin["units"]}
|
||||
reasons = [u.get("reason", "") for u in result["units"] if u["unit"] in selected]
|
||||
sandbox_failed = args.executor == "mixed-ax" and result.get("container_exit") == 1 and "AX sandbox failed twice" in reasons
|
||||
if lane == "ax" and (sandbox_failed or any("JVM out of memory" in reason for reason in reasons)):
|
||||
result.update(fallback="vm-sandbox" if sandbox_failed else "vm-oom", fallback_pending=True)
|
||||
write_json(args.out / pin_name(pin), result)
|
||||
fallbacks.append((pin, result))
|
||||
continue
|
||||
failures |= failed
|
||||
selected = {u["unit"] for u in pin["units"]}
|
||||
counts.update(u["status"] for u in result["units"] if u["unit"] in selected)
|
||||
progress.update(
|
||||
pins_done=progress["pins_done"] + 1, last_pin={"repo": pin["repo"], "pin": pin["pin"]}, statuses=dict(counts)
|
||||
)
|
||||
idle = Counter(lanes) - Counter(lane for _, lane in active.values())
|
||||
# Give a waiting fallback priority over overflow on the single VM lane.
|
||||
for lane, count in idle.items():
|
||||
for _ in range(count):
|
||||
launch(lane)
|
||||
if args.executor in {"ax", "mixed-ax"}:
|
||||
progress["fallbacks_pending"] = len(fallbacks)
|
||||
persist()
|
||||
return failures
|
||||
|
||||
|
||||
def main():
|
||||
|
||||
@@ -1,50 +1,184 @@
|
||||
"""Engine workload adapter for the shared AX transport."""
|
||||
"""Host-only AX Task transport; credentials never enter task manifests."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import io
|
||||
import ipaddress
|
||||
import json
|
||||
import os
|
||||
import queue
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
from datetime import timedelta
|
||||
from pathlib import Path
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
import yaml
|
||||
from engine_trace_k8s import pack
|
||||
|
||||
from benchmarks.ax.batch import AXLane, Item, SandboxFailure, Workload, manifest
|
||||
from benchmarks.ax.batch import task_state as task_state
|
||||
from openultrasast.plane.memory import open_store as open_store
|
||||
from openultrasast.config import load_dotenv
|
||||
from openultrasast.plane.memory import PRESIGNED_ARCHIVE_SUFFIX, S3Store, open_store
|
||||
|
||||
|
||||
class EgressError(ValueError):
|
||||
"""Safe diagnostic without presigned URL contents."""
|
||||
|
||||
|
||||
class SandboxFailure(Exception):
|
||||
"""A terminal AX task without an authoritative result object."""
|
||||
|
||||
|
||||
def public_endpoint(url):
|
||||
"""Reject endpoints the actor's hostname-based HTTPS egress policy cannot reach."""
|
||||
parts = urlsplit(url)
|
||||
host = parts.hostname or ""
|
||||
try:
|
||||
ipaddress.ip_address(host)
|
||||
is_ip = True
|
||||
except ValueError:
|
||||
is_ip = False
|
||||
if (
|
||||
parts.scheme != "https"
|
||||
or not host
|
||||
or is_ip
|
||||
or "." not in host
|
||||
or host.endswith((".localhost", ".local", ".internal"))
|
||||
or parts.port not in (None, 443)
|
||||
or parts.username
|
||||
or parts.password
|
||||
):
|
||||
raise EgressError(
|
||||
"AX requires a public HTTPS DNS hostname in S3_ENDPOINT and presigned URLs: "
|
||||
"the egress policy allows the store hostname on port 443 only (no IP or plain HTTP)"
|
||||
)
|
||||
return host.lower()
|
||||
|
||||
|
||||
def task_manifest(name, args, urls):
|
||||
return manifest(
|
||||
name, engine_workload(args, lambda *_: None), engine_item({}, "pin", b"", args), urls, args.atespace, validate_name=False
|
||||
if not re.fullmatch(r"[^\s@]+@sha256:[0-9a-f]{64}", args.image):
|
||||
raise ValueError("AX --image must be pinned by sha256 digest")
|
||||
if set(urls) != {"SOURCE_URL", "QUESTIONS_URL", "RESULT_URL"}:
|
||||
raise ValueError("AX Task accepts only source, questions and result presigned URLs")
|
||||
for url in urls.values():
|
||||
public_endpoint(url)
|
||||
env = [{"name": key, "value": value} for key, value in urls.items()]
|
||||
env.extend(
|
||||
{"name": key, "value": str(value)}
|
||||
for key, value in {"DEADLINE": args.deadline, "QUESTION_DEADLINE": args.question_deadline}.items()
|
||||
)
|
||||
if len(json.dumps(env).encode()) >= 32768:
|
||||
raise ValueError("AX Task env exceeds the 32 KB limit")
|
||||
return {
|
||||
"apiVersion": "ax.io/v1alpha1",
|
||||
"kind": "Task",
|
||||
"metadata": {"name": name, "atespace": args.atespace},
|
||||
"spec": {"image": args.image, "env": env, "command": ["python3", "/app/benchmarks/learn/engine_task_entry.py"]},
|
||||
}
|
||||
|
||||
|
||||
def engine_workload(args, validate):
|
||||
return Workload(args.image, frozenset({"SOURCE_URL", "QUESTIONS_URL", "RESULT_URL"}), args.deadline, validate)
|
||||
def task_state(text, name):
|
||||
"""Accept AX YAML/JSON output and its human-readable tasks table."""
|
||||
try:
|
||||
doc = yaml.safe_load(text)
|
||||
except yaml.YAMLError:
|
||||
doc = None
|
||||
rows = doc.get("items", doc.get("tasks", [doc])) if isinstance(doc, dict) else doc if isinstance(doc, list) else []
|
||||
for row in rows:
|
||||
if not isinstance(row, dict) or row.get("metadata", {}).get("name", row.get("name")) != name:
|
||||
continue
|
||||
status = row.get("status", {})
|
||||
ip = next(
|
||||
(status.get(key) or row.get(key) for key in ("workerIP", "workerIp", "worker_ip", "ip") if status.get(key) or row.get(key)),
|
||||
None,
|
||||
)
|
||||
worker = status.get("worker")
|
||||
if not ip and isinstance(worker, dict):
|
||||
ip = worker.get("ip") or worker.get("address")
|
||||
return str(status.get("phase", row.get("phase", ""))), ip
|
||||
lines = [line.split() for line in text.splitlines() if line.strip()]
|
||||
if lines:
|
||||
headers = [re.sub(r"[^a-z]", "", cell.lower()) for cell in lines[0]]
|
||||
for cells in lines[1:]:
|
||||
if name not in cells:
|
||||
continue
|
||||
row = dict(zip(headers, cells, strict=False))
|
||||
phase = row.get("phase", row.get("status", ""))
|
||||
ip = row.get("workerip") or row.get("ip")
|
||||
if not ip:
|
||||
for cell in cells:
|
||||
try:
|
||||
ipaddress.ip_address(cell)
|
||||
ip = cell
|
||||
break
|
||||
except ValueError:
|
||||
pass
|
||||
return phase, None if ip in {"<none>", "-", ""} else ip
|
||||
listed = isinstance(doc, (dict, list)) or bool(lines and "name" in headers and {"phase", "status"}.intersection(headers))
|
||||
return "Missing" if listed else "", None
|
||||
|
||||
|
||||
def engine_item(pin, pin_id, source, args):
|
||||
return Item(
|
||||
pin_id,
|
||||
{"SOURCE_URL": source, "QUESTIONS_URL": json.dumps(pin).encode()},
|
||||
"result.dat",
|
||||
("python3", "/app/benchmarks/learn/engine_task_entry.py"),
|
||||
{"DEADLINE": str(args.deadline), "QUESTION_DEADLINE": str(args.question_deadline)},
|
||||
"",
|
||||
{"SOURCE_URL": "source.dat", "QUESTIONS_URL": "questions.json"},
|
||||
)
|
||||
def bounded_read(read, timeout):
|
||||
"""Do not let SDK retries postpone task teardown beyond the pin deadline.
|
||||
|
||||
Only the read runs in a daemon thread; it cannot publish a late result to disk.
|
||||
No executor shutdown waits for an SDK call whose socket has stopped responding.
|
||||
"""
|
||||
if timeout <= 0:
|
||||
raise TimeoutError("AX result deadline")
|
||||
replies = queue.Queue(maxsize=1)
|
||||
|
||||
def fetch():
|
||||
try:
|
||||
replies.put((True, read()))
|
||||
except Exception as exc:
|
||||
replies.put((False, exc))
|
||||
|
||||
threading.Thread(target=fetch, daemon=True, name="engine-ax-result-read").start()
|
||||
try:
|
||||
ok, result = replies.get(timeout=timeout)
|
||||
except queue.Empty:
|
||||
raise TimeoutError("AX result deadline") from None
|
||||
if not ok:
|
||||
raise result
|
||||
return result
|
||||
|
||||
|
||||
class AX(AXLane):
|
||||
class AX:
|
||||
def __init__(self, args, store=None, pause=time.sleep, clock=time.monotonic):
|
||||
super().__init__(args, engine_workload(args, lambda *_: None), store, pause, clock)
|
||||
self.args = args
|
||||
self.pause, self.clock = pause, clock
|
||||
self.run = uuid.uuid4().hex
|
||||
load_dotenv()
|
||||
self.host = public_endpoint(os.environ.get("S3_ENDPOINT", ""))
|
||||
# Validate the image before opening a store, whose constructor probes the bucket.
|
||||
task_manifest("validate", args, {key: f"https://{self.host}/" for key in ("SOURCE_URL", "QUESTIONS_URL", "RESULT_URL")})
|
||||
self.store = store if store is not None else open_store()
|
||||
if store is None and not isinstance(self.store, S3Store):
|
||||
raise ValueError("AX requires OUSAST_MEMORY=s3://bucket[/prefix]")
|
||||
|
||||
def ax(self, *args, manifest=None, timeout=30):
|
||||
env = dict(os.environ)
|
||||
if self.args.kubeconfig:
|
||||
env["KUBECONFIG"] = str(Path(self.args.kubeconfig).expanduser())
|
||||
done = subprocess.run(
|
||||
[str(Path(self.args.ax_bin).expanduser()), *args, *(["--atespace", self.args.atespace] if args[0] != "apply" else [])],
|
||||
input=json.dumps(manifest) if manifest is not None else None,
|
||||
env=env,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=max(0.01, timeout),
|
||||
)
|
||||
if done.returncode:
|
||||
# AX errors may echo manifest URLs; report only verb and exit code.
|
||||
raise ValueError(f"ax {args[0]} failed (exit {done.returncode})")
|
||||
return done
|
||||
|
||||
def receive(self, data, output, pin):
|
||||
# Validate in isolation: malformed results must never become usable checkpoints.
|
||||
@@ -86,15 +220,11 @@ class AX(AXLane):
|
||||
shutil.copytree(root, output, dirs_exist_ok=True)
|
||||
|
||||
def execute(self, checkout, output, pin, pin_id):
|
||||
import copy
|
||||
|
||||
self = copy.copy(self)
|
||||
self.workload = engine_workload(self.args, lambda data, target: self.receive(data, target, pin))
|
||||
attempts = []
|
||||
try:
|
||||
for attempt in range(2):
|
||||
try:
|
||||
return self._execute_attempt(engine_item(pin, pin_id, pack(checkout), self.args), output, attempt, attempts)
|
||||
return self._execute_attempt(checkout, output, pin, pin_id, attempt, attempts)
|
||||
except SandboxFailure:
|
||||
if attempt == 0:
|
||||
self.pause(10)
|
||||
@@ -106,3 +236,84 @@ class AX(AXLane):
|
||||
record = json.loads(result_path.read_text())
|
||||
record["ax_attempts"] = attempts
|
||||
result_path.write_text(json.dumps(record))
|
||||
|
||||
def _execute_attempt(self, checkout, output, pin, pin_id, attempt, attempts):
|
||||
digest = hashlib.sha256(f"{self.run}/{pin_id}/attempt-{attempt}".encode()).hexdigest()[:12]
|
||||
slug = re.sub(r"[^a-z0-9]+", "-", str(pin_id).lower()).strip("-")[:22]
|
||||
name = f"ousast-engine-{slug or 'pin'}-{digest}"
|
||||
key_id = hashlib.sha256(str(pin_id).encode()).hexdigest()
|
||||
prefix = f"engine-queue/{self.run}/{key_id}/attempt-{attempt}/"
|
||||
keys = [prefix + suffix for suffix in (f"source{PRESIGNED_ARCHIVE_SUFFIX}", "questions.json", f"result{PRESIGNED_ARCHIVE_SUFFIX}")]
|
||||
expires = timedelta(seconds=self.args.deadline + 600)
|
||||
worker_ip = None
|
||||
submitted = False
|
||||
started = self.clock()
|
||||
phase = ""
|
||||
try:
|
||||
self.store._put(keys[0], pack(checkout))
|
||||
self.store._put(keys[1], json.dumps(pin).encode())
|
||||
urls = {
|
||||
"SOURCE_URL": self.store.presign_get(keys[0], expires),
|
||||
"QUESTIONS_URL": self.store.presign_get(keys[1], expires),
|
||||
"RESULT_URL": self.store.presign_put(keys[2], expires),
|
||||
}
|
||||
if any(public_endpoint(url) != self.host for url in urls.values()):
|
||||
raise EgressError("AX presign hostname must match S3_ENDPOINT for the store egress policy")
|
||||
manifest = task_manifest(name, self.args, urls)
|
||||
end = self.clock() + self.args.deadline
|
||||
# An apply timeout can still have created the task; cleanup must cover it.
|
||||
submitted = True
|
||||
self.ax("apply", "-f", "-", manifest=manifest, timeout=min(30, self.args.deadline))
|
||||
while self.clock() < end:
|
||||
result = bounded_read(lambda: self.store._get(keys[2]), end - self.clock())
|
||||
if self.clock() >= end:
|
||||
return None, worker_ip
|
||||
phase = ""
|
||||
try:
|
||||
state = self.ax("get", "tasks", timeout=min(30, end - self.clock())).stdout
|
||||
phase, ip = task_state(state, name)
|
||||
worker_ip = ip or worker_ip
|
||||
except (ValueError, OSError, subprocess.SubprocessError):
|
||||
# The object is authoritative even when AX status is temporarily unavailable.
|
||||
pass
|
||||
if self.clock() >= end:
|
||||
return None, worker_ip
|
||||
if result is not None:
|
||||
self.receive(result[0], output, pin)
|
||||
return subprocess.CompletedProcess([], 0, "", ""), worker_ip
|
||||
if phase.lower() in {"failed", "missing"}:
|
||||
# The worker may have uploaded while we were reading AX status.
|
||||
result = bounded_read(lambda: self.store._get(keys[2]), end - self.clock())
|
||||
if self.clock() >= end:
|
||||
return None, worker_ip
|
||||
if result is not None:
|
||||
self.receive(result[0], output, pin)
|
||||
return subprocess.CompletedProcess([], 0, "", ""), worker_ip
|
||||
raise SandboxFailure
|
||||
self.pause(min(2, max(0, end - self.clock())))
|
||||
return None, worker_ip
|
||||
except (subprocess.TimeoutExpired, TimeoutError):
|
||||
return None, worker_ip
|
||||
except SandboxFailure:
|
||||
raise
|
||||
except Exception as exc:
|
||||
# Never expose exception strings which may contain object-scoped bearer URLs.
|
||||
detail = str(exc) if isinstance(exc, EgressError) else type(exc).__name__
|
||||
return subprocess.CompletedProcess([], 1, "", "AX transport failed: " + detail), worker_ip
|
||||
finally:
|
||||
attempts.append({"name": name, "phase": phase, "seconds": self.clock() - started})
|
||||
errors = []
|
||||
if submitted:
|
||||
try:
|
||||
self.ax("delete", "task", name)
|
||||
except Exception as exc:
|
||||
# A successful list already confirmed a disappeared task is gone.
|
||||
if phase != "Missing":
|
||||
errors.append("task: " + type(exc).__name__)
|
||||
for key in keys:
|
||||
try:
|
||||
self.store._delete(key)
|
||||
except Exception as exc:
|
||||
errors.append("object: " + type(exc).__name__)
|
||||
if errors:
|
||||
raise RuntimeError("AX cleanup failed; stop dispatching: " + ", ".join(errors))
|
||||
|
||||
@@ -1,86 +0,0 @@
|
||||
{
|
||||
"count": 1029,
|
||||
"guard": "green",
|
||||
"sha256": "5a644a1e93c68a0a35c794e653286d0532b7eb8d757123ccce5f08b11bf9a4a2",
|
||||
"sources": {
|
||||
"harvest": {
|
||||
"bytes": 501031,
|
||||
"count": 467,
|
||||
"files": 3,
|
||||
"known_empty": false,
|
||||
"rows": 446,
|
||||
"skipped_values": 446
|
||||
},
|
||||
"measurements": {
|
||||
"bytes": 21169903,
|
||||
"count": 278,
|
||||
"files": 221,
|
||||
"known_empty": false,
|
||||
"rows": 376,
|
||||
"skipped_values": 681
|
||||
},
|
||||
"memory_local": {
|
||||
"bytes": 8084523,
|
||||
"count": 192,
|
||||
"files": 776,
|
||||
"known_empty": false,
|
||||
"rows": 759,
|
||||
"skipped_values": 2
|
||||
},
|
||||
"memory_s3": {
|
||||
"bytes": 15928676,
|
||||
"count": 572,
|
||||
"files": 1499,
|
||||
"known_empty": false,
|
||||
"rows": 1490,
|
||||
"skipped_values": 2
|
||||
},
|
||||
"pairs": {
|
||||
"bytes": 1680480,
|
||||
"count": 728,
|
||||
"files": 28,
|
||||
"known_empty": false,
|
||||
"rows": 900,
|
||||
"skipped_values": 1260
|
||||
},
|
||||
"pointers": {
|
||||
"bytes": 892448,
|
||||
"count": 537,
|
||||
"files": 7,
|
||||
"known_empty": false,
|
||||
"rows": 586,
|
||||
"skipped_values": 666
|
||||
},
|
||||
"populations": {
|
||||
"bytes": 32431,
|
||||
"count": 28,
|
||||
"files": 2,
|
||||
"known_empty": false,
|
||||
"rows": 28,
|
||||
"skipped_values": 0
|
||||
},
|
||||
"sweep": {
|
||||
"bytes": 34006439,
|
||||
"count": 1029,
|
||||
"files": 1619,
|
||||
"known_empty": false,
|
||||
"rows": 2155,
|
||||
"skipped_values": 0
|
||||
},
|
||||
"units": {
|
||||
"bytes": 319641,
|
||||
"count": 229,
|
||||
"files": 9,
|
||||
"known_empty": false,
|
||||
"rows": 642,
|
||||
"skipped_values": 0
|
||||
}
|
||||
},
|
||||
"test_exit_code": 0,
|
||||
"tests": {
|
||||
"errors": 0,
|
||||
"failures": 0,
|
||||
"skipped": 1,
|
||||
"tests": 41
|
||||
}
|
||||
}
|
||||
@@ -1,41 +0,0 @@
|
||||
{
|
||||
"commit": "48ae918dec25119142ae781607ba99306ae5ec17",
|
||||
"fixture_bytes": 15374,
|
||||
"fixture_sha256": "dd3c774a47d553105ed85b9cea8810736c233f870e5a015559a1524d5db5a14d",
|
||||
"implementation_sha256": {
|
||||
"benchmarks/ax/batch.py": "37aecac93beb25c02826e81eacaf064fbebb447a998fe139f6d94a07035b8685",
|
||||
"benchmarks/unseen/replay.py": "32efb3e0b9e9d247f2780b5f3ceeb01b4c2cbda3656c75a741f9925621cfdfbc",
|
||||
"benchmarks/unseen/replay_entry.py": "994d8e239eb505dc2c5bad4cdb8b5d45967a8a2aa9ef48fb4c2f20ba45736280",
|
||||
"benchmarks/unseen/score.py": "56e0a7b34482b5d84df39824b8ee447c283e35bd932f38cfe692cf84cb52cef7"
|
||||
},
|
||||
"network_proofs": {
|
||||
"4.4": "coordinator_pending",
|
||||
"5.4": "coordinator_pending"
|
||||
},
|
||||
"scope": "offline synthetic scorer only",
|
||||
"score_command_exit": 0,
|
||||
"scorer_output_sha256": "77f154eedb6a4acd098ab63a0b17434291f0d226dabe8ac66f2e0b362154d002",
|
||||
"scorer_test_exit": 0,
|
||||
"scorer_tests": 14,
|
||||
"synthetic_records": 20,
|
||||
"uncommitted": true,
|
||||
"validation": {
|
||||
"generic_dispatcher_tests": {
|
||||
"exit_code": 0,
|
||||
"passed": 5
|
||||
},
|
||||
"mypy": 0,
|
||||
"proof_and_scorer_tests": {
|
||||
"exit_code": 0,
|
||||
"passed": 16
|
||||
},
|
||||
"requested_pytest": {
|
||||
"exit_code": 0,
|
||||
"passed": 279,
|
||||
"skipped": 8
|
||||
},
|
||||
"reserved_repository_guard": 0,
|
||||
"ruff_check": 0,
|
||||
"ruff_format_check": 0
|
||||
}
|
||||
}
|
||||
@@ -1,312 +0,0 @@
|
||||
# Unseen evaluation, group 1
|
||||
|
||||
The coordinator runs task 1.5 from the repository root:
|
||||
|
||||
```sh
|
||||
export PATH=/home/mc/Source/OpenUltraSAST/.venv/bin:$PATH
|
||||
export PYTHONPATH=$PWD/src
|
||||
python -m benchmarks.unseen.eligibility --s3-store s3:// --verify-tests --output benchmarks/measurements/2026-10-04-unseen-01-used-set/record.json
|
||||
```
|
||||
|
||||
This is the networked command; implementation tests use fakes. It loads `.env` with
|
||||
`openultrasast.config.load_dotenv` (exported values win), opens the local FileStore and
|
||||
S3 MemoryStore through `openultrasast.plane.memory.open_store(..., read_only=True)`,
|
||||
and uses `GH_TOKEN` for GitHub GET resolution. `s3://` selects `S3_BUCKET`; pass
|
||||
`--s3-store s3://BUCKET/PREFIX` for a prefixed store and `--local-store file:///PATH`
|
||||
for a different local store. Read-only S3 opening skips the normal write/Select probe;
|
||||
collection uses LIST/GET only. Existing normal store opening is unchanged.
|
||||
|
||||
Pair catalogs/recipes include local pointer rows (`vendored = false`, including
|
||||
`fix_repo`). Additional external manifests can be supplied with repeatable
|
||||
`--pointer-manifest PATH`. Registered units in memory and experiment YAML are followed;
|
||||
missing units with a registered digest fail. Plans without frozen units are not used data.
|
||||
Experiment units identify repositories in `group`. Binary measurement artifacts are
|
||||
read and counted; structured JSON, TOML and multi-document YAML are parsed strictly.
|
||||
|
||||
Each source records files/objects read, bytes read, repository-bearing rows (distinct
|
||||
names per input), and distinct repository counts. The digest is SHA-256 of sorted,
|
||||
normalized names joined with newlines. Source counts overlap. An empty source fails;
|
||||
only after establishing it is empty may the coordinator pass `--known-empty LABEL`.
|
||||
The output records that declaration. Missing/unreadable inputs still fail.
|
||||
|
||||
GitHub redirects add the requested and canonical names, plus parent/source fork
|
||||
identities. Used names are also resolved so an old used name excludes a renamed
|
||||
candidate. GitHub cannot enumerate all historical names: injected clients may supply
|
||||
`former_names`; actual historical aliases come from used references and redirects.
|
||||
A 404 for a historical/synthetic used identity retains its literal identity; other
|
||||
API failures stop collection. Candidate resolution always fails closed on a 404.
|
||||
`resolve(candidate, api)` and `rejection(resolution, used, selected_networks)` are
|
||||
available to the later sourcing task. Names stay in memory; output contains only
|
||||
counts, labels and SHA-256 digests. Output files are created exclusively, never overwritten.
|
||||
|
||||
`--verify-tests` runs the three group-1 test modules with both output streams discarded,
|
||||
including the existing population guard. Only aggregate test counts and the exit code
|
||||
enter the record; a failed test prevents the record. V3 paths are excluded before
|
||||
eligibility reads; only the existing guard handles v3 inside its black-box test process.
|
||||
|
||||
Population reservations retain their existing sweep over `benchmarks/`, `src/`, and
|
||||
`tests/`. Pool reservations also cover `plane/`. Each reservation exempts only its own
|
||||
directory; population checks include all of `benchmarks/unseen/private/`. This preserves
|
||||
the existing population behavior while adding pools (the hard rule for this increment).
|
||||
|
||||
All `pool-pN.toml` files require a matching `freeze-pN.json`; drafts use `draft-pN.toml`.
|
||||
The freeze digest is SHA-256 of parsed public TOML encoded as UTF-8 JSON with sorted
|
||||
keys, compact separators and `ensure_ascii=False`. Private entries must be represented
|
||||
by opaque digests in that public manifest, as task 3.2 requires.
|
||||
|
||||
The ledger exposes `append(path, row)` and `read(path)`. Rows contain `slice`, `decision`
|
||||
(an opaque decision or experiment label), `purpose`, ISO `date`, `result_digest`, and
|
||||
`freeze_digest`. Appends lock and validate the existing chain, reject duplicates or a
|
||||
changed freeze, and reject qualification of an informed slice for the same decision.
|
||||
Keep the published final digest to authenticate history: the chain detects row edits,
|
||||
but cannot independently prove that an entire history was not replaced or truncated.
|
||||
|
||||
## Sourcing and extraction (tasks 2.1–2.4)
|
||||
|
||||
The coordinator's networked task **2.5** is now:
|
||||
|
||||
```sh
|
||||
export PATH=/home/mc/Source/OpenUltraSAST/.venv/bin:$PATH
|
||||
export PYTHONPATH=$PWD/src
|
||||
python -m benchmarks.unseen.draft --root "$PWD" --s3-store s3:// --seed 20260601 \
|
||||
--cache /home/mc/ousast-results/plane/unseen-draft --max-hours 8
|
||||
```
|
||||
|
||||
Export `GH_TOKEN` first, or supply it through the root `.env` (exported values win).
|
||||
The same read-only local/S3 stores, `--pointer-manifest`, and explicit `--known-empty`
|
||||
options as task 1.5 apply. Missing or unreadable sources fail. The used set is rebuilt
|
||||
and canonicalized before candidate selection, including redirects and fork networks.
|
||||
No population v3 file is opened and no guard/bisection is run by this command; that is
|
||||
reserved for group 3. An existing draft is never overwritten.
|
||||
|
||||
The command writes `--cache/journal/draft-p1.toml` and
|
||||
`--cache/journal/private-draft-p1.toml` only after all sourcing stages complete.
|
||||
Both stay outside the repository, with mode 0600. The public-format file contains permissive entries
|
||||
and, for every private entry, only an opaque ID, slice, private marker, and SHA-256
|
||||
of canonical JSON containing its URL and ordered `[base, head]` pairs. Private entries
|
||||
carry the complete records, including licences, in the private-format file.
|
||||
Licensing is re-read through GitHub's licence endpoint **at the actual extracted
|
||||
vulnerability replay head**; the candidate's earlier fix-parent classification is
|
||||
provisional. An explicit SPDX declaration in the pinned file takes precedence over
|
||||
the API's classification. Missing/unknown/compound licences remain private.
|
||||
|
||||
Three seeded slices each contain 100 repositories, one vulnerability range and twelve
|
||||
ordinary changes per repository. Each ecosystem/post-cutoff stratum differs from its
|
||||
proportional per-slice target by less than one repository. Post-cutoff candidates are
|
||||
attempted first within each ecosystem. Below 300 eligible repositories the command
|
||||
exits 1 with counts and does not write a partial draft or relax any selection bar.
|
||||
The group-2 measured exit and label confirmation remain coordinator tasks 2.5/2.6;
|
||||
passing implementation tests does not assert that 300 eligible repositories exist.
|
||||
|
||||
Clones and checkpoints default to `results_root()/unseen-draft` (normally
|
||||
`~/ousast-results/plane/unseen-draft`), overridable by `--cache /outside/path`.
|
||||
The cache cannot lie inside `--root`. One opaque, disposable clone exists at a time;
|
||||
it is removed on success, rejection, or failure. `--filter=blob:none --no-checkout`
|
||||
keeps the commit graph needed for temporal thirds, blame, and release-tag ancestry,
|
||||
and fetches source blobs on demand. This is a **partial clone**, not a shallow history.
|
||||
The 500 MiB disk cap is checked after Git commands, including lazy blob fetches;
|
||||
one in-flight command can transiently exceed it before rejection and cleanup.
|
||||
Git output is captured, never forwarded. Each repository must read a nonempty source
|
||||
blob before extraction. Summary instrumentation includes source read operations and
|
||||
actual bytes, clone count, cumulative/peak disk bytes, wall time, and logical API calls
|
||||
and HTTP attempts. Zero source bytes or a failed Git read is an instrument failure,
|
||||
not an eligibility drop. Stdout is a JSON summary of counts/digests only; no names or
|
||||
URLs enter stdout or a measurement record. Raw per-change data stays in the external journal and manifests.
|
||||
|
||||
Extraction uses Git and syntax boundaries, with no scanner, engine, or model. Python
|
||||
uses stdlib AST; Java/PHP/JS/TS use conservative named-function brace matching. Ambiguous
|
||||
or unparseable functions can reject a candidate. The first removed code line per
|
||||
fix-touched function is the SZZ sink proxy; pure-addition fixes use its declaration.
|
||||
Normalized text absence, non-root/non-merge ancestry, file/line limits, and the OSV
|
||||
release bracket are checked before accepting `introducing`. Matching tags are `VERSION`
|
||||
or `vVERSION`; an absent/ambiguous tag or missing prior release cannot certify the
|
||||
bracket. After at most five walk-backs, `last_touch` uses `git log -L` with the resolved
|
||||
numeric function extent (also works where Git lacks a language-specific function driver).
|
||||
Method, bias, fallback reasons, head function/range, files/hunks, family, advisory, and
|
||||
cutoff flag are recorded. Task 2.6 must review every third `introducing` label.
|
||||
|
||||
All reviewed advisories are enumerated to exclude **all** known repository fixes from
|
||||
ordinary changes, including old advisories and advisories outside the four candidate
|
||||
ecosystems. Fix URLs are canonicalized before indexing so renames cannot hide them.
|
||||
The ordinary filter uses only the design's narrowed security pattern. It counts
|
||||
rejections (including oversized commits and repositories failing the 40-commit floor),
|
||||
checks both sides for touches to known vulnerable functions, and inspects generated
|
||||
headers and inherited `.gitattributes`. Sampling allocates twelve draws proportionally
|
||||
across three history-time thirds and four source-size buckets, with one per third.
|
||||
|
||||
For offline tests, inject GitHub/OSV clients, the clone context, and `UsedSet` into
|
||||
`draft.run`; HTTP transports and Git subprocess runners also accept fakes. Tests use
|
||||
sanitized REST-shaped data and a temporary synthetic local Git history. No implementation
|
||||
test contacts GitHub, OSV, a model, or a cluster. HTTP retries are bounded to five attempts;
|
||||
rate-limit reset/Retry-After waits and transient 5xx backoff are tested with fake clocks
|
||||
and sleepers. A stale candidate 404 is counted; other service failures fail the run.
|
||||
|
||||
### Checkpoint and resume
|
||||
|
||||
Replay reads bounded JSONL records into a disposable SQLite working store, with a 2 MiB
|
||||
page cache. Advisory descriptions, non-fix references and vulnerable-version payloads
|
||||
are discarded; package identities, CWEs, status, publication dates and every fix link
|
||||
are retained. Repository metadata keeps canonical/fork identities and eligibility fields.
|
||||
Advisory ordering, fix indexes and checkpoint lookups stay on disk, independent of the
|
||||
number of journal records. The original seeded ordering and ordinary draws are preserved.
|
||||
Per-repository history and exclusion sets also spill to private temporary storage, which
|
||||
is removed on success or failure. Budget disk space for the journal, working store and
|
||||
spooled Git output as well as the clone.
|
||||
|
||||
To bound individual reads, journal records are limited to 16 MiB, decoded Git command
|
||||
output to 8 MiB, history messages to 1 MiB and source blobs to 512 KiB. Exceeding a limit
|
||||
fails the instrument explicitly; it never counts as an empty history or a clean result.
|
||||
The synthetic 20,000-advisory regression checks a 64 MiB traced-memory ceiling across
|
||||
replay, enumeration, ordering and fix indexing; this is not a live coordinator RSS measurement.
|
||||
|
||||
The memory fix can migrate the exact preceding design fingerprints automatically,
|
||||
keeping `inputs-before-memory-fix.json` and leaving the JSONL journal untouched. All
|
||||
other input/design mismatches still refuse resume. This compatibility exception must
|
||||
be removed if a later change alters selection or sampling semantics.
|
||||
|
||||
The command above uses journal directory
|
||||
`/home/mc/ousast-results/plane/unseen-draft/journal`. Run the same command to resume.
|
||||
`--max-hours` is a per-invocation budget; changing it does not invalidate checkpoints.
|
||||
Create `journal/STOP` to stop between units, then remove it before resuming. A page,
|
||||
repository resolution, candidate decision, or repository extraction already running
|
||||
finishes first, so the time budget is not a hard timeout on a large repository.
|
||||
Clean stops exit 0 with `status: stopped`; failures and shortfalls exit 1. Completion
|
||||
exits 0 with `status: draft`. No partial manifest is published.
|
||||
|
||||
`inputs.json` binds the seed, source-labelled used-set digest, known-empty declarations,
|
||||
clone cap, and design fingerprints (sourcing/extraction/eligibility/publication code and
|
||||
family taxonomy). A mismatch refuses resume and prints only the changed parameter
|
||||
labels. Use a new external cache for a deliberately changed design or population.
|
||||
The used set is rebuilt each invocation; repository resolutions, including 404s, are
|
||||
reused from the journal. Live GitHub metadata for unfinished units can still change;
|
||||
completed pages and decisions are the durable sourcing snapshot.
|
||||
|
||||
Each completed unit is appended to its stage JSONL and fsynced before work advances:
|
||||
`advisories.jsonl` (pages), `resolution.jsonl` (alias/fork metadata),
|
||||
`eligibility.jsonl` (candidate decisions), `extraction.jsonl` (repository/advisory
|
||||
results, vulnerable range, ordinary draws, pinned licence, method and counts), and
|
||||
`completion.jsonl` (final manifest digest). A lock permits only one writer.
|
||||
`progress.json` is atomically rewritten after every unit with stage counts, last unit,
|
||||
last observed GitHub rate-limit headers, and cumulative active elapsed seconds.
|
||||
These files may contain identities and must stay outside version control. Stdout and
|
||||
repository records contain only counts, digests and fixed labels.
|
||||
|
||||
Replaying completed units makes no GitHub/OSV or clone calls. Final manifests are
|
||||
reconstructed deterministically from journal results; an interrupted publication can
|
||||
be retried, but a different existing manifest is refused. Completed JSONL rows are
|
||||
never rewritten. Malformed or torn records fail closed instead of being interpreted
|
||||
as completed work. Tests interrupt each stage with fakes, then compare resumed and
|
||||
uninterrupted manifests byte for byte.
|
||||
|
||||
### Planning estimate for task 2.5 (not measured)
|
||||
|
||||
Let A be all reviewed advisories, R distinct fix-linked repository identities, U used
|
||||
identities not already cached, C candidate advisories passing early filters, and K
|
||||
repositories surviving extraction. GitHub requests are approximately
|
||||
`ceil(A/100) + R + U + 2*C + K` (enumeration, canonical repositories, fix metadata,
|
||||
provisional licence, final licence), plus retries. OSV requests are at most C. Git
|
||||
fetches and lazy blob transfers are additional network operations, not GitHub API calls.
|
||||
|
||||
For a planning scenario of A=40,000, R=15,000, U≈1,000, C=600–1,500, K=300:
|
||||
about **18,000–20,000 GitHub calls** and **600–1,500 OSV calls**. The all-advisory index
|
||||
makes this substantially more than 300 repository lookups. At an assumed 5,000-request
|
||||
hourly allowance, allow several rate-limit windows. Actual token limits can differ.
|
||||
For 300–1,000 attempted clones averaging 20–100 MB after source hydration, plan roughly
|
||||
**6–100 GB transferred**, with one clone occupying about 500 MiB at most between commands
|
||||
plus the retained journal and manifests after completion. Transfer volume is an estimate; disk bytes
|
||||
are only a proxy. A working planning range is **8–48 hours**, potentially longer on
|
||||
large histories: every first-parent commit is inspected and lazy blobs can dominate.
|
||||
These are assumptions, not evidence from a network run. The command reports actual
|
||||
counts and wall time; task 2.5 establishes the measured estimate.
|
||||
|
||||
## Dispatcher, replay and scoring (groups 4–6)
|
||||
|
||||
`benchmarks.ax.batch` owns Task transport and the shared scheduler. New workloads supply
|
||||
`Item` input bytes/commands and a `Workload` image, URL allowlist, deadline and result
|
||||
validator. Two AX slots and one VM slot are the default; sandbox death gets one fresh
|
||||
Task before VM fallback, OOM goes straight to VM, and instrument failures get one rerun.
|
||||
Cleanup failure prevents further launches. Create `STOP` in the output directory to stop
|
||||
launching; remove it and repeat the command to resume matching input digests. The legacy
|
||||
engine adapter retains its `questions.json` object name for compatibility; replay inputs
|
||||
and outputs use `.dat`. Both use the same object polling and cleanup implementation.
|
||||
|
||||
Replay reads `pool-pN.toml`, verifies its adjacent `freeze-pN.json` canonical digest,
|
||||
selects `--slice`, and resolves private pointers through `private/pool-pN.toml` (or
|
||||
`--private-manifest`). It never reads a draft. `--dry-run` prints only counts and digests.
|
||||
The five-change proof uses a separate frozen JSON manifest from this project's own
|
||||
source-changing history. Raw output must be outside the repository.
|
||||
|
||||
Each output root contains `inputs.json` and digest-keyed records and artifact directories.
|
||||
The scorer accepts that root directly, joins change labels, and includes missing or stale
|
||||
results as instrument failures. It refuses a slice above 5% failure after reruns. Example:
|
||||
|
||||
```sh
|
||||
python -m benchmarks.unseen.score "$HOME/ousast-results/unseen/p1/1/default" \
|
||||
--out "$HOME/ousast-results/unseen/p1/1/default-score.json"
|
||||
```
|
||||
|
||||
For paired binary arms add `--against BASELINE_ROOT --slice-id p1/2`; `--informed` forces
|
||||
`unseen.gate=not_run`. The later experiment integration owns ledger verification; callers
|
||||
of `gate` must pass that ledger decision. Score-bearing arms use `budgets` with fixed
|
||||
thresholds and `gate(..., binary=False, threshold=..., threshold_fixed=True)`. In-slice
|
||||
thresholds are exploratory and cannot qualify adoption. JVM wall time is explicitly the
|
||||
sum of the hook artifact's scan build/query times, not a separately observed process timer.
|
||||
|
||||
### Coordinator task 4.4
|
||||
|
||||
Run from the merged repository root. Set `ENGINE_IMAGE` to the **engine** digest printed
|
||||
by Engine image CI, `ENGINE_UNITS` and `ENGINE_EXAMPLES` to the frozen input files for the
|
||||
two stored pins, `ENGINE_BASELINE` to their stored record directory, and `AX_KUBECONFIG`
|
||||
to the operator's kube-ax kubeconfig. Host store credentials stay on the VM; use the public
|
||||
HTTPS `S3_ENDPOINT=https://files.because-security.com` and the operator's `OUSAST_MEMORY`.
|
||||
Use a fresh output directory, so the smoke executes instead of resuming prior records.
|
||||
|
||||
```sh
|
||||
export PATH=/home/mc/Source/OpenUltraSAST/.venv/bin:$PATH
|
||||
export PYTHONPATH=$PWD/src
|
||||
export S3_ENDPOINT=https://files.because-security.com
|
||||
python benchmarks/learn/engine_trace.py \
|
||||
--units "$ENGINE_UNITS" --examples "$ENGINE_EXAMPLES" \
|
||||
--executor ax --parallel 2 --limit 2 --image "$ENGINE_IMAGE" \
|
||||
--kubeconfig "$AX_KUBECONFIG" --out "$HOME/ousast-results/unseen/proof-04"
|
||||
python -m benchmarks.unseen.proof engine --image "$ENGINE_IMAGE" \
|
||||
--records "$HOME/ousast-results/unseen/proof-04" --baseline "$ENGINE_BASELINE" \
|
||||
--out "benchmarks/measurements/$(date +%F)-unseen-04-dispatcher/record.json"
|
||||
```
|
||||
|
||||
### Coordinator task 5.4
|
||||
|
||||
Merge before running this proof. `.github/workflows/engine-image.yml` builds the `engine`
|
||||
and `replay` targets on push to main; it publishes `ousast-engine-task` and
|
||||
`ousast-replay-task` separately and prints their digests. Set `REPLAY_IMAGE` to the latter
|
||||
full `ghcr.io/...@sha256:...` reference. No manual image build is needed. Set
|
||||
`KIND_KUBECONFIG` and `AX_KUBECONFIG` to the corresponding AX kubeconfigs; the kind lane
|
||||
uses the identical Task API/manifest, never a Kubernetes Job.
|
||||
|
||||
```sh
|
||||
export PATH=/home/mc/Source/OpenUltraSAST/.venv/bin:$PATH
|
||||
export PYTHONPATH=$PWD/src
|
||||
export S3_ENDPOINT=https://files.because-security.com
|
||||
python -m benchmarks.unseen.replay --proof-history \
|
||||
--manifest "$HOME/ousast-results/unseen/proof-05-inputs.json"
|
||||
python -m benchmarks.unseen.replay --manifest "$HOME/ousast-results/unseen/proof-05-inputs.json" \
|
||||
--image "$REPLAY_IMAGE" --lane kind --kubeconfig "$KIND_KUBECONFIG" \
|
||||
--out "$HOME/ousast-results/unseen/proof-05-kind"
|
||||
python -m benchmarks.unseen.replay --manifest "$HOME/ousast-results/unseen/proof-05-inputs.json" \
|
||||
--image "$REPLAY_IMAGE" --lane ax --kubeconfig "$AX_KUBECONFIG" \
|
||||
--out "$HOME/ousast-results/unseen/proof-05-ax"
|
||||
python -m benchmarks.unseen.replay --manifest "$HOME/ousast-results/unseen/proof-05-inputs.json" \
|
||||
--image "$REPLAY_IMAGE" --lane docker \
|
||||
--out "$HOME/ousast-results/unseen/proof-05-vm"
|
||||
python -m benchmarks.unseen.proof replay --image "$REPLAY_IMAGE" \
|
||||
--kind-records "$HOME/ousast-results/unseen/proof-05-kind" \
|
||||
--ax-records "$HOME/ousast-results/unseen/proof-05-ax" \
|
||||
--vm-records "$HOME/ousast-results/unseen/proof-05-vm" \
|
||||
--out "benchmarks/measurements/$(date +%F)-unseen-05-replay-task/record.json"
|
||||
```
|
||||
|
||||
The record command fails if fewer than five results exist, identities differ, bytes/JVM
|
||||
proof is missing, or quick-tier findings differ. It does not convert a quick-only run
|
||||
without a JVM into an engine proof. Pool replays use `--lane mixed` (default) for the two
|
||||
AX slots and single VM overflow/fallback lane; `--arm long_deadline` changes only the
|
||||
hook deadline to 300 seconds and is a separate input digest.
|
||||
@@ -1,493 +0,0 @@
|
||||
"""Coordinator-only sourcing command: python -m benchmarks.unseen.draft --help.
|
||||
|
||||
All network and git operations are injected at the run() boundary. Only counts,
|
||||
digests and fixed failure codes go to stdout; identity-bearing output stays in the
|
||||
private journal outside the repository. No population guard is run here.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import copy
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import random
|
||||
import subprocess
|
||||
import tempfile
|
||||
import time
|
||||
from collections import Counter, defaultdict
|
||||
from contextlib import contextmanager
|
||||
from dataclasses import asdict
|
||||
from pathlib import Path
|
||||
|
||||
from . import eligibility, extract, source
|
||||
from .extract import Git, InstrumentFailure
|
||||
from .journal import Journal, ResolvedGitHub, ResumeMismatch, Stopped, atomic, digest, external
|
||||
from .storage import AdvisoryRows, DiskSet, FixIndex, working_store
|
||||
|
||||
|
||||
def canonical(value) -> bytes:
|
||||
return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode()
|
||||
|
||||
|
||||
def private_digest(row: dict) -> str:
|
||||
return hashlib.sha256(canonical({"url": row["url"], "commits": [[c["base"], c["head"]] for c in row["changes"]]})).hexdigest()
|
||||
|
||||
|
||||
def assign_slices(rows: list[dict], *, seed: int) -> list[dict]:
|
||||
if len(rows) != 300 or len({r["repository"] for r in rows}) != 300:
|
||||
raise ValueError("pool_size")
|
||||
for row in rows:
|
||||
kinds = Counter(c["kind"] for c in row["changes"])
|
||||
if kinds != {"vulnerability": 1, "ordinary": 12}:
|
||||
raise ValueError("change_count")
|
||||
if row["license_class"] not in {"permissive", "private"}:
|
||||
raise ValueError("license_class")
|
||||
if row["license_class"] == "permissive" and row["license_spdx"] not in source.PERMISSIVE:
|
||||
raise ValueError("license_class")
|
||||
rng = random.Random(seed)
|
||||
groups = defaultdict(list)
|
||||
for row in sorted(rows, key=lambda r: r["repository"]):
|
||||
groups[row["ecosystem"], row["post_cutoff"]].append(row)
|
||||
sizes = Counter({1: 0, 2: 0, 3: 0})
|
||||
assigned = []
|
||||
for key in sorted(groups):
|
||||
group = groups[key]
|
||||
rng.shuffle(group)
|
||||
quotient, remainder = divmod(len(group), 3)
|
||||
# Give each stratum its floor on all slices, then fill globally least-full slices.
|
||||
quotas = {i: quotient for i in sizes}
|
||||
order = list(sizes)
|
||||
rng.shuffle(order)
|
||||
order.sort(key=lambda i: sizes[i])
|
||||
for i in order[:remainder]:
|
||||
quotas[i] += 1
|
||||
offset = 0
|
||||
for slice_id in sorted(quotas):
|
||||
for row in group[offset : offset + quotas[slice_id]]:
|
||||
digest = private_digest(row)
|
||||
assigned.append({**row, "id": "r-" + digest[:24], "slice": slice_id})
|
||||
sizes[slice_id] += quotas[slice_id]
|
||||
offset += quotas[slice_id]
|
||||
if set(sizes.values()) != {100}:
|
||||
raise ValueError("slice_size")
|
||||
return sorted(assigned, key=lambda r: (r["slice"], r["id"]))
|
||||
|
||||
|
||||
def toml(value) -> str:
|
||||
"""Inline TOML values keep nested change records structured without a new dependency."""
|
||||
if isinstance(value, bool):
|
||||
return "true" if value else "false"
|
||||
if isinstance(value, str):
|
||||
return json.dumps(value, ensure_ascii=False)
|
||||
if isinstance(value, int):
|
||||
return str(value)
|
||||
if isinstance(value, list):
|
||||
return "[" + ", ".join(toml(v) for v in value) + "]"
|
||||
if isinstance(value, dict):
|
||||
return "{" + ", ".join(json.dumps(k) + " = " + toml(v) for k, v in sorted(value.items()) if v is not None) + "}"
|
||||
raise ValueError("unsupported_manifest_type")
|
||||
|
||||
|
||||
def manifest(rows: list[dict], seed: int) -> str:
|
||||
lines = ['status = "draft"', 'pool = "p1"', f"seed = {seed}"]
|
||||
for row in rows:
|
||||
lines.append("\n[[repository]]")
|
||||
lines.extend(f"{key} = {toml(value)}" for key, value in sorted(row.items()) if value is not None)
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def write_draft(root: Path, assigned: list[dict], *, seed: int) -> tuple[Path, Path]:
|
||||
public_path = root / "benchmarks/unseen/draft-p1.toml"
|
||||
private_path = root / "benchmarks/unseen/private/draft-p1.toml"
|
||||
if public_path.exists() or private_path.exists():
|
||||
raise FileExistsError("draft_exists")
|
||||
public, private = [], []
|
||||
for row in assigned:
|
||||
if row["license_class"] == "permissive":
|
||||
public.append(row)
|
||||
else:
|
||||
private.append(row)
|
||||
public.append({"id": row["id"], "sha256": private_digest(row), "slice": row["slice"], "private": True})
|
||||
private_path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
# Exclusive creates protect drafts awaiting human label review. Private first: never
|
||||
# publish a pointer whose private manifest failed to write.
|
||||
with private_path.open("x") as handle:
|
||||
os.chmod(private_path, 0o600)
|
||||
handle.write(manifest(private, seed))
|
||||
with public_path.open("x") as handle:
|
||||
handle.write(manifest(public, seed))
|
||||
return public_path, private_path
|
||||
|
||||
|
||||
class Clones:
|
||||
"""One disposable blobless clone at a time; no name in cache paths or subprocess output."""
|
||||
|
||||
def __init__(self, root: Path, cache: Path, *, runner=None, limit_bytes=500 * 1024 * 1024):
|
||||
self.cache = external(root, cache)
|
||||
self.runner = runner or subprocess.run
|
||||
self.limit_bytes = limit_bytes
|
||||
self.counts = Counter()
|
||||
|
||||
@contextmanager
|
||||
def open(self, url: str):
|
||||
self.cache.mkdir(parents=True, exist_ok=True)
|
||||
started = time.monotonic()
|
||||
with tempfile.TemporaryDirectory(prefix="clone-", dir=self.cache) as directory:
|
||||
path = Path(directory)
|
||||
git = Git(path, runner=self.runner, limit_bytes=self.limit_bytes)
|
||||
try:
|
||||
# Full commit graph is needed for temporal thirds and OSV brackets; blobs
|
||||
# are fetched lazily. This is a partial clone, not a truncated history.
|
||||
done = self.runner(
|
||||
["git", "-c", "credential.helper=", "clone", "--quiet", "--filter=blob:none", "--no-checkout", url, str(path)],
|
||||
stdin=subprocess.DEVNULL,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
check=False,
|
||||
timeout=180,
|
||||
env={**os.environ, "GIT_TERMINAL_PROMPT": "0"},
|
||||
)
|
||||
if done.returncode:
|
||||
raise InstrumentFailure("clone_failed")
|
||||
self.counts["clones"] += 1
|
||||
if git.disk_bytes() > self.limit_bytes:
|
||||
raise InstrumentFailure("clone_size_cap")
|
||||
yield git
|
||||
finally:
|
||||
size = git.disk_bytes()
|
||||
self.counts["clone_bytes"] += size
|
||||
self.counts["peak_clone_bytes"] = max(self.counts["peak_clone_bytes"], size)
|
||||
self.counts["source_bytes_read"] += git.bytes_read
|
||||
self.counts["source_files_read"] += git.files_read
|
||||
self.counts["clone_wall_seconds"] += round(time.monotonic() - started, 3)
|
||||
|
||||
|
||||
def prove_source(git, revision: str) -> dict:
|
||||
files = git.run("ls-tree", "-r", "--name-only", revision).splitlines()
|
||||
for path in files:
|
||||
if Path(path).suffix in extract.LANGUAGES and not extract.generated_path(path):
|
||||
text = git.blob(revision, path)
|
||||
if text:
|
||||
return {"files": 1, "bytes": getattr(git, "last_blob_bytes", len(text.encode()))}
|
||||
raise InstrumentFailure("zero_source_bytes")
|
||||
|
||||
|
||||
def advisory_index(rows, github, *, db):
|
||||
"""Include ALL enumerated fixes, even old, unmapped, multi-fix and other ecosystems.
|
||||
|
||||
Canonicalize link names as well, so an old advisory URL survives repository renames.
|
||||
A 404 is not treated as proof that an advisory is unrelated to a selected repository.
|
||||
"""
|
||||
index = FixIndex(db)
|
||||
for row in rows:
|
||||
for name, sha in source.fix_links(row):
|
||||
try:
|
||||
resolved = eligibility.resolve(name, github)
|
||||
except eligibility.RepositoryNotFound:
|
||||
index.add(name, sha)
|
||||
continue
|
||||
index.add(resolved.full_name, sha)
|
||||
return index
|
||||
|
||||
|
||||
def repository_changes(git, candidate, fixes: set[str], *, seed: int):
|
||||
proof = prove_source(git, candidate.parent)
|
||||
change = extract.introducing(git, candidate)
|
||||
with working_store() as db:
|
||||
known = DiskSet(db, "known_functions")
|
||||
expanded = DiskSet(db, "expanded_fixes")
|
||||
pending = DiskSet(db, "pending_fixes")
|
||||
pending.update(fixes)
|
||||
pending.add(candidate.fix)
|
||||
for fix in pending:
|
||||
# Explicitly fetch detached advisory commits unavailable on default-branch history.
|
||||
# All git including this possible network call still goes through the injected client.
|
||||
git.run("fetch", "--quiet", "--filter=blob:none", "origin", fix)
|
||||
full = git.run("rev-parse", fix).strip()
|
||||
expanded.add(full)
|
||||
parents = git.parents(full)
|
||||
for parent in parents:
|
||||
sites = extract.fix_sites(git, full, parent, global_ok=True)
|
||||
known.update((site["path"], site["function"]) for site in sites)
|
||||
rows = extract.ordinary_history(git, "HEAD")
|
||||
ordinary, exclusions = extract.draw_ordinary(rows, expanded, known, seed=seed)
|
||||
return [change, *ordinary], exclusions, proof
|
||||
|
||||
|
||||
def inputs(used, seed, known_empty, clones):
|
||||
from openultrasast.model.taxonomy import DEFAULT_FAMILIES_PATH
|
||||
|
||||
parameters = {
|
||||
"seed": seed,
|
||||
"used_set_digest": digest({label: sorted(names) for label, names in used.sources.items()}),
|
||||
"known_empty": sorted(known_empty),
|
||||
"clone_limit_bytes": getattr(clones, "limit_bytes", 500 * 1024 * 1024),
|
||||
"journal_version": 1,
|
||||
}
|
||||
# Hardcoded design thresholds and algorithms are covered as well as taxonomy data.
|
||||
for name in ("draft", "source", "extract", "eligibility", "journal", "storage"):
|
||||
parameters["design_" + name] = hashlib.sha256(Path(__file__).with_name(name + ".py").read_bytes()).hexdigest()
|
||||
parameters["design_taxonomy"] = hashlib.sha256(DEFAULT_FAMILIES_PATH.read_bytes()).hexdigest()
|
||||
return parameters
|
||||
|
||||
|
||||
# The pre-memory-fix implementation has identical ordering/selection semantics. Only
|
||||
# this exact fingerprint set may migrate; changed seeds, taxonomy or used sets still fail.
|
||||
LEGACY_DESIGNS = {
|
||||
"design_draft": "906bf36de11761931b2c1b62a2ea8da3c8124f9d754455f1a0acf29a6db576a3",
|
||||
"design_source": "01c896bd0418faf0d073dc2d5cb6dbc5d7903b95b5e8216c9022bc9985326a9c",
|
||||
"design_extract": "593d954d9c22b835e1d349df6b575707a39098dca2eff4425f67a0de14c6525e",
|
||||
"design_journal": "c97ad0761ea5bbfe65db816b13dfa79d93608ea03bcd5848f70686ed9157bdf7",
|
||||
}
|
||||
|
||||
|
||||
def legacy_inputs(parameters):
|
||||
return {key: LEGACY_DESIGNS.get(key, value) for key, value in parameters.items() if key != "design_storage"}
|
||||
|
||||
|
||||
def enumerate_advisories(github, journal):
|
||||
page, records = 1, AdvisoryRows(journal.db, "advisory_records")
|
||||
while True:
|
||||
key = str(page)
|
||||
if key not in journal.rows["advisories"]:
|
||||
journal.boundary()
|
||||
result = github.advisory_page(page)
|
||||
journal.put("advisories", key, result)
|
||||
result = journal.rows["advisories"][key]
|
||||
for row in result["rows"]:
|
||||
records[row["ghsa_id"]] = row
|
||||
if result["done"]:
|
||||
break
|
||||
page += 1
|
||||
if not records:
|
||||
raise InstrumentFailure("empty_advisories")
|
||||
return records
|
||||
|
||||
|
||||
def decide(row, github, osv, used, selected_networks):
|
||||
try:
|
||||
links = source.fix_links(row)
|
||||
if len(links) != 1:
|
||||
raise source.Rejected("fix_links")
|
||||
name, _ = next(iter(links))
|
||||
resolved = eligibility.resolve(name, github)
|
||||
rejected_by = eligibility.rejection(resolved, used, set(selected_networks))
|
||||
if rejected_by:
|
||||
raise source.Rejected("used_" + rejected_by)
|
||||
candidate = source.candidate(row, github, osv)
|
||||
return {"candidate": asdict(candidate), "network": resolved.network}
|
||||
except source.Rejected as exc:
|
||||
return {"rejected": str(exc)}
|
||||
except eligibility.RepositoryNotFound:
|
||||
return {"rejected": "repository_or_commit_not_found"}
|
||||
|
||||
|
||||
def extract_repository(candidate, github, clones, fixes, seed):
|
||||
before = dict(clones.counts)
|
||||
try:
|
||||
with clones.open(candidate.url) as git:
|
||||
git.run("fetch", "--quiet", "--filter=blob:none", "origin", candidate.fix)
|
||||
changes, exclusions, proof = repository_changes(git, candidate, fixes, seed=seed)
|
||||
pinned = changes[0]["head"]
|
||||
license_record = github.get_license(candidate.repository, pinned)
|
||||
spdx, license_class = source.license_info(license_record)
|
||||
entry = {
|
||||
**asdict(candidate),
|
||||
"license_spdx": spdx,
|
||||
"license_class": license_class,
|
||||
"license_ref": pinned,
|
||||
"license_path": license_record.get("path", ""),
|
||||
"changes": changes,
|
||||
"instrument": proof,
|
||||
}
|
||||
result = {"entry": entry, "ordinary_exclusions": exclusions}
|
||||
except source.Rejected as exc:
|
||||
result = {"rejected": str(exc), "ordinary_exclusions": getattr(exc, "counts", {})}
|
||||
except eligibility.RepositoryNotFound:
|
||||
result = {"rejected": "repository_or_commit_not_found", "ordinary_exclusions": {}}
|
||||
result["counts"] = {key: value - before.get(key, 0) for key, value in clones.counts.items()}
|
||||
return result
|
||||
|
||||
|
||||
def publish(journal, assigned, seed):
|
||||
"""Idempotent publication from completed journal data, never inside the checkout."""
|
||||
public, private = [], []
|
||||
for row in assigned:
|
||||
if row["license_class"] == "permissive":
|
||||
public.append(row)
|
||||
else:
|
||||
private.append(row)
|
||||
public.append({"id": row["id"], "sha256": private_digest(row), "slice": row["slice"], "private": True})
|
||||
for name, rows in (("private-draft-p1.toml", private), ("draft-p1.toml", public)):
|
||||
path = journal.path / name
|
||||
data = manifest(rows, seed).encode()
|
||||
if path.exists():
|
||||
if path.read_bytes() != data:
|
||||
raise ValueError("draft_exists")
|
||||
else:
|
||||
atomic(path, data)
|
||||
|
||||
|
||||
def run(
|
||||
root: Path,
|
||||
*,
|
||||
github,
|
||||
osv,
|
||||
clones,
|
||||
used: eligibility.UsedSet,
|
||||
seed: int,
|
||||
known_empty=None,
|
||||
journal_path=None,
|
||||
max_hours=None,
|
||||
clock=None,
|
||||
) -> dict:
|
||||
started = time.monotonic()
|
||||
report = {"status": "sourcing", "seed": seed, "candidates": {}, "rejections": {}, "ordinary_exclusions": {}}
|
||||
counts, rejected, ordinary_counts = Counter(), Counter(), Counter()
|
||||
checkpoint = None
|
||||
try:
|
||||
known_empty = set(known_empty or ())
|
||||
used.report(known_empty)
|
||||
cache = getattr(clones, "cache", root.parent / (root.name + "-cache"))
|
||||
path = external(root, journal_path or cache / "journal")
|
||||
checkpoint = Journal(
|
||||
path, max_hours=max_hours, clock=clock, rate_limit=lambda: getattr(getattr(github, "transport", None), "rate_limit", {})
|
||||
)
|
||||
parameters = inputs(used, seed, known_empty, clones)
|
||||
with checkpoint.open(parameters, compatible_parameters=legacy_inputs(parameters)):
|
||||
checkpoint.boundary()
|
||||
api = ResolvedGitHub(github, checkpoint)
|
||||
used = copy.deepcopy(used) # canonicalization must not mutate the resume input
|
||||
eligibility.canonicalize_used(used, api)
|
||||
report["used_set"] = used.report(known_empty)
|
||||
rows = enumerate_advisories(api, checkpoint)
|
||||
report["advisories"] = len(rows)
|
||||
fixes_by_repo = advisory_index(rows, api, db=checkpoint.db)
|
||||
selected_networks, entries = set(), []
|
||||
for row in source.ordered_advisories(rows, seed):
|
||||
checkpoint.boundary()
|
||||
key = row["ghsa_id"]
|
||||
counts[source.ecosystem(row)] += 1
|
||||
if key not in checkpoint.rows["eligibility"]:
|
||||
decision = decide(row, api, osv, used, selected_networks)
|
||||
checkpoint.put("eligibility", key, decision)
|
||||
decision = checkpoint.rows["eligibility"][key]
|
||||
if "rejected" in decision:
|
||||
rejected[decision["rejected"]] += 1
|
||||
continue
|
||||
candidate = source.Candidate(**decision["candidate"])
|
||||
# Multiple advisories for one repository can have different fix/range inputs.
|
||||
extraction_key = digest({"repository": candidate.repository, "advisory": key, "fix": candidate.fix})
|
||||
if extraction_key not in checkpoint.rows["extraction"]:
|
||||
checkpoint.boundary()
|
||||
result = extract_repository(candidate, api, clones, fixes_by_repo.get(candidate.repository, set()), seed)
|
||||
checkpoint.put("extraction", extraction_key, result)
|
||||
result = checkpoint.rows["extraction"][extraction_key]
|
||||
ordinary_counts.update(result["ordinary_exclusions"])
|
||||
if "rejected" in result:
|
||||
rejected[result["rejected"]] += 1
|
||||
continue
|
||||
selected_networks.add(decision["network"])
|
||||
entries.append(result["entry"])
|
||||
if len(entries) == 300:
|
||||
break
|
||||
report.update(
|
||||
{
|
||||
"repositories": len(entries),
|
||||
"candidates": dict(counts),
|
||||
"rejections": dict(rejected),
|
||||
"ordinary_exclusions": dict(ordinary_counts),
|
||||
}
|
||||
)
|
||||
checkpoint.boundary()
|
||||
if len(entries) != 300:
|
||||
report["status"] = "insufficient_repositories"
|
||||
checkpoint.progress(report["status"])
|
||||
return report
|
||||
assigned = assign_slices(entries, seed=seed)
|
||||
completion = {"manifest_sha256": digest(assigned), "repositories": len(entries)}
|
||||
if "all" not in checkpoint.rows["completion"]:
|
||||
checkpoint.put("completion", "all", completion)
|
||||
elif checkpoint.rows["completion"]["all"] != completion:
|
||||
raise ValueError("journal_completion_mismatch")
|
||||
checkpoint.boundary()
|
||||
publish(checkpoint, assigned, seed)
|
||||
report.update(
|
||||
{
|
||||
"status": "draft",
|
||||
"changes": 3900,
|
||||
"slices": 3,
|
||||
"license_split": dict(Counter(e["license_class"] for e in entries)),
|
||||
"methods": dict(Counter(e["changes"][0]["method"] for e in entries)),
|
||||
"post_cutoff": sum(e["post_cutoff"] for e in entries),
|
||||
**completion,
|
||||
}
|
||||
)
|
||||
checkpoint.progress("draft")
|
||||
return report
|
||||
except ResumeMismatch as exc:
|
||||
report.update({"status": "resume_refused", "changed_parameters": exc.parameters})
|
||||
return report
|
||||
except Stopped as exc:
|
||||
report.update({"status": "stopped", "reason": str(exc)})
|
||||
return report
|
||||
except Exception as exc:
|
||||
report.update({"status": "instrument_failure", "error_type": type(exc).__name__})
|
||||
return report
|
||||
finally:
|
||||
report.update({"candidates": dict(counts), "rejections": dict(rejected), "ordinary_exclusions": dict(ordinary_counts)})
|
||||
report["instrument"] = dict(clones.counts)
|
||||
report["wall_seconds"] = round(time.monotonic() - started, 3)
|
||||
report["github_calls"] = github.calls
|
||||
report["osv_calls"] = osv.calls
|
||||
report["github_http_attempts"] = getattr(getattr(github, "transport", None), "attempts", github.calls)
|
||||
report["osv_http_attempts"] = getattr(getattr(osv, "transport", None), "attempts", osv.calls)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
from openultrasast import config
|
||||
from openultrasast.plane import memory
|
||||
from openultrasast.plane.reconciler import results_root
|
||||
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--root", type=Path, default=Path.cwd())
|
||||
parser.add_argument("--cache", type=Path)
|
||||
parser.add_argument("--max-hours", type=float, help="Stop between units after this many hours per invocation")
|
||||
parser.add_argument("--seed", type=int, default=20260601)
|
||||
parser.add_argument("--local-store")
|
||||
parser.add_argument("--s3-store", default="s3://")
|
||||
parser.add_argument("--pointer-manifest", type=Path, action="append", default=[])
|
||||
parser.add_argument("--known-empty", choices=(*eligibility.SOURCES, "memory_local", "memory_s3"), action="append", default=[])
|
||||
args = parser.parse_args(argv)
|
||||
try:
|
||||
config.load_dotenv(args.root / ".env")
|
||||
# Refuse an existing draft before doing any network work.
|
||||
if any((args.root / p).exists() for p in ("benchmarks/unseen/draft-p1.toml", "benchmarks/unseen/private/draft-p1.toml")):
|
||||
raise FileExistsError("draft_exists")
|
||||
clones = Clones(args.root, args.cache or results_root() / "unseen-draft")
|
||||
stores = {
|
||||
"local": memory.open_store(args.local_store or "file://" + str(results_root() / "memory"), read_only=True),
|
||||
"s3": memory.open_store(args.s3_store, read_only=True),
|
||||
}
|
||||
used = eligibility.build_used(args.root, stores=stores, pointer_manifests=args.pointer_manifest)
|
||||
report = run(
|
||||
args.root,
|
||||
github=source.GitHub(os.environ.get("GH_TOKEN", "")),
|
||||
osv=source.OSV(),
|
||||
clones=clones,
|
||||
used=used,
|
||||
seed=args.seed,
|
||||
known_empty=set(args.known_empty),
|
||||
journal_path=clones.cache / "journal",
|
||||
max_hours=args.max_hours,
|
||||
)
|
||||
except Exception as exc:
|
||||
report = {"status": "instrument_failure", "error_type": type(exc).__name__}
|
||||
print(json.dumps(report, sort_keys=True))
|
||||
return 0 if report["status"] in {"draft", "stopped"} else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,379 +0,0 @@
|
||||
"""Read-only used-set collection. Only counts, source labels and digests leave this module's CLI.
|
||||
|
||||
Run from the repository root: python -m benchmarks.unseen.eligibility --help.
|
||||
Population v3 is exclusively the existing guard's responsibility.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
import tomllib
|
||||
from collections.abc import Iterable, Mapping
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
import yaml
|
||||
|
||||
from openultrasast.plane.memory import MemoryStore
|
||||
|
||||
URL = re.compile(r"github\.com/([\w.-]+/[\w.-]+)", re.I)
|
||||
NAME = re.compile(r"[\w.-]+/[\w.-]+")
|
||||
SOURCES = ("pairs", "pointers", "populations", "harvest", "units", "measurements", "sweep")
|
||||
|
||||
|
||||
def normalize(value: str) -> str:
|
||||
text = value.strip().lower().rstrip("/")
|
||||
if NAME.fullmatch(text):
|
||||
name = text.removesuffix(".git")
|
||||
else:
|
||||
# Structured fields must be identities, not prose containing a URL.
|
||||
if any(char.isspace() for char in text):
|
||||
raise ValueError("invalid_repository")
|
||||
url = urlsplit(text if "://" in text else "https://" + text)
|
||||
if url.scheme not in {"http", "https", "ssh", "git"} or not url.hostname or "." not in url.hostname:
|
||||
raise ValueError("invalid_repository")
|
||||
parts = url.path.strip("/").split("/")
|
||||
if len(parts) < 2:
|
||||
raise ValueError("invalid_repository")
|
||||
name = "/".join(parts[:2]).removesuffix(".git")
|
||||
if not NAME.fullmatch(name):
|
||||
raise ValueError("invalid_repository")
|
||||
if url.hostname != "github.com":
|
||||
return url.hostname + "/" + name
|
||||
if not NAME.fullmatch(name) or any(part in {".", ".."} for part in name.split("/")):
|
||||
raise ValueError("invalid_repository")
|
||||
return name
|
||||
|
||||
|
||||
def excluded(path: Path) -> bool:
|
||||
return any("v3" in part.lower() or part in {".git", "__pycache__", ".venv"} for part in path.parts)
|
||||
|
||||
|
||||
def repositories(value: object, counts: dict[str, int] | None = None) -> set[str]:
|
||||
"""Structured repo fields include bare owner/name pointers; URLs also occur in prose."""
|
||||
found = set()
|
||||
if isinstance(value, dict):
|
||||
for key, item in value.items():
|
||||
if key in {"repo", "fix_repo", "repository", "full_name", "origin"} or (key == "group" and "unit" in value):
|
||||
try:
|
||||
if not isinstance(item, str):
|
||||
raise ValueError("invalid_repository")
|
||||
found.add(normalize(item))
|
||||
except ValueError:
|
||||
if counts is not None:
|
||||
counts["skipped_values"] = counts.get("skipped_values", 0) + 1
|
||||
else:
|
||||
found.update(repositories(item, counts))
|
||||
elif isinstance(value, list):
|
||||
for item in value:
|
||||
found.update(repositories(item, counts))
|
||||
elif isinstance(value, str):
|
||||
found.update(normalize(m[1]) for m in URL.finditer(value))
|
||||
return found
|
||||
|
||||
|
||||
def pointer_repositories(value: object, counts: dict[str, int] | None = None) -> set[str]:
|
||||
if isinstance(value, dict):
|
||||
if value.get("vendored") is False:
|
||||
return repositories(value, counts)
|
||||
return set().union(*(pointer_repositories(v, counts) for v in value.values()))
|
||||
if isinstance(value, list):
|
||||
return set().union(*(pointer_repositories(v, counts) for v in value))
|
||||
return set()
|
||||
|
||||
|
||||
def parse(path: Path, raw: bytes) -> object:
|
||||
structured = path.suffix in {".toml", ".json", ".jsonl", ".yaml", ".yml"}
|
||||
text = raw.decode("utf-8", errors="strict" if structured else "ignore")
|
||||
if path.suffix == ".toml":
|
||||
return tomllib.loads(text)
|
||||
if path.suffix == ".json":
|
||||
return json.loads(text)
|
||||
if path.suffix == ".jsonl":
|
||||
return [json.loads(line) for line in text.splitlines() if line.strip()]
|
||||
if path.suffix in {".yaml", ".yml"}:
|
||||
documents = list(yaml.safe_load_all(text))
|
||||
return documents[0] if len(documents) == 1 else documents
|
||||
return text
|
||||
|
||||
|
||||
@dataclass
|
||||
class UsedSet:
|
||||
sources: dict[str, set[str]] = field(default_factory=lambda: {label: set() for label in SOURCES})
|
||||
instrument: dict[str, dict[str, int]] = field(default_factory=dict)
|
||||
|
||||
def add(self, label: str, names: set[str], size: int, files: int = 1, skipped_values: int = 0) -> None:
|
||||
self.sources.setdefault(label, set()).update(names)
|
||||
counts = self.instrument.setdefault(label, {"files": 0, "bytes": 0, "rows": 0, "skipped_values": 0})
|
||||
counts["files"] += files
|
||||
counts["bytes"] += size
|
||||
counts["rows"] += len(names)
|
||||
counts["skipped_values"] += skipped_values
|
||||
|
||||
def report(self, known_empty: set[str] | None = None) -> dict:
|
||||
known_empty = known_empty or set()
|
||||
if known_empty - self.sources.keys():
|
||||
raise ValueError("unknown_source_label")
|
||||
records = {}
|
||||
for label, names in self.sources.items():
|
||||
counts = self.instrument.get(label, {"files": 0, "bytes": 0, "rows": 0, "skipped_values": 0})
|
||||
if not counts["rows"] and label not in known_empty:
|
||||
raise ValueError("empty_source:" + label)
|
||||
records[label] = {**counts, "count": len(names), "known_empty": label in known_empty}
|
||||
names = set().union(*self.sources.values())
|
||||
return {"sources": records, "count": len(names), "sha256": hashlib.sha256("\n".join(sorted(names)).encode()).hexdigest()}
|
||||
|
||||
|
||||
def build_used(root: Path, *, stores: Mapping[str, MemoryStore] | None = None, pointer_manifests: Iterable[Path] = ()) -> UsedSet:
|
||||
used = UsedSet()
|
||||
|
||||
def read(label: str, paths: Iterable[Path], structured: bool = True) -> list[object]:
|
||||
values = []
|
||||
for path in sorted(set(paths)):
|
||||
if excluded(path):
|
||||
continue
|
||||
raw = path.read_bytes() # failure is an instrument failure, never a quiet empty source
|
||||
value = parse(path, raw) if structured else raw.decode("utf-8", errors="ignore")
|
||||
counts: dict[str, int] = {}
|
||||
names = repositories(value, counts)
|
||||
used.add(label, names, len(raw), skipped_values=counts.get("skipped_values", 0))
|
||||
if label == "pairs":
|
||||
pointer_counts: dict[str, int] = {}
|
||||
pointers = pointer_repositories(value, pointer_counts)
|
||||
if pointers or pointer_counts:
|
||||
used.add("pointers", pointers, len(raw), skipped_values=pointer_counts.get("skipped_values", 0))
|
||||
values.append(value)
|
||||
return values
|
||||
|
||||
pairs = root / "benchmarks/pairs"
|
||||
read("pairs", (p for p in pairs.rglob("*") if p.is_file() and p.suffix in {".toml", ".json", ".jsonl", ".csv", ".yaml", ".yml"}))
|
||||
read("pointers", pointer_manifests)
|
||||
read("populations", (p for p in (root / "benchmarks/independent").glob("population-v[12].toml")))
|
||||
read("harvest", (p for directory in pairs.glob("advisory-fixes*") for p in directory.rglob("*.toml")))
|
||||
units = set((root / "plane/experiments").glob("*.units.jsonl"))
|
||||
for path in (root / "plane/experiments").glob("*.y*ml"):
|
||||
if excluded(path):
|
||||
continue
|
||||
raw = path.read_bytes()
|
||||
manifest = parse(path, raw)
|
||||
counts = {}
|
||||
names = repositories(manifest, counts)
|
||||
used.add("units", names, len(raw), skipped_values=counts.get("skipped_values", 0))
|
||||
if isinstance(manifest, dict) and manifest.get("units_file"):
|
||||
declared = root / Path(manifest["units_file"]).expanduser()
|
||||
if declared.exists(): # an unregistered plan may not have frozen units yet
|
||||
units.add(declared)
|
||||
for label, store in (stores or {}).items():
|
||||
if label not in {"local", "s3"}:
|
||||
raise ValueError("unknown_store_label")
|
||||
source = "memory_" + label
|
||||
used.add(source, set(), 0, 0)
|
||||
# Read actual objects, not a serialized row size masquerading as bytes read.
|
||||
for key in store._keys("repos/"):
|
||||
if not key.endswith(".jsonl") or excluded(Path(key)):
|
||||
continue
|
||||
result = store._get(key)
|
||||
if result is None:
|
||||
raise ValueError("missing_memory_object")
|
||||
raw = result[0]
|
||||
rows = parse(Path("rows.jsonl"), raw)
|
||||
names = set()
|
||||
counts = {}
|
||||
for row in rows:
|
||||
if row.get("kind") == "example" and "repo" in row:
|
||||
names.update(repositories({"repo": row["repo"]}, counts))
|
||||
if row.get("kind") == "experiment" and row.get("units_file"):
|
||||
registered = root / Path(row["units_file"]).expanduser()
|
||||
if row.get("units_digest") or registered.exists():
|
||||
units.add(registered)
|
||||
used.add(source, names, len(raw), skipped_values=counts.get("skipped_values", 0))
|
||||
read("units", units)
|
||||
read("measurements", (p for top in ("measurements", "experiments") for p in (root / "benchmarks" / top).rglob("*") if p.is_file()))
|
||||
read(
|
||||
"sweep",
|
||||
(
|
||||
p
|
||||
for top in ("benchmarks", "src", "tests", "plane", ".kiro")
|
||||
for p in (root / top).rglob("*")
|
||||
if p.is_file() and not p.is_relative_to(root / "benchmarks/unseen") and not excluded(p)
|
||||
),
|
||||
structured=False,
|
||||
)
|
||||
return used
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Resolution:
|
||||
full_name: str
|
||||
names: frozenset[str]
|
||||
network: str
|
||||
|
||||
|
||||
def resolve(candidate: str, api) -> Resolution:
|
||||
name = normalize(candidate)
|
||||
if name.count("/") > 1:
|
||||
return Resolution(name, frozenset({name}), name)
|
||||
data = api.get_repo(name)
|
||||
canonical = normalize(data["full_name"])
|
||||
names = {name, canonical, *(normalize(n) for n in data.get("former_names", []))}
|
||||
for key in ("parent", "source"):
|
||||
if data.get(key):
|
||||
names.add(normalize(data[key]["full_name"]))
|
||||
network = normalize((data.get("source") or data.get("parent") or data)["full_name"])
|
||||
return Resolution(canonical, frozenset(names), network)
|
||||
|
||||
|
||||
def canonicalize_used(used: UsedSet, api) -> None:
|
||||
"""Resolve old used names too: GitHub exposes redirects, not a complete rename history."""
|
||||
cache = {}
|
||||
for names in used.sources.values():
|
||||
for name in sorted(names):
|
||||
if name not in cache:
|
||||
try:
|
||||
cache[name] = resolve(name, api)
|
||||
except RepositoryNotFound:
|
||||
cache[name] = Resolution(name, frozenset({name}), name)
|
||||
names.update(cache[name].names)
|
||||
|
||||
|
||||
def rejection(candidate: Resolution, used: UsedSet, selected: set[str]) -> str | None:
|
||||
for label, names in used.sources.items():
|
||||
if candidate.names & names:
|
||||
return label
|
||||
if candidate.network in selected:
|
||||
return "fork_network"
|
||||
selected.add(candidate.network)
|
||||
return None
|
||||
|
||||
|
||||
class RepositoryNotFound(ValueError):
|
||||
"""A historical or synthetic used identity no longer resolves; retain it verbatim."""
|
||||
|
||||
|
||||
class GitHub:
|
||||
"""Coordinator-only GET client. Exceptions never disclose request URLs or tokens."""
|
||||
|
||||
def __init__(self, token: str):
|
||||
if not token:
|
||||
raise ValueError("missing_gh_token")
|
||||
self.token = token
|
||||
|
||||
def get_repo(self, name: str) -> dict:
|
||||
from urllib.error import HTTPError
|
||||
from urllib.request import Request, urlopen
|
||||
|
||||
request = Request(
|
||||
"https://api.github.com/repos/" + normalize(name),
|
||||
headers={
|
||||
"Authorization": "Bearer " + self.token,
|
||||
"Accept": "application/vnd.github+json",
|
||||
},
|
||||
)
|
||||
try:
|
||||
with urlopen(request, timeout=30) as response:
|
||||
return json.load(response)
|
||||
except HTTPError as exc:
|
||||
if exc.code == 404:
|
||||
raise RepositoryNotFound("github_not_found") from None
|
||||
raise ValueError("github_resolution_failed") from None
|
||||
except Exception:
|
||||
raise ValueError("github_resolution_failed") from None
|
||||
|
||||
|
||||
def verify_tests(root: Path) -> dict:
|
||||
"""The coordinator runs the guard as a black box; only exit codes and counts escape."""
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from xml.etree import ElementTree
|
||||
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
report = Path(directory) / "tests.xml"
|
||||
done = subprocess.run(
|
||||
[
|
||||
sys.executable,
|
||||
"-m",
|
||||
"pytest",
|
||||
"-q",
|
||||
"-p",
|
||||
"no:cacheprovider",
|
||||
"tests/test_unseen_eligibility.py",
|
||||
"tests/test_unseen_ledger.py",
|
||||
"tests/test_independent_population.py",
|
||||
"--junitxml=" + str(report),
|
||||
],
|
||||
cwd=root,
|
||||
stdin=subprocess.DEVNULL,
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL,
|
||||
check=False,
|
||||
)
|
||||
if done.returncode != 0:
|
||||
raise ValueError("guard_or_tests_failed")
|
||||
suites = ElementTree.parse(report).getroot().iter("testsuite")
|
||||
counts = {key: 0 for key in ("tests", "failures", "errors", "skipped")}
|
||||
for suite in suites:
|
||||
for key in counts:
|
||||
counts[key] += int(suite.attrib[key])
|
||||
if not counts["tests"]:
|
||||
raise ValueError("empty_test_run")
|
||||
return {"guard": "green", "test_exit_code": done.returncode, "tests": counts}
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
import os
|
||||
|
||||
from openultrasast import config
|
||||
from openultrasast.plane import memory
|
||||
from openultrasast.plane.reconciler import results_root
|
||||
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--root", type=Path, default=Path.cwd())
|
||||
parser.add_argument("--local-store", default=None)
|
||||
parser.add_argument("--s3-store", default="s3://")
|
||||
parser.add_argument("--pointer-manifest", type=Path, action="append", default=[])
|
||||
parser.add_argument("--known-empty", choices=(*SOURCES, "memory_local", "memory_s3"), action="append", default=[])
|
||||
parser.add_argument("--output", type=Path)
|
||||
parser.add_argument("--verify-tests", action="store_true")
|
||||
args = parser.parse_args(argv)
|
||||
step = "open_stores"
|
||||
try:
|
||||
config.load_dotenv(args.root / ".env")
|
||||
stores = {
|
||||
"local": memory.open_store(args.local_store or "file://" + str(results_root() / "memory"), read_only=True),
|
||||
"s3": memory.open_store(args.s3_store, read_only=True),
|
||||
}
|
||||
step = "build_used"
|
||||
used = build_used(args.root, stores=stores, pointer_manifests=args.pointer_manifest)
|
||||
step = "report"
|
||||
used.report(set(args.known_empty)) # prove inputs before API work
|
||||
step = "canonicalize"
|
||||
canonicalize_used(used, GitHub(os.environ.get("GH_TOKEN", "")))
|
||||
step = "report"
|
||||
report = used.report(set(args.known_empty))
|
||||
if args.verify_tests:
|
||||
step = "verify_tests"
|
||||
report.update(verify_tests(args.root))
|
||||
step = "report"
|
||||
text = json.dumps(report, sort_keys=True, indent=2) + "\n"
|
||||
if args.output:
|
||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||
with args.output.open("x") as handle:
|
||||
handle.write(text)
|
||||
print(text, end="")
|
||||
return 0
|
||||
except Exception as exc:
|
||||
# Source paths, store exceptions and JSON parser errors can contain repository names.
|
||||
report = {"status": "instrument_failure", "step": step, "type": type(exc).__name__}
|
||||
if str(exc) in {"empty_source:" + label for label in (*SOURCES, "memory_local", "memory_s3")}:
|
||||
report["source"] = str(exc).partition(":")[2]
|
||||
print(json.dumps(report))
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,519 +0,0 @@
|
||||
"""Git-only change extraction; clients and subprocess execution are injectable.
|
||||
|
||||
No scanner, engine or model runs here. Function boundaries are syntax, not a security
|
||||
judgement: the first removed code line of a fix is the SZZ sink proxy. Ambiguous sites
|
||||
are dropped, and the recorded SZZ bias remains subject to task 2.6's label check.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import ast
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
import tempfile
|
||||
from contextlib import contextmanager
|
||||
from pathlib import Path
|
||||
|
||||
from .source import Candidate, Rejected
|
||||
from .storage import working_store
|
||||
|
||||
LANGUAGES = {".py", ".js", ".jsx", ".mjs", ".cjs", ".ts", ".tsx", ".php", ".java"}
|
||||
HUNK = re.compile(r"^@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@", re.M)
|
||||
SHA = re.compile(r"[a-f0-9]{40}")
|
||||
BIAS = {
|
||||
"introducing": "SZZ last-writer proxy: refactors, moves and reformats can misidentify the introduction in either direction.",
|
||||
"last_touch": "Pessimistic lower bound: the flaw may already exist in base, so a delta hook can correctly stay quiet.",
|
||||
}
|
||||
|
||||
|
||||
class InstrumentFailure(ValueError):
|
||||
"""A failed read or command must never masquerade as an empty history."""
|
||||
|
||||
|
||||
class Git:
|
||||
def __init__(self, path: Path, *, runner=None, limit_bytes: int | None = None):
|
||||
self.path = path
|
||||
self.runner = runner or subprocess.run
|
||||
self.bytes_read = 0
|
||||
self.files_read = 0
|
||||
self.last_blob_bytes = 0
|
||||
self.limit_bytes = limit_bytes
|
||||
|
||||
@contextmanager
|
||||
def output(self, *args: str, allowed=(0,)):
|
||||
# Git can expand a tiny pack into enormous stdout. Spool before decoding, including
|
||||
# stderr, so neither subprocess.communicate nor an error can consume the VM's RAM.
|
||||
with tempfile.TemporaryFile() as stdout, tempfile.TemporaryFile() as stderr:
|
||||
try:
|
||||
done = self.runner(
|
||||
["git", "-c", "core.quotePath=false", "-C", str(self.path), *args],
|
||||
stdin=subprocess.DEVNULL,
|
||||
stdout=stdout,
|
||||
stderr=stderr,
|
||||
timeout=180,
|
||||
check=False,
|
||||
)
|
||||
except Exception:
|
||||
raise InstrumentFailure("git_launch_failed") from None
|
||||
if done.returncode not in allowed:
|
||||
raise InstrumentFailure("git_exit_" + str(done.returncode))
|
||||
if self.limit_bytes is not None and self.disk_bytes() > self.limit_bytes:
|
||||
raise InstrumentFailure("clone_size_cap")
|
||||
# Injected runners may return bytes instead of writing the supplied file.
|
||||
if done.stdout is not None:
|
||||
stdout.write(done.stdout)
|
||||
stdout.seek(0)
|
||||
yield stdout
|
||||
|
||||
def run(self, *args: str, allowed=(0,)) -> str:
|
||||
with self.output(*args, allowed=allowed) as stdout:
|
||||
raw = stdout.read(8 * 1024 * 1024 + 1)
|
||||
if len(raw) > 8 * 1024 * 1024:
|
||||
raise InstrumentFailure("git_output_size_cap")
|
||||
self.last_blob_bytes = len(raw)
|
||||
return raw.decode("utf-8", errors="replace")
|
||||
|
||||
def history_entries(self, head):
|
||||
with self.output("log", "--first-parent", "--format=%H%x00%P%x00%ct%x00%an%x00%B%x1e", head) as stdout:
|
||||
pending = b""
|
||||
while chunk := stdout.read(64 * 1024):
|
||||
entries = (pending + chunk).split(b"\x1e")
|
||||
pending = entries.pop()
|
||||
for entry in entries:
|
||||
if len(entry) > 1024 * 1024:
|
||||
raise InstrumentFailure("git_history_record_size_cap")
|
||||
yield entry.decode("utf-8", errors="replace")
|
||||
if len(pending) > 1024 * 1024:
|
||||
raise InstrumentFailure("git_history_record_size_cap")
|
||||
if pending.strip():
|
||||
raise InstrumentFailure("invalid_history")
|
||||
|
||||
def disk_bytes(self) -> int:
|
||||
return sum(p.stat().st_size for p in self.path.rglob("*") if p.is_file() and not p.is_symlink())
|
||||
|
||||
def parents(self, sha: str) -> list[str]:
|
||||
return self.run("rev-list", "--parents", "-n", "1", sha).split()[1:]
|
||||
|
||||
def blob(self, sha: str, path: str) -> str | None:
|
||||
# Missing is established by a successful tree read, never by swallowing a show error.
|
||||
if not self.run("ls-tree", sha, "--", path).strip():
|
||||
return None
|
||||
size = int(self.run("cat-file", "-s", f"{sha}:{path}").strip())
|
||||
if size > 512 * 1024:
|
||||
raise InstrumentFailure("source_blob_size_cap")
|
||||
text = self.run("show", f"{sha}:{path}")
|
||||
if Path(path).suffix in LANGUAGES:
|
||||
self.files_read += 1
|
||||
self.bytes_read += self.last_blob_bytes
|
||||
return text
|
||||
|
||||
def stats(self, base: str, head: str) -> list[tuple[str, int, int]]:
|
||||
rows = []
|
||||
for line in self.run("diff", "--no-renames", "--numstat", base, head, "--").splitlines():
|
||||
added, removed, path = line.split("\t", 2)
|
||||
rows.append((path, int(added) if added.isdigit() else 0, int(removed) if removed.isdigit() else 0))
|
||||
return rows
|
||||
|
||||
def hunks(self, base: str, head: str, path: str) -> list[tuple[int, int, int, int]]:
|
||||
diff = self.run("diff", "--no-ext-diff", "--no-renames", "-U0", base, head, "--", path)
|
||||
return [(int(m[1]), int(m[2] or 1), int(m[3]), int(m[4] or 1)) for m in HUNK.finditer(diff)]
|
||||
|
||||
def ancestor(self, older: str, newer: str) -> bool:
|
||||
# rev-list avoids interpreting a command failure as a negative ancestry result.
|
||||
return not self.run("rev-list", "--max-count=1", older, "--not", newer).strip()
|
||||
|
||||
def blame(self, revision: str, path: str, line: int) -> tuple[str, str, int]:
|
||||
text = self.run("blame", "-w", "-M", "-C", "--porcelain", f"-L{line},{line}", revision, "--", path)
|
||||
header = text.splitlines()[0].split()
|
||||
if len(header) < 3 or not SHA.fullmatch(header[0]):
|
||||
raise InstrumentFailure("invalid_blame")
|
||||
filename = next((row[9:] for row in text.splitlines() if row.startswith("filename ")), path)
|
||||
if filename.startswith('"'):
|
||||
filename = json.loads(filename)
|
||||
return header[0], filename, int(header[1])
|
||||
|
||||
|
||||
def functions(path: str, text: str) -> list[tuple[str, int, int]]:
|
||||
"""Conservative syntax boundaries. No scanner import; unresolved sites are rejected."""
|
||||
if Path(path).suffix == ".py":
|
||||
try:
|
||||
tree = ast.parse(text)
|
||||
except SyntaxError:
|
||||
raise Rejected("function_parse") from None
|
||||
return [
|
||||
(n.name, n.lineno, n.end_lineno or n.lineno) for n in ast.walk(tree) if isinstance(n, ast.FunctionDef | ast.AsyncFunctionDef)
|
||||
]
|
||||
# Blank comments and quoted strings before brace matching. Keep newlines/offsets.
|
||||
masked = re.sub(
|
||||
r'/\*.*?\*/|//[^\n]*|\#[^\n]*|"(?:\\.|[^"\\])*"|\'(?:\\.|[^\'\\])*\'|`(?:\\.|[^`\\])*`',
|
||||
lambda m: "".join("\n" if c == "\n" else " " for c in m[0]),
|
||||
text,
|
||||
flags=re.S,
|
||||
)
|
||||
declaration = re.compile(
|
||||
r"\bfunction\s*&?\s*(\w+)\s*\([^)]*\)\s*(?::[^\n{]+)?\s*\{|"
|
||||
r"\b(?:const|let|var)\s+(\w+)\s*=\s*(?:async\s*)?(?:\([^)]*\)|\w+)\s*=>\s*\{|"
|
||||
r"\b([\w$]+)\s*\([^;{}]*\)\s*(?:throws\s+[\w., ]+)?\s*\{"
|
||||
)
|
||||
found = []
|
||||
for match in declaration.finditer(masked):
|
||||
name = next(g for g in match.groups() if g)
|
||||
if name in {"if", "while", "for", "switch", "catch", "with", "synchronized"}:
|
||||
continue
|
||||
depth = 1
|
||||
end = match.end()
|
||||
while end < len(masked) and depth:
|
||||
depth += (masked[end] == "{") - (masked[end] == "}")
|
||||
end += 1
|
||||
if depth == 0:
|
||||
found.append((name, masked.count("\n", 0, match.start()) + 1, masked.count("\n", 0, end) + 1))
|
||||
return found
|
||||
|
||||
|
||||
def enclosing(path: str, text: str, line: int, *, global_ok=False) -> tuple[str, int, int] | None:
|
||||
spans = [s for s in functions(path, text) if s[1] <= line <= s[2]]
|
||||
if spans:
|
||||
return min(spans, key=lambda s: s[2] - s[1])
|
||||
return ("<global>", 1, max(1, len(text.splitlines()))) if global_ok else None
|
||||
|
||||
|
||||
def normalized(text: str) -> str:
|
||||
return re.sub(r"\s+", "", text)
|
||||
|
||||
|
||||
def site_at(git, revision: str, path: str, name: str):
|
||||
text = git.blob(revision, path)
|
||||
if text is None:
|
||||
return None
|
||||
spans = [("<global>", 1, max(1, len(text.splitlines())))] if name == "<global>" else [s for s in functions(path, text) if s[0] == name]
|
||||
if len(spans) != 1:
|
||||
return None
|
||||
return text, spans[0]
|
||||
|
||||
|
||||
def fix_sites(git, fix: str, parent: str, *, global_ok=False) -> list[dict]:
|
||||
sites = []
|
||||
for path, _, _ in git.stats(parent, fix):
|
||||
if Path(path).suffix not in LANGUAGES:
|
||||
continue
|
||||
text = git.blob(parent, path)
|
||||
if text is None:
|
||||
continue
|
||||
lines = text.splitlines()
|
||||
for start, count, _, _ in git.hunks(parent, fix, path):
|
||||
for line in range(max(1, start), max(1, start) + max(1, count)):
|
||||
if line > len(lines):
|
||||
continue
|
||||
span = enclosing(path, text, line, global_ok=global_ok)
|
||||
if span is None:
|
||||
continue
|
||||
sink_line = line if count else span[1]
|
||||
sink = lines[sink_line - 1]
|
||||
if not sink.strip() or sink.lstrip().startswith(("#", "//", "/*", "*")):
|
||||
continue
|
||||
key = (path, span[0])
|
||||
if any((s["path"], s["function"]) == key for s in sites):
|
||||
continue
|
||||
sites.append({"path": path, "function": span[0], "line": sink_line, "text": sink, "absence": count == 0, "span": span})
|
||||
return sites
|
||||
|
||||
|
||||
def version_bracket(git, candidate: Candidate, revision: str) -> bool:
|
||||
if not candidate.first_affected:
|
||||
return True
|
||||
tags = git.run("tag", "--sort=version:refname").splitlines()
|
||||
matching = [t for t in tags if t.removeprefix("v") == candidate.first_affected]
|
||||
if len(matching) != 1:
|
||||
return False
|
||||
affected = matching[0]
|
||||
earlier = [t for t in tags[: tags.index(affected)] if git.ancestor(t, affected)]
|
||||
return bool(earlier) and git.ancestor(revision, affected) and not git.ancestor(revision, earlier[-1])
|
||||
|
||||
|
||||
def change_files(git, base: str, head: str) -> list[dict]:
|
||||
return [
|
||||
{"path": path, "head_lines": [[max(1, new), max(1, new + max(count, 1) - 1)] for _, _, new, count in git.hunks(base, head, path)]}
|
||||
for path, _, _ in git.stats(base, head)
|
||||
]
|
||||
|
||||
|
||||
def introducing(git, candidate: Candidate) -> dict:
|
||||
from collections import Counter
|
||||
|
||||
if git.parents(candidate.fix) != [candidate.parent]:
|
||||
raise Rejected("fix_parents")
|
||||
sites = fix_sites(git, candidate.fix, candidate.parent, global_ok=candidate.family == "config_secrets")
|
||||
reasons = Counter()
|
||||
|
||||
def record(method, base, head, path, name):
|
||||
resolved = site_at(git, head, path, name)
|
||||
if resolved is None:
|
||||
return None
|
||||
return {
|
||||
"kind": "vulnerability",
|
||||
"base": base,
|
||||
"head": head,
|
||||
"method": method,
|
||||
"family": candidate.family,
|
||||
"function": name,
|
||||
"path": path,
|
||||
"function_lines": list(resolved[1][1:]),
|
||||
"files": change_files(git, base, head),
|
||||
"post_cutoff": candidate.post_cutoff,
|
||||
"advisory": candidate.advisory,
|
||||
"bias": BIAS[method],
|
||||
"fallback_reasons": dict(reasons),
|
||||
}
|
||||
|
||||
for site in sites:
|
||||
revision, path, line = candidate.parent, site["path"], site["line"]
|
||||
for _ in range(6): # initial blame plus at most five walk-back steps
|
||||
blamed, blamed_path, blamed_line = git.blame(revision, path, line)
|
||||
parents = git.parents(blamed)
|
||||
if len(parents) != 1:
|
||||
reasons["merge_or_root"] += 1
|
||||
break
|
||||
base = parents[0]
|
||||
head_site = site_at(git, blamed, blamed_path, site["function"])
|
||||
if head_site is None:
|
||||
reasons["unresolved_function"] += 1
|
||||
break
|
||||
parent_text = git.blob(base, blamed_path)
|
||||
if parent_text is not None and site["function"] != "<global>":
|
||||
matching = [span for span in functions(blamed_path, parent_text) if span[0] == site["function"]]
|
||||
if len(matching) > 1:
|
||||
reasons["ambiguous_parent"] += 1
|
||||
break
|
||||
old_site = site_at(git, base, blamed_path, site["function"])
|
||||
occurrences = []
|
||||
if old_site:
|
||||
text, (_, start, end) = old_site
|
||||
occurrences = [
|
||||
i for i, value in enumerate(text.splitlines(), 1) if start <= i <= end and normalized(value) == normalized(site["text"])
|
||||
]
|
||||
if occurrences:
|
||||
reasons["text_present"] += 1
|
||||
revision, path, line = base, blamed_path, occurrences[0]
|
||||
continue
|
||||
stats = git.stats(base, blamed)
|
||||
if len(stats) > 50 or sum(a + d for _, a, d in stats) > 2000:
|
||||
reasons["size_cap"] += 1
|
||||
break
|
||||
if not version_bracket(git, candidate, blamed):
|
||||
reasons["version_bracket"] += 1
|
||||
break
|
||||
result = record("introducing", base, blamed, blamed_path, site["function"])
|
||||
if result:
|
||||
return result
|
||||
else:
|
||||
reasons["walkback_limit"] += 1
|
||||
for site in sites:
|
||||
# Numeric function extent implements log -L for all five languages, including
|
||||
# languages for which Git has no built-in :function: driver.
|
||||
_, start, end = site["span"]
|
||||
history = git.run("log", "--format=%H", "--no-patch", f"-L{start},{end}:{site['path']}", candidate.parent).splitlines()
|
||||
if not history:
|
||||
continue
|
||||
latest = history[0]
|
||||
parents = git.parents(latest)
|
||||
if len(parents) != 1:
|
||||
continue
|
||||
result = record("last_touch", parents[0], candidate.parent, site["path"], site["function"])
|
||||
if result:
|
||||
return result
|
||||
raise Rejected("no_range")
|
||||
|
||||
|
||||
SECURITY = re.compile(r"secur|cve|ghsa|vuln|inject|xss|csrf|ssrf|exploit|sanitiz|escap", re.I)
|
||||
DEPENDENCY = re.compile(r"\bbump\b|\bupgrade .* from\b", re.I)
|
||||
MANIFESTS = {
|
||||
"package.json",
|
||||
"package-lock.json",
|
||||
"yarn.lock",
|
||||
"pnpm-lock.yaml",
|
||||
"composer.json",
|
||||
"composer.lock",
|
||||
"pom.xml",
|
||||
"build.gradle",
|
||||
"build.gradle.kts",
|
||||
"gradle.lockfile",
|
||||
"pyproject.toml",
|
||||
"setup.py",
|
||||
"setup.cfg",
|
||||
"Pipfile",
|
||||
"Pipfile.lock",
|
||||
"poetry.lock",
|
||||
"uv.lock",
|
||||
}
|
||||
|
||||
|
||||
def generated_path(path: str) -> bool:
|
||||
return bool({"vendor", "node_modules", "dist"} & set(Path(path).parts)) or path.endswith(".min.js")
|
||||
|
||||
|
||||
def manifest(path: str) -> bool:
|
||||
name = Path(path).name
|
||||
return name in MANIFESTS or name.endswith(".lock") or bool(re.fullmatch(r"requirements[^/]*\.txt", name))
|
||||
|
||||
|
||||
def ordinary_rejection(row: dict, fixes: set[str], known_functions: set[tuple[str, str]]) -> str | None:
|
||||
if len(row["parents"]) != 1:
|
||||
return "merge_or_root"
|
||||
if row["head"] in fixes:
|
||||
return "advisory_fix"
|
||||
if SECURITY.search(row["message"]):
|
||||
return "security"
|
||||
if re.search(r"^Revert|This reverts commit", row["message"], re.I | re.M):
|
||||
return "revert"
|
||||
if re.search(r"dependabot|renovate|\[bot\]", row["author"], re.I) or DEPENDENCY.search(row["message"]):
|
||||
return "dependency"
|
||||
files = row["files"]
|
||||
if files and all(manifest(f["path"]) for f in files):
|
||||
return "manifest_only"
|
||||
source = [f for f in files if Path(f["path"]).suffix in LANGUAGES and not manifest(f["path"])]
|
||||
if not source:
|
||||
return "no_source"
|
||||
if all(f["generated"] or generated_path(f["path"]) for f in source):
|
||||
return "generated"
|
||||
if any((f["path"], name) in known_functions for f in files for name in f["functions"]):
|
||||
return "vulnerable_function"
|
||||
if any((f["path"], "<global>") in known_functions for f in files):
|
||||
return "vulnerable_function"
|
||||
if sum(f["added"] + f["removed"] for f in source) > 1000:
|
||||
return "size_cap"
|
||||
return None
|
||||
|
||||
|
||||
def generated(git, revision: str, path: str, text: str) -> bool:
|
||||
import fnmatch
|
||||
|
||||
if generated_path(path):
|
||||
return True
|
||||
header = "\n".join(text.splitlines()[:20])
|
||||
if re.search(r"(?im)^\s*(?:#|//|/\*|\*|<!--).*?(?:@generated|auto.?generated|generated\s+(?:by|file|code)|do not edit)", header):
|
||||
return True
|
||||
result = False
|
||||
parts = Path(path).parts
|
||||
for index in range(len(parts)):
|
||||
directory = "/".join(parts[:index])
|
||||
attributes = git.blob(revision, (directory + "/" if directory else "") + ".gitattributes") or ""
|
||||
relative = "/".join(parts[index:])
|
||||
for line in attributes.splitlines():
|
||||
tokens = line.split()
|
||||
if not tokens or tokens[0].startswith("#"):
|
||||
continue
|
||||
pattern = tokens[0].lstrip("/")
|
||||
target = relative if "/" in pattern else Path(relative).name
|
||||
if fnmatch.fnmatchcase(target, pattern):
|
||||
for token in tokens[1:]:
|
||||
if token in {"linguist-generated", "linguist-generated=true"}:
|
||||
result = True
|
||||
elif token in {"-linguist-generated", "!linguist-generated", "linguist-generated=false"}:
|
||||
result = False
|
||||
return result
|
||||
|
||||
|
||||
def ordinary_history(git, head: str):
|
||||
"""Yield one commit at a time; no repository-wide blob or diff retention."""
|
||||
for entry in git.history_entries(head):
|
||||
if not entry.strip():
|
||||
continue
|
||||
fields = entry.strip().split("\x00")
|
||||
if len(fields) != 5 or not SHA.fullmatch(fields[0]):
|
||||
raise InstrumentFailure("invalid_history")
|
||||
sha, parents, timestamp, author, message = fields
|
||||
row = {"head": sha, "parents": parents.split(), "timestamp": int(timestamp), "author": author, "message": message, "files": []}
|
||||
if len(row["parents"]) != 1:
|
||||
yield row
|
||||
continue
|
||||
base = row["parents"][0]
|
||||
for path, added, removed in git.stats(base, sha):
|
||||
hunks = git.hunks(base, sha, path)
|
||||
info = {
|
||||
"path": path,
|
||||
"added": added,
|
||||
"removed": removed,
|
||||
"head_lines": [[max(1, n), max(1, n + max(c, 1) - 1)] for _, _, n, c in hunks],
|
||||
"functions": [],
|
||||
"generated": False,
|
||||
}
|
||||
if Path(path).suffix in LANGUAGES:
|
||||
# Inspect both sides: deleting a vulnerable function is still touching it.
|
||||
for revision, side in ((base, 0), (sha, 2)):
|
||||
text = git.blob(revision, path)
|
||||
if text is None:
|
||||
continue
|
||||
spans = functions(path, text)
|
||||
for hunk in hunks:
|
||||
start, count = hunk[side : side + 2]
|
||||
for name, lo, hi in spans:
|
||||
if max(1, start) <= hi and start + max(count, 1) - 1 >= lo:
|
||||
info["functions"].append(name)
|
||||
info["generated"] |= generated(git, revision, path, text)
|
||||
row["files"].append(info)
|
||||
yield row
|
||||
|
||||
|
||||
def draw_ordinary(rows, fixes: set[str], known_functions: set[tuple[str, str]], *, seed: int) -> tuple[list[dict], dict]:
|
||||
"""Spill eligible history to disk; keep the original seeded stratified draw exactly."""
|
||||
with working_store() as db:
|
||||
return _draw_ordinary(db, rows, fixes, known_functions, seed=seed)
|
||||
|
||||
|
||||
def _draw_ordinary(db, rows, fixes, known_functions, *, seed):
|
||||
import random
|
||||
from collections import Counter
|
||||
|
||||
counts = Counter()
|
||||
earliest = latest = None
|
||||
db.execute("CREATE TABLE eligible (head TEXT, timestamp INTEGER, bucket INTEGER, value TEXT, third INTEGER)")
|
||||
for row in rows:
|
||||
timestamp = row["timestamp"]
|
||||
earliest = timestamp if earliest is None else min(earliest, timestamp)
|
||||
latest = timestamp if latest is None else max(latest, timestamp)
|
||||
reason = ordinary_rejection(row, fixes, known_functions)
|
||||
counts[reason or "eligible"] += 1
|
||||
if reason is not None:
|
||||
continue
|
||||
size = sum(f["added"] + f["removed"] for f in row["files"] if Path(f["path"]).suffix in LANGUAGES and not manifest(f["path"]))
|
||||
bucket = 0 if size <= 10 else 1 if size <= 50 else 2 if size <= 200 else 3
|
||||
value = {
|
||||
"kind": "ordinary",
|
||||
"base": row["parents"][0],
|
||||
"head": row["head"],
|
||||
"method": "first_parent",
|
||||
"size_bucket": bucket,
|
||||
"files": [{"path": f["path"], "head_lines": f["head_lines"]} for f in row["files"]],
|
||||
}
|
||||
db.execute("INSERT INTO eligible VALUES (?, ?, ?, ?, NULL)", (row["head"], timestamp, bucket, json.dumps(value)))
|
||||
if counts["eligible"] < 40:
|
||||
failure = Rejected("ordinary_floor")
|
||||
failure.counts = dict(counts)
|
||||
raise failure
|
||||
db.execute("UPDATE eligible SET third = min(2, 3 * (timestamp - ?) / ?)", (earliest, max(1, latest - earliest + 1)))
|
||||
cells = {
|
||||
(third, bucket): count for third, bucket, count in db.execute("SELECT third, bucket, count(*) FROM eligible GROUP BY third, bucket")
|
||||
}
|
||||
if {t for t, _ in cells} != {0, 1, 2}:
|
||||
failure = Rejected("time_thirds")
|
||||
failure.counts = dict(counts)
|
||||
raise failure
|
||||
allocation = Counter()
|
||||
targets = {cell: 12 * count / counts["eligible"] for cell, count in cells.items()}
|
||||
for third in range(3):
|
||||
cell = max(sorted(c for c in cells if c[0] == third), key=lambda c: targets[c])
|
||||
allocation[cell] = 1
|
||||
while sum(allocation.values()) < 12:
|
||||
cell = max(sorted(c for c in cells if allocation[c] < cells[c]), key=lambda c: targets[c] - allocation[c])
|
||||
allocation[cell] += 1
|
||||
db.execute("CREATE INDEX eligible_order ON eligible(third, bucket, head)")
|
||||
rng = random.Random(seed)
|
||||
drawn = []
|
||||
for cell in sorted(cells):
|
||||
for position in rng.sample(range(cells[cell]), allocation[cell]):
|
||||
(value,) = db.execute(
|
||||
"SELECT value FROM eligible WHERE third=? AND bucket=? ORDER BY head LIMIT 1 OFFSET ?", (*cell, position)
|
||||
).fetchone()
|
||||
drawn.append({**json.loads(value), "time_third": cell[0]})
|
||||
return drawn, dict(counts)
|
||||
@@ -1,9 +0,0 @@
|
||||
{
|
||||
"ghsa_id": "GHSA-test-test-test",
|
||||
"published_at": "2026-06-02T00:00:00Z",
|
||||
"type": "reviewed",
|
||||
"withdrawn_at": null,
|
||||
"cwes": [{"cwe_id": "CWE-89"}],
|
||||
"vulnerabilities": [{"package": {"ecosystem": "pip", "name": "fixture"}}],
|
||||
"references": ["https://github.com/fixture/project/commit/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"]
|
||||
}
|
||||
@@ -1,860 +0,0 @@
|
||||
[
|
||||
{
|
||||
"id": "0",
|
||||
"repo": "0",
|
||||
"kind": "ordinary",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": [
|
||||
{
|
||||
"cwe": "CWE-89",
|
||||
"path": "app.py",
|
||||
"line": 10
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "1",
|
||||
"repo": "0",
|
||||
"kind": "ordinary",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "2",
|
||||
"repo": "1",
|
||||
"kind": "ordinary",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "3",
|
||||
"repo": "1",
|
||||
"kind": "ordinary",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "4",
|
||||
"repo": "2",
|
||||
"kind": "ordinary",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "5",
|
||||
"repo": "2",
|
||||
"kind": "ordinary",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "6",
|
||||
"repo": "3",
|
||||
"kind": "ordinary",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "7",
|
||||
"repo": "3",
|
||||
"kind": "ordinary",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "8",
|
||||
"repo": "4",
|
||||
"kind": "ordinary",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "9",
|
||||
"repo": "4",
|
||||
"kind": "ordinary",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "10",
|
||||
"repo": "5",
|
||||
"kind": "ordinary",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "11",
|
||||
"repo": "5",
|
||||
"kind": "ordinary",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "12",
|
||||
"repo": "6",
|
||||
"kind": "ordinary",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "13",
|
||||
"repo": "6",
|
||||
"kind": "ordinary",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "14",
|
||||
"repo": "7",
|
||||
"kind": "ordinary",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "15",
|
||||
"repo": "7",
|
||||
"kind": "ordinary",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "16",
|
||||
"repo": "8",
|
||||
"kind": "vulnerability",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": [
|
||||
{
|
||||
"cwe": "CWE-89",
|
||||
"path": "app.py",
|
||||
"line": 10
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "17",
|
||||
"repo": "8",
|
||||
"kind": "vulnerability",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": [
|
||||
{
|
||||
"cwe": "CWE-89",
|
||||
"path": "app.py",
|
||||
"line": 10
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "18",
|
||||
"repo": "9",
|
||||
"kind": "vulnerability",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "19",
|
||||
"repo": "9",
|
||||
"kind": "vulnerability",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
@@ -1,215 +0,0 @@
|
||||
"""Private, append-only sourcing checkpoints. No identity-bearing data leaves this directory."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import fcntl
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import time
|
||||
from contextlib import contextmanager
|
||||
from pathlib import Path
|
||||
|
||||
from .source import compact_advisory, compact_repository
|
||||
from .storage import DiskMap, working_store
|
||||
|
||||
MAX_RECORD_BYTES = 16 * 1024 * 1024
|
||||
|
||||
|
||||
def compact(stage, value):
|
||||
if stage == "advisories":
|
||||
return {"rows": [compact_advisory(row) for row in value["rows"]], "done": value["done"]}
|
||||
if stage == "resolution" and value is not None:
|
||||
return compact_repository(value)
|
||||
return value
|
||||
|
||||
|
||||
STAGES = ("advisories", "resolution", "eligibility", "extraction", "completion")
|
||||
|
||||
|
||||
def encoded(value) -> bytes:
|
||||
return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode()
|
||||
|
||||
|
||||
def digest(value) -> str:
|
||||
return hashlib.sha256(encoded(value)).hexdigest()
|
||||
|
||||
|
||||
class ResumeMismatch(ValueError):
|
||||
def __init__(self, parameters):
|
||||
self.parameters = parameters
|
||||
super().__init__("resume_inputs_changed")
|
||||
|
||||
|
||||
class Stopped(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def external(root: Path, path: Path) -> Path:
|
||||
path = path.expanduser().resolve()
|
||||
# Also reject the actual checkout when a different --root is supplied.
|
||||
if any(path.is_relative_to(p.resolve()) for p in (root, Path(__file__).resolve().parents[2])):
|
||||
raise ValueError("cache_inside_repository")
|
||||
return path
|
||||
|
||||
|
||||
def atomic(path: Path, payload: bytes) -> None:
|
||||
temporary = path.with_suffix(path.suffix + ".tmp")
|
||||
with temporary.open("wb") as handle:
|
||||
os.chmod(temporary, 0o600)
|
||||
handle.write(payload)
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
temporary.replace(path)
|
||||
sync_directory(path.parent)
|
||||
|
||||
|
||||
def sync_directory(path: Path) -> None:
|
||||
fd = os.open(path, os.O_RDONLY | os.O_DIRECTORY)
|
||||
try:
|
||||
os.fsync(fd)
|
||||
finally:
|
||||
os.close(fd)
|
||||
|
||||
|
||||
class Journal:
|
||||
def __init__(self, path: Path, *, max_hours=None, clock=None, rate_limit=None):
|
||||
if max_hours is not None and (not math.isfinite(max_hours) or max_hours < 0):
|
||||
raise ValueError("invalid_max_hours")
|
||||
self.path = path
|
||||
self.clock = clock or time.monotonic
|
||||
self.started = self.clock()
|
||||
self.max_seconds = None if max_hours is None else max_hours * 3600
|
||||
self.rate_limit = rate_limit or (lambda: {})
|
||||
self.rows = {}
|
||||
self.elapsed_before = 0.0
|
||||
self.last = None
|
||||
self.previous_rate = {}
|
||||
|
||||
@contextmanager
|
||||
def open(self, parameters, *, compatible_parameters=None):
|
||||
self.path.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
with (self.path / "LOCK").open("a") as lock:
|
||||
try:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except BlockingIOError:
|
||||
raise ValueError("journal_busy") from None
|
||||
metadata = self.path / "inputs.json"
|
||||
migrating = False
|
||||
if metadata.exists():
|
||||
old = json.loads(metadata.read_bytes())
|
||||
# Only current, fixed parameter labels can enter stdout.
|
||||
changed = [key for key in parameters if old.get(key) != parameters[key]]
|
||||
if old.keys() != parameters.keys():
|
||||
changed.append("parameter_schema")
|
||||
if changed:
|
||||
if old != compatible_parameters:
|
||||
raise ResumeMismatch(changed)
|
||||
migrating = True
|
||||
else:
|
||||
if any(self.path.glob("*.jsonl")):
|
||||
raise ValueError("journal_missing_inputs")
|
||||
atomic(metadata, encoded(parameters))
|
||||
with working_store(self.path) as self.db:
|
||||
self.rows = {stage: DiskMap(self.db, stage) for stage in STAGES}
|
||||
self._replay()
|
||||
if migrating:
|
||||
atomic(self.path / "inputs-before-memory-fix.json", encoded(old))
|
||||
atomic(metadata, encoded(parameters))
|
||||
self.progress("sourcing")
|
||||
yield self
|
||||
|
||||
def _replay(self):
|
||||
for stage in STAGES:
|
||||
path = self.path / (stage + ".jsonl")
|
||||
if not path.exists():
|
||||
continue
|
||||
with path.open("rb") as handle:
|
||||
while line := handle.readline(MAX_RECORD_BYTES + 1):
|
||||
if len(line) > MAX_RECORD_BYTES:
|
||||
raise ValueError("journal_record_size_cap")
|
||||
# Fail closed on torn/corrupt writes; never silently lose a completed unit.
|
||||
if not line.endswith(b"\n"):
|
||||
raise ValueError("journal_incomplete_record")
|
||||
row = json.loads(line)
|
||||
if row["id"] in self.rows[stage]:
|
||||
raise ValueError("journal_duplicate_unit")
|
||||
self.rows[stage][row["id"]] = compact(stage, row["value"])
|
||||
if row["elapsed_seconds"] >= self.elapsed_before:
|
||||
self.elapsed_before = row["elapsed_seconds"]
|
||||
self.last = {"stage": stage, "id": row["id"]}
|
||||
self.previous_rate = row["rate_limit"]
|
||||
|
||||
def boundary(self):
|
||||
reason = None
|
||||
if (self.path / "STOP").exists():
|
||||
reason = "STOP"
|
||||
elif self.max_seconds is not None and self.clock() - self.started >= self.max_seconds:
|
||||
reason = "max_hours"
|
||||
if reason:
|
||||
self.progress("stopped", reason=reason)
|
||||
raise Stopped(reason)
|
||||
|
||||
def elapsed(self):
|
||||
return self.elapsed_before + self.clock() - self.started
|
||||
|
||||
def progress(self, status, **extra):
|
||||
atomic(
|
||||
self.path / "progress.json",
|
||||
encoded(
|
||||
{
|
||||
"status": status,
|
||||
"counts": {stage: len(rows) for stage, rows in self.rows.items()},
|
||||
"last_unit": self.last,
|
||||
"rate_limit": self.rate_limit() or self.previous_rate,
|
||||
"elapsed_seconds": self.elapsed(),
|
||||
**extra,
|
||||
}
|
||||
),
|
||||
)
|
||||
|
||||
def put(self, stage, key, value):
|
||||
if key in self.rows[stage]:
|
||||
raise ValueError("journal_duplicate_unit")
|
||||
value = compact(stage, value)
|
||||
row = {"id": key, "value": value, "elapsed_seconds": self.elapsed(), "rate_limit": self.rate_limit() or self.previous_rate}
|
||||
path = self.path / (stage + ".jsonl")
|
||||
with path.open("ab") as handle:
|
||||
os.chmod(path, 0o600)
|
||||
payload = encoded(row) + b"\n"
|
||||
if len(payload) > MAX_RECORD_BYTES:
|
||||
raise ValueError("journal_record_size_cap")
|
||||
handle.write(payload)
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
sync_directory(self.path)
|
||||
self.rows[stage][key] = value
|
||||
self.last = {"stage": stage, "id": key}
|
||||
self.progress("sourcing")
|
||||
|
||||
|
||||
class ResolvedGitHub:
|
||||
"""Persist successful repository lookups and 404s used by both eligibility and the fix index."""
|
||||
|
||||
def __init__(self, github, journal):
|
||||
self.github, self.journal = github, journal
|
||||
|
||||
def __getattr__(self, name):
|
||||
return getattr(self.github, name)
|
||||
|
||||
def get_repo(self, name):
|
||||
from .eligibility import RepositoryNotFound, normalize
|
||||
|
||||
key = normalize(name)
|
||||
if key not in self.journal.rows["resolution"]:
|
||||
self.journal.boundary()
|
||||
try:
|
||||
value = self.github.get_repo(key)
|
||||
except RepositoryNotFound:
|
||||
value = None
|
||||
self.journal.put("resolution", key, value)
|
||||
value = self.journal.rows["resolution"][key]
|
||||
if value is None:
|
||||
raise RepositoryNotFound("not_found")
|
||||
return value
|
||||
@@ -1,88 +0,0 @@
|
||||
"""Append-only slice usage; a hash chain detects rewritten rows, not a rebuilt entire history.
|
||||
|
||||
Keep the published final digest to authenticate a history against truncation or wholesale replacement.
|
||||
Only opaque decision/experiment labels belong here; repository identities are never accepted.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import fcntl
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
from datetime import date
|
||||
from pathlib import Path
|
||||
|
||||
FIELDS = {"slice", "decision", "purpose", "date", "result_digest", "freeze_digest"}
|
||||
PURPOSES = {"baseline", "exploratory", "informed", "qualifies"}
|
||||
ZERO = "0" * 64
|
||||
|
||||
|
||||
def digest(row: dict) -> str:
|
||||
return hashlib.sha256(json.dumps(row, sort_keys=True, separators=(",", ":")).encode()).hexdigest()
|
||||
|
||||
|
||||
def validate(row: dict) -> None:
|
||||
if set(row) != FIELDS or row["purpose"] not in PURPOSES:
|
||||
raise ValueError("invalid_usage_fields")
|
||||
if type(row["slice"]) is not int or row["slice"] < 1:
|
||||
raise ValueError("invalid_slice")
|
||||
if not isinstance(row["decision"], str) or not re.fullmatch(r"[A-Za-z0-9_-]+", row["decision"]):
|
||||
raise ValueError("invalid_decision_label")
|
||||
for key in ("result_digest", "freeze_digest"):
|
||||
if not isinstance(row[key], str) or not re.fullmatch(r"[0-9a-f]{64}", row[key]):
|
||||
raise ValueError("invalid_digest")
|
||||
try:
|
||||
if date.fromisoformat(row["date"]).isoformat() != row["date"]:
|
||||
raise ValueError
|
||||
except (ValueError, TypeError):
|
||||
raise ValueError("invalid_date") from None
|
||||
|
||||
|
||||
def _read(text: str) -> list[dict]:
|
||||
previous = ZERO
|
||||
rows = []
|
||||
if text and not text.endswith("\n"):
|
||||
raise ValueError("broken_digest_chain")
|
||||
for line in text.splitlines():
|
||||
try:
|
||||
row = json.loads(line)
|
||||
stored = row.pop("digest")
|
||||
if row["previous"] != previous or digest(row) != stored:
|
||||
raise ValueError
|
||||
validate({k: v for k, v in row.items() if k != "previous"})
|
||||
except (ValueError, KeyError, TypeError, AttributeError):
|
||||
raise ValueError("broken_digest_chain") from None
|
||||
rows.append({**row, "digest": stored})
|
||||
previous = stored
|
||||
return rows
|
||||
|
||||
|
||||
def read(path: Path) -> list[dict]:
|
||||
if not path.exists():
|
||||
return []
|
||||
with path.open() as handle:
|
||||
fcntl.flock(handle, fcntl.LOCK_SH)
|
||||
return _read(handle.read())
|
||||
|
||||
|
||||
def append(path: Path, row: dict) -> str:
|
||||
validate(row)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with path.open("a+") as handle:
|
||||
fcntl.flock(handle, fcntl.LOCK_EX)
|
||||
handle.seek(0)
|
||||
rows = _read(handle.read())
|
||||
if any(all(old[key] == row[key] for key in FIELDS) for old in rows):
|
||||
raise ValueError("duplicate_usage")
|
||||
if rows and rows[0]["freeze_digest"] != row["freeze_digest"]:
|
||||
raise ValueError("freeze_digest_changed")
|
||||
if row["purpose"] == "qualifies" and any(
|
||||
old["slice"] == row["slice"] and old["decision"] == row["decision"] and old["purpose"] == "informed" for old in rows
|
||||
):
|
||||
raise ValueError("slice_informed_decision")
|
||||
entry = {**row, "previous": rows[-1]["digest"] if rows else ZERO}
|
||||
sha = digest(entry)
|
||||
handle.write(json.dumps({**entry, "digest": sha}, sort_keys=True) + "\n")
|
||||
handle.flush()
|
||||
return sha
|
||||
@@ -1,128 +0,0 @@
|
||||
"""Build anonymous coordinator proof records from completed dispatcher outputs."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def digest(value):
|
||||
return hashlib.sha256(json.dumps(value, sort_keys=True).encode()).hexdigest()
|
||||
|
||||
|
||||
def records(root):
|
||||
return [json.loads(path.read_text()) for path in sorted(root.glob("*.json")) if path.name not in {"progress.json", "inputs.json"}]
|
||||
|
||||
|
||||
def replay_proof(roots):
|
||||
lanes = {lane: records(root) for lane, root in roots.items()}
|
||||
if any(len(rows) != 5 for rows in lanes.values()):
|
||||
raise ValueError("proof requires exactly five completed changes per lane")
|
||||
keys = [{row["input_digest"] for row in rows} for rows in lanes.values()]
|
||||
if not all(key == keys[0] for key in keys):
|
||||
raise ValueError("lane input identities differ")
|
||||
quick = {}
|
||||
evidence = {}
|
||||
for lane, rows in lanes.items():
|
||||
quick[lane] = {}
|
||||
evidence[lane] = []
|
||||
for row in rows:
|
||||
inst = row.get("instrument", {})
|
||||
if (
|
||||
row.get("status") not in {"quick_only", "engine_covered"}
|
||||
or not inst.get("hook_bytes_read")
|
||||
or inst.get("suspect_fast")
|
||||
or inst.get("jvm_wall_seconds", 0) < 5
|
||||
):
|
||||
raise ValueError("proof instrument has unread input or implausible JVM time")
|
||||
quick[lane][row["input_digest"]] = digest([t.get("findings", []) for t in row["push"].get("quick_tier", [])])
|
||||
evidence[lane].append(
|
||||
{
|
||||
"input_digest": row["input_digest"],
|
||||
"seconds": inst["seconds"],
|
||||
"jvm_seconds": inst["jvm_wall_seconds"],
|
||||
"bytes": inst["hook_bytes_read"],
|
||||
"exit_code": inst["hook_exit"],
|
||||
}
|
||||
)
|
||||
equal = all(value == next(iter(quick.values())) for value in quick.values())
|
||||
if not equal:
|
||||
raise ValueError("quick-tier findings differ across lanes")
|
||||
return {"lanes": evidence, "quick_tier_equal": True, "quick_digests": quick, "cleanup": "completed-before-checkpoint"}
|
||||
|
||||
|
||||
def engine_proof(root, baseline):
|
||||
current = records(root)
|
||||
old = {digest([r["repo"], r["pin"]]): r for r in records(baseline) if "repo" in r and "pin" in r}
|
||||
current = [r for r in current if "repo" in r and "pin" in r]
|
||||
if len(current) != 2:
|
||||
raise ValueError("engine proof requires exactly two pins")
|
||||
evidence = []
|
||||
|
||||
def stable(record):
|
||||
return [
|
||||
{key: u.get(key) for key in ("unit", "status", "questions_asked", "questions_completed", "witness_rows", "traces")}
|
||||
for u in record["units"]
|
||||
]
|
||||
|
||||
for row in current:
|
||||
key = digest([row["repo"], row["pin"]])
|
||||
previous = old[key]
|
||||
instruments = [u.get("instrument", {}) for u in row["units"]]
|
||||
if (
|
||||
not row.get("done")
|
||||
or not any(i.get("bytes", 0) > 0 for i in instruments)
|
||||
or not any(stage.get("seconds", 0) >= 5 for i in instruments for stage in i.get("jvm", []))
|
||||
):
|
||||
raise ValueError("engine proof lacks bytes or realistic JVM time")
|
||||
evidence.append(
|
||||
{
|
||||
"pin_digest": key,
|
||||
"seconds": row.get("seconds"),
|
||||
"record_digest": digest(stable(row)),
|
||||
"baseline_digest": digest(stable(previous)),
|
||||
"equal": stable(row) == stable(previous),
|
||||
"instrument": [
|
||||
{
|
||||
"bytes": u.get("instrument", {}).get("bytes", 0),
|
||||
"cpg_bytes": u.get("instrument", {}).get("cpg_bytes", 0),
|
||||
"jvm": [
|
||||
{key: stage.get(key) for key in ("seconds", "exit_code", "success", "heap_mb")}
|
||||
for stage in u.get("instrument", {}).get("jvm", [])
|
||||
],
|
||||
}
|
||||
for u in row["units"]
|
||||
],
|
||||
}
|
||||
)
|
||||
return {"pins": evidence, "differences": sum(not e["equal"] for e in evidence), "cleanup": "completed-before-checkpoint"}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("kind", choices=("engine", "replay"))
|
||||
parser.add_argument("--image", required=True)
|
||||
parser.add_argument("--out", type=Path, required=True)
|
||||
parser.add_argument("--records", type=Path)
|
||||
parser.add_argument("--baseline", type=Path)
|
||||
parser.add_argument("--kind-records", type=Path)
|
||||
parser.add_argument("--ax-records", type=Path)
|
||||
parser.add_argument("--vm-records", type=Path)
|
||||
args = parser.parse_args()
|
||||
result = (
|
||||
engine_proof(args.records, args.baseline)
|
||||
if args.kind == "engine"
|
||||
else replay_proof({"kind": args.kind_records, "ax": args.ax_records, "vm": args.vm_records})
|
||||
)
|
||||
result.update(
|
||||
image=args.image, commit=subprocess.run(["git", "rev-parse", "HEAD"], capture_output=True, text=True, check=True).stdout.strip()
|
||||
)
|
||||
args.out.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.out.write_text(json.dumps(result, sort_keys=True, indent=2) + "\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,193 +0,0 @@
|
||||
"""Dispatch frozen unseen changes or a five-change project-history proof."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import subprocess
|
||||
import tarfile
|
||||
import tomllib
|
||||
from pathlib import Path
|
||||
|
||||
from benchmarks.ax.batch import AXLane, DockerLane, Item, Workload, dispatch, write_json
|
||||
from benchmarks.unseen.score import coverage
|
||||
|
||||
|
||||
def make_item(change, arm, image):
|
||||
if arm not in {"default", "long_deadline"}:
|
||||
raise ValueError("unknown replay arm")
|
||||
identity = hashlib.sha256(str(change["id"]).encode()).hexdigest()[:14]
|
||||
spec = {**change, "id": identity, "hook_flags": [] if arm == "default" else ["--deadline", "300"], "image": image}
|
||||
data = json.dumps(spec, sort_keys=True).encode()
|
||||
digest = hashlib.sha256(data).hexdigest()
|
||||
return Item(
|
||||
"replay-" + identity,
|
||||
{"SPEC_URL": data},
|
||||
"result.dat",
|
||||
("python3", "/app/benchmarks/unseen/replay_entry.py"),
|
||||
{},
|
||||
digest,
|
||||
{"SPEC_URL": "spec.dat"},
|
||||
)
|
||||
|
||||
|
||||
def validate_result(data, output):
|
||||
with tarfile.open(fileobj=io.BytesIO(data)) as archive:
|
||||
members = archive.getmembers()
|
||||
required = {"push.json", "push.txt", "engine.json", "instrument.json"}
|
||||
if {m.name for m in members} != required or any(not m.isfile() for m in members):
|
||||
raise ValueError("replay result requires push, engine and instrument.json")
|
||||
payload = {m.name: archive.extractfile(m).read() for m in members}
|
||||
instrument = json.loads(payload["instrument.json"])
|
||||
push = json.loads(payload["push.json"])
|
||||
record = {"instrument": instrument, "push": push}
|
||||
record["status"] = coverage(record)
|
||||
# Transport/scheduler needs an explicit OOM signal, not a quiet failed replay.
|
||||
if "OutOfMemoryError" in json.dumps(push) or "JVM out of memory" in json.dumps(push):
|
||||
record["status"] = "oom"
|
||||
output.mkdir(parents=True, exist_ok=True)
|
||||
for name, body in payload.items():
|
||||
(output / name).write_bytes(body)
|
||||
(output / "result.dat").write_bytes(data)
|
||||
write_json(output / "result.json", record)
|
||||
return record
|
||||
|
||||
|
||||
def proof_changes(root):
|
||||
"""Only the operator's project history; no pool data or fetches."""
|
||||
|
||||
def git(*args):
|
||||
return subprocess.run(["git", "-C", str(root), *args], capture_output=True, text=True, check=True).stdout.strip()
|
||||
|
||||
changes = []
|
||||
for head in git("rev-list", "--no-merges", "--max-count=100", "HEAD", "--", "src").splitlines():
|
||||
base = git("rev-parse", head + "^")
|
||||
changed = git("diff", "--name-only", base, head).splitlines()
|
||||
if not any(name.startswith("src/") and name.endswith(".py") for name in changed):
|
||||
continue
|
||||
changes.append(
|
||||
{
|
||||
"id": hashlib.sha256(head.encode()).hexdigest(),
|
||||
"repository_url": "https://github.com/norandom/OpenUltraSAST.git",
|
||||
"repo": "project-proof",
|
||||
"kind": "ordinary",
|
||||
"base": base,
|
||||
"head": head,
|
||||
}
|
||||
)
|
||||
if len(changes) == 5:
|
||||
break
|
||||
if len(changes) != 5:
|
||||
raise ValueError("history lacks five source changes")
|
||||
return changes
|
||||
|
||||
|
||||
def canonical(value):
|
||||
return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode()
|
||||
|
||||
|
||||
def load_slice(path, slice_id, private_path=None):
|
||||
if path.suffix != ".toml":
|
||||
frozen = json.loads(path.read_text())
|
||||
changes = frozen["changes"]
|
||||
if frozen["sha256"] != hashlib.sha256(json.dumps(changes, sort_keys=True).encode()).hexdigest():
|
||||
raise ValueError("frozen changes digest mismatch")
|
||||
return changes
|
||||
if not path.name.startswith("pool-"):
|
||||
raise ValueError("replay requires a frozen pool, never a draft")
|
||||
pool = tomllib.loads(path.read_text())
|
||||
freeze = json.loads(path.with_name("freeze-" + path.stem.removeprefix("pool-") + ".json").read_text())
|
||||
if hashlib.sha256(canonical(pool)).hexdigest() != freeze["freeze_digest"]:
|
||||
raise ValueError("frozen pool digest mismatch")
|
||||
private_path = private_path or path.parent / "private" / path.name
|
||||
private = {}
|
||||
if private_path.exists():
|
||||
private = {row["id"]: row for row in tomllib.loads(private_path.read_text()).get("repository", [])}
|
||||
changes = []
|
||||
for public in pool.get("repository", []):
|
||||
if str(public["slice"]) != str(slice_id):
|
||||
continue
|
||||
row = public
|
||||
if public.get("private"):
|
||||
row = private.get(public["id"])
|
||||
if row is None:
|
||||
raise ValueError("missing private pool entry")
|
||||
payload = {"url": row["url"], "commits": [[c["base"], c["head"]] for c in row["changes"]]}
|
||||
if hashlib.sha256(canonical(payload)).hexdigest() != public["sha256"]:
|
||||
raise ValueError("private pool digest mismatch")
|
||||
for change in row["changes"]:
|
||||
identity = hashlib.sha256(canonical([public["id"], change["kind"], change["base"], change["head"]])).hexdigest()
|
||||
changes.append(
|
||||
{
|
||||
**change,
|
||||
"id": identity,
|
||||
"repo": public["id"],
|
||||
"repository_url": row["url"],
|
||||
"freeze_digest": freeze["freeze_digest"],
|
||||
"slice": slice_id,
|
||||
}
|
||||
)
|
||||
if not changes:
|
||||
raise ValueError("frozen slice has no changes")
|
||||
return changes
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--manifest", type=Path, required=True, help="frozen JSON: changes array plus sha256 of canonical changes JSON")
|
||||
parser.add_argument("--proof-history", action="store_true", help="write a five-change own-history manifest, then exit")
|
||||
parser.add_argument("--image")
|
||||
parser.add_argument("--private-manifest", type=Path)
|
||||
parser.add_argument("--dry-run", action="store_true")
|
||||
parser.add_argument("--arm", choices=("default", "long_deadline"), default="default")
|
||||
parser.add_argument("--pool", default="p1")
|
||||
parser.add_argument("--slice", default="1")
|
||||
parser.add_argument("--out", type=Path)
|
||||
parser.add_argument("--lane", choices=("ax", "kind", "docker", "mixed"), default="mixed")
|
||||
parser.add_argument("--ax-bin", default="ax")
|
||||
parser.add_argument("--kubeconfig", type=Path)
|
||||
parser.add_argument("--atespace", default="default")
|
||||
args = parser.parse_args()
|
||||
if args.proof_history:
|
||||
changes = proof_changes(Path.cwd())
|
||||
write_json(args.manifest, {"changes": changes, "sha256": hashlib.sha256(json.dumps(changes, sort_keys=True).encode()).hexdigest()})
|
||||
return 0
|
||||
if not args.image:
|
||||
parser.error("--image digest is required")
|
||||
changes = load_slice(args.manifest, args.slice, args.private_manifest)
|
||||
items = [make_item(change, args.arm, args.image) for change in changes]
|
||||
if len({i.id for i in items}) != len(items):
|
||||
parser.error("duplicate change identity")
|
||||
args.out = args.out or Path.home() / "ousast-results/unseen" / args.pool / args.slice / args.arm
|
||||
if args.dry_run:
|
||||
print(json.dumps({"items": len(items), "input_digest": hashlib.sha256(canonical([i.input_digest for i in items])).hexdigest()}))
|
||||
return 0
|
||||
if args.out.resolve().is_relative_to(Path(__file__).resolve().parents[2]):
|
||||
parser.error("raw replay output must be outside the repository")
|
||||
write_json(
|
||||
args.out / "inputs.json",
|
||||
[{"item_id": item.id, "change": change, "input_digest": item.input_digest} for item, change in zip(items, changes, strict=True)],
|
||||
)
|
||||
workload = Workload(args.image, frozenset({"SPEC_URL", "RESULT_URL"}), 600 if args.arm == "default" else 900, validate_result)
|
||||
lanes = ("ax", "ax", "docker") if args.lane == "mixed" else (args.lane,) * (2 if args.lane in {"ax", "kind"} else 1)
|
||||
executors = {}
|
||||
for lane in set(lanes):
|
||||
transport = (DockerLane if lane == "docker" else AXLane)(args, workload)
|
||||
|
||||
def execute(item, output, attempt, transport=transport):
|
||||
record = {**json.loads(item.inputs["SPEC_URL"]), **transport(item, output, attempt)}
|
||||
if (output / "result.dat").exists():
|
||||
# Durable archive lives separately from attempt staging objects, which are deleted.
|
||||
key = f"unseen/{item.input_digest}/result.dat"
|
||||
transport.store._put(key, (output / "result.dat").read_bytes())
|
||||
record["result_object"] = key
|
||||
return record
|
||||
|
||||
executors[lane] = execute
|
||||
return 2 if dispatch(items, workload, args.out, executors, lanes=lanes) else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,219 +0,0 @@
|
||||
"""Credential-free replay actor. Standard library only; PUT is completion."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "learn"))
|
||||
from engine_task_entry import transfer
|
||||
|
||||
CLUSTER_FLAGS = (
|
||||
"-Xms1700m -Xmx1700m -XX:MaxMetaspaceSize=256m -XX:ReservedCodeCacheSize=128m "
|
||||
"-XX:MaxDirectMemorySize=256m -Xss512k -XX:ActiveProcessorCount=2 -XX:+UseG1GC"
|
||||
)
|
||||
|
||||
|
||||
class ReplayFailure(Exception):
|
||||
"""Locally controlled diagnostic only."""
|
||||
|
||||
|
||||
def validate_spec(spec):
|
||||
parts = urlsplit(spec["repository_url"])
|
||||
if (
|
||||
parts.scheme != "https"
|
||||
or not parts.hostname
|
||||
or "." not in parts.hostname
|
||||
or parts.username
|
||||
or parts.password
|
||||
or parts.query
|
||||
or parts.fragment
|
||||
):
|
||||
raise ReplayFailure("repository must be public HTTPS")
|
||||
if parts.port not in (None, 443) or parts.hostname.endswith((".local", ".internal", ".localhost")):
|
||||
raise ReplayFailure("repository must be public HTTPS")
|
||||
import ipaddress
|
||||
|
||||
try:
|
||||
ipaddress.ip_address(parts.hostname)
|
||||
except ValueError:
|
||||
pass
|
||||
else:
|
||||
raise ReplayFailure("repository must use public DNS")
|
||||
if any(not re.fullmatch("[0-9a-f]{40,64}", spec[key]) for key in ("base", "head")):
|
||||
raise ReplayFailure("invalid revision")
|
||||
if spec.get("hook_flags", []) not in ([], ["--deadline", "300"]):
|
||||
raise ReplayFailure("unsupported hook flags")
|
||||
|
||||
|
||||
def main(*, work_root=Path("/tmp"), runner=subprocess.run, heap_profile="cluster", clock=time.monotonic):
|
||||
output = work_root / "out"
|
||||
output.mkdir(parents=True, exist_ok=True)
|
||||
instrument = {
|
||||
"status": "instrument_failure",
|
||||
"head_verified": False,
|
||||
"base_verified": False,
|
||||
"changed_bytes": {},
|
||||
"checkout_files": 0,
|
||||
"checkout_bytes": 0,
|
||||
"hook_bytes_read": 0,
|
||||
"jvm_wall_seconds": 0,
|
||||
"jvm_time_basis": "scan build_seconds + query_seconds",
|
||||
"hook_exit": None,
|
||||
"timings": {},
|
||||
"heap_profile": heap_profile,
|
||||
}
|
||||
spec = {}
|
||||
step = "GET spec"
|
||||
started = clock()
|
||||
|
||||
def run(command, *, timeout=180, env=None):
|
||||
before = clock()
|
||||
result = runner(command, capture_output=True, text=True, timeout=timeout, env=env)
|
||||
instrument["timings"][step] = instrument["timings"].get(step, 0) + clock() - before
|
||||
if result.returncode:
|
||||
raise ReplayFailure(f"exit {result.returncode}")
|
||||
return result
|
||||
|
||||
try:
|
||||
spec_path = work_root / "spec.dat"
|
||||
transfer(os.environ["SPEC_URL"], spec_path, readiness_budget=60)
|
||||
step = "validate spec"
|
||||
spec = json.loads(spec_path.read_text())
|
||||
instrument["image"] = spec.get("image")
|
||||
validate_spec(spec)
|
||||
case = work_root / "case"
|
||||
case.mkdir()
|
||||
git = ["git", "-c", "core.hooksPath=/dev/null", "-c", "protocol.file.allow=never", "-C", str(case)]
|
||||
step = "git init"
|
||||
run([*git, "init", "-q"])
|
||||
run([*git, "remote", "add", "origin", spec["repository_url"]])
|
||||
# One total fetch budget, independent of the hook deadline.
|
||||
fetch_end = clock() + 180
|
||||
for key in ("head", "base"):
|
||||
step = "fetch " + key
|
||||
remaining = fetch_end - clock()
|
||||
if remaining <= 0:
|
||||
raise ReplayFailure("fetch deadline")
|
||||
run([*git, "fetch", "--no-tags", "--depth", "1", "origin", spec[key]], timeout=remaining)
|
||||
step = "verify " + key
|
||||
run([*git, "cat-file", "-e", spec[key] + "^{commit}"])
|
||||
instrument[key + "_verified"] = True
|
||||
step = "checkout"
|
||||
run([*git, "checkout", "--detach", spec["head"]])
|
||||
paths = run([*git, "ls-files", "-z"]).stdout.split("\0")
|
||||
for name in filter(None, paths):
|
||||
path = case / name
|
||||
if path.is_file() and not path.is_symlink():
|
||||
data = path.read_bytes()
|
||||
instrument["checkout_files"] += 1
|
||||
instrument["checkout_bytes"] += len(data)
|
||||
if instrument["checkout_bytes"] > 500 * 1024 * 1024:
|
||||
raise ReplayFailure("checkout size limit")
|
||||
step = "changed files"
|
||||
changed = run([*git, "diff", "--name-only", "--diff-filter=ACMRT", "-z", spec["base"], spec["head"]]).stdout.split("\0")
|
||||
for name in filter(None, changed):
|
||||
path = case / name
|
||||
if path.is_file() and not path.is_symlink() and path.resolve().is_relative_to(case.resolve()):
|
||||
instrument["changed_bytes"][name] = len(path.read_bytes())
|
||||
if not instrument["checkout_bytes"] or not sum(instrument["changed_bytes"].values()):
|
||||
instrument["suspect_fast"] = True
|
||||
raise ReplayFailure("no changed bytes read")
|
||||
step = "pre-push"
|
||||
env = {
|
||||
key: value for key, value in os.environ.items() if key not in {"SPEC_URL", "RESULT_URL", "_JAVA_OPTIONS", "JDK_JAVA_OPTIONS"}
|
||||
}
|
||||
env["OPENULTRASAST_CPG_HEAP_MB"] = "1700" if heap_profile == "cluster" else "2560"
|
||||
env["JAVA_TOOL_OPTIONS"] = CLUSTER_FLAGS if heap_profile == "cluster" else "-Xmx2560m"
|
||||
command = [
|
||||
"ousast",
|
||||
"pre-push",
|
||||
str(case),
|
||||
"--base",
|
||||
spec["base"],
|
||||
"--head",
|
||||
spec["head"],
|
||||
"--artifact",
|
||||
str(output / "push.json"),
|
||||
*spec.get("hook_flags", []),
|
||||
]
|
||||
before = clock()
|
||||
completed = runner(command, capture_output=True, text=True, timeout=330 if spec.get("hook_flags") else 60, env=env)
|
||||
instrument["timings"][step] = clock() - before
|
||||
instrument["hook_exit"] = completed.returncode
|
||||
(output / "push.txt").write_text(completed.stdout)
|
||||
if completed.returncode not in (0, 1):
|
||||
raise ReplayFailure(f"hook exit {completed.returncode}")
|
||||
step = "read push artifact"
|
||||
push = json.loads((output / "push.json").read_text())
|
||||
instrument["hook_bytes_read"] = sum(t.get("bytes_read", 0) for t in push.get("quick_tier", []))
|
||||
scans = push.get("scans", [])
|
||||
instrument["jvm_wall_seconds"] = sum(s.get("scan", {}).get(k, 0) for s in scans for k in ("build_seconds", "query_seconds"))
|
||||
uncovered = "language_not_covered" in json.dumps(push.get("admission", {}).get("coverage_reasons", []))
|
||||
partitions = [part for scan in scans for part in scan.get("scan", {}).get("partitions", [])]
|
||||
unsupported_only = (
|
||||
uncovered
|
||||
and instrument["hook_bytes_read"] == 0
|
||||
and bool(partitions)
|
||||
and all(part.get("status") == "unsupported" for part in partitions)
|
||||
)
|
||||
instrument["suspect_fast"] = bool(scans) and not unsupported_only and instrument["jvm_wall_seconds"] < 5
|
||||
if instrument["suspect_fast"] or (instrument["hook_bytes_read"] <= 0 and not uncovered):
|
||||
instrument["suspect_fast"] = True
|
||||
raise ReplayFailure("suspect_fast or unread hook input")
|
||||
engine = {
|
||||
"repository": spec["repository_url"],
|
||||
"revisions": {key: spec[key] for key in ("base", "head")},
|
||||
"scans": scans,
|
||||
"admission": push.get("admission", {}),
|
||||
}
|
||||
(output / "engine.json").write_text(json.dumps(engine))
|
||||
instrument["status"] = "ok"
|
||||
except Exception as exc:
|
||||
detail = (
|
||||
f"HTTP {exc.code}"
|
||||
if isinstance(exc, urllib.error.HTTPError)
|
||||
else str(exc)
|
||||
if isinstance(exc, ReplayFailure)
|
||||
else type(exc).__name__
|
||||
)
|
||||
instrument["failure"] = {"step": step, "status": detail}
|
||||
# A crash artifact must never be mistaken for a completed scan.
|
||||
if not (output / "push.json").exists():
|
||||
(output / "push.json").write_text("{}")
|
||||
if not (output / "push.txt").exists():
|
||||
(output / "push.txt").write_text("")
|
||||
if not (output / "engine.json").exists():
|
||||
(output / "engine.json").write_text("{}")
|
||||
instrument["seconds"] = clock() - started
|
||||
(output / "instrument.json").write_text(json.dumps(instrument))
|
||||
try:
|
||||
archive = work_root / "result.dat"
|
||||
with tarfile.open(archive, "w") as tar:
|
||||
for name in ("push.json", "push.txt", "engine.json", "instrument.json"):
|
||||
tar.add(output / name, arcname=name)
|
||||
request = urllib.request.Request(
|
||||
os.environ["RESULT_URL"], data=archive.read_bytes(), method="PUT", headers={"Content-Type": "application/x-tar"}
|
||||
)
|
||||
transfer(request, readiness_budget=60)
|
||||
except Exception as exc:
|
||||
detail = f"HTTP {exc.code}" if isinstance(exc, urllib.error.HTTPError) else type(exc).__name__
|
||||
print("PUT result: " + detail, file=sys.stderr)
|
||||
return 1
|
||||
return int(instrument["status"] != "ok")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--heap-profile", choices=("cluster", "vm"), default="cluster")
|
||||
raise SystemExit(main(heap_profile=parser.parse_args().heap_profile))
|
||||
@@ -1,308 +0,0 @@
|
||||
"""Push-level unseen evaluation, explicit coverage and paired adoption evidence."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import random
|
||||
from collections import Counter, defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
from openultrasast.model.taxonomy import load_families
|
||||
from openultrasast.push.report import CAPABILITY_VETOES
|
||||
|
||||
STATES = ("instrument_failure", "unanalysable", "quick_only", "engine_covered")
|
||||
|
||||
|
||||
def coverage(record):
|
||||
instrument = record.get("instrument", {})
|
||||
push = record.get("push")
|
||||
if (
|
||||
not isinstance(push, dict)
|
||||
or not instrument.get("head_verified")
|
||||
or not instrument.get("base_verified")
|
||||
or instrument.get("suspect_fast")
|
||||
or instrument.get("failure")
|
||||
or instrument.get("hook_exit") not in (0, 1)
|
||||
or sum(instrument.get("changed_bytes", {}).values()) <= 0
|
||||
):
|
||||
return "instrument_failure"
|
||||
if "language_not_covered" in json.dumps([push.get("skipped", []), push.get("admission", {}).get("coverage_reasons", [])]) and not any(
|
||||
tier.get("bytes_read", 0) for tier in push.get("quick_tier", [])
|
||||
):
|
||||
return "unanalysable"
|
||||
if instrument.get("hook_bytes_read", 0) <= 0:
|
||||
return "instrument_failure"
|
||||
changed = set(instrument["changed_bytes"])
|
||||
if any(
|
||||
outcome.get("status") == "completed" and outcome.get("identity", {}).get("path") in changed
|
||||
for scan in push.get("scans", [])
|
||||
if scan.get("side") == "head"
|
||||
for outcome in scan.get("scan", {}).get("question_outcomes", [])
|
||||
):
|
||||
return "engine_covered"
|
||||
return "quick_only"
|
||||
|
||||
|
||||
def findings(record):
|
||||
push = record.get("push", {})
|
||||
admission = push.get("admission", {})
|
||||
for defect in admission.get("defects", []):
|
||||
for path, line in defect.get("locations", []):
|
||||
yield "alerts", {**defect, "path": path, "line": line}
|
||||
seen = set()
|
||||
for disposition in admission.get("dispositions", []):
|
||||
delta = disposition.get("candidate", {}).get("delta", {})
|
||||
operation = delta.get("head_operation")
|
||||
if (
|
||||
disposition.get("admitted")
|
||||
or delta.get("novelty") not in {"new", "worsened"}
|
||||
or not set(disposition.get("reasons", [])) <= CAPABILITY_VETOES
|
||||
or operation is None
|
||||
or delta.get("witness") is None
|
||||
or delta.get("defect_id") in seen
|
||||
):
|
||||
continue
|
||||
seen.add(delta.get("defect_id"))
|
||||
yield "advisory", {**operation, "family": delta.get("family")}
|
||||
taxonomy = load_families()
|
||||
for tier in push.get("quick_tier", []):
|
||||
for finding in tier.get("findings", []):
|
||||
family = taxonomy.family_of_cwe(finding.get("cwe") or "")
|
||||
yield "quick", {**finding, "family": family.id if family else None}
|
||||
|
||||
|
||||
def location_match(record, finding):
|
||||
for file in record.get("files", []):
|
||||
if finding.get("path") != file["path"]:
|
||||
continue
|
||||
line = finding.get("line", 0)
|
||||
if any(start - 10 <= line <= end + 10 for start, end in file.get("head_lines", [])):
|
||||
return True
|
||||
if record.get("family") != "config_secrets" and file["path"] == record.get("path", file["path"]):
|
||||
function = record.get("function")
|
||||
if function and function != "<global>" and finding.get("function") == function:
|
||||
return True
|
||||
span = record.get("function_lines") or file.get("function_lines")
|
||||
if span and span[0] <= line <= span[1]:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def catches(record):
|
||||
return sorted(
|
||||
{
|
||||
stream
|
||||
for stream, finding in findings(record)
|
||||
if finding.get("family") == record.get("family") and location_match(record, finding)
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def false_alarm(record, *, alerts_only=False):
|
||||
return any(not alerts_only or stream == "alerts" for stream, _ in findings(record))
|
||||
|
||||
|
||||
def wilson(successes, n):
|
||||
if n == 0:
|
||||
return [None, None]
|
||||
p, z = successes / n, 1.96
|
||||
centre = (p + z * z / (2 * n)) / (1 + z * z / n)
|
||||
half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
|
||||
return [max(0, centre - half), min(1, centre + half)]
|
||||
|
||||
|
||||
def required_n(p, width):
|
||||
for n in range(1, 1000000):
|
||||
low, high = wilson(p * n, n)
|
||||
if max(p - low, high - p) <= width:
|
||||
return n
|
||||
raise ValueError("requested precision exceeds sizing range")
|
||||
|
||||
|
||||
def rate(values):
|
||||
n, k = len(values), sum(values)
|
||||
return {"n": n, "count": k, "rate": k / n if n else None, "wilson95": wilson(k, n)}
|
||||
|
||||
|
||||
def cluster_interval(rows, *, seed=0, draws=2000):
|
||||
clusters = defaultdict(list)
|
||||
for repo, value in rows:
|
||||
clusters[repo].append(value)
|
||||
if not clusters:
|
||||
return [None, None]
|
||||
rng = random.Random(seed)
|
||||
groups = list(clusters.values())
|
||||
samples = []
|
||||
for _ in range(draws):
|
||||
values = [value for group in rng.choices(groups, k=len(groups)) for value in group]
|
||||
samples.append(sum(values) / len(values))
|
||||
samples.sort()
|
||||
return [samples[int(0.025 * (draws - 1))], samples[int(0.975 * (draws - 1))]]
|
||||
|
||||
|
||||
def icc(rows):
|
||||
clusters = defaultdict(list)
|
||||
for repo, value in rows:
|
||||
clusters[repo].append(value)
|
||||
n, k = len(rows), len(clusters)
|
||||
if k < 2 or n <= k:
|
||||
return None
|
||||
mean = sum(v for _, v in rows) / n
|
||||
between = sum(len(v) * (sum(v) / len(v) - mean) ** 2 for v in clusters.values()) / (k - 1)
|
||||
within = sum(sum((x - sum(v) / len(v)) ** 2 for x in v) for v in clusters.values()) / (n - k)
|
||||
size = (n - sum(len(v) ** 2 for v in clusters.values()) / n) / (k - 1)
|
||||
denominator = between + (size - 1) * within
|
||||
return (between - within) / denominator if denominator else 0.0
|
||||
|
||||
|
||||
def mcnemar(a, b):
|
||||
pairs = list(zip(a, b, strict=True))
|
||||
losses = sum(x and not y for x, y in pairs)
|
||||
gains = sum(y and not x for x, y in pairs)
|
||||
n = losses + gains
|
||||
return min(1.0, 2 * sum(math.comb(n, i) for i in range(min(losses, gains) + 1)) / 2**n) if n else 1.0
|
||||
|
||||
|
||||
def score(records, *, seed=0):
|
||||
counts = Counter(coverage(r) for r in records)
|
||||
if not records or counts["instrument_failure"] / len(records) > 0.05:
|
||||
raise ValueError("slice exceeds 5% instrument_failure (or is empty)")
|
||||
usable = [r for r in records if coverage(r) != "instrument_failure"]
|
||||
|
||||
def metrics(rows):
|
||||
ordinary = [r for r in rows if r["kind"] == "ordinary"]
|
||||
vulnerable = [r for r in rows if r["kind"] != "ordinary"]
|
||||
clustered = [(r["repo"], int(false_alarm(r))) for r in ordinary]
|
||||
return {
|
||||
"catch": rate([bool(catches(r)) for r in vulnerable]),
|
||||
"false_alarm": {
|
||||
**rate([false_alarm(r) for r in ordinary]),
|
||||
"cluster95": cluster_interval(clustered, seed=seed),
|
||||
"icc": icc(clustered),
|
||||
},
|
||||
"alerts_only": rate([false_alarm(r, alerts_only=True) for r in ordinary]),
|
||||
}
|
||||
|
||||
breakdown = {}
|
||||
for key in ("family", "method", "post_cutoff"):
|
||||
groups = defaultdict(list)
|
||||
for r in usable:
|
||||
groups[str(r.get(key, "unknown"))].append(r)
|
||||
breakdown[key] = {key: metrics(rows) for key, rows in sorted(groups.items())}
|
||||
breakdown["coverage"] = {state: metrics([r for r in usable if coverage(r) == state]) for state in STATES[1:]}
|
||||
breakdown["stream"] = {
|
||||
stream: rate([stream in catches(r) for r in usable if r["kind"] != "ordinary"]) for stream in ("alerts", "advisory", "quick")
|
||||
}
|
||||
return {
|
||||
**metrics(usable),
|
||||
"total": len(records),
|
||||
"coverage": {state: counts[state] for state in STATES},
|
||||
"breakdowns": breakdown,
|
||||
"confirmed": metrics([r for r in usable if r.get("label_check", r.get("confirmed")) in ("confirmed", True)]),
|
||||
}
|
||||
|
||||
|
||||
def budgets(rows, *, thresholds=None, binary=False):
|
||||
ordinary = [r for r in rows if r["kind"] == "ordinary"]
|
||||
vulnerable = [r for r in rows if r["kind"] != "ordinary"]
|
||||
if binary:
|
||||
fa = rate([r["prediction"] for r in ordinary])
|
||||
return {
|
||||
"catch": rate([r["prediction"] for r in vulnerable]),
|
||||
"false_alarm": fa,
|
||||
"budgets_met": [f"{budget}%" for budget in (1, 5, 10) if fa["rate"] is not None and fa["rate"] <= budget / 100],
|
||||
}
|
||||
output = {}
|
||||
for budget in (1, 5, 10):
|
||||
key = f"{budget}%"
|
||||
if thresholds is not None and key not in thresholds:
|
||||
continue
|
||||
if not ordinary:
|
||||
output[key] = {"label": "not_run"}
|
||||
continue
|
||||
if thresholds is not None:
|
||||
threshold = thresholds[key]
|
||||
else:
|
||||
candidates = sorted({r["score"] for r in rows})
|
||||
candidates.append(math.nextafter(candidates[-1], math.inf))
|
||||
threshold = next(t for t in candidates if sum(r["score"] >= t for r in ordinary) / len(ordinary) <= budget / 100)
|
||||
output[key] = {
|
||||
"threshold": threshold,
|
||||
"label": "fixed" if thresholds is not None else "exploratory",
|
||||
"catch": rate([r["score"] >= threshold for r in vulnerable]),
|
||||
"false_alarm": rate([r["score"] >= threshold for r in ordinary]),
|
||||
}
|
||||
return output
|
||||
|
||||
|
||||
def gate(a, b, *, slice_id=None, informed=False, binary=True, threshold=None, threshold_fixed=False, seed=0):
|
||||
not_run = {"gate": "not_run"}
|
||||
if not slice_id or informed or not a or not b or (not binary and (not threshold_fixed or threshold is None)):
|
||||
return not_run
|
||||
try:
|
||||
score(a, seed=seed)
|
||||
score(b, seed=seed)
|
||||
except ValueError:
|
||||
return not_run
|
||||
aa, bb = {r["id"]: r for r in a}, {r["id"]: r for r in b}
|
||||
if len(aa) != len(a) or len(bb) != len(b) or aa.keys() != bb.keys():
|
||||
return not_run
|
||||
pairs = [
|
||||
(aa[key], bb[key]) for key in sorted(aa) if coverage(aa[key]) != "instrument_failure" and coverage(bb[key]) != "instrument_failure"
|
||||
]
|
||||
if any(x["kind"] != y["kind"] or x["repo"] != y["repo"] for x, y in pairs):
|
||||
return not_run
|
||||
vulnerable = [(x, y) for x, y in pairs if x["kind"] != "ordinary"]
|
||||
ordinary = [(x, y) for x, y in pairs if x["kind"] == "ordinary"]
|
||||
if not vulnerable or not ordinary:
|
||||
return not_run
|
||||
ca = [bool(catches(x)) for x, _ in vulnerable]
|
||||
cb = [bool(catches(y)) if binary else y["score"] >= threshold for _, y in vulnerable]
|
||||
fa = [false_alarm(x) for x, _ in ordinary]
|
||||
fb = [false_alarm(y) if binary else y["score"] >= threshold for _, y in ordinary]
|
||||
catch_p, fa_p = mcnemar(ca, cb), mcnemar(fa, fb)
|
||||
regression = (sum(cb) < sum(ca) and catch_p < 0.05) or (binary and sum(fb) > sum(fa) and fa_p < 0.05)
|
||||
return {
|
||||
"gate": "regression" if regression else "pass",
|
||||
"slice": slice_id,
|
||||
"catch": {"difference": sum(cb) / len(cb) - sum(ca) / len(ca), "mcnemar_p": catch_p},
|
||||
"false_alarm": {
|
||||
"difference": sum(fb) / len(fb) - sum(fa) / len(fa),
|
||||
"mcnemar_p": fa_p,
|
||||
"cluster95": cluster_interval([(x["repo"], int(v) - int(u)) for (x, _), u, v in zip(ordinary, fa, fb, strict=True)], seed=seed),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def load_records(path):
|
||||
if path.is_file():
|
||||
return json.loads(path.read_text())
|
||||
rows = []
|
||||
for entry in json.loads((path / "inputs.json").read_text()):
|
||||
result = path / (hashlib.sha256(entry["item_id"].encode()).hexdigest() + ".json")
|
||||
record = json.loads(result.read_text()) if result.exists() else {}
|
||||
if record.get("input_digest") != entry["input_digest"]:
|
||||
record = {}
|
||||
rows.append({**entry["change"], **record, "id": entry["change"]["id"]})
|
||||
return rows
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("records", type=Path, help="JSON array of joined change/instrument/push records")
|
||||
parser.add_argument("--out", type=Path, required=True)
|
||||
parser.add_argument("--against", type=Path, help="production-arm records for a paired adoption gate")
|
||||
parser.add_argument("--slice-id")
|
||||
parser.add_argument("--informed", action="store_true")
|
||||
args = parser.parse_args()
|
||||
current = load_records(args.records)
|
||||
result = score(current)
|
||||
result["unseen"] = gate(load_records(args.against) if args.against else [], current, slice_id=args.slice_id, informed=args.informed)
|
||||
args.out.write_text(json.dumps(result, sort_keys=True, indent=2) + "\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,305 +0,0 @@
|
||||
"""Advisory selection. Network adapters are constructed only by the coordinator CLI."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
import random
|
||||
import re
|
||||
from collections import OrderedDict
|
||||
from dataclasses import dataclass
|
||||
from datetime import date
|
||||
from urllib.parse import urlencode, urlsplit
|
||||
|
||||
from openultrasast.model.taxonomy import load_families
|
||||
|
||||
from .eligibility import RepositoryNotFound, normalize
|
||||
|
||||
ECOSYSTEMS = ("pip", "npm", "maven", "composer")
|
||||
OSV_ECOSYSTEMS = dict(zip(ECOSYSTEMS, ("PyPI", "npm", "Maven", "Packagist"), strict=True))
|
||||
FAMILIES = {"injection", "path", "deserialization", "access_control", "output_encoding", "untrusted_destination", "config_secrets"}
|
||||
PERMISSIVE = {"MIT", "Apache-2.0", "BSD-2-Clause", "BSD-3-Clause", "ISC", "0BSD", "Unlicense", "Zlib", "PostgreSQL", "BSL-1.0"}
|
||||
FIX = re.compile(r"https://github\.com/([\w.-]+/[\w.-]+)/commit/([a-fA-F0-9]{7,40})(?:[/?#].*)?$", re.I)
|
||||
|
||||
|
||||
class Rejected(ValueError):
|
||||
"""Expected eligibility rejection; messages are fixed reason codes, never names."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Candidate:
|
||||
repository: str
|
||||
url: str
|
||||
fix: str
|
||||
parent: str
|
||||
advisory: str
|
||||
ecosystem: str
|
||||
family: str
|
||||
post_cutoff: bool
|
||||
first_affected: str | None
|
||||
license_spdx: str
|
||||
license_class: str
|
||||
license_ref: str
|
||||
license_path: str
|
||||
|
||||
|
||||
def compact_advisory(row: dict) -> dict:
|
||||
"""Keep exactly the input fields used by selection and the all-advisory fix index."""
|
||||
return {
|
||||
**{key: row[key] for key in ("ghsa_id", "published_at", "type", "withdrawn_at") if key in row},
|
||||
"cwes": [{"cwe_id": c["cwe_id"]} for c in row.get("cwes", [])],
|
||||
"references": [f"https://github.com/{name}/commit/{sha}" for name, sha in sorted(fix_links(row))],
|
||||
"vulnerabilities": [{"package": {key: v["package"][key] for key in ("ecosystem", "name")}} for v in row.get("vulnerabilities", [])],
|
||||
}
|
||||
|
||||
|
||||
def compact_repository(row: dict) -> dict:
|
||||
return {
|
||||
**{key: row[key] for key in ("full_name", "private", "size", "clone_url", "former_names") if key in row},
|
||||
**{key: {"full_name": row[key]["full_name"]} for key in ("parent", "source") if row.get(key)},
|
||||
}
|
||||
|
||||
|
||||
def post_cutoff(row: dict) -> bool:
|
||||
return date.fromisoformat(row["published_at"][:10]) > date(2026, 6, 1)
|
||||
|
||||
|
||||
def fix_links(row: dict) -> set[tuple[str, str]]:
|
||||
links = set()
|
||||
for reference in row.get("references", []):
|
||||
url = reference if isinstance(reference, str) else reference.get("url", "")
|
||||
match = FIX.fullmatch(url)
|
||||
if match:
|
||||
links.add((normalize(match[1]), match[2].lower()))
|
||||
return links
|
||||
|
||||
|
||||
def ecosystem(row: dict) -> str | None:
|
||||
values = {v["package"]["ecosystem"].lower() for v in row.get("vulnerabilities", [])}
|
||||
return next((e for e in ECOSYSTEMS if e in values), None)
|
||||
|
||||
|
||||
def ordered_advisories(rows: list[dict], seed: int) -> list[dict]:
|
||||
"""Stable seeded ordering, post-cutoff first in each ecosystem; round-robin strata."""
|
||||
from .storage import AdvisoryRows
|
||||
|
||||
if isinstance(rows, AdvisoryRows):
|
||||
return rows.ordered(seed)
|
||||
rng = random.Random(seed)
|
||||
groups = []
|
||||
for eco in ECOSYSTEMS:
|
||||
group = sorted((r for r in rows if ecosystem(r) == eco), key=lambda r: r["ghsa_id"])
|
||||
rng.shuffle(group)
|
||||
group.sort(key=lambda r: not post_cutoff(r))
|
||||
groups.append(group)
|
||||
return [group[i] for i in range(max(map(len, groups), default=0)) for group in groups if i < len(group)]
|
||||
|
||||
|
||||
def first_affected(record: dict, row: dict, eco: str) -> str | None:
|
||||
packages = {v["package"]["name"] for v in row["vulnerabilities"] if v["package"]["ecosystem"].lower() == eco}
|
||||
versions = set()
|
||||
for affected in record.get("affected", []):
|
||||
package = affected.get("package", {})
|
||||
if package.get("ecosystem") != OSV_ECOSYSTEMS[eco] or package.get("name") not in packages:
|
||||
continue
|
||||
for span in affected.get("ranges", []):
|
||||
if span.get("type") not in {"ECOSYSTEM", "SEMVER"}:
|
||||
continue
|
||||
for event in span.get("events", []):
|
||||
if event.get("introduced") not in {None, "0"}:
|
||||
versions.add(event["introduced"])
|
||||
if len(versions) > 1:
|
||||
raise Rejected("ambiguous_version_bracket")
|
||||
return next(iter(versions), None)
|
||||
|
||||
|
||||
def license_info(record: dict) -> tuple[str, str]:
|
||||
# Explicit pinned-file SPDX wins. For GitHub-recognized full texts its license endpoint
|
||||
# provides the detected SPDX. Unknown/custom/compound declarations stay private.
|
||||
declared = re.search(r"SPDX-License-Identifier:\s*([^\r\n]+)", record.get("text", ""))
|
||||
spdx = declared[1].strip().removesuffix("*/").strip() if declared else record.get("spdx_id") or "NONE"
|
||||
return spdx, "permissive" if spdx in PERMISSIVE else "private"
|
||||
|
||||
|
||||
def candidate(row: dict, github, osv) -> Candidate:
|
||||
if date.fromisoformat(row["published_at"][:10]) < date(2019, 1, 1):
|
||||
raise Rejected("date")
|
||||
if row.get("type", "reviewed") != "reviewed" or row.get("withdrawn_at"):
|
||||
raise Rejected("advisory_status")
|
||||
eco = ecosystem(row)
|
||||
if eco is None:
|
||||
raise Rejected("ecosystem")
|
||||
taxonomy = load_families()
|
||||
families = {f.id for c in row.get("cwes", []) if (f := taxonomy.family_of_cwe(c["cwe_id"])) and f.id in FAMILIES}
|
||||
if len(families) != 1:
|
||||
raise Rejected("family")
|
||||
links = fix_links(row)
|
||||
if len(links) != 1:
|
||||
raise Rejected("fix_links")
|
||||
name, fix = next(iter(links))
|
||||
repo = github.get_repo(name)
|
||||
url = urlsplit(repo.get("clone_url", ""))
|
||||
if (
|
||||
repo.get("private", True)
|
||||
or not 0 <= repo.get("size", -1) <= 500 * 1024
|
||||
or url.scheme != "https"
|
||||
or url.hostname != "github.com"
|
||||
or url.username
|
||||
or url.password
|
||||
or url.query
|
||||
or url.fragment
|
||||
or normalize(repo["clone_url"]) != normalize(repo["full_name"])
|
||||
):
|
||||
raise Rejected("repository")
|
||||
commit = github.get_commit(repo["full_name"], fix)
|
||||
if len(commit["parents"]) != 1:
|
||||
raise Rejected("parents")
|
||||
parent = commit["parents"][0]["sha"]
|
||||
license_record = github.get_license(repo["full_name"], parent)
|
||||
spdx, license_class = license_info(license_record)
|
||||
return Candidate(
|
||||
normalize(repo["full_name"]),
|
||||
repo["clone_url"],
|
||||
commit["sha"],
|
||||
parent,
|
||||
row["ghsa_id"],
|
||||
eco,
|
||||
next(iter(families)),
|
||||
post_cutoff(row),
|
||||
first_affected(osv.get(row["ghsa_id"]), row, eco),
|
||||
spdx,
|
||||
license_class,
|
||||
parent,
|
||||
license_record.get("path", ""),
|
||||
)
|
||||
|
||||
|
||||
class JSONTransport:
|
||||
"""Coordinator GET transport with bounded retries; all I/O and waits are injectable."""
|
||||
|
||||
def __init__(self, *, opener=None, sleeper=None, clock=None):
|
||||
import time
|
||||
from urllib.request import urlopen
|
||||
|
||||
self.opener = opener or urlopen
|
||||
self.sleeper = sleeper or time.sleep
|
||||
self.clock = clock or time.time
|
||||
self.attempts = 0
|
||||
self.rate_limit = {}
|
||||
|
||||
def get(self, url: str, headers: dict) -> object:
|
||||
from urllib.error import HTTPError
|
||||
from urllib.request import Request
|
||||
|
||||
for attempt in range(5):
|
||||
self.attempts += 1
|
||||
try:
|
||||
with self.opener(Request(url, headers=headers), timeout=60) as response:
|
||||
self.record_rate(getattr(response, "headers", {}))
|
||||
return json.load(response)
|
||||
except HTTPError as exc:
|
||||
self.record_rate(exc.headers)
|
||||
if exc.code == 404:
|
||||
raise RepositoryNotFound("not_found") from None
|
||||
limited = exc.code == 429 or (
|
||||
exc.code == 403 and (exc.headers.get("X-RateLimit-Remaining") == "0" or exc.headers.get("Retry-After"))
|
||||
)
|
||||
transient = exc.code in {500, 502, 503, 504}
|
||||
if attempt == 4 or not (limited or transient):
|
||||
raise ValueError("http_failure") from None
|
||||
try:
|
||||
delay = float(exc.headers.get("Retry-After", "0"))
|
||||
if limited and not delay:
|
||||
delay = max(60, float(exc.headers.get("X-RateLimit-Reset", "0")) - self.clock() + 1)
|
||||
delay = max(delay, 2**attempt)
|
||||
if delay > 3605:
|
||||
raise ValueError("retry_window")
|
||||
except (TypeError, ValueError):
|
||||
raise ValueError("http_retry_failed") from None
|
||||
self.sleeper(delay)
|
||||
except Exception:
|
||||
raise ValueError("http_failure") from None
|
||||
raise ValueError("http_failure")
|
||||
|
||||
def record_rate(self, headers):
|
||||
self.rate_limit = {
|
||||
key: headers.get(header)
|
||||
for key, header in (
|
||||
("remaining", "X-RateLimit-Remaining"),
|
||||
("limit", "X-RateLimit-Limit"),
|
||||
("reset", "X-RateLimit-Reset"),
|
||||
("retry_after", "Retry-After"),
|
||||
)
|
||||
if headers.get(header) is not None
|
||||
}
|
||||
|
||||
|
||||
class GitHub:
|
||||
def __init__(self, token: str, transport=None):
|
||||
if not token:
|
||||
raise ValueError("missing_gh_token")
|
||||
self.transport = transport or JSONTransport()
|
||||
self.headers = {"Authorization": "Bearer " + token, "Accept": "application/vnd.github+json"}
|
||||
self.repos: OrderedDict[str, dict] = OrderedDict()
|
||||
self.calls = 0
|
||||
|
||||
def get(self, path: str):
|
||||
self.calls += 1
|
||||
return self.transport.get("https://api.github.com" + path, self.headers)
|
||||
|
||||
def get_repo(self, name: str) -> dict:
|
||||
name = normalize(name)
|
||||
if name not in self.repos:
|
||||
self.repos[name] = compact_repository(self.get("/repos/" + name))
|
||||
if len(self.repos) > 128:
|
||||
self.repos.popitem(last=False)
|
||||
self.repos.move_to_end(name)
|
||||
return self.repos[name]
|
||||
|
||||
def get_commit(self, name: str, sha: str) -> dict:
|
||||
return self.get("/repos/" + normalize(name) + "/commits/" + sha)
|
||||
|
||||
def get_license(self, name: str, ref: str) -> dict:
|
||||
try:
|
||||
row = self.get("/repos/" + normalize(name) + "/license?" + urlencode({"ref": ref}))
|
||||
except RepositoryNotFound:
|
||||
return {"spdx_id": "NONE", "text": "", "path": ""}
|
||||
return {
|
||||
"spdx_id": (row.get("license") or {}).get("spdx_id"),
|
||||
"path": row.get("path", ""),
|
||||
"text": base64.b64decode(row.get("content", "")).decode("utf-8", errors="replace"),
|
||||
}
|
||||
|
||||
def advisory_page(self, page: int) -> dict:
|
||||
# Oldest first: newly published advisories do not shift already persisted pages.
|
||||
rows = self.get(
|
||||
"/advisories?" + urlencode({"type": "reviewed", "per_page": 100, "page": page, "sort": "published", "direction": "asc"})
|
||||
)
|
||||
if not isinstance(rows, list):
|
||||
raise ValueError("invalid_advisory_page")
|
||||
return {"rows": rows, "done": len(rows) < 100}
|
||||
|
||||
def advisories(self):
|
||||
# Enumerate every reviewed advisory, including non-candidate fixes.
|
||||
page = 1
|
||||
while True:
|
||||
result = self.advisory_page(page)
|
||||
yield from result["rows"]
|
||||
if result["done"]:
|
||||
break
|
||||
page += 1
|
||||
|
||||
|
||||
class OSV:
|
||||
def __init__(self, transport=None):
|
||||
self.transport = transport or JSONTransport()
|
||||
self.calls = 0
|
||||
|
||||
def get(self, identifier: str) -> dict:
|
||||
if not re.fullmatch(r"GHSA-[\w-]+", identifier):
|
||||
raise ValueError("invalid_advisory_id")
|
||||
self.calls += 1
|
||||
try:
|
||||
return self.transport.get("https://api.osv.dev/v1/vulns/" + identifier, {})
|
||||
except RepositoryNotFound:
|
||||
return {} # No OSV bracket is distinct from a failed service call.
|
||||
@@ -1,170 +0,0 @@
|
||||
"""Disposable private SQLite working sets; JSONL remains the durable source of truth."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sqlite3
|
||||
import tempfile
|
||||
from collections.abc import Mapping, Sequence, Set
|
||||
from contextlib import contextmanager
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
@contextmanager
|
||||
def working_store(directory=None):
|
||||
# TemporaryDirectory is mode 0700. No identity-bearing scratch data enters the checkout.
|
||||
with tempfile.TemporaryDirectory(prefix="sourcing-", dir=directory) as temporary:
|
||||
db = sqlite3.connect(Path(temporary) / "working.sqlite")
|
||||
try:
|
||||
db.execute("PRAGMA journal_mode=OFF")
|
||||
db.execute("PRAGMA synchronous=OFF")
|
||||
db.execute("PRAGMA temp_store=FILE")
|
||||
db.execute("PRAGMA cache_size=-2048")
|
||||
db.execute("PRAGMA mmap_size=0")
|
||||
db.execute("CREATE TABLE records (kind TEXT, key TEXT, value TEXT, PRIMARY KEY (kind, key)) WITHOUT ROWID")
|
||||
db.execute("CREATE TABLE sequences (kind TEXT, position INTEGER, value TEXT, PRIMARY KEY (kind, position)) WITHOUT ROWID")
|
||||
yield db
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
class DiskMap(Mapping):
|
||||
def __init__(self, db, kind):
|
||||
self.db, self.kind = db, kind
|
||||
self.count = 0
|
||||
|
||||
def __len__(self):
|
||||
return self.count
|
||||
|
||||
def __iter__(self):
|
||||
for (key,) in self.db.execute("SELECT key FROM records WHERE kind=? ORDER BY key", (self.kind,)):
|
||||
yield key
|
||||
|
||||
def __contains__(self, key):
|
||||
return self.db.execute("SELECT 1 FROM records WHERE kind=? AND key=?", (self.kind, key)).fetchone() is not None
|
||||
|
||||
def __getitem__(self, key):
|
||||
row = self.db.execute("SELECT value FROM records WHERE kind=? AND key=?", (self.kind, key)).fetchone()
|
||||
if row is None:
|
||||
raise KeyError(key)
|
||||
return json.loads(row[0])
|
||||
|
||||
def __setitem__(self, key, value):
|
||||
if key not in self:
|
||||
self.count += 1
|
||||
self.db.execute("INSERT OR REPLACE INTO records VALUES (?, ?, ?)", (self.kind, key, json.dumps(value)))
|
||||
|
||||
def values(self):
|
||||
for (value,) in self.db.execute("SELECT value FROM records WHERE kind=? ORDER BY key", (self.kind,)):
|
||||
yield json.loads(value)
|
||||
|
||||
|
||||
class DiskSequence(Sequence):
|
||||
def __init__(self, db, kind):
|
||||
self.db, self.kind = db, kind
|
||||
self.count = 0
|
||||
|
||||
def __len__(self):
|
||||
return self.count
|
||||
|
||||
def __getitem__(self, position):
|
||||
if position < 0:
|
||||
position += self.count
|
||||
row = self.db.execute("SELECT value FROM sequences WHERE kind=? AND position=?", (self.kind, position)).fetchone()
|
||||
if row is None:
|
||||
raise IndexError(position)
|
||||
return json.loads(row[0])
|
||||
|
||||
def __setitem__(self, position, value):
|
||||
self.db.execute("INSERT OR REPLACE INTO sequences VALUES (?, ?, ?)", (self.kind, position, json.dumps(value)))
|
||||
|
||||
def append(self, value):
|
||||
self[self.count] = value
|
||||
self.count += 1
|
||||
|
||||
def __iter__(self):
|
||||
for (value,) in self.db.execute("SELECT value FROM sequences WHERE kind=? ORDER BY position", (self.kind,)):
|
||||
yield json.loads(value)
|
||||
|
||||
|
||||
class DiskSet(Set):
|
||||
def __init__(self, db, kind):
|
||||
self.rows = DiskMap(db, kind)
|
||||
|
||||
def __len__(self):
|
||||
return len(self.rows)
|
||||
|
||||
def __contains__(self, value):
|
||||
return json.dumps(value) in self.rows
|
||||
|
||||
def __iter__(self):
|
||||
for key in self.rows:
|
||||
value = json.loads(key)
|
||||
yield tuple(value) if isinstance(value, list) else value
|
||||
|
||||
def add(self, value):
|
||||
self.rows[json.dumps(value)] = None
|
||||
|
||||
def update(self, values):
|
||||
for value in values:
|
||||
self.add(value)
|
||||
|
||||
|
||||
class AdvisoryRows(DiskMap):
|
||||
def __iter__(self):
|
||||
return self.values()
|
||||
|
||||
def ordered(self, seed):
|
||||
import random
|
||||
|
||||
from .source import ECOSYSTEMS, ecosystem
|
||||
|
||||
rng = random.Random(seed)
|
||||
groups = [DiskSequence(self.db, "order_" + eco) for eco in ECOSYSTEMS]
|
||||
for row in self.values(): # key order is the original ghsa_id sort
|
||||
eco = ecosystem(row)
|
||||
if eco in ECOSYSTEMS:
|
||||
groups[ECOSYSTEMS.index(eco)].append(row)
|
||||
for group in groups:
|
||||
rng.shuffle(group) # Same Fisher-Yates draws as the in-memory implementation.
|
||||
cursors = [
|
||||
self.db.execute(
|
||||
"SELECT value FROM sequences WHERE kind=? "
|
||||
"ORDER BY substr(json_extract(value, '$.published_at'), 1, 10) <= '2026-06-01', position",
|
||||
(group.kind,),
|
||||
)
|
||||
for group in groups
|
||||
]
|
||||
for _ in range(max(map(len, groups), default=0)):
|
||||
for cursor in cursors:
|
||||
row = cursor.fetchone()
|
||||
if row is not None:
|
||||
yield json.loads(row[0])
|
||||
|
||||
|
||||
class FixIndex:
|
||||
def __init__(self, db):
|
||||
self.db = db
|
||||
db.execute("CREATE TABLE fixes (repository TEXT, sha TEXT, PRIMARY KEY (repository, sha)) WITHOUT ROWID")
|
||||
|
||||
def add(self, repository, sha):
|
||||
self.db.execute("INSERT OR IGNORE INTO fixes VALUES (?, ?)", (repository, sha))
|
||||
|
||||
def get(self, repository, default=None):
|
||||
fixes = RepositoryFixes(self.db, repository)
|
||||
return fixes if fixes else default
|
||||
|
||||
|
||||
class RepositoryFixes(Set):
|
||||
def __init__(self, db, repository):
|
||||
self.db, self.repository = db, repository
|
||||
|
||||
def __len__(self):
|
||||
return self.db.execute("SELECT count(*) FROM fixes WHERE repository=?", (self.repository,)).fetchone()[0]
|
||||
|
||||
def __iter__(self):
|
||||
for (sha,) in self.db.execute("SELECT sha FROM fixes WHERE repository=? ORDER BY sha", (self.repository,)):
|
||||
yield sha
|
||||
|
||||
def __contains__(self, sha):
|
||||
return self.db.execute("SELECT 1 FROM fixes WHERE repository=? AND sha=?", (self.repository, sha)).fetchone() is not None
|
||||
@@ -2,7 +2,7 @@
|
||||
# preserve its runner as PID 1 and put the engine command in the Task manifest.
|
||||
FROM eclipse-temurin:21-jre-jammy AS jre
|
||||
|
||||
FROM ghcr.io/norandom/ax-task-runner:v0.3.1 AS engine
|
||||
FROM ghcr.io/norandom/ax-task-runner:v0.3.1
|
||||
LABEL org.opencontainers.image.licenses="Apache-2.0" \
|
||||
org.opencontainers.image.source="https://github.com/norandom/OpenUltraSAST"
|
||||
|
||||
@@ -50,11 +50,3 @@ RUN python3 -m venv /venv \
|
||||
&& /venv/bin/python -I -c 'import pathlib, runpy, sys; sys.modules.update(yaml=None, boto3=None); sys.path.insert(0, "/app/benchmarks/learn"); p = pathlib.Path("/app/benchmarks/learn/engine_trace_worker.py"); print("Worker import smoke bytes:", len(p.read_bytes())); runpy.run_path(str(p), run_name="engine_worker_import_smoke")'
|
||||
|
||||
# Do not set ENTRYPOINT or CMD: both belong to the upstream AX runner.
|
||||
|
||||
FROM engine AS replay
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends git \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
COPY benchmarks/unseen/replay_entry.py ./benchmarks/unseen/replay_entry.py
|
||||
RUN /venv/bin/pip install --no-cache-dir . \
|
||||
&& /venv/bin/ousast pre-push --help > /dev/null
|
||||
|
||||
@@ -33,6 +33,7 @@ semantic = [
|
||||
"tree-sitter-c>=0.23",
|
||||
"tree-sitter-cpp>=0.23",
|
||||
"tree-sitter-java>=0.23",
|
||||
"tree-sitter-php>=0.23",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
|
||||
@@ -15,6 +15,14 @@ from typing import Any
|
||||
|
||||
|
||||
def add_commands(learn_sub: Any) -> None:
|
||||
mechanisms = learn_sub.add_parser("mechanisms", help="deterministic candidate discharge facts (no admission)")
|
||||
mechanisms_sub = mechanisms.add_subparsers(dest="mechanisms_command", required=True)
|
||||
mine = mechanisms_sub.add_parser("mine", help="mine fix pairs; no model or fact-table writes")
|
||||
mine.add_argument("--memory", help="the example store (default OUSAST_MEMORY)")
|
||||
mine.add_argument("--out", type=Path, required=True, help="candidate JSONL under the results root, outside this repository")
|
||||
mine.add_argument("--profile", choices=("static", "plane"), default="static")
|
||||
mine.add_argument("--spec", type=Path, help="decision engine compile spec defining the fold seed/split")
|
||||
mine.add_argument("--root", type=Path, default=Path("."), help="repository root for the counts-only measurement")
|
||||
memory = learn_sub.add_parser("memory", help="the decision engine's local memory of labelled examples")
|
||||
memory_sub = memory.add_subparsers(dest="memory_command", required=True)
|
||||
build = memory_sub.add_parser("build", help="label + feature record + excerpt -> example rows (no model)")
|
||||
@@ -97,6 +105,8 @@ def add_commands(learn_sub: Any) -> None:
|
||||
|
||||
|
||||
def run(args: argparse.Namespace) -> int:
|
||||
if args.learn_command == "mechanisms":
|
||||
return _mechanisms_mine(args)
|
||||
if args.learn_command == "audit-leaks":
|
||||
return _audit_leaks(args)
|
||||
if args.learn_command == "memory" and args.memory_command == "build":
|
||||
@@ -111,6 +121,25 @@ def run(args: argparse.Namespace) -> int:
|
||||
return 2
|
||||
|
||||
|
||||
def _mechanisms_mine(args: argparse.Namespace) -> int:
|
||||
from ..plane.memory import MemoryStoreError, open_store
|
||||
from ..plane.reconciler import results_root
|
||||
from .mechanisms import mine_store, write_result
|
||||
|
||||
try:
|
||||
root = args.root.resolve()
|
||||
out = args.out.resolve()
|
||||
if out.is_relative_to(root) or not out.is_relative_to(results_root().resolve()):
|
||||
raise ValueError("--out must be under the results root and outside the repository")
|
||||
result = mine_store(open_store(args.memory), profile=args.profile, spec_path=args.spec)
|
||||
write_result(result, out, root=root, results=results_root())
|
||||
except (ValueError, OSError, MemoryStoreError) as exc:
|
||||
print(f"learn mechanisms mine: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
print(json.dumps(result.record, sort_keys=True))
|
||||
return 0 if result.record["pairs_mined"] else 2
|
||||
|
||||
|
||||
def _experiment(args: argparse.Namespace) -> int:
|
||||
"""register | units | run | analyse. Exit 2 on a refused manifest (unregistered, modified, uncommitted), 3 when a
|
||||
ceiling stops the run (``unfinished``: a rerun resumes from the outcome rows)."""
|
||||
|
||||
@@ -0,0 +1,612 @@
|
||||
"""Deterministic discharge proposals, never admitted facts.
|
||||
|
||||
The shared tree-sitter IR supplies function/call identity. Its CST supplies bounded
|
||||
expression containment and structured if branches; loops, exceptions, alias writes,
|
||||
and unmodelled expressions fail closed. Parameters and fact-table sources are
|
||||
potential taint, as in learn.slice. A return is an output boundary only for
|
||||
output_encoding (a proposal, not a claim that every return renders HTML).
|
||||
|
||||
Patterns use zero-based argument positions. Only >=2 independent repository
|
||||
proposers survive. Each fold carries its own training evidence; global evidence
|
||||
must never be used to admit a fact for a held-out fold. No fact table/store writes.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import ast
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
from collections import Counter, defaultdict
|
||||
from collections.abc import Callable, Mapping, Sequence
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from textwrap import dedent
|
||||
from typing import Any
|
||||
|
||||
from ..model.specs import taint_specs
|
||||
from ..plane.memory import MemoryStore, repo_key
|
||||
from ..semantic.cst import _KIND, _walk, file_ir_from_tree
|
||||
from ..semantic.extra import grammar_for
|
||||
from ..semantic.facts import load_facts
|
||||
from ..semantic.ir import FileIR, trailing_name
|
||||
from ..semantic.taint import _call_matches, _source_in
|
||||
from .examples import Example
|
||||
from .folds import Fold, compile_split, folds_digest, outer_folds
|
||||
|
||||
RECORD = Path("benchmarks/measurements/2026-10-04-fix-mechanisms-mining/record.json")
|
||||
SUPPORTED = frozenset({"python", "javascript", "typescript", "tsx", "php"})
|
||||
_NUMBERED = re.compile(r"^\s*(\d+)(?: (.*))?$")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Pair:
|
||||
id: str
|
||||
repository: str
|
||||
group: str
|
||||
language: str
|
||||
family: str
|
||||
vulnerable: str
|
||||
fixed: str
|
||||
source: str = "pairs"
|
||||
frameworks: tuple[str, ...] = ()
|
||||
package: str = ""
|
||||
example_ids: tuple[str, ...] = ()
|
||||
|
||||
|
||||
@dataclass
|
||||
class MiningResult:
|
||||
candidates: list[dict[str, Any]]
|
||||
record: dict[str, Any]
|
||||
|
||||
|
||||
class Unsupported(ValueError):
|
||||
"""A counted skip, never an empty successful analysis."""
|
||||
|
||||
|
||||
def _text(node: Any) -> str:
|
||||
return bytes(node.text).decode("utf-8")
|
||||
|
||||
|
||||
def _hash(value: str) -> str:
|
||||
return hashlib.sha256(value.encode()).hexdigest()
|
||||
|
||||
|
||||
def _canonical(value: Any) -> str:
|
||||
return json.dumps(value, sort_keys=True, separators=(",", ":"))
|
||||
|
||||
|
||||
def _parse(text: str, language: str) -> tuple[Any, FileIR]:
|
||||
if language not in SUPPORTED:
|
||||
raise Unsupported("unsupported_language")
|
||||
grammar = grammar_for(language)
|
||||
if grammar is None:
|
||||
raise Unsupported("grammar_unavailable")
|
||||
lines = text.split("\ndiff:\n", 1)[0].splitlines()
|
||||
matches = [_NUMBERED.match(line) for line in lines]
|
||||
if any(matches):
|
||||
if not all(matches) or len({m[1] for m in matches if m}) != len(lines):
|
||||
raise Unsupported("unparseable")
|
||||
lines = [(m[2] or "") for m in matches if m]
|
||||
code = dedent("\n".join(lines))
|
||||
if language == "php" and not code.lstrip().startswith("<?php"):
|
||||
code = "<?php\n" + code
|
||||
from tree_sitter import Parser
|
||||
|
||||
tree = Parser(grammar).parse(code.encode())
|
||||
ir = file_ir_from_tree(path="excerpt", text=code, language=language, tree=tree)
|
||||
if not ir.parse_ok or not code.strip():
|
||||
raise Unsupported("unparseable")
|
||||
return tree, ir
|
||||
|
||||
|
||||
def _callee(name: str, variables: set[str]) -> str:
|
||||
name = name.strip().replace("->", ".").replace("::", ".")
|
||||
if "(" in name or not re.fullmatch(r"[\w$\\]+(?:\.[\w$]+)*", name):
|
||||
raise Unsupported("unsupported_shape")
|
||||
if name.split(".")[0] in variables or name.startswith("$"):
|
||||
if "." not in name:
|
||||
raise Unsupported("unsupported_shape")
|
||||
return trailing_name(name)
|
||||
return name.lstrip("\\")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Flow:
|
||||
roots: frozenset[str] = frozenset()
|
||||
patterns: frozenset[str] = frozenset()
|
||||
|
||||
|
||||
def _union(flows: Sequence[Flow]) -> Flow:
|
||||
return Flow(frozenset().union(*(f.roots for f in flows)), frozenset().union(*(f.patterns for f in flows)))
|
||||
|
||||
|
||||
@dataclass
|
||||
class Site:
|
||||
callee: str
|
||||
flow: Flow
|
||||
arguments: dict[str, Any]
|
||||
slot: tuple[int, ...]
|
||||
guards: frozenset[str]
|
||||
|
||||
|
||||
class Analysis:
|
||||
"""Small structured walker over the shipped IR's call sites, not a new parser."""
|
||||
|
||||
def __init__(self, language: str, family: str, function: Any, ir: Any) -> None:
|
||||
self.language, self.family = language, family
|
||||
self.kinds = _KIND[language]
|
||||
self.calls = {call.order: call for call in ir.calls}
|
||||
self.variables = set(ir.params) | {bind.name for bind in ir.binds}
|
||||
self.states = {name: Flow(frozenset({name})) for name in ir.params}
|
||||
self.sites: list[Site] = []
|
||||
self.facts = load_facts().for_language(language)
|
||||
self.spec = taint_specs(language=language).get(family)
|
||||
self.function = function
|
||||
|
||||
def name(self, node: Any) -> str:
|
||||
call = self.calls.get(node.end_byte)
|
||||
if call is None:
|
||||
raise Unsupported("unsupported_shape")
|
||||
return _callee(call.name, self.variables)
|
||||
|
||||
def args(self, node: Any) -> list[Any]:
|
||||
args = node.child_by_field_name("arguments")
|
||||
if args is None:
|
||||
args = next((c for c in node.named_children if c.type in self.kinds["args"]), None)
|
||||
return list(args.named_children) if args else []
|
||||
|
||||
def expr(self, node: Any, slot: tuple[int, ...], guards: frozenset[str]) -> Flow:
|
||||
kind = node.type
|
||||
children = list(node.named_children)
|
||||
if kind in {"identifier", "variable_name"}:
|
||||
return self.states.get(_text(node), Flow())
|
||||
if kind in {"string", "string_literal", "integer", "float", "number", "true", "false", "null", "none", "boolean"}:
|
||||
return Flow()
|
||||
if kind in self.kinds["call"]:
|
||||
name = self.name(node)
|
||||
args = self.args(node)
|
||||
values = [self.expr(arg, slot + (i,), guards) for i, arg in enumerate(args)]
|
||||
flow = _union(values)
|
||||
if _source_in(self.calls[node.end_byte].name + "(", self.facts):
|
||||
flow = Flow(flow.roots | {"<source:" + name + ">"}, flow.patterns)
|
||||
arguments = _arguments(args)
|
||||
self.sites.append(Site(name, flow, arguments, slot, guards))
|
||||
added = {_canonical({"callee": name, "argument": i}) for i, value in enumerate(values) if value.roots}
|
||||
return Flow(flow.roots, flow.patterns | added)
|
||||
if kind in {"attribute", "member_expression", "subscript", "subscript_expression", "member_access_expression"}:
|
||||
flow = self.expr(children[0], slot, guards) if children else Flow()
|
||||
if _source_in(_text(node), self.facts):
|
||||
return Flow(flow.roots | {"<source>"}, flow.patterns)
|
||||
return flow
|
||||
if kind in {"keyword_argument", "argument", "parenthesized_expression", "unary_expression", "not_operator"}:
|
||||
return self.expr(children[-1], slot, guards) if children else Flow()
|
||||
if kind in {
|
||||
"binary_expression",
|
||||
"binary_operator",
|
||||
"comparison_operator",
|
||||
"list",
|
||||
"tuple",
|
||||
"array",
|
||||
"array_creation_expression",
|
||||
"array_element_initializer",
|
||||
"pair",
|
||||
"dictionary",
|
||||
"object",
|
||||
}:
|
||||
return _union([self.expr(child, slot + (i,), guards) for i, child in enumerate(children)])
|
||||
if kind in {"property_identifier", "name", "comment", "string_content", "escape_sequence"}:
|
||||
return Flow()
|
||||
raise Unsupported("unsupported_shape")
|
||||
|
||||
def predicate(self, node: Any, truth: bool) -> frozenset[str]:
|
||||
while node.type == "parenthesized_expression":
|
||||
node = node.named_children[0]
|
||||
if node.type in {"not_operator", "unary_expression"}:
|
||||
if not _text(node).lstrip().startswith(("not ", "!")):
|
||||
raise Unsupported("unsupported_shape")
|
||||
return self.predicate(node.named_children[-1], not truth)
|
||||
if node.type not in self.kinds["call"]:
|
||||
raise Unsupported("unsupported_shape")
|
||||
name = self.name(node)
|
||||
predicates = []
|
||||
for index, arg in enumerate(self.args(node)):
|
||||
# Evaluating a condition is not an output path or a sanitizer.
|
||||
value = self.expr(arg, (), frozenset())
|
||||
if value.roots:
|
||||
predicates.append(
|
||||
_canonical(
|
||||
{"callee": name, "argument": index, "predicate": "truthy" if truth else "falsy", "_roots": sorted(value.roots)}
|
||||
)
|
||||
)
|
||||
return frozenset(predicates)
|
||||
|
||||
def block(self, node: Any, prefix: tuple[int, ...] = (), guards: frozenset[str] = frozenset()) -> bool:
|
||||
children = list(node.named_children) if node.type in {"block", "statement_block", "compound_statement"} else [node]
|
||||
for index, statement in enumerate(children):
|
||||
slot = prefix + (index,)
|
||||
kind = statement.type
|
||||
if kind in self.kinds["function"] or kind in {"comment", "pass_statement"}:
|
||||
continue
|
||||
if kind == "if_statement":
|
||||
condition = statement.child_by_field_name("condition")
|
||||
consequence = statement.child_by_field_name("consequence") or statement.child_by_field_name("body")
|
||||
alternative = statement.child_by_field_name("alternative")
|
||||
if condition is None or consequence is None or alternative is not None:
|
||||
raise Unsupported("unsupported_shape")
|
||||
before = dict(self.states)
|
||||
yes, no = self.predicate(condition, True), self.predicate(condition, False)
|
||||
exits = self.block(consequence, slot, guards | yes)
|
||||
# Branch writes need SSA/joins; no invented textual-order dataflow.
|
||||
if self.states != before:
|
||||
raise Unsupported("unsupported_shape")
|
||||
if exits:
|
||||
guards |= no
|
||||
continue
|
||||
if kind in {"return_statement", "echo_statement", "throw_statement", "raise_statement"}:
|
||||
flow = _union([self.expr(c, slot, guards) for c in statement.named_children])
|
||||
if self.family == "output_encoding" and kind in {"return_statement", "echo_statement"}:
|
||||
self.sites.append(Site("<return>" if kind == "return_statement" else "echo", flow, {}, slot, guards))
|
||||
if kind != "echo_statement":
|
||||
return True
|
||||
continue
|
||||
if kind in {"lexical_declaration", "variable_declaration", "expression_statement"}:
|
||||
if self.block(statement.named_children[0], slot, guards):
|
||||
return True
|
||||
if len(statement.named_children) != 1:
|
||||
raise Unsupported("unsupported_shape")
|
||||
continue
|
||||
if kind in self.kinds["assign"]:
|
||||
left = statement.child_by_field_name("left") or statement.child_by_field_name("name")
|
||||
right = statement.child_by_field_name("right") or statement.child_by_field_name("value")
|
||||
if left is None or right is None or left.type not in {"identifier", "variable_name"}:
|
||||
raise Unsupported("unsupported_shape")
|
||||
if kind.startswith("augmented"):
|
||||
raise Unsupported("unsupported_shape")
|
||||
self.states[_text(left)] = self.expr(right, slot, guards)
|
||||
continue
|
||||
if kind in self.kinds["call"]:
|
||||
self.expr(statement, slot, guards)
|
||||
if self.name(statement) in {"abort", "exit", "die"}:
|
||||
return True
|
||||
continue
|
||||
raise Unsupported("unsupported_shape")
|
||||
return False
|
||||
|
||||
def run(self) -> list[Site]:
|
||||
body = self.function.child_by_field_name("body")
|
||||
if body is None:
|
||||
raise Unsupported("unsupported_shape")
|
||||
self.block(body)
|
||||
return self.sites
|
||||
|
||||
def sink(self, site: Site) -> bool:
|
||||
return (self.family == "output_encoding" and site.callee in {"<return>", "echo"}) or bool(
|
||||
self.spec and any(_call_matches(site.callee, sink) for sink in self.spec.sinks)
|
||||
)
|
||||
|
||||
|
||||
def _literal(node: Any) -> tuple[bool, Any]:
|
||||
text = _text(node)
|
||||
if text.lower() in {"true", "false", "null", "none"}:
|
||||
return True, {"true": True, "false": False, "null": None, "none": None}[text.lower()]
|
||||
if node.type not in {"string", "string_literal", "integer", "number", "float"}:
|
||||
return False, None
|
||||
try:
|
||||
value = ast.literal_eval(text)
|
||||
except (ValueError, SyntaxError):
|
||||
return False, None
|
||||
return (True, value) if isinstance(value, str | int | float | bool) else (False, None)
|
||||
|
||||
|
||||
def _arguments(args: list[Any]) -> dict[str, Any]:
|
||||
result: dict[str, Any] = {}
|
||||
for position, original in enumerate(args):
|
||||
arg = original.named_children[-1] if original.type == "argument" else original
|
||||
key = ""
|
||||
if arg.type == "keyword_argument":
|
||||
key = _text(arg.named_children[0])
|
||||
arg = arg.named_children[-1]
|
||||
ok, value = _literal(arg)
|
||||
if ok:
|
||||
result[_canonical({"key": key} if key else {"argument": position, "key": key})] = value
|
||||
elif arg.type in {"array_creation_expression", "dictionary", "object"}:
|
||||
for entry in arg.named_children:
|
||||
if entry.type not in {"array_element_initializer", "pair"} or len(entry.named_children) != 2:
|
||||
continue
|
||||
valid_key, key_value = _literal(entry.named_children[0])
|
||||
valid_value, value = _literal(entry.named_children[-1])
|
||||
if valid_key and isinstance(key_value, str) and valid_value:
|
||||
result[_canonical({"argument": position, "key": key_value})] = value
|
||||
return result
|
||||
|
||||
|
||||
def _functions(tree: Any, ir: FileIR) -> dict[str, tuple[Any, Any]]:
|
||||
nodes = [n for n in _walk(tree.root_node) if n.type in _KIND[ir.language]["function"]]
|
||||
functions = list(ir.functions[1:])
|
||||
if not functions or len({f.name for f in functions}) != len(functions):
|
||||
raise Unsupported("unsupported_shape")
|
||||
return {f.name: (node, f) for node, f in zip(nodes, functions, strict=True)}
|
||||
|
||||
|
||||
def _proposals(before: Analysis, after: Analysis) -> set[tuple[str, str]]:
|
||||
old, new = before.run(), after.run()
|
||||
result: set[tuple[str, str]] = set()
|
||||
old_sinks = [s for s in old if before.sink(s) and s.flow.roots]
|
||||
for sink in old_sinks:
|
||||
# Ambiguous repeated operations are not aligned by textual similarity.
|
||||
matches = [s for s in new if s.callee == sink.callee and s.flow.roots == sink.flow.roots]
|
||||
if len([s for s in old_sinks if s.callee == sink.callee]) != 1:
|
||||
raise Unsupported("ambiguous_sink")
|
||||
if len(matches) == 1:
|
||||
fixed = matches[0]
|
||||
result.update(("sanitizer", p) for p in fixed.flow.patterns - sink.flow.patterns)
|
||||
# Only predicates involving a value on this sink's path are retained below.
|
||||
for encoded in fixed.guards - sink.guards:
|
||||
predicate = json.loads(encoded)
|
||||
roots = set(predicate.pop("_roots"))
|
||||
if roots & fixed.flow.roots:
|
||||
result.add(("guard", _canonical(predicate)))
|
||||
for key, value in fixed.arguments.items():
|
||||
if key not in sink.arguments or _canonical(sink.arguments[key]) != _canonical(value):
|
||||
result.add(("safe_argument", _canonical({"callee": fixed.callee, **json.loads(key), "value": value})))
|
||||
elif not matches:
|
||||
replacements = [s for s in new if s.flow.roots == sink.flow.roots]
|
||||
if len(replacements) == 1 and replacements[0].slot == sink.slot and replacements[0].callee != sink.callee:
|
||||
result.add(("safe_alternative", _canonical({"callee": sink.callee, "replacement": replacements[0].callee})))
|
||||
elif replacements:
|
||||
raise Unsupported("ambiguous_sink")
|
||||
else:
|
||||
raise Unsupported("ambiguous_sink")
|
||||
return result
|
||||
|
||||
|
||||
def _known(language: str, family: str, form: str, pattern: Mapping[str, Any]) -> bool:
|
||||
spec = taint_specs(language=language).get(family)
|
||||
if spec is None:
|
||||
return False
|
||||
name = str(pattern.get("replacement") if form == "safe_alternative" else pattern.get("callee"))
|
||||
if form == "guard" and pattern.get("predicate") != "truthy":
|
||||
return False
|
||||
names = spec.guards if form == "guard" else spec.sanitizers + spec.quoted_sanitizers
|
||||
# Shape sinks alone do not establish a particular keyword/literal restriction.
|
||||
return form != "safe_argument" and any(_call_matches(name, known) for known in names)
|
||||
|
||||
|
||||
def _project_specific(pattern: Mapping[str, Any], pair: Pair, local: set[str]) -> str | None:
|
||||
names = [str(pattern.get(k, "")) for k in ("callee", "replacement")]
|
||||
if any(trailing_name(n) in local for n in names if n):
|
||||
return "local_definition"
|
||||
identities = [pair.repository.rstrip("/").removesuffix(".git").rsplit("/", 1)[-1], pair.package]
|
||||
compact = re.sub(r"[^a-z0-9]", "", _canonical(pattern).lower())
|
||||
if any((token := re.sub(r"[^a-z0-9]", "", identity.lower())) and token in compact for identity in identities):
|
||||
return "project_identifier"
|
||||
return None
|
||||
|
||||
|
||||
def mine_pairs(pairs: Sequence[Pair], *, folds: Sequence[Fold] = ()) -> MiningResult:
|
||||
skips: Counter[str] = Counter()
|
||||
parsed: dict[int, tuple[dict[str, tuple[Any, Any]], dict[str, tuple[Any, Any]]]] = {}
|
||||
definitions: dict[tuple[str, str], set[str]] = defaultdict(set)
|
||||
for index, pair in enumerate(pairs):
|
||||
try:
|
||||
before = _functions(*_parse(pair.vulnerable, pair.language))
|
||||
after = _functions(*_parse(pair.fixed, pair.language))
|
||||
parsed[index] = before, after
|
||||
definitions[repo_key(pair.repository), pair.language].update(before.keys() | after.keys())
|
||||
except Unsupported as exc:
|
||||
skips[str(exc)] += 1
|
||||
proposals: dict[tuple[str, str, str, str], list[Pair]] = defaultdict(list)
|
||||
mined = 0
|
||||
for index, (old, new) in parsed.items():
|
||||
pair = pairs[index]
|
||||
try:
|
||||
common = old.keys() & new.keys()
|
||||
if not common:
|
||||
raise Unsupported("unmatched_function")
|
||||
found: set[tuple[str, str]] = set()
|
||||
for name in sorted(common):
|
||||
found |= _proposals(Analysis(pair.language, pair.family, *old[name]), Analysis(pair.language, pair.family, *new[name]))
|
||||
mined += 1
|
||||
if not found:
|
||||
skips["no_supported_change"] += 1
|
||||
for form, pattern in sorted(found):
|
||||
reason = _project_specific(json.loads(pattern), pair, definitions[repo_key(pair.repository), pair.language])
|
||||
if reason:
|
||||
skips[reason] += 1
|
||||
else:
|
||||
proposals[pair.language, pair.family, form, pattern].append(pair)
|
||||
except Unsupported as exc:
|
||||
skips[str(exc)] += 1
|
||||
candidates = []
|
||||
cells: dict[str, dict[str, int]] = {}
|
||||
tops: dict[str, list[dict[str, Any]]] = defaultdict(list)
|
||||
for (language, family, form, encoded), evidence in sorted(proposals.items()):
|
||||
repos = {_hash(repo_key(p.repository)) for p in evidence}
|
||||
pattern = json.loads(encoded)
|
||||
known = _known(language, family, form, pattern)
|
||||
cell = cells.setdefault(f"{language}/{family}/{form}", {"total": 0, "at_least_two_repositories": 0, "known": 0})
|
||||
cell["total"] += 1
|
||||
cell["known"] += int(known)
|
||||
if len(repos) < 2:
|
||||
skips["fewer_than_two_repositories"] += 1
|
||||
continue
|
||||
cell["at_least_two_repositories"] += 1
|
||||
candidate = {
|
||||
"language": language,
|
||||
"family": family,
|
||||
"form": form,
|
||||
"pattern": pattern,
|
||||
"known": known,
|
||||
"proposing_pairs": sorted({p.id for p in evidence}),
|
||||
"proposing_examples": {p.id: sorted(p.example_ids) for p in sorted(evidence, key=lambda p: p.id)},
|
||||
"repository_count": len(repos),
|
||||
"repository_ids": sorted(repos),
|
||||
}
|
||||
training = []
|
||||
for fold in folds:
|
||||
held_repos = {repo_key(p.repository) for p in pairs if p.group in fold.eval_groups}
|
||||
train = [
|
||||
p
|
||||
for p in evidence
|
||||
if p.group not in fold.eval_groups
|
||||
and repo_key(p.repository) not in held_repos
|
||||
and p.source != fold.held_source
|
||||
and (not fold.held_framework or fold.held_framework not in p.frameworks)
|
||||
]
|
||||
train_repos = sorted({_hash(repo_key(p.repository)) for p in train})
|
||||
training.append(
|
||||
{
|
||||
"fold": fold.name,
|
||||
"repository_count": len(train_repos),
|
||||
"repository_ids": train_repos,
|
||||
"proposing_pairs": sorted({p.id for p in train}),
|
||||
"supported": len(train_repos) >= 2,
|
||||
"eligible": len(train_repos) >= 2 and not known and fold.status == "ok",
|
||||
}
|
||||
)
|
||||
candidate["folds"] = training
|
||||
candidates.append(candidate)
|
||||
tops[form].append({"language": language, "family": family, "pattern": pattern, "repository_count": len(repos)})
|
||||
return MiningResult(
|
||||
candidates,
|
||||
{
|
||||
"pairs_read": len(pairs),
|
||||
"pairs_mined": mined,
|
||||
"skips": dict(sorted(skips.items())),
|
||||
"candidates": cells,
|
||||
"top_patterns": {
|
||||
form: sorted(rows, key=lambda r: (-r["repository_count"], _canonical(r)))[:10] for form, rows in sorted(tops.items())
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def pairs_from_rows(
|
||||
rows: Sequence[Mapping[str, Any]], excerpt_text: Callable[[str], str | None], *, profile: str = "static"
|
||||
) -> tuple[list[Pair], Counter[str], int]:
|
||||
"""Never pair on repository group alone. Legacy rows use an unambiguous candidate.
|
||||
|
||||
Explicit pair/source_ref keys take precedence; repeated revisions/profiles and
|
||||
missing sides are counted, not guessed. Delta rows duplicate pin evidence.
|
||||
"""
|
||||
from .labels import load_sources
|
||||
|
||||
allowed = {source.id for source in load_sources().sources}
|
||||
buckets: dict[tuple[str, str, str, str, str, str], list[Mapping[str, Any]]] = defaultdict(list)
|
||||
skips: Counter[str] = Counter()
|
||||
for row in rows:
|
||||
if row.get("profile", "static") != profile or row.get("unit", "pin") != "pin":
|
||||
continue
|
||||
source = str(row.get("source", ""))
|
||||
if source not in allowed or row.get("origin") == "user" or row.get("conditional"):
|
||||
skips["excluded_source"] += 1
|
||||
continue
|
||||
repo, group = str(row.get("repo") or ""), str(row.get("group") or "")
|
||||
candidate = str(row.get("candidate") or "")
|
||||
if not repo or not group or "/" not in repo or not candidate:
|
||||
skips["missing_pair_identity"] += 1
|
||||
continue
|
||||
explicit = str(row.get("pair_id") or row.get("pair") or row.get("source_ref") or "")
|
||||
identity = explicit or candidate
|
||||
key = (repo_key(repo), group, source, str(row.get("language", "")), str(row.get("family", "")), identity)
|
||||
buckets[key].append(row)
|
||||
pairs: list[Pair] = []
|
||||
for key, members in sorted(buckets.items()):
|
||||
sides = {label: [r for r in members if r.get("label") == label] for label in (0, 1)}
|
||||
if len(members) != 2 or any(len(side) != 1 for side in sides.values()):
|
||||
skips["incomplete_pair" if any(not s for s in sides.values()) else "ambiguous_pair"] += 1
|
||||
continue
|
||||
fixed, vulnerable = sides[0][0], sides[1][0]
|
||||
if any(r.get("pin_role", role) != role for r, role in ((fixed, "fixed"), (vulnerable, "vulnerable"))):
|
||||
skips["label_side_mismatch"] += 1
|
||||
continue
|
||||
texts = [excerpt_text(str(r.get("excerpt_sha") or "")) for r in (vulnerable, fixed)]
|
||||
if any(not text for text in texts):
|
||||
skips["unreadable_excerpt"] += 1
|
||||
continue
|
||||
repo, group, source, language, family, identity = key
|
||||
if not any(vulnerable.get(k) for k in ("pair_id", "pair", "source_ref")):
|
||||
identity = "legacy-" + _hash(_canonical(key))
|
||||
pairs.append(
|
||||
Pair(
|
||||
identity,
|
||||
repo,
|
||||
group,
|
||||
language,
|
||||
family,
|
||||
texts[0] or "",
|
||||
texts[1] or "",
|
||||
source,
|
||||
tuple(sorted(set(vulnerable.get("frameworks") or ()) | set(fixed.get("frameworks") or ()))),
|
||||
str(vulnerable.get("package") or ""),
|
||||
tuple(sorted(str(r["id"]) for r in members if r.get("id"))),
|
||||
)
|
||||
)
|
||||
return pairs, skips, len(buckets)
|
||||
|
||||
|
||||
def mine_store(store: MemoryStore, *, profile: str = "static", spec_path: Path | None = None) -> MiningResult:
|
||||
from .compile import load_spec
|
||||
from .labels import load_sources
|
||||
|
||||
rows = [r.row for r in store.rows(kind="example")]
|
||||
allowed = {source.id for source in load_sources().sources}
|
||||
examples = [
|
||||
Example.from_row(row)
|
||||
for row in rows
|
||||
if row.get("profile") == profile and row.get("source") in allowed and row.get("origin") != "user" and not row.get("conditional")
|
||||
]
|
||||
spec = load_spec(spec_path) if spec_path else load_spec()
|
||||
split = compile_split(examples, fraction=float(spec["split"].get("compile_fraction", 0.25)), seed=spec.seed)
|
||||
folds = outer_folds(examples, seed=spec.seed, exclude=split.groups)
|
||||
read_bytes = 0
|
||||
blobs = 0
|
||||
|
||||
def excerpt_text(sha: str) -> str | None:
|
||||
nonlocal read_bytes, blobs
|
||||
if not re.fullmatch(r"[0-9a-f]{64}", sha):
|
||||
return None
|
||||
data = store.get_blob("excerpts", sha)
|
||||
if data is None:
|
||||
return None
|
||||
read_bytes += len(data)
|
||||
blobs += 1
|
||||
try:
|
||||
return data.decode("utf-8")
|
||||
except UnicodeDecodeError:
|
||||
return None
|
||||
|
||||
pairs, skips, pairs_read = pairs_from_rows(rows, excerpt_text, profile=profile)
|
||||
result = mine_pairs(pairs, folds=folds)
|
||||
combined = Counter(result.record["skips"]) + skips
|
||||
result.record.update(
|
||||
{
|
||||
"rows_read": len(rows),
|
||||
"pairs_read": pairs_read,
|
||||
"complete_pairs": len(pairs),
|
||||
"excerpt_blobs_read": blobs,
|
||||
"excerpt_bytes_read": read_bytes,
|
||||
"skips": dict(sorted(combined.items())),
|
||||
"folds_digest": folds_digest(split, folds),
|
||||
"compile_spec_sha256": spec.sha256,
|
||||
"fold_seed": spec.seed,
|
||||
"profile": profile,
|
||||
"status": "mined" if result.record["pairs_mined"] else "no_mineable_pairs",
|
||||
}
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def write_result(result: MiningResult, out: Path, *, root: Path, results: Path) -> None:
|
||||
"""Counts-only checked-in record; candidate evidence must stay under results outside the repo."""
|
||||
out, root, results = out.resolve(), root.resolve(), results.resolve()
|
||||
if out.is_relative_to(root) or not out.is_relative_to(results):
|
||||
raise ValueError("--out must be under the results root and outside the repository")
|
||||
record = root / RECORD
|
||||
if not record.resolve().is_relative_to(root):
|
||||
raise ValueError("measurement record must stay inside the repository")
|
||||
out.parent.mkdir(parents=True, exist_ok=True)
|
||||
out.write_text("".join(_canonical(candidate) + "\n" for candidate in result.candidates), encoding="utf-8")
|
||||
record.parent.mkdir(parents=True, exist_ok=True)
|
||||
record.write_text(json.dumps(result.record, indent=2, sort_keys=True) + "\n", encoding="utf-8")
|
||||
@@ -615,12 +615,9 @@ class S3Store(MemoryStore):
|
||||
presigned URLs. An admin configures the bucket once; opening a store verifies it (:meth:`verify_bucket`) and
|
||||
refuses a bucket without versioning, the ``runs/`` expiry rule or S3 Select. There is no local fallback."""
|
||||
|
||||
read_only: bool = False
|
||||
|
||||
def __init__(self, client: ObjectClient, bucket: str, prefix: str = "", *, read_only: bool = False) -> None:
|
||||
def __init__(self, client: ObjectClient, bucket: str, prefix: str = "") -> None:
|
||||
self.client, self.bucket, self.prefix = client, bucket, prefix.strip("/")
|
||||
self.read_only = read_only
|
||||
if not read_only and self.describe() not in _VERIFIED:
|
||||
if self.describe() not in _VERIFIED:
|
||||
self.verify_bucket()
|
||||
_VERIFIED.add(self.describe())
|
||||
|
||||
@@ -634,11 +631,9 @@ class S3Store(MemoryStore):
|
||||
return self.client.get(self._k(key))
|
||||
|
||||
def _put(self, key: str, data: bytes, labels: Mapping[str, str] | None = None) -> None:
|
||||
self._writable()
|
||||
self.client.put(self._k(key), data, dict(labels or {}))
|
||||
|
||||
def _delete(self, key: str) -> None:
|
||||
self._writable()
|
||||
self.client.delete(self._k(key))
|
||||
|
||||
def _keys(self, prefix: str) -> list[str]:
|
||||
@@ -652,16 +647,11 @@ class S3Store(MemoryStore):
|
||||
tagged = self.client.tags(self._k(key)).get("kind")
|
||||
return tagged is None or tagged == "*" or kind in tagged.split()
|
||||
|
||||
def _writable(self) -> None:
|
||||
if self.read_only:
|
||||
raise MemoryStoreError("read-only store")
|
||||
|
||||
def verify_bucket(self) -> None:
|
||||
"""Check the admin's one-time setup, writing only the capability probe: versioning ``Enabled``, an enabled
|
||||
"""Read-only check that the admin's one-time setup is in place: versioning ``Enabled``, an enabled
|
||||
lifecycle rule expiring ``<prefix>/runs/`` and none expiring the whole store, S3 Select answering a probe
|
||||
over JSON Lines, and the probe's object tags readable. Every missing piece is named with its admin action
|
||||
in one :class:`MemoryStoreError`."""
|
||||
self._writable()
|
||||
runs = self._k(RUNS_PREFIX)
|
||||
endpoint = '--endpoint-url "$S3_ENDPOINT"'
|
||||
missing: list[str] = []
|
||||
@@ -749,7 +739,6 @@ class S3Store(MemoryStore):
|
||||
) from exc
|
||||
|
||||
def presign_put(self, key: str, expires: timedelta = timedelta(hours=1)) -> str:
|
||||
self._writable()
|
||||
return self.client.presign("PUT", self._k(key), expires)
|
||||
|
||||
def presign_get(self, key: str, expires: timedelta = timedelta(hours=1)) -> str:
|
||||
@@ -926,10 +915,9 @@ def s3_settings(environ: Mapping[str, str] | None = None) -> dict[str, Any]:
|
||||
return settings
|
||||
|
||||
|
||||
def open_store(spec: str | None = None, environ: Mapping[str, str] | None = None, *, read_only: bool = False) -> MemoryStore:
|
||||
def open_store(spec: str | None = None, environ: Mapping[str, str] | None = None) -> MemoryStore:
|
||||
"""The store ``spec`` (else ``OUSAST_MEMORY``, else ``file://<results_root()>/memory``) names. ``s3://`` with no
|
||||
bucket uses ``S3_BUCKET``. ``read_only`` skips the S3 write/Select capability probe and refuses S3 mutations;
|
||||
it is intended for GET/list inventory readers, which do not need Select. Default opening is unchanged."""
|
||||
bucket uses ``S3_BUCKET``."""
|
||||
from .reconciler import results_root
|
||||
|
||||
env = environ if environ is not None else os.environ
|
||||
@@ -944,7 +932,7 @@ def open_store(spec: str | None = None, environ: Mapping[str, str] | None = None
|
||||
if not bucket:
|
||||
raise MemoryStoreError("OUSAST_MEMORY=s3:// names no bucket and S3_BUCKET is unset: set one of them")
|
||||
settings = s3_settings(environ)
|
||||
return S3Store(S3Client(bucket=bucket, **settings), bucket, parts.path, read_only=read_only)
|
||||
return S3Store(S3Client(bucket=bucket, **settings), bucket, parts.path)
|
||||
raise MemoryStoreError(f"OUSAST_MEMORY must be file:///<path> or s3://<bucket>[/<prefix>], got {text!r}")
|
||||
|
||||
|
||||
|
||||
@@ -43,6 +43,14 @@ _KIND = {
|
||||
_KIND["cpp"] = _KIND["c"]
|
||||
_KIND["typescript"] = _KIND["javascript"]
|
||||
_KIND["tsx"] = _KIND["javascript"]
|
||||
_KIND["php"] = {
|
||||
"function": frozenset({"function_definition", "method_declaration"}),
|
||||
"assign": frozenset({"assignment_expression", "augmented_assignment_expression"}),
|
||||
"call": frozenset({"function_call_expression", "member_call_expression", "scoped_call_expression"}),
|
||||
"args": frozenset({"arguments"}),
|
||||
"string": frozenset({"string", "encapsed_string"}),
|
||||
"params": frozenset({"formal_parameters"}),
|
||||
}
|
||||
|
||||
_CONST = frozenset(
|
||||
{
|
||||
@@ -183,6 +191,11 @@ def _call(node: Any, kinds: dict[str, frozenset[str]]) -> CallSite:
|
||||
|
||||
|
||||
def _callee_name(node: Any, args_node: Any) -> str:
|
||||
if node.type in {"member_call_expression", "scoped_call_expression"}:
|
||||
receiver = node.child_by_field_name("object") or node.child_by_field_name("scope")
|
||||
name = node.child_by_field_name("name")
|
||||
if receiver is not None and name is not None:
|
||||
return f"{_text(receiver)}->{_text(name)}"
|
||||
idents = [
|
||||
_text(child)
|
||||
for child in getattr(node, "named_children", ()) or ()
|
||||
@@ -207,7 +220,7 @@ def _call_name(node: Any, kinds: dict[str, frozenset[str]]) -> str | None:
|
||||
|
||||
def _function_name(node: Any) -> str:
|
||||
for child in getattr(node, "named_children", ()) or ():
|
||||
if child.type == "identifier":
|
||||
if child.type in {"identifier", "name"}:
|
||||
return _text(child)
|
||||
if child.type == "function_declarator":
|
||||
for grandchild in getattr(child, "named_children", ()) or ():
|
||||
@@ -240,7 +253,7 @@ def _left_name(node: Any) -> str:
|
||||
def _identifiers(node: Any) -> tuple[str, ...]:
|
||||
names: list[str] = []
|
||||
for child in _walk(node):
|
||||
if child.type == "identifier":
|
||||
if child.type in {"identifier", "variable_name"}:
|
||||
names.append(_text(child))
|
||||
return tuple(names)
|
||||
|
||||
|
||||
@@ -17,6 +17,7 @@ _GRAMMAR_MODULES = {
|
||||
"c": ("tree_sitter_c", "language"),
|
||||
"cpp": ("tree_sitter_cpp", "language"),
|
||||
"java": ("tree_sitter_java", "language"),
|
||||
"php": ("tree_sitter_php", "language_php"),
|
||||
}
|
||||
_PROBE_ENV = "OPENULTRASAST_TREE_SITTER_PROBE"
|
||||
_PROBE_OFF = frozenset({"0", "off", "false", "unavailable", "no"})
|
||||
|
||||
@@ -1,159 +0,0 @@
|
||||
"""Offline shared dispatcher contracts."""
|
||||
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
|
||||
from benchmarks.ax.batch import Item, Workload, dispatch, manifest
|
||||
|
||||
IMAGE = "registry/task@sha256:" + "a" * 64
|
||||
|
||||
|
||||
def item(identity="one"):
|
||||
return Item(identity, {"SPEC_URL": b"{}"}, "result.dat", ("python3", "/entry.py"), {}, identity)
|
||||
|
||||
|
||||
def workload():
|
||||
return Workload(IMAGE, frozenset({"SPEC_URL", "RESULT_URL"}), 900, lambda data, output: {})
|
||||
|
||||
|
||||
def test_manifest_contract():
|
||||
doc = manifest(
|
||||
"ousast-engine-replay-abc",
|
||||
workload(),
|
||||
item(),
|
||||
{"SPEC_URL": "https://files.example/spec.dat", "RESULT_URL": "https://files.example/result.dat"},
|
||||
)
|
||||
assert {v["name"] for v in doc["spec"]["env"]} == {"SPEC_URL", "RESULT_URL"}
|
||||
with pytest.raises(ValueError):
|
||||
manifest("bad", workload(), item(), {})
|
||||
bad = item()
|
||||
bad.extra_env["AWS_SECRET_ACCESS_KEY"] = "secret"
|
||||
with pytest.raises(ValueError):
|
||||
manifest(
|
||||
"ousast-engine-test",
|
||||
workload(),
|
||||
bad,
|
||||
{"SPEC_URL": "https://files.example/spec.dat", "RESULT_URL": "https://files.example/result.dat"},
|
||||
)
|
||||
|
||||
|
||||
def test_dispatch_retry_fallback_resume_stop(tmp_path):
|
||||
calls = []
|
||||
|
||||
def lane(it, output, attempt):
|
||||
calls.append(("ax", attempt))
|
||||
return {"status": "sandbox_failure"}
|
||||
|
||||
def vm(it, output, attempt):
|
||||
calls.append(("docker", attempt))
|
||||
return {"status": "ok"}
|
||||
|
||||
assert not dispatch([item()], workload(), tmp_path, {"ax": lane, "docker": vm}, lanes=("ax", "ax", "docker"), overflow=False)
|
||||
assert [c[0] for c in calls] == ["ax", "ax", "docker"]
|
||||
assert not dispatch([item()], workload(), tmp_path, {"ax": lane, "docker": vm})
|
||||
assert len(calls) == 3
|
||||
(tmp_path / "STOP").touch()
|
||||
dispatch([item("two")], workload(), tmp_path, {"ax": lane, "docker": vm})
|
||||
assert len(calls) == 3
|
||||
assert json.loads((tmp_path / "progress.json").read_text())["pins_in_flight"] == 0
|
||||
|
||||
|
||||
def test_cleanup_failure_halts(tmp_path):
|
||||
calls = []
|
||||
|
||||
def broken(it, output, attempt):
|
||||
calls.append(it.id)
|
||||
raise RuntimeError("cleanup failed")
|
||||
|
||||
with pytest.raises(RuntimeError, match="cleanup"):
|
||||
dispatch([item("one"), item("two")], workload(), tmp_path, {"ax": broken}, lanes=("ax",))
|
||||
assert calls == ["one"]
|
||||
|
||||
|
||||
def test_ax_lane_fake_cli_and_store(tmp_path, monkeypatch):
|
||||
import subprocess
|
||||
from types import SimpleNamespace
|
||||
|
||||
from benchmarks.ax.batch import AXLane
|
||||
|
||||
calls = []
|
||||
|
||||
class Store:
|
||||
objects = {}
|
||||
|
||||
def _put(self, key, data):
|
||||
self.objects[key] = data
|
||||
|
||||
def _get(self, key):
|
||||
return (b"completed", None)
|
||||
|
||||
def _delete(self, key):
|
||||
self.objects.pop(key, None)
|
||||
|
||||
def presign_get(self, key, expiry):
|
||||
assert key.endswith(".dat")
|
||||
return "https://files.example/" + key + "?signature=secret"
|
||||
|
||||
presign_put = presign_get
|
||||
|
||||
def run(command, **kwargs):
|
||||
calls.append((command, kwargs))
|
||||
if command[1] == "get":
|
||||
name = json.loads(calls[0][1]["input"])["metadata"]["name"]
|
||||
return subprocess.CompletedProcess(command, 0, json.dumps({"metadata": {"name": name}, "status": {"phase": "Running"}}), "")
|
||||
return subprocess.CompletedProcess(command, 0, "", "")
|
||||
|
||||
monkeypatch.setattr(subprocess, "run", run)
|
||||
monkeypatch.setenv("S3_ENDPOINT", "https://files.example")
|
||||
|
||||
def validate(data, output):
|
||||
assert data == b"completed"
|
||||
(output / "result.json").write_text('{"status":"ok"}')
|
||||
|
||||
work = Workload(IMAGE, frozenset({"SPEC_URL", "RESULT_URL"}), 10, validate)
|
||||
lane = AXLane(SimpleNamespace(ax_bin="fake-ax", kubeconfig=None, atespace="default"), work, Store())
|
||||
assert lane(item(), tmp_path, 0)["status"] == "ok"
|
||||
doc = json.loads(calls[0][1]["input"])
|
||||
assert doc["metadata"]["name"].startswith("ousast-engine-") and len(doc["metadata"]["name"]) <= 49
|
||||
assert {e["name"] for e in doc["spec"]["env"]} == {"SPEC_URL", "RESULT_URL"}
|
||||
assert calls[-1][0][1:3] == ["delete", "task"]
|
||||
assert not lane.store.objects
|
||||
|
||||
|
||||
def test_bounded_lanes_oom_and_instrument_retry(tmp_path):
|
||||
import threading
|
||||
import time
|
||||
from collections import Counter
|
||||
|
||||
active, peak, calls = Counter(), Counter(), []
|
||||
lock = threading.Lock()
|
||||
|
||||
def execute(lane):
|
||||
def run(it, output, attempt):
|
||||
with lock:
|
||||
calls.append((it.id, lane, attempt))
|
||||
active[lane] += 1
|
||||
peak[lane] = max(peak[lane], active[lane])
|
||||
time.sleep(0.01)
|
||||
with lock:
|
||||
active[lane] -= 1
|
||||
return {"status": "oom" if lane == "ax" else "ok"}
|
||||
|
||||
return run
|
||||
|
||||
assert not dispatch([item(str(i)) for i in range(9)], workload(), tmp_path, {"ax": execute("ax"), "docker": execute("docker")})
|
||||
assert peak["ax"] == 2 and peak["docker"] == 1
|
||||
assert all(attempt == 0 for _, _, attempt in calls)
|
||||
attempts = []
|
||||
|
||||
def failed(it, output, attempt):
|
||||
attempts.append(attempt)
|
||||
return {"status": "instrument_failure"}
|
||||
|
||||
assert dispatch([item("fail")], workload(), tmp_path, {"ax": failed}, lanes=("ax",))
|
||||
assert attempts == [0, 1]
|
||||
@@ -8,8 +8,6 @@ anywhere else, which is how an evaluation set quietly becomes a development one.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
import tomllib
|
||||
from pathlib import Path
|
||||
@@ -66,131 +64,17 @@ def test_a_multi_language_repository_is_reserved(path: Path) -> None:
|
||||
assert members and all(by_id[m]["language"] == "php" and "javascript" in by_id[m].get("also", []) for m in members)
|
||||
|
||||
|
||||
def _pool_manifests() -> list[Path]:
|
||||
directory = ROOT / "benchmarks/unseen"
|
||||
return sorted([*directory.glob("*.toml"), *directory.glob("private/*.toml")])
|
||||
|
||||
|
||||
def _pool_names() -> set[str]:
|
||||
names = set()
|
||||
for path in _pool_manifests():
|
||||
text = path.read_text().lower()
|
||||
names.update(m.rstrip("/").removesuffix(".git") for m in re.findall(r"github\.com/([\w.-]+/[\w.-]+)", text))
|
||||
return names
|
||||
|
||||
|
||||
def _population_names() -> set[str]:
|
||||
reserved = {case["repo"].rstrip("/").lower() for path in POPULATIONS for case in _population(path)["case"]}
|
||||
return {url.rsplit("/", 2)[-2] + "/" + url.rsplit("/", 1)[-1] for url in reserved}
|
||||
|
||||
|
||||
def test_no_reserved_repository_is_referenced_by_the_development_tree() -> None:
|
||||
reservations = [(ROOT / "benchmarks/independent", _population_names()), (ROOT / "benchmarks/unseen", _pool_names())]
|
||||
offenders = 0
|
||||
counts = {}
|
||||
for top in ("benchmarks", "src", "tests", "plane"):
|
||||
reserved = {case["repo"].rstrip("/").lower() for path in POPULATIONS for case in _population(path)["case"]}
|
||||
names = {url.rsplit("/", 2)[-2] + "/" + url.rsplit("/", 1)[-1] for url in reserved}
|
||||
offenders = []
|
||||
for top in ("benchmarks", "src", "tests"):
|
||||
for path in (ROOT / top).rglob("*"):
|
||||
if not path.is_file() or path == Path(__file__) or path.suffix in {".pyc", ".bin", ".gz"}:
|
||||
continue
|
||||
# Preserve the population guard's existing scope; plane is newly reserved for pools.
|
||||
active = [
|
||||
(directory, names)
|
||||
for directory, names in reservations
|
||||
if directory not in path.parents and names and not (top == "plane" and directory == ROOT / "benchmarks/independent")
|
||||
]
|
||||
if not active:
|
||||
if not path.is_file() or POPULATION.parent in path.parents or path == Path(__file__) or path.suffix in {".pyc", ".bin", ".gz"}:
|
||||
continue
|
||||
try:
|
||||
text = path.read_text(errors="ignore").lower()
|
||||
except OSError:
|
||||
continue
|
||||
count = sum(1 for _, names in active for name in names if f"github.com/{name}" in text)
|
||||
offenders += count
|
||||
counts[top] = counts.get(top, 0) + count
|
||||
assert not offenders, f"reserved repository referenced outside its reservation: count={offenders}, sources={counts}"
|
||||
|
||||
|
||||
def test_pool_and_population_names_are_disjoint() -> None:
|
||||
count = len(_population_names() & _pool_names())
|
||||
assert not count, f"overlapping reservations: count={count}"
|
||||
|
||||
|
||||
def manifest_digest(path: Path) -> str:
|
||||
"""Canonical parsed TOML, UTF-8 JSON with sorted keys and compact separators.
|
||||
|
||||
Private entries are covered by their opaque pointers' digests in the public manifest.
|
||||
The freeze writer must use the same encoding (task 3.2).
|
||||
"""
|
||||
raw = json.dumps(tomllib.loads(path.read_text()), sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode()
|
||||
return hashlib.sha256(raw).hexdigest()
|
||||
|
||||
|
||||
def check_pool_digest(path: Path) -> None:
|
||||
record = path.with_name("freeze-" + path.stem.removeprefix("pool-") + ".json")
|
||||
expected = json.loads(record.read_text())["freeze_digest"]
|
||||
assert manifest_digest(path) == expected, "frozen pool digest mismatch"
|
||||
|
||||
|
||||
def test_frozen_pool_manifest_digest() -> None:
|
||||
pools = sorted((ROOT / "benchmarks/unseen").glob("pool-p*.toml"))
|
||||
if not pools:
|
||||
pytest.skip("no pool frozen")
|
||||
for path in pools:
|
||||
check_pool_digest(path)
|
||||
|
||||
|
||||
def test_pool_guard_controls(tmp_path, monkeypatch) -> None:
|
||||
"""A synthetic reservation proves green, then red outside its own directory."""
|
||||
population = tmp_path / "benchmarks/independent/population-v1.toml"
|
||||
pool = tmp_path / "benchmarks/unseen/pool-p1.toml"
|
||||
population.parent.mkdir(parents=True)
|
||||
pool.parent.mkdir(parents=True)
|
||||
prefix = "https://" + "github.com/"
|
||||
population.write_text('[[case]]\nrepo = "' + prefix + 'synthetic/population"\n')
|
||||
pool.write_text('[[repository]]\nrepo = "' + prefix + 'synthetic/pool"\n')
|
||||
monkeypatch.setattr(__import__(__name__), "ROOT", tmp_path)
|
||||
monkeypatch.setattr(__import__(__name__), "POPULATIONS", [population])
|
||||
test_no_reserved_repository_is_referenced_by_the_development_tree()
|
||||
leak = tmp_path / "benchmarks/measurements/fixture.txt"
|
||||
leak.parent.mkdir(parents=True)
|
||||
leak.write_text(prefix + "synthetic/pool")
|
||||
with pytest.raises(AssertionError, match="count=1"):
|
||||
test_no_reserved_repository_is_referenced_by_the_development_tree()
|
||||
leak.unlink()
|
||||
plane = tmp_path / "plane/fixture.txt"
|
||||
plane.parent.mkdir()
|
||||
plane.write_text(prefix + "synthetic/pool")
|
||||
with pytest.raises(AssertionError, match="count=1"):
|
||||
test_no_reserved_repository_is_referenced_by_the_development_tree()
|
||||
plane.write_text(prefix + "synthetic/population")
|
||||
test_no_reserved_repository_is_referenced_by_the_development_tree()
|
||||
plane.unlink()
|
||||
private = pool.parent / "private/draft-p1.toml"
|
||||
private.parent.mkdir()
|
||||
private.write_text('[[repository]]\nrepo = "' + prefix + 'synthetic/population"\n')
|
||||
with pytest.raises(AssertionError, match="count=2"):
|
||||
test_no_reserved_repository_is_referenced_by_the_development_tree()
|
||||
|
||||
|
||||
def test_pool_digest_control(tmp_path) -> None:
|
||||
pool = tmp_path / "pool-p1.toml"
|
||||
pool.write_text('status = "frozen"\n[[repository]]\nid = "opaque"\n')
|
||||
record = tmp_path / "freeze-p1.json"
|
||||
record.write_text(json.dumps({"freeze_digest": manifest_digest(pool)}))
|
||||
check_pool_digest(pool)
|
||||
pool.write_text(pool.read_text() + "slice = 2\n")
|
||||
with pytest.raises(AssertionError, match="digest"):
|
||||
check_pool_digest(pool)
|
||||
|
||||
|
||||
def test_frozen_pool_discovery_cannot_be_disabled(tmp_path, monkeypatch) -> None:
|
||||
pool = tmp_path / "benchmarks/unseen/pool-p1.toml"
|
||||
pool.parent.mkdir(parents=True)
|
||||
pool.write_text('status = "frozen"\n')
|
||||
record = pool.with_name("freeze-p1.json")
|
||||
record.write_text(json.dumps({"freeze_digest": manifest_digest(pool)}))
|
||||
monkeypatch.setattr(__import__(__name__), "ROOT", tmp_path)
|
||||
test_frozen_pool_manifest_digest()
|
||||
pool.write_text('status = "draft"\n')
|
||||
with pytest.raises(AssertionError, match="digest"):
|
||||
test_frozen_pool_manifest_digest()
|
||||
offenders += [f"{path.relative_to(ROOT)}: {name}" for name in names if f"github.com/{name}" in text]
|
||||
assert not offenders, "a reserved repository is referenced outside the reservation: " + "; ".join(offenders[:10])
|
||||
|
||||
@@ -0,0 +1,289 @@
|
||||
"""Authored fix pairs only: no store, models, containers, or population files."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import json
|
||||
from dataclasses import replace
|
||||
|
||||
import pytest
|
||||
|
||||
from openultrasast.learn.folds import Fold
|
||||
from openultrasast.learn.mechanisms import Pair, mine_pairs
|
||||
|
||||
|
||||
def pair(before: str, after: str, *, language: str = "python", family: str = "output_encoding", repo: str = "owner/alpha") -> Pair:
|
||||
return Pair("pair-" + repo, repo, repo, language, family, before, after)
|
||||
|
||||
|
||||
def repeated(p: Pair, **kwargs):
|
||||
return mine_pairs([p, replace(p, id="pair-beta", repository="owner/beta", group="owner/beta")], **kwargs)
|
||||
|
||||
|
||||
def test_learn_mechanisms_sanitizer_return_and_def_use():
|
||||
p = pair("def f(x):\n return x", "def f(x):\n y = html.escape(x)\n return y")
|
||||
result = repeated(p)
|
||||
assert [(c["form"], c["pattern"]) for c in result.candidates] == [("sanitizer", {"callee": "html.escape", "argument": 0})]
|
||||
assert result.candidates[0]["repository_count"] == 2
|
||||
assert "owner/alpha" not in json.dumps(result.record)
|
||||
assert result.record["pairs_mined"] == 2
|
||||
|
||||
|
||||
def test_learn_mechanisms_javascript():
|
||||
p = pair("function f(x) { res.send(x); }", "function f(x) { const y = escapeHtml(x); res.send(y); }", language="javascript")
|
||||
assert repeated(p).candidates[0]["pattern"] == {"callee": "escapeHtml", "argument": 0}
|
||||
|
||||
|
||||
def test_learn_mechanisms_guard_and_known():
|
||||
p = pair(
|
||||
"def f(u):\n redirect(u)",
|
||||
"def f(u):\n if not is_safe_url(u):\n abort()\n redirect(u)",
|
||||
family="untrusted_destination",
|
||||
)
|
||||
candidate = repeated(p).candidates[0]
|
||||
assert candidate["form"] == "guard"
|
||||
assert candidate["pattern"] == {"callee": "is_safe_url", "argument": 0, "predicate": "truthy"}
|
||||
assert candidate["known"] is True
|
||||
assert not any(f["eligible"] for f in candidate["folds"])
|
||||
|
||||
|
||||
@pytest.mark.skipif(importlib.util.find_spec("tree_sitter_php") is None, reason="PHP tree-sitter wheel unavailable")
|
||||
def test_learn_mechanisms_php_safe_argument_and_echo():
|
||||
p = pair(
|
||||
"function f($d) { return unserialize($d); }",
|
||||
"function f($d) { return unserialize($d, ['allowed_classes' => false]); }",
|
||||
language="php",
|
||||
family="deserialization",
|
||||
)
|
||||
assert repeated(p).candidates[0]["pattern"] == {"callee": "unserialize", "argument": 1, "key": "allowed_classes", "value": False}
|
||||
assert repeated(p).candidates[0]["form"] == "safe_argument"
|
||||
p = pair("function f($x) { echo $x; }", "function f($x) { echo htmlspecialchars($x); }", language="php")
|
||||
assert repeated(p).candidates[0]["form"] == "sanitizer"
|
||||
|
||||
|
||||
def test_learn_mechanisms_safe_alternative():
|
||||
p = pair("def f(x):\n return yaml.load(x)", "def f(x):\n return yaml.safe_load(x)", family="deserialization")
|
||||
candidate = repeated(p).candidates[0]
|
||||
assert candidate["form"] == "safe_alternative"
|
||||
assert candidate["pattern"] == {"callee": "yaml.load", "replacement": "yaml.safe_load"}
|
||||
assert candidate["known"] is True
|
||||
|
||||
|
||||
def test_learn_mechanisms_keyword():
|
||||
p = pair("def f(x):\n subprocess.run(x, shell=True)", "def f(x):\n subprocess.run(x, shell=False)", family="injection")
|
||||
result = repeated(p)
|
||||
assert any(c["pattern"] == {"callee": "subprocess.run", "key": "shell", "value": False} for c in result.candidates)
|
||||
|
||||
|
||||
def test_learn_mechanisms_one_repository_and_local_helpers_rejected():
|
||||
p = pair("def f(x):\n return x", "def f(x):\n return cleanse(x)")
|
||||
assert mine_pairs([p]).candidates == []
|
||||
local = replace(p, fixed="def cleanse(x):\n return x\ndef f(x):\n return cleanse(x)")
|
||||
result = repeated(local)
|
||||
assert result.candidates == []
|
||||
assert result.record["skips"]["local_definition"] == 2
|
||||
assert repeated(replace(p, fixed="def f(x):\n return alpha_escape(x)")).candidates == []
|
||||
|
||||
|
||||
def test_learn_mechanisms_folds_exclude_proposers():
|
||||
p = pair("def f(x):\n return x", "def f(x):\n return cleanse(x)")
|
||||
folds = [Fold("outer", "outer-0", frozenset({p.group})), Fold("outer", "outer-1", frozenset({"elsewhere"}))]
|
||||
candidate = repeated(p, folds=folds).candidates[0]
|
||||
assert [(f["repository_count"], f["eligible"]) for f in candidate["folds"]] == [(1, False), (2, True)]
|
||||
assert candidate["folds"][0]["proposing_pairs"] == ["pair-beta"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"fixed",
|
||||
[
|
||||
"def f(x):\n cleanse(x)\n return x",
|
||||
"def f(x):\n y = cleanse(x)\n y = 'constant'\n return y",
|
||||
"def f(x):\n if is_safe_url(x):\n pass\n redirect(x)",
|
||||
],
|
||||
)
|
||||
def test_learn_mechanisms_unrelated_calls_are_not_candidates(fixed):
|
||||
before = "def f(x):\n redirect(x)" if "redirect" in fixed else "def f(x):\n return x"
|
||||
family = "untrusted_destination" if "redirect" in fixed else "output_encoding"
|
||||
assert repeated(pair(before, fixed, family=family)).candidates == []
|
||||
|
||||
|
||||
def test_learn_mechanisms_skips_and_determinism():
|
||||
p = pair("def f(x):\n return x", "def f(x):\n return cleanse(x)")
|
||||
assert repeated(p) == repeated(p)
|
||||
assert repeated(replace(p, fixed="def broken(")).record["skips"]["unparseable"] == 2
|
||||
assert repeated(replace(p, language="ruby")).record["skips"]["unsupported_language"] == 2
|
||||
assert (
|
||||
repeated(replace(p, fixed="def f(x):\n for y in x:\n cleanse(y)\n return x")).record["skips"]["unsupported_shape"] == 2
|
||||
)
|
||||
|
||||
|
||||
def test_learn_mechanisms_guard_must_test_sink_value():
|
||||
p = pair(
|
||||
"def f(x, u):\n redirect(u)",
|
||||
"def f(x, u):\n if not check(x):\n abort()\n redirect(u)",
|
||||
family="untrusted_destination",
|
||||
)
|
||||
assert repeated(p).candidates == []
|
||||
|
||||
|
||||
def test_learn_mechanisms_existing_path_call_is_not_new():
|
||||
p = pair("def f(x):\n return html.escape(x)", "def f(x):\n y = html.escape(x)\n return y")
|
||||
assert repeated(p).candidates == []
|
||||
|
||||
|
||||
def test_learn_mechanisms_missing_grammar_is_counted(monkeypatch):
|
||||
monkeypatch.setattr("openultrasast.learn.mechanisms.grammar_for", lambda language: None)
|
||||
result = repeated(pair("def f(x):\n return x", "def f(x):\n return cleanse(x)"))
|
||||
assert result.candidates == []
|
||||
assert result.record["skips"] == {"grammar_unavailable": 2}
|
||||
|
||||
|
||||
def test_learn_mechanisms_pair_identity_and_unread_inputs():
|
||||
from openultrasast.learn.mechanisms import pairs_from_rows
|
||||
|
||||
base = {
|
||||
"repo": "owner/alpha",
|
||||
"group": "owner/alpha",
|
||||
"source": "pairs",
|
||||
"candidate": "app.py::f",
|
||||
"language": "python",
|
||||
"family": "output_encoding",
|
||||
"profile": "static",
|
||||
"unit": "pin",
|
||||
}
|
||||
rows = [{**base, "label": label, "excerpt_sha": str(label)} for label in (0, 1)]
|
||||
pairs, skips, count = pairs_from_rows(rows, lambda sha: "def f(x):\n return x")
|
||||
assert len(pairs) == count == 1 and not skips
|
||||
assert pairs_from_rows(rows + [rows[0]], lambda sha: "code")[1] == {"ambiguous_pair": 1}
|
||||
assert pairs_from_rows(rows[:1], lambda sha: "code")[1] == {"incomplete_pair": 1}
|
||||
assert pairs_from_rows(rows, lambda sha: None)[1] == {"unreadable_excerpt": 1}
|
||||
forbidden = [{**row, "source": "population-v3"} for row in rows]
|
||||
assert pairs_from_rows(forbidden, lambda sha: pytest.fail("must not read excluded excerpts"))[1] == {"excluded_source": 2}
|
||||
|
||||
|
||||
def memory_store(tmp_path):
|
||||
from openultrasast.plane.memory import FileStore, row_id
|
||||
|
||||
store = FileStore(tmp_path / "memory")
|
||||
rows = []
|
||||
for repo in ("owner/alpha", "owner/beta", "owner/gamma"):
|
||||
for label in (0, 1):
|
||||
code = "def f(x):\n return x" if label else "def f(x):\n return cleanse(x)"
|
||||
sha = store.put_blob("excerpts", code.encode())
|
||||
rows.append(
|
||||
{
|
||||
"id": row_id("example", "test", "test", f"{repo}:{label}"),
|
||||
"kind": "example",
|
||||
"repo": repo,
|
||||
"pin": str(label) * 40,
|
||||
"run": "test",
|
||||
"task": "test",
|
||||
"population": "pairs",
|
||||
"split": "pairs",
|
||||
"image": "host",
|
||||
"candidate": "app.py::f",
|
||||
"family": "output_encoding",
|
||||
"language": "python",
|
||||
"profile": "static",
|
||||
"label": label,
|
||||
"source": "pairs",
|
||||
"group": repo,
|
||||
"unit": "pin",
|
||||
"x": {},
|
||||
"instruments": {},
|
||||
"excerpt_sha": sha,
|
||||
}
|
||||
)
|
||||
store.put_rows(rows)
|
||||
return store
|
||||
|
||||
|
||||
def test_learn_mechanisms_store_cli_and_privacy(tmp_path, monkeypatch):
|
||||
from openultrasast.cli import main
|
||||
from openultrasast.learn.mechanisms import RECORD, mine_store
|
||||
|
||||
store = memory_store(tmp_path)
|
||||
result = mine_store(store)
|
||||
assert result.record["complete_pairs"] == 3
|
||||
assert result.record["excerpt_bytes_read"] > 0
|
||||
assert len(result.candidates) == 1
|
||||
assert len(result.candidates[0]["folds"]) == 5
|
||||
monkeypatch.setenv("OUSAST_RESULTS", str(tmp_path / "results"))
|
||||
root = tmp_path / "repo"
|
||||
out = tmp_path / "results" / "plane" / "mining" / "candidates.jsonl"
|
||||
assert main(["learn", "mechanisms", "mine", "--memory", f"file://{tmp_path / 'memory'}", "--out", str(out), "--root", str(root)]) == 0
|
||||
record = (root / RECORD).read_text()
|
||||
assert all(repo not in record for repo in ("alpha", "beta", "gamma"))
|
||||
assert json.loads(out.read_text())["repository_count"] == 3
|
||||
assert main(["learn", "mechanisms", "mine", "--memory", f"file://{tmp_path / 'empty'}", "--out", str(out), "--root", str(root)]) == 2
|
||||
assert json.loads((root / RECORD).read_text())["status"] == "no_mineable_pairs"
|
||||
|
||||
|
||||
def test_learn_mechanisms_output_location_rejected(tmp_path):
|
||||
from openultrasast.learn.mechanisms import MiningResult, write_result
|
||||
|
||||
root = tmp_path / "repo"
|
||||
with pytest.raises(ValueError, match="outside"):
|
||||
write_result(MiningResult([], {}), root / "candidate.jsonl", root=root, results=tmp_path)
|
||||
assert not root.exists()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("signature,source", [("", "input()"), ("request", "request.get_json()")])
|
||||
def test_learn_mechanisms_source_calls(signature, source):
|
||||
p = pair(f"def f({signature}):\n return {source}", f"def f({signature}):\n return html.escape({source})")
|
||||
assert repeated(p).candidates[0]["pattern"] == {"callee": "html.escape", "argument": 0}
|
||||
|
||||
|
||||
def test_learn_mechanisms_inserted_log_is_not_sink_replacement():
|
||||
p = pair("def f(x):\n yaml.load(x)", "def f(x):\n logger.info(x)\n yaml.safe_load(x)", family="deserialization")
|
||||
result = repeated(p)
|
||||
assert result.candidates == []
|
||||
assert result.record["skips"]["ambiguous_sink"] == 2
|
||||
|
||||
|
||||
def test_learn_mechanisms_keyword_reordering_is_not_change():
|
||||
p = pair(
|
||||
"def f(x):\n subprocess.run(x, shell=False, timeout=10)",
|
||||
"def f(x):\n subprocess.run(x, timeout=10, shell=False)",
|
||||
family="injection",
|
||||
)
|
||||
assert repeated(p).candidates == []
|
||||
|
||||
|
||||
def test_learn_mechanisms_legacy_pair_ids_are_repository_scoped(tmp_path):
|
||||
from openultrasast.learn.mechanisms import mine_store
|
||||
|
||||
result = mine_store(memory_store(tmp_path))
|
||||
candidate = result.candidates[0]
|
||||
assert len(candidate["proposing_pairs"]) == 3
|
||||
assert all(len(ids) == 2 for ids in candidate["proposing_examples"].values())
|
||||
assert len({i for ids in candidate["proposing_examples"].values() for i in ids}) == 6
|
||||
|
||||
|
||||
def test_learn_mechanisms_runtime_callable_is_unsupported():
|
||||
p = pair("def f(x, clean):\n return x", "def f(x, clean):\n return clean(x)")
|
||||
result = repeated(p)
|
||||
assert result.candidates == []
|
||||
assert result.record["skips"]["unsupported_shape"] == 2
|
||||
|
||||
|
||||
def test_learn_mechanisms_language_support_is_not_pooled():
|
||||
p = pair("def f(x):\n return x", "def f(x):\n return cleanse(x)")
|
||||
js = pair("function f(x) { return x; }", "function f(x) { return cleanse(x); }", language="javascript", repo="owner/beta")
|
||||
assert mine_pairs([p, js]).candidates == []
|
||||
|
||||
|
||||
def test_learn_mechanisms_guard_inside_branch_and_numbered_excerpt():
|
||||
p = pair(
|
||||
" 10 def f(x):\n 11 redirect(x)",
|
||||
" 20 def f(x):\n 21 if is_safe_url(x):\n 22 redirect(x)",
|
||||
family="untrusted_destination",
|
||||
)
|
||||
candidate = repeated(p).candidates[0]
|
||||
assert candidate["form"] == "guard" and candidate["pattern"]["predicate"] == "truthy"
|
||||
|
||||
|
||||
@pytest.mark.skipif(importlib.util.find_spec("tree_sitter_php") is None, reason="PHP tree-sitter wheel unavailable")
|
||||
def test_learn_mechanisms_php_receiver_normalization():
|
||||
p = pair("function f($x, $helper) { echo $x; }", "function f($x, $helper) { echo $helper->escape($x); }", language="php")
|
||||
assert repeated(p).candidates[0]["pattern"] == {"callee": "escape", "argument": 0}
|
||||
@@ -1,527 +0,0 @@
|
||||
"""Draft and coordinator tests contain synthetic identities only and use injected clients."""
|
||||
|
||||
import copy
|
||||
import importlib
|
||||
import sys
|
||||
import tomllib
|
||||
from collections import Counter
|
||||
from contextlib import contextmanager
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
d = importlib.import_module("benchmarks.unseen.draft")
|
||||
s = importlib.import_module("benchmarks.unseen.source")
|
||||
e = importlib.import_module("benchmarks.unseen.eligibility")
|
||||
|
||||
|
||||
def entries():
|
||||
rows = []
|
||||
for i in range(300):
|
||||
changes = [{"kind": "vulnerability", "method": "introducing", "base": "a" * 40, "head": "b" * 40}]
|
||||
changes += [{"kind": "ordinary", "method": "first_parent", "base": f"{j:040x}", "head": f"{j + 20:040x}"} for j in range(12)]
|
||||
rows.append(
|
||||
{
|
||||
"repository": f"fixture/project{i:03d}",
|
||||
"url": f"https://github.com/fixture/project{i:03d}",
|
||||
"ecosystem": s.ECOSYSTEMS[i % 4],
|
||||
"post_cutoff": i % 7 == 0,
|
||||
"license_spdx": "MIT" if i % 2 else "GPL-3.0",
|
||||
"license_class": "permissive" if i % 2 else "private",
|
||||
"license_ref": "b" * 40,
|
||||
"changes": changes,
|
||||
}
|
||||
)
|
||||
return rows
|
||||
|
||||
|
||||
def test_draft_split_digest_and_balanced_slices(tmp_path):
|
||||
rows = entries()
|
||||
assigned = d.assign_slices(rows, seed=19)
|
||||
assert Counter(r["slice"] for r in assigned) == {1: 100, 2: 100, 3: 100}
|
||||
assert assigned == d.assign_slices(list(reversed(rows)), seed=19)
|
||||
strata = Counter((r["ecosystem"], r["post_cutoff"]) for r in assigned)
|
||||
for slice_id in (1, 2, 3):
|
||||
counts = Counter((r["ecosystem"], r["post_cutoff"]) for r in assigned if r["slice"] == slice_id)
|
||||
assert all(abs(counts[key] - total / 3) < 1 for key, total in strata.items())
|
||||
public, private = d.write_draft(tmp_path, assigned, seed=19)
|
||||
public_text = public.read_text()
|
||||
shared = tomllib.loads(public_text)
|
||||
hidden = tomllib.loads(private.read_text())
|
||||
assert len(shared["repository"]) == 300 and len(hidden["repository"]) == 150
|
||||
assert sum(len(r["changes"]) for r in shared["repository"] if "changes" in r) == 1950
|
||||
for row in hidden["repository"]:
|
||||
assert row["url"] not in public_text and row["repository"] not in public_text
|
||||
pointer = next(p for p in shared["repository"] if p["id"] == row["id"])
|
||||
assert set(pointer) == {"id", "sha256", "slice", "private"}
|
||||
assert pointer["sha256"] == d.private_digest(row)
|
||||
changed = copy.deepcopy(row)
|
||||
changed["changes"][0]["head"] = "c" * 40
|
||||
assert d.private_digest(changed) != pointer["sha256"]
|
||||
assert private.is_relative_to(tmp_path / "benchmarks/unseen/private")
|
||||
with pytest.raises(FileExistsError):
|
||||
d.write_draft(tmp_path, assigned, seed=19)
|
||||
|
||||
|
||||
def test_draft_requires_full_size_and_valid_changes():
|
||||
with pytest.raises(ValueError, match="pool_size"):
|
||||
d.assign_slices(entries()[:299], seed=1)
|
||||
rows = entries()
|
||||
rows[0]["changes"].pop()
|
||||
with pytest.raises(ValueError, match="change_count"):
|
||||
d.assign_slices(rows, seed=1)
|
||||
|
||||
|
||||
def test_cache_is_outside_repository_deleted_and_partial(tmp_path):
|
||||
import subprocess
|
||||
|
||||
root, cache = tmp_path / "repo", tmp_path / "cache"
|
||||
root.mkdir()
|
||||
with pytest.raises(ValueError, match="cache_inside_repository"):
|
||||
d.Clones(root, root / "cache")
|
||||
calls = []
|
||||
|
||||
def runner(argv, **kwargs):
|
||||
calls.append(argv)
|
||||
if "clone" in argv:
|
||||
target = Path(argv[-1])
|
||||
(target / ".git").mkdir()
|
||||
(target / ".git/pack").write_bytes(b"123")
|
||||
return subprocess.CompletedProcess(argv, 0, b"", b"")
|
||||
|
||||
clones = d.Clones(root, cache, runner=runner)
|
||||
with clones.open("https://github.com/fixture/project") as git:
|
||||
assert git.path.exists()
|
||||
clone_path = git.path
|
||||
assert not clone_path.exists()
|
||||
assert "--filter=blob:none" in calls[0] and "--no-checkout" in calls[0]
|
||||
assert clones.counts["clones"] == 1 and clones.counts["clone_bytes"] == 3
|
||||
|
||||
|
||||
def test_clone_failure_sanitized_and_cleaned(tmp_path, capsys):
|
||||
import subprocess
|
||||
|
||||
def runner(argv, **kwargs):
|
||||
return subprocess.CompletedProcess(argv, 128, b"", b"private identity")
|
||||
|
||||
clones = d.Clones(tmp_path / "repo", tmp_path / "cache", runner=runner)
|
||||
with pytest.raises(d.InstrumentFailure), clones.open("https://github.com/fixture/project"):
|
||||
pass
|
||||
assert list((tmp_path / "cache").iterdir()) == []
|
||||
assert not capsys.readouterr().out
|
||||
|
||||
|
||||
class FakeAPI:
|
||||
calls = 0
|
||||
|
||||
def __init__(self, count=300):
|
||||
self.count = count
|
||||
self.licenses = []
|
||||
|
||||
def get_repo(self, name):
|
||||
self.calls += 1
|
||||
if name == "fixture/old":
|
||||
name = "fixture/project000"
|
||||
repo = {"full_name": name, "private": False, "size": 2, "clone_url": "https://github.com/" + name}
|
||||
if name == "fixture/fork":
|
||||
repo["source"] = {"full_name": "fixture/project001"}
|
||||
return repo
|
||||
|
||||
def get_commit(self, name, sha):
|
||||
self.calls += 1
|
||||
return {"sha": sha, "parents": [{"sha": "b" * 40}]}
|
||||
|
||||
def get_license(self, name, ref):
|
||||
self.licenses.append((name, ref))
|
||||
self.calls += 1
|
||||
# P is MIT but actual introducing head is GPL: public output must use V.
|
||||
return {"text": "", "spdx_id": "MIT" if ref == "b" * 40 else "GPL-3.0", "path": "LICENSE"}
|
||||
|
||||
def advisory_page(self, page):
|
||||
self.calls += 1
|
||||
rows = list(self.advisories())[(page - 1) * 100 : page * 100]
|
||||
return {"rows": rows, "done": len(rows) < 100}
|
||||
|
||||
def advisories(self):
|
||||
for i in range(self.count):
|
||||
yield {
|
||||
"ghsa_id": f"GHSA-test-test-{i:04d}",
|
||||
"published_at": "2026-06-02T00:00:00Z",
|
||||
"cwes": [{"cwe_id": "CWE-89"}],
|
||||
"vulnerabilities": [{"package": {"ecosystem": "pip", "name": "fixture"}}],
|
||||
"references": [f"https://github.com/fixture/project{i:03d}/commit/" + "a" * 40],
|
||||
}
|
||||
|
||||
|
||||
class FakeOSV:
|
||||
calls = 0
|
||||
|
||||
def get(self, identifier):
|
||||
self.calls += 1
|
||||
return {}
|
||||
|
||||
|
||||
class FakeClones:
|
||||
def __init__(self):
|
||||
self.counts = Counter()
|
||||
self.commands = []
|
||||
|
||||
@contextmanager
|
||||
def open(self, url):
|
||||
self.counts["clones"] += 1
|
||||
yield self
|
||||
|
||||
def run(self, *args):
|
||||
self.commands.append(args)
|
||||
return ""
|
||||
|
||||
|
||||
def test_coordinator_end_to_end_fakes_pinned_license_and_no_names(tmp_path, monkeypatch, capsys):
|
||||
rows = entries()
|
||||
changes = copy.deepcopy(rows[0]["changes"])
|
||||
changes[0]["head"] = "c" * 40
|
||||
monkeypatch.setattr(d, "repository_changes", lambda *a, **kw: (changes, {"security": 4, "eligible": 40}, {"bytes": 12, "files": 1}))
|
||||
used, api, clones = e.UsedSet(), FakeAPI(), FakeClones()
|
||||
report = d.run(tmp_path, github=api, osv=FakeOSV(), clones=clones, used=used, seed=7, known_empty=set(used.sources))
|
||||
assert report["status"] == "draft" and report["changes"] == 3900
|
||||
assert report["license_split"] == {"private": 300}
|
||||
assert report["ordinary_exclusions"]["security"] == 1200
|
||||
assert len([ref for _, ref in api.licenses if ref == "c" * 40]) == 300
|
||||
public = (tmp_path.parent / (tmp_path.name + "-cache") / "journal/draft-p1.toml").read_text()
|
||||
assert "fixture/" not in public
|
||||
import json
|
||||
|
||||
assert "fixture/" not in json.dumps(report)
|
||||
assert capsys.readouterr().out == ""
|
||||
|
||||
|
||||
def test_used_rename_and_fork_resolution_and_shortfall(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(d, "repository_changes", lambda *a, **kw: (entries()[0]["changes"], {}, {"bytes": 1, "files": 1}))
|
||||
used, api, clones = e.UsedSet(), FakeAPI(2), FakeClones()
|
||||
used.add("pairs", {"fixture/old"}, 10)
|
||||
# project000 resolves from used alias; project001 is reached via used fork's source.
|
||||
used.add("pairs", {"fixture/fork"}, 10)
|
||||
report = d.run(tmp_path, github=api, osv=FakeOSV(), clones=clones, used=used, seed=7, known_empty=set(used.sources) - {"pairs"})
|
||||
assert report["status"] == "insufficient_repositories"
|
||||
assert report["rejections"] == {"used_pairs": 2}
|
||||
assert report["repositories"] == 0 and not clones.counts
|
||||
assert not (tmp_path / "benchmarks/unseen/draft-p1.toml").exists()
|
||||
|
||||
|
||||
def test_zero_source_is_instrument_failure_not_rejection(tmp_path, monkeypatch):
|
||||
def unread(*args, **kwargs):
|
||||
raise d.InstrumentFailure("zero_source_bytes")
|
||||
|
||||
monkeypatch.setattr(d, "repository_changes", unread)
|
||||
used = e.UsedSet()
|
||||
report = d.run(tmp_path, github=FakeAPI(1), osv=FakeOSV(), clones=FakeClones(), used=used, seed=1, known_empty=set(used.sources))
|
||||
assert report["status"] == "instrument_failure" and not report["rejections"]
|
||||
|
||||
|
||||
def test_cli_uses_environment_and_injected_stores(tmp_path, monkeypatch, capsys):
|
||||
from openultrasast import config
|
||||
from openultrasast.plane import memory
|
||||
|
||||
seen = []
|
||||
monkeypatch.setenv("GH_TOKEN", "fake-secret")
|
||||
monkeypatch.setattr(config, "load_dotenv", lambda path: None)
|
||||
monkeypatch.setattr(memory, "open_store", lambda value, **kw: seen.append((value, kw)) or object())
|
||||
monkeypatch.setattr(d.eligibility, "build_used", lambda *a, **kw: e.UsedSet())
|
||||
monkeypatch.setattr(d.source, "GitHub", lambda token: seen.append(token) or FakeAPI(0))
|
||||
monkeypatch.setattr(d.source, "OSV", FakeOSV)
|
||||
args = ["--root", str(tmp_path), "--cache", str(tmp_path.parent / "clone-cache")]
|
||||
for label in e.SOURCES:
|
||||
args += ["--known-empty", label]
|
||||
assert d.main(args) == 1
|
||||
output = capsys.readouterr().out
|
||||
assert "instrument_failure" in output and "fake-secret" not in output and "fixture/" not in output
|
||||
assert seen[-1] == "fake-secret"
|
||||
assert all(kwargs == {"read_only": True} for _, kwargs in seen[:-1])
|
||||
|
||||
|
||||
def test_repository_changes_excludes_all_advisory_sites(monkeypatch):
|
||||
calls = []
|
||||
|
||||
class FakeGit:
|
||||
def run(self, *args):
|
||||
calls.append(args)
|
||||
if args[0] == "ls-tree":
|
||||
return "app.py\n"
|
||||
return args[-1] if args[0] == "rev-parse" else ""
|
||||
|
||||
def blob(self, revision, path):
|
||||
return "def handler():\n return 1\n"
|
||||
|
||||
def parents(self, fix):
|
||||
return ["b" * 40]
|
||||
|
||||
candidate = s.Candidate(
|
||||
"fixture/project",
|
||||
"https://github.com/fixture/project",
|
||||
"a" * 40,
|
||||
"b" * 40,
|
||||
"GHSA-test-test-test",
|
||||
"pip",
|
||||
"injection",
|
||||
True,
|
||||
None,
|
||||
"MIT",
|
||||
"permissive",
|
||||
"b" * 40,
|
||||
"LICENSE",
|
||||
)
|
||||
monkeypatch.setattr(d.extract, "introducing", lambda *a: {"head": "c" * 40})
|
||||
monkeypatch.setattr(d.extract, "fix_sites", lambda git, fix, parent, **kw: [{"path": "app.py", "function": fix}])
|
||||
monkeypatch.setattr(d.extract, "ordinary_history", lambda *a: [])
|
||||
|
||||
def draw(rows, fixes, known, **kwargs):
|
||||
assert fixes == {"a" * 40, "d" * 40}
|
||||
assert known == {("app.py", "a" * 40), ("app.py", "d" * 40)}
|
||||
return [], {}
|
||||
|
||||
monkeypatch.setattr(d.extract, "draw_ordinary", draw)
|
||||
_, _, proof = d.repository_changes(FakeGit(), candidate, {"d" * 40}, seed=1)
|
||||
assert proof["bytes"] > 0
|
||||
assert len([c for c in calls if c[0] == "fetch"]) == 2
|
||||
|
||||
|
||||
def test_candidate_404_is_counted_but_service_failure_aborts(tmp_path):
|
||||
class Missing(FakeAPI):
|
||||
def get_commit(self, name, sha):
|
||||
raise e.RepositoryNotFound("not_found")
|
||||
|
||||
class Broken(FakeAPI):
|
||||
def get_commit(self, name, sha):
|
||||
raise ValueError("private URL must never escape")
|
||||
|
||||
for api, status in [(Missing(1), "insufficient_repositories"), (Broken(1), "instrument_failure")]:
|
||||
used = e.UsedSet()
|
||||
report = d.run(
|
||||
tmp_path,
|
||||
github=api,
|
||||
osv=FakeOSV(),
|
||||
clones=FakeClones(),
|
||||
used=used,
|
||||
seed=1,
|
||||
known_empty=set(used.sources),
|
||||
journal_path=tmp_path.parent / (tmp_path.name + type(api).__name__) / "journal",
|
||||
)
|
||||
assert report["status"] == status
|
||||
if status == "insufficient_repositories":
|
||||
assert report["rejections"] == {"repository_or_commit_not_found": 1}
|
||||
assert "private URL" not in str(report)
|
||||
|
||||
|
||||
def test_shortfall_keeps_ordinary_exclusion_counts(tmp_path, monkeypatch):
|
||||
def too_few(*args, **kwargs):
|
||||
exc = s.Rejected("ordinary_floor")
|
||||
exc.counts = {"eligible": 39, "size_cap": 10}
|
||||
raise exc
|
||||
|
||||
monkeypatch.setattr(d, "repository_changes", too_few)
|
||||
used = e.UsedSet()
|
||||
report = d.run(tmp_path, github=FakeAPI(1), osv=FakeOSV(), clones=FakeClones(), used=used, seed=1, known_empty=set(used.sources))
|
||||
assert report["ordinary_exclusions"] == {"eligible": 39, "size_cap": 10}
|
||||
|
||||
|
||||
def test_clone_disk_cap_cleans_up(tmp_path):
|
||||
import subprocess
|
||||
|
||||
def runner(argv, **kwargs):
|
||||
(Path(argv[-1]) / "big").write_bytes(b"12345")
|
||||
return subprocess.CompletedProcess(argv, 0, b"", b"")
|
||||
|
||||
clones = d.Clones(tmp_path / "repo", tmp_path / "cache", runner=runner, limit_bytes=4)
|
||||
with pytest.raises(d.InstrumentFailure, match="clone_size_cap"), clones.open("https://github.com/fixture/project"):
|
||||
pass
|
||||
assert list((tmp_path / "cache").iterdir()) == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize("stage,n", [("advisories", 2), ("resolution", 103), ("eligibility", 17), ("extraction", 23), ("completion", 1)])
|
||||
def test_checkpoint_interrupt_resume_matches_uninterrupted(tmp_path, monkeypatch, stage, n):
|
||||
import json
|
||||
|
||||
journal_module = importlib.import_module("benchmarks.unseen.journal")
|
||||
# Large fake populations exercise replay; durability is checked separately below.
|
||||
monkeypatch.setattr(journal_module.os, "fsync", lambda fd: None)
|
||||
root = tmp_path / "repo"
|
||||
full, interrupted = tmp_path / "full/journal", tmp_path / "interrupted/journal"
|
||||
changes = entries()[0]["changes"]
|
||||
monkeypatch.setattr(d, "repository_changes", lambda *a, **kw: (changes, {"eligible": 40}, {"bytes": 12, "files": 1}))
|
||||
used = e.UsedSet()
|
||||
|
||||
def run(path, api=None, clones=None):
|
||||
return d.run(
|
||||
root,
|
||||
github=api or FakeAPI(),
|
||||
osv=FakeOSV(),
|
||||
clones=clones or FakeClones(),
|
||||
used=used,
|
||||
seed=7,
|
||||
known_empty=set(used.sources),
|
||||
journal_path=path,
|
||||
)
|
||||
|
||||
assert run(full)["status"] == "draft"
|
||||
original = d.Journal.put
|
||||
|
||||
def interrupt(self, current, key, value):
|
||||
original(self, current, key, value)
|
||||
if current == stage and len(self.rows[current]) == n:
|
||||
raise KeyboardInterrupt
|
||||
|
||||
monkeypatch.setattr(d.Journal, "put", interrupt)
|
||||
with pytest.raises(KeyboardInterrupt):
|
||||
run(interrupted)
|
||||
assert not (interrupted / "draft-p1.toml").exists()
|
||||
before = {p.name: p.read_bytes() for p in interrupted.glob("*.jsonl")}
|
||||
progress = json.loads((interrupted / "progress.json").read_text())
|
||||
assert progress["counts"][stage] == n
|
||||
assert progress["last_unit"]["stage"] == stage
|
||||
assert progress["elapsed_seconds"] >= 0
|
||||
assert "rate_limit" in progress
|
||||
monkeypatch.setattr(d.Journal, "put", original)
|
||||
clones = FakeClones()
|
||||
assert run(interrupted, clones=clones)["status"] == "draft"
|
||||
assert clones.counts["clones"] == 300 - progress["counts"]["extraction"]
|
||||
for name in ("draft-p1.toml", "private-draft-p1.toml"):
|
||||
assert (full / name).read_bytes() == (interrupted / name).read_bytes()
|
||||
assert all((interrupted / name).read_bytes().startswith(data) for name, data in before.items())
|
||||
assert not root.exists() # no names, manifests, or other records in the checkout
|
||||
api, clones = FakeAPI(), FakeClones()
|
||||
assert run(interrupted, api, clones)["status"] == "draft"
|
||||
assert api.calls == 0 and clones.counts == {}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("parameter", ["seed", "used_set_digest", "clone_limit_bytes", "design_extract"])
|
||||
def test_resume_refuses_changed_inputs_before_network(tmp_path, monkeypatch, parameter):
|
||||
path, root = tmp_path / "cache/journal", tmp_path / "repo"
|
||||
used = e.UsedSet()
|
||||
kwargs = dict(
|
||||
root=root,
|
||||
github=FakeAPI(),
|
||||
osv=FakeOSV(),
|
||||
clones=FakeClones(),
|
||||
used=used,
|
||||
seed=7,
|
||||
known_empty=set(used.sources),
|
||||
journal_path=path,
|
||||
max_hours=0,
|
||||
)
|
||||
assert d.run(**kwargs)["status"] == "stopped"
|
||||
if parameter == "seed":
|
||||
kwargs["seed"] = 8
|
||||
elif parameter == "used_set_digest":
|
||||
used.add("pairs", {"fixture/new"}, 12)
|
||||
elif parameter == "clone_limit_bytes":
|
||||
kwargs["clones"].limit_bytes = 12
|
||||
else:
|
||||
original = d.inputs
|
||||
monkeypatch.setattr(d, "inputs", lambda *a: {**original(*a), "design_extract": "changed"})
|
||||
report = d.run(**kwargs)
|
||||
assert report["status"] == "resume_refused"
|
||||
assert report["changed_parameters"] == [parameter]
|
||||
assert kwargs["github"].calls == 0
|
||||
|
||||
|
||||
def test_stop_between_units_and_progress_rate_state(tmp_path, monkeypatch):
|
||||
import json
|
||||
from types import SimpleNamespace
|
||||
|
||||
path, root = tmp_path / "cache/journal", tmp_path / "repo"
|
||||
api = FakeAPI(1)
|
||||
api.transport = SimpleNamespace(rate_limit={"remaining": "27", "reset": "1234"}, attempts=0)
|
||||
used = e.UsedSet()
|
||||
original = d.Journal.put
|
||||
|
||||
def stop(self, stage, key, value):
|
||||
original(self, stage, key, value)
|
||||
if stage == "eligibility":
|
||||
(self.path / "STOP").touch()
|
||||
|
||||
monkeypatch.setattr(d.Journal, "put", stop)
|
||||
clones = FakeClones()
|
||||
report = d.run(root, github=api, osv=FakeOSV(), clones=clones, used=used, seed=1, known_empty=set(used.sources), journal_path=path)
|
||||
assert report["status"] == "stopped" and report["reason"] == "STOP"
|
||||
assert clones.counts == {}
|
||||
progress = json.loads((path / "progress.json").read_text())
|
||||
assert progress["status"] == "stopped" and progress["counts"]["eligibility"] == 1
|
||||
assert progress["rate_limit"] == api.transport.rate_limit
|
||||
assert not (path / "draft-p1.toml").exists()
|
||||
|
||||
|
||||
def test_max_hours_stops_after_completed_page(tmp_path):
|
||||
import json
|
||||
|
||||
now = [0.0]
|
||||
|
||||
class TimedAPI(FakeAPI):
|
||||
def advisory_page(self, page):
|
||||
result = super().advisory_page(page)
|
||||
now[0] += 3601
|
||||
return result
|
||||
|
||||
path = tmp_path / "cache/journal"
|
||||
api, used = TimedAPI(), e.UsedSet()
|
||||
report = d.run(
|
||||
tmp_path / "repo",
|
||||
github=api,
|
||||
osv=FakeOSV(),
|
||||
clones=FakeClones(),
|
||||
used=used,
|
||||
seed=1,
|
||||
known_empty=set(used.sources),
|
||||
journal_path=path,
|
||||
max_hours=1,
|
||||
clock=lambda: now[0],
|
||||
)
|
||||
assert report["status"] == "stopped" and report["reason"] == "max_hours"
|
||||
assert api.calls == 1
|
||||
progress = json.loads((path / "progress.json").read_text())
|
||||
assert progress["counts"]["advisories"] == 1
|
||||
assert progress["elapsed_seconds"] == 3601
|
||||
|
||||
|
||||
def test_journal_cannot_be_inside_repository(tmp_path):
|
||||
used = e.UsedSet()
|
||||
api = FakeAPI()
|
||||
report = d.run(
|
||||
tmp_path,
|
||||
github=api,
|
||||
osv=FakeOSV(),
|
||||
clones=FakeClones(),
|
||||
used=used,
|
||||
seed=1,
|
||||
known_empty=set(used.sources),
|
||||
journal_path=tmp_path / "journal",
|
||||
)
|
||||
assert report["status"] == "instrument_failure" and api.calls == 0
|
||||
assert not (tmp_path / "journal").exists()
|
||||
|
||||
|
||||
def test_journal_fsync_lock_and_corruption(tmp_path, monkeypatch):
|
||||
import json
|
||||
|
||||
j = importlib.import_module("benchmarks.unseen.journal")
|
||||
original = j.os.fsync
|
||||
synced = []
|
||||
|
||||
def fsync(fd):
|
||||
synced.append(fd)
|
||||
original(fd)
|
||||
|
||||
monkeypatch.setattr(j.os, "fsync", fsync)
|
||||
path = tmp_path / "journal"
|
||||
with j.Journal(path).open({"seed": 1}) as journal:
|
||||
before = len(synced)
|
||||
journal.put("advisories", "1", {"rows": [], "done": True})
|
||||
# Appended JSONL, directory, atomic progress file, directory.
|
||||
assert len(synced) - before == 4
|
||||
assert json.loads((path / "progress.json").read_text())["counts"]["advisories"] == 1
|
||||
with pytest.raises(ValueError, match="journal_busy"), j.Journal(path).open({"seed": 1}):
|
||||
pass
|
||||
with (path / "advisories.jsonl").open("ab") as handle:
|
||||
handle.write(b'{"id":')
|
||||
with pytest.raises(ValueError, match="journal_incomplete_record"), j.Journal(path).open({"seed": 1}):
|
||||
pass
|
||||
@@ -1,293 +0,0 @@
|
||||
"""Offline eligibility fixtures; no real repository identities are needed."""
|
||||
|
||||
import io
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from openultrasast.plane.memory import FileStore
|
||||
|
||||
with pytest.MonkeyPatch.context() as patch:
|
||||
patch.syspath_prepend(str(Path(__file__).resolve().parents[1] / "benchmarks/unseen"))
|
||||
import eligibility as e
|
||||
|
||||
|
||||
def plant(root, path, text):
|
||||
target = root / path
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_text(text)
|
||||
return target
|
||||
|
||||
|
||||
def test_sources_and_reserved_input_never_opened(tmp_path, monkeypatch, capsys):
|
||||
sources = {
|
||||
"benchmarks/pairs/a/catalog.toml": ("pairs", "pair"),
|
||||
"benchmarks/pairs/a/recipes.toml": ("pairs", "recipe"),
|
||||
"benchmarks/pairs/a/dataset.json": ("pairs", "dataset"),
|
||||
"benchmarks/pairs/advisory-fixes-b/catalog.toml": ("harvest", "harvest"),
|
||||
"benchmarks/independent/population-v1.toml": ("populations", "first"),
|
||||
"benchmarks/independent/population-v2.toml": ("populations", "second"),
|
||||
"plane/experiments/a.units.jsonl": ("units", "unit"),
|
||||
"benchmarks/measurements/a/record.json": ("measurements", "measurement"),
|
||||
"benchmarks/experiments/a.json": ("measurements", "experiment"),
|
||||
"src/example.txt": ("sweep", "sweep"),
|
||||
}
|
||||
for path, (_, name) in sources.items():
|
||||
text = f'repo = "https://github.com/fixture/{name}"' if path.endswith(".toml") else json.dumps({"repo": f"fixture/{name}"})
|
||||
if name == "sweep":
|
||||
text = "https://github.com/fixture/sweep"
|
||||
plant(tmp_path, path, text)
|
||||
pointer = plant(tmp_path, "local/pointers.toml", '[[pair]]\nrepo = "fixture/pointer"\nfix_repo = "fixture/fix"\nvendored = false')
|
||||
plant(tmp_path, "benchmarks/independent/population-v3.toml", "do not read")
|
||||
plant(tmp_path, "benchmarks/independent/results-v3.json", "do not read")
|
||||
original = io.open
|
||||
|
||||
def guarded(file, *args, **kwargs):
|
||||
if "v3" in str(file):
|
||||
pytest.fail("reserved input opened")
|
||||
return original(file, *args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(io, "open", guarded)
|
||||
store = FileStore(tmp_path / "memory")
|
||||
plant(tmp_path, "memory/repos/opaque/pin.jsonl", json.dumps({"id": "1", "kind": "example", "repo": "fixture/memory"}) + "\n")
|
||||
used = e.build_used(tmp_path, stores={"local": store}, pointer_manifests=[pointer])
|
||||
for label, name in sources.values():
|
||||
assert f"fixture/{name}" in used.sources[label]
|
||||
assert {"fixture/pointer", "fixture/fix"} <= used.sources["pointers"]
|
||||
assert "fixture/memory" in used.sources["memory_local"]
|
||||
report = used.report()
|
||||
assert report["sources"]["memory_local"]["bytes"] > 0
|
||||
assert all(item["files"] > 0 and item["bytes"] > 0 for item in report["sources"].values())
|
||||
assert "fixture/" not in json.dumps(report)
|
||||
assert capsys.readouterr().out == ""
|
||||
assert e.normalize("HTTPS://GitHub.com/Fixture/Pair.GIT/") == "fixture/pair"
|
||||
assert len({e.normalize(s) for s in ["fixture/pair", "Fixture/Pair.git/", "github.com/fixture/pair"]}) == 1
|
||||
|
||||
|
||||
def test_empty_sources_fail_unless_declared(tmp_path):
|
||||
used = e.build_used(tmp_path)
|
||||
with pytest.raises(ValueError, match="empty_source"):
|
||||
used.report()
|
||||
assert used.report(known_empty=set(used.sources))["count"] == 0
|
||||
|
||||
|
||||
class FakeAPI:
|
||||
def __init__(self, records):
|
||||
self.records = records
|
||||
|
||||
def get_repo(self, name):
|
||||
return self.records[name]
|
||||
|
||||
|
||||
def test_resolution_rejects_used_renames_and_forks():
|
||||
used = e.UsedSet()
|
||||
used.add("pairs", {"fixture/used"}, 10)
|
||||
api = FakeAPI(
|
||||
{
|
||||
"fixture/used": {"full_name": "fixture/used"},
|
||||
"fixture/old": {"full_name": "fixture/new", "former_names": ["fixture/used"]},
|
||||
"fixture/fork": {"full_name": "fixture/fork", "parent": {"full_name": "fixture/used"}, "source": {"full_name": "fixture/used"}},
|
||||
"fixture/a": {"full_name": "fixture/a", "source": {"full_name": "fixture/root"}},
|
||||
"fixture/b": {"full_name": "fixture/b", "source": {"full_name": "fixture/root"}},
|
||||
}
|
||||
)
|
||||
selected = set()
|
||||
for name in ("used", "old", "fork"):
|
||||
resolved = e.resolve("fixture/" + name, api)
|
||||
assert e.rejection(resolved, used, selected) == "pairs"
|
||||
assert e.rejection(e.resolve("fixture/a", api), used, selected) is None
|
||||
assert e.rejection(e.resolve("fixture/b", api), used, selected) == "fork_network"
|
||||
|
||||
|
||||
def test_registered_units_and_read_only_store(tmp_path):
|
||||
units = plant(tmp_path, "local/units.jsonl", json.dumps({"repo": "fixture/registered"}))
|
||||
store = FileStore(tmp_path / "memory")
|
||||
plant(tmp_path, "memory/repos/opaque/pin.jsonl", json.dumps({"kind": "experiment", "units_file": str(units)}))
|
||||
used = e.build_used(tmp_path, stores={"s3": store})
|
||||
assert "fixture/registered" in used.sources["units"]
|
||||
|
||||
|
||||
def test_cli_injected_store_and_credentials(tmp_path, monkeypatch, capsys):
|
||||
from openultrasast import config
|
||||
from openultrasast.plane import memory
|
||||
|
||||
calls = []
|
||||
monkeypatch.setattr(config, "load_dotenv", lambda path: calls.append("dotenv"))
|
||||
monkeypatch.setenv("GH_TOKEN", "test-token")
|
||||
store = FileStore(tmp_path / "store")
|
||||
monkeypatch.setattr(memory, "open_store", lambda spec, **kwargs: calls.append(spec) or store)
|
||||
monkeypatch.setattr(e, "GitHub", lambda token: calls.append(token) or FakeAPI({}))
|
||||
args = ["--root", str(tmp_path), "--s3-store", "s3://", "--local-store", "file:///unused"]
|
||||
for label in (*e.SOURCES, "memory_local", "memory_s3"):
|
||||
args += ["--known-empty", label]
|
||||
assert e.main(args) == 0
|
||||
assert calls == ["dotenv", "file:///unused", "s3://", "test-token"]
|
||||
output = capsys.readouterr().out
|
||||
assert json.loads(output)["count"] == 0
|
||||
assert "test-token" not in output
|
||||
|
||||
|
||||
def test_s3_open_is_read_only(tmp_path, monkeypatch):
|
||||
from openultrasast.plane import memory
|
||||
|
||||
class Objects:
|
||||
def keys(self, prefix):
|
||||
return ["repos/opaque/pin.jsonl"]
|
||||
|
||||
def get(self, key):
|
||||
return json.dumps({"kind": "example", "repo": "fixture/stored"}).encode(), "1"
|
||||
|
||||
def __getattr__(self, name):
|
||||
raise AssertionError("unexpected_store_operation")
|
||||
|
||||
monkeypatch.setattr(memory, "S3Client", lambda **kwargs: Objects())
|
||||
monkeypatch.setattr(memory, "s3_settings", lambda env: {})
|
||||
store = memory.open_store("s3://fixture-bucket", read_only=True)
|
||||
used = e.build_used(tmp_path, stores={"s3": store})
|
||||
assert used.sources["memory_s3"] == {"fixture/stored"}
|
||||
with pytest.raises(memory.MemoryStoreError, match="read.only"):
|
||||
store._put("key", b"value")
|
||||
with pytest.raises(memory.MemoryStoreError, match="read.only"):
|
||||
store._delete("key")
|
||||
|
||||
|
||||
def test_non_repository_provenance_and_missing_used_identity(tmp_path):
|
||||
plant(tmp_path, "benchmarks/pairs/recipes.toml", 'repo = "provenance-label"')
|
||||
assert not e.build_used(tmp_path).sources["pairs"]
|
||||
used = e.UsedSet()
|
||||
used.add("pairs", {"fixture/old", "fixture/nonexistent"}, 1)
|
||||
|
||||
class API:
|
||||
def get_repo(self, name):
|
||||
if name.endswith("nonexistent"):
|
||||
raise e.RepositoryNotFound
|
||||
return {"full_name": "fixture/new"}
|
||||
|
||||
e.canonicalize_used(used, API())
|
||||
assert "fixture/nonexistent" in used.sources["pairs"]
|
||||
assert e.rejection(e.resolve("fixture/new", API()), used, set()) == "pairs"
|
||||
|
||||
|
||||
def test_binary_measurement_and_local_pointer_discovery(tmp_path):
|
||||
target = tmp_path / "benchmarks/measurements/a/image.bin"
|
||||
target.parent.mkdir(parents=True)
|
||||
target.write_bytes(b"\xff\x00")
|
||||
plant(tmp_path, "benchmarks/pairs/a/recipes.toml", '[[pair]]\nrepo = "fixture/pointer"\nvendored = false')
|
||||
used = e.build_used(tmp_path)
|
||||
assert used.instrument["measurements"]["bytes"] == 2
|
||||
assert used.sources["pointers"] == {"fixture/pointer"}
|
||||
|
||||
|
||||
def test_coordinator_verification_is_black_box(tmp_path, monkeypatch):
|
||||
import subprocess
|
||||
from types import SimpleNamespace
|
||||
|
||||
def run(command, **kwargs):
|
||||
assert kwargs["stdout"] == kwargs["stderr"] == kwargs["stdin"] == subprocess.DEVNULL
|
||||
path = Path(next(part.split("=", 1)[1] for part in command if part.startswith("--junitxml=")))
|
||||
path.write_text('<testsuites><testsuite tests="25" failures="0" errors="0" skipped="1"/></testsuites>')
|
||||
return SimpleNamespace(returncode=0)
|
||||
|
||||
monkeypatch.setattr(subprocess, "run", run)
|
||||
assert e.verify_tests(tmp_path)["tests"]["tests"] == 25
|
||||
monkeypatch.setattr(subprocess, "run", lambda *args, **kwargs: SimpleNamespace(returncode=2))
|
||||
with pytest.raises(ValueError, match="guard_or_tests_failed"):
|
||||
e.verify_tests(tmp_path)
|
||||
|
||||
|
||||
def test_registered_unit_group_is_repository(tmp_path):
|
||||
plant(tmp_path, "plane/experiments/a.units.jsonl", json.dumps({"unit": "opaque", "group": "fixture/unit", "label": 1}))
|
||||
assert "fixture/unit" in e.build_used(tmp_path).sources["units"]
|
||||
|
||||
|
||||
def test_multidocument_measurement_and_missing_registered_units(tmp_path):
|
||||
plant(tmp_path, "benchmarks/measurements/a/run.yaml", "repo: fixture/first\n---\nrepo: fixture/second\n")
|
||||
assert e.build_used(tmp_path).sources["measurements"] == {"fixture/first", "fixture/second"}
|
||||
store = FileStore(tmp_path / "memory")
|
||||
plant(
|
||||
tmp_path,
|
||||
"memory/repos/opaque/pin.jsonl",
|
||||
json.dumps(
|
||||
{
|
||||
"kind": "experiment",
|
||||
"units_file": "absent.units.jsonl",
|
||||
"units_digest": "a" * 64,
|
||||
}
|
||||
),
|
||||
)
|
||||
with pytest.raises(FileNotFoundError):
|
||||
e.build_used(tmp_path, stores={"local": store})
|
||||
|
||||
|
||||
def test_catalog_skips_fixture_and_preserves_host_identity(tmp_path, capsys):
|
||||
plant(
|
||||
tmp_path,
|
||||
"benchmarks/pairs/catalog.toml",
|
||||
'[[pair]]\norigin = "in-tree-fixture"\nvendored = false\n[[pair]]\nrepo = "https://GitLab.com/Fixture/Private.git"\n',
|
||||
)
|
||||
used = e.build_used(tmp_path)
|
||||
assert used.sources["pairs"] == {"gitlab.com/fixture/private"}
|
||||
assert used.instrument["pairs"]["skipped_values"] == 1
|
||||
assert used.instrument["pointers"]["skipped_values"] == 1
|
||||
assert used.sources["sweep"] == set()
|
||||
e.canonicalize_used(used, FakeAPI({})) # Non-GitHub identities never reach this API.
|
||||
assert e.rejection(e.resolve("https://gitlab.com/fixture/private", FakeAPI({})), used, set()) == "pairs"
|
||||
report = used.report(set(e.SOURCES) - {"pairs"})
|
||||
assert report["sources"]["pairs"]["skipped_values"] == 1
|
||||
assert "fixture" not in json.dumps(report)
|
||||
assert capsys.readouterr() == ("", "")
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("value", "expected"),
|
||||
[
|
||||
("https://bitbucket.org/Fixture/Repo.git", "bitbucket.org/fixture/repo"),
|
||||
("https://git.example.org/Fixture/Repo", "git.example.org/fixture/repo"),
|
||||
("https://github.com/Fixture/Repo/tree/main", "fixture/repo"),
|
||||
("Fixture/Repo.git/", "fixture/repo"),
|
||||
],
|
||||
)
|
||||
def test_repository_hosts(value, expected):
|
||||
assert e.normalize(value) == expected
|
||||
assert e.normalize(expected) == expected
|
||||
|
||||
|
||||
def test_memory_values_are_skipped_and_counted(tmp_path, capsys):
|
||||
rows = [
|
||||
{"kind": "example", "repo": value}
|
||||
for value in ["local-slice", "free text https://github.com/fixture/prose", None, "https://bitbucket.org/fixture/repo"]
|
||||
]
|
||||
plant(tmp_path, "memory/repos/opaque/pin.jsonl", "\n".join(json.dumps(row) for row in rows))
|
||||
used = e.build_used(tmp_path, stores={"local": FileStore(tmp_path / "memory")})
|
||||
assert used.sources["memory_local"] == {"bitbucket.org/fixture/repo"}
|
||||
assert used.instrument["memory_local"]["skipped_values"] == 3
|
||||
assert capsys.readouterr() == ("", "")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("step", ["open_stores", "build_used", "report", "canonicalize", "verify_tests"])
|
||||
def test_cli_failure_reports_only_step_and_exception_type(tmp_path, monkeypatch, capsys, step):
|
||||
from openultrasast import config
|
||||
from openultrasast.plane import memory
|
||||
|
||||
def fail(*args, **kwargs):
|
||||
raise RuntimeError("sensitive/identity")
|
||||
|
||||
monkeypatch.setattr(config, "load_dotenv", lambda path: None)
|
||||
monkeypatch.setattr(memory, "open_store", lambda *args, **kwargs: object())
|
||||
monkeypatch.setattr(e, "build_used", lambda *args, **kwargs: e.UsedSet())
|
||||
monkeypatch.setattr(e.UsedSet, "report", lambda *args, **kwargs: {})
|
||||
monkeypatch.setattr(e, "GitHub", lambda token: object())
|
||||
monkeypatch.setattr(e, "canonicalize_used", lambda *args: None)
|
||||
monkeypatch.setattr(e, "verify_tests", lambda *args: {})
|
||||
if step == "open_stores":
|
||||
monkeypatch.setattr(memory, "open_store", fail)
|
||||
elif step == "report":
|
||||
monkeypatch.setattr(e.UsedSet, "report", fail)
|
||||
else:
|
||||
monkeypatch.setattr(e, "canonicalize_used" if step == "canonicalize" else step, fail)
|
||||
assert e.main(["--root", str(tmp_path), "--verify-tests"]) == 1
|
||||
output = capsys.readouterr()
|
||||
assert output.err == ""
|
||||
assert json.loads(output.out) == {"status": "instrument_failure", "step": step, "type": "RuntimeError"}
|
||||
assert "sensitive" not in output.out
|
||||
@@ -1,314 +0,0 @@
|
||||
"""Offline extraction: an injected local Git client over a synthetic temporary history."""
|
||||
|
||||
import importlib
|
||||
import subprocess
|
||||
import sys
|
||||
from dataclasses import replace
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
e = importlib.import_module("benchmarks.unseen.extract")
|
||||
s = importlib.import_module("benchmarks.unseen.source")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def history(tmp_path):
|
||||
calls = []
|
||||
|
||||
def runner(argv, **kwargs):
|
||||
assert not {"clone", "fetch", "push", "pull"} & set(argv)
|
||||
calls.append(argv)
|
||||
return subprocess.run(argv, **kwargs)
|
||||
|
||||
git = e.Git(tmp_path, runner=runner)
|
||||
git.run("init", "-q")
|
||||
git.run("config", "user.name", "Fixture")
|
||||
git.run("config", "user.email", "[email protected]")
|
||||
|
||||
def commit(text, message="ordinary", extra=None):
|
||||
(tmp_path / "app.py").write_text(text)
|
||||
for path, content in (extra or {}).items():
|
||||
(tmp_path / path).write_text(content)
|
||||
git.run("add", ".")
|
||||
git.run("commit", "-q", "-m", message)
|
||||
return git.run("rev-parse", "HEAD").strip()
|
||||
|
||||
root = commit("def handler(value):\n return value\n")
|
||||
git.run("tag", "v1.0")
|
||||
intro = commit("def handler(value):\n return eval(value)\n")
|
||||
git.run("tag", "v1.1")
|
||||
parent = commit("def handler(value):\n return eval( value )\n", "format")
|
||||
fix = commit("def handler(value):\n return int(value)\n", "security fix")
|
||||
candidate = s.Candidate(
|
||||
"fixture/project",
|
||||
"https://github.com/fixture/project",
|
||||
fix,
|
||||
parent,
|
||||
"GHSA-test-test-test",
|
||||
"pip",
|
||||
"injection",
|
||||
True,
|
||||
"1.1",
|
||||
"MIT",
|
||||
"permissive",
|
||||
parent,
|
||||
"LICENSE",
|
||||
)
|
||||
return git, commit, candidate, root, intro, parent, calls
|
||||
|
||||
|
||||
def test_introducing_with_reformat_walkback(history):
|
||||
git, _, candidate, root, intro, _, calls = history
|
||||
change = e.introducing(git, candidate)
|
||||
assert change["method"] == "introducing"
|
||||
assert (change["base"], change["head"]) == (root, intro)
|
||||
assert change["function"] == "handler"
|
||||
assert change["family"] == "injection" and change["post_cutoff"]
|
||||
assert change["files"][0]["head_lines"] == [[2, 2]]
|
||||
assert change["bias"] and change["advisory"] == candidate.advisory
|
||||
assert any("-w" in args and "-M" in args and "-C" in args for args in calls)
|
||||
assert git.bytes_read > 0
|
||||
|
||||
|
||||
def test_bracket_rejection_uses_fallback(history):
|
||||
git, _, candidate, _, _, parent, _ = history
|
||||
change = e.introducing(git, replace(candidate, first_affected="1.0"))
|
||||
assert change["method"] == "last_touch" and change["head"] == parent
|
||||
assert change["fallback_reasons"]["version_bracket"] >= 1
|
||||
|
||||
|
||||
def test_size_cap_and_merge_rejection(history):
|
||||
git, _, candidate, _, _, _, _ = history
|
||||
|
||||
class Oversize:
|
||||
def __getattr__(self, name):
|
||||
return getattr(git, name)
|
||||
|
||||
def stats(self, base, head):
|
||||
return [("app.py", 2001, 0)]
|
||||
|
||||
assert e.introducing(Oversize(), candidate)["method"] == "last_touch"
|
||||
|
||||
class Merge(Oversize):
|
||||
def parents(self, sha):
|
||||
parents = git.parents(sha)
|
||||
return parents + ["f" * 40] if sha != candidate.fix else parents
|
||||
|
||||
with pytest.raises(s.Rejected, match="no_range"):
|
||||
e.introducing(Merge(), candidate)
|
||||
|
||||
|
||||
def test_pure_addition_blames_function_declaration(history):
|
||||
git, commit, candidate, _, _, _, _ = history
|
||||
git.run("checkout", "-q", candidate.parent)
|
||||
parent = commit("def handler(value):\n return eval(value)\n\ndef added(value):\n return value\n")
|
||||
fix = commit("def handler(value):\n return eval(value)\n\ndef added(value):\n assert value\n return value\n")
|
||||
change = e.introducing(git, replace(candidate, fix=fix, parent=parent, first_affected=None, family="access_control"))
|
||||
assert change["head"] == parent and change["method"] == "introducing"
|
||||
assert change["function"] == "added"
|
||||
|
||||
|
||||
def test_git_failures_are_sanitized_and_not_empty(tmp_path):
|
||||
def failed(argv, **kwargs):
|
||||
return subprocess.CompletedProcess(argv, 128, b"", b"private repository secret")
|
||||
|
||||
with pytest.raises(e.InstrumentFailure, match="git_exit_128"):
|
||||
e.Git(tmp_path, runner=failed).run("log")
|
||||
|
||||
|
||||
def test_walkback_is_bounded_and_file_cap_falls_back(history):
|
||||
git, _, candidate, _, _, parent, _ = history
|
||||
|
||||
class FileCap:
|
||||
def __getattr__(self, name):
|
||||
return getattr(git, name)
|
||||
|
||||
def stats(self, base, head):
|
||||
return [("app.py", 1, 0)] * 51
|
||||
|
||||
result = e.introducing(FileCap(), candidate)
|
||||
assert result["method"] == "last_touch"
|
||||
assert result["fallback_reasons"]["size_cap"]
|
||||
|
||||
class Walk:
|
||||
calls = 0
|
||||
|
||||
def __getattr__(self, name):
|
||||
return getattr(git, name)
|
||||
|
||||
def blame(self, revision, path, line):
|
||||
self.calls += 1
|
||||
return parent, path, 2
|
||||
|
||||
def parents(self, revision):
|
||||
return [parent] if revision != candidate.fix else [candidate.parent]
|
||||
|
||||
walk = Walk()
|
||||
result = e.introducing(walk, candidate)
|
||||
assert walk.calls == 6
|
||||
assert result["fallback_reasons"]["walkback_limit"] == 1
|
||||
|
||||
|
||||
def test_root_rejected_and_head_function_resolved(history):
|
||||
git, _, candidate, root, _, _, _ = history
|
||||
|
||||
class Root:
|
||||
def __getattr__(self, name):
|
||||
return getattr(git, name)
|
||||
|
||||
def blame(self, revision, path, line):
|
||||
return root, path, 1
|
||||
|
||||
result = e.introducing(Root(), candidate)
|
||||
assert result["method"] == "last_touch"
|
||||
assert result["fallback_reasons"]["merge_or_root"] == 1
|
||||
assert result["function_lines"] == [1, 2]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"path,text",
|
||||
[
|
||||
("a.php", "<?php\nfunction handle($x) {\n return $x;\n}\n"),
|
||||
("a.js", "function handle(x) {\n return x;\n}\n"),
|
||||
("a.ts", "const handle = (x) => {\n return x;\n}\n"),
|
||||
("A.java", "class A {\n public String handle(String x) {\n return x;\n }\n}\n"),
|
||||
],
|
||||
)
|
||||
def test_supported_function_boundaries(path, text):
|
||||
ranges = e.functions(path, text)
|
||||
assert len(ranges) == 1 and ranges[0][0] == "handle"
|
||||
assert ranges[0][2] < len(text.splitlines()) + 1
|
||||
|
||||
|
||||
def test_ambiguous_parent_is_not_proof_of_absence(history):
|
||||
git, commit, candidate, _, _, _, _ = history
|
||||
duplicate = (
|
||||
"class A:\n def handler(self, value):\n return eval(value)\n\n"
|
||||
"class B:\n def handler(self, value):\n return value\n"
|
||||
)
|
||||
commit(duplicate)
|
||||
parent = commit("class A:\n def handler(self, value):\n return eval( value )\n")
|
||||
fix = commit("class A:\n def handler(self, value):\n return int(value)\n")
|
||||
selected = replace(candidate, fix=fix, parent=parent, first_affected=None)
|
||||
|
||||
class BlameLast:
|
||||
def __getattr__(self, name):
|
||||
return getattr(git, name)
|
||||
|
||||
def blame(self, revision, path, line):
|
||||
return parent, path, 3
|
||||
|
||||
result = e.introducing(BlameLast(), selected)
|
||||
assert result["method"] == "last_touch"
|
||||
assert result["fallback_reasons"]["ambiguous_parent"] == 1
|
||||
|
||||
|
||||
def ordinary(index, **overrides):
|
||||
row = {
|
||||
"head": f"{index:040x}",
|
||||
"parents": ["f" * 40],
|
||||
"timestamp": index * 100,
|
||||
"message": "update query token session valid",
|
||||
"author": "Human",
|
||||
"files": [{"path": "app.py", "added": 5, "removed": 0, "head_lines": [[2, 6]], "functions": ["ordinary"], "generated": False}],
|
||||
}
|
||||
row.update(overrides)
|
||||
return row
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"message",
|
||||
[
|
||||
"security fix",
|
||||
"CVE-2026",
|
||||
"GHSA-test",
|
||||
"vulnerability",
|
||||
"injection fix",
|
||||
"XSS",
|
||||
"CSRF",
|
||||
"SSRF",
|
||||
"exploit",
|
||||
"sanitize",
|
||||
"escaping",
|
||||
"Revert changes",
|
||||
"This reverts commit abc",
|
||||
"bump package",
|
||||
"upgrade package from 1 to 2",
|
||||
],
|
||||
)
|
||||
def test_ordinary_message_exclusions(message):
|
||||
assert e.ordinary_rejection(ordinary(1, message=message), set(), set()) is not None
|
||||
|
||||
|
||||
def test_narrow_filter_and_every_other_exclusion():
|
||||
assert e.ordinary_rejection(ordinary(1), set(), set()) is None
|
||||
assert e.ordinary_rejection(ordinary(1, parents=["a", "b"]), set(), set()) == "merge_or_root"
|
||||
assert e.ordinary_rejection(ordinary(1, author="dependabot[bot]"), set(), set()) == "dependency"
|
||||
assert e.ordinary_rejection(ordinary(1, author="renovate"), set(), set()) == "dependency"
|
||||
assert e.ordinary_rejection(ordinary(1), {f"{1:040x}"}, set()) == "advisory_fix"
|
||||
assert e.ordinary_rejection(ordinary(1), set(), {("app.py", "ordinary")}) == "vulnerable_function"
|
||||
row = ordinary(1)
|
||||
row["files"][0]["added"] = 1001
|
||||
assert e.ordinary_rejection(row, set(), set()) == "size_cap"
|
||||
for path, reason in [
|
||||
("README.md", "no_source"),
|
||||
("package.json", "manifest_only"),
|
||||
("package-lock.json", "manifest_only"),
|
||||
("vendor/a.py", "generated"),
|
||||
("node_modules/a.js", "generated"),
|
||||
("dist/a.js", "generated"),
|
||||
("app.min.js", "generated"),
|
||||
]:
|
||||
row = ordinary(1)
|
||||
row["files"][0]["path"] = path
|
||||
assert e.ordinary_rejection(row, set(), set()) == reason
|
||||
row = ordinary(1)
|
||||
row["files"][0]["generated"] = True
|
||||
assert e.ordinary_rejection(row, set(), set()) == "generated"
|
||||
|
||||
|
||||
def test_draw_seed_proportional_cells_floor_and_thirds():
|
||||
rows = [ordinary(i) for i in range(120)]
|
||||
for i, row in enumerate(rows):
|
||||
row["files"][0]["added"] = [5, 20, 100, 500][i % 4]
|
||||
drawn, counts = e.draw_ordinary(rows, set(), set(), seed=19)
|
||||
assert len(drawn) == 12 and len({r["head"] for r in drawn}) == 12
|
||||
assert {r["time_third"] for r in drawn} == {0, 1, 2}
|
||||
assert len({(r["time_third"], r["size_bucket"]) for r in drawn}) == 12
|
||||
assert drawn == e.draw_ordinary(list(reversed(rows)), set(), set(), seed=19)[0]
|
||||
assert counts["eligible"] == 120
|
||||
with pytest.raises(s.Rejected, match="ordinary_floor"):
|
||||
e.draw_ordinary(rows[:39], set(), set(), seed=19)
|
||||
# Thirds come from the history's time span, not rank among eligible changes.
|
||||
with pytest.raises(s.Rejected, match="time_thirds"):
|
||||
e.draw_ordinary([ordinary(i) for i in range(40)] + [ordinary(10000, message="security")], set(), set(), seed=1)
|
||||
|
||||
|
||||
def test_git_ordinary_collection_generated_attributes_and_headers(history):
|
||||
git, commit, _, _, _, _, _ = history
|
||||
commit("# Generated by fixture\nx = 2\n", extra={".gitattributes": "other.py linguist-generated=true\n", "other.py": "x = 1\n"})
|
||||
rows = list(e.ordinary_history(git, "HEAD"))
|
||||
assert rows and rows[0]["files"]
|
||||
files = {f["path"]: f for f in rows[0]["files"]}
|
||||
assert files["app.py"]["generated"] and files["other.py"]["generated"]
|
||||
assert files["app.py"]["head_lines"]
|
||||
|
||||
|
||||
def test_attribute_unset_and_mixed_generated_source(history):
|
||||
git, commit, _, _, _, _, _ = history
|
||||
sha = commit("x = 1\n", extra={".gitattributes": "*.py linguist-generated\napp.py -linguist-generated\n"})
|
||||
assert not e.generated(git, sha, "app.py", "x = 1\n")
|
||||
row = ordinary(1)
|
||||
row["files"].append({**row["files"][0], "path": "dist/b.js", "generated": True})
|
||||
assert e.ordinary_rejection(row, set(), set()) is None
|
||||
|
||||
|
||||
def test_unbalanced_cells_remain_proportional_and_seed_changes_draw():
|
||||
rows = [ordinary(i) for i in range(120)]
|
||||
for i, row in enumerate(rows):
|
||||
row["files"][0]["added"] = 5 if i % 4 else 100
|
||||
draw, _ = e.draw_ordinary(rows, set(), set(), seed=1)
|
||||
assert len([r for r in draw if r["size_bucket"] == 0]) == 9
|
||||
assert draw != e.draw_ordinary(rows, set(), set(), seed=2)[0]
|
||||
@@ -1,16 +0,0 @@
|
||||
"""Static image contract; no Docker invocation."""
|
||||
|
||||
import hashlib
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
ENGINE_SHA = "06ec67d411cd7abbfcd0735b1c2a70e4f3eb65ddcfeef89975709c664f59ad42"
|
||||
|
||||
|
||||
def test_image_stages():
|
||||
text = (ROOT / "plane/Dockerfile.engine-task").read_text()
|
||||
engine, replay = text.split("FROM engine AS replay", 1)
|
||||
original = engine.replace("v0.3.1 AS engine", "v0.3.1").rstrip() + "\n"
|
||||
assert hashlib.sha256(original.encode()).hexdigest() == ENGINE_SHA
|
||||
assert "FROM ghcr.io/norandom/ax-task-runner:v0.3.1 AS engine" in engine
|
||||
assert "git" in replay and "replay_entry.py" in replay and "ousast pre-push --help" in replay
|
||||
@@ -1,63 +0,0 @@
|
||||
"""Append-only use and tamper detection, with opaque fixture identifiers."""
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
with pytest.MonkeyPatch.context() as patch:
|
||||
patch.syspath_prepend(str(Path(__file__).resolve().parents[1] / "benchmarks/unseen"))
|
||||
import ledger
|
||||
|
||||
|
||||
def row(**extra):
|
||||
return {
|
||||
"slice": 1,
|
||||
"decision": "decision-001",
|
||||
"purpose": "baseline",
|
||||
"date": "2026-10-04",
|
||||
"result_digest": "a" * 64,
|
||||
"freeze_digest": "b" * 64,
|
||||
**extra,
|
||||
}
|
||||
|
||||
|
||||
def test_append_and_refuse_rewrite(tmp_path):
|
||||
path = tmp_path / "usage-p1.jsonl"
|
||||
ledger.append(path, row())
|
||||
original = path.read_bytes()
|
||||
ledger.append(path, row(purpose="exploratory"))
|
||||
assert path.read_bytes().startswith(original)
|
||||
assert len(ledger.read(path)) == 2
|
||||
with pytest.raises(ValueError, match="duplicate"):
|
||||
ledger.append(path, row())
|
||||
|
||||
|
||||
def test_informed_slice_cannot_qualify_same_decision(tmp_path):
|
||||
path = tmp_path / "usage-p1.jsonl"
|
||||
ledger.append(path, row(purpose="informed"))
|
||||
before = path.read_bytes()
|
||||
with pytest.raises(ValueError, match="informed"):
|
||||
ledger.append(path, row(purpose="qualifies"))
|
||||
assert path.read_bytes() == before
|
||||
ledger.append(path, row(purpose="qualifies", slice=2))
|
||||
ledger.append(path, row(purpose="qualifies", decision="decision-002"))
|
||||
|
||||
|
||||
def test_rewritten_row_breaks_digest_chain(tmp_path):
|
||||
path = tmp_path / "usage-p1.jsonl"
|
||||
ledger.append(path, row())
|
||||
ledger.append(path, row(purpose="exploratory"))
|
||||
rows = [json.loads(line) for line in path.read_text().splitlines()]
|
||||
rows[0]["purpose"] = "informed"
|
||||
path.write_text("".join(json.dumps(r) + "\n" for r in rows))
|
||||
with pytest.raises(ValueError, match="chain"):
|
||||
ledger.read(path)
|
||||
with pytest.raises(ValueError, match="chain"):
|
||||
ledger.append(path, row(slice=2))
|
||||
|
||||
|
||||
@pytest.mark.parametrize("extra", [{"decision": "owner/name"}, {"purpose": "other"}, {"freeze_digest": "bad"}, {"slice": 0}])
|
||||
def test_invalid_rows_refused(tmp_path, extra):
|
||||
with pytest.raises(ValueError):
|
||||
ledger.append(tmp_path / "usage-p1.jsonl", row(**extra))
|
||||
@@ -1,244 +0,0 @@
|
||||
"""Offline memory and semantic regression checks using synthetic identities only."""
|
||||
|
||||
import importlib
|
||||
import json
|
||||
import sys
|
||||
import tracemalloc
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
d = importlib.import_module("benchmarks.unseen.draft")
|
||||
s = importlib.import_module("benchmarks.unseen.source")
|
||||
j = importlib.import_module("benchmarks.unseen.journal")
|
||||
storage = importlib.import_module("benchmarks.unseen.storage")
|
||||
|
||||
|
||||
def advisory(index):
|
||||
return {
|
||||
"ghsa_id": f"GHSA-test-{index:08d}",
|
||||
"published_at": "2026-06-02T00:00:00Z" if index % 3 else "2025-01-01T00:00:00Z",
|
||||
"type": "reviewed",
|
||||
"withdrawn_at": None,
|
||||
"description": "An affected handler accepts untrusted input. " * 400,
|
||||
"cwes": [{"cwe_id": "CWE-89", "name": "SQL injection"}],
|
||||
"references": [
|
||||
{"url": f"https://github.com/fixture/project{index}/commit/{index:040x}"},
|
||||
*[{"url": f"https://example.invalid/details/{n}"} for n in range(30)],
|
||||
],
|
||||
"vulnerabilities": [
|
||||
{
|
||||
"package": {"ecosystem": s.ECOSYSTEMS[index % 4], "name": f"package-{n}"},
|
||||
"vulnerable_version_range": ", ".join(f">= {v}.0, < {v}.9" for v in range(50)),
|
||||
"first_patched_version": "50.0",
|
||||
}
|
||||
for n in range(8)
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
class OfflineAPI:
|
||||
def advisory_page(self, page):
|
||||
pytest.fail("A complete journal must not fetch advisories")
|
||||
|
||||
def get_repo(self, name):
|
||||
return {"full_name": name}
|
||||
|
||||
|
||||
def test_replay_20000_advisories_has_bounded_peak_memory(tmp_path):
|
||||
path = tmp_path / "journal"
|
||||
path.mkdir()
|
||||
(path / "inputs.json").write_text('{"seed": 19}')
|
||||
journal_file = path / "advisories.jsonl"
|
||||
# Write old, unprojected REST pages without ever building the entire fixture in RAM.
|
||||
with journal_file.open("w") as handle:
|
||||
for page in range(200):
|
||||
json.dump(
|
||||
{
|
||||
"id": str(page + 1),
|
||||
"value": {"rows": [advisory(i) for i in range(page * 100, (page + 1) * 100)], "done": page == 199},
|
||||
"elapsed_seconds": page,
|
||||
"rate_limit": {"remaining": "10"},
|
||||
},
|
||||
handle,
|
||||
)
|
||||
handle.write("\n")
|
||||
with journal_file.open("rb") as handle:
|
||||
first_line = handle.readline()
|
||||
assert len(json.loads(first_line)["value"]["rows"]) == 100
|
||||
assert len(first_line) > 2 * 1024 * 1024
|
||||
del first_line
|
||||
input_bytes = journal_file.stat().st_size
|
||||
assert input_bytes > 400 * 1024 * 1024
|
||||
print(f"journal_bytes_read={input_bytes}")
|
||||
tracemalloc.start()
|
||||
try:
|
||||
with j.Journal(path).open({"seed": 19}) as journal:
|
||||
assert len(journal.rows["advisories"]) == 200
|
||||
assert journal.elapsed_before == 199
|
||||
rows = d.enumerate_advisories(OfflineAPI(), journal)
|
||||
assert len(rows) == 20_000
|
||||
index = d.advisory_index(rows, OfflineAPI(), db=journal.db)
|
||||
assert index.get("fixture/project19999") == {f"{19999:040x}"}
|
||||
count = 0
|
||||
for row in s.ordered_advisories(rows, 19):
|
||||
assert "description" not in row
|
||||
assert all(set(v) == {"package"} for v in row["vulnerabilities"])
|
||||
count += 1
|
||||
assert count == 20_000
|
||||
assert journal.last == {"stage": "advisories", "id": "200"}
|
||||
_, peak = tracemalloc.get_traced_memory()
|
||||
finally:
|
||||
tracemalloc.stop()
|
||||
print(f"replay_enumeration_order_index_peak_bytes={peak}")
|
||||
assert peak < 64 * 1024 * 1024
|
||||
assert not list(path.glob("sourcing-*"))
|
||||
|
||||
|
||||
def test_disk_order_matches_original_seeded_order_and_duplicate_last_wins(tmp_path):
|
||||
raw = [advisory(i) for i in range(93)]
|
||||
raw[0]["vulnerabilities"] = [{"package": {"ecosystem": "other", "name": "fixture"}}]
|
||||
# One advisory lists multiple ecosystems; retain the original ecosystem priority.
|
||||
raw[1]["vulnerabilities"].append({"package": {"ecosystem": "pip", "name": "extra"}})
|
||||
expected = [s.compact_advisory(row) for row in raw]
|
||||
with storage.working_store(tmp_path) as db:
|
||||
rows = storage.AdvisoryRows(db, "advisories")
|
||||
for row in reversed(expected):
|
||||
rows[row["ghsa_id"]] = row
|
||||
rows[expected[2]["ghsa_id"]] = expected[2]
|
||||
assert len(rows) == len(expected)
|
||||
assert list(s.ordered_advisories(rows, 17)) == s.ordered_advisories(expected, 17)
|
||||
|
||||
|
||||
def test_compaction_preserves_candidate_and_all_fix_links():
|
||||
fixtures = importlib.import_module("test_unseen_source")
|
||||
row = fixtures.advisory()
|
||||
assert s.candidate(s.compact_advisory(row), fixtures.API(), fixtures.OSV()) == s.candidate(row, fixtures.API(), fixtures.OSV())
|
||||
row["references"] += [
|
||||
"https://github.com/fixture/old/commit/ABCDEF1#diff",
|
||||
{"url": "https://github.com/fixture/other/commit/" + "b" * 40},
|
||||
]
|
||||
assert s.fix_links(s.compact_advisory(row)) == s.fix_links(row)
|
||||
repo = {
|
||||
"full_name": "fixture/new",
|
||||
"private": False,
|
||||
"size": 10,
|
||||
"clone_url": "https://github.com/fixture/new",
|
||||
"source": {"full_name": "fixture/source", "description": "large"},
|
||||
"parent": {"full_name": "fixture/parent", "description": "large"},
|
||||
"former_names": ["fixture/old"],
|
||||
}
|
||||
eligibility = importlib.import_module("benchmarks.unseen.eligibility")
|
||||
|
||||
class API:
|
||||
def __init__(self, data):
|
||||
self.data = data
|
||||
|
||||
def get_repo(self, name):
|
||||
return self.data
|
||||
|
||||
assert eligibility.resolve("fixture/old", API(repo)) == eligibility.resolve("fixture/old", API(s.compact_repository(repo)))
|
||||
|
||||
|
||||
def test_legacy_journal_migration_keeps_payload_and_refuses_changed_parameters(tmp_path):
|
||||
fixtures = importlib.import_module("test_unseen_draft")
|
||||
used = fixtures.e.UsedSet()
|
||||
parameters = d.inputs(used, 19, set(used.sources), fixtures.FakeClones())
|
||||
legacy = d.legacy_inputs(parameters)
|
||||
path = tmp_path / "journal"
|
||||
with j.Journal(path).open(legacy) as journal:
|
||||
journal.put("advisories", "1", {"rows": [advisory(1)], "done": True})
|
||||
original = (path / "advisories.jsonl").read_bytes()
|
||||
for changed in ({**parameters, "seed": 20}, {**parameters, "design_taxonomy": "different"}):
|
||||
with pytest.raises(j.ResumeMismatch), j.Journal(path).open(changed, compatible_parameters=d.legacy_inputs(changed)):
|
||||
pass
|
||||
with j.Journal(path).open(parameters, compatible_parameters=legacy) as journal:
|
||||
assert journal.rows["advisories"]["1"]["rows"][0]["ghsa_id"] == advisory(1)["ghsa_id"]
|
||||
assert json.loads((path / "inputs.json").read_text()) == parameters
|
||||
assert json.loads((path / "inputs-before-memory-fix.json").read_text()) == legacy
|
||||
assert (path / "advisories.jsonl").read_bytes() == original
|
||||
|
||||
|
||||
def test_replay_duplicate_torn_and_oversized_records_fail_closed(tmp_path, monkeypatch):
|
||||
path = tmp_path / "journal"
|
||||
with j.Journal(path).open({}) as journal:
|
||||
journal.put("advisories", "1", {"rows": [], "done": True})
|
||||
record = (path / "advisories.jsonl").read_bytes()
|
||||
for data, reason in ((record * 2, "duplicate_unit"), (record[:-1], "incomplete_record")):
|
||||
(path / "advisories.jsonl").write_bytes(data)
|
||||
with pytest.raises(ValueError, match=reason), j.Journal(path).open({}):
|
||||
pass
|
||||
assert not list(path.glob("sourcing-*"))
|
||||
monkeypatch.setattr(j, "MAX_RECORD_BYTES", 16)
|
||||
(path / "advisories.jsonl").write_bytes(record)
|
||||
with pytest.raises(ValueError, match="record_size_cap"), j.Journal(path).open({}):
|
||||
pass
|
||||
|
||||
|
||||
def test_ordinary_sampling_streams_payloads_and_preserves_frozen_draw():
|
||||
fixtures = importlib.import_module("test_unseen_extract")
|
||||
extract = fixtures.e
|
||||
rows = [fixtures.ordinary(i) for i in range(120)]
|
||||
for i, row in enumerate(rows):
|
||||
row["files"][0]["added"] = [5, 20, 100, 500][i % 4]
|
||||
# Captured from the pre-memory-fix implementation, not a second copy of its algorithm.
|
||||
drawn, counts = extract.draw_ordinary(iter(rows), set(), set(), seed=19)
|
||||
assert [int(row["head"], 16) for row in drawn] == [0, 33, 6, 35, 52, 65, 62, 75, 96, 117, 90, 119]
|
||||
assert counts == {"eligible": 120}
|
||||
|
||||
def history():
|
||||
for i in range(20_000):
|
||||
row = fixtures.ordinary(i)
|
||||
row["message"] = "ordinary change " * 1500
|
||||
yield row
|
||||
|
||||
tracemalloc.start()
|
||||
try:
|
||||
drawn, counts = extract.draw_ordinary(history(), set(), set(), seed=19)
|
||||
_, peak = tracemalloc.get_traced_memory()
|
||||
finally:
|
||||
tracemalloc.stop()
|
||||
assert counts == {"eligible": 20_000}
|
||||
assert len(drawn) == 12
|
||||
assert peak < 16 * 1024 * 1024
|
||||
|
||||
|
||||
def test_git_spools_large_output_and_refuses_large_blob_before_reading(tmp_path):
|
||||
import subprocess
|
||||
|
||||
extract = importlib.import_module("benchmarks.unseen.extract")
|
||||
|
||||
def runner(argv, **kwargs):
|
||||
assert kwargs["stdout"] != subprocess.PIPE
|
||||
assert kwargs["stderr"] != subprocess.PIPE
|
||||
for _ in range(9):
|
||||
kwargs["stdout"].write(b"x" * 1024 * 1024)
|
||||
return subprocess.CompletedProcess(argv, 0)
|
||||
|
||||
with pytest.raises(extract.InstrumentFailure, match="git_output_size_cap"):
|
||||
extract.Git(tmp_path, runner=runner).run("log")
|
||||
calls = []
|
||||
git = extract.Git(tmp_path)
|
||||
|
||||
def run(*args):
|
||||
calls.append(args[0])
|
||||
return "app.py" if args[0] == "ls-tree" else str(512 * 1024 + 1)
|
||||
|
||||
git.run = run
|
||||
with pytest.raises(extract.InstrumentFailure, match="source_blob_size_cap"):
|
||||
git.blob("a" * 40, "app.py")
|
||||
assert calls == ["ls-tree", "cat-file"]
|
||||
assert git.bytes_read == git.files_read == 0
|
||||
|
||||
|
||||
def test_repository_cache_is_bounded_and_drops_payloads():
|
||||
class Transport:
|
||||
def get(self, url, headers):
|
||||
return {"full_name": url.rsplit("/", 2)[-2] + "/" + url.rsplit("/", 1)[-1], "description": "large" * 1000}
|
||||
|
||||
github = s.GitHub("offline", transport=Transport())
|
||||
for i in range(200):
|
||||
github.get_repo(f"fixture/project{i}")
|
||||
assert len(github.repos) == 128
|
||||
assert all("description" not in row for row in github.repos.values())
|
||||
@@ -1,56 +0,0 @@
|
||||
"""Proof records fail closed using synthetic local results."""
|
||||
|
||||
import importlib
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
p = importlib.import_module("benchmarks.unseen.proof")
|
||||
|
||||
|
||||
def test_replay_proof(tmp_path):
|
||||
roots = {}
|
||||
for lane in ("kind", "ax", "vm"):
|
||||
root = tmp_path / lane
|
||||
root.mkdir()
|
||||
roots[lane] = root
|
||||
(root / "inputs.json").write_text("[]")
|
||||
for i in range(5):
|
||||
(root / f"{i}.json").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"input_digest": str(i),
|
||||
"status": "engine_covered",
|
||||
"instrument": {"hook_bytes_read": 20, "jvm_wall_seconds": 8, "seconds": 30, "hook_exit": 0},
|
||||
"push": {"quick_tier": [{"findings": []}]},
|
||||
}
|
||||
)
|
||||
)
|
||||
assert p.replay_proof(roots)["quick_tier_equal"]
|
||||
path = roots["vm"] / "0.json"
|
||||
record = json.loads(path.read_text())
|
||||
record["instrument"]["jvm_wall_seconds"] = 0
|
||||
path.write_text(json.dumps(record))
|
||||
with pytest.raises(ValueError, match="instrument"):
|
||||
p.replay_proof(roots)
|
||||
|
||||
|
||||
def test_engine_proof_counts_differences_without_names(tmp_path):
|
||||
current, baseline = tmp_path / "current", tmp_path / "baseline"
|
||||
current.mkdir()
|
||||
baseline.mkdir()
|
||||
for i in range(2):
|
||||
record = {
|
||||
"repo": "synthetic",
|
||||
"pin": str(i),
|
||||
"done": True,
|
||||
"seconds": 20,
|
||||
"units": [{"unit": str(i), "status": "path", "instrument": {"bytes": 20, "jvm": [{"seconds": 8, "success": True}]}}],
|
||||
}
|
||||
(current / f"{i}.json").write_text(json.dumps(record))
|
||||
(baseline / f"{i}.json").write_text(json.dumps(record))
|
||||
proof = p.engine_proof(current, baseline)
|
||||
assert proof["differences"] == 0 and "synthetic" not in json.dumps(proof)
|
||||
@@ -1,79 +0,0 @@
|
||||
"""Replay workload privacy, validation and input identity."""
|
||||
|
||||
import importlib
|
||||
import io
|
||||
import json
|
||||
import sys
|
||||
import tarfile
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
r = importlib.import_module("benchmarks.unseen.replay")
|
||||
|
||||
|
||||
def test_items_private_and_arm_identity():
|
||||
change = {
|
||||
"id": "repo-name-secret",
|
||||
"repo": "owner/private-name",
|
||||
"repository_url": "https://public.example/private-name",
|
||||
"base": "a" * 40,
|
||||
"head": "b" * 40,
|
||||
"files": [{"path": "app.py", "head_lines": [[1, 2]]}],
|
||||
}
|
||||
image = "registry/replay@sha256:" + "a" * 64
|
||||
a = r.make_item(change, "default", image)
|
||||
b = r.make_item(change, "long_deadline", image)
|
||||
assert a.input_digest != b.input_digest
|
||||
assert "private-name" not in a.id and "repo-name" not in a.id
|
||||
assert set(a.inputs) == {"SPEC_URL"} and not a.extra_env
|
||||
spec = json.loads(a.inputs["SPEC_URL"])
|
||||
assert spec["repository_url"] == change["repository_url"]
|
||||
assert json.loads(b.inputs["SPEC_URL"])["hook_flags"] == ["--deadline", "300"]
|
||||
|
||||
|
||||
def test_validator_requires_instrument(tmp_path):
|
||||
buffer = io.BytesIO()
|
||||
with tarfile.open(fileobj=buffer, mode="w") as tar:
|
||||
data = b"{}"
|
||||
info = tarfile.TarInfo("push.json")
|
||||
info.size = len(data)
|
||||
tar.addfile(info, io.BytesIO(data))
|
||||
with pytest.raises(ValueError, match="instrument"):
|
||||
r.validate_result(buffer.getvalue(), tmp_path)
|
||||
|
||||
|
||||
def test_frozen_toml_slice_and_scoring_join(tmp_path):
|
||||
import hashlib
|
||||
|
||||
from benchmarks.unseen.score import load_records
|
||||
|
||||
pool = tmp_path / "pool-p1.toml"
|
||||
pool.write_text("""pool="p1"
|
||||
[[repository]]
|
||||
id="opaque-a"
|
||||
slice=1
|
||||
url="https://public.example/repo-a"
|
||||
changes=[{kind="ordinary",base="aaa",head="bbb",files=[]}]
|
||||
[[repository]]
|
||||
id="opaque-b"
|
||||
slice=2
|
||||
url="https://public.example/repo-b"
|
||||
changes=[{kind="ordinary",base="ccc",head="ddd",files=[]}]
|
||||
""")
|
||||
import tomllib
|
||||
|
||||
freeze = hashlib.sha256(r.canonical(tomllib.loads(pool.read_text()))).hexdigest()
|
||||
(tmp_path / "freeze-p1.json").write_text(json.dumps({"freeze_digest": freeze}))
|
||||
changes = r.load_slice(pool, 2)
|
||||
assert len(changes) == 1 and changes[0]["repo"] == "opaque-b"
|
||||
(tmp_path / "inputs.json").write_text(json.dumps([{"item_id": "replay-opaque", "change": changes[0], "input_digest": "digest"}]))
|
||||
assert load_records(tmp_path)[0]["kind"] == "ordinary"
|
||||
assert "instrument" not in load_records(tmp_path)[0] # Missing result remains a measured failure.
|
||||
result = tmp_path / (hashlib.sha256(b"replay-opaque").hexdigest() + ".json")
|
||||
result.write_text(json.dumps({"input_digest": "digest", "instrument": {"head_verified": True}}))
|
||||
assert load_records(tmp_path)[0]["instrument"]["head_verified"]
|
||||
pool.write_text(pool.read_text() + "changed=true\n")
|
||||
with pytest.raises(ValueError, match="digest"):
|
||||
r.load_slice(pool, 2)
|
||||
@@ -1,136 +0,0 @@
|
||||
"""Replay entry uses real local git and an injected transfer, never a network."""
|
||||
|
||||
import importlib
|
||||
import io
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
e = importlib.import_module("benchmarks.unseen.replay_entry")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def local_repo(tmp_path):
|
||||
repo = tmp_path / "repo"
|
||||
repo.mkdir()
|
||||
|
||||
def git(*args):
|
||||
return subprocess.run(["git", "-C", str(repo), *args], capture_output=True, text=True, check=True).stdout.strip()
|
||||
|
||||
git("init", "-q")
|
||||
git("config", "user.email", "[email protected]")
|
||||
git("config", "user.name", "Fixture")
|
||||
(repo / "app.py").write_text("print(1)\n")
|
||||
git("add", "app.py")
|
||||
git("commit", "-qm", "base")
|
||||
base = git("rev-parse", "HEAD")
|
||||
(repo / "app.py").write_text("print(2)\n")
|
||||
git("commit", "-qam", "head")
|
||||
return repo, base, git("rev-parse", "HEAD")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("failure", [None, "missing", "fetch", "crash", "fast", "unread", "unsupported"])
|
||||
def test_entry(local_repo, tmp_path, monkeypatch, failure):
|
||||
repo, base, head = local_repo
|
||||
spec = {
|
||||
"repository_url": "https://public.example/project.git",
|
||||
"base": base,
|
||||
"head": head,
|
||||
"id": "opaque",
|
||||
"image": "digest",
|
||||
"hook_flags": [],
|
||||
}
|
||||
if failure == "missing":
|
||||
spec["head"] = "a" * 40
|
||||
received = {}
|
||||
|
||||
def transfer(request, destination=None, **kwargs):
|
||||
assert kwargs["readiness_budget"] == 60
|
||||
if destination:
|
||||
destination.write_text(json.dumps(spec))
|
||||
else:
|
||||
with tarfile.open(fileobj=io.BytesIO(request.data)) as archive:
|
||||
received.update({m.name: archive.extractfile(m).read() for m in archive.getmembers()})
|
||||
|
||||
monkeypatch.setenv("SPEC_URL", "https://files.example/spec.dat?secret")
|
||||
monkeypatch.setenv("RESULT_URL", "https://files.example/result.dat?secret")
|
||||
monkeypatch.setattr(e, "transfer", transfer)
|
||||
real_run = subprocess.run
|
||||
|
||||
def run(command, **kwargs):
|
||||
if command[0] == "ousast":
|
||||
assert command[command.index("--base") + 1] == base
|
||||
assert kwargs["env"]["OPENULTRASAST_CPG_HEAP_MB"] == "1700"
|
||||
if failure == "crash":
|
||||
return subprocess.CompletedProcess(command, 2, "", "crash https://secret")
|
||||
target = Path(command[command.index("--artifact") + 1])
|
||||
target.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"quick_tier": [{"files": ["app.py"], "bytes_read": 0 if failure == "unread" else 9, "findings": []}],
|
||||
"scans": [
|
||||
{
|
||||
"side": "head",
|
||||
"scan": {"build_seconds": 0.1 if failure == "fast" else 8, "query_seconds": 2, "question_outcomes": []},
|
||||
}
|
||||
],
|
||||
}
|
||||
)
|
||||
)
|
||||
if failure == "unsupported":
|
||||
target.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"quick_tier": [{"files": [], "bytes_read": 0, "findings": []}],
|
||||
"admission": {"coverage_reasons": ["language_not_covered:rust:1:1"]},
|
||||
"scans": [
|
||||
{
|
||||
"side": "head",
|
||||
"scan": {
|
||||
"build_seconds": 0.01,
|
||||
"query_seconds": 0,
|
||||
"partitions": [
|
||||
{
|
||||
"language": "rust",
|
||||
"frontend": None,
|
||||
"paths": ["app.rs"],
|
||||
"source_bytes": 0,
|
||||
"status": "unsupported",
|
||||
}
|
||||
],
|
||||
},
|
||||
}
|
||||
],
|
||||
}
|
||||
)
|
||||
)
|
||||
return subprocess.CompletedProcess(command, 0, "Skipped: deadline\n", "")
|
||||
command = [
|
||||
str(repo) if arg == spec["repository_url"] else "protocol.file.allow=always" if arg == "protocol.file.allow=never" else arg
|
||||
for arg in command
|
||||
]
|
||||
if failure == "fetch" and "fetch" in command:
|
||||
return subprocess.CompletedProcess(command, 1, "", "fetch secret")
|
||||
return real_run(command, **kwargs)
|
||||
|
||||
result = e.main(work_root=tmp_path / "work", runner=run)
|
||||
instrument = json.loads(received["instrument.json"])
|
||||
assert instrument["image"] == "digest"
|
||||
assert b"secret" not in received["instrument.json"]
|
||||
if failure == "unsupported":
|
||||
from benchmarks.unseen.score import coverage
|
||||
|
||||
assert result == 0 and not instrument["suspect_fast"]
|
||||
assert coverage({"instrument": instrument, "push": json.loads(received["push.json"])}) == "unanalysable"
|
||||
elif failure:
|
||||
assert result == 1 and instrument["status"] == "instrument_failure"
|
||||
else:
|
||||
assert result == 0 and instrument["changed_bytes"] == {"app.py": 9}
|
||||
assert instrument["head_verified"] and instrument["base_verified"]
|
||||
assert instrument["jvm_wall_seconds"] == 10
|
||||
assert b"Skipped:" in received["push.txt"]
|
||||
@@ -1,127 +0,0 @@
|
||||
"""Synthetic push-level statistical controls; no corpus reads."""
|
||||
|
||||
import importlib
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
s = importlib.import_module("benchmarks.unseen.score")
|
||||
|
||||
|
||||
def record(kind="introducing", **updates):
|
||||
return {
|
||||
"id": "a",
|
||||
"repo": "opaque",
|
||||
"kind": kind,
|
||||
"family": "injection",
|
||||
"function": "handler",
|
||||
"files": [{"path": "app.py", "head_lines": [[20, 22]]}],
|
||||
"instrument": {
|
||||
"head_verified": True,
|
||||
"base_verified": True,
|
||||
"changed_bytes": {"app.py": 20},
|
||||
"hook_exit": 0,
|
||||
"hook_bytes_read": 20,
|
||||
},
|
||||
"push": {"quick_tier": [{"bytes_read": 20, "findings": []}]},
|
||||
**updates,
|
||||
}
|
||||
|
||||
|
||||
def test_coverage_and_failure_denominator():
|
||||
r = record()
|
||||
assert s.coverage(r) == "quick_only"
|
||||
r["push"]["scans"] = [{"side": "head", "scan": {"question_outcomes": [{"status": "completed", "identity": {"path": "app.py"}}]}}]
|
||||
assert s.coverage(r) == "engine_covered"
|
||||
r["instrument"]["suspect_fast"] = True
|
||||
assert s.coverage(r) == "instrument_failure"
|
||||
assert s.coverage(record(push={"skipped": ["language_not_covered:other:1:1"]})) == "unanalysable"
|
||||
assert s.coverage({}) == "instrument_failure"
|
||||
with pytest.raises(ValueError, match="5%"):
|
||||
s.score([record(), r])
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"p,width,n", [(0.05, 0.03, 315), (0.1, 0.03, 483), (0.2, 0.03, 756), (0.1, 0.1, 62), (0.25, 0.1, 88), (0.4, 0.1, 97)]
|
||||
)
|
||||
def test_sample_sizes(p, width, n):
|
||||
assert s.required_n(p, width) == n
|
||||
|
||||
|
||||
def test_catch_and_all_stream_false_alarm():
|
||||
r = record()
|
||||
r["push"]["quick_tier"][0]["findings"] = [{"cwe": "CWE-89", "path": "app.py", "line": 20}]
|
||||
assert s.catches(r) == ["quick"]
|
||||
r["push"]["quick_tier"][0]["findings"][0]["line"] = 100
|
||||
assert not s.catches(r)
|
||||
r["kind"] = "ordinary"
|
||||
assert s.false_alarm(r)
|
||||
assert not s.false_alarm(r, alerts_only=True)
|
||||
|
||||
|
||||
def test_location_rules():
|
||||
assert s.location_match(record(family="access_control"), {"path": "app.py", "function": "handler", "line": 200})
|
||||
assert not s.location_match(record(family="config_secrets"), {"path": "app.py", "function": "handler", "line": 200})
|
||||
assert s.location_match(record(family="config_secrets"), {"path": "app.py", "line": 30})
|
||||
assert not s.location_match(record(), {"path": "else.py", "function": "handler", "line": 20})
|
||||
|
||||
|
||||
def test_bootstrap_icc_mcnemar():
|
||||
rows = [("a", 1), ("a", 1), ("b", 0), ("b", 0)]
|
||||
assert s.cluster_interval(rows, seed=7) == s.cluster_interval(rows, seed=7)
|
||||
assert s.icc(rows) == pytest.approx(1)
|
||||
assert s.mcnemar([True] * 6, [False] * 6) == pytest.approx(0.03125)
|
||||
assert s.mcnemar([True], [True]) == 1
|
||||
|
||||
|
||||
def test_budgets():
|
||||
rows = [{"kind": "ordinary", "score": x} for x in range(100)] + [{"kind": "introducing", "score": 99}]
|
||||
result = s.budgets(rows)
|
||||
assert result["5%"]["threshold"] == 95
|
||||
assert result["5%"]["label"] == "exploratory"
|
||||
assert s.budgets(rows, thresholds={"5%": 96})["5%"]["label"] == "fixed"
|
||||
binary = s.budgets([{"kind": "ordinary", "prediction": False}, {"kind": "introducing", "prediction": True}], binary=True)
|
||||
assert binary["budgets_met"] == ["1%", "5%", "10%"]
|
||||
|
||||
|
||||
def test_gate():
|
||||
a = [record(kind="ordinary", id=str(i), repo=str(i)) for i in range(10)] + [record(id="v")]
|
||||
b = [
|
||||
record(kind="ordinary", id=str(i), repo=str(i), push={"quick_tier": [{"findings": [{"cwe": "CWE-89", "path": "x", "line": 1}]}]})
|
||||
for i in range(10)
|
||||
] + [record(id="v")]
|
||||
assert s.gate(a, b, slice_id="p1/2")["gate"] == "regression"
|
||||
assert s.gate(a, b)["gate"] == "not_run"
|
||||
assert s.gate(a, b, slice_id="p1/2", informed=True)["gate"] == "not_run"
|
||||
assert s.gate(a, [{}], slice_id="p1/2")["gate"] == "not_run"
|
||||
|
||||
|
||||
def test_real_artifact_coverage_and_location_binding():
|
||||
r = record(path="app.py", function_lines=[1, 100], files=[{"path": "other.py", "head_lines": [[200, 210]]}])
|
||||
assert not s.location_match(r, {"path": "other.py", "line": 40})
|
||||
assert (
|
||||
s.coverage(
|
||||
record(
|
||||
instrument={"head_verified": True, "base_verified": True, "changed_bytes": {"x": 20}, "hook_exit": 0, "hook_bytes_read": 0},
|
||||
push={"admission": {"coverage_reasons": ["language_not_covered:unknown:1:1"]}},
|
||||
)
|
||||
)
|
||||
== "unanalysable"
|
||||
)
|
||||
r["push"]["quick_tier"][0]["findings"] = [{"cwe": None, "path": "other.py", "line": 200}]
|
||||
assert not s.catches(r)
|
||||
assert s.false_alarm(r)
|
||||
|
||||
|
||||
def test_committed_synthetic_fixture():
|
||||
import json
|
||||
|
||||
fixture = Path(__file__).resolve().parents[1] / "benchmarks/unseen/fixtures/scorer.json"
|
||||
payload = fixture.read_bytes()
|
||||
assert len(payload) > 0
|
||||
report = s.score(json.loads(payload))
|
||||
assert report["catch"]["n"] == 4 and report["catch"]["count"] == 2
|
||||
assert report["false_alarm"]["n"] == 16 and report["false_alarm"]["count"] == 1
|
||||
assert report["coverage"]["quick_only"] == 20
|
||||
@@ -1,182 +0,0 @@
|
||||
"""Sanitized REST-shaped advisory fixtures; every external client is a fake."""
|
||||
|
||||
import copy
|
||||
import importlib
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
s = importlib.import_module("benchmarks.unseen.source")
|
||||
|
||||
|
||||
def advisory():
|
||||
path = Path(__file__).resolve().parents[1] / "benchmarks/unseen/fixtures/advisory.json"
|
||||
return json.loads(path.read_bytes())
|
||||
|
||||
|
||||
class API:
|
||||
def __init__(self):
|
||||
self.parents = ["b" * 40]
|
||||
self.spdx = "MIT"
|
||||
self.license_text = "SPDX-License-Identifier: MIT"
|
||||
self.calls = []
|
||||
self.repo = {"full_name": "fixture/project", "private": False, "size": 100, "clone_url": "https://github.com/fixture/project.git"}
|
||||
|
||||
def get_repo(self, name):
|
||||
return copy.deepcopy(self.repo)
|
||||
|
||||
def get_commit(self, name, sha):
|
||||
return {"sha": sha, "parents": [{"sha": p} for p in self.parents]}
|
||||
|
||||
def get_license(self, name, ref):
|
||||
self.calls.append((name, ref))
|
||||
return {"text": self.license_text, "spdx_id": self.spdx, "path": "LICENSE"}
|
||||
|
||||
|
||||
class OSV:
|
||||
def get(self, identifier):
|
||||
return {
|
||||
"affected": [
|
||||
{
|
||||
"package": {"ecosystem": "PyPI", "name": "fixture"},
|
||||
"ranges": [{"type": "ECOSYSTEM", "events": [{"introduced": "1.2"}, {"fixed": "1.3"}]}],
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
def test_candidate_cwe_cutoff_bracket_and_pinned_license():
|
||||
api = API()
|
||||
candidate = s.candidate(advisory(), api, OSV())
|
||||
assert candidate.family == "injection" and candidate.post_cutoff
|
||||
assert candidate.first_affected == "1.2"
|
||||
assert candidate.license_class == "permissive"
|
||||
assert api.calls == [("fixture/project", "b" * 40)]
|
||||
raw = advisory()
|
||||
raw["published_at"] = "2026-06-01T12:00:00Z"
|
||||
assert not s.candidate(raw, api, OSV()).post_cutoff
|
||||
raw["published_at"] = "2018-12-31T00:00:00Z"
|
||||
with pytest.raises(s.Rejected, match="date"):
|
||||
s.candidate(raw, api, OSV())
|
||||
|
||||
|
||||
@pytest.mark.parametrize("spdx", ["GPL-3.0", "AGPL-3.0-only", "LGPL-2.1", "NOASSERTION", "", "BUSL-1.1", "MIT OR GPL-3.0"])
|
||||
def test_private_licenses(spdx):
|
||||
api = API()
|
||||
api.spdx = spdx
|
||||
api.license_text = ""
|
||||
assert s.candidate(advisory(), api, OSV()).license_class == "private"
|
||||
|
||||
|
||||
def test_license_file_overrides_api_and_missing_file_is_private():
|
||||
api = API()
|
||||
api.license_text = "SPDX-License-Identifier: GPL-3.0-only"
|
||||
assert s.candidate(advisory(), api, OSV()).license_class == "private"
|
||||
api.license_text = ""
|
||||
api.spdx = "Apache-2.0"
|
||||
assert s.candidate(advisory(), api, OSV()).license_spdx == "Apache-2.0"
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"mutation,reason",
|
||||
[
|
||||
(lambda a, api: api.parents.append("c" * 40), "parents"),
|
||||
(lambda a, api: a["references"].append("https://github.com/fixture/other/commit/" + "d" * 40), "fix_links"),
|
||||
(lambda a, api: a.update(cwes=[{"cwe_id": "CWE-99999"}]), "family"),
|
||||
(lambda a, api: api.repo.update(size=512001), "repository"),
|
||||
(lambda a, api: api.repo.update(private=True), "repository"),
|
||||
(lambda a, api: api.repo.update(clone_url="ssh://fixture/project"), "repository"),
|
||||
],
|
||||
)
|
||||
def test_rejections(mutation, reason):
|
||||
api, raw = API(), advisory()
|
||||
mutation(raw, api)
|
||||
with pytest.raises(s.Rejected, match=reason):
|
||||
s.candidate(raw, api, OSV())
|
||||
|
||||
|
||||
def test_post_cutoff_first_within_ecosystem_is_seeded():
|
||||
rows = [advisory() for _ in range(12)]
|
||||
for i, row in enumerate(rows):
|
||||
row["ghsa_id"] += str(i)
|
||||
if i % 2:
|
||||
row["published_at"] = "2025-01-01T00:00:00Z"
|
||||
ordered = s.ordered_advisories(rows, 17)
|
||||
assert ordered == s.ordered_advisories(list(reversed(rows)), 17)
|
||||
assert all(s.post_cutoff(row) for row in ordered[:6])
|
||||
|
||||
|
||||
def test_http_adapters_with_fake_transport():
|
||||
import base64
|
||||
from urllib.parse import parse_qs, urlsplit
|
||||
|
||||
class Transport:
|
||||
def __init__(self):
|
||||
self.calls = []
|
||||
|
||||
def get(self, url, headers):
|
||||
self.calls.append((url, headers))
|
||||
parsed = urlsplit(url)
|
||||
if parsed.path == "/advisories":
|
||||
page = int(parse_qs(parsed.query)["page"][0])
|
||||
return [advisory()] * (100 if page == 1 else 1)
|
||||
if "/license" in parsed.path:
|
||||
assert parse_qs(parsed.query)["ref"] == ["b" * 40]
|
||||
return {
|
||||
"license": {"spdx_id": "MIT"},
|
||||
"path": "LICENSE",
|
||||
"content": base64.b64encode(b"SPDX-License-Identifier: MIT").decode(),
|
||||
}
|
||||
if "/commits/" in parsed.path:
|
||||
return {"sha": "a" * 40, "parents": [{"sha": "b" * 40}]}
|
||||
if parsed.hostname == "api.osv.dev":
|
||||
return OSV().get("ignored")
|
||||
return API().repo
|
||||
|
||||
transport = Transport()
|
||||
api = s.GitHub("fake-token", transport)
|
||||
assert len(list(api.advisories())) == 101
|
||||
assert s.candidate(advisory(), api, s.OSV(transport)).first_affected == "1.2"
|
||||
assert all(headers["Authorization"] == "Bearer fake-token" for url, headers in transport.calls if "api.github.com" in url)
|
||||
count = api.calls
|
||||
api.get_repo("fixture/project")
|
||||
assert api.calls == count
|
||||
|
||||
|
||||
def test_http_failures_are_not_empty_results():
|
||||
class Transport:
|
||||
def get(self, url, headers):
|
||||
raise ValueError("http_failure")
|
||||
|
||||
with pytest.raises(ValueError, match="http_failure"):
|
||||
list(s.GitHub("fake", Transport()).advisories())
|
||||
with pytest.raises(ValueError, match="http_failure"):
|
||||
s.OSV(Transport()).get("GHSA-test-test-test")
|
||||
|
||||
|
||||
def test_transport_rate_limit_retry_and_sanitized_failure():
|
||||
import io
|
||||
from urllib.error import HTTPError
|
||||
|
||||
waits, attempts = [], []
|
||||
|
||||
def opener(request, timeout):
|
||||
attempts.append(request)
|
||||
if len(attempts) == 1:
|
||||
raise HTTPError(request.full_url, 403, "private message", {"X-RateLimit-Remaining": "0", "X-RateLimit-Reset": "120"}, None)
|
||||
return io.BytesIO(b'{"ok": true}')
|
||||
|
||||
transport = s.JSONTransport(opener=opener, sleeper=waits.append, clock=lambda: 100)
|
||||
assert transport.get("https://example.invalid", {}) == {"ok": True}
|
||||
assert waits == [60] and transport.attempts == 2
|
||||
|
||||
def down(request, timeout):
|
||||
raise HTTPError(request.full_url, 503, "private name", {}, None)
|
||||
|
||||
transport = s.JSONTransport(opener=down, sleeper=waits.append)
|
||||
with pytest.raises(ValueError, match="^http_failure$"):
|
||||
transport.get("https://example.invalid", {})
|
||||
assert transport.attempts == 5
|
||||
Reference in New Issue
Block a user