mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 19:59:26 +00:00
Compare commits
32
Commits
@@ -0,0 +1,69 @@
|
||||
name: Engine image
|
||||
on:
|
||||
workflow_dispatch:
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
- 'src/**'
|
||||
- 'benchmarks/learn/engine_trace*.py'
|
||||
- 'benchmarks/learn/engine_task_entry.py'
|
||||
- 'benchmarks/unseen/replay_entry.py'
|
||||
- 'benchmarks/learn/engine_worker_service.py'
|
||||
- 'Dockerfile'
|
||||
- 'plane/Dockerfile.engine-task'
|
||||
- 'ops/frontend-retention/install.py'
|
||||
- 'pyproject.toml'
|
||||
- '.github/workflows/engine-image.yml'
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
concurrency:
|
||||
group: engine-image-${{ github.ref }}
|
||||
cancel-in-progress: false
|
||||
jobs:
|
||||
image:
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- package: openultrasast
|
||||
dockerfile: Dockerfile
|
||||
- package: ousast-engine-task
|
||||
dockerfile: plane/Dockerfile.engine-task
|
||||
target: engine
|
||||
- package: ousast-replay-task
|
||||
dockerfile: plane/Dockerfile.engine-task
|
||||
target: replay
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
- uses: docker/setup-buildx-action@v3
|
||||
- uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
- id: names
|
||||
env:
|
||||
OWNER: ${{ github.repository_owner }}
|
||||
PACKAGE: ${{ matrix.package }}
|
||||
run: |
|
||||
echo "image=ghcr.io/${OWNER,,}/$PACKAGE" >> "$GITHUB_OUTPUT"
|
||||
echo "sha=sha-${GITHUB_SHA:0:7}" >> "$GITHUB_OUTPUT"
|
||||
- id: build
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
file: ${{ matrix.dockerfile }}
|
||||
target: ${{ matrix.target }}
|
||||
platforms: linux/amd64
|
||||
push: true
|
||||
tags: |
|
||||
${{ steps.names.outputs.image }}:${{ steps.names.outputs.sha }}
|
||||
${{ steps.names.outputs.image }}:main
|
||||
- env:
|
||||
IMAGE: ${{ steps.names.outputs.image }}
|
||||
DIGEST: ${{ steps.build.outputs.digest }}
|
||||
run: |
|
||||
echo "$IMAGE@$DIGEST"
|
||||
echo "$IMAGE@$DIGEST" >> "$GITHUB_STEP_SUMMARY"
|
||||
@@ -36,3 +36,6 @@ workspace/
|
||||
|
||||
# MkDocs build output (mkdocs build)
|
||||
site/
|
||||
|
||||
# Unseen evaluation pointers for non-permissive repositories.
|
||||
benchmarks/unseen/private/
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
# Plan: learn fix mechanisms into the engine (draft for review, 2026-10-04)
|
||||
|
||||
Status: approved 2026-10-04 by the maintainer. Increment 1 = discharge facts only; coverage additions (Req 7) come later.
|
||||
Decisions: minimum 2 proposing repositories before a candidate is tried; admission per language (not pooled by callee name).
|
||||
|
||||
## Why
|
||||
|
||||
The Joern trace run over exp-005's 501 units shows the engine cannot tell a vulnerable function from its
|
||||
fix. In the 105 pairs where both sides were analysed, a taint path appears on both sides in 12, on the
|
||||
vulnerable side only in 2, on the fixed side only in 0, and on neither in 91
|
||||
(`benchmarks/measurements/2026-10-04-joern-slice-leak-audit/record.json`).
|
||||
|
||||
So the fix is invisible to the engine. Either the flow survives and the engine does not know the fix's
|
||||
mechanism (an `allowed_classes` argument, a path normalisation check, an escaping call), or the engine's
|
||||
sources and sinks miss the function on both sides. A classifier fed this evidence cannot improve on it.
|
||||
|
||||
The corpus now holds 710 complete fix pairs (`2026-10-04-decision-engine-harvest-3/record.json`). Each fix
|
||||
diff shows what was added. That is the material the engine lacks.
|
||||
|
||||
## Goal
|
||||
|
||||
Grow the engine's discharge knowledge (sanitizers, guards, safe-argument forms) and, second, its source and
|
||||
sink coverage, from the fix diffs. Admit a fact only when it improves held-out separation. Spend no model
|
||||
money.
|
||||
|
||||
## Requirements (proposed)
|
||||
|
||||
1. **Mining.** From each fix pair, extract what the fixed function adds on the vulnerable flow: new calls,
|
||||
new conditions guarding the sink, new or changed sink arguments. Record each as a candidate fact with its
|
||||
language, family, form (sanitizer call, guard predicate, safe argument, sink restriction) and the pairs
|
||||
that propose it. Mining is deterministic (tree-sitter diff of the two function bodies). No model calls.
|
||||
2. **Generalisation is explicit.** A candidate is a pattern, not a repository name: a callee name, an
|
||||
argument key, a predicate shape. Candidates naming project-specific identifiers are rejected (Req 8: no
|
||||
application-specific trees).
|
||||
3. **Admission by held-out separation.** Folds are by repository, as in the decision engine. A candidate is
|
||||
admitted for a fold only if, on that fold's held-out pairs, adding it to the fact tables:
|
||||
- increases pairs where the vulnerable side keeps a path and the fixed side loses it or becomes
|
||||
sanitized;
|
||||
- and does not remove any held-out vulnerable-side path (no recall loss).
|
||||
A candidate proposed by fewer than 2 distinct repositories in the training folds is not tried.
|
||||
4. **No train-on-test.** Mining and admission never read held-out pairs; the closed-loop leak recorded in
|
||||
memory (`closed-loop-train-on-test-leak`) is the failure this rule exists to prevent. Population v3 is
|
||||
never read.
|
||||
5. **Measurement by the trace runner.** Each admission round re-runs the engine on the affected pins only
|
||||
(the runner's resume and selection already support this; on k3s the worker Deployment runs them in
|
||||
parallel). The outcome is the pair-asymmetry table per family, before and after, on held-out folds.
|
||||
6. **Gate before production.** An admitted fact enters `src/openultrasast/ruleset/semantic/*.toml` only after
|
||||
the detection gates (`gate`, `map_gate`, `pair_gate`) show no regression, and only as a registered
|
||||
experiment whose decision rule was fixed before the run.
|
||||
7. **Coverage second.** After discharge facts, the same loop proposes source and sink additions from pairs
|
||||
where neither side has a path (91 of 105 today), admitted under the same held-out rule.
|
||||
|
||||
## How it relates to what exists
|
||||
|
||||
- `corpus-seeded-mechanisms` already has a closed loop that admits and retracts mechanisms by leave-one-out.
|
||||
This plan reuses its lever and store, and points it at discharge facts mined from real fix diffs instead
|
||||
of variant search.
|
||||
- `benchmarks/pairs/mechanisms.toml` (23 mechanisms) labels what kind of bug a pair is. The new candidates
|
||||
describe what a fix does. They live in the fact tables, not in the mechanism vocabulary.
|
||||
|
||||
## Measures of success
|
||||
|
||||
- Pair asymmetry on held-out folds: vulnerable-only paths rise from 2 of 105 toward a substantial share,
|
||||
with zero fixed-only paths.
|
||||
- No held-out recall loss on vulnerable sides.
|
||||
- Then the decision engine is re-run with the new engine evidence (exp-005 design), and the pooled block gate
|
||||
(exp-004 design) is re-measured.
|
||||
|
||||
## Cost and time
|
||||
|
||||
- Model cost: $0 (mining and admission are deterministic; the classifier re-run afterwards is a separate,
|
||||
approved spend).
|
||||
- Engine time: one pass over affected pins per admission round; about 2 hours per full pass on k3s plus the
|
||||
VM, about 6 hours on the VM alone.
|
||||
|
||||
## Open questions for the maintainer
|
||||
|
||||
1. Should coverage additions (Req 7) be in the same increment or a later one?
|
||||
2. Minimum proposing repositories: 2 is the proposal. Higher is safer, lower finds more.
|
||||
3. Admission per family and language, or pooled across languages for the same callee name?
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"feature_name": "learned-decision-engine",
|
||||
"created_at": "2026-09-30T14:00:00Z",
|
||||
"updated_at": "2026-10-01T10:20:00Z",
|
||||
"updated_at": "2026-10-04T18:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"approvals": {
|
||||
|
||||
@@ -8,6 +8,10 @@ memory, compiled DSPy-style in-house); tasks 1-5 are unaffected, tasks 6-10 are
|
||||
against the provider balance read just before it, and none starts with a ceiling above that balance. No test makes
|
||||
a model or embedding call.
|
||||
|
||||
Direction 2026-10-04: `.kiro/steering/roadmap.md` (section of 2026-10-04, phases G0-G5) is the authority. Every
|
||||
change is judged on the unseen-repository pool (G0, spec `unseen-repo-evaluation`) by catch rate and false-alarm
|
||||
rate per push. Within-pair AUC stays a development signal. Tasks 11-14 below follow that roadmap.
|
||||
|
||||
- [x] 1. Schema and features
|
||||
- `learn/schema.py` (closed, typed allow-list; no identities; missing != zero), `learn/features.py` (host
|
||||
builder), new memory `KINDS`, `plane/tasks/features.py` and the `remember` input; tests enforcing the
|
||||
@@ -45,6 +49,11 @@ a model or embedding call.
|
||||
- _Requirements: 1.1_
|
||||
|
||||
- [ ] 6. Program, local memory and compilation (design sections 4-5)
|
||||
- Status 2026-10-04: 6.1-6.6 are done. 6.7 is partial: leave-one-source-out, the 0/25/50% memory-size points and
|
||||
the source-count curve are not run. Around this task the experiment machinery grew: rule arms
|
||||
(`precision_bound_block`, `pooled_block_gate` with a family floor) in `learn/experiments.py`, the response-cache
|
||||
fill with a hard ceiling and a dry-run estimate in `learn/fill.py`, and the in-function slice in `learn/slice.py`.
|
||||
These ran exp-003, exp-004 and exp-005 (7.5-7.7). Memory now holds 710 complete fix pairs (7.9).
|
||||
- [x] 6.1 Excerpts and the example store (no model)
|
||||
- `learn/excerpt.py` (function span at the label's pin, bounds 80/4,000 for candidates and 40/2,000 for
|
||||
examples, numbered lines, no path/repository/commit, advisory ids and security wording in comments
|
||||
@@ -151,6 +160,28 @@ a model or embedding call.
|
||||
rows' null plane-only features (1f525ff), the 788 embedding vectors absent from RustFS (copied from the file
|
||||
store), the result row's column-type collision under S3 Select (780caaa). Further program variants (`k = 5` vs
|
||||
`k = 1`, the confidence source) as new manifests.
|
||||
- [x] 7.5 exp-003 run and recorded 2026-10-03 ($0, replay only): **inconclusive**. Selective blocking (a per-family
|
||||
raw-score threshold chosen on training folds by its Wilson lower bound) offers no BLOCK on held-out paired units
|
||||
in any family; the incumbent stays. 211 of 353 frozen units evaluated (142 replay misses). Proving 0.95 needs at
|
||||
least 73 clean flags. Result: `benchmarks/experiments/exp-003-precision-bound-blocking/result.json`.
|
||||
- [x] 7.6 exp-004 run and recorded 2026-10-03: **inconclusive**. One pooled BLOCK threshold over all families with a
|
||||
family floor (Req 6.4). No outer fold found a threshold, pooled or per family; B offers no BLOCK. 501 units,
|
||||
replay at $0 after a cache fill of $1.24 (ceiling $2; the dry-run estimate over-priced 4.5x). Result and record:
|
||||
`benchmarks/experiments/exp-004-pooled-block-gate/result.json`, `record.json`.
|
||||
- [x] 7.7 exp-005 run and recorded 2026-10-03: **inconclusive**. The in-function def-use slice in the prompt of the
|
||||
six slice-2 programs. Within-pair AUC 0.762 -> 0.746 (CI [-0.063, 0.032], spans 0); unsure rate 0.29 -> 0.24
|
||||
(favours B); usd per candidate slightly higher. A $0 leak audit came first
|
||||
(`benchmarks/measurements/2026-10-03-graph-slice-leak-audit/record.json`). Result:
|
||||
`benchmarks/experiments/exp-005-graph-slice/result.json`.
|
||||
- [x] 7.8 Negatives harvest 2026-10-03 ($0.0014 embeddings, no chat-model call): 98 real fix pairs whose function
|
||||
survives the fix, from reviewed advisories, as pointer-pair corpus `advisory-fixes-2`; memory 937 -> 1133
|
||||
examples. Record: `benchmarks/measurements/2026-10-03-decision-engine-negatives-harvest/record.json`.
|
||||
- [x] 7.9 Harvest 3, 2026-10-04 ($0.003 embeddings, no chat-model call): 251 pairs as `advisory-fixes-3`; complete
|
||||
pairs in memory 459 -> 710. Record: `benchmarks/measurements/2026-10-04-decision-engine-harvest-3/record.json`.
|
||||
- [x] 7.10 Joern trace run over exp-005's 501 units and its leak audit, 2026-10-04 ($0): no leak flagged. In the 105
|
||||
pairs where both sides were asked, a path appears on both sides in 12, on the vulnerable side only in 2, on the
|
||||
fixed side only in 0, and on neither in 91. The engine cannot see what a fix changed. Record:
|
||||
`benchmarks/measurements/2026-10-04-joern-slice-leak-audit/record.json`.
|
||||
- _Requirements: 3.6, 4.1-4.4_
|
||||
|
||||
- [ ] 8. Integration
|
||||
@@ -176,3 +207,50 @@ a model or embedding call.
|
||||
`result-v3-engine.json` is committed. Nothing between the two commits may touch the program, memory snapshot,
|
||||
features, priors or thresholds. This spec's code never opens v3 files.
|
||||
- _Requirements: 5.3_
|
||||
|
||||
- [ ] 11. Fix mechanisms (plan approved 2026-10-04; see `fix-mechanisms-plan.md`)
|
||||
- Roadmap G2 arm (`.kiro/steering/roadmap.md`, section of 2026-10-04): one candidate improvement, not the spine.
|
||||
It is judged on the unseen-repository pool (G0, spec `unseen-repo-evaluation`) by catch rate and false-alarm
|
||||
rate per push. **Paused after 11.1 until the G0 pool exists.**
|
||||
- [ ] 11.1 Mine candidate discharge facts from the 710 fix diffs (sanitizer calls, guarding conditions, safe sink
|
||||
arguments, safe replacements); deterministic, $0. In progress.
|
||||
- [ ] 11.2 Admission by held-out separation per outer fold: a candidate turns "path on both sides" into "path on the
|
||||
vulnerable side only" on folds it was not mined from, removes no held-out vulnerable path, has at least 2
|
||||
proposing repositories, and is admitted per language.
|
||||
- [ ] 11.3 Engine re-run on the affected pins (kube-ax lanes plus the VM lane).
|
||||
- [ ] 11.4 Registered experiment, with the detection gates green, before `ruleset/semantic/*.toml` changes.
|
||||
- [ ] 11.5 Coverage increment: source and sink additions for pairs where neither side has a path (later).
|
||||
- [ ] 11.6 TypeScript admission in the engine (branch `engine-typescript`).
|
||||
- _Requirements: 4.1-4.4, 7.1, 8.1_
|
||||
|
||||
- [ ] 12. exp-006: Joern witness slice in the prompt **(budget go-ahead, about $2-4)**
|
||||
- Planned, roadmap G1/G2. Against the current prompt, after a $0 leak audit, on re-frozen units. Judged on the
|
||||
unseen-repository pool, not on within-pair AUC alone.
|
||||
- _Requirements: 4.1-4.4, 7.1_
|
||||
|
||||
- [ ] 13. exp-007: the combiner ($0 replay)
|
||||
- Planned, roadmap G1/G2. Three or four weights (score, engine entailment, neighbour agreement) over cached
|
||||
responses. Judged on the unseen-repository pool.
|
||||
- _Requirements: 4.1-4.4, 7.1_
|
||||
|
||||
- [ ] 14. Re-run the pooled block gate on re-frozen units
|
||||
- Planned, roadmap G3. The exp-004 design with the family floor (Req 6.4), together with the pool's false-alarm
|
||||
rate per family.
|
||||
- _Requirements: 6.4, 7.3_
|
||||
|
||||
- [ ] 15. Trained classifier arm (maintainer decision 2026-10-04: train a model, not hand-set profiles; judged on the
|
||||
unseen-repo pool, roadmap G1)
|
||||
- [ ] 15.1 Features: code embedding of the excerpt (stored vectors) + every allow-listed signal; no hand-picked
|
||||
profile. Automated screening: a signal that separates pair sides by itself on training folds above the leak
|
||||
threshold is dropped for that fold (the leak audit as a filter, fitted per fold).
|
||||
- [ ] 15.2 Models: regularised logistic regression and gradient-boosted trees, fitted per repository-held-out outer
|
||||
fold with inner-fold hyper-parameter search; per-family and pooled variants.
|
||||
- [ ] 15.3 Scores on held-out folds: within-pair AUC, pooled AUC, top-k precision, Wilson bound at the pooled
|
||||
threshold; compared with the current LLM programs on the same units.
|
||||
- [ ] 15.4 Scored on the unseen-repo pool when G0 exists: catch rate at fixed false-alarm budgets.
|
||||
- _Requirements: 3 (revised 2026-10-04 by maintainer decision), 6, 7_
|
||||
|
||||
- [ ] 16. Jointly optimised LLM program arm (judged on the unseen-repo pool)
|
||||
- Search input fields, retrieval parameters, k, instruction and demonstrations together (DSPy MIPRO-style) on
|
||||
training folds; no hand-set input profile. Budget go-ahead before paid search.
|
||||
- _Requirements: 3, 6, 7_
|
||||
|
||||
@@ -1,5 +1,20 @@
|
||||
# Design Document
|
||||
|
||||
> **Status 2026-10-04.** Groups 1-2 are done on kind (records `benchmarks/measurements/2026-10-02-k8s-01-profile/`
|
||||
> and `2026-10-02-k8s-02-store-delivery/`). Task 2.6 is paused. The production target changed. The operator runs a
|
||||
> kube-ax cluster (k3s, Agent Substrate v0.3.0, AX v0.3.1, 2 nodes). There we may only submit AX Tasks: no k8s
|
||||
> object creation, no secrets, no access to `ate-system`. The engine now runs there as AX Tasks through
|
||||
> `benchmarks/learn/engine_trace_ax.py`. The engine-task image (`plane/Dockerfile.engine-task`) is built FROM
|
||||
> `ghcr.io/norandom/ax-task-runner:v0.3.1`. Tasks read and write through presigned HTTPS links, are named
|
||||
> `ousast-engine-*`, and run a 1.7 GB heap with `ActiveProcessorCount=2` and one resident Joern session per pin. A
|
||||
> pin whose sandbox dies is retried once under a new name, then handed to the VM lane (2.5 GB heap). Egress is
|
||||
> operator-managed: HTTPS 443 to public hosts is open; LAN and cluster-internal addresses are blocked. Archive
|
||||
> names (`.tar`, `.gz`, `.zip`, `.bin`) failed through the store host's Cloudflare cache until the operator added a
|
||||
> cache-bypass rule; presigned objects are staged as `.dat` as a safeguard. Groups 3-8 are **to re-plan** against
|
||||
> that cluster under roadmap G4 (`.kiro/steering/roadmap.md`, section of 2026-10-04). Their text below is kept
|
||||
> as written. G4 (detection on push for any repository: `ousast plane scan --base --head`, engine pins on kube-ax,
|
||||
> model calls on the VM or CI) is the new target for group 7, sized by what G1-G3 keep.
|
||||
|
||||
## Overview
|
||||
|
||||
The plane keeps its shape: `ousast plane run` is the reconciler, ax is the only executor, every task is one
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"feature_name": "plane-on-kubernetes",
|
||||
"created_at": "2026-10-02T20:00:00Z",
|
||||
"updated_at": "2026-10-02T22:00:00Z",
|
||||
"updated_at": "2026-10-04T18:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"approvals": {
|
||||
|
||||
@@ -1,5 +1,20 @@
|
||||
# Implementation Plan
|
||||
|
||||
> **Status 2026-10-04.** Groups 1-2 are done on kind (records `benchmarks/measurements/2026-10-02-k8s-01-profile/`
|
||||
> and `2026-10-02-k8s-02-store-delivery/`). Task 2.6 is paused. The production target changed. The operator runs a
|
||||
> kube-ax cluster (k3s, Agent Substrate v0.3.0, AX v0.3.1, 2 nodes). There we may only submit AX Tasks: no k8s
|
||||
> object creation, no secrets, no access to `ate-system`. The engine now runs there as AX Tasks through
|
||||
> `benchmarks/learn/engine_trace_ax.py`. The engine-task image (`plane/Dockerfile.engine-task`) is built FROM
|
||||
> `ghcr.io/norandom/ax-task-runner:v0.3.1`. Tasks read and write through presigned HTTPS links, are named
|
||||
> `ousast-engine-*`, and run a 1.7 GB heap with `ActiveProcessorCount=2` and one resident Joern session per pin. A
|
||||
> pin whose sandbox dies is retried once under a new name, then handed to the VM lane (2.5 GB heap). Egress is
|
||||
> operator-managed: HTTPS 443 to public hosts is open; LAN and cluster-internal addresses are blocked. Archive
|
||||
> names (`.tar`, `.gz`, `.zip`, `.bin`) failed through the store host's Cloudflare cache until the operator added a
|
||||
> cache-bypass rule; presigned objects are staged as `.dat` as a safeguard. Groups 3-8 are **to re-plan** against
|
||||
> that cluster under roadmap G4 (`.kiro/steering/roadmap.md`, section of 2026-10-04). Their text below is kept
|
||||
> as written. G4 (detection on push for any repository: `ousast plane scan --base --head`, engine pins on kube-ax,
|
||||
> model calls on the VM or CI) is the new target for group 7, sized by what G1-G3 keep.
|
||||
|
||||
One task group per increment of `design.md` section 7, in that order; a later group never starts before the
|
||||
earlier group's kind proof is recorded. Every task ships with its tests; a task is done when its tests pass in
|
||||
the host suite, ruff and mypy are clean, and nothing in `src/`, `plane/`, `ops/` (minus `ops/ax/up.sh` and
|
||||
@@ -9,7 +24,7 @@ its kind proof and the record it writes under `benchmarks/measurements/<date>-k8
|
||||
(commit, images, the instrument checks, the numbers). Steps that call a model are marked **paid: needs a budget
|
||||
go-ahead** with a ceiling; steps that need a hand on a cluster that does not exist yet are marked **manual**.
|
||||
|
||||
- [ ] 1. Profile and literals (design section 1; open-question checks 2, 3, 7)
|
||||
- [x] 1. Profile and literals (design section 1; open-question checks 2, 3, 7)
|
||||
- [x] 1.1 `PlaneProfile`
|
||||
- `src/openultrasast/plane/profile.py`: frozen dataclass with the fields of design section 1; `load_profile()`
|
||||
reads `ops/k8s/profiles/<OUSAST_PLANE_PROFILE>.toml` (a name or a path; default `kind`), then applies
|
||||
@@ -107,7 +122,7 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
|
||||
prints the table. Instrument: the tar's byte size from the store equals the extracted files' total.
|
||||
- `benchmarks/measurements/<date>-k8s-02-store-delivery/record.json`: bytes, PUT-to-poll seconds, rules.
|
||||
- _Requirements: 3.1, 3.2, 7.1_
|
||||
- [ ] 2.6 Remove the receiver (last; after 2.5 is recorded)
|
||||
- [ ] 2.6 Remove the receiver (last; after 2.5 is recorded) -- paused 2026-10-04
|
||||
- Delete `Receiver`, `_Handler`, `receiver_address`, `receiver_cluster_ip`, `RECEIVER_HOST`, `RECEIVER_PORT`
|
||||
from `egress.py`; `OUSAST_ARTIFACT_URL/DIAL/HOST/PORT` from `runner.py` and `reconciler.py`; the
|
||||
`OUSAST_DELIVERY` switch; `ops/ax/receiver-service.yaml.tmpl` and the `up.sh` receiver step; the EndpointSlice
|
||||
@@ -117,7 +132,7 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
|
||||
`receiver_removed: <commit>`.
|
||||
- _Requirements: 3.1, 1.3_
|
||||
|
||||
- [ ] 3. `--exec remote` from a CI-shaped principal (design section 2; open-question check 1)
|
||||
- [ ] 3. `--exec remote` from a CI-shaped principal (design section 2; open-question check 1) -- to re-plan (kube-ax, roadmap G4)
|
||||
- [ ] 3.1 Execution mode and RBAC manifests
|
||||
- `cli.py`: `ousast plane run --exec local|remote` (default `profile.exec`); `reconciler.Ax` passes the
|
||||
profile's context to `ax` (its context flag, read from `ax --help` and pinned in a test); `remote` refuses a
|
||||
@@ -150,7 +165,7 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
|
||||
status, usd spent.
|
||||
- _Requirements: 2.1, 2.2, 2.3, 2.4, 7.1_
|
||||
|
||||
- [ ] 4. Images on GHCR (design section 8)
|
||||
- [ ] 4. Images on GHCR (design section 8) -- to re-plan (kube-ax, roadmap G4)
|
||||
- [ ] 4.1 `images.yml`
|
||||
- `.github/workflows/images.yml` on tags `v*`: `docker/build-push-action` for `plane/Dockerfile.runner` ->
|
||||
`ghcr.io/norandom/ousast-runner:<tag>` and the root `Dockerfile` -> `ghcr.io/norandom/ousast-engine:<tag>`;
|
||||
@@ -178,7 +193,7 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
|
||||
- `benchmarks/measurements/<date>-k8s-04-ghcr-images/record.json`: image sizes, pull seconds, visibility.
|
||||
- _Requirements: 1.1, 4.2, 7.1_
|
||||
|
||||
- [ ] 5. The engine as a plane task (design section 4)
|
||||
- [ ] 5. The engine as a plane task (design section 4) -- to re-plan (kube-ax, roadmap G4)
|
||||
- [ ] 5.1 `cpg/dump.py` and `tasks/engine.py`
|
||||
- Lift `benchmarks/push/finding_dump.py`'s scan into `src/openultrasast/cpg/dump.py` (the script becomes a
|
||||
thin caller; `ops/README.md:78` command unchanged). `src/openultrasast/plane/tasks/engine.py`: env
|
||||
@@ -208,7 +223,7 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
|
||||
placement, image size and pull seconds.
|
||||
- _Requirements: 4.1, 4.2, 7.1_
|
||||
|
||||
- [ ] 6. Pools, autoscaling, node checklist (design section 5)
|
||||
- [ ] 6. Pools, autoscaling, node checklist (design section 5) -- to re-plan (kube-ax, roadmap G4)
|
||||
- [ ] 6.1 `plane/k8s.py` and the manifests
|
||||
- `src/openultrasast/plane/k8s.py`: `render(profile, out)` writes `workerpool-default.yaml`,
|
||||
`workerpool-engine.yaml`, `ax-server-snapshots.yaml` (patch with `profile.snapshots_bucket`), `rbac-ci.yaml`;
|
||||
@@ -232,7 +247,7 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
|
||||
- `benchmarks/measurements/<date>-k8s-06-pools/record.json`: footprint table, securityContext, dry-run result.
|
||||
- _Requirements: 5.1, 5.2, 7.1_
|
||||
|
||||
- [ ] 7. `ousast plane scan` (design section 6)
|
||||
- [ ] 7. `ousast plane scan` (design section 6) -- to re-plan (kube-ax, roadmap G4)
|
||||
- [ ] 7.1 Workspaces and candidates from URL + commits
|
||||
- `generate.scan_cases(url, base, head, cache, *, all=False)`: shallow clone into the case cache, two pinned
|
||||
Workspaces, head-side hunks via `fix_ranges(side="new")`, functions via `repo_facts.enclosing` over
|
||||
@@ -265,7 +280,7 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
|
||||
seconds from submit to verdict, whether the known CVE site is among the agreed.
|
||||
- _Requirements: 6.1, 6.2, 6.3, 7.1_
|
||||
|
||||
- [ ] 8. Runbook (design section 7, Req 7)
|
||||
- [ ] 8. Runbook (design section 7, Req 7) -- to re-plan (kube-ax, roadmap G4)
|
||||
- [ ] 8.1 `docs/deployment.md` as the runbook
|
||||
- Sections: prerequisites for k3s (Substrate, ax with the snapshots patch, GHCR and the pull secret, buckets,
|
||||
the CI SA and its grant from 3.2, the node checklist of 6.2), the profile switch, `--exec remote` from a laptop
|
||||
|
||||
@@ -623,6 +623,7 @@ Preserve all completed task evidence below.
|
||||
- [x] 17. Operational readiness (2026-10-02)
|
||||
- **Named by the 2026-10-02 operational-readiness audit**, maintainer-approved as the first increment. The maintainer's definition: a user installs the hook on ANY repository and gets honest, advisory-only results; nothing ever blocks a push. The audit found the plumbing sound (snapshots, resolution, deadline accounting, advisory exit 0, the JSON artifact) and five things in the way: the hook can block a push, quick rules are not part of pre-push, the engine cannot finish a 26-file repository inside 30 s, the terminal hides every skip reason, and there is no supported install path without the source checkout. Before numbers for every item are the audit's live runs on records (Python), serve-static (JavaScript) and unrolled/secure (Go), each with one planted commit.
|
||||
- This group is plumbing and honesty. It enables no capability, edits no rule for a planted sample, and leaves graph reuse (the engine speed work decided in 11.6) as a later task.
|
||||
- Record: `benchmarks/measurements/2026-10-02-pre-push-operational-1/record.json` (see 17.7).
|
||||
- _Done 2026-10-02:_ all seven subtasks done. The engine still does not fit the 30 s hook deadline (graph reuse, the 11.6 direction, stays a later task), and plane verdicts after the push were not in this increment.
|
||||
- _Requirements: 9.1–9.6_
|
||||
- [x] 17.1 Never block a push the user did not choose to block
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
# Brief: does the safety net work on repositories it has never seen?
|
||||
|
||||
Maintainer, 2026-10-04: "we want to spread the safety net across repos in general, unseen code. that was the purpose
|
||||
of the AB tests to see if our approach is generalizeable." And: "we should run our tests / runs in kube. now we can
|
||||
speed it up and design the workloads for ax directly. kind is only a fallback."
|
||||
|
||||
## Why this spec exists
|
||||
|
||||
The A/B experiments so far (exp-003, exp-004, exp-005) score the decision engine on repository-held-out folds of known
|
||||
fix pairs. That measures whether the vulnerable side of a pair scores above its fixed side. It does not measure what a
|
||||
safety net on an arbitrary repository lives on:
|
||||
|
||||
- on an ordinary push, does it stay quiet? (false-alarm rate)
|
||||
- on a push that introduces a real vulnerability, does it say so? (catch rate)
|
||||
|
||||
Populations v1 and v2 measured the engine scan on fresh cases once (1 of 11, 0 of 17) and are spent. Nothing since has
|
||||
measured push-level behaviour on unseen repositories.
|
||||
|
||||
## What this spec delivers
|
||||
|
||||
1. A frozen pool of repositories never used anywhere in development, with two kinds of change each: real
|
||||
vulnerability-introducing commits and ordinary commits.
|
||||
2. A push-level metric: each change replayed through the hook exactly as a user would run it, and through other
|
||||
arms (decision engine, engine evidence) where an experiment asks.
|
||||
3. Execution designed for the kube-ax cluster from the start: one push replay is one AX Task; kind and the VM are
|
||||
fallbacks.
|
||||
4. A baseline for today's hook, then a table of arms scored on the same pool.
|
||||
|
||||
## Not in scope
|
||||
|
||||
Changing detection. This spec only builds the measurement. Improvements are separate, registered experiments judged
|
||||
by it.
|
||||
@@ -0,0 +1,431 @@
|
||||
# Design Document
|
||||
|
||||
## Overview
|
||||
|
||||
This spec builds the instrument for roadmap G0: a frozen pool of repositories that no development step has seen,
|
||||
and a push-level score on it. The score is what a user of the hook sees. On a vulnerability-introducing push, did
|
||||
the hook say so (catch rate)? On an ordinary push, did it say anything (false-alarm rate)? Replays run as AX Tasks
|
||||
on kube-ax. kind and the VM run the same tasks as fallbacks.
|
||||
|
||||
Four facts from the code shape the design:
|
||||
|
||||
- **Populations already have a freeze-and-guard pattern.** `benchmarks/independent/freeze.py` verifies cases by
|
||||
reading git only, `evaluate.py` scores with fixed rules and Wilson intervals, and
|
||||
`tests/test_independent_population.py` fails when a reserved repository is named anywhere in `benchmarks/`,
|
||||
`src/` or `tests/` outside its directory. Harvest batch 3 used that guard as a black box: candidates written,
|
||||
guard run with output discarded, subsets bisected on red. That is how v3 stays unread.
|
||||
- **The hook can already replay a change.** `ousast pre-push PATH --base B --head H --artifact F` resolves two
|
||||
local commits (`snapshot.py`, `resolve_replay`: no fetch, no merge-base), runs quick rules and the engine inside
|
||||
one deadline (default 30 s, engine inline) and writes a JSON artifact. The terminal text has one `Skipped:` line
|
||||
per skipped check. The artifact holds three finding streams: admitted defects, advisory engine findings
|
||||
(new or worsened, vetoed only by capability) and quick-rule matches with their CWE.
|
||||
- **The AX pattern works but lives in one script.** `engine_trace_ax.py` (manifest, egress checks, presigned
|
||||
objects, retry once under a new name, cleanup) and the lane scheduler in `engine_trace.py` (`run_plan`: fixed
|
||||
lanes, fallback queue, `progress.json`, `STOP`, resume by input digest) are engine-specific.
|
||||
- **Placement on kube-ax is random.** One worker per node, shared by all actors. Two heavy tasks fit per worker,
|
||||
but only two concurrent tasks in all guarantee at most two on one worker.
|
||||
|
||||
### Goals
|
||||
|
||||
- A pool frozen by code against every used source (Req 1), with two kinds of change per repository (Req 2).
|
||||
- A scorer whose numbers mean what the hook shows, with silent zeros visible (Req 3).
|
||||
- One replay is one AX Task via a dispatcher shared with the engine (Req 4); a baseline and fresh slices (Req 5).
|
||||
|
||||
### Non-Goals
|
||||
|
||||
Detection changes, population v3, model calls in the cluster, languages the hook does not analyse.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
M[pool manifest slice] --> D[batch dispatcher on VM]
|
||||
D --> P[presign spec GET and record PUT]
|
||||
P --> T[AX Task ousast-engine-replay]
|
||||
T --> G[shallow fetch base and head from GitHub]
|
||||
G --> H[ousast pre-push --base --head]
|
||||
H --> R[record tar: artifact, text, instrument]
|
||||
R --> S[(store via presigned PUT)]
|
||||
D -->|sandbox died twice or OOM| V[VM docker lane]
|
||||
D -->|cluster down| K[kind lane, same tasks]
|
||||
V --> S
|
||||
K --> S
|
||||
S --> C[scorer: catch, false alarm, coverage]
|
||||
S --> A[model arms on VM, budgeted]
|
||||
A --> C
|
||||
C --> O[record: counts and digests]
|
||||
```
|
||||
|
||||
## Components and Interfaces
|
||||
|
||||
### 1. Pool sourcing and eligibility (Req 1)
|
||||
|
||||
**Decision 1.1: sources and ecosystems.** Candidates come from GitHub reviewed advisories (`gh api /advisories`,
|
||||
the enumeration harvest-3 used) for pip, npm, maven and composer, published 2019 or later, whose CWE maps to one of
|
||||
the seven families through the project's family CWE table. Each needs a linked fix commit with one parent in one
|
||||
repository. OSV records for the same advisory give the first affected version. That version is a cross-check for
|
||||
the introducing commit (section 2). Reason: these four ecosystems are the five languages the hook analyses
|
||||
(Python, JavaScript, TypeScript, PHP, Java). GHSA links commits and CWEs; OSV adds version brackets. Rejected:
|
||||
drawing from every ecosystem (Go, Ruby, C#). That would mostly measure missing frontends, which is already known.
|
||||
Coverage still reports uncovered files inside the drawn repositories. Rejected: NVD and published fix datasets.
|
||||
NVD links commits weakly, and the datasets are mined corpora that development or model training has likely seen.
|
||||
|
||||
**Decision 1.2: eligibility by code against every used source.** `benchmarks/unseen/eligibility.py` builds the
|
||||
used set before the draw. It never prints names; it records counts per source and a digest of the set.
|
||||
|
||||
| Source (Req 1.1) | Where it is read |
|
||||
| --- | --- |
|
||||
| Pair corpora | every catalog, recipe and dataset file under `benchmarks/pairs/`, plus the local pointer-pair manifests |
|
||||
| Populations v1, v2 | `population-v1.toml`, `population-v2.toml`, read directly |
|
||||
| Population v3 | never opened; the existing guard test only (below) |
|
||||
| Harvest records | the batch catalogs under `benchmarks/pairs/advisory-fixes*` (the records hold counts only) |
|
||||
| Memory examples | `example` rows of the local FileStore and the S3 store (`repo` field) |
|
||||
| Experiment units | `plane/experiments/*.units.jsonl` and registered units files |
|
||||
| Measurement records | everything under `benchmarks/measurements/` and `benchmarks/experiments/` |
|
||||
| Anything else | a text sweep for `github.com/<owner>/<name>` over `benchmarks/`, `src/`, `tests/`, `plane/`, `.kiro/` |
|
||||
|
||||
Names are normalised (lower case, no `.git`, no trailing slash). Each candidate is resolved through the GitHub API
|
||||
to its canonical `full_name`, its former names (redirects) and its fork network (`parent`, `source`). A candidate
|
||||
is out if any of these is in the used set. A fork shares code with what it was forked from.
|
||||
|
||||
v3 is checked like harvest-3 did. The draft manifest is written into `benchmarks/unseen/`. Then
|
||||
`pytest -q tests/test_independent_population.py -k referenced` runs with its output discarded. The guard already
|
||||
sweeps `benchmarks/` outside the population directory. On red, the candidate list is bisected by rewriting the
|
||||
draft until green, and the dropped count is recorded. No v3 name is ever printed or read by a person.
|
||||
|
||||
**Decision 1.3: post-cutoff preference.** An advisory published after 2026-06-01 (harvest-3's cutoff for the
|
||||
decision engine's model) is flagged `post_cutoff`. Within each ecosystem stratum post-cutoff advisories are drawn
|
||||
first. Results are reported for both groups. Reason: a post-cutoff advisory cannot be in the model's training
|
||||
data as a known vulnerability. Rejected: post-cutoff only. Harvest-3 kept 12 post-cutoff pairs out of 251, and
|
||||
three slices of 100 repositories cannot be filled from that window.
|
||||
|
||||
**Decision 1.4: licences.** The licence is read at the pinned head commit (LICENSE file, then the API's SPDX id).
|
||||
Only permissive licences are published by name: MIT, Apache-2.0, BSD-2-Clause, BSD-3-Clause, ISC, 0BSD, Unlicense,
|
||||
Zlib, PostgreSQL, BSL-1.0. Everything else (copyleft, source-available, none) goes to the gitignored
|
||||
`benchmarks/unseen/private/pool-p1.toml`. The committed manifest then holds an opaque id and the sha256 of
|
||||
(URL, commits) for that entry, so the freeze digest still covers it. Reason: Req 1.4 read literally. Rejected:
|
||||
"OSI-approved" as the bar, as population v2 used. It is wider than the requirement says (open question 1,
|
||||
resolved 2026-10-04: copyleft stays private).
|
||||
|
||||
Also required: anonymous HTTPS, at most 500 MB, at least 40 eligible ordinary commits, one per fork network.
|
||||
|
||||
### 2. Change extraction (Req 2.1, 2.2)
|
||||
|
||||
Extraction runs on the VM over a full `--filter=blob:none` clone in a scratch cache. It uses git only, like
|
||||
`freeze.py`. No scanner, engine or model touches a pool repository before the freeze.
|
||||
|
||||
**Decision 2.1: the vulnerability-introducing range.** Let F be the fix and P its parent. The vulnerable lines are
|
||||
the old-side lines of F's hunks in the vulnerable function. For a pure-addition fix (an absence bug such as a
|
||||
missing guard), they are the vulnerable function's own lines.
|
||||
|
||||
1. **`introducing` (preferred).** `git blame -w -M -C --porcelain P -L <lines> -- <file>`. The candidate V is the
|
||||
commit that last wrote the sink line (for absence bugs, the function's declaration). V is accepted if all hold:
|
||||
at V's first parent the sink line's normalised text is absent from the file's vulnerable function; V is not a
|
||||
merge or root commit; V changes at most 50 files and 2,000 lines; when OSV names a first affected version, V is
|
||||
reachable from that tag and not from the previous release tag. If the text is present at V's parent, blame
|
||||
walks back from there, at most five steps. The replay is `base = V^`, `head = V`.
|
||||
2. **`last_touch` (fallback).** When no V passes, `git log -L :<function>:<file> P` gives the newest commit L
|
||||
before F that changed the vulnerable function. The replay is `base = L^`, `head = P`. The vulnerable code is in
|
||||
head, and the range contains a change to the vulnerable function.
|
||||
|
||||
Each change records its method, family, vulnerable function (resolved in head), head-side changed lines,
|
||||
`post_cutoff` and the advisory id.
|
||||
|
||||
Bias of each, stated in the record. `introducing` inherits SZZ noise: blame finds the last writer, not the author
|
||||
of the flaw. A refactor, a move the heuristics miss or a reformat can stand in for the real introduction. The
|
||||
absence check, the walk-back and the version bracket cut this down, but do not remove it. The bias runs both
|
||||
ways: a wrong V may introduce nothing (a miss is not the hook's fault) or may move a flaw (the delta-based hook is
|
||||
right to call it existing). `last_touch` is pessimistic for a delta hook. The flaw is often already in base, so
|
||||
the hook's novelty rule correctly stays quiet and the change counts as a miss. Catch is therefore reported per
|
||||
method. The headline is `introducing`; `last_touch` is a lower bound. Rejected: fix-parent against a fixed old
|
||||
base (for example 50 commits back). The range often misses the vulnerable function entirely and grows without
|
||||
bound on busy repositories.
|
||||
|
||||
Label check: every third `introducing` change is read by an agent before the freeze, with no model call, as
|
||||
harvest-3 reviewed diffs. It is marked `confirmed`, `refactor` or `unclear`. The confirmed share is the label
|
||||
noise estimate. The scorer also reports catch on the confirmed subset.
|
||||
|
||||
**Decision 2.2: ordinary commits.** The candidates are non-merge commits on the first-parent line of the default
|
||||
branch that change at least one source file of an analysed language. Excluded:
|
||||
|
||||
- explicit security signals in the message: `secur|cve|ghsa|vuln|inject|xss|csrf|ssrf|exploit|sanitiz|escap`;
|
||||
- reverts (`^Revert`, `This reverts commit`);
|
||||
- dependency bumps (bot authors such as dependabot and renovate, `bump|upgrade .* from`, or only manifest and lock
|
||||
files changed);
|
||||
- commits whose source changes are all generated or vendored (`vendor/`, `node_modules/`, `dist/`, `*.min.js`,
|
||||
lock files, `linguist-generated` in `.gitattributes`, "generated" headers);
|
||||
- any fix commit of any advisory of the repository, and any commit touching a known vulnerable function of it;
|
||||
- commits over 1,000 changed source lines (counted, reported).
|
||||
|
||||
Twelve are drawn per repository with a seeded random draw. Strata are three time thirds of the repository's
|
||||
history crossed with four size buckets of changed source lines (1-10, 11-50, 51-200, 201-1,000). Allocation is
|
||||
proportional to the repository's own commits per cell, with at least one per time third. Each change records base
|
||||
(its first parent), head, changed files and line ranges. Reason: proportional allocation keeps the false-alarm
|
||||
rate a per-push rate as a user meets it. Rejected: `freeze.py`'s broad `SECURITY_WORDS`. It also drops commits
|
||||
saying "query", "token", "session" or "valid", which removes the ordinary commits where false alarms are most
|
||||
likely, and the rate would look better than it is. Rejected: docs-only commits. They are trivially quiet and
|
||||
would dilute the rate.
|
||||
|
||||
Label noise: an ordinary commit may itself introduce an unknown vulnerability, so a flag counted as a false alarm
|
||||
may be true. The false-alarm rate stays as Req 3.2 defines it. Flagged ordinary changes are adjudicated beside it
|
||||
(all if 30 or fewer, else every third) against protocol-v2's standard, and the adjudicated share is reported.
|
||||
|
||||
### 3. Sizing and slices (Req 2.3, 5.2)
|
||||
|
||||
Wilson 95% interval, z = 1.96: centre (p + z²/2n)/(1 + z²/n), half width z·sqrt(p(1-p)/n + z²/4n²)/(1 + z²/n).
|
||||
Taking the larger distance from p as the width, the smallest n that meets the bar, computed exactly:
|
||||
|
||||
| Rate | Assumed p | Bar | n needed |
|
||||
| --- | --- | --- | --- |
|
||||
| false alarm | 0.05 | +/- 3 points | 315 |
|
||||
| false alarm | 0.10 | +/- 3 points | 483 |
|
||||
| false alarm | 0.20 | +/- 3 points | 756 |
|
||||
| catch | 0.10 | +/- 10 points | 62 |
|
||||
| catch | 0.25 | +/- 10 points | 88 |
|
||||
| catch | 0.40 | +/- 10 points | 97 |
|
||||
|
||||
Catch: 100 vulnerability-introducing changes per slice, one per repository. At p = 0.4, Wilson gives
|
||||
[0.309, 0.498], within 10 points. One per repository keeps them independent.
|
||||
|
||||
False alarm: ordinary changes of one repository are correlated (noisy code flags often). Wilson assumes
|
||||
independence, so the count is inflated by a design effect 1 + (m - 1)·ICC. With m = 12 per repository and an
|
||||
assumed ICC of 0.05, that is 1.55. 100 repositories × 12 = 1,200 changes, effective n 774 ≥ 756. The plain Wilson
|
||||
interval at p = 0.2, n = 1,200 is about +/- 2.3 points. A repository-cluster bootstrap interval is reported beside
|
||||
it, and the ICC measured on slice 1 is recorded. If it exceeds 0.05, the next pool sizes up. A frozen slice is
|
||||
never changed.
|
||||
|
||||
**Decision 3.1: slices.** Pool p1 has three slices of 100 repositories each (100 + 1,200 changes per slice; 300
|
||||
repositories and 3,900 changes in all). Repositories are assigned by a seeded shuffle, stratified by ecosystem and
|
||||
`post_cutoff`. Slice 1 serves the baseline and the G1 arms table. Slices 2 and 3 serve the first two adoption
|
||||
decisions. When they are spent, pool p2 is drawn by the same code, and it excludes p1. Reason: each slice alone
|
||||
meets the Req 2.3 widths, so any decision can stand on one slice. Rejected: one large pool scored whole. Every
|
||||
decision would then see every repository, which Req 5.2 forbids. Rejected: ten small slices. None would meet the
|
||||
widths. Per family, 100 changes split over seven families give intervals of about +/- 20 to 25 points. They are
|
||||
reported, not sized (open question 3).
|
||||
|
||||
Use ledger: `benchmarks/unseen/usage-p1.jsonl` is append-only, one row per use (slice, decision or experiment id,
|
||||
date, result digest). A slice with a row whose purpose is `informed` cannot appear again as `qualifies` for that
|
||||
change. A test checks this.
|
||||
|
||||
### 4. Scoring rules (Req 3.1, 3.2, 3.3)
|
||||
|
||||
Each replay uses the hook's defaults: `--deadline 30`, `--engine inline`, `--max-regions 500`, `--mode advisory`.
|
||||
That is what a user's push gets. The text output is kept with every `Skipped:` line, and so is the artifact.
|
||||
|
||||
**Decision 4.0: a registered long-deadline arm** (maintainer decision 2026-10-04, open question 5). Beside the
|
||||
default replay, slice 1 is replayed once more with `--deadline 300` and every other flag unchanged. The arm is
|
||||
registered (manifest committed, `unseen: {pool: p1, slice: 1}`) before its first replay. G1 then reports both
|
||||
what the user sees (30 s) and what the engine could catch given time (300 s), so a miss caused by the deadline is
|
||||
told apart from a miss by the engine. It is a measurement arm, not a proposed default: adopting a longer hook
|
||||
deadline would be a separate decision under 7.2.
|
||||
|
||||
**Decision 4.1: catch, per family.** A finding counts if its family is the change's family. Engine streams carry a
|
||||
family; a quick-rule match maps its CWE through the family CWE table. Its head location must also be in the
|
||||
vulnerable function or within 10 lines of the change's head-side hunks (protocol v1's window).
|
||||
|
||||
| Family | Location rule |
|
||||
| --- | --- |
|
||||
| injection, path, deserialization, output_encoding, untrusted_destination | sink line in the vulnerable function, or the changed lines +/- 10 |
|
||||
| access_control | the vulnerable function itself (the handler that lacks the check), or the changed lines +/- 10 |
|
||||
| config_secrets | the changed file and lines +/- 10 (file scope; function `<global>`) |
|
||||
|
||||
All three streams count: admitted defects, advisory engine findings and quick-rule matches. Each catch also names
|
||||
the stream that made it. Reason: the hook shows all three to the user. Rejected: admitted defects only. That
|
||||
scores the blocking path, which G3 decides separately, and would hide what users read.
|
||||
|
||||
**Decision 4.2: false alarm.** An ordinary change is a false alarm when any of the three streams holds at least one
|
||||
finding, of any family, anywhere. The alerts-only rate (admitted defects) is reported beside it.
|
||||
|
||||
**Decision 4.3: coverage, so zeros cannot hide.** Every change gets exactly one state:
|
||||
|
||||
- `instrument_failure`: no record, a failed fetch, the head commit not verified, or zero bytes of the changed files
|
||||
read. It is re-run once, then reported. It never counts as quiet or missed.
|
||||
- `unanalysable`: no changed file is in a language the hook covers (the hook's `language_not_covered` notice).
|
||||
- `quick_only`: the quick tier read the changed files, but no engine question completed on them.
|
||||
- `engine_covered`: at least one engine question completed on a changed file.
|
||||
|
||||
The task entry proves its input as AGENTS.md requires. It records files and bytes in the checkout, bytes of each
|
||||
changed file, the JVM's wall time and the hook's exit code. A replay faster than a JVM can start is flagged
|
||||
`suspect_fast` and goes to the failure path. Catch and false alarm are reported over all changes and over each
|
||||
coverage state.
|
||||
|
||||
### 5. The AX workload (Req 4)
|
||||
|
||||
**Decision 5.1: the image.** `plane/Dockerfile.engine-task` gains a second stage, `FROM engine AS replay`, built
|
||||
with `--target replay` and pinned by digest. It adds `git`, the package with its runtime dependencies and
|
||||
`benchmarks/unseen/replay_entry.py`. Joern, php-cli, Java 21 and the frontend-retention patch come from the engine
|
||||
stage, so both lanes run the same Joern. The import smoke runs `ousast pre-push --help`. The root image stays
|
||||
`ghcr.io/norandom/ax-task-runner:v0.3.1`. Rejected: a separate Dockerfile. It would duplicate the Joern install,
|
||||
and two Joern copies are two instruments that drift. Rejected: one stage for both. The engine image's isolated
|
||||
venv without yaml and boto3 is proven, and replay dependencies would change its digest on every bump.
|
||||
|
||||
**Decision 5.2: the task entry.** `replay_entry.py` is standard library only, like `engine_task_entry.py`, and
|
||||
reuses its `transfer` (egress readiness wait, three attempts, no URL text in errors).
|
||||
|
||||
1. GET `SPEC_URL` (presigned, `.json`): repository URL, base, head, change id, hook flags.
|
||||
2. `git init`, then `git fetch --depth 1 origin <head>` and `git fetch --depth 1 origin <base>` over HTTPS
|
||||
(GitHub serves commits by SHA). Verify both with `git cat-file -e`. The fetch is bounded at 180 s; the 500 MB
|
||||
repository cap bounds its size.
|
||||
3. Run `ousast pre-push <checkout> --base <base> --head <head> --artifact /tmp/out/push.json` with defaults, the
|
||||
backend heap at 1700 MB and the cluster JVM flags of `engine_trace_worker.py`. Capture stdout.
|
||||
4. Pack `push.json`, `push.txt`, `engine.json` (the head and base scan records and witness rows from the artifact,
|
||||
keyed by repository and revision, for engine-evidence arms) and `instrument.json` (section 4.3, image digest,
|
||||
step timings). PUT to `RESULT_URL` as `.dat`. The PUT is completion.
|
||||
|
||||
On any error it PUTs a failure record naming the step and HTTP status, as the engine entry does. Task names are
|
||||
`ousast-engine-replay-<slug>-<digest>`, the prefix the operator's egress watcher matches. No key or store
|
||||
credential enters the manifest. Only the two presigned URLs do, and the spec object keeps repository names out of
|
||||
AX status and cluster telemetry.
|
||||
|
||||
**Decision 5.3: the generic dispatcher.** `benchmarks/ax/batch.py` takes over from `engine_trace_ax.AX` and
|
||||
`engine_trace.run_plan`:
|
||||
|
||||
- `Item`: id, input objects (name to bytes), the result object name, command, extra env, input digest.
|
||||
- `Workload`: image digest, allowed URL env names, deadline, how to validate a received result.
|
||||
- Lanes: `ax` (manifest, apply, poll the result object and AX phase, delete, clean objects), `kind` (the same
|
||||
manifest against the kind AX), `docker` (the VM image, one container at a time).
|
||||
- Scheduler: two AX lanes plus one VM lane by default. A sandbox death is retried once under a new name; a second
|
||||
death or a JVM OOM goes to the VM fallback queue, which beats overflow on the VM lane. `progress.json` is written
|
||||
only by the scheduler. A `STOP` file halts new launches. Resume skips items whose stored record has the same
|
||||
input digest. Cleanup failure stops dispatching.
|
||||
|
||||
The engine pass becomes a workload over it, with the same tests. Reason: the egress, secrecy and cleanup logic took
|
||||
several fixes and must exist once. Rejected: a replay-specific copy of `engine_trace_ax.py`. Rejected: putting it in
|
||||
`src/` now. G4's `ousast plane scan` may need it there, and it moves when G4 is planned.
|
||||
|
||||
Concurrency is two AX tasks. Placement is random, and only two guarantees two per worker at most. Four would need
|
||||
the operator to add placement (open question 2, resolved 2026-10-04: stays at two, no placement change
|
||||
requested). Estimated wall time per slice: about 75 s per replay (fetch, 30 s hook, setup), 1,300 replays over
|
||||
three lanes, about 9 hours. The long-deadline arm (4.0) is bounded at about 345 s per replay; at that bound 1,300
|
||||
replays over three lanes take about 42 hours, less where the hook finishes early. It runs with `STOP` and resume
|
||||
across sessions.
|
||||
|
||||
**Decision 5.4: model arms on the VM.** Arms that need DeepSeek (the decision engine, the combiner) read the stored
|
||||
records on the VM. They build candidates from the head side's changed functions, score them, and take the change
|
||||
score as the maximum. Each runs under its experiment manifest's `budget_usd` through the metered client, dry run
|
||||
first, with responses cached. Nothing in the cluster calls a model.
|
||||
|
||||
**kind (Req 4.5)** runs the identical manifest and image through the `kind` lane. Every new task is proven there
|
||||
before kube-ax, as `plane-on-kubernetes` requires.
|
||||
|
||||
### 6. Records and the guard (Req 1.2, 1.3, 5.1)
|
||||
|
||||
`benchmarks/unseen/` holds `pool-p1.toml` (repositories, licences, slices, changes with base, head, method,
|
||||
family, function, lines), `freeze-p1.json` (used-set digests and counts per source, guard result, label-check
|
||||
counts, freeze digest = sha256 of the canonical manifest), `protocol-p1.md` (the scoring rules of section 4,
|
||||
committed before the first replay) and `usage-p1.jsonl`. The `private/` directory is gitignored.
|
||||
|
||||
The guard grows by one parametrised reservation. The existing sweep becomes a list of (reservation directory,
|
||||
names): `benchmarks/independent/` for populations and `benchmarks/unseen/` for pools, including the private
|
||||
manifest when present. It also sweeps `plane/`. A second test asserts that pool and population names are disjoint,
|
||||
and prints only a count on failure. A third asserts that a frozen pool's digest matches `freeze-p1.json`, so the
|
||||
pool is never edited. Raw per-change records stay outside the repository (`~/ousast-results/unseen/` and the
|
||||
store). Committed result records under `benchmarks/measurements/` carry counts, rates, intervals and digests,
|
||||
never names.
|
||||
|
||||
### 7. Metrics and reporting (Req 3.3, 3.4, 5.3)
|
||||
|
||||
`benchmarks/unseen/score.py` reports per family and overall: catch and false-alarm rate with Wilson 95%
|
||||
intervals, the cluster bootstrap interval for false alarms, coverage counts per state, catch per method,
|
||||
post-cutoff and per stream, and the alerts-only rate.
|
||||
|
||||
**Decision 7.1: arms at fixed budgets.** An arm with a score is thresholded per budget (1%, 5%, 10%). The
|
||||
threshold is the lowest score whose false-alarm rate on the slice's ordinary changes stays within the budget.
|
||||
Catch at that threshold is reported with its Wilson interval and the realised false-alarm rate. That in-slice
|
||||
threshold is labelled exploratory. For a qualification, the threshold is fixed beforehand on an earlier slice or
|
||||
on held-out folds, and it is applied unchanged. A binary arm (today's hook) has one operating point, reported with
|
||||
the budgets it meets. Rejected: AUC over changes (it averages operating points no user runs).
|
||||
|
||||
**Decision 7.2: next to held-out folds.** Experiment manifests registered after this spec lands carry
|
||||
`unseen: {pool, slice}`. `analyse` writes an `unseen` block into `result.json` next to the fold estimates: catch at
|
||||
each budget for each arm, and the paired difference B - A with an exact McNemar test on the 100 changes. The
|
||||
fold rule decides as today. The slice is recorded in the usage ledger.
|
||||
|
||||
**Decision 7.3: adoption gate on the unseen slice** (amended 2026-10-04, maintainer decision; Req 5.4; open
|
||||
question 4 resolved). Reporting is not enough. An arm B is adopted over the production arm A only if, on the
|
||||
untouched slice that qualifies it, both hold:
|
||||
|
||||
- catch at the 5% budget, with B's threshold fixed beforehand (7.1), is not significantly lower than A's: exact
|
||||
two-sided McNemar on the paired vulnerability-introducing changes, alpha 0.05;
|
||||
- for an arm with one operating point (a binary arm such as a hook setting), its false-alarm rate is not
|
||||
significantly higher than A's: exact two-sided McNemar on the paired ordinary changes, alpha 0.05, with the
|
||||
repository-cluster bootstrap interval of the difference reported beside it, since those changes are clustered.
|
||||
|
||||
`analyse` writes `unseen.gate: pass | regression | not_run` into `result.json`. `not_run` (no slice, slice
|
||||
already `informed` for this change, or more than 5% `instrument_failure`) blocks adoption like `regression`. The
|
||||
fold rule and this gate must both pass.
|
||||
|
||||
### 8. Increments
|
||||
|
||||
1. **Used set and guard.** `eligibility.py`, the extended guard and the usage-ledger test. Exit: guard green on an
|
||||
empty pool; a planted used repository, a renamed one and a fork are each rejected in tests.
|
||||
2. **Sourcing and extraction.** Candidates, both change kinds and the label check, on the VM. Exit: a draft for 300
|
||||
repositories with method counts, the confirmed share and exclusion counts.
|
||||
3. **Freeze p1.** Guard bisection for v3, then manifest, freeze record and protocol committed. Exit: freeze digest
|
||||
committed and guard green.
|
||||
4. **Dispatcher.** `benchmarks/ax/batch.py`, with the engine pass moved onto it. Exit: existing tests green and a
|
||||
two-pin engine smoke on kube-ax matching its stored records.
|
||||
5. **Replay task.** Image stage and entry, proven on kind, then kube-ax on five changes from this repository's own
|
||||
history (never pool changes). Exit: instrument fields show bytes read and realistic JVM time, and quick-tier
|
||||
output matches the VM lane.
|
||||
6. **Scorer.** `score.py` on synthetic records, including every coverage state and an instrument failure. Exit:
|
||||
unit tests green.
|
||||
7. **Baseline.** Today's hook, from a frozen export of HEAD, on slice 1 (1,300 replays). Exit: the record committed
|
||||
(counts and digests only) and a `baseline` row in the usage ledger. That is G0's exit.
|
||||
8. **Arms on slice 1 (G1)** (added 2026-10-04 with decisions 4.0 and 7.3). The registered long-deadline arm,
|
||||
the engine-evidence arm from the stored records, the model arms on the VM (5.4) under their budgets, and the
|
||||
`unseen` block with the adoption gate in `analyse`. Exit: one committed table of arms on slice 1 at the 1/5/10%
|
||||
budgets (counts, rates, intervals, digests), and `exploratory` rows in the usage ledger.
|
||||
|
||||
## Error Handling
|
||||
|
||||
A failed fetch, a missing commit or a hook crash gives a failure record, never an empty success. A dead sandbox is
|
||||
retried once, then the VM runs it. A store or AX cleanup error stops the dispatcher. A slice with more than 5%
|
||||
`instrument_failure` after re-runs is not scored; the cause is fixed first.
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
Unit tests: used-set normalisation (case, `.git`, renames, forks), blame acceptance on a synthetic repository,
|
||||
ordinary-commit exclusions, Wilson values against section 3, budget thresholds, coverage states, the task manifest
|
||||
and dispatcher resume and fallback with a fake lane. Guard tests run in CI; cluster runs stay manual.
|
||||
|
||||
## Requirement Traceability
|
||||
|
||||
| Requirement | Design |
|
||||
| --- | --- |
|
||||
| 1.1 eligibility against every source, v3 via the guard | 1.2 |
|
||||
| 1.2 frozen manifest, never edited | 6, increment 3 |
|
||||
| 1.3 pool directory, guard extended | 6 |
|
||||
| 1.4 licences, private manifest | 1.4 |
|
||||
| 2.1 introducing changes, family, function, post-cutoff | 1.1, 1.3, 2.1 |
|
||||
| 2.2 ordinary changes across time and size | 2.2 |
|
||||
| 2.3 sizes for interval widths | 3 |
|
||||
| 3.1 default replay, skipped checks kept | 4, 5.2 |
|
||||
| 3.2 catch and false-alarm rules | 4.1, 4.2 |
|
||||
| 3.3 per family, Wilson, coverage | 4.3, 7 |
|
||||
| 3.4 arms at 1/5/10% budgets | 4.0, 7.1, increment 8 |
|
||||
| 4.1 one change one Task, prefix, image by digest | 5.1, 5.2 |
|
||||
| 4.2 HTTPS fetch, presigned record, no keys | 5.2 |
|
||||
| 4.3 generic dispatcher, concurrency, retry, fallback, resume | 5.3 |
|
||||
| 4.4 model arms on the VM within budgets | 5.4 |
|
||||
| 4.5 kind runs the same tasks | 5.3, 5 (kind paragraph) |
|
||||
| 5.1 baseline on slice 1, counts and digests | 6, increment 7 |
|
||||
| 5.2 fresh slice per decision | 3.1, usage ledger |
|
||||
| 5.3 experiments report unseen next to folds | 7.2 |
|
||||
| 5.4 adoption needs no significant unseen regression | 7.3 |
|
||||
|
||||
## Open Questions
|
||||
|
||||
Maintainer decisions of 2026-10-04 resolve questions 1, 2, 4 and 5; question 3 stays open for G3.
|
||||
|
||||
1. **Resolved 2026-10-04.** Licences: only permissively licensed repositories are published by name. Copyleft and
|
||||
all other licences go to the gitignored private manifest, covered in the committed manifest by an opaque id and
|
||||
a digest (Req 1.4 as written; decision 1.4). Population v2's OSI-approved bar is not used.
|
||||
2. **Resolved 2026-10-04.** Concurrency stays at two AX lanes plus the VM lane. No operator placement change is
|
||||
requested (5.3).
|
||||
3. Per-family BLOCK (G3) needs about 100 changes per family. Pool slices together, or draw a family-enriched pool
|
||||
for G3?
|
||||
4. **Resolved 2026-10-04.** An adoption requires no significant regression on the unseen slice, not only a report
|
||||
(Req 5.4, amended; decision 7.3).
|
||||
5. **Resolved 2026-10-04.** G1 adds a registered long-deadline arm (`--deadline 300`) next to the default 30 s, so
|
||||
the table shows both what the user sees and what the engine could catch (decision 4.0, increment 8).
|
||||
@@ -0,0 +1,107 @@
|
||||
# Requirements Document
|
||||
|
||||
## Introduction
|
||||
|
||||
The safety net is meant for repositories it has never seen. This spec builds the instrument that measures that: a
|
||||
frozen pool of unseen repositories, a push-level metric (catch rate and false-alarm rate as the hook reports them),
|
||||
and an execution path that runs the replays as AX Tasks on the kube-ax cluster. Context: `brief.md`, the roadmap's
|
||||
G0-G1 (`.kiro/steering/roadmap.md`).
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: pool selection and freezing; change extraction (vulnerability-introducing and ordinary commits);
|
||||
replay of each change through `ousast pre-push` and through registered experiment arms; the metric and its
|
||||
confidence intervals; AX Task execution with kind/VM fallback; the baseline for today's hook; rotation of pool
|
||||
slices.
|
||||
- **Out of scope**: any detection change; population v3 (stays reserved and unread); model calls inside the cluster.
|
||||
- **Adjacent expectations**: `learned-decision-engine` experiments use this pool as their primary unseen-repo score;
|
||||
`pre-push-safety-net` owns the hook being replayed; `plane-on-kubernetes` and the AX executor
|
||||
(`benchmarks/learn/engine_trace_ax.py`) supply the execution pattern; the reserved-repository guard covers every
|
||||
population and this pool.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement 1: A pool no development step has touched
|
||||
|
||||
**User Story:** As the maintainer, I want the evaluation repositories to be ones nothing in development has seen, so a
|
||||
good number means the approach generalises.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. A repository is eligible only if it appears in no pair corpus, no population (v1, v2, v3), no harvest record, no
|
||||
memory example, no experiment unit and no measurement record. Eligibility is checked by code against every source
|
||||
before the pool is frozen; v3 is checked through the existing guard without being opened.
|
||||
2. The pool is frozen (a committed manifest with repository URLs, commit SHAs, licences and the freeze digest) before
|
||||
any change in it is replayed. A frozen pool is never edited; a correction is a new pool.
|
||||
3. Pool repositories are referenced only inside the pool's own directory (like `benchmarks/independent/`), and the
|
||||
reserved-repository guard is extended to them.
|
||||
4. Only permissively licensed repositories are published by name; others may be used with their names kept out of
|
||||
the repository (pointers in a local, gitignored manifest), recorded as such.
|
||||
|
||||
### Requirement 2: Two kinds of change per repository
|
||||
|
||||
**User Story:** As the maintainer, I want both vulnerable and ordinary pushes from the same repositories, so the
|
||||
false-alarm rate is measured on the code the catch rate is measured on.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. Vulnerability-introducing changes: from public advisories with a linked fix, the commit range that introduced the
|
||||
vulnerable code where it can be determined, else the fix's parent compared with an earlier base; each labelled
|
||||
with family and the vulnerable function. Post-model-cutoff advisories are preferred and flagged.
|
||||
2. Ordinary changes: commits from the same repositories' histories with no security fix, revert or advisory link,
|
||||
sampled across time and size; each recorded with its changed files and lines.
|
||||
3. Per repository, at least one vulnerability-introducing change and several ordinary changes; the pool's totals are
|
||||
sized so the false-alarm rate's 95% interval is no wider than +/- 3 points and the catch rate's no wider than
|
||||
+/- 10 points (sizes computed in the design).
|
||||
|
||||
### Requirement 3: The push-level metric
|
||||
|
||||
**User Story:** As the maintainer, I want the score to be what a user would see, so the number means something.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. Each change is replayed as `ousast pre-push --base <base> --head <head>` with the hook's default settings, in a
|
||||
clean checkout, with the record of every skipped check kept.
|
||||
2. Catch: a vulnerability-introducing change counts as caught when the hook (or the arm) reports a finding of the
|
||||
right family on the vulnerable function or its changed lines. False alarm: an ordinary change counts as a false
|
||||
alarm when the hook reports any advisory finding.
|
||||
3. Reported per family and overall: catch rate, false-alarm rate, and both with 95% Wilson intervals; plus coverage
|
||||
(changes where the hook could analyse the changed language) so a silent zero is visible.
|
||||
4. Arms (decision engine, combiner, engine evidence, fix mechanisms) are scored on the same replays: catch rate at
|
||||
fixed false-alarm budgets (1%, 5%, 10%), so arms are compared at equal noise.
|
||||
|
||||
### Requirement 4: Execution on kube-ax, designed for AX
|
||||
|
||||
**User Story:** As the maintainer, I want the replays to run on the cluster, so measurements are fast and the VM is
|
||||
free.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. One replayed change is one AX Task, named with the `ousast-engine-` prefix the operator's egress watcher matches,
|
||||
using an image built `FROM ghcr.io/norandom/ax-task-runner:v0.3.1` with this package, Joern and php-cli, pinned by
|
||||
digest.
|
||||
2. The task fetches the repository at the needed commits itself over HTTPS (public hosts are reachable from actors)
|
||||
and writes its record through a presigned link; no API keys or store credentials enter the cluster.
|
||||
3. A generic AX batch dispatcher serves both this replay and the existing engine passes: items to Tasks, at most the
|
||||
cluster's safe concurrency (two heavy tasks per worker), retry once on sandbox death, then a VM or kind fallback
|
||||
lane; progress and resume as in the engine runner.
|
||||
4. Arms that need model calls take the cluster's records and make their calls on the VM, within registered budgets.
|
||||
5. kind runs the same tasks unchanged as the local fallback.
|
||||
|
||||
### Requirement 5: Baseline, rotation and use
|
||||
|
||||
**User Story:** As the maintainer, I want a baseline now and a fresh slice for each decision, so the pool is not
|
||||
worn out by tuning.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The first use is a baseline of today's hook on the first slice; its record is committed (counts and digests only,
|
||||
no repository names outside the pool directory).
|
||||
2. A slice that informed a change can no longer qualify that change; each adoption decision draws a new, untouched
|
||||
slice, recorded with its freeze digest.
|
||||
3. Every learned-decision-engine experiment from now on reports its unseen-repo score from this pool next to its
|
||||
held-out-fold score.
|
||||
4. An arm is adopted only if, on the untouched slice that qualifies it, it shows no significant regression against
|
||||
the current production arm: catch at the 5% false-alarm budget not significantly lower, and for an arm with one
|
||||
operating point the false-alarm rate not significantly higher (exact paired tests, two-sided, alpha 0.05).
|
||||
Reporting the unseen score alone is not enough to adopt. *(amended 2026-10-04, maintainer decision)*
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"feature_name": "unseen-repo-evaluation",
|
||||
"created_at": "2026-10-04T18:00:00Z",
|
||||
"updated_at": "2026-10-04T20:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"design": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"tasks": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": true
|
||||
}
|
||||
@@ -0,0 +1,384 @@
|
||||
# Implementation Plan
|
||||
|
||||
One task group per increment of `design.md` section 8, in that order. A group starts only after the previous
|
||||
group's exit is recorded. Maintainer decisions of 2026-10-04 are folded in: a registered long-deadline arm
|
||||
(`--deadline 300`) beside the hook's default 30 s (design 4.0, group 8); adoption needs no significant regression
|
||||
on the unseen slice (Req 5.4 as amended, design 7.3, tasks 6.4 and 8.2); only permissively licensed repositories
|
||||
are published by name, everything else goes to the gitignored private manifest under a digest (Req 1.4, design
|
||||
1.4); concurrency stays at two AX lanes plus the VM lane (design 5.3).
|
||||
|
||||
Execution rules:
|
||||
|
||||
- **Who does what.** Codex implements every coding task offline: no network, no model, no store, no cluster. The
|
||||
coordinator runs every networked step (GitHub and OSV API calls, git fetches and clones, the S3 store, image
|
||||
pushes, kind and kube-ax). Those tasks are marked **(coordinator, network)**. Steps that need the coordinator but
|
||||
no network (agent label reads, adjudication, the v3 guard bisection, commits) are marked **(coordinator)**.
|
||||
Steps that spend model money are marked **paid: needs a budget go-ahead**. Each starts only after the maintainer
|
||||
confirms a ceiling against the provider balance read just before it.
|
||||
- **Done means.** A coding task is done when its tests pass in the host suite, `ruff check`, `ruff format --check`
|
||||
and `mypy` are clean, and the reserved-repository guard is green. No test makes a network, model or cluster call.
|
||||
- **Names stay in the pool directory.** Pool repository names and URLs appear only under `benchmarks/unseen/`
|
||||
(copyleft and other non-permissive ones only under the gitignored `benchmarks/unseen/private/`). Raw per-change
|
||||
records live in `~/ousast-results/unseen/` and the store, never in the repository.
|
||||
- **Population v3 is never opened.** It is checked only through the guard test, run as a black box with its
|
||||
output discarded.
|
||||
- **Every group ends with a measured exit** and a record at
|
||||
`benchmarks/measurements/<date>-unseen-NN-<slug>/record.json`: commit, image digests where used, the instrument
|
||||
checks (bytes read, JVM time, exit codes), counts and digests. No repository names. The guard sweeps these
|
||||
records.
|
||||
- **Instrument first** (AGENTS.md). A zero, an empty draft or a fast run is treated as an instrument failure until
|
||||
bytes read and wall time prove otherwise.
|
||||
|
||||
- [x] 1. Used set and guard (design 1.2, 6; increment 1)
|
||||
- [x] 1.1 Used-set builder
|
||||
- `benchmarks/unseen/eligibility.py`: normalises names (lower case, no `.git`, no trailing slash) and builds the
|
||||
used set from every source of design table 1.2: pair catalogs, recipes and datasets under `benchmarks/pairs/`
|
||||
plus the local pointer-pair manifests as `benchmarks/pairs/catalog_gen.py` reads them; `population-v1.toml`
|
||||
and `population-v2.toml`; the advisory-fix batch catalogs; memory `example` rows (`repo` field) through an
|
||||
injected `MemoryStore` (`FileStore` locally, the S3 store when the coordinator passes one); experiment units
|
||||
under `plane/experiments/`; `benchmarks/measurements/` and `benchmarks/experiments/`; a text sweep for
|
||||
`github.com/<owner>/<name>` over `benchmarks/`, `src/`, `tests/`, `plane/` and `.kiro/`. It never reads
|
||||
population v3 and never prints a name. Output: counts per source and the sha256 of the sorted set.
|
||||
- `tests/test_unseen_eligibility.py`: a synthetic tree with one repository per source, each found; case, `.git`
|
||||
and trailing-slash variants collapse; stdout and the output JSON contain no name; a v3-shaped file planted in
|
||||
the synthetic tree is not opened (open is patched to fail on it).
|
||||
- _Requirements: 1.1_
|
||||
- [x] 1.2 Canonical resolution, renames and forks
|
||||
- `eligibility.py`: `resolve(candidate, api)` with an injected GitHub client returns the canonical `full_name`,
|
||||
former names (redirects) and the fork network (`parent`, `source`). A candidate is ineligible when any of them
|
||||
is in the used set; one candidate per fork network is kept.
|
||||
- Tests with a fake client: a planted used repository, a renamed one and a fork of a used one are each
|
||||
rejected; two forks of one network keep one; the rejection reason is a source label, not a name.
|
||||
- _Requirements: 1.1_
|
||||
- [x] 1.3 Guard extended to pools
|
||||
- `tests/test_independent_population.py`: the sweep becomes a list of (reservation directory, names):
|
||||
`benchmarks/independent/` for populations, `benchmarks/unseen/` for pools (public manifests, plus
|
||||
`benchmarks/unseen/private/*.toml` when present). The sweep also covers `plane/`. New tests: pool and
|
||||
population names are disjoint (failure prints a count only); a frozen pool's manifest digest equals its
|
||||
`freeze-pN.json` (skipped while no pool is frozen). The existing v3 handling is unchanged.
|
||||
- Each reservation is exempt only inside its own directory: population names (v3 included) are still swept in
|
||||
`benchmarks/unseen/`, the gitignored `private/` files included. Task 3.1 depends on this.
|
||||
- `.gitignore`: `benchmarks/unseen/private/`.
|
||||
- Evidence: the guard is green with an empty `benchmarks/unseen/`, and red when a test fixture names a pool
|
||||
repository in `benchmarks/measurements/` (shown once in the record).
|
||||
- _Requirements: 1.2, 1.3, 1.4_
|
||||
- [x] 1.4 Usage ledger
|
||||
- `benchmarks/unseen/ledger.py`: append-only `usage-pN.jsonl`, one row per use (slice, decision or experiment
|
||||
id, purpose `baseline | exploratory | informed | qualifies`, date, result digest, freeze digest). It refuses a
|
||||
`qualifies` row for a change whose slice already has an `informed` row for it, and refuses to rewrite rows.
|
||||
- `tests/test_unseen_ledger.py`: append, the refusal, a rewritten row detected by a digest chain.
|
||||
- _Requirements: 5.2_
|
||||
- [x] 1.5 Used-set measurement and record 01 **(coordinator, network)**
|
||||
- Run `eligibility.py` against the real sources, the S3 memory store included (read only).
|
||||
- Instrument: each source reports files read and bytes; a source with zero rows fails the run unless it is
|
||||
known empty.
|
||||
- `benchmarks/measurements/<date>-unseen-01-used-set/record.json`: counts per source, used-set digest, guard
|
||||
result (green on the empty pool), test counts of 1.1-1.4.
|
||||
- _Requirements: 1.1, 1.3_
|
||||
|
||||
- [ ] 2. Sourcing and extraction (design 1.1, 1.3, 1.4, 2, 3; increment 2)
|
||||
- [x] 2.1 Advisory candidates and repository filters
|
||||
- `benchmarks/unseen/source.py`: from GHSA advisory JSON (pip, npm, maven, composer; published 2019 or later)
|
||||
keep those whose CWE maps to a family through `src/openultrasast/model/taxonomy.py` and that link one fix
|
||||
commit with one parent in one repository; attach the OSV first affected version; flag `post_cutoff` when
|
||||
published after 2026-06-01. Repository filters: anonymous HTTPS, at most 500 MB, licence at the pinned head
|
||||
(LICENSE file, then the API's SPDX id) classed `permissive` (the design 1.4 list) or `private`.
|
||||
- `tests/test_unseen_source.py` on recorded fixture JSON (no network): CWE mapping, multi-parent and
|
||||
multi-repository fixes dropped, cutoff flag, licence classing (GPL, AGPL, LGPL, none and source-available go
|
||||
private).
|
||||
- _Requirements: 1.4, 2.1_
|
||||
- [x] 2.2 Vulnerability-introducing range
|
||||
- `benchmarks/unseen/extract.py`: the `introducing` method of design 2.1 (blame of the sink line, or the
|
||||
declaration for an absence bug; absence at V's first parent; no merge or root; at most 50 files and 2,000
|
||||
lines; OSV version bracket when present; walk-back of at most five steps) and the `last_touch` fallback. Each
|
||||
change records method, family, vulnerable function resolved in head, head-side lines, `post_cutoff`, advisory
|
||||
id. Git only; no scanner or engine.
|
||||
- `tests/test_unseen_extract.py` on a synthetic repository built in a temp dir: accepted V; a reformat walked
|
||||
back; a merge rejected; size cap; bracket rejection; fallback chosen when no V passes.
|
||||
- _Requirements: 2.1_
|
||||
- [x] 2.3 Ordinary-commit draw
|
||||
- `extract.py`: first-parent, non-merge commits touching an analysed language; the exclusions of design 2.2
|
||||
(security words of the narrow pattern, reverts, bot and bump commits, manifest- or lock-only, generated or
|
||||
vendored, advisory fix commits, commits touching a known vulnerable function, more than 1,000 source lines,
|
||||
counted). Seeded draw of 12 per repository over three time thirds by four size buckets, proportional, at least
|
||||
one per third; a repository with fewer than 40 eligible commits is dropped.
|
||||
- Tests: each exclusion; the narrow pattern keeps "query", "token", "session"; allocation and the seed
|
||||
reproduce the same draw; the 40-commit floor.
|
||||
- _Requirements: 2.2, 2.3_
|
||||
- [x] 2.4 Draft manifest and slices
|
||||
- `benchmarks/unseen/draft.py`: writes `benchmarks/unseen/draft-p1.toml` (permissive entries by name) and
|
||||
`benchmarks/unseen/private/draft-p1.toml` (the rest); the public file holds an opaque id and
|
||||
sha256(URL, commits) per private entry. Seeded shuffle into three slices of 100 repositories, stratified by
|
||||
ecosystem and `post_cutoff`; each repository has one introducing change and 12 ordinary changes.
|
||||
- Tests: the digest covers private entries; no private URL in the public file; the same seed gives the same
|
||||
slices; per-slice strata within one repository of proportional.
|
||||
- _Requirements: 1.2, 1.4, 2.3, 5.2_
|
||||
- [ ] 2.5 Sourcing run **(coordinator, network)**
|
||||
- `gh api /advisories` enumeration, OSV lookups, canonical resolution against the used set (1.2), blobless
|
||||
clones into a scratch cache, extraction (2.2, 2.3), draft (2.4).
|
||||
- Instrument: clones counted with bytes on disk; a repository whose clone reads zero source files is an
|
||||
instrument failure, not a drop. Fewer than 300 eligible repositories stops the run with counts; the
|
||||
eligibility bars are not relaxed to fill it.
|
||||
- Observable: a draft of 300 repositories (3,900 changes) on disk.
|
||||
- _Requirements: 1.1, 1.4, 2.1, 2.2, 2.3_
|
||||
- [ ] 2.6 Label check and record 02 **(coordinator)**
|
||||
- Every third `introducing` change read by an agent, no model call, marked `confirmed | refactor | unclear`
|
||||
in the draft.
|
||||
- `benchmarks/measurements/<date>-unseen-02-draft/record.json`: candidates per ecosystem, rejections per used
|
||||
source, licence split (public, private), method counts, post-cutoff counts, confirmed share, ordinary-commit
|
||||
exclusion counts. No names.
|
||||
- _Requirements: 2.1, 2.2_
|
||||
|
||||
- [ ] 3. Freeze p1 (design 1.2, 6; increment 3)
|
||||
- [ ] 3.1 Guard bisection tool
|
||||
- `benchmarks/unseen/bisect_guard.py`: rewrites the draft with a candidate subset, runs
|
||||
`.venv/bin/pytest -q -p no:cacheprovider tests/test_independent_population.py -k referenced` with stdout and
|
||||
stderr discarded, reads only the exit code, and bisects until green. Reports the dropped count only.
|
||||
- `tests/test_unseen_bisect.py` with an injected fake guard (a hidden set): finds and drops exactly the hidden
|
||||
members; never logs a candidate; an exit code other than 0 or 1 aborts (instrument failure).
|
||||
- _Requirements: 1.1_
|
||||
- [ ] 3.2 Freeze writer
|
||||
- `benchmarks/unseen/freeze.py`: canonical `pool-p1.toml` (and the private counterpart) from the bisected draft;
|
||||
`freeze-p1.json` with used-set digest and per-source counts, guard result, dropped count, label-check counts
|
||||
and the freeze digest (sha256 of the canonical manifest). Refuses to write over a frozen pool; a correction
|
||||
is `p2`.
|
||||
- Tests: digest stable under key order; overwrite refused; the guard's digest test (1.3) passes on a fixture
|
||||
pool.
|
||||
- _Requirements: 1.2_
|
||||
- [ ] 3.3 Protocol p1
|
||||
- `benchmarks/unseen/protocol-p1.md`: the scoring rules of design section 4 (default flags, decision 4.0's
|
||||
long-deadline arm, catch and location rules, false alarm, coverage states), the statistics of section 3 and
|
||||
7, and the adoption gate of 7.3, stated before any replay.
|
||||
- Observable: the file exists and its sha256 is listed in `freeze-p1.json`.
|
||||
- _Requirements: 3.1, 3.2, 3.3, 3.4, 5.4_
|
||||
- [ ] 3.4 Freeze run and record 03 **(coordinator)**
|
||||
- Run the bisection (3.1), then the freeze (3.2). Guard green, digest test active.
|
||||
- `benchmarks/measurements/<date>-unseen-03-freeze/record.json`: freeze digest, protocol digest, dropped
|
||||
count, final counts per slice and kind.
|
||||
- _Requirements: 1.1, 1.2, 1.3_
|
||||
|
||||
- [ ] 4. Dispatcher (design 5.3; increment 4)
|
||||
- [x] 4.1 Items, workloads and the AX lane
|
||||
- `benchmarks/ax/batch.py`: `Item` (id, input objects, result object name, command, extra env, input digest),
|
||||
`Workload` (image digest, allowed URL env names, deadline, result validator) and the `ax` lane, moved from
|
||||
`benchmarks/learn/engine_trace_ax.py`: manifest render, egress readiness, presigned GET and PUT as `.dat`,
|
||||
apply, poll result object and AX phase, delete, object cleanup. The `kind` lane is the same lane with the
|
||||
kind context.
|
||||
- `tests/test_ax_batch.py` with a fake `ax` CLI and a fake store: manifest holds only the allowed URL env
|
||||
names and no credential; names carry the `ousast-engine-` prefix; cleanup on success and failure.
|
||||
- _Requirements: 4.1, 4.2, 4.5_
|
||||
- [x] 4.2 Scheduler, fallback and resume
|
||||
- `batch.py`: two AX lanes and one VM `docker` lane by default (one container at a time); sandbox death retried
|
||||
once under a new name, a second death or a JVM OOM goes to the VM fallback queue ahead of overflow;
|
||||
`progress.json` written by the scheduler only; a `STOP` file halts launches; resume skips items whose stored
|
||||
record has the same input digest; a cleanup failure stops dispatching.
|
||||
- Tests with fake lanes: retry then fallback; OOM straight to fallback; resume skips; `STOP`; cleanup failure
|
||||
halts; never more than two AX tasks in flight.
|
||||
- _Requirements: 4.3_
|
||||
- [x] 4.3 Engine pass on the dispatcher
|
||||
- `benchmarks/learn/engine_trace_ax.py` and `engine_trace.run_plan` become an engine `Workload` over `batch.py`;
|
||||
no behaviour change.
|
||||
- Observable: `tests/test_ax_executor.py` and `tests/test_engine_trace.py` pass with their assertions unchanged
|
||||
(moved imports only).
|
||||
- _Requirements: 4.3_
|
||||
- [ ] 4.4 Engine smoke on kube-ax and record 04 **(coordinator, network)**
|
||||
- Two pins through the new dispatcher on kube-ax, compared with their stored records.
|
||||
- `benchmarks/measurements/<date>-unseen-04-dispatcher/record.json`: image digest, per-pin wall time and JVM
|
||||
time, record digests equal to the stored ones (or the diff counted), cleanup result.
|
||||
- _Requirements: 4.3_
|
||||
|
||||
- [ ] 5. Replay task (design 5.1, 5.2; increment 5)
|
||||
- [x] 5.1 Image stage
|
||||
- `plane/Dockerfile.engine-task`: second stage `FROM engine AS replay` adding `git`, the package with its
|
||||
runtime dependencies and `benchmarks/unseen/replay_entry.py`; smoke `ousast pre-push --help`. The root image
|
||||
stays `ghcr.io/norandom/ax-task-runner:v0.3.1`; the engine stage is unchanged.
|
||||
- `tests/test_unseen_image.py`: parses the Dockerfile; the `replay` stage builds from `engine`; the root image
|
||||
pin; the engine stage's text is byte-identical to before.
|
||||
- _Requirements: 4.1_
|
||||
- [x] 5.2 Task entry
|
||||
- `benchmarks/unseen/replay_entry.py`, standard library only, reusing `engine_task_entry.transfer`: GET the spec;
|
||||
`git init` and depth-1 fetches of head and base over HTTPS (180 s bound), verified with `git cat-file -e`; run
|
||||
`ousast pre-push` with the spec's hook flags (defaults, or `--deadline 300` for the long-deadline arm), heap
|
||||
1700 MB and the cluster JVM flags; pack `push.json`, `push.txt`, `engine.json`, `instrument.json` (files and
|
||||
bytes in the checkout, bytes of each changed file, JVM wall time, hook exit code, step timings, image
|
||||
digest); PUT as `.dat`. Any error PUTs a failure record naming the step and status, never URL text.
|
||||
- `tests/test_unseen_replay_entry.py`: a local bare repository and a localhost presign stub; success record
|
||||
fields; a missing commit, a failed fetch and a hook crash each give a failure record; zero bytes read and a
|
||||
replay faster than JVM start are flagged `suspect_fast`.
|
||||
- _Requirements: 3.1, 4.1, 4.2_
|
||||
- [x] 5.3 Replay workload
|
||||
- `benchmarks/unseen/replay.py`: builds `Item`s from a frozen slice and an arm (`default` or `long_deadline`);
|
||||
the spec object carries the repository URL so names stay out of AX status and telemetry; task names
|
||||
`ousast-engine-replay-<slug>-<digest>` with an opaque slug; the validator rejects a record without
|
||||
`instrument.json`; records go to `~/ousast-results/unseen/<pool>/<slice>/<arm>/` and the store.
|
||||
- Tests: no repository name in any task name or manifest; only the two presigned URLs in the env; input
|
||||
digest changes with the arm's flags; `instrument_failure` items queued for one re-run.
|
||||
- _Requirements: 3.1, 4.1, 4.2, 4.3_
|
||||
- [ ] 5.4 Proof on kind, then kube-ax, and record 05 **(coordinator, network)**
|
||||
- Build and push the `replay` target, pinned by digest. Five changes from this repository's own history (never
|
||||
pool changes) on kind, then the same five on kube-ax, then on the VM lane.
|
||||
- Exit: instrument fields show bytes read and realistic JVM time on every lane; quick-tier findings equal
|
||||
across lanes.
|
||||
- `benchmarks/measurements/<date>-unseen-05-replay-task/record.json`: image digest, per-lane timings, bytes,
|
||||
exit codes, the quick-tier comparison.
|
||||
- _Requirements: 4.1, 4.2, 4.5_
|
||||
|
||||
- [x] 6. Scorer (design 3, 4, 7.1, 7.3; increment 6)
|
||||
- [x] 6.1 Coverage states
|
||||
- `benchmarks/unseen/score.py`: each record gets one state (`instrument_failure`, `unanalysable`, `quick_only`,
|
||||
`engine_covered`) from `instrument.json` and the artifact; `suspect_fast` goes to failure; a slice above 5%
|
||||
`instrument_failure` after re-runs is refused.
|
||||
- `tests/test_unseen_score.py` on synthetic records: every state; the refusal; a failure never counts as quiet
|
||||
or missed.
|
||||
- _Requirements: 3.3_
|
||||
- [x] 6.2 Catch and false-alarm rules
|
||||
- `score.py`: family match (engine family, quick-rule CWE through `taxonomy.py`), the location table of design
|
||||
4.1, all three streams with the stream named per catch; false alarm on any finding of any stream; the
|
||||
alerts-only rate.
|
||||
- Tests: right family wrong place misses; `access_control` on the handler; `config_secrets` file scope; a
|
||||
quick-rule-only catch; one advisory finding on an ordinary change is a false alarm.
|
||||
- _Requirements: 3.2_
|
||||
- [x] 6.3 Rates, intervals and breakdowns
|
||||
- `score.py`: Wilson 95% intervals; repository-cluster bootstrap for false alarms; ICC; breakdowns per family,
|
||||
method, `post_cutoff`, stream, coverage state and the confirmed subset.
|
||||
- Tests: the n values of design section 3 (315, 483, 756, 62, 88, 97) reproduced; bootstrap seeded; ICC on a
|
||||
synthetic clustered set.
|
||||
- _Requirements: 2.3, 3.3_
|
||||
- [x] 6.4 Budgets, paired tests, the gate and record 06
|
||||
- `score.py`: per-budget thresholds (1%, 5%, 10%), labelled `exploratory` in-slice and `fixed` when supplied;
|
||||
a binary arm's single operating point with the budgets it meets; exact two-sided McNemar for catch and for
|
||||
false alarm; `gate(A, B)` returning `pass | regression | not_run` per design 7.3.
|
||||
- Tests: thresholds on a synthetic score set; McNemar exact values; a binary arm with a significant
|
||||
false-alarm increase gives `regression`; missing slice or excess failures give `not_run`.
|
||||
- `benchmarks/measurements/<date>-unseen-06-scorer/record.json`: test counts and the digest of the scorer's
|
||||
output on the committed synthetic fixture.
|
||||
- _Requirements: 3.4, 5.4_
|
||||
|
||||
- [ ] 7. Baseline (design 4, 6; increment 7; G0 exit)
|
||||
- [ ] 7.1 Frozen source and dry run **(coordinator, network)**
|
||||
- Export HEAD to a frozen tree; build and push the `replay` image from it, pinned by digest.
|
||||
- `benchmarks/push/python_overhead.py` run with the stub engine to size per-replay Python cost; dispatcher dry
|
||||
run on slice 1 lists 1,300 items without launching.
|
||||
- _Requirements: 5.1, 4.1_
|
||||
- [ ] 7.2 Baseline replay of slice 1 **(coordinator, network)**
|
||||
- Default arm, 1,300 replays over two AX lanes and the VM lane (about 9 hours); `instrument_failure` items
|
||||
re-run once; the run stops for a cause above 5% failures.
|
||||
- Observable: 1,300 records in `~/ousast-results/unseen/p1/1/default/` and the store, each with
|
||||
`instrument.json`.
|
||||
- _Requirements: 3.1, 4.3, 5.1_
|
||||
- [ ] 7.3 Adjudication **(coordinator)**
|
||||
- Flagged ordinary changes adjudicated against protocol-v2's standard (all if 30 or fewer, else every third);
|
||||
the adjudicated share kept beside the false-alarm rate. ICC measured; above 0.05 noted as "next pool sizes
|
||||
up".
|
||||
- _Requirements: 3.2, 2.3_
|
||||
- [ ] 7.4 Record 07 and the ledger **(coordinator)**
|
||||
- `benchmarks/measurements/<date>-unseen-07-baseline/record.json`: `score.py` output (counts, rates, Wilson and
|
||||
bootstrap intervals, coverage per state, per family, method, stream), freeze digest, image digest, commit.
|
||||
`usage-p1.jsonl` row `baseline`.
|
||||
- Observable: guard green over the record; G0's exit met.
|
||||
- _Requirements: 5.1, 5.2, 3.3_
|
||||
|
||||
- [ ] 8. Arms on slice 1 (design 4.0, 5.4, 7.1-7.3; increment 8; G1)
|
||||
- [ ] 8.1 Registered long-deadline arm
|
||||
- `plane/experiments/` manifest for the long-deadline arm: hook flags with `--deadline 300`, budget 0,
|
||||
`unseen: {pool: p1, slice: 1}`, committed before its first replay. `src/openultrasast/learn/experiments.py`
|
||||
accepts `unseen` and a hook-flag arm.
|
||||
- `tests/test_learn_experiments.py`: the manifest loads; an `unseen` block without pool or slice is refused.
|
||||
- _Requirements: 3.4, 5.3_
|
||||
- [ ] 8.2 `unseen` block and adoption gate in `analyse`
|
||||
- `experiments.py` `analyse`: writes `unseen` into `result.json` (catch at each budget per arm, paired B - A,
|
||||
McNemar, `gate`) from `score.py`; checks the usage ledger; adoption requires the fold rule and
|
||||
`gate == pass`.
|
||||
- Tests: `regression` and `not_run` each block adoption; an `informed` slice cannot qualify; result JSON shape.
|
||||
- _Requirements: 5.2, 5.3, 5.4_
|
||||
- [ ] 8.3 Engine-evidence arm
|
||||
- `benchmarks/unseen/arms.py`: a score per change from `engine.json` (witness rows on head-side changed
|
||||
functions), no model; plugs into the budget thresholds of 6.4.
|
||||
- Tests on synthetic records: score monotone in witness strength; a change without engine coverage gets no
|
||||
score and is reported, not counted quiet.
|
||||
- _Requirements: 3.4_
|
||||
- [ ] 8.4 Model-arm adapter
|
||||
- `arms.py`: candidates from head-side changed functions of stored records, scored by the current decision
|
||||
engine program (and the combiner when its program exists; else its row reads `not_built`); change score is the
|
||||
maximum; metered client, dry run first, response cache, the manifest's `budget_usd` as the hard ceiling.
|
||||
- Tests with a scripted client: dry-run estimate, ceiling stop, cache replay; no model call.
|
||||
- _Requirements: 4.4, 3.4_
|
||||
- [ ] 8.5 Long-deadline replay of slice 1 **(coordinator, network)**
|
||||
- 1,300 replays of the `long_deadline` arm, bounded at about 345 s each (up to about 42 hours over three
|
||||
lanes), with `STOP` and resume across sessions; failures re-run once.
|
||||
- Observable: 1,300 records under `~/ousast-results/unseen/p1/1/long_deadline/`, JVM time in each.
|
||||
- _Requirements: 3.1, 3.4, 4.3_
|
||||
- [ ] 8.6 Model arms on slice 1 **(coordinator, network; paid: needs a budget go-ahead)**
|
||||
- Dry run gives the candidate count; estimate at the injection slice's measured cost (about $0.008 per
|
||||
candidate at k = 5); the maintainer sets the ceiling against the balance read just before. Then the decision
|
||||
engine arm (and the combiner if built) on the VM.
|
||||
- _Requirements: 4.4, 3.4_
|
||||
- [ ] 8.7 Record 08: the arms table **(coordinator)**
|
||||
- `benchmarks/measurements/<date>-unseen-08-arms/record.json`: per arm (hook 30 s, hook 300 s, engine evidence,
|
||||
decision engine, combiner or `not_built`) catch at 1/5/10% budgets with Wilson intervals and realised
|
||||
false-alarm rates, coverage per state, spend; `exploratory` rows in `usage-p1.jsonl`.
|
||||
- Observable: one committed table of arms on slice 1; G1's table exists.
|
||||
- _Requirements: 3.4, 5.2, 5.3_
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- 2026-10-04: Tasks 1.1–1.4 implemented offline and independently reviewed. Task 1.5 remains for the
|
||||
coordinator; no networked measurement or commit was performed. Command and API contracts:
|
||||
`benchmarks/unseen/README.md`. Offline evidence:
|
||||
`benchmarks/measurements/2026-10-04-unseen-01-offline/record.json`.
|
||||
- Population guard scope is preserved under the maintainer's hard rule: benchmarks/src/tests for
|
||||
populations, plus plane for pools. Each directory exemption is reservation-specific, including private
|
||||
pool manifests. Extending population scope to plane exposed 30 existing references; no historical
|
||||
records were changed or exempted inside the existing population scope.
|
||||
- S3 inventory opens through `open_store(read_only=True)` because ordinary store construction writes a
|
||||
Select capability probe. Default store behavior is unchanged; inventory uses LIST/GET only.
|
||||
- The freeze writer must match the canonical TOML digest encoding documented in the pool README and
|
||||
exercised by `test_independent_population.py`; final pool discovery does not trust mutable status.
|
||||
|
||||
### Group 2 implementation evidence (2026-10-04)
|
||||
|
||||
Tasks 2.1–2.4 are implemented offline and independently reviewed. Task 2.5 is
|
||||
`python -m benchmarks.unseen.draft --root "$PWD" --s3-store s3:// --seed 20260601`
|
||||
with the documented PATH/PYTHONPATH and GH_TOKEN environment. No networked draw,
|
||||
label check, group-2 measurement record, guard bisection, or commit was performed.
|
||||
The group header remains pending its coordinator exits 2.5/2.6.
|
||||
|
||||
The command rebuilds the group-1 used set, resolves renamed/fork repositories,
|
||||
uses disposable external blobless clones, re-reads licences at extracted heads,
|
||||
and emits counts/digests only. The README documents injected clients, method bias,
|
||||
the hard 300-repository floor, and an explicitly unmeasured resource estimate.
|
||||
Independent review caught and fixed ambiguous parent-function absence and stale
|
||||
candidate-404 handling; regression tests cover both.
|
||||
|
||||
Validation: requested pytest selection 103 passed, 1 skipped, exit 0 (population-test
|
||||
output discarded); `ruff check .` exit 0; `ruff format --check .` exit 0 (411 files);
|
||||
`mypy src/openultrasast` exit 0 (160 source files). The CLI `--help` smoke exited 0.
|
||||
Source-read instrumentation printed 20,101 bytes for the extraction implementation.
|
||||
|
||||
### Groups 4–6 offline implementation (2026-10-04)
|
||||
|
||||
Tasks 4.1–4.3, 5.1–5.3 and 6.1–6.4 are implemented and independently reviewed.
|
||||
Tasks 4.4 and 5.4 remain coordinator-owned network proofs; group headers 4 and 5 stay
|
||||
pending those exits. No commit, network request, Docker, AX or kubectl command was run.
|
||||
The sourcing job's draft.py, source.py, extract.py and journal.py are unchanged.
|
||||
|
||||
The shared dispatcher retains the engine adapter's historical questions.json object
|
||||
name to preserve its unchanged assertions; replay uses spec.dat and result.dat.
|
||||
The engine Dockerfile body is unchanged except its required AS engine alias; CI builds
|
||||
engine and replay separately. S3Store's read_only class default preserves existing
|
||||
constructor-bypassing transport fixtures; initialized read-only stores still refuse writes.
|
||||
|
||||
Exact coordinator commands and input contracts are in benchmarks/unseen/README.md,
|
||||
including frozen TOML/private pointer verification, scoring joins, own-history proof
|
||||
preparation and anonymous measurement-record generation. JVM wall time is the hook's
|
||||
aggregate scan build/query timing and is labelled as such. Unsupported-only partitions
|
||||
with explicit language_not_covered evidence do not pretend to have launched a JVM.
|
||||
|
||||
Record 06 contains the committed synthetic fixture and scorer-output digests. The
|
||||
reserved-repository guard was run as a black box with both output streams discarded.
|
||||
Independent review: APPROVED; all reported integration and coverage defects resolved.
|
||||
|
||||
Final verification: requested pytest command 279 passed, 8 skipped, exit 0; separate
|
||||
`tests/test_ax_batch.py` 5 passed, exit 0; `ruff check .`, `ruff format --check .`,
|
||||
`mypy src/openultrasast`, and the output-discarded reserved-repository guard all exit 0.
|
||||
The own-history proof preparer and replay dry run read five items and exit 0.
|
||||
@@ -1,8 +1,23 @@
|
||||
# Overview — what this is, and what is actually true of it
|
||||
|
||||
## Current state, 2026-10-02
|
||||
## Current state, 2026-10-04
|
||||
|
||||
Everything below this section is dated history; where it disagrees with this section, this section wins. The
|
||||
direction lives in `roadmap.md`, section of 2026-10-04 (phases G0-G5).
|
||||
|
||||
- **Goal.** A pre-push safety net that works on repositories it has never seen. Two numbers per push decide it:
|
||||
catch rate on real vulnerability-introducing changes, and false-alarm rate on ordinary changes. Neither has been
|
||||
measured yet; the unseen-repository pool (G0, spec `unseen-repo-evaluation`) comes first.
|
||||
- **Correction, 2026-10-04.** Infrastructure and engine work had drifted from that goal. It is now done only when
|
||||
it removes a bottleneck for those measurements.
|
||||
- **Engine.** Joern runs on the operator's kube-ax cluster as AX Tasks, with a VM fallback lane; kind stays the
|
||||
development profile. It is almost blind to fixes: paths differ between sides in 2 of 105 both-asked pairs.
|
||||
- **Decision engine.** Ranks but cannot block (exp-003 to exp-005 inconclusive; 710 fix pairs in memory). Fix
|
||||
mechanisms is one G2 arm, paused after mining until the pool exists.
|
||||
- **Hook.** `ousast pre-push` is operational and advisory on any repository; rollout stays NO-GO.
|
||||
|
||||
## State, 2026-10-02 (superseded by the section above)
|
||||
|
||||
Everything below this section is dated history; where it disagrees with this section, this section wins.
|
||||
User-facing documentation: `README.md`, `docs/architecture.md`, `docs/evaluation.md`, `docs/decision-engine.md`,
|
||||
`ops/ax/README.md`.
|
||||
|
||||
|
||||
@@ -1,5 +1,100 @@
|
||||
# Roadmap
|
||||
|
||||
## Roadmap to detection with the decision engine on kube-ax (2026-10-04)
|
||||
|
||||
This section supersedes the ordering below it. The older sections stay as history.
|
||||
|
||||
### Destination
|
||||
|
||||
On every push, from a hook or CI, OpenUltraSAST answers for the changed code. The answer is advisory for every
|
||||
family, and BLOCK only for families whose precision has been proven on repositories the system never saw. The work
|
||||
is split by what each place is good at:
|
||||
|
||||
- **kube-ax cluster** (operator k3s, Agent Substrate v0.3.0, AX v0.3.1, 2 nodes): the Joern engine, as AX Tasks.
|
||||
No API keys and no store credentials in the cluster; tasks read and write through short-lived presigned links.
|
||||
- **The VM or CI runner**: candidate selection, the decision engine's model calls (DeepSeek), the memory store,
|
||||
and the fallback engine lane (2.5 GB heap) for pins that do not fit the cluster's 1.7 GB heap.
|
||||
- **kind** stays the local development profile, with the same code and manifests.
|
||||
|
||||
### The question everything answers
|
||||
|
||||
Does the safety net work on repositories it has never seen? That is the purpose of the A/B experiments, and every
|
||||
phase below is judged by it. On an unseen repository almost every push is an ordinary change, so two numbers decide
|
||||
whether the safety net is useful, measured on the hook's advisory output exactly as a user would see it:
|
||||
|
||||
1. **Catch rate:** on real vulnerability-introducing changes in unseen repositories, how often it flags the change.
|
||||
2. **False-alarm rate:** on ordinary changes in the same unseen repositories, how often it flags anything.
|
||||
|
||||
Within-pair AUC on known fix pairs answers neither. It stays a development signal, not the goal.
|
||||
|
||||
### Where we are (2026-10-04)
|
||||
|
||||
| Area | State | Evidence |
|
||||
| --- | --- | --- |
|
||||
| Generalisation measured so far | The classifier ranks above chance on repository-held-out folds (within-pair AUC about 0.76) but cannot reach the block bar; the engine-based scan failed fresh populations (v1 1/11, v2 0/17); push-level catch and false-alarm rates on unseen repositories have never been measured | exp-003/004/005, `results-v1.json`, `results-v2.json` |
|
||||
| Pre-push hook | Operational and advisory on any repository | `2026-10-02-pre-push-operational-1/record.json` |
|
||||
| Labelled data | 710 complete real fix pairs | `2026-10-04-decision-engine-harvest-3/record.json` |
|
||||
| Engine | Almost blind to fixes (paths differ between sides in 2 of 105 both-asked pairs) | `2026-10-04-joern-slice-leak-audit/record.json` |
|
||||
| Infrastructure | Engine runs on kube-ax as AX Tasks with a VM fallback; kind for development | memory `kube-ax-cluster` |
|
||||
|
||||
Correction recorded 2026-10-04: the previous two days went mostly into infrastructure and engine internals that did
|
||||
not move a generalisation number. From here, infrastructure and engine work is done only when it removes a
|
||||
bottleneck for the measurements below.
|
||||
|
||||
### Phases
|
||||
|
||||
**G0. The unseen-repository pool and the push-level metric** (new spec `unseen-repo-evaluation`)
|
||||
|
||||
1. A frozen pool of repositories never used for training, tuning, mining or harvesting, and not in any population.
|
||||
Two kinds of change per repository: real vulnerability-introducing commits (the parent of a fix, post-cutoff where
|
||||
possible) and ordinary commits from the same history.
|
||||
2. The metric: replay each change through `ousast pre-push` (and the decision engine where an arm uses it) and
|
||||
record catch rate and false-alarm rate, with confidence intervals, per family and overall.
|
||||
3. Rotation: the pool is spent per decision, like the populations; a new slice is drawn for the next one.
|
||||
|
||||
Exit: the pool is frozen and the current production hook has a measured baseline on it.
|
||||
|
||||
**G1. Re-score the arms we have on the pool**
|
||||
|
||||
Current hook (quick rules + engine), decision-engine advisory (current programs), combiner, engine evidence. Each is
|
||||
an arm with a registered decision rule, scored by catch rate at a fixed false-alarm budget.
|
||||
|
||||
Exit: one measured table of arms on unseen code; keep only what generalises.
|
||||
|
||||
**G2. Improve what moves the unseen-repo numbers**
|
||||
|
||||
Candidates, each admitted only if it improves the pool metric on a fresh slice: fix mechanisms in the engine
|
||||
(approved plan, now one arm, not the spine), engine coverage and TypeScript, decision-engine prompt and retrieval
|
||||
changes, more labelled pairs from more distinct repositories.
|
||||
|
||||
**G3. Decide BLOCK per family** on the pool's false-alarm rate and the pooled precision bound (Req 6.4).
|
||||
|
||||
**G4. Detection on push for any repository** (`plane-on-kubernetes` P4 work: `ousast plane scan --base --head`,
|
||||
engine pins on kube-ax, model calls on the VM or CI), sized by what G1-G3 kept.
|
||||
|
||||
**G5. Qualification and rollout:** adopt, commit `prediction-v3.json`, spend population v3 once, advisory rollout,
|
||||
BLOCK only for families that passed.
|
||||
|
||||
### Cross-cutting rules
|
||||
|
||||
- No hand-tuning per miss. Every detection change is a registered experiment or an admission by held-out
|
||||
separation.
|
||||
- Records carry counts, digests and pattern names, never repository names: the reserved-repository guard covers
|
||||
every population.
|
||||
- Instrument first: a zero from an unread input is a failed run (AGENTS.md).
|
||||
- Cluster stability: sandbox deaths are retried once, then sent to the VM lane; the operator owns workers, egress
|
||||
and Substrate.
|
||||
|
||||
### Spec map
|
||||
|
||||
| Spec | Role in this roadmap | State |
|
||||
| --- | --- | --- |
|
||||
| `unseen-repo-evaluation` (new) | G0, G1: the pool and the push-level metric | To specify |
|
||||
| `learned-decision-engine` | G1-G3, G5; fix-mechanisms plan is one G2 arm | Implementation; experiments 3-5 recorded |
|
||||
| `plane-on-kubernetes` | G4; the engine executors for kube-ax | Groups 1-2 done (kind); production path re-scoped to the operator's kube-ax (AX Tasks), groups 3-8 to re-plan |
|
||||
| `pre-push-safety-net` | the hook replayed in G0-G1; M4-M5 in G5 | Operational increment 1 done; M4-M5 open |
|
||||
| `ai-service-plane` | The plane the above runs on | Complete |
|
||||
|
||||
## Status, 2026-10-02
|
||||
|
||||
The milestone table below dates from 2026-09-14. Since then (`.kiro/specs/pre-push-safety-net/release-milestones.md`):
|
||||
|
||||
+1
-1
@@ -52,7 +52,7 @@ COPY benchmarks ./benchmarks
|
||||
# The core install is zero-dependency by design; `semantic` adds the tree-sitter grammars the candidate
|
||||
# enumerator uses. The LLM endpoint stays optional and is configured by environment, never baked in.
|
||||
RUN python3 -m venv /venv \
|
||||
&& /venv/bin/pip install --no-cache-dir -e ".[semantic]"
|
||||
&& /venv/bin/pip install --no-cache-dir -e ".[semantic,s3]"
|
||||
ENV PATH="/venv/bin:${PATH}"
|
||||
|
||||
# Analysis never needs root.
|
||||
|
||||
@@ -363,3 +363,81 @@ own terms:
|
||||
- Joern (Apache-2.0) is downloaded at image build time.
|
||||
- The tree-sitter wheels of the `semantic` extra are MIT.
|
||||
- The published benchmark cases were licence-checked one by one. v1's OpenCVE case is BUSL-1.1, see [RELEASE_NOTES.md](RELEASE_NOTES.md).
|
||||
|
||||
## Engine traces on kube-ax
|
||||
|
||||
`benchmarks/learn/engine_trace.py --executor ax` submits AX v0.3.1 Tasks with
|
||||
`ax apply -f -` and removes them with `ax delete task NAME`. The operator supplies
|
||||
kubeconfig and AX access; the dispatcher creates no Kubernetes resources or secrets.
|
||||
The `ousast-engine-task` GHCR package **must be made Public once** in its package
|
||||
settings so the actors can pull it without registry credentials.
|
||||
|
||||
The task image contains the current engine and worker code. Rebuild it after code
|
||||
changes and pass its digest with `--image`; no analyzer download is needed. The host
|
||||
materializes each pin as usual and uploads source and prepared questions under
|
||||
`engine-queue/<run>/<pin>/`. Only object-scoped presigned URLs and deadline parameters
|
||||
enter Task env. Credentials stay on the coordinator. Configure `OUSAST_MEMORY=s3://…`
|
||||
and the store credentials in the coordinator's `.env` or environment, with
|
||||
`S3_ENDPOINT=https://files.because-security.com`. Presigned links expire after the
|
||||
pin deadline plus ten minutes. IP and HTTP endpoints are rejected: the operator's
|
||||
actor EgressPolicy must allow **files.because-security.com:443**.
|
||||
|
||||
Completion is the result object, even while AX reports Running. Failed phases and
|
||||
host deadlines produce explicit failure/timeout rows. Task and staging-object cleanup
|
||||
runs on success and failure; cleanup failure stops further dispatch. Per-pin files,
|
||||
`--rerun-status`, resume and `progress.json` retain the existing trace-runner behavior.
|
||||
AX results include `executor: "ax"`, `worker_ip`, and the image digest. Placement is
|
||||
random; `--parallel` defaults to two AX tasks and cannot exceed two.
|
||||
|
||||
AX workers use `--heap-profile cluster` with
|
||||
`-Xms1700m -Xmx1700m -XX:MaxMetaspaceSize=256m -XX:ReservedCodeCacheSize=128m -XX:MaxDirectMemorySize=256m -Xss512k -XX:ActiveProcessorCount=1 -XX:+UseG1GC`
|
||||
on every frontend/query JVM; instruments record the flags. A `JVM out of memory`
|
||||
result is automatically re-queued to one local Docker lane with `--heap-profile vm`
|
||||
(2560m), recording `fallback: "vm-oom"` and the AX attempt. `mixed-ax` also uses this
|
||||
one Docker lane for overflow, prioritizing pending OOM fallbacks. Thus a mixed run has
|
||||
up to two AX tasks plus one Docker container. Use `--docker-image` for the locally
|
||||
available VM image; Docker never pulls it implicitly. A STOP file prevents new
|
||||
launches; interrupted fallback queues are recovered on resume.
|
||||
|
||||
Coordinator commands, from this worktree (the AX binary and kubeconfig remain
|
||||
configurable). Build/push the actual uncommitted tree and read its digest:
|
||||
|
||||
```bash
|
||||
cd /home/mc/Source/OpenUltraSAST/.claude/worktrees/engine-ax
|
||||
export PATH=/home/mc/Source/OpenUltraSAST/.venv/bin:$PATH
|
||||
export PYTHONPATH="$PWD/src"
|
||||
TASK_TAG="ghcr.io/norandom/ousast-engine-task:sha-$(git rev-parse --short=7 HEAD)"
|
||||
docker buildx build --platform linux/amd64 -f plane/Dockerfile.engine-task \
|
||||
-t "$TASK_TAG" -t ghcr.io/norandom/ousast-engine-task:main \
|
||||
--push --metadata-file /tmp/ousast-engine-task-build.json .
|
||||
export TASK_IMAGE="ghcr.io/norandom/ousast-engine-task@$(python -c 'import json; print(json.load(open("/tmp/ousast-engine-task-build.json"))["containerimage.digest"])')"
|
||||
printf '%s\n' "$TASK_IMAGE"
|
||||
docker build -f Dockerfile -t openultrasast:engine-ax .
|
||||
```
|
||||
|
||||
The Engine image workflow also builds/pushes both images and prints each digest;
|
||||
CI needs the changes published before dispatch. The local build above includes
|
||||
uncommitted changes without publishing a git commit.
|
||||
|
||||
With `KUBECONFIG` pointing to the operator's kube-ax config and store settings in
|
||||
`.env`, run a one-pin smoke using the default experiment units and local example
|
||||
memory (or add `--examples /path/to/frozen-examples.jsonl`):
|
||||
|
||||
```bash
|
||||
: "${KUBECONFIG:?Set KUBECONFIG to the operator kube-ax configuration}"
|
||||
python benchmarks/learn/engine_trace.py --executor ax \
|
||||
--ax-bin "$HOME/.local/ax-v0.3.1/ax" --kubeconfig "$KUBECONFIG" \
|
||||
--atespace default --image "$TASK_IMAGE" --docker-image openultrasast:engine-ax \
|
||||
--parallel 1 --limit 1 --deadline 900 \
|
||||
--out "$HOME/ousast-results/plane/engine-ax-smoke"
|
||||
```
|
||||
|
||||
Then run the mixed lanes (repeat to resume; defaults re-run failed/timeout pins):
|
||||
|
||||
```bash
|
||||
python benchmarks/learn/engine_trace.py --executor mixed-ax \
|
||||
--ax-bin "$HOME/.local/ax-v0.3.1/ax" --kubeconfig "$KUBECONFIG" \
|
||||
--atespace default --image "$TASK_IMAGE" --docker-image openultrasast:engine-ax \
|
||||
--parallel 2 --deadline 900 --rerun-status failed,timeout \
|
||||
--out "$HOME/ousast-results/plane/engine-ax-mixed"
|
||||
```
|
||||
|
||||
@@ -0,0 +1,434 @@
|
||||
"""Host-only AX Task transport; credentials never enter task manifests."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import ipaddress
|
||||
import json
|
||||
import os
|
||||
import queue
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
from collections import Counter, deque
|
||||
from concurrent.futures import FIRST_COMPLETED, ThreadPoolExecutor, wait
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import timedelta
|
||||
from pathlib import Path
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
import yaml
|
||||
|
||||
from openultrasast.config import load_dotenv
|
||||
from openultrasast.plane.memory import PRESIGNED_ARCHIVE_SUFFIX, S3Store, open_store
|
||||
|
||||
|
||||
class EgressError(ValueError):
|
||||
"""Safe diagnostic without presigned URL contents."""
|
||||
|
||||
|
||||
class SandboxFailure(Exception):
|
||||
"""A terminal AX task without an authoritative result object."""
|
||||
|
||||
|
||||
def public_endpoint(url):
|
||||
"""Reject endpoints the actor's hostname-based HTTPS egress policy cannot reach."""
|
||||
parts = urlsplit(url)
|
||||
host = parts.hostname or ""
|
||||
try:
|
||||
ipaddress.ip_address(host)
|
||||
is_ip = True
|
||||
except ValueError:
|
||||
is_ip = False
|
||||
if (
|
||||
parts.scheme != "https"
|
||||
or not host
|
||||
or is_ip
|
||||
or "." not in host
|
||||
or host.endswith((".localhost", ".local", ".internal"))
|
||||
or parts.port not in (None, 443)
|
||||
or parts.username
|
||||
or parts.password
|
||||
):
|
||||
raise EgressError(
|
||||
"AX requires a public HTTPS DNS hostname in S3_ENDPOINT and presigned URLs: "
|
||||
"the egress policy allows the store hostname on port 443 only (no IP or plain HTTP)"
|
||||
)
|
||||
return host.lower()
|
||||
|
||||
|
||||
def task_state(text, name):
|
||||
"""Accept AX YAML/JSON output and its human-readable tasks table."""
|
||||
try:
|
||||
doc = yaml.safe_load(text)
|
||||
except yaml.YAMLError:
|
||||
doc = None
|
||||
rows = doc.get("items", doc.get("tasks", [doc])) if isinstance(doc, dict) else doc if isinstance(doc, list) else []
|
||||
for row in rows:
|
||||
if not isinstance(row, dict) or row.get("metadata", {}).get("name", row.get("name")) != name:
|
||||
continue
|
||||
status = row.get("status", {})
|
||||
ip = next(
|
||||
(status.get(key) or row.get(key) for key in ("workerIP", "workerIp", "worker_ip", "ip") if status.get(key) or row.get(key)),
|
||||
None,
|
||||
)
|
||||
worker = status.get("worker")
|
||||
if not ip and isinstance(worker, dict):
|
||||
ip = worker.get("ip") or worker.get("address")
|
||||
return str(status.get("phase", row.get("phase", ""))), ip
|
||||
lines = [line.split() for line in text.splitlines() if line.strip()]
|
||||
if lines:
|
||||
headers = [re.sub(r"[^a-z]", "", cell.lower()) for cell in lines[0]]
|
||||
for cells in lines[1:]:
|
||||
if name not in cells:
|
||||
continue
|
||||
row = dict(zip(headers, cells, strict=False))
|
||||
phase = row.get("phase", row.get("status", ""))
|
||||
ip = row.get("workerip") or row.get("ip")
|
||||
if not ip:
|
||||
for cell in cells:
|
||||
try:
|
||||
ipaddress.ip_address(cell)
|
||||
ip = cell
|
||||
break
|
||||
except ValueError:
|
||||
pass
|
||||
return phase, None if ip in {"<none>", "-", ""} else ip
|
||||
listed = isinstance(doc, (dict, list)) or bool(lines and "name" in headers and {"phase", "status"}.intersection(headers))
|
||||
return "Missing" if listed else "", None
|
||||
|
||||
|
||||
def bounded_read(read, timeout):
|
||||
"""Do not let SDK retries postpone task teardown beyond the pin deadline.
|
||||
|
||||
Only the read runs in a daemon thread; it cannot publish a late result to disk.
|
||||
No executor shutdown waits for an SDK call whose socket has stopped responding.
|
||||
"""
|
||||
if timeout <= 0:
|
||||
raise TimeoutError("AX result deadline")
|
||||
replies = queue.Queue(maxsize=1)
|
||||
|
||||
def fetch():
|
||||
try:
|
||||
replies.put((True, read()))
|
||||
except Exception as exc:
|
||||
replies.put((False, exc))
|
||||
|
||||
threading.Thread(target=fetch, daemon=True, name="engine-ax-result-read").start()
|
||||
try:
|
||||
ok, result = replies.get(timeout=timeout)
|
||||
except queue.Empty:
|
||||
raise TimeoutError("AX result deadline") from None
|
||||
if not ok:
|
||||
raise result
|
||||
return result
|
||||
|
||||
|
||||
class AXLane:
|
||||
def __init__(self, args, workload, store=None, pause=time.sleep, clock=time.monotonic):
|
||||
self.workload = workload
|
||||
self.args = args
|
||||
self.pause, self.clock = pause, clock
|
||||
self.run = uuid.uuid4().hex
|
||||
load_dotenv()
|
||||
self.host = public_endpoint(os.environ.get("S3_ENDPOINT", ""))
|
||||
# Validate the image before opening a store, whose constructor probes the bucket.
|
||||
validate_image(workload.image)
|
||||
self.store = store if store is not None else open_store()
|
||||
if store is None and not isinstance(self.store, S3Store):
|
||||
raise ValueError("AX requires OUSAST_MEMORY=s3://bucket[/prefix]")
|
||||
|
||||
def __call__(self, item, output, attempt):
|
||||
attempts = []
|
||||
try:
|
||||
done, worker = self._execute_attempt(item, output, attempt, attempts)
|
||||
except SandboxFailure:
|
||||
return {"status": "sandbox_failure", "ax_attempts": attempts}
|
||||
if done is None or done.returncode:
|
||||
return {"status": "instrument_failure", "ax_attempts": attempts}
|
||||
record = json.loads((output / "result.json").read_text())
|
||||
return {**record, "worker_ip": worker, "ax_attempts": attempts}
|
||||
|
||||
def ax(self, *args, manifest=None, timeout=30):
|
||||
env = dict(os.environ)
|
||||
if self.args.kubeconfig:
|
||||
env["KUBECONFIG"] = str(Path(self.args.kubeconfig).expanduser())
|
||||
done = subprocess.run(
|
||||
[str(Path(self.args.ax_bin).expanduser()), *args, *(["--atespace", self.args.atespace] if args[0] != "apply" else [])],
|
||||
input=json.dumps(manifest) if manifest is not None else None,
|
||||
env=env,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=max(0.01, timeout),
|
||||
)
|
||||
if done.returncode:
|
||||
# AX errors may echo manifest URLs; report only verb and exit code.
|
||||
raise ValueError(f"ax {args[0]} failed (exit {done.returncode})")
|
||||
return done
|
||||
|
||||
def _execute_attempt(self, item, output, attempt, attempts):
|
||||
pin_id = item.id
|
||||
digest = hashlib.sha256(f"{self.run}/{pin_id}/attempt-{attempt}".encode()).hexdigest()[:12]
|
||||
slug = re.sub(r"[^a-z0-9]+", "-", str(pin_id).lower()).strip("-")[:22]
|
||||
name = f"ousast-engine-{slug or 'pin'}-{digest}"
|
||||
key_id = hashlib.sha256(str(pin_id).encode()).hexdigest()
|
||||
prefix = f"engine-queue/{self.run}/{key_id}/attempt-{attempt}/"
|
||||
keys = [prefix + item.object_names.get(key, key.lower() + PRESIGNED_ARCHIVE_SUFFIX) for key in item.inputs]
|
||||
keys.append(prefix + item.result_object)
|
||||
expires = timedelta(seconds=self.workload.deadline + 600)
|
||||
worker_ip = None
|
||||
submitted = False
|
||||
started = self.clock()
|
||||
phase = ""
|
||||
try:
|
||||
for key, data in zip(keys, item.inputs.values(), strict=False):
|
||||
self.store._put(key, data)
|
||||
urls = {env: self.store.presign_get(key, expires) for env, key in zip(item.inputs, keys, strict=False)}
|
||||
urls["RESULT_URL"] = self.store.presign_put(keys[-1], expires)
|
||||
if any(public_endpoint(url) != self.host for url in urls.values()):
|
||||
raise EgressError("AX presign hostname must match S3_ENDPOINT for the store egress policy")
|
||||
document = manifest(name, self.workload, item, urls, self.args.atespace)
|
||||
end = self.clock() + self.workload.deadline
|
||||
# An apply timeout can still have created the task; cleanup must cover it.
|
||||
submitted = True
|
||||
self.ax("apply", "-f", "-", manifest=document, timeout=min(30, self.workload.deadline))
|
||||
while self.clock() < end:
|
||||
result = bounded_read(lambda: self.store._get(keys[-1]), end - self.clock())
|
||||
if self.clock() >= end:
|
||||
return None, worker_ip
|
||||
phase = ""
|
||||
try:
|
||||
state = self.ax("get", "tasks", timeout=min(30, end - self.clock())).stdout
|
||||
phase, ip = task_state(state, name)
|
||||
worker_ip = ip or worker_ip
|
||||
except (ValueError, OSError, subprocess.SubprocessError):
|
||||
# The object is authoritative even when AX status is temporarily unavailable.
|
||||
pass
|
||||
if self.clock() >= end:
|
||||
return None, worker_ip
|
||||
if result is not None:
|
||||
self.workload.validate(result[0], output)
|
||||
return subprocess.CompletedProcess([], 0, "", ""), worker_ip
|
||||
if phase.lower() in {"failed", "missing"}:
|
||||
# The worker may have uploaded while we were reading AX status.
|
||||
result = bounded_read(lambda: self.store._get(keys[-1]), end - self.clock())
|
||||
if self.clock() >= end:
|
||||
return None, worker_ip
|
||||
if result is not None:
|
||||
self.workload.validate(result[0], output)
|
||||
return subprocess.CompletedProcess([], 0, "", ""), worker_ip
|
||||
raise SandboxFailure
|
||||
self.pause(min(2, max(0, end - self.clock())))
|
||||
return None, worker_ip
|
||||
except (subprocess.TimeoutExpired, TimeoutError):
|
||||
return None, worker_ip
|
||||
except SandboxFailure:
|
||||
raise
|
||||
except Exception as exc:
|
||||
# Never expose exception strings which may contain object-scoped bearer URLs.
|
||||
detail = str(exc) if isinstance(exc, EgressError) else type(exc).__name__
|
||||
return subprocess.CompletedProcess([], 1, "", "AX transport failed: " + detail), worker_ip
|
||||
finally:
|
||||
attempts.append({"name": name, "phase": phase, "seconds": self.clock() - started})
|
||||
errors = []
|
||||
if submitted:
|
||||
try:
|
||||
self.ax("delete", "task", name)
|
||||
except Exception as exc:
|
||||
# A successful list already confirmed a disappeared task is gone.
|
||||
if phase != "Missing":
|
||||
errors.append("task: " + type(exc).__name__)
|
||||
for key in keys:
|
||||
try:
|
||||
self.store._delete(key)
|
||||
except Exception as exc:
|
||||
errors.append("object: " + type(exc).__name__)
|
||||
if errors:
|
||||
raise RuntimeError("AX cleanup failed; stop dispatching: " + ", ".join(errors))
|
||||
|
||||
|
||||
@dataclass
|
||||
class Item:
|
||||
id: str
|
||||
inputs: dict[str, bytes]
|
||||
result_object: str
|
||||
command: tuple[str, ...]
|
||||
extra_env: dict[str, str]
|
||||
input_digest: str
|
||||
object_names: dict[str, str] = field(default_factory=dict)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Workload:
|
||||
image: str
|
||||
url_env: frozenset[str]
|
||||
deadline: float
|
||||
validate: object
|
||||
|
||||
|
||||
def validate_image(image):
|
||||
if not re.fullmatch(r"[^\s@]+@sha256:[0-9a-f]{64}", image):
|
||||
raise ValueError("AX --image must be pinned by sha256 digest")
|
||||
|
||||
|
||||
def manifest(name, workload, item, urls, atespace="default", *, validate_name=True):
|
||||
validate_image(workload.image)
|
||||
if validate_name and (not re.fullmatch(r"ousast-engine-[a-z0-9-]+", name) or len(name.encode()) > 49):
|
||||
raise ValueError("invalid AX task name")
|
||||
if set(urls) != workload.url_env:
|
||||
raise ValueError("unexpected URL environment names")
|
||||
# Extra env is deliberately a small numeric execution contract, never inherited credentials.
|
||||
if set(item.extra_env) - {"DEADLINE", "QUESTION_DEADLINE"} or any(
|
||||
not re.fullmatch(r"[0-9]+(?:\.[0-9]+)?", str(v)) for v in item.extra_env.values()
|
||||
):
|
||||
raise ValueError("unexpected task environment")
|
||||
for url in urls.values():
|
||||
public_endpoint(url)
|
||||
env = [{"name": key, "value": str(value)} for key, value in {**urls, **item.extra_env}.items()]
|
||||
if len(json.dumps(env).encode()) >= 32768:
|
||||
raise ValueError("AX Task env exceeds the 32 KB limit")
|
||||
return {
|
||||
"apiVersion": "ax.io/v1alpha1",
|
||||
"kind": "Task",
|
||||
"metadata": {"name": name, "atespace": atespace},
|
||||
"spec": {"image": workload.image, "env": env, "command": list(item.command)},
|
||||
}
|
||||
|
||||
|
||||
def write_json(path, record):
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
temporary = path.with_suffix(".tmp")
|
||||
temporary.write_text(json.dumps(record, sort_keys=True) + "\n")
|
||||
temporary.replace(path)
|
||||
|
||||
|
||||
def schedule(pending, lanes, out, execute, complete, *, fallbacks=(), overflow=True, progress=None):
|
||||
"""The sole progress writer; callbacks carry workload-specific records, never transport."""
|
||||
if sum(lane in {"ax", "kind"} for lane in lanes) > 2 or lanes.count("docker") > 1:
|
||||
raise ValueError("at most two AX lanes and one VM lane")
|
||||
pending, fallbacks = deque(pending), deque(fallbacks)
|
||||
progress = progress if progress is not None else {"pins_done": 0, "pins_total": len(pending) + len(fallbacks)}
|
||||
progress["lanes"] = dict(Counter(lanes))
|
||||
active = {}
|
||||
failures = False
|
||||
|
||||
def persist():
|
||||
progress.update(pins_in_flight=len(active), active_lanes=dict(Counter(lane for _, lane in active.values())))
|
||||
if any(lane in {"ax", "kind"} for lane in lanes):
|
||||
progress["fallbacks_pending"] = len(fallbacks)
|
||||
write_json(out / "progress.json", progress)
|
||||
|
||||
with ThreadPoolExecutor(max_workers=len(lanes)) as pool:
|
||||
|
||||
def launch(lane):
|
||||
if (out / "STOP").exists():
|
||||
return
|
||||
if lane == "docker" and fallbacks:
|
||||
item, previous = fallbacks.popleft()
|
||||
elif pending and (lane != "docker" or overflow):
|
||||
item, previous = pending.popleft(), None
|
||||
else:
|
||||
return
|
||||
active[pool.submit(execute, item, lane, previous)] = item, lane
|
||||
|
||||
for lane in lanes:
|
||||
launch(lane)
|
||||
persist()
|
||||
try:
|
||||
while active:
|
||||
finished, _ = wait(active, return_when=FIRST_COMPLETED)
|
||||
# Resolve ALL completed futures before launching; cleanup exceptions stop launches.
|
||||
results = [(active.pop(future), future.result()) for future in finished]
|
||||
for (item, lane), result in results:
|
||||
failed, fallback = complete(item, lane, result, progress)
|
||||
if fallback is not None:
|
||||
fallbacks.append((item, fallback))
|
||||
else:
|
||||
failures |= failed
|
||||
progress["pins_done"] += 1
|
||||
for lane, count in (Counter(lanes) - Counter(lane for _, lane in active.values())).items():
|
||||
for _ in range(count):
|
||||
launch(lane)
|
||||
persist()
|
||||
finally:
|
||||
persist()
|
||||
return failures
|
||||
|
||||
|
||||
def dispatch(items, workload, out, executors, *, lanes=("ax", "ax", "docker"), overflow=True):
|
||||
"""Digest checkpoints, one instrument rerun, sandbox retry, priority VM fallback."""
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
pending, fallbacks = [], []
|
||||
|
||||
def path(item):
|
||||
return out / (hashlib.sha256(item.id.encode()).hexdigest() + ".json")
|
||||
|
||||
for item in items:
|
||||
previous = json.loads(path(item).read_text()) if path(item).exists() else {}
|
||||
if previous.get("input_digest") == item.input_digest:
|
||||
if previous.get("fallback_pending"):
|
||||
fallbacks.append((item, previous))
|
||||
continue
|
||||
if previous.get("done"):
|
||||
continue
|
||||
pending.append(item)
|
||||
|
||||
def execute(item, lane, previous):
|
||||
output = out / hashlib.sha256(item.id.encode()).hexdigest()
|
||||
output.mkdir(exist_ok=True)
|
||||
attempts = []
|
||||
for attempt in range(2):
|
||||
attempt_output = output / ("attempt-" + uuid.uuid4().hex)
|
||||
attempt_output.mkdir()
|
||||
record = executors[lane](item, attempt_output, attempt)
|
||||
for artifact in attempt_output.iterdir():
|
||||
if artifact.is_file():
|
||||
shutil.copyfile(artifact, output / artifact.name)
|
||||
attempts.append({"lane": lane, "status": record.get("status"), "attempt": attempt, "transport": record.get("ax_attempts", [])})
|
||||
if record.get("status") not in {"sandbox_failure", "instrument_failure"}:
|
||||
break
|
||||
return {**record, "attempts": (previous or {}).get("attempts", []) + attempts, "input_digest": item.input_digest, "done": True}
|
||||
|
||||
def complete(item, lane, record, progress):
|
||||
fallback = lane in {"ax", "kind"} and record.get("status") in {"sandbox_failure", "oom"} and "docker" in lanes
|
||||
record["fallback_pending"] = fallback
|
||||
write_json(path(item), record)
|
||||
progress["last_item"] = hashlib.sha256(item.id.encode()).hexdigest()
|
||||
return record.get("status") not in {"ok", "unanalysable", "quick_only", "engine_covered"}, record if fallback else None
|
||||
|
||||
return schedule(pending, lanes, out, execute, complete, fallbacks=fallbacks, overflow=overflow)
|
||||
|
||||
|
||||
class DockerLane(AXLane):
|
||||
"""VM lane uses the same presigned-object protocol and pinned replay image."""
|
||||
|
||||
def _execute_attempt(self, item, output, attempt, attempts):
|
||||
# Reuse all object/validation/cleanup logic, replacing only the task runner.
|
||||
import copy
|
||||
|
||||
lane = copy.copy(self)
|
||||
lane.ax = lane._docker_command
|
||||
lane._docker_name = None
|
||||
return AXLane._execute_attempt(lane, item, output, attempt, attempts)
|
||||
|
||||
def _docker_command(self, *args, manifest=None, timeout=30):
|
||||
if args[0] == "apply":
|
||||
self._docker_name = manifest["metadata"]["name"]
|
||||
command = ["docker", "run", "-d", "--name", self._docker_name, "--entrypoint", manifest["spec"]["command"][0]]
|
||||
for env in manifest["spec"]["env"]:
|
||||
command += ["-e", env["name"] + "=" + env["value"]]
|
||||
command += [manifest["spec"]["image"], *manifest["spec"]["command"][1:], "--heap-profile", "vm"]
|
||||
elif args[0] == "get":
|
||||
command = ["docker", "inspect", "--format", "{{.State.Running}} {{.State.ExitCode}}", self._docker_name]
|
||||
else:
|
||||
command = ["docker", "rm", "-f", self._docker_name or args[2]]
|
||||
done = subprocess.run(command, capture_output=True, text=True, timeout=timeout)
|
||||
if done.returncode:
|
||||
raise ValueError("docker command failed")
|
||||
if args[0] == "get":
|
||||
phase = "Running" if done.stdout.startswith("true") else "Failed"
|
||||
done.stdout = json.dumps({"metadata": {"name": self._docker_name}, "status": {"phase": phase}})
|
||||
return done
|
||||
@@ -0,0 +1,173 @@
|
||||
"""AX task entry: object-scoped transfers and stdlib only; result PUT is completion."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import http.client
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
class TaskEntryError(RuntimeError):
|
||||
"""A task-entry diagnostic containing only locally controlled text."""
|
||||
|
||||
|
||||
def transfer(request, destination=None, *, readiness_budget=None):
|
||||
"""Wait for actor egress on startup; bound later transfers to three attempts."""
|
||||
deadline = time.monotonic() + readiness_budget if readiness_budget is not None else None
|
||||
delay = 0.5
|
||||
attempt = 0
|
||||
while True:
|
||||
remaining = deadline - time.monotonic() if deadline is not None else 60
|
||||
if remaining <= 0:
|
||||
raise TaskEntryError("egress not open after 60 s")
|
||||
attempt += 1
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=min(60, remaining)) as response:
|
||||
if destination is not None:
|
||||
with destination.open("wb") as output:
|
||||
shutil.copyfileobj(response, output)
|
||||
return
|
||||
except urllib.error.HTTPError as exc:
|
||||
exc.close()
|
||||
if exc.code != 403 and not 500 <= exc.code < 600:
|
||||
raise
|
||||
if deadline is None and attempt >= 3:
|
||||
raise
|
||||
except (urllib.error.URLError, OSError, http.client.HTTPException):
|
||||
if deadline is None and attempt >= 3:
|
||||
raise
|
||||
if deadline is not None:
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
raise TaskEntryError("egress not open after 60 s") from None
|
||||
time.sleep(min(delay, remaining))
|
||||
delay = min(delay * 2, 8)
|
||||
else:
|
||||
time.sleep(attempt)
|
||||
|
||||
|
||||
def download(url, destination, *, readiness_budget=None):
|
||||
transfer(url, destination, readiness_budget=readiness_budget)
|
||||
|
||||
|
||||
def unpack(source, destination):
|
||||
"""Reject links and traversal; source archives contain only regular files/directories."""
|
||||
with tarfile.open(source) as archive:
|
||||
for member in archive.getmembers():
|
||||
target = (destination / member.name).resolve()
|
||||
if not target.is_relative_to(destination.resolve()) or not (member.isfile() or member.isdir()):
|
||||
raise ValueError("unsafe source archive member")
|
||||
archive.extractall(destination, filter="data")
|
||||
|
||||
|
||||
def failure_record(pin, reason):
|
||||
return {
|
||||
**pin,
|
||||
"done": True,
|
||||
"status": "failed",
|
||||
"reason": reason,
|
||||
"units": [
|
||||
{
|
||||
**unit,
|
||||
"status": "failed" if unit.get("supported", True) else "unsupported",
|
||||
"reason": reason,
|
||||
"instrument": {"files": [], "bytes": 0, "jvm": []},
|
||||
"questions": {},
|
||||
"questions_asked": [],
|
||||
"questions_completed": [],
|
||||
"witness_rows": [],
|
||||
"traces": [],
|
||||
}
|
||||
for unit in pin.get("units", [])
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def main(*, work_root=Path("/tmp"), worker_script=None, readiness_budget=60):
|
||||
pin = {}
|
||||
failed = False
|
||||
output = work_root / "out"
|
||||
result = output / "result.json"
|
||||
step = "prepare output"
|
||||
try:
|
||||
output.mkdir(parents=True, exist_ok=True)
|
||||
result.unlink(missing_ok=True)
|
||||
questions = work_root / "questions.json"
|
||||
step = "GET questions"
|
||||
download(os.environ["QUESTIONS_URL"], questions, readiness_budget=readiness_budget)
|
||||
step = "parse questions"
|
||||
pin = json.loads(questions.read_text())
|
||||
if not isinstance(pin, dict):
|
||||
pin = {}
|
||||
raise ValueError("questions object must contain a prepared pin")
|
||||
source = work_root / "source.tar"
|
||||
step = "GET source"
|
||||
download(os.environ["SOURCE_URL"], source)
|
||||
step = "unpack source"
|
||||
case = work_root / "case"
|
||||
case.mkdir(parents=True, exist_ok=True)
|
||||
unpack(source, case)
|
||||
step = "configure worker"
|
||||
deadline = int(os.environ.get("DEADLINE", "900"))
|
||||
command = [
|
||||
sys.executable,
|
||||
str(worker_script or Path(__file__).with_name("engine_trace_worker.py")),
|
||||
"--worker",
|
||||
str(questions),
|
||||
"--root",
|
||||
str(case),
|
||||
"--output",
|
||||
str(result),
|
||||
"--heap-profile",
|
||||
"cluster",
|
||||
"--deadline",
|
||||
str(deadline),
|
||||
"--question-deadline",
|
||||
os.environ.get("QUESTION_DEADLINE", "120"),
|
||||
]
|
||||
step = "run worker"
|
||||
completed = subprocess.run(command, timeout=deadline + 5, check=False)
|
||||
if completed.returncode:
|
||||
raise TaskEntryError(f"worker exited {completed.returncode}")
|
||||
step = "read worker result"
|
||||
if not result.is_file() or not json.loads(result.read_text()).get("done"):
|
||||
raise TaskEntryError("worker did not write a completed result")
|
||||
except Exception as exc:
|
||||
failed = True
|
||||
# Avoid including exception text containing a presigned URL in durable output.
|
||||
if isinstance(exc, urllib.error.HTTPError):
|
||||
detail = f"HTTP {exc.code}"
|
||||
else:
|
||||
detail = str(exc) if isinstance(exc, TaskEntryError) else type(exc).__name__
|
||||
reason = f"{step}: {detail}"
|
||||
try:
|
||||
output.mkdir(parents=True, exist_ok=True)
|
||||
result.write_text(json.dumps(failure_record(pin, reason)) + "\n")
|
||||
except Exception:
|
||||
print("task entry could not write failure result", file=sys.stderr)
|
||||
step = "pack result"
|
||||
try:
|
||||
archive = work_root / "result.tar"
|
||||
with tarfile.open(archive, "w") as tar:
|
||||
for path in sorted(output.iterdir()):
|
||||
tar.add(path, arcname=path.name)
|
||||
data = archive.read_bytes()
|
||||
step = "PUT result"
|
||||
request = urllib.request.Request(os.environ["RESULT_URL"], data=data, method="PUT", headers={"Content-Type": "application/x-tar"})
|
||||
transfer(request, readiness_budget=readiness_budget if failed else None)
|
||||
except Exception as exc:
|
||||
detail = f"HTTP {exc.code}" if isinstance(exc, urllib.error.HTTPError) else type(exc).__name__
|
||||
print(f"{step}: {detail}; dispatcher deadline remains authoritative", file=sys.stderr)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,985 @@
|
||||
"""Resumable labelled-function traces with Docker and Kubernetes execution lanes.
|
||||
|
||||
Pair blob identities resolve through catalogs to real source commits. The analyzer
|
||||
is git archive HEAD src plus a hashed snapshot of this uncommitted tool.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import ast
|
||||
import copy
|
||||
import fcntl
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import textwrap
|
||||
import threading
|
||||
import time
|
||||
import tomllib
|
||||
from collections import Counter, defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
from engine_trace_k8s import IMAGE, Kubernetes
|
||||
from engine_trace_queue import Dispatcher
|
||||
from engine_trace_worker import EXCLUDED_DIRS, safe_path, unit_record, write_json
|
||||
|
||||
from openultrasast.learn.examples import read_jsonl
|
||||
from openultrasast.learn.labels import load_sources, repo_name
|
||||
from openultrasast.model.specs import taint_specs
|
||||
from openultrasast.model.taint import request_params
|
||||
from openultrasast.pairs import _materialize_side, load_pair_catalog
|
||||
from openultrasast.plane.engine_alerts import export_pin
|
||||
from openultrasast.plane.memory import FileStore
|
||||
from openultrasast.plane.tasks.alerts import engine_languages
|
||||
from openultrasast.preprocess import detect_language
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
from benchmarks.ax.batch import schedule
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
DEFAULT_OUT = Path.home() / "ousast-results/plane/engine-trace"
|
||||
SNAPSHOT_FILES = (
|
||||
"benchmarks/learn/engine_trace.py",
|
||||
"benchmarks/learn/engine_trace_worker.py",
|
||||
"benchmarks/learn/engine_trace_k8s.py",
|
||||
"benchmarks/learn/engine_trace_queue.py",
|
||||
"benchmarks/learn/engine_trace_ax.py",
|
||||
"benchmarks/ax/batch.py",
|
||||
"benchmarks/learn/engine_task_entry.py",
|
||||
"benchmarks/learn/engine_worker_service.py",
|
||||
"benchmarks/learn/engine_trace_parse.py",
|
||||
"src/openultrasast/model/taint.py",
|
||||
"src/openultrasast/cpg/queries/taint.sc",
|
||||
)
|
||||
MATERIALIZATION_VERSION = 3
|
||||
EXPORT_CAP = 20 * 1024 * 1024
|
||||
|
||||
STATUSES = ("path", "asked-nothing", "failed", "timeout", "unsupported")
|
||||
|
||||
|
||||
def digest(value):
|
||||
return hashlib.sha256(json.dumps(value, sort_keys=True).encode()).hexdigest()
|
||||
|
||||
|
||||
def pin_name(pin):
|
||||
return digest([pin["repo"], pin["pin"]]) + ".json"
|
||||
|
||||
|
||||
def join_units(units, examples, include_typescript=False):
|
||||
index = {row["id"]: row for row in examples}
|
||||
missing = [u["unit"] for u in units if u["unit"] not in index]
|
||||
if missing:
|
||||
raise ValueError(f"example snapshot missing {len(missing)}/{len(units)} unit IDs; supply --examples JSONL; first: {missing[0]}")
|
||||
covered = engine_languages() - {"typescript"}
|
||||
if include_typescript:
|
||||
covered |= {"typescript"}
|
||||
result = []
|
||||
for unit in units:
|
||||
example = index[unit["unit"]]
|
||||
if any(example[key] != unit[key] for key in ("family", "label", "group")):
|
||||
raise ValueError(f"unit/example identity mismatch: {unit['unit']}")
|
||||
file, sep, function = example["candidate"].partition("::")
|
||||
if not sep or not function:
|
||||
raise ValueError(f"unit has no labelled function: {unit['unit']}")
|
||||
language = detect_language(Path(file)) or example.get("language", "other")
|
||||
model_language = "javascript" if language == "typescript" else language
|
||||
spec = taint_specs(language=model_language).get(unit["family"]) if language in covered else None
|
||||
result.append(
|
||||
{
|
||||
**unit,
|
||||
"repo": example["repo"],
|
||||
"pin": example["pin"],
|
||||
"file": file,
|
||||
"function": function,
|
||||
"language": language,
|
||||
"source": example["source"],
|
||||
"excerpt": example.get("excerpt", example["source"] == "pairs" or example["source"].startswith("advisory-fixes")),
|
||||
"pin_role": example.get("pin_role", ""),
|
||||
"supported": spec is not None,
|
||||
"reason": "" if spec else f"no admitted taint model: {language}/{unit['family']}",
|
||||
}
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def make_plan(
|
||||
units,
|
||||
out,
|
||||
only_family=None,
|
||||
limit=0,
|
||||
*,
|
||||
include_typescript=False,
|
||||
only_source=(),
|
||||
rerun_status=("failed",),
|
||||
allow_excerpt_fallback=False,
|
||||
):
|
||||
grouped = defaultdict(list)
|
||||
for unit in units:
|
||||
if (not only_family or unit["family"] == only_family) and (not only_source or unit["source"] in only_source):
|
||||
# Include admission options even when a unit's support happens to stay
|
||||
# unchanged. Old results cannot mark a different selection as complete.
|
||||
identity = {
|
||||
"unit": unit,
|
||||
"include_typescript": include_typescript,
|
||||
"materialization_version": MATERIALIZATION_VERSION,
|
||||
"allow_excerpt_fallback": allow_excerpt_fallback,
|
||||
}
|
||||
grouped[unit["repo"], unit["pin"]].append({**unit, "input_digest": digest(identity)})
|
||||
plan = []
|
||||
for (repo, pin), rows in sorted(grouped.items()):
|
||||
item = {
|
||||
"repo": repo,
|
||||
"pin": pin,
|
||||
"materialization_version": MATERIALIZATION_VERSION,
|
||||
"units": sorted(rows, key=lambda r: r["unit"]),
|
||||
}
|
||||
item["selection"] = digest(item["units"])
|
||||
done = out / pin_name(item)
|
||||
if done.exists():
|
||||
old = json.loads(done.read_text())
|
||||
recorded = {r["unit"]: r.get("input_digest") for r in old.get("units", [])}
|
||||
if (
|
||||
old.get("done")
|
||||
and not old.get("fallback_pending")
|
||||
and old.get("materialization_version") == MATERIALIZATION_VERSION
|
||||
and not any(r.get("status") in rerun_status for r in old.get("units", []) if r["unit"] in {u["unit"] for u in rows})
|
||||
and all(recorded.get(r["unit"]) == r["input_digest"] for r in rows)
|
||||
):
|
||||
continue
|
||||
sources = {r["source"] for r in rows}
|
||||
# Inventory medians: harvest sides 31s, advisory sides 47s; v1 pins 182s, v2 pins 498s.
|
||||
item["estimated_seconds"] = (
|
||||
(47 if any(s.startswith("advisory") for s in sources) else 31)
|
||||
if any(r.get("excerpt") for r in rows) or any(s.startswith("advisory") for s in sources)
|
||||
else (182 if sources == {"population-v1"} else 498)
|
||||
)
|
||||
if not any(r["supported"] for r in rows):
|
||||
item["estimated_seconds"] = 0
|
||||
plan.append(item)
|
||||
# limit counts runnable pins, not unsupported records.
|
||||
if limit:
|
||||
chosen = []
|
||||
count = 0
|
||||
for pin in plan:
|
||||
runnable = any(r["supported"] for r in pin["units"])
|
||||
if runnable and count >= limit:
|
||||
continue
|
||||
chosen.append(pin)
|
||||
count += runnable
|
||||
return chosen
|
||||
return plan
|
||||
|
||||
|
||||
class Inputs:
|
||||
"""Resolve spent corpora; fetch pair source commits into the host cache."""
|
||||
|
||||
def __init__(self, root, cache, allow_excerpt_fallback=False):
|
||||
self.cache = cache
|
||||
self.allow_excerpt_fallback = allow_excerpt_fallback
|
||||
self.catalog_rows = defaultdict(list)
|
||||
for catalog in (root / "benchmarks/pairs").rglob("catalog.toml"):
|
||||
for item in tomllib.loads(catalog.read_text()).get("pair", []):
|
||||
self.catalog_rows[item["name"]].append((catalog.parent.name, item))
|
||||
self.pairs = []
|
||||
self.repos = defaultdict(list)
|
||||
sources = load_sources()
|
||||
for source in sources.sources:
|
||||
if source.kind == "pairs":
|
||||
self.pairs.extend(load_pair_catalog(root / source.files[0]))
|
||||
elif source.kind == "population":
|
||||
if source.id not in {"population-v1", "population-v2"}:
|
||||
continue
|
||||
data = tomllib.loads((root / source.files[0]).read_text())
|
||||
for case in data.get("case", []):
|
||||
self.repos[repo_name(case["repo"])].append((cache / "independent" / case["id"], case))
|
||||
elif source.kind == "recipes":
|
||||
for file in source.files:
|
||||
data = tomllib.loads((root / file).read_text())
|
||||
recipe = data.get("repo", data)
|
||||
# Recipes carry a top-level URL and a pinned checkout cache.
|
||||
if isinstance(recipe, dict):
|
||||
self.repos[repo_name(str(recipe.get("url", "")))].append((cache / "repos" / str(data.get("name", "")), data))
|
||||
self.hashes = {}
|
||||
|
||||
def pair_for(self, row):
|
||||
if row["pin_role"] not in {"fixed", "vulnerable", "vuln"}:
|
||||
raise ValueError(f"ambiguous pair side: {row['pin_role']}")
|
||||
side = "fixed" if row["pin_role"] == "fixed" else "vuln"
|
||||
candidates = [
|
||||
c for c in self.pairs if c.relpath == row["file"] and repo_name(c.repo or f"{c.slice}:{c.name}") == repo_name(row["repo"])
|
||||
]
|
||||
matches = []
|
||||
for case in candidates:
|
||||
path = case.fixed_file if side == "fixed" else case.vuln_file
|
||||
if path.is_file():
|
||||
if path not in self.hashes:
|
||||
data = path.read_bytes()
|
||||
# Harvest hashes decoded text, so universal newline conversion is part
|
||||
# of its identity (one recorded ThreatByte excerpt uses CRLF).
|
||||
normalized = path.read_text().encode("utf-8")
|
||||
self.hashes[path] = {
|
||||
hashlib.sha1(b"blob " + str(len(body)).encode() + b"\0" + body).hexdigest() for body in (data, normalized)
|
||||
}
|
||||
if row["pin"] in self.hashes[path]:
|
||||
matches.append(case)
|
||||
if len(matches) == 1:
|
||||
return matches[0], side
|
||||
if len(candidates) == 1:
|
||||
# Earlier stores used label_pin rather than the excerpt digest.
|
||||
from openultrasast.learn.examples import label_pin
|
||||
|
||||
case = candidates[0]
|
||||
if row["pin"] == label_pin({"source": "pairs", "source_ref": case.name, "pin_role": row["pin_role"]}):
|
||||
return case, side
|
||||
raise ValueError(f"cannot uniquely resolve excerpt {row['repo']}@{row['pin']}:{row['file']}")
|
||||
|
||||
def catalog_for(self, case):
|
||||
matches = [
|
||||
(source, item)
|
||||
for source, item in self.catalog_rows[case.name]
|
||||
if item.get("relpath") == case.relpath and repo_name(item.get("repo", "")) == repo_name(case.repo)
|
||||
]
|
||||
if len(matches) != 1:
|
||||
raise ValueError(f"ambiguous catalog source for {case.name}")
|
||||
return matches[0]
|
||||
|
||||
def source_identity(self, case, side):
|
||||
_, item = self.catalog_for(case)
|
||||
parent, commit = item.get("parent", ""), item.get("commit", "")
|
||||
if side not in {"vuln", "fixed"} or not all(re.fullmatch(r"[0-9a-fA-F]{40}", sha) for sha in (parent, commit)) or parent == commit:
|
||||
raise ValueError(f"ambiguous catalog commits for {case.name}: need distinct parent and commit SHAs")
|
||||
repo = item.get("repo", "")
|
||||
if not repo:
|
||||
raise ValueError(f"missing catalog repository for {case.name}")
|
||||
return repo, parent if side == "vuln" else commit
|
||||
|
||||
def fetch_source(self, repo, commit, roots):
|
||||
clone = self.cache / "trace-sources" / digest([repo, commit, roots])
|
||||
clone.mkdir(parents=True, exist_ok=True)
|
||||
url = repo if repo.startswith("https://") else f"https://github.com/{repo.removesuffix('.git')}.git"
|
||||
env = {**os.environ, "GIT_TERMINAL_PROMPT": "0"}
|
||||
env.pop("GIT_NO_LAZY_FETCH", None) # sparse checkout may fetch the selected blobs
|
||||
|
||||
def git(*args):
|
||||
return subprocess.run(["git", "-C", str(clone), *args], env=env, capture_output=True, text=True, check=True, timeout=300)
|
||||
|
||||
git("init")
|
||||
git("config", "remote.origin.url", url)
|
||||
git("config", "remote.origin.promisor", "true")
|
||||
git("config", "remote.origin.partialclonefilter", "blob:none")
|
||||
git("fetch", "--depth", "1", "--filter=blob:none", url, commit)
|
||||
git("sparse-checkout", "init", "--cone")
|
||||
git("sparse-checkout", "set", "--", *roots)
|
||||
git("checkout", "--detach", "--force", commit)
|
||||
actual = git("rev-parse", "HEAD").stdout.strip()
|
||||
if actual != commit:
|
||||
raise ValueError(f"fetched commit mismatch: {actual} != {commit}")
|
||||
return clone
|
||||
|
||||
def export_local(self, clone, pin, target):
|
||||
with tempfile.TemporaryDirectory(prefix="trace-export-", dir=target.parent) as scratch:
|
||||
export_pin(clone, pin["pin"], Path(scratch))
|
||||
roots = sorted({source_root(r["file"], r["language"]) for r in pin["units"]})
|
||||
proof = export_sources(Path(scratch), target, pin["units"], roots)
|
||||
return {**proof, "mode": "source", "commit": pin["pin"]}
|
||||
|
||||
def materialize(self, pin, target):
|
||||
rows = pin["units"]
|
||||
for row in rows:
|
||||
safe_path(target, row["file"])
|
||||
if row.get("mapping_error"):
|
||||
raise ValueError(row["mapping_error"])
|
||||
if any(r.get("excerpt") or r["source"].startswith("advisory-fixes") for r in rows):
|
||||
resolved = [self.pair_for(row) for row in rows]
|
||||
identities = set()
|
||||
catalog_files = []
|
||||
for case, side in resolved:
|
||||
_, item = self.catalog_for(case)
|
||||
parent, commit = item.get("parent", ""), item.get("commit", "")
|
||||
if parent != commit and all(re.fullmatch(r"[0-9a-fA-F]{40}", sha) for sha in (parent, commit)):
|
||||
identities.add(self.source_identity(case, side))
|
||||
else:
|
||||
file = case.fixed_file if side == "fixed" else case.vuln_file
|
||||
data = file.read_text()
|
||||
if is_catalog_excerpt(data):
|
||||
data = parseable_excerpt(file, data)
|
||||
mode = "excerpt-parseable"
|
||||
else:
|
||||
# Validate BOTH twins before treating this as a complete-file pair.
|
||||
for twin in (case.vuln_file, case.fixed_file):
|
||||
verify_catalog_file(twin)
|
||||
mode = "catalog-file"
|
||||
catalog_files.append((case, side, mode, data))
|
||||
if catalog_files:
|
||||
if identities:
|
||||
raise ValueError("mixed source and catalog-file materialization at one pair pin")
|
||||
for case, side, mode, data in catalog_files:
|
||||
_materialize_side(target, case, side=side)
|
||||
if mode == "excerpt-parseable":
|
||||
safe_path(target, case.relpath).write_text(data)
|
||||
verify_functions(target, rows)
|
||||
modes = {case.relpath: mode for case, _, mode, _ in catalog_files}
|
||||
for row in rows:
|
||||
if modes[row["file"]] == "excerpt-parseable":
|
||||
row["evidence_scope"] = "in-function-only"
|
||||
return {
|
||||
"mode": "excerpt-parseable" if "excerpt-parseable" in modes.values() else "catalog-file",
|
||||
"files": sum(p.is_file() for p in target.rglob("*")),
|
||||
"bytes": sum(p.stat().st_size for p in target.rglob("*") if p.is_file()),
|
||||
"roots": sorted({source_root(r["file"], r["language"]) for r in rows}),
|
||||
}
|
||||
if len(identities) != 1:
|
||||
raise ValueError("ambiguous source commits at one pair pin")
|
||||
repo, commit = identities.pop()
|
||||
roots = sorted({source_root(row["file"], row["language"]) for row in rows})
|
||||
try:
|
||||
clone = self.fetch_source(repo, commit, roots)
|
||||
proof = export_sources(clone, target, rows, roots)
|
||||
except (OSError, ValueError, subprocess.SubprocessError) as exc:
|
||||
if not self.allow_excerpt_fallback:
|
||||
raise
|
||||
shutil.rmtree(target, ignore_errors=True)
|
||||
for case, side in resolved:
|
||||
_materialize_side(target, case, side=side)
|
||||
return {"mode": "excerpt-fallback", "reason": str(exc), "repo": repo, "commit": commit}
|
||||
verify_functions(target, rows)
|
||||
return {**proof, "mode": "source", "repo": repo, "commit": commit}
|
||||
if all(r["source"] == "dev-php" for r in rows):
|
||||
# Export the exact requested commit from the local recipe checkout.
|
||||
for recipe in (ROOT / "benchmarks/repos").glob("*.toml"):
|
||||
data = tomllib.loads(recipe.read_text())
|
||||
if repo_name(str(data.get("url", ""))) == repo_name(pin["repo"]):
|
||||
checkout = self.cache / "repos" / data["name"] / pin["pin"][:12]
|
||||
return self.export_local(checkout, pin, target)
|
||||
for clone, _ in self.repos[repo_name(pin["repo"])]:
|
||||
check = subprocess.run(["git", "-C", str(clone), "cat-file", "-e", pin["pin"] + "^{commit}"], capture_output=True)
|
||||
if check.returncode == 0:
|
||||
return self.export_local(clone, pin, target)
|
||||
raise ValueError(f"no local clone contains {pin['repo']}@{pin['pin']}")
|
||||
|
||||
|
||||
def is_catalog_excerpt(data):
|
||||
# Older Python exports have this header but no upstream_start field.
|
||||
return re.match(r"\s*(?://|#)\s*Provenance:", data) is not None
|
||||
|
||||
|
||||
def parseable_excerpt(path, data):
|
||||
"""Dedent the code independently of its unindented provenance comments."""
|
||||
language = detect_language(path)
|
||||
lines = data.splitlines(keepends=True)
|
||||
start = 0
|
||||
while start < len(lines) and (not lines[start].strip() or lines[start].lstrip().startswith(("#", "//"))):
|
||||
start += 1
|
||||
header = "".join(lines[:start])
|
||||
if language == "python":
|
||||
header = re.sub(r"(?m)^(\s*)//", r"\1#", header)
|
||||
data = header + textwrap.dedent("".join(lines[start:]))
|
||||
if not "".join(lines[start:]).strip():
|
||||
raise ValueError(f"excerpt does not parse: empty code ({path.name})")
|
||||
if language == "python":
|
||||
try:
|
||||
ast.parse(data, filename=str(path))
|
||||
except SyntaxError as exc:
|
||||
raise ValueError(f"excerpt does not parse: {exc}") from exc
|
||||
else:
|
||||
from openultrasast.semantic.cst import parse_with_cst
|
||||
|
||||
parsed = parse_with_cst(str(path), data, language)
|
||||
if parsed is None:
|
||||
raise ValueError(f"excerpt does not parse: no cheap parser for {language} ({path.name})")
|
||||
if not parsed.parse_ok:
|
||||
raise ValueError(f"excerpt does not parse: {language} {parsed.reason} ({path.name})")
|
||||
return data
|
||||
|
||||
|
||||
def verify_catalog_file(path):
|
||||
"""Admit complete source only; syntactically valid provenance excerpts still fail."""
|
||||
data = path.read_text()
|
||||
if is_catalog_excerpt(data):
|
||||
raise ValueError(f"catalog side is an upstream excerpt without distinct commits: {path.name}")
|
||||
language = detect_language(path)
|
||||
if not data.strip():
|
||||
raise ValueError(f"catalog side is empty: {path.name}")
|
||||
if language == "python":
|
||||
try:
|
||||
ast.parse(data, filename=str(path))
|
||||
except SyntaxError as exc:
|
||||
raise ValueError(f"catalog side is not complete Python source: {path.name}: {exc.msg}") from exc
|
||||
else:
|
||||
from openultrasast.semantic.extra import grammar_for
|
||||
|
||||
grammar = grammar_for(language)
|
||||
if grammar is None:
|
||||
raise ValueError(f"cannot verify complete catalog source for {language}: {path.name}")
|
||||
from tree_sitter import Parser
|
||||
|
||||
if Parser(grammar).parse(data.encode()).root_node.has_error:
|
||||
raise ValueError(f"catalog side is not complete {language} source: {path.name}")
|
||||
|
||||
|
||||
def source_root(file, language):
|
||||
parts = Path(file).parts
|
||||
if language == "java":
|
||||
for i in range(len(parts) - 2):
|
||||
if parts[i : i + 3] == ("src", "main", "java"):
|
||||
return "/".join(parts[: i + 3])
|
||||
# Monorepo and source-layout containers are not themselves a package.
|
||||
if len(parts) > 2 and parts[0] in {"packages", "apps", "src", "web"}:
|
||||
return "/".join(parts[:2])
|
||||
return parts[0] if len(parts) > 1 else "."
|
||||
|
||||
|
||||
def local_imports(source, rows):
|
||||
"""Cheap one-level Python imports from labelled files, without importing code."""
|
||||
found = set()
|
||||
for row in rows:
|
||||
if row["language"] != "python":
|
||||
continue
|
||||
file = safe_path(source, row["file"])
|
||||
try:
|
||||
tree = ast.parse(file.read_bytes())
|
||||
except (SyntaxError, ValueError):
|
||||
continue
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.Import):
|
||||
modules, bases = [alias.name for alias in node.names], [source, source / "src"]
|
||||
elif isinstance(node, ast.ImportFrom):
|
||||
modules = [node.module or ""]
|
||||
modules += [".".join(filter(None, (node.module, alias.name))) for alias in node.names if alias.name != "*"]
|
||||
bases = [file.parents[node.level - 1]] if node.level and node.level <= len(file.parents) else [source, source / "src"]
|
||||
else:
|
||||
continue
|
||||
for base in bases:
|
||||
for module in modules:
|
||||
path = base.joinpath(*module.split("."))
|
||||
for candidate in (path.with_suffix(".py"), path / "__init__.py"):
|
||||
if candidate.is_relative_to(source) and candidate.is_file() and not candidate.is_symlink():
|
||||
rel = candidate.relative_to(source).as_posix()
|
||||
if not any(part in EXCLUDED_DIRS for part in Path(rel).parts):
|
||||
found.add(rel)
|
||||
return found
|
||||
|
||||
|
||||
def export_sources(source, target, rows, roots=(".",), cap=EXPORT_CAP):
|
||||
"""Copy labels intact, narrowing oversized roots before bounding context bytes."""
|
||||
labelled = {r["file"] for r in rows}
|
||||
languages = {r["language"] for r in rows}
|
||||
files = set()
|
||||
protected = {parent.as_posix() for rel in labelled for parent in Path(rel).parents}
|
||||
for directory, dirs, names in os.walk(source):
|
||||
dirs[:] = sorted(
|
||||
d
|
||||
for d in dirs
|
||||
if d != ".git"
|
||||
and not (Path(directory) / d).is_symlink()
|
||||
and (d not in EXCLUDED_DIRS or (Path(directory) / d).relative_to(source).as_posix() in protected)
|
||||
)
|
||||
for name in names:
|
||||
file = Path(directory) / name
|
||||
rel = file.relative_to(source).as_posix()
|
||||
excluded = any(part in EXCLUDED_DIRS for part in Path(rel).parts[:-1])
|
||||
if not file.is_symlink() and (
|
||||
rel in labelled
|
||||
or (
|
||||
not excluded
|
||||
and detect_language(file) in languages
|
||||
and any(root == "." or rel == root or rel.startswith(root + "/") for root in roots)
|
||||
)
|
||||
):
|
||||
files.add(rel)
|
||||
missing = labelled - files
|
||||
if missing:
|
||||
raise ValueError(f"labelled files missing or excluded from export: {sorted(missing)}")
|
||||
imports = local_imports(source, rows)
|
||||
files.update(imports)
|
||||
sizes = {rel: safe_path(source, rel).stat().st_size for rel in files}
|
||||
original_roots = sorted(roots)
|
||||
roots = original_roots
|
||||
# Avoid lexicographically truncating a whole large package such as vllm.
|
||||
if sum(sizes.values()) > cap:
|
||||
roots = sorted({str(Path(rel).parent) for rel in labelled})
|
||||
files = {
|
||||
rel for rel in files if rel in labelled or rel in imports or any(root == "." or rel.startswith(root + "/") for root in roots)
|
||||
}
|
||||
proof = {
|
||||
"bytes": 0,
|
||||
"files": 0,
|
||||
"cap_bytes": cap,
|
||||
"cap_hit": False,
|
||||
"excludes": sorted(EXCLUDED_DIRS),
|
||||
"roots": roots,
|
||||
"requested_roots": original_roots,
|
||||
"scope_narrowed": roots != original_roots,
|
||||
"import_files": sorted(imports),
|
||||
}
|
||||
for rel in sorted(files, key=lambda rel: (rel not in labelled, rel)):
|
||||
file = safe_path(source, rel)
|
||||
size = sizes[rel]
|
||||
# The cap bounds context only. No labelled source is ever cut or omitted.
|
||||
if proof["bytes"] + size > cap:
|
||||
proof["cap_hit"] = True
|
||||
if rel not in labelled:
|
||||
continue
|
||||
dest = safe_path(target, rel)
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
dest.write_bytes(file.read_bytes())
|
||||
proof["bytes"] += size
|
||||
proof["files"] += 1
|
||||
return proof
|
||||
|
||||
|
||||
def verify_functions(root, rows):
|
||||
for row in rows:
|
||||
data = safe_path(root, row["file"]).read_text(errors="replace")
|
||||
if not re.search(r"(?<![\w$])" + re.escape(row["function"]) + r"(?![\w$])", data):
|
||||
raise ValueError(f"labelled function not found in fetched source: {row['file']}::{row['function']}")
|
||||
|
||||
|
||||
def freeze_source(target):
|
||||
target.mkdir(parents=True)
|
||||
archive = target.parent / "source.tar"
|
||||
with archive.open("wb") as stream:
|
||||
subprocess.run(["git", "-C", str(ROOT), "archive", "HEAD", "src"], stdout=stream, check=True)
|
||||
subprocess.run(["tar", "-xf", str(archive), "-C", str(target)], check=True)
|
||||
archive.unlink()
|
||||
overlay = {}
|
||||
for name in SNAPSHOT_FILES:
|
||||
data = (ROOT / name).read_bytes()
|
||||
dest = target / name
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
dest.write_bytes(data)
|
||||
overlay[name] = hashlib.sha256(data).hexdigest()
|
||||
head = subprocess.run(["git", "-C", str(ROOT), "rev-parse", "HEAD"], capture_output=True, text=True, check=True).stdout.strip()
|
||||
return {"head": head, "overlay_sha256": overlay}
|
||||
|
||||
|
||||
def docker_command(source, checkout, out, name, image, deadline, question_deadline):
|
||||
return [
|
||||
"docker",
|
||||
"run",
|
||||
"--rm",
|
||||
"--init",
|
||||
"--name",
|
||||
name,
|
||||
"--network",
|
||||
"none",
|
||||
"--memory",
|
||||
"3g",
|
||||
"--pull",
|
||||
"never",
|
||||
"--entrypoint",
|
||||
"python",
|
||||
"-e",
|
||||
"PYTHONPATH=/frozen/src",
|
||||
"-v",
|
||||
f"{source}:/frozen:ro",
|
||||
"-v",
|
||||
f"{checkout}:/case:ro",
|
||||
"-v",
|
||||
f"{out}:/out",
|
||||
image,
|
||||
"/frozen/benchmarks/learn/engine_trace_worker.py",
|
||||
"--worker",
|
||||
"/out/pin.json",
|
||||
"--deadline",
|
||||
str(deadline),
|
||||
"--heap-profile",
|
||||
"vm",
|
||||
"--question-deadline",
|
||||
str(question_deadline),
|
||||
]
|
||||
|
||||
|
||||
def prepare_questions(pin, root, question_deadline):
|
||||
"""Prepare metadata on the host, before entering the dependency-minimal image."""
|
||||
active = [u for u in pin["units"] if u["supported"]]
|
||||
from openultrasast.learn.features import entry_names
|
||||
from openultrasast.mapping import php_hook_callbacks
|
||||
from openultrasast.plane.harvest import declares
|
||||
from openultrasast.preprocess import preprocess_repository
|
||||
|
||||
_, targets = preprocess_repository(root)
|
||||
entries, _ = entry_names(root, {u["file"] for u in active})
|
||||
hooks = ";".join(f"{h}:{f}" for h, functions in php_hook_callbacks(root, targets).items() for f in functions)
|
||||
requests = {}
|
||||
for u in active:
|
||||
specs = taint_specs(language="javascript" if u["language"] == "typescript" else u["language"])
|
||||
if not declares(safe_path(root, u["file"]).read_text(errors="replace").splitlines(), u["file"], u["function"]):
|
||||
raise ValueError(f"labelled function not declared: {u['file']}::{u['function']}")
|
||||
excerpt = u.get("evidence_scope") == "in-function-only" or pin.get("materialization", {}).get("mode") == "excerpt-fallback"
|
||||
params = request_params(
|
||||
specs[u["family"]],
|
||||
file=u["file"],
|
||||
function=u["function"],
|
||||
trace=True,
|
||||
parameter_sources=excerpt or u["function"] in entries,
|
||||
call_depth=0 if excerpt else 3,
|
||||
hook_callbacks=hooks,
|
||||
)
|
||||
params["questionDeadline"] = str(question_deadline)
|
||||
requests[u["unit"]] = params
|
||||
return requests
|
||||
|
||||
|
||||
def run_pin(pin, source, provenance, inputs, out, args):
|
||||
executor = getattr(args, "executor", "docker")
|
||||
node = None
|
||||
with tempfile.TemporaryDirectory(prefix="trace-pin-", dir=out) as scratch:
|
||||
scratch = Path(scratch)
|
||||
checkout, output = scratch / "case", scratch / "output"
|
||||
output.mkdir()
|
||||
try:
|
||||
with getattr(args, "input_lock", threading.Lock()):
|
||||
pin = {**pin, "materialization": inputs.materialize(pin, checkout)}
|
||||
write_json(output / "pin.json", {**pin, "questions": prepare_questions(pin, checkout, args.question_deadline)})
|
||||
name = "ousast-trace-" + pin_name(pin)[:16]
|
||||
started = time.monotonic()
|
||||
if executor in {"k8s", "k8s-jobs"}:
|
||||
done, node = args.kubernetes.execute(source, checkout, output, json.loads((output / "pin.json").read_text()))
|
||||
status, reason = "failed", done.stderr
|
||||
if done.returncode:
|
||||
(output / "container.log").write_text(reason)
|
||||
elif executor == "ax":
|
||||
done, node = args.ax_dispatcher.execute(checkout, output, json.loads((output / "pin.json").read_text()), pin_name(pin)[:-5])
|
||||
status = "timeout" if done is None else "failed"
|
||||
reason = f"AX deadline {args.deadline}s" if done is None else done.stderr
|
||||
elif executor == "queue":
|
||||
done, node = args.dispatcher.execute(checkout, output, json.loads((output / "pin.json").read_text()), pin_name(pin)[:-5])
|
||||
status, reason = "timeout", f"queue wait deadline {args.queue_timeout}s"
|
||||
else:
|
||||
command = docker_command(source, checkout, output, name, args.image, args.deadline, args.question_deadline)
|
||||
started = time.monotonic()
|
||||
done = None
|
||||
try:
|
||||
done = subprocess.run(command, capture_output=True, text=True, timeout=args.deadline)
|
||||
status, reason = "failed", f"container exit {done.returncode}: {done.stderr[-1000:]}"
|
||||
(output / "container.log").write_text(done.stdout + "\n" + done.stderr)
|
||||
except subprocess.TimeoutExpired:
|
||||
done = None
|
||||
status, reason = "timeout", f"container deadline {args.deadline}s"
|
||||
finally:
|
||||
# Never start the next pin unless this container is known to be gone.
|
||||
try:
|
||||
removed = subprocess.run(["docker", "rm", "-f", name], capture_output=True, timeout=20)
|
||||
except subprocess.SubprocessError as exc:
|
||||
raise RuntimeError(f"cannot confirm container {name} stopped; refusing another launch") from exc
|
||||
if done is None and removed.returncode != 0:
|
||||
raise RuntimeError(f"cannot remove timed-out container {name}; refusing another launch")
|
||||
result_path = output / "result.json"
|
||||
if not result_path.exists() and reason != "AX sandbox failed twice":
|
||||
reason = f"container produced no result: {reason}"
|
||||
record = json.loads(result_path.read_text()) if result_path.exists() else {**pin, "units": []}
|
||||
if not record.get("done"):
|
||||
previous = {u["unit"]: u for u in record["units"]}
|
||||
record["units"] = []
|
||||
for unit in pin["units"]:
|
||||
row = previous.get(unit["unit"], unit_record(unit, status))
|
||||
if row["status"] not in {"path", "asked-nothing"}:
|
||||
row.update(
|
||||
status=status if unit["supported"] else "unsupported", reason=reason if unit["supported"] else unit["reason"]
|
||||
)
|
||||
record["units"].append(row)
|
||||
if executor != "queue":
|
||||
record["container_exit"] = done.returncode if done else None
|
||||
else:
|
||||
record.setdefault("container_exit", None)
|
||||
if done is not None and done.returncode != 0:
|
||||
for row in record["units"]:
|
||||
if row["supported"]:
|
||||
row.update(status="failed", reason=reason)
|
||||
record.update(
|
||||
done=True,
|
||||
seconds=time.monotonic() - started,
|
||||
analyzer={"image": args.image, "code": "baked-in-image"} if executor == "ax" else provenance,
|
||||
image=record.get("image", args.image) if executor == "queue" else args.image,
|
||||
)
|
||||
if (output / "container.log").exists():
|
||||
shutil.copyfile(output / "container.log", out / pin_name(pin).replace(".json", ".log"))
|
||||
except (OSError, ValueError, subprocess.SubprocessError) as exc:
|
||||
record = {
|
||||
**pin,
|
||||
"done": True,
|
||||
"analyzer": provenance,
|
||||
"units": [unit_record(u, "failed" if u["supported"] else "unsupported", str(exc)) for u in pin["units"]],
|
||||
}
|
||||
if executor == "ax":
|
||||
record.update(executor="ax", worker=node, worker_ip=node, image=args.image)
|
||||
record["analyzer"] = {"image": args.image, "code": "baked-in-image"}
|
||||
if (output / "ax_attempts.json").exists():
|
||||
record["ax_attempts"] = json.loads((output / "ax_attempts.json").read_text())
|
||||
if getattr(args, "fallback", None):
|
||||
record.update(executor="docker", fallback=args.fallback, ax_attempt=args.ax_attempt, fallback_pending=False)
|
||||
if "ax_attempts" in args.ax_attempt:
|
||||
record["ax_attempts"] = args.ax_attempt["ax_attempts"]
|
||||
if executor in {"k8s", "k8s-jobs"}:
|
||||
record.update(executor="k8s", node=node, image=args.image)
|
||||
if executor == "queue":
|
||||
record.update(executor="queue", worker=record.get("worker", node), pod=record.get("pod"))
|
||||
save_pin(out / pin_name(pin), record)
|
||||
print(
|
||||
json.dumps({"repo": pin["repo"], "pin": pin["pin"], "statuses": dict(Counter(u["status"] for u in record["units"]))}),
|
||||
flush=True,
|
||||
)
|
||||
return any(u["status"] in {"failed", "timeout"} for u in record["units"])
|
||||
|
||||
|
||||
def save_pin(path, record):
|
||||
if path.exists():
|
||||
old = json.loads(path.read_text())
|
||||
selected = {u["unit"] for u in record["units"]}
|
||||
record["units"].extend(u for u in old.get("units", []) if u["unit"] not in selected)
|
||||
write_json(path, record)
|
||||
|
||||
|
||||
def summary(out):
|
||||
if not out.is_dir():
|
||||
raise ValueError(f"results root does not exist: {out}")
|
||||
family, source = defaultdict(Counter), defaultdict(Counter)
|
||||
pairs = defaultdict(dict)
|
||||
units = {}
|
||||
result_files = 0
|
||||
for path in sorted(out.glob("*.json")):
|
||||
record = json.loads(path.read_text())
|
||||
if not record.get("done"):
|
||||
continue
|
||||
result_files += 1
|
||||
for unit in record["units"]:
|
||||
if unit["unit"] in units:
|
||||
raise ValueError(f"duplicate result for {unit['unit']}")
|
||||
units[unit["unit"]] = unit
|
||||
family[unit["family"]][unit["status"]] += 1
|
||||
source[unit["source"]][unit["status"]] += 1
|
||||
if unit["pair"]:
|
||||
pairs[unit["family"], unit["pair"]][unit["label"]] = unit["status"]
|
||||
asymmetry = Counter()
|
||||
completed = Counter()
|
||||
for sides in pairs.values():
|
||||
if set(sides) != {0, 1}:
|
||||
asymmetry["incomplete-pair"] += 1
|
||||
continue
|
||||
key = {(True, False): "vulnerable-only", (False, True): "fixed-only", (True, True): "both", (False, False): "neither"}[
|
||||
sides[1] == "path", sides[0] == "path"
|
||||
]
|
||||
asymmetry[key] += 1
|
||||
if all(s in {"path", "asked-nothing"} for s in sides.values()):
|
||||
completed[key] += 1
|
||||
return {
|
||||
"result_files_read": result_files,
|
||||
"units": len(units),
|
||||
"by_family": {k: {s: v[s] for s in STATUSES} for k, v in sorted(family.items())},
|
||||
"by_source": {k: {s: v[s] for s in STATUSES} for k, v in sorted(source.items())},
|
||||
"pair_asymmetry": dict(asymmetry),
|
||||
"pair_asymmetry_both_answered": dict(completed),
|
||||
}
|
||||
|
||||
|
||||
def run_plan(plan, source, provenance, inputs, args):
|
||||
"""One task per fixed executor lane; progress is written only by the scheduler."""
|
||||
lanes = ["k8s"] * (args.parallel - 1) + ["docker"] if args.executor == "mixed" else [args.executor] * args.parallel
|
||||
if args.executor in {"ax", "mixed-ax"}:
|
||||
lanes = ["ax"] * args.parallel + ["docker"]
|
||||
args.input_lock = threading.Lock()
|
||||
progress = {
|
||||
"pins_done": 0,
|
||||
"pins_total": len(plan),
|
||||
"last_pin": None,
|
||||
"statuses": {},
|
||||
"lanes": dict(Counter(lanes)),
|
||||
"pins_in_flight": 0,
|
||||
"active_lanes": {},
|
||||
}
|
||||
counts = Counter()
|
||||
pending = list(plan)
|
||||
fallbacks = []
|
||||
if args.executor in {"ax", "mixed-ax"}:
|
||||
pending.clear()
|
||||
for pin in plan:
|
||||
path = args.out / pin_name(pin)
|
||||
old = json.loads(path.read_text()) if path.exists() else {}
|
||||
recorded = {u["unit"]: u.get("input_digest") for u in old.get("units", [])}
|
||||
if old.get("fallback_pending") and all(recorded.get(u["unit"]) == u.get("input_digest") for u in pin["units"]):
|
||||
fallbacks.append((pin, old))
|
||||
else:
|
||||
pending.append(pin)
|
||||
|
||||
def execute(pin, lane, attempt=None):
|
||||
if not any(u["supported"] for u in pin["units"]):
|
||||
save_pin(
|
||||
args.out / pin_name(pin), {**pin, "done": True, "units": [unit_record(u, "unsupported", u["reason"]) for u in pin["units"]]}
|
||||
)
|
||||
return False
|
||||
options = copy.copy(args)
|
||||
options.executor = lane
|
||||
if attempt is not None:
|
||||
options.fallback = attempt.get("fallback", "vm-oom")
|
||||
options.ax_attempt = attempt
|
||||
if lane == "docker":
|
||||
options.image = getattr(args, "docker_image", args.image)
|
||||
return run_pin(pin, source, provenance, inputs, args.out, options)
|
||||
|
||||
def complete(pin, lane, failed, progress):
|
||||
result = json.loads((args.out / pin_name(pin)).read_text())
|
||||
selected = {u["unit"] for u in pin["units"]}
|
||||
reasons = [u.get("reason", "") for u in result["units"] if u["unit"] in selected]
|
||||
sandbox_failed = args.executor == "mixed-ax" and result.get("container_exit") == 1 and "AX sandbox failed twice" in reasons
|
||||
if lane == "ax" and (sandbox_failed or any("JVM out of memory" in reason for reason in reasons)):
|
||||
result.update(fallback="vm-sandbox" if sandbox_failed else "vm-oom", fallback_pending=True)
|
||||
write_json(args.out / pin_name(pin), result)
|
||||
return False, result
|
||||
counts.update(u["status"] for u in result["units"] if u["unit"] in selected)
|
||||
progress.update(last_pin={"repo": pin["repo"], "pin": pin["pin"]}, statuses=dict(counts))
|
||||
return failed, None
|
||||
|
||||
return schedule(pending, lanes, args.out, execute, complete, fallbacks=fallbacks, overflow=args.executor != "ax", progress=progress)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--units", type=Path, default=ROOT / "plane/experiments/exp-005-graph-slice.units.jsonl")
|
||||
parser.add_argument("--examples", type=Path, help="frozen example JSONL instead of the local memory store")
|
||||
parser.add_argument("--memory", type=Path, default=Path.home() / "ousast-results/plane/memory")
|
||||
parser.add_argument("--cache", type=Path, default=Path.home() / ".cache/openultrasast")
|
||||
parser.add_argument("--out", type=Path, default=DEFAULT_OUT)
|
||||
parser.add_argument("--image", help="default: openultrasast:dev for Docker; public 2.0.1 image for k8s/mixed")
|
||||
parser.add_argument("--executor", choices=("docker", "k8s", "k8s-jobs", "mixed", "queue", "ax", "mixed-ax"), default="docker")
|
||||
parser.add_argument("--queue-status", action="store_true")
|
||||
parser.add_argument("--queue-timeout", type=int, default=3600, help="total queue wait including execution, seconds")
|
||||
parser.add_argument("--parallel", type=int, help="AX lanes (default 2, maximum 2); otherwise total lanes (default 1)")
|
||||
parser.add_argument("--ax-bin", default="ax")
|
||||
parser.add_argument("--kubeconfig", type=Path, help="AX kubeconfig; otherwise inherited KUBECONFIG/default CLI config")
|
||||
parser.add_argument("--atespace", default="default")
|
||||
parser.add_argument("--docker-image", default="openultrasast:dev", help="local VM image for AX OOM fallback/mixed overflow")
|
||||
parser.add_argument("--kube-context")
|
||||
parser.add_argument("--kube-namespace", default="ousast-engine")
|
||||
parser.add_argument("--keep-jobs", action="store_true", help="retain Jobs, Secrets and staging objects for debugging")
|
||||
parser.add_argument("--k8s-check", action="store_true")
|
||||
parser.add_argument("--deadline", type=int, default=900)
|
||||
parser.add_argument("--question-deadline", type=int, default=120)
|
||||
parser.add_argument("--limit", type=int, default=0)
|
||||
parser.add_argument("--only-family")
|
||||
parser.add_argument("--only-source", default="", help="comma-separated source catalog IDs")
|
||||
parser.add_argument("--rerun-status", default="failed,timeout", help="comma-separated statuses to re-run (including timeout)")
|
||||
parser.add_argument(
|
||||
"--allow-excerpt-fallback", action="store_true", help="allow fragments only if fetching/exporting resolved source fails"
|
||||
)
|
||||
parser.add_argument("--include-typescript", action="store_true")
|
||||
parser.add_argument("--dry-run", action="store_true")
|
||||
parser.add_argument("--summary", action="store_true")
|
||||
args = parser.parse_args()
|
||||
if args.parallel is None:
|
||||
args.parallel = 2 if args.executor in {"ax", "mixed-ax"} else 1
|
||||
if args.executor in {"ax", "mixed-ax"}:
|
||||
if args.parallel > 2:
|
||||
parser.error("AX allows at most 2 engine tasks in flight")
|
||||
if not args.image or not re.fullmatch(r"[^\s@]+@sha256:[0-9a-f]{64}", args.image):
|
||||
parser.error("AX requires --image by sha256 digest")
|
||||
if args.queue_status:
|
||||
print(json.dumps(Dispatcher(args).status(), indent=2))
|
||||
return 0
|
||||
if args.queue_timeout <= 0:
|
||||
parser.error("--queue-timeout must be positive")
|
||||
if args.executor == "queue" and (not args.image or "@sha256:" not in args.image):
|
||||
parser.error("queue requires --image with the deployed image digest")
|
||||
if args.executor == "queue" and args.deadline > 900:
|
||||
parser.error("queue deadline must be <= Deployment termination grace (900s)")
|
||||
if min(args.deadline, args.question_deadline) <= 0 or args.limit < 0:
|
||||
parser.error("deadlines must be positive; limit must be nonnegative")
|
||||
if args.parallel < 1 or (args.executor == "mixed" and args.parallel < 2):
|
||||
parser.error("--parallel must be positive; mixed needs at least 2 total lanes")
|
||||
if (args.executor in {"k8s", "k8s-jobs", "mixed"} or args.k8s_check) and not args.kube_context:
|
||||
parser.error("--kube-context is required for Kubernetes")
|
||||
args.image = args.image or (IMAGE if args.executor != "docker" or args.k8s_check else "openultrasast:dev")
|
||||
if args.k8s_check:
|
||||
return Kubernetes(args).check()
|
||||
rerun_status = set(filter(None, args.rerun_status.split(",")))
|
||||
if rerun_status - set(STATUSES):
|
||||
parser.error("unknown --rerun-status value")
|
||||
if args.summary:
|
||||
print(json.dumps(summary(args.out), indent=2))
|
||||
return 0
|
||||
# Even inherited opt-ins must not make catalog resolution fetch.
|
||||
os.environ["GIT_NO_LAZY_FETCH"] = "1"
|
||||
os.environ.pop("OPENULTRASAST_PAIRS_NETWORK", None)
|
||||
os.environ.pop("OPENULTRASAST_REPOS_NETWORK", None)
|
||||
examples = read_jsonl(args.examples) if args.examples else [r.row for r in FileStore(args.memory).rows(kind="example")]
|
||||
try:
|
||||
units = join_units(read_jsonl(args.units), examples, args.include_typescript)
|
||||
except ValueError as exc:
|
||||
parser.error(str(exc))
|
||||
inputs = Inputs(ROOT, args.cache, args.allow_excerpt_fallback)
|
||||
# Resolve pair source names for leak-audit coverage before calculating the resume key.
|
||||
for unit in units:
|
||||
if unit.get("excerpt") or unit["source"].startswith("advisory-fixes"):
|
||||
try:
|
||||
case, _ = inputs.pair_for(unit)
|
||||
catalog_source, _ = inputs.catalog_for(case)
|
||||
unit["source"] = catalog_source if catalog_source.startswith("advisory-fixes") else case.slice
|
||||
unit["excerpt"] = True
|
||||
except ValueError as exc:
|
||||
unit["mapping_error"] = str(exc)
|
||||
plan = make_plan(
|
||||
units,
|
||||
args.out,
|
||||
args.only_family,
|
||||
args.limit,
|
||||
include_typescript=args.include_typescript,
|
||||
only_source=set(filter(None, args.only_source.split(","))),
|
||||
rerun_status=rerun_status,
|
||||
allow_excerpt_fallback=args.allow_excerpt_fallback,
|
||||
)
|
||||
# Planning is read-only, including unsupported pins. All persistence stays below this return.
|
||||
if args.dry_run:
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"pins": len(plan),
|
||||
"runnable_pins": sum(any(u["supported"] for u in p["units"]) for p in plan),
|
||||
"units": sum(len(p["units"]) for p in plan),
|
||||
"estimated_seconds": sum(p["estimated_seconds"] for p in plan),
|
||||
"estimate_basis": "inventory medians: excerpt 31/47s, repository 182/498s; not a deadline guarantee",
|
||||
"plan": plan,
|
||||
},
|
||||
indent=2,
|
||||
)
|
||||
)
|
||||
return 0
|
||||
args.out.mkdir(parents=True, exist_ok=True)
|
||||
lock = (args.out / ".lock").open("w")
|
||||
try:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except BlockingIOError:
|
||||
parser.error("another trace runner holds this results root")
|
||||
failures = False
|
||||
if args.executor not in {"ax", "mixed-ax"}:
|
||||
args.docker_image = args.image
|
||||
if args.executor in {"docker", "mixed"} and any(u["supported"] for pin in plan for u in pin["units"]):
|
||||
image = subprocess.run(
|
||||
["docker", "image", "inspect", "--format", "{{.Id}}", args.image], capture_output=True, text=True, check=True
|
||||
)
|
||||
args.docker_image = image.stdout.strip()
|
||||
if not args.docker_image:
|
||||
raise ValueError("Docker returned no local image identity")
|
||||
if args.executor in {"k8s", "k8s-jobs", "mixed"} and any(u["supported"] for pin in plan for u in pin["units"]):
|
||||
args.kubernetes = Kubernetes(args)
|
||||
args.kubernetes.namespace()
|
||||
if args.executor == "queue":
|
||||
args.dispatcher = Dispatcher(args)
|
||||
if args.executor in {"ax", "mixed-ax"}:
|
||||
from engine_trace_ax import AX
|
||||
|
||||
args.ax_dispatcher = AX(args)
|
||||
with tempfile.TemporaryDirectory(prefix="trace-source-", dir=args.out) as scratch:
|
||||
source = Path(scratch) / "frozen"
|
||||
provenance = {"image": args.image, "executor": "queue"} if args.executor == "queue" else freeze_source(source)
|
||||
failures = run_plan(plan, source, provenance, inputs, args)
|
||||
return 2 if failures else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,108 @@
|
||||
"""Engine workload adapter for the shared AX transport."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
import json
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
import tempfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
from engine_trace_k8s import pack
|
||||
|
||||
from benchmarks.ax.batch import AXLane, Item, SandboxFailure, Workload, manifest
|
||||
from benchmarks.ax.batch import task_state as task_state
|
||||
from openultrasast.plane.memory import open_store as open_store
|
||||
|
||||
|
||||
def task_manifest(name, args, urls):
|
||||
return manifest(
|
||||
name, engine_workload(args, lambda *_: None), engine_item({}, "pin", b"", args), urls, args.atespace, validate_name=False
|
||||
)
|
||||
|
||||
|
||||
def engine_workload(args, validate):
|
||||
return Workload(args.image, frozenset({"SOURCE_URL", "QUESTIONS_URL", "RESULT_URL"}), args.deadline, validate)
|
||||
|
||||
|
||||
def engine_item(pin, pin_id, source, args):
|
||||
return Item(
|
||||
pin_id,
|
||||
{"SOURCE_URL": source, "QUESTIONS_URL": json.dumps(pin).encode()},
|
||||
"result.dat",
|
||||
("python3", "/app/benchmarks/learn/engine_task_entry.py"),
|
||||
{"DEADLINE": str(args.deadline), "QUESTION_DEADLINE": str(args.question_deadline)},
|
||||
"",
|
||||
{"SOURCE_URL": "source.dat", "QUESTIONS_URL": "questions.json"},
|
||||
)
|
||||
|
||||
|
||||
class AX(AXLane):
|
||||
def __init__(self, args, store=None, pause=time.sleep, clock=time.monotonic):
|
||||
super().__init__(args, engine_workload(args, lambda *_: None), store, pause, clock)
|
||||
|
||||
def receive(self, data, output, pin):
|
||||
# Validate in isolation: malformed results must never become usable checkpoints.
|
||||
with tempfile.TemporaryDirectory(prefix="engine-ax-result-") as temporary:
|
||||
root = Path(temporary)
|
||||
with tarfile.open(fileobj=io.BytesIO(data)) as archive:
|
||||
archive.extractall(root, filter="data")
|
||||
record = json.loads((root / "result.json").read_text())
|
||||
units = record.get("units", [])
|
||||
if record.get("done") and record.get("status") == "failed" and not units:
|
||||
# Questions download can fail before the actor learns any unit identities.
|
||||
# Keep that explicit failure, filling identities only from the host input.
|
||||
record = {
|
||||
**pin,
|
||||
**record,
|
||||
"units": [
|
||||
{
|
||||
**unit,
|
||||
"status": "failed" if unit["supported"] else "unsupported",
|
||||
"reason": record.get("reason", "AX task entry failed"),
|
||||
"instrument": {"files": [], "bytes": 0, "cpg_path": None, "cpg_bytes": 0, "jvm": []},
|
||||
"questions": {},
|
||||
"questions_asked": [],
|
||||
"questions_completed": [],
|
||||
"witness_rows": [],
|
||||
"traces": [],
|
||||
}
|
||||
for unit in pin["units"]
|
||||
],
|
||||
}
|
||||
units = record["units"]
|
||||
(root / "result.json").write_text(json.dumps(record))
|
||||
expected = [u["unit"] for u in pin["units"]]
|
||||
actual = [u["unit"] for u in units]
|
||||
if not record.get("done") or sorted(actual) != sorted(expected):
|
||||
raise ValueError("AX Task returned incomplete or mismatched units")
|
||||
output.mkdir(parents=True, exist_ok=True)
|
||||
# Keep the artifacts without interpreting any remote paths as host destinations.
|
||||
shutil.copytree(root, output, dirs_exist_ok=True)
|
||||
|
||||
def execute(self, checkout, output, pin, pin_id):
|
||||
import copy
|
||||
|
||||
self = copy.copy(self)
|
||||
self.workload = engine_workload(self.args, lambda data, target: self.receive(data, target, pin))
|
||||
attempts = []
|
||||
try:
|
||||
for attempt in range(2):
|
||||
try:
|
||||
return self._execute_attempt(engine_item(pin, pin_id, pack(checkout), self.args), output, attempt, attempts)
|
||||
except SandboxFailure:
|
||||
if attempt == 0:
|
||||
self.pause(10)
|
||||
return subprocess.CompletedProcess([], 1, "", "AX sandbox failed twice"), None
|
||||
finally:
|
||||
(output / "ax_attempts.json").write_text(json.dumps(attempts))
|
||||
result_path = output / "result.json"
|
||||
if result_path.exists():
|
||||
record = json.loads(result_path.read_text())
|
||||
record["ax_attempts"] = attempts
|
||||
result_path.write_text(json.dumps(record))
|
||||
@@ -0,0 +1,259 @@
|
||||
"""Host Kubernetes transport; the embedded bootstrap uses only the standard library."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
import tarfile
|
||||
import time
|
||||
import uuid
|
||||
from datetime import timedelta
|
||||
|
||||
from openultrasast.plane.memory import PRESIGNED_ARCHIVE_SUFFIX, S3Store, open_store
|
||||
|
||||
IMAGE = "ghcr.io/norandom/openultrasast:2.0.1"
|
||||
BOOTSTRAP = r"""
|
||||
import io, json, os, pathlib, subprocess, sys, tarfile, urllib.request
|
||||
|
||||
def get(name):
|
||||
with urllib.request.urlopen(os.environ[name], timeout=120) as response:
|
||||
return response.read()
|
||||
|
||||
def put(data):
|
||||
request = urllib.request.Request(os.environ['RESULT_URL'], data=data, method='PUT')
|
||||
with urllib.request.urlopen(request, timeout=120) as response:
|
||||
response.read()
|
||||
|
||||
try:
|
||||
if os.environ.get('CHECK') == '1':
|
||||
data = get('SOURCE_URL')
|
||||
assert len(data) == 1024
|
||||
subprocess.run([sys.executable, '-c', 'print(42)'], check=True)
|
||||
put(data)
|
||||
else:
|
||||
for name, target in [('SOURCE_URL', '/case'), ('ANALYZER_URL', '/frozen')]:
|
||||
pathlib.Path(target).mkdir(parents=True, exist_ok=True)
|
||||
with tarfile.open(fileobj=io.BytesIO(get(name))) as archive:
|
||||
archive.extractall(target, filter='data')
|
||||
out = pathlib.Path('/out')
|
||||
out.mkdir(exist_ok=True)
|
||||
(out / 'pin.json').write_text(os.environ['PIN'])
|
||||
env = dict(os.environ, PYTHONPATH='/frozen/src')
|
||||
for key in ['SOURCE_URL', 'ANALYZER_URL', 'RESULT_URL']:
|
||||
env.pop(key, None)
|
||||
with (out / 'container.log').open('w') as log:
|
||||
done = subprocess.run([sys.executable, '/frozen/benchmarks/learn/engine_trace_worker.py',
|
||||
'--worker', '/out/pin.json', '--deadline', os.environ['DEADLINE'],
|
||||
'--question-deadline', os.environ['QUESTION_DEADLINE']], env=env, stdout=log, stderr=log)
|
||||
(out / 'exit.json').write_text(json.dumps(done.returncode))
|
||||
buffer = io.BytesIO()
|
||||
with tarfile.open(fileobj=buffer, mode='w') as archive:
|
||||
for path in out.iterdir():
|
||||
archive.add(path, arcname=path.name)
|
||||
put(buffer.getvalue())
|
||||
sys.exit(done.returncode)
|
||||
except Exception as exc:
|
||||
# urllib exceptions can contain bearer URLs. Never print exception text.
|
||||
print('engine bootstrap failed: ' + type(exc).__name__, file=sys.stderr)
|
||||
sys.exit(1)
|
||||
"""
|
||||
|
||||
|
||||
def pack(path):
|
||||
buffer = io.BytesIO()
|
||||
with tarfile.open(fileobj=buffer, mode="w") as archive:
|
||||
for child in sorted(path.iterdir()):
|
||||
archive.add(child, arcname=child.name)
|
||||
return buffer.getvalue()
|
||||
|
||||
|
||||
def manifests(name, run, args, urls, pin, check=False):
|
||||
labels = {"engine-trace-run": run, "engine-trace-job": name, "engine-trace": "true"}
|
||||
secret = {
|
||||
"apiVersion": "v1",
|
||||
"kind": "Secret",
|
||||
"metadata": {"name": name, "namespace": args.kube_namespace, "labels": labels},
|
||||
"type": "Opaque",
|
||||
"stringData": urls,
|
||||
}
|
||||
env = [{"name": key, "valueFrom": {"secretKeyRef": {"name": name, "key": key}}} for key in urls]
|
||||
env += [
|
||||
{"name": key, "value": value}
|
||||
for key, value in {
|
||||
"PIN": json.dumps(pin),
|
||||
"DEADLINE": str(args.deadline),
|
||||
"QUESTION_DEADLINE": str(args.question_deadline),
|
||||
"CHECK": "1" if check else "0",
|
||||
}.items()
|
||||
]
|
||||
job = {
|
||||
"apiVersion": "batch/v1",
|
||||
"kind": "Job",
|
||||
"metadata": {"name": name, "namespace": args.kube_namespace, "labels": labels},
|
||||
"spec": {
|
||||
"activeDeadlineSeconds": args.deadline,
|
||||
"backoffLimit": 0,
|
||||
"ttlSecondsAfterFinished": 86400,
|
||||
"template": {
|
||||
"metadata": {"labels": labels},
|
||||
"spec": {
|
||||
"restartPolicy": "Never",
|
||||
"automountServiceAccountToken": False,
|
||||
"securityContext": {"runAsUser": 1000, "runAsGroup": 1000, "fsGroup": 1000},
|
||||
"volumes": [{"name": name, "emptyDir": {}} for name in ("case", "frozen", "out")],
|
||||
"affinity": {
|
||||
"podAntiAffinity": {
|
||||
"requiredDuringSchedulingIgnoredDuringExecution": [
|
||||
{
|
||||
"labelSelector": {"matchLabels": {"engine-trace": "true"}},
|
||||
"topologyKey": "kubernetes.io/hostname",
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"containers": [
|
||||
{
|
||||
"name": "engine",
|
||||
"image": args.image,
|
||||
"command": ["python", "-c", BOOTSTRAP],
|
||||
"env": env,
|
||||
"volumeMounts": [{"name": name, "mountPath": "/" + name} for name in ("case", "frozen", "out")],
|
||||
"resources": {"requests": {"memory": "3Gi", "cpu": "1"}, "limits": {"memory": "3Gi"}},
|
||||
}
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
return job, secret
|
||||
|
||||
|
||||
def redact(text):
|
||||
return re.sub(r"https?://[^\s\"'<>]+", "[redacted URL]", text)
|
||||
|
||||
|
||||
class Kubernetes:
|
||||
def __init__(self, args, store=None):
|
||||
self.args = args
|
||||
self.run = uuid.uuid4().hex
|
||||
self.store = store if store is not None else open_store()
|
||||
if not isinstance(self.store, S3Store):
|
||||
raise ValueError("Kubernetes requires OUSAST_MEMORY=s3://bucket[/prefix]")
|
||||
|
||||
def kubectl(self, *args, manifest=None, check=True):
|
||||
result = subprocess.run(
|
||||
["kubectl", "--context", self.args.kube_context, "--namespace", self.args.kube_namespace, *args],
|
||||
input=json.dumps(manifest) if manifest is not None else None,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=30,
|
||||
)
|
||||
if check and result.returncode:
|
||||
# kubectl can echo submitted Secret data on errors; never expose stderr.
|
||||
raise ValueError(f"kubectl {args[0]} failed (exit {result.returncode})")
|
||||
return result
|
||||
|
||||
def namespace(self):
|
||||
self.kubectl("get", "nodes", "-o", "json")
|
||||
self.kubectl("apply", "-f", "-", manifest={"apiVersion": "v1", "kind": "Namespace", "metadata": {"name": self.args.kube_namespace}})
|
||||
|
||||
def execute(self, source, checkout, output, pin, check=False):
|
||||
name = "engine-" + uuid.uuid4().hex[:24]
|
||||
pin_key = hashlib.sha256(json.dumps([pin.get("repo"), pin.get("pin")]).encode()).hexdigest()
|
||||
prefix = f"engine-jobs/{self.run}/{pin_key}/"
|
||||
keys = [prefix + name + PRESIGNED_ARCHIVE_SUFFIX for name in ("source", "analyzer", "result")]
|
||||
expiry = timedelta(seconds=self.args.deadline + 900)
|
||||
node = None
|
||||
try:
|
||||
self.store._put(keys[0], b"x" * 1024 if check else pack(checkout))
|
||||
self.store._put(keys[1], b"" if check else pack(source))
|
||||
urls = {
|
||||
"SOURCE_URL": self.store.presign_get(keys[0], expiry),
|
||||
"ANALYZER_URL": self.store.presign_get(keys[1], expiry),
|
||||
"RESULT_URL": self.store.presign_put(keys[2], expiry),
|
||||
}
|
||||
job, secret = manifests(name, self.run, self.args, urls, pin, check)
|
||||
# create avoids kubectl's last-applied annotation duplicating Secret values.
|
||||
created = json.loads(self.kubectl("create", "-f", "-", "-o", "json", manifest=job).stdout)
|
||||
secret["metadata"]["ownerReferences"] = [
|
||||
{
|
||||
"apiVersion": "batch/v1",
|
||||
"kind": "Job",
|
||||
"name": name,
|
||||
"uid": created["metadata"]["uid"],
|
||||
}
|
||||
]
|
||||
self.kubectl("create", "-f", "-", manifest=secret)
|
||||
end = time.monotonic() + self.args.deadline + 60
|
||||
reason = "host deadline waiting for Job"
|
||||
while time.monotonic() < end:
|
||||
state = json.loads(self.kubectl("get", "job", name, "-o", "json").stdout).get("status", {})
|
||||
pods = json.loads(self.kubectl("get", "pods", "-l", f"engine-trace-job={name}", "-o", "json").stdout)
|
||||
node = next((p.get("spec", {}).get("nodeName") for p in pods.get("items", []) if p.get("spec", {}).get("nodeName")), node)
|
||||
result = self.store._get(keys[2])
|
||||
failed = next((c for c in state.get("conditions", []) if c.get("type") == "Failed" and c.get("status") == "True"), None)
|
||||
if failed or state.get("failed"):
|
||||
reason = (failed or {}).get("reason", "Job failed") + ": " + (failed or {}).get("message", "")
|
||||
break
|
||||
if state.get("succeeded") and result is not None:
|
||||
if check:
|
||||
if result[0] != b"x" * 1024:
|
||||
raise ValueError("round-trip bytes mismatch")
|
||||
else:
|
||||
with tarfile.open(fileobj=io.BytesIO(result[0])) as archive:
|
||||
archive.extractall(output, filter="data")
|
||||
if not (output / "result.json").is_file():
|
||||
raise ValueError("Job produced no result.json")
|
||||
record = json.loads((output / "result.json").read_text())
|
||||
if not record.get("done") or {u["unit"] for u in record["units"]} != {u["unit"] for u in pin["units"]}:
|
||||
raise ValueError("Job returned incomplete or mismatched units")
|
||||
return subprocess.CompletedProcess([], 0, "", ""), node
|
||||
time.sleep(2)
|
||||
logs = self.kubectl("logs", f"job/{name}", "--tail=40", check=False)
|
||||
reason = redact(reason + "\n" + logs.stdout + logs.stderr)
|
||||
return subprocess.CompletedProcess([], 1, "", reason), node
|
||||
except Exception as exc:
|
||||
reason = "Kubernetes transport failed: " + type(exc).__name__ + ": " + redact(str(exc))
|
||||
try:
|
||||
logs = self.kubectl("logs", f"job/{name}", "--tail=40", check=False)
|
||||
reason += "\n" + redact(logs.stdout + logs.stderr)
|
||||
except (OSError, ValueError, subprocess.SubprocessError):
|
||||
pass
|
||||
return subprocess.CompletedProcess([], 1, "", reason), node
|
||||
finally:
|
||||
if not self.args.keep_jobs:
|
||||
# Attempt every cleanup even if an earlier deletion fails.
|
||||
errors = []
|
||||
for kind in ("job", "secret"):
|
||||
try:
|
||||
self.kubectl("delete", kind, name, "--ignore-not-found", "--wait=true", "--timeout=30s")
|
||||
except Exception as exc:
|
||||
errors.append(type(exc).__name__)
|
||||
for key in keys:
|
||||
try:
|
||||
self.store._delete(key)
|
||||
except Exception as exc:
|
||||
errors.append(type(exc).__name__)
|
||||
if errors:
|
||||
raise RuntimeError("engine Job cleanup failed: " + ", ".join(errors))
|
||||
|
||||
def check(self):
|
||||
self.namespace()
|
||||
nodes = json.loads(self.kubectl("get", "nodes", "-o", "json").stdout)
|
||||
for node in nodes["items"]:
|
||||
print(json.dumps({"node": node["metadata"]["name"], "allocatable_memory": node["status"]["allocatable"]["memory"]}))
|
||||
done, node = self.execute(None, None, None, {}, check=True)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"k8s_check": "passed" if done.returncode == 0 else "failed",
|
||||
"node": node,
|
||||
"reason": done.stderr,
|
||||
"round_trip_bytes": 1024 if done.returncode == 0 else 0,
|
||||
}
|
||||
)
|
||||
)
|
||||
return done.returncode
|
||||
@@ -0,0 +1,51 @@
|
||||
"""Lossless parsing of the optional shortest-path trace, with explicit evidence limits."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections.abc import Mapping
|
||||
from typing import Any
|
||||
|
||||
_STEP = re.compile(r"^(.*):(-?\d+) \[([^\]]+)\] ([\s\S]*)$")
|
||||
|
||||
|
||||
def parse_trace(row: Mapping[str, Any]) -> dict[str, Any]:
|
||||
"""Preserve row-level discharge evidence; never guess which node sanitized a path.
|
||||
|
||||
The query emits basename-only files and truncated code. A bound can be off-path;
|
||||
it remains in ``guard`` even when no trace step exactly matches it. Invalid steps
|
||||
raise instead of turning an unreadable trace into an empty path.
|
||||
"""
|
||||
raw = row.get("trace", [])
|
||||
if not isinstance(raw, list):
|
||||
raise ValueError("trace must be an array")
|
||||
steps = []
|
||||
bound = str(row.get("bound") or "")
|
||||
for index, text in enumerate(raw):
|
||||
match = _STEP.fullmatch(text) if isinstance(text, str) else None
|
||||
if match is None:
|
||||
raise ValueError(f"invalid trace step: {text!r}")
|
||||
location, line, node_label, code = match.groups()
|
||||
file, separator, method = location.partition(":")
|
||||
roles = ["source"] if index == 0 else ["step"]
|
||||
if index == len(raw) - 1:
|
||||
roles = ["source", "sink"] if index == 0 else ["sink"]
|
||||
if bound and code == bound:
|
||||
roles.append("guard")
|
||||
steps.append(
|
||||
{
|
||||
"file": file,
|
||||
"method": method if separator else "?",
|
||||
"line": int(line),
|
||||
"node_label": node_label,
|
||||
"code": code,
|
||||
"roles": roles,
|
||||
"raw": text,
|
||||
}
|
||||
)
|
||||
return {
|
||||
"steps": steps,
|
||||
"sanitized": bool(row.get("sanitized", False)),
|
||||
"sanitizer": {"on_path": bool(row.get("sanitized", False)), "step": None},
|
||||
"guard": {"text": bound, "bounded": bool(row.get("bounded", False))},
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
"""Host queue transport: source and task metadata only, never analyzer code."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
import time
|
||||
import uuid
|
||||
|
||||
from engine_worker_service import PENDING, client_from_env, pack, put, read, result_key, status
|
||||
|
||||
from openultrasast.plane.memory import PRESIGNED_ARCHIVE_SUFFIX
|
||||
|
||||
|
||||
class Dispatcher:
|
||||
def __init__(self, args, client=None, *, pause=time.sleep, clock=time.monotonic):
|
||||
self.client = client if client is not None else client_from_env()
|
||||
self.run = uuid.uuid4().hex
|
||||
self.args, self.pause, self.clock = args, pause, clock
|
||||
|
||||
def execute(self, checkout, output, pin, pin_id):
|
||||
task_id = f"{self.run}-{pin_id}"
|
||||
task = dict(
|
||||
run=self.run,
|
||||
id=pin_id,
|
||||
task_id=task_id,
|
||||
pin=pin,
|
||||
input=f"engine-queue/inputs/{self.run}/{pin_id}/source{PRESIGNED_ARCHIVE_SUFFIX}",
|
||||
image=self.args.image,
|
||||
deadline=self.args.deadline,
|
||||
question_deadline=self.args.question_deadline,
|
||||
)
|
||||
try:
|
||||
self.client.put(task["input"], pack(checkout), {})
|
||||
put(self.client, PENDING + task_id, task)
|
||||
end = self.clock() + self.args.queue_timeout
|
||||
while self.clock() < end:
|
||||
result = read(self.client, result_key(task))
|
||||
if result is not None:
|
||||
(output / "result.json").write_text(json.dumps(result))
|
||||
self.client.delete(task["input"])
|
||||
return subprocess.CompletedProcess([], 0, "", ""), result.get("worker")
|
||||
self.pause(2)
|
||||
# Removing pending cancels work that has not been claimed; retain source for a live lease.
|
||||
self.client.delete(PENDING + task_id)
|
||||
return None, None
|
||||
except Exception as exc:
|
||||
# SDK errors can contain endpoint URLs or credentials. Do not persist their text.
|
||||
raise OSError("queue transport failed") from exc
|
||||
|
||||
def status(self):
|
||||
return status(self.client)
|
||||
@@ -0,0 +1,644 @@
|
||||
"""Container-only Joern trace worker; host prepares all corpus and question metadata.
|
||||
|
||||
Keep this module independent of learn/plane and their optional dependencies. The
|
||||
frozen host invokes this file directly in the shipped, offline analyzer image.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import contextlib
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import selectors
|
||||
import signal
|
||||
import subprocess
|
||||
import tempfile
|
||||
import time
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
|
||||
from engine_trace_parse import parse_trace
|
||||
|
||||
from openultrasast.cpg.backend import DATAFLOW_OVERLAY, JoernBackend, _render, extract_payload
|
||||
from openultrasast.cpg.session import EngineSession
|
||||
from openultrasast.model.contracts import ExecutionBudget
|
||||
|
||||
|
||||
def write_json(path, value):
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = path.with_suffix(".tmp")
|
||||
tmp.write_text(json.dumps(value, indent=2, sort_keys=True) + "\n")
|
||||
tmp.replace(path)
|
||||
|
||||
|
||||
def safe_path(root, name):
|
||||
path = (root / name).resolve()
|
||||
if not path.is_relative_to(root.resolve()):
|
||||
raise ValueError(f"path escapes input root: {name}")
|
||||
return path
|
||||
|
||||
|
||||
FRONTENDS = {"php2cpg", "pysrc2cpg", "jssrc2cpg", "javasrc2cpg", "c2cpg", "joern-parse"}
|
||||
MIN_CPG_BYTES = 1024
|
||||
HEAP_MB = 2560
|
||||
CLUSTER_JVM_FLAGS = (
|
||||
"-Xms1700m -Xmx1700m -XX:MaxMetaspaceSize=256m -XX:ReservedCodeCacheSize=128m "
|
||||
"-XX:MaxDirectMemorySize=256m -Xss512k -XX:ActiveProcessorCount=2 -XX:+UseG1GC"
|
||||
)
|
||||
ENGINE_BUG = "engine bug: jssrc2cpg ObjectPropertyCallLinker"
|
||||
EXCLUDED_DIRS = frozenset({"tests", "test", "docs", "node_modules", "vendor", "dist", "build", "templates", "migrations"})
|
||||
|
||||
|
||||
def instrument_failure(proof):
|
||||
if any(stage.get("oom") for stage in proof.get("jvm", [])):
|
||||
return "JVM out of memory"
|
||||
if not proof.get("files") or not proof.get("bytes"):
|
||||
return "instrument read zero files or zero bytes"
|
||||
if proof.get("cpg_bytes", 0) <= MIN_CPG_BYTES:
|
||||
return "frontend wrote no CPG"
|
||||
for stage in proof.get("jvm", []):
|
||||
if stage["command"] in FRONTENDS:
|
||||
if stage.get("cpg_bytes", 0) <= MIN_CPG_BYTES:
|
||||
return "frontend wrote no CPG"
|
||||
elif (
|
||||
stage["command"] == "joern" or (stage["command"] == "joern-session" and stage.get("step") == "start" and stage.get("success"))
|
||||
) and stage["seconds"] < 5:
|
||||
return f"implausibly fast JVM step: {stage['command']} in {stage['seconds']:.3f}s (<5s)"
|
||||
if not proof.get("jvm"):
|
||||
return "no JVM step recorded"
|
||||
return ""
|
||||
|
||||
|
||||
class MeasuredBackend(JoernBackend):
|
||||
"""Use the shipped backend with process and resident-session watchdogs."""
|
||||
|
||||
def __init__(self, deadline, question_deadline, checkpoint, heap_profile="vm"):
|
||||
if heap_profile not in {"vm", "cluster"}:
|
||||
raise ValueError(f"unknown heap profile: {heap_profile}")
|
||||
self.heap_profile = heap_profile
|
||||
super().__init__(
|
||||
build_timeout=deadline,
|
||||
query_timeout=deadline,
|
||||
session_transport=False,
|
||||
heap_mb=1700 if heap_profile == "cluster" else HEAP_MB,
|
||||
)
|
||||
self.deadline = time.monotonic() + deadline
|
||||
self.question_deadline = question_deadline
|
||||
self.checkpoint = checkpoint
|
||||
self.stages = []
|
||||
self.asked = []
|
||||
self.answers = {}
|
||||
self.completion_counts = Counter()
|
||||
self.timed_out = False
|
||||
self.fatal = ""
|
||||
self.labelled_files = []
|
||||
self.retry_pending = False
|
||||
self.retries = []
|
||||
self.session_fallback = ""
|
||||
|
||||
def _jvm_env(self):
|
||||
env = super()._jvm_env()
|
||||
if self.heap_profile == "cluster":
|
||||
# Bound every JVM, including the child spawned by the Joern launcher.
|
||||
env["JAVA_TOOL_OPTIONS"] = CLUSTER_JVM_FLAGS
|
||||
env.pop("_JAVA_OPTIONS", None)
|
||||
env.pop("JDK_JAVA_OPTIONS", None)
|
||||
return env
|
||||
|
||||
def _session_stage(self, name, action):
|
||||
started = time.monotonic()
|
||||
result = None
|
||||
try:
|
||||
result = action()
|
||||
return result
|
||||
finally:
|
||||
self.stages.append(
|
||||
{
|
||||
"command": "joern-session",
|
||||
"step": name,
|
||||
"success": result is not None and result is not False,
|
||||
"seconds": time.monotonic() - started,
|
||||
"heap_mb": self._heap_mb(),
|
||||
"java_tool_options": self._jvm_env()["JAVA_TOOL_OPTIONS"],
|
||||
}
|
||||
)
|
||||
self.checkpoint()
|
||||
|
||||
def _fallback_session(self, reason):
|
||||
self.session_fallback = reason
|
||||
if self._session is not None:
|
||||
detail = self._session.log_tail()
|
||||
if "OutOfMemoryError" in detail:
|
||||
self.fatal = "JVM out of memory"
|
||||
self.stages[-1]["oom"] = True
|
||||
self.stages[-1]["output_tail"] = detail
|
||||
self.close_session()
|
||||
self.checkpoint()
|
||||
|
||||
def _session_for(self, cpg_path):
|
||||
if self.session_fallback or self.fatal or self.timed_out:
|
||||
return None
|
||||
try:
|
||||
if self._session is None:
|
||||
self._session = EngineSession(
|
||||
cpg_path.parent / "engine-session",
|
||||
ExecutionBudget(self.deadline, 5),
|
||||
self._jvm_env(),
|
||||
startup_allowance_seconds=min(180, max(20, (self.deadline - time.monotonic()) / 3)),
|
||||
)
|
||||
if not self._session_stage("start", self._session.start):
|
||||
self._fallback_session(self._session.failure or "session_start_failed")
|
||||
return None
|
||||
if self._session.loaded_graph != str(cpg_path):
|
||||
if not self._session_stage("load", lambda: self._session.load(cpg_path, timeout=self.query_timeout)):
|
||||
self._fallback_session(self._session.failure or "session_load_failed")
|
||||
return None
|
||||
self._defined.clear()
|
||||
return self._session
|
||||
except (OSError, RuntimeError) as exc:
|
||||
self._fallback_session(f"session_start_or_load:{exc}")
|
||||
return None
|
||||
|
||||
def _session_payload(self, cpg_path, query, params):
|
||||
session = self._session_for(cpg_path)
|
||||
if session is None:
|
||||
return None
|
||||
end = min(self.deadline, time.monotonic() + self.query_timeout)
|
||||
question_end = end
|
||||
prior_asked = list(self.asked)
|
||||
prior_answers = dict(self.answers)
|
||||
prior_counts = self.completion_counts.copy()
|
||||
consumed = 0
|
||||
streamed = ""
|
||||
|
||||
def observe(output):
|
||||
nonlocal consumed, question_end, streamed
|
||||
streamed = output
|
||||
lines = output[consumed:].splitlines(keepends=True)
|
||||
for line in lines:
|
||||
if not line.endswith("\n"):
|
||||
break
|
||||
consumed += len(line)
|
||||
try:
|
||||
event = json.loads(line)
|
||||
except ValueError:
|
||||
continue
|
||||
if not isinstance(event, dict):
|
||||
continue
|
||||
if "__question__" in event:
|
||||
self.asked.append(event["__question__"])
|
||||
question_end = min(end, time.monotonic() + self.question_deadline)
|
||||
if "id" in event and "rows" in event:
|
||||
self.answers[event["id"]] = event["rows"]
|
||||
self.completion_counts[event["id"]] += 1
|
||||
question_end = end
|
||||
self.checkpoint()
|
||||
if time.monotonic() >= min(end, question_end):
|
||||
self.timed_out = True
|
||||
return False
|
||||
return True
|
||||
|
||||
def run():
|
||||
source = (self.queries_dir / f"{query}.sc").read_text()
|
||||
# Same shipped script/parameters as JoernBackend, using the graph loaded above.
|
||||
if source.count("importCpg(cpgFile)") != 1:
|
||||
return None
|
||||
symbol = f"ousast_{query}"
|
||||
if query not in self._defined:
|
||||
source = source.replace("importCpg(cpgFile)", "()", 1)
|
||||
source = source.replace("@main def exec(", f"def {symbol}(", 1)
|
||||
if session.define(source, timeout=max(0, end - time.monotonic())) is None:
|
||||
return None
|
||||
self._defined.add(query)
|
||||
arguments = ", ".join(f"{name} = {json.dumps(_render(value))}" for name, value in sorted(params.items()))
|
||||
session.output_observer = observe
|
||||
try:
|
||||
answer = session.evaluate(f"{symbol}({arguments})", timeout=max(0, end - time.monotonic()))
|
||||
finally:
|
||||
session.output_observer = None
|
||||
if answer is not None:
|
||||
observe(answer.stdout)
|
||||
return answer.stdout
|
||||
return None
|
||||
|
||||
failure = ""
|
||||
try:
|
||||
body = self._session_stage(query, run)
|
||||
except (OSError, RuntimeError) as exc:
|
||||
body = None
|
||||
failure = f"session_{query}:{exc}"
|
||||
if self.timed_out:
|
||||
self.close_session()
|
||||
return streamed
|
||||
parsed = extract_payload(body) if body is not None else None
|
||||
if not isinstance(parsed, dict):
|
||||
# Discard markers from an unverified attempt before replaying the script.
|
||||
self.asked[:] = prior_asked
|
||||
self.answers = prior_answers
|
||||
self.completion_counts = prior_counts
|
||||
self._fallback_session(failure or session.failure or f"session_{query}_payload_missing")
|
||||
return None
|
||||
if query == "census":
|
||||
self.stages[-1]["files"] = int(parsed.get("files", 0))
|
||||
if self.stages[-1]["files"] <= 0:
|
||||
self.fatal = "query census read zero files"
|
||||
return body
|
||||
|
||||
def _apply_overlays(self, cpg_path, scratch):
|
||||
session = self._session_for(cpg_path)
|
||||
result = False
|
||||
if session is not None:
|
||||
body = self._session_payload(cpg_path, "overlay", {"cpgFile": str(cpg_path)})
|
||||
saved = session.scratch / "workspace" / cpg_path.name / "cpg.bin"
|
||||
if body is not None and saved.is_file():
|
||||
import shutil
|
||||
|
||||
shutil.copyfile(saved, cpg_path)
|
||||
result = True
|
||||
elif body is not None:
|
||||
self._fallback_session("session_overlay_saved_graph_missing")
|
||||
if not result:
|
||||
result = super()._apply_overlays(cpg_path, scratch)
|
||||
# An optional overlay failure must not prevent the taint-only attempt.
|
||||
if self.fatal == ENGINE_BUG and not self.timed_out:
|
||||
self.retry_pending = True
|
||||
self.fatal = ""
|
||||
return result
|
||||
|
||||
def _batch_once(self, cpg_path, query, requests):
|
||||
if self.fatal or self.timed_out:
|
||||
return None
|
||||
if not self.retry_pending:
|
||||
answers = super()._batch_once(cpg_path, query, requests)
|
||||
if self.fatal != ENGINE_BUG:
|
||||
return answers
|
||||
if query != "taint" or self.timed_out or self.retries:
|
||||
self.fatal = ENGINE_BUG
|
||||
return None
|
||||
self.retry_pending = False
|
||||
self.fatal = ""
|
||||
retry = {"reason": ENGINE_BUG, "mode": "taint-only", "completed": False}
|
||||
self.retries.append(retry)
|
||||
# Do not reuse rows/markers from the failed attempt as completion evidence.
|
||||
self.asked.clear()
|
||||
self.answers.clear()
|
||||
self.completion_counts.clear()
|
||||
original = self.queries_dir
|
||||
with tempfile.TemporaryDirectory(prefix="trace-taint-", dir=cpg_path.parent) as scratch:
|
||||
script = (original / "taint.sc").read_text()
|
||||
marker = "importCpg(cpgFile)"
|
||||
if script.count(marker) != 1:
|
||||
self.fatal = ENGINE_BUG
|
||||
return None
|
||||
# Loading without enhancement skips the failing JS postprocessing passes.
|
||||
# A raw graph without dataflow cannot establish a negative answer.
|
||||
script = script.replace(
|
||||
marker,
|
||||
"importCpg(cpgFile, enhance = false)\n"
|
||||
f' require(cpg.metaData.overlays.l.contains("{DATAFLOW_OVERLAY}"), '
|
||||
'"taint-only retry requires saved dataflowOss overlay")',
|
||||
)
|
||||
self.queries_dir = Path(scratch)
|
||||
(self.queries_dir / "taint.sc").write_text(script)
|
||||
try:
|
||||
answers = super()._batch_once(cpg_path, query, requests)
|
||||
finally:
|
||||
self.queries_dir = original
|
||||
retry["completed"] = answers is not None and not self.fatal and not self.timed_out
|
||||
if not retry["completed"]:
|
||||
self.fatal = ENGINE_BUG
|
||||
return answers
|
||||
|
||||
def _run(self, command, *, timeout, cwd=None):
|
||||
if self.fatal or self.timed_out:
|
||||
return None
|
||||
if time.monotonic() >= self.deadline:
|
||||
self.timed_out = True
|
||||
return None
|
||||
started = time.monotonic()
|
||||
end = min(self.deadline, started + timeout)
|
||||
question_end = end
|
||||
binary = Path(command[0]).name
|
||||
frontend = binary in FRONTENDS
|
||||
query = binary == "joern" and "--script" in command
|
||||
output = bytearray()
|
||||
pending = b""
|
||||
process = subprocess.Popen(
|
||||
command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, cwd=cwd, env=self._jvm_env(), start_new_session=True
|
||||
)
|
||||
try:
|
||||
with selectors.DefaultSelector() as selector:
|
||||
selector.register(process.stdout, selectors.EVENT_READ)
|
||||
while selector.get_map():
|
||||
if time.monotonic() >= min(end, question_end):
|
||||
self.timed_out = True
|
||||
break
|
||||
for key, _ in selector.select(min(0.2, max(0, min(end, question_end) - time.monotonic()))):
|
||||
chunk = os.read(key.fileobj.fileno(), 65536)
|
||||
if not chunk:
|
||||
selector.unregister(key.fileobj)
|
||||
continue
|
||||
output.extend(chunk)
|
||||
pending += chunk
|
||||
while b"\n" in pending:
|
||||
line, pending = pending.split(b"\n", 1)
|
||||
try:
|
||||
event = json.loads(line)
|
||||
except ValueError:
|
||||
continue
|
||||
if not isinstance(event, dict):
|
||||
continue
|
||||
if "__question__" in event:
|
||||
self.asked.append(event["__question__"])
|
||||
question_end = min(end, time.monotonic() + self.question_deadline)
|
||||
if "id" in event and "rows" in event:
|
||||
self.answers[event["id"]] = event["rows"]
|
||||
self.completion_counts[event["id"]] += 1
|
||||
question_end = end
|
||||
self.checkpoint()
|
||||
if not self.timed_out:
|
||||
process.wait(timeout=max(0.01, end - time.monotonic()))
|
||||
finally:
|
||||
with contextlib.suppress(ProcessLookupError):
|
||||
os.killpg(process.pid, signal.SIGKILL)
|
||||
process.wait(timeout=5)
|
||||
process.stdout.close()
|
||||
elapsed = time.monotonic() - started
|
||||
if frontend or query:
|
||||
stage = {
|
||||
"command": binary,
|
||||
"argv": command,
|
||||
"heap_mb": self._heap_mb(),
|
||||
"java_tool_options": self._jvm_env()["JAVA_TOOL_OPTIONS"],
|
||||
"seconds": elapsed,
|
||||
"exit": process.returncode,
|
||||
"oom": b"OutOfMemoryError" in output,
|
||||
}
|
||||
lines = output.decode(errors="replace").splitlines()
|
||||
stage["labelled_file_logs"] = {
|
||||
file: [line for line in lines if file in line or Path(file).name in line]
|
||||
for file in self.labelled_files
|
||||
if any(file in line or Path(file).name in line for line in lines)
|
||||
}
|
||||
self.stages.append(stage)
|
||||
if query and process.returncode != 0:
|
||||
# stderr is merged into stdout above; drop JVM frames before taking
|
||||
# the tail so they cannot crowd out the exception's message.
|
||||
lines = [
|
||||
line.strip()
|
||||
for line in output.decode(errors="replace").splitlines()
|
||||
if line.strip() and not line.strip().startswith(("at ", "... "))
|
||||
]
|
||||
stage["output_tail"] = "\n".join(lines[-20:])
|
||||
if not self.timed_out:
|
||||
self.fatal = f"joern query exited {process.returncode}"
|
||||
if frontend:
|
||||
# Capture the exact frontend output before build failure cleanup or overlays.
|
||||
output_arg = next((i + 1 for i, arg in enumerate(command[:-1]) if arg in {"-o", "--output"}), None)
|
||||
cpg_path = (Path(cwd or Path.cwd()) / command[output_arg]).resolve() if output_arg is not None else None
|
||||
stage["cpg_path"] = str(cpg_path) if cpg_path is not None else None
|
||||
stage["cpg_bytes"] = cpg_path.stat().st_size if cpg_path is not None and cpg_path.is_file() else 0
|
||||
if stage["cpg_bytes"] <= MIN_CPG_BYTES:
|
||||
self.fatal = "frontend wrote no CPG"
|
||||
elif elapsed < 5 and not self.fatal:
|
||||
self.fatal = f"implausibly fast JVM step: {command[0]} in {elapsed:.3f}s (<5s)"
|
||||
if b"ObjectPropertyCallLinker failed" in output and b"Assignment statement with 3 arguments" in output:
|
||||
self.fatal = ENGINE_BUG
|
||||
stage["engine_bug"] = ENGINE_BUG
|
||||
if b"OutOfMemoryError" in output:
|
||||
self.fatal = "JVM out of memory"
|
||||
self.checkpoint()
|
||||
return subprocess.CompletedProcess(command, process.returncode if not self.fatal else 1, output.decode(errors="replace"), "")
|
||||
|
||||
|
||||
def unit_record(unit, status, reason="", **extra):
|
||||
return {
|
||||
**unit,
|
||||
"status": status,
|
||||
"reason": reason,
|
||||
"witness_rows": [],
|
||||
"traces": [],
|
||||
"instrument": {"files": [], "bytes": 0, "cpg_path": None, "cpg_bytes": 0, "jvm": []},
|
||||
"questions": {},
|
||||
"questions_asked": [],
|
||||
"questions_completed": [],
|
||||
**extra,
|
||||
}
|
||||
|
||||
|
||||
def worker_language(pin, root, out, deadline, question_deadline, persist=write_json, heap_profile="vm"):
|
||||
proof = {
|
||||
"files": [],
|
||||
"bytes": 0,
|
||||
"cpg_path": None,
|
||||
"cpg_bytes": 0,
|
||||
"jvm": [],
|
||||
"heap_mb": 1700 if heap_profile == "cluster" else HEAP_MB,
|
||||
"heap_profile": heap_profile,
|
||||
"java_tool_options": CLUSTER_JVM_FLAGS
|
||||
if heap_profile == "cluster"
|
||||
else f"{os.environ.get('JAVA_TOOL_OPTIONS', '').strip()} -Xmx{HEAP_MB}m".strip(),
|
||||
"excludes": sorted(EXCLUDED_DIRS),
|
||||
"materialization": pin.get("materialization", {}),
|
||||
}
|
||||
record = {
|
||||
**pin,
|
||||
"done": False,
|
||||
"instrument": proof,
|
||||
"units": [unit_record(u, "unsupported" if not u["supported"] else "timeout", u["reason"]) for u in pin["units"]],
|
||||
}
|
||||
|
||||
def checkpoint():
|
||||
# Checkpoints retain raw completed rows even if the outer container deadline fires.
|
||||
# They remain timeout until the finished JVM's instrument proof is validated.
|
||||
for stage in backend.stages:
|
||||
if "cpg_path" in stage:
|
||||
proof["cpg_path"] = stage["cpg_path"]
|
||||
proof["cpg_bytes"] = stage["cpg_bytes"]
|
||||
if getattr(backend, "session_fallback", ""):
|
||||
proof["session_fallback"] = backend.session_fallback
|
||||
for row in record["units"]:
|
||||
rid = row["unit"]
|
||||
row["instrument"] = proof
|
||||
row["questions_asked"] = [rid] if rid in backend.asked else []
|
||||
if rid in backend.answers:
|
||||
row["questions_completed"] = [rid]
|
||||
row["witness_rows"] = [r for r in backend.answers[rid] if isinstance(r, dict) and "kind" not in r and "sink" in r]
|
||||
persist(out, record)
|
||||
|
||||
backend = MeasuredBackend(deadline, question_deadline, checkpoint, **({"heap_profile": heap_profile} if heap_profile != "vm" else {}))
|
||||
proof["jvm"] = backend.stages
|
||||
proof["retries"] = getattr(backend, "retries", [])
|
||||
backend.labelled_files = sorted({u["file"] for u in pin["units"] if u["supported"]})
|
||||
cpg = None
|
||||
try:
|
||||
active = [u for u in pin["units"] if u["supported"]]
|
||||
for file in sorted({u["file"] for u in active}):
|
||||
data = safe_path(root, file).read_bytes()
|
||||
proof["files"].append({"file": file, "bytes": len(data), "sha256": hashlib.sha256(data).hexdigest()})
|
||||
proof["bytes"] += len(data)
|
||||
print(f"read {file}: {len(data)} bytes", flush=True)
|
||||
if not proof["bytes"]:
|
||||
raise ValueError("instrument read zero files or zero bytes")
|
||||
languages = {"javascript" if u["language"] == "typescript" else u["language"] for u in active}
|
||||
if len(languages) != 1:
|
||||
raise ValueError(f"one pin has multiple frontend languages: {sorted(languages)}")
|
||||
language = next(iter(languages))
|
||||
requests = {rid: {key: _render(value) for key, value in params.items()} for rid, params in pin["questions"].items()}
|
||||
if set(requests) != {u["unit"] for u in active}:
|
||||
raise ValueError("prepared questions do not match supported units")
|
||||
record["questions"] = requests
|
||||
for unit_row in record["units"]:
|
||||
unit_row["questions"] = {unit_row["unit"]: requests[unit_row["unit"]]} if unit_row["unit"] in requests else {}
|
||||
persist(out, record)
|
||||
cpg = backend.build(root, language=language)
|
||||
if cpg is None:
|
||||
raise ValueError(backend.fatal or backend.last_failure or ("pin deadline" if backend.timed_out else "CPG build failed"))
|
||||
if proof["cpg_path"] is None:
|
||||
proof["cpg_path"] = str(cpg.cpg_path)
|
||||
proof["cpg_bytes"] = cpg.cpg_path.stat().st_size if cpg.cpg_path.is_file() else 0
|
||||
failure = instrument_failure(proof)
|
||||
if failure:
|
||||
raise ValueError(failure)
|
||||
if cpg.run_batch_once is None:
|
||||
raise ValueError("backend offers no batch operation")
|
||||
answers = cpg.run_batch_once("taint", requests) or {}
|
||||
record["census"] = answers.get("__census__", [])
|
||||
failure = backend.fatal or instrument_failure(proof)
|
||||
if not backend.asked and not backend.timed_out:
|
||||
failure = failure or "query produced no markers"
|
||||
census = record["census"]
|
||||
if census and int(census[0].get("files", 0)) <= 0:
|
||||
failure = failure or "query census read zero files"
|
||||
elif not census and not backend.timed_out:
|
||||
failure = failure or "query census is missing"
|
||||
for index, unit in enumerate(pin["units"]):
|
||||
if not unit["supported"]:
|
||||
continue
|
||||
rid = unit["unit"]
|
||||
extra = {
|
||||
"questions": {rid: requests[rid]},
|
||||
"questions_asked": [rid] if rid in backend.asked else [],
|
||||
"questions_completed": [rid] if rid in answers else [],
|
||||
"instrument": proof,
|
||||
}
|
||||
rows = answers.get(rid)
|
||||
# A batch across backend shards is complete only when every shard answered.
|
||||
# The legacy merger can return a union even when one shard was killed.
|
||||
shards = int(census[0].get("shards", 1)) if census else 1
|
||||
shard_incomplete = shards > 1 and getattr(backend, "completion_counts", {}).get(rid, 0) < shards
|
||||
if rows is not None:
|
||||
raw_flows = [r for r in rows if isinstance(r, dict) and "kind" not in r and "sink" in r]
|
||||
extra.update(witness_rows=raw_flows, traces=[parse_trace(r) for r in raw_flows])
|
||||
if shard_incomplete:
|
||||
extra["questions_completed"] = []
|
||||
names = census[0].get("file_names", []) if census else []
|
||||
file_missing = not any(str(p).removeprefix("./") == unit["file"] or str(p).endswith("/" + unit["file"]) for p in names)
|
||||
dropped = file_missing and rows is not None
|
||||
if failure or dropped:
|
||||
status, reason = "failed", failure or "labelled file was dropped or absent from query census"
|
||||
if dropped:
|
||||
logs = [line for stage in backend.stages for line in stage.get("labelled_file_logs", {}).get(unit["file"], [])]
|
||||
reason += "\n" + ("\n".join(dict.fromkeys(logs)) if logs else "frontend emitted no log lines mentioning labelled file")
|
||||
elif rows is None or shard_incomplete:
|
||||
status, reason = ("timeout" if backend.timed_out else "failed"), "question did not complete"
|
||||
elif not rows and rid not in backend.asked:
|
||||
status, reason = "failed", "query produced no markers"
|
||||
else:
|
||||
flows = [r for r in rows if isinstance(r, dict) and "kind" not in r and "sink" in r]
|
||||
if any(not row.get("trace") for row in flows):
|
||||
raise ValueError("taint witness returned without requested trace")
|
||||
extra.update(witness_rows=flows, traces=[parse_trace(r) for r in flows])
|
||||
status, reason = ("path" if flows else "asked-nothing"), ""
|
||||
record["units"][index] = unit_record(unit, status, reason, **extra)
|
||||
except (OSError, ValueError, RuntimeError) as exc:
|
||||
for index, unit in enumerate(pin["units"]):
|
||||
if unit["supported"]:
|
||||
record["units"][index] = unit_record(
|
||||
unit,
|
||||
"timeout" if backend.timed_out and not backend.fatal else "failed",
|
||||
str(exc),
|
||||
instrument=proof,
|
||||
questions={unit["unit"]: record["questions"][unit["unit"]]} if unit["unit"] in record.get("questions", {}) else {},
|
||||
witness_rows=[r for r in backend.answers.get(unit["unit"], []) if isinstance(r, dict) and "sink" in r],
|
||||
questions_completed=[unit["unit"]] if unit["unit"] in backend.answers else [],
|
||||
questions_asked=[unit["unit"]] if unit["unit"] in backend.asked else [],
|
||||
)
|
||||
finally:
|
||||
if hasattr(backend, "close_session"):
|
||||
backend.close_session()
|
||||
if cpg and cpg.cleanup:
|
||||
cpg.cleanup()
|
||||
for stage in backend.stages:
|
||||
if stage["command"] == "joern" and stage.get("exit") and "output_tail" in stage:
|
||||
detail = f"joern query exited {stage['exit']}: {stage['output_tail'] or 'no output'}"
|
||||
for row in record["units"]:
|
||||
if row["supported"] and row["status"] in {"failed", "timeout"}:
|
||||
row["reason"] = f"{row['reason']}\n{detail}".strip()
|
||||
record["questions_asked"] = list(dict.fromkeys(backend.asked))
|
||||
record["questions_completed"] = [rid for u in record["units"] for rid in u["questions_completed"]]
|
||||
record["done"] = True
|
||||
persist(out, record)
|
||||
return record
|
||||
|
||||
|
||||
def worker(pin, root, out, deadline, question_deadline, heap_profile="vm"):
|
||||
"""Run language partitions sequentially under one shared container deadline."""
|
||||
languages = sorted({"javascript" if u["language"] == "typescript" else u["language"] for u in pin["units"] if u["supported"]})
|
||||
if len(languages) <= 1:
|
||||
return worker_language(pin, root, out, deadline, question_deadline, heap_profile=heap_profile)
|
||||
end = time.monotonic() + deadline
|
||||
record = {
|
||||
**pin,
|
||||
"done": False,
|
||||
"units": [
|
||||
unit_record(u, "timeout" if u["supported"] else "unsupported", "pin deadline" if u["supported"] else u["reason"])
|
||||
for u in pin["units"]
|
||||
],
|
||||
"instruments": {},
|
||||
}
|
||||
|
||||
def persist(_, partial):
|
||||
updates = {u["unit"]: u for u in partial["units"]}
|
||||
record["units"] = [updates.get(u["unit"], u) for u in record["units"]]
|
||||
record["instruments"][language] = partial["instrument"]
|
||||
record["questions_asked"] = [rid for u in record["units"] for rid in u["questions_asked"]]
|
||||
record["questions_completed"] = [rid for u in record["units"] for rid in u["questions_completed"]]
|
||||
write_json(out, record)
|
||||
|
||||
write_json(out, record)
|
||||
for language in languages:
|
||||
remaining = end - time.monotonic()
|
||||
if remaining <= 0:
|
||||
break
|
||||
units = [
|
||||
u for u in pin["units"] if u["supported"] and ("javascript" if u["language"] == "typescript" else u["language"]) == language
|
||||
]
|
||||
questions = {u["unit"]: pin["questions"][u["unit"]] for u in units}
|
||||
worker_language(
|
||||
{**pin, "units": units, "questions": questions}, root, out, remaining, question_deadline, persist, heap_profile=heap_profile
|
||||
)
|
||||
record["done"] = True
|
||||
write_json(out, record)
|
||||
return record
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--worker", type=Path, required=True)
|
||||
parser.add_argument("--deadline", type=int, default=900)
|
||||
parser.add_argument("--question-deadline", type=int, default=120)
|
||||
parser.add_argument("--heap-profile", choices=("cluster", "vm"), default="vm")
|
||||
parser.add_argument("--root", type=Path, default=Path("/case"))
|
||||
parser.add_argument("--output", type=Path, default=Path("/out/result.json"))
|
||||
args = parser.parse_args()
|
||||
if min(args.deadline, args.question_deadline) <= 0:
|
||||
parser.error("deadlines must be positive")
|
||||
worker(json.loads(args.worker.read_text()), args.root, args.output, args.deadline, args.question_deadline, args.heap_profile)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,315 @@
|
||||
"""Image-resident queue workers. Requires strongly consistent S3 GET/PUT/LIST.
|
||||
|
||||
Per-task Lamport bakery tickets use only ordinary object operations. A choosing
|
||||
marker precedes ticket selection; contenders wait for choosing peers and lower
|
||||
(ticket, worker) pairs. Late arrivals cannot displace a running owner. Leases
|
||||
assume bounded clock skew (less than MARGIN / 2) and a worker killed by its deadline.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import contextlib
|
||||
import ctypes
|
||||
import io
|
||||
import json
|
||||
import logging
|
||||
import multiprocessing
|
||||
import os
|
||||
import signal
|
||||
import socket
|
||||
import tarfile
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
from pathlib import Path
|
||||
|
||||
from engine_trace_worker import unit_record, worker
|
||||
|
||||
from openultrasast.plane.memory import S3Client, s3_settings
|
||||
|
||||
PENDING = "engine-queue/pending/"
|
||||
CLAIMED = "engine-queue/claimed/"
|
||||
RESULTS = "engine-results/"
|
||||
MARGIN = 60
|
||||
LOG = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def client_from_env():
|
||||
settings = s3_settings(os.environ)
|
||||
bucket = os.environ.get("S3_BUCKET")
|
||||
if not bucket:
|
||||
raise ValueError("S3_BUCKET is required")
|
||||
return S3Client(bucket=bucket, **settings)
|
||||
|
||||
|
||||
def read(client, key):
|
||||
value = client.get(key)
|
||||
return json.loads(value[0]) if value else None
|
||||
|
||||
|
||||
def put(client, key, value):
|
||||
client.put(key, json.dumps(value).encode(), {})
|
||||
|
||||
|
||||
def result_key(task):
|
||||
return f"{RESULTS}{task['run']}/{task['id']}.json"
|
||||
|
||||
|
||||
def pack(root):
|
||||
stream = io.BytesIO()
|
||||
with tarfile.open(fileobj=stream, mode="w") as archive:
|
||||
archive.add(root, arcname=".")
|
||||
return stream.getvalue()
|
||||
|
||||
|
||||
def unpack(data, root):
|
||||
with tarfile.open(fileobj=io.BytesIO(data)) as archive:
|
||||
# Source snapshots need only regular files and directories, never links/devices.
|
||||
for member in archive.getmembers():
|
||||
target = (root / member.name).resolve()
|
||||
if not target.is_relative_to(root.resolve()) or not (member.isfile() or member.isdir()):
|
||||
raise ValueError("unsafe source archive")
|
||||
archive.extractall(root, filter="data")
|
||||
|
||||
|
||||
class Queue:
|
||||
def __init__(self, client, identity=None, *, now=time.time, pause=time.sleep, stop=None):
|
||||
self.client = client
|
||||
self.identity = identity or f"{socket.gethostname()}-{uuid.uuid4().hex}"
|
||||
self.now, self.pause = now, pause
|
||||
self.stop = stop or threading.Event()
|
||||
|
||||
def markers(self, task_id=None):
|
||||
result = []
|
||||
for key in self.client.keys(CLAIMED):
|
||||
if task_id is None or key.rsplit("/", 1)[-1] == task_id:
|
||||
marker = read(self.client, key)
|
||||
if marker:
|
||||
result.append((key, marker))
|
||||
return result
|
||||
|
||||
def tasks(self):
|
||||
tasks = {}
|
||||
for key in self.client.keys(PENDING):
|
||||
task = read(self.client, key)
|
||||
if task:
|
||||
tasks[task["task_id"]] = task
|
||||
# A dead worker may have removed pending already. Its task remains in the lease.
|
||||
for _, marker in self.markers():
|
||||
if marker["expires"] < self.now():
|
||||
task = marker["task"]
|
||||
tasks.setdefault(task["task_id"], task)
|
||||
return list(tasks.values())
|
||||
|
||||
def claim(self, task):
|
||||
if read(self.client, result_key(task)) is not None or self.stop.is_set():
|
||||
return None
|
||||
key = f"{CLAIMED}{self.identity}/{task['task_id']}"
|
||||
marker = dict(task=task, ticket=0, expires=self.now() + task["deadline"] + MARGIN)
|
||||
put(self.client, key, marker)
|
||||
try:
|
||||
live = [m for _, m in self.markers(task["task_id"]) if m["expires"] > self.now()]
|
||||
marker["ticket"] = 1 + max((m["ticket"] for m in live), default=0)
|
||||
put(self.client, key, marker)
|
||||
while not self.stop.is_set():
|
||||
# Do not start once the execution budget no longer fits within our lease.
|
||||
if self.now() >= marker["expires"] - task["deadline"] - MARGIN / 2:
|
||||
break
|
||||
peers = [(k, m) for k, m in self.markers(task["task_id"]) if k != key and m["expires"] > self.now()]
|
||||
if any(m["ticket"] == 0 for _, m in peers):
|
||||
self.pause(0.1)
|
||||
continue
|
||||
if any((m["ticket"], k) < (marker["ticket"], key) for k, m in peers):
|
||||
break # loser backs off and can try another pending task
|
||||
if read(self.client, key) != marker or read(self.client, result_key(task)) is not None:
|
||||
break
|
||||
self.client.delete(PENDING + task["task_id"])
|
||||
return key
|
||||
except Exception:
|
||||
self.client.delete(key)
|
||||
raise
|
||||
self.client.delete(key)
|
||||
return None
|
||||
|
||||
def process_one(self, execute=None):
|
||||
for task in self.tasks():
|
||||
key = self.claim(task)
|
||||
if key is None:
|
||||
continue
|
||||
try:
|
||||
lease = read(self.client, key)
|
||||
if lease is None:
|
||||
continue
|
||||
budget = min(task["deadline"], lease["expires"] - self.now() - MARGIN / 2)
|
||||
if budget <= 0:
|
||||
continue # Leave the expired lease's task available to recovery.
|
||||
result = (execute or execute_task)(self.client, {**task, "runtime_budget": budget})
|
||||
result.update(executor="queue", worker=self.identity, pod=socket.gethostname(), image=os.environ.get("ENGINE_IMAGE", ""))
|
||||
put(self.client, result_key(task), result)
|
||||
# Publish the result before removing either the input or the claim.
|
||||
self.client.delete(task["input"])
|
||||
for old_key, old in self.markers(task["task_id"]):
|
||||
if old_key == key or old["expires"] < self.now():
|
||||
self.client.delete(old_key)
|
||||
return True
|
||||
except Exception:
|
||||
# Keep the lease and task for recovery, without logging credential-bearing errors.
|
||||
LOG.error("task failed; lease retained for recovery")
|
||||
return False
|
||||
return False
|
||||
|
||||
|
||||
def _child(pin, root, out, deadline, question_deadline):
|
||||
os.setsid()
|
||||
signal.signal(signal.SIGTERM, signal.SIG_DFL)
|
||||
worker(pin, root, out, deadline, question_deadline)
|
||||
|
||||
|
||||
def adopt_orphans():
|
||||
# Linux PR_SET_CHILD_SUBREAPER: escaped Joern sessions reparent here, not
|
||||
# outside the service. There is exactly one task process tree per worker.
|
||||
if ctypes.CDLL(None, use_errno=True).prctl(36, 1, 0, 0, 0) != 0:
|
||||
raise OSError("cannot enable task descendant cleanup")
|
||||
|
||||
|
||||
def kill_children():
|
||||
"""Freeze parents before walking children, including new-session JVMs."""
|
||||
seen = set()
|
||||
|
||||
def freeze(parent):
|
||||
for entry in Path("/proc").iterdir():
|
||||
if not entry.name.isdigit() or int(entry.name) in seen:
|
||||
continue
|
||||
try:
|
||||
# comm can contain spaces or parentheses. Fields after it begin at state.
|
||||
fields = (entry / "stat").read_text().rsplit(")", 1)[1].split()
|
||||
if int(fields[1]) != parent:
|
||||
continue
|
||||
pid = int(entry.name)
|
||||
os.kill(pid, signal.SIGSTOP)
|
||||
seen.add(pid)
|
||||
freeze(pid)
|
||||
except (FileNotFoundError, ProcessLookupError, PermissionError):
|
||||
continue
|
||||
|
||||
# A child that exited during discovery can reparent descendants to this
|
||||
# subreaper. Repeat until every remaining descendant is frozen.
|
||||
while True:
|
||||
count = len(seen)
|
||||
freeze(os.getpid())
|
||||
if len(seen) == count:
|
||||
break
|
||||
for pid in seen:
|
||||
with contextlib.suppress(ProcessLookupError):
|
||||
os.kill(pid, signal.SIGKILL)
|
||||
|
||||
|
||||
def execute_task(client, task):
|
||||
adopt_orphans()
|
||||
pin = task["pin"]
|
||||
started = time.monotonic()
|
||||
budget = task.get("runtime_budget", task["deadline"])
|
||||
status, reason, code = "failed", "worker produced no result", None
|
||||
with tempfile.TemporaryDirectory(prefix="engine-task-") as scratch:
|
||||
root, out = Path(scratch) / "case", Path(scratch) / "result.json"
|
||||
root.mkdir()
|
||||
try:
|
||||
if task["deadline"] > 900 or task["deadline"] <= 0:
|
||||
raise ValueError("deadline exceeds deployment grace")
|
||||
if task.get("image") != os.environ.get("ENGINE_IMAGE"):
|
||||
raise ValueError("task image differs from deployed image")
|
||||
source = client.get(task["input"])
|
||||
if source is None:
|
||||
raise ValueError("missing source")
|
||||
unpack(source[0], root)
|
||||
# The shipped worker also checks frontend input and graph output for every language.
|
||||
files = [p for p in root.rglob("*") if p.is_file()]
|
||||
if not files:
|
||||
raise ValueError("empty source")
|
||||
LOG.info("input proof: first file bytes=%d", len(files[0].read_bytes()))
|
||||
if time.monotonic() - started >= budget:
|
||||
status, reason = "timeout", "task input deadline"
|
||||
raise TimeoutError(reason)
|
||||
process = multiprocessing.get_context("fork").Process(target=_child, args=(pin, root, out, budget, task["question_deadline"]))
|
||||
process.start()
|
||||
try:
|
||||
process.join(max(0, budget - (time.monotonic() - started)))
|
||||
if process.is_alive():
|
||||
status, reason = "timeout", "task deadline"
|
||||
else:
|
||||
reason = f"worker exit {process.exitcode}"
|
||||
code = process.exitcode
|
||||
finally:
|
||||
# Joern creates separate sessions. Kill the whole descendant tree,
|
||||
# not merely the Python wrapper's process group.
|
||||
kill_children()
|
||||
process.join()
|
||||
# Reap adopted JVMs and their launchers after the direct child.
|
||||
while True:
|
||||
try:
|
||||
os.waitpid(-1, 0)
|
||||
except ChildProcessError:
|
||||
break
|
||||
record = json.loads(out.read_text()) if out.exists() else {**pin, "units": []}
|
||||
except Exception:
|
||||
record = {**pin, "units": []}
|
||||
reason = "worker input or execution failed"
|
||||
if not record.get("done") or code != 0:
|
||||
previous = {u["unit"]: u for u in record["units"]}
|
||||
record["units"] = [
|
||||
previous[u["unit"]]
|
||||
if code is None and previous.get(u["unit"], {}).get("status") in {"path", "asked-nothing"}
|
||||
else unit_record(u, status if u["supported"] else "unsupported", reason if u["supported"] else u["reason"])
|
||||
for u in pin["units"]
|
||||
]
|
||||
record.update(done=True, container_exit=code, seconds=time.monotonic() - started)
|
||||
return record
|
||||
|
||||
|
||||
def status(client):
|
||||
return {
|
||||
"pending": len(client.keys(PENDING)),
|
||||
"claimed": len(client.keys(CLAIMED)),
|
||||
"done": len([k for k in client.keys(RESULTS) if k.endswith(".json")]),
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--port", type=int, default=8080)
|
||||
args = parser.parse_args()
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
stop = threading.Event()
|
||||
signal.signal(signal.SIGTERM, lambda *_: stop.set())
|
||||
signal.signal(signal.SIGINT, lambda *_: stop.set())
|
||||
client = client_from_env()
|
||||
queue = Queue(client, stop=stop)
|
||||
|
||||
class Health(BaseHTTPRequestHandler):
|
||||
def do_GET(self):
|
||||
self.send_response(200 if self.path == "/healthz" and not stop.is_set() else 503)
|
||||
self.end_headers()
|
||||
|
||||
def log_message(self, *_):
|
||||
pass
|
||||
|
||||
server = ThreadingHTTPServer(("0.0.0.0", args.port), Health)
|
||||
threading.Thread(target=server.serve_forever, daemon=True).start()
|
||||
try:
|
||||
while not stop.is_set():
|
||||
try:
|
||||
if not queue.process_one():
|
||||
stop.wait(2)
|
||||
except Exception:
|
||||
LOG.error("queue unavailable; retrying")
|
||||
stop.wait(5)
|
||||
finally:
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,239 @@
|
||||
"""Offline audit of completed Joern results; never launches an analyzer or reads corpora.
|
||||
|
||||
Missing results stay in coverage, not signal denominators. Observed non-path rows
|
||||
score zero for path features. Sink kinds are one-hot indicators, never ordinal
|
||||
category IDs. Both signed and best-direction metrics expose reversed shortcuts.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import ast
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
|
||||
from slice_leak_audit import THRESHOLD, complete_pairs, metrics
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
UNITS = ROOT / "plane/experiments/exp-005-graph-slice.units.jsonl"
|
||||
RESULTS = Path.home() / "ousast-results/plane/engine-trace"
|
||||
OUTPUT = ROOT / "benchmarks/measurements/2026-10-04-joern-slice-leak-audit/record.json"
|
||||
STATUSES = ("path", "asked-nothing", "failed", "timeout", "unsupported", "missing")
|
||||
ASKED = {"path", "asked-nothing"}
|
||||
SIGNALS = ("has_path", "shortest_path_steps", "sanitized", "guard_present", "asked")
|
||||
ASYMMETRY = ("vulnerable_only", "fixed_only", "both", "neither", "not_both_asked")
|
||||
|
||||
|
||||
def materialization_version():
|
||||
# Read the producer's literal without importing its execution dependencies.
|
||||
tree = ast.parse(Path(__file__).with_name("engine_trace.py").read_bytes())
|
||||
for node in tree.body:
|
||||
if isinstance(node, ast.Assign) and any(isinstance(t, ast.Name) and t.id == "MATERIALIZATION_VERSION" for t in node.targets):
|
||||
return ast.literal_eval(node.value)
|
||||
raise ValueError("Producer has no materialization version")
|
||||
|
||||
|
||||
def features(row):
|
||||
path = row["status"] == "path"
|
||||
traces = row.get("traces", []) if path else []
|
||||
candidates = [(i, t) for i, t in enumerate(traces) if t.get("steps")]
|
||||
index, trace = min(candidates, key=lambda it: len(it[1]["steps"])) if candidates else (None, {})
|
||||
witnesses = row.get("witness_rows", [])
|
||||
witness = witnesses[index] if index is not None and index < len(witnesses) else {}
|
||||
guard = trace.get("guard", {})
|
||||
sink = witness.get("sinkKind")
|
||||
if not sink:
|
||||
# Retain only the call target: arguments, literals and code are not kinds.
|
||||
match = re.search(r"([\w:$>.\\-]+)\s*\(", str(witness.get("sink", "")))
|
||||
sink = match[1] if match else "unknown"
|
||||
return {
|
||||
"has_path": int(path),
|
||||
"shortest_path_steps": len(trace["steps"]) if trace else None if path else 0,
|
||||
"sanitized": int(bool(trace.get("sanitized", witness.get("sanitized", False)))) if trace else None if path else 0,
|
||||
"guard_present": int(
|
||||
bool(guard.get("text") or guard.get("bounded") or witness.get("bound") or witness.get("bounded"))
|
||||
or any("guard" in s.get("roles", []) for s in trace.get("steps", []))
|
||||
)
|
||||
if trace
|
||||
else None
|
||||
if path
|
||||
else 0,
|
||||
"sink_kind": str(sink) if path else "no-path",
|
||||
"asked": int(row["status"] in ASKED),
|
||||
}
|
||||
|
||||
|
||||
def scored(rows, signal):
|
||||
eligible = [r for r in rows if r[signal] is not None]
|
||||
result = metrics(eligible, signal)
|
||||
result["excluded_rows"] = len(rows) - len(eligible)
|
||||
result["flagged"] = (result["pairs"] >= 10 and (result["best_direction_within_pair_accuracy"] or 0) >= THRESHOLD) or (
|
||||
result["rows"] >= 10 and (result["best_direction_pooled_auc"] or 0) >= THRESHOLD
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def signal_metrics(rows, kinds):
|
||||
return {
|
||||
**{s: scored(rows, s) for s in SIGNALS},
|
||||
"sink_kind": {kind: scored([{**r, "is_kind": int(r["sink_kind"] == kind)} for r in rows], "is_kind") for kind in kinds},
|
||||
}
|
||||
|
||||
|
||||
def scope_report(rows, kinds):
|
||||
pairs = complete_pairs(rows)
|
||||
asymmetry = Counter()
|
||||
honest = []
|
||||
for v, f in pairs:
|
||||
if not (v["asked"] and f["asked"]):
|
||||
asymmetry["not_both_asked"] += 1
|
||||
else:
|
||||
honest.extend((v, f))
|
||||
asymmetry[
|
||||
"both"
|
||||
if v["has_path"] and f["has_path"]
|
||||
else "vulnerable_only"
|
||||
if v["has_path"]
|
||||
else "fixed_only"
|
||||
if f["has_path"]
|
||||
else "neither"
|
||||
] += 1
|
||||
observed = [r for r in rows if r["status"] != "missing"]
|
||||
return {
|
||||
"rows": len(rows),
|
||||
"coverage": {s: sum(r["status"] == s for r in rows) for s in STATUSES},
|
||||
"complete_pairs": len(pairs),
|
||||
"pair_asymmetry": {s: asymmetry[s] for s in ASYMMETRY},
|
||||
"both_asked_pairs": len(honest) // 2,
|
||||
"signals": signal_metrics(observed, kinds),
|
||||
"both_asked_signals": signal_metrics(honest, kinds),
|
||||
}
|
||||
|
||||
|
||||
def audit(units, results, *, expected_rows=501, expected_pairs=195):
|
||||
if len(units) != expected_rows or len({u["unit"] for u in units}) != expected_rows:
|
||||
raise ValueError(f"Expected {expected_rows} unique units")
|
||||
if len(complete_pairs(units)) != expected_pairs:
|
||||
raise ValueError(f"Expected {expected_pairs} complete pairs")
|
||||
if not results.is_dir():
|
||||
raise ValueError(f"Results directory is unreadable: {results}")
|
||||
version = materialization_version()
|
||||
unit_index = {u["unit"]: u for u in units}
|
||||
matched = {}
|
||||
files = []
|
||||
counts = Counter()
|
||||
for path in sorted(results.glob("*.json")):
|
||||
raw = path.read_bytes()
|
||||
record = json.loads(raw) # Corruption fails loudly; producer uses atomic replace.
|
||||
if not isinstance(record, dict) or "repo" not in record or "pin" not in record:
|
||||
continue
|
||||
stale = record.get("materialization_version") != version
|
||||
unfinished = record.get("done") is not True
|
||||
counts["stale"] += stale
|
||||
counts["unfinished"] += unfinished
|
||||
counts["skipped"] += stale or unfinished
|
||||
files.append((path.name, len(raw)))
|
||||
if stale or unfinished:
|
||||
continue
|
||||
counts["accepted_files"] += 1
|
||||
counts["result_rows_read"] += len(record["units"])
|
||||
for row in record["units"]:
|
||||
if row["unit"] not in unit_index:
|
||||
continue
|
||||
unit = unit_index[row["unit"]]
|
||||
if any(row.get(k) != unit.get(k) for k in ("family", "label", "pair")):
|
||||
raise ValueError(f"Result metadata mismatch: {row['unit']}")
|
||||
if row["status"] not in STATUSES[:-1] or not row.get("source"):
|
||||
raise ValueError(f"Invalid result status/source: {row['unit']}")
|
||||
if row["unit"] in matched:
|
||||
raise ValueError(f"Duplicate current result: {row['unit']}")
|
||||
matched[row["unit"]] = {
|
||||
**unit,
|
||||
"source": row["source"],
|
||||
"status": row["status"],
|
||||
"pin_role": row.get("pin_role"),
|
||||
**features(row),
|
||||
}
|
||||
rows = [
|
||||
matched.get(u["unit"], {**u, "source": u.get("source", "unknown"), "status": "missing", **features({"status": "missing"})})
|
||||
for u in units
|
||||
]
|
||||
kinds = sorted({r["sink_kind"] for r in rows if r["status"] != "missing"})
|
||||
scopes = {"overall": scope_report(rows, kinds)}
|
||||
for field in ("family", "source"):
|
||||
scopes[f"by_{field}"] = {
|
||||
value: scope_report([r for r in rows if r[field] == value], kinds) for value in sorted({r[field] for r in rows})
|
||||
}
|
||||
return {
|
||||
"cost_usd": 0,
|
||||
"materialization_version": version,
|
||||
"threshold": THRESHOLD,
|
||||
"minimum_samples": 10,
|
||||
"unit_rows_read": len(units),
|
||||
"matched_rows": len(matched),
|
||||
"inputs": {
|
||||
**{k: counts[k] for k in ("accepted_files", "result_rows_read", "stale", "unfinished", "skipped")},
|
||||
"files_read": len(files),
|
||||
"bytes_read": sum(size for _, size in files),
|
||||
# Hash a canonical inventory; filenames and paths may identify repositories.
|
||||
"files_sha256": hashlib.sha256(json.dumps(sorted(files), separators=(",", ":")).encode()).hexdigest(),
|
||||
},
|
||||
"semantics": {
|
||||
"signals": "Shortest nonempty trace (first on ties); its sanitizer, guard and corresponding witness sink call target.",
|
||||
"missing": "Coverage only; absent source metadata is unknown. Skipped result units are never consumed.",
|
||||
"non_path": "Observed non-path rows score zero; path rows lacking trace evidence have null trace features.",
|
||||
"asked": "path or asked-nothing=1; failed, timeout, unsupported=0; missing excluded from metrics.",
|
||||
"asymmetry": "Disjoint categories; not_both_asked takes precedence over path asymmetry.",
|
||||
"flags": "Best of both directions, >=0.70 with >=10 pairs for pair accuracy or >=10 rows for AUC.",
|
||||
"sink_kind": "One binary indicator per observed call target, including no-path/unknown; no ordinal encoding.",
|
||||
"limits": "Exploratory associations, not proof of leakage; running input snapshot, no multiple-testing correction.",
|
||||
},
|
||||
**scopes,
|
||||
"units": [{k: r[k] for k in ("unit", "pair", "family", "label", "fold", "source", "status", *SIGNALS, "sink_kind")} for r in rows],
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--results", type=Path, default=RESULTS)
|
||||
parser.add_argument("--units", type=Path, default=UNITS)
|
||||
parser.add_argument("--output", type=Path, default=OUTPUT)
|
||||
args = parser.parse_args()
|
||||
raw = args.units.read_bytes()
|
||||
units = [json.loads(line) for line in raw.splitlines() if line.strip()]
|
||||
report = audit(units, args.results)
|
||||
report["units_input"] = {"bytes": len(raw), "sha256": hashlib.sha256(raw).hexdigest()}
|
||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.output.write_text(json.dumps(report, indent=2, sort_keys=True) + "\n")
|
||||
overall, inputs = report["overall"], report["inputs"]
|
||||
lines = [
|
||||
f"Joern leak audit: $0; materialization_version={report['materialization_version']}",
|
||||
f"Units read: {len(units)}; bytes={len(raw)}; file={args.units}",
|
||||
f"Result files read: {inputs['files_read']}; bytes={inputs['bytes_read']}; accepted={inputs['accepted_files']}",
|
||||
f"Skipped: {inputs['skipped']} (stale={inputs['stale']}, unfinished={inputs['unfinished']}; counts overlap)",
|
||||
f"Result rows read: {inputs['result_rows_read']}; matched={report['matched_rows']}",
|
||||
f"Coverage: {overall['coverage']}",
|
||||
f"Pairs: {overall['complete_pairs']}; both asked={overall['both_asked_pairs']}",
|
||||
f"Asymmetry: {overall['pair_asymmetry']}",
|
||||
]
|
||||
for signal in (*SIGNALS, "sink_kind"):
|
||||
|
||||
def brief(value, signal=signal):
|
||||
if signal == "sink_kind":
|
||||
return {k: brief_metric(m) for k, m in value.items()}
|
||||
return brief_metric(value)
|
||||
|
||||
lines.append(f"{signal}: all={brief(overall['signals'][signal])}; both asked={brief(overall['both_asked_signals'][signal])}")
|
||||
lines.append(f"Family/source metrics and input inventory digest: {args.output}")
|
||||
print("\n".join(lines))
|
||||
|
||||
|
||||
def brief_metric(metric):
|
||||
return {k: metric[k] for k in ("rows", "pairs", "within_pair_accuracy", "pooled_auc", "flagged")}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -141,7 +141,9 @@ def audit(store: MemoryStore, units: Sequence[Mapping[str, Any]], *, expected_ro
|
||||
"limitations": (
|
||||
"Flat IR: no dominance/guards, textual branch order, potential parameter inputs, conservative call returns; no alias analysis."
|
||||
),
|
||||
"units": rows,
|
||||
"units": [
|
||||
{k: r[k] for k in ("unit", "pair", "family", "label", "fold", "source", "status", "has_flow", "slice_length")} for r in rows
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,326 @@
|
||||
{
|
||||
"created": "2026-10-04",
|
||||
"note": "COUNTS ONLY. Batch 3 of real (vulnerable function, fixed function) pairs whose function survives the fix, mined at zero model cost from reviewed GitHub advisories (pip, npm, maven, composer) and added as pointer-pair corpus benchmarks/pairs/advisory-fixes-3 (source `advisory-fixes-3` in learn/sources.toml); labelled, given static feature records, built into the memory store and embedded. No chat-model call, no Docker, no Joern. Candidate, path, function and repository names stay in the catalog; the pairs the reserved-population guard flagged are withheld (found by bisection with the test output discarded; names never printed).",
|
||||
"method": {
|
||||
"enumeration": "gh api /advisories?type=reviewed&ecosystem=<eco>&cwes=<family CWEs>&published=>2023-01-01 per ecosystem (pip, npm, maven, composer) x six compiled families; Go, Ruby and C# not enumerated; newest first",
|
||||
"advisories_enumerated": 8531,
|
||||
"advisories_published_after_2026_06_01": 1855,
|
||||
"automated_screen": "as batch 2 (single family by CWE; linked fix commit, at most 3, one repository; repository not named anywhere under benchmarks/ incl. batches 1-2; excluded v3 file names never opened; no WordPress plugin; one pair per repository; SPDX licence permissive; one-parent commit; the function holding most changed code lines inside its own matcher span, declared once on both sides, body changed, >= 25% of the file's changed code lines) plus three widenings: (a) an advisory linking only a merged pull request uses its single commit or its squash/rebase merge commit; (b) commits of up to 6 non-test source files / 240 changed lines (batch 2: 4 / 120); (c) a second pass over advisories whose CWEs span two families, the family then chosen from the function's diff on review",
|
||||
"file_reads": "commit metadata and the two versions of each changed source file through the contents API into scratch (no clone, no scanner, engine, plane Run or model on these repositories); scratch deleted after",
|
||||
"manual_review": "every automated candidate's in-function diff read (agent, no model call); kept only when the named function itself holds the fix; a weak inline check replaced by a stronger helper call counts as the fix, a function that only forwards a flag or calls a fix made elsewhere does not",
|
||||
"reserved_population_guard": "pytest -q tests/ -k 'reserved or v3' with output discarded, bisected over catalog subsets (18 runs; baseline with an empty batch green); final catalog green",
|
||||
"rename_check": "canonical full_name of every kept repository from the API: no two kept pairs share a repository, none renamed into a name the benchmarks use",
|
||||
"mechanism": "family default as batch 2 (injection/output_encoding/deserialization source_reaches_sink, path path_join_user_input, access_control missing_auth_guard, untrusted_destination validation_strength for CWE-601 else source_reaches_sink); provenance = agent"
|
||||
},
|
||||
"screened": {
|
||||
"advisories_examined": 8805,
|
||||
"outcomes": {
|
||||
"no fix commit linked": 3480,
|
||||
"repository already in benchmarks/ (populations v1/v2 or a pair corpus)": 1993,
|
||||
"licence not permissive or unrecorded": 1249,
|
||||
"repository already kept (one pair per repository)": 657,
|
||||
"kept": 398,
|
||||
"CWEs span several families": 274,
|
||||
"no function declared on both sides with a changed body": 191,
|
||||
"large commit (refactor or release, not the fix)": 173,
|
||||
"fix spread over several commits or repositories": 167,
|
||||
"merge or root commit": 115,
|
||||
"no source file changed": 42,
|
||||
"language outside the feature vocabulary (go/ruby/c#/...)": 32,
|
||||
"manual review: the chosen function does not hold the fix (comment-only, misattributed, partial or family mismatch)": 19,
|
||||
"repository unavailable": 9,
|
||||
"WordPress plugin (population v3 safety)": 4,
|
||||
"fix commit unavailable": 2
|
||||
},
|
||||
"outcomes_note": "single-family pass (screen 2, every advisory examined, no quota met) plus the two-family pass; `kept` = automated candidates; `manual review: the chosen function does not hold the fix (comment-only, misattributed, partial or family mismatch)` = the screen re-meeting a function the 2026-10-03 review had dropped",
|
||||
"candidates_manually_reviewed": 412,
|
||||
"kept_after_review": 255,
|
||||
"review_drops": {
|
||||
"total": 157,
|
||||
"change in the function unrelated to the advisory (refactor, rename, formatting, another advisory)": 33,
|
||||
"function misattributed (matcher span overrun or an undetected enclosing declaration)": 33,
|
||||
"partial: the function only calls or configures the real fix": 29,
|
||||
"function diff not small (rewrite or refactor around the fix)": 28,
|
||||
"unsure": 17,
|
||||
"family mismatch": 11,
|
||||
"same fix as a kept pair in a fork or sibling repository (would leak across repository folds)": 3,
|
||||
"WordPress plugin (population v3 safety; missed by the automated screen)": 2,
|
||||
"reserved-population guard (flagged by the 2026-10-03 guard run; name not printed)": 1
|
||||
},
|
||||
"one_pair_per_repository_drops": 3,
|
||||
"reserved_population_drops": 2,
|
||||
"kept": 251
|
||||
},
|
||||
"kept": {
|
||||
"total": 251,
|
||||
"by_family": {
|
||||
"access_control": 20,
|
||||
"deserialization": 9,
|
||||
"injection": 56,
|
||||
"output_encoding": 69,
|
||||
"path": 71,
|
||||
"untrusted_destination": 26
|
||||
},
|
||||
"post_cutoff_by_family": {
|
||||
"injection": 5,
|
||||
"output_encoding": 5,
|
||||
"path": 2
|
||||
},
|
||||
"post_cutoff_total": 12,
|
||||
"by_language": {
|
||||
"java": 37,
|
||||
"javascript": 37,
|
||||
"php": 46,
|
||||
"python": 83,
|
||||
"typescript": 48
|
||||
},
|
||||
"post_cutoff_by_language": {
|
||||
"python": 6,
|
||||
"typescript": 6
|
||||
},
|
||||
"by_family_language": {
|
||||
"access_control:java": 5,
|
||||
"access_control:javascript": 3,
|
||||
"access_control:php": 7,
|
||||
"access_control:python": 2,
|
||||
"access_control:typescript": 3,
|
||||
"deserialization:java": 2,
|
||||
"deserialization:php": 1,
|
||||
"deserialization:python": 6,
|
||||
"injection:java": 5,
|
||||
"injection:javascript": 7,
|
||||
"injection:php": 9,
|
||||
"injection:python": 24,
|
||||
"injection:typescript": 11,
|
||||
"output_encoding:java": 7,
|
||||
"output_encoding:javascript": 20,
|
||||
"output_encoding:php": 12,
|
||||
"output_encoding:python": 14,
|
||||
"output_encoding:typescript": 16,
|
||||
"path:java": 10,
|
||||
"path:javascript": 4,
|
||||
"path:php": 10,
|
||||
"path:python": 34,
|
||||
"path:typescript": 13,
|
||||
"untrusted_destination:java": 8,
|
||||
"untrusted_destination:javascript": 3,
|
||||
"untrusted_destination:php": 7,
|
||||
"untrusted_destination:python": 3,
|
||||
"untrusted_destination:typescript": 5
|
||||
},
|
||||
"by_ecosystem": {
|
||||
"composer": 46,
|
||||
"maven": 37,
|
||||
"npm": 81,
|
||||
"pip": 87
|
||||
},
|
||||
"licences": {
|
||||
"Apache-2.0": 80,
|
||||
"BSD-2-Clause": 3,
|
||||
"BSD-3-Clause": 18,
|
||||
"ISC": 1,
|
||||
"MIT": 147,
|
||||
"MPL-2.0": 1,
|
||||
"Unlicense": 1
|
||||
},
|
||||
"split": {
|
||||
"holdout": 124,
|
||||
"train": 127
|
||||
},
|
||||
"published_by_year": {
|
||||
"2023": 50,
|
||||
"2024": 52,
|
||||
"2025": 66,
|
||||
"2026": 83
|
||||
},
|
||||
"published_range": [
|
||||
"2023-01-04",
|
||||
"2026-10-01"
|
||||
],
|
||||
"via_pull_request": 13,
|
||||
"commit_over_120_source_lines": 10,
|
||||
"two_family_advisory_family_chosen_on_review": 3,
|
||||
"verification": {
|
||||
"verified": 251,
|
||||
"excerpt_bytes_read": 1297516
|
||||
},
|
||||
"targets": {
|
||||
"asked": "injection and untrusted_destination >= 60 each; path, access_control, output_encoding, deserialization >= 30 where advisories allow; >= 250 total",
|
||||
"met": {
|
||||
"access_control": false,
|
||||
"deserialization": false,
|
||||
"injection": false,
|
||||
"output_encoding": true,
|
||||
"path": true,
|
||||
"untrusted_destination": false
|
||||
},
|
||||
"limit": "the reviewed-advisory pool, not the review: every advisory since 2023-01-01 in the four ecosystems was examined; the post-cutoff pool was largely spent by batch 2 (its repositories are now excluded), and untrusted_destination/access_control/deserialization advisories mostly lack a linked single fix commit or sit in repositories the benchmarks already use"
|
||||
}
|
||||
},
|
||||
"labels": {
|
||||
"command": "ousast learn labels --root . --out <scratch> --snapshot <scratch>",
|
||||
"rows": [
|
||||
6981,
|
||||
7985
|
||||
],
|
||||
"pairs_rows": [
|
||||
1936,
|
||||
2940
|
||||
],
|
||||
"new_pair_rows": 1004
|
||||
},
|
||||
"features": {
|
||||
"command": "benchmarks/learn/build_features.py --labels <the 1004 new rows> --units <merged units.json of `ousast plane harvest --catalog benchmarks/pairs/advisory-fixes-3/catalog.toml` (manifests written, no Run started)> --catalog benchmarks/pairs/advisory-fixes-3/catalog.toml --kinds pairs",
|
||||
"records": {
|
||||
"plane:access_control": 40,
|
||||
"plane:deserialization": 18,
|
||||
"plane:injection": 112,
|
||||
"plane:output_encoding": 138,
|
||||
"plane:path": 142,
|
||||
"plane:untrusted_destination": 52,
|
||||
"static:access_control": 40,
|
||||
"static:deserialization": 18,
|
||||
"static:injection": 112,
|
||||
"static:output_encoding": 138,
|
||||
"static:path": 142,
|
||||
"static:untrusted_destination": 52
|
||||
},
|
||||
"instrument_states_plane_all_families": {
|
||||
"agree:missing": 164,
|
||||
"agree:none": 338,
|
||||
"delta:not_applicable": 502,
|
||||
"engine:missing": 406,
|
||||
"engine:none": 96,
|
||||
"entry_points:not_applicable": 502,
|
||||
"facts:not_applicable": 502,
|
||||
"language:ran": 502,
|
||||
"model_sinks:missing": 502,
|
||||
"quick:ran": 502,
|
||||
"roles:ran": 502,
|
||||
"source:ran": 502,
|
||||
"verify:missing": 164,
|
||||
"verify:none": 338
|
||||
},
|
||||
"note": "quick rules and deterministic roles ran on each side; engine, verify, agree and model sinks never ran for these pairs (missing/none: null features, never zeros)"
|
||||
},
|
||||
"memory_build": {
|
||||
"command": "ousast learn memory build --labels <labels> --units <merged units.json> --profile static --root . (exit 2: adjudications 9 of 43 anonymous functions, unchanged since 2026-10-01; rows kept)",
|
||||
"examples": [
|
||||
1133,
|
||||
1635
|
||||
],
|
||||
"pairs_examples": [
|
||||
968,
|
||||
1470
|
||||
],
|
||||
"pairs_counts_after": {
|
||||
"absent_fixed_side": 0,
|
||||
"examples": 1470,
|
||||
"labels": 2940,
|
||||
"no_excerpt": 0,
|
||||
"no_record": 1470,
|
||||
"recorded": 1470,
|
||||
"unread": 0
|
||||
}
|
||||
},
|
||||
"complete_pairs": {
|
||||
"how": "store example rows (profile static) keyed as learn.experiments.freeze_units pairs them: (repo, candidate, family) holding both labels; whole memory, every family",
|
||||
"before": 459,
|
||||
"after": 710,
|
||||
"by_family": {
|
||||
"access_control": [
|
||||
37,
|
||||
57
|
||||
],
|
||||
"config_secrets": [
|
||||
12,
|
||||
12
|
||||
],
|
||||
"deserialization": [
|
||||
32,
|
||||
41
|
||||
],
|
||||
"injection": [
|
||||
68,
|
||||
124
|
||||
],
|
||||
"memory": [
|
||||
149,
|
||||
149
|
||||
],
|
||||
"output_encoding": [
|
||||
77,
|
||||
146
|
||||
],
|
||||
"path": [
|
||||
38,
|
||||
109
|
||||
],
|
||||
"unknown": [
|
||||
2,
|
||||
2
|
||||
],
|
||||
"untrusted_destination": [
|
||||
44,
|
||||
70
|
||||
]
|
||||
},
|
||||
"exp_004_005_units": "195 complete pairs inside exp-004/005's frozen 501 units (six families, evaluation folds); those units are frozen and were not regenerated here"
|
||||
},
|
||||
"positives_only_groups": {
|
||||
"before": 99,
|
||||
"after": 99,
|
||||
"groups_both_by_family": {
|
||||
"access_control": [
|
||||
37,
|
||||
57
|
||||
],
|
||||
"config_secrets": [
|
||||
2,
|
||||
2
|
||||
],
|
||||
"deserialization": [
|
||||
32,
|
||||
41
|
||||
],
|
||||
"injection": [
|
||||
53,
|
||||
109
|
||||
],
|
||||
"memory": [
|
||||
5,
|
||||
5
|
||||
],
|
||||
"output_encoding": [
|
||||
38,
|
||||
107
|
||||
],
|
||||
"path": [
|
||||
38,
|
||||
109
|
||||
],
|
||||
"unknown": [
|
||||
2,
|
||||
2
|
||||
],
|
||||
"untrusted_destination": [
|
||||
42,
|
||||
68
|
||||
]
|
||||
}
|
||||
},
|
||||
"embeddings": {
|
||||
"command": "ousast learn memory embed --budget-usd 0.20 (dry run first: 144,806 estimated tokens, $0.0029)",
|
||||
"requested": 1479,
|
||||
"cached": 1004,
|
||||
"embedded": 475,
|
||||
"calls": 15,
|
||||
"tokens": 150339,
|
||||
"usd": 0.003007
|
||||
},
|
||||
"spend": {
|
||||
"chat_model_calls": 0,
|
||||
"embedding_usd": 0.003007,
|
||||
"github_api": "metadata and raw contents only"
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,86 @@
|
||||
{
|
||||
"count": 1029,
|
||||
"guard": "green",
|
||||
"sha256": "5a644a1e93c68a0a35c794e653286d0532b7eb8d757123ccce5f08b11bf9a4a2",
|
||||
"sources": {
|
||||
"harvest": {
|
||||
"bytes": 501031,
|
||||
"count": 467,
|
||||
"files": 3,
|
||||
"known_empty": false,
|
||||
"rows": 446,
|
||||
"skipped_values": 446
|
||||
},
|
||||
"measurements": {
|
||||
"bytes": 21169903,
|
||||
"count": 278,
|
||||
"files": 221,
|
||||
"known_empty": false,
|
||||
"rows": 376,
|
||||
"skipped_values": 681
|
||||
},
|
||||
"memory_local": {
|
||||
"bytes": 8084523,
|
||||
"count": 192,
|
||||
"files": 776,
|
||||
"known_empty": false,
|
||||
"rows": 759,
|
||||
"skipped_values": 2
|
||||
},
|
||||
"memory_s3": {
|
||||
"bytes": 15928676,
|
||||
"count": 572,
|
||||
"files": 1499,
|
||||
"known_empty": false,
|
||||
"rows": 1490,
|
||||
"skipped_values": 2
|
||||
},
|
||||
"pairs": {
|
||||
"bytes": 1680480,
|
||||
"count": 728,
|
||||
"files": 28,
|
||||
"known_empty": false,
|
||||
"rows": 900,
|
||||
"skipped_values": 1260
|
||||
},
|
||||
"pointers": {
|
||||
"bytes": 892448,
|
||||
"count": 537,
|
||||
"files": 7,
|
||||
"known_empty": false,
|
||||
"rows": 586,
|
||||
"skipped_values": 666
|
||||
},
|
||||
"populations": {
|
||||
"bytes": 32431,
|
||||
"count": 28,
|
||||
"files": 2,
|
||||
"known_empty": false,
|
||||
"rows": 28,
|
||||
"skipped_values": 0
|
||||
},
|
||||
"sweep": {
|
||||
"bytes": 34006439,
|
||||
"count": 1029,
|
||||
"files": 1619,
|
||||
"known_empty": false,
|
||||
"rows": 2155,
|
||||
"skipped_values": 0
|
||||
},
|
||||
"units": {
|
||||
"bytes": 319641,
|
||||
"count": 229,
|
||||
"files": 9,
|
||||
"known_empty": false,
|
||||
"rows": 642,
|
||||
"skipped_values": 0
|
||||
}
|
||||
},
|
||||
"test_exit_code": 0,
|
||||
"tests": {
|
||||
"errors": 0,
|
||||
"failures": 0,
|
||||
"skipped": 1,
|
||||
"tests": 41
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
{
|
||||
"commit": "48ae918dec25119142ae781607ba99306ae5ec17",
|
||||
"fixture_bytes": 15374,
|
||||
"fixture_sha256": "dd3c774a47d553105ed85b9cea8810736c233f870e5a015559a1524d5db5a14d",
|
||||
"implementation_sha256": {
|
||||
"benchmarks/ax/batch.py": "37aecac93beb25c02826e81eacaf064fbebb447a998fe139f6d94a07035b8685",
|
||||
"benchmarks/unseen/replay.py": "32efb3e0b9e9d247f2780b5f3ceeb01b4c2cbda3656c75a741f9925621cfdfbc",
|
||||
"benchmarks/unseen/replay_entry.py": "994d8e239eb505dc2c5bad4cdb8b5d45967a8a2aa9ef48fb4c2f20ba45736280",
|
||||
"benchmarks/unseen/score.py": "56e0a7b34482b5d84df39824b8ee447c283e35bd932f38cfe692cf84cb52cef7"
|
||||
},
|
||||
"network_proofs": {
|
||||
"4.4": "coordinator_pending",
|
||||
"5.4": "coordinator_pending"
|
||||
},
|
||||
"scope": "offline synthetic scorer only",
|
||||
"score_command_exit": 0,
|
||||
"scorer_output_sha256": "77f154eedb6a4acd098ab63a0b17434291f0d226dabe8ac66f2e0b362154d002",
|
||||
"scorer_test_exit": 0,
|
||||
"scorer_tests": 14,
|
||||
"synthetic_records": 20,
|
||||
"uncommitted": true,
|
||||
"validation": {
|
||||
"generic_dispatcher_tests": {
|
||||
"exit_code": 0,
|
||||
"passed": 5
|
||||
},
|
||||
"mypy": 0,
|
||||
"proof_and_scorer_tests": {
|
||||
"exit_code": 0,
|
||||
"passed": 16
|
||||
},
|
||||
"requested_pytest": {
|
||||
"exit_code": 0,
|
||||
"passed": 279,
|
||||
"skipped": 8
|
||||
},
|
||||
"reserved_repository_guard": 0,
|
||||
"ruff_check": 0,
|
||||
"ruff_format_check": 0
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,312 @@
|
||||
# Unseen evaluation, group 1
|
||||
|
||||
The coordinator runs task 1.5 from the repository root:
|
||||
|
||||
```sh
|
||||
export PATH=/home/mc/Source/OpenUltraSAST/.venv/bin:$PATH
|
||||
export PYTHONPATH=$PWD/src
|
||||
python -m benchmarks.unseen.eligibility --s3-store s3:// --verify-tests --output benchmarks/measurements/2026-10-04-unseen-01-used-set/record.json
|
||||
```
|
||||
|
||||
This is the networked command; implementation tests use fakes. It loads `.env` with
|
||||
`openultrasast.config.load_dotenv` (exported values win), opens the local FileStore and
|
||||
S3 MemoryStore through `openultrasast.plane.memory.open_store(..., read_only=True)`,
|
||||
and uses `GH_TOKEN` for GitHub GET resolution. `s3://` selects `S3_BUCKET`; pass
|
||||
`--s3-store s3://BUCKET/PREFIX` for a prefixed store and `--local-store file:///PATH`
|
||||
for a different local store. Read-only S3 opening skips the normal write/Select probe;
|
||||
collection uses LIST/GET only. Existing normal store opening is unchanged.
|
||||
|
||||
Pair catalogs/recipes include local pointer rows (`vendored = false`, including
|
||||
`fix_repo`). Additional external manifests can be supplied with repeatable
|
||||
`--pointer-manifest PATH`. Registered units in memory and experiment YAML are followed;
|
||||
missing units with a registered digest fail. Plans without frozen units are not used data.
|
||||
Experiment units identify repositories in `group`. Binary measurement artifacts are
|
||||
read and counted; structured JSON, TOML and multi-document YAML are parsed strictly.
|
||||
|
||||
Each source records files/objects read, bytes read, repository-bearing rows (distinct
|
||||
names per input), and distinct repository counts. The digest is SHA-256 of sorted,
|
||||
normalized names joined with newlines. Source counts overlap. An empty source fails;
|
||||
only after establishing it is empty may the coordinator pass `--known-empty LABEL`.
|
||||
The output records that declaration. Missing/unreadable inputs still fail.
|
||||
|
||||
GitHub redirects add the requested and canonical names, plus parent/source fork
|
||||
identities. Used names are also resolved so an old used name excludes a renamed
|
||||
candidate. GitHub cannot enumerate all historical names: injected clients may supply
|
||||
`former_names`; actual historical aliases come from used references and redirects.
|
||||
A 404 for a historical/synthetic used identity retains its literal identity; other
|
||||
API failures stop collection. Candidate resolution always fails closed on a 404.
|
||||
`resolve(candidate, api)` and `rejection(resolution, used, selected_networks)` are
|
||||
available to the later sourcing task. Names stay in memory; output contains only
|
||||
counts, labels and SHA-256 digests. Output files are created exclusively, never overwritten.
|
||||
|
||||
`--verify-tests` runs the three group-1 test modules with both output streams discarded,
|
||||
including the existing population guard. Only aggregate test counts and the exit code
|
||||
enter the record; a failed test prevents the record. V3 paths are excluded before
|
||||
eligibility reads; only the existing guard handles v3 inside its black-box test process.
|
||||
|
||||
Population reservations retain their existing sweep over `benchmarks/`, `src/`, and
|
||||
`tests/`. Pool reservations also cover `plane/`. Each reservation exempts only its own
|
||||
directory; population checks include all of `benchmarks/unseen/private/`. This preserves
|
||||
the existing population behavior while adding pools (the hard rule for this increment).
|
||||
|
||||
All `pool-pN.toml` files require a matching `freeze-pN.json`; drafts use `draft-pN.toml`.
|
||||
The freeze digest is SHA-256 of parsed public TOML encoded as UTF-8 JSON with sorted
|
||||
keys, compact separators and `ensure_ascii=False`. Private entries must be represented
|
||||
by opaque digests in that public manifest, as task 3.2 requires.
|
||||
|
||||
The ledger exposes `append(path, row)` and `read(path)`. Rows contain `slice`, `decision`
|
||||
(an opaque decision or experiment label), `purpose`, ISO `date`, `result_digest`, and
|
||||
`freeze_digest`. Appends lock and validate the existing chain, reject duplicates or a
|
||||
changed freeze, and reject qualification of an informed slice for the same decision.
|
||||
Keep the published final digest to authenticate history: the chain detects row edits,
|
||||
but cannot independently prove that an entire history was not replaced or truncated.
|
||||
|
||||
## Sourcing and extraction (tasks 2.1–2.4)
|
||||
|
||||
The coordinator's networked task **2.5** is now:
|
||||
|
||||
```sh
|
||||
export PATH=/home/mc/Source/OpenUltraSAST/.venv/bin:$PATH
|
||||
export PYTHONPATH=$PWD/src
|
||||
python -m benchmarks.unseen.draft --root "$PWD" --s3-store s3:// --seed 20260601 \
|
||||
--cache /home/mc/ousast-results/plane/unseen-draft --max-hours 8
|
||||
```
|
||||
|
||||
Export `GH_TOKEN` first, or supply it through the root `.env` (exported values win).
|
||||
The same read-only local/S3 stores, `--pointer-manifest`, and explicit `--known-empty`
|
||||
options as task 1.5 apply. Missing or unreadable sources fail. The used set is rebuilt
|
||||
and canonicalized before candidate selection, including redirects and fork networks.
|
||||
No population v3 file is opened and no guard/bisection is run by this command; that is
|
||||
reserved for group 3. An existing draft is never overwritten.
|
||||
|
||||
The command writes `--cache/journal/draft-p1.toml` and
|
||||
`--cache/journal/private-draft-p1.toml` only after all sourcing stages complete.
|
||||
Both stay outside the repository, with mode 0600. The public-format file contains permissive entries
|
||||
and, for every private entry, only an opaque ID, slice, private marker, and SHA-256
|
||||
of canonical JSON containing its URL and ordered `[base, head]` pairs. Private entries
|
||||
carry the complete records, including licences, in the private-format file.
|
||||
Licensing is re-read through GitHub's licence endpoint **at the actual extracted
|
||||
vulnerability replay head**; the candidate's earlier fix-parent classification is
|
||||
provisional. An explicit SPDX declaration in the pinned file takes precedence over
|
||||
the API's classification. Missing/unknown/compound licences remain private.
|
||||
|
||||
Three seeded slices each contain 100 repositories, one vulnerability range and twelve
|
||||
ordinary changes per repository. Each ecosystem/post-cutoff stratum differs from its
|
||||
proportional per-slice target by less than one repository. Post-cutoff candidates are
|
||||
attempted first within each ecosystem. Below 300 eligible repositories the command
|
||||
exits 1 with counts and does not write a partial draft or relax any selection bar.
|
||||
The group-2 measured exit and label confirmation remain coordinator tasks 2.5/2.6;
|
||||
passing implementation tests does not assert that 300 eligible repositories exist.
|
||||
|
||||
Clones and checkpoints default to `results_root()/unseen-draft` (normally
|
||||
`~/ousast-results/plane/unseen-draft`), overridable by `--cache /outside/path`.
|
||||
The cache cannot lie inside `--root`. One opaque, disposable clone exists at a time;
|
||||
it is removed on success, rejection, or failure. `--filter=blob:none --no-checkout`
|
||||
keeps the commit graph needed for temporal thirds, blame, and release-tag ancestry,
|
||||
and fetches source blobs on demand. This is a **partial clone**, not a shallow history.
|
||||
The 500 MiB disk cap is checked after Git commands, including lazy blob fetches;
|
||||
one in-flight command can transiently exceed it before rejection and cleanup.
|
||||
Git output is captured, never forwarded. Each repository must read a nonempty source
|
||||
blob before extraction. Summary instrumentation includes source read operations and
|
||||
actual bytes, clone count, cumulative/peak disk bytes, wall time, and logical API calls
|
||||
and HTTP attempts. Zero source bytes or a failed Git read is an instrument failure,
|
||||
not an eligibility drop. Stdout is a JSON summary of counts/digests only; no names or
|
||||
URLs enter stdout or a measurement record. Raw per-change data stays in the external journal and manifests.
|
||||
|
||||
Extraction uses Git and syntax boundaries, with no scanner, engine, or model. Python
|
||||
uses stdlib AST; Java/PHP/JS/TS use conservative named-function brace matching. Ambiguous
|
||||
or unparseable functions can reject a candidate. The first removed code line per
|
||||
fix-touched function is the SZZ sink proxy; pure-addition fixes use its declaration.
|
||||
Normalized text absence, non-root/non-merge ancestry, file/line limits, and the OSV
|
||||
release bracket are checked before accepting `introducing`. Matching tags are `VERSION`
|
||||
or `vVERSION`; an absent/ambiguous tag or missing prior release cannot certify the
|
||||
bracket. After at most five walk-backs, `last_touch` uses `git log -L` with the resolved
|
||||
numeric function extent (also works where Git lacks a language-specific function driver).
|
||||
Method, bias, fallback reasons, head function/range, files/hunks, family, advisory, and
|
||||
cutoff flag are recorded. Task 2.6 must review every third `introducing` label.
|
||||
|
||||
All reviewed advisories are enumerated to exclude **all** known repository fixes from
|
||||
ordinary changes, including old advisories and advisories outside the four candidate
|
||||
ecosystems. Fix URLs are canonicalized before indexing so renames cannot hide them.
|
||||
The ordinary filter uses only the design's narrowed security pattern. It counts
|
||||
rejections (including oversized commits and repositories failing the 40-commit floor),
|
||||
checks both sides for touches to known vulnerable functions, and inspects generated
|
||||
headers and inherited `.gitattributes`. Sampling allocates twelve draws proportionally
|
||||
across three history-time thirds and four source-size buckets, with one per third.
|
||||
|
||||
For offline tests, inject GitHub/OSV clients, the clone context, and `UsedSet` into
|
||||
`draft.run`; HTTP transports and Git subprocess runners also accept fakes. Tests use
|
||||
sanitized REST-shaped data and a temporary synthetic local Git history. No implementation
|
||||
test contacts GitHub, OSV, a model, or a cluster. HTTP retries are bounded to five attempts;
|
||||
rate-limit reset/Retry-After waits and transient 5xx backoff are tested with fake clocks
|
||||
and sleepers. A stale candidate 404 is counted; other service failures fail the run.
|
||||
|
||||
### Checkpoint and resume
|
||||
|
||||
Replay reads bounded JSONL records into a disposable SQLite working store, with a 2 MiB
|
||||
page cache. Advisory descriptions, non-fix references and vulnerable-version payloads
|
||||
are discarded; package identities, CWEs, status, publication dates and every fix link
|
||||
are retained. Repository metadata keeps canonical/fork identities and eligibility fields.
|
||||
Advisory ordering, fix indexes and checkpoint lookups stay on disk, independent of the
|
||||
number of journal records. The original seeded ordering and ordinary draws are preserved.
|
||||
Per-repository history and exclusion sets also spill to private temporary storage, which
|
||||
is removed on success or failure. Budget disk space for the journal, working store and
|
||||
spooled Git output as well as the clone.
|
||||
|
||||
To bound individual reads, journal records are limited to 16 MiB, decoded Git command
|
||||
output to 8 MiB, history messages to 1 MiB and source blobs to 512 KiB. Exceeding a limit
|
||||
fails the instrument explicitly; it never counts as an empty history or a clean result.
|
||||
The synthetic 20,000-advisory regression checks a 64 MiB traced-memory ceiling across
|
||||
replay, enumeration, ordering and fix indexing; this is not a live coordinator RSS measurement.
|
||||
|
||||
The memory fix can migrate the exact preceding design fingerprints automatically,
|
||||
keeping `inputs-before-memory-fix.json` and leaving the JSONL journal untouched. All
|
||||
other input/design mismatches still refuse resume. This compatibility exception must
|
||||
be removed if a later change alters selection or sampling semantics.
|
||||
|
||||
The command above uses journal directory
|
||||
`/home/mc/ousast-results/plane/unseen-draft/journal`. Run the same command to resume.
|
||||
`--max-hours` is a per-invocation budget; changing it does not invalidate checkpoints.
|
||||
Create `journal/STOP` to stop between units, then remove it before resuming. A page,
|
||||
repository resolution, candidate decision, or repository extraction already running
|
||||
finishes first, so the time budget is not a hard timeout on a large repository.
|
||||
Clean stops exit 0 with `status: stopped`; failures and shortfalls exit 1. Completion
|
||||
exits 0 with `status: draft`. No partial manifest is published.
|
||||
|
||||
`inputs.json` binds the seed, source-labelled used-set digest, known-empty declarations,
|
||||
clone cap, and design fingerprints (sourcing/extraction/eligibility/publication code and
|
||||
family taxonomy). A mismatch refuses resume and prints only the changed parameter
|
||||
labels. Use a new external cache for a deliberately changed design or population.
|
||||
The used set is rebuilt each invocation; repository resolutions, including 404s, are
|
||||
reused from the journal. Live GitHub metadata for unfinished units can still change;
|
||||
completed pages and decisions are the durable sourcing snapshot.
|
||||
|
||||
Each completed unit is appended to its stage JSONL and fsynced before work advances:
|
||||
`advisories.jsonl` (pages), `resolution.jsonl` (alias/fork metadata),
|
||||
`eligibility.jsonl` (candidate decisions), `extraction.jsonl` (repository/advisory
|
||||
results, vulnerable range, ordinary draws, pinned licence, method and counts), and
|
||||
`completion.jsonl` (final manifest digest). A lock permits only one writer.
|
||||
`progress.json` is atomically rewritten after every unit with stage counts, last unit,
|
||||
last observed GitHub rate-limit headers, and cumulative active elapsed seconds.
|
||||
These files may contain identities and must stay outside version control. Stdout and
|
||||
repository records contain only counts, digests and fixed labels.
|
||||
|
||||
Replaying completed units makes no GitHub/OSV or clone calls. Final manifests are
|
||||
reconstructed deterministically from journal results; an interrupted publication can
|
||||
be retried, but a different existing manifest is refused. Completed JSONL rows are
|
||||
never rewritten. Malformed or torn records fail closed instead of being interpreted
|
||||
as completed work. Tests interrupt each stage with fakes, then compare resumed and
|
||||
uninterrupted manifests byte for byte.
|
||||
|
||||
### Planning estimate for task 2.5 (not measured)
|
||||
|
||||
Let A be all reviewed advisories, R distinct fix-linked repository identities, U used
|
||||
identities not already cached, C candidate advisories passing early filters, and K
|
||||
repositories surviving extraction. GitHub requests are approximately
|
||||
`ceil(A/100) + R + U + 2*C + K` (enumeration, canonical repositories, fix metadata,
|
||||
provisional licence, final licence), plus retries. OSV requests are at most C. Git
|
||||
fetches and lazy blob transfers are additional network operations, not GitHub API calls.
|
||||
|
||||
For a planning scenario of A=40,000, R=15,000, U≈1,000, C=600–1,500, K=300:
|
||||
about **18,000–20,000 GitHub calls** and **600–1,500 OSV calls**. The all-advisory index
|
||||
makes this substantially more than 300 repository lookups. At an assumed 5,000-request
|
||||
hourly allowance, allow several rate-limit windows. Actual token limits can differ.
|
||||
For 300–1,000 attempted clones averaging 20–100 MB after source hydration, plan roughly
|
||||
**6–100 GB transferred**, with one clone occupying about 500 MiB at most between commands
|
||||
plus the retained journal and manifests after completion. Transfer volume is an estimate; disk bytes
|
||||
are only a proxy. A working planning range is **8–48 hours**, potentially longer on
|
||||
large histories: every first-parent commit is inspected and lazy blobs can dominate.
|
||||
These are assumptions, not evidence from a network run. The command reports actual
|
||||
counts and wall time; task 2.5 establishes the measured estimate.
|
||||
|
||||
## Dispatcher, replay and scoring (groups 4–6)
|
||||
|
||||
`benchmarks.ax.batch` owns Task transport and the shared scheduler. New workloads supply
|
||||
`Item` input bytes/commands and a `Workload` image, URL allowlist, deadline and result
|
||||
validator. Two AX slots and one VM slot are the default; sandbox death gets one fresh
|
||||
Task before VM fallback, OOM goes straight to VM, and instrument failures get one rerun.
|
||||
Cleanup failure prevents further launches. Create `STOP` in the output directory to stop
|
||||
launching; remove it and repeat the command to resume matching input digests. The legacy
|
||||
engine adapter retains its `questions.json` object name for compatibility; replay inputs
|
||||
and outputs use `.dat`. Both use the same object polling and cleanup implementation.
|
||||
|
||||
Replay reads `pool-pN.toml`, verifies its adjacent `freeze-pN.json` canonical digest,
|
||||
selects `--slice`, and resolves private pointers through `private/pool-pN.toml` (or
|
||||
`--private-manifest`). It never reads a draft. `--dry-run` prints only counts and digests.
|
||||
The five-change proof uses a separate frozen JSON manifest from this project's own
|
||||
source-changing history. Raw output must be outside the repository.
|
||||
|
||||
Each output root contains `inputs.json` and digest-keyed records and artifact directories.
|
||||
The scorer accepts that root directly, joins change labels, and includes missing or stale
|
||||
results as instrument failures. It refuses a slice above 5% failure after reruns. Example:
|
||||
|
||||
```sh
|
||||
python -m benchmarks.unseen.score "$HOME/ousast-results/unseen/p1/1/default" \
|
||||
--out "$HOME/ousast-results/unseen/p1/1/default-score.json"
|
||||
```
|
||||
|
||||
For paired binary arms add `--against BASELINE_ROOT --slice-id p1/2`; `--informed` forces
|
||||
`unseen.gate=not_run`. The later experiment integration owns ledger verification; callers
|
||||
of `gate` must pass that ledger decision. Score-bearing arms use `budgets` with fixed
|
||||
thresholds and `gate(..., binary=False, threshold=..., threshold_fixed=True)`. In-slice
|
||||
thresholds are exploratory and cannot qualify adoption. JVM wall time is explicitly the
|
||||
sum of the hook artifact's scan build/query times, not a separately observed process timer.
|
||||
|
||||
### Coordinator task 4.4
|
||||
|
||||
Run from the merged repository root. Set `ENGINE_IMAGE` to the **engine** digest printed
|
||||
by Engine image CI, `ENGINE_UNITS` and `ENGINE_EXAMPLES` to the frozen input files for the
|
||||
two stored pins, `ENGINE_BASELINE` to their stored record directory, and `AX_KUBECONFIG`
|
||||
to the operator's kube-ax kubeconfig. Host store credentials stay on the VM; use the public
|
||||
HTTPS `S3_ENDPOINT=https://files.because-security.com` and the operator's `OUSAST_MEMORY`.
|
||||
Use a fresh output directory, so the smoke executes instead of resuming prior records.
|
||||
|
||||
```sh
|
||||
export PATH=/home/mc/Source/OpenUltraSAST/.venv/bin:$PATH
|
||||
export PYTHONPATH=$PWD/src
|
||||
export S3_ENDPOINT=https://files.because-security.com
|
||||
python benchmarks/learn/engine_trace.py \
|
||||
--units "$ENGINE_UNITS" --examples "$ENGINE_EXAMPLES" \
|
||||
--executor ax --parallel 2 --limit 2 --image "$ENGINE_IMAGE" \
|
||||
--kubeconfig "$AX_KUBECONFIG" --out "$HOME/ousast-results/unseen/proof-04"
|
||||
python -m benchmarks.unseen.proof engine --image "$ENGINE_IMAGE" \
|
||||
--records "$HOME/ousast-results/unseen/proof-04" --baseline "$ENGINE_BASELINE" \
|
||||
--out "benchmarks/measurements/$(date +%F)-unseen-04-dispatcher/record.json"
|
||||
```
|
||||
|
||||
### Coordinator task 5.4
|
||||
|
||||
Merge before running this proof. `.github/workflows/engine-image.yml` builds the `engine`
|
||||
and `replay` targets on push to main; it publishes `ousast-engine-task` and
|
||||
`ousast-replay-task` separately and prints their digests. Set `REPLAY_IMAGE` to the latter
|
||||
full `ghcr.io/...@sha256:...` reference. No manual image build is needed. Set
|
||||
`KIND_KUBECONFIG` and `AX_KUBECONFIG` to the corresponding AX kubeconfigs; the kind lane
|
||||
uses the identical Task API/manifest, never a Kubernetes Job.
|
||||
|
||||
```sh
|
||||
export PATH=/home/mc/Source/OpenUltraSAST/.venv/bin:$PATH
|
||||
export PYTHONPATH=$PWD/src
|
||||
export S3_ENDPOINT=https://files.because-security.com
|
||||
python -m benchmarks.unseen.replay --proof-history \
|
||||
--manifest "$HOME/ousast-results/unseen/proof-05-inputs.json"
|
||||
python -m benchmarks.unseen.replay --manifest "$HOME/ousast-results/unseen/proof-05-inputs.json" \
|
||||
--image "$REPLAY_IMAGE" --lane kind --kubeconfig "$KIND_KUBECONFIG" \
|
||||
--out "$HOME/ousast-results/unseen/proof-05-kind"
|
||||
python -m benchmarks.unseen.replay --manifest "$HOME/ousast-results/unseen/proof-05-inputs.json" \
|
||||
--image "$REPLAY_IMAGE" --lane ax --kubeconfig "$AX_KUBECONFIG" \
|
||||
--out "$HOME/ousast-results/unseen/proof-05-ax"
|
||||
python -m benchmarks.unseen.replay --manifest "$HOME/ousast-results/unseen/proof-05-inputs.json" \
|
||||
--image "$REPLAY_IMAGE" --lane docker \
|
||||
--out "$HOME/ousast-results/unseen/proof-05-vm"
|
||||
python -m benchmarks.unseen.proof replay --image "$REPLAY_IMAGE" \
|
||||
--kind-records "$HOME/ousast-results/unseen/proof-05-kind" \
|
||||
--ax-records "$HOME/ousast-results/unseen/proof-05-ax" \
|
||||
--vm-records "$HOME/ousast-results/unseen/proof-05-vm" \
|
||||
--out "benchmarks/measurements/$(date +%F)-unseen-05-replay-task/record.json"
|
||||
```
|
||||
|
||||
The record command fails if fewer than five results exist, identities differ, bytes/JVM
|
||||
proof is missing, or quick-tier findings differ. It does not convert a quick-only run
|
||||
without a JVM into an engine proof. Pool replays use `--lane mixed` (default) for the two
|
||||
AX slots and single VM overflow/fallback lane; `--arm long_deadline` changes only the
|
||||
hook deadline to 300 seconds and is a separate input digest.
|
||||
@@ -0,0 +1,493 @@
|
||||
"""Coordinator-only sourcing command: python -m benchmarks.unseen.draft --help.
|
||||
|
||||
All network and git operations are injected at the run() boundary. Only counts,
|
||||
digests and fixed failure codes go to stdout; identity-bearing output stays in the
|
||||
private journal outside the repository. No population guard is run here.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import copy
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import random
|
||||
import subprocess
|
||||
import tempfile
|
||||
import time
|
||||
from collections import Counter, defaultdict
|
||||
from contextlib import contextmanager
|
||||
from dataclasses import asdict
|
||||
from pathlib import Path
|
||||
|
||||
from . import eligibility, extract, source
|
||||
from .extract import Git, InstrumentFailure
|
||||
from .journal import Journal, ResolvedGitHub, ResumeMismatch, Stopped, atomic, digest, external
|
||||
from .storage import AdvisoryRows, DiskSet, FixIndex, working_store
|
||||
|
||||
|
||||
def canonical(value) -> bytes:
|
||||
return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode()
|
||||
|
||||
|
||||
def private_digest(row: dict) -> str:
|
||||
return hashlib.sha256(canonical({"url": row["url"], "commits": [[c["base"], c["head"]] for c in row["changes"]]})).hexdigest()
|
||||
|
||||
|
||||
def assign_slices(rows: list[dict], *, seed: int) -> list[dict]:
|
||||
if len(rows) != 300 or len({r["repository"] for r in rows}) != 300:
|
||||
raise ValueError("pool_size")
|
||||
for row in rows:
|
||||
kinds = Counter(c["kind"] for c in row["changes"])
|
||||
if kinds != {"vulnerability": 1, "ordinary": 12}:
|
||||
raise ValueError("change_count")
|
||||
if row["license_class"] not in {"permissive", "private"}:
|
||||
raise ValueError("license_class")
|
||||
if row["license_class"] == "permissive" and row["license_spdx"] not in source.PERMISSIVE:
|
||||
raise ValueError("license_class")
|
||||
rng = random.Random(seed)
|
||||
groups = defaultdict(list)
|
||||
for row in sorted(rows, key=lambda r: r["repository"]):
|
||||
groups[row["ecosystem"], row["post_cutoff"]].append(row)
|
||||
sizes = Counter({1: 0, 2: 0, 3: 0})
|
||||
assigned = []
|
||||
for key in sorted(groups):
|
||||
group = groups[key]
|
||||
rng.shuffle(group)
|
||||
quotient, remainder = divmod(len(group), 3)
|
||||
# Give each stratum its floor on all slices, then fill globally least-full slices.
|
||||
quotas = {i: quotient for i in sizes}
|
||||
order = list(sizes)
|
||||
rng.shuffle(order)
|
||||
order.sort(key=lambda i: sizes[i])
|
||||
for i in order[:remainder]:
|
||||
quotas[i] += 1
|
||||
offset = 0
|
||||
for slice_id in sorted(quotas):
|
||||
for row in group[offset : offset + quotas[slice_id]]:
|
||||
digest = private_digest(row)
|
||||
assigned.append({**row, "id": "r-" + digest[:24], "slice": slice_id})
|
||||
sizes[slice_id] += quotas[slice_id]
|
||||
offset += quotas[slice_id]
|
||||
if set(sizes.values()) != {100}:
|
||||
raise ValueError("slice_size")
|
||||
return sorted(assigned, key=lambda r: (r["slice"], r["id"]))
|
||||
|
||||
|
||||
def toml(value) -> str:
|
||||
"""Inline TOML values keep nested change records structured without a new dependency."""
|
||||
if isinstance(value, bool):
|
||||
return "true" if value else "false"
|
||||
if isinstance(value, str):
|
||||
return json.dumps(value, ensure_ascii=False)
|
||||
if isinstance(value, int):
|
||||
return str(value)
|
||||
if isinstance(value, list):
|
||||
return "[" + ", ".join(toml(v) for v in value) + "]"
|
||||
if isinstance(value, dict):
|
||||
return "{" + ", ".join(json.dumps(k) + " = " + toml(v) for k, v in sorted(value.items()) if v is not None) + "}"
|
||||
raise ValueError("unsupported_manifest_type")
|
||||
|
||||
|
||||
def manifest(rows: list[dict], seed: int) -> str:
|
||||
lines = ['status = "draft"', 'pool = "p1"', f"seed = {seed}"]
|
||||
for row in rows:
|
||||
lines.append("\n[[repository]]")
|
||||
lines.extend(f"{key} = {toml(value)}" for key, value in sorted(row.items()) if value is not None)
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def write_draft(root: Path, assigned: list[dict], *, seed: int) -> tuple[Path, Path]:
|
||||
public_path = root / "benchmarks/unseen/draft-p1.toml"
|
||||
private_path = root / "benchmarks/unseen/private/draft-p1.toml"
|
||||
if public_path.exists() or private_path.exists():
|
||||
raise FileExistsError("draft_exists")
|
||||
public, private = [], []
|
||||
for row in assigned:
|
||||
if row["license_class"] == "permissive":
|
||||
public.append(row)
|
||||
else:
|
||||
private.append(row)
|
||||
public.append({"id": row["id"], "sha256": private_digest(row), "slice": row["slice"], "private": True})
|
||||
private_path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
# Exclusive creates protect drafts awaiting human label review. Private first: never
|
||||
# publish a pointer whose private manifest failed to write.
|
||||
with private_path.open("x") as handle:
|
||||
os.chmod(private_path, 0o600)
|
||||
handle.write(manifest(private, seed))
|
||||
with public_path.open("x") as handle:
|
||||
handle.write(manifest(public, seed))
|
||||
return public_path, private_path
|
||||
|
||||
|
||||
class Clones:
|
||||
"""One disposable blobless clone at a time; no name in cache paths or subprocess output."""
|
||||
|
||||
def __init__(self, root: Path, cache: Path, *, runner=None, limit_bytes=500 * 1024 * 1024):
|
||||
self.cache = external(root, cache)
|
||||
self.runner = runner or subprocess.run
|
||||
self.limit_bytes = limit_bytes
|
||||
self.counts = Counter()
|
||||
|
||||
@contextmanager
|
||||
def open(self, url: str):
|
||||
self.cache.mkdir(parents=True, exist_ok=True)
|
||||
started = time.monotonic()
|
||||
with tempfile.TemporaryDirectory(prefix="clone-", dir=self.cache) as directory:
|
||||
path = Path(directory)
|
||||
git = Git(path, runner=self.runner, limit_bytes=self.limit_bytes)
|
||||
try:
|
||||
# Full commit graph is needed for temporal thirds and OSV brackets; blobs
|
||||
# are fetched lazily. This is a partial clone, not a truncated history.
|
||||
done = self.runner(
|
||||
["git", "-c", "credential.helper=", "clone", "--quiet", "--filter=blob:none", "--no-checkout", url, str(path)],
|
||||
stdin=subprocess.DEVNULL,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
check=False,
|
||||
timeout=180,
|
||||
env={**os.environ, "GIT_TERMINAL_PROMPT": "0"},
|
||||
)
|
||||
if done.returncode:
|
||||
raise InstrumentFailure("clone_failed")
|
||||
self.counts["clones"] += 1
|
||||
if git.disk_bytes() > self.limit_bytes:
|
||||
raise InstrumentFailure("clone_size_cap")
|
||||
yield git
|
||||
finally:
|
||||
size = git.disk_bytes()
|
||||
self.counts["clone_bytes"] += size
|
||||
self.counts["peak_clone_bytes"] = max(self.counts["peak_clone_bytes"], size)
|
||||
self.counts["source_bytes_read"] += git.bytes_read
|
||||
self.counts["source_files_read"] += git.files_read
|
||||
self.counts["clone_wall_seconds"] += round(time.monotonic() - started, 3)
|
||||
|
||||
|
||||
def prove_source(git, revision: str) -> dict:
|
||||
files = git.run("ls-tree", "-r", "--name-only", revision).splitlines()
|
||||
for path in files:
|
||||
if Path(path).suffix in extract.LANGUAGES and not extract.generated_path(path):
|
||||
text = git.blob(revision, path)
|
||||
if text:
|
||||
return {"files": 1, "bytes": getattr(git, "last_blob_bytes", len(text.encode()))}
|
||||
raise InstrumentFailure("zero_source_bytes")
|
||||
|
||||
|
||||
def advisory_index(rows, github, *, db):
|
||||
"""Include ALL enumerated fixes, even old, unmapped, multi-fix and other ecosystems.
|
||||
|
||||
Canonicalize link names as well, so an old advisory URL survives repository renames.
|
||||
A 404 is not treated as proof that an advisory is unrelated to a selected repository.
|
||||
"""
|
||||
index = FixIndex(db)
|
||||
for row in rows:
|
||||
for name, sha in source.fix_links(row):
|
||||
try:
|
||||
resolved = eligibility.resolve(name, github)
|
||||
except eligibility.RepositoryNotFound:
|
||||
index.add(name, sha)
|
||||
continue
|
||||
index.add(resolved.full_name, sha)
|
||||
return index
|
||||
|
||||
|
||||
def repository_changes(git, candidate, fixes: set[str], *, seed: int):
|
||||
proof = prove_source(git, candidate.parent)
|
||||
change = extract.introducing(git, candidate)
|
||||
with working_store() as db:
|
||||
known = DiskSet(db, "known_functions")
|
||||
expanded = DiskSet(db, "expanded_fixes")
|
||||
pending = DiskSet(db, "pending_fixes")
|
||||
pending.update(fixes)
|
||||
pending.add(candidate.fix)
|
||||
for fix in pending:
|
||||
# Explicitly fetch detached advisory commits unavailable on default-branch history.
|
||||
# All git including this possible network call still goes through the injected client.
|
||||
git.run("fetch", "--quiet", "--filter=blob:none", "origin", fix)
|
||||
full = git.run("rev-parse", fix).strip()
|
||||
expanded.add(full)
|
||||
parents = git.parents(full)
|
||||
for parent in parents:
|
||||
sites = extract.fix_sites(git, full, parent, global_ok=True)
|
||||
known.update((site["path"], site["function"]) for site in sites)
|
||||
rows = extract.ordinary_history(git, "HEAD")
|
||||
ordinary, exclusions = extract.draw_ordinary(rows, expanded, known, seed=seed)
|
||||
return [change, *ordinary], exclusions, proof
|
||||
|
||||
|
||||
def inputs(used, seed, known_empty, clones):
|
||||
from openultrasast.model.taxonomy import DEFAULT_FAMILIES_PATH
|
||||
|
||||
parameters = {
|
||||
"seed": seed,
|
||||
"used_set_digest": digest({label: sorted(names) for label, names in used.sources.items()}),
|
||||
"known_empty": sorted(known_empty),
|
||||
"clone_limit_bytes": getattr(clones, "limit_bytes", 500 * 1024 * 1024),
|
||||
"journal_version": 1,
|
||||
}
|
||||
# Hardcoded design thresholds and algorithms are covered as well as taxonomy data.
|
||||
for name in ("draft", "source", "extract", "eligibility", "journal", "storage"):
|
||||
parameters["design_" + name] = hashlib.sha256(Path(__file__).with_name(name + ".py").read_bytes()).hexdigest()
|
||||
parameters["design_taxonomy"] = hashlib.sha256(DEFAULT_FAMILIES_PATH.read_bytes()).hexdigest()
|
||||
return parameters
|
||||
|
||||
|
||||
# The pre-memory-fix implementation has identical ordering/selection semantics. Only
|
||||
# this exact fingerprint set may migrate; changed seeds, taxonomy or used sets still fail.
|
||||
LEGACY_DESIGNS = {
|
||||
"design_draft": "906bf36de11761931b2c1b62a2ea8da3c8124f9d754455f1a0acf29a6db576a3",
|
||||
"design_source": "01c896bd0418faf0d073dc2d5cb6dbc5d7903b95b5e8216c9022bc9985326a9c",
|
||||
"design_extract": "593d954d9c22b835e1d349df6b575707a39098dca2eff4425f67a0de14c6525e",
|
||||
"design_journal": "c97ad0761ea5bbfe65db816b13dfa79d93608ea03bcd5848f70686ed9157bdf7",
|
||||
}
|
||||
|
||||
|
||||
def legacy_inputs(parameters):
|
||||
return {key: LEGACY_DESIGNS.get(key, value) for key, value in parameters.items() if key != "design_storage"}
|
||||
|
||||
|
||||
def enumerate_advisories(github, journal):
|
||||
page, records = 1, AdvisoryRows(journal.db, "advisory_records")
|
||||
while True:
|
||||
key = str(page)
|
||||
if key not in journal.rows["advisories"]:
|
||||
journal.boundary()
|
||||
result = github.advisory_page(page)
|
||||
journal.put("advisories", key, result)
|
||||
result = journal.rows["advisories"][key]
|
||||
for row in result["rows"]:
|
||||
records[row["ghsa_id"]] = row
|
||||
if result["done"]:
|
||||
break
|
||||
page += 1
|
||||
if not records:
|
||||
raise InstrumentFailure("empty_advisories")
|
||||
return records
|
||||
|
||||
|
||||
def decide(row, github, osv, used, selected_networks):
|
||||
try:
|
||||
links = source.fix_links(row)
|
||||
if len(links) != 1:
|
||||
raise source.Rejected("fix_links")
|
||||
name, _ = next(iter(links))
|
||||
resolved = eligibility.resolve(name, github)
|
||||
rejected_by = eligibility.rejection(resolved, used, set(selected_networks))
|
||||
if rejected_by:
|
||||
raise source.Rejected("used_" + rejected_by)
|
||||
candidate = source.candidate(row, github, osv)
|
||||
return {"candidate": asdict(candidate), "network": resolved.network}
|
||||
except source.Rejected as exc:
|
||||
return {"rejected": str(exc)}
|
||||
except eligibility.RepositoryNotFound:
|
||||
return {"rejected": "repository_or_commit_not_found"}
|
||||
|
||||
|
||||
def extract_repository(candidate, github, clones, fixes, seed):
|
||||
before = dict(clones.counts)
|
||||
try:
|
||||
with clones.open(candidate.url) as git:
|
||||
git.run("fetch", "--quiet", "--filter=blob:none", "origin", candidate.fix)
|
||||
changes, exclusions, proof = repository_changes(git, candidate, fixes, seed=seed)
|
||||
pinned = changes[0]["head"]
|
||||
license_record = github.get_license(candidate.repository, pinned)
|
||||
spdx, license_class = source.license_info(license_record)
|
||||
entry = {
|
||||
**asdict(candidate),
|
||||
"license_spdx": spdx,
|
||||
"license_class": license_class,
|
||||
"license_ref": pinned,
|
||||
"license_path": license_record.get("path", ""),
|
||||
"changes": changes,
|
||||
"instrument": proof,
|
||||
}
|
||||
result = {"entry": entry, "ordinary_exclusions": exclusions}
|
||||
except source.Rejected as exc:
|
||||
result = {"rejected": str(exc), "ordinary_exclusions": getattr(exc, "counts", {})}
|
||||
except eligibility.RepositoryNotFound:
|
||||
result = {"rejected": "repository_or_commit_not_found", "ordinary_exclusions": {}}
|
||||
result["counts"] = {key: value - before.get(key, 0) for key, value in clones.counts.items()}
|
||||
return result
|
||||
|
||||
|
||||
def publish(journal, assigned, seed):
|
||||
"""Idempotent publication from completed journal data, never inside the checkout."""
|
||||
public, private = [], []
|
||||
for row in assigned:
|
||||
if row["license_class"] == "permissive":
|
||||
public.append(row)
|
||||
else:
|
||||
private.append(row)
|
||||
public.append({"id": row["id"], "sha256": private_digest(row), "slice": row["slice"], "private": True})
|
||||
for name, rows in (("private-draft-p1.toml", private), ("draft-p1.toml", public)):
|
||||
path = journal.path / name
|
||||
data = manifest(rows, seed).encode()
|
||||
if path.exists():
|
||||
if path.read_bytes() != data:
|
||||
raise ValueError("draft_exists")
|
||||
else:
|
||||
atomic(path, data)
|
||||
|
||||
|
||||
def run(
|
||||
root: Path,
|
||||
*,
|
||||
github,
|
||||
osv,
|
||||
clones,
|
||||
used: eligibility.UsedSet,
|
||||
seed: int,
|
||||
known_empty=None,
|
||||
journal_path=None,
|
||||
max_hours=None,
|
||||
clock=None,
|
||||
) -> dict:
|
||||
started = time.monotonic()
|
||||
report = {"status": "sourcing", "seed": seed, "candidates": {}, "rejections": {}, "ordinary_exclusions": {}}
|
||||
counts, rejected, ordinary_counts = Counter(), Counter(), Counter()
|
||||
checkpoint = None
|
||||
try:
|
||||
known_empty = set(known_empty or ())
|
||||
used.report(known_empty)
|
||||
cache = getattr(clones, "cache", root.parent / (root.name + "-cache"))
|
||||
path = external(root, journal_path or cache / "journal")
|
||||
checkpoint = Journal(
|
||||
path, max_hours=max_hours, clock=clock, rate_limit=lambda: getattr(getattr(github, "transport", None), "rate_limit", {})
|
||||
)
|
||||
parameters = inputs(used, seed, known_empty, clones)
|
||||
with checkpoint.open(parameters, compatible_parameters=legacy_inputs(parameters)):
|
||||
checkpoint.boundary()
|
||||
api = ResolvedGitHub(github, checkpoint)
|
||||
used = copy.deepcopy(used) # canonicalization must not mutate the resume input
|
||||
eligibility.canonicalize_used(used, api)
|
||||
report["used_set"] = used.report(known_empty)
|
||||
rows = enumerate_advisories(api, checkpoint)
|
||||
report["advisories"] = len(rows)
|
||||
fixes_by_repo = advisory_index(rows, api, db=checkpoint.db)
|
||||
selected_networks, entries = set(), []
|
||||
for row in source.ordered_advisories(rows, seed):
|
||||
checkpoint.boundary()
|
||||
key = row["ghsa_id"]
|
||||
counts[source.ecosystem(row)] += 1
|
||||
if key not in checkpoint.rows["eligibility"]:
|
||||
decision = decide(row, api, osv, used, selected_networks)
|
||||
checkpoint.put("eligibility", key, decision)
|
||||
decision = checkpoint.rows["eligibility"][key]
|
||||
if "rejected" in decision:
|
||||
rejected[decision["rejected"]] += 1
|
||||
continue
|
||||
candidate = source.Candidate(**decision["candidate"])
|
||||
# Multiple advisories for one repository can have different fix/range inputs.
|
||||
extraction_key = digest({"repository": candidate.repository, "advisory": key, "fix": candidate.fix})
|
||||
if extraction_key not in checkpoint.rows["extraction"]:
|
||||
checkpoint.boundary()
|
||||
result = extract_repository(candidate, api, clones, fixes_by_repo.get(candidate.repository, set()), seed)
|
||||
checkpoint.put("extraction", extraction_key, result)
|
||||
result = checkpoint.rows["extraction"][extraction_key]
|
||||
ordinary_counts.update(result["ordinary_exclusions"])
|
||||
if "rejected" in result:
|
||||
rejected[result["rejected"]] += 1
|
||||
continue
|
||||
selected_networks.add(decision["network"])
|
||||
entries.append(result["entry"])
|
||||
if len(entries) == 300:
|
||||
break
|
||||
report.update(
|
||||
{
|
||||
"repositories": len(entries),
|
||||
"candidates": dict(counts),
|
||||
"rejections": dict(rejected),
|
||||
"ordinary_exclusions": dict(ordinary_counts),
|
||||
}
|
||||
)
|
||||
checkpoint.boundary()
|
||||
if len(entries) != 300:
|
||||
report["status"] = "insufficient_repositories"
|
||||
checkpoint.progress(report["status"])
|
||||
return report
|
||||
assigned = assign_slices(entries, seed=seed)
|
||||
completion = {"manifest_sha256": digest(assigned), "repositories": len(entries)}
|
||||
if "all" not in checkpoint.rows["completion"]:
|
||||
checkpoint.put("completion", "all", completion)
|
||||
elif checkpoint.rows["completion"]["all"] != completion:
|
||||
raise ValueError("journal_completion_mismatch")
|
||||
checkpoint.boundary()
|
||||
publish(checkpoint, assigned, seed)
|
||||
report.update(
|
||||
{
|
||||
"status": "draft",
|
||||
"changes": 3900,
|
||||
"slices": 3,
|
||||
"license_split": dict(Counter(e["license_class"] for e in entries)),
|
||||
"methods": dict(Counter(e["changes"][0]["method"] for e in entries)),
|
||||
"post_cutoff": sum(e["post_cutoff"] for e in entries),
|
||||
**completion,
|
||||
}
|
||||
)
|
||||
checkpoint.progress("draft")
|
||||
return report
|
||||
except ResumeMismatch as exc:
|
||||
report.update({"status": "resume_refused", "changed_parameters": exc.parameters})
|
||||
return report
|
||||
except Stopped as exc:
|
||||
report.update({"status": "stopped", "reason": str(exc)})
|
||||
return report
|
||||
except Exception as exc:
|
||||
report.update({"status": "instrument_failure", "error_type": type(exc).__name__})
|
||||
return report
|
||||
finally:
|
||||
report.update({"candidates": dict(counts), "rejections": dict(rejected), "ordinary_exclusions": dict(ordinary_counts)})
|
||||
report["instrument"] = dict(clones.counts)
|
||||
report["wall_seconds"] = round(time.monotonic() - started, 3)
|
||||
report["github_calls"] = github.calls
|
||||
report["osv_calls"] = osv.calls
|
||||
report["github_http_attempts"] = getattr(getattr(github, "transport", None), "attempts", github.calls)
|
||||
report["osv_http_attempts"] = getattr(getattr(osv, "transport", None), "attempts", osv.calls)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
from openultrasast import config
|
||||
from openultrasast.plane import memory
|
||||
from openultrasast.plane.reconciler import results_root
|
||||
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--root", type=Path, default=Path.cwd())
|
||||
parser.add_argument("--cache", type=Path)
|
||||
parser.add_argument("--max-hours", type=float, help="Stop between units after this many hours per invocation")
|
||||
parser.add_argument("--seed", type=int, default=20260601)
|
||||
parser.add_argument("--local-store")
|
||||
parser.add_argument("--s3-store", default="s3://")
|
||||
parser.add_argument("--pointer-manifest", type=Path, action="append", default=[])
|
||||
parser.add_argument("--known-empty", choices=(*eligibility.SOURCES, "memory_local", "memory_s3"), action="append", default=[])
|
||||
args = parser.parse_args(argv)
|
||||
try:
|
||||
config.load_dotenv(args.root / ".env")
|
||||
# Refuse an existing draft before doing any network work.
|
||||
if any((args.root / p).exists() for p in ("benchmarks/unseen/draft-p1.toml", "benchmarks/unseen/private/draft-p1.toml")):
|
||||
raise FileExistsError("draft_exists")
|
||||
clones = Clones(args.root, args.cache or results_root() / "unseen-draft")
|
||||
stores = {
|
||||
"local": memory.open_store(args.local_store or "file://" + str(results_root() / "memory"), read_only=True),
|
||||
"s3": memory.open_store(args.s3_store, read_only=True),
|
||||
}
|
||||
used = eligibility.build_used(args.root, stores=stores, pointer_manifests=args.pointer_manifest)
|
||||
report = run(
|
||||
args.root,
|
||||
github=source.GitHub(os.environ.get("GH_TOKEN", "")),
|
||||
osv=source.OSV(),
|
||||
clones=clones,
|
||||
used=used,
|
||||
seed=args.seed,
|
||||
known_empty=set(args.known_empty),
|
||||
journal_path=clones.cache / "journal",
|
||||
max_hours=args.max_hours,
|
||||
)
|
||||
except Exception as exc:
|
||||
report = {"status": "instrument_failure", "error_type": type(exc).__name__}
|
||||
print(json.dumps(report, sort_keys=True))
|
||||
return 0 if report["status"] in {"draft", "stopped"} else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,379 @@
|
||||
"""Read-only used-set collection. Only counts, source labels and digests leave this module's CLI.
|
||||
|
||||
Run from the repository root: python -m benchmarks.unseen.eligibility --help.
|
||||
Population v3 is exclusively the existing guard's responsibility.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
import tomllib
|
||||
from collections.abc import Iterable, Mapping
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
import yaml
|
||||
|
||||
from openultrasast.plane.memory import MemoryStore
|
||||
|
||||
URL = re.compile(r"github\.com/([\w.-]+/[\w.-]+)", re.I)
|
||||
NAME = re.compile(r"[\w.-]+/[\w.-]+")
|
||||
SOURCES = ("pairs", "pointers", "populations", "harvest", "units", "measurements", "sweep")
|
||||
|
||||
|
||||
def normalize(value: str) -> str:
|
||||
text = value.strip().lower().rstrip("/")
|
||||
if NAME.fullmatch(text):
|
||||
name = text.removesuffix(".git")
|
||||
else:
|
||||
# Structured fields must be identities, not prose containing a URL.
|
||||
if any(char.isspace() for char in text):
|
||||
raise ValueError("invalid_repository")
|
||||
url = urlsplit(text if "://" in text else "https://" + text)
|
||||
if url.scheme not in {"http", "https", "ssh", "git"} or not url.hostname or "." not in url.hostname:
|
||||
raise ValueError("invalid_repository")
|
||||
parts = url.path.strip("/").split("/")
|
||||
if len(parts) < 2:
|
||||
raise ValueError("invalid_repository")
|
||||
name = "/".join(parts[:2]).removesuffix(".git")
|
||||
if not NAME.fullmatch(name):
|
||||
raise ValueError("invalid_repository")
|
||||
if url.hostname != "github.com":
|
||||
return url.hostname + "/" + name
|
||||
if not NAME.fullmatch(name) or any(part in {".", ".."} for part in name.split("/")):
|
||||
raise ValueError("invalid_repository")
|
||||
return name
|
||||
|
||||
|
||||
def excluded(path: Path) -> bool:
|
||||
return any("v3" in part.lower() or part in {".git", "__pycache__", ".venv"} for part in path.parts)
|
||||
|
||||
|
||||
def repositories(value: object, counts: dict[str, int] | None = None) -> set[str]:
|
||||
"""Structured repo fields include bare owner/name pointers; URLs also occur in prose."""
|
||||
found = set()
|
||||
if isinstance(value, dict):
|
||||
for key, item in value.items():
|
||||
if key in {"repo", "fix_repo", "repository", "full_name", "origin"} or (key == "group" and "unit" in value):
|
||||
try:
|
||||
if not isinstance(item, str):
|
||||
raise ValueError("invalid_repository")
|
||||
found.add(normalize(item))
|
||||
except ValueError:
|
||||
if counts is not None:
|
||||
counts["skipped_values"] = counts.get("skipped_values", 0) + 1
|
||||
else:
|
||||
found.update(repositories(item, counts))
|
||||
elif isinstance(value, list):
|
||||
for item in value:
|
||||
found.update(repositories(item, counts))
|
||||
elif isinstance(value, str):
|
||||
found.update(normalize(m[1]) for m in URL.finditer(value))
|
||||
return found
|
||||
|
||||
|
||||
def pointer_repositories(value: object, counts: dict[str, int] | None = None) -> set[str]:
|
||||
if isinstance(value, dict):
|
||||
if value.get("vendored") is False:
|
||||
return repositories(value, counts)
|
||||
return set().union(*(pointer_repositories(v, counts) for v in value.values()))
|
||||
if isinstance(value, list):
|
||||
return set().union(*(pointer_repositories(v, counts) for v in value))
|
||||
return set()
|
||||
|
||||
|
||||
def parse(path: Path, raw: bytes) -> object:
|
||||
structured = path.suffix in {".toml", ".json", ".jsonl", ".yaml", ".yml"}
|
||||
text = raw.decode("utf-8", errors="strict" if structured else "ignore")
|
||||
if path.suffix == ".toml":
|
||||
return tomllib.loads(text)
|
||||
if path.suffix == ".json":
|
||||
return json.loads(text)
|
||||
if path.suffix == ".jsonl":
|
||||
return [json.loads(line) for line in text.splitlines() if line.strip()]
|
||||
if path.suffix in {".yaml", ".yml"}:
|
||||
documents = list(yaml.safe_load_all(text))
|
||||
return documents[0] if len(documents) == 1 else documents
|
||||
return text
|
||||
|
||||
|
||||
@dataclass
|
||||
class UsedSet:
|
||||
sources: dict[str, set[str]] = field(default_factory=lambda: {label: set() for label in SOURCES})
|
||||
instrument: dict[str, dict[str, int]] = field(default_factory=dict)
|
||||
|
||||
def add(self, label: str, names: set[str], size: int, files: int = 1, skipped_values: int = 0) -> None:
|
||||
self.sources.setdefault(label, set()).update(names)
|
||||
counts = self.instrument.setdefault(label, {"files": 0, "bytes": 0, "rows": 0, "skipped_values": 0})
|
||||
counts["files"] += files
|
||||
counts["bytes"] += size
|
||||
counts["rows"] += len(names)
|
||||
counts["skipped_values"] += skipped_values
|
||||
|
||||
def report(self, known_empty: set[str] | None = None) -> dict:
|
||||
known_empty = known_empty or set()
|
||||
if known_empty - self.sources.keys():
|
||||
raise ValueError("unknown_source_label")
|
||||
records = {}
|
||||
for label, names in self.sources.items():
|
||||
counts = self.instrument.get(label, {"files": 0, "bytes": 0, "rows": 0, "skipped_values": 0})
|
||||
if not counts["rows"] and label not in known_empty:
|
||||
raise ValueError("empty_source:" + label)
|
||||
records[label] = {**counts, "count": len(names), "known_empty": label in known_empty}
|
||||
names = set().union(*self.sources.values())
|
||||
return {"sources": records, "count": len(names), "sha256": hashlib.sha256("\n".join(sorted(names)).encode()).hexdigest()}
|
||||
|
||||
|
||||
def build_used(root: Path, *, stores: Mapping[str, MemoryStore] | None = None, pointer_manifests: Iterable[Path] = ()) -> UsedSet:
|
||||
used = UsedSet()
|
||||
|
||||
def read(label: str, paths: Iterable[Path], structured: bool = True) -> list[object]:
|
||||
values = []
|
||||
for path in sorted(set(paths)):
|
||||
if excluded(path):
|
||||
continue
|
||||
raw = path.read_bytes() # failure is an instrument failure, never a quiet empty source
|
||||
value = parse(path, raw) if structured else raw.decode("utf-8", errors="ignore")
|
||||
counts: dict[str, int] = {}
|
||||
names = repositories(value, counts)
|
||||
used.add(label, names, len(raw), skipped_values=counts.get("skipped_values", 0))
|
||||
if label == "pairs":
|
||||
pointer_counts: dict[str, int] = {}
|
||||
pointers = pointer_repositories(value, pointer_counts)
|
||||
if pointers or pointer_counts:
|
||||
used.add("pointers", pointers, len(raw), skipped_values=pointer_counts.get("skipped_values", 0))
|
||||
values.append(value)
|
||||
return values
|
||||
|
||||
pairs = root / "benchmarks/pairs"
|
||||
read("pairs", (p for p in pairs.rglob("*") if p.is_file() and p.suffix in {".toml", ".json", ".jsonl", ".csv", ".yaml", ".yml"}))
|
||||
read("pointers", pointer_manifests)
|
||||
read("populations", (p for p in (root / "benchmarks/independent").glob("population-v[12].toml")))
|
||||
read("harvest", (p for directory in pairs.glob("advisory-fixes*") for p in directory.rglob("*.toml")))
|
||||
units = set((root / "plane/experiments").glob("*.units.jsonl"))
|
||||
for path in (root / "plane/experiments").glob("*.y*ml"):
|
||||
if excluded(path):
|
||||
continue
|
||||
raw = path.read_bytes()
|
||||
manifest = parse(path, raw)
|
||||
counts = {}
|
||||
names = repositories(manifest, counts)
|
||||
used.add("units", names, len(raw), skipped_values=counts.get("skipped_values", 0))
|
||||
if isinstance(manifest, dict) and manifest.get("units_file"):
|
||||
declared = root / Path(manifest["units_file"]).expanduser()
|
||||
if declared.exists(): # an unregistered plan may not have frozen units yet
|
||||
units.add(declared)
|
||||
for label, store in (stores or {}).items():
|
||||
if label not in {"local", "s3"}:
|
||||
raise ValueError("unknown_store_label")
|
||||
source = "memory_" + label
|
||||
used.add(source, set(), 0, 0)
|
||||
# Read actual objects, not a serialized row size masquerading as bytes read.
|
||||
for key in store._keys("repos/"):
|
||||
if not key.endswith(".jsonl") or excluded(Path(key)):
|
||||
continue
|
||||
result = store._get(key)
|
||||
if result is None:
|
||||
raise ValueError("missing_memory_object")
|
||||
raw = result[0]
|
||||
rows = parse(Path("rows.jsonl"), raw)
|
||||
names = set()
|
||||
counts = {}
|
||||
for row in rows:
|
||||
if row.get("kind") == "example" and "repo" in row:
|
||||
names.update(repositories({"repo": row["repo"]}, counts))
|
||||
if row.get("kind") == "experiment" and row.get("units_file"):
|
||||
registered = root / Path(row["units_file"]).expanduser()
|
||||
if row.get("units_digest") or registered.exists():
|
||||
units.add(registered)
|
||||
used.add(source, names, len(raw), skipped_values=counts.get("skipped_values", 0))
|
||||
read("units", units)
|
||||
read("measurements", (p for top in ("measurements", "experiments") for p in (root / "benchmarks" / top).rglob("*") if p.is_file()))
|
||||
read(
|
||||
"sweep",
|
||||
(
|
||||
p
|
||||
for top in ("benchmarks", "src", "tests", "plane", ".kiro")
|
||||
for p in (root / top).rglob("*")
|
||||
if p.is_file() and not p.is_relative_to(root / "benchmarks/unseen") and not excluded(p)
|
||||
),
|
||||
structured=False,
|
||||
)
|
||||
return used
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Resolution:
|
||||
full_name: str
|
||||
names: frozenset[str]
|
||||
network: str
|
||||
|
||||
|
||||
def resolve(candidate: str, api) -> Resolution:
|
||||
name = normalize(candidate)
|
||||
if name.count("/") > 1:
|
||||
return Resolution(name, frozenset({name}), name)
|
||||
data = api.get_repo(name)
|
||||
canonical = normalize(data["full_name"])
|
||||
names = {name, canonical, *(normalize(n) for n in data.get("former_names", []))}
|
||||
for key in ("parent", "source"):
|
||||
if data.get(key):
|
||||
names.add(normalize(data[key]["full_name"]))
|
||||
network = normalize((data.get("source") or data.get("parent") or data)["full_name"])
|
||||
return Resolution(canonical, frozenset(names), network)
|
||||
|
||||
|
||||
def canonicalize_used(used: UsedSet, api) -> None:
|
||||
"""Resolve old used names too: GitHub exposes redirects, not a complete rename history."""
|
||||
cache = {}
|
||||
for names in used.sources.values():
|
||||
for name in sorted(names):
|
||||
if name not in cache:
|
||||
try:
|
||||
cache[name] = resolve(name, api)
|
||||
except RepositoryNotFound:
|
||||
cache[name] = Resolution(name, frozenset({name}), name)
|
||||
names.update(cache[name].names)
|
||||
|
||||
|
||||
def rejection(candidate: Resolution, used: UsedSet, selected: set[str]) -> str | None:
|
||||
for label, names in used.sources.items():
|
||||
if candidate.names & names:
|
||||
return label
|
||||
if candidate.network in selected:
|
||||
return "fork_network"
|
||||
selected.add(candidate.network)
|
||||
return None
|
||||
|
||||
|
||||
class RepositoryNotFound(ValueError):
|
||||
"""A historical or synthetic used identity no longer resolves; retain it verbatim."""
|
||||
|
||||
|
||||
class GitHub:
|
||||
"""Coordinator-only GET client. Exceptions never disclose request URLs or tokens."""
|
||||
|
||||
def __init__(self, token: str):
|
||||
if not token:
|
||||
raise ValueError("missing_gh_token")
|
||||
self.token = token
|
||||
|
||||
def get_repo(self, name: str) -> dict:
|
||||
from urllib.error import HTTPError
|
||||
from urllib.request import Request, urlopen
|
||||
|
||||
request = Request(
|
||||
"https://api.github.com/repos/" + normalize(name),
|
||||
headers={
|
||||
"Authorization": "Bearer " + self.token,
|
||||
"Accept": "application/vnd.github+json",
|
||||
},
|
||||
)
|
||||
try:
|
||||
with urlopen(request, timeout=30) as response:
|
||||
return json.load(response)
|
||||
except HTTPError as exc:
|
||||
if exc.code == 404:
|
||||
raise RepositoryNotFound("github_not_found") from None
|
||||
raise ValueError("github_resolution_failed") from None
|
||||
except Exception:
|
||||
raise ValueError("github_resolution_failed") from None
|
||||
|
||||
|
||||
def verify_tests(root: Path) -> dict:
|
||||
"""The coordinator runs the guard as a black box; only exit codes and counts escape."""
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from xml.etree import ElementTree
|
||||
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
report = Path(directory) / "tests.xml"
|
||||
done = subprocess.run(
|
||||
[
|
||||
sys.executable,
|
||||
"-m",
|
||||
"pytest",
|
||||
"-q",
|
||||
"-p",
|
||||
"no:cacheprovider",
|
||||
"tests/test_unseen_eligibility.py",
|
||||
"tests/test_unseen_ledger.py",
|
||||
"tests/test_independent_population.py",
|
||||
"--junitxml=" + str(report),
|
||||
],
|
||||
cwd=root,
|
||||
stdin=subprocess.DEVNULL,
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL,
|
||||
check=False,
|
||||
)
|
||||
if done.returncode != 0:
|
||||
raise ValueError("guard_or_tests_failed")
|
||||
suites = ElementTree.parse(report).getroot().iter("testsuite")
|
||||
counts = {key: 0 for key in ("tests", "failures", "errors", "skipped")}
|
||||
for suite in suites:
|
||||
for key in counts:
|
||||
counts[key] += int(suite.attrib[key])
|
||||
if not counts["tests"]:
|
||||
raise ValueError("empty_test_run")
|
||||
return {"guard": "green", "test_exit_code": done.returncode, "tests": counts}
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
import os
|
||||
|
||||
from openultrasast import config
|
||||
from openultrasast.plane import memory
|
||||
from openultrasast.plane.reconciler import results_root
|
||||
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--root", type=Path, default=Path.cwd())
|
||||
parser.add_argument("--local-store", default=None)
|
||||
parser.add_argument("--s3-store", default="s3://")
|
||||
parser.add_argument("--pointer-manifest", type=Path, action="append", default=[])
|
||||
parser.add_argument("--known-empty", choices=(*SOURCES, "memory_local", "memory_s3"), action="append", default=[])
|
||||
parser.add_argument("--output", type=Path)
|
||||
parser.add_argument("--verify-tests", action="store_true")
|
||||
args = parser.parse_args(argv)
|
||||
step = "open_stores"
|
||||
try:
|
||||
config.load_dotenv(args.root / ".env")
|
||||
stores = {
|
||||
"local": memory.open_store(args.local_store or "file://" + str(results_root() / "memory"), read_only=True),
|
||||
"s3": memory.open_store(args.s3_store, read_only=True),
|
||||
}
|
||||
step = "build_used"
|
||||
used = build_used(args.root, stores=stores, pointer_manifests=args.pointer_manifest)
|
||||
step = "report"
|
||||
used.report(set(args.known_empty)) # prove inputs before API work
|
||||
step = "canonicalize"
|
||||
canonicalize_used(used, GitHub(os.environ.get("GH_TOKEN", "")))
|
||||
step = "report"
|
||||
report = used.report(set(args.known_empty))
|
||||
if args.verify_tests:
|
||||
step = "verify_tests"
|
||||
report.update(verify_tests(args.root))
|
||||
step = "report"
|
||||
text = json.dumps(report, sort_keys=True, indent=2) + "\n"
|
||||
if args.output:
|
||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||
with args.output.open("x") as handle:
|
||||
handle.write(text)
|
||||
print(text, end="")
|
||||
return 0
|
||||
except Exception as exc:
|
||||
# Source paths, store exceptions and JSON parser errors can contain repository names.
|
||||
report = {"status": "instrument_failure", "step": step, "type": type(exc).__name__}
|
||||
if str(exc) in {"empty_source:" + label for label in (*SOURCES, "memory_local", "memory_s3")}:
|
||||
report["source"] = str(exc).partition(":")[2]
|
||||
print(json.dumps(report))
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,519 @@
|
||||
"""Git-only change extraction; clients and subprocess execution are injectable.
|
||||
|
||||
No scanner, engine or model runs here. Function boundaries are syntax, not a security
|
||||
judgement: the first removed code line of a fix is the SZZ sink proxy. Ambiguous sites
|
||||
are dropped, and the recorded SZZ bias remains subject to task 2.6's label check.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import ast
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
import tempfile
|
||||
from contextlib import contextmanager
|
||||
from pathlib import Path
|
||||
|
||||
from .source import Candidate, Rejected
|
||||
from .storage import working_store
|
||||
|
||||
LANGUAGES = {".py", ".js", ".jsx", ".mjs", ".cjs", ".ts", ".tsx", ".php", ".java"}
|
||||
HUNK = re.compile(r"^@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@", re.M)
|
||||
SHA = re.compile(r"[a-f0-9]{40}")
|
||||
BIAS = {
|
||||
"introducing": "SZZ last-writer proxy: refactors, moves and reformats can misidentify the introduction in either direction.",
|
||||
"last_touch": "Pessimistic lower bound: the flaw may already exist in base, so a delta hook can correctly stay quiet.",
|
||||
}
|
||||
|
||||
|
||||
class InstrumentFailure(ValueError):
|
||||
"""A failed read or command must never masquerade as an empty history."""
|
||||
|
||||
|
||||
class Git:
|
||||
def __init__(self, path: Path, *, runner=None, limit_bytes: int | None = None):
|
||||
self.path = path
|
||||
self.runner = runner or subprocess.run
|
||||
self.bytes_read = 0
|
||||
self.files_read = 0
|
||||
self.last_blob_bytes = 0
|
||||
self.limit_bytes = limit_bytes
|
||||
|
||||
@contextmanager
|
||||
def output(self, *args: str, allowed=(0,)):
|
||||
# Git can expand a tiny pack into enormous stdout. Spool before decoding, including
|
||||
# stderr, so neither subprocess.communicate nor an error can consume the VM's RAM.
|
||||
with tempfile.TemporaryFile() as stdout, tempfile.TemporaryFile() as stderr:
|
||||
try:
|
||||
done = self.runner(
|
||||
["git", "-c", "core.quotePath=false", "-C", str(self.path), *args],
|
||||
stdin=subprocess.DEVNULL,
|
||||
stdout=stdout,
|
||||
stderr=stderr,
|
||||
timeout=180,
|
||||
check=False,
|
||||
)
|
||||
except Exception:
|
||||
raise InstrumentFailure("git_launch_failed") from None
|
||||
if done.returncode not in allowed:
|
||||
raise InstrumentFailure("git_exit_" + str(done.returncode))
|
||||
if self.limit_bytes is not None and self.disk_bytes() > self.limit_bytes:
|
||||
raise InstrumentFailure("clone_size_cap")
|
||||
# Injected runners may return bytes instead of writing the supplied file.
|
||||
if done.stdout is not None:
|
||||
stdout.write(done.stdout)
|
||||
stdout.seek(0)
|
||||
yield stdout
|
||||
|
||||
def run(self, *args: str, allowed=(0,)) -> str:
|
||||
with self.output(*args, allowed=allowed) as stdout:
|
||||
raw = stdout.read(8 * 1024 * 1024 + 1)
|
||||
if len(raw) > 8 * 1024 * 1024:
|
||||
raise InstrumentFailure("git_output_size_cap")
|
||||
self.last_blob_bytes = len(raw)
|
||||
return raw.decode("utf-8", errors="replace")
|
||||
|
||||
def history_entries(self, head):
|
||||
with self.output("log", "--first-parent", "--format=%H%x00%P%x00%ct%x00%an%x00%B%x1e", head) as stdout:
|
||||
pending = b""
|
||||
while chunk := stdout.read(64 * 1024):
|
||||
entries = (pending + chunk).split(b"\x1e")
|
||||
pending = entries.pop()
|
||||
for entry in entries:
|
||||
if len(entry) > 1024 * 1024:
|
||||
raise InstrumentFailure("git_history_record_size_cap")
|
||||
yield entry.decode("utf-8", errors="replace")
|
||||
if len(pending) > 1024 * 1024:
|
||||
raise InstrumentFailure("git_history_record_size_cap")
|
||||
if pending.strip():
|
||||
raise InstrumentFailure("invalid_history")
|
||||
|
||||
def disk_bytes(self) -> int:
|
||||
return sum(p.stat().st_size for p in self.path.rglob("*") if p.is_file() and not p.is_symlink())
|
||||
|
||||
def parents(self, sha: str) -> list[str]:
|
||||
return self.run("rev-list", "--parents", "-n", "1", sha).split()[1:]
|
||||
|
||||
def blob(self, sha: str, path: str) -> str | None:
|
||||
# Missing is established by a successful tree read, never by swallowing a show error.
|
||||
if not self.run("ls-tree", sha, "--", path).strip():
|
||||
return None
|
||||
size = int(self.run("cat-file", "-s", f"{sha}:{path}").strip())
|
||||
if size > 512 * 1024:
|
||||
raise InstrumentFailure("source_blob_size_cap")
|
||||
text = self.run("show", f"{sha}:{path}")
|
||||
if Path(path).suffix in LANGUAGES:
|
||||
self.files_read += 1
|
||||
self.bytes_read += self.last_blob_bytes
|
||||
return text
|
||||
|
||||
def stats(self, base: str, head: str) -> list[tuple[str, int, int]]:
|
||||
rows = []
|
||||
for line in self.run("diff", "--no-renames", "--numstat", base, head, "--").splitlines():
|
||||
added, removed, path = line.split("\t", 2)
|
||||
rows.append((path, int(added) if added.isdigit() else 0, int(removed) if removed.isdigit() else 0))
|
||||
return rows
|
||||
|
||||
def hunks(self, base: str, head: str, path: str) -> list[tuple[int, int, int, int]]:
|
||||
diff = self.run("diff", "--no-ext-diff", "--no-renames", "-U0", base, head, "--", path)
|
||||
return [(int(m[1]), int(m[2] or 1), int(m[3]), int(m[4] or 1)) for m in HUNK.finditer(diff)]
|
||||
|
||||
def ancestor(self, older: str, newer: str) -> bool:
|
||||
# rev-list avoids interpreting a command failure as a negative ancestry result.
|
||||
return not self.run("rev-list", "--max-count=1", older, "--not", newer).strip()
|
||||
|
||||
def blame(self, revision: str, path: str, line: int) -> tuple[str, str, int]:
|
||||
text = self.run("blame", "-w", "-M", "-C", "--porcelain", f"-L{line},{line}", revision, "--", path)
|
||||
header = text.splitlines()[0].split()
|
||||
if len(header) < 3 or not SHA.fullmatch(header[0]):
|
||||
raise InstrumentFailure("invalid_blame")
|
||||
filename = next((row[9:] for row in text.splitlines() if row.startswith("filename ")), path)
|
||||
if filename.startswith('"'):
|
||||
filename = json.loads(filename)
|
||||
return header[0], filename, int(header[1])
|
||||
|
||||
|
||||
def functions(path: str, text: str) -> list[tuple[str, int, int]]:
|
||||
"""Conservative syntax boundaries. No scanner import; unresolved sites are rejected."""
|
||||
if Path(path).suffix == ".py":
|
||||
try:
|
||||
tree = ast.parse(text)
|
||||
except SyntaxError:
|
||||
raise Rejected("function_parse") from None
|
||||
return [
|
||||
(n.name, n.lineno, n.end_lineno or n.lineno) for n in ast.walk(tree) if isinstance(n, ast.FunctionDef | ast.AsyncFunctionDef)
|
||||
]
|
||||
# Blank comments and quoted strings before brace matching. Keep newlines/offsets.
|
||||
masked = re.sub(
|
||||
r'/\*.*?\*/|//[^\n]*|\#[^\n]*|"(?:\\.|[^"\\])*"|\'(?:\\.|[^\'\\])*\'|`(?:\\.|[^`\\])*`',
|
||||
lambda m: "".join("\n" if c == "\n" else " " for c in m[0]),
|
||||
text,
|
||||
flags=re.S,
|
||||
)
|
||||
declaration = re.compile(
|
||||
r"\bfunction\s*&?\s*(\w+)\s*\([^)]*\)\s*(?::[^\n{]+)?\s*\{|"
|
||||
r"\b(?:const|let|var)\s+(\w+)\s*=\s*(?:async\s*)?(?:\([^)]*\)|\w+)\s*=>\s*\{|"
|
||||
r"\b([\w$]+)\s*\([^;{}]*\)\s*(?:throws\s+[\w., ]+)?\s*\{"
|
||||
)
|
||||
found = []
|
||||
for match in declaration.finditer(masked):
|
||||
name = next(g for g in match.groups() if g)
|
||||
if name in {"if", "while", "for", "switch", "catch", "with", "synchronized"}:
|
||||
continue
|
||||
depth = 1
|
||||
end = match.end()
|
||||
while end < len(masked) and depth:
|
||||
depth += (masked[end] == "{") - (masked[end] == "}")
|
||||
end += 1
|
||||
if depth == 0:
|
||||
found.append((name, masked.count("\n", 0, match.start()) + 1, masked.count("\n", 0, end) + 1))
|
||||
return found
|
||||
|
||||
|
||||
def enclosing(path: str, text: str, line: int, *, global_ok=False) -> tuple[str, int, int] | None:
|
||||
spans = [s for s in functions(path, text) if s[1] <= line <= s[2]]
|
||||
if spans:
|
||||
return min(spans, key=lambda s: s[2] - s[1])
|
||||
return ("<global>", 1, max(1, len(text.splitlines()))) if global_ok else None
|
||||
|
||||
|
||||
def normalized(text: str) -> str:
|
||||
return re.sub(r"\s+", "", text)
|
||||
|
||||
|
||||
def site_at(git, revision: str, path: str, name: str):
|
||||
text = git.blob(revision, path)
|
||||
if text is None:
|
||||
return None
|
||||
spans = [("<global>", 1, max(1, len(text.splitlines())))] if name == "<global>" else [s for s in functions(path, text) if s[0] == name]
|
||||
if len(spans) != 1:
|
||||
return None
|
||||
return text, spans[0]
|
||||
|
||||
|
||||
def fix_sites(git, fix: str, parent: str, *, global_ok=False) -> list[dict]:
|
||||
sites = []
|
||||
for path, _, _ in git.stats(parent, fix):
|
||||
if Path(path).suffix not in LANGUAGES:
|
||||
continue
|
||||
text = git.blob(parent, path)
|
||||
if text is None:
|
||||
continue
|
||||
lines = text.splitlines()
|
||||
for start, count, _, _ in git.hunks(parent, fix, path):
|
||||
for line in range(max(1, start), max(1, start) + max(1, count)):
|
||||
if line > len(lines):
|
||||
continue
|
||||
span = enclosing(path, text, line, global_ok=global_ok)
|
||||
if span is None:
|
||||
continue
|
||||
sink_line = line if count else span[1]
|
||||
sink = lines[sink_line - 1]
|
||||
if not sink.strip() or sink.lstrip().startswith(("#", "//", "/*", "*")):
|
||||
continue
|
||||
key = (path, span[0])
|
||||
if any((s["path"], s["function"]) == key for s in sites):
|
||||
continue
|
||||
sites.append({"path": path, "function": span[0], "line": sink_line, "text": sink, "absence": count == 0, "span": span})
|
||||
return sites
|
||||
|
||||
|
||||
def version_bracket(git, candidate: Candidate, revision: str) -> bool:
|
||||
if not candidate.first_affected:
|
||||
return True
|
||||
tags = git.run("tag", "--sort=version:refname").splitlines()
|
||||
matching = [t for t in tags if t.removeprefix("v") == candidate.first_affected]
|
||||
if len(matching) != 1:
|
||||
return False
|
||||
affected = matching[0]
|
||||
earlier = [t for t in tags[: tags.index(affected)] if git.ancestor(t, affected)]
|
||||
return bool(earlier) and git.ancestor(revision, affected) and not git.ancestor(revision, earlier[-1])
|
||||
|
||||
|
||||
def change_files(git, base: str, head: str) -> list[dict]:
|
||||
return [
|
||||
{"path": path, "head_lines": [[max(1, new), max(1, new + max(count, 1) - 1)] for _, _, new, count in git.hunks(base, head, path)]}
|
||||
for path, _, _ in git.stats(base, head)
|
||||
]
|
||||
|
||||
|
||||
def introducing(git, candidate: Candidate) -> dict:
|
||||
from collections import Counter
|
||||
|
||||
if git.parents(candidate.fix) != [candidate.parent]:
|
||||
raise Rejected("fix_parents")
|
||||
sites = fix_sites(git, candidate.fix, candidate.parent, global_ok=candidate.family == "config_secrets")
|
||||
reasons = Counter()
|
||||
|
||||
def record(method, base, head, path, name):
|
||||
resolved = site_at(git, head, path, name)
|
||||
if resolved is None:
|
||||
return None
|
||||
return {
|
||||
"kind": "vulnerability",
|
||||
"base": base,
|
||||
"head": head,
|
||||
"method": method,
|
||||
"family": candidate.family,
|
||||
"function": name,
|
||||
"path": path,
|
||||
"function_lines": list(resolved[1][1:]),
|
||||
"files": change_files(git, base, head),
|
||||
"post_cutoff": candidate.post_cutoff,
|
||||
"advisory": candidate.advisory,
|
||||
"bias": BIAS[method],
|
||||
"fallback_reasons": dict(reasons),
|
||||
}
|
||||
|
||||
for site in sites:
|
||||
revision, path, line = candidate.parent, site["path"], site["line"]
|
||||
for _ in range(6): # initial blame plus at most five walk-back steps
|
||||
blamed, blamed_path, blamed_line = git.blame(revision, path, line)
|
||||
parents = git.parents(blamed)
|
||||
if len(parents) != 1:
|
||||
reasons["merge_or_root"] += 1
|
||||
break
|
||||
base = parents[0]
|
||||
head_site = site_at(git, blamed, blamed_path, site["function"])
|
||||
if head_site is None:
|
||||
reasons["unresolved_function"] += 1
|
||||
break
|
||||
parent_text = git.blob(base, blamed_path)
|
||||
if parent_text is not None and site["function"] != "<global>":
|
||||
matching = [span for span in functions(blamed_path, parent_text) if span[0] == site["function"]]
|
||||
if len(matching) > 1:
|
||||
reasons["ambiguous_parent"] += 1
|
||||
break
|
||||
old_site = site_at(git, base, blamed_path, site["function"])
|
||||
occurrences = []
|
||||
if old_site:
|
||||
text, (_, start, end) = old_site
|
||||
occurrences = [
|
||||
i for i, value in enumerate(text.splitlines(), 1) if start <= i <= end and normalized(value) == normalized(site["text"])
|
||||
]
|
||||
if occurrences:
|
||||
reasons["text_present"] += 1
|
||||
revision, path, line = base, blamed_path, occurrences[0]
|
||||
continue
|
||||
stats = git.stats(base, blamed)
|
||||
if len(stats) > 50 or sum(a + d for _, a, d in stats) > 2000:
|
||||
reasons["size_cap"] += 1
|
||||
break
|
||||
if not version_bracket(git, candidate, blamed):
|
||||
reasons["version_bracket"] += 1
|
||||
break
|
||||
result = record("introducing", base, blamed, blamed_path, site["function"])
|
||||
if result:
|
||||
return result
|
||||
else:
|
||||
reasons["walkback_limit"] += 1
|
||||
for site in sites:
|
||||
# Numeric function extent implements log -L for all five languages, including
|
||||
# languages for which Git has no built-in :function: driver.
|
||||
_, start, end = site["span"]
|
||||
history = git.run("log", "--format=%H", "--no-patch", f"-L{start},{end}:{site['path']}", candidate.parent).splitlines()
|
||||
if not history:
|
||||
continue
|
||||
latest = history[0]
|
||||
parents = git.parents(latest)
|
||||
if len(parents) != 1:
|
||||
continue
|
||||
result = record("last_touch", parents[0], candidate.parent, site["path"], site["function"])
|
||||
if result:
|
||||
return result
|
||||
raise Rejected("no_range")
|
||||
|
||||
|
||||
SECURITY = re.compile(r"secur|cve|ghsa|vuln|inject|xss|csrf|ssrf|exploit|sanitiz|escap", re.I)
|
||||
DEPENDENCY = re.compile(r"\bbump\b|\bupgrade .* from\b", re.I)
|
||||
MANIFESTS = {
|
||||
"package.json",
|
||||
"package-lock.json",
|
||||
"yarn.lock",
|
||||
"pnpm-lock.yaml",
|
||||
"composer.json",
|
||||
"composer.lock",
|
||||
"pom.xml",
|
||||
"build.gradle",
|
||||
"build.gradle.kts",
|
||||
"gradle.lockfile",
|
||||
"pyproject.toml",
|
||||
"setup.py",
|
||||
"setup.cfg",
|
||||
"Pipfile",
|
||||
"Pipfile.lock",
|
||||
"poetry.lock",
|
||||
"uv.lock",
|
||||
}
|
||||
|
||||
|
||||
def generated_path(path: str) -> bool:
|
||||
return bool({"vendor", "node_modules", "dist"} & set(Path(path).parts)) or path.endswith(".min.js")
|
||||
|
||||
|
||||
def manifest(path: str) -> bool:
|
||||
name = Path(path).name
|
||||
return name in MANIFESTS or name.endswith(".lock") or bool(re.fullmatch(r"requirements[^/]*\.txt", name))
|
||||
|
||||
|
||||
def ordinary_rejection(row: dict, fixes: set[str], known_functions: set[tuple[str, str]]) -> str | None:
|
||||
if len(row["parents"]) != 1:
|
||||
return "merge_or_root"
|
||||
if row["head"] in fixes:
|
||||
return "advisory_fix"
|
||||
if SECURITY.search(row["message"]):
|
||||
return "security"
|
||||
if re.search(r"^Revert|This reverts commit", row["message"], re.I | re.M):
|
||||
return "revert"
|
||||
if re.search(r"dependabot|renovate|\[bot\]", row["author"], re.I) or DEPENDENCY.search(row["message"]):
|
||||
return "dependency"
|
||||
files = row["files"]
|
||||
if files and all(manifest(f["path"]) for f in files):
|
||||
return "manifest_only"
|
||||
source = [f for f in files if Path(f["path"]).suffix in LANGUAGES and not manifest(f["path"])]
|
||||
if not source:
|
||||
return "no_source"
|
||||
if all(f["generated"] or generated_path(f["path"]) for f in source):
|
||||
return "generated"
|
||||
if any((f["path"], name) in known_functions for f in files for name in f["functions"]):
|
||||
return "vulnerable_function"
|
||||
if any((f["path"], "<global>") in known_functions for f in files):
|
||||
return "vulnerable_function"
|
||||
if sum(f["added"] + f["removed"] for f in source) > 1000:
|
||||
return "size_cap"
|
||||
return None
|
||||
|
||||
|
||||
def generated(git, revision: str, path: str, text: str) -> bool:
|
||||
import fnmatch
|
||||
|
||||
if generated_path(path):
|
||||
return True
|
||||
header = "\n".join(text.splitlines()[:20])
|
||||
if re.search(r"(?im)^\s*(?:#|//|/\*|\*|<!--).*?(?:@generated|auto.?generated|generated\s+(?:by|file|code)|do not edit)", header):
|
||||
return True
|
||||
result = False
|
||||
parts = Path(path).parts
|
||||
for index in range(len(parts)):
|
||||
directory = "/".join(parts[:index])
|
||||
attributes = git.blob(revision, (directory + "/" if directory else "") + ".gitattributes") or ""
|
||||
relative = "/".join(parts[index:])
|
||||
for line in attributes.splitlines():
|
||||
tokens = line.split()
|
||||
if not tokens or tokens[0].startswith("#"):
|
||||
continue
|
||||
pattern = tokens[0].lstrip("/")
|
||||
target = relative if "/" in pattern else Path(relative).name
|
||||
if fnmatch.fnmatchcase(target, pattern):
|
||||
for token in tokens[1:]:
|
||||
if token in {"linguist-generated", "linguist-generated=true"}:
|
||||
result = True
|
||||
elif token in {"-linguist-generated", "!linguist-generated", "linguist-generated=false"}:
|
||||
result = False
|
||||
return result
|
||||
|
||||
|
||||
def ordinary_history(git, head: str):
|
||||
"""Yield one commit at a time; no repository-wide blob or diff retention."""
|
||||
for entry in git.history_entries(head):
|
||||
if not entry.strip():
|
||||
continue
|
||||
fields = entry.strip().split("\x00")
|
||||
if len(fields) != 5 or not SHA.fullmatch(fields[0]):
|
||||
raise InstrumentFailure("invalid_history")
|
||||
sha, parents, timestamp, author, message = fields
|
||||
row = {"head": sha, "parents": parents.split(), "timestamp": int(timestamp), "author": author, "message": message, "files": []}
|
||||
if len(row["parents"]) != 1:
|
||||
yield row
|
||||
continue
|
||||
base = row["parents"][0]
|
||||
for path, added, removed in git.stats(base, sha):
|
||||
hunks = git.hunks(base, sha, path)
|
||||
info = {
|
||||
"path": path,
|
||||
"added": added,
|
||||
"removed": removed,
|
||||
"head_lines": [[max(1, n), max(1, n + max(c, 1) - 1)] for _, _, n, c in hunks],
|
||||
"functions": [],
|
||||
"generated": False,
|
||||
}
|
||||
if Path(path).suffix in LANGUAGES:
|
||||
# Inspect both sides: deleting a vulnerable function is still touching it.
|
||||
for revision, side in ((base, 0), (sha, 2)):
|
||||
text = git.blob(revision, path)
|
||||
if text is None:
|
||||
continue
|
||||
spans = functions(path, text)
|
||||
for hunk in hunks:
|
||||
start, count = hunk[side : side + 2]
|
||||
for name, lo, hi in spans:
|
||||
if max(1, start) <= hi and start + max(count, 1) - 1 >= lo:
|
||||
info["functions"].append(name)
|
||||
info["generated"] |= generated(git, revision, path, text)
|
||||
row["files"].append(info)
|
||||
yield row
|
||||
|
||||
|
||||
def draw_ordinary(rows, fixes: set[str], known_functions: set[tuple[str, str]], *, seed: int) -> tuple[list[dict], dict]:
|
||||
"""Spill eligible history to disk; keep the original seeded stratified draw exactly."""
|
||||
with working_store() as db:
|
||||
return _draw_ordinary(db, rows, fixes, known_functions, seed=seed)
|
||||
|
||||
|
||||
def _draw_ordinary(db, rows, fixes, known_functions, *, seed):
|
||||
import random
|
||||
from collections import Counter
|
||||
|
||||
counts = Counter()
|
||||
earliest = latest = None
|
||||
db.execute("CREATE TABLE eligible (head TEXT, timestamp INTEGER, bucket INTEGER, value TEXT, third INTEGER)")
|
||||
for row in rows:
|
||||
timestamp = row["timestamp"]
|
||||
earliest = timestamp if earliest is None else min(earliest, timestamp)
|
||||
latest = timestamp if latest is None else max(latest, timestamp)
|
||||
reason = ordinary_rejection(row, fixes, known_functions)
|
||||
counts[reason or "eligible"] += 1
|
||||
if reason is not None:
|
||||
continue
|
||||
size = sum(f["added"] + f["removed"] for f in row["files"] if Path(f["path"]).suffix in LANGUAGES and not manifest(f["path"]))
|
||||
bucket = 0 if size <= 10 else 1 if size <= 50 else 2 if size <= 200 else 3
|
||||
value = {
|
||||
"kind": "ordinary",
|
||||
"base": row["parents"][0],
|
||||
"head": row["head"],
|
||||
"method": "first_parent",
|
||||
"size_bucket": bucket,
|
||||
"files": [{"path": f["path"], "head_lines": f["head_lines"]} for f in row["files"]],
|
||||
}
|
||||
db.execute("INSERT INTO eligible VALUES (?, ?, ?, ?, NULL)", (row["head"], timestamp, bucket, json.dumps(value)))
|
||||
if counts["eligible"] < 40:
|
||||
failure = Rejected("ordinary_floor")
|
||||
failure.counts = dict(counts)
|
||||
raise failure
|
||||
db.execute("UPDATE eligible SET third = min(2, 3 * (timestamp - ?) / ?)", (earliest, max(1, latest - earliest + 1)))
|
||||
cells = {
|
||||
(third, bucket): count for third, bucket, count in db.execute("SELECT third, bucket, count(*) FROM eligible GROUP BY third, bucket")
|
||||
}
|
||||
if {t for t, _ in cells} != {0, 1, 2}:
|
||||
failure = Rejected("time_thirds")
|
||||
failure.counts = dict(counts)
|
||||
raise failure
|
||||
allocation = Counter()
|
||||
targets = {cell: 12 * count / counts["eligible"] for cell, count in cells.items()}
|
||||
for third in range(3):
|
||||
cell = max(sorted(c for c in cells if c[0] == third), key=lambda c: targets[c])
|
||||
allocation[cell] = 1
|
||||
while sum(allocation.values()) < 12:
|
||||
cell = max(sorted(c for c in cells if allocation[c] < cells[c]), key=lambda c: targets[c] - allocation[c])
|
||||
allocation[cell] += 1
|
||||
db.execute("CREATE INDEX eligible_order ON eligible(third, bucket, head)")
|
||||
rng = random.Random(seed)
|
||||
drawn = []
|
||||
for cell in sorted(cells):
|
||||
for position in rng.sample(range(cells[cell]), allocation[cell]):
|
||||
(value,) = db.execute(
|
||||
"SELECT value FROM eligible WHERE third=? AND bucket=? ORDER BY head LIMIT 1 OFFSET ?", (*cell, position)
|
||||
).fetchone()
|
||||
drawn.append({**json.loads(value), "time_third": cell[0]})
|
||||
return drawn, dict(counts)
|
||||
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"ghsa_id": "GHSA-test-test-test",
|
||||
"published_at": "2026-06-02T00:00:00Z",
|
||||
"type": "reviewed",
|
||||
"withdrawn_at": null,
|
||||
"cwes": [{"cwe_id": "CWE-89"}],
|
||||
"vulnerabilities": [{"package": {"ecosystem": "pip", "name": "fixture"}}],
|
||||
"references": ["https://github.com/fixture/project/commit/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"]
|
||||
}
|
||||
@@ -0,0 +1,860 @@
|
||||
[
|
||||
{
|
||||
"id": "0",
|
||||
"repo": "0",
|
||||
"kind": "ordinary",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": [
|
||||
{
|
||||
"cwe": "CWE-89",
|
||||
"path": "app.py",
|
||||
"line": 10
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "1",
|
||||
"repo": "0",
|
||||
"kind": "ordinary",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "2",
|
||||
"repo": "1",
|
||||
"kind": "ordinary",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "3",
|
||||
"repo": "1",
|
||||
"kind": "ordinary",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "4",
|
||||
"repo": "2",
|
||||
"kind": "ordinary",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "5",
|
||||
"repo": "2",
|
||||
"kind": "ordinary",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "6",
|
||||
"repo": "3",
|
||||
"kind": "ordinary",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "7",
|
||||
"repo": "3",
|
||||
"kind": "ordinary",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "8",
|
||||
"repo": "4",
|
||||
"kind": "ordinary",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "9",
|
||||
"repo": "4",
|
||||
"kind": "ordinary",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "10",
|
||||
"repo": "5",
|
||||
"kind": "ordinary",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "11",
|
||||
"repo": "5",
|
||||
"kind": "ordinary",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "12",
|
||||
"repo": "6",
|
||||
"kind": "ordinary",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "13",
|
||||
"repo": "6",
|
||||
"kind": "ordinary",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "14",
|
||||
"repo": "7",
|
||||
"kind": "ordinary",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "15",
|
||||
"repo": "7",
|
||||
"kind": "ordinary",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "16",
|
||||
"repo": "8",
|
||||
"kind": "vulnerability",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": [
|
||||
{
|
||||
"cwe": "CWE-89",
|
||||
"path": "app.py",
|
||||
"line": 10
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "17",
|
||||
"repo": "8",
|
||||
"kind": "vulnerability",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": [
|
||||
{
|
||||
"cwe": "CWE-89",
|
||||
"path": "app.py",
|
||||
"line": 10
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "18",
|
||||
"repo": "9",
|
||||
"kind": "vulnerability",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "19",
|
||||
"repo": "9",
|
||||
"kind": "vulnerability",
|
||||
"family": "injection",
|
||||
"method": "introducing",
|
||||
"post_cutoff": true,
|
||||
"label_check": "confirmed",
|
||||
"path": "app.py",
|
||||
"function": "handler",
|
||||
"files": [
|
||||
{
|
||||
"path": "app.py",
|
||||
"head_lines": [
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"instrument": {
|
||||
"head_verified": true,
|
||||
"base_verified": true,
|
||||
"changed_bytes": {
|
||||
"app.py": 20
|
||||
},
|
||||
"hook_bytes_read": 20,
|
||||
"hook_exit": 0
|
||||
},
|
||||
"push": {
|
||||
"quick_tier": [
|
||||
{
|
||||
"files": [
|
||||
"app.py"
|
||||
],
|
||||
"bytes_read": 20,
|
||||
"findings": []
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,215 @@
|
||||
"""Private, append-only sourcing checkpoints. No identity-bearing data leaves this directory."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import fcntl
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import time
|
||||
from contextlib import contextmanager
|
||||
from pathlib import Path
|
||||
|
||||
from .source import compact_advisory, compact_repository
|
||||
from .storage import DiskMap, working_store
|
||||
|
||||
MAX_RECORD_BYTES = 16 * 1024 * 1024
|
||||
|
||||
|
||||
def compact(stage, value):
|
||||
if stage == "advisories":
|
||||
return {"rows": [compact_advisory(row) for row in value["rows"]], "done": value["done"]}
|
||||
if stage == "resolution" and value is not None:
|
||||
return compact_repository(value)
|
||||
return value
|
||||
|
||||
|
||||
STAGES = ("advisories", "resolution", "eligibility", "extraction", "completion")
|
||||
|
||||
|
||||
def encoded(value) -> bytes:
|
||||
return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode()
|
||||
|
||||
|
||||
def digest(value) -> str:
|
||||
return hashlib.sha256(encoded(value)).hexdigest()
|
||||
|
||||
|
||||
class ResumeMismatch(ValueError):
|
||||
def __init__(self, parameters):
|
||||
self.parameters = parameters
|
||||
super().__init__("resume_inputs_changed")
|
||||
|
||||
|
||||
class Stopped(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def external(root: Path, path: Path) -> Path:
|
||||
path = path.expanduser().resolve()
|
||||
# Also reject the actual checkout when a different --root is supplied.
|
||||
if any(path.is_relative_to(p.resolve()) for p in (root, Path(__file__).resolve().parents[2])):
|
||||
raise ValueError("cache_inside_repository")
|
||||
return path
|
||||
|
||||
|
||||
def atomic(path: Path, payload: bytes) -> None:
|
||||
temporary = path.with_suffix(path.suffix + ".tmp")
|
||||
with temporary.open("wb") as handle:
|
||||
os.chmod(temporary, 0o600)
|
||||
handle.write(payload)
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
temporary.replace(path)
|
||||
sync_directory(path.parent)
|
||||
|
||||
|
||||
def sync_directory(path: Path) -> None:
|
||||
fd = os.open(path, os.O_RDONLY | os.O_DIRECTORY)
|
||||
try:
|
||||
os.fsync(fd)
|
||||
finally:
|
||||
os.close(fd)
|
||||
|
||||
|
||||
class Journal:
|
||||
def __init__(self, path: Path, *, max_hours=None, clock=None, rate_limit=None):
|
||||
if max_hours is not None and (not math.isfinite(max_hours) or max_hours < 0):
|
||||
raise ValueError("invalid_max_hours")
|
||||
self.path = path
|
||||
self.clock = clock or time.monotonic
|
||||
self.started = self.clock()
|
||||
self.max_seconds = None if max_hours is None else max_hours * 3600
|
||||
self.rate_limit = rate_limit or (lambda: {})
|
||||
self.rows = {}
|
||||
self.elapsed_before = 0.0
|
||||
self.last = None
|
||||
self.previous_rate = {}
|
||||
|
||||
@contextmanager
|
||||
def open(self, parameters, *, compatible_parameters=None):
|
||||
self.path.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
with (self.path / "LOCK").open("a") as lock:
|
||||
try:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except BlockingIOError:
|
||||
raise ValueError("journal_busy") from None
|
||||
metadata = self.path / "inputs.json"
|
||||
migrating = False
|
||||
if metadata.exists():
|
||||
old = json.loads(metadata.read_bytes())
|
||||
# Only current, fixed parameter labels can enter stdout.
|
||||
changed = [key for key in parameters if old.get(key) != parameters[key]]
|
||||
if old.keys() != parameters.keys():
|
||||
changed.append("parameter_schema")
|
||||
if changed:
|
||||
if old != compatible_parameters:
|
||||
raise ResumeMismatch(changed)
|
||||
migrating = True
|
||||
else:
|
||||
if any(self.path.glob("*.jsonl")):
|
||||
raise ValueError("journal_missing_inputs")
|
||||
atomic(metadata, encoded(parameters))
|
||||
with working_store(self.path) as self.db:
|
||||
self.rows = {stage: DiskMap(self.db, stage) for stage in STAGES}
|
||||
self._replay()
|
||||
if migrating:
|
||||
atomic(self.path / "inputs-before-memory-fix.json", encoded(old))
|
||||
atomic(metadata, encoded(parameters))
|
||||
self.progress("sourcing")
|
||||
yield self
|
||||
|
||||
def _replay(self):
|
||||
for stage in STAGES:
|
||||
path = self.path / (stage + ".jsonl")
|
||||
if not path.exists():
|
||||
continue
|
||||
with path.open("rb") as handle:
|
||||
while line := handle.readline(MAX_RECORD_BYTES + 1):
|
||||
if len(line) > MAX_RECORD_BYTES:
|
||||
raise ValueError("journal_record_size_cap")
|
||||
# Fail closed on torn/corrupt writes; never silently lose a completed unit.
|
||||
if not line.endswith(b"\n"):
|
||||
raise ValueError("journal_incomplete_record")
|
||||
row = json.loads(line)
|
||||
if row["id"] in self.rows[stage]:
|
||||
raise ValueError("journal_duplicate_unit")
|
||||
self.rows[stage][row["id"]] = compact(stage, row["value"])
|
||||
if row["elapsed_seconds"] >= self.elapsed_before:
|
||||
self.elapsed_before = row["elapsed_seconds"]
|
||||
self.last = {"stage": stage, "id": row["id"]}
|
||||
self.previous_rate = row["rate_limit"]
|
||||
|
||||
def boundary(self):
|
||||
reason = None
|
||||
if (self.path / "STOP").exists():
|
||||
reason = "STOP"
|
||||
elif self.max_seconds is not None and self.clock() - self.started >= self.max_seconds:
|
||||
reason = "max_hours"
|
||||
if reason:
|
||||
self.progress("stopped", reason=reason)
|
||||
raise Stopped(reason)
|
||||
|
||||
def elapsed(self):
|
||||
return self.elapsed_before + self.clock() - self.started
|
||||
|
||||
def progress(self, status, **extra):
|
||||
atomic(
|
||||
self.path / "progress.json",
|
||||
encoded(
|
||||
{
|
||||
"status": status,
|
||||
"counts": {stage: len(rows) for stage, rows in self.rows.items()},
|
||||
"last_unit": self.last,
|
||||
"rate_limit": self.rate_limit() or self.previous_rate,
|
||||
"elapsed_seconds": self.elapsed(),
|
||||
**extra,
|
||||
}
|
||||
),
|
||||
)
|
||||
|
||||
def put(self, stage, key, value):
|
||||
if key in self.rows[stage]:
|
||||
raise ValueError("journal_duplicate_unit")
|
||||
value = compact(stage, value)
|
||||
row = {"id": key, "value": value, "elapsed_seconds": self.elapsed(), "rate_limit": self.rate_limit() or self.previous_rate}
|
||||
path = self.path / (stage + ".jsonl")
|
||||
with path.open("ab") as handle:
|
||||
os.chmod(path, 0o600)
|
||||
payload = encoded(row) + b"\n"
|
||||
if len(payload) > MAX_RECORD_BYTES:
|
||||
raise ValueError("journal_record_size_cap")
|
||||
handle.write(payload)
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
sync_directory(self.path)
|
||||
self.rows[stage][key] = value
|
||||
self.last = {"stage": stage, "id": key}
|
||||
self.progress("sourcing")
|
||||
|
||||
|
||||
class ResolvedGitHub:
|
||||
"""Persist successful repository lookups and 404s used by both eligibility and the fix index."""
|
||||
|
||||
def __init__(self, github, journal):
|
||||
self.github, self.journal = github, journal
|
||||
|
||||
def __getattr__(self, name):
|
||||
return getattr(self.github, name)
|
||||
|
||||
def get_repo(self, name):
|
||||
from .eligibility import RepositoryNotFound, normalize
|
||||
|
||||
key = normalize(name)
|
||||
if key not in self.journal.rows["resolution"]:
|
||||
self.journal.boundary()
|
||||
try:
|
||||
value = self.github.get_repo(key)
|
||||
except RepositoryNotFound:
|
||||
value = None
|
||||
self.journal.put("resolution", key, value)
|
||||
value = self.journal.rows["resolution"][key]
|
||||
if value is None:
|
||||
raise RepositoryNotFound("not_found")
|
||||
return value
|
||||
@@ -0,0 +1,88 @@
|
||||
"""Append-only slice usage; a hash chain detects rewritten rows, not a rebuilt entire history.
|
||||
|
||||
Keep the published final digest to authenticate a history against truncation or wholesale replacement.
|
||||
Only opaque decision/experiment labels belong here; repository identities are never accepted.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import fcntl
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
from datetime import date
|
||||
from pathlib import Path
|
||||
|
||||
FIELDS = {"slice", "decision", "purpose", "date", "result_digest", "freeze_digest"}
|
||||
PURPOSES = {"baseline", "exploratory", "informed", "qualifies"}
|
||||
ZERO = "0" * 64
|
||||
|
||||
|
||||
def digest(row: dict) -> str:
|
||||
return hashlib.sha256(json.dumps(row, sort_keys=True, separators=(",", ":")).encode()).hexdigest()
|
||||
|
||||
|
||||
def validate(row: dict) -> None:
|
||||
if set(row) != FIELDS or row["purpose"] not in PURPOSES:
|
||||
raise ValueError("invalid_usage_fields")
|
||||
if type(row["slice"]) is not int or row["slice"] < 1:
|
||||
raise ValueError("invalid_slice")
|
||||
if not isinstance(row["decision"], str) or not re.fullmatch(r"[A-Za-z0-9_-]+", row["decision"]):
|
||||
raise ValueError("invalid_decision_label")
|
||||
for key in ("result_digest", "freeze_digest"):
|
||||
if not isinstance(row[key], str) or not re.fullmatch(r"[0-9a-f]{64}", row[key]):
|
||||
raise ValueError("invalid_digest")
|
||||
try:
|
||||
if date.fromisoformat(row["date"]).isoformat() != row["date"]:
|
||||
raise ValueError
|
||||
except (ValueError, TypeError):
|
||||
raise ValueError("invalid_date") from None
|
||||
|
||||
|
||||
def _read(text: str) -> list[dict]:
|
||||
previous = ZERO
|
||||
rows = []
|
||||
if text and not text.endswith("\n"):
|
||||
raise ValueError("broken_digest_chain")
|
||||
for line in text.splitlines():
|
||||
try:
|
||||
row = json.loads(line)
|
||||
stored = row.pop("digest")
|
||||
if row["previous"] != previous or digest(row) != stored:
|
||||
raise ValueError
|
||||
validate({k: v for k, v in row.items() if k != "previous"})
|
||||
except (ValueError, KeyError, TypeError, AttributeError):
|
||||
raise ValueError("broken_digest_chain") from None
|
||||
rows.append({**row, "digest": stored})
|
||||
previous = stored
|
||||
return rows
|
||||
|
||||
|
||||
def read(path: Path) -> list[dict]:
|
||||
if not path.exists():
|
||||
return []
|
||||
with path.open() as handle:
|
||||
fcntl.flock(handle, fcntl.LOCK_SH)
|
||||
return _read(handle.read())
|
||||
|
||||
|
||||
def append(path: Path, row: dict) -> str:
|
||||
validate(row)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with path.open("a+") as handle:
|
||||
fcntl.flock(handle, fcntl.LOCK_EX)
|
||||
handle.seek(0)
|
||||
rows = _read(handle.read())
|
||||
if any(all(old[key] == row[key] for key in FIELDS) for old in rows):
|
||||
raise ValueError("duplicate_usage")
|
||||
if rows and rows[0]["freeze_digest"] != row["freeze_digest"]:
|
||||
raise ValueError("freeze_digest_changed")
|
||||
if row["purpose"] == "qualifies" and any(
|
||||
old["slice"] == row["slice"] and old["decision"] == row["decision"] and old["purpose"] == "informed" for old in rows
|
||||
):
|
||||
raise ValueError("slice_informed_decision")
|
||||
entry = {**row, "previous": rows[-1]["digest"] if rows else ZERO}
|
||||
sha = digest(entry)
|
||||
handle.write(json.dumps({**entry, "digest": sha}, sort_keys=True) + "\n")
|
||||
handle.flush()
|
||||
return sha
|
||||
@@ -0,0 +1,128 @@
|
||||
"""Build anonymous coordinator proof records from completed dispatcher outputs."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def digest(value):
|
||||
return hashlib.sha256(json.dumps(value, sort_keys=True).encode()).hexdigest()
|
||||
|
||||
|
||||
def records(root):
|
||||
return [json.loads(path.read_text()) for path in sorted(root.glob("*.json")) if path.name not in {"progress.json", "inputs.json"}]
|
||||
|
||||
|
||||
def replay_proof(roots):
|
||||
lanes = {lane: records(root) for lane, root in roots.items()}
|
||||
if any(len(rows) != 5 for rows in lanes.values()):
|
||||
raise ValueError("proof requires exactly five completed changes per lane")
|
||||
keys = [{row["input_digest"] for row in rows} for rows in lanes.values()]
|
||||
if not all(key == keys[0] for key in keys):
|
||||
raise ValueError("lane input identities differ")
|
||||
quick = {}
|
||||
evidence = {}
|
||||
for lane, rows in lanes.items():
|
||||
quick[lane] = {}
|
||||
evidence[lane] = []
|
||||
for row in rows:
|
||||
inst = row.get("instrument", {})
|
||||
if (
|
||||
row.get("status") not in {"quick_only", "engine_covered"}
|
||||
or not inst.get("hook_bytes_read")
|
||||
or inst.get("suspect_fast")
|
||||
or inst.get("jvm_wall_seconds", 0) < 5
|
||||
):
|
||||
raise ValueError("proof instrument has unread input or implausible JVM time")
|
||||
quick[lane][row["input_digest"]] = digest([t.get("findings", []) for t in row["push"].get("quick_tier", [])])
|
||||
evidence[lane].append(
|
||||
{
|
||||
"input_digest": row["input_digest"],
|
||||
"seconds": inst["seconds"],
|
||||
"jvm_seconds": inst["jvm_wall_seconds"],
|
||||
"bytes": inst["hook_bytes_read"],
|
||||
"exit_code": inst["hook_exit"],
|
||||
}
|
||||
)
|
||||
equal = all(value == next(iter(quick.values())) for value in quick.values())
|
||||
if not equal:
|
||||
raise ValueError("quick-tier findings differ across lanes")
|
||||
return {"lanes": evidence, "quick_tier_equal": True, "quick_digests": quick, "cleanup": "completed-before-checkpoint"}
|
||||
|
||||
|
||||
def engine_proof(root, baseline):
|
||||
current = records(root)
|
||||
old = {digest([r["repo"], r["pin"]]): r for r in records(baseline) if "repo" in r and "pin" in r}
|
||||
current = [r for r in current if "repo" in r and "pin" in r]
|
||||
if len(current) != 2:
|
||||
raise ValueError("engine proof requires exactly two pins")
|
||||
evidence = []
|
||||
|
||||
def stable(record):
|
||||
return [
|
||||
{key: u.get(key) for key in ("unit", "status", "questions_asked", "questions_completed", "witness_rows", "traces")}
|
||||
for u in record["units"]
|
||||
]
|
||||
|
||||
for row in current:
|
||||
key = digest([row["repo"], row["pin"]])
|
||||
previous = old[key]
|
||||
instruments = [u.get("instrument", {}) for u in row["units"]]
|
||||
if (
|
||||
not row.get("done")
|
||||
or not any(i.get("bytes", 0) > 0 for i in instruments)
|
||||
or not any(stage.get("seconds", 0) >= 5 for i in instruments for stage in i.get("jvm", []))
|
||||
):
|
||||
raise ValueError("engine proof lacks bytes or realistic JVM time")
|
||||
evidence.append(
|
||||
{
|
||||
"pin_digest": key,
|
||||
"seconds": row.get("seconds"),
|
||||
"record_digest": digest(stable(row)),
|
||||
"baseline_digest": digest(stable(previous)),
|
||||
"equal": stable(row) == stable(previous),
|
||||
"instrument": [
|
||||
{
|
||||
"bytes": u.get("instrument", {}).get("bytes", 0),
|
||||
"cpg_bytes": u.get("instrument", {}).get("cpg_bytes", 0),
|
||||
"jvm": [
|
||||
{key: stage.get(key) for key in ("seconds", "exit_code", "success", "heap_mb")}
|
||||
for stage in u.get("instrument", {}).get("jvm", [])
|
||||
],
|
||||
}
|
||||
for u in row["units"]
|
||||
],
|
||||
}
|
||||
)
|
||||
return {"pins": evidence, "differences": sum(not e["equal"] for e in evidence), "cleanup": "completed-before-checkpoint"}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("kind", choices=("engine", "replay"))
|
||||
parser.add_argument("--image", required=True)
|
||||
parser.add_argument("--out", type=Path, required=True)
|
||||
parser.add_argument("--records", type=Path)
|
||||
parser.add_argument("--baseline", type=Path)
|
||||
parser.add_argument("--kind-records", type=Path)
|
||||
parser.add_argument("--ax-records", type=Path)
|
||||
parser.add_argument("--vm-records", type=Path)
|
||||
args = parser.parse_args()
|
||||
result = (
|
||||
engine_proof(args.records, args.baseline)
|
||||
if args.kind == "engine"
|
||||
else replay_proof({"kind": args.kind_records, "ax": args.ax_records, "vm": args.vm_records})
|
||||
)
|
||||
result.update(
|
||||
image=args.image, commit=subprocess.run(["git", "rev-parse", "HEAD"], capture_output=True, text=True, check=True).stdout.strip()
|
||||
)
|
||||
args.out.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.out.write_text(json.dumps(result, sort_keys=True, indent=2) + "\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,193 @@
|
||||
"""Dispatch frozen unseen changes or a five-change project-history proof."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import subprocess
|
||||
import tarfile
|
||||
import tomllib
|
||||
from pathlib import Path
|
||||
|
||||
from benchmarks.ax.batch import AXLane, DockerLane, Item, Workload, dispatch, write_json
|
||||
from benchmarks.unseen.score import coverage
|
||||
|
||||
|
||||
def make_item(change, arm, image):
|
||||
if arm not in {"default", "long_deadline"}:
|
||||
raise ValueError("unknown replay arm")
|
||||
identity = hashlib.sha256(str(change["id"]).encode()).hexdigest()[:14]
|
||||
spec = {**change, "id": identity, "hook_flags": [] if arm == "default" else ["--deadline", "300"], "image": image}
|
||||
data = json.dumps(spec, sort_keys=True).encode()
|
||||
digest = hashlib.sha256(data).hexdigest()
|
||||
return Item(
|
||||
"replay-" + identity,
|
||||
{"SPEC_URL": data},
|
||||
"result.dat",
|
||||
("python3", "/app/benchmarks/unseen/replay_entry.py"),
|
||||
{},
|
||||
digest,
|
||||
{"SPEC_URL": "spec.dat"},
|
||||
)
|
||||
|
||||
|
||||
def validate_result(data, output):
|
||||
with tarfile.open(fileobj=io.BytesIO(data)) as archive:
|
||||
members = archive.getmembers()
|
||||
required = {"push.json", "push.txt", "engine.json", "instrument.json"}
|
||||
if {m.name for m in members} != required or any(not m.isfile() for m in members):
|
||||
raise ValueError("replay result requires push, engine and instrument.json")
|
||||
payload = {m.name: archive.extractfile(m).read() for m in members}
|
||||
instrument = json.loads(payload["instrument.json"])
|
||||
push = json.loads(payload["push.json"])
|
||||
record = {"instrument": instrument, "push": push}
|
||||
record["status"] = coverage(record)
|
||||
# Transport/scheduler needs an explicit OOM signal, not a quiet failed replay.
|
||||
if "OutOfMemoryError" in json.dumps(push) or "JVM out of memory" in json.dumps(push):
|
||||
record["status"] = "oom"
|
||||
output.mkdir(parents=True, exist_ok=True)
|
||||
for name, body in payload.items():
|
||||
(output / name).write_bytes(body)
|
||||
(output / "result.dat").write_bytes(data)
|
||||
write_json(output / "result.json", record)
|
||||
return record
|
||||
|
||||
|
||||
def proof_changes(root):
|
||||
"""Only the operator's project history; no pool data or fetches."""
|
||||
|
||||
def git(*args):
|
||||
return subprocess.run(["git", "-C", str(root), *args], capture_output=True, text=True, check=True).stdout.strip()
|
||||
|
||||
changes = []
|
||||
for head in git("rev-list", "--no-merges", "--max-count=100", "HEAD", "--", "src").splitlines():
|
||||
base = git("rev-parse", head + "^")
|
||||
changed = git("diff", "--name-only", base, head).splitlines()
|
||||
if not any(name.startswith("src/") and name.endswith(".py") for name in changed):
|
||||
continue
|
||||
changes.append(
|
||||
{
|
||||
"id": hashlib.sha256(head.encode()).hexdigest(),
|
||||
"repository_url": "https://github.com/norandom/OpenUltraSAST.git",
|
||||
"repo": "project-proof",
|
||||
"kind": "ordinary",
|
||||
"base": base,
|
||||
"head": head,
|
||||
}
|
||||
)
|
||||
if len(changes) == 5:
|
||||
break
|
||||
if len(changes) != 5:
|
||||
raise ValueError("history lacks five source changes")
|
||||
return changes
|
||||
|
||||
|
||||
def canonical(value):
|
||||
return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode()
|
||||
|
||||
|
||||
def load_slice(path, slice_id, private_path=None):
|
||||
if path.suffix != ".toml":
|
||||
frozen = json.loads(path.read_text())
|
||||
changes = frozen["changes"]
|
||||
if frozen["sha256"] != hashlib.sha256(json.dumps(changes, sort_keys=True).encode()).hexdigest():
|
||||
raise ValueError("frozen changes digest mismatch")
|
||||
return changes
|
||||
if not path.name.startswith("pool-"):
|
||||
raise ValueError("replay requires a frozen pool, never a draft")
|
||||
pool = tomllib.loads(path.read_text())
|
||||
freeze = json.loads(path.with_name("freeze-" + path.stem.removeprefix("pool-") + ".json").read_text())
|
||||
if hashlib.sha256(canonical(pool)).hexdigest() != freeze["freeze_digest"]:
|
||||
raise ValueError("frozen pool digest mismatch")
|
||||
private_path = private_path or path.parent / "private" / path.name
|
||||
private = {}
|
||||
if private_path.exists():
|
||||
private = {row["id"]: row for row in tomllib.loads(private_path.read_text()).get("repository", [])}
|
||||
changes = []
|
||||
for public in pool.get("repository", []):
|
||||
if str(public["slice"]) != str(slice_id):
|
||||
continue
|
||||
row = public
|
||||
if public.get("private"):
|
||||
row = private.get(public["id"])
|
||||
if row is None:
|
||||
raise ValueError("missing private pool entry")
|
||||
payload = {"url": row["url"], "commits": [[c["base"], c["head"]] for c in row["changes"]]}
|
||||
if hashlib.sha256(canonical(payload)).hexdigest() != public["sha256"]:
|
||||
raise ValueError("private pool digest mismatch")
|
||||
for change in row["changes"]:
|
||||
identity = hashlib.sha256(canonical([public["id"], change["kind"], change["base"], change["head"]])).hexdigest()
|
||||
changes.append(
|
||||
{
|
||||
**change,
|
||||
"id": identity,
|
||||
"repo": public["id"],
|
||||
"repository_url": row["url"],
|
||||
"freeze_digest": freeze["freeze_digest"],
|
||||
"slice": slice_id,
|
||||
}
|
||||
)
|
||||
if not changes:
|
||||
raise ValueError("frozen slice has no changes")
|
||||
return changes
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--manifest", type=Path, required=True, help="frozen JSON: changes array plus sha256 of canonical changes JSON")
|
||||
parser.add_argument("--proof-history", action="store_true", help="write a five-change own-history manifest, then exit")
|
||||
parser.add_argument("--image")
|
||||
parser.add_argument("--private-manifest", type=Path)
|
||||
parser.add_argument("--dry-run", action="store_true")
|
||||
parser.add_argument("--arm", choices=("default", "long_deadline"), default="default")
|
||||
parser.add_argument("--pool", default="p1")
|
||||
parser.add_argument("--slice", default="1")
|
||||
parser.add_argument("--out", type=Path)
|
||||
parser.add_argument("--lane", choices=("ax", "kind", "docker", "mixed"), default="mixed")
|
||||
parser.add_argument("--ax-bin", default="ax")
|
||||
parser.add_argument("--kubeconfig", type=Path)
|
||||
parser.add_argument("--atespace", default="default")
|
||||
args = parser.parse_args()
|
||||
if args.proof_history:
|
||||
changes = proof_changes(Path.cwd())
|
||||
write_json(args.manifest, {"changes": changes, "sha256": hashlib.sha256(json.dumps(changes, sort_keys=True).encode()).hexdigest()})
|
||||
return 0
|
||||
if not args.image:
|
||||
parser.error("--image digest is required")
|
||||
changes = load_slice(args.manifest, args.slice, args.private_manifest)
|
||||
items = [make_item(change, args.arm, args.image) for change in changes]
|
||||
if len({i.id for i in items}) != len(items):
|
||||
parser.error("duplicate change identity")
|
||||
args.out = args.out or Path.home() / "ousast-results/unseen" / args.pool / args.slice / args.arm
|
||||
if args.dry_run:
|
||||
print(json.dumps({"items": len(items), "input_digest": hashlib.sha256(canonical([i.input_digest for i in items])).hexdigest()}))
|
||||
return 0
|
||||
if args.out.resolve().is_relative_to(Path(__file__).resolve().parents[2]):
|
||||
parser.error("raw replay output must be outside the repository")
|
||||
write_json(
|
||||
args.out / "inputs.json",
|
||||
[{"item_id": item.id, "change": change, "input_digest": item.input_digest} for item, change in zip(items, changes, strict=True)],
|
||||
)
|
||||
workload = Workload(args.image, frozenset({"SPEC_URL", "RESULT_URL"}), 600 if args.arm == "default" else 900, validate_result)
|
||||
lanes = ("ax", "ax", "docker") if args.lane == "mixed" else (args.lane,) * (2 if args.lane in {"ax", "kind"} else 1)
|
||||
executors = {}
|
||||
for lane in set(lanes):
|
||||
transport = (DockerLane if lane == "docker" else AXLane)(args, workload)
|
||||
|
||||
def execute(item, output, attempt, transport=transport):
|
||||
record = {**json.loads(item.inputs["SPEC_URL"]), **transport(item, output, attempt)}
|
||||
if (output / "result.dat").exists():
|
||||
# Durable archive lives separately from attempt staging objects, which are deleted.
|
||||
key = f"unseen/{item.input_digest}/result.dat"
|
||||
transport.store._put(key, (output / "result.dat").read_bytes())
|
||||
record["result_object"] = key
|
||||
return record
|
||||
|
||||
executors[lane] = execute
|
||||
return 2 if dispatch(items, workload, args.out, executors, lanes=lanes) else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,219 @@
|
||||
"""Credential-free replay actor. Standard library only; PUT is completion."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "learn"))
|
||||
from engine_task_entry import transfer
|
||||
|
||||
CLUSTER_FLAGS = (
|
||||
"-Xms1700m -Xmx1700m -XX:MaxMetaspaceSize=256m -XX:ReservedCodeCacheSize=128m "
|
||||
"-XX:MaxDirectMemorySize=256m -Xss512k -XX:ActiveProcessorCount=2 -XX:+UseG1GC"
|
||||
)
|
||||
|
||||
|
||||
class ReplayFailure(Exception):
|
||||
"""Locally controlled diagnostic only."""
|
||||
|
||||
|
||||
def validate_spec(spec):
|
||||
parts = urlsplit(spec["repository_url"])
|
||||
if (
|
||||
parts.scheme != "https"
|
||||
or not parts.hostname
|
||||
or "." not in parts.hostname
|
||||
or parts.username
|
||||
or parts.password
|
||||
or parts.query
|
||||
or parts.fragment
|
||||
):
|
||||
raise ReplayFailure("repository must be public HTTPS")
|
||||
if parts.port not in (None, 443) or parts.hostname.endswith((".local", ".internal", ".localhost")):
|
||||
raise ReplayFailure("repository must be public HTTPS")
|
||||
import ipaddress
|
||||
|
||||
try:
|
||||
ipaddress.ip_address(parts.hostname)
|
||||
except ValueError:
|
||||
pass
|
||||
else:
|
||||
raise ReplayFailure("repository must use public DNS")
|
||||
if any(not re.fullmatch("[0-9a-f]{40,64}", spec[key]) for key in ("base", "head")):
|
||||
raise ReplayFailure("invalid revision")
|
||||
if spec.get("hook_flags", []) not in ([], ["--deadline", "300"]):
|
||||
raise ReplayFailure("unsupported hook flags")
|
||||
|
||||
|
||||
def main(*, work_root=Path("/tmp"), runner=subprocess.run, heap_profile="cluster", clock=time.monotonic):
|
||||
output = work_root / "out"
|
||||
output.mkdir(parents=True, exist_ok=True)
|
||||
instrument = {
|
||||
"status": "instrument_failure",
|
||||
"head_verified": False,
|
||||
"base_verified": False,
|
||||
"changed_bytes": {},
|
||||
"checkout_files": 0,
|
||||
"checkout_bytes": 0,
|
||||
"hook_bytes_read": 0,
|
||||
"jvm_wall_seconds": 0,
|
||||
"jvm_time_basis": "scan build_seconds + query_seconds",
|
||||
"hook_exit": None,
|
||||
"timings": {},
|
||||
"heap_profile": heap_profile,
|
||||
}
|
||||
spec = {}
|
||||
step = "GET spec"
|
||||
started = clock()
|
||||
|
||||
def run(command, *, timeout=180, env=None):
|
||||
before = clock()
|
||||
result = runner(command, capture_output=True, text=True, timeout=timeout, env=env)
|
||||
instrument["timings"][step] = instrument["timings"].get(step, 0) + clock() - before
|
||||
if result.returncode:
|
||||
raise ReplayFailure(f"exit {result.returncode}")
|
||||
return result
|
||||
|
||||
try:
|
||||
spec_path = work_root / "spec.dat"
|
||||
transfer(os.environ["SPEC_URL"], spec_path, readiness_budget=60)
|
||||
step = "validate spec"
|
||||
spec = json.loads(spec_path.read_text())
|
||||
instrument["image"] = spec.get("image")
|
||||
validate_spec(spec)
|
||||
case = work_root / "case"
|
||||
case.mkdir()
|
||||
git = ["git", "-c", "core.hooksPath=/dev/null", "-c", "protocol.file.allow=never", "-C", str(case)]
|
||||
step = "git init"
|
||||
run([*git, "init", "-q"])
|
||||
run([*git, "remote", "add", "origin", spec["repository_url"]])
|
||||
# One total fetch budget, independent of the hook deadline.
|
||||
fetch_end = clock() + 180
|
||||
for key in ("head", "base"):
|
||||
step = "fetch " + key
|
||||
remaining = fetch_end - clock()
|
||||
if remaining <= 0:
|
||||
raise ReplayFailure("fetch deadline")
|
||||
run([*git, "fetch", "--no-tags", "--depth", "1", "origin", spec[key]], timeout=remaining)
|
||||
step = "verify " + key
|
||||
run([*git, "cat-file", "-e", spec[key] + "^{commit}"])
|
||||
instrument[key + "_verified"] = True
|
||||
step = "checkout"
|
||||
run([*git, "checkout", "--detach", spec["head"]])
|
||||
paths = run([*git, "ls-files", "-z"]).stdout.split("\0")
|
||||
for name in filter(None, paths):
|
||||
path = case / name
|
||||
if path.is_file() and not path.is_symlink():
|
||||
data = path.read_bytes()
|
||||
instrument["checkout_files"] += 1
|
||||
instrument["checkout_bytes"] += len(data)
|
||||
if instrument["checkout_bytes"] > 500 * 1024 * 1024:
|
||||
raise ReplayFailure("checkout size limit")
|
||||
step = "changed files"
|
||||
changed = run([*git, "diff", "--name-only", "--diff-filter=ACMRT", "-z", spec["base"], spec["head"]]).stdout.split("\0")
|
||||
for name in filter(None, changed):
|
||||
path = case / name
|
||||
if path.is_file() and not path.is_symlink() and path.resolve().is_relative_to(case.resolve()):
|
||||
instrument["changed_bytes"][name] = len(path.read_bytes())
|
||||
if not instrument["checkout_bytes"] or not sum(instrument["changed_bytes"].values()):
|
||||
instrument["suspect_fast"] = True
|
||||
raise ReplayFailure("no changed bytes read")
|
||||
step = "pre-push"
|
||||
env = {
|
||||
key: value for key, value in os.environ.items() if key not in {"SPEC_URL", "RESULT_URL", "_JAVA_OPTIONS", "JDK_JAVA_OPTIONS"}
|
||||
}
|
||||
env["OPENULTRASAST_CPG_HEAP_MB"] = "1700" if heap_profile == "cluster" else "2560"
|
||||
env["JAVA_TOOL_OPTIONS"] = CLUSTER_FLAGS if heap_profile == "cluster" else "-Xmx2560m"
|
||||
command = [
|
||||
"ousast",
|
||||
"pre-push",
|
||||
str(case),
|
||||
"--base",
|
||||
spec["base"],
|
||||
"--head",
|
||||
spec["head"],
|
||||
"--artifact",
|
||||
str(output / "push.json"),
|
||||
*spec.get("hook_flags", []),
|
||||
]
|
||||
before = clock()
|
||||
completed = runner(command, capture_output=True, text=True, timeout=330 if spec.get("hook_flags") else 60, env=env)
|
||||
instrument["timings"][step] = clock() - before
|
||||
instrument["hook_exit"] = completed.returncode
|
||||
(output / "push.txt").write_text(completed.stdout)
|
||||
if completed.returncode not in (0, 1):
|
||||
raise ReplayFailure(f"hook exit {completed.returncode}")
|
||||
step = "read push artifact"
|
||||
push = json.loads((output / "push.json").read_text())
|
||||
instrument["hook_bytes_read"] = sum(t.get("bytes_read", 0) for t in push.get("quick_tier", []))
|
||||
scans = push.get("scans", [])
|
||||
instrument["jvm_wall_seconds"] = sum(s.get("scan", {}).get(k, 0) for s in scans for k in ("build_seconds", "query_seconds"))
|
||||
uncovered = "language_not_covered" in json.dumps(push.get("admission", {}).get("coverage_reasons", []))
|
||||
partitions = [part for scan in scans for part in scan.get("scan", {}).get("partitions", [])]
|
||||
unsupported_only = (
|
||||
uncovered
|
||||
and instrument["hook_bytes_read"] == 0
|
||||
and bool(partitions)
|
||||
and all(part.get("status") == "unsupported" for part in partitions)
|
||||
)
|
||||
instrument["suspect_fast"] = bool(scans) and not unsupported_only and instrument["jvm_wall_seconds"] < 5
|
||||
if instrument["suspect_fast"] or (instrument["hook_bytes_read"] <= 0 and not uncovered):
|
||||
instrument["suspect_fast"] = True
|
||||
raise ReplayFailure("suspect_fast or unread hook input")
|
||||
engine = {
|
||||
"repository": spec["repository_url"],
|
||||
"revisions": {key: spec[key] for key in ("base", "head")},
|
||||
"scans": scans,
|
||||
"admission": push.get("admission", {}),
|
||||
}
|
||||
(output / "engine.json").write_text(json.dumps(engine))
|
||||
instrument["status"] = "ok"
|
||||
except Exception as exc:
|
||||
detail = (
|
||||
f"HTTP {exc.code}"
|
||||
if isinstance(exc, urllib.error.HTTPError)
|
||||
else str(exc)
|
||||
if isinstance(exc, ReplayFailure)
|
||||
else type(exc).__name__
|
||||
)
|
||||
instrument["failure"] = {"step": step, "status": detail}
|
||||
# A crash artifact must never be mistaken for a completed scan.
|
||||
if not (output / "push.json").exists():
|
||||
(output / "push.json").write_text("{}")
|
||||
if not (output / "push.txt").exists():
|
||||
(output / "push.txt").write_text("")
|
||||
if not (output / "engine.json").exists():
|
||||
(output / "engine.json").write_text("{}")
|
||||
instrument["seconds"] = clock() - started
|
||||
(output / "instrument.json").write_text(json.dumps(instrument))
|
||||
try:
|
||||
archive = work_root / "result.dat"
|
||||
with tarfile.open(archive, "w") as tar:
|
||||
for name in ("push.json", "push.txt", "engine.json", "instrument.json"):
|
||||
tar.add(output / name, arcname=name)
|
||||
request = urllib.request.Request(
|
||||
os.environ["RESULT_URL"], data=archive.read_bytes(), method="PUT", headers={"Content-Type": "application/x-tar"}
|
||||
)
|
||||
transfer(request, readiness_budget=60)
|
||||
except Exception as exc:
|
||||
detail = f"HTTP {exc.code}" if isinstance(exc, urllib.error.HTTPError) else type(exc).__name__
|
||||
print("PUT result: " + detail, file=sys.stderr)
|
||||
return 1
|
||||
return int(instrument["status"] != "ok")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--heap-profile", choices=("cluster", "vm"), default="cluster")
|
||||
raise SystemExit(main(heap_profile=parser.parse_args().heap_profile))
|
||||
@@ -0,0 +1,308 @@
|
||||
"""Push-level unseen evaluation, explicit coverage and paired adoption evidence."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import random
|
||||
from collections import Counter, defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
from openultrasast.model.taxonomy import load_families
|
||||
from openultrasast.push.report import CAPABILITY_VETOES
|
||||
|
||||
STATES = ("instrument_failure", "unanalysable", "quick_only", "engine_covered")
|
||||
|
||||
|
||||
def coverage(record):
|
||||
instrument = record.get("instrument", {})
|
||||
push = record.get("push")
|
||||
if (
|
||||
not isinstance(push, dict)
|
||||
or not instrument.get("head_verified")
|
||||
or not instrument.get("base_verified")
|
||||
or instrument.get("suspect_fast")
|
||||
or instrument.get("failure")
|
||||
or instrument.get("hook_exit") not in (0, 1)
|
||||
or sum(instrument.get("changed_bytes", {}).values()) <= 0
|
||||
):
|
||||
return "instrument_failure"
|
||||
if "language_not_covered" in json.dumps([push.get("skipped", []), push.get("admission", {}).get("coverage_reasons", [])]) and not any(
|
||||
tier.get("bytes_read", 0) for tier in push.get("quick_tier", [])
|
||||
):
|
||||
return "unanalysable"
|
||||
if instrument.get("hook_bytes_read", 0) <= 0:
|
||||
return "instrument_failure"
|
||||
changed = set(instrument["changed_bytes"])
|
||||
if any(
|
||||
outcome.get("status") == "completed" and outcome.get("identity", {}).get("path") in changed
|
||||
for scan in push.get("scans", [])
|
||||
if scan.get("side") == "head"
|
||||
for outcome in scan.get("scan", {}).get("question_outcomes", [])
|
||||
):
|
||||
return "engine_covered"
|
||||
return "quick_only"
|
||||
|
||||
|
||||
def findings(record):
|
||||
push = record.get("push", {})
|
||||
admission = push.get("admission", {})
|
||||
for defect in admission.get("defects", []):
|
||||
for path, line in defect.get("locations", []):
|
||||
yield "alerts", {**defect, "path": path, "line": line}
|
||||
seen = set()
|
||||
for disposition in admission.get("dispositions", []):
|
||||
delta = disposition.get("candidate", {}).get("delta", {})
|
||||
operation = delta.get("head_operation")
|
||||
if (
|
||||
disposition.get("admitted")
|
||||
or delta.get("novelty") not in {"new", "worsened"}
|
||||
or not set(disposition.get("reasons", [])) <= CAPABILITY_VETOES
|
||||
or operation is None
|
||||
or delta.get("witness") is None
|
||||
or delta.get("defect_id") in seen
|
||||
):
|
||||
continue
|
||||
seen.add(delta.get("defect_id"))
|
||||
yield "advisory", {**operation, "family": delta.get("family")}
|
||||
taxonomy = load_families()
|
||||
for tier in push.get("quick_tier", []):
|
||||
for finding in tier.get("findings", []):
|
||||
family = taxonomy.family_of_cwe(finding.get("cwe") or "")
|
||||
yield "quick", {**finding, "family": family.id if family else None}
|
||||
|
||||
|
||||
def location_match(record, finding):
|
||||
for file in record.get("files", []):
|
||||
if finding.get("path") != file["path"]:
|
||||
continue
|
||||
line = finding.get("line", 0)
|
||||
if any(start - 10 <= line <= end + 10 for start, end in file.get("head_lines", [])):
|
||||
return True
|
||||
if record.get("family") != "config_secrets" and file["path"] == record.get("path", file["path"]):
|
||||
function = record.get("function")
|
||||
if function and function != "<global>" and finding.get("function") == function:
|
||||
return True
|
||||
span = record.get("function_lines") or file.get("function_lines")
|
||||
if span and span[0] <= line <= span[1]:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def catches(record):
|
||||
return sorted(
|
||||
{
|
||||
stream
|
||||
for stream, finding in findings(record)
|
||||
if finding.get("family") == record.get("family") and location_match(record, finding)
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def false_alarm(record, *, alerts_only=False):
|
||||
return any(not alerts_only or stream == "alerts" for stream, _ in findings(record))
|
||||
|
||||
|
||||
def wilson(successes, n):
|
||||
if n == 0:
|
||||
return [None, None]
|
||||
p, z = successes / n, 1.96
|
||||
centre = (p + z * z / (2 * n)) / (1 + z * z / n)
|
||||
half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
|
||||
return [max(0, centre - half), min(1, centre + half)]
|
||||
|
||||
|
||||
def required_n(p, width):
|
||||
for n in range(1, 1000000):
|
||||
low, high = wilson(p * n, n)
|
||||
if max(p - low, high - p) <= width:
|
||||
return n
|
||||
raise ValueError("requested precision exceeds sizing range")
|
||||
|
||||
|
||||
def rate(values):
|
||||
n, k = len(values), sum(values)
|
||||
return {"n": n, "count": k, "rate": k / n if n else None, "wilson95": wilson(k, n)}
|
||||
|
||||
|
||||
def cluster_interval(rows, *, seed=0, draws=2000):
|
||||
clusters = defaultdict(list)
|
||||
for repo, value in rows:
|
||||
clusters[repo].append(value)
|
||||
if not clusters:
|
||||
return [None, None]
|
||||
rng = random.Random(seed)
|
||||
groups = list(clusters.values())
|
||||
samples = []
|
||||
for _ in range(draws):
|
||||
values = [value for group in rng.choices(groups, k=len(groups)) for value in group]
|
||||
samples.append(sum(values) / len(values))
|
||||
samples.sort()
|
||||
return [samples[int(0.025 * (draws - 1))], samples[int(0.975 * (draws - 1))]]
|
||||
|
||||
|
||||
def icc(rows):
|
||||
clusters = defaultdict(list)
|
||||
for repo, value in rows:
|
||||
clusters[repo].append(value)
|
||||
n, k = len(rows), len(clusters)
|
||||
if k < 2 or n <= k:
|
||||
return None
|
||||
mean = sum(v for _, v in rows) / n
|
||||
between = sum(len(v) * (sum(v) / len(v) - mean) ** 2 for v in clusters.values()) / (k - 1)
|
||||
within = sum(sum((x - sum(v) / len(v)) ** 2 for x in v) for v in clusters.values()) / (n - k)
|
||||
size = (n - sum(len(v) ** 2 for v in clusters.values()) / n) / (k - 1)
|
||||
denominator = between + (size - 1) * within
|
||||
return (between - within) / denominator if denominator else 0.0
|
||||
|
||||
|
||||
def mcnemar(a, b):
|
||||
pairs = list(zip(a, b, strict=True))
|
||||
losses = sum(x and not y for x, y in pairs)
|
||||
gains = sum(y and not x for x, y in pairs)
|
||||
n = losses + gains
|
||||
return min(1.0, 2 * sum(math.comb(n, i) for i in range(min(losses, gains) + 1)) / 2**n) if n else 1.0
|
||||
|
||||
|
||||
def score(records, *, seed=0):
|
||||
counts = Counter(coverage(r) for r in records)
|
||||
if not records or counts["instrument_failure"] / len(records) > 0.05:
|
||||
raise ValueError("slice exceeds 5% instrument_failure (or is empty)")
|
||||
usable = [r for r in records if coverage(r) != "instrument_failure"]
|
||||
|
||||
def metrics(rows):
|
||||
ordinary = [r for r in rows if r["kind"] == "ordinary"]
|
||||
vulnerable = [r for r in rows if r["kind"] != "ordinary"]
|
||||
clustered = [(r["repo"], int(false_alarm(r))) for r in ordinary]
|
||||
return {
|
||||
"catch": rate([bool(catches(r)) for r in vulnerable]),
|
||||
"false_alarm": {
|
||||
**rate([false_alarm(r) for r in ordinary]),
|
||||
"cluster95": cluster_interval(clustered, seed=seed),
|
||||
"icc": icc(clustered),
|
||||
},
|
||||
"alerts_only": rate([false_alarm(r, alerts_only=True) for r in ordinary]),
|
||||
}
|
||||
|
||||
breakdown = {}
|
||||
for key in ("family", "method", "post_cutoff"):
|
||||
groups = defaultdict(list)
|
||||
for r in usable:
|
||||
groups[str(r.get(key, "unknown"))].append(r)
|
||||
breakdown[key] = {key: metrics(rows) for key, rows in sorted(groups.items())}
|
||||
breakdown["coverage"] = {state: metrics([r for r in usable if coverage(r) == state]) for state in STATES[1:]}
|
||||
breakdown["stream"] = {
|
||||
stream: rate([stream in catches(r) for r in usable if r["kind"] != "ordinary"]) for stream in ("alerts", "advisory", "quick")
|
||||
}
|
||||
return {
|
||||
**metrics(usable),
|
||||
"total": len(records),
|
||||
"coverage": {state: counts[state] for state in STATES},
|
||||
"breakdowns": breakdown,
|
||||
"confirmed": metrics([r for r in usable if r.get("label_check", r.get("confirmed")) in ("confirmed", True)]),
|
||||
}
|
||||
|
||||
|
||||
def budgets(rows, *, thresholds=None, binary=False):
|
||||
ordinary = [r for r in rows if r["kind"] == "ordinary"]
|
||||
vulnerable = [r for r in rows if r["kind"] != "ordinary"]
|
||||
if binary:
|
||||
fa = rate([r["prediction"] for r in ordinary])
|
||||
return {
|
||||
"catch": rate([r["prediction"] for r in vulnerable]),
|
||||
"false_alarm": fa,
|
||||
"budgets_met": [f"{budget}%" for budget in (1, 5, 10) if fa["rate"] is not None and fa["rate"] <= budget / 100],
|
||||
}
|
||||
output = {}
|
||||
for budget in (1, 5, 10):
|
||||
key = f"{budget}%"
|
||||
if thresholds is not None and key not in thresholds:
|
||||
continue
|
||||
if not ordinary:
|
||||
output[key] = {"label": "not_run"}
|
||||
continue
|
||||
if thresholds is not None:
|
||||
threshold = thresholds[key]
|
||||
else:
|
||||
candidates = sorted({r["score"] for r in rows})
|
||||
candidates.append(math.nextafter(candidates[-1], math.inf))
|
||||
threshold = next(t for t in candidates if sum(r["score"] >= t for r in ordinary) / len(ordinary) <= budget / 100)
|
||||
output[key] = {
|
||||
"threshold": threshold,
|
||||
"label": "fixed" if thresholds is not None else "exploratory",
|
||||
"catch": rate([r["score"] >= threshold for r in vulnerable]),
|
||||
"false_alarm": rate([r["score"] >= threshold for r in ordinary]),
|
||||
}
|
||||
return output
|
||||
|
||||
|
||||
def gate(a, b, *, slice_id=None, informed=False, binary=True, threshold=None, threshold_fixed=False, seed=0):
|
||||
not_run = {"gate": "not_run"}
|
||||
if not slice_id or informed or not a or not b or (not binary and (not threshold_fixed or threshold is None)):
|
||||
return not_run
|
||||
try:
|
||||
score(a, seed=seed)
|
||||
score(b, seed=seed)
|
||||
except ValueError:
|
||||
return not_run
|
||||
aa, bb = {r["id"]: r for r in a}, {r["id"]: r for r in b}
|
||||
if len(aa) != len(a) or len(bb) != len(b) or aa.keys() != bb.keys():
|
||||
return not_run
|
||||
pairs = [
|
||||
(aa[key], bb[key]) for key in sorted(aa) if coverage(aa[key]) != "instrument_failure" and coverage(bb[key]) != "instrument_failure"
|
||||
]
|
||||
if any(x["kind"] != y["kind"] or x["repo"] != y["repo"] for x, y in pairs):
|
||||
return not_run
|
||||
vulnerable = [(x, y) for x, y in pairs if x["kind"] != "ordinary"]
|
||||
ordinary = [(x, y) for x, y in pairs if x["kind"] == "ordinary"]
|
||||
if not vulnerable or not ordinary:
|
||||
return not_run
|
||||
ca = [bool(catches(x)) for x, _ in vulnerable]
|
||||
cb = [bool(catches(y)) if binary else y["score"] >= threshold for _, y in vulnerable]
|
||||
fa = [false_alarm(x) for x, _ in ordinary]
|
||||
fb = [false_alarm(y) if binary else y["score"] >= threshold for _, y in ordinary]
|
||||
catch_p, fa_p = mcnemar(ca, cb), mcnemar(fa, fb)
|
||||
regression = (sum(cb) < sum(ca) and catch_p < 0.05) or (binary and sum(fb) > sum(fa) and fa_p < 0.05)
|
||||
return {
|
||||
"gate": "regression" if regression else "pass",
|
||||
"slice": slice_id,
|
||||
"catch": {"difference": sum(cb) / len(cb) - sum(ca) / len(ca), "mcnemar_p": catch_p},
|
||||
"false_alarm": {
|
||||
"difference": sum(fb) / len(fb) - sum(fa) / len(fa),
|
||||
"mcnemar_p": fa_p,
|
||||
"cluster95": cluster_interval([(x["repo"], int(v) - int(u)) for (x, _), u, v in zip(ordinary, fa, fb, strict=True)], seed=seed),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def load_records(path):
|
||||
if path.is_file():
|
||||
return json.loads(path.read_text())
|
||||
rows = []
|
||||
for entry in json.loads((path / "inputs.json").read_text()):
|
||||
result = path / (hashlib.sha256(entry["item_id"].encode()).hexdigest() + ".json")
|
||||
record = json.loads(result.read_text()) if result.exists() else {}
|
||||
if record.get("input_digest") != entry["input_digest"]:
|
||||
record = {}
|
||||
rows.append({**entry["change"], **record, "id": entry["change"]["id"]})
|
||||
return rows
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("records", type=Path, help="JSON array of joined change/instrument/push records")
|
||||
parser.add_argument("--out", type=Path, required=True)
|
||||
parser.add_argument("--against", type=Path, help="production-arm records for a paired adoption gate")
|
||||
parser.add_argument("--slice-id")
|
||||
parser.add_argument("--informed", action="store_true")
|
||||
args = parser.parse_args()
|
||||
current = load_records(args.records)
|
||||
result = score(current)
|
||||
result["unseen"] = gate(load_records(args.against) if args.against else [], current, slice_id=args.slice_id, informed=args.informed)
|
||||
args.out.write_text(json.dumps(result, sort_keys=True, indent=2) + "\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,305 @@
|
||||
"""Advisory selection. Network adapters are constructed only by the coordinator CLI."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
import random
|
||||
import re
|
||||
from collections import OrderedDict
|
||||
from dataclasses import dataclass
|
||||
from datetime import date
|
||||
from urllib.parse import urlencode, urlsplit
|
||||
|
||||
from openultrasast.model.taxonomy import load_families
|
||||
|
||||
from .eligibility import RepositoryNotFound, normalize
|
||||
|
||||
ECOSYSTEMS = ("pip", "npm", "maven", "composer")
|
||||
OSV_ECOSYSTEMS = dict(zip(ECOSYSTEMS, ("PyPI", "npm", "Maven", "Packagist"), strict=True))
|
||||
FAMILIES = {"injection", "path", "deserialization", "access_control", "output_encoding", "untrusted_destination", "config_secrets"}
|
||||
PERMISSIVE = {"MIT", "Apache-2.0", "BSD-2-Clause", "BSD-3-Clause", "ISC", "0BSD", "Unlicense", "Zlib", "PostgreSQL", "BSL-1.0"}
|
||||
FIX = re.compile(r"https://github\.com/([\w.-]+/[\w.-]+)/commit/([a-fA-F0-9]{7,40})(?:[/?#].*)?$", re.I)
|
||||
|
||||
|
||||
class Rejected(ValueError):
|
||||
"""Expected eligibility rejection; messages are fixed reason codes, never names."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Candidate:
|
||||
repository: str
|
||||
url: str
|
||||
fix: str
|
||||
parent: str
|
||||
advisory: str
|
||||
ecosystem: str
|
||||
family: str
|
||||
post_cutoff: bool
|
||||
first_affected: str | None
|
||||
license_spdx: str
|
||||
license_class: str
|
||||
license_ref: str
|
||||
license_path: str
|
||||
|
||||
|
||||
def compact_advisory(row: dict) -> dict:
|
||||
"""Keep exactly the input fields used by selection and the all-advisory fix index."""
|
||||
return {
|
||||
**{key: row[key] for key in ("ghsa_id", "published_at", "type", "withdrawn_at") if key in row},
|
||||
"cwes": [{"cwe_id": c["cwe_id"]} for c in row.get("cwes", [])],
|
||||
"references": [f"https://github.com/{name}/commit/{sha}" for name, sha in sorted(fix_links(row))],
|
||||
"vulnerabilities": [{"package": {key: v["package"][key] for key in ("ecosystem", "name")}} for v in row.get("vulnerabilities", [])],
|
||||
}
|
||||
|
||||
|
||||
def compact_repository(row: dict) -> dict:
|
||||
return {
|
||||
**{key: row[key] for key in ("full_name", "private", "size", "clone_url", "former_names") if key in row},
|
||||
**{key: {"full_name": row[key]["full_name"]} for key in ("parent", "source") if row.get(key)},
|
||||
}
|
||||
|
||||
|
||||
def post_cutoff(row: dict) -> bool:
|
||||
return date.fromisoformat(row["published_at"][:10]) > date(2026, 6, 1)
|
||||
|
||||
|
||||
def fix_links(row: dict) -> set[tuple[str, str]]:
|
||||
links = set()
|
||||
for reference in row.get("references", []):
|
||||
url = reference if isinstance(reference, str) else reference.get("url", "")
|
||||
match = FIX.fullmatch(url)
|
||||
if match:
|
||||
links.add((normalize(match[1]), match[2].lower()))
|
||||
return links
|
||||
|
||||
|
||||
def ecosystem(row: dict) -> str | None:
|
||||
values = {v["package"]["ecosystem"].lower() for v in row.get("vulnerabilities", [])}
|
||||
return next((e for e in ECOSYSTEMS if e in values), None)
|
||||
|
||||
|
||||
def ordered_advisories(rows: list[dict], seed: int) -> list[dict]:
|
||||
"""Stable seeded ordering, post-cutoff first in each ecosystem; round-robin strata."""
|
||||
from .storage import AdvisoryRows
|
||||
|
||||
if isinstance(rows, AdvisoryRows):
|
||||
return rows.ordered(seed)
|
||||
rng = random.Random(seed)
|
||||
groups = []
|
||||
for eco in ECOSYSTEMS:
|
||||
group = sorted((r for r in rows if ecosystem(r) == eco), key=lambda r: r["ghsa_id"])
|
||||
rng.shuffle(group)
|
||||
group.sort(key=lambda r: not post_cutoff(r))
|
||||
groups.append(group)
|
||||
return [group[i] for i in range(max(map(len, groups), default=0)) for group in groups if i < len(group)]
|
||||
|
||||
|
||||
def first_affected(record: dict, row: dict, eco: str) -> str | None:
|
||||
packages = {v["package"]["name"] for v in row["vulnerabilities"] if v["package"]["ecosystem"].lower() == eco}
|
||||
versions = set()
|
||||
for affected in record.get("affected", []):
|
||||
package = affected.get("package", {})
|
||||
if package.get("ecosystem") != OSV_ECOSYSTEMS[eco] or package.get("name") not in packages:
|
||||
continue
|
||||
for span in affected.get("ranges", []):
|
||||
if span.get("type") not in {"ECOSYSTEM", "SEMVER"}:
|
||||
continue
|
||||
for event in span.get("events", []):
|
||||
if event.get("introduced") not in {None, "0"}:
|
||||
versions.add(event["introduced"])
|
||||
if len(versions) > 1:
|
||||
raise Rejected("ambiguous_version_bracket")
|
||||
return next(iter(versions), None)
|
||||
|
||||
|
||||
def license_info(record: dict) -> tuple[str, str]:
|
||||
# Explicit pinned-file SPDX wins. For GitHub-recognized full texts its license endpoint
|
||||
# provides the detected SPDX. Unknown/custom/compound declarations stay private.
|
||||
declared = re.search(r"SPDX-License-Identifier:\s*([^\r\n]+)", record.get("text", ""))
|
||||
spdx = declared[1].strip().removesuffix("*/").strip() if declared else record.get("spdx_id") or "NONE"
|
||||
return spdx, "permissive" if spdx in PERMISSIVE else "private"
|
||||
|
||||
|
||||
def candidate(row: dict, github, osv) -> Candidate:
|
||||
if date.fromisoformat(row["published_at"][:10]) < date(2019, 1, 1):
|
||||
raise Rejected("date")
|
||||
if row.get("type", "reviewed") != "reviewed" or row.get("withdrawn_at"):
|
||||
raise Rejected("advisory_status")
|
||||
eco = ecosystem(row)
|
||||
if eco is None:
|
||||
raise Rejected("ecosystem")
|
||||
taxonomy = load_families()
|
||||
families = {f.id for c in row.get("cwes", []) if (f := taxonomy.family_of_cwe(c["cwe_id"])) and f.id in FAMILIES}
|
||||
if len(families) != 1:
|
||||
raise Rejected("family")
|
||||
links = fix_links(row)
|
||||
if len(links) != 1:
|
||||
raise Rejected("fix_links")
|
||||
name, fix = next(iter(links))
|
||||
repo = github.get_repo(name)
|
||||
url = urlsplit(repo.get("clone_url", ""))
|
||||
if (
|
||||
repo.get("private", True)
|
||||
or not 0 <= repo.get("size", -1) <= 500 * 1024
|
||||
or url.scheme != "https"
|
||||
or url.hostname != "github.com"
|
||||
or url.username
|
||||
or url.password
|
||||
or url.query
|
||||
or url.fragment
|
||||
or normalize(repo["clone_url"]) != normalize(repo["full_name"])
|
||||
):
|
||||
raise Rejected("repository")
|
||||
commit = github.get_commit(repo["full_name"], fix)
|
||||
if len(commit["parents"]) != 1:
|
||||
raise Rejected("parents")
|
||||
parent = commit["parents"][0]["sha"]
|
||||
license_record = github.get_license(repo["full_name"], parent)
|
||||
spdx, license_class = license_info(license_record)
|
||||
return Candidate(
|
||||
normalize(repo["full_name"]),
|
||||
repo["clone_url"],
|
||||
commit["sha"],
|
||||
parent,
|
||||
row["ghsa_id"],
|
||||
eco,
|
||||
next(iter(families)),
|
||||
post_cutoff(row),
|
||||
first_affected(osv.get(row["ghsa_id"]), row, eco),
|
||||
spdx,
|
||||
license_class,
|
||||
parent,
|
||||
license_record.get("path", ""),
|
||||
)
|
||||
|
||||
|
||||
class JSONTransport:
|
||||
"""Coordinator GET transport with bounded retries; all I/O and waits are injectable."""
|
||||
|
||||
def __init__(self, *, opener=None, sleeper=None, clock=None):
|
||||
import time
|
||||
from urllib.request import urlopen
|
||||
|
||||
self.opener = opener or urlopen
|
||||
self.sleeper = sleeper or time.sleep
|
||||
self.clock = clock or time.time
|
||||
self.attempts = 0
|
||||
self.rate_limit = {}
|
||||
|
||||
def get(self, url: str, headers: dict) -> object:
|
||||
from urllib.error import HTTPError
|
||||
from urllib.request import Request
|
||||
|
||||
for attempt in range(5):
|
||||
self.attempts += 1
|
||||
try:
|
||||
with self.opener(Request(url, headers=headers), timeout=60) as response:
|
||||
self.record_rate(getattr(response, "headers", {}))
|
||||
return json.load(response)
|
||||
except HTTPError as exc:
|
||||
self.record_rate(exc.headers)
|
||||
if exc.code == 404:
|
||||
raise RepositoryNotFound("not_found") from None
|
||||
limited = exc.code == 429 or (
|
||||
exc.code == 403 and (exc.headers.get("X-RateLimit-Remaining") == "0" or exc.headers.get("Retry-After"))
|
||||
)
|
||||
transient = exc.code in {500, 502, 503, 504}
|
||||
if attempt == 4 or not (limited or transient):
|
||||
raise ValueError("http_failure") from None
|
||||
try:
|
||||
delay = float(exc.headers.get("Retry-After", "0"))
|
||||
if limited and not delay:
|
||||
delay = max(60, float(exc.headers.get("X-RateLimit-Reset", "0")) - self.clock() + 1)
|
||||
delay = max(delay, 2**attempt)
|
||||
if delay > 3605:
|
||||
raise ValueError("retry_window")
|
||||
except (TypeError, ValueError):
|
||||
raise ValueError("http_retry_failed") from None
|
||||
self.sleeper(delay)
|
||||
except Exception:
|
||||
raise ValueError("http_failure") from None
|
||||
raise ValueError("http_failure")
|
||||
|
||||
def record_rate(self, headers):
|
||||
self.rate_limit = {
|
||||
key: headers.get(header)
|
||||
for key, header in (
|
||||
("remaining", "X-RateLimit-Remaining"),
|
||||
("limit", "X-RateLimit-Limit"),
|
||||
("reset", "X-RateLimit-Reset"),
|
||||
("retry_after", "Retry-After"),
|
||||
)
|
||||
if headers.get(header) is not None
|
||||
}
|
||||
|
||||
|
||||
class GitHub:
|
||||
def __init__(self, token: str, transport=None):
|
||||
if not token:
|
||||
raise ValueError("missing_gh_token")
|
||||
self.transport = transport or JSONTransport()
|
||||
self.headers = {"Authorization": "Bearer " + token, "Accept": "application/vnd.github+json"}
|
||||
self.repos: OrderedDict[str, dict] = OrderedDict()
|
||||
self.calls = 0
|
||||
|
||||
def get(self, path: str):
|
||||
self.calls += 1
|
||||
return self.transport.get("https://api.github.com" + path, self.headers)
|
||||
|
||||
def get_repo(self, name: str) -> dict:
|
||||
name = normalize(name)
|
||||
if name not in self.repos:
|
||||
self.repos[name] = compact_repository(self.get("/repos/" + name))
|
||||
if len(self.repos) > 128:
|
||||
self.repos.popitem(last=False)
|
||||
self.repos.move_to_end(name)
|
||||
return self.repos[name]
|
||||
|
||||
def get_commit(self, name: str, sha: str) -> dict:
|
||||
return self.get("/repos/" + normalize(name) + "/commits/" + sha)
|
||||
|
||||
def get_license(self, name: str, ref: str) -> dict:
|
||||
try:
|
||||
row = self.get("/repos/" + normalize(name) + "/license?" + urlencode({"ref": ref}))
|
||||
except RepositoryNotFound:
|
||||
return {"spdx_id": "NONE", "text": "", "path": ""}
|
||||
return {
|
||||
"spdx_id": (row.get("license") or {}).get("spdx_id"),
|
||||
"path": row.get("path", ""),
|
||||
"text": base64.b64decode(row.get("content", "")).decode("utf-8", errors="replace"),
|
||||
}
|
||||
|
||||
def advisory_page(self, page: int) -> dict:
|
||||
# Oldest first: newly published advisories do not shift already persisted pages.
|
||||
rows = self.get(
|
||||
"/advisories?" + urlencode({"type": "reviewed", "per_page": 100, "page": page, "sort": "published", "direction": "asc"})
|
||||
)
|
||||
if not isinstance(rows, list):
|
||||
raise ValueError("invalid_advisory_page")
|
||||
return {"rows": rows, "done": len(rows) < 100}
|
||||
|
||||
def advisories(self):
|
||||
# Enumerate every reviewed advisory, including non-candidate fixes.
|
||||
page = 1
|
||||
while True:
|
||||
result = self.advisory_page(page)
|
||||
yield from result["rows"]
|
||||
if result["done"]:
|
||||
break
|
||||
page += 1
|
||||
|
||||
|
||||
class OSV:
|
||||
def __init__(self, transport=None):
|
||||
self.transport = transport or JSONTransport()
|
||||
self.calls = 0
|
||||
|
||||
def get(self, identifier: str) -> dict:
|
||||
if not re.fullmatch(r"GHSA-[\w-]+", identifier):
|
||||
raise ValueError("invalid_advisory_id")
|
||||
self.calls += 1
|
||||
try:
|
||||
return self.transport.get("https://api.osv.dev/v1/vulns/" + identifier, {})
|
||||
except RepositoryNotFound:
|
||||
return {} # No OSV bracket is distinct from a failed service call.
|
||||
@@ -0,0 +1,170 @@
|
||||
"""Disposable private SQLite working sets; JSONL remains the durable source of truth."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sqlite3
|
||||
import tempfile
|
||||
from collections.abc import Mapping, Sequence, Set
|
||||
from contextlib import contextmanager
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
@contextmanager
|
||||
def working_store(directory=None):
|
||||
# TemporaryDirectory is mode 0700. No identity-bearing scratch data enters the checkout.
|
||||
with tempfile.TemporaryDirectory(prefix="sourcing-", dir=directory) as temporary:
|
||||
db = sqlite3.connect(Path(temporary) / "working.sqlite")
|
||||
try:
|
||||
db.execute("PRAGMA journal_mode=OFF")
|
||||
db.execute("PRAGMA synchronous=OFF")
|
||||
db.execute("PRAGMA temp_store=FILE")
|
||||
db.execute("PRAGMA cache_size=-2048")
|
||||
db.execute("PRAGMA mmap_size=0")
|
||||
db.execute("CREATE TABLE records (kind TEXT, key TEXT, value TEXT, PRIMARY KEY (kind, key)) WITHOUT ROWID")
|
||||
db.execute("CREATE TABLE sequences (kind TEXT, position INTEGER, value TEXT, PRIMARY KEY (kind, position)) WITHOUT ROWID")
|
||||
yield db
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
class DiskMap(Mapping):
|
||||
def __init__(self, db, kind):
|
||||
self.db, self.kind = db, kind
|
||||
self.count = 0
|
||||
|
||||
def __len__(self):
|
||||
return self.count
|
||||
|
||||
def __iter__(self):
|
||||
for (key,) in self.db.execute("SELECT key FROM records WHERE kind=? ORDER BY key", (self.kind,)):
|
||||
yield key
|
||||
|
||||
def __contains__(self, key):
|
||||
return self.db.execute("SELECT 1 FROM records WHERE kind=? AND key=?", (self.kind, key)).fetchone() is not None
|
||||
|
||||
def __getitem__(self, key):
|
||||
row = self.db.execute("SELECT value FROM records WHERE kind=? AND key=?", (self.kind, key)).fetchone()
|
||||
if row is None:
|
||||
raise KeyError(key)
|
||||
return json.loads(row[0])
|
||||
|
||||
def __setitem__(self, key, value):
|
||||
if key not in self:
|
||||
self.count += 1
|
||||
self.db.execute("INSERT OR REPLACE INTO records VALUES (?, ?, ?)", (self.kind, key, json.dumps(value)))
|
||||
|
||||
def values(self):
|
||||
for (value,) in self.db.execute("SELECT value FROM records WHERE kind=? ORDER BY key", (self.kind,)):
|
||||
yield json.loads(value)
|
||||
|
||||
|
||||
class DiskSequence(Sequence):
|
||||
def __init__(self, db, kind):
|
||||
self.db, self.kind = db, kind
|
||||
self.count = 0
|
||||
|
||||
def __len__(self):
|
||||
return self.count
|
||||
|
||||
def __getitem__(self, position):
|
||||
if position < 0:
|
||||
position += self.count
|
||||
row = self.db.execute("SELECT value FROM sequences WHERE kind=? AND position=?", (self.kind, position)).fetchone()
|
||||
if row is None:
|
||||
raise IndexError(position)
|
||||
return json.loads(row[0])
|
||||
|
||||
def __setitem__(self, position, value):
|
||||
self.db.execute("INSERT OR REPLACE INTO sequences VALUES (?, ?, ?)", (self.kind, position, json.dumps(value)))
|
||||
|
||||
def append(self, value):
|
||||
self[self.count] = value
|
||||
self.count += 1
|
||||
|
||||
def __iter__(self):
|
||||
for (value,) in self.db.execute("SELECT value FROM sequences WHERE kind=? ORDER BY position", (self.kind,)):
|
||||
yield json.loads(value)
|
||||
|
||||
|
||||
class DiskSet(Set):
|
||||
def __init__(self, db, kind):
|
||||
self.rows = DiskMap(db, kind)
|
||||
|
||||
def __len__(self):
|
||||
return len(self.rows)
|
||||
|
||||
def __contains__(self, value):
|
||||
return json.dumps(value) in self.rows
|
||||
|
||||
def __iter__(self):
|
||||
for key in self.rows:
|
||||
value = json.loads(key)
|
||||
yield tuple(value) if isinstance(value, list) else value
|
||||
|
||||
def add(self, value):
|
||||
self.rows[json.dumps(value)] = None
|
||||
|
||||
def update(self, values):
|
||||
for value in values:
|
||||
self.add(value)
|
||||
|
||||
|
||||
class AdvisoryRows(DiskMap):
|
||||
def __iter__(self):
|
||||
return self.values()
|
||||
|
||||
def ordered(self, seed):
|
||||
import random
|
||||
|
||||
from .source import ECOSYSTEMS, ecosystem
|
||||
|
||||
rng = random.Random(seed)
|
||||
groups = [DiskSequence(self.db, "order_" + eco) for eco in ECOSYSTEMS]
|
||||
for row in self.values(): # key order is the original ghsa_id sort
|
||||
eco = ecosystem(row)
|
||||
if eco in ECOSYSTEMS:
|
||||
groups[ECOSYSTEMS.index(eco)].append(row)
|
||||
for group in groups:
|
||||
rng.shuffle(group) # Same Fisher-Yates draws as the in-memory implementation.
|
||||
cursors = [
|
||||
self.db.execute(
|
||||
"SELECT value FROM sequences WHERE kind=? "
|
||||
"ORDER BY substr(json_extract(value, '$.published_at'), 1, 10) <= '2026-06-01', position",
|
||||
(group.kind,),
|
||||
)
|
||||
for group in groups
|
||||
]
|
||||
for _ in range(max(map(len, groups), default=0)):
|
||||
for cursor in cursors:
|
||||
row = cursor.fetchone()
|
||||
if row is not None:
|
||||
yield json.loads(row[0])
|
||||
|
||||
|
||||
class FixIndex:
|
||||
def __init__(self, db):
|
||||
self.db = db
|
||||
db.execute("CREATE TABLE fixes (repository TEXT, sha TEXT, PRIMARY KEY (repository, sha)) WITHOUT ROWID")
|
||||
|
||||
def add(self, repository, sha):
|
||||
self.db.execute("INSERT OR IGNORE INTO fixes VALUES (?, ?)", (repository, sha))
|
||||
|
||||
def get(self, repository, default=None):
|
||||
fixes = RepositoryFixes(self.db, repository)
|
||||
return fixes if fixes else default
|
||||
|
||||
|
||||
class RepositoryFixes(Set):
|
||||
def __init__(self, db, repository):
|
||||
self.db, self.repository = db, repository
|
||||
|
||||
def __len__(self):
|
||||
return self.db.execute("SELECT count(*) FROM fixes WHERE repository=?", (self.repository,)).fetchone()[0]
|
||||
|
||||
def __iter__(self):
|
||||
for (sha,) in self.db.execute("SELECT sha FROM fixes WHERE repository=? ORDER BY sha", (self.repository,)):
|
||||
yield sha
|
||||
|
||||
def __contains__(self, sha):
|
||||
return self.db.execute("SELECT 1 FROM fixes WHERE repository=? AND sha=?", (self.repository, sha)).fetchone() is not None
|
||||
+18
-3
@@ -17,6 +17,21 @@ This page uses two words strictly:
|
||||
been made.** Section B is guidance, not a record. (kind runs a Kubernetes cluster inside Docker
|
||||
containers on one machine.)
|
||||
|
||||
The kind profile runs on a 7.7 GB server VM. Observability is off by default
|
||||
(`kind_observability = false` in `ops/k8s/profiles/kind.toml`). This saves the measured
|
||||
930 MiB used by `otel-system`, about 40% of the cluster's 2.3 GB. The smoke Run still
|
||||
completes with those deployments stopped. To enable the stack, run
|
||||
`OUSAST_KIND_OBSERVABILITY=1 ops/ax/up.sh` or set `kind_observability = true` in the profile.
|
||||
|
||||
`up.sh` stages copied Substrate manifests in a temporary checkout view and installs through
|
||||
`hack/install-ate-kind.sh`. The local `ops/ax/substrate-kind-lean/` overlay removes the collector,
|
||||
Prometheus, Jaeger and their supporting resources before installation. This preserves the
|
||||
installer's secret, CRD and image setup. The upstream checkout is unchanged. The shared
|
||||
`ate-otel-config` disables trace, metric and log export and the OTEL SDK; it removes the old
|
||||
collector endpoint and metric timing keys. On an existing cluster, `up.sh` scales the three
|
||||
observability deployments to zero, replaces the ConfigMap data and rolls its consumers when
|
||||
the configuration changes. Opting in restores the upstream ConfigMap and observability manifests.
|
||||
|
||||
## What runs where today (implemented)
|
||||
|
||||
```mermaid
|
||||
@@ -357,7 +372,7 @@ This section answers the maintainer's questions from the code and the upstream c
|
||||
|
||||
The decisions of 2026-10-02 set the target: **two profiles of one code base**.
|
||||
|
||||
- `kind` is this laptop. It stays a first-class local execution mode. Every change is proven here first.
|
||||
- `kind` is this server VM. It stays a first-class local execution mode. Every change is proven here first.
|
||||
- `k3s` is production. (k3s is a small Kubernetes distribution.)
|
||||
|
||||
Everything marked **planned** is not built. The rest is what runs on this host today.
|
||||
@@ -368,7 +383,7 @@ flowchart LR
|
||||
hook["ousast pre-push: local, 30 s deadline"]
|
||||
cli["ousast plane run, remote mode (planned)"]
|
||||
end
|
||||
subgraph k3s ["k3s cluster (the kind profile is the same picture on one laptop)"]
|
||||
subgraph k3s ["k3s cluster (the kind profile is the same picture on one server VM)"]
|
||||
subgraph axns ["Namespace ax-system"]
|
||||
axs["ax-server Deployment + Redis"]
|
||||
end
|
||||
@@ -403,7 +418,7 @@ This picture has no inbound path:
|
||||
- the tasks push their artifacts to the store through the egress gateway;
|
||||
- the CLI reads the store.
|
||||
|
||||
Today's EndpointSlice back to a laptop (`ops/ax/receiver-service.yaml.tmpl`) belongs to the kind
|
||||
Today's EndpointSlice back to the server VM (`ops/ax/receiver-service.yaml.tmpl`) belongs to the kind
|
||||
profile only. It does not appear in production.
|
||||
|
||||
### ax, Kubernetes and Agent Substrate in three sentences
|
||||
|
||||
+1
-1
@@ -159,7 +159,7 @@ flowchart LR
|
||||
ax is a Kubernetes application over Agent Substrate. Every Task runs the one runner image as an actor. Each actor
|
||||
occupies a whole worker. This code base targets two profiles:
|
||||
|
||||
- `kind` on a laptop (today).
|
||||
- `kind` on a server VM (today).
|
||||
- `k3s` in production (planned). There the same CLI submits Runs remotely, and the tasks deliver to the S3 store.
|
||||
|
||||
[Production topology: ax on a Kubernetes cluster](deployment.md#production-topology-ax-on-a-kubernetes-cluster)
|
||||
|
||||
@@ -214,3 +214,42 @@ bucket `OUSAST_MEMORY_TEST_BUCKET`, else `S3_BUCKET`. The tests:
|
||||
- never configure the bucket.
|
||||
|
||||
More on the host setup: [ax on this host](ops/ax/README.md).
|
||||
|
||||
## Scoped account for engine workers
|
||||
|
||||
Create a **new account** for the k3s engine Deployment. Attach only this policy.
|
||||
Do not give it the existing memory-store account's policies.
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Sid": "ListEnginePrefixes",
|
||||
"Effect": "Allow",
|
||||
"Action": "s3:ListBucket",
|
||||
"Resource": "arn:aws:s3:::sast-memory",
|
||||
"Condition": {
|
||||
"StringLike": {
|
||||
"s3:prefix": ["engine-queue/", "engine-queue/*", "engine-results/", "engine-results/*"]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"Sid": "EngineObjectsOnly",
|
||||
"Effect": "Allow",
|
||||
"Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
|
||||
"Resource": [
|
||||
"arn:aws:s3:::sast-memory/engine-queue/*",
|
||||
"arn:aws:s3:::sast-memory/engine-results/*"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
The cluster can read and write only queued engine inputs, claims, and results.
|
||||
It cannot list or read the rest of the memory store. It cannot change bucket
|
||||
settings. The host queue dispatcher can use the same scoped account. Store its
|
||||
keys in the `ousast-engine-s3` Secret; no key values belong in manifests.
|
||||
Deployment and coordinator commands are in `ops/k8s/engine/README.md`.
|
||||
|
||||
+16
-1
@@ -17,6 +17,21 @@ in `~/.cache/ousast/ax-src/`. The cluster's context, registry and pool sizes com
|
||||
(`ops/k8s/profiles/kind-images.json`), which the templates are re-pinned from and which is committed with them.
|
||||
`up.sh` and `smoke-run.sh` run the checkout's code with `OUSAST_PYTHON` (default `.venv/bin/python`).
|
||||
|
||||
The kind profile runs on a 7.7 GB server VM. Observability is off by default
|
||||
(`kind_observability = false` in `ops/k8s/profiles/kind.toml`). This saves the measured
|
||||
930 MiB used by `otel-system`, about 40% of the cluster's 2.3 GB. The smoke Run still
|
||||
completes with those deployments stopped. To enable the stack, run
|
||||
`OUSAST_KIND_OBSERVABILITY=1 ops/ax/up.sh` or set `kind_observability = true` in the profile.
|
||||
|
||||
`up.sh` stages copied Substrate manifests in a temporary checkout view and installs through
|
||||
`hack/install-ate-kind.sh`. The local `ops/ax/substrate-kind-lean/` overlay removes the collector,
|
||||
Prometheus, Jaeger and their supporting resources before installation. This preserves the
|
||||
installer's secret, CRD and image setup. The upstream checkout is unchanged. The shared
|
||||
`ate-otel-config` disables trace, metric and log export and the OTEL SDK; it removes the old
|
||||
collector endpoint and metric timing keys. On an existing cluster, `up.sh` scales the three
|
||||
observability deployments to zero, replaces the ConfigMap data and rolls its consumers when
|
||||
the configuration changes. Opting in restores the upstream ConfigMap and observability manifests.
|
||||
|
||||
## How a Run executes
|
||||
|
||||
A Run (`plane/runs/*.yaml`, kind `openultrasast.io/v1alpha1 Run`) is a DAG of steps; each step names an ax Task
|
||||
@@ -191,7 +206,7 @@ configure the bucket.
|
||||
No deployment outside this host's kind cluster has been made. The step-by-step guidance, with what is implemented
|
||||
and what is planned, is `docs/deployment.md` (the site page "Deployment"). The Model and Workspace manifests under
|
||||
`plane/` move as they are and the Task templates need their image reference re-pinned to your registry; these parts
|
||||
assume this laptop and must change first (checked 2026-10-02):
|
||||
assume this server VM and must change first (checked 2026-10-02):
|
||||
|
||||
| Assumption | Where | Needed in a real cluster |
|
||||
|---|---|---|
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Stage an installer checkout with lean kind manifests; never modify the source checkout.
|
||||
|
||||
Usage: python ops/ax/prepare-substrate.py SOURCE EMPTY_DESTINATION
|
||||
The destination lives only for an install/render. Source/build inputs are symlinked;
|
||||
manifests are copied because the installer uses fixed paths (including the OTEL ConfigMap).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import shutil
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
|
||||
def prepare(source: Path, destination: Path) -> None:
|
||||
source = source.resolve()
|
||||
destination.mkdir(parents=True, exist_ok=True)
|
||||
if any(destination.iterdir()):
|
||||
raise ValueError(f"staging directory must be empty: {destination}")
|
||||
for entry in source.iterdir():
|
||||
if entry.name != "manifests":
|
||||
(destination / entry.name).symlink_to(entry, target_is_directory=entry.is_dir())
|
||||
shutil.copytree(source / "manifests", destination / "manifests")
|
||||
install = destination / "manifests/ate-install"
|
||||
shutil.copytree(install / "kind", install / "kind-upstream")
|
||||
overlay = Path(__file__).resolve().parent / "substrate-kind-lean"
|
||||
shutil.copytree(overlay, install / "kind", dirs_exist_ok=True)
|
||||
# The installer applies this file directly before it renders the bundle.
|
||||
# Keep the strategic-merge directive only in the overlay's patch file.
|
||||
config = yaml.safe_load((overlay / "ate-otel-config.yaml").read_text())
|
||||
config["data"].pop("$patch")
|
||||
(install / "kind/disabled-otel-config.yaml").write_text(yaml.safe_dump(config))
|
||||
(install / "kind/ate-otel-config.yaml").rename(install / "kind/otel-config-patch.yaml")
|
||||
(install / "kind/disabled-otel-config.yaml").rename(install / "kind/ate-otel-config.yaml")
|
||||
kustomization = install / "kind/kustomization.yaml"
|
||||
kustomization.write_text(kustomization.read_text().replace("path: ate-otel-config.yaml", "path: otel-config-patch.yaml"))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
prepare(Path(sys.argv[1]), Path(sys.argv[2]))
|
||||
@@ -0,0 +1,11 @@
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: ate-otel-config
|
||||
namespace: ate-system
|
||||
data:
|
||||
$patch: replace
|
||||
OTEL_TRACES_EXPORTER: none
|
||||
OTEL_METRICS_EXPORTER: none
|
||||
OTEL_LOGS_EXPORTER: none
|
||||
OTEL_SDK_DISABLED: "true"
|
||||
@@ -0,0 +1,44 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
# prepare-substrate.py stages the upstream kind overlay beside this directory.
|
||||
resources:
|
||||
- ../kind-upstream
|
||||
patches:
|
||||
- target:
|
||||
namespace: otel-system
|
||||
patch: |-
|
||||
$patch: delete
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: unused
|
||||
- target:
|
||||
kind: Namespace
|
||||
name: otel-system
|
||||
patch: |-
|
||||
$patch: delete
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: otel-system
|
||||
- target:
|
||||
kind: ClusterRole
|
||||
name: ate-prometheus
|
||||
patch: |-
|
||||
$patch: delete
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: ate-prometheus
|
||||
- target:
|
||||
kind: RoleBinding
|
||||
name: ate-prometheus
|
||||
namespace: ate-system
|
||||
patch: |-
|
||||
$patch: delete
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: ate-prometheus
|
||||
namespace: ate-system
|
||||
- path: ate-otel-config.yaml
|
||||
+61
-1
@@ -15,6 +15,7 @@ profile() { PYTHONPATH="$REPO_ROOT/src" "$PY" -m openultrasast.plane.profile --p
|
||||
CTX="$(profile kube_context)" # kind names its context kind-<cluster>
|
||||
export KIND_CLUSTER_NAME="${CTX#kind-}"
|
||||
export KO_DOCKER_REPO="${KO_DOCKER_REPO:-$(profile registry)}"
|
||||
OBSERVABILITY="$(profile kind_observability)"
|
||||
IMAGES_FILE="$(profile images)" # the digest pins the reconciler and the templates read
|
||||
SRC="${OUSAST_AX_SRC:-$HOME/.cache/ousast/ax-src}"
|
||||
|
||||
@@ -37,9 +38,68 @@ else
|
||||
fi
|
||||
|
||||
step "agent substrate (ate-system)"
|
||||
# The upstream installer has fixed manifest paths and also creates secrets/CRDs.
|
||||
# Give it a temporary checkout view so it never installs the observability resources.
|
||||
install_substrate() (
|
||||
if [ "$OBSERVABILITY" = true ]; then
|
||||
cd "$SRC/substrate"
|
||||
else
|
||||
# Symlinked build inputs must not make the staged view a different binary version.
|
||||
export VERSION="${VERSION:-$(git -C "$SRC/substrate" describe --tags --always --dirty)}"
|
||||
staged="$(mktemp -d)"
|
||||
trap 'rm -rf "$staged"' EXIT
|
||||
"$PY" "$REPO_ROOT/ops/ax/prepare-substrate.py" "$SRC/substrate" "$staged"
|
||||
cd "$staged"
|
||||
fi
|
||||
KUBECTL_CONTEXT="$CTX" hack/install-ate-kind.sh --deploy-ate-system
|
||||
)
|
||||
if ready ate-system; then echo "substrate ready; kept"; else
|
||||
(cd "$SRC/substrate" && hack/install-ate-kind.sh --deploy-ate-system)
|
||||
install_substrate
|
||||
fi
|
||||
# Reconcile the choice even when the existing control plane is already ready.
|
||||
if [ "$OBSERVABILITY" = true ]; then
|
||||
for manifest in otel-collector prometheus; do
|
||||
kubectl --context "$CTX" apply -f "$SRC/substrate/manifests/ate-install/kind/$manifest.yaml"
|
||||
done
|
||||
otel_config="$SRC/substrate/manifests/ate-install/kind/ate-otel-config.yaml"
|
||||
else
|
||||
otel_config="$REPO_ROOT/ops/ax/substrate-kind-lean/ate-otel-config.yaml"
|
||||
for deployment in opentelemetry-collector prometheus jaeger; do
|
||||
existing="$(kubectl --context "$CTX" -n otel-system get deployment "$deployment" --ignore-not-found -o name)"
|
||||
if [ -n "$existing" ]; then
|
||||
kubectl --context "$CTX" -n otel-system scale "$existing" --replicas=0
|
||||
fi
|
||||
done
|
||||
echo "kind observability off: existing otel-system deployments scaled to 0; opt in with OUSAST_KIND_OBSERVABILITY=1 ops/ax/up.sh"
|
||||
fi
|
||||
# Replace data to remove stale exporter/endpoint keys in either direction. ConfigMap
|
||||
# envFrom is read at pod start; a content annotation triggers only changed rollouts.
|
||||
otel_data="$("$PY" - "$otel_config" <<'PYCONFIG'
|
||||
import json
|
||||
import sys
|
||||
import yaml
|
||||
config = yaml.safe_load(open(sys.argv[1]))
|
||||
config["data"].pop("$patch", None)
|
||||
print(json.dumps(config["data"], sort_keys=True))
|
||||
PYCONFIG
|
||||
)"
|
||||
kubectl --context "$CTX" -n ate-system patch configmap ate-otel-config --type=json \
|
||||
-p "[{\"op\":\"replace\",\"path\":\"/data\",\"value\":$otel_data}]"
|
||||
otel_hash="$(printf '%s' "$otel_data" | sha256sum | cut -d ' ' -f1)"
|
||||
otel_workloads="$(kubectl --context "$CTX" -n ate-system get deployments,daemonsets -o json | "$PY" -c '
|
||||
import json, sys
|
||||
for item in json.load(sys.stdin)["items"]:
|
||||
spec = item["spec"]["template"]["spec"]
|
||||
containers = spec.get("containers", []) + spec.get("initContainers", [])
|
||||
if any(env.get("configMapRef", {}).get("name") == "ate-otel-config"
|
||||
for container in containers for env in container.get("envFrom", [])):
|
||||
print(item["kind"].lower() + "/" + item["metadata"]["name"])
|
||||
')"
|
||||
for workload in $otel_workloads; do
|
||||
kubectl --context "$CTX" -n ate-system patch "$workload" --type=merge \
|
||||
-p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"ousast.io/otel-config\":\"$otel_hash\"}}}}}"
|
||||
kubectl --context "$CTX" -n ate-system rollout status "$workload" --timeout=300s
|
||||
done
|
||||
kubectl --context "$CTX" -n ate-system get pods
|
||||
|
||||
step "egress gateway (atenet-egress, agentgateway variant: one prebuilt image, nothing built)"
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
# Engine workers on k3s
|
||||
|
||||
Build the new **Engine image** workflow on main (or dispatch it), then copy its
|
||||
`ghcr.io/<owner>/openultrasast@sha256:...` job-summary value. The old September
|
||||
image does not include boto3 or this service. The release workflow still builds
|
||||
version tags. The Dockerfile copies the worker and parser under `/app/benchmarks/learn`
|
||||
and installs `.[semantic,s3]`; no analyzer tarball is sent by the dispatcher.
|
||||
|
||||
Use two schedulable nodes with at least 3 GiB available each. Required hostname
|
||||
anti-affinity places one worker on each node. Updates use Recreate so a third
|
||||
worker does not wait forever for a third node. Local scratch uses the container's
|
||||
filesystem, with no host mount or Kubernetes API access.
|
||||
|
||||
Create a **new scoped account** using the policy in `docs/rustfs.md`. Export its
|
||||
`S3_ENDPOINT`, `S3_BUCKET=sast-memory`, `AWS_ACCESS_KEY_ID`, and
|
||||
`AWS_SECRET_ACCESS_KEY` in the coordinator shell. Do not use the memory-store
|
||||
account. The following commands are for the coordinator; they were not run by the
|
||||
implementation agent.
|
||||
|
||||
```sh
|
||||
export ENGINE_IMAGE='ghcr.io/<owner>/openultrasast@sha256:<digest-from-workflow>'
|
||||
# From the repository root, edit the digest parameter without changing the template:
|
||||
export ENGINE_OVERLAY="$(mktemp -d /tmp/ousast-engine.XXXXXX)"
|
||||
cp ops/k8s/engine/{namespace,deployment,kustomization}.yaml "$ENGINE_OVERLAY/"
|
||||
python - <<'PY'
|
||||
import os
|
||||
from pathlib import Path
|
||||
image, digest = os.environ['ENGINE_IMAGE'].split('@', 1)
|
||||
p = Path(os.environ['ENGINE_OVERLAY']) / 'kustomization.yaml'
|
||||
p.write_text(p.read_text().replace('ghcr.io/OWNER/openultrasast', image)
|
||||
.replace('sha256:REPLACE_WITH_ENGINE_IMAGE_DIGEST', digest))
|
||||
PY
|
||||
kubectl apply -f ops/k8s/engine/namespace.yaml
|
||||
kubectl -n ousast-engine create secret generic ousast-engine-s3 \
|
||||
--from-literal=S3_ENDPOINT="$S3_ENDPOINT" \
|
||||
--from-literal=S3_BUCKET="$S3_BUCKET" \
|
||||
--from-literal=AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" \
|
||||
--from-literal=AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY"
|
||||
kubectl apply -k "$ENGINE_OVERLAY"
|
||||
kubectl -n ousast-engine rollout status deployment/ousast-engine-worker
|
||||
kubectl -n ousast-engine get pods -o wide
|
||||
|
||||
export PATH=/home/mc/Source/OpenUltraSAST/.venv/bin:$PATH
|
||||
export PYTHONPATH="$PWD/src"
|
||||
python benchmarks/learn/engine_trace.py --queue-status
|
||||
python benchmarks/learn/engine_trace.py --executor queue \
|
||||
--image "$ENGINE_IMAGE" --parallel 2 --deadline 900 --queue-timeout 3600 \
|
||||
--units plane/experiments/exp-005-graph-slice.units.jsonl \
|
||||
--out "$HOME/ousast-results/plane/engine-trace-queue"
|
||||
```
|
||||
|
||||
The dispatcher uses the existing local memory examples by default. Add
|
||||
`--examples /path/to/examples.jsonl` for a frozen example snapshot. Materialization
|
||||
uses locally cached source pins, just as the Docker executor does. `--executor
|
||||
k8s-jobs` preserves the old Job lane (`k8s` is its legacy alias); `queue` requires
|
||||
no kubectl access. Resume skips completed local units and retries failed or timed
|
||||
out units. `progress.json`, `STOP`, filtering, and `--summary` retain their existing
|
||||
meaning. Each invocation uses a new run ID, so an old remote result cannot satisfy
|
||||
a new retry. The dispatcher records worker and pod identity in each pin JSON.
|
||||
|
||||
Tasks contain the pin, units, prepared questions, source object key, and deadlines.
|
||||
Results live at `engine-results/<run>/<pin-id>.json`; inputs are removed after
|
||||
publication. The Docker lane retains JSON and a log, not a result tar, so the queue
|
||||
also returns JSON. Queue timeout records timeout locally and removes pending;
|
||||
an already claimed task finishes remotely and cleans its input. Orphaned inputs
|
||||
from a failed submission can be removed under `engine-queue/inputs/` after checking
|
||||
that no pending task or claim references them.
|
||||
|
||||
Claims use per-worker choosing markers followed by ordered tickets. A worker waits
|
||||
for choosing peers and every lower ticket before removing pending and starting.
|
||||
This requires strongly consistent GET/PUT/LIST from RustFS, including paginated
|
||||
LIST, and synchronized node clocks with skew below 30 seconds. Half of the 60-second lease margin is reserved for clock
|
||||
skew; the remaining execution budget is checked again after claiming.
|
||||
It does not require conditional writes, object tags, S3 Select, or bucket-admin
|
||||
permissions. A dead worker's task is recovered from its expired claim. Execution
|
||||
runs in a child process. Linux subreaping and a frozen descendant walk kill its
|
||||
entire tree, including Joern's separate sessions, at the task deadline before the
|
||||
lease expires. As with any unfenced lease, a node suspended beyond its lease must be
|
||||
terminated before it rejoins; the protocol cannot guarantee exclusivity against
|
||||
arbitrary process suspension or an inconsistent object store.
|
||||
|
||||
SIGTERM stops new claims and lets the current task finish. The 900-second
|
||||
termination grace matches the maximum task deadline. Keep these values aligned
|
||||
if changing the deployment. `/healthz` serves both probes and becomes unready on
|
||||
shutdown. Errors omit SDK exception text to avoid exposing credentials.
|
||||
@@ -0,0 +1,63 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: ousast-engine-worker
|
||||
spec:
|
||||
replicas: 2
|
||||
strategy:
|
||||
type: Recreate
|
||||
selector:
|
||||
matchLabels:
|
||||
app: ousast-engine-worker
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: ousast-engine-worker
|
||||
spec:
|
||||
automountServiceAccountToken: false
|
||||
terminationGracePeriodSeconds: 900
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
affinity:
|
||||
podAntiAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
- labelSelector:
|
||||
matchLabels:
|
||||
app: ousast-engine-worker
|
||||
topologyKey: kubernetes.io/hostname
|
||||
containers:
|
||||
- name: worker
|
||||
image: engine-image
|
||||
command: [python, /app/benchmarks/learn/engine_worker_service.py]
|
||||
envFrom:
|
||||
- secretRef:
|
||||
name: ousast-engine-s3
|
||||
env:
|
||||
- name: ENGINE_IMAGE
|
||||
value: engine-image
|
||||
resources:
|
||||
requests:
|
||||
memory: 3Gi
|
||||
cpu: '1'
|
||||
limits:
|
||||
memory: 3Gi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop: [ALL]
|
||||
ports:
|
||||
- name: health
|
||||
containerPort: 8080
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /healthz
|
||||
port: health
|
||||
periodSeconds: 5
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /healthz
|
||||
port: health
|
||||
initialDelaySeconds: 15
|
||||
periodSeconds: 10
|
||||
@@ -0,0 +1,22 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
namespace: ousast-engine
|
||||
resources:
|
||||
- namespace.yaml
|
||||
- deployment.yaml
|
||||
# Deliberately non-pullable until the coordinator supplies the workflow's digest.
|
||||
images:
|
||||
- name: engine-image
|
||||
newName: ghcr.io/OWNER/openultrasast
|
||||
digest: sha256:REPLACE_WITH_ENGINE_IMAGE_DIGEST
|
||||
replacements:
|
||||
- source:
|
||||
kind: Deployment
|
||||
name: ousast-engine-worker
|
||||
fieldPath: spec.template.spec.containers.0.image
|
||||
targets:
|
||||
- select:
|
||||
kind: Deployment
|
||||
name: ousast-engine-worker
|
||||
fieldPaths:
|
||||
- spec.template.spec.containers.0.env.0.value
|
||||
@@ -0,0 +1,4 @@
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: ousast-engine
|
||||
@@ -0,0 +1,9 @@
|
||||
# Reference only; excluded from kustomization. Create using the README command.
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: ousast-engine-s3
|
||||
namespace: ousast-engine
|
||||
type: Opaque
|
||||
stringData: {}
|
||||
# Required keys: S3_ENDPOINT, S3_BUCKET, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
|
||||
@@ -1,3 +1,3 @@
|
||||
{
|
||||
"runner": "localhost:5001/ousast-runner@sha256:d2a55293999cc34581ef1f02a1ffb0c2dd8fc0cbe001ac952df81600e868f019"
|
||||
"runner": "localhost:5001/ousast-runner@sha256:784bc6afc869c679c4966d04d3f9653a8d379726b899809590cc83327f4038e3"
|
||||
}
|
||||
|
||||
@@ -1,8 +1,9 @@
|
||||
# The kind profile: the plane on this laptop's single-node kind cluster, which ops/ax/up.sh creates and which is
|
||||
# The kind profile: the plane on this server VM's single-node kind cluster, which ops/ax/up.sh creates and which is
|
||||
# the development default (plane-on-kubernetes design section 1). Values only; a credential never goes here
|
||||
# (keys stay in the environment or .env). Every field is overridable by OUSAST_<FIELD> in the environment, a pool
|
||||
# by OUSAST_POOL_<NAME>_<FIELD>. This file may name the kind cluster; the code must not (tests/test_plane_literals.py).
|
||||
|
||||
kind_observability = false # opt in with OUSAST_KIND_OBSERVABILITY=1
|
||||
exec = "local" # the reconciler runs on this host against the cluster below
|
||||
kube_context = "kind-ousast" # kind's context for KIND_CLUSTER_NAME=ousast
|
||||
registry = "localhost:5001" # the kind-local registry up.sh creates (Substrate rewrites it for the node)
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
# Build from the repository root. AX v0.3.1 starts spec.command immediately;
|
||||
# preserve its runner as PID 1 and put the engine command in the Task manifest.
|
||||
FROM eclipse-temurin:21-jre-jammy AS jre
|
||||
|
||||
FROM ghcr.io/norandom/ax-task-runner:v0.3.1 AS engine
|
||||
LABEL org.opencontainers.image.licenses="Apache-2.0" \
|
||||
org.opencontainers.image.source="https://github.com/norandom/OpenUltraSAST"
|
||||
|
||||
USER root
|
||||
ARG JOERN_VERSION=v4.0.625
|
||||
ARG ARCHIVE=joern-cli-linux-x86_64.zip
|
||||
|
||||
# The AX runner is Python 3.12 / Debian slim. Copy the Java 21 runtime from
|
||||
# Jammy (older glibc than Debian bookworm) instead of relying on distro JRE versions.
|
||||
COPY --from=jre /opt/java/openjdk /opt/java/openjdk
|
||||
ENV JAVA_HOME=/opt/java/openjdk
|
||||
ENV PATH="/venv/bin:/opt/java/openjdk/bin:/opt/joern-cli:${PATH}"
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends \
|
||||
php-cli unzip curl ca-certificates libstdc++6 zlib1g \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
RUN set -eux; \
|
||||
cd /tmp; \
|
||||
curl -fsSL -O "https://github.com/joernio/joern/releases/download/${JOERN_VERSION}/${ARCHIVE}"; \
|
||||
curl -fsSL -O "https://github.com/joernio/joern/releases/download/${JOERN_VERSION}/${ARCHIVE}.sha512"; \
|
||||
sed 's@ target/@ @' "${ARCHIVE}.sha512" > checksum.sha512; \
|
||||
sha512sum -c checksum.sha512; \
|
||||
unzip -q "${ARCHIVE}" -d /opt; \
|
||||
rm -f "${ARCHIVE}" "${ARCHIVE}.sha512" checksum.sha512; \
|
||||
/opt/joern-cli/joern-parse --help > /dev/null
|
||||
|
||||
RUN if [ ! -e /opt/joern-cli/jssrc2cpg ] && [ -x /opt/joern-cli/jssrc2cpg.sh ]; then \
|
||||
ln -s jssrc2cpg.sh /opt/joern-cli/jssrc2cpg; \
|
||||
fi
|
||||
COPY ops/frontend-retention/install.py /tmp/frontend-retention-install.py
|
||||
RUN python3 /tmp/frontend-retention-install.py /opt/joern-cli \
|
||||
&& rm /tmp/frontend-retention-install.py
|
||||
|
||||
WORKDIR /app
|
||||
COPY pyproject.toml README.md LICENSE ./
|
||||
COPY src ./src
|
||||
COPY benchmarks/learn/engine_task_entry.py \
|
||||
benchmarks/learn/engine_trace_worker.py \
|
||||
benchmarks/learn/engine_trace_parse.py ./benchmarks/learn/
|
||||
# Worker imports are standard-library plus engine modules. Isolate them from
|
||||
# the AX runner's Python environment and omit PyYAML, boto3 and all extras.
|
||||
RUN python3 -m venv /venv \
|
||||
&& /venv/bin/pip install --no-cache-dir --no-deps . \
|
||||
&& /venv/bin/python -I -c 'import pathlib, runpy, sys; sys.modules.update(yaml=None, boto3=None); sys.path.insert(0, "/app/benchmarks/learn"); p = pathlib.Path("/app/benchmarks/learn/engine_trace_worker.py"); print("Worker import smoke bytes:", len(p.read_bytes())); runpy.run_path(str(p), run_name="engine_worker_import_smoke")'
|
||||
|
||||
# Do not set ENTRYPOINT or CMD: both belong to the upstream AX runner.
|
||||
|
||||
FROM engine AS replay
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends git \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
COPY benchmarks/unseen/replay_entry.py ./benchmarks/unseen/replay_entry.py
|
||||
RUN /venv/bin/pip install --no-cache-dir . \
|
||||
&& /venv/bin/ousast pre-push --help > /dev/null
|
||||
@@ -53,7 +53,8 @@
|
||||
}
|
||||
|
||||
val calls = {
|
||||
val all = cpg.call.filter(c => matches(c.name, c.code)).l
|
||||
// CPS-IT/mailqueue Extension::KEY has a null .method; exclude methodless calls before attribution.
|
||||
val all = cpg.call.filter(c => Option(c.method).nonEmpty && matches(c.name, c.code)).l
|
||||
val inFile = if (fileS.isEmpty) all else all.filter(_.method.filename.endsWith(fileS))
|
||||
if (function.isEmpty) inFile else inFile.filter(_.method.name == function)
|
||||
}
|
||||
|
||||
@@ -172,7 +172,8 @@
|
||||
// repository this was measured on, and unscoped rows attributed the model's unguarded query to the
|
||||
// handler. The sibling comparison is a statement about a FILE contradicting itself, which is what the
|
||||
// rung's justification says, so the file is also the right scope for it.
|
||||
val allOperations = cpg.call.filter(isOperation).l
|
||||
// CPS-IT/mailqueue Extension::KEY has a null .method; exclude methodless calls before attribution.
|
||||
val allOperations = cpg.call.filter(c => Option(c.method).nonEmpty).filter(isOperation).l
|
||||
val opCalls = if (fileS.isEmpty) allOperations else allOperations.filter(_.method.filename.endsWith(fileS))
|
||||
|
||||
// ---- The module boundary an application puts between its route and its query --------------------------
|
||||
@@ -236,7 +237,7 @@
|
||||
if (fileS.isEmpty) Nil
|
||||
else
|
||||
fileMethods(fileS).flatMap { m =>
|
||||
m.call.l.flatMap { via =>
|
||||
m.call.l.filter(c => Option(c.method).nonEmpty).flatMap { via =>
|
||||
via.callee.l
|
||||
.filter(_.isExternal)
|
||||
.flatMap(stub => stubTarget(stub.fullName))
|
||||
|
||||
@@ -56,7 +56,8 @@
|
||||
contextFilter: String = "",
|
||||
contextPaths: String = "",
|
||||
requests: String = "",
|
||||
requestsFile: String = ""
|
||||
requestsFile: String = "",
|
||||
trace: String = ""
|
||||
) = {
|
||||
importCpg(cpgFile)
|
||||
|
||||
@@ -225,6 +226,7 @@
|
||||
// module region exists to cover the code that is in NO function, so its membership is direct.
|
||||
val moduleScope = function == "<module>"
|
||||
|
||||
// CPS-IT/mailqueue Extension::KEY is a class-level PHP call with a null .method; exclude it from scope.
|
||||
def nestedInLabeled(m: io.shiftleft.codepropertygraph.generated.nodes.Method): Boolean =
|
||||
if (moduleScope) m.name == function
|
||||
else
|
||||
@@ -495,7 +497,7 @@
|
||||
case c: io.shiftleft.codepropertygraph.generated.nodes.Call if originAnchors.contains(c.name) => true
|
||||
case c: io.shiftleft.codepropertygraph.generated.nodes.Call if concatenations.contains(c.name) =>
|
||||
c.argument.l.sortBy(_.argumentIndex).headOption.exists(a => fixesOrigin(a, depth + 1))
|
||||
case i: io.shiftleft.codepropertygraph.generated.nodes.Identifier =>
|
||||
case i: io.shiftleft.codepropertygraph.generated.nodes.Identifier if i.method != null =>
|
||||
// A variable fixes the origin when EVERY assignment to it in the method does.
|
||||
val writes = i.method.ast.isCall.nameExact(ASSIGNMENT).l.filter(_.argument.l.sortBy(_.argumentIndex).headOption.exists {
|
||||
case t: io.shiftleft.codepropertygraph.generated.nodes.Identifier => t.name == i.name
|
||||
@@ -789,7 +791,7 @@
|
||||
case _ => false
|
||||
}
|
||||
ternary || (end :: elements.filter(stored)).distinctBy(_.id).exists {
|
||||
case cfg: io.shiftleft.codepropertygraph.generated.nodes.CfgNode =>
|
||||
case cfg: io.shiftleft.codepropertygraph.generated.nodes.CfgNode if cfg.method != null =>
|
||||
val method = cfg.method
|
||||
method.ast.isControlStructure.controlStructureType("IF").l.exists { cs =>
|
||||
cs.condition.l.headOption.exists { cond =>
|
||||
@@ -897,7 +899,7 @@
|
||||
def fieldSourceNodes =
|
||||
if (fieldSrc != "true") Iterator.empty
|
||||
else {
|
||||
val reads = cpg.call.nameExact(FIELD_ACCESS).filter(c => inScope(c.method)).l.filterNot(writeTarget)
|
||||
val reads = cpg.call.nameExact(FIELD_ACCESS).filter(c => Option(c.method).exists(inScope)).l.filterNot(writeTarget)
|
||||
if (reads.isEmpty) Iterator.empty
|
||||
else reads.filter(r => taintedPrefixes(r.code.trim, fileS, r.method) && !overwrittenBefore(r)).iterator
|
||||
}
|
||||
@@ -957,7 +959,7 @@
|
||||
val framework: List[io.shiftleft.codepropertygraph.generated.nodes.CfgNode] =
|
||||
callback.ast.isCall.filter(isSourceCall).l
|
||||
val fields: List[io.shiftleft.codepropertygraph.generated.nodes.CfgNode] =
|
||||
callback.ast.isCall.nameExact(FIELD_ACCESS).l.filter(r => taintedPrefixes(r.code.trim, callback.filename, r.method))
|
||||
callback.ast.isCall.nameExact(FIELD_ACCESS).l.filter(r => Option(r.method).exists(m => taintedPrefixes(r.code.trim, callback.filename, m)))
|
||||
framework ++ fields
|
||||
}
|
||||
|
||||
@@ -1012,7 +1014,7 @@
|
||||
def hookSourceNodes =
|
||||
if (hookTable.isEmpty || hookApply.isEmpty) Iterator.empty
|
||||
else {
|
||||
val applies = cpg.call.filter(c => hookApply.contains(c.name)).filter(c => inScope(c.method)).l
|
||||
val applies = cpg.call.filter(c => hookApply.contains(c.name)).filter(c => Option(c.method).exists(inScope)).l
|
||||
if (applies.isEmpty) Iterator.empty
|
||||
else {
|
||||
val keys = applies.flatMap { call =>
|
||||
@@ -1023,7 +1025,7 @@
|
||||
else
|
||||
cpg.call
|
||||
.nameExact(INDEX_ACCESS)
|
||||
.filter(c => inScope(c.method))
|
||||
.filter(c => Option(c.method).exists(inScope))
|
||||
.l
|
||||
.filter(access => keys.contains(keyOf(access)))
|
||||
// ...and actually derived from the filtered value, not merely sharing a key name with it.
|
||||
@@ -1107,7 +1109,7 @@
|
||||
def sinkCalls =
|
||||
sinkCandidatesMemo
|
||||
.getOrElseUpdate(sinksS, cpg.call.filter(c => sinkNames.exists(n => sinkMatches(c, n))).l)
|
||||
.filter(c => inScope(c.method))
|
||||
.filter(c => Option(c.method).exists(inScope))
|
||||
|
||||
// ---- stage one of the two-stage join WITHOUT dataflow, for evidence mode (flow-aware-ranking, phase 2) --
|
||||
//
|
||||
@@ -1169,7 +1171,7 @@
|
||||
)
|
||||
|
||||
def fieldCarriedKind: String =
|
||||
strongest(cpg.call.nameExact(FIELD_ACCESS).filter(c => inScope(c.method)).l.map(r => prefixKind(r.code.trim, fedFields(r.method.filename))))
|
||||
strongest(cpg.call.nameExact(FIELD_ACCESS).filter(c => Option(c.method).exists(inScope)).l.map(r => prefixKind(r.code.trim, fedFields(r.method.filename))))
|
||||
|
||||
def callbackFedKind(callback: String): String =
|
||||
callbackFedMemo.getOrElseUpdate(
|
||||
@@ -1186,7 +1188,7 @@
|
||||
def hookCarriedKind: String =
|
||||
if (hookTable.isEmpty || hookApply.isEmpty) ""
|
||||
else
|
||||
strongest(cpg.call.filter(c => hookApply.contains(c.name)).filter(c => inScope(c.method)).l.flatMap { call =>
|
||||
strongest(cpg.call.filter(c => hookApply.contains(c.name)).filter(c => Option(c.method).exists(inScope)).l.flatMap { call =>
|
||||
val hook = call.argument.l.headOption.map(a => unquote(a.code)).getOrElse("")
|
||||
hookTable.getOrElse(hook, Nil).map(callbackFedKind)
|
||||
})
|
||||
@@ -1232,7 +1234,7 @@
|
||||
else {
|
||||
val confirmed = cpg.call
|
||||
.nameExact(FIELD_ACCESS)
|
||||
.filter(c => inScope(c.method))
|
||||
.filter(c => Option(c.method).exists(inScope))
|
||||
.l
|
||||
.exists(r => prefixKind(r.code.trim, fedFields(r.method.filename)) == "parameter" && taintedPrefixes(r.code.trim, r.method.filename, r.method))
|
||||
if (confirmed) "source" else "parameter"
|
||||
@@ -1276,9 +1278,9 @@
|
||||
// Field carrying is deliberately same-file in the existing abstraction. Retain
|
||||
// its file context conservatively, never claim every method carries the value.
|
||||
val fieldFiles = if (fieldCarriedKind.isEmpty) Set.empty[String] else
|
||||
cpg.call.nameExact(FIELD_ACCESS).filter(c => inScope(c.method)).l.map(_.method.filename).toSet
|
||||
cpg.call.nameExact(FIELD_ACCESS).filter(c => Option(c.method).exists(inScope)).l.map(_.method.filename).toSet
|
||||
val callbacks = if (hookApply.isEmpty) Nil else cpg.call.filter(c => hookApply.contains(c.name))
|
||||
.filter(c => inScope(c.method)).l.flatMap { call =>
|
||||
.filter(c => Option(c.method).exists(inScope)).l.flatMap { call =>
|
||||
val hook = call.argument.l.headOption.map(a => unquote(a.code)).getOrElse("")
|
||||
hookTable.getOrElse(hook, Nil).flatMap(name => cpg.method.nameExact(name).filterNot(_.isExternal).l)
|
||||
}.distinct
|
||||
@@ -1400,7 +1402,7 @@
|
||||
def traced(flow: io.joern.dataflowengineoss.language.Path): ujson.Arr =
|
||||
ujson.Arr.from(spliced(flow.elements.l).map { node =>
|
||||
val where = node match {
|
||||
case c: io.shiftleft.codepropertygraph.generated.nodes.CfgNode => c.method.filename.split("/").last + ":" + c.method.name
|
||||
case c: io.shiftleft.codepropertygraph.generated.nodes.CfgNode => Option(c.method).map(m => m.filename.split("/").last + ":" + m.name).getOrElse("?")
|
||||
case _ => "?"
|
||||
}
|
||||
s"$where:${node.lineNumber.getOrElse(-1)} [${node.label}] ${node.code.take(120)}"
|
||||
@@ -1435,7 +1437,7 @@
|
||||
"sinkArg0Literal" -> arg0Literal,
|
||||
"bounded" -> bounds.nonEmpty,
|
||||
"bound" -> bounds.headOption.getOrElse("").take(120),
|
||||
"inLabeledScope" -> (function.isEmpty || nestedInLabeled(sink.method) || reachableMethods.contains(sink.method.fullName))
|
||||
"inLabeledScope" -> (function.isEmpty || Option(sink.method).exists(nestedInLabeled) || reachableMethods.contains(sink.method.fullName))
|
||||
)
|
||||
if (traceS == "true") row("trace") = traced(shortest)
|
||||
row
|
||||
@@ -1489,6 +1491,12 @@
|
||||
// collapsed the sizer to three sink visits per JVM start on the plugin.
|
||||
val started = System.nanoTime()
|
||||
def field(name: String): String = req.obj.get(name).map(_.str).getOrElse("")
|
||||
// Opt-in heartbeat for the trace runner's per-question process watchdog.
|
||||
// Killing the JVM preserves completed streamed rows without unsafe query threads.
|
||||
if (field("questionDeadline").nonEmpty) {
|
||||
println(ujson.write(ujson.Obj("__question__" -> id)))
|
||||
System.out.flush()
|
||||
}
|
||||
val paramSrc = req.obj.get("parameterSources").map(_.str).getOrElse("false")
|
||||
val fieldParamSrc = req.obj.get("fieldParameterSources").map(_.str).getOrElse(paramSrc)
|
||||
val fieldSrc = req.obj.get("fieldSources").map(_.str).getOrElse("true")
|
||||
@@ -1532,7 +1540,7 @@
|
||||
} else {
|
||||
println("---OUSAST-CPG-BEGIN---")
|
||||
println(
|
||||
ujson.write(ujson.Arr(rowsFor(sources, sinks, sanitizers, hookCallbacks, dispatchApply, dispatchValue, function, parameterSources, parameterSources, fieldSources, evidenceOnly, file, callDepth, boundedSinks): _*))
|
||||
ujson.write(ujson.Arr(rowsFor(sources, sinks, sanitizers, hookCallbacks, dispatchApply, dispatchValue, function, parameterSources, parameterSources, fieldSources, evidenceOnly, file, callDepth, boundedSinks, traceS = trace): _*))
|
||||
)
|
||||
}
|
||||
println("---OUSAST-CPG-END---")
|
||||
|
||||
@@ -51,11 +51,13 @@
|
||||
val sourceMemo = scala.collection.mutable.Map.empty[String, Boolean]
|
||||
def nearSource(c: io.shiftleft.codepropertygraph.generated.nodes.Call): Boolean =
|
||||
if (sourcePatterns.isEmpty) false
|
||||
else
|
||||
// CPS-IT/mailqueue Extension::KEY has a null .method; it cannot be near a method source.
|
||||
else Option(c.method).exists { m =>
|
||||
sourceMemo.getOrElseUpdate(
|
||||
c.method.fullName,
|
||||
c.method.ast.isCall.exists(x => sourcePatterns.exists(p => boundedPattern(p).matcher(x.code).find()))
|
||||
m.fullName,
|
||||
m.ast.isCall.exists(x => sourcePatterns.exists(p => boundedPattern(p).matcher(x.code).find()))
|
||||
)
|
||||
}
|
||||
|
||||
def isModelled(c: io.shiftleft.codepropertygraph.generated.nodes.Call): Boolean = {
|
||||
val callee = calleeText(c.code)
|
||||
|
||||
@@ -31,10 +31,12 @@ import os
|
||||
import signal
|
||||
import socket
|
||||
import subprocess
|
||||
import threading
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
import uuid
|
||||
from collections.abc import Callable
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
@@ -78,6 +80,7 @@ class EngineSession:
|
||||
# with nothing: measured as a 25.3 s "build" of which only 4.5 s was the frontend.
|
||||
startup_allowance_seconds: float = 20.0
|
||||
work_reserve_seconds: float = 10.0
|
||||
output_observer: Callable[[str], bool] | None = None
|
||||
failure: str = ""
|
||||
startup_seconds: float = 0.0
|
||||
loaded_graph: str = ""
|
||||
@@ -199,7 +202,34 @@ class EngineSession:
|
||||
target = self.scratch / f"answer-{nonce}.txt"
|
||||
wrapped = _redirected(code, target, nonce)
|
||||
started = time.monotonic()
|
||||
body = self._post(wrapped, timeout=remaining)
|
||||
if self.output_observer is None:
|
||||
body = self._post(wrapped, timeout=remaining)
|
||||
else:
|
||||
# Only HTTP runs in the helper; observation and cancellation stay on the caller.
|
||||
done = threading.Event()
|
||||
response: list[dict[str, object] | None] = []
|
||||
|
||||
def post() -> None:
|
||||
try:
|
||||
response.append(self._post(wrapped, timeout=remaining))
|
||||
finally:
|
||||
done.set()
|
||||
|
||||
threading.Thread(target=post, daemon=True).start()
|
||||
while True:
|
||||
finished = done.wait(0.05)
|
||||
try:
|
||||
first, _, output = target.read_text(errors="replace").partition("\n")
|
||||
except OSError:
|
||||
first, output = "", ""
|
||||
if not self.output_observer(output if first.strip() == nonce else ""):
|
||||
self._poison("session_observer_cancelled")
|
||||
self.close()
|
||||
target.unlink(missing_ok=True)
|
||||
return None
|
||||
if finished:
|
||||
break
|
||||
body = response[0] if response else None
|
||||
seconds = time.monotonic() - started
|
||||
self.requests += 1
|
||||
if body is None:
|
||||
|
||||
@@ -82,6 +82,18 @@ recorded = "2026-10-03"
|
||||
status = "development"
|
||||
exclude_tiers = ["title"]
|
||||
|
||||
# Batch 3: more advisory fixes whose function survives the fix, weighted toward injection and untrusted_destination.
|
||||
# Same slice, cache layout and checks; screening record
|
||||
# benchmarks/measurements/2026-10-04-decision-engine-harvest-3/record.json.
|
||||
[[source]]
|
||||
id = "advisory-fixes-3"
|
||||
kind = "pairs"
|
||||
file = "benchmarks/pairs/advisory-fixes-3/catalog.toml"
|
||||
evaluation = ["benchmarks/pairs/advisory-fixes-3/verification.json"]
|
||||
recorded = "2026-10-04"
|
||||
status = "development"
|
||||
exclude_tiers = ["title"]
|
||||
|
||||
[[source]]
|
||||
id = "adjudications"
|
||||
kind = "adjudications"
|
||||
|
||||
@@ -28,6 +28,7 @@ def request_params(
|
||||
call_depth: int = 0,
|
||||
hook_callbacks: str = "",
|
||||
field_parameter_sources: bool | None = None,
|
||||
trace: bool = False,
|
||||
) -> dict[str, object]:
|
||||
"""The query parameters this arbiter sends. Shared with the batcher so there is one definition, not two.
|
||||
|
||||
@@ -40,6 +41,7 @@ def request_params(
|
||||
means unscoped, which is what the single-region pair path still uses.
|
||||
"""
|
||||
return {
|
||||
**({"trace": "true"} if trace else {}),
|
||||
"sources": spec.sources,
|
||||
"sinks": spec.sinks,
|
||||
"sanitizers": spec.sanitizers,
|
||||
|
||||
@@ -586,6 +586,10 @@ class ObjectClient(Protocol):
|
||||
|
||||
_VERIFIED: set[str] = set()
|
||||
PROBE_KEY = "_probe/select.jsonl"
|
||||
# Live finding: files.because-security.com denies presigned GETs ending in .tar, .gz,
|
||||
# .zip or .bin (HTTP 403 AccessDenied). Use .dat for archive objects; bytes stay tar.
|
||||
PRESIGNED_ARCHIVE_SUFFIX = ".dat" # engine executors only; the plane keeps output.tar
|
||||
|
||||
RUNS_PREFIX = "runs/"
|
||||
RUNS_RULE_ID = "ousast-runs-expiry"
|
||||
RUNS_EXPIRE_DAYS = 30
|
||||
@@ -611,9 +615,12 @@ class S3Store(MemoryStore):
|
||||
presigned URLs. An admin configures the bucket once; opening a store verifies it (:meth:`verify_bucket`) and
|
||||
refuses a bucket without versioning, the ``runs/`` expiry rule or S3 Select. There is no local fallback."""
|
||||
|
||||
def __init__(self, client: ObjectClient, bucket: str, prefix: str = "") -> None:
|
||||
read_only: bool = False
|
||||
|
||||
def __init__(self, client: ObjectClient, bucket: str, prefix: str = "", *, read_only: bool = False) -> None:
|
||||
self.client, self.bucket, self.prefix = client, bucket, prefix.strip("/")
|
||||
if self.describe() not in _VERIFIED:
|
||||
self.read_only = read_only
|
||||
if not read_only and self.describe() not in _VERIFIED:
|
||||
self.verify_bucket()
|
||||
_VERIFIED.add(self.describe())
|
||||
|
||||
@@ -627,9 +634,11 @@ class S3Store(MemoryStore):
|
||||
return self.client.get(self._k(key))
|
||||
|
||||
def _put(self, key: str, data: bytes, labels: Mapping[str, str] | None = None) -> None:
|
||||
self._writable()
|
||||
self.client.put(self._k(key), data, dict(labels or {}))
|
||||
|
||||
def _delete(self, key: str) -> None:
|
||||
self._writable()
|
||||
self.client.delete(self._k(key))
|
||||
|
||||
def _keys(self, prefix: str) -> list[str]:
|
||||
@@ -643,11 +652,16 @@ class S3Store(MemoryStore):
|
||||
tagged = self.client.tags(self._k(key)).get("kind")
|
||||
return tagged is None or tagged == "*" or kind in tagged.split()
|
||||
|
||||
def _writable(self) -> None:
|
||||
if self.read_only:
|
||||
raise MemoryStoreError("read-only store")
|
||||
|
||||
def verify_bucket(self) -> None:
|
||||
"""Read-only check that the admin's one-time setup is in place: versioning ``Enabled``, an enabled
|
||||
"""Check the admin's one-time setup, writing only the capability probe: versioning ``Enabled``, an enabled
|
||||
lifecycle rule expiring ``<prefix>/runs/`` and none expiring the whole store, S3 Select answering a probe
|
||||
over JSON Lines, and the probe's object tags readable. Every missing piece is named with its admin action
|
||||
in one :class:`MemoryStoreError`."""
|
||||
self._writable()
|
||||
runs = self._k(RUNS_PREFIX)
|
||||
endpoint = '--endpoint-url "$S3_ENDPOINT"'
|
||||
missing: list[str] = []
|
||||
@@ -735,6 +749,7 @@ class S3Store(MemoryStore):
|
||||
) from exc
|
||||
|
||||
def presign_put(self, key: str, expires: timedelta = timedelta(hours=1)) -> str:
|
||||
self._writable()
|
||||
return self.client.presign("PUT", self._k(key), expires)
|
||||
|
||||
def presign_get(self, key: str, expires: timedelta = timedelta(hours=1)) -> str:
|
||||
@@ -911,9 +926,10 @@ def s3_settings(environ: Mapping[str, str] | None = None) -> dict[str, Any]:
|
||||
return settings
|
||||
|
||||
|
||||
def open_store(spec: str | None = None, environ: Mapping[str, str] | None = None) -> MemoryStore:
|
||||
def open_store(spec: str | None = None, environ: Mapping[str, str] | None = None, *, read_only: bool = False) -> MemoryStore:
|
||||
"""The store ``spec`` (else ``OUSAST_MEMORY``, else ``file://<results_root()>/memory``) names. ``s3://`` with no
|
||||
bucket uses ``S3_BUCKET``."""
|
||||
bucket uses ``S3_BUCKET``. ``read_only`` skips the S3 write/Select capability probe and refuses S3 mutations;
|
||||
it is intended for GET/list inventory readers, which do not need Select. Default opening is unchanged."""
|
||||
from .reconciler import results_root
|
||||
|
||||
env = environ if environ is not None else os.environ
|
||||
@@ -928,7 +944,7 @@ def open_store(spec: str | None = None, environ: Mapping[str, str] | None = None
|
||||
if not bucket:
|
||||
raise MemoryStoreError("OUSAST_MEMORY=s3:// names no bucket and S3_BUCKET is unset: set one of them")
|
||||
settings = s3_settings(environ)
|
||||
return S3Store(S3Client(bucket=bucket, **settings), bucket, parts.path)
|
||||
return S3Store(S3Client(bucket=bucket, **settings), bucket, parts.path, read_only=read_only)
|
||||
raise MemoryStoreError(f"OUSAST_MEMORY must be file:///<path> or s3://<bucket>[/<prefix>], got {text!r}")
|
||||
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
"""The plane's configuration surface: one ``PlaneProfile`` per cluster (plane-on-kubernetes design section 1).
|
||||
|
||||
A profile is a TOML file under ``ops/k8s/profiles/`` (``kind.toml`` for the laptop's kind cluster, ``k3s.toml`` for
|
||||
A profile is a TOML file under ``ops/k8s/profiles/`` (``kind.toml`` for the server VM's kind cluster, ``k3s.toml`` for
|
||||
the production cluster) holding every value that differs between clusters: the execution mode, the kube context,
|
||||
the registry and the digest-pinned images file, the router and ax-server addresses (empty: tunnel through the
|
||||
Kubernetes API), the memory store, the atespace, ax's snapshot bucket, the worker pools and the image pull secret.
|
||||
@@ -59,6 +59,7 @@ class PlaneProfile:
|
||||
name: str
|
||||
path: Path
|
||||
exec: str = "local"
|
||||
kind_observability: bool = False
|
||||
kube_context: str = ""
|
||||
registry: str = ""
|
||||
images: Path = Path("images.json")
|
||||
@@ -135,6 +136,10 @@ def _reject_secret(where: str, key: str, value: object, declared: bool = False)
|
||||
|
||||
|
||||
def _coerce(where: str, key: str, value: object, kind: str) -> Any:
|
||||
if kind == "bool":
|
||||
if not isinstance(value, bool):
|
||||
raise ProfileError(f"{where}: {key} must be a boolean, got {value!r}")
|
||||
return value
|
||||
if kind == "int":
|
||||
if isinstance(value, bool) or not isinstance(value, int):
|
||||
raise ProfileError(f"{where}: {key} must be an integer, got {value!r}")
|
||||
@@ -172,6 +177,10 @@ def _env_overrides(environ: Mapping[str, str], where: str, values: dict[str, Any
|
||||
raw = environ.get(_ENV_PREFIX + name.upper())
|
||||
if raw is not None and raw != "":
|
||||
value: Any = raw
|
||||
if kind == "bool":
|
||||
if raw.lower() not in ("true", "false", "1", "0"):
|
||||
raise ProfileError(f"{where}: {_ENV_PREFIX}{name.upper()} must be true/false or 1/0, got {raw!r}")
|
||||
value = raw.lower() in ("true", "1")
|
||||
if kind == "int":
|
||||
try:
|
||||
value = int(raw)
|
||||
@@ -241,7 +250,8 @@ def _lookup(profile: PlaneProfile, dotted: str) -> str:
|
||||
return str(getattr(profile.pools[pool_name], key))
|
||||
if head not in _SCALARS or rest:
|
||||
raise ProfileError(f"{profile.path}: no such field {dotted!r}")
|
||||
return str(getattr(profile, head))
|
||||
value = getattr(profile, head)
|
||||
return str(value).lower() if isinstance(value, bool) else str(value)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
|
||||
@@ -21,8 +21,9 @@ from openultrasast.pairs import load_pair_catalog
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
PAIRS = ROOT / "benchmarks" / "pairs"
|
||||
# batch 1 (2026-10-01) and batch 2 (2026-10-03, functions that survive the fix): one source each, same rules
|
||||
BATCHES = {"advisory-fixes": PAIRS / "advisory-fixes", "advisory-fixes-2": PAIRS / "advisory-fixes-2"}
|
||||
# batch 1 (2026-10-01), batches 2 (2026-10-03) and 3 (2026-10-04, functions that survive the fix): one source each,
|
||||
# same rules
|
||||
BATCHES = {name: PAIRS / name for name in ("advisory-fixes", "advisory-fixes-2", "advisory-fixes-3")}
|
||||
SHA = re.compile(r"[0-9a-f]{40}")
|
||||
PERMISSIVE_OR_COPYLEFT = re.compile(
|
||||
r"^(MIT|BSD-[23]-Clause|Apache-2\.0|ISC|Zlib|0BSD|Unlicense|MPL-[12]\.[01]"
|
||||
@@ -106,10 +107,18 @@ def test_no_pair_repeats_a_repository_the_benchmarks_already_name(source_id: str
|
||||
PERMISSIVE = {"MIT", "BSD-2-Clause", "BSD-3-Clause", "Apache-2.0", "ISC", "MPL-2.0", "Unlicense"}
|
||||
|
||||
|
||||
def test_batch_two_is_permissive_and_dated_against_the_cutoff() -> None:
|
||||
"""Batch 2 publishes permissive licences only and records each advisory's publication against the 2026-06-01
|
||||
@pytest.mark.parametrize(("source_id", "batch"), [("advisory-fixes-2", "2026-10-03"), ("advisory-fixes-3", "2026-10-04")])
|
||||
def test_later_batches_are_permissive_and_dated_against_the_cutoff(source_id: str, batch: str) -> None:
|
||||
"""Batches 2 and 3 publish permissive licences only and record each advisory's publication against the 2026-06-01
|
||||
cutoff, so memorisation can be measured on the post-cutoff pairs."""
|
||||
for item in _raw(BATCHES["advisory-fixes-2"]):
|
||||
for item in _raw(BATCHES[source_id]):
|
||||
assert item["license"] in PERMISSIVE, item["name"]
|
||||
assert item["batch"] == "2026-10-03", item["name"]
|
||||
assert item["batch"] == batch, item["name"]
|
||||
assert item["post_cutoff"] is (item["published_at"][:10] >= "2026-06-01"), item["name"]
|
||||
|
||||
|
||||
def test_no_repository_repeats_across_batches() -> None:
|
||||
seen: Counter[str] = Counter()
|
||||
for folder in BATCHES.values():
|
||||
seen.update({repo_name(item["repo"]) for item in _raw(folder)})
|
||||
assert [repo for repo, n in seen.items() if n > 1] == []
|
||||
|
||||
@@ -0,0 +1,159 @@
|
||||
"""Offline shared dispatcher contracts."""
|
||||
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
|
||||
from benchmarks.ax.batch import Item, Workload, dispatch, manifest
|
||||
|
||||
IMAGE = "registry/task@sha256:" + "a" * 64
|
||||
|
||||
|
||||
def item(identity="one"):
|
||||
return Item(identity, {"SPEC_URL": b"{}"}, "result.dat", ("python3", "/entry.py"), {}, identity)
|
||||
|
||||
|
||||
def workload():
|
||||
return Workload(IMAGE, frozenset({"SPEC_URL", "RESULT_URL"}), 900, lambda data, output: {})
|
||||
|
||||
|
||||
def test_manifest_contract():
|
||||
doc = manifest(
|
||||
"ousast-engine-replay-abc",
|
||||
workload(),
|
||||
item(),
|
||||
{"SPEC_URL": "https://files.example/spec.dat", "RESULT_URL": "https://files.example/result.dat"},
|
||||
)
|
||||
assert {v["name"] for v in doc["spec"]["env"]} == {"SPEC_URL", "RESULT_URL"}
|
||||
with pytest.raises(ValueError):
|
||||
manifest("bad", workload(), item(), {})
|
||||
bad = item()
|
||||
bad.extra_env["AWS_SECRET_ACCESS_KEY"] = "secret"
|
||||
with pytest.raises(ValueError):
|
||||
manifest(
|
||||
"ousast-engine-test",
|
||||
workload(),
|
||||
bad,
|
||||
{"SPEC_URL": "https://files.example/spec.dat", "RESULT_URL": "https://files.example/result.dat"},
|
||||
)
|
||||
|
||||
|
||||
def test_dispatch_retry_fallback_resume_stop(tmp_path):
|
||||
calls = []
|
||||
|
||||
def lane(it, output, attempt):
|
||||
calls.append(("ax", attempt))
|
||||
return {"status": "sandbox_failure"}
|
||||
|
||||
def vm(it, output, attempt):
|
||||
calls.append(("docker", attempt))
|
||||
return {"status": "ok"}
|
||||
|
||||
assert not dispatch([item()], workload(), tmp_path, {"ax": lane, "docker": vm}, lanes=("ax", "ax", "docker"), overflow=False)
|
||||
assert [c[0] for c in calls] == ["ax", "ax", "docker"]
|
||||
assert not dispatch([item()], workload(), tmp_path, {"ax": lane, "docker": vm})
|
||||
assert len(calls) == 3
|
||||
(tmp_path / "STOP").touch()
|
||||
dispatch([item("two")], workload(), tmp_path, {"ax": lane, "docker": vm})
|
||||
assert len(calls) == 3
|
||||
assert json.loads((tmp_path / "progress.json").read_text())["pins_in_flight"] == 0
|
||||
|
||||
|
||||
def test_cleanup_failure_halts(tmp_path):
|
||||
calls = []
|
||||
|
||||
def broken(it, output, attempt):
|
||||
calls.append(it.id)
|
||||
raise RuntimeError("cleanup failed")
|
||||
|
||||
with pytest.raises(RuntimeError, match="cleanup"):
|
||||
dispatch([item("one"), item("two")], workload(), tmp_path, {"ax": broken}, lanes=("ax",))
|
||||
assert calls == ["one"]
|
||||
|
||||
|
||||
def test_ax_lane_fake_cli_and_store(tmp_path, monkeypatch):
|
||||
import subprocess
|
||||
from types import SimpleNamespace
|
||||
|
||||
from benchmarks.ax.batch import AXLane
|
||||
|
||||
calls = []
|
||||
|
||||
class Store:
|
||||
objects = {}
|
||||
|
||||
def _put(self, key, data):
|
||||
self.objects[key] = data
|
||||
|
||||
def _get(self, key):
|
||||
return (b"completed", None)
|
||||
|
||||
def _delete(self, key):
|
||||
self.objects.pop(key, None)
|
||||
|
||||
def presign_get(self, key, expiry):
|
||||
assert key.endswith(".dat")
|
||||
return "https://files.example/" + key + "?signature=secret"
|
||||
|
||||
presign_put = presign_get
|
||||
|
||||
def run(command, **kwargs):
|
||||
calls.append((command, kwargs))
|
||||
if command[1] == "get":
|
||||
name = json.loads(calls[0][1]["input"])["metadata"]["name"]
|
||||
return subprocess.CompletedProcess(command, 0, json.dumps({"metadata": {"name": name}, "status": {"phase": "Running"}}), "")
|
||||
return subprocess.CompletedProcess(command, 0, "", "")
|
||||
|
||||
monkeypatch.setattr(subprocess, "run", run)
|
||||
monkeypatch.setenv("S3_ENDPOINT", "https://files.example")
|
||||
|
||||
def validate(data, output):
|
||||
assert data == b"completed"
|
||||
(output / "result.json").write_text('{"status":"ok"}')
|
||||
|
||||
work = Workload(IMAGE, frozenset({"SPEC_URL", "RESULT_URL"}), 10, validate)
|
||||
lane = AXLane(SimpleNamespace(ax_bin="fake-ax", kubeconfig=None, atespace="default"), work, Store())
|
||||
assert lane(item(), tmp_path, 0)["status"] == "ok"
|
||||
doc = json.loads(calls[0][1]["input"])
|
||||
assert doc["metadata"]["name"].startswith("ousast-engine-") and len(doc["metadata"]["name"]) <= 49
|
||||
assert {e["name"] for e in doc["spec"]["env"]} == {"SPEC_URL", "RESULT_URL"}
|
||||
assert calls[-1][0][1:3] == ["delete", "task"]
|
||||
assert not lane.store.objects
|
||||
|
||||
|
||||
def test_bounded_lanes_oom_and_instrument_retry(tmp_path):
|
||||
import threading
|
||||
import time
|
||||
from collections import Counter
|
||||
|
||||
active, peak, calls = Counter(), Counter(), []
|
||||
lock = threading.Lock()
|
||||
|
||||
def execute(lane):
|
||||
def run(it, output, attempt):
|
||||
with lock:
|
||||
calls.append((it.id, lane, attempt))
|
||||
active[lane] += 1
|
||||
peak[lane] = max(peak[lane], active[lane])
|
||||
time.sleep(0.01)
|
||||
with lock:
|
||||
active[lane] -= 1
|
||||
return {"status": "oom" if lane == "ax" else "ok"}
|
||||
|
||||
return run
|
||||
|
||||
assert not dispatch([item(str(i)) for i in range(9)], workload(), tmp_path, {"ax": execute("ax"), "docker": execute("docker")})
|
||||
assert peak["ax"] == 2 and peak["docker"] == 1
|
||||
assert all(attempt == 0 for _, _, attempt in calls)
|
||||
attempts = []
|
||||
|
||||
def failed(it, output, attempt):
|
||||
attempts.append(attempt)
|
||||
return {"status": "instrument_failure"}
|
||||
|
||||
assert dispatch([item("fail")], workload(), tmp_path, {"ax": failed}, lanes=("ax",))
|
||||
assert attempts == [0, 1]
|
||||
@@ -0,0 +1,406 @@
|
||||
"""AX transport controls using an executable shim and an in-memory object store."""
|
||||
|
||||
import io
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
|
||||
with pytest.MonkeyPatch.context() as patch:
|
||||
patch.syspath_prepend(str(Path(__file__).resolve().parents[1] / "benchmarks/learn"))
|
||||
import engine_trace_ax as ax
|
||||
|
||||
|
||||
IMAGE = "ghcr.io/norandom/ousast-engine-task@sha256:" + "a" * 64
|
||||
PIN = {"repo": "owner/repo", "pin": "123", "units": [{"unit": "a", "supported": True}], "questions": {"a": {}}}
|
||||
|
||||
|
||||
def archive(record):
|
||||
buffer = io.BytesIO()
|
||||
with tarfile.open(fileobj=buffer, mode="w") as tar:
|
||||
data = json.dumps(record).encode()
|
||||
info = tarfile.TarInfo("result.json")
|
||||
info.size = len(data)
|
||||
tar.addfile(info, io.BytesIO(data))
|
||||
return buffer.getvalue()
|
||||
|
||||
|
||||
class Store:
|
||||
def __init__(self, result=None):
|
||||
self.result = result
|
||||
self.objects = {}
|
||||
self.deleted = []
|
||||
self.expiries = []
|
||||
|
||||
def _put(self, key, data):
|
||||
self.objects[key] = data
|
||||
|
||||
def _get(self, key):
|
||||
return (self.result, None) if self.result else None
|
||||
|
||||
def _delete(self, key):
|
||||
self.deleted.append(key)
|
||||
self.objects.pop(key, None)
|
||||
|
||||
def presign_get(self, key, expiry):
|
||||
self.expiries.append(expiry.total_seconds())
|
||||
return "https://files.because-security.com/bucket/" + key + "?signature=secret"
|
||||
|
||||
presign_put = presign_get
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def setup(tmp_path, monkeypatch):
|
||||
log = tmp_path / "calls.jsonl"
|
||||
shim = tmp_path / "fake-ax"
|
||||
shim.write_text("#!/bin/sh\nexec " + sys.executable + " " + str(tmp_path / "shim.py") + ' "$@"\n')
|
||||
shim.chmod(0o755)
|
||||
(tmp_path / "shim.py").write_text("""import json, os, sys
|
||||
from pathlib import Path
|
||||
args = sys.argv[1:]
|
||||
with Path(os.environ["AX_LOG"]).open("a") as log:
|
||||
log.write(json.dumps({"args": args, "input": sys.stdin.read() if args[0] == "apply" else "",
|
||||
"kubeconfig": os.environ.get("KUBECONFIG")}) + "\\n")
|
||||
if args[0] == "get" and os.environ.get("AX_GET_FAIL"):
|
||||
sys.exit(1)
|
||||
if args[0] == "get":
|
||||
applies = [json.loads(line) for line in Path(os.environ["AX_LOG"]).read_text().splitlines()
|
||||
if json.loads(line)["args"][0] == "apply"]
|
||||
manifest = json.loads(applies[-1]["input"])
|
||||
phases = os.environ.get("AX_PHASE", "Running").split(",")
|
||||
phase = phases[min(len(applies) - 1, len(phases) - 1)]
|
||||
print(json.dumps({"items": [{"metadata": manifest["metadata"], "status": {
|
||||
"phase": phase, "workerIP": "10.0.0.2"}}] if phase != "Missing" else []}))
|
||||
if args[0] == "delete" and os.environ.get("AX_DELETE_FAIL"):
|
||||
sys.exit(1)
|
||||
""")
|
||||
monkeypatch.setenv("AX_LOG", str(log))
|
||||
monkeypatch.setenv("S3_ENDPOINT", "https://files.because-security.com")
|
||||
args = SimpleNamespace(
|
||||
ax_bin=str(shim), kubeconfig="/operator/kubeconfig", atespace="default", image=IMAGE, deadline=4, question_deadline=2
|
||||
)
|
||||
source = tmp_path / "case"
|
||||
source.mkdir()
|
||||
(source / "app.py").write_text("print(42)\n")
|
||||
output = tmp_path / "out"
|
||||
output.mkdir()
|
||||
now = [0]
|
||||
|
||||
def pause(seconds):
|
||||
now[0] += seconds
|
||||
|
||||
return args, source, output, log, pause, lambda: now[0]
|
||||
|
||||
|
||||
def test_ax_executor_success_while_running_and_cleanup(setup):
|
||||
args, source, output, log, pause, clock = setup
|
||||
store = Store(archive({**PIN, "done": True}))
|
||||
executor = ax.AX(args, store, pause=pause, clock=clock)
|
||||
done, ip = executor.execute(source, output, PIN, "owner/repo/123")
|
||||
assert done.returncode == 0
|
||||
assert ip == "10.0.0.2"
|
||||
assert json.loads((output / "result.json").read_text())["done"]
|
||||
calls = [json.loads(line) for line in log.read_text().splitlines()]
|
||||
assert calls[0]["args"] == ["apply", "-f", "-"]
|
||||
assert calls[-1]["args"][:2] == ["delete", "task"]
|
||||
assert calls[-1]["args"][-2:] == ["--atespace", args.atespace]
|
||||
assert all(c["kubeconfig"] == args.kubeconfig for c in calls)
|
||||
manifest = json.loads(calls[0]["input"])
|
||||
assert manifest["apiVersion"] == "ax.io/v1alpha1"
|
||||
assert manifest["kind"] == "Task"
|
||||
assert len(manifest["metadata"]["name"]) <= 49
|
||||
assert manifest["spec"]["image"] == IMAGE
|
||||
env = {e["name"]: e["value"] for e in manifest["spec"]["env"]}
|
||||
assert set(env) == {"SOURCE_URL", "QUESTIONS_URL", "RESULT_URL", "DEADLINE", "QUESTION_DEADLINE"}
|
||||
assert len(json.dumps(manifest["spec"]["env"]).encode()) < 32768
|
||||
assert "AWS" not in calls[0]["input"]
|
||||
assert env["SOURCE_URL"].split("?")[0].endswith("/source.dat")
|
||||
assert env["RESULT_URL"].split("?")[0].endswith("/result.dat")
|
||||
assert env["QUESTIONS_URL"].split("?")[0].endswith("/questions.json")
|
||||
assert {key.rsplit("/", 1)[-1] for key in store.deleted} == {"source.dat", "questions.json", "result.dat"}
|
||||
assert store.expiries == [604] * 3
|
||||
assert len(store.deleted) == 3 and not store.objects
|
||||
|
||||
|
||||
def test_ax_executor_timeout_deletes_task(setup):
|
||||
args, source, output, log, pause, clock = setup
|
||||
store = Store()
|
||||
done, _ = ax.AX(args, store, pause=pause, clock=clock).execute(source, output, PIN, "pin")
|
||||
assert done is None
|
||||
assert json.loads(log.read_text().splitlines()[-1])["args"][:2] == ["delete", "task"]
|
||||
assert len(store.deleted) == 3
|
||||
|
||||
|
||||
def test_ax_executor_failed_phase_deletes_task(setup, monkeypatch):
|
||||
args, source, output, log, pause, clock = setup
|
||||
monkeypatch.setenv("AX_PHASE", "Failed")
|
||||
done, _ = ax.AX(args, Store(), pause=pause, clock=clock).execute(source, output, PIN, "pin")
|
||||
assert done.returncode == 1 and done.stderr == "AX sandbox failed twice"
|
||||
attempts = json.loads((output / "ax_attempts.json").read_text())
|
||||
assert [a["phase"] for a in attempts] == ["Failed", "Failed"]
|
||||
assert json.loads(log.read_text().splitlines()[-1])["args"][:2] == ["delete", "task"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("phase", ["Failed", "Missing"])
|
||||
def test_ax_executor_retries_with_new_name_links_and_backoff(setup, monkeypatch, phase):
|
||||
args, source, output, log, pause, clock = setup
|
||||
monkeypatch.setenv("AX_PHASE", phase + ",Succeeded")
|
||||
store = Store()
|
||||
store._get = lambda key: (archive({**PIN, "done": True}), None) if "/attempt-1/" in key else None
|
||||
pauses = []
|
||||
|
||||
def backoff(seconds):
|
||||
calls = [json.loads(line) for line in log.read_text().splitlines()]
|
||||
assert calls[-1]["args"][:2] == ["delete", "task"]
|
||||
pauses.append(seconds)
|
||||
pause(seconds)
|
||||
|
||||
done, _ = ax.AX(args, store, pause=backoff, clock=clock).execute(source, output, PIN, "x" * 100)
|
||||
assert done.returncode == 0
|
||||
record = json.loads((output / "result.json").read_text())
|
||||
attempts = record["ax_attempts"]
|
||||
assert record["done"] and [a["phase"] for a in attempts] == [phase, "Succeeded"]
|
||||
assert len({a["name"] for a in attempts}) == 2
|
||||
assert all(a["name"].startswith("ousast-engine-") and len(a["name"]) <= 49 and a["seconds"] >= 0 for a in attempts)
|
||||
assert pauses == [10]
|
||||
calls = [json.loads(line) for line in log.read_text().splitlines()]
|
||||
manifests = [json.loads(c["input"]) for c in calls if c["args"][0] == "apply"]
|
||||
links = [{e["name"]: e["value"] for e in m["spec"]["env"]} for m in manifests]
|
||||
assert all(links[0][key] != links[1][key] for key in ("SOURCE_URL", "QUESTIONS_URL", "RESULT_URL"))
|
||||
assert len(store.expiries) == 6 and len(store.deleted) == 6
|
||||
|
||||
|
||||
@pytest.mark.parametrize("late", [False, True])
|
||||
def test_ax_executor_failed_result_is_never_retried(setup, monkeypatch, late):
|
||||
args, source, output, log, pause, clock = setup
|
||||
monkeypatch.setenv("AX_PHASE", "Failed")
|
||||
store = Store(archive({**PIN, "done": True, "status": "failed"}))
|
||||
original_get = store._get
|
||||
reads = []
|
||||
|
||||
def get(key):
|
||||
reads.append(key)
|
||||
return None if late and len(reads) == 1 else original_get(key)
|
||||
|
||||
store._get = get
|
||||
done, _ = ax.AX(args, store, pause=pause, clock=clock).execute(source, output, PIN, "pin")
|
||||
record = json.loads((output / "result.json").read_text())
|
||||
assert done.returncode == 0 and record["status"] == "failed"
|
||||
assert len(record["ax_attempts"]) == 1
|
||||
assert sum(json.loads(line)["args"][0] == "apply" for line in log.read_text().splitlines()) == 1
|
||||
|
||||
|
||||
@pytest.mark.parametrize("mode", ["ax", "mixed-ax"])
|
||||
def test_ax_sandbox_failed_twice_scheduler(setup, monkeypatch, mode):
|
||||
with monkeypatch.context() as patch:
|
||||
patch.syspath_prepend(str(Path(__file__).resolve().parents[1] / "benchmarks/learn"))
|
||||
import engine_trace as trace
|
||||
|
||||
args, source, output, log, pause, clock = setup
|
||||
monkeypatch.setenv("AX_PHASE", "Failed")
|
||||
args.executor, args.parallel, args.out, args.docker_image = mode, 1, output, "vm-image"
|
||||
args.ax_dispatcher = ax.AX(args, Store(), pause=pause, clock=clock)
|
||||
monkeypatch.setattr(trace, "prepare_questions", lambda *_: PIN["questions"])
|
||||
|
||||
class Inputs:
|
||||
def materialize(self, pin, target):
|
||||
target.mkdir()
|
||||
data = (source / "app.py").read_bytes()
|
||||
(target / "app.py").write_bytes(data)
|
||||
return {"files": 1, "bytes": len(data)}
|
||||
|
||||
original_run = subprocess.run
|
||||
docker_calls = []
|
||||
|
||||
def run(command, **kwargs):
|
||||
if command[0] == args.ax_bin:
|
||||
return original_run(command, **kwargs)
|
||||
assert command[:2] in (["docker", "run"], ["docker", "rm"])
|
||||
docker_calls.append(command)
|
||||
if command[1] == "run":
|
||||
assert command[command.index("--heap-profile") + 1] == "vm"
|
||||
assert "vm-image" in command
|
||||
target = Path(next(arg.removesuffix(":/out") for arg in command if arg.endswith(":/out")))
|
||||
prepared = json.loads((target / "pin.json").read_text())
|
||||
trace.write_json(
|
||||
target / "result.json",
|
||||
{**prepared, "done": True, "units": [trace.unit_record(u, "asked-nothing") for u in prepared["units"]]},
|
||||
)
|
||||
return subprocess.CompletedProcess(command, 0, "", "")
|
||||
|
||||
monkeypatch.setattr(subprocess, "run", run)
|
||||
assert trace.run_plan([PIN], source, {}, Inputs(), args) is (mode == "ax")
|
||||
record = json.loads((output / trace.pin_name(PIN)).read_text())
|
||||
assert [a["phase"] for a in record["ax_attempts"]] == ["Failed", "Failed"]
|
||||
assert sum(json.loads(line)["args"][0] == "apply" for line in log.read_text().splitlines()) == 2
|
||||
if mode == "mixed-ax":
|
||||
assert [command[1] for command in docker_calls] == ["run", "rm"]
|
||||
assert record["executor"] == "docker" and record["fallback"] == "vm-sandbox"
|
||||
assert record["units"][0]["status"] == "asked-nothing"
|
||||
assert record["ax_attempt"]["ax_attempts"] == record["ax_attempts"]
|
||||
else:
|
||||
assert not docker_calls
|
||||
assert record["units"][0]["status"] == "failed"
|
||||
assert record["units"][0]["reason"] == "AX sandbox failed twice"
|
||||
|
||||
|
||||
def test_ax_executor_delete_failure_is_fatal_and_still_cleans_objects(setup, monkeypatch):
|
||||
args, source, output, _, pause, clock = setup
|
||||
monkeypatch.setenv("AX_DELETE_FAIL", "1")
|
||||
store = Store(archive({**PIN, "done": True}))
|
||||
with pytest.raises(RuntimeError, match="cleanup"):
|
||||
ax.AX(args, store, pause=pause, clock=clock).execute(source, output, PIN, "pin")
|
||||
assert len(store.deleted) == 3
|
||||
|
||||
|
||||
@pytest.mark.parametrize("endpoint", ["http://files.because-security.com", "https://127.0.0.1", "https://[::1]", "https://localhost"])
|
||||
def test_ax_executor_endpoint_rejected_before_store_open(setup, monkeypatch, endpoint):
|
||||
args = setup[0]
|
||||
monkeypatch.setenv("S3_ENDPOINT", endpoint)
|
||||
monkeypatch.setattr(ax, "open_store", lambda: pytest.fail("store opened before validation"))
|
||||
with pytest.raises(ValueError, match="egress policy"):
|
||||
ax.AX(args)
|
||||
|
||||
|
||||
def test_ax_executor_rejects_tag_and_large_env(setup):
|
||||
args = setup[0]
|
||||
urls = {key: "https://files.because-security.com/" for key in ("SOURCE_URL", "QUESTIONS_URL", "RESULT_URL")}
|
||||
args.image = "image:latest"
|
||||
with pytest.raises(ValueError, match="digest"):
|
||||
ax.task_manifest("name", args, urls)
|
||||
args.image = IMAGE
|
||||
urls["SOURCE_URL"] += "x" * 32768
|
||||
with pytest.raises(ValueError, match="32 KB"):
|
||||
ax.task_manifest("name", args, urls)
|
||||
|
||||
|
||||
def test_ax_executor_rejects_mismatched_result(setup):
|
||||
args, source, output, _, pause, clock = setup
|
||||
store = Store(archive({"done": True, "units": []}))
|
||||
done, _ = ax.AX(args, store, pause=pause, clock=clock).execute(source, output, PIN, "pin")
|
||||
assert done.returncode == 1
|
||||
assert not (output / "result.json").exists()
|
||||
|
||||
|
||||
def test_ax_executor_table_worker_ip():
|
||||
assert ax.task_state("NAME ATESPACE PHASE WORKER-IP\nours default Running 10.2.3.4\n", "ours") == ("Running", "10.2.3.4")
|
||||
|
||||
|
||||
def test_ax_executor_missing_requires_readable_task_list():
|
||||
assert ax.task_state('{"items": []}', "ours") == ("Missing", None)
|
||||
assert ax.task_state("NAME ATESPACE PHASE WORKER-IP\n", "ours") == ("Missing", None)
|
||||
assert ax.task_state("", "ours") == ("", None)
|
||||
assert ax.task_state("temporarily unavailable", "ours") == ("", None)
|
||||
|
||||
|
||||
def test_ax_executor_status_query_failure_is_not_disappearance(setup, monkeypatch):
|
||||
args, source, output, log, pause, clock = setup
|
||||
monkeypatch.setenv("AX_GET_FAIL", "1")
|
||||
done, _ = ax.AX(args, Store(), pause=pause, clock=clock).execute(source, output, PIN, "pin")
|
||||
assert done is None
|
||||
assert sum(json.loads(line)["args"][0] == "apply" for line in log.read_text().splitlines()) == 1
|
||||
|
||||
|
||||
def test_ax_executor_entry_download_failure_restores_host_units(setup):
|
||||
args, source, output, _, pause, clock = setup
|
||||
store = Store(archive({"done": True, "status": "failed", "reason": "questions download failed", "units": []}))
|
||||
done, _ = ax.AX(args, store, pause=pause, clock=clock).execute(source, output, PIN, "pin")
|
||||
assert done.returncode == 0
|
||||
unit = json.loads((output / "result.json").read_text())["units"][0]
|
||||
assert unit["unit"] == "a" and unit["status"] == "failed"
|
||||
assert unit["reason"] == "questions download failed"
|
||||
|
||||
|
||||
def test_ax_executor_presign_host_must_match_egress_policy(setup):
|
||||
args, source, output, log, pause, clock = setup
|
||||
store = Store()
|
||||
store.presign_put = lambda key, expiry: "https://another-store.example/result"
|
||||
done, _ = ax.AX(args, store, pause=pause, clock=clock).execute(source, output, PIN, "pin")
|
||||
assert done.returncode == 1 and "egress policy" in done.stderr
|
||||
assert not log.exists()
|
||||
assert len(store.deleted) == 3
|
||||
|
||||
|
||||
def test_ax_executor_get_failure_does_not_lose_result(setup, monkeypatch):
|
||||
args, source, output, _, pause, clock = setup
|
||||
monkeypatch.setenv("AX_GET_FAIL", "1")
|
||||
done, _ = ax.AX(args, Store(archive({**PIN, "done": True})), pause=pause, clock=clock).execute(source, output, PIN, "pin")
|
||||
assert done.returncode == 0
|
||||
|
||||
|
||||
def test_ax_executor_absent_worker_ip():
|
||||
assert ax.task_state("NAME ATESPACE PHASE ACTOR WORKER-IP AGE\nours other Pending actor <none> 1s\n", "ours") == ("Pending", None)
|
||||
|
||||
|
||||
def test_ax_executor_late_store_result_times_out(setup):
|
||||
args, source, output, log, pause, clock = setup
|
||||
store = Store(archive({**PIN, "done": True}))
|
||||
original_get = store._get
|
||||
|
||||
def late_get(key):
|
||||
pause(args.deadline)
|
||||
return original_get(key)
|
||||
|
||||
store._get = late_get
|
||||
done, _ = ax.AX(args, store, pause=pause, clock=clock).execute(source, output, PIN, "pin")
|
||||
assert done is None
|
||||
assert not (output / "result.json").exists()
|
||||
assert json.loads(log.read_text().splitlines()[-1])["args"][:2] == ["delete", "task"]
|
||||
|
||||
|
||||
def test_ax_executor_late_status_times_out(setup):
|
||||
args, source, output, _, pause, clock = setup
|
||||
executor = ax.AX(args, Store(archive({**PIN, "done": True})), pause=pause, clock=clock)
|
||||
original_ax = executor.ax
|
||||
|
||||
def late_ax(*command, **kwargs):
|
||||
result = original_ax(*command, **kwargs)
|
||||
if command[0] == "get":
|
||||
pause(args.deadline)
|
||||
return result
|
||||
|
||||
executor.ax = late_ax
|
||||
done, _ = executor.execute(source, output, PIN, "pin")
|
||||
assert done is None
|
||||
assert not (output / "result.json").exists()
|
||||
|
||||
|
||||
def test_ax_executor_blocking_store_read_cannot_delay_task_deletion(setup):
|
||||
import threading
|
||||
import time
|
||||
|
||||
args, source, output, log, _, _ = setup
|
||||
args.deadline = 0.15
|
||||
release = threading.Event()
|
||||
store = Store()
|
||||
store._get = lambda key: release.wait(5)
|
||||
started = time.monotonic()
|
||||
try:
|
||||
done, _ = ax.AX(args, store).execute(source, output, PIN, "pin")
|
||||
assert done is None
|
||||
assert time.monotonic() - started < 2
|
||||
assert json.loads(log.read_text().splitlines()[-1])["args"][:2] == ["delete", "task"]
|
||||
finally:
|
||||
release.set()
|
||||
|
||||
|
||||
def test_ax_task_names_reserve_watcher_prefix_and_fit_actor_limit(setup):
|
||||
args, source, output, log, pause, clock = setup
|
||||
executor = ax.AX(args, Store(archive({**PIN, "done": True})), pause=pause, clock=clock)
|
||||
# Sanitization and truncation must not collapse distinct pins.
|
||||
for pin_id in ["A/B", "a-b", "x" * 100 + "1", "x" * 100 + "2", "☃"]:
|
||||
done, _ = executor.execute(source, output, PIN, pin_id)
|
||||
assert done.returncode == 0
|
||||
manifests = [json.loads(call["input"]) for line in log.read_text().splitlines() if (call := json.loads(line))["args"][0] == "apply"]
|
||||
names = [manifest["metadata"]["name"] for manifest in manifests]
|
||||
assert len(set(names)) == 5
|
||||
for manifest, name in zip(manifests, names, strict=True):
|
||||
assert manifest["kind"] == "Task"
|
||||
assert manifest["metadata"]["atespace"] == "default"
|
||||
assert name.startswith("ousast-engine-")
|
||||
assert len(name.encode()) <= 49
|
||||
assert len((name + "-tmpl-12345678").encode()) <= 63
|
||||
@@ -0,0 +1,130 @@
|
||||
"""Offline overlay contract and installer staging; no Kubernetes executable needed."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import runpy
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
import yaml
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
OVERLAY = ROOT / "ops/ax/substrate-kind-lean"
|
||||
DISABLED = {
|
||||
"OTEL_TRACES_EXPORTER": "none",
|
||||
"OTEL_METRICS_EXPORTER": "none",
|
||||
"OTEL_LOGS_EXPORTER": "none",
|
||||
"OTEL_SDK_DISABLED": "true",
|
||||
}
|
||||
|
||||
|
||||
def test_ax_lean_overlay_removes_observability_and_replaces_export_config() -> None:
|
||||
overlay = yaml.safe_load((OVERLAY / "kustomization.yaml").read_text())
|
||||
assert overlay["resources"] == ["../kind-upstream"]
|
||||
deletes = [p for p in overlay["patches"] if "target" in p]
|
||||
assert [p["target"] for p in deletes] == [
|
||||
{"namespace": "otel-system"},
|
||||
{"kind": "Namespace", "name": "otel-system"},
|
||||
{"kind": "ClusterRole", "name": "ate-prometheus"},
|
||||
{"kind": "RoleBinding", "name": "ate-prometheus", "namespace": "ate-system"},
|
||||
]
|
||||
assert all(yaml.safe_load(p["patch"])["$patch"] == "delete" for p in deletes)
|
||||
config = yaml.safe_load((OVERLAY / overlay["patches"][-1]["path"]).read_text())
|
||||
assert config["metadata"] == {"name": "ate-otel-config", "namespace": "ate-system"}
|
||||
# Whole-data replacement drops the upstream endpoint, interval and timeout,
|
||||
# including any future keys that would otherwise survive a merge.
|
||||
assert config["data"] == {"$patch": "replace", **DISABLED}
|
||||
|
||||
|
||||
def test_ax_staging_preserves_upstream_and_installer_prerequisites(tmp_path: Path) -> None:
|
||||
prepare = runpy.run_path(str(ROOT / "ops/ax/prepare-substrate.py"))["prepare"]
|
||||
source = tmp_path / "source"
|
||||
kind = source / "manifests/ate-install/kind"
|
||||
kind.mkdir(parents=True)
|
||||
original = "resources: [otel-collector.yaml, prometheus.yaml, ate-otel-config.yaml]\n"
|
||||
(kind / "kustomization.yaml").write_text(original)
|
||||
(kind / "ate-otel-config.yaml").write_text("upstream config")
|
||||
(kind / "postgres").mkdir()
|
||||
(kind / "postgres/kustomization.yaml").write_text("resources: [../../postgres]\n")
|
||||
(source / "go.mod").write_text("module test\n")
|
||||
(source / "hack").mkdir()
|
||||
(source / "hack/install-ate-kind.sh").write_text("upstream installer")
|
||||
stage = tmp_path / "stage"
|
||||
prepare(source, stage)
|
||||
install = stage / "manifests/ate-install"
|
||||
assert (install / "kind-upstream/kustomization.yaml").read_text() == original
|
||||
assert (install / "kind/postgres/kustomization.yaml").read_text() == "resources: [../../postgres]\n"
|
||||
assert (kind / "kustomization.yaml").read_text() == original
|
||||
assert (kind / "ate-otel-config.yaml").read_text() == "upstream config"
|
||||
assert (stage / "hack/install-ate-kind.sh").read_text() == "upstream installer"
|
||||
assert (stage / "go.mod").is_symlink()
|
||||
config = yaml.safe_load((install / "kind/ate-otel-config.yaml").read_text())
|
||||
assert config["data"] == DISABLED # direct installer apply must not contain $patch
|
||||
overlay = yaml.safe_load((install / "kind/kustomization.yaml").read_text())
|
||||
patch = yaml.safe_load((install / "kind" / overlay["patches"][-1]["path"]).read_text())
|
||||
assert patch["data"] == {"$patch": "replace", **DISABLED}
|
||||
with pytest.raises(ValueError, match="must be empty"):
|
||||
prepare(source, stage)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("enabled", [False, True])
|
||||
@pytest.mark.parametrize("existing", [False, True])
|
||||
def test_ax_reconciles_observability_on_ready_clusters(tmp_path: Path, enabled: bool, existing: bool) -> None:
|
||||
"""Execute the reconciliation shell with a function double, never real kubectl."""
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
script = (ROOT / "ops/ax/up.sh").read_text()
|
||||
section = script.split("# Reconcile the choice", 1)[1].split('kubectl --context "$CTX" -n ate-system get pods', 1)[0]
|
||||
section = "# Reconcile the choice" + section
|
||||
source = tmp_path / "substrate/manifests/ate-install/kind"
|
||||
source.mkdir(parents=True)
|
||||
upstream_data = {"OTEL_EXPORTER_OTLP_ENDPOINT": "http://collector:4317", "OTEL_LOGS_EXPORTER": "otlp"}
|
||||
(source / "ate-otel-config.yaml").write_text(yaml.safe_dump({"data": upstream_data}))
|
||||
log = tmp_path / "calls"
|
||||
consumer = {
|
||||
"kind": "Deployment",
|
||||
"metadata": {"name": "ate-controller"},
|
||||
"spec": {"template": {"spec": {"containers": [{"envFrom": [{"configMapRef": {"name": "ate-otel-config"}}]}]}}},
|
||||
}
|
||||
nonconsumer = {"kind": "Deployment", "metadata": {"name": "unrelated"}, "spec": {"template": {"spec": {"containers": []}}}}
|
||||
mock = """
|
||||
set -euo pipefail
|
||||
kubectl() {
|
||||
printf '%s\\t' "$@" >> "$CALL_LOG"
|
||||
printf '\\n' >> "$CALL_LOG"
|
||||
case " $* " in
|
||||
*" get deployment "*)
|
||||
if [ "$EXISTING" = 1 ]; then printf 'deployment.apps/%s\\n' "$7"; fi ;;
|
||||
*" get deployments,daemonsets "*) printf '%s' "$WORKLOADS" ;;
|
||||
esac
|
||||
return 0
|
||||
}
|
||||
"""
|
||||
env = {
|
||||
**os.environ,
|
||||
"OBSERVABILITY": str(enabled).lower(),
|
||||
"EXISTING": str(int(existing)),
|
||||
"PY": sys.executable,
|
||||
"CTX": "test-context",
|
||||
"SRC": str(tmp_path),
|
||||
"REPO_ROOT": str(ROOT),
|
||||
"CALL_LOG": str(log),
|
||||
"WORKLOADS": json.dumps({"items": [consumer, nonconsumer]}),
|
||||
}
|
||||
result = subprocess.run(["bash", "-c", mock + section], env=env, text=True, capture_output=True)
|
||||
assert result.returncode == 0, result.stderr
|
||||
calls = [line.rstrip("\t").split("\t") for line in log.read_text().splitlines()]
|
||||
scales = [call for call in calls if "scale" in call]
|
||||
assert len(scales) == (3 if existing and not enabled else 0)
|
||||
assert all(call[-1] == "--replicas=0" for call in scales)
|
||||
applies = [call for call in calls if "apply" in call]
|
||||
assert len(applies) == (2 if enabled else 0)
|
||||
config_patch = next(call[-1] for call in calls if "configmap" in call)
|
||||
assert json.loads(config_patch) == [{"op": "replace", "path": "/data", "value": upstream_data if enabled else DISABLED}]
|
||||
assert not any("deployment/unrelated" in call for call in calls)
|
||||
assert any("deployment/ate-controller" in call for call in calls)
|
||||
if not enabled:
|
||||
assert "OUSAST_KIND_OBSERVABILITY=1" in result.stdout
|
||||
@@ -20,6 +20,7 @@ host's pytest mounted in, which needs no network and installs nothing:
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
@@ -28,6 +29,18 @@ import pytest
|
||||
QUERIES = Path("src/openultrasast/cpg/queries")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("name", ["taint", "dominance", "config", "census", "overlay", "vocabulary"])
|
||||
def test_query_scope_checks_guard_nullable_methods(name: str) -> None:
|
||||
"""PHP class constants can have no method; one unsafe scope check aborts the batch."""
|
||||
script = QUERIES / f"{name}.sc"
|
||||
source = script.read_text()
|
||||
assert source, f"{script} is empty"
|
||||
# Require Option(node.method).exists(predicate), keeping the null guard at the call site.
|
||||
unsafe = re.compile(r"\b(?:inScope|nestedInLabeled)\s*\(\s*[\w.]+\s*\.\s*method\s*\)")
|
||||
sites = [source.count("\n", 0, match.start()) + 1 for match in unsafe.finditer(source)]
|
||||
assert not sites, f"{script}: unguarded method scope checks at lines {sites}"
|
||||
|
||||
|
||||
def _joern() -> str | None:
|
||||
import shutil
|
||||
|
||||
|
||||
@@ -343,3 +343,42 @@ def test_a_startup_wait_is_bounded_by_its_allowance_not_the_deadline(tmp_path, f
|
||||
# Bounded by the allowance, nowhere near the 600-second deadline.
|
||||
assert time.monotonic() - started < 10.0
|
||||
live.close()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("cancel", [False, True])
|
||||
def test_live_receipt_observation_without_socket_or_jvm(tmp_path, monkeypatch, cancel):
|
||||
import threading
|
||||
from types import SimpleNamespace
|
||||
|
||||
live = EngineSession(tmp_path, ExecutionBudget(time.monotonic() + 10, 1), {})
|
||||
live._process = SimpleNamespace(poll=lambda: None)
|
||||
stopped = threading.Event()
|
||||
observed = []
|
||||
|
||||
def close():
|
||||
live._process = None
|
||||
stopped.set()
|
||||
|
||||
def post(code, **kwargs):
|
||||
target = code.split('java.nio.file.Path.of("', 1)[1].split('"', 1)[0]
|
||||
nonce = code.split('__ousastPath, "', 1)[1].split("\\n", 1)[0]
|
||||
Path(target).write_text(nonce + '\n{"__question__":"a"}\n')
|
||||
if cancel:
|
||||
assert stopped.wait(2)
|
||||
return {"success": True}
|
||||
|
||||
def observe(output):
|
||||
observed.append(output)
|
||||
return not cancel
|
||||
|
||||
monkeypatch.setattr(live, "_post", post)
|
||||
monkeypatch.setattr(live, "close", close)
|
||||
live.output_observer = observe
|
||||
answer = live.evaluate("work()", timeout=5)
|
||||
assert any("__question__" in output for output in observed)
|
||||
if cancel:
|
||||
assert answer is None and live.failure == "session_observer_cancelled"
|
||||
assert stopped.is_set()
|
||||
else:
|
||||
assert answer.stdout == '{"__question__":"a"}\n'
|
||||
assert not list(tmp_path.glob("answer-*.txt"))
|
||||
|
||||
@@ -0,0 +1,291 @@
|
||||
"""Offline queue protocol, image imports and deployment contract controls."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
import yaml
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
with pytest.MonkeyPatch.context() as patch:
|
||||
patch.syspath_prepend(str(ROOT / "benchmarks/learn"))
|
||||
service = importlib.import_module("engine_worker_service")
|
||||
transport = importlib.import_module("engine_trace_queue")
|
||||
|
||||
|
||||
class FakeS3:
|
||||
def __init__(self):
|
||||
self.objects = {}
|
||||
self.lock = threading.Lock()
|
||||
self.choosing = None
|
||||
|
||||
def get(self, key):
|
||||
with self.lock:
|
||||
return (self.objects[key], None) if key in self.objects else None
|
||||
|
||||
def put(self, key, data, labels):
|
||||
with self.lock:
|
||||
self.objects[key] = data
|
||||
if self.choosing and key.startswith(service.CLAIMED) and json.loads(data)["ticket"] == 0:
|
||||
self.choosing.wait(timeout=5)
|
||||
|
||||
def delete(self, key):
|
||||
with self.lock:
|
||||
self.objects.pop(key, None)
|
||||
|
||||
def keys(self, prefix):
|
||||
with self.lock:
|
||||
return sorted(k for k in self.objects if k.startswith(prefix))
|
||||
|
||||
|
||||
def task():
|
||||
return dict(
|
||||
run="run",
|
||||
id="pin",
|
||||
task_id="run-pin",
|
||||
input="engine-queue/inputs/run/pin/source.dat",
|
||||
deadline=10,
|
||||
question_deadline=2,
|
||||
pin={"repo": "owner/repo", "pin": "commit", "units": []},
|
||||
)
|
||||
|
||||
|
||||
def test_engine_worker_racing_claimants_run_exactly_once():
|
||||
store = FakeS3()
|
||||
item = task()
|
||||
service.put(store, service.PENDING + item["task_id"], item)
|
||||
store.choosing = threading.Barrier(2)
|
||||
called = []
|
||||
|
||||
def execute(client, entry):
|
||||
called.append(entry["task_id"])
|
||||
time.sleep(0.05)
|
||||
return {**entry["pin"], "done": True, "instrument": {"bytes": 11}}
|
||||
|
||||
with ThreadPoolExecutor(2) as pool:
|
||||
futures = [pool.submit(service.Queue(store, identity).process_one, execute) for identity in ["worker-b", "worker-a"]]
|
||||
results = [f.result(timeout=10) for f in futures]
|
||||
assert sum(results) == 1
|
||||
assert called == ["run-pin"]
|
||||
result = service.read(store, service.result_key(item))
|
||||
assert result["worker"] in {"worker-a", "worker-b"}
|
||||
assert not store.keys(service.CLAIMED)
|
||||
assert not store.keys(service.PENDING)
|
||||
|
||||
|
||||
def test_engine_worker_expired_lease_recovered_without_pending():
|
||||
store = FakeS3()
|
||||
item = task()
|
||||
service.put(store, "engine-queue/claimed/dead/run-pin", dict(task=item, ticket=1, expires=99))
|
||||
queue = service.Queue(store, "new", now=lambda: 100)
|
||||
assert queue.process_one(lambda *_: {**item["pin"], "done": True})
|
||||
assert service.read(store, service.result_key(item))["worker"] == "new"
|
||||
assert not store.keys(service.CLAIMED)
|
||||
|
||||
|
||||
def test_engine_worker_late_arrival_cannot_displace_running_owner():
|
||||
store = FakeS3()
|
||||
item = task()
|
||||
first = service.Queue(store, "z-first")
|
||||
key = first.claim(item)
|
||||
assert key
|
||||
stop = threading.Event()
|
||||
later = service.Queue(store, "a-later", stop=stop)
|
||||
with ThreadPoolExecutor(1) as pool:
|
||||
future = pool.submit(later.claim, item)
|
||||
assert future.result(timeout=3) is None
|
||||
assert service.read(store, key) is not None
|
||||
service.put(store, service.result_key(item), {"done": True})
|
||||
store.delete(key)
|
||||
|
||||
|
||||
def test_engine_queue_dispatch_roundtrip_and_timeout(tmp_path):
|
||||
store = FakeS3()
|
||||
checkout, output = tmp_path / "case", tmp_path / "out"
|
||||
checkout.mkdir()
|
||||
output.mkdir()
|
||||
(checkout / "app.py").write_text("print(1)\n")
|
||||
args = SimpleNamespace(deadline=10, question_deadline=2, queue_timeout=2, image="image@sha256:abc")
|
||||
unit = {"unit": "u", "supported": True, "reason": "", "family": "injection"}
|
||||
pin = {"repo": "owner/repo", "pin": "sha", "units": [unit], "questions": []}
|
||||
|
||||
def execute(client, item):
|
||||
assert item["input"].endswith("/source.dat")
|
||||
unpacked = tmp_path / "unpacked"
|
||||
unpacked.mkdir()
|
||||
service.unpack(client.get(item["input"])[0], unpacked)
|
||||
assert (unpacked / "app.py").read_bytes() == b"print(1)\n"
|
||||
return {
|
||||
**item["pin"],
|
||||
"done": True,
|
||||
"container_exit": 0,
|
||||
"units": [service.unit_record(unit, "asked-nothing")],
|
||||
"instrument": {"bytes": 9, "heap_mb": 2560},
|
||||
}
|
||||
|
||||
queue = service.Queue(store, "pod-1")
|
||||
dispatcher = transport.Dispatcher(args, store, pause=lambda _: queue.process_one(execute))
|
||||
completed, worker = dispatcher.execute(checkout, output, pin, "pin")
|
||||
assert completed.returncode == 0
|
||||
assert worker == "pod-1"
|
||||
result = json.loads((output / "result.json").read_text())
|
||||
assert result["executor"] == "queue"
|
||||
assert result["units"][0]["status"] == "asked-nothing"
|
||||
assert result["instrument"]["heap_mb"] == 2560
|
||||
assert dispatcher.status() == {"pending": 0, "claimed": 0, "done": 1}
|
||||
assert not store.keys("engine-queue/inputs/")
|
||||
ticks = iter([0, 3])
|
||||
dispatcher = transport.Dispatcher(args, store, clock=lambda: next(ticks))
|
||||
assert dispatcher.execute(checkout, output, pin, "other") == (None, None)
|
||||
assert not store.keys(service.PENDING)
|
||||
|
||||
|
||||
def test_engine_worker_imports_without_yaml():
|
||||
code = "import sys; sys.modules['yaml'] = None; import engine_worker_service; import engine_trace_worker"
|
||||
done = subprocess.run(
|
||||
[sys.executable, "-c", code],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env={**os.environ, "PYTHONPATH": f"{ROOT / 'src'}:{ROOT / 'benchmarks/learn'}"},
|
||||
)
|
||||
assert done.returncode == 0, done.stderr
|
||||
|
||||
|
||||
def test_engine_k8s_render_contract():
|
||||
base = ROOT / "ops/k8s/engine"
|
||||
config = yaml.safe_load((base / "kustomization.yaml").read_text())
|
||||
# Render this base's resource inclusion, namespace, image and replacement transforms offline.
|
||||
resources = [yaml.safe_load((base / name).read_text()) for name in config["resources"]]
|
||||
deployment = next(r for r in resources if r["kind"] == "Deployment")
|
||||
deployment["metadata"]["namespace"] = config["namespace"]
|
||||
image = config["images"][0]
|
||||
pod = deployment["spec"]["template"]["spec"]
|
||||
container = pod["containers"][0]
|
||||
assert container["image"] == image["name"]
|
||||
container["image"] = image["newName"] + "@" + image["digest"]
|
||||
replacement = config["replacements"][0]
|
||||
assert replacement["source"]["fieldPath"] == "spec.template.spec.containers.0.image"
|
||||
assert replacement["targets"][0]["fieldPaths"] == ["spec.template.spec.containers.0.env.0.value"]
|
||||
container["env"][0]["value"] = container["image"]
|
||||
assert "@sha256:" in container["image"]
|
||||
assert deployment["spec"]["replicas"] == 2
|
||||
assert container["resources"] == {"requests": {"memory": "3Gi", "cpu": "1"}, "limits": {"memory": "3Gi"}}
|
||||
affinity = pod["affinity"]["podAntiAffinity"]["requiredDuringSchedulingIgnoredDuringExecution"][0]
|
||||
assert affinity["topologyKey"] == "kubernetes.io/hostname"
|
||||
assert affinity["labelSelector"]["matchLabels"] == deployment["spec"]["template"]["metadata"]["labels"]
|
||||
assert pod["terminationGracePeriodSeconds"] == 900
|
||||
assert container["envFrom"] == [{"secretRef": {"name": "ousast-engine-s3"}}]
|
||||
for probe in ["readinessProbe", "livenessProbe"]:
|
||||
assert container[probe]["httpGet"] == {"path": "/healthz", "port": "health"}
|
||||
assert not pod["automountServiceAccountToken"]
|
||||
assert pod["securityContext"]["runAsNonRoot"]
|
||||
assert "hostPath" not in json.dumps(resources)
|
||||
assert "privileged" not in json.dumps(resources)
|
||||
assert all(r["kind"] != "Secret" for r in resources)
|
||||
secret = yaml.safe_load((base / "secret.example.yaml").read_text())
|
||||
assert secret["stringData"] == {}
|
||||
assert "AWS_SECRET_ACCESS_KEY" not in json.dumps(resources)
|
||||
|
||||
|
||||
def test_engine_image_and_rustfs_policy():
|
||||
dockerfile = (ROOT / "Dockerfile").read_text()
|
||||
assert "COPY benchmarks ./benchmarks" in dockerfile
|
||||
assert ".[semantic,s3]" in dockerfile
|
||||
for name in ["engine_trace_worker.py", "engine_trace_parse.py", "engine_worker_service.py"]:
|
||||
assert (ROOT / "benchmarks/learn" / name).stat().st_size > 0
|
||||
doc = (ROOT / "docs/rustfs.md").read_text().split("## Scoped account for engine workers")[1]
|
||||
policy = json.loads(doc.split("```json\n")[1].split("```")[0])
|
||||
listing, objects = policy["Statement"]
|
||||
assert listing["Action"] == "s3:ListBucket"
|
||||
assert set(listing["Condition"]["StringLike"]["s3:prefix"]) == {
|
||||
"engine-queue/",
|
||||
"engine-queue/*",
|
||||
"engine-results/",
|
||||
"engine-results/*",
|
||||
}
|
||||
assert set(objects["Action"]) == {"s3:GetObject", "s3:PutObject", "s3:DeleteObject"}
|
||||
assert set(objects["Resource"]) == {"arn:aws:s3:::sast-memory/engine-queue/*", "arn:aws:s3:::sast-memory/engine-results/*"}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("hang", [False, True])
|
||||
def test_engine_worker_process_deadline_and_shipped_schema(tmp_path, monkeypatch, hang):
|
||||
store = FakeS3()
|
||||
item = task()
|
||||
item["image"] = "image@sha256:abc"
|
||||
item["deadline"] = 0.1 if hang else 5
|
||||
item["pin"]["units"] = [{"unit": "u", "supported": True, "reason": ""}]
|
||||
(tmp_path / "app.py").write_text("print(1)\n")
|
||||
store.put(item["input"], service.pack(tmp_path), {})
|
||||
monkeypatch.setenv("ENGINE_IMAGE", item["image"])
|
||||
|
||||
def fake_worker(pin, root, out, deadline, question_deadline):
|
||||
assert (root / "app.py").read_bytes() == b"print(1)\n"
|
||||
if hang:
|
||||
time.sleep(10)
|
||||
out.write_text(json.dumps({**pin, "done": True, "units": [service.unit_record(pin["units"][0], "asked-nothing")]}))
|
||||
|
||||
monkeypatch.setattr(service, "worker", fake_worker)
|
||||
result = service.execute_task(store, item)
|
||||
assert result["done"]
|
||||
assert result["units"][0]["status"] == ("timeout" if hang else "asked-nothing")
|
||||
assert result["container_exit"] == (None if hang else 0)
|
||||
assert result["seconds"] < 5
|
||||
|
||||
|
||||
def test_engine_worker_stopping_does_not_claim():
|
||||
store = FakeS3()
|
||||
item = task()
|
||||
service.put(store, service.PENDING + item["task_id"], item)
|
||||
stop = threading.Event()
|
||||
stop.set()
|
||||
assert not service.Queue(store, stop=stop).process_one(lambda *_: pytest.fail("claimed after SIGTERM"))
|
||||
assert store.keys(service.PENDING)
|
||||
assert not store.keys(service.CLAIMED)
|
||||
|
||||
|
||||
def test_engine_worker_rejects_archive_links_and_escape(tmp_path):
|
||||
import io
|
||||
import tarfile
|
||||
|
||||
for name, kind in [("../escape", tarfile.REGTYPE), ("link", tarfile.SYMTYPE)]:
|
||||
stream = io.BytesIO()
|
||||
with tarfile.open(fileobj=stream, mode="w") as archive:
|
||||
member = tarfile.TarInfo(name)
|
||||
member.type = kind
|
||||
member.linkname = "/etc/passwd"
|
||||
archive.addfile(member)
|
||||
with pytest.raises(ValueError, match="unsafe source archive"):
|
||||
service.unpack(stream.getvalue(), tmp_path)
|
||||
|
||||
|
||||
def test_engine_worker_timeout_kills_separate_session_descendant(tmp_path, monkeypatch):
|
||||
store = FakeS3()
|
||||
item = task()
|
||||
item.update(image="image@sha256:abc", deadline=0.5)
|
||||
item["pin"]["units"] = [{"unit": "u", "supported": True, "reason": ""}]
|
||||
(tmp_path / "app.py").write_text("print(1)\n")
|
||||
store.put(item["input"], service.pack(tmp_path), {})
|
||||
monkeypatch.setenv("ENGINE_IMAGE", item["image"])
|
||||
pid_file = tmp_path / "escaped-pid"
|
||||
|
||||
def fake_worker(*_):
|
||||
child = subprocess.Popen([sys.executable, "-c", "import time; time.sleep(30)"], start_new_session=True)
|
||||
pid_file.write_text(str(child.pid))
|
||||
time.sleep(30)
|
||||
|
||||
monkeypatch.setattr(service, "worker", fake_worker)
|
||||
result = service.execute_task(store, item)
|
||||
assert result["units"][0]["status"] == "timeout"
|
||||
assert pid_file.exists(), "instrument: fake worker never launched its descendant"
|
||||
pid = int(pid_file.read_text())
|
||||
assert not Path(f"/proc/{pid}").exists(), "escaped session survived the task deadline"
|
||||
@@ -0,0 +1,234 @@
|
||||
"""Offline AX worker entry controls; HTTP is loopback only."""
|
||||
|
||||
import importlib.util
|
||||
import io
|
||||
import json
|
||||
import tarfile
|
||||
import threading
|
||||
import time
|
||||
import urllib.error
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
SCRIPTS = Path(__file__).resolve().parents[1] / "benchmarks/learn"
|
||||
|
||||
|
||||
def load(name):
|
||||
with pytest.MonkeyPatch.context() as patch:
|
||||
patch.syspath_prepend(str(SCRIPTS))
|
||||
spec = importlib.util.spec_from_file_location(name, SCRIPTS / f"{name}.py")
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def test_engine_cluster_heap_flags(monkeypatch):
|
||||
worker = load("engine_trace_worker")
|
||||
monkeypatch.setenv("JAVA_TOOL_OPTIONS", "-Xmx9999m")
|
||||
backend = worker.MeasuredBackend(30, 10, lambda: None, heap_profile="cluster")
|
||||
flags = (
|
||||
"-Xms1700m -Xmx1700m -XX:MaxMetaspaceSize=256m -XX:ReservedCodeCacheSize=128m "
|
||||
"-XX:MaxDirectMemorySize=256m -Xss512k -XX:ActiveProcessorCount=2 -XX:+UseG1GC"
|
||||
)
|
||||
assert backend._jvm_env()["JAVA_TOOL_OPTIONS"] == flags
|
||||
assert backend._heap_flag() == "-J-Xmx1700m"
|
||||
assert worker.MeasuredBackend(30, 10, lambda: None)._heap_flag() == "-J-Xmx2560m"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("readiness", ["ready", "delayed", "closed", "source_denied"])
|
||||
@pytest.mark.parametrize("failure", [False, True])
|
||||
@pytest.mark.parametrize("transport", ["memory", "loopback"])
|
||||
def test_task_entry_http_roundtrip(tmp_path, monkeypatch, failure, transport, readiness):
|
||||
entry = load("engine_task_entry")
|
||||
archive = io.BytesIO()
|
||||
with tarfile.open(fileobj=archive, mode="w") as tar:
|
||||
data = b"print('input proof')\n"
|
||||
info = tarfile.TarInfo("app.py")
|
||||
info.size = len(data)
|
||||
tar.addfile(info, io.BytesIO(data))
|
||||
pin = {"pin": "abc", "units": [{"unit": "u", "supported": True}], "questions": {"u": {}}}
|
||||
uploads = []
|
||||
gets = []
|
||||
puts = []
|
||||
started = time.monotonic()
|
||||
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
def do_GET(self):
|
||||
gets.append((self.path, time.monotonic() - started))
|
||||
if readiness == "source_denied" and self.path == "/source":
|
||||
self.send_response(403)
|
||||
self.end_headers()
|
||||
return
|
||||
if readiness == "closed" or (readiness == "delayed" and gets[-1][1] < 3):
|
||||
self.send_response(403 if len(gets) % 2 else 503)
|
||||
self.end_headers()
|
||||
return
|
||||
data = archive.getvalue() if self.path == "/source" else json.dumps(pin).encode()
|
||||
self.send_response(200)
|
||||
self.end_headers()
|
||||
self.wfile.write(data)
|
||||
|
||||
def do_PUT(self):
|
||||
body = self.rfile.read(int(self.headers["Content-Length"]))
|
||||
puts.append(time.monotonic())
|
||||
if readiness == "closed" and len(puts) == 1:
|
||||
self.send_response(403)
|
||||
self.end_headers()
|
||||
return
|
||||
uploads.append(body)
|
||||
self.send_response(200)
|
||||
self.end_headers()
|
||||
|
||||
def log_message(self, *args):
|
||||
pass
|
||||
|
||||
fake_worker = tmp_path / "worker.py"
|
||||
fake_worker.write_text(
|
||||
"import argparse,json,pathlib,sys\np=argparse.ArgumentParser()\n"
|
||||
"for key in ['worker','root','output','deadline','question-deadline','heap-profile']: p.add_argument('--'+key)\n"
|
||||
"a=p.parse_args()\nassert a.heap_profile == 'cluster'\n"
|
||||
"assert (pathlib.Path(a.root)/'app.py').stat().st_size > 0\n"
|
||||
+ ("sys.exit(7)\n" if failure else "pathlib.Path(a.output).write_text(json.dumps({'done':True,'units':[]}))\n")
|
||||
)
|
||||
if transport == "loopback":
|
||||
try:
|
||||
server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
|
||||
except PermissionError:
|
||||
pytest.skip("sandbox prohibits loopback sockets; in-memory HTTP roundtrip still runs")
|
||||
thread = threading.Thread(target=server.serve_forever, daemon=True)
|
||||
thread.start()
|
||||
base = f"http://127.0.0.1:{server.server_port}"
|
||||
else:
|
||||
base = "https://fake-store.invalid"
|
||||
|
||||
def urlopen(request, timeout):
|
||||
method = request.get_method() if hasattr(request, "get_method") else "GET"
|
||||
url = request.full_url if hasattr(request, "full_url") else request
|
||||
body = request.data or b"" if method == "PUT" else b""
|
||||
raw = f"{method} {url.removeprefix(base)} HTTP/1.0\r\nContent-Length: {len(body)}\r\n\r\n".encode() + body
|
||||
|
||||
class Socket:
|
||||
def __init__(self):
|
||||
self.sent = bytearray()
|
||||
|
||||
def makefile(self, *args):
|
||||
return io.BytesIO(raw)
|
||||
|
||||
def sendall(self, data):
|
||||
self.sent.extend(data)
|
||||
|
||||
connection = Socket()
|
||||
Handler(connection, ("127.0.0.1", 0), None)
|
||||
headers, payload = bytes(connection.sent).split(b"\r\n\r\n", 1)
|
||||
status = int(headers.split()[1])
|
||||
if status != 200:
|
||||
raise urllib.error.HTTPError(url, status, "denied", {}, io.BytesIO(payload))
|
||||
return io.BytesIO(payload)
|
||||
|
||||
monkeypatch.setattr(entry.urllib.request, "urlopen", urlopen)
|
||||
try:
|
||||
for name, suffix in [("SOURCE_URL", "source"), ("QUESTIONS_URL", "questions"), ("RESULT_URL", "result")]:
|
||||
monkeypatch.setenv(name, base + "/" + suffix)
|
||||
budget = 0.7 if readiness == "closed" else 60
|
||||
assert entry.main(work_root=tmp_path / "work", worker_script=fake_worker, readiness_budget=budget) == 0
|
||||
if readiness == "delayed":
|
||||
assert len(gets) >= 5
|
||||
assert gets[-1][1] >= 3
|
||||
if readiness == "closed":
|
||||
assert 0.7 <= puts[0] - started < 1.5
|
||||
assert len(puts) == 2
|
||||
assert puts[1] - puts[0] >= 0.5
|
||||
assert all(path == "/questions" for path, _ in gets)
|
||||
finally:
|
||||
if transport == "loopback":
|
||||
server.shutdown()
|
||||
thread.join()
|
||||
server.server_close()
|
||||
assert len(uploads) == 1
|
||||
with tarfile.open(fileobj=io.BytesIO(uploads[0])) as tar:
|
||||
result = json.load(tar.extractfile("result.json"))
|
||||
assert result["done"]
|
||||
if readiness == "closed":
|
||||
assert result["status"] == "failed"
|
||||
assert result["reason"] == "GET questions: egress not open after 60 s"
|
||||
assert json.loads((tmp_path / "work/out/result.json").read_text()) == result
|
||||
elif readiness == "source_denied":
|
||||
assert result["status"] == "failed"
|
||||
assert result["reason"] == "GET source: HTTP 403"
|
||||
assert result["units"][0]["reason"] == "GET source: HTTP 403"
|
||||
assert base not in json.dumps(result)
|
||||
assert json.loads((tmp_path / "work/out/result.json").read_text()) == result
|
||||
elif failure:
|
||||
assert result["units"][0]["status"] == "failed"
|
||||
assert "7" in result["units"][0]["reason"]
|
||||
|
||||
|
||||
def test_engine_cluster_profile_recorded_on_input_failure(tmp_path):
|
||||
worker = load("engine_trace_worker")
|
||||
pin = {
|
||||
"units": [{"unit": "u", "file": "missing.py", "language": "python", "supported": True, "reason": ""}],
|
||||
"questions": {"u": {}},
|
||||
}
|
||||
result = worker.worker(pin, tmp_path, tmp_path / "result.json", 30, 10, heap_profile="cluster")
|
||||
assert result["units"][0]["status"] == "failed"
|
||||
assert result["instrument"]["heap_profile"] == "cluster"
|
||||
assert result["instrument"]["heap_mb"] == 1700
|
||||
assert result["instrument"]["java_tool_options"] == worker.CLUSTER_JVM_FLAGS
|
||||
|
||||
|
||||
def test_task_entry_rejects_tar_traversal(tmp_path):
|
||||
entry = load("engine_task_entry")
|
||||
source = tmp_path / "source.tar"
|
||||
with tarfile.open(source, "w") as archive:
|
||||
member = tarfile.TarInfo("../escape")
|
||||
member.size = 1
|
||||
archive.addfile(member, io.BytesIO(b"x"))
|
||||
with pytest.raises(ValueError, match="unsafe"):
|
||||
entry.unpack(source, tmp_path / "case")
|
||||
assert not (tmp_path / "escape").exists()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("error", [urllib.error.URLError("refused"), ConnectionResetError("reset")])
|
||||
def test_readiness_connection_errors_back_off_to_cap(tmp_path, monkeypatch, error):
|
||||
entry = load("engine_task_entry")
|
||||
now = [0.0]
|
||||
pauses = []
|
||||
timeouts = []
|
||||
|
||||
def pause(seconds):
|
||||
pauses.append(seconds)
|
||||
now[0] += seconds
|
||||
|
||||
def refused(request, timeout):
|
||||
timeouts.append(timeout)
|
||||
raise error
|
||||
|
||||
monkeypatch.setattr(entry.time, "monotonic", lambda: now[0])
|
||||
monkeypatch.setattr(entry.time, "sleep", pause)
|
||||
monkeypatch.setattr(entry.urllib.request, "urlopen", refused)
|
||||
with pytest.raises(RuntimeError, match="^egress not open after 60 s$"):
|
||||
entry.download("https://fake.invalid", tmp_path / "input", readiness_budget=60)
|
||||
assert now[0] == 60
|
||||
assert pauses[:5] == [0.5, 1, 2, 4, 8]
|
||||
assert max(pauses) == 8
|
||||
assert timeouts[-1] == pauses[-1]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("status, attempts", [(403, 3), (503, 3), (404, 1)])
|
||||
def test_later_transfers_keep_short_retry(monkeypatch, status, attempts):
|
||||
entry = load("engine_task_entry")
|
||||
calls = []
|
||||
pauses = []
|
||||
|
||||
def refused(request, timeout):
|
||||
calls.append(timeout)
|
||||
raise urllib.error.HTTPError(request, status, "denied", {}, io.BytesIO())
|
||||
|
||||
monkeypatch.setattr(entry.time, "sleep", pauses.append)
|
||||
monkeypatch.setattr(entry.urllib.request, "urlopen", refused)
|
||||
with pytest.raises(urllib.error.HTTPError):
|
||||
entry.transfer("https://fake.invalid")
|
||||
assert len(calls) == attempts
|
||||
assert pauses == list(range(1, attempts))
|
||||
File diff suppressed because it is too large
Load Diff
@@ -8,6 +8,8 @@ anywhere else, which is how an evaluation set quietly becomes a development one.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
import tomllib
|
||||
from pathlib import Path
|
||||
@@ -64,17 +66,131 @@ def test_a_multi_language_repository_is_reserved(path: Path) -> None:
|
||||
assert members and all(by_id[m]["language"] == "php" and "javascript" in by_id[m].get("also", []) for m in members)
|
||||
|
||||
|
||||
def test_no_reserved_repository_is_referenced_by_the_development_tree() -> None:
|
||||
def _pool_manifests() -> list[Path]:
|
||||
directory = ROOT / "benchmarks/unseen"
|
||||
return sorted([*directory.glob("*.toml"), *directory.glob("private/*.toml")])
|
||||
|
||||
|
||||
def _pool_names() -> set[str]:
|
||||
names = set()
|
||||
for path in _pool_manifests():
|
||||
text = path.read_text().lower()
|
||||
names.update(m.rstrip("/").removesuffix(".git") for m in re.findall(r"github\.com/([\w.-]+/[\w.-]+)", text))
|
||||
return names
|
||||
|
||||
|
||||
def _population_names() -> set[str]:
|
||||
reserved = {case["repo"].rstrip("/").lower() for path in POPULATIONS for case in _population(path)["case"]}
|
||||
names = {url.rsplit("/", 2)[-2] + "/" + url.rsplit("/", 1)[-1] for url in reserved}
|
||||
offenders = []
|
||||
for top in ("benchmarks", "src", "tests"):
|
||||
return {url.rsplit("/", 2)[-2] + "/" + url.rsplit("/", 1)[-1] for url in reserved}
|
||||
|
||||
|
||||
def test_no_reserved_repository_is_referenced_by_the_development_tree() -> None:
|
||||
reservations = [(ROOT / "benchmarks/independent", _population_names()), (ROOT / "benchmarks/unseen", _pool_names())]
|
||||
offenders = 0
|
||||
counts = {}
|
||||
for top in ("benchmarks", "src", "tests", "plane"):
|
||||
for path in (ROOT / top).rglob("*"):
|
||||
if not path.is_file() or POPULATION.parent in path.parents or path == Path(__file__) or path.suffix in {".pyc", ".bin", ".gz"}:
|
||||
if not path.is_file() or path == Path(__file__) or path.suffix in {".pyc", ".bin", ".gz"}:
|
||||
continue
|
||||
# Preserve the population guard's existing scope; plane is newly reserved for pools.
|
||||
active = [
|
||||
(directory, names)
|
||||
for directory, names in reservations
|
||||
if directory not in path.parents and names and not (top == "plane" and directory == ROOT / "benchmarks/independent")
|
||||
]
|
||||
if not active:
|
||||
continue
|
||||
try:
|
||||
text = path.read_text(errors="ignore").lower()
|
||||
except OSError:
|
||||
continue
|
||||
offenders += [f"{path.relative_to(ROOT)}: {name}" for name in names if f"github.com/{name}" in text]
|
||||
assert not offenders, "a reserved repository is referenced outside the reservation: " + "; ".join(offenders[:10])
|
||||
count = sum(1 for _, names in active for name in names if f"github.com/{name}" in text)
|
||||
offenders += count
|
||||
counts[top] = counts.get(top, 0) + count
|
||||
assert not offenders, f"reserved repository referenced outside its reservation: count={offenders}, sources={counts}"
|
||||
|
||||
|
||||
def test_pool_and_population_names_are_disjoint() -> None:
|
||||
count = len(_population_names() & _pool_names())
|
||||
assert not count, f"overlapping reservations: count={count}"
|
||||
|
||||
|
||||
def manifest_digest(path: Path) -> str:
|
||||
"""Canonical parsed TOML, UTF-8 JSON with sorted keys and compact separators.
|
||||
|
||||
Private entries are covered by their opaque pointers' digests in the public manifest.
|
||||
The freeze writer must use the same encoding (task 3.2).
|
||||
"""
|
||||
raw = json.dumps(tomllib.loads(path.read_text()), sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode()
|
||||
return hashlib.sha256(raw).hexdigest()
|
||||
|
||||
|
||||
def check_pool_digest(path: Path) -> None:
|
||||
record = path.with_name("freeze-" + path.stem.removeprefix("pool-") + ".json")
|
||||
expected = json.loads(record.read_text())["freeze_digest"]
|
||||
assert manifest_digest(path) == expected, "frozen pool digest mismatch"
|
||||
|
||||
|
||||
def test_frozen_pool_manifest_digest() -> None:
|
||||
pools = sorted((ROOT / "benchmarks/unseen").glob("pool-p*.toml"))
|
||||
if not pools:
|
||||
pytest.skip("no pool frozen")
|
||||
for path in pools:
|
||||
check_pool_digest(path)
|
||||
|
||||
|
||||
def test_pool_guard_controls(tmp_path, monkeypatch) -> None:
|
||||
"""A synthetic reservation proves green, then red outside its own directory."""
|
||||
population = tmp_path / "benchmarks/independent/population-v1.toml"
|
||||
pool = tmp_path / "benchmarks/unseen/pool-p1.toml"
|
||||
population.parent.mkdir(parents=True)
|
||||
pool.parent.mkdir(parents=True)
|
||||
prefix = "https://" + "github.com/"
|
||||
population.write_text('[[case]]\nrepo = "' + prefix + 'synthetic/population"\n')
|
||||
pool.write_text('[[repository]]\nrepo = "' + prefix + 'synthetic/pool"\n')
|
||||
monkeypatch.setattr(__import__(__name__), "ROOT", tmp_path)
|
||||
monkeypatch.setattr(__import__(__name__), "POPULATIONS", [population])
|
||||
test_no_reserved_repository_is_referenced_by_the_development_tree()
|
||||
leak = tmp_path / "benchmarks/measurements/fixture.txt"
|
||||
leak.parent.mkdir(parents=True)
|
||||
leak.write_text(prefix + "synthetic/pool")
|
||||
with pytest.raises(AssertionError, match="count=1"):
|
||||
test_no_reserved_repository_is_referenced_by_the_development_tree()
|
||||
leak.unlink()
|
||||
plane = tmp_path / "plane/fixture.txt"
|
||||
plane.parent.mkdir()
|
||||
plane.write_text(prefix + "synthetic/pool")
|
||||
with pytest.raises(AssertionError, match="count=1"):
|
||||
test_no_reserved_repository_is_referenced_by_the_development_tree()
|
||||
plane.write_text(prefix + "synthetic/population")
|
||||
test_no_reserved_repository_is_referenced_by_the_development_tree()
|
||||
plane.unlink()
|
||||
private = pool.parent / "private/draft-p1.toml"
|
||||
private.parent.mkdir()
|
||||
private.write_text('[[repository]]\nrepo = "' + prefix + 'synthetic/population"\n')
|
||||
with pytest.raises(AssertionError, match="count=2"):
|
||||
test_no_reserved_repository_is_referenced_by_the_development_tree()
|
||||
|
||||
|
||||
def test_pool_digest_control(tmp_path) -> None:
|
||||
pool = tmp_path / "pool-p1.toml"
|
||||
pool.write_text('status = "frozen"\n[[repository]]\nid = "opaque"\n')
|
||||
record = tmp_path / "freeze-p1.json"
|
||||
record.write_text(json.dumps({"freeze_digest": manifest_digest(pool)}))
|
||||
check_pool_digest(pool)
|
||||
pool.write_text(pool.read_text() + "slice = 2\n")
|
||||
with pytest.raises(AssertionError, match="digest"):
|
||||
check_pool_digest(pool)
|
||||
|
||||
|
||||
def test_frozen_pool_discovery_cannot_be_disabled(tmp_path, monkeypatch) -> None:
|
||||
pool = tmp_path / "benchmarks/unseen/pool-p1.toml"
|
||||
pool.parent.mkdir(parents=True)
|
||||
pool.write_text('status = "frozen"\n')
|
||||
record = pool.with_name("freeze-p1.json")
|
||||
record.write_text(json.dumps({"freeze_digest": manifest_digest(pool)}))
|
||||
monkeypatch.setattr(__import__(__name__), "ROOT", tmp_path)
|
||||
test_frozen_pool_manifest_digest()
|
||||
pool.write_text('status = "draft"\n')
|
||||
with pytest.raises(AssertionError, match="digest"):
|
||||
test_frozen_pool_manifest_digest()
|
||||
|
||||
@@ -0,0 +1,190 @@
|
||||
"""Synthetic, offline result snapshots; no analyzer, model or corpus access."""
|
||||
|
||||
import hashlib
|
||||
import importlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def audit_module(monkeypatch):
|
||||
monkeypatch.syspath_prepend(str(Path(__file__).resolve().parents[1] / "benchmarks/learn"))
|
||||
return importlib.import_module("joern_leak_audit")
|
||||
|
||||
|
||||
def units(count=12):
|
||||
return [
|
||||
{"unit": f"{pair}-{label}", "pair": str(pair), "label": label, "family": "injection", "group": "g", "fold": "f"}
|
||||
for pair in range(count)
|
||||
for label in (1, 0)
|
||||
]
|
||||
|
||||
|
||||
def write_result(root, module, rows, name="pin", *, done=True, version=None):
|
||||
path = root / f"{name}.json"
|
||||
path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"repo": "repo",
|
||||
"pin": name,
|
||||
"done": done,
|
||||
"materialization_version": module.materialization_version() if version is None else version,
|
||||
"units": rows,
|
||||
}
|
||||
)
|
||||
)
|
||||
return path
|
||||
|
||||
|
||||
def result(unit, status):
|
||||
return {
|
||||
**unit,
|
||||
"source": "authored",
|
||||
"status": status,
|
||||
"pin_role": "vulnerable" if unit["label"] else "fixed",
|
||||
"traces": [{"steps": [{"roles": ["source"]}, {"roles": ["sink"]}], "sanitized": False, "guard": {"text": "", "bounded": False}}]
|
||||
if status == "path"
|
||||
else [],
|
||||
"witness_rows": [{"sink": "execute(x)"}] if status == "path" else [],
|
||||
}
|
||||
|
||||
|
||||
def run(module, root, population):
|
||||
return module.audit(population, root, expected_rows=len(population), expected_pairs=len(population) // 2)
|
||||
|
||||
|
||||
def test_skips_stale_unfinished_and_proves_read(audit_module, tmp_path):
|
||||
population = units(2)
|
||||
paths = [
|
||||
write_result(tmp_path, audit_module, [result(population[0], "path")], "valid"),
|
||||
write_result(tmp_path, audit_module, [result(population[1], "path")], "stale", version=-1),
|
||||
write_result(tmp_path, audit_module, [result(population[2], "path")], "unfinished", done=False),
|
||||
write_result(tmp_path, audit_module, [result(population[3], "path")], "both", version=-1, done=False),
|
||||
]
|
||||
report = run(audit_module, tmp_path, population)
|
||||
evidence = report["inputs"]
|
||||
assert (evidence["stale"], evidence["unfinished"], evidence["skipped"]) == (2, 2, 3)
|
||||
assert evidence["files_read"] == 4 and evidence["accepted_files"] == evidence["result_rows_read"] == 1
|
||||
assert evidence["bytes_read"] == sum(p.stat().st_size for p in paths)
|
||||
assert report["matched_rows"] == 1
|
||||
assert report["overall"]["coverage"]["missing"] == 3
|
||||
assert report["by_source"]["unknown"]["rows"] == 3
|
||||
assert report["overall"]["signals"]["has_path"]["rows"] == 1
|
||||
assert report["overall"]["both_asked_signals"]["has_path"]["pooled_auc"] is None
|
||||
|
||||
|
||||
def test_asymmetry_and_honest_denominator(audit_module, tmp_path):
|
||||
population = units(6)
|
||||
statuses = [
|
||||
"path",
|
||||
"asked-nothing",
|
||||
"asked-nothing",
|
||||
"path",
|
||||
"path",
|
||||
"path",
|
||||
"asked-nothing",
|
||||
"asked-nothing",
|
||||
"path",
|
||||
"failed",
|
||||
"unsupported",
|
||||
"timeout",
|
||||
]
|
||||
write_result(tmp_path, audit_module, [result(u, s) for u, s in zip(population, statuses, strict=True)])
|
||||
report = run(audit_module, tmp_path, population)
|
||||
overall = report["overall"]
|
||||
assert overall["pair_asymmetry"] == {"vulnerable_only": 1, "fixed_only": 1, "both": 1, "neither": 1, "not_both_asked": 2}
|
||||
assert overall["both_asked_pairs"] == 4
|
||||
assert overall["both_asked_signals"]["asked"]["within_pair_accuracy"] == 0.5
|
||||
assert overall["coverage"] == {"path": 5, "asked-nothing": 4, "failed": 1, "timeout": 1, "unsupported": 1, "missing": 0}
|
||||
assert report["by_family"]["injection"] == overall == report["by_source"]["authored"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("balanced", [False, True])
|
||||
def test_constructed_leak_and_balanced_set(audit_module, tmp_path, balanced):
|
||||
population = units()
|
||||
rows = [result(u, "path" if u["label"] == (int(u["pair"]) % 2 if balanced else 1) else "asked-nothing") for u in population]
|
||||
write_result(tmp_path, audit_module, rows)
|
||||
overall = run(audit_module, tmp_path, population)["overall"]
|
||||
for denominator in ("signals", "both_asked_signals"):
|
||||
signals = overall[denominator]
|
||||
for name in ("has_path", "shortest_path_steps"):
|
||||
assert signals[name]["flagged"] is not balanced
|
||||
assert signals[name]["within_pair_accuracy"] == signals[name]["pooled_auc"] == (0.5 if balanced else 1)
|
||||
assert signals["sink_kind"]["execute"]["flagged"] is not balanced
|
||||
assert not signals["asked"]["flagged"]
|
||||
assert not signals["sanitized"]["flagged"]
|
||||
assert not signals["guard_present"]["flagged"]
|
||||
|
||||
|
||||
def test_minimum_samples_reverse_and_source_proxy(audit_module, tmp_path):
|
||||
population = units(2)
|
||||
write_result(tmp_path, audit_module, [result(u, "path" if not u["label"] else "asked-nothing") for u in population])
|
||||
assert not run(audit_module, tmp_path, population)["overall"]["signals"]["has_path"]["flagged"]
|
||||
population = units(12)
|
||||
rows = []
|
||||
for u in population:
|
||||
row = result(u, "path" if not u["label"] else "unsupported")
|
||||
row["source"] = "positive" if u["label"] else "negative"
|
||||
rows.append(row)
|
||||
write_result(tmp_path, audit_module, rows)
|
||||
report = run(audit_module, tmp_path, population)
|
||||
assert report["overall"]["signals"]["asked"]["flagged"]
|
||||
assert report["overall"]["signals"]["asked"]["pooled_auc"] == 0
|
||||
assert report["overall"]["both_asked_pairs"] == 0
|
||||
assert report["by_source"]["positive"]["signals"]["asked"]["pooled_auc"] is None
|
||||
|
||||
|
||||
def test_shortest_trace_evidence_and_missing_trace(audit_module):
|
||||
row = result(units(1)[0], "path")
|
||||
row["traces"].insert(0, {"steps": [{}] * 5, "sanitized": False})
|
||||
row["witness_rows"].insert(0, {"sink": "longer(x)"})
|
||||
row["traces"][1].update(sanitized=True, guard={"bounded": True, "text": "off-path bound"})
|
||||
features = audit_module.features(row)
|
||||
assert (features["shortest_path_steps"], features["sanitized"], features["guard_present"], features["sink_kind"]) == (
|
||||
2,
|
||||
1,
|
||||
1,
|
||||
"execute",
|
||||
)
|
||||
row["traces"] = []
|
||||
assert audit_module.features(row)["shortest_path_steps"] is None
|
||||
|
||||
|
||||
def test_bad_inputs_fail_loudly(audit_module, tmp_path):
|
||||
population = units(1)
|
||||
write_result(tmp_path, audit_module, [result(population[0], "path")] * 2)
|
||||
with pytest.raises(ValueError, match="Duplicate"):
|
||||
run(audit_module, tmp_path, population)
|
||||
write_result(tmp_path, audit_module, [{**result(population[0], "path"), "label": 0}])
|
||||
with pytest.raises(ValueError, match="metadata mismatch"):
|
||||
run(audit_module, tmp_path, population)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("done", [False, True])
|
||||
def test_record_uses_hashes_without_repository_identity(audit_module, tmp_path, done):
|
||||
repo = "https://github.com/owner/name"
|
||||
population = [{**u, "group": "owner/name", "repo": repo, "source": "population-v1"} for u in units(1)]
|
||||
root = tmp_path / "github.com" / "owner" / "name"
|
||||
root.mkdir(parents=True)
|
||||
paths = []
|
||||
for i, u in enumerate(population):
|
||||
row = {**result(u, "path"), "source": "population-v1"}
|
||||
path = write_result(root, audit_module, [row], f"name-{1 - i}", done=done)
|
||||
record = json.loads(path.read_text())
|
||||
record.update(repo=repo, pin="owner/name")
|
||||
path.write_text(json.dumps(record))
|
||||
paths.append(path)
|
||||
report = run(audit_module, root, population)
|
||||
serialized = json.dumps(report)
|
||||
assert repo not in serialized and "owner/name" not in serialized
|
||||
assert "files" not in report["inputs"]
|
||||
inventory = sorted((p.name, p.stat().st_size) for p in paths)
|
||||
assert report["inputs"]["files_sha256"] == hashlib.sha256(json.dumps(inventory, separators=(",", ":")).encode()).hexdigest()
|
||||
assert report["inputs"]["files_read"] == 2
|
||||
assert report["inputs"]["bytes_read"] == sum(size for _, size in inventory)
|
||||
assert report["inputs"]["result_rows_read"] == (2 if done else 0)
|
||||
assert report["by_source"]["population-v1"]["rows"] == 2
|
||||
assert [r["unit"] for r in report["units"]] == [u["unit"] for u in population]
|
||||
assert [r["pair"] for r in report["units"]] == [u["pair"] for u in population]
|
||||
@@ -150,7 +150,9 @@ def test_learn_slice_audit_reads_and_never_writes(audit_module: ModuleType, monk
|
||||
|
||||
raw = b" 10 def f(x):\n 11 execute(x)\n"
|
||||
digest = hashlib.sha256(raw).hexdigest()
|
||||
example = replace(corpus(groups=1)[0], excerpt_sha=digest, source="pairs", language="python", family="injection")
|
||||
example = replace(
|
||||
corpus(groups=1)[0], excerpt_sha=digest, source="pairs", language="python", family="injection", group="github.com/owner/name"
|
||||
)
|
||||
monkeypatch.setattr(audit_module, "load_examples", lambda store: [example])
|
||||
|
||||
class ReadOnly:
|
||||
@@ -161,6 +163,8 @@ def test_learn_slice_audit_reads_and_never_writes(audit_module: ModuleType, monk
|
||||
units = [{"family": example.family, "label": example.label, "group": example.group, "unit": example.id, "pair": None, "fold": "f"}]
|
||||
report = audit_module.audit(ReadOnly(), units, expected_rows=1, expected_pairs=0)
|
||||
assert report["example_rows_read"] == report["matched_rows"] == report["excerpts_read"] == 1
|
||||
assert "owner/name" not in json.dumps(report)
|
||||
assert report["units"][0]["unit"] == example.id
|
||||
assert report["excerpt_bytes"] == len(raw)
|
||||
assert report["coverage"]["injection"]["flow"] == 1
|
||||
|
||||
|
||||
@@ -156,3 +156,27 @@ def test_profile_is_frozen_and_typed() -> None:
|
||||
assert isinstance(loaded, PlaneProfile)
|
||||
with pytest.raises(AttributeError):
|
||||
loaded.exec = "remote" # type: ignore[misc]
|
||||
|
||||
|
||||
def test_kind_observability_defaults_off() -> None:
|
||||
assert load_profile("kind", environ={}).kind_observability is False
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("raw", "expected"), [("1", True), ("true", True), ("0", False), ("FALSE", False)])
|
||||
def test_kind_observability_environment_override(tmp_path: Path, raw: str, expected: bool) -> None:
|
||||
path = write_profile(tmp_path, MINIMAL + f"kind_observability = {str(not expected).lower()}\n")
|
||||
assert load_profile(str(path), environ={"OUSAST_KIND_OBSERVABILITY": raw}).kind_observability is expected
|
||||
|
||||
|
||||
def test_kind_observability_validation_and_print(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]
|
||||
) -> None:
|
||||
path = write_profile(tmp_path, MINIMAL + 'kind_observability = "false"\n')
|
||||
with pytest.raises(ProfileError, match="kind_observability must be a boolean"):
|
||||
load_profile(str(path), environ={})
|
||||
path = write_profile(tmp_path)
|
||||
with pytest.raises(ProfileError, match="OUSAST_KIND_OBSERVABILITY must be"):
|
||||
load_profile(str(path), environ={"OUSAST_KIND_OBSERVABILITY": "maybe"})
|
||||
monkeypatch.setenv("OUSAST_KIND_OBSERVABILITY", "1")
|
||||
assert profile_module.main(["--profile", str(path), "--print", "kind_observability"]) == 0
|
||||
assert capsys.readouterr().out == "true\n"
|
||||
|
||||
@@ -0,0 +1,527 @@
|
||||
"""Draft and coordinator tests contain synthetic identities only and use injected clients."""
|
||||
|
||||
import copy
|
||||
import importlib
|
||||
import sys
|
||||
import tomllib
|
||||
from collections import Counter
|
||||
from contextlib import contextmanager
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
d = importlib.import_module("benchmarks.unseen.draft")
|
||||
s = importlib.import_module("benchmarks.unseen.source")
|
||||
e = importlib.import_module("benchmarks.unseen.eligibility")
|
||||
|
||||
|
||||
def entries():
|
||||
rows = []
|
||||
for i in range(300):
|
||||
changes = [{"kind": "vulnerability", "method": "introducing", "base": "a" * 40, "head": "b" * 40}]
|
||||
changes += [{"kind": "ordinary", "method": "first_parent", "base": f"{j:040x}", "head": f"{j + 20:040x}"} for j in range(12)]
|
||||
rows.append(
|
||||
{
|
||||
"repository": f"fixture/project{i:03d}",
|
||||
"url": f"https://github.com/fixture/project{i:03d}",
|
||||
"ecosystem": s.ECOSYSTEMS[i % 4],
|
||||
"post_cutoff": i % 7 == 0,
|
||||
"license_spdx": "MIT" if i % 2 else "GPL-3.0",
|
||||
"license_class": "permissive" if i % 2 else "private",
|
||||
"license_ref": "b" * 40,
|
||||
"changes": changes,
|
||||
}
|
||||
)
|
||||
return rows
|
||||
|
||||
|
||||
def test_draft_split_digest_and_balanced_slices(tmp_path):
|
||||
rows = entries()
|
||||
assigned = d.assign_slices(rows, seed=19)
|
||||
assert Counter(r["slice"] for r in assigned) == {1: 100, 2: 100, 3: 100}
|
||||
assert assigned == d.assign_slices(list(reversed(rows)), seed=19)
|
||||
strata = Counter((r["ecosystem"], r["post_cutoff"]) for r in assigned)
|
||||
for slice_id in (1, 2, 3):
|
||||
counts = Counter((r["ecosystem"], r["post_cutoff"]) for r in assigned if r["slice"] == slice_id)
|
||||
assert all(abs(counts[key] - total / 3) < 1 for key, total in strata.items())
|
||||
public, private = d.write_draft(tmp_path, assigned, seed=19)
|
||||
public_text = public.read_text()
|
||||
shared = tomllib.loads(public_text)
|
||||
hidden = tomllib.loads(private.read_text())
|
||||
assert len(shared["repository"]) == 300 and len(hidden["repository"]) == 150
|
||||
assert sum(len(r["changes"]) for r in shared["repository"] if "changes" in r) == 1950
|
||||
for row in hidden["repository"]:
|
||||
assert row["url"] not in public_text and row["repository"] not in public_text
|
||||
pointer = next(p for p in shared["repository"] if p["id"] == row["id"])
|
||||
assert set(pointer) == {"id", "sha256", "slice", "private"}
|
||||
assert pointer["sha256"] == d.private_digest(row)
|
||||
changed = copy.deepcopy(row)
|
||||
changed["changes"][0]["head"] = "c" * 40
|
||||
assert d.private_digest(changed) != pointer["sha256"]
|
||||
assert private.is_relative_to(tmp_path / "benchmarks/unseen/private")
|
||||
with pytest.raises(FileExistsError):
|
||||
d.write_draft(tmp_path, assigned, seed=19)
|
||||
|
||||
|
||||
def test_draft_requires_full_size_and_valid_changes():
|
||||
with pytest.raises(ValueError, match="pool_size"):
|
||||
d.assign_slices(entries()[:299], seed=1)
|
||||
rows = entries()
|
||||
rows[0]["changes"].pop()
|
||||
with pytest.raises(ValueError, match="change_count"):
|
||||
d.assign_slices(rows, seed=1)
|
||||
|
||||
|
||||
def test_cache_is_outside_repository_deleted_and_partial(tmp_path):
|
||||
import subprocess
|
||||
|
||||
root, cache = tmp_path / "repo", tmp_path / "cache"
|
||||
root.mkdir()
|
||||
with pytest.raises(ValueError, match="cache_inside_repository"):
|
||||
d.Clones(root, root / "cache")
|
||||
calls = []
|
||||
|
||||
def runner(argv, **kwargs):
|
||||
calls.append(argv)
|
||||
if "clone" in argv:
|
||||
target = Path(argv[-1])
|
||||
(target / ".git").mkdir()
|
||||
(target / ".git/pack").write_bytes(b"123")
|
||||
return subprocess.CompletedProcess(argv, 0, b"", b"")
|
||||
|
||||
clones = d.Clones(root, cache, runner=runner)
|
||||
with clones.open("https://github.com/fixture/project") as git:
|
||||
assert git.path.exists()
|
||||
clone_path = git.path
|
||||
assert not clone_path.exists()
|
||||
assert "--filter=blob:none" in calls[0] and "--no-checkout" in calls[0]
|
||||
assert clones.counts["clones"] == 1 and clones.counts["clone_bytes"] == 3
|
||||
|
||||
|
||||
def test_clone_failure_sanitized_and_cleaned(tmp_path, capsys):
|
||||
import subprocess
|
||||
|
||||
def runner(argv, **kwargs):
|
||||
return subprocess.CompletedProcess(argv, 128, b"", b"private identity")
|
||||
|
||||
clones = d.Clones(tmp_path / "repo", tmp_path / "cache", runner=runner)
|
||||
with pytest.raises(d.InstrumentFailure), clones.open("https://github.com/fixture/project"):
|
||||
pass
|
||||
assert list((tmp_path / "cache").iterdir()) == []
|
||||
assert not capsys.readouterr().out
|
||||
|
||||
|
||||
class FakeAPI:
|
||||
calls = 0
|
||||
|
||||
def __init__(self, count=300):
|
||||
self.count = count
|
||||
self.licenses = []
|
||||
|
||||
def get_repo(self, name):
|
||||
self.calls += 1
|
||||
if name == "fixture/old":
|
||||
name = "fixture/project000"
|
||||
repo = {"full_name": name, "private": False, "size": 2, "clone_url": "https://github.com/" + name}
|
||||
if name == "fixture/fork":
|
||||
repo["source"] = {"full_name": "fixture/project001"}
|
||||
return repo
|
||||
|
||||
def get_commit(self, name, sha):
|
||||
self.calls += 1
|
||||
return {"sha": sha, "parents": [{"sha": "b" * 40}]}
|
||||
|
||||
def get_license(self, name, ref):
|
||||
self.licenses.append((name, ref))
|
||||
self.calls += 1
|
||||
# P is MIT but actual introducing head is GPL: public output must use V.
|
||||
return {"text": "", "spdx_id": "MIT" if ref == "b" * 40 else "GPL-3.0", "path": "LICENSE"}
|
||||
|
||||
def advisory_page(self, page):
|
||||
self.calls += 1
|
||||
rows = list(self.advisories())[(page - 1) * 100 : page * 100]
|
||||
return {"rows": rows, "done": len(rows) < 100}
|
||||
|
||||
def advisories(self):
|
||||
for i in range(self.count):
|
||||
yield {
|
||||
"ghsa_id": f"GHSA-test-test-{i:04d}",
|
||||
"published_at": "2026-06-02T00:00:00Z",
|
||||
"cwes": [{"cwe_id": "CWE-89"}],
|
||||
"vulnerabilities": [{"package": {"ecosystem": "pip", "name": "fixture"}}],
|
||||
"references": [f"https://github.com/fixture/project{i:03d}/commit/" + "a" * 40],
|
||||
}
|
||||
|
||||
|
||||
class FakeOSV:
|
||||
calls = 0
|
||||
|
||||
def get(self, identifier):
|
||||
self.calls += 1
|
||||
return {}
|
||||
|
||||
|
||||
class FakeClones:
|
||||
def __init__(self):
|
||||
self.counts = Counter()
|
||||
self.commands = []
|
||||
|
||||
@contextmanager
|
||||
def open(self, url):
|
||||
self.counts["clones"] += 1
|
||||
yield self
|
||||
|
||||
def run(self, *args):
|
||||
self.commands.append(args)
|
||||
return ""
|
||||
|
||||
|
||||
def test_coordinator_end_to_end_fakes_pinned_license_and_no_names(tmp_path, monkeypatch, capsys):
|
||||
rows = entries()
|
||||
changes = copy.deepcopy(rows[0]["changes"])
|
||||
changes[0]["head"] = "c" * 40
|
||||
monkeypatch.setattr(d, "repository_changes", lambda *a, **kw: (changes, {"security": 4, "eligible": 40}, {"bytes": 12, "files": 1}))
|
||||
used, api, clones = e.UsedSet(), FakeAPI(), FakeClones()
|
||||
report = d.run(tmp_path, github=api, osv=FakeOSV(), clones=clones, used=used, seed=7, known_empty=set(used.sources))
|
||||
assert report["status"] == "draft" and report["changes"] == 3900
|
||||
assert report["license_split"] == {"private": 300}
|
||||
assert report["ordinary_exclusions"]["security"] == 1200
|
||||
assert len([ref for _, ref in api.licenses if ref == "c" * 40]) == 300
|
||||
public = (tmp_path.parent / (tmp_path.name + "-cache") / "journal/draft-p1.toml").read_text()
|
||||
assert "fixture/" not in public
|
||||
import json
|
||||
|
||||
assert "fixture/" not in json.dumps(report)
|
||||
assert capsys.readouterr().out == ""
|
||||
|
||||
|
||||
def test_used_rename_and_fork_resolution_and_shortfall(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(d, "repository_changes", lambda *a, **kw: (entries()[0]["changes"], {}, {"bytes": 1, "files": 1}))
|
||||
used, api, clones = e.UsedSet(), FakeAPI(2), FakeClones()
|
||||
used.add("pairs", {"fixture/old"}, 10)
|
||||
# project000 resolves from used alias; project001 is reached via used fork's source.
|
||||
used.add("pairs", {"fixture/fork"}, 10)
|
||||
report = d.run(tmp_path, github=api, osv=FakeOSV(), clones=clones, used=used, seed=7, known_empty=set(used.sources) - {"pairs"})
|
||||
assert report["status"] == "insufficient_repositories"
|
||||
assert report["rejections"] == {"used_pairs": 2}
|
||||
assert report["repositories"] == 0 and not clones.counts
|
||||
assert not (tmp_path / "benchmarks/unseen/draft-p1.toml").exists()
|
||||
|
||||
|
||||
def test_zero_source_is_instrument_failure_not_rejection(tmp_path, monkeypatch):
|
||||
def unread(*args, **kwargs):
|
||||
raise d.InstrumentFailure("zero_source_bytes")
|
||||
|
||||
monkeypatch.setattr(d, "repository_changes", unread)
|
||||
used = e.UsedSet()
|
||||
report = d.run(tmp_path, github=FakeAPI(1), osv=FakeOSV(), clones=FakeClones(), used=used, seed=1, known_empty=set(used.sources))
|
||||
assert report["status"] == "instrument_failure" and not report["rejections"]
|
||||
|
||||
|
||||
def test_cli_uses_environment_and_injected_stores(tmp_path, monkeypatch, capsys):
|
||||
from openultrasast import config
|
||||
from openultrasast.plane import memory
|
||||
|
||||
seen = []
|
||||
monkeypatch.setenv("GH_TOKEN", "fake-secret")
|
||||
monkeypatch.setattr(config, "load_dotenv", lambda path: None)
|
||||
monkeypatch.setattr(memory, "open_store", lambda value, **kw: seen.append((value, kw)) or object())
|
||||
monkeypatch.setattr(d.eligibility, "build_used", lambda *a, **kw: e.UsedSet())
|
||||
monkeypatch.setattr(d.source, "GitHub", lambda token: seen.append(token) or FakeAPI(0))
|
||||
monkeypatch.setattr(d.source, "OSV", FakeOSV)
|
||||
args = ["--root", str(tmp_path), "--cache", str(tmp_path.parent / "clone-cache")]
|
||||
for label in e.SOURCES:
|
||||
args += ["--known-empty", label]
|
||||
assert d.main(args) == 1
|
||||
output = capsys.readouterr().out
|
||||
assert "instrument_failure" in output and "fake-secret" not in output and "fixture/" not in output
|
||||
assert seen[-1] == "fake-secret"
|
||||
assert all(kwargs == {"read_only": True} for _, kwargs in seen[:-1])
|
||||
|
||||
|
||||
def test_repository_changes_excludes_all_advisory_sites(monkeypatch):
|
||||
calls = []
|
||||
|
||||
class FakeGit:
|
||||
def run(self, *args):
|
||||
calls.append(args)
|
||||
if args[0] == "ls-tree":
|
||||
return "app.py\n"
|
||||
return args[-1] if args[0] == "rev-parse" else ""
|
||||
|
||||
def blob(self, revision, path):
|
||||
return "def handler():\n return 1\n"
|
||||
|
||||
def parents(self, fix):
|
||||
return ["b" * 40]
|
||||
|
||||
candidate = s.Candidate(
|
||||
"fixture/project",
|
||||
"https://github.com/fixture/project",
|
||||
"a" * 40,
|
||||
"b" * 40,
|
||||
"GHSA-test-test-test",
|
||||
"pip",
|
||||
"injection",
|
||||
True,
|
||||
None,
|
||||
"MIT",
|
||||
"permissive",
|
||||
"b" * 40,
|
||||
"LICENSE",
|
||||
)
|
||||
monkeypatch.setattr(d.extract, "introducing", lambda *a: {"head": "c" * 40})
|
||||
monkeypatch.setattr(d.extract, "fix_sites", lambda git, fix, parent, **kw: [{"path": "app.py", "function": fix}])
|
||||
monkeypatch.setattr(d.extract, "ordinary_history", lambda *a: [])
|
||||
|
||||
def draw(rows, fixes, known, **kwargs):
|
||||
assert fixes == {"a" * 40, "d" * 40}
|
||||
assert known == {("app.py", "a" * 40), ("app.py", "d" * 40)}
|
||||
return [], {}
|
||||
|
||||
monkeypatch.setattr(d.extract, "draw_ordinary", draw)
|
||||
_, _, proof = d.repository_changes(FakeGit(), candidate, {"d" * 40}, seed=1)
|
||||
assert proof["bytes"] > 0
|
||||
assert len([c for c in calls if c[0] == "fetch"]) == 2
|
||||
|
||||
|
||||
def test_candidate_404_is_counted_but_service_failure_aborts(tmp_path):
|
||||
class Missing(FakeAPI):
|
||||
def get_commit(self, name, sha):
|
||||
raise e.RepositoryNotFound("not_found")
|
||||
|
||||
class Broken(FakeAPI):
|
||||
def get_commit(self, name, sha):
|
||||
raise ValueError("private URL must never escape")
|
||||
|
||||
for api, status in [(Missing(1), "insufficient_repositories"), (Broken(1), "instrument_failure")]:
|
||||
used = e.UsedSet()
|
||||
report = d.run(
|
||||
tmp_path,
|
||||
github=api,
|
||||
osv=FakeOSV(),
|
||||
clones=FakeClones(),
|
||||
used=used,
|
||||
seed=1,
|
||||
known_empty=set(used.sources),
|
||||
journal_path=tmp_path.parent / (tmp_path.name + type(api).__name__) / "journal",
|
||||
)
|
||||
assert report["status"] == status
|
||||
if status == "insufficient_repositories":
|
||||
assert report["rejections"] == {"repository_or_commit_not_found": 1}
|
||||
assert "private URL" not in str(report)
|
||||
|
||||
|
||||
def test_shortfall_keeps_ordinary_exclusion_counts(tmp_path, monkeypatch):
|
||||
def too_few(*args, **kwargs):
|
||||
exc = s.Rejected("ordinary_floor")
|
||||
exc.counts = {"eligible": 39, "size_cap": 10}
|
||||
raise exc
|
||||
|
||||
monkeypatch.setattr(d, "repository_changes", too_few)
|
||||
used = e.UsedSet()
|
||||
report = d.run(tmp_path, github=FakeAPI(1), osv=FakeOSV(), clones=FakeClones(), used=used, seed=1, known_empty=set(used.sources))
|
||||
assert report["ordinary_exclusions"] == {"eligible": 39, "size_cap": 10}
|
||||
|
||||
|
||||
def test_clone_disk_cap_cleans_up(tmp_path):
|
||||
import subprocess
|
||||
|
||||
def runner(argv, **kwargs):
|
||||
(Path(argv[-1]) / "big").write_bytes(b"12345")
|
||||
return subprocess.CompletedProcess(argv, 0, b"", b"")
|
||||
|
||||
clones = d.Clones(tmp_path / "repo", tmp_path / "cache", runner=runner, limit_bytes=4)
|
||||
with pytest.raises(d.InstrumentFailure, match="clone_size_cap"), clones.open("https://github.com/fixture/project"):
|
||||
pass
|
||||
assert list((tmp_path / "cache").iterdir()) == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize("stage,n", [("advisories", 2), ("resolution", 103), ("eligibility", 17), ("extraction", 23), ("completion", 1)])
|
||||
def test_checkpoint_interrupt_resume_matches_uninterrupted(tmp_path, monkeypatch, stage, n):
|
||||
import json
|
||||
|
||||
journal_module = importlib.import_module("benchmarks.unseen.journal")
|
||||
# Large fake populations exercise replay; durability is checked separately below.
|
||||
monkeypatch.setattr(journal_module.os, "fsync", lambda fd: None)
|
||||
root = tmp_path / "repo"
|
||||
full, interrupted = tmp_path / "full/journal", tmp_path / "interrupted/journal"
|
||||
changes = entries()[0]["changes"]
|
||||
monkeypatch.setattr(d, "repository_changes", lambda *a, **kw: (changes, {"eligible": 40}, {"bytes": 12, "files": 1}))
|
||||
used = e.UsedSet()
|
||||
|
||||
def run(path, api=None, clones=None):
|
||||
return d.run(
|
||||
root,
|
||||
github=api or FakeAPI(),
|
||||
osv=FakeOSV(),
|
||||
clones=clones or FakeClones(),
|
||||
used=used,
|
||||
seed=7,
|
||||
known_empty=set(used.sources),
|
||||
journal_path=path,
|
||||
)
|
||||
|
||||
assert run(full)["status"] == "draft"
|
||||
original = d.Journal.put
|
||||
|
||||
def interrupt(self, current, key, value):
|
||||
original(self, current, key, value)
|
||||
if current == stage and len(self.rows[current]) == n:
|
||||
raise KeyboardInterrupt
|
||||
|
||||
monkeypatch.setattr(d.Journal, "put", interrupt)
|
||||
with pytest.raises(KeyboardInterrupt):
|
||||
run(interrupted)
|
||||
assert not (interrupted / "draft-p1.toml").exists()
|
||||
before = {p.name: p.read_bytes() for p in interrupted.glob("*.jsonl")}
|
||||
progress = json.loads((interrupted / "progress.json").read_text())
|
||||
assert progress["counts"][stage] == n
|
||||
assert progress["last_unit"]["stage"] == stage
|
||||
assert progress["elapsed_seconds"] >= 0
|
||||
assert "rate_limit" in progress
|
||||
monkeypatch.setattr(d.Journal, "put", original)
|
||||
clones = FakeClones()
|
||||
assert run(interrupted, clones=clones)["status"] == "draft"
|
||||
assert clones.counts["clones"] == 300 - progress["counts"]["extraction"]
|
||||
for name in ("draft-p1.toml", "private-draft-p1.toml"):
|
||||
assert (full / name).read_bytes() == (interrupted / name).read_bytes()
|
||||
assert all((interrupted / name).read_bytes().startswith(data) for name, data in before.items())
|
||||
assert not root.exists() # no names, manifests, or other records in the checkout
|
||||
api, clones = FakeAPI(), FakeClones()
|
||||
assert run(interrupted, api, clones)["status"] == "draft"
|
||||
assert api.calls == 0 and clones.counts == {}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("parameter", ["seed", "used_set_digest", "clone_limit_bytes", "design_extract"])
|
||||
def test_resume_refuses_changed_inputs_before_network(tmp_path, monkeypatch, parameter):
|
||||
path, root = tmp_path / "cache/journal", tmp_path / "repo"
|
||||
used = e.UsedSet()
|
||||
kwargs = dict(
|
||||
root=root,
|
||||
github=FakeAPI(),
|
||||
osv=FakeOSV(),
|
||||
clones=FakeClones(),
|
||||
used=used,
|
||||
seed=7,
|
||||
known_empty=set(used.sources),
|
||||
journal_path=path,
|
||||
max_hours=0,
|
||||
)
|
||||
assert d.run(**kwargs)["status"] == "stopped"
|
||||
if parameter == "seed":
|
||||
kwargs["seed"] = 8
|
||||
elif parameter == "used_set_digest":
|
||||
used.add("pairs", {"fixture/new"}, 12)
|
||||
elif parameter == "clone_limit_bytes":
|
||||
kwargs["clones"].limit_bytes = 12
|
||||
else:
|
||||
original = d.inputs
|
||||
monkeypatch.setattr(d, "inputs", lambda *a: {**original(*a), "design_extract": "changed"})
|
||||
report = d.run(**kwargs)
|
||||
assert report["status"] == "resume_refused"
|
||||
assert report["changed_parameters"] == [parameter]
|
||||
assert kwargs["github"].calls == 0
|
||||
|
||||
|
||||
def test_stop_between_units_and_progress_rate_state(tmp_path, monkeypatch):
|
||||
import json
|
||||
from types import SimpleNamespace
|
||||
|
||||
path, root = tmp_path / "cache/journal", tmp_path / "repo"
|
||||
api = FakeAPI(1)
|
||||
api.transport = SimpleNamespace(rate_limit={"remaining": "27", "reset": "1234"}, attempts=0)
|
||||
used = e.UsedSet()
|
||||
original = d.Journal.put
|
||||
|
||||
def stop(self, stage, key, value):
|
||||
original(self, stage, key, value)
|
||||
if stage == "eligibility":
|
||||
(self.path / "STOP").touch()
|
||||
|
||||
monkeypatch.setattr(d.Journal, "put", stop)
|
||||
clones = FakeClones()
|
||||
report = d.run(root, github=api, osv=FakeOSV(), clones=clones, used=used, seed=1, known_empty=set(used.sources), journal_path=path)
|
||||
assert report["status"] == "stopped" and report["reason"] == "STOP"
|
||||
assert clones.counts == {}
|
||||
progress = json.loads((path / "progress.json").read_text())
|
||||
assert progress["status"] == "stopped" and progress["counts"]["eligibility"] == 1
|
||||
assert progress["rate_limit"] == api.transport.rate_limit
|
||||
assert not (path / "draft-p1.toml").exists()
|
||||
|
||||
|
||||
def test_max_hours_stops_after_completed_page(tmp_path):
|
||||
import json
|
||||
|
||||
now = [0.0]
|
||||
|
||||
class TimedAPI(FakeAPI):
|
||||
def advisory_page(self, page):
|
||||
result = super().advisory_page(page)
|
||||
now[0] += 3601
|
||||
return result
|
||||
|
||||
path = tmp_path / "cache/journal"
|
||||
api, used = TimedAPI(), e.UsedSet()
|
||||
report = d.run(
|
||||
tmp_path / "repo",
|
||||
github=api,
|
||||
osv=FakeOSV(),
|
||||
clones=FakeClones(),
|
||||
used=used,
|
||||
seed=1,
|
||||
known_empty=set(used.sources),
|
||||
journal_path=path,
|
||||
max_hours=1,
|
||||
clock=lambda: now[0],
|
||||
)
|
||||
assert report["status"] == "stopped" and report["reason"] == "max_hours"
|
||||
assert api.calls == 1
|
||||
progress = json.loads((path / "progress.json").read_text())
|
||||
assert progress["counts"]["advisories"] == 1
|
||||
assert progress["elapsed_seconds"] == 3601
|
||||
|
||||
|
||||
def test_journal_cannot_be_inside_repository(tmp_path):
|
||||
used = e.UsedSet()
|
||||
api = FakeAPI()
|
||||
report = d.run(
|
||||
tmp_path,
|
||||
github=api,
|
||||
osv=FakeOSV(),
|
||||
clones=FakeClones(),
|
||||
used=used,
|
||||
seed=1,
|
||||
known_empty=set(used.sources),
|
||||
journal_path=tmp_path / "journal",
|
||||
)
|
||||
assert report["status"] == "instrument_failure" and api.calls == 0
|
||||
assert not (tmp_path / "journal").exists()
|
||||
|
||||
|
||||
def test_journal_fsync_lock_and_corruption(tmp_path, monkeypatch):
|
||||
import json
|
||||
|
||||
j = importlib.import_module("benchmarks.unseen.journal")
|
||||
original = j.os.fsync
|
||||
synced = []
|
||||
|
||||
def fsync(fd):
|
||||
synced.append(fd)
|
||||
original(fd)
|
||||
|
||||
monkeypatch.setattr(j.os, "fsync", fsync)
|
||||
path = tmp_path / "journal"
|
||||
with j.Journal(path).open({"seed": 1}) as journal:
|
||||
before = len(synced)
|
||||
journal.put("advisories", "1", {"rows": [], "done": True})
|
||||
# Appended JSONL, directory, atomic progress file, directory.
|
||||
assert len(synced) - before == 4
|
||||
assert json.loads((path / "progress.json").read_text())["counts"]["advisories"] == 1
|
||||
with pytest.raises(ValueError, match="journal_busy"), j.Journal(path).open({"seed": 1}):
|
||||
pass
|
||||
with (path / "advisories.jsonl").open("ab") as handle:
|
||||
handle.write(b'{"id":')
|
||||
with pytest.raises(ValueError, match="journal_incomplete_record"), j.Journal(path).open({"seed": 1}):
|
||||
pass
|
||||
@@ -0,0 +1,293 @@
|
||||
"""Offline eligibility fixtures; no real repository identities are needed."""
|
||||
|
||||
import io
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from openultrasast.plane.memory import FileStore
|
||||
|
||||
with pytest.MonkeyPatch.context() as patch:
|
||||
patch.syspath_prepend(str(Path(__file__).resolve().parents[1] / "benchmarks/unseen"))
|
||||
import eligibility as e
|
||||
|
||||
|
||||
def plant(root, path, text):
|
||||
target = root / path
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_text(text)
|
||||
return target
|
||||
|
||||
|
||||
def test_sources_and_reserved_input_never_opened(tmp_path, monkeypatch, capsys):
|
||||
sources = {
|
||||
"benchmarks/pairs/a/catalog.toml": ("pairs", "pair"),
|
||||
"benchmarks/pairs/a/recipes.toml": ("pairs", "recipe"),
|
||||
"benchmarks/pairs/a/dataset.json": ("pairs", "dataset"),
|
||||
"benchmarks/pairs/advisory-fixes-b/catalog.toml": ("harvest", "harvest"),
|
||||
"benchmarks/independent/population-v1.toml": ("populations", "first"),
|
||||
"benchmarks/independent/population-v2.toml": ("populations", "second"),
|
||||
"plane/experiments/a.units.jsonl": ("units", "unit"),
|
||||
"benchmarks/measurements/a/record.json": ("measurements", "measurement"),
|
||||
"benchmarks/experiments/a.json": ("measurements", "experiment"),
|
||||
"src/example.txt": ("sweep", "sweep"),
|
||||
}
|
||||
for path, (_, name) in sources.items():
|
||||
text = f'repo = "https://github.com/fixture/{name}"' if path.endswith(".toml") else json.dumps({"repo": f"fixture/{name}"})
|
||||
if name == "sweep":
|
||||
text = "https://github.com/fixture/sweep"
|
||||
plant(tmp_path, path, text)
|
||||
pointer = plant(tmp_path, "local/pointers.toml", '[[pair]]\nrepo = "fixture/pointer"\nfix_repo = "fixture/fix"\nvendored = false')
|
||||
plant(tmp_path, "benchmarks/independent/population-v3.toml", "do not read")
|
||||
plant(tmp_path, "benchmarks/independent/results-v3.json", "do not read")
|
||||
original = io.open
|
||||
|
||||
def guarded(file, *args, **kwargs):
|
||||
if "v3" in str(file):
|
||||
pytest.fail("reserved input opened")
|
||||
return original(file, *args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(io, "open", guarded)
|
||||
store = FileStore(tmp_path / "memory")
|
||||
plant(tmp_path, "memory/repos/opaque/pin.jsonl", json.dumps({"id": "1", "kind": "example", "repo": "fixture/memory"}) + "\n")
|
||||
used = e.build_used(tmp_path, stores={"local": store}, pointer_manifests=[pointer])
|
||||
for label, name in sources.values():
|
||||
assert f"fixture/{name}" in used.sources[label]
|
||||
assert {"fixture/pointer", "fixture/fix"} <= used.sources["pointers"]
|
||||
assert "fixture/memory" in used.sources["memory_local"]
|
||||
report = used.report()
|
||||
assert report["sources"]["memory_local"]["bytes"] > 0
|
||||
assert all(item["files"] > 0 and item["bytes"] > 0 for item in report["sources"].values())
|
||||
assert "fixture/" not in json.dumps(report)
|
||||
assert capsys.readouterr().out == ""
|
||||
assert e.normalize("HTTPS://GitHub.com/Fixture/Pair.GIT/") == "fixture/pair"
|
||||
assert len({e.normalize(s) for s in ["fixture/pair", "Fixture/Pair.git/", "github.com/fixture/pair"]}) == 1
|
||||
|
||||
|
||||
def test_empty_sources_fail_unless_declared(tmp_path):
|
||||
used = e.build_used(tmp_path)
|
||||
with pytest.raises(ValueError, match="empty_source"):
|
||||
used.report()
|
||||
assert used.report(known_empty=set(used.sources))["count"] == 0
|
||||
|
||||
|
||||
class FakeAPI:
|
||||
def __init__(self, records):
|
||||
self.records = records
|
||||
|
||||
def get_repo(self, name):
|
||||
return self.records[name]
|
||||
|
||||
|
||||
def test_resolution_rejects_used_renames_and_forks():
|
||||
used = e.UsedSet()
|
||||
used.add("pairs", {"fixture/used"}, 10)
|
||||
api = FakeAPI(
|
||||
{
|
||||
"fixture/used": {"full_name": "fixture/used"},
|
||||
"fixture/old": {"full_name": "fixture/new", "former_names": ["fixture/used"]},
|
||||
"fixture/fork": {"full_name": "fixture/fork", "parent": {"full_name": "fixture/used"}, "source": {"full_name": "fixture/used"}},
|
||||
"fixture/a": {"full_name": "fixture/a", "source": {"full_name": "fixture/root"}},
|
||||
"fixture/b": {"full_name": "fixture/b", "source": {"full_name": "fixture/root"}},
|
||||
}
|
||||
)
|
||||
selected = set()
|
||||
for name in ("used", "old", "fork"):
|
||||
resolved = e.resolve("fixture/" + name, api)
|
||||
assert e.rejection(resolved, used, selected) == "pairs"
|
||||
assert e.rejection(e.resolve("fixture/a", api), used, selected) is None
|
||||
assert e.rejection(e.resolve("fixture/b", api), used, selected) == "fork_network"
|
||||
|
||||
|
||||
def test_registered_units_and_read_only_store(tmp_path):
|
||||
units = plant(tmp_path, "local/units.jsonl", json.dumps({"repo": "fixture/registered"}))
|
||||
store = FileStore(tmp_path / "memory")
|
||||
plant(tmp_path, "memory/repos/opaque/pin.jsonl", json.dumps({"kind": "experiment", "units_file": str(units)}))
|
||||
used = e.build_used(tmp_path, stores={"s3": store})
|
||||
assert "fixture/registered" in used.sources["units"]
|
||||
|
||||
|
||||
def test_cli_injected_store_and_credentials(tmp_path, monkeypatch, capsys):
|
||||
from openultrasast import config
|
||||
from openultrasast.plane import memory
|
||||
|
||||
calls = []
|
||||
monkeypatch.setattr(config, "load_dotenv", lambda path: calls.append("dotenv"))
|
||||
monkeypatch.setenv("GH_TOKEN", "test-token")
|
||||
store = FileStore(tmp_path / "store")
|
||||
monkeypatch.setattr(memory, "open_store", lambda spec, **kwargs: calls.append(spec) or store)
|
||||
monkeypatch.setattr(e, "GitHub", lambda token: calls.append(token) or FakeAPI({}))
|
||||
args = ["--root", str(tmp_path), "--s3-store", "s3://", "--local-store", "file:///unused"]
|
||||
for label in (*e.SOURCES, "memory_local", "memory_s3"):
|
||||
args += ["--known-empty", label]
|
||||
assert e.main(args) == 0
|
||||
assert calls == ["dotenv", "file:///unused", "s3://", "test-token"]
|
||||
output = capsys.readouterr().out
|
||||
assert json.loads(output)["count"] == 0
|
||||
assert "test-token" not in output
|
||||
|
||||
|
||||
def test_s3_open_is_read_only(tmp_path, monkeypatch):
|
||||
from openultrasast.plane import memory
|
||||
|
||||
class Objects:
|
||||
def keys(self, prefix):
|
||||
return ["repos/opaque/pin.jsonl"]
|
||||
|
||||
def get(self, key):
|
||||
return json.dumps({"kind": "example", "repo": "fixture/stored"}).encode(), "1"
|
||||
|
||||
def __getattr__(self, name):
|
||||
raise AssertionError("unexpected_store_operation")
|
||||
|
||||
monkeypatch.setattr(memory, "S3Client", lambda **kwargs: Objects())
|
||||
monkeypatch.setattr(memory, "s3_settings", lambda env: {})
|
||||
store = memory.open_store("s3://fixture-bucket", read_only=True)
|
||||
used = e.build_used(tmp_path, stores={"s3": store})
|
||||
assert used.sources["memory_s3"] == {"fixture/stored"}
|
||||
with pytest.raises(memory.MemoryStoreError, match="read.only"):
|
||||
store._put("key", b"value")
|
||||
with pytest.raises(memory.MemoryStoreError, match="read.only"):
|
||||
store._delete("key")
|
||||
|
||||
|
||||
def test_non_repository_provenance_and_missing_used_identity(tmp_path):
|
||||
plant(tmp_path, "benchmarks/pairs/recipes.toml", 'repo = "provenance-label"')
|
||||
assert not e.build_used(tmp_path).sources["pairs"]
|
||||
used = e.UsedSet()
|
||||
used.add("pairs", {"fixture/old", "fixture/nonexistent"}, 1)
|
||||
|
||||
class API:
|
||||
def get_repo(self, name):
|
||||
if name.endswith("nonexistent"):
|
||||
raise e.RepositoryNotFound
|
||||
return {"full_name": "fixture/new"}
|
||||
|
||||
e.canonicalize_used(used, API())
|
||||
assert "fixture/nonexistent" in used.sources["pairs"]
|
||||
assert e.rejection(e.resolve("fixture/new", API()), used, set()) == "pairs"
|
||||
|
||||
|
||||
def test_binary_measurement_and_local_pointer_discovery(tmp_path):
|
||||
target = tmp_path / "benchmarks/measurements/a/image.bin"
|
||||
target.parent.mkdir(parents=True)
|
||||
target.write_bytes(b"\xff\x00")
|
||||
plant(tmp_path, "benchmarks/pairs/a/recipes.toml", '[[pair]]\nrepo = "fixture/pointer"\nvendored = false')
|
||||
used = e.build_used(tmp_path)
|
||||
assert used.instrument["measurements"]["bytes"] == 2
|
||||
assert used.sources["pointers"] == {"fixture/pointer"}
|
||||
|
||||
|
||||
def test_coordinator_verification_is_black_box(tmp_path, monkeypatch):
|
||||
import subprocess
|
||||
from types import SimpleNamespace
|
||||
|
||||
def run(command, **kwargs):
|
||||
assert kwargs["stdout"] == kwargs["stderr"] == kwargs["stdin"] == subprocess.DEVNULL
|
||||
path = Path(next(part.split("=", 1)[1] for part in command if part.startswith("--junitxml=")))
|
||||
path.write_text('<testsuites><testsuite tests="25" failures="0" errors="0" skipped="1"/></testsuites>')
|
||||
return SimpleNamespace(returncode=0)
|
||||
|
||||
monkeypatch.setattr(subprocess, "run", run)
|
||||
assert e.verify_tests(tmp_path)["tests"]["tests"] == 25
|
||||
monkeypatch.setattr(subprocess, "run", lambda *args, **kwargs: SimpleNamespace(returncode=2))
|
||||
with pytest.raises(ValueError, match="guard_or_tests_failed"):
|
||||
e.verify_tests(tmp_path)
|
||||
|
||||
|
||||
def test_registered_unit_group_is_repository(tmp_path):
|
||||
plant(tmp_path, "plane/experiments/a.units.jsonl", json.dumps({"unit": "opaque", "group": "fixture/unit", "label": 1}))
|
||||
assert "fixture/unit" in e.build_used(tmp_path).sources["units"]
|
||||
|
||||
|
||||
def test_multidocument_measurement_and_missing_registered_units(tmp_path):
|
||||
plant(tmp_path, "benchmarks/measurements/a/run.yaml", "repo: fixture/first\n---\nrepo: fixture/second\n")
|
||||
assert e.build_used(tmp_path).sources["measurements"] == {"fixture/first", "fixture/second"}
|
||||
store = FileStore(tmp_path / "memory")
|
||||
plant(
|
||||
tmp_path,
|
||||
"memory/repos/opaque/pin.jsonl",
|
||||
json.dumps(
|
||||
{
|
||||
"kind": "experiment",
|
||||
"units_file": "absent.units.jsonl",
|
||||
"units_digest": "a" * 64,
|
||||
}
|
||||
),
|
||||
)
|
||||
with pytest.raises(FileNotFoundError):
|
||||
e.build_used(tmp_path, stores={"local": store})
|
||||
|
||||
|
||||
def test_catalog_skips_fixture_and_preserves_host_identity(tmp_path, capsys):
|
||||
plant(
|
||||
tmp_path,
|
||||
"benchmarks/pairs/catalog.toml",
|
||||
'[[pair]]\norigin = "in-tree-fixture"\nvendored = false\n[[pair]]\nrepo = "https://GitLab.com/Fixture/Private.git"\n',
|
||||
)
|
||||
used = e.build_used(tmp_path)
|
||||
assert used.sources["pairs"] == {"gitlab.com/fixture/private"}
|
||||
assert used.instrument["pairs"]["skipped_values"] == 1
|
||||
assert used.instrument["pointers"]["skipped_values"] == 1
|
||||
assert used.sources["sweep"] == set()
|
||||
e.canonicalize_used(used, FakeAPI({})) # Non-GitHub identities never reach this API.
|
||||
assert e.rejection(e.resolve("https://gitlab.com/fixture/private", FakeAPI({})), used, set()) == "pairs"
|
||||
report = used.report(set(e.SOURCES) - {"pairs"})
|
||||
assert report["sources"]["pairs"]["skipped_values"] == 1
|
||||
assert "fixture" not in json.dumps(report)
|
||||
assert capsys.readouterr() == ("", "")
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("value", "expected"),
|
||||
[
|
||||
("https://bitbucket.org/Fixture/Repo.git", "bitbucket.org/fixture/repo"),
|
||||
("https://git.example.org/Fixture/Repo", "git.example.org/fixture/repo"),
|
||||
("https://github.com/Fixture/Repo/tree/main", "fixture/repo"),
|
||||
("Fixture/Repo.git/", "fixture/repo"),
|
||||
],
|
||||
)
|
||||
def test_repository_hosts(value, expected):
|
||||
assert e.normalize(value) == expected
|
||||
assert e.normalize(expected) == expected
|
||||
|
||||
|
||||
def test_memory_values_are_skipped_and_counted(tmp_path, capsys):
|
||||
rows = [
|
||||
{"kind": "example", "repo": value}
|
||||
for value in ["local-slice", "free text https://github.com/fixture/prose", None, "https://bitbucket.org/fixture/repo"]
|
||||
]
|
||||
plant(tmp_path, "memory/repos/opaque/pin.jsonl", "\n".join(json.dumps(row) for row in rows))
|
||||
used = e.build_used(tmp_path, stores={"local": FileStore(tmp_path / "memory")})
|
||||
assert used.sources["memory_local"] == {"bitbucket.org/fixture/repo"}
|
||||
assert used.instrument["memory_local"]["skipped_values"] == 3
|
||||
assert capsys.readouterr() == ("", "")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("step", ["open_stores", "build_used", "report", "canonicalize", "verify_tests"])
|
||||
def test_cli_failure_reports_only_step_and_exception_type(tmp_path, monkeypatch, capsys, step):
|
||||
from openultrasast import config
|
||||
from openultrasast.plane import memory
|
||||
|
||||
def fail(*args, **kwargs):
|
||||
raise RuntimeError("sensitive/identity")
|
||||
|
||||
monkeypatch.setattr(config, "load_dotenv", lambda path: None)
|
||||
monkeypatch.setattr(memory, "open_store", lambda *args, **kwargs: object())
|
||||
monkeypatch.setattr(e, "build_used", lambda *args, **kwargs: e.UsedSet())
|
||||
monkeypatch.setattr(e.UsedSet, "report", lambda *args, **kwargs: {})
|
||||
monkeypatch.setattr(e, "GitHub", lambda token: object())
|
||||
monkeypatch.setattr(e, "canonicalize_used", lambda *args: None)
|
||||
monkeypatch.setattr(e, "verify_tests", lambda *args: {})
|
||||
if step == "open_stores":
|
||||
monkeypatch.setattr(memory, "open_store", fail)
|
||||
elif step == "report":
|
||||
monkeypatch.setattr(e.UsedSet, "report", fail)
|
||||
else:
|
||||
monkeypatch.setattr(e, "canonicalize_used" if step == "canonicalize" else step, fail)
|
||||
assert e.main(["--root", str(tmp_path), "--verify-tests"]) == 1
|
||||
output = capsys.readouterr()
|
||||
assert output.err == ""
|
||||
assert json.loads(output.out) == {"status": "instrument_failure", "step": step, "type": "RuntimeError"}
|
||||
assert "sensitive" not in output.out
|
||||
@@ -0,0 +1,314 @@
|
||||
"""Offline extraction: an injected local Git client over a synthetic temporary history."""
|
||||
|
||||
import importlib
|
||||
import subprocess
|
||||
import sys
|
||||
from dataclasses import replace
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
e = importlib.import_module("benchmarks.unseen.extract")
|
||||
s = importlib.import_module("benchmarks.unseen.source")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def history(tmp_path):
|
||||
calls = []
|
||||
|
||||
def runner(argv, **kwargs):
|
||||
assert not {"clone", "fetch", "push", "pull"} & set(argv)
|
||||
calls.append(argv)
|
||||
return subprocess.run(argv, **kwargs)
|
||||
|
||||
git = e.Git(tmp_path, runner=runner)
|
||||
git.run("init", "-q")
|
||||
git.run("config", "user.name", "Fixture")
|
||||
git.run("config", "user.email", "[email protected]")
|
||||
|
||||
def commit(text, message="ordinary", extra=None):
|
||||
(tmp_path / "app.py").write_text(text)
|
||||
for path, content in (extra or {}).items():
|
||||
(tmp_path / path).write_text(content)
|
||||
git.run("add", ".")
|
||||
git.run("commit", "-q", "-m", message)
|
||||
return git.run("rev-parse", "HEAD").strip()
|
||||
|
||||
root = commit("def handler(value):\n return value\n")
|
||||
git.run("tag", "v1.0")
|
||||
intro = commit("def handler(value):\n return eval(value)\n")
|
||||
git.run("tag", "v1.1")
|
||||
parent = commit("def handler(value):\n return eval( value )\n", "format")
|
||||
fix = commit("def handler(value):\n return int(value)\n", "security fix")
|
||||
candidate = s.Candidate(
|
||||
"fixture/project",
|
||||
"https://github.com/fixture/project",
|
||||
fix,
|
||||
parent,
|
||||
"GHSA-test-test-test",
|
||||
"pip",
|
||||
"injection",
|
||||
True,
|
||||
"1.1",
|
||||
"MIT",
|
||||
"permissive",
|
||||
parent,
|
||||
"LICENSE",
|
||||
)
|
||||
return git, commit, candidate, root, intro, parent, calls
|
||||
|
||||
|
||||
def test_introducing_with_reformat_walkback(history):
|
||||
git, _, candidate, root, intro, _, calls = history
|
||||
change = e.introducing(git, candidate)
|
||||
assert change["method"] == "introducing"
|
||||
assert (change["base"], change["head"]) == (root, intro)
|
||||
assert change["function"] == "handler"
|
||||
assert change["family"] == "injection" and change["post_cutoff"]
|
||||
assert change["files"][0]["head_lines"] == [[2, 2]]
|
||||
assert change["bias"] and change["advisory"] == candidate.advisory
|
||||
assert any("-w" in args and "-M" in args and "-C" in args for args in calls)
|
||||
assert git.bytes_read > 0
|
||||
|
||||
|
||||
def test_bracket_rejection_uses_fallback(history):
|
||||
git, _, candidate, _, _, parent, _ = history
|
||||
change = e.introducing(git, replace(candidate, first_affected="1.0"))
|
||||
assert change["method"] == "last_touch" and change["head"] == parent
|
||||
assert change["fallback_reasons"]["version_bracket"] >= 1
|
||||
|
||||
|
||||
def test_size_cap_and_merge_rejection(history):
|
||||
git, _, candidate, _, _, _, _ = history
|
||||
|
||||
class Oversize:
|
||||
def __getattr__(self, name):
|
||||
return getattr(git, name)
|
||||
|
||||
def stats(self, base, head):
|
||||
return [("app.py", 2001, 0)]
|
||||
|
||||
assert e.introducing(Oversize(), candidate)["method"] == "last_touch"
|
||||
|
||||
class Merge(Oversize):
|
||||
def parents(self, sha):
|
||||
parents = git.parents(sha)
|
||||
return parents + ["f" * 40] if sha != candidate.fix else parents
|
||||
|
||||
with pytest.raises(s.Rejected, match="no_range"):
|
||||
e.introducing(Merge(), candidate)
|
||||
|
||||
|
||||
def test_pure_addition_blames_function_declaration(history):
|
||||
git, commit, candidate, _, _, _, _ = history
|
||||
git.run("checkout", "-q", candidate.parent)
|
||||
parent = commit("def handler(value):\n return eval(value)\n\ndef added(value):\n return value\n")
|
||||
fix = commit("def handler(value):\n return eval(value)\n\ndef added(value):\n assert value\n return value\n")
|
||||
change = e.introducing(git, replace(candidate, fix=fix, parent=parent, first_affected=None, family="access_control"))
|
||||
assert change["head"] == parent and change["method"] == "introducing"
|
||||
assert change["function"] == "added"
|
||||
|
||||
|
||||
def test_git_failures_are_sanitized_and_not_empty(tmp_path):
|
||||
def failed(argv, **kwargs):
|
||||
return subprocess.CompletedProcess(argv, 128, b"", b"private repository secret")
|
||||
|
||||
with pytest.raises(e.InstrumentFailure, match="git_exit_128"):
|
||||
e.Git(tmp_path, runner=failed).run("log")
|
||||
|
||||
|
||||
def test_walkback_is_bounded_and_file_cap_falls_back(history):
|
||||
git, _, candidate, _, _, parent, _ = history
|
||||
|
||||
class FileCap:
|
||||
def __getattr__(self, name):
|
||||
return getattr(git, name)
|
||||
|
||||
def stats(self, base, head):
|
||||
return [("app.py", 1, 0)] * 51
|
||||
|
||||
result = e.introducing(FileCap(), candidate)
|
||||
assert result["method"] == "last_touch"
|
||||
assert result["fallback_reasons"]["size_cap"]
|
||||
|
||||
class Walk:
|
||||
calls = 0
|
||||
|
||||
def __getattr__(self, name):
|
||||
return getattr(git, name)
|
||||
|
||||
def blame(self, revision, path, line):
|
||||
self.calls += 1
|
||||
return parent, path, 2
|
||||
|
||||
def parents(self, revision):
|
||||
return [parent] if revision != candidate.fix else [candidate.parent]
|
||||
|
||||
walk = Walk()
|
||||
result = e.introducing(walk, candidate)
|
||||
assert walk.calls == 6
|
||||
assert result["fallback_reasons"]["walkback_limit"] == 1
|
||||
|
||||
|
||||
def test_root_rejected_and_head_function_resolved(history):
|
||||
git, _, candidate, root, _, _, _ = history
|
||||
|
||||
class Root:
|
||||
def __getattr__(self, name):
|
||||
return getattr(git, name)
|
||||
|
||||
def blame(self, revision, path, line):
|
||||
return root, path, 1
|
||||
|
||||
result = e.introducing(Root(), candidate)
|
||||
assert result["method"] == "last_touch"
|
||||
assert result["fallback_reasons"]["merge_or_root"] == 1
|
||||
assert result["function_lines"] == [1, 2]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"path,text",
|
||||
[
|
||||
("a.php", "<?php\nfunction handle($x) {\n return $x;\n}\n"),
|
||||
("a.js", "function handle(x) {\n return x;\n}\n"),
|
||||
("a.ts", "const handle = (x) => {\n return x;\n}\n"),
|
||||
("A.java", "class A {\n public String handle(String x) {\n return x;\n }\n}\n"),
|
||||
],
|
||||
)
|
||||
def test_supported_function_boundaries(path, text):
|
||||
ranges = e.functions(path, text)
|
||||
assert len(ranges) == 1 and ranges[0][0] == "handle"
|
||||
assert ranges[0][2] < len(text.splitlines()) + 1
|
||||
|
||||
|
||||
def test_ambiguous_parent_is_not_proof_of_absence(history):
|
||||
git, commit, candidate, _, _, _, _ = history
|
||||
duplicate = (
|
||||
"class A:\n def handler(self, value):\n return eval(value)\n\n"
|
||||
"class B:\n def handler(self, value):\n return value\n"
|
||||
)
|
||||
commit(duplicate)
|
||||
parent = commit("class A:\n def handler(self, value):\n return eval( value )\n")
|
||||
fix = commit("class A:\n def handler(self, value):\n return int(value)\n")
|
||||
selected = replace(candidate, fix=fix, parent=parent, first_affected=None)
|
||||
|
||||
class BlameLast:
|
||||
def __getattr__(self, name):
|
||||
return getattr(git, name)
|
||||
|
||||
def blame(self, revision, path, line):
|
||||
return parent, path, 3
|
||||
|
||||
result = e.introducing(BlameLast(), selected)
|
||||
assert result["method"] == "last_touch"
|
||||
assert result["fallback_reasons"]["ambiguous_parent"] == 1
|
||||
|
||||
|
||||
def ordinary(index, **overrides):
|
||||
row = {
|
||||
"head": f"{index:040x}",
|
||||
"parents": ["f" * 40],
|
||||
"timestamp": index * 100,
|
||||
"message": "update query token session valid",
|
||||
"author": "Human",
|
||||
"files": [{"path": "app.py", "added": 5, "removed": 0, "head_lines": [[2, 6]], "functions": ["ordinary"], "generated": False}],
|
||||
}
|
||||
row.update(overrides)
|
||||
return row
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"message",
|
||||
[
|
||||
"security fix",
|
||||
"CVE-2026",
|
||||
"GHSA-test",
|
||||
"vulnerability",
|
||||
"injection fix",
|
||||
"XSS",
|
||||
"CSRF",
|
||||
"SSRF",
|
||||
"exploit",
|
||||
"sanitize",
|
||||
"escaping",
|
||||
"Revert changes",
|
||||
"This reverts commit abc",
|
||||
"bump package",
|
||||
"upgrade package from 1 to 2",
|
||||
],
|
||||
)
|
||||
def test_ordinary_message_exclusions(message):
|
||||
assert e.ordinary_rejection(ordinary(1, message=message), set(), set()) is not None
|
||||
|
||||
|
||||
def test_narrow_filter_and_every_other_exclusion():
|
||||
assert e.ordinary_rejection(ordinary(1), set(), set()) is None
|
||||
assert e.ordinary_rejection(ordinary(1, parents=["a", "b"]), set(), set()) == "merge_or_root"
|
||||
assert e.ordinary_rejection(ordinary(1, author="dependabot[bot]"), set(), set()) == "dependency"
|
||||
assert e.ordinary_rejection(ordinary(1, author="renovate"), set(), set()) == "dependency"
|
||||
assert e.ordinary_rejection(ordinary(1), {f"{1:040x}"}, set()) == "advisory_fix"
|
||||
assert e.ordinary_rejection(ordinary(1), set(), {("app.py", "ordinary")}) == "vulnerable_function"
|
||||
row = ordinary(1)
|
||||
row["files"][0]["added"] = 1001
|
||||
assert e.ordinary_rejection(row, set(), set()) == "size_cap"
|
||||
for path, reason in [
|
||||
("README.md", "no_source"),
|
||||
("package.json", "manifest_only"),
|
||||
("package-lock.json", "manifest_only"),
|
||||
("vendor/a.py", "generated"),
|
||||
("node_modules/a.js", "generated"),
|
||||
("dist/a.js", "generated"),
|
||||
("app.min.js", "generated"),
|
||||
]:
|
||||
row = ordinary(1)
|
||||
row["files"][0]["path"] = path
|
||||
assert e.ordinary_rejection(row, set(), set()) == reason
|
||||
row = ordinary(1)
|
||||
row["files"][0]["generated"] = True
|
||||
assert e.ordinary_rejection(row, set(), set()) == "generated"
|
||||
|
||||
|
||||
def test_draw_seed_proportional_cells_floor_and_thirds():
|
||||
rows = [ordinary(i) for i in range(120)]
|
||||
for i, row in enumerate(rows):
|
||||
row["files"][0]["added"] = [5, 20, 100, 500][i % 4]
|
||||
drawn, counts = e.draw_ordinary(rows, set(), set(), seed=19)
|
||||
assert len(drawn) == 12 and len({r["head"] for r in drawn}) == 12
|
||||
assert {r["time_third"] for r in drawn} == {0, 1, 2}
|
||||
assert len({(r["time_third"], r["size_bucket"]) for r in drawn}) == 12
|
||||
assert drawn == e.draw_ordinary(list(reversed(rows)), set(), set(), seed=19)[0]
|
||||
assert counts["eligible"] == 120
|
||||
with pytest.raises(s.Rejected, match="ordinary_floor"):
|
||||
e.draw_ordinary(rows[:39], set(), set(), seed=19)
|
||||
# Thirds come from the history's time span, not rank among eligible changes.
|
||||
with pytest.raises(s.Rejected, match="time_thirds"):
|
||||
e.draw_ordinary([ordinary(i) for i in range(40)] + [ordinary(10000, message="security")], set(), set(), seed=1)
|
||||
|
||||
|
||||
def test_git_ordinary_collection_generated_attributes_and_headers(history):
|
||||
git, commit, _, _, _, _, _ = history
|
||||
commit("# Generated by fixture\nx = 2\n", extra={".gitattributes": "other.py linguist-generated=true\n", "other.py": "x = 1\n"})
|
||||
rows = list(e.ordinary_history(git, "HEAD"))
|
||||
assert rows and rows[0]["files"]
|
||||
files = {f["path"]: f for f in rows[0]["files"]}
|
||||
assert files["app.py"]["generated"] and files["other.py"]["generated"]
|
||||
assert files["app.py"]["head_lines"]
|
||||
|
||||
|
||||
def test_attribute_unset_and_mixed_generated_source(history):
|
||||
git, commit, _, _, _, _, _ = history
|
||||
sha = commit("x = 1\n", extra={".gitattributes": "*.py linguist-generated\napp.py -linguist-generated\n"})
|
||||
assert not e.generated(git, sha, "app.py", "x = 1\n")
|
||||
row = ordinary(1)
|
||||
row["files"].append({**row["files"][0], "path": "dist/b.js", "generated": True})
|
||||
assert e.ordinary_rejection(row, set(), set()) is None
|
||||
|
||||
|
||||
def test_unbalanced_cells_remain_proportional_and_seed_changes_draw():
|
||||
rows = [ordinary(i) for i in range(120)]
|
||||
for i, row in enumerate(rows):
|
||||
row["files"][0]["added"] = 5 if i % 4 else 100
|
||||
draw, _ = e.draw_ordinary(rows, set(), set(), seed=1)
|
||||
assert len([r for r in draw if r["size_bucket"] == 0]) == 9
|
||||
assert draw != e.draw_ordinary(rows, set(), set(), seed=2)[0]
|
||||
@@ -0,0 +1,16 @@
|
||||
"""Static image contract; no Docker invocation."""
|
||||
|
||||
import hashlib
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
ENGINE_SHA = "06ec67d411cd7abbfcd0735b1c2a70e4f3eb65ddcfeef89975709c664f59ad42"
|
||||
|
||||
|
||||
def test_image_stages():
|
||||
text = (ROOT / "plane/Dockerfile.engine-task").read_text()
|
||||
engine, replay = text.split("FROM engine AS replay", 1)
|
||||
original = engine.replace("v0.3.1 AS engine", "v0.3.1").rstrip() + "\n"
|
||||
assert hashlib.sha256(original.encode()).hexdigest() == ENGINE_SHA
|
||||
assert "FROM ghcr.io/norandom/ax-task-runner:v0.3.1 AS engine" in engine
|
||||
assert "git" in replay and "replay_entry.py" in replay and "ousast pre-push --help" in replay
|
||||
@@ -0,0 +1,63 @@
|
||||
"""Append-only use and tamper detection, with opaque fixture identifiers."""
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
with pytest.MonkeyPatch.context() as patch:
|
||||
patch.syspath_prepend(str(Path(__file__).resolve().parents[1] / "benchmarks/unseen"))
|
||||
import ledger
|
||||
|
||||
|
||||
def row(**extra):
|
||||
return {
|
||||
"slice": 1,
|
||||
"decision": "decision-001",
|
||||
"purpose": "baseline",
|
||||
"date": "2026-10-04",
|
||||
"result_digest": "a" * 64,
|
||||
"freeze_digest": "b" * 64,
|
||||
**extra,
|
||||
}
|
||||
|
||||
|
||||
def test_append_and_refuse_rewrite(tmp_path):
|
||||
path = tmp_path / "usage-p1.jsonl"
|
||||
ledger.append(path, row())
|
||||
original = path.read_bytes()
|
||||
ledger.append(path, row(purpose="exploratory"))
|
||||
assert path.read_bytes().startswith(original)
|
||||
assert len(ledger.read(path)) == 2
|
||||
with pytest.raises(ValueError, match="duplicate"):
|
||||
ledger.append(path, row())
|
||||
|
||||
|
||||
def test_informed_slice_cannot_qualify_same_decision(tmp_path):
|
||||
path = tmp_path / "usage-p1.jsonl"
|
||||
ledger.append(path, row(purpose="informed"))
|
||||
before = path.read_bytes()
|
||||
with pytest.raises(ValueError, match="informed"):
|
||||
ledger.append(path, row(purpose="qualifies"))
|
||||
assert path.read_bytes() == before
|
||||
ledger.append(path, row(purpose="qualifies", slice=2))
|
||||
ledger.append(path, row(purpose="qualifies", decision="decision-002"))
|
||||
|
||||
|
||||
def test_rewritten_row_breaks_digest_chain(tmp_path):
|
||||
path = tmp_path / "usage-p1.jsonl"
|
||||
ledger.append(path, row())
|
||||
ledger.append(path, row(purpose="exploratory"))
|
||||
rows = [json.loads(line) for line in path.read_text().splitlines()]
|
||||
rows[0]["purpose"] = "informed"
|
||||
path.write_text("".join(json.dumps(r) + "\n" for r in rows))
|
||||
with pytest.raises(ValueError, match="chain"):
|
||||
ledger.read(path)
|
||||
with pytest.raises(ValueError, match="chain"):
|
||||
ledger.append(path, row(slice=2))
|
||||
|
||||
|
||||
@pytest.mark.parametrize("extra", [{"decision": "owner/name"}, {"purpose": "other"}, {"freeze_digest": "bad"}, {"slice": 0}])
|
||||
def test_invalid_rows_refused(tmp_path, extra):
|
||||
with pytest.raises(ValueError):
|
||||
ledger.append(tmp_path / "usage-p1.jsonl", row(**extra))
|
||||
@@ -0,0 +1,244 @@
|
||||
"""Offline memory and semantic regression checks using synthetic identities only."""
|
||||
|
||||
import importlib
|
||||
import json
|
||||
import sys
|
||||
import tracemalloc
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
d = importlib.import_module("benchmarks.unseen.draft")
|
||||
s = importlib.import_module("benchmarks.unseen.source")
|
||||
j = importlib.import_module("benchmarks.unseen.journal")
|
||||
storage = importlib.import_module("benchmarks.unseen.storage")
|
||||
|
||||
|
||||
def advisory(index):
|
||||
return {
|
||||
"ghsa_id": f"GHSA-test-{index:08d}",
|
||||
"published_at": "2026-06-02T00:00:00Z" if index % 3 else "2025-01-01T00:00:00Z",
|
||||
"type": "reviewed",
|
||||
"withdrawn_at": None,
|
||||
"description": "An affected handler accepts untrusted input. " * 400,
|
||||
"cwes": [{"cwe_id": "CWE-89", "name": "SQL injection"}],
|
||||
"references": [
|
||||
{"url": f"https://github.com/fixture/project{index}/commit/{index:040x}"},
|
||||
*[{"url": f"https://example.invalid/details/{n}"} for n in range(30)],
|
||||
],
|
||||
"vulnerabilities": [
|
||||
{
|
||||
"package": {"ecosystem": s.ECOSYSTEMS[index % 4], "name": f"package-{n}"},
|
||||
"vulnerable_version_range": ", ".join(f">= {v}.0, < {v}.9" for v in range(50)),
|
||||
"first_patched_version": "50.0",
|
||||
}
|
||||
for n in range(8)
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
class OfflineAPI:
|
||||
def advisory_page(self, page):
|
||||
pytest.fail("A complete journal must not fetch advisories")
|
||||
|
||||
def get_repo(self, name):
|
||||
return {"full_name": name}
|
||||
|
||||
|
||||
def test_replay_20000_advisories_has_bounded_peak_memory(tmp_path):
|
||||
path = tmp_path / "journal"
|
||||
path.mkdir()
|
||||
(path / "inputs.json").write_text('{"seed": 19}')
|
||||
journal_file = path / "advisories.jsonl"
|
||||
# Write old, unprojected REST pages without ever building the entire fixture in RAM.
|
||||
with journal_file.open("w") as handle:
|
||||
for page in range(200):
|
||||
json.dump(
|
||||
{
|
||||
"id": str(page + 1),
|
||||
"value": {"rows": [advisory(i) for i in range(page * 100, (page + 1) * 100)], "done": page == 199},
|
||||
"elapsed_seconds": page,
|
||||
"rate_limit": {"remaining": "10"},
|
||||
},
|
||||
handle,
|
||||
)
|
||||
handle.write("\n")
|
||||
with journal_file.open("rb") as handle:
|
||||
first_line = handle.readline()
|
||||
assert len(json.loads(first_line)["value"]["rows"]) == 100
|
||||
assert len(first_line) > 2 * 1024 * 1024
|
||||
del first_line
|
||||
input_bytes = journal_file.stat().st_size
|
||||
assert input_bytes > 400 * 1024 * 1024
|
||||
print(f"journal_bytes_read={input_bytes}")
|
||||
tracemalloc.start()
|
||||
try:
|
||||
with j.Journal(path).open({"seed": 19}) as journal:
|
||||
assert len(journal.rows["advisories"]) == 200
|
||||
assert journal.elapsed_before == 199
|
||||
rows = d.enumerate_advisories(OfflineAPI(), journal)
|
||||
assert len(rows) == 20_000
|
||||
index = d.advisory_index(rows, OfflineAPI(), db=journal.db)
|
||||
assert index.get("fixture/project19999") == {f"{19999:040x}"}
|
||||
count = 0
|
||||
for row in s.ordered_advisories(rows, 19):
|
||||
assert "description" not in row
|
||||
assert all(set(v) == {"package"} for v in row["vulnerabilities"])
|
||||
count += 1
|
||||
assert count == 20_000
|
||||
assert journal.last == {"stage": "advisories", "id": "200"}
|
||||
_, peak = tracemalloc.get_traced_memory()
|
||||
finally:
|
||||
tracemalloc.stop()
|
||||
print(f"replay_enumeration_order_index_peak_bytes={peak}")
|
||||
assert peak < 64 * 1024 * 1024
|
||||
assert not list(path.glob("sourcing-*"))
|
||||
|
||||
|
||||
def test_disk_order_matches_original_seeded_order_and_duplicate_last_wins(tmp_path):
|
||||
raw = [advisory(i) for i in range(93)]
|
||||
raw[0]["vulnerabilities"] = [{"package": {"ecosystem": "other", "name": "fixture"}}]
|
||||
# One advisory lists multiple ecosystems; retain the original ecosystem priority.
|
||||
raw[1]["vulnerabilities"].append({"package": {"ecosystem": "pip", "name": "extra"}})
|
||||
expected = [s.compact_advisory(row) for row in raw]
|
||||
with storage.working_store(tmp_path) as db:
|
||||
rows = storage.AdvisoryRows(db, "advisories")
|
||||
for row in reversed(expected):
|
||||
rows[row["ghsa_id"]] = row
|
||||
rows[expected[2]["ghsa_id"]] = expected[2]
|
||||
assert len(rows) == len(expected)
|
||||
assert list(s.ordered_advisories(rows, 17)) == s.ordered_advisories(expected, 17)
|
||||
|
||||
|
||||
def test_compaction_preserves_candidate_and_all_fix_links():
|
||||
fixtures = importlib.import_module("test_unseen_source")
|
||||
row = fixtures.advisory()
|
||||
assert s.candidate(s.compact_advisory(row), fixtures.API(), fixtures.OSV()) == s.candidate(row, fixtures.API(), fixtures.OSV())
|
||||
row["references"] += [
|
||||
"https://github.com/fixture/old/commit/ABCDEF1#diff",
|
||||
{"url": "https://github.com/fixture/other/commit/" + "b" * 40},
|
||||
]
|
||||
assert s.fix_links(s.compact_advisory(row)) == s.fix_links(row)
|
||||
repo = {
|
||||
"full_name": "fixture/new",
|
||||
"private": False,
|
||||
"size": 10,
|
||||
"clone_url": "https://github.com/fixture/new",
|
||||
"source": {"full_name": "fixture/source", "description": "large"},
|
||||
"parent": {"full_name": "fixture/parent", "description": "large"},
|
||||
"former_names": ["fixture/old"],
|
||||
}
|
||||
eligibility = importlib.import_module("benchmarks.unseen.eligibility")
|
||||
|
||||
class API:
|
||||
def __init__(self, data):
|
||||
self.data = data
|
||||
|
||||
def get_repo(self, name):
|
||||
return self.data
|
||||
|
||||
assert eligibility.resolve("fixture/old", API(repo)) == eligibility.resolve("fixture/old", API(s.compact_repository(repo)))
|
||||
|
||||
|
||||
def test_legacy_journal_migration_keeps_payload_and_refuses_changed_parameters(tmp_path):
|
||||
fixtures = importlib.import_module("test_unseen_draft")
|
||||
used = fixtures.e.UsedSet()
|
||||
parameters = d.inputs(used, 19, set(used.sources), fixtures.FakeClones())
|
||||
legacy = d.legacy_inputs(parameters)
|
||||
path = tmp_path / "journal"
|
||||
with j.Journal(path).open(legacy) as journal:
|
||||
journal.put("advisories", "1", {"rows": [advisory(1)], "done": True})
|
||||
original = (path / "advisories.jsonl").read_bytes()
|
||||
for changed in ({**parameters, "seed": 20}, {**parameters, "design_taxonomy": "different"}):
|
||||
with pytest.raises(j.ResumeMismatch), j.Journal(path).open(changed, compatible_parameters=d.legacy_inputs(changed)):
|
||||
pass
|
||||
with j.Journal(path).open(parameters, compatible_parameters=legacy) as journal:
|
||||
assert journal.rows["advisories"]["1"]["rows"][0]["ghsa_id"] == advisory(1)["ghsa_id"]
|
||||
assert json.loads((path / "inputs.json").read_text()) == parameters
|
||||
assert json.loads((path / "inputs-before-memory-fix.json").read_text()) == legacy
|
||||
assert (path / "advisories.jsonl").read_bytes() == original
|
||||
|
||||
|
||||
def test_replay_duplicate_torn_and_oversized_records_fail_closed(tmp_path, monkeypatch):
|
||||
path = tmp_path / "journal"
|
||||
with j.Journal(path).open({}) as journal:
|
||||
journal.put("advisories", "1", {"rows": [], "done": True})
|
||||
record = (path / "advisories.jsonl").read_bytes()
|
||||
for data, reason in ((record * 2, "duplicate_unit"), (record[:-1], "incomplete_record")):
|
||||
(path / "advisories.jsonl").write_bytes(data)
|
||||
with pytest.raises(ValueError, match=reason), j.Journal(path).open({}):
|
||||
pass
|
||||
assert not list(path.glob("sourcing-*"))
|
||||
monkeypatch.setattr(j, "MAX_RECORD_BYTES", 16)
|
||||
(path / "advisories.jsonl").write_bytes(record)
|
||||
with pytest.raises(ValueError, match="record_size_cap"), j.Journal(path).open({}):
|
||||
pass
|
||||
|
||||
|
||||
def test_ordinary_sampling_streams_payloads_and_preserves_frozen_draw():
|
||||
fixtures = importlib.import_module("test_unseen_extract")
|
||||
extract = fixtures.e
|
||||
rows = [fixtures.ordinary(i) for i in range(120)]
|
||||
for i, row in enumerate(rows):
|
||||
row["files"][0]["added"] = [5, 20, 100, 500][i % 4]
|
||||
# Captured from the pre-memory-fix implementation, not a second copy of its algorithm.
|
||||
drawn, counts = extract.draw_ordinary(iter(rows), set(), set(), seed=19)
|
||||
assert [int(row["head"], 16) for row in drawn] == [0, 33, 6, 35, 52, 65, 62, 75, 96, 117, 90, 119]
|
||||
assert counts == {"eligible": 120}
|
||||
|
||||
def history():
|
||||
for i in range(20_000):
|
||||
row = fixtures.ordinary(i)
|
||||
row["message"] = "ordinary change " * 1500
|
||||
yield row
|
||||
|
||||
tracemalloc.start()
|
||||
try:
|
||||
drawn, counts = extract.draw_ordinary(history(), set(), set(), seed=19)
|
||||
_, peak = tracemalloc.get_traced_memory()
|
||||
finally:
|
||||
tracemalloc.stop()
|
||||
assert counts == {"eligible": 20_000}
|
||||
assert len(drawn) == 12
|
||||
assert peak < 16 * 1024 * 1024
|
||||
|
||||
|
||||
def test_git_spools_large_output_and_refuses_large_blob_before_reading(tmp_path):
|
||||
import subprocess
|
||||
|
||||
extract = importlib.import_module("benchmarks.unseen.extract")
|
||||
|
||||
def runner(argv, **kwargs):
|
||||
assert kwargs["stdout"] != subprocess.PIPE
|
||||
assert kwargs["stderr"] != subprocess.PIPE
|
||||
for _ in range(9):
|
||||
kwargs["stdout"].write(b"x" * 1024 * 1024)
|
||||
return subprocess.CompletedProcess(argv, 0)
|
||||
|
||||
with pytest.raises(extract.InstrumentFailure, match="git_output_size_cap"):
|
||||
extract.Git(tmp_path, runner=runner).run("log")
|
||||
calls = []
|
||||
git = extract.Git(tmp_path)
|
||||
|
||||
def run(*args):
|
||||
calls.append(args[0])
|
||||
return "app.py" if args[0] == "ls-tree" else str(512 * 1024 + 1)
|
||||
|
||||
git.run = run
|
||||
with pytest.raises(extract.InstrumentFailure, match="source_blob_size_cap"):
|
||||
git.blob("a" * 40, "app.py")
|
||||
assert calls == ["ls-tree", "cat-file"]
|
||||
assert git.bytes_read == git.files_read == 0
|
||||
|
||||
|
||||
def test_repository_cache_is_bounded_and_drops_payloads():
|
||||
class Transport:
|
||||
def get(self, url, headers):
|
||||
return {"full_name": url.rsplit("/", 2)[-2] + "/" + url.rsplit("/", 1)[-1], "description": "large" * 1000}
|
||||
|
||||
github = s.GitHub("offline", transport=Transport())
|
||||
for i in range(200):
|
||||
github.get_repo(f"fixture/project{i}")
|
||||
assert len(github.repos) == 128
|
||||
assert all("description" not in row for row in github.repos.values())
|
||||
@@ -0,0 +1,56 @@
|
||||
"""Proof records fail closed using synthetic local results."""
|
||||
|
||||
import importlib
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
p = importlib.import_module("benchmarks.unseen.proof")
|
||||
|
||||
|
||||
def test_replay_proof(tmp_path):
|
||||
roots = {}
|
||||
for lane in ("kind", "ax", "vm"):
|
||||
root = tmp_path / lane
|
||||
root.mkdir()
|
||||
roots[lane] = root
|
||||
(root / "inputs.json").write_text("[]")
|
||||
for i in range(5):
|
||||
(root / f"{i}.json").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"input_digest": str(i),
|
||||
"status": "engine_covered",
|
||||
"instrument": {"hook_bytes_read": 20, "jvm_wall_seconds": 8, "seconds": 30, "hook_exit": 0},
|
||||
"push": {"quick_tier": [{"findings": []}]},
|
||||
}
|
||||
)
|
||||
)
|
||||
assert p.replay_proof(roots)["quick_tier_equal"]
|
||||
path = roots["vm"] / "0.json"
|
||||
record = json.loads(path.read_text())
|
||||
record["instrument"]["jvm_wall_seconds"] = 0
|
||||
path.write_text(json.dumps(record))
|
||||
with pytest.raises(ValueError, match="instrument"):
|
||||
p.replay_proof(roots)
|
||||
|
||||
|
||||
def test_engine_proof_counts_differences_without_names(tmp_path):
|
||||
current, baseline = tmp_path / "current", tmp_path / "baseline"
|
||||
current.mkdir()
|
||||
baseline.mkdir()
|
||||
for i in range(2):
|
||||
record = {
|
||||
"repo": "synthetic",
|
||||
"pin": str(i),
|
||||
"done": True,
|
||||
"seconds": 20,
|
||||
"units": [{"unit": str(i), "status": "path", "instrument": {"bytes": 20, "jvm": [{"seconds": 8, "success": True}]}}],
|
||||
}
|
||||
(current / f"{i}.json").write_text(json.dumps(record))
|
||||
(baseline / f"{i}.json").write_text(json.dumps(record))
|
||||
proof = p.engine_proof(current, baseline)
|
||||
assert proof["differences"] == 0 and "synthetic" not in json.dumps(proof)
|
||||
@@ -0,0 +1,79 @@
|
||||
"""Replay workload privacy, validation and input identity."""
|
||||
|
||||
import importlib
|
||||
import io
|
||||
import json
|
||||
import sys
|
||||
import tarfile
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
r = importlib.import_module("benchmarks.unseen.replay")
|
||||
|
||||
|
||||
def test_items_private_and_arm_identity():
|
||||
change = {
|
||||
"id": "repo-name-secret",
|
||||
"repo": "owner/private-name",
|
||||
"repository_url": "https://public.example/private-name",
|
||||
"base": "a" * 40,
|
||||
"head": "b" * 40,
|
||||
"files": [{"path": "app.py", "head_lines": [[1, 2]]}],
|
||||
}
|
||||
image = "registry/replay@sha256:" + "a" * 64
|
||||
a = r.make_item(change, "default", image)
|
||||
b = r.make_item(change, "long_deadline", image)
|
||||
assert a.input_digest != b.input_digest
|
||||
assert "private-name" not in a.id and "repo-name" not in a.id
|
||||
assert set(a.inputs) == {"SPEC_URL"} and not a.extra_env
|
||||
spec = json.loads(a.inputs["SPEC_URL"])
|
||||
assert spec["repository_url"] == change["repository_url"]
|
||||
assert json.loads(b.inputs["SPEC_URL"])["hook_flags"] == ["--deadline", "300"]
|
||||
|
||||
|
||||
def test_validator_requires_instrument(tmp_path):
|
||||
buffer = io.BytesIO()
|
||||
with tarfile.open(fileobj=buffer, mode="w") as tar:
|
||||
data = b"{}"
|
||||
info = tarfile.TarInfo("push.json")
|
||||
info.size = len(data)
|
||||
tar.addfile(info, io.BytesIO(data))
|
||||
with pytest.raises(ValueError, match="instrument"):
|
||||
r.validate_result(buffer.getvalue(), tmp_path)
|
||||
|
||||
|
||||
def test_frozen_toml_slice_and_scoring_join(tmp_path):
|
||||
import hashlib
|
||||
|
||||
from benchmarks.unseen.score import load_records
|
||||
|
||||
pool = tmp_path / "pool-p1.toml"
|
||||
pool.write_text("""pool="p1"
|
||||
[[repository]]
|
||||
id="opaque-a"
|
||||
slice=1
|
||||
url="https://public.example/repo-a"
|
||||
changes=[{kind="ordinary",base="aaa",head="bbb",files=[]}]
|
||||
[[repository]]
|
||||
id="opaque-b"
|
||||
slice=2
|
||||
url="https://public.example/repo-b"
|
||||
changes=[{kind="ordinary",base="ccc",head="ddd",files=[]}]
|
||||
""")
|
||||
import tomllib
|
||||
|
||||
freeze = hashlib.sha256(r.canonical(tomllib.loads(pool.read_text()))).hexdigest()
|
||||
(tmp_path / "freeze-p1.json").write_text(json.dumps({"freeze_digest": freeze}))
|
||||
changes = r.load_slice(pool, 2)
|
||||
assert len(changes) == 1 and changes[0]["repo"] == "opaque-b"
|
||||
(tmp_path / "inputs.json").write_text(json.dumps([{"item_id": "replay-opaque", "change": changes[0], "input_digest": "digest"}]))
|
||||
assert load_records(tmp_path)[0]["kind"] == "ordinary"
|
||||
assert "instrument" not in load_records(tmp_path)[0] # Missing result remains a measured failure.
|
||||
result = tmp_path / (hashlib.sha256(b"replay-opaque").hexdigest() + ".json")
|
||||
result.write_text(json.dumps({"input_digest": "digest", "instrument": {"head_verified": True}}))
|
||||
assert load_records(tmp_path)[0]["instrument"]["head_verified"]
|
||||
pool.write_text(pool.read_text() + "changed=true\n")
|
||||
with pytest.raises(ValueError, match="digest"):
|
||||
r.load_slice(pool, 2)
|
||||
@@ -0,0 +1,136 @@
|
||||
"""Replay entry uses real local git and an injected transfer, never a network."""
|
||||
|
||||
import importlib
|
||||
import io
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
e = importlib.import_module("benchmarks.unseen.replay_entry")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def local_repo(tmp_path):
|
||||
repo = tmp_path / "repo"
|
||||
repo.mkdir()
|
||||
|
||||
def git(*args):
|
||||
return subprocess.run(["git", "-C", str(repo), *args], capture_output=True, text=True, check=True).stdout.strip()
|
||||
|
||||
git("init", "-q")
|
||||
git("config", "user.email", "[email protected]")
|
||||
git("config", "user.name", "Fixture")
|
||||
(repo / "app.py").write_text("print(1)\n")
|
||||
git("add", "app.py")
|
||||
git("commit", "-qm", "base")
|
||||
base = git("rev-parse", "HEAD")
|
||||
(repo / "app.py").write_text("print(2)\n")
|
||||
git("commit", "-qam", "head")
|
||||
return repo, base, git("rev-parse", "HEAD")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("failure", [None, "missing", "fetch", "crash", "fast", "unread", "unsupported"])
|
||||
def test_entry(local_repo, tmp_path, monkeypatch, failure):
|
||||
repo, base, head = local_repo
|
||||
spec = {
|
||||
"repository_url": "https://public.example/project.git",
|
||||
"base": base,
|
||||
"head": head,
|
||||
"id": "opaque",
|
||||
"image": "digest",
|
||||
"hook_flags": [],
|
||||
}
|
||||
if failure == "missing":
|
||||
spec["head"] = "a" * 40
|
||||
received = {}
|
||||
|
||||
def transfer(request, destination=None, **kwargs):
|
||||
assert kwargs["readiness_budget"] == 60
|
||||
if destination:
|
||||
destination.write_text(json.dumps(spec))
|
||||
else:
|
||||
with tarfile.open(fileobj=io.BytesIO(request.data)) as archive:
|
||||
received.update({m.name: archive.extractfile(m).read() for m in archive.getmembers()})
|
||||
|
||||
monkeypatch.setenv("SPEC_URL", "https://files.example/spec.dat?secret")
|
||||
monkeypatch.setenv("RESULT_URL", "https://files.example/result.dat?secret")
|
||||
monkeypatch.setattr(e, "transfer", transfer)
|
||||
real_run = subprocess.run
|
||||
|
||||
def run(command, **kwargs):
|
||||
if command[0] == "ousast":
|
||||
assert command[command.index("--base") + 1] == base
|
||||
assert kwargs["env"]["OPENULTRASAST_CPG_HEAP_MB"] == "1700"
|
||||
if failure == "crash":
|
||||
return subprocess.CompletedProcess(command, 2, "", "crash https://secret")
|
||||
target = Path(command[command.index("--artifact") + 1])
|
||||
target.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"quick_tier": [{"files": ["app.py"], "bytes_read": 0 if failure == "unread" else 9, "findings": []}],
|
||||
"scans": [
|
||||
{
|
||||
"side": "head",
|
||||
"scan": {"build_seconds": 0.1 if failure == "fast" else 8, "query_seconds": 2, "question_outcomes": []},
|
||||
}
|
||||
],
|
||||
}
|
||||
)
|
||||
)
|
||||
if failure == "unsupported":
|
||||
target.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"quick_tier": [{"files": [], "bytes_read": 0, "findings": []}],
|
||||
"admission": {"coverage_reasons": ["language_not_covered:rust:1:1"]},
|
||||
"scans": [
|
||||
{
|
||||
"side": "head",
|
||||
"scan": {
|
||||
"build_seconds": 0.01,
|
||||
"query_seconds": 0,
|
||||
"partitions": [
|
||||
{
|
||||
"language": "rust",
|
||||
"frontend": None,
|
||||
"paths": ["app.rs"],
|
||||
"source_bytes": 0,
|
||||
"status": "unsupported",
|
||||
}
|
||||
],
|
||||
},
|
||||
}
|
||||
],
|
||||
}
|
||||
)
|
||||
)
|
||||
return subprocess.CompletedProcess(command, 0, "Skipped: deadline\n", "")
|
||||
command = [
|
||||
str(repo) if arg == spec["repository_url"] else "protocol.file.allow=always" if arg == "protocol.file.allow=never" else arg
|
||||
for arg in command
|
||||
]
|
||||
if failure == "fetch" and "fetch" in command:
|
||||
return subprocess.CompletedProcess(command, 1, "", "fetch secret")
|
||||
return real_run(command, **kwargs)
|
||||
|
||||
result = e.main(work_root=tmp_path / "work", runner=run)
|
||||
instrument = json.loads(received["instrument.json"])
|
||||
assert instrument["image"] == "digest"
|
||||
assert b"secret" not in received["instrument.json"]
|
||||
if failure == "unsupported":
|
||||
from benchmarks.unseen.score import coverage
|
||||
|
||||
assert result == 0 and not instrument["suspect_fast"]
|
||||
assert coverage({"instrument": instrument, "push": json.loads(received["push.json"])}) == "unanalysable"
|
||||
elif failure:
|
||||
assert result == 1 and instrument["status"] == "instrument_failure"
|
||||
else:
|
||||
assert result == 0 and instrument["changed_bytes"] == {"app.py": 9}
|
||||
assert instrument["head_verified"] and instrument["base_verified"]
|
||||
assert instrument["jvm_wall_seconds"] == 10
|
||||
assert b"Skipped:" in received["push.txt"]
|
||||
@@ -0,0 +1,127 @@
|
||||
"""Synthetic push-level statistical controls; no corpus reads."""
|
||||
|
||||
import importlib
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
s = importlib.import_module("benchmarks.unseen.score")
|
||||
|
||||
|
||||
def record(kind="introducing", **updates):
|
||||
return {
|
||||
"id": "a",
|
||||
"repo": "opaque",
|
||||
"kind": kind,
|
||||
"family": "injection",
|
||||
"function": "handler",
|
||||
"files": [{"path": "app.py", "head_lines": [[20, 22]]}],
|
||||
"instrument": {
|
||||
"head_verified": True,
|
||||
"base_verified": True,
|
||||
"changed_bytes": {"app.py": 20},
|
||||
"hook_exit": 0,
|
||||
"hook_bytes_read": 20,
|
||||
},
|
||||
"push": {"quick_tier": [{"bytes_read": 20, "findings": []}]},
|
||||
**updates,
|
||||
}
|
||||
|
||||
|
||||
def test_coverage_and_failure_denominator():
|
||||
r = record()
|
||||
assert s.coverage(r) == "quick_only"
|
||||
r["push"]["scans"] = [{"side": "head", "scan": {"question_outcomes": [{"status": "completed", "identity": {"path": "app.py"}}]}}]
|
||||
assert s.coverage(r) == "engine_covered"
|
||||
r["instrument"]["suspect_fast"] = True
|
||||
assert s.coverage(r) == "instrument_failure"
|
||||
assert s.coverage(record(push={"skipped": ["language_not_covered:other:1:1"]})) == "unanalysable"
|
||||
assert s.coverage({}) == "instrument_failure"
|
||||
with pytest.raises(ValueError, match="5%"):
|
||||
s.score([record(), r])
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"p,width,n", [(0.05, 0.03, 315), (0.1, 0.03, 483), (0.2, 0.03, 756), (0.1, 0.1, 62), (0.25, 0.1, 88), (0.4, 0.1, 97)]
|
||||
)
|
||||
def test_sample_sizes(p, width, n):
|
||||
assert s.required_n(p, width) == n
|
||||
|
||||
|
||||
def test_catch_and_all_stream_false_alarm():
|
||||
r = record()
|
||||
r["push"]["quick_tier"][0]["findings"] = [{"cwe": "CWE-89", "path": "app.py", "line": 20}]
|
||||
assert s.catches(r) == ["quick"]
|
||||
r["push"]["quick_tier"][0]["findings"][0]["line"] = 100
|
||||
assert not s.catches(r)
|
||||
r["kind"] = "ordinary"
|
||||
assert s.false_alarm(r)
|
||||
assert not s.false_alarm(r, alerts_only=True)
|
||||
|
||||
|
||||
def test_location_rules():
|
||||
assert s.location_match(record(family="access_control"), {"path": "app.py", "function": "handler", "line": 200})
|
||||
assert not s.location_match(record(family="config_secrets"), {"path": "app.py", "function": "handler", "line": 200})
|
||||
assert s.location_match(record(family="config_secrets"), {"path": "app.py", "line": 30})
|
||||
assert not s.location_match(record(), {"path": "else.py", "function": "handler", "line": 20})
|
||||
|
||||
|
||||
def test_bootstrap_icc_mcnemar():
|
||||
rows = [("a", 1), ("a", 1), ("b", 0), ("b", 0)]
|
||||
assert s.cluster_interval(rows, seed=7) == s.cluster_interval(rows, seed=7)
|
||||
assert s.icc(rows) == pytest.approx(1)
|
||||
assert s.mcnemar([True] * 6, [False] * 6) == pytest.approx(0.03125)
|
||||
assert s.mcnemar([True], [True]) == 1
|
||||
|
||||
|
||||
def test_budgets():
|
||||
rows = [{"kind": "ordinary", "score": x} for x in range(100)] + [{"kind": "introducing", "score": 99}]
|
||||
result = s.budgets(rows)
|
||||
assert result["5%"]["threshold"] == 95
|
||||
assert result["5%"]["label"] == "exploratory"
|
||||
assert s.budgets(rows, thresholds={"5%": 96})["5%"]["label"] == "fixed"
|
||||
binary = s.budgets([{"kind": "ordinary", "prediction": False}, {"kind": "introducing", "prediction": True}], binary=True)
|
||||
assert binary["budgets_met"] == ["1%", "5%", "10%"]
|
||||
|
||||
|
||||
def test_gate():
|
||||
a = [record(kind="ordinary", id=str(i), repo=str(i)) for i in range(10)] + [record(id="v")]
|
||||
b = [
|
||||
record(kind="ordinary", id=str(i), repo=str(i), push={"quick_tier": [{"findings": [{"cwe": "CWE-89", "path": "x", "line": 1}]}]})
|
||||
for i in range(10)
|
||||
] + [record(id="v")]
|
||||
assert s.gate(a, b, slice_id="p1/2")["gate"] == "regression"
|
||||
assert s.gate(a, b)["gate"] == "not_run"
|
||||
assert s.gate(a, b, slice_id="p1/2", informed=True)["gate"] == "not_run"
|
||||
assert s.gate(a, [{}], slice_id="p1/2")["gate"] == "not_run"
|
||||
|
||||
|
||||
def test_real_artifact_coverage_and_location_binding():
|
||||
r = record(path="app.py", function_lines=[1, 100], files=[{"path": "other.py", "head_lines": [[200, 210]]}])
|
||||
assert not s.location_match(r, {"path": "other.py", "line": 40})
|
||||
assert (
|
||||
s.coverage(
|
||||
record(
|
||||
instrument={"head_verified": True, "base_verified": True, "changed_bytes": {"x": 20}, "hook_exit": 0, "hook_bytes_read": 0},
|
||||
push={"admission": {"coverage_reasons": ["language_not_covered:unknown:1:1"]}},
|
||||
)
|
||||
)
|
||||
== "unanalysable"
|
||||
)
|
||||
r["push"]["quick_tier"][0]["findings"] = [{"cwe": None, "path": "other.py", "line": 200}]
|
||||
assert not s.catches(r)
|
||||
assert s.false_alarm(r)
|
||||
|
||||
|
||||
def test_committed_synthetic_fixture():
|
||||
import json
|
||||
|
||||
fixture = Path(__file__).resolve().parents[1] / "benchmarks/unseen/fixtures/scorer.json"
|
||||
payload = fixture.read_bytes()
|
||||
assert len(payload) > 0
|
||||
report = s.score(json.loads(payload))
|
||||
assert report["catch"]["n"] == 4 and report["catch"]["count"] == 2
|
||||
assert report["false_alarm"]["n"] == 16 and report["false_alarm"]["count"] == 1
|
||||
assert report["coverage"]["quick_only"] == 20
|
||||
@@ -0,0 +1,182 @@
|
||||
"""Sanitized REST-shaped advisory fixtures; every external client is a fake."""
|
||||
|
||||
import copy
|
||||
import importlib
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
s = importlib.import_module("benchmarks.unseen.source")
|
||||
|
||||
|
||||
def advisory():
|
||||
path = Path(__file__).resolve().parents[1] / "benchmarks/unseen/fixtures/advisory.json"
|
||||
return json.loads(path.read_bytes())
|
||||
|
||||
|
||||
class API:
|
||||
def __init__(self):
|
||||
self.parents = ["b" * 40]
|
||||
self.spdx = "MIT"
|
||||
self.license_text = "SPDX-License-Identifier: MIT"
|
||||
self.calls = []
|
||||
self.repo = {"full_name": "fixture/project", "private": False, "size": 100, "clone_url": "https://github.com/fixture/project.git"}
|
||||
|
||||
def get_repo(self, name):
|
||||
return copy.deepcopy(self.repo)
|
||||
|
||||
def get_commit(self, name, sha):
|
||||
return {"sha": sha, "parents": [{"sha": p} for p in self.parents]}
|
||||
|
||||
def get_license(self, name, ref):
|
||||
self.calls.append((name, ref))
|
||||
return {"text": self.license_text, "spdx_id": self.spdx, "path": "LICENSE"}
|
||||
|
||||
|
||||
class OSV:
|
||||
def get(self, identifier):
|
||||
return {
|
||||
"affected": [
|
||||
{
|
||||
"package": {"ecosystem": "PyPI", "name": "fixture"},
|
||||
"ranges": [{"type": "ECOSYSTEM", "events": [{"introduced": "1.2"}, {"fixed": "1.3"}]}],
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
def test_candidate_cwe_cutoff_bracket_and_pinned_license():
|
||||
api = API()
|
||||
candidate = s.candidate(advisory(), api, OSV())
|
||||
assert candidate.family == "injection" and candidate.post_cutoff
|
||||
assert candidate.first_affected == "1.2"
|
||||
assert candidate.license_class == "permissive"
|
||||
assert api.calls == [("fixture/project", "b" * 40)]
|
||||
raw = advisory()
|
||||
raw["published_at"] = "2026-06-01T12:00:00Z"
|
||||
assert not s.candidate(raw, api, OSV()).post_cutoff
|
||||
raw["published_at"] = "2018-12-31T00:00:00Z"
|
||||
with pytest.raises(s.Rejected, match="date"):
|
||||
s.candidate(raw, api, OSV())
|
||||
|
||||
|
||||
@pytest.mark.parametrize("spdx", ["GPL-3.0", "AGPL-3.0-only", "LGPL-2.1", "NOASSERTION", "", "BUSL-1.1", "MIT OR GPL-3.0"])
|
||||
def test_private_licenses(spdx):
|
||||
api = API()
|
||||
api.spdx = spdx
|
||||
api.license_text = ""
|
||||
assert s.candidate(advisory(), api, OSV()).license_class == "private"
|
||||
|
||||
|
||||
def test_license_file_overrides_api_and_missing_file_is_private():
|
||||
api = API()
|
||||
api.license_text = "SPDX-License-Identifier: GPL-3.0-only"
|
||||
assert s.candidate(advisory(), api, OSV()).license_class == "private"
|
||||
api.license_text = ""
|
||||
api.spdx = "Apache-2.0"
|
||||
assert s.candidate(advisory(), api, OSV()).license_spdx == "Apache-2.0"
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"mutation,reason",
|
||||
[
|
||||
(lambda a, api: api.parents.append("c" * 40), "parents"),
|
||||
(lambda a, api: a["references"].append("https://github.com/fixture/other/commit/" + "d" * 40), "fix_links"),
|
||||
(lambda a, api: a.update(cwes=[{"cwe_id": "CWE-99999"}]), "family"),
|
||||
(lambda a, api: api.repo.update(size=512001), "repository"),
|
||||
(lambda a, api: api.repo.update(private=True), "repository"),
|
||||
(lambda a, api: api.repo.update(clone_url="ssh://fixture/project"), "repository"),
|
||||
],
|
||||
)
|
||||
def test_rejections(mutation, reason):
|
||||
api, raw = API(), advisory()
|
||||
mutation(raw, api)
|
||||
with pytest.raises(s.Rejected, match=reason):
|
||||
s.candidate(raw, api, OSV())
|
||||
|
||||
|
||||
def test_post_cutoff_first_within_ecosystem_is_seeded():
|
||||
rows = [advisory() for _ in range(12)]
|
||||
for i, row in enumerate(rows):
|
||||
row["ghsa_id"] += str(i)
|
||||
if i % 2:
|
||||
row["published_at"] = "2025-01-01T00:00:00Z"
|
||||
ordered = s.ordered_advisories(rows, 17)
|
||||
assert ordered == s.ordered_advisories(list(reversed(rows)), 17)
|
||||
assert all(s.post_cutoff(row) for row in ordered[:6])
|
||||
|
||||
|
||||
def test_http_adapters_with_fake_transport():
|
||||
import base64
|
||||
from urllib.parse import parse_qs, urlsplit
|
||||
|
||||
class Transport:
|
||||
def __init__(self):
|
||||
self.calls = []
|
||||
|
||||
def get(self, url, headers):
|
||||
self.calls.append((url, headers))
|
||||
parsed = urlsplit(url)
|
||||
if parsed.path == "/advisories":
|
||||
page = int(parse_qs(parsed.query)["page"][0])
|
||||
return [advisory()] * (100 if page == 1 else 1)
|
||||
if "/license" in parsed.path:
|
||||
assert parse_qs(parsed.query)["ref"] == ["b" * 40]
|
||||
return {
|
||||
"license": {"spdx_id": "MIT"},
|
||||
"path": "LICENSE",
|
||||
"content": base64.b64encode(b"SPDX-License-Identifier: MIT").decode(),
|
||||
}
|
||||
if "/commits/" in parsed.path:
|
||||
return {"sha": "a" * 40, "parents": [{"sha": "b" * 40}]}
|
||||
if parsed.hostname == "api.osv.dev":
|
||||
return OSV().get("ignored")
|
||||
return API().repo
|
||||
|
||||
transport = Transport()
|
||||
api = s.GitHub("fake-token", transport)
|
||||
assert len(list(api.advisories())) == 101
|
||||
assert s.candidate(advisory(), api, s.OSV(transport)).first_affected == "1.2"
|
||||
assert all(headers["Authorization"] == "Bearer fake-token" for url, headers in transport.calls if "api.github.com" in url)
|
||||
count = api.calls
|
||||
api.get_repo("fixture/project")
|
||||
assert api.calls == count
|
||||
|
||||
|
||||
def test_http_failures_are_not_empty_results():
|
||||
class Transport:
|
||||
def get(self, url, headers):
|
||||
raise ValueError("http_failure")
|
||||
|
||||
with pytest.raises(ValueError, match="http_failure"):
|
||||
list(s.GitHub("fake", Transport()).advisories())
|
||||
with pytest.raises(ValueError, match="http_failure"):
|
||||
s.OSV(Transport()).get("GHSA-test-test-test")
|
||||
|
||||
|
||||
def test_transport_rate_limit_retry_and_sanitized_failure():
|
||||
import io
|
||||
from urllib.error import HTTPError
|
||||
|
||||
waits, attempts = [], []
|
||||
|
||||
def opener(request, timeout):
|
||||
attempts.append(request)
|
||||
if len(attempts) == 1:
|
||||
raise HTTPError(request.full_url, 403, "private message", {"X-RateLimit-Remaining": "0", "X-RateLimit-Reset": "120"}, None)
|
||||
return io.BytesIO(b'{"ok": true}')
|
||||
|
||||
transport = s.JSONTransport(opener=opener, sleeper=waits.append, clock=lambda: 100)
|
||||
assert transport.get("https://example.invalid", {}) == {"ok": True}
|
||||
assert waits == [60] and transport.attempts == 2
|
||||
|
||||
def down(request, timeout):
|
||||
raise HTTPError(request.full_url, 503, "private name", {}, None)
|
||||
|
||||
transport = s.JSONTransport(opener=down, sleeper=waits.append)
|
||||
with pytest.raises(ValueError, match="^http_failure$"):
|
||||
transport.get("https://example.invalid", {})
|
||||
assert transport.attempts == 5
|
||||
Reference in New Issue
Block a user