spec(unseen-repo-evaluation): design and tasks approved; run through the baseline

Decisions: long-deadline arm (300 s), adoption needs no unseen regression
(Req 5.4), permissive repositories by name only, two AX lanes. Also adds the
trained-classifier and joint-optimisation arms to learned-decision-engine.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

.kiro/specs/learned-decision-engine/tasks.md

@@ -237,3 +237,20 @@ rate per push. Within-pair AUC stays a development signal. Tasks 11-14 below fol
   - Planned, roadmap G3. The exp-004 design with the family floor (Req 6.4), together with the pool's false-alarm
     rate per family.
   - _Requirements: 6.4, 7.3_
+
+- [ ] 15. Trained classifier arm (maintainer decision 2026-10-04: train a model, not hand-set profiles; judged on the
+  unseen-repo pool, roadmap G1)
+  - [ ] 15.1 Features: code embedding of the excerpt (stored vectors) + every allow-listed signal; no hand-picked
+    profile. Automated screening: a signal that separates pair sides by itself on training folds above the leak
+    threshold is dropped for that fold (the leak audit as a filter, fitted per fold).
+  - [ ] 15.2 Models: regularised logistic regression and gradient-boosted trees, fitted per repository-held-out outer
+    fold with inner-fold hyper-parameter search; per-family and pooled variants.
+  - [ ] 15.3 Scores on held-out folds: within-pair AUC, pooled AUC, top-k precision, Wilson bound at the pooled
+    threshold; compared with the current LLM programs on the same units.
+  - [ ] 15.4 Scored on the unseen-repo pool when G0 exists: catch rate at fixed false-alarm budgets.
+  - _Requirements: 3 (revised 2026-10-04 by maintainer decision), 6, 7_
+
+- [ ] 16. Jointly optimised LLM program arm (judged on the unseen-repo pool)
+  - Search input fields, retrieval parameters, k, instruction and demonstrations together (DSPy MIPRO-style) on
+    training folds; no hand-set input profile. Budget go-ahead before paid search.
+  - _Requirements: 3, 6, 7_

.kiro/specs/unseen-repo-evaluation/design.md

@@ -0,0 +1,431 @@
+# Design Document
+
+## Overview
+
+This spec builds the instrument for roadmap G0: a frozen pool of repositories that no development step has seen,
+and a push-level score on it. The score is what a user of the hook sees. On a vulnerability-introducing push, did
+the hook say so (catch rate)? On an ordinary push, did it say anything (false-alarm rate)? Replays run as AX Tasks
+on kube-ax. kind and the VM run the same tasks as fallbacks.
+
+Four facts from the code shape the design:
+
+- **Populations already have a freeze-and-guard pattern.** `benchmarks/independent/freeze.py` verifies cases by
+  reading git only, `evaluate.py` scores with fixed rules and Wilson intervals, and
+  `tests/test_independent_population.py` fails when a reserved repository is named anywhere in `benchmarks/`,
+  `src/` or `tests/` outside its directory. Harvest batch 3 used that guard as a black box: candidates written,
+  guard run with output discarded, subsets bisected on red. That is how v3 stays unread.
+- **The hook can already replay a change.** `ousast pre-push PATH --base B --head H --artifact F` resolves two
+  local commits (`snapshot.py`, `resolve_replay`: no fetch, no merge-base), runs quick rules and the engine inside
+  one deadline (default 30 s, engine inline) and writes a JSON artifact. The terminal text has one `Skipped:` line
+  per skipped check. The artifact holds three finding streams: admitted defects, advisory engine findings
+  (new or worsened, vetoed only by capability) and quick-rule matches with their CWE.
+- **The AX pattern works but lives in one script.** `engine_trace_ax.py` (manifest, egress checks, presigned
+  objects, retry once under a new name, cleanup) and the lane scheduler in `engine_trace.py` (`run_plan`: fixed
+  lanes, fallback queue, `progress.json`, `STOP`, resume by input digest) are engine-specific.
+- **Placement on kube-ax is random.** One worker per node, shared by all actors. Two heavy tasks fit per worker,
+  but only two concurrent tasks in all guarantee at most two on one worker.
+
+### Goals
+
+- A pool frozen by code against every used source (Req 1), with two kinds of change per repository (Req 2).
+- A scorer whose numbers mean what the hook shows, with silent zeros visible (Req 3).
+- One replay is one AX Task via a dispatcher shared with the engine (Req 4); a baseline and fresh slices (Req 5).
+
+### Non-Goals
+
+Detection changes, population v3, model calls in the cluster, languages the hook does not analyse.
+
+## Architecture
+
+```mermaid
+flowchart LR
+    M[pool manifest slice] --> D[batch dispatcher on VM]
+    D --> P[presign spec GET and record PUT]
+    P --> T[AX Task ousast-engine-replay]
+    T --> G[shallow fetch base and head from GitHub]
+    G --> H[ousast pre-push --base --head]
+    H --> R[record tar: artifact, text, instrument]
+    R --> S[(store via presigned PUT)]
+    D -->|sandbox died twice or OOM| V[VM docker lane]
+    D -->|cluster down| K[kind lane, same tasks]
+    V --> S
+    K --> S
+    S --> C[scorer: catch, false alarm, coverage]
+    S --> A[model arms on VM, budgeted]
+    A --> C
+    C --> O[record: counts and digests]
+```
+
+## Components and Interfaces
+
+### 1. Pool sourcing and eligibility (Req 1)
+
+**Decision 1.1: sources and ecosystems.** Candidates come from GitHub reviewed advisories (`gh api /advisories`,
+the enumeration harvest-3 used) for pip, npm, maven and composer, published 2019 or later, whose CWE maps to one of
+the seven families through the project's family CWE table. Each needs a linked fix commit with one parent in one
+repository. OSV records for the same advisory give the first affected version. That version is a cross-check for
+the introducing commit (section 2). Reason: these four ecosystems are the five languages the hook analyses
+(Python, JavaScript, TypeScript, PHP, Java). GHSA links commits and CWEs; OSV adds version brackets. Rejected:
+drawing from every ecosystem (Go, Ruby, C#). That would mostly measure missing frontends, which is already known.
+Coverage still reports uncovered files inside the drawn repositories. Rejected: NVD and published fix datasets.
+NVD links commits weakly, and the datasets are mined corpora that development or model training has likely seen.
+
+**Decision 1.2: eligibility by code against every used source.** `benchmarks/unseen/eligibility.py` builds the
+used set before the draw. It never prints names; it records counts per source and a digest of the set.
+
+| Source (Req 1.1) | Where it is read |
+| --- | --- |
+| Pair corpora | every catalog, recipe and dataset file under `benchmarks/pairs/`, plus the local pointer-pair manifests |
+| Populations v1, v2 | `population-v1.toml`, `population-v2.toml`, read directly |
+| Population v3 | never opened; the existing guard test only (below) |
+| Harvest records | the batch catalogs under `benchmarks/pairs/advisory-fixes*` (the records hold counts only) |
+| Memory examples | `example` rows of the local FileStore and the S3 store (`repo` field) |
+| Experiment units | `plane/experiments/*.units.jsonl` and registered units files |
+| Measurement records | everything under `benchmarks/measurements/` and `benchmarks/experiments/` |
+| Anything else | a text sweep for `github.com/<owner>/<name>` over `benchmarks/`, `src/`, `tests/`, `plane/`, `.kiro/` |
+
+Names are normalised (lower case, no `.git`, no trailing slash). Each candidate is resolved through the GitHub API
+to its canonical `full_name`, its former names (redirects) and its fork network (`parent`, `source`). A candidate
+is out if any of these is in the used set. A fork shares code with what it was forked from.
+
+v3 is checked like harvest-3 did. The draft manifest is written into `benchmarks/unseen/`. Then
+`pytest -q tests/test_independent_population.py -k referenced` runs with its output discarded. The guard already
+sweeps `benchmarks/` outside the population directory. On red, the candidate list is bisected by rewriting the
+draft until green, and the dropped count is recorded. No v3 name is ever printed or read by a person.
+
+**Decision 1.3: post-cutoff preference.** An advisory published after 2026-06-01 (harvest-3's cutoff for the
+decision engine's model) is flagged `post_cutoff`. Within each ecosystem stratum post-cutoff advisories are drawn
+first. Results are reported for both groups. Reason: a post-cutoff advisory cannot be in the model's training
+data as a known vulnerability. Rejected: post-cutoff only. Harvest-3 kept 12 post-cutoff pairs out of 251, and
+three slices of 100 repositories cannot be filled from that window.
+
+**Decision 1.4: licences.** The licence is read at the pinned head commit (LICENSE file, then the API's SPDX id).
+Only permissive licences are published by name: MIT, Apache-2.0, BSD-2-Clause, BSD-3-Clause, ISC, 0BSD, Unlicense,
+Zlib, PostgreSQL, BSL-1.0. Everything else (copyleft, source-available, none) goes to the gitignored
+`benchmarks/unseen/private/pool-p1.toml`. The committed manifest then holds an opaque id and the sha256 of
+(URL, commits) for that entry, so the freeze digest still covers it. Reason: Req 1.4 read literally. Rejected:
+"OSI-approved" as the bar, as population v2 used. It is wider than the requirement says (open question 1,
+resolved 2026-10-04: copyleft stays private).
+
+Also required: anonymous HTTPS, at most 500 MB, at least 40 eligible ordinary commits, one per fork network.
+
+### 2. Change extraction (Req 2.1, 2.2)
+
+Extraction runs on the VM over a full `--filter=blob:none` clone in a scratch cache. It uses git only, like
+`freeze.py`. No scanner, engine or model touches a pool repository before the freeze.
+
+**Decision 2.1: the vulnerability-introducing range.** Let F be the fix and P its parent. The vulnerable lines are
+the old-side lines of F's hunks in the vulnerable function. For a pure-addition fix (an absence bug such as a
+missing guard), they are the vulnerable function's own lines.
+
+1. **`introducing` (preferred).** `git blame -w -M -C --porcelain P -L <lines> -- <file>`. The candidate V is the
+   commit that last wrote the sink line (for absence bugs, the function's declaration). V is accepted if all hold:
+   at V's first parent the sink line's normalised text is absent from the file's vulnerable function; V is not a
+   merge or root commit; V changes at most 50 files and 2,000 lines; when OSV names a first affected version, V is
+   reachable from that tag and not from the previous release tag. If the text is present at V's parent, blame
+   walks back from there, at most five steps. The replay is `base = V^`, `head = V`.
+2. **`last_touch` (fallback).** When no V passes, `git log -L :<function>:<file> P` gives the newest commit L
+   before F that changed the vulnerable function. The replay is `base = L^`, `head = P`. The vulnerable code is in
+   head, and the range contains a change to the vulnerable function.
+
+Each change records its method, family, vulnerable function (resolved in head), head-side changed lines,
+`post_cutoff` and the advisory id.
+
+Bias of each, stated in the record. `introducing` inherits SZZ noise: blame finds the last writer, not the author
+of the flaw. A refactor, a move the heuristics miss or a reformat can stand in for the real introduction. The
+absence check, the walk-back and the version bracket cut this down, but do not remove it. The bias runs both
+ways: a wrong V may introduce nothing (a miss is not the hook's fault) or may move a flaw (the delta-based hook is
+right to call it existing). `last_touch` is pessimistic for a delta hook. The flaw is often already in base, so
+the hook's novelty rule correctly stays quiet and the change counts as a miss. Catch is therefore reported per
+method. The headline is `introducing`; `last_touch` is a lower bound. Rejected: fix-parent against a fixed old
+base (for example 50 commits back). The range often misses the vulnerable function entirely and grows without
+bound on busy repositories.
+
+Label check: every third `introducing` change is read by an agent before the freeze, with no model call, as
+harvest-3 reviewed diffs. It is marked `confirmed`, `refactor` or `unclear`. The confirmed share is the label
+noise estimate. The scorer also reports catch on the confirmed subset.
+
+**Decision 2.2: ordinary commits.** The candidates are non-merge commits on the first-parent line of the default
+branch that change at least one source file of an analysed language. Excluded:
+
+- explicit security signals in the message: `secur|cve|ghsa|vuln|inject|xss|csrf|ssrf|exploit|sanitiz|escap`;
+- reverts (`^Revert`, `This reverts commit`);
+- dependency bumps (bot authors such as dependabot and renovate, `bump|upgrade .* from`, or only manifest and lock
+  files changed);
+- commits whose source changes are all generated or vendored (`vendor/`, `node_modules/`, `dist/`, `*.min.js`,
+  lock files, `linguist-generated` in `.gitattributes`, "generated" headers);
+- any fix commit of any advisory of the repository, and any commit touching a known vulnerable function of it;
+- commits over 1,000 changed source lines (counted, reported).
+
+Twelve are drawn per repository with a seeded random draw. Strata are three time thirds of the repository's
+history crossed with four size buckets of changed source lines (1-10, 11-50, 51-200, 201-1,000). Allocation is
+proportional to the repository's own commits per cell, with at least one per time third. Each change records base
+(its first parent), head, changed files and line ranges. Reason: proportional allocation keeps the false-alarm
+rate a per-push rate as a user meets it. Rejected: `freeze.py`'s broad `SECURITY_WORDS`. It also drops commits
+saying "query", "token", "session" or "valid", which removes the ordinary commits where false alarms are most
+likely, and the rate would look better than it is. Rejected: docs-only commits. They are trivially quiet and
+would dilute the rate.
+
+Label noise: an ordinary commit may itself introduce an unknown vulnerability, so a flag counted as a false alarm
+may be true. The false-alarm rate stays as Req 3.2 defines it. Flagged ordinary changes are adjudicated beside it
+(all if 30 or fewer, else every third) against protocol-v2's standard, and the adjudicated share is reported.
+
+### 3. Sizing and slices (Req 2.3, 5.2)
+
+Wilson 95% interval, z = 1.96: centre (p + z²/2n)/(1 + z²/n), half width z·sqrt(p(1-p)/n + z²/4n²)/(1 + z²/n).
+Taking the larger distance from p as the width, the smallest n that meets the bar, computed exactly:
+
+| Rate | Assumed p | Bar | n needed |
+| --- | --- | --- | --- |
+| false alarm | 0.05 | +/- 3 points | 315 |
+| false alarm | 0.10 | +/- 3 points | 483 |
+| false alarm | 0.20 | +/- 3 points | 756 |
+| catch | 0.10 | +/- 10 points | 62 |
+| catch | 0.25 | +/- 10 points | 88 |
+| catch | 0.40 | +/- 10 points | 97 |
+
+Catch: 100 vulnerability-introducing changes per slice, one per repository. At p = 0.4, Wilson gives
+[0.309, 0.498], within 10 points. One per repository keeps them independent.
+
+False alarm: ordinary changes of one repository are correlated (noisy code flags often). Wilson assumes
+independence, so the count is inflated by a design effect 1 + (m - 1)·ICC. With m = 12 per repository and an
+assumed ICC of 0.05, that is 1.55. 100 repositories × 12 = 1,200 changes, effective n 774 ≥ 756. The plain Wilson
+interval at p = 0.2, n = 1,200 is about +/- 2.3 points. A repository-cluster bootstrap interval is reported beside
+it, and the ICC measured on slice 1 is recorded. If it exceeds 0.05, the next pool sizes up. A frozen slice is
+never changed.
+
+**Decision 3.1: slices.** Pool p1 has three slices of 100 repositories each (100 + 1,200 changes per slice; 300
+repositories and 3,900 changes in all). Repositories are assigned by a seeded shuffle, stratified by ecosystem and
+`post_cutoff`. Slice 1 serves the baseline and the G1 arms table. Slices 2 and 3 serve the first two adoption
+decisions. When they are spent, pool p2 is drawn by the same code, and it excludes p1. Reason: each slice alone
+meets the Req 2.3 widths, so any decision can stand on one slice. Rejected: one large pool scored whole. Every
+decision would then see every repository, which Req 5.2 forbids. Rejected: ten small slices. None would meet the
+widths. Per family, 100 changes split over seven families give intervals of about +/- 20 to 25 points. They are
+reported, not sized (open question 3).
+
+Use ledger: `benchmarks/unseen/usage-p1.jsonl` is append-only, one row per use (slice, decision or experiment id,
+date, result digest). A slice with a row whose purpose is `informed` cannot appear again as `qualifies` for that
+change. A test checks this.
+
+### 4. Scoring rules (Req 3.1, 3.2, 3.3)
+
+Each replay uses the hook's defaults: `--deadline 30`, `--engine inline`, `--max-regions 500`, `--mode advisory`.
+That is what a user's push gets. The text output is kept with every `Skipped:` line, and so is the artifact.
+
+**Decision 4.0: a registered long-deadline arm** (maintainer decision 2026-10-04, open question 5). Beside the
+default replay, slice 1 is replayed once more with `--deadline 300` and every other flag unchanged. The arm is
+registered (manifest committed, `unseen: {pool: p1, slice: 1}`) before its first replay. G1 then reports both
+what the user sees (30 s) and what the engine could catch given time (300 s), so a miss caused by the deadline is
+told apart from a miss by the engine. It is a measurement arm, not a proposed default: adopting a longer hook
+deadline would be a separate decision under 7.2.
+
+**Decision 4.1: catch, per family.** A finding counts if its family is the change's family. Engine streams carry a
+family; a quick-rule match maps its CWE through the family CWE table. Its head location must also be in the
+vulnerable function or within 10 lines of the change's head-side hunks (protocol v1's window).
+
+| Family | Location rule |
+| --- | --- |
+| injection, path, deserialization, output_encoding, untrusted_destination | sink line in the vulnerable function, or the changed lines +/- 10 |
+| access_control | the vulnerable function itself (the handler that lacks the check), or the changed lines +/- 10 |
+| config_secrets | the changed file and lines +/- 10 (file scope; function `<global>`) |
+
+All three streams count: admitted defects, advisory engine findings and quick-rule matches. Each catch also names
+the stream that made it. Reason: the hook shows all three to the user. Rejected: admitted defects only. That
+scores the blocking path, which G3 decides separately, and would hide what users read.
+
+**Decision 4.2: false alarm.** An ordinary change is a false alarm when any of the three streams holds at least one
+finding, of any family, anywhere. The alerts-only rate (admitted defects) is reported beside it.
+
+**Decision 4.3: coverage, so zeros cannot hide.** Every change gets exactly one state:
+
+- `instrument_failure`: no record, a failed fetch, the head commit not verified, or zero bytes of the changed files
+  read. It is re-run once, then reported. It never counts as quiet or missed.
+- `unanalysable`: no changed file is in a language the hook covers (the hook's `language_not_covered` notice).
+- `quick_only`: the quick tier read the changed files, but no engine question completed on them.
+- `engine_covered`: at least one engine question completed on a changed file.
+
+The task entry proves its input as AGENTS.md requires. It records files and bytes in the checkout, bytes of each
+changed file, the JVM's wall time and the hook's exit code. A replay faster than a JVM can start is flagged
+`suspect_fast` and goes to the failure path. Catch and false alarm are reported over all changes and over each
+coverage state.
+
+### 5. The AX workload (Req 4)
+
+**Decision 5.1: the image.** `plane/Dockerfile.engine-task` gains a second stage, `FROM engine AS replay`, built
+with `--target replay` and pinned by digest. It adds `git`, the package with its runtime dependencies and
+`benchmarks/unseen/replay_entry.py`. Joern, php-cli, Java 21 and the frontend-retention patch come from the engine
+stage, so both lanes run the same Joern. The import smoke runs `ousast pre-push --help`. The root image stays
+`ghcr.io/norandom/ax-task-runner:v0.3.1`. Rejected: a separate Dockerfile. It would duplicate the Joern install,
+and two Joern copies are two instruments that drift. Rejected: one stage for both. The engine image's isolated
+venv without yaml and boto3 is proven, and replay dependencies would change its digest on every bump.
+
+**Decision 5.2: the task entry.** `replay_entry.py` is standard library only, like `engine_task_entry.py`, and
+reuses its `transfer` (egress readiness wait, three attempts, no URL text in errors).
+
+1. GET `SPEC_URL` (presigned, `.json`): repository URL, base, head, change id, hook flags.
+2. `git init`, then `git fetch --depth 1 origin <head>` and `git fetch --depth 1 origin <base>` over HTTPS
+   (GitHub serves commits by SHA). Verify both with `git cat-file -e`. The fetch is bounded at 180 s; the 500 MB
+   repository cap bounds its size.
+3. Run `ousast pre-push <checkout> --base <base> --head <head> --artifact /tmp/out/push.json` with defaults, the
+   backend heap at 1700 MB and the cluster JVM flags of `engine_trace_worker.py`. Capture stdout.
+4. Pack `push.json`, `push.txt`, `engine.json` (the head and base scan records and witness rows from the artifact,
+   keyed by repository and revision, for engine-evidence arms) and `instrument.json` (section 4.3, image digest,
+   step timings). PUT to `RESULT_URL` as `.dat`. The PUT is completion.
+
+On any error it PUTs a failure record naming the step and HTTP status, as the engine entry does. Task names are
+`ousast-engine-replay-<slug>-<digest>`, the prefix the operator's egress watcher matches. No key or store
+credential enters the manifest. Only the two presigned URLs do, and the spec object keeps repository names out of
+AX status and cluster telemetry.
+
+**Decision 5.3: the generic dispatcher.** `benchmarks/ax/batch.py` takes over from `engine_trace_ax.AX` and
+`engine_trace.run_plan`:
+
+- `Item`: id, input objects (name to bytes), the result object name, command, extra env, input digest.
+- `Workload`: image digest, allowed URL env names, deadline, how to validate a received result.
+- Lanes: `ax` (manifest, apply, poll the result object and AX phase, delete, clean objects), `kind` (the same
+  manifest against the kind AX), `docker` (the VM image, one container at a time).
+- Scheduler: two AX lanes plus one VM lane by default. A sandbox death is retried once under a new name; a second
+  death or a JVM OOM goes to the VM fallback queue, which beats overflow on the VM lane. `progress.json` is written
+  only by the scheduler. A `STOP` file halts new launches. Resume skips items whose stored record has the same
+  input digest. Cleanup failure stops dispatching.
+
+The engine pass becomes a workload over it, with the same tests. Reason: the egress, secrecy and cleanup logic took
+several fixes and must exist once. Rejected: a replay-specific copy of `engine_trace_ax.py`. Rejected: putting it in
+`src/` now. G4's `ousast plane scan` may need it there, and it moves when G4 is planned.
+
+Concurrency is two AX tasks. Placement is random, and only two guarantees two per worker at most. Four would need
+the operator to add placement (open question 2, resolved 2026-10-04: stays at two, no placement change
+requested). Estimated wall time per slice: about 75 s per replay (fetch, 30 s hook, setup), 1,300 replays over
+three lanes, about 9 hours. The long-deadline arm (4.0) is bounded at about 345 s per replay; at that bound 1,300
+replays over three lanes take about 42 hours, less where the hook finishes early. It runs with `STOP` and resume
+across sessions.
+
+**Decision 5.4: model arms on the VM.** Arms that need DeepSeek (the decision engine, the combiner) read the stored
+records on the VM. They build candidates from the head side's changed functions, score them, and take the change
+score as the maximum. Each runs under its experiment manifest's `budget_usd` through the metered client, dry run
+first, with responses cached. Nothing in the cluster calls a model.
+
+**kind (Req 4.5)** runs the identical manifest and image through the `kind` lane. Every new task is proven there
+before kube-ax, as `plane-on-kubernetes` requires.
+
+### 6. Records and the guard (Req 1.2, 1.3, 5.1)
+
+`benchmarks/unseen/` holds `pool-p1.toml` (repositories, licences, slices, changes with base, head, method,
+family, function, lines), `freeze-p1.json` (used-set digests and counts per source, guard result, label-check
+counts, freeze digest = sha256 of the canonical manifest), `protocol-p1.md` (the scoring rules of section 4,
+committed before the first replay) and `usage-p1.jsonl`. The `private/` directory is gitignored.
+
+The guard grows by one parametrised reservation. The existing sweep becomes a list of (reservation directory,
+names): `benchmarks/independent/` for populations and `benchmarks/unseen/` for pools, including the private
+manifest when present. It also sweeps `plane/`. A second test asserts that pool and population names are disjoint,
+and prints only a count on failure. A third asserts that a frozen pool's digest matches `freeze-p1.json`, so the
+pool is never edited. Raw per-change records stay outside the repository (`~/ousast-results/unseen/` and the
+store). Committed result records under `benchmarks/measurements/` carry counts, rates, intervals and digests,
+never names.
+
+### 7. Metrics and reporting (Req 3.3, 3.4, 5.3)
+
+`benchmarks/unseen/score.py` reports per family and overall: catch and false-alarm rate with Wilson 95%
+intervals, the cluster bootstrap interval for false alarms, coverage counts per state, catch per method,
+post-cutoff and per stream, and the alerts-only rate.
+
+**Decision 7.1: arms at fixed budgets.** An arm with a score is thresholded per budget (1%, 5%, 10%). The
+threshold is the lowest score whose false-alarm rate on the slice's ordinary changes stays within the budget.
+Catch at that threshold is reported with its Wilson interval and the realised false-alarm rate. That in-slice
+threshold is labelled exploratory. For a qualification, the threshold is fixed beforehand on an earlier slice or
+on held-out folds, and it is applied unchanged. A binary arm (today's hook) has one operating point, reported with
+the budgets it meets. Rejected: AUC over changes (it averages operating points no user runs).
+
+**Decision 7.2: next to held-out folds.** Experiment manifests registered after this spec lands carry
+`unseen: {pool, slice}`. `analyse` writes an `unseen` block into `result.json` next to the fold estimates: catch at
+each budget for each arm, and the paired difference B - A with an exact McNemar test on the 100 changes. The
+fold rule decides as today. The slice is recorded in the usage ledger.
+
+**Decision 7.3: adoption gate on the unseen slice** (amended 2026-10-04, maintainer decision; Req 5.4; open
+question 4 resolved). Reporting is not enough. An arm B is adopted over the production arm A only if, on the
+untouched slice that qualifies it, both hold:
+
+- catch at the 5% budget, with B's threshold fixed beforehand (7.1), is not significantly lower than A's: exact
+  two-sided McNemar on the paired vulnerability-introducing changes, alpha 0.05;
+- for an arm with one operating point (a binary arm such as a hook setting), its false-alarm rate is not
+  significantly higher than A's: exact two-sided McNemar on the paired ordinary changes, alpha 0.05, with the
+  repository-cluster bootstrap interval of the difference reported beside it, since those changes are clustered.
+
+`analyse` writes `unseen.gate: pass | regression | not_run` into `result.json`. `not_run` (no slice, slice
+already `informed` for this change, or more than 5% `instrument_failure`) blocks adoption like `regression`. The
+fold rule and this gate must both pass.
+
+### 8. Increments
+
+1. **Used set and guard.** `eligibility.py`, the extended guard and the usage-ledger test. Exit: guard green on an
+   empty pool; a planted used repository, a renamed one and a fork are each rejected in tests.
+2. **Sourcing and extraction.** Candidates, both change kinds and the label check, on the VM. Exit: a draft for 300
+   repositories with method counts, the confirmed share and exclusion counts.
+3. **Freeze p1.** Guard bisection for v3, then manifest, freeze record and protocol committed. Exit: freeze digest
+   committed and guard green.
+4. **Dispatcher.** `benchmarks/ax/batch.py`, with the engine pass moved onto it. Exit: existing tests green and a
+   two-pin engine smoke on kube-ax matching its stored records.
+5. **Replay task.** Image stage and entry, proven on kind, then kube-ax on five changes from this repository's own
+   history (never pool changes). Exit: instrument fields show bytes read and realistic JVM time, and quick-tier
+   output matches the VM lane.
+6. **Scorer.** `score.py` on synthetic records, including every coverage state and an instrument failure. Exit:
+   unit tests green.
+7. **Baseline.** Today's hook, from a frozen export of HEAD, on slice 1 (1,300 replays). Exit: the record committed
+   (counts and digests only) and a `baseline` row in the usage ledger. That is G0's exit.
+8. **Arms on slice 1 (G1)** (added 2026-10-04 with decisions 4.0 and 7.3). The registered long-deadline arm,
+   the engine-evidence arm from the stored records, the model arms on the VM (5.4) under their budgets, and the
+   `unseen` block with the adoption gate in `analyse`. Exit: one committed table of arms on slice 1 at the 1/5/10%
+   budgets (counts, rates, intervals, digests), and `exploratory` rows in the usage ledger.
+
+## Error Handling
+
+A failed fetch, a missing commit or a hook crash gives a failure record, never an empty success. A dead sandbox is
+retried once, then the VM runs it. A store or AX cleanup error stops the dispatcher. A slice with more than 5%
+`instrument_failure` after re-runs is not scored; the cause is fixed first.
+
+## Testing Strategy
+
+Unit tests: used-set normalisation (case, `.git`, renames, forks), blame acceptance on a synthetic repository,
+ordinary-commit exclusions, Wilson values against section 3, budget thresholds, coverage states, the task manifest
+and dispatcher resume and fallback with a fake lane. Guard tests run in CI; cluster runs stay manual.
+
+## Requirement Traceability
+
+| Requirement | Design |
+| --- | --- |
+| 1.1 eligibility against every source, v3 via the guard | 1.2 |
+| 1.2 frozen manifest, never edited | 6, increment 3 |
+| 1.3 pool directory, guard extended | 6 |
+| 1.4 licences, private manifest | 1.4 |
+| 2.1 introducing changes, family, function, post-cutoff | 1.1, 1.3, 2.1 |
+| 2.2 ordinary changes across time and size | 2.2 |
+| 2.3 sizes for interval widths | 3 |
+| 3.1 default replay, skipped checks kept | 4, 5.2 |
+| 3.2 catch and false-alarm rules | 4.1, 4.2 |
+| 3.3 per family, Wilson, coverage | 4.3, 7 |
+| 3.4 arms at 1/5/10% budgets | 4.0, 7.1, increment 8 |
+| 4.1 one change one Task, prefix, image by digest | 5.1, 5.2 |
+| 4.2 HTTPS fetch, presigned record, no keys | 5.2 |
+| 4.3 generic dispatcher, concurrency, retry, fallback, resume | 5.3 |
+| 4.4 model arms on the VM within budgets | 5.4 |
+| 4.5 kind runs the same tasks | 5.3, 5 (kind paragraph) |
+| 5.1 baseline on slice 1, counts and digests | 6, increment 7 |
+| 5.2 fresh slice per decision | 3.1, usage ledger |
+| 5.3 experiments report unseen next to folds | 7.2 |
+| 5.4 adoption needs no significant unseen regression | 7.3 |
+
+## Open Questions
+
+Maintainer decisions of 2026-10-04 resolve questions 1, 2, 4 and 5; question 3 stays open for G3.
+
+1. **Resolved 2026-10-04.** Licences: only permissively licensed repositories are published by name. Copyleft and
+   all other licences go to the gitignored private manifest, covered in the committed manifest by an opaque id and
+   a digest (Req 1.4 as written; decision 1.4). Population v2's OSI-approved bar is not used.
+2. **Resolved 2026-10-04.** Concurrency stays at two AX lanes plus the VM lane. No operator placement change is
+   requested (5.3).
+3. Per-family BLOCK (G3) needs about 100 changes per family. Pool slices together, or draw a family-enriched pool
+   for G3?
+4. **Resolved 2026-10-04.** An adoption requires no significant regression on the unseen slice, not only a report
+   (Req 5.4, amended; decision 7.3).
+5. **Resolved 2026-10-04.** G1 adds a registered long-deadline arm (`--deadline 300`) next to the default 30 s, so
+   the table shows both what the user sees and what the engine could catch (decision 4.0, increment 8).

.kiro/specs/unseen-repo-evaluation/requirements.md

@@ -101,3 +101,7 @@ worn out by tuning.
    slice, recorded with its freeze digest.
 3. Every learned-decision-engine experiment from now on reports its unseen-repo score from this pool next to its
    held-out-fold score.
+4. An arm is adopted only if, on the untouched slice that qualifies it, it shows no significant regression against
+   the current production arm: catch at the 5% false-alarm budget not significantly lower, and for an arm with one
+   operating point the false-alarm rate not significantly higher (exact paired tests, two-sided, alpha 0.05).
+   Reporting the unseen score alone is not enough to adopt. *(amended 2026-10-04, maintainer decision)*

.kiro/specs/unseen-repo-evaluation/spec.json

@@ -1,22 +1,22 @@
 {
   "feature_name": "unseen-repo-evaluation",
   "created_at": "2026-10-04T18:00:00Z",
-  "updated_at": "2026-10-04T18:30:00Z",
+  "updated_at": "2026-10-04T20:00:00Z",
   "language": "en",
-  "phase": "requirements-approved",
+  "phase": "implementation",
   "approvals": {
     "requirements": {
       "generated": true,
       "approved": true
     },
     "design": {
-      "generated": false,
-      "approved": false
+      "generated": true,
+      "approved": true
     },
     "tasks": {
-      "generated": false,
-      "approved": false
+      "generated": true,
+      "approved": true
     }
   },
-  "ready_for_implementation": false
+  "ready_for_implementation": true
 }

.kiro/specs/unseen-repo-evaluation/tasks.md

@@ -0,0 +1,321 @@
+# Implementation Plan
+
+One task group per increment of `design.md` section 8, in that order. A group starts only after the previous
+group's exit is recorded. Maintainer decisions of 2026-10-04 are folded in: a registered long-deadline arm
+(`--deadline 300`) beside the hook's default 30 s (design 4.0, group 8); adoption needs no significant regression
+on the unseen slice (Req 5.4 as amended, design 7.3, tasks 6.4 and 8.2); only permissively licensed repositories
+are published by name, everything else goes to the gitignored private manifest under a digest (Req 1.4, design
+1.4); concurrency stays at two AX lanes plus the VM lane (design 5.3).
+
+Execution rules:
+
+- **Who does what.** Codex implements every coding task offline: no network, no model, no store, no cluster. The
+  coordinator runs every networked step (GitHub and OSV API calls, git fetches and clones, the S3 store, image
+  pushes, kind and kube-ax). Those tasks are marked **(coordinator, network)**. Steps that need the coordinator but
+  no network (agent label reads, adjudication, the v3 guard bisection, commits) are marked **(coordinator)**.
+  Steps that spend model money are marked **paid: needs a budget go-ahead**. Each starts only after the maintainer
+  confirms a ceiling against the provider balance read just before it.
+- **Done means.** A coding task is done when its tests pass in the host suite, `ruff check`, `ruff format --check`
+  and `mypy` are clean, and the reserved-repository guard is green. No test makes a network, model or cluster call.
+- **Names stay in the pool directory.** Pool repository names and URLs appear only under `benchmarks/unseen/`
+  (copyleft and other non-permissive ones only under the gitignored `benchmarks/unseen/private/`). Raw per-change
+  records live in `~/ousast-results/unseen/` and the store, never in the repository.
+- **Population v3 is never opened.** It is checked only through the guard test, run as a black box with its
+  output discarded.
+- **Every group ends with a measured exit** and a record at
+  `benchmarks/measurements/<date>-unseen-NN-<slug>/record.json`: commit, image digests where used, the instrument
+  checks (bytes read, JVM time, exit codes), counts and digests. No repository names. The guard sweeps these
+  records.
+- **Instrument first** (AGENTS.md). A zero, an empty draft or a fast run is treated as an instrument failure until
+  bytes read and wall time prove otherwise.
+
+- [ ] 1. Used set and guard (design 1.2, 6; increment 1)
+- [ ] 1.1 Used-set builder
+  - `benchmarks/unseen/eligibility.py`: normalises names (lower case, no `.git`, no trailing slash) and builds the
+    used set from every source of design table 1.2: pair catalogs, recipes and datasets under `benchmarks/pairs/`
+    plus the local pointer-pair manifests as `benchmarks/pairs/catalog_gen.py` reads them; `population-v1.toml`
+    and `population-v2.toml`; the advisory-fix batch catalogs; memory `example` rows (`repo` field) through an
+    injected `MemoryStore` (`FileStore` locally, the S3 store when the coordinator passes one); experiment units
+    under `plane/experiments/`; `benchmarks/measurements/` and `benchmarks/experiments/`; a text sweep for
+    `github.com/<owner>/<name>` over `benchmarks/`, `src/`, `tests/`, `plane/` and `.kiro/`. It never reads
+    population v3 and never prints a name. Output: counts per source and the sha256 of the sorted set.
+  - `tests/test_unseen_eligibility.py`: a synthetic tree with one repository per source, each found; case, `.git`
+    and trailing-slash variants collapse; stdout and the output JSON contain no name; a v3-shaped file planted in
+    the synthetic tree is not opened (open is patched to fail on it).
+  - _Requirements: 1.1_
+- [ ] 1.2 Canonical resolution, renames and forks
+  - `eligibility.py`: `resolve(candidate, api)` with an injected GitHub client returns the canonical `full_name`,
+    former names (redirects) and the fork network (`parent`, `source`). A candidate is ineligible when any of them
+    is in the used set; one candidate per fork network is kept.
+  - Tests with a fake client: a planted used repository, a renamed one and a fork of a used one are each
+    rejected; two forks of one network keep one; the rejection reason is a source label, not a name.
+  - _Requirements: 1.1_
+- [ ] 1.3 Guard extended to pools
+  - `tests/test_independent_population.py`: the sweep becomes a list of (reservation directory, names):
+    `benchmarks/independent/` for populations, `benchmarks/unseen/` for pools (public manifests, plus
+    `benchmarks/unseen/private/*.toml` when present). The sweep also covers `plane/`. New tests: pool and
+    population names are disjoint (failure prints a count only); a frozen pool's manifest digest equals its
+    `freeze-pN.json` (skipped while no pool is frozen). The existing v3 handling is unchanged.
+  - Each reservation is exempt only inside its own directory: population names (v3 included) are still swept in
+    `benchmarks/unseen/`, the gitignored `private/` files included. Task 3.1 depends on this.
+  - `.gitignore`: `benchmarks/unseen/private/`.
+  - Evidence: the guard is green with an empty `benchmarks/unseen/`, and red when a test fixture names a pool
+    repository in `benchmarks/measurements/` (shown once in the record).
+  - _Requirements: 1.2, 1.3, 1.4_
+- [ ] 1.4 Usage ledger
+  - `benchmarks/unseen/ledger.py`: append-only `usage-pN.jsonl`, one row per use (slice, decision or experiment
+    id, purpose `baseline | exploratory | informed | qualifies`, date, result digest, freeze digest). It refuses a
+    `qualifies` row for a change whose slice already has an `informed` row for it, and refuses to rewrite rows.
+  - `tests/test_unseen_ledger.py`: append, the refusal, a rewritten row detected by a digest chain.
+  - _Requirements: 5.2_
+- [ ] 1.5 Used-set measurement and record 01 **(coordinator, network)**
+  - Run `eligibility.py` against the real sources, the S3 memory store included (read only).
+  - Instrument: each source reports files read and bytes; a source with zero rows fails the run unless it is
+    known empty.
+  - `benchmarks/measurements/<date>-unseen-01-used-set/record.json`: counts per source, used-set digest, guard
+    result (green on the empty pool), test counts of 1.1-1.4.
+  - _Requirements: 1.1, 1.3_
+
+- [ ] 2. Sourcing and extraction (design 1.1, 1.3, 1.4, 2, 3; increment 2)
+- [ ] 2.1 Advisory candidates and repository filters
+  - `benchmarks/unseen/source.py`: from GHSA advisory JSON (pip, npm, maven, composer; published 2019 or later)
+    keep those whose CWE maps to a family through `src/openultrasast/model/taxonomy.py` and that link one fix
+    commit with one parent in one repository; attach the OSV first affected version; flag `post_cutoff` when
+    published after 2026-06-01. Repository filters: anonymous HTTPS, at most 500 MB, licence at the pinned head
+    (LICENSE file, then the API's SPDX id) classed `permissive` (the design 1.4 list) or `private`.
+  - `tests/test_unseen_source.py` on recorded fixture JSON (no network): CWE mapping, multi-parent and
+    multi-repository fixes dropped, cutoff flag, licence classing (GPL, AGPL, LGPL, none and source-available go
+    private).
+  - _Requirements: 1.4, 2.1_
+- [ ] 2.2 Vulnerability-introducing range
+  - `benchmarks/unseen/extract.py`: the `introducing` method of design 2.1 (blame of the sink line, or the
+    declaration for an absence bug; absence at V's first parent; no merge or root; at most 50 files and 2,000
+    lines; OSV version bracket when present; walk-back of at most five steps) and the `last_touch` fallback. Each
+    change records method, family, vulnerable function resolved in head, head-side lines, `post_cutoff`, advisory
+    id. Git only; no scanner or engine.
+  - `tests/test_unseen_extract.py` on a synthetic repository built in a temp dir: accepted V; a reformat walked
+    back; a merge rejected; size cap; bracket rejection; fallback chosen when no V passes.
+  - _Requirements: 2.1_
+- [ ] 2.3 Ordinary-commit draw
+  - `extract.py`: first-parent, non-merge commits touching an analysed language; the exclusions of design 2.2
+    (security words of the narrow pattern, reverts, bot and bump commits, manifest- or lock-only, generated or
+    vendored, advisory fix commits, commits touching a known vulnerable function, more than 1,000 source lines,
+    counted). Seeded draw of 12 per repository over three time thirds by four size buckets, proportional, at least
+    one per third; a repository with fewer than 40 eligible commits is dropped.
+  - Tests: each exclusion; the narrow pattern keeps "query", "token", "session"; allocation and the seed
+    reproduce the same draw; the 40-commit floor.
+  - _Requirements: 2.2, 2.3_
+- [ ] 2.4 Draft manifest and slices
+  - `benchmarks/unseen/draft.py`: writes `benchmarks/unseen/draft-p1.toml` (permissive entries by name) and
+    `benchmarks/unseen/private/draft-p1.toml` (the rest); the public file holds an opaque id and
+    sha256(URL, commits) per private entry. Seeded shuffle into three slices of 100 repositories, stratified by
+    ecosystem and `post_cutoff`; each repository has one introducing change and 12 ordinary changes.
+  - Tests: the digest covers private entries; no private URL in the public file; the same seed gives the same
+    slices; per-slice strata within one repository of proportional.
+  - _Requirements: 1.2, 1.4, 2.3, 5.2_
+- [ ] 2.5 Sourcing run **(coordinator, network)**
+  - `gh api /advisories` enumeration, OSV lookups, canonical resolution against the used set (1.2), blobless
+    clones into a scratch cache, extraction (2.2, 2.3), draft (2.4).
+  - Instrument: clones counted with bytes on disk; a repository whose clone reads zero source files is an
+    instrument failure, not a drop. Fewer than 300 eligible repositories stops the run with counts; the
+    eligibility bars are not relaxed to fill it.
+  - Observable: a draft of 300 repositories (3,900 changes) on disk.
+  - _Requirements: 1.1, 1.4, 2.1, 2.2, 2.3_
+- [ ] 2.6 Label check and record 02 **(coordinator)**
+  - Every third `introducing` change read by an agent, no model call, marked `confirmed | refactor | unclear`
+    in the draft.
+  - `benchmarks/measurements/<date>-unseen-02-draft/record.json`: candidates per ecosystem, rejections per used
+    source, licence split (public, private), method counts, post-cutoff counts, confirmed share, ordinary-commit
+    exclusion counts. No names.
+  - _Requirements: 2.1, 2.2_
+
+- [ ] 3. Freeze p1 (design 1.2, 6; increment 3)
+- [ ] 3.1 Guard bisection tool
+  - `benchmarks/unseen/bisect_guard.py`: rewrites the draft with a candidate subset, runs
+    `.venv/bin/pytest -q -p no:cacheprovider tests/test_independent_population.py -k referenced` with stdout and
+    stderr discarded, reads only the exit code, and bisects until green. Reports the dropped count only.
+  - `tests/test_unseen_bisect.py` with an injected fake guard (a hidden set): finds and drops exactly the hidden
+    members; never logs a candidate; an exit code other than 0 or 1 aborts (instrument failure).
+  - _Requirements: 1.1_
+- [ ] 3.2 Freeze writer
+  - `benchmarks/unseen/freeze.py`: canonical `pool-p1.toml` (and the private counterpart) from the bisected draft;
+    `freeze-p1.json` with used-set digest and per-source counts, guard result, dropped count, label-check counts
+    and the freeze digest (sha256 of the canonical manifest). Refuses to write over a frozen pool; a correction
+    is `p2`.
+  - Tests: digest stable under key order; overwrite refused; the guard's digest test (1.3) passes on a fixture
+    pool.
+  - _Requirements: 1.2_
+- [ ] 3.3 Protocol p1
+  - `benchmarks/unseen/protocol-p1.md`: the scoring rules of design section 4 (default flags, decision 4.0's
+    long-deadline arm, catch and location rules, false alarm, coverage states), the statistics of section 3 and
+    7, and the adoption gate of 7.3, stated before any replay.
+  - Observable: the file exists and its sha256 is listed in `freeze-p1.json`.
+  - _Requirements: 3.1, 3.2, 3.3, 3.4, 5.4_
+- [ ] 3.4 Freeze run and record 03 **(coordinator)**
+  - Run the bisection (3.1), then the freeze (3.2). Guard green, digest test active.
+  - `benchmarks/measurements/<date>-unseen-03-freeze/record.json`: freeze digest, protocol digest, dropped
+    count, final counts per slice and kind.
+  - _Requirements: 1.1, 1.2, 1.3_
+
+- [ ] 4. Dispatcher (design 5.3; increment 4)
+- [ ] 4.1 Items, workloads and the AX lane
+  - `benchmarks/ax/batch.py`: `Item` (id, input objects, result object name, command, extra env, input digest),
+    `Workload` (image digest, allowed URL env names, deadline, result validator) and the `ax` lane, moved from
+    `benchmarks/learn/engine_trace_ax.py`: manifest render, egress readiness, presigned GET and PUT as `.dat`,
+    apply, poll result object and AX phase, delete, object cleanup. The `kind` lane is the same lane with the
+    kind context.
+  - `tests/test_ax_batch.py` with a fake `ax` CLI and a fake store: manifest holds only the allowed URL env
+    names and no credential; names carry the `ousast-engine-` prefix; cleanup on success and failure.
+  - _Requirements: 4.1, 4.2, 4.5_
+- [ ] 4.2 Scheduler, fallback and resume
+  - `batch.py`: two AX lanes and one VM `docker` lane by default (one container at a time); sandbox death retried
+    once under a new name, a second death or a JVM OOM goes to the VM fallback queue ahead of overflow;
+    `progress.json` written by the scheduler only; a `STOP` file halts launches; resume skips items whose stored
+    record has the same input digest; a cleanup failure stops dispatching.
+  - Tests with fake lanes: retry then fallback; OOM straight to fallback; resume skips; `STOP`; cleanup failure
+    halts; never more than two AX tasks in flight.
+  - _Requirements: 4.3_
+- [ ] 4.3 Engine pass on the dispatcher
+  - `benchmarks/learn/engine_trace_ax.py` and `engine_trace.run_plan` become an engine `Workload` over `batch.py`;
+    no behaviour change.
+  - Observable: `tests/test_ax_executor.py` and `tests/test_engine_trace.py` pass with their assertions unchanged
+    (moved imports only).
+  - _Requirements: 4.3_
+- [ ] 4.4 Engine smoke on kube-ax and record 04 **(coordinator, network)**
+  - Two pins through the new dispatcher on kube-ax, compared with their stored records.
+  - `benchmarks/measurements/<date>-unseen-04-dispatcher/record.json`: image digest, per-pin wall time and JVM
+    time, record digests equal to the stored ones (or the diff counted), cleanup result.
+  - _Requirements: 4.3_
+
+- [ ] 5. Replay task (design 5.1, 5.2; increment 5)
+- [ ] 5.1 Image stage
+  - `plane/Dockerfile.engine-task`: second stage `FROM engine AS replay` adding `git`, the package with its
+    runtime dependencies and `benchmarks/unseen/replay_entry.py`; smoke `ousast pre-push --help`. The root image
+    stays `ghcr.io/norandom/ax-task-runner:v0.3.1`; the engine stage is unchanged.
+  - `tests/test_unseen_image.py`: parses the Dockerfile; the `replay` stage builds from `engine`; the root image
+    pin; the engine stage's text is byte-identical to before.
+  - _Requirements: 4.1_
+- [ ] 5.2 Task entry
+  - `benchmarks/unseen/replay_entry.py`, standard library only, reusing `engine_task_entry.transfer`: GET the spec;
+    `git init` and depth-1 fetches of head and base over HTTPS (180 s bound), verified with `git cat-file -e`; run
+    `ousast pre-push` with the spec's hook flags (defaults, or `--deadline 300` for the long-deadline arm), heap
+    1700 MB and the cluster JVM flags; pack `push.json`, `push.txt`, `engine.json`, `instrument.json` (files and
+    bytes in the checkout, bytes of each changed file, JVM wall time, hook exit code, step timings, image
+    digest); PUT as `.dat`. Any error PUTs a failure record naming the step and status, never URL text.
+  - `tests/test_unseen_replay_entry.py`: a local bare repository and a localhost presign stub; success record
+    fields; a missing commit, a failed fetch and a hook crash each give a failure record; zero bytes read and a
+    replay faster than JVM start are flagged `suspect_fast`.
+  - _Requirements: 3.1, 4.1, 4.2_
+- [ ] 5.3 Replay workload
+  - `benchmarks/unseen/replay.py`: builds `Item`s from a frozen slice and an arm (`default` or `long_deadline`);
+    the spec object carries the repository URL so names stay out of AX status and telemetry; task names
+    `ousast-engine-replay-<slug>-<digest>` with an opaque slug; the validator rejects a record without
+    `instrument.json`; records go to `~/ousast-results/unseen/<pool>/<slice>/<arm>/` and the store.
+  - Tests: no repository name in any task name or manifest; only the two presigned URLs in the env; input
+    digest changes with the arm's flags; `instrument_failure` items queued for one re-run.
+  - _Requirements: 3.1, 4.1, 4.2, 4.3_
+- [ ] 5.4 Proof on kind, then kube-ax, and record 05 **(coordinator, network)**
+  - Build and push the `replay` target, pinned by digest. Five changes from this repository's own history (never
+    pool changes) on kind, then the same five on kube-ax, then on the VM lane.
+  - Exit: instrument fields show bytes read and realistic JVM time on every lane; quick-tier findings equal
+    across lanes.
+  - `benchmarks/measurements/<date>-unseen-05-replay-task/record.json`: image digest, per-lane timings, bytes,
+    exit codes, the quick-tier comparison.
+  - _Requirements: 4.1, 4.2, 4.5_
+
+- [ ] 6. Scorer (design 3, 4, 7.1, 7.3; increment 6)
+- [ ] 6.1 Coverage states
+  - `benchmarks/unseen/score.py`: each record gets one state (`instrument_failure`, `unanalysable`, `quick_only`,
+    `engine_covered`) from `instrument.json` and the artifact; `suspect_fast` goes to failure; a slice above 5%
+    `instrument_failure` after re-runs is refused.
+  - `tests/test_unseen_score.py` on synthetic records: every state; the refusal; a failure never counts as quiet
+    or missed.
+  - _Requirements: 3.3_
+- [ ] 6.2 Catch and false-alarm rules
+  - `score.py`: family match (engine family, quick-rule CWE through `taxonomy.py`), the location table of design
+    4.1, all three streams with the stream named per catch; false alarm on any finding of any stream; the
+    alerts-only rate.
+  - Tests: right family wrong place misses; `access_control` on the handler; `config_secrets` file scope; a
+    quick-rule-only catch; one advisory finding on an ordinary change is a false alarm.
+  - _Requirements: 3.2_
+- [ ] 6.3 Rates, intervals and breakdowns
+  - `score.py`: Wilson 95% intervals; repository-cluster bootstrap for false alarms; ICC; breakdowns per family,
+    method, `post_cutoff`, stream, coverage state and the confirmed subset.
+  - Tests: the n values of design section 3 (315, 483, 756, 62, 88, 97) reproduced; bootstrap seeded; ICC on a
+    synthetic clustered set.
+  - _Requirements: 2.3, 3.3_
+- [ ] 6.4 Budgets, paired tests, the gate and record 06
+  - `score.py`: per-budget thresholds (1%, 5%, 10%), labelled `exploratory` in-slice and `fixed` when supplied;
+    a binary arm's single operating point with the budgets it meets; exact two-sided McNemar for catch and for
+    false alarm; `gate(A, B)` returning `pass | regression | not_run` per design 7.3.
+  - Tests: thresholds on a synthetic score set; McNemar exact values; a binary arm with a significant
+    false-alarm increase gives `regression`; missing slice or excess failures give `not_run`.
+  - `benchmarks/measurements/<date>-unseen-06-scorer/record.json`: test counts and the digest of the scorer's
+    output on the committed synthetic fixture.
+  - _Requirements: 3.4, 5.4_
+
+- [ ] 7. Baseline (design 4, 6; increment 7; G0 exit)
+- [ ] 7.1 Frozen source and dry run **(coordinator, network)**
+  - Export HEAD to a frozen tree; build and push the `replay` image from it, pinned by digest.
+  - `benchmarks/push/python_overhead.py` run with the stub engine to size per-replay Python cost; dispatcher dry
+    run on slice 1 lists 1,300 items without launching.
+  - _Requirements: 5.1, 4.1_
+- [ ] 7.2 Baseline replay of slice 1 **(coordinator, network)**
+  - Default arm, 1,300 replays over two AX lanes and the VM lane (about 9 hours); `instrument_failure` items
+    re-run once; the run stops for a cause above 5% failures.
+  - Observable: 1,300 records in `~/ousast-results/unseen/p1/1/default/` and the store, each with
+    `instrument.json`.
+  - _Requirements: 3.1, 4.3, 5.1_
+- [ ] 7.3 Adjudication **(coordinator)**
+  - Flagged ordinary changes adjudicated against protocol-v2's standard (all if 30 or fewer, else every third);
+    the adjudicated share kept beside the false-alarm rate. ICC measured; above 0.05 noted as "next pool sizes
+    up".
+  - _Requirements: 3.2, 2.3_
+- [ ] 7.4 Record 07 and the ledger **(coordinator)**
+  - `benchmarks/measurements/<date>-unseen-07-baseline/record.json`: `score.py` output (counts, rates, Wilson and
+    bootstrap intervals, coverage per state, per family, method, stream), freeze digest, image digest, commit.
+    `usage-p1.jsonl` row `baseline`.
+  - Observable: guard green over the record; G0's exit met.
+  - _Requirements: 5.1, 5.2, 3.3_
+
+- [ ] 8. Arms on slice 1 (design 4.0, 5.4, 7.1-7.3; increment 8; G1)
+- [ ] 8.1 Registered long-deadline arm
+  - `plane/experiments/` manifest for the long-deadline arm: hook flags with `--deadline 300`, budget 0,
+    `unseen: {pool: p1, slice: 1}`, committed before its first replay. `src/openultrasast/learn/experiments.py`
+    accepts `unseen` and a hook-flag arm.
+  - `tests/test_learn_experiments.py`: the manifest loads; an `unseen` block without pool or slice is refused.
+  - _Requirements: 3.4, 5.3_
+- [ ] 8.2 `unseen` block and adoption gate in `analyse`
+  - `experiments.py` `analyse`: writes `unseen` into `result.json` (catch at each budget per arm, paired B - A,
+    McNemar, `gate`) from `score.py`; checks the usage ledger; adoption requires the fold rule and
+    `gate == pass`.
+  - Tests: `regression` and `not_run` each block adoption; an `informed` slice cannot qualify; result JSON shape.
+  - _Requirements: 5.2, 5.3, 5.4_
+- [ ] 8.3 Engine-evidence arm
+  - `benchmarks/unseen/arms.py`: a score per change from `engine.json` (witness rows on head-side changed
+    functions), no model; plugs into the budget thresholds of 6.4.
+  - Tests on synthetic records: score monotone in witness strength; a change without engine coverage gets no
+    score and is reported, not counted quiet.
+  - _Requirements: 3.4_
+- [ ] 8.4 Model-arm adapter
+  - `arms.py`: candidates from head-side changed functions of stored records, scored by the current decision
+    engine program (and the combiner when its program exists; else its row reads `not_built`); change score is the
+    maximum; metered client, dry run first, response cache, the manifest's `budget_usd` as the hard ceiling.
+  - Tests with a scripted client: dry-run estimate, ceiling stop, cache replay; no model call.
+  - _Requirements: 4.4, 3.4_
+- [ ] 8.5 Long-deadline replay of slice 1 **(coordinator, network)**
+  - 1,300 replays of the `long_deadline` arm, bounded at about 345 s each (up to about 42 hours over three
+    lanes), with `STOP` and resume across sessions; failures re-run once.
+  - Observable: 1,300 records under `~/ousast-results/unseen/p1/1/long_deadline/`, JVM time in each.
+  - _Requirements: 3.1, 3.4, 4.3_
+- [ ] 8.6 Model arms on slice 1 **(coordinator, network; paid: needs a budget go-ahead)**
+  - Dry run gives the candidate count; estimate at the injection slice's measured cost (about $0.008 per
+    candidate at k = 5); the maintainer sets the ceiling against the balance read just before. Then the decision
+    engine arm (and the combiner if built) on the VM.
+  - _Requirements: 4.4, 3.4_
+- [ ] 8.7 Record 08: the arms table **(coordinator)**
+  - `benchmarks/measurements/<date>-unseen-08-arms/record.json`: per arm (hook 30 s, hook 300 s, engine evidence,
+    decision engine, combiner or `not_built`) catch at 1/5/10% budgets with Wilson intervals and realised
+    false-alarm rates, coverage per state, spend; `exploratory` rows in `usage-p1.jsonl`.
+  - Observable: one committed table of arms on slice 1; G1's table exists.
+  - _Requirements: 3.4, 5.2, 5.3_