steering: roadmap re-centred on generalisation to unseen repositories (G0-G5); specs updated; unseen-repo-evaluation brief and requirements (approved)

Maintainer 2026-10-04: the safety net is for unseen code and the A/B tests exist to prove the approach generalises; infrastructure work had drifted from that. Runs are designed for AX on kube-ax, kind is the fallback. Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-18 04:29:27 +00:00 · 2026-10-04 17:34:57 +02:00
parent ee422b63e0
commit d4467f4236
11 changed files with 371 additions and 12 deletions
@@ -1,7 +1,7 @@
 {
  "feature_name": "learned-decision-engine",
  "created_at": "2026-09-30T14:00:00Z",
-  "updated_at": "2026-10-01T10:20:00Z",
+  "updated_at": "2026-10-04T18:00:00Z",
  "language": "en",
  "phase": "implementation",
  "approvals": {
@@ -8,6 +8,10 @@ memory, compiled DSPy-style in-house); tasks 1-5 are unaffected, tasks 6-10 are
 against the provider balance read just before it, and none starts with a ceiling above that balance. No test makes
 a model or embedding call.

+Direction 2026-10-04: `.kiro/steering/roadmap.md` (section of 2026-10-04, phases G0-G5) is the authority. Every
+change is judged on the unseen-repository pool (G0, spec `unseen-repo-evaluation`) by catch rate and false-alarm
+rate per push. Within-pair AUC stays a development signal. Tasks 11-14 below follow that roadmap.
+
 - [x] 1. Schema and features
  - `learn/schema.py` (closed, typed allow-list; no identities; missing != zero), `learn/features.py` (host
    builder), new memory `KINDS`, `plane/tasks/features.py` and the `remember` input; tests enforcing the
@@ -45,6 +49,11 @@ a model or embedding call.
  - _Requirements: 1.1_

 - [ ] 6. Program, local memory and compilation (design sections 4-5)
+  - Status 2026-10-04: 6.1-6.6 are done. 6.7 is partial: leave-one-source-out, the 0/25/50% memory-size points and
+    the source-count curve are not run. Around this task the experiment machinery grew: rule arms
+    (`precision_bound_block`, `pooled_block_gate` with a family floor) in `learn/experiments.py`, the response-cache
+    fill with a hard ceiling and a dry-run estimate in `learn/fill.py`, and the in-function slice in `learn/slice.py`.
+    These ran exp-003, exp-004 and exp-005 (7.5-7.7). Memory now holds 710 complete fix pairs (7.9).
  - [x] 6.1 Excerpts and the example store (no model)
    - `learn/excerpt.py` (function span at the label's pin, bounds 80/4,000 for candidates and 40/2,000 for
      examples, numbered lines, no path/repository/commit, advisory ids and security wording in comments
@@ -151,6 +160,28 @@ a model or embedding call.
    rows' null plane-only features (1f525ff), the 788 embedding vectors absent from RustFS (copied from the file
    store), the result row's column-type collision under S3 Select (780caaa). Further program variants (`k = 5` vs
    `k = 1`, the confidence source) as new manifests.
+  - [x] 7.5 exp-003 run and recorded 2026-10-03 ($0, replay only): **inconclusive**. Selective blocking (a per-family
+    raw-score threshold chosen on training folds by its Wilson lower bound) offers no BLOCK on held-out paired units
+    in any family; the incumbent stays. 211 of 353 frozen units evaluated (142 replay misses). Proving 0.95 needs at
+    least 73 clean flags. Result: `benchmarks/experiments/exp-003-precision-bound-blocking/result.json`.
+  - [x] 7.6 exp-004 run and recorded 2026-10-03: **inconclusive**. One pooled BLOCK threshold over all families with a
+    family floor (Req 6.4). No outer fold found a threshold, pooled or per family; B offers no BLOCK. 501 units,
+    replay at $0 after a cache fill of $1.24 (ceiling $2; the dry-run estimate over-priced 4.5x). Result and record:
+    `benchmarks/experiments/exp-004-pooled-block-gate/result.json`, `record.json`.
+  - [x] 7.7 exp-005 run and recorded 2026-10-03: **inconclusive**. The in-function def-use slice in the prompt of the
+    six slice-2 programs. Within-pair AUC 0.762 -> 0.746 (CI [-0.063, 0.032], spans 0); unsure rate 0.29 -> 0.24
+    (favours B); usd per candidate slightly higher. A $0 leak audit came first
+    (`benchmarks/measurements/2026-10-03-graph-slice-leak-audit/record.json`). Result:
+    `benchmarks/experiments/exp-005-graph-slice/result.json`.
+  - [x] 7.8 Negatives harvest 2026-10-03 ($0.0014 embeddings, no chat-model call): 98 real fix pairs whose function
+    survives the fix, from reviewed advisories, as pointer-pair corpus `advisory-fixes-2`; memory 937 -> 1133
+    examples. Record: `benchmarks/measurements/2026-10-03-decision-engine-negatives-harvest/record.json`.
+  - [x] 7.9 Harvest 3, 2026-10-04 ($0.003 embeddings, no chat-model call): 251 pairs as `advisory-fixes-3`; complete
+    pairs in memory 459 -> 710. Record: `benchmarks/measurements/2026-10-04-decision-engine-harvest-3/record.json`.
+  - [x] 7.10 Joern trace run over exp-005's 501 units and its leak audit, 2026-10-04 ($0): no leak flagged. In the 105
+    pairs where both sides were asked, a path appears on both sides in 12, on the vulnerable side only in 2, on the
+    fixed side only in 0, and on neither in 91. The engine cannot see what a fix changed. Record:
+    `benchmarks/measurements/2026-10-04-joern-slice-leak-audit/record.json`.
  - _Requirements: 3.6, 4.1-4.4_

 - [ ] 8. Integration
@@ -176,3 +207,33 @@ a model or embedding call.
    `result-v3-engine.json` is committed. Nothing between the two commits may touch the program, memory snapshot,
    features, priors or thresholds. This spec's code never opens v3 files.
  - _Requirements: 5.3_
+
+- [ ] 11. Fix mechanisms (plan approved 2026-10-04; see `fix-mechanisms-plan.md`)
+  - Roadmap G2 arm (`.kiro/steering/roadmap.md`, section of 2026-10-04): one candidate improvement, not the spine.
+    It is judged on the unseen-repository pool (G0, spec `unseen-repo-evaluation`) by catch rate and false-alarm
+    rate per push. **Paused after 11.1 until the G0 pool exists.**
+  - [ ] 11.1 Mine candidate discharge facts from the 710 fix diffs (sanitizer calls, guarding conditions, safe sink
+    arguments, safe replacements); deterministic, $0. In progress.
+  - [ ] 11.2 Admission by held-out separation per outer fold: a candidate turns "path on both sides" into "path on the
+    vulnerable side only" on folds it was not mined from, removes no held-out vulnerable path, has at least 2
+    proposing repositories, and is admitted per language.
+  - [ ] 11.3 Engine re-run on the affected pins (kube-ax lanes plus the VM lane).
+  - [ ] 11.4 Registered experiment, with the detection gates green, before `ruleset/semantic/*.toml` changes.
+  - [ ] 11.5 Coverage increment: source and sink additions for pairs where neither side has a path (later).
+  - [ ] 11.6 TypeScript admission in the engine (branch `engine-typescript`).
+  - _Requirements: 4.1-4.4, 7.1, 8.1_
+
+- [ ] 12. exp-006: Joern witness slice in the prompt **(budget go-ahead, about $2-4)**
+  - Planned, roadmap G1/G2. Against the current prompt, after a $0 leak audit, on re-frozen units. Judged on the
+    unseen-repository pool, not on within-pair AUC alone.
+  - _Requirements: 4.1-4.4, 7.1_
+
+- [ ] 13. exp-007: the combiner ($0 replay)
+  - Planned, roadmap G1/G2. Three or four weights (score, engine entailment, neighbour agreement) over cached
+    responses. Judged on the unseen-repository pool.
+  - _Requirements: 4.1-4.4, 7.1_
+
+- [ ] 14. Re-run the pooled block gate on re-frozen units
+  - Planned, roadmap G3. The exp-004 design with the family floor (Req 6.4), together with the pool's false-alarm
+    rate per family.
+  - _Requirements: 6.4, 7.3_
@@ -1,5 +1,20 @@
 # Design Document

+> **Status 2026-10-04.** Groups 1-2 are done on kind (records `benchmarks/measurements/2026-10-02-k8s-01-profile/`
+> and `2026-10-02-k8s-02-store-delivery/`). Task 2.6 is paused. The production target changed. The operator runs a
+> kube-ax cluster (k3s, Agent Substrate v0.3.0, AX v0.3.1, 2 nodes). There we may only submit AX Tasks: no k8s
+> object creation, no secrets, no access to `ate-system`. The engine now runs there as AX Tasks through
+> `benchmarks/learn/engine_trace_ax.py`. The engine-task image (`plane/Dockerfile.engine-task`) is built FROM
+> `ghcr.io/norandom/ax-task-runner:v0.3.1`. Tasks read and write through presigned HTTPS links, are named
+> `ousast-engine-*`, and run a 1.7 GB heap with `ActiveProcessorCount=2` and one resident Joern session per pin. A
+> pin whose sandbox dies is retried once under a new name, then handed to the VM lane (2.5 GB heap). Egress is
+> operator-managed: HTTPS 443 to public hosts is open; LAN and cluster-internal addresses are blocked. Archive
+> names (`.tar`, `.gz`, `.zip`, `.bin`) failed through the store host's Cloudflare cache until the operator added a
+> cache-bypass rule; presigned objects are staged as `.dat` as a safeguard. Groups 3-8 are **to re-plan** against
+> that cluster under roadmap G4 (`.kiro/steering/roadmap.md`, section of 2026-10-04). Their text below is kept
+> as written. G4 (detection on push for any repository: `ousast plane scan --base --head`, engine pins on kube-ax,
+> model calls on the VM or CI) is the new target for group 7, sized by what G1-G3 keep.
+
 ## Overview

 The plane keeps its shape: `ousast plane run` is the reconciler, ax is the only executor, every task is one
@@ -1,7 +1,7 @@
 {
  "feature_name": "plane-on-kubernetes",
  "created_at": "2026-10-02T20:00:00Z",
-  "updated_at": "2026-10-02T22:00:00Z",
+  "updated_at": "2026-10-04T18:00:00Z",
  "language": "en",
  "phase": "implementation",
  "approvals": {
@@ -1,5 +1,20 @@
 # Implementation Plan

+> **Status 2026-10-04.** Groups 1-2 are done on kind (records `benchmarks/measurements/2026-10-02-k8s-01-profile/`
+> and `2026-10-02-k8s-02-store-delivery/`). Task 2.6 is paused. The production target changed. The operator runs a
+> kube-ax cluster (k3s, Agent Substrate v0.3.0, AX v0.3.1, 2 nodes). There we may only submit AX Tasks: no k8s
+> object creation, no secrets, no access to `ate-system`. The engine now runs there as AX Tasks through
+> `benchmarks/learn/engine_trace_ax.py`. The engine-task image (`plane/Dockerfile.engine-task`) is built FROM
+> `ghcr.io/norandom/ax-task-runner:v0.3.1`. Tasks read and write through presigned HTTPS links, are named
+> `ousast-engine-*`, and run a 1.7 GB heap with `ActiveProcessorCount=2` and one resident Joern session per pin. A
+> pin whose sandbox dies is retried once under a new name, then handed to the VM lane (2.5 GB heap). Egress is
+> operator-managed: HTTPS 443 to public hosts is open; LAN and cluster-internal addresses are blocked. Archive
+> names (`.tar`, `.gz`, `.zip`, `.bin`) failed through the store host's Cloudflare cache until the operator added a
+> cache-bypass rule; presigned objects are staged as `.dat` as a safeguard. Groups 3-8 are **to re-plan** against
+> that cluster under roadmap G4 (`.kiro/steering/roadmap.md`, section of 2026-10-04). Their text below is kept
+> as written. G4 (detection on push for any repository: `ousast plane scan --base --head`, engine pins on kube-ax,
+> model calls on the VM or CI) is the new target for group 7, sized by what G1-G3 keep.
+
 One task group per increment of `design.md` section 7, in that order; a later group never starts before the
 earlier group's kind proof is recorded. Every task ships with its tests; a task is done when its tests pass in
 the host suite, ruff and mypy are clean, and nothing in `src/`, `plane/`, `ops/` (minus `ops/ax/up.sh` and
@@ -9,7 +24,7 @@ its kind proof and the record it writes under `benchmarks/measurements/<date>-k8
 (commit, images, the instrument checks, the numbers). Steps that call a model are marked **paid: needs a budget
 go-ahead** with a ceiling; steps that need a hand on a cluster that does not exist yet are marked **manual**.

- [ ] 1. Profile and literals (design section 1; open-question checks 2, 3, 7)
+- [x] 1. Profile and literals (design section 1; open-question checks 2, 3, 7)
 - [x] 1.1 `PlaneProfile`
  - `src/openultrasast/plane/profile.py`: frozen dataclass with the fields of design section 1; `load_profile()`
    reads `ops/k8s/profiles/<OUSAST_PLANE_PROFILE>.toml` (a name or a path; default `kind`), then applies
@@ -107,7 +122,7 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
    prints the table. Instrument: the tar's byte size from the store equals the extracted files' total.
  - `benchmarks/measurements/<date>-k8s-02-store-delivery/record.json`: bytes, PUT-to-poll seconds, rules.
  - _Requirements: 3.1, 3.2, 7.1_
- [ ] 2.6 Remove the receiver (last; after 2.5 is recorded)
+- [ ] 2.6 Remove the receiver (last; after 2.5 is recorded) -- paused 2026-10-04
  - Delete `Receiver`, `_Handler`, `receiver_address`, `receiver_cluster_ip`, `RECEIVER_HOST`, `RECEIVER_PORT`
    from `egress.py`; `OUSAST_ARTIFACT_URL/DIAL/HOST/PORT` from `runner.py` and `reconciler.py`; the
    `OUSAST_DELIVERY` switch; `ops/ax/receiver-service.yaml.tmpl` and the `up.sh` receiver step; the EndpointSlice
@@ -117,7 +132,7 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
    `receiver_removed: <commit>`.
  - _Requirements: 3.1, 1.3_

- [ ] 3. `--exec remote` from a CI-shaped principal (design section 2; open-question check 1)
+- [ ] 3. `--exec remote` from a CI-shaped principal (design section 2; open-question check 1) -- to re-plan (kube-ax, roadmap G4)
 - [ ] 3.1 Execution mode and RBAC manifests
  - `cli.py`: `ousast plane run --exec local|remote` (default `profile.exec`); `reconciler.Ax` passes the
    profile's context to `ax` (its context flag, read from `ax --help` and pinned in a test); `remote` refuses a
@@ -150,7 +165,7 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
    status, usd spent.
  - _Requirements: 2.1, 2.2, 2.3, 2.4, 7.1_

- [ ] 4. Images on GHCR (design section 8)
+- [ ] 4. Images on GHCR (design section 8) -- to re-plan (kube-ax, roadmap G4)
 - [ ] 4.1 `images.yml`
  - `.github/workflows/images.yml` on tags `v*`: `docker/build-push-action` for `plane/Dockerfile.runner` ->
    `ghcr.io/norandom/ousast-runner:<tag>` and the root `Dockerfile` -> `ghcr.io/norandom/ousast-engine:<tag>`;
@@ -178,7 +193,7 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
  - `benchmarks/measurements/<date>-k8s-04-ghcr-images/record.json`: image sizes, pull seconds, visibility.
  - _Requirements: 1.1, 4.2, 7.1_

- [ ] 5. The engine as a plane task (design section 4)
+- [ ] 5. The engine as a plane task (design section 4) -- to re-plan (kube-ax, roadmap G4)
 - [ ] 5.1 `cpg/dump.py` and `tasks/engine.py`
  - Lift `benchmarks/push/finding_dump.py`'s scan into `src/openultrasast/cpg/dump.py` (the script becomes a
    thin caller; `ops/README.md:78` command unchanged). `src/openultrasast/plane/tasks/engine.py`: env
@@ -208,7 +223,7 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
    placement, image size and pull seconds.
  - _Requirements: 4.1, 4.2, 7.1_

- [ ] 6. Pools, autoscaling, node checklist (design section 5)
+- [ ] 6. Pools, autoscaling, node checklist (design section 5) -- to re-plan (kube-ax, roadmap G4)
 - [ ] 6.1 `plane/k8s.py` and the manifests
  - `src/openultrasast/plane/k8s.py`: `render(profile, out)` writes `workerpool-default.yaml`,
    `workerpool-engine.yaml`, `ax-server-snapshots.yaml` (patch with `profile.snapshots_bucket`), `rbac-ci.yaml`;
@@ -232,7 +247,7 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
  - `benchmarks/measurements/<date>-k8s-06-pools/record.json`: footprint table, securityContext, dry-run result.
  - _Requirements: 5.1, 5.2, 7.1_

- [ ] 7. `ousast plane scan` (design section 6)
+- [ ] 7. `ousast plane scan` (design section 6) -- to re-plan (kube-ax, roadmap G4)
 - [ ] 7.1 Workspaces and candidates from URL + commits
  - `generate.scan_cases(url, base, head, cache, *, all=False)`: shallow clone into the case cache, two pinned
    Workspaces, head-side hunks via `fix_ranges(side="new")`, functions via `repo_facts.enclosing` over
@@ -265,7 +280,7 @@ go-ahead** with a ceiling; steps that need a hand on a cluster that does not exi
    seconds from submit to verdict, whether the known CVE site is among the agreed.
  - _Requirements: 6.1, 6.2, 6.3, 7.1_

- [ ] 8. Runbook (design section 7, Req 7)
+- [ ] 8. Runbook (design section 7, Req 7) -- to re-plan (kube-ax, roadmap G4)
 - [ ] 8.1 `docs/deployment.md` as the runbook
  - Sections: prerequisites for k3s (Substrate, ax with the snapshots patch, GHCR and the pull secret, buckets,
    the CI SA and its grant from 3.2, the node checklist of 6.2), the profile switch, `--exec remote` from a laptop
@@ -623,6 +623,7 @@ Preserve all completed task evidence below.
 - [x] 17. Operational readiness (2026-10-02)
  - **Named by the 2026-10-02 operational-readiness audit**, maintainer-approved as the first increment. The maintainer's definition: a user installs the hook on ANY repository and gets honest, advisory-only results; nothing ever blocks a push. The audit found the plumbing sound (snapshots, resolution, deadline accounting, advisory exit 0, the JSON artifact) and five things in the way: the hook can block a push, quick rules are not part of pre-push, the engine cannot finish a 26-file repository inside 30 s, the terminal hides every skip reason, and there is no supported install path without the source checkout. Before numbers for every item are the audit's live runs on records (Python), serve-static (JavaScript) and unrolled/secure (Go), each with one planted commit.
  - This group is plumbing and honesty. It enables no capability, edits no rule for a planted sample, and leaves graph reuse (the engine speed work decided in 11.6) as a later task.
+  - Record: `benchmarks/measurements/2026-10-02-pre-push-operational-1/record.json` (see 17.7).
  - _Done 2026-10-02:_ all seven subtasks done. The engine still does not fit the 30 s hook deadline (graph reuse, the 11.6 direction, stays a later task), and plane verdicts after the push were not in this increment.
  - _Requirements: 9.1–9.6_
 - [x] 17.1 Never block a push the user did not choose to block
@@ -0,0 +1,32 @@
+# Brief: does the safety net work on repositories it has never seen?
+
+Maintainer, 2026-10-04: "we want to spread the safety net across repos in general, unseen code. that was the purpose
+of the AB tests to see if our approach is generalizeable." And: "we should run our tests / runs in kube. now we can
+speed it up and design the workloads for ax directly. kind is only a fallback."
+
+## Why this spec exists
+
+The A/B experiments so far (exp-003, exp-004, exp-005) score the decision engine on repository-held-out folds of known
+fix pairs. That measures whether the vulnerable side of a pair scores above its fixed side. It does not measure what a
+safety net on an arbitrary repository lives on:
+
+- on an ordinary push, does it stay quiet? (false-alarm rate)
+- on a push that introduces a real vulnerability, does it say so? (catch rate)
+
+Populations v1 and v2 measured the engine scan on fresh cases once (1 of 11, 0 of 17) and are spent. Nothing since has
+measured push-level behaviour on unseen repositories.
+
+## What this spec delivers
+
+1. A frozen pool of repositories never used anywhere in development, with two kinds of change each: real
+   vulnerability-introducing commits and ordinary commits.
+2. A push-level metric: each change replayed through the hook exactly as a user would run it, and through other
+   arms (decision engine, engine evidence) where an experiment asks.
+3. Execution designed for the kube-ax cluster from the start: one push replay is one AX Task; kind and the VM are
+   fallbacks.
+4. A baseline for today's hook, then a table of arms scored on the same pool.
+
+## Not in scope
+
+Changing detection. This spec only builds the measurement. Improvements are separate, registered experiments judged
+by it.
@@ -0,0 +1,103 @@
+# Requirements Document
+
+## Introduction
+
+The safety net is meant for repositories it has never seen. This spec builds the instrument that measures that: a
+frozen pool of unseen repositories, a push-level metric (catch rate and false-alarm rate as the hook reports them),
+and an execution path that runs the replays as AX Tasks on the kube-ax cluster. Context: `brief.md`, the roadmap's
+G0-G1 (`.kiro/steering/roadmap.md`).
+
+## Boundary Context
+
+- **In scope**: pool selection and freezing; change extraction (vulnerability-introducing and ordinary commits);
+  replay of each change through `ousast pre-push` and through registered experiment arms; the metric and its
+  confidence intervals; AX Task execution with kind/VM fallback; the baseline for today's hook; rotation of pool
+  slices.
+- **Out of scope**: any detection change; population v3 (stays reserved and unread); model calls inside the cluster.
+- **Adjacent expectations**: `learned-decision-engine` experiments use this pool as their primary unseen-repo score;
+  `pre-push-safety-net` owns the hook being replayed; `plane-on-kubernetes` and the AX executor
+  (`benchmarks/learn/engine_trace_ax.py`) supply the execution pattern; the reserved-repository guard covers every
+  population and this pool.
+
+## Requirements
+
+### Requirement 1: A pool no development step has touched
+
+**User Story:** As the maintainer, I want the evaluation repositories to be ones nothing in development has seen, so a
+good number means the approach generalises.
+
+#### Acceptance Criteria
+
+1. A repository is eligible only if it appears in no pair corpus, no population (v1, v2, v3), no harvest record, no
+   memory example, no experiment unit and no measurement record. Eligibility is checked by code against every source
+   before the pool is frozen; v3 is checked through the existing guard without being opened.
+2. The pool is frozen (a committed manifest with repository URLs, commit SHAs, licences and the freeze digest) before
+   any change in it is replayed. A frozen pool is never edited; a correction is a new pool.
+3. Pool repositories are referenced only inside the pool's own directory (like `benchmarks/independent/`), and the
+   reserved-repository guard is extended to them.
+4. Only permissively licensed repositories are published by name; others may be used with their names kept out of
+   the repository (pointers in a local, gitignored manifest), recorded as such.
+
+### Requirement 2: Two kinds of change per repository
+
+**User Story:** As the maintainer, I want both vulnerable and ordinary pushes from the same repositories, so the
+false-alarm rate is measured on the code the catch rate is measured on.
+
+#### Acceptance Criteria
+
+1. Vulnerability-introducing changes: from public advisories with a linked fix, the commit range that introduced the
+   vulnerable code where it can be determined, else the fix's parent compared with an earlier base; each labelled
+   with family and the vulnerable function. Post-model-cutoff advisories are preferred and flagged.
+2. Ordinary changes: commits from the same repositories' histories with no security fix, revert or advisory link,
+   sampled across time and size; each recorded with its changed files and lines.
+3. Per repository, at least one vulnerability-introducing change and several ordinary changes; the pool's totals are
+   sized so the false-alarm rate's 95% interval is no wider than +/- 3 points and the catch rate's no wider than
+   +/- 10 points (sizes computed in the design).
+
+### Requirement 3: The push-level metric
+
+**User Story:** As the maintainer, I want the score to be what a user would see, so the number means something.
+
+#### Acceptance Criteria
+
+1. Each change is replayed as `ousast pre-push --base <base> --head <head>` with the hook's default settings, in a
+   clean checkout, with the record of every skipped check kept.
+2. Catch: a vulnerability-introducing change counts as caught when the hook (or the arm) reports a finding of the
+   right family on the vulnerable function or its changed lines. False alarm: an ordinary change counts as a false
+   alarm when the hook reports any advisory finding.
+3. Reported per family and overall: catch rate, false-alarm rate, and both with 95% Wilson intervals; plus coverage
+   (changes where the hook could analyse the changed language) so a silent zero is visible.
+4. Arms (decision engine, combiner, engine evidence, fix mechanisms) are scored on the same replays: catch rate at
+   fixed false-alarm budgets (1%, 5%, 10%), so arms are compared at equal noise.
+
+### Requirement 4: Execution on kube-ax, designed for AX
+
+**User Story:** As the maintainer, I want the replays to run on the cluster, so measurements are fast and the VM is
+free.
+
+#### Acceptance Criteria
+
+1. One replayed change is one AX Task, named with the `ousast-engine-` prefix the operator's egress watcher matches,
+   using an image built `FROM ghcr.io/norandom/ax-task-runner:v0.3.1` with this package, Joern and php-cli, pinned by
+   digest.
+2. The task fetches the repository at the needed commits itself over HTTPS (public hosts are reachable from actors)
+   and writes its record through a presigned link; no API keys or store credentials enter the cluster.
+3. A generic AX batch dispatcher serves both this replay and the existing engine passes: items to Tasks, at most the
+   cluster's safe concurrency (two heavy tasks per worker), retry once on sandbox death, then a VM or kind fallback
+   lane; progress and resume as in the engine runner.
+4. Arms that need model calls take the cluster's records and make their calls on the VM, within registered budgets.
+5. kind runs the same tasks unchanged as the local fallback.
+
+### Requirement 5: Baseline, rotation and use
+
+**User Story:** As the maintainer, I want a baseline now and a fresh slice for each decision, so the pool is not
+worn out by tuning.
+
+#### Acceptance Criteria
+
+1. The first use is a baseline of today's hook on the first slice; its record is committed (counts and digests only,
+   no repository names outside the pool directory).
+2. A slice that informed a change can no longer qualify that change; each adoption decision draws a new, untouched
+   slice, recorded with its freeze digest.
+3. Every learned-decision-engine experiment from now on reports its unseen-repo score from this pool next to its
+   held-out-fold score.
@@ -0,0 +1,22 @@
+{
+  "feature_name": "unseen-repo-evaluation",
+  "created_at": "2026-10-04T18:00:00Z",
+  "updated_at": "2026-10-04T18:30:00Z",
+  "language": "en",
+  "phase": "requirements-approved",
+  "approvals": {
+    "requirements": {
+      "generated": true,
+      "approved": true
+    },
+    "design": {
+      "generated": false,
+      "approved": false
+    },
+    "tasks": {
+      "generated": false,
+      "approved": false
+    }
+  },
+  "ready_for_implementation": false
+}