From 60512fff8841e1a01c6008de7247941b0ec6b800 Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Sun, 4 Oct 2026 18:48:13 +0200 Subject: [PATCH] spec(unseen-repo-evaluation): design and tasks approved; run through the baseline Decisions: long-deadline arm (300 s), adoption needs no unseen regression (Req 5.4), permissive repositories by name only, two AX lanes. Also adds the trained-classifier and joint-optimisation arms to learned-decision-engine. Co-Authored-By: Claude Opus 5.5 --- .kiro/specs/learned-decision-engine/tasks.md | 17 + .kiro/specs/unseen-repo-evaluation/design.md | 431 ++++++++++++++++++ .../unseen-repo-evaluation/requirements.md | 4 + .kiro/specs/unseen-repo-evaluation/spec.json | 14 +- .kiro/specs/unseen-repo-evaluation/tasks.md | 321 +++++++++++++ 5 files changed, 780 insertions(+), 7 deletions(-) create mode 100644 .kiro/specs/unseen-repo-evaluation/design.md create mode 100644 .kiro/specs/unseen-repo-evaluation/tasks.md diff --git a/.kiro/specs/learned-decision-engine/tasks.md b/.kiro/specs/learned-decision-engine/tasks.md index 4519cd6..c980443 100644 --- a/.kiro/specs/learned-decision-engine/tasks.md +++ b/.kiro/specs/learned-decision-engine/tasks.md @@ -237,3 +237,20 @@ rate per push. Within-pair AUC stays a development signal. Tasks 11-14 below fol - Planned, roadmap G3. The exp-004 design with the family floor (Req 6.4), together with the pool's false-alarm rate per family. - _Requirements: 6.4, 7.3_ + +- [ ] 15. Trained classifier arm (maintainer decision 2026-10-04: train a model, not hand-set profiles; judged on the + unseen-repo pool, roadmap G1) + - [ ] 15.1 Features: code embedding of the excerpt (stored vectors) + every allow-listed signal; no hand-picked + profile. Automated screening: a signal that separates pair sides by itself on training folds above the leak + threshold is dropped for that fold (the leak audit as a filter, fitted per fold). + - [ ] 15.2 Models: regularised logistic regression and gradient-boosted trees, fitted per repository-held-out outer + fold with inner-fold hyper-parameter search; per-family and pooled variants. + - [ ] 15.3 Scores on held-out folds: within-pair AUC, pooled AUC, top-k precision, Wilson bound at the pooled + threshold; compared with the current LLM programs on the same units. + - [ ] 15.4 Scored on the unseen-repo pool when G0 exists: catch rate at fixed false-alarm budgets. + - _Requirements: 3 (revised 2026-10-04 by maintainer decision), 6, 7_ + +- [ ] 16. Jointly optimised LLM program arm (judged on the unseen-repo pool) + - Search input fields, retrieval parameters, k, instruction and demonstrations together (DSPy MIPRO-style) on + training folds; no hand-set input profile. Budget go-ahead before paid search. + - _Requirements: 3, 6, 7_ diff --git a/.kiro/specs/unseen-repo-evaluation/design.md b/.kiro/specs/unseen-repo-evaluation/design.md new file mode 100644 index 0000000..4b7b492 --- /dev/null +++ b/.kiro/specs/unseen-repo-evaluation/design.md @@ -0,0 +1,431 @@ +# Design Document + +## Overview + +This spec builds the instrument for roadmap G0: a frozen pool of repositories that no development step has seen, +and a push-level score on it. The score is what a user of the hook sees. On a vulnerability-introducing push, did +the hook say so (catch rate)? On an ordinary push, did it say anything (false-alarm rate)? Replays run as AX Tasks +on kube-ax. kind and the VM run the same tasks as fallbacks. + +Four facts from the code shape the design: + +- **Populations already have a freeze-and-guard pattern.** `benchmarks/independent/freeze.py` verifies cases by + reading git only, `evaluate.py` scores with fixed rules and Wilson intervals, and + `tests/test_independent_population.py` fails when a reserved repository is named anywhere in `benchmarks/`, + `src/` or `tests/` outside its directory. Harvest batch 3 used that guard as a black box: candidates written, + guard run with output discarded, subsets bisected on red. That is how v3 stays unread. +- **The hook can already replay a change.** `ousast pre-push PATH --base B --head H --artifact F` resolves two + local commits (`snapshot.py`, `resolve_replay`: no fetch, no merge-base), runs quick rules and the engine inside + one deadline (default 30 s, engine inline) and writes a JSON artifact. The terminal text has one `Skipped:` line + per skipped check. The artifact holds three finding streams: admitted defects, advisory engine findings + (new or worsened, vetoed only by capability) and quick-rule matches with their CWE. +- **The AX pattern works but lives in one script.** `engine_trace_ax.py` (manifest, egress checks, presigned + objects, retry once under a new name, cleanup) and the lane scheduler in `engine_trace.py` (`run_plan`: fixed + lanes, fallback queue, `progress.json`, `STOP`, resume by input digest) are engine-specific. +- **Placement on kube-ax is random.** One worker per node, shared by all actors. Two heavy tasks fit per worker, + but only two concurrent tasks in all guarantee at most two on one worker. + +### Goals + +- A pool frozen by code against every used source (Req 1), with two kinds of change per repository (Req 2). +- A scorer whose numbers mean what the hook shows, with silent zeros visible (Req 3). +- One replay is one AX Task via a dispatcher shared with the engine (Req 4); a baseline and fresh slices (Req 5). + +### Non-Goals + +Detection changes, population v3, model calls in the cluster, languages the hook does not analyse. + +## Architecture + +```mermaid +flowchart LR + M[pool manifest slice] --> D[batch dispatcher on VM] + D --> P[presign spec GET and record PUT] + P --> T[AX Task ousast-engine-replay] + T --> G[shallow fetch base and head from GitHub] + G --> H[ousast pre-push --base --head] + H --> R[record tar: artifact, text, instrument] + R --> S[(store via presigned PUT)] + D -->|sandbox died twice or OOM| V[VM docker lane] + D -->|cluster down| K[kind lane, same tasks] + V --> S + K --> S + S --> C[scorer: catch, false alarm, coverage] + S --> A[model arms on VM, budgeted] + A --> C + C --> O[record: counts and digests] +``` + +## Components and Interfaces + +### 1. Pool sourcing and eligibility (Req 1) + +**Decision 1.1: sources and ecosystems.** Candidates come from GitHub reviewed advisories (`gh api /advisories`, +the enumeration harvest-3 used) for pip, npm, maven and composer, published 2019 or later, whose CWE maps to one of +the seven families through the project's family CWE table. Each needs a linked fix commit with one parent in one +repository. OSV records for the same advisory give the first affected version. That version is a cross-check for +the introducing commit (section 2). Reason: these four ecosystems are the five languages the hook analyses +(Python, JavaScript, TypeScript, PHP, Java). GHSA links commits and CWEs; OSV adds version brackets. Rejected: +drawing from every ecosystem (Go, Ruby, C#). That would mostly measure missing frontends, which is already known. +Coverage still reports uncovered files inside the drawn repositories. Rejected: NVD and published fix datasets. +NVD links commits weakly, and the datasets are mined corpora that development or model training has likely seen. + +**Decision 1.2: eligibility by code against every used source.** `benchmarks/unseen/eligibility.py` builds the +used set before the draw. It never prints names; it records counts per source and a digest of the set. + +| Source (Req 1.1) | Where it is read | +| --- | --- | +| Pair corpora | every catalog, recipe and dataset file under `benchmarks/pairs/`, plus the local pointer-pair manifests | +| Populations v1, v2 | `population-v1.toml`, `population-v2.toml`, read directly | +| Population v3 | never opened; the existing guard test only (below) | +| Harvest records | the batch catalogs under `benchmarks/pairs/advisory-fixes*` (the records hold counts only) | +| Memory examples | `example` rows of the local FileStore and the S3 store (`repo` field) | +| Experiment units | `plane/experiments/*.units.jsonl` and registered units files | +| Measurement records | everything under `benchmarks/measurements/` and `benchmarks/experiments/` | +| Anything else | a text sweep for `github.com//` over `benchmarks/`, `src/`, `tests/`, `plane/`, `.kiro/` | + +Names are normalised (lower case, no `.git`, no trailing slash). Each candidate is resolved through the GitHub API +to its canonical `full_name`, its former names (redirects) and its fork network (`parent`, `source`). A candidate +is out if any of these is in the used set. A fork shares code with what it was forked from. + +v3 is checked like harvest-3 did. The draft manifest is written into `benchmarks/unseen/`. Then +`pytest -q tests/test_independent_population.py -k referenced` runs with its output discarded. The guard already +sweeps `benchmarks/` outside the population directory. On red, the candidate list is bisected by rewriting the +draft until green, and the dropped count is recorded. No v3 name is ever printed or read by a person. + +**Decision 1.3: post-cutoff preference.** An advisory published after 2026-06-01 (harvest-3's cutoff for the +decision engine's model) is flagged `post_cutoff`. Within each ecosystem stratum post-cutoff advisories are drawn +first. Results are reported for both groups. Reason: a post-cutoff advisory cannot be in the model's training +data as a known vulnerability. Rejected: post-cutoff only. Harvest-3 kept 12 post-cutoff pairs out of 251, and +three slices of 100 repositories cannot be filled from that window. + +**Decision 1.4: licences.** The licence is read at the pinned head commit (LICENSE file, then the API's SPDX id). +Only permissive licences are published by name: MIT, Apache-2.0, BSD-2-Clause, BSD-3-Clause, ISC, 0BSD, Unlicense, +Zlib, PostgreSQL, BSL-1.0. Everything else (copyleft, source-available, none) goes to the gitignored +`benchmarks/unseen/private/pool-p1.toml`. The committed manifest then holds an opaque id and the sha256 of +(URL, commits) for that entry, so the freeze digest still covers it. Reason: Req 1.4 read literally. Rejected: +"OSI-approved" as the bar, as population v2 used. It is wider than the requirement says (open question 1, +resolved 2026-10-04: copyleft stays private). + +Also required: anonymous HTTPS, at most 500 MB, at least 40 eligible ordinary commits, one per fork network. + +### 2. Change extraction (Req 2.1, 2.2) + +Extraction runs on the VM over a full `--filter=blob:none` clone in a scratch cache. It uses git only, like +`freeze.py`. No scanner, engine or model touches a pool repository before the freeze. + +**Decision 2.1: the vulnerability-introducing range.** Let F be the fix and P its parent. The vulnerable lines are +the old-side lines of F's hunks in the vulnerable function. For a pure-addition fix (an absence bug such as a +missing guard), they are the vulnerable function's own lines. + +1. **`introducing` (preferred).** `git blame -w -M -C --porcelain P -L -- `. The candidate V is the + commit that last wrote the sink line (for absence bugs, the function's declaration). V is accepted if all hold: + at V's first parent the sink line's normalised text is absent from the file's vulnerable function; V is not a + merge or root commit; V changes at most 50 files and 2,000 lines; when OSV names a first affected version, V is + reachable from that tag and not from the previous release tag. If the text is present at V's parent, blame + walks back from there, at most five steps. The replay is `base = V^`, `head = V`. +2. **`last_touch` (fallback).** When no V passes, `git log -L :: P` gives the newest commit L + before F that changed the vulnerable function. The replay is `base = L^`, `head = P`. The vulnerable code is in + head, and the range contains a change to the vulnerable function. + +Each change records its method, family, vulnerable function (resolved in head), head-side changed lines, +`post_cutoff` and the advisory id. + +Bias of each, stated in the record. `introducing` inherits SZZ noise: blame finds the last writer, not the author +of the flaw. A refactor, a move the heuristics miss or a reformat can stand in for the real introduction. The +absence check, the walk-back and the version bracket cut this down, but do not remove it. The bias runs both +ways: a wrong V may introduce nothing (a miss is not the hook's fault) or may move a flaw (the delta-based hook is +right to call it existing). `last_touch` is pessimistic for a delta hook. The flaw is often already in base, so +the hook's novelty rule correctly stays quiet and the change counts as a miss. Catch is therefore reported per +method. The headline is `introducing`; `last_touch` is a lower bound. Rejected: fix-parent against a fixed old +base (for example 50 commits back). The range often misses the vulnerable function entirely and grows without +bound on busy repositories. + +Label check: every third `introducing` change is read by an agent before the freeze, with no model call, as +harvest-3 reviewed diffs. It is marked `confirmed`, `refactor` or `unclear`. The confirmed share is the label +noise estimate. The scorer also reports catch on the confirmed subset. + +**Decision 2.2: ordinary commits.** The candidates are non-merge commits on the first-parent line of the default +branch that change at least one source file of an analysed language. Excluded: + +- explicit security signals in the message: `secur|cve|ghsa|vuln|inject|xss|csrf|ssrf|exploit|sanitiz|escap`; +- reverts (`^Revert`, `This reverts commit`); +- dependency bumps (bot authors such as dependabot and renovate, `bump|upgrade .* from`, or only manifest and lock + files changed); +- commits whose source changes are all generated or vendored (`vendor/`, `node_modules/`, `dist/`, `*.min.js`, + lock files, `linguist-generated` in `.gitattributes`, "generated" headers); +- any fix commit of any advisory of the repository, and any commit touching a known vulnerable function of it; +- commits over 1,000 changed source lines (counted, reported). + +Twelve are drawn per repository with a seeded random draw. Strata are three time thirds of the repository's +history crossed with four size buckets of changed source lines (1-10, 11-50, 51-200, 201-1,000). Allocation is +proportional to the repository's own commits per cell, with at least one per time third. Each change records base +(its first parent), head, changed files and line ranges. Reason: proportional allocation keeps the false-alarm +rate a per-push rate as a user meets it. Rejected: `freeze.py`'s broad `SECURITY_WORDS`. It also drops commits +saying "query", "token", "session" or "valid", which removes the ordinary commits where false alarms are most +likely, and the rate would look better than it is. Rejected: docs-only commits. They are trivially quiet and +would dilute the rate. + +Label noise: an ordinary commit may itself introduce an unknown vulnerability, so a flag counted as a false alarm +may be true. The false-alarm rate stays as Req 3.2 defines it. Flagged ordinary changes are adjudicated beside it +(all if 30 or fewer, else every third) against protocol-v2's standard, and the adjudicated share is reported. + +### 3. Sizing and slices (Req 2.3, 5.2) + +Wilson 95% interval, z = 1.96: centre (p + z²/2n)/(1 + z²/n), half width z·sqrt(p(1-p)/n + z²/4n²)/(1 + z²/n). +Taking the larger distance from p as the width, the smallest n that meets the bar, computed exactly: + +| Rate | Assumed p | Bar | n needed | +| --- | --- | --- | --- | +| false alarm | 0.05 | +/- 3 points | 315 | +| false alarm | 0.10 | +/- 3 points | 483 | +| false alarm | 0.20 | +/- 3 points | 756 | +| catch | 0.10 | +/- 10 points | 62 | +| catch | 0.25 | +/- 10 points | 88 | +| catch | 0.40 | +/- 10 points | 97 | + +Catch: 100 vulnerability-introducing changes per slice, one per repository. At p = 0.4, Wilson gives +[0.309, 0.498], within 10 points. One per repository keeps them independent. + +False alarm: ordinary changes of one repository are correlated (noisy code flags often). Wilson assumes +independence, so the count is inflated by a design effect 1 + (m - 1)·ICC. With m = 12 per repository and an +assumed ICC of 0.05, that is 1.55. 100 repositories × 12 = 1,200 changes, effective n 774 ≥ 756. The plain Wilson +interval at p = 0.2, n = 1,200 is about +/- 2.3 points. A repository-cluster bootstrap interval is reported beside +it, and the ICC measured on slice 1 is recorded. If it exceeds 0.05, the next pool sizes up. A frozen slice is +never changed. + +**Decision 3.1: slices.** Pool p1 has three slices of 100 repositories each (100 + 1,200 changes per slice; 300 +repositories and 3,900 changes in all). Repositories are assigned by a seeded shuffle, stratified by ecosystem and +`post_cutoff`. Slice 1 serves the baseline and the G1 arms table. Slices 2 and 3 serve the first two adoption +decisions. When they are spent, pool p2 is drawn by the same code, and it excludes p1. Reason: each slice alone +meets the Req 2.3 widths, so any decision can stand on one slice. Rejected: one large pool scored whole. Every +decision would then see every repository, which Req 5.2 forbids. Rejected: ten small slices. None would meet the +widths. Per family, 100 changes split over seven families give intervals of about +/- 20 to 25 points. They are +reported, not sized (open question 3). + +Use ledger: `benchmarks/unseen/usage-p1.jsonl` is append-only, one row per use (slice, decision or experiment id, +date, result digest). A slice with a row whose purpose is `informed` cannot appear again as `qualifies` for that +change. A test checks this. + +### 4. Scoring rules (Req 3.1, 3.2, 3.3) + +Each replay uses the hook's defaults: `--deadline 30`, `--engine inline`, `--max-regions 500`, `--mode advisory`. +That is what a user's push gets. The text output is kept with every `Skipped:` line, and so is the artifact. + +**Decision 4.0: a registered long-deadline arm** (maintainer decision 2026-10-04, open question 5). Beside the +default replay, slice 1 is replayed once more with `--deadline 300` and every other flag unchanged. The arm is +registered (manifest committed, `unseen: {pool: p1, slice: 1}`) before its first replay. G1 then reports both +what the user sees (30 s) and what the engine could catch given time (300 s), so a miss caused by the deadline is +told apart from a miss by the engine. It is a measurement arm, not a proposed default: adopting a longer hook +deadline would be a separate decision under 7.2. + +**Decision 4.1: catch, per family.** A finding counts if its family is the change's family. Engine streams carry a +family; a quick-rule match maps its CWE through the family CWE table. Its head location must also be in the +vulnerable function or within 10 lines of the change's head-side hunks (protocol v1's window). + +| Family | Location rule | +| --- | --- | +| injection, path, deserialization, output_encoding, untrusted_destination | sink line in the vulnerable function, or the changed lines +/- 10 | +| access_control | the vulnerable function itself (the handler that lacks the check), or the changed lines +/- 10 | +| config_secrets | the changed file and lines +/- 10 (file scope; function ``) | + +All three streams count: admitted defects, advisory engine findings and quick-rule matches. Each catch also names +the stream that made it. Reason: the hook shows all three to the user. Rejected: admitted defects only. That +scores the blocking path, which G3 decides separately, and would hide what users read. + +**Decision 4.2: false alarm.** An ordinary change is a false alarm when any of the three streams holds at least one +finding, of any family, anywhere. The alerts-only rate (admitted defects) is reported beside it. + +**Decision 4.3: coverage, so zeros cannot hide.** Every change gets exactly one state: + +- `instrument_failure`: no record, a failed fetch, the head commit not verified, or zero bytes of the changed files + read. It is re-run once, then reported. It never counts as quiet or missed. +- `unanalysable`: no changed file is in a language the hook covers (the hook's `language_not_covered` notice). +- `quick_only`: the quick tier read the changed files, but no engine question completed on them. +- `engine_covered`: at least one engine question completed on a changed file. + +The task entry proves its input as AGENTS.md requires. It records files and bytes in the checkout, bytes of each +changed file, the JVM's wall time and the hook's exit code. A replay faster than a JVM can start is flagged +`suspect_fast` and goes to the failure path. Catch and false alarm are reported over all changes and over each +coverage state. + +### 5. The AX workload (Req 4) + +**Decision 5.1: the image.** `plane/Dockerfile.engine-task` gains a second stage, `FROM engine AS replay`, built +with `--target replay` and pinned by digest. It adds `git`, the package with its runtime dependencies and +`benchmarks/unseen/replay_entry.py`. Joern, php-cli, Java 21 and the frontend-retention patch come from the engine +stage, so both lanes run the same Joern. The import smoke runs `ousast pre-push --help`. The root image stays +`ghcr.io/norandom/ax-task-runner:v0.3.1`. Rejected: a separate Dockerfile. It would duplicate the Joern install, +and two Joern copies are two instruments that drift. Rejected: one stage for both. The engine image's isolated +venv without yaml and boto3 is proven, and replay dependencies would change its digest on every bump. + +**Decision 5.2: the task entry.** `replay_entry.py` is standard library only, like `engine_task_entry.py`, and +reuses its `transfer` (egress readiness wait, three attempts, no URL text in errors). + +1. GET `SPEC_URL` (presigned, `.json`): repository URL, base, head, change id, hook flags. +2. `git init`, then `git fetch --depth 1 origin ` and `git fetch --depth 1 origin ` over HTTPS + (GitHub serves commits by SHA). Verify both with `git cat-file -e`. The fetch is bounded at 180 s; the 500 MB + repository cap bounds its size. +3. Run `ousast pre-push --base --head --artifact /tmp/out/push.json` with defaults, the + backend heap at 1700 MB and the cluster JVM flags of `engine_trace_worker.py`. Capture stdout. +4. Pack `push.json`, `push.txt`, `engine.json` (the head and base scan records and witness rows from the artifact, + keyed by repository and revision, for engine-evidence arms) and `instrument.json` (section 4.3, image digest, + step timings). PUT to `RESULT_URL` as `.dat`. The PUT is completion. + +On any error it PUTs a failure record naming the step and HTTP status, as the engine entry does. Task names are +`ousast-engine-replay--`, the prefix the operator's egress watcher matches. No key or store +credential enters the manifest. Only the two presigned URLs do, and the spec object keeps repository names out of +AX status and cluster telemetry. + +**Decision 5.3: the generic dispatcher.** `benchmarks/ax/batch.py` takes over from `engine_trace_ax.AX` and +`engine_trace.run_plan`: + +- `Item`: id, input objects (name to bytes), the result object name, command, extra env, input digest. +- `Workload`: image digest, allowed URL env names, deadline, how to validate a received result. +- Lanes: `ax` (manifest, apply, poll the result object and AX phase, delete, clean objects), `kind` (the same + manifest against the kind AX), `docker` (the VM image, one container at a time). +- Scheduler: two AX lanes plus one VM lane by default. A sandbox death is retried once under a new name; a second + death or a JVM OOM goes to the VM fallback queue, which beats overflow on the VM lane. `progress.json` is written + only by the scheduler. A `STOP` file halts new launches. Resume skips items whose stored record has the same + input digest. Cleanup failure stops dispatching. + +The engine pass becomes a workload over it, with the same tests. Reason: the egress, secrecy and cleanup logic took +several fixes and must exist once. Rejected: a replay-specific copy of `engine_trace_ax.py`. Rejected: putting it in +`src/` now. G4's `ousast plane scan` may need it there, and it moves when G4 is planned. + +Concurrency is two AX tasks. Placement is random, and only two guarantees two per worker at most. Four would need +the operator to add placement (open question 2, resolved 2026-10-04: stays at two, no placement change +requested). Estimated wall time per slice: about 75 s per replay (fetch, 30 s hook, setup), 1,300 replays over +three lanes, about 9 hours. The long-deadline arm (4.0) is bounded at about 345 s per replay; at that bound 1,300 +replays over three lanes take about 42 hours, less where the hook finishes early. It runs with `STOP` and resume +across sessions. + +**Decision 5.4: model arms on the VM.** Arms that need DeepSeek (the decision engine, the combiner) read the stored +records on the VM. They build candidates from the head side's changed functions, score them, and take the change +score as the maximum. Each runs under its experiment manifest's `budget_usd` through the metered client, dry run +first, with responses cached. Nothing in the cluster calls a model. + +**kind (Req 4.5)** runs the identical manifest and image through the `kind` lane. Every new task is proven there +before kube-ax, as `plane-on-kubernetes` requires. + +### 6. Records and the guard (Req 1.2, 1.3, 5.1) + +`benchmarks/unseen/` holds `pool-p1.toml` (repositories, licences, slices, changes with base, head, method, +family, function, lines), `freeze-p1.json` (used-set digests and counts per source, guard result, label-check +counts, freeze digest = sha256 of the canonical manifest), `protocol-p1.md` (the scoring rules of section 4, +committed before the first replay) and `usage-p1.jsonl`. The `private/` directory is gitignored. + +The guard grows by one parametrised reservation. The existing sweep becomes a list of (reservation directory, +names): `benchmarks/independent/` for populations and `benchmarks/unseen/` for pools, including the private +manifest when present. It also sweeps `plane/`. A second test asserts that pool and population names are disjoint, +and prints only a count on failure. A third asserts that a frozen pool's digest matches `freeze-p1.json`, so the +pool is never edited. Raw per-change records stay outside the repository (`~/ousast-results/unseen/` and the +store). Committed result records under `benchmarks/measurements/` carry counts, rates, intervals and digests, +never names. + +### 7. Metrics and reporting (Req 3.3, 3.4, 5.3) + +`benchmarks/unseen/score.py` reports per family and overall: catch and false-alarm rate with Wilson 95% +intervals, the cluster bootstrap interval for false alarms, coverage counts per state, catch per method, +post-cutoff and per stream, and the alerts-only rate. + +**Decision 7.1: arms at fixed budgets.** An arm with a score is thresholded per budget (1%, 5%, 10%). The +threshold is the lowest score whose false-alarm rate on the slice's ordinary changes stays within the budget. +Catch at that threshold is reported with its Wilson interval and the realised false-alarm rate. That in-slice +threshold is labelled exploratory. For a qualification, the threshold is fixed beforehand on an earlier slice or +on held-out folds, and it is applied unchanged. A binary arm (today's hook) has one operating point, reported with +the budgets it meets. Rejected: AUC over changes (it averages operating points no user runs). + +**Decision 7.2: next to held-out folds.** Experiment manifests registered after this spec lands carry +`unseen: {pool, slice}`. `analyse` writes an `unseen` block into `result.json` next to the fold estimates: catch at +each budget for each arm, and the paired difference B - A with an exact McNemar test on the 100 changes. The +fold rule decides as today. The slice is recorded in the usage ledger. + +**Decision 7.3: adoption gate on the unseen slice** (amended 2026-10-04, maintainer decision; Req 5.4; open +question 4 resolved). Reporting is not enough. An arm B is adopted over the production arm A only if, on the +untouched slice that qualifies it, both hold: + +- catch at the 5% budget, with B's threshold fixed beforehand (7.1), is not significantly lower than A's: exact + two-sided McNemar on the paired vulnerability-introducing changes, alpha 0.05; +- for an arm with one operating point (a binary arm such as a hook setting), its false-alarm rate is not + significantly higher than A's: exact two-sided McNemar on the paired ordinary changes, alpha 0.05, with the + repository-cluster bootstrap interval of the difference reported beside it, since those changes are clustered. + +`analyse` writes `unseen.gate: pass | regression | not_run` into `result.json`. `not_run` (no slice, slice +already `informed` for this change, or more than 5% `instrument_failure`) blocks adoption like `regression`. The +fold rule and this gate must both pass. + +### 8. Increments + +1. **Used set and guard.** `eligibility.py`, the extended guard and the usage-ledger test. Exit: guard green on an + empty pool; a planted used repository, a renamed one and a fork are each rejected in tests. +2. **Sourcing and extraction.** Candidates, both change kinds and the label check, on the VM. Exit: a draft for 300 + repositories with method counts, the confirmed share and exclusion counts. +3. **Freeze p1.** Guard bisection for v3, then manifest, freeze record and protocol committed. Exit: freeze digest + committed and guard green. +4. **Dispatcher.** `benchmarks/ax/batch.py`, with the engine pass moved onto it. Exit: existing tests green and a + two-pin engine smoke on kube-ax matching its stored records. +5. **Replay task.** Image stage and entry, proven on kind, then kube-ax on five changes from this repository's own + history (never pool changes). Exit: instrument fields show bytes read and realistic JVM time, and quick-tier + output matches the VM lane. +6. **Scorer.** `score.py` on synthetic records, including every coverage state and an instrument failure. Exit: + unit tests green. +7. **Baseline.** Today's hook, from a frozen export of HEAD, on slice 1 (1,300 replays). Exit: the record committed + (counts and digests only) and a `baseline` row in the usage ledger. That is G0's exit. +8. **Arms on slice 1 (G1)** (added 2026-10-04 with decisions 4.0 and 7.3). The registered long-deadline arm, + the engine-evidence arm from the stored records, the model arms on the VM (5.4) under their budgets, and the + `unseen` block with the adoption gate in `analyse`. Exit: one committed table of arms on slice 1 at the 1/5/10% + budgets (counts, rates, intervals, digests), and `exploratory` rows in the usage ledger. + +## Error Handling + +A failed fetch, a missing commit or a hook crash gives a failure record, never an empty success. A dead sandbox is +retried once, then the VM runs it. A store or AX cleanup error stops the dispatcher. A slice with more than 5% +`instrument_failure` after re-runs is not scored; the cause is fixed first. + +## Testing Strategy + +Unit tests: used-set normalisation (case, `.git`, renames, forks), blame acceptance on a synthetic repository, +ordinary-commit exclusions, Wilson values against section 3, budget thresholds, coverage states, the task manifest +and dispatcher resume and fallback with a fake lane. Guard tests run in CI; cluster runs stay manual. + +## Requirement Traceability + +| Requirement | Design | +| --- | --- | +| 1.1 eligibility against every source, v3 via the guard | 1.2 | +| 1.2 frozen manifest, never edited | 6, increment 3 | +| 1.3 pool directory, guard extended | 6 | +| 1.4 licences, private manifest | 1.4 | +| 2.1 introducing changes, family, function, post-cutoff | 1.1, 1.3, 2.1 | +| 2.2 ordinary changes across time and size | 2.2 | +| 2.3 sizes for interval widths | 3 | +| 3.1 default replay, skipped checks kept | 4, 5.2 | +| 3.2 catch and false-alarm rules | 4.1, 4.2 | +| 3.3 per family, Wilson, coverage | 4.3, 7 | +| 3.4 arms at 1/5/10% budgets | 4.0, 7.1, increment 8 | +| 4.1 one change one Task, prefix, image by digest | 5.1, 5.2 | +| 4.2 HTTPS fetch, presigned record, no keys | 5.2 | +| 4.3 generic dispatcher, concurrency, retry, fallback, resume | 5.3 | +| 4.4 model arms on the VM within budgets | 5.4 | +| 4.5 kind runs the same tasks | 5.3, 5 (kind paragraph) | +| 5.1 baseline on slice 1, counts and digests | 6, increment 7 | +| 5.2 fresh slice per decision | 3.1, usage ledger | +| 5.3 experiments report unseen next to folds | 7.2 | +| 5.4 adoption needs no significant unseen regression | 7.3 | + +## Open Questions + +Maintainer decisions of 2026-10-04 resolve questions 1, 2, 4 and 5; question 3 stays open for G3. + +1. **Resolved 2026-10-04.** Licences: only permissively licensed repositories are published by name. Copyleft and + all other licences go to the gitignored private manifest, covered in the committed manifest by an opaque id and + a digest (Req 1.4 as written; decision 1.4). Population v2's OSI-approved bar is not used. +2. **Resolved 2026-10-04.** Concurrency stays at two AX lanes plus the VM lane. No operator placement change is + requested (5.3). +3. Per-family BLOCK (G3) needs about 100 changes per family. Pool slices together, or draw a family-enriched pool + for G3? +4. **Resolved 2026-10-04.** An adoption requires no significant regression on the unseen slice, not only a report + (Req 5.4, amended; decision 7.3). +5. **Resolved 2026-10-04.** G1 adds a registered long-deadline arm (`--deadline 300`) next to the default 30 s, so + the table shows both what the user sees and what the engine could catch (decision 4.0, increment 8). diff --git a/.kiro/specs/unseen-repo-evaluation/requirements.md b/.kiro/specs/unseen-repo-evaluation/requirements.md index 45f2427..4f9e326 100644 --- a/.kiro/specs/unseen-repo-evaluation/requirements.md +++ b/.kiro/specs/unseen-repo-evaluation/requirements.md @@ -101,3 +101,7 @@ worn out by tuning. slice, recorded with its freeze digest. 3. Every learned-decision-engine experiment from now on reports its unseen-repo score from this pool next to its held-out-fold score. +4. An arm is adopted only if, on the untouched slice that qualifies it, it shows no significant regression against + the current production arm: catch at the 5% false-alarm budget not significantly lower, and for an arm with one + operating point the false-alarm rate not significantly higher (exact paired tests, two-sided, alpha 0.05). + Reporting the unseen score alone is not enough to adopt. *(amended 2026-10-04, maintainer decision)* diff --git a/.kiro/specs/unseen-repo-evaluation/spec.json b/.kiro/specs/unseen-repo-evaluation/spec.json index a818711..9418b06 100644 --- a/.kiro/specs/unseen-repo-evaluation/spec.json +++ b/.kiro/specs/unseen-repo-evaluation/spec.json @@ -1,22 +1,22 @@ { "feature_name": "unseen-repo-evaluation", "created_at": "2026-10-04T18:00:00Z", - "updated_at": "2026-10-04T18:30:00Z", + "updated_at": "2026-10-04T20:00:00Z", "language": "en", - "phase": "requirements-approved", + "phase": "implementation", "approvals": { "requirements": { "generated": true, "approved": true }, "design": { - "generated": false, - "approved": false + "generated": true, + "approved": true }, "tasks": { - "generated": false, - "approved": false + "generated": true, + "approved": true } }, - "ready_for_implementation": false + "ready_for_implementation": true } diff --git a/.kiro/specs/unseen-repo-evaluation/tasks.md b/.kiro/specs/unseen-repo-evaluation/tasks.md new file mode 100644 index 0000000..124060e --- /dev/null +++ b/.kiro/specs/unseen-repo-evaluation/tasks.md @@ -0,0 +1,321 @@ +# Implementation Plan + +One task group per increment of `design.md` section 8, in that order. A group starts only after the previous +group's exit is recorded. Maintainer decisions of 2026-10-04 are folded in: a registered long-deadline arm +(`--deadline 300`) beside the hook's default 30 s (design 4.0, group 8); adoption needs no significant regression +on the unseen slice (Req 5.4 as amended, design 7.3, tasks 6.4 and 8.2); only permissively licensed repositories +are published by name, everything else goes to the gitignored private manifest under a digest (Req 1.4, design +1.4); concurrency stays at two AX lanes plus the VM lane (design 5.3). + +Execution rules: + +- **Who does what.** Codex implements every coding task offline: no network, no model, no store, no cluster. The + coordinator runs every networked step (GitHub and OSV API calls, git fetches and clones, the S3 store, image + pushes, kind and kube-ax). Those tasks are marked **(coordinator, network)**. Steps that need the coordinator but + no network (agent label reads, adjudication, the v3 guard bisection, commits) are marked **(coordinator)**. + Steps that spend model money are marked **paid: needs a budget go-ahead**. Each starts only after the maintainer + confirms a ceiling against the provider balance read just before it. +- **Done means.** A coding task is done when its tests pass in the host suite, `ruff check`, `ruff format --check` + and `mypy` are clean, and the reserved-repository guard is green. No test makes a network, model or cluster call. +- **Names stay in the pool directory.** Pool repository names and URLs appear only under `benchmarks/unseen/` + (copyleft and other non-permissive ones only under the gitignored `benchmarks/unseen/private/`). Raw per-change + records live in `~/ousast-results/unseen/` and the store, never in the repository. +- **Population v3 is never opened.** It is checked only through the guard test, run as a black box with its + output discarded. +- **Every group ends with a measured exit** and a record at + `benchmarks/measurements/-unseen-NN-/record.json`: commit, image digests where used, the instrument + checks (bytes read, JVM time, exit codes), counts and digests. No repository names. The guard sweeps these + records. +- **Instrument first** (AGENTS.md). A zero, an empty draft or a fast run is treated as an instrument failure until + bytes read and wall time prove otherwise. + +- [ ] 1. Used set and guard (design 1.2, 6; increment 1) +- [ ] 1.1 Used-set builder + - `benchmarks/unseen/eligibility.py`: normalises names (lower case, no `.git`, no trailing slash) and builds the + used set from every source of design table 1.2: pair catalogs, recipes and datasets under `benchmarks/pairs/` + plus the local pointer-pair manifests as `benchmarks/pairs/catalog_gen.py` reads them; `population-v1.toml` + and `population-v2.toml`; the advisory-fix batch catalogs; memory `example` rows (`repo` field) through an + injected `MemoryStore` (`FileStore` locally, the S3 store when the coordinator passes one); experiment units + under `plane/experiments/`; `benchmarks/measurements/` and `benchmarks/experiments/`; a text sweep for + `github.com//` over `benchmarks/`, `src/`, `tests/`, `plane/` and `.kiro/`. It never reads + population v3 and never prints a name. Output: counts per source and the sha256 of the sorted set. + - `tests/test_unseen_eligibility.py`: a synthetic tree with one repository per source, each found; case, `.git` + and trailing-slash variants collapse; stdout and the output JSON contain no name; a v3-shaped file planted in + the synthetic tree is not opened (open is patched to fail on it). + - _Requirements: 1.1_ +- [ ] 1.2 Canonical resolution, renames and forks + - `eligibility.py`: `resolve(candidate, api)` with an injected GitHub client returns the canonical `full_name`, + former names (redirects) and the fork network (`parent`, `source`). A candidate is ineligible when any of them + is in the used set; one candidate per fork network is kept. + - Tests with a fake client: a planted used repository, a renamed one and a fork of a used one are each + rejected; two forks of one network keep one; the rejection reason is a source label, not a name. + - _Requirements: 1.1_ +- [ ] 1.3 Guard extended to pools + - `tests/test_independent_population.py`: the sweep becomes a list of (reservation directory, names): + `benchmarks/independent/` for populations, `benchmarks/unseen/` for pools (public manifests, plus + `benchmarks/unseen/private/*.toml` when present). The sweep also covers `plane/`. New tests: pool and + population names are disjoint (failure prints a count only); a frozen pool's manifest digest equals its + `freeze-pN.json` (skipped while no pool is frozen). The existing v3 handling is unchanged. + - Each reservation is exempt only inside its own directory: population names (v3 included) are still swept in + `benchmarks/unseen/`, the gitignored `private/` files included. Task 3.1 depends on this. + - `.gitignore`: `benchmarks/unseen/private/`. + - Evidence: the guard is green with an empty `benchmarks/unseen/`, and red when a test fixture names a pool + repository in `benchmarks/measurements/` (shown once in the record). + - _Requirements: 1.2, 1.3, 1.4_ +- [ ] 1.4 Usage ledger + - `benchmarks/unseen/ledger.py`: append-only `usage-pN.jsonl`, one row per use (slice, decision or experiment + id, purpose `baseline | exploratory | informed | qualifies`, date, result digest, freeze digest). It refuses a + `qualifies` row for a change whose slice already has an `informed` row for it, and refuses to rewrite rows. + - `tests/test_unseen_ledger.py`: append, the refusal, a rewritten row detected by a digest chain. + - _Requirements: 5.2_ +- [ ] 1.5 Used-set measurement and record 01 **(coordinator, network)** + - Run `eligibility.py` against the real sources, the S3 memory store included (read only). + - Instrument: each source reports files read and bytes; a source with zero rows fails the run unless it is + known empty. + - `benchmarks/measurements/-unseen-01-used-set/record.json`: counts per source, used-set digest, guard + result (green on the empty pool), test counts of 1.1-1.4. + - _Requirements: 1.1, 1.3_ + +- [ ] 2. Sourcing and extraction (design 1.1, 1.3, 1.4, 2, 3; increment 2) +- [ ] 2.1 Advisory candidates and repository filters + - `benchmarks/unseen/source.py`: from GHSA advisory JSON (pip, npm, maven, composer; published 2019 or later) + keep those whose CWE maps to a family through `src/openultrasast/model/taxonomy.py` and that link one fix + commit with one parent in one repository; attach the OSV first affected version; flag `post_cutoff` when + published after 2026-06-01. Repository filters: anonymous HTTPS, at most 500 MB, licence at the pinned head + (LICENSE file, then the API's SPDX id) classed `permissive` (the design 1.4 list) or `private`. + - `tests/test_unseen_source.py` on recorded fixture JSON (no network): CWE mapping, multi-parent and + multi-repository fixes dropped, cutoff flag, licence classing (GPL, AGPL, LGPL, none and source-available go + private). + - _Requirements: 1.4, 2.1_ +- [ ] 2.2 Vulnerability-introducing range + - `benchmarks/unseen/extract.py`: the `introducing` method of design 2.1 (blame of the sink line, or the + declaration for an absence bug; absence at V's first parent; no merge or root; at most 50 files and 2,000 + lines; OSV version bracket when present; walk-back of at most five steps) and the `last_touch` fallback. Each + change records method, family, vulnerable function resolved in head, head-side lines, `post_cutoff`, advisory + id. Git only; no scanner or engine. + - `tests/test_unseen_extract.py` on a synthetic repository built in a temp dir: accepted V; a reformat walked + back; a merge rejected; size cap; bracket rejection; fallback chosen when no V passes. + - _Requirements: 2.1_ +- [ ] 2.3 Ordinary-commit draw + - `extract.py`: first-parent, non-merge commits touching an analysed language; the exclusions of design 2.2 + (security words of the narrow pattern, reverts, bot and bump commits, manifest- or lock-only, generated or + vendored, advisory fix commits, commits touching a known vulnerable function, more than 1,000 source lines, + counted). Seeded draw of 12 per repository over three time thirds by four size buckets, proportional, at least + one per third; a repository with fewer than 40 eligible commits is dropped. + - Tests: each exclusion; the narrow pattern keeps "query", "token", "session"; allocation and the seed + reproduce the same draw; the 40-commit floor. + - _Requirements: 2.2, 2.3_ +- [ ] 2.4 Draft manifest and slices + - `benchmarks/unseen/draft.py`: writes `benchmarks/unseen/draft-p1.toml` (permissive entries by name) and + `benchmarks/unseen/private/draft-p1.toml` (the rest); the public file holds an opaque id and + sha256(URL, commits) per private entry. Seeded shuffle into three slices of 100 repositories, stratified by + ecosystem and `post_cutoff`; each repository has one introducing change and 12 ordinary changes. + - Tests: the digest covers private entries; no private URL in the public file; the same seed gives the same + slices; per-slice strata within one repository of proportional. + - _Requirements: 1.2, 1.4, 2.3, 5.2_ +- [ ] 2.5 Sourcing run **(coordinator, network)** + - `gh api /advisories` enumeration, OSV lookups, canonical resolution against the used set (1.2), blobless + clones into a scratch cache, extraction (2.2, 2.3), draft (2.4). + - Instrument: clones counted with bytes on disk; a repository whose clone reads zero source files is an + instrument failure, not a drop. Fewer than 300 eligible repositories stops the run with counts; the + eligibility bars are not relaxed to fill it. + - Observable: a draft of 300 repositories (3,900 changes) on disk. + - _Requirements: 1.1, 1.4, 2.1, 2.2, 2.3_ +- [ ] 2.6 Label check and record 02 **(coordinator)** + - Every third `introducing` change read by an agent, no model call, marked `confirmed | refactor | unclear` + in the draft. + - `benchmarks/measurements/-unseen-02-draft/record.json`: candidates per ecosystem, rejections per used + source, licence split (public, private), method counts, post-cutoff counts, confirmed share, ordinary-commit + exclusion counts. No names. + - _Requirements: 2.1, 2.2_ + +- [ ] 3. Freeze p1 (design 1.2, 6; increment 3) +- [ ] 3.1 Guard bisection tool + - `benchmarks/unseen/bisect_guard.py`: rewrites the draft with a candidate subset, runs + `.venv/bin/pytest -q -p no:cacheprovider tests/test_independent_population.py -k referenced` with stdout and + stderr discarded, reads only the exit code, and bisects until green. Reports the dropped count only. + - `tests/test_unseen_bisect.py` with an injected fake guard (a hidden set): finds and drops exactly the hidden + members; never logs a candidate; an exit code other than 0 or 1 aborts (instrument failure). + - _Requirements: 1.1_ +- [ ] 3.2 Freeze writer + - `benchmarks/unseen/freeze.py`: canonical `pool-p1.toml` (and the private counterpart) from the bisected draft; + `freeze-p1.json` with used-set digest and per-source counts, guard result, dropped count, label-check counts + and the freeze digest (sha256 of the canonical manifest). Refuses to write over a frozen pool; a correction + is `p2`. + - Tests: digest stable under key order; overwrite refused; the guard's digest test (1.3) passes on a fixture + pool. + - _Requirements: 1.2_ +- [ ] 3.3 Protocol p1 + - `benchmarks/unseen/protocol-p1.md`: the scoring rules of design section 4 (default flags, decision 4.0's + long-deadline arm, catch and location rules, false alarm, coverage states), the statistics of section 3 and + 7, and the adoption gate of 7.3, stated before any replay. + - Observable: the file exists and its sha256 is listed in `freeze-p1.json`. + - _Requirements: 3.1, 3.2, 3.3, 3.4, 5.4_ +- [ ] 3.4 Freeze run and record 03 **(coordinator)** + - Run the bisection (3.1), then the freeze (3.2). Guard green, digest test active. + - `benchmarks/measurements/-unseen-03-freeze/record.json`: freeze digest, protocol digest, dropped + count, final counts per slice and kind. + - _Requirements: 1.1, 1.2, 1.3_ + +- [ ] 4. Dispatcher (design 5.3; increment 4) +- [ ] 4.1 Items, workloads and the AX lane + - `benchmarks/ax/batch.py`: `Item` (id, input objects, result object name, command, extra env, input digest), + `Workload` (image digest, allowed URL env names, deadline, result validator) and the `ax` lane, moved from + `benchmarks/learn/engine_trace_ax.py`: manifest render, egress readiness, presigned GET and PUT as `.dat`, + apply, poll result object and AX phase, delete, object cleanup. The `kind` lane is the same lane with the + kind context. + - `tests/test_ax_batch.py` with a fake `ax` CLI and a fake store: manifest holds only the allowed URL env + names and no credential; names carry the `ousast-engine-` prefix; cleanup on success and failure. + - _Requirements: 4.1, 4.2, 4.5_ +- [ ] 4.2 Scheduler, fallback and resume + - `batch.py`: two AX lanes and one VM `docker` lane by default (one container at a time); sandbox death retried + once under a new name, a second death or a JVM OOM goes to the VM fallback queue ahead of overflow; + `progress.json` written by the scheduler only; a `STOP` file halts launches; resume skips items whose stored + record has the same input digest; a cleanup failure stops dispatching. + - Tests with fake lanes: retry then fallback; OOM straight to fallback; resume skips; `STOP`; cleanup failure + halts; never more than two AX tasks in flight. + - _Requirements: 4.3_ +- [ ] 4.3 Engine pass on the dispatcher + - `benchmarks/learn/engine_trace_ax.py` and `engine_trace.run_plan` become an engine `Workload` over `batch.py`; + no behaviour change. + - Observable: `tests/test_ax_executor.py` and `tests/test_engine_trace.py` pass with their assertions unchanged + (moved imports only). + - _Requirements: 4.3_ +- [ ] 4.4 Engine smoke on kube-ax and record 04 **(coordinator, network)** + - Two pins through the new dispatcher on kube-ax, compared with their stored records. + - `benchmarks/measurements/-unseen-04-dispatcher/record.json`: image digest, per-pin wall time and JVM + time, record digests equal to the stored ones (or the diff counted), cleanup result. + - _Requirements: 4.3_ + +- [ ] 5. Replay task (design 5.1, 5.2; increment 5) +- [ ] 5.1 Image stage + - `plane/Dockerfile.engine-task`: second stage `FROM engine AS replay` adding `git`, the package with its + runtime dependencies and `benchmarks/unseen/replay_entry.py`; smoke `ousast pre-push --help`. The root image + stays `ghcr.io/norandom/ax-task-runner:v0.3.1`; the engine stage is unchanged. + - `tests/test_unseen_image.py`: parses the Dockerfile; the `replay` stage builds from `engine`; the root image + pin; the engine stage's text is byte-identical to before. + - _Requirements: 4.1_ +- [ ] 5.2 Task entry + - `benchmarks/unseen/replay_entry.py`, standard library only, reusing `engine_task_entry.transfer`: GET the spec; + `git init` and depth-1 fetches of head and base over HTTPS (180 s bound), verified with `git cat-file -e`; run + `ousast pre-push` with the spec's hook flags (defaults, or `--deadline 300` for the long-deadline arm), heap + 1700 MB and the cluster JVM flags; pack `push.json`, `push.txt`, `engine.json`, `instrument.json` (files and + bytes in the checkout, bytes of each changed file, JVM wall time, hook exit code, step timings, image + digest); PUT as `.dat`. Any error PUTs a failure record naming the step and status, never URL text. + - `tests/test_unseen_replay_entry.py`: a local bare repository and a localhost presign stub; success record + fields; a missing commit, a failed fetch and a hook crash each give a failure record; zero bytes read and a + replay faster than JVM start are flagged `suspect_fast`. + - _Requirements: 3.1, 4.1, 4.2_ +- [ ] 5.3 Replay workload + - `benchmarks/unseen/replay.py`: builds `Item`s from a frozen slice and an arm (`default` or `long_deadline`); + the spec object carries the repository URL so names stay out of AX status and telemetry; task names + `ousast-engine-replay--` with an opaque slug; the validator rejects a record without + `instrument.json`; records go to `~/ousast-results/unseen////` and the store. + - Tests: no repository name in any task name or manifest; only the two presigned URLs in the env; input + digest changes with the arm's flags; `instrument_failure` items queued for one re-run. + - _Requirements: 3.1, 4.1, 4.2, 4.3_ +- [ ] 5.4 Proof on kind, then kube-ax, and record 05 **(coordinator, network)** + - Build and push the `replay` target, pinned by digest. Five changes from this repository's own history (never + pool changes) on kind, then the same five on kube-ax, then on the VM lane. + - Exit: instrument fields show bytes read and realistic JVM time on every lane; quick-tier findings equal + across lanes. + - `benchmarks/measurements/-unseen-05-replay-task/record.json`: image digest, per-lane timings, bytes, + exit codes, the quick-tier comparison. + - _Requirements: 4.1, 4.2, 4.5_ + +- [ ] 6. Scorer (design 3, 4, 7.1, 7.3; increment 6) +- [ ] 6.1 Coverage states + - `benchmarks/unseen/score.py`: each record gets one state (`instrument_failure`, `unanalysable`, `quick_only`, + `engine_covered`) from `instrument.json` and the artifact; `suspect_fast` goes to failure; a slice above 5% + `instrument_failure` after re-runs is refused. + - `tests/test_unseen_score.py` on synthetic records: every state; the refusal; a failure never counts as quiet + or missed. + - _Requirements: 3.3_ +- [ ] 6.2 Catch and false-alarm rules + - `score.py`: family match (engine family, quick-rule CWE through `taxonomy.py`), the location table of design + 4.1, all three streams with the stream named per catch; false alarm on any finding of any stream; the + alerts-only rate. + - Tests: right family wrong place misses; `access_control` on the handler; `config_secrets` file scope; a + quick-rule-only catch; one advisory finding on an ordinary change is a false alarm. + - _Requirements: 3.2_ +- [ ] 6.3 Rates, intervals and breakdowns + - `score.py`: Wilson 95% intervals; repository-cluster bootstrap for false alarms; ICC; breakdowns per family, + method, `post_cutoff`, stream, coverage state and the confirmed subset. + - Tests: the n values of design section 3 (315, 483, 756, 62, 88, 97) reproduced; bootstrap seeded; ICC on a + synthetic clustered set. + - _Requirements: 2.3, 3.3_ +- [ ] 6.4 Budgets, paired tests, the gate and record 06 + - `score.py`: per-budget thresholds (1%, 5%, 10%), labelled `exploratory` in-slice and `fixed` when supplied; + a binary arm's single operating point with the budgets it meets; exact two-sided McNemar for catch and for + false alarm; `gate(A, B)` returning `pass | regression | not_run` per design 7.3. + - Tests: thresholds on a synthetic score set; McNemar exact values; a binary arm with a significant + false-alarm increase gives `regression`; missing slice or excess failures give `not_run`. + - `benchmarks/measurements/-unseen-06-scorer/record.json`: test counts and the digest of the scorer's + output on the committed synthetic fixture. + - _Requirements: 3.4, 5.4_ + +- [ ] 7. Baseline (design 4, 6; increment 7; G0 exit) +- [ ] 7.1 Frozen source and dry run **(coordinator, network)** + - Export HEAD to a frozen tree; build and push the `replay` image from it, pinned by digest. + - `benchmarks/push/python_overhead.py` run with the stub engine to size per-replay Python cost; dispatcher dry + run on slice 1 lists 1,300 items without launching. + - _Requirements: 5.1, 4.1_ +- [ ] 7.2 Baseline replay of slice 1 **(coordinator, network)** + - Default arm, 1,300 replays over two AX lanes and the VM lane (about 9 hours); `instrument_failure` items + re-run once; the run stops for a cause above 5% failures. + - Observable: 1,300 records in `~/ousast-results/unseen/p1/1/default/` and the store, each with + `instrument.json`. + - _Requirements: 3.1, 4.3, 5.1_ +- [ ] 7.3 Adjudication **(coordinator)** + - Flagged ordinary changes adjudicated against protocol-v2's standard (all if 30 or fewer, else every third); + the adjudicated share kept beside the false-alarm rate. ICC measured; above 0.05 noted as "next pool sizes + up". + - _Requirements: 3.2, 2.3_ +- [ ] 7.4 Record 07 and the ledger **(coordinator)** + - `benchmarks/measurements/-unseen-07-baseline/record.json`: `score.py` output (counts, rates, Wilson and + bootstrap intervals, coverage per state, per family, method, stream), freeze digest, image digest, commit. + `usage-p1.jsonl` row `baseline`. + - Observable: guard green over the record; G0's exit met. + - _Requirements: 5.1, 5.2, 3.3_ + +- [ ] 8. Arms on slice 1 (design 4.0, 5.4, 7.1-7.3; increment 8; G1) +- [ ] 8.1 Registered long-deadline arm + - `plane/experiments/` manifest for the long-deadline arm: hook flags with `--deadline 300`, budget 0, + `unseen: {pool: p1, slice: 1}`, committed before its first replay. `src/openultrasast/learn/experiments.py` + accepts `unseen` and a hook-flag arm. + - `tests/test_learn_experiments.py`: the manifest loads; an `unseen` block without pool or slice is refused. + - _Requirements: 3.4, 5.3_ +- [ ] 8.2 `unseen` block and adoption gate in `analyse` + - `experiments.py` `analyse`: writes `unseen` into `result.json` (catch at each budget per arm, paired B - A, + McNemar, `gate`) from `score.py`; checks the usage ledger; adoption requires the fold rule and + `gate == pass`. + - Tests: `regression` and `not_run` each block adoption; an `informed` slice cannot qualify; result JSON shape. + - _Requirements: 5.2, 5.3, 5.4_ +- [ ] 8.3 Engine-evidence arm + - `benchmarks/unseen/arms.py`: a score per change from `engine.json` (witness rows on head-side changed + functions), no model; plugs into the budget thresholds of 6.4. + - Tests on synthetic records: score monotone in witness strength; a change without engine coverage gets no + score and is reported, not counted quiet. + - _Requirements: 3.4_ +- [ ] 8.4 Model-arm adapter + - `arms.py`: candidates from head-side changed functions of stored records, scored by the current decision + engine program (and the combiner when its program exists; else its row reads `not_built`); change score is the + maximum; metered client, dry run first, response cache, the manifest's `budget_usd` as the hard ceiling. + - Tests with a scripted client: dry-run estimate, ceiling stop, cache replay; no model call. + - _Requirements: 4.4, 3.4_ +- [ ] 8.5 Long-deadline replay of slice 1 **(coordinator, network)** + - 1,300 replays of the `long_deadline` arm, bounded at about 345 s each (up to about 42 hours over three + lanes), with `STOP` and resume across sessions; failures re-run once. + - Observable: 1,300 records under `~/ousast-results/unseen/p1/1/long_deadline/`, JVM time in each. + - _Requirements: 3.1, 3.4, 4.3_ +- [ ] 8.6 Model arms on slice 1 **(coordinator, network; paid: needs a budget go-ahead)** + - Dry run gives the candidate count; estimate at the injection slice's measured cost (about $0.008 per + candidate at k = 5); the maintainer sets the ceiling against the balance read just before. Then the decision + engine arm (and the combiner if built) on the VM. + - _Requirements: 4.4, 3.4_ +- [ ] 8.7 Record 08: the arms table **(coordinator)** + - `benchmarks/measurements/-unseen-08-arms/record.json`: per arm (hook 30 s, hook 300 s, engine evidence, + decision engine, combiner or `not_built`) catch at 1/5/10% budgets with Wilson intervals and realised + false-alarm rates, coverage per state, spend; `exploratory` rows in `usage-p1.jsonl`. + - Observable: one committed table of arms on slice 1; G1's table exists. + - _Requirements: 3.4, 5.2, 5.3_