From d03590148f4392aa62a7206214835ac47d17e400 Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Sun, 11 Oct 2026 08:17:06 +0200 Subject: [PATCH] docs(g0): record the 2026-10-11 state and readiness plan; bucket-first + image provenance Fold the night's learnings into the roadmap, steering and the unseen-repo-evaluation spec so the process is reproducible and the project is "ready now": - roadmap.md: a 2026-10-11 state table (pool p1 frozen; harvest hardened + bucket-first; replay-image provenance pinned; baseline-on-kube next) and a 5-step readiness plan to record 07. - storage-streaming.md: RustFS-over-local as the default (local disk is the scarce resource); the implemented ousast-corpus layout; replay outputs belong on RustFS; the AX store endpoint is operator-owned. - design.md: an "Amendments (2026-10-11)" section (bucket-first persistence, ~2.2% rate-bound yield, machine-label record 07, pinned replay image provenance). - tasks.md: a folded-in amendment block (pool p1 frozen + freeze recipe; bucket-first; CI replay image; machine-label baseline; kube lane + ax/store as operator-owned infra). Facts stated are verified (measured yield, freeze digest, GHCR anon pull, batch.py's endpoint check); infra capability/policy is left to the operator, not assumed. The frozen pool artifacts themselves are not committed here (a separate samples/licensing decision). Co-Authored-By: Claude Opus 4.8 --- .kiro/specs/unseen-repo-evaluation/design.md | 30 ++++++++++++++++++ .kiro/specs/unseen-repo-evaluation/tasks.md | 18 +++++++++++ .kiro/steering/roadmap.md | 30 ++++++++++++++++++ .kiro/steering/storage-streaming.md | 33 +++++++++++++++++--- 4 files changed, 107 insertions(+), 4 deletions(-) diff --git a/.kiro/specs/unseen-repo-evaluation/design.md b/.kiro/specs/unseen-repo-evaluation/design.md index 76db1b3..80b9c33 100644 --- a/.kiro/specs/unseen-repo-evaluation/design.md +++ b/.kiro/specs/unseen-repo-evaluation/design.md @@ -430,6 +430,36 @@ and dispatcher resume and fallback with a fake lane. Guard tests run in CI; clus | 5.3 experiments report unseen next to folds | 7.2 | | 5.4 adoption needs no significant unseen regression | 7.3 | +## Amendments (2026-10-11) + +Folded in after the G0 pool was frozen; the original sections above stay as the design of record, these refine it +where reality differed. + +- **Persistence is bucket-first, not local disk.** The draw streams every stage to the `ousast-corpus` RustFS + bucket and resumes from it, so a restart (even after a code change) is cheap. Layout: `unseen//decisions/` + (accept/reject per repo, key `sha256(repo+fix)`, idempotent), `unseen//resolved/` (per-advisory + resolution), `unseen//advisories/list.json`, `unseen/repos/` (get_repo cache, **not** seed-scoped, shared + across seeds), and `unseen//pool/` (the frozen pool + provenance — the "don't redo sourcing" archive). The + local journal is a within-process cursor only. Implementation: `benchmarks/unseen/bank.py` (`CorpusBank`). The + VM's local disk is the scarce resource; run artifacts and replay outputs belong on RustFS (see steering + `storage-streaming.md`). Credentials: `.env` `CORPUS_S3_*`; boto3 needs path-style + SigV4. +- **Harvest cost is GitHub-rate-limit bound, measured yield ~2.2%.** Through the strict change criteria (1 vuln + + 12 stratified ordinary, >=40 eligible commits) ~2.2% of candidates are accepted, so "100 repos" is ~4,500 + candidate attempts at ~4 GitHub calls each — the bottleneck is the 5,000/hr API limit, not CPU or cores. Pure + GitHub-API harvesting cannot replace the clone: the positive is the SZZ introducing commit and the negatives are + stratified over full history, both needing repo history. The convergence fix was a full-objects clone (dropping + `--filter=blob:none`, which caused ~600 per-blob lazy fetches) plus the bucket caches above. +- **Record 07 is a machine-label baseline.** The pool's 100 vulnerability labels are SZZ-derived and unchecked + (`freeze-p1.json` `label_check_counts`: 100 unchecked). The G0 baseline is reported as a machine-label number + with that caveat; a human label pass is a later quality refinement, required before any published precision + claim, not before record 07. +- **Replay image provenance is pinned.** CI `engine-image.yml` builds `ghcr.io/norandom/ousast-replay-task` (the + replay harness, which `COPY`s `/app/benchmarks/unseen/replay_entry.py`) and `ousast-engine-task` (the Joern + hook), publicly pullable by digest. `ousast-runner` (the 248 MB ax-task-runner, no `/app/benchmarks`, no Joern) + is **not** the replay image. The AX lane's store check (`benchmarks/ax/batch.py`) requires a public-HTTPS DNS + presign-capable endpoint; how that and the cluster's store access are provided is operator-owned infra, not + assumed here. + ## Open Questions Maintainer decisions of 2026-10-04 resolve questions 1, 2, 4 and 5; question 3 stays open for G3. diff --git a/.kiro/specs/unseen-repo-evaluation/tasks.md b/.kiro/specs/unseen-repo-evaluation/tasks.md index 3ddd7a7..9aa822f 100644 --- a/.kiro/specs/unseen-repo-evaluation/tasks.md +++ b/.kiro/specs/unseen-repo-evaluation/tasks.md @@ -7,6 +7,24 @@ on the unseen slice (Req 5.4 as amended, design 7.3, tasks 6.4 and 8.2); only pe are published by name, everything else goes to the gitignored private manifest under a digest (Req 1.4, design 1.4); concurrency stays at two AX lanes plus the VM lane (design 5.3). +**2026-10-11 amendments folded in** (see design.md "Amendments (2026-10-11)"): + +- **Pool p1 is frozen.** `benchmarks/unseen/pool-p1.toml` = 100 repos, slice 1; `freeze-p1.json` guard=0, + dropped=0, deterministic (manifest `6bd2fa5d`); archived to `ousast-corpus/unseen/20260601/pool/`. Groups 1-3 are + effectively done (the draw, draft and freeze ran end to end); formal records 02/03 were not written as separate + `record.json` files — the freeze was run directly (`bisect_guard` then `freeze`). Freeze recipe: `bisect_guard + --root $PWD --draft $CACHE/journal/draft-p1.toml` then `freeze --root $PWD --used-report ` where the file + is the draw report's `used_set` object (not the whole report). +- **Persistence is bucket-first** (`ousast-corpus`, `benchmarks/unseen/bank.py`); the local journal is a + within-process cursor only. Raw per-change replay records go to **RustFS**, not `~/ousast-results/` on local disk + — amend tasks 5.3/7.2's `--out` accordingly. +- **Group 5 image note:** the replay image is the CI-built `ghcr.io/norandom/ousast-replay-task` (+ engine-task), + publicly pullable by digest — not a local export-HEAD build and not `ousast-runner`. Task 7.1's "build and push + the replay image" is satisfied by the CI `engine-image.yml` run; use that digest. +- **Record 07 is a machine-label baseline** (labels SZZ-derived, 100 unchecked); state the caveat in the record. +- **Baseline lane (7.2):** run on kube (`--lane ax`). The `ax` binary and the AX-lane store endpoint are + operator-owned infra to stand up before 7.2; do not assume the cluster/network state. + Execution rules: - **Who does what.** Codex implements every coding task offline: no network, no model, no store, no cluster. The diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index 3780862..4fd7c13 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -41,6 +41,36 @@ Correction recorded 2026-10-04: the previous two days went mostly into infrastru not move a generalisation number. From here, infrastructure and engine work is done only when it removes a bottleneck for the measurements below. +### Where we are (2026-10-11) + +The G0 measurement is now runnable: the unseen pool is **frozen** and the whole pipeline is hardened and +**bucket-first**. The baseline replay is the one remaining step, to run on kube. + +| Area | State | Evidence | +| --- | --- | --- | +| Unseen pool (G0.1) | **Frozen.** `pool-p1.toml` = 100 repos (slice 1); `freeze-p1.json` guard=0, dropped=0; deterministic (manifest `6bd2fa5d`). License 49 permissive / 51 private; methods 81 introducing / 19 last_touch. | `benchmarks/unseen/pool-p1.toml`; archived `ousast-corpus/unseen/20260601/pool/` | +| Labels | 100 machine-derived (SZZ), **unchecked**. Record 07 is a **machine-label baseline** (caveat stated); a human label pass is a later refinement, before any published claim. | `freeze-p1.json` label_check_counts | +| Harvest | Rebuilt to converge. Pure GitHub-API harvesting is a dead end for this benchmark (it needs the SZZ introducing commit + 12 stratified history negatives, not one commit). The fix: a full-objects clone (drop `--filter=blob:none`, which caused ~600 per-blob lazy fetches) + bucket-first durability + a get_repo cache that paid the 4,499-name used-set canonicalization once. Yield ~2.2%; **GitHub-rate-limit bound, not CPU**. | 95bed70, 857263d, 21aebe8, 7b711a6 | +| Persistence | **Bucket-first, not local disk.** `ousast-corpus` is the source of truth: enumeration, resolution, get_repo, and accept/reject decisions all stream to and resume from it. The host disk (~9-13 GB free) is the scarce resource; RustFS/S3 has the room, so run artifacts and **replay outputs go to RustFS too**, not `~/ousast-results` on local disk. | `storage-streaming.md`, `bank.py` | +| Replay image | Provenance pinned. CI `engine-image.yml` builds `ousast-replay-task` (harness with `/app/benchmarks/unseen/replay_entry.py`) + `ousast-engine-task` (Joern hook), publicly pullable by digest. The old `ousast-runner:dev` (248 MB ax-runner, no `/app/benchmarks`, no Joern) is the **wrong** image for replay. | CI run 38116281676 | +| Baseline (G0.2) | **Not yet.** Replay reads the frozen pool cleanly (dry-run = 1,300 changes, `freeze_digest` verified). Remaining: mirror CI's passing `replay` job wiring, and stand up the kube lane (`ax` binary + a public-HTTPS presign-capable store endpoint; GHCR pull already public). | CI `replay` job (passed) | + +Lesson, recorded so it is not re-paid: sourcing and engine-image work cost real time. The durable takeaway now lives +in the pipeline and these docs — bucket-first persistence, pinned image provenance, a measured yield — so the next +pool or arm is cheap, not a rediscovery. + +### Readiness plan (to G0 exit = record 07) + +1. **Kube replay lane (infra, operator-owned):** install the `ax` binary; provide the store endpoint the AX lane + needs. The only thing settled in-repo is a code fact: `benchmarks/ax/batch.py` currently expects a public-HTTPS + DNS, presign-capable S3 endpoint. How that endpoint and the cluster's access to the store are provided is an + operator/AX decision, not an assumption to make here. GHCR pull of the images is verified public. +2. **Unblock the replay** by mirroring CI's passing `replay` job (replay-task + engine-task wired, store + network). +3. **Run slice-1 baseline on kube:** `--lane ax --arm default`, 1,300 changes, no model cost, resumable; outputs to + RustFS. +4. **Adjudicate (7.3) -> record 07 (7.4)** = the measured machine-label unseen baseline = **G0 exit**. +5. Then **G1 arms** (DeepSeek decision engine) on the frozen pool. + ### Phases **G0. The unseen-repository pool and the push-level metric** (new spec `unseen-repo-evaluation`) diff --git a/.kiro/steering/storage-streaming.md b/.kiro/steering/storage-streaming.md index 60ec6ca..8305122 100644 --- a/.kiro/steering/storage-streaming.md +++ b/.kiro/steering/storage-streaming.md @@ -48,8 +48,33 @@ buckets rather than inventing a new transport. New buckets and their grants are admin** action — the store credentials cannot create buckets or set lifecycle (confirmed by the `AccessDenied` on the 2026-10-09 lifecycle apply). -## Status +## Default: RustFS over local disk -Direction recorded; planned as a follow-up spec, not retrofitted into an in-flight run. The current -unseen draw finishes to its local journal and then archives its result to `sast-pools` once that -bucket exists. +Local disk on the VM is the scarce resource (it runs near full; the draw stops itself under 2 GB free). +RustFS/S3 has far more room. So the default for any run's durable state and large artifacts is the bucket, +not local disk. Local files are a rebuildable scratch cache only (the live git clone, a within-process +journal). A long run must be resumable purely from the bucket, and its outputs — including replay result +records — are written to RustFS, not `~/ousast-results` on local disk. + +## Status (implemented 2026-10-10/11) + +Bucket-first persistence is live, not a future spec. The unseen draw streams every stage to the +`ousast-corpus` bucket (`http://10.0.0.54:9001`, creds in `.env` as `CORPUS_S3_*`; boto3 needs path-style + +SigV4) and resumes from it: + +- `unseen//decisions/.json` — accept/reject per repository (idempotent; content dedup). +- `unseen//resolved/.json` — per-advisory resolution. +- `unseen//advisories/list.json` — the enumerated advisory list. +- `unseen/repos/.json` — get_repo cache, **not** seed-scoped (shared across seeds/runs), so the + thousands of used-set canonicalization lookups are paid once ever. +- `unseen//pool/` — the frozen pool and its provenance (`pool-p1.toml`, `private-pool-p1.toml`, + `freeze-p1.json`, `bisect-p1.json`, `protocol-p1.md`): the durable "don't redo the sourcing" archive. + +Implementation: `benchmarks/unseen/bank.py` (`CorpusBank`, `corpus_settings`). The old local journal is a +within-process cursor only; the bucket is the cross-run source of truth, so a restart (even after a code +change) resumes cheaply. Still to do and consistent with the default above: replay result records +(`replay.py --out`) currently default to `~/ousast-results/unseen/...` on local disk — point them at RustFS +for the slice-1 baseline and after. The AX (kube) lane already uses presigned object transfer, which is the +same principle. One in-repo code fact: `benchmarks/ax/batch.py` currently expects a public-HTTPS DNS, +presign-capable endpoint. How the store endpoint and the cluster's access to it are provided is operator/AX-owned +infra, not something this doc should presume.