mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-18 20:49:28 +00:00
docs(g0): record the 2026-10-11 state and readiness plan; bucket-first + image provenance
Fold the night's learnings into the roadmap, steering and the unseen-repo-evaluation spec so the process is reproducible and the project is "ready now": - roadmap.md: a 2026-10-11 state table (pool p1 frozen; harvest hardened + bucket-first; replay-image provenance pinned; baseline-on-kube next) and a 5-step readiness plan to record 07. - storage-streaming.md: RustFS-over-local as the default (local disk is the scarce resource); the implemented ousast-corpus layout; replay outputs belong on RustFS; the AX store endpoint is operator-owned. - design.md: an "Amendments (2026-10-11)" section (bucket-first persistence, ~2.2% rate-bound yield, machine-label record 07, pinned replay image provenance). - tasks.md: a folded-in amendment block (pool p1 frozen + freeze recipe; bucket-first; CI replay image; machine-label baseline; kube lane + ax/store as operator-owned infra). Facts stated are verified (measured yield, freeze digest, GHCR anon pull, batch.py's endpoint check); infra capability/policy is left to the operator, not assumed. The frozen pool artifacts themselves are not committed here (a separate samples/licensing decision). Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
7b711a6150
commit
d03590148f
@@ -430,6 +430,36 @@ and dispatcher resume and fallback with a fake lane. Guard tests run in CI; clus
|
||||
| 5.3 experiments report unseen next to folds | 7.2 |
|
||||
| 5.4 adoption needs no significant unseen regression | 7.3 |
|
||||
|
||||
## Amendments (2026-10-11)
|
||||
|
||||
Folded in after the G0 pool was frozen; the original sections above stay as the design of record, these refine it
|
||||
where reality differed.
|
||||
|
||||
- **Persistence is bucket-first, not local disk.** The draw streams every stage to the `ousast-corpus` RustFS
|
||||
bucket and resumes from it, so a restart (even after a code change) is cheap. Layout: `unseen/<seed>/decisions/`
|
||||
(accept/reject per repo, key `sha256(repo+fix)`, idempotent), `unseen/<seed>/resolved/` (per-advisory
|
||||
resolution), `unseen/<seed>/advisories/list.json`, `unseen/repos/` (get_repo cache, **not** seed-scoped, shared
|
||||
across seeds), and `unseen/<seed>/pool/` (the frozen pool + provenance — the "don't redo sourcing" archive). The
|
||||
local journal is a within-process cursor only. Implementation: `benchmarks/unseen/bank.py` (`CorpusBank`). The
|
||||
VM's local disk is the scarce resource; run artifacts and replay outputs belong on RustFS (see steering
|
||||
`storage-streaming.md`). Credentials: `.env` `CORPUS_S3_*`; boto3 needs path-style + SigV4.
|
||||
- **Harvest cost is GitHub-rate-limit bound, measured yield ~2.2%.** Through the strict change criteria (1 vuln +
|
||||
12 stratified ordinary, >=40 eligible commits) ~2.2% of candidates are accepted, so "100 repos" is ~4,500
|
||||
candidate attempts at ~4 GitHub calls each — the bottleneck is the 5,000/hr API limit, not CPU or cores. Pure
|
||||
GitHub-API harvesting cannot replace the clone: the positive is the SZZ introducing commit and the negatives are
|
||||
stratified over full history, both needing repo history. The convergence fix was a full-objects clone (dropping
|
||||
`--filter=blob:none`, which caused ~600 per-blob lazy fetches) plus the bucket caches above.
|
||||
- **Record 07 is a machine-label baseline.** The pool's 100 vulnerability labels are SZZ-derived and unchecked
|
||||
(`freeze-p1.json` `label_check_counts`: 100 unchecked). The G0 baseline is reported as a machine-label number
|
||||
with that caveat; a human label pass is a later quality refinement, required before any published precision
|
||||
claim, not before record 07.
|
||||
- **Replay image provenance is pinned.** CI `engine-image.yml` builds `ghcr.io/norandom/ousast-replay-task` (the
|
||||
replay harness, which `COPY`s `/app/benchmarks/unseen/replay_entry.py`) and `ousast-engine-task` (the Joern
|
||||
hook), publicly pullable by digest. `ousast-runner` (the 248 MB ax-task-runner, no `/app/benchmarks`, no Joern)
|
||||
is **not** the replay image. The AX lane's store check (`benchmarks/ax/batch.py`) requires a public-HTTPS DNS
|
||||
presign-capable endpoint; how that and the cluster's store access are provided is operator-owned infra, not
|
||||
assumed here.
|
||||
|
||||
## Open Questions
|
||||
|
||||
Maintainer decisions of 2026-10-04 resolve questions 1, 2, 4 and 5; question 3 stays open for G3.
|
||||
|
||||
@@ -7,6 +7,24 @@ on the unseen slice (Req 5.4 as amended, design 7.3, tasks 6.4 and 8.2); only pe
|
||||
are published by name, everything else goes to the gitignored private manifest under a digest (Req 1.4, design
|
||||
1.4); concurrency stays at two AX lanes plus the VM lane (design 5.3).
|
||||
|
||||
**2026-10-11 amendments folded in** (see design.md "Amendments (2026-10-11)"):
|
||||
|
||||
- **Pool p1 is frozen.** `benchmarks/unseen/pool-p1.toml` = 100 repos, slice 1; `freeze-p1.json` guard=0,
|
||||
dropped=0, deterministic (manifest `6bd2fa5d`); archived to `ousast-corpus/unseen/20260601/pool/`. Groups 1-3 are
|
||||
effectively done (the draw, draft and freeze ran end to end); formal records 02/03 were not written as separate
|
||||
`record.json` files — the freeze was run directly (`bisect_guard` then `freeze`). Freeze recipe: `bisect_guard
|
||||
--root $PWD --draft $CACHE/journal/draft-p1.toml` then `freeze --root $PWD --used-report <FILE>` where the file
|
||||
is the draw report's `used_set` object (not the whole report).
|
||||
- **Persistence is bucket-first** (`ousast-corpus`, `benchmarks/unseen/bank.py`); the local journal is a
|
||||
within-process cursor only. Raw per-change replay records go to **RustFS**, not `~/ousast-results/` on local disk
|
||||
— amend tasks 5.3/7.2's `--out` accordingly.
|
||||
- **Group 5 image note:** the replay image is the CI-built `ghcr.io/norandom/ousast-replay-task` (+ engine-task),
|
||||
publicly pullable by digest — not a local export-HEAD build and not `ousast-runner`. Task 7.1's "build and push
|
||||
the replay image" is satisfied by the CI `engine-image.yml` run; use that digest.
|
||||
- **Record 07 is a machine-label baseline** (labels SZZ-derived, 100 unchecked); state the caveat in the record.
|
||||
- **Baseline lane (7.2):** run on kube (`--lane ax`). The `ax` binary and the AX-lane store endpoint are
|
||||
operator-owned infra to stand up before 7.2; do not assume the cluster/network state.
|
||||
|
||||
Execution rules:
|
||||
|
||||
- **Who does what.** Codex implements every coding task offline: no network, no model, no store, no cluster. The
|
||||
|
||||
Reference in New Issue
Block a user