mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-17 12:09:27 +00:00
Marius CiepluchandClaude Opus 4.8
d03590148f
Fold the night's learnings into the roadmap, steering and the unseen-repo-evaluation spec so the process is reproducible and the project is "ready now": - roadmap.md: a 2026-10-11 state table (pool p1 frozen; harvest hardened + bucket-first; replay-image provenance pinned; baseline-on-kube next) and a 5-step readiness plan to record 07. - storage-streaming.md: RustFS-over-local as the default (local disk is the scarce resource); the implemented ousast-corpus layout; replay outputs belong on RustFS; the AX store endpoint is operator-owned. - design.md: an "Amendments (2026-10-11)" section (bucket-first persistence, ~2.2% rate-bound yield, machine-label record 07, pinned replay image provenance). - tasks.md: a folded-in amendment block (pool p1 frozen + freeze recipe; bucket-first; CI replay image; machine-label baseline; kube lane + ax/store as operator-owned infra). Facts stated are verified (measured yield, freeze digest, GHCR anon pull, batch.py's endpoint check); infra capability/policy is left to the operator, not assumed. The frozen pool artifacts themselves are not committed here (a separate samples/licensing decision). Co-Authored-By: Claude Opus 4.8 <[email protected]>
81 lines
5.2 KiB
Markdown
81 lines
5.2 KiB
Markdown
# Stream run artifacts to RustFS, one bucket per purpose
|
|
|
|
Recorded 2026-10-09 from the maintainer's direction. The host's local disk is the scarce
|
|
resource (it runs near full, and the draw stops itself at under 2 GB free). Runs should stream
|
|
their durable artifacts to RustFS instead of accumulating them on local disk, and should spread
|
|
them across **purpose-specific buckets** rather than one shared bucket. This relieves the disk and
|
|
simplifies runs: local state becomes mostly a scratch cache that can be rebuilt or resumed from the
|
|
object store.
|
|
|
|
## Why per-purpose buckets
|
|
|
|
One shared bucket (`sast-memory`) already bit us: an append-rewritten key on a versioned bucket
|
|
with no noncurrent-version expiry ballooned to >10 GB (see the `engine-queue` command-upload fix and
|
|
the lifecycle rules added 2026-10-09). Separate buckets let each kind of artifact carry the
|
|
lifecycle it actually needs, and keep a transient firehose from polluting durable data.
|
|
|
|
Proposed split (names illustrative; the admin creates them and grants the store key):
|
|
|
|
- `sast-memory` — the plane memory store (existing): facts, excerpts, embeddings, index. Durable,
|
|
versioned, with the noncurrent-version lifecycle now in place.
|
|
- `sast-pools` — frozen unseen pools and their provenance: `pool-pN.toml`, `freeze-pN.json`, and the
|
|
draw journal needed to resume/reproduce. Durable, versioned. This is also the "don't redo the
|
|
sourcing" archive.
|
|
- `sast-runs` (or `sast-journals`) — ephemeral run and queue state (engine-queue, replay results,
|
|
per-run journals). Aggressive lifecycle: short current expiry + 1-day noncurrent expiry, since it
|
|
is a transient firehose.
|
|
- optionally `sast-traces` — engine/replay traces, if kept.
|
|
|
|
## Principles
|
|
|
|
- **Stream, don't hoard.** Write a run's durable outputs to its bucket as they are produced; keep
|
|
only a working scratch (e.g. the live git clone, which cannot be streamed) on local disk, and let
|
|
the disk guard stay as the backstop.
|
|
- **Lifecycle per bucket**, tuned to the artifact: durable+versioned for pools/memory;
|
|
short+noncurrent-expiry for transient run/queue state. Never a versioned bucket without a
|
|
noncurrent-version rule.
|
|
- **Resume from the object store**, so a wiped or small local disk does not lose a long run.
|
|
- **Least privilege per bucket.** A run gets a key scoped to the bucket it writes; the store never
|
|
creates or configures a bucket (admin does that once), consistent with the current S3Store.
|
|
- Content discipline unchanged: no secrets, no exploit payloads; pool manifests keep the existing
|
|
public/private (licence) split.
|
|
|
|
## Foundation and dependency
|
|
|
|
`src/openultrasast/plane/memory.py` already speaks boto3 to RustFS (S3Store) with startup
|
|
verification of versioning/lifecycle/Select/tags; this direction extends that pattern to more
|
|
buckets rather than inventing a new transport. New buckets and their grants are an **operator
|
|
admin** action — the store credentials cannot create buckets or set lifecycle (confirmed by the
|
|
`AccessDenied` on the 2026-10-09 lifecycle apply).
|
|
|
|
## Default: RustFS over local disk
|
|
|
|
Local disk on the VM is the scarce resource (it runs near full; the draw stops itself under 2 GB free).
|
|
RustFS/S3 has far more room. So the default for any run's durable state and large artifacts is the bucket,
|
|
not local disk. Local files are a rebuildable scratch cache only (the live git clone, a within-process
|
|
journal). A long run must be resumable purely from the bucket, and its outputs — including replay result
|
|
records — are written to RustFS, not `~/ousast-results` on local disk.
|
|
|
|
## Status (implemented 2026-10-10/11)
|
|
|
|
Bucket-first persistence is live, not a future spec. The unseen draw streams every stage to the
|
|
`ousast-corpus` bucket (`http://10.0.0.54:9001`, creds in `.env` as `CORPUS_S3_*`; boto3 needs path-style +
|
|
SigV4) and resumes from it:
|
|
|
|
- `unseen/<seed>/decisions/<sha256(repo+fix)>.json` — accept/reject per repository (idempotent; content dedup).
|
|
- `unseen/<seed>/resolved/<sha256(ghsa)>.json` — per-advisory resolution.
|
|
- `unseen/<seed>/advisories/list.json` — the enumerated advisory list.
|
|
- `unseen/repos/<sha256(name)>.json` — get_repo cache, **not** seed-scoped (shared across seeds/runs), so the
|
|
thousands of used-set canonicalization lookups are paid once ever.
|
|
- `unseen/<seed>/pool/` — the frozen pool and its provenance (`pool-p1.toml`, `private-pool-p1.toml`,
|
|
`freeze-p1.json`, `bisect-p1.json`, `protocol-p1.md`): the durable "don't redo the sourcing" archive.
|
|
|
|
Implementation: `benchmarks/unseen/bank.py` (`CorpusBank`, `corpus_settings`). The old local journal is a
|
|
within-process cursor only; the bucket is the cross-run source of truth, so a restart (even after a code
|
|
change) resumes cheaply. Still to do and consistent with the default above: replay result records
|
|
(`replay.py --out`) currently default to `~/ousast-results/unseen/...` on local disk — point them at RustFS
|
|
for the slice-1 baseline and after. The AX (kube) lane already uses presigned object transfer, which is the
|
|
same principle. One in-repo code fact: `benchmarks/ax/batch.py` currently expects a public-HTTPS DNS,
|
|
presign-capable endpoint. How the store endpoint and the cluster's access to it are provided is operator/AX-owned
|
|
infra, not something this doc should presume.
|