mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00

Files

T

Marius CiepluchandClaude Opus 4.8 d03590148f docs(g0): record the 2026-10-11 state and readiness plan; bucket-first + image provenance

Fold the night's learnings into the roadmap, steering and the
unseen-repo-evaluation spec so the process is reproducible and the
project is "ready now":

- roadmap.md: a 2026-10-11 state table (pool p1 frozen; harvest hardened
  + bucket-first; replay-image provenance pinned; baseline-on-kube next)
  and a 5-step readiness plan to record 07.
- storage-streaming.md: RustFS-over-local as the default (local disk is
  the scarce resource); the implemented ousast-corpus layout; replay
  outputs belong on RustFS; the AX store endpoint is operator-owned.
- design.md: an "Amendments (2026-10-11)" section (bucket-first
  persistence, ~2.2% rate-bound yield, machine-label record 07, pinned
  replay image provenance).
- tasks.md: a folded-in amendment block (pool p1 frozen + freeze recipe;
  bucket-first; CI replay image; machine-label baseline; kube lane +
  ax/store as operator-owned infra).

Facts stated are verified (measured yield, freeze digest, GHCR anon pull,
batch.py's endpoint check); infra capability/policy is left to the
operator, not assumed. The frozen pool artifacts themselves are not
committed here (a separate samples/licensing decision).

Co-Authored-By: Claude Opus 4.8 <[email protected]>

2026-10-11 08:17:06 +02:00

5.2 KiB

Raw Blame History

Stream run artifacts to RustFS, one bucket per purpose

Recorded 2026-10-09 from the maintainer's direction. The host's local disk is the scarce resource (it runs near full, and the draw stops itself at under 2 GB free). Runs should stream their durable artifacts to RustFS instead of accumulating them on local disk, and should spread them across purpose-specific buckets rather than one shared bucket. This relieves the disk and simplifies runs: local state becomes mostly a scratch cache that can be rebuilt or resumed from the object store.

Why per-purpose buckets

One shared bucket (sast-memory) already bit us: an append-rewritten key on a versioned bucket with no noncurrent-version expiry ballooned to >10 GB (see the engine-queue command-upload fix and the lifecycle rules added 2026-10-09). Separate buckets let each kind of artifact carry the lifecycle it actually needs, and keep a transient firehose from polluting durable data.

Proposed split (names illustrative; the admin creates them and grants the store key):

sast-memory — the plane memory store (existing): facts, excerpts, embeddings, index. Durable, versioned, with the noncurrent-version lifecycle now in place.
sast-pools — frozen unseen pools and their provenance: pool-pN.toml, freeze-pN.json, and the draw journal needed to resume/reproduce. Durable, versioned. This is also the "don't redo the sourcing" archive.
sast-runs (or sast-journals) — ephemeral run and queue state (engine-queue, replay results, per-run journals). Aggressive lifecycle: short current expiry + 1-day noncurrent expiry, since it is a transient firehose.
optionally sast-traces — engine/replay traces, if kept.

Principles

Stream, don't hoard. Write a run's durable outputs to its bucket as they are produced; keep only a working scratch (e.g. the live git clone, which cannot be streamed) on local disk, and let the disk guard stay as the backstop.
Lifecycle per bucket, tuned to the artifact: durable+versioned for pools/memory; short+noncurrent-expiry for transient run/queue state. Never a versioned bucket without a noncurrent-version rule.
Resume from the object store, so a wiped or small local disk does not lose a long run.
Least privilege per bucket. A run gets a key scoped to the bucket it writes; the store never creates or configures a bucket (admin does that once), consistent with the current S3Store.
Content discipline unchanged: no secrets, no exploit payloads; pool manifests keep the existing public/private (licence) split.

Foundation and dependency

src/openultrasast/plane/memory.py already speaks boto3 to RustFS (S3Store) with startup verification of versioning/lifecycle/Select/tags; this direction extends that pattern to more buckets rather than inventing a new transport. New buckets and their grants are an operator admin action — the store credentials cannot create buckets or set lifecycle (confirmed by the AccessDenied on the 2026-10-09 lifecycle apply).

Default: RustFS over local disk

Local disk on the VM is the scarce resource (it runs near full; the draw stops itself under 2 GB free). RustFS/S3 has far more room. So the default for any run's durable state and large artifacts is the bucket, not local disk. Local files are a rebuildable scratch cache only (the live git clone, a within-process journal). A long run must be resumable purely from the bucket, and its outputs — including replay result records — are written to RustFS, not ~/ousast-results on local disk.

Status (implemented 2026-10-10/11)

Bucket-first persistence is live, not a future spec. The unseen draw streams every stage to the ousast-corpus bucket (http://10.0.0.54:9001, creds in .env as CORPUS_S3_*; boto3 needs path-style + SigV4) and resumes from it:

unseen/<seed>/decisions/<sha256(repo+fix)>.json — accept/reject per repository (idempotent; content dedup).
unseen/<seed>/resolved/<sha256(ghsa)>.json — per-advisory resolution.
unseen/<seed>/advisories/list.json — the enumerated advisory list.
unseen/repos/<sha256(name)>.json — get_repo cache, not seed-scoped (shared across seeds/runs), so the thousands of used-set canonicalization lookups are paid once ever.
unseen/<seed>/pool/ — the frozen pool and its provenance (pool-p1.toml, private-pool-p1.toml, freeze-p1.json, bisect-p1.json, protocol-p1.md): the durable "don't redo the sourcing" archive.

Implementation: benchmarks/unseen/bank.py (CorpusBank, corpus_settings). The old local journal is a within-process cursor only; the bucket is the cross-run source of truth, so a restart (even after a code change) resumes cheaply. Still to do and consistent with the default above: replay result records (replay.py --out) currently default to ~/ousast-results/unseen/... on local disk — point them at RustFS for the slice-1 baseline and after. The AX (kube) lane already uses presigned object transfer, which is the same principle. One in-repo code fact: benchmarks/ax/batch.py currently expects a public-HTTPS DNS, presign-capable endpoint. How the store endpoint and the cluster's access to it are provided is operator/AX-owned infra, not something this doc should presume.

5.2 KiB Raw Blame History