mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-21 05:59:29 +00:00
refactor(memory): S3-generic store on boto3; RustFS is the tested server (no MinIO naming)
The plane's memory store no longer names one vendor. `MinioStore`/`SdkClient`/ `minio_settings`/`MINIO_KEYS` become `S3Store`/`S3Client`/`s3_settings`/`S3_KEYS`; the URL scheme is `s3://<bucket>[/<prefix>]` (`s3://` alone uses `S3_BUCKET`); the settings are `S3_ENDPOINT` (a full URL), `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, optional `AWS_SESSION_TOKEN` and `S3_REGION`. The client is boto3 (extra `s3`, imported lazily): SigV4, path-style addressing, SDK checksum trailers off, paginated listing, versions and delete markers listed and deleted by version (`purge`, used by the contract tests' teardown so the bucket keeps nothing of a run). Behaviour is unchanged: the store never configures its bucket and verifies versioning, the `runs/` expiry rule, S3 Select and tag reads at startup; no local fallback; every row carries every queryable field with "" placeholders. The admin hints read `--endpoint-url "$S3_ENDPOINT"`. The fake client still reproduces RustFS's two Select failures (unbound leading-row field, null-typed column). Docs: rustfs.md stays the tested server's setup page (policy JSON unchanged, env names updated, any S3-compatible store with versioning, lifecycle, tags and Select works); memory.md, ops/ax/README.md, README.md, steering and the two current spec designs use the S3 naming. Retired specs and committed measurements are untouched. Real server (RustFS, OUSAST_MEMORY_TEST_S3=1): test_plane_memory + test_learn_compile 75 passed, 1 skipped. Store copy of the live local store to a temporary prefix: every count matched (776 row objects, 2982 rows, 22 facts, 788 excerpts, 2601 responses, 2 programs, 1793 index entries), 4190 versions purged afterwards. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
285b6a367f
commit
51d064e381
@@ -218,19 +218,19 @@ Ingest is idempotent: an `index.jsonl` hit skips the delivery, and rows carry a
|
||||
refuses to write when `shutil.disk_usage` reports less than 1 GiB free. The disk is at 98% today, so it names
|
||||
the store path in the error.
|
||||
|
||||
**Storage backends (maintainer, 2026-09-30: "think of using minio features, including its json support. i have a
|
||||
server").** The store is written against one interface, `MemoryStore` (`put_row`, `put_facts`, `get_facts`,
|
||||
**Storage backends (maintainer, 2026-09-30: asked to use the object-store features of the S3 server he runs,
|
||||
including its JSON support).** The store is written against one interface, `MemoryStore` (`put_row`, `put_facts`, `get_facts`,
|
||||
`rows(repo=..., pin=..., kind=..., where=...)`, `index`), with two backends of identical behaviour:
|
||||
|
||||
- `FileStore`: the layout above, for this laptop and the unit tests.
|
||||
- `MinioStore`: the maintainer's MinIO server, and the store of record once the plane runs on a separate
|
||||
- `S3Store`: an S3-compatible bucket (boto3; RustFS is the tested server), and the store of record once the plane runs on a separate
|
||||
Kubernetes cluster. The layout maps onto one bucket with the same key paths:
|
||||
- **Object metadata and tags** on every row object: `repo`, `pin`, `kind`, `family`, `run`, `population`,
|
||||
`split`. Rules list by prefix (`repos/<repo>/`) and filter by tag instead of scanning the bucket.
|
||||
- **Server-side JSON queries**: `rows(where=...)` pushes the filter down with S3 Select
|
||||
(`SelectObjectContent`, JSON Lines input, JSON output), so a proposal rule reads matching rows, not whole
|
||||
files. Support is probed once per process against the server (a small Select on a known object); when the
|
||||
server lacks it (some MinIO releases removed S3 Select), the backend fetches and filters locally, logs the
|
||||
server lacks it, the backend fetches and filters locally, logs the
|
||||
fallback once, and returns the same rows. A query never silently returns nothing because a feature is
|
||||
missing.
|
||||
- **Versioning** on the bucket: each proposal's provenance records the object key and version id of every
|
||||
@@ -240,16 +240,16 @@ server").** The store is written against one interface, `MemoryStore` (`put_row`
|
||||
`facts/` and `proposals/` are kept. The near-full laptop disk stops being the store's limit.
|
||||
- **Presigned URLs** (the step that removes the host receiver for the Kubernetes move, not built in this
|
||||
spec): the reconciler can hand a task presigned PUT URLs for its outputs and GET URLs for its inputs in the
|
||||
start request, so tasks write to MinIO directly and never hold credentials. The interface keeps
|
||||
start request, so tasks write to the bucket directly and never hold credentials. The interface keeps
|
||||
`presign_put`/`presign_get` so that follow-on changes the reconciler, not the store.
|
||||
- **Bucket notifications** (follow-on): an event on new `repos/` rows can trigger the loop Run
|
||||
(event-driven loop engineering).
|
||||
- **Configuration**: `OUSAST_MEMORY` selects the backend (`file:///path` or `minio://<bucket>`); the MinIO
|
||||
endpoint and credentials come from `.env` (`MINIO_ENDPOINT`, `MINIO_ACCESS_KEY`, `MINIO_SECRET_KEY`,
|
||||
`MINIO_SECURE`), never from a manifest, never printed. Client: the `minio` Python SDK as an optional extra
|
||||
(`openultrasast[minio]`); the core install stays PyYAML-only.
|
||||
- **Tests**: the same contract tests run against `FileStore` and against `MinioStore` pointed at a MinIO
|
||||
endpoint (skipped unless `OUSAST_MEMORY_TEST_MINIO` is set), including the Select probe with its fallback
|
||||
- **Configuration**: `OUSAST_MEMORY` selects the backend (`file:///path` or `s3://<bucket>[/<prefix>]`); the S3
|
||||
endpoint and credentials come from `.env` (`S3_ENDPOINT`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`,
|
||||
optional `S3_REGION`), never from a manifest, never printed. Client: boto3 as an optional extra
|
||||
(`openultrasast[s3]`); the core install stays PyYAML-only.
|
||||
- **Tests**: the same contract tests run against `FileStore` and against `S3Store` pointed at an S3-compatible
|
||||
endpoint (skipped unless `OUSAST_MEMORY_TEST_S3` is set), including the Select probe with its fallback
|
||||
forced, versioned provenance, and metadata filters. The disk-space refusal applies to `FileStore` only.
|
||||
|
||||
**`remember` task** (`plane/tasks/remember.py`, no Model, budget `{usd: 0, calls: 0}`, one per case). It reads the
|
||||
|
||||
@@ -13,9 +13,9 @@ and passed its fixture test. Every task is a commit gated on the full suite's ow
|
||||
- _Requirements: 3.1, 5.1_
|
||||
|
||||
- [x] 2. Memory store and the `remember` task
|
||||
- `plane/memory.py`: the `MemoryStore` interface with `FileStore` and `MinioStore` (object metadata/tags,
|
||||
- `plane/memory.py`: the `MemoryStore` interface with `FileStore` and `S3Store` (object metadata/tags,
|
||||
S3 Select JSON queries probed with a local fallback, versioned provenance, lifecycle, presign stubs;
|
||||
`OUSAST_MEMORY` selects; MinIO settings from `.env`; `minio` SDK as an optional extra); JSON rows per
|
||||
`OUSAST_MEMORY` selects; S3 settings from `.env`; boto3 as an optional extra); JSON rows per
|
||||
repository + pin, facts by content hash, `FileStore` ingest refuses below 1 GiB free,
|
||||
`plane/tasks/remember.py` (model-free), `reconciler.mark_done` under the run lock, seed and ingest wiring in
|
||||
`cli._plane`, generator annotations; fact reuse when repository, pin, candidate set and runner image match.
|
||||
|
||||
Reference in New Issue
Block a user