spec(harnessx-removal): memory store on MinIO -- metadata, S3 Select JSON queries, versioned provenance

Maintainer: use MinIO features including its JSON support; a server exists.
One MemoryStore interface with FileStore (laptop, tests) and MinioStore (the
store of record for the separate Kubernetes deployment): tags for indexing,
S3 Select pushed-down queries probed with a local fallback, versioning for
proposal provenance, lifecycle for raw outputs, presigned URLs and bucket
notifications kept for the follow-on that removes the host receiver.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-09-30 10:37:14 +02:00
co-authored by Claude Opus 5.5
parent e19687a379
commit b3561b5a20
2 changed files with 38 additions and 1 deletions
+34
View File
@@ -218,6 +218,40 @@ Ingest is idempotent: an `index.jsonl` hit skips the delivery, and rows carry a
refuses to write when `shutil.disk_usage` reports less than 1 GiB free. The disk is at 98% today, so it names
the store path in the error.
**Storage backends (maintainer, 2026-09-30: "think of using minio features, including its json support. i have a
server").** The store is written against one interface, `MemoryStore` (`put_row`, `put_facts`, `get_facts`,
`rows(repo=..., pin=..., kind=..., where=...)`, `index`), with two backends of identical behaviour:
- `FileStore`: the layout above, for this laptop and the unit tests.
- `MinioStore`: the maintainer's MinIO server, and the store of record once the plane runs on a separate
Kubernetes cluster. The layout maps onto one bucket with the same key paths:
- **Object metadata and tags** on every row object: `repo`, `pin`, `kind`, `family`, `run`, `population`,
`split`. Rules list by prefix (`repos/<repo>/`) and filter by tag instead of scanning the bucket.
- **Server-side JSON queries**: `rows(where=...)` pushes the filter down with S3 Select
(`SelectObjectContent`, JSON Lines input, JSON output), so a proposal rule reads matching rows, not whole
files. Support is probed once per process against the server (a small Select on a known object); when the
server lacks it (some MinIO releases removed S3 Select), the backend fetches and filters locally, logs the
fallback once, and returns the same rows. A query never silently returns nothing because a feature is
missing.
- **Versioning** on the bucket: each proposal's provenance records the object key and version id of every
memory row it cites, so its evidence cannot change underneath it (Req 6.4). Object lock (governance mode) is
optional for the `proposals/` prefix.
- **Lifecycle rules**: raw run outputs under `runs/` expire after a configurable number of days; `repos/`,
`facts/` and `proposals/` are kept. The near-full laptop disk stops being the store's limit.
- **Presigned URLs** (the step that removes the host receiver for the Kubernetes move, not built in this
spec): the reconciler can hand a task presigned PUT URLs for its outputs and GET URLs for its inputs in the
start request, so tasks write to MinIO directly and never hold credentials. The interface keeps
`presign_put`/`presign_get` so that follow-on changes the reconciler, not the store.
- **Bucket notifications** (follow-on): an event on new `repos/` rows can trigger the loop Run
(event-driven loop engineering).
- **Configuration**: `OUSAST_MEMORY` selects the backend (`file:///path` or `minio://<bucket>`); the MinIO
endpoint and credentials come from `.env` (`MINIO_ENDPOINT`, `MINIO_ACCESS_KEY`, `MINIO_SECRET_KEY`,
`MINIO_SECURE`), never from a manifest, never printed. Client: the `minio` Python SDK as an optional extra
(`openultrasast[minio]`); the core install stays PyYAML-only.
- **Tests**: the same contract tests run against `FileStore` and against `MinioStore` pointed at a MinIO
endpoint (skipped unless `OUSAST_MEMORY_TEST_MINIO` is set), including the Select probe with its fallback
forced, versioned provenance, and metadata filters. The disk-space refusal applies to `FileStore` only.
**`remember` task** (`plane/tasks/remember.py`, no Model, budget `{usd: 0, calls: 0}`, one per case). It reads the
case's delivered artifacts as inputs: `facts.json`, the `units.jsonl` of `va`/`vb`/`vc`, `final/agreed.json` and
`final/disputed.json` with the per-candidate rows, and `alerts.jsonl`. It takes the repository URL and pin from
+4 -1
View File
@@ -13,7 +13,10 @@ and passed its fixture test. Every task is a commit gated on the full suite's ow
- _Requirements: 3.1, 5.1_
- [ ] 2. Memory store and the `remember` task
- `plane/memory.py` (JSON rows per repository + pin, facts by content hash, ingest refuses below 1 GiB free),
- `plane/memory.py`: the `MemoryStore` interface with `FileStore` and `MinioStore` (object metadata/tags,
S3 Select JSON queries probed with a local fallback, versioned provenance, lifecycle, presign stubs;
`OUSAST_MEMORY` selects; MinIO settings from `.env`; `minio` SDK as an optional extra); JSON rows per
repository + pin, facts by content hash, `FileStore` ingest refuses below 1 GiB free,
`plane/tasks/remember.py` (model-free), `reconciler.mark_done` under the run lock, seed and ingest wiring in
`cli._plane`, generator annotations; fact reuse when repository, pin, candidate set and runner image match.
- Fixture tests: rows from a recorded Run, reuse on an unchanged pin, recompute on a changed one, disk refusal.