From b3561b5a20401f94d2615511888f95b14e63d7c0 Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Wed, 30 Sep 2026 10:37:14 +0200 Subject: [PATCH] spec(harnessx-removal): memory store on MinIO -- metadata, S3 Select JSON queries, versioned provenance Maintainer: use MinIO features including its JSON support; a server exists. One MemoryStore interface with FileStore (laptop, tests) and MinioStore (the store of record for the separate Kubernetes deployment): tags for indexing, S3 Select pushed-down queries probed with a local fallback, versioning for proposal provenance, lifecycle for raw outputs, presigned URLs and bucket notifications kept for the follow-on that removes the host receiver. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo --- .kiro/specs/harnessx-removal/design.md | 34 ++++++++++++++++++++++++++ .kiro/specs/harnessx-removal/tasks.md | 5 +++- 2 files changed, 38 insertions(+), 1 deletion(-) diff --git a/.kiro/specs/harnessx-removal/design.md b/.kiro/specs/harnessx-removal/design.md index 21ce4a3..74f3a3d 100644 --- a/.kiro/specs/harnessx-removal/design.md +++ b/.kiro/specs/harnessx-removal/design.md @@ -218,6 +218,40 @@ Ingest is idempotent: an `index.jsonl` hit skips the delivery, and rows carry a refuses to write when `shutil.disk_usage` reports less than 1 GiB free. The disk is at 98% today, so it names the store path in the error. +**Storage backends (maintainer, 2026-09-30: "think of using minio features, including its json support. i have a +server").** The store is written against one interface, `MemoryStore` (`put_row`, `put_facts`, `get_facts`, +`rows(repo=..., pin=..., kind=..., where=...)`, `index`), with two backends of identical behaviour: + +- `FileStore`: the layout above, for this laptop and the unit tests. +- `MinioStore`: the maintainer's MinIO server, and the store of record once the plane runs on a separate + Kubernetes cluster. The layout maps onto one bucket with the same key paths: + - **Object metadata and tags** on every row object: `repo`, `pin`, `kind`, `family`, `run`, `population`, + `split`. Rules list by prefix (`repos//`) and filter by tag instead of scanning the bucket. + - **Server-side JSON queries**: `rows(where=...)` pushes the filter down with S3 Select + (`SelectObjectContent`, JSON Lines input, JSON output), so a proposal rule reads matching rows, not whole + files. Support is probed once per process against the server (a small Select on a known object); when the + server lacks it (some MinIO releases removed S3 Select), the backend fetches and filters locally, logs the + fallback once, and returns the same rows. A query never silently returns nothing because a feature is + missing. + - **Versioning** on the bucket: each proposal's provenance records the object key and version id of every + memory row it cites, so its evidence cannot change underneath it (Req 6.4). Object lock (governance mode) is + optional for the `proposals/` prefix. + - **Lifecycle rules**: raw run outputs under `runs/` expire after a configurable number of days; `repos/`, + `facts/` and `proposals/` are kept. The near-full laptop disk stops being the store's limit. + - **Presigned URLs** (the step that removes the host receiver for the Kubernetes move, not built in this + spec): the reconciler can hand a task presigned PUT URLs for its outputs and GET URLs for its inputs in the + start request, so tasks write to MinIO directly and never hold credentials. The interface keeps + `presign_put`/`presign_get` so that follow-on changes the reconciler, not the store. + - **Bucket notifications** (follow-on): an event on new `repos/` rows can trigger the loop Run + (event-driven loop engineering). +- **Configuration**: `OUSAST_MEMORY` selects the backend (`file:///path` or `minio://`); the MinIO + endpoint and credentials come from `.env` (`MINIO_ENDPOINT`, `MINIO_ACCESS_KEY`, `MINIO_SECRET_KEY`, + `MINIO_SECURE`), never from a manifest, never printed. Client: the `minio` Python SDK as an optional extra + (`openultrasast[minio]`); the core install stays PyYAML-only. +- **Tests**: the same contract tests run against `FileStore` and against `MinioStore` pointed at a MinIO + endpoint (skipped unless `OUSAST_MEMORY_TEST_MINIO` is set), including the Select probe with its fallback + forced, versioned provenance, and metadata filters. The disk-space refusal applies to `FileStore` only. + **`remember` task** (`plane/tasks/remember.py`, no Model, budget `{usd: 0, calls: 0}`, one per case). It reads the case's delivered artifacts as inputs: `facts.json`, the `units.jsonl` of `va`/`vb`/`vc`, `final/agreed.json` and `final/disputed.json` with the per-candidate rows, and `alerts.jsonl`. It takes the repository URL and pin from diff --git a/.kiro/specs/harnessx-removal/tasks.md b/.kiro/specs/harnessx-removal/tasks.md index 6b98eb8..63ea7b3 100644 --- a/.kiro/specs/harnessx-removal/tasks.md +++ b/.kiro/specs/harnessx-removal/tasks.md @@ -13,7 +13,10 @@ and passed its fixture test. Every task is a commit gated on the full suite's ow - _Requirements: 3.1, 5.1_ - [ ] 2. Memory store and the `remember` task - - `plane/memory.py` (JSON rows per repository + pin, facts by content hash, ingest refuses below 1 GiB free), + - `plane/memory.py`: the `MemoryStore` interface with `FileStore` and `MinioStore` (object metadata/tags, + S3 Select JSON queries probed with a local fallback, versioned provenance, lifecycle, presign stubs; + `OUSAST_MEMORY` selects; MinIO settings from `.env`; `minio` SDK as an optional extra); JSON rows per + repository + pin, facts by content hash, `FileStore` ingest refuses below 1 GiB free, `plane/tasks/remember.py` (model-free), `reconciler.mark_done` under the run lock, seed and ingest wiring in `cli._plane`, generator annotations; fact reuse when repository, pin, candidate set and runner image match. - Fixture tests: rows from a recorded Run, reuse on an unchanged pin, recompute on a changed one, disk refusal.