mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
spec(harnessx-removal): memory store on MinIO -- metadata, S3 Select JSON queries, versioned provenance
Maintainer: use MinIO features including its JSON support; a server exists. One MemoryStore interface with FileStore (laptop, tests) and MinioStore (the store of record for the separate Kubernetes deployment): tags for indexing, S3 Select pushed-down queries probed with a local fallback, versioning for proposal provenance, lifecycle for raw outputs, presigned URLs and bucket notifications kept for the follow-on that removes the host receiver. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
e19687a379
commit
b3561b5a20
@@ -218,6 +218,40 @@ Ingest is idempotent: an `index.jsonl` hit skips the delivery, and rows carry a
|
|||||||
refuses to write when `shutil.disk_usage` reports less than 1 GiB free. The disk is at 98% today, so it names
|
refuses to write when `shutil.disk_usage` reports less than 1 GiB free. The disk is at 98% today, so it names
|
||||||
the store path in the error.
|
the store path in the error.
|
||||||
|
|
||||||
|
**Storage backends (maintainer, 2026-09-30: "think of using minio features, including its json support. i have a
|
||||||
|
server").** The store is written against one interface, `MemoryStore` (`put_row`, `put_facts`, `get_facts`,
|
||||||
|
`rows(repo=..., pin=..., kind=..., where=...)`, `index`), with two backends of identical behaviour:
|
||||||
|
|
||||||
|
- `FileStore`: the layout above, for this laptop and the unit tests.
|
||||||
|
- `MinioStore`: the maintainer's MinIO server, and the store of record once the plane runs on a separate
|
||||||
|
Kubernetes cluster. The layout maps onto one bucket with the same key paths:
|
||||||
|
- **Object metadata and tags** on every row object: `repo`, `pin`, `kind`, `family`, `run`, `population`,
|
||||||
|
`split`. Rules list by prefix (`repos/<repo>/`) and filter by tag instead of scanning the bucket.
|
||||||
|
- **Server-side JSON queries**: `rows(where=...)` pushes the filter down with S3 Select
|
||||||
|
(`SelectObjectContent`, JSON Lines input, JSON output), so a proposal rule reads matching rows, not whole
|
||||||
|
files. Support is probed once per process against the server (a small Select on a known object); when the
|
||||||
|
server lacks it (some MinIO releases removed S3 Select), the backend fetches and filters locally, logs the
|
||||||
|
fallback once, and returns the same rows. A query never silently returns nothing because a feature is
|
||||||
|
missing.
|
||||||
|
- **Versioning** on the bucket: each proposal's provenance records the object key and version id of every
|
||||||
|
memory row it cites, so its evidence cannot change underneath it (Req 6.4). Object lock (governance mode) is
|
||||||
|
optional for the `proposals/` prefix.
|
||||||
|
- **Lifecycle rules**: raw run outputs under `runs/` expire after a configurable number of days; `repos/`,
|
||||||
|
`facts/` and `proposals/` are kept. The near-full laptop disk stops being the store's limit.
|
||||||
|
- **Presigned URLs** (the step that removes the host receiver for the Kubernetes move, not built in this
|
||||||
|
spec): the reconciler can hand a task presigned PUT URLs for its outputs and GET URLs for its inputs in the
|
||||||
|
start request, so tasks write to MinIO directly and never hold credentials. The interface keeps
|
||||||
|
`presign_put`/`presign_get` so that follow-on changes the reconciler, not the store.
|
||||||
|
- **Bucket notifications** (follow-on): an event on new `repos/` rows can trigger the loop Run
|
||||||
|
(event-driven loop engineering).
|
||||||
|
- **Configuration**: `OUSAST_MEMORY` selects the backend (`file:///path` or `minio://<bucket>`); the MinIO
|
||||||
|
endpoint and credentials come from `.env` (`MINIO_ENDPOINT`, `MINIO_ACCESS_KEY`, `MINIO_SECRET_KEY`,
|
||||||
|
`MINIO_SECURE`), never from a manifest, never printed. Client: the `minio` Python SDK as an optional extra
|
||||||
|
(`openultrasast[minio]`); the core install stays PyYAML-only.
|
||||||
|
- **Tests**: the same contract tests run against `FileStore` and against `MinioStore` pointed at a MinIO
|
||||||
|
endpoint (skipped unless `OUSAST_MEMORY_TEST_MINIO` is set), including the Select probe with its fallback
|
||||||
|
forced, versioned provenance, and metadata filters. The disk-space refusal applies to `FileStore` only.
|
||||||
|
|
||||||
**`remember` task** (`plane/tasks/remember.py`, no Model, budget `{usd: 0, calls: 0}`, one per case). It reads the
|
**`remember` task** (`plane/tasks/remember.py`, no Model, budget `{usd: 0, calls: 0}`, one per case). It reads the
|
||||||
case's delivered artifacts as inputs: `facts.json`, the `units.jsonl` of `va`/`vb`/`vc`, `final/agreed.json` and
|
case's delivered artifacts as inputs: `facts.json`, the `units.jsonl` of `va`/`vb`/`vc`, `final/agreed.json` and
|
||||||
`final/disputed.json` with the per-candidate rows, and `alerts.jsonl`. It takes the repository URL and pin from
|
`final/disputed.json` with the per-candidate rows, and `alerts.jsonl`. It takes the repository URL and pin from
|
||||||
|
|||||||
@@ -13,7 +13,10 @@ and passed its fixture test. Every task is a commit gated on the full suite's ow
|
|||||||
- _Requirements: 3.1, 5.1_
|
- _Requirements: 3.1, 5.1_
|
||||||
|
|
||||||
- [ ] 2. Memory store and the `remember` task
|
- [ ] 2. Memory store and the `remember` task
|
||||||
- `plane/memory.py` (JSON rows per repository + pin, facts by content hash, ingest refuses below 1 GiB free),
|
- `plane/memory.py`: the `MemoryStore` interface with `FileStore` and `MinioStore` (object metadata/tags,
|
||||||
|
S3 Select JSON queries probed with a local fallback, versioned provenance, lifecycle, presign stubs;
|
||||||
|
`OUSAST_MEMORY` selects; MinIO settings from `.env`; `minio` SDK as an optional extra); JSON rows per
|
||||||
|
repository + pin, facts by content hash, `FileStore` ingest refuses below 1 GiB free,
|
||||||
`plane/tasks/remember.py` (model-free), `reconciler.mark_done` under the run lock, seed and ingest wiring in
|
`plane/tasks/remember.py` (model-free), `reconciler.mark_done` under the run lock, seed and ingest wiring in
|
||||||
`cli._plane`, generator annotations; fact reuse when repository, pin, candidate set and runner image match.
|
`cli._plane`, generator annotations; fact reuse when repository, pin, candidate set and runner image match.
|
||||||
- Fixture tests: rows from a recorded Run, reuse on an unchanged pin, recompute on a changed one, disk refusal.
|
- Fixture tests: rows from a recorded Run, reuse on an unchanged pin, recompute on a changed one, disk refusal.
|
||||||
|
|||||||
Reference in New Issue
Block a user