refactor(memory): S3-generic store on boto3; RustFS is the tested server (no MinIO naming)

The plane's memory store no longer names one vendor. `MinioStore`/`SdkClient`/
`minio_settings`/`MINIO_KEYS` become `S3Store`/`S3Client`/`s3_settings`/`S3_KEYS`;
the URL scheme is `s3://<bucket>[/<prefix>]` (`s3://` alone uses `S3_BUCKET`); the
settings are `S3_ENDPOINT` (a full URL), `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`,
optional `AWS_SESSION_TOKEN` and `S3_REGION`. The client is boto3 (extra `s3`,
imported lazily): SigV4, path-style addressing, SDK checksum trailers off, paginated
listing, versions and delete markers listed and deleted by version (`purge`, used by
the contract tests' teardown so the bucket keeps nothing of a run).

Behaviour is unchanged: the store never configures its bucket and verifies
versioning, the `runs/` expiry rule, S3 Select and tag reads at startup; no local
fallback; every row carries every queryable field with "" placeholders. The admin
hints read `--endpoint-url "$S3_ENDPOINT"`. The fake client still reproduces RustFS's
two Select failures (unbound leading-row field, null-typed column).

Docs: rustfs.md stays the tested server's setup page (policy JSON unchanged, env
names updated, any S3-compatible store with versioning, lifecycle, tags and Select
works); memory.md, ops/ax/README.md, README.md, steering and the two current spec
designs use the S3 naming. Retired specs and committed measurements are untouched.

Real server (RustFS, OUSAST_MEMORY_TEST_S3=1): test_plane_memory + test_learn_compile
75 passed, 1 skipped. Store copy of the live local store to a temporary prefix: every
count matched (776 row objects, 2982 rows, 22 facts, 788 excerpts, 2601 responses,
2 programs, 1793 index entries), 4190 versions purged afterwards.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-10-02 10:15:19 +02:00
co-authored by Claude Fable 5.1
parent 285b6a367f
commit 51d064e381
22 changed files with 410 additions and 437 deletions
+11 -11
View File
@@ -218,19 +218,19 @@ Ingest is idempotent: an `index.jsonl` hit skips the delivery, and rows carry a
refuses to write when `shutil.disk_usage` reports less than 1 GiB free. The disk is at 98% today, so it names
the store path in the error.
**Storage backends (maintainer, 2026-09-30: "think of using minio features, including its json support. i have a
server").** The store is written against one interface, `MemoryStore` (`put_row`, `put_facts`, `get_facts`,
**Storage backends (maintainer, 2026-09-30: asked to use the object-store features of the S3 server he runs,
including its JSON support).** The store is written against one interface, `MemoryStore` (`put_row`, `put_facts`, `get_facts`,
`rows(repo=..., pin=..., kind=..., where=...)`, `index`), with two backends of identical behaviour:
- `FileStore`: the layout above, for this laptop and the unit tests.
- `MinioStore`: the maintainer's MinIO server, and the store of record once the plane runs on a separate
- `S3Store`: an S3-compatible bucket (boto3; RustFS is the tested server), and the store of record once the plane runs on a separate
Kubernetes cluster. The layout maps onto one bucket with the same key paths:
- **Object metadata and tags** on every row object: `repo`, `pin`, `kind`, `family`, `run`, `population`,
`split`. Rules list by prefix (`repos/<repo>/`) and filter by tag instead of scanning the bucket.
- **Server-side JSON queries**: `rows(where=...)` pushes the filter down with S3 Select
(`SelectObjectContent`, JSON Lines input, JSON output), so a proposal rule reads matching rows, not whole
files. Support is probed once per process against the server (a small Select on a known object); when the
server lacks it (some MinIO releases removed S3 Select), the backend fetches and filters locally, logs the
server lacks it, the backend fetches and filters locally, logs the
fallback once, and returns the same rows. A query never silently returns nothing because a feature is
missing.
- **Versioning** on the bucket: each proposal's provenance records the object key and version id of every
@@ -240,16 +240,16 @@ server").** The store is written against one interface, `MemoryStore` (`put_row`
`facts/` and `proposals/` are kept. The near-full laptop disk stops being the store's limit.
- **Presigned URLs** (the step that removes the host receiver for the Kubernetes move, not built in this
spec): the reconciler can hand a task presigned PUT URLs for its outputs and GET URLs for its inputs in the
start request, so tasks write to MinIO directly and never hold credentials. The interface keeps
start request, so tasks write to the bucket directly and never hold credentials. The interface keeps
`presign_put`/`presign_get` so that follow-on changes the reconciler, not the store.
- **Bucket notifications** (follow-on): an event on new `repos/` rows can trigger the loop Run
(event-driven loop engineering).
- **Configuration**: `OUSAST_MEMORY` selects the backend (`file:///path` or `minio://<bucket>`); the MinIO
endpoint and credentials come from `.env` (`MINIO_ENDPOINT`, `MINIO_ACCESS_KEY`, `MINIO_SECRET_KEY`,
`MINIO_SECURE`), never from a manifest, never printed. Client: the `minio` Python SDK as an optional extra
(`openultrasast[minio]`); the core install stays PyYAML-only.
- **Tests**: the same contract tests run against `FileStore` and against `MinioStore` pointed at a MinIO
endpoint (skipped unless `OUSAST_MEMORY_TEST_MINIO` is set), including the Select probe with its fallback
- **Configuration**: `OUSAST_MEMORY` selects the backend (`file:///path` or `s3://<bucket>[/<prefix>]`); the S3
endpoint and credentials come from `.env` (`S3_ENDPOINT`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`,
optional `S3_REGION`), never from a manifest, never printed. Client: boto3 as an optional extra
(`openultrasast[s3]`); the core install stays PyYAML-only.
- **Tests**: the same contract tests run against `FileStore` and against `S3Store` pointed at an S3-compatible
endpoint (skipped unless `OUSAST_MEMORY_TEST_S3` is set), including the Select probe with its fallback
forced, versioned provenance, and metadata filters. The disk-space refusal applies to `FileStore` only.
**`remember` task** (`plane/tasks/remember.py`, no Model, budget `{usd: 0, calls: 0}`, one per case). It reads the
+2 -2
View File
@@ -13,9 +13,9 @@ and passed its fixture test. Every task is a commit gated on the full suite's ow
- _Requirements: 3.1, 5.1_
- [x] 2. Memory store and the `remember` task
- `plane/memory.py`: the `MemoryStore` interface with `FileStore` and `MinioStore` (object metadata/tags,
- `plane/memory.py`: the `MemoryStore` interface with `FileStore` and `S3Store` (object metadata/tags,
S3 Select JSON queries probed with a local fallback, versioned provenance, lifecycle, presign stubs;
`OUSAST_MEMORY` selects; MinIO settings from `.env`; `minio` SDK as an optional extra); JSON rows per
`OUSAST_MEMORY` selects; S3 settings from `.env`; boto3 as an optional extra); JSON rows per
repository + pin, facts by content hash, `FileStore` ingest refuses below 1 GiB free,
`plane/tasks/remember.py` (model-free), `reconciler.mark_done` under the run lock, seed and ingest wiring in
`cli._plane`, generator annotations; fact reuse when repository, pin, candidate set and runner image match.
@@ -38,7 +38,7 @@ Six facts from the repository and the recorded results shape the design:
(`push/policy.py:802-823`: "No shipped declaration is enabled"), so `pre-push` blocks nothing today. The
engine's operating point replaces the capability-eligibility reasons, and the structural reasons stay.
- **The memory store is the system of record and has the needed shape.** Rows are keyed by repository and pin,
with a closed `KINDS` tuple (`plane/memory.py:52`). There are two backends (FileStore, MinioStore) behind one
with a closed `KINDS` tuple (`plane/memory.py:52`). There are two backends (FileStore, S3Store) behind one
interface (`plane/memory.py:176-291`), content-addressed blobs (`put_facts`, `plane/memory.py:249-255`), and a
train-on-test `Guard` (`improve/memory.py:95-133`). Features, labels, decisions, model artifacts and experiment
outcomes become new row kinds and blob prefixes. No second store is added.
@@ -604,7 +604,7 @@ that hits its ceiling records `unfinished` and produces no artifact.
`sha256(model id, Model parameters digest, messages, temperature, sample index, json_object)`. Every random choice
(compile split, demo draws, tie-breaks, subsets) is seeded from `compile.toml`. Re-running a compile on the same
store snapshot with the cache reproduces the artifact byte for byte at $0 (`test_learn_compile.py::
test_reproducible_from_cache`, on FileStore and on MinioStore gated as today). Without the cache, temperature 0 is
test_reproducible_from_cache`, on FileStore and on S3Store gated as today). Without the cache, temperature 0 is
not a determinism guarantee at the provider; the artifact records the cache digest so a re-run can say whether it
was replayed or re-asked.
@@ -700,7 +700,7 @@ and a top-up. No single run may start whose ceiling exceeds the balance read jus
A compiled program is one canonical JSON (sorted keys, floats with `repr`), a content-addressed blob
`programs/<sha256>.json`, with `programs/index.jsonl` (every version) and `programs/adopted.json` (profile -> sha;
a single overwritten object, versioned under MinIO, history in the index on FileStore). On MinIO the object is
a single overwritten object, versioned on S3, history in the index on FileStore). On S3 the object is
tagged `kind=program` and `profile`.
```
@@ -1038,7 +1038,7 @@ No test makes a model or embedding call: the chat and embedding clients are scri
and stops at its ceiling; the response cache key and replay.
- `tests/test_learn_compile.py`: bootstrap keeps only correct, confident, identity-free rationales; instruction
candidates scored by the balanced Brier metric with ties to fewer tokens; compile split disjoint from evaluation
folds; `test_reproducible_from_cache` (byte-identical artifact on FileStore, and on MinioStore gated as today);
folds; `test_reproducible_from_cache` (byte-identical artifact on FileStore, and on S3Store gated as today);
ceiling -> `unfinished`, no artifact.
- `tests/test_learn_folds.py`: no group in both an evaluation fold and the compile split or retrieval pool, for
outer, leave-one-source-out and leave-one-framework-out folds; determinism by seed.
+1 -1
View File
@@ -47,7 +47,7 @@ a model or embedding call.
redacted, delta diff); `learn/examples.py` and `ousast learn memory build` (label + feature record +
excerpt -> `example` row; refuses `origin: user` and sources outside `sources.toml`; exits 2 when more than
10% of a source has no excerpt); `example` added to `KINDS`; `put_blob`/`get_blob` on both backends.
- Tests: `test_learn_excerpt.py`, example-builder refusals, blob round trip on FileStore (MinIO gated).
- Tests: `test_learn_excerpt.py`, example-builder refusals, blob round trip on FileStore (S3 gated).
- _Requirements: 3.2, 3.6, 7.2_
- [x] 6.2 Metered embeddings and the cache
- `plane/models/openrouter-embedding.yaml` (`input_per_m: 0.02`), `MeteredEmbeddingClient` in
+1 -1
View File
@@ -15,7 +15,7 @@ User-facing documentation: `README.md`, `docs/architecture.md`, `docs/evaluation
- **Agentic plane.** google/ax over Agent Substrate on kind is the only executor of model work over whole
repositories (`ousast plane run|status|doctor|workspaces|remember|alerts-engine|harvest`). Credentials travel
only in the start request through `atenet-router`; egress is per task; budgets are per task; rows go to the
memory store (`OUSAST_MEMORY`: FileStore, or MinioStore on the maintainer's MinIO).
memory store (`OUSAST_MEMORY`: FileStore, or S3Store on an S3-compatible bucket; RustFS is the tested server).
- **Removed.** HarnessX (retired 2026-09-30): its config keys fail, a `[harnessx]` section warns. The LLM judge
of the verifier and the LLM fusion panels are gone; fusion is deterministic.
- **Improvement.** `ousast improve --memory` proposes rule-status edits from plane memory through the unchanged