refactor(memory): S3-generic store on boto3; RustFS is the tested server (no MinIO naming)

The plane's memory store no longer names one vendor. `MinioStore`/`SdkClient`/
`minio_settings`/`MINIO_KEYS` become `S3Store`/`S3Client`/`s3_settings`/`S3_KEYS`;
the URL scheme is `s3://<bucket>[/<prefix>]` (`s3://` alone uses `S3_BUCKET`); the
settings are `S3_ENDPOINT` (a full URL), `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`,
optional `AWS_SESSION_TOKEN` and `S3_REGION`. The client is boto3 (extra `s3`,
imported lazily): SigV4, path-style addressing, SDK checksum trailers off, paginated
listing, versions and delete markers listed and deleted by version (`purge`, used by
the contract tests' teardown so the bucket keeps nothing of a run).

Behaviour is unchanged: the store never configures its bucket and verifies
versioning, the `runs/` expiry rule, S3 Select and tag reads at startup; no local
fallback; every row carries every queryable field with "" placeholders. The admin
hints read `--endpoint-url "$S3_ENDPOINT"`. The fake client still reproduces RustFS's
two Select failures (unbound leading-row field, null-typed column).

Docs: rustfs.md stays the tested server's setup page (policy JSON unchanged, env
names updated, any S3-compatible store with versioning, lifecycle, tags and Select
works); memory.md, ops/ax/README.md, README.md, steering and the two current spec
designs use the S3 naming. Retired specs and committed measurements are untouched.

Real server (RustFS, OUSAST_MEMORY_TEST_S3=1): test_plane_memory + test_learn_compile
75 passed, 1 skipped. Store copy of the live local store to a temporary prefix: every
count matched (776 row objects, 2982 rows, 22 facts, 788 excerpts, 2601 responses,
2 programs, 1793 index entries), 4190 versions purged afterwards.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-10-02 10:15:19 +02:00
co-authored by Claude Fable 5.1
parent 285b6a367f
commit 51d064e381
22 changed files with 410 additions and 437 deletions
@@ -38,7 +38,7 @@ Six facts from the repository and the recorded results shape the design:
(`push/policy.py:802-823`: "No shipped declaration is enabled"), so `pre-push` blocks nothing today. The
engine's operating point replaces the capability-eligibility reasons, and the structural reasons stay.
- **The memory store is the system of record and has the needed shape.** Rows are keyed by repository and pin,
with a closed `KINDS` tuple (`plane/memory.py:52`). There are two backends (FileStore, MinioStore) behind one
with a closed `KINDS` tuple (`plane/memory.py:52`). There are two backends (FileStore, S3Store) behind one
interface (`plane/memory.py:176-291`), content-addressed blobs (`put_facts`, `plane/memory.py:249-255`), and a
train-on-test `Guard` (`improve/memory.py:95-133`). Features, labels, decisions, model artifacts and experiment
outcomes become new row kinds and blob prefixes. No second store is added.
@@ -604,7 +604,7 @@ that hits its ceiling records `unfinished` and produces no artifact.
`sha256(model id, Model parameters digest, messages, temperature, sample index, json_object)`. Every random choice
(compile split, demo draws, tie-breaks, subsets) is seeded from `compile.toml`. Re-running a compile on the same
store snapshot with the cache reproduces the artifact byte for byte at $0 (`test_learn_compile.py::
test_reproducible_from_cache`, on FileStore and on MinioStore gated as today). Without the cache, temperature 0 is
test_reproducible_from_cache`, on FileStore and on S3Store gated as today). Without the cache, temperature 0 is
not a determinism guarantee at the provider; the artifact records the cache digest so a re-run can say whether it
was replayed or re-asked.
@@ -700,7 +700,7 @@ and a top-up. No single run may start whose ceiling exceeds the balance read jus
A compiled program is one canonical JSON (sorted keys, floats with `repr`), a content-addressed blob
`programs/<sha256>.json`, with `programs/index.jsonl` (every version) and `programs/adopted.json` (profile -> sha;
a single overwritten object, versioned under MinIO, history in the index on FileStore). On MinIO the object is
a single overwritten object, versioned on S3, history in the index on FileStore). On S3 the object is
tagged `kind=program` and `profile`.
```
@@ -1038,7 +1038,7 @@ No test makes a model or embedding call: the chat and embedding clients are scri
and stops at its ceiling; the response cache key and replay.
- `tests/test_learn_compile.py`: bootstrap keeps only correct, confident, identity-free rationales; instruction
candidates scored by the balanced Brier metric with ties to fewer tokens; compile split disjoint from evaluation
folds; `test_reproducible_from_cache` (byte-identical artifact on FileStore, and on MinioStore gated as today);
folds; `test_reproducible_from_cache` (byte-identical artifact on FileStore, and on S3Store gated as today);
ceiling -> `unfinished`, no artifact.
- `tests/test_learn_folds.py`: no group in both an evaluation fold and the compile split or retrieval pool, for
outer, leave-one-source-out and leave-one-framework-out folds; determinism by seed.
+1 -1
View File
@@ -47,7 +47,7 @@ a model or embedding call.
redacted, delta diff); `learn/examples.py` and `ousast learn memory build` (label + feature record +
excerpt -> `example` row; refuses `origin: user` and sources outside `sources.toml`; exits 2 when more than
10% of a source has no excerpt); `example` added to `KINDS`; `put_blob`/`get_blob` on both backends.
- Tests: `test_learn_excerpt.py`, example-builder refusals, blob round trip on FileStore (MinIO gated).
- Tests: `test_learn_excerpt.py`, example-builder refusals, blob round trip on FileStore (S3 gated).
- _Requirements: 3.2, 3.6, 7.2_
- [x] 6.2 Metered embeddings and the cache
- `plane/models/openrouter-embedding.yaml` (`input_per_m: 0.02`), `MeteredEmbeddingClient` in