mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
refactor(memory): S3-generic store on boto3; RustFS is the tested server (no MinIO naming)
The plane's memory store no longer names one vendor. `MinioStore`/`SdkClient`/ `minio_settings`/`MINIO_KEYS` become `S3Store`/`S3Client`/`s3_settings`/`S3_KEYS`; the URL scheme is `s3://<bucket>[/<prefix>]` (`s3://` alone uses `S3_BUCKET`); the settings are `S3_ENDPOINT` (a full URL), `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, optional `AWS_SESSION_TOKEN` and `S3_REGION`. The client is boto3 (extra `s3`, imported lazily): SigV4, path-style addressing, SDK checksum trailers off, paginated listing, versions and delete markers listed and deleted by version (`purge`, used by the contract tests' teardown so the bucket keeps nothing of a run). Behaviour is unchanged: the store never configures its bucket and verifies versioning, the `runs/` expiry rule, S3 Select and tag reads at startup; no local fallback; every row carries every queryable field with "" placeholders. The admin hints read `--endpoint-url "$S3_ENDPOINT"`. The fake client still reproduces RustFS's two Select failures (unbound leading-row field, null-typed column). Docs: rustfs.md stays the tested server's setup page (policy JSON unchanged, env names updated, any S3-compatible store with versioning, lifecycle, tags and Select works); memory.md, ops/ax/README.md, README.md, steering and the two current spec designs use the S3 naming. Retired specs and committed measurements are untouched. Real server (RustFS, OUSAST_MEMORY_TEST_S3=1): test_plane_memory + test_learn_compile 75 passed, 1 skipped. Store copy of the live local store to a temporary prefix: every count matched (776 row objects, 2982 rows, 22 facts, 788 excerpts, 2601 responses, 2 programs, 1793 index entries), 4190 versions purged afterwards. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
285b6a367f
commit
51d064e381
@@ -218,19 +218,19 @@ Ingest is idempotent: an `index.jsonl` hit skips the delivery, and rows carry a
|
||||
refuses to write when `shutil.disk_usage` reports less than 1 GiB free. The disk is at 98% today, so it names
|
||||
the store path in the error.
|
||||
|
||||
**Storage backends (maintainer, 2026-09-30: "think of using minio features, including its json support. i have a
|
||||
server").** The store is written against one interface, `MemoryStore` (`put_row`, `put_facts`, `get_facts`,
|
||||
**Storage backends (maintainer, 2026-09-30: asked to use the object-store features of the S3 server he runs,
|
||||
including its JSON support).** The store is written against one interface, `MemoryStore` (`put_row`, `put_facts`, `get_facts`,
|
||||
`rows(repo=..., pin=..., kind=..., where=...)`, `index`), with two backends of identical behaviour:
|
||||
|
||||
- `FileStore`: the layout above, for this laptop and the unit tests.
|
||||
- `MinioStore`: the maintainer's MinIO server, and the store of record once the plane runs on a separate
|
||||
- `S3Store`: an S3-compatible bucket (boto3; RustFS is the tested server), and the store of record once the plane runs on a separate
|
||||
Kubernetes cluster. The layout maps onto one bucket with the same key paths:
|
||||
- **Object metadata and tags** on every row object: `repo`, `pin`, `kind`, `family`, `run`, `population`,
|
||||
`split`. Rules list by prefix (`repos/<repo>/`) and filter by tag instead of scanning the bucket.
|
||||
- **Server-side JSON queries**: `rows(where=...)` pushes the filter down with S3 Select
|
||||
(`SelectObjectContent`, JSON Lines input, JSON output), so a proposal rule reads matching rows, not whole
|
||||
files. Support is probed once per process against the server (a small Select on a known object); when the
|
||||
server lacks it (some MinIO releases removed S3 Select), the backend fetches and filters locally, logs the
|
||||
server lacks it, the backend fetches and filters locally, logs the
|
||||
fallback once, and returns the same rows. A query never silently returns nothing because a feature is
|
||||
missing.
|
||||
- **Versioning** on the bucket: each proposal's provenance records the object key and version id of every
|
||||
@@ -240,16 +240,16 @@ server").** The store is written against one interface, `MemoryStore` (`put_row`
|
||||
`facts/` and `proposals/` are kept. The near-full laptop disk stops being the store's limit.
|
||||
- **Presigned URLs** (the step that removes the host receiver for the Kubernetes move, not built in this
|
||||
spec): the reconciler can hand a task presigned PUT URLs for its outputs and GET URLs for its inputs in the
|
||||
start request, so tasks write to MinIO directly and never hold credentials. The interface keeps
|
||||
start request, so tasks write to the bucket directly and never hold credentials. The interface keeps
|
||||
`presign_put`/`presign_get` so that follow-on changes the reconciler, not the store.
|
||||
- **Bucket notifications** (follow-on): an event on new `repos/` rows can trigger the loop Run
|
||||
(event-driven loop engineering).
|
||||
- **Configuration**: `OUSAST_MEMORY` selects the backend (`file:///path` or `minio://<bucket>`); the MinIO
|
||||
endpoint and credentials come from `.env` (`MINIO_ENDPOINT`, `MINIO_ACCESS_KEY`, `MINIO_SECRET_KEY`,
|
||||
`MINIO_SECURE`), never from a manifest, never printed. Client: the `minio` Python SDK as an optional extra
|
||||
(`openultrasast[minio]`); the core install stays PyYAML-only.
|
||||
- **Tests**: the same contract tests run against `FileStore` and against `MinioStore` pointed at a MinIO
|
||||
endpoint (skipped unless `OUSAST_MEMORY_TEST_MINIO` is set), including the Select probe with its fallback
|
||||
- **Configuration**: `OUSAST_MEMORY` selects the backend (`file:///path` or `s3://<bucket>[/<prefix>]`); the S3
|
||||
endpoint and credentials come from `.env` (`S3_ENDPOINT`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`,
|
||||
optional `S3_REGION`), never from a manifest, never printed. Client: boto3 as an optional extra
|
||||
(`openultrasast[s3]`); the core install stays PyYAML-only.
|
||||
- **Tests**: the same contract tests run against `FileStore` and against `S3Store` pointed at an S3-compatible
|
||||
endpoint (skipped unless `OUSAST_MEMORY_TEST_S3` is set), including the Select probe with its fallback
|
||||
forced, versioned provenance, and metadata filters. The disk-space refusal applies to `FileStore` only.
|
||||
|
||||
**`remember` task** (`plane/tasks/remember.py`, no Model, budget `{usd: 0, calls: 0}`, one per case). It reads the
|
||||
|
||||
@@ -13,9 +13,9 @@ and passed its fixture test. Every task is a commit gated on the full suite's ow
|
||||
- _Requirements: 3.1, 5.1_
|
||||
|
||||
- [x] 2. Memory store and the `remember` task
|
||||
- `plane/memory.py`: the `MemoryStore` interface with `FileStore` and `MinioStore` (object metadata/tags,
|
||||
- `plane/memory.py`: the `MemoryStore` interface with `FileStore` and `S3Store` (object metadata/tags,
|
||||
S3 Select JSON queries probed with a local fallback, versioned provenance, lifecycle, presign stubs;
|
||||
`OUSAST_MEMORY` selects; MinIO settings from `.env`; `minio` SDK as an optional extra); JSON rows per
|
||||
`OUSAST_MEMORY` selects; S3 settings from `.env`; boto3 as an optional extra); JSON rows per
|
||||
repository + pin, facts by content hash, `FileStore` ingest refuses below 1 GiB free,
|
||||
`plane/tasks/remember.py` (model-free), `reconciler.mark_done` under the run lock, seed and ingest wiring in
|
||||
`cli._plane`, generator annotations; fact reuse when repository, pin, candidate set and runner image match.
|
||||
|
||||
@@ -38,7 +38,7 @@ Six facts from the repository and the recorded results shape the design:
|
||||
(`push/policy.py:802-823`: "No shipped declaration is enabled"), so `pre-push` blocks nothing today. The
|
||||
engine's operating point replaces the capability-eligibility reasons, and the structural reasons stay.
|
||||
- **The memory store is the system of record and has the needed shape.** Rows are keyed by repository and pin,
|
||||
with a closed `KINDS` tuple (`plane/memory.py:52`). There are two backends (FileStore, MinioStore) behind one
|
||||
with a closed `KINDS` tuple (`plane/memory.py:52`). There are two backends (FileStore, S3Store) behind one
|
||||
interface (`plane/memory.py:176-291`), content-addressed blobs (`put_facts`, `plane/memory.py:249-255`), and a
|
||||
train-on-test `Guard` (`improve/memory.py:95-133`). Features, labels, decisions, model artifacts and experiment
|
||||
outcomes become new row kinds and blob prefixes. No second store is added.
|
||||
@@ -604,7 +604,7 @@ that hits its ceiling records `unfinished` and produces no artifact.
|
||||
`sha256(model id, Model parameters digest, messages, temperature, sample index, json_object)`. Every random choice
|
||||
(compile split, demo draws, tie-breaks, subsets) is seeded from `compile.toml`. Re-running a compile on the same
|
||||
store snapshot with the cache reproduces the artifact byte for byte at $0 (`test_learn_compile.py::
|
||||
test_reproducible_from_cache`, on FileStore and on MinioStore gated as today). Without the cache, temperature 0 is
|
||||
test_reproducible_from_cache`, on FileStore and on S3Store gated as today). Without the cache, temperature 0 is
|
||||
not a determinism guarantee at the provider; the artifact records the cache digest so a re-run can say whether it
|
||||
was replayed or re-asked.
|
||||
|
||||
@@ -700,7 +700,7 @@ and a top-up. No single run may start whose ceiling exceeds the balance read jus
|
||||
|
||||
A compiled program is one canonical JSON (sorted keys, floats with `repr`), a content-addressed blob
|
||||
`programs/<sha256>.json`, with `programs/index.jsonl` (every version) and `programs/adopted.json` (profile -> sha;
|
||||
a single overwritten object, versioned under MinIO, history in the index on FileStore). On MinIO the object is
|
||||
a single overwritten object, versioned on S3, history in the index on FileStore). On S3 the object is
|
||||
tagged `kind=program` and `profile`.
|
||||
|
||||
```
|
||||
@@ -1038,7 +1038,7 @@ No test makes a model or embedding call: the chat and embedding clients are scri
|
||||
and stops at its ceiling; the response cache key and replay.
|
||||
- `tests/test_learn_compile.py`: bootstrap keeps only correct, confident, identity-free rationales; instruction
|
||||
candidates scored by the balanced Brier metric with ties to fewer tokens; compile split disjoint from evaluation
|
||||
folds; `test_reproducible_from_cache` (byte-identical artifact on FileStore, and on MinioStore gated as today);
|
||||
folds; `test_reproducible_from_cache` (byte-identical artifact on FileStore, and on S3Store gated as today);
|
||||
ceiling -> `unfinished`, no artifact.
|
||||
- `tests/test_learn_folds.py`: no group in both an evaluation fold and the compile split or retrieval pool, for
|
||||
outer, leave-one-source-out and leave-one-framework-out folds; determinism by seed.
|
||||
|
||||
@@ -47,7 +47,7 @@ a model or embedding call.
|
||||
redacted, delta diff); `learn/examples.py` and `ousast learn memory build` (label + feature record +
|
||||
excerpt -> `example` row; refuses `origin: user` and sources outside `sources.toml`; exits 2 when more than
|
||||
10% of a source has no excerpt); `example` added to `KINDS`; `put_blob`/`get_blob` on both backends.
|
||||
- Tests: `test_learn_excerpt.py`, example-builder refusals, blob round trip on FileStore (MinIO gated).
|
||||
- Tests: `test_learn_excerpt.py`, example-builder refusals, blob round trip on FileStore (S3 gated).
|
||||
- _Requirements: 3.2, 3.6, 7.2_
|
||||
- [x] 6.2 Metered embeddings and the cache
|
||||
- `plane/models/openrouter-embedding.yaml` (`input_per_m: 0.02`), `MeteredEmbeddingClient` in
|
||||
|
||||
@@ -15,7 +15,7 @@ User-facing documentation: `README.md`, `docs/architecture.md`, `docs/evaluation
|
||||
- **Agentic plane.** google/ax over Agent Substrate on kind is the only executor of model work over whole
|
||||
repositories (`ousast plane run|status|doctor|workspaces|remember|alerts-engine|harvest`). Credentials travel
|
||||
only in the start request through `atenet-router`; egress is per task; budgets are per task; rows go to the
|
||||
memory store (`OUSAST_MEMORY`: FileStore, or MinioStore on the maintainer's MinIO).
|
||||
memory store (`OUSAST_MEMORY`: FileStore, or S3Store on an S3-compatible bucket; RustFS is the tested server).
|
||||
- **Removed.** HarnessX (retired 2026-09-30): its config keys fail, a `[harnessx]` section warns. The LLM judge
|
||||
of the verifier and the LLM fusion panels are gone; fusion is deterministic.
|
||||
- **Improvement.** `ousast improve --memory` proposes rule-status edits from plane memory through the unchanged
|
||||
|
||||
Reference in New Issue
Block a user