mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
refactor(memory): S3-generic store on boto3; RustFS is the tested server (no MinIO naming)
The plane's memory store no longer names one vendor. `MinioStore`/`SdkClient`/ `minio_settings`/`MINIO_KEYS` become `S3Store`/`S3Client`/`s3_settings`/`S3_KEYS`; the URL scheme is `s3://<bucket>[/<prefix>]` (`s3://` alone uses `S3_BUCKET`); the settings are `S3_ENDPOINT` (a full URL), `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, optional `AWS_SESSION_TOKEN` and `S3_REGION`. The client is boto3 (extra `s3`, imported lazily): SigV4, path-style addressing, SDK checksum trailers off, paginated listing, versions and delete markers listed and deleted by version (`purge`, used by the contract tests' teardown so the bucket keeps nothing of a run). Behaviour is unchanged: the store never configures its bucket and verifies versioning, the `runs/` expiry rule, S3 Select and tag reads at startup; no local fallback; every row carries every queryable field with "" placeholders. The admin hints read `--endpoint-url "$S3_ENDPOINT"`. The fake client still reproduces RustFS's two Select failures (unbound leading-row field, null-typed column). Docs: rustfs.md stays the tested server's setup page (policy JSON unchanged, env names updated, any S3-compatible store with versioning, lifecycle, tags and Select works); memory.md, ops/ax/README.md, README.md, steering and the two current spec designs use the S3 naming. Retired specs and committed measurements are untouched. Real server (RustFS, OUSAST_MEMORY_TEST_S3=1): test_plane_memory + test_learn_compile 75 passed, 1 skipped. Store copy of the live local store to a temporary prefix: every count matched (776 row objects, 2982 rows, 22 facts, 788 excerpts, 2601 responses, 2 programs, 1793 index entries), 4190 versions purged afterwards. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
285b6a367f
commit
51d064e381
@@ -11,7 +11,7 @@ from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from learn_fixtures import ScriptedChat, by_id, corpus, default_answer, example
|
||||
from test_plane_memory import FakeObjects, _real_minio
|
||||
from test_plane_memory import FakeObjects, _real_s3
|
||||
|
||||
from openultrasast.learn import compile as compile_module
|
||||
from openultrasast.learn.compile import (
|
||||
@@ -34,7 +34,7 @@ from openultrasast.learn.examples import Example
|
||||
from openultrasast.learn.folds import Fold, FoldError
|
||||
from openultrasast.learn.program import Caller, Demo
|
||||
from openultrasast.plane.budget import MeteredClient
|
||||
from openultrasast.plane.memory import FileStore, MemoryStore, MinioStore
|
||||
from openultrasast.plane.memory import FileStore, MemoryStore, S3Store
|
||||
|
||||
PRICES = {"cache_hit_per_m": 0.014, "input_per_m": 0.44, "output_per_m": 1.32}
|
||||
MEMORY = corpus(groups=24, per_group=4)
|
||||
@@ -58,14 +58,14 @@ def answer(messages: Sequence[Mapping[str, object]], temperature: float | None)
|
||||
return json.dumps(data)
|
||||
|
||||
|
||||
@pytest.fixture(params=["file", "fake-minio", "minio"])
|
||||
@pytest.fixture(params=["file", "fake-s3-select", "s3"])
|
||||
def store(request: pytest.FixtureRequest, tmp_path: Path) -> Iterator[MemoryStore]:
|
||||
if request.param == "file":
|
||||
yield FileStore(tmp_path / "memory")
|
||||
elif request.param == "minio":
|
||||
yield from _real_minio()
|
||||
elif request.param == "s3":
|
||||
yield from _real_s3()
|
||||
else:
|
||||
yield MinioStore(FakeObjects("works"), "bucket", "p")
|
||||
yield S3Store(FakeObjects("works"), "bucket", "p")
|
||||
|
||||
|
||||
def _caller(chat: ScriptedChat, store: MemoryStore | None = None, **budget: object) -> Caller:
|
||||
|
||||
Reference in New Issue
Block a user