Commit Graph
4 Commits
Author SHA1 Message Date
Marius CiepluchandClaude Fable 5.1 51d064e381 refactor(memory): S3-generic store on boto3; RustFS is the tested server (no MinIO naming)
The plane's memory store no longer names one vendor. `MinioStore`/`SdkClient`/
`minio_settings`/`MINIO_KEYS` become `S3Store`/`S3Client`/`s3_settings`/`S3_KEYS`;
the URL scheme is `s3://<bucket>[/<prefix>]` (`s3://` alone uses `S3_BUCKET`); the
settings are `S3_ENDPOINT` (a full URL), `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`,
optional `AWS_SESSION_TOKEN` and `S3_REGION`. The client is boto3 (extra `s3`,
imported lazily): SigV4, path-style addressing, SDK checksum trailers off, paginated
listing, versions and delete markers listed and deleted by version (`purge`, used by
the contract tests' teardown so the bucket keeps nothing of a run).

Behaviour is unchanged: the store never configures its bucket and verifies
versioning, the `runs/` expiry rule, S3 Select and tag reads at startup; no local
fallback; every row carries every queryable field with "" placeholders. The admin
hints read `--endpoint-url "$S3_ENDPOINT"`. The fake client still reproduces RustFS's
two Select failures (unbound leading-row field, null-typed column).

Docs: rustfs.md stays the tested server's setup page (policy JSON unchanged, env
names updated, any S3-compatible store with versioning, lifecycle, tags and Select
works); memory.md, ops/ax/README.md, README.md, steering and the two current spec
designs use the S3 naming. Retired specs and committed measurements are untouched.

Real server (RustFS, OUSAST_MEMORY_TEST_S3=1): test_plane_memory + test_learn_compile
75 passed, 1 skipped. Store copy of the live local store to a temporary prefix: every
count matched (776 row objects, 2982 rows, 22 facts, 788 excerpts, 2601 responses,
2 programs, 1793 index entries), 4190 versions purged afterwards.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 10:15:19 +02:00
Marius CiepluchandClaude Opus 5.5 132d3ef45c feat(memory): every row carries its kind's queryable fields
RustFS's S3 Select infers an object's schema from its first 1000 rows: a where
field absent there is unbound (EvaluatorBindingDoesNotExist), and a column null
in all of them but set later fails the whole object (JSONParsingError, measured:
999 leading nulls answer, 1000 fail). plane/memory.py now declares per kind the
fields a where may name (QUERY_FIELDS: facts candidates_digest, verdict
candidate/final, example profile, plus the ROW_FIELDS every row has) and writes
their union on every row at write time as strings, "" where one does not apply
(null measured to break). rows() refuses an unknown kind, an undeclared where
field and where = null before any read. `ousast plane memory-normalise
[--store URL] [--dry-run]` rewrites pre-rule rows once, idempotently, with
counts. The fake object client mimics both RustFS failures.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 08:03:39 +02:00
Marius CiepluchandClaude Opus 5.5 e26fe4b608 fix(memory): the S3 store verifies its bucket and never falls back
Maintainer decisions 2026-10-02: "it has the featurs. fallback not desired" and
"Admin sets it once; agent verifies".

- MinioStore never configures its bucket: configure()/enable_versioning()/
  expire() are gone; the client protocol only reads versioning and lifecycle.
- verify_bucket() runs when a MinioStore is opened (once per process per
  store): versioning Enabled, an enabled lifecycle rule expiring <prefix>/runs/
  (none expiring the whole store), S3 Select answering a probe, object tags
  readable. Every missing piece is listed in one MemoryStoreError with the
  aws-cli admin command or the permission to grant.
- No S3 Select fallback: a server without Select is refused at open, and a
  failing Select on one object raises (naming RustFS's leading-row schema
  inference when the field is unbound) instead of filtering locally.
- MINIO_REGION is passed through when set.
- Tests: fakes for versioning off/unreadable, lifecycle missing/disabled/
  tag-narrowed/whole-store, Select missing, tags unreadable, and the pass case;
  the real-server contract uses OUSAST_MEMORY_TEST_BUCKET (else MINIO_BUCKET)
  and never writes bucket configuration.
- ops/ax/README.md: RustFS support, the admin one-time setup, the agent policy.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-02 07:26:01 +02:00
Marius CiepluchandClaude Opus 5.5 52d28b5121 feat(plane): memory store (file and MinIO) and the remember task (harnessx-removal task 2)
plane/memory.py: the MemoryStore interface with FileStore (index.jsonl, facts by
content hash, repos/<host>__<owner>__<name>/<pin>.jsonl, idempotent ingest,
deterministic row ids, refusal below 1 GiB free naming the path) and MinioStore
(object metadata and tags, S3 Select pushdown probed once per process with a
logged local fallback, object version ids on every row read, versioning and
runs/ lifecycle helper, presigned URLs). OUSAST_MEMORY selects the backend;
MINIO_* settings come from .env; the minio SDK is the optional extra `minio`.

plane/tasks/remember.py: model-free rows (facts, verdict, unit_cost, alert)
from a case's artifacts; remember_run computes them on the host for a recorded
run without writing into it. reconciler.mark_done (477 lines total), seed
before and ingest after `ousast plane run`, `ousast plane remember <run>`.
The generator annotates each repo-facts Task with openultrasast.io/memory-key
(repo, pin, candidates digest, image digest) and the Run with its population
and split; plane/ regenerated.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-09-30 11:10:28 +02:00