marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

refactor(memory): S3-generic store on boto3; RustFS is the tested server (no MinIO naming)

Browse Source 
The plane's memory store no longer names one vendor. `MinioStore`/`SdkClient`/
`minio_settings`/`MINIO_KEYS` become `S3Store`/`S3Client`/`s3_settings`/`S3_KEYS`;
the URL scheme is `s3://<bucket>[/<prefix>]` (`s3://` alone uses `S3_BUCKET`); the
settings are `S3_ENDPOINT` (a full URL), `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`,
optional `AWS_SESSION_TOKEN` and `S3_REGION`. The client is boto3 (extra `s3`,
imported lazily): SigV4, path-style addressing, SDK checksum trailers off, paginated
listing, versions and delete markers listed and deleted by version (`purge`, used by
the contract tests' teardown so the bucket keeps nothing of a run).

Behaviour is unchanged: the store never configures its bucket and verifies
versioning, the `runs/` expiry rule, S3 Select and tag reads at startup; no local
fallback; every row carries every queryable field with "" placeholders. The admin
hints read `--endpoint-url "$S3_ENDPOINT"`. The fake client still reproduces RustFS's
two Select failures (unbound leading-row field, null-typed column).

Docs: rustfs.md stays the tested server's setup page (policy JSON unchanged, env
names updated, any S3-compatible store with versioning, lifecycle, tags and Select
works); memory.md, ops/ax/README.md, README.md, steering and the two current spec
designs use the S3 naming. Retired specs and committed measurements are untouched.

Real server (RustFS, OUSAST_MEMORY_TEST_S3=1): test_plane_memory + test_learn_compile
75 passed, 1 skipped. Store copy of the live local store to a temporary prefix: every
count matched (776 row objects, 2982 rows, 22 facts, 788 excerpts, 2601 responses,
2 programs, 1793 index entries), 4190 versions purged afterwards.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-10-02 10:15:19 +02:00
co-authored by Claude Fable 5.1
parent 285b6a367f
commit 51d064e381
22 changed files with 410 additions and 437 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/memory.md
+5 -5
View File
@@ -96,17 +96,17 @@ flowchart LR
`OUSAST_MEMORY` selects the backend (`open_store`):
| | `FileStore` | `MinioStore` (S3: RustFS or MinIO) |
| | `FileStore` | `S3Store` (any S3-compatible store; RustFS is the tested server) |
| --- | --- | --- |
| Selected by | `file:///path`, or nothing (default `$OUSAST_RESULTS/plane/memory`) | `minio://<bucket>[/<prefix>]` |
| Needs | a local disk; refuses to write below 1 GiB free | the `minio` extra, `MINIO_*` settings from `.env` or the environment |
| Selected by | `file:///path`, or nothing (default `$OUSAST_RESULTS/plane/memory`) | `s3://<bucket>[/<prefix>]`, or `s3://` for the bucket in `S3_BUCKET` |
| Needs | a local disk; refuses to write below 1 GiB free | the `s3` extra (boto3); `S3_ENDPOINT`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` (optional `S3_REGION`, `AWS_SESSION_TOKEN`) from `.env` or the environment |
| Filtered reads | read and filter locally | S3 Select pushdown (`SelectObjectContent` over JSON Lines); kind filter by object tag |
| Object version cited by provenance | the file's content hash | the bucket's object version id |
| Bucket setup | none | done once by an admin, verified at every start ([RustFS setup](rustfs.md)) |
### Verify at startup, no fallback
The S3 store never configures its bucket. Each time a `MinioStore` is opened for a bucket and
The S3 store never configures its bucket. Each time an `S3Store` is opened for a bucket and
prefix (once per process), `verify_bucket()` checks, using reads plus one small probe object at
`<prefix>/_probe/select.jsonl`:
@@ -125,7 +125,7 @@ messages are listed on [RustFS setup](rustfs.md#what-the-store-verifies-at-start
RustFS's Select infers an object's JSON schema from its leading rows, so a `where` field missing
there fails with `EvaluatorBindingDoesNotExist` even when a later row has it (measured
2026-10-02; `memory.py`, `MinioStore._select`). On main today the store turns that failure into a
2026-10-02; `memory.py`, `S3Store._select`). On main today the store turns that failure into a
`MemoryStoreError` naming the field, never into an empty answer. A fixed set of queryable fields
per row kind, so that every filtered field is present in every row of its kind, is in progress
and not on main as of this page.
docs/rustfs.md
+40 -31
View File
@@ -1,9 +1,11 @@
# RustFS setup for the memory store
The plane's memory store can live in an S3-compatible bucket: `OUSAST_MEMORY=minio://sast-memory`
selects `MinioStore` (`src/openultrasast/plane/memory.py`), which works with RustFS and MinIO. The
maintainer's server is RustFS at `files.because-security.com`, bucket `sast-memory`, store at the
bucket root. What the store keeps there is described on [Memory](memory.md).
The plane's memory store can live in an S3-compatible bucket: `OUSAST_MEMORY=s3://sast-memory`
selects `S3Store` (`src/openultrasast/plane/memory.py`), which talks to the server through boto3.
Any S3-compatible store with bucket versioning, lifecycle rules, object tags and S3 Select works;
RustFS is the server the contract tests run against. The maintainer's server is RustFS at
`files.because-security.com`, bucket `sast-memory`, store at the bucket root. What the store keeps
there is described on [Memory](memory.md).
**The store never configures its bucket.** An admin sets the bucket up once with admin
credentials. The agent account the store runs as can read the bucket's settings but not change
@@ -15,7 +17,7 @@ flowchart LR
admin["Admin, once: versioning on, runs/ expiry rule"] --> bucket[("Bucket sast-memory")]
agent["Agent account: read settings, read and write objects and tags"] --> verify{"verify_bucket at startup"}
bucket --> verify
verify -- "all present" --> store["MinioStore serves reads and writes"]
verify -- "all present" --> store["S3Store serves reads and writes"]
verify -- "anything missing" --> refuse["MemoryStoreError naming each missing piece and its admin command"]
```
@@ -106,36 +108,43 @@ removing the expiry rule needs the admin.
## The `.env` settings
The store reads its endpoint and credentials from `.env` in the working directory or from the
environment (`minio_settings`), never from a manifest, and never prints them.
environment (`s3_settings`), never from a manifest, and never prints them. The names are the ones
the AWS SDKs and CLI use for credentials, so one `.env` serves both the store and `aws s3api`.
| Variable | Value for this server | Notes |
| --- | --- | --- |
| `OUSAST_MEMORY` | `minio://sast-memory` | `minio://<bucket>[/<prefix>]`; unset means the local `FileStore` |
| `MINIO_ENDPOINT` | `files.because-security.com` | host[:port] without a scheme (the `minio` SDK's endpoint); required |
| `MINIO_ACCESS_KEY` | the agent account's access key | required |
| `MINIO_SECRET_KEY` | the agent account's secret key | required |
| `MINIO_SECURE` | `true` | TLS; `0`, `false`, `no` or `off` turn it off; default `true` |
| `MINIO_REGION` | optional | spares a GetBucketLocation call |
| `OUSAST_MEMORY` | `s3://sast-memory` | `s3://<bucket>[/<prefix>]`; `s3://` alone uses `S3_BUCKET`; unset means the local `FileStore` |
| `S3_ENDPOINT` | `https://files.because-security.com` | the full URL, scheme included (`https://host[:port]`); required |
| `AWS_ACCESS_KEY_ID` | the agent account's access key | required |
| `AWS_SECRET_ACCESS_KEY` | the agent account's secret key | required |
| `AWS_SESSION_TOKEN` | optional | for temporary credentials |
| `S3_REGION` | optional (`us-east-1` when unset) | the SigV4 signing region; spares a GetBucketLocation call |
| `S3_BUCKET` | `sast-memory` | the bucket `s3://` names when the URL has none; also the real-server tests' default bucket |
The client signs with SigV4 and uses path-style addressing (`https://host/bucket/key`), which
RustFS needs. The SDK's own checksum trailers are off, because not every S3-compatible server
accepts them.
- **Never commit `.env`.** It is gitignored (`.gitignore`: `.env`, `.env.*`, except
`.env.example`). `.env.example` lists these names with empty values.
- **`.env` never overrides a variable already exported in your shell** (`config.load_dotenv`).
A stale `export MINIO_SECRET_KEY=...` in your profile silently wins over the file; `unset` it
when a change in `.env` seems to have no effect.
- The store needs the `minio` extra (`uv sync --extra minio`); without it opening the store fails
with `OUSAST_MEMORY=minio://... needs the minio SDK: install openultrasast[minio]`. A missing
variable fails with `OUSAST_MEMORY=minio://... needs MINIO_ENDPOINT, MINIO_ACCESS_KEY,
MINIO_SECRET_KEY in .env or the environment` (naming only the missing ones).
A stale `export AWS_SECRET_ACCESS_KEY=...` in your profile silently wins over the file; `unset`
it when a change in `.env` seems to have no effect.
- The store needs the `s3` extra (`uv sync --extra s3`, which installs boto3); without it opening
the store fails with `OUSAST_MEMORY=s3://... needs boto3: install openultrasast[s3]`. A missing
variable fails with `OUSAST_MEMORY=s3://... needs S3_ENDPOINT, AWS_ACCESS_KEY_ID,
AWS_SECRET_ACCESS_KEY in .env or the environment` (naming only the missing ones); an endpoint
without a scheme fails with `S3_ENDPOINT must be a URL with a scheme (https://host[:port])`.
## What the store verifies at startup
`MinioStore.verify_bucket()` runs once per process for each bucket and prefix. It reads the
`S3Store.verify_bucket()` runs once per process for each bucket and prefix. It reads the
versioning status and the lifecycle rules, writes a two-line probe object at
`_probe/select.jsonl` (under the prefix, if any), queries it with S3 Select and reads its tags.
Every missing piece is collected into one error:
```text
memory minio://sast-memory: the bucket is not set up for the store. The store never configures its bucket (an admin sets it once, the store verifies); missing:
memory s3://sast-memory: the bucket is not set up for the store. The store never configures its bucket (an admin sets it once, the store verifies); missing:
- <one line per missing piece>
```
@@ -145,16 +154,15 @@ server's error code and the first 200 characters of its message):
| Check | Refusal line |
| --- | --- |
| versioning unreadable | `versioning cannot be read (<failure>): the agent needs s3:GetBucketVersioning on arn:aws:s3:::sast-memory, and versioning must be Enabled` |
| versioning off | `versioning is Off, it must be Enabled (provenance cites object versions). Admin: aws s3api put-bucket-versioning --endpoint-url "$MINIO_ENDPOINT" --bucket sast-memory --versioning-configuration Status=Enabled` (the actual status, e.g. `Suspended`, replaces `Off` when set) |
| versioning off | `versioning is Off, it must be Enabled (provenance cites object versions). Admin: aws s3api put-bucket-versioning --endpoint-url "$S3_ENDPOINT" --bucket sast-memory --versioning-configuration Status=Enabled` (the actual status, e.g. `Suspended`, replaces `Off` when set) |
| lifecycle unreadable | `the lifecycle configuration cannot be read (<failure>): the agent needs s3:GetLifecycleConfiguration on arn:aws:s3:::sast-memory, and a rule must expire runs/` |
| a rule expires everything | `lifecycle rule(s) <rule ids> expire the whole store (repos/, facts/ and the index would be deleted); limit them to runs/` |
| no `runs/` rule | `no enabled lifecycle rule expires runs/ after a number of days. Admin (merged with any rules the bucket already has): aws s3api put-bucket-lifecycle-configuration --endpoint-url "$MINIO_ENDPOINT" --bucket sast-memory --lifecycle-configuration '{"Rules":[{"ID":"ousast-runs-expiry","Status":"Enabled","Filter":{"Prefix":"runs/"},"Expiration":{"Days":30}}]}'` |
| no `runs/` rule | `no enabled lifecycle rule expires runs/ after a number of days. Admin (merged with any rules the bucket already has): aws s3api put-bucket-lifecycle-configuration --endpoint-url "$S3_ENDPOINT" --bucket sast-memory --lifecycle-configuration '{"Rules":[{"ID":"ousast-runs-expiry","Status":"Enabled","Filter":{"Prefix":"runs/"},"Expiration":{"Days":30}}]}'` |
| tags unreadable | `object tags cannot be read on _probe/select.jsonl (<failure>): the store filters row objects by their kind tag, so the agent needs s3:GetObjectTagging on arn:aws:s3:::sast-memory/*` |
| S3 Select missing | `S3 Select (SelectObjectContent over JSON Lines) does not answer on _probe/select.jsonl (<failure>). The store requires it and has no local fallback: use a server with S3 Select (RustFS has it; some MinIO releases removed it) and let the agent s3:PutObject and s3:GetObject` |
| S3 Select missing | `S3 Select (SelectObjectContent over JSON Lines) does not answer on _probe/select.jsonl (<failure>). The store requires it and has no local fallback: use a server with S3 Select (RustFS has it; not every S3-compatible server does) and let the agent s3:PutObject and s3:GetObject` |
The admin commands inside the messages use `--endpoint-url "$MINIO_ENDPOINT"`; with the
host-only `MINIO_ENDPOINT` above, pass the full URL (`https://files.because-security.com`) as in
the steps on this page.
The admin commands inside the messages use `--endpoint-url "$S3_ENDPOINT"`, which is the full
URL from `.env`, so they run as written once `.env` is exported.
There is **no fallback**: the store does not start in a degraded mode, does not drop to a local
store, and does not fetch and filter objects itself.
@@ -168,7 +176,7 @@ present at row 5000). The store does not read that as "no rows", since that answ
drop matches; it raises:
```text
memory minio://sast-memory: S3 Select on <key> failed (EvaluatorBindingDoesNotExist: ...) (a field of [<fields>] is absent from the object's leading rows, which the server reads as its schema); the store has no local fallback
memory s3://sast-memory: S3 Select on <key> failed (EvaluatorBindingDoesNotExist: ...) (a field of [<fields>] is absent from the object's leading rows, which the server reads as its schema); the store has no local fallback
```
The rule that follows: a field the store filters on must be present in every row of its kind
@@ -178,7 +186,8 @@ per kind in the store is in progress and not on main as of this page (see
## Checks against the real server
The contract tests that talk to a real server run only with `OUSAST_MEMORY_TEST_MINIO=1` (bucket
`OUSAST_MEMORY_TEST_BUCKET`, else `MINIO_BUCKET`). They verify the bucket at its root and write
only under a fresh `contract-<id>/` prefix; they never configure the bucket. More on the host
setup: [ax on this host](ops/ax/README.md).
The contract tests that talk to a real server run only with `OUSAST_MEMORY_TEST_S3=1` (bucket
`OUSAST_MEMORY_TEST_BUCKET`, else `S3_BUCKET`). They verify the bucket at its root, write only
under a fresh `contract-<id>/` prefix, and remove every version and delete marker under it
afterwards; they never configure the bucket. More on the host setup:
[ax on this host](ops/ax/README.md).