mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
Marius CiepluchandClaude Fable 5.1
51d064e381
The plane's memory store no longer names one vendor. `MinioStore`/`SdkClient`/ `minio_settings`/`MINIO_KEYS` become `S3Store`/`S3Client`/`s3_settings`/`S3_KEYS`; the URL scheme is `s3://<bucket>[/<prefix>]` (`s3://` alone uses `S3_BUCKET`); the settings are `S3_ENDPOINT` (a full URL), `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, optional `AWS_SESSION_TOKEN` and `S3_REGION`. The client is boto3 (extra `s3`, imported lazily): SigV4, path-style addressing, SDK checksum trailers off, paginated listing, versions and delete markers listed and deleted by version (`purge`, used by the contract tests' teardown so the bucket keeps nothing of a run). Behaviour is unchanged: the store never configures its bucket and verifies versioning, the `runs/` expiry rule, S3 Select and tag reads at startup; no local fallback; every row carries every queryable field with "" placeholders. The admin hints read `--endpoint-url "$S3_ENDPOINT"`. The fake client still reproduces RustFS's two Select failures (unbound leading-row field, null-typed column). Docs: rustfs.md stays the tested server's setup page (policy JSON unchanged, env names updated, any S3-compatible store with versioning, lifecycle, tags and Select works); memory.md, ops/ax/README.md, README.md, steering and the two current spec designs use the S3 naming. Retired specs and committed measurements are untouched. Real server (RustFS, OUSAST_MEMORY_TEST_S3=1): test_plane_memory + test_learn_compile 75 passed, 1 skipped. Store copy of the live local store to a temporary prefix: every count matched (776 row objects, 2982 rows, 22 facts, 788 excerpts, 2601 responses, 2 programs, 1793 index entries), 4190 versions purged afterwards. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
132 lines
7.8 KiB
Markdown
132 lines
7.8 KiB
Markdown
# Memory
|
|
|
|
One store, keyed by repository and pin, holds what plane runs and the decision engine learned.
|
|
The code is `src/openultrasast/plane/memory.py`; the same layout serves both backends.
|
|
How the stored rows feed detection is on [Memory and detection](memory-and-detection.md).
|
|
|
|
## What is stored
|
|
|
|
```
|
|
index.jsonl one row per ingested (run, task): sha256 of its rows, row count
|
|
facts/<sha256>.json repository facts by content (identical facts stored once)
|
|
repos/<host>__<owner>__<name>/<pin>.jsonl the rows of one repository + 40-hex pin, sorted by id
|
|
excerpts/<sha256>.txt code excerpts, by content
|
|
embeddings/<model>/<excerpt sha>.json an excerpt's embedding vector, per embedding model
|
|
responses/<request sha>.json cached model responses
|
|
programs/<program id>.json compiled decision-engine programs
|
|
```
|
|
|
|
**Rows.** Every row carries `id`, `kind`, `repo`, `pin`, `run`, `task`, `population`, `split`
|
|
and `image` (`ROW_FIELDS`); `id` is the sha256 of (kind, run, task, subject), so a repeated row
|
|
replaces itself. The kinds (`KINDS`) and who writes them:
|
|
|
|
| Kind | Written by | Holds |
|
|
| --- | --- | --- |
|
|
| `facts` | `remember` task | the facts entry of a case (points at `facts/<sha256>.json`; reused by memory key) |
|
|
| `verdict` | `remember` task | per candidate: each verify pass's answer, the final outcome, whether a tie-break ran, cost |
|
|
| `unit_cost` | `remember` task | per verify pass and file: usd, calls and token usage |
|
|
| `alert` | `remember` task | a quick-scan or engine alert on the vulnerable or fixed pin (loop Runs) |
|
|
| `coverage` | `remember` task | per language and pin: whether any rule could have fired there |
|
|
| `features` | `remember` task, from the `features` task | one feature record per candidate and family |
|
|
| `proposal_outcome` | `ousast improve --memory` (not with `--dry-run`), and the loop's `improve` step | what happened to each memory proposal in a round |
|
|
| `example` | `ousast learn memory build` | a labelled case: label + feature record + excerpt sha (no rationale) |
|
|
| `label`, `decision`, `experiment`, `arm_outcome`, `experiment_result` | reserved for later decision-engine tasks | declared in `KINDS`; not written by the code on main yet |
|
|
|
|
**Blobs** (`put_blob`) are content-addressed: a 64-hex name under a fixed prefix. Excerpts are
|
|
written by `learn memory build` (`learn/examples.py`), embeddings by `learn memory embed`
|
|
(`learn/embeddings.py`, keyed by excerpt sha so an excerpt is embedded once), responses by the
|
|
program's caller (`learn/program.py`, keyed by the sha256 of model, parameters digest, messages,
|
|
temperature, sample and format, so a rerun replays at no cost), and compiled programs by
|
|
`learn compile` (`learn/compile.py`).
|
|
|
|
## Write path
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
tasks["Plane tasks: facts, verify a/b/c, agree, alerts, features"] --> rem["remember task: memory.jsonl"]
|
|
rem --> recv["Delivered to the host with the run's artifacts"]
|
|
recv --> ingest["ingest after ousast plane run, or ousast plane remember RUN"]
|
|
loop["Loop improve step: proposal_outcome"] --> ingest
|
|
improve["ousast improve --memory: proposal_outcome"] --> put
|
|
build["ousast learn memory build: labels + features + excerpts"] --> put["put_rows, put_blob"]
|
|
embed["ousast learn memory embed"] --> put
|
|
compile["ousast learn compile, evaluate"] --> put
|
|
ingest --> idx{"index.jsonl: same run, task and sha256?"}
|
|
idx -- "yes" --> skip["Skipped"]
|
|
idx -- "no" --> put
|
|
put --> store[("Memory store")]
|
|
```
|
|
|
|
- `ingest` validates every row first: a malformed file fails before anything is written.
|
|
- Rows are merged into their repository + pin object by `id`. Every object is written whole and the
|
|
index last, so an interrupted ingest leaves nothing a repeat would not repair.
|
|
- Each row object carries tags (`TAG_FIELDS`: `repo`, `pin`, `kind`, `family`, `run`,
|
|
`population`, `split`) listing the distinct values of its rows (`*` when they exceed 256
|
|
characters).
|
|
- The plane's store failures never change a run's result: an ingest can be repeated by hand
|
|
(`cli.py`, `_plane_memory`).
|
|
|
|
## Read path
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
store[("Memory store")] --> seed["seed before a Run"]
|
|
seed --> reuse["Facts reused by memory key: repo, pin, candidates, runner image"]
|
|
seed --> snap["memory-snapshot: rows after the train-on-test guard"]
|
|
store --> examples["learn: load example rows, excerpts, embeddings"]
|
|
examples --> retr["Retrieval of similar labelled cases"]
|
|
store --> responses["Response cache: replay at no cost"]
|
|
store --> imp["ousast improve --memory: alert, verdict, coverage rows"]
|
|
imp --> guard["Train-on-test guard"]
|
|
guard --> rules["Rules M1, M2 to rule-status proposals"]
|
|
rules --> gate{"Unchanged validator and gate"}
|
|
```
|
|
|
|
- **Seed** (`memory.seed`): a Run task whose `openultrasast.io/memory-key` matches stored facts gets
|
|
them and is marked done; a changed pin, candidate set or runner image finds nothing and
|
|
recomputes. The loop's `memory-snapshot` is seeded the same way, because a sandboxed task cannot
|
|
read the store.
|
|
- **Retrieval** (`learn/retrieve.py`): see
|
|
[Memory and detection](memory-and-detection.md#3-retrieval-of-similar-labelled-cases).
|
|
- **`improve --memory`** (`improve/memory.py`): rows from holdout pairs, from the gated manifest's
|
|
own cases and from any `--qualify-population` are dropped before any rule sees them. See
|
|
[Architecture](architecture.md#proposals-from-plane-memory).
|
|
|
|
## Backends: FileStore and the S3 store
|
|
|
|
`OUSAST_MEMORY` selects the backend (`open_store`):
|
|
|
|
| | `FileStore` | `S3Store` (any S3-compatible store; RustFS is the tested server) |
|
|
| --- | --- | --- |
|
|
| Selected by | `file:///path`, or nothing (default `$OUSAST_RESULTS/plane/memory`) | `s3://<bucket>[/<prefix>]`, or `s3://` for the bucket in `S3_BUCKET` |
|
|
| Needs | a local disk; refuses to write below 1 GiB free | the `s3` extra (boto3); `S3_ENDPOINT`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` (optional `S3_REGION`, `AWS_SESSION_TOKEN`) from `.env` or the environment |
|
|
| Filtered reads | read and filter locally | S3 Select pushdown (`SelectObjectContent` over JSON Lines); kind filter by object tag |
|
|
| Object version cited by provenance | the file's content hash | the bucket's object version id |
|
|
| Bucket setup | none | done once by an admin, verified at every start ([RustFS setup](rustfs.md)) |
|
|
|
|
### Verify at startup, no fallback
|
|
|
|
The S3 store never configures its bucket. Each time an `S3Store` is opened for a bucket and
|
|
prefix (once per process), `verify_bucket()` checks, using reads plus one small probe object at
|
|
`<prefix>/_probe/select.jsonl`:
|
|
|
|
1. versioning is `Enabled` (provenance cites object versions);
|
|
2. an enabled lifecycle rule expires `<prefix>/runs/` after a number of days, as a plain prefix
|
|
filter, and no enabled rule expires the whole store;
|
|
3. S3 Select answers a probe query over JSON Lines;
|
|
4. the probe's object tags are readable.
|
|
|
|
Anything missing is named, with the admin command that fixes it, in one `MemoryStoreError`, and
|
|
the store does not start. There is no local or fetch-and-filter fallback: a filtered read that
|
|
S3 Select cannot answer raises a `MemoryStoreError` instead of returning "no rows". The exact
|
|
messages are listed on [RustFS setup](rustfs.md#what-the-store-verifies-at-startup).
|
|
|
|
### Queryable fields per kind (in progress)
|
|
|
|
RustFS's Select infers an object's JSON schema from its leading rows, so a `where` field missing
|
|
there fails with `EvaluatorBindingDoesNotExist` even when a later row has it (measured
|
|
2026-10-02; `memory.py`, `S3Store._select`). On main today the store turns that failure into a
|
|
`MemoryStoreError` naming the field, never into an empty answer. A fixed set of queryable fields
|
|
per row kind, so that every filtered field is present in every row of its kind, is in progress
|
|
and not on main as of this page.
|