# Memory One store, keyed by repository and pin, holds what plane runs and the decision engine learned. The code is `src/openultrasast/plane/memory.py`; the same layout serves both backends. How the stored rows feed detection is on [Memory and detection](memory-and-detection.md). ## What is stored ``` index.jsonl one row per ingested (run, task): sha256 of its rows, row count facts/.json repository facts by content (identical facts stored once) repos/____/.jsonl the rows of one repository + 40-hex pin, sorted by id excerpts/.txt code excerpts, by content embeddings//.json an excerpt's embedding vector, per embedding model responses/.json cached model responses programs/.json compiled decision-engine programs ``` **Rows.** Every row carries `id`, `kind`, `repo`, `pin`, `run`, `task`, `population`, `split` and `image` (`ROW_FIELDS`); `id` is the sha256 of (kind, run, task, subject), so a repeated row replaces itself. The kinds (`KINDS`) and who writes them: | Kind | Written by | Holds | | --- | --- | --- | | `facts` | `remember` task | the facts entry of a case (points at `facts/.json`; reused by memory key) | | `verdict` | `remember` task | per candidate: each verify pass's answer, the final outcome, whether a tie-break ran, cost | | `unit_cost` | `remember` task | per verify pass and file: usd, calls and token usage | | `alert` | `remember` task | a quick-scan or engine alert on the vulnerable or fixed pin (loop Runs) | | `coverage` | `remember` task | per language and pin: whether any rule could have fired there | | `features` | `remember` task, from the `features` task | one feature record per candidate and family | | `proposal_outcome` | `ousast improve --memory` (not with `--dry-run`), and the loop's `improve` step | what happened to each memory proposal in a round | | `example` | `ousast learn memory build` | a labelled case: label + feature record + excerpt sha (no rationale) | | `label`, `decision`, `experiment`, `arm_outcome`, `experiment_result` | reserved for later decision-engine tasks | declared in `KINDS`; not written by the code on main yet | **Blobs** (`put_blob`) are content-addressed: a 64-hex name under a fixed prefix. Excerpts are written by `learn memory build` (`learn/examples.py`), embeddings by `learn memory embed` (`learn/embeddings.py`, keyed by excerpt sha so an excerpt is embedded once), responses by the program's caller (`learn/program.py`, keyed by the sha256 of model, parameters digest, messages, temperature, sample and format, so a rerun replays at no cost), and compiled programs by `learn compile` (`learn/compile.py`). ## Write path ```mermaid flowchart LR tasks["Plane tasks: facts, verify a/b/c, agree, alerts, features"] --> rem["remember task: memory.jsonl"] rem --> recv["Delivered to the host with the run's artifacts"] recv --> ingest["ingest after ousast plane run, or ousast plane remember RUN"] loop["Loop improve step: proposal_outcome"] --> ingest improve["ousast improve --memory: proposal_outcome"] --> put build["ousast learn memory build: labels + features + excerpts"] --> put["put_rows, put_blob"] embed["ousast learn memory embed"] --> put compile["ousast learn compile, evaluate"] --> put ingest --> idx{"index.jsonl: same run, task and sha256?"} idx -- "yes" --> skip["Skipped"] idx -- "no" --> put put --> store[("Memory store")] ``` - `ingest` validates every row first: a malformed file fails before anything is written. - Rows are merged into their repository + pin object by `id`. Every object is written whole and the index last, so an interrupted ingest leaves nothing a repeat would not repair. - Each row object carries tags (`TAG_FIELDS`: `repo`, `pin`, `kind`, `family`, `run`, `population`, `split`) listing the distinct values of its rows (`*` when they exceed 256 characters). - The plane's store failures never change a run's result: an ingest can be repeated by hand (`cli.py`, `_plane_memory`). ## Read path ```mermaid flowchart LR store[("Memory store")] --> seed["seed before a Run"] seed --> reuse["Facts reused by memory key: repo, pin, candidates, runner image"] seed --> snap["memory-snapshot: rows after the train-on-test guard"] store --> examples["learn: load example rows, excerpts, embeddings"] examples --> retr["Retrieval of similar labelled cases"] store --> responses["Response cache: replay at no cost"] store --> imp["ousast improve --memory: alert, verdict, coverage rows"] imp --> guard["Train-on-test guard"] guard --> rules["Rules M1, M2 to rule-status proposals"] rules --> gate{"Unchanged validator and gate"} ``` - **Seed** (`memory.seed`): a Run task whose `openultrasast.io/memory-key` matches stored facts gets them and is marked done; a changed pin, candidate set or runner image finds nothing and recomputes. The loop's `memory-snapshot` is seeded the same way, because a sandboxed task cannot read the store. - **Retrieval** (`learn/retrieve.py`): see [Memory and detection](memory-and-detection.md#3-retrieval-of-similar-labelled-cases). - **`improve --memory`** (`improve/memory.py`): rows from holdout pairs, from the gated manifest's own cases and from any `--qualify-population` are dropped before any rule sees them. See [Architecture](architecture.md#proposals-from-plane-memory). ## Backends: FileStore and the S3 store `OUSAST_MEMORY` selects the backend (`open_store`): | | `FileStore` | `MinioStore` (S3: RustFS or MinIO) | | --- | --- | --- | | Selected by | `file:///path`, or nothing (default `$OUSAST_RESULTS/plane/memory`) | `minio://[/]` | | Needs | a local disk; refuses to write below 1 GiB free | the `minio` extra, `MINIO_*` settings from `.env` or the environment | | Filtered reads | read and filter locally | S3 Select pushdown (`SelectObjectContent` over JSON Lines); kind filter by object tag | | Object version cited by provenance | the file's content hash | the bucket's object version id | | Bucket setup | none | done once by an admin, verified at every start ([RustFS setup](rustfs.md)) | ### Verify at startup, no fallback The S3 store never configures its bucket. Each time a `MinioStore` is opened for a bucket and prefix (once per process), `verify_bucket()` checks, using reads plus one small probe object at `/_probe/select.jsonl`: 1. versioning is `Enabled` (provenance cites object versions); 2. an enabled lifecycle rule expires `/runs/` after a number of days, as a plain prefix filter, and no enabled rule expires the whole store; 3. S3 Select answers a probe query over JSON Lines; 4. the probe's object tags are readable. Anything missing is named, with the admin command that fixes it, in one `MemoryStoreError`, and the store does not start. There is no local or fetch-and-filter fallback: a filtered read that S3 Select cannot answer raises a `MemoryStoreError` instead of returning "no rows". The exact messages are listed on [RustFS setup](rustfs.md#what-the-store-verifies-at-startup). ### Queryable fields per kind (in progress) RustFS's Select infers an object's JSON schema from its leading rows, so a `where` field missing there fails with `EvaluatorBindingDoesNotExist` even when a later row has it (measured 2026-10-02; `memory.py`, `MinioStore._select`). On main today the store turns that failure into a `MemoryStoreError` naming the field, never into an empty answer. A fixed set of queryable fields per row kind, so that every filtered field is present in every row of its kind, is in progress and not on main as of this page.