mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
Marius CiepluchandClaude Fable 5.1
fbc3220bbe
learn/experiments.py: YAML manifests under plane/experiments/<id>.yaml, digest-checked at register, run and analyse time; `experiment`, `arm_outcome` and `experiment_result` rows; paired arms in a seeded repository order with a per-unit coin; arm A replay-only at $0; canary replicates under a fresh cache salt; repository-cluster paired bootstrap, exact McNemar, two O'Brien-Fleming looks, adopt/reject/equivalent/ inconclusive. `ousast learn experiment register|units|run|analyse`. Program: `ProgramSpec.sampling = retrieval_ensemble` (k samples at temperature 0 over k disjoint retrieved example sets; default unchanged); `Caller.salt`; `retrieve(exclude=)`. verify: OUSAST_VERIFY_CALLERS=off. Manifests exp-001-known-callers (plane arms, units pending) and exp-002-retrieval-ensemble (slice-2 incumbents, ceiling $4) -- not run. 18 scripted tests; module audit regenerated (151 modules). Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
135 lines
8.3 KiB
Markdown
135 lines
8.3 KiB
Markdown
# Memory
|
|
|
|
One store, keyed by repository and pin, holds what plane runs and the decision engine learned.
|
|
The code is `src/openultrasast/plane/memory.py`; the same layout serves both backends.
|
|
How the stored rows feed detection is on [Memory and detection](memory-and-detection.md).
|
|
|
|
## What is stored
|
|
|
|
```
|
|
index.jsonl one row per ingested (run, task): sha256 of its rows, row count
|
|
facts/<sha256>.json repository facts by content (identical facts stored once)
|
|
repos/<host>__<owner>__<name>/<pin>.jsonl the rows of one repository + 40-hex pin, sorted by id
|
|
excerpts/<sha256>.txt code excerpts, by content
|
|
embeddings/<model>/<excerpt sha>.json an excerpt's embedding vector, per embedding model
|
|
responses/<request sha>.json cached model responses
|
|
programs/<program id>.json compiled decision-engine programs
|
|
```
|
|
|
|
**Rows.** Every row carries `id`, `kind`, `repo`, `pin`, `run`, `task`, `population`, `split`
|
|
and `image` (`ROW_FIELDS`); `id` is the sha256 of (kind, run, task, subject), so a repeated row
|
|
replaces itself. The kinds (`KINDS`) and who writes them:
|
|
|
|
| Kind | Written by | Holds |
|
|
| --- | --- | --- |
|
|
| `facts` | `remember` task | the facts entry of a case (points at `facts/<sha256>.json`; reused by memory key) |
|
|
| `verdict` | `remember` task | per candidate: each verify pass's answer, the final outcome, whether a tie-break ran, cost |
|
|
| `unit_cost` | `remember` task | per verify pass and file: usd, calls and token usage |
|
|
| `alert` | `remember` task | a quick-scan or engine alert on the vulnerable or fixed pin (loop Runs) |
|
|
| `coverage` | `remember` task | per language and pin: whether any rule could have fired there |
|
|
| `features` | `remember` task, from the `features` task | one feature record per candidate and family |
|
|
| `proposal_outcome` | `ousast improve --memory` (not with `--dry-run`), and the loop's `improve` step | what happened to each memory proposal in a round |
|
|
| `example` | `ousast learn memory build` | a labelled case: label + feature record + excerpt sha (no rationale) |
|
|
| `experiment` | `ousast learn experiment register` (task `register`) and `run` (task `run`) | a registered manifest's digests and provenance; a run's counts-only summary. Repo `experiments/<id>`, pin the registering commit |
|
|
| `arm_outcome` | `ousast learn experiment run` | one decision per (unit, arm, replicate): score, verdict, usd, order |
|
|
| `experiment_result` | `ousast learn experiment analyse --record` | the paired estimates, the looks and the adoption verdict (counts and intervals only) |
|
|
| `label`, `decision` | reserved for later decision-engine tasks | declared in `KINDS`; not written by the code on main yet |
|
|
|
|
**Blobs** (`put_blob`) are content-addressed: a 64-hex name under a fixed prefix. Excerpts are
|
|
written by `learn memory build` (`learn/examples.py`), embeddings by `learn memory embed`
|
|
(`learn/embeddings.py`, keyed by excerpt sha so an excerpt is embedded once), responses by the
|
|
program's caller (`learn/program.py`, keyed by the sha256 of model, parameters digest, messages,
|
|
temperature, sample and format, so a rerun replays at no cost), and compiled programs by
|
|
`learn compile` (`learn/compile.py`).
|
|
|
|
## Write path
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
tasks["Plane tasks: facts, verify a/b/c, agree, alerts, features"] --> rem["remember task: memory.jsonl"]
|
|
rem --> recv["Delivered to the host with the run's artifacts"]
|
|
recv --> ingest["ingest after ousast plane run, or ousast plane remember RUN"]
|
|
loop["Loop improve step: proposal_outcome"] --> ingest
|
|
improve["ousast improve --memory: proposal_outcome"] --> put
|
|
build["ousast learn memory build: labels + features + excerpts"] --> put["put_rows, put_blob"]
|
|
embed["ousast learn memory embed"] --> put
|
|
compile["ousast learn compile, evaluate"] --> put
|
|
ingest --> idx{"index.jsonl: same run, task and sha256?"}
|
|
idx -- "yes" --> skip["Skipped"]
|
|
idx -- "no" --> put
|
|
put --> store[("Memory store")]
|
|
```
|
|
|
|
- `ingest` validates every row first: a malformed file fails before anything is written.
|
|
- Rows are merged into their repository + pin object by `id`. Every object is written whole and the
|
|
index last, so an interrupted ingest leaves nothing a repeat would not repair.
|
|
- Each row object carries tags (`TAG_FIELDS`: `repo`, `pin`, `kind`, `family`, `run`,
|
|
`population`, `split`) listing the distinct values of its rows (`*` when they exceed 256
|
|
characters).
|
|
- The plane's store failures never change a run's result: an ingest can be repeated by hand
|
|
(`cli.py`, `_plane_memory`).
|
|
|
|
## Read path
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
store[("Memory store")] --> seed["seed before a Run"]
|
|
seed --> reuse["Facts reused by memory key: repo, pin, candidates, runner image"]
|
|
seed --> snap["memory-snapshot: rows after the train-on-test guard"]
|
|
store --> examples["learn: load example rows, excerpts, embeddings"]
|
|
examples --> retr["Retrieval of similar labelled cases"]
|
|
store --> responses["Response cache: replay at no cost"]
|
|
store --> imp["ousast improve --memory: alert, verdict, coverage rows"]
|
|
imp --> guard["Train-on-test guard"]
|
|
guard --> rules["Rules M1, M2 to rule-status proposals"]
|
|
rules --> gate{"Unchanged validator and gate"}
|
|
```
|
|
|
|
- **Seed** (`memory.seed`): a Run task whose `openultrasast.io/memory-key` matches stored facts gets
|
|
them and is marked done; a changed pin, candidate set or runner image finds nothing and
|
|
recomputes. The loop's `memory-snapshot` is seeded the same way, because a sandboxed task cannot
|
|
read the store.
|
|
- **Retrieval** (`learn/retrieve.py`): see
|
|
[Memory and detection](memory-and-detection.md#3-retrieval-of-similar-labelled-cases).
|
|
- **`improve --memory`** (`improve/memory.py`): rows from holdout pairs, from the gated manifest's
|
|
own cases and from any `--qualify-population` are dropped before any rule sees them. See
|
|
[Architecture](architecture.md#proposals-from-plane-memory).
|
|
|
|
## Backends: FileStore and the S3 store
|
|
|
|
`OUSAST_MEMORY` selects the backend (`open_store`):
|
|
|
|
| | `FileStore` | `S3Store` (any S3-compatible store; RustFS is the tested server) |
|
|
| --- | --- | --- |
|
|
| Selected by | `file:///path`, or nothing (default `$OUSAST_RESULTS/plane/memory`) | `s3://<bucket>[/<prefix>]`, or `s3://` for the bucket in `S3_BUCKET` |
|
|
| Needs | a local disk; refuses to write below 1 GiB free | the `s3` extra (boto3); `S3_ENDPOINT`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` (optional `S3_REGION`, `AWS_SESSION_TOKEN`) from `.env` or the environment |
|
|
| Filtered reads | read and filter locally | S3 Select pushdown (`SelectObjectContent` over JSON Lines); kind filter by object tag |
|
|
| Object version cited by provenance | the file's content hash | the bucket's object version id |
|
|
| Bucket setup | none | done once by an admin, verified at every start ([RustFS setup](rustfs.md)) |
|
|
|
|
### Verify at startup, no fallback
|
|
|
|
The S3 store never configures its bucket. Each time an `S3Store` is opened for a bucket and
|
|
prefix (once per process), `verify_bucket()` checks, using reads plus one small probe object at
|
|
`<prefix>/_probe/select.jsonl`:
|
|
|
|
1. versioning is `Enabled` (provenance cites object versions);
|
|
2. an enabled lifecycle rule expires `<prefix>/runs/` after a number of days, as a plain prefix
|
|
filter, and no enabled rule expires the whole store;
|
|
3. S3 Select answers a probe query over JSON Lines;
|
|
4. the probe's object tags are readable.
|
|
|
|
Anything missing is named, with the admin command that fixes it, in one `MemoryStoreError`, and
|
|
the store does not start. There is no local or fetch-and-filter fallback: a filtered read that
|
|
S3 Select cannot answer raises a `MemoryStoreError` instead of returning "no rows". The exact
|
|
messages are listed on [RustFS setup](rustfs.md#what-the-store-verifies-at-startup).
|
|
|
|
### Queryable fields per kind
|
|
|
|
RustFS's Select infers an object's JSON schema from its leading rows, so a `where` field missing
|
|
there fails with `EvaluatorBindingDoesNotExist` even when a later row has it, and a column that is
|
|
null in the first 1000 rows breaks Select on the whole object (measured 2026-10-02; `memory.py`,
|
|
`S3Store._select`). The store therefore writes every row with its kind's full set of queryable
|
|
fields (`QUERY_FIELDS`), using `""` where a field does not apply, and refuses a `where` on an
|
|
undeclared field. Existing rows are rewritten once with `ousast plane memory-normalise`.
|