marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

merge(learn): A/B experiment runner; exp-001 and exp-002 registered (decision-engine task 7)

Browse Source 
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-10-02 16:37:57 +02:00
co-authored by Claude Fable 5.1
parent 5aac7eac36 3b91a52788
commit 4c8c3ce1c4
16 changed files with 1706 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/decision-engine.md
+1
View File
@@ -125,6 +125,7 @@ reduction of the score). No result exists; this page carries one only once a rec
| `ousast learn evaluate --program ID` | outer-fold evaluation, cross-fitted calibration, nested operating points |
| `ousast learn curve --program ID` | the metric over memory size on a fixed candidate subset |
| `ousast learn audit-leaks` | how well each signal alone separates the sides of a pair (no model) |
| `ousast learn experiment register\|units\|run\|analyse MANIFEST` | A/B experiments from a pre-registered manifest (`plane/experiments/<id>.yaml`): record its digest, freeze the units (no model), run the arms paired (arm A may replay from the cache at $0), analyse (repository bootstrap, exact McNemar, two looks, the adoption verdict) |
The memory store is the plane's (`OUSAST_MEMORY`; see [ops/ax/README.md](ops/ax/README.md)).
The harvests that produced the plane signals ran as plane Runs generated by
docs/memory.md
+4 -1
View File
@@ -30,7 +30,10 @@ replaces itself. The kinds (`KINDS`) and who writes them:
| `features` | `remember` task, from the `features` task | one feature record per candidate and family |
| `proposal_outcome` | `ousast improve --memory` (not with `--dry-run`), and the loop's `improve` step | what happened to each memory proposal in a round |
| `example` | `ousast learn memory build` | a labelled case: label + feature record + excerpt sha (no rationale) |
| `label`, `decision`, `experiment`, `arm_outcome`, `experiment_result` | reserved for later decision-engine tasks | declared in `KINDS`; not written by the code on main yet |
| `experiment` | `ousast learn experiment register` (task `register`) and `run` (task `run`) | a registered manifest's digests and provenance; a run's counts-only summary. Repo `experiments/<id>`, pin the registering commit |
| `arm_outcome` | `ousast learn experiment run` | one decision per (unit, arm, replicate): score, verdict, usd, order |
| `experiment_result` | `ousast learn experiment analyse --record` | the paired estimates, the looks and the adoption verdict (counts and intervals only) |
| `label`, `decision` | reserved for later decision-engine tasks | declared in `KINDS`; not written by the code on main yet |
**Blobs** (`put_blob`) are content-addressed: a 64-hex name under a fixed prefix. Excerpts are
written by `learn memory build` (`learn/examples.py`), embeddings by `learn memory embed`