mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
feat(learn): A/B experiment runner -- registered manifests, paired arms, bootstrap and McNemar (decision-engine task 7)
learn/experiments.py: YAML manifests under plane/experiments/<id>.yaml, digest-checked at register, run and analyse time; `experiment`, `arm_outcome` and `experiment_result` rows; paired arms in a seeded repository order with a per-unit coin; arm A replay-only at $0; canary replicates under a fresh cache salt; repository-cluster paired bootstrap, exact McNemar, two O'Brien-Fleming looks, adopt/reject/equivalent/ inconclusive. `ousast learn experiment register|units|run|analyse`. Program: `ProgramSpec.sampling = retrieval_ensemble` (k samples at temperature 0 over k disjoint retrieved example sets; default unchanged); `Caller.salt`; `retrieve(exclude=)`. verify: OUSAST_VERIFY_CALLERS=off. Manifests exp-001-known-callers (plane arms, units pending) and exp-002-retrieval-ensemble (slice-2 incumbents, ceiling $4) -- not run. 18 scripted tests; module audit regenerated (151 modules). Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
a90e570f48
commit
fbc3220bbe
@@ -79,6 +79,7 @@ v3 check are still ahead.
|
||||
| `ousast learn evaluate --program ID` | outer-fold evaluation, cross-fitted calibration, nested operating points |
|
||||
| `ousast learn curve --program ID` | the metric over memory size on a fixed candidate subset |
|
||||
| `ousast learn audit-leaks` | how well each signal alone separates the sides of a pair (no model) |
|
||||
| `ousast learn experiment register\|units\|run\|analyse MANIFEST` | A/B experiments from a pre-registered manifest (`plane/experiments/<id>.yaml`): record its digest, freeze the units (no model), run the arms paired (arm A may replay from the cache at $0), analyse (repository bootstrap, exact McNemar, two looks, the adoption verdict) |
|
||||
|
||||
The memory store is the plane's (`OUSAST_MEMORY`; see [ops/ax/README.md](ops/ax/README.md)).
|
||||
The harvest that produced the plane signals runs as plane Runs generated by
|
||||
|
||||
Reference in New Issue
Block a user