marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-18 04:29:27 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(learn): A/B experiment runner -- registered manifests, paired arms, bootstrap and McNemar (decision-engine task 7)

Browse Source 
learn/experiments.py: YAML manifests under plane/experiments/<id>.yaml,
digest-checked at register, run and analyse time; `experiment`,
`arm_outcome` and `experiment_result` rows; paired arms in a seeded
repository order with a per-unit coin; arm A replay-only at $0; canary
replicates under a fresh cache salt; repository-cluster paired bootstrap,
exact McNemar, two O'Brien-Fleming looks, adopt/reject/equivalent/
inconclusive. `ousast learn experiment register|units|run|analyse`.
Program: `ProgramSpec.sampling = retrieval_ensemble` (k samples at
temperature 0 over k disjoint retrieved example sets; default unchanged);
`Caller.salt`; `retrieve(exclude=)`. verify: OUSAST_VERIFY_CALLERS=off.
Manifests exp-001-known-callers (plane arms, units pending) and
exp-002-retrieval-ensemble (slice-2 incumbents, ceiling $4) -- not run.
18 scripted tests; module audit regenerated (151 modules).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-10-02 16:04:33 +02:00
co-authored by Claude Fable 5.1
parent a90e570f48
commit fbc3220bbe
14 changed files with 1681 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/steering/overview.md
+2 -2
View File
@@ -215,5 +215,5 @@ Counted 2026-10-02 from each spec's `tasks.md` checkboxes and `spec.json`.
> and evolutionary proposals come from the plane memory. The HarnessX specs in the table above are history.
As of 2026-09-09: 97 source modules, 90 load-bearing and 7 standalone capabilities; 814 tests.
150 source modules, 145 load-bearing and 5 standalone capabilities, 0 orphaned
(`benchmarks/measurements/2026-09-08-module-audit.json`, regenerated 2026-10-02); 1,964 tests collected.
151 source modules, 146 load-bearing and 5 standalone capabilities, 0 orphaned
(`benchmarks/measurements/2026-09-08-module-audit.json`, regenerated 2026-10-02); 2,009 tests collected.