feat(learn): A/B experiment runner -- registered manifests, paired arms, bootstrap and McNemar (decision-engine task 7)

learn/experiments.py: YAML manifests under plane/experiments/<id>.yaml,
digest-checked at register, run and analyse time; `experiment`,
`arm_outcome` and `experiment_result` rows; paired arms in a seeded
repository order with a per-unit coin; arm A replay-only at $0; canary
replicates under a fresh cache salt; repository-cluster paired bootstrap,
exact McNemar, two O'Brien-Fleming looks, adopt/reject/equivalent/
inconclusive. `ousast learn experiment register|units|run|analyse`.
Program: `ProgramSpec.sampling = retrieval_ensemble` (k samples at
temperature 0 over k disjoint retrieved example sets; default unchanged);
`Caller.salt`; `retrieve(exclude=)`. verify: OUSAST_VERIFY_CALLERS=off.
Manifests exp-001-known-callers (plane arms, units pending) and
exp-002-retrieval-ensemble (slice-2 incumbents, ceiling $4) -- not run.
18 scripted tests; module audit regenerated (151 modules).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-10-02 16:04:33 +02:00
co-authored by Claude Fable 5.1
parent a90e570f48
commit fbc3220bbe
14 changed files with 1681 additions and 35 deletions
+2 -1
View File
@@ -159,9 +159,10 @@ def callers_line(rows: Sequence[Mapping[str, Any]], cap: int = CALLERS_CAP) -> s
def hunt_prompt(path: str, group: Sequence[Candidate], family: str, facts: Mapping[str, Any] | None) -> str:
lines = []
callers_on = os.environ.get("OUSAST_VERIFY_CALLERS", "on") != "off" # arm B of exp-001 (design section 6) turns the line off
for _, function, line in group:
lines.append(f"- function `{function}` around line {line}")
known = callers_line(callers_of(facts, path, function))
known = callers_line(callers_of(facts, path, function)) if callers_on else ""
if known:
lines.append(f" {known}")
listing = "\n".join(lines)