feat(learn): A/B experiment runner -- registered manifests, paired arms, bootstrap and McNemar (decision-engine task 7)

learn/experiments.py: YAML manifests under plane/experiments/<id>.yaml,
digest-checked at register, run and analyse time; `experiment`,
`arm_outcome` and `experiment_result` rows; paired arms in a seeded
repository order with a per-unit coin; arm A replay-only at $0; canary
replicates under a fresh cache salt; repository-cluster paired bootstrap,
exact McNemar, two O'Brien-Fleming looks, adopt/reject/equivalent/
inconclusive. `ousast learn experiment register|units|run|analyse`.
Program: `ProgramSpec.sampling = retrieval_ensemble` (k samples at
temperature 0 over k disjoint retrieved example sets; default unchanged);
`Caller.salt`; `retrieve(exclude=)`. verify: OUSAST_VERIFY_CALLERS=off.
Manifests exp-001-known-callers (plane arms, units pending) and
exp-002-retrieval-ensemble (slice-2 incumbents, ceiling $4) -- not run.
18 scripted tests; module audit regenerated (151 modules).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo

plane/experiments/exp-001-known-callers.yaml

@@ -0,0 +1,34 @@
+# Experiment exp-001 (learned-decision-engine design section 6, Requirement 4.4): the known-callers prompt line.
+# Registered before it runs; `ousast learn experiment register` records this file's sha256, and a changed file is a new
+# experiment. Pre-registered expectation: with about 55 positive units the paired MDE is about 0.15, the observed gap
+# 0.10 needs about 115 -- an inconclusive result is the likely and honest outcome; a follow-up needs new units.
+id: exp-001-known-callers
+hypothesis: >
+  Removing the known-callers line from the verify prompt changes the rate at which verify flags a true site in any
+  pass (plane-increment-2 recorded 17 of 20 declared sites flagged with the line vs 19 without, confounded).
+kind: plane                 # arms are plane Run configurations; the plane's experiment-outcome task writes the arm_outcome rows
+arms:
+  A:
+    label: verify as today (known-callers line on; CALLERS_CAP 8)
+    task_env: {}
+  B:
+    label: verify with the known-callers line off
+    task_env: {OUSAST_VERIFY_CALLERS: "off"}   # everything else identical: model, PER_HUNT, MAX_STEPS, facts, runner image
+unit: candidate
+pairing: paired             # every unit runs in both arms
+cluster: repository         # the resampling unit of the bootstrap
+order: {randomise: per_case, seed: 1}
+metric:
+  primary: "flag_rate[label=1]"                     # flagged in any pass, over the positive units (the 17-vs-19 metric)
+  secondary: ["flag_rate[label=0]", "usd_per_candidate"]   # false flags on fixed-pin negatives; cost
+test: {primary: cluster_bootstrap_paired_diff, secondary: exact_mcnemar, alpha: 0.05, sides: 2, resamples: 10000}
+mde: 0.15
+stopping: {looks: [0.5, 1.0], boundary: obrien_fleming}   # |z| >= 2.797 at half the repositories, 1.977 at the end
+budget_usd: {total: 15, per_arm: 7.5}          # ~$7.7 expected: 552 hunts x $0.014
+families: [injection, access_control, path, untrusted_destination]   # the verify families
+# The development corpus, not v2 (v2 generated the hypothesis): population v1 (11 cases), pmpro, MediaWiki, WP Statistics,
+# and the 27 full-repository pointer pairs of the verify families -- 41 repositories. Candidates are generated model-free
+# (fix-changed functions plus the top engine- and quick-alert functions, up to 8 per pin, on both pins) and frozen here;
+# only units whose arm-A callers line is non-empty are treated. The file does not exist yet: registration records the
+# units as `pending`, and the experiment cannot run until it is frozen, committed and registered again.
+units_file: plane/experiments/exp-001-known-callers.units.jsonl

plane/experiments/exp-002-retrieval-ensemble.yaml

@@ -0,0 +1,41 @@
+# Experiment exp-002 (learned-decision-engine design section 6, engine-level experiments): variance reduction.
+# Registered before it runs; `ousast learn experiment register` records this file's sha256, and a changed file is a new
+# experiment. The incumbents are the slice-2 compiled programs (benchmarks/measurements/2026-10-02-decision-engine-slice-2),
+# whose canary agreement on a re-run was 0.83 (injection, 30 candidates) and 0.83 (access_control).
+id: exp-002-retrieval-ensemble
+hypothesis: >
+  Drawing the k = 5 samples from five disjoint retrieved example sets at temperature 0 (a retrieval ensemble) makes the
+  decision more repeatable than five temperature samples over one example set, without hurting the ranking within a
+  candidate pair, at no higher cost per candidate.
+kind: program
+programs:                   # the incumbents: one compiled program per family (slice 2, record.json `program.id`)
+  injection: b207a0710e0abf053b3b9397b25f9ee1cab4679144865e3a82c1b9be2c7b4982
+  access_control: 745cb1d76bbb09dc1574d333d2118489be62e39886517894d3699fa565b0e8c2
+arms:
+  A:
+    label: today's sampling -- k = 5, sample 1 at temperature 0 and samples 2..5 at 0.7, one retrieved example set
+    spec: {}                 # the compiled program as it is
+    replay_only: false       # replicate 0 replays the evaluation's cached responses at $0; replicate 1 (the canary) is paid
+    repeats: 2
+    budget_usd: 1.0
+  B:
+    label: retrieval ensemble -- k = 5 at temperature 0, each sample over a disjoint retrieved example set
+    spec: {sampling: retrieval_ensemble}   # the one declared difference (ProgramSpec.sampling)
+    repeats: 2
+    budget_usd: 3.0
+unit: candidate
+pairing: paired             # every candidate in both arms, same memory, same outer folds
+cluster: repository
+order: {randomise: per_case, seed: 2}
+metric:
+  primary:
+    - {name: canary_agreement, target: 0.90}   # replicate 0 and 1 give the same verdict; B must reach 0.90
+    - within_pair_auc                           # the vulnerable side of a candidate pair scores above the fixed side
+  secondary: [usd_per_candidate, auc, advisory_recall, advisory_precision]
+test: {primary: cluster_bootstrap_paired_diff, secondary: exact_mcnemar, alpha: 0.05, sides: 2, resamples: 10000}
+mde: 0.10                   # agreement 0.83 -> 0.93 on ~147 paired candidates
+stopping: {looks: [0.5, 1.0], boundary: obrien_fleming}
+budget_usd: {total: 4, per_arm: {A: 1.0, B: 3.0}}   # ~$0.5-0.7 per arm-pass of 147 candidates at k = 5; 3 paid passes
+families: [injection, access_control]
+folds: {kind: outer, k: 5, seed: 0, compile_fraction: 0.25}   # the incumbent evaluation's folds (learn/compile.toml seed)
+units_file: plane/experiments/exp-002-retrieval-ensemble.units.jsonl   # frozen by `ousast learn experiment units`