bench(learn): injection first slice -- compiled AI classifier, out-of-repo evaluation (decision-engine 6.2, 6.6)

6.2: the 724 static examples' 616 distinct excerpts embedded with openai/text-embedding-3-small via
OpenRouter, 198,819 tokens, $0.0040 metered (ceiling $1).

6.6, family injection only, memory = every family (at most 2 contrast examples per prompt), input
profile v1: smoke fixed k = 5 (repeat prefix-cache hit share 0.96); one compile (program c1237a49,
proposed instruction 5, balanced Brier 0.128 on 13 C_val candidates vs 0.199 baseline); outer grouped
5-fold, 110 out-of-repository candidates in 29 groups: ADVISORY recall 0.90 [0.82, 0.97], precision
0.54 [0.44, 0.65], AUC 0.74; BLOCK not offered (ECE 0.064 > 0.05; no fold reached a Wilson lower bound
of 0.95); canary re-ask agreement 0.83. $0.91 metered at list price; DeepSeek balance $18.20 -> $17.90.

Instrument fixes found on the way, each with a test:
- memory build: pair sides join the record stored under their harvest unit pin (--units), and only the
  built profile's records are read (static and plane shared a key; 85 -> 724 examples);
- retrieval: stage 1 per side (family / contrast) and at most 2 contrast examples, never padded;
- program: the JSON answer contract is part of the signature, appended to proposed instructions
  (DeepSeek JSON mode refused a prompt without "json");
- compile: candidate_families for a one-family compile; compile.toml k = 5 from the smoke rule.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo

.kiro/specs/learned-decision-engine/design.md

@@ -529,7 +529,8 @@ explicit rather than imputed:
   block). Prior features (`FeatureSpec.prior`) are dropped when `priors = off`. Weights are a compile-time
   setting only if an A/B shows a gain; the default is uniform.
 - Candidates in the eligible pool (after the boundary filter) are ranked by `D`, ties by `sha256(example_id +
-  seed)`; the first `n1 = 40` go to stage 2.
+  seed)`; the first `n1 = 40` of the candidate's family and the first `n1` of the other families go to stage 2
+  (per side since 2026-10-01, so a minority family reaches stage 2; Risks, "First paid slice").
 
 **Stage 2: code-embedding re-rank.** Cosine similarity of the candidate's excerpt embedding with each stage-1
 example's. The re-rank score is `r = lam * (1 - D) + (1 - lam) * cos`, with `lam` a compiled setting in {0, 0.5}
@@ -1205,6 +1206,27 @@ start only after the maintainer confirms their budget against the balance read j
   Pre-registering that and recording "inconclusive" is the intended behaviour. The failure mode to avoid is
   extending an experiment after looking.
 
+- **First paid slice, 2026-10-01 (tasks 6.2, 6.6; injection only).** Four instrument defects surfaced before any
+  number was believed, each fixed with a test:
+  - *The memory build joined no pair.* The harvest stores a pair side's record under its unit pin (the blob sha1 of
+    the side's excerpt), not under `label_pin`; `ousast learn memory build --units <harvest units.json>` joins them.
+    And a candidate's `static` and `plane` records share a key, so reading both let one overwrite the other and
+    silently drop about half the examples; the build now reads its own profile only. 85 -> 724 examples.
+  - *Contrast was unbounded.* The balance rule padded a short pick with other families (up to 5 of 6), and stage 1
+    took the 40 nearest of all families, so a minority family often never reached stage 2. Now stage 1 takes the
+    `n1` nearest of the candidate's family and the `n1` nearest others, and at most 2 contrast examples are shown
+    whenever the pool holds the family (maintainer: "allow 2 contrast"); never padded.
+  - *Proposed instructions dropped the answer contract*, and DeepSeek's JSON mode refuses a prompt without the word
+    "json" (HTTP 400, the first compile stopped there). The contract (`ANSWER_FORMAT`) is now part of the signature
+    and appended to any instruction that lacks it; the baseline prefix is byte-identical.
+  - The adjudication source fails the 10% excerpt gate: 9 of its 43 candidates are anonymous functions (`<lambda>N`,
+    `<module>`) that `function_span` cannot find by name; the clones were read. Open.
+  The memory is small: 724 static examples (only harvested candidates have feature records; the 97 advisory-fix pairs
+  have none yet), injection 140 (75/65), so `C_val` had 13 injection candidates and the compile metric is noisy.
+  The measured ranking signal is real but weak at the recall target: AUC 0.74 out of repository, ADVISORY at recall
+  0.90 has precision 0.54 (prevalence 0.53); the majority `vulnerable` verdict alone has precision 0.86, recall 0.43.
+  Canary re-ask agreement at temperature 0 was 0.83, below the 0.9 the adoption check requires.
+
 ## Maintainer decisions at design approval (2026-09-30)
 
 - Design approved as written.

.kiro/specs/learned-decision-engine/tasks.md

@@ -54,6 +54,8 @@ a model or embedding call.
       `plane/budget.py`, usage kept by `parse_embedding_response`; cache `embeddings/<model>/<excerpt_sha>.json`;
       `ousast learn memory embed`; scripted-client tests.
     - **(budget go-ahead)** embedding the store: ~5.0 M tokens, ~$0.10, ceiling $1 (OpenRouter account).
+      Done 2026-10-01: 616 distinct excerpts of the 724 static examples, 198,819 tokens, 20 calls, $0.0040 metered
+      (`benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`).
     - _Requirements: 3.2_
   - [x] 6.3 Retrieval and the evaluation boundary (no model)
     - `learn/retrieve.py`: per-instrument Gower distance with explicit missing, `n1 = 40`, embedding re-rank
@@ -82,13 +84,21 @@ a model or embedding call.
     - Tests: `test_learn_compile.py` (`test_reproducible_from_cache` on both backends), `test_learn_calibrate.py`,
       `test_learn_decide.py`.
     - _Requirements: 3.3, 3.5, 5.2, 7.3, 7.4_
-  - [ ] 6.6 First paid slice **(budget go-ahead: ~$5 expected, ceiling $7)**
+  - [x] 6.6 First paid slice **(budget go-ahead: ~$5 expected, ceiling $7)**
     - Smoke run on 20 candidates (~$0.10): real token counts, prefix-cache hit share, parse rate; it fixes `k`
       (5 if samples 2..k hit the cache for >= 80% of their prompt tokens, else 1) before anything else is spent.
     - One compile per profile with data (`static` first; ~$2.2, ceiling $3).
     - Outer-fold evaluation, priors off, evaluable families only (675 candidates: ~$2.0 at k = 1, ~$3.0 at k = 5),
       with the 30-candidate canary set; report under `benchmarks/measurements/<date>-decision-engine-v1/` (counts
       and intervals only, no identities).
+    - Done 2026-10-01 for family **injection only** (the maintainer's scope for this slice; memory holds every family,
+      at most 2 contrast examples per prompt): smoke fixed `k = 5` (repeat hit share 0.96); program `c1237a49...`
+      (proposed instruction 5, balanced Brier 0.128 on 13 `C_val` candidates vs 0.199 baseline); 110 out-of-repository
+      candidates in 29 groups: ADVISORY recall 0.90 [0.82, 0.97], precision 0.54 [0.44, 0.65]; AUC of `s` 0.74 [0.65,
+      0.85]; BLOCK not offered (ECE 0.064 > 0.05, and no fold reached a Wilson lower bound of 0.95); canary agreement
+      0.83; $0.91 metered at list price (provider balance $18.20 -> $17.90). Record:
+      `benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`. The other evaluable families
+      are not evaluated yet.
     - _Requirements: 3.1-3.5, 5.2, 7.3, 7.4_
   - [ ] 6.7 Second paid slice **(budget go-ahead: ~$7 expected, ceiling $10; needs a top-up)**
     - Leave-one-source-out (~$1.4-2.1), leave-one-framework-out where a framework reaches 10 groups ($0 today),

benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json

@@ -0,0 +1,524 @@
+{
+ "calibration": {
+  "ece": 0.0636,
+  "holds": false,
+  "intercept": -0.0179,
+  "intercept_ci": [
+   -0.5344,
+   0.4288
+  ],
+  "n": 110,
+  "offered": {
+   "advisory": {
+    "offered": true,
+    "reason": null
+   },
+   "block": {
+    "offered": false,
+    "reason": "calibration"
+   }
+  },
+  "reliability": [
+   {
+    "mean_p": 0.2891,
+    "n": 22,
+    "positive_rate": 0.3636
+   },
+   {
+    "mean_p": 0.4015,
+    "n": 22,
+    "positive_rate": 0.3182
+   },
+   {
+    "mean_p": 0.4867,
+    "n": 22,
+    "positive_rate": 0.4545
+   },
+   {
+    "mean_p": 0.6075,
+    "n": 22,
+    "positive_rate": 0.6818
+   },
+   {
+    "mean_p": 0.8718,
+    "n": 22,
+    "positive_rate": 0.8182
+   }
+  ],
+  "rule": "holds when the slope CI contains 1, the intercept CI contains 0 and ECE <= 0.05",
+  "slope": 0.9155,
+  "slope_ci": [
+   0.5808,
+   1.6419
+  ]
+ },
+ "canary": {
+  "agreement": 0.8333333333333334,
+  "canary": 30,
+  "mean_abs_s_diff": 0.03833333333333333,
+  "reparsed": 30,
+  "set_digest": "f0888f459e9daa9175c8d3ba8df75631f66f52598d38904cda8ef6ed9c22de15",
+  "spend": null
+ },
+ "compile_metric": {
+  "baseline_instruction": 0.198998,
+  "c_boot_candidates": 17,
+  "c_val_candidates": 13,
+  "chosen": 0.127636,
+  "configurations": [
+   {
+    "demo_set": 0,
+    "instruction": 0,
+    "prompt_tokens": 2271,
+    "score": 0.198998,
+    "secondary": {
+     "recall_at_precision_0.90": 0.571429,
+     "unsure_share": 0.230769
+    }
+   },
+   {
+    "demo_set": 0,
+    "instruction": 1,
+    "prompt_tokens": 2248,
+    "score": 0.170483,
+    "secondary": {
+     "recall_at_precision_0.90": 0.571429,
+     "unsure_share": 0.307692
+    }
+   },
+   {
+    "demo_set": 0,
+    "instruction": 2,
+    "prompt_tokens": 2239,
+    "score": 0.145871,
+    "secondary": {
+     "recall_at_precision_0.90": 0.714286,
+     "unsure_share": 0.307692
+    }
+   },
+   {
+    "demo_set": 0,
+    "instruction": 3,
+    "prompt_tokens": 2236,
+    "score": 0.17766,
+    "secondary": {
+     "recall_at_precision_0.90": 0.428571,
+     "unsure_share": 0.692308
+    }
+   },
+   {
+    "demo_set": 0,
+    "instruction": 4,
+    "prompt_tokens": 2222,
+    "score": 0.19376,
+    "secondary": {
+     "recall_at_precision_0.90": 0.571429,
+     "unsure_share": 0.692308
+    }
+   },
+   {
+    "demo_set": 0,
+    "instruction": 5,
+    "prompt_tokens": 2238,
+    "score": 0.127636,
+    "secondary": {
+     "recall_at_precision_0.90": 0.714286,
+     "unsure_share": 0.0
+    }
+   },
+   {
+    "demo_set": 0,
+    "instruction": 6,
+    "prompt_tokens": 2238,
+    "score": 0.156588,
+    "secondary": {
+     "recall_at_precision_0.90": 0.571429,
+     "unsure_share": 0.384615
+    }
+   },
+   {
+    "demo_set": 1,
+    "instruction": 5,
+    "prompt_tokens": 2137,
+    "score": 0.169213,
+    "secondary": {
+     "recall_at_precision_0.90": 0.571429,
+     "unsure_share": 0.0
+    }
+   },
+   {
+    "demo_set": 2,
+    "instruction": 5,
+    "prompt_tokens": 2118,
+    "score": 0.129923,
+    "secondary": {
+     "recall_at_precision_0.90": 0.714286,
+     "unsure_share": 0.076923
+    }
+   },
+   {
+    "demo_set": 3,
+    "instruction": 5,
+    "prompt_tokens": 2455,
+    "score": 0.128863,
+    "secondary": {
+     "recall_at_precision_0.90": 0.857143,
+     "unsure_share": 0.0
+    }
+   }
+  ],
+  "name": "balanced_brier_v1 (lower is better)",
+  "on": "C_val, injection candidates"
+ },
+ "created": "2026-10-01",
+ "embeddings_6_2": {
+  "calls": 20,
+  "ceiling_usd": 1.0,
+  "embedded": 616,
+  "examples_with_vector": 724,
+  "excerpts": 616,
+  "metered_usd": 0.003976,
+  "model": "openai/text-embedding-3-small",
+  "route": "OpenRouter",
+  "tokens": 198819
+ },
+ "evaluation": {
+  "advisory": {
+   "precision": {
+    "ci": [
+     0.4444444444444444,
+     0.6477272727272727
+    ],
+    "value": 0.5416666666666666
+   },
+   "recall": {
+    "ci": [
+     0.8245614035087719,
+     0.967741935483871
+    ],
+    "value": 0.896551724137931
+   }
+  },
+  "advisory_threshold_by_fold": {
+   "outer-0": 0.2933,
+   "outer-1": 0.3203,
+   "outer-2": 0.2933,
+   "outer-3": 0.3479,
+   "outer-4": 0.3432
+  },
+  "auc_raw_score": {
+   "ci": [
+    0.6451,
+    0.8527
+   ],
+   "value": 0.7382
+  },
+  "block": {
+   "offered": false,
+   "reason": "calibration"
+  },
+  "block_reason_by_fold": {
+   "outer-0": "precision_unreachable",
+   "outer-1": "precision_unreachable",
+   "outer-2": "precision_unreachable",
+   "outer-3": "precision_unreachable",
+   "outer-4": "precision_unreachable"
+  },
+  "by_language": {
+   "javascript": {
+    "advisory": {
+     "precision": {
+      "ci": [
+       0.3333333333333333,
+       0.6226415094339622
+      ],
+      "value": 0.5
+     },
+     "recall": {
+      "ci": [
+       0.6153846153846154,
+       0.9375
+      ],
+      "value": 0.8095238095238095
+     }
+    },
+    "block": {
+     "recall": {
+      "ci": [
+       0.0,
+       0.0
+      ],
+      "value": 0.0
+     }
+    },
+    "candidates": 41,
+    "groups": 15,
+    "status": "pooled"
+   },
+   "php": {
+    "status": "insufficient data"
+   },
+   "python": {
+    "status": "insufficient data"
+   },
+   "typescript": {
+    "status": "insufficient data"
+   }
+  },
+  "candidates": 110,
+  "compile_groups": 25,
+  "folds": {
+   "outer-0": {
+    "groups": 15,
+    "status": "ok"
+   },
+   "outer-1": {
+    "groups": 15,
+    "status": "ok"
+   },
+   "outer-2": {
+    "groups": 15,
+    "status": "ok"
+   },
+   "outer-3": {
+    "groups": 14,
+    "status": "ok"
+   },
+   "outer-4": {
+    "groups": 14,
+    "status": "ok"
+   }
+  },
+  "groups": 29,
+  "intervals": "repository-cluster bootstrap 95%",
+  "majority_vulnerable_verdict": {
+   "precision": {
+    "ci": [
+     0.7241,
+     1.0
+    ],
+    "value": 0.8621
+   },
+   "recall": {
+    "ci": [
+     0.2917,
+     0.6222
+    ],
+    "value": 0.431
+   }
+  },
+  "parse_failed_share": 0.0,
+  "positives": 58,
+  "status": "ok",
+  "undecided": 0,
+  "verdict_by_label": {
+   "0:not_vulnerable": 42,
+   "0:unsure": 6,
+   "0:vulnerable": 4,
+   "1:not_vulnerable": 27,
+   "1:unsure": 6,
+   "1:vulnerable": 25
+  }
+ },
+ "instruments": {
+  "balance": {
+   "ok": 110
+  },
+  "candidates": 110,
+  "candidates_with_vector": 110,
+  "contrast_per_candidate": {
+   "0": 24,
+   "1": 24,
+   "2": 62
+  },
+  "demos_per_candidate": {
+   "3": 110
+  },
+  "distinct_s": 52,
+  "examples_per_candidate": {
+   "6": 110
+  },
+  "excerpt_bytes_min_mean_max": [
+   38,
+   1166,
+   1975
+  ],
+  "modes": {
+   "signals+embeddings": 110
+  },
+  "near_duplicates_dropped": 0,
+  "no_excerpt": 0,
+  "parse_failed": 0,
+  "pool_mean": 636.4,
+  "prompt_chars_mean": 25480
+ },
+ "labels": {
+  "same_counts_as": "2026-10-01-decision-engine-labels-advisory-fixes/labels.json",
+  "sha256": "92f1856c90aaebfaaf0a738c3e4e3a5a730135391b2ea5a3ca2b5987db7430ee"
+ },
+ "memory_build": {
+  "by_source": {
+   "adjudications": {
+    "absent_fixed_side": 0,
+    "examples": 34,
+    "labels": 43,
+    "no_excerpt": 9,
+    "no_record": 0,
+    "recorded": 43,
+    "unread": 0
+   },
+   "dev-php": {
+    "absent_fixed_side": 0,
+    "examples": 3,
+    "labels": 3,
+    "no_excerpt": 0,
+    "no_record": 0,
+    "recorded": 3,
+    "unread": 0
+   },
+   "pairs": {
+    "absent_fixed_side": 0,
+    "examples": 560,
+    "labels": 1544,
+    "no_excerpt": 0,
+    "no_record": 984,
+    "recorded": 560,
+    "unread": 0
+   },
+   "population-v1": {
+    "absent_fixed_side": 0,
+    "examples": 85,
+    "labels": 170,
+    "no_excerpt": 0,
+    "no_record": 85,
+    "recorded": 85,
+    "unread": 0
+   },
+   "population-v2": {
+    "absent_fixed_side": 0,
+    "examples": 42,
+    "labels": 86,
+    "no_excerpt": 1,
+    "no_record": 43,
+    "recorded": 43,
+    "unread": 0
+   }
+  },
+  "examples": 724,
+  "examples_by_family_label": {
+   "access_control:1": 13,
+   "config_secrets:0": 13,
+   "config_secrets:1": 18,
+   "deserialization:1": 20,
+   "injection:0": 65,
+   "injection:1": 75,
+   "memory:0": 171,
+   "memory:1": 171,
+   "output_encoding:0": 40,
+   "output_encoding:1": 56,
+   "path:0": 1,
+   "path:1": 13,
+   "unknown:0": 2,
+   "unknown:1": 2,
+   "untrusted_destination:0": 19,
+   "untrusted_destination:1": 45
+  },
+  "examples_by_source_label": {
+   "adjudications:0": 32,
+   "adjudications:1": 2,
+   "dev-php:1": 3,
+   "pairs:0": 226,
+   "pairs:1": 334,
+   "population-v1:0": 35,
+   "population-v1:1": 50,
+   "population-v2:0": 18,
+   "population-v2:1": 24
+  },
+  "excerpt_bytes_total": 817773,
+  "excerpts_read": 724,
+  "failing_reason": "adjudications: 9 of 43 recorded candidates are anonymous functions (<lambda>N, <module>) that function_span cannot locate by name; the clones were read (unread 0). Exit 2 recorded, rows kept.",
+  "failing_sources": [
+   "adjudications"
+  ],
+  "profile": "static",
+  "skipped": {
+   "conditional label (condition not evaluated)": 4743
+  }
+ },
+ "note": "COUNTS ONLY. Decision-engine tasks 6.2 and 6.6, family INJECTION only: memory = every family's static examples (retrieval may show up to 2 contrast examples of other families), candidates = injection. Input profile v1 (engine and facts.function_lines withheld), priors off. Out-of-repository: compile split (25% of groups) never evaluated; outer grouped 5-fold over the rest; Platt map cross-fitted on the other folds; operating points nested.",
+ "program": {
+  "bootstrap": {
+   "candidates": 17,
+   "kept": 11,
+   "rejected": {
+    "not confident": 1,
+    "rationale names an identity": 1,
+    "wrong or unsure": 4
+   }
+  },
+  "classify": {
+   "confidence": "vote_mean",
+   "inputs": "v1",
+   "k": 5,
+   "max_output_tokens": 300,
+   "model": "deepseek-flash",
+   "model_params_digest": "f31055fe086f97cf687b9f2198637556ec8b9eee5624f8004a178c502166735a",
+   "temperature": [
+    0.0,
+    0.7
+   ],
+   "withheld": [
+    "engine",
+    "facts.function_lines"
+   ]
+  },
+  "data": {
+   "examples": 724,
+   "groups": 98,
+   "sources": [
+    "adjudications",
+    "dev-php",
+    "pairs",
+    "population-v1",
+    "population-v2"
+   ]
+  },
+  "demos": 3,
+  "id": "c1237a49e19d2477aa8521b182271fa9d6eae165d391aa0f2a4c4c6d6a825a8b",
+  "instruction_source": "proposed:5"
+ },
+ "replay_check": "evaluate re-run replay-only on the committed code: 550 responses replayed, $0, identical overall and calibration",
+ "smoke": {
+  "candidates": 17,
+  "k5_repeat_hit_share_measured": 0.96,
+  "k_fixed": 5,
+  "k_rule": "k = 5 when samples 2..k hit the prefix cache for >= 0.8 of their prompt tokens",
+  "parse_failed": 0
+ },
+ "spend": {
+  "balance_note": "the provider charged ~1/3 of the list-price meter (balance moved $0.30 for $0.91 metered): DeepSeek's off-peak discount window, presumably; the meter prices at list",
+  "ceilings_usd": {
+   "6_2": 1.0,
+   "6_6": 7.0
+  },
+  "cost_per_candidate_evaluate_usd": 0.003215,
+  "deepseek_balance_usd": {
+   "after": 17.9,
+   "before": 18.2
+  },
+  "deepseek_tokens": {
+   "completion_tokens": 155480,
+   "prompt_cache_hit_tokens": 8601840,
+   "prompt_tokens": 9936337
+  },
+  "metered_at_list_price_usd": {
+   "canary": 0.010158,
+   "compile": 0.429194,
+   "compile_first_attempt_failed_400": 0.049047,
+   "deepseek_total_6_6": 0.912838,
+   "embed_6_2_openrouter": 0.003976,
+   "evaluate": 0.353626,
+   "smoke_deepseek": 0.070812
+  }
+ }
+}

src/openultrasast/learn/cli.py

@@ -25,6 +25,7 @@ def add_commands(learn_sub: Any) -> None:
     build.add_argument("--cache", type=Path, help="clones and checkouts (default ~/.cache/openultrasast)")
     build.add_argument("--memory", help="the store (default OUSAST_MEMORY, then results_root()/memory)")
     build.add_argument("--profile", choices=("static", "plane"), default="static")
+    build.add_argument("--units", type=Path, help="the harvest units.json: the pin each pair side's record is stored under")
     build.add_argument("--dry-run", action="store_true", help="count, write nothing")
     embed = memory_sub.add_parser("embed", help="embed the stored excerpts (OpenRouter, metered; cached by excerpt sha)")
     embed.add_argument("--memory", help="the store (default OUSAST_MEMORY, then results_root()/memory)")
@@ -209,19 +210,21 @@ def _memory_embed(args: argparse.Namespace) -> int:
 def _memory_build(args: argparse.Namespace) -> int:
     """Exit 2 on a refused label (``origin: user``, a source outside the list) or a source with >10% unread excerpts."""
     from ..plane.memory import open_store
-    from .examples import ExampleBuildError, SideReader, build_examples, read_jsonl, records_by_key
+    from .examples import ExampleBuildError, SideReader, build_examples, pair_pins, read_jsonl, records_by_key
     from .labels import DEFAULT_CACHE, DEFAULT_SOURCES, load_sources
 
     sources = load_sources(args.sources or DEFAULT_SOURCES)
     labels = read_jsonl(args.labels)
     store = None if args.dry_run else open_store(args.memory)
-    rows = [row for path in args.features for row in read_jsonl(path)]
+    rows = [row for path in args.features for row in read_jsonl(path) if row.get("profile", args.profile) == args.profile]
     if store is not None:
-        rows += [r.row for r in store.rows(kind="features")]
+        # one profile only: the static and plane records of a candidate share a key, and the other would overwrite it
+        rows += [r.row for r in store.rows(kind="features") if r.row.get("profile") == args.profile]
     reader = SideReader(args.root, args.cache or DEFAULT_CACHE, sources, labels)
     try:
         build = build_examples(
             labels, records_by_key(rows), reader, sources=sources, store=store, profile=args.profile, license_of=reader.license,
+            pins=pair_pins(json.loads(args.units.read_text(encoding="utf-8"))) if args.units is not None else None,
         )  # fmt: skip
     except ExampleBuildError as exc:
         print(f"learn memory build: {exc}", file=sys.stderr)

src/openultrasast/learn/compile.py

@@ -354,12 +354,14 @@ def compile_program(
     vectors: Mapping[str, Sequence[float]] | None = None,
     identities: Callable[[Example], Iterable[str]] | None = None,
     families: Sequence[str] = (),
+    candidate_families: Sequence[str] = (),
     store: MemoryStore | None = None,
     created: str = "",
     code_commit: str = "",
 ) -> CompileResult:
     """Compile one profile's program from ``memory`` (module docstring). Deterministic given the memory, the spec and
-    the response cache."""
+    the response cache. ``candidate_families`` limits the bootstrap and validation candidates to those families (a
+    one-family slice); retrieval still draws on every family of the compile split, for contrast examples."""
     seed = spec.seed
     pool = [e for e in memory if e.profile == profile]
     split: CompileSplit = compile_split(pool, fraction=float(spec["split"].get("compile_fraction", 0.25)),
@@ -368,8 +370,9 @@ def compile_program(
     assert_disjoint(split, evaluation)
     compile_memory = [e for e in pool if e.group in split.groups]
     fold = Fold("compile", "compile", frozenset(), phase="compile")
-    boot = _order([e for e in compile_memory if e.group in split.boot], seed, "boot")[: int(spec["bootstrap"].get("max_candidates", 150))]
+    asked = [e for e in compile_memory if not candidate_families or e.family in candidate_families]
-    val = _order([e for e in compile_memory if e.group in split.val], seed, "val")[: int(spec["search"].get("val_candidates", 60))]
+    boot = _order([e for e in asked if e.group in split.boot], seed, "boot")[: int(spec["bootstrap"].get("max_candidates", 150))]
+    val = _order([e for e in asked if e.group in split.val], seed, "val")[: int(spec["search"].get("val_candidates", 60))]
     if not boot or not val:
         raise CompileError(f"compile split too small: {len(boot)} bootstrap and {len(val)} validation candidates")
     retrieval, classify = spec["retrieval"], spec["classify"]

src/openultrasast/learn/compile.toml

@@ -29,7 +29,9 @@ val_candidates = 60
 
 [classify]
 model = "deepseek-flash"
-k = 1                        # k and lam are A/B variants (design section 6), not searched here
+k = 5                        # fixed by the smoke run before any compile spend (task 6.6): samples 2..5 hit the
+                             # provider prefix cache for 0.96 of their prompt tokens (>= 0.8), 2026-10-01. k and lam
+                             # remain A/B variants (design section 6), not searched here
 max_output_tokens = 300
 inputs = "v1"                # the input profile: the engine instrument is withheld from prompts and retrieval (2026-10-01)
 

src/openultrasast/learn/examples.py

@@ -22,7 +22,9 @@ Memory examples carry **no rationale** (code, signals, roles, family and label o
   ``fix_function`` (a different handler of the same snapshot).
 
 Pair labels have no commit pin (the corpus holds excerpts); their envelope pin is :func:`label_pin`, a stable
-40-hex digest of the pair and its side, so the vulnerable and the fixed side stay distinct rows.
+40-hex digest of the pair and its side, so the vulnerable and the fixed side stay distinct rows. The harvest keyed a
+pair side's feature record by its unit's pin instead (the blob sha1 of the side's excerpt, ``plane/harvest.py``):
+:func:`pair_pins` reads that mapping from the harvest ``units.json`` and the build uses it when given.
 """
 
 from __future__ import annotations
@@ -62,6 +64,11 @@ def label_pin(label: Mapping[str, Any]) -> str:
     return hashlib.sha1(subject.encode("utf-8")).hexdigest()  # noqa: S324 -- a name, not a security digest
 
 
+def pair_pins(units: Mapping[str, Mapping[str, Any]]) -> dict[tuple[str, str], str]:
+    """Harvest ``units.json`` -> ``(case reference, side) -> pin`` for the pair units (the pin their records carry)."""
+    return {(str(u["ref"]), str(u["side"])): str(u["pin"]) for u in units.values() if u.get("source") == "pairs"}
+
+
 def record_key(repo: str, pin: str, candidate: str, family: str, unit: str) -> tuple[str, str, str, str, str]:
     return repo_key(repo), pin, candidate, family, unit
 
@@ -127,9 +134,11 @@ def build_examples(
     license_of: Callable[[Mapping[str, Any]], str] | None = None,
     function_of: Callable[[Mapping[str, Any], str], str | None] | None = None,
     bounds: Bounds = EXAMPLE,
+    pins: Mapping[tuple[str, str], str] | None = None,
 ) -> MemoryBuild:
     """Example rows for the verified labels that have a feature record (``records`` by :func:`record_key`) and an
-    excerpt; with ``store`` the excerpt blobs and the rows are written. Delta units append the base->head diff."""
+    excerpt; with ``store`` the excerpt blobs and the rows are written. Delta units append the base->head diff.
+    ``pins`` (:func:`pair_pins`) gives a pair side the pin its harvest record was stored under."""
     rows = [_as_dict(label) for label in labels]
     allowed = {s.id for s in sources.sources} | {d for d, kind in DERIVED_SOURCES.items() if any(s.kind == kind for s in sources.sources)}
     for row in rows:
@@ -145,7 +154,9 @@ def build_examples(
             continue
         count = build.sources.setdefault(str(label["source"]), SourceCount())
         count.labels += 1
-        pin = label_pin(label)
+        side = (str(label.get("source_ref")), str(label.get("pin_role") or "vulnerable"))
+        pin = (pins or {}).get(side) if label.get("source") == "pairs" else None
+        pin = pin or label_pin(label)
         candidate, family, unit = str(label["candidate"]), str(label["family"]), str(label["unit"])
         record = records.get(record_key(str(label["repo"]), pin, candidate, family, unit))
         if record is None or record.get("profile") != profile:
@@ -307,5 +318,5 @@ def records_by_key(rows: Iterable[Mapping[str, Any]]) -> dict[tuple[str, str, st
 
 __all__ = [
     "EXAMPLE_KIND", "NO_EXCERPT_LIMIT", "Example", "ExampleBuildError", "MemoryBuild", "SideReader", "build_examples",
-    "label_pin", "label_set_digest", "load_examples", "read_jsonl", "record_key", "records_by_key",
+    "label_pin", "label_set_digest", "load_examples", "pair_pins", "read_jsonl", "record_key", "records_by_key",
 ]  # fmt: skip

src/openultrasast/learn/program.py

@@ -87,16 +87,21 @@ SIGNATURE = Signature(
     )
 )
 
-BASELINE_INSTRUCTION = (
+# The answer contract is part of the signature, not of the searched instruction: a proposed instruction that leaves it
-    "You decide whether a function contains an exploitable security vulnerability of the given family.\n"
+# out still gets it (``render_prefix``), so every prompt asks for JSON (DeepSeek's JSON mode refuses a prompt that
-    "You see the function's code (numbered lines), the signals static instruments emitted for it, and the roles\n"
+# never says "json") and every answer can parse.
-    "inferred for its calls. Signals are evidence, not verdicts: rules over-report, and a missing instrument is not a\n"
+ANSWER_FORMAT = (
-    "negative. Similar labelled functions from other repositories follow as examples; reason about this code.\n"
     'Answer with one JSON object: {"verdict": "vulnerable" | "not_vulnerable" | "unsure", "family": <family or\n'
     '"none">, "confidence": <probability in [0, 1] that your verdict is right>, "rationale": <at most 3 sentences>,\n'
     '"cited_lines": [<line numbers of the code that support the verdict>]}. Cite at least one line. Say "unsure"\n'
     "when the code does not decide it."
 )
+BASELINE_INSTRUCTION = (
+    "You decide whether a function contains an exploitable security vulnerability of the given family.\n"
+    "You see the function's code (numbered lines), the signals static instruments emitted for it, and the roles\n"
+    "inferred for its calls. Signals are evidence, not verdicts: rules over-report, and a missing instrument is not a\n"
+    "negative. Similar labelled functions from other repositories follow as examples; reason about this code.\n" + ANSWER_FORMAT
+)
 
 
 # --- rendering ---------------------------------------------------------------------------------------------------------
@@ -232,7 +237,8 @@ ExcerptText = Callable[[str], str | None]
 
 def render_prefix(spec: ProgramSpec, demos: Sequence[Demo], examples: Mapping[str, Example], excerpt_text: ExcerptText) -> str:
     """The instruction and the demonstrations: identical for every candidate of one program and fold."""
-    parts = [spec.instruction.strip()]
+    instruction = spec.instruction.strip()
+    parts = [instruction if ANSWER_FORMAT in instruction else f"{instruction}\n{ANSWER_FORMAT}"]
     for number, demo in enumerate(demos, 1):
         example = examples[demo.example_id]
         code = excerpt_text(demo.excerpt_sha) or ""
@@ -481,7 +487,8 @@ class Program:
 
 
 __all__ = [
-    "BASELINE_INSTRUCTION", "PARSE_FAILED", "SIGNATURE", "UNSURE", "VERDICTS", "Answer", "Caller", "Candidate", "Decision", "Demo",
+    "ANSWER_FORMAT", "BASELINE_INSTRUCTION", "PARSE_FAILED", "SIGNATURE", "UNSURE", "VERDICTS", "Answer", "Caller", "Candidate",
+    "Decision", "Demo",
     "Program", "ProgramSpec", "Prompt", "ReplayMiss", "aggregate", "classify", "label_word", "line_numbers",
     "params_digest", "parse_answer", "render", "render_prefix", "render_roles", "render_signals", "request_key",
 ]  # fmt: skip

src/openultrasast/learn/retrieve.py

@@ -11,12 +11,13 @@ profile, computed per instrument block so that missing is explicit, never impute
 - ``D`` is the uniform mean over the profile's blocks; prior features are dropped with ``priors = "off"``, and the blocks
   of instruments the input profile withholds (``inputs``, default ``v1``: the engine) are left out entirely.
 
-Ties break by ``sha256(example id + seed)``; the first ``n1 = 40`` go to stage 2, a cosine re-rank of the code
+Ties break by ``sha256(example id + seed)``; the first ``n1 = 40`` of the candidate's family and the first ``n1``
+of the other families go to stage 2, a cosine re-rank of the code
 embeddings, ``r = lam (1 - D) + (1 - lam) cos``. Without the target's vector the whole retrieval is
 ``signals_only``; an example without a vector scores ``cos = 1 - D`` (its signal similarity, so it keeps its
 stage-1 standing). The **balance rule** then takes ``k_ret = 6``: at most ``k/2`` per label, at most 2 per
-repository group, at most 2 from other families while the candidate's family has examples left; ``short`` when
+repository group, at most 2 contrast examples from other families whenever the pool holds the candidate's family
-the pool cannot fill it.
+(never padded with more); ``short`` when the pool cannot fill it.
 
 **The boundary.** :func:`eligible` is the only way an example enters a prompt -- retrieved examples and compiled
 demonstrations alike: not the target's group, not a group the fold evaluates, not the held-out source, not the
@@ -43,6 +44,7 @@ LAM = 0.5
 NEAR_DUPLICATE_COS = 0.98
 UNBOUNDED_CAP = 10.0
 PER_GROUP = 2
+CONTRAST = 2  # contrast examples from other families, at most, while the pool holds the target's family
 
 
 class BoundaryViolation(AssertionError):
@@ -153,35 +155,26 @@ def _tie(example_id: str, seed: int) -> str:
 
 
 def balance(ranked: Sequence[Example], family: str, k: int = K_RET) -> tuple[list[Example], str]:
-    """Take up to ``k`` in rank order: <= k/2 per label, <= 2 per group, <= 2 other families while the target's family
+    """Take up to ``k`` in rank order: <= k/2 per label, <= 2 per group, and at most 2 contrast examples from other
-    has examples left; then, if short, fill from other families under the label and group caps."""
+    families whenever the pool holds any example of the target's family (maintainer, 2026-10-01: "allow 2 contrast").
-    per_label, other_cap = k // 2, min(2, k)
+    A pool without the target's family takes from the others under the label and group caps. ``short`` when the
+    pool cannot fill ``k`` or a label minimum: what exists is taken, never padded with more contrast."""
+    per_label = k // 2
+    other_cap = min(CONTRAST, k) if any(e.family == family for e in ranked) else k
     taken: list[Example] = []
     labels: dict[int, int] = {}
     groups: dict[str, int] = {}
-
-    def fits(example: Example) -> bool:
-        return labels.get(example.label, 0) < per_label and groups.get(example.group, 0) < PER_GROUP
-
-    def take(example: Example) -> None:
-        taken.append(example)
-        labels[example.label] = labels.get(example.label, 0) + 1
-        groups[example.group] = groups.get(example.group, 0) + 1
-
     others = 0
     for example in ranked:
         if len(taken) == k:
             break
         if example.family != family and others >= other_cap:
             continue
-        if fits(example):
+        if labels.get(example.label, 0) < per_label and groups.get(example.group, 0) < PER_GROUP:
             others += example.family != family
-            take(example)
+            taken.append(example)
-    for example in ranked:
+            labels[example.label] = labels.get(example.label, 0) + 1
-        if len(taken) == k:
+            groups[example.group] = groups.get(example.group, 0) + 1
-            break
-        if example.id not in {e.id for e in taken} and fits(example):
-            take(example)
     pool_labels = {e.label for e in ranked}
     short = len(taken) < k or any(labels.get(lab, 0) < min(2, per_label) for lab in (0, 1) if lab in pool_labels)
     return taken, "short" if short else "ok"
@@ -202,10 +195,14 @@ def retrieve(
 ) -> Retrieval:
     """The ``k`` balanced examples for ``target`` from the eligible part of ``pool`` (module docstring)."""
     candidates = [e for e in pool if e.profile == target.profile and eligible(e, target, fold)]
-    scored = sorted(
+    ranked_all = sorted(
         ((gower(target.x, target.instruments, e.x, e.instruments, target.profile, priors=priors, inputs=inputs), e) for e in candidates),
         key=lambda pair: (pair[0], _tie(pair[1].id, seed)),
-    )[:n1]
+    )
+    # stage 1 per side: the n1 nearest of the target's family and the n1 nearest contrast examples, so a family that is a
+    # minority of the memory still reaches stage 2 (the balance rule then takes at most 2 contrast examples)
+    own = [pair for pair in ranked_all if pair[1].family == target.family][:n1]
+    scored = own + [pair for pair in ranked_all if pair[1].family != target.family][:n1]
     mode = "signals+embeddings" if target.vector is not None and vectors else "signals_only"
     near = without = 0
     reranked: list[tuple[float, str, Example]] = []

tests/test_learn_compile.py

@@ -6,7 +6,7 @@ byte for byte at $0 (``test_reproducible_from_cache``); the permissive-licence f
 from __future__ import annotations
 
 import json
-from collections.abc import Iterator, Mapping, Sequence
+from collections.abc import Iterable, Iterator, Mapping, Sequence
 from pathlib import Path
 
 import pytest
@@ -30,6 +30,7 @@ from openultrasast.learn.compile import (
     permissive,
     program_id,
 )
+from openultrasast.learn.examples import Example
 from openultrasast.learn.folds import Fold, FoldError
 from openultrasast.learn.program import Caller, Demo
 from openultrasast.plane.budget import MeteredClient
@@ -191,3 +192,23 @@ def test_only_permissively_licensed_demonstrations_are_packaged() -> None:
     assert [d["example_id"] for d in packaged["demos"]] == [mit.id, apache.id]
     assert packaged["program_id"] == program_id(packaged) != artifact["program_id"]
     assert packaged["packaged"]["from"] == artifact["program_id"]
+
+
+def test_a_one_family_compile_asks_only_that_family(monkeypatch: pytest.MonkeyPatch) -> None:
+    """``candidate_families`` limits the bootstrap and validation candidates; the compile memory keeps every family
+    (retrieval's contrast examples), so the artifact's data counts are those of the whole pool."""
+    asked: dict[str, set[str]] = {}
+    real = compile_module._order
+
+    def spy(items: Iterable[Example], seed: int, salt: str) -> list[Example]:
+        items = list(items)
+        asked.setdefault(salt, set()).update(e.family for e in items)
+        return real(items, seed, salt)
+
+    monkeypatch.setattr(compile_module, "_order", spy)
+    result = compile_program(MEMORY, _caller(ScriptedChat(answer)), excerpt_text, spec=SPEC, candidate_families=("injection",))
+    assert result.status == "done" and result.artifact is not None
+    assert asked["boot"] == asked["val"] == {"injection"}
+    assert result.artifact["data"]["examples"] == len(MEMORY)
+    with pytest.raises(CompileError, match="compile split too small"):
+        compile_program(MEMORY, _caller(ScriptedChat(answer)), excerpt_text, spec=SPEC, candidate_families=("memory",))

tests/test_learn_examples.py

@@ -18,6 +18,7 @@ from openultrasast.learn.examples import (
     build_examples,
     label_pin,
     load_examples,
+    pair_pins,
     record_key,
     records_by_key,
 )
@@ -155,3 +156,18 @@ def test_a_fixed_side_without_the_function_is_a_label_defect_not_an_unread_side(
     assert count.absent_fixed_side == 1 and count.no_excerpt == 0 and build.failing() == []
     unread = build_examples([fixed], records_by_key([feature(fixed)]), lambda lab, role: None, sources=SOURCES)
     assert unread.sources["population-v1"].unread == 1 and unread.failing() == ["population-v1"]
+
+
+def test_pair_sides_join_the_record_stored_under_their_harvest_unit_pin() -> None:
+    """The harvest stores a pair side's record under its unit pin (the blob sha1 of the side's excerpt), not under
+    :func:`label_pin`; ``pins`` from the harvest ``units.json`` joins them, and without it the record is not found."""
+    lab = label(source="pairs", source_ref="case-9", pin="")
+    unit_pin = "a" * 40
+    stored = {**feature(lab), "pin": unit_pin}
+    units = {"h-1": {"source": "pairs", "ref": "case-9", "side": "vulnerable", "pin": unit_pin, "repo": lab["repo"]},
+             "h-2": {"source": "population-v1", "ref": "case-9", "side": "vulnerable", "pin": PIN, "repo": lab["repo"]}}  # fmt: skip
+    pins = pair_pins(units)
+    assert pins == {("case-9", "vulnerable"): unit_pin}
+    assert build_examples([lab], records_by_key([stored]), lines_of, sources=SOURCES).summary()["examples"] == 0
+    build = build_examples([lab], records_by_key([stored]), lines_of, sources=SOURCES, pins=pins)
+    assert build.summary()["examples"] == 1 and build.rows[0]["pin"] == unit_pin

tests/test_learn_program.py

@@ -15,6 +15,8 @@ from learn_fixtures import ScriptedChat, by_id, corpus, x_record
 from openultrasast.learn import program as program_module
 from openultrasast.learn.folds import outer_folds
 from openultrasast.learn.program import (
+    ANSWER_FORMAT,
+    BASELINE_INSTRUCTION,
     PARSE_FAILED,
     Answer,
     Caller,
@@ -121,6 +123,15 @@ def test_the_prefix_is_byte_identical_across_candidates() -> None:
     assert "Demonstration 1:" in a.prefix and a.messages[1] != b.messages[1]
 
 
+def test_every_instruction_carries_the_answer_contract() -> None:
+    """A proposed instruction rarely restates the JSON contract; the prefix adds it, once, so JSON mode is accepted
+    (DeepSeek refuses a prompt without the word "json") and the answer parses. The baseline prefix is unchanged."""
+    proposed, _ = program(instruction="Weigh the sanitiser before the sink.").prepare(candidate(), FOLD)
+    baseline, _ = program().prepare(candidate(), FOLD)
+    assert proposed.prefix.startswith("Weigh the sanitiser before the sink.\n") and proposed.prefix.count(ANSWER_FORMAT) == 1
+    assert baseline.prefix.count(ANSWER_FORMAT) == 1 and baseline.prefix.startswith(BASELINE_INSTRUCTION)
+
+
 def test_parsing_valid_invalid_and_out_of_range_citations() -> None:
     lines = frozenset({40, 41})
     ok = parse_answer(

tests/test_learn_retrieve.py

@@ -8,7 +8,7 @@ from dataclasses import replace
 import pytest
 from learn_fixtures import by_id, corpus, example, x_record
 
-from openultrasast.learn.folds import compile_split, leave_one_framework_out, leave_one_source_out, outer_folds
+from openultrasast.learn.folds import Fold, compile_split, leave_one_framework_out, leave_one_source_out, outer_folds
 from openultrasast.learn.retrieve import (
     BoundaryViolation,
     Target,
@@ -102,6 +102,25 @@ def test_balance_caps_labels_groups_and_other_families() -> None:
     crowded = [example(f"c{i}", group="one/repo", label=i % 2) for i in range(10)]
     taken, state = balance(crowded, "injection", 6)
     assert len(taken) == 2 and state == "short"  # at most 2 per repository group
+    scarce = [example("own", group="own/repo", label=1)] + [
+        example(f"o{i}", group=f"o{i}/r", label=i % 2, family="path") for i in range(10)
+    ]
+    taken, state = balance(scarce, "injection", 6)
+    assert sum(e.family != "injection" for e in taken) == 2 and len(taken) == 3 and state == "short"  # never padded
+    absent = [example(f"a{i}", group=f"a{i}/r", label=i % 2, family="path") for i in range(10)]
+    taken, state = balance(absent, "injection", 6)
+    assert len(taken) == 6 and state == "ok"  # no example of the family at all: the others fill under the caps
+
+
+def test_a_minority_family_reaches_stage_two_and_contrast_stays_at_two() -> None:
+    """Stage 1 keeps the n1 nearest of the target's family beside the n1 nearest others: 50 contrast examples nearer
+    in signals cannot push the family out, and the balance rule takes at most 2 of them."""
+    others = [example(f"o{i}", group=f"o{i}/r", label=i % 2, family="path", hits=0) for i in range(50)]
+    own = [example(f"own{i}", group=f"own{i}/r", label=i % 2, hits=6) for i in range(4)]
+    fold = Fold("compile", "compile", frozenset(), phase="compile")
+    got = retrieve(_target(hits=0), [*others, *own], fold, n1=40, k=6)
+    assert sum(e.family == "injection" for e in got.examples) == 4
+    assert sum(e.family != "injection" for e in got.examples) == 2
 
 
 def test_near_duplicates_drop_in_evaluation_and_stay_in_deployment() -> None: