bench(learn): slice 2 -- six families out-of-repo

Decision-engine task 6.7 (partial). After the advisory-fix harvest (memory 724 -> 904 static examples),
one compile per evaluable family and an outer grouped 5-fold evaluation of that family's own candidates,
input profile v1, priors off, k = 5, cross-fitted Platt, nested operating points (repository-cluster
bootstrap 95%):

  family                 n   AUC(s)              ADVISORY R / P   'vulnerable' P / R   ECE    canary
  injection            110   0.77 [0.67, 0.88]   0.88 / 0.57      0.84 / 0.36          0.093  0.77
  untrusted_dest.       62   0.85 [0.76, 0.94]   0.84 / 0.73      0.96 / 0.66          0.071  0.87
  path                  35   0.89 [0.79, 0.97]   0.86 / 0.69      0.89 / 0.76          0.119  0.90
  output_encoding       44   0.91 [0.82, 0.97]   0.86 / 0.81      0.94 / 0.55          0.045  0.90
  deserialization       46   0.90 [0.82, 0.97]   0.83 / 0.76      0.93 / 0.83          0.095  0.93
  access_control        37   0.96 [0.90, 1.00]   0.91 / 0.91      1.00 / 0.61          0.076  0.83

BLOCK offered for none: calibration fails everywhere except output_encoding, where no fold reaches a
Wilson lower bound of 0.95. Within paired groups (a function and its own fix) AUC is lower, 0.72-0.93:
every single-label group in the new families holds positives only. Injection learning-curve point,
paired (same program, candidates, folds): memory 724 -> 904, AUC 0.76 -> 0.77 (no injection example
was added). $2.88 metered DeepSeek (ceiling $8; balance 17.73 -> 16.73), $0.0009 embeddings.

Instrument fix: compiled demonstrations now follow the near-duplicate rule (cos >= 0.98) that retrieval
and the pre-call assertion apply; the assertion had stopped the untrusted_destination evaluation.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo

.kiro/specs/learned-decision-engine/design.md

@@ -1227,6 +1227,15 @@ start only after the maintainer confirms their budget against the balance read j
   0.90 has precision 0.54 (prevalence 0.53); the majority `vulnerable` verdict alone has precision 0.86, recall 0.43.
   Canary re-ask agreement at temperature 0 was 0.83, below the 0.9 the adoption check requires.
 
+- **Slice 2, 2026-10-02 (six families, after the advisory-fix harvest).** Two instrument defects stopped runs before any
+  number: the memory build's side reader knew only the first pair-kind source (all 194 advisory sides unread), and
+  compiled demonstrations skipped the near-duplicate rule that retrieval and the pre-call assertion apply (the
+  assertion stopped the untrusted_destination evaluation). Both fixed with tests. A confound to read with the new
+  families' numbers: every single-label repository group in their evaluation holds positives only (the pre-advisory
+  corpus had no negatives there), so pooled AUC mixes "vulnerable vs its own fix" with "vulnerable vs another
+  repository's fix"; the record reports the paired-group AUC beside it (0.72-0.93, lower than pooled 0.77-0.96).
+  Memorisation is visible: a verify pass named the advisory's CVE id for a public fix.
+
 ## Maintainer decisions at design approval (2026-09-30)
 
 - Design approved as written.

.kiro/specs/learned-decision-engine/tasks.md

@@ -107,6 +107,12 @@ a model or embedding call.
   - [ ] 6.7 Second paid slice **(budget go-ahead: ~$7 expected, ceiling $10; needs a top-up)**
     - Leave-one-source-out (~$1.4-2.1), leave-one-framework-out where a framework reaches 10 groups ($0 today),
       learning curves over memory size (0/25/50/100%) and number of sources (~$5.2), recorded beside 6.6.
+    - Partial 2026-10-02 (`benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`): after the
+      advisory-fix harvest (memory 724 -> 904), one compile per evaluable family and its outer-fold evaluation for all
+      six (injection, untrusted_destination, path, output_encoding, deserialization, access_control); out-of-repository
+      AUC 0.77-0.96, BLOCK offered for none (calibration holds only for output_encoding, where no fold reaches the
+      Wilson bound); one memory-size point for injection (724 vs 904 examples, paired: AUC 0.76 vs 0.77). $2.88
+      metered (ceiling $8). Not done: leave-one-source-out, the 0/25/50% curve points, the source-count curve.
     - _Requirements: 5.1, 5.2, 8.3_
 
 - [ ] 7. Experiments (exp-001 and program variants need budget go-ahead)

src/openultrasast/learn/program.py

@@ -475,7 +475,8 @@ class Program:
         target = candidate.target
         got = retrieve(target, self.memory, fold, vectors=self.vectors, n1=self.spec.n1, k=self.spec.k_ret, lam=self.spec.lam, seed=seed,
                        priors=self.spec.priors, inputs=self.spec.inputs)  # fmt: skip
-        usable = [Demo.from_dict(d) for d in demonstrations_for([d.as_dict() for d in self.spec.demos], self.index, target, fold)]
+        demos = demonstrations_for([d.as_dict() for d in self.spec.demos], self.index, target, fold, vectors=self.vectors)
+        usable = [Demo.from_dict(d) for d in demos]
         prompt = render(self.spec, candidate, got.examples, usable, self.index, self.excerpt_text)
         assert_boundary([*prompt.example_ids, *prompt.demo_ids], self.index, target, fold, vectors=self.vectors)
         return prompt, got

src/openultrasast/learn/retrieve.py

@@ -240,19 +240,32 @@ def assert_boundary(
 
 
 def demonstrations_for(
-    demos: Sequence[Mapping[str, Any]], examples: Mapping[str, Example], target: Target, fold: Fold
+    demos: Sequence[Mapping[str, Any]],
+    examples: Mapping[str, Example],
+    target: Target,
+    fold: Fold,
+    *,
+    vectors: Mapping[str, Sequence[float]] | None = None,
 ) -> list[Mapping[str, Any]]:
     """The compiled demonstrations usable for ``target``: an ineligible one is swapped for its alternate when that
-    is eligible, and dropped otherwise (leave-one-framework-out on a demonstration's framework)."""
+    is eligible, and dropped otherwise (leave-one-framework-out on a demonstration's framework; in evaluation, a
+    demonstration whose excerpt is a near-duplicate of the target's, the same rule as retrieval and
+    :func:`assert_boundary`)."""
+
+    def ok(example: Example | None) -> bool:
+        if example is None:
+            return False
+        vector = (vectors or {}).get(example.excerpt_sha)
+        cos = cosine(target.vector, vector) if target.vector is not None and vector is not None else None
+        return eligible(example, target, fold, cos=cos)
+
     out: list[Mapping[str, Any]] = []
     for demo in demos:
-        example = examples.get(str(demo["example_id"]))
-        if example is not None and eligible(example, target, fold):
+        if ok(examples.get(str(demo["example_id"]))):
             out.append(demo)
             continue
         alternate = demo.get("alternate")
-        alt = examples.get(str(alternate.get("example_id"))) if isinstance(alternate, Mapping) else None
-        if isinstance(alternate, Mapping) and alt is not None and eligible(alt, target, fold):
+        if isinstance(alternate, Mapping) and ok(examples.get(str(alternate.get("example_id")))):
             out.append(alternate)
     return out
 

tests/test_learn_retrieve.py

@@ -136,6 +136,24 @@ def test_near_duplicates_drop_in_evaluation_and_stay_in_deployment() -> None:
     assert kept.near_duplicates == 0 and twin.id in {e.id for e in kept.examples}
 
 
+def test_a_near_duplicate_demonstration_is_swapped_in_evaluation_and_kept_in_deployment() -> None:
+    """A compiled demonstration whose excerpt is a near-duplicate of the target's (cos >= 0.98) is ineligible in
+    evaluation, like a retrieved one: swapped for its alternate, or dropped. Before 2026-10-02 only the pre-call
+    assertion checked it, and the advisory-fix slice stopped there (BoundaryViolation: near duplicate)."""
+    twin = example("demo-twin", group="fork/repo", label=1)
+    alt = example("demo-alt", group="a/b", label=1)
+    vectors = {twin.excerpt_sha: [1.0, 0.0], alt.excerpt_sha: [0.0, 1.0]}
+    target = replace(_target(), vector=[1.0, 0.0])
+    evaluation = replace(outer_folds([twin], k=1)[0], eval_groups=frozenset())
+    demos = [{"example_id": twin.id, "alternate": {"example_id": alt.id}}]
+    usable = demonstrations_for(demos, by_id([twin, alt]), target, evaluation, vectors=vectors)
+    assert [d["example_id"] for d in usable] == [alt.id]
+    assert_boundary([d["example_id"] for d in usable], by_id([twin, alt]), target, evaluation, vectors=vectors)
+    assert demonstrations_for([{"example_id": twin.id, "alternate": None}], by_id([twin]), target, evaluation, vectors=vectors) == []
+    deployed, _ = deployment_fold("https://github.com/someone/else", ["fork/repo"])
+    assert [d["example_id"] for d in demonstrations_for(demos, by_id([twin, alt]), target, deployed, vectors=vectors)] == [twin.id]
+
+
 def test_deployment_excludes_the_scanned_repository_when_it_is_a_corpus_group() -> None:
     own = example("own", group="acme/webapp", label=1)
     fold, note = deployment_fold("https://github.com/Acme/WebApp.git", ["acme/webapp", "x/y"])