fix(learning-harness): the round-3 review findings, remediated RED-first

Eight of nine group-3 tasks were rejected. Each defect was hiding a number,
and the numbers are why the loop was not learning:

- The fixed side was judged against the *vulnerable* side's function
  ranges. 191 of 227 pairs with a resolvable labeled span (84%) have
  different spans on the two sides, so a leak was recorded as silence,
  always in the direction that flatters Youden.
- Round zero measured every family with the generalist. `family:unknown`
  earns no credit against any real family, so every family but `unknown`
  baselined at a structural zero and every noise floor with it.
- The cross-family sweep counted absolute failures instead of negative
  flips. On a corpus where most pairs fail at round zero that rejects
  every round for damage it did not do. Unscorable rows voted too.
- K runs was declared and never threaded; `pairs --k-runs` was parsed and
  discarded, and its only test grepped `--help`.
- The meta-agent proposer was a stub, so `learning round` could never
  propose anything. It now composes MetaAgent.evolve against a scratch
  copy of the family directory, and `--proposals FILE` runs a recorded
  sequence offline.
- `refused_holdout` was a vocabulary word no code path could produce, and
  the covering test accepted either outcome.
- Cost was metered once, after everything had been paid for; a stage that
  raised left no record at all.
- The corrected lever number was published beside the earlier one rather
  than in place of it, and the test named for that property never checked.
- Refusals were `continue` statements: `Refusal.degradation()` existed and
  nothing called it. A train-split identical twin was reported as
  `split:train`, which is untrue.

Test repairs that each changed a number: two split tests measured an
`x = 1` fixture that teaches nothing whatever the rule says; two CLI
scoreboard tests asserted against all-zero output produced by a stub that
answers twice and puts its finding outside the labeled function.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG

.kiro/specs/learning-harness/tasks.md

@@ -78,59 +78,59 @@
   - _Requirements: 9.3, 9.6, 9.7, 11.3_
   - _Boundary: Journal_
 
-- [ ] 3. Integration: split, scoring paths, rounds, proposer, publish, CLI
+- [x] 3. Integration: split, scoring paths, rounds, proposer, publish, CLI
 - [x] 3.1 One teacher rule across every learning path
   - Teachers are vendored, gating-tier, train-split, scorable pairs; a refusal names any candidate taught by or touching a holdout pair; the exporter, leave-one-out, per-profile evaluation, mechanism-profile evaluation, mechanism admission and pair signals all obtain teachers through the rule and record refusals as degradations; the mechanism lever's holdout number is re-measured under the rule and its raw artifact committed for the publish task.
   - Observable: leave-one-out on the toy slice seeds from train pairs only; a candidate shape taught by a holdout pair is refused by name; the re-measured artifact is in the measurements directory.
   - _Depends: 1.3_
   - _Requirements: 5.1, 5.2, 5.3_
 
-- [ ] 3.2 Family scoring on the pairs hunter path
+- [x] 3.2 Family scoring on the pairs hunter path
   - The pair scorer's hunter path takes an injected family matcher and leak rule from the learning package, classifies each pair once, runs the admitted family detectors plus the generalist K times per side, and emits family outcomes and metrics; the overlay and inventory paths stay untouched.
   - Observable: the pairs command with a scripted client on the shared fixture reports a family outcome per pair; no text-token matching remains on the hunter path; the pair gate output is byte-identical to before.
   - _Depends: 2.1, 2.4, 2.5, 2.6_
   - _Requirements: 3.1, 3.2, 3.3, 3.5, 11.5_
 
-- [ ] 3.3 Round zero and the noise floor
+- [x] 3.3 Round zero and the noise floor
   - Clone the current hunter prompt into every family configuration; run each family K = 5 times over the scorable vendored pairs of the selected slices at deterministic settings; record the per-family negative-flip rate between runs as that family's regression budget; persist noise floors and round-zero artifacts keyed by detector model so another model can be baselined and compared without touching configurations; the web families over vibe-py and agent-vfc are the default selection and the memory-safety family over vfc is selectable but never gates.
   - Observable: with a scripted client round zero writes a noise-floor file with one budget per family and a round-zero report per slice; a second baseline under another model name writes a separate artifact and the comparison table lists both.
   - _Depends: 3.2_
   - _Requirements: 8.1, 8.2, 8.3, 8.4_
 
-- [ ] 3.4 Family detectors, verifiers and tags in scans
+- [x] 3.4 Family detectors, verifiers and tags in scans
   - In a standard scan each classified region runs its admitted family detectors plus the generalist from the directories round zero wrote (a missing family directory is a recorded degradation and that family's detector is skipped), claims pass through the family's verifier, findings carry family, detector and verifier tags in reports and SARIF, and the manifest gains a learning block with taxonomy version, per-family counts, the classifier report and the current round; the detection and map gates stay byte-identical.
   - Observable: a standard scan of the shared fixture with a scripted client writes findings tagged with a family, a detector version and a verifier rung, and the manifest block; the detection and map gate outputs diff empty.
   - _Depends: 3.3, 2.6_
   - _Requirements: 6.3, 7.1, 11.5_
 
-- [ ] 3.5 Proposer protocol, scripted proposer and the HarnessX proposer
+- [x] 3.5 Proposer protocol, scripted proposer and the HarnessX proposer
   - Failure facts for one family: misses and leaks with pair, function, sink or check, line, verifier outcome and run outcomes, plus the rejected buffer and the current configuration; a proposal is one hypothesis, one lever (tool, checklist, counterexample, memory, prompt), one file-level change inside the family directory, and predicted affected and at-risk families; the proposer never receives holdout inputs, outputs or scores.
   - A scripted proposer for tests and offline rounds; a HarnessX proposer composing the meta-agent with write roots limited to the family directory, degrading to a recorded reason without the extra.
   - Observable: the scripted proposer returns a proposal from a facts fixture; the HarnessX proposer is skipped with the reason recorded when the extra is absent and, when present, writes only inside the family directory; the write-root list excludes the verifier and corpus directories.
   - _Depends: 2.5, 2.8_
   - _Requirements: 9.1, 9.2, 5.4, 7.6_
 
-- [ ] 3.6 Acceptance rule and directory revert
+- [x] 3.6 Acceptance rule and directory revert
   - Acceptance requires train and holdout non-negative with one strictly positive and every other family within its budget; ties reject; a family directory snapshot restores byte for byte on rejection or overrun; refusals for holdout contact, length cap and lever set are outcomes with reasons.
   - Observable: a helpful change is accepted, a tie is rejected, a change that breaches another family's budget is rejected with the family named, and a revert restores the directory byte for byte.
   - Observable: every family other than the target has a byte-identical directory before and after the round, whatever the outcome.
   - _Depends: 3.3_
   - _Requirements: 6.2, 9.5_
 
-- [ ] 3.7 Evolve round orchestration, cost and journal
+- [x] 3.7 Evolve round orchestration, cost and journal
   - One round: facts, proposal, refusals, train minibatch, target-family holdout, sweep over every other family's holdout, K-run scoring at every stage as the scorer requires, accept (freeze, bump version, archive) or revert; cost metered per stage with revert and a recorded reason on overrun; a journal record and the rejected buffer written for every outcome; a round directory holds the snapshot, proposal, scores and trajectories and a repeated round number is refused.
   - Observable: with the scripted proposer each of accepted, rejected, budget-breach and cost-overrun leaves one journal record with attribution and a round directory; a repeated round number is refused.
   - The round directory holds the pair identities behind the attribution counts and every detector trajectory, redacted, so a round can be replayed and its attribution audited; every stage scores at least three runs per pair.
   - _Depends: 3.5, 3.6_
   - _Requirements: 3.5, 9.4, 9.6, 9.7, 9.8, 9.9, 11.3_
 
-- [ ] 3.8 Publish from artifacts
+- [x] 3.8 Publish from artifacts
   - One entry point regenerates every family and slice number from committed artifacts, writes a markdown table with scorable and unscorable counts, taxonomy version, noise floor and cost per point beside every number, replaces the roadmap's measured section between markers, and writes the raw artifacts under the measurements directory.
   - Observable: running the publish entry point twice without new artifacts produces byte-identical output; the corrected mechanism-lever number from the re-measurement artifact replaces the earlier roadmap value.
   - _Depends: 3.1, 3.3_
   - _Requirements: 11.1, 11.2, 4.5, 5.3_
 
-- [ ] 3.9 Learning subcommands
+- [x] 3.9 Learning subcommands
   - Subcommands for classify, score, baseline, round and publish with the flags the design names; the pairs hunter path gains the K-runs flag; every step runs under the harness runtime with degradations for missing extras, endpoints or sandbox.
   - Observable: each subcommand exits zero on the shared fixture with a scripted client and writes its artifact; the roadmap section written by the publish subcommand carries the command line that produced it; a missing endpoint records the reason and the remaining steps complete.
   - _Depends: 3.7, 3.8_
@@ -219,3 +219,17 @@
   - Three existing tests encoded the old corpus and now encode the new one: the TypeScript parse pin (the lensops excerpt parses now), the classifier report (`labeled` 0 -> 4, the four seeded rows; the six `title`-tier agent rows carry a family but are not labels a maintainer stands behind), and the hygiene report ordering.
   - **New honesty finding, recorded in the hygiene artifact as `cross_split_duplicates`: nine of the ten duplicate groups sit on both sides of the split.** Two forks of one project (`openniw-jobs-0fc947` train, `openniw-jobs-b74de3` holdout) and eight CVEs harvested twice under different names are byte-identical, so the train row teaches the holdout row exactly. `split_by_repository` cannot see it — the duplicates are different repositories. `duplicate_groups` is now ordered by name so the committed report does not reshuffle when one excerpt is re-harvested.
   - Pre-existing drift the regeneration corrected, unrelated to this task: `catalog.toml` in git disagreed with a fresh `catalog_gen` run at the group-3 tip on two derived labels (`origin` -> `createApplication`, `normalizeDomain` -> `getGoogleProvider`). Verified in a detached worktree at `7a33756` before touching anything.
+- Group 3 (2026-09-06). Tasks 3.1-3.9 implemented in `86ed54c`, `fcda1ee`, `e8e6baf`, `127dbef`, `ad7c98e`, `346cc70`, `16d9bc2`, `b955251`, `b8ad42c`, `7a33756`. RED counts, derived by the reviewer against each task's parent commit: 3.1 4/6, 3.2 6/6, 3.3 6/6, 3.4 3/5, 3.5 10/10, 3.6 12/12, 3.7 9/9, 3.8 7/7, 3.9 7/7. Deviations from the design, recorded here because they were not written down at the time: `learning.split` duck-types `PairCase` through `getattr` rather than importing it; `pairs.py` reaches `learning` through function-local imports rather than the design's injected callable; `run_baseline` takes a scan factory where the design named a client. One flaky test found and fixed in `346cc70` (`_is_vulnerable_side` keyed on `"-v" in str(root)`, which random temporary directory names matched about once in eight; harness noise recorded as detector noise would inflate every family's regression budget).
+- Review round 3 (2026-09-06) APPROVED 3.4; REJECTED 3.1, 3.2, 3.3, 3.5, 3.6, 3.7, 3.8, 3.9. The reviewer ran every check in detached worktrees at the group tip against `71efc9c`, derived the RED phase per task, mutation-tested `decide`, the teacher rule, the snapshot restore and the round's revert paths (7 mutants on the acceptance rule, 6 killed and 1 equivalent), ran the three flakiness-prone files 61 times with varied `PYTHONHASHSEED` without a failure, and confirmed the three gate outputs byte-identical and the extra-free matrix unchanged. Every rejection was remediated RED-first; the numbers each defect was hiding are recorded below because they are the point of the spec.
+  - **3.2, the fixed side was judged against the vulnerable side's line ranges.** A fix that adds a guard above the handler moves its first line, and the reviewer measured that **191 of 227 pairs with a resolvable labeled span (84%) have different spans on the two sides**. A finding inside the fixed handler was therefore read as outside it: a leak recorded as silence, always in the direction that flatters Youden. `score_pair_family` now takes `fixed_ranges`; the fixed side's parse was already being computed and thrown away.
+  - **3.2 and 3.9, K runs was declared and never threaded.** `_evaluate_hunter_pair(..., runs=1)` had one call site that passed nothing, `evaluate_catalog` had no parameter at all, and `pairs --k-runs` was parsed and discarded; the only test grepped `--help`. K now flows `pairs --k-runs` -> `evaluate_catalog` -> `evaluate_pair` -> `_evaluate_hunter_pair`, defaults to three and refuses less, and the test counts scans per side.
+  - **3.2, the generalist could never be scored.** `pairs --hunter` fed the class-aware scorer findings with no `family:` tag, so it reported recall 0 by construction. `run_tool_hunter` now reads an optional `family` field into the tag (the design said it would) and `make_hunter_scan(taxonomy=...)` asks for one from the closed list. A generalist that names no family is unanswerable; one that names a wrong family is counted as noise, which is the honest treatment.
+  - **3.3 and 3.9, round zero measured every family with the generalist.** `run_baseline` took one `scan`, and the CLI handed it the `unknown` configuration; since `related(X, "unknown")` is `lateral` by the taxonomy's own decision that abstention earns no credit, **every family except `unknown` baselined at a structural zero** and every noise floor with it. `run_baseline` now takes a `scan_factory` and runs each family's own configuration, reports per slice as well as per family, and refuses fewer than three runs. This is the decision the group-2 review asked for on `unknown` serving as both abstention and generalist: it is the generalist for *scanning*, and it is never credited as a family for *scoring*.
+  - **3.5, the meta-agent proposer was a stub** that returned None after confirming the extra was present, so `ousast learning round` could never propose anything at any time. It now composes `MetaAgent.evolve` behind `harness_ext` against a scratch **copy** of the family directory — the live tree is never touched, because the round snapshots only after a proposal exists — and builds the proposal from the diff of that copy, so the one-file rule and the lever whitelist are enforced on what the agent did rather than on what it claimed. `learning round --proposals FILE` runs a recorded sequence offline, which is what makes the command exercisable without the extra at all.
+  - **3.6 and 3.1, the holdout refusal existed only as a vocabulary word.** `journal.py` documented `refused_holdout` and no code path could produce it; the covering test asserted `outcome == "refused_holdout" or outcome == "rejected"`, which passes whichever way the code behaves. The round now runs `refuse_if_holdout` over every corpus pair the proposal names — in its predictions and in the file it writes — and journals `refused_holdout` with the pair names. A memory entry that quotes a held-out pair looks exactly like a legitimate hard negative until the holdout number is read as if it meant something.
+  - **3.7, the sweep counted absolute failures, not negative flips.** Req 9.5 compares against the pre-change state; the code counted every pair of another family that was not `pair_correct` and compared that to a budget derived from the run-to-run flip rate. On the real corpus, where most pairs fail at round zero, that rejects **every** round for damage it did not do — a harness that then reports it cannot improve. Unscorable rows voted too, against Req 4.1. The sweep is now scored before and after the change, `_negative_flips` counts only pairs that were right and are not, unscorable rows leave the count, and attribution is derived from the same comparison instead of from absolute failures.
+  - **3.7, cost was metered once**, after the minibatch, the holdout and the whole cross-family sweep had all been paid for; Req 9.8 says stop, revert and record. The cap is now read after every stage. A stage that raises is journalled as `reverted`/`stage_failed` with its round directory before the exception leaves, because an exception that loses the record also loses the money the round already spent. A rejected round's version is archived: Req 9.7 exists so a configuration that lost overall but is the only one that ever won some pair is not forgotten.
+  - **3.8, the corrected lever number was published beside the earlier one, not in place of it** (Req 5.3), and the test named for that property never checked it. `publish` now rewrites the superseded claim to point at the regenerated section, idempotently, and the test asserts the old sentence is gone. Run against the real roadmap: the 2026-09-05 claim "human holdout profile pair_correct 0 -> 6 of 17, Youden 0 -> +0.353" now says it was measured with holdout pairs teaching and names the artifact that corrects it.
+  - **3.1, refusals were `continue` statements.** `Refusal.degradation()` existed and nothing called it. The mechanism lever now records every candidate it declined with the holdout pairs that taught it, leave-one-out records the pairs it kept out of the teaching set, and `improve` carries both into the round outcome. `semantic.seed` reported `split:train` for a train-split identical twin, which is simply untrue; an unscorable row now reports its own reason. `evaluate_profiles` and `evaluate_mechanism_profiles` are deliberately split-blind and now say so in their docstrings: they measure a ruleset and never learn from a pair, so a filter there would only shrink the evidence.
+  - Test-quality repairs the review asked for, each of which changed a number: two split tests were measuring an `x = 1` fixture that teaches nothing whatever the rule says, so both arms now use a pair that really carries a shape and the lever test asserts the same shape *is* admitted when a train pair taught it; the two CLI scoreboard tests asserted against output that was all zeros, and now use a side-aware detector stub (the shipped `scripted` stub answers twice, then goes quiet, and puts its finding outside the labeled function — three separate ways to make a family number zero); `ScriptedChatClient` gained an opt-in `cycle` so a stub cannot silently run out of answers mid-run; the sandbox-candidate assertion in `tests/test_learning_scan.py` compared verdict paths against a prefix no implementation would produce and now compares finding ids; the manifest's classifier block reported `measure_classifier([])`, all zeros, and now reports what this scan actually routed.
+  - Carry-forward, unchanged and still capping every holdout number: the nineteen straddling repositories, and — found during task 4.1 — **nine of the ten duplicate groups sit on both sides of the split**, which the repository-level split cannot see because the duplicates are different repositories.

.kiro/steering/roadmap.md

@@ -51,7 +51,7 @@ Leave-one-out (corpus-seeded-mechanisms, 2026-09-05; the corpus as a teacher, ne
 | github | 6 | 0 | 6 | 0.000 | 0 | advisory tier |
 | vfc-js | 17 | 0 | 17 | 0.000 | 0 | advisory tier |
 
-The loop is closed: `ousast mechanisms export` writes candidates, `ousast improve --pair-catalog` admits the ones that recover a holdout pair without leaking (per-profile clause over variant search; `mechanism_leak`/`mechanism_no_gain` revert byte for byte), the scan searches the admitted store and reports name the mechanism and its known fix. Measured 2026-09-05 on the `python-vulnerable` benchmark target with the vibe-py candidates: 4 records admitted in round 1 (`execute/1 parameter` taught by 7 pairs among them), human holdout profile pair_correct 0 -> 6 of 17, Youden 0 -> +0.353, round 2 `no_proposals`.
+The loop is closed: `ousast mechanisms export` writes candidates, `ousast improve --pair-catalog` admits the ones that recover a holdout pair without leaking (per-profile clause over variant search; `mechanism_leak`/`mechanism_no_gain` revert byte for byte), the scan searches the admitted store and reports name the mechanism and its known fix. Measured 2026-09-05 on the `python-vulnerable` benchmark target with the vibe-py candidates: 4 records admitted in round 1 (`execute/1 parameter` taught by 7 pairs among them), superseded by the learning-harness section below: that number was measured with holdout pairs teaching (see `benchmarks/measurements/2026-09-06-mechanism-lever-split.json`).
 
 Measured 2026-09-05 after review tiers (Req 9) and pointer slices (Req 10): vibe-py 115 = 35 vendored human (13/35, Youden +37.1%) + 80 `agent` pointer pairs over the 40 LLM-generated Real-Vuln repositories (0/80, Youden 0.0; the LLM-repo baseline, scored by the nightly `pairs --slice vibe-py --pointers` only, never in tests or gates); agent-vfc 59 = 29 vendored + 30 pointer, tier `title`, 1/59, Youden 0.0, 2 unresolved labels; sast, vfc-js, vfc unchanged; detection/map/local-pair gate outputs byte-identical to the pre-spec commit. Only `seeded`/`reviewed` pairs gate the improve loop. Carry-forwards: `language_unsupported` loss counter and multi-line C/C++ declarator rescue (overlay-ir-completeness), `profiles_ungated` reporting and an offline test for the agent-vfc pointer builder (corpus-seeded-mechanisms).
 
@@ -89,3 +89,13 @@ sast: java-sqli moved MISS→LEAK (coverage counted on both sides; the fixed twi
 - [ ] overlay-ir-completeness -- Shared CST walker completeness (C `init_declarator`, lvalue paths for field stores, per-grammar literal kinds, local ERROR-node tolerance, multi-line C/C++ declarators the walker names `<anon>` and `functions.py`'s one-line rescue cannot name: `chromium-cve-2022-2162`, `openssl-cve-2025-69421`; a Scorer-side multi-line-aware rescue is the cheaper alternative) plus Python parity with stdlib ast (f-string constness, augmented assignment, kwargs); injection-regime facts: container/transformer taint, `weak_literals` consulted, parameter sources off for injection. Measured on sast and vibe-py. Dependencies: pair-corpus-honesty
 - [ ] authorization-obligations -- Absence bugs as obligations, not flows: obligated operations (protected read/write, privileged action, security setting) and discharger kinds (path guard, identity constraint from authenticated context, ownership check, non-permissive value) as closed facts; obligation shapes learned from trusted pairs; the application's policy recovered from sibling-handler consistency and an optional declared, versioned policy file; violations evaluated on the path (function-local degraded mode without it) with `consistency_violation` / `declared_policy_violation` rungs below proof; `obligations` improve lever under the holdout gate; the hunter adjudicates intent only. Measured leave-one-out on the vibe-py and agent-vfc absence pairs (today 0 detected; 15 of 29 agent-vfc rows). Dependencies: reachability-flow-model, guard-dominance-regime, corpus-seeded-mechanisms. **Requirements drafted 2026-09-05, unapproved.**
 - [ ] guard-dominance-regime -- Guard and Use IR records; guard dominance by nesting or early exit; memory sinks (memcpy/malloc/deref) with operand roles, nullable callees, out-param transformers, fixed-array capacity; same regime applied to web absence bugs (route auth guard, identity from auth context). Consumes path records. Measured on vfc, vfc-js, agent-vfc. Dependencies: reachability-flow-model, overlay-ir-completeness
+
+<!-- learning-harness:begin -->
+
+## Classified detector numbers
+
+Generated by `uv run ousast learning publish`.
+
+No baseline has been recorded yet, so there is nothing to publish. Run round zero first; a number without the run that produced it is a claim, not a measurement.
+
+<!-- learning-harness:end -->

src/openultrasast/cli.py

@@ -7,7 +7,7 @@ from collections.abc import Callable, Sequence
 from dataclasses import dataclass, replace
 from pathlib import Path
 from time import perf_counter
-from typing import Any
+from typing import TYPE_CHECKING, Any
 
 from . import tool_hunter
 from .benchmark import (
@@ -100,6 +100,10 @@ class ScanOutcome:
     exit_code: int
 
 
+if TYPE_CHECKING:
+    from .learning.proposer import Proposer
+
+
 def main(argv: list[str] | None = None) -> int:
     load_dotenv()
     parser = argparse.ArgumentParser(prog="ousast")
@@ -223,6 +227,12 @@ def main(argv: list[str] | None = None) -> int:
         if name == "round":
             command.add_argument("--family", required=True)
             command.add_argument("--cost-cap-usd", type=float, default=None)
+            command.add_argument(
+                "--proposals",
+                type=Path,
+                default=None,
+                help="a JSON list of proposals to try in order; without it the round asks the meta-agent",
+            )
         if name == "publish":
             command.add_argument("--measurements", type=Path, default=Path("benchmarks/measurements"))
             command.add_argument("--roadmap", type=Path, default=Path(".kiro/steering/roadmap.md"))
@@ -263,6 +273,7 @@ def main(argv: list[str] | None = None) -> int:
             pointers=args.pointers,
             loo=args.loo,
             loo_out=args.loo_out,
+            k_runs=args.k_runs,
         )
     if args.command == "mechanisms":
         return _mechanisms_export(args.catalog, args.slice, args.store, json_out=args.json)
@@ -954,6 +965,7 @@ def _pairs(
     pointers: bool = False,
     loo: bool = False,
     loo_out: Path | None = None,
+    k_runs: int | None = None,
 ) -> int:
     if not catalog.exists() or not catalog.is_file():
         raise SystemExit(f"pair catalog is not a file: {catalog}")
@@ -967,8 +979,10 @@ def _pairs(
         model = hunter_model or config.models.hunter or ""
         client = tool_hunter.resolve_hunter_client()
         if model and client is not None:
-            scan = make_hunter_scan(client, model)
-    result = evaluate_catalog(cases, hunter=scan, pointers=True if pointers else None)
+            from .learning.families import load_families
+
+            scan = make_hunter_scan(client, model, taxonomy=load_families())
+    result = evaluate_catalog(cases, hunter=scan, pointers=True if pointers else None, k_runs=k_runs)
     if json_out:
         print(json.dumps(result_payload(result), indent=2, sort_keys=True))
         return 0
@@ -1125,7 +1139,7 @@ def _run_family_detectors(
     suspicion here: the verifier writes a tag, and the evidence ladder is another boundary's to move.
     """
     del cases
-    from .learning.classify import classify_region, measure_classifier
+    from .learning.classify import classify_region
     from .learning.detectors import DetectorConfigError, Region, load_family_configs, run_region_detectors
     from .learning.endpoint import resolve_chat_endpoint, resolve_models
     from .learning.families import FamiliesError, load_families
@@ -1152,6 +1166,8 @@ def _run_family_detectors(
     model = resolve_models(config)[0]  # type: ignore[arg-type]
     findings: list[StaticFinding] = []
     per_family: dict[str, int] = {}
+    per_tier: dict[str, int] = {}
+    abstained = 0
     seen: set[str] = set()
     for hotspot in hotspots:
         path = str(getattr(hotspot, "path", ""))
@@ -1161,6 +1177,8 @@ def _run_family_detectors(
             continue
         seen.add(key)
         answer = classify_region(root, path, function=function, taxonomy=taxonomy, client=None)
+        per_tier[answer.tier] = per_tier.get(answer.tier, 0) + 1
+        abstained += 0 if answer.families else 1
         region = Region(path=path, function=str(function) if function else None, families=answer.families)
 
         def _run(region: Region = region) -> list[StaticFinding]:
@@ -1172,11 +1190,19 @@ def _run_family_detectors(
             verification = verifier_for(taxonomy.by_id(family_id)).verify(finding, root=root, language="python")
             findings.append(apply_verification(finding, verification))
             per_family[family_id] = per_family.get(family_id, 0) + 1
-    report = measure_classifier([], taxonomy, client=None)
     payload: dict[str, object] = {
         "taxonomy_version": taxonomy.version,
         "families": per_family,
-        "classifier": report.to_dict(),
+        # What the classifier did on *this* scan. Scoring it needs labels a maintainer stands behind, which a scan
+        # of someone else's repository does not have, so the block reports the routing rather than an accuracy.
+        "classifier": {
+            "regions": len(seen),
+            "per_tier": dict(sorted(per_tier.items())),
+            "abstained": abstained,
+            # Req 2.6/3.7: say out loud that no family declares a parent at this taxonomy version, so the
+            # partial credit for a parent-child relation never applied to any number here.
+            "hierarchical_credit": taxonomy.hierarchical,
+        },
         "round": None,
         "endpoint": endpoint.provider,
         "regions": len(seen),
@@ -1226,7 +1252,6 @@ def _learning_run(args: argparse.Namespace, cases: Sequence[PairCase], taxonomy:
     from .learning.detectors import load_family_configs, write_default_configs
     from .learning.endpoint import resolve_chat_endpoint, resolve_models
     from .learning.journal import Archive, LearningJournal
-    from .learning.proposer import HarnessXProposer
     from .learning.rounds import MIN_K_RUNS, load_noise_floors, run_baseline, run_learning_round
     from .learning.scoring import aggregate
     from .tool_hunter import _SYSTEM_PROMPT
@@ -1259,7 +1284,9 @@ def _learning_run(args: argparse.Namespace, cases: Sequence[PairCase], taxonomy:
         from .learning.rounds import score_case
 
         scores = [
-            score_case(case, factory(configs[_label(case)]), taxonomy=taxonomy, runs=k_runs) for case in cases if _label(case) in configs
+            score_case(case, factory(configs[_label(case, taxonomy)]), taxonomy=taxonomy, runs=k_runs, family=_label(case, taxonomy))
+            for case in cases
+            if _label(case, taxonomy) in configs
         ]
         families = {name: block.to_dict() for name, block in aggregate(scores, taxonomy=taxonomy).items()}
         if args.json:
@@ -1273,7 +1300,7 @@ def _learning_run(args: argparse.Namespace, cases: Sequence[PairCase], taxonomy:
             cases,
             taxonomy=taxonomy,
             configs_dir=configs_dir,
-            scan=factory(configs.get("unknown") or next(iter(configs.values()))),
+            scan_factory=factory,
             model=model,
             out_dir=out,
             k_runs=k_runs,
@@ -1286,7 +1313,7 @@ def _learning_run(args: argparse.Namespace, cases: Sequence[PairCase], taxonomy:
         family=args.family,
         taxonomy=taxonomy,
         configs_dir=configs_dir,
-        proposer=HarnessXProposer(configs_dir=configs_dir, model=model),
+        proposer=_proposer(args, configs_dir, model),
         scan_factory=factory,
         model=model,
         journal=journal,
@@ -1296,16 +1323,44 @@ def _learning_run(args: argparse.Namespace, cases: Sequence[PairCase], taxonomy:
         cost_cap_usd=args.cost_cap_usd or config.learning.round_cost_cap_usd,
         minibatch=config.learning.minibatch,
         k_runs=k_runs,
+        # Every other family the corpus actually carries, so a change that reaches beyond its own directory is
+        # measured rather than assumed impossible (Req 9.5).
+        sweep_families=tuple(sorted({_label(case, taxonomy) for case in cases} & set(configs) - {args.family})),
     )
     print(f"learning round {record.round} {record.family}: {record.outcome} ({record.reason})")
     return 0
 
 
-def _label(case: PairCase) -> str:
-    for row in case.expected:
-        if row.family:
-            return str(row.family)
-    return "unknown"
+def _label(case: PairCase, taxonomy: object) -> str:
+    """The family that routes this pair: the maintainer's label when there is one, else the classifier's answer.
+
+    Reading only the declared label would send every unlabeled row to the generalist, whose findings earn no credit
+    against any real family, so a scoreboard built that way reports structural zeros (Req 2.1, 8.2)."""
+    from .learning.rounds import _family_of
+
+    return _family_of(case, taxonomy)  # type: ignore[arg-type]
+
+
+def _proposer(args: argparse.Namespace, configs_dir: Path, model: str) -> Proposer:
+    """`--proposals FILE` runs a recorded sequence offline; without it the round asks the meta-agent."""
+    from .learning.proposer import HarnessXProposer, Proposal, ScriptedProposer
+
+    path = getattr(args, "proposals", None)
+    if path is None:
+        return HarnessXProposer(configs_dir=configs_dir, model=model)
+    rows = json.loads(Path(path).read_text(encoding="utf-8"))
+    return ScriptedProposer(
+        [
+            Proposal(
+                hypothesis=str(row["hypothesis"]),
+                lever=row["lever"],
+                change=dict(row["change"]),
+                predicted_affected=tuple(row.get("predicted_affected", ())),
+                predicted_at_risk=tuple(row.get("predicted_at_risk", ())),
+            )
+            for row in rows
+        ]
+    )
 
 
 def _first_source(root: Path) -> str:

src/openultrasast/improve/evolve.py

@@ -52,6 +52,8 @@ class RoundOutcome:
     profiles_under_minimum: list[str] = field(default_factory=list)
     per_profile_before: dict[str, dict[str, float]] = field(default_factory=dict)
     per_profile_after: dict[str, dict[str, float]] = field(default_factory=dict)
+    # learning-harness Req 5.2: the candidates this round declined to learn from, by pair name.
+    degradations: list[dict[str, object]] = field(default_factory=list)
 
 
 MAX_PROFILES = 4
@@ -64,6 +66,10 @@ def evaluate_profiles(
     """Pair-corpus metrics per provenance profile under ``rules`` (holdout split, no inventory sidecar).
 
     Only ``seeded`` and ``reviewed`` pairs gate (Req 9.3); ``advisory`` and ``title`` pairs are scored by ``pairs``, never here.
+
+    Deliberately split-blind: this *measures* a ruleset, it never learns from a pair, so every pair is scored and the
+    teacher rule has nothing to say here. What the rule governs is the store these rules were built from
+    (`semantic.seed`, `semantic.loo`), and a filter here would only shrink the evidence (learning-harness Req 5.1).
     """
     from ..pairs import GATING_TIERS, PairCase, evaluate_catalog, select_tier, select_vendored
 
@@ -84,7 +90,10 @@ def evaluate_profiles(
 
 
 def evaluate_mechanism_profiles(pair_cases: Sequence[object], store_path: Path) -> dict[str, dict[str, float]]:
-    """Per-profile pair metrics of variant search with the given scan-time store over the holdout pairs (vendored, gating tiers)."""
+    """Per-profile pair metrics of variant search with the given scan-time store over the holdout pairs (vendored, gating tiers).
+
+    Split-blind for the same reason as `evaluate_profiles`: the store is the thing the teacher rule governs, and this
+    function only reads it (learning-harness Req 5.1)."""
     from ..pairs import GATING_TIERS, PairCase, select_tier, select_vendored
     from ..semantic.loo import score_pair_with_store
     from ..semantic.mechanisms import MechanismStore
@@ -110,7 +119,12 @@ def evaluate_mechanism_profiles(pair_cases: Sequence[object], store_path: Path)
     return result
 
 
-def propose_mechanism_edits(pair_cases: Sequence[object], candidates: object, store_path: Path) -> list[MechanismEdit]:
+def propose_mechanism_edits(
+    pair_cases: Sequence[object],
+    candidates: object,
+    store_path: Path,
+    degradations: list[dict[str, object]] | None = None,
+) -> list[MechanismEdit]:
     """Req 5.3: admit a candidate that recovers a currently missed holdout pair without leaking; retract an admitted leaker."""
     from ..pairs import GATING_TIERS, PairCase, select_tier, select_vendored
     from ..semantic.loo import score_pair_with_store
@@ -139,8 +153,13 @@ def propose_mechanism_edits(pair_cases: Sequence[object], candidates: object, st
     for record in corpus_mechanisms(candidates.load()):  # type: ignore[attr-defined]
         if record.id in admitted or not missed:
             continue
-        if refuse_if_holdout(record.pairs, cases_all) is not None:
-            continue  # learning-harness Req 5.2: a shape a holdout pair taught can never recover that pair
+        refusal = refuse_if_holdout(record.pairs, cases_all)
+        if refusal is not None:
+            # learning-harness Req 5.2: a shape a holdout pair taught can never recover that pair, and the run
+            # says which pairs it was rather than showing a shorter list with no reason.
+            if degradations is not None:
+                degradations.append(refusal.degradation())
+            continue
         with tempfile.TemporaryDirectory(prefix="ousast-lever-") as scratch:
             trial = MechanismStore(Path(scratch) / "trial.jsonl")
             trial.append(record)
@@ -278,6 +297,7 @@ def run_round(
     edits = propose_status_edits(before.result.metrics.per_rule, ruleset_by_id, current_ledger, blocked)
     # corpus-seeded-mechanisms Req 5: the `mechanisms` lever proposes from the holdout pairs, gated like every other edit.
     mechanism_edits: list[MechanismEdit] = []
+    refusals: list[dict[str, object]] = []
     candidates_store = None
     if mechanism_candidates is not None and mechanism_store is not None and pair_cases:
         from ..semantic.mechanisms import MechanismStore
@@ -286,12 +306,12 @@ def run_round(
         mechanism_edits = (
             list(scripted_mechanism_edits)
             if scripted_mechanism_edits is not None
-            else propose_mechanism_edits(pair_cases, candidates_store, mechanism_store)
+            else propose_mechanism_edits(pair_cases, candidates_store, mechanism_store, degradations=refusals)
         )
         blocked_mechanisms = {edit.key() for edit in mechanism_edits} & blocked
         mechanism_edits = [edit for edit in mechanism_edits if edit.key() not in blocked_mechanisms]
     if not edits and not mechanism_edits:
-        return _outcome(round_index, accepted=False, reason="no_proposals", edits=[], before=before, after=before)
+        return _outcome(round_index, accepted=False, reason="no_proposals", edits=[], before=before, after=before, degradations=refusals)
 
     validator = EvolveValidator()
     try:
@@ -314,6 +334,7 @@ def run_round(
             before=before,
             after=before,
             mechanism_edits=mechanism_edits,
+            degradations=refusals,
         )
 
     candidate = edits_to_ledger(edits, current_ledger)
@@ -321,7 +342,9 @@ def run_round(
         _load_with(ruleset_dir, candidate)  # replay smoke gate: candidate ruleset must boot
     except Exception as exc:  # noqa: BLE001 — any boot failure rejects the round
         _record(journal_path, round_index, edits, "rejected", before, before, f"replay_failed: {exc}")
-        return _outcome(round_index, accepted=False, reason="replay_failed", edits=edits, before=before, after=before)
+        return _outcome(
+            round_index, accepted=False, reason="replay_failed", edits=edits, before=before, after=before, degradations=refusals
+        )
 
     after_findings, after_rules = _scan(target, ruleset_dir, candidate, policy)
     after = _evaluate(target, manifest, after_findings, after_rules, policy)
@@ -395,7 +418,14 @@ def run_round(
         candidates=candidates_store,
     )
     outcome = _outcome(
-        round_index, accepted=accepted, reason=reason, edits=edits, before=before, after=after, mechanism_edits=mechanism_edits
+        round_index,
+        accepted=accepted,
+        reason=reason,
+        edits=edits,
+        before=before,
+        after=after,
+        mechanism_edits=mechanism_edits,
+        degradations=refusals,
     )
     return RoundOutcome(
         **{
@@ -513,6 +543,7 @@ def _outcome(
     before: _Eval,
     after: _Eval,
     mechanism_edits: list[MechanismEdit] | None = None,
+    degradations: list[dict[str, object]] | None = None,
 ) -> RoundOutcome:
     return RoundOutcome(
         round=round_index,
@@ -520,6 +551,7 @@ def _outcome(
         reason=reason,
         edits=edits,
         mechanism_edits=list(mechanism_edits or []),
+        degradations=list(degradations or []),
         recall_before=before.recall,
         recall_after=after.recall,
         fp_before=before.fp_rate,

src/openultrasast/learning/proposer.py

@@ -12,7 +12,9 @@ construction rather than by promise, and the verifier code and the corpus stay o
 from __future__ import annotations
 
 import json
-from collections.abc import Mapping, Sequence
+import shutil
+import tempfile
+from collections.abc import Callable, Mapping, Sequence
 from dataclasses import dataclass, field
 from pathlib import Path
 from typing import Literal, Protocol
@@ -25,7 +27,7 @@ from .scoring import PairFamilyScore
 Lever = Literal["tool", "checklist", "counterexample", "memory", "prompt"]
 LEVERS: tuple[Lever, ...] = ("tool", "checklist", "counterexample", "memory", "prompt")
 # The only files a round may write, one per lever. Anything else is a proposal reaching beyond its family.
-EDITABLE_FILES = {
+EDITABLE_FILES: dict[Lever, str] = {
     "tool": "family.toml",
     "checklist": "checklist.md",
     "counterexample": "counterexamples.jsonl",
@@ -159,31 +161,109 @@ class ScriptedProposer:
         return self.proposals[self.used - 1]
 
 
+MetaAgentCall = Callable[..., str]
+
+
 @dataclass
 class HarnessXProposer:
-    """The meta-agent, with write access limited to the family directory it is targeting."""
+    """The meta-agent, with write access limited to a copy of the family directory it is targeting.
+
+    It never edits the live tree. The round snapshots the family directory *after* a proposal exists, so a proposer
+    that wrote in place would have changed the configuration before there was anything to restore, and a rejection
+    could not put it back. The agent works in a scratch copy and the proposal is the diff of that copy, which also
+    means the one-file rule and the lever whitelist are enforced on what the agent actually did rather than on what
+    it claimed (Req 9.1, 9.2, 5.4).
+    """
 
     configs_dir: Path
     model: str
     reason: str = ""
+    agent: MetaAgentCall | None = None  # injected in tests; None composes MetaAgent behind harness_ext
 
     def write_roots(self, family: str) -> tuple[Path, ...]:
         return write_roots(self.configs_dir, family)
 
     def propose(self, facts: FailureFacts) -> Proposal | None:
-        if not has_harnessx():
-            self.reason = "harnessx_unavailable"
+        call = self.agent
+        if call is None:
+            if not has_harnessx():
+                self.reason = "harnessx_unavailable"
+                return None
+            try:
+                call = _meta_agent_call()
+            except Exception:  # noqa: BLE001 — a proposer that cannot start proposes nothing and says so
+                self.reason = "harnessx_unavailable"
+                return None
+        family_dir = self.configs_dir / facts.family
+        with tempfile.TemporaryDirectory(prefix="ousast-propose-") as scratch:
+            workspace = Path(scratch) / facts.family
+            shutil.copytree(family_dir, workspace)
+            before = _directory_text(workspace)
+            try:
+                hypothesis = call(workspace=workspace, facts=facts, model=self.model)
+            except Exception as exc:  # noqa: BLE001 — an agent that failed proposed nothing, and the round says why
+                self.reason = f"meta_agent_failed: {type(exc).__name__}"
+                return None
+            after = _directory_text(workspace)
+        changed = sorted(name for name in set(before) | set(after) if before.get(name) != after.get(name))
+        if not changed:
+            self.reason = "no_change_proposed"
             return None
-        try:
-            from ..harness_ext import require_harnessx
+        if len(changed) > 1:
+            self.reason = f"a round changes one file, not {len(changed)}: {', '.join(changed)}"
+            return None
+        name = changed[0]
+        lever: Lever | None = next((key for key, filename in EDITABLE_FILES.items() if filename == name), None)
+        if lever is None or name not in after:
+            self.reason = f"{name!r} is not a file any lever writes"
+            return None
+        self.reason = ""
+        return Proposal(
+            hypothesis=str(hypothesis or "").strip() or f"meta-agent edit to {name}",
+            lever=lever,
+            change={name: after[name]},
+            predicted_affected=tuple(item.pair for item in facts.misses),
+            predicted_at_risk=tuple(item.pair for item in facts.leaks),
+        )
 
-            require_harnessx()
-        except Exception:  # noqa: BLE001 — a proposer that cannot start proposes nothing and says so
-            self.reason = "harnessx_unavailable"
-            return None
-        self.reason = "harnessx_proposer_not_wired"  # the evolve call lands with the live rounds (task 5.3)
-        del facts
-        return None
+
+def _directory_text(directory: Path) -> dict[str, str]:
+    """Every readable file directly inside ``directory``, by name. Nested paths are not a lever's business."""
+    out: dict[str, str] = {}
+    for path in sorted(directory.iterdir()) if directory.is_dir() else []:
+        if path.is_file():
+            try:
+                out[path.name] = path.read_text(encoding="utf-8")
+            except (OSError, UnicodeDecodeError):
+                continue
+    return out
+
+
+def _meta_agent_call() -> MetaAgentCall:
+    """Compose `MetaAgent.evolve` behind `harness_ext`, writing only inside the scratch copy.
+
+    `evolve` is a coroutine and returns a HarnessX config path; what this proposer takes from it is the state of the
+    workspace afterwards, so the return value is only used for the hypothesis line.
+    """
+    from ..harness_ext import require_harnessx
+
+    require_harnessx()
+
+    def call(*, workspace: Path, facts: FailureFacts, model: str) -> str:
+        import asyncio
+
+        from harnessx.meta_harness.agent import MetaAgent  # type: ignore[import-not-found]
+
+        trajectories = workspace.parent / "trajectories"
+        trajectories.mkdir(parents=True, exist_ok=True)
+        (trajectories / "facts.json").write_text(facts_prompt(facts), encoding="utf-8")
+        agent = MetaAgent(inner_model=model, allowed_write_roots=(workspace,))  # type: ignore[arg-type]
+        output = workspace.parent / "meta-out"
+        output.mkdir(parents=True, exist_ok=True)
+        asyncio.run(agent.evolve(current_config=workspace, trajectories_dir=trajectories, output_dir=output))
+        return f"meta-agent round for {facts.family}"
+
+    return call
 
 
 def _fact(score: PairFamilyScore, other: int) -> PairFact:

src/openultrasast/learning/publish.py

@@ -12,6 +12,7 @@ rather than appended, so the file cannot accumulate stale numbers beside fresh o
 from __future__ import annotations
 
 import json
+import re
 from collections.abc import Mapping
 from dataclasses import dataclass, field
 from datetime import UTC, datetime
@@ -69,6 +70,7 @@ def publish(*, learning_dir: Path, measurements_dir: Path, roadmap: Path, date:
         )
         artifacts.append(artifact)
     _replace_section(roadmap, table)
+    _supersede(roadmap)
     return PublishReport(command=COMMAND, table=table, families=families, models=tuple(sorted(models)), artifacts=tuple(artifacts))
 
 
@@ -169,6 +171,32 @@ def _cost(cost: float, correct: int) -> str:
     return f"{cost / correct:.2f}" if cost and correct else "-"
 
 
+# Claims the split correction supersedes, as they were written before holdout pairs stopped teaching. Req 5.3 says
+# the corrected number is published *in place of* the earlier one; a document that carries both asserts two numbers
+# for one measurement and the reader has no way to tell which run is current.
+_SUPERSEDED = (
+    re.compile(r"human holdout profile pair_correct \d+ -> \d+ of \d+, Youden [-+]?\d+ -> [-+]?[\d.]+(?:, round \d+ `[^`]+`)?\."),
+    re.compile(r"The lever recovered \d+ of \d+ holdout pairs at Youden [-+][\d.]+\."),
+)
+_SUPERSEDED_NOTE = (
+    "superseded by the learning-harness section below: that number was measured with holdout pairs teaching "
+    "(see `benchmarks/measurements/2026-09-06-mechanism-lever-split.json`)."
+)
+
+
+def _supersede(roadmap: Path) -> None:
+    """Rewrite every superseded lever claim to point at the regenerated section. Idempotent: the pattern is gone
+    afterwards, so a second run changes nothing."""
+    if not roadmap.is_file():
+        return
+    text = roadmap.read_text(encoding="utf-8")
+    updated = text
+    for pattern in _SUPERSEDED:
+        updated = pattern.sub(_SUPERSEDED_NOTE, updated)
+    if updated != text:
+        roadmap.write_text(updated, encoding="utf-8")
+
+
 def _replace_section(roadmap: Path, table: str) -> None:
     body = f"{BEGIN}\n\n## Classified detector numbers\n\n{table.rstrip()}\n\n{END}\n"
     text = roadmap.read_text(encoding="utf-8") if roadmap.is_file() else ""

src/openultrasast/learning/rounds.py

@@ -68,6 +68,9 @@ class BaselineReport:
     slices: tuple[str, ...]
     floors: dict[str, NoiseFloor] = field(default_factory=dict)
     metrics: dict[str, dict[str, object]] = field(default_factory=dict)
+    # `<slice>/<family>`: the budgets stay per family, but vibe-py and agent-vfc are different worlds and one
+    # pooled number hides which of them moved (Req 8.4).
+    per_slice: dict[str, dict[str, object]] = field(default_factory=dict)
     scores: tuple[PairFamilyScore, ...] = ()
     cost_usd: float = 0.0
 
@@ -80,6 +83,7 @@ class BaselineReport:
             "cost_usd": self.cost_usd,
             "floors": {name: floor.to_dict() for name, floor in sorted(self.floors.items())},
             "metrics": {name: dict(block) for name, block in sorted(self.metrics.items())},
+            "per_slice": {name: dict(block) for name, block in sorted(self.per_slice.items())},
         }
 
 
@@ -88,7 +92,7 @@ def run_baseline(
     *,
     taxonomy: FamilyTaxonomy,
     configs_dir: Path,
-    scan: Scan,
+    scan_factory: Callable[[FamilyConfig], Scan],
     model: str,
     out_dir: Path,
     slices: Sequence[str] = DEFAULT_SLICES,
@@ -96,13 +100,29 @@ def run_baseline(
     prompt: str | None = None,
     spent_usd: Callable[[], float] | None = None,
 ) -> BaselineReport:
-    """Clone one detector into every family, measure each family against itself, and write the artifacts."""
-    from ..tool_hunter import _SYSTEM_PROMPT
+    """Clone one detector into every family, measure each family **with its own configuration**, write the artifacts.
 
+    The factory is the point. One shared `scan` would score every family with one detector, and since the generalist
+    tags its findings `family:unknown` — which earns no credit against any real family, by the taxonomy's own
+    decision that abstention is not a partial answer — every family but `unknown` would baseline at a structural
+    zero and the noise floors would all be nil (Req 8.2).
+    """
+    from ..tool_hunter import _SYSTEM_PROMPT
+    from .detectors import load_family_configs
+
+    if k_runs < MIN_K_RUNS:
+        raise ValueError(f"round zero scores at least three runs per pair; {k_runs} was asked for")
     if not configs_dir.exists():
         write_default_configs(configs_dir, taxonomy, prompt=prompt or _SYSTEM_PROMPT, version="0")
+    configs = load_family_configs(configs_dir, taxonomy)
     selected = [case for case in cases if case.slice in set(slices)]
-    scores = [score_case(case, scan, taxonomy=taxonomy, runs=k_runs) for case in selected]
+    scores = []
+    for case in selected:
+        name = _family_of(case, taxonomy)
+        config = configs.get(name)
+        if config is None:
+            continue  # a family with no configuration is not measured rather than measured with someone else's
+        scores.append(score_case(case, scan_factory(config), taxonomy=taxonomy, runs=k_runs, family=name))
     metrics = aggregate(scores, taxonomy=taxonomy)
     floors = {name: _floor(name, block, [item for item in scores if item.family == name], k_runs) for name, block in metrics.items()}
     report = BaselineReport(
@@ -112,6 +132,7 @@ def run_baseline(
         slices=tuple(slices),
         floors=floors,
         metrics={name: block.to_dict() for name, block in metrics.items()},
+        per_slice={name: block.to_dict() for name, block in aggregate(scores, taxonomy=taxonomy, per_slice=True).items()},
         scores=tuple(scores),
         cost_usd=spent_usd() if spent_usd is not None else 0.0,
     )
@@ -141,9 +162,12 @@ def score_case(case: PairCase, scan: Scan, *, taxonomy: FamilyTaxonomy, runs: in
         vuln_root = _materialize_side(Path(scratch) / "vuln", case, side="vuln")
         fix_root = _materialize_side(Path(scratch) / "fixed", case, side="fixed")
         ranges = _overlay_scan(vuln_root).ranges
+        fixed_ranges = _overlay_scan(fix_root).ranges
         vuln_runs = [scan(vuln_root) for _ in range(max(runs, 1))]
         fix_runs = [scan(fix_root) for _ in range(max(runs, 1))]
-    return score_pair_family(case, name, vuln_runs, fix_runs, ranges=ranges, taxonomy=taxonomy, parse_ok=bool(ranges))
+    return score_pair_family(
+        case, name, vuln_runs, fix_runs, ranges=ranges, taxonomy=taxonomy, parse_ok=bool(ranges), fixed_ranges=fixed_ranges
+    )
 
 
 def compare_baselines(out_dir: Path) -> dict[str, dict[str, dict[str, object]]]:
@@ -249,11 +273,10 @@ def run_learning_round(
     real changes. The proposer never sees a held-out pair, and the family directory is snapshotted before
     the change so any outcome other than acceptance leaves the tree exactly as it was found.
     """
-    from ..redaction import redact_secrets
     from .acceptance import DirectorySnapshot, bump_version, decide, within_budget
     from .detectors import load_family_configs
     from .proposer import apply_proposal, build_failure_facts, refuse_proposal
-    from .split import is_teacher
+    from .split import is_teacher, refuse_if_holdout
 
     if k_runs < MIN_K_RUNS:
         raise ValueError(f"a stage scores at least three runs per pair; {k_runs} was asked for")
@@ -268,33 +291,73 @@ def run_learning_round(
     train = [case for case in cases if is_teacher(case) and _family_of(case, taxonomy) == family]
     holdout = [case for case in cases if case.split == "holdout" and _family_of(case, taxonomy) == family]
     batch = train[:minibatch]
+    spent = 0.0
 
     def _score(rows: Sequence[PairCase], scan: Scan, name: str) -> list[PairFamilyScore]:
         return [score_case(case, scan, taxonomy=taxonomy, runs=k_runs, family=name) for case in rows]
 
+    def _over_cap() -> bool:
+        """Req 9.8: cost is read after every stage, so a round stops rather than finishing and then complaining."""
+        nonlocal spent
+        spent = spent_usd() if spent_usd is not None else 0.0
+        return not within_budget(spent, cost_cap_usd)
+
     before_scan = scan_factory(config)
     before_train = _score(batch, before_scan, family)
     before_hold = _score(holdout, before_scan, family)
+    record = _round_record(round_number, family, None, taxonomy, config)
+    if _over_cap():
+        return _finish(
+            journal, directory, record, outcome="reverted_cost", reason="cost_cap_exceeded", cost=spent, trajectories=trajectories
+        )
     facts = build_failure_facts(before_train, family=family, config=config, journal=journal, train_pairs={case.name for case in train})
     proposal = proposer.propose(facts)  # type: ignore[attr-defined]
     record = _round_record(round_number, family, proposal, taxonomy, config)
     if proposal is None:
-        return _finish(journal, directory, record, outcome="rejected", reason="no_proposal", trajectories=trajectories)
+        return _finish(journal, directory, record, outcome="rejected", reason="no_proposal", cost=spent, trajectories=trajectories)
     refusal = refuse_proposal(proposal, config)
     if refusal is not None:
-        return _finish(journal, directory, record, outcome="rejected", reason=refusal, proposal=proposal, trajectories=trajectories)
+        return _finish(
+            journal, directory, record, outcome="rejected", reason=refusal, proposal=proposal, cost=spent, trajectories=trajectories
+        )
+    leak = refuse_if_holdout(_pairs_named_by(proposal, cases), cases)
+    if leak is not None:
+        # Req 5.2: a change a holdout pair taught, or that names one, is refused by name rather than measured.
+        return _finish(
+            journal,
+            directory,
+            record,
+            outcome="refused_holdout",
+            reason=leak.degradation()["detail"],  # type: ignore[arg-type]
+            proposal=proposal,
+            cost=spent,
+            trajectories=trajectories,
+        )
+    # The sweep's "before" is measured with the unchanged configurations, so what it later reports is what this
+    # round flipped and not what was already failing (Req 9.5).
+    before_sweep = _sweep_scores(cases, taxonomy, configs, scan_factory, sweep_families, family, k_runs)
+    if _over_cap():
+        return _finish(
+            journal,
+            directory,
+            record,
+            outcome="reverted_cost",
+            reason="cost_cap_exceeded",
+            proposal=proposal,
+            cost=spent,
+            trajectories=trajectories,
+        )
     snapshot = DirectorySnapshot.of(configs_dir / family)
     apply_proposal(proposal, configs_dir / family)
+    version = str(int(config.version) + 1) if config.version.isdigit() else f"{config.version}+1"
     try:
         after_config = load_family_configs(configs_dir, taxonomy)[family]
         after_scan = scan_factory(after_config)
         after_train = _score(batch, after_scan, family)
         after_hold = _score(holdout, after_scan, family)
-        sweep, flipped_unpredicted = _sweep(cases, taxonomy, configs, scan_factory, sweep_families, family, k_runs)
-        cost = spent_usd() if spent_usd is not None else 0.0
-        train_delta = _delta(before_train, after_train)
-        holdout_delta = _delta(before_hold, after_hold)
-        if not within_budget(cost, cost_cap_usd):
+        for score in after_train + after_hold:
+            archive.record(family, version, score)  # type: ignore[attr-defined]
+        if _over_cap():
             snapshot.restore()
             return _finish(
                 journal,
@@ -303,17 +366,34 @@ def run_learning_round(
                 outcome="reverted_cost",
                 reason="cost_cap_exceeded",
                 proposal=proposal,
-                cost=cost,
+                cost=spent,
                 scores=after_train + after_hold,
                 trajectories=trajectories,
             )
-        verdict = decide(target=family, train_delta=train_delta, holdout_delta=holdout_delta, sweep=sweep, floors=floors)  # type: ignore[arg-type]
+        after_sweep = _sweep_scores(cases, taxonomy, configs, scan_factory, sweep_families, family, k_runs)
+        sweep, flipped_unpredicted = _negative_flips(before_sweep, after_sweep)
+        train_delta = _delta(before_train, after_train)
+        holdout_delta = _delta(before_hold, after_hold)
         flipped = _flipped(before_train + before_hold, after_train + after_hold)
         attribution = Attribution(
             flipped_predicted=len(flipped),
             flipped_unpredicted=len(flipped_unpredicted),
             precision=len(flipped) / (len(flipped) + len(flipped_unpredicted)) if flipped or flipped_unpredicted else 0.0,
         )
+        if _over_cap():
+            snapshot.restore()
+            return _finish(
+                journal,
+                directory,
+                record,
+                outcome="reverted_cost",
+                reason="cost_cap_exceeded",
+                proposal=proposal,
+                cost=spent,
+                scores=after_train + after_hold,
+                trajectories=trajectories,
+            )
+        verdict = decide(target=family, train_delta=train_delta, holdout_delta=holdout_delta, sweep=sweep, floors=floors)  # type: ignore[arg-type]
         if not verdict.accepted:
             snapshot.restore()
             return _finish(
@@ -323,7 +403,7 @@ def run_learning_round(
                 outcome="rejected",
                 reason=verdict.reason,
                 proposal=proposal,
-                cost=cost,
+                cost=spent,
                 train_delta=train_delta,
                 holdout_delta=holdout_delta,
                 sweep=sweep,
@@ -333,10 +413,7 @@ def run_learning_round(
                 scores=after_train + after_hold,
                 trajectories=trajectories,
             )
-        version = str(int(config.version) + 1) if config.version.isdigit() else f"{config.version}+1"
         bump_version(configs_dir / family, version)
-        for score in after_train + after_hold:
-            archive.record(family, version, score)  # type: ignore[attr-defined]
         return _finish(
             journal,
             directory,
@@ -344,7 +421,7 @@ def run_learning_round(
             outcome="accepted",
             reason=verdict.reason,
             proposal=proposal,
-            cost=cost,
+            cost=spent,
             train_delta=train_delta,
             holdout_delta=holdout_delta,
             sweep=sweep,
@@ -356,13 +433,37 @@ def run_learning_round(
             trajectories=trajectories,
         )
     except Exception:
+        # A stage that raises has already spent money and already changed the tree. Both facts belong in the
+        # journal before the exception leaves, or the next round starts from a state nothing describes.
         snapshot.restore()
+        _finish(
+            journal,
+            directory,
+            record,
+            outcome="reverted",
+            reason="stage_failed",
+            proposal=proposal,
+            cost=spent,
+            trajectories=trajectories,
+        )
         raise
-    finally:
-        del redact_secrets
 
 
-def _sweep(
+def _pairs_named_by(proposal: Proposal, cases: Sequence[PairCase]) -> tuple[str, ...]:
+    """Every corpus pair this proposal names, in its predictions or in the file it writes.
+
+    A memory entry that quotes a held-out pair is the loop learning the answer instead of the shape, and it looks
+    exactly like a legitimate hard negative until the holdout number is read as if it meant something."""
+    haystack = "\n".join([*proposal.predicted_affected, *proposal.predicted_at_risk, *proposal.change.values()])
+    named = []
+    for case in cases:
+        name = str(getattr(case, "name", ""))
+        if name and re.search(rf"(?<![\w-]){re.escape(name)}(?![\w-])", haystack):
+            named.append(name)
+    return tuple(sorted(set(named)))
+
+
+def _sweep_scores(
     cases: Sequence[PairCase],
     taxonomy: FamilyTaxonomy,
     configs: Mapping[str, FamilyConfig],
@@ -370,23 +471,34 @@ def _sweep(
     families: Sequence[str],
     target: str,
     k_runs: int,
-) -> tuple[dict[str, int], list[str]]:
-    """Every other family's held-out pairs, so a change cannot help one family by quietly breaking another."""
-    flips: dict[str, int] = {}
-    unpredicted: list[str] = []
+) -> dict[str, dict[str, str]]:
+    """Every other family's held-out pairs, family by family, as `{family: {pair: outcome}}`.
+
+    Called once before the change and once after it. Scoring both sides is the only way to say which pairs *this*
+    round broke: on a real corpus most pairs are failing already, and counting failures would reject every round
+    for damage it did not do (Req 9.5). Unscorable rows are dropped here rather than counted as failures (Req 4.1).
+    """
+    outcomes: dict[str, dict[str, str]] = {}
     for name in families:
         if name == target or name not in configs:
             continue
         rows = [case for case in cases if case.split == "holdout" and _family_of(case, taxonomy) == name]
         scan = scan_factory(configs[name])
-        broken = [
-            item
-            for item in (score_case(case, scan, taxonomy=taxonomy, runs=k_runs, family=name) for case in rows)
-            if item.outcome != "pair_correct"
-        ]
-        flips[name] = len(broken)
-        unpredicted.extend(item.pair for item in broken)
-    return flips, unpredicted
+        scored = [score_case(case, scan, taxonomy=taxonomy, runs=k_runs, family=name) for case in rows]
+        outcomes[name] = {item.pair: item.outcome for item in scored if item.outcome != "unscorable"}
+    return outcomes
+
+
+def _negative_flips(before: Mapping[str, Mapping[str, str]], after: Mapping[str, Mapping[str, str]]) -> tuple[dict[str, int], list[str]]:
+    """Pairs that were right before this round and are not right after it, per family and by name."""
+    flips: dict[str, int] = {}
+    broken: list[str] = []
+    for name, was in before.items():
+        now = after.get(name, {})
+        lost = sorted(pair for pair, outcome in was.items() if outcome == "pair_correct" and now.get(pair) != "pair_correct")
+        flips[name] = len(lost)
+        broken.extend(lost)
+    return flips, broken
 
 
 def _delta(before: Sequence[PairFamilyScore], after: Sequence[PairFamilyScore]) -> float:
@@ -400,8 +512,9 @@ def _delta(before: Sequence[PairFamilyScore], after: Sequence[PairFamilyScore])
 
 
 def _flipped(before: Sequence[PairFamilyScore], after: Sequence[PairFamilyScore]) -> list[str]:
+    """Pairs this round turned right, by name and in a stable order."""
     was = {item.pair: item.outcome for item in before}
-    return [item.pair for item in after if item.outcome == "pair_correct" and was.get(item.pair) not in {"pair_correct", None}]
+    return sorted(item.pair for item in after if item.outcome == "pair_correct" and was.get(item.pair) not in {"pair_correct", None})
 
 
 def _round_record(

src/openultrasast/learning/scoring.py

@@ -58,14 +58,21 @@ def score_pair_family(
     taxonomy: FamilyTaxonomy,
     parse_ok: bool = True,
     entry_points: Sequence[str] | None = None,
+    fixed_ranges: FunctionRanges | None = None,
 ) -> PairFamilyScore:
-    """Score one pair for one family over K runs of the detector on each side."""
+    """Score one pair for one family over K runs of the detector on each side.
+
+    ``fixed_ranges`` is the *fixed* side's parse. A fix that adds a guard above the handler moves its first line,
+    so judging the fixed side against the vulnerable side's span reads a finding inside the fixed handler as
+    outside it: a leak recorded as silence, and always in the direction that flatters the number.
+    """
     name = str(getattr(case, "name", "?"))
     slice_name = str(getattr(case, "slice", "") or "")
     reason = unscorable_reason(case, parse_ok=parse_ok, ranges=ranges, entry_points=entry_points, family=family)
     if reason is not None:
         return PairFamilyScore(pair=name, family=family, slice=slice_name, runs=(), outcome="unscorable", unscorable_reason=reason)
     spans = _spans(case, ranges)
+    fixed_spans = _spans(case, fixed_ranges) if fixed_ranges is not None else spans
     outcomes: list[FamilyOutcome] = []
     other_vuln = other_fixed = fabricated_vuln = fabricated_fixed = 0
     rung: Rung = "suspicion"
@@ -73,9 +80,9 @@ def score_pair_family(
         vuln = list(vuln_runs[index]) if index < len(vuln_runs) else []
         fixed = list(fixed_runs[index]) if index < len(fixed_runs) else []
         hits_vuln = _hits(vuln, family, spans, taxonomy)
-        hits_fixed = _hits(fixed, family, spans, taxonomy)
+        hits_fixed = _hits(fixed, family, fixed_spans, taxonomy)
         other_vuln += _others(vuln, family, spans, taxonomy)
-        other_fixed += _others(fixed, family, spans, taxonomy)
+        other_fixed += _others(fixed, family, fixed_spans, taxonomy)
         fabricated_vuln += _fabricated(vuln, taxonomy)
         fabricated_fixed += _fabricated(fixed, taxonomy)
         rung = _best_rung(rung, hits_vuln)

src/openultrasast/pairs.py

@@ -417,12 +417,18 @@ def _review_tier(item: dict[str, object], slice_name: str, name: str) -> tuple[s
     return tier, "" if pending else reviewer
 
 
-def evaluate_pair(case: PairCase, *, hunter: HunterScan | None = None, ruleset: tuple[PatternRule, ...] | None = None) -> PairOutcome:
+def evaluate_pair(
+    case: PairCase,
+    *,
+    hunter: HunterScan | None = None,
+    ruleset: tuple[PatternRule, ...] | None = None,
+    k_runs: int | None = None,
+) -> PairOutcome:
     with tempfile.TemporaryDirectory(prefix="ousast-pair-") as scratch:
         vuln_root = _materialize_side(Path(scratch) / "vuln", case, side="vuln")
         fix_root = _materialize_side(Path(scratch) / "fixed", case, side="fixed")
         if hunter is not None:
-            return _evaluate_hunter_pair(case, vuln_root, fix_root, hunter)
+            return _evaluate_hunter_pair(case, vuln_root, fix_root, hunter, _k_runs(k_runs))
         if case.slice in OVERLAY_SLICES:
             return _evaluate_overlay_pair(case, vuln_root, fix_root, ruleset)
         return _evaluate_inventory_pair(case, vuln_root, fix_root, ruleset=ruleset)
@@ -435,12 +441,15 @@ def evaluate_catalog(
     ruleset: tuple[PatternRule, ...] | None = None,
     inventory_sidecar: bool = True,
     pointers: bool | None = None,
+    k_runs: int | None = None,
 ) -> PairEvalResult:
     """Score every case. ``ruleset`` lets the improve loop score a candidate ledger; ``inventory_sidecar=False`` skips the second scan.
 
     Pointer pairs (``vendored = false``) are harvested into the cache when ``pointers`` (or OPENULTRASAST_PAIRS_NETWORK=1) allows the
     network, otherwise skipped with a ``pointer_pair_skipped`` degradation (Req 10.2, 10.3).
     """
+    if hunter is not None:
+        _k_runs(k_runs)  # refuse an impossible K before spending a single call
     degradations: list[dict[str, object]] = []
     ready: list[PairCase] = []
     for case in cases:
@@ -459,7 +468,7 @@ def evaluate_catalog(
         if inventory_sidecar:
             scorers[slice_name]["inventory"] = _metrics(tuple(_inventory_only(case, ruleset) for case in slice_cases))
         if hunter is not None:
-            hunted = tuple(evaluate_pair(case, hunter=hunter) for case in slice_cases)
+            hunted = tuple(evaluate_pair(case, hunter=hunter, k_runs=k_runs) for case in slice_cases)
             hunter_outcomes.extend(hunted)
             scorers[slice_name]["hunter"] = _metrics(hunted)
     if hunter is None and scorers:
@@ -938,10 +947,23 @@ def _with_functions(root: Path, findings: list[StaticFinding]) -> list[StaticFin
     return out
 
 
-def make_hunter_scan(client: object, model: str, *, max_steps: int | None = None) -> HunterScan:
-    """Run the tool hunter over every file of an isolated pair tree (Req 1.7)."""
+def make_hunter_scan(client: object, model: str, *, max_steps: int | None = None, taxonomy: object | None = None) -> HunterScan:
+    """Run the tool hunter over every file of an isolated pair tree (Req 1.7).
+
+    With a ``taxonomy`` the hunter is asked to name the family of every finding, from the closed list, and the answer
+    becomes the finding's `family:` tag. The class-aware scorer credits a detection only to a family, so a generalist
+    that names none can never be scored — it would report recall zero by construction rather than by measurement.
+    """
     from .complexity.map import Hotspot
-    from .tool_hunter import DEFAULT_MAX_STEPS, run_tool_hunter
+    from .tool_hunter import _SYSTEM_PROMPT, DEFAULT_MAX_STEPS, run_tool_hunter
+
+    prompt = _SYSTEM_PROMPT
+    if taxonomy is not None:
+        ids = ", ".join(family.id for family in taxonomy.families)  # type: ignore[attr-defined]
+        prompt = (
+            f'{_SYSTEM_PROMPT}\n\nGive every finding a "family" field, chosen from exactly this list: {ids}. '
+            'Use "unknown" when none of them fits; do not invent a name.'
+        )
 
     def scan(root: Path) -> list[StaticFinding]:
         hotspots = [
@@ -957,7 +979,17 @@ def make_hunter_scan(client: object, model: str, *, max_steps: int | None = None
             )
             for target in _targets(root)
         ]
-        return _with_functions(root, run_tool_hunter(root, hotspots, client=client, model=model, max_steps=max_steps or DEFAULT_MAX_STEPS))  # type: ignore[arg-type]
+        return _with_functions(
+            root,
+            run_tool_hunter(
+                root,
+                hotspots,
+                client=client,  # type: ignore[arg-type]
+                model=model,
+                max_steps=max_steps or DEFAULT_MAX_STEPS,
+                system_prompt=prompt,
+            ),
+        )
 
     return scan
 
@@ -1097,14 +1129,23 @@ def _evaluate_overlay_pair(case: PairCase, vuln_root: Path, fix_root: Path, rule
     )
 
 
-def _evaluate_hunter_pair(case: PairCase, vuln_root: Path, fix_root: Path, hunter: HunterScan, runs: int = 1) -> PairOutcome:
+def _k_runs(value: int | None) -> int:
+    """Req 3.5: three runs per pair per side is the floor, and the default, because one run misses most changes."""
+    from .learning.rounds import MIN_K_RUNS
+
+    runs = MIN_K_RUNS if value is None else int(value)
+    if runs < MIN_K_RUNS:
+        raise ValueError(f"the hunter path scores at least three runs per pair; {runs} was asked for")
+    return runs
+
+
+def _evaluate_hunter_pair(case: PairCase, vuln_root: Path, fix_root: Path, hunter: HunterScan, runs: int) -> PairOutcome:
     """Score the model-driven hunter by family and function (learning-harness Req 3.1-3.3, 3.5).
 
     Detection is a finding of the labeled family inside the labeled function; a finding of another family
     or outside it is counted and never called a leak; the words in a finding are never read. A pair that
     cannot be scored is reported with its reason and the detector is never run on it.
     """
-    from .learning.classify import classify_pair
     from .learning.families import load_families
     from .learning.scoring import score_pair_family, unscorable_reason
 
@@ -1115,10 +1156,9 @@ def _evaluate_hunter_pair(case: PairCase, vuln_root: Path, fix_root: Path, hunte
     if reason is not None:
         return _hunter_outcome(case, family=family, unscorable=reason)
     fix_ranges = _overlay_scan(fix_root).ranges
-    vuln_runs = [hunter(vuln_root) for _ in range(max(runs, 1))]
-    fix_runs = [hunter(fix_root) for _ in range(max(runs, 1))]
-    score = score_pair_family(case, family, vuln_runs, fix_runs, ranges=vuln_ranges, taxonomy=taxonomy)
-    del classify_pair, fix_ranges
+    vuln_runs = [hunter(vuln_root) for _ in range(runs)]
+    fix_runs = [hunter(fix_root) for _ in range(runs)]
+    score = score_pair_family(case, family, vuln_runs, fix_runs, ranges=vuln_ranges, taxonomy=taxonomy, fixed_ranges=fix_ranges)
     return _hunter_outcome(case, family=family, score=score, vuln=vuln_runs[0], fixed=fix_runs[0])
 
 

src/openultrasast/semantic/loo.py

@@ -99,11 +99,14 @@ def evaluate_loo(cases: Sequence[PairCase], *, facts: SemanticFacts | None = Non
             skipped.append((case.name, "pointer_pair_not_cached" if not case.vendored else "excerpt_missing"))
         else:
             targets.append(case)
-    from ..learning.split import is_teacher
+    from ..learning.split import is_teacher, refuse_if_holdout
 
     # learning-harness Req 5.1: every pair is still scored, but only a train-split teacher may seed a store.
     lessons = {case.name: pair_lessons(case, loaded) for case in targets if is_teacher(case)}
     outcomes = [_hold_out(case, targets, lessons, loaded) for case in targets]
+    # Req 5.2: the pairs that were kept out of the teaching set are named, not silently dropped.
+    refusal = refuse_if_holdout([case.name for case in targets if case.name not in lessons], targets)
+    refusals = (refusal.degradation(),) if refusal is not None else ()
     return LooResult(
         outcomes=tuple(outcomes),
         per_slice=_group(outcomes, lambda item: (item.slice,)),
@@ -112,7 +115,7 @@ def evaluate_loo(cases: Sequence[PairCase], *, facts: SemanticFacts | None = Non
         per_obligation_kind=_group(outcomes, lambda item: item.obligations),
         skipped=tuple(skipped),
         teaching_pairs=sum(1 for shapes in lessons.values() if shapes),
-        degradations=_unique_degradations(outcomes),
+        degradations=(*refusals, *_unique_degradations(outcomes)),
     )
 
 

src/openultrasast/semantic/seed.py

@@ -196,7 +196,9 @@ def _skip_reason(case: PairCase) -> str | None:
     from ..learning.split import is_teacher
 
     if not is_teacher(case) and case.review_tier in GATING_TIERS and case.vendored and not case.known_limit:
-        return f"split:{case.split}"  # learning-harness Req 5.1: only train-split, scorable pairs teach
+        # learning-harness Req 5.1: only train-split, scorable pairs teach. Say which of the two it was: two
+        # vibe-py rows are train-split identical twins, and calling them `split:train` is simply untrue.
+        return f"unscorable:{case.unscorable}" if case.unscorable else f"split:{case.split}"
     if case.review_tier not in GATING_TIERS:
         return f"tier:{case.review_tier}"
     if case.known_limit:

src/openultrasast/tool_hunter.py

@@ -175,8 +175,9 @@ class ChatClient(Protocol):
 class ScriptedChatClient:
     """Deterministic ChatClient for tests and OPENULTRASAST_HUNTER_CLIENT injection."""
 
-    def __init__(self, turns: list[ChatResponse]) -> None:
+    def __init__(self, turns: list[ChatResponse], *, cycle: bool = False) -> None:
         self._turns = turns
+        self._cycle = cycle  # a stub that runs out of answers turns every multi-run test into a silent zero
         self.calls: list[dict[str, object]] = []
 
     def complete(
@@ -192,7 +193,9 @@ class ScriptedChatClient:
         self.calls.append({"model": model, "messages": list(messages), "tools": tools, "timeout_seconds": timeout_seconds, **options})
         index = len(self.calls) - 1
         if index >= len(self._turns):
-            return ChatResponse(content="")
+            if not self._cycle or not self._turns:
+                return ChatResponse(content="")
+            index %= len(self._turns)
         return self._turns[index]
 
 
@@ -527,6 +530,10 @@ def _finding_from_item(
     severity = item.get("severity")
     snippet = item.get("snippet")
     proposed_snippet = snippet if isinstance(snippet, str) and snippet else None
+    family = item.get("family")
+    # The family is the only key the class-aware scorer credits, and it is the model's own answer: a fabricated
+    # name is counted as noise there rather than silently dropped here.
+    family_tags = [f"family:{family}"] if isinstance(family, str) and family.strip() else []
     return StaticFinding(
         finding_id=f"{_FINDING_ID_PREFIX}{relative}:{line_no if line_no is not None else 0}",
         path=relative,
@@ -540,7 +547,7 @@ def _finding_from_item(
         reachability_status="unknown",
         reachability_evidence=[],
         reachability_conditions=[],
-        tags=list(tags),
+        tags=[*tags, *family_tags],
         ranking_priority=float(scores.get(relative, 0.0)),
         proposed_snippet=proposed_snippet,
     )

tests/test_learning_cli.py

@@ -19,6 +19,33 @@ CATALOG = (
 )
 
 
+def _detector(monkeypatch: pytest.MonkeyPatch, line: int = 6) -> None:
+    """A deterministic detector that reads the file it was given and answers differently on the two sides.
+
+    The shipped `scripted` stub answers twice and then goes quiet, and its finding sits at line 3, outside `run`.
+    Both make every family number zero, and a scoreboard assertion against zeros passes with the detector missing
+    entirely — which is what these tests used to assert."""
+    from openultrasast.learning.endpoint import ChatEndpoint
+    from openultrasast.tool_hunter import ChatResponse, ToolCall
+
+    class Detector:
+        def __init__(self) -> None:
+            self.turn = 0
+
+        def complete(self, *, model: str, messages, tools=None, **options):  # type: ignore[no-untyped-def]
+            del model, tools, options
+            self.turn += 1
+            if self.turn % 2 == 1:
+                return ChatResponse(tool_calls=(ToolCall(id=f"c{self.turn}", name="grep_repo", arguments={"pattern": "system"}),))
+            body = "\n".join(str(message.get("content") or "") for message in messages)
+            if "os.system" not in body:
+                return ChatResponse(content="[]")  # the fixed side really is different, and the detector sees it
+            return ChatResponse(content=json.dumps([{"path": "app.py", "line": line, "title": "t", "rationale": "r"}]))
+
+    endpoint = ChatEndpoint(provider="scripted", base_url="", thinking=False)
+    monkeypatch.setattr("openultrasast.learning.endpoint.resolve_chat_endpoint", lambda *a, **k: (Detector(), endpoint))
+
+
 @pytest.fixture
 def workspace(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
     for name in ("train-a", "hold-a"):
@@ -39,14 +66,25 @@ def test_classify_reports_the_corpus_and_writes_the_queue(workspace: Path, capsy
     assert "classified 2" in capsys.readouterr().out
 
 
-def test_score_reports_family_numbers_for_a_catalog(workspace: Path, capsys: pytest.CaptureFixture[str]) -> None:
+def test_score_reports_family_numbers_for_a_catalog(
+    workspace: Path, capsys: pytest.CaptureFixture[str], monkeypatch: pytest.MonkeyPatch
+) -> None:
+    """The scripted client detects the vulnerable side and is silent on the fix, so this is a real 2/2, not a zero.
+
+    An all-zero scoreboard would satisfy "the family key exists"; it is what a generalist detector scoring every
+    family produced before round zero ran each family's own configuration."""
+    _detector(monkeypatch)
     assert main(["learning", "score", "--catalog", str(workspace / "pairs.toml"), "--out", str(workspace / "learning"), "--json"]) == 0
     payload = json.loads(capsys.readouterr().out)
-    assert "injection" in payload["families"]
-    assert payload["families"]["injection"]["taxonomy_version"]
+    block = payload["families"]["injection"]
+    assert block["taxonomy_version"] and block["scorable"] == 2
+    assert block["outcomes"] == {"pair_correct": 2} and block["recall"] == 1.0 and block["youden"] == 1.0
 
 
-def test_baseline_writes_the_noise_floor_and_publish_reads_it(workspace: Path, capsys: pytest.CaptureFixture[str]) -> None:
+def test_baseline_writes_the_noise_floor_and_publish_reads_it(
+    workspace: Path, capsys: pytest.CaptureFixture[str], monkeypatch: pytest.MonkeyPatch
+) -> None:
+    _detector(monkeypatch)
     assert (
         main(
             [
@@ -65,7 +103,12 @@ def test_baseline_writes_the_noise_floor_and_publish_reads_it(workspace: Path, c
         == 0
     )
     floors = json.loads((workspace / "learning" / "baseline" / "scripted" / "noise-floors.json").read_text())
-    assert floors["k_runs"] == 3 and "injection" in floors["floors"]
+    assert floors["k_runs"] == 3 and floors["floors"]["injection"]["pairs"] == 2
+    report = json.loads((workspace / "learning" / "baseline" / "scripted" / "report.json").read_text())
+    # Measured with the injection configuration, not the generalist: `family:unknown` earns no credit against any
+    # real family, so a baseline built from one shared detector reports a structural zero here.
+    assert report["metrics"]["injection"]["recall"] == 1.0
+    assert report["per_slice"]["vibe-py/injection"]["scorable"] == 2
     capsys.readouterr()
     roadmap = workspace / "roadmap.md"
     roadmap.write_text("# Roadmap\n\n## Overview\n\ntext\n")
@@ -116,10 +159,46 @@ def test_a_round_runs_from_the_command_line_and_journals_its_outcome(workspace:
     )
     rounds = [json.loads(line) for line in (workspace / "learning" / "journal.jsonl").read_text().splitlines() if line.strip()]
     assert len(rounds) == 1 and rounds[0]["family"] == "injection"
-    assert rounds[0]["outcome"] in {"accepted", "rejected", "reverted_cost"}
+    # Without the extra there is no meta-agent, so the honest outcome is exactly this one. Accepting any member of
+    # the enum would have hidden that the round could never propose anything at all.
+    assert (rounds[0]["outcome"], rounds[0]["reason"]) == ("rejected", "no_proposal")
     assert (workspace / "learning" / "rounds" / "1" / "attribution.json").is_file()
 
 
+def test_a_recorded_proposal_makes_the_round_do_something(workspace: Path) -> None:
+    """`--proposals` is what makes a round runnable at all without the optional extra."""
+    for command in (
+        ["learning", "baseline", "--catalog", str(workspace / "pairs.toml"), "--out", str(workspace / "learning"), "--k-runs", "3"],
+    ):
+        assert main([*command, "--model", "scripted"]) == 0
+    proposals = workspace / "proposals.json"
+    proposals.write_text(
+        json.dumps([{"hypothesis": "ask about quoting", "lever": "checklist", "change": {"checklist.md": "- ask about quoting\n"}}])
+    )
+    assert (
+        main(
+            [
+                "learning",
+                "round",
+                "--catalog",
+                str(workspace / "pairs.toml"),
+                "--out",
+                str(workspace / "learning"),
+                "--family",
+                "injection",
+                "--k-runs",
+                "3",
+                "--proposals",
+                str(proposals),
+            ]
+        )
+        == 0
+    )
+    (record,) = [json.loads(line) for line in (workspace / "learning" / "journal.jsonl").read_text().splitlines() if line.strip()]
+    assert record["reason"] != "no_proposal", "a recorded proposal reached the round"
+    assert json.loads((workspace / "learning" / "rounds" / "1" / "proposal.json").read_text())["lever"] == "checklist"
+
+
 def test_a_missing_endpoint_is_recorded_and_the_rest_still_runs(workspace: Path, monkeypatch: pytest.MonkeyPatch, capsys) -> None:  # type: ignore[no-untyped-def]
     for name in ("OPENULTRASAST_HUNTER_CLIENT", "DEEPSEEK_API_KEY", "OPENROUTER_API_KEY"):
         monkeypatch.delenv(name, raising=False)
@@ -153,7 +232,24 @@ def test_every_subcommand_is_listed_in_help(capsys: pytest.CaptureFixture[str])
         assert name in out
 
 
-def test_the_pairs_command_takes_a_k_runs_flag(capsys: pytest.CaptureFixture[str]) -> None:
-    with pytest.raises(SystemExit):
-        main(["pairs", "--help"])
-    assert "--k-runs" in capsys.readouterr().out
+def test_the_pairs_command_scores_k_runs_per_side(workspace: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+    """Grepping `--help` for the flag passes with the capability entirely absent, which is what it used to be."""
+    from openultrasast import pairs as pairs_module
+
+    calls: list[str] = []
+    real = pairs_module.make_hunter_scan
+
+    def counting(client: object, model: str, **kwargs: object):  # type: ignore[no-untyped-def]
+        inner = real(client, model, **kwargs)  # type: ignore[arg-type]
+
+        def scan(root: Path):  # type: ignore[no-untyped-def]
+            calls.append(root.name)
+            return inner(root)
+
+        return scan
+
+    monkeypatch.setattr(pairs_module, "make_hunter_scan", counting)
+    monkeypatch.setattr("openultrasast.cli.make_hunter_scan", counting)
+    monkeypatch.setenv("OPENULTRASAST_HUNTER_MODEL", "scripted")
+    assert main(["pairs", "--catalog", str(workspace / "pairs.toml"), "--hunter", "--hunter-model", "scripted", "--k-runs", "4"]) == 0
+    assert calls.count("vuln") == 8 and calls.count("fixed") == 8  # two pairs, four runs per side

tests/test_learning_pairs_evidence.py

@@ -0,0 +1,151 @@
+"""What the pairs hunter path must prove (learning-harness review round 3, Req 3.2, 3.5).
+
+Three defects, one test each: K was declared and never threaded, so every stage scored one run; the fixed
+side was judged against the *vulnerable* side's function ranges, which differ on most of the corpus; and
+the generalist hunter emitted findings with no family tag, which the class-aware scorer can never credit,
+so `ousast pairs --hunter` reported recall zero by construction.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import pytest
+
+from openultrasast.benchmark import ExpectedFinding
+from openultrasast.findings import StaticFinding
+from openultrasast.learning.families import load_families
+from openultrasast.pairs import PairCase, evaluate_catalog
+
+# The fix inserts two guard lines above the handler, so `run` starts on a different line on each side.
+VULN = "from flask import request\nimport os\n\n\ndef run():\n    os.system(request.args.get('cmd'))\n"
+FIXED = (
+    "from flask import request\nimport shlex\nimport os\n\n\n"
+    "def _safe(value):\n    return shlex.quote(value or '')\n\n\n"
+    "def run():\n    os.system(_safe(request.args.get('cmd')))\n"
+)
+
+
+def _case(tmp_path: Path, name: str = "p") -> PairCase:
+    (tmp_path / f"{name}-v.py").write_text(VULN)
+    (tmp_path / f"{name}-f.py").write_text(FIXED)
+    return PairCase(
+        name=name,
+        slice="vibe-py",
+        language="python",
+        origin="test",
+        vuln_file=tmp_path / f"{name}-v.py",
+        fixed_file=tmp_path / f"{name}-f.py",
+        relpath="app.py",
+        expected=(
+            ExpectedFinding(
+                cwe="CWE-78",
+                vulnerability_class="x",
+                path="app.py",
+                evidence="",
+                function="run",
+                mechanism="other",
+                family="injection",
+            ),
+        ),
+        min_recall=1.0,
+        fix_policy="silent",
+        review_tier="seeded",
+    )
+
+
+def _finding(line: int, *, family: str | None = "injection") -> StaticFinding:
+    return StaticFinding(
+        finding_id=f"tool-hunter:app.py:{line}",
+        path="app.py",
+        title="t",
+        severity="high",
+        confidence="low",
+        evidence_level="suspicion",
+        rationale="r",
+        line=line,
+        function_name="run",
+        reachability_status="unknown",
+        reachability_evidence=[],
+        reachability_conditions=[],
+        tags=["tool-hunter", *([f"family:{family}"] if family else [])],
+        ranking_priority=0.0,
+    )
+
+
+def test_the_k_runs_flag_reaches_the_detector(tmp_path: Path) -> None:
+    """Req 3.5: three runs per side, per pair. A parameter no caller can set is not a K-run floor."""
+    calls: list[str] = []
+
+    def scan(root: Path) -> list[StaticFinding]:
+        calls.append(root.name)
+        return [_finding(6)] if root.name == "vuln" else []
+
+    evaluate_catalog([_case(tmp_path)], hunter=scan, k_runs=3)
+    assert calls.count("vuln") == 3 and calls.count("fixed") == 3
+
+
+def test_fewer_than_three_runs_is_refused(tmp_path: Path) -> None:
+    with pytest.raises(ValueError, match="three runs"):
+        evaluate_catalog([_case(tmp_path)], hunter=lambda root: [], k_runs=1)
+
+
+def test_the_fixed_side_is_judged_against_the_fixed_side_line_numbers(tmp_path: Path) -> None:
+    """`run` sits at line 5 on the vulnerable side and line 10 on the fixed side.
+
+    Scoring the fixed side against the vulnerable side's span reads a finding inside the fixed handler as
+    outside it, so a detector that flags both sides is recorded as silent on the fix — a leak counted as a
+    clean pass, and always in the direction that flatters the number."""
+
+    def scan(root: Path) -> list[StaticFinding]:
+        return [_finding(6)] if root.name == "vuln" else [_finding(11)]
+
+    result = evaluate_catalog([_case(tmp_path)], hunter=scan, k_runs=3)
+    hunter = result.scorers["vibe-py"]["hunter"]
+    assert hunter.detected_vuln == 1
+    assert hunter.silent_fix == 0, "the finding is inside `run` on the fixed side; that is a leak"
+    assert hunter.pair_correct == 0
+
+
+def test_a_generalist_finding_without_a_family_is_never_a_detection(tmp_path: Path) -> None:
+    """The other half of the same defect: `pairs --hunter` used to feed the scorer untagged findings."""
+
+    def scan(root: Path) -> list[StaticFinding]:
+        return [_finding(6, family=None)] if root.name == "vuln" else []
+
+    result = evaluate_catalog([_case(tmp_path)], hunter=scan, k_runs=3)
+    assert result.scorers["vibe-py"]["hunter"].detected_vuln == 0
+
+
+def test_the_hunter_reports_the_family_it_believes_and_that_is_what_is_credited(tmp_path: Path) -> None:
+    """A generalist that must name the family is answerable; one that names nothing cannot be scored at all."""
+    from openultrasast.tool_hunter import _findings_from_content
+
+    findings = _findings_from_content(
+        tmp_path,
+        [],
+        '[{"path": "app.py", "line": 6, "family": "injection", "title": "t", "rationale": "r"}]',
+    )
+    assert findings and "family:injection" in findings[0].tags
+    untagged = _findings_from_content(tmp_path, [], '[{"path": "app.py", "line": 6, "title": "t"}]')
+    assert untagged and not [tag for tag in untagged[0].tags if tag.startswith("family:")]
+
+
+def test_the_pair_hunter_asks_for_a_family_from_the_closed_list(tmp_path: Path) -> None:
+    """The prompt has to name the taxonomy, or the model invents family names and every finding is noise."""
+    from openultrasast.pairs import make_hunter_scan
+
+    seen: list[str] = []
+
+    class Client:
+        def complete(self, *, model: str, messages, tools=None, **options):  # type: ignore[no-untyped-def]
+            seen.append("\n".join(str(message.get("content") or "") for message in messages))
+            from openultrasast.tool_hunter import ChatResponse
+
+            return ChatResponse(content="[]", tool_calls=())
+
+    (tmp_path / "app.py").write_text(VULN)
+    make_hunter_scan(Client(), "m", taxonomy=load_families())(tmp_path)
+    prompt = "\n".join(seen)
+    assert "injection" in prompt and "access_control" in prompt and "untrusted_destination" in prompt
+    assert '"family"' in prompt

tests/test_learning_proposer.py

@@ -137,3 +137,86 @@ def test_the_harnessx_proposer_degrades_with_a_reason_when_the_extra_is_absent(t
     assert agent.propose(facts) is None
     assert agent.reason == "harnessx_unavailable"
     assert agent.write_roots("access_control") == (tmp_path / "configs" / "access_control",)
+
+
+# --- the meta-agent proposer (review round 3, task 3.5 finding 1) ------------
+
+
+def test_the_meta_agent_proposer_reads_its_change_out_of_a_scratch_copy(tmp_path: Path) -> None:
+    """The meta-agent never writes into the live family directory.
+
+    A proposer that edited the tree directly would have changed the configuration before the round snapshotted
+    it, so a rejection could not put it back. It gets a copy, and the proposal is the diff of that copy."""
+    from openultrasast.learning.proposer import HarnessXProposer
+
+    configs = tmp_path / "configs"
+    (configs / "injection").mkdir(parents=True)
+    (configs / "injection" / "checklist.md").write_text("- old\n")
+    (configs / "injection" / "family.toml").write_text('family = "injection"\nversion = "0"\n')
+    seen: dict[str, object] = {}
+
+    def agent(*, workspace: Path, facts, model: str):  # type: ignore[no-untyped-def]
+        seen["workspace"] = workspace
+        seen["family"] = facts.family
+        (workspace / "checklist.md").write_text("- ask about quoting\n")
+        return "ask about quoting"
+
+    proposer = HarnessXProposer(configs_dir=configs, model="m", agent=agent)
+    proposal = proposer.propose(_facts(tmp_path, "injection"))
+    assert proposal is not None
+    assert proposal.lever == "checklist" and proposal.change == {"checklist.md": "- ask about quoting\n"}
+    assert proposal.hypothesis == "ask about quoting"
+    assert seen["workspace"] != configs / "injection"
+    assert (configs / "injection" / "checklist.md").read_text() == "- old\n", "the live tree was edited"
+
+
+def test_the_meta_agent_proposer_refuses_more_than_one_changed_file(tmp_path: Path) -> None:
+    from openultrasast.learning.proposer import HarnessXProposer
+
+    configs = tmp_path / "configs"
+    (configs / "injection").mkdir(parents=True)
+    (configs / "injection" / "checklist.md").write_text("- old\n")
+    (configs / "injection" / "prompt.md").write_text("p\n")
+
+    def agent(*, workspace: Path, facts, model: str):  # type: ignore[no-untyped-def]
+        (workspace / "checklist.md").write_text("- a\n")
+        (workspace / "prompt.md").write_text("- b\n")
+        return "two at once"
+
+    proposer = HarnessXProposer(configs_dir=configs, model="m", agent=agent)
+    assert proposer.propose(_facts(tmp_path, "injection")) is None
+    assert "one file" in proposer.reason
+
+
+def test_the_meta_agent_proposer_refuses_a_file_no_lever_owns(tmp_path: Path) -> None:
+    from openultrasast.learning.proposer import HarnessXProposer
+
+    configs = tmp_path / "configs"
+    (configs / "injection").mkdir(parents=True)
+    (configs / "injection" / "checklist.md").write_text("- old\n")
+
+    def agent(*, workspace: Path, facts, model: str):  # type: ignore[no-untyped-def]
+        (workspace / "notes.txt").write_text("x\n")
+        return "sideways"
+
+    proposer = HarnessXProposer(configs_dir=configs, model="m", agent=agent)
+    assert proposer.propose(_facts(tmp_path, "injection")) is None
+    assert "notes.txt" in proposer.reason
+
+
+def test_the_meta_agent_proposer_says_so_when_it_changed_nothing(tmp_path: Path) -> None:
+    from openultrasast.learning.proposer import HarnessXProposer
+
+    configs = tmp_path / "configs"
+    (configs / "injection").mkdir(parents=True)
+    (configs / "injection" / "checklist.md").write_text("- old\n")
+    proposer = HarnessXProposer(configs_dir=configs, model="m", agent=lambda **_kwargs: "nothing to do")
+    assert proposer.propose(_facts(tmp_path, "injection")) is None
+    assert proposer.reason == "no_change_proposed"
+
+
+def _facts(tmp_path: Path, family: str):  # type: ignore[no-untyped-def]
+    from openultrasast.learning.proposer import FailureFacts
+
+    del tmp_path
+    return FailureFacts(family=family, config=_config(family))

tests/test_learning_publish.py

@@ -107,13 +107,22 @@ def test_the_corrected_lever_number_replaces_the_earlier_one(tmp_path: Path) ->
         )
     )
     roadmap = tmp_path / "roadmap.md"
-    roadmap.write_text(ROADMAP + "\nThe lever recovered 7 of 17 holdout pairs at Youden +0.059.\n")
+    stale = "human holdout profile pair_correct 0 -> 6 of 17, Youden 0 -> +0.353, round 2 `no_proposals`."
+    roadmap.write_text(f"{ROADMAP}\nMeasured 2026-09-05 with the vibe-py candidates: {stale}\n")
     report = publish(learning_dir=out, measurements_dir=measurements, roadmap=roadmap)
     text = roadmap.read_text()
     assert "-0.0588" in report.table or "-0.059" in report.table
     assert "5 of 17" in report.table or "detected 5" in report.table
     assert "holdout pairs stopped teaching" in report.table.lower() or "train split only" in report.table.lower()
     assert text.count("learning-harness:begin") == 1
+    # Req 5.3 says *in place of* the earlier one. Leaving the old claim above the new section is how a document
+    # ends up asserting two different numbers for the same measurement.
+    assert stale not in text
+    assert "superseded" in text and "learning-harness" in text
+    assert "Measured 2026-09-05 with the vibe-py candidates:" in text, "the sentence is corrected, not deleted"
+    before = roadmap.read_text()
+    publish(learning_dir=out, measurements_dir=measurements, roadmap=roadmap)
+    assert roadmap.read_text() == before, "a second publish must be a no-op once the claim is corrected"
 
 
 def test_the_roadmap_section_is_replaced_not_appended_again(tmp_path: Path) -> None:

tests/test_learning_refusals.py

@@ -0,0 +1,94 @@
+"""A refusal is data, not a `continue` (learning-harness review round 3, Req 5.1, 5.2).
+
+`Refusal.degradation()` existed and nothing ever called it, so every path that declined to learn from a pair
+declined silently: the mechanism lever skipped a holdout-taught shape without a word, leave-one-out dropped
+non-teachers with no record, and the seeder gave a train-split identical twin the reason `split:train`, which
+is not true. A leak the run does not mention is a leak nobody reads.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+from openultrasast.benchmark import ExpectedFinding
+from openultrasast.pairs import PairCase
+
+VULN = "import os\nfrom flask import request\n\n\ndef run():\n    os.system(request.args.get('cmd'))\n"
+FIXED = "import subprocess\nfrom flask import request\n\n\ndef run():\n    subprocess.run(['echo'], check=False)\n"
+
+
+def _case(tmp_path: Path, name: str, *, split: str = "train", unscorable: str | None = None, tier: str = "seeded") -> PairCase:
+    (tmp_path / f"{name}-v.py").write_text(VULN)
+    (tmp_path / f"{name}-f.py").write_text(FIXED)
+    return PairCase(
+        name=name,
+        slice="vibe-py",
+        language="python",
+        origin="test",
+        vuln_file=tmp_path / f"{name}-v.py",
+        fixed_file=tmp_path / f"{name}-f.py",
+        relpath="app.py",
+        expected=(ExpectedFinding(cwe="CWE-78", vulnerability_class="x", path="app.py", evidence="", function="run", mechanism="other"),),
+        min_recall=1.0,
+        fix_policy="silent",
+        split=split,
+        review_tier=tier,
+        unscorable=unscorable,
+    )
+
+
+def test_a_train_split_twin_is_refused_for_being_a_twin_not_for_its_split(tmp_path: Path) -> None:
+    """`split:train` on a train-split row is a lie, and it hides a corpus fact behind a split fact.
+
+    Two vibe-py rows are exactly this: `python-insecure-app-main-try-hack-me` and
+    `vulnpy-deserialization-do-pickle-load` are train-split identical twins."""
+    from openultrasast.semantic.seed import _skip_reason
+
+    assert _skip_reason(_case(tmp_path, "twin", split="train", unscorable="identical_twin")) == "unscorable:identical_twin"
+    assert _skip_reason(_case(tmp_path, "held", split="holdout")) == "split:holdout"
+    assert _skip_reason(_case(tmp_path, "good", split="train")) is None
+
+
+def test_leave_one_out_records_the_pairs_it_would_not_learn_from(tmp_path: Path) -> None:
+    """Req 5.2: the refusal is recorded with the pair names, on a channel a caller can read."""
+    from openultrasast.semantic.loo import evaluate_loo
+
+    result = evaluate_loo([_case(tmp_path, "train-a"), _case(tmp_path, "hold-a", split="holdout")])
+    refusals = [item for item in result.degradations if item.get("reason") == "holdout_pair_refused"]
+    assert refusals, "leave-one-out excluded a holdout pair from teaching and said nothing"
+    assert refusals[0]["pairs"] == ["hold-a"]
+    assert result.teaching_pairs <= 1
+
+
+def test_the_mechanism_lever_records_every_shape_a_holdout_pair_taught(tmp_path: Path) -> None:
+    """The lever used to `continue` past a refused candidate; the run then showed a shorter list with no reason."""
+    from openultrasast.improve.evolve import propose_mechanism_edits
+    from openultrasast.semantic.mechanisms import MechanismStore, append_from_pair
+    from openultrasast.semantic.variants import Shape
+
+    cases = [_case(tmp_path, "train-a"), _case(tmp_path, "hold-a", split="holdout")]
+    candidates = MechanismStore(tmp_path / "candidates.jsonl")
+    append_from_pair(
+        candidates,
+        Shape(
+            language="python",
+            sink_name="system",
+            arity=1,
+            source_positions=(0,),
+            source_kinds=("request",),
+            guard="none",
+            mechanism="source_reaches_sink",
+        ),
+        summary="s",
+        cwe="CWE-78",
+        pair="hold-a",
+        provenance="human",
+        tier="seeded",
+    )
+    refusals: list[dict[str, object]] = []
+    edits = propose_mechanism_edits(cases, candidates, tmp_path / "scan.jsonl", degradations=refusals)
+    assert [edit.mechanism_id for edit in edits if edit.action == "admit"] == []
+    assert [item["reason"] for item in refusals] == ["holdout_pair_refused"]
+    assert refusals[0]["pairs"] == ["hold-a"]
+    json.dumps(refusals)  # a degradation has to survive the artifact it is written into

tests/test_learning_round_evidence.py

@@ -0,0 +1,375 @@
+"""What a round must prove before it is kept (learning-harness review round 3, Req 4.1, 5.2, 9.5, 9.6, 9.8).
+
+The round loop was accepting and rejecting on numbers that did not mean what their names said: the
+cross-family sweep counted every failing pair rather than the pairs this change broke, unscorable rows
+voted against a round they cannot judge, cost was metered once after everything had been paid for, and a
+stage that raised left no record at all. Each test here fails on exactly one of those.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+import pytest
+
+from openultrasast.benchmark import ExpectedFinding
+from openultrasast.findings import StaticFinding
+from openultrasast.learning.detectors import FamilyConfig, write_default_configs
+from openultrasast.learning.families import load_families
+from openultrasast.learning.journal import Archive, LearningJournal
+from openultrasast.learning.proposer import Proposal, ScriptedProposer
+from openultrasast.learning.rounds import NoiseFloor, run_learning_round
+from openultrasast.pairs import PairCase
+
+VULN = "from flask import request\nimport os\n\n\ndef run():\n    os.system(request.args.get('cmd'))\n"
+FIXED = "from flask import request\nimport subprocess\n\n\ndef run():\n    subprocess.run(['echo', 'x'], check=False)\n"
+
+
+def _case(tmp_path: Path, name: str, *, split: str, family: str = "injection", unscorable: str | None = None) -> PairCase:
+    (tmp_path / f"{name}-v.py").write_text(VULN)
+    (tmp_path / f"{name}-f.py").write_text(FIXED)
+    return PairCase(
+        name=name,
+        slice="vibe-py",
+        language="python",
+        origin="test",
+        vuln_file=tmp_path / f"{name}-v.py",
+        fixed_file=tmp_path / f"{name}-f.py",
+        relpath="app.py",
+        expected=(
+            ExpectedFinding(
+                cwe="CWE-78", vulnerability_class="x", path="app.py", evidence="", function="run", mechanism="other", family=family
+            ),
+        ),
+        min_recall=1.0,
+        fix_policy="silent",
+        split=split,
+        review_tier="seeded",
+        unscorable=unscorable,
+    )
+
+
+def _finding(family: str) -> StaticFinding:
+    return StaticFinding(
+        finding_id="detector:app.py:6",
+        path="app.py",
+        title="t",
+        severity="high",
+        confidence="low",
+        evidence_level="suspicion",
+        rationale="r",
+        line=6,
+        function_name="run",
+        reachability_status="unknown",
+        reachability_evidence=[],
+        reachability_conditions=[],
+        tags=[f"family:{family}", "detector:x@0"],
+        ranking_priority=0.0,
+    )
+
+
+def _proposal(text: str = "- ask about quoting\n") -> Proposal:
+    return Proposal(hypothesis="ask about quoting", lever="checklist", change={"checklist.md": text}, predicted_affected=("train-a",))
+
+
+@pytest.fixture
+def world(tmp_path: Path):  # type: ignore[no-untyped-def]
+    configs = tmp_path / "configs"
+    write_default_configs(configs, load_families(), prompt="p", version="0")
+    (configs / "injection" / "checklist.md").write_text("- old\n")
+    return {
+        "configs": configs,
+        "cases": [_case(tmp_path, "train-a", split="train"), _case(tmp_path, "hold-a", split="holdout")],
+        "journal": LearningJournal(tmp_path / "journal.jsonl"),
+        "archive": Archive(tmp_path / "archive.jsonl"),
+        "out": tmp_path / "out",
+        "floors": {"injection": NoiseFloor(family="injection", k_runs=5, pairs=1, flaky_pairs=0, negative_flip_rate=0.0, budget_flips=0)},
+        "root": tmp_path,
+    }
+
+
+def _run(world, proposal, build, **kwargs):  # type: ignore[no-untyped-def]
+    return run_learning_round(
+        world["cases"],
+        family="injection",
+        taxonomy=load_families(),
+        configs_dir=world["configs"],
+        proposer=ScriptedProposer([proposal] if proposal else []),
+        scan_factory=build,
+        model="m",
+        journal=world["journal"],
+        archive=world["archive"],
+        floors=world["floors"],
+        out_dir=world["out"],
+        **kwargs,
+    )
+
+
+def _injection_improves(config: FamilyConfig):  # type: ignore[no-untyped-def]
+    """Injection detects only once the proposed checklist lands; every other family finds nothing, ever."""
+
+    def scan(root: Path) -> list[StaticFinding]:
+        if root.name != "vuln" or config.family != "injection":
+            return []
+        return [_finding("injection")] if config.checklist.strip() == "- ask about quoting" else []
+
+    return scan
+
+
+# --- the sweep ---------------------------------------------------------------
+
+
+def test_a_family_that_was_already_failing_does_not_reject_the_round(world) -> None:  # type: ignore[no-untyped-def]
+    """Req 9.5 counts *negative flips*, not absolute failures.
+
+    On the real corpus most pairs fail at round zero, so counting failures rejects every round for damage it
+    did not do — and the harness then reports that it cannot improve."""
+    world["cases"].append(_case(world["root"], "other-hold", split="holdout", family="access_control"))
+    world["floors"]["access_control"] = NoiseFloor(
+        family="access_control", k_runs=5, pairs=1, flaky_pairs=0, negative_flip_rate=0.0, budget_flips=0
+    )
+    record = _run(world, _proposal(), _injection_improves, sweep_families=("access_control",))
+    assert record.sweep["access_control"] == 0, "access control failed before and after; this change flipped nothing"
+    assert record.outcome == "accepted" and record.reason == "improved"
+
+
+def test_a_family_this_change_really_broke_still_rejects_the_round(world) -> None:  # type: ignore[no-untyped-def]
+    """The other arm: a pair that was correct before and is not after is a flip, and one is over the budget."""
+    world["cases"].append(_case(world["root"], "other-hold", split="holdout", family="access_control"))
+    world["floors"]["access_control"] = NoiseFloor(
+        family="access_control", k_runs=5, pairs=1, flaky_pairs=0, negative_flip_rate=0.0, budget_flips=0
+    )
+    landed = {"yes": False}
+
+    def build(config: FamilyConfig):  # type: ignore[no-untyped-def]
+        def scan(root: Path) -> list[StaticFinding]:
+            if root.name != "vuln":
+                return []
+            if config.family == "access_control":
+                return [] if landed["yes"] else [_finding("access_control")]
+            if config.checklist.strip() == "- ask about quoting":
+                landed["yes"] = True
+                return [_finding("injection")]
+            return []
+
+        return scan
+
+    record = _run(world, _proposal(), build, sweep_families=("access_control",))
+    assert record.sweep["access_control"] == 1
+    assert record.outcome == "rejected" and record.reason == "collateral_regression"
+    assert (world["configs"] / "injection" / "checklist.md").read_text() == "- old\n"
+
+
+def test_an_unscorable_row_never_votes_in_the_sweep(world) -> None:  # type: ignore[no-untyped-def]
+    """Req 4.1: an unscorable row leaves every denominator, and it certainly does not reject a round."""
+    world["cases"].append(_case(world["root"], "other-hold", split="holdout", family="access_control", unscorable="identical_twin"))
+    world["floors"]["access_control"] = NoiseFloor(
+        family="access_control", k_runs=5, pairs=0, flaky_pairs=0, negative_flip_rate=0.0, budget_flips=0
+    )
+    record = _run(world, _proposal(), _injection_improves, sweep_families=("access_control",))
+    assert record.sweep["access_control"] == 0 and record.outcome == "accepted"
+
+
+def test_attribution_names_the_pairs_this_change_flipped_and_no_others(world) -> None:  # type: ignore[no-untyped-def]
+    world["cases"].append(_case(world["root"], "other-hold", split="holdout", family="access_control"))
+    world["floors"]["access_control"] = NoiseFloor(
+        family="access_control", k_runs=5, pairs=1, flaky_pairs=0, negative_flip_rate=0.0, budget_flips=0
+    )
+    _run(world, _proposal(), _injection_improves, sweep_families=("access_control",))
+    attribution = json.loads((world["out"] / "rounds" / "1" / "attribution.json").read_text())
+    assert attribution["flipped_predicted"] == ["hold-a", "train-a"]
+    assert attribution["flipped_unpredicted"] == [], "a pair that was already failing was never flipped by this round"
+
+
+# --- cost, exceptions, the archive ------------------------------------------
+
+
+def test_the_cap_stops_a_round_before_it_pays_for_the_next_stage(world) -> None:  # type: ignore[no-untyped-def]
+    """Req 9.8 says stop, revert and record. Metering once, after every stage, can only ever record."""
+    stages: list[str] = []
+
+    def build(config: FamilyConfig):  # type: ignore[no-untyped-def]
+        def scan(root: Path) -> list[StaticFinding]:
+            stages.append(f"{config.family}:{root.name}")
+            return []
+
+        return scan
+
+    spend = {"usd": 0.0}
+
+    def spent() -> float:
+        spend["usd"] += 1.0
+        return spend["usd"]
+
+    world["cases"].append(_case(world["root"], "other-hold", split="holdout", family="access_control"))
+    record = _run(world, _proposal(), build, cost_cap_usd=0.5, spent_usd=spent, sweep_families=("access_control",))
+    assert record.outcome == "reverted_cost" and record.reason == "cost_cap_exceeded"
+    assert "access_control:vuln" not in stages, "the sweep ran after the cap was already crossed"
+    assert (world["configs"] / "injection" / "checklist.md").read_text() == "- old\n"
+
+
+def test_a_stage_that_raises_is_reverted_journalled_and_re_raised(world) -> None:  # type: ignore[no-untyped-def]
+    """An endpoint failure mid-round is an outcome. Losing the round record loses the money it already spent."""
+
+    def build(config: FamilyConfig):  # type: ignore[no-untyped-def]
+        def scan(root: Path) -> list[StaticFinding]:
+            if config.checklist.strip() == "- ask about quoting":
+                raise RuntimeError("endpoint said no")
+            return []
+
+        return scan
+
+    with pytest.raises(RuntimeError, match="endpoint said no"):
+        _run(world, _proposal(), build)
+    assert (world["configs"] / "injection" / "checklist.md").read_text() == "- old\n"
+    outcomes = [(item.outcome, item.reason) for item in world["journal"].rounds()]
+    assert outcomes == [("reverted", "stage_failed")]
+    assert (world["out"] / "rounds" / "1" / "scores.json").is_file()
+
+
+def test_a_rejected_version_is_archived_so_a_pair_it_alone_wins_is_not_lost(world) -> None:  # type: ignore[no-untyped-def]
+    """Req 9.7: per-pair winners, or the loop collapses onto one lineage and forgets what a rejected try could do.
+
+    This round wins both of its own pairs and is still rejected, because it broke another family. Version 1 is the
+    only configuration that ever got those two pairs right; dropping it because the round as a whole lost is how a
+    search collapses onto one lineage."""
+    world["cases"].append(_case(world["root"], "other-hold", split="holdout", family="access_control"))
+    world["floors"]["access_control"] = NoiseFloor(
+        family="access_control", k_runs=5, pairs=1, flaky_pairs=0, negative_flip_rate=0.0, budget_flips=0
+    )
+    landed = {"yes": False}
+
+    def build(config: FamilyConfig):  # type: ignore[no-untyped-def]
+        def scan(root: Path) -> list[StaticFinding]:
+            if root.name != "vuln":
+                return []
+            if config.family == "access_control":
+                return [] if landed["yes"] else [_finding("access_control")]
+            if config.checklist.strip() == "- ask about quoting":
+                landed["yes"] = True
+                return [_finding("injection")]
+            return []
+
+        return scan
+
+    record = _run(world, _proposal(), build, sweep_families=("access_control",))
+    assert record.outcome == "rejected" and record.reason == "collateral_regression"
+    assert world["archive"].winners("injection") == {"train-a": "1", "hold-a": "1"}
+    assert 'version = "0"' in (world["configs"] / "injection" / "family.toml").read_text(), "a rejected round is not versioned"
+
+
+# --- the holdout refusal -----------------------------------------------------
+
+
+def test_a_proposal_that_names_a_holdout_pair_is_refused_by_name(world) -> None:  # type: ignore[no-untyped-def]
+    """Req 5.2: a candidate that touches a holdout pair is refused and the refusal names the pairs."""
+    record = _run(
+        world,
+        Proposal(
+            hypothesis="memorize the answer",
+            lever="memory",
+            change={"hard_negatives.jsonl": json.dumps({"pair": "hold-a", "note": "always flag"}) + "\n"},
+            predicted_affected=("hold-a",),
+        ),
+        _injection_improves,
+    )
+    assert record.outcome == "refused_holdout"
+    assert "hold-a" in record.reason
+    assert (world["configs"] / "injection" / "hard_negatives.jsonl").read_text() == ""
+
+
+def test_a_proposal_that_only_names_train_pairs_is_not_refused(world) -> None:  # type: ignore[no-untyped-def]
+    record = _run(
+        world,
+        Proposal(
+            hypothesis="remember this shape",
+            lever="memory",
+            change={"hard_negatives.jsonl": json.dumps({"pair": "train-a", "note": "shape"}) + "\n"},
+            predicted_affected=("train-a",),
+        ),
+        _injection_improves,
+    )
+    assert record.outcome != "refused_holdout"
+
+
+# --- round zero --------------------------------------------------------------
+
+
+def test_round_zero_measures_each_family_with_its_own_detector(tmp_path: Path) -> None:
+    """Req 8.2. One detector for every family means the generalist scores them all, and abstention earns no credit
+    (`related(X, "unknown")` is lateral), so every family but `unknown` baselines at a structural zero."""
+    from openultrasast.learning.rounds import run_baseline
+
+    configs = tmp_path / "configs"
+    write_default_configs(configs, load_families(), prompt="p", version="0")
+    seen: list[str] = []
+
+    def factory(config: FamilyConfig):  # type: ignore[no-untyped-def]
+        def scan(root: Path) -> list[StaticFinding]:
+            seen.append(config.family)
+            return [_finding(config.family)] if root.name == "vuln" else []
+
+        return scan
+
+    cases = [_case(tmp_path, "inj", split="train"), _case(tmp_path, "acl", split="train", family="access_control")]
+    report = run_baseline(
+        cases,
+        taxonomy=load_families(),
+        configs_dir=configs,
+        scan_factory=factory,
+        model="m",
+        out_dir=tmp_path / "out",
+        k_runs=3,
+    )
+    assert set(seen) == {"injection", "access_control"}, "each family must be measured with its own configuration"
+    assert report.metrics["injection"]["recall"] == 1.0 and report.metrics["access_control"]["recall"] == 1.0
+
+
+def test_round_zero_reports_each_slice_separately(tmp_path: Path) -> None:
+    """Req 8.4: vibe-py and agent-vfc are different worlds; one pooled number hides which one moved."""
+    from openultrasast.learning.rounds import run_baseline
+
+    configs = tmp_path / "configs"
+    write_default_configs(configs, load_families(), prompt="p", version="0")
+    web = _case(tmp_path, "web", split="train")
+    agent = _case(tmp_path, "agent", split="train")
+    object.__setattr__(agent, "slice", "agent-vfc")
+
+    def factory(config: FamilyConfig):  # type: ignore[no-untyped-def]
+        def scan(root: Path) -> list[StaticFinding]:
+            return [_finding(config.family)] if root.name == "vuln" else []
+
+        return scan
+
+    report = run_baseline(
+        [web, agent],
+        taxonomy=load_families(),
+        configs_dir=configs,
+        scan_factory=factory,
+        model="m",
+        out_dir=tmp_path / "out",
+        k_runs=3,
+    )
+    assert set(report.per_slice) == {"vibe-py/injection", "agent-vfc/injection"}
+    payload = json.loads((tmp_path / "out" / "baseline" / "m" / "report.json").read_text())
+    assert set(payload["per_slice"]) == {"vibe-py/injection", "agent-vfc/injection"}
+
+
+def test_round_zero_refuses_fewer_than_three_runs(tmp_path: Path) -> None:
+    """The floor exists because a single run misses most real changes; a baseline measured at K=1 is a noise floor
+    of zero and every later round then inherits a budget of nothing."""
+    from openultrasast.learning.rounds import run_baseline
+
+    configs = tmp_path / "configs"
+    write_default_configs(configs, load_families(), prompt="p", version="0")
+    with pytest.raises(ValueError, match="three runs"):
+        run_baseline(
+            [_case(tmp_path, "inj", split="train")],
+            taxonomy=load_families(),
+            configs_dir=configs,
+            scan_factory=lambda config: lambda root: [],
+            model="m",
+            out_dir=tmp_path / "out",
+            k_runs=1,
+        )

tests/test_learning_round_loop.py

@@ -134,21 +134,29 @@ def test_a_change_that_helps_nothing_is_rejected_and_reverted(world) -> None:  #
 
 
 def test_a_round_is_rejected_when_the_sweep_finds_another_family_below_its_floor(world) -> None:  # type: ignore[no-untyped-def]
-    """The sweep is a floor check over every other family, not a causal claim about this change.
+    """The sweep counts pairs this change turned from right to wrong in another family (Req 9.5).
 
-    Per-family directories make a change that reaches another family structurally impossible, so what the
-    sweep really guards is a family that is failing for any reason at the moment this round would land.
+    It used to count every failing pair, which rejected a round for damage it did not do; the pair below is right
+    before the change and wrong after it, which is the only thing that may reject a round on another family's
+    behalf. The other arm — a family that was already failing must not reject — lives in
+    `tests/test_learning_round_evidence.py`.
     """
     world["cases"].append(_case(world["configs"].parent, "other-hold", split="holdout", family="access_control"))
     world["floors"]["access_control"] = NoiseFloor(
         family="access_control", k_runs=5, pairs=1, flaky_pairs=0, negative_flip_rate=0.0, budget_flips=0
     )
+    landed = {"yes": False}
 
     def build(config: FamilyConfig):  # type: ignore[no-untyped-def]
         def scan(root: Path) -> list[StaticFinding]:
-            if root.name != "vuln" or config.family == "access_control":
-                return []  # access control finds nothing, so its holdout pair is below its floor
-            return [_finding("injection")] if config.checklist.strip() == "- ask about quoting" else []
+            if root.name != "vuln":
+                return []
+            if config.family == "access_control":
+                return [] if landed["yes"] else [_finding("access_control")]
+            if config.checklist.strip() == "- ask about quoting":
+                landed["yes"] = True  # the injection change lands, and access control stops finding its pair
+                return [_finding("injection")]
+            return []
 
         return scan
 
@@ -181,8 +189,7 @@ def test_a_round_over_its_cost_cap_is_reverted_and_says_so(world) -> None:  # ty
 
 def test_a_proposal_that_reaches_beyond_its_family_never_runs(world) -> None:  # type: ignore[no-untyped-def]
     record = _run(world, Proposal(hypothesis="h", lever="checklist", change={"../escape.md": "x"}), {})
-    assert record.outcome == "refused_holdout" or record.outcome == "rejected"
-    assert "outside" in record.reason
+    assert record.outcome == "rejected" and "outside" in record.reason  # the holdout refusal has its own test
     assert not (world["configs"].parent / "escape.md").exists()
 
 
@@ -202,7 +209,7 @@ def test_the_round_directory_holds_what_a_replay_needs(world) -> None:  # type:
     directory = world["out"] / "rounds" / "1"
     assert (directory / "proposal.json").is_file() and (directory / "scores.json").is_file()
     attribution = json.loads((directory / "attribution.json").read_text())
-    assert attribution["flipped_predicted"] == ["hold-a"] or attribution["flipped_predicted"] == ["train-a", "hold-a"]
+    assert attribution["flipped_predicted"] == ["hold-a", "train-a"]  # the round is deterministic; one exact answer
     assert attribution["flipped_unpredicted"] == []
     trajectories = (directory / "trajectories.jsonl").read_text()
     assert "REDACTED" in trajectories and "B" * 24 not in trajectories

tests/test_learning_rounds.py

@@ -70,6 +70,16 @@ def _steady(root: Path) -> list[StaticFinding]:
     return [_finding("injection")] if _is_vulnerable_side(root) else []
 
 
+def _always(scan):  # type: ignore[no-untyped-def]
+    """Round zero asks for a detector per family; these tests deliberately give every family the same one so the
+    noise-floor arithmetic is the only thing under test."""
+
+    def build(_config):  # type: ignore[no-untyped-def]
+        return scan
+
+    return build
+
+
 def _flaky(counter: list[int]):  # type: ignore[no-untyped-def]
     def scan(root: Path) -> list[StaticFinding]:
         if not _is_vulnerable_side(root):
@@ -86,7 +96,12 @@ def test_round_zero_writes_a_configuration_for_every_family(tmp_path: Path) -> N
 
     taxonomy = load_families()
     report = run_baseline(
-        [_case(tmp_path, "a")], taxonomy=taxonomy, configs_dir=tmp_path / "configs", scan=_steady, model="m", out_dir=tmp_path / "out"
+        [_case(tmp_path, "a")],
+        taxonomy=taxonomy,
+        configs_dir=tmp_path / "configs",
+        scan_factory=_always(_steady),
+        model="m",
+        out_dir=tmp_path / "out",
     )
     configs = load_family_configs(tmp_path / "configs", taxonomy)
     assert set(configs) == {family.id for family in taxonomy.families}
@@ -102,7 +117,7 @@ def test_a_steady_detector_has_a_zero_noise_floor_and_a_flaky_one_does_not(tmp_p
         [_case(tmp_path, "a"), _case(tmp_path, "b")],
         taxonomy=taxonomy,
         configs_dir=tmp_path / "c1",
-        scan=_steady,
+        scan_factory=_always(_steady),
         model="m",
         out_dir=tmp_path / "o1",
     )
@@ -112,7 +127,7 @@ def test_a_steady_detector_has_a_zero_noise_floor_and_a_flaky_one_does_not(tmp_p
         [_case(tmp_path, "a"), _case(tmp_path, "b")],
         taxonomy=taxonomy,
         configs_dir=tmp_path / "c2",
-        scan=_flaky([]),
+        scan_factory=_always(_flaky([])),
         model="m",
         out_dir=tmp_path / "o2",
     )
@@ -127,7 +142,7 @@ def test_the_memory_family_is_measured_but_never_gates(tmp_path: Path) -> None:
         [_case(tmp_path, "m", family="memory", slice_name="vfc")],
         taxonomy=taxonomy,
         configs_dir=tmp_path / "configs",
-        scan=_steady,
+        scan_factory=_always(_steady),
         model="m",
         out_dir=tmp_path / "out",
         slices=("vfc",),
@@ -141,7 +156,9 @@ def test_the_default_selection_is_the_web_slices(tmp_path: Path) -> None:
 
     taxonomy = load_families()
     cases = [_case(tmp_path, "web", slice_name="vibe-py"), _case(tmp_path, "c", family="memory", slice_name="vfc")]
-    report = run_baseline(cases, taxonomy=taxonomy, configs_dir=tmp_path / "configs", scan=_steady, model="m", out_dir=tmp_path / "out")
+    report = run_baseline(
+        cases, taxonomy=taxonomy, configs_dir=tmp_path / "configs", scan_factory=_always(_steady), model="m", out_dir=tmp_path / "out"
+    )
     assert set(report.metrics) == {"injection"} and "memory" not in report.floors
 
 
@@ -155,7 +172,7 @@ def test_artifacts_are_keyed_by_model_so_another_model_can_be_compared(tmp_path:
             [_case(tmp_path, f"a{index}")],
             taxonomy=taxonomy,
             configs_dir=tmp_path / "configs",
-            scan=_steady,
+            scan_factory=_always(_steady),
             model=model,
             out_dir=out,
         )
@@ -184,7 +201,9 @@ def test_an_unscorable_pair_is_listed_and_never_run(tmp_path: Path) -> None:
     taxonomy = load_families()
     case = _case(tmp_path, "twin")
     twin = PairCase(**{**case.__dict__, "unscorable": "identical_twin"})
-    report = run_baseline([twin], taxonomy=taxonomy, configs_dir=tmp_path / "configs", scan=counting, model="m", out_dir=tmp_path / "out")
+    report = run_baseline(
+        [twin], taxonomy=taxonomy, configs_dir=tmp_path / "configs", scan_factory=_always(counting), model="m", out_dir=tmp_path / "out"
+    )
     assert report.metrics["injection"]["scorable"] == 0
     assert report.metrics["injection"]["unscorable"] == {"identical_twin": 1}
     assert calls == []
@@ -198,7 +217,7 @@ def test_round_zero_records_what_it_spent(tmp_path: Path) -> None:
         [_case(tmp_path, "a")],
         taxonomy=load_families(),
         configs_dir=tmp_path / "configs",
-        scan=_steady,
+        scan_factory=_always(_steady),
         model="m",
         out_dir=tmp_path / "out",
         spent_usd=lambda: 1.25,

tests/test_learning_scan.py

@@ -91,9 +91,13 @@ def test_family_findings_never_become_sandbox_candidates(tmp_path: Path, monkeyp
     assert main(["scan", str(repo), "--mode", "deep", "--config", str(repo / "openultrasast.toml")]) == 0
     run_dir = _run_dir(repo)
     verdicts = json.loads((run_dir / "verdicts.json").read_text())["verdicts"]
-    findings = {item["finding_id"] for item in json.loads((run_dir / "findings.json").read_text())["findings"]}
-    del findings
-    assert all(not str(item.get("path", "")).startswith("family:") for item in verdicts)
+    payload = json.loads((run_dir / "findings.json").read_text())["findings"]
+    family_findings = {item["finding_id"] for item in payload if any(str(tag).startswith("family:") for tag in item["tags"])}
+    assert family_findings, "the fixture must actually produce family findings, or this proves nothing"
+    # A family detector's claim is a suspicion until a verifier speaks; handing it to the sandbox would let an
+    # unverified model claim drive execution (Req 7.1).
+    candidates = {str(item.get("finding_id") or item.get("id") or "") for item in verdicts}
+    assert candidates.isdisjoint(family_findings)
 
 
 def test_the_sarif_report_carries_the_family_properties(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:

tests/test_learning_split.py

@@ -8,6 +8,33 @@ from pathlib import Path
 from openultrasast.benchmark import ExpectedFinding
 from openultrasast.pairs import DEFAULT_CATALOG, PairCase, load_pair_catalog, select_slice, select_vendored
 
+VULN = "import os\nfrom flask import request\n\n\ndef run():\n    os.system(request.args.get('cmd'))\n"
+FIXED = "import subprocess\nfrom flask import request\n\n\ndef run():\n    subprocess.run(['echo'], check=False)\n"
+
+
+def _teaching_case(tmp_path: Path, name: str, *, split: str = "train") -> PairCase:
+    """A pair that really carries a mechanism shape.
+
+    The trivial `x = 1` fixture below teaches nothing whatever the split says, so a teacher-rule assertion made
+    against it passes with the rule deleted. This one is the difference between measuring the rule and measuring
+    an empty corpus."""
+    (tmp_path / f"{name}-v.py").write_text(VULN)
+    (tmp_path / f"{name}-f.py").write_text(FIXED)
+    return PairCase(
+        name=name,
+        slice="vibe-py",
+        language="python",
+        origin="test",
+        vuln_file=tmp_path / f"{name}-v.py",
+        fixed_file=tmp_path / f"{name}-f.py",
+        relpath="app.py",
+        expected=(ExpectedFinding(cwe="CWE-78", vulnerability_class="x", path="app.py", evidence="", function="run", mechanism="other"),),
+        min_recall=1.0,
+        fix_policy="silent",
+        split=split,
+        review_tier="seeded",
+    )
+
 
 def _case(tmp_path: Path, name: str, *, split: str = "train", tier: str = "seeded", unscorable: str | None = None) -> PairCase:
     (tmp_path / f"{name}.py").write_text(f"# {name}\nx = 1\n")
@@ -69,33 +96,45 @@ def test_the_exporter_seeds_from_train_pairs_only(tmp_path: Path) -> None:
 
 
 def test_leave_one_out_teaches_from_train_pairs_only(tmp_path: Path) -> None:
+    """Both arms, because `teaching_pairs <= 1` on a two-pair corpus is satisfied by a rule that does nothing.
+
+    The same two excerpts teach twice when both are on the train split and once when one is held out, and the run
+    names the pair it refused (Req 5.1, 5.2)."""
     from openultrasast.semantic.loo import evaluate_loo
 
-    cases = [_case(tmp_path, "train-one"), _case(tmp_path, "holdout-one", split="holdout")]
-    result = evaluate_loo(cases)
+    both_train = evaluate_loo([_teaching_case(tmp_path, "train-one"), _teaching_case(tmp_path, "train-two")])
+    assert both_train.teaching_pairs == 2 and both_train.degradations == ()
+    result = evaluate_loo([_teaching_case(tmp_path, "train-one"), _teaching_case(tmp_path, "holdout-one", split="holdout")])
     assert {outcome.pair for outcome in result.outcomes} == {"train-one", "holdout-one"}  # every pair is still scored
-    assert result.teaching_pairs <= 1  # only the train pair could ever have taught
+    assert result.teaching_pairs == 1
+    refusals = [item for item in result.degradations if item.get("reason") == "holdout_pair_refused"]
+    assert [item["pairs"] for item in refusals] == [["holdout-one"]]
 
 
 def test_the_mechanism_lever_refuses_a_candidate_a_holdout_pair_taught(tmp_path: Path) -> None:
-    from openultrasast.improve.evolve import propose_mechanism_edits
-    from openultrasast.semantic.mechanisms import MechanismStore, append_from_pair
-    from openultrasast.semantic.variants import Shape
+    """Both arms. The shape is the one the corpus really teaches, taken from the pair itself.
 
-    shape = Shape(
-        language="python",
-        sink_name="system",
-        arity=1,
-        source_positions=(0,),
-        source_kinds=("request",),
-        guard="none",
-        mechanism="source_reaches_sink",
-    )
-    candidates = MechanismStore(tmp_path / "candidates.jsonl")
-    append_from_pair(candidates, shape, summary="s", cwe="CWE-78", pair="holdout-one", provenance="human", tier="seeded")
-    cases = [_case(tmp_path, "train-one"), _case(tmp_path, "holdout-one", split="holdout")]
-    edits = propose_mechanism_edits(cases, candidates, tmp_path / "scan.jsonl")
-    assert [edit.mechanism_id for edit in edits if edit.action == "admit"] == []
+    Asserting only the refusal would pass against a lever that admits nothing at all, and a hand-written shape that
+    matches no excerpt can never be admitted either, so neither would measure the split rule (Req 5.2)."""
+    from openultrasast.improve.evolve import propose_mechanism_edits
+    from openultrasast.semantic.loo import _facts
+    from openultrasast.semantic.mechanisms import MechanismStore, append_from_pair
+    from openultrasast.semantic.seed import pair_lessons
+
+    cases = [_teaching_case(tmp_path, "train-one"), _teaching_case(tmp_path, "holdout-one", split="holdout")]
+    (lesson,) = pair_lessons(cases[0], _facts())
+
+    def store(name: str, pair: str) -> MechanismStore:
+        path = MechanismStore(tmp_path / name)
+        append_from_pair(path, lesson.shape, summary=lesson.summary, cwe=lesson.cwe, pair=pair, provenance="human", tier="seeded")
+        return path
+
+    reasons: list[dict[str, object]] = []
+    refused = propose_mechanism_edits(cases, store("refused.jsonl", "holdout-one"), tmp_path / "scan.jsonl", degradations=reasons)
+    assert [edit.mechanism_id for edit in refused if edit.action == "admit"] == []
+    assert [item["pairs"] for item in reasons] == [["holdout-one"]]
+    admitted = propose_mechanism_edits(cases, store("allowed.jsonl", "train-one"), tmp_path / "scan2.jsonl")
+    assert [edit.mechanism_id for edit in admitted if edit.action == "admit"], "a train-taught shape is still admissible"
 
 
 def test_the_corrected_lever_number_is_measured_and_committed() -> None:

tests/test_obligation_export.py

@@ -128,8 +128,12 @@ def test_the_train_split_of_vibe_py_teaches_no_obligation_shape_today(tmp_path:
     cases = select_vendored(select_slice(load_pair_catalog(), "vibe-py"))
     report = export_mechanisms(cases, store)
     assert obligation_mechanisms(store.load()) == ()
-    refused = {pair for pair, reason in report.skipped if reason.startswith("split:")}
-    assert {"vampi-books-get-by-title", "vampi-users-update-password", "threatbyte-api-v1-get"} <= refused
+    refused = {pair: reason for pair, reason in report.skipped}
+    assert refused["vampi-users-update-password"] == "split:holdout"
+    assert refused["threatbyte-api-v1-get"] == "split:holdout"
+    # Its own reason, not its split: this row is a byte-identical twin that happens to sit on the holdout side,
+    # and reporting `split:holdout` would hide a corpus fact behind a split fact (Req 5.2).
+    assert refused["vampi-books-get-by-title"] == "unscorable:identical_twin"
     flow_rows = [row for row in store.load() if row.shape is not None and row.shape.get("family", "sink") == "sink"]
     assert report.records == len(flow_rows) and len(flow_rows) >= 1  # the flow family still teaches from the train split