fix(learning-harness): the round-4 review findings

Three defects, two of them the same shape as ones already rejected once:

- The meta-agent proposer still could not propose. `MetaAgent(inner_model=)`
  takes a ModelConfig; it was handed a model name, a `# type: ignore` hid
  that from mypy, and a bare `except Exception` turned the resulting
  AttributeError into `meta_agent_failed` -- which reads exactly like the
  model declining. The composition is now a seam a test can pin, and a
  construction error is re-raised rather than reported as a quiet round.
- `ousast pairs --hunter` crashed in its documented form: the K floor landed
  with `--k-runs` still defaulting to 1. The covering test always passed
  `--k-runs 4`, which is why nothing noticed.
- 28 rows (12% of the corpus) resolved a span on the vulnerable side and
  none on the fixed side, so every fixed-side finding read as outside the
  labeled function and the pair scored as silence whatever the detector
  said. `fix_function` now reaches the catalog and the scorer, and the
  generator derives it from the fixed excerpt for the 25 trap rows that
  never declared one. Measured after: 0 rows.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG

.kiro/specs/learning-harness/tasks.md

@@ -238,3 +238,8 @@
   - **The fixture-difficulty check** (`learning/difficulty.py`), on three axes the corpus really varies on: binding hops from the identity source to the constrained use, registration distance (0 decorator on the handler, 1 statement elsewhere in the file, 2 another document), and guard distance (0 decorator, 1 call in the body, 2 nothing visible). The comparison is per axis over the whole set, because a suite covers a bug shape with several handlers between them. **Measured first, and it failed: the shared obligations fixture reached (1, 0, 2) where the vibe-py absence rows reach (5, 2, 2).** Every shared handler carried its own route decorator, so nothing in the suite exercised a handler registered from another document and no chain was longer than one binding — the checker was passing on a simpler world than the one it meets. `from_spec` was added to the fixture: the connexion shape those rows really have, a token validated by a call rather than a decorator and five bindings between the token and the constraint. The verdict is committed as `benchmarks/measurements/2026-09-06-fixture-difficulty.json` with both reaches and the history of the first measurement. One existing expectation moved with it (`from_spec` reads the user row and then the book: two real operations).
 - Task 4.3 (2026-09-06, RED first: 10 failing tests in `tests/test_learning_cutoff.py`; then green, 11). The catalog generator fills `fix_date` from the recipe at the precision the recipe has: a full ISO day when one was recorded, otherwise the year. Every harvested recipe carries a year and almost none carries a day, and writing `-01-01` would turn a year into a claim about a month. A row therefore counts as post-cutoff only when its **earliest possible** date is after the cutoff, so a coarse date can under-claim the clean slice but never over-claim it — and over-claiming is the direction that would turn contamination into a result. `PairFamilyScore` carries the date, `aggregate(cutoff=...)` reports the same numbers over the post-cutoff rows beside the whole number and never instead of it, undated rows are counted on their own, and `publish` prints the slice with the cutoff it used. Without a configured cutoff every scoreboard says so rather than inventing one; `[learning].cutoff_date` has no default because guessing a model's training cutoff and publishing a contamination number from the guess would be worse than reporting none. Corpus dates as harvested: vibe-py all 2026, agent-vfc 67 of 69 in 2026, vfc spread 2014-2026, vfc-js 2014-2021.
 - Task 5.1 (2026-09-06, RED first: 3 of 9 tests in `tests/test_degradation_matrix.py`; then green, 8 after one test was deleted rather than kept). The three gate outputs are captured at `71efc9c` — the commit before the first line of this feature — as `benchmarks/measurements/2026-09-06-gate-baseline.txt`, and a test reproduces them byte for byte on every run, in both matrices. Until now byte-identity was something a reviewer checked by hand against a worktree; it is now mechanical and runs in CI, which is what makes it a gate rather than a habit. Six absences are each exercised once with the reason they record: HarnessX (`harnessx_unavailable`), the chat endpoint (`learning_endpoint_unavailable`, and the run still writes its artifact), the family configurations (`learning_configs_unavailable`), the network for pointer pairs (`pointer_pair_skipped`), the semantic extra (`language_unsupported` rather than a guess), and the sandbox (`sandbox_unavailable`, so a claim nothing could run is not confused with a claim that failed). One test written for this task was **deleted before the commit**: it asserted that this file contained a `without_<capability>` name per capability, which measures the file's own naming and nothing about the code.
+- Review round 4 (2026-09-06) APPROVED 4.1, 4.2, 4.3; REJECTED the round-3 remediation on three counts. The reviewer re-derived the RED phase for group 4 (24 of 26 for 4.1, 3 for 4.2, 11 of 11 for 4.3 — **every count in the notes above was understated**, corrected here), killed 18 of 19 mutants, verified the 4.1 observable independently (2 of 10 access-control rows named at `7a33756`, 9 of 10 at `3e71e9e`, the tenth stating its reason), confirmed `_python_route_access` changed no access level anywhere in the corpus and `.mjs` changed exactly one snapshot, and reproduced the fixture-difficulty artifact number for number including its failing arm. Remediated RED-first:
+  - **The meta-agent proposer still could not propose.** `MetaAgent(inner_model=...)` takes a `ModelConfig`; it was handed a model name, a `# type: ignore` hid that from mypy, and a bare `except Exception` turned the resulting `AttributeError` into the degradation reason `meta_agent_failed` — which reads exactly like the model declining. Every other HarnessX composition site in the repository builds the config properly. `_meta_agent_class` and `_model_config` are now seams a test can pin without the extra, and a `TypeError`/`AttributeError`/`ImportError` from the call is re-raised: "we called the agent wrong" is not "the agent had nothing to propose".
+  - **`ousast pairs --hunter` crashed in its documented form.** The K floor landed with `--k-runs` still defaulting to 1, so the flag's absence raised `ValueError` from inside `evaluate_catalog`. The default is `None` (three), and a breach exits naming the flag. The covering test always passed `--k-runs 4`, which is why nothing noticed.
+  - **28 rows resolved a span on the vulnerable side and none on the fixed side** (12% of the corpus), so every fixed-side finding read as outside the labeled function and the pair scored as silence whatever the detector said — the same flattering direction as the range defect, one-seventh the size. `fix_function` reached only the harvester; it now reaches the catalog and `PairCase`, `_spans` uses it for the fixed side, and the catalog generator derives it from the fixed excerpt itself for the 25 vibe-py trap rows that never declared one. A row that still resolves no fixed-side span reports `unresolved_label:fixed` instead of scoring silent. Measured after: **0 rows**.
+  - Suggestions applied: one spelling per corpus fact everywhere (`PairCase.unscorable` verbatim, no invented prefix); leave-one-out records non-teachers withheld for tier or an unscorable reason, not only holdout refusals; the hygiene test asserts the declared twins still hash equal, so a declaration that stopped being true would not be invisible; `is_post_cutoff` refuses a cutoff coarser than a day, which was the one over-claim path that existed, and month precision is pinned; `UNSCORABLE_REASONS` covers the whole vocabulary rather than `identical_twin` alone; the zero-run score carries its date; `_pairs_named_by` filters by substring before the regex.

benchmarks/pairs/catalog_gen.py

@@ -93,6 +93,9 @@ def catalog_text(slice_name: str, recipes: list[dict[str, object]]) -> str:
             lines.append(f'reviewer = "{_q(reviewer)}"')
         if recipe.get("known_limit"):
             lines.append(f'known_limit = "{_q(recipe["known_limit"])}"')
+        if recipe.get("fix_function"):
+            # Req 3.2: the fixed side of a trap pair is a different handler, and the scorer has to know which one
+            lines.append(f'fix_function = "{_q(recipe["fix_function"])}"')
         fix_date = _fix_date(recipe)
         if fix_date:
             lines.append(f'fix_date = "{_q(fix_date)}"')  # Req 10.5: the post-cutoff slice needs a date to sort by
@@ -399,6 +402,27 @@ def _stamp_function(path: Path, function: str) -> None:
         path.write_text(updated)
 
 
+def derive_fix_function(slice_root: Path, recipe: dict[str, object]) -> str:
+    """The function the fixed excerpt actually holds, when it is not the labeled one.
+
+    Real-Vuln-Benchmark traps take their fixed side from a different, correctly guarded handler of the same
+    snapshot. Without this the fixed side resolves no span, every finding there counts as outside the labeled
+    function, and the pair scores as silence instead of as a leak — always in the direction that flatters the
+    number (learning-harness Req 3.2)."""
+    _vuln, fixed = excerpt_rel(recipe)
+    path = slice_root / fixed
+    if not path.is_file():
+        return ""
+    ranges = _ranges(path, _language(recipe))
+    if not ranges:
+        return ""
+    labeled = str(recipe.get("function", "") or "")
+    names = [name for name, _start, _end in ranges]
+    if labeled in names or len(set(names)) != 1:
+        return ""  # the fixed side holds the labeled function, or holds several and the recipe must say which
+    return names[0]
+
+
 def generate(slice_name: str, *, prune: bool = False) -> tuple[Path, int]:
     slice_root = ROOT / slice_name
     loaded = [dict(item) for item in tomllib.loads((slice_root / "recipes.toml").read_text()).get("recipe", [])]
@@ -419,6 +443,10 @@ def generate(slice_name: str, *, prune: bool = False) -> tuple[Path, int]:
                 recipe["function"] = derived
                 _stamp_function(slice_root / vuln, derived)
                 _stamp_function(slice_root / fixed, derived)
+            if not recipe.get("fix_function"):
+                derived = derive_fix_function(slice_root, recipe)
+                if derived:
+                    recipe["fix_function"] = derived
             recipes.append(recipe)  # `reviewer = "pending"` rows load at tier `title` (Req 9.4); no separate queue file
         else:
             print(f"skip {recipe['name']}: excerpt missing (harvest failed or not fetched)")

benchmarks/pairs/vfc/catalog.toml

@@ -492,6 +492,7 @@ fix_policy = "silent"
 provenance = "human"
 split = "holdout"
 review_tier = "advisory"
+fix_function = "OSSL_TRACE_BEGIN"
 fix_date = "2025"
 
 [[pair.expected]]

benchmarks/pairs/vibe-py/catalog.toml

@@ -2153,6 +2153,7 @@ fix_policy = "silent"
 provenance = "human"
 split = "holdout"
 review_tier = "seeded"
+fix_function = "mutate"
 fix_date = "2026"
 
 [[pair.expected]]
@@ -2183,6 +2184,7 @@ fix_policy = "silent"
 provenance = "human"
 split = "train"
 review_tier = "seeded"
+fix_function = "mutate"
 fix_date = "2026"
 
 [[pair.expected]]
@@ -2213,6 +2215,7 @@ fix_policy = "silent"
 provenance = "human"
 split = "holdout"
 review_tier = "seeded"
+fix_function = "parse_field"
 fix_date = "2026"
 
 [[pair.expected]]
@@ -2243,6 +2246,7 @@ fix_policy = "silent"
 provenance = "human"
 split = "train"
 review_tier = "seeded"
+fix_function = "forgot_password"
 fix_date = "2026"
 
 [[pair.expected]]
@@ -2273,6 +2277,7 @@ fix_policy = "silent"
 provenance = "human"
 split = "holdout"
 review_tier = "seeded"
+fix_function = "parse_field"
 fix_date = "2026"
 
 [[pair.expected]]
@@ -2333,6 +2338,7 @@ fix_policy = "silent"
 provenance = "human"
 split = "holdout"
 review_tier = "seeded"
+fix_function = "run"
 fix_date = "2026"
 
 [[pair.expected]]
@@ -2363,6 +2369,7 @@ fix_policy = "silent"
 provenance = "human"
 split = "train"
 review_tier = "seeded"
+fix_function = "run"
 fix_date = "2026"
 
 [[pair.expected]]
@@ -2424,6 +2431,7 @@ fix_policy = "silent"
 provenance = "human"
 split = "train"
 review_tier = "seeded"
+fix_function = "get_by_username"
 fix_date = "2026"
 
 [[pair.expected]]
@@ -2454,6 +2462,7 @@ fix_policy = "silent"
 provenance = "human"
 split = "holdout"
 review_tier = "seeded"
+fix_function = "get_by_username"
 fix_date = "2026"
 
 [[pair.expected]]
@@ -2484,6 +2493,7 @@ fix_policy = "silent"
 provenance = "human"
 split = "train"
 review_tier = "seeded"
+fix_function = "get_by_username"
 fix_date = "2026"
 
 [[pair.expected]]
@@ -2514,6 +2524,7 @@ fix_policy = "silent"
 provenance = "human"
 split = "holdout"
 review_tier = "seeded"
+fix_function = "do_login"
 fix_date = "2026"
 
 [[pair.expected]]
@@ -2544,6 +2555,7 @@ fix_policy = "silent"
 provenance = "human"
 split = "train"
 review_tier = "seeded"
+fix_function = "do_login"
 fix_date = "2026"
 
 [[pair.expected]]
@@ -2574,6 +2586,7 @@ fix_policy = "silent"
 provenance = "human"
 split = "holdout"
 review_tier = "seeded"
+fix_function = "do_signup"
 fix_date = "2026"
 
 [[pair.expected]]
@@ -3366,6 +3379,7 @@ fix_policy = "silent"
 provenance = "human"
 split = "train"
 review_tier = "seeded"
+fix_function = "post"
 fix_date = "2026"
 
 [[pair.expected]]
@@ -3397,6 +3411,7 @@ fix_policy = "silent"
 provenance = "human"
 split = "holdout"
 review_tier = "seeded"
+fix_function = "post"
 fix_date = "2026"
 
 [[pair.expected]]
@@ -3428,6 +3443,7 @@ fix_policy = "silent"
 provenance = "human"
 split = "train"
 review_tier = "seeded"
+fix_function = "login"
 fix_date = "2026"
 
 [[pair.expected]]
@@ -3495,6 +3511,7 @@ fix_policy = "silent"
 provenance = "human"
 split = "train"
 review_tier = "seeded"
+fix_function = "register_user"
 fix_date = "2026"
 
 [[pair.expected]]
@@ -3525,6 +3542,7 @@ fix_policy = "silent"
 provenance = "human"
 split = "holdout"
 review_tier = "seeded"
+fix_function = "register_user"
 fix_date = "2026"
 
 [[pair.context]]
@@ -3561,6 +3579,7 @@ fix_policy = "silent"
 provenance = "human"
 split = "train"
 review_tier = "seeded"
+fix_function = "nosql_return_users_from_username"
 fix_date = "2026"
 
 [[pair.expected]]
@@ -3591,6 +3610,7 @@ fix_policy = "silent"
 provenance = "human"
 split = "holdout"
 review_tier = "seeded"
+fix_function = "nosql_return_users_from_username"
 fix_date = "2026"
 
 [[pair.expected]]
@@ -3621,6 +3641,7 @@ fix_policy = "silent"
 provenance = "human"
 split = "train"
 review_tier = "seeded"
+fix_function = "nosql_return_users_from_username"
 fix_date = "2026"
 
 [[pair.expected]]
@@ -3651,6 +3672,7 @@ fix_policy = "silent"
 provenance = "human"
 split = "holdout"
 review_tier = "seeded"
+fix_function = "get_post"
 fix_date = "2026"
 
 [[pair.expected]]
@@ -3681,6 +3703,7 @@ fix_policy = "silent"
 provenance = "human"
 split = "train"
 review_tier = "seeded"
+fix_function = "get_post"
 fix_date = "2026"
 
 [[pair.expected]]
@@ -3711,6 +3734,7 @@ fix_policy = "silent"
 provenance = "human"
 split = "train"
 review_tier = "seeded"
+fix_function = "_execute"
 fix_date = "2026"
 
 [[pair.expected]]
@@ -3741,6 +3765,7 @@ fix_policy = "silent"
 provenance = "human"
 split = "holdout"
 review_tier = "seeded"
+fix_function = "_execute"
 fix_date = "2026"
 
 [[pair.expected]]
@@ -3802,6 +3827,7 @@ fix_policy = "silent"
 provenance = "human"
 split = "holdout"
 review_tier = "seeded"
+fix_function = "get_posts"
 fix_date = "2026"
 
 [[pair.expected]]
@@ -3832,6 +3858,7 @@ fix_policy = "silent"
 provenance = "human"
 split = "train"
 review_tier = "seeded"
+fix_function = "get_posts"
 fix_date = "2026"
 
 [[pair.expected]]
@@ -3862,6 +3889,7 @@ fix_policy = "silent"
 provenance = "human"
 split = "holdout"
 review_tier = "seeded"
+fix_function = "get_posts"
 fix_date = "2026"
 
 [[pair.expected]]

src/openultrasast/cli.py

@@ -189,7 +189,9 @@ def main(argv: list[str] | None = None) -> int:
     )
     pairs.add_argument("--loo-out", type=Path, default=None, help="where to write loo.json (default: reports/loo.json)")
     pairs.add_argument("--json", action="store_true", help="print the pair scoreboard as JSON")
-    pairs.add_argument("--k-runs", type=int, default=1, help="runs per pair on the hunter path; the family scorer needs at least three")
+    pairs.add_argument(
+        "--k-runs", type=int, default=None, help="runs per pair on the hunter path; never fewer than three, and three by default"
+    )
 
     subparsers.add_parser("mcp", help="run the narrow MCP server over stdio for OpenCode integration")
     mechanisms = subparsers.add_parser("mechanisms", help="maintainer: mechanism memory seeded from trusted pairs")
@@ -982,7 +984,10 @@ def _pairs(
             from .learning.families import load_families
 
             scan = make_hunter_scan(client, model, taxonomy=load_families())
-    result = evaluate_catalog(cases, hunter=scan, pointers=True if pointers else None, k_runs=k_runs)
+    try:
+        result = evaluate_catalog(cases, hunter=scan, pointers=True if pointers else None, k_runs=k_runs)
+    except ValueError as exc:
+        raise SystemExit(f"--k-runs: {exc}") from exc
     if json_out:
         print(json.dumps(result_payload(result), indent=2, sort_keys=True))
         return 0

src/openultrasast/learning/proposer.py

@@ -201,6 +201,10 @@ class HarnessXProposer:
             before = _directory_text(workspace)
             try:
                 hypothesis = call(workspace=workspace, facts=facts, model=self.model)
+            except (TypeError, AttributeError, ImportError):
+                # "we called the agent wrong" is not "the agent declined", and folding one into the other is how a
+                # broken composition reads as a quiet round for as long as nobody looks.
+                raise
             except Exception as exc:  # noqa: BLE001 — an agent that failed proposed nothing, and the round says why
                 self.reason = f"meta_agent_failed: {type(exc).__name__}"
                 return None
@@ -239,6 +243,26 @@ def _directory_text(directory: Path) -> dict[str, str]:
     return out
 
 
+def _meta_agent_class() -> type:
+    """`MetaAgent` behind `harness_ext`. A seam, so the argument contract can be pinned without the extra."""
+    from ..harness_ext import require_harnessx
+
+    require_harnessx()
+    from harnessx.meta_harness.agent import MetaAgent  # type: ignore[import-not-found]
+
+    return MetaAgent  # type: ignore[no-any-return]
+
+
+def _model_config(model: str, provider: str = "anthropic") -> object:
+    """`MetaAgent(inner_model=...)` takes a `ModelConfig`, never a model name; the name goes to the provider."""
+    from ..harness_ext import build_provider, require_harnessx
+
+    require_harnessx()
+    from harnessx.core.model_config import ModelConfig  # type: ignore[import-not-found]
+
+    return ModelConfig(main=build_provider(model, provider))
+
+
 def _meta_agent_call() -> MetaAgentCall:
     """Compose `MetaAgent.evolve` behind `harness_ext`, writing only inside the scratch copy.
 
@@ -252,12 +276,10 @@ def _meta_agent_call() -> MetaAgentCall:
     def call(*, workspace: Path, facts: FailureFacts, model: str) -> str:
         import asyncio
 
-        from harnessx.meta_harness.agent import MetaAgent  # type: ignore[import-not-found]
-
         trajectories = workspace.parent / "trajectories"
         trajectories.mkdir(parents=True, exist_ok=True)
         (trajectories / "facts.json").write_text(facts_prompt(facts), encoding="utf-8")
-        agent = MetaAgent(inner_model=model, allowed_write_roots=(workspace,))  # type: ignore[arg-type]
+        agent = _meta_agent_class()(inner_model=_model_config(model), allowed_write_roots=(workspace,))
         output = workspace.parent / "meta-out"
         output.mkdir(parents=True, exist_ok=True)
         asyncio.run(agent.evolve(current_config=workspace, trajectories_dir=trajectories, output_dir=output))

src/openultrasast/learning/rounds.py

@@ -457,11 +457,13 @@ def _pairs_named_by(proposal: Proposal, cases: Sequence[PairCase]) -> tuple[str,
     A memory entry that quotes a held-out pair is the loop learning the answer instead of the shape, and it looks
     exactly like a legitimate hard negative until the holdout number is read as if it meant something."""
     haystack = "\n".join([*proposal.predicted_affected, *proposal.predicted_at_risk, *proposal.change.values()])
-    named = []
-    for case in cases:
-        name = str(getattr(case, "name", ""))
-        if name and re.search(rf"(?<![\w-]){re.escape(name)}(?![\w-])", haystack):
-            named.append(name)
+    # Substring first, regex only for the handful that survive it: the corpus is a few hundred rows and a round
+    # runs this once, but a per-round sweep of one regex per row is the kind of thing that stops being free.
+    named = [
+        name
+        for case in cases
+        if (name := str(getattr(case, "name", ""))) and name in haystack and re.search(rf"(?<![\w-]){re.escape(name)}(?![\w-])", haystack)
+    ]
     return tuple(sorted(set(named)))
 
 

src/openultrasast/learning/scoring.py

@@ -34,9 +34,9 @@ def is_post_cutoff(fix_date: str, cutoff: str) -> bool:
     A year-precision row counts only when the whole year is after the cutoff, so a coarse date can under-claim the
     clean slice but never over-claim it — and over-claiming is the one that would turn contamination into a result.
     """
-    if not fix_date or not cutoff:
-        return False
-    earliest = fix_date if len(fix_date) == 10 else f"{fix_date[:4]}-01-01"
+    if not fix_date or len(cutoff) != 10:
+        return False  # a cutoff coarser than a day is the one over-claim path there is, so it is refused outright
+    earliest = fix_date if len(fix_date) == 10 else f"{fix_date}-01-01"[:10]
     return earliest > cutoff
 
 
@@ -96,7 +96,22 @@ def score_pair_family(
             unscorable_reason=reason,
         )
     spans = _spans(case, ranges)
-    fixed_spans = _spans(case, fixed_ranges) if fixed_ranges is not None else spans
+    if fixed_ranges is None:
+        fixed_spans = spans
+    else:
+        # A trap pair's fixed side holds a different, correctly guarded handler; `fix_function` names it. Without a
+        # span there, every fixed-side finding reads as outside the labeled function and the pair scores as silence.
+        fixed_spans = _spans(case, fixed_ranges, function=str(getattr(case, "fix_function", "") or ""))
+        if not fixed_spans:
+            return PairFamilyScore(
+                pair=name,
+                family=family,
+                slice=slice_name,
+                fix_date=str(getattr(case, "fix_date", "") or ""),
+                runs=(),
+                outcome="unscorable",
+                unscorable_reason="unresolved_label:fixed",
+            )
     outcomes: list[FamilyOutcome] = []
     other_vuln = other_fixed = fabricated_vuln = fabricated_fixed = 0
     rung: Rung = "suspicion"
@@ -112,7 +127,15 @@ def score_pair_family(
         rung = _best_rung(rung, hits_vuln)
         outcomes.append(_outcome(bool(hits_vuln), bool(hits_fixed)))
     if not outcomes:
-        return PairFamilyScore(pair=name, family=family, slice=slice_name, runs=(), outcome="unscorable", unscorable_reason="no_runs")
+        return PairFamilyScore(
+            pair=name,
+            family=family,
+            slice=slice_name,
+            fix_date=str(getattr(case, "fix_date", "") or ""),
+            runs=(),
+            outcome="unscorable",
+            unscorable_reason="no_runs",
+        )
     majority = Counter(outcomes).most_common(1)[0][0]
     return PairFamilyScore(
         pair=name,
@@ -170,9 +193,9 @@ def unscorable_reason(
     return None
 
 
-def _spans(case: object, ranges: FunctionRanges) -> tuple[tuple[str, int, int], ...]:
-    """The (path, start, end) spans of the labeled functions, from the parsed vulnerable side."""
-    wanted = {str(getattr(row, "function", "") or "") for row in getattr(case, "expected", ()) or ()}
+def _spans(case: object, ranges: FunctionRanges, *, function: str = "") -> tuple[tuple[str, int, int], ...]:
+    """The (path, start, end) spans of the labeled functions. ``function`` overrides the label for the fixed side."""
+    wanted = {function} if function else {str(getattr(row, "function", "") or "") for row in getattr(case, "expected", ()) or ()}
     wanted.discard("")
     found: list[tuple[str, int, int]] = []
     for path, entries in ranges.items():

src/openultrasast/pairs.py

@@ -131,6 +131,12 @@ class PairCase:
     unscorable: str | None = None  # computed: identical_twin, known_limit:<reason> (learning-harness Req 4.1)
     split_declared: bool = True  # False when the catalog row named no split, so the split helper may assign one
     fix_date: str = ""  # ISO date of the fixing commit; the catalog generator fills it, the split helper orders by it
+    # The function the *fixed* side actually holds, when it is not the labeled one. Real-Vuln-Benchmark traps take
+    # their fixed side from a different, correctly guarded handler of the same snapshot, and without this the fixed
+    # side resolves no span at all: every finding there reads as outside the labeled function, so the pair scores as
+    # silence rather than as a leak (learning-harness Req 3.2, 4.1).
+    fix_function: str = ""
+
     recipe: tuple[tuple[str, object], ...] = ()  # pointer pairs carry their harvest recipe instead of excerpts
     # (relpath, vuln document, fixed document): files that register the handler from outside its own module — an
     # OpenAPI operation, a router table. Materialized beside the excerpt so the mapper can name the handler at all
@@ -292,6 +298,7 @@ def _load_catalog_file(path: Path) -> tuple[PairCase, ...]:
                 unscorable=_unscorable_reason(known_limit, vuln_file, fixed_file, vendored=vendored),
                 split_declared="split" in item,
                 fix_date=str(item.get("fix_date", "")),
+                fix_function=str(item.get("fix_function", "")),
                 review_tier=review_tier,
                 reviewer=reviewer,
                 vendored=vendored,
@@ -625,8 +632,10 @@ def _body_digest(path: Path) -> str | None:
     return hashlib.sha256(body.encode()).hexdigest() if body else None
 
 
-# The reasons a row may carry verbatim from the catalog, because the loader computes them under the same names.
-UNSCORABLE_REASONS = frozenset({"identical_twin"})
+# The reasons a row may carry verbatim from the catalog, because something else computes them under the same
+# names. A maintainer declaring one of these is stating the fact the computation would have found, so the two must
+# not end up as two buckets in the same report (learning-harness Req 4.1, 10.3).
+UNSCORABLE_REASONS = frozenset({"identical_twin", "unsupported_language", "unresolved_label", "missing_context"})
 
 
 def _unscorable_reason(known_limit: object, vuln_file: Path, fixed_file: Path, *, vendored: bool) -> str | None:

src/openultrasast/semantic/loo.py

@@ -93,20 +93,31 @@ def evaluate_loo(cases: Sequence[PairCase], *, facts: SemanticFacts | None = Non
     targets: list[PairCase] = []
     skipped: list[tuple[str, str]] = []
     for case in cases:
-        if case.known_limit:
-            skipped.append((case.name, f"known_limit:{case.known_limit}"))
+        if case.unscorable or case.known_limit:
+            # `PairCase.unscorable` verbatim when the loader computed one: one spelling per corpus fact, and it
+            # already carries the `known_limit:` prefix when the limit is a maintainer's own word rather than one
+            # of the vocabulary's. A case built by hand may carry only the declaration.
+            skipped.append((case.name, case.unscorable or f"known_limit:{case.known_limit}"))
         elif not case.vuln_file.is_file() or not case.fixed_file.is_file():
             skipped.append((case.name, "pointer_pair_not_cached" if not case.vendored else "excerpt_missing"))
         else:
             targets.append(case)
-    from ..learning.split import is_teacher, refuse_if_holdout
+    from ..learning.split import GATING_TIERS, holdout_names, is_teacher, refuse_if_holdout
 
     # learning-harness Req 5.1: every pair is still scored, but only a train-split teacher may seed a store.
     lessons = {case.name: pair_lessons(case, loaded) for case in targets if is_teacher(case)}
     outcomes = [_hold_out(case, targets, lessons, loaded) for case in targets]
-    # Req 5.2: the pairs that were kept out of the teaching set are named, not silently dropped.
-    refusal = refuse_if_holdout([case.name for case in targets if case.name not in lessons], targets)
+    # Req 5.2: the holdout pairs kept out of the teaching set are named, not silently dropped. Non-teachers held
+    # back for their tier or their unscorable reason are already in `skipped` with that reason.
+    excluded = [case.name for case in targets if case.name not in lessons]
+    refusal = refuse_if_holdout(excluded, targets)
     refusals = (refusal.degradation(),) if refusal is not None else ()
+    withheld = tuple(
+        (case.name, "tier" if case.review_tier not in GATING_TIERS else "unscorable")
+        for case in targets
+        if case.name in set(excluded) - set(holdout_names(targets))
+    )
+    skipped.extend((name, f"not_a_teacher:{why}") for name, why in withheld)
     return LooResult(
         outcomes=tuple(outcomes),
         per_slice=_group(outcomes, lambda item: (item.slice,)),

src/openultrasast/semantic/seed.py

@@ -197,12 +197,12 @@ def _skip_reason(case: PairCase) -> str | None:
 
     if case.review_tier not in GATING_TIERS:
         return f"tier:{case.review_tier}"
-    if case.unscorable:
-        # One spelling per corpus fact. `identical_twin` declared in the catalog and `identical_twin` computed from
-        # the bytes are the same reason, and calling one of them `split:train` is simply untrue (Req 5.1, 10.3).
-        return f"unscorable:{case.unscorable}"
-    if case.known_limit:
-        return f"known_limit:{case.known_limit}"
+    if case.unscorable or case.known_limit:
+        # `PairCase.unscorable` verbatim when the loader computed one, the same string every other path reports:
+        # `identical_twin` declared in the catalog and `identical_twin` computed from the bytes are one fact, and
+        # calling either of them `split:train` is simply untrue. A case built by hand may carry only the
+        # declaration (Req 5.1, 10.3).
+        return case.unscorable or f"known_limit:{case.known_limit}"
     if not is_teacher(case) and case.vendored:
         return f"split:{case.split}"
     if not case.vuln_file.is_file() or not case.fixed_file.is_file():

tests/test_learning_cutoff.py

@@ -44,6 +44,9 @@ def test_every_vendored_row_carries_the_date_its_recipe_recorded() -> None:
         ("2026", "2026-03-01", False),  # a year-precision row that might be before it is not claimed
         ("2026-04-02", "2026-03-01", True),
         ("2024", "2025-07-01", False),
+        ("2026-05", "2026-03-01", True),  # month precision: the earliest day of May is after 1 March
+        ("2026-05", "2026-06-01", False),  # and before 1 June, so the row is not claimed
+        ("2025", "2025", False),  # a cutoff coarser than a day is refused rather than compared
         ("", "2025-07-01", False),
     ],
 )

tests/test_learning_refusals.py

@@ -45,7 +45,7 @@ def test_a_train_split_twin_is_refused_for_being_a_twin_not_for_its_split(tmp_pa
     `vulnpy-deserialization-do-pickle-load` are train-split identical twins."""
     from openultrasast.semantic.seed import _skip_reason
 
-    assert _skip_reason(_case(tmp_path, "twin", split="train", unscorable="identical_twin")) == "unscorable:identical_twin"
+    assert _skip_reason(_case(tmp_path, "twin", split="train", unscorable="identical_twin")) == "identical_twin"
     assert _skip_reason(_case(tmp_path, "held", split="holdout")) == "split:holdout"
     assert _skip_reason(_case(tmp_path, "good", split="train")) is None
 

tests/test_learning_round4_evidence.py

@@ -0,0 +1,247 @@
+"""Three defects the round-4 review found, one test each (learning-harness, Req 3.2, 4.1, 9.1).
+
+Two of them were the same shape as defects already rejected once: a proposer that cannot propose because the call
+is wrong and the error is swallowed, and a K-run floor that turned a documented command into a traceback. The
+third is the fixed-side half of the range defect — 28 rows resolve a span on the vulnerable side and none on the
+fixed side, so every finding there reads as outside the labeled function and the pair scores as silence.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import pytest
+
+from openultrasast.benchmark import ExpectedFinding
+from openultrasast.findings import StaticFinding
+from openultrasast.harness_ext import has_harnessx
+from openultrasast.learning.families import load_families
+from openultrasast.pairs import DEFAULT_CATALOG, PairCase, load_pair_catalog, select_vendored
+from openultrasast.semantic.extra import has_semantic_extra
+
+VULN = "from flask import request\nimport os\n\n\ndef run():\n    os.system(request.args.get('cmd'))\n"
+FIXED_OTHER = "from flask import request\n\n\ndef unrelated():\n    return 'ok'\n"
+
+
+def _finding(line: int) -> StaticFinding:
+    return StaticFinding(
+        finding_id=f"tool-hunter:app.py:{line}",
+        path="app.py",
+        title="t",
+        severity="high",
+        confidence="low",
+        evidence_level="suspicion",
+        rationale="r",
+        line=line,
+        function_name="run",
+        reachability_status="unknown",
+        reachability_evidence=[],
+        reachability_conditions=[],
+        tags=["tool-hunter", "family:injection"],
+        ranking_priority=0.0,
+    )
+
+
+# --- the meta-agent call -----------------------------------------------------
+
+
+@pytest.mark.skipif(not has_harnessx(), reason="the composition seam only runs when the extra is importable")
+def test_the_meta_agent_is_built_with_a_model_config_not_a_model_name(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+    """`MetaAgent(inner_model=...)` wants a `ModelConfig`; a string reaches `self.inner_model.agentic(...)` and dies.
+
+    Every other HarnessX composition site in this repository builds the config properly. Here the mistake was
+    invisible because a `# type: ignore` hid it from mypy and a bare `except Exception` turned it into the
+    degradation reason `meta_agent_failed`, which reads exactly like the model declining."""
+    from openultrasast.learning import proposer as proposer_module
+
+    seen: dict[str, object] = {}
+
+    class _Agent:
+        def __init__(self, **kwargs: object) -> None:
+            seen.update(kwargs)
+
+        async def evolve(self, **kwargs: object) -> Path:
+            seen["evolve"] = kwargs
+            workspace = kwargs["current_config"]
+            assert isinstance(workspace, Path)
+            (workspace / "checklist.md").write_text("- from the meta agent\n")
+            return workspace
+
+    monkeypatch.setattr(proposer_module, "_meta_agent_class", lambda: _Agent)
+    monkeypatch.setattr(proposer_module, "_model_config", lambda model: f"ModelConfig({model})")
+    (tmp_path / "injection").mkdir()
+    (tmp_path / "injection" / "checklist.md").write_text("- old\n")
+    proposal = proposer_module.HarnessXProposer(configs_dir=tmp_path, model="m").propose(_facts())
+    assert proposal is not None and proposal.change == {"checklist.md": "- from the meta agent\n"}
+    assert seen["inner_model"] == "ModelConfig(m)", "the agent was handed a model name where it wanted a config"
+    assert seen["allowed_write_roots"] != (tmp_path / "injection",), "the live family directory is never a write root"
+
+
+@pytest.mark.skipif(not has_harnessx(), reason="the composition seam only runs when the extra is importable")
+def test_a_meta_agent_built_wrongly_is_not_reported_as_a_declined_proposal(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+    """ "We called the agent wrong" and "the agent had nothing to propose" are different facts."""
+    from openultrasast.learning import proposer as proposer_module
+
+    def _boom() -> object:
+        raise TypeError("inner_model must be a ModelConfig")
+
+    monkeypatch.setattr(proposer_module, "_meta_agent_class", _boom)
+    (tmp_path / "injection").mkdir()
+    proposer = proposer_module.HarnessXProposer(configs_dir=tmp_path, model="m")
+    with pytest.raises(TypeError, match="ModelConfig"):
+        proposer.propose(_facts())
+
+
+def _facts():  # type: ignore[no-untyped-def]
+    from openultrasast.learning.detectors import FamilyConfig
+    from openultrasast.learning.proposer import FailureFacts
+
+    return FailureFacts(
+        family="injection",
+        config=FamilyConfig(
+            family="injection",
+            version="0",
+            prompt="p",
+            checklist="- old",
+            tools=("read_file",),
+            max_steps=1,
+            max_cost_usd=0.1,
+            max_chars=100,
+        ),
+    )
+
+
+# --- the pairs command in its documented form --------------------------------
+
+
+def test_the_pairs_hunter_path_runs_without_a_k_runs_flag(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+    """The floor is a default, not a trap: `ousast pairs --hunter` must work as documented."""
+    from openultrasast.cli import main
+
+    (tmp_path / "a-v.py").write_text(VULN)
+    (tmp_path / "a-f.py").write_text(FIXED_OTHER)
+    (tmp_path / "pairs.toml").write_text(
+        '[[pair]]\nname = "a"\nslice = "vibe-py"\nlanguage = "python"\nvuln = "a-v.py"\nfixed = "a-f.py"\n'
+        'relpath = "app.py"\nsplit = "train"\nreview_tier = "seeded"\n\n'
+        '[[pair.expected]]\ncwe = "CWE-78"\nclass = "x"\npath = "app.py"\nfunction = "run"\nmechanism = "other"\nfamily = "injection"\n'
+    )
+    monkeypatch.setenv("OPENULTRASAST_HUNTER_CLIENT", "scripted")
+    assert main(["pairs", "--catalog", str(tmp_path / "pairs.toml"), "--hunter", "--hunter-model", "scripted"]) == 0
+
+
+def test_a_k_below_the_floor_names_the_flag_instead_of_raising(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+    from openultrasast.cli import main
+
+    (tmp_path / "pairs.toml").write_text("")
+    monkeypatch.setenv("OPENULTRASAST_HUNTER_CLIENT", "scripted")
+    with pytest.raises(SystemExit, match="k-runs"):
+        main(["pairs", "--catalog", str(tmp_path / "pairs.toml"), "--hunter", "--hunter-model", "scripted", "--k-runs", "1"])
+
+
+# --- the fixed side of a trap pair -------------------------------------------
+
+
+def test_a_trap_pair_is_scored_against_the_function_its_fixed_side_really_holds(tmp_path: Path) -> None:
+    """A Real-Vuln-Benchmark trap takes its fixed side from a different, correctly guarded handler.
+
+    With no span for it, every fixed-side finding counts as outside the labeled function and the pair scores as
+    silence — the same flattering direction as the range defect, for the 28 rows that resolve one side only."""
+    from openultrasast.learning.scoring import score_pair_family
+
+    (tmp_path / "v.py").write_text(VULN)
+    (tmp_path / "f.py").write_text(FIXED_OTHER)
+    case = PairCase(
+        name="trap",
+        slice="vibe-py",
+        language="python",
+        origin="test",
+        vuln_file=tmp_path / "v.py",
+        fixed_file=tmp_path / "f.py",
+        relpath="app.py",
+        expected=(
+            ExpectedFinding(
+                cwe="CWE-78", vulnerability_class="x", path="app.py", evidence="", function="run", mechanism="other", family="injection"
+            ),
+        ),
+        min_recall=1.0,
+        fix_policy="silent",
+        fix_function="unrelated",
+    )
+    score = score_pair_family(
+        case,
+        "injection",
+        [[_finding(6)]],
+        [[_finding(5)]],
+        ranges={"app.py": (("run", 5, 6),)},
+        fixed_ranges={"app.py": (("unrelated", 4, 5),)},
+        taxonomy=load_families(),
+    )
+    assert score.outcome == "both_flagged", "the finding is inside the fixed side's own handler; that is a leak"
+
+
+def test_a_row_whose_fixed_side_resolves_no_span_says_so_instead_of_scoring_silent(tmp_path: Path) -> None:
+    from openultrasast.learning.scoring import score_pair_family
+
+    (tmp_path / "v.py").write_text(VULN)
+    (tmp_path / "f.py").write_text(FIXED_OTHER)
+    case = PairCase(
+        name="orphan",
+        slice="vibe-py",
+        language="python",
+        origin="test",
+        vuln_file=tmp_path / "v.py",
+        fixed_file=tmp_path / "f.py",
+        relpath="app.py",
+        expected=(
+            ExpectedFinding(
+                cwe="CWE-78", vulnerability_class="x", path="app.py", evidence="", function="run", mechanism="other", family="injection"
+            ),
+        ),
+        min_recall=1.0,
+        fix_policy="silent",
+    )
+    score = score_pair_family(
+        case,
+        "injection",
+        [[_finding(6)]],
+        [[]],
+        ranges={"app.py": (("run", 5, 6),)},
+        fixed_ranges={"app.py": (("unrelated", 4, 5),)},
+        taxonomy=load_families(),
+    )
+    assert score.outcome == "unscorable" and score.unscorable_reason == "unresolved_label:fixed"
+
+
+def test_the_trap_rows_carry_their_fixed_side_function_in_the_catalog() -> None:
+    """`fix_function` lived only in the recipes; the scorer could not see which handler the fixed side holds."""
+    cases = {case.name: case for case in select_vendored(load_pair_catalog(DEFAULT_CATALOG))}
+    assert cases["threatbyte-api-v1-get"].fix_function == "post"
+    assert cases["threatbyte-api-v1-delete"].fix_function == "post"
+    assert cases["vampi-users-update-password"].fix_function == "register_user"
+    assert cases["vampi-books-get-by-title"].fix_function == "", "a row whose two sides are one function names none"
+
+
+@pytest.mark.semantic
+@pytest.mark.skipif(not has_semantic_extra(), reason="resolving a span on both sides needs the tree-sitter grammars")
+def test_no_vendored_row_resolves_one_side_and_not_the_other() -> None:
+    """The corpus-wide form of the same defect: 28 rows resolved a vulnerable span and no fixed-side span.
+
+    Every one of them therefore scored the fixed side as silent by construction, whatever the detector said. The
+    catalog generator now derives `fix_function` from the fixed excerpt itself, so the span exists and the leak is
+    visible."""
+    import tempfile
+
+    from openultrasast.learning.scoring import _spans
+    from openultrasast.pairs import _materialize_side, _overlay_scan
+
+    orphans = []
+    for case in select_vendored(load_pair_catalog(DEFAULT_CATALOG)):
+        if case.unscorable:
+            continue
+        with tempfile.TemporaryDirectory() as scratch:
+            vuln = _materialize_side(Path(scratch) / "vuln", case, side="vuln")
+            fixed = _materialize_side(Path(scratch) / "fixed", case, side="fixed")
+            vuln_ranges, fixed_ranges = _overlay_scan(vuln).ranges, _overlay_scan(fixed).ranges
+        if _spans(case, vuln_ranges) and not _spans(case, fixed_ranges, function=case.fix_function):
+            orphans.append(case.name)
+    assert orphans == [], f"rows whose fixed side can only ever score as silence: {orphans}"

tests/test_loo.py

@@ -89,7 +89,9 @@ def test_leave_one_out_seeds_only_trusted_pairs_and_skips_known_limit(tmp_path:
     assert set(by_name) == {"a", "b"}  # known_limit excluded entirely
     assert not by_name["a"].detected  # b is advisory: it cannot teach a
     assert by_name["b"].detected  # a (seeded) teaches b, which is scored as a held-out target
-    assert result.skipped == (("c", "known_limit:engine"),)
+    # `b` is excluded from teaching for its tier and says so; without that line a pair that silently stopped
+    # teaching would look identical to one that never could (Req 5.2).
+    assert result.skipped == (("c", "known_limit:engine"), ("b", "not_a_teacher:tier"))
 
 
 def test_cli_pairs_loo_writes_the_artifact_offline(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:

tests/test_obligation_export.py

@@ -133,7 +133,7 @@ def test_the_train_split_of_vibe_py_teaches_no_obligation_shape_today(tmp_path:
     assert refused["threatbyte-api-v1-get"] == "split:holdout"
     # Its own reason, not its split: this row is a byte-identical twin that happens to sit on the holdout side,
     # and reporting `split:holdout` would hide a corpus fact behind a split fact (Req 5.2).
-    assert refused["vampi-books-get-by-title"] == "unscorable:identical_twin"
+    assert refused["vampi-books-get-by-title"] == "identical_twin"
     flow_rows = [row for row in store.load() if row.shape is not None and row.shape.get("family", "sink") == "sink"]
     assert report.records == len(flow_rows) and len(flow_rows) >= 1  # the flow family still teaches from the train split
 

tests/test_pair_hygiene.py

@@ -26,6 +26,13 @@ def test_identical_twins_are_unscorable_and_no_row_is_dropped() -> None:
     assert per_slice == VENDORED_PER_SLICE  # nothing dropped by the new accounting
     twins = {case.name for case in cases if case.unscorable == "identical_twin"}
     assert twins == IDENTICAL_TWINS
+    # The declaration and the bytes must still agree. A declared limit that stopped being true after a re-harvest
+    # would otherwise be invisible: the mirror image of the risk declaring it was meant to close.
+    from openultrasast.pairs import _body_digest
+
+    declared = [case for case in cases if case.known_limit == "identical_twin"]
+    assert {case.name for case in declared} == IDENTICAL_TWINS
+    assert all(_body_digest(case.vuln_file) == _body_digest(case.fixed_file) for case in declared)
     assert all(case.unscorable is None for case in cases if case.name not in IDENTICAL_TWINS and not case.known_limit)