fix(learning-harness): the round-3 review findings, remediated RED-first

Eight of nine group-3 tasks were rejected. Each defect was hiding a number,
and the numbers are why the loop was not learning:

- The fixed side was judged against the *vulnerable* side's function
  ranges. 191 of 227 pairs with a resolvable labeled span (84%) have
  different spans on the two sides, so a leak was recorded as silence,
  always in the direction that flatters Youden.
- Round zero measured every family with the generalist. `family:unknown`
  earns no credit against any real family, so every family but `unknown`
  baselined at a structural zero and every noise floor with it.
- The cross-family sweep counted absolute failures instead of negative
  flips. On a corpus where most pairs fail at round zero that rejects
  every round for damage it did not do. Unscorable rows voted too.
- K runs was declared and never threaded; `pairs --k-runs` was parsed and
  discarded, and its only test grepped `--help`.
- The meta-agent proposer was a stub, so `learning round` could never
  propose anything. It now composes MetaAgent.evolve against a scratch
  copy of the family directory, and `--proposals FILE` runs a recorded
  sequence offline.
- `refused_holdout` was a vocabulary word no code path could produce, and
  the covering test accepted either outcome.
- Cost was metered once, after everything had been paid for; a stage that
  raised left no record at all.
- The corrected lever number was published beside the earlier one rather
  than in place of it, and the test named for that property never checked.
- Refusals were `continue` statements: `Refusal.degradation()` existed and
  nothing called it. A train-split identical twin was reported as
  `split:train`, which is untrue.

Test repairs that each changed a number: two split tests measured an
`x = 1` fixture that teaches nothing whatever the rule says; two CLI
scoreboard tests asserted against all-zero output produced by a stub that
answers twice and puts its finding outside the labeled function.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG

.kiro/specs/learning-harness/tasks.md

@@ -78,59 +78,59 @@
   - _Requirements: 9.3, 9.6, 9.7, 11.3_
   - _Boundary: Journal_
 
-- [ ] 3. Integration: split, scoring paths, rounds, proposer, publish, CLI
+- [x] 3. Integration: split, scoring paths, rounds, proposer, publish, CLI
 - [x] 3.1 One teacher rule across every learning path
   - Teachers are vendored, gating-tier, train-split, scorable pairs; a refusal names any candidate taught by or touching a holdout pair; the exporter, leave-one-out, per-profile evaluation, mechanism-profile evaluation, mechanism admission and pair signals all obtain teachers through the rule and record refusals as degradations; the mechanism lever's holdout number is re-measured under the rule and its raw artifact committed for the publish task.
   - Observable: leave-one-out on the toy slice seeds from train pairs only; a candidate shape taught by a holdout pair is refused by name; the re-measured artifact is in the measurements directory.
   - _Depends: 1.3_
   - _Requirements: 5.1, 5.2, 5.3_
 
-- [ ] 3.2 Family scoring on the pairs hunter path
+- [x] 3.2 Family scoring on the pairs hunter path
   - The pair scorer's hunter path takes an injected family matcher and leak rule from the learning package, classifies each pair once, runs the admitted family detectors plus the generalist K times per side, and emits family outcomes and metrics; the overlay and inventory paths stay untouched.
   - Observable: the pairs command with a scripted client on the shared fixture reports a family outcome per pair; no text-token matching remains on the hunter path; the pair gate output is byte-identical to before.
   - _Depends: 2.1, 2.4, 2.5, 2.6_
   - _Requirements: 3.1, 3.2, 3.3, 3.5, 11.5_
 
-- [ ] 3.3 Round zero and the noise floor
+- [x] 3.3 Round zero and the noise floor
   - Clone the current hunter prompt into every family configuration; run each family K = 5 times over the scorable vendored pairs of the selected slices at deterministic settings; record the per-family negative-flip rate between runs as that family's regression budget; persist noise floors and round-zero artifacts keyed by detector model so another model can be baselined and compared without touching configurations; the web families over vibe-py and agent-vfc are the default selection and the memory-safety family over vfc is selectable but never gates.
   - Observable: with a scripted client round zero writes a noise-floor file with one budget per family and a round-zero report per slice; a second baseline under another model name writes a separate artifact and the comparison table lists both.
   - _Depends: 3.2_
   - _Requirements: 8.1, 8.2, 8.3, 8.4_
 
-- [ ] 3.4 Family detectors, verifiers and tags in scans
+- [x] 3.4 Family detectors, verifiers and tags in scans
   - In a standard scan each classified region runs its admitted family detectors plus the generalist from the directories round zero wrote (a missing family directory is a recorded degradation and that family's detector is skipped), claims pass through the family's verifier, findings carry family, detector and verifier tags in reports and SARIF, and the manifest gains a learning block with taxonomy version, per-family counts, the classifier report and the current round; the detection and map gates stay byte-identical.
   - Observable: a standard scan of the shared fixture with a scripted client writes findings tagged with a family, a detector version and a verifier rung, and the manifest block; the detection and map gate outputs diff empty.
   - _Depends: 3.3, 2.6_
   - _Requirements: 6.3, 7.1, 11.5_
 
-- [ ] 3.5 Proposer protocol, scripted proposer and the HarnessX proposer
+- [x] 3.5 Proposer protocol, scripted proposer and the HarnessX proposer
   - Failure facts for one family: misses and leaks with pair, function, sink or check, line, verifier outcome and run outcomes, plus the rejected buffer and the current configuration; a proposal is one hypothesis, one lever (tool, checklist, counterexample, memory, prompt), one file-level change inside the family directory, and predicted affected and at-risk families; the proposer never receives holdout inputs, outputs or scores.
   - A scripted proposer for tests and offline rounds; a HarnessX proposer composing the meta-agent with write roots limited to the family directory, degrading to a recorded reason without the extra.
   - Observable: the scripted proposer returns a proposal from a facts fixture; the HarnessX proposer is skipped with the reason recorded when the extra is absent and, when present, writes only inside the family directory; the write-root list excludes the verifier and corpus directories.
   - _Depends: 2.5, 2.8_
   - _Requirements: 9.1, 9.2, 5.4, 7.6_
 
-- [ ] 3.6 Acceptance rule and directory revert
+- [x] 3.6 Acceptance rule and directory revert
   - Acceptance requires train and holdout non-negative with one strictly positive and every other family within its budget; ties reject; a family directory snapshot restores byte for byte on rejection or overrun; refusals for holdout contact, length cap and lever set are outcomes with reasons.
   - Observable: a helpful change is accepted, a tie is rejected, a change that breaches another family's budget is rejected with the family named, and a revert restores the directory byte for byte.
   - Observable: every family other than the target has a byte-identical directory before and after the round, whatever the outcome.
   - _Depends: 3.3_
   - _Requirements: 6.2, 9.5_
 
-- [ ] 3.7 Evolve round orchestration, cost and journal
+- [x] 3.7 Evolve round orchestration, cost and journal
   - One round: facts, proposal, refusals, train minibatch, target-family holdout, sweep over every other family's holdout, K-run scoring at every stage as the scorer requires, accept (freeze, bump version, archive) or revert; cost metered per stage with revert and a recorded reason on overrun; a journal record and the rejected buffer written for every outcome; a round directory holds the snapshot, proposal, scores and trajectories and a repeated round number is refused.
   - Observable: with the scripted proposer each of accepted, rejected, budget-breach and cost-overrun leaves one journal record with attribution and a round directory; a repeated round number is refused.
   - The round directory holds the pair identities behind the attribution counts and every detector trajectory, redacted, so a round can be replayed and its attribution audited; every stage scores at least three runs per pair.
   - _Depends: 3.5, 3.6_
   - _Requirements: 3.5, 9.4, 9.6, 9.7, 9.8, 9.9, 11.3_
 
-- [ ] 3.8 Publish from artifacts
+- [x] 3.8 Publish from artifacts
   - One entry point regenerates every family and slice number from committed artifacts, writes a markdown table with scorable and unscorable counts, taxonomy version, noise floor and cost per point beside every number, replaces the roadmap's measured section between markers, and writes the raw artifacts under the measurements directory.
   - Observable: running the publish entry point twice without new artifacts produces byte-identical output; the corrected mechanism-lever number from the re-measurement artifact replaces the earlier roadmap value.
   - _Depends: 3.1, 3.3_
   - _Requirements: 11.1, 11.2, 4.5, 5.3_
 
-- [ ] 3.9 Learning subcommands
+- [x] 3.9 Learning subcommands
   - Subcommands for classify, score, baseline, round and publish with the flags the design names; the pairs hunter path gains the K-runs flag; every step runs under the harness runtime with degradations for missing extras, endpoints or sandbox.
   - Observable: each subcommand exits zero on the shared fixture with a scripted client and writes its artifact; the roadmap section written by the publish subcommand carries the command line that produced it; a missing endpoint records the reason and the remaining steps complete.
   - _Depends: 3.7, 3.8_
@@ -219,3 +219,17 @@
   - Three existing tests encoded the old corpus and now encode the new one: the TypeScript parse pin (the lensops excerpt parses now), the classifier report (`labeled` 0 -> 4, the four seeded rows; the six `title`-tier agent rows carry a family but are not labels a maintainer stands behind), and the hygiene report ordering.
   - **New honesty finding, recorded in the hygiene artifact as `cross_split_duplicates`: nine of the ten duplicate groups sit on both sides of the split.** Two forks of one project (`openniw-jobs-0fc947` train, `openniw-jobs-b74de3` holdout) and eight CVEs harvested twice under different names are byte-identical, so the train row teaches the holdout row exactly. `split_by_repository` cannot see it — the duplicates are different repositories. `duplicate_groups` is now ordered by name so the committed report does not reshuffle when one excerpt is re-harvested.
   - Pre-existing drift the regeneration corrected, unrelated to this task: `catalog.toml` in git disagreed with a fresh `catalog_gen` run at the group-3 tip on two derived labels (`origin` -> `createApplication`, `normalizeDomain` -> `getGoogleProvider`). Verified in a detached worktree at `7a33756` before touching anything.
+- Group 3 (2026-09-06). Tasks 3.1-3.9 implemented in `86ed54c`, `fcda1ee`, `e8e6baf`, `127dbef`, `ad7c98e`, `346cc70`, `16d9bc2`, `b955251`, `b8ad42c`, `7a33756`. RED counts, derived by the reviewer against each task's parent commit: 3.1 4/6, 3.2 6/6, 3.3 6/6, 3.4 3/5, 3.5 10/10, 3.6 12/12, 3.7 9/9, 3.8 7/7, 3.9 7/7. Deviations from the design, recorded here because they were not written down at the time: `learning.split` duck-types `PairCase` through `getattr` rather than importing it; `pairs.py` reaches `learning` through function-local imports rather than the design's injected callable; `run_baseline` takes a scan factory where the design named a client. One flaky test found and fixed in `346cc70` (`_is_vulnerable_side` keyed on `"-v" in str(root)`, which random temporary directory names matched about once in eight; harness noise recorded as detector noise would inflate every family's regression budget).
+- Review round 3 (2026-09-06) APPROVED 3.4; REJECTED 3.1, 3.2, 3.3, 3.5, 3.6, 3.7, 3.8, 3.9. The reviewer ran every check in detached worktrees at the group tip against `71efc9c`, derived the RED phase per task, mutation-tested `decide`, the teacher rule, the snapshot restore and the round's revert paths (7 mutants on the acceptance rule, 6 killed and 1 equivalent), ran the three flakiness-prone files 61 times with varied `PYTHONHASHSEED` without a failure, and confirmed the three gate outputs byte-identical and the extra-free matrix unchanged. Every rejection was remediated RED-first; the numbers each defect was hiding are recorded below because they are the point of the spec.
+  - **3.2, the fixed side was judged against the vulnerable side's line ranges.** A fix that adds a guard above the handler moves its first line, and the reviewer measured that **191 of 227 pairs with a resolvable labeled span (84%) have different spans on the two sides**. A finding inside the fixed handler was therefore read as outside it: a leak recorded as silence, always in the direction that flatters Youden. `score_pair_family` now takes `fixed_ranges`; the fixed side's parse was already being computed and thrown away.
+  - **3.2 and 3.9, K runs was declared and never threaded.** `_evaluate_hunter_pair(..., runs=1)` had one call site that passed nothing, `evaluate_catalog` had no parameter at all, and `pairs --k-runs` was parsed and discarded; the only test grepped `--help`. K now flows `pairs --k-runs` -> `evaluate_catalog` -> `evaluate_pair` -> `_evaluate_hunter_pair`, defaults to three and refuses less, and the test counts scans per side.
+  - **3.2, the generalist could never be scored.** `pairs --hunter` fed the class-aware scorer findings with no `family:` tag, so it reported recall 0 by construction. `run_tool_hunter` now reads an optional `family` field into the tag (the design said it would) and `make_hunter_scan(taxonomy=...)` asks for one from the closed list. A generalist that names no family is unanswerable; one that names a wrong family is counted as noise, which is the honest treatment.
+  - **3.3 and 3.9, round zero measured every family with the generalist.** `run_baseline` took one `scan`, and the CLI handed it the `unknown` configuration; since `related(X, "unknown")` is `lateral` by the taxonomy's own decision that abstention earns no credit, **every family except `unknown` baselined at a structural zero** and every noise floor with it. `run_baseline` now takes a `scan_factory` and runs each family's own configuration, reports per slice as well as per family, and refuses fewer than three runs. This is the decision the group-2 review asked for on `unknown` serving as both abstention and generalist: it is the generalist for *scanning*, and it is never credited as a family for *scoring*.
+  - **3.5, the meta-agent proposer was a stub** that returned None after confirming the extra was present, so `ousast learning round` could never propose anything at any time. It now composes `MetaAgent.evolve` behind `harness_ext` against a scratch **copy** of the family directory — the live tree is never touched, because the round snapshots only after a proposal exists — and builds the proposal from the diff of that copy, so the one-file rule and the lever whitelist are enforced on what the agent did rather than on what it claimed. `learning round --proposals FILE` runs a recorded sequence offline, which is what makes the command exercisable without the extra at all.
+  - **3.6 and 3.1, the holdout refusal existed only as a vocabulary word.** `journal.py` documented `refused_holdout` and no code path could produce it; the covering test asserted `outcome == "refused_holdout" or outcome == "rejected"`, which passes whichever way the code behaves. The round now runs `refuse_if_holdout` over every corpus pair the proposal names — in its predictions and in the file it writes — and journals `refused_holdout` with the pair names. A memory entry that quotes a held-out pair looks exactly like a legitimate hard negative until the holdout number is read as if it meant something.
+  - **3.7, the sweep counted absolute failures, not negative flips.** Req 9.5 compares against the pre-change state; the code counted every pair of another family that was not `pair_correct` and compared that to a budget derived from the run-to-run flip rate. On the real corpus, where most pairs fail at round zero, that rejects **every** round for damage it did not do — a harness that then reports it cannot improve. Unscorable rows voted too, against Req 4.1. The sweep is now scored before and after the change, `_negative_flips` counts only pairs that were right and are not, unscorable rows leave the count, and attribution is derived from the same comparison instead of from absolute failures.
+  - **3.7, cost was metered once**, after the minibatch, the holdout and the whole cross-family sweep had all been paid for; Req 9.8 says stop, revert and record. The cap is now read after every stage. A stage that raises is journalled as `reverted`/`stage_failed` with its round directory before the exception leaves, because an exception that loses the record also loses the money the round already spent. A rejected round's version is archived: Req 9.7 exists so a configuration that lost overall but is the only one that ever won some pair is not forgotten.
+  - **3.8, the corrected lever number was published beside the earlier one, not in place of it** (Req 5.3), and the test named for that property never checked it. `publish` now rewrites the superseded claim to point at the regenerated section, idempotently, and the test asserts the old sentence is gone. Run against the real roadmap: the 2026-09-05 claim "human holdout profile pair_correct 0 -> 6 of 17, Youden 0 -> +0.353" now says it was measured with holdout pairs teaching and names the artifact that corrects it.
+  - **3.1, refusals were `continue` statements.** `Refusal.degradation()` existed and nothing called it. The mechanism lever now records every candidate it declined with the holdout pairs that taught it, leave-one-out records the pairs it kept out of the teaching set, and `improve` carries both into the round outcome. `semantic.seed` reported `split:train` for a train-split identical twin, which is simply untrue; an unscorable row now reports its own reason. `evaluate_profiles` and `evaluate_mechanism_profiles` are deliberately split-blind and now say so in their docstrings: they measure a ruleset and never learn from a pair, so a filter there would only shrink the evidence.
+  - Test-quality repairs the review asked for, each of which changed a number: two split tests were measuring an `x = 1` fixture that teaches nothing whatever the rule says, so both arms now use a pair that really carries a shape and the lever test asserts the same shape *is* admitted when a train pair taught it; the two CLI scoreboard tests asserted against output that was all zeros, and now use a side-aware detector stub (the shipped `scripted` stub answers twice, then goes quiet, and puts its finding outside the labeled function — three separate ways to make a family number zero); `ScriptedChatClient` gained an opt-in `cycle` so a stub cannot silently run out of answers mid-run; the sandbox-candidate assertion in `tests/test_learning_scan.py` compared verdict paths against a prefix no implementation would produce and now compares finding ids; the manifest's classifier block reported `measure_classifier([])`, all zeros, and now reports what this scan actually routed.
+  - Carry-forward, unchanged and still capping every holdout number: the nineteen straddling repositories, and — found during task 4.1 — **nine of the ten duplicate groups sit on both sides of the split**, which the repository-level split cannot see because the duplicates are different repositories.

.kiro/steering/roadmap.md

@@ -51,7 +51,7 @@ Leave-one-out (corpus-seeded-mechanisms, 2026-09-05; the corpus as a teacher, ne
 | github | 6 | 0 | 6 | 0.000 | 0 | advisory tier |
 | vfc-js | 17 | 0 | 17 | 0.000 | 0 | advisory tier |
 
-The loop is closed: `ousast mechanisms export` writes candidates, `ousast improve --pair-catalog` admits the ones that recover a holdout pair without leaking (per-profile clause over variant search; `mechanism_leak`/`mechanism_no_gain` revert byte for byte), the scan searches the admitted store and reports name the mechanism and its known fix. Measured 2026-09-05 on the `python-vulnerable` benchmark target with the vibe-py candidates: 4 records admitted in round 1 (`execute/1 parameter` taught by 7 pairs among them), human holdout profile pair_correct 0 -> 6 of 17, Youden 0 -> +0.353, round 2 `no_proposals`.
+The loop is closed: `ousast mechanisms export` writes candidates, `ousast improve --pair-catalog` admits the ones that recover a holdout pair without leaking (per-profile clause over variant search; `mechanism_leak`/`mechanism_no_gain` revert byte for byte), the scan searches the admitted store and reports name the mechanism and its known fix. Measured 2026-09-05 on the `python-vulnerable` benchmark target with the vibe-py candidates: 4 records admitted in round 1 (`execute/1 parameter` taught by 7 pairs among them), superseded by the learning-harness section below: that number was measured with holdout pairs teaching (see `benchmarks/measurements/2026-09-06-mechanism-lever-split.json`).
 
 Measured 2026-09-05 after review tiers (Req 9) and pointer slices (Req 10): vibe-py 115 = 35 vendored human (13/35, Youden +37.1%) + 80 `agent` pointer pairs over the 40 LLM-generated Real-Vuln repositories (0/80, Youden 0.0; the LLM-repo baseline, scored by the nightly `pairs --slice vibe-py --pointers` only, never in tests or gates); agent-vfc 59 = 29 vendored + 30 pointer, tier `title`, 1/59, Youden 0.0, 2 unresolved labels; sast, vfc-js, vfc unchanged; detection/map/local-pair gate outputs byte-identical to the pre-spec commit. Only `seeded`/`reviewed` pairs gate the improve loop. Carry-forwards: `language_unsupported` loss counter and multi-line C/C++ declarator rescue (overlay-ir-completeness), `profiles_ungated` reporting and an offline test for the agent-vfc pointer builder (corpus-seeded-mechanisms).
 
@@ -89,3 +89,13 @@ sast: java-sqli moved MISS→LEAK (coverage counted on both sides; the fixed twi
 - [ ] overlay-ir-completeness -- Shared CST walker completeness (C `init_declarator`, lvalue paths for field stores, per-grammar literal kinds, local ERROR-node tolerance, multi-line C/C++ declarators the walker names `<anon>` and `functions.py`'s one-line rescue cannot name: `chromium-cve-2022-2162`, `openssl-cve-2025-69421`; a Scorer-side multi-line-aware rescue is the cheaper alternative) plus Python parity with stdlib ast (f-string constness, augmented assignment, kwargs); injection-regime facts: container/transformer taint, `weak_literals` consulted, parameter sources off for injection. Measured on sast and vibe-py. Dependencies: pair-corpus-honesty
 - [ ] authorization-obligations -- Absence bugs as obligations, not flows: obligated operations (protected read/write, privileged action, security setting) and discharger kinds (path guard, identity constraint from authenticated context, ownership check, non-permissive value) as closed facts; obligation shapes learned from trusted pairs; the application's policy recovered from sibling-handler consistency and an optional declared, versioned policy file; violations evaluated on the path (function-local degraded mode without it) with `consistency_violation` / `declared_policy_violation` rungs below proof; `obligations` improve lever under the holdout gate; the hunter adjudicates intent only. Measured leave-one-out on the vibe-py and agent-vfc absence pairs (today 0 detected; 15 of 29 agent-vfc rows). Dependencies: reachability-flow-model, guard-dominance-regime, corpus-seeded-mechanisms. **Requirements drafted 2026-09-05, unapproved.**
 - [ ] guard-dominance-regime -- Guard and Use IR records; guard dominance by nesting or early exit; memory sinks (memcpy/malloc/deref) with operand roles, nullable callees, out-param transformers, fixed-array capacity; same regime applied to web absence bugs (route auth guard, identity from auth context). Consumes path records. Measured on vfc, vfc-js, agent-vfc. Dependencies: reachability-flow-model, overlay-ir-completeness
+
+<!-- learning-harness:begin -->
+
+## Classified detector numbers
+
+Generated by `uv run ousast learning publish`.
+
+No baseline has been recorded yet, so there is nothing to publish. Run round zero first; a number without the run that produced it is a claim, not a measurement.
+
+<!-- learning-harness:end -->