fix(learning-harness): the round-3 review findings, remediated RED-first

Eight of nine group-3 tasks were rejected. Each defect was hiding a number,
and the numbers are why the loop was not learning:

- The fixed side was judged against the *vulnerable* side's function
  ranges. 191 of 227 pairs with a resolvable labeled span (84%) have
  different spans on the two sides, so a leak was recorded as silence,
  always in the direction that flatters Youden.
- Round zero measured every family with the generalist. `family:unknown`
  earns no credit against any real family, so every family but `unknown`
  baselined at a structural zero and every noise floor with it.
- The cross-family sweep counted absolute failures instead of negative
  flips. On a corpus where most pairs fail at round zero that rejects
  every round for damage it did not do. Unscorable rows voted too.
- K runs was declared and never threaded; `pairs --k-runs` was parsed and
  discarded, and its only test grepped `--help`.
- The meta-agent proposer was a stub, so `learning round` could never
  propose anything. It now composes MetaAgent.evolve against a scratch
  copy of the family directory, and `--proposals FILE` runs a recorded
  sequence offline.
- `refused_holdout` was a vocabulary word no code path could produce, and
  the covering test accepted either outcome.
- Cost was metered once, after everything had been paid for; a stage that
  raised left no record at all.
- The corrected lever number was published beside the earlier one rather
  than in place of it, and the test named for that property never checked.
- Refusals were `continue` statements: `Refusal.degradation()` existed and
  nothing called it. A train-split identical twin was reported as
  `split:train`, which is untrue.

Test repairs that each changed a number: two split tests measured an
`x = 1` fixture that teaches nothing whatever the rule says; two CLI
scoreboard tests asserted against all-zero output produced by a stub that
answers twice and puts its finding outside the labeled function.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-06 13:03:06 +02:00
co-authored by Claude Opus 5
parent c2d9c4ad01
commit 1843ef9a02
23 changed files with 1431 additions and 159 deletions
+10 -1
View File
@@ -107,13 +107,22 @@ def test_the_corrected_lever_number_replaces_the_earlier_one(tmp_path: Path) ->
)
)
roadmap = tmp_path / "roadmap.md"
roadmap.write_text(ROADMAP + "\nThe lever recovered 7 of 17 holdout pairs at Youden +0.059.\n")
stale = "human holdout profile pair_correct 0 -> 6 of 17, Youden 0 -> +0.353, round 2 `no_proposals`."
roadmap.write_text(f"{ROADMAP}\nMeasured 2026-09-05 with the vibe-py candidates: {stale}\n")
report = publish(learning_dir=out, measurements_dir=measurements, roadmap=roadmap)
text = roadmap.read_text()
assert "-0.0588" in report.table or "-0.059" in report.table
assert "5 of 17" in report.table or "detected 5" in report.table
assert "holdout pairs stopped teaching" in report.table.lower() or "train split only" in report.table.lower()
assert text.count("learning-harness:begin") == 1
# Req 5.3 says *in place of* the earlier one. Leaving the old claim above the new section is how a document
# ends up asserting two different numbers for the same measurement.
assert stale not in text
assert "superseded" in text and "learning-harness" in text
assert "Measured 2026-09-05 with the vibe-py candidates:" in text, "the sentence is corrected, not deleted"
before = roadmap.read_text()
publish(learning_dir=out, measurements_dir=measurements, roadmap=roadmap)
assert roadmap.read_text() == before, "a second publish must be a no-op once the claim is corrected"
def test_the_roadmap_section_is_replaced_not_appended_again(tmp_path: Path) -> None: