docs(learning-harness): round 5, and what it says about the experiment itself

Round 5 re-ran round 1's content through the lever it should have used --
counterexamples, rendered to the detector as "Worked examples" rather than
"Known non-issues, do not report these" -- and was rejected because one pair
of eight moved, at p = 1.000.

So the experiment I designed to give the demo lever a fair test could not
give it one. Correcting my authoring error changed the reason for the verdict
and not the verdict, because the rule cannot tell a lever that helps from a
lever that does nothing.

Five rounds, $2.17, zero decisions distinguishable from chance.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG

.kiro/specs/learning-harness/tasks.md

@@ -247,7 +247,7 @@
   - **28 rows resolved a span on the vulnerable side and none on the fixed side** (12% of the corpus), so every fixed-side finding read as outside the labeled function and the pair scored as silence whatever the detector said — the same flattering direction as the range defect, one-seventh the size. `fix_function` reached only the harvester; it now reaches the catalog and `PairCase`, `_spans` uses it for the fixed side, and the catalog generator derives it from the fixed excerpt itself for the 25 vibe-py trap rows that never declared one. A row that still resolves no fixed-side span reports `unresolved_label:fixed` instead of scoring silent. Measured after: **0 rows**.
   - Suggestions applied: one spelling per corpus fact everywhere (`PairCase.unscorable` verbatim, no invented prefix); leave-one-out records non-teachers withheld for tier or an unscorable reason, not only holdout refusals; the hygiene test asserts the declared twins still hash equal, so a declaration that stopped being true would not be invisible; `is_post_cutoff` refuses a cutoff coarser than a day, which was the one over-claim path that existed, and month precision is pinned; `UNSCORABLE_REASONS` covers the whole vocabulary rather than `identical_twin` alone; the zero-run score carries its date; `_pairs_named_by` filters by substring before the regex.
 - Carry-forward, measured 2026-09-06 after group 4 (`benchmarks/measurements/2026-09-06-trajectory-divergence.json`): **the K-run numbers this spec publishes are ensemble votes, not repeated measurements.** Across 120 runs of the injection family at temperature 0 with thinking disabled there was not one repeated tool-call trajectory, and outcome disagreement tracks trajectory divergence (30.9% at low similarity, 4.2% at mid). Consequences for anyone reading these numbers: Req 3.5's K is an ensemble over distinct search procedures; Req 8.2's noise floor measures how often an unconstrained search reaches the sink rather than the detector's uncertainty, so Req 9.5's regression budget grows with the search's own wandering; and the instability is entirely one-sided — every pair produced a single finding set on the fixed side, so specificity is solid and recall carries the caveat. `.kiro/specs/detector-optimizer/brief.md` proposes the repair, and it is a program change rather than a prompt change.
-- Task 5.3 (2026-09-07, four rounds on `injection`, $1.76, `benchmarks/measurements/2026-09-07-evolve-rounds-injection.json`). The task was amended to target injection because access_control has one train teacher.
+- Task 5.3 (2026-09-07, five rounds on `injection`, $2.17, `benchmarks/measurements/2026-09-07-evolve-rounds-injection.json`). The task was amended to target injection because access_control has one train teacher.
 
   | round | lever | outcome | train | p | holdout | p | distinguishable from noise |
   |---|---|---|---|---|---|---|---|
@@ -255,14 +255,15 @@
   | 2 | checklist | rejected `train_regressed` | 0+/1− | 1.000 | 1+/1− | 1.000 | no |
   | 3 | prompt | **accepted `improved` → v1** | 2+/0− | 0.500 | 2+/1− | 1.000 | no |
   | 4 | meta-agent | rejected `no_proposal` | — | — | — | — | never ran |
+  | 5 | counterexample | rejected `train_regressed` | 0+/1− | 1.000 | 1+/3− | 0.625 | no |
 
-  **Nothing the loop decided carried evidence, in either direction.** Round 2 rejected a change because **one pair of eight** moved, at p = 1.000 — the maximum the sign test can return. Round 3 accepted one because two of eight improved and none regressed, at p = 0.500, and that acceptance is now versioned as v1 and will be built on. On a family whose measured floor says **11 of 20 pairs disagree with themselves between runs of the unchanged detector**, both readings are what changing nothing looks like.
+  **Nothing the loop decided carried evidence, in either direction — five rounds, zero reliable decisions.** Round 2 rejected a change because **one pair of eight** moved, at p = 1.000 — the maximum the sign test can return. Round 3 accepted one because two of eight improved and none regressed, at p = 0.500, and that acceptance is now versioned as v1 and will be built on. On a family whose measured floor says **11 of 20 pairs disagree with themselves between runs of the unchanged detector**, both readings are what changing nothing looks like.
 
   I overstated this once and correct it here: after rounds 1 and 2 I wrote that the rule was "close to structurally incapable of accepting anything". Round 3 accepted. The accurate statement is symmetric and worse — the rule accepts on noise as readily as it rejects on it.
 
   - **Escalation to the maintainer, not applied here.** Req 9.5's threshold rule (reject if either side regresses, accept if one improves and neither regresses, tie rejects) should be replaced by an acceptance that survives the measured variance. `Archive` already holds the per-pair Pareto frontier and `acceptance.reliability` already computes the test; neither decides anything. The honest caveat is that at a minibatch of 8 and a holdout of 10 a reliability-gated rule would accept almost nothing either: **the corpus is the binding constraint, not the rule**, and no acceptance rule fixes a train split of twelve.
   - **The v1 configuration rests on p = 0.500** and should probably be reverted before any further round builds on it. Left in place, and flagged, because reverting an accepted round is the maintainer's call.
-  - Round 1's rejection was correct but for my error, not the loop's insight: the `memory` lever writes `hard_negatives.jsonl`, which the detector renders as *"Known non-issues, do not report these"*, and I fed it the four vulnerability shapes the train split teaches. Round 5 re-runs the same content through `counterexamples` (*"Worked examples"*), which is the lever I meant.
+  - Round 1's rejection was correct but for my error, not the loop's insight: the `memory` lever writes `hard_negatives.jsonl`, which the detector renders as *"Known non-issues, do not report these"*, and I fed it the four vulnerability shapes the train split teaches. Round 5 re-ran the same content through `counterexamples` (*"Worked examples"*), which is the lever I meant — and was rejected because **one pair of eight** moved, at p = 1.000. **The experiment I designed to give the demo lever a fair test could not give it one.** The rule cannot distinguish a lever that helps from a lever that does nothing, so correcting my authoring error changed the reason for the verdict and not the verdict.
   - What did work: the cross-family sweep caught a genuine negative flip in `access_control` in round 2 — a real regression found by the before/after comparison that replaced the absolute-failure count, which under the old rule would have rejected rounds for families that were already failing.
 
 - **The finding that outranks the rounds (maintainer, 2026-09-07: "the error is not the model, the error is the elsewhere").** Eighteen defects were found in this session's measurement and round work. Every one was in this repository; none was the model's. The list is kept because it should change how the next bad number is triaged: the prose parser that discarded findings the model had already reached; the CLI that never read the cost meter it published; the baseline that lost everything on an exception, and again on a timeout; two keys that collapsed a slice into another, in two different modules; the pattern ruleset at a 12.1% ceiling; a harvester that truncated an excerpt at both ends; a family shape that could not see a permissive default inside a call argument; a lever wired to the file that says *do not report these*; a provider seam hard-coded to a vendor with no key; a proposer reason the journal threw away; a retry that did not cover a body which stopped mid-read; an exception path wrapped around half a round; log-probabilities read from the wrong object; `.mjs` absent from the language table; and an access-level check that matched prose inside a decorator. Three of them were the same shape — fixing an instance and missing the class. **The default assumption for an unexplained number is that we broke it.** `deepseek-v4-pro` scoring 46.7% against flash's 83.3% is unexplained and is therefore, by this prior, ours until shown otherwise.

benchmarks/measurements/2026-09-07-evolve-rounds-injection.json

@@ -10,16 +10,17 @@
     "caveats": [
       "Round 1's rejection was correct, but for an authoring error rather than the loop's insight: the memory lever writes hard_negatives.jsonl, rendered to the detector as 'Known non-issues, do not report these', and it was fed the four vulnerability shapes the train split teaches.",
       "Round 4 never proposed anything: the meta-agent composed correctly and failed authentication, because the provider seam was hard-coded to anthropic while the detector talks to DeepSeek.",
-      "The minibatch is 8 and the holdout 10. A sign test cannot reach p<0.05 below n=6 discordant pairs, so at this corpus size a reliability-gated rule would accept almost nothing either. The corpus is the binding constraint, not the rule."
+      "The minibatch is 8 and the holdout 10. A sign test cannot reach p<0.05 below n=6 discordant pairs, so at this corpus size a reliability-gated rule would accept almost nothing either. The corpus is the binding constraint, not the rule.",
+      "Round 5 re-ran round 1's content through the lever it should have used -- counterexamples, rendered as 'Worked examples' -- and was rejected because one pair of eight moved, at p=1.000. The experiment designed to give the demo lever a fair test could not give it one: the rule cannot tell a lever that helps from a lever that does nothing."
     ],
     "conclusion": "No decision was distinguishable from chance, in either direction. Two rejections at p=0.250 and p=1.000, one acceptance at p=0.500 that versioned the configuration to v1. On a family where 11 of 20 pairs disagree with themselves between runs of the unchanged detector, the threshold rule in Req 9.5 -- reject if any pair regresses, accept if one improves and none regress -- is reading noise in both directions and reporting it as progress.",
-    "decisions": 4,
+    "decisions": 5,
     "distinguishable_from_noise": 0,
     "injection_noise_floor_at_round_zero": {
       "flaky_pairs": 11,
       "scorable": 20
     },
-    "total_cost_usd": 1.76
+    "total_cost_usd": 2.17
   },
   "rounds": [
     {
@@ -151,6 +152,40 @@
         "reliable": false,
         "worse": 0
       }
+    },
+    {
+      "attribution": {
+        "flipped_predicted": 1,
+        "flipped_unpredicted": 0,
+        "precision": 1.0
+      },
+      "config_version_after": "1",
+      "cost_usd": 0.41933022399999986,
+      "holdout": {
+        "better": 1,
+        "delta": -0.25,
+        "p_value": 0.625,
+        "reliable": false,
+        "worse": 3
+      },
+      "lever": "counterexample",
+      "outcome": "rejected",
+      "reason": "train_regressed",
+      "round": 5,
+      "sweep": {
+        "access_control": 0,
+        "config_secrets": 0,
+        "deserialization": 0,
+        "output_encoding": 0,
+        "path": 0
+      },
+      "train": {
+        "better": 0,
+        "delta": -0.125,
+        "p_value": 1.0,
+        "reliable": false,
+        "worse": 1
+      }
     }
   ],
   "slice": "vibe-py"