diff --git a/.kiro/specs/learning-harness/tasks.md b/.kiro/specs/learning-harness/tasks.md index 23dbfa3..6dc3a37 100644 --- a/.kiro/specs/learning-harness/tasks.md +++ b/.kiro/specs/learning-harness/tasks.md @@ -247,7 +247,7 @@ - **28 rows resolved a span on the vulnerable side and none on the fixed side** (12% of the corpus), so every fixed-side finding read as outside the labeled function and the pair scored as silence whatever the detector said — the same flattering direction as the range defect, one-seventh the size. `fix_function` reached only the harvester; it now reaches the catalog and `PairCase`, `_spans` uses it for the fixed side, and the catalog generator derives it from the fixed excerpt itself for the 25 vibe-py trap rows that never declared one. A row that still resolves no fixed-side span reports `unresolved_label:fixed` instead of scoring silent. Measured after: **0 rows**. - Suggestions applied: one spelling per corpus fact everywhere (`PairCase.unscorable` verbatim, no invented prefix); leave-one-out records non-teachers withheld for tier or an unscorable reason, not only holdout refusals; the hygiene test asserts the declared twins still hash equal, so a declaration that stopped being true would not be invisible; `is_post_cutoff` refuses a cutoff coarser than a day, which was the one over-claim path that existed, and month precision is pinned; `UNSCORABLE_REASONS` covers the whole vocabulary rather than `identical_twin` alone; the zero-run score carries its date; `_pairs_named_by` filters by substring before the regex. - Carry-forward, measured 2026-09-06 after group 4 (`benchmarks/measurements/2026-09-06-trajectory-divergence.json`): **the K-run numbers this spec publishes are ensemble votes, not repeated measurements.** Across 120 runs of the injection family at temperature 0 with thinking disabled there was not one repeated tool-call trajectory, and outcome disagreement tracks trajectory divergence (30.9% at low similarity, 4.2% at mid). Consequences for anyone reading these numbers: Req 3.5's K is an ensemble over distinct search procedures; Req 8.2's noise floor measures how often an unconstrained search reaches the sink rather than the detector's uncertainty, so Req 9.5's regression budget grows with the search's own wandering; and the instability is entirely one-sided — every pair produced a single finding set on the fixed side, so specificity is solid and recall carries the caveat. `.kiro/specs/detector-optimizer/brief.md` proposes the repair, and it is a program change rather than a prompt change. -- Task 5.3 (2026-09-07, four rounds on `injection`, $1.76, `benchmarks/measurements/2026-09-07-evolve-rounds-injection.json`). The task was amended to target injection because access_control has one train teacher. +- Task 5.3 (2026-09-07, five rounds on `injection`, $2.17, `benchmarks/measurements/2026-09-07-evolve-rounds-injection.json`). The task was amended to target injection because access_control has one train teacher. | round | lever | outcome | train | p | holdout | p | distinguishable from noise | |---|---|---|---|---|---|---|---| @@ -255,14 +255,15 @@ | 2 | checklist | rejected `train_regressed` | 0+/1− | 1.000 | 1+/1− | 1.000 | no | | 3 | prompt | **accepted `improved` → v1** | 2+/0− | 0.500 | 2+/1− | 1.000 | no | | 4 | meta-agent | rejected `no_proposal` | — | — | — | — | never ran | + | 5 | counterexample | rejected `train_regressed` | 0+/1− | 1.000 | 1+/3− | 0.625 | no | - **Nothing the loop decided carried evidence, in either direction.** Round 2 rejected a change because **one pair of eight** moved, at p = 1.000 — the maximum the sign test can return. Round 3 accepted one because two of eight improved and none regressed, at p = 0.500, and that acceptance is now versioned as v1 and will be built on. On a family whose measured floor says **11 of 20 pairs disagree with themselves between runs of the unchanged detector**, both readings are what changing nothing looks like. + **Nothing the loop decided carried evidence, in either direction — five rounds, zero reliable decisions.** Round 2 rejected a change because **one pair of eight** moved, at p = 1.000 — the maximum the sign test can return. Round 3 accepted one because two of eight improved and none regressed, at p = 0.500, and that acceptance is now versioned as v1 and will be built on. On a family whose measured floor says **11 of 20 pairs disagree with themselves between runs of the unchanged detector**, both readings are what changing nothing looks like. I overstated this once and correct it here: after rounds 1 and 2 I wrote that the rule was "close to structurally incapable of accepting anything". Round 3 accepted. The accurate statement is symmetric and worse — the rule accepts on noise as readily as it rejects on it. - **Escalation to the maintainer, not applied here.** Req 9.5's threshold rule (reject if either side regresses, accept if one improves and neither regresses, tie rejects) should be replaced by an acceptance that survives the measured variance. `Archive` already holds the per-pair Pareto frontier and `acceptance.reliability` already computes the test; neither decides anything. The honest caveat is that at a minibatch of 8 and a holdout of 10 a reliability-gated rule would accept almost nothing either: **the corpus is the binding constraint, not the rule**, and no acceptance rule fixes a train split of twelve. - **The v1 configuration rests on p = 0.500** and should probably be reverted before any further round builds on it. Left in place, and flagged, because reverting an accepted round is the maintainer's call. - - Round 1's rejection was correct but for my error, not the loop's insight: the `memory` lever writes `hard_negatives.jsonl`, which the detector renders as *"Known non-issues, do not report these"*, and I fed it the four vulnerability shapes the train split teaches. Round 5 re-runs the same content through `counterexamples` (*"Worked examples"*), which is the lever I meant. + - Round 1's rejection was correct but for my error, not the loop's insight: the `memory` lever writes `hard_negatives.jsonl`, which the detector renders as *"Known non-issues, do not report these"*, and I fed it the four vulnerability shapes the train split teaches. Round 5 re-ran the same content through `counterexamples` (*"Worked examples"*), which is the lever I meant — and was rejected because **one pair of eight** moved, at p = 1.000. **The experiment I designed to give the demo lever a fair test could not give it one.** The rule cannot distinguish a lever that helps from a lever that does nothing, so correcting my authoring error changed the reason for the verdict and not the verdict. - What did work: the cross-family sweep caught a genuine negative flip in `access_control` in round 2 — a real regression found by the before/after comparison that replaced the absolute-failure count, which under the old rule would have rejected rounds for families that were already failing. - **The finding that outranks the rounds (maintainer, 2026-09-07: "the error is not the model, the error is the elsewhere").** Eighteen defects were found in this session's measurement and round work. Every one was in this repository; none was the model's. The list is kept because it should change how the next bad number is triaged: the prose parser that discarded findings the model had already reached; the CLI that never read the cost meter it published; the baseline that lost everything on an exception, and again on a timeout; two keys that collapsed a slice into another, in two different modules; the pattern ruleset at a 12.1% ceiling; a harvester that truncated an excerpt at both ends; a family shape that could not see a permissive default inside a call argument; a lever wired to the file that says *do not report these*; a provider seam hard-coded to a vendor with no key; a proposer reason the journal threw away; a retry that did not cover a body which stopped mid-read; an exception path wrapped around half a round; log-probabilities read from the wrong object; `.mjs` absent from the language table; and an access-level check that matched prose inside a decorator. Three of them were the same shape — fixing an instance and missing the class. **The default assumption for an unexplained number is that we broke it.** `deepseek-v4-pro` scoring 46.7% against flash's 83.3% is unexplained and is therefore, by this prior, ours until shown otherwise. diff --git a/benchmarks/measurements/2026-09-07-evolve-rounds-injection.json b/benchmarks/measurements/2026-09-07-evolve-rounds-injection.json index f54df8d..ef1a1ac 100644 --- a/benchmarks/measurements/2026-09-07-evolve-rounds-injection.json +++ b/benchmarks/measurements/2026-09-07-evolve-rounds-injection.json @@ -10,16 +10,17 @@ "caveats": [ "Round 1's rejection was correct, but for an authoring error rather than the loop's insight: the memory lever writes hard_negatives.jsonl, rendered to the detector as 'Known non-issues, do not report these', and it was fed the four vulnerability shapes the train split teaches.", "Round 4 never proposed anything: the meta-agent composed correctly and failed authentication, because the provider seam was hard-coded to anthropic while the detector talks to DeepSeek.", - "The minibatch is 8 and the holdout 10. A sign test cannot reach p<0.05 below n=6 discordant pairs, so at this corpus size a reliability-gated rule would accept almost nothing either. The corpus is the binding constraint, not the rule." + "The minibatch is 8 and the holdout 10. A sign test cannot reach p<0.05 below n=6 discordant pairs, so at this corpus size a reliability-gated rule would accept almost nothing either. The corpus is the binding constraint, not the rule.", + "Round 5 re-ran round 1's content through the lever it should have used -- counterexamples, rendered as 'Worked examples' -- and was rejected because one pair of eight moved, at p=1.000. The experiment designed to give the demo lever a fair test could not give it one: the rule cannot tell a lever that helps from a lever that does nothing." ], "conclusion": "No decision was distinguishable from chance, in either direction. Two rejections at p=0.250 and p=1.000, one acceptance at p=0.500 that versioned the configuration to v1. On a family where 11 of 20 pairs disagree with themselves between runs of the unchanged detector, the threshold rule in Req 9.5 -- reject if any pair regresses, accept if one improves and none regress -- is reading noise in both directions and reporting it as progress.", - "decisions": 4, + "decisions": 5, "distinguishable_from_noise": 0, "injection_noise_floor_at_round_zero": { "flaky_pairs": 11, "scorable": 20 }, - "total_cost_usd": 1.76 + "total_cost_usd": 2.17 }, "rounds": [ { @@ -151,6 +152,40 @@ "reliable": false, "worse": 0 } + }, + { + "attribution": { + "flipped_predicted": 1, + "flipped_unpredicted": 0, + "precision": 1.0 + }, + "config_version_after": "1", + "cost_usd": 0.41933022399999986, + "holdout": { + "better": 1, + "delta": -0.25, + "p_value": 0.625, + "reliable": false, + "worse": 3 + }, + "lever": "counterexample", + "outcome": "rejected", + "reason": "train_regressed", + "round": 5, + "sweep": { + "access_control": 0, + "config_secrets": 0, + "deserialization": 0, + "output_encoding": 0, + "path": 0 + }, + "train": { + "better": 0, + "delta": -0.125, + "p_value": 1.0, + "reliable": false, + "worse": 1 + } } ], "slice": "vibe-py"