From 78924316be1c02c53b8ec4a671b7b0849095097b Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Mon, 7 Sep 2026 07:39:48 +0200 Subject: [PATCH] docs(learning-harness): the reading of the first evolve rounds Four rounds on injection, $1.76. Not one decision carried evidence, in either direction. Round 2 rejected a change because one pair of eight moved, at p = 1.000 -- the maximum the sign test can return. Round 3 accepted one at p = 0.500 and that acceptance is now versioned as v1. On a family where 11 of 20 pairs disagree with themselves between runs of the unchanged detector, both readings are what changing nothing looks like. I corrected an overstatement in the notes: after two rejections I wrote that the rule was near-incapable of accepting. Round 3 accepted. The accurate statement is symmetric and worse -- it accepts on noise as readily as it rejects on it. Escalated, not applied: Req 9.5's threshold rule should be replaced by an acceptance that survives the measured variance. The Pareto archive and the sign test both exist and neither decides anything. The honest caveat is that at a minibatch of 8 a reliability-gated rule would accept almost nothing either -- the corpus is the binding constraint, not the rule. And the finding that outranks the rounds, in the maintainer's words: the error is not the model, the error is the elsewhere. Eighteen defects this session, every one in this repository, three of them the same shape of fixing an instance and missing the class. The default assumption for an unexplained number is now that we broke it -- which puts pro's 46.7% against flash's 83.3% on our side of the ledger until shown otherwise. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG --- .kiro/specs/learning-harness/tasks.md | 19 +++ .../2026-09-07-evolve-rounds-injection.json | 157 ++++++++++++++++++ 2 files changed, 176 insertions(+) create mode 100644 benchmarks/measurements/2026-09-07-evolve-rounds-injection.json diff --git a/.kiro/specs/learning-harness/tasks.md b/.kiro/specs/learning-harness/tasks.md index 9c421bc..23dbfa3 100644 --- a/.kiro/specs/learning-harness/tasks.md +++ b/.kiro/specs/learning-harness/tasks.md @@ -247,3 +247,22 @@ - **28 rows resolved a span on the vulnerable side and none on the fixed side** (12% of the corpus), so every fixed-side finding read as outside the labeled function and the pair scored as silence whatever the detector said — the same flattering direction as the range defect, one-seventh the size. `fix_function` reached only the harvester; it now reaches the catalog and `PairCase`, `_spans` uses it for the fixed side, and the catalog generator derives it from the fixed excerpt itself for the 25 vibe-py trap rows that never declared one. A row that still resolves no fixed-side span reports `unresolved_label:fixed` instead of scoring silent. Measured after: **0 rows**. - Suggestions applied: one spelling per corpus fact everywhere (`PairCase.unscorable` verbatim, no invented prefix); leave-one-out records non-teachers withheld for tier or an unscorable reason, not only holdout refusals; the hygiene test asserts the declared twins still hash equal, so a declaration that stopped being true would not be invisible; `is_post_cutoff` refuses a cutoff coarser than a day, which was the one over-claim path that existed, and month precision is pinned; `UNSCORABLE_REASONS` covers the whole vocabulary rather than `identical_twin` alone; the zero-run score carries its date; `_pairs_named_by` filters by substring before the regex. - Carry-forward, measured 2026-09-06 after group 4 (`benchmarks/measurements/2026-09-06-trajectory-divergence.json`): **the K-run numbers this spec publishes are ensemble votes, not repeated measurements.** Across 120 runs of the injection family at temperature 0 with thinking disabled there was not one repeated tool-call trajectory, and outcome disagreement tracks trajectory divergence (30.9% at low similarity, 4.2% at mid). Consequences for anyone reading these numbers: Req 3.5's K is an ensemble over distinct search procedures; Req 8.2's noise floor measures how often an unconstrained search reaches the sink rather than the detector's uncertainty, so Req 9.5's regression budget grows with the search's own wandering; and the instability is entirely one-sided — every pair produced a single finding set on the fixed side, so specificity is solid and recall carries the caveat. `.kiro/specs/detector-optimizer/brief.md` proposes the repair, and it is a program change rather than a prompt change. +- Task 5.3 (2026-09-07, four rounds on `injection`, $1.76, `benchmarks/measurements/2026-09-07-evolve-rounds-injection.json`). The task was amended to target injection because access_control has one train teacher. + + | round | lever | outcome | train | p | holdout | p | distinguishable from noise | + |---|---|---|---|---|---|---|---| + | 1 | memory | rejected `train_regressed` | 0+/3− | 0.250 | 1+/4− | 0.375 | no | + | 2 | checklist | rejected `train_regressed` | 0+/1− | 1.000 | 1+/1− | 1.000 | no | + | 3 | prompt | **accepted `improved` → v1** | 2+/0− | 0.500 | 2+/1− | 1.000 | no | + | 4 | meta-agent | rejected `no_proposal` | — | — | — | — | never ran | + + **Nothing the loop decided carried evidence, in either direction.** Round 2 rejected a change because **one pair of eight** moved, at p = 1.000 — the maximum the sign test can return. Round 3 accepted one because two of eight improved and none regressed, at p = 0.500, and that acceptance is now versioned as v1 and will be built on. On a family whose measured floor says **11 of 20 pairs disagree with themselves between runs of the unchanged detector**, both readings are what changing nothing looks like. + + I overstated this once and correct it here: after rounds 1 and 2 I wrote that the rule was "close to structurally incapable of accepting anything". Round 3 accepted. The accurate statement is symmetric and worse — the rule accepts on noise as readily as it rejects on it. + + - **Escalation to the maintainer, not applied here.** Req 9.5's threshold rule (reject if either side regresses, accept if one improves and neither regresses, tie rejects) should be replaced by an acceptance that survives the measured variance. `Archive` already holds the per-pair Pareto frontier and `acceptance.reliability` already computes the test; neither decides anything. The honest caveat is that at a minibatch of 8 and a holdout of 10 a reliability-gated rule would accept almost nothing either: **the corpus is the binding constraint, not the rule**, and no acceptance rule fixes a train split of twelve. + - **The v1 configuration rests on p = 0.500** and should probably be reverted before any further round builds on it. Left in place, and flagged, because reverting an accepted round is the maintainer's call. + - Round 1's rejection was correct but for my error, not the loop's insight: the `memory` lever writes `hard_negatives.jsonl`, which the detector renders as *"Known non-issues, do not report these"*, and I fed it the four vulnerability shapes the train split teaches. Round 5 re-runs the same content through `counterexamples` (*"Worked examples"*), which is the lever I meant. + - What did work: the cross-family sweep caught a genuine negative flip in `access_control` in round 2 — a real regression found by the before/after comparison that replaced the absolute-failure count, which under the old rule would have rejected rounds for families that were already failing. + +- **The finding that outranks the rounds (maintainer, 2026-09-07: "the error is not the model, the error is the elsewhere").** Eighteen defects were found in this session's measurement and round work. Every one was in this repository; none was the model's. The list is kept because it should change how the next bad number is triaged: the prose parser that discarded findings the model had already reached; the CLI that never read the cost meter it published; the baseline that lost everything on an exception, and again on a timeout; two keys that collapsed a slice into another, in two different modules; the pattern ruleset at a 12.1% ceiling; a harvester that truncated an excerpt at both ends; a family shape that could not see a permissive default inside a call argument; a lever wired to the file that says *do not report these*; a provider seam hard-coded to a vendor with no key; a proposer reason the journal threw away; a retry that did not cover a body which stopped mid-read; an exception path wrapped around half a round; log-probabilities read from the wrong object; `.mjs` absent from the language table; and an access-level check that matched prose inside a decorator. Three of them were the same shape — fixing an instance and missing the class. **The default assumption for an unexplained number is that we broke it.** `deepseek-v4-pro` scoring 46.7% against flash's 83.3% is unexplained and is therefore, by this prior, ours until shown otherwise. diff --git a/benchmarks/measurements/2026-09-07-evolve-rounds-injection.json b/benchmarks/measurements/2026-09-07-evolve-rounds-injection.json new file mode 100644 index 0000000..f54df8d --- /dev/null +++ b/benchmarks/measurements/2026-09-07-evolve-rounds-injection.json @@ -0,0 +1,157 @@ +{ + "command": "uv run ousast learning round --family injection --slice vibe-py --k-runs 3 --model deepseek-v4-flash", + "family": "injection", + "generated": "2026-09-07", + "k_runs": 3, + "minibatch": 8, + "model": "deepseek-v4-flash", + "note": "learning-harness task 5.3, amended to target injection because access_control has one train teacher. Each round records the two-sided sign test over the pairs that actually moved, beside the decision the threshold rule made. The question these rounds were run to answer is whether any of those decisions carried evidence.", + "reading": { + "caveats": [ + "Round 1's rejection was correct, but for an authoring error rather than the loop's insight: the memory lever writes hard_negatives.jsonl, rendered to the detector as 'Known non-issues, do not report these', and it was fed the four vulnerability shapes the train split teaches.", + "Round 4 never proposed anything: the meta-agent composed correctly and failed authentication, because the provider seam was hard-coded to anthropic while the detector talks to DeepSeek.", + "The minibatch is 8 and the holdout 10. A sign test cannot reach p<0.05 below n=6 discordant pairs, so at this corpus size a reliability-gated rule would accept almost nothing either. The corpus is the binding constraint, not the rule." + ], + "conclusion": "No decision was distinguishable from chance, in either direction. Two rejections at p=0.250 and p=1.000, one acceptance at p=0.500 that versioned the configuration to v1. On a family where 11 of 20 pairs disagree with themselves between runs of the unchanged detector, the threshold rule in Req 9.5 -- reject if any pair regresses, accept if one improves and none regress -- is reading noise in both directions and reporting it as progress.", + "decisions": 4, + "distinguishable_from_noise": 0, + "injection_noise_floor_at_round_zero": { + "flaky_pairs": 11, + "scorable": 20 + }, + "total_cost_usd": 1.76 + }, + "rounds": [ + { + "attribution": { + "flipped_predicted": 1, + "flipped_unpredicted": 0, + "precision": 1.0 + }, + "config_version_after": "0", + "cost_usd": 0.5971665280000001, + "holdout": { + "better": 1, + "delta": -0.375, + "p_value": 0.375, + "reliable": false, + "worse": 4 + }, + "lever": "memory", + "outcome": "rejected", + "reason": "train_regressed", + "round": 1, + "sweep": { + "access_control": 0, + "config_secrets": 0, + "deserialization": 0, + "output_encoding": 0, + "path": 0 + }, + "train": { + "better": 0, + "delta": -0.375, + "p_value": 0.25, + "reliable": false, + "worse": 3 + } + }, + { + "attribution": { + "flipped_predicted": 1, + "flipped_unpredicted": 1, + "precision": 0.5 + }, + "config_version_after": "0", + "cost_usd": 0.5615603119999997, + "holdout": { + "better": 1, + "delta": 0.0, + "p_value": 1.0, + "reliable": false, + "worse": 1 + }, + "lever": "checklist", + "outcome": "rejected", + "reason": "train_regressed", + "round": 2, + "sweep": { + "access_control": 1, + "config_secrets": 0, + "deserialization": 0, + "output_encoding": 0, + "path": 0 + }, + "train": { + "better": 0, + "delta": -0.125, + "p_value": 1.0, + "reliable": false, + "worse": 1 + } + }, + { + "attribution": { + "flipped_predicted": 4, + "flipped_unpredicted": 0, + "precision": 1.0 + }, + "config_version_after": "1", + "cost_usd": 0.47976149600000073, + "holdout": { + "better": 2, + "delta": 0.125, + "p_value": 1.0, + "reliable": false, + "worse": 1 + }, + "lever": "prompt", + "outcome": "accepted", + "reason": "improved", + "round": 3, + "sweep": { + "access_control": 0, + "config_secrets": 0, + "deserialization": 0, + "output_encoding": 0, + "path": 0 + }, + "train": { + "better": 2, + "delta": 0.25, + "p_value": 0.5, + "reliable": false, + "worse": 0 + } + }, + { + "attribution": { + "flipped_predicted": 0, + "flipped_unpredicted": 0, + "precision": 0.0 + }, + "config_version_after": "1", + "cost_usd": 0.11708116000000007, + "holdout": { + "better": 0, + "delta": 0.0, + "p_value": 1.0, + "reliable": false, + "worse": 0 + }, + "lever": null, + "outcome": "rejected", + "reason": "no_proposal", + "round": 4, + "sweep": {}, + "train": { + "better": 0, + "delta": 0.0, + "p_value": 1.0, + "reliable": false, + "worse": 0 + } + } + ], + "slice": "vibe-py" +}