docs: reflect the trajectory measurement back onto the approved spec

learning-harness divides cleanly. The measurement layer stands and is worth
more than before -- it is what made a $0.23 experiment able to answer 'is
our own instability structural?'. What was answering the wrong question:
Req 3.5's K (an ensemble of distinct search procedures, not a repeated
measurement), Req 8.2's noise floor (it measures how often an unconstrained
search reaches the sink, so the regression budget grows with the search's own
wandering), Req 9.5's acceptance rule, and the sign test added this session.

And the highest-leverage change in either spec turns out not to be a prompt,
a lever or an optimizer: stop asking the model to locate the sink. 555 grep
calls against 71 entry_points, while the CST and the entry-point mapper
already know where the sinks are.

Limits stated in the brief: twelve pairs, one family, one model; the
high-similarity bucket is two run pairs and its 0% is decoration.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-06 15:50:44 +02:00
co-authored by Claude Opus 5
parent 5e2070bc17
commit 4e6afa862b
2 changed files with 49 additions and 0 deletions
+48
View File
@@ -211,6 +211,54 @@ procedures — and a regression budget derived from it is a budget for the searc
the budget and the sign test are all statistical treatment for a structural problem, and the 0% and 4.2%
rows of that table say what happens when the structure is fixed instead.
## Reflection on the approved specification
Read against that measurement, `learning-harness` divides cleanly into a part that stands and a part that
was answering the wrong question.
**Stands, and is worth more now than before.** The closed taxonomy; class-aware pair scoring; the split
enforced by construction; the unscorable vocabulary and its denominators; verifier-gated reporting; the
corpus repairs; the publication machinery; the per-pair archive. None of it depends on the noise framing,
and all of it is what made this experiment possible to run and cost $0.23. A measurement layer that can
answer "is our own instability structural?" in twenty minutes is the thing that was missing before.
**Was answering the wrong question.**
- **Req 3.5, K ≥ 3 runs per pair.** The premise is variance reduction over a repeated measurement. There is
no repeated measurement: 120 runs, 120 trajectories. K runs is an *ensemble of distinct search
procedures*, and a majority vote over it is an unweighted ensemble, not an estimate. It should either be
renamed and published as an ensemble — which changes what every family number means — or dropped once the
search is constrained.
- **Req 8.2, the noise floor as the regression budget.** The floor measures how often an unconstrained
search happens to reach the sink. Deriving a regression budget from it means the worse the search wanders,
the more collateral damage a round is licensed to do. That is backwards, and it is why injection's budget
is 11 of 20.
- **Req 9.5, the acceptance rule.** Already flagged for reading the holdout every round. Now also: it
compares deltas between two ensemble votes, and calls the difference an improvement.
- **The sign test and `reliable_change`.** Added this session in good faith; they test whether an ensemble's
vote moved, which is not the question either.
**What the data adds that neither spec anticipated.**
- **The instability is entirely one-sided.** All twelve pairs produced exactly one finding set on the fixed
side. Specificity is solid and recall is the noisy half, so the leak ceiling and the false-positive
machinery are sound and it is the recall numbers that carry the caveat.
- **`grep_repo` is the variance engine** — 555 calls against 181 `read_file` and 71 `entry_points`. The
single highest-leverage change in either spec is not a prompt, a lever or an optimiser: **stop asking the
model to locate the sink.** The CST, the entry-point mapper, `flows` and `obligations` already know. Give
the detector the candidate sites and ask it the question only a model can answer — does untrusted input
reach this one, and is there a guard.
- **The loop is partly self-correcting**, so this should be a large win rather than a marginal one: 78
distinct trajectories collapse to one to three finding sets per pair, and half the pairs are already
stable despite never repeating an investigation.
**Limits of the experiment, stated so the conclusion is not over-read.** Twelve pairs, one family, one
slice, one model. The high-similarity bucket holds only two run pairs, so its 0% is decoration; the
defensible number is the mid bucket's 4.2% over 24 run pairs against 30.9% over 94. And the direction of
causation is inferred, not established — similar trajectories may be a symptom of easy pairs rather than a
cause of agreement. The way to settle that is the constrained detector itself: fix the investigation, and
if the floor does not collapse, this reading was wrong.
## The constraint that outranks the algorithm
Both algorithms assume a dataset. GEPA splits `D_train` into `D_feedback` and `D_pareto`; SIMBA's default