learning-harness divides cleanly. The measurement layer stands and is worth
more than before -- it is what made a $0.23 experiment able to answer 'is
our own instability structural?'. What was answering the wrong question:
Req 3.5's K (an ensemble of distinct search procedures, not a repeated
measurement), Req 8.2's noise floor (it measures how often an unconstrained
search reaches the sink, so the regression budget grows with the search's own
wandering), Req 9.5's acceptance rule, and the sign test added this session.
And the highest-leverage change in either spec turns out not to be a prompt,
a lever or an optimizer: stop asking the model to locate the sink. 555 grep
calls against 71 entry_points, while the CST and the entry-point mapper
already know where the sinks are.
Limits stated in the brief: twelve pairs, one family, one model; the
high-similarity bucket is two run pairs and its 0% is decoration.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Twelve injection pairs, both sides, K=5, temperature 0, thinking disabled,
recording every tool call. $0.23.
The statistic I designed for could not be computed: across 120 runs there was
not one repeated trajectory -- 4.67 and 4.92 distinct trajectories per side
out of 5, and 78 distinct tool-name sequences ignoring arguments entirely.
The detector is not sampling one measurement five times; it is running five
different investigations and voting on the results.
As a gradient the relationship is monotone: runs whose trajectories were
similar (>=0.75 Jaccard) disagreed 0% of the time, mid-similarity 4.2%,
low-similarity 30.9%.
Three more facts. The search is grep: 555 grep_repo calls against 181
read_file, 71 entry_points, 43 find_refs, 13 read_definition -- the model is
guessing patterns to locate sinks the CST and the entry-point mapper already
know. All the instability is on the vulnerable side; every pair produced
exactly one finding set on the fixed side, so the noise is in detection and
none of it in false positives. And findings converge much more than
trajectories do: 78 trajectories, one to three finding sets per pair.
The floor is therefore not measuring uncertainty about a question. It is
measuring how often an unconstrained search happens to reach the sink.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The harness treats the detector as a black box emitting a noisy scalar, and
K runs, majority voting, the measured floor, the regression budget and the
sign test all follow from that. It is the stance you take when you cannot
see inside what you are measuring, and we can.
Temperature is already 0 and thinking is disabled, and 11 of 20 injection
pairs still disagree with themselves. The freedom is in the trajectory --
which files get read, which greps get run -- and we built statistics to
average over it instead of removing it.
Three DSPy constructs attack the source: constraints enforced as predicates
at generation time (we already compute every one of them, after the fact, at
scoring time), typed decomposition of the single ReAct loop into steps whose
deterministic parts we already answer without a model, and best-of-N against
a checkable reward instead of a majority vote that discards the
disagreement. The cheaper the detector model, the more of the return sits
there and the less in prompt evolution -- the opposite of the order this
brief first proposed.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
SIMBA ranks instances by max-to-min score gap across variants, which turns
our noise floor from an obstacle into an instance sampler: the 11 injection
pairs that disagree with themselves are the pairs where a rule change can
flip an outcome. Its two alternating strategies -- append a demo, append a
rule -- are the split our levers already have.
GEPA's reflection meta-prompt is a specification for the feedback function,
not just a prompt: it asks the reflector to extract domain facts the model
cannot re-derive later, which means feedback must carry which sink, which
span and which side leaked, not a verdict.
And the finding that outranks both: GEPA splits a trainset, SIMBA batches 32.
Injection has 12 train teachers. These algorithms are sample-efficient in
rollouts, not in instances, and they will not shortcut a corpus of twelve.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The learning harness built the measurement honestly and then hill-climbed on
it by hand. Four review rounds' worth of defects in that loop were defects of
its shape, not its code.
Read from GEPA's Algorithm 1 and 2 rather than from a summary, three things
bear directly on ours: the acceptance gate is deliberately cheap and noisy,
because the Pareto frontier -- not the gate -- is what protects against a bad
accept; selection is stochastic and diversity-preserving, which is what stops
the search collapsing onto one lineage; and the expensive full evaluation is
spent only on candidates that already passed the cheap gate.
Two consequences for us are honesty findings, not performance ones. Our
acceptance rule reads the holdout every round, so repeated rounds overfit the
test set -- the same leak class this project already measured once, one level
up. And the proposer is handed a score where the published work hands it
textual feedback from the evaluation trace, which our evaluator produces and
then discards.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
