SIMBA ranks instances by max-to-min score gap across variants, which turns
our noise floor from an obstacle into an instance sampler: the 11 injection
pairs that disagree with themselves are the pairs where a rule change can
flip an outcome. Its two alternating strategies -- append a demo, append a
rule -- are the split our levers already have.
GEPA's reflection meta-prompt is a specification for the feedback function,
not just a prompt: it asks the reflector to extract domain facts the model
cannot re-derive later, which means feedback must carry which sink, which
span and which side leaked, not a verdict.
And the finding that outranks both: GEPA splits a trainset, SIMBA batches 32.
Injection has 12 train teachers. These algorithms are sample-efficient in
rollouts, not in instances, and they will not shortcut a corpus of twelve.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The learning harness built the measurement honestly and then hill-climbed on
it by hand. Four review rounds' worth of defects in that loop were defects of
its shape, not its code.
Read from GEPA's Algorithm 1 and 2 rather than from a summary, three things
bear directly on ours: the acceptance gate is deliberately cheap and noisy,
because the Pareto frontier -- not the gate -- is what protects against a bad
accept; selection is stochastic and diversity-preserving, which is what stops
the search collapsing onto one lineage; and the expensive full evaluation is
spent only on candidates that already passed the cheap gate.
Two consequences for us are honesty findings, not performance ones. Our
acceptance rule reads the holdout every round, so repeated rounds overfit the
test set -- the same leak class this project already measured once, one level
up. And the proposer is handed a score where the published work hands it
textual feedback from the evaluation trace, which our evaluator produces and
then discards.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
