Task 3.3. Before anything learns, every family gets the same detector and is
measured against itself. Running a pair several times at the same settings does
not give the same answer, because the chat provider offers no seed, so the
spread between runs is real and unavoidable. That spread is the family's noise
floor and it becomes the budget a later round must stay inside before a
cross-family change counts as a regression. Choosing a budget instead of
measuring one would be a guess dressed as a threshold.
A pair the corpus already marked unscorable is never run: spending model calls
on a row that cannot be scored buys nothing and would put its noise into a floor
where it does not belong.
Artifacts are keyed by detector model, so a second model is baselined by
changing the model and running again, and the comparison table reads both side
by side. A model swap never touches a configuration, which is what makes the two
numbers comparable and reconsidering the model cheap. The web slices are the
default selection and the memory-safety family is measured with its floor marked
as never gating a round.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
