Task 2.2. The report covers the whole corpus and scores the classifier only
where a maintainer label exists. Agreement uses the classifier's first answer,
the oracle number uses the best of its answers, and the random baseline is a
uniform pick over the nine real families, so a router error is visible on its
own rather than hidden inside a detector number. Credit is hierarchical: full
for the same family, half for a parent or child, none for a lateral or
fabricated id.
When it is being measured, the classifier no longer reads the declared label it
is scored against, otherwise agreement would be the classifier reading back its
own answer sheet. With no maintainer labels in the corpus yet the report says
the reference set is empty instead of inventing a score, and every disagreement
is queued once with both answers and the deciding tier for a human to settle.
Measured on the 277 vendored pairs: tier one places 275 of them without a single
model call, only two land in unknown, and the 176 memory-safety rows resolve to
one family instead of reading as an unclassified catch-all.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
