mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-17 03:59:28 +00:00
Marius CiepluchandClaude Opus 5
09898529bd
Task 2.2. The report covers the whole corpus and scores the classifier only where a maintainer label exists. Agreement uses the classifier's first answer, the oracle number uses the best of its answers, and the random baseline is a uniform pick over the nine real families, so a router error is visible on its own rather than hidden inside a detector number. Credit is hierarchical: full for the same family, half for a parent or child, none for a lateral or fabricated id. When it is being measured, the classifier no longer reads the declared label it is scored against, otherwise agreement would be the classifier reading back its own answer sheet. With no maintainer labels in the corpus yet the report says the reference set is empty instead of inventing a score, and every disagreement is queued once with both answers and the deciding tier for a human to settle. Measured on the 277 vendored pairs: tier one places 275 of them without a single model call, only two land in unknown, and the 176 memory-safety rows resolve to one family instead of reading as an unclassified catch-all. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG