The tool hunter's family tag is the model's free text (a live run returned
family:sql-parser and family:readonly-enforcement), so every arm (b) defect
scored family "unknown" and score.catches() could never credit the hunter
even when it landed on the labelled file.
hunter_arm now classifies each finding into the closed taxonomy vocabulary
with the decision-engine classifier (learn.program.classify with
families=FAMILY_IDS minus unknown): a numbered, clamped excerpt around the
finding sets valid_lines, the canonical ANSWER_FORMAT is reused, an off-list
family parses to "none" and lands as "unknown". The benchmark label is never
a fallback. The defect keeps the raw hunter family, title and rationale plus
the classifier's verdict, confidence, rationale, cited lines and parse
failures for audit; the record counts classifier calls and families.
--no-classify keeps the old heuristic path.
14 new offline tests, 67 green with the benchmark tests.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
benchmarks/valid/hunter_arm.py runs the shipped deep-mode tool hunter
(tool_hunter.run_tool_hunter via resolve_hunter_client) over each frozen
change off-cluster: detached worktree at head with git hooks disabled,
hotspots scoped to the changed files, instrument rule on every hotspot
(bytes read printed, empty or unreadable fails the change loudly), model id
from the config resolver, family from the taxonomy, never from the label.
Records land in the exact shape score.py reads (push.admission.defects with
locations), resumable per change.
`combine` joins arm (a) replay records with arm (b) hunter records per change
id, dedupes defects by (path, line, family), keeps (a)'s instrument, records
which arms were present, and writes a file score.py loads directly.
9 offline tests (stub hunter, real scorer), 53 green with the benchmark
tests. No network, no key handling, no live run.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
