The training clones are sparse-checkout and a worktree inherits it; without this the hunter arm raises
unreadable_hotspot on every change of the access-control pool. Same helper and checkout proof as the hunt runner.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Opt-in filter on the classifier verdict for hunter-sourced defects; the default
keeps everything. Each combined record notes the kept verdicts and the counts.
Live check on the pilot: (a)+(b) filtered to 'vulnerable' gives recall 5/12 and
54.4 false alarms per 100 ordinary pushes, matching the scratch numbers.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Live run: every classification failed with ValueError because Caller.ask
reads client.usage as a dict (the learn CLI always hands it a MeteredClient)
while a bare DeepSeekChatClient keeps usage as a list of rows. build_caller
now wraps a client that carries a usage list in MeteredClient; scripted and
already-metered clients pass through unchanged.
Regression test runs classify_family through the real DeepSeekChatClient
wrapper with a stub transport (the module fixture that stubs classify is
restored for it). The error field stays the exception type only, since a
message can carry a model key.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The tool hunter's family tag is the model's free text (a live run returned
family:sql-parser and family:readonly-enforcement), so every arm (b) defect
scored family "unknown" and score.catches() could never credit the hunter
even when it landed on the labelled file.
hunter_arm now classifies each finding into the closed taxonomy vocabulary
with the decision-engine classifier (learn.program.classify with
families=FAMILY_IDS minus unknown): a numbered, clamped excerpt around the
finding sets valid_lines, the canonical ANSWER_FORMAT is reused, an off-list
family parses to "none" and lands as "unknown". The benchmark label is never
a fallback. The defect keeps the raw hunter family, title and rationale plus
the classifier's verdict, confidence, rationale, cited lines and parse
failures for audit; the record counts classifier calls and families.
--no-classify keeps the old heuristic path.
14 new offline tests, 67 green with the benchmark tests.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
benchmarks/valid/hunter_arm.py runs the shipped deep-mode tool hunter
(tool_hunter.run_tool_hunter via resolve_hunter_client) over each frozen
change off-cluster: detached worktree at head with git hooks disabled,
hotspots scoped to the changed files, instrument rule on every hotspot
(bytes read printed, empty or unreadable fails the change loudly), model id
from the config resolver, family from the taxonomy, never from the label.
Records land in the exact shape score.py reads (push.admission.defects with
locations), resumable per change.
`combine` joins arm (a) replay records with arm (b) hunter records per change
id, dedupes defects by (path, line, family), keeps (a)'s instrument, records
which arms were present, and writes a file score.py loads directly.
9 offline tests (stub hunter, real scorer), 53 green with the benchmark
tests. No network, no key handling, no live run.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
