Files
Open_Ultra_SAST/benchmarks
Marius CiepluchandClaude Fable 5.1 d37d9634c6 fix(benchmark): classify hunter findings into the taxonomy before scoring arm (b)
The tool hunter's family tag is the model's free text (a live run returned
family:sql-parser and family:readonly-enforcement), so every arm (b) defect
scored family "unknown" and score.catches() could never credit the hunter
even when it landed on the labelled file.

hunter_arm now classifies each finding into the closed taxonomy vocabulary
with the decision-engine classifier (learn.program.classify with
families=FAMILY_IDS minus unknown): a numbered, clamped excerpt around the
finding sets valid_lines, the canonical ANSWER_FORMAT is reused, an off-list
family parses to "none" and lands as "unknown". The benchmark label is never
a fallback. The defect keeps the raw hunter family, title and rationale plus
the classifier's verdict, confidence, rationale, cited lines and parse
failures for audit; the record counts classifier calls and families.
--no-classify keeps the old heuristic path.

14 new offline tests, 67 green with the benchmark tests.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-12 20:42:08 +02:00
..