feat(learning-harness): three-tier auto-classifier

Task 2.1. Tier one decides from what the repository already knows: a declared
family label, an obligation kind, a mechanism id, the rule a finding came from,
or the labeled weakness, and it never spends a model call. Tier two asks one
closed question in JSON mode with thinking disabled, and abstains when the
answer names an id outside the taxonomy or the reply's mean token
log-probability says the model was guessing. Tier three is abstention. Every
answer records which tier decided it.

Answers are multi-label and come back in taxonomy order, so two runs of the
same question compare equal. The classifier proposes and never writes a label,
and it owns validation of the labels the corpus carries, because the corpus
stores a family as an opaque string.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-06 09:25:45 +02:00
co-authored by Claude Opus 5
parent 6965c39b7b
commit 37af831b0d
3 changed files with 393 additions and 1 deletions
+3 -1
View File
@@ -186,8 +186,10 @@ class ScriptedChatClient:
messages: list[dict[str, object]],
tools: list[dict[str, object]],
timeout_seconds: int = 60,
**options: object,
) -> ChatResponse:
self.calls.append({"model": model, "messages": list(messages), "tools": tools, "timeout_seconds": timeout_seconds})
# `options` carries the adapter's extras (json_object, logprobs) so a test can assert what was asked for.
self.calls.append({"model": model, "messages": list(messages), "tools": tools, "timeout_seconds": timeout_seconds, **options})
index = len(self.calls) - 1
if index >= len(self._turns):
return ChatResponse(content="")