mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-18 20:49:28 +00:00
feat(learning-harness): three-tier auto-classifier
Task 2.1. Tier one decides from what the repository already knows: a declared family label, an obligation kind, a mechanism id, the rule a finding came from, or the labeled weakness, and it never spends a model call. Tier two asks one closed question in JSON mode with thinking disabled, and abstains when the answer names an id outside the taxonomy or the reply's mean token log-probability says the model was guessing. Tier three is abstention. Every answer records which tier decided it. Answers are multi-label and come back in taxonomy order, so two runs of the same question compare equal. The classifier proposes and never writes a label, and it owns validation of the labels the corpus carries, because the corpus stores a family as an opaque string. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
co-authored by
Claude Opus 5
parent
6965c39b7b
commit
37af831b0d
@@ -186,8 +186,10 @@ class ScriptedChatClient:
|
||||
messages: list[dict[str, object]],
|
||||
tools: list[dict[str, object]],
|
||||
timeout_seconds: int = 60,
|
||||
**options: object,
|
||||
) -> ChatResponse:
|
||||
self.calls.append({"model": model, "messages": list(messages), "tools": tools, "timeout_seconds": timeout_seconds})
|
||||
# `options` carries the adapter's extras (json_object, logprobs) so a test can assert what was asked for.
|
||||
self.calls.append({"model": model, "messages": list(messages), "tools": tools, "timeout_seconds": timeout_seconds, **options})
|
||||
index = len(self.calls) - 1
|
||||
if index >= len(self._turns):
|
||||
return ChatResponse(content="")
|
||||
|
||||
Reference in New Issue
Block a user