mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 19:59:26 +00:00
fix(learning-harness): logprobs are on the choice, not the message
OpenAI-shaped providers put `logprobs` beside the message rather than inside it, so the confidence signal the classifier is meant to abstain on was always None -- a knob that reads 'never unsure' whatever the model said. Measured against DeepSeek while checking: it returns the structure but not the values. Every chosen token comes back at logprob 0.0 and every alternative at -9999.0, on a deliberately uncertain prompt as much as a certain one. The field is there; the information is not. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
co-authored by
Claude Opus 5
parent
ce89b0dbe0
commit
1e4f212868
@@ -189,12 +189,19 @@ def _provider_of(base_url: str) -> Provider:
|
||||
|
||||
|
||||
def _message_of(payload: object) -> Mapping[str, object]:
|
||||
"""The reply message, carrying the choice's `logprobs` with it.
|
||||
|
||||
OpenAI-shaped providers put `logprobs` on the *choice*, beside the message rather than inside it. Reading only
|
||||
the message meant the confidence signal was always absent, so a knob meant to let the classifier abstain when
|
||||
it is unsure read as "never unsure" whatever the model said."""
|
||||
if isinstance(payload, Mapping):
|
||||
choices = payload.get("choices")
|
||||
if isinstance(choices, Sequence) and choices and isinstance(choices[0], Mapping):
|
||||
message = choices[0].get("message")
|
||||
choice = choices[0]
|
||||
message = choice.get("message")
|
||||
if isinstance(message, Mapping):
|
||||
return message
|
||||
logprobs = choice.get("logprobs")
|
||||
return {**message, "logprobs": logprobs} if logprobs is not None else message
|
||||
return {}
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user