mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-17 20:19:29 +00:00
fix(hunter): ask for the final answer in plain text; DeepSeek's JSON mode lost it
The final-answer turn requested the provider's JSON mode with the tools still
attached. DeepSeek (deepseek-flash) then answered with the format itself
(`{"type": "json_object"}`), wrapped the array (`{"type": "json_object",
"value": [...]}`), or emitted its tool-call markup as text. A hunt that had found
the vulnerability reported nothing.
Measured on the v2 sink verifier with the declared vulnerable functions: 0 of 7
confirmed in JSON mode, 5 of 7 asked plainly (6 of 7 at 8 steps). The model named
Medplum's actual redirect-URI prefix-match bug, LMDeploy's redirect bypass of
_is_safe_url, and ContextForge's unsandboxed Jinja2.
The final turn is now plain text without tools. The parser also reads a
`{"value": [...]}` wrapper. Two tests that pinned JSON mode now pin the plain
turn, and a new test reproduces the wrapped answer.
Invalidates as measurements: the hunter experiment on v2 (c4a0bd8) and the first
sink-verifier run. Both used this turn.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
267bca5232
commit
3ffb8b4c54
@@ -179,7 +179,9 @@ def test_the_loop_asks_once_more_for_the_answer_instead_of_dropping_prose() -> N
|
||||
findings = run_tool_hunter(root, [], client=client, model="m", max_steps=3)
|
||||
assert [item.line for item in findings] == [2]
|
||||
final = client.calls[-1]
|
||||
assert final.get("json_object") is True, "the last turn asks the provider for JSON, it does not hope for it"
|
||||
# Asked for JSON in the prompt, not through the provider's JSON mode: with tools attached, DeepSeek's JSON mode
|
||||
# returned `{"type": "json_object"}` and lost the answers (tests/test_tool_hunter.py pins the shapes).
|
||||
assert not final.get("json_object") and final.get("tools") == [], "the last turn is plain text without tools"
|
||||
text = "\n".join(str(message.get("content") or "") for message in final["messages"])
|
||||
assert "JSON" in text and prose in text, "the prose it already wrote is what the final turn is asked to format"
|
||||
|
||||
|
||||
Reference in New Issue
Block a user