marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 11:49:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(hunter): ask for the final answer in plain text; DeepSeek's JSON mode lost it

Browse Source 
The final-answer turn requested the provider's JSON mode with the tools still
attached. DeepSeek (deepseek-flash) then answered with the format itself
(`{"type": "json_object"}`), wrapped the array (`{"type": "json_object",
"value": [...]}`), or emitted its tool-call markup as text. A hunt that had found
the vulnerability reported nothing.

Measured on the v2 sink verifier with the declared vulnerable functions: 0 of 7
confirmed in JSON mode, 5 of 7 asked plainly (6 of 7 at 8 steps). The model named
Medplum's actual redirect-URI prefix-match bug, LMDeploy's redirect bypass of
_is_safe_url, and ContextForge's unsandboxed Jinja2.

The final turn is now plain text without tools. The parser also reads a
`{"value": [...]}` wrapper. Two tests that pinned JSON mode now pin the plain
turn, and a new test reproduces the wrapped answer.

Invalidates as measurements: the hunter experiment on v2 (c4a0bd8) and the first
sink-verifier run. Both used this turn.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-09-28 20:10:57 +02:00
co-authored by Claude Opus 5.5
parent 267bca5232
commit 3ffb8b4c54
3 changed files with 48 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/openultrasast/tool_hunter.py
+10 -1
View File
@@ -396,7 +396,13 @@ def _final_answer(
),
},
]
response = client.complete(model=model, messages=messages, tools=schemas, json_object=True)
# Plain text, no tools. In JSON mode with tools attached, DeepSeek answered the final turn with the format
# itself (`{"type": "json_object"}`), wrapped the array (`{"type": "json_object", "value": [...]}`) or emitted
# its tool-call markup as text -- and a hunt that had found the bug reported nothing. Measured on the v2 sink
# verifier: 0 of 7 declared vulnerable functions confirmed in JSON mode, 5 of 7 asked plainly (6 of 7 at 8
# steps). The parser already takes a bare or fenced array.
del schemas
response = client.complete(model=model, messages=messages, tools=[], json_object=False)
return _findings_from_content(root, hotspots, response.content or "", tags)
@@ -548,6 +554,9 @@ def _findings_from_content(
items = payload
elif isinstance(payload, dict) and isinstance(payload.get("findings"), list):
items = payload["findings"]
elif isinstance(payload, dict) and isinstance(payload.get("value"), list):
# DeepSeek in JSON mode wrapped the array it was asked for as `{"type": "json_object", "value": [...]}`.
items = payload["value"]
else:
return []
scores = {hotspot.path: hotspot.score for hotspot in hotspots}