The final-answer turn requested the provider's JSON mode with the tools still
attached. DeepSeek (deepseek-flash) then answered with the format itself
(`{"type": "json_object"}`), wrapped the array (`{"type": "json_object",
"value": [...]}`), or emitted its tool-call markup as text. A hunt that had found
the vulnerability reported nothing.
Measured on the v2 sink verifier with the declared vulnerable functions: 0 of 7
confirmed in JSON mode, 5 of 7 asked plainly (6 of 7 at 8 steps). The model named
Medplum's actual redirect-URI prefix-match bug, LMDeploy's redirect bypass of
_is_safe_url, and ContextForge's unsandboxed Jinja2.
The final turn is now plain text without tools. The parser also reads a
`{"value": [...]}` wrapper. Two tests that pinned JSON mode now pin the plain
turn, and a new test reproduces the wrapped answer.
Invalidates as measurements: the hunter experiment on v2 (c4a0bd8) and the first
sink-verifier run. Both used this turn.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Measured live against DeepSeek on one pair: the model spends its step budget
on tools and then answers in a paragraph that names the sink, the argument
and the line -- which parses as nothing. A detector that found the bug and
reported zero is indistinguishable from one that found nothing, and every
family in round zero would have baselined at zero for this reason alone.
The loop now takes one more turn when the hunt ends in prose or runs out of
steps, asking the provider for JSON and handing back the reasoning the model
already produced. It formats an answer that was reached; it never asks for a
new one, and a hunt that called no tool never gets there.
Same pair, same model, before and after: recall 0.00 -> 1.00, Youden 0.00 ->
1.00 over three runs per side.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
A mechanism claimed by two families now fails as loudly as a weakness claimed
twice, with a named offender. Two behaviours that were correct but unasserted
are pinned: the obligations tool refuses a path outside the repository, and a
provider that refuses to build degrades to no endpoint instead of raising into
a scan. The TypeScript parse assertion is now exact rather than a quarter-of-
the-slice allowance, which surfaces one real corpus defect: a TSX excerpt
harvested as a truncated destructuring parameter list is not valid TypeScript
on its own, and belongs to the re-harvest rather than to the grammar.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 1.5. Four tools join the three the hunter already had. Resolving a
definition by symbol returns its file, line span and body, decorators included,
so a detector stops guessing what a callee or a route handler does. Listing
entry points, flows and obligations answers with identifiers and line spans
only, so a structural tool can never become a second way to dump the
repository. Every path a caller supplies is clamped to the scan root first.
The loop now takes the system prompt, the user prompt, the files to start from,
the tool subset, the character limit and the tags its findings carry, each
defaulting to today's behaviour, and it replays the provider's reasoning field
on the next turn because the provider otherwise drops its own chain of thought
across a tool call. A detector configuration is exactly this set of knobs.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
