mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
Marius CiepluchandClaude Opus 5
6dbb0e6422
The first like-for-like comparison with the LLM hunter, on the same 30 vibe-py
pairs and the same metric.
hunter (K=5) pipeline (K=1)
pair-correct 76.7% 73.3%
leak 12.3% 6.7%
cost ~$0.69 $0.0126
determinism none entailed band deterministic
Half the leak, 55x cheaper, a fifth of the model calls, one pair short of the
hunter's recall (22 against 23 of 30). And understated: all six both_silent
pairs are access_control and config_secrets, which had no pipeline path to
their arbiters when this ran. A corrected run supersedes those rows.
The rung split says what is doing the work: 15 suspicion to 9 model_entailed on
the vulnerable side. The LLM supplies most of the recall; the model supplies
the precision, halving the leak by dropping claims it could contradict. That is
the division of labour the architecture was designed around and the first
evidence it pays.
It also revises an earlier conclusion of mine. "Detection has regressed" was
measured with the proposer disconnected -- an arbiter compared against a
detector. With the enumerator wired in, the comparison reverses on precision
and cost while holding recall.
And it makes the strategy concrete: the model arbitrates about a third of
findings today, and every suspicion converted to model_entailed is leak removed
for free and one fewer model call. Sink coverage and interprocedural resolution
now have a measured payoff rather than a hoped-for one.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG