mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 11:49:28 +00:00
Marius CiepluchandClaude Opus 5
c74e101d10
hunter (K=5) pipeline (K=1)
pair-correct 76.7% 86.7%
leak 12.3% 6.7%
detection -- 93.3%
cost ~$0.69 $0.0067
Ten points of recall over the hunter, half the leak, a fifth of the model calls,
~100x cheaper. 26 of 30 pairs correct.
Four runs, and each step came from a measurement contradicting an assumption --
twice one I had stated:
1 73.3% the absence and config families had no pipeline path at all
2 73.3% routing fixed and the result IDENTICAL, which was the clue: the
arbiter was reached and its verdict thrown away, because every
corroboration was put to the judge as a taint question
3 80.0% residual question phrased for the arbiter that raised it
4 86.7% aiohttp Application(debug=True) and RedisStorage(httponly=False)
added to the settings table
The division of labour keeps moving the right way: model_entailed 9 -> 11 while
suspicion held at 15, so both findings gained in run 4 came from the MODEL, not
the LLM, and cost fell from $0.0126 to $0.0067 as work moved onto the free
deterministic path. That is the payoff curve for fact-table coverage, measured
rather than argued: closing a gap moves findings out of the LLM's column into
the model's, improving precision and cost at once.
Recorded corrections and limits, because the numbers are only worth what the
caveats are. The three silent config pairs were called corpus defects on a
similarity threshold borrowed from the OWASP minimal-fix analysis; it does not
apply to a slice that deliberately pairs a vulnerable function with a different
safe one, where 36% and 34% pairs both score pair_correct. Two were a settings
gap and now split. And permissive_values is a flat literal list with no keyword
association, holding both True and False -- debug=True is dangerous where
httponly=True is safe, and the model cannot tell them apart. It did not bite
here, and it is a real limit of the fact schema.
Four remaining misses, each with a named cause. Four runs cost about $0.04.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG