mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
Marius CiepluchandClaude Opus 5
83ab533fc4
Three corrections and one design requirement, from measurement and from maintainer direction. MEASURED. 4,463 PMPro regions carry five distinct ranks, and 3,742 of them -- 83.8% -- share rank 0.30, ordered alphabetically by path. That tier spans positions 159..3,900, and 341 of the 500 examined regions come from it: two thirds of the budget spent on an arbitrary alphabetical slice. It is not a weak ranker, it is barely a ranker. CORRECTED. CVE-2023-23488's region is at position 390 of 4,463 -- INSIDE the budget and examined. I stated repeatedly that it ranked ~1036 and was never reached; that number came from before the array-callable and method-recogniser fixes moved it. The reason it goes unreported is the taint query timing out, so no region got a verdict at all. Ranking is a real problem; it is not this miss, and conflating them would have sent the fix to the wrong place. DIRECTION. Ranking is the LLM's job, not a tie-breaker for a residue. The judgement "is this worth looking at" is a judgement about unfamiliar code. Facts and generic features stay as cheap evidence for the model to read and as the fallback when none is configured. AND THE LOOP IS CLOSED, which is worse than train-on-test and was the sharper point. A region the ranker excludes is never arbitrated, so it never produces a label, so the ranker never learns it was wrong -- the feedback is censored by the policy being trained, and every run confirms the ordering it already had. Optimising on that makes it more confident, not more correct. Breaking the loop is therefore a design requirement, with two cheap mechanisms recorded: an exploration slice (~10% of budget sampled from outside the top-K, the only labels the policy did not choose) and full labelling on repositories small enough to arbitrate every region, which is policy-independent ground truth. Inverse-propensity weighting is noted as available if the ranking is made stochastic, and as unnecessary otherwise. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG