mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 11:49:28 +00:00
docs(contributor-scan): the ranker is 84% one tier, and the loop is closed
Three corrections and one design requirement, from measurement and from maintainer direction. MEASURED. 4,463 PMPro regions carry five distinct ranks, and 3,742 of them -- 83.8% -- share rank 0.30, ordered alphabetically by path. That tier spans positions 159..3,900, and 341 of the 500 examined regions come from it: two thirds of the budget spent on an arbitrary alphabetical slice. It is not a weak ranker, it is barely a ranker. CORRECTED. CVE-2023-23488's region is at position 390 of 4,463 -- INSIDE the budget and examined. I stated repeatedly that it ranked ~1036 and was never reached; that number came from before the array-callable and method-recogniser fixes moved it. The reason it goes unreported is the taint query timing out, so no region got a verdict at all. Ranking is a real problem; it is not this miss, and conflating them would have sent the fix to the wrong place. DIRECTION. Ranking is the LLM's job, not a tie-breaker for a residue. The judgement "is this worth looking at" is a judgement about unfamiliar code. Facts and generic features stay as cheap evidence for the model to read and as the fallback when none is configured. AND THE LOOP IS CLOSED, which is worse than train-on-test and was the sharper point. A region the ranker excludes is never arbitrated, so it never produces a label, so the ranker never learns it was wrong -- the feedback is censored by the policy being trained, and every run confirms the ordering it already had. Optimising on that makes it more confident, not more correct. Breaking the loop is therefore a design requirement, with two cheap mechanisms recorded: an exploration slice (~10% of budget sampled from outside the top-K, the only labels the policy did not choose) and full labelling on repositories small enough to arbitrate every region, which is policy-independent ground truth. Inverse-propensity weighting is noted as available if the ranking is made stochastic, and as unnecessary otherwise. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
co-authored by
Claude Opus 5
parent
5abc6cd6f6
commit
83ab533fc4
@@ -25,15 +25,23 @@ DEEPSEEK_BASE_URL = "https://api.deepseek.com" # no version suffix: the platfor
|
||||
DEEPSEEK_KEY_ENV = "DEEPSEEK_API_KEY"
|
||||
DEEPSEEK_BASE_ENV = "DEEPSEEK_BASE_URL"
|
||||
OPENROUTER_BASE_URL = "https://openrouter.ai/api/v1"
|
||||
DEFAULT_DETECTOR_MODEL = "deepseek-v4-flash"
|
||||
DEFAULT_JUDGE_MODEL = "deepseek-v4-pro"
|
||||
# V4 Pro is discontinued: DeepSeek announced V4.1 Flash on 2026-09-10 and routes Pro requests to it from
|
||||
# 12:00 Beijing time on 2026-09-14, billed at Flash's price. So the judge tier is no longer a more expensive
|
||||
# model -- detector and judge are the same model, and any claim in this repo that the judge is a *stronger*
|
||||
# second opinion is now a claim about prompt and evidence, not about model class.
|
||||
DEFAULT_DETECTOR_MODEL = "deepseek-v4.1-flash"
|
||||
DEFAULT_JUDGE_MODEL = "deepseek-v4.1-flash"
|
||||
Provider = Literal["deepseek", "openrouter", "custom", "scripted"]
|
||||
_SCRIPTED_FLAGS = frozenset({"scripted", "script", "dump", "dump-only", "unsafe", "unsafe-snippet"})
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Prices:
|
||||
"""USD per million tokens. DeepSeek platform pricing, peak rate, recorded 2026-09-06."""
|
||||
"""USD per million tokens. DeepSeek platform pricing, peak rate, recorded 2026-09-06.
|
||||
|
||||
The older ids are kept so a recorded run from before the V4.1 Flash migration still costs out correctly;
|
||||
a manifest that cannot price its own model is a manifest nobody can audit.
|
||||
"""
|
||||
|
||||
cache_hit_per_m: float
|
||||
input_per_m: float
|
||||
@@ -41,6 +49,8 @@ class Prices:
|
||||
|
||||
|
||||
_PRICES: dict[str, Prices] = {
|
||||
# V4.1 Flash is priced as Flash; Pro requests are routed here and billed at this rate from 2026-09-14.
|
||||
"deepseek-v4.1-flash": Prices(cache_hit_per_m=0.014, input_per_m=0.44, output_per_m=1.32),
|
||||
"deepseek-v4-flash": Prices(cache_hit_per_m=0.014, input_per_m=0.44, output_per_m=1.32),
|
||||
"deepseek-v4-flash-vision-exp": Prices(cache_hit_per_m=0.014, input_per_m=0.44, output_per_m=1.32),
|
||||
"deepseek-v4-pro": Prices(cache_hit_per_m=0.044, input_per_m=1.32, output_per_m=3.96),
|
||||
|
||||
Reference in New Issue
Block a user