docs(contributor-scan): the ranker is 84% one tier, and the loop is closed

Three corrections and one design requirement, from measurement and from
maintainer direction.

MEASURED. 4,463 PMPro regions carry five distinct ranks, and 3,742 of them --
83.8% -- share rank 0.30, ordered alphabetically by path. That tier spans
positions 159..3,900, and 341 of the 500 examined regions come from it: two
thirds of the budget spent on an arbitrary alphabetical slice. It is not a
weak ranker, it is barely a ranker.

CORRECTED. CVE-2023-23488's region is at position 390 of 4,463 -- INSIDE the
budget and examined. I stated repeatedly that it ranked ~1036 and was never
reached; that number came from before the array-callable and method-recogniser
fixes moved it. The reason it goes unreported is the taint query timing out,
so no region got a verdict at all. Ranking is a real problem; it is not this
miss, and conflating them would have sent the fix to the wrong place.

DIRECTION. Ranking is the LLM's job, not a tie-breaker for a residue. The
judgement "is this worth looking at" is a judgement about unfamiliar code.
Facts and generic features stay as cheap evidence for the model to read and as
the fallback when none is configured.

AND THE LOOP IS CLOSED, which is worse than train-on-test and was the sharper
point. A region the ranker excludes is never arbitrated, so it never produces
a label, so the ranker never learns it was wrong -- the feedback is censored by
the policy being trained, and every run confirms the ordering it already had.
Optimising on that makes it more confident, not more correct.

Breaking the loop is therefore a design requirement, with two cheap mechanisms
recorded: an exploration slice (~10% of budget sampled from outside the top-K,
the only labels the policy did not choose) and full labelling on repositories
small enough to arbitrate every region, which is policy-independent ground
truth. Inverse-propensity weighting is noted as available if the ranking is
made stochastic, and as unnecessary otherwise.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-10 15:49:57 +02:00
co-authored by Claude Opus 5
parent 5abc6cd6f6
commit 83ab533fc4
3 changed files with 62 additions and 16 deletions
+13 -3
View File
@@ -25,15 +25,23 @@ DEEPSEEK_BASE_URL = "https://api.deepseek.com" # no version suffix: the platfor
DEEPSEEK_KEY_ENV = "DEEPSEEK_API_KEY"
DEEPSEEK_BASE_ENV = "DEEPSEEK_BASE_URL"
OPENROUTER_BASE_URL = "https://openrouter.ai/api/v1"
DEFAULT_DETECTOR_MODEL = "deepseek-v4-flash"
DEFAULT_JUDGE_MODEL = "deepseek-v4-pro"
# V4 Pro is discontinued: DeepSeek announced V4.1 Flash on 2026-09-10 and routes Pro requests to it from
# 12:00 Beijing time on 2026-09-14, billed at Flash's price. So the judge tier is no longer a more expensive
# model -- detector and judge are the same model, and any claim in this repo that the judge is a *stronger*
# second opinion is now a claim about prompt and evidence, not about model class.
DEFAULT_DETECTOR_MODEL = "deepseek-v4.1-flash"
DEFAULT_JUDGE_MODEL = "deepseek-v4.1-flash"
Provider = Literal["deepseek", "openrouter", "custom", "scripted"]
_SCRIPTED_FLAGS = frozenset({"scripted", "script", "dump", "dump-only", "unsafe", "unsafe-snippet"})
@dataclass(frozen=True)
class Prices:
"""USD per million tokens. DeepSeek platform pricing, peak rate, recorded 2026-09-06."""
"""USD per million tokens. DeepSeek platform pricing, peak rate, recorded 2026-09-06.
The older ids are kept so a recorded run from before the V4.1 Flash migration still costs out correctly;
a manifest that cannot price its own model is a manifest nobody can audit.
"""
cache_hit_per_m: float
input_per_m: float
@@ -41,6 +49,8 @@ class Prices:
_PRICES: dict[str, Prices] = {
# V4.1 Flash is priced as Flash; Pro requests are routed here and billed at this rate from 2026-09-14.
"deepseek-v4.1-flash": Prices(cache_hit_per_m=0.014, input_per_m=0.44, output_per_m=1.32),
"deepseek-v4-flash": Prices(cache_hit_per_m=0.014, input_per_m=0.44, output_per_m=1.32),
"deepseek-v4-flash-vision-exp": Prices(cache_hit_per_m=0.014, input_per_m=0.44, output_per_m=1.32),
"deepseek-v4-pro": Prices(cache_hit_per_m=0.044, input_per_m=1.32, output_per_m=3.96),