docs(contributor-scan): the ranker is 84% one tier, and the loop is closed

Three corrections and one design requirement, from measurement and from
maintainer direction.

MEASURED. 4,463 PMPro regions carry five distinct ranks, and 3,742 of them --
83.8% -- share rank 0.30, ordered alphabetically by path. That tier spans
positions 159..3,900, and 341 of the 500 examined regions come from it: two
thirds of the budget spent on an arbitrary alphabetical slice. It is not a
weak ranker, it is barely a ranker.

CORRECTED. CVE-2023-23488's region is at position 390 of 4,463 -- INSIDE the
budget and examined. I stated repeatedly that it ranked ~1036 and was never
reached; that number came from before the array-callable and method-recogniser
fixes moved it. The reason it goes unreported is the taint query timing out,
so no region got a verdict at all. Ranking is a real problem; it is not this
miss, and conflating them would have sent the fix to the wrong place.

DIRECTION. Ranking is the LLM's job, not a tie-breaker for a residue. The
judgement "is this worth looking at" is a judgement about unfamiliar code.
Facts and generic features stay as cheap evidence for the model to read and as
the fallback when none is configured.

AND THE LOOP IS CLOSED, which is worse than train-on-test and was the sharper
point. A region the ranker excludes is never arbitrated, so it never produces
a label, so the ranker never learns it was wrong -- the feedback is censored by
the policy being trained, and every run confirms the ordering it already had.
Optimising on that makes it more confident, not more correct.

Breaking the loop is therefore a design requirement, with two cheap mechanisms
recorded: an exploration slice (~10% of budget sampled from outside the top-K,
the only labels the policy did not choose) and full labelling on repositories
small enough to arbitrate every region, which is policy-independent ground
truth. Inverse-propensity weighting is noted as available if the ranking is
made stochastic, and as unnecessary otherwise.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG

.kiro/specs/contributor-scan/tasks.md

@@ -849,12 +849,26 @@
     access levels to numbers and knows nothing about any framework. The EVIDENCE feeding it does not:
     `mapping._wordpress_hook_access` is a Python function containing the literals `wp_ajax_nopriv_` and
     `wp_ajax_`, and `_rest_route_access` contains `__return_true`. One framework, written in code.
-  - **The consequence is not "slightly worse on other frameworks", it is no ordering at all.** Anything the
-    classifier does not recognise falls to `review-required` = 0.30, and within a tie the order is
-    alphabetical by path -- arbitrary. On Paid Memberships Pro that is where CVE-2023-23488's region sits,
-    below every REST handler at 0.80, at position ~1036 of 4,463. With a 500-region budget the arbiter never
-    sees it. A Laravel, Symfony or Django codebase would have EVERY region in that tier: 4,463 regions
-    ordered by filename.
+  - **The consequence is not "slightly worse on other frameworks", it is no ordering at all.** Measured on
+    Paid Memberships Pro, 4,463 regions carry **five distinct ranks**:
+
+    | rank | regions | |
+    |---|---|---|
+    | 1.00 | 79 | 1.8% |
+    | 0.80 | 80 | 1.8% |
+    | **0.30** | **3,742** | **83.8%** -- one tier, ordered ALPHABETICALLY BY PATH |
+    | 0.15 | 464 | 10.4% |
+    | 0.10 | 98 | 2.2% |
+
+    The 0.30 tier spans positions 159 to 3,900, and **341 of the 500 regions examined come from it** -- two
+    thirds of the budget spent on an arbitrary alphabetical slice. A Laravel or Django codebase would have
+    every region in that tier: thousands, ordered by filename.
+  - **A correction that matters, because it was stated repeatedly and was wrong.** CVE-2023-23488's region
+    is at position **390 of 4,463** -- INSIDE the 500-region budget, and examined. The earlier claim that it
+    ranked ~1036 and was never reached came from a measurement taken before the array-callable and
+    method-recogniser fixes moved it. The reason it is not reported is that the taint query TIMED OUT
+    (`query_failed`, all 2,500 requests), so no region got a verdict, well ranked or not. Ranking is a real
+    problem; it is not this miss.
   - The fix is three layers, and the LLM is only the third:
     - **Facts, not code.** Framework vocabulary moves into the semantic tables the way `[[dispatch]]` already
       did today. An `[[access]]` fact declares hook-name patterns and their access level, so
@@ -868,8 +882,11 @@
       WordPress rule. This is the layer that would have moved CVE-2023-23488's region on its own, and it
       should be built before any model is involved -- otherwise the LLM's contribution cannot be measured
       against anything.
-    - **The LLM for the residue.** Regions the facts cannot classify and the features cannot separate. It
-      sees the feature vector plus a short excerpt and orders them.
+    - **The LLM does the ranking.** Not a tie-breaker for a residue -- the ordering job itself. It sees the
+      feature vector plus a short excerpt and orders regions, because the thing being judged ("is this
+      worth looking at") is a judgement about unfamiliar code, which is what a model is for and what a
+      hardcoded table demonstrably is not. Layers one and two remain as CHEAP EVIDENCE for it to read and
+      as the fallback when no model is configured, not as the primary ranker.
   - **Why ranking is a safe place for a model, and adjudication is not.** A ranker cannot manufacture a
     finding. It decides what is LOOKED AT, never what is reported: the arbiters still entail or stay silent,
     the three gates stay byte-identical, and a misranked region costs recall rather than producing a false
@@ -878,10 +895,27 @@
   - Constraints that belong in the design before any model budget is spent:
     - **Cost.** One call per region is 4,463 calls a scan and nobody runs that in CI. The model ranks a
       prefiltered candidate set in batches, or it is not shipped.
-    - **Leak.** Four pinned CVEs is a tiny training set and this project has the scar already -- see the
-      closed-loop train-on-test finding, where the improve lever learned from holdout pairs and inflated its
-      own numbers. Optimisation must be leave-one-repository-out and the reported figure must come from a
-      plugin the optimiser never saw.
+    - **Leak, and it is worse than train-on-test.** Two distinct problems, and only the first is the usual
+      one:
+      1. **Overfitting.** Four pinned CVEs is not a training set. This needs N projects and
+         leave-one-repository-out, with the reported figure from a plugin the optimiser never saw. The
+         project already has the scar -- the closed-loop finding where the improve lever learned from
+         holdout pairs and inflated its own numbers.
+      2. **The loop is CLOSED: the policy determines its own training data.** A region the ranker does not
+         put in the budget is never arbitrated, so it never produces a label, so the ranker never learns it
+         was wrong about it. The feedback is censored by the very policy being trained, and every run
+         confirms the ordering it already had. Optimising on that signal makes the ranker more confident,
+         not more correct.
+    - **Breaking the loop is a design requirement, not a refinement.** Two mechanisms, both cheap:
+      - **An exploration slice.** Reserve a fraction of the budget -- 10% is a reasonable start -- for
+        regions sampled from OUTSIDE the top-K. Those are the only labels the ranker did not choose, and
+        they are what makes the training signal unbiased. It also has an honest side effect: the scan reports
+        that some of its budget went to exploration rather than to its own best guesses.
+      - **Full labelling on a small corpus.** For repositories small enough to arbitrate EVERY region --
+        VAmPI at 25 regions, a single plugin file -- the ground truth is complete and policy-independent.
+        Expensive per repository and bounded in number, which is exactly the right shape for a reference set.
+      If the ranking is made stochastic, inverse-propensity weighting is available too, but exploration plus
+      a small fully-labelled corpus is simpler and does not require the ranker to be probabilistic.
     - **Reproducibility.** Two runs of one repository must agree, or a baseline diff means nothing. The
       ranking is recorded in the manifest so a change in what was examined is auditable rather than
       invisible.

src/openultrasast/model/endpoint.py

@@ -25,15 +25,23 @@ DEEPSEEK_BASE_URL = "https://api.deepseek.com"  # no version suffix: the platfor
 DEEPSEEK_KEY_ENV = "DEEPSEEK_API_KEY"
 DEEPSEEK_BASE_ENV = "DEEPSEEK_BASE_URL"
 OPENROUTER_BASE_URL = "https://openrouter.ai/api/v1"
-DEFAULT_DETECTOR_MODEL = "deepseek-v4-flash"
-DEFAULT_JUDGE_MODEL = "deepseek-v4-pro"
+# V4 Pro is discontinued: DeepSeek announced V4.1 Flash on 2026-09-10 and routes Pro requests to it from
+# 12:00 Beijing time on 2026-09-14, billed at Flash's price. So the judge tier is no longer a more expensive
+# model -- detector and judge are the same model, and any claim in this repo that the judge is a *stronger*
+# second opinion is now a claim about prompt and evidence, not about model class.
+DEFAULT_DETECTOR_MODEL = "deepseek-v4.1-flash"
+DEFAULT_JUDGE_MODEL = "deepseek-v4.1-flash"
 Provider = Literal["deepseek", "openrouter", "custom", "scripted"]
 _SCRIPTED_FLAGS = frozenset({"scripted", "script", "dump", "dump-only", "unsafe", "unsafe-snippet"})
 
 
 @dataclass(frozen=True)
 class Prices:
-    """USD per million tokens. DeepSeek platform pricing, peak rate, recorded 2026-09-06."""
+    """USD per million tokens. DeepSeek platform pricing, peak rate, recorded 2026-09-06.
+
+    The older ids are kept so a recorded run from before the V4.1 Flash migration still costs out correctly;
+    a manifest that cannot price its own model is a manifest nobody can audit.
+    """
 
     cache_hit_per_m: float
     input_per_m: float
@@ -41,6 +49,8 @@ class Prices:
 
 
 _PRICES: dict[str, Prices] = {
+    # V4.1 Flash is priced as Flash; Pro requests are routed here and billed at this rate from 2026-09-14.
+    "deepseek-v4.1-flash": Prices(cache_hit_per_m=0.014, input_per_m=0.44, output_per_m=1.32),
     "deepseek-v4-flash": Prices(cache_hit_per_m=0.014, input_per_m=0.44, output_per_m=1.32),
     "deepseek-v4-flash-vision-exp": Prices(cache_hit_per_m=0.014, input_per_m=0.44, output_per_m=1.32),
     "deepseek-v4-pro": Prices(cache_hit_per_m=0.044, input_per_m=1.32, output_per_m=3.96),

tests/test_model_endpoint.py

@@ -93,7 +93,9 @@ def test_the_config_can_point_the_chat_endpoint_somewhere_else(monkeypatch: pyte
     resolved = resolve_chat_endpoint(config)
     assert resolved is not None and resolved[1].base_url == "http://localhost:11434/v1"
     detector, judge = resolve_models(config)
-    assert detector == "deepseek-v4-flash" and judge == "deepseek-v4-pro"
+    # The default detector follows DEFAULT_DETECTOR_MODEL; the judge here is pinned by config to a legacy id,
+    # which must keep working so an old run can be reproduced and re-priced.
+    assert detector == "deepseek-v4.1-flash" and judge == "deepseek-v4-pro"
 
 
 def test_the_adapter_disables_thinking_asks_for_json_and_keeps_the_reasoning_field() -> None: