feat(ranking): phase 0 -- the baseline, the evidence instrument, and a 7x taint speedup found on the way

flow-aware-ranking phase 0: measure before changing anything.

THE BASELINE, committed as benchmarks/ranking/baselines/2026-09-10-access-rank.json
from benchmarks/ranking/position.py, which sorts regions exactly as the scan does:

    pmpro         4,463 regions  83.8% at 0.30   CVE at 390   budget_at_recall 391
    mwwpform        983 regions  81.7% at 0.30   CVE at 512   budget_at_recall 513
    wpstatistics  1,673 regions  75.3% at 0.30   CVE at 503   budget_at_recall 504
    vampi            25 regions                  BOLA at 15, 10; SQLI reached transitively

That corrects the brief's own cross-assessment: "concentration, not inclusion"
was measured on pmpro alone. On the other two plugins the CVE sits OUTSIDE a
500-region budget and is never examined; the slices found those CVEs only
because a slice has too few regions to exclude anything.

THE INSTRUMENT: `evidenceOnly` mode on the taint query -- what is in scope,
which sink calls, what shape, whether a sanitizer wraps THEIR argument, whether
a source is local or reachable -- returning before reachableByFlows is ever
called. model/evidence.py turns its rows into a vector per (region, family)
and a tier; scan.py records every pair's tier on the result and acts on none of
it. Every later phase is measured against a record made by scans that did not
know they were being measured.

    2,500 pairs in 98.9s incl. JVM load, ~40 ms per pair
    tier 0 -- no sink of that family in reach: 2,161 of 2,500 = 86%

Two earlier attempts ran past an hour each with no payload, and both were my
fault: repository-wide walks repeated per request -- the sink scan, the
reachable-method BFS, familyInRepo. Memoised across the batch.

AND THE SAME FIX REACHES THE DATAFLOW MODE. sinkCalls and reachableMethods do
not depend on the family, and were recomputed for all 2,500 requests. The
identical 250-request set with both joins on:

    2,026.3s -> 287.0s      8,105 -> 1,148 ms per request      same 5,779 rows

Most of what 5.13 measured as "the base taint query" was never dataflow. A 7x
speedup with nothing about the analysis changed; both corrections recorded in
the specs. Combined with 86% tier 0, the whole-repository taint query that
timed out at 40 minutes is ~339 real requests at ~1.1s: about six minutes.

Also: the module audit manifest is regenerated (91 load-bearing, 7 standalone,
0 orphaned) because evidence.py is now imported by the scan, and the
failed-query test is order-independent since the evidence pass records its own
failure before taint's.

869 tests pass; query compile test 4 passed; gates byte-identical.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG

src/openultrasast/model/scan.py

@@ -26,6 +26,7 @@ from pathlib import Path
 from typing import Any
 
 from ..cpg.backend import CpgResult
+from .evidence import evidence_from_rows
 from .candidates import enumerate_candidates
 from .config_value import request_params as config_params
 from .dominance import request_params as dominance_params
@@ -64,6 +65,11 @@ class ScanBudget:
 
     max_model_calls: int = 200
     max_regions: int = 500
+    # Compute the evidence tier of every (region, family) pair and RECORD it. Phase 0 of flow-aware-ranking:
+    # the instrument runs on every real scan so later phases are measurable, and it changes nothing about
+    # what is asked or in what order. One extra query invocation without dataflow -- a JVM load, then
+    # milliseconds per pair.
+    tiering: bool = True
 
 
 @dataclass(frozen=True)
@@ -87,6 +93,10 @@ class ModelScanResult:
     # would remove) or CPGQL evaluation (which it would not). That is the whole of the server-mode decision.
     query_seconds_by_kind: Mapping[str, float] = field(default_factory=dict)
     degradations: tuple[Mapping[str, object], ...] = ()
+    # The evidence tier of each (path, function, family) pair the scan collected, and the count per tier.
+    # Recorded, not acted on: this is what "the ranker got better" will be measured against.
+    tiers: tuple[tuple[str, str, str, int], ...] = ()
+    tier_counts: Mapping[int, int] = field(default_factory=dict)
 
     @property
     def arbitrate_seconds(self) -> float:
@@ -176,11 +186,44 @@ def scan_repository(
                 continue
             work.append((f"{len(work)}", region, spec))
 
+    # Phase 1b: the evidence tier of every taint pair, before any dataflow is asked. Same query, `evidenceOnly`,
+    # no `reachableByFlows`. Recorded on the result and NOT used to reorder or skip anything -- that is a
+    # later phase, and it must be measured against this record before it is allowed to change behaviour.
+    tiers: list[tuple[str, str, str, int]] = []
+    per_kind: dict[str, float] = {}
+    if limits.tiering and callable(getattr(cpg, "run_batch", None)):
+        grouped_taint = _grouped(work, hook_callbacks=hooks).get("taint", {})
+        if grouped_taint:
+            evidence_requests = {rid: {**dict(params), "evidenceOnly": "true"} for rid, params in grouped_taint.items()}
+            tier_started = time.monotonic()
+            answered_evidence = cpg.run_batch("taint", evidence_requests)
+            per_kind["evidence"] = round(time.monotonic() - tier_started, 2)
+            if answered_evidence is None:
+                degradations.append({"stage": "model", "reason": "query_failed", "kind": "evidence", "requests": len(evidence_requests)})
+            else:
+                answered_evidence.pop("__census__", None)
+                by_rid = {rid: (region, spec) for rid, region, spec in work}
+                for rid, rows in answered_evidence.items():
+                    pair = by_rid.get(rid)
+                    if pair is None:
+                        continue
+                    region, spec = pair
+                    evidence = evidence_from_rows(
+                        rows,
+                        entry=_is_entry_point(region),
+                        # Rank 1.0 is the public tier. Provenance -- declared by a route contract versus
+                        # inferred from an absent decorator -- is not yet carried on ScanRegion, so this is
+                        # the proxy phase 0 has; the brief names it as the field to thread through next.
+                        access_declared_public=region.rank >= 1.0,
+                        bound_names=getattr(spec, "safe_shape_sinks", ()),
+                    )
+                    if evidence is not None:
+                        tiers.append((region.path, region.function or "", getattr(spec, "family", ""), evidence.tier))
+
     # Phase 2: ONE invocation per query kind. JVM startup dominates a repository scan -- a call per region
     # per family put a ten-line file at four minutes and a thousand regions at roughly fifty hours -- while
     # the queries themselves are milliseconds once the CPG is loaded.
     rows_by_id: dict[str, list[object]] = {}
-    per_kind: dict[str, float] = {}
     census_reported = False
     query_started = time.monotonic()
     for kind, requests in _grouped(work, hook_callbacks=hooks).items():
@@ -293,6 +336,8 @@ def scan_repository(
         query_seconds=query_seconds,
         query_seconds_by_kind=per_kind,
         degradations=tuple(degradations),
+        tiers=tuple(tiers),
+        tier_counts={tier: sum(1 for t in tiers if t[3] == tier) for tier in sorted({t[3] for t in tiers})},
     )