docs(flow-aware-ranking): brief -- rank pairs by evidence, in phases that each must prove themselves

A taint request costs 6.63s, a 637-file plugin yields 4,463 regions, and 83.8%
of them share one rank ordered alphabetically. The ranker is not weak, it is
looking at the wrong thing.

The evidence that this is tractable, from one plugin:

    getMemberOrderByCode  (holds CVE-2023-23488)  rank 0.30
        $wpdb->get_var("... WHERE code = '" . $code . "' LIMIT 1")
    getMemberOrderByID    (sibling)               rank 0.30
        $wpdb->get_row("... WHERE id = '$id' LIMIT 1")
    pmpro_rest_api_get_membership_level           rank 0.80
        no SQL sink at all

A region with no sink outranks two holding raw concatenated SQL. The signal is
in the sink call's own text -- interpolated versus bound -- and nothing in the
ranker reads it.

And the detail that decides whether a keyword rule suffices: `esc_sql` IS in
scope for the CVE's function, applied to a different statement. "A sanitizer is
present, rank it lower" demotes exactly the right answer. Knowing which call it
wraps is reading, not matching -- which is the case for a model rather than a
table.

Unit: (region, family) PAIRS, tiered by cheap evidence. Tier 0 (no sink of that
family in scope) is the only tier that excludes rather than orders, so it alone
must be exact; above it, being wrong costs position rather than the finding.
That bound is what makes a model safe here -- a ranker cannot manufacture a
finding.

Six phases, each shipping only if it beats the previous one on a repository it
was not developed against: baseline, exact pruning (the phase that makes a scan
finish), heuristic tiers, the LLM ranker, breaking the closed loop, then prompt
optimisation. Phase 2 carries an explicit possibility that the model is not
needed for tiering -- and knowing that is worth more than assuming it. Phase 5
comes last because an optimiser run against a censored metric optimises the
censorship.

Records what the prompt must carry, and that the question is never "is this
vulnerable" -- that is the arbiter's job and the one thing a ranker must not be
asked.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG

.kiro/specs/contributor-scan/tasks.md

@@ -897,7 +897,31 @@
     method-recogniser fixes moved it. The reason it is not reported is that the taint query TIMED OUT
     (`query_failed`, all 2,500 requests), so no region got a verdict, well ranked or not. Ranking is a real
     problem; it is not this miss.
-  - The fix is three layers, and the LLM is only the third:
+  - **Reframed 2026-09-10, and this is the shape the whole thing should take.** Pruning impossible questions
+    (5.14) is not a separate optimisation, it is TIER 0 of the ranker. The two are one mechanism at different
+    resolutions, and once that is seen the unit of ranking changes:
+
+    > **Rank (region, family) PAIRS, not regions — and rank them by evidence that a taint flow could exist.**
+
+    Today the ranker orders regions by declared access and then asks all five families of each, so a region
+    is examined for `deserialization` in a repository that never calls `unserialize`. The expensive question
+    is "does a source reach a sink of family F in this scope", and it costs 6.63 seconds. Every cheap
+    approximation of that question is available for free, from the fact tables and a text scan:
+
+    | tier | evidence | what it means |
+    |---|---|---|
+    | **0** | no sink of F in scope | **never ask** -- provably empty, cannot lose a finding |
+    | 1 | sink present, no source in scope or reachable | ask last |
+    | 2 | sink and source present, a sanitizer between them | may still corroborate |
+    | 3 | sink and source, no sanitizer, entry point or declared-public | ask first |
+
+    Tier 0 must be EXACT -- it is the only tier that excludes rather than orders, so it may only hold pairs
+    that provably cannot yield a flow. Everything above it is ordering, where being wrong costs position
+    rather than the finding.
+  - That makes the ranker a **cheap conservative approximation of the query it schedules**, which is the
+    right relationship: the ranker's job is to spend an expensive budget where the cheap evidence is
+    strongest, and its errors are bounded by construction because it cannot exclude anything above tier 0.
+  - The three layers below then describe HOW a tier is decided, not what the unit is:
     - **Facts, not code.** Framework vocabulary moves into the semantic tables the way `[[dispatch]]` already
       did today. An `[[access]]` fact declares hook-name patterns and their access level, so
       `wp_ajax_nopriv_*` is a row rather than an `if`. Covers every framework somebody has written facts for,

.kiro/specs/flow-aware-ranking/brief.md

@@ -0,0 +1,104 @@
+# Brief: flow-aware-ranking — spend the expensive query where the cheap evidence is strongest
+
+**Status:** brief only, unapproved.
+
+## Why now
+
+> "we need a taint flow aware ranker, that is capable to put high risk areas into tiers"
+> "can you engineer a prompt with enough information to steer the llm making ranker decisions"
+> "this can be done, but it requires an iterative concept and clear plans"
+
+A taint request costs **6.63 seconds** (contributor-scan 5.13). A 637-file plugin yields **4,463 regions**, and
+the current ranker gives 83.8% of them the same rank, ordered alphabetically by path. So the budget is spent
+arbitrarily and the scan does not finish.
+
+The ranker is not weak. It is looking at the wrong thing.
+
+## The evidence that this is tractable
+
+Three regions from one plugin, with what the ranker sees today and what is actually in the code:
+
+| region | rank today | sink in scope |
+|---|---|---|
+| `getMemberOrderByCode` — **holds CVE-2023-23488** | 0.30 | `$wpdb->get_var("... WHERE code = '" . $code . "' LIMIT 1")` |
+| `getMemberOrderByID` — sibling | 0.30 | `$wpdb->get_row("... WHERE id = '$id' LIMIT 1")` |
+| `pmpro_rest_api_get_membership_level` | **0.80** | none — no SQL sink at all |
+
+A region with no sink outranks two holding raw concatenated SQL. The discriminating signal is in the sink
+call's own source text — interpolated versus bound — and nothing in the ranker reads it.
+
+One detail decides whether a keyword rule suffices, and it does not: `esc_sql` IS in scope for the CVE's
+function, applied to a different statement. "A sanitizer is present, rank it lower" demotes precisely the
+right answer. Knowing which call it wraps is reading, not matching.
+
+## The unit, and the tiers
+
+**Rank (region, family) PAIRS, not regions.** The expensive question is "does a source reach a sink of family
+F here"; every cheap approximation of it is free from the fact tables and a text scan.
+
+| tier | evidence | |
+|---|---|---|
+| **0** | no sink of F in scope | **never ask** — exact, cannot lose a finding |
+| 1 | sink present, no source in scope or reachable | ask last |
+| 2 | sink and source, a cleansing call on THAT call | may still corroborate |
+| 3 | sink and source, interpolated shape, no cleansing on that call, entry point or declared public | ask first |
+
+Tier 0 is the only tier that excludes rather than orders, so it alone must be exact. Above it, being wrong
+costs position, not the finding. That bound is what makes a model safe here: **a ranker cannot manufacture a
+finding**, it decides what is looked at, and `regions_truncated` already reports what went unexamined.
+
+## The iterative concept
+
+Each phase ships only if it beats the previous one **on a repository the change was not developed against**.
+A phase that cannot show that is not an improvement, it is a story.
+
+- **Phase 0 — the oracle and the baseline.** The evidence extractor (no model) and a scoring harness over the
+  pinned CVEs, reporting each one's rank position. Gate: reproduces today's numbers — CVE-2023-23488 at
+  position 390 of 4,463, 83.8% of regions in one tier. Without a baseline nothing later is measurable.
+- **Phase 1 — tier 0 only, no model.** Exact pruning. Gate: 53% fewer taint requests, and `pmpro`'s taint
+  query COMPLETES. This is the phase that turns a scan that never finishes into one that does.
+- **Phase 2 — heuristic tiers, still no model.** Sink shape, cleansing on the call rather than in the file,
+  source presence, declared access. Gate: every pinned CVE's region inside the examined budget, measured on
+  all pinned repositories. **If this passes, the model may not be needed for tiering at all** — and knowing
+  that is worth more than assuming it.
+- **Phase 3 — the LLM ranker.** Same evidence, ordered by a model. Gate: beats phase 2 leave-one-repository-out.
+  If it does not, phase 2 ships and phase 3 does not.
+- **Phase 4 — break the closed loop.** An exploration slice (~10% of budget sampled outside the top-K) and
+  full labelling on repositories small enough to arbitrate every region. Gate: a measured estimate of the
+  selection bias in phase 3's numbers.
+- **Phase 5 — prompt optimisation (GEPA / SIMBA / DSPy).** Only after 3 and 4. An optimiser run against a
+  censored metric optimises the censorship.
+
+## What the prompt has to carry
+
+The question put to the model is **not** "is this vulnerable" — that is adjudication, it is the arbiter's job,
+and it is the one thing a ranker must never be asked. It is:
+
+> Given a fixed budget of expensive dataflow queries, which of these sites are worth spending it on?
+
+and the evidence it needs is textual and already available: the sink calls in scope **with their source
+text**, the untrusted-input expressions found, the cleansing calls and **which call each one wraps**, the
+sink's shape (bound versus interpolated), the declared access and its provenance, and whether the region is
+reachable from a request handler.
+
+## Constraints, from measurement and from prior scars
+
+- **Cost.** One model call per region is 4,463 calls a scan. Batch over a prefilter, or it does not ship.
+- **The loop is closed.** A region the ranker excludes is never arbitrated, never labelled, so the ranker
+  never learns it was wrong. Every run confirms the ordering it already had.
+- **N projects.** Four pinned CVEs is not a training set. This project already has the closed-loop
+  train-on-test scar; the reported figure must come from a repository the optimiser never saw.
+- **Determinism.** Two runs of one repository must agree or a baseline diff is meaningless. The ranking is
+  recorded in the manifest.
+- **Degradation.** With no model configured, phases 0–2 still rank, and the report says which ranker ordered
+  the scan.
+
+## Open questions for the design phase
+
+1. Where does the evidence extractor live — the mapper, or a new pass? It needs fact tables and file text but
+   no CPG, which suggests it belongs beside `mapping` rather than in `model/`.
+2. Is tier 0 computed per file (exact at `callDepth=0`) or per reachable set (exact at any depth)? The
+   repository-wide form — no sink of F anywhere — is exact at any depth and removes a fifth of the budget on
+   its own.
+3. How many pinned repositories are needed before leave-one-out means anything? Four is not enough.
+4. Does the arbiter's own verdict feed back as a label, and if so how is the selection bias corrected?

.kiro/specs/flow-aware-ranking/spec.json

@@ -0,0 +1,7 @@
+{
+  "name": "flow-aware-ranking",
+  "status": "brief",
+  "approved": false,
+  "summary": "Rank (region, family) pairs by cheap evidence of a possible taint flow, so an expensive query is spent where it can pay.",
+  "depends_on": ["contributor-scan"]
+}