diff --git a/.kiro/specs/contributor-scan/tasks.md b/.kiro/specs/contributor-scan/tasks.md index e7b9a38..8d0e4f8 100644 --- a/.kiro/specs/contributor-scan/tasks.md +++ b/.kiro/specs/contributor-scan/tasks.md @@ -897,7 +897,31 @@ method-recogniser fixes moved it. The reason it is not reported is that the taint query TIMED OUT (`query_failed`, all 2,500 requests), so no region got a verdict, well ranked or not. Ranking is a real problem; it is not this miss. - - The fix is three layers, and the LLM is only the third: + - **Reframed 2026-09-10, and this is the shape the whole thing should take.** Pruning impossible questions + (5.14) is not a separate optimisation, it is TIER 0 of the ranker. The two are one mechanism at different + resolutions, and once that is seen the unit of ranking changes: + + > **Rank (region, family) PAIRS, not regions — and rank them by evidence that a taint flow could exist.** + + Today the ranker orders regions by declared access and then asks all five families of each, so a region + is examined for `deserialization` in a repository that never calls `unserialize`. The expensive question + is "does a source reach a sink of family F in this scope", and it costs 6.63 seconds. Every cheap + approximation of that question is available for free, from the fact tables and a text scan: + + | tier | evidence | what it means | + |---|---|---| + | **0** | no sink of F in scope | **never ask** -- provably empty, cannot lose a finding | + | 1 | sink present, no source in scope or reachable | ask last | + | 2 | sink and source present, a sanitizer between them | may still corroborate | + | 3 | sink and source, no sanitizer, entry point or declared-public | ask first | + + Tier 0 must be EXACT -- it is the only tier that excludes rather than orders, so it may only hold pairs + that provably cannot yield a flow. Everything above it is ordering, where being wrong costs position + rather than the finding. + - That makes the ranker a **cheap conservative approximation of the query it schedules**, which is the + right relationship: the ranker's job is to spend an expensive budget where the cheap evidence is + strongest, and its errors are bounded by construction because it cannot exclude anything above tier 0. + - The three layers below then describe HOW a tier is decided, not what the unit is: - **Facts, not code.** Framework vocabulary moves into the semantic tables the way `[[dispatch]]` already did today. An `[[access]]` fact declares hook-name patterns and their access level, so `wp_ajax_nopriv_*` is a row rather than an `if`. Covers every framework somebody has written facts for, diff --git a/.kiro/specs/flow-aware-ranking/brief.md b/.kiro/specs/flow-aware-ranking/brief.md new file mode 100644 index 0000000..a3f2898 --- /dev/null +++ b/.kiro/specs/flow-aware-ranking/brief.md @@ -0,0 +1,104 @@ +# Brief: flow-aware-ranking — spend the expensive query where the cheap evidence is strongest + +**Status:** brief only, unapproved. + +## Why now + +> "we need a taint flow aware ranker, that is capable to put high risk areas into tiers" +> "can you engineer a prompt with enough information to steer the llm making ranker decisions" +> "this can be done, but it requires an iterative concept and clear plans" + +A taint request costs **6.63 seconds** (contributor-scan 5.13). A 637-file plugin yields **4,463 regions**, and +the current ranker gives 83.8% of them the same rank, ordered alphabetically by path. So the budget is spent +arbitrarily and the scan does not finish. + +The ranker is not weak. It is looking at the wrong thing. + +## The evidence that this is tractable + +Three regions from one plugin, with what the ranker sees today and what is actually in the code: + +| region | rank today | sink in scope | +|---|---|---| +| `getMemberOrderByCode` — **holds CVE-2023-23488** | 0.30 | `$wpdb->get_var("... WHERE code = '" . $code . "' LIMIT 1")` | +| `getMemberOrderByID` — sibling | 0.30 | `$wpdb->get_row("... WHERE id = '$id' LIMIT 1")` | +| `pmpro_rest_api_get_membership_level` | **0.80** | none — no SQL sink at all | + +A region with no sink outranks two holding raw concatenated SQL. The discriminating signal is in the sink +call's own source text — interpolated versus bound — and nothing in the ranker reads it. + +One detail decides whether a keyword rule suffices, and it does not: `esc_sql` IS in scope for the CVE's +function, applied to a different statement. "A sanitizer is present, rank it lower" demotes precisely the +right answer. Knowing which call it wraps is reading, not matching. + +## The unit, and the tiers + +**Rank (region, family) PAIRS, not regions.** The expensive question is "does a source reach a sink of family +F here"; every cheap approximation of it is free from the fact tables and a text scan. + +| tier | evidence | | +|---|---|---| +| **0** | no sink of F in scope | **never ask** — exact, cannot lose a finding | +| 1 | sink present, no source in scope or reachable | ask last | +| 2 | sink and source, a cleansing call on THAT call | may still corroborate | +| 3 | sink and source, interpolated shape, no cleansing on that call, entry point or declared public | ask first | + +Tier 0 is the only tier that excludes rather than orders, so it alone must be exact. Above it, being wrong +costs position, not the finding. That bound is what makes a model safe here: **a ranker cannot manufacture a +finding**, it decides what is looked at, and `regions_truncated` already reports what went unexamined. + +## The iterative concept + +Each phase ships only if it beats the previous one **on a repository the change was not developed against**. +A phase that cannot show that is not an improvement, it is a story. + +- **Phase 0 — the oracle and the baseline.** The evidence extractor (no model) and a scoring harness over the + pinned CVEs, reporting each one's rank position. Gate: reproduces today's numbers — CVE-2023-23488 at + position 390 of 4,463, 83.8% of regions in one tier. Without a baseline nothing later is measurable. +- **Phase 1 — tier 0 only, no model.** Exact pruning. Gate: 53% fewer taint requests, and `pmpro`'s taint + query COMPLETES. This is the phase that turns a scan that never finishes into one that does. +- **Phase 2 — heuristic tiers, still no model.** Sink shape, cleansing on the call rather than in the file, + source presence, declared access. Gate: every pinned CVE's region inside the examined budget, measured on + all pinned repositories. **If this passes, the model may not be needed for tiering at all** — and knowing + that is worth more than assuming it. +- **Phase 3 — the LLM ranker.** Same evidence, ordered by a model. Gate: beats phase 2 leave-one-repository-out. + If it does not, phase 2 ships and phase 3 does not. +- **Phase 4 — break the closed loop.** An exploration slice (~10% of budget sampled outside the top-K) and + full labelling on repositories small enough to arbitrate every region. Gate: a measured estimate of the + selection bias in phase 3's numbers. +- **Phase 5 — prompt optimisation (GEPA / SIMBA / DSPy).** Only after 3 and 4. An optimiser run against a + censored metric optimises the censorship. + +## What the prompt has to carry + +The question put to the model is **not** "is this vulnerable" — that is adjudication, it is the arbiter's job, +and it is the one thing a ranker must never be asked. It is: + +> Given a fixed budget of expensive dataflow queries, which of these sites are worth spending it on? + +and the evidence it needs is textual and already available: the sink calls in scope **with their source +text**, the untrusted-input expressions found, the cleansing calls and **which call each one wraps**, the +sink's shape (bound versus interpolated), the declared access and its provenance, and whether the region is +reachable from a request handler. + +## Constraints, from measurement and from prior scars + +- **Cost.** One model call per region is 4,463 calls a scan. Batch over a prefilter, or it does not ship. +- **The loop is closed.** A region the ranker excludes is never arbitrated, never labelled, so the ranker + never learns it was wrong. Every run confirms the ordering it already had. +- **N projects.** Four pinned CVEs is not a training set. This project already has the closed-loop + train-on-test scar; the reported figure must come from a repository the optimiser never saw. +- **Determinism.** Two runs of one repository must agree or a baseline diff is meaningless. The ranking is + recorded in the manifest. +- **Degradation.** With no model configured, phases 0–2 still rank, and the report says which ranker ordered + the scan. + +## Open questions for the design phase + +1. Where does the evidence extractor live — the mapper, or a new pass? It needs fact tables and file text but + no CPG, which suggests it belongs beside `mapping` rather than in `model/`. +2. Is tier 0 computed per file (exact at `callDepth=0`) or per reachable set (exact at any depth)? The + repository-wide form — no sink of F anywhere — is exact at any depth and removes a fifth of the budget on + its own. +3. How many pinned repositories are needed before leave-one-out means anything? Four is not enough. +4. Does the arbiter's own verdict feed back as a label, and if so how is the selection bias corrected? diff --git a/.kiro/specs/flow-aware-ranking/spec.json b/.kiro/specs/flow-aware-ranking/spec.json new file mode 100644 index 0000000..b4ec0ae --- /dev/null +++ b/.kiro/specs/flow-aware-ranking/spec.json @@ -0,0 +1,7 @@ +{ + "name": "flow-aware-ranking", + "status": "brief", + "approved": false, + "summary": "Rank (region, family) pairs by cheap evidence of a possible taint flow, so an expensive query is spent where it can pay.", + "depends_on": ["contributor-scan"] +}