pmpro, 637 files, the product's own build path after 60963f5: build 97 s with overlays, evidence 76 s,
taint 4,105 s for 348 real requests (11.8 s each), arbitrate 99 s, peak child RSS 2.4 GB. The CVE is at
class.memberorder.php:936 among 250 entailed findings across 40 files -- the number the flow-aware-ranking
spec now has to concentrate. callDepth measured on the same graph: a 1.4x time term and a 100x payload
term (3,838 rows for ten requests at depth 3 against 38 at depth 1).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
A pair with no sink of its family in reach cannot yield a flow, in any
language under any dataflow model, so its taint request is never issued. The
count is recorded as requests_pruned -- exact, so not a degradation -- and a
test asserts that every other pair's request still goes out.
637-file plugin, 500-region budget
tier_counts {0: 2161, 1: 60, 2: 20, 3: 211, 4: 48}
pruned 2,161 of 2,500 = 86% evidence pass 94.5s
taint 339 requests -> 2,400s TIMEOUT
Gate met and exceeded; the scan still does not finish, and the reason corrects
phase 0. "1,148 ms per request" was an average over a 250-request set that was
88% tier 0 and answered instantly; the ~30 real requests in it cost ~9.5s each.
Pruning removes exactly the cheap questions and keeps the expensive ones, so it
cannot make them faster. The honest cost of a taint question that CAN have an
answer is 7-10s at callDepth=3, and that is what phase 2 and 5.13 design
against -- not 1.1s, and not 6.6s either. Both specs corrected.
869 tests pass.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
flow-aware-ranking phase 0: measure before changing anything.
THE BASELINE, committed as benchmarks/ranking/baselines/2026-09-10-access-rank.json
from benchmarks/ranking/position.py, which sorts regions exactly as the scan does:
pmpro 4,463 regions 83.8% at 0.30 CVE at 390 budget_at_recall 391
mwwpform 983 regions 81.7% at 0.30 CVE at 512 budget_at_recall 513
wpstatistics 1,673 regions 75.3% at 0.30 CVE at 503 budget_at_recall 504
vampi 25 regions BOLA at 15, 10; SQLI reached transitively
That corrects the brief's own cross-assessment: "concentration, not inclusion"
was measured on pmpro alone. On the other two plugins the CVE sits OUTSIDE a
500-region budget and is never examined; the slices found those CVEs only
because a slice has too few regions to exclude anything.
THE INSTRUMENT: `evidenceOnly` mode on the taint query -- what is in scope,
which sink calls, what shape, whether a sanitizer wraps THEIR argument, whether
a source is local or reachable -- returning before reachableByFlows is ever
called. model/evidence.py turns its rows into a vector per (region, family)
and a tier; scan.py records every pair's tier on the result and acts on none of
it. Every later phase is measured against a record made by scans that did not
know they were being measured.
2,500 pairs in 98.9s incl. JVM load, ~40 ms per pair
tier 0 -- no sink of that family in reach: 2,161 of 2,500 = 86%
Two earlier attempts ran past an hour each with no payload, and both were my
fault: repository-wide walks repeated per request -- the sink scan, the
reachable-method BFS, familyInRepo. Memoised across the batch.
AND THE SAME FIX REACHES THE DATAFLOW MODE. sinkCalls and reachableMethods do
not depend on the family, and were recomputed for all 2,500 requests. The
identical 250-request set with both joins on:
2,026.3s -> 287.0s 8,105 -> 1,148 ms per request same 5,779 rows
Most of what 5.13 measured as "the base taint query" was never dataflow. A 7x
speedup with nothing about the analysis changed; both corrections recorded in
the specs. Combined with 86% tier 0, the whole-repository taint query that
timed out at 40 minutes is ~339 real requests at ~1.1s: about six minutes.
Also: the module audit manifest is regenerated (91 load-bearing, 7 standalone,
0 orphaned) because evidence.py is now imported by the scan, and the
failed-query test is order-independent since the evidence pass records its own
failure before taint's.
869 tests pass; query compile test 4 passed; gates byte-identical.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Read against today's measurements the brief had two errors and one gap. They
are corrected in place and kept on the record, because the corrections are
themselves results.
Phase 1's gate was wrong. Pruning removes 53% of requests, and 1,177 x 6.63s is
still 2.2 hours -- it halves the problem and does not finish the scan. Saying
otherwise would send someone looking for a bug in a phase that did exactly what
it can.
Ranking's job is concentration, not inclusion. CVE-2023-23488's region is at
position 390, inside the budget, examined. It goes unreported because 500
regions are unaffordable at 6.63s per family per region, not because it was
left out. The metric is therefore budget_at_recall -- the smallest K holding
every pinned CVE -- and phase 2's gate is that K below 50.
The gap: the heuristics were named, not defined, and an undefined heuristic
cannot be checked for language leakage. Now defined as an evidence vector over
(region, family) pairs whose every field is stated in terms of fact-table
categories -- source, sink, sanitizer, dispatch, access -- so a heuristic that
mentions `$wpdb` has by construction left the vector.
The architectural decision that makes it agnostic rather than merely
neutral-in-intent: the vector is a CPG query WITHOUT dataflow, a sibling of
taint.sc that never calls reachableByFlows. Tiering is the CPG query without
dataflow; arbitration is the query with it. Same graph, same facts, two costs.
The graph normalises syntax across frontends, so "this argument is a literal"
means the same thing in PHP, Python and C.
Five tiers as predicates over the vector. Only tier 0 excludes, and its
predicate has a proof: a family with no sink in reach cannot produce a flow in
any language under any dataflow model. Its scope must match the arbiter's
callDepth or it is not exact and must not be called tier 0. Everything above
orders, where being wrong costs position, which regions_truncated reports.
Where the model helps is narrower than "ranking": ambiguous cleansing (esc_sql
two statements earlier into a variable), unfamiliar sink shapes as an ordering
opinion never a fact, and within-tier ties. Where it must not be allowed:
computing tier 0, seeing the question "is this vulnerable", or seeing the
arbiter's own verdict on the same run.
Six metrics so every phase can fail, and budget_at_recall is the one that
decides the release question: below 50 and the model is a refinement, above
and the model is the plan.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
A taint request costs 6.63s, a 637-file plugin yields 4,463 regions, and 83.8%
of them share one rank ordered alphabetically. The ranker is not weak, it is
looking at the wrong thing.
The evidence that this is tractable, from one plugin:
getMemberOrderByCode (holds CVE-2023-23488) rank 0.30
$wpdb->get_var("... WHERE code = '" . $code . "' LIMIT 1")
getMemberOrderByID (sibling) rank 0.30
$wpdb->get_row("... WHERE id = '$id' LIMIT 1")
pmpro_rest_api_get_membership_level rank 0.80
no SQL sink at all
A region with no sink outranks two holding raw concatenated SQL. The signal is
in the sink call's own text -- interpolated versus bound -- and nothing in the
ranker reads it.
And the detail that decides whether a keyword rule suffices: `esc_sql` IS in
scope for the CVE's function, applied to a different statement. "A sanitizer is
present, rank it lower" demotes exactly the right answer. Knowing which call it
wraps is reading, not matching -- which is the case for a model rather than a
table.
Unit: (region, family) PAIRS, tiered by cheap evidence. Tier 0 (no sink of that
family in scope) is the only tier that excludes rather than orders, so it alone
must be exact; above it, being wrong costs position rather than the finding.
That bound is what makes a model safe here -- a ranker cannot manufacture a
finding.
Six phases, each shipping only if it beats the previous one on a repository it
was not developed against: baseline, exact pruning (the phase that makes a scan
finish), heuristic tiers, the LLM ranker, breaking the closed loop, then prompt
optimisation. Phase 2 carries an explicit possibility that the model is not
needed for tiering -- and knowing that is worth more than assuming it. Phase 5
comes last because an optimiser run against a censored metric optimises the
censorship.
Records what the prompt must carry, and that the question is never "is this
vulnerable" -- that is the arbiter's job and the one thing a ranker must not be
asked.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
