`Evidence.score` orders pairs within a tier by one table of weights (open sinks, cleansed sinks, local /
carried / near sources, entry, declared public), capped so volume stops counting, and zero for tier 0 so
the weights can rank but never manufacture. `ScanBudget.order_by_evidence` runs the evidence pass over
every region and cuts `max_regions` from each region's best (tier, score); `order_by_evidence()` is the
one ordering function, used by the scan and by `benchmarks/ranking/position.py --evidence`. Off by
default until the position benchmark says where the pinned CVEs land under it.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
A pair with no sink of its family in reach cannot yield a flow, in any
language under any dataflow model, so its taint request is never issued. The
count is recorded as requests_pruned -- exact, so not a degradation -- and a
test asserts that every other pair's request still goes out.
637-file plugin, 500-region budget
tier_counts {0: 2161, 1: 60, 2: 20, 3: 211, 4: 48}
pruned 2,161 of 2,500 = 86% evidence pass 94.5s
taint 339 requests -> 2,400s TIMEOUT
Gate met and exceeded; the scan still does not finish, and the reason corrects
phase 0. "1,148 ms per request" was an average over a 250-request set that was
88% tier 0 and answered instantly; the ~30 real requests in it cost ~9.5s each.
Pruning removes exactly the cheap questions and keeps the expensive ones, so it
cannot make them faster. The honest cost of a taint question that CAN have an
answer is 7-10s at callDepth=3, and that is what phase 2 and 5.13 design
against -- not 1.1s, and not 6.6s either. Both specs corrected.
869 tests pass.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
flow-aware-ranking phase 0: measure before changing anything.
THE BASELINE, committed as benchmarks/ranking/baselines/2026-09-10-access-rank.json
from benchmarks/ranking/position.py, which sorts regions exactly as the scan does:
pmpro 4,463 regions 83.8% at 0.30 CVE at 390 budget_at_recall 391
mwwpform 983 regions 81.7% at 0.30 CVE at 512 budget_at_recall 513
wpstatistics 1,673 regions 75.3% at 0.30 CVE at 503 budget_at_recall 504
vampi 25 regions BOLA at 15, 10; SQLI reached transitively
That corrects the brief's own cross-assessment: "concentration, not inclusion"
was measured on pmpro alone. On the other two plugins the CVE sits OUTSIDE a
500-region budget and is never examined; the slices found those CVEs only
because a slice has too few regions to exclude anything.
THE INSTRUMENT: `evidenceOnly` mode on the taint query -- what is in scope,
which sink calls, what shape, whether a sanitizer wraps THEIR argument, whether
a source is local or reachable -- returning before reachableByFlows is ever
called. model/evidence.py turns its rows into a vector per (region, family)
and a tier; scan.py records every pair's tier on the result and acts on none of
it. Every later phase is measured against a record made by scans that did not
know they were being measured.
2,500 pairs in 98.9s incl. JVM load, ~40 ms per pair
tier 0 -- no sink of that family in reach: 2,161 of 2,500 = 86%
Two earlier attempts ran past an hour each with no payload, and both were my
fault: repository-wide walks repeated per request -- the sink scan, the
reachable-method BFS, familyInRepo. Memoised across the batch.
AND THE SAME FIX REACHES THE DATAFLOW MODE. sinkCalls and reachableMethods do
not depend on the family, and were recomputed for all 2,500 requests. The
identical 250-request set with both joins on:
2,026.3s -> 287.0s 8,105 -> 1,148 ms per request same 5,779 rows
Most of what 5.13 measured as "the base taint query" was never dataflow. A 7x
speedup with nothing about the analysis changed; both corrections recorded in
the specs. Combined with 86% tier 0, the whole-repository taint query that
timed out at 40 minutes is ~339 real requests at ~1.1s: about six minutes.
Also: the module audit manifest is regenerated (91 load-bearing, 7 standalone,
0 orphaned) because evidence.py is now imported by the scan, and the
failed-query test is order-independent since the evidence pass records its own
failure before taint's.
869 tests pass; query compile test 4 passed; gates byte-identical.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Paid Memberships Pro yields 4,463 regions from 637 files. The default budget
examines 500 of them -- 11% of the repository -- and nothing in the output
said so. A report whose silence covers the other 89% invites that silence to
be read as a clean bill of health.
`budget_exhausted` is a different fact and already reported: that is the JUDGE
running out on work already collected. This is the work never being collected.
Renders in "What could not be analysed" as: only the N highest-ranked of M
regions were examined, and nothing above says anything about the rest.
850 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 5.7, and it was the wrong question.
**The 117k-line pmpro CPG does not load.** Importing it throws in a
ForkJoinParallelCpgPassWithAccumulator during overlay application, so every
query against it returns nothing. It is not a call-graph question at all.
Isolated by running the ARBITER against two prebuilt CPGs rather than
writing a sixth ad-hoc Scala probe, four of which had already failed on my
own mistakes:
small wpdb CPG get_order_by_code -> 1 model_entailed
pmpro CPG getMemberOrderByCode -> 0
Both have their source and sink in ONE function, both were asked with
parameter sources and depth 3, same arbiter and same spec. The CPG is the
only difference.
That clears the facts, which is worth stating: `$wpdb` sinks work, the
parameter-source rule works, and a PHP parameter reaching `$wpdb->get_var`
through a concat is entailed. Whether php2cpg resolves `new X()` and
`$this->method()` is still unmeasured -- it cannot be measured on a CPG
that will not load -- and is recorded as 5.10 along with finding the size
boundary.
**And the defect underneath it, which is the fourth quiet failure found on
this one checkout.** `query_batch` returned `{}` for a failure and `{}` for
a genuine empty answer, so a caller could not tell "could not ask" from
"nothing found". Every query on this scan failed at load and it reported
500 regions examined, no findings, and no degradation: a vulnerable plugin,
clean.
A failed batch now returns None, unparseable output counts as a failure
rather than an empty answer, and the driver records a `query_failed`
degradation naming the kind and the request count.
This is the failure mode the rung ladder structurally cannot catch. Every
other honesty mechanism here labels a VERDICT -- suspicion,
regions_unjudged, regions_unasked, stated family limits -- and when no
verdict is produced there is nothing to label.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 2.9. Task 2.4 let a region's question follow the call
graph, and the moment it did, many entry points could reach one shared
sink: libpng reported the same site twenty-six times. A contributor shown
the same defect twenty-six times learns to scroll past it.
Findings are deduplicated on (site, family), keeping the strongest rung
and counting the regions that reached it -- "reached from 26 regions" is
the useful half of that observation and one row is the readable half. Two
families at one site stay two findings, because `echo $_GET[...]` is an
injection question and an output-encoding question with different answers.
The duplication that motivated this was removed by another route: the 26
rows were all the `fgets`-matched-as-`gets` false positive, and the sink
matcher fix took them out. So this changes no current number and is a
structural guarantee rather than a measured reduction -- the mechanism that
produced them is untouched, and only the finding that made it visible is
gone.
+25 lines for a guarantee, in a session that has been asked to watch its
growth. The audit is unchanged at 97 modules, 90 load-bearing and 7
standalone, with no orphans.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 2.2 for libpng and the 2.3 decision it exists to inform.
89,082 lines of C across 93 files, under a 2GB heap cap:
CPG build 16.1s (25s cold)
Joern queries 250.0s
arbitration 122.0s (105 calls, $0.0122)
whole scan 446s
peak RSS 911MB
regions 121, all arbitrated, 32 unasked
CVE-2025-66293 out_of_scope, as the recipe declares
Memory is not the constraint a fleet runner would hit: 911MB leaves room
for the rest of the job on a 2-vCPU/7GB runner. The memory pressure seen
while building this was concurrent JVMs on one desktop, not repository
size.
**Server mode: no.** The query phase splits taint 243.3s / config 10.7s
over two invocations, and fixed overhead measured on the same CPG is ~13s
per invocation. So ~26s of the 250s is JVM startup and ~224s is CPGQL
evaluation: a warm server removes 6% of the scan and none of the part that
costs. The driver now records query seconds per kind, because a single
total cannot tell startup from evaluation and the two answer this
oppositely.
The small repository pointed the other way. vampi's 34.3s over three kinds
is roughly three startups and almost no evaluation, which made a warm JVM
look worth having; the ratio inverts with size. That inversion is why the
task was sequenced after a real repository rather than decided on a
520-line file.
On whether one whole-repository CPG is the only shape for a horizontal
fleet: no, and it is also not where the pressure is. The build
parallelises and costs 16s; the query does not shard and costs 250s, so
splitting a repository across runners divides the cheap phase -- and after
2.4 a shard boundary hides exactly the cross-module flows that matter.
Scoping to a contributor's diff cuts the expensive phase instead. 911MB
for 89k lines extrapolates to roughly 5GB at a million lines, which would
exceed a standard runner, so the ceiling is real and further out than this
checkout can locate.
Two defects recorded rather than fixed, both invisible at vampi scale:
2.9, all 26 entailed findings are the same site reported 26 times, because
regions that share a sink each emit their own; and 2.10, 22 of the 30
highest-ranked regions are libpng's `contrib/` example programs rather than
the library.
Also ignores Joern's `workspace/`, which a maintainer querying a CPG by
hand from the repository root will create.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 2.4, closing the no-go task 2.2 recorded.
VAmPI's SQL injection is a request parameter reaching `get_by_username`
in api_views/users.py, passed to `User.get_user` in models/user_model.py,
interpolated into a query string and executed there. Source and sink are
in different functions and different modules. Regions were per-function,
so neither region saw both ends and the engine was silent on the most
ordinary shape a real injection bug has.
`reachableByFlows` was never the limitation -- it follows calls already.
The limitation was the QUESTION. Sinks were filtered to the labeled
function's own body, so a crossing flow had no sink to end at.
Three changes, each with a reason it is sound rather than convenient:
**An entry point is the trust boundary.** parameter_sources was off for
repository scans because a repository's parameters are not all attacker
input. An entry point's are, by definition -- that is what makes it an
entry point. The driver turns it on only for regions the mapper found and
named, and leaves it off for walker fallbacks and for named functions that
are not entry points.
**Sinks may sit below the handler.** taint.sc admits sinks in methods
reachable from the labeled method through the call graph, bounded at
ENTRY_POINT_CALL_DEPTH = 3: handler -> service -> data access, which is
where these bugs live. Unbounded would make the question about most of the
repository rather than about this entry point, and make it unaffordable.
**The finding names the sink, not the asker.** A crossing flow ends in
another file and another function, so Verdict carries the evidence's own
`path:line:function` and the site uses it. Reporting
`models/user_model.py:73:get_by_username` would name a place that does not
exist: line 73 is inside get_user.
Also: labeled methods are now resolved by name AND file. `get_user` exists
in more than one module of a real repository, which a corpus of one-file
excerpts could never surface.
Measured on the same checkout and budget as 2.2:
VAmPI-SQLI missed -> model_entailed at models/user_model.py:73:get_user
known found 0 of 2 before 2.2's fixes, 1 of 2 in 2.2, 2 of 2 now
entailed 7 -> 8
query time 40.9s -> 37.4s
peak RSS 869MB -> 827MB
cost $0.0067, unchanged
Asking the bigger question cost nothing measurable here: file scoping and
call-graph following pull in opposite directions and roughly cancel at this
size. That is one 520-line repository, and VAmPI is now the repository this
change was developed against -- the exact overfitting shape Req 8 exists to
catch. The depth bound and the entry-point rule mean nothing general until
a checkout nobody tuned against says so.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 2.2. VAmPI at a pinned commit, scanned whole rather than
as excerpts, with benchmarks/repos/measure.py recording build time, query
time, arbitration time, peak memory, cost, unjudged regions, and whether
each known vulnerability was found and at which rung.
The corpus could not have found any of this. Every defect below needs a
repository to become visible, and three of them were invisible behind a
passing test suite.
**The batch path was unreachable.** query_batch was implemented, tested and
committed while CpgResult exposed only `run`, so the driver's capability
check found nothing and fell back to one JVM per region-family: the first
attempt at this measurement ran eighteen minutes on 520 lines, issuing
`--param function=` one at a time. Every test called query_batch directly,
so none of them noticed. CpgResult now carries run_batch and a test asserts
the capability is reachable through the object the driver is handed.
The earlier 275s->106s improvement attributed to batching came from region
deduplication alone.
**The budget stopped free work.** It bounds model calls, but was enforced
by breaking the region loop -- spending a paid limit to stop unpaid CPG
arbitration. 6 of 22 regions were examined and VAmPI's BOLA, which the
graph entails for nothing, was never reached. It now withholds the judge
only. 22 of 22 regions arbitrated, 17 counted as regions_unasked (silent
because unaffordable is not silent because decided), and the BOLA is
entailed by the model layer.
**A region with no function asked about the whole repository.** Sending
function="" matched every hit in the tree and the driver attributed that
answer to whoever asked: one `host='0.0.0.0'` in app.py became eight
identical entailed findings in eight files that do not contain it. Queries
now carry the region's file. Entailed findings 14 -> 7, query time
70.9s -> 40.9s. The single-request form stays unscoped so the committed
pair measurements remain reproducible.
**The JVM had no heap bound.** Joern claimed a quarter of physical RAM and
runs were killed under memory pressure at different points, so the numbers
described the host rather than the engine. -J-Xmx2048m by default,
OPENULTRASAST_CPG_HEAP_MB to raise it.
Also: arbitrated findings carried `function:line` where the report layer
read `path:line`, so every entailed finding reached a contributor with a
function name where its file belonged; and the witness line was parsed by
splitting on a comma, which produced sites like
`config.py:17): permissive literal '0.0.0.0':?`.
Reading: GO on one CPG per repository -- 6.2s to build, 177s end to end,
869MB, $0.0067, nothing infeasible. NO-GO on region-scoped taint as the
only shape: VAmPI's SQL injection runs from a handler parameter into a
function in another module, and a per-function region sees neither end.
That is the ordinary shape of a real injection bug, so it is recorded as
task 2.4 along with three smaller defects this run exposed.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
JVM startup dominated a scan: a call per region per family put a ten-line
file at 275s over three regions, and would have put a thousand regions at
roughly fifty hours. The queries themselves are milliseconds once the CPG
is loaded, so the cost was almost entirely process starts.
scan_repository() is now collect -> batch -> arbitrate. The three .sc
scripts accept a `requests` JSON map and answer all of them in one run,
keeping the single-request form so the committed measurements stay
reproducible byte-for-byte. Arbiters are untouched: each is handed a
CpgResult whose `run` returns its own prefetched rows.
The batcher builds parameters by calling request_params() extracted from
taint.py, dominance.py and config_value.py rather than reconstructing
them. A second copy that drifts from the arbiter's is the exact shape of
the five family-assumption bugs the predecessor spec spent a session
finding; there is now one definition per family and the batcher borrows
it.
Regions are also deduplicated by (path, function), collapsing the
nameless entry-point rows that were producing `model:injection:?:10`.
Measured on the same ten-line file: 275s -> 106s, three duplicate
findings -> one `model:injection:ping:10`, rung unchanged at
model_entailed. Gates byte-identical.
Server mode is recorded as task 2.3, deliberately sequenced after the
repository measurement: whether the remaining ~100s floor is worth a
process model, a weakened subprocess seam and a Scala-evaluating HTTP
endpoint depends on a build-time-to-query-time ratio only 2.2 produces.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Pair scoring could rely on a labelled function; a contributor cannot supply one
-- which code is risky is the question they are asking. Regions now come from
entry points, with a file-level fallback that says it is one, and each carries
only the families its shape and language admit. Ranking uses what
EntryPointRecord already decided (access_level, trust_boundary) rather than a
second opinion computed from the same evidence.
A family is offered only where its arbiter could answer: access_control needs a
handler, so a library file with no entry point is not asked about obligations it
has no siblings to be inconsistent with.
The driver makes the two costs that were free at pair scale explicit. ONE CPG
per repository, reused across regions -- a build is 5-30s on a forty-line
excerpt and per-region builds would make this unusable. And a budget that is a
TOTAL for the run rather than the enumerator's per-region cap of eight, spent
highest-rank-first, with whatever it does not reach counted into
regions_unjudged. A tool that silently analyses a tenth of a repository and
reports a clean result is worse than one that says what it skipped.
Two flaws the tests caught, both mine and both about ownership: the budget
counted calls by reading a client's own usage log, so it silently never tripped
for a client that keeps none -- the driver now counts its own spend through a
wrapper; and the driver trusted its caller to have sorted, when the order the
budget is spent in belongs to whoever holds the budget.
The third was worse and only surfaced because a test failed for the right
reason: the driver passed candidates=() and never enumerated, so a repository
scan would have reported only what the arbiters establish unaided and never
populated the suspicion band -- which supplied 15 of 26 findings in the pair
measurement. It enumerates now.
Sibling audit (Req 8.5) done before writing: family routing lives in exactly one
production place, pipeline._arbitrate, dispatching on spec type. regions_for
returns family names and the driver resolves specs, so that stays true.
750 passing, mypy clean over 96 files, gates byte-identical, zero orphans.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Marius Ciepluch