From f3f41e2fa90b06466ba29efa8c099e763d1ee9fa Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Thu, 10 Sep 2026 09:06:03 +0200 Subject: [PATCH] docs: the 637-file wall is the taint query, not memory and not vendored code Follow-up measurements, because the obvious explanations were all wrong. Not memory: at 2GB AND 4GB the CPG never finishes loading, with no OutOfMemoryError either time, both runs stopping in the same place -- Braintree\Util. What costs is Joern's dataflow overlay, recomputed per import. Not scope, though scope is real and worth fixing on its own terms: 429 of the 637 files are vendored SDKs (Braintree, Stripe, each with its own LICENSE) and 31 are tests. PMPro's own code is 177 files, 62k lines. Auditing a plugin by scanning the payment SDK it bundles is wrong regardless of cost -- those findings are not the plugin's bugs -- and it is the same idea shipped.py already applies to Makefile.am, which PHP has no equivalent of. Scoping to authored code helps and does not fix it: regions 4,463 -> 1,324 total 996.6s -> 843.7s findings still 0 600.3s of that 843.7s is the taint query hitting QUERY_TIMEOUT_SECONDS (300s) on each of two shards. Two timeouts, not work. So the wall is the per-request cost of the taint query at 2,500 requests. The next measurement is what the two-stage joins cost at scale -- 1.6x on a slice (62s -> 97s) and plausibly the dominant term here. Raising the timeout before knowing that would only buy a slower way to find out. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG --- .kiro/specs/contributor-scan/tasks.md | 22 ++++++++++++++++++++-- benchmarks/repos/README.md | 18 ++++++++++++++++-- 2 files changed, 36 insertions(+), 4 deletions(-) diff --git a/.kiro/specs/contributor-scan/tasks.md b/.kiro/specs/contributor-scan/tasks.md index ba05545..d6458d9 100644 --- a/.kiro/specs/contributor-scan/tasks.md +++ b/.kiro/specs/contributor-scan/tasks.md @@ -656,8 +656,26 @@ - Two defects the measurement exposed and fixed along the way -- both mine, both invisible at slice scale: `hookCallbacks` was repeated into every request (42.5MB of which 41.7MB was one string), and candidates were enumerated for a judge that was never going to be asked (737s -> 19.8s of arbitration). - - Still open: the heap a 637-file PHP CPG actually needs; whether the query survives with it; and nothing - here recovers a cross-shard flow. + - **It is not a heap problem, and it is not the payload.** At 2GB *and* 4GB the CPG never finishes + loading -- no `OutOfMemoryError`, both runs hit a 900s ceiling, and both died in the same place, + `Braintree\Util`. What costs is Joern's dataflow overlay, recomputed on every import. + - **Two thirds of that repository is not its own code.** Of 637 files, **429 are vendored SDKs** (Braintree + and Stripe, each shipping its own LICENSE and README) and 31 are tests; PMPro's own code is **177 files, + 62k lines**. Auditing a plugin by scanning the payment SDK it bundles is wrong on its own terms -- those + findings are not the plugin's bugs -- and it is what makes the graph unaffordable. This is the same idea + `shipped.py` applies to `Makefile.am`, which PHP has no equivalent of. + - **Scoping to authored code helps and does not fix it.** 177 files: + + regions 4,463 -> 1,324 total 996.6s -> 843.7s peak RSS 1,132 -> 1,316 MB + findings 0 CVE-2023-23488 still not reported + + The taint query hits `QUERY_TIMEOUT_SECONDS` (300s) on **each** shard -- 600.3s of the 843.7s is two + timeouts, not work. So the wall is the per-request cost of the taint query at 2,500 requests, and + nothing about memory, the build, or the vendored code removes it. + - The next measurement is therefore the per-request cost itself, and specifically what the two-stage joins + cost at scale: they were 1.6x on a slice (62s -> 97s) and are plausibly the dominant term here. Until + that is known, raising the timeout would only buy a slower way to find out. + - Still open: that measurement; and nothing here recovers a cross-shard flow. - Still open: wiring that exclusion loop, the 637-file build with it in place, and whether the REST handler two calls away can reach the sink -- the slice found it through a same-function flow, not across the object. diff --git a/benchmarks/repos/README.md b/benchmarks/repos/README.md index 0592f39..206bd6b 100644 --- a/benchmarks/repos/README.md +++ b/benchmarks/repos/README.md @@ -216,8 +216,22 @@ obvious question is whether any of it survives on the whole plugin. Measured on | CVE-2023-23488 | **not reported** | The build is affordable; a CI runner can do this without a vertical-scale machine, which was the open -question. The scan is not: the taint query dies in `ForkJoinPool` after ten minutes, and so does a bare -`cpg.file.size` census, so it is the CPG under a 2 GB heap rather than the request payload. +question. The scan is not. + +It is worth being precise about *why*, because the obvious answers are all wrong. It is not memory: at 2 GB +and at 4 GB the CPG never finishes loading, with no `OutOfMemoryError` in either case, both runs stopping in +the same place — `Braintree\Util`. It is not the request payload either. What costs is Joern's dataflow +overlay, recomputed on every import, over a graph two thirds of which is not the plugin's code at all: +**429 of the 637 files are vendored SDKs** (Braintree, Stripe) and 31 are tests, leaving 177 files of PMPro's +own. Auditing a plugin by scanning the payment SDK it bundles is wrong on its own terms. + +Scoping to those 177 files helps and does not fix it: + + regions 4,463 -> 1,324 total 996.6s -> 843.7s findings still 0 + +600.3s of that 843.7s is the taint query hitting its 300s timeout on each of two shards — two timeouts, not +work. The wall is the per-request cost of the taint query, and neither memory, nor the build, nor dropping +the vendored code moves it. So the honest statement of PHP detection today is that **the slices find the CVEs and the repository does not** — one file finds CVE-2023-23488, two files find it with a shard, and 637 files find nothing at all.