From c03944f7dd9a4565f3f4a4cdfc619ce08ee7ca5d Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Thu, 10 Sep 2026 08:49:07 +0200 Subject: [PATCH] docs(benchmarks): the whole-plugin envelope, next to the slice results The class-by-class table is measured on one- and two-file slices, which invites the reading that the engine scans WordPress plugins. It does not, and the README should not let that stand unqualified. 637 files, 117k lines: the build is affordable (33s, 1.2GB, a 1.6MB CPG) and a CI runner can do it. The scan is not -- 996.6s to decide nothing, with the taint query dying in ForkJoinPool and 500 of 4,463 regions examined. What the run does correctly is refuse to look clean: files_unparsed, query_failed, cpg_sharded and the region cap all reach the reader. A scan that decides nothing and says so is a different artifact from one that decides nothing quietly. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG --- benchmarks/repos/README.md | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/benchmarks/repos/README.md b/benchmarks/repos/README.md index 4c9a920..0592f39 100644 --- a/benchmarks/repos/README.md +++ b/benchmarks/repos/README.md @@ -200,6 +200,40 @@ That is the argument for why a corpus of benchmarks is not enough on its own, ma stayed byte-identical through all of it. A corpus tells you a verdict is right; it cannot tell you the verdict was reached by looking at anything. +## The whole-plugin envelope + +The class-by-class table above is measured on slices — one or two files carrying both ends of a CVE. The +obvious question is whether any of it survives on the whole plugin. Measured on `pmpro`, 637 files and +117k lines: + +| | | +|---|---| +| preprocess + entry-point mapping | 107s → 673 targets, 4,412 entry points, **4,463 regions** | +| frontend build, one attempt | 33s, peak RSS 1,217 MB, a 1.6 MB CPG | +| full scan: build / query / arbitrate | 255.9s / 720.9s / 19.8s — **996.6s total**, peak RSS 1,132 MB | +| regions examined | **500 of 4,463 (11%)** — the default cap | +| findings | **0** | +| CVE-2023-23488 | **not reported** | + +The build is affordable; a CI runner can do this without a vertical-scale machine, which was the open +question. The scan is not: the taint query dies in `ForkJoinPool` after ten minutes, and so does a bare +`cpg.file.size` census, so it is the CPG under a 2 GB heap rather than the request payload. + +So the honest statement of PHP detection today is that **the slices find the CVEs and the repository does +not** — one file finds CVE-2023-23488, two files find it with a shard, and 637 files find nothing at all. +No release claim about scanning a WordPress plugin can rest on this. + +What the run does do correctly is refuse to look clean. It reports `files_unparsed`, `query_failed` for +2,500 regions, `cpg_sharded`, and the region cap that hid 89% of the repository — all of which appear in +the report under "What could not be analysed". A scan that decides nothing and says so is a different +artifact from a scan that decides nothing quietly, and the whole point of the rung ladder is that the +second one must not be possible. + +It also found two defects that slice-scale measurement never would have. `hookCallbacks` describes the +repository rather than the region and was being repeated into every request — 42.5 MB of which 41.7 MB was +one string, now 1.78 MB. And candidates were enumerated for a judge that was never going to be asked, which +was 737 of the 996 seconds; arbitration is now 19.8s. + ## Baselines Task 6.1 turns a scan of a pinned checkout into a committed baseline, so a change to the fact tables or the