docs(benchmarks): the whole-plugin envelope, next to the slice results

The class-by-class table is measured on one- and two-file slices, which
invites the reading that the engine scans WordPress plugins. It does not, and
the README should not let that stand unqualified.

637 files, 117k lines: the build is affordable (33s, 1.2GB, a 1.6MB CPG) and a
CI runner can do it. The scan is not -- 996.6s to decide nothing, with the
taint query dying in ForkJoinPool and 500 of 4,463 regions examined.

What the run does correctly is refuse to look clean: files_unparsed,
query_failed, cpg_sharded and the region cap all reach the reader. A scan that
decides nothing and says so is a different artifact from one that decides
nothing quietly.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-10 08:49:07 +02:00
co-authored by Claude Opus 5
parent ebf37a9b3c
commit c03944f7dd
+34
View File
@@ -200,6 +200,40 @@ That is the argument for why a corpus of benchmarks is not enough on its own, ma
stayed byte-identical through all of it. A corpus tells you a verdict is right; it cannot tell you the
verdict was reached by looking at anything.
## The whole-plugin envelope
The class-by-class table above is measured on slices — one or two files carrying both ends of a CVE. The
obvious question is whether any of it survives on the whole plugin. Measured on `pmpro`, 637 files and
117k lines:
| | |
|---|---|
| preprocess + entry-point mapping | 107s → 673 targets, 4,412 entry points, **4,463 regions** |
| frontend build, one attempt | 33s, peak RSS 1,217 MB, a 1.6 MB CPG |
| full scan: build / query / arbitrate | 255.9s / 720.9s / 19.8s — **996.6s total**, peak RSS 1,132 MB |
| regions examined | **500 of 4,463 (11%)** — the default cap |
| findings | **0** |
| CVE-2023-23488 | **not reported** |
The build is affordable; a CI runner can do this without a vertical-scale machine, which was the open
question. The scan is not: the taint query dies in `ForkJoinPool` after ten minutes, and so does a bare
`cpg.file.size` census, so it is the CPG under a 2 GB heap rather than the request payload.
So the honest statement of PHP detection today is that **the slices find the CVEs and the repository does
not** — one file finds CVE-2023-23488, two files find it with a shard, and 637 files find nothing at all.
No release claim about scanning a WordPress plugin can rest on this.
What the run does do correctly is refuse to look clean. It reports `files_unparsed`, `query_failed` for
2,500 regions, `cpg_sharded`, and the region cap that hid 89% of the repository — all of which appear in
the report under "What could not be analysed". A scan that decides nothing and says so is a different
artifact from a scan that decides nothing quietly, and the whole point of the rung ladder is that the
second one must not be possible.
It also found two defects that slice-scale measurement never would have. `hookCallbacks` describes the
repository rather than the region and was being repeated into every request — 42.5 MB of which 41.7 MB was
one string, now 1.78 MB. And candidates were enumerated for a judge that was never going to be asked, which
was 737 of the 996 seconds; arbitration is now 19.8s.
## Baselines
Task 6.1 turns a scan of a pinned checkout into a committed baseline, so a change to the fact tables or the