mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-18 04:29:27 +00:00
docs(benchmarks): the whole-plugin envelope, next to the slice results
The class-by-class table is measured on one- and two-file slices, which invites the reading that the engine scans WordPress plugins. It does not, and the README should not let that stand unqualified. 637 files, 117k lines: the build is affordable (33s, 1.2GB, a 1.6MB CPG) and a CI runner can do it. The scan is not -- 996.6s to decide nothing, with the taint query dying in ForkJoinPool and 500 of 4,463 regions examined. What the run does correctly is refuse to look clean: files_unparsed, query_failed, cpg_sharded and the region cap all reach the reader. A scan that decides nothing and says so is a different artifact from one that decides nothing quietly. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
co-authored by
Claude Opus 5
parent
ebf37a9b3c
commit
c03944f7dd
@@ -200,6 +200,40 @@ That is the argument for why a corpus of benchmarks is not enough on its own, ma
|
|||||||
stayed byte-identical through all of it. A corpus tells you a verdict is right; it cannot tell you the
|
stayed byte-identical through all of it. A corpus tells you a verdict is right; it cannot tell you the
|
||||||
verdict was reached by looking at anything.
|
verdict was reached by looking at anything.
|
||||||
|
|
||||||
|
## The whole-plugin envelope
|
||||||
|
|
||||||
|
The class-by-class table above is measured on slices — one or two files carrying both ends of a CVE. The
|
||||||
|
obvious question is whether any of it survives on the whole plugin. Measured on `pmpro`, 637 files and
|
||||||
|
117k lines:
|
||||||
|
|
||||||
|
| | |
|
||||||
|
|---|---|
|
||||||
|
| preprocess + entry-point mapping | 107s → 673 targets, 4,412 entry points, **4,463 regions** |
|
||||||
|
| frontend build, one attempt | 33s, peak RSS 1,217 MB, a 1.6 MB CPG |
|
||||||
|
| full scan: build / query / arbitrate | 255.9s / 720.9s / 19.8s — **996.6s total**, peak RSS 1,132 MB |
|
||||||
|
| regions examined | **500 of 4,463 (11%)** — the default cap |
|
||||||
|
| findings | **0** |
|
||||||
|
| CVE-2023-23488 | **not reported** |
|
||||||
|
|
||||||
|
The build is affordable; a CI runner can do this without a vertical-scale machine, which was the open
|
||||||
|
question. The scan is not: the taint query dies in `ForkJoinPool` after ten minutes, and so does a bare
|
||||||
|
`cpg.file.size` census, so it is the CPG under a 2 GB heap rather than the request payload.
|
||||||
|
|
||||||
|
So the honest statement of PHP detection today is that **the slices find the CVEs and the repository does
|
||||||
|
not** — one file finds CVE-2023-23488, two files find it with a shard, and 637 files find nothing at all.
|
||||||
|
No release claim about scanning a WordPress plugin can rest on this.
|
||||||
|
|
||||||
|
What the run does do correctly is refuse to look clean. It reports `files_unparsed`, `query_failed` for
|
||||||
|
2,500 regions, `cpg_sharded`, and the region cap that hid 89% of the repository — all of which appear in
|
||||||
|
the report under "What could not be analysed". A scan that decides nothing and says so is a different
|
||||||
|
artifact from a scan that decides nothing quietly, and the whole point of the rung ladder is that the
|
||||||
|
second one must not be possible.
|
||||||
|
|
||||||
|
It also found two defects that slice-scale measurement never would have. `hookCallbacks` describes the
|
||||||
|
repository rather than the region and was being repeated into every request — 42.5 MB of which 41.7 MB was
|
||||||
|
one string, now 1.78 MB. And candidates were enumerated for a judge that was never going to be asked, which
|
||||||
|
was 737 of the 996 seconds; arbitration is now 19.8s.
|
||||||
|
|
||||||
## Baselines
|
## Baselines
|
||||||
|
|
||||||
Task 6.1 turns a scan of a pinned checkout into a committed baseline, so a change to the fact tables or the
|
Task 6.1 turns a scan of a pinned checkout into a committed baseline, so a change to the fact tables or the
|
||||||
|
|||||||
Reference in New Issue
Block a user