fix(cpg): build php through the frontend and retry while it drops files

The intermittent empty-graph builds that cost four manual retries to measure
WP Statistics at all.

First, a correction: the explanation this repo carried was wrong. It said
php-parse's stderr banners interleave into block-buffered stdout and corrupt
the JSON. Capturing the exact bytes php2cpg receives -- by putting a logging
wrapper on `php` -- shows its input arriving complete and valid: two
well-formed JSON documents, 5,176,594 bytes, byte-identical across runs, with
no banner anywhere inside them. The defect is in php2cpg's own reading of that
input, and nothing outside it repairs a given attempt.

Three fixes were tried against the old theory and each changed nothing, which
is how the theory died:

  * an unbuffered PHP shim via --php-parser-bin        0 of 6 builds usable
  * compacting the JSON, 5,176,594 -> 943,028 bytes    0 of 6
  * -J-Djava.util.concurrent.ForkJoinPool.common.parallelism=1   0 of 6

What is true is that the failure is INTERMITTENT, and that `joern-parse` is
both worse at it and silent about it. Measured on one WordPress slice:

    joern-parse   5 of 8 builds usable, and says nothing when they are not
    php2cpg      11 of 12 builds usable, and names every file it dropped

So php builds through the frontend directly and retries while it reports
dropped files. Retrying is sound only because the failure is intermittent AND
reported -- retrying a silent failure would be superstition; retrying a
reported one is just using the report. It gives up after four attempts and
reports what is still missing rather than pretending a partial graph is whole.

    wp-vuln through the driver   before: 4+ manual retries to get one build
                                 after:  6 of 6, 3 entailed findings each time

joern-parse remains the path for every other language, and the fallback when
no frontend is on PATH.

843 tests pass; gates byte-identical; mwwpform, pmpro and VAmPI unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-09 21:11:12 +02:00
co-authored by Claude Opus 5
parent d6908f18ba
commit 1ce7e9db9b
4 changed files with 156 additions and 10 deletions
+18 -4
View File
@@ -580,10 +580,24 @@
One oversized file empties the whole graph, and `joern-parse` propagates none of the warnings -- thirteen
lines of output, exit 0, "Successfully wrote graph". It is also a **race**, not a threshold: five builds
of the same two-file input gave three graphs and two empty ones.
- Two things follow. The engine now says so (`cpg_empty`, from a census carried on the batched query; see
`cpg/backend.py`), which is what turned this from a clean bill of health into a diagnosis. And the
mitigation is per-file, not per-repository: `--exclude` the file whose dump is oversized and the rest of
the tree builds, so the coverage disclosure should name the excluded files rather than the build failing.
- **Corrected 2026-09-09, and the first diagnosis above was WRONG.** The banner-interleaving story is not
what happens. Capturing the exact bytes php2cpg receives -- by putting a logging wrapper on `php` -- shows
the parser's output arriving COMPLETE and VALID: two well-formed JSON documents, 5,176,594 bytes,
byte-identical across runs. The defect is inside php2cpg's own reading of that input, and nothing outside
it repairs a given attempt. Tried and each changed nothing: an unbuffered PHP shim, compacting the JSON
to a fifth of its size (5,176,594 -> 943,028), and `ForkJoinPool.common.parallelism=1`.
- What IS true and useful: the failure is **intermittent**, and `joern-parse` is both worse at it and
silent about it. On one WordPress slice, measured:
joern-parse 5 of 8 builds usable, and says nothing when they are not
php2cpg 11 of 12 builds usable, and names every file it dropped
- So php now builds through the frontend directly and retries while it reports dropped files
(`_build_with_retries`, 4 attempts). Retrying is only sound because the failure is intermittent AND
reported -- retrying a silent failure would be superstition. Driver-level result on the slice that
previously needed four manual retries: **6 of 6 builds usable**, 3 entailed findings each time.
- Still open: a file that fails on every attempt is still reported rather than analysed, and `--exclude`
plus a second CPG for the excluded files is the remaining lever.
- Still open: wiring that exclusion loop, the 637-file build with it in place, and whether the REST handler
two calls away can reach the sink -- the slice found it through a same-function flow, not across the
object.