mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 11:49:28 +00:00
fix(cpg): build php through the frontend and retry while it drops files
The intermittent empty-graph builds that cost four manual retries to measure
WP Statistics at all.
First, a correction: the explanation this repo carried was wrong. It said
php-parse's stderr banners interleave into block-buffered stdout and corrupt
the JSON. Capturing the exact bytes php2cpg receives -- by putting a logging
wrapper on `php` -- shows its input arriving complete and valid: two
well-formed JSON documents, 5,176,594 bytes, byte-identical across runs, with
no banner anywhere inside them. The defect is in php2cpg's own reading of that
input, and nothing outside it repairs a given attempt.
Three fixes were tried against the old theory and each changed nothing, which
is how the theory died:
* an unbuffered PHP shim via --php-parser-bin 0 of 6 builds usable
* compacting the JSON, 5,176,594 -> 943,028 bytes 0 of 6
* -J-Djava.util.concurrent.ForkJoinPool.common.parallelism=1 0 of 6
What is true is that the failure is INTERMITTENT, and that `joern-parse` is
both worse at it and silent about it. Measured on one WordPress slice:
joern-parse 5 of 8 builds usable, and says nothing when they are not
php2cpg 11 of 12 builds usable, and names every file it dropped
So php builds through the frontend directly and retries while it reports
dropped files. Retrying is sound only because the failure is intermittent AND
reported -- retrying a silent failure would be superstition; retrying a
reported one is just using the report. It gives up after four attempts and
reports what is still missing rather than pretending a partial graph is whole.
wp-vuln through the driver before: 4+ manual retries to get one build
after: 6 of 6, 3 entailed findings each time
joern-parse remains the path for every other language, and the fallback when
no frontend is on PATH.
843 tests pass; gates byte-identical; mwwpform, pmpro and VAmPI unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
co-authored by
Claude Opus 5
parent
d6908f18ba
commit
1ce7e9db9b
@@ -177,9 +177,24 @@ is still draining from a block-buffered stdout. The banner lands inside the JSON
|
||||
`joern-parse` propagates none of it: thirteen lines of output, exit status 0, "Successfully wrote graph".
|
||||
One oversized file empties the entire graph, so this was never the scale limit it looked like.
|
||||
|
||||
It is also a **race**, not a threshold. Five builds of the same two-file input: three produced a graph, two
|
||||
came back empty. Before the batched query started carrying a census of the graph, two of every five scans of
|
||||
that plugin would have reported a clean bill of health.
|
||||
It is also **intermittent**, not a threshold. Five builds of the same two-file input: three produced a graph,
|
||||
two came back empty. Before the batched query started carrying a census of the graph, two of every five scans
|
||||
of that plugin would have reported a clean bill of health.
|
||||
|
||||
The first explanation offered here — stderr banners interleaving into buffered stdout — was wrong, and worth
|
||||
recording as wrong. Capturing the exact bytes php2cpg receives shows the parser's output arriving complete
|
||||
and valid, byte-identical across runs; the defect is inside php2cpg's own reading of it. An unbuffered shim,
|
||||
compacting the JSON to a fifth of its size, and pinning ForkJoin parallelism each changed nothing.
|
||||
|
||||
What did work is not a repair but a measurement. `joern-parse` is worse at this than the frontend it calls,
|
||||
and silent about it:
|
||||
|
||||
joern-parse 5 of 8 builds usable, and says nothing when they are not
|
||||
php2cpg 11 of 12 builds usable, and names every file it dropped
|
||||
|
||||
So PHP builds through `php2cpg` directly and retries while it reports dropped files. Retrying is sound only
|
||||
because the failure is intermittent *and* reported — retrying a silent failure would be superstition. The
|
||||
slice that previously needed four manual retries now builds 6 times out of 6.
|
||||
|
||||
That is the argument for why a corpus of benchmarks is not enough on its own, made concretely. Every gate
|
||||
stayed byte-identical through all of it. A corpus tells you a verdict is right; it cannot tell you the
|
||||
|
||||
Reference in New Issue
Block a user