fix(cpg): a graph the frontend emptied must not read as a clean repository

Scanning a two-file WordPress slice returned 64 regions examined, zero
findings and zero degradations. The graph it examined had no methods in it.

php2cpg 4.0.623 feeds several files to one `php-parse` process and reads its
stdout and stderr merged. php-parse writes a `====> File <next>:` banner to
stderr the moment it starts the next file, while the previous file's JSON --
megabytes of it -- is still draining from a block-buffered stdout. The banner
lands inside the JSON, ujson reports

    expected json value or } got "=" at index 5011645

and the frontend drops that file. Reproducible on real plugin code, and the
size that matters is one file's AST dump, not the repository's:

    class.memberorder.php alone (1834 lines)     -> 142KB CPG, no failures
    the same file + a two-line file              ->   5.7KB CPG, BOTH dropped

One oversized file empties the whole graph. `joern-parse` then does not
propagate any of the frontend's warnings: 13 lines of output, exit status 0,
"Successfully wrote graph". This is what made a 637-file plugin unscannable
and reported it as clean -- it was never a scale limit.

Two fixes, because the failure arrives on two different paths:

  * `_unparsed_files` reads the per-file warnings the frontend does emit, and
    the scan reports them as `files_unparsed`. This covers the direct-frontend
    path where the warnings are visible.

  * The batched query now carries a `__census__` of the graph -- method and
    file counts -- under a key no request id can collide with, and a scan over
    a graph with no methods degrades as `cpg_empty`. This covers joern-parse,
    which shows nothing. It rides the batch rather than costing its own
    invocation, because avoiding a JVM start is the entire reason batching
    exists.

Neither invents a verdict. There is no rung for "the graph was empty" because
the ladder labels verdicts and none was produced, which is exactly why it has
to be a degradation.

    pmpro two-file slice  before: 0 findings, degradations=[]
                          after:  0 findings, degradations=[cpg_empty]

Gates byte-identical; 840 tests pass.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-09 18:07:28 +02:00
co-authored by Claude Opus 5
parent 37a6594703
commit e0442d146c
7 changed files with 179 additions and 10 deletions
+23
View File
@@ -126,6 +126,16 @@ def scan_repository(
degradations=({"stage": "model", "reason": "cpg_build_failed"},),
)
# A CPG the frontend built with files missing is not the repository the caller asked about, and no
# verdict over it can say anything about the code that was dropped. There is no rung for that, because
# the ladder labels verdicts and none was ever produced here -- so it is a degradation, named, with the
# files listed. See `cpg.backend._unparsed_files` for how a frontend drops a file while exiting 0.
if getattr(cpg, "unparsed", ()): # a backend need not carry the field
unparsed = tuple(cpg.unparsed)
degradations.append(
{"stage": "model", "reason": "files_unparsed", "count": len(unparsed), "files": [Path(name).name for name in unparsed[:20]]}
)
# Count calls here rather than reading a client's own meter: the budget is this driver's contract and
# must hold for any client, including one that keeps no usage log.
counted = _CountingClient(client) if client is not None else None
@@ -150,6 +160,7 @@ def scan_repository(
# the queries themselves are milliseconds once the CPG is loaded.
rows_by_id: dict[str, list[object]] = {}
per_kind: dict[str, float] = {}
census_reported = False
query_started = time.monotonic()
for kind, requests in _grouped(work).items():
kind_started = time.monotonic()
@@ -162,6 +173,18 @@ def scan_repository(
# failed at CPG load and the scan reported 500 regions examined with nothing found.
degradations.append({"stage": "model", "reason": "query_failed", "kind": kind, "requests": len(requests)})
else:
# The graph census rides the batch under a reserved key. A frontend that fails every file
# still exits 0 with a valid CPG containing nothing, and `joern-parse` does not propagate
# its warnings, so without this a scan of an EMPTY graph is indistinguishable from a scan
# that found nothing -- which is how a two-file WordPress slice reported a clean bill of
# health over a 12KB graph with no methods in it at all.
census = answered_batch.pop("__census__", None)
if census and not census_reported:
entry = census[0] if isinstance(census[0], Mapping) else {}
methods = int(str(entry.get("methods", "0")) or 0)
if methods == 0:
degradations.append({"stage": "model", "reason": "cpg_empty", "files": int(str(entry.get("files", "0")) or 0)})
census_reported = True
rows_by_id.update(answered_batch)
else: # a backend without batching still works, one call at a time
for rid, params in requests.items():