fix(cpg): retry a failed build through the language frontend

Scanning a 637-file WordPress plugin reported 1577 regions with 0 scanned
and `cpg_build_failed`. The build was not slow or out of memory -- it
failed in 42 seconds -- and `php2cpg` run directly on the same tree
produced a 4.3MB CPG with no errors at all.

The difference is that `joern-parse` detects the language and then shells
out to the frontend in a separate process. On this tree it got as far as
applying overlays and then threw, while still writing an unusable cpg.bin.
Joern's own output recommends the direct route for a large codebase:

  "Invoking CPG generator in a separate process... If you are importing a
   large codebase (and/or running into memory issues), please try:
   invoke the frontend: php2cpg -J-Xmx3072m /tmp/src -o cpg.bin"

So a failed `joern-parse` now retries through `<language>2cpg`, with the
driver passing the dominant region language. A language with no mapping
fails closed exactly as before -- no guessing at a frontend name.

This is the documented fallback rather than a workaround, and it was
diagnosed rather than assumed: the standalone build was run first, and only
after it succeeded on the identical tree was the retry added.

Gates byte-identical.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-09 16:19:32 +02:00
co-authored by Claude Opus 5
parent 922ed84053
commit 9c50e17c11
3 changed files with 101 additions and 4 deletions
+14 -1
View File
@@ -110,7 +110,12 @@ def scan_repository(
taxonomy = load_families()
build_started = time.monotonic()
cpg = backend.build(root)
# The dominant region language, so a failed `joern-parse` can retry through that frontend directly.
counts: dict[str, int] = {}
for region in regions:
counts[region.language] = counts.get(region.language, 0) + 1
dominant = max(counts, key=lambda name: (counts[name], name)) if counts else ""
cpg = _build(backend, root, dominant)
build_seconds = round(time.monotonic() - build_started, 2)
if cpg is None:
return ModelScanResult(
@@ -223,6 +228,14 @@ def scan_repository(
)
def _build(backend: Any, root: Path, language: str) -> Any:
"""Build through the backend, passing the language when the backend can use it."""
try:
return backend.build(root, language=language)
except TypeError: # a backend from before the retry existed, including every test double
return backend.build(root)
def _prefetched(rows: list[object]): # type: ignore[no-untyped-def]
"""A `run` that ignores the question and returns rows already fetched in the batch."""