mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-18 20:49:28 +00:00
fix(cpg): a graph the frontend emptied must not read as a clean repository
Scanning a two-file WordPress slice returned 64 regions examined, zero
findings and zero degradations. The graph it examined had no methods in it.
php2cpg 4.0.623 feeds several files to one `php-parse` process and reads its
stdout and stderr merged. php-parse writes a `====> File <next>:` banner to
stderr the moment it starts the next file, while the previous file's JSON --
megabytes of it -- is still draining from a block-buffered stdout. The banner
lands inside the JSON, ujson reports
expected json value or } got "=" at index 5011645
and the frontend drops that file. Reproducible on real plugin code, and the
size that matters is one file's AST dump, not the repository's:
class.memberorder.php alone (1834 lines) -> 142KB CPG, no failures
the same file + a two-line file -> 5.7KB CPG, BOTH dropped
One oversized file empties the whole graph. `joern-parse` then does not
propagate any of the frontend's warnings: 13 lines of output, exit status 0,
"Successfully wrote graph". This is what made a 637-file plugin unscannable
and reported it as clean -- it was never a scale limit.
Two fixes, because the failure arrives on two different paths:
* `_unparsed_files` reads the per-file warnings the frontend does emit, and
the scan reports them as `files_unparsed`. This covers the direct-frontend
path where the warnings are visible.
* The batched query now carries a `__census__` of the graph -- method and
file counts -- under a key no request id can collide with, and a scan over
a graph with no methods degrades as `cpg_empty`. This covers joern-parse,
which shows nothing. It rides the batch rather than costing its own
invocation, because avoiding a JVM start is the entire reason batching
exists.
Neither invents a verdict. There is no rung for "the graph was empty" because
the ladder labels verdicts and none was produced, which is exactly why it has
to be a degradation.
pmpro two-file slice before: 0 findings, degradations=[]
after: 0 findings, degradations=[cpg_empty]
Gates byte-identical; 840 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
co-authored by
Claude Opus 5
parent
37a6594703
commit
e0442d146c
@@ -126,6 +126,16 @@ def scan_repository(
|
||||
degradations=({"stage": "model", "reason": "cpg_build_failed"},),
|
||||
)
|
||||
|
||||
# A CPG the frontend built with files missing is not the repository the caller asked about, and no
|
||||
# verdict over it can say anything about the code that was dropped. There is no rung for that, because
|
||||
# the ladder labels verdicts and none was ever produced here -- so it is a degradation, named, with the
|
||||
# files listed. See `cpg.backend._unparsed_files` for how a frontend drops a file while exiting 0.
|
||||
if getattr(cpg, "unparsed", ()): # a backend need not carry the field
|
||||
unparsed = tuple(cpg.unparsed)
|
||||
degradations.append(
|
||||
{"stage": "model", "reason": "files_unparsed", "count": len(unparsed), "files": [Path(name).name for name in unparsed[:20]]}
|
||||
)
|
||||
|
||||
# Count calls here rather than reading a client's own meter: the budget is this driver's contract and
|
||||
# must hold for any client, including one that keeps no usage log.
|
||||
counted = _CountingClient(client) if client is not None else None
|
||||
@@ -150,6 +160,7 @@ def scan_repository(
|
||||
# the queries themselves are milliseconds once the CPG is loaded.
|
||||
rows_by_id: dict[str, list[object]] = {}
|
||||
per_kind: dict[str, float] = {}
|
||||
census_reported = False
|
||||
query_started = time.monotonic()
|
||||
for kind, requests in _grouped(work).items():
|
||||
kind_started = time.monotonic()
|
||||
@@ -162,6 +173,18 @@ def scan_repository(
|
||||
# failed at CPG load and the scan reported 500 regions examined with nothing found.
|
||||
degradations.append({"stage": "model", "reason": "query_failed", "kind": kind, "requests": len(requests)})
|
||||
else:
|
||||
# The graph census rides the batch under a reserved key. A frontend that fails every file
|
||||
# still exits 0 with a valid CPG containing nothing, and `joern-parse` does not propagate
|
||||
# its warnings, so without this a scan of an EMPTY graph is indistinguishable from a scan
|
||||
# that found nothing -- which is how a two-file WordPress slice reported a clean bill of
|
||||
# health over a 12KB graph with no methods in it at all.
|
||||
census = answered_batch.pop("__census__", None)
|
||||
if census and not census_reported:
|
||||
entry = census[0] if isinstance(census[0], Mapping) else {}
|
||||
methods = int(str(entry.get("methods", "0")) or 0)
|
||||
if methods == 0:
|
||||
degradations.append({"stage": "model", "reason": "cpg_empty", "files": int(str(entry.get("files", "0")) or 0)})
|
||||
census_reported = True
|
||||
rows_by_id.update(answered_batch)
|
||||
else: # a backend without batching still works, one call at a time
|
||||
for rid, params in requests.items():
|
||||
|
||||
Reference in New Issue
Block a user