perf(model): one Joern invocation per query kind, not one per region

JVM startup dominated a scan: a call per region per family put a ten-line
file at 275s over three regions, and would have put a thousand regions at
roughly fifty hours. The queries themselves are milliseconds once the CPG
is loaded, so the cost was almost entirely process starts.

scan_repository() is now collect -> batch -> arbitrate. The three .sc
scripts accept a `requests` JSON map and answer all of them in one run,
keeping the single-request form so the committed measurements stay
reproducible byte-for-byte. Arbiters are untouched: each is handed a
CpgResult whose `run` returns its own prefetched rows.

The batcher builds parameters by calling request_params() extracted from
taint.py, dominance.py and config_value.py rather than reconstructing
them. A second copy that drifts from the arbiter's is the exact shape of
the five family-assumption bugs the predecessor spec spent a session
finding; there is now one definition per family and the batcher borrows
it.

Regions are also deduplicated by (path, function), collapsing the
nameless entry-point rows that were producing `model:injection:?:10`.

Measured on the same ten-line file: 275s -> 106s, three duplicate
findings -> one `model:injection:ping:10`, rung unchanged at
model_entailed. Gates byte-identical.

Server mode is recorded as task 2.3, deliberately sequenced after the
repository measurement: whether the remaining ~100s floor is worth a
process model, a weakened subprocess seam and a Scala-evaluating HTTP
endpoint depends on a build-time-to-query-time ratio only 2.2 produces.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-09 07:57:35 +02:00
co-authored by Claude Opus 5
parent e67f3ee2f0
commit c35426e186
13 changed files with 412 additions and 76 deletions
+37
View File
@@ -116,6 +116,43 @@ class JoernBackend:
return None
return CpgResult(cpg_path=cpg_path, run=lambda query, params: self.query(cpg_path, query, params))
def query_batch(self, cpg_path: Path, query: str, requests: Mapping[str, Mapping[str, object]]) -> dict[str, list[object]]:
"""Run ONE script invocation carrying many requests, keyed back to their ids.
JVM startup, not CPG construction, is what dominates a repository scan: every ``joern --script`` call
starts a JVM of roughly thirty seconds, and issuing one per region per family put a ten-line Python
file at four minutes and a thousand regions at about fifty hours. The CPG is already built and loaded
by then; the work itself is milliseconds. So the batch carries the whole scan's questions in one
parameter and the script loops over them.
Fails closed like every other path here: an engine that could not answer returns ``{}``, never an
empty result per request, because "no rows" and "could not ask" must stay distinguishable.
"""
script = self.queries_dir / f"{query}.sc"
if not script.is_file():
logger.warning("no such cpg query: %s", script)
return {}
payload = json.dumps({rid: {k: _render(v) for k, v in req.items()} for rid, req in requests.items()})
command = [
shutil.which("joern") or "joern",
"--script",
str(script),
"--param",
f"cpgFile={cpg_path}",
"--param",
f"requests={payload}",
]
completed = self._run(command, timeout=self.query_timeout, cwd=cpg_path.parent)
if completed is None or completed.returncode != 0:
detail = (completed.stderr or "")[-400:] if completed is not None else "timeout"
logger.warning("cpg batch %s failed: %s", query, detail)
return {}
parsed = extract_payload(completed.stdout or "")
if not isinstance(parsed, Mapping):
return {}
# A request the engine did not answer is absent, not empty: the caller must be able to tell them apart.
return {str(rid): list(rows) for rid, rows in parsed.items() if isinstance(rows, list)}
def query(self, cpg_path: Path, query: str, params: Mapping[str, object]) -> object | None:
"""Run a shipped CPGQL script against a built CPG and return its fenced JSON payload."""
script = self.queries_dir / f"{query}.sc"