mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-19 13:10:03 +00:00
perf(model): one Joern invocation per query kind, not one per region
JVM startup dominated a scan: a call per region per family put a ten-line file at 275s over three regions, and would have put a thousand regions at roughly fifty hours. The queries themselves are milliseconds once the CPG is loaded, so the cost was almost entirely process starts. scan_repository() is now collect -> batch -> arbitrate. The three .sc scripts accept a `requests` JSON map and answer all of them in one run, keeping the single-request form so the committed measurements stay reproducible byte-for-byte. Arbiters are untouched: each is handed a CpgResult whose `run` returns its own prefetched rows. The batcher builds parameters by calling request_params() extracted from taint.py, dominance.py and config_value.py rather than reconstructing them. A second copy that drifts from the arbiter's is the exact shape of the five family-assumption bugs the predecessor spec spent a session finding; there is now one definition per family and the batcher borrows it. Regions are also deduplicated by (path, function), collapsing the nameless entry-point rows that were producing `model:injection:?:10`. Measured on the same ten-line file: 275s -> 106s, three duplicate findings -> one `model:injection:ping:10`, rung unchanged at model_entailed. Gates byte-identical. Server mode is recorded as task 2.3, deliberately sequenced after the repository measurement: whether the remaining ~100s floor is worth a process model, a weakened subprocess seam and a Scala-evaluating HTTP endpoint depends on a build-time-to-query-time ratio only 2.2 produces. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
co-authored by
Claude Opus 5
parent
e67f3ee2f0
commit
c35426e186
@@ -116,6 +116,43 @@ class JoernBackend:
|
||||
return None
|
||||
return CpgResult(cpg_path=cpg_path, run=lambda query, params: self.query(cpg_path, query, params))
|
||||
|
||||
def query_batch(self, cpg_path: Path, query: str, requests: Mapping[str, Mapping[str, object]]) -> dict[str, list[object]]:
|
||||
"""Run ONE script invocation carrying many requests, keyed back to their ids.
|
||||
|
||||
JVM startup, not CPG construction, is what dominates a repository scan: every ``joern --script`` call
|
||||
starts a JVM of roughly thirty seconds, and issuing one per region per family put a ten-line Python
|
||||
file at four minutes and a thousand regions at about fifty hours. The CPG is already built and loaded
|
||||
by then; the work itself is milliseconds. So the batch carries the whole scan's questions in one
|
||||
parameter and the script loops over them.
|
||||
|
||||
Fails closed like every other path here: an engine that could not answer returns ``{}``, never an
|
||||
empty result per request, because "no rows" and "could not ask" must stay distinguishable.
|
||||
"""
|
||||
script = self.queries_dir / f"{query}.sc"
|
||||
if not script.is_file():
|
||||
logger.warning("no such cpg query: %s", script)
|
||||
return {}
|
||||
payload = json.dumps({rid: {k: _render(v) for k, v in req.items()} for rid, req in requests.items()})
|
||||
command = [
|
||||
shutil.which("joern") or "joern",
|
||||
"--script",
|
||||
str(script),
|
||||
"--param",
|
||||
f"cpgFile={cpg_path}",
|
||||
"--param",
|
||||
f"requests={payload}",
|
||||
]
|
||||
completed = self._run(command, timeout=self.query_timeout, cwd=cpg_path.parent)
|
||||
if completed is None or completed.returncode != 0:
|
||||
detail = (completed.stderr or "")[-400:] if completed is not None else "timeout"
|
||||
logger.warning("cpg batch %s failed: %s", query, detail)
|
||||
return {}
|
||||
parsed = extract_payload(completed.stdout or "")
|
||||
if not isinstance(parsed, Mapping):
|
||||
return {}
|
||||
# A request the engine did not answer is absent, not empty: the caller must be able to tell them apart.
|
||||
return {str(rid): list(rows) for rid, rows in parsed.items() if isinstance(rows, list)}
|
||||
|
||||
def query(self, cpg_path: Path, query: str, params: Mapping[str, object]) -> object | None:
|
||||
"""Run a shipped CPGQL script against a built CPG and return its fenced JSON payload."""
|
||||
script = self.queries_dir / f"{query}.sc"
|
||||
|
||||
Reference in New Issue
Block a user