mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-19 21:19:28 +00:00
perf(model): one Joern invocation per query kind, not one per region
JVM startup dominated a scan: a call per region per family put a ten-line file at 275s over three regions, and would have put a thousand regions at roughly fifty hours. The queries themselves are milliseconds once the CPG is loaded, so the cost was almost entirely process starts. scan_repository() is now collect -> batch -> arbitrate. The three .sc scripts accept a `requests` JSON map and answer all of them in one run, keeping the single-request form so the committed measurements stay reproducible byte-for-byte. Arbiters are untouched: each is handed a CpgResult whose `run` returns its own prefetched rows. The batcher builds parameters by calling request_params() extracted from taint.py, dominance.py and config_value.py rather than reconstructing them. A second copy that drifts from the arbiter's is the exact shape of the five family-assumption bugs the predecessor spec spent a session finding; there is now one definition per family and the batcher borrows it. Regions are also deduplicated by (path, function), collapsing the nameless entry-point rows that were producing `model:injection:?:10`. Measured on the same ten-line file: 275s -> 106s, three duplicate findings -> one `model:injection:ping:10`, rung unchanged at model_entailed. Gates byte-identical. Server mode is recorded as task 2.3, deliberately sequenced after the repository measurement: whether the remaining ~100s floor is worth a process model, a weakened subprocess seam and a Scala-evaluating HTTP endpoint depends on a build-time-to-query-time ratio only 2.2 produces. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
co-authored by
Claude Opus 5
parent
e67f3ee2f0
commit
c35426e186
@@ -58,7 +58,7 @@
|
|||||||
- _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 8.4_
|
- _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 8.4_
|
||||||
- _Depends: 1.2_
|
- _Depends: 1.2_
|
||||||
|
|
||||||
- [ ] 1.4 Batch the queries — one Joern invocation per scan, not one per region and family
|
- [x] 1.4 Batch the queries — one Joern invocation per scan, not one per region and family
|
||||||
- **Why this is in group 1 rather than group 2.** Wiring 1.3 exposed that the cost model in the design was
|
- **Why this is in group 1 rather than group 2.** Wiring 1.3 exposed that the cost model in the design was
|
||||||
half right. One CPG per scan was the right call, but every *query* is its own `joern --script` JVM launch
|
half right. One CPG per scan was the right call, but every *query* is its own `joern --script` JVM launch
|
||||||
of roughly 30 seconds, and the driver issues one per region per family. A single ten-line Python file
|
of roughly 30 seconds, and the driver issues one per region per family. A single ten-line Python file
|
||||||
@@ -88,15 +88,36 @@
|
|||||||
- _Depends: 1.3_
|
- _Depends: 1.3_
|
||||||
|
|
||||||
- [ ] 2.2 The first repository measurement
|
- [ ] 2.2 The first repository measurement
|
||||||
- Run the full scan against that checkout and commit the artifact: CPG build time, scan wall-clock, peak
|
- Run the full scan against that checkout and commit the artifact: CPG build time, query time and total
|
||||||
memory, findings by rung, cost, `regions_unjudged`, and **whether the known CVE was found and at which
|
scan wall-clock as three separate numbers, peak memory, findings by rung, cost, `regions_unjudged`,
|
||||||
rung**. If the repository cannot be processed in its envelope, record the stage it failed at.
|
and **whether the known CVE was found and at which rung**. The build/query split is separated because
|
||||||
|
2.3 is decided on that ratio and cannot be decided on a single wall-clock figure. If the repository cannot be processed in its envelope, record the stage it failed at.
|
||||||
- This is the go/no-go for the shape of the rest: if one CPG per repository is infeasible, the region and
|
- This is the go/no-go for the shape of the rest: if one CPG per repository is infeasible, the region and
|
||||||
budget design changes and phases 3–6 should not be built on it first.
|
budget design changes and phases 3–6 should not be built on it first.
|
||||||
- Observable: the artifact exists with every field; the reading states go or no-go and why.
|
- Observable: the artifact exists with every field; the reading states go or no-go and why.
|
||||||
- _Requirements: 4.2, 4.3, 4.4, 4.5_
|
- _Requirements: 4.2, 4.3, 4.4, 4.5_
|
||||||
- _Depends: 2.1_
|
- _Depends: 2.1_
|
||||||
|
|
||||||
|
- [ ] 2.3 Decide on Joern server mode, on 2.2's numbers
|
||||||
|
- Joern offers `--server` with `--server-host`, `--server-port` and basic auth: one warm JVM with the CPG
|
||||||
|
loaded once, and queries in milliseconds after that. It would remove the three-to-four JVM starts
|
||||||
|
(~100s) that remain per scan now that 1.4 has batched the queries.
|
||||||
|
- **Deliberately sequenced after the measurement, because deciding it now would be optimising before
|
||||||
|
measuring.** Batching already moved query cost from O(regions) to O(1) per scan; whether the remaining
|
||||||
|
floor matters is a ratio 2.2 produces and nothing before it does. If a real CPG build takes ten minutes,
|
||||||
|
a 100s query floor is noise and server mode buys nothing. If the build is thirty seconds, the floor is
|
||||||
|
3x the build and server mode is the obvious next move.
|
||||||
|
- Costs to weigh against that ratio: process lifecycle (start, health, shutdown, orphan reaping -- stray
|
||||||
|
JVMs had to be killed by hand twice while building the batching); the seam weakening from "the subprocess
|
||||||
|
is the only Joern touch-point" to "subprocess or HTTP client", which is the promise that has kept the
|
||||||
|
backend replaceable; and that a CPGQL endpoint evaluates Scala, so it is arbitrary code execution on the
|
||||||
|
analysis host -- acceptable inside the shipped container, which runs `network_mode: none`, and much less
|
||||||
|
so as a port listening on a contributor's laptop.
|
||||||
|
- Observable: a recorded decision that cites 2.2's build-time-to-query-time split. A preference does not
|
||||||
|
close this task; a ratio does.
|
||||||
|
- _Requirements: 4.2_
|
||||||
|
- _Depends: 2.2_
|
||||||
|
|
||||||
## Group 3 — Ship it
|
## Group 3 — Ship it
|
||||||
|
|
||||||
- [ ] 3.1 Contributor output
|
- [ ] 3.1 Contributor output
|
||||||
|
|||||||
@@ -116,6 +116,43 @@ class JoernBackend:
|
|||||||
return None
|
return None
|
||||||
return CpgResult(cpg_path=cpg_path, run=lambda query, params: self.query(cpg_path, query, params))
|
return CpgResult(cpg_path=cpg_path, run=lambda query, params: self.query(cpg_path, query, params))
|
||||||
|
|
||||||
|
def query_batch(self, cpg_path: Path, query: str, requests: Mapping[str, Mapping[str, object]]) -> dict[str, list[object]]:
|
||||||
|
"""Run ONE script invocation carrying many requests, keyed back to their ids.
|
||||||
|
|
||||||
|
JVM startup, not CPG construction, is what dominates a repository scan: every ``joern --script`` call
|
||||||
|
starts a JVM of roughly thirty seconds, and issuing one per region per family put a ten-line Python
|
||||||
|
file at four minutes and a thousand regions at about fifty hours. The CPG is already built and loaded
|
||||||
|
by then; the work itself is milliseconds. So the batch carries the whole scan's questions in one
|
||||||
|
parameter and the script loops over them.
|
||||||
|
|
||||||
|
Fails closed like every other path here: an engine that could not answer returns ``{}``, never an
|
||||||
|
empty result per request, because "no rows" and "could not ask" must stay distinguishable.
|
||||||
|
"""
|
||||||
|
script = self.queries_dir / f"{query}.sc"
|
||||||
|
if not script.is_file():
|
||||||
|
logger.warning("no such cpg query: %s", script)
|
||||||
|
return {}
|
||||||
|
payload = json.dumps({rid: {k: _render(v) for k, v in req.items()} for rid, req in requests.items()})
|
||||||
|
command = [
|
||||||
|
shutil.which("joern") or "joern",
|
||||||
|
"--script",
|
||||||
|
str(script),
|
||||||
|
"--param",
|
||||||
|
f"cpgFile={cpg_path}",
|
||||||
|
"--param",
|
||||||
|
f"requests={payload}",
|
||||||
|
]
|
||||||
|
completed = self._run(command, timeout=self.query_timeout, cwd=cpg_path.parent)
|
||||||
|
if completed is None or completed.returncode != 0:
|
||||||
|
detail = (completed.stderr or "")[-400:] if completed is not None else "timeout"
|
||||||
|
logger.warning("cpg batch %s failed: %s", query, detail)
|
||||||
|
return {}
|
||||||
|
parsed = extract_payload(completed.stdout or "")
|
||||||
|
if not isinstance(parsed, Mapping):
|
||||||
|
return {}
|
||||||
|
# A request the engine did not answer is absent, not empty: the caller must be able to tell them apart.
|
||||||
|
return {str(rid): list(rows) for rid, rows in parsed.items() if isinstance(rows, list)}
|
||||||
|
|
||||||
def query(self, cpg_path: Path, query: str, params: Mapping[str, object]) -> object | None:
|
def query(self, cpg_path: Path, query: str, params: Mapping[str, object]) -> object | None:
|
||||||
"""Run a shipped CPGQL script against a built CPG and return its fenced JSON payload."""
|
"""Run a shipped CPGQL script against a built CPG and return its fenced JSON payload."""
|
||||||
script = self.queries_dir / f"{query}.sc"
|
script = self.queries_dir / f"{query}.sc"
|
||||||
|
|||||||
@@ -7,11 +7,16 @@
|
|||||||
// Parameters: cpgFile, settings, function (optional).
|
// Parameters: cpgFile, settings, function (optional).
|
||||||
// Output: fenced JSON array of {setting, line, method, args, literalArgs}.
|
// Output: fenced JSON array of {setting, line, method, args, literalArgs}.
|
||||||
|
|
||||||
@main def exec(cpgFile: String, settings: String, function: String = "") = {
|
// BATCHED, like taint.sc. Single-request form kept for the committed measurements.
|
||||||
|
|
||||||
|
@main def exec(cpgFile: String, settings: String = "", function: String = "", requests: String = "") = {
|
||||||
importCpg(cpgFile)
|
importCpg(cpgFile)
|
||||||
|
|
||||||
def split(raw: String): List[String] = raw.split(",").map(_.trim).filter(_.nonEmpty).toList
|
def split(raw: String): List[String] = raw.split(",").map(_.trim).filter(_.nonEmpty).toList
|
||||||
val names = split(settings)
|
|
||||||
|
def rowsFor(settingsS: String, functionS: String): List[ujson.Obj] = {
|
||||||
|
val names = split(settingsS)
|
||||||
|
val function = functionS
|
||||||
|
|
||||||
def matches(name: String): Boolean = names.exists(n => name == n || name == n.split("\\.").last)
|
def matches(name: String): Boolean = names.exists(n => name == n || name == n.split("\\.").last)
|
||||||
|
|
||||||
@@ -38,7 +43,19 @@
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
rows
|
||||||
|
}
|
||||||
|
|
||||||
println("---OUSAST-CPG-BEGIN---")
|
println("---OUSAST-CPG-BEGIN---")
|
||||||
println(ujson.write(ujson.Arr(rows: _*)))
|
if (requests.nonEmpty) {
|
||||||
|
val parsed = ujson.read(requests).obj
|
||||||
|
val answers = parsed.map { case (id, req) =>
|
||||||
|
def field(name: String): String = req.obj.get(name).map(_.str).getOrElse("")
|
||||||
|
id -> ujson.Arr(rowsFor(field("settings"), field("function")): _*)
|
||||||
|
}
|
||||||
|
println(ujson.write(ujson.Obj.from(answers)))
|
||||||
|
} else {
|
||||||
|
println(ujson.write(ujson.Arr(rowsFor(settings, function): _*)))
|
||||||
|
}
|
||||||
println("---OUSAST-CPG-END---")
|
println("---OUSAST-CPG-END---")
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -22,13 +22,26 @@
|
|||||||
//
|
//
|
||||||
// Output: a fenced JSON array of {operation, opLine, opMethod, dominatingGuards}.
|
// Output: a fenced JSON array of {operation, opLine, opMethod, dominatingGuards}.
|
||||||
|
|
||||||
@main def exec(cpgFile: String, operations: String, dischargers: String, function: String = "") = {
|
// BATCHED, like taint.sc: `requests` is {id: {operations, dischargers, function}} and the output is
|
||||||
|
// {id: [row, ...]}. One JVM start answers a whole scan. The single-request form is kept so the committed
|
||||||
|
// measurements remain reproducible against the same call shape.
|
||||||
|
|
||||||
|
@main def exec(
|
||||||
|
cpgFile: String,
|
||||||
|
operations: String = "",
|
||||||
|
dischargers: String = "",
|
||||||
|
function: String = "",
|
||||||
|
requests: String = ""
|
||||||
|
) = {
|
||||||
importCpg(cpgFile)
|
importCpg(cpgFile)
|
||||||
|
|
||||||
def split(raw: String): List[String] = raw.split(",").map(_.trim).filter(_.nonEmpty).toList
|
def split(raw: String): List[String] = raw.split(",").map(_.trim).filter(_.nonEmpty).toList
|
||||||
|
|
||||||
val opNames = split(operations)
|
def rowsFor(operationsS: String, dischargersS: String, functionS: String): List[ujson.Obj] = {
|
||||||
val guardTokens = split(dischargers)
|
|
||||||
|
val opNames = split(operationsS)
|
||||||
|
val guardTokens = split(dischargersS)
|
||||||
|
val function = functionS
|
||||||
|
|
||||||
// Word-boundary matching, never substring. A discharger token like `user` matched as a substring hits
|
// Word-boundary matching, never substring. A discharger token like `user` matched as a substring hits
|
||||||
// `users` inside "SELECT * FROM users WHERE id = ?", marking a textbook IDOR as guarded -- a silent false
|
// `users` inside "SELECT * FROM users WHERE id = ?", marking a textbook IDOR as guarded -- a silent false
|
||||||
@@ -69,7 +82,19 @@
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
rows
|
||||||
|
}
|
||||||
|
|
||||||
println("---OUSAST-CPG-BEGIN---")
|
println("---OUSAST-CPG-BEGIN---")
|
||||||
println(ujson.write(ujson.Arr(rows: _*)))
|
if (requests.nonEmpty) {
|
||||||
|
val parsed = ujson.read(requests).obj
|
||||||
|
val answers = parsed.map { case (id, req) =>
|
||||||
|
def field(name: String): String = req.obj.get(name).map(_.str).getOrElse("")
|
||||||
|
id -> ujson.Arr(rowsFor(field("operations"), field("dischargers"), field("function")): _*)
|
||||||
|
}
|
||||||
|
println(ujson.write(ujson.Obj.from(answers)))
|
||||||
|
} else {
|
||||||
|
println(ujson.write(ujson.Arr(rowsFor(operations, dischargers, function): _*)))
|
||||||
|
}
|
||||||
println("---OUSAST-CPG-END---")
|
println("---OUSAST-CPG-END---")
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -22,18 +22,29 @@
|
|||||||
// Output: a fenced JSON array of {sink, sinkLine, sinkMethod, source, sanitized, length}. The fence exists
|
// Output: a fenced JSON array of {sink, sinkLine, sinkMethod, source, sanitized, length}. The fence exists
|
||||||
// because Joern prints a banner, pass logs and a prompt around whatever a script emits.
|
// because Joern prints a banner, pass logs and a prompt around whatever a script emits.
|
||||||
|
|
||||||
|
// BATCHED. `requests` is a JSON object of {id: {sources, sinks, sanitizers, function, parameterSources}} and
|
||||||
|
// the output is {id: [row, ...]}. One invocation answers a whole scan's questions, because the JVM start
|
||||||
|
// (~30s) dwarfs the queries themselves once the CPG is loaded -- one call per region per family put a
|
||||||
|
// ten-line file at four minutes and a thousand regions at roughly fifty hours.
|
||||||
|
//
|
||||||
|
// The single-request form is kept below it: the pair harnesses and every measurement committed so far use it,
|
||||||
|
// and changing their call shape would silently invalidate comparisons against those artifacts.
|
||||||
|
|
||||||
@main def exec(
|
@main def exec(
|
||||||
cpgFile: String,
|
cpgFile: String,
|
||||||
sources: String,
|
sources: String = "",
|
||||||
sinks: String,
|
sinks: String = "",
|
||||||
sanitizers: String = "",
|
sanitizers: String = "",
|
||||||
function: String = "",
|
function: String = "",
|
||||||
parameterSources: String = "false"
|
parameterSources: String = "false",
|
||||||
|
requests: String = ""
|
||||||
) = {
|
) = {
|
||||||
importCpg(cpgFile)
|
importCpg(cpgFile)
|
||||||
|
|
||||||
def split(raw: String): List[String] = raw.split(",").map(_.trim).filter(_.nonEmpty).toList
|
def split(raw: String): List[String] = raw.split(",").map(_.trim).filter(_.nonEmpty).toList
|
||||||
|
|
||||||
|
def rowsFor(sourcesS: String, sinksS: String, sanitizersS: String, functionS: String, paramSrc: String): List[ujson.Obj] = {
|
||||||
|
|
||||||
// Word-boundary matching, never substring. `resolveUrl` contains `resolve`, so a bare-substring sanitizer
|
// Word-boundary matching, never substring. `resolveUrl` contains `resolve`, so a bare-substring sanitizer
|
||||||
// test marked the VULNERABLE flow sanitized and the verdict fell back to a captured `res` that names no
|
// test marked the VULNERABLE flow sanitized and the verdict fell back to a captured `res` that names no
|
||||||
// attacker input. Same bug class as the dominance query's guard matching; fixed in both.
|
// attacker input. Same bug class as the dominance query's guard matching; fixed in both.
|
||||||
@@ -45,9 +56,11 @@
|
|||||||
.find()
|
.find()
|
||||||
)
|
)
|
||||||
|
|
||||||
val sourcePatterns = split(sources)
|
val sourcePatterns = split(sourcesS)
|
||||||
val sinkNames = split(sinks)
|
val sinkNames = split(sinksS)
|
||||||
val sanitizerNames = split(sanitizers)
|
val sanitizerNames = split(sanitizersS)
|
||||||
|
val function = functionS
|
||||||
|
val parameterSources = paramSrc
|
||||||
|
|
||||||
// A source is any call whose code contains one of the patterns. Matching on `code` rather than a method
|
// A source is any call whose code contains one of the patterns. Matching on `code` rather than a method
|
||||||
// name keeps `request.args["x"]` (an indexAccess over a fieldAccess) and `req.body.name` both reachable
|
// name keeps `request.args["x"]` (an indexAccess over a fieldAccess) and `req.body.name` both reachable
|
||||||
@@ -126,7 +139,20 @@
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
rows
|
||||||
|
}
|
||||||
|
|
||||||
println("---OUSAST-CPG-BEGIN---")
|
println("---OUSAST-CPG-BEGIN---")
|
||||||
println(ujson.write(ujson.Arr(rows: _*)))
|
if (requests.nonEmpty) {
|
||||||
|
val parsed = ujson.read(requests).obj
|
||||||
|
val answers = parsed.map { case (id, req) =>
|
||||||
|
def field(name: String): String = req.obj.get(name).map(_.str).getOrElse("")
|
||||||
|
val paramSrc = req.obj.get("parameterSources").map(_.str).getOrElse("false")
|
||||||
|
id -> ujson.Arr(rowsFor(field("sources"), field("sinks"), field("sanitizers"), field("function"), paramSrc): _*)
|
||||||
|
}
|
||||||
|
println(ujson.write(ujson.Obj.from(answers)))
|
||||||
|
} else {
|
||||||
|
println(ujson.write(ujson.Arr(rowsFor(sources, sinks, sanitizers, function, parameterSources): _*)))
|
||||||
|
}
|
||||||
println("---OUSAST-CPG-END---")
|
println("---OUSAST-CPG-END---")
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -19,9 +19,14 @@ from .ladder import Rung, Verdict
|
|||||||
from .specs import ConfigSpec
|
from .specs import ConfigSpec
|
||||||
|
|
||||||
|
|
||||||
|
def request_params(spec: ConfigSpec, *, function: str = "") -> dict[str, object]:
|
||||||
|
"""The query parameters this arbiter sends. Shared with the batcher; see the note in `taint.request_params`."""
|
||||||
|
return {"settings": spec.settings, "function": function}
|
||||||
|
|
||||||
|
|
||||||
def verdict(cpg: CpgResult, spec: ConfigSpec, *, function: str = "") -> Verdict | None:
|
def verdict(cpg: CpgResult, spec: ConfigSpec, *, function: str = "") -> Verdict | None:
|
||||||
"""The verdict for a security setting in ``function``, or ``None`` when nothing is established."""
|
"""The verdict for a security setting in ``function``, or ``None`` when nothing is established."""
|
||||||
rows = cpg.run("config", {"settings": spec.settings, "function": function})
|
rows = cpg.run("config", request_params(spec, function=function))
|
||||||
settings = _settings(rows, spec, function=function)
|
settings = _settings(rows, spec, function=function)
|
||||||
if not settings:
|
if not settings:
|
||||||
return None
|
return None
|
||||||
@@ -79,4 +84,4 @@ def _settings(rows: object, spec: ConfigSpec, *, function: str) -> list[dict[str
|
|||||||
return sorted(kept, key=lambda row: (str(row["method"]), str(row["line"]), str(row["setting"])))
|
return sorted(kept, key=lambda row: (str(row["method"]), str(row["line"]), str(row["setting"])))
|
||||||
|
|
||||||
|
|
||||||
__all__ = ["verdict"]
|
__all__ = ["request_params", "verdict"]
|
||||||
|
|||||||
@@ -26,12 +26,14 @@ from .ladder import Rung, Verdict
|
|||||||
from .specs import DominanceSpec
|
from .specs import DominanceSpec
|
||||||
|
|
||||||
|
|
||||||
|
def request_params(spec: DominanceSpec, *, function: str = "") -> dict[str, object]:
|
||||||
|
"""The query parameters this arbiter sends. Shared with the batcher; see the note in `taint.request_params`."""
|
||||||
|
return {"operations": spec.operations, "dischargers": spec.dischargers, "function": function}
|
||||||
|
|
||||||
|
|
||||||
def verdict(cpg: CpgResult, spec: DominanceSpec, *, function: str = "") -> Verdict | None:
|
def verdict(cpg: CpgResult, spec: DominanceSpec, *, function: str = "") -> Verdict | None:
|
||||||
"""The verdict for the obligated operation in ``function``, or ``None`` when there is nothing to say."""
|
"""The verdict for the obligated operation in ``function``, or ``None`` when there is nothing to say."""
|
||||||
rows = cpg.run(
|
rows = cpg.run("dominance", request_params(spec, function=function))
|
||||||
"dominance",
|
|
||||||
{"operations": spec.operations, "dischargers": spec.dischargers, "function": function},
|
|
||||||
)
|
|
||||||
operations = _operations(rows)
|
operations = _operations(rows)
|
||||||
if not operations:
|
if not operations:
|
||||||
return None
|
return None
|
||||||
@@ -85,4 +87,4 @@ def _operations(rows: object) -> list[dict[str, object]]:
|
|||||||
return kept
|
return kept
|
||||||
|
|
||||||
|
|
||||||
__all__ = ["verdict"]
|
__all__ = ["request_params", "verdict"]
|
||||||
|
|||||||
@@ -64,8 +64,11 @@ def regions_for(entries: Sequence[object], targets: Sequence[object]) -> tuple[S
|
|||||||
if language:
|
if language:
|
||||||
by_path[str(getattr(target, "path", ""))] = language
|
by_path[str(getattr(target, "path", ""))] = language
|
||||||
|
|
||||||
regions: list[ScanRegion] = []
|
# One region per (path, function). The mapper can emit several entry points for a single file -- on a
|
||||||
covered: set[str] = set()
|
# ten-line Flask file it produced three, one named and two nameless -- and turning each into a region
|
||||||
|
# scanned the same code repeatedly, multiplying Joern invocations for no new coverage. Where several
|
||||||
|
# entries agree on a region, the highest access rank wins, because that is the one worth looking at first.
|
||||||
|
best: dict[tuple[str, str | None], ScanRegion] = {}
|
||||||
for entry in entries:
|
for entry in entries:
|
||||||
path = str(getattr(entry, "path", ""))
|
path = str(getattr(entry, "path", ""))
|
||||||
language = by_path.get(path)
|
language = by_path.get(path)
|
||||||
@@ -74,17 +77,18 @@ def regions_for(entries: Sequence[object], targets: Sequence[object]) -> tuple[S
|
|||||||
families = _families(language, has_handler=True)
|
families = _families(language, has_handler=True)
|
||||||
if not families:
|
if not families:
|
||||||
continue
|
continue
|
||||||
covered.add(path)
|
function = getattr(entry, "function_name", None) or None
|
||||||
regions.append(
|
rank = _ACCESS_RANK.get(str(getattr(entry, "access_level", "")), 0.5)
|
||||||
ScanRegion(
|
key = (path, function)
|
||||||
path=path,
|
current = best.get(key)
|
||||||
function=(getattr(entry, "function_name", None) or None),
|
if current is None or rank > current.rank:
|
||||||
language=language,
|
best[key] = ScanRegion(path=path, function=function, language=language, families=families, rank=rank, source="entry_point")
|
||||||
families=families,
|
|
||||||
rank=_ACCESS_RANK.get(str(getattr(entry, "access_level", "")), 0.5),
|
# A nameless entry point IS the file. Where a named region already covers that file it adds nothing but
|
||||||
source="entry_point",
|
# a second full-file pass, so it is dropped -- unless the file has no named region at all.
|
||||||
)
|
named_paths = {path for path, function in best if function is not None}
|
||||||
)
|
regions = [region for (path, function), region in best.items() if function is not None or path not in named_paths]
|
||||||
|
covered = {region.path for region in regions}
|
||||||
|
|
||||||
for path, language in by_path.items():
|
for path, language in by_path.items():
|
||||||
if path in covered:
|
if path in covered:
|
||||||
|
|||||||
@@ -24,13 +24,21 @@ from dataclasses import dataclass, field
|
|||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
|
from ..cpg.backend import CpgResult
|
||||||
from .candidates import enumerate_candidates
|
from .candidates import enumerate_candidates
|
||||||
|
from .config_value import request_params as config_params
|
||||||
|
from .dominance import request_params as dominance_params
|
||||||
from .ladder import Rung
|
from .ladder import Rung
|
||||||
from .pipeline import ModelFinding, scan_region
|
from .pipeline import ModelFinding, scan_region
|
||||||
from .regions import ScanRegion
|
from .regions import ScanRegion
|
||||||
from .specs import ConfigSpec, DominanceSpec, TaintSpec, config_specs, dominance_specs, taint_specs
|
from .specs import ConfigSpec, DominanceSpec, TaintSpec, config_specs, dominance_specs, taint_specs
|
||||||
|
from .taint import request_params as taint_params
|
||||||
from .taxonomy import load_families
|
from .taxonomy import load_families
|
||||||
|
|
||||||
|
# The three arbiters a region can be routed to. Naming the union keeps the batcher honest: `object`
|
||||||
|
# would let a fourth spec kind reach `_grouped` and silently fall through to the taint branch.
|
||||||
|
ArbiterSpec = TaintSpec | DominanceSpec | ConfigSpec
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
_RUNG_ORDER = {Rung.ENTAILED: 0, Rung.CORROBORATED: 1, Rung.SUSPICION: 2, Rung.EXECUTION_CONFIRMED: -1}
|
_RUNG_ORDER = {Rung.ENTAILED: 0, Rung.CORROBORATED: 1, Rung.SUSPICION: 2, Rung.EXECUTION_CONFIRMED: -1}
|
||||||
@@ -90,23 +98,45 @@ def scan_repository(
|
|||||||
# spent in belongs to whoever holds the budget.
|
# spent in belongs to whoever holds the budget.
|
||||||
ordered_regions = sorted(regions, key=lambda r: (-r.rank, r.path, r.function or ""))
|
ordered_regions = sorted(regions, key=lambda r: (-r.rank, r.path, r.function or ""))
|
||||||
|
|
||||||
for region in ordered_regions:
|
# Phase 1: collect the whole scan's questions, for at most `max_regions` regions. Nothing is asked yet.
|
||||||
if scanned >= limits.max_regions or (counted is not None and counted.calls >= limits.max_model_calls):
|
work: list[tuple[str, ScanRegion, ArbiterSpec]] = []
|
||||||
degradations.append({"stage": "model", "reason": "budget_exhausted", "regions_unjudged": len(regions) - scanned})
|
for region in ordered_regions[: limits.max_regions]:
|
||||||
break
|
|
||||||
for family in region.families:
|
for family in region.families:
|
||||||
spec = _spec_for(family, region.language)
|
spec = _spec_for(family, region.language)
|
||||||
if spec is None:
|
if spec is None:
|
||||||
continue
|
continue
|
||||||
|
work.append((f"{len(work)}", region, spec))
|
||||||
|
|
||||||
|
# Phase 2: ONE invocation per query kind. JVM startup dominates a repository scan -- a call per region
|
||||||
|
# per family put a ten-line file at four minutes and a thousand regions at roughly fifty hours -- while
|
||||||
|
# the queries themselves are milliseconds once the CPG is loaded.
|
||||||
|
rows_by_id: dict[str, list[object]] = {}
|
||||||
|
for kind, requests in _grouped(work).items():
|
||||||
|
batch = getattr(cpg, "run_batch", None)
|
||||||
|
if callable(batch):
|
||||||
|
rows_by_id.update(batch(kind, requests) or {})
|
||||||
|
else: # a backend without batching still works, one call at a time
|
||||||
|
for rid, params in requests.items():
|
||||||
|
answered = cpg.run(kind, params)
|
||||||
|
rows_by_id[rid] = list(answered) if isinstance(answered, list) else []
|
||||||
|
|
||||||
|
# Phase 3: arbitrate from the rows already in hand. The arbiters are unchanged: each is handed a
|
||||||
|
# CpgResult that simply returns its own prefetched rows.
|
||||||
|
#
|
||||||
|
# `judged` counts regions actually ARBITRATED, not regions collected: a region the budget stopped before
|
||||||
|
# is unjudged, and reporting it as scanned is the silent truncation this exists to prevent.
|
||||||
|
judged: set[tuple[str, str | None]] = set()
|
||||||
|
for rid, region, spec in work:
|
||||||
|
if counted is not None and counted.calls >= limits.max_model_calls:
|
||||||
|
degradations.append({"stage": "model", "reason": "budget_exhausted", "regions_unjudged": max(len(regions) - len(judged), 0)})
|
||||||
|
break
|
||||||
|
family = getattr(spec, "family", "")
|
||||||
|
prefetched = CpgResult(cpg_path=cpg.cpg_path, run=_prefetched(rows_by_id.get(rid, [])))
|
||||||
try:
|
try:
|
||||||
# Enumerate the region's candidate sites so the judge has something to answer about where the
|
|
||||||
# model resolves nothing. Without this a repository scan reports only what taint, dominance
|
|
||||||
# and constant abstraction establish unaided -- and the suspicion band, which supplied 15 of
|
|
||||||
# 26 findings in the pair measurement, would never be populated at all.
|
|
||||||
enumerated = enumerate_candidates(root, region, family, taxonomy=taxonomy)
|
enumerated = enumerate_candidates(root, region, family, taxonomy=taxonomy)
|
||||||
candidates = [{"id": c.id, "text": c.text, "line": c.line} for c in enumerated.candidates]
|
candidates = [{"id": c.id, "text": c.text, "line": c.line} for c in enumerated.candidates]
|
||||||
found = scan_region(
|
found = scan_region(
|
||||||
cpg,
|
prefetched,
|
||||||
spec,
|
spec,
|
||||||
function=region.function or "",
|
function=region.function or "",
|
||||||
client=counted,
|
client=counted,
|
||||||
@@ -118,8 +148,10 @@ def scan_repository(
|
|||||||
logger.warning("model scan failed for %s: %s", region.path, exc)
|
logger.warning("model scan failed for %s: %s", region.path, exc)
|
||||||
degradations.append({"stage": "model", "reason": "region_failed", "path": region.path})
|
degradations.append({"stage": "model", "reason": "region_failed", "path": region.path})
|
||||||
continue
|
continue
|
||||||
|
judged.add((region.path, region.function))
|
||||||
collected.extend((finding, region.rank) for finding in found)
|
collected.extend((finding, region.rank) for finding in found)
|
||||||
scanned += 1
|
|
||||||
|
scanned = len(judged)
|
||||||
|
|
||||||
findings = _ordered(collected)
|
findings = _ordered(collected)
|
||||||
return ModelScanResult(
|
return ModelScanResult(
|
||||||
@@ -134,7 +166,36 @@ def scan_repository(
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def _spec_for(family: str, language: str) -> TaintSpec | DominanceSpec | ConfigSpec | None:
|
def _prefetched(rows: list[object]): # type: ignore[no-untyped-def]
|
||||||
|
"""A `run` that ignores the question and returns rows already fetched in the batch."""
|
||||||
|
|
||||||
|
def run(query: str, params: Mapping[str, object]) -> object:
|
||||||
|
del query, params
|
||||||
|
return rows
|
||||||
|
|
||||||
|
return run
|
||||||
|
|
||||||
|
|
||||||
|
def _grouped(work: Sequence[tuple[str, ScanRegion, ArbiterSpec]]) -> dict[str, dict[str, Mapping[str, object]]]:
|
||||||
|
"""The work, keyed by query kind, with each request built by the arbiter's OWN parameter function.
|
||||||
|
|
||||||
|
Building the parameters here instead would be a second copy that drifts from the arbiter's -- the exact
|
||||||
|
shape of the five bugs the predecessor spent a session finding.
|
||||||
|
"""
|
||||||
|
grouped: dict[str, dict[str, Mapping[str, object]]] = {}
|
||||||
|
for rid, region, spec in work:
|
||||||
|
function = region.function or ""
|
||||||
|
if isinstance(spec, DominanceSpec):
|
||||||
|
kind, params = "dominance", dominance_params(spec, function=function)
|
||||||
|
elif isinstance(spec, ConfigSpec):
|
||||||
|
kind, params = "config", config_params(spec, function=function)
|
||||||
|
else:
|
||||||
|
kind, params = "taint", taint_params(spec, function=function, parameter_sources=False)
|
||||||
|
grouped.setdefault(kind, {})[rid] = params
|
||||||
|
return grouped
|
||||||
|
|
||||||
|
|
||||||
|
def _spec_for(family: str, language: str) -> ArbiterSpec | None:
|
||||||
"""The spec whose arbiter can decide this family. Routing stays a lookup; dispatch stays in `_arbitrate`."""
|
"""The spec whose arbiter can decide this family. Routing stays a lookup; dispatch stays in `_arbitrate`."""
|
||||||
if family == "access_control":
|
if family == "access_control":
|
||||||
return dominance_specs(language=language).get(family)
|
return dominance_specs(language=language).get(family)
|
||||||
|
|||||||
@@ -19,6 +19,21 @@ from .ladder import Rung, Verdict
|
|||||||
from .specs import TaintSpec
|
from .specs import TaintSpec
|
||||||
|
|
||||||
|
|
||||||
|
def request_params(spec: TaintSpec, *, function: str = "", parameter_sources: bool = False) -> dict[str, object]:
|
||||||
|
"""The query parameters this arbiter sends. Shared with the batcher so there is one definition, not two.
|
||||||
|
|
||||||
|
Two copies of a parameter dict is precisely the duplication that produced five bugs of one shape in the
|
||||||
|
predecessor: a change made in one place and not its sibling.
|
||||||
|
"""
|
||||||
|
return {
|
||||||
|
"sources": spec.sources,
|
||||||
|
"sinks": spec.sinks,
|
||||||
|
"sanitizers": spec.sanitizers,
|
||||||
|
"function": function,
|
||||||
|
"parameterSources": "true" if parameter_sources else "false",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
def verdict(cpg: CpgResult, spec: TaintSpec, *, function: str = "", parameter_sources: bool = False) -> Verdict | None:
|
def verdict(cpg: CpgResult, spec: TaintSpec, *, function: str = "", parameter_sources: bool = False) -> Verdict | None:
|
||||||
"""The strongest verdict the taint query supports for ``spec`` in ``function``, or ``None`` to stay at suspicion.
|
"""The strongest verdict the taint query supports for ``spec`` in ``function``, or ``None`` to stay at suspicion.
|
||||||
|
|
||||||
@@ -26,16 +41,7 @@ def verdict(cpg: CpgResult, spec: TaintSpec, *, function: str = "", parameter_so
|
|||||||
for a function-level pair, where the function boundary *is* the trust boundary, and wrong for a whole
|
for a function-level pair, where the function boundary *is* the trust boundary, and wrong for a whole
|
||||||
repository, where most parameters carry internal values — so it is off by default and the caller opts in.
|
repository, where most parameters carry internal values — so it is off by default and the caller opts in.
|
||||||
"""
|
"""
|
||||||
rows = cpg.run(
|
rows = cpg.run("taint", request_params(spec, function=function, parameter_sources=parameter_sources))
|
||||||
"taint",
|
|
||||||
{
|
|
||||||
"sources": spec.sources,
|
|
||||||
"sinks": spec.sinks,
|
|
||||||
"sanitizers": spec.sanitizers,
|
|
||||||
"function": function,
|
|
||||||
"parameterSources": "true" if parameter_sources else "false",
|
|
||||||
},
|
|
||||||
)
|
|
||||||
flows = _flows(rows, function=function)
|
flows = _flows(rows, function=function)
|
||||||
# A sink whose *shape* is safe is the fix, not the bug. `execute(sql, params)` binds rather than
|
# A sink whose *shape* is safe is the fix, not the bug. `execute(sql, params)` binds rather than
|
||||||
# interpolates and `printf("literal", x)` has a constant format string, so the flow that reaches them is
|
# interpolates and `printf("literal", x)` has a constant format string, so the flow that reaches them is
|
||||||
@@ -146,4 +152,4 @@ def _witness(flow: Mapping[str, object]) -> str:
|
|||||||
return f"{flow['source']} -> {flow['sink']} (line {line}, {flow['length']} steps)"
|
return f"{flow['source']} -> {flow['sink']} (line {line}, {flow['length']} steps)"
|
||||||
|
|
||||||
|
|
||||||
__all__ = ["verdict"]
|
__all__ = ["request_params", "verdict"]
|
||||||
|
|||||||
@@ -105,3 +105,60 @@ def test_the_query_scripts_are_shipped_beside_the_seam() -> None:
|
|||||||
"""The CPGQL is ours and is data: it lives with the package, not inlined in a Python string."""
|
"""The CPGQL is ours and is data: it lives with the package, not inlined in a Python string."""
|
||||||
queries = Path("src/openultrasast/cpg/queries")
|
queries = Path("src/openultrasast/cpg/queries")
|
||||||
assert (queries / "taint.sc").is_file(), "the injection taint query must ship with the package"
|
assert (queries / "taint.sc").is_file(), "the injection taint query must ship with the package"
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_batch_is_one_invocation_carrying_many_requests(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
"""The cost that dominates a repository scan is JVM startup, not CPG construction.
|
||||||
|
|
||||||
|
Every `joern --script` call starts a JVM (~30s). The driver issued one per region per family, so a
|
||||||
|
ten-line Python file -- six families, one region -- took over four minutes, and 1000 regions would take
|
||||||
|
~50 hours. Batching collapses that to one invocation per query kind.
|
||||||
|
"""
|
||||||
|
from openultrasast.cpg.backend import BEGIN, END, JoernBackend
|
||||||
|
|
||||||
|
calls: list[list[str]] = []
|
||||||
|
|
||||||
|
class _Done:
|
||||||
|
returncode = 0
|
||||||
|
stderr = ""
|
||||||
|
stdout = f'{BEGIN}\n{{"r1": [{{"sink": "os.system(x)"}}], "r2": []}}\n{END}\n'
|
||||||
|
|
||||||
|
def runner(command, **kwargs): # type: ignore[no-untyped-def]
|
||||||
|
calls.append(list(command))
|
||||||
|
return _Done()
|
||||||
|
|
||||||
|
backend = JoernBackend(runner=runner)
|
||||||
|
rows = backend.query_batch(
|
||||||
|
Path("/tmp/c.bin"),
|
||||||
|
"taint",
|
||||||
|
{"r1": {"sources": ("request.args",), "sinks": ("os.system",)}, "r2": {"sources": ("request.args",), "sinks": ("eval",)}},
|
||||||
|
)
|
||||||
|
assert len(calls) == 1, "a batch must be ONE invocation, whatever it carries"
|
||||||
|
assert rows["r1"] and rows["r2"] == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_failed_batch_yields_no_rows_for_any_request(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
"""Fail closed, as everywhere else: an engine that could not answer must not read as 'nothing here'."""
|
||||||
|
from openultrasast.cpg.backend import JoernBackend
|
||||||
|
|
||||||
|
class _Bad:
|
||||||
|
returncode = 1
|
||||||
|
stdout = ""
|
||||||
|
stderr = "boom"
|
||||||
|
|
||||||
|
backend = JoernBackend(runner=lambda c, **k: _Bad())
|
||||||
|
assert backend.query_batch(Path("/tmp/c.bin"), "taint", {"r1": {}}) == {}
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_batch_result_that_omits_a_request_returns_nothing_for_it() -> None:
|
||||||
|
"""A missing key is not an empty answer: the caller must be able to tell them apart."""
|
||||||
|
from openultrasast.cpg.backend import BEGIN, END, JoernBackend
|
||||||
|
|
||||||
|
class _Partial:
|
||||||
|
returncode = 0
|
||||||
|
stderr = ""
|
||||||
|
stdout = f'{BEGIN}\n{{"r1": []}}\n{END}\n'
|
||||||
|
|
||||||
|
backend = JoernBackend(runner=lambda c, **k: _Partial())
|
||||||
|
rows = backend.query_batch(Path("/tmp/c.bin"), "taint", {"r1": {}, "r2": {}})
|
||||||
|
assert rows.get("r1") == [] and "r2" not in rows
|
||||||
|
|||||||
@@ -116,3 +116,37 @@ def test_an_unsupported_language_yields_no_region() -> None:
|
|||||||
from openultrasast.model.regions import regions_for
|
from openultrasast.model.regions import regions_for
|
||||||
|
|
||||||
assert regions_for([], [_target(path="x.rb", language="ruby")]) == ()
|
assert regions_for([], [_target(path="x.rb", language="ruby")]) == ()
|
||||||
|
|
||||||
|
|
||||||
|
def test_nameless_entry_points_do_not_duplicate_a_file_already_covered() -> None:
|
||||||
|
"""The mapper can emit several entry points for one file, some with no function name.
|
||||||
|
|
||||||
|
On a ten-line Flask file it produced three: `ping`, and two nameless ones. Turning each into a region
|
||||||
|
scanned the same file three times and multiplied the Joern invocations by three for no new coverage.
|
||||||
|
A nameless entry point is the file; if a named region already covers that file, it adds nothing.
|
||||||
|
"""
|
||||||
|
from openultrasast.model.regions import regions_for
|
||||||
|
|
||||||
|
entries = [
|
||||||
|
_entry(function="ping", access="public"),
|
||||||
|
_entry(function=None, access="local-only"),
|
||||||
|
_entry(function=None, access="public"),
|
||||||
|
]
|
||||||
|
regions = regions_for(entries, [_target()])
|
||||||
|
assert len(regions) == 1
|
||||||
|
assert regions[0].function == "ping"
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_file_with_only_nameless_entry_points_still_gets_one_region() -> None:
|
||||||
|
from openultrasast.model.regions import regions_for
|
||||||
|
|
||||||
|
regions = regions_for([_entry(function=None), _entry(function=None)], [_target()])
|
||||||
|
assert len(regions) == 1 and regions[0].function is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_two_named_handlers_in_one_file_are_two_regions() -> None:
|
||||||
|
"""Deduping must not collapse genuinely distinct handlers."""
|
||||||
|
from openultrasast.model.regions import regions_for
|
||||||
|
|
||||||
|
regions = regions_for([_entry(function="a"), _entry(function="b")], [_target()])
|
||||||
|
assert sorted(r.function for r in regions) == ["a", "b"]
|
||||||
|
|||||||
@@ -145,3 +145,44 @@ def test_a_region_is_only_asked_about_families_it_carries() -> None:
|
|||||||
|
|
||||||
scan_repository(Path("/repo"), (_region(families=("injection",)),), backend=_B(), client=None, model="")
|
scan_repository(Path("/repo"), (_region(families=("injection",)),), backend=_B(), client=None, model="")
|
||||||
assert "dominance" not in asked and "config" not in asked
|
assert "dominance" not in asked and "config" not in asked
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_driver_issues_one_query_invocation_per_kind_not_one_per_region(tmp_path) -> None:
|
||||||
|
"""Task 1.4: JVM startup, not CPG construction, is what dominates a repository scan.
|
||||||
|
|
||||||
|
Before batching the driver issued one `joern --script` call per region per family. A ten-line Python file
|
||||||
|
admits six families, so it cost six JVM launches and over four minutes; a thousand regions would have
|
||||||
|
taken about fifty hours. The whole scan's questions now go in one invocation per query kind.
|
||||||
|
"""
|
||||||
|
from openultrasast.model.scan import scan_repository
|
||||||
|
|
||||||
|
invocations: list[str] = []
|
||||||
|
|
||||||
|
class _CountingBackend:
|
||||||
|
def available(self): # type: ignore[no-untyped-def]
|
||||||
|
return True
|
||||||
|
|
||||||
|
def build(self, root): # type: ignore[no-untyped-def]
|
||||||
|
from openultrasast.cpg.backend import CpgResult
|
||||||
|
|
||||||
|
def run(query, params): # type: ignore[no-untyped-def]
|
||||||
|
invocations.append(f"single:{query}")
|
||||||
|
return []
|
||||||
|
|
||||||
|
def run_batch(query, requests): # type: ignore[no-untyped-def]
|
||||||
|
invocations.append(f"batch:{query}")
|
||||||
|
return {rid: [] for rid in requests}
|
||||||
|
|
||||||
|
result = CpgResult(cpg_path=Path("c.bin"), run=run)
|
||||||
|
object.__setattr__(result, "run_batch", run_batch)
|
||||||
|
return result
|
||||||
|
|
||||||
|
source = "import os\n\n\ndef run(cmd):\n os.system(cmd)\n"
|
||||||
|
for i in range(30):
|
||||||
|
(tmp_path / f"f{i}.py").write_text(source)
|
||||||
|
regions = tuple(_region(path=f"f{i}.py", families=("injection", "path")) for i in range(30))
|
||||||
|
|
||||||
|
scan_repository(tmp_path, regions, backend=_CountingBackend(), client=None, model="")
|
||||||
|
assert not any(i.startswith("single:") for i in invocations), f"unbatched calls remain: {invocations[:4]}"
|
||||||
|
# 30 regions x 2 families would have been 60 invocations; batching makes it one per query kind.
|
||||||
|
assert len(invocations) <= 3, f"expected one invocation per query kind, got {len(invocations)}"
|
||||||
|
|||||||
Reference in New Issue
Block a user