fix(cpg): the real-request cost, attributed and removed -- overlays once, seeds scoped, data tables out

Task 5.13, measured on ten real pmpro pairs at callDepth 1 with each join half switched separately and
the 54 s JVM fixed cost subtracted: both halves off 11.1 s/request, field half 10.9 s, hook half past
49 s (timeout). Stage timers put it in `hookSeedsIn("")`: every field access in all 7,061 methods put
through the field join before the first callback was answered. Seeds are now the callback's own body
and file, which is where WP Statistics's `set_current_page` and `$this->rest_hits = ...` both live.
Both halves on: 9.7 s/request. `sourceNodes` is also materialised once per request instead of once
per sink.

The WP Statistics re-verification then exposed the larger term. A `joern --script` run is two JVMs and
`-J-Xmx` sizes only the launcher, so the heap knob governed nothing; the backend now sets
JAVA_TOOL_OPTIONS too. And since PHP builds went through php2cpg directly (1ce7e9d, an hour after the
hook half was verified from a joern-parse build), every query batch has recomputed the dataflow overlay
at load -- 54 s on pmpro, and on WP Statistics an OutOfMemoryError inside OssDataFlow after 21 minutes
at 2 GB, no completion in 25 minutes at 4 GB, and no help from --max-num-def at 1000 or 200. Two
vendored browser-profile tables, one array literal each, were 60% of that graph.

Three changes, all named in the log: `queries/overlay.sc` applies the overlays once at build time and
the saved graph replaces the raw one (joern-parse --overlaysonly NPEs in 4.0.623); a source file over
1 MB is excluded as a data table and reported in `unparsed`; and the seed scoping above. After: the WP
Statistics graph reloads in 8 s, the `record` request that timed out at 520 s answers in 39 s including
the JVM, and CVE-2022-25148 is entailed again with `sourceKind = hook`. A raw-vs-saved payload diff on
pmpro shows identical evidence keys and fewer duplicate flow rows.

Also: the backend's own unit tests left fifteen ousast-cpg-* directories in /tmp per run; conftest now
keeps tempfile under the test's directory. The compile test covers overlay.sc.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG

src/openultrasast/cpg/backend.py

@@ -87,6 +87,15 @@ _FRONTENDS = {
 _PREFER_FRONTEND = frozenset({"php"})
 # The extensions that count as source for a language, used to ask whether a graph is missing anything.
 _LANGUAGE_EXTENSIONS: dict[str, tuple[str, ...]] = {"php": (".php",)}
+
+# A source file above this size is a data table, not code: WP Statistics carries two vendored browser
+# profile files of 1.6 MB and 1.5 MB, each one array literal, and they were 60% of the graph (9.6 MB with
+# them, 3.6 MB without). The reaching-definitions overlay does not merely slow down on them, it never
+# finishes -- 25 minutes at 4 GB, OutOfMemoryError inside `initGen` at every cap `--max-num-def` offers,
+# because a single assignment of a hundred-thousand-element literal is ONE definition. Nothing flows
+# through a lookup table that a scan could name, and the file is named in the log so the exclusion is
+# visible rather than silent. Task 5.13.
+MAX_SOURCE_BYTES = 1_000_000
 # Frontends that shell out to a separate interpreter, and the command that proves it can read a file.
 # `php2cpg` drives PHP-Parser through whatever `php` is on PATH, so the frontend is only as good as that
 # interpreter's view of the filesystem -- and a `php` that cannot see the tree does not fail, it produces an
@@ -287,6 +296,7 @@ class JoernBackend:
             retried = self._build_with_frontend(root, cpg_path, scratch, language)
             if retried is None:
                 return None
+            self._apply_overlays(cpg_path, scratch)
             unparsed = retried
         if unparsed:
             logger.warning("the frontend could not parse %d file(s) under %s: %s", len(unparsed), root, ", ".join(unparsed[:5]))
@@ -298,6 +308,22 @@ class JoernBackend:
             cleanup=dispose,
         )
 
+    def _files_too_large_to_flow(self, root: Path, language: str) -> tuple[str, ...]:
+        """Source files over `MAX_SOURCE_BYTES`, relative to the root, in the language the build is about."""
+        extensions = _LANGUAGE_EXTENSIONS.get(language.lower())
+        if not extensions:
+            return ()
+        found: list[str] = []
+        for path in sorted(root.rglob("*")):
+            if path.suffix.lower() not in extensions or not path.is_file():
+                continue
+            try:
+                if path.stat().st_size > MAX_SOURCE_BYTES:
+                    found.append(str(path.relative_to(root)))
+            except OSError:
+                continue
+        return tuple(found)
+
     def _files_with_frontend_defect(self, root: Path, language: str) -> tuple[str, ...]:
         """Files carrying a construct this frontend miscompiles, found by reading the source.
 
@@ -386,6 +412,16 @@ class JoernBackend:
                 root,
                 ", ".join(Path(name).name for name in defective[:5]),
             )
+        oversized = self._files_too_large_to_flow(root, language)
+        if oversized:
+            logger.warning(
+                "excluding %d file(s) under %s over %d bytes (data tables the dataflow overlay never finishes on): %s",
+                len(oversized),
+                root,
+                MAX_SOURCE_BYTES,
+                ", ".join(f"{Path(name).name} ({(root / name).stat().st_size // 1024} KB)" for name in oversized[:5]),
+            )
+            defective = (*defective, *oversized)
 
         main = scratch / "cpg.bin"
         dropped = self._build_with_retries(root, main, scratch, language, exclude=defective)
@@ -522,8 +558,52 @@ class JoernBackend:
         merged["__census__"] = [{"methods": str(methods), "files": str(files), "shards": str(len(shards))}]
         return merged
 
+    def _apply_overlays(self, cpg_path: Path, scratch: Path) -> bool:
+        """Run Joern's default overlays ONCE, at build time, and keep the result in the graph.
+
+        A frontend writes a raw graph. Everything that answers a dataflow question -- the call graph, the
+        type layer, the reaching-definitions pass `OssDataFlow` -- is an OVERLAY, and `importCpg` computes
+        every overlay the graph lacks each time a JVM opens it. `joern-parse` applies them once and saves;
+        `php2cpg` output has none, so from the day PHP builds went through the frontend directly
+        (2026-09-09, 1ce7e9d) every query batch has paid the whole pass again before its first request.
+        Measured: 54 s of fixed cost per batch on a 637-file plugin, and on WP Statistics the pass alone
+        ran the default 2 GB heap out of memory after 21 minutes -- a graph the same tool had entailed
+        CVE-2022-25148 on, one hour before the switch, from a `joern-parse` build. Task 5.13.
+
+        `queries/overlay.sc` is that once: `importCpg`, `save`, and the saved graph replaces the raw one
+        under the same path. (`joern-parse --overlaysonly` is the obvious tool and is broken in 4.0.623.)
+        A failure here is logged and the raw graph kept, so the scan degrades to the old cost rather than
+        losing its graph.
+        """
+        joern = shutil.which("joern")
+        if joern is None:
+            return False
+        script = self.queries_dir / "overlay.sc"
+        command = [joern, self._heap_flag(), "--script", str(script), "--param", f"cpgFile={cpg_path}"]
+        completed = self._run(command, timeout=self.build_timeout, cwd=scratch)
+        workspace = scratch / "workspace"
+        saved = workspace / cpg_path.name / "cpg.bin"
+        try:
+            answered = completed is not None and completed.returncode == 0 and extract_payload(completed.stdout or "") is not None
+            if not answered or not saved.is_file():
+                detail = (completed.stderr or completed.stdout or "")[-400:] if completed is not None else "timeout"
+                logger.warning("overlays could not be applied to %s, keeping the raw graph: %s", cpg_path, detail)
+                return False
+            saved.replace(cpg_path)
+            return True
+        finally:
+            shutil.rmtree(workspace, ignore_errors=True)
+
     def _build_with_retries(
         self, root: Path, cpg_path: Path, scratch: Path, language: str, exclude: Sequence[str] = ()
+    ) -> tuple[str, ...] | None:
+        outcome = self._build_with_retries_raw(root, cpg_path, scratch, language, exclude)
+        if outcome is not None:
+            self._apply_overlays(cpg_path, scratch)
+        return outcome
+
+    def _build_with_retries_raw(
+        self, root: Path, cpg_path: Path, scratch: Path, language: str, exclude: Sequence[str] = ()
     ) -> tuple[str, ...] | None:
         """Build through the frontend, trying again while it reports files it could not parse.
 
@@ -657,12 +737,30 @@ class JoernBackend:
             return None
         return extract_payload(completed.stdout or "")
 
-    def _heap_flag(self) -> str:
-        """``-J-Xmx``: both launchers forward ``-J`` arguments to the JVM."""
+    def _heap_mb(self) -> int:
         if self.heap_mb > 0:
-            return f"-J-Xmx{self.heap_mb}m"
+            return self.heap_mb
         configured = os.environ.get(HEAP_ENV, "").strip()
-        return f"-J-Xmx{configured if configured.isdigit() else CPG_HEAP_MB}m"
+        return int(configured) if configured.isdigit() else CPG_HEAP_MB
+
+    def _heap_flag(self) -> str:
+        """``-J-Xmx``: both launchers forward ``-J`` arguments to the JVM they start."""
+        return f"-J-Xmx{self._heap_mb()}m"
+
+    def _jvm_env(self) -> dict[str, str]:
+        """The heap for the JVM that does the work, which is NOT the one ``-J`` reaches.
+
+        ``joern --script`` starts a launcher JVM and forks a second one to run the script
+        (``replpp.scripting.NonForkingScriptRunner``, despite the name), and the ``-J-Xmx`` goes to the
+        launcher only: 163 MB resident, next to a worker with no ``-Xmx`` at all sitting at 2.26 GB -- the
+        JVM default of a quarter of physical memory. So ``OPENULTRASAST_CPG_HEAP_MB`` governed nothing on an
+        8 GB machine and would silently take 16 GB on a 64 GB one. ``JAVA_TOOL_OPTIONS`` is read by every
+        JVM at start-up, forked ones included, and is appended to rather than replaced so an operator's own
+        options survive.
+        """
+        env = dict(os.environ)
+        env["JAVA_TOOL_OPTIONS"] = f"{env.get('JAVA_TOOL_OPTIONS', '').strip()} -Xmx{self._heap_mb()}m".strip()
+        return env
 
     def _run(self, command: list[str], *, timeout: int, cwd: Path | None = None) -> subprocess.CompletedProcess[str] | None:
         """Run one Joern command, and on timeout kill the whole PROCESS GROUP rather than the child.
@@ -689,6 +787,7 @@ class JoernBackend:
                 stderr=subprocess.PIPE,
                 text=True,
                 cwd=str(cwd) if cwd else None,
+                env=self._jvm_env(),
                 start_new_session=True,  # its own process group, so the JVM dies with the script
             ) as process:
                 try:

src/openultrasast/cpg/queries/overlay.sc

@@ -0,0 +1,27 @@
+// Apply Joern's default overlays ONCE, at build time, and save them into the graph (contributor-scan 5.13).
+//
+// A frontend writes a raw graph. The call graph, the type layer and the reaching-definitions pass that
+// every dataflow question runs on are OVERLAYS, and `importCpg` computes each one the graph lacks every
+// time a JVM opens it. `joern-parse` applies them once and saves; a frontend run directly does not. So
+// from the day PHP builds went through php2cpg (2026-09-09, 1ce7e9d) every query batch recomputed the whole
+// dataflow layer before its first request: 54 s of fixed cost on a 637-file plugin, and on WP Statistics a
+// pass that ran the heap out after 21 minutes -- a graph the same tool had entailed CVE-2022-25148 on, an
+// hour before the switch, from a joern-parse build.
+//
+// Measured after: the saved graph reloads in 8 s where the raw one took 64 s, and the request that had
+// timed out at 520 s answered in 39 s including the JVM. `joern-parse --overlaysonly` would be the
+// obvious tool and is broken in 4.0.623 (NullPointerException in a post-processing step that needs the
+// frontend it did not run), so the once is this script.
+//
+// `save` writes to `workspace/<cpg name>/cpg.bin` under the working directory; the driver moves that over
+// the raw graph.
+//
+// Parameters: cpgFile.
+// Output: fenced JSON {files, methods}.
+@main def exec(cpgFile: String) = {
+  importCpg(cpgFile)
+  save
+  println("---OUSAST-CPG-BEGIN---")
+  println(ujson.write(ujson.Obj("files" -> cpg.file.size.toString, "methods" -> cpg.method.size.toString)))
+  println("---OUSAST-CPG-END---")
+}

src/openultrasast/cpg/queries/taint.sc

@@ -354,23 +354,21 @@
   // every registered callback "return tainted" and the join degenerates into the complete graph this design
   // exists to refuse. Only a real framework source, or a field already shown to hold one, counts.
   //
-  // They are also NOT scoped to the requesting file, which the field half is. A hook registry is global by
-  // construction: the whole point of the edge is that `set_current_page` lives in one file and the
-  // `apply_filters` that calls it lives in another. Scoping these seeds the way the field half scopes its
-  // own defeats exactly the join being built, and the query returned nothing until this was separated.
-  val hookSeedsByFile = scala.collection.mutable.Map.empty[String, List[io.shiftleft.codepropertygraph.generated.nodes.CfgNode]]
-
-  def hookSeedsIn(fileName: String) =
-    hookSeedsByFile.getOrElseUpdate(
-      fileName, {
-        val methods = methodsIn(fileName)
-        val framework: List[io.shiftleft.codepropertygraph.generated.nodes.CfgNode] =
-          methods.flatMap(_.ast.isCall.filter(c => sourcePatterns.exists(p => c.code.contains(p))).l)
-        val fields: List[io.shiftleft.codepropertygraph.generated.nodes.CfgNode] =
-          methods.flatMap(_.ast.isCall.nameExact(FIELD_ACCESS).l).filter(r => taintedPrefixes(r.code.trim, fileName))
-        framework ++ fields
-      }
-    )
+  // They are NOT scoped to the requesting file, which the field half is: a hook registry is global by
+  // construction, `set_current_page` lives in one file and the `apply_filters` that calls it in another,
+  // and the query returned nothing while they were. They ARE scoped to the CALLBACK's file. The first
+  // version scoped them to nothing -- every field access in every method of the repository, each put through
+  // the field join -- and on a 637-file plugin that step alone did not finish in 270 s, for a callback whose
+  // sources are, by the join's own logic, the reads in its body and the assignments in its class. WP
+  // Statistics is the case that has to survive: the callback and `$this->rest_hits = ...` share a file,
+  // and only the sink is elsewhere. Task 5.13 has the numbers.
+  def hookSeedsOf(callback: io.shiftleft.codepropertygraph.generated.nodes.Method) = {
+    val framework: List[io.shiftleft.codepropertygraph.generated.nodes.CfgNode] =
+      callback.ast.isCall.filter(c => sourcePatterns.exists(p => c.code.contains(p))).l
+    val fields: List[io.shiftleft.codepropertygraph.generated.nodes.CfgNode] =
+      callback.ast.isCall.nameExact(FIELD_ACCESS).l.filter(r => taintedPrefixes(r.code.trim, callback.filename))
+    framework ++ fields
+  }
 
   // Half one is asked PER ARRAY KEY, not per callback, and that distinction is the whole result.
   //
@@ -385,39 +383,40 @@
   // and not a detection. The key is a literal at both ends -- written `"id"` in the callback and `['id']`
   // at the sink -- so it joins the same way the hook name and the field name do. Third use of one idea.
   val INDEX_ACCESS = "<operator>.indexAccess"
-  val hookKeysMemo = scala.collection.mutable.Map.empty[(String, String), Set[String]]
+  val hookKeysMemo = scala.collection.mutable.Map.empty[String, Set[String]]
 
   def keyOf(access: io.shiftleft.codepropertygraph.generated.nodes.Call): String =
     access.argument.l.lift(1).map(a => unquote(a.code)).getOrElse("")
 
-  def taintedKeysOf(fileName: String, callback: String): Set[String] =
+  def taintedKeysOf(callback: String): Set[String] =
     hookKeysMemo.getOrElseUpdate(
-      (fileName, callback), {
-        val seeds = hookSeedsIn(fileName)
-        if (seeds.isEmpty) Set.empty[String]
-        else
-          cpg.method
-            .nameExact(callback)
-            .ast
-            .isCall
-            .nameExact(ASSIGNMENT)
-            .l
-            .flatMap { assignment =>
-              val args = assignment.argument.l
-              if (args.size < 2) None
-              else
-                args.head match {
-                  case target: io.shiftleft.codepropertygraph.generated.nodes.Call if target.name == INDEX_ACCESS =>
-                    val key = keyOf(target)
-                    if (key.isEmpty) None
-                    else if (args(1).start.reachableByFlows(seeds.iterator).l.exists(f => !f.elements.l.exists(sanitizesHere)))
-                      Some(key)
-                    else None
-                  case _ => None
-                }
-            }
-            .toSet
-      }
+      callback,
+      cpg.method
+        .nameExact(callback)
+        .l
+        .flatMap { method =>
+          val seeds = hookSeedsOf(method)
+          if (seeds.isEmpty) Nil
+          else
+            method.ast.isCall
+              .nameExact(ASSIGNMENT)
+              .l
+              .flatMap { assignment =>
+                val args = assignment.argument.l
+                if (args.size < 2) None
+                else
+                  args.head match {
+                    case target: io.shiftleft.codepropertygraph.generated.nodes.Call if target.name == INDEX_ACCESS =>
+                      val key = keyOf(target)
+                      if (key.isEmpty) None
+                      else if (args(1).start.reachableByFlows(seeds.iterator).l.exists(f => !f.elements.l.exists(sanitizesHere)))
+                        Some(key)
+                      else None
+                    case _ => None
+                  }
+              }
+        }
+        .toSet
     )
 
   def hookSourceNodes =
@@ -428,7 +427,7 @@
       else {
         val keys = applies.flatMap { call =>
           val hook = call.argument.l.headOption.map(a => unquote(a.code)).getOrElse("")
-          hookTable.getOrElse(hook, Nil).flatMap(cb => taintedKeysOf("", cb))
+          hookTable.getOrElse(hook, Nil).flatMap(taintedKeysOf)
         }.toSet
         if (keys.isEmpty) Iterator.empty
         else
@@ -443,7 +442,16 @@
       }
     }
 
-  def sourceNodes = frameworkSources.l.iterator ++ parameterNodes ++ fieldSourceNodes ++ hookSourceNodes
+  // Materialised ONCE per request, because every one of the four is a `def` that walks the graph --
+  // `frameworkSources` scans every call in the repository, `hookSourceNodes` runs one `reachableBy` per
+  // candidate index access -- and `sourceNodes` is consumed once per SINK. A file-scope region with 346 sinks
+  // paid for 346 repository scans and 346 hook joins. Measured on ten real pmpro pairs at callDepth 1:
+  // 16.5 s per request with the hook half off, past 54 s with it on, before this; see task 5.13.
+  lazy val frameworkList = frameworkSources.l
+  lazy val parameterList = parameterNodes.toList
+  lazy val fieldList     = fieldSourceNodes.toList
+  lazy val hookList      = hookSourceNodes.toList
+  def sourceNodes = frameworkList.iterator ++ parameterList.iterator ++ fieldList.iterator ++ hookList.iterator
 
   // WHICH KIND of source a flow started from, reported per row so the driver can tell them apart.
   //
@@ -453,9 +461,9 @@
   // present, which is right for parameters and wrong for these: on WP Statistics, seven sanitized
   // `$_SERVER["REQUEST_URI"]` flows displaced the unsanitized hook flow carrying CVE-2022-25148, and the
   // finding disappeared between the query and the verdict.
-  lazy val frameworkIds = frameworkSources.l.map(_.id).toSet
-  lazy val fieldIds     = fieldSourceNodes.map(_.id).toSet
-  lazy val hookIds      = hookSourceNodes.map(_.id).toSet
+  lazy val frameworkIds = frameworkList.map(_.id).toSet
+  lazy val fieldIds     = fieldList.map(_.id).toSet
+  lazy val hookIds      = hookList.map(_.id).toSet
 
   def sourceKindOf(head: Option[io.shiftleft.codepropertygraph.generated.nodes.AstNode]): String =
     head match {