mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-19 13:10:03 +00:00
fix(cpg): the real-request cost, attributed and removed -- overlays once, seeds scoped, data tables out
Task 5.13, measured on ten real pmpro pairs at callDepth 1 with each join half switched separately and
the 54 s JVM fixed cost subtracted: both halves off 11.1 s/request, field half 10.9 s, hook half past
49 s (timeout). Stage timers put it in `hookSeedsIn("")`: every field access in all 7,061 methods put
through the field join before the first callback was answered. Seeds are now the callback's own body
and file, which is where WP Statistics's `set_current_page` and `$this->rest_hits = ...` both live.
Both halves on: 9.7 s/request. `sourceNodes` is also materialised once per request instead of once
per sink.
The WP Statistics re-verification then exposed the larger term. A `joern --script` run is two JVMs and
`-J-Xmx` sizes only the launcher, so the heap knob governed nothing; the backend now sets
JAVA_TOOL_OPTIONS too. And since PHP builds went through php2cpg directly (1ce7e9d, an hour after the
hook half was verified from a joern-parse build), every query batch has recomputed the dataflow overlay
at load -- 54 s on pmpro, and on WP Statistics an OutOfMemoryError inside OssDataFlow after 21 minutes
at 2 GB, no completion in 25 minutes at 4 GB, and no help from --max-num-def at 1000 or 200. Two
vendored browser-profile tables, one array literal each, were 60% of that graph.
Three changes, all named in the log: `queries/overlay.sc` applies the overlays once at build time and
the saved graph replaces the raw one (joern-parse --overlaysonly NPEs in 4.0.623); a source file over
1 MB is excluded as a data table and reported in `unparsed`; and the seed scoping above. After: the WP
Statistics graph reloads in 8 s, the `record` request that timed out at 520 s answers in 39 s including
the JVM, and CVE-2022-25148 is entailed again with `sourceKind = hook`. A raw-vs-saved payload diff on
pmpro shows identical evidence keys and fewer duplicate flow rows.
Also: the backend's own unit tests left fifteen ousast-cpg-* directories in /tmp per run; conftest now
keeps tempfile under the test's directory. The compile test covers overlay.sc.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
9a8e9f083f
commit
60963f565c
@@ -0,0 +1,27 @@
|
||||
// Apply Joern's default overlays ONCE, at build time, and save them into the graph (contributor-scan 5.13).
|
||||
//
|
||||
// A frontend writes a raw graph. The call graph, the type layer and the reaching-definitions pass that
|
||||
// every dataflow question runs on are OVERLAYS, and `importCpg` computes each one the graph lacks every
|
||||
// time a JVM opens it. `joern-parse` applies them once and saves; a frontend run directly does not. So
|
||||
// from the day PHP builds went through php2cpg (2026-09-09, 1ce7e9d) every query batch recomputed the whole
|
||||
// dataflow layer before its first request: 54 s of fixed cost on a 637-file plugin, and on WP Statistics a
|
||||
// pass that ran the heap out after 21 minutes -- a graph the same tool had entailed CVE-2022-25148 on, an
|
||||
// hour before the switch, from a joern-parse build.
|
||||
//
|
||||
// Measured after: the saved graph reloads in 8 s where the raw one took 64 s, and the request that had
|
||||
// timed out at 520 s answered in 39 s including the JVM. `joern-parse --overlaysonly` would be the
|
||||
// obvious tool and is broken in 4.0.623 (NullPointerException in a post-processing step that needs the
|
||||
// frontend it did not run), so the once is this script.
|
||||
//
|
||||
// `save` writes to `workspace/<cpg name>/cpg.bin` under the working directory; the driver moves that over
|
||||
// the raw graph.
|
||||
//
|
||||
// Parameters: cpgFile.
|
||||
// Output: fenced JSON {files, methods}.
|
||||
@main def exec(cpgFile: String) = {
|
||||
importCpg(cpgFile)
|
||||
save
|
||||
println("---OUSAST-CPG-BEGIN---")
|
||||
println(ujson.write(ujson.Obj("files" -> cpg.file.size.toString, "methods" -> cpg.method.size.toString)))
|
||||
println("---OUSAST-CPG-END---")
|
||||
}
|
||||
Reference in New Issue
Block a user