marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-19 13:10:03 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(cpg): portion the work instead of capping it -- a kill keeps what finished

Browse Source 
A hard per-invocation ceiling is all-or-nothing. A batch killed at 301 s lost every answer it had
already computed, and the retry paid a fresh JVM start per part to ask them all again: measured, 116
requests killed at 300 s, then eight restarts carrying the same everywhere-sinks in every part.

Three coordinated changes make portioning, not the cap, the thing that governs cost.

  The taint query STREAMS. The census first, then one line per request as it is answered, flushed as
  it goes, inside the unchanged fence. The other queries do not stream and need not; the driver
  reads both forms, and a legacy single document parses exactly as before.

  The driver HARVESTS a kill. The process group is terminated first -- a first cut read the pipe
  before killing and a pipe whose writer is alive never reaches end-of-file; the suite own
  hanging-engine control sat on that read for twenty-eight minutes -- then the pipe is drained and
  every complete line becomes an answer, marked partial. The retry asks only the remainder.

  Portion sizing ADAPTS. After each portion the observed seconds per sink visit, less the fixed
  start, resizes the rest toward a 120 s target; a portion that did not answer halves the budget
  rather than being fitted, because its true cost is unknown and above the ceiling. A portion cut
  down to fit puts its remainder back at the front of the queue.

A census LINE carries nothing but the census. A first cut misread a legacy document that carried the
census beside its answers as a census line and dropped every answer it sat beside; a control caught
it. Task 7.8 in contributor-scan.

Verified: 1,423 host tests pass, nineteen skipped, including new controls for the assembler in both
forms and under a kill, for a retry that asks only the remainder, and for the sizer. Not yet
verified: the joern-gated streaming check and the plugin measurement, which wait for the transfer
replay to free the machine and are stated as open in the task.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-09-22 21:03:04 +02:00
co-authored by Claude Fable 5.1
parent 1c9d7f501f
commit a8b226a5c8
7 changed files with 316 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/openultrasast/cpg/queries/taint.sc
+18 -10
View File
@@ -874,15 +874,23 @@
)
}
println("---OUSAST-CPG-BEGIN---")
if (requestsJson.nonEmpty) {
val parsed = ujson.read(requestsJson).obj
val answers = parsed.map { case (id, req) =>
// STREAMED: the census first, then one line per request as it is answered, flushed as it goes. What
// this buys is what a kill keeps. Built as one object and printed at the end, a batch killed at its
// ceiling lost every answer it had finished -- 116 requests at 300 s, then eight JVM starts to ask them
// all again. Printed as it goes, the same kill keeps everything that finished, the driver harvests it,
// and only the unfinished requests are asked again. The fence is unchanged; the driver reads both forms.
val census = ujson.Arr(ujson.Obj("methods" -> cpg.method.size.toString, "files" -> cpg.file.size.toString, "file_names" -> ujson.Arr.from(cpg.file.name.l)))
println("---OUSAST-CPG-BEGIN---")
println(ujson.write(ujson.Obj("__census__" -> census)))
System.out.flush()
parsed.foreach { case (id, req) =>
def field(name: String): String = req.obj.get(name).map(_.str).getOrElse("")
val paramSrc = req.obj.get("parameterSources").map(_.str).getOrElse("false")
val fieldSrc = req.obj.get("fieldSources").map(_.str).getOrElse("true")
val evidence = req.obj.get("evidenceOnly").map(_.str).getOrElse("false")
id -> ujson.Arr(
val rows = ujson.Arr(
rowsFor(
field("sources"),
field("sinks"),
@@ -902,15 +910,15 @@
field("contextEvidence")
): _*
)
println(ujson.write(ujson.Obj("id" -> id, "rows" -> rows)))
System.out.flush()
}
// A census of the graph, under a key no request id can collide with (ids are numbers). A frontend can
// fail every file and STILL exit 0 with a valid, empty CPG -- `joern-parse` does not even propagate the
// per-file warnings -- so "no rows" and "no graph" are indistinguishable to the driver without this.
// It rides the batch rather than costing its own invocation, because JVM startup is what this whole
// batching design exists to avoid.
val census = ujson.Arr(ujson.Obj("methods" -> cpg.method.size.toString, "files" -> cpg.file.size.toString, "file_names" -> ujson.Arr.from(cpg.file.name.l)))
println(ujson.write(ujson.Obj.from(answers.toSeq :+ ("__census__" -> census))))
// The census rides the stream as its FIRST line rather than costing its own invocation, because JVM
// startup is what this whole batching design exists to avoid -- and first because a frontend that fails
// every file still exits 0 with an empty graph, and the driver must see that even when the process is
// killed before any request answers.
} else {
println("---OUSAST-CPG-BEGIN---")
println(
ujson.write(ujson.Arr(rowsFor(sources, sinks, sanitizers, hookCallbacks, dispatchApply, function, parameterSources, fieldSources, evidenceOnly, file, callDepth, boundedSinks): _*))
)