mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-20 21:49:28 +00:00
feat(scan): size portions from measured per-answer cost and isolate the kill culprit
The 7.8 trace showed the completion loss was a sizer death spiral, not expensive families: after two kills the budget fell to 3 sink visits and never recovered, because portions finishing in 25-30 s sat under an assumed 70 s fixed start and read as "no information", so the budget could only shrink. - taint.sc streams each answer's own cost (`ms`); the backend carries it as a reserved `__timing__` list through shards, partitions and reuse. - The sizer budgets predicted seconds from per-family seconds-per-visit rates learned from those costs, and measures the fixed start as the wall time the answers do not explain (median), instead of assuming it. - A kill names its culprit: the first unanswered id in request order. The ids behind it never started and are re-queued at once; the culprit is asked alone after the rest. One that still fails alone after more than a portion's target is reported as `query_too_expensive`, apart from `query_failed`. - Portions grow id by id from the queue head, and a single-portion scan now goes through the queue too, so a kill there re-asks its remainder. - Fix: the reuse wrapper wrapped only `run_batch`, so since 7.8 every portioned query bypassed the graph-reuse cache and its source-root normalization. `run_batch_once` is wrapped too. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
c1a689f4b3
commit
1b28465a61
@@ -886,6 +886,12 @@
|
||||
println(ujson.write(ujson.Obj("__census__" -> census)))
|
||||
System.out.flush()
|
||||
parsed.foreach { case (id, req) =>
|
||||
// What THIS answer cost, measured where it is spent. The driver sizes its portions from it, learns the
|
||||
// fixed start as the wall time the answers do not account for, and -- when a portion is killed -- knows
|
||||
// exactly which request was running, because every one before it reported its own time. A per-portion
|
||||
// wall clock cannot tell one 90 s request from thirty 3 s ones, and fitting a single rate to both
|
||||
// collapsed the sizer to three sink visits per JVM start on the plugin.
|
||||
val started = System.nanoTime()
|
||||
def field(name: String): String = req.obj.get(name).map(_.str).getOrElse("")
|
||||
val paramSrc = req.obj.get("parameterSources").map(_.str).getOrElse("false")
|
||||
val fieldSrc = req.obj.get("fieldSources").map(_.str).getOrElse("true")
|
||||
@@ -910,7 +916,7 @@
|
||||
field("contextEvidence")
|
||||
): _*
|
||||
)
|
||||
println(ujson.write(ujson.Obj("id" -> id, "rows" -> rows)))
|
||||
println(ujson.write(ujson.Obj("id" -> id, "rows" -> rows, "ms" -> ((System.nanoTime() - started) / 1000000L).toDouble)))
|
||||
System.out.flush()
|
||||
}
|
||||
// The census rides the stream as its FIRST line rather than costing its own invocation, because JVM
|
||||
|
||||
Reference in New Issue
Block a user