mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
Marius CiepluchandClaude Opus 4.8
c1a689f4b3
Final end-to-end on the 500-region PHP subject: findings identical at 125 either way, completion slightly lower under portioning (131 of 406 vs the hard-cap-plus-reserve 153), wall time higher (2,982 vs 2,338 s). Told straight: portioning does not beat the cap on throughput here; it matches it on what it finds, and what it buys is kill-safety -- a killed batch keeps its streamed answers instead of losing them. Reaching the same 125 findings is the evidence that streaming loses nothing the cap kept. Completion did not rise because the requeue re-asks a killed portion remainder, and on this subject those are the expensive output-encoding questions that keep hitting the ceiling in smaller pieces, spending budget without completing. Telling a transient kill from a too-expensive family is the next lever, not more portioning. Neither approach is release-ready anyway: 32-38% completion against a 95% gate on development data that qualifies nothing. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
28 lines
2.5 KiB
JSON
28 lines
2.5 KiB
JSON
{
|
|
"schema_version": 1,
|
|
"experiment": "portioning-vs-hard-cap-v1",
|
|
"date": "2026-09-23",
|
|
"task": "contributor-scan 7.8",
|
|
"claim_scope": "Portioning with streamed harvest against the hard-cap-plus-reserve baseline, on one PHP subject at 500 regions, 3600 s.",
|
|
"hard_cap_7_7": {
|
|
"completed": 153,
|
|
"of": 406,
|
|
"findings": 125,
|
|
"seconds": 2338.2
|
|
},
|
|
"portioned_7_8": {
|
|
"completed": 131,
|
|
"of": 406,
|
|
"findings": 125,
|
|
"seconds": 2982.0
|
|
},
|
|
"reading": "A mixed result, told straight. Findings are IDENTICAL, 125 either way -- the detection output does not change. Completion is slightly LOWER under portioning, 131 of 406 against 153, and it used more wall time, 2,982 s against 2,338. So on this subject portioning does not beat hard-cap-plus-reserve on throughput; it matches it on what it finds.",
|
|
"what_portioning_actually_buys": "Kill-safety, which is what the maintainer asked for. Under the hard cap a batch killed at the ceiling loses every answer it computed; under streaming it keeps them. The two approaches reaching the same 125 findings is the evidence that streaming loses nothing the cap kept -- and it no longer discards a near-complete batch's work when the ceiling falls mid-answer. The completion gap is not a detection gap.",
|
|
"why_completion_did_not_rise": "The requeue re-asks a killed portion's remainder, and on this subject those remainders are the expensive output-encoding questions that keep hitting the ceiling in smaller and smaller pieces, spending budget without completing. The arbitration reserve then stops the query phase (query_budget_reserved) with those still unresolved. A cap that simply drops them, plus the reserve, happened to leave more budget for the questions that could complete. Honest read: requeue helps a transient kill and does not help a genuinely-too-expensive family, and telling those apart is the next lever, not more portioning.",
|
|
"neither_is_release_ready": "131 or 153 of 406 is 32-38% completion on a bounded 500-region PHP run; the M4 gate is 95%. Neither approach makes PHP release-qualifying, and this subject is development data that qualifies nothing. The value delivered here is robustness of the engine path, not a PHP capability.",
|
|
"limits": [
|
|
"One subject, one language, one budget. The comparison may invert on a repository whose expensive family is rarer.",
|
|
"The reserve share and the portion target are first values, not tuned; the completion gap is within the range those could move."
|
|
]
|
|
}
|