diff --git a/.kiro/specs/contributor-scan/tasks.md b/.kiro/specs/contributor-scan/tasks.md index 5c01ae9..8c09e8f 100644 --- a/.kiro/specs/contributor-scan/tasks.md +++ b/.kiro/specs/contributor-scan/tasks.md @@ -1284,7 +1284,7 @@ the ontology describes libraries, frameworks and language constructs, never repo - _Status 2026-09-22:_ implemented; host controls pass for the assembler in both forms and under a kill, for the remainder-only retry, and for the sizer; the joern-gated check passes on the real engine — the taint query streams one line per request, and the comment-sink and literal-destination controls hold under the streamed form. The plugin measurement is running on a frozen export of the committed source. - _Two things the measurement exposed on the way, both fixed:_ the evidence pass was one unportioned batch protected only by the ceiling — 22,125 requests on a plugin push transaction — and now runs in portions of 4,000; and a first cut of the harvest read a timed-out process's pipe BEFORE killing it, which blocks forever because a pipe whose writer is alive never reaches end-of-file. The suite's own hanging-engine control caught that after twenty minutes of what I mistook for CPU starvation. - _A stated limit:_ the reuse layer's re-query of `missing` ids calls the backend batch directly and so bypasses sink-weighted portioning. Those re-queries are the small remainder of a cached scan, and streaming still keeps what a killed one finished, so it is recorded here rather than fixed. - - _End-to-end tuning, 2026-09-23, the honest version._ The first end-to-end run REGRESSED completion (64 of 406 against 7.7's 153) while finding as many defects, and each fix exposed the next: the adaptive sizer read one portion at 953 s because it timed `run_batch`, whose own split-retry folded in, and collapsed; giving the portioner a single-attempt path (`run_batch_once`) fixed the timing but only after it was wired on the PARTITIONED graph too, which every plugin uses and which the first wiring missed; and even then completion sat at 56 of 406 because the portioner iterated a fixed list and dropped the remainders of killed portions, stopping at 1,263 s of a 3,600 s budget. The queue re-asks those remainders. The sizer now adapts both ways in the trace — 268 up to 535 after a cheap portion, halved to 267 to 133 on kills — and re-asks what a kill left. Each step is unit-tested: the assembler in both forms and under a kill, the single-attempt path, coverage-halving, and queue termination. The end-to-end completion number is being re-measured on a frozen tree; findings rose to 141 and the declared target is detected under the streamed form. + - _End-to-end tuning, 2026-09-23, the honest version._ The first end-to-end run REGRESSED completion (64 of 406 against 7.7's 153) while finding as many defects, and each fix exposed the next: the adaptive sizer read one portion at 953 s because it timed `run_batch`, whose own split-retry folded in, and collapsed; giving the portioner a single-attempt path (`run_batch_once`) fixed the timing but only after it was wired on the PARTITIONED graph too, which every plugin uses and which the first wiring missed; and even then completion sat at 56 of 406 because the portioner iterated a fixed list and dropped the remainders of killed portions, stopping at 1,263 s of a 3,600 s budget. The queue re-asks those remainders. The sizer now adapts both ways in the trace — 268 up to 535 after a cheap portion, halved to 267 to 133 on kills — and re-asks what a kill left. Each step is unit-tested: the assembler in both forms and under a kill, the single-attempt path, coverage-halving, and queue termination. _Final end-to-end 2026-09-23, mixed and told straight._ Against the hard-cap-plus-reserve baseline on the same 500-region PHP run: findings IDENTICAL at 125 either way, completion slightly LOWER under portioning (131 of 406 against 153), wall time higher (2,982 s against 2,338). Portioning does not beat the cap on throughput here; it matches it on what it finds. What it buys is kill-safety -- the maintainer's actual ask -- and reaching the same 125 findings is the evidence that streaming loses nothing the cap kept while no longer discarding a near-complete batch when the ceiling falls mid-answer. Completion did not rise because the requeue re-asks a killed portion's remainder, and on this subject those are the expensive output-encoding questions that keep hitting the ceiling in smaller pieces; telling a transient kill from a genuinely-too-expensive family is the next lever, not more portioning. Neither approach is release-ready regardless: 32-38% completion against a 95% gate, on development data that qualifies nothing. Evidence: `benchmarks/measurements/2026-09-23-portioning-vs-hard-cap.json`. - _Boundary: cpg backend, contributor-scan scan budget, taint query_ - _Depends: 7.7_ - _Requirements: 4.1, 6.3, 8.1_ diff --git a/benchmarks/measurements/2026-09-23-portioning-vs-hard-cap.json b/benchmarks/measurements/2026-09-23-portioning-vs-hard-cap.json new file mode 100644 index 0000000..6832e61 --- /dev/null +++ b/benchmarks/measurements/2026-09-23-portioning-vs-hard-cap.json @@ -0,0 +1,27 @@ +{ + "schema_version": 1, + "experiment": "portioning-vs-hard-cap-v1", + "date": "2026-09-23", + "task": "contributor-scan 7.8", + "claim_scope": "Portioning with streamed harvest against the hard-cap-plus-reserve baseline, on one PHP subject at 500 regions, 3600 s.", + "hard_cap_7_7": { + "completed": 153, + "of": 406, + "findings": 125, + "seconds": 2338.2 + }, + "portioned_7_8": { + "completed": 131, + "of": 406, + "findings": 125, + "seconds": 2982.0 + }, + "reading": "A mixed result, told straight. Findings are IDENTICAL, 125 either way -- the detection output does not change. Completion is slightly LOWER under portioning, 131 of 406 against 153, and it used more wall time, 2,982 s against 2,338. So on this subject portioning does not beat hard-cap-plus-reserve on throughput; it matches it on what it finds.", + "what_portioning_actually_buys": "Kill-safety, which is what the maintainer asked for. Under the hard cap a batch killed at the ceiling loses every answer it computed; under streaming it keeps them. The two approaches reaching the same 125 findings is the evidence that streaming loses nothing the cap kept -- and it no longer discards a near-complete batch's work when the ceiling falls mid-answer. The completion gap is not a detection gap.", + "why_completion_did_not_rise": "The requeue re-asks a killed portion's remainder, and on this subject those remainders are the expensive output-encoding questions that keep hitting the ceiling in smaller and smaller pieces, spending budget without completing. The arbitration reserve then stops the query phase (query_budget_reserved) with those still unresolved. A cap that simply drops them, plus the reserve, happened to leave more budget for the questions that could complete. Honest read: requeue helps a transient kill and does not help a genuinely-too-expensive family, and telling those apart is the next lever, not more portioning.", + "neither_is_release_ready": "131 or 153 of 406 is 32-38% completion on a bounded 500-region PHP run; the M4 gate is 95%. Neither approach makes PHP release-qualifying, and this subject is development data that qualifies nothing. The value delivered here is robustness of the engine path, not a PHP capability.", + "limits": [ + "One subject, one language, one budget. The comparison may invert on a repository whose expensive family is rarer.", + "The reserve share and the portion target are first values, not tuned; the completion gap is within the range those could move." + ] +}