fix(scan): the portioner asks single attempts and reads coverage, not wall time

End-to-end, the adaptive portioner regressed completion -- 64 of 406 against 7.7 hard-cap 153 --
while finding the same defects. The cause: the portioner timed each portion with run_batch, which
splits on failure internally, so one portion folded its own eight-way retry into its number and read
953 s. The sizer fitted a per-visit rate from that and collapsed its budget to 14 visits, spending
the back half of the deadline on tiny portions.

Two portioning layers were fighting. The portioner is now the sole retry layer: it asks
run_batch_once, a single invocation whose kill returns the answers it streamed, so its wall time is
the cost of one attempt. And it observes COVERAGE, not emptiness -- a portion that came back short
of the ids it asked halves the budget, because it was too big for the ceiling; only a whole answer
gives a clean per-visit sample to fit from.

run_batch_once is new on CpgResult, wired on the disposable and sharded paths; a backend without it
falls back to run_batch unchanged. Host controls cover the single-attempt path and the
halve-on-short-answer rule.

1,424 tests pass, nineteen skipped. The end-to-end plugin re-measure on a frozen tree is the open
item; the mechanism (a kill keeps what finished) and the sizer logic are unit-verified.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-09-22 23:00:14 +02:00
co-authored by Claude Opus 4.8
parent 2e96ccb8ea
commit 0fef399869
3 changed files with 65 additions and 4 deletions
+12 -2
View File
@@ -427,8 +427,13 @@ def _scan_repository_impl(
kind_started = time.monotonic()
batch = getattr(cpg, "run_batch", None)
if callable(batch):
# The portioner is the retry layer, so it asks in SINGLE attempts: a portion timed with
# `run_batch` folds that method's own split-retry into the number, which once read 953 s for one
# portion and collapsed the sizer. `run_batch_once` is one invocation whose kill returns what it
# streamed. A backend without it falls back to `run_batch`, unchanged.
once = getattr(cpg, "run_batch_once", None)
answered_batch = _batched(
batch,
once if callable(once) else batch,
kind,
requests,
weights=_sink_weights(work, evidence_by_pair),
@@ -910,7 +915,12 @@ def _batched(
chunk = sizer.fit(chunk, pending=chunks[index + 1 :])
started = time.monotonic()
answer = batch(kind, chunk)
sizer.observe(chunk, time.monotonic() - started, answered=bool(answer))
# WHOLE, not merely non-empty. A killed portion streams some answers and returns them, which is the
# point -- but its time is the ceiling, not the cost of the work, so fitting a per-visit rate from it
# is meaningless. Only a portion that answered every id it was asked gives a clean sample; a short
# one halves the budget, because it was too big for the ceiling.
whole = answer is not None and all(rid in answer for rid in chunk)
sizer.observe(chunk, time.monotonic() - started, answered=whole)
if answer is None:
continue
answered_any = True