fix(scan): prune a tier-zero question unless ITS partition is the incomplete one

The mechanism already existed. A question whose evidence vector says no sink of its family is in
reach cannot produce a flow, and tier-zero pruning defers it. That was switched off for the whole
scan whenever any of twelve degradation reasons appeared, and a multi-language repository always
carries two of them, so the measured plugin arbitrated every question it could ask. Third instance
this week of one blast radius, and the invariant written into the design on Monday named it.

A prune blocker that names a partition now withholds pruning from that partition questions and from
nobody else. cross_partition_semantics_unresolved leaves the blocker set entirely, for the reason it
left the integrity gaps: a PHP family sink is PHP, so a second language cannot be why its vector saw
none.

Measured on Paid Memberships Pro. At 25 regions arbitration falls from 127 questions to 34 with 93
recorded deferred as tier_zero, and 19 findings before and after, identical family by family. At 500
it falls from 2,526 to 406 with 2,120 deferred.

It is not sufficient, and that is the useful half. The 500-region run still fails in the same way at
the same cost, because 406 questions at roughly one and a half seconds each still exceed the
300-second per-batch ceiling. This lever moved the volume; the ceiling is a separate limit.

The census now records what a scan skipped and why, per family, so a cheaper scan and a narrower one
stop looking identical: a family whose every question was pruned reports no sink of this family in
reach rather than region offered, no question asked.

1,420 tests pass, eighteen skipped.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-09-22 08:06:15 +02:00
co-authored by Claude Opus 5
parent b0500ecfa0
commit 3b6e642433
4 changed files with 105 additions and 22 deletions
+3
View File
@@ -1239,6 +1239,9 @@ the ontology describes libraries, frameworks and language constructs, never repo
- The structural pass answers 17,700 requests in 44 s; arbitration costs one to two seconds per question, and a 673-file plugin asks 2,526 of them. Gate arbitration on the evidence vector already computed: a question whose scope holds no modelled sink, or no reachable source, cannot produce a flow and does not need the expensive pass.
- This attacks the scale limit from the opposite side to graph reuse and can be sized today from data already on disk, by counting how many of those 2,526 questions had any sink in scope at all.
- Done when the arbitrated count falls without any finding being lost on the measured subjects, and the questions skipped are recorded as skipped rather than as answered.
- _Implemented and measured 2026-09-22, and the mechanism already existed._ Tier-zero pruning defers a question whose evidence vector says no sink of its family is in reach. It was switched off for the WHOLE SCAN whenever any of twelve degradation reasons appeared, and a multi-language repository always carries two of them, so the measured plugin arbitrated every question it could ask. Third instance this week of the same blast radius. A prune blocker that names a partition now withholds pruning from that partition's questions and nobody else's, and `cross_partition_semantics_unresolved` leaves the blocker set entirely for the reason it left the integrity gaps.
- _The arbitrated count falls and no finding is lost._ At 25 regions: 127 questions to 34, with 93 recorded deferred as `tier_zero`, and 19 findings before and after, identical family by family. At 500: 2,526 to 406, with 2,120 deferred. A family whose every question was pruned now reports "no sink of this family in reach" rather than "region offered, no question asked", because nothing to ask is a different statement from never offered.
- **It is not sufficient, and that is the useful half of the result.** The 500-region run still fails in the same way at the same cost, because 406 questions at roughly one and a half seconds each still exceed the 300-second per-batch ceiling. This lever moved the volume; the ceiling is a separate limit and pre-push 13.3 still cannot reach the population that holds its target without it. Evidence: `benchmarks/measurements/2026-09-22-tier-zero-pruning.json`.
- _Boundary: contributor-scan scan budget_
- _Requirements: 4.1, 6.3, 8.1_
- [ ] 7.4 One matcher, used by every query
@@ -0,0 +1,54 @@
{
"schema_version": 1,
"experiment": "tier-zero-pruning-scoped-v1",
"date": "2026-09-22",
"task": "contributor-scan 7.3",
"claim_scope": "One change to when tier-zero pruning applies, measured on Paid Memberships Pro at two population sizes.",
"what_was_wrong": "Tier-zero pruning already existed: a question whose evidence vector says no sink of its family is in reach cannot produce a flow and is deferred. It was switched off for the WHOLE SCAN whenever any of twelve degradation reasons appeared, and a multi-language repository always carries two of them, so the plugin arbitrated every question it could ask. Third instance this week of one blast radius.",
"the_change": "A prune blocker that names a partition withholds pruning from that partition's questions and from nobody else's. `cross_partition_semantics_unresolved` leaves the blocker set entirely, for the reason it left the integrity gaps: a PHP family's sink is PHP, so a second language cannot be why its vector saw none.",
"measured": {
"25 regions, before": {
"questions_arbitrated": 127,
"questions_deferred": 0,
"questions_completed": 120,
"findings": 19,
"query_seconds": 248.97,
"timed_out": false
},
"25 regions, after": {
"questions_arbitrated": 34,
"questions_deferred": 93,
"questions_completed": 27,
"findings": 19,
"query_seconds": 257.34,
"timed_out": false
},
"500 regions, before": {
"questions_arbitrated": 2526,
"questions_deferred": 0,
"questions_completed": 0,
"findings": null,
"query_seconds": 363.41,
"timed_out": true
},
"500 regions, after": {
"questions_arbitrated": 406,
"questions_deferred": 2120,
"questions_completed": 0,
"findings": 0,
"query_seconds": 375.72,
"timed_out": true
}
},
"reading": "Pruning works and is not sufficient. At 25 regions arbitration falls from 127 questions to 34 with all 19 findings identical, family by family. At 500 it falls from 2,526 to 406, and the run fails in the same way at the same cost, because 406 questions at roughly one and a half seconds each still exceed the 300-second per-batch ceiling. The lever moved the volume; the ceiling is a separate limit.",
"no_finding_lost": {
"at_25_regions": "19 before and 19 after, identical per family: injection 1, output_encoding 18, the rest zero",
"at_500_regions": "neither run completed a question, so neither establishes anything to compare"
},
"skips_are_recorded": "2,120 questions deferred with reason tier_zero at 500 regions, 93 at 25. A family whose every question was pruned now reports 'no sink of this family in reach' rather than 'region offered, no question asked', because nothing to ask is a different statement from never offered.",
"limits": [
"One subject. The saving depends entirely on how much of a repository holds no sink of a given family, which is a property of the code and the ontology together.",
"Pruning is only defined for taint families; dominance and configuration questions are unaffected.",
"The 500-region run still fails, so this task removes a cause rather than the symptom."
]
}
+11
View File
@@ -97,8 +97,16 @@ def main() -> int:
# WHY an outcome is not completed. Counting only the statuses left three separate causes looking
# identical, and each one had to be found by reading code rather than by reading the record.
"outcome_reason": dict(collections.Counter(f"{q.status}:{q.reason}" for q in result.question_outcomes)),
# A question the scan chose not to arbitrate is recorded as skipped, with why. Without this a cheaper
# scan and a narrower one look identical, and the whole point of skipping is that it is not the same.
"deferred": dict(collections.Counter(item.reason for item in (result.scope.deferred if result.scope else ()))),
}
# A family whose every question was pruned asked nothing, and "nothing to ask" is a different statement
# from "never offered". Counting the skips per family is what keeps those apart in the verdict below.
deferred_by_family: dict[str, collections.Counter[str]] = collections.defaultdict(collections.Counter)
for item in result.scope.deferred if result.scope else ():
deferred_by_family[item.identity.family][item.reason] += 1
asked = collections.Counter(q.identity.family for q in result.question_outcomes)
completed = collections.Counter(q.identity.family for q in result.question_outcomes if q.status == "completed")
found: dict[str, list[dict[str, Any]]] = collections.defaultdict(list)
@@ -133,9 +141,12 @@ def main() -> int:
# job is to say what each family finds reported nothing for a repository that had found a real
# SQL injection. A question-outcome count is a statement about bookkeeping; a finding is the
# thing being censused, and it cannot be hidden behind the other.
"deferred": dict(deferred_by_family.get(family, {})),
"verdict": (
"establishes findings"
if found.get(family)
else "no sink of this family in reach"
if not asked.get(family) and deferred_by_family.get(family, {}).get("tier_zero")
else "region offered, no question asked"
if not asked.get(family)
else "asked but no question completed"
+37 -22
View File
@@ -357,28 +357,38 @@ def _scan_repository_impl(
# Authoritative selection is made once, from the actual final ranker order.
# Stable question IDs are the IDs sent to every execution request below.
scope_limits = (
replace(limits, prune_tier0=False)
if any(
d.get("reason")
in {
"files_unparsed",
"cpg_empty",
"partition_file_census_incomplete",
"partition_file_census_unavailable",
"cpg_sharded",
"cross_partition_semantics_unresolved",
"vendor_semantics_unresolved",
"symlink_context_unresolved",
"frontend_unsupported",
"source_unreadable",
"ambiguous_frontend_path",
"typescript_property_support_unvalidated",
}
for d in degradations
)
else limits
)
#
# A tier-zero vector says no sink of this family is in reach, and a question with no sink cannot produce a
# flow. Pruning those is what makes a large population affordable: arbitration costs one to two seconds a
# question and a 673-file plugin asks 2,526 of them, while the structural pass that produced the vectors
# answered 17,700 requests in 44 s.
#
# It is switched off where the graph might be missing the very sink the vector did not see -- an unparsed
# file, an excluded tree, a shard that cannot see its neighbour. That is right, and it was applied to the
# whole scan, which is not: a census gap belongs to ONE partition, and a JavaScript file missing from the
# JavaScript graph is no reason to arbitrate every tier-zero question a PHP family asks. On the measured
# plugin that single distinction is the difference between 2,526 arbitrated questions and the few hundred
# that could say anything.
#
# `cross_partition_semantics_unresolved` leaves this set entirely, for the reason it left the integrity
# gaps: a PHP family's sink is PHP, so the existence of a second language cannot be why its vector saw none.
prune_blockers = {
"files_unparsed",
"cpg_empty",
"partition_file_census_incomplete",
"partition_file_census_unavailable",
"cpg_sharded",
"vendor_semantics_unresolved",
"symlink_context_unresolved",
"frontend_unsupported",
"source_unreadable",
"ambiguous_frontend_path",
"typescript_property_support_unvalidated",
}
blocking = [d for d in degradations if d.get("reason") in prune_blockers]
# A blocker that names no partition is about the whole read and stops pruning everywhere.
scope_limits = replace(limits, prune_tier0=False) if any(not d.get("census_language") for d in blocking) else limits
unprunable_languages = frozenset(str(d["census_language"]) for d in blocking if d.get("census_language"))
scope, work = _scope_work(
ordered_regions,
evidence_by_pair,
@@ -389,6 +399,7 @@ def _scan_repository_impl(
change_context=change_context,
context_gaps=context_gaps,
execution_budget=execution_budget,
unprunable_languages=unprunable_languages,
)
outcomes: dict[str, QuestionOutcome] = {}
@@ -649,6 +660,9 @@ def _scope_work(
change_context: ChangeContext | None = None,
context_gaps: set[QuestionIdentity] | None = None,
execution_budget: ExecutionBudget | None = None,
# Languages whose graph might be missing the sink their vectors did not see. Pruning is withheld from
# their questions and from nobody else's.
unprunable_languages: frozenset[str] = frozenset(),
) -> tuple[ScopeDecision, list[tuple[str, ScanRegion, ArbiterSpec]]]:
"""Consume the ranker's final order once; the returned work IS selected scope."""
selected: list[RankedQuestion] = []
@@ -707,6 +721,7 @@ def _scope_work(
if index >= limits.max_regions
else "tier_zero"
if limits.prune_tier0
and identity.language not in unprunable_languages
and identity not in (context_gaps or set())
and isinstance(spec, TaintSpec)
and vector is not None