unseen: bucket-cache get_repo (kill the repeated used-set canonicalization tax)

The real bottleneck was not enumeration or extraction: canonicalize_used
resolves the entire exclusion set (measured: 4,499 names) via serial
GitHub get_repo on every launch, before any harvesting -- ~30 min of
network-latency-bound work that also burns ~4,499 of GitHub's 5,000/hr
budget before real work starts (hence the low CPU and the "never
finishes"). The earlier resolution-cache only banked via
BankedGitHub.resolve, but canonicalize_used calls eligibility.resolve ->
get_repo directly, so it was never cached and re-ran every time.

Cache get_repo at the real choke point. CorpusBank gains cache_repo /
get_cached_repo storing one object per normalized name at
<prefix>/repos/<sha256(name)>.json -- NOT seed-scoped, since repo
metadata is seed-independent, so every seed/run shares it. 404s are
cached as None too (a found-but-None hit) so known-missing repos are not
re-fetched. BankedGitHub.get_repo now checks the local journal, then the
bucket, and only then GitHub (banking the result). canonicalize_used,
advisory_index and candidate resolution all funnel through get_repo, so
all of them are covered.

Effect: the first run pays the lookups once and banks them; every later
launch reads them from the bucket (no GitHub, no rate-limit spend), so
the rate budget goes to actual candidate harvesting and restarts are
cheap. Proven live against ousast-corpus (repos/ fills 1:1 with
resolutions). 206 unseen tests + new repo-cache test pass, ruff clean.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-10-10 17:10:19 +02:00
co-authored by Claude Opus 4.8
parent 21aebe8538
commit a3a38f1b9f
3 changed files with 60 additions and 0 deletions
+20
View File
@@ -218,6 +218,26 @@ class BankedGitHub(ResolvedGitHub):
self.resolutions = bank.repositories(contract=RESOLUTION_CONTRACT)
journal.rows["resolution"].update({name: body["metadata"] if body is not None else None for name, body in self.resolutions.items()})
def get_repo(self, name):
# Bucket-cache get_repo (the universal resolution choke point) so the
# ~thousands of used-set canonicalization lookups and candidate resolves
# are not re-fetched from GitHub every run. 404s are cached as None too.
key = eligibility.normalize(name)
if key not in self.journal.rows["resolution"]:
found, value = self.bank.get_cached_repo(key, contract=RESOLUTION_CONTRACT)
if not found:
self.journal.boundary()
try:
value = self.github.get_repo(key)
except eligibility.RepositoryNotFound:
value = None
self.bank.cache_repo(key, value, contract=RESOLUTION_CONTRACT)
self.journal.put("resolution", key, value)
value = self.journal.rows["resolution"][key]
if value is None:
raise eligibility.RepositoryNotFound("not_found")
return value
def resolve(self, name):
key = eligibility.normalize(name)
if key not in self.resolutions: