mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
unseen: bucket-cache get_repo (kill the repeated used-set canonicalization tax)
The real bottleneck was not enumeration or extraction: canonicalize_used resolves the entire exclusion set (measured: 4,499 names) via serial GitHub get_repo on every launch, before any harvesting -- ~30 min of network-latency-bound work that also burns ~4,499 of GitHub's 5,000/hr budget before real work starts (hence the low CPU and the "never finishes"). The earlier resolution-cache only banked via BankedGitHub.resolve, but canonicalize_used calls eligibility.resolve -> get_repo directly, so it was never cached and re-ran every time. Cache get_repo at the real choke point. CorpusBank gains cache_repo / get_cached_repo storing one object per normalized name at <prefix>/repos/<sha256(name)>.json -- NOT seed-scoped, since repo metadata is seed-independent, so every seed/run shares it. 404s are cached as None too (a found-but-None hit) so known-missing repos are not re-fetched. BankedGitHub.get_repo now checks the local journal, then the bucket, and only then GitHub (banking the result). canonicalize_used, advisory_index and candidate resolution all funnel through get_repo, so all of them are covered. Effect: the first run pays the lookups once and banks them; every later launch reads them from the bucket (no GitHub, no rate-limit spend), so the rate budget goes to actual candidate harvesting and restarts are cheap. Proven live against ousast-corpus (repos/ fills 1:1 with resolutions). 206 unseen tests + new repo-cache test pass, ruff clean. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
21aebe8538
commit
a3a38f1b9f
@@ -218,6 +218,26 @@ class BankedGitHub(ResolvedGitHub):
|
||||
self.resolutions = bank.repositories(contract=RESOLUTION_CONTRACT)
|
||||
journal.rows["resolution"].update({name: body["metadata"] if body is not None else None for name, body in self.resolutions.items()})
|
||||
|
||||
def get_repo(self, name):
|
||||
# Bucket-cache get_repo (the universal resolution choke point) so the
|
||||
# ~thousands of used-set canonicalization lookups and candidate resolves
|
||||
# are not re-fetched from GitHub every run. 404s are cached as None too.
|
||||
key = eligibility.normalize(name)
|
||||
if key not in self.journal.rows["resolution"]:
|
||||
found, value = self.bank.get_cached_repo(key, contract=RESOLUTION_CONTRACT)
|
||||
if not found:
|
||||
self.journal.boundary()
|
||||
try:
|
||||
value = self.github.get_repo(key)
|
||||
except eligibility.RepositoryNotFound:
|
||||
value = None
|
||||
self.bank.cache_repo(key, value, contract=RESOLUTION_CONTRACT)
|
||||
self.journal.put("resolution", key, value)
|
||||
value = self.journal.rows["resolution"][key]
|
||||
if value is None:
|
||||
raise eligibility.RepositoryNotFound("not_found")
|
||||
return value
|
||||
|
||||
def resolve(self, name):
|
||||
key = eligibility.normalize(name)
|
||||
if key not in self.resolutions:
|
||||
|
||||
Reference in New Issue
Block a user