mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 11:49:28 +00:00
Marius CiepluchandClaude Opus 4.8
a3a38f1b9f
The real bottleneck was not enumeration or extraction: canonicalize_used resolves the entire exclusion set (measured: 4,499 names) via serial GitHub get_repo on every launch, before any harvesting -- ~30 min of network-latency-bound work that also burns ~4,499 of GitHub's 5,000/hr budget before real work starts (hence the low CPU and the "never finishes"). The earlier resolution-cache only banked via BankedGitHub.resolve, but canonicalize_used calls eligibility.resolve -> get_repo directly, so it was never cached and re-ran every time. Cache get_repo at the real choke point. CorpusBank gains cache_repo / get_cached_repo storing one object per normalized name at <prefix>/repos/<sha256(name)>.json -- NOT seed-scoped, since repo metadata is seed-independent, so every seed/run shares it. 404s are cached as None too (a found-but-None hit) so known-missing repos are not re-fetched. BankedGitHub.get_repo now checks the local journal, then the bucket, and only then GitHub (banking the result). canonicalize_used, advisory_index and candidate resolution all funnel through get_repo, so all of them are covered. Effect: the first run pays the lookups once and banks them; every later launch reads them from the bucket (no GitHub, no rate-limit spend), so the rate budget goes to actual candidate harvesting and restarts are cheap. Proven live against ousast-corpus (repos/ fills 1:1 with resolutions). 206 unseen tests + new repo-cache test pass, ruff clean. Co-Authored-By: Claude Opus 4.8 <[email protected]>