Marius Ciepluch and Claude Opus 4.8
a3a38f1b9f
unseen: bucket-cache get_repo (kill the repeated used-set canonicalization tax)
...
The real bottleneck was not enumeration or extraction: canonicalize_used
resolves the entire exclusion set (measured: 4,499 names) via serial
GitHub get_repo on every launch, before any harvesting -- ~30 min of
network-latency-bound work that also burns ~4,499 of GitHub's 5,000/hr
budget before real work starts (hence the low CPU and the "never
finishes"). The earlier resolution-cache only banked via
BankedGitHub.resolve, but canonicalize_used calls eligibility.resolve ->
get_repo directly, so it was never cached and re-ran every time.
Cache get_repo at the real choke point. CorpusBank gains cache_repo /
get_cached_repo storing one object per normalized name at
<prefix>/repos/<sha256(name)>.json -- NOT seed-scoped, since repo
metadata is seed-independent, so every seed/run shares it. 404s are
cached as None too (a found-but-None hit) so known-missing repos are not
re-fetched. BankedGitHub.get_repo now checks the local journal, then the
bucket, and only then GitHub (banking the result). canonicalize_used,
advisory_index and candidate resolution all funnel through get_repo, so
all of them are covered.
Effect: the first run pays the lookups once and banks them; every later
launch reads them from the bucket (no GitHub, no rate-limit spend), so
the rate budget goes to actual candidate harvesting and restarts are
cheap. Proven live against ousast-corpus (repos/ fills 1:1 with
resolutions). 206 unseen tests + new repo-cache test pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected] >
2026-10-10 17:10:19 +02:00
..
2026-10-05 14:45:48 +02:00
2026-06-22 17:13:02 +02:00
2026-10-02 20:26:46 +02:00
2026-10-01 10:13:31 +02:00
2026-10-02 21:05:48 +02:00
2026-09-29 19:41:35 +02:00
2026-10-04 11:57:34 +02:00
2026-10-02 19:48:01 +02:00
2026-10-06 11:15:09 +02:00
2026-10-06 14:51:07 +02:00
2026-10-04 12:03:19 +02:00
2026-09-30 15:05:47 +02:00
2026-06-18 10:06:21 +02:00
2026-09-08 12:15:54 +02:00
2026-09-05 10:45:43 +02:00
2026-09-03 20:33:48 +02:00
2026-09-04 00:18:33 +02:00
2026-09-03 20:33:48 +02:00
2026-09-03 20:05:13 +02:00
2026-10-02 19:45:37 +02:00
2026-09-23 08:51:12 +02:00
2026-09-26 07:44:33 +02:00
2026-10-03 22:53:28 +02:00
2026-10-04 16:41:22 +02:00
2026-09-08 12:15:54 +02:00
2026-06-22 17:13:02 +02:00
2026-09-24 11:08:22 +02:00
2026-10-02 10:15:19 +02:00
2026-10-04 16:18:01 +02:00
2026-10-04 16:41:22 +02:00
2026-10-05 08:14:57 +02:00
2026-09-23 19:36:32 +02:00
2026-06-22 15:27:16 +02:00
2026-09-14 08:07:06 +02:00
2026-09-30 12:13:26 +02:00
2026-10-02 07:22:35 +02:00
2026-09-25 15:29:43 +02:00
2026-09-13 17:08:49 +02:00
2026-06-22 20:02:34 +02:00
2026-06-18 10:06:21 +02:00
2026-09-28 20:10:57 +02:00
2026-09-28 20:15:14 +02:00
2026-06-18 10:59:25 +02:00
2026-09-30 13:44:32 +02:00
2026-09-05 10:45:43 +02:00
2026-09-30 18:15:52 +02:00
2026-10-04 19:17:35 +02:00
2026-10-02 19:32:46 +02:00
2026-10-04 14:39:36 +02:00
2026-09-30 18:49:05 +02:00
2026-10-03 14:41:55 +02:00
2026-10-01 10:36:34 +02:00
2026-10-02 10:15:19 +02:00
2026-10-01 10:36:34 +02:00
2026-10-02 10:15:19 +02:00
2026-10-02 17:24:49 +02:00
2026-10-02 16:12:08 +02:00
2026-10-03 15:06:08 +02:00
2026-10-03 15:06:08 +02:00
2026-10-01 10:05:19 +02:00
2026-10-01 11:03:54 +02:00
2026-10-02 15:25:21 +02:00
2026-10-01 20:40:41 +02:00
2026-10-02 11:33:16 +02:00
2026-10-01 14:04:28 +02:00
2026-10-03 09:40:58 +02:00
2026-09-30 18:38:04 +02:00
2026-10-04 14:39:36 +02:00
2026-09-04 11:53:06 +02:00
2026-09-09 17:33:36 +02:00
2026-09-09 13:15:54 +02:00
2026-09-25 14:30:36 +02:00
2026-09-20 16:30:04 +02:00
2026-09-09 13:09:47 +02:00
2026-09-28 09:10:02 +02:00
2026-09-11 13:45:00 +02:00
2026-09-08 19:46:58 +02:00
2026-09-14 08:07:06 +02:00
2026-09-09 09:27:19 +02:00
2026-09-17 14:04:39 +02:00
2026-09-08 19:46:58 +02:00
2026-09-09 07:42:44 +02:00
2026-09-14 08:07:06 +02:00
2026-09-25 15:13:30 +02:00
2026-09-09 12:45:21 +02:00
2026-09-09 16:13:17 +02:00
2026-09-09 14:30:48 +02:00
2026-09-09 15:08:01 +02:00
2026-10-02 07:22:35 +02:00
2026-09-04 11:50:44 +02:00
2026-09-08 12:15:54 +02:00
2026-09-05 19:38:04 +02:00
2026-09-06 07:24:13 +02:00
2026-09-06 07:24:13 +02:00
2026-09-06 13:09:56 +02:00
2026-09-06 07:24:13 +02:00
2026-09-06 07:24:13 +02:00
2026-09-06 07:24:13 +02:00
2026-09-05 19:38:04 +02:00
2026-09-06 07:24:13 +02:00
2026-09-08 12:15:54 +02:00
2026-09-06 07:24:13 +02:00
2026-09-08 19:46:58 +02:00
2026-09-08 19:46:58 +02:00
2026-09-30 18:49:05 +02:00
2026-06-21 20:44:08 +02:00
2026-09-29 19:04:52 +02:00
2026-09-29 17:41:09 +02:00
2026-10-02 21:49:58 +02:00
2026-10-02 21:29:32 +02:00
2026-09-30 15:05:47 +02:00
2026-09-30 18:38:04 +02:00
2026-10-01 19:40:58 +02:00
2026-10-02 20:43:40 +02:00
2026-10-02 20:32:51 +02:00
2026-09-30 18:38:04 +02:00
2026-10-01 09:56:52 +02:00
2026-10-02 10:15:19 +02:00
2026-10-04 12:03:19 +02:00
2026-10-02 21:49:58 +02:00
2026-09-30 13:44:32 +02:00
2026-09-29 17:55:34 +02:00
2026-10-02 20:26:46 +02:00
2026-10-02 21:12:19 +02:00
2026-10-02 08:07:35 +02:00
2026-09-30 08:19:46 +02:00
2026-09-29 19:49:28 +02:00
2026-06-22 12:31:35 +02:00
2026-06-17 21:13:51 +02:00
2026-09-23 19:05:08 +02:00
2026-09-05 10:45:43 +02:00
2026-09-07 07:37:41 +02:00
2026-09-17 09:56:59 +02:00
2026-09-13 18:03:31 +02:00
2026-10-02 22:31:14 +02:00
2026-09-13 16:16:19 +02:00
2026-09-23 14:12:47 +02:00
2026-09-12 21:12:45 +02:00
2026-09-13 19:32:43 +02:00
2026-09-13 15:24:06 +02:00
2026-10-02 22:51:22 +02:00
2026-09-13 19:06:02 +02:00
2026-10-02 21:57:42 +02:00
2026-09-12 18:45:54 +02:00
2026-10-02 22:51:22 +02:00
2026-09-13 17:45:16 +02:00
2026-10-02 22:24:24 +02:00
2026-09-21 13:50:38 +02:00
2026-10-02 22:18:14 +02:00
2026-09-13 18:43:13 +02:00
2026-09-13 13:25:44 +02:00
2026-09-23 07:55:33 +02:00
2026-09-13 17:08:49 +02:00
2026-09-12 19:04:20 +02:00
2026-09-12 21:12:45 +02:00
2026-10-02 21:57:42 +02:00
2026-06-18 07:57:06 +02:00
2026-06-22 20:02:34 +02:00
2026-09-03 22:36:02 +02:00
2026-09-03 22:48:10 +02:00
2026-09-03 22:12:45 +02:00
2026-09-03 22:59:01 +02:00
2026-09-30 12:45:29 +02:00
2026-10-02 22:10:33 +02:00
2026-09-09 08:10:59 +02:00
2026-09-30 18:49:05 +02:00
2026-06-17 21:13:51 +02:00
2026-09-03 23:43:15 +02:00
2026-09-03 23:43:15 +02:00
2026-09-30 12:13:26 +02:00
2026-09-30 12:13:26 +02:00
2026-06-22 15:27:16 +02:00
2026-06-22 15:41:17 +02:00
2026-10-05 21:07:58 +02:00
2026-10-05 14:26:43 +02:00
2026-10-05 14:26:43 +02:00
2026-10-06 17:07:24 +02:00
2026-10-07 12:03:45 +02:00
2026-10-06 14:50:14 +02:00
2026-10-06 14:37:58 +02:00
2026-10-06 11:15:09 +02:00
2026-10-09 14:03:30 +02:00
2026-10-05 20:40:13 +02:00
2026-10-05 17:12:46 +02:00
2026-10-07 12:03:45 +02:00
2026-10-05 17:26:06 +02:00
2026-10-05 21:37:56 +02:00
2026-10-06 14:37:58 +02:00
2026-10-06 11:15:09 +02:00
2026-10-06 12:43:11 +02:00
2026-10-06 09:37:47 +02:00
2026-09-04 15:07:35 +02:00
2026-09-30 12:18:42 +02:00
2026-09-04 15:07:35 +02:00
2026-09-08 19:46:58 +02:00
2026-09-30 18:49:05 +02:00
2026-06-22 19:51:04 +02:00
2026-09-04 11:49:44 +02:00
2026-09-04 11:54:01 +02:00
2026-09-03 23:14:00 +02:00
2026-09-08 19:46:58 +02:00
2026-10-05 20:40:13 +02:00
2026-09-30 12:13:26 +02:00
2026-09-06 12:39:55 +02:00
2026-10-10 17:10:19 +02:00
2026-10-10 07:54:49 +02:00
2026-10-05 08:14:57 +02:00
2026-10-10 16:33:29 +02:00
2026-10-04 19:17:35 +02:00
2026-10-10 14:46:18 +02:00
2026-10-10 07:54:49 +02:00
2026-10-04 20:19:14 +02:00
2026-10-04 19:17:35 +02:00
2026-10-05 08:14:57 +02:00
2026-10-04 20:19:14 +02:00
2026-10-04 20:19:14 +02:00
2026-10-04 20:19:14 +02:00
2026-10-08 10:26:11 +02:00
2026-06-18 10:06:21 +02:00