mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
Marius CiepluchandClaude Opus 4.8
79f04400f6
Extraction is CPU-bound Python (git-output parsing, HUNK regex, ast.parse of hundreds of source files per repo). A ThreadPoolExecutor cannot parallelize it: live profiling showed one thread pegging a core at 99.7% while three cores sat idle and the others waited on the GIL, so six "workers" barely beat serial (~0.25 entries/min). Switch ordered_extractions to a ProcessPoolExecutor with a fork context (copy-on-write memory; lets the existing in-memory test fakes cross into children without pickling). Effective workers are clamped to the core count. The parent still owns the journal, REST (finish_license), decide, counters, the fork-network dedup, strict seed-order acceptance, the systemic guard, and the graceful drain. Children run extract_repository with github=None on picklable args and return their own count deltas (threads had shared a Counter; processes return counts in the result). A broken pool is replaced; a collateral worker_crash is a skip outside the guard, a task-specific extract_worker_failure counts toward it. Determinism is unchanged and tested with REAL processes: workers=6 == workers=1 byte-identical (fork collisions, out-of-order completion, resume). 208 unseen tests pass, ruff clean. Co-Authored-By: Claude Opus 4.8 <[email protected]>