mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
unseen: parallelize extraction + batch per-repo blob fetch
Serial extraction at ~147s/repo made a 300-repo pool take days, and ~626 lazy per-file blob faults per clone pushed slow repos past the 240s deadline (a draw reached 21 entries from 347 attempts, 160 of them extraction_timeout). Both fixed. Parallel: a bounded ThreadPoolExecutor (--extract-workers, default 6) runs clones concurrently, each into its own clone with a private counter. The main thread alone owns the journal, REST (license via finish_license), counters, and acceptance. Entries are accepted in STRICT SEED ORDER with the fork-network dedup applied at acceptance, so the pool is byte- identical to the serial draw (tested: workers=6 == workers=1, including fork collisions, out-of-order completion, and resume). The systemic guard and resilient skips are unchanged; a graceful stop drains started work to the journal for resume. Batched blobs: repository_changes fetches the fixes together and Git.materialize fetches all missing blobs for the read revisions in one `fetch --no-filter --stdin` (rev-list --missing=print), instead of per- file promisor faults. Content is byte-identical (tested; >=12 fetches collapsed to 1). 202 unseen tests pass, ruff clean. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
4f7eedddf1
commit
65714405eb
@@ -53,12 +53,12 @@ class Git:
|
||||
raise ExtractionTimeout("extraction_timeout")
|
||||
return min(180, remaining)
|
||||
|
||||
def run(self, *args: str, allowed=(0,)) -> str:
|
||||
def run(self, *args: str, allowed=(0,), input_data: bytes | None = None) -> str:
|
||||
timeout = self.command_timeout()
|
||||
try:
|
||||
done = self.runner(
|
||||
["git", "-c", "core.quotePath=false", "-C", str(self.path), *args],
|
||||
stdin=subprocess.DEVNULL,
|
||||
**({"stdin": subprocess.DEVNULL} if input_data is None else {"input": input_data}),
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
timeout=timeout,
|
||||
@@ -75,6 +75,38 @@ class Git:
|
||||
self.last_blob_bytes = len(done.stdout)
|
||||
return done.stdout.decode("utf-8", errors="replace")
|
||||
|
||||
def materialize(self, revisions: list[str]) -> None:
|
||||
"""Fetch missing historical blobs in one pack, without changing content reads.
|
||||
|
||||
ordinary_history inspects the complete first-parent history, including both
|
||||
sides, attributes and non-source diffs. Blame -M/-C and log -L can walk all
|
||||
ancestors and copy sources. Include the reachable trees, not only tip files,
|
||||
so these commands keep exactly their original semantics. This is a safe
|
||||
superset (merge-side trees included), still subject to the disk/deadline caps.
|
||||
Missing-object enumeration itself does not trigger lazy blob fetches.
|
||||
"""
|
||||
data = ("\n".join(revisions) + "\n").encode()
|
||||
objects = self.run("rev-list", "--objects", "--missing=print", "--stdin", input_data=data)
|
||||
missing = sorted({line[1:] for line in objects.splitlines() if line.startswith("?")})
|
||||
if not all(SHA.fullmatch(oid) for oid in missing):
|
||||
raise InstrumentFailure("invalid_missing_object")
|
||||
if missing:
|
||||
# stdin avoids argv limits; explicit object wants avoid one promisor
|
||||
# request per show/blame. --no-filter overrides clone's stored filter.
|
||||
self.run(
|
||||
"fetch",
|
||||
"--quiet",
|
||||
"--no-tags",
|
||||
"--no-write-fetch-head",
|
||||
"--no-filter",
|
||||
"--stdin",
|
||||
"origin",
|
||||
input_data=("\n".join(missing) + "\n").encode(),
|
||||
)
|
||||
remaining = self.run("rev-list", "--objects", "--missing=print", "--stdin", input_data=data)
|
||||
if any(line.startswith("?") for line in remaining.splitlines()):
|
||||
raise InstrumentFailure("missing_blobs_after_fetch")
|
||||
|
||||
def disk_bytes(self) -> int:
|
||||
# git churns .git/objects/pack (incl. transient *.rev) during on-demand blob
|
||||
# fetches, so a path rglob just listed can vanish before stat(): skip it.
|
||||
|
||||
Reference in New Issue
Block a user