mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-18 12:39:28 +00:00
fix(cpg): reclaim the scratch tree, and sweep what earlier runs abandoned
Every build allocated a temp directory for the graph, its request files and
joern's own working copy of the CPG. Nothing ever reclaimed it. A day of
scanning left 1,055 directories and 355MB in /tmp on a disk already at 95%,
and the runs that leaked hardest were the ones that FAILED -- which is exactly
backwards, since a failed scan has the least to show for the space it took.
Two mechanisms, because they cover different deaths:
* `CpgResult.cleanup` disposes of the tree, and `scan_repository` calls it
when the graph has answered everything it is going to. Never raises: a scan
that found things must not fail while tidying up.
* A sweep at build time removes trees older than six hours. A process killed
mid-scan never runs its cleanup -- which is how most of the 1,055 died --
so the disposal alone is not enough. Only directories under the system temp
root carrying this module's own prefix are touched, and a scan running in
another process right now is left alone.
Note for the record what was NOT a product leak: 6.3GB of /tmp/joern-cli-*
was three copies of the Joern CLI I made by hand during today's container
experiments, and 63MB of workspace/ in the repo came from running joern from
the repo root instead of a scratch directory. The product already runs joern
with cwd set to the scratch tree, so its workspace is contained and now
disposed with it. Both are cleaned up; neither was the code's fault, and
conflating my litter with its leak would have hidden the real one.
Disk went from 6.1GB free to 19GB.
857 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
co-authored by
Claude Opus 5
parent
37eca758d4
commit
f496f41be1
@@ -277,6 +277,9 @@ def scan_repository(
|
||||
scanned = len(judged)
|
||||
|
||||
findings = _ordered(_deduplicated(collected))
|
||||
# The graph has answered everything it is going to. Its scratch tree holds the CPG, the request files and
|
||||
# joern's own working copy -- tens of megabytes per scan -- and nothing else reclaims it.
|
||||
_dispose(cpg)
|
||||
return ModelScanResult(
|
||||
findings=findings,
|
||||
by_rung=_tally(findings),
|
||||
@@ -293,6 +296,16 @@ def scan_repository(
|
||||
)
|
||||
|
||||
|
||||
def _dispose(cpg: object) -> None:
|
||||
"""Release a built CPG's scratch tree. Never raises: cleanup must not turn a good scan into a failure."""
|
||||
cleanup = getattr(cpg, "cleanup", None)
|
||||
if callable(cleanup):
|
||||
try:
|
||||
cleanup()
|
||||
except Exception as exc: # noqa: BLE001 -- a scan that found things must not fail while tidying up
|
||||
logger.warning("could not remove the cpg scratch directory: %s", exc)
|
||||
|
||||
|
||||
def _build(backend: Any, root: Path, language: str) -> Any:
|
||||
"""Build through the backend, passing the language when the backend can use it."""
|
||||
try:
|
||||
|
||||
Reference in New Issue
Block a user