marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 11:49:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

perf(hunter): read a repository's source once per state, not once per tool call

Browse Source 
Each grep_repo and find_refs call re-enumerated the tree (walk, ignore rules,
vendored checks) and re-read every file. The v2 sink verifier spent almost three
hours on its first case (pgAdmin, about 1,800 files) at 108% CPU with one open
connection. It was CPU-bound in these tools, not waiting on the model.

Source lines are now memoised per root, keyed by path and modification time. A hit
is re-validated file by file, because an edit inside a subdirectory does not touch
the root's mtime; a test covers one enumeration and a detected edit.

verify_sinks.py now runs candidates in worker processes, not threads, and appends
each candidate's result to a per-case log, so a stopped run resumes where it left
off. GoPay: 12 candidates in 14 s, the real site found.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-09-28 18:30:09 +02:00
co-authored by Claude Opus 5.5
parent 023c24535f
commit 267bca5232
3 changed files with 117 additions and 38 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/test_hunter_tools.py
+22
View File
@@ -121,3 +121,25 @@ def test_find_refs_rejects_mapping_path_escape(tmp_path: Path) -> None:
find_refs(root, "passwd", [{"path": "../etc/passwd", "name": "passwd"}])
with pytest.raises(PathEscapesRepo):
find_refs(root, "passwd", [{"path": "/etc/passwd", "name": "passwd"}])
def test_grep_reads_the_tree_once_and_sees_an_edit(tmp_path, monkeypatch) -> None:
"""Every call re-enumerated and re-read the repository; now once per root state, re-validated per file."""
import os
from openultrasast import hunter_tools
sub = tmp_path / "pkg"
sub.mkdir()
source = sub / "app.py"
source.write_text("def run(q):\n return query(q)\n")
calls = []
real = hunter_tools._iter_clamped_source_files
monkeypatch.setattr(hunter_tools, "_iter_clamped_source_files", lambda root: calls.append(root) or real(root))
assert [m["line"] for m in hunter_tools.grep_repo(tmp_path, r"query\(", max_matches=5)] == [2]
assert [m["line"] for m in hunter_tools.grep_repo(tmp_path, r"def run", max_matches=5)] == [1]
assert len(calls) == 1, "the tree was enumerated again for an unchanged root"
source.write_text("def run(q):\n x = 1\n return query(q)\n")
stamp = source.stat().st_mtime_ns + 10_000_000
os.utime(source, ns=(stamp, stamp))
assert [m["line"] for m in hunter_tools.grep_repo(tmp_path, r"query\(", max_matches=5)] == [3], "a stale read survived an edit"