perf(push): find the Python-side scale walls all at once, and remove them

The loop the maintainer called out: four scale defects in the PHP transfer
replay, found one per two-hour Joern run, all Python bookkeeping between
engine calls. benchmarks/push/python_overhead.py ends that: the real
replay() on the real PMPro pins, with a JoernBackend whose graphs are stubs
and whose queries answer instantly with rows recorded from the real engine.
Engine time is zero, so everything measured is ours, in about a minute,
under the sampling profiler. Its first pass named two more walls:

- Region discovery re-parsed the ruleset for every region: 89,127 TOML
  parses, 275 s of a 339 s discovery, twice per push. The fact, obligation
  and taxonomy loaders now keep the parsed (immutable) result keyed by each
  file's name, size and mtime -- an edit in place or an added file reloads.
  Discovery 339 s -> 7 s; output digest identical on PMPro.
- affected_context decoded every span's and every line anchor's path for
  every context row -- a changed file has an anchor per unchanged line --
  78% of a stubbed push. Decoded once per call, anchors indexed by line and
  bisected, original order kept. A randomised control compares it against
  the row-by-row matching over 200 contexts (>100 attachments).

Result: the whole stubbed PMPro push, 27,542 answered requests and 2,688
outcomes, runs in 58.5 s of Python. What is left is the engine.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-09-23 19:36:32 +02:00
co-authored by Claude Opus 5.5
parent c629fea89a
commit 6b85a60ff1
7 changed files with 338 additions and 15 deletions
+19 -1
View File
@@ -90,7 +90,7 @@ class FamilyTaxonomy:
return "lateral"
def load_families(path: Path | None = None, *, require_all: bool = True) -> FamilyTaxonomy:
def _load_families_uncached(path: Path | None = None, *, require_all: bool = True) -> FamilyTaxonomy:
"""Load the taxonomy. ``require_all`` demands the full closed set in order (the shipped file);
a scoped fixture may declare a subset, but never a family outside ``FAMILY_IDS``."""
source = path if path is not None else DEFAULT_FAMILIES_PATH
@@ -192,3 +192,21 @@ __all__ = [
"Relation",
"load_families",
]
# Parsed once per file state; see `semantic.facts.load_facts` for why and for the key.
_FAMILIES_MEMO: dict[tuple[object, ...], FamilyTaxonomy] = {}
def load_families(path: Path | None = None, *, require_all: bool = True) -> FamilyTaxonomy:
"""Load the taxonomy. ``require_all`` demands the full closed set in order (the shipped file);
a scoped fixture may declare a subset, but never a family outside ``FAMILY_IDS``."""
source = path if path is not None else DEFAULT_FAMILIES_PATH
try:
info = source.stat()
except OSError:
return _load_families_uncached(path, require_all=require_all)
key = (str(source.resolve()), info.st_size, info.st_mtime_ns, require_all)
if key not in _FAMILIES_MEMO:
_FAMILIES_MEMO[key] = _load_families_uncached(path, require_all=require_all)
return _FAMILIES_MEMO[key]