diff --git a/.kiro/specs/pre-push-safety-net/design.md b/.kiro/specs/pre-push-safety-net/design.md index 7244621..2d1d04e 100644 --- a/.kiro/specs/pre-push-safety-net/design.md +++ b/.kiro/specs/pre-push-safety-net/design.md @@ -265,6 +265,21 @@ results. A query/fact change invalidates its answers without automatically rebui input/frontend/overlay semantics are unchanged. This supports iteration on the core as well as repeated push checks; it does not authorize reusing stale evidence after a source change. +Preparation is also a reusable artifact boundary. Task 5.2 measured PMPro exhausting +30 seconds after materialization and before graph construction. Cache complete discovery +by immutable source tree identity plus discovery code, shipping/layout facts and configuration; +retain snapshot readability proofs and declaration/boundary records. Reuse must preserve +repository-relative locations and must never deserialize repository-controlled executable objects. +Measure preparation, graph and query hits separately. An identical-tip hit demonstrates reuse, +not representative changed-code latency. Dependency/configuration changes invalidate affected +preparation and graph/query context even when an individual source file is unchanged. + +Rollout approval remains an evidence decision under Requirement 6.5 and evaluation.md. +Group 6 establishes correctness and bounded reuse; group 8 must still demonstrate the joint +changed-code latency, completion, precision and recall gates on declared populations. Missing +independent positive/fixed/benign populations or an empty capability registry cannot yield GO. +No threshold, population denominator or alert eligibility rule is relaxed by this amendment. + Only validated, complete artifacts can satisfy complete-result lookups. Partial artifacts may be retained for diagnostics but never promoted by cache presence. Publish atomically from private temporary paths under a per-key lock; another push may wait only within its deadline. Apply a diff --git a/.kiro/specs/pre-push-safety-net/implementation-group-6.md b/.kiro/specs/pre-push-safety-net/implementation-group-6.md new file mode 100644 index 0000000..8aeb0fb --- /dev/null +++ b/.kiro/specs/pre-push-safety-net/implementation-group-6.md @@ -0,0 +1,39 @@ +# Group 6 implementation and verification + +The maintainer authorized 6.1–6.4 together on 2026-09-13. A fresh implementer dispatch +failed with `agent thread limit reached`; the explicit kiro-impl manual fallback applies. +Reviews below are main-context adversarial reviews, not independent reviewer evidence. +The design amendment makes measured preparation/discovery costs explicit. Rollout gates +and the empty default capability registry are unchanged. + +## Task 6.1 brief + +Requirements 4.2, 4.3, 4.4 and 7.3; design: Cache and Engine Boundary. +Backend-owned identities bind readable source paths/bytes, declarations, exclusions, +frontend/engine versions, overlay implementation and options. Complete graphs require a +named source census, methods, calls and the dataflow overlay. Loading checks identity and +streamed graph digest, then issues a disposable copy through the existing CpgResult driver +contract. Old structural manifests without source paths cannot qualify. Sharded or partial +graphs cannot qualify. Query locations normalize to repository-relative paths. + +## Status Report +- STATUS: READY_FOR_REVIEW +- TASK: 6.1 +- RED_PHASE_OUTPUT: `/tmp/group6-1-red.log` and `/tmp/group6-1-flag-off.log`: four failures before implementation/with the temporary switch OFF. ON passed; switch removed and five final tests passed. +- TESTS_RUN: 105 focused tests passed; full suite 1246 passed, 9 skipped (105.31s), followed by the final additional driver integration test and focused rerun. Ruff and mypy passed. +- EVIDENCE: source-mounted nonroot Docker Joern 4.0.625, network disabled, 3 GiB limit: read 45 JavaScript bytes, census 1 file/4 methods/3 calls, sealed graph loaded through a disposable lease, original digest unchanged after query and cleanup. This is a correctness control, not a latency benchmark. +- CONCERNS: persistent publication and runner selection belong to 6.2–6.4; this task provides the validated backend lease contract. Main-context review fallback. + +## Review Verdict +- VERDICT: APPROVED +- TASK: 6.1 +- MECHANICAL_RESULTS: focused tests, full regression suite, ruff, mypy and diff whitespace checks passed; no introduced placeholders or secrets. +- FINDINGS: review repaired unbounded full-graph buffering to streamed hashing/copy; unreadable source walks fail explicitly; single PHP graph census passes through unchanged while sharded graphs cannot seal. Existing driver accepts and releases the leased CpgResult. +- SUMMARY: Backend identity and isolated leases satisfy task 6.1 within the approved boundary. + +## Verification Result +- STATUS: VERIFIED +- CLAIM_TYPE: TASK +- CLAIM: Task 6.1 identity and lease behavior, not persistent runner reuse or rollout readiness. +- EVIDENCE: tests/test_graph_reuse.py, 105 focused passing tests, full suite and real Joern control above; measurement JSON retained in benchmarks/measurements/2026-09-13-graph-lease-control.json. +- GAPS: Warm push feasibility and independent capability eligibility remain unproven. diff --git a/.kiro/specs/pre-push-safety-net/spec.json b/.kiro/specs/pre-push-safety-net/spec.json index 7a4a2c8..cd3c6ea 100644 --- a/.kiro/specs/pre-push-safety-net/spec.json +++ b/.kiro/specs/pre-push-safety-net/spec.json @@ -65,6 +65,11 @@ "at": "2026-09-13T12:40:57.961577+00:00", "source": "approved", "action": "Authorized group 5 after the group 4 completion summary: explicit transaction replay, then the first PMPro/frozen-core Node experiment. Review and fresh evidence remain required." + }, + { + "at": "2026-09-13T13:58:52.336506+00:00", + "source": "continue with group 6 as a whole. do we need to amend the spec to get a go for the rollout", + "action": "Authorized group 6 (6.1\u20136.4). Clarified complete preparation/discovery reuse after measured group 5 bottleneck; rollout gates remain unchanged. Manual Kiro review fallback remains necessary after a fresh agent dispatch hit the thread limit." } ], "task_review": { diff --git a/.kiro/specs/pre-push-safety-net/tasks.md b/.kiro/specs/pre-push-safety-net/tasks.md index 9deaacb..123b78d 100644 --- a/.kiro/specs/pre-push-safety-net/tasks.md +++ b/.kiro/specs/pre-push-safety-net/tasks.md @@ -161,7 +161,7 @@ hook latency is admitted. - _Requirements: 6.2, 6.3, 6.4, 6.5, 8.1, 8.2, 8.3, 8.4_ - [ ] 6. Reuse valid graphs and results without stale conclusions -- [ ] 6.1 Validate graph artifact identity and leases +- [x] 6.1 Validate graph artifact identity and leases - Have the backend own source/declaration/exclusion digests, frontend/engine/overlay options, graph census and complete/partial identity. - Permit validated graph leases in the generic driver while preserving the fresh-build path; isolate potentially mutating loads and retain bounded cleanup. - Done when identical valid inputs can reuse a graph while any changed graph input or failed census rejects the lease; original cached artifacts survive query loading unchanged. @@ -181,6 +181,7 @@ hook latency is admitted. - _Boundary: Push cache, policy integration_ - _Requirements: 3.4, 4.3, 4.4, 6.4, 7.2, 7.3_ - [ ] 6.4 Integrate cache reuse into the shared push deadline + - Reuse complete preparation/discovery by immutable tree and discovery/fact/configuration identity, retaining readable snapshot and declaration/boundary records; measure its costs and hits separately from graph/query reuse. - Reuse identical snapshots/unchanged declared units across refs without merging different comparisons; changed units rebuild and stale graph relationships remain hints only. - Account for lookup, lock wait, lease, build/query and result persistence in the same deadline; preserve completed outcomes on cancellation. - Done when cached and uncached replay produce equal evidence/admission for the same completed questions under compatible semantics, with full equivalence under a sufficiently large controlled budget. Under the hook deadline, verify truthful extra completion/deferred scope from cache reuse rather than requiring identical scope; no stale answer survives a dependency/configuration edit. diff --git a/benchmarks/measurements/2026-09-13-graph-lease-control.json b/benchmarks/measurements/2026-09-13-graph-lease-control.json new file mode 100644 index 0000000..9b7bd13 --- /dev/null +++ b/benchmarks/measurements/2026-09-13-graph-lease-control.json @@ -0,0 +1,59 @@ +{ + "source_bytes": 45, + "artifact": { + "identity": { + "source_digest": "aab4a6878883db96c5eb1d0dd98e160e30d16fa2b215735bdac6ff5a41f9783d", + "declarations_digest": "none", + "exclusions_digest": "none", + "unit": "repository", + "language": "javascript", + "engine_version": "4.0.625", + "frontend": "jssrc2cpg", + "frontend_version": "4.0.625", + "overlay_semantics": [ + "dataflowOss" + ], + "overlay_options": [ + [ + "implementation", + "96d0a93a3edb7e4a999a380b5823cd6d593ae2a49cf2a503f923483eda43989a" + ] + ], + "frontend_options": [ + [ + "include_tests", + "True" + ], + [ + "java_options", + "default" + ] + ] + }, + "graph_digest": "a02bef8f7d4d1f2c420cb9bb4c5cc9df2fe86d86ad4e429021c95ad76a996873", + "source_bytes": 45, + "census": { + "files": 1, + "methods": 4, + "calls": 3 + }, + "completeness": "complete", + "diagnostics": [], + "source_root": "/tmp/tmpb4ff84v8", + "source_paths": [ + "a.js" + ] + }, + "lease_census": { + "files": "1", + "methods": "4", + "calls": "3", + "file_names": [ + "a.js" + ], + "overlays": "base,controlflow,typerel,callgraph,dataflowOss", + "maxHeapMB": "2048" + }, + "original_unchanged": true, + "execution": "source-mounted openultrasast:dev image 7f71174daa9e; nonroot, network none, memory 3g; correctness control, not latency" +} diff --git a/src/openultrasast/cpg/artifact.py b/src/openultrasast/cpg/artifact.py index cb76d69..c998992 100644 --- a/src/openultrasast/cpg/artifact.py +++ b/src/openultrasast/cpg/artifact.py @@ -1,7 +1,7 @@ """Backend-owned graph provenance and live resource boundary (pre-push Req 4.3, 7.3). -Structural validity is necessary but insufficient for reuse. Task 6.1 will validate -actual source bytes, graph digest/census and engine compatibility before issuing leases. +Structural manifests carry provenance; the backend reads source bytes and seals a named +graph census, then verifies graph bytes and compatibility before issuing isolated leases. """ from __future__ import annotations @@ -54,6 +54,8 @@ class GraphArtifact(Contract): census: GraphCensus completeness: Literal["complete", "partial"] diagnostics: tuple[str, ...] + source_root: str = "relative" + source_paths: tuple[str, ...] = () def __post_init__(self) -> None: super().__post_init__() @@ -71,7 +73,7 @@ class GraphLease(Protocol): The backend validates the artifact before issuing the lease and isolates loading when it may mutate graph bytes. The consumer owns releasing it exactly once; the backend's release is idempotent and bounded by the supplied monotonic deadline. - A cached artifact survives release. Implementations belong to task 6.1. + A cached artifact survives release. """ @property @@ -81,3 +83,108 @@ class GraphLease(Protocol): def path(self) -> Path: ... def close(self, *, deadline_monotonic: float) -> None: ... + + +def digest_value(value: object) -> str: + import hashlib + import json + + return hashlib.sha256(json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=True).encode()).hexdigest() + + +def read_bytes(path: Path, deadline: float) -> bytes: + """Read a regular local file without following a substituted final symlink.""" + import os + import stat + import time + + chunks: list[bytes] = [] + fd = os.open(path, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK) + with os.fdopen(fd, "rb") as stream: + if not stat.S_ISREG(os.fstat(stream.fileno()).st_mode): + raise ValueError("artifact_not_regular") + while True: + if time.monotonic() >= deadline: + raise TimeoutError("deadline_exhausted") + chunk = stream.read(1024 * 1024) + if not chunk: + return b"".join(chunks) + chunks.append(chunk) + + +def source_inventory(root: Path, deadline: float) -> tuple[str, int, tuple[str, ...]]: + import hashlib + import os + import time + + rows = [] + size = 0 + if root.is_symlink(): + raise ValueError("symlink_source_root") + + def unreadable(error: OSError) -> None: + raise error + + for directory, dirs, files in os.walk(root, followlinks=False, onerror=unreadable): + if time.monotonic() >= deadline: + raise TimeoutError("deadline_exhausted") + if any((Path(directory) / name).is_symlink() for name in dirs): + raise ValueError("symlink_source") + for name in sorted(files): + path = Path(directory) / name + data = read_bytes(path, deadline) + size += len(data) + rows.append((path.relative_to(root).as_posix(), hashlib.sha256(data).hexdigest())) + if not rows or not size: + raise ValueError("source_unreadable_or_empty") + rows.sort() + return digest_value(rows), size, tuple(path for path, _ in rows) + + +@dataclass +class IsolatedGraphLease: + artifact: GraphArtifact + path: Path + _closed: bool = False + + def close(self, *, deadline_monotonic: float) -> None: + from .backend import _DeadlineExpired, _remove_owned_tree + + if self._closed: + return + try: + _remove_owned_tree(self.path.parent, deadline_monotonic) + except (OSError, _DeadlineExpired): + return + self._closed = True + + +def graph_bytes(path: Path, deadline: float, destination: Path | None = None) -> str: + """Hash and optionally copy a graph in bounded chunks; never buffer the full graph.""" + import hashlib + import os + import stat + import time + from contextlib import ExitStack + + digest = hashlib.sha256() + count = 0 + with ExitStack() as stack: + fd = os.open(path, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK) + source = stack.enter_context(os.fdopen(fd, "rb")) + if not stat.S_ISREG(os.fstat(fd).st_mode): + raise ValueError("graph_not_regular") + output = stack.enter_context(destination.open("xb")) if destination is not None else None + while True: + if time.monotonic() >= deadline: + raise TimeoutError("deadline_exhausted") + chunk = source.read(1024 * 1024) + if not chunk: + break + count += len(chunk) + digest.update(chunk) + if output is not None: + output.write(chunk) + if not count: + raise ValueError("graph_empty") + return digest.hexdigest() diff --git a/src/openultrasast/cpg/backend.py b/src/openultrasast/cpg/backend.py index 1504164..c62e19e 100644 --- a/src/openultrasast/cpg/backend.py +++ b/src/openultrasast/cpg/backend.py @@ -32,9 +32,10 @@ import time from collections.abc import Callable, Mapping, Sequence from dataclasses import dataclass, field, replace from pathlib import Path -from typing import Protocol +from typing import Any, Protocol from ..model.contracts import ExecutionBudget +from .artifact import GraphArtifact, GraphCensus, GraphIdentity logger = logging.getLogger(__name__) @@ -311,6 +312,118 @@ class JoernBackend: self.last_failure = session.last_failure self._diagnostics = list(session._diagnostics) + def graph_identity( + self, + root: Path, + *, + language: str, + declarations_digest: str, + exclusions_digest: str, + unit: str = "repository", + execution_budget: ExecutionBudget | None = None, + ) -> GraphIdentity: + from .artifact import digest_value, read_bytes, source_inventory + + deadline = execution_budget.deadline_monotonic if execution_budget else float("inf") + version = joern_version() + if version is None: + raise ValueError("engine_identity_unknown") + engine = ".".join(map(str, version)) + source, _, _ = source_inventory(root, deadline) + code = digest_value([read_bytes(Path(__file__), deadline).hex(), read_bytes(self.queries_dir / "overlay.sc", deadline).hex()]) + return GraphIdentity( + source, + declarations_digest, + exclusions_digest, + unit, + language, + engine, + _FRONTENDS.get(language, "joern-parse"), + engine, + (DATAFLOW_OVERLAY,), + (("implementation", code),), + (("include_tests", str(self.include_tests)), ("java_options", os.environ.get("JAVA_TOOL_OPTIONS", "default") or "default")), + ) + + def describe_graph( + self, result: CpgResult, root: Path, identity: GraphIdentity, *, execution_budget: ExecutionBudget | None = None + ) -> GraphArtifact | None: + """Seal only a named, complete, single graph census bound to bytes after loading.""" + from .artifact import graph_bytes, source_inventory + + budget = execution_budget or self.execution_budget + deadline = budget.deadline_monotonic if budget else float("inf") + try: + digest, size, paths = source_inventory(root, deadline) + if digest != identity.source_digest or result.unparsed: + return None + census = result.run("census", {}) + if not isinstance(census, Mapping) or DATAFLOW_OVERLAY not in str(census.get("overlays", "")).split(","): + return None + names = census.get("file_names") + if not isinstance(names, list) or not all(isinstance(p, str) for p in names): + return None + normalized = {p.removeprefix(str(root) + "/").removeprefix("./") for p in names} + if not set(paths) <= normalized or int(str(census.get("shards", 1))) != 1: + return None + counts = GraphCensus(*(int(str(census[k])) for k in ("files", "methods", "calls"))) + if counts.files < len(paths) or counts.methods < 1: + return None + return GraphArtifact(identity, graph_bytes(result.cpg_path, deadline), size, counts, "complete", (), str(root), paths) + except (OSError, ValueError, KeyError, TimeoutError): + return None + + def load_graph( + self, + path: Path, + artifact: GraphArtifact, + expected: GraphIdentity, + *, + execution_budget: ExecutionBudget | None = None, + ) -> CpgResult | None: + """Validate sealed bytes, then query only a disposable private copy.""" + from .artifact import IsolatedGraphLease, graph_bytes + + budget = execution_budget or self.execution_budget + deadline = budget.deadline_monotonic if budget else float("inf") + cleanup_deadline = deadline + budget.cancellation_allowance_seconds if budget else float("inf") + scratch = None + try: + if artifact.identity != expected or artifact.completeness != "complete" or not artifact.source_paths: + return None + if artifact.census.files < len(artifact.source_paths) or artifact.census.methods < 1: + return None + scratch = Path(tempfile.mkdtemp(prefix="ousast-lease-")) + working = scratch / "cpg.bin" + if graph_bytes(path, deadline, working) != artifact.graph_digest: + raise ValueError("graph_digest_mismatch") + session = replace(self, execution_budget=budget) + lease = IsolatedGraphLease(artifact, working) + + def normalize(value: object) -> Any: + if isinstance(value, str): + return value.replace(artifact.source_root + "/", "") + if isinstance(value, list): + return [normalize(item) for item in value] + if isinstance(value, dict): + return {key: normalize(item) for key, item in value.items()} + return value + + return CpgResult( + working, + lambda kind, params: normalize(session.query(working, kind, params)), + lambda kind, requests: normalize(session.query_batch(working, kind, requests)), + cleanup=lambda: lease.close(deadline_monotonic=cleanup_deadline), + execution_diagnostics=lambda: tuple(session._diagnostics), + ) + except (OSError, ValueError, TimeoutError): + if scratch is not None: + try: + _remove_owned_tree(scratch, cleanup_deadline) + except (OSError, _DeadlineExpired): + self._note("scratch_cleanup_incomplete") + return None + def _build_impl(self, root: Path, *, language: str = "", exclude: Sequence[str] = ()) -> CpgResult | None: """Build a CPG for ``root``. ``None`` on any failure — the caller degrades, never guesses. @@ -671,6 +784,8 @@ class JoernBackend: def query_across(self, shards: Sequence[Path], query: str, params: Mapping[str, object]) -> object | None: """One question over every shard, answers concatenated. ``None`` only when no shard could answer.""" + if query == "census": + return self.query(shards[0], query, params) if len(shards) == 1 else None merged: list[object] = [] answered = 0 for shard in shards: diff --git a/src/openultrasast/cpg/queries/census.sc b/src/openultrasast/cpg/queries/census.sc index dc190ed..f3eaf8d 100644 --- a/src/openultrasast/cpg/queries/census.sc +++ b/src/openultrasast/cpg/queries/census.sc @@ -23,6 +23,8 @@ ujson.Obj( "files" -> cpg.file.size.toString, "methods" -> cpg.method.size.toString, + "calls" -> cpg.call.size.toString, + "file_names" -> ujson.Arr.from(cpg.file.name.l), "overlays" -> cpg.metaData.overlays.l.mkString(","), "maxHeapMB" -> (Runtime.getRuntime.maxMemory / 1048576).toString ) diff --git a/tests/test_graph_reuse.py b/tests/test_graph_reuse.py new file mode 100644 index 0000000..544e0ef --- /dev/null +++ b/tests/test_graph_reuse.py @@ -0,0 +1,90 @@ +"""Real byte identity and isolated graph lease boundaries, without launching Joern.""" + +import time +from dataclasses import replace + +import pytest + +from openultrasast.cpg.backend import CpgResult, JoernBackend +from openultrasast.model.contracts import ExecutionBudget + + +def prepared(tmp_path, monkeypatch): + import openultrasast.cpg.backend as module + + monkeypatch.setattr(module, "joern_version", lambda: (4, 0, 625)) + root = tmp_path / "source" + root.mkdir() + (root / "a.js").write_text("function f(x) { return x; }") + graph = tmp_path / "cpg.bin" + graph.write_bytes(b"graph original") + backend = JoernBackend() + census = {"files": "1", "methods": "1", "calls": "0", "file_names": [str(root / "a.js")], "overlays": "dataflowOss"} + monkeypatch.setattr(JoernBackend, "query", lambda *args: census) + identity = backend.graph_identity(root, language="javascript", declarations_digest="declarations", exclusions_digest="exclusions") + result = CpgResult(graph, lambda *_: census) + artifact = backend.describe_graph(result, root, identity) + return backend, root, graph, identity, artifact + + +def test_identity_reads_content_and_rejects_changed_inputs(tmp_path, monkeypatch): + backend, root, graph, identity, artifact = prepared(tmp_path, monkeypatch) + assert artifact.source_bytes == (root / "a.js").stat().st_size > 0 + lease = backend.load_graph(graph, artifact, identity) + assert lease is not None + lease.cleanup() + (root / "a.js").write_text("changed") + changed = backend.graph_identity(root, language="javascript", declarations_digest="declarations", exclusions_digest="exclusions") + assert changed != identity + assert backend.load_graph(graph, artifact, changed) is None + for field in ("declarations_digest", "exclusions_digest", "engine_version", "frontend_version"): + assert backend.load_graph(graph, artifact, replace(identity, **{field: "changed"})) is None + + +def test_lease_isolates_loads_and_cleanup_preserves_original(tmp_path, monkeypatch): + backend, root, graph, identity, artifact = prepared(tmp_path, monkeypatch) + leased = backend.load_graph(graph, artifact, identity) + assert leased is not None and leased.cpg_path != graph + leased.cpg_path.write_bytes(b"query mutated working copy") + assert graph.read_bytes() == b"graph original" + leased.cleanup() + leased.cleanup() + assert not leased.cpg_path.exists() and graph.exists() + + +def test_corruption_partial_census_and_deadline_are_misses(tmp_path, monkeypatch): + backend, root, graph, identity, artifact = prepared(tmp_path, monkeypatch) + graph.write_bytes(b"corrupted") + assert backend.load_graph(graph, artifact, identity) is None + graph.write_bytes(b"graph original") + assert backend.load_graph(graph, replace(artifact, completeness="partial", diagnostics=("dropped_file",)), identity) is None + budget = ExecutionBudget(time.monotonic() - 1, 0.1) + assert backend.load_graph(graph, artifact, identity, execution_budget=budget) is None + bad = CpgResult(graph, lambda *_: {"files": "1", "methods": "1", "calls": "0", "file_names": [], "overlays": "dataflowOss"}) + assert backend.describe_graph(bad, root, identity) is None + + +def test_graph_identity_rejects_unreadable_or_symlink_sources(tmp_path, monkeypatch): + backend, root, *_ = prepared(tmp_path, monkeypatch) + (root / "link.js").symlink_to(root / "a.js") + with pytest.raises((ValueError, OSError)): + backend.graph_identity(root, language="javascript", declarations_digest="d", exclusions_digest="e") + + +def test_generic_driver_accepts_backend_validated_lease(tmp_path, monkeypatch): + from openultrasast.model.scan import _build, _dispose + + backend, root, graph, identity, artifact = prepared(tmp_path, monkeypatch) + budget = ExecutionBudget(time.monotonic() + 5, 0.2) + + class ReusingBackend: + def build(self, source, *, language, exclude, execution_budget): + assert source == root and language == "javascript" + assert execution_budget is budget + return backend.load_graph(graph, artifact, identity, execution_budget=execution_budget) + + result = _build(ReusingBackend(), root, "javascript", execution_budget=budget) + assert result is not None + assert result.run("census", {})["file_names"] == ["a.js"] + _dispose(result) + assert graph.read_bytes() == b"graph original"