mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-18 12:39:28 +00:00
feat(pre-push-safety-net): validate graph identity and isolated leases
This commit is contained in:
@@ -265,6 +265,21 @@ results. A query/fact change invalidates its answers without automatically rebui
|
|||||||
input/frontend/overlay semantics are unchanged. This supports iteration on the core as well as
|
input/frontend/overlay semantics are unchanged. This supports iteration on the core as well as
|
||||||
repeated push checks; it does not authorize reusing stale evidence after a source change.
|
repeated push checks; it does not authorize reusing stale evidence after a source change.
|
||||||
|
|
||||||
|
Preparation is also a reusable artifact boundary. Task 5.2 measured PMPro exhausting
|
||||||
|
30 seconds after materialization and before graph construction. Cache complete discovery
|
||||||
|
by immutable source tree identity plus discovery code, shipping/layout facts and configuration;
|
||||||
|
retain snapshot readability proofs and declaration/boundary records. Reuse must preserve
|
||||||
|
repository-relative locations and must never deserialize repository-controlled executable objects.
|
||||||
|
Measure preparation, graph and query hits separately. An identical-tip hit demonstrates reuse,
|
||||||
|
not representative changed-code latency. Dependency/configuration changes invalidate affected
|
||||||
|
preparation and graph/query context even when an individual source file is unchanged.
|
||||||
|
|
||||||
|
Rollout approval remains an evidence decision under Requirement 6.5 and evaluation.md.
|
||||||
|
Group 6 establishes correctness and bounded reuse; group 8 must still demonstrate the joint
|
||||||
|
changed-code latency, completion, precision and recall gates on declared populations. Missing
|
||||||
|
independent positive/fixed/benign populations or an empty capability registry cannot yield GO.
|
||||||
|
No threshold, population denominator or alert eligibility rule is relaxed by this amendment.
|
||||||
|
|
||||||
Only validated, complete artifacts can satisfy complete-result lookups. Partial artifacts may be
|
Only validated, complete artifacts can satisfy complete-result lookups. Partial artifacts may be
|
||||||
retained for diagnostics but never promoted by cache presence. Publish atomically from private
|
retained for diagnostics but never promoted by cache presence. Publish atomically from private
|
||||||
temporary paths under a per-key lock; another push may wait only within its deadline. Apply a
|
temporary paths under a per-key lock; another push may wait only within its deadline. Apply a
|
||||||
|
|||||||
@@ -0,0 +1,39 @@
|
|||||||
|
# Group 6 implementation and verification
|
||||||
|
|
||||||
|
The maintainer authorized 6.1–6.4 together on 2026-09-13. A fresh implementer dispatch
|
||||||
|
failed with `agent thread limit reached`; the explicit kiro-impl manual fallback applies.
|
||||||
|
Reviews below are main-context adversarial reviews, not independent reviewer evidence.
|
||||||
|
The design amendment makes measured preparation/discovery costs explicit. Rollout gates
|
||||||
|
and the empty default capability registry are unchanged.
|
||||||
|
|
||||||
|
## Task 6.1 brief
|
||||||
|
|
||||||
|
Requirements 4.2, 4.3, 4.4 and 7.3; design: Cache and Engine Boundary.
|
||||||
|
Backend-owned identities bind readable source paths/bytes, declarations, exclusions,
|
||||||
|
frontend/engine versions, overlay implementation and options. Complete graphs require a
|
||||||
|
named source census, methods, calls and the dataflow overlay. Loading checks identity and
|
||||||
|
streamed graph digest, then issues a disposable copy through the existing CpgResult driver
|
||||||
|
contract. Old structural manifests without source paths cannot qualify. Sharded or partial
|
||||||
|
graphs cannot qualify. Query locations normalize to repository-relative paths.
|
||||||
|
|
||||||
|
## Status Report
|
||||||
|
- STATUS: READY_FOR_REVIEW
|
||||||
|
- TASK: 6.1
|
||||||
|
- RED_PHASE_OUTPUT: `/tmp/group6-1-red.log` and `/tmp/group6-1-flag-off.log`: four failures before implementation/with the temporary switch OFF. ON passed; switch removed and five final tests passed.
|
||||||
|
- TESTS_RUN: 105 focused tests passed; full suite 1246 passed, 9 skipped (105.31s), followed by the final additional driver integration test and focused rerun. Ruff and mypy passed.
|
||||||
|
- EVIDENCE: source-mounted nonroot Docker Joern 4.0.625, network disabled, 3 GiB limit: read 45 JavaScript bytes, census 1 file/4 methods/3 calls, sealed graph loaded through a disposable lease, original digest unchanged after query and cleanup. This is a correctness control, not a latency benchmark.
|
||||||
|
- CONCERNS: persistent publication and runner selection belong to 6.2–6.4; this task provides the validated backend lease contract. Main-context review fallback.
|
||||||
|
|
||||||
|
## Review Verdict
|
||||||
|
- VERDICT: APPROVED
|
||||||
|
- TASK: 6.1
|
||||||
|
- MECHANICAL_RESULTS: focused tests, full regression suite, ruff, mypy and diff whitespace checks passed; no introduced placeholders or secrets.
|
||||||
|
- FINDINGS: review repaired unbounded full-graph buffering to streamed hashing/copy; unreadable source walks fail explicitly; single PHP graph census passes through unchanged while sharded graphs cannot seal. Existing driver accepts and releases the leased CpgResult.
|
||||||
|
- SUMMARY: Backend identity and isolated leases satisfy task 6.1 within the approved boundary.
|
||||||
|
|
||||||
|
## Verification Result
|
||||||
|
- STATUS: VERIFIED
|
||||||
|
- CLAIM_TYPE: TASK
|
||||||
|
- CLAIM: Task 6.1 identity and lease behavior, not persistent runner reuse or rollout readiness.
|
||||||
|
- EVIDENCE: tests/test_graph_reuse.py, 105 focused passing tests, full suite and real Joern control above; measurement JSON retained in benchmarks/measurements/2026-09-13-graph-lease-control.json.
|
||||||
|
- GAPS: Warm push feasibility and independent capability eligibility remain unproven.
|
||||||
@@ -65,6 +65,11 @@
|
|||||||
"at": "2026-09-13T12:40:57.961577+00:00",
|
"at": "2026-09-13T12:40:57.961577+00:00",
|
||||||
"source": "approved",
|
"source": "approved",
|
||||||
"action": "Authorized group 5 after the group 4 completion summary: explicit transaction replay, then the first PMPro/frozen-core Node experiment. Review and fresh evidence remain required."
|
"action": "Authorized group 5 after the group 4 completion summary: explicit transaction replay, then the first PMPro/frozen-core Node experiment. Review and fresh evidence remain required."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"at": "2026-09-13T13:58:52.336506+00:00",
|
||||||
|
"source": "continue with group 6 as a whole. do we need to amend the spec to get a go for the rollout",
|
||||||
|
"action": "Authorized group 6 (6.1\u20136.4). Clarified complete preparation/discovery reuse after measured group 5 bottleneck; rollout gates remain unchanged. Manual Kiro review fallback remains necessary after a fresh agent dispatch hit the thread limit."
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"task_review": {
|
"task_review": {
|
||||||
|
|||||||
@@ -161,7 +161,7 @@ hook latency is admitted.
|
|||||||
- _Requirements: 6.2, 6.3, 6.4, 6.5, 8.1, 8.2, 8.3, 8.4_
|
- _Requirements: 6.2, 6.3, 6.4, 6.5, 8.1, 8.2, 8.3, 8.4_
|
||||||
|
|
||||||
- [ ] 6. Reuse valid graphs and results without stale conclusions
|
- [ ] 6. Reuse valid graphs and results without stale conclusions
|
||||||
- [ ] 6.1 Validate graph artifact identity and leases
|
- [x] 6.1 Validate graph artifact identity and leases
|
||||||
- Have the backend own source/declaration/exclusion digests, frontend/engine/overlay options, graph census and complete/partial identity.
|
- Have the backend own source/declaration/exclusion digests, frontend/engine/overlay options, graph census and complete/partial identity.
|
||||||
- Permit validated graph leases in the generic driver while preserving the fresh-build path; isolate potentially mutating loads and retain bounded cleanup.
|
- Permit validated graph leases in the generic driver while preserving the fresh-build path; isolate potentially mutating loads and retain bounded cleanup.
|
||||||
- Done when identical valid inputs can reuse a graph while any changed graph input or failed census rejects the lease; original cached artifacts survive query loading unchanged.
|
- Done when identical valid inputs can reuse a graph while any changed graph input or failed census rejects the lease; original cached artifacts survive query loading unchanged.
|
||||||
@@ -181,6 +181,7 @@ hook latency is admitted.
|
|||||||
- _Boundary: Push cache, policy integration_
|
- _Boundary: Push cache, policy integration_
|
||||||
- _Requirements: 3.4, 4.3, 4.4, 6.4, 7.2, 7.3_
|
- _Requirements: 3.4, 4.3, 4.4, 6.4, 7.2, 7.3_
|
||||||
- [ ] 6.4 Integrate cache reuse into the shared push deadline
|
- [ ] 6.4 Integrate cache reuse into the shared push deadline
|
||||||
|
- Reuse complete preparation/discovery by immutable tree and discovery/fact/configuration identity, retaining readable snapshot and declaration/boundary records; measure its costs and hits separately from graph/query reuse.
|
||||||
- Reuse identical snapshots/unchanged declared units across refs without merging different comparisons; changed units rebuild and stale graph relationships remain hints only.
|
- Reuse identical snapshots/unchanged declared units across refs without merging different comparisons; changed units rebuild and stale graph relationships remain hints only.
|
||||||
- Account for lookup, lock wait, lease, build/query and result persistence in the same deadline; preserve completed outcomes on cancellation.
|
- Account for lookup, lock wait, lease, build/query and result persistence in the same deadline; preserve completed outcomes on cancellation.
|
||||||
- Done when cached and uncached replay produce equal evidence/admission for the same completed questions under compatible semantics, with full equivalence under a sufficiently large controlled budget. Under the hook deadline, verify truthful extra completion/deferred scope from cache reuse rather than requiring identical scope; no stale answer survives a dependency/configuration edit.
|
- Done when cached and uncached replay produce equal evidence/admission for the same completed questions under compatible semantics, with full equivalence under a sufficiently large controlled budget. Under the hook deadline, verify truthful extra completion/deferred scope from cache reuse rather than requiring identical scope; no stale answer survives a dependency/configuration edit.
|
||||||
|
|||||||
@@ -0,0 +1,59 @@
|
|||||||
|
{
|
||||||
|
"source_bytes": 45,
|
||||||
|
"artifact": {
|
||||||
|
"identity": {
|
||||||
|
"source_digest": "aab4a6878883db96c5eb1d0dd98e160e30d16fa2b215735bdac6ff5a41f9783d",
|
||||||
|
"declarations_digest": "none",
|
||||||
|
"exclusions_digest": "none",
|
||||||
|
"unit": "repository",
|
||||||
|
"language": "javascript",
|
||||||
|
"engine_version": "4.0.625",
|
||||||
|
"frontend": "jssrc2cpg",
|
||||||
|
"frontend_version": "4.0.625",
|
||||||
|
"overlay_semantics": [
|
||||||
|
"dataflowOss"
|
||||||
|
],
|
||||||
|
"overlay_options": [
|
||||||
|
[
|
||||||
|
"implementation",
|
||||||
|
"96d0a93a3edb7e4a999a380b5823cd6d593ae2a49cf2a503f923483eda43989a"
|
||||||
|
]
|
||||||
|
],
|
||||||
|
"frontend_options": [
|
||||||
|
[
|
||||||
|
"include_tests",
|
||||||
|
"True"
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"java_options",
|
||||||
|
"default"
|
||||||
|
]
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"graph_digest": "a02bef8f7d4d1f2c420cb9bb4c5cc9df2fe86d86ad4e429021c95ad76a996873",
|
||||||
|
"source_bytes": 45,
|
||||||
|
"census": {
|
||||||
|
"files": 1,
|
||||||
|
"methods": 4,
|
||||||
|
"calls": 3
|
||||||
|
},
|
||||||
|
"completeness": "complete",
|
||||||
|
"diagnostics": [],
|
||||||
|
"source_root": "/tmp/tmpb4ff84v8",
|
||||||
|
"source_paths": [
|
||||||
|
"a.js"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"lease_census": {
|
||||||
|
"files": "1",
|
||||||
|
"methods": "4",
|
||||||
|
"calls": "3",
|
||||||
|
"file_names": [
|
||||||
|
"a.js"
|
||||||
|
],
|
||||||
|
"overlays": "base,controlflow,typerel,callgraph,dataflowOss",
|
||||||
|
"maxHeapMB": "2048"
|
||||||
|
},
|
||||||
|
"original_unchanged": true,
|
||||||
|
"execution": "source-mounted openultrasast:dev image 7f71174daa9e; nonroot, network none, memory 3g; correctness control, not latency"
|
||||||
|
}
|
||||||
@@ -1,7 +1,7 @@
|
|||||||
"""Backend-owned graph provenance and live resource boundary (pre-push Req 4.3, 7.3).
|
"""Backend-owned graph provenance and live resource boundary (pre-push Req 4.3, 7.3).
|
||||||
|
|
||||||
Structural validity is necessary but insufficient for reuse. Task 6.1 will validate
|
Structural manifests carry provenance; the backend reads source bytes and seals a named
|
||||||
actual source bytes, graph digest/census and engine compatibility before issuing leases.
|
graph census, then verifies graph bytes and compatibility before issuing isolated leases.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
@@ -54,6 +54,8 @@ class GraphArtifact(Contract):
|
|||||||
census: GraphCensus
|
census: GraphCensus
|
||||||
completeness: Literal["complete", "partial"]
|
completeness: Literal["complete", "partial"]
|
||||||
diagnostics: tuple[str, ...]
|
diagnostics: tuple[str, ...]
|
||||||
|
source_root: str = "relative"
|
||||||
|
source_paths: tuple[str, ...] = ()
|
||||||
|
|
||||||
def __post_init__(self) -> None:
|
def __post_init__(self) -> None:
|
||||||
super().__post_init__()
|
super().__post_init__()
|
||||||
@@ -71,7 +73,7 @@ class GraphLease(Protocol):
|
|||||||
The backend validates the artifact before issuing the lease and isolates loading
|
The backend validates the artifact before issuing the lease and isolates loading
|
||||||
when it may mutate graph bytes. The consumer owns releasing it exactly once; the
|
when it may mutate graph bytes. The consumer owns releasing it exactly once; the
|
||||||
backend's release is idempotent and bounded by the supplied monotonic deadline.
|
backend's release is idempotent and bounded by the supplied monotonic deadline.
|
||||||
A cached artifact survives release. Implementations belong to task 6.1.
|
A cached artifact survives release.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
@property
|
@property
|
||||||
@@ -81,3 +83,108 @@ class GraphLease(Protocol):
|
|||||||
def path(self) -> Path: ...
|
def path(self) -> Path: ...
|
||||||
|
|
||||||
def close(self, *, deadline_monotonic: float) -> None: ...
|
def close(self, *, deadline_monotonic: float) -> None: ...
|
||||||
|
|
||||||
|
|
||||||
|
def digest_value(value: object) -> str:
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
|
||||||
|
return hashlib.sha256(json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=True).encode()).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def read_bytes(path: Path, deadline: float) -> bytes:
|
||||||
|
"""Read a regular local file without following a substituted final symlink."""
|
||||||
|
import os
|
||||||
|
import stat
|
||||||
|
import time
|
||||||
|
|
||||||
|
chunks: list[bytes] = []
|
||||||
|
fd = os.open(path, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK)
|
||||||
|
with os.fdopen(fd, "rb") as stream:
|
||||||
|
if not stat.S_ISREG(os.fstat(stream.fileno()).st_mode):
|
||||||
|
raise ValueError("artifact_not_regular")
|
||||||
|
while True:
|
||||||
|
if time.monotonic() >= deadline:
|
||||||
|
raise TimeoutError("deadline_exhausted")
|
||||||
|
chunk = stream.read(1024 * 1024)
|
||||||
|
if not chunk:
|
||||||
|
return b"".join(chunks)
|
||||||
|
chunks.append(chunk)
|
||||||
|
|
||||||
|
|
||||||
|
def source_inventory(root: Path, deadline: float) -> tuple[str, int, tuple[str, ...]]:
|
||||||
|
import hashlib
|
||||||
|
import os
|
||||||
|
import time
|
||||||
|
|
||||||
|
rows = []
|
||||||
|
size = 0
|
||||||
|
if root.is_symlink():
|
||||||
|
raise ValueError("symlink_source_root")
|
||||||
|
|
||||||
|
def unreadable(error: OSError) -> None:
|
||||||
|
raise error
|
||||||
|
|
||||||
|
for directory, dirs, files in os.walk(root, followlinks=False, onerror=unreadable):
|
||||||
|
if time.monotonic() >= deadline:
|
||||||
|
raise TimeoutError("deadline_exhausted")
|
||||||
|
if any((Path(directory) / name).is_symlink() for name in dirs):
|
||||||
|
raise ValueError("symlink_source")
|
||||||
|
for name in sorted(files):
|
||||||
|
path = Path(directory) / name
|
||||||
|
data = read_bytes(path, deadline)
|
||||||
|
size += len(data)
|
||||||
|
rows.append((path.relative_to(root).as_posix(), hashlib.sha256(data).hexdigest()))
|
||||||
|
if not rows or not size:
|
||||||
|
raise ValueError("source_unreadable_or_empty")
|
||||||
|
rows.sort()
|
||||||
|
return digest_value(rows), size, tuple(path for path, _ in rows)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class IsolatedGraphLease:
|
||||||
|
artifact: GraphArtifact
|
||||||
|
path: Path
|
||||||
|
_closed: bool = False
|
||||||
|
|
||||||
|
def close(self, *, deadline_monotonic: float) -> None:
|
||||||
|
from .backend import _DeadlineExpired, _remove_owned_tree
|
||||||
|
|
||||||
|
if self._closed:
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
_remove_owned_tree(self.path.parent, deadline_monotonic)
|
||||||
|
except (OSError, _DeadlineExpired):
|
||||||
|
return
|
||||||
|
self._closed = True
|
||||||
|
|
||||||
|
|
||||||
|
def graph_bytes(path: Path, deadline: float, destination: Path | None = None) -> str:
|
||||||
|
"""Hash and optionally copy a graph in bounded chunks; never buffer the full graph."""
|
||||||
|
import hashlib
|
||||||
|
import os
|
||||||
|
import stat
|
||||||
|
import time
|
||||||
|
from contextlib import ExitStack
|
||||||
|
|
||||||
|
digest = hashlib.sha256()
|
||||||
|
count = 0
|
||||||
|
with ExitStack() as stack:
|
||||||
|
fd = os.open(path, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK)
|
||||||
|
source = stack.enter_context(os.fdopen(fd, "rb"))
|
||||||
|
if not stat.S_ISREG(os.fstat(fd).st_mode):
|
||||||
|
raise ValueError("graph_not_regular")
|
||||||
|
output = stack.enter_context(destination.open("xb")) if destination is not None else None
|
||||||
|
while True:
|
||||||
|
if time.monotonic() >= deadline:
|
||||||
|
raise TimeoutError("deadline_exhausted")
|
||||||
|
chunk = source.read(1024 * 1024)
|
||||||
|
if not chunk:
|
||||||
|
break
|
||||||
|
count += len(chunk)
|
||||||
|
digest.update(chunk)
|
||||||
|
if output is not None:
|
||||||
|
output.write(chunk)
|
||||||
|
if not count:
|
||||||
|
raise ValueError("graph_empty")
|
||||||
|
return digest.hexdigest()
|
||||||
|
|||||||
@@ -32,9 +32,10 @@ import time
|
|||||||
from collections.abc import Callable, Mapping, Sequence
|
from collections.abc import Callable, Mapping, Sequence
|
||||||
from dataclasses import dataclass, field, replace
|
from dataclasses import dataclass, field, replace
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Protocol
|
from typing import Any, Protocol
|
||||||
|
|
||||||
from ..model.contracts import ExecutionBudget
|
from ..model.contracts import ExecutionBudget
|
||||||
|
from .artifact import GraphArtifact, GraphCensus, GraphIdentity
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
@@ -311,6 +312,118 @@ class JoernBackend:
|
|||||||
self.last_failure = session.last_failure
|
self.last_failure = session.last_failure
|
||||||
self._diagnostics = list(session._diagnostics)
|
self._diagnostics = list(session._diagnostics)
|
||||||
|
|
||||||
|
def graph_identity(
|
||||||
|
self,
|
||||||
|
root: Path,
|
||||||
|
*,
|
||||||
|
language: str,
|
||||||
|
declarations_digest: str,
|
||||||
|
exclusions_digest: str,
|
||||||
|
unit: str = "repository",
|
||||||
|
execution_budget: ExecutionBudget | None = None,
|
||||||
|
) -> GraphIdentity:
|
||||||
|
from .artifact import digest_value, read_bytes, source_inventory
|
||||||
|
|
||||||
|
deadline = execution_budget.deadline_monotonic if execution_budget else float("inf")
|
||||||
|
version = joern_version()
|
||||||
|
if version is None:
|
||||||
|
raise ValueError("engine_identity_unknown")
|
||||||
|
engine = ".".join(map(str, version))
|
||||||
|
source, _, _ = source_inventory(root, deadline)
|
||||||
|
code = digest_value([read_bytes(Path(__file__), deadline).hex(), read_bytes(self.queries_dir / "overlay.sc", deadline).hex()])
|
||||||
|
return GraphIdentity(
|
||||||
|
source,
|
||||||
|
declarations_digest,
|
||||||
|
exclusions_digest,
|
||||||
|
unit,
|
||||||
|
language,
|
||||||
|
engine,
|
||||||
|
_FRONTENDS.get(language, "joern-parse"),
|
||||||
|
engine,
|
||||||
|
(DATAFLOW_OVERLAY,),
|
||||||
|
(("implementation", code),),
|
||||||
|
(("include_tests", str(self.include_tests)), ("java_options", os.environ.get("JAVA_TOOL_OPTIONS", "default") or "default")),
|
||||||
|
)
|
||||||
|
|
||||||
|
def describe_graph(
|
||||||
|
self, result: CpgResult, root: Path, identity: GraphIdentity, *, execution_budget: ExecutionBudget | None = None
|
||||||
|
) -> GraphArtifact | None:
|
||||||
|
"""Seal only a named, complete, single graph census bound to bytes after loading."""
|
||||||
|
from .artifact import graph_bytes, source_inventory
|
||||||
|
|
||||||
|
budget = execution_budget or self.execution_budget
|
||||||
|
deadline = budget.deadline_monotonic if budget else float("inf")
|
||||||
|
try:
|
||||||
|
digest, size, paths = source_inventory(root, deadline)
|
||||||
|
if digest != identity.source_digest or result.unparsed:
|
||||||
|
return None
|
||||||
|
census = result.run("census", {})
|
||||||
|
if not isinstance(census, Mapping) or DATAFLOW_OVERLAY not in str(census.get("overlays", "")).split(","):
|
||||||
|
return None
|
||||||
|
names = census.get("file_names")
|
||||||
|
if not isinstance(names, list) or not all(isinstance(p, str) for p in names):
|
||||||
|
return None
|
||||||
|
normalized = {p.removeprefix(str(root) + "/").removeprefix("./") for p in names}
|
||||||
|
if not set(paths) <= normalized or int(str(census.get("shards", 1))) != 1:
|
||||||
|
return None
|
||||||
|
counts = GraphCensus(*(int(str(census[k])) for k in ("files", "methods", "calls")))
|
||||||
|
if counts.files < len(paths) or counts.methods < 1:
|
||||||
|
return None
|
||||||
|
return GraphArtifact(identity, graph_bytes(result.cpg_path, deadline), size, counts, "complete", (), str(root), paths)
|
||||||
|
except (OSError, ValueError, KeyError, TimeoutError):
|
||||||
|
return None
|
||||||
|
|
||||||
|
def load_graph(
|
||||||
|
self,
|
||||||
|
path: Path,
|
||||||
|
artifact: GraphArtifact,
|
||||||
|
expected: GraphIdentity,
|
||||||
|
*,
|
||||||
|
execution_budget: ExecutionBudget | None = None,
|
||||||
|
) -> CpgResult | None:
|
||||||
|
"""Validate sealed bytes, then query only a disposable private copy."""
|
||||||
|
from .artifact import IsolatedGraphLease, graph_bytes
|
||||||
|
|
||||||
|
budget = execution_budget or self.execution_budget
|
||||||
|
deadline = budget.deadline_monotonic if budget else float("inf")
|
||||||
|
cleanup_deadline = deadline + budget.cancellation_allowance_seconds if budget else float("inf")
|
||||||
|
scratch = None
|
||||||
|
try:
|
||||||
|
if artifact.identity != expected or artifact.completeness != "complete" or not artifact.source_paths:
|
||||||
|
return None
|
||||||
|
if artifact.census.files < len(artifact.source_paths) or artifact.census.methods < 1:
|
||||||
|
return None
|
||||||
|
scratch = Path(tempfile.mkdtemp(prefix="ousast-lease-"))
|
||||||
|
working = scratch / "cpg.bin"
|
||||||
|
if graph_bytes(path, deadline, working) != artifact.graph_digest:
|
||||||
|
raise ValueError("graph_digest_mismatch")
|
||||||
|
session = replace(self, execution_budget=budget)
|
||||||
|
lease = IsolatedGraphLease(artifact, working)
|
||||||
|
|
||||||
|
def normalize(value: object) -> Any:
|
||||||
|
if isinstance(value, str):
|
||||||
|
return value.replace(artifact.source_root + "/", "")
|
||||||
|
if isinstance(value, list):
|
||||||
|
return [normalize(item) for item in value]
|
||||||
|
if isinstance(value, dict):
|
||||||
|
return {key: normalize(item) for key, item in value.items()}
|
||||||
|
return value
|
||||||
|
|
||||||
|
return CpgResult(
|
||||||
|
working,
|
||||||
|
lambda kind, params: normalize(session.query(working, kind, params)),
|
||||||
|
lambda kind, requests: normalize(session.query_batch(working, kind, requests)),
|
||||||
|
cleanup=lambda: lease.close(deadline_monotonic=cleanup_deadline),
|
||||||
|
execution_diagnostics=lambda: tuple(session._diagnostics),
|
||||||
|
)
|
||||||
|
except (OSError, ValueError, TimeoutError):
|
||||||
|
if scratch is not None:
|
||||||
|
try:
|
||||||
|
_remove_owned_tree(scratch, cleanup_deadline)
|
||||||
|
except (OSError, _DeadlineExpired):
|
||||||
|
self._note("scratch_cleanup_incomplete")
|
||||||
|
return None
|
||||||
|
|
||||||
def _build_impl(self, root: Path, *, language: str = "", exclude: Sequence[str] = ()) -> CpgResult | None:
|
def _build_impl(self, root: Path, *, language: str = "", exclude: Sequence[str] = ()) -> CpgResult | None:
|
||||||
"""Build a CPG for ``root``. ``None`` on any failure — the caller degrades, never guesses.
|
"""Build a CPG for ``root``. ``None`` on any failure — the caller degrades, never guesses.
|
||||||
|
|
||||||
@@ -671,6 +784,8 @@ class JoernBackend:
|
|||||||
|
|
||||||
def query_across(self, shards: Sequence[Path], query: str, params: Mapping[str, object]) -> object | None:
|
def query_across(self, shards: Sequence[Path], query: str, params: Mapping[str, object]) -> object | None:
|
||||||
"""One question over every shard, answers concatenated. ``None`` only when no shard could answer."""
|
"""One question over every shard, answers concatenated. ``None`` only when no shard could answer."""
|
||||||
|
if query == "census":
|
||||||
|
return self.query(shards[0], query, params) if len(shards) == 1 else None
|
||||||
merged: list[object] = []
|
merged: list[object] = []
|
||||||
answered = 0
|
answered = 0
|
||||||
for shard in shards:
|
for shard in shards:
|
||||||
|
|||||||
@@ -23,6 +23,8 @@
|
|||||||
ujson.Obj(
|
ujson.Obj(
|
||||||
"files" -> cpg.file.size.toString,
|
"files" -> cpg.file.size.toString,
|
||||||
"methods" -> cpg.method.size.toString,
|
"methods" -> cpg.method.size.toString,
|
||||||
|
"calls" -> cpg.call.size.toString,
|
||||||
|
"file_names" -> ujson.Arr.from(cpg.file.name.l),
|
||||||
"overlays" -> cpg.metaData.overlays.l.mkString(","),
|
"overlays" -> cpg.metaData.overlays.l.mkString(","),
|
||||||
"maxHeapMB" -> (Runtime.getRuntime.maxMemory / 1048576).toString
|
"maxHeapMB" -> (Runtime.getRuntime.maxMemory / 1048576).toString
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -0,0 +1,90 @@
|
|||||||
|
"""Real byte identity and isolated graph lease boundaries, without launching Joern."""
|
||||||
|
|
||||||
|
import time
|
||||||
|
from dataclasses import replace
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from openultrasast.cpg.backend import CpgResult, JoernBackend
|
||||||
|
from openultrasast.model.contracts import ExecutionBudget
|
||||||
|
|
||||||
|
|
||||||
|
def prepared(tmp_path, monkeypatch):
|
||||||
|
import openultrasast.cpg.backend as module
|
||||||
|
|
||||||
|
monkeypatch.setattr(module, "joern_version", lambda: (4, 0, 625))
|
||||||
|
root = tmp_path / "source"
|
||||||
|
root.mkdir()
|
||||||
|
(root / "a.js").write_text("function f(x) { return x; }")
|
||||||
|
graph = tmp_path / "cpg.bin"
|
||||||
|
graph.write_bytes(b"graph original")
|
||||||
|
backend = JoernBackend()
|
||||||
|
census = {"files": "1", "methods": "1", "calls": "0", "file_names": [str(root / "a.js")], "overlays": "dataflowOss"}
|
||||||
|
monkeypatch.setattr(JoernBackend, "query", lambda *args: census)
|
||||||
|
identity = backend.graph_identity(root, language="javascript", declarations_digest="declarations", exclusions_digest="exclusions")
|
||||||
|
result = CpgResult(graph, lambda *_: census)
|
||||||
|
artifact = backend.describe_graph(result, root, identity)
|
||||||
|
return backend, root, graph, identity, artifact
|
||||||
|
|
||||||
|
|
||||||
|
def test_identity_reads_content_and_rejects_changed_inputs(tmp_path, monkeypatch):
|
||||||
|
backend, root, graph, identity, artifact = prepared(tmp_path, monkeypatch)
|
||||||
|
assert artifact.source_bytes == (root / "a.js").stat().st_size > 0
|
||||||
|
lease = backend.load_graph(graph, artifact, identity)
|
||||||
|
assert lease is not None
|
||||||
|
lease.cleanup()
|
||||||
|
(root / "a.js").write_text("changed")
|
||||||
|
changed = backend.graph_identity(root, language="javascript", declarations_digest="declarations", exclusions_digest="exclusions")
|
||||||
|
assert changed != identity
|
||||||
|
assert backend.load_graph(graph, artifact, changed) is None
|
||||||
|
for field in ("declarations_digest", "exclusions_digest", "engine_version", "frontend_version"):
|
||||||
|
assert backend.load_graph(graph, artifact, replace(identity, **{field: "changed"})) is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_lease_isolates_loads_and_cleanup_preserves_original(tmp_path, monkeypatch):
|
||||||
|
backend, root, graph, identity, artifact = prepared(tmp_path, monkeypatch)
|
||||||
|
leased = backend.load_graph(graph, artifact, identity)
|
||||||
|
assert leased is not None and leased.cpg_path != graph
|
||||||
|
leased.cpg_path.write_bytes(b"query mutated working copy")
|
||||||
|
assert graph.read_bytes() == b"graph original"
|
||||||
|
leased.cleanup()
|
||||||
|
leased.cleanup()
|
||||||
|
assert not leased.cpg_path.exists() and graph.exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_corruption_partial_census_and_deadline_are_misses(tmp_path, monkeypatch):
|
||||||
|
backend, root, graph, identity, artifact = prepared(tmp_path, monkeypatch)
|
||||||
|
graph.write_bytes(b"corrupted")
|
||||||
|
assert backend.load_graph(graph, artifact, identity) is None
|
||||||
|
graph.write_bytes(b"graph original")
|
||||||
|
assert backend.load_graph(graph, replace(artifact, completeness="partial", diagnostics=("dropped_file",)), identity) is None
|
||||||
|
budget = ExecutionBudget(time.monotonic() - 1, 0.1)
|
||||||
|
assert backend.load_graph(graph, artifact, identity, execution_budget=budget) is None
|
||||||
|
bad = CpgResult(graph, lambda *_: {"files": "1", "methods": "1", "calls": "0", "file_names": [], "overlays": "dataflowOss"})
|
||||||
|
assert backend.describe_graph(bad, root, identity) is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_graph_identity_rejects_unreadable_or_symlink_sources(tmp_path, monkeypatch):
|
||||||
|
backend, root, *_ = prepared(tmp_path, monkeypatch)
|
||||||
|
(root / "link.js").symlink_to(root / "a.js")
|
||||||
|
with pytest.raises((ValueError, OSError)):
|
||||||
|
backend.graph_identity(root, language="javascript", declarations_digest="d", exclusions_digest="e")
|
||||||
|
|
||||||
|
|
||||||
|
def test_generic_driver_accepts_backend_validated_lease(tmp_path, monkeypatch):
|
||||||
|
from openultrasast.model.scan import _build, _dispose
|
||||||
|
|
||||||
|
backend, root, graph, identity, artifact = prepared(tmp_path, monkeypatch)
|
||||||
|
budget = ExecutionBudget(time.monotonic() + 5, 0.2)
|
||||||
|
|
||||||
|
class ReusingBackend:
|
||||||
|
def build(self, source, *, language, exclude, execution_budget):
|
||||||
|
assert source == root and language == "javascript"
|
||||||
|
assert execution_budget is budget
|
||||||
|
return backend.load_graph(graph, artifact, identity, execution_budget=execution_budget)
|
||||||
|
|
||||||
|
result = _build(ReusingBackend(), root, "javascript", execution_budget=budget)
|
||||||
|
assert result is not None
|
||||||
|
assert result.run("census", {})["file_names"] == ["a.js"]
|
||||||
|
_dispose(result)
|
||||||
|
assert graph.read_bytes() == b"graph original"
|
||||||
Reference in New Issue
Block a user