diff --git a/.kiro/specs/pre-push-safety-net/brief.md b/.kiro/specs/pre-push-safety-net/brief.md index bd0d992..57de10b 100644 --- a/.kiro/specs/pre-push-safety-net/brief.md +++ b/.kiro/specs/pre-push-safety-net/brief.md @@ -1,6 +1,6 @@ # Brief: pre-push-safety-net -Status: requirements, design and tasks approved on 2026-09-12; group 1 and task 2.1 verified complete; group 2 implementation underway. Language: English. +Status: requirements, design and tasks approved on 2026-09-12; groups 1 and 2 verified complete (7/27 subtasks); groups 3–8 pending. Language: English. ## Problem @@ -11,8 +11,8 @@ maintainer explicitly rejects noisy advisory output as well as noisy blocking ou ## Current State The scan driver and Joern arbiters exist, with demonstrated PHP/Python cases. Immutable push-tip -resolution is implemented and verified. Isolated source materialization and change context are -under implementation; persistent graph caching and an end-to-end deadline remain pending. +resolution, isolated source materialization and lexical change context are implemented and verified. +Ranker integration, persistent graph caching and an end-to-end deadline remain pending. The driver builds a whole-root graph before cutting the region budget. The latest PMPro ranking experiment alone spends about 130 seconds building and 296 seconds extracting evidence. A manually selected two-file slice finding a CVE does not demonstrate automatic incremental analysis. diff --git a/.kiro/specs/pre-push-safety-net/design.md b/.kiro/specs/pre-push-safety-net/design.md index 8c911f5..d1e71fe 100644 --- a/.kiro/specs/pre-push-safety-net/design.md +++ b/.kiro/specs/pre-push-safety-net/design.md @@ -192,6 +192,12 @@ automatic fetch inside the hook. Peel annotated tags to commits; other objects a Deletion updates require no target scan. Deduplicate identical target graphs, but retain separate comparisons when the same target has different bases. Multiple refs share one deadline. +The snapshot implementation uses `SnapshotAdapter.resolve_updates`, the owned `materialize` +context manager, and `compare`. Preparation has explicit file, source-byte, tree/diff-output and +line-mapping limits. Materialization verifies blob identity and records omitted symlinks, gitlinks +and LFS pointers. Its completeness flag describes source preparation, not security coverage. +The shipped implementation uses Linux directory-relative cleanup that does not follow links. + Materialize tracked blobs directly into isolated scratch storage; do not checkout over the user's tree or run filters/build scripts. Use NUL-delimited tree/diff records and argument arrays. Include deletions as impact seeds and retain rename attribution. Symlinks are not followed outside the @@ -206,6 +212,17 @@ an alternative inclusion policy. The existing ranker produces `ScopeDecision`: s IDs, their priority/evidence, deferred question IDs with reasons, and unresolved context boundaries. Identifiers include unit/language/path/function/family to avoid collisions across partitions. +Adapter-produced contexts declare `path_encoding="filesystem-bytes-hex"` for structural paths +(changed/deleted/declaration paths, spans, renames and line correspondences). Use +`ChangeContext.decode_path` when matching canonical question paths; semantic relationship +`QuestionIdentity` paths retain their normal encoding. Existing text-context construction remains +compatible. This preserves unusual and non-UTF-8 filesystem names without ambiguous display escaping. +The caller supplies known declaration/configuration paths as raw bytes; an unavailable declaration +inventory is explicitly unresolved. Unique identical lines outside edit hunks provide lexical +correspondence even after a declaration rename or guard removal. Repeated lines, unmatched moves +and ambiguous rename attribution remain unresolved; these anchors do not prove semantic function +identity, guard effects or defect novelty. Those decisions remain with tasks 3.3 and 4.1. + Third-party vendor exclusions apply before both preprocessing targets and graph construction. Tests remain available but unshipped, consistent with the approved layout decision. The ranker cannot reintroduce vendor sources. Known external summaries are versioned facts; without a relevant diff --git a/.kiro/specs/pre-push-safety-net/implementation-group-2.md b/.kiro/specs/pre-push-safety-net/implementation-group-2.md new file mode 100644 index 0000000..9a62961 --- /dev/null +++ b/.kiro/specs/pre-push-safety-net/implementation-group-2.md @@ -0,0 +1,73 @@ +# Group 2 implementation verification + +Date: 2026-09-12. Scope: tasks 2.1–2.3. All three tasks received independent +APPROVED reviews after task-local verification. The specification remains in +implementation: 7/27 executable subtasks are complete, with groups 3–8 pending. + +## Verification result + +- STATUS: VERIFIED +- CLAIM_TYPE: TASK +- CLAIM: Group 2 resolves immutable pushed revisions, materializes isolated tracked + source, and supplies bounded lexical change evidence to the shared contract. +- EVIDENCE: Fresh final `.venv/bin/pytest -q`: 1,069 passed, 9 skipped (exit 0). + Focused snapshot/contracts/audit suite: 148 passed. Global Ruff check and format + check passed (218 files formatted); touched-module mypy and compileall passed. + Detection, split-sink map and local pair regression gates each passed. +- GAPS: This is not feature-level GO. Ranker integration, graph/vendor partitions, + semantic change attribution, persistent caching and the operational hook remain + scheduled work. No hook detection, precision or latency claim follows. + +## Task evidence + +| Task | Verified result | +| --- | --- | +| 2.1 | Exact supplied base/head comparisons, configured unique merge bases, tag peeling, special-update dispositions and all ref associations. SHA-1/SHA-256 tests include replacement/graft suppression and missing promisor objects without fetching. Review caught Unicode-aware protocol splitting; six regression cases now retain Git-valid Unicode refs. Full suite at completion: 997 passed, 9 skipped. Commit `05da202`. | +| 2.2 | Direct bounded blob reads with object hashes, reversible binary paths, explicit symlink/gitlink/LFS limits, and owned scratch cleanup. Dirty/non-HEAD success and failure paths preserve live files, index, refs and hooks. Independent hanging-child cancellation removed scratch in 0.202 s. Full suite: 1,037 passed, 9 skipped. Commit `97fd633`. | +| 2.3 | NUL-safe comparison metadata, both-side changed spans, declared configuration paths, renamed-path attribution and unique unchanged-line anchors. PHP/JavaScript declaration-renaming and removed-guard fixtures preserve the unchanged operation's lexical anchor. Repeated lines, unmatched/ambiguous moves, binary and unavailable inputs retain explicit uncertainty. Independent review additionally checked 40 real-Git edits with 4,880 base bytes read. | + +The initial disabled implementations failed the relevant tests before implementation; +expanded tests passed after flag removal. Final normalization also fixed formatting +in the earlier push-foundation files without changing their behavior. + +## Real repository and packaged checks + +`benchmarks/measurements/2026-09-12-snapshot-materialization.json` records a lab +materialization of commit `05da202`: 1,157 tracked files and 8,737,198 source bytes. +Every materialized file was reopened and checked for length and SHA256; the pinned +2,547-byte `pyproject.toml` also matched its Git blob directly. Scratch removal and +unchanged live status/index were verified. Preparation took 5.783 seconds under an +explicit 120-second lab budget. This is preparation cost, not complete hook latency. +The artifact's source hashes match committed task 2.2 implementation `97fd633`. + +After final source formatting, `docker compose build ousast` exited 0. Image: +`sha256:2a0dd7943ca9b9c84adc821990d38bff13a7824c7d5a9f94d07c38cb04ce333f`. +Seven packaged source-file hashes matched the workspace. The non-root, +network-disabled `ops/smoke_snapshot.py` run verified 103 JavaScript bytes and 46 +PHP bytes, exact pushed comparisons, renamed/removed spans and declared configuration +context in a dirty non-HEAD checkout. Scratch was removed and live state preserved. +The packaged CLI help also exited 0. Retained evidence: +`benchmarks/measurements/2026-09-12-packaged-snapshot-smoke.json`. +Reproduction instructions are in `ops/README.md`. + +## Remaining limits and next group + +Full-tree `.venv/bin/mypy src/openultrasast` still exits 1 with the same three +pre-existing errors in `model/scan.py` at 299, 302 and 310, checking 108 source files. +No additional type errors were reported. The nine skipped tests remain outside +demonstrated coverage. + +Snapshot completeness describes materialized source only. The snapshot contains the +tracked population; task 3.4 applies vendor exclusion and supported language partitions +before graph construction and target selection. Cleanup currently targets the shipped +Linux environment. The resolver's transaction-wide deadline integration remains pending. + +Change-context structural paths use `filesystem-bytes-hex`; consumers must decode them +with `ChangeContext.decode_path` before matching canonical question paths. Declaration +paths come from the caller's project discovery; an unavailable inventory is recorded. +Lexical correspondence does not prove semantic function identity, a guard's effect, +or defect novelty. Tasks 3.3 and 4.1 own those decisions. + +Next: group 3 integrates the shared execution budget, exact existing-ranker scope, +affected first-party relationships and vendor/language boundaries. The first complete +PMPro/Node experiment remains task 5.2, before persistent caching in group 6. diff --git a/.kiro/specs/pre-push-safety-net/research.md b/.kiro/specs/pre-push-safety-net/research.md index a5fa2c4..ca4572f 100644 --- a/.kiro/specs/pre-push-safety-net/research.md +++ b/.kiro/specs/pre-push-safety-net/research.md @@ -6,7 +6,7 @@ - User direction: a scalable pre-push safety net as AI accelerates development; WordPress is one case. - Clarifications: advisory must be actionable, not SAST noise; the ranker is the scope mechanism; third-party vendor code must remain outside targets and graphs. -- Requirements, design and tasks are now approved, and group 1 implementation is verified complete; see +- Requirements, design and tasks are now approved, and groups 1 and 2 are verified complete; see `tasks.md` for verified completion. The discovery log below preserves the earlier drafting state and measurements. No hook latency claim is made. diff --git a/.kiro/specs/pre-push-safety-net/spec.json b/.kiro/specs/pre-push-safety-net/spec.json index d663e47..183bd75 100644 --- a/.kiro/specs/pre-push-safety-net/spec.json +++ b/.kiro/specs/pre-push-safety-net/spec.json @@ -1,7 +1,7 @@ { "feature_name": "pre-push-safety-net", "created_at": "2026-09-12T15:35:00Z", - "updated_at": "2026-09-12T18:32:19.033933Z", + "updated_at": "2026-09-12T19:09:25.547910Z", "language": "en", "phase": "implementation", "approvals": { @@ -19,7 +19,7 @@ } }, "ready_for_implementation": true, - "implementation_approval_note": "Groups 1 and 2 explicitly authorized by the maintainer. Group 1 verified complete; this run implements group 2 (2.1\u20132.3). Remaining task plan retains approval, with execution sequenced by the spec.", + "implementation_approval_note": "Groups 1 and 2 explicitly authorized by the maintainer and verified complete after independent review. Remaining task plan retains approval, with execution sequenced by the spec. Implementation of the complete hook is still pending.", "depends_on": [ "contributor-scan" ], @@ -61,11 +61,17 @@ "1.1", "1.2", "1.3", - "1.4" + "1.4", + "2.1", + "2.2", + "2.3" ], - "remaining_tasks": 23, - "verification_record": "implementation-group-1.md", - "next_group": 2, - "active_group": 2 + "remaining_tasks": 20, + "verification_record": "implementation-group-2.md", + "verification_records": [ + "implementation-group-1.md", + "implementation-group-2.md" + ], + "next_group": 3 } } diff --git a/.kiro/specs/pre-push-safety-net/tasks.md b/.kiro/specs/pre-push-safety-net/tasks.md index b4e60fd..24581a5 100644 --- a/.kiro/specs/pre-push-safety-net/tasks.md +++ b/.kiro/specs/pre-push-safety-net/tasks.md @@ -1,8 +1,8 @@ # Implementation Plan: pre-push-safety-net -Status: task plan approved by the maintainer's "approve." on 2026-09-12. Group 1 (tasks 1.1–1.4) -is complete after independent review and fresh verification. Remaining groups are pending; -the hook and capability admission are not yet implemented. See `implementation-group-1.md`. +Status: task plan approved by the maintainer's "approve." on 2026-09-12. Groups 1 and 2 (tasks 1.1–2.3) +are complete after independent review and fresh verification. Groups 3–8 remain pending; +the hook and capability admission are not yet implemented. See `implementation-group-1.md` and `implementation-group-2.md`. ## Execution contract @@ -55,7 +55,7 @@ the hook and capability admission are not yet implemented. See `implementation-g - _Requirements: 4.1, 4.3, 5.3, 5.4, 5.5, 7.1, 7.3_ - _Verified 2026-09-12:_ independent review APPROVED; 965 tests passed, 9 skipped; touched-source Ruff/mypy passed. Parent rebuilt the local image and verified non-root, network-disabled CLI startup, all contract imports, an 83-byte configuration round trip and invalid-budget rejection. Comparison-owned completion preserves different bases for a shared head. Contracts are structural; runtime snapshot/cache/runner/admission implementations remain in later tasks. -- [ ] 2. Resolve immutable pushed content +- [x] 2. Resolve immutable pushed content - [x] 2.1 Resolve every supplied ref update and comparison - Parse Git's complete update input using object-format-aware validation; resolve existing tips, new-branch configured bases, force-pushes, merges, peeled tags, deleted refs and unsupported objects. - Deduplicate equal targets without merging different base comparisons; retain every ref association. Do not silently substitute HEAD or fetch missing objects. @@ -70,12 +70,13 @@ the hook and capability admission are not yet implemented. See `implementation-g - _Boundary: Snapshot Adapter_ - _Requirements: 1.1, 1.2, 1.3, 2.3, 7.1, 7.3_ - _Verified 2026-09-12:_ independent review APPROVED; 1,037 tests passed, 9 skipped; focused 116 passed; touched Ruff/mypy passed. Real SHA-1/SHA-256 fixtures verify binary-safe paths, exact blobs, no filters, explicit symlink/gitlink/LFS limits, cancellation and unchanged live state. Independent hanging-child cleanup took 0.202 s. Parent reopened and checked all 1,157 materialized repository files (8,737,198 bytes), with scratch removed and live state preserved; evidence `benchmarks/measurements/2026-09-12-snapshot-materialization.json` records a 5.783 s preparation-only lab run, not end-to-end hook latency. -- [ ] 2.3 Attach changed spans and deletion/rename context +- [x] 2.3 Attach changed spans and deletion/rename context - Compare recorded base/head objects using NUL-safe path handling; retain removed spans, renamed identities and declaration/config changes. - Supply change context to the shared ranker contract rather than selecting an independent changed-files-only scope. - Done when renamed functions retain correspondence and a removed guard remains an impact seed for an unchanged operation, with missing correspondence explicitly unresolved. - _Boundary: Snapshot Adapter_ - _Requirements: 1.4, 2.1, 2.4, 3.4_ + - _Verified 2026-09-12:_ independent review APPROVED; fresh final suite 1,069 passed, 9 skipped; focused 148 passed. Global Ruff/format, touched mypy and regression gates passed. PHP/JavaScript fixtures retain removed spans and unchanged-operation anchors across declaration renames; uncertain moves, repeated lines, binary and unavailable inputs remain unresolved. Forty independent real-Git comparisons checked exact spans/anchors. Network-disabled image smoke preserved committed PHP/JavaScript bytes and dirty non-HEAD state, with cleanup verified; see `implementation-group-2.md`. Lexical anchors do not establish semantic impact or novelty. - [ ] 3. Integrate bounded execution and ranker scope with the existing engine - [ ] 3.1 (P) Enforce one remaining-time budget in the backend diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index 3a14545..6da404c 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -46,13 +46,14 @@ engine are out of scope. - [ ] pre-push-safety-net — Immutable pushed tips, existing-ranker integration, compatible artifact reuse, total deadline, change attribution, actionability admission and compact hook behavior. Dependencies: contributor-scan. Requirements/design approved 2026-09-12; tasks generated - (8 groups, 27 executable subtasks), independently reviewed and approved; group 1 is complete (4/27 tasks). + (8 groups, 27 executable subtasks), independently reviewed and approved; groups 1 and 2 are complete (7/27 tasks). ### Next concrete step -Group 1 has passed independent review: reproducible PHP/JavaScript runtime, pinned PMPro/Node inputs, -honest scoring and shared contracts. Continue with group 2's immutable push snapshots under the -approved task plan; see `pre-push-safety-net/implementation-group-1.md` for evidence and limits. +Groups 1 and 2 have passed independent review: reproducible runtime, pinned inputs, honest scoring, +shared contracts, immutable push snapshots and lexical change context. Continue with group 3: one +execution budget, exact ranker scope, affected relationships and vendor/language partitions. See +`pre-push-safety-net/implementation-group-2.md` for evidence and limits. The first complete experiment is 5.2, before persistent caching and hook installation work. Keep PMPro as the primary core-optimization example, then freeze the core and prove a changed security flow and its fixed twin on a JavaScript/Express Node API workload through diff --git a/.kiro/steering/safety-net.md b/.kiro/steering/safety-net.md index 5c1fe57..d220077 100644 --- a/.kiro/steering/safety-net.md +++ b/.kiro/steering/safety-net.md @@ -79,5 +79,6 @@ updates as a shortcut to an unmeasured latency promise. The product direction above is user-requested. The amended requirements, design and initial evaluation profile in `pre-push-safety-net` were approved for task generation on 2026-09-12 ("ok, next kiro phase"). The maintainer then approved the task plan and starting group 1 -("approve."). Implementation is authorized under that plan; task completion still requires +("approve.") and then group 2 ("ok, next kiro phase, approved"). Groups 1 and 2 are now verified +complete. Implementation is authorized under that plan; task completion still requires independent review and fresh evidence. Hook performance and capability admission remain unproven. diff --git a/benchmarks/measurements/2026-09-08-module-audit.json b/benchmarks/measurements/2026-09-08-module-audit.json index 9850103..be47a4a 100644 --- a/benchmarks/measurements/2026-09-08-module-audit.json +++ b/benchmarks/measurements/2026-09-08-module-audit.json @@ -4,7 +4,7 @@ "standalone_capability": 9 }, "modules": 107, - "note": "Refreshed 2026-09-12 after pre-push-safety-net task 2.1: local immutable ref resolution is spec-owned; runner integration remains pending.", + "note": "Refreshed 2026-09-12 after pre-push-safety-net group 2: immutable comparison, materialization and lexical change context are spec-owned; ranker/runner integration remains pending.", "orphaned": [], "rows": [ { @@ -351,7 +351,8 @@ "model.specs", "model.taint", "preprocess", - "push.contracts" + "push.contracts", + "push.snapshot" ], "module": "model", "reason": "reachable from gate, map_gate, pair_gate or the CLI" @@ -389,7 +390,8 @@ { "classification": "load_bearing", "importers": [ - "push.contracts" + "push.contracts", + "push.snapshot" ], "module": "model.contracts", "reason": "pre-push-safety-net Req 4.1/7.3: generic deadline, change and exact scope contracts; integration pending" diff --git a/benchmarks/measurements/2026-09-12-packaged-snapshot-smoke.json b/benchmarks/measurements/2026-09-12-packaged-snapshot-smoke.json new file mode 100644 index 0000000..13b4549 --- /dev/null +++ b/benchmarks/measurements/2026-09-12-packaged-snapshot-smoke.json @@ -0,0 +1,106 @@ +{ + "purpose": "packaged snapshot and change-context smoke, not vulnerability detection or hook latency", + "uid": 1000, + "git_version": "git version 2.43.0", + "source_bytes": { + "javascript": 103, + "php": 46 + }, + "elapsed_seconds": 0.066, + "snapshot_complete": true, + "live_state_preserved": true, + "scratch_removed": true, + "change_context": { + "base_revision": "571abb8f43ec7b3fa8de233be65114f57cb0e77b", + "head_revision": "9dfa656cc7e5b8f07737ab155d0fb23d90d90e07", + "changed_paths": [ + "6f6c642d6170692e6a73", + "6170692e6a73", + "7061636b6167652e6a736f6e" + ], + "deleted_paths": [], + "spans": [ + { + "path": "6f6c642d6170692e6a73", + "start_line": 1, + "end_line": 2, + "side": "base" + }, + { + "path": "6170692e6a73", + "start_line": 1, + "end_line": 1, + "side": "head" + }, + { + "path": "7061636b6167652e6a736f6e", + "start_line": 1, + "end_line": 1, + "side": "base" + }, + { + "path": "7061636b6167652e6a736f6e", + "start_line": 1, + "end_line": 1, + "side": "head" + } + ], + "renames": [ + { + "base_path": "6f6c642d6170692e6a73", + "head_path": "6170692e6a73" + } + ], + "declaration_paths": [ + "7061636b6167652e6a736f6e" + ], + "relationships": [], + "unresolved_boundaries": [ + "7061636b6167652e6a736f6e:line_correspondence_unavailable" + ], + "path_encoding": "filesystem-bytes-hex", + "line_correspondences": [ + { + "base_path": "6f6c642d6170692e6a73", + "head_path": "6170692e6a73", + "base_start_line": 3, + "base_end_line": 3, + "head_start_line": 2 + }, + { + "base_path": "6f6c642d6170692e6a73", + "head_path": "6170692e6a73", + "base_start_line": 4, + "base_end_line": 4, + "head_start_line": 3 + }, + { + "base_path": "6f6c642d6170692e6a73", + "head_path": "6170692e6a73", + "base_start_line": 5, + "base_end_line": 5, + "head_start_line": 4 + }, + { + "base_path": "6f6c642d6170692e6a73", + "head_path": "6170692e6a73", + "base_start_line": 6, + "base_end_line": 6, + "head_start_line": 5 + } + ] + }, + "image_id": "sha256:2a0dd7943ca9b9c84adc821990d38bff13a7824c7d5a9f94d07c38cb04ce333f", + "implementation_sha256": { + "src/openultrasast/push/snapshot.py": "c6891f6de9221f63e534372c3009190d227172ca0b006bfb68bb6cffd7bf38f3", + "src/openultrasast/model/contracts.py": "89ea70185f9dabd6a264d1cd79b3e0c73161f5af4f29fd6f391d27f357439bbf", + "src/openultrasast/push/contracts.py": "a0bbef3137cd1e43c46b45209e5997ca89d46f60460402333c7ef37b8f0aec99", + "src/openultrasast/config.py": "1c2dd83f5245222c4be3efff6952a2f26984f2743193aa17f536e2a87af75872", + "src/openultrasast/contracts.py": "646f944f44a203315944ab8939aaec2609c3a045469c6dab303441d675373fe1", + "src/openultrasast/cpg/artifact.py": "8d6fd7af3db55b22e5b150ecd322984291db4c847c45da0ee11d12be06181257", + "src/openultrasast/push_scoring.py": "085ca153732b8b900f162408ce834fccacccf1187788ba2b6f436dd65c68f1b7" + }, + "smoke_script_sha256": "e2402ceb5105d323388b1e2bc3dc2ceb01e73b75acdadbfede5656af9b070e7c", + "packaged_source_matches_workspace": true, + "cli_help_exit_code": 0 +} diff --git a/ops/README.md b/ops/README.md index 7b17d68..be18021 100644 --- a/ops/README.md +++ b/ops/README.md @@ -127,3 +127,17 @@ PHP, ABAP and Ghidra binaries. We currently model **python, javascript, java, c* **PHP works but needs a PHP interpreter**: `php2cpg` drives PHP-Parser and shells out to `php`, so on a host without it a PHP tree fails to build with an opaque "Process exited with code 1". Set `php_frontend: True` in the inventory to install `php-cli` alongside the JVM. + +## Snapshot preparation smoke + +After building the image, exercise immutable Git comparisons and isolated source +materialization without network access: + +```bash +docker run --rm -i --network none --entrypoint python openultrasast:dev - < ops/smoke_snapshot.py +``` + +The smoke reads committed PHP/JavaScript bytes from a dirty, non-HEAD checkout, +checks rename/deletion and declared configuration context, and verifies scratch +cleanup and unchanged live state. Its JSON reports snapshot preparation only; +it does not run the detector, install a hook, or establish hook latency. diff --git a/ops/smoke_snapshot.py b/ops/smoke_snapshot.py new file mode 100644 index 0000000..7993366 --- /dev/null +++ b/ops/smoke_snapshot.py @@ -0,0 +1,98 @@ +"""Check packaged Git snapshot preparation; no graph, detector or hook-latency claim.""" + +from __future__ import annotations + +import json +import os +import subprocess +import tempfile +import time +from pathlib import Path + +from openultrasast.model.contracts import ExecutionBudget +from openultrasast.push.snapshot import SnapshotAdapter + + +def main() -> None: + assert os.getuid() != 0, "exercise the shipped non-root runtime" + with tempfile.TemporaryDirectory(prefix="ousast-push-smoke-") as temporary: + repo = Path(temporary) / "repository" + repo.mkdir() + + def git(*args: str) -> bytes: + return subprocess.run(["git", "-C", str(repo), *args], check=True, capture_output=True, timeout=10).stdout + + def commit(message: str) -> str: + git("add", ".") + git("commit", "-m", message) + return git("rev-parse", "HEAD").decode().strip() + + git("init") + git("config", "user.name", "Snapshot smoke") + git("config", "user.email", "snapshot@example.invalid") + git("config", "core.hooksPath", os.devnull) + before = ( + b"function before(req, res) {\n if (!authorized(req)) return;\n" + b" const input = req.body;\n const value = input.value;\n res.send(value);\n}\n" + ) + after = before.replace(b"function before", b"function after").replace(b" if (!authorized(req)) return;\n", b"") + php = b" None: + super().__post_init__() + if self.base_start_line < 1 or self.base_end_line < self.base_start_line or self.head_start_line < 1: + raise ValueError("line correspondence requires positive ordered lines") + + @dataclass(frozen=True) class ChangeContext(Contract): base_revision: str | None @@ -82,6 +105,39 @@ class ChangeContext(Contract): declaration_paths: tuple[str, ...] relationships: tuple[AffectedRelationship, ...] unresolved_boundaries: tuple[str, ...] + # Filesystem paths can be whitespace-only or non-UTF8. Adapter-produced contexts use + # lowercase raw-byte hex for change paths, spans, renames and line anchors. + # Relationship QuestionIdentity paths retain canonical question encoding; they + # are not encoded or decoded by this field. + path_encoding: Literal["text", "filesystem-bytes-hex"] = "text" + line_correspondences: tuple[LineCorrespondence, ...] = () + + def decode_path(self, path: str) -> str: + """Return the filesystem spelling for matching ordinary question paths. + + Raw undecodable filename bytes survive through filesystem surrogate escapes; + serialization should retain the authoritative encoded contract path instead. + """ + return os.fsdecode(bytes.fromhex(path)) if self.path_encoding == "filesystem-bytes-hex" else path + + def __post_init__(self) -> None: + super().__post_init__() + if self.path_encoding == "filesystem-bytes-hex": + paths = [ + *self.changed_paths, + *self.deleted_paths, + *self.declaration_paths, + *(s.path for s in self.spans), + *(p for r in self.renames for p in (r.base_path, r.head_path)), + *(p for m in self.line_correspondences for p in (m.base_path, m.head_path)), + ] + for path in paths: + try: + raw = bytes.fromhex(path) + except ValueError as error: + raise ValueError("invalid raw filesystem path hex") from error + if not raw or b"\0" in raw or raw.hex() != path: + raise ValueError("invalid raw filesystem path hex") @dataclass(frozen=True) diff --git a/src/openultrasast/push/contracts.py b/src/openultrasast/push/contracts.py index 76e74b8..a8ff338 100644 --- a/src/openultrasast/push/contracts.py +++ b/src/openultrasast/push/contracts.py @@ -3,6 +3,7 @@ Git resolution, admission policy and execution are separate later implementations. No default result status can turn an unexecuted contract into a complete negative. """ + from __future__ import annotations import re @@ -92,8 +93,17 @@ class UpdateResolution(Contract): """One record per supplied line, including repeated ref associations.""" update: PushUpdate - disposition: Literal["ready", "deleted", "missing_target", "unsupported_target", "missing_base", "unsupported_base", - "no_merge_base", "ambiguous_base", "resolution_error"] + disposition: Literal[ + "ready", + "deleted", + "missing_target", + "unsupported_target", + "missing_base", + "unsupported_base", + "no_merge_base", + "ambiguous_base", + "resolution_error", + ] head_oid: str | None base_oid: str | None base_reason: str diff --git a/src/openultrasast/push/snapshot.py b/src/openultrasast/push/snapshot.py index a3f25dd..5c823d2 100644 --- a/src/openultrasast/push/snapshot.py +++ b/src/openultrasast/push/snapshot.py @@ -1,8 +1,9 @@ """Resolve supplied push tips using local Git objects, never the live checkout. No fetch, replacement objects, checkout, filters or project commands are used. -Snapshots contain exact tracked blobs; changed spans are a subsequent adapter task. +Snapshots and change evidence come from exact tracked objects, not working-tree files. """ + from __future__ import annotations import hashlib @@ -14,12 +15,19 @@ import stat import subprocess import tempfile import time +from collections import Counter from collections.abc import Callable, Iterator from contextlib import contextmanager, suppress from dataclasses import dataclass, replace from pathlib import Path -from openultrasast.model.contracts import ExecutionBudget +from openultrasast.model.contracts import ( + ChangeContext, + ChangedSpan, + ExecutionBudget, + LineCorrespondence, + PathRename, +) from openultrasast.push.contracts import ( PushComparison, PushResolution, @@ -41,9 +49,16 @@ class SnapshotAdapter: self.comparison_base = comparison_base # Hook-inherited Git variables may redirect objects or worktrees elsewhere. self._env = {key: value for key, value in os.environ.items() if not key.startswith("GIT_")} - self._env.update(GIT_NO_REPLACE_OBJECTS="1", GIT_NO_LAZY_FETCH="1", GIT_TERMINAL_PROMPT="0", - GIT_CONFIG_NOSYSTEM="1", GIT_CONFIG_GLOBAL=os.devnull, GIT_ALLOW_PROTOCOL="", GIT_OPTIONAL_LOCKS="0", - GIT_GRAFT_FILE=os.devnull) + self._env.update( + GIT_NO_REPLACE_OBJECTS="1", + GIT_NO_LAZY_FETCH="1", + GIT_TERMINAL_PROMPT="0", + GIT_CONFIG_NOSYSTEM="1", + GIT_CONFIG_GLOBAL=os.devnull, + GIT_ALLOW_PROTOCOL="", + GIT_OPTIONAL_LOCKS="0", + GIT_GRAFT_FILE=os.devnull, + ) self.object_format = self._required("rev-parse", "--show-object-format=storage") # Ask the installed Git for its object ID width rather than assuming SHA-1. self._oid_width = len(self._required("hash-object", "--stdin", input="")) @@ -57,17 +72,21 @@ class SnapshotAdapter: def _repository_path(self, *args: str) -> Path: # Binary framing also matters for the repository's own whitespace/non-UTF8 name. - raw = self._bounded_git(("rev-parse", *args), 65536, - ExecutionBudget(time.monotonic() + 10.0, 2.0), None, []) + raw = self._bounded_git(("rev-parse", *args), 65536, ExecutionBudget(time.monotonic() + 10.0, 2.0), None, []) return Path(os.fsdecode(raw.removesuffix(b"\n"))).resolve() def _git(self, *args: str, input: str | None = None) -> subprocess.CompletedProcess[str]: try: return subprocess.run( - ["git", "-c", "protocol.allow=never", "-c", "core.hooksPath=" + os.devnull, - "-C", str(self.repository), *args], - input=input, capture_output=True, text=True, encoding="utf-8", errors="surrogateescape", - env=self._env, timeout=10, check=False, + ["git", "-c", "protocol.allow=never", "-c", "core.hooksPath=" + os.devnull, "-C", str(self.repository), *args], + input=input, + capture_output=True, + text=True, + encoding="utf-8", + errors="surrogateescape", + env=self._env, + timeout=10, + check=False, ) except (OSError, subprocess.TimeoutExpired) as error: raise SnapshotInputError(f"local Git resolution failed: {type(error).__name__}") from error @@ -91,8 +110,7 @@ class SnapshotAdapter: if len(parts) != 4 or any(not part for part in parts) or any(ord(char) < 32 for char in record): raise SnapshotInputError(f"invalid pre-push update at line {number}") local_ref, local_oid, remote_ref, remote_oid = parts - if any(re.fullmatch(r"[0-9a-f]{" + str(self._oid_width) + r"}", oid) is None - for oid in (local_oid, remote_oid)): + if any(re.fullmatch(r"[0-9a-f]{" + str(self._oid_width) + r"}", oid) is None for oid in (local_oid, remote_oid)): raise SnapshotInputError(f"invalid {self.object_format} object ID at line {number}") if self._git("check-ref-format", remote_ref).returncode or not remote_ref.startswith("refs/"): raise SnapshotInputError(f"invalid destination ref at line {number}") @@ -127,8 +145,9 @@ class SnapshotAdapter: return UpdateResolution(update, "deleted", None, None, "ref_deleted_no_analysis") head, kind = self._commit(update.local_oid) if head is None: - return UpdateResolution(update, "missing_target" if kind == "missing" else "unsupported_target", - None, None, "target_not_a_local_commit") + return UpdateResolution( + update, "missing_target" if kind == "missing" else "unsupported_target", None, None, "target_not_a_local_commit" + ) if update.remote_oid != self._zero: base, kind = self._commit(update.remote_oid) disposition = "ready" if base else ("missing_base" if kind == "missing" else "unsupported_base") @@ -141,8 +160,9 @@ class SnapshotAdapter: return UpdateResolution(update, "missing_base", head, None, "configured_base_unavailable") anchor, kind = self._commit(configured.stdout.strip()) if anchor is None: - return UpdateResolution(update, "missing_base" if kind == "missing" else "unsupported_base", - head, None, "configured_base_not_a_local_commit") + return UpdateResolution( + update, "missing_base" if kind == "missing" else "unsupported_base", head, None, "configured_base_not_a_local_commit" + ) reason += ":" + anchor merged = self._git("merge-base", "--all", anchor, head) if merged.returncode not in (0, 1): @@ -179,10 +199,225 @@ class SnapshotAdapter: comparisons.append(PushComparison(*key, refs=(update.remote_ref,))) return PushResolution(self.object_format, tuple(records), tuple(comparisons), tuple(targets)) + def compare( + self, + comparison: PushComparison, + *, + declaration_paths: tuple[bytes, ...] | None = None, + budget: ExecutionBudget | None = None, + limits: SnapshotLimits | None = None, + cancelled: Callable[[], bool] | None = None, + ) -> ChangeContext: + """Supply bounded lexical change evidence, never select analysis scope. + + Change paths, spans, renames and line anchors use raw path hex. Canonical + question paths in relationships retain their own encoding. Declarations are + supplied by project discovery (including configuration/manifests); None + records unknown coverage. Unchanged-line anchors are not function identity + or security proof. Semantic relationships belong to frontend/ranker work. + """ + budget = budget or ExecutionBudget(time.monotonic() + 30.0, 2.0) + limits = limits or SnapshotLimits() + if declaration_paths is not None and ( + type(declaration_paths) is not tuple or any(type(p) is not bytes or not p or b"\0" in p for p in declaration_paths) + ): + raise ValueError("declaration paths must be a tuple of raw nonempty Git paths") + boundaries = ["declaration_paths_unavailable"] if declaration_paths is None else [] + changed: list[str] = [] + deleted: list[str] = [] + spans: list[ChangedSpan] = [] + renames: list[PathRename] = [] + mappings: list[LineCorrespondence] = [] + cleanup: list[float] = [] + total_bytes = 0 + remaining_diff = limits.max_diff_bytes + remaining_lines = limits.max_mapping_lines + try: + if comparison.base_oid is None: + raise SnapshotInputError("comparison_base_unavailable") + for oid in (comparison.base_oid, comparison.head_oid): + if re.fullmatch(r"[0-9a-f]{" + str(self._oid_width) + r"}", oid) is None: + raise SnapshotInputError("comparison_requires_immutable_commit_oid") + if self._bounded_git(("cat-file", "-t", oid), 64, budget, cancelled, cleanup) != b"commit\n": + raise SnapshotInputError("comparison_requires_commit_objects") + raw = self._bounded_git( + ( + "diff-tree", + "-r", + "--no-commit-id", + "--raw", + "-z", + "--no-abbrev", + "--no-ext-diff", + "--no-textconv", + "--no-relative", + "--ignore-submodules=none", + "--find-renames=50%", + "-l1000", + comparison.base_oid, + comparison.head_oid, + "--", + ), + min(limits.max_tree_bytes, remaining_diff), + budget, + cancelled, + cleanup, + ) + remaining_diff -= len(raw) + records = _diff_records(raw, limits.max_files) + statuses = {record[4] for record in records} + if {"A", "D"} <= statuses: + # Rename thresholds/limits can leave a moved declaration as A+D. + # Absence of a Git pairing is not proof of unrelated/new behavior. + boundaries.append("unmatched_added_deleted_path_correspondence") + old_blob_counts = Counter(record[2] for record in records if record[4] != "A") + new_blob_counts = Counter(record[3] for record in records if record[4] != "D") + for mode_base, mode_head, oid_base, oid_head, status, path_base, path_head in records: + _check_budget(budget, cancelled) + paths = tuple(p for p in (path_base, path_head) if p is not None) + for path in paths: + if path.hex() not in changed: + changed.append(path.hex()) + if status == "D": + assert path_base is not None + deleted.append(path_base.hex()) + if status == "R": + assert path_base is not None and path_head is not None + renames.append(PathRename(path_base.hex(), path_head.hex())) + label = (path_head or path_base or b"?").hex() + ambiguous_rename = status == "R" and (old_blob_counts[oid_base] > 1 or new_blob_counts[oid_head] > 1) + if ambiguous_rename: + boundaries.append(label + ":ambiguous_path_rename_correspondence") + try: + if any(len(p) > limits.max_path_bytes or len(p.split(b"/")) > limits.max_path_depth for p in paths): + raise SnapshotInputError("unsupported_path") + contents: list[bytes] = [] + for mode, oid in ((mode_base, oid_base), (mode_head, oid_head)): + if mode == "000000": + contents.append(b"") + continue + if mode not in ("100644", "100755"): + raise SnapshotInputError("unsupported_diff_entry") + size = int(self._bounded_git(("cat-file", "-s", oid), 64, budget, cancelled, cleanup)) + if size < 0 or size > min(limits.max_blob_bytes, limits.max_total_bytes - total_bytes): + raise SnapshotInputError("source_byte_limit") + data = self._bounded_git(("cat-file", "blob", oid), size, budget, cancelled, cleanup) + if ( + len(data) != size + or hashlib.new(self.object_format, b"blob " + str(size).encode() + b"\0" + data).hexdigest() != oid + ): + raise SnapshotInputError("blob_identity_mismatch") + total_bytes += size + if data.startswith(b"version https://git-lfs.github.com/spec/v1\n"): + raise SnapshotInputError("lfs_content_unavailable") + if b"\0" in data: + raise SnapshotInputError("binary_line_correspondence_unavailable") + contents.append(data) + before, after = contents + # LF is the only source-line delimiter; Unicode and CR remain data. + line_count = sum(data.count(b"\n") + int(bool(data) and not data.endswith(b"\n")) for data in (before, after)) + if line_count > remaining_lines: + raise SnapshotInputError("line_mapping_limit") + remaining_lines -= line_count + old_lines, new_lines = _source_lines(before), _source_lines(after) + if path_base is None: + if new_lines: + spans.append(ChangedSpan(label, 1, len(new_lines), "head")) + continue + if path_head is None: + if old_lines: + spans.append(ChangedSpan(label, 1, len(old_lines), "base")) + continue + patch = self._bounded_git( + ( + "diff", + "--no-ext-diff", + "--no-textconv", + "--no-color", + "--no-relative", + "--text", + "--unified=0", + "--inter-hunk-context=0", + "--diff-algorithm=myers", + "--no-indent-heuristic", + oid_base, + oid_head, + "--", + ), + remaining_diff, + budget, + cancelled, + cleanup, + ) + remaining_diff -= len(patch) + hunks = [ + (int(a), int(b) if b else 1, int(c), int(d) if d else 1) + for a, b, c, d in re.findall(rb"^@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@", patch, re.M) + ] + if before != after and not hunks: + raise SnapshotInputError("diff_hunks_unavailable") + pairs: list[tuple[int, int]] = [] + old_cursor = new_cursor = 0 + for old_start, old_count, new_start, new_count in hunks: + # Git uses the preceding line for a zero-width edit interval. + old_index = old_start - 1 if old_count else old_start + new_index = new_start - 1 if new_count else new_start + if ( + old_index < old_cursor + or new_index < new_cursor + or old_index + old_count > len(old_lines) + or new_index + new_count > len(new_lines) + ): + raise SnapshotInputError("invalid_diff_hunks") + pairs.extend(_equal_line_pairs(old_lines, new_lines, old_cursor, old_index, new_cursor, new_index)) + if old_count: + spans.append(ChangedSpan(path_base.hex(), old_start, old_start + old_count - 1, "base")) + if new_count: + spans.append(ChangedSpan(path_head.hex(), new_start, new_start + new_count - 1, "head")) + old_cursor, new_cursor = old_index + old_count, new_index + new_count + pairs.extend(_equal_line_pairs(old_lines, new_lines, old_cursor, len(old_lines), new_cursor, len(new_lines))) + old_counts, new_counts = Counter(old_lines), Counter(new_lines) + ambiguous = False + for old_index, new_index in () if ambiguous_rename else pairs: + _check_budget(budget, cancelled) + content = old_lines[old_index] + if old_counts[content] != 1 or new_counts[content] != 1: + ambiguous = True + continue + mappings.append(LineCorrespondence(path_base.hex(), path_head.hex(), old_index + 1, old_index + 1, new_index + 1)) + if ambiguous: + boundaries.append(label + ":ambiguous_line_correspondence") + if before and after and not any(m.base_path == path_base.hex() for m in mappings): + boundaries.append(label + ":line_correspondence_unavailable") + except (SnapshotInputError, ValueError) as error: + boundaries.append(label + ":" + str(error)) + if str(error) in ("deadline_exhausted", "cancelled", "git_cleanup_deadline_exhausted"): + break + except (SnapshotInputError, ValueError) as error: + boundaries.append(str(error)) + declared = {p.hex() for p in declaration_paths or ()} + return ChangeContext( + comparison.base_oid, + comparison.head_oid, + tuple(changed), + tuple(deleted), + tuple(spans), + tuple(renames), + tuple(p for p in changed if p in declared), + (), + tuple(boundaries), + "filesystem-bytes-hex", + tuple(mappings), + ) + @contextmanager def materialize( - self, commit_oid: str, *, budget: ExecutionBudget | None = None, - limits: SnapshotLimits | None = None, scratch_parent: Path | None = None, + self, + commit_oid: str, + *, + budget: ExecutionBudget | None = None, + limits: SnapshotLimits | None = None, + scratch_parent: Path | None = None, cancelled: Callable[[], bool] | None = None, ) -> Iterator[MaterializedSnapshot]: """Own one disposable tracked-object tree, with no checkout/filter execution. @@ -208,8 +443,9 @@ class SnapshotAdapter: try: if self._bounded_git(("cat-file", "-t", commit_oid), 64, budget, cancelled, cleanup_deadline) != b"commit\n": raise SnapshotInputError("unsupported_snapshot_object") - listing = self._bounded_git(("ls-tree", "-r", "-z", "--full-tree", commit_oid), - limits.max_tree_bytes, budget, cancelled, cleanup_deadline) + listing = self._bounded_git( + ("ls-tree", "-r", "-z", "--full-tree", commit_oid), limits.max_tree_bytes, budget, cancelled, cleanup_deadline + ) records = listing.split(b"\0") if records[-1] != b"": raise SnapshotInputError("truncated_tree_metadata") @@ -223,9 +459,14 @@ class SnapshotAdapter: mode, kind, oid = metadata.split(b" ") path_hex = path.hex() components = path.split(b"/") - if (not path or path.startswith(b"/") or any(part in (b"", b".", b"..") for part in components) - or len(path) > limits.max_path_bytes or len(components) > limits.max_path_depth - or any(part.lower() == b".git" for part in components)): + if ( + not path + or path.startswith(b"/") + or any(part in (b"", b".", b"..") for part in components) + or len(path) > limits.max_path_bytes + or len(components) > limits.max_path_depth + or any(part.lower() == b".git" for part in components) + ): boundaries.append(SnapshotBoundary(path_hex or None, "unsupported_path")) continue if mode == b"120000": @@ -256,27 +497,33 @@ class SnapshotAdapter: with target.open("xb") as output: output.write(data) target.chmod(0o555 if mode == b"100755" else 0o444) - files.append(SnapshotFile(path_hex, blob_oid, mode.decode("ascii"), size, - hashlib.sha256(data).hexdigest())) + files.append(SnapshotFile(path_hex, blob_oid, mode.decode("ascii"), size, hashlib.sha256(data).hexdigest())) except SnapshotInputError as error: boundaries.append(SnapshotBoundary(path_hex, str(error))) if str(error) in ("deadline_exhausted", "cancelled", "git_output_limit", "git_cleanup_deadline_exhausted"): break except (SnapshotInputError, OSError, ValueError) as error: boundaries.append(SnapshotBoundary(None, str(error) or type(error).__name__)) - yield MaterializedSnapshot(root, SnapshotManifest(commit_oid, self.object_format, tuple(files), - tuple(boundaries), tree_complete, total)) + yield MaterializedSnapshot( + root, SnapshotManifest(commit_oid, self.object_format, tuple(files), tuple(boundaries), tree_complete, total) + ) finally: _remove_scratch(root, _cleanup_deadline(budget, cleanup_deadline)) - def _bounded_git(self, args: tuple[str, ...], limit: int, budget: ExecutionBudget, - cancelled: Callable[[], bool] | None, cleanup_deadline: list[float]) -> bytes: + def _bounded_git( + self, + args: tuple[str, ...], + limit: int, + budget: ExecutionBudget, + cancelled: Callable[[], bool] | None, + cleanup_deadline: list[float], + ) -> bytes: _check_budget(budget, cancelled) - command = ["git", "-c", "protocol.allow=never", "-c", "core.hooksPath=" + os.devnull, - "-C", str(self.repository), *args] + command = ["git", "-c", "protocol.allow=never", "-c", "core.hooksPath=" + os.devnull, "-C", str(self.repository), *args] try: - process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, - stdin=subprocess.DEVNULL, env=self._env, start_new_session=True) + process = subprocess.Popen( + command, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, stdin=subprocess.DEVNULL, env=self._env, start_new_session=True + ) except OSError as error: raise SnapshotInputError("git_launch_failed") from error output = bytearray() @@ -322,6 +569,8 @@ class SnapshotLimits: max_tree_bytes: int = 4 * 1024 * 1024 max_path_bytes: int = 4095 max_path_depth: int = 64 + max_diff_bytes: int = 8 * 1024 * 1024 + max_mapping_lines: int = 100000 def __post_init__(self) -> None: for value in vars(self).values(): @@ -364,6 +613,7 @@ def _remove_scratch(root: Path, deadline: float) -> None: os.rmdir(name, dir_fd=descriptor) else: os.unlink(name, dir_fd=descriptor) + if root.is_symlink(): root.unlink() return @@ -373,3 +623,46 @@ def _remove_scratch(root: Path, deadline: float) -> None: finally: os.close(descriptor) root.rmdir() + + +def _diff_records(raw: bytes, max_files: int) -> list[tuple[str, str, str, str, str, bytes | None, bytes | None]]: + """Parse only NUL-framed raw metadata; quoted patch filenames are never parsed.""" + if raw and not raw.endswith(b"\0"): + raise SnapshotInputError("truncated_diff_metadata") + fields = iter(raw.split(b"\0")[:-1]) + result = [] + try: + for metadata in fields: + old_mode, new_mode, old_oid, new_oid, status = metadata.decode("ascii").split(" ") + if not old_mode.startswith(":") or status[0] not in "ADMRT": + raise SnapshotInputError("unsupported_diff_metadata") + first = next(fields) + if not first: + raise SnapshotInputError("empty_diff_path") + second = next(fields) if status.startswith("R") else first + if not second: + raise SnapshotInputError("empty_diff_path") + result.append( + (old_mode[1:], new_mode, old_oid, new_oid, status[0], None if status == "A" else first, None if status == "D" else second) + ) + if len(result) > max_files: + raise SnapshotInputError("file_count_limit") + except (ValueError, StopIteration, IndexError) as error: + raise SnapshotInputError("invalid_diff_metadata") from error + return result + + +def _source_lines(data: bytes) -> list[bytes]: + if not data: + return [] + # Keep LF in each record so a final-newline change cannot look unchanged. + lines = data.split(b"\n") + return [line + b"\n" for line in lines[:-1]] + ([lines[-1]] if lines[-1] else []) + + +def _equal_line_pairs( + old: list[bytes], new: list[bytes], old_start: int, old_end: int, new_start: int, new_end: int +) -> list[tuple[int, int]]: + if old[old_start:old_end] != new[new_start:new_end]: + raise SnapshotInputError("diff_line_identity_mismatch") + return [(old_start + offset, new_start + offset) for offset in range(old_end - old_start)] diff --git a/src/openultrasast/push_scoring.py b/src/openultrasast/push_scoring.py index 49ba508..69da13d 100644 --- a/src/openultrasast/push_scoring.py +++ b/src/openultrasast/push_scoring.py @@ -122,7 +122,10 @@ def _case( and record.get("coverage") == "complete" ) negative = any( - q.get("status") == "answered" and q.get("outcome") == "negative" and q.get("target") == target and isinstance(q.get("witness"), str) + q.get("status") == "answered" + and q.get("outcome") == "negative" + and q.get("target") == target + and isinstance(q.get("witness"), str) and bool(q["witness"].strip()) for q in queries.values() ) diff --git a/tests/test_push_contracts.py b/tests/test_push_contracts.py index 81ef9aa..a1e9886 100644 --- a/tests/test_push_contracts.py +++ b/tests/test_push_contracts.py @@ -1,4 +1,5 @@ """Approved task 1.4 contracts: persist identities without inventing completed analysis.""" + import ast import json from dataclasses import FrozenInstanceError, replace @@ -35,9 +36,11 @@ def comparison() -> PushComparison: def test_config_round_trip_and_ordinary_scan_compatibility(tmp_path: Path) -> None: baseline = load_config() path = tmp_path / "settings.toml" - path.write_text('[push]\ncomparison_base = "refs/heads/main"\ndeadline_seconds = 11.5\n' - 'cancellation_allowance_seconds = 1\ncache_max_bytes = 4096\nmode = "blocking"\n' - 'incomplete_coverage_policy = "block"\n') + path.write_text( + '[push]\ncomparison_base = "refs/heads/main"\ndeadline_seconds = 11.5\n' + 'cancellation_allowance_seconds = 1\ncache_max_bytes = 4096\nmode = "blocking"\n' + 'incomplete_coverage_policy = "block"\n' + ) configured = load_config(path) assert configured.push == PushConfig("refs/heads/main", 11.5, 1, 4096, "blocking", "block") assert PushConfig.from_payload(json.loads(json.dumps(config_payload(configured)["push"]))) == configured.push @@ -48,19 +51,32 @@ def test_config_round_trip_and_ordinary_scan_compatibility(tmp_path: Path) -> No assert baseline.push == PushConfig() assert baseline.push.deadline_seconds == 30 assert baseline.push.cancellation_allowance_seconds == 2 - assert baseline.push.cache_max_bytes == 2 * 1024 ** 3 + assert baseline.push.cache_max_bytes == 2 * 1024**3 assert baseline.push.mode == "advisory" assert baseline.push.incomplete_coverage_policy == "allow" -@pytest.mark.parametrize("setting", [ - 'push = "bad"', 'push = []', '[push]\ndeadline_seconds = true', '[push]\ndeadline_seconds = "30"', - '[push]\ndeadline_seconds = nan', '[push]\ndeadline_seconds = inf', '[push]\ndeadline_seconds = 0', - '[push]\ncancellation_allowance_seconds = -1', '[push]\ncache_max_bytes = 0', - '[push]\ncache_max_bytes = 1.5', '[push]\ncache_max_bytes = true', '[push]\nmode = "nag"', - '[push]\nincomplete_coverage_policy = "ignore"', '[push]\ncomparison_base = ""', - '[push]\ncomparison_base = 1', '[push]\ndeadline_second = 60', -]) +@pytest.mark.parametrize( + "setting", + [ + 'push = "bad"', + "push = []", + "[push]\ndeadline_seconds = true", + '[push]\ndeadline_seconds = "30"', + "[push]\ndeadline_seconds = nan", + "[push]\ndeadline_seconds = inf", + "[push]\ndeadline_seconds = 0", + "[push]\ncancellation_allowance_seconds = -1", + "[push]\ncache_max_bytes = 0", + "[push]\ncache_max_bytes = 1.5", + "[push]\ncache_max_bytes = true", + '[push]\nmode = "nag"', + '[push]\nincomplete_coverage_policy = "ignore"', + '[push]\ncomparison_base = ""', + "[push]\ncomparison_base = 1", + "[push]\ndeadline_second = 60", + ], +) def test_push_configuration_rejects_unsafe_coercion(tmp_path: Path, setting: str) -> None: path = tmp_path / "settings.toml" path.write_text(setting) @@ -70,8 +86,17 @@ def test_push_configuration_rejects_unsafe_coercion(tmp_path: Path, setting: str def test_contract_round_trips_preserve_partition_ref_and_comparison_identity() -> None: update = PushUpdate("refs/heads/topic", "a" * 64, "refs/heads/main", "b" * 64) - context = ChangeContext("b" * 64, "a" * 64, ("src/api.js",), ("src/guard.js",), - (ChangedSpan("src/guard.js", 3, 8, "base"),), (), ("package.json",), (), ("dynamic dispatch",)) + context = ChangeContext( + "b" * 64, + "a" * 64, + ("src/api.js",), + ("src/guard.js",), + (ChangedSpan("src/guard.js", 3, 8, "base"),), + (), + ("package.json",), + (), + ("dynamic dispatch",), + ) for value in (update, comparison(), context, scope()): assert type(value).from_payload(json.loads(json.dumps(value.to_payload()))) == value assert question() != question("php", "src/api.js") @@ -82,15 +107,21 @@ def test_contract_round_trips_preserve_partition_ref_and_comparison_identity() - update.local_oid = "other" # type: ignore[misc] -@pytest.mark.parametrize("finding,coverage,disposition", [ - ("none", "incomplete", "allow"), ("none", "unavailable", "block"), - ("actionable", "incomplete", "allow"), ("actionable", "complete_within_scope", "block"), -]) +@pytest.mark.parametrize( + "finding,coverage,disposition", + [ + ("none", "incomplete", "allow"), + ("none", "unavailable", "block"), + ("actionable", "incomplete", "allow"), + ("actionable", "complete_within_scope", "block"), + ], +) def test_result_statuses_are_independent(finding: str, coverage: str, disposition: str) -> None: payload = { "analyses": [ComparisonAnalysis(comparison(), (scope(),), (question(),), coverage).to_payload()], "finding_status": finding, - "coverage_status": coverage, "push_disposition": disposition, + "coverage_status": coverage, + "push_disposition": disposition, "actionable_defect_ids": ["defect-1"] if finding == "actionable" else [], } result = PushResult.from_payload(payload) @@ -104,8 +135,11 @@ def test_incomplete_scope_cannot_be_serialized_as_complete_negative() -> None: result = ComparisonAnalysis(comparison(), (scope(),), (question(),), "complete_within_scope") assert ComparisonAnalysis.from_payload(result.to_payload()) == result deferred = DeferredQuestion(question("php", "index.php"), "deadline", ("unanswered",)) - for changed in (replace(scope(), deferred=(deferred,)), replace(scope(), unresolved_boundaries=("vendor call",)), - replace(scope(), population_complete=False)): + for changed in ( + replace(scope(), deferred=(deferred,)), + replace(scope(), unresolved_boundaries=("vendor call",)), + replace(scope(), population_complete=False), + ): with pytest.raises(ValueError): replace(result, scopes=(changed,)) with pytest.raises(ValueError): @@ -118,19 +152,32 @@ def test_incomplete_scope_cannot_be_serialized_as_complete_negative() -> None: replace(scope(), deferred=(DeferredQuestion(question(), "deadline", ()),)) -@pytest.mark.parametrize("value", [ - {"local_ref": "x", "local_oid": "a", "remote_ref": "y", "remote_oid": "b", "extra": 1}, - {"local_ref": "x", "local_oid": 1, "remote_ref": "y", "remote_oid": "b"}, - {"local_ref": "x", "local_oid": "a", "remote_ref": "y"}, -]) +@pytest.mark.parametrize( + "value", + [ + {"local_ref": "x", "local_oid": "a", "remote_ref": "y", "remote_oid": "b", "extra": 1}, + {"local_ref": "x", "local_oid": 1, "remote_ref": "y", "remote_oid": "b"}, + {"local_ref": "x", "local_oid": "a", "remote_ref": "y"}, + ], +) def test_malformed_identity_payloads_fail(value: dict[str, object]) -> None: with pytest.raises(ValueError): PushUpdate.from_payload(value) def test_graph_provenance_round_trip_and_live_lease_separation() -> None: - identity = GraphIdentity("source-digest", "declarations", "exclusions", "api", "javascript", - "4.0.625", "jssrc2cpg", "4.0.625", ("dataflowOss-v1",), (("option", "value"),)) + identity = GraphIdentity( + "source-digest", + "declarations", + "exclusions", + "api", + "javascript", + "4.0.625", + "jssrc2cpg", + "4.0.625", + ("dataflowOss-v1",), + (("option", "value"),), + ) artifact = GraphArtifact(identity, "graph-sha256", 128, GraphCensus(1, 2, 3), "complete", ()) assert GraphArtifact.from_payload(json.loads(json.dumps(artifact.to_payload()))) == artifact assert replace(identity, exclusions_digest="changed") != identity @@ -189,10 +236,16 @@ def test_deletion_only_result_is_explicitly_not_applicable() -> None: replace(result, coverage_status="complete_within_scope") -@pytest.mark.parametrize("field,value", [ - ("finding_status", "clean"), ("coverage_status", "secure"), ("push_disposition", "warn"), - ("finding_status", True), ("coverage_status", None), -]) +@pytest.mark.parametrize( + "field,value", + [ + ("finding_status", "clean"), + ("coverage_status", "secure"), + ("push_disposition", "warn"), + ("finding_status", True), + ("coverage_status", None), + ], +) def test_result_rejects_unknown_status_vocabulary(field: str, value: object) -> None: payload = PushResult((), "none", "not_applicable", "allow", ()).to_payload() payload[field] = value @@ -201,8 +254,19 @@ def test_result_rejects_unknown_status_vocabulary(field: str, value: object) -> def test_graph_partial_provenance_stays_partial_and_options_are_preserved() -> None: - identity = GraphIdentity("sources", "decls", "excludes", "core", "c", "4.0.625", "c2cpg", "4.0.625", - ("dataflowOss-v1",), (("depth", "2"),), (("include", "src"),)) + identity = GraphIdentity( + "sources", + "decls", + "excludes", + "core", + "c", + "4.0.625", + "c2cpg", + "4.0.625", + ("dataflowOss-v1",), + (("depth", "2"),), + (("include", "src"),), + ) artifact = GraphArtifact(identity, "digest", 0, GraphCensus(0, 0, 0), "partial", ("source unreadable",)) restored = GraphArtifact.from_payload(json.loads(json.dumps(artifact.to_payload()))) assert restored == artifact diff --git a/tests/test_push_snapshot.py b/tests/test_push_snapshot.py index 72d5983..23cf982 100644 --- a/tests/test_push_snapshot.py +++ b/tests/test_push_snapshot.py @@ -1,4 +1,5 @@ """Exercise the snapshot resolver with real local Git objects, not mocked answers.""" + import subprocess from pathlib import Path @@ -41,7 +42,9 @@ def test_preserves_every_ref_and_distinct_comparisons(repo: Path) -> None: result = SnapshotAdapter(repo).resolve_updates(line(head, base) + line(head, other, "force") + line(head, base, "alias")) assert result.targets == (head,) assert [(c.head_oid, c.base_oid, c.refs) for c in result.comparisons] == [ - (head, base, ("refs/heads/main", "refs/heads/alias")), (head, other, ("refs/heads/force",))] + (head, base, ("refs/heads/main", "refs/heads/alias")), + (head, other, ("refs/heads/force",)), + ] assert [r.disposition for r in result.updates] == ["ready"] * 3 assert len(result.updates) == 3 @@ -119,9 +122,16 @@ def test_missing_and_unsupported_objects_are_explicit(repo: Path) -> None: def test_bad_protocol_never_silently_skips_lines(repo: Path) -> None: from openultrasast.push.snapshot import SnapshotInputError + head = commit(repo, "head") - for invalid in ("\n", "only three fields\n", line(head[:-1], head), line(head, head, "bad..ref"), - line("0" * len(head), head), f"(delete) {head} refs/heads/main {head}\n"): + for invalid in ( + "\n", + "only three fields\n", + line(head[:-1], head), + line(head, head, "bad..ref"), + line("0" * len(head), head), + f"(delete) {head} refs/heads/main {head}\n", + ): with pytest.raises(SnapshotInputError): SnapshotAdapter(repo).resolve_updates(line(head, head) + invalid) @@ -234,25 +244,35 @@ def binary_commit(repo: Path, entries: list[tuple[bytes, bytes, bytes]]) -> str: oid = data kind = b"commit" else: - oid = subprocess.run(["git", "-C", str(repo), "hash-object", "-w", "--stdin"], - input=data, capture_output=True, check=True).stdout.rstrip(b"\n") + oid = subprocess.run( + ["git", "-C", str(repo), "hash-object", "-w", "--stdin"], input=data, capture_output=True, check=True + ).stdout.rstrip(b"\n") kind = b"blob" assert int(git(repo, "cat-file", "-s", oid.decode())) == len(data) records.append(mode + b" " + kind + b" " + oid + b"\t" + path + b"\0") - tree = subprocess.run(["git", "-C", str(repo), "mktree", "-z"], input=b"".join(records), - capture_output=True, check=True).stdout.decode().strip() + tree = ( + subprocess.run(["git", "-C", str(repo), "mktree", "-z"], input=b"".join(records), capture_output=True, check=True) + .stdout.decode() + .strip() + ) return git(repo, "commit-tree", tree, input="binary fixture") def test_snapshot_exact_binary_paths_and_no_filters(repo: Path) -> None: import os - entries = [(b"100644", path, data) for path, data in [ - (b"handler.php", b"res.send(req.body.x));\n"), - (b"newline\nname\t.js", b"\x00\xffbinary\r\n"), ("\u03b1.js".encode(), b"unicode"), - (b"\xff.js", b"invalid UTF8 name"), (b" ", b"space"), - (b".gitattributes", b"* filter=trap\n"), - ]] + + entries = [ + (b"100644", path, data) + for path, data in [ + (b"handler.php", b"res.send(req.body.x));\n"), + (b"newline\nname\t.js", b"\x00\xffbinary\r\n"), + ("\u03b1.js".encode(), b"unicode"), + (b"\xff.js", b"invalid UTF8 name"), + (b" ", b"space"), + (b".gitattributes", b"* filter=trap\n"), + ] + ] oid = binary_commit(repo, entries) sentinel = repo / "FILTER_RAN" git(repo, "config", "filter.trap.smudge", "touch " + str(sentinel)) @@ -268,24 +288,38 @@ def test_snapshot_exact_binary_paths_and_no_filters(repo: Path) -> None: def test_snapshot_omits_symlinks_gitlinks_and_lfs(repo: Path) -> None: target = commit(repo, "submodule") - oid = binary_commit(repo, [ - (b"120000", b"outside", b"/etc/passwd"), - (b"160000", b"submodule", target.encode()), - (b"100644", b"large.js", b"version https://git-lfs.github.com/spec/v1\noid sha256:" + b"0" * 64 + b"\nsize 3\n"), - ]) + oid = binary_commit( + repo, + [ + (b"120000", b"outside", b"/etc/passwd"), + (b"160000", b"submodule", target.encode()), + (b"100644", b"large.js", b"version https://git-lfs.github.com/spec/v1\noid sha256:" + b"0" * 64 + b"\nsize 3\n"), + ], + ) with SnapshotAdapter(repo).materialize(oid) as snapshot: assert not snapshot.manifest.complete assert snapshot.manifest.tree_complete assert {b.reason for b in snapshot.manifest.boundaries} == { - "symlink_not_materialized", "gitlink_not_materialized", "lfs_content_unavailable"} + "symlink_not_materialized", + "gitlink_not_materialized", + "lfs_content_unavailable", + } assert list(snapshot.root.iterdir()) == [] -@pytest.mark.parametrize("limit,reason", [("max_files", "file_count_limit"), - ("max_tree_bytes", "git_output_limit"), ("max_blob_bytes", "source_byte_limit"), - ("max_total_bytes", "source_byte_limit"), ("max_path_bytes", "unsupported_path")]) +@pytest.mark.parametrize( + "limit,reason", + [ + ("max_files", "file_count_limit"), + ("max_tree_bytes", "git_output_limit"), + ("max_blob_bytes", "source_byte_limit"), + ("max_total_bytes", "source_byte_limit"), + ("max_path_bytes", "unsupported_path"), + ], +) def test_snapshot_limits_are_explicit(repo: Path, limit: str, reason: str) -> None: from openultrasast.push.snapshot import SnapshotLimits + oid = binary_commit(repo, [(b"100644", b"one.js", b"123"), (b"100644", b"two.php", b"456")]) with SnapshotAdapter(repo).materialize(oid, limits=SnapshotLimits(**{limit: 1})) as snapshot: root = snapshot.root @@ -301,6 +335,7 @@ def test_snapshot_failure_preserves_live_state_and_cleans(repo: Path, failure: s from openultrasast.model.contracts import ExecutionBudget from openultrasast.push.snapshot import SnapshotInputError + oid = commit(repo, "object bytes") git(repo, "update-ref", "HEAD", oid) (repo / "file.js").write_bytes(b"staged") @@ -313,10 +348,12 @@ def test_snapshot_failure_preserves_live_state_and_cleans(repo: Path, failure: s (repo / ".git/objects" / blob[:2] / blob[2:]).unlink() before = {p.relative_to(repo): p.read_bytes() for p in repo.rglob("*") if p.is_file()} calls = 0 + def cancel() -> bool: nonlocal calls calls += 1 return failure == "cancel" and calls >= 5 + budget = ExecutionBudget(time.monotonic() + (-1 if failure == "deadline" else 10), 2.0) root = None try: @@ -337,6 +374,7 @@ def test_snapshot_failure_preserves_live_state_and_cleans(repo: Path, failure: s def test_snapshot_rejects_live_scratch_parent(repo: Path) -> None: from openultrasast.push.snapshot import SnapshotInputError + oid = commit(repo, "input") for parent in (repo, repo / ".git", repo / ".git/objects"): with pytest.raises(SnapshotInputError, match="outside"), SnapshotAdapter(repo).materialize(oid, scratch_parent=parent): @@ -359,13 +397,16 @@ def test_snapshot_missing_tree_and_unsupported_object_are_not_clean(repo: Path) def test_snapshot_refuses_oversize_blob_before_reading_it(repo: Path, monkeypatch: pytest.MonkeyPatch) -> None: from openultrasast.push.snapshot import SnapshotLimits + oid = commit(repo, "large payload") adapter = SnapshotAdapter(repo) original = adapter._bounded_git commands = [] + def capture(args, *rest): commands.append(args) return original(args, *rest) + monkeypatch.setattr(adapter, "_bounded_git", capture) with adapter.materialize(oid, limits=SnapshotLimits(max_blob_bytes=1)) as snapshot: assert not snapshot.manifest.complete @@ -392,14 +433,17 @@ def test_snapshot_deadline_kills_slow_git_and_cleans(repo: Path, monkeypatch: py import time from openultrasast.model.contracts import ExecutionBudget + oid = commit(repo, "input") adapter = SnapshotAdapter(repo) real_popen = subprocess.Popen processes = [] + def slow_git(command, **kwargs): process = real_popen([sys.executable, "-c", "import time; time.sleep(30)"], **kwargs) processes.append(process) return process + monkeypatch.setattr(subprocess, "Popen", slow_git) start = time.monotonic() with adapter.materialize(oid, budget=ExecutionBudget(start + 0.1, 1.0)) as snapshot: @@ -412,6 +456,7 @@ def test_snapshot_deadline_kills_slow_git_and_cleans(repo: Path, monkeypatch: py def test_snapshot_manifest_rejects_forged_census(repo: Path) -> None: from dataclasses import replace + oid = commit(repo, "input") with SnapshotAdapter(repo).materialize(oid) as snapshot: manifest = snapshot.manifest @@ -426,6 +471,7 @@ def test_snapshot_manifest_rejects_forged_census(repo: Path) -> None: def test_snapshot_subdirectory_adapter_cannot_scratch_in_live_parent(repo: Path) -> None: from openultrasast.push.snapshot import SnapshotInputError + oid = commit(repo, "input") nested = repo / "nested" nested.mkdir() @@ -436,12 +482,178 @@ def test_snapshot_subdirectory_adapter_cannot_scratch_in_live_parent(repo: Path) def test_snapshot_write_failure_cleans_partial_scratch(repo: Path, monkeypatch: pytest.MonkeyPatch) -> None: oid = commit(repo, "input") original = Path.open + def broken(path, mode="r", *args, **kwargs): if mode == "xb": raise OSError("scratch_write_failed") return original(path, mode, *args, **kwargs) + monkeypatch.setattr(Path, "open", broken) with SnapshotAdapter(repo).materialize(oid) as snapshot: assert not snapshot.manifest.complete assert any(b.reason == "scratch_write_failed" for b in snapshot.manifest.boundaries) assert not snapshot.root.exists() + + +def comparison(base: str | None, head: str): + from openultrasast.push.contracts import PushComparison + + return PushComparison(head, base, "supplied_remote_tip", ("refs/heads/main",)) + + +@pytest.mark.parametrize( + "extension,prefix,guard,sink", + [ + (b"php", b" None: + path = b"handler." + extension + old = prefix + b"function oldHandler() {\n" + guard + sink + b"}\n" + new = prefix + b"function newHandler() {\n" + sink + b"}\n" + base = binary_commit(repo, [(b"100644", path, old)]) + head = binary_commit(repo, [(b"100644", path, new)]) + result = SnapshotAdapter(repo).compare(comparison(base, head), declaration_paths=()) + assert result.base_revision == base and result.head_revision == head + assert result.path_encoding == "filesystem-bytes-hex" + assert result.changed_paths == (path.hex(),) + base_guard = old.split(b"\n").index(guard.rstrip(b"\n")) + 1 + assert any(s.side == "base" and s.start_line <= base_guard <= s.end_line for s in result.spans) + base_sink = old.split(b"\n").index(sink.rstrip(b"\n")) + 1 + head_sink = new.split(b"\n").index(sink.rstrip(b"\n")) + 1 + assert any( + m.base_start_line <= base_sink <= m.base_end_line and m.head_start_line + base_sink - m.base_start_line == head_sink + for m in result.line_correspondences + ) + assert not any(s.side == "head" and s.start_line <= head_sink <= s.end_line for s in result.spans) + assert result.relationships == () # Snapshot evidence cannot invent semantic call/guard edges. + assert type(result).from_payload(result.to_payload()) == result + + +def test_change_context_nul_paths_renames_deletions_and_declarations(repo: Path) -> None: + old_path, new_path = b"old\n\xff.js", b"new\t\xfe.js" + payload = b"function unchanged() {\n uniqueOperation();\n}\n" + base = binary_commit( + repo, [(b"100644", old_path, payload), (b"100644", b" ", b"deleted\n"), (b"100644", b"package.json", b'{"mode":"old"}\n')] + ) + head = binary_commit(repo, [(b"100644", new_path, payload), (b"100644", b"package.json", b'{"mode":"new"}\n')]) + result = SnapshotAdapter(repo).compare(comparison(base, head), declaration_paths=(b"package.json",)) + assert {(r.base_path, r.head_path) for r in result.renames} == {(old_path.hex(), new_path.hex())} + assert result.deleted_paths == (b" ".hex(),) + assert result.declaration_paths == (b"package.json".hex(),) + assert any(m.base_path == old_path.hex() and m.head_path == new_path.hex() for m in result.line_correspondences) + assert any(s.path == b" ".hex() and s.side == "base" for s in result.spans) + + +def test_change_context_unknown_base_binary_and_ambiguous_lines_are_unresolved(repo: Path) -> None: + base = binary_commit(repo, [(b"100644", b"api.js", b"old();\nsink();\nsink();\n"), (b"100644", b"binary", b"\x00old")]) + head = binary_commit(repo, [(b"100644", b"api.js", b"new();\nsink();\nsink();\n"), (b"100644", b"binary", b"\x00new")]) + missing = SnapshotAdapter(repo).compare(comparison(None, head)) + assert "comparison_base_unavailable" in missing.unresolved_boundaries + result = SnapshotAdapter(repo).compare(comparison(base, head)) + assert "declaration_paths_unavailable" in result.unresolved_boundaries + assert any("ambiguous_line_correspondence" in b for b in result.unresolved_boundaries) + assert any("binary_line_correspondence_unavailable" in b for b in result.unresolved_boundaries) + assert not result.line_correspondences + + +def test_change_context_is_bounded_and_never_executes_diff_helpers(repo: Path) -> None: + from openultrasast.push.snapshot import SnapshotLimits + + marker = repo / "DIFF_RAN" + git(repo, "config", "diff.external", "touch " + str(marker)) + git(repo, "config", "diff.trap.textconv", "touch " + str(marker)) + (repo / ".gitattributes").write_text("* diff=trap\n") + base, head = commit(repo, "old\n"), commit(repo, "new\n") + result = SnapshotAdapter(repo).compare(comparison(base, head), declaration_paths=()) + assert result.spans + assert not marker.exists() + limited = SnapshotAdapter(repo).compare(comparison(base, head), limits=SnapshotLimits(max_blob_bytes=1)) + assert any("source_byte_limit" in b for b in limited.unresolved_boundaries) + cancelled = SnapshotAdapter(repo).compare(comparison(base, head), cancelled=lambda: True) + assert "cancelled" in cancelled.unresolved_boundaries + + +def test_change_context_moved_repeated_operations_do_not_claim_identity(repo: Path) -> None: + base = binary_commit(repo, [(b"100644", b"api.js", b"function a() {\n sink();\n}\nfunction b() {\n sink();\n}\n")]) + head = binary_commit(repo, [(b"100644", b"api.js", b"function b() {\n sink();\n}\nfunction renamed() {\n sink();\n}\n")]) + result = SnapshotAdapter(repo).compare(comparison(base, head), declaration_paths=()) + assert any("ambiguous_line_correspondence" in b for b in result.unresolved_boundaries) + assert not any(m.base_start_line in (2, 5) for m in result.line_correspondences) + assert result.decode_path(result.changed_paths[0]) == "api.js" + + +@pytest.mark.parametrize("limit", ["max_total_bytes", "max_diff_bytes", "max_mapping_lines", "max_files"]) +def test_change_context_total_limits_are_explicit(repo: Path, limit: str) -> None: + from openultrasast.push.snapshot import SnapshotLimits + + base = binary_commit(repo, [(b"100644", b"a.js", b"old\n"), (b"100644", b"b.js", b"old2\n")]) + head = binary_commit(repo, [(b"100644", b"a.js", b"new\n"), (b"100644", b"b.js", b"new2\n")]) + result = SnapshotAdapter(repo).compare(comparison(base, head), declaration_paths=(), limits=SnapshotLimits(**{limit: 1})) + assert result.unresolved_boundaries + assert not result.line_correspondences + + +def test_change_context_missing_blob_and_invalid_revisions_are_not_clean(repo: Path) -> None: + base, head = commit(repo, "old\n"), commit(repo, "new\n") + blob = git(repo, "rev-parse", base + ":file.js") + (repo / ".git/objects" / blob[:2] / blob[2:]).unlink() + result = SnapshotAdapter(repo).compare(comparison(base, head), declaration_paths=()) + assert result.unresolved_boundaries + assert not result.spans and not result.line_correspondences + result = SnapshotAdapter(repo).compare(comparison("HEAD", head), declaration_paths=()) + assert result.unresolved_boundaries == ("comparison_requires_immutable_commit_oid",) + + +def test_change_context_lf_lines_and_final_newline_changes(repo: Path) -> None: + # CR, vertical tab and Unicode line separators are source bytes, not LF lines. + before = b"first\x0b\xe2\x80\xa8\rline\noperation();\n" + after = b"changed\x0b\xe2\x80\xa8\rline\noperation();\n" + base = binary_commit(repo, [(b"100644", b"api.js", before)]) + head = binary_commit(repo, [(b"100644", b"api.js", after)]) + result = SnapshotAdapter(repo).compare(comparison(base, head), declaration_paths=()) + assert [(s.start_line, s.end_line) for s in result.spans] == [(1, 1), (1, 1)] + assert [(m.base_start_line, m.head_start_line) for m in result.line_correspondences] == [(2, 2)] + no_newline = binary_commit(repo, [(b"100644", b"api.js", after[:-1])]) + result = SnapshotAdapter(repo).compare(comparison(head, no_newline), declaration_paths=()) + assert any(s.side == "head" and s.start_line == 2 for s in result.spans) + assert not any(m.head_start_line == 2 for m in result.line_correspondences) + + +def test_change_context_zero_changes_are_real_equal_objects(repo: Path) -> None: + oid = commit(repo, "source read by Git\n") + result = SnapshotAdapter(repo).compare(comparison(oid, oid), declaration_paths=()) + assert result.changed_paths == result.spans == result.unresolved_boundaries == () + + +def test_change_context_generic_hex_contract_validation(repo: Path) -> None: + import os + from dataclasses import replace + + oid = binary_commit(repo, [(b"100644", b" \xff", b"a\n")]) + new = binary_commit(repo, [(b"100644", b" \xff", b"b\n")]) + context = SnapshotAdapter(repo).compare(comparison(oid, new), declaration_paths=()) + assert os.fsencode(context.decode_path(context.changed_paths[0])) == b" \xff" + for invalid in ("zz", "00", "FF", "20 ff"): + with pytest.raises(ValueError): + replace(context, changed_paths=(invalid,)) + + +def test_change_context_unmatched_move_is_not_proven_new_behavior(repo: Path) -> None: + base = binary_commit(repo, [(b"100644", b"old.js", b"function old() {\n guard();\n sink();\n}\n")]) + head = binary_commit(repo, [(b"100644", b"new.js", b"function renamed() {\n sink();\n newCall();\n}\n")]) + result = SnapshotAdapter(repo).compare(comparison(base, head), declaration_paths=()) + assert not result.renames + assert "unmatched_added_deleted_path_correspondence" in result.unresolved_boundaries + assert {span.side for span in result.spans} == {"base", "head"} + + +def test_change_context_ambiguous_identical_file_renames_preserve_gap(repo: Path) -> None: + payload = b"function original() {\n operation();\n}\n" + base = binary_commit(repo, [(b"100644", b"a.js", payload), (b"100644", b"b.js", payload)]) + head = binary_commit(repo, [(b"100644", b"c.js", payload), (b"100644", b"d.js", payload)]) + result = SnapshotAdapter(repo).compare(comparison(base, head), declaration_paths=()) + assert result.renames # Git's attribution retained, with its ambiguity alongside. + assert not result.line_correspondences + assert any("ambiguous_path_rename_correspondence" in b for b in result.unresolved_boundaries)