mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
feat(pre-push-safety-net): replay immutable revisions through the full transaction
This commit is contained in:
@@ -0,0 +1,66 @@
|
|||||||
|
# Implementation verification: group 5
|
||||||
|
|
||||||
|
Group 5 was authorized by the maintainer's “approved” after group 4 completion.
|
||||||
|
Task 5.1 is verified complete (15/27 total); task 5.2 is next.
|
||||||
|
Real capability eligibility and rollout readiness remain unproven.
|
||||||
|
|
||||||
|
## Execution and review mode
|
||||||
|
|
||||||
|
Fresh implementer dispatch for 5.1 returned `agent thread limit reached`.
|
||||||
|
The [Kiro implementation skill](../../../.opencode/skills/kiro-impl/SKILL.md) explicitly
|
||||||
|
states: “If multi-agent is not available, fall back to manual mode execution for all tasks.”
|
||||||
|
Its manual review rule permits review in the main context when a fresh reviewer is unavailable.
|
||||||
|
This run therefore uses manual implementation and adversarial review, not independent review.
|
||||||
|
|
||||||
|
## Task 5.1 brief
|
||||||
|
|
||||||
|
Wire explicit immutable base/head replay through the existing mapper, ranker, driver,
|
||||||
|
base comparison and admission/report contracts. Preserve the shared elapsed deadline,
|
||||||
|
source/index/ref state, completed evidence on failure, and separate coverage/exit policy.
|
||||||
|
Requirements: 1.1, 2.1, 3.1, 3.4, 4.1, 4.2, 5.5, 7.1, 7.3. Design sections are named:
|
||||||
|
Snapshot Adapter, Ranker Scope Contract, Delta and Actionability Policy, Result and Hook
|
||||||
|
Adapter, Cache and Engine Boundary. No new detector, ranking weights, framework facts,
|
||||||
|
cache or hook installation belongs to this task.
|
||||||
|
|
||||||
|
### Integration findings
|
||||||
|
|
||||||
|
- Snapshot resolution previously used fresh per-command timeouts. An optional shared
|
||||||
|
execution budget now covers constructor probes and explicit revision resolution.
|
||||||
|
- Discovery and provenance are bounded in a Linux child process. The parent drains output
|
||||||
|
incrementally, checks the deadline, kills a stalled worker and requires a valid receipt;
|
||||||
|
exit zero with no result is failure. No project code or engine runs in this child.
|
||||||
|
- The first real controlled PHP replay found the exact request-to-eval witness but novelty
|
||||||
|
was unknown: comparison policy treated another question's context gap as global. The
|
||||||
|
owning task 4.1 policy was repaired here with a failing regression first. Only recognized
|
||||||
|
boundaries attached to a different recorded question can be ignored for that candidate;
|
||||||
|
aggregate coverage retains every gap. Own-question and global boundaries still prevent
|
||||||
|
supported novelty. This is policy ownership repair, not a new upstream graph abstraction.
|
||||||
|
- Artifacts must resolve outside the analyzed repository. The runner refuses a destination
|
||||||
|
that could overwrite live source/index/ref state and preserves the real enforcement result.
|
||||||
|
|
||||||
|
### Evidence in progress
|
||||||
|
|
||||||
|
Initial OFF-state tests: `/tmp/ousast-replay-red.log`, six failures before implementation.
|
||||||
|
Policy ownership regression: `/tmp/ousast-replay-policy-red.log`, one failure before repair.
|
||||||
|
Live-source overwrite regression: `/tmp/ousast-replay-review-red.log`, reproduced before repair.
|
||||||
|
Final validation: 1,236 tests passed, nine skipped; Ruff/format, mypy (112 source files), compileall and all three regression gates passed. Final packaged CLI boot and the controlled positive replay passed. See `benchmarks/measurements/2026-09-13-replay-smoke.json`.
|
||||||
|
|
||||||
|
|
||||||
|
Additional integration repairs retain snapshot omissions in engine change context and distinguish
|
||||||
|
base questions proven tier zero from unfinished queries. The exact required base question still
|
||||||
|
must have completed, valid evidence; budget-deferred work still prevents a complete comparison.
|
||||||
|
RED evidence: `/tmp/ousast-replay-tierzero-red.log` and `/tmp/ousast-replay-snapshot-red.log`.
|
||||||
|
|
||||||
|
## Task 5.1 review verdict
|
||||||
|
- VERDICT: APPROVED
|
||||||
|
- TASK: 5.1
|
||||||
|
- MECHANICAL_RESULTS: Full suite exit 0 (1,236 passed, 9 skipped); Ruff/format/mypy/compileall and detection/map/pair gates PASS; no new placeholder markers or secrets; boundary WITHIN; behavioral RED VERIFIED.
|
||||||
|
- FINDINGS: Integration defects above repaired with regressions; no outstanding task-blocking finding.
|
||||||
|
- SUMMARY: Manual adversarial review approves bounded local replay; this is not independent review or rollout admission.
|
||||||
|
|
||||||
|
## Task 5.1 verification
|
||||||
|
- STATUS: VERIFIED
|
||||||
|
- CLAIM_TYPE: TASK
|
||||||
|
- CLAIM: Explicit local replay traverses the production mapper/ranker/arbiter/comparison/report path and retains uncertainty and shared deadlines.
|
||||||
|
- EVIDENCE: Final packaged controlled security change yields new/source_connection_absent_from_comparable_base. Fixed twin emits no modeled head finding. The packaged deadline control returns incomplete in 30.785 seconds, within 30 + 2 seconds. Exact final package policy/core/facts/query digests match the workspace. Complete commands and captured raw answers live in the measurement artifact.
|
||||||
|
- GAPS: Aggregate coverage remains incomplete, real capabilities are disabled, and lab timings do not satisfy the latency gate. Source-mounted fixed and earlier packaged deadline provenance is explicitly separated in the artifact. PMPro/Node transfer is task 5.2, not inferred from a small PHP control.
|
||||||
@@ -1,7 +1,7 @@
|
|||||||
{
|
{
|
||||||
"feature_name": "pre-push-safety-net",
|
"feature_name": "pre-push-safety-net",
|
||||||
"created_at": "2026-09-12T15:35:00Z",
|
"created_at": "2026-09-12T15:35:00Z",
|
||||||
"updated_at": "2026-09-13T11:21:07.667199+00:00",
|
"updated_at": "2026-09-13T12:50:25.249104+00:00",
|
||||||
"language": "en",
|
"language": "en",
|
||||||
"phase": "implementation",
|
"phase": "implementation",
|
||||||
"approvals": {
|
"approvals": {
|
||||||
@@ -19,7 +19,7 @@
|
|||||||
}
|
}
|
||||||
},
|
},
|
||||||
"ready_for_implementation": true,
|
"ready_for_implementation": true,
|
||||||
"implementation_approval_note": "Groups 1\u20134 verified complete. Tasks 4.1/4.2 independently approved; 4.3 used the documented Kiro manual review fallback after agent thread limit. Remaining tasks require review and fresh verification.",
|
"implementation_approval_note": "Group 5 authorized. Task5.1 verified through manual Kiro fallback review; 5.2 next. Real capabilities remain disabled.",
|
||||||
"depends_on": [
|
"depends_on": [
|
||||||
"contributor-scan"
|
"contributor-scan"
|
||||||
],
|
],
|
||||||
@@ -60,6 +60,11 @@
|
|||||||
"at": "2026-09-13T10:26:47.307154+00:00",
|
"at": "2026-09-13T10:26:47.307154+00:00",
|
||||||
"source": "move on to group 4 as a whole",
|
"source": "move on to group 4 as a whole",
|
||||||
"action": "Authorized all of group 4 (4.1\u20134.3), sequential implementation, independent task review and fresh verification. Groups 5\u20138 remain pending."
|
"action": "Authorized all of group 4 (4.1\u20134.3), sequential implementation, independent task review and fresh verification. Groups 5\u20138 remain pending."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"at": "2026-09-13T12:40:57.961577+00:00",
|
||||||
|
"source": "approved",
|
||||||
|
"action": "Authorized group 5 after the group 4 completion summary: explicit transaction replay, then the first PMPro/frozen-core Node experiment. Review and fresh evidence remain required."
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"task_review": {
|
"task_review": {
|
||||||
@@ -86,15 +91,17 @@
|
|||||||
"3.4",
|
"3.4",
|
||||||
"4.1",
|
"4.1",
|
||||||
"4.2",
|
"4.2",
|
||||||
"4.3"
|
"4.3",
|
||||||
|
"5.1"
|
||||||
],
|
],
|
||||||
"remaining_tasks": 13,
|
"remaining_tasks": 12,
|
||||||
"verification_record": "implementation-group-4.md",
|
"verification_record": "implementation-group-5.md",
|
||||||
"verification_records": [
|
"verification_records": [
|
||||||
"implementation-group-1.md",
|
"implementation-group-1.md",
|
||||||
"implementation-group-2.md",
|
"implementation-group-2.md",
|
||||||
"implementation-group-3.md",
|
"implementation-group-3.md",
|
||||||
"implementation-group-4.md"
|
"implementation-group-4.md",
|
||||||
|
"implementation-group-5.md"
|
||||||
],
|
],
|
||||||
"next_group": 5,
|
"next_group": 5,
|
||||||
"blocked_tasks": []
|
"blocked_tasks": []
|
||||||
|
|||||||
@@ -3,7 +3,7 @@
|
|||||||
Status: task plan approved by the maintainer on 2026-09-12. Groups 1–4 (tasks 1.1–4.3)
|
Status: task plan approved by the maintainer on 2026-09-12. Groups 1–4 (tasks 1.1–4.3)
|
||||||
are complete after task reviews and fresh verification (14/27 subtasks). Task 4.3 used
|
are complete after task reviews and fresh verification (14/27 subtasks). Task 4.3 used
|
||||||
the documented manual review fallback after agent dispatch became unavailable.
|
the documented manual review fallback after agent dispatch became unavailable.
|
||||||
Groups 5–8 remain pending; no production capability or hook latency is admitted.
|
Task 5.1 is also verified (15/27 total); task 5.2 and groups 6–8 remain pending; no production capability or hook latency is admitted.
|
||||||
|
|
||||||
## Execution contract
|
## Execution contract
|
||||||
|
|
||||||
@@ -140,10 +140,11 @@ Groups 5–8 remain pending; no production capability or hook latency is admitte
|
|||||||
- _Requirements: 3.1, 3.2, 3.3, 5.1, 5.2, 5.5, 7.3_
|
- _Requirements: 3.1, 3.2, 3.3, 5.1, 5.2, 5.5, 7.3_
|
||||||
|
|
||||||
- [ ] 5. Prove the first end-to-end experiment before cache optimization
|
- [ ] 5. Prove the first end-to-end experiment before cache optimization
|
||||||
- [ ] 5.1 Wire explicit base/head replay through the full transaction
|
- [x] 5.1 Wire explicit base/head replay through the full transaction
|
||||||
- Connect immutable snapshots, ranker decisions, existing arbiter, targeted comparison, admission diagnostics and compact artifacts through an explicit local replay command.
|
- Connect immutable snapshots, ranker decisions, existing arbiter, targeted comparison, admission diagnostics and compact artifacts through an explicit local replay command.
|
||||||
- Apply the shared budget to preparation and both revisions; no model/service or installed hook is required. All runtime errors preserve coverage state.
|
- Apply the shared budget to preparation and both revisions; no model/service or installed hook is required. All runtime errors preserve coverage state.
|
||||||
- Done when a controlled security change and fixed twin traverse the production runner through the existing ranker and produce replayable results with truthful deadlines.
|
- Done when a controlled security change and fixed twin traverse the production runner through the existing ranker and produce replayable results with truthful deadlines.
|
||||||
|
- _Verified 2026-09-13:_ manual Kiro fallback review APPROVED; 1,236 tests passed, nine skipped; final packaged security replay and fixed/deadline controls verified. See `implementation-group-5.md`.
|
||||||
- _Boundary: Push runner, CLI, Snapshot Adapter, engine and policy integration_
|
- _Boundary: Push runner, CLI, Snapshot Adapter, engine and policy integration_
|
||||||
- _Depends: 1.3, 3.1, 3.4, 4.3_
|
- _Depends: 1.3, 3.1, 3.4, 4.3_
|
||||||
- _Requirements: 1.1, 2.1, 3.1, 3.4, 4.1, 4.2, 5.5, 7.1, 7.3_
|
- _Requirements: 1.1, 2.1, 3.1, 3.4, 4.1, 4.2, 5.5, 7.1, 7.3_
|
||||||
|
|||||||
@@ -1,10 +1,10 @@
|
|||||||
{
|
{
|
||||||
"counts": {
|
"counts": {
|
||||||
"load_bearing": 101,
|
"load_bearing": 102,
|
||||||
"standalone_capability": 9
|
"standalone_capability": 9
|
||||||
},
|
},
|
||||||
"modules": 110,
|
"modules": 111,
|
||||||
"note": "Refreshed after pre-push-safety-net task 4.3: compact reporting preserves full evidence and decisions; transaction replay remains task 5.1.",
|
"note": "Refreshed for task 5.1: explicit local replay integrates the existing ranker and comparison policy.",
|
||||||
"orphaned": [],
|
"orphaned": [],
|
||||||
"rows": [
|
"rows": [
|
||||||
{
|
{
|
||||||
@@ -104,6 +104,7 @@
|
|||||||
"model.endpoint",
|
"model.endpoint",
|
||||||
"pairs",
|
"pairs",
|
||||||
"push.policy",
|
"push.policy",
|
||||||
|
"push.runner",
|
||||||
"regress.candidate",
|
"regress.candidate",
|
||||||
"regress.recipes",
|
"regress.recipes",
|
||||||
"run"
|
"run"
|
||||||
@@ -135,7 +136,8 @@
|
|||||||
"model.pipeline",
|
"model.pipeline",
|
||||||
"model.scan",
|
"model.scan",
|
||||||
"model.taint",
|
"model.taint",
|
||||||
"push.policy"
|
"push.policy",
|
||||||
|
"push.runner"
|
||||||
],
|
],
|
||||||
"module": "cpg",
|
"module": "cpg",
|
||||||
"reason": "model-grounded-detection Req 4: the CPG seam package"
|
"reason": "model-grounded-detection Req 4: the CPG seam package"
|
||||||
@@ -157,7 +159,8 @@
|
|||||||
"model.pipeline",
|
"model.pipeline",
|
||||||
"model.scan",
|
"model.scan",
|
||||||
"model.taint",
|
"model.taint",
|
||||||
"push.policy"
|
"push.policy",
|
||||||
|
"push.runner"
|
||||||
],
|
],
|
||||||
"module": "cpg.backend",
|
"module": "cpg.backend",
|
||||||
"reason": "model-grounded-detection Req 4.1/4.5: the single Joern subprocess boundary"
|
"reason": "model-grounded-detection Req 4.1/4.5: the single Joern subprocess boundary"
|
||||||
@@ -327,6 +330,7 @@
|
|||||||
"model.scan",
|
"model.scan",
|
||||||
"pairs",
|
"pairs",
|
||||||
"preprocess",
|
"preprocess",
|
||||||
|
"push.runner",
|
||||||
"stage_processors"
|
"stage_processors"
|
||||||
],
|
],
|
||||||
"module": "mapping",
|
"module": "mapping",
|
||||||
@@ -362,6 +366,7 @@
|
|||||||
"push.contracts",
|
"push.contracts",
|
||||||
"push.policy",
|
"push.policy",
|
||||||
"push.report",
|
"push.report",
|
||||||
|
"push.runner",
|
||||||
"push.snapshot"
|
"push.snapshot"
|
||||||
],
|
],
|
||||||
"module": "model",
|
"module": "model",
|
||||||
@@ -407,6 +412,7 @@
|
|||||||
"push.contracts",
|
"push.contracts",
|
||||||
"push.policy",
|
"push.policy",
|
||||||
"push.report",
|
"push.report",
|
||||||
|
"push.runner",
|
||||||
"push.snapshot"
|
"push.snapshot"
|
||||||
],
|
],
|
||||||
"module": "model.contracts",
|
"module": "model.contracts",
|
||||||
@@ -497,7 +503,8 @@
|
|||||||
"cli",
|
"cli",
|
||||||
"model.layout",
|
"model.layout",
|
||||||
"model.scan",
|
"model.scan",
|
||||||
"push.policy"
|
"push.policy",
|
||||||
|
"push.runner"
|
||||||
],
|
],
|
||||||
"module": "model.regions",
|
"module": "model.regions",
|
||||||
"reason": "contributor-scan Req 2: regions from entry points, with the families each admits"
|
"reason": "contributor-scan Req 2: regions from entry points, with the families each admits"
|
||||||
@@ -515,7 +522,8 @@
|
|||||||
"importers": [
|
"importers": [
|
||||||
"cli",
|
"cli",
|
||||||
"push.policy",
|
"push.policy",
|
||||||
"push.report"
|
"push.report",
|
||||||
|
"push.runner"
|
||||||
],
|
],
|
||||||
"module": "model.scan",
|
"module": "model.scan",
|
||||||
"reason": "contributor-scan Req 3: the repository driver -- one CPG, one budget, ranked spend"
|
"reason": "contributor-scan Req 3: the repository driver -- one CPG, one budget, ranked spend"
|
||||||
@@ -523,7 +531,8 @@
|
|||||||
{
|
{
|
||||||
"classification": "load_bearing",
|
"classification": "load_bearing",
|
||||||
"importers": [
|
"importers": [
|
||||||
"cli"
|
"cli",
|
||||||
|
"push.runner"
|
||||||
],
|
],
|
||||||
"module": "model.shipped",
|
"module": "model.shipped",
|
||||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
|
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
|
||||||
@@ -630,6 +639,7 @@
|
|||||||
"model.partitions",
|
"model.partitions",
|
||||||
"model.scan",
|
"model.scan",
|
||||||
"pairs",
|
"pairs",
|
||||||
|
"push.runner",
|
||||||
"rank",
|
"rank",
|
||||||
"regress.candidate",
|
"regress.candidate",
|
||||||
"semantic.overlay",
|
"semantic.overlay",
|
||||||
@@ -668,8 +678,10 @@
|
|||||||
{
|
{
|
||||||
"classification": "load_bearing",
|
"classification": "load_bearing",
|
||||||
"importers": [
|
"importers": [
|
||||||
|
"cli",
|
||||||
"push.policy",
|
"push.policy",
|
||||||
"push.report",
|
"push.report",
|
||||||
|
"push.runner",
|
||||||
"push.snapshot"
|
"push.snapshot"
|
||||||
],
|
],
|
||||||
"module": "push",
|
"module": "push",
|
||||||
@@ -680,6 +692,7 @@
|
|||||||
"importers": [
|
"importers": [
|
||||||
"push.policy",
|
"push.policy",
|
||||||
"push.report",
|
"push.report",
|
||||||
|
"push.runner",
|
||||||
"push.snapshot"
|
"push.snapshot"
|
||||||
],
|
],
|
||||||
"module": "push.contracts",
|
"module": "push.contracts",
|
||||||
@@ -688,20 +701,33 @@
|
|||||||
{
|
{
|
||||||
"classification": "load_bearing",
|
"classification": "load_bearing",
|
||||||
"importers": [
|
"importers": [
|
||||||
"push.report"
|
"push.report",
|
||||||
|
"push.runner"
|
||||||
],
|
],
|
||||||
"module": "push.policy",
|
"module": "push.policy",
|
||||||
"reason": "pre-push-safety-net Req 2.4/3.1/3.4/4.1: targeted same-semantics base comparison; replay integration pending"
|
"reason": "pre-push-safety-net Req 2.4/3.1/3.4/4.1: targeted same-semantics base comparison; replay integration pending"
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"classification": "load_bearing",
|
"classification": "load_bearing",
|
||||||
"importers": [],
|
"importers": [
|
||||||
|
"push.runner"
|
||||||
|
],
|
||||||
"module": "push.report",
|
"module": "push.report",
|
||||||
"reason": "pre-push-safety-net Req 5.1/5.2/5.5/7.3: compact output and complete artifacts; runner integration pending"
|
"reason": "pre-push-safety-net Req 5.1/5.2/5.5/7.3: compact output and complete artifacts; runner integration pending"
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"classification": "load_bearing",
|
"classification": "load_bearing",
|
||||||
"importers": [],
|
"importers": [
|
||||||
|
"cli"
|
||||||
|
],
|
||||||
|
"module": "push.runner",
|
||||||
|
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"classification": "load_bearing",
|
||||||
|
"importers": [
|
||||||
|
"push.runner"
|
||||||
|
],
|
||||||
"module": "push.snapshot",
|
"module": "push.snapshot",
|
||||||
"reason": "pre-push-safety-net Req 1.1/1.3/1.4/7.1: local immutable ref resolution; runner integration pending"
|
"reason": "pre-push-safety-net Req 1.1/1.3/1.4/7.1: local immutable ref resolution; runner integration pending"
|
||||||
},
|
},
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
@@ -242,3 +242,27 @@ allowance; a stalled writer is killed. A hard cancellation can leave a private
|
|||||||
`.partial-` file, which is never the published artifact and cannot establish a
|
`.partial-` file, which is never the published artifact and cannot establish a
|
||||||
completed result. Serialization/write failure emits one notice without replacing
|
completed result. Serialization/write failure emits one notice without replacing
|
||||||
the finding, coverage or push decision.
|
the finding, coverage or push decision.
|
||||||
|
|
||||||
|
## Explicit local replay
|
||||||
|
|
||||||
|
Task 5.1 adds `ousast pre-push REPOSITORY --base BASE --head HEAD --artifact OUTSIDE_REPOSITORY.json`.
|
||||||
|
Both revisions must already exist locally. The command records their resolved commit identities,
|
||||||
|
materializes tracked object bytes in private scratch storage, discovers regions through the existing
|
||||||
|
mapper, and runs the existing evidence ranker, arbiter, targeted base comparison and admission policy.
|
||||||
|
It installs no hook and requires no model endpoint. All real capabilities remain experimental and
|
||||||
|
ineligible for normal alerts until task 8.3; diagnostic candidates remain in the JSON artifact.
|
||||||
|
|
||||||
|
`--deadline` defaults to 30 seconds, `--cancellation-allowance` to 2 seconds, and `--max-regions`
|
||||||
|
to 500. Preparation, both revisions and reporting share that elapsed budget. `--mode blocking`
|
||||||
|
and `--incomplete-coverage block` explicitly select enforcement; default advisory exit zero means
|
||||||
|
allow, never complete analysis. This initial replay interface requires the artifact outside the
|
||||||
|
analyzed repository, including resolved symlink destinations, to preserve the live source and index.
|
||||||
|
Artifact failure cannot change the actual allow/block decision.
|
||||||
|
|
||||||
|
`ops/smoke_replay.py --case security|fixed` exercises a real controlled PHP change through this
|
||||||
|
production runner, without supplying hand-selected regions. Its default 900-second budget is a
|
||||||
|
separately labeled runtime laboratory check, not hook latency. `--deadline 30` exercises cancellation;
|
||||||
|
it does not demand a positive after an interrupted query. Run with the network-disabled packaged
|
||||||
|
image, mounting the smoke script read-only as for the other smoke scripts. Snapshot manifests
|
||||||
|
record verified immutable source bytes; full scan records retain exact selected/deferred questions,
|
||||||
|
raw answers, per-question gaps, change context and candidate dispositions.
|
||||||
|
|||||||
@@ -0,0 +1,89 @@
|
|||||||
|
"""Real production replay on a controlled PHP security change and its fixed twin."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import subprocess
|
||||||
|
import tempfile
|
||||||
|
import time
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from openultrasast.config import PushConfig
|
||||||
|
from openultrasast.push.runner import replay
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument("--case", choices=("security", "fixed"), required=True)
|
||||||
|
parser.add_argument("--deadline", type=float, default=900)
|
||||||
|
args = parser.parse_args()
|
||||||
|
with tempfile.TemporaryDirectory(prefix="ousast-replay-smoke-") as temporary:
|
||||||
|
root = Path(temporary) / "repo"
|
||||||
|
root.mkdir()
|
||||||
|
|
||||||
|
def git(*parts: str) -> str:
|
||||||
|
return subprocess.run(["git", "-C", str(root), *parts], check=True, capture_output=True, timeout=10).stdout.decode().strip()
|
||||||
|
|
||||||
|
git("init")
|
||||||
|
git("config", "user.name", "Replay smoke")
|
||||||
|
git("config", "user.email", "[email protected]")
|
||||||
|
git("config", "core.hooksPath", os.devnull)
|
||||||
|
source = root / "api.php"
|
||||||
|
before = '<?php\nfunction handler() {\n $value = "fixed";\n eval($value);\n}\nhandler();\n'
|
||||||
|
after = before.replace('$value = "fixed";', '$value = $_GET["value"];')
|
||||||
|
source.write_text(before)
|
||||||
|
git("add", "api.php")
|
||||||
|
git("commit", "-m", "fixed")
|
||||||
|
fixed = git("rev-parse", "HEAD")
|
||||||
|
source.write_text(after)
|
||||||
|
git("add", "api.php")
|
||||||
|
git("commit", "-m", "security")
|
||||||
|
security = git("rev-parse", "HEAD")
|
||||||
|
base, head = (fixed, security) if args.case == "security" else (security, fixed)
|
||||||
|
# Verify readable immutable source before trusting any reported zeros.
|
||||||
|
readable = git("show", head + ":api.php")
|
||||||
|
assert len(readable.encode()) > 0
|
||||||
|
source.write_text("dirty live content")
|
||||||
|
index = (root / ".git/index").read_bytes()
|
||||||
|
status = git("status", "--porcelain")
|
||||||
|
artifact = Path(temporary) / "replay.json"
|
||||||
|
start = time.monotonic()
|
||||||
|
delivery = replay(root, base=base, head=head, artifact=artifact, config=PushConfig(deadline_seconds=args.deadline))
|
||||||
|
payload = json.loads(artifact.read_text()) if artifact.exists() else None
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"case": args.case,
|
||||||
|
"source_bytes": len(readable.encode()),
|
||||||
|
"elapsed_seconds": time.monotonic() - start,
|
||||||
|
"exit_code": delivery.exit_code,
|
||||||
|
"report_error": delivery.error,
|
||||||
|
"reporting_seconds": delivery.reporting_seconds,
|
||||||
|
"text": delivery.text,
|
||||||
|
"artifact": payload,
|
||||||
|
},
|
||||||
|
indent=2,
|
||||||
|
),
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
assert source.read_text() == "dirty live content"
|
||||||
|
assert (root / ".git/index").read_bytes() == index
|
||||||
|
assert git("status", "--porcelain") == status
|
||||||
|
assert payload and payload["scans"], "production runner produced no scan evidence"
|
||||||
|
assert delivery.result.finding_status == "none" and delivery.exit_code == 0
|
||||||
|
if args.deadline >= 900:
|
||||||
|
head_scan = next(r["scan"] for r in payload["scans"] if r["side"] == "head")
|
||||||
|
assert head_scan["scope"]["ranking_mode"] == "evidence"
|
||||||
|
if args.case == "security":
|
||||||
|
assert any(d["candidate"]["delta"]["novelty"] in ("new", "worsened") for d in payload["admission"]["dispositions"]), (
|
||||||
|
"supported security control missing: inspect upstream evidence before claiming replay complete"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
assert not head_scan["findings"], "fixed twin retained a modeled finding"
|
||||||
|
print(json.dumps({"verified": True, "case": args.case}), flush=True)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -110,6 +110,17 @@ def main(argv: list[str] | None = None) -> int:
|
|||||||
parser = argparse.ArgumentParser(prog="ousast")
|
parser = argparse.ArgumentParser(prog="ousast")
|
||||||
subparsers = parser.add_subparsers(dest="command", required=True)
|
subparsers = parser.add_subparsers(dest="command", required=True)
|
||||||
|
|
||||||
|
replay_parser = subparsers.add_parser("pre-push", help="replay explicit local base/head revisions (experimental)")
|
||||||
|
replay_parser.add_argument("path", type=Path)
|
||||||
|
replay_parser.add_argument("--base", required=True)
|
||||||
|
replay_parser.add_argument("--head", required=True)
|
||||||
|
replay_parser.add_argument("--artifact", type=Path, required=True)
|
||||||
|
replay_parser.add_argument("--deadline", type=float, default=30.0)
|
||||||
|
replay_parser.add_argument("--cancellation-allowance", type=float, default=2.0)
|
||||||
|
replay_parser.add_argument("--max-regions", type=int, default=500)
|
||||||
|
replay_parser.add_argument("--mode", choices=("advisory", "blocking"), default="advisory")
|
||||||
|
replay_parser.add_argument("--incomplete-coverage", choices=("allow", "block"), default="allow")
|
||||||
|
|
||||||
scan = subparsers.add_parser("scan", help="scan a local repository")
|
scan = subparsers.add_parser("scan", help="scan a local repository")
|
||||||
scan.add_argument("path", type=Path)
|
scan.add_argument("path", type=Path)
|
||||||
scan.add_argument("--mode", choices=("quick", "standard", "deep"), default="quick")
|
scan.add_argument("--mode", choices=("quick", "standard", "deep"), default="quick")
|
||||||
@@ -189,6 +200,24 @@ def main(argv: list[str] | None = None) -> int:
|
|||||||
candidates.add_argument("--json", action="store_true")
|
candidates.add_argument("--json", action="store_true")
|
||||||
|
|
||||||
args = parser.parse_args(argv)
|
args = parser.parse_args(argv)
|
||||||
|
if args.command == "pre-push":
|
||||||
|
from .config import PushConfig
|
||||||
|
from .push.runner import replay
|
||||||
|
|
||||||
|
try:
|
||||||
|
settings = PushConfig(
|
||||||
|
deadline_seconds=args.deadline,
|
||||||
|
cancellation_allowance_seconds=args.cancellation_allowance,
|
||||||
|
mode=args.mode,
|
||||||
|
incomplete_coverage_policy=args.incomplete_coverage,
|
||||||
|
)
|
||||||
|
delivery = replay(
|
||||||
|
args.path, base=args.base, head=args.head, artifact=args.artifact, config=settings, max_regions=args.max_regions
|
||||||
|
)
|
||||||
|
except ValueError as error:
|
||||||
|
parser.error(str(error))
|
||||||
|
print(delivery.text, end="")
|
||||||
|
return delivery.exit_code
|
||||||
if args.command == "scan":
|
if args.command == "scan":
|
||||||
return _scan(args.path, args.config, args.mode, args.fail_on)
|
return _scan(args.path, args.config, args.mode, args.fail_on)
|
||||||
if args.command == "index":
|
if args.command == "index":
|
||||||
|
|||||||
@@ -195,9 +195,25 @@ def _valid_answer(scan: ModelScanResult, question: QuestionIdentity, operations:
|
|||||||
return len(rows) == sum(op.question == question for op in operations)
|
return len(rows) == sum(op.question == question for op in operations)
|
||||||
|
|
||||||
|
|
||||||
def _context_boundaries(scan: ModelScanResult) -> tuple[str, ...]:
|
def _context_boundaries(scan: ModelScanResult, question: QuestionIdentity | None = None) -> tuple[str, ...]:
|
||||||
# A missing scheduling vector is not missing arbiter evidence after its question completed.
|
# Aggregate coverage keeps every gap. A candidate may ignore only a recognized
|
||||||
return tuple(b for b in scan.scope.unresolved_boundaries if not b.startswith("evidence_unknown:")) if scan.scope else ("scope_missing",)
|
# question-owned boundary attached to a different, recorded question. Unknown
|
||||||
|
# ownership and graph/declaration gaps remain global.
|
||||||
|
if scan.scope is None:
|
||||||
|
return ("scope_missing",)
|
||||||
|
owned_prefixes = (
|
||||||
|
"context_projection_unavailable:contributor-scan:",
|
||||||
|
"context_scope_empty:contributor-scan:",
|
||||||
|
"dynamic_external_or_depth_context_unresolved:contributor-scan:",
|
||||||
|
)
|
||||||
|
others = {q.identity.question_id for q in scan.scope.selected if q.identity != question}
|
||||||
|
others.update(q.identity.question_id for q in scan.scope.deferred if q.identity != question)
|
||||||
|
return tuple(
|
||||||
|
boundary
|
||||||
|
for boundary in scan.scope.unresolved_boundaries
|
||||||
|
if not boundary.startswith("evidence_unknown:")
|
||||||
|
and not (question is not None and boundary.startswith(owned_prefixes) and boundary.rsplit(":", 1)[-1] in others)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _counterpart(question: QuestionIdentity, context: ChangeContext) -> QuestionIdentity:
|
def _counterpart(question: QuestionIdentity, context: ChangeContext) -> QuestionIdentity:
|
||||||
@@ -268,7 +284,7 @@ def compare_evidence(
|
|||||||
base
|
base
|
||||||
and base.scope
|
and base.scope
|
||||||
and base.scope.population_complete
|
and base.scope.population_complete
|
||||||
and not base.scope.deferred
|
and all(question.reason == "tier_zero" for question in base.scope.deferred)
|
||||||
and not _context_boundaries(base)
|
and not _context_boundaries(base)
|
||||||
and not base.degradations
|
and not base.degradations
|
||||||
and _valid_answer(base, counterpart, base_ops)
|
and _valid_answer(base, counterpart, base_ops)
|
||||||
@@ -285,7 +301,7 @@ def compare_evidence(
|
|||||||
reason = "change_context_unresolved"
|
reason = "change_context_unresolved"
|
||||||
elif (
|
elif (
|
||||||
not head.scope
|
not head.scope
|
||||||
or _context_boundaries(head)
|
or _context_boundaries(head, op.question)
|
||||||
or head.degradations
|
or head.degradations
|
||||||
or not _valid_answer(head, op.question, head_ops)
|
or not _valid_answer(head, op.question, head_ops)
|
||||||
or op.question not in {q.identity for q in head.scope.selected}
|
or op.question not in {q.identity for q in head.scope.selected}
|
||||||
|
|||||||
@@ -15,9 +15,9 @@ from pathlib import Path
|
|||||||
from types import MappingProxyType
|
from types import MappingProxyType
|
||||||
from typing import Literal
|
from typing import Literal
|
||||||
|
|
||||||
from openultrasast.model.contracts import ExecutionBudget
|
from openultrasast.model.contracts import ChangeContext, ExecutionBudget
|
||||||
from openultrasast.model.scan import ModelScanResult
|
from openultrasast.model.scan import ModelScanResult
|
||||||
from openultrasast.push.contracts import PushComparison, PushResolution, PushResult
|
from openultrasast.push.contracts import PushComparison, PushResolution, PushResult, SnapshotManifest
|
||||||
from openultrasast.push.policy import AdmissionResult, _context_boundaries
|
from openultrasast.push.policy import AdmissionResult, _context_boundaries
|
||||||
|
|
||||||
|
|
||||||
@@ -36,6 +36,8 @@ class PushReport:
|
|||||||
provenance: Mapping[str, str]
|
provenance: Mapping[str, str]
|
||||||
timings: Mapping[str, float]
|
timings: Mapping[str, float]
|
||||||
resolution: PushResolution | None = None
|
resolution: PushResolution | None = None
|
||||||
|
snapshots: tuple[SnapshotManifest, ...] = ()
|
||||||
|
change_context: ChangeContext | None = None
|
||||||
|
|
||||||
def __post_init__(self) -> None:
|
def __post_init__(self) -> None:
|
||||||
object.__setattr__(self, "provenance", MappingProxyType(dict(self.provenance)))
|
object.__setattr__(self, "provenance", MappingProxyType(dict(self.provenance)))
|
||||||
@@ -109,6 +111,8 @@ def _write_artifact(report: PushReport, target: Path, temporary: Path, connectio
|
|||||||
with os.fdopen(fd, "w", encoding="utf-8") as stream:
|
with os.fdopen(fd, "w", encoding="utf-8") as stream:
|
||||||
payload = {
|
payload = {
|
||||||
"schema_version": 1,
|
"schema_version": 1,
|
||||||
|
"snapshots": [item.to_payload() for item in report.snapshots],
|
||||||
|
"change_context": report.change_context.to_payload() if report.change_context else None,
|
||||||
"result": report.result.to_payload(),
|
"result": report.result.to_payload(),
|
||||||
"admission": report.admission.to_payload(),
|
"admission": report.admission.to_payload(),
|
||||||
"scans": [asdict(record) for record in report.scans],
|
"scans": [asdict(record) for record in report.scans],
|
||||||
@@ -223,7 +227,9 @@ def render_report(report: PushReport, *, artifact: Path | None, error: str | Non
|
|||||||
notices.append(
|
notices.append(
|
||||||
"Some checks could not be completed. Review the comparison details and resolve the recorded gaps before retrying."
|
"Some checks could not be completed. Review the comparison details and resolve the recorded gaps before retrying."
|
||||||
)
|
)
|
||||||
if error:
|
if error == "artifact_inside_repository":
|
||||||
|
notices.append("Details could not be saved. Choose an artifact path outside the analyzed repository.")
|
||||||
|
elif error:
|
||||||
notices.append("Details could not be saved. Retry with a writable artifact path and available reporting time.")
|
notices.append("Details could not be saved. Retry with a writable artifact path and available reporting time.")
|
||||||
if notices:
|
if notices:
|
||||||
lines.append("Notice: " + " ".join(notices))
|
lines.append("Notice: " + " ".join(notices))
|
||||||
|
|||||||
@@ -0,0 +1,279 @@
|
|||||||
|
"""Bounded, local base/head replay through the existing repository driver."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import multiprocessing
|
||||||
|
import os
|
||||||
|
import pickle
|
||||||
|
import selectors
|
||||||
|
import time
|
||||||
|
from collections.abc import Callable
|
||||||
|
from dataclasses import asdict, replace
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any, Literal, TypeVar, cast
|
||||||
|
|
||||||
|
from openultrasast.config import PushConfig
|
||||||
|
from openultrasast.cpg.backend import joern_version, resolve_cpg_backend
|
||||||
|
from openultrasast.mapping import analyze_entry_points
|
||||||
|
from openultrasast.model.contracts import ExecutionBudget
|
||||||
|
from openultrasast.model.regions import ScanRegion, regions_for
|
||||||
|
from openultrasast.model.scan import ScanBudget, scan_repository
|
||||||
|
from openultrasast.model.shipped import declared_sources
|
||||||
|
from openultrasast.preprocess import build_file_target, enumerate_source_files
|
||||||
|
from openultrasast.push.contracts import ComparisonAnalysis, SnapshotManifest
|
||||||
|
from openultrasast.push.policy import (
|
||||||
|
AdmissionCandidate,
|
||||||
|
AdmissionResult,
|
||||||
|
CapabilityKey,
|
||||||
|
admission_push_result,
|
||||||
|
admit_candidates,
|
||||||
|
compare_targeted_base,
|
||||||
|
semantics_digest,
|
||||||
|
)
|
||||||
|
from openultrasast.push.report import PushReport, ReportDelivery, ScanRecord, _completed_scan, deliver_report, render_report
|
||||||
|
from openultrasast.push.snapshot import SnapshotAdapter
|
||||||
|
|
||||||
|
_T = TypeVar("_T")
|
||||||
|
|
||||||
|
|
||||||
|
def _prepare(function: Callable[[], _T], budget: ExecutionBudget) -> _T:
|
||||||
|
"""Bound pure local discovery/metadata work, including serialization, on Linux.
|
||||||
|
|
||||||
|
Read incrementally while the child writes; no pipe-buffer deadlock or unbounded
|
||||||
|
blocking recv. The child never launches engines or project code.
|
||||||
|
"""
|
||||||
|
if time.monotonic() >= budget.deadline_monotonic:
|
||||||
|
raise TimeoutError("deadline_exhausted")
|
||||||
|
reader, writer = os.pipe()
|
||||||
|
|
||||||
|
def child() -> None:
|
||||||
|
os.close(reader)
|
||||||
|
try:
|
||||||
|
payload: tuple[bool, Any]
|
||||||
|
try:
|
||||||
|
payload = (True, function())
|
||||||
|
except Exception as error:
|
||||||
|
payload = (False, type(error).__name__)
|
||||||
|
with os.fdopen(writer, "wb") as stream:
|
||||||
|
pickle.dump(payload, stream, protocol=5)
|
||||||
|
finally:
|
||||||
|
os._exit(0)
|
||||||
|
|
||||||
|
process = multiprocessing.get_context("fork").Process(target=child)
|
||||||
|
data = bytearray()
|
||||||
|
try:
|
||||||
|
process.start()
|
||||||
|
os.close(writer)
|
||||||
|
writer = -1
|
||||||
|
with selectors.DefaultSelector() as selector:
|
||||||
|
selector.register(reader, selectors.EVENT_READ)
|
||||||
|
while selector.get_map():
|
||||||
|
remaining = budget.deadline_monotonic - time.monotonic()
|
||||||
|
if remaining <= 0:
|
||||||
|
raise TimeoutError("deadline_exhausted")
|
||||||
|
for key, _ in selector.select(min(0.05, remaining)):
|
||||||
|
chunk = os.read(key.fd, 65536)
|
||||||
|
if not chunk:
|
||||||
|
selector.unregister(key.fileobj)
|
||||||
|
else:
|
||||||
|
data.extend(chunk)
|
||||||
|
if len(data) > 32 * 1024**2:
|
||||||
|
raise ValueError("preparation_output_limit")
|
||||||
|
process.join(max(0, budget.deadline_monotonic - time.monotonic()))
|
||||||
|
if process.is_alive() or process.exitcode != 0 or not data:
|
||||||
|
raise RuntimeError("preparation_worker_failed")
|
||||||
|
success, result = pickle.loads(data) # trusted child, never repository-provided pickle
|
||||||
|
if not success:
|
||||||
|
raise RuntimeError("preparation_failed:" + result)
|
||||||
|
return cast(_T, result)
|
||||||
|
finally:
|
||||||
|
os.close(reader)
|
||||||
|
if writer >= 0:
|
||||||
|
os.close(writer)
|
||||||
|
if process.pid is not None:
|
||||||
|
if process.is_alive():
|
||||||
|
process.kill()
|
||||||
|
process.join(max(0, budget.deadline_monotonic + budget.cancellation_allowance_seconds - time.monotonic()))
|
||||||
|
if not process.is_alive():
|
||||||
|
process.close()
|
||||||
|
|
||||||
|
|
||||||
|
def _discover(root: Path) -> tuple[tuple[ScanRegion, ...], tuple[bytes, ...]]:
|
||||||
|
# Same proposer and declaration-based shipping rules as ordinary scans. Do not
|
||||||
|
# call preprocess's live-HEAD probe on a materialized, intentionally Git-free tree.
|
||||||
|
files = enumerate_source_files(root)
|
||||||
|
targets = [build_file_target(root, path) for path in files]
|
||||||
|
regions = regions_for(analyze_entry_points(root, targets), targets, shipped=declared_sources(root))
|
||||||
|
# Conservatively seed every tracked declaration/non-source file. This is change
|
||||||
|
# context only, never a graph allowlist or a second scope selector.
|
||||||
|
sources = set(files)
|
||||||
|
declarations = tuple(os.fsencode(str(p.relative_to(root))) for p in root.rglob("*") if p.is_file() and p not in sources)
|
||||||
|
return regions, declarations
|
||||||
|
|
||||||
|
|
||||||
|
def _provenance(repository: Path, scan_budget: ScanBudget) -> dict[str, str]:
|
||||||
|
root = Path(__file__).resolve().parents[1]
|
||||||
|
|
||||||
|
def digest(paths: list[Path]) -> str:
|
||||||
|
hasher = hashlib.sha256()
|
||||||
|
for path in sorted(paths):
|
||||||
|
hasher.update(str(path.relative_to(root)).encode())
|
||||||
|
hasher.update(path.read_bytes())
|
||||||
|
return hasher.hexdigest()
|
||||||
|
|
||||||
|
version = joern_version()
|
||||||
|
engine = "joern-" + (".".join(map(str, version)) if version else "unavailable_or_unidentified")
|
||||||
|
return {
|
||||||
|
"repository": str(repository.absolute()),
|
||||||
|
"engine": engine,
|
||||||
|
"facts": digest(list((root / "ruleset").rglob("*.toml"))),
|
||||||
|
"queries": digest(list((root / "cpg/queries").glob("*.sc"))),
|
||||||
|
"policy": digest(list((root / "push").glob("*.py"))),
|
||||||
|
"core": digest(list((root / "model").glob("*.py")) + list((root / "cpg").glob("*.py"))),
|
||||||
|
"semantics": semantics_digest(engine_identity=engine, options=asdict(scan_budget)),
|
||||||
|
"ranking_mode": "evidence",
|
||||||
|
"model": "disabled",
|
||||||
|
"capabilities": "empty_registry_pending_task_8.3",
|
||||||
|
"cache": "cold_no_reuse",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def replay(
|
||||||
|
repository: Path,
|
||||||
|
*,
|
||||||
|
base: str,
|
||||||
|
head: str,
|
||||||
|
artifact: Path,
|
||||||
|
config: PushConfig | None = None,
|
||||||
|
backend: Any = None,
|
||||||
|
max_regions: int = 500,
|
||||||
|
) -> ReportDelivery:
|
||||||
|
"""Run a single explicit comparison; exit policy never stands for coverage."""
|
||||||
|
config = config or PushConfig()
|
||||||
|
if max_regions < 1:
|
||||||
|
raise ValueError("max_regions must be positive")
|
||||||
|
started = time.monotonic()
|
||||||
|
budget = ExecutionBudget(started + config.deadline_seconds, config.cancellation_allowance_seconds)
|
||||||
|
scan_budget = ScanBudget(max_model_calls=0, max_regions=max_regions, order_by_evidence=True)
|
||||||
|
records: list[ScanRecord] = []
|
||||||
|
manifests: list[SnapshotManifest] = []
|
||||||
|
reasons: list[str] = []
|
||||||
|
admission = AdmissionResult((), (), ())
|
||||||
|
comparison = None
|
||||||
|
context = None
|
||||||
|
timings: dict[str, float] = {
|
||||||
|
"deadline_seconds": config.deadline_seconds,
|
||||||
|
"cancellation_allowance_seconds": config.cancellation_allowance_seconds,
|
||||||
|
}
|
||||||
|
provenance = {key: "unavailable_preparation_incomplete" for key in ("engine", "facts", "queries", "policy")}
|
||||||
|
provenance.update(repository=str(repository.absolute()), model="disabled", requested_base=base, requested_head=head)
|
||||||
|
stage = "resolution"
|
||||||
|
stage_started = started
|
||||||
|
try:
|
||||||
|
adapter = SnapshotAdapter(repository, execution_budget=budget)
|
||||||
|
comparison = adapter.resolve_replay(base, head)
|
||||||
|
timings[stage + "_seconds"] = time.monotonic() - stage_started
|
||||||
|
stage, stage_started = "provenance", time.monotonic()
|
||||||
|
provenance.update(_prepare(lambda: _provenance(repository, scan_budget), budget))
|
||||||
|
timings[stage + "_seconds"] = time.monotonic() - stage_started
|
||||||
|
stage, stage_started = "preparation", time.monotonic()
|
||||||
|
with adapter.materialize(comparison.head_oid, budget=budget) as tip:
|
||||||
|
manifests.append(tip.manifest)
|
||||||
|
assert comparison.base_oid is not None
|
||||||
|
with adapter.materialize(comparison.base_oid, budget=budget) as old:
|
||||||
|
manifests.append(old.manifest)
|
||||||
|
for manifest in manifests:
|
||||||
|
reasons.extend("snapshot:" + boundary.reason for boundary in manifest.boundaries)
|
||||||
|
head_regions, head_declarations = _prepare(lambda: _discover(tip.root), budget)
|
||||||
|
base_regions, base_declarations = _prepare(lambda: _discover(old.root), budget)
|
||||||
|
context = adapter.compare(comparison, declaration_paths=tuple(set(head_declarations + base_declarations)), budget=budget)
|
||||||
|
context = replace(context, unresolved_boundaries=tuple(dict.fromkeys((*context.unresolved_boundaries, *reasons))))
|
||||||
|
reasons.extend(context.unresolved_boundaries)
|
||||||
|
timings[stage + "_seconds"] = time.monotonic() - stage_started
|
||||||
|
stage, stage_started = "head", time.monotonic()
|
||||||
|
head_scan = scan_repository(
|
||||||
|
tip.root,
|
||||||
|
head_regions,
|
||||||
|
backend=backend or resolve_cpg_backend(),
|
||||||
|
budget=scan_budget,
|
||||||
|
execution_budget=budget,
|
||||||
|
ranking_mode="evidence",
|
||||||
|
unit="repository",
|
||||||
|
population_complete=all(m.complete for m in manifests),
|
||||||
|
change_context=context,
|
||||||
|
)
|
||||||
|
records.append(ScanRecord(comparison, "head", head_scan))
|
||||||
|
timings[stage + "_seconds"] = time.monotonic() - stage_started
|
||||||
|
stage, stage_started = "base", time.monotonic()
|
||||||
|
delta = compare_targeted_base(
|
||||||
|
old.root,
|
||||||
|
head=head_scan,
|
||||||
|
head_regions=head_regions,
|
||||||
|
base_regions=base_regions,
|
||||||
|
context=context,
|
||||||
|
backend=backend or resolve_cpg_backend(),
|
||||||
|
execution_budget=budget,
|
||||||
|
scan_budget=scan_budget,
|
||||||
|
head_semantics=provenance["semantics"],
|
||||||
|
base_semantics=provenance["semantics"],
|
||||||
|
)
|
||||||
|
if delta.base_scan is not None:
|
||||||
|
records.append(ScanRecord(comparison, "base", delta.base_scan))
|
||||||
|
reasons.extend(delta.coverage_reasons)
|
||||||
|
timings[stage + "_seconds"] = time.monotonic() - stage_started
|
||||||
|
stage, stage_started = "admission", time.monotonic()
|
||||||
|
candidates = tuple(
|
||||||
|
AdmissionCandidate(
|
||||||
|
c,
|
||||||
|
CapabilityKey(
|
||||||
|
c.head_operation.question.language if c.head_operation else "unknown",
|
||||||
|
"unspecified",
|
||||||
|
"unspecified",
|
||||||
|
c.family,
|
||||||
|
c.head_operation.mechanism if c.head_operation else "taint",
|
||||||
|
"unreviewed",
|
||||||
|
delta.semantics,
|
||||||
|
),
|
||||||
|
comparison,
|
||||||
|
tuple(reasons),
|
||||||
|
)
|
||||||
|
for c in delta.candidates
|
||||||
|
)
|
||||||
|
admission = _prepare(lambda: admit_candidates(candidates), budget)
|
||||||
|
timings[stage + "_seconds"] = time.monotonic() - stage_started
|
||||||
|
stage, stage_started = "cleanup", time.monotonic()
|
||||||
|
timings[stage + "_seconds"] = time.monotonic() - stage_started
|
||||||
|
except Exception as error:
|
||||||
|
reasons.append(stage + "_failed:" + type(error).__name__)
|
||||||
|
timings[stage + "_seconds"] = time.monotonic() - stage_started
|
||||||
|
if time.monotonic() >= budget.deadline_monotonic:
|
||||||
|
reasons.append("deadline_exhausted")
|
||||||
|
head_records = [r.scan for r in records if r.side == "head"]
|
||||||
|
for record in records:
|
||||||
|
if record.scan.scope is not None:
|
||||||
|
reasons.extend(record.scan.scope.unresolved_boundaries)
|
||||||
|
if not records or not all(_completed_scan(r.scan) for r in records):
|
||||||
|
reasons.append("analysis_incomplete")
|
||||||
|
if not any(r.side == "base" for r in records):
|
||||||
|
reasons.append("base_comparison_unavailable")
|
||||||
|
admission = replace(admission, coverage_reasons=tuple(dict.fromkeys((*admission.coverage_reasons, *reasons))))
|
||||||
|
analyses: tuple[ComparisonAnalysis, ...] = ()
|
||||||
|
if comparison is not None:
|
||||||
|
scopes = tuple(scan.scope for scan in head_records if scan.scope is not None)
|
||||||
|
completed = tuple(outcome.identity for scan in head_records for outcome in scan.question_outcomes if outcome.status == "completed")
|
||||||
|
coverage: Literal["unavailable", "incomplete", "complete_within_scope"] = (
|
||||||
|
"unavailable" if not records else "incomplete" if admission.coverage_reasons else "complete_within_scope"
|
||||||
|
)
|
||||||
|
analyses = (ComparisonAnalysis(comparison, scopes, completed, coverage),)
|
||||||
|
result = admission_push_result(admission, analyses, config=config)
|
||||||
|
timings["total_seconds"] = time.monotonic() - started
|
||||||
|
report = PushReport(result, admission, tuple(records), provenance, timings, snapshots=tuple(manifests), change_context=context)
|
||||||
|
try:
|
||||||
|
artifact.resolve().relative_to(repository.resolve())
|
||||||
|
except ValueError:
|
||||||
|
pass
|
||||||
|
else:
|
||||||
|
write_error = "artifact_inside_repository"
|
||||||
|
return ReportDelivery(result, render_report(report, artifact=None, error=write_error), None, write_error)
|
||||||
|
return deliver_report(report, artifact, execution_budget=budget)
|
||||||
@@ -44,7 +44,8 @@ class SnapshotInputError(ValueError):
|
|||||||
|
|
||||||
|
|
||||||
class SnapshotAdapter:
|
class SnapshotAdapter:
|
||||||
def __init__(self, repository: Path, *, comparison_base: str | None = None) -> None:
|
def __init__(self, repository: Path, *, comparison_base: str | None = None, execution_budget: ExecutionBudget | None = None) -> None:
|
||||||
|
self.execution_budget = execution_budget
|
||||||
self.repository = repository.resolve()
|
self.repository = repository.resolve()
|
||||||
self.comparison_base = comparison_base
|
self.comparison_base = comparison_base
|
||||||
# Hook-inherited Git variables may redirect objects or worktrees elsewhere.
|
# Hook-inherited Git variables may redirect objects or worktrees elsewhere.
|
||||||
@@ -72,10 +73,22 @@ class SnapshotAdapter:
|
|||||||
|
|
||||||
def _repository_path(self, *args: str) -> Path:
|
def _repository_path(self, *args: str) -> Path:
|
||||||
# Binary framing also matters for the repository's own whitespace/non-UTF8 name.
|
# Binary framing also matters for the repository's own whitespace/non-UTF8 name.
|
||||||
raw = self._bounded_git(("rev-parse", *args), 65536, ExecutionBudget(time.monotonic() + 10.0, 2.0), None, [])
|
raw = self._bounded_git(
|
||||||
|
("rev-parse", *args), 65536, self.execution_budget or ExecutionBudget(time.monotonic() + 10.0, 2.0), None, []
|
||||||
|
)
|
||||||
return Path(os.fsdecode(raw.removesuffix(b"\n"))).resolve()
|
return Path(os.fsdecode(raw.removesuffix(b"\n"))).resolve()
|
||||||
|
|
||||||
def _git(self, *args: str, input: str | None = None) -> subprocess.CompletedProcess[str]:
|
def _git(self, *args: str, input: str | None = None) -> subprocess.CompletedProcess[str]:
|
||||||
|
if self.execution_budget is not None:
|
||||||
|
if input not in (None, ""):
|
||||||
|
raise SnapshotInputError("bounded_resolution_requires_empty_stdin")
|
||||||
|
try:
|
||||||
|
raw = self._bounded_git(tuple(args), 16 * 1024**2, self.execution_budget, None, [])
|
||||||
|
return subprocess.CompletedProcess(args, 0, raw.decode("utf-8", "surrogateescape"), "")
|
||||||
|
except SnapshotInputError as error:
|
||||||
|
if str(error).startswith("local_git_") and "_failed_exit_" in str(error):
|
||||||
|
return subprocess.CompletedProcess(args, 1, "", "")
|
||||||
|
raise
|
||||||
try:
|
try:
|
||||||
return subprocess.run(
|
return subprocess.run(
|
||||||
["git", "-c", "protocol.allow=never", "-c", "core.hooksPath=" + os.devnull, "-C", str(self.repository), *args],
|
["git", "-c", "protocol.allow=never", "-c", "core.hooksPath=" + os.devnull, "-C", str(self.repository), *args],
|
||||||
@@ -97,6 +110,17 @@ class SnapshotAdapter:
|
|||||||
raise SnapshotInputError(f"local Git {args[0]} failed (exit {result.returncode})")
|
raise SnapshotInputError(f"local Git {args[0]} failed (exit {result.returncode})")
|
||||||
return result.stdout.strip()
|
return result.stdout.strip()
|
||||||
|
|
||||||
|
def resolve_replay(self, base: str, head: str) -> PushComparison:
|
||||||
|
"""Resolve explicitly supplied local revisions without inferred baselines or fetching."""
|
||||||
|
resolved = []
|
||||||
|
for revision in (base, head):
|
||||||
|
oid = self._required("rev-parse", "--verify", "--end-of-options", revision)
|
||||||
|
commit, kind = self._commit(oid)
|
||||||
|
if commit is None:
|
||||||
|
raise SnapshotInputError("replay_revision_" + kind)
|
||||||
|
resolved.append(commit)
|
||||||
|
return PushComparison(resolved[1], resolved[0], "explicit_local_replay", ("refs/heads/replay",))
|
||||||
|
|
||||||
def parse_updates(self, data: str) -> tuple[PushUpdate, ...]:
|
def parse_updates(self, data: str) -> tuple[PushUpdate, ...]:
|
||||||
"""Validate every record before attempting resolution; malformed input fails closed."""
|
"""Validate every record before attempting resolution; malformed input fails closed."""
|
||||||
updates = []
|
updates = []
|
||||||
|
|||||||
@@ -206,3 +206,35 @@ def test_non_string_operation_cannot_prove_base_absence():
|
|||||||
row = json.loads(scan(line=5).question_outcomes[0].raw_rows_json)[0]
|
row = json.loads(scan(line=5).question_outcomes[0].raw_rows_json)[0]
|
||||||
row["sink"] = 42
|
row["sink"] = 42
|
||||||
assert delta(scan(), scan(line=5, rows=[row])).novelty == "unknown"
|
assert delta(scan(), scan(line=5, rows=[row])).novelty == "unknown"
|
||||||
|
|
||||||
|
|
||||||
|
def test_unrelated_question_context_gap_does_not_poison_completed_comparison():
|
||||||
|
head = scan()
|
||||||
|
other = replace(head.scope.selected[0].identity, family="path")
|
||||||
|
gap = "dynamic_external_or_depth_context_unresolved:contributor-scan:" + other.question_id
|
||||||
|
head = replace(
|
||||||
|
head,
|
||||||
|
scope=replace(head.scope, selected=(*head.scope.selected, RankedQuestion(other, 0.0, ())), unresolved_boundaries=(gap,)),
|
||||||
|
question_outcomes=(*head.question_outcomes, QuestionOutcome(other, "unresolved", "change_context_incomplete", "[]")),
|
||||||
|
)
|
||||||
|
assert delta(head, scan(rows=[], findings=False)).novelty == "new"
|
||||||
|
own_gap = gap.rsplit(":", 1)[0] + ":" + head.scope.selected[0].identity.question_id
|
||||||
|
assert (
|
||||||
|
delta(replace(head, scope=replace(head.scope, unresolved_boundaries=(own_gap,))), scan(rows=[], findings=False)).novelty
|
||||||
|
== "unknown"
|
||||||
|
)
|
||||||
|
assert (
|
||||||
|
delta(replace(head, scope=replace(head.scope, unresolved_boundaries=("graph_incomplete",))), scan(rows=[], findings=False)).novelty
|
||||||
|
== "unknown"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_proven_tier_zero_other_families_do_not_invalidate_base_answer():
|
||||||
|
from openultrasast.model.contracts import DeferredQuestion
|
||||||
|
|
||||||
|
base = scan(rows=[], findings=False)
|
||||||
|
other = replace(base.scope.selected[0].identity, family="path")
|
||||||
|
base = replace(base, scope=replace(base.scope, deferred=(DeferredQuestion(other, "tier_zero", ()),)))
|
||||||
|
assert delta(scan(), base).novelty == "new"
|
||||||
|
base = replace(base, scope=replace(base.scope, deferred=(DeferredQuestion(other, "budget_exhausted", ()),)))
|
||||||
|
assert delta(scan(), base).novelty == "unknown"
|
||||||
|
|||||||
@@ -0,0 +1,198 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import subprocess
|
||||||
|
import time
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from openultrasast.config import PushConfig
|
||||||
|
from openultrasast.cpg.backend import NullBackend
|
||||||
|
from openultrasast.push.runner import replay
|
||||||
|
|
||||||
|
|
||||||
|
def history(tmp_path):
|
||||||
|
root = tmp_path / "repo"
|
||||||
|
root.mkdir()
|
||||||
|
|
||||||
|
def git(*args):
|
||||||
|
return subprocess.run(["git", "-C", str(root), *args], check=True, capture_output=True).stdout.strip().decode()
|
||||||
|
|
||||||
|
git("init")
|
||||||
|
git("config", "user.name", "Replay test")
|
||||||
|
git("config", "user.email", "[email protected]")
|
||||||
|
git("config", "core.hooksPath", os.devnull)
|
||||||
|
source = root / "api.js"
|
||||||
|
source.write_text('function handler(req, res) {\n const value = "fixed";\n eval(value);\n}\napp.post("/run", handler);\n')
|
||||||
|
git("add", "api.js")
|
||||||
|
git("commit", "-m", "base")
|
||||||
|
base = git("rev-parse", "HEAD")
|
||||||
|
source.write_text(source.read_text().replace('"fixed"', "req.body.value"))
|
||||||
|
git("add", "api.js")
|
||||||
|
git("commit", "-m", "head")
|
||||||
|
head = git("rev-parse", "HEAD")
|
||||||
|
source.write_text("dirty source must never be scanned")
|
||||||
|
return root, base, head, git
|
||||||
|
|
||||||
|
|
||||||
|
def test_replay_unavailable_preserves_input_and_records_scope(tmp_path):
|
||||||
|
root, base, head, git = history(tmp_path)
|
||||||
|
before = (root / ".git/index").read_bytes(), git("status", "--porcelain")
|
||||||
|
artifact = tmp_path / "result.json"
|
||||||
|
delivery = replay(root, base=base, head=head, artifact=artifact, backend=NullBackend())
|
||||||
|
data = json.loads(artifact.read_text())
|
||||||
|
assert delivery.exit_code == 0
|
||||||
|
assert delivery.result.coverage_status in ("unavailable", "incomplete")
|
||||||
|
assert data["result"]["analyses"][0]["comparison"]["base_oid"] == base
|
||||||
|
assert data["result"]["analyses"][0]["comparison"]["head_oid"] == head
|
||||||
|
assert data["scans"][0]["scan"]["scope"]["ranking_mode"] == "evidence"
|
||||||
|
assert data["provenance"]["model"] == "disabled"
|
||||||
|
assert data["snapshots"][0]["bytes_read"] > 0
|
||||||
|
assert (root / "api.js").read_text() == "dirty source must never be scanned"
|
||||||
|
assert ((root / ".git/index").read_bytes(), git("status", "--porcelain")) == before
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_revision_is_unavailable_not_clean(tmp_path):
|
||||||
|
root, base, head, _ = history(tmp_path)
|
||||||
|
result = replay(root, base="missing-ref", head=head, artifact=tmp_path / "missing.json")
|
||||||
|
assert result.result.coverage_status == "unavailable"
|
||||||
|
assert result.exit_code == 0
|
||||||
|
assert "Notice:" in result.text
|
||||||
|
|
||||||
|
|
||||||
|
def test_expired_preparation_obeys_total_deadline(tmp_path):
|
||||||
|
root, base, head, _ = history(tmp_path)
|
||||||
|
start = time.monotonic()
|
||||||
|
result = replay(
|
||||||
|
root,
|
||||||
|
base=base,
|
||||||
|
head=head,
|
||||||
|
artifact=tmp_path / "expired.json",
|
||||||
|
config=PushConfig(deadline_seconds=0.001, cancellation_allowance_seconds=0.5),
|
||||||
|
)
|
||||||
|
assert time.monotonic() - start < 0.55
|
||||||
|
assert result.result.coverage_status != "complete_within_scope"
|
||||||
|
|
||||||
|
|
||||||
|
def test_strict_incomplete_policy_and_artifact_failure(tmp_path):
|
||||||
|
root, base, head, _ = history(tmp_path)
|
||||||
|
result = replay(
|
||||||
|
root,
|
||||||
|
base=base,
|
||||||
|
head=head,
|
||||||
|
artifact=tmp_path,
|
||||||
|
config=PushConfig(mode="blocking", incomplete_coverage_policy="block"),
|
||||||
|
backend=NullBackend(),
|
||||||
|
)
|
||||||
|
assert result.exit_code == 1 and result.error and result.artifact is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_hanging_discovery_is_bounded(tmp_path, monkeypatch):
|
||||||
|
from openultrasast.push import runner
|
||||||
|
|
||||||
|
root, base, head, _ = history(tmp_path)
|
||||||
|
monkeypatch.setattr(runner, "_discover", lambda *args: time.sleep(5))
|
||||||
|
start = time.monotonic()
|
||||||
|
result = replay(
|
||||||
|
root,
|
||||||
|
base=base,
|
||||||
|
head=head,
|
||||||
|
artifact=tmp_path / "hung.json",
|
||||||
|
config=PushConfig(deadline_seconds=0.3, cancellation_allowance_seconds=0.5),
|
||||||
|
)
|
||||||
|
assert time.monotonic() - start < 0.85
|
||||||
|
assert result.result.coverage_status == "unavailable"
|
||||||
|
assert "deadline" in (tmp_path / "hung.json").read_text()
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_exposes_replay(tmp_path, capsys):
|
||||||
|
from openultrasast.cli import main
|
||||||
|
|
||||||
|
root, base, head, _ = history(tmp_path)
|
||||||
|
assert (
|
||||||
|
main(["pre-push", str(root), "--base", base, "--head", head, "--artifact", str(tmp_path / "cli.json"), "--deadline", "0.001"]) == 0
|
||||||
|
)
|
||||||
|
assert "Notice:" in capsys.readouterr().out
|
||||||
|
|
||||||
|
|
||||||
|
def test_replay_artifact_cannot_overwrite_live_repository(tmp_path):
|
||||||
|
root, base, head, _ = history(tmp_path)
|
||||||
|
source = root / "api.js"
|
||||||
|
before = source.read_bytes()
|
||||||
|
result = replay(root, base=base, head=head, artifact=source, backend=NullBackend())
|
||||||
|
assert result.artifact is None and result.error == "artifact_inside_repository"
|
||||||
|
assert source.read_bytes() == before
|
||||||
|
|
||||||
|
|
||||||
|
def test_snapshot_resolution_uses_shared_deadline(tmp_path, monkeypatch):
|
||||||
|
from openultrasast.model.contracts import ExecutionBudget
|
||||||
|
from openultrasast.push.snapshot import SnapshotAdapter, SnapshotInputError
|
||||||
|
|
||||||
|
bin_dir = tmp_path / "bin"
|
||||||
|
bin_dir.mkdir()
|
||||||
|
launcher = bin_dir / "git"
|
||||||
|
launcher.write_text("#!/bin/sh\nexec sleep 5\n")
|
||||||
|
launcher.chmod(0o755)
|
||||||
|
monkeypatch.setenv("PATH", str(bin_dir) + os.pathsep + os.environ["PATH"])
|
||||||
|
started = time.monotonic()
|
||||||
|
with pytest.raises(SnapshotInputError, match="deadline"):
|
||||||
|
SnapshotAdapter(tmp_path, execution_budget=ExecutionBudget(started + 0.1, 0.5))
|
||||||
|
assert time.monotonic() - started < 0.65
|
||||||
|
|
||||||
|
|
||||||
|
def test_preparation_exit_zero_without_receipt_is_failure():
|
||||||
|
from openultrasast.model.contracts import ExecutionBudget
|
||||||
|
from openultrasast.push.runner import _prepare
|
||||||
|
|
||||||
|
with pytest.raises(RuntimeError, match="worker_failed"):
|
||||||
|
_prepare(lambda: os._exit(0), ExecutionBudget(time.monotonic() + 1, 0.5))
|
||||||
|
|
||||||
|
|
||||||
|
def test_base_failure_retains_completed_head_evidence(tmp_path, monkeypatch):
|
||||||
|
from test_push_policy import scan
|
||||||
|
|
||||||
|
from openultrasast.push import runner
|
||||||
|
|
||||||
|
root, base, head, _ = history(tmp_path)
|
||||||
|
seen = []
|
||||||
|
|
||||||
|
def head_scan(root, regions, **kwargs):
|
||||||
|
assert (root / "api.js").read_bytes() and regions
|
||||||
|
assert kwargs["ranking_mode"] == "evidence" and kwargs["change_context"].head_revision == head
|
||||||
|
seen.append(kwargs["execution_budget"])
|
||||||
|
return scan(path="api.js")
|
||||||
|
|
||||||
|
def failed_base(*args, **kwargs):
|
||||||
|
assert kwargs["execution_budget"] is seen[0]
|
||||||
|
raise RuntimeError("failed comparison")
|
||||||
|
|
||||||
|
monkeypatch.setattr(runner, "scan_repository", head_scan)
|
||||||
|
monkeypatch.setattr(runner, "compare_targeted_base", failed_base)
|
||||||
|
artifact = tmp_path / "partial.json"
|
||||||
|
result = replay(root, base=base, head=head, artifact=artifact)
|
||||||
|
data = json.loads(artifact.read_text())
|
||||||
|
assert result.result.coverage_status == "incomplete"
|
||||||
|
assert len(data["scans"][0]["scan"]["findings"]) == 1
|
||||||
|
assert len(data["result"]["analyses"][0]["completed_questions"]) == 1
|
||||||
|
assert "base_failed:RuntimeError" in data["admission"]["coverage_reasons"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_preparation_large_result_does_not_deadlock():
|
||||||
|
from openultrasast.model.contracts import ExecutionBudget
|
||||||
|
from openultrasast.push.runner import _prepare
|
||||||
|
|
||||||
|
assert len(_prepare(lambda: b"x" * 1_000_000, ExecutionBudget(time.monotonic() + 2, 0.5))) == 1_000_000
|
||||||
|
|
||||||
|
|
||||||
|
def test_snapshot_boundaries_reach_engine_change_context(tmp_path):
|
||||||
|
root, base, head, git = history(tmp_path)
|
||||||
|
(root / "external.js").symlink_to("/unavailable/external.js")
|
||||||
|
git("add", "external.js")
|
||||||
|
git("commit", "-m", "external context")
|
||||||
|
head = git("rev-parse", "HEAD")
|
||||||
|
artifact = tmp_path / "boundary.json"
|
||||||
|
result = replay(root, base=base, head=head, artifact=artifact, backend=NullBackend())
|
||||||
|
data = json.loads(artifact.read_text())
|
||||||
|
assert result.result.coverage_status == "incomplete"
|
||||||
|
assert "snapshot:symlink_not_materialized" in data["change_context"]["unresolved_boundaries"]
|
||||||
Reference in New Issue
Block a user