feat(pre-push-safety-net): render bounded reports and preserve full evidence

This commit is contained in:
Marius Ciepluch
2026-09-13 13:25:44 +02:00
parent d0e0ecad78
commit 6b12b73411
13 changed files with 3105 additions and 35 deletions
+1 -1
View File
@@ -1,6 +1,6 @@
# Brief: pre-push-safety-net # Brief: pre-push-safety-net
Status: requirements, design and tasks approved on 2026-09-12; groups 1–3 verified complete (11/27 subtasks), including the approved contributor-scan frontend-retention prerequisite; groups 4–8 pending. Language: English. Status: requirements, design and tasks approved; groups 1–4 verified complete (14/27 subtasks). Groups 5–8 pending; real capability eligibility and hook latency remain unproven. Language: English.
## Problem ## Problem
@@ -4,9 +4,10 @@ Date: 2026-09-13. The maintainer authorized group 4 as a whole. Scope is tasks
4.1–4.3: targeted novelty comparison, actionability admission and compact reporting. 4.1–4.3: targeted novelty comparison, actionability admission and compact reporting.
Groups 1–3 are verified at `6ba2b06`; full transaction replay remains task 5.1. Groups 1–3 are verified at `6ba2b06`; full transaction replay remains task 5.1.
Status: implementation in progress. Task completion requires independent review Status: tasks 4.1–4.3 are VERIFIED (TASK scope), bringing the spec to 14/27 tasks.
and fresh evidence. No production capability is enabled for normal hook alerts; Tasks 4.1 and 4.2 passed independent review. Task 4.3 used the Kiro manual review
capability admission still requires task 8.3 evaluation. fallback after the agent service rejected further dispatch. No production capability
is enabled for normal hook alerts; actual admission still requires task 8.3 evaluation.
## Verification boundary ## Verification boundary
@@ -62,3 +63,64 @@ default registry. Explicitly synthetic PHP/WordPress and JavaScript/Express cont
receive equivalent admission and enforcement behavior. They do not qualify any receive equivalent admission and enforcement behavior. They do not qualify any
real capability; task 8.3 still owns actual eligibility. Task 4.2 brings progress to real capability; task 8.3 still owns actual eligibility. Task 4.2 brings progress to
13/27 tasks. Compact reporting remains in this authorized group. 13/27 tasks. Compact reporting remains in this authorized group.
## Task 4.3 verification
The agent service rejected the task 4.3 dispatch with `agent thread limit reached`.
The [kiro-impl skill](../../../.opencode/skills/kiro-impl/SKILL.md) explicitly provides:
“If multi-agent is not available, fall back to manual mode execution for all tasks.”
Implementation, review and completion checks therefore ran in the main context.
This is a manual review record, not an independent-review claim.
OFF controls produced seven relevant failures; ON and unflagged reporting tests
passed. Manual review found and repaired an invalid artifact path escaping the
notice path and a complete comparison carrying failed base evidence. A writer
exiting zero without a receipt is rejected. Stalled-writer tests prove cancellation,
preservation of an existing artifact and no surviving child. Other controls cover
all candidate/scan preservation, five defects with only three displayed, separate
coverage/enforcement, escaping and bounded long text, and immutable provenance.
## Review Verdict
- VERDICT: APPROVED
- TASK: 4.3
- REVIEW_MODE: Kiro manual fallback; agent dispatch unavailable.
- MECHANICAL_RESULTS: final full suite 1,222 passed, nine skipped in 102.39 seconds;
Ruff and format (238 files), mypy (111 source files), compileall, diff checks and
detection/map/pair gates PASS. No new placeholders or hardcoded secrets.
- FINDINGS: Invalid destination and failed-base completeness issues repaired and
verified with failing controls followed by passing tests. No outstanding task finding.
- BOUNDARY: Reporting consumes existing admission and engine records; no new
detector, ranker, graph mutation, provider or installed hook.
The image build and CLI boot passed. The packaged integration control rechecked
captured actual PHP/JavaScript answers through current comparison/admission/reporting,
preserving complete scan payloads and default diagnostic-only output. Five explicitly
synthetic admitted defects retain all five records, display three summaries and keep
the block decision. Artifact write failure also retains that decision. Packaged
policy/reporter hashes match the workspace. See
`benchmarks/measurements/2026-09-13-compact-report-smoke.json`.
Publication is atomic from mode-0600 temporary files. A separate Linux process
bounds serialization and filesystem work by the remaining transaction/report
allowance. Hard cancellation can leave a private unpublished partial file; it is
never presented as the completed artifact. Caller-supplied stage timings are
preserved, and the delivery receipt measures publication/rendering separately.
Observed packaged delivery costs (about 0.02–0.04 seconds for these controls) do not
establish whole-hook latency.
## Group integration validation
- DECISION: GO
- VALIDATION_SCOPE: Tasks 4.1–4.3 only; no full-feature or release admission.
- CROSS_TASK_CONTRACTS: Actual captured engine evidence → bound delta → empty-registry
diagnostics → compact report and complete artifact PASS. Explicitly synthetic
calibrated controls → admission → advisory/blocking → capped rendering PASS.
- REQUIREMENTS: Group-owned portions of 2.4, 3.1–3.5, 4.1, 5.1–5.5, 7.3 and 8.1 are
covered by the implementation and controls above. Full transaction wiring and
representative capability/latency evaluation remain later tasks.
- DESIGN_ALIGNMENT: Existing arbiters and ranker retained; policy and report live
in their planned push-adapter boundary. No orphaned module or upstream detector change.
- BLOCKED_TASKS: None in group 4. Automatic dominance change-context projection is
a documented upstream integration limit, not erased by these declared controls.
- NEXT: Group 5 full replay and first bounded experiment. Complete that experiment
before persistent graph reuse in group 6. Groups 5–8 remain pending.
+6 -5
View File
@@ -1,7 +1,7 @@
{ {
"feature_name": "pre-push-safety-net", "feature_name": "pre-push-safety-net",
"created_at": "2026-09-12T15:35:00Z", "created_at": "2026-09-12T15:35:00Z",
"updated_at": "2026-09-13T10:26:47.307154+00:00", "updated_at": "2026-09-13T11:21:07.667199+00:00",
"language": "en", "language": "en",
"phase": "implementation", "phase": "implementation",
"approvals": { "approvals": {
@@ -19,7 +19,7 @@
} }
}, },
"ready_for_implementation": true, "ready_for_implementation": true,
"implementation_approval_note": "Groups 1\u20133 verified complete. Maintainer explicitly authorized group 4 as a whole; each task requires independent review and fresh verification.", "implementation_approval_note": "Groups 1\u20134 verified complete. Tasks 4.1/4.2 independently approved; 4.3 used the documented Kiro manual review fallback after agent thread limit. Remaining tasks require review and fresh verification.",
"depends_on": [ "depends_on": [
"contributor-scan" "contributor-scan"
], ],
@@ -85,9 +85,10 @@
"3.3", "3.3",
"3.4", "3.4",
"4.1", "4.1",
"4.2" "4.2",
"4.3"
], ],
"remaining_tasks": 14, "remaining_tasks": 13,
"verification_record": "implementation-group-4.md", "verification_record": "implementation-group-4.md",
"verification_records": [ "verification_records": [
"implementation-group-1.md", "implementation-group-1.md",
@@ -95,7 +96,7 @@
"implementation-group-3.md", "implementation-group-3.md",
"implementation-group-4.md" "implementation-group-4.md"
], ],
"next_group": 4, "next_group": 5,
"blocked_tasks": [] "blocked_tasks": []
} }
} }
+7 -5
View File
@@ -1,8 +1,9 @@
# Implementation Plan: pre-push-safety-net # Implementation Plan: pre-push-safety-net
Status: task plan approved by the maintainer's "approve." on 2026-09-12. Groups 1–3 (tasks 1.1–3.4) Status: task plan approved by the maintainer on 2026-09-12. Groups 1–4 (tasks 1.1–4.3)
are complete after independent review and fresh verification (13/27 subtasks, including tasks 4.1–4.2). Task 4.3 and groups 5–8 remain pending; are complete after task reviews and fresh verification (14/27 subtasks). Task 4.3 used
the hook and capability admission are not yet implemented. See `implementation-group-1.md` and `implementation-group-2.md`. the documented manual review fallback after agent dispatch became unavailable.
Groups 5–8 remain pending; no production capability or hook latency is admitted.
## Execution contract ## Execution contract
@@ -112,7 +113,7 @@ the hook and capability admission are not yet implemented. See `implementation-g
- _Requirements: 2.2, 2.3, 7.3, 8.2, 8.4_ - _Requirements: 2.2, 2.3, 7.3, 8.2, 8.4_
- _Verified 2026-09-13:_ independent review APPROVED; 1,151 passed, nine skipped; Ruff/format/mypy pass. Real PHP/JavaScript graphs retain tests/browser source, omit vendor nodes and share one ranker limit. Named census survives PHP aggregation; missing source names and unreadable discovery inputs remain explicit. Python exercises all three batch census schemas. Evidence: `benchmarks/measurements/2026-09-13-partition-scope-smoke.json`. No hook capability or latency claim. - _Verified 2026-09-13:_ independent review APPROVED; 1,151 passed, nine skipped; Ruff/format/mypy pass. Real PHP/JavaScript graphs retain tests/browser source, omit vendor nodes and share one ranker limit. Named census survives PHP aggregation; missing source names and unreadable discovery inputs remain explicit. Python exercises all three batch census schemas. Evidence: `benchmarks/measurements/2026-09-13-partition-scope-smoke.json`. No hook capability or latency claim.
- [ ] 4. Implement change attribution and actionable output - [x] 4. Implement change attribution and actionable output
- [x] 4.1 Compare targeted base/head evidence for defect novelty - [x] 4.1 Compare targeted base/head evidence for defect novelty
- Establish stable mechanism/operation identity with rename/span mapping and collect comparable base evidence under the same semantics and remaining deadline. - Establish stable mechanism/operation identity with rename/span mapping and collect comparable base evidence under the same semantics and remaining deadline.
- Distinguish new, worsened, unchanged and unknown; an absent finding from a failed/truncated base is unknown. Support removed guards and new source connections at unchanged sinks. - Distinguish new, worsened, unchanged and unknown; an absent finding from a failed/truncated base is unknown. Support removed guards and new source connections at unchanged sinks.
@@ -130,10 +131,11 @@ the hook and capability admission are not yet implemented. See `implementation-g
- _Verified 2026-09-13:_ independent review APPROVED after operation/context-binding repair; 1,207 tests passed, nine skipped; default real-evidence controls remain diagnostic and synthetic PHP/JavaScript admission is equivalent. See `implementation-group-4.md`. - _Verified 2026-09-13:_ independent review APPROVED after operation/context-binding repair; 1,207 tests passed, nine skipped; default real-evidence controls remain diagnostic and synthetic PHP/JavaScript admission is equivalent. See `implementation-group-4.md`.
- _Boundary: Delta and Actionability Policy_ - _Boundary: Delta and Actionability Policy_
- _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5, 5.3, 5.4, 8.1_ - _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5, 5.3, 5.4, 8.1_
- [ ] 4.3 Render compact results and preserve complete artifacts - [x] 4.3 Render compact results and preserve complete artifacts
- Render up to three actionable summaries plus remaining count; emit at most one no-finding status and one aggregated incomplete notice with a useful next action. - Render up to three actionable summaries plus remaining count; emit at most one no-finding status and one aggregated incomplete notice with a useful next action.
- Persist all defects, candidate dispositions, exact scope, identities, capability/ranker provenance and timings with separate finding/coverage/push status. - Persist all defects, candidate dispositions, exact scope, identities, capability/ranker provenance and timings with separate finding/coverage/push status.
- Done when the display cap does not change the push decision, incomplete analysis cannot read as secure, and artifact-write failure produces a bounded notice while retaining the real decision. - Done when the display cap does not change the push decision, incomplete analysis cannot read as secure, and artifact-write failure produces a bounded notice while retaining the real decision.
- _Verified 2026-09-13:_ manual Kiro review fallback APPROVED after dispatch thread limit; 1,222 tests passed, nine skipped. Packaged comparison/admission/report smoke and CLI boot passed; full records retained with bounded output/write failure handling. See `implementation-group-4.md`.
- _Boundary: Result and Hook Adapter_ - _Boundary: Result and Hook Adapter_
- _Requirements: 3.1, 3.2, 3.3, 5.1, 5.2, 5.5, 7.3_ - _Requirements: 3.1, 3.2, 3.3, 5.1, 5.2, 5.5, 7.3_
+10 -8
View File
@@ -46,17 +46,19 @@ engine are out of scope.
- [ ] pre-push-safety-net — Immutable pushed tips, existing-ranker integration, compatible artifact - [ ] pre-push-safety-net — Immutable pushed tips, existing-ranker integration, compatible artifact
reuse, total deadline, change attribution, actionability admission and compact hook behavior. reuse, total deadline, change attribution, actionability admission and compact hook behavior.
Dependencies: contributor-scan. Requirements/design approved 2026-09-12; tasks generated Dependencies: contributor-scan. Requirements/design approved 2026-09-12; tasks generated
(8 groups, 27 executable subtasks), independently reviewed and approved; groups 1–3 are complete (11/27 tasks), including the resolved frontend-retention prerequisite. (8 groups, 27 executable subtasks), independently reviewed and approved; groups 1–4 are complete (14/27 tasks), including the resolved frontend-retention prerequisite.
### Next concrete step ### Next concrete step
Groups 1–3 have passed independent review: runtime, inputs, scoring, snapshots, Groups 1–4 are verified: runtime, inputs, scoring, snapshots, shared deadline,
shared deadline, exact ranker scope, change-context evidence and vendor-free language ranker scope, change context, vendor-free partitions, novelty comparison, actionability
partitions. The approved contributor-scan prerequisite 2.18 (`51fabef`) repairs both admission and compact reporting. Tasks 4.1/4.2 passed independent review; 4.3 used the
JavaScript test filters; real graphs retain first-party tests and exclude vendor code. Kiro manual review fallback after the agent service reached its thread limit.
Task 3.4 also repairs unreadable-input handling and preserves named PHP census evidence. Group 5 is next: wire explicit full transaction replay, then run the first bounded
Group 4 is next: novelty comparison, actionability admission and compact advisory output. experiment. Real capabilities remain disabled pending independent evaluation.
See `pre-push-safety-net/implementation-group-3.md` for evidence and limits. Automatic dominance-only change-context projection remains an upstream limit to
retain during replay integration. See `pre-push-safety-net/implementation-group-4.md`
for verification and scope limits.
The first complete experiment is 5.2, before persistent caching and hook The first complete experiment is 5.2, before persistent caching and hook
installation work. Keep PMPro as the primary core-optimization example, then freeze the core and installation work. Keep PMPro as the primary core-optimization example, then freeze the core and
prove a changed security flow and its fixed twin on a JavaScript/Express Node API workload through prove a changed security flow and its fixed twin on a JavaScript/Express Node API workload through
+8 -7
View File
@@ -79,10 +79,11 @@ updates as a shortcut to an unmeasured latency promise.
The product direction above is user-requested. The amended requirements, design and initial The product direction above is user-requested. The amended requirements, design and initial
evaluation profile in `pre-push-safety-net` were approved for task generation on 2026-09-12 evaluation profile in `pre-push-safety-net` were approved for task generation on 2026-09-12
("ok, next kiro phase"). The maintainer then approved the task plan and starting group 1 ("ok, next kiro phase"). The maintainer then approved the task plan and starting group 1
("approve.") and then group 2 ("ok, next kiro phase, approved"). Groups 1–3 are now verified ("approve.") and then group 2 ("ok, next kiro phase, approved"). Groups 1–4 are now verified
complete (11/27 total). The maintainer approved the frontend-retention prerequisite; complete (14/27 total), including the approved frontend-retention prerequisite.
contributor-scan 2.18 and pre-push task 3.4 have both passed independent review and Group 4 adds bound novelty, operation-specific admission and compact artifacts;
fresh verification. Group 4 is next: novelty comparison, actionability admission and 4.1/4.2 passed independent review and 4.3 used the documented Kiro manual fallback
compact advisory output. Implementation is authorized under the approved plan; after agent dispatch became unavailable. Group 5 is next for full replay and the
task completion still requires independent review and fresh evidence. Hook performance first bounded experiment. Task completion still requires review and fresh evidence.
and capability admission remain unproven. Real capability eligibility, automatic dominance context projection and hook
performance remain unproven; the default capability registry is empty.
@@ -1,10 +1,10 @@
{ {
"counts": { "counts": {
"load_bearing": 100, "load_bearing": 101,
"standalone_capability": 9 "standalone_capability": 9
}, },
"modules": 109, "modules": 110,
"note": "Refreshed after pre-push-safety-net task 4.1: targeted base comparison reuses existing arbiters and shared execution budget; replay integration remains task 5.1.", "note": "Refreshed after pre-push-safety-net task 4.3: compact reporting preserves full evidence and decisions; transaction replay remains task 5.1.",
"orphaned": [], "orphaned": [],
"rows": [ "rows": [
{ {
@@ -103,6 +103,7 @@
"hunter_tools", "hunter_tools",
"model.endpoint", "model.endpoint",
"pairs", "pairs",
"push.policy",
"regress.candidate", "regress.candidate",
"regress.recipes", "regress.recipes",
"run" "run"
@@ -360,6 +361,7 @@
"preprocess", "preprocess",
"push.contracts", "push.contracts",
"push.policy", "push.policy",
"push.report",
"push.snapshot" "push.snapshot"
], ],
"module": "model", "module": "model",
@@ -404,6 +406,7 @@
"model.scan", "model.scan",
"push.contracts", "push.contracts",
"push.policy", "push.policy",
"push.report",
"push.snapshot" "push.snapshot"
], ],
"module": "model.contracts", "module": "model.contracts",
@@ -511,7 +514,8 @@
"classification": "load_bearing", "classification": "load_bearing",
"importers": [ "importers": [
"cli", "cli",
"push.policy" "push.policy",
"push.report"
], ],
"module": "model.scan", "module": "model.scan",
"reason": "contributor-scan Req 3: the repository driver -- one CPG, one budget, ranked spend" "reason": "contributor-scan Req 3: the repository driver -- one CPG, one budget, ranked spend"
@@ -534,7 +538,8 @@
"model.regions", "model.regions",
"model.report", "model.report",
"model.scan", "model.scan",
"model.taint" "model.taint",
"push.policy"
], ],
"module": "model.specs", "module": "model.specs",
"reason": "model-grounded-detection Req 7.2-7.4: the security vocabularies the CPG queries are parameterised by" "reason": "model-grounded-detection Req 7.2-7.4: the security vocabularies the CPG queries are parameterised by"
@@ -663,6 +668,8 @@
{ {
"classification": "load_bearing", "classification": "load_bearing",
"importers": [ "importers": [
"push.policy",
"push.report",
"push.snapshot" "push.snapshot"
], ],
"module": "push", "module": "push",
@@ -671,6 +678,8 @@
{ {
"classification": "load_bearing", "classification": "load_bearing",
"importers": [ "importers": [
"push.policy",
"push.report",
"push.snapshot" "push.snapshot"
], ],
"module": "push.contracts", "module": "push.contracts",
@@ -678,10 +687,18 @@
}, },
{ {
"classification": "load_bearing", "classification": "load_bearing",
"importers": [], "importers": [
"push.report"
],
"module": "push.policy", "module": "push.policy",
"reason": "pre-push-safety-net Req 2.4/3.1/3.4/4.1: targeted same-semantics base comparison; replay integration pending" "reason": "pre-push-safety-net Req 2.4/3.1/3.4/4.1: targeted same-semantics base comparison; replay integration pending"
}, },
{
"classification": "load_bearing",
"importers": [],
"module": "push.report",
"reason": "pre-push-safety-net Req 5.1/5.2/5.5/7.3: compact output and complete artifacts; runner integration pending"
},
{ {
"classification": "load_bearing", "classification": "load_bearing",
"importers": [], "importers": [],
File diff suppressed because it is too large Load Diff
+22
View File
@@ -220,3 +220,25 @@ rechecks its actual engine answers through the current policy. With no evaluated
capabilities supplied, every candidate stays diagnostic and the result separates capabilities supplied, every candidate stays diagnostic and the result separates
`none` findings, `incomplete` coverage and `allow` disposition. This offline control `none` findings, `incomplete` coverage and `allow` disposition. This offline control
executes no new graph query and does not qualify any capability for normal alerts. executes no new graph query and does not qualify any capability for normal alerts.
## Compact reporting integration
After rebuilding the image, exercise comparison → admission → artifact/report
using captured actual engine answers and explicitly synthetic eligibility controls:
```bash
docker run --rm -i --network none --entrypoint python openultrasast:dev - < ops/smoke_report.py
```
The control verifies full scan-record preservation, default diagnostic-only behavior,
a three-summary display cap that retains all five synthetic defects, and a write
failure that preserves blocking. It runs no new Joern query and admits no real
capability. The receipt records reporting duration separately from the captured
analysis timings.
Artifacts publish atomically from mode-0600 temporary files. The Linux writer runs
in a separate process bounded by the transaction deadline and its remaining report
allowance; a stalled writer is killed. A hard cancellation can leave a private
`.partial-` file, which is never the published artifact and cannot establish a
completed result. Serialization/write failure emits one notice without replacing
the finding, coverage or push decision.
+177
View File
@@ -0,0 +1,177 @@
"""Packaged comparison/admission/report integration over captured and synthetic evidence."""
from __future__ import annotations
import hashlib
import json
import tempfile
import time
from dataclasses import asdict, replace
from pathlib import Path
from openultrasast.config import PushConfig
from openultrasast.model.contracts import ChangeContext, ExecutionBudget, FamilyCoverage, QuestionOutcome, RankedQuestion, ScopeDecision
from openultrasast.model.ladder import Rung
from openultrasast.model.partitions import PartitionCoverage
from openultrasast.model.pipeline import ModelFinding
from openultrasast.model.scan import ModelScanResult
from openultrasast.push.contracts import ComparisonAnalysis, PushComparison
from openultrasast.push.policy import (
AdmissionCandidate,
CapabilityKey,
admission_push_result,
admit_candidates,
compare_evidence,
)
from openultrasast.push.report import PushReport, ScanRecord, deliver_report
def restore(data: dict) -> ModelScanResult:
values = dict(data)
values["findings"] = tuple(ModelFinding(**{**f, "rung": Rung(f["rung"])}) for f in data["findings"])
values["scope"] = ScopeDecision.from_payload(data["scope"]) if data["scope"] else None
values["question_outcomes"] = tuple(QuestionOutcome.from_payload(q) for q in data["question_outcomes"])
values["change_context"] = ChangeContext.from_payload(data["change_context"]) if data["change_context"] else None
values["family_coverage"] = tuple(FamilyCoverage.from_payload(c) for c in data["family_coverage"])
values["partitions"] = tuple(
PartitionCoverage(**{**p, **{k: tuple(p[k]) for k in ("paths", "boundaries", "excluded_paths", "unshipped_paths")}})
for p in data["partitions"]
)
values["degradations"] = tuple(data["degradations"])
values["tiers"] = tuple(tuple(t) for t in data["tiers"])
values["tier_counts"] = {int(k): v for k, v in data["tier_counts"].items()}
result = ModelScanResult(**values)
assert json.loads(json.dumps(asdict(result))) == data, "record restoration lost evidence"
return result
def main() -> None:
measured = Path("benchmarks/measurements")
raw = (measured / "2026-09-13-delta-comparison-smoke.json").read_bytes()
controls = json.loads(raw)
synthetic_bytes = (measured / "2026-09-13-admission-policy-smoke.json").read_bytes()
assert raw and synthetic_bytes
synthetic = json.loads(synthetic_bytes)["synthetic_controls"]["javascript"]
policy = Path("src/openultrasast/push/policy.py").read_bytes()
reporter = Path("src/openultrasast/push/report.py").read_bytes()
provenance = {
"repository": "captured immutable Git fixture comparisons",
"engine": controls["runtime"]["image"],
"facts": "covered by recorded comparison semantics",
"queries": "covered by recorded comparison semantics",
"policy": hashlib.sha256(policy + reporter).hexdigest(),
}
summary = {
"purpose": "Packaged reporting over captured actual engine evidence and explicitly synthetic admission controls; no new Joern run",
"input_bytes": len(raw) + len(synthetic_bytes),
"comparison_artifact_sha256": hashlib.sha256(raw).hexdigest(),
"policy_sha256": hashlib.sha256(policy).hexdigest(),
"report_sha256": hashlib.sha256(reporter).hexdigest(),
"cases": {},
}
with tempfile.TemporaryDirectory(prefix="ousast-report-smoke-") as directory:
root = Path(directory)
def emit(name: str, report: PushReport, *, fail: bool = False) -> None:
target = root / (name + ".json")
if fail:
target.mkdir()
delivery = deliver_report(report, target, execution_budget=ExecutionBudget(time.monotonic() + 10, 1.0))
assert delivery.result == report.result and delivery.exit_code == int(report.result.push_disposition == "block")
assert delivery.text.count("Notice:") <= 1
artifact = json.loads(target.read_text()) if delivery.artifact else None
if fail:
assert delivery.error and delivery.artifact is None and target.is_dir()
else:
assert delivery.error is None and artifact is not None
assert artifact["result"] == report.result.to_payload()
assert artifact["admission"] == report.admission.to_payload()
assert artifact["scans"] == json.loads(json.dumps([asdict(s) for s in report.scans]))
summary["cases"][name] = {
"exit_code": delivery.exit_code,
"text": delivery.text,
"error": delivery.error,
"reporting_seconds": delivery.reporting_seconds,
"artifact": artifact,
}
for name, record in controls["cases"].items():
evidence = record["evidence"]
head = restore(evidence["head"])
base = restore(evidence["delta"]["base_scan"])
context = ChangeContext.from_payload(evidence["context"])
comparison = PushComparison.from_payload(evidence["comparison"])
semantics = evidence["delta"]["semantics"]
deltas = compare_evidence(head, base, context=context, head_semantics=semantics, base_semantics=semantics)
assert deltas and all(d.novelty == evidence["expected_novelty"] for d in deltas)
candidates = []
for delta in deltas:
op = delta.head_operation
assert op is not None
key = CapabilityKey(op.question.language, "unspecified", "unspecified", delta.family, op.mechanism, "unreviewed", semantics)
candidates.append(AdmissionCandidate(delta, key, comparison))
admission = admit_candidates(candidates)
assert not admission.defects and head.scope is not None
analysis = ComparisonAnalysis(
comparison, (head.scope,), tuple(q.identity for q in head.question_outcomes), "complete_within_scope"
)
result = admission_push_result(admission, (analysis,))
assert result.coverage_status == "incomplete"
report = PushReport(
result,
admission,
(ScanRecord(comparison, "head", head), ScanRecord(comparison, "base", base)),
{**provenance, "comparison_semantics": semantics},
{"total_seconds": record["completion"]["elapsed_seconds"]},
)
emit(name, report)
# These five normalized controls test display/decision separation only.
from openultrasast.push.policy import CandidateDisposition
disposition = CandidateDisposition.from_payload(synthetic["admission"]["dispositions"][0])
cap = disposition.capability_evaluations[0]
candidate = disposition.candidate
candidates = []
questions = []
findings = []
outcomes = []
for i in range(5):
old = candidate.delta.head_operation
assert old is not None
q = replace(old.question, path=f"handler{i}.src")
op = replace(old, question=q, path=q.path)
delta = replace(
candidate.delta,
defect_id=f"synthetic-defect-{i}",
head_operation=op,
site=f"{q.path}:{op.line}:handler",
change_evidence=(f"head:{q.path}:2-2",),
)
candidates.append(replace(candidate, delta=delta))
questions.append(q)
findings.append(ModelFinding(delta.site, delta.family, Rung(delta.rung), delta.witness or "synthetic witness"))
outcomes.append(QuestionOutcome(q, "completed", "synthetic-normalized-control", "[]"))
scope = ScopeDecision("synthetic-ranker", "static", True, tuple(RankedQuestion(q, 1.0, ()) for q in questions), (), ())
scan = ModelScanResult(findings=tuple(findings), scope=scope, question_outcomes=tuple(outcomes))
base_scan = replace(scan, findings=())
analysis = ComparisonAnalysis(candidate.comparison, (scope,), tuple(questions), "complete_within_scope")
admission = admit_candidates(candidates, capabilities=(cap,))
assert len(admission.defects) == 5
result = admission_push_result(admission, (analysis,), config=PushConfig(mode="blocking"))
report = PushReport(
result,
admission,
(ScanRecord(candidate.comparison, "head", scan), ScanRecord(candidate.comparison, "base", base_scan)),
{**provenance, "repository": "synthetic normalized policy controls", "engine": "synthetic; no engine execution"},
{"total_seconds": 0.0},
)
emit("synthetic-display-cap", report)
text = summary["cases"]["synthetic-display-cap"]["text"]
assert text.count("- injection at ") == 3 and "2 more" in text
emit("synthetic-write-failure", report, fail=True)
print(json.dumps(summary, indent=2))
if __name__ == "__main__":
main()
+1
View File
@@ -28,6 +28,7 @@ ROOTS = ("gate", "map_gate", "pair_gate", "cli", "__main__")
# Modules that stand alone: their own CLI subcommand or external protocol, deliberately off the scan path. # Modules that stand alone: their own CLI subcommand or external protocol, deliberately off the scan path.
SPEC_OWNED = { SPEC_OWNED = {
"push.report": "pre-push-safety-net Req 5.1/5.2/5.5/7.3: compact output and complete artifacts; runner integration pending",
"push.policy": "pre-push-safety-net Req 2.4/3.1/3.4/4.1: targeted same-semantics base comparison; replay integration pending", "push.policy": "pre-push-safety-net Req 2.4/3.1/3.4/4.1: targeted same-semantics base comparison; replay integration pending",
"push.snapshot": "pre-push-safety-net Req 1.1/1.3/1.4/7.1: local immutable ref resolution; runner integration pending", "push.snapshot": "pre-push-safety-net Req 1.1/1.3/1.4/7.1: local immutable ref resolution; runner integration pending",
"contracts": "pre-push-safety-net Req 7.3: strict shared manifest serialization, independent of push", "contracts": "pre-push-safety-net Req 7.3: strict shared manifest serialization, independent of push",
+242
View File
@@ -0,0 +1,242 @@
"""Compact presentation and complete artifacts for an already-decided push."""
from __future__ import annotations
import json
import math
import multiprocessing
import os
import time
import uuid
from collections.abc import Mapping
from dataclasses import asdict, dataclass
from multiprocessing.connection import Connection
from pathlib import Path
from types import MappingProxyType
from typing import Literal
from openultrasast.model.contracts import ExecutionBudget
from openultrasast.model.scan import ModelScanResult
from openultrasast.push.contracts import PushComparison, PushResolution, PushResult
from openultrasast.push.policy import AdmissionResult, _context_boundaries
@dataclass(frozen=True)
class ScanRecord:
comparison: PushComparison
side: Literal["base", "head"]
scan: ModelScanResult
@dataclass(frozen=True)
class PushReport:
result: PushResult
admission: AdmissionResult
scans: tuple[ScanRecord, ...]
provenance: Mapping[str, str]
timings: Mapping[str, float]
resolution: PushResolution | None = None
def __post_init__(self) -> None:
object.__setattr__(self, "provenance", MappingProxyType(dict(self.provenance)))
object.__setattr__(self, "timings", MappingProxyType(dict(self.timings)))
ids = tuple(defect.defect_id for defect in self.admission.defects)
if ids != self.result.actionable_defect_ids:
raise ValueError("report must retain the exact admitted defect set and order")
if any(
not all((d.witnesses, d.locations, d.consequences, d.repairs, d.change_evidence, d.comparisons)) for d in self.admission.defects
):
raise ValueError("actionable summaries require witnesses, locations, consequence, repair and change evidence")
admitted = {d.candidate.delta.defect_id for d in self.admission.dispositions if d.admitted}
if admitted != set(ids):
raise ValueError("report must retain every admitted candidate disposition")
comparisons = {analysis.comparison for analysis in self.result.analyses}
if self.result.coverage_status == "complete_within_scope" and self.admission.coverage_reasons:
raise ValueError("complete comparison cannot hide admission coverage gaps")
if any(record.comparison not in comparisons or record.side not in ("base", "head") for record in self.scans):
raise ValueError("scan evidence must belong to an exact reported comparison and side")
for analysis in self.result.analyses:
heads = [record.scan for record in self.scans if record.comparison == analysis.comparison and record.side == "head"]
if any(scope not in [scan.scope for scan in heads] for scope in analysis.scopes):
raise ValueError("reported scope requires the actual head scan evidence")
completed = {outcome.identity for scan in heads for outcome in scan.question_outcomes if outcome.status == "completed"}
if not set(analysis.completed_questions) <= completed:
raise ValueError("completed questions require completed recorded outcomes")
if analysis.coverage_status == "complete_within_scope":
bases = [record.scan for record in self.scans if record.comparison == analysis.comparison and record.side == "base"]
if not bases or any(not _completed_scan(scan) for scan in (*heads, *bases)):
raise ValueError("complete comparison requires completed head and base evidence")
required = {"repository", "engine", "facts", "queries", "policy"}
if not required <= self.provenance.keys() or any(not isinstance(v, str) or not v.strip() for v in self.provenance.values()):
raise ValueError("report requires repository, engine, facts, queries and policy provenance")
if "total_seconds" not in self.timings or any(
type(value) not in (int, float) or not math.isfinite(value) or value < 0 for value in self.timings.values()
):
raise ValueError("report requires finite nonnegative timings including total_seconds")
@dataclass(frozen=True)
class ReportDelivery:
result: PushResult
text: str
artifact: Path | None
error: str | None
reporting_seconds: float = 0.0
@property
def exit_code(self) -> int:
return int(self.result.push_disposition == "block")
def _completed_scan(scan: ModelScanResult) -> bool:
return bool(
scan.scope
and scan.scope.population_complete
and not scan.scope.deferred
and not _context_boundaries(scan)
and not scan.degradations
and {q.identity for q in scan.scope.selected} == {q.identity for q in scan.question_outcomes if q.status == "completed"}
and all(q.status == "completed" for q in scan.question_outcomes)
)
def _write_artifact(report: PushReport, target: Path, temporary: Path, connection: Connection) -> None:
"""Only this bounded child performs serialization or filesystem operations."""
try:
target.parent.mkdir(parents=True, exist_ok=True)
fd = os.open(temporary, os.O_CREAT | os.O_EXCL | os.O_WRONLY | os.O_NOFOLLOW, 0o600)
try:
with os.fdopen(fd, "w", encoding="utf-8") as stream:
payload = {
"schema_version": 1,
"result": report.result.to_payload(),
"admission": report.admission.to_payload(),
"scans": [asdict(record) for record in report.scans],
"provenance": dict(report.provenance),
"timings": dict(report.timings),
"timing_scope": "Caller stage measurements; publication and rendering duration is returned in ReportDelivery.",
"resolution": report.resolution.to_payload() if report.resolution else None,
}
json.dump(payload, stream, ensure_ascii=True, allow_nan=False, indent=2)
stream.write("\n")
stream.flush()
os.fsync(stream.fileno())
os.replace(temporary, target)
finally:
temporary.unlink(missing_ok=True)
connection.send(None)
except Exception as error:
connection.send("artifact_write_failed:" + type(error).__name__)
finally:
connection.close()
def _save(report: PushReport, target: Path, budget: ExecutionBudget) -> str | None:
# Persist completed evidence during the remaining cancellation/report allowance.
# Never restart that allowance for a late caller.
end = budget.deadline_monotonic + budget.cancellation_allowance_seconds
if time.monotonic() >= end:
return "report_deadline_exhausted"
reserve = min(0.1, budget.cancellation_allowance_seconds / 2)
reader = writer = process = None
try:
temporary = target.with_name("." + target.name + ".partial-" + uuid.uuid4().hex)
context = multiprocessing.get_context("fork")
reader, writer = context.Pipe(duplex=False)
process = context.Process(target=_write_artifact, args=(report, target, temporary, writer), daemon=True)
process.start()
writer.close()
process.join(max(0.0, end - time.monotonic() - reserve))
if process.is_alive():
process.kill()
process.join(max(0.0, end - time.monotonic()))
return "report_deadline_exhausted"
if process.exitcode != 0 or not reader.poll():
return "artifact_writer_failed"
error = reader.recv()
return None if error is None else error if isinstance(error, str) else "artifact_writer_failed"
except (OSError, ValueError, EOFError, RuntimeError):
return "artifact_writer_unavailable"
finally:
if writer is not None:
writer.close()
if reader is not None:
reader.close()
if process is not None:
if process.is_alive():
process.kill()
process.join(max(0.0, end - time.monotonic()))
if not process.is_alive():
process.close()
def _short(value: str, limit: int = 180) -> str:
# Bound before escaping: repository-controlled names/witnesses cannot add lines
# or terminal escapes. Full spelling and every witness remain in the artifact.
clipped = value if len(value) <= limit else value[: limit // 2] + "..." + value[-limit // 2 :]
return json.dumps(clipped, ensure_ascii=True)[1:-1]
def render_report(report: PushReport, *, artifact: Path | None, error: str | None = None) -> str:
lines: list[str] = []
defects = report.admission.defects
for defect in defects[:3]:
path, line = defect.locations[0]
delta = next(
disposition.candidate.delta
for disposition in report.admission.dispositions
if disposition.admitted and disposition.candidate.delta.defect_id == defect.defect_id
)
change = (
"A protection present in the base is missing here."
if delta.reason == "discharge_removed"
else "This change connects input to the operation."
)
lines.extend(
(
f"- {_short(defect.family)} at {_short(path, 4096)}:{line}: {_short(defect.consequences[0])}",
f" Evidence: {_short(defect.witnesses[0])}; Change: {change} ({_short(defect.change_evidence[0])})",
f" Fix: {_short(defect.repairs[0])}",
)
)
if len(defects) > 3:
lines.append(f"{len(defects) - 3} more actionable defects retained in the detailed result.")
if not defects:
coverage = report.result.coverage_status
if coverage == "complete_within_scope":
completed = sum(len(analysis.completed_questions) for analysis in report.result.analyses)
lines.append(f"No actionable defects in {completed} completed checks within the recorded scope.")
elif coverage == "not_applicable":
lines.append("No applicable code updates to check.")
elif coverage == "unavailable":
lines.append("No analysis result is available.")
else:
lines.append("No actionable defects reported; analysis is incomplete.")
notices = []
if report.result.coverage_status in ("incomplete", "unavailable"):
reasons = report.admission.coverage_reasons
if any("deadline" in reason for reason in reasons):
notices.append("Analysis did not finish. Rerun with a longer analysis deadline.")
elif any("capability" in reason for reason in reasons):
notices.append("Some checks are not enabled for normal alerts. Review capability coverage in the details.")
else:
notices.append(
"Some checks could not be completed. Review the comparison details and resolve the recorded gaps before retrying."
)
if error:
notices.append("Details could not be saved. Retry with a writable artifact path and available reporting time.")
if notices:
lines.append("Notice: " + " ".join(notices))
if artifact is not None:
lines.append("Details: " + _short(str(artifact), 4096))
if report.result.push_disposition == "block":
lines.append("Push blocked by the configured policy.")
return "\n".join(lines) + "\n"
def deliver_report(report: PushReport, artifact: Path, *, execution_budget: ExecutionBudget) -> ReportDelivery:
started = time.monotonic()
error = _save(report, artifact, execution_budget)
saved = artifact if error is None else None
text = render_report(report, artifact=saved, error=error)
return ReportDelivery(report.result, text, saved, error, time.monotonic() - started)
+182
View File
@@ -0,0 +1,182 @@
"""Presentation bounds cannot alter admission, evidence, or enforcement."""
import json
import os
import time
from dataclasses import replace
import pytest
from test_push_admission import control
from openultrasast.config import PushConfig
from openultrasast.model.contracts import ExecutionBudget, QuestionOutcome
from openultrasast.model.scan import ModelScanResult
from openultrasast.push.policy import admission_push_result, admit_candidates
from openultrasast.push.report import PushReport, ScanRecord, deliver_report
def report_with(count=1, *, admitted=True, blocking=False):
candidate, capability, analysis = control()
candidates = tuple(replace(candidate, delta=replace(candidate.delta, defect_id=f"defect-{i}")) for i in range(count))
admission = admit_candidates(candidates, capabilities=(capability,) if admitted else ())
result = admission_push_result(admission, (analysis,), config=PushConfig(mode="blocking" if blocking else "advisory"))
q = analysis.completed_questions[0]
scan = ModelScanResult(scope=analysis.scopes[0], question_outcomes=(QuestionOutcome(q, "completed", "answered", "[]"),))
records = (ScanRecord(analysis.comparison, "head", scan), ScanRecord(analysis.comparison, "base", scan))
provenance = {key: "synthetic-control" for key in ("repository", "engine", "facts", "queries", "policy")}
return PushReport(result, admission, records, provenance, {"total_seconds": 0.1})
def budget(seconds=3.0):
return ExecutionBudget(time.monotonic() + seconds, 0.2)
def test_display_cap_preserves_all_decisions_and_complete_artifact(tmp_path):
report = report_with(5, blocking=True)
delivered = deliver_report(report, tmp_path / "run.json", execution_budget=budget())
assert delivered.exit_code == 1 and delivered.result == report.result
assert delivered.text.count("- injection at ") == 3 and "2 more" in delivered.text
assert "Evidence:" in delivered.text and "Change:" in delivered.text and "Fix:" in delivered.text
data = json.loads(delivered.artifact.read_text())
assert len(data["admission"]["defects"]) == 5 and len(data["admission"]["dispositions"]) == 5
assert data["result"]["push_disposition"] == "block"
assert data["scans"][0]["scan"]["question_outcomes"][0]["raw_rows_json"] == "[]"
assert data["scans"][0]["scan"]["scope"]["selected"][0]["identity"]["path"] == "handler.src"
assert data["provenance"] == report.provenance and data["timings"] == report.timings
assert os.stat(delivered.artifact).st_mode & 0o777 == 0o600
def test_one_incomplete_notice_and_one_status_without_claiming_secure(tmp_path):
report = report_with(8, admitted=False)
delivered = deliver_report(report, tmp_path / "run.json", execution_budget=budget())
assert delivered.exit_code == 0 and delivered.result.coverage_status == "incomplete"
assert delivered.text.count("Notice:") == 1
assert delivered.text.count("No actionable defects") == 1
assert "Review" in delivered.text and str(delivered.artifact) in delivered.text
assert "secure" not in delivered.text.lower() and "clean" not in delivered.text.lower()
def test_complete_empty_check_is_bounded_by_scope(tmp_path):
delivered = deliver_report(report_with(0), tmp_path / "run.json", execution_budget=budget())
assert "1 completed" in delivered.text and "within" in delivered.text
assert "Notice:" not in delivered.text and "secure" not in delivered.text.lower()
def test_write_failure_retains_existing_artifact_and_block_decision(tmp_path):
target = tmp_path / "directory"
target.mkdir()
delivered = deliver_report(report_with(blocking=True), target, execution_budget=budget())
assert delivered.exit_code == 1 and delivered.artifact is None and delivered.error
assert delivered.text.count("Notice:") == 1 and "writable" in delivered.text
assert target.is_dir()
def test_invalid_serializable_detail_fails_without_losing_decision(tmp_path):
report = report_with(blocking=True)
broken = replace(report.scans[0], scan=replace(report.scans[0].scan, by_rung={"invalid_metadata": object()}))
report = replace(report, scans=(broken, report.scans[1]))
delivered = deliver_report(report, tmp_path / "run.json", execution_budget=budget())
assert delivered.exit_code == 1 and delivered.artifact is None and delivered.error
def test_mismatched_result_or_missing_evidence_is_rejected():
report = report_with()
with pytest.raises(ValueError):
replace(report, admission=admit_candidates(()))
with pytest.raises(ValueError):
replace(report, scans=())
with pytest.raises(ValueError):
replace(report, provenance={"engine": "only"})
with pytest.raises(ValueError):
replace(report, timings={"total_seconds": float("nan")})
def test_repository_control_characters_cannot_inject_terminal_notices(tmp_path):
report = report_with()
defect = replace(report.admission.defects[0], witnesses=("input\nNotice: fake\x1b[31m",))
report = replace(report, admission=replace(report.admission, defects=(defect,)))
delivered = deliver_report(report, tmp_path / "run.json", execution_budget=budget())
assert "\nNotice: fake" not in delivered.text and "\x1b" not in delivered.text
assert json.loads(delivered.artifact.read_text())["admission"]["defects"][0]["witnesses"] == ["input\nNotice: fake\x1b[31m"]
def test_expired_reporting_allowance_still_returns_real_decision(tmp_path):
delivered = deliver_report(report_with(blocking=True), tmp_path / "run.json", execution_budget=ExecutionBudget(0.0, 0.1))
assert delivered.exit_code == 1 and delivered.error == "report_deadline_exhausted"
assert delivered.text.count("Notice:") == 1
def test_stalled_writer_is_cancelled_without_overwriting_previous_artifact(tmp_path, monkeypatch):
import multiprocessing
import openultrasast.push.report as module
def stall(*args):
time.sleep(10)
target = tmp_path / "run.json"
target.write_text("previous-complete-artifact")
children = {p.pid for p in multiprocessing.active_children()}
monkeypatch.setattr(module, "_write_artifact", stall)
started = time.monotonic()
delivered = deliver_report(report_with(blocking=True), target, execution_budget=budget(0.05))
assert time.monotonic() - started < 0.6
assert delivered.exit_code == 1 and delivered.error == "report_deadline_exhausted"
assert target.read_text() == "previous-complete-artifact"
assert {p.pid for p in multiprocessing.active_children()} == children
def test_long_terminal_values_are_bounded_but_artifact_keeps_full_evidence(tmp_path):
report = report_with()
witness = "request.input->" + "x" * 100_000
defect = replace(report.admission.defects[0], witnesses=(witness,))
report = replace(report, admission=replace(report.admission, defects=(defect,)))
delivered = deliver_report(report, tmp_path / "run.json", execution_budget=budget())
assert len(delivered.text) < 3000 and witness not in delivered.text
assert json.loads(delivered.artifact.read_text())["admission"]["defects"][0]["witnesses"] == [witness]
def test_unavailable_and_not_applicable_never_read_as_completed(tmp_path):
from openultrasast.push.contracts import PushResult
from openultrasast.push.policy import AdmissionResult
for coverage, expected_notice in [("unavailable", 1), ("not_applicable", 0)]:
report = report_with(0)
result = PushResult((), "none", coverage, "allow", ())
report = replace(report, result=result, admission=AdmissionResult((), (), ()), scans=())
delivered = deliver_report(report, tmp_path / (coverage + ".json"), execution_budget=budget())
assert "completed checks" not in delivered.text and delivered.text.count("Notice:") == expected_notice
assert delivered.result.coverage_status == coverage
def test_provenance_and_timings_are_frozen_after_construction():
report = report_with()
with pytest.raises(TypeError):
report.provenance["engine"] = "different"
with pytest.raises(TypeError):
report.timings["total_seconds"] = 0.0
def test_writer_exiting_zero_without_receipt_is_not_success(tmp_path, monkeypatch):
import openultrasast.push.report as module
def empty_writer(report, target, temporary, connection):
connection.close()
monkeypatch.setattr(module, "_write_artifact", empty_writer)
delivered = deliver_report(report_with(blocking=True), tmp_path / "run.json", execution_budget=budget())
assert delivered.artifact is None and delivered.error and delivered.exit_code == 1
def test_invalid_artifact_destination_is_a_notice_not_an_exception():
from pathlib import Path
delivered = deliver_report(report_with(blocking=True), Path("/"), execution_budget=budget())
assert delivered.artifact is None and delivered.error and delivered.exit_code == 1
def test_failed_base_evidence_cannot_be_presented_as_complete():
report = report_with(0)
failed = replace(report.scans[1], scan=ModelScanResult(degradations=({"reason": "cpg_build_failed"},)))
with pytest.raises(ValueError, match="complete comparison"):
replace(report, scans=(report.scans[0], failed))