mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
feat(pre-push-safety-net): render bounded reports and preserve full evidence
This commit is contained in:
@@ -1,6 +1,6 @@
|
|||||||
# Brief: pre-push-safety-net
|
# Brief: pre-push-safety-net
|
||||||
|
|
||||||
Status: requirements, design and tasks approved on 2026-09-12; groups 1–3 verified complete (11/27 subtasks), including the approved contributor-scan frontend-retention prerequisite; groups 4–8 pending. Language: English.
|
Status: requirements, design and tasks approved; groups 1–4 verified complete (14/27 subtasks). Groups 5–8 pending; real capability eligibility and hook latency remain unproven. Language: English.
|
||||||
|
|
||||||
## Problem
|
## Problem
|
||||||
|
|
||||||
|
|||||||
@@ -4,9 +4,10 @@ Date: 2026-09-13. The maintainer authorized group 4 as a whole. Scope is tasks
|
|||||||
4.1–4.3: targeted novelty comparison, actionability admission and compact reporting.
|
4.1–4.3: targeted novelty comparison, actionability admission and compact reporting.
|
||||||
Groups 1–3 are verified at `6ba2b06`; full transaction replay remains task 5.1.
|
Groups 1–3 are verified at `6ba2b06`; full transaction replay remains task 5.1.
|
||||||
|
|
||||||
Status: implementation in progress. Task completion requires independent review
|
Status: tasks 4.1–4.3 are VERIFIED (TASK scope), bringing the spec to 14/27 tasks.
|
||||||
and fresh evidence. No production capability is enabled for normal hook alerts;
|
Tasks 4.1 and 4.2 passed independent review. Task 4.3 used the Kiro manual review
|
||||||
capability admission still requires task 8.3 evaluation.
|
fallback after the agent service rejected further dispatch. No production capability
|
||||||
|
is enabled for normal hook alerts; actual admission still requires task 8.3 evaluation.
|
||||||
|
|
||||||
## Verification boundary
|
## Verification boundary
|
||||||
|
|
||||||
@@ -62,3 +63,64 @@ default registry. Explicitly synthetic PHP/WordPress and JavaScript/Express cont
|
|||||||
receive equivalent admission and enforcement behavior. They do not qualify any
|
receive equivalent admission and enforcement behavior. They do not qualify any
|
||||||
real capability; task 8.3 still owns actual eligibility. Task 4.2 brings progress to
|
real capability; task 8.3 still owns actual eligibility. Task 4.2 brings progress to
|
||||||
13/27 tasks. Compact reporting remains in this authorized group.
|
13/27 tasks. Compact reporting remains in this authorized group.
|
||||||
|
|
||||||
|
## Task 4.3 verification
|
||||||
|
|
||||||
|
The agent service rejected the task 4.3 dispatch with `agent thread limit reached`.
|
||||||
|
The [kiro-impl skill](../../../.opencode/skills/kiro-impl/SKILL.md) explicitly provides:
|
||||||
|
“If multi-agent is not available, fall back to manual mode execution for all tasks.”
|
||||||
|
Implementation, review and completion checks therefore ran in the main context.
|
||||||
|
This is a manual review record, not an independent-review claim.
|
||||||
|
|
||||||
|
OFF controls produced seven relevant failures; ON and unflagged reporting tests
|
||||||
|
passed. Manual review found and repaired an invalid artifact path escaping the
|
||||||
|
notice path and a complete comparison carrying failed base evidence. A writer
|
||||||
|
exiting zero without a receipt is rejected. Stalled-writer tests prove cancellation,
|
||||||
|
preservation of an existing artifact and no surviving child. Other controls cover
|
||||||
|
all candidate/scan preservation, five defects with only three displayed, separate
|
||||||
|
coverage/enforcement, escaping and bounded long text, and immutable provenance.
|
||||||
|
|
||||||
|
## Review Verdict
|
||||||
|
- VERDICT: APPROVED
|
||||||
|
- TASK: 4.3
|
||||||
|
- REVIEW_MODE: Kiro manual fallback; agent dispatch unavailable.
|
||||||
|
- MECHANICAL_RESULTS: final full suite 1,222 passed, nine skipped in 102.39 seconds;
|
||||||
|
Ruff and format (238 files), mypy (111 source files), compileall, diff checks and
|
||||||
|
detection/map/pair gates PASS. No new placeholders or hardcoded secrets.
|
||||||
|
- FINDINGS: Invalid destination and failed-base completeness issues repaired and
|
||||||
|
verified with failing controls followed by passing tests. No outstanding task finding.
|
||||||
|
- BOUNDARY: Reporting consumes existing admission and engine records; no new
|
||||||
|
detector, ranker, graph mutation, provider or installed hook.
|
||||||
|
|
||||||
|
The image build and CLI boot passed. The packaged integration control rechecked
|
||||||
|
captured actual PHP/JavaScript answers through current comparison/admission/reporting,
|
||||||
|
preserving complete scan payloads and default diagnostic-only output. Five explicitly
|
||||||
|
synthetic admitted defects retain all five records, display three summaries and keep
|
||||||
|
the block decision. Artifact write failure also retains that decision. Packaged
|
||||||
|
policy/reporter hashes match the workspace. See
|
||||||
|
`benchmarks/measurements/2026-09-13-compact-report-smoke.json`.
|
||||||
|
|
||||||
|
Publication is atomic from mode-0600 temporary files. A separate Linux process
|
||||||
|
bounds serialization and filesystem work by the remaining transaction/report
|
||||||
|
allowance. Hard cancellation can leave a private unpublished partial file; it is
|
||||||
|
never presented as the completed artifact. Caller-supplied stage timings are
|
||||||
|
preserved, and the delivery receipt measures publication/rendering separately.
|
||||||
|
Observed packaged delivery costs (about 0.02–0.04 seconds for these controls) do not
|
||||||
|
establish whole-hook latency.
|
||||||
|
|
||||||
|
## Group integration validation
|
||||||
|
|
||||||
|
- DECISION: GO
|
||||||
|
- VALIDATION_SCOPE: Tasks 4.1–4.3 only; no full-feature or release admission.
|
||||||
|
- CROSS_TASK_CONTRACTS: Actual captured engine evidence → bound delta → empty-registry
|
||||||
|
diagnostics → compact report and complete artifact PASS. Explicitly synthetic
|
||||||
|
calibrated controls → admission → advisory/blocking → capped rendering PASS.
|
||||||
|
- REQUIREMENTS: Group-owned portions of 2.4, 3.1–3.5, 4.1, 5.1–5.5, 7.3 and 8.1 are
|
||||||
|
covered by the implementation and controls above. Full transaction wiring and
|
||||||
|
representative capability/latency evaluation remain later tasks.
|
||||||
|
- DESIGN_ALIGNMENT: Existing arbiters and ranker retained; policy and report live
|
||||||
|
in their planned push-adapter boundary. No orphaned module or upstream detector change.
|
||||||
|
- BLOCKED_TASKS: None in group 4. Automatic dominance change-context projection is
|
||||||
|
a documented upstream integration limit, not erased by these declared controls.
|
||||||
|
- NEXT: Group 5 full replay and first bounded experiment. Complete that experiment
|
||||||
|
before persistent graph reuse in group 6. Groups 5–8 remain pending.
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
{
|
{
|
||||||
"feature_name": "pre-push-safety-net",
|
"feature_name": "pre-push-safety-net",
|
||||||
"created_at": "2026-09-12T15:35:00Z",
|
"created_at": "2026-09-12T15:35:00Z",
|
||||||
"updated_at": "2026-09-13T10:26:47.307154+00:00",
|
"updated_at": "2026-09-13T11:21:07.667199+00:00",
|
||||||
"language": "en",
|
"language": "en",
|
||||||
"phase": "implementation",
|
"phase": "implementation",
|
||||||
"approvals": {
|
"approvals": {
|
||||||
@@ -19,7 +19,7 @@
|
|||||||
}
|
}
|
||||||
},
|
},
|
||||||
"ready_for_implementation": true,
|
"ready_for_implementation": true,
|
||||||
"implementation_approval_note": "Groups 1\u20133 verified complete. Maintainer explicitly authorized group 4 as a whole; each task requires independent review and fresh verification.",
|
"implementation_approval_note": "Groups 1\u20134 verified complete. Tasks 4.1/4.2 independently approved; 4.3 used the documented Kiro manual review fallback after agent thread limit. Remaining tasks require review and fresh verification.",
|
||||||
"depends_on": [
|
"depends_on": [
|
||||||
"contributor-scan"
|
"contributor-scan"
|
||||||
],
|
],
|
||||||
@@ -85,9 +85,10 @@
|
|||||||
"3.3",
|
"3.3",
|
||||||
"3.4",
|
"3.4",
|
||||||
"4.1",
|
"4.1",
|
||||||
"4.2"
|
"4.2",
|
||||||
|
"4.3"
|
||||||
],
|
],
|
||||||
"remaining_tasks": 14,
|
"remaining_tasks": 13,
|
||||||
"verification_record": "implementation-group-4.md",
|
"verification_record": "implementation-group-4.md",
|
||||||
"verification_records": [
|
"verification_records": [
|
||||||
"implementation-group-1.md",
|
"implementation-group-1.md",
|
||||||
@@ -95,7 +96,7 @@
|
|||||||
"implementation-group-3.md",
|
"implementation-group-3.md",
|
||||||
"implementation-group-4.md"
|
"implementation-group-4.md"
|
||||||
],
|
],
|
||||||
"next_group": 4,
|
"next_group": 5,
|
||||||
"blocked_tasks": []
|
"blocked_tasks": []
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,8 +1,9 @@
|
|||||||
# Implementation Plan: pre-push-safety-net
|
# Implementation Plan: pre-push-safety-net
|
||||||
|
|
||||||
Status: task plan approved by the maintainer's "approve." on 2026-09-12. Groups 1–3 (tasks 1.1–3.4)
|
Status: task plan approved by the maintainer on 2026-09-12. Groups 1–4 (tasks 1.1–4.3)
|
||||||
are complete after independent review and fresh verification (13/27 subtasks, including tasks 4.1–4.2). Task 4.3 and groups 5–8 remain pending;
|
are complete after task reviews and fresh verification (14/27 subtasks). Task 4.3 used
|
||||||
the hook and capability admission are not yet implemented. See `implementation-group-1.md` and `implementation-group-2.md`.
|
the documented manual review fallback after agent dispatch became unavailable.
|
||||||
|
Groups 5–8 remain pending; no production capability or hook latency is admitted.
|
||||||
|
|
||||||
## Execution contract
|
## Execution contract
|
||||||
|
|
||||||
@@ -112,7 +113,7 @@ the hook and capability admission are not yet implemented. See `implementation-g
|
|||||||
- _Requirements: 2.2, 2.3, 7.3, 8.2, 8.4_
|
- _Requirements: 2.2, 2.3, 7.3, 8.2, 8.4_
|
||||||
- _Verified 2026-09-13:_ independent review APPROVED; 1,151 passed, nine skipped; Ruff/format/mypy pass. Real PHP/JavaScript graphs retain tests/browser source, omit vendor nodes and share one ranker limit. Named census survives PHP aggregation; missing source names and unreadable discovery inputs remain explicit. Python exercises all three batch census schemas. Evidence: `benchmarks/measurements/2026-09-13-partition-scope-smoke.json`. No hook capability or latency claim.
|
- _Verified 2026-09-13:_ independent review APPROVED; 1,151 passed, nine skipped; Ruff/format/mypy pass. Real PHP/JavaScript graphs retain tests/browser source, omit vendor nodes and share one ranker limit. Named census survives PHP aggregation; missing source names and unreadable discovery inputs remain explicit. Python exercises all three batch census schemas. Evidence: `benchmarks/measurements/2026-09-13-partition-scope-smoke.json`. No hook capability or latency claim.
|
||||||
|
|
||||||
- [ ] 4. Implement change attribution and actionable output
|
- [x] 4. Implement change attribution and actionable output
|
||||||
- [x] 4.1 Compare targeted base/head evidence for defect novelty
|
- [x] 4.1 Compare targeted base/head evidence for defect novelty
|
||||||
- Establish stable mechanism/operation identity with rename/span mapping and collect comparable base evidence under the same semantics and remaining deadline.
|
- Establish stable mechanism/operation identity with rename/span mapping and collect comparable base evidence under the same semantics and remaining deadline.
|
||||||
- Distinguish new, worsened, unchanged and unknown; an absent finding from a failed/truncated base is unknown. Support removed guards and new source connections at unchanged sinks.
|
- Distinguish new, worsened, unchanged and unknown; an absent finding from a failed/truncated base is unknown. Support removed guards and new source connections at unchanged sinks.
|
||||||
@@ -130,10 +131,11 @@ the hook and capability admission are not yet implemented. See `implementation-g
|
|||||||
- _Verified 2026-09-13:_ independent review APPROVED after operation/context-binding repair; 1,207 tests passed, nine skipped; default real-evidence controls remain diagnostic and synthetic PHP/JavaScript admission is equivalent. See `implementation-group-4.md`.
|
- _Verified 2026-09-13:_ independent review APPROVED after operation/context-binding repair; 1,207 tests passed, nine skipped; default real-evidence controls remain diagnostic and synthetic PHP/JavaScript admission is equivalent. See `implementation-group-4.md`.
|
||||||
- _Boundary: Delta and Actionability Policy_
|
- _Boundary: Delta and Actionability Policy_
|
||||||
- _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5, 5.3, 5.4, 8.1_
|
- _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5, 5.3, 5.4, 8.1_
|
||||||
- [ ] 4.3 Render compact results and preserve complete artifacts
|
- [x] 4.3 Render compact results and preserve complete artifacts
|
||||||
- Render up to three actionable summaries plus remaining count; emit at most one no-finding status and one aggregated incomplete notice with a useful next action.
|
- Render up to three actionable summaries plus remaining count; emit at most one no-finding status and one aggregated incomplete notice with a useful next action.
|
||||||
- Persist all defects, candidate dispositions, exact scope, identities, capability/ranker provenance and timings with separate finding/coverage/push status.
|
- Persist all defects, candidate dispositions, exact scope, identities, capability/ranker provenance and timings with separate finding/coverage/push status.
|
||||||
- Done when the display cap does not change the push decision, incomplete analysis cannot read as secure, and artifact-write failure produces a bounded notice while retaining the real decision.
|
- Done when the display cap does not change the push decision, incomplete analysis cannot read as secure, and artifact-write failure produces a bounded notice while retaining the real decision.
|
||||||
|
- _Verified 2026-09-13:_ manual Kiro review fallback APPROVED after dispatch thread limit; 1,222 tests passed, nine skipped. Packaged comparison/admission/report smoke and CLI boot passed; full records retained with bounded output/write failure handling. See `implementation-group-4.md`.
|
||||||
- _Boundary: Result and Hook Adapter_
|
- _Boundary: Result and Hook Adapter_
|
||||||
- _Requirements: 3.1, 3.2, 3.3, 5.1, 5.2, 5.5, 7.3_
|
- _Requirements: 3.1, 3.2, 3.3, 5.1, 5.2, 5.5, 7.3_
|
||||||
|
|
||||||
|
|||||||
@@ -46,17 +46,19 @@ engine are out of scope.
|
|||||||
- [ ] pre-push-safety-net — Immutable pushed tips, existing-ranker integration, compatible artifact
|
- [ ] pre-push-safety-net — Immutable pushed tips, existing-ranker integration, compatible artifact
|
||||||
reuse, total deadline, change attribution, actionability admission and compact hook behavior.
|
reuse, total deadline, change attribution, actionability admission and compact hook behavior.
|
||||||
Dependencies: contributor-scan. Requirements/design approved 2026-09-12; tasks generated
|
Dependencies: contributor-scan. Requirements/design approved 2026-09-12; tasks generated
|
||||||
(8 groups, 27 executable subtasks), independently reviewed and approved; groups 1–3 are complete (11/27 tasks), including the resolved frontend-retention prerequisite.
|
(8 groups, 27 executable subtasks), independently reviewed and approved; groups 1–4 are complete (14/27 tasks), including the resolved frontend-retention prerequisite.
|
||||||
|
|
||||||
### Next concrete step
|
### Next concrete step
|
||||||
|
|
||||||
Groups 1–3 have passed independent review: runtime, inputs, scoring, snapshots,
|
Groups 1–4 are verified: runtime, inputs, scoring, snapshots, shared deadline,
|
||||||
shared deadline, exact ranker scope, change-context evidence and vendor-free language
|
ranker scope, change context, vendor-free partitions, novelty comparison, actionability
|
||||||
partitions. The approved contributor-scan prerequisite 2.18 (`51fabef`) repairs both
|
admission and compact reporting. Tasks 4.1/4.2 passed independent review; 4.3 used the
|
||||||
JavaScript test filters; real graphs retain first-party tests and exclude vendor code.
|
Kiro manual review fallback after the agent service reached its thread limit.
|
||||||
Task 3.4 also repairs unreadable-input handling and preserves named PHP census evidence.
|
Group 5 is next: wire explicit full transaction replay, then run the first bounded
|
||||||
Group 4 is next: novelty comparison, actionability admission and compact advisory output.
|
experiment. Real capabilities remain disabled pending independent evaluation.
|
||||||
See `pre-push-safety-net/implementation-group-3.md` for evidence and limits.
|
Automatic dominance-only change-context projection remains an upstream limit to
|
||||||
|
retain during replay integration. See `pre-push-safety-net/implementation-group-4.md`
|
||||||
|
for verification and scope limits.
|
||||||
The first complete experiment is 5.2, before persistent caching and hook
|
The first complete experiment is 5.2, before persistent caching and hook
|
||||||
installation work. Keep PMPro as the primary core-optimization example, then freeze the core and
|
installation work. Keep PMPro as the primary core-optimization example, then freeze the core and
|
||||||
prove a changed security flow and its fixed twin on a JavaScript/Express Node API workload through
|
prove a changed security flow and its fixed twin on a JavaScript/Express Node API workload through
|
||||||
|
|||||||
@@ -79,10 +79,11 @@ updates as a shortcut to an unmeasured latency promise.
|
|||||||
The product direction above is user-requested. The amended requirements, design and initial
|
The product direction above is user-requested. The amended requirements, design and initial
|
||||||
evaluation profile in `pre-push-safety-net` were approved for task generation on 2026-09-12
|
evaluation profile in `pre-push-safety-net` were approved for task generation on 2026-09-12
|
||||||
("ok, next kiro phase"). The maintainer then approved the task plan and starting group 1
|
("ok, next kiro phase"). The maintainer then approved the task plan and starting group 1
|
||||||
("approve.") and then group 2 ("ok, next kiro phase, approved"). Groups 1–3 are now verified
|
("approve.") and then group 2 ("ok, next kiro phase, approved"). Groups 1–4 are now verified
|
||||||
complete (11/27 total). The maintainer approved the frontend-retention prerequisite;
|
complete (14/27 total), including the approved frontend-retention prerequisite.
|
||||||
contributor-scan 2.18 and pre-push task 3.4 have both passed independent review and
|
Group 4 adds bound novelty, operation-specific admission and compact artifacts;
|
||||||
fresh verification. Group 4 is next: novelty comparison, actionability admission and
|
4.1/4.2 passed independent review and 4.3 used the documented Kiro manual fallback
|
||||||
compact advisory output. Implementation is authorized under the approved plan;
|
after agent dispatch became unavailable. Group 5 is next for full replay and the
|
||||||
task completion still requires independent review and fresh evidence. Hook performance
|
first bounded experiment. Task completion still requires review and fresh evidence.
|
||||||
and capability admission remain unproven.
|
Real capability eligibility, automatic dominance context projection and hook
|
||||||
|
performance remain unproven; the default capability registry is empty.
|
||||||
|
|||||||
@@ -1,10 +1,10 @@
|
|||||||
{
|
{
|
||||||
"counts": {
|
"counts": {
|
||||||
"load_bearing": 100,
|
"load_bearing": 101,
|
||||||
"standalone_capability": 9
|
"standalone_capability": 9
|
||||||
},
|
},
|
||||||
"modules": 109,
|
"modules": 110,
|
||||||
"note": "Refreshed after pre-push-safety-net task 4.1: targeted base comparison reuses existing arbiters and shared execution budget; replay integration remains task 5.1.",
|
"note": "Refreshed after pre-push-safety-net task 4.3: compact reporting preserves full evidence and decisions; transaction replay remains task 5.1.",
|
||||||
"orphaned": [],
|
"orphaned": [],
|
||||||
"rows": [
|
"rows": [
|
||||||
{
|
{
|
||||||
@@ -103,6 +103,7 @@
|
|||||||
"hunter_tools",
|
"hunter_tools",
|
||||||
"model.endpoint",
|
"model.endpoint",
|
||||||
"pairs",
|
"pairs",
|
||||||
|
"push.policy",
|
||||||
"regress.candidate",
|
"regress.candidate",
|
||||||
"regress.recipes",
|
"regress.recipes",
|
||||||
"run"
|
"run"
|
||||||
@@ -360,6 +361,7 @@
|
|||||||
"preprocess",
|
"preprocess",
|
||||||
"push.contracts",
|
"push.contracts",
|
||||||
"push.policy",
|
"push.policy",
|
||||||
|
"push.report",
|
||||||
"push.snapshot"
|
"push.snapshot"
|
||||||
],
|
],
|
||||||
"module": "model",
|
"module": "model",
|
||||||
@@ -404,6 +406,7 @@
|
|||||||
"model.scan",
|
"model.scan",
|
||||||
"push.contracts",
|
"push.contracts",
|
||||||
"push.policy",
|
"push.policy",
|
||||||
|
"push.report",
|
||||||
"push.snapshot"
|
"push.snapshot"
|
||||||
],
|
],
|
||||||
"module": "model.contracts",
|
"module": "model.contracts",
|
||||||
@@ -511,7 +514,8 @@
|
|||||||
"classification": "load_bearing",
|
"classification": "load_bearing",
|
||||||
"importers": [
|
"importers": [
|
||||||
"cli",
|
"cli",
|
||||||
"push.policy"
|
"push.policy",
|
||||||
|
"push.report"
|
||||||
],
|
],
|
||||||
"module": "model.scan",
|
"module": "model.scan",
|
||||||
"reason": "contributor-scan Req 3: the repository driver -- one CPG, one budget, ranked spend"
|
"reason": "contributor-scan Req 3: the repository driver -- one CPG, one budget, ranked spend"
|
||||||
@@ -534,7 +538,8 @@
|
|||||||
"model.regions",
|
"model.regions",
|
||||||
"model.report",
|
"model.report",
|
||||||
"model.scan",
|
"model.scan",
|
||||||
"model.taint"
|
"model.taint",
|
||||||
|
"push.policy"
|
||||||
],
|
],
|
||||||
"module": "model.specs",
|
"module": "model.specs",
|
||||||
"reason": "model-grounded-detection Req 7.2-7.4: the security vocabularies the CPG queries are parameterised by"
|
"reason": "model-grounded-detection Req 7.2-7.4: the security vocabularies the CPG queries are parameterised by"
|
||||||
@@ -663,6 +668,8 @@
|
|||||||
{
|
{
|
||||||
"classification": "load_bearing",
|
"classification": "load_bearing",
|
||||||
"importers": [
|
"importers": [
|
||||||
|
"push.policy",
|
||||||
|
"push.report",
|
||||||
"push.snapshot"
|
"push.snapshot"
|
||||||
],
|
],
|
||||||
"module": "push",
|
"module": "push",
|
||||||
@@ -671,6 +678,8 @@
|
|||||||
{
|
{
|
||||||
"classification": "load_bearing",
|
"classification": "load_bearing",
|
||||||
"importers": [
|
"importers": [
|
||||||
|
"push.policy",
|
||||||
|
"push.report",
|
||||||
"push.snapshot"
|
"push.snapshot"
|
||||||
],
|
],
|
||||||
"module": "push.contracts",
|
"module": "push.contracts",
|
||||||
@@ -678,10 +687,18 @@
|
|||||||
},
|
},
|
||||||
{
|
{
|
||||||
"classification": "load_bearing",
|
"classification": "load_bearing",
|
||||||
"importers": [],
|
"importers": [
|
||||||
|
"push.report"
|
||||||
|
],
|
||||||
"module": "push.policy",
|
"module": "push.policy",
|
||||||
"reason": "pre-push-safety-net Req 2.4/3.1/3.4/4.1: targeted same-semantics base comparison; replay integration pending"
|
"reason": "pre-push-safety-net Req 2.4/3.1/3.4/4.1: targeted same-semantics base comparison; replay integration pending"
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
"classification": "load_bearing",
|
||||||
|
"importers": [],
|
||||||
|
"module": "push.report",
|
||||||
|
"reason": "pre-push-safety-net Req 5.1/5.2/5.5/7.3: compact output and complete artifacts; runner integration pending"
|
||||||
|
},
|
||||||
{
|
{
|
||||||
"classification": "load_bearing",
|
"classification": "load_bearing",
|
||||||
"importers": [],
|
"importers": [],
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
@@ -220,3 +220,25 @@ rechecks its actual engine answers through the current policy. With no evaluated
|
|||||||
capabilities supplied, every candidate stays diagnostic and the result separates
|
capabilities supplied, every candidate stays diagnostic and the result separates
|
||||||
`none` findings, `incomplete` coverage and `allow` disposition. This offline control
|
`none` findings, `incomplete` coverage and `allow` disposition. This offline control
|
||||||
executes no new graph query and does not qualify any capability for normal alerts.
|
executes no new graph query and does not qualify any capability for normal alerts.
|
||||||
|
|
||||||
|
## Compact reporting integration
|
||||||
|
|
||||||
|
After rebuilding the image, exercise comparison → admission → artifact/report
|
||||||
|
using captured actual engine answers and explicitly synthetic eligibility controls:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker run --rm -i --network none --entrypoint python openultrasast:dev - < ops/smoke_report.py
|
||||||
|
```
|
||||||
|
|
||||||
|
The control verifies full scan-record preservation, default diagnostic-only behavior,
|
||||||
|
a three-summary display cap that retains all five synthetic defects, and a write
|
||||||
|
failure that preserves blocking. It runs no new Joern query and admits no real
|
||||||
|
capability. The receipt records reporting duration separately from the captured
|
||||||
|
analysis timings.
|
||||||
|
|
||||||
|
Artifacts publish atomically from mode-0600 temporary files. The Linux writer runs
|
||||||
|
in a separate process bounded by the transaction deadline and its remaining report
|
||||||
|
allowance; a stalled writer is killed. A hard cancellation can leave a private
|
||||||
|
`.partial-` file, which is never the published artifact and cannot establish a
|
||||||
|
completed result. Serialization/write failure emits one notice without replacing
|
||||||
|
the finding, coverage or push decision.
|
||||||
|
|||||||
@@ -0,0 +1,177 @@
|
|||||||
|
"""Packaged comparison/admission/report integration over captured and synthetic evidence."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import tempfile
|
||||||
|
import time
|
||||||
|
from dataclasses import asdict, replace
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from openultrasast.config import PushConfig
|
||||||
|
from openultrasast.model.contracts import ChangeContext, ExecutionBudget, FamilyCoverage, QuestionOutcome, RankedQuestion, ScopeDecision
|
||||||
|
from openultrasast.model.ladder import Rung
|
||||||
|
from openultrasast.model.partitions import PartitionCoverage
|
||||||
|
from openultrasast.model.pipeline import ModelFinding
|
||||||
|
from openultrasast.model.scan import ModelScanResult
|
||||||
|
from openultrasast.push.contracts import ComparisonAnalysis, PushComparison
|
||||||
|
from openultrasast.push.policy import (
|
||||||
|
AdmissionCandidate,
|
||||||
|
CapabilityKey,
|
||||||
|
admission_push_result,
|
||||||
|
admit_candidates,
|
||||||
|
compare_evidence,
|
||||||
|
)
|
||||||
|
from openultrasast.push.report import PushReport, ScanRecord, deliver_report
|
||||||
|
|
||||||
|
|
||||||
|
def restore(data: dict) -> ModelScanResult:
|
||||||
|
values = dict(data)
|
||||||
|
values["findings"] = tuple(ModelFinding(**{**f, "rung": Rung(f["rung"])}) for f in data["findings"])
|
||||||
|
values["scope"] = ScopeDecision.from_payload(data["scope"]) if data["scope"] else None
|
||||||
|
values["question_outcomes"] = tuple(QuestionOutcome.from_payload(q) for q in data["question_outcomes"])
|
||||||
|
values["change_context"] = ChangeContext.from_payload(data["change_context"]) if data["change_context"] else None
|
||||||
|
values["family_coverage"] = tuple(FamilyCoverage.from_payload(c) for c in data["family_coverage"])
|
||||||
|
values["partitions"] = tuple(
|
||||||
|
PartitionCoverage(**{**p, **{k: tuple(p[k]) for k in ("paths", "boundaries", "excluded_paths", "unshipped_paths")}})
|
||||||
|
for p in data["partitions"]
|
||||||
|
)
|
||||||
|
values["degradations"] = tuple(data["degradations"])
|
||||||
|
values["tiers"] = tuple(tuple(t) for t in data["tiers"])
|
||||||
|
values["tier_counts"] = {int(k): v for k, v in data["tier_counts"].items()}
|
||||||
|
result = ModelScanResult(**values)
|
||||||
|
assert json.loads(json.dumps(asdict(result))) == data, "record restoration lost evidence"
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
measured = Path("benchmarks/measurements")
|
||||||
|
raw = (measured / "2026-09-13-delta-comparison-smoke.json").read_bytes()
|
||||||
|
controls = json.loads(raw)
|
||||||
|
synthetic_bytes = (measured / "2026-09-13-admission-policy-smoke.json").read_bytes()
|
||||||
|
assert raw and synthetic_bytes
|
||||||
|
synthetic = json.loads(synthetic_bytes)["synthetic_controls"]["javascript"]
|
||||||
|
policy = Path("src/openultrasast/push/policy.py").read_bytes()
|
||||||
|
reporter = Path("src/openultrasast/push/report.py").read_bytes()
|
||||||
|
provenance = {
|
||||||
|
"repository": "captured immutable Git fixture comparisons",
|
||||||
|
"engine": controls["runtime"]["image"],
|
||||||
|
"facts": "covered by recorded comparison semantics",
|
||||||
|
"queries": "covered by recorded comparison semantics",
|
||||||
|
"policy": hashlib.sha256(policy + reporter).hexdigest(),
|
||||||
|
}
|
||||||
|
summary = {
|
||||||
|
"purpose": "Packaged reporting over captured actual engine evidence and explicitly synthetic admission controls; no new Joern run",
|
||||||
|
"input_bytes": len(raw) + len(synthetic_bytes),
|
||||||
|
"comparison_artifact_sha256": hashlib.sha256(raw).hexdigest(),
|
||||||
|
"policy_sha256": hashlib.sha256(policy).hexdigest(),
|
||||||
|
"report_sha256": hashlib.sha256(reporter).hexdigest(),
|
||||||
|
"cases": {},
|
||||||
|
}
|
||||||
|
with tempfile.TemporaryDirectory(prefix="ousast-report-smoke-") as directory:
|
||||||
|
root = Path(directory)
|
||||||
|
|
||||||
|
def emit(name: str, report: PushReport, *, fail: bool = False) -> None:
|
||||||
|
target = root / (name + ".json")
|
||||||
|
if fail:
|
||||||
|
target.mkdir()
|
||||||
|
delivery = deliver_report(report, target, execution_budget=ExecutionBudget(time.monotonic() + 10, 1.0))
|
||||||
|
assert delivery.result == report.result and delivery.exit_code == int(report.result.push_disposition == "block")
|
||||||
|
assert delivery.text.count("Notice:") <= 1
|
||||||
|
artifact = json.loads(target.read_text()) if delivery.artifact else None
|
||||||
|
if fail:
|
||||||
|
assert delivery.error and delivery.artifact is None and target.is_dir()
|
||||||
|
else:
|
||||||
|
assert delivery.error is None and artifact is not None
|
||||||
|
assert artifact["result"] == report.result.to_payload()
|
||||||
|
assert artifact["admission"] == report.admission.to_payload()
|
||||||
|
assert artifact["scans"] == json.loads(json.dumps([asdict(s) for s in report.scans]))
|
||||||
|
summary["cases"][name] = {
|
||||||
|
"exit_code": delivery.exit_code,
|
||||||
|
"text": delivery.text,
|
||||||
|
"error": delivery.error,
|
||||||
|
"reporting_seconds": delivery.reporting_seconds,
|
||||||
|
"artifact": artifact,
|
||||||
|
}
|
||||||
|
|
||||||
|
for name, record in controls["cases"].items():
|
||||||
|
evidence = record["evidence"]
|
||||||
|
head = restore(evidence["head"])
|
||||||
|
base = restore(evidence["delta"]["base_scan"])
|
||||||
|
context = ChangeContext.from_payload(evidence["context"])
|
||||||
|
comparison = PushComparison.from_payload(evidence["comparison"])
|
||||||
|
semantics = evidence["delta"]["semantics"]
|
||||||
|
deltas = compare_evidence(head, base, context=context, head_semantics=semantics, base_semantics=semantics)
|
||||||
|
assert deltas and all(d.novelty == evidence["expected_novelty"] for d in deltas)
|
||||||
|
candidates = []
|
||||||
|
for delta in deltas:
|
||||||
|
op = delta.head_operation
|
||||||
|
assert op is not None
|
||||||
|
key = CapabilityKey(op.question.language, "unspecified", "unspecified", delta.family, op.mechanism, "unreviewed", semantics)
|
||||||
|
candidates.append(AdmissionCandidate(delta, key, comparison))
|
||||||
|
admission = admit_candidates(candidates)
|
||||||
|
assert not admission.defects and head.scope is not None
|
||||||
|
analysis = ComparisonAnalysis(
|
||||||
|
comparison, (head.scope,), tuple(q.identity for q in head.question_outcomes), "complete_within_scope"
|
||||||
|
)
|
||||||
|
result = admission_push_result(admission, (analysis,))
|
||||||
|
assert result.coverage_status == "incomplete"
|
||||||
|
report = PushReport(
|
||||||
|
result,
|
||||||
|
admission,
|
||||||
|
(ScanRecord(comparison, "head", head), ScanRecord(comparison, "base", base)),
|
||||||
|
{**provenance, "comparison_semantics": semantics},
|
||||||
|
{"total_seconds": record["completion"]["elapsed_seconds"]},
|
||||||
|
)
|
||||||
|
emit(name, report)
|
||||||
|
|
||||||
|
# These five normalized controls test display/decision separation only.
|
||||||
|
from openultrasast.push.policy import CandidateDisposition
|
||||||
|
|
||||||
|
disposition = CandidateDisposition.from_payload(synthetic["admission"]["dispositions"][0])
|
||||||
|
cap = disposition.capability_evaluations[0]
|
||||||
|
candidate = disposition.candidate
|
||||||
|
candidates = []
|
||||||
|
questions = []
|
||||||
|
findings = []
|
||||||
|
outcomes = []
|
||||||
|
for i in range(5):
|
||||||
|
old = candidate.delta.head_operation
|
||||||
|
assert old is not None
|
||||||
|
q = replace(old.question, path=f"handler{i}.src")
|
||||||
|
op = replace(old, question=q, path=q.path)
|
||||||
|
delta = replace(
|
||||||
|
candidate.delta,
|
||||||
|
defect_id=f"synthetic-defect-{i}",
|
||||||
|
head_operation=op,
|
||||||
|
site=f"{q.path}:{op.line}:handler",
|
||||||
|
change_evidence=(f"head:{q.path}:2-2",),
|
||||||
|
)
|
||||||
|
candidates.append(replace(candidate, delta=delta))
|
||||||
|
questions.append(q)
|
||||||
|
findings.append(ModelFinding(delta.site, delta.family, Rung(delta.rung), delta.witness or "synthetic witness"))
|
||||||
|
outcomes.append(QuestionOutcome(q, "completed", "synthetic-normalized-control", "[]"))
|
||||||
|
scope = ScopeDecision("synthetic-ranker", "static", True, tuple(RankedQuestion(q, 1.0, ()) for q in questions), (), ())
|
||||||
|
scan = ModelScanResult(findings=tuple(findings), scope=scope, question_outcomes=tuple(outcomes))
|
||||||
|
base_scan = replace(scan, findings=())
|
||||||
|
analysis = ComparisonAnalysis(candidate.comparison, (scope,), tuple(questions), "complete_within_scope")
|
||||||
|
admission = admit_candidates(candidates, capabilities=(cap,))
|
||||||
|
assert len(admission.defects) == 5
|
||||||
|
result = admission_push_result(admission, (analysis,), config=PushConfig(mode="blocking"))
|
||||||
|
report = PushReport(
|
||||||
|
result,
|
||||||
|
admission,
|
||||||
|
(ScanRecord(candidate.comparison, "head", scan), ScanRecord(candidate.comparison, "base", base_scan)),
|
||||||
|
{**provenance, "repository": "synthetic normalized policy controls", "engine": "synthetic; no engine execution"},
|
||||||
|
{"total_seconds": 0.0},
|
||||||
|
)
|
||||||
|
emit("synthetic-display-cap", report)
|
||||||
|
text = summary["cases"]["synthetic-display-cap"]["text"]
|
||||||
|
assert text.count("- injection at ") == 3 and "2 more" in text
|
||||||
|
emit("synthetic-write-failure", report, fail=True)
|
||||||
|
print(json.dumps(summary, indent=2))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -28,6 +28,7 @@ ROOTS = ("gate", "map_gate", "pair_gate", "cli", "__main__")
|
|||||||
|
|
||||||
# Modules that stand alone: their own CLI subcommand or external protocol, deliberately off the scan path.
|
# Modules that stand alone: their own CLI subcommand or external protocol, deliberately off the scan path.
|
||||||
SPEC_OWNED = {
|
SPEC_OWNED = {
|
||||||
|
"push.report": "pre-push-safety-net Req 5.1/5.2/5.5/7.3: compact output and complete artifacts; runner integration pending",
|
||||||
"push.policy": "pre-push-safety-net Req 2.4/3.1/3.4/4.1: targeted same-semantics base comparison; replay integration pending",
|
"push.policy": "pre-push-safety-net Req 2.4/3.1/3.4/4.1: targeted same-semantics base comparison; replay integration pending",
|
||||||
"push.snapshot": "pre-push-safety-net Req 1.1/1.3/1.4/7.1: local immutable ref resolution; runner integration pending",
|
"push.snapshot": "pre-push-safety-net Req 1.1/1.3/1.4/7.1: local immutable ref resolution; runner integration pending",
|
||||||
"contracts": "pre-push-safety-net Req 7.3: strict shared manifest serialization, independent of push",
|
"contracts": "pre-push-safety-net Req 7.3: strict shared manifest serialization, independent of push",
|
||||||
|
|||||||
@@ -0,0 +1,242 @@
|
|||||||
|
"""Compact presentation and complete artifacts for an already-decided push."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import math
|
||||||
|
import multiprocessing
|
||||||
|
import os
|
||||||
|
import time
|
||||||
|
import uuid
|
||||||
|
from collections.abc import Mapping
|
||||||
|
from dataclasses import asdict, dataclass
|
||||||
|
from multiprocessing.connection import Connection
|
||||||
|
from pathlib import Path
|
||||||
|
from types import MappingProxyType
|
||||||
|
from typing import Literal
|
||||||
|
|
||||||
|
from openultrasast.model.contracts import ExecutionBudget
|
||||||
|
from openultrasast.model.scan import ModelScanResult
|
||||||
|
from openultrasast.push.contracts import PushComparison, PushResolution, PushResult
|
||||||
|
from openultrasast.push.policy import AdmissionResult, _context_boundaries
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class ScanRecord:
|
||||||
|
comparison: PushComparison
|
||||||
|
side: Literal["base", "head"]
|
||||||
|
scan: ModelScanResult
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class PushReport:
|
||||||
|
result: PushResult
|
||||||
|
admission: AdmissionResult
|
||||||
|
scans: tuple[ScanRecord, ...]
|
||||||
|
provenance: Mapping[str, str]
|
||||||
|
timings: Mapping[str, float]
|
||||||
|
resolution: PushResolution | None = None
|
||||||
|
|
||||||
|
def __post_init__(self) -> None:
|
||||||
|
object.__setattr__(self, "provenance", MappingProxyType(dict(self.provenance)))
|
||||||
|
object.__setattr__(self, "timings", MappingProxyType(dict(self.timings)))
|
||||||
|
ids = tuple(defect.defect_id for defect in self.admission.defects)
|
||||||
|
if ids != self.result.actionable_defect_ids:
|
||||||
|
raise ValueError("report must retain the exact admitted defect set and order")
|
||||||
|
if any(
|
||||||
|
not all((d.witnesses, d.locations, d.consequences, d.repairs, d.change_evidence, d.comparisons)) for d in self.admission.defects
|
||||||
|
):
|
||||||
|
raise ValueError("actionable summaries require witnesses, locations, consequence, repair and change evidence")
|
||||||
|
admitted = {d.candidate.delta.defect_id for d in self.admission.dispositions if d.admitted}
|
||||||
|
if admitted != set(ids):
|
||||||
|
raise ValueError("report must retain every admitted candidate disposition")
|
||||||
|
comparisons = {analysis.comparison for analysis in self.result.analyses}
|
||||||
|
if self.result.coverage_status == "complete_within_scope" and self.admission.coverage_reasons:
|
||||||
|
raise ValueError("complete comparison cannot hide admission coverage gaps")
|
||||||
|
if any(record.comparison not in comparisons or record.side not in ("base", "head") for record in self.scans):
|
||||||
|
raise ValueError("scan evidence must belong to an exact reported comparison and side")
|
||||||
|
for analysis in self.result.analyses:
|
||||||
|
heads = [record.scan for record in self.scans if record.comparison == analysis.comparison and record.side == "head"]
|
||||||
|
if any(scope not in [scan.scope for scan in heads] for scope in analysis.scopes):
|
||||||
|
raise ValueError("reported scope requires the actual head scan evidence")
|
||||||
|
completed = {outcome.identity for scan in heads for outcome in scan.question_outcomes if outcome.status == "completed"}
|
||||||
|
if not set(analysis.completed_questions) <= completed:
|
||||||
|
raise ValueError("completed questions require completed recorded outcomes")
|
||||||
|
if analysis.coverage_status == "complete_within_scope":
|
||||||
|
bases = [record.scan for record in self.scans if record.comparison == analysis.comparison and record.side == "base"]
|
||||||
|
if not bases or any(not _completed_scan(scan) for scan in (*heads, *bases)):
|
||||||
|
raise ValueError("complete comparison requires completed head and base evidence")
|
||||||
|
required = {"repository", "engine", "facts", "queries", "policy"}
|
||||||
|
if not required <= self.provenance.keys() or any(not isinstance(v, str) or not v.strip() for v in self.provenance.values()):
|
||||||
|
raise ValueError("report requires repository, engine, facts, queries and policy provenance")
|
||||||
|
if "total_seconds" not in self.timings or any(
|
||||||
|
type(value) not in (int, float) or not math.isfinite(value) or value < 0 for value in self.timings.values()
|
||||||
|
):
|
||||||
|
raise ValueError("report requires finite nonnegative timings including total_seconds")
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class ReportDelivery:
|
||||||
|
result: PushResult
|
||||||
|
text: str
|
||||||
|
artifact: Path | None
|
||||||
|
error: str | None
|
||||||
|
reporting_seconds: float = 0.0
|
||||||
|
|
||||||
|
@property
|
||||||
|
def exit_code(self) -> int:
|
||||||
|
return int(self.result.push_disposition == "block")
|
||||||
|
|
||||||
|
|
||||||
|
def _completed_scan(scan: ModelScanResult) -> bool:
|
||||||
|
return bool(
|
||||||
|
scan.scope
|
||||||
|
and scan.scope.population_complete
|
||||||
|
and not scan.scope.deferred
|
||||||
|
and not _context_boundaries(scan)
|
||||||
|
and not scan.degradations
|
||||||
|
and {q.identity for q in scan.scope.selected} == {q.identity for q in scan.question_outcomes if q.status == "completed"}
|
||||||
|
and all(q.status == "completed" for q in scan.question_outcomes)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _write_artifact(report: PushReport, target: Path, temporary: Path, connection: Connection) -> None:
|
||||||
|
"""Only this bounded child performs serialization or filesystem operations."""
|
||||||
|
try:
|
||||||
|
target.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
fd = os.open(temporary, os.O_CREAT | os.O_EXCL | os.O_WRONLY | os.O_NOFOLLOW, 0o600)
|
||||||
|
try:
|
||||||
|
with os.fdopen(fd, "w", encoding="utf-8") as stream:
|
||||||
|
payload = {
|
||||||
|
"schema_version": 1,
|
||||||
|
"result": report.result.to_payload(),
|
||||||
|
"admission": report.admission.to_payload(),
|
||||||
|
"scans": [asdict(record) for record in report.scans],
|
||||||
|
"provenance": dict(report.provenance),
|
||||||
|
"timings": dict(report.timings),
|
||||||
|
"timing_scope": "Caller stage measurements; publication and rendering duration is returned in ReportDelivery.",
|
||||||
|
"resolution": report.resolution.to_payload() if report.resolution else None,
|
||||||
|
}
|
||||||
|
json.dump(payload, stream, ensure_ascii=True, allow_nan=False, indent=2)
|
||||||
|
stream.write("\n")
|
||||||
|
stream.flush()
|
||||||
|
os.fsync(stream.fileno())
|
||||||
|
os.replace(temporary, target)
|
||||||
|
finally:
|
||||||
|
temporary.unlink(missing_ok=True)
|
||||||
|
connection.send(None)
|
||||||
|
except Exception as error:
|
||||||
|
connection.send("artifact_write_failed:" + type(error).__name__)
|
||||||
|
finally:
|
||||||
|
connection.close()
|
||||||
|
|
||||||
|
|
||||||
|
def _save(report: PushReport, target: Path, budget: ExecutionBudget) -> str | None:
|
||||||
|
# Persist completed evidence during the remaining cancellation/report allowance.
|
||||||
|
# Never restart that allowance for a late caller.
|
||||||
|
end = budget.deadline_monotonic + budget.cancellation_allowance_seconds
|
||||||
|
if time.monotonic() >= end:
|
||||||
|
return "report_deadline_exhausted"
|
||||||
|
reserve = min(0.1, budget.cancellation_allowance_seconds / 2)
|
||||||
|
reader = writer = process = None
|
||||||
|
try:
|
||||||
|
temporary = target.with_name("." + target.name + ".partial-" + uuid.uuid4().hex)
|
||||||
|
context = multiprocessing.get_context("fork")
|
||||||
|
reader, writer = context.Pipe(duplex=False)
|
||||||
|
process = context.Process(target=_write_artifact, args=(report, target, temporary, writer), daemon=True)
|
||||||
|
process.start()
|
||||||
|
writer.close()
|
||||||
|
process.join(max(0.0, end - time.monotonic() - reserve))
|
||||||
|
if process.is_alive():
|
||||||
|
process.kill()
|
||||||
|
process.join(max(0.0, end - time.monotonic()))
|
||||||
|
return "report_deadline_exhausted"
|
||||||
|
if process.exitcode != 0 or not reader.poll():
|
||||||
|
return "artifact_writer_failed"
|
||||||
|
error = reader.recv()
|
||||||
|
return None if error is None else error if isinstance(error, str) else "artifact_writer_failed"
|
||||||
|
except (OSError, ValueError, EOFError, RuntimeError):
|
||||||
|
return "artifact_writer_unavailable"
|
||||||
|
finally:
|
||||||
|
if writer is not None:
|
||||||
|
writer.close()
|
||||||
|
if reader is not None:
|
||||||
|
reader.close()
|
||||||
|
if process is not None:
|
||||||
|
if process.is_alive():
|
||||||
|
process.kill()
|
||||||
|
process.join(max(0.0, end - time.monotonic()))
|
||||||
|
if not process.is_alive():
|
||||||
|
process.close()
|
||||||
|
|
||||||
|
|
||||||
|
def _short(value: str, limit: int = 180) -> str:
|
||||||
|
# Bound before escaping: repository-controlled names/witnesses cannot add lines
|
||||||
|
# or terminal escapes. Full spelling and every witness remain in the artifact.
|
||||||
|
clipped = value if len(value) <= limit else value[: limit // 2] + "..." + value[-limit // 2 :]
|
||||||
|
return json.dumps(clipped, ensure_ascii=True)[1:-1]
|
||||||
|
|
||||||
|
|
||||||
|
def render_report(report: PushReport, *, artifact: Path | None, error: str | None = None) -> str:
|
||||||
|
lines: list[str] = []
|
||||||
|
defects = report.admission.defects
|
||||||
|
for defect in defects[:3]:
|
||||||
|
path, line = defect.locations[0]
|
||||||
|
delta = next(
|
||||||
|
disposition.candidate.delta
|
||||||
|
for disposition in report.admission.dispositions
|
||||||
|
if disposition.admitted and disposition.candidate.delta.defect_id == defect.defect_id
|
||||||
|
)
|
||||||
|
change = (
|
||||||
|
"A protection present in the base is missing here."
|
||||||
|
if delta.reason == "discharge_removed"
|
||||||
|
else "This change connects input to the operation."
|
||||||
|
)
|
||||||
|
lines.extend(
|
||||||
|
(
|
||||||
|
f"- {_short(defect.family)} at {_short(path, 4096)}:{line}: {_short(defect.consequences[0])}",
|
||||||
|
f" Evidence: {_short(defect.witnesses[0])}; Change: {change} ({_short(defect.change_evidence[0])})",
|
||||||
|
f" Fix: {_short(defect.repairs[0])}",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if len(defects) > 3:
|
||||||
|
lines.append(f"{len(defects) - 3} more actionable defects retained in the detailed result.")
|
||||||
|
if not defects:
|
||||||
|
coverage = report.result.coverage_status
|
||||||
|
if coverage == "complete_within_scope":
|
||||||
|
completed = sum(len(analysis.completed_questions) for analysis in report.result.analyses)
|
||||||
|
lines.append(f"No actionable defects in {completed} completed checks within the recorded scope.")
|
||||||
|
elif coverage == "not_applicable":
|
||||||
|
lines.append("No applicable code updates to check.")
|
||||||
|
elif coverage == "unavailable":
|
||||||
|
lines.append("No analysis result is available.")
|
||||||
|
else:
|
||||||
|
lines.append("No actionable defects reported; analysis is incomplete.")
|
||||||
|
notices = []
|
||||||
|
if report.result.coverage_status in ("incomplete", "unavailable"):
|
||||||
|
reasons = report.admission.coverage_reasons
|
||||||
|
if any("deadline" in reason for reason in reasons):
|
||||||
|
notices.append("Analysis did not finish. Rerun with a longer analysis deadline.")
|
||||||
|
elif any("capability" in reason for reason in reasons):
|
||||||
|
notices.append("Some checks are not enabled for normal alerts. Review capability coverage in the details.")
|
||||||
|
else:
|
||||||
|
notices.append(
|
||||||
|
"Some checks could not be completed. Review the comparison details and resolve the recorded gaps before retrying."
|
||||||
|
)
|
||||||
|
if error:
|
||||||
|
notices.append("Details could not be saved. Retry with a writable artifact path and available reporting time.")
|
||||||
|
if notices:
|
||||||
|
lines.append("Notice: " + " ".join(notices))
|
||||||
|
if artifact is not None:
|
||||||
|
lines.append("Details: " + _short(str(artifact), 4096))
|
||||||
|
if report.result.push_disposition == "block":
|
||||||
|
lines.append("Push blocked by the configured policy.")
|
||||||
|
return "\n".join(lines) + "\n"
|
||||||
|
|
||||||
|
|
||||||
|
def deliver_report(report: PushReport, artifact: Path, *, execution_budget: ExecutionBudget) -> ReportDelivery:
|
||||||
|
started = time.monotonic()
|
||||||
|
error = _save(report, artifact, execution_budget)
|
||||||
|
saved = artifact if error is None else None
|
||||||
|
text = render_report(report, artifact=saved, error=error)
|
||||||
|
return ReportDelivery(report.result, text, saved, error, time.monotonic() - started)
|
||||||
@@ -0,0 +1,182 @@
|
|||||||
|
"""Presentation bounds cannot alter admission, evidence, or enforcement."""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import time
|
||||||
|
from dataclasses import replace
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from test_push_admission import control
|
||||||
|
|
||||||
|
from openultrasast.config import PushConfig
|
||||||
|
from openultrasast.model.contracts import ExecutionBudget, QuestionOutcome
|
||||||
|
from openultrasast.model.scan import ModelScanResult
|
||||||
|
from openultrasast.push.policy import admission_push_result, admit_candidates
|
||||||
|
from openultrasast.push.report import PushReport, ScanRecord, deliver_report
|
||||||
|
|
||||||
|
|
||||||
|
def report_with(count=1, *, admitted=True, blocking=False):
|
||||||
|
candidate, capability, analysis = control()
|
||||||
|
candidates = tuple(replace(candidate, delta=replace(candidate.delta, defect_id=f"defect-{i}")) for i in range(count))
|
||||||
|
admission = admit_candidates(candidates, capabilities=(capability,) if admitted else ())
|
||||||
|
result = admission_push_result(admission, (analysis,), config=PushConfig(mode="blocking" if blocking else "advisory"))
|
||||||
|
q = analysis.completed_questions[0]
|
||||||
|
scan = ModelScanResult(scope=analysis.scopes[0], question_outcomes=(QuestionOutcome(q, "completed", "answered", "[]"),))
|
||||||
|
records = (ScanRecord(analysis.comparison, "head", scan), ScanRecord(analysis.comparison, "base", scan))
|
||||||
|
provenance = {key: "synthetic-control" for key in ("repository", "engine", "facts", "queries", "policy")}
|
||||||
|
return PushReport(result, admission, records, provenance, {"total_seconds": 0.1})
|
||||||
|
|
||||||
|
|
||||||
|
def budget(seconds=3.0):
|
||||||
|
return ExecutionBudget(time.monotonic() + seconds, 0.2)
|
||||||
|
|
||||||
|
|
||||||
|
def test_display_cap_preserves_all_decisions_and_complete_artifact(tmp_path):
|
||||||
|
report = report_with(5, blocking=True)
|
||||||
|
delivered = deliver_report(report, tmp_path / "run.json", execution_budget=budget())
|
||||||
|
assert delivered.exit_code == 1 and delivered.result == report.result
|
||||||
|
assert delivered.text.count("- injection at ") == 3 and "2 more" in delivered.text
|
||||||
|
assert "Evidence:" in delivered.text and "Change:" in delivered.text and "Fix:" in delivered.text
|
||||||
|
data = json.loads(delivered.artifact.read_text())
|
||||||
|
assert len(data["admission"]["defects"]) == 5 and len(data["admission"]["dispositions"]) == 5
|
||||||
|
assert data["result"]["push_disposition"] == "block"
|
||||||
|
assert data["scans"][0]["scan"]["question_outcomes"][0]["raw_rows_json"] == "[]"
|
||||||
|
assert data["scans"][0]["scan"]["scope"]["selected"][0]["identity"]["path"] == "handler.src"
|
||||||
|
assert data["provenance"] == report.provenance and data["timings"] == report.timings
|
||||||
|
assert os.stat(delivered.artifact).st_mode & 0o777 == 0o600
|
||||||
|
|
||||||
|
|
||||||
|
def test_one_incomplete_notice_and_one_status_without_claiming_secure(tmp_path):
|
||||||
|
report = report_with(8, admitted=False)
|
||||||
|
delivered = deliver_report(report, tmp_path / "run.json", execution_budget=budget())
|
||||||
|
assert delivered.exit_code == 0 and delivered.result.coverage_status == "incomplete"
|
||||||
|
assert delivered.text.count("Notice:") == 1
|
||||||
|
assert delivered.text.count("No actionable defects") == 1
|
||||||
|
assert "Review" in delivered.text and str(delivered.artifact) in delivered.text
|
||||||
|
assert "secure" not in delivered.text.lower() and "clean" not in delivered.text.lower()
|
||||||
|
|
||||||
|
|
||||||
|
def test_complete_empty_check_is_bounded_by_scope(tmp_path):
|
||||||
|
delivered = deliver_report(report_with(0), tmp_path / "run.json", execution_budget=budget())
|
||||||
|
assert "1 completed" in delivered.text and "within" in delivered.text
|
||||||
|
assert "Notice:" not in delivered.text and "secure" not in delivered.text.lower()
|
||||||
|
|
||||||
|
|
||||||
|
def test_write_failure_retains_existing_artifact_and_block_decision(tmp_path):
|
||||||
|
target = tmp_path / "directory"
|
||||||
|
target.mkdir()
|
||||||
|
delivered = deliver_report(report_with(blocking=True), target, execution_budget=budget())
|
||||||
|
assert delivered.exit_code == 1 and delivered.artifact is None and delivered.error
|
||||||
|
assert delivered.text.count("Notice:") == 1 and "writable" in delivered.text
|
||||||
|
assert target.is_dir()
|
||||||
|
|
||||||
|
|
||||||
|
def test_invalid_serializable_detail_fails_without_losing_decision(tmp_path):
|
||||||
|
report = report_with(blocking=True)
|
||||||
|
broken = replace(report.scans[0], scan=replace(report.scans[0].scan, by_rung={"invalid_metadata": object()}))
|
||||||
|
report = replace(report, scans=(broken, report.scans[1]))
|
||||||
|
delivered = deliver_report(report, tmp_path / "run.json", execution_budget=budget())
|
||||||
|
assert delivered.exit_code == 1 and delivered.artifact is None and delivered.error
|
||||||
|
|
||||||
|
|
||||||
|
def test_mismatched_result_or_missing_evidence_is_rejected():
|
||||||
|
report = report_with()
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
replace(report, admission=admit_candidates(()))
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
replace(report, scans=())
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
replace(report, provenance={"engine": "only"})
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
replace(report, timings={"total_seconds": float("nan")})
|
||||||
|
|
||||||
|
|
||||||
|
def test_repository_control_characters_cannot_inject_terminal_notices(tmp_path):
|
||||||
|
report = report_with()
|
||||||
|
defect = replace(report.admission.defects[0], witnesses=("input\nNotice: fake\x1b[31m",))
|
||||||
|
report = replace(report, admission=replace(report.admission, defects=(defect,)))
|
||||||
|
delivered = deliver_report(report, tmp_path / "run.json", execution_budget=budget())
|
||||||
|
assert "\nNotice: fake" not in delivered.text and "\x1b" not in delivered.text
|
||||||
|
assert json.loads(delivered.artifact.read_text())["admission"]["defects"][0]["witnesses"] == ["input\nNotice: fake\x1b[31m"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_expired_reporting_allowance_still_returns_real_decision(tmp_path):
|
||||||
|
delivered = deliver_report(report_with(blocking=True), tmp_path / "run.json", execution_budget=ExecutionBudget(0.0, 0.1))
|
||||||
|
assert delivered.exit_code == 1 and delivered.error == "report_deadline_exhausted"
|
||||||
|
assert delivered.text.count("Notice:") == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_stalled_writer_is_cancelled_without_overwriting_previous_artifact(tmp_path, monkeypatch):
|
||||||
|
import multiprocessing
|
||||||
|
|
||||||
|
import openultrasast.push.report as module
|
||||||
|
|
||||||
|
def stall(*args):
|
||||||
|
time.sleep(10)
|
||||||
|
|
||||||
|
target = tmp_path / "run.json"
|
||||||
|
target.write_text("previous-complete-artifact")
|
||||||
|
children = {p.pid for p in multiprocessing.active_children()}
|
||||||
|
monkeypatch.setattr(module, "_write_artifact", stall)
|
||||||
|
started = time.monotonic()
|
||||||
|
delivered = deliver_report(report_with(blocking=True), target, execution_budget=budget(0.05))
|
||||||
|
assert time.monotonic() - started < 0.6
|
||||||
|
assert delivered.exit_code == 1 and delivered.error == "report_deadline_exhausted"
|
||||||
|
assert target.read_text() == "previous-complete-artifact"
|
||||||
|
assert {p.pid for p in multiprocessing.active_children()} == children
|
||||||
|
|
||||||
|
|
||||||
|
def test_long_terminal_values_are_bounded_but_artifact_keeps_full_evidence(tmp_path):
|
||||||
|
report = report_with()
|
||||||
|
witness = "request.input->" + "x" * 100_000
|
||||||
|
defect = replace(report.admission.defects[0], witnesses=(witness,))
|
||||||
|
report = replace(report, admission=replace(report.admission, defects=(defect,)))
|
||||||
|
delivered = deliver_report(report, tmp_path / "run.json", execution_budget=budget())
|
||||||
|
assert len(delivered.text) < 3000 and witness not in delivered.text
|
||||||
|
assert json.loads(delivered.artifact.read_text())["admission"]["defects"][0]["witnesses"] == [witness]
|
||||||
|
|
||||||
|
|
||||||
|
def test_unavailable_and_not_applicable_never_read_as_completed(tmp_path):
|
||||||
|
from openultrasast.push.contracts import PushResult
|
||||||
|
from openultrasast.push.policy import AdmissionResult
|
||||||
|
|
||||||
|
for coverage, expected_notice in [("unavailable", 1), ("not_applicable", 0)]:
|
||||||
|
report = report_with(0)
|
||||||
|
result = PushResult((), "none", coverage, "allow", ())
|
||||||
|
report = replace(report, result=result, admission=AdmissionResult((), (), ()), scans=())
|
||||||
|
delivered = deliver_report(report, tmp_path / (coverage + ".json"), execution_budget=budget())
|
||||||
|
assert "completed checks" not in delivered.text and delivered.text.count("Notice:") == expected_notice
|
||||||
|
assert delivered.result.coverage_status == coverage
|
||||||
|
|
||||||
|
|
||||||
|
def test_provenance_and_timings_are_frozen_after_construction():
|
||||||
|
report = report_with()
|
||||||
|
with pytest.raises(TypeError):
|
||||||
|
report.provenance["engine"] = "different"
|
||||||
|
with pytest.raises(TypeError):
|
||||||
|
report.timings["total_seconds"] = 0.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_writer_exiting_zero_without_receipt_is_not_success(tmp_path, monkeypatch):
|
||||||
|
import openultrasast.push.report as module
|
||||||
|
|
||||||
|
def empty_writer(report, target, temporary, connection):
|
||||||
|
connection.close()
|
||||||
|
|
||||||
|
monkeypatch.setattr(module, "_write_artifact", empty_writer)
|
||||||
|
delivered = deliver_report(report_with(blocking=True), tmp_path / "run.json", execution_budget=budget())
|
||||||
|
assert delivered.artifact is None and delivered.error and delivered.exit_code == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_invalid_artifact_destination_is_a_notice_not_an_exception():
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
delivered = deliver_report(report_with(blocking=True), Path("/"), execution_budget=budget())
|
||||||
|
assert delivered.artifact is None and delivered.error and delivered.exit_code == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_failed_base_evidence_cannot_be_presented_as_complete():
|
||||||
|
report = report_with(0)
|
||||||
|
failed = replace(report.scans[1], scan=ModelScanResult(degradations=({"reason": "cpg_build_failed"},)))
|
||||||
|
with pytest.raises(ValueError, match="complete comparison"):
|
||||||
|
replace(report, scans=(report.scans[0], failed))
|
||||||
Reference in New Issue
Block a user