mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
feat(pre-push-safety-net): render bounded reports and preserve full evidence
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# Brief: pre-push-safety-net
|
||||
|
||||
Status: requirements, design and tasks approved on 2026-09-12; groups 1–3 verified complete (11/27 subtasks), including the approved contributor-scan frontend-retention prerequisite; groups 4–8 pending. Language: English.
|
||||
Status: requirements, design and tasks approved; groups 1–4 verified complete (14/27 subtasks). Groups 5–8 pending; real capability eligibility and hook latency remain unproven. Language: English.
|
||||
|
||||
## Problem
|
||||
|
||||
|
||||
@@ -4,9 +4,10 @@ Date: 2026-09-13. The maintainer authorized group 4 as a whole. Scope is tasks
|
||||
4.1–4.3: targeted novelty comparison, actionability admission and compact reporting.
|
||||
Groups 1–3 are verified at `6ba2b06`; full transaction replay remains task 5.1.
|
||||
|
||||
Status: implementation in progress. Task completion requires independent review
|
||||
and fresh evidence. No production capability is enabled for normal hook alerts;
|
||||
capability admission still requires task 8.3 evaluation.
|
||||
Status: tasks 4.1–4.3 are VERIFIED (TASK scope), bringing the spec to 14/27 tasks.
|
||||
Tasks 4.1 and 4.2 passed independent review. Task 4.3 used the Kiro manual review
|
||||
fallback after the agent service rejected further dispatch. No production capability
|
||||
is enabled for normal hook alerts; actual admission still requires task 8.3 evaluation.
|
||||
|
||||
## Verification boundary
|
||||
|
||||
@@ -62,3 +63,64 @@ default registry. Explicitly synthetic PHP/WordPress and JavaScript/Express cont
|
||||
receive equivalent admission and enforcement behavior. They do not qualify any
|
||||
real capability; task 8.3 still owns actual eligibility. Task 4.2 brings progress to
|
||||
13/27 tasks. Compact reporting remains in this authorized group.
|
||||
|
||||
## Task 4.3 verification
|
||||
|
||||
The agent service rejected the task 4.3 dispatch with `agent thread limit reached`.
|
||||
The [kiro-impl skill](../../../.opencode/skills/kiro-impl/SKILL.md) explicitly provides:
|
||||
“If multi-agent is not available, fall back to manual mode execution for all tasks.”
|
||||
Implementation, review and completion checks therefore ran in the main context.
|
||||
This is a manual review record, not an independent-review claim.
|
||||
|
||||
OFF controls produced seven relevant failures; ON and unflagged reporting tests
|
||||
passed. Manual review found and repaired an invalid artifact path escaping the
|
||||
notice path and a complete comparison carrying failed base evidence. A writer
|
||||
exiting zero without a receipt is rejected. Stalled-writer tests prove cancellation,
|
||||
preservation of an existing artifact and no surviving child. Other controls cover
|
||||
all candidate/scan preservation, five defects with only three displayed, separate
|
||||
coverage/enforcement, escaping and bounded long text, and immutable provenance.
|
||||
|
||||
## Review Verdict
|
||||
- VERDICT: APPROVED
|
||||
- TASK: 4.3
|
||||
- REVIEW_MODE: Kiro manual fallback; agent dispatch unavailable.
|
||||
- MECHANICAL_RESULTS: final full suite 1,222 passed, nine skipped in 102.39 seconds;
|
||||
Ruff and format (238 files), mypy (111 source files), compileall, diff checks and
|
||||
detection/map/pair gates PASS. No new placeholders or hardcoded secrets.
|
||||
- FINDINGS: Invalid destination and failed-base completeness issues repaired and
|
||||
verified with failing controls followed by passing tests. No outstanding task finding.
|
||||
- BOUNDARY: Reporting consumes existing admission and engine records; no new
|
||||
detector, ranker, graph mutation, provider or installed hook.
|
||||
|
||||
The image build and CLI boot passed. The packaged integration control rechecked
|
||||
captured actual PHP/JavaScript answers through current comparison/admission/reporting,
|
||||
preserving complete scan payloads and default diagnostic-only output. Five explicitly
|
||||
synthetic admitted defects retain all five records, display three summaries and keep
|
||||
the block decision. Artifact write failure also retains that decision. Packaged
|
||||
policy/reporter hashes match the workspace. See
|
||||
`benchmarks/measurements/2026-09-13-compact-report-smoke.json`.
|
||||
|
||||
Publication is atomic from mode-0600 temporary files. A separate Linux process
|
||||
bounds serialization and filesystem work by the remaining transaction/report
|
||||
allowance. Hard cancellation can leave a private unpublished partial file; it is
|
||||
never presented as the completed artifact. Caller-supplied stage timings are
|
||||
preserved, and the delivery receipt measures publication/rendering separately.
|
||||
Observed packaged delivery costs (about 0.02–0.04 seconds for these controls) do not
|
||||
establish whole-hook latency.
|
||||
|
||||
## Group integration validation
|
||||
|
||||
- DECISION: GO
|
||||
- VALIDATION_SCOPE: Tasks 4.1–4.3 only; no full-feature or release admission.
|
||||
- CROSS_TASK_CONTRACTS: Actual captured engine evidence → bound delta → empty-registry
|
||||
diagnostics → compact report and complete artifact PASS. Explicitly synthetic
|
||||
calibrated controls → admission → advisory/blocking → capped rendering PASS.
|
||||
- REQUIREMENTS: Group-owned portions of 2.4, 3.1–3.5, 4.1, 5.1–5.5, 7.3 and 8.1 are
|
||||
covered by the implementation and controls above. Full transaction wiring and
|
||||
representative capability/latency evaluation remain later tasks.
|
||||
- DESIGN_ALIGNMENT: Existing arbiters and ranker retained; policy and report live
|
||||
in their planned push-adapter boundary. No orphaned module or upstream detector change.
|
||||
- BLOCKED_TASKS: None in group 4. Automatic dominance change-context projection is
|
||||
a documented upstream integration limit, not erased by these declared controls.
|
||||
- NEXT: Group 5 full replay and first bounded experiment. Complete that experiment
|
||||
before persistent graph reuse in group 6. Groups 5–8 remain pending.
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"feature_name": "pre-push-safety-net",
|
||||
"created_at": "2026-09-12T15:35:00Z",
|
||||
"updated_at": "2026-09-13T10:26:47.307154+00:00",
|
||||
"updated_at": "2026-09-13T11:21:07.667199+00:00",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"approvals": {
|
||||
@@ -19,7 +19,7 @@
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": true,
|
||||
"implementation_approval_note": "Groups 1\u20133 verified complete. Maintainer explicitly authorized group 4 as a whole; each task requires independent review and fresh verification.",
|
||||
"implementation_approval_note": "Groups 1\u20134 verified complete. Tasks 4.1/4.2 independently approved; 4.3 used the documented Kiro manual review fallback after agent thread limit. Remaining tasks require review and fresh verification.",
|
||||
"depends_on": [
|
||||
"contributor-scan"
|
||||
],
|
||||
@@ -85,9 +85,10 @@
|
||||
"3.3",
|
||||
"3.4",
|
||||
"4.1",
|
||||
"4.2"
|
||||
"4.2",
|
||||
"4.3"
|
||||
],
|
||||
"remaining_tasks": 14,
|
||||
"remaining_tasks": 13,
|
||||
"verification_record": "implementation-group-4.md",
|
||||
"verification_records": [
|
||||
"implementation-group-1.md",
|
||||
@@ -95,7 +96,7 @@
|
||||
"implementation-group-3.md",
|
||||
"implementation-group-4.md"
|
||||
],
|
||||
"next_group": 4,
|
||||
"next_group": 5,
|
||||
"blocked_tasks": []
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,8 +1,9 @@
|
||||
# Implementation Plan: pre-push-safety-net
|
||||
|
||||
Status: task plan approved by the maintainer's "approve." on 2026-09-12. Groups 1–3 (tasks 1.1–3.4)
|
||||
are complete after independent review and fresh verification (13/27 subtasks, including tasks 4.1–4.2). Task 4.3 and groups 5–8 remain pending;
|
||||
the hook and capability admission are not yet implemented. See `implementation-group-1.md` and `implementation-group-2.md`.
|
||||
Status: task plan approved by the maintainer on 2026-09-12. Groups 1–4 (tasks 1.1–4.3)
|
||||
are complete after task reviews and fresh verification (14/27 subtasks). Task 4.3 used
|
||||
the documented manual review fallback after agent dispatch became unavailable.
|
||||
Groups 5–8 remain pending; no production capability or hook latency is admitted.
|
||||
|
||||
## Execution contract
|
||||
|
||||
@@ -112,7 +113,7 @@ the hook and capability admission are not yet implemented. See `implementation-g
|
||||
- _Requirements: 2.2, 2.3, 7.3, 8.2, 8.4_
|
||||
- _Verified 2026-09-13:_ independent review APPROVED; 1,151 passed, nine skipped; Ruff/format/mypy pass. Real PHP/JavaScript graphs retain tests/browser source, omit vendor nodes and share one ranker limit. Named census survives PHP aggregation; missing source names and unreadable discovery inputs remain explicit. Python exercises all three batch census schemas. Evidence: `benchmarks/measurements/2026-09-13-partition-scope-smoke.json`. No hook capability or latency claim.
|
||||
|
||||
- [ ] 4. Implement change attribution and actionable output
|
||||
- [x] 4. Implement change attribution and actionable output
|
||||
- [x] 4.1 Compare targeted base/head evidence for defect novelty
|
||||
- Establish stable mechanism/operation identity with rename/span mapping and collect comparable base evidence under the same semantics and remaining deadline.
|
||||
- Distinguish new, worsened, unchanged and unknown; an absent finding from a failed/truncated base is unknown. Support removed guards and new source connections at unchanged sinks.
|
||||
@@ -130,10 +131,11 @@ the hook and capability admission are not yet implemented. See `implementation-g
|
||||
- _Verified 2026-09-13:_ independent review APPROVED after operation/context-binding repair; 1,207 tests passed, nine skipped; default real-evidence controls remain diagnostic and synthetic PHP/JavaScript admission is equivalent. See `implementation-group-4.md`.
|
||||
- _Boundary: Delta and Actionability Policy_
|
||||
- _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5, 5.3, 5.4, 8.1_
|
||||
- [ ] 4.3 Render compact results and preserve complete artifacts
|
||||
- [x] 4.3 Render compact results and preserve complete artifacts
|
||||
- Render up to three actionable summaries plus remaining count; emit at most one no-finding status and one aggregated incomplete notice with a useful next action.
|
||||
- Persist all defects, candidate dispositions, exact scope, identities, capability/ranker provenance and timings with separate finding/coverage/push status.
|
||||
- Done when the display cap does not change the push decision, incomplete analysis cannot read as secure, and artifact-write failure produces a bounded notice while retaining the real decision.
|
||||
- _Verified 2026-09-13:_ manual Kiro review fallback APPROVED after dispatch thread limit; 1,222 tests passed, nine skipped. Packaged comparison/admission/report smoke and CLI boot passed; full records retained with bounded output/write failure handling. See `implementation-group-4.md`.
|
||||
- _Boundary: Result and Hook Adapter_
|
||||
- _Requirements: 3.1, 3.2, 3.3, 5.1, 5.2, 5.5, 7.3_
|
||||
|
||||
|
||||
@@ -46,17 +46,19 @@ engine are out of scope.
|
||||
- [ ] pre-push-safety-net — Immutable pushed tips, existing-ranker integration, compatible artifact
|
||||
reuse, total deadline, change attribution, actionability admission and compact hook behavior.
|
||||
Dependencies: contributor-scan. Requirements/design approved 2026-09-12; tasks generated
|
||||
(8 groups, 27 executable subtasks), independently reviewed and approved; groups 1–3 are complete (11/27 tasks), including the resolved frontend-retention prerequisite.
|
||||
(8 groups, 27 executable subtasks), independently reviewed and approved; groups 1–4 are complete (14/27 tasks), including the resolved frontend-retention prerequisite.
|
||||
|
||||
### Next concrete step
|
||||
|
||||
Groups 1–3 have passed independent review: runtime, inputs, scoring, snapshots,
|
||||
shared deadline, exact ranker scope, change-context evidence and vendor-free language
|
||||
partitions. The approved contributor-scan prerequisite 2.18 (`51fabef`) repairs both
|
||||
JavaScript test filters; real graphs retain first-party tests and exclude vendor code.
|
||||
Task 3.4 also repairs unreadable-input handling and preserves named PHP census evidence.
|
||||
Group 4 is next: novelty comparison, actionability admission and compact advisory output.
|
||||
See `pre-push-safety-net/implementation-group-3.md` for evidence and limits.
|
||||
Groups 1–4 are verified: runtime, inputs, scoring, snapshots, shared deadline,
|
||||
ranker scope, change context, vendor-free partitions, novelty comparison, actionability
|
||||
admission and compact reporting. Tasks 4.1/4.2 passed independent review; 4.3 used the
|
||||
Kiro manual review fallback after the agent service reached its thread limit.
|
||||
Group 5 is next: wire explicit full transaction replay, then run the first bounded
|
||||
experiment. Real capabilities remain disabled pending independent evaluation.
|
||||
Automatic dominance-only change-context projection remains an upstream limit to
|
||||
retain during replay integration. See `pre-push-safety-net/implementation-group-4.md`
|
||||
for verification and scope limits.
|
||||
The first complete experiment is 5.2, before persistent caching and hook
|
||||
installation work. Keep PMPro as the primary core-optimization example, then freeze the core and
|
||||
prove a changed security flow and its fixed twin on a JavaScript/Express Node API workload through
|
||||
|
||||
@@ -79,10 +79,11 @@ updates as a shortcut to an unmeasured latency promise.
|
||||
The product direction above is user-requested. The amended requirements, design and initial
|
||||
evaluation profile in `pre-push-safety-net` were approved for task generation on 2026-09-12
|
||||
("ok, next kiro phase"). The maintainer then approved the task plan and starting group 1
|
||||
("approve.") and then group 2 ("ok, next kiro phase, approved"). Groups 1–3 are now verified
|
||||
complete (11/27 total). The maintainer approved the frontend-retention prerequisite;
|
||||
contributor-scan 2.18 and pre-push task 3.4 have both passed independent review and
|
||||
fresh verification. Group 4 is next: novelty comparison, actionability admission and
|
||||
compact advisory output. Implementation is authorized under the approved plan;
|
||||
task completion still requires independent review and fresh evidence. Hook performance
|
||||
and capability admission remain unproven.
|
||||
("approve.") and then group 2 ("ok, next kiro phase, approved"). Groups 1–4 are now verified
|
||||
complete (14/27 total), including the approved frontend-retention prerequisite.
|
||||
Group 4 adds bound novelty, operation-specific admission and compact artifacts;
|
||||
4.1/4.2 passed independent review and 4.3 used the documented Kiro manual fallback
|
||||
after agent dispatch became unavailable. Group 5 is next for full replay and the
|
||||
first bounded experiment. Task completion still requires review and fresh evidence.
|
||||
Real capability eligibility, automatic dominance context projection and hook
|
||||
performance remain unproven; the default capability registry is empty.
|
||||
|
||||
Reference in New Issue
Block a user