mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
feat(pre-push-safety-net): replay immutable revisions through the full transaction
This commit is contained in:
@@ -0,0 +1,66 @@
|
||||
# Implementation verification: group 5
|
||||
|
||||
Group 5 was authorized by the maintainer's “approved” after group 4 completion.
|
||||
Task 5.1 is verified complete (15/27 total); task 5.2 is next.
|
||||
Real capability eligibility and rollout readiness remain unproven.
|
||||
|
||||
## Execution and review mode
|
||||
|
||||
Fresh implementer dispatch for 5.1 returned `agent thread limit reached`.
|
||||
The [Kiro implementation skill](../../../.opencode/skills/kiro-impl/SKILL.md) explicitly
|
||||
states: “If multi-agent is not available, fall back to manual mode execution for all tasks.”
|
||||
Its manual review rule permits review in the main context when a fresh reviewer is unavailable.
|
||||
This run therefore uses manual implementation and adversarial review, not independent review.
|
||||
|
||||
## Task 5.1 brief
|
||||
|
||||
Wire explicit immutable base/head replay through the existing mapper, ranker, driver,
|
||||
base comparison and admission/report contracts. Preserve the shared elapsed deadline,
|
||||
source/index/ref state, completed evidence on failure, and separate coverage/exit policy.
|
||||
Requirements: 1.1, 2.1, 3.1, 3.4, 4.1, 4.2, 5.5, 7.1, 7.3. Design sections are named:
|
||||
Snapshot Adapter, Ranker Scope Contract, Delta and Actionability Policy, Result and Hook
|
||||
Adapter, Cache and Engine Boundary. No new detector, ranking weights, framework facts,
|
||||
cache or hook installation belongs to this task.
|
||||
|
||||
### Integration findings
|
||||
|
||||
- Snapshot resolution previously used fresh per-command timeouts. An optional shared
|
||||
execution budget now covers constructor probes and explicit revision resolution.
|
||||
- Discovery and provenance are bounded in a Linux child process. The parent drains output
|
||||
incrementally, checks the deadline, kills a stalled worker and requires a valid receipt;
|
||||
exit zero with no result is failure. No project code or engine runs in this child.
|
||||
- The first real controlled PHP replay found the exact request-to-eval witness but novelty
|
||||
was unknown: comparison policy treated another question's context gap as global. The
|
||||
owning task 4.1 policy was repaired here with a failing regression first. Only recognized
|
||||
boundaries attached to a different recorded question can be ignored for that candidate;
|
||||
aggregate coverage retains every gap. Own-question and global boundaries still prevent
|
||||
supported novelty. This is policy ownership repair, not a new upstream graph abstraction.
|
||||
- Artifacts must resolve outside the analyzed repository. The runner refuses a destination
|
||||
that could overwrite live source/index/ref state and preserves the real enforcement result.
|
||||
|
||||
### Evidence in progress
|
||||
|
||||
Initial OFF-state tests: `/tmp/ousast-replay-red.log`, six failures before implementation.
|
||||
Policy ownership regression: `/tmp/ousast-replay-policy-red.log`, one failure before repair.
|
||||
Live-source overwrite regression: `/tmp/ousast-replay-review-red.log`, reproduced before repair.
|
||||
Final validation: 1,236 tests passed, nine skipped; Ruff/format, mypy (112 source files), compileall and all three regression gates passed. Final packaged CLI boot and the controlled positive replay passed. See `benchmarks/measurements/2026-09-13-replay-smoke.json`.
|
||||
|
||||
|
||||
Additional integration repairs retain snapshot omissions in engine change context and distinguish
|
||||
base questions proven tier zero from unfinished queries. The exact required base question still
|
||||
must have completed, valid evidence; budget-deferred work still prevents a complete comparison.
|
||||
RED evidence: `/tmp/ousast-replay-tierzero-red.log` and `/tmp/ousast-replay-snapshot-red.log`.
|
||||
|
||||
## Task 5.1 review verdict
|
||||
- VERDICT: APPROVED
|
||||
- TASK: 5.1
|
||||
- MECHANICAL_RESULTS: Full suite exit 0 (1,236 passed, 9 skipped); Ruff/format/mypy/compileall and detection/map/pair gates PASS; no new placeholder markers or secrets; boundary WITHIN; behavioral RED VERIFIED.
|
||||
- FINDINGS: Integration defects above repaired with regressions; no outstanding task-blocking finding.
|
||||
- SUMMARY: Manual adversarial review approves bounded local replay; this is not independent review or rollout admission.
|
||||
|
||||
## Task 5.1 verification
|
||||
- STATUS: VERIFIED
|
||||
- CLAIM_TYPE: TASK
|
||||
- CLAIM: Explicit local replay traverses the production mapper/ranker/arbiter/comparison/report path and retains uncertainty and shared deadlines.
|
||||
- EVIDENCE: Final packaged controlled security change yields new/source_connection_absent_from_comparable_base. Fixed twin emits no modeled head finding. The packaged deadline control returns incomplete in 30.785 seconds, within 30 + 2 seconds. Exact final package policy/core/facts/query digests match the workspace. Complete commands and captured raw answers live in the measurement artifact.
|
||||
- GAPS: Aggregate coverage remains incomplete, real capabilities are disabled, and lab timings do not satisfy the latency gate. Source-mounted fixed and earlier packaged deadline provenance is explicitly separated in the artifact. PMPro/Node transfer is task 5.2, not inferred from a small PHP control.
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"feature_name": "pre-push-safety-net",
|
||||
"created_at": "2026-09-12T15:35:00Z",
|
||||
"updated_at": "2026-09-13T11:21:07.667199+00:00",
|
||||
"updated_at": "2026-09-13T12:50:25.249104+00:00",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"approvals": {
|
||||
@@ -19,7 +19,7 @@
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": true,
|
||||
"implementation_approval_note": "Groups 1\u20134 verified complete. Tasks 4.1/4.2 independently approved; 4.3 used the documented Kiro manual review fallback after agent thread limit. Remaining tasks require review and fresh verification.",
|
||||
"implementation_approval_note": "Group 5 authorized. Task5.1 verified through manual Kiro fallback review; 5.2 next. Real capabilities remain disabled.",
|
||||
"depends_on": [
|
||||
"contributor-scan"
|
||||
],
|
||||
@@ -60,6 +60,11 @@
|
||||
"at": "2026-09-13T10:26:47.307154+00:00",
|
||||
"source": "move on to group 4 as a whole",
|
||||
"action": "Authorized all of group 4 (4.1\u20134.3), sequential implementation, independent task review and fresh verification. Groups 5\u20138 remain pending."
|
||||
},
|
||||
{
|
||||
"at": "2026-09-13T12:40:57.961577+00:00",
|
||||
"source": "approved",
|
||||
"action": "Authorized group 5 after the group 4 completion summary: explicit transaction replay, then the first PMPro/frozen-core Node experiment. Review and fresh evidence remain required."
|
||||
}
|
||||
],
|
||||
"task_review": {
|
||||
@@ -86,15 +91,17 @@
|
||||
"3.4",
|
||||
"4.1",
|
||||
"4.2",
|
||||
"4.3"
|
||||
"4.3",
|
||||
"5.1"
|
||||
],
|
||||
"remaining_tasks": 13,
|
||||
"verification_record": "implementation-group-4.md",
|
||||
"remaining_tasks": 12,
|
||||
"verification_record": "implementation-group-5.md",
|
||||
"verification_records": [
|
||||
"implementation-group-1.md",
|
||||
"implementation-group-2.md",
|
||||
"implementation-group-3.md",
|
||||
"implementation-group-4.md"
|
||||
"implementation-group-4.md",
|
||||
"implementation-group-5.md"
|
||||
],
|
||||
"next_group": 5,
|
||||
"blocked_tasks": []
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
Status: task plan approved by the maintainer on 2026-09-12. Groups 1–4 (tasks 1.1–4.3)
|
||||
are complete after task reviews and fresh verification (14/27 subtasks). Task 4.3 used
|
||||
the documented manual review fallback after agent dispatch became unavailable.
|
||||
Groups 5–8 remain pending; no production capability or hook latency is admitted.
|
||||
Task 5.1 is also verified (15/27 total); task 5.2 and groups 6–8 remain pending; no production capability or hook latency is admitted.
|
||||
|
||||
## Execution contract
|
||||
|
||||
@@ -140,10 +140,11 @@ Groups 5–8 remain pending; no production capability or hook latency is admitte
|
||||
- _Requirements: 3.1, 3.2, 3.3, 5.1, 5.2, 5.5, 7.3_
|
||||
|
||||
- [ ] 5. Prove the first end-to-end experiment before cache optimization
|
||||
- [ ] 5.1 Wire explicit base/head replay through the full transaction
|
||||
- [x] 5.1 Wire explicit base/head replay through the full transaction
|
||||
- Connect immutable snapshots, ranker decisions, existing arbiter, targeted comparison, admission diagnostics and compact artifacts through an explicit local replay command.
|
||||
- Apply the shared budget to preparation and both revisions; no model/service or installed hook is required. All runtime errors preserve coverage state.
|
||||
- Done when a controlled security change and fixed twin traverse the production runner through the existing ranker and produce replayable results with truthful deadlines.
|
||||
- _Verified 2026-09-13:_ manual Kiro fallback review APPROVED; 1,236 tests passed, nine skipped; final packaged security replay and fixed/deadline controls verified. See `implementation-group-5.md`.
|
||||
- _Boundary: Push runner, CLI, Snapshot Adapter, engine and policy integration_
|
||||
- _Depends: 1.3, 3.1, 3.4, 4.3_
|
||||
- _Requirements: 1.1, 2.1, 3.1, 3.4, 4.1, 4.2, 5.5, 7.1, 7.3_
|
||||
|
||||
Reference in New Issue
Block a user