feat(pre-push-safety-net): measure frozen PMPro and Node replay feasibility

.kiro/specs/pre-push-safety-net/brief.md

@@ -1,6 +1,6 @@
 # Brief: pre-push-safety-net
 
-Status: requirements, design and tasks approved; groups 1–4 verified complete (14/27 subtasks). Groups 5–8 pending; real capability eligibility and hook latency remain unproven. Language: English.
+Status: requirements, design and tasks approved; groups 1–5 implemented and verified (16/27 subtasks). The first PMPro/Node experiment is measured NO-GO: six cold timeouts, no completed target checks. Groups 6–8 pending; normal capabilities remain disabled and warm hook latency unproven. Language: English.
 
 ## Problem
 

.kiro/specs/pre-push-safety-net/implementation-group-5.md

@@ -1,7 +1,7 @@
 # Implementation verification: group 5
 
 Group 5 was authorized by the maintainer's “approved” after group 4 completion.
-Task 5.1 is verified complete (15/27 total); task 5.2 is next.
+Tasks 5.1–5.2 are implemented and verified (16/27 total); the measured feasibility decision is NO-GO.
 Real capability eligibility and rollout readiness remain unproven.
 
 ## Execution and review mode
@@ -105,3 +105,80 @@ facts or detector semantics changed. The additional regression failed before rep
 - CLAIM: The reproduced Node cancellation path retains diagnostics and returns within 30 + 2 seconds.
 - EVIDENCE: Final packaged real Node replay returned in 30.1255 seconds with one complete diagnostic artifact and incomplete coverage. Final policy/core/facts/query hashes match the workspace. See `benchmarks/measurements/2026-09-13-replay-deadline-repair.json` for initial/partial-repair traces and final raw artifact.
 - GAPS: This proves the reproduced cancellation path, not representative warm performance or security coverage. Normal capabilities remain disabled.
+
+## Task 5.2 brief and measurement method
+
+Requirements 6.2–6.5 and 8.1–8.4: execute the prepared PMPro and Node security/fixed/benign
+changes through full replay, freeze the core before transfer, retain unresolved populations,
+and make an explicit feasibility decision. This task owns the evaluation harness, not new
+engine abstractions, framework facts, ranking weights or normal alert eligibility.
+
+The harness freezes the full input population and runtime profile before invoking the CLI.
+Authored variants are reproducible commits in a private Git object store, with byte/hash
+checks and the rest of each tree retained. The original cache is mounted read-only. All six
+cases run under 30 seconds plus a 2-second cancellation/report allowance, with complete CLI
+elapsed time measured. Exact artifacts/process exits remain separate from the scorer's
+witness association. Empty completed rows cannot manufacture a witnessed negative.
+
+Initial OFF-state evidence: `/tmp/ousast-experiment-red.log`, three failed acceptance tests.
+After implementation and flag removal, the same three tests pass, including whole-tree
+retention, unchanged source/index, reproducible authored identity, bad-hash refusal,
+timeout accounting and absence of fabricated negative evidence.
+
+The first v1 trial is retained as diagnostic history. Its Node post-deadline failures
+triggered the predecessor repair above; PMPro measurements overlapped regression tests.
+The v2 profile is frozen anew after repair; final timed runs have no competing test/build
+workload from this session. Both profiles and their distinct core/policy hashes are retained.
+The reserved Ghost workload remains untouched. No framework/fact adaptation or ranking
+retune is made from these Node results.
+
+## Final experiment: measured NO-GO
+
+[Joint scorecard](../../../benchmarks/measurements/2026-09-13-first-replay-experiment.json)
+records the final v2 profile, all metrics, hardware, exact case identities, initial trial
+and final-trial bundle hashes. The two gzip bundles are JSON mappings from each original
+filename to its exact UTF-8 contents; decompression was checked against every original file.
+They preserve all detailed CLI artifacts, stdout/stderr, pins, profiles and scorer inputs.
+
+| Workload | Cases | Whole CLI elapsed | Reached analysis | Result |
+|---|---:|---|---|---|
+| PMPro | 3 | 30.61–30.64 s | No; both trees materialized, discovery/preparation expired before change context | Timeout, no completed target check |
+| NodeGoat | 3 | 30.55–30.58 s | Driver reached after 7.78–7.96 s preparation; deadline consumed in head analysis/planning | Timeout, 436 deferred questions per case |
+
+All six processes exited 0 (advisory allow), saved artifacts and explicitly reported
+incomplete analysis. The harness exited **1 with a completed NO-GO decision**, distinct
+from an instrument crash. Both the initial and post-PMPro freeze match each final runtime
+and the current workspace's core/facts/query/policy hashes. CPU: Intel Core i7-8700,
+four exposed CPUs, 3 GiB container memory limit, network disabled. Tests/builds were not
+running during these final measurements. Filesystem caches were not flushed.
+
+The scorer retains all 11 unresolved cases; six were executed and the regression,
+unsupported-envelope and reserved populations were not scanned. No security change was
+recovered within budget, no witnessed fixed-side negative was established, and no normal
+alert was admitted. Precision has denominator zero and is undefined. The two benign
+pushes emitted no alert, but an empty registry cannot establish product usefulness.
+There are no warm-changed or identical-tip samples. Cold sample p50 is 30.58 s and p95
+30.64 s for six one-shot deadline-limited runs; these are not warm latency acceptance evidence.
+
+The negative result points first to preparation/discovery cost for PMPro, before graph
+persistence can help that path. Group6 can implement valid reuse, including compatible
+preparation work, but must not claim scale from identical-tip hits or bypass the ranker
+with another slicer. No new abstract capability or incremental graph mutation is justified
+by this measurement. Normal capability qualification remains task8.3.
+
+## Task 5.2 review verdict
+- VERDICT: APPROVED
+- TASK: 5.2
+- MECHANICAL_RESULTS: Full suite 1,242 passed, nine skipped, exit 0; Ruff/format (243 files), mypy (112 source files), compileall and detection/map/pair gates PASS; three behavioral OFF-state failures reproduced before implementation; no new placeholder markers/secrets; evaluation boundary respected.
+- FINDINGS: Initial instrument failures repaired in their owning driver/replay integration before the final freeze. All six final runtime provenance records match the frozen profile; scorer replay and complete compressed-artifact round trips reproduce exactly. No outstanding implementation blocker.
+- SUMMARY: Manual review approves the executed diagnostic experiment and truthful NO-GO assessment; it does not approve usefulness, warm latency or capability admission.
+
+## Group 5 verification and integration assessment
+- STATUS: VERIFIED
+- CLAIM_TYPE: TASK
+- CLAIM: Tasks5.1–5.2 implement full local replay and execute the first frozen PMPro/Node experiment with truthful evidence, failures, deadline accounting and an explicit feasibility decision.
+- EVIDENCE: Controlled security/fixed replay, packaged deadline repair, 1,242 passing tests, fresh static/regression checks, six final real CLI replays, exact frozen runtime hashes and reproducible 11-case scorecard. Task5.2 acceptance requires publishing the actual decision; this decision is NO-GO.
+- GAPS: No completed target checks in the 30-second cold workload, no warm measurements, no independent eligibility, and remaining groups6–8. The first usefulness/quality/latency milestone has not passed; the feature is not complete.
+
+Implementation and measurement integration: GO for the scoped work in group5.
+Product feasibility and rollout: **NO-GO**. Kiro state is 16/27 tasks verified; group6 next.

.kiro/specs/pre-push-safety-net/spec.json

@@ -1,7 +1,7 @@
 {
   "feature_name": "pre-push-safety-net",
   "created_at": "2026-09-12T15:35:00Z",
-  "updated_at": "2026-09-13T13:11:42.850127+00:00",
+  "updated_at": "2026-09-13T13:22:55.634332+00:00",
   "language": "en",
   "phase": "implementation",
   "approvals": {
@@ -19,7 +19,7 @@
     }
   },
   "ready_for_implementation": true,
-  "implementation_approval_note": "Group5 authorized. Task5.1 and its observed deadline regression repaired and verified with manual Kiro fallback; task5.2 resumes under a newly frozen core identity.",
+  "implementation_approval_note": "Groups1\u20135 implemented and verified (16/27). Group5 used manual Kiro fallback review, repaired observed predecessor deadline bugs, and executed a newly frozen PMPro/Node experiment. Measured feasibility NO-GO; real capabilities remain disabled. Groups6\u20138 pending.",
   "depends_on": [
     "contributor-scan"
   ],
@@ -92,9 +92,10 @@
       "4.1",
       "4.2",
       "4.3",
-      "5.1"
+      "5.1",
+      "5.2"
     ],
-    "remaining_tasks": 12,
+    "remaining_tasks": 11,
     "verification_record": "implementation-group-5.md",
     "verification_records": [
       "implementation-group-1.md",
@@ -103,7 +104,8 @@
       "implementation-group-4.md",
       "implementation-group-5.md"
     ],
-    "next_group": 5,
-    "blocked_tasks": []
+    "next_group": 6,
+    "blocked_tasks": [],
+    "feasibility": "NO-GO: six cold replay timeouts; no completed supported target checks; warm performance and eligibility unproven"
   }
 }

.kiro/specs/pre-push-safety-net/tasks.md

@@ -1,9 +1,11 @@
 # Implementation Plan: pre-push-safety-net
 
-Status: task plan approved by the maintainer on 2026-09-12. Groups 1–4 (tasks 1.1–4.3)
-are complete after task reviews and fresh verification (14/27 subtasks). Task 4.3 used
-the documented manual review fallback after agent dispatch became unavailable.
-Task 5.1 is also verified (15/27 total); task 5.2 and groups 6–8 remain pending; no production capability or hook latency is admitted.
+Status: task plan approved by the maintainer on 2026-09-12. Groups 1–5 (tasks 1.1–5.2)
+are implemented, reviewed and verified (16/27 subtasks). Tasks 4.3 and group 5 used the
+documented Kiro manual review fallback after agent dispatch reached its thread limit.
+The first experiment is measured NO-GO: all six cold PMPro/Node cases timed out without
+completed target checks. Groups 6–8 remain pending; no production capability or warm
+hook latency is admitted.
 
 ## Execution contract
 
@@ -139,7 +141,7 @@ Task 5.1 is also verified (15/27 total); task 5.2 and groups 6–8 remain pendin
   - _Boundary: Result and Hook Adapter_
   - _Requirements: 3.1, 3.2, 3.3, 5.1, 5.2, 5.5, 7.3_
 
-- [ ] 5. Prove the first end-to-end experiment before cache optimization
+- [x] 5. Prove the first end-to-end experiment before cache optimization
 - [x] 5.1 Wire explicit base/head replay through the full transaction
   - Connect immutable snapshots, ranker decisions, existing arbiter, targeted comparison, admission diagnostics and compact artifacts through an explicit local replay command.
   - Apply the shared budget to preparation and both revisions; no model/service or installed hook is required. All runtime errors preserve coverage state.
@@ -148,11 +150,12 @@ Task 5.1 is also verified (15/27 total); task 5.2 and groups 6–8 remain pendin
   - _Boundary: Push runner, CLI, Snapshot Adapter, engine and policy integration_
   - _Depends: 1.3, 3.1, 3.4, 4.3_
   - _Requirements: 1.1, 2.1, 3.1, 3.4, 4.1, 4.2, 5.5, 7.1, 7.3_
-- [ ] 5.2 Measure PMPro and frozen-core Node transfer
+- [x] 5.2 Measure PMPro and frozen-core Node transfer
   - _Prerequisite repaired 2026-09-13:_ 3.1/5.1 deadline regression verified in the packaged Node replay; re-freeze the experiment after this core identity change.
   - Replay each prepared security/fixed/benign case, first using the declared hook deadline; record timeouts and stage costs. A separately named extended lab run may obtain witnesses but cannot satisfy the hook latency gate.
   - Freeze the core after PMPro, run Node with existing facts, and classify instrument, mapping, fact, scope and domain gaps without tuning on an untouched holdout.
   - Done when one joint artifact reports actual evidence and outcomes for both workloads, with unresolved cases retained and an explicit feasibility decision; no normal capability is enabled here.
+  - _Verified 2026-09-13:_ manual Kiro review APPROVED; six final CLI cases measured under unchanged core, all timed out in 30.55–30.64 seconds with artifacts retained. Full 11-case population retained; feasibility NO-GO, no eligibility enabled. 1,242 tests passed, nine skipped. See `implementation-group-5.md`.
   - _Boundary: Evaluation harness, full-run integration_
   - _Depends: 1.2, 1.3, 5.1_
   - _Requirements: 6.2, 6.3, 6.4, 6.5, 8.1, 8.2, 8.3, 8.4_
@@ -234,3 +237,9 @@ Task 5.1 is also verified (15/27 total); task 5.2 and groups 6–8 remain pendin
   - Done when a reproducible validation artifact states GO only with all mandatory integration and joint quality/latency gates satisfied; otherwise it names the remaining NO-GO conditions without marking the feature complete.
   - _Boundary: Packaged runtime, full feature validation integration_
   - _Requirements: 3.1, 3.4, 5.1, 5.2, 5.3, 5.4, 5.5, 6.5, 7.1, 7.3, 7.4, 8.3, 8.4_
+
+
+## Implementation Notes
+
+- Task5.2 measured PMPro timing out after both snapshots were materialized but before change context/graph analysis. Group6 must account for compatible snapshot/discovery reuse as well as graph reuse; identical-tip hits cannot substitute for the later representative changed-code latency gate. No new detector, scope heuristic or incremental CPG mutation is implied.
+- Cancellation must cover semantic planning and reporting, not only engine subprocesses: `_collect` and `_scope_work` now consume the shared deadline, and replay instantiates the budget-aware backend once. Preserve the 400-question failure-census and real Node artifact-retention regressions.

.kiro/steering/roadmap.md

@@ -46,28 +46,33 @@ engine are out of scope.
 - [ ] pre-push-safety-net — Immutable pushed tips, existing-ranker integration, compatible artifact
   reuse, total deadline, change attribution, actionability admission and compact hook behavior.
   Dependencies: contributor-scan. Requirements/design approved 2026-09-12; tasks generated
-  (8 groups, 27 executable subtasks), independently reviewed and approved; groups 1–4 are complete (14/27 tasks), including the resolved frontend-retention prerequisite.
+  (8 groups, 27 executable subtasks), independently reviewed and approved; groups 1–5 are implemented and verified (16/27 tasks), including the resolved frontend-retention prerequisite.
 
 ### Next concrete step
 
-Groups 1–4 are verified: runtime, inputs, scoring, snapshots, shared deadline,
-ranker scope, change context, vendor-free partitions, novelty comparison, actionability
-admission and compact reporting. Tasks 4.1/4.2 passed independent review; 4.3 used the
-Kiro manual review fallback after the agent service reached its thread limit.
-Group 5 is next: wire explicit full transaction replay, then run the first bounded
-experiment. Real capabilities remain disabled pending independent evaluation.
-Automatic dominance-only change-context projection remains an upstream limit to
-retain during replay integration. See `pre-push-safety-net/implementation-group-4.md`
-for verification and scope limits.
-The first complete experiment is 5.2, before persistent caching and hook
-installation work. Keep PMPro as the primary core-optimization example, then freeze the core and
-prove a changed security flow and its fixed twin on a JavaScript/Express Node API workload through
-the same ranker, with a real deadline and vendor-free graph. Include benign pushes and retain Python
-regressions. `pre-push-safety-net/evaluation.md` proposes NodeGoat for integration, a separate untouched
-Node API repository for generalization, and libpng for C envelope/abstraction-limit evidence. Shared
-abstract evidence and decisions are mandatory; framework facts/adaptations are reported separately.
-Measure realistic changed-code cache behavior before more ranker tuning. Broader runtime and
-capability admission follow that result; an LLM ranker is optional.
+Groups 1–5 now connect immutable replay to the existing ranker, detector, comparison,
+admission and compact artifacts. The first frozen PMPro/Node experiment is **NO-GO**:
+all six cold security/fixed/benign cases timed out without completed target checks.
+After repairing post-deadline semantic planning and a repeated availability probe,
+every final CLI run saved its artifact and returned in 30.55–30.64 seconds. PMPro spent
+its deadline in preparation/discovery after materializing both trees; Node reached the
+driver but did not complete checks. The exact 11-case population remains unresolved.
+No normal capability is enabled and these six cold samples establish no warm p95 claim.
+
+Group 6 is next for valid graph/result identity, leases and reuse. Account for snapshot
+and discovery costs before treating graph persistence as the sole optimization; record
+compatible preparation reuse alongside graph reuse. Representative changed-code warm
+latency and coverage must still be measured later, without substituting identical-tip hits.
+Do not change ranking weights, add a second source-slicer or assume incremental Joern
+mutation from this negative performance evidence.
+
+Tasks 4.3 and group 5 used the documented Kiro manual review fallback after the agent
+thread limit; this is not independent review. See `pre-push-safety-net/implementation-group-5.md`
+and `benchmarks/measurements/2026-09-13-first-replay-experiment.json` for provenance,
+initial instrument failures, their repairs and the newly frozen final trial. The NodeGoat
+teaching workload did not cause a framework/fact adaptation or ranking retune. Ghost remains
+untouched; independent PHP/Python eligibility populations and C/C++ property support remain
+separate gaps. Automatic dominance-only context projection is still an upstream limit.
 
 ## Historical roadmap narrative — September 9–11
 

.kiro/steering/safety-net.md

@@ -79,11 +79,16 @@ updates as a shortcut to an unmeasured latency promise.
 The product direction above is user-requested. The amended requirements, design and initial
 evaluation profile in `pre-push-safety-net` were approved for task generation on 2026-09-12
 ("ok, next kiro phase"). The maintainer then approved the task plan and starting group 1
-("approve.") and then group 2 ("ok, next kiro phase, approved"). Groups 1–4 are now verified
-complete (14/27 total), including the approved frontend-retention prerequisite.
-Group 4 adds bound novelty, operation-specific admission and compact artifacts;
-4.1/4.2 passed independent review and 4.3 used the documented Kiro manual fallback
-after agent dispatch became unavailable. Group 5 is next for full replay and the
-first bounded experiment. Task completion still requires review and fresh evidence.
-Real capability eligibility, automatic dominance context projection and hook
-performance remain unproven; the default capability registry is empty.
+("approve.") and then group 2 ("ok, next kiro phase, approved"). Groups 1–5 are now
+implemented, reviewed and verified (16/27), including the frontend-retention prerequisite
+and the deadline regression exposed by full Node replay. Tasks 4.3 and group 5 used the
+explicit Kiro manual review fallback after the agent thread limit.
+
+The first newly frozen PMPro/Node experiment is measured **NO-GO**: all six cold cases
+timed out without completed supported target checks, while retaining all 11 declared cases.
+The repaired CLI saved all artifacts and returned in 30.55–30.64 seconds. PMPro exhausted
+the deadline in preparation/discovery before graph analysis, so compatible preparation
+reuse matters alongside graph persistence. Group 6 is next; warm changed-code performance
+and useful coverage remain mandatory later gates. No ranking/fact adaptation or holdout
+inspection occurred in the final transfer run. Real capability eligibility and automatic
+dominance context projection remain unproven; the default capability registry is empty.