feat(pre-push-safety-net): measure frozen PMPro and Node replay feasibility

.kiro/steering/roadmap.md

@@ -46,28 +46,33 @@ engine are out of scope.
 - [ ] pre-push-safety-net — Immutable pushed tips, existing-ranker integration, compatible artifact
   reuse, total deadline, change attribution, actionability admission and compact hook behavior.
   Dependencies: contributor-scan. Requirements/design approved 2026-09-12; tasks generated
-  (8 groups, 27 executable subtasks), independently reviewed and approved; groups 1–4 are complete (14/27 tasks), including the resolved frontend-retention prerequisite.
+  (8 groups, 27 executable subtasks), independently reviewed and approved; groups 1–5 are implemented and verified (16/27 tasks), including the resolved frontend-retention prerequisite.
 
 ### Next concrete step
 
-Groups 1–4 are verified: runtime, inputs, scoring, snapshots, shared deadline,
-ranker scope, change context, vendor-free partitions, novelty comparison, actionability
-admission and compact reporting. Tasks 4.1/4.2 passed independent review; 4.3 used the
-Kiro manual review fallback after the agent service reached its thread limit.
-Group 5 is next: wire explicit full transaction replay, then run the first bounded
-experiment. Real capabilities remain disabled pending independent evaluation.
-Automatic dominance-only change-context projection remains an upstream limit to
-retain during replay integration. See `pre-push-safety-net/implementation-group-4.md`
-for verification and scope limits.
-The first complete experiment is 5.2, before persistent caching and hook
-installation work. Keep PMPro as the primary core-optimization example, then freeze the core and
-prove a changed security flow and its fixed twin on a JavaScript/Express Node API workload through
-the same ranker, with a real deadline and vendor-free graph. Include benign pushes and retain Python
-regressions. `pre-push-safety-net/evaluation.md` proposes NodeGoat for integration, a separate untouched
-Node API repository for generalization, and libpng for C envelope/abstraction-limit evidence. Shared
-abstract evidence and decisions are mandatory; framework facts/adaptations are reported separately.
-Measure realistic changed-code cache behavior before more ranker tuning. Broader runtime and
-capability admission follow that result; an LLM ranker is optional.
+Groups 1–5 now connect immutable replay to the existing ranker, detector, comparison,
+admission and compact artifacts. The first frozen PMPro/Node experiment is **NO-GO**:
+all six cold security/fixed/benign cases timed out without completed target checks.
+After repairing post-deadline semantic planning and a repeated availability probe,
+every final CLI run saved its artifact and returned in 30.55–30.64 seconds. PMPro spent
+its deadline in preparation/discovery after materializing both trees; Node reached the
+driver but did not complete checks. The exact 11-case population remains unresolved.
+No normal capability is enabled and these six cold samples establish no warm p95 claim.
+
+Group 6 is next for valid graph/result identity, leases and reuse. Account for snapshot
+and discovery costs before treating graph persistence as the sole optimization; record
+compatible preparation reuse alongside graph reuse. Representative changed-code warm
+latency and coverage must still be measured later, without substituting identical-tip hits.
+Do not change ranking weights, add a second source-slicer or assume incremental Joern
+mutation from this negative performance evidence.
+
+Tasks 4.3 and group 5 used the documented Kiro manual review fallback after the agent
+thread limit; this is not independent review. See `pre-push-safety-net/implementation-group-5.md`
+and `benchmarks/measurements/2026-09-13-first-replay-experiment.json` for provenance,
+initial instrument failures, their repairs and the newly frozen final trial. The NodeGoat
+teaching workload did not cause a framework/fact adaptation or ranking retune. Ghost remains
+untouched; independent PHP/Python eligibility populations and C/C++ property support remain
+separate gaps. Automatic dominance-only context projection is still an upstream limit.
 
 ## Historical roadmap narrative — September 9–11
 

.kiro/steering/safety-net.md

@@ -79,11 +79,16 @@ updates as a shortcut to an unmeasured latency promise.
 The product direction above is user-requested. The amended requirements, design and initial
 evaluation profile in `pre-push-safety-net` were approved for task generation on 2026-09-12
 ("ok, next kiro phase"). The maintainer then approved the task plan and starting group 1
-("approve.") and then group 2 ("ok, next kiro phase, approved"). Groups 1–4 are now verified
-complete (14/27 total), including the approved frontend-retention prerequisite.
-Group 4 adds bound novelty, operation-specific admission and compact artifacts;
-4.1/4.2 passed independent review and 4.3 used the documented Kiro manual fallback
-after agent dispatch became unavailable. Group 5 is next for full replay and the
-first bounded experiment. Task completion still requires review and fresh evidence.
-Real capability eligibility, automatic dominance context projection and hook
-performance remain unproven; the default capability registry is empty.
+("approve.") and then group 2 ("ok, next kiro phase, approved"). Groups 1–5 are now
+implemented, reviewed and verified (16/27), including the frontend-retention prerequisite
+and the deadline regression exposed by full Node replay. Tasks 4.3 and group 5 used the
+explicit Kiro manual review fallback after the agent thread limit.
+
+The first newly frozen PMPro/Node experiment is measured **NO-GO**: all six cold cases
+timed out without completed supported target checks, while retaining all 11 declared cases.
+The repaired CLI saved all artifacts and returned in 30.55–30.64 seconds. PMPro exhausted
+the deadline in preparation/discovery before graph analysis, so compatible preparation
+reuse matters alongside graph persistence. Group 6 is next; warm changed-code performance
+and useful coverage remain mandatory later gates. No ranking/fact adaptation or holdout
+inspection occurred in the final transfer run. Real capability eligibility and automatic
+dominance context projection remain unproven; the default capability registry is empty.