diff --git a/.kiro/specs/pre-push-safety-net/tasks.md b/.kiro/specs/pre-push-safety-net/tasks.md index e21ac0d..efbc73d 100644 --- a/.kiro/specs/pre-push-safety-net/tasks.md +++ b/.kiro/specs/pre-push-safety-net/tasks.md @@ -385,7 +385,7 @@ Preserve all completed task evidence below. - _Depends: 11.3_ - _Requirements: 4.1, 4.2, 6.2, 6.5_ - _Verified 2026-09-17:_ 1,399 tests passed, nine skipped; Ruff/format/mypy passed. The overlay, the last fixed cost bypassing the session, now runs in-session and the saved graph is verified rather than assumed. The first session-enabled replay then showed correct detections and no speedup, and a targeted probe disproved the sequence hypothesis before finding the real defect: a build runs on a copy of the backend, so the session it started was never closed, leaking a JVM holding the configured heap, twice per transaction, enough to exhaust the container. With that fixed the session stays alive for the whole transaction and no `session_query_failed` appears. Measured on the same pins and budget, the vulnerable revision went from 326.4 s to 243.3 s, head query time from 63.6 s to 32.0 s and base from 85.1 s to 51.2 s, with dominance and configuration each falling from about 14 s to about 4 s. Detections unchanged. Evidence: `benchmarks/measurements/2026-09-17-nodegoat-session-transport.json`. -- [ ] 11.5 Rerun the frozen seven-class runtime profile and decide +- [x] 11.5 Rerun the frozen seven-class runtime profile and decide - Rerun the unchanged seven-class profile: identical-tip, function edit, dependency edit, configuration edit, cold, growth and multi-ref, three samples each, retaining every timeout and unresolved check. Identical-tip reuse cannot substitute for warm changed-code work. - Verify the M1 outcome still holds on the same three revisions through the production rules, so speed is not bought with a lost detection. - Apply the existing gates without alteration: representative warm changed-code p95 at most 30 s, cancellation and reporting at most 2 s, supported-check completion at least 95%. A failed gate stays NO-GO and names the dominant remaining cost. @@ -393,6 +393,7 @@ Preserve all completed task evidence below. - _Boundary: Evaluation harness, runtime integration_ - _Depends: 11.4_ - _Requirements: 4.1, 4.2, 6.2, 6.3, 6.5, 7.3_ + - _Measured 2026-09-17, verdict NO-GO:_ the unchanged seven-class profile, three samples each, session enabled. Every changed-code workload still times out at the 30-second deadline, and identical-tip got worse, p50 rising from 18.5 s to 24.2 s. The cause is recorded in the artifacts rather than inferred: 13 transactions report `session_unavailable:session_startup_timeout`, so under the real deadline the session usually never starts and everything falls back. Startup is 9 to 13 seconds, about a third of the whole budget, spent before any analysis. Where a session did start, on cached-graph transactions with little remaining work, it cost more than it saved. Correctness is unchanged: the same three revisions through the production rules with the session on reproduce the M1 verdict. Evidence: `benchmarks/measurements/2026-09-17-session-runtime-profile.json`. **Dominant remaining cost: engine startup itself, which a transaction-owned session relocates but cannot remove.** - [ ] 12. Prerequisites for independent qualification found during M1a and M1b - [ ] 12.1 Extract the operation symbol from the operation, not from the statement @@ -408,6 +409,15 @@ Preserve all completed task evidence below. - _Boundary: contributor-scan query/context evidence_ - _Requirements: 2.1, 2.2, 8.1, 8.2_ +- [ ] 11.6 Decide what to do about engine startup, which the session cannot remove + - **Named by the 11.5 measurement.** A transaction-owned session amortizes engine startup across the invocations inside one transaction, and a warm changed-code push does not have enough of them: startup is 9 to 13 seconds of a 30-second budget, so the session either fails to start or costs more than it saves. Group 11 as designed cannot reach the gate. + - The options are a decision for the maintainer, not an implementation detail, because two of them change the product's shape. A session shared across transactions is a resident local service, which the accepted milestone spec explicitly does not imply. A smaller budget for what a push analyses is a scope change, not a speed change. A faster cold path would mean attacking the frontend and overlay costs directly, which the session does not touch. + - Record the measured cost of each candidate before choosing, and keep the 30-second and 2-second gates unchanged; a gate met by analysing less is not the same result. + - Done when one record states the candidates, their measured or estimated costs, their effect on the product's shape, and the maintainer's choice. + - _Boundary: contributor-scan runtime, release direction_ + - _Depends: 11.5_ + - _Requirements: 4.1, 4.2, 6.5_ + ## Implementation Notes - Task5.2 measured PMPro timing out after both snapshots were materialized but before change context/graph analysis. Group6 must account for compatible snapshot/discovery reuse as well as graph reuse; identical-tip hits cannot substitute for the later representative changed-code latency gate. No new detector, scope heuristic or incremental CPG mutation is implied. diff --git a/benchmarks/measurements/2026-09-17-session-runtime-profile.json b/benchmarks/measurements/2026-09-17-session-runtime-profile.json new file mode 100644 index 0000000..fe199b2 --- /dev/null +++ b/benchmarks/measurements/2026-09-17-session-runtime-profile.json @@ -0,0 +1,147 @@ +{ + "workloads": { + "identical-tip": { + "population": 3, + "observed": 3, + "missing": 0, + "latency_samples": 3, + "p50_seconds": 24.163701707031578, + "p95_seconds": 24.394183435942978, + "timeouts": 0, + "completed_targets": 0, + "expected_targets": 3, + "complete_target_cases": 0, + "completion": 0.0, + "complete_coverage_cases": 0, + "cancellation_overrun_seconds": 0, + "pass": false + }, + "function-edit": { + "population": 3, + "observed": 3, + "missing": 0, + "latency_samples": 3, + "p50_seconds": 30.771415333962068, + "p95_seconds": 30.935717942891642, + "timeouts": 3, + "completed_targets": 0, + "expected_targets": 3, + "complete_target_cases": 0, + "completion": 0.0, + "complete_coverage_cases": 0, + "cancellation_overrun_seconds": 0.9357179428916425, + "pass": false + }, + "dependency-edit": { + "population": 3, + "observed": 3, + "missing": 0, + "latency_samples": 3, + "p50_seconds": 30.8792759970529, + "p95_seconds": 30.88528792699799, + "timeouts": 3, + "completed_targets": 0, + "expected_targets": 3, + "complete_target_cases": 0, + "completion": 0.0, + "complete_coverage_cases": 0, + "cancellation_overrun_seconds": 0.8852879269979894, + "pass": false + }, + "configuration-edit": { + "population": 3, + "observed": 3, + "missing": 0, + "latency_samples": 3, + "p50_seconds": 30.830487370025367, + "p95_seconds": 30.94464882405009, + "timeouts": 3, + "completed_targets": 0, + "expected_targets": 3, + "complete_target_cases": 0, + "completion": 0.0, + "complete_coverage_cases": 0, + "cancellation_overrun_seconds": 0.9446488240500912, + "pass": false + }, + "cold": { + "population": 3, + "observed": 3, + "missing": 0, + "latency_samples": 3, + "p50_seconds": 30.70144796196837, + "p95_seconds": 30.723933225031942, + "timeouts": 3, + "completed_targets": 0, + "expected_targets": 3, + "complete_target_cases": 0, + "completion": 0.0, + "complete_coverage_cases": 0, + "cancellation_overrun_seconds": 0.7239332250319421, + "pass": false + }, + "growth": { + "population": 3, + "observed": 3, + "missing": 0, + "latency_samples": 3, + "p50_seconds": 31.13915658602491, + "p95_seconds": 31.186890079989098, + "timeouts": 3, + "completed_targets": 0, + "expected_targets": 3, + "complete_target_cases": 0, + "completion": 0.0, + "complete_coverage_cases": 0, + "cancellation_overrun_seconds": 1.186890079989098, + "pass": false + }, + "multi-ref": { + "population": 3, + "observed": 3, + "missing": 0, + "latency_samples": 3, + "p50_seconds": 30.744394473033026, + "p95_seconds": 30.833070866996422, + "timeouts": 3, + "completed_targets": 0, + "expected_targets": 6, + "complete_target_cases": 0, + "completion": 0.0, + "complete_coverage_cases": 0, + "cancellation_overrun_seconds": 0.8330708669964224, + "pass": false + } + }, + "runtime_verdict": "NO-GO", + "rollout": "NO-GO", + "qualification_limit": "Development NodeGoat, n=3 per workload; independent quality and multi-project performance not established.", + "missing_workloads": [], + "profile_sha256": "077254b581db671f6732f72bf15efc7f12d3b3c72d9de16be3033624860a2989", + "priming_completed": false, + "bundle": { + "path": "2026-09-17-session-runtime-profile.json.gz", + "sha256": "34d0b23e5abd052bdd060a2db922f8ad2eb21d0229f81368c8205842e0934ced", + "files": 108 + }, + "experiment": "session-runtime-profile-v1", + "milestone": "M2 task 11.5", + "transport": "engine session enabled via OUSAST_ENGINE_SESSION=1", + "session_outcomes": { + "session_unavailable:session_startup_timeout": 13, + "session_define_failed:census": 1 + }, + "finding": "Under the real 30-second deadline the session almost never starts: 13 transactions recorded session_startup_timeout and fell back to the disposable path. Session startup measured 9 to 13 seconds, roughly a third of the whole budget, and it is spent before any analysis. Where a session did start, on identical-tip transactions whose graphs are cached and whose remaining work is small, it made things worse: p50 rose from 18.5 s in the pre-session profile to 24.2 s. A transaction-owned session relocates engine startup rather than removing it, and it can only amortize that cost across many engine invocations, which a warm changed-code transaction does not have.", + "comparison_with_pre_session_profile": { + "identical-tip p50": { + "before": 18.5, + "after": 24.2 + }, + "changed-code workloads": "all still time out at the 30-second deadline, before and after" + }, + "correctness": "Unchanged. The same three pinned revisions through the production rules with the session enabled reproduce the M1 verdict exactly: benchmarks/measurements/2026-09-17-nodegoat-session-transport.json.", + "limits": [ + "Development NodeGoat, three samples per workload; not independent qualification.", + "Priming completed in 236 s without completing its target, so warm graph reuse is partial." + ] +} diff --git a/benchmarks/measurements/2026-09-17-session-runtime-profile.json.gz b/benchmarks/measurements/2026-09-17-session-runtime-profile.json.gz new file mode 100644 index 0000000..03944a6 Binary files /dev/null and b/benchmarks/measurements/2026-09-17-session-runtime-profile.json.gz differ