feat(pre-push-safety-net): reuse compatible artifacts throughout replay

This commit is contained in:
Marius Ciepluch
2026-09-13 17:08:49 +02:00
parent 0ab5d56618
commit 530b9c9879
21 changed files with 2522 additions and 54 deletions
+13 -7
View File
@@ -1,6 +1,6 @@
# Brief: pre-push-safety-net
Status: requirements, design and tasks approved; groups 1–5 implemented and verified (16/27 subtasks). The first PMPro/Node experiment is measured NO-GO: six cold timeouts, no completed target checks. Groups 6–8 pending; normal capabilities remain disabled and warm hook latency unproven. Language: English.
Status: requirements, design and tasks approved; groups 1–6 implemented and verified (20/27 subtasks). Compatible preparation/graph/query reuse is demonstrated, including packaged JavaScript evidence equivalence. Groups 7–8 pending. Rollout remains NO-GO: changed-code feasibility and independent capability qualification are not established; normal capabilities remain disabled. Language: English.
## Problem
@@ -10,12 +10,18 @@ maintainer explicitly rejects noisy advisory output as well as noisy blocking ou
## Current State
The scan driver and Joern arbiters exist, with demonstrated PHP/Python cases. Immutable push-tip
resolution, isolated source materialization and lexical change context are implemented and verified.
Ranker integration, persistent graph caching and an end-to-end deadline remain pending.
The driver builds a whole-root graph before cutting the region budget. The latest PMPro ranking
experiment alone spends about 130 seconds building and 296 seconds extracting evidence. A manually
selected two-file slice finding a CVE does not demonstrate automatic incremental analysis.
The existing ranker, Joern arbiters, immutable replay, change comparison, actionability policy
and absolute deadline are integrated. Vendor code is physically excluded from frontend graphs
and targets. Explicit local caching reuses complete discovery, validated isolated graph copies
and compatible query evidence while reapplying current admission. Changed source/declarations
invalidate affected graph units; no incremental Joern mutation is implemented.
Group 5's PMPro/Node cold experiment remains NO-GO. Group 6 adds a packaged JavaScript
correctness control with equal cached/uncached evidence and admission; an identical-revision
hit is fast, while the changed-code deadline remains a separate unmet gate. Automatic
context projection and independent capability eligibility remain explicit limits. See
implementation-group-6.md and the committed reuse control evidence. Historical whole-PMPro
ranking costs and manually selected CVE slices do not establish automatic push latency.
## Desired Outcome
+5 -3
View File
@@ -147,6 +147,7 @@ New files (proposed, not created by this documentation change):
| `src/openultrasast/push/contracts.py` | Immutable ref/comparison identities and comparison-owned completion/status records |
| `src/openultrasast/push/snapshot.py` | Parse ref updates, resolve bases, materialize blobs, produce change context |
| `src/openultrasast/push/cache.py` | Artifact keys, validity, atomic publication and eviction |
| `src/openultrasast/push/reuse.py` | Complete discovery reuse and backend graph/query reuse through the existing driver |
| `src/openultrasast/push/policy.py` | Delta identity, actionability admission and push disposition |
| `src/openultrasast/push/runner.py` | Shared deadline, ranker/engine orchestration, per-ref result aggregation |
| `src/openultrasast/push/report.py` | Compact terminal result and reproducible push manifest |
@@ -157,8 +158,9 @@ Contract configuration (task 1.4) uses `[push]`: optional `comparison_base`, pos
`deadline_seconds` (30), positive `cancellation_allowance_seconds` (2), positive integer
`cache_max_bytes` (2 GiB), `mode` (`advisory` by default, or `blocking`) and independent
`incomplete_coverage_policy` (`allow` by default, or `block`). Unknown keys, malformed
sections and coerced/nonfinite numeric values are rejected. These settings are inert until
runner integration; they do not enable a hook or admit experimental capabilities. The cache
sections and coerced/nonfinite numeric values are rejected. Experimental replay consumes the deadline, mode, coverage policy and optional cache budget.
An explicit `--cache-dir` enables local reuse; omission retains a cold control. These settings
do not install a hook or admit experimental capabilities. The cache
limit is a local storage budget, not a latency claim. Completion records are owned by an
exact comparison so reused head graphs cannot merge evidence from different bases. Live
leases have no JSON constructor; only the backend can validate and issue them in task 6.1.
@@ -251,7 +253,7 @@ cleanup consume the same execution deadline and cancellation allowance as the ba
Runtime origin is `unspecified` when no declaration establishes browser/server scope;
JavaScript presence alone does not admit an Express capability. TypeScript and C/C++
retain explicit capability limits. These records prepare later artifact manifests;
they do not implement persistent graph storage.
task 6 now binds them to backend-owned persistent graph manifests and isolated leases.
Graph key: digest of every included blob/path and applicable declarations, frontend/engine version,
overlay semantics/options, language, and exclusions. Query key additionally includes graph key,
@@ -103,3 +103,96 @@ the cache never confers eligibility. JSON decoding is non-executable and bounded
- CLAIM: Task 6.3 semantic keys and completed comparison evidence reuse.
- EVIDENCE: production policy integration and cache key tests, 1260 passing suite tests and final static checks.
- GAPS: Graph/query/preparation reuse must still be connected to replay under 6.4; no rollout claim.
## Task 6.4 brief
Requirements 1.1, 2.2, 4.1, 4.2, 4.3, 4.4 and 7.3; design: Push Runner and
Cache and Engine Boundary. Explicit `--cache-dir` integrates the three cache layers;
omitting it preserves a cold replay control. Complete immutable snapshots are still read
and verified by the adapter before persisted discovery can be used. Equal base/head revisions
share one live materialization. Discovery keys include exact file identities and shipped code/facts;
dependency/declaration changes invalidate them. Graph units remain the existing physical frontend
partitions. Source changes rebuild the affected unit; all declaration and exclusion context is
bound. There is no incremental graph mutation or changed-file scope selector.
The runner uses backend-issued copies and per-question cached raw answers, then runs the existing
ranker, arbiter, comparison and current admission. Partial answers are not promoted to complete
lookups. Complete answers already returned by the engine remain available when persistence cannot
finish. Query keys include actual parameters and graph bytes; graph keys do not include unrelated
ranking/security-fact versions. Layout/exclusion facts and frontend adaptation bytes do affect them.
Integration review repaired the task 6.1 identity to include installed frontend/adaptation bytes,
rather than relying only on Joern's release label. This is an owning-contract repair, not a new
frontend or a detector change.
The first source-mounted/package integration trial is retained separately from the final
frozen image run. It showed equal cached/uncached evidence and admission (including one
new JavaScript injection candidate), a 1.23-second identical-revision hit, and a 30.48-second
dependency-edit timeout. Those numbers are preliminary control evidence, not the final image
or rollout measurements. Final review also binds installed runtime identity into admission
semantics, while avoiding a second backend instance or an unbudgeted availability probe.
The cache entry lease ends after its validated bytes are copied into the backend's independent
working lease. The working copy remains valid if storage later evicts the original entry;
queries never operate on the stored original. Partial fresh answers cannot borrow a cached
answer's census. Pure discovery remains in the existing bounded child process; persisted
JSON is decoded as data, never executable objects.
## Status Report
- STATUS: READY_FOR_REVIEW
- TASK: 6.4
- RED_PHASE_OUTPUT: `/tmp/group6-4-red.log`: graph/query reuse and replay discovery/cache-location tests failed before integration (temporary switch OFF). Enabled implementation passed; switch removed. Additional regressions cover mixed cached/fresh census failure and installed runtime provenance.
- TESTS_RUN: 89 final focused tests passed; full suite 1268 passed, 9 skipped in 108.74s. Ruff, formatting, mypy (114 source files), compileall, detection/map/pair gates and Docker build/packaged CLI smoke passed.
- EVIDENCE: final packaged JavaScript control exited 0 with exact cached/uncached scope, question outcomes, findings and admission equality. One new injection is retained as diagnostic evidence and denied normal admission by the empty registry. Source/dependency edits invalidate graph reuse and retain timed-out coverage artifacts.
- CONCERNS: identical-revision hits are development correctness controls, not representative warm p95 or capability qualification. Main-context review fallback remains explicit.
## Review Verdict
- VERDICT: APPROVED
- TASK: 6.4
- MECHANICAL_RESULTS: canonical tests and regression gates passed; final image reached the usable CLI and completed real replay controls. No introduced placeholders, secrets or boundary violations found.
- FINDINGS: installation-byte identity now also binds comparison/admission provenance. The predecessor no-unbudgeted-probe regression caught an unnecessary backend instance in provenance; the repair uses a pure runtime-identity function and the final regression passes. Mixed fresh/cached answers require the fresh query's own valid census before being retained as complete. Snapshot reads, discovery, ranker/driver selection, vendor boundary and independent push/coverage status remain intact.
- SUMMARY: Task 6.4 integrates valid local reuse and truthful misses under the existing transaction deadline without changing detection or admission rules.
## Verification Result
- STATUS: VERIFIED
- CLAIM_TYPE: TASK
- CLAIM: Group 6 implementation (6.1–6.4), with compatible replay evidence and admission equivalence.
- EVIDENCE: final tests/static checks, native packaged Joern controls, and committed measurement record below.
- GAPS: Rollout remains NO-GO. Groups 7–8, representative changed-code completion/p95 and independent capability qualification remain pending. Automatic projection limitations are unchanged.
## Final packaged evidence
Image: `sha256:6622b846996989a321d81bfa6c25e792e3f217984d7ede66f830cff1206a634c`.
Nonroot, network disabled, 3 GiB memory; controls ran serially without concurrent
tests/builds. Filesystem cache was not flushed. Each case is one development control,
not a representative p95 population.
| Run | Full CLI seconds | Graph hits | Query-answer hits |
|---|---:|---:|---:|
| populate | 154.848 | 0 | 0 |
| warm-identical | 2.002 | 2 | 37 |
| uncached | 130.432 | 0 | 0 |
| dependency-edit | 30.458 | 0 | 0 |
| one-function-edit | 30.479 | 0 | 0 |
| uncached, same revisions, 30s budget | 30.606 | 0 | 0 |
Cached and uncached laboratory runs have **equal scope, question outcomes, findings and
admission**, including the known introduced JavaScript injection. The 30-second identical
cached run retains that same evidence: one completed supported head injection check plus
four base comparison checks. The same revisions without caching complete zero checks in
the 30-second control. All six CLI artifacts were retained, including timed-out edits.
Warm source/dependency edits correctly invalidate the affected graph and produce incomplete
coverage; they do not replay the old answer. The empty normal capability registry denies the
new diagnostic candidate for `capability_unavailable`; this is not a clean repository verdict.
The manifest, full artifacts and stdout/stderr are retained in
`benchmarks/measurements/2026-09-13-reuse-control.json` and the associated final/initial
compressed bundles. Each bundle maps original filenames to exact UTF-8 content; decompression
was compared with the original files and SHA-256 hashes recorded. The committed reusable
harness is `benchmarks/push/reuse_control.py`; the extra same-revision 30-second uncached
command is recorded in the final bundle.
**Group 6 implementation: verified. Rollout: NO-GO.** Changed-code feasibility remains
unmet in these controls, and independent precision/recall/capability populations have not
qualified. The spec now explicitly requires stage-level investigation of remaining timeouts
under task 8.2, retaining its original thresholds and population rules. Group 7 is next;
no hook has been installed and no normal capability enabled.
+13 -8
View File
@@ -1,7 +1,7 @@
{
"feature_name": "pre-push-safety-net",
"created_at": "2026-09-12T15:35:00Z",
"updated_at": "2026-09-13T13:22:55.634332+00:00",
"updated_at": "2026-09-13T15:06:56.856952+00:00",
"language": "en",
"phase": "implementation",
"approvals": {
@@ -19,7 +19,7 @@
}
},
"ready_for_implementation": true,
"implementation_approval_note": "Groups1\u20135 implemented and verified (16/27). Group5 used manual Kiro fallback review, repaired observed predecessor deadline bugs, and executed a newly frozen PMPro/Node experiment. Measured feasibility NO-GO; real capabilities remain disabled. Groups6\u20138 pending.",
"implementation_approval_note": "Groups 1\u20136 implemented and verified (20/27); group 6 authorized as a whole. Main-context Kiro review fallback after fresh dispatch thread limit. Valid reuse is demonstrated; real capability registry remains empty and rollout gates remain unmet. Groups 7\u20138 pending.",
"depends_on": [
"contributor-scan"
],
@@ -98,19 +98,24 @@
"4.2",
"4.3",
"5.1",
"5.2"
"5.2",
"6.1",
"6.2",
"6.3",
"6.4"
],
"remaining_tasks": 11,
"verification_record": "implementation-group-5.md",
"remaining_tasks": 7,
"verification_record": "implementation-group-6.md",
"verification_records": [
"implementation-group-1.md",
"implementation-group-2.md",
"implementation-group-3.md",
"implementation-group-4.md",
"implementation-group-5.md"
"implementation-group-5.md",
"implementation-group-6.md"
],
"next_group": 6,
"next_group": 7,
"blocked_tasks": [],
"feasibility": "NO-GO: six cold replay timeouts; no completed supported target checks; warm performance and eligibility unproven"
"feasibility": "NO-GO for rollout: compatible identical-revision reuse verified; representative changed-code p95/completion and independent alert eligibility remain unmet."
}
}
+12 -8
View File
@@ -1,11 +1,12 @@
# Implementation Plan: pre-push-safety-net
Status: task plan approved by the maintainer on 2026-09-12. Groups 1–5 (tasks 1.1–5.2)
are implemented, reviewed and verified (16/27 subtasks). Tasks 4.3 and group 5 used the
documented Kiro manual review fallback after agent dispatch reached its thread limit.
The first experiment is measured NO-GO: all six cold PMPro/Node cases timed out without
completed target checks. Groups 6–8 remain pending; no production capability or warm
hook latency is admitted.
Status: task plan approved by the maintainer on 2026-09-12. Groups 1–6 (tasks 1.1–6.4)
are implemented, reviewed and verified (20/27 subtasks). Tasks 4.3 and groups 5–6 used the
explicit Kiro manual review fallback after fresh agent dispatch reached its thread limit.
Group 5's six cold PMPro/Node timeouts remain historical NO-GO evidence. Group 6 verifies
compatible reuse with equal evidence/admission in a packaged authored JavaScript control;
its identical-revision run completed in 2.00s. This is not representative warm p95 or
independent capability qualification. Groups 7–8 remain pending; rollout remains NO-GO.
## Execution contract
@@ -160,7 +161,7 @@ hook latency is admitted.
- _Depends: 1.2, 1.3, 5.1_
- _Requirements: 6.2, 6.3, 6.4, 6.5, 8.1, 8.2, 8.3, 8.4_
- [ ] 6. Reuse valid graphs and results without stale conclusions
- [x] 6. Reuse valid graphs and results without stale conclusions
- [x] 6.1 Validate graph artifact identity and leases
- Have the backend own source/declaration/exclusion digests, frontend/engine/overlay options, graph census and complete/partial identity.
- Permit validated graph leases in the generic driver while preserving the fresh-build path; isolate potentially mutating loads and retain bounded cleanup.
@@ -180,7 +181,7 @@ hook latency is admitted.
- Done when source, context, fact, query, mode, policy and optional model changes invalidate the appropriate layer, while an identical compatible request returns the same evidence and provenance.
- _Boundary: Push cache, policy integration_
- _Requirements: 3.4, 4.3, 4.4, 6.4, 7.2, 7.3_
- [ ] 6.4 Integrate cache reuse into the shared push deadline
- [x] 6.4 Integrate cache reuse into the shared push deadline
- Reuse complete preparation/discovery by immutable tree and discovery/fact/configuration identity, retaining readable snapshot and declaration/boundary records; measure its costs and hits separately from graph/query reuse.
- Reuse identical snapshots/unchanged declared units across refs without merging different comparisons; changed units rebuild and stale graph relationships remain hints only.
- Account for lookup, lock wait, lease, build/query and result persistence in the same deadline; preserve completed outcomes on cancellation.
@@ -219,6 +220,7 @@ hook latency is admitted.
- _Requirements: 1.1, 1.2, 1.3, 1.4, 2.2, 2.3, 4.2, 4.4, 5.2, 5.5, 7.1, 7.4, 8.2_
- [ ] 8.2 Measure realistic cache and deadline feasibility
- Measure identical-tip reuse, a warm one-function edit, dependency/configuration edits, cold starts, repository growth and multi-ref workloads on declared hardware.
- If reuse correctness passes but changed-code checks time out, identify preparation, frontend/overlay/census and query costs separately and route the remaining optimization to its owning boundary before rollout GO. Preserve the declared gates and do not substitute identical-tip latency.
- Publish p50/p95, timeouts, cancellation allowance, completed/deferred population and stage costs; the denominator is fixed before ranker selection.
- Done when the versioned scorecard evaluates the accepted latency/coverage gates without hiding timeouts or calling repeated-identical-tip hits representative warm edits; a failed gate remains NO-GO for rollout.
- _Boundary: Evaluation harness, runtime integration_
@@ -244,3 +246,5 @@ hook latency is admitted.
- Task5.2 measured PMPro timing out after both snapshots were materialized but before change context/graph analysis. Group6 must account for compatible snapshot/discovery reuse as well as graph reuse; identical-tip hits cannot substitute for the later representative changed-code latency gate. No new detector, scope heuristic or incremental CPG mutation is implied.
- Cancellation must cover semantic planning and reporting, not only engine subprocesses: `_collect` and `_scope_work` now consume the shared deadline, and replay instantiates the budget-aware backend once. Preserve the 400-question failure-census and real Node artifact-retention regressions.
- Group 6: complete discovery/graph/query reuse is explicit via `--cache-dir`; source/declaration edits invalidate affected units. Installed frontend/adaptation bytes now bind graph and admission provenance. A valid cached census cannot qualify a fresh answer that omitted its own census. Identical-revision timing is not the changed-code rollout gate.
+13 -8
View File
@@ -46,7 +46,7 @@ engine are out of scope.
- [ ] pre-push-safety-net — Immutable pushed tips, existing-ranker integration, compatible artifact
reuse, total deadline, change attribution, actionability admission and compact hook behavior.
Dependencies: contributor-scan. Requirements/design approved 2026-09-12; tasks generated
(8 groups, 27 executable subtasks), independently reviewed and approved; groups 1–5 are implemented and verified (16/27 tasks), including the resolved frontend-retention prerequisite.
(8 groups, 27 executable subtasks), independently reviewed and approved; groups 1–6 are implemented and verified (20/27 tasks), including the resolved frontend-retention prerequisite.
### Next concrete step
@@ -59,14 +59,19 @@ its deadline in preparation/discovery after materializing both trees; Node reach
driver but did not complete checks. The exact 11-case population remains unresolved.
No normal capability is enabled and these six cold samples establish no warm p95 claim.
Group 6 is next for valid graph/result identity, leases and reuse. Account for snapshot
and discovery costs before treating graph persistence as the sole optimization; record
compatible preparation reuse alongside graph reuse. Representative changed-code warm
latency and coverage must still be measured later, without substituting identical-tip hits.
Do not change ranking weights, add a second source-slicer or assume incremental Joern
mutation from this negative performance evidence.
Group 6 now provides complete preparation/discovery, graph and query reuse through explicit
`--cache-dir`, bounded publication and isolated graph leases. A packaged authored JavaScript
control preserved the same evidence/admission as an uncached run; its identical-revision
replay took 2.00s. Source/declaration changes invalidate affected units. This correctness
control does not establish representative changed-code p95, independent alert precision or
eligibility. Rollout remains NO-GO and normal capabilities remain disabled.
Tasks 4.3 and group 5 used the documented Kiro manual review fallback after the agent
Group 7 is next for the real multi-ref push interface and opt-in hook integration. Group 8
must still pass the joint gates on declared populations. Keep the existing thresholds and
profile preparation, graph construction and queries separately when investigating misses;
do not substitute identical-tip hits for warm changed-code performance.
Tasks 4.3 and groups 5–6 used the documented Kiro manual review fallback after the agent
thread limit; this is not independent review. See `pre-push-safety-net/implementation-group-5.md`
and `benchmarks/measurements/2026-09-13-first-replay-experiment.json` for provenance,
initial instrument failures, their repairs and the newly frozen final trial. The NodeGoat
+9 -4
View File
@@ -79,16 +79,21 @@ updates as a shortcut to an unmeasured latency promise.
The product direction above is user-requested. The amended requirements, design and initial
evaluation profile in `pre-push-safety-net` were approved for task generation on 2026-09-12
("ok, next kiro phase"). The maintainer then approved the task plan and starting group 1
("approve.") and then group 2 ("ok, next kiro phase, approved"). Groups 1–5 are now
implemented, reviewed and verified (16/27), including the frontend-retention prerequisite
and the deadline regression exposed by full Node replay. Tasks 4.3 and group 5 used the
("approve.") and then group 2 ("ok, next kiro phase, approved"). Groups 1–6 are now
implemented, reviewed and verified (20/27), including the frontend-retention prerequisite
and the deadline regression exposed by full Node replay. Tasks 4.3 and groups 5–6 used the
explicit Kiro manual review fallback after the agent thread limit.
The first newly frozen PMPro/Node experiment is measured **NO-GO**: all six cold cases
timed out without completed supported target checks, while retaining all 11 declared cases.
The repaired CLI saved all artifacts and returned in 30.55–30.64 seconds. PMPro exhausted
the deadline in preparation/discovery before graph analysis, so compatible preparation
reuse matters alongside graph persistence. Group 6 is next; warm changed-code performance
reuse matters alongside graph persistence. Group 6 now provides compatible local reuse; warm changed-code performance
and useful coverage remain mandatory later gates. No ranking/fact adaptation or holdout
inspection occurred in the final transfer run. Real capability eligibility and automatic
dominance context projection remain unproven; the default capability registry is empty.
Group 6 verified equal cached/uncached evidence and current admission in a packaged authored
JavaScript control. Source/declaration edits invalidate affected units. Identical-revision
latency does not qualify representative changed-code performance. Requirements and rollout
thresholds remain unchanged; groups 7–8 and independent capability admission remain pending.