feat(pre-push-safety-net): establish shared push and engine contracts

This commit is contained in:
Marius Ciepluch
2026-09-12 19:21:31 +02:00
parent 051f3b0058
commit 14a2a60e52
18 changed files with 810 additions and 25 deletions
+1 -1
View File
@@ -1,6 +1,6 @@
# Brief: pre-push-safety-net
Status: requirements, design and tasks approved on 2026-09-12; group 1 implementation started. Language: English.
Status: requirements, design and tasks approved on 2026-09-12; group 1 implementation verified complete; groups 2–8 pending. Language: English.
## Problem
+16 -2
View File
@@ -132,8 +132,8 @@ Concrete workload roles and required paired changes are in `evaluation.md`.
| Storage | Local content-addressed directories and JSON manifests | Atomic reusable artifacts, no database |
| Presentation | Existing finding records plus compact hook renderer | Separate findings and coverage |
Align Docker, Compose and provisioning with the evaluated Joern version before any deployment
claim. The old defaults are 4.0.623. Pin alignment is predecessor packaging work, not completed here.
Docker, Compose and provisioning defaults were aligned to 4.0.625 in task 1.1, with a verified
local PHP/JavaScript image smoke. Host deployment and hook performance remain separate claims.
## File Structure Plan
@@ -141,6 +141,10 @@ New files (proposed, not created by this documentation change):
| Path | Responsibility |
|---|---|
| `src/openultrasast/contracts.py` | Strict immutable manifest serialization shared by engine and adapters |
| `src/openultrasast/model/contracts.py` | Generic deadline, change context and exact selected/deferred scope contracts |
| `src/openultrasast/cpg/artifact.py` | Backend-owned graph identity, census and live lease protocol |
| `src/openultrasast/push/contracts.py` | Immutable ref/comparison identities and comparison-owned completion/status records |
| `src/openultrasast/push/snapshot.py` | Parse ref updates, resolve bases, materialize blobs, produce change context |
| `src/openultrasast/push/cache.py` | Artifact keys, validity, atomic publication and eviction |
| `src/openultrasast/push/policy.py` | Delta identity, actionability admission and push disposition |
@@ -149,6 +153,16 @@ New files (proposed, not created by this documentation change):
| `benchmarks/push/README.md` and `benchmarks/push/measure.py` | Labeled push recipes and joint quality/latency measurement |
| `tests/test_push_snapshot.py`, `tests/test_push_cache.py`, `tests/test_push_policy.py`, `tests/test_push_runner.py` | Contract and integration checks below |
Contract configuration (task 1.4) uses `[push]`: optional `comparison_base`, positive
`deadline_seconds` (30), positive `cancellation_allowance_seconds` (2), positive integer
`cache_max_bytes` (2 GiB), `mode` (`advisory` by default, or `blocking`) and independent
`incomplete_coverage_policy` (`allow` by default, or `block`). Unknown keys, malformed
sections and coerced/nonfinite numeric values are rejected. These settings are inert until
runner integration; they do not enable a hook or admit experimental capabilities. The cache
limit is a local storage budget, not a latency claim. Completion records are owned by an
exact comparison so reused head graphs cannot merge evidence from different bases. Live
leases have no JSON constructor; only the backend can validate and issue them in task 6.1.
Modified files: `cli.py` and `config.py` add explicit push command/configuration;
`model/regions.py` carries change context without Git-specific logic; `model/scan.py` consumes the
ranker's selected questions and returns exact selected/deferred IDs, deadline and reusable-graph
@@ -0,0 +1,60 @@
# Group 1 implementation verification
Date: 2026-09-12. Scope: tasks 1.1–1.4 only. All four tasks received independent
review with an APPROVED verdict and fresh task-local verification. Requirements,
design and tasks remain approved; implementation remains in progress (4/27 subtasks).
## Verification result
- STATUS: VERIFIED
- CLAIM_TYPE: TASK
- CLAIM: Group 1 establishes the reproducible runtime, evaluation inputs, honest
scoring and shared push/engine contracts.
- EVIDENCE: Final independent review ran the full suite: 965 passed, 9 skipped.
Focused contracts/configuration tests: 53 passed. Ruff and mypy passed for the
seven touched contract/configuration modules. `git diff --check` passed.
- GAPS: This is not feature-level GO. The pre-push runtime, persistent cache and
capability admission are not implemented. No end-to-end latency or precision
claim follows from these tests.
## Task evidence
| Task | Result and retained evidence |
| --- | --- |
| 1.1 | Joern 4.0.625 packaged and exercised with native PHP and JavaScript. Actual source bytes, graph census and query witnesses were verified; unreadable input failed explicitly. See `benchmarks/measurements/2026-09-12-engine-runtime-smoke.json`. Suite at review: 888 passed, 9 skipped. Commit `56cfc9d`. |
| 1.2 | Pinned PMPro and NodeGoat vulnerable, repaired and benign recipes; explicit authored-repair provenance. Offline validation read 506,502 bytes: 8 snapshots ready, 1 missing; 7 cases ready, 4 missing. Missing prerequisites cause exit 1. See `benchmarks/push/input-validation.json`. Suite: 902 passed, 9 skipped. Commit `8acada0`. |
| 1.3 | Frozen evaluation identity and uncertainty-preserving scorer. VAmPI retains all three known defects; unmatched SQL injection is not silently called transitively covered. Empty runs remain unmeasured. See `benchmarks/ranking/baselines/2026-09-12-vampi-position-v2.json` and `benchmarks/push/diagnostic-profile-v1.json`. Suite: 925 passed, 9 skipped. Commit `051f3b0`. |
| 1.4 | Strict immutable configuration, budget, question, scope, graph artifact and comparison-owned result contracts. Incomplete/unexecuted work cannot become complete through serialization. Existing scan configuration remains compatible; runtime enforcement is pending. Final suite: 965 passed, 9 skipped. |
## Packaged runtime and limitations
After the final source changes, `docker compose build ousast` exited 0. Final local
image: `sha256:92d311097075a2092a2e362b5470d706734f9de35de7820c3de5eb1ad91b17d5`.
A network-disabled container imported the new contracts, read an 83-byte TOML
configuration, round-tripped its push settings, rejected a NaN deadline, and ran
`python -m openultrasast.cli --help` successfully. Runtime: UID 1000, Python 3.12.3.
The earlier graph smoke records its own image identity; the backend did not change
after that smoke. The final packaging check does not repeat or replace graph evidence.
Full-tree `.venv/bin/mypy src/openultrasast` still exits 1 with three pre-existing
errors in `model/scan.py` at lines 299, 302 and 310 (candidate redefinition and
incompatible candidate types). It checked 107 source files with no additional errors.
Skipped tests and the unavailable evaluation prerequisites remain outside demonstrated
coverage. Ghost is a reserved workload, not a scanned or admitted positive population;
independent paired/holdout evidence remains required per capability.
## Design direction and next work
The ranker remains the scope mechanism. Vendor exclusion applies to graph inputs
as well as targets. PMPro guides core development and JavaScript/Express tests
transfer; Python regressions and C-family abstraction limits remain explicit.
Persist validated immutable graphs first. Separate graph, query/evidence and ranked
result identities so ranking-only changes can reuse compatible graphs and evidence.
Differential graph mutation is a future experiment requiring equivalence to cold
rebuilds, including deletions, renames, dependent edges and overlays. No stable
Joern node-ID assumption or partial-file update is accepted as proof of correctness.
Next: group 2 immutable push snapshots and comparison resolution, then the first
complete PMPro/Node experiment at task 5.2. Persistent caching remains group 6;
normal advisory capability admission remains gated by task 8.3.
+3 -1
View File
@@ -6,7 +6,9 @@
- User direction: a scalable pre-push safety net as AI accelerates development; WordPress is one case.
- Clarifications: advisory must be actionable, not SAST noise; the ranker is the scope mechanism;
third-party vendor code must remain outside targets and graphs.
- Requirements and design are proposals, not approved or implemented. No hook latency claim is made.
- Requirements, design and tasks are now approved, and group 1 implementation is verified complete; see
`tasks.md` for verified completion. The discovery log below preserves the earlier drafting state
and measurements. No hook latency claim is made.
## Research Log
+12 -1
View File
@@ -1,7 +1,7 @@
{
"feature_name": "pre-push-safety-net",
"created_at": "2026-09-12T15:35:00Z",
"updated_at": "2026-09-12T16:20:40.814596Z",
"updated_at": "2026-09-12T17:21:05.783007Z",
"language": "en",
"phase": "implementation",
"approvals": {
@@ -50,5 +50,16 @@
"executable_subtasks": 27,
"requirements_covered": 35,
"record": "task-review.md"
},
"implementation_progress": {
"completed_tasks": [
"1.1",
"1.2",
"1.3",
"1.4"
],
"remaining_tasks": 23,
"verification_record": "implementation-group-1.md",
"next_group": 2
}
}
+6 -4
View File
@@ -1,7 +1,8 @@
# Implementation Plan: pre-push-safety-net
Status: task plan approved by the maintainer's "approve." on 2026-09-12. Implementation has
started with group 1 (tasks 1.1–1.4); each task remains pending until independent review and fresh verification pass.
Status: task plan approved by the maintainer's "approve." on 2026-09-12. Group 1 (tasks 1.1–1.4)
is complete after independent review and fresh verification. Remaining groups are pending;
the hook and capability admission are not yet implemented. See `implementation-group-1.md`.
## Execution contract
@@ -21,7 +22,7 @@ started with group 1 (tasks 1.1–1.4); each task remains pending until independ
- Sequential order is the default. Only 3.1 may run alongside group 2 after its explicit prerequisite;
shared backend/driver, cache and report edits otherwise remain serial. No optional acceptance tests.
- [ ] 1. Establish the runtime, evaluation inputs and shared contracts
- [x] 1. Establish the runtime, evaluation inputs and shared contracts
- [x] 1.1 Reproduce the evaluated engine environment
- Align the shipped image, Compose and host-install defaults with Joern 4.0.625; preserve version override and checksum verification.
- Exercise PHP with the native interpreter and the installed JavaScript frontend; open a source and the frontend parser where applicable, recording byte counts and graph census.
@@ -45,13 +46,14 @@ started with group 1 (tasks 1.1–1.4); each task remains pending until independ
- _Boundary: Evaluation harness, repository benchmark integration_
- _Requirements: 6.2, 6.3, 6.4, 6.5, 7.3_
- _Verified 2026-09-12:_ independent review APPROVED; 925 tests passed, 9 skipped. Parent reproduced the frozen profile and actual VAmPI static rerun (15,942 pinned bytes, all 3 targets retained, SQLI unmatched, no transitive/detection claim, exit 1). Empty-run scoring retains 11 unresolved cases and explicitly reports unmeasured/0 recorded cases. Historical baselines remain unchanged with a caveat; new scorer consumes associated execution/review evidence and does not implement replay or admission.
- [ ] 1.4 Establish typed push and engine integration contracts
- [x] 1.4 Establish typed push and engine integration contracts
- Add importable push components and validated configuration for comparison base, shared deadline/cancellation allowance, local cache limit, advisory/blocking and separate coverage policy.
- Define ownership of update/comparison, change context, selected/deferred scope, graph identity/lease and separate finding/coverage/push statuses as described in design.
- Keep generic engine contracts outside the push package; leave ordinary scan defaults compatible and capabilities experimental until evaluated.
- Done when contract/config round trips preserve all identities/statuses, invalid values are rejected, and the engine has no dependency on the push package.
- _Boundary: Push runner, Cache and Engine Boundary, CLI/config integration_
- _Requirements: 4.1, 4.3, 5.3, 5.4, 5.5, 7.1, 7.3_
- _Verified 2026-09-12:_ independent review APPROVED; 965 tests passed, 9 skipped; touched-source Ruff/mypy passed. Parent rebuilt the local image and verified non-root, network-disabled CLI startup, all contract imports, an 83-byte configuration round trip and invalid-budget rejection. Comparison-owned completion preserves different bases for a shared head. Contracts are structural; runtime snapshot/cache/runner/admission implementations remain in later tasks.
- [ ] 2. Resolve immutable pushed content
- [ ] 2.1 Resolve every supplied ref update and comparison
+8 -2
View File
@@ -1,17 +1,23 @@
# Overview — what this is, and what is actually true of it
**Current direction, 2026-09-12:** an actionable pre-push security safety net for AI-accelerated
development across supported ecosystems. See `safety-net.md` and the proposed `pre-push-safety-net`
development across supported ecosystems. See `safety-net.md` and the approved `pre-push-safety-net`
spec. The ranker remains the scope mechanism; third-party vendor code stays outside targets and
graphs. Advisory must meet the same actionability bar as blocking.
**Historical scoreboard below:** it was written September 9. Later commits record a completing
whole-PMPro scan and Joern 4.0.625 ranking measurements (82/91/54 on the three plugins), while
default installation pins remain 4.0.623. The pre-push workflow, changed-code latency and actionable
default installation pins are now aligned to 4.0.625 with verified PHP/JavaScript image smoke.
The pre-push workflow, changed-code latency and actionable
precision are not demonstrated by those results. In particular, the manually selected two-file
slice below is a useful experiment, not evidence that an automatic push scope would select the
same context. Current scope/status corrections live in `flow-aware-ranking/brief.md` and the roadmap.
The new scoring instrument retains missing targets and unanswered queries. Historical rank positions
do not establish detection recall; VAmPI's old budget of 16 omitted its unmatched SQLI target. The
corrected positional run keeps that target unresolved. Labeled PMPro/Node inputs and a frozen
diagnostic profile are committed; independent capability evaluation is still required.
**Written 2026-09-09.** Fourteen specs have accumulated and the roadmap still described a pre-Joern world, so
this is the synthesis above them: what the system does, what has been measured, and what is not true yet.
Every number here comes from a committed artifact in `benchmarks/measurements/`.
+8 -7
View File
@@ -21,8 +21,9 @@ engine are out of scope.
- Whole-PMPro analysis has completed with its known CVE; the earlier "does not finish" diagnosis
is superseded. The recorded 72-minute scan is not a pre-push performance result.
- The latest Joern 4.0.625 + layout ranking budgets are PMPro 82, WP Statistics 91, MW WP Form 54,
VAmPI 16. `<50` remains unmet; VAmPI's transitive SQLI is not measured by this position metric.
- Historical Joern 4.0.625 + layout position budgets are PMPro 82, WP Statistics 91, MW WP Form 54.
`<50` remains unmet. VAmPI's old 16 omitted SQLI; the corrected instrument retains the unmatched
target and reports no full-target budget or detection recall without the required evidence.
- Equal-score tie-breaking alone cannot bring PMPro or WP Statistics under 50. The existing
ranker is useful, but its measured ordering is not enabled by the ordinary CLI caller.
- Docker/Compose/ops now default to 4.0.625; task 1.1 passed local PHP/JavaScript image smoke and independent review. No host deployment or hook latency claim follows from that runtime evidence.
@@ -45,14 +46,14 @@ engine are out of scope.
- [ ] pre-push-safety-net — Immutable pushed tips, existing-ranker integration, compatible artifact
reuse, total deadline, change attribution, actionability admission and compact hook behavior.
Dependencies: contributor-scan. Requirements/design approved 2026-09-12; tasks generated
(8 groups, 27 executable subtasks), independently reviewed and approved; group 1 implementation started.
(8 groups, 27 executable subtasks), independently reviewed and approved; group 1 is complete (4/27 tasks).
### Next concrete step
The maintainer approved `pre-push-safety-net/tasks.md` and starting group 1. Requirements and design
were accepted when the maintainer requested the next Kiro phase; the 27-subtask plan passed independent
sanity review. Complete group 1 (reproducible runtime, readable PMPro/Node inputs, honest scoring
and shared contracts). The first complete experiment is 5.2, before persistent caching and hook
Group 1 has passed independent review: reproducible PHP/JavaScript runtime, pinned PMPro/Node inputs,
honest scoring and shared contracts. Continue with group 2's immutable push snapshots under the
approved task plan; see `pre-push-safety-net/implementation-group-1.md` for evidence and limits.
The first complete experiment is 5.2, before persistent caching and hook
installation work. Keep PMPro as the primary core-optimization example, then freeze the core and
prove a changed security flow and its fixed twin on a JavaScript/Express Node API workload through
the same ranker, with a real deadline and vendor-free graph. Include benign pushes and retain Python
+3 -2
View File
@@ -78,5 +78,6 @@ updates as a shortcut to an unmeasured latency promise.
The product direction above is user-requested. The amended requirements, design and initial
evaluation profile in `pre-push-safety-net` were approved for task generation on 2026-09-12
("ok, next kiro phase"). Task approval remains pending. This is not authorization to implement
or a claim that hook performance or capability admission has been demonstrated.
("ok, next kiro phase"). The maintainer then approved the task plan and starting group 1
("approve."). Implementation is authorized under that plan; task completion still requires
independent review and fresh evidence. Hook performance and capability admission remain unproven.