mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-17 03:59:28 +00:00
feat(pre-push-safety-net): establish shared push and engine contracts
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# Brief: pre-push-safety-net
|
||||
|
||||
Status: requirements, design and tasks approved on 2026-09-12; group 1 implementation started. Language: English.
|
||||
Status: requirements, design and tasks approved on 2026-09-12; group 1 implementation verified complete; groups 2–8 pending. Language: English.
|
||||
|
||||
## Problem
|
||||
|
||||
|
||||
@@ -132,8 +132,8 @@ Concrete workload roles and required paired changes are in `evaluation.md`.
|
||||
| Storage | Local content-addressed directories and JSON manifests | Atomic reusable artifacts, no database |
|
||||
| Presentation | Existing finding records plus compact hook renderer | Separate findings and coverage |
|
||||
|
||||
Align Docker, Compose and provisioning with the evaluated Joern version before any deployment
|
||||
claim. The old defaults are 4.0.623. Pin alignment is predecessor packaging work, not completed here.
|
||||
Docker, Compose and provisioning defaults were aligned to 4.0.625 in task 1.1, with a verified
|
||||
local PHP/JavaScript image smoke. Host deployment and hook performance remain separate claims.
|
||||
|
||||
## File Structure Plan
|
||||
|
||||
@@ -141,6 +141,10 @@ New files (proposed, not created by this documentation change):
|
||||
|
||||
| Path | Responsibility |
|
||||
|---|---|
|
||||
| `src/openultrasast/contracts.py` | Strict immutable manifest serialization shared by engine and adapters |
|
||||
| `src/openultrasast/model/contracts.py` | Generic deadline, change context and exact selected/deferred scope contracts |
|
||||
| `src/openultrasast/cpg/artifact.py` | Backend-owned graph identity, census and live lease protocol |
|
||||
| `src/openultrasast/push/contracts.py` | Immutable ref/comparison identities and comparison-owned completion/status records |
|
||||
| `src/openultrasast/push/snapshot.py` | Parse ref updates, resolve bases, materialize blobs, produce change context |
|
||||
| `src/openultrasast/push/cache.py` | Artifact keys, validity, atomic publication and eviction |
|
||||
| `src/openultrasast/push/policy.py` | Delta identity, actionability admission and push disposition |
|
||||
@@ -149,6 +153,16 @@ New files (proposed, not created by this documentation change):
|
||||
| `benchmarks/push/README.md` and `benchmarks/push/measure.py` | Labeled push recipes and joint quality/latency measurement |
|
||||
| `tests/test_push_snapshot.py`, `tests/test_push_cache.py`, `tests/test_push_policy.py`, `tests/test_push_runner.py` | Contract and integration checks below |
|
||||
|
||||
Contract configuration (task 1.4) uses `[push]`: optional `comparison_base`, positive
|
||||
`deadline_seconds` (30), positive `cancellation_allowance_seconds` (2), positive integer
|
||||
`cache_max_bytes` (2 GiB), `mode` (`advisory` by default, or `blocking`) and independent
|
||||
`incomplete_coverage_policy` (`allow` by default, or `block`). Unknown keys, malformed
|
||||
sections and coerced/nonfinite numeric values are rejected. These settings are inert until
|
||||
runner integration; they do not enable a hook or admit experimental capabilities. The cache
|
||||
limit is a local storage budget, not a latency claim. Completion records are owned by an
|
||||
exact comparison so reused head graphs cannot merge evidence from different bases. Live
|
||||
leases have no JSON constructor; only the backend can validate and issue them in task 6.1.
|
||||
|
||||
Modified files: `cli.py` and `config.py` add explicit push command/configuration;
|
||||
`model/regions.py` carries change context without Git-specific logic; `model/scan.py` consumes the
|
||||
ranker's selected questions and returns exact selected/deferred IDs, deadline and reusable-graph
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
# Group 1 implementation verification
|
||||
|
||||
Date: 2026-09-12. Scope: tasks 1.1–1.4 only. All four tasks received independent
|
||||
review with an APPROVED verdict and fresh task-local verification. Requirements,
|
||||
design and tasks remain approved; implementation remains in progress (4/27 subtasks).
|
||||
|
||||
## Verification result
|
||||
|
||||
- STATUS: VERIFIED
|
||||
- CLAIM_TYPE: TASK
|
||||
- CLAIM: Group 1 establishes the reproducible runtime, evaluation inputs, honest
|
||||
scoring and shared push/engine contracts.
|
||||
- EVIDENCE: Final independent review ran the full suite: 965 passed, 9 skipped.
|
||||
Focused contracts/configuration tests: 53 passed. Ruff and mypy passed for the
|
||||
seven touched contract/configuration modules. `git diff --check` passed.
|
||||
- GAPS: This is not feature-level GO. The pre-push runtime, persistent cache and
|
||||
capability admission are not implemented. No end-to-end latency or precision
|
||||
claim follows from these tests.
|
||||
|
||||
## Task evidence
|
||||
|
||||
| Task | Result and retained evidence |
|
||||
| --- | --- |
|
||||
| 1.1 | Joern 4.0.625 packaged and exercised with native PHP and JavaScript. Actual source bytes, graph census and query witnesses were verified; unreadable input failed explicitly. See `benchmarks/measurements/2026-09-12-engine-runtime-smoke.json`. Suite at review: 888 passed, 9 skipped. Commit `56cfc9d`. |
|
||||
| 1.2 | Pinned PMPro and NodeGoat vulnerable, repaired and benign recipes; explicit authored-repair provenance. Offline validation read 506,502 bytes: 8 snapshots ready, 1 missing; 7 cases ready, 4 missing. Missing prerequisites cause exit 1. See `benchmarks/push/input-validation.json`. Suite: 902 passed, 9 skipped. Commit `8acada0`. |
|
||||
| 1.3 | Frozen evaluation identity and uncertainty-preserving scorer. VAmPI retains all three known defects; unmatched SQL injection is not silently called transitively covered. Empty runs remain unmeasured. See `benchmarks/ranking/baselines/2026-09-12-vampi-position-v2.json` and `benchmarks/push/diagnostic-profile-v1.json`. Suite: 925 passed, 9 skipped. Commit `051f3b0`. |
|
||||
| 1.4 | Strict immutable configuration, budget, question, scope, graph artifact and comparison-owned result contracts. Incomplete/unexecuted work cannot become complete through serialization. Existing scan configuration remains compatible; runtime enforcement is pending. Final suite: 965 passed, 9 skipped. |
|
||||
|
||||
## Packaged runtime and limitations
|
||||
|
||||
After the final source changes, `docker compose build ousast` exited 0. Final local
|
||||
image: `sha256:92d311097075a2092a2e362b5470d706734f9de35de7820c3de5eb1ad91b17d5`.
|
||||
A network-disabled container imported the new contracts, read an 83-byte TOML
|
||||
configuration, round-tripped its push settings, rejected a NaN deadline, and ran
|
||||
`python -m openultrasast.cli --help` successfully. Runtime: UID 1000, Python 3.12.3.
|
||||
The earlier graph smoke records its own image identity; the backend did not change
|
||||
after that smoke. The final packaging check does not repeat or replace graph evidence.
|
||||
|
||||
Full-tree `.venv/bin/mypy src/openultrasast` still exits 1 with three pre-existing
|
||||
errors in `model/scan.py` at lines 299, 302 and 310 (candidate redefinition and
|
||||
incompatible candidate types). It checked 107 source files with no additional errors.
|
||||
Skipped tests and the unavailable evaluation prerequisites remain outside demonstrated
|
||||
coverage. Ghost is a reserved workload, not a scanned or admitted positive population;
|
||||
independent paired/holdout evidence remains required per capability.
|
||||
|
||||
## Design direction and next work
|
||||
|
||||
The ranker remains the scope mechanism. Vendor exclusion applies to graph inputs
|
||||
as well as targets. PMPro guides core development and JavaScript/Express tests
|
||||
transfer; Python regressions and C-family abstraction limits remain explicit.
|
||||
|
||||
Persist validated immutable graphs first. Separate graph, query/evidence and ranked
|
||||
result identities so ranking-only changes can reuse compatible graphs and evidence.
|
||||
Differential graph mutation is a future experiment requiring equivalence to cold
|
||||
rebuilds, including deletions, renames, dependent edges and overlays. No stable
|
||||
Joern node-ID assumption or partial-file update is accepted as proof of correctness.
|
||||
|
||||
Next: group 2 immutable push snapshots and comparison resolution, then the first
|
||||
complete PMPro/Node experiment at task 5.2. Persistent caching remains group 6;
|
||||
normal advisory capability admission remains gated by task 8.3.
|
||||
@@ -6,7 +6,9 @@
|
||||
- User direction: a scalable pre-push safety net as AI accelerates development; WordPress is one case.
|
||||
- Clarifications: advisory must be actionable, not SAST noise; the ranker is the scope mechanism;
|
||||
third-party vendor code must remain outside targets and graphs.
|
||||
- Requirements and design are proposals, not approved or implemented. No hook latency claim is made.
|
||||
- Requirements, design and tasks are now approved, and group 1 implementation is verified complete; see
|
||||
`tasks.md` for verified completion. The discovery log below preserves the earlier drafting state
|
||||
and measurements. No hook latency claim is made.
|
||||
|
||||
## Research Log
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"feature_name": "pre-push-safety-net",
|
||||
"created_at": "2026-09-12T15:35:00Z",
|
||||
"updated_at": "2026-09-12T16:20:40.814596Z",
|
||||
"updated_at": "2026-09-12T17:21:05.783007Z",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"approvals": {
|
||||
@@ -50,5 +50,16 @@
|
||||
"executable_subtasks": 27,
|
||||
"requirements_covered": 35,
|
||||
"record": "task-review.md"
|
||||
},
|
||||
"implementation_progress": {
|
||||
"completed_tasks": [
|
||||
"1.1",
|
||||
"1.2",
|
||||
"1.3",
|
||||
"1.4"
|
||||
],
|
||||
"remaining_tasks": 23,
|
||||
"verification_record": "implementation-group-1.md",
|
||||
"next_group": 2
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
# Implementation Plan: pre-push-safety-net
|
||||
|
||||
Status: task plan approved by the maintainer's "approve." on 2026-09-12. Implementation has
|
||||
started with group 1 (tasks 1.1–1.4); each task remains pending until independent review and fresh verification pass.
|
||||
Status: task plan approved by the maintainer's "approve." on 2026-09-12. Group 1 (tasks 1.1–1.4)
|
||||
is complete after independent review and fresh verification. Remaining groups are pending;
|
||||
the hook and capability admission are not yet implemented. See `implementation-group-1.md`.
|
||||
|
||||
## Execution contract
|
||||
|
||||
@@ -21,7 +22,7 @@ started with group 1 (tasks 1.1–1.4); each task remains pending until independ
|
||||
- Sequential order is the default. Only 3.1 may run alongside group 2 after its explicit prerequisite;
|
||||
shared backend/driver, cache and report edits otherwise remain serial. No optional acceptance tests.
|
||||
|
||||
- [ ] 1. Establish the runtime, evaluation inputs and shared contracts
|
||||
- [x] 1. Establish the runtime, evaluation inputs and shared contracts
|
||||
- [x] 1.1 Reproduce the evaluated engine environment
|
||||
- Align the shipped image, Compose and host-install defaults with Joern 4.0.625; preserve version override and checksum verification.
|
||||
- Exercise PHP with the native interpreter and the installed JavaScript frontend; open a source and the frontend parser where applicable, recording byte counts and graph census.
|
||||
@@ -45,13 +46,14 @@ started with group 1 (tasks 1.1–1.4); each task remains pending until independ
|
||||
- _Boundary: Evaluation harness, repository benchmark integration_
|
||||
- _Requirements: 6.2, 6.3, 6.4, 6.5, 7.3_
|
||||
- _Verified 2026-09-12:_ independent review APPROVED; 925 tests passed, 9 skipped. Parent reproduced the frozen profile and actual VAmPI static rerun (15,942 pinned bytes, all 3 targets retained, SQLI unmatched, no transitive/detection claim, exit 1). Empty-run scoring retains 11 unresolved cases and explicitly reports unmeasured/0 recorded cases. Historical baselines remain unchanged with a caveat; new scorer consumes associated execution/review evidence and does not implement replay or admission.
|
||||
- [ ] 1.4 Establish typed push and engine integration contracts
|
||||
- [x] 1.4 Establish typed push and engine integration contracts
|
||||
- Add importable push components and validated configuration for comparison base, shared deadline/cancellation allowance, local cache limit, advisory/blocking and separate coverage policy.
|
||||
- Define ownership of update/comparison, change context, selected/deferred scope, graph identity/lease and separate finding/coverage/push statuses as described in design.
|
||||
- Keep generic engine contracts outside the push package; leave ordinary scan defaults compatible and capabilities experimental until evaluated.
|
||||
- Done when contract/config round trips preserve all identities/statuses, invalid values are rejected, and the engine has no dependency on the push package.
|
||||
- _Boundary: Push runner, Cache and Engine Boundary, CLI/config integration_
|
||||
- _Requirements: 4.1, 4.3, 5.3, 5.4, 5.5, 7.1, 7.3_
|
||||
- _Verified 2026-09-12:_ independent review APPROVED; 965 tests passed, 9 skipped; touched-source Ruff/mypy passed. Parent rebuilt the local image and verified non-root, network-disabled CLI startup, all contract imports, an 83-byte configuration round trip and invalid-budget rejection. Comparison-owned completion preserves different bases for a shared head. Contracts are structural; runtime snapshot/cache/runner/admission implementations remain in later tasks.
|
||||
|
||||
- [ ] 2. Resolve immutable pushed content
|
||||
- [ ] 2.1 Resolve every supplied ref update and comparison
|
||||
|
||||
@@ -1,17 +1,23 @@
|
||||
# Overview — what this is, and what is actually true of it
|
||||
|
||||
**Current direction, 2026-09-12:** an actionable pre-push security safety net for AI-accelerated
|
||||
development across supported ecosystems. See `safety-net.md` and the proposed `pre-push-safety-net`
|
||||
development across supported ecosystems. See `safety-net.md` and the approved `pre-push-safety-net`
|
||||
spec. The ranker remains the scope mechanism; third-party vendor code stays outside targets and
|
||||
graphs. Advisory must meet the same actionability bar as blocking.
|
||||
|
||||
**Historical scoreboard below:** it was written September 9. Later commits record a completing
|
||||
whole-PMPro scan and Joern 4.0.625 ranking measurements (82/91/54 on the three plugins), while
|
||||
default installation pins remain 4.0.623. The pre-push workflow, changed-code latency and actionable
|
||||
default installation pins are now aligned to 4.0.625 with verified PHP/JavaScript image smoke.
|
||||
The pre-push workflow, changed-code latency and actionable
|
||||
precision are not demonstrated by those results. In particular, the manually selected two-file
|
||||
slice below is a useful experiment, not evidence that an automatic push scope would select the
|
||||
same context. Current scope/status corrections live in `flow-aware-ranking/brief.md` and the roadmap.
|
||||
|
||||
The new scoring instrument retains missing targets and unanswered queries. Historical rank positions
|
||||
do not establish detection recall; VAmPI's old budget of 16 omitted its unmatched SQLI target. The
|
||||
corrected positional run keeps that target unresolved. Labeled PMPro/Node inputs and a frozen
|
||||
diagnostic profile are committed; independent capability evaluation is still required.
|
||||
|
||||
**Written 2026-09-09.** Fourteen specs have accumulated and the roadmap still described a pre-Joern world, so
|
||||
this is the synthesis above them: what the system does, what has been measured, and what is not true yet.
|
||||
Every number here comes from a committed artifact in `benchmarks/measurements/`.
|
||||
|
||||
@@ -21,8 +21,9 @@ engine are out of scope.
|
||||
|
||||
- Whole-PMPro analysis has completed with its known CVE; the earlier "does not finish" diagnosis
|
||||
is superseded. The recorded 72-minute scan is not a pre-push performance result.
|
||||
- The latest Joern 4.0.625 + layout ranking budgets are PMPro 82, WP Statistics 91, MW WP Form 54,
|
||||
VAmPI 16. `<50` remains unmet; VAmPI's transitive SQLI is not measured by this position metric.
|
||||
- Historical Joern 4.0.625 + layout position budgets are PMPro 82, WP Statistics 91, MW WP Form 54.
|
||||
`<50` remains unmet. VAmPI's old 16 omitted SQLI; the corrected instrument retains the unmatched
|
||||
target and reports no full-target budget or detection recall without the required evidence.
|
||||
- Equal-score tie-breaking alone cannot bring PMPro or WP Statistics under 50. The existing
|
||||
ranker is useful, but its measured ordering is not enabled by the ordinary CLI caller.
|
||||
- Docker/Compose/ops now default to 4.0.625; task 1.1 passed local PHP/JavaScript image smoke and independent review. No host deployment or hook latency claim follows from that runtime evidence.
|
||||
@@ -45,14 +46,14 @@ engine are out of scope.
|
||||
- [ ] pre-push-safety-net — Immutable pushed tips, existing-ranker integration, compatible artifact
|
||||
reuse, total deadline, change attribution, actionability admission and compact hook behavior.
|
||||
Dependencies: contributor-scan. Requirements/design approved 2026-09-12; tasks generated
|
||||
(8 groups, 27 executable subtasks), independently reviewed and approved; group 1 implementation started.
|
||||
(8 groups, 27 executable subtasks), independently reviewed and approved; group 1 is complete (4/27 tasks).
|
||||
|
||||
### Next concrete step
|
||||
|
||||
The maintainer approved `pre-push-safety-net/tasks.md` and starting group 1. Requirements and design
|
||||
were accepted when the maintainer requested the next Kiro phase; the 27-subtask plan passed independent
|
||||
sanity review. Complete group 1 (reproducible runtime, readable PMPro/Node inputs, honest scoring
|
||||
and shared contracts). The first complete experiment is 5.2, before persistent caching and hook
|
||||
Group 1 has passed independent review: reproducible PHP/JavaScript runtime, pinned PMPro/Node inputs,
|
||||
honest scoring and shared contracts. Continue with group 2's immutable push snapshots under the
|
||||
approved task plan; see `pre-push-safety-net/implementation-group-1.md` for evidence and limits.
|
||||
The first complete experiment is 5.2, before persistent caching and hook
|
||||
installation work. Keep PMPro as the primary core-optimization example, then freeze the core and
|
||||
prove a changed security flow and its fixed twin on a JavaScript/Express Node API workload through
|
||||
the same ranker, with a real deadline and vendor-free graph. Include benign pushes and retain Python
|
||||
|
||||
@@ -78,5 +78,6 @@ updates as a shortcut to an unmeasured latency promise.
|
||||
|
||||
The product direction above is user-requested. The amended requirements, design and initial
|
||||
evaluation profile in `pre-push-safety-net` were approved for task generation on 2026-09-12
|
||||
("ok, next kiro phase"). Task approval remains pending. This is not authorization to implement
|
||||
or a claim that hook performance or capability admission has been demonstrated.
|
||||
("ok, next kiro phase"). The maintainer then approved the task plan and starting group 1
|
||||
("approve."). Implementation is authorized under that plan; task completion still requires
|
||||
independent review and fresh evidence. Hook performance and capability admission remain unproven.
|
||||
|
||||
Reference in New Issue
Block a user