mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
feat(pre-push-safety-net): reuse compatible artifacts throughout replay
This commit is contained in:
@@ -1,6 +1,6 @@
|
|||||||
# Brief: pre-push-safety-net
|
# Brief: pre-push-safety-net
|
||||||
|
|
||||||
Status: requirements, design and tasks approved; groups 1–5 implemented and verified (16/27 subtasks). The first PMPro/Node experiment is measured NO-GO: six cold timeouts, no completed target checks. Groups 6–8 pending; normal capabilities remain disabled and warm hook latency unproven. Language: English.
|
Status: requirements, design and tasks approved; groups 1–6 implemented and verified (20/27 subtasks). Compatible preparation/graph/query reuse is demonstrated, including packaged JavaScript evidence equivalence. Groups 7–8 pending. Rollout remains NO-GO: changed-code feasibility and independent capability qualification are not established; normal capabilities remain disabled. Language: English.
|
||||||
|
|
||||||
## Problem
|
## Problem
|
||||||
|
|
||||||
@@ -10,12 +10,18 @@ maintainer explicitly rejects noisy advisory output as well as noisy blocking ou
|
|||||||
|
|
||||||
## Current State
|
## Current State
|
||||||
|
|
||||||
The scan driver and Joern arbiters exist, with demonstrated PHP/Python cases. Immutable push-tip
|
The existing ranker, Joern arbiters, immutable replay, change comparison, actionability policy
|
||||||
resolution, isolated source materialization and lexical change context are implemented and verified.
|
and absolute deadline are integrated. Vendor code is physically excluded from frontend graphs
|
||||||
Ranker integration, persistent graph caching and an end-to-end deadline remain pending.
|
and targets. Explicit local caching reuses complete discovery, validated isolated graph copies
|
||||||
The driver builds a whole-root graph before cutting the region budget. The latest PMPro ranking
|
and compatible query evidence while reapplying current admission. Changed source/declarations
|
||||||
experiment alone spends about 130 seconds building and 296 seconds extracting evidence. A manually
|
invalidate affected graph units; no incremental Joern mutation is implemented.
|
||||||
selected two-file slice finding a CVE does not demonstrate automatic incremental analysis.
|
|
||||||
|
Group 5's PMPro/Node cold experiment remains NO-GO. Group 6 adds a packaged JavaScript
|
||||||
|
correctness control with equal cached/uncached evidence and admission; an identical-revision
|
||||||
|
hit is fast, while the changed-code deadline remains a separate unmet gate. Automatic
|
||||||
|
context projection and independent capability eligibility remain explicit limits. See
|
||||||
|
implementation-group-6.md and the committed reuse control evidence. Historical whole-PMPro
|
||||||
|
ranking costs and manually selected CVE slices do not establish automatic push latency.
|
||||||
|
|
||||||
## Desired Outcome
|
## Desired Outcome
|
||||||
|
|
||||||
|
|||||||
@@ -147,6 +147,7 @@ New files (proposed, not created by this documentation change):
|
|||||||
| `src/openultrasast/push/contracts.py` | Immutable ref/comparison identities and comparison-owned completion/status records |
|
| `src/openultrasast/push/contracts.py` | Immutable ref/comparison identities and comparison-owned completion/status records |
|
||||||
| `src/openultrasast/push/snapshot.py` | Parse ref updates, resolve bases, materialize blobs, produce change context |
|
| `src/openultrasast/push/snapshot.py` | Parse ref updates, resolve bases, materialize blobs, produce change context |
|
||||||
| `src/openultrasast/push/cache.py` | Artifact keys, validity, atomic publication and eviction |
|
| `src/openultrasast/push/cache.py` | Artifact keys, validity, atomic publication and eviction |
|
||||||
|
| `src/openultrasast/push/reuse.py` | Complete discovery reuse and backend graph/query reuse through the existing driver |
|
||||||
| `src/openultrasast/push/policy.py` | Delta identity, actionability admission and push disposition |
|
| `src/openultrasast/push/policy.py` | Delta identity, actionability admission and push disposition |
|
||||||
| `src/openultrasast/push/runner.py` | Shared deadline, ranker/engine orchestration, per-ref result aggregation |
|
| `src/openultrasast/push/runner.py` | Shared deadline, ranker/engine orchestration, per-ref result aggregation |
|
||||||
| `src/openultrasast/push/report.py` | Compact terminal result and reproducible push manifest |
|
| `src/openultrasast/push/report.py` | Compact terminal result and reproducible push manifest |
|
||||||
@@ -157,8 +158,9 @@ Contract configuration (task 1.4) uses `[push]`: optional `comparison_base`, pos
|
|||||||
`deadline_seconds` (30), positive `cancellation_allowance_seconds` (2), positive integer
|
`deadline_seconds` (30), positive `cancellation_allowance_seconds` (2), positive integer
|
||||||
`cache_max_bytes` (2 GiB), `mode` (`advisory` by default, or `blocking`) and independent
|
`cache_max_bytes` (2 GiB), `mode` (`advisory` by default, or `blocking`) and independent
|
||||||
`incomplete_coverage_policy` (`allow` by default, or `block`). Unknown keys, malformed
|
`incomplete_coverage_policy` (`allow` by default, or `block`). Unknown keys, malformed
|
||||||
sections and coerced/nonfinite numeric values are rejected. These settings are inert until
|
sections and coerced/nonfinite numeric values are rejected. Experimental replay consumes the deadline, mode, coverage policy and optional cache budget.
|
||||||
runner integration; they do not enable a hook or admit experimental capabilities. The cache
|
An explicit `--cache-dir` enables local reuse; omission retains a cold control. These settings
|
||||||
|
do not install a hook or admit experimental capabilities. The cache
|
||||||
limit is a local storage budget, not a latency claim. Completion records are owned by an
|
limit is a local storage budget, not a latency claim. Completion records are owned by an
|
||||||
exact comparison so reused head graphs cannot merge evidence from different bases. Live
|
exact comparison so reused head graphs cannot merge evidence from different bases. Live
|
||||||
leases have no JSON constructor; only the backend can validate and issue them in task 6.1.
|
leases have no JSON constructor; only the backend can validate and issue them in task 6.1.
|
||||||
@@ -251,7 +253,7 @@ cleanup consume the same execution deadline and cancellation allowance as the ba
|
|||||||
Runtime origin is `unspecified` when no declaration establishes browser/server scope;
|
Runtime origin is `unspecified` when no declaration establishes browser/server scope;
|
||||||
JavaScript presence alone does not admit an Express capability. TypeScript and C/C++
|
JavaScript presence alone does not admit an Express capability. TypeScript and C/C++
|
||||||
retain explicit capability limits. These records prepare later artifact manifests;
|
retain explicit capability limits. These records prepare later artifact manifests;
|
||||||
they do not implement persistent graph storage.
|
task 6 now binds them to backend-owned persistent graph manifests and isolated leases.
|
||||||
|
|
||||||
Graph key: digest of every included blob/path and applicable declarations, frontend/engine version,
|
Graph key: digest of every included blob/path and applicable declarations, frontend/engine version,
|
||||||
overlay semantics/options, language, and exclusions. Query key additionally includes graph key,
|
overlay semantics/options, language, and exclusions. Query key additionally includes graph key,
|
||||||
|
|||||||
@@ -103,3 +103,96 @@ the cache never confers eligibility. JSON decoding is non-executable and bounded
|
|||||||
- CLAIM: Task 6.3 semantic keys and completed comparison evidence reuse.
|
- CLAIM: Task 6.3 semantic keys and completed comparison evidence reuse.
|
||||||
- EVIDENCE: production policy integration and cache key tests, 1260 passing suite tests and final static checks.
|
- EVIDENCE: production policy integration and cache key tests, 1260 passing suite tests and final static checks.
|
||||||
- GAPS: Graph/query/preparation reuse must still be connected to replay under 6.4; no rollout claim.
|
- GAPS: Graph/query/preparation reuse must still be connected to replay under 6.4; no rollout claim.
|
||||||
|
|
||||||
|
## Task 6.4 brief
|
||||||
|
|
||||||
|
Requirements 1.1, 2.2, 4.1, 4.2, 4.3, 4.4 and 7.3; design: Push Runner and
|
||||||
|
Cache and Engine Boundary. Explicit `--cache-dir` integrates the three cache layers;
|
||||||
|
omitting it preserves a cold replay control. Complete immutable snapshots are still read
|
||||||
|
and verified by the adapter before persisted discovery can be used. Equal base/head revisions
|
||||||
|
share one live materialization. Discovery keys include exact file identities and shipped code/facts;
|
||||||
|
dependency/declaration changes invalidate them. Graph units remain the existing physical frontend
|
||||||
|
partitions. Source changes rebuild the affected unit; all declaration and exclusion context is
|
||||||
|
bound. There is no incremental graph mutation or changed-file scope selector.
|
||||||
|
|
||||||
|
The runner uses backend-issued copies and per-question cached raw answers, then runs the existing
|
||||||
|
ranker, arbiter, comparison and current admission. Partial answers are not promoted to complete
|
||||||
|
lookups. Complete answers already returned by the engine remain available when persistence cannot
|
||||||
|
finish. Query keys include actual parameters and graph bytes; graph keys do not include unrelated
|
||||||
|
ranking/security-fact versions. Layout/exclusion facts and frontend adaptation bytes do affect them.
|
||||||
|
Integration review repaired the task 6.1 identity to include installed frontend/adaptation bytes,
|
||||||
|
rather than relying only on Joern's release label. This is an owning-contract repair, not a new
|
||||||
|
frontend or a detector change.
|
||||||
|
|
||||||
|
The first source-mounted/package integration trial is retained separately from the final
|
||||||
|
frozen image run. It showed equal cached/uncached evidence and admission (including one
|
||||||
|
new JavaScript injection candidate), a 1.23-second identical-revision hit, and a 30.48-second
|
||||||
|
dependency-edit timeout. Those numbers are preliminary control evidence, not the final image
|
||||||
|
or rollout measurements. Final review also binds installed runtime identity into admission
|
||||||
|
semantics, while avoiding a second backend instance or an unbudgeted availability probe.
|
||||||
|
|
||||||
|
The cache entry lease ends after its validated bytes are copied into the backend's independent
|
||||||
|
working lease. The working copy remains valid if storage later evicts the original entry;
|
||||||
|
queries never operate on the stored original. Partial fresh answers cannot borrow a cached
|
||||||
|
answer's census. Pure discovery remains in the existing bounded child process; persisted
|
||||||
|
JSON is decoded as data, never executable objects.
|
||||||
|
|
||||||
|
## Status Report
|
||||||
|
- STATUS: READY_FOR_REVIEW
|
||||||
|
- TASK: 6.4
|
||||||
|
- RED_PHASE_OUTPUT: `/tmp/group6-4-red.log`: graph/query reuse and replay discovery/cache-location tests failed before integration (temporary switch OFF). Enabled implementation passed; switch removed. Additional regressions cover mixed cached/fresh census failure and installed runtime provenance.
|
||||||
|
- TESTS_RUN: 89 final focused tests passed; full suite 1268 passed, 9 skipped in 108.74s. Ruff, formatting, mypy (114 source files), compileall, detection/map/pair gates and Docker build/packaged CLI smoke passed.
|
||||||
|
- EVIDENCE: final packaged JavaScript control exited 0 with exact cached/uncached scope, question outcomes, findings and admission equality. One new injection is retained as diagnostic evidence and denied normal admission by the empty registry. Source/dependency edits invalidate graph reuse and retain timed-out coverage artifacts.
|
||||||
|
- CONCERNS: identical-revision hits are development correctness controls, not representative warm p95 or capability qualification. Main-context review fallback remains explicit.
|
||||||
|
|
||||||
|
## Review Verdict
|
||||||
|
- VERDICT: APPROVED
|
||||||
|
- TASK: 6.4
|
||||||
|
- MECHANICAL_RESULTS: canonical tests and regression gates passed; final image reached the usable CLI and completed real replay controls. No introduced placeholders, secrets or boundary violations found.
|
||||||
|
- FINDINGS: installation-byte identity now also binds comparison/admission provenance. The predecessor no-unbudgeted-probe regression caught an unnecessary backend instance in provenance; the repair uses a pure runtime-identity function and the final regression passes. Mixed fresh/cached answers require the fresh query's own valid census before being retained as complete. Snapshot reads, discovery, ranker/driver selection, vendor boundary and independent push/coverage status remain intact.
|
||||||
|
- SUMMARY: Task 6.4 integrates valid local reuse and truthful misses under the existing transaction deadline without changing detection or admission rules.
|
||||||
|
|
||||||
|
## Verification Result
|
||||||
|
- STATUS: VERIFIED
|
||||||
|
- CLAIM_TYPE: TASK
|
||||||
|
- CLAIM: Group 6 implementation (6.1–6.4), with compatible replay evidence and admission equivalence.
|
||||||
|
- EVIDENCE: final tests/static checks, native packaged Joern controls, and committed measurement record below.
|
||||||
|
- GAPS: Rollout remains NO-GO. Groups 7–8, representative changed-code completion/p95 and independent capability qualification remain pending. Automatic projection limitations are unchanged.
|
||||||
|
|
||||||
|
## Final packaged evidence
|
||||||
|
|
||||||
|
Image: `sha256:6622b846996989a321d81bfa6c25e792e3f217984d7ede66f830cff1206a634c`.
|
||||||
|
Nonroot, network disabled, 3 GiB memory; controls ran serially without concurrent
|
||||||
|
tests/builds. Filesystem cache was not flushed. Each case is one development control,
|
||||||
|
not a representative p95 population.
|
||||||
|
|
||||||
|
| Run | Full CLI seconds | Graph hits | Query-answer hits |
|
||||||
|
|---|---:|---:|---:|
|
||||||
|
| populate | 154.848 | 0 | 0 |
|
||||||
|
| warm-identical | 2.002 | 2 | 37 |
|
||||||
|
| uncached | 130.432 | 0 | 0 |
|
||||||
|
| dependency-edit | 30.458 | 0 | 0 |
|
||||||
|
| one-function-edit | 30.479 | 0 | 0 |
|
||||||
|
| uncached, same revisions, 30s budget | 30.606 | 0 | 0 |
|
||||||
|
|
||||||
|
Cached and uncached laboratory runs have **equal scope, question outcomes, findings and
|
||||||
|
admission**, including the known introduced JavaScript injection. The 30-second identical
|
||||||
|
cached run retains that same evidence: one completed supported head injection check plus
|
||||||
|
four base comparison checks. The same revisions without caching complete zero checks in
|
||||||
|
the 30-second control. All six CLI artifacts were retained, including timed-out edits.
|
||||||
|
Warm source/dependency edits correctly invalidate the affected graph and produce incomplete
|
||||||
|
coverage; they do not replay the old answer. The empty normal capability registry denies the
|
||||||
|
new diagnostic candidate for `capability_unavailable`; this is not a clean repository verdict.
|
||||||
|
|
||||||
|
The manifest, full artifacts and stdout/stderr are retained in
|
||||||
|
`benchmarks/measurements/2026-09-13-reuse-control.json` and the associated final/initial
|
||||||
|
compressed bundles. Each bundle maps original filenames to exact UTF-8 content; decompression
|
||||||
|
was compared with the original files and SHA-256 hashes recorded. The committed reusable
|
||||||
|
harness is `benchmarks/push/reuse_control.py`; the extra same-revision 30-second uncached
|
||||||
|
command is recorded in the final bundle.
|
||||||
|
|
||||||
|
**Group 6 implementation: verified. Rollout: NO-GO.** Changed-code feasibility remains
|
||||||
|
unmet in these controls, and independent precision/recall/capability populations have not
|
||||||
|
qualified. The spec now explicitly requires stage-level investigation of remaining timeouts
|
||||||
|
under task 8.2, retaining its original thresholds and population rules. Group 7 is next;
|
||||||
|
no hook has been installed and no normal capability enabled.
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
{
|
{
|
||||||
"feature_name": "pre-push-safety-net",
|
"feature_name": "pre-push-safety-net",
|
||||||
"created_at": "2026-09-12T15:35:00Z",
|
"created_at": "2026-09-12T15:35:00Z",
|
||||||
"updated_at": "2026-09-13T13:22:55.634332+00:00",
|
"updated_at": "2026-09-13T15:06:56.856952+00:00",
|
||||||
"language": "en",
|
"language": "en",
|
||||||
"phase": "implementation",
|
"phase": "implementation",
|
||||||
"approvals": {
|
"approvals": {
|
||||||
@@ -19,7 +19,7 @@
|
|||||||
}
|
}
|
||||||
},
|
},
|
||||||
"ready_for_implementation": true,
|
"ready_for_implementation": true,
|
||||||
"implementation_approval_note": "Groups1\u20135 implemented and verified (16/27). Group5 used manual Kiro fallback review, repaired observed predecessor deadline bugs, and executed a newly frozen PMPro/Node experiment. Measured feasibility NO-GO; real capabilities remain disabled. Groups6\u20138 pending.",
|
"implementation_approval_note": "Groups 1\u20136 implemented and verified (20/27); group 6 authorized as a whole. Main-context Kiro review fallback after fresh dispatch thread limit. Valid reuse is demonstrated; real capability registry remains empty and rollout gates remain unmet. Groups 7\u20138 pending.",
|
||||||
"depends_on": [
|
"depends_on": [
|
||||||
"contributor-scan"
|
"contributor-scan"
|
||||||
],
|
],
|
||||||
@@ -98,19 +98,24 @@
|
|||||||
"4.2",
|
"4.2",
|
||||||
"4.3",
|
"4.3",
|
||||||
"5.1",
|
"5.1",
|
||||||
"5.2"
|
"5.2",
|
||||||
|
"6.1",
|
||||||
|
"6.2",
|
||||||
|
"6.3",
|
||||||
|
"6.4"
|
||||||
],
|
],
|
||||||
"remaining_tasks": 11,
|
"remaining_tasks": 7,
|
||||||
"verification_record": "implementation-group-5.md",
|
"verification_record": "implementation-group-6.md",
|
||||||
"verification_records": [
|
"verification_records": [
|
||||||
"implementation-group-1.md",
|
"implementation-group-1.md",
|
||||||
"implementation-group-2.md",
|
"implementation-group-2.md",
|
||||||
"implementation-group-3.md",
|
"implementation-group-3.md",
|
||||||
"implementation-group-4.md",
|
"implementation-group-4.md",
|
||||||
"implementation-group-5.md"
|
"implementation-group-5.md",
|
||||||
|
"implementation-group-6.md"
|
||||||
],
|
],
|
||||||
"next_group": 6,
|
"next_group": 7,
|
||||||
"blocked_tasks": [],
|
"blocked_tasks": [],
|
||||||
"feasibility": "NO-GO: six cold replay timeouts; no completed supported target checks; warm performance and eligibility unproven"
|
"feasibility": "NO-GO for rollout: compatible identical-revision reuse verified; representative changed-code p95/completion and independent alert eligibility remain unmet."
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,11 +1,12 @@
|
|||||||
# Implementation Plan: pre-push-safety-net
|
# Implementation Plan: pre-push-safety-net
|
||||||
|
|
||||||
Status: task plan approved by the maintainer on 2026-09-12. Groups 1–5 (tasks 1.1–5.2)
|
Status: task plan approved by the maintainer on 2026-09-12. Groups 1–6 (tasks 1.1–6.4)
|
||||||
are implemented, reviewed and verified (16/27 subtasks). Tasks 4.3 and group 5 used the
|
are implemented, reviewed and verified (20/27 subtasks). Tasks 4.3 and groups 5–6 used the
|
||||||
documented Kiro manual review fallback after agent dispatch reached its thread limit.
|
explicit Kiro manual review fallback after fresh agent dispatch reached its thread limit.
|
||||||
The first experiment is measured NO-GO: all six cold PMPro/Node cases timed out without
|
Group 5's six cold PMPro/Node timeouts remain historical NO-GO evidence. Group 6 verifies
|
||||||
completed target checks. Groups 6–8 remain pending; no production capability or warm
|
compatible reuse with equal evidence/admission in a packaged authored JavaScript control;
|
||||||
hook latency is admitted.
|
its identical-revision run completed in 2.00s. This is not representative warm p95 or
|
||||||
|
independent capability qualification. Groups 7–8 remain pending; rollout remains NO-GO.
|
||||||
|
|
||||||
## Execution contract
|
## Execution contract
|
||||||
|
|
||||||
@@ -160,7 +161,7 @@ hook latency is admitted.
|
|||||||
- _Depends: 1.2, 1.3, 5.1_
|
- _Depends: 1.2, 1.3, 5.1_
|
||||||
- _Requirements: 6.2, 6.3, 6.4, 6.5, 8.1, 8.2, 8.3, 8.4_
|
- _Requirements: 6.2, 6.3, 6.4, 6.5, 8.1, 8.2, 8.3, 8.4_
|
||||||
|
|
||||||
- [ ] 6. Reuse valid graphs and results without stale conclusions
|
- [x] 6. Reuse valid graphs and results without stale conclusions
|
||||||
- [x] 6.1 Validate graph artifact identity and leases
|
- [x] 6.1 Validate graph artifact identity and leases
|
||||||
- Have the backend own source/declaration/exclusion digests, frontend/engine/overlay options, graph census and complete/partial identity.
|
- Have the backend own source/declaration/exclusion digests, frontend/engine/overlay options, graph census and complete/partial identity.
|
||||||
- Permit validated graph leases in the generic driver while preserving the fresh-build path; isolate potentially mutating loads and retain bounded cleanup.
|
- Permit validated graph leases in the generic driver while preserving the fresh-build path; isolate potentially mutating loads and retain bounded cleanup.
|
||||||
@@ -180,7 +181,7 @@ hook latency is admitted.
|
|||||||
- Done when source, context, fact, query, mode, policy and optional model changes invalidate the appropriate layer, while an identical compatible request returns the same evidence and provenance.
|
- Done when source, context, fact, query, mode, policy and optional model changes invalidate the appropriate layer, while an identical compatible request returns the same evidence and provenance.
|
||||||
- _Boundary: Push cache, policy integration_
|
- _Boundary: Push cache, policy integration_
|
||||||
- _Requirements: 3.4, 4.3, 4.4, 6.4, 7.2, 7.3_
|
- _Requirements: 3.4, 4.3, 4.4, 6.4, 7.2, 7.3_
|
||||||
- [ ] 6.4 Integrate cache reuse into the shared push deadline
|
- [x] 6.4 Integrate cache reuse into the shared push deadline
|
||||||
- Reuse complete preparation/discovery by immutable tree and discovery/fact/configuration identity, retaining readable snapshot and declaration/boundary records; measure its costs and hits separately from graph/query reuse.
|
- Reuse complete preparation/discovery by immutable tree and discovery/fact/configuration identity, retaining readable snapshot and declaration/boundary records; measure its costs and hits separately from graph/query reuse.
|
||||||
- Reuse identical snapshots/unchanged declared units across refs without merging different comparisons; changed units rebuild and stale graph relationships remain hints only.
|
- Reuse identical snapshots/unchanged declared units across refs without merging different comparisons; changed units rebuild and stale graph relationships remain hints only.
|
||||||
- Account for lookup, lock wait, lease, build/query and result persistence in the same deadline; preserve completed outcomes on cancellation.
|
- Account for lookup, lock wait, lease, build/query and result persistence in the same deadline; preserve completed outcomes on cancellation.
|
||||||
@@ -219,6 +220,7 @@ hook latency is admitted.
|
|||||||
- _Requirements: 1.1, 1.2, 1.3, 1.4, 2.2, 2.3, 4.2, 4.4, 5.2, 5.5, 7.1, 7.4, 8.2_
|
- _Requirements: 1.1, 1.2, 1.3, 1.4, 2.2, 2.3, 4.2, 4.4, 5.2, 5.5, 7.1, 7.4, 8.2_
|
||||||
- [ ] 8.2 Measure realistic cache and deadline feasibility
|
- [ ] 8.2 Measure realistic cache and deadline feasibility
|
||||||
- Measure identical-tip reuse, a warm one-function edit, dependency/configuration edits, cold starts, repository growth and multi-ref workloads on declared hardware.
|
- Measure identical-tip reuse, a warm one-function edit, dependency/configuration edits, cold starts, repository growth and multi-ref workloads on declared hardware.
|
||||||
|
- If reuse correctness passes but changed-code checks time out, identify preparation, frontend/overlay/census and query costs separately and route the remaining optimization to its owning boundary before rollout GO. Preserve the declared gates and do not substitute identical-tip latency.
|
||||||
- Publish p50/p95, timeouts, cancellation allowance, completed/deferred population and stage costs; the denominator is fixed before ranker selection.
|
- Publish p50/p95, timeouts, cancellation allowance, completed/deferred population and stage costs; the denominator is fixed before ranker selection.
|
||||||
- Done when the versioned scorecard evaluates the accepted latency/coverage gates without hiding timeouts or calling repeated-identical-tip hits representative warm edits; a failed gate remains NO-GO for rollout.
|
- Done when the versioned scorecard evaluates the accepted latency/coverage gates without hiding timeouts or calling repeated-identical-tip hits representative warm edits; a failed gate remains NO-GO for rollout.
|
||||||
- _Boundary: Evaluation harness, runtime integration_
|
- _Boundary: Evaluation harness, runtime integration_
|
||||||
@@ -244,3 +246,5 @@ hook latency is admitted.
|
|||||||
|
|
||||||
- Task5.2 measured PMPro timing out after both snapshots were materialized but before change context/graph analysis. Group6 must account for compatible snapshot/discovery reuse as well as graph reuse; identical-tip hits cannot substitute for the later representative changed-code latency gate. No new detector, scope heuristic or incremental CPG mutation is implied.
|
- Task5.2 measured PMPro timing out after both snapshots were materialized but before change context/graph analysis. Group6 must account for compatible snapshot/discovery reuse as well as graph reuse; identical-tip hits cannot substitute for the later representative changed-code latency gate. No new detector, scope heuristic or incremental CPG mutation is implied.
|
||||||
- Cancellation must cover semantic planning and reporting, not only engine subprocesses: `_collect` and `_scope_work` now consume the shared deadline, and replay instantiates the budget-aware backend once. Preserve the 400-question failure-census and real Node artifact-retention regressions.
|
- Cancellation must cover semantic planning and reporting, not only engine subprocesses: `_collect` and `_scope_work` now consume the shared deadline, and replay instantiates the budget-aware backend once. Preserve the 400-question failure-census and real Node artifact-retention regressions.
|
||||||
|
|
||||||
|
- Group 6: complete discovery/graph/query reuse is explicit via `--cache-dir`; source/declaration edits invalidate affected units. Installed frontend/adaptation bytes now bind graph and admission provenance. A valid cached census cannot qualify a fresh answer that omitted its own census. Identical-revision timing is not the changed-code rollout gate.
|
||||||
|
|||||||
@@ -46,7 +46,7 @@ engine are out of scope.
|
|||||||
- [ ] pre-push-safety-net — Immutable pushed tips, existing-ranker integration, compatible artifact
|
- [ ] pre-push-safety-net — Immutable pushed tips, existing-ranker integration, compatible artifact
|
||||||
reuse, total deadline, change attribution, actionability admission and compact hook behavior.
|
reuse, total deadline, change attribution, actionability admission and compact hook behavior.
|
||||||
Dependencies: contributor-scan. Requirements/design approved 2026-09-12; tasks generated
|
Dependencies: contributor-scan. Requirements/design approved 2026-09-12; tasks generated
|
||||||
(8 groups, 27 executable subtasks), independently reviewed and approved; groups 1–5 are implemented and verified (16/27 tasks), including the resolved frontend-retention prerequisite.
|
(8 groups, 27 executable subtasks), independently reviewed and approved; groups 1–6 are implemented and verified (20/27 tasks), including the resolved frontend-retention prerequisite.
|
||||||
|
|
||||||
### Next concrete step
|
### Next concrete step
|
||||||
|
|
||||||
@@ -59,14 +59,19 @@ its deadline in preparation/discovery after materializing both trees; Node reach
|
|||||||
driver but did not complete checks. The exact 11-case population remains unresolved.
|
driver but did not complete checks. The exact 11-case population remains unresolved.
|
||||||
No normal capability is enabled and these six cold samples establish no warm p95 claim.
|
No normal capability is enabled and these six cold samples establish no warm p95 claim.
|
||||||
|
|
||||||
Group 6 is next for valid graph/result identity, leases and reuse. Account for snapshot
|
Group 6 now provides complete preparation/discovery, graph and query reuse through explicit
|
||||||
and discovery costs before treating graph persistence as the sole optimization; record
|
`--cache-dir`, bounded publication and isolated graph leases. A packaged authored JavaScript
|
||||||
compatible preparation reuse alongside graph reuse. Representative changed-code warm
|
control preserved the same evidence/admission as an uncached run; its identical-revision
|
||||||
latency and coverage must still be measured later, without substituting identical-tip hits.
|
replay took 2.00s. Source/declaration changes invalidate affected units. This correctness
|
||||||
Do not change ranking weights, add a second source-slicer or assume incremental Joern
|
control does not establish representative changed-code p95, independent alert precision or
|
||||||
mutation from this negative performance evidence.
|
eligibility. Rollout remains NO-GO and normal capabilities remain disabled.
|
||||||
|
|
||||||
Tasks 4.3 and group 5 used the documented Kiro manual review fallback after the agent
|
Group 7 is next for the real multi-ref push interface and opt-in hook integration. Group 8
|
||||||
|
must still pass the joint gates on declared populations. Keep the existing thresholds and
|
||||||
|
profile preparation, graph construction and queries separately when investigating misses;
|
||||||
|
do not substitute identical-tip hits for warm changed-code performance.
|
||||||
|
|
||||||
|
Tasks 4.3 and groups 5–6 used the documented Kiro manual review fallback after the agent
|
||||||
thread limit; this is not independent review. See `pre-push-safety-net/implementation-group-5.md`
|
thread limit; this is not independent review. See `pre-push-safety-net/implementation-group-5.md`
|
||||||
and `benchmarks/measurements/2026-09-13-first-replay-experiment.json` for provenance,
|
and `benchmarks/measurements/2026-09-13-first-replay-experiment.json` for provenance,
|
||||||
initial instrument failures, their repairs and the newly frozen final trial. The NodeGoat
|
initial instrument failures, their repairs and the newly frozen final trial. The NodeGoat
|
||||||
|
|||||||
@@ -79,16 +79,21 @@ updates as a shortcut to an unmeasured latency promise.
|
|||||||
The product direction above is user-requested. The amended requirements, design and initial
|
The product direction above is user-requested. The amended requirements, design and initial
|
||||||
evaluation profile in `pre-push-safety-net` were approved for task generation on 2026-09-12
|
evaluation profile in `pre-push-safety-net` were approved for task generation on 2026-09-12
|
||||||
("ok, next kiro phase"). The maintainer then approved the task plan and starting group 1
|
("ok, next kiro phase"). The maintainer then approved the task plan and starting group 1
|
||||||
("approve.") and then group 2 ("ok, next kiro phase, approved"). Groups 1–5 are now
|
("approve.") and then group 2 ("ok, next kiro phase, approved"). Groups 1–6 are now
|
||||||
implemented, reviewed and verified (16/27), including the frontend-retention prerequisite
|
implemented, reviewed and verified (20/27), including the frontend-retention prerequisite
|
||||||
and the deadline regression exposed by full Node replay. Tasks 4.3 and group 5 used the
|
and the deadline regression exposed by full Node replay. Tasks 4.3 and groups 5–6 used the
|
||||||
explicit Kiro manual review fallback after the agent thread limit.
|
explicit Kiro manual review fallback after the agent thread limit.
|
||||||
|
|
||||||
The first newly frozen PMPro/Node experiment is measured **NO-GO**: all six cold cases
|
The first newly frozen PMPro/Node experiment is measured **NO-GO**: all six cold cases
|
||||||
timed out without completed supported target checks, while retaining all 11 declared cases.
|
timed out without completed supported target checks, while retaining all 11 declared cases.
|
||||||
The repaired CLI saved all artifacts and returned in 30.55–30.64 seconds. PMPro exhausted
|
The repaired CLI saved all artifacts and returned in 30.55–30.64 seconds. PMPro exhausted
|
||||||
the deadline in preparation/discovery before graph analysis, so compatible preparation
|
the deadline in preparation/discovery before graph analysis, so compatible preparation
|
||||||
reuse matters alongside graph persistence. Group 6 is next; warm changed-code performance
|
reuse matters alongside graph persistence. Group 6 now provides compatible local reuse; warm changed-code performance
|
||||||
and useful coverage remain mandatory later gates. No ranking/fact adaptation or holdout
|
and useful coverage remain mandatory later gates. No ranking/fact adaptation or holdout
|
||||||
inspection occurred in the final transfer run. Real capability eligibility and automatic
|
inspection occurred in the final transfer run. Real capability eligibility and automatic
|
||||||
dominance context projection remain unproven; the default capability registry is empty.
|
dominance context projection remain unproven; the default capability registry is empty.
|
||||||
|
|
||||||
|
Group 6 verified equal cached/uncached evidence and current admission in a packaged authored
|
||||||
|
JavaScript control. Source/declaration edits invalidate affected units. Identical-revision
|
||||||
|
latency does not qualify representative changed-code performance. Requirements and rollout
|
||||||
|
thresholds remain unchanged; groups 7–8 and independent capability admission remain pending.
|
||||||
|
|||||||
@@ -1,10 +1,10 @@
|
|||||||
{
|
{
|
||||||
"counts": {
|
"counts": {
|
||||||
"load_bearing": 103,
|
"load_bearing": 104,
|
||||||
"standalone_capability": 9
|
"standalone_capability": 9
|
||||||
},
|
},
|
||||||
"modules": 112,
|
"modules": 113,
|
||||||
"note": "Refreshed for task 6.2: bounded local artifact storage, integrated by task 6.4.",
|
"note": "Refreshed for group 6: replay reaches compatible graph, query, discovery and comparison reuse.",
|
||||||
"orphaned": [],
|
"orphaned": [],
|
||||||
"rows": [
|
"rows": [
|
||||||
{
|
{
|
||||||
@@ -118,6 +118,7 @@
|
|||||||
"config",
|
"config",
|
||||||
"cpg.artifact",
|
"cpg.artifact",
|
||||||
"model.contracts",
|
"model.contracts",
|
||||||
|
"push.cache",
|
||||||
"push.contracts",
|
"push.contracts",
|
||||||
"push.policy"
|
"push.policy"
|
||||||
],
|
],
|
||||||
@@ -139,6 +140,7 @@
|
|||||||
"model.taint",
|
"model.taint",
|
||||||
"push.cache",
|
"push.cache",
|
||||||
"push.policy",
|
"push.policy",
|
||||||
|
"push.reuse",
|
||||||
"push.runner"
|
"push.runner"
|
||||||
],
|
],
|
||||||
"module": "cpg",
|
"module": "cpg",
|
||||||
@@ -148,7 +150,10 @@
|
|||||||
"classification": "load_bearing",
|
"classification": "load_bearing",
|
||||||
"importers": [
|
"importers": [
|
||||||
"cpg.backend",
|
"cpg.backend",
|
||||||
"push.cache"
|
"push.cache",
|
||||||
|
"push.policy",
|
||||||
|
"push.reuse",
|
||||||
|
"push.runner"
|
||||||
],
|
],
|
||||||
"module": "cpg.artifact",
|
"module": "cpg.artifact",
|
||||||
"reason": "pre-push-safety-net Req 4.3/7.3: backend graph identity and live lease contract; validation integration pending"
|
"reason": "pre-push-safety-net Req 4.3/7.3: backend graph identity and live lease contract; validation integration pending"
|
||||||
@@ -167,6 +172,7 @@
|
|||||||
"model.taint",
|
"model.taint",
|
||||||
"push.cache",
|
"push.cache",
|
||||||
"push.policy",
|
"push.policy",
|
||||||
|
"push.reuse",
|
||||||
"push.runner"
|
"push.runner"
|
||||||
],
|
],
|
||||||
"module": "cpg.backend",
|
"module": "cpg.backend",
|
||||||
@@ -374,6 +380,7 @@
|
|||||||
"push.contracts",
|
"push.contracts",
|
||||||
"push.policy",
|
"push.policy",
|
||||||
"push.report",
|
"push.report",
|
||||||
|
"push.reuse",
|
||||||
"push.runner",
|
"push.runner",
|
||||||
"push.snapshot"
|
"push.snapshot"
|
||||||
],
|
],
|
||||||
@@ -421,6 +428,7 @@
|
|||||||
"push.contracts",
|
"push.contracts",
|
||||||
"push.policy",
|
"push.policy",
|
||||||
"push.report",
|
"push.report",
|
||||||
|
"push.reuse",
|
||||||
"push.runner",
|
"push.runner",
|
||||||
"push.snapshot"
|
"push.snapshot"
|
||||||
],
|
],
|
||||||
@@ -484,7 +492,8 @@
|
|||||||
"importers": [
|
"importers": [
|
||||||
"model.partitions",
|
"model.partitions",
|
||||||
"model.scan",
|
"model.scan",
|
||||||
"preprocess"
|
"preprocess",
|
||||||
|
"push.runner"
|
||||||
],
|
],
|
||||||
"module": "model.layout",
|
"module": "model.layout",
|
||||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
|
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
|
||||||
@@ -513,6 +522,7 @@
|
|||||||
"model.layout",
|
"model.layout",
|
||||||
"model.scan",
|
"model.scan",
|
||||||
"push.policy",
|
"push.policy",
|
||||||
|
"push.reuse",
|
||||||
"push.runner"
|
"push.runner"
|
||||||
],
|
],
|
||||||
"module": "model.regions",
|
"module": "model.regions",
|
||||||
@@ -690,6 +700,7 @@
|
|||||||
"cli",
|
"cli",
|
||||||
"push.policy",
|
"push.policy",
|
||||||
"push.report",
|
"push.report",
|
||||||
|
"push.reuse",
|
||||||
"push.runner",
|
"push.runner",
|
||||||
"push.snapshot"
|
"push.snapshot"
|
||||||
],
|
],
|
||||||
@@ -698,7 +709,11 @@
|
|||||||
},
|
},
|
||||||
{
|
{
|
||||||
"classification": "load_bearing",
|
"classification": "load_bearing",
|
||||||
"importers": [],
|
"importers": [
|
||||||
|
"push.policy",
|
||||||
|
"push.reuse",
|
||||||
|
"push.runner"
|
||||||
|
],
|
||||||
"module": "push.cache",
|
"module": "push.cache",
|
||||||
"reason": "pre-push-safety-net task 6.2: bounded artifact publication and leased eviction; runner integration in 6.4"
|
"reason": "pre-push-safety-net task 6.2: bounded artifact publication and leased eviction; runner integration in 6.4"
|
||||||
},
|
},
|
||||||
@@ -707,6 +722,7 @@
|
|||||||
"importers": [
|
"importers": [
|
||||||
"push.policy",
|
"push.policy",
|
||||||
"push.report",
|
"push.report",
|
||||||
|
"push.reuse",
|
||||||
"push.runner",
|
"push.runner",
|
||||||
"push.snapshot"
|
"push.snapshot"
|
||||||
],
|
],
|
||||||
@@ -730,6 +746,14 @@
|
|||||||
"module": "push.report",
|
"module": "push.report",
|
||||||
"reason": "pre-push-safety-net Req 5.1/5.2/5.5/7.3: compact output and complete artifacts; runner integration pending"
|
"reason": "pre-push-safety-net Req 5.1/5.2/5.5/7.3: compact output and complete artifacts; runner integration pending"
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
"classification": "load_bearing",
|
||||||
|
"importers": [
|
||||||
|
"push.runner"
|
||||||
|
],
|
||||||
|
"module": "push.reuse",
|
||||||
|
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
|
||||||
|
},
|
||||||
{
|
{
|
||||||
"classification": "load_bearing",
|
"classification": "load_bearing",
|
||||||
"importers": [
|
"importers": [
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
Binary file not shown.
Binary file not shown.
@@ -137,3 +137,24 @@ flushed. Six one-shot runs are diagnostic samples, not representative warm p95 e
|
|||||||
Normal capabilities remain disabled, so zero alerts do not establish precision or useful
|
Normal capabilities remain disabled, so zero alerts do not establish precision or useful
|
||||||
recall. A separately named longer lab run may obtain witnesses, but cannot replace the
|
recall. A separately named longer lab run may obtain witnesses, but cannot replace the
|
||||||
30-second measurements or satisfy the hook's latency gate.
|
30-second measurements or satisfy the hook's latency gate.
|
||||||
|
|
||||||
|
### Group 6 reuse correctness control
|
||||||
|
|
||||||
|
`reuse_control.py` creates an isolated, authored JavaScript history and invokes the packaged
|
||||||
|
replay command for a cache-populating run, a 30-second identical-revision run, an uncached
|
||||||
|
control, and dependency/one-function edits. It retains every artifact and checks equal evidence
|
||||||
|
and current admission for compatible completed questions. This is development control data,
|
||||||
|
not the untouched eligibility population or the representative warm p95 gate.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
docker run --rm --network none --memory 3g \
|
||||||
|
-v /path/to/empty/results:/results \
|
||||||
|
-v "$PWD/benchmarks/push/reuse_control.py:/control.py:ro" \
|
||||||
|
--entrypoint python openultrasast:dev /control.py --out /results
|
||||||
|
```
|
||||||
|
|
||||||
|
The output directory must be empty and writable by the image's nonroot user. Cold runs use a
|
||||||
|
600-second laboratory budget; the identical-revision and changed-code controls retain the
|
||||||
|
30-second deadline and 2-second cancellation allowance. Default normal capabilities remain
|
||||||
|
empty. A passing reuse control cannot authorize rollout or turn unresolved coverage into clean
|
||||||
|
analysis; groups 7–8 still own the real interface and joint evaluation.
|
||||||
|
|||||||
@@ -0,0 +1,133 @@
|
|||||||
|
"""Packaged JavaScript reuse correctness control, not capability qualification.
|
||||||
|
|
||||||
|
Run with --out pointing to an empty directory outside any analyzed repository.
|
||||||
|
Authored development code is never an independent precision/recall population.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import subprocess
|
||||||
|
import time
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument("--out", type=Path, required=True)
|
||||||
|
args = parser.parse_args()
|
||||||
|
out = args.out.resolve()
|
||||||
|
out.mkdir(parents=True, exist_ok=True)
|
||||||
|
if any(out.iterdir()):
|
||||||
|
parser.error("output directory must be empty")
|
||||||
|
repo = out / "repo"
|
||||||
|
repo.mkdir()
|
||||||
|
|
||||||
|
|
||||||
|
def git(*args):
|
||||||
|
return subprocess.run(["git", "-C", str(repo), *args], check=True, capture_output=True, text=True).stdout.strip()
|
||||||
|
|
||||||
|
|
||||||
|
git("init")
|
||||||
|
git("config", "user.name", "Replay control")
|
||||||
|
git("config", "user.email", "[email protected]")
|
||||||
|
git("config", "core.hooksPath", "/dev/null")
|
||||||
|
source = repo / "api.js"
|
||||||
|
source.write_text('function handler(req, res) {\n const value = "fixed";\n eval(value);\n}\napp.post("/run", handler);\n')
|
||||||
|
assert len(source.read_bytes()) > 0
|
||||||
|
git("add", "api.js")
|
||||||
|
git("commit", "-m", "fixed")
|
||||||
|
base = git("rev-parse", "HEAD")
|
||||||
|
source.write_text(source.read_text().replace('"fixed"', "req.body.value"))
|
||||||
|
git("add", "api.js")
|
||||||
|
git("commit", "-m", "introduce")
|
||||||
|
head = git("rev-parse", "HEAD")
|
||||||
|
runs = []
|
||||||
|
|
||||||
|
|
||||||
|
def run(name, deadline, cache=True, tip=head):
|
||||||
|
artifact = out / (name + ".json")
|
||||||
|
cmd = [
|
||||||
|
"python",
|
||||||
|
"-m",
|
||||||
|
"openultrasast.cli",
|
||||||
|
"pre-push",
|
||||||
|
str(repo),
|
||||||
|
"--base",
|
||||||
|
base,
|
||||||
|
"--head",
|
||||||
|
tip,
|
||||||
|
"--artifact",
|
||||||
|
str(artifact),
|
||||||
|
"--deadline",
|
||||||
|
str(deadline),
|
||||||
|
"--max-regions",
|
||||||
|
"20",
|
||||||
|
]
|
||||||
|
if cache:
|
||||||
|
cmd += ["--cache-dir", str(out / "cache")]
|
||||||
|
start = time.monotonic()
|
||||||
|
process = subprocess.run(cmd, capture_output=True, text=True, timeout=deadline + 10)
|
||||||
|
elapsed = time.monotonic() - start
|
||||||
|
(out / (name + ".stdout")).write_text(process.stdout)
|
||||||
|
(out / (name + ".stderr")).write_text(process.stderr)
|
||||||
|
assert artifact.is_file(), (process.returncode, process.stdout, process.stderr)
|
||||||
|
data = json.loads(artifact.read_text())
|
||||||
|
run = {
|
||||||
|
"name": name,
|
||||||
|
"elapsed_seconds": elapsed,
|
||||||
|
"exit_code": process.returncode,
|
||||||
|
"timings": data["timings"],
|
||||||
|
"result": data["result"],
|
||||||
|
"artifact": str(artifact),
|
||||||
|
}
|
||||||
|
runs.append(run)
|
||||||
|
(out / "runs.json").write_text(json.dumps(runs, indent=2))
|
||||||
|
print(json.dumps(run), flush=True)
|
||||||
|
return data
|
||||||
|
|
||||||
|
|
||||||
|
populated = run("populate", 600)
|
||||||
|
warm = run("warm-identical", 30)
|
||||||
|
cold = run("uncached", 600, False)
|
||||||
|
(repo / "package.json").write_text('{"dependencies":{"express":"5.0.0"}}\n')
|
||||||
|
git("add", "package.json")
|
||||||
|
git("commit", "-m", "dependency change")
|
||||||
|
changed = git("rev-parse", "HEAD")
|
||||||
|
run("dependency-edit", 30, tip=changed)
|
||||||
|
git("rm", "package.json")
|
||||||
|
source.write_text(source.read_text().replace("req.body.value", 'req.body.value + "!"'))
|
||||||
|
git("add", "api.js")
|
||||||
|
git("commit", "-m", "one function edit")
|
||||||
|
edited = git("rev-parse", "HEAD")
|
||||||
|
run("one-function-edit", 30, tip=edited)
|
||||||
|
|
||||||
|
|
||||||
|
def evidence(data):
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
"side": s["side"],
|
||||||
|
"scope": s["scan"]["scope"],
|
||||||
|
"questions": s["scan"]["question_outcomes"],
|
||||||
|
"findings": s["scan"]["findings"],
|
||||||
|
"degradations": s["scan"]["degradations"],
|
||||||
|
}
|
||||||
|
for s in data["scans"]
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
summary = {
|
||||||
|
"cold_populated_evidence_equal": evidence(cold) == evidence(populated),
|
||||||
|
"cold_warm_evidence_equal": evidence(cold) == evidence(warm),
|
||||||
|
"admission_equal": cold["admission"] == warm["admission"],
|
||||||
|
"runs": runs,
|
||||||
|
"source_bytes": len(source.read_bytes()),
|
||||||
|
"rollout": "not qualified; authored development control only",
|
||||||
|
}
|
||||||
|
(out / "summary.json").write_text(json.dumps(summary, indent=2))
|
||||||
|
print(json.dumps(summary), flush=True)
|
||||||
|
assert summary["cold_populated_evidence_equal"] and summary["cold_warm_evidence_equal"] and summary["admission_equal"]
|
||||||
|
assert warm["timings"]["graph_hits"] == 2 and warm["timings"]["query_hits"] > 0
|
||||||
|
assert any(item["candidate"]["delta"]["novelty"] == "new" for item in warm["admission"]["dispositions"])
|
||||||
|
assert not any(data["admission"]["defects"] for data in (populated, warm, cold))
|
||||||
|
assert runs[1]["elapsed_seconds"] <= 32
|
||||||
|
assert all(
|
||||||
|
r["timings"]["graph_hits"] <= 1 and r["elapsed_seconds"] <= 32 for r in runs if r["name"] in ("dependency-edit", "one-function-edit")
|
||||||
|
)
|
||||||
@@ -266,3 +266,19 @@ it does not demand a positive after an interrupted query. Run with the network-d
|
|||||||
image, mounting the smoke script read-only as for the other smoke scripts. Snapshot manifests
|
image, mounting the smoke script read-only as for the other smoke scripts. Snapshot manifests
|
||||||
record verified immutable source bytes; full scan records retain exact selected/deferred questions,
|
record verified immutable source bytes; full scan records retain exact selected/deferred questions,
|
||||||
raw answers, per-question gaps, change context and candidate dispositions.
|
raw answers, per-question gaps, change context and candidate dispositions.
|
||||||
|
|
||||||
|
Experimental replay can reuse compatible local preparation, graphs and query evidence:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
ousast pre-push /path/to/repo --base BASE --head HEAD \
|
||||||
|
--artifact /path/outside/repo/result.json \
|
||||||
|
--cache-dir /path/outside/repo/private-cache
|
||||||
|
```
|
||||||
|
|
||||||
|
Omit `--cache-dir` for a cold control. The cache directory must be owned by the current
|
||||||
|
user, private (0700), and outside the analyzed repository. Its default payload/manifest
|
||||||
|
budget is 2 GiB (`PushConfig.cache_max_bytes` for API callers). Cache permission to reuse
|
||||||
|
is separate from alert eligibility: current admission is always applied. No hook is installed
|
||||||
|
by replay. Artifact timings record discovery, graph and query hits; identical-tip hits do
|
||||||
|
not establish representative changed-code latency. Corrupt, partial or incompatible entries
|
||||||
|
are misses, with remaining work constrained by the same push deadline.
|
||||||
|
|||||||
@@ -115,6 +115,9 @@ def main(argv: list[str] | None = None) -> int:
|
|||||||
replay_parser.add_argument("--base", required=True)
|
replay_parser.add_argument("--base", required=True)
|
||||||
replay_parser.add_argument("--head", required=True)
|
replay_parser.add_argument("--head", required=True)
|
||||||
replay_parser.add_argument("--artifact", type=Path, required=True)
|
replay_parser.add_argument("--artifact", type=Path, required=True)
|
||||||
|
replay_parser.add_argument(
|
||||||
|
"--cache-dir", type=Path, help="reuse compatible local artifacts in a private directory outside the repository"
|
||||||
|
)
|
||||||
replay_parser.add_argument("--deadline", type=float, default=30.0)
|
replay_parser.add_argument("--deadline", type=float, default=30.0)
|
||||||
replay_parser.add_argument("--cancellation-allowance", type=float, default=2.0)
|
replay_parser.add_argument("--cancellation-allowance", type=float, default=2.0)
|
||||||
replay_parser.add_argument("--max-regions", type=int, default=500)
|
replay_parser.add_argument("--max-regions", type=int, default=500)
|
||||||
@@ -212,7 +215,13 @@ def main(argv: list[str] | None = None) -> int:
|
|||||||
incomplete_coverage_policy=args.incomplete_coverage,
|
incomplete_coverage_policy=args.incomplete_coverage,
|
||||||
)
|
)
|
||||||
delivery = replay(
|
delivery = replay(
|
||||||
args.path, base=args.base, head=args.head, artifact=args.artifact, config=settings, max_regions=args.max_regions
|
args.path,
|
||||||
|
base=args.base,
|
||||||
|
head=args.head,
|
||||||
|
artifact=args.artifact,
|
||||||
|
config=settings,
|
||||||
|
max_regions=args.max_regions,
|
||||||
|
cache_dir=args.cache_dir,
|
||||||
)
|
)
|
||||||
except ValueError as error:
|
except ValueError as error:
|
||||||
parser.error(str(error))
|
parser.error(str(error))
|
||||||
|
|||||||
@@ -266,6 +266,41 @@ class NullBackend:
|
|||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def engine_runtime_identity(
|
||||||
|
*, include_tests: bool = True, language: str | None = None, execution_budget: ExecutionBudget | None = None
|
||||||
|
) -> str:
|
||||||
|
"""Installed frontend/adaptation bytes, also used by comparison eligibility provenance."""
|
||||||
|
from .artifact import digest_value, graph_bytes
|
||||||
|
|
||||||
|
budget = execution_budget
|
||||||
|
deadline = budget.deadline_monotonic if budget else float("inf")
|
||||||
|
launcher = shutil.which("joern")
|
||||||
|
version = joern_version()
|
||||||
|
installation = []
|
||||||
|
if launcher is not None and version is not None:
|
||||||
|
home = Path(launcher).resolve().parent
|
||||||
|
engine = ".".join(map(str, version))
|
||||||
|
frontends = {_FRONTENDS.get(language, "joern-parse")} if language else set(_FRONTENDS.values())
|
||||||
|
candidates = {home / f"lib/io.joern.joern-cli-{engine}.jar", Path(launcher).resolve()}
|
||||||
|
for frontend in frontends:
|
||||||
|
candidates.add(home / f"frontends/{frontend}/lib/io.joern.{frontend}-{engine}.jar")
|
||||||
|
frontend_launcher = shutil.which(frontend)
|
||||||
|
if frontend_launcher is not None:
|
||||||
|
candidates.add(Path(frontend_launcher).resolve())
|
||||||
|
if "jssrc2cpg" in frontends:
|
||||||
|
candidates.update({home / "ousast-frontend-retention-v1/manifest.json", home / "frontends/jssrc2cpg/bin/astgen/astgen-linux"})
|
||||||
|
for candidate in sorted(candidates):
|
||||||
|
installation.append((candidate.name, graph_bytes(candidate, deadline) if candidate.is_file() else "absent"))
|
||||||
|
return digest_value(
|
||||||
|
{
|
||||||
|
"version": version,
|
||||||
|
"files": installation,
|
||||||
|
"include_tests": include_tests,
|
||||||
|
"java_options": os.environ.get("JAVA_TOOL_OPTIONS", ""),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class JoernBackend:
|
class JoernBackend:
|
||||||
"""``joern-parse`` then ``joern --script``, both out of process and both timeout-bounded."""
|
"""``joern-parse`` then ``joern --script``, both out of process and both timeout-bounded."""
|
||||||
@@ -324,7 +359,8 @@ class JoernBackend:
|
|||||||
) -> GraphIdentity:
|
) -> GraphIdentity:
|
||||||
from .artifact import digest_value, read_bytes, source_inventory
|
from .artifact import digest_value, read_bytes, source_inventory
|
||||||
|
|
||||||
deadline = execution_budget.deadline_monotonic if execution_budget else float("inf")
|
budget = execution_budget or self.execution_budget
|
||||||
|
deadline = budget.deadline_monotonic if budget else float("inf")
|
||||||
version = joern_version()
|
version = joern_version()
|
||||||
if version is None:
|
if version is None:
|
||||||
raise ValueError("engine_identity_unknown")
|
raise ValueError("engine_identity_unknown")
|
||||||
@@ -342,7 +378,16 @@ class JoernBackend:
|
|||||||
engine,
|
engine,
|
||||||
(DATAFLOW_OVERLAY,),
|
(DATAFLOW_OVERLAY,),
|
||||||
(("implementation", code),),
|
(("implementation", code),),
|
||||||
(("include_tests", str(self.include_tests)), ("java_options", os.environ.get("JAVA_TOOL_OPTIONS", "default") or "default")),
|
(
|
||||||
|
("include_tests", str(self.include_tests)),
|
||||||
|
(
|
||||||
|
"installation",
|
||||||
|
engine_runtime_identity(
|
||||||
|
include_tests=self.include_tests, language=language, execution_budget=execution_budget or self.execution_budget
|
||||||
|
),
|
||||||
|
),
|
||||||
|
("java_options", os.environ.get("JAVA_TOOL_OPTIONS", "default") or "default"),
|
||||||
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
def describe_graph(
|
def describe_graph(
|
||||||
|
|||||||
@@ -0,0 +1,202 @@
|
|||||||
|
"""Compatible replay reuse; the existing ranker and detector remain decision owners."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import math
|
||||||
|
import os
|
||||||
|
import time
|
||||||
|
from collections.abc import Callable, Mapping, Sequence
|
||||||
|
from dataclasses import asdict, replace
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from openultrasast.cpg.artifact import GraphArtifact, digest_value
|
||||||
|
from openultrasast.cpg.backend import CpgResult, JoernBackend
|
||||||
|
from openultrasast.model.contracts import ExecutionBudget
|
||||||
|
from openultrasast.model.regions import ScanRegion
|
||||||
|
from openultrasast.push.cache import ArtifactCache, SemanticKeys
|
||||||
|
from openultrasast.push.contracts import SnapshotManifest
|
||||||
|
|
||||||
|
|
||||||
|
class ReusingBackend:
|
||||||
|
def __init__(self, backend: JoernBackend, cache: ArtifactCache, semantics: SemanticKeys, *, declarations: str, exclusions: str) -> None:
|
||||||
|
self.backend = backend
|
||||||
|
self.cache = cache
|
||||||
|
self.semantics = semantics
|
||||||
|
self.declarations = declarations
|
||||||
|
self.exclusions = exclusions
|
||||||
|
self.last_failure = ""
|
||||||
|
self.graph_hits = 0
|
||||||
|
self.query_hits = 0
|
||||||
|
|
||||||
|
def build(self, root: Path, *, language: str, exclude: Sequence[str], execution_budget: ExecutionBudget) -> CpgResult | None:
|
||||||
|
identity = None
|
||||||
|
artifact = None
|
||||||
|
result = None
|
||||||
|
try:
|
||||||
|
identity = self.backend.graph_identity(
|
||||||
|
root,
|
||||||
|
language=language,
|
||||||
|
declarations_digest=self.declarations,
|
||||||
|
exclusions_digest=digest_value([self.exclusions, sorted(exclude)]),
|
||||||
|
execution_budget=execution_budget,
|
||||||
|
)
|
||||||
|
key = digest_value({"layer": "graph-v1", "identity": identity.to_payload()})
|
||||||
|
with self.cache.lookup(key, execution_budget) as entry:
|
||||||
|
if entry is not None:
|
||||||
|
artifact = GraphArtifact.from_payload(entry.metadata)
|
||||||
|
result = self.backend.load_graph(entry.path, artifact, identity, execution_budget=execution_budget)
|
||||||
|
if result is not None:
|
||||||
|
self.graph_hits += 1
|
||||||
|
except (OSError, ValueError, TimeoutError):
|
||||||
|
artifact = None
|
||||||
|
if result is None:
|
||||||
|
artifact = None
|
||||||
|
result = self.backend.build(root, language=language, exclude=exclude, execution_budget=execution_budget)
|
||||||
|
self.last_failure = self.backend.last_failure
|
||||||
|
if result is None:
|
||||||
|
return None
|
||||||
|
if identity is not None and time.monotonic() < execution_budget.deadline_monotonic:
|
||||||
|
artifact = self.backend.describe_graph(result, root, identity, execution_budget=execution_budget)
|
||||||
|
if artifact is not None:
|
||||||
|
self.cache.publish(key, result.cpg_path, artifact.to_payload(), execution_budget)
|
||||||
|
if artifact is None:
|
||||||
|
return result
|
||||||
|
return self._answers(result, artifact, execution_budget)
|
||||||
|
|
||||||
|
def _answers(self, result: CpgResult, artifact: GraphArtifact, budget: ExecutionBudget) -> CpgResult:
|
||||||
|
def normalize(value: Any) -> Any:
|
||||||
|
if isinstance(value, str):
|
||||||
|
return value.replace(artifact.source_root + "/", "")
|
||||||
|
if isinstance(value, list):
|
||||||
|
return [normalize(item) for item in value]
|
||||||
|
if isinstance(value, dict):
|
||||||
|
return {key: normalize(item) for key, item in value.items()}
|
||||||
|
return value
|
||||||
|
|
||||||
|
def batch(kind: str, requests: Mapping[str, Mapping[str, object]]) -> dict[str, list[object]] | None:
|
||||||
|
answers: dict[str, list[object]] = {}
|
||||||
|
missing = {}
|
||||||
|
keys = {}
|
||||||
|
for rid, request in requests.items():
|
||||||
|
if time.monotonic() >= budget.deadline_monotonic:
|
||||||
|
break
|
||||||
|
key = self.semantics.query(
|
||||||
|
graph=digest_value({"identity": artifact.identity.to_payload(), "bytes": artifact.graph_digest}),
|
||||||
|
kind=kind,
|
||||||
|
request=request,
|
||||||
|
context={"paths": artifact.source_paths, "declarations": artifact.identity.declarations_digest},
|
||||||
|
)
|
||||||
|
keys[rid] = key
|
||||||
|
cached = self.cache.get_json(key, kind="query", budget=budget)
|
||||||
|
if isinstance(cached, dict) and isinstance(cached.get("rows"), list) and self._census(cached.get("census"), artifact):
|
||||||
|
answers[rid] = cached["rows"]
|
||||||
|
answers["__census__"] = cached["census"]
|
||||||
|
self.query_hits += 1
|
||||||
|
else:
|
||||||
|
missing[rid] = request
|
||||||
|
if missing and time.monotonic() < budget.deadline_monotonic:
|
||||||
|
raw: Any = result.run_batch(kind, missing) if result.run_batch else None
|
||||||
|
if raw is None and result.run_batch is None:
|
||||||
|
raw = {rid: result.run(kind, params) for rid, params in missing.items()}
|
||||||
|
if isinstance(raw, dict):
|
||||||
|
normalized = normalize(raw)
|
||||||
|
census = normalized.get("__census__")
|
||||||
|
valid = self._census(census, artifact)
|
||||||
|
if not valid:
|
||||||
|
return answers or None
|
||||||
|
answers["__census__"] = census
|
||||||
|
for rid in missing:
|
||||||
|
rows = normalized.get(rid)
|
||||||
|
if isinstance(rows, list):
|
||||||
|
answers[rid] = rows
|
||||||
|
self.cache.put_json(keys[rid], {"rows": rows, "census": census}, kind="query", budget=budget, complete=valid)
|
||||||
|
return answers or None
|
||||||
|
|
||||||
|
def run(kind: str, params: Mapping[str, object]) -> object | None:
|
||||||
|
if kind == "census":
|
||||||
|
value: object = normalize(result.run(kind, params))
|
||||||
|
return value
|
||||||
|
rows = batch(kind, {"one": params})
|
||||||
|
return rows.get("one") if rows is not None else None
|
||||||
|
|
||||||
|
return replace(result, run=run, run_batch=batch)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _census(value: object, artifact: GraphArtifact) -> bool:
|
||||||
|
if not isinstance(value, list) or len(value) != 1 or not isinstance(value[0], dict):
|
||||||
|
return False
|
||||||
|
row = value[0]
|
||||||
|
names = row.get("file_names")
|
||||||
|
try:
|
||||||
|
return (
|
||||||
|
isinstance(names, list)
|
||||||
|
and all(isinstance(name, str) for name in names)
|
||||||
|
and set(artifact.source_paths) <= set(names)
|
||||||
|
and int(str(row.get("methods", 0))) > 0
|
||||||
|
and int(str(row.get("files", 0))) >= len(artifact.source_paths)
|
||||||
|
and int(str(row.get("shards", 1))) == 1
|
||||||
|
)
|
||||||
|
except (ValueError, TypeError):
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def discovery(
|
||||||
|
root: Path,
|
||||||
|
manifest: SnapshotManifest,
|
||||||
|
*,
|
||||||
|
cache: ArtifactCache | None,
|
||||||
|
identity: str,
|
||||||
|
budget: ExecutionBudget,
|
||||||
|
discover: Callable[[Path], tuple[tuple[ScanRegion, ...], tuple[bytes, ...]]],
|
||||||
|
) -> tuple[tuple[ScanRegion, ...], tuple[bytes, ...], bool]:
|
||||||
|
"""Persist pure discovery only after the adapter proves the complete snapshot readable."""
|
||||||
|
key = digest_value(
|
||||||
|
{
|
||||||
|
"layer": "discovery-v1",
|
||||||
|
"files": [item.to_payload() for item in manifest.files],
|
||||||
|
"object_format": manifest.object_format,
|
||||||
|
"identity": identity,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
raw = cache.get_json(key, kind="discovery", budget=budget) if cache and manifest.complete else None
|
||||||
|
if isinstance(raw, dict):
|
||||||
|
try:
|
||||||
|
regions = []
|
||||||
|
for item in raw["regions"]:
|
||||||
|
values = dict(item)
|
||||||
|
values["families"] = tuple(values["families"])
|
||||||
|
region = ScanRegion(**values)
|
||||||
|
if (
|
||||||
|
not all(isinstance(v, str) for v in (region.path, region.language, region.source, *region.families))
|
||||||
|
or region.function is not None
|
||||||
|
and not isinstance(region.function, str)
|
||||||
|
or type(region.shipped) is not bool
|
||||||
|
or type(region.rank) not in (int, float)
|
||||||
|
or not math.isfinite(region.rank)
|
||||||
|
):
|
||||||
|
raise ValueError("invalid_discovery_region")
|
||||||
|
if os.fsencode(region.path) not in {item.path_bytes for item in manifest.files}:
|
||||||
|
raise ValueError("discovery_path_outside_snapshot")
|
||||||
|
regions.append(region)
|
||||||
|
declarations = tuple(bytes.fromhex(item) for item in raw["declarations"])
|
||||||
|
if not set(declarations) <= {item.path_bytes for item in manifest.files}:
|
||||||
|
raise ValueError("invalid_declarations")
|
||||||
|
return tuple(regions), declarations, True
|
||||||
|
except (KeyError, ValueError, TypeError):
|
||||||
|
pass
|
||||||
|
found, declarations = discover(root)
|
||||||
|
if cache and manifest.complete:
|
||||||
|
cache.put_json(
|
||||||
|
key,
|
||||||
|
{"regions": [asdict(region) for region in found], "declarations": [item.hex() for item in declarations]},
|
||||||
|
kind="discovery",
|
||||||
|
budget=budget,
|
||||||
|
complete=True,
|
||||||
|
)
|
||||||
|
return found, declarations, False
|
||||||
|
|
||||||
|
|
||||||
|
def declaration_identity(manifest: SnapshotManifest, paths: tuple[bytes, ...]) -> str:
|
||||||
|
declared = set(paths)
|
||||||
|
return digest_value([item.to_payload() for item in manifest.files if item.path_bytes in declared])
|
||||||
@@ -9,18 +9,22 @@ import pickle
|
|||||||
import selectors
|
import selectors
|
||||||
import time
|
import time
|
||||||
from collections.abc import Callable
|
from collections.abc import Callable
|
||||||
|
from contextlib import nullcontext
|
||||||
from dataclasses import asdict, replace
|
from dataclasses import asdict, replace
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any, Literal, TypeVar, cast
|
from typing import Any, Literal, TypeVar, cast
|
||||||
|
|
||||||
from openultrasast.config import PushConfig
|
from openultrasast.config import PushConfig
|
||||||
from openultrasast.cpg.backend import JoernBackend, joern_version
|
from openultrasast.cpg.artifact import digest_value
|
||||||
|
from openultrasast.cpg.backend import JoernBackend, engine_runtime_identity, joern_version
|
||||||
from openultrasast.mapping import analyze_entry_points
|
from openultrasast.mapping import analyze_entry_points
|
||||||
from openultrasast.model.contracts import ExecutionBudget
|
from openultrasast.model.contracts import ExecutionBudget
|
||||||
|
from openultrasast.model.layout import layout_facts
|
||||||
from openultrasast.model.regions import ScanRegion, regions_for
|
from openultrasast.model.regions import ScanRegion, regions_for
|
||||||
from openultrasast.model.scan import ScanBudget, scan_repository
|
from openultrasast.model.scan import ScanBudget, scan_repository
|
||||||
from openultrasast.model.shipped import declared_sources
|
from openultrasast.model.shipped import declared_sources
|
||||||
from openultrasast.preprocess import build_file_target, enumerate_source_files
|
from openultrasast.preprocess import build_file_target, enumerate_source_files
|
||||||
|
from openultrasast.push.cache import ArtifactCache, SemanticKeys
|
||||||
from openultrasast.push.contracts import ComparisonAnalysis, SnapshotManifest
|
from openultrasast.push.contracts import ComparisonAnalysis, SnapshotManifest
|
||||||
from openultrasast.push.policy import (
|
from openultrasast.push.policy import (
|
||||||
AdmissionCandidate,
|
AdmissionCandidate,
|
||||||
@@ -32,6 +36,7 @@ from openultrasast.push.policy import (
|
|||||||
semantics_digest,
|
semantics_digest,
|
||||||
)
|
)
|
||||||
from openultrasast.push.report import PushReport, ReportDelivery, ScanRecord, _completed_scan, deliver_report, render_report
|
from openultrasast.push.report import PushReport, ReportDelivery, ScanRecord, _completed_scan, deliver_report, render_report
|
||||||
|
from openultrasast.push.reuse import ReusingBackend, declaration_identity, discovery
|
||||||
from openultrasast.push.snapshot import SnapshotAdapter
|
from openultrasast.push.snapshot import SnapshotAdapter
|
||||||
|
|
||||||
_T = TypeVar("_T")
|
_T = TypeVar("_T")
|
||||||
@@ -124,14 +129,25 @@ def _provenance(repository: Path, scan_budget: ScanBudget) -> dict[str, str]:
|
|||||||
|
|
||||||
version = joern_version()
|
version = joern_version()
|
||||||
engine = "joern-" + (".".join(map(str, version)) if version else "unavailable_or_unidentified")
|
engine = "joern-" + (".".join(map(str, version)) if version else "unavailable_or_unidentified")
|
||||||
|
runtime = engine_runtime_identity()
|
||||||
|
integration = digest(list((root / "cpg").glob("*.py")))
|
||||||
return {
|
return {
|
||||||
"repository": str(repository.absolute()),
|
"repository": str(repository.absolute()),
|
||||||
|
"engine_runtime": runtime,
|
||||||
"engine": engine,
|
"engine": engine,
|
||||||
"facts": digest(list((root / "ruleset").rglob("*.toml"))),
|
"facts": digest(list((root / "ruleset").rglob("*.toml"))),
|
||||||
|
"graph_layout": digest_value(
|
||||||
|
{
|
||||||
|
"facts": [asdict(row) for row in layout_facts()],
|
||||||
|
"implementation": digest([root / "model/layout.py", root / "model/partitions.py"]),
|
||||||
|
}
|
||||||
|
),
|
||||||
"queries": digest(list((root / "cpg/queries").glob("*.sc"))),
|
"queries": digest(list((root / "cpg/queries").glob("*.sc"))),
|
||||||
"policy": digest(list((root / "push").glob("*.py"))),
|
"policy": digest(list((root / "push").glob("*.py"))),
|
||||||
"core": digest(list((root / "model").glob("*.py")) + list((root / "cpg").glob("*.py"))),
|
"core": digest(list((root / "model").glob("*.py")) + list((root / "cpg").glob("*.py"))),
|
||||||
"semantics": semantics_digest(engine_identity=engine, options=asdict(scan_budget)),
|
"semantics": semantics_digest(
|
||||||
|
engine_identity=engine, options={**asdict(scan_budget), "engine_runtime": runtime, "engine_integration": integration}
|
||||||
|
),
|
||||||
"ranking_mode": "evidence",
|
"ranking_mode": "evidence",
|
||||||
"model": "disabled",
|
"model": "disabled",
|
||||||
"capabilities": "empty_registry_pending_task_8.3",
|
"capabilities": "empty_registry_pending_task_8.3",
|
||||||
@@ -139,6 +155,13 @@ def _provenance(repository: Path, scan_budget: ScanBudget) -> dict[str, str]:
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _discovery_identity() -> str:
|
||||||
|
root = Path(__file__).resolve().parents[1]
|
||||||
|
# Discovery invokes the shipped mapper, preprocessing, layout and family specs.
|
||||||
|
# Hash code bytes, never a mutable import name or a repository-provided version.
|
||||||
|
return digest_value([(p.relative_to(root).as_posix(), hashlib.sha256(p.read_bytes()).hexdigest()) for p in sorted(root.rglob("*.py"))])
|
||||||
|
|
||||||
|
|
||||||
def replay(
|
def replay(
|
||||||
repository: Path,
|
repository: Path,
|
||||||
*,
|
*,
|
||||||
@@ -148,16 +171,22 @@ def replay(
|
|||||||
config: PushConfig | None = None,
|
config: PushConfig | None = None,
|
||||||
backend: Any = None,
|
backend: Any = None,
|
||||||
max_regions: int = 500,
|
max_regions: int = 500,
|
||||||
|
cache_dir: Path | None = None,
|
||||||
) -> ReportDelivery:
|
) -> ReportDelivery:
|
||||||
"""Run a single explicit comparison; exit policy never stands for coverage."""
|
"""Run a single explicit comparison; exit policy never stands for coverage."""
|
||||||
config = config or PushConfig()
|
config = config or PushConfig()
|
||||||
if max_regions < 1:
|
if max_regions < 1:
|
||||||
raise ValueError("max_regions must be positive")
|
raise ValueError("max_regions must be positive")
|
||||||
|
if cache_dir is not None and cache_dir.resolve().is_relative_to(repository.resolve()):
|
||||||
|
raise ValueError("cache directory must be outside the analyzed repository")
|
||||||
started = time.monotonic()
|
started = time.monotonic()
|
||||||
budget = ExecutionBudget(started + config.deadline_seconds, config.cancellation_allowance_seconds)
|
budget = ExecutionBudget(started + config.deadline_seconds, config.cancellation_allowance_seconds)
|
||||||
scan_budget = ScanBudget(max_model_calls=0, max_regions=max_regions, order_by_evidence=True)
|
scan_budget = ScanBudget(max_model_calls=0, max_regions=max_regions, order_by_evidence=True)
|
||||||
if backend is None:
|
if backend is None:
|
||||||
backend = JoernBackend()
|
backend = JoernBackend()
|
||||||
|
cache = None
|
||||||
|
cache_semantics = None
|
||||||
|
reused: list[ReusingBackend] = []
|
||||||
records: list[ScanRecord] = []
|
records: list[ScanRecord] = []
|
||||||
manifests: list[SnapshotManifest] = []
|
manifests: list[SnapshotManifest] = []
|
||||||
reasons: list[str] = []
|
reasons: list[str] = []
|
||||||
@@ -179,16 +208,66 @@ def replay(
|
|||||||
stage, stage_started = "provenance", time.monotonic()
|
stage, stage_started = "provenance", time.monotonic()
|
||||||
provenance.update(_prepare(lambda: _provenance(repository, scan_budget), budget))
|
provenance.update(_prepare(lambda: _provenance(repository, scan_budget), budget))
|
||||||
timings[stage + "_seconds"] = time.monotonic() - stage_started
|
timings[stage + "_seconds"] = time.monotonic() - stage_started
|
||||||
|
if cache_dir is not None:
|
||||||
|
cache = ArtifactCache(cache_dir, max_bytes=config.cache_max_bytes)
|
||||||
|
cache_semantics = SemanticKeys(
|
||||||
|
provenance["facts"],
|
||||||
|
provenance["queries"],
|
||||||
|
digest_value({"query_parameters": "complete-joern-request-v1"}),
|
||||||
|
digest_value({"ranking_mode": "evidence", "options": asdict(scan_budget)}),
|
||||||
|
provenance["policy"],
|
||||||
|
provenance["model"],
|
||||||
|
provenance["capabilities"],
|
||||||
|
config.mode,
|
||||||
|
)
|
||||||
|
provenance["cache"] = "validated_local_reuse"
|
||||||
|
discovery_identity = digest_value({"code": _prepare(lambda: _discovery_identity(), budget), "facts": provenance["facts"]})
|
||||||
stage, stage_started = "preparation", time.monotonic()
|
stage, stage_started = "preparation", time.monotonic()
|
||||||
with adapter.materialize(comparison.head_oid, budget=budget) as tip:
|
with adapter.materialize(comparison.head_oid, budget=budget) as tip:
|
||||||
manifests.append(tip.manifest)
|
manifests.append(tip.manifest)
|
||||||
assert comparison.base_oid is not None
|
assert comparison.base_oid is not None
|
||||||
with adapter.materialize(comparison.base_oid, budget=budget) as old:
|
old_context = (
|
||||||
|
nullcontext(tip) if comparison.base_oid == comparison.head_oid else adapter.materialize(comparison.base_oid, budget=budget)
|
||||||
|
)
|
||||||
|
with old_context as old:
|
||||||
manifests.append(old.manifest)
|
manifests.append(old.manifest)
|
||||||
for manifest in manifests:
|
for manifest in manifests:
|
||||||
reasons.extend("snapshot:" + boundary.reason for boundary in manifest.boundaries)
|
reasons.extend("snapshot:" + boundary.reason for boundary in manifest.boundaries)
|
||||||
head_regions, head_declarations = _prepare(lambda: _discover(tip.root), budget)
|
timings["snapshot_seconds"] = time.monotonic() - stage_started
|
||||||
base_regions, base_declarations = _prepare(lambda: _discover(old.root), budget)
|
timings["snapshot_reuses"] = int(comparison.base_oid == comparison.head_oid)
|
||||||
|
discovery_started = time.monotonic()
|
||||||
|
head_regions, head_declarations, head_hit = _prepare(
|
||||||
|
lambda: discovery(tip.root, tip.manifest, cache=cache, identity=discovery_identity, budget=budget, discover=_discover),
|
||||||
|
budget,
|
||||||
|
)
|
||||||
|
base_regions, base_declarations, base_hit = _prepare(
|
||||||
|
lambda: discovery(old.root, old.manifest, cache=cache, identity=discovery_identity, budget=budget, discover=_discover),
|
||||||
|
budget,
|
||||||
|
)
|
||||||
|
timings["discovery_seconds"] = time.monotonic() - discovery_started
|
||||||
|
timings["discovery_hits"] = int(head_hit) + int(base_hit)
|
||||||
|
head_backend = base_backend = backend
|
||||||
|
if (
|
||||||
|
cache is not None
|
||||||
|
and cache_semantics is not None
|
||||||
|
and isinstance(backend, JoernBackend)
|
||||||
|
and all(m.complete for m in manifests)
|
||||||
|
):
|
||||||
|
head_backend = ReusingBackend(
|
||||||
|
backend,
|
||||||
|
cache,
|
||||||
|
cache_semantics,
|
||||||
|
declarations=declaration_identity(tip.manifest, head_declarations),
|
||||||
|
exclusions=provenance["graph_layout"],
|
||||||
|
)
|
||||||
|
base_backend = ReusingBackend(
|
||||||
|
backend,
|
||||||
|
cache,
|
||||||
|
cache_semantics,
|
||||||
|
declarations=declaration_identity(old.manifest, base_declarations),
|
||||||
|
exclusions=provenance["graph_layout"],
|
||||||
|
)
|
||||||
|
reused.extend((head_backend, base_backend))
|
||||||
context = adapter.compare(comparison, declaration_paths=tuple(set(head_declarations + base_declarations)), budget=budget)
|
context = adapter.compare(comparison, declaration_paths=tuple(set(head_declarations + base_declarations)), budget=budget)
|
||||||
context = replace(context, unresolved_boundaries=tuple(dict.fromkeys((*context.unresolved_boundaries, *reasons))))
|
context = replace(context, unresolved_boundaries=tuple(dict.fromkeys((*context.unresolved_boundaries, *reasons))))
|
||||||
reasons.extend(context.unresolved_boundaries)
|
reasons.extend(context.unresolved_boundaries)
|
||||||
@@ -197,7 +276,7 @@ def replay(
|
|||||||
head_scan = scan_repository(
|
head_scan = scan_repository(
|
||||||
tip.root,
|
tip.root,
|
||||||
head_regions,
|
head_regions,
|
||||||
backend=backend,
|
backend=head_backend,
|
||||||
budget=scan_budget,
|
budget=scan_budget,
|
||||||
execution_budget=budget,
|
execution_budget=budget,
|
||||||
ranking_mode="evidence",
|
ranking_mode="evidence",
|
||||||
@@ -214,11 +293,13 @@ def replay(
|
|||||||
head_regions=head_regions,
|
head_regions=head_regions,
|
||||||
base_regions=base_regions,
|
base_regions=base_regions,
|
||||||
context=context,
|
context=context,
|
||||||
backend=backend,
|
backend=base_backend,
|
||||||
execution_budget=budget,
|
execution_budget=budget,
|
||||||
scan_budget=scan_budget,
|
scan_budget=scan_budget,
|
||||||
head_semantics=provenance["semantics"],
|
head_semantics=provenance["semantics"],
|
||||||
base_semantics=provenance["semantics"],
|
base_semantics=provenance["semantics"],
|
||||||
|
cache=cache,
|
||||||
|
cache_semantics=cache_semantics,
|
||||||
)
|
)
|
||||||
if delta.base_scan is not None:
|
if delta.base_scan is not None:
|
||||||
records.append(ScanRecord(comparison, "base", delta.base_scan))
|
records.append(ScanRecord(comparison, "base", delta.base_scan))
|
||||||
@@ -251,6 +332,8 @@ def replay(
|
|||||||
timings[stage + "_seconds"] = time.monotonic() - stage_started
|
timings[stage + "_seconds"] = time.monotonic() - stage_started
|
||||||
if time.monotonic() >= budget.deadline_monotonic:
|
if time.monotonic() >= budget.deadline_monotonic:
|
||||||
reasons.append("deadline_exhausted")
|
reasons.append("deadline_exhausted")
|
||||||
|
timings["graph_hits"] = sum(item.graph_hits for item in reused)
|
||||||
|
timings["query_hits"] = sum(item.query_hits for item in reused)
|
||||||
head_records = [r.scan for r in records if r.side == "head"]
|
head_records = [r.scan for r in records if r.side == "head"]
|
||||||
for record in records:
|
for record in records:
|
||||||
if record.scan.scope is not None:
|
if record.scan.scope is not None:
|
||||||
|
|||||||
@@ -88,3 +88,21 @@ def test_generic_driver_accepts_backend_validated_lease(tmp_path, monkeypatch):
|
|||||||
assert result.run("census", {})["file_names"] == ["a.js"]
|
assert result.run("census", {})["file_names"] == ["a.js"]
|
||||||
_dispose(result)
|
_dispose(result)
|
||||||
assert graph.read_bytes() == b"graph original"
|
assert graph.read_bytes() == b"graph original"
|
||||||
|
|
||||||
|
|
||||||
|
def test_installed_frontend_adaptation_changes_graph_identity(tmp_path, monkeypatch):
|
||||||
|
import openultrasast.cpg.backend as module
|
||||||
|
|
||||||
|
backend, root, _, identity, _ = prepared(tmp_path, monkeypatch)
|
||||||
|
home = tmp_path / "engine"
|
||||||
|
home.mkdir()
|
||||||
|
launcher = home / "joern"
|
||||||
|
launcher.write_text("launcher")
|
||||||
|
jar = home / "frontends/jssrc2cpg/lib/io.joern.jssrc2cpg-4.0.625.jar"
|
||||||
|
jar.parent.mkdir(parents=True)
|
||||||
|
jar.write_bytes(b"frontend-one")
|
||||||
|
monkeypatch.setattr(module.shutil, "which", lambda name: str(launcher) if name == "joern" else None)
|
||||||
|
first = backend.graph_identity(root, language="javascript", declarations_digest="d", exclusions_digest="e")
|
||||||
|
jar.write_bytes(b"frontend-two")
|
||||||
|
second = backend.graph_identity(root, language="javascript", declarations_digest="d", exclusions_digest="e")
|
||||||
|
assert first != second
|
||||||
|
|||||||
@@ -0,0 +1,89 @@
|
|||||||
|
"""Live integration of validated graph and query reuse under one deadline."""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import time
|
||||||
|
|
||||||
|
from openultrasast.cpg.backend import CpgResult, JoernBackend
|
||||||
|
from openultrasast.model.contracts import ExecutionBudget
|
||||||
|
from openultrasast.push.cache import ArtifactCache, SemanticKeys
|
||||||
|
from openultrasast.push.reuse import ReusingBackend
|
||||||
|
|
||||||
|
|
||||||
|
def test_backend_reuses_graph_and_answers_then_invalidates_source_and_dependencies(tmp_path, monkeypatch):
|
||||||
|
import openultrasast.cpg.backend as module
|
||||||
|
|
||||||
|
monkeypatch.setattr(module, "joern_version", lambda: (4, 0, 625))
|
||||||
|
builds = []
|
||||||
|
queries = []
|
||||||
|
|
||||||
|
class Engine(JoernBackend):
|
||||||
|
def build(self, root, *, language, exclude, execution_budget):
|
||||||
|
builds.append(root)
|
||||||
|
path = tmp_path / f"graph-{len(builds)}.json"
|
||||||
|
paths = [str(p) for p in root.glob("*.js")]
|
||||||
|
path.write_text(json.dumps(paths))
|
||||||
|
return CpgResult(path, lambda kind, params: self.query(path, kind, params), lambda kind, req: self.query_batch(path, kind, req))
|
||||||
|
|
||||||
|
def query(self, path, kind, params):
|
||||||
|
names = json.loads(path.read_text())
|
||||||
|
return {"files": len(names), "methods": 2, "calls": 1, "file_names": names, "overlays": "dataflowOss"}
|
||||||
|
|
||||||
|
def query_batch(self, path, kind, requests):
|
||||||
|
queries.append((kind, requests))
|
||||||
|
return {**{rid: [] for rid in requests}, "__census__": [self.query(path, "census", {})]}
|
||||||
|
|
||||||
|
root = tmp_path / "source"
|
||||||
|
root.mkdir()
|
||||||
|
(root / "a.js").write_text("function f() {}")
|
||||||
|
cache = ArtifactCache(tmp_path / "cache", max_bytes=100000)
|
||||||
|
semantics = SemanticKeys("facts", "queries", "config", "ranking", "admission", "disabled", "empty", "advisory")
|
||||||
|
deadline = ExecutionBudget(time.monotonic() + 10, 0.2)
|
||||||
|
backend = ReusingBackend(Engine(), cache, semantics, declarations="dependencies", exclusions="exclusions")
|
||||||
|
|
||||||
|
def run():
|
||||||
|
result = backend.build(root, language="javascript", exclude=(), execution_budget=deadline)
|
||||||
|
assert result is not None
|
||||||
|
answer = result.run_batch("taint", {"q": {"file": "a.js", "callDepth": 3}})
|
||||||
|
if result.cleanup:
|
||||||
|
result.cleanup()
|
||||||
|
return answer
|
||||||
|
|
||||||
|
expected = run()
|
||||||
|
assert run() == expected
|
||||||
|
assert len(builds) == 1 and len(queries) == 1
|
||||||
|
(root / "a.js").write_text("function g() {}")
|
||||||
|
run()
|
||||||
|
assert len(builds) == 2 and len(queries) == 2
|
||||||
|
backend.declarations = "changed-dependency"
|
||||||
|
run()
|
||||||
|
assert len(builds) == 3 and len(queries) == 3
|
||||||
|
backend.semantics = SemanticKeys("changed-facts", "queries", "config", "ranking", "admission", "disabled", "empty", "advisory")
|
||||||
|
run()
|
||||||
|
assert len(builds) == 3 and len(queries) == 4
|
||||||
|
|
||||||
|
|
||||||
|
def test_fresh_missing_census_cannot_borrow_a_cached_query_census(tmp_path):
|
||||||
|
from openultrasast.cpg.artifact import GraphArtifact, GraphCensus, GraphIdentity
|
||||||
|
|
||||||
|
identity = GraphIdentity(
|
||||||
|
"source", "declarations", "exclude", "repository", "javascript", "engine", "frontend", "version", ("dataflowOss",), ()
|
||||||
|
)
|
||||||
|
artifact = GraphArtifact(identity, "graph-bytes", 10, GraphCensus(1, 1, 1), "complete", (), "/old", ("a.js",))
|
||||||
|
cache = ArtifactCache(tmp_path / "cache", max_bytes=100000)
|
||||||
|
semantics = SemanticKeys("facts", "queries", "config", "ranking", "admission", "disabled", "empty", "advisory")
|
||||||
|
wrapper = ReusingBackend(JoernBackend(), cache, semantics, declarations="d", exclusions="e")
|
||||||
|
census = [{"files": 1, "methods": 1, "file_names": ["a.js"]}]
|
||||||
|
calls = []
|
||||||
|
|
||||||
|
def batch(kind, requests):
|
||||||
|
calls.append(requests)
|
||||||
|
result = {rid: [] for rid in requests}
|
||||||
|
if len(calls) == 1:
|
||||||
|
result["__census__"] = census
|
||||||
|
return result
|
||||||
|
|
||||||
|
graph = CpgResult(tmp_path / "graph", lambda *_: None, batch)
|
||||||
|
result = wrapper._answers(graph, artifact, ExecutionBudget(time.monotonic() + 5, 0.2))
|
||||||
|
assert "q" in result.run_batch("taint", {"q": {"file": "a.js"}})
|
||||||
|
answer = result.run_batch("taint", {"q": {"file": "a.js"}, "fresh": {"file": "a.js", "callDepth": 3}})
|
||||||
|
assert "q" in answer and "fresh" not in answer
|
||||||
@@ -218,3 +218,84 @@ def test_default_replay_does_not_launch_unbudgeted_availability_probe(tmp_path,
|
|||||||
monkeypatch.setattr(runner, "JoernBackend", Backend, raising=False)
|
monkeypatch.setattr(runner, "JoernBackend", Backend, raising=False)
|
||||||
result = replay(root, base=base, head=head, artifact=tmp_path / "probe.json")
|
result = replay(root, base=base, head=head, artifact=tmp_path / "probe.json")
|
||||||
assert builds and result.result.coverage_status == "incomplete"
|
assert builds and result.result.coverage_status == "incomplete"
|
||||||
|
|
||||||
|
|
||||||
|
def test_replay_persists_discovery_without_skipping_readable_snapshots(tmp_path, monkeypatch):
|
||||||
|
from openultrasast.push import runner
|
||||||
|
|
||||||
|
root, base, head, _ = history(tmp_path)
|
||||||
|
cache = tmp_path / "cache"
|
||||||
|
cold = tmp_path / "cold.json"
|
||||||
|
warm = tmp_path / "warm.json"
|
||||||
|
replay(root, base=base, head=head, artifact=cold, backend=NullBackend(), cache_dir=cache)
|
||||||
|
|
||||||
|
def should_reuse(*args):
|
||||||
|
raise AssertionError("discovery should be cached")
|
||||||
|
|
||||||
|
monkeypatch.setattr(runner, "_discover", should_reuse)
|
||||||
|
replay(root, base=base, head=head, artifact=warm, backend=NullBackend(), cache_dir=cache)
|
||||||
|
a, b = json.loads(cold.read_text()), json.loads(warm.read_text())
|
||||||
|
assert a["result"] == b["result"]
|
||||||
|
assert a["snapshots"] == b["snapshots"]
|
||||||
|
assert b["timings"]["discovery_hits"] == 2
|
||||||
|
assert b["snapshots"][0]["bytes_read"] > 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_cache_cannot_write_inside_repository(tmp_path):
|
||||||
|
root, base, head, git = history(tmp_path)
|
||||||
|
before = git("status", "--porcelain")
|
||||||
|
with pytest.raises(ValueError, match="cache.*outside"):
|
||||||
|
replay(root, base=base, head=head, artifact=tmp_path / "out.json", cache_dir=root / "cache")
|
||||||
|
assert git("status", "--porcelain") == before
|
||||||
|
|
||||||
|
|
||||||
|
def test_cached_discovery_invalidates_source_and_dependency_edits(tmp_path):
|
||||||
|
root, base, head, git = history(tmp_path)
|
||||||
|
cache = tmp_path / "cache"
|
||||||
|
replay(root, base=base, head=head, artifact=tmp_path / "initial.json", backend=NullBackend(), cache_dir=cache)
|
||||||
|
(root / "api.js").write_text("function replacement(req) { eval(req.body.other); }")
|
||||||
|
git("add", "api.js")
|
||||||
|
git("commit", "-m", "change source")
|
||||||
|
edited = git("rev-parse", "HEAD")
|
||||||
|
replay(root, base=base, head=edited, artifact=tmp_path / "edited.json", backend=NullBackend(), cache_dir=cache)
|
||||||
|
data = json.loads((tmp_path / "edited.json").read_text())
|
||||||
|
assert data["timings"]["discovery_hits"] == 1
|
||||||
|
(root / "package.json").write_text('{"dependencies":{"express":"5"}}')
|
||||||
|
git("add", "package.json")
|
||||||
|
git("commit", "-m", "change dependencies")
|
||||||
|
dependency = git("rev-parse", "HEAD")
|
||||||
|
replay(root, base=base, head=dependency, artifact=tmp_path / "dependency.json", backend=NullBackend(), cache_dir=cache)
|
||||||
|
data = json.loads((tmp_path / "dependency.json").read_text())
|
||||||
|
assert data["timings"]["discovery_hits"] == 1
|
||||||
|
assert data["change_context"]["head_revision"] == dependency
|
||||||
|
|
||||||
|
|
||||||
|
def test_identical_revision_materializes_once_and_reuses_discovery(tmp_path, monkeypatch):
|
||||||
|
from openultrasast.push.snapshot import SnapshotAdapter
|
||||||
|
|
||||||
|
root, base, _, _ = history(tmp_path)
|
||||||
|
original = SnapshotAdapter.materialize
|
||||||
|
calls = []
|
||||||
|
|
||||||
|
def counted(self, oid, **kwargs):
|
||||||
|
calls.append(oid)
|
||||||
|
return original(self, oid, **kwargs)
|
||||||
|
|
||||||
|
monkeypatch.setattr(SnapshotAdapter, "materialize", counted)
|
||||||
|
replay(root, base=base, head=base, artifact=tmp_path / "same.json", backend=NullBackend(), cache_dir=tmp_path / "cache")
|
||||||
|
data = json.loads((tmp_path / "same.json").read_text())
|
||||||
|
assert calls == [base]
|
||||||
|
assert data["timings"]["discovery_hits"] == 1
|
||||||
|
assert len(data["snapshots"]) == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_admission_provenance_invalidates_when_installed_runtime_changes(tmp_path, monkeypatch):
|
||||||
|
from openultrasast.model.scan import ScanBudget
|
||||||
|
from openultrasast.push import runner
|
||||||
|
|
||||||
|
monkeypatch.setattr(runner, "engine_runtime_identity", lambda: "first-installation")
|
||||||
|
first = runner._provenance(tmp_path, ScanBudget())
|
||||||
|
monkeypatch.setattr(runner, "engine_runtime_identity", lambda: "second-installation")
|
||||||
|
second = runner._provenance(tmp_path, ScanBudget())
|
||||||
|
assert first["semantics"] != second["semantics"]
|
||||||
|
assert first["facts"] == second["facts"] and first["graph_layout"] == second["graph_layout"]
|
||||||
|
|||||||
Reference in New Issue
Block a user