marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(pre-push-safety-net): reuse compatible artifacts throughout replay

Browse Source
This commit is contained in:
Marius Ciepluch
2026-09-13 17:08:49 +02:00
parent 0ab5d56618
commit 530b9c9879
21 changed files with 2522 additions and 54 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/specs/pre-push-safety-net/brief.md
+13 -7
View File
@@ -1,6 +1,6 @@
# Brief: pre-push-safety-net
Status: requirements, design and tasks approved; groups 1–5 implemented and verified (16/27 subtasks). The first PMPro/Node experiment is measured NO-GO: six cold timeouts, no completed target checks. Groups 6–8 pending; normal capabilities remain disabled and warm hook latency unproven. Language: English.
Status: requirements, design and tasks approved; groups 1–6 implemented and verified (20/27 subtasks). Compatible preparation/graph/query reuse is demonstrated, including packaged JavaScript evidence equivalence. Groups 7–8 pending. Rollout remains NO-GO: changed-code feasibility and independent capability qualification are not established; normal capabilities remain disabled. Language: English.
## Problem
@@ -10,12 +10,18 @@ maintainer explicitly rejects noisy advisory output as well as noisy blocking ou
## Current State
The scan driver and Joern arbiters exist, with demonstrated PHP/Python cases. Immutable push-tip
resolution, isolated source materialization and lexical change context are implemented and verified.
Ranker integration, persistent graph caching and an end-to-end deadline remain pending.
The driver builds a whole-root graph before cutting the region budget. The latest PMPro ranking
experiment alone spends about 130 seconds building and 296 seconds extracting evidence. A manually
selected two-file slice finding a CVE does not demonstrate automatic incremental analysis.
The existing ranker, Joern arbiters, immutable replay, change comparison, actionability policy
and absolute deadline are integrated. Vendor code is physically excluded from frontend graphs
and targets. Explicit local caching reuses complete discovery, validated isolated graph copies
and compatible query evidence while reapplying current admission. Changed source/declarations
invalidate affected graph units; no incremental Joern mutation is implemented.
Group 5's PMPro/Node cold experiment remains NO-GO. Group 6 adds a packaged JavaScript
correctness control with equal cached/uncached evidence and admission; an identical-revision
hit is fast, while the changed-code deadline remains a separate unmet gate. Automatic
context projection and independent capability eligibility remain explicit limits. See
implementation-group-6.md and the committed reuse control evidence. Historical whole-PMPro
ranking costs and manually selected CVE slices do not establish automatic push latency.
## Desired Outcome
.kiro/specs/pre-push-safety-net/design.md
+5 -3
View File
@@ -147,6 +147,7 @@ New files (proposed, not created by this documentation change):
| `src/openultrasast/push/contracts.py` | Immutable ref/comparison identities and comparison-owned completion/status records |
| `src/openultrasast/push/snapshot.py` | Parse ref updates, resolve bases, materialize blobs, produce change context |
| `src/openultrasast/push/cache.py` | Artifact keys, validity, atomic publication and eviction |
| `src/openultrasast/push/reuse.py` | Complete discovery reuse and backend graph/query reuse through the existing driver |
| `src/openultrasast/push/policy.py` | Delta identity, actionability admission and push disposition |
| `src/openultrasast/push/runner.py` | Shared deadline, ranker/engine orchestration, per-ref result aggregation |
| `src/openultrasast/push/report.py` | Compact terminal result and reproducible push manifest |
@@ -157,8 +158,9 @@ Contract configuration (task 1.4) uses `[push]`: optional `comparison_base`, pos
`deadline_seconds` (30), positive `cancellation_allowance_seconds` (2), positive integer
`cache_max_bytes` (2 GiB), `mode` (`advisory` by default, or `blocking`) and independent
`incomplete_coverage_policy` (`allow` by default, or `block`). Unknown keys, malformed
sections and coerced/nonfinite numeric values are rejected. These settings are inert until
runner integration; they do not enable a hook or admit experimental capabilities. The cache
sections and coerced/nonfinite numeric values are rejected. Experimental replay consumes the deadline, mode, coverage policy and optional cache budget.
An explicit `--cache-dir` enables local reuse; omission retains a cold control. These settings
do not install a hook or admit experimental capabilities. The cache
limit is a local storage budget, not a latency claim. Completion records are owned by an
exact comparison so reused head graphs cannot merge evidence from different bases. Live
leases have no JSON constructor; only the backend can validate and issue them in task 6.1.
@@ -251,7 +253,7 @@ cleanup consume the same execution deadline and cancellation allowance as the ba
Runtime origin is `unspecified` when no declaration establishes browser/server scope;
JavaScript presence alone does not admit an Express capability. TypeScript and C/C++
retain explicit capability limits. These records prepare later artifact manifests;
they do not implement persistent graph storage.
task 6 now binds them to backend-owned persistent graph manifests and isolated leases.
Graph key: digest of every included blob/path and applicable declarations, frontend/engine version,
overlay semantics/options, language, and exclusions. Query key additionally includes graph key,
.kiro/specs/pre-push-safety-net/implementation-group-6.md
+93
View File
@@ -103,3 +103,96 @@ the cache never confers eligibility. JSON decoding is non-executable and bounded
- CLAIM: Task 6.3 semantic keys and completed comparison evidence reuse.
- EVIDENCE: production policy integration and cache key tests, 1260 passing suite tests and final static checks.
- GAPS: Graph/query/preparation reuse must still be connected to replay under 6.4; no rollout claim.
## Task 6.4 brief
Requirements 1.1, 2.2, 4.1, 4.2, 4.3, 4.4 and 7.3; design: Push Runner and
Cache and Engine Boundary. Explicit `--cache-dir` integrates the three cache layers;
omitting it preserves a cold replay control. Complete immutable snapshots are still read
and verified by the adapter before persisted discovery can be used. Equal base/head revisions
share one live materialization. Discovery keys include exact file identities and shipped code/facts;
dependency/declaration changes invalidate them. Graph units remain the existing physical frontend
partitions. Source changes rebuild the affected unit; all declaration and exclusion context is
bound. There is no incremental graph mutation or changed-file scope selector.
The runner uses backend-issued copies and per-question cached raw answers, then runs the existing
ranker, arbiter, comparison and current admission. Partial answers are not promoted to complete
lookups. Complete answers already returned by the engine remain available when persistence cannot
finish. Query keys include actual parameters and graph bytes; graph keys do not include unrelated
ranking/security-fact versions. Layout/exclusion facts and frontend adaptation bytes do affect them.
Integration review repaired the task 6.1 identity to include installed frontend/adaptation bytes,
rather than relying only on Joern's release label. This is an owning-contract repair, not a new
frontend or a detector change.
The first source-mounted/package integration trial is retained separately from the final
frozen image run. It showed equal cached/uncached evidence and admission (including one
new JavaScript injection candidate), a 1.23-second identical-revision hit, and a 30.48-second
dependency-edit timeout. Those numbers are preliminary control evidence, not the final image
or rollout measurements. Final review also binds installed runtime identity into admission
semantics, while avoiding a second backend instance or an unbudgeted availability probe.
The cache entry lease ends after its validated bytes are copied into the backend's independent
working lease. The working copy remains valid if storage later evicts the original entry;
queries never operate on the stored original. Partial fresh answers cannot borrow a cached
answer's census. Pure discovery remains in the existing bounded child process; persisted
JSON is decoded as data, never executable objects.
## Status Report
- STATUS: READY_FOR_REVIEW
- TASK: 6.4
- RED_PHASE_OUTPUT: `/tmp/group6-4-red.log`: graph/query reuse and replay discovery/cache-location tests failed before integration (temporary switch OFF). Enabled implementation passed; switch removed. Additional regressions cover mixed cached/fresh census failure and installed runtime provenance.
- TESTS_RUN: 89 final focused tests passed; full suite 1268 passed, 9 skipped in 108.74s. Ruff, formatting, mypy (114 source files), compileall, detection/map/pair gates and Docker build/packaged CLI smoke passed.
- EVIDENCE: final packaged JavaScript control exited 0 with exact cached/uncached scope, question outcomes, findings and admission equality. One new injection is retained as diagnostic evidence and denied normal admission by the empty registry. Source/dependency edits invalidate graph reuse and retain timed-out coverage artifacts.
- CONCERNS: identical-revision hits are development correctness controls, not representative warm p95 or capability qualification. Main-context review fallback remains explicit.
## Review Verdict
- VERDICT: APPROVED
- TASK: 6.4
- MECHANICAL_RESULTS: canonical tests and regression gates passed; final image reached the usable CLI and completed real replay controls. No introduced placeholders, secrets or boundary violations found.
- FINDINGS: installation-byte identity now also binds comparison/admission provenance. The predecessor no-unbudgeted-probe regression caught an unnecessary backend instance in provenance; the repair uses a pure runtime-identity function and the final regression passes. Mixed fresh/cached answers require the fresh query's own valid census before being retained as complete. Snapshot reads, discovery, ranker/driver selection, vendor boundary and independent push/coverage status remain intact.
- SUMMARY: Task 6.4 integrates valid local reuse and truthful misses under the existing transaction deadline without changing detection or admission rules.
## Verification Result
- STATUS: VERIFIED
- CLAIM_TYPE: TASK
- CLAIM: Group 6 implementation (6.1–6.4), with compatible replay evidence and admission equivalence.
- EVIDENCE: final tests/static checks, native packaged Joern controls, and committed measurement record below.
- GAPS: Rollout remains NO-GO. Groups 7–8, representative changed-code completion/p95 and independent capability qualification remain pending. Automatic projection limitations are unchanged.
## Final packaged evidence
Image: `sha256:6622b846996989a321d81bfa6c25e792e3f217984d7ede66f830cff1206a634c`.
Nonroot, network disabled, 3 GiB memory; controls ran serially without concurrent
tests/builds. Filesystem cache was not flushed. Each case is one development control,
not a representative p95 population.
| Run | Full CLI seconds | Graph hits | Query-answer hits |
|---|---:|---:|---:|
| populate | 154.848 | 0 | 0 |
| warm-identical | 2.002 | 2 | 37 |
| uncached | 130.432 | 0 | 0 |
| dependency-edit | 30.458 | 0 | 0 |
| one-function-edit | 30.479 | 0 | 0 |
| uncached, same revisions, 30s budget | 30.606 | 0 | 0 |
Cached and uncached laboratory runs have **equal scope, question outcomes, findings and
admission**, including the known introduced JavaScript injection. The 30-second identical
cached run retains that same evidence: one completed supported head injection check plus
four base comparison checks. The same revisions without caching complete zero checks in
the 30-second control. All six CLI artifacts were retained, including timed-out edits.
Warm source/dependency edits correctly invalidate the affected graph and produce incomplete
coverage; they do not replay the old answer. The empty normal capability registry denies the
new diagnostic candidate for `capability_unavailable`; this is not a clean repository verdict.
The manifest, full artifacts and stdout/stderr are retained in
`benchmarks/measurements/2026-09-13-reuse-control.json` and the associated final/initial
compressed bundles. Each bundle maps original filenames to exact UTF-8 content; decompression
was compared with the original files and SHA-256 hashes recorded. The committed reusable
harness is `benchmarks/push/reuse_control.py`; the extra same-revision 30-second uncached
command is recorded in the final bundle.
**Group 6 implementation: verified. Rollout: NO-GO.** Changed-code feasibility remains
unmet in these controls, and independent precision/recall/capability populations have not
qualified. The spec now explicitly requires stage-level investigation of remaining timeouts
under task 8.2, retaining its original thresholds and population rules. Group 7 is next;
no hook has been installed and no normal capability enabled.
.kiro/specs/pre-push-safety-net/spec.json
+13 -8
View File
@@ -1,7 +1,7 @@
{
"feature_name": "pre-push-safety-net",
"created_at": "2026-09-12T15:35:00Z",
"updated_at": "2026-09-13T13:22:55.634332+00:00",
"updated_at": "2026-09-13T15:06:56.856952+00:00",
"language": "en",
"phase": "implementation",
"approvals": {
@@ -19,7 +19,7 @@
}
},
"ready_for_implementation": true,
"implementation_approval_note": "Groups1\u20135 implemented and verified (16/27). Group5 used manual Kiro fallback review, repaired observed predecessor deadline bugs, and executed a newly frozen PMPro/Node experiment. Measured feasibility NO-GO; real capabilities remain disabled. Groups6\u20138 pending.",
"implementation_approval_note": "Groups 1\u20136 implemented and verified (20/27); group 6 authorized as a whole. Main-context Kiro review fallback after fresh dispatch thread limit. Valid reuse is demonstrated; real capability registry remains empty and rollout gates remain unmet. Groups 7\u20138 pending.",
"depends_on": [
"contributor-scan"
],
@@ -98,19 +98,24 @@
"4.2",
"4.3",
"5.1",
"5.2"
"5.2",
"6.1",
"6.2",
"6.3",
"6.4"
],
"remaining_tasks": 11,
"verification_record": "implementation-group-5.md",
"remaining_tasks": 7,
"verification_record": "implementation-group-6.md",
"verification_records": [
"implementation-group-1.md",
"implementation-group-2.md",
"implementation-group-3.md",
"implementation-group-4.md",
"implementation-group-5.md"
"implementation-group-5.md",
"implementation-group-6.md"
],
"next_group": 6,
"next_group": 7,
"blocked_tasks": [],
"feasibility": "NO-GO: six cold replay timeouts; no completed supported target checks; warm performance and eligibility unproven"
"feasibility": "NO-GO for rollout: compatible identical-revision reuse verified; representative changed-code p95/completion and independent alert eligibility remain unmet."
}
}
.kiro/specs/pre-push-safety-net/tasks.md
+12 -8
View File
@@ -1,11 +1,12 @@
# Implementation Plan: pre-push-safety-net
Status: task plan approved by the maintainer on 2026-09-12. Groups 1–5 (tasks 1.1–5.2)
are implemented, reviewed and verified (16/27 subtasks). Tasks 4.3 and group 5 used the
documented Kiro manual review fallback after agent dispatch reached its thread limit.
The first experiment is measured NO-GO: all six cold PMPro/Node cases timed out without
completed target checks. Groups 6–8 remain pending; no production capability or warm
hook latency is admitted.
Status: task plan approved by the maintainer on 2026-09-12. Groups 1–6 (tasks 1.1–6.4)
are implemented, reviewed and verified (20/27 subtasks). Tasks 4.3 and groups 5–6 used the
explicit Kiro manual review fallback after fresh agent dispatch reached its thread limit.
Group 5's six cold PMPro/Node timeouts remain historical NO-GO evidence. Group 6 verifies
compatible reuse with equal evidence/admission in a packaged authored JavaScript control;
its identical-revision run completed in 2.00s. This is not representative warm p95 or
independent capability qualification. Groups 7–8 remain pending; rollout remains NO-GO.
## Execution contract
@@ -160,7 +161,7 @@ hook latency is admitted.
- _Depends: 1.2, 1.3, 5.1_
- _Requirements: 6.2, 6.3, 6.4, 6.5, 8.1, 8.2, 8.3, 8.4_
- [ ] 6. Reuse valid graphs and results without stale conclusions
- [x] 6. Reuse valid graphs and results without stale conclusions
- [x] 6.1 Validate graph artifact identity and leases
- Have the backend own source/declaration/exclusion digests, frontend/engine/overlay options, graph census and complete/partial identity.
- Permit validated graph leases in the generic driver while preserving the fresh-build path; isolate potentially mutating loads and retain bounded cleanup.
@@ -180,7 +181,7 @@ hook latency is admitted.
- Done when source, context, fact, query, mode, policy and optional model changes invalidate the appropriate layer, while an identical compatible request returns the same evidence and provenance.
- _Boundary: Push cache, policy integration_
- _Requirements: 3.4, 4.3, 4.4, 6.4, 7.2, 7.3_
- [ ] 6.4 Integrate cache reuse into the shared push deadline
- [x] 6.4 Integrate cache reuse into the shared push deadline
- Reuse complete preparation/discovery by immutable tree and discovery/fact/configuration identity, retaining readable snapshot and declaration/boundary records; measure its costs and hits separately from graph/query reuse.
- Reuse identical snapshots/unchanged declared units across refs without merging different comparisons; changed units rebuild and stale graph relationships remain hints only.
- Account for lookup, lock wait, lease, build/query and result persistence in the same deadline; preserve completed outcomes on cancellation.
@@ -219,6 +220,7 @@ hook latency is admitted.
- _Requirements: 1.1, 1.2, 1.3, 1.4, 2.2, 2.3, 4.2, 4.4, 5.2, 5.5, 7.1, 7.4, 8.2_
- [ ] 8.2 Measure realistic cache and deadline feasibility
- Measure identical-tip reuse, a warm one-function edit, dependency/configuration edits, cold starts, repository growth and multi-ref workloads on declared hardware.
- If reuse correctness passes but changed-code checks time out, identify preparation, frontend/overlay/census and query costs separately and route the remaining optimization to its owning boundary before rollout GO. Preserve the declared gates and do not substitute identical-tip latency.
- Publish p50/p95, timeouts, cancellation allowance, completed/deferred population and stage costs; the denominator is fixed before ranker selection.
- Done when the versioned scorecard evaluates the accepted latency/coverage gates without hiding timeouts or calling repeated-identical-tip hits representative warm edits; a failed gate remains NO-GO for rollout.
- _Boundary: Evaluation harness, runtime integration_
@@ -244,3 +246,5 @@ hook latency is admitted.
- Task5.2 measured PMPro timing out after both snapshots were materialized but before change context/graph analysis. Group6 must account for compatible snapshot/discovery reuse as well as graph reuse; identical-tip hits cannot substitute for the later representative changed-code latency gate. No new detector, scope heuristic or incremental CPG mutation is implied.
- Cancellation must cover semantic planning and reporting, not only engine subprocesses: `_collect` and `_scope_work` now consume the shared deadline, and replay instantiates the budget-aware backend once. Preserve the 400-question failure-census and real Node artifact-retention regressions.
- Group 6: complete discovery/graph/query reuse is explicit via `--cache-dir`; source/declaration edits invalidate affected units. Installed frontend/adaptation bytes now bind graph and admission provenance. A valid cached census cannot qualify a fresh answer that omitted its own census. Identical-revision timing is not the changed-code rollout gate.
.kiro/steering/roadmap.md
+13 -8
View File
@@ -46,7 +46,7 @@ engine are out of scope.
- [ ] pre-push-safety-net — Immutable pushed tips, existing-ranker integration, compatible artifact
reuse, total deadline, change attribution, actionability admission and compact hook behavior.
Dependencies: contributor-scan. Requirements/design approved 2026-09-12; tasks generated
(8 groups, 27 executable subtasks), independently reviewed and approved; groups 1–5 are implemented and verified (16/27 tasks), including the resolved frontend-retention prerequisite.
(8 groups, 27 executable subtasks), independently reviewed and approved; groups 1–6 are implemented and verified (20/27 tasks), including the resolved frontend-retention prerequisite.
### Next concrete step
@@ -59,14 +59,19 @@ its deadline in preparation/discovery after materializing both trees; Node reach
driver but did not complete checks. The exact 11-case population remains unresolved.
No normal capability is enabled and these six cold samples establish no warm p95 claim.
Group 6 is next for valid graph/result identity, leases and reuse. Account for snapshot
and discovery costs before treating graph persistence as the sole optimization; record
compatible preparation reuse alongside graph reuse. Representative changed-code warm
latency and coverage must still be measured later, without substituting identical-tip hits.
Do not change ranking weights, add a second source-slicer or assume incremental Joern
mutation from this negative performance evidence.
Group 6 now provides complete preparation/discovery, graph and query reuse through explicit
`--cache-dir`, bounded publication and isolated graph leases. A packaged authored JavaScript
control preserved the same evidence/admission as an uncached run; its identical-revision
replay took 2.00s. Source/declaration changes invalidate affected units. This correctness
control does not establish representative changed-code p95, independent alert precision or
eligibility. Rollout remains NO-GO and normal capabilities remain disabled.
Tasks 4.3 and group 5 used the documented Kiro manual review fallback after the agent
Group 7 is next for the real multi-ref push interface and opt-in hook integration. Group 8
must still pass the joint gates on declared populations. Keep the existing thresholds and
profile preparation, graph construction and queries separately when investigating misses;
do not substitute identical-tip hits for warm changed-code performance.
Tasks 4.3 and groups 5–6 used the documented Kiro manual review fallback after the agent
thread limit; this is not independent review. See `pre-push-safety-net/implementation-group-5.md`
and `benchmarks/measurements/2026-09-13-first-replay-experiment.json` for provenance,
initial instrument failures, their repairs and the newly frozen final trial. The NodeGoat
.kiro/steering/safety-net.md
+9 -4
View File
@@ -79,16 +79,21 @@ updates as a shortcut to an unmeasured latency promise.
The product direction above is user-requested. The amended requirements, design and initial
evaluation profile in `pre-push-safety-net` were approved for task generation on 2026-09-12
("ok, next kiro phase"). The maintainer then approved the task plan and starting group 1
("approve.") and then group 2 ("ok, next kiro phase, approved"). Groups 1–5 are now
implemented, reviewed and verified (16/27), including the frontend-retention prerequisite
and the deadline regression exposed by full Node replay. Tasks 4.3 and group 5 used the
("approve.") and then group 2 ("ok, next kiro phase, approved"). Groups 1–6 are now
implemented, reviewed and verified (20/27), including the frontend-retention prerequisite
and the deadline regression exposed by full Node replay. Tasks 4.3 and groups 5–6 used the
explicit Kiro manual review fallback after the agent thread limit.
The first newly frozen PMPro/Node experiment is measured **NO-GO**: all six cold cases
timed out without completed supported target checks, while retaining all 11 declared cases.
The repaired CLI saved all artifacts and returned in 30.55–30.64 seconds. PMPro exhausted
the deadline in preparation/discovery before graph analysis, so compatible preparation
reuse matters alongside graph persistence. Group 6 is next; warm changed-code performance
reuse matters alongside graph persistence. Group 6 now provides compatible local reuse; warm changed-code performance
and useful coverage remain mandatory later gates. No ranking/fact adaptation or holdout
inspection occurred in the final transfer run. Real capability eligibility and automatic
dominance context projection remain unproven; the default capability registry is empty.
Group 6 verified equal cached/uncached evidence and current admission in a packaged authored
JavaScript control. Source/declaration edits invalidate affected units. Identical-revision
latency does not qualify representative changed-code performance. Requirements and rollout
thresholds remain unchanged; groups 7–8 and independent capability admission remain pending.
benchmarks/measurements/2026-09-08-module-audit.json
+30 -6
View File
@@ -1,10 +1,10 @@
{
"counts": {
"load_bearing": 103,
"load_bearing": 104,
"standalone_capability": 9
},
"modules": 112,
"note": "Refreshed for task 6.2: bounded local artifact storage, integrated by task 6.4.",
"modules": 113,
"note": "Refreshed for group 6: replay reaches compatible graph, query, discovery and comparison reuse.",
"orphaned": [],
"rows": [
{
@@ -118,6 +118,7 @@
"config",
"cpg.artifact",
"model.contracts",
"push.cache",
"push.contracts",
"push.policy"
],
@@ -139,6 +140,7 @@
"model.taint",
"push.cache",
"push.policy",
"push.reuse",
"push.runner"
],
"module": "cpg",
@@ -148,7 +150,10 @@
"classification": "load_bearing",
"importers": [
"cpg.backend",
"push.cache"
"push.cache",
"push.policy",
"push.reuse",
"push.runner"
],
"module": "cpg.artifact",
"reason": "pre-push-safety-net Req 4.3/7.3: backend graph identity and live lease contract; validation integration pending"
@@ -167,6 +172,7 @@
"model.taint",
"push.cache",
"push.policy",
"push.reuse",
"push.runner"
],
"module": "cpg.backend",
@@ -374,6 +380,7 @@
"push.contracts",
"push.policy",
"push.report",
"push.reuse",
"push.runner",
"push.snapshot"
],
@@ -421,6 +428,7 @@
"push.contracts",
"push.policy",
"push.report",
"push.reuse",
"push.runner",
"push.snapshot"
],
@@ -484,7 +492,8 @@
"importers": [
"model.partitions",
"model.scan",
"preprocess"
"preprocess",
"push.runner"
],
"module": "model.layout",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
@@ -513,6 +522,7 @@
"model.layout",
"model.scan",
"push.policy",
"push.reuse",
"push.runner"
],
"module": "model.regions",
@@ -690,6 +700,7 @@
"cli",
"push.policy",
"push.report",
"push.reuse",
"push.runner",
"push.snapshot"
],
@@ -698,7 +709,11 @@
},
{
"classification": "load_bearing",
"importers": [],
"importers": [
"push.policy",
"push.reuse",
"push.runner"
],
"module": "push.cache",
"reason": "pre-push-safety-net task 6.2: bounded artifact publication and leased eviction; runner integration in 6.4"
},
@@ -707,6 +722,7 @@
"importers": [
"push.policy",
"push.report",
"push.reuse",
"push.runner",
"push.snapshot"
],
@@ -730,6 +746,14 @@
"module": "push.report",
"reason": "pre-push-safety-net Req 5.1/5.2/5.5/7.3: compact output and complete artifacts; runner integration pending"
},
{
"classification": "load_bearing",
"importers": [
"push.runner"
],
"module": "push.reuse",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
benchmarks/measurements/2026-09-13-reuse-control.json
+1627
View File
File diff suppressed because it is too large Load Diff
benchmarks/measurements/2026-09-13-reuse-final-control.json.gz
BIN
View File
Binary file not shown.
benchmarks/measurements/2026-09-13-reuse-initial-control.json.gz
BIN
View File
Binary file not shown.
benchmarks/push/README.md
+21
View File
@@ -137,3 +137,24 @@ flushed. Six one-shot runs are diagnostic samples, not representative warm p95 e
Normal capabilities remain disabled, so zero alerts do not establish precision or useful
recall. A separately named longer lab run may obtain witnesses, but cannot replace the
30-second measurements or satisfy the hook's latency gate.
### Group 6 reuse correctness control
`reuse_control.py` creates an isolated, authored JavaScript history and invokes the packaged
replay command for a cache-populating run, a 30-second identical-revision run, an uncached
control, and dependency/one-function edits. It retains every artifact and checks equal evidence
and current admission for compatible completed questions. This is development control data,
not the untouched eligibility population or the representative warm p95 gate.
```sh
docker run --rm --network none --memory 3g \
-v /path/to/empty/results:/results \
-v "$PWD/benchmarks/push/reuse_control.py:/control.py:ro" \
--entrypoint python openultrasast:dev /control.py --out /results
```
The output directory must be empty and writable by the image's nonroot user. Cold runs use a
600-second laboratory budget; the identical-revision and changed-code controls retain the
30-second deadline and 2-second cancellation allowance. Default normal capabilities remain
empty. A passing reuse control cannot authorize rollout or turn unresolved coverage into clean
analysis; groups 7–8 still own the real interface and joint evaluation.
benchmarks/push/reuse_control.py
+133
View File
@@ -0,0 +1,133 @@
"""Packaged JavaScript reuse correctness control, not capability qualification.
Run with --out pointing to an empty directory outside any analyzed repository.
Authored development code is never an independent precision/recall population.
"""
import argparse
import json
import subprocess
import time
from pathlib import Path
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--out", type=Path, required=True)
args = parser.parse_args()
out = args.out.resolve()
out.mkdir(parents=True, exist_ok=True)
if any(out.iterdir()):
parser.error("output directory must be empty")
repo = out / "repo"
repo.mkdir()
def git(*args):
return subprocess.run(["git", "-C", str(repo), *args], check=True, capture_output=True, text=True).stdout.strip()
git("init")
git("config", "user.name", "Replay control")
git("config", "user.email", "[email protected]")
git("config", "core.hooksPath", "/dev/null")
source = repo / "api.js"
source.write_text('function handler(req, res) {\n const value = "fixed";\n eval(value);\n}\napp.post("/run", handler);\n')
assert len(source.read_bytes()) > 0
git("add", "api.js")
git("commit", "-m", "fixed")
base = git("rev-parse", "HEAD")
source.write_text(source.read_text().replace('"fixed"', "req.body.value"))
git("add", "api.js")
git("commit", "-m", "introduce")
head = git("rev-parse", "HEAD")
runs = []
def run(name, deadline, cache=True, tip=head):
artifact = out / (name + ".json")
cmd = [
"python",
"-m",
"openultrasast.cli",
"pre-push",
str(repo),
"--base",
base,
"--head",
tip,
"--artifact",
str(artifact),
"--deadline",
str(deadline),
"--max-regions",
"20",
]
if cache:
cmd += ["--cache-dir", str(out / "cache")]
start = time.monotonic()
process = subprocess.run(cmd, capture_output=True, text=True, timeout=deadline + 10)
elapsed = time.monotonic() - start
(out / (name + ".stdout")).write_text(process.stdout)
(out / (name + ".stderr")).write_text(process.stderr)
assert artifact.is_file(), (process.returncode, process.stdout, process.stderr)
data = json.loads(artifact.read_text())
run = {
"name": name,
"elapsed_seconds": elapsed,
"exit_code": process.returncode,
"timings": data["timings"],
"result": data["result"],
"artifact": str(artifact),
}
runs.append(run)
(out / "runs.json").write_text(json.dumps(runs, indent=2))
print(json.dumps(run), flush=True)
return data
populated = run("populate", 600)
warm = run("warm-identical", 30)
cold = run("uncached", 600, False)
(repo / "package.json").write_text('{"dependencies":{"express":"5.0.0"}}\n')
git("add", "package.json")
git("commit", "-m", "dependency change")
changed = git("rev-parse", "HEAD")
run("dependency-edit", 30, tip=changed)
git("rm", "package.json")
source.write_text(source.read_text().replace("req.body.value", 'req.body.value + "!"'))
git("add", "api.js")
git("commit", "-m", "one function edit")
edited = git("rev-parse", "HEAD")
run("one-function-edit", 30, tip=edited)
def evidence(data):
return [
{
"side": s["side"],
"scope": s["scan"]["scope"],
"questions": s["scan"]["question_outcomes"],
"findings": s["scan"]["findings"],
"degradations": s["scan"]["degradations"],
}
for s in data["scans"]
]
summary = {
"cold_populated_evidence_equal": evidence(cold) == evidence(populated),
"cold_warm_evidence_equal": evidence(cold) == evidence(warm),
"admission_equal": cold["admission"] == warm["admission"],
"runs": runs,
"source_bytes": len(source.read_bytes()),
"rollout": "not qualified; authored development control only",
}
(out / "summary.json").write_text(json.dumps(summary, indent=2))
print(json.dumps(summary), flush=True)
assert summary["cold_populated_evidence_equal"] and summary["cold_warm_evidence_equal"] and summary["admission_equal"]
assert warm["timings"]["graph_hits"] == 2 and warm["timings"]["query_hits"] > 0
assert any(item["candidate"]["delta"]["novelty"] == "new" for item in warm["admission"]["dispositions"])
assert not any(data["admission"]["defects"] for data in (populated, warm, cold))
assert runs[1]["elapsed_seconds"] <= 32
assert all(
r["timings"]["graph_hits"] <= 1 and r["elapsed_seconds"] <= 32 for r in runs if r["name"] in ("dependency-edit", "one-function-edit")
)
ops/README.md
+16
View File
@@ -266,3 +266,19 @@ it does not demand a positive after an interrupted query. Run with the network-d
image, mounting the smoke script read-only as for the other smoke scripts. Snapshot manifests
record verified immutable source bytes; full scan records retain exact selected/deferred questions,
raw answers, per-question gaps, change context and candidate dispositions.
Experimental replay can reuse compatible local preparation, graphs and query evidence:
```sh
ousast pre-push /path/to/repo --base BASE --head HEAD \
--artifact /path/outside/repo/result.json \
--cache-dir /path/outside/repo/private-cache
```
Omit `--cache-dir` for a cold control. The cache directory must be owned by the current
user, private (0700), and outside the analyzed repository. Its default payload/manifest
budget is 2 GiB (`PushConfig.cache_max_bytes` for API callers). Cache permission to reuse
is separate from alert eligibility: current admission is always applied. No hook is installed
by replay. Artifact timings record discovery, graph and query hits; identical-tip hits do
not establish representative changed-code latency. Corrupt, partial or incompatible entries
are misses, with remaining work constrained by the same push deadline.
src/openultrasast/cli.py
+10 -1
View File
@@ -115,6 +115,9 @@ def main(argv: list[str] | None = None) -> int:
replay_parser.add_argument("--base", required=True)
replay_parser.add_argument("--head", required=True)
replay_parser.add_argument("--artifact", type=Path, required=True)
replay_parser.add_argument(
"--cache-dir", type=Path, help="reuse compatible local artifacts in a private directory outside the repository"
)
replay_parser.add_argument("--deadline", type=float, default=30.0)
replay_parser.add_argument("--cancellation-allowance", type=float, default=2.0)
replay_parser.add_argument("--max-regions", type=int, default=500)
@@ -212,7 +215,13 @@ def main(argv: list[str] | None = None) -> int:
incomplete_coverage_policy=args.incomplete_coverage,
)
delivery = replay(
args.path, base=args.base, head=args.head, artifact=args.artifact, config=settings, max_regions=args.max_regions
args.path,
base=args.base,
head=args.head,
artifact=args.artifact,
config=settings,
max_regions=args.max_regions,
cache_dir=args.cache_dir,
)
except ValueError as error:
parser.error(str(error))
src/openultrasast/cpg/backend.py
+47 -2
View File
@@ -266,6 +266,41 @@ class NullBackend:
return None
def engine_runtime_identity(
*, include_tests: bool = True, language: str | None = None, execution_budget: ExecutionBudget | None = None
) -> str:
"""Installed frontend/adaptation bytes, also used by comparison eligibility provenance."""
from .artifact import digest_value, graph_bytes
budget = execution_budget
deadline = budget.deadline_monotonic if budget else float("inf")
launcher = shutil.which("joern")
version = joern_version()
installation = []
if launcher is not None and version is not None:
home = Path(launcher).resolve().parent
engine = ".".join(map(str, version))
frontends = {_FRONTENDS.get(language, "joern-parse")} if language else set(_FRONTENDS.values())
candidates = {home / f"lib/io.joern.joern-cli-{engine}.jar", Path(launcher).resolve()}
for frontend in frontends:
candidates.add(home / f"frontends/{frontend}/lib/io.joern.{frontend}-{engine}.jar")
frontend_launcher = shutil.which(frontend)
if frontend_launcher is not None:
candidates.add(Path(frontend_launcher).resolve())
if "jssrc2cpg" in frontends:
candidates.update({home / "ousast-frontend-retention-v1/manifest.json", home / "frontends/jssrc2cpg/bin/astgen/astgen-linux"})
for candidate in sorted(candidates):
installation.append((candidate.name, graph_bytes(candidate, deadline) if candidate.is_file() else "absent"))
return digest_value(
{
"version": version,
"files": installation,
"include_tests": include_tests,
"java_options": os.environ.get("JAVA_TOOL_OPTIONS", ""),
}
)
@dataclass
class JoernBackend:
"""``joern-parse`` then ``joern --script``, both out of process and both timeout-bounded."""
@@ -324,7 +359,8 @@ class JoernBackend:
) -> GraphIdentity:
from .artifact import digest_value, read_bytes, source_inventory
deadline = execution_budget.deadline_monotonic if execution_budget else float("inf")
budget = execution_budget or self.execution_budget
deadline = budget.deadline_monotonic if budget else float("inf")
version = joern_version()
if version is None:
raise ValueError("engine_identity_unknown")
@@ -342,7 +378,16 @@ class JoernBackend:
engine,
(DATAFLOW_OVERLAY,),
(("implementation", code),),
(("include_tests", str(self.include_tests)), ("java_options", os.environ.get("JAVA_TOOL_OPTIONS", "default") or "default")),
(
("include_tests", str(self.include_tests)),
(
"installation",
engine_runtime_identity(
include_tests=self.include_tests, language=language, execution_budget=execution_budget or self.execution_budget
),
),
("java_options", os.environ.get("JAVA_TOOL_OPTIONS", "default") or "default"),
),
)
def describe_graph(
src/openultrasast/push/reuse.py
+202
View File
@@ -0,0 +1,202 @@
"""Compatible replay reuse; the existing ranker and detector remain decision owners."""
from __future__ import annotations
import math
import os
import time
from collections.abc import Callable, Mapping, Sequence
from dataclasses import asdict, replace
from pathlib import Path
from typing import Any
from openultrasast.cpg.artifact import GraphArtifact, digest_value
from openultrasast.cpg.backend import CpgResult, JoernBackend
from openultrasast.model.contracts import ExecutionBudget
from openultrasast.model.regions import ScanRegion
from openultrasast.push.cache import ArtifactCache, SemanticKeys
from openultrasast.push.contracts import SnapshotManifest
class ReusingBackend:
def __init__(self, backend: JoernBackend, cache: ArtifactCache, semantics: SemanticKeys, *, declarations: str, exclusions: str) -> None:
self.backend = backend
self.cache = cache
self.semantics = semantics
self.declarations = declarations
self.exclusions = exclusions
self.last_failure = ""
self.graph_hits = 0
self.query_hits = 0
def build(self, root: Path, *, language: str, exclude: Sequence[str], execution_budget: ExecutionBudget) -> CpgResult | None:
identity = None
artifact = None
result = None
try:
identity = self.backend.graph_identity(
root,
language=language,
declarations_digest=self.declarations,
exclusions_digest=digest_value([self.exclusions, sorted(exclude)]),
execution_budget=execution_budget,
)
key = digest_value({"layer": "graph-v1", "identity": identity.to_payload()})
with self.cache.lookup(key, execution_budget) as entry:
if entry is not None:
artifact = GraphArtifact.from_payload(entry.metadata)
result = self.backend.load_graph(entry.path, artifact, identity, execution_budget=execution_budget)
if result is not None:
self.graph_hits += 1
except (OSError, ValueError, TimeoutError):
artifact = None
if result is None:
artifact = None
result = self.backend.build(root, language=language, exclude=exclude, execution_budget=execution_budget)
self.last_failure = self.backend.last_failure
if result is None:
return None
if identity is not None and time.monotonic() < execution_budget.deadline_monotonic:
artifact = self.backend.describe_graph(result, root, identity, execution_budget=execution_budget)
if artifact is not None:
self.cache.publish(key, result.cpg_path, artifact.to_payload(), execution_budget)
if artifact is None:
return result
return self._answers(result, artifact, execution_budget)
def _answers(self, result: CpgResult, artifact: GraphArtifact, budget: ExecutionBudget) -> CpgResult:
def normalize(value: Any) -> Any:
if isinstance(value, str):
return value.replace(artifact.source_root + "/", "")
if isinstance(value, list):
return [normalize(item) for item in value]
if isinstance(value, dict):
return {key: normalize(item) for key, item in value.items()}
return value
def batch(kind: str, requests: Mapping[str, Mapping[str, object]]) -> dict[str, list[object]] | None:
answers: dict[str, list[object]] = {}
missing = {}
keys = {}
for rid, request in requests.items():
if time.monotonic() >= budget.deadline_monotonic:
break
key = self.semantics.query(
graph=digest_value({"identity": artifact.identity.to_payload(), "bytes": artifact.graph_digest}),
kind=kind,
request=request,
context={"paths": artifact.source_paths, "declarations": artifact.identity.declarations_digest},
)
keys[rid] = key
cached = self.cache.get_json(key, kind="query", budget=budget)
if isinstance(cached, dict) and isinstance(cached.get("rows"), list) and self._census(cached.get("census"), artifact):
answers[rid] = cached["rows"]
answers["__census__"] = cached["census"]
self.query_hits += 1
else:
missing[rid] = request
if missing and time.monotonic() < budget.deadline_monotonic:
raw: Any = result.run_batch(kind, missing) if result.run_batch else None
if raw is None and result.run_batch is None:
raw = {rid: result.run(kind, params) for rid, params in missing.items()}
if isinstance(raw, dict):
normalized = normalize(raw)
census = normalized.get("__census__")
valid = self._census(census, artifact)
if not valid:
return answers or None
answers["__census__"] = census
for rid in missing:
rows = normalized.get(rid)
if isinstance(rows, list):
answers[rid] = rows
self.cache.put_json(keys[rid], {"rows": rows, "census": census}, kind="query", budget=budget, complete=valid)
return answers or None
def run(kind: str, params: Mapping[str, object]) -> object | None:
if kind == "census":
value: object = normalize(result.run(kind, params))
return value
rows = batch(kind, {"one": params})
return rows.get("one") if rows is not None else None
return replace(result, run=run, run_batch=batch)
@staticmethod
def _census(value: object, artifact: GraphArtifact) -> bool:
if not isinstance(value, list) or len(value) != 1 or not isinstance(value[0], dict):
return False
row = value[0]
names = row.get("file_names")
try:
return (
isinstance(names, list)
and all(isinstance(name, str) for name in names)
and set(artifact.source_paths) <= set(names)
and int(str(row.get("methods", 0))) > 0
and int(str(row.get("files", 0))) >= len(artifact.source_paths)
and int(str(row.get("shards", 1))) == 1
)
except (ValueError, TypeError):
return False
def discovery(
root: Path,
manifest: SnapshotManifest,
*,
cache: ArtifactCache | None,
identity: str,
budget: ExecutionBudget,
discover: Callable[[Path], tuple[tuple[ScanRegion, ...], tuple[bytes, ...]]],
) -> tuple[tuple[ScanRegion, ...], tuple[bytes, ...], bool]:
"""Persist pure discovery only after the adapter proves the complete snapshot readable."""
key = digest_value(
{
"layer": "discovery-v1",
"files": [item.to_payload() for item in manifest.files],
"object_format": manifest.object_format,
"identity": identity,
}
)
raw = cache.get_json(key, kind="discovery", budget=budget) if cache and manifest.complete else None
if isinstance(raw, dict):
try:
regions = []
for item in raw["regions"]:
values = dict(item)
values["families"] = tuple(values["families"])
region = ScanRegion(**values)
if (
not all(isinstance(v, str) for v in (region.path, region.language, region.source, *region.families))
or region.function is not None
and not isinstance(region.function, str)
or type(region.shipped) is not bool
or type(region.rank) not in (int, float)
or not math.isfinite(region.rank)
):
raise ValueError("invalid_discovery_region")
if os.fsencode(region.path) not in {item.path_bytes for item in manifest.files}:
raise ValueError("discovery_path_outside_snapshot")
regions.append(region)
declarations = tuple(bytes.fromhex(item) for item in raw["declarations"])
if not set(declarations) <= {item.path_bytes for item in manifest.files}:
raise ValueError("invalid_declarations")
return tuple(regions), declarations, True
except (KeyError, ValueError, TypeError):
pass
found, declarations = discover(root)
if cache and manifest.complete:
cache.put_json(
key,
{"regions": [asdict(region) for region in found], "declarations": [item.hex() for item in declarations]},
kind="discovery",
budget=budget,
complete=True,
)
return found, declarations, False
def declaration_identity(manifest: SnapshotManifest, paths: tuple[bytes, ...]) -> str:
declared = set(paths)
return digest_value([item.to_payload() for item in manifest.files if item.path_bytes in declared])
src/openultrasast/push/runner.py
+90 -7
View File
@@ -9,18 +9,22 @@ import pickle
import selectors
import time
from collections.abc import Callable
from contextlib import nullcontext
from dataclasses import asdict, replace
from pathlib import Path
from typing import Any, Literal, TypeVar, cast
from openultrasast.config import PushConfig
from openultrasast.cpg.backend import JoernBackend, joern_version
from openultrasast.cpg.artifact import digest_value
from openultrasast.cpg.backend import JoernBackend, engine_runtime_identity, joern_version
from openultrasast.mapping import analyze_entry_points
from openultrasast.model.contracts import ExecutionBudget
from openultrasast.model.layout import layout_facts
from openultrasast.model.regions import ScanRegion, regions_for
from openultrasast.model.scan import ScanBudget, scan_repository
from openultrasast.model.shipped import declared_sources
from openultrasast.preprocess import build_file_target, enumerate_source_files
from openultrasast.push.cache import ArtifactCache, SemanticKeys
from openultrasast.push.contracts import ComparisonAnalysis, SnapshotManifest
from openultrasast.push.policy import (
AdmissionCandidate,
@@ -32,6 +36,7 @@ from openultrasast.push.policy import (
semantics_digest,
)
from openultrasast.push.report import PushReport, ReportDelivery, ScanRecord, _completed_scan, deliver_report, render_report
from openultrasast.push.reuse import ReusingBackend, declaration_identity, discovery
from openultrasast.push.snapshot import SnapshotAdapter
_T = TypeVar("_T")
@@ -124,14 +129,25 @@ def _provenance(repository: Path, scan_budget: ScanBudget) -> dict[str, str]:
version = joern_version()
engine = "joern-" + (".".join(map(str, version)) if version else "unavailable_or_unidentified")
runtime = engine_runtime_identity()
integration = digest(list((root / "cpg").glob("*.py")))
return {
"repository": str(repository.absolute()),
"engine_runtime": runtime,
"engine": engine,
"facts": digest(list((root / "ruleset").rglob("*.toml"))),
"graph_layout": digest_value(
{
"facts": [asdict(row) for row in layout_facts()],
"implementation": digest([root / "model/layout.py", root / "model/partitions.py"]),
}
),
"queries": digest(list((root / "cpg/queries").glob("*.sc"))),
"policy": digest(list((root / "push").glob("*.py"))),
"core": digest(list((root / "model").glob("*.py")) + list((root / "cpg").glob("*.py"))),
"semantics": semantics_digest(engine_identity=engine, options=asdict(scan_budget)),
"semantics": semantics_digest(
engine_identity=engine, options={**asdict(scan_budget), "engine_runtime": runtime, "engine_integration": integration}
),
"ranking_mode": "evidence",
"model": "disabled",
"capabilities": "empty_registry_pending_task_8.3",
@@ -139,6 +155,13 @@ def _provenance(repository: Path, scan_budget: ScanBudget) -> dict[str, str]:
}
def _discovery_identity() -> str:
root = Path(__file__).resolve().parents[1]
# Discovery invokes the shipped mapper, preprocessing, layout and family specs.
# Hash code bytes, never a mutable import name or a repository-provided version.
return digest_value([(p.relative_to(root).as_posix(), hashlib.sha256(p.read_bytes()).hexdigest()) for p in sorted(root.rglob("*.py"))])
def replay(
repository: Path,
*,
@@ -148,16 +171,22 @@ def replay(
config: PushConfig | None = None,
backend: Any = None,
max_regions: int = 500,
cache_dir: Path | None = None,
) -> ReportDelivery:
"""Run a single explicit comparison; exit policy never stands for coverage."""
config = config or PushConfig()
if max_regions < 1:
raise ValueError("max_regions must be positive")
if cache_dir is not None and cache_dir.resolve().is_relative_to(repository.resolve()):
raise ValueError("cache directory must be outside the analyzed repository")
started = time.monotonic()
budget = ExecutionBudget(started + config.deadline_seconds, config.cancellation_allowance_seconds)
scan_budget = ScanBudget(max_model_calls=0, max_regions=max_regions, order_by_evidence=True)
if backend is None:
backend = JoernBackend()
cache = None
cache_semantics = None
reused: list[ReusingBackend] = []
records: list[ScanRecord] = []
manifests: list[SnapshotManifest] = []
reasons: list[str] = []
@@ -179,16 +208,66 @@ def replay(
stage, stage_started = "provenance", time.monotonic()
provenance.update(_prepare(lambda: _provenance(repository, scan_budget), budget))
timings[stage + "_seconds"] = time.monotonic() - stage_started
if cache_dir is not None:
cache = ArtifactCache(cache_dir, max_bytes=config.cache_max_bytes)
cache_semantics = SemanticKeys(
provenance["facts"],
provenance["queries"],
digest_value({"query_parameters": "complete-joern-request-v1"}),
digest_value({"ranking_mode": "evidence", "options": asdict(scan_budget)}),
provenance["policy"],
provenance["model"],
provenance["capabilities"],
config.mode,
)
provenance["cache"] = "validated_local_reuse"
discovery_identity = digest_value({"code": _prepare(lambda: _discovery_identity(), budget), "facts": provenance["facts"]})
stage, stage_started = "preparation", time.monotonic()
with adapter.materialize(comparison.head_oid, budget=budget) as tip:
manifests.append(tip.manifest)
assert comparison.base_oid is not None
with adapter.materialize(comparison.base_oid, budget=budget) as old:
old_context = (
nullcontext(tip) if comparison.base_oid == comparison.head_oid else adapter.materialize(comparison.base_oid, budget=budget)
)
with old_context as old:
manifests.append(old.manifest)
for manifest in manifests:
reasons.extend("snapshot:" + boundary.reason for boundary in manifest.boundaries)
head_regions, head_declarations = _prepare(lambda: _discover(tip.root), budget)
base_regions, base_declarations = _prepare(lambda: _discover(old.root), budget)
timings["snapshot_seconds"] = time.monotonic() - stage_started
timings["snapshot_reuses"] = int(comparison.base_oid == comparison.head_oid)
discovery_started = time.monotonic()
head_regions, head_declarations, head_hit = _prepare(
lambda: discovery(tip.root, tip.manifest, cache=cache, identity=discovery_identity, budget=budget, discover=_discover),
budget,
)
base_regions, base_declarations, base_hit = _prepare(
lambda: discovery(old.root, old.manifest, cache=cache, identity=discovery_identity, budget=budget, discover=_discover),
budget,
)
timings["discovery_seconds"] = time.monotonic() - discovery_started
timings["discovery_hits"] = int(head_hit) + int(base_hit)
head_backend = base_backend = backend
if (
cache is not None
and cache_semantics is not None
and isinstance(backend, JoernBackend)
and all(m.complete for m in manifests)
):
head_backend = ReusingBackend(
backend,
cache,
cache_semantics,
declarations=declaration_identity(tip.manifest, head_declarations),
exclusions=provenance["graph_layout"],
)
base_backend = ReusingBackend(
backend,
cache,
cache_semantics,
declarations=declaration_identity(old.manifest, base_declarations),
exclusions=provenance["graph_layout"],
)
reused.extend((head_backend, base_backend))
context = adapter.compare(comparison, declaration_paths=tuple(set(head_declarations + base_declarations)), budget=budget)
context = replace(context, unresolved_boundaries=tuple(dict.fromkeys((*context.unresolved_boundaries, *reasons))))
reasons.extend(context.unresolved_boundaries)
@@ -197,7 +276,7 @@ def replay(
head_scan = scan_repository(
tip.root,
head_regions,
backend=backend,
backend=head_backend,
budget=scan_budget,
execution_budget=budget,
ranking_mode="evidence",
@@ -214,11 +293,13 @@ def replay(
head_regions=head_regions,
base_regions=base_regions,
context=context,
backend=backend,
backend=base_backend,
execution_budget=budget,
scan_budget=scan_budget,
head_semantics=provenance["semantics"],
base_semantics=provenance["semantics"],
cache=cache,
cache_semantics=cache_semantics,
)
if delta.base_scan is not None:
records.append(ScanRecord(comparison, "base", delta.base_scan))
@@ -251,6 +332,8 @@ def replay(
timings[stage + "_seconds"] = time.monotonic() - stage_started
if time.monotonic() >= budget.deadline_monotonic:
reasons.append("deadline_exhausted")
timings["graph_hits"] = sum(item.graph_hits for item in reused)
timings["query_hits"] = sum(item.query_hits for item in reused)
head_records = [r.scan for r in records if r.side == "head"]
for record in records:
if record.scan.scope is not None:
tests/test_graph_reuse.py
+18
View File
@@ -88,3 +88,21 @@ def test_generic_driver_accepts_backend_validated_lease(tmp_path, monkeypatch):
assert result.run("census", {})["file_names"] == ["a.js"]
_dispose(result)
assert graph.read_bytes() == b"graph original"
def test_installed_frontend_adaptation_changes_graph_identity(tmp_path, monkeypatch):
import openultrasast.cpg.backend as module
backend, root, _, identity, _ = prepared(tmp_path, monkeypatch)
home = tmp_path / "engine"
home.mkdir()
launcher = home / "joern"
launcher.write_text("launcher")
jar = home / "frontends/jssrc2cpg/lib/io.joern.jssrc2cpg-4.0.625.jar"
jar.parent.mkdir(parents=True)
jar.write_bytes(b"frontend-one")
monkeypatch.setattr(module.shutil, "which", lambda name: str(launcher) if name == "joern" else None)
first = backend.graph_identity(root, language="javascript", declarations_digest="d", exclusions_digest="e")
jar.write_bytes(b"frontend-two")
second = backend.graph_identity(root, language="javascript", declarations_digest="d", exclusions_digest="e")
assert first != second
tests/test_push_reuse.py
+89
View File
@@ -0,0 +1,89 @@
"""Live integration of validated graph and query reuse under one deadline."""
import json
import time
from openultrasast.cpg.backend import CpgResult, JoernBackend
from openultrasast.model.contracts import ExecutionBudget
from openultrasast.push.cache import ArtifactCache, SemanticKeys
from openultrasast.push.reuse import ReusingBackend
def test_backend_reuses_graph_and_answers_then_invalidates_source_and_dependencies(tmp_path, monkeypatch):
import openultrasast.cpg.backend as module
monkeypatch.setattr(module, "joern_version", lambda: (4, 0, 625))
builds = []
queries = []
class Engine(JoernBackend):
def build(self, root, *, language, exclude, execution_budget):
builds.append(root)
path = tmp_path / f"graph-{len(builds)}.json"
paths = [str(p) for p in root.glob("*.js")]
path.write_text(json.dumps(paths))
return CpgResult(path, lambda kind, params: self.query(path, kind, params), lambda kind, req: self.query_batch(path, kind, req))
def query(self, path, kind, params):
names = json.loads(path.read_text())
return {"files": len(names), "methods": 2, "calls": 1, "file_names": names, "overlays": "dataflowOss"}
def query_batch(self, path, kind, requests):
queries.append((kind, requests))
return {**{rid: [] for rid in requests}, "__census__": [self.query(path, "census", {})]}
root = tmp_path / "source"
root.mkdir()
(root / "a.js").write_text("function f() {}")
cache = ArtifactCache(tmp_path / "cache", max_bytes=100000)
semantics = SemanticKeys("facts", "queries", "config", "ranking", "admission", "disabled", "empty", "advisory")
deadline = ExecutionBudget(time.monotonic() + 10, 0.2)
backend = ReusingBackend(Engine(), cache, semantics, declarations="dependencies", exclusions="exclusions")
def run():
result = backend.build(root, language="javascript", exclude=(), execution_budget=deadline)
assert result is not None
answer = result.run_batch("taint", {"q": {"file": "a.js", "callDepth": 3}})
if result.cleanup:
result.cleanup()
return answer
expected = run()
assert run() == expected
assert len(builds) == 1 and len(queries) == 1
(root / "a.js").write_text("function g() {}")
run()
assert len(builds) == 2 and len(queries) == 2
backend.declarations = "changed-dependency"
run()
assert len(builds) == 3 and len(queries) == 3
backend.semantics = SemanticKeys("changed-facts", "queries", "config", "ranking", "admission", "disabled", "empty", "advisory")
run()
assert len(builds) == 3 and len(queries) == 4
def test_fresh_missing_census_cannot_borrow_a_cached_query_census(tmp_path):
from openultrasast.cpg.artifact import GraphArtifact, GraphCensus, GraphIdentity
identity = GraphIdentity(
"source", "declarations", "exclude", "repository", "javascript", "engine", "frontend", "version", ("dataflowOss",), ()
)
artifact = GraphArtifact(identity, "graph-bytes", 10, GraphCensus(1, 1, 1), "complete", (), "/old", ("a.js",))
cache = ArtifactCache(tmp_path / "cache", max_bytes=100000)
semantics = SemanticKeys("facts", "queries", "config", "ranking", "admission", "disabled", "empty", "advisory")
wrapper = ReusingBackend(JoernBackend(), cache, semantics, declarations="d", exclusions="e")
census = [{"files": 1, "methods": 1, "file_names": ["a.js"]}]
calls = []
def batch(kind, requests):
calls.append(requests)
result = {rid: [] for rid in requests}
if len(calls) == 1:
result["__census__"] = census
return result
graph = CpgResult(tmp_path / "graph", lambda *_: None, batch)
result = wrapper._answers(graph, artifact, ExecutionBudget(time.monotonic() + 5, 0.2))
assert "q" in result.run_batch("taint", {"q": {"file": "a.js"}})
answer = result.run_batch("taint", {"q": {"file": "a.js"}, "fresh": {"file": "a.js", "callDepth": 3}})
assert "q" in answer and "fresh" not in answer
tests/test_push_runner.py
+81
View File
@@ -218,3 +218,84 @@ def test_default_replay_does_not_launch_unbudgeted_availability_probe(tmp_path,
monkeypatch.setattr(runner, "JoernBackend", Backend, raising=False)
result = replay(root, base=base, head=head, artifact=tmp_path / "probe.json")
assert builds and result.result.coverage_status == "incomplete"
def test_replay_persists_discovery_without_skipping_readable_snapshots(tmp_path, monkeypatch):
from openultrasast.push import runner
root, base, head, _ = history(tmp_path)
cache = tmp_path / "cache"
cold = tmp_path / "cold.json"
warm = tmp_path / "warm.json"
replay(root, base=base, head=head, artifact=cold, backend=NullBackend(), cache_dir=cache)
def should_reuse(*args):
raise AssertionError("discovery should be cached")
monkeypatch.setattr(runner, "_discover", should_reuse)
replay(root, base=base, head=head, artifact=warm, backend=NullBackend(), cache_dir=cache)
a, b = json.loads(cold.read_text()), json.loads(warm.read_text())
assert a["result"] == b["result"]
assert a["snapshots"] == b["snapshots"]
assert b["timings"]["discovery_hits"] == 2
assert b["snapshots"][0]["bytes_read"] > 0
def test_cache_cannot_write_inside_repository(tmp_path):
root, base, head, git = history(tmp_path)
before = git("status", "--porcelain")
with pytest.raises(ValueError, match="cache.*outside"):
replay(root, base=base, head=head, artifact=tmp_path / "out.json", cache_dir=root / "cache")
assert git("status", "--porcelain") == before
def test_cached_discovery_invalidates_source_and_dependency_edits(tmp_path):
root, base, head, git = history(tmp_path)
cache = tmp_path / "cache"
replay(root, base=base, head=head, artifact=tmp_path / "initial.json", backend=NullBackend(), cache_dir=cache)
(root / "api.js").write_text("function replacement(req) { eval(req.body.other); }")
git("add", "api.js")
git("commit", "-m", "change source")
edited = git("rev-parse", "HEAD")
replay(root, base=base, head=edited, artifact=tmp_path / "edited.json", backend=NullBackend(), cache_dir=cache)
data = json.loads((tmp_path / "edited.json").read_text())
assert data["timings"]["discovery_hits"] == 1
(root / "package.json").write_text('{"dependencies":{"express":"5"}}')
git("add", "package.json")
git("commit", "-m", "change dependencies")
dependency = git("rev-parse", "HEAD")
replay(root, base=base, head=dependency, artifact=tmp_path / "dependency.json", backend=NullBackend(), cache_dir=cache)
data = json.loads((tmp_path / "dependency.json").read_text())
assert data["timings"]["discovery_hits"] == 1
assert data["change_context"]["head_revision"] == dependency
def test_identical_revision_materializes_once_and_reuses_discovery(tmp_path, monkeypatch):
from openultrasast.push.snapshot import SnapshotAdapter
root, base, _, _ = history(tmp_path)
original = SnapshotAdapter.materialize
calls = []
def counted(self, oid, **kwargs):
calls.append(oid)
return original(self, oid, **kwargs)
monkeypatch.setattr(SnapshotAdapter, "materialize", counted)
replay(root, base=base, head=base, artifact=tmp_path / "same.json", backend=NullBackend(), cache_dir=tmp_path / "cache")
data = json.loads((tmp_path / "same.json").read_text())
assert calls == [base]
assert data["timings"]["discovery_hits"] == 1
assert len(data["snapshots"]) == 2
def test_admission_provenance_invalidates_when_installed_runtime_changes(tmp_path, monkeypatch):
from openultrasast.model.scan import ScanBudget
from openultrasast.push import runner
monkeypatch.setattr(runner, "engine_runtime_identity", lambda: "first-installation")
first = runner._provenance(tmp_path, ScanBudget())
monkeypatch.setattr(runner, "engine_runtime_identity", lambda: "second-installation")
second = runner._provenance(tmp_path, ScanBudget())
assert first["semantics"] != second["semantics"]
assert first["facts"] == second["facts"] and first["graph_layout"] == second["graph_layout"]