mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
feat(pre-push-safety-net): compare targeted base and head evidence
This commit is contained in:
@@ -0,0 +1,41 @@
|
|||||||
|
# Group 4 implementation and verification
|
||||||
|
|
||||||
|
Date: 2026-09-13. The maintainer authorized group 4 as a whole. Scope is tasks
|
||||||
|
4.1–4.3: targeted novelty comparison, actionability admission and compact reporting.
|
||||||
|
Groups 1–3 are verified at `6ba2b06`; full transaction replay remains task 5.1.
|
||||||
|
|
||||||
|
Status: implementation in progress. Task completion requires independent review
|
||||||
|
and fresh evidence. No production capability is enabled for normal hook alerts;
|
||||||
|
capability admission still requires task 8.3 evaluation.
|
||||||
|
|
||||||
|
## Verification boundary
|
||||||
|
|
||||||
|
Exercise production policy functions with immutable Git snapshots and actual engine
|
||||||
|
answers, plus deterministic error-path and equivalent-origin controls. Prove source
|
||||||
|
bytes were read before interpreting empty answers. Synthetic evaluated capabilities
|
||||||
|
exercise admission mechanics only and never establish real capability eligibility.
|
||||||
|
The established gates protect detector regressions; these controls do not establish
|
||||||
|
production hook latency, broad alert precision or feature-level readiness.
|
||||||
|
|
||||||
|
## Task 4.1 verification
|
||||||
|
|
||||||
|
VERIFIED (TASK scope), independent review APPROVED. Full suite: 1,167 passed,
|
||||||
|
nine skipped in 122.44 seconds; global Ruff/format (233 files), mypy (110 source
|
||||||
|
files), all three regression gates and diff checks passed. The first OFF control
|
||||||
|
failed on novelty behavior. Review regressions reproduced false NEW classifications
|
||||||
|
from wrong-mechanism base rows and non-string operations; both now stay unknown.
|
||||||
|
|
||||||
|
`benchmarks/measurements/2026-09-13-delta-comparison-smoke.json` records actual
|
||||||
|
JavaScript guard removal as worsened, unchanged backlog moved by comments as
|
||||||
|
unchanged, and a PHP source newly connected to a mapped unchanged sink as new.
|
||||||
|
All inputs were read and both sides' questions completed. Current reviewed policy
|
||||||
|
was independently rechecked against captured actual answers after the schema-only
|
||||||
|
repair; the artifact preserves exact source hashes and that narrower provenance.
|
||||||
|
Concurrent lab costs were 214.81, 214.52 and 291.01 seconds respectively under
|
||||||
|
900-second budgets. These are not hook latency measurements.
|
||||||
|
|
||||||
|
Automatic dominance-only change-context projection remains an explicit upstream
|
||||||
|
limitation for the later replay integration. These controls use declared questions
|
||||||
|
and Git context directly; no missing-coverage result was erased. Ambiguous source
|
||||||
|
and identifier changes remain unknown. Task 4.1 brings progress to 12/27 tasks;
|
||||||
|
admission and rendering remain pending within this authorized group.
|
||||||
@@ -178,3 +178,22 @@ incomplete coverage. A synthetic graph file must never compensate for an omitted
|
|||||||
real source, and a wrapper must not discard evidence supplied by a frontend.
|
real source, and a wrapper must not discard evidence supplied by a frontend.
|
||||||
See `implementation-group-3.md` for final evidence. Eleven of 27 tasks are complete;
|
See `implementation-group-3.md` for final evidence. Eleven of 27 tasks are complete;
|
||||||
change attribution, actionability admission and compact output are next.
|
change attribution, actionability admission and compact output are next.
|
||||||
|
|
||||||
|
## Group 4 comparison controls (2026-09-13)
|
||||||
|
|
||||||
|
Real JavaScript dominance rows distinguish a guarded base operation from an
|
||||||
|
unguarded head operation while retaining a guarded sibling. The direct task 4.1
|
||||||
|
comparator can consume these answers with declared questions and immutable Git
|
||||||
|
span correspondence. This does not establish automatic context projection for
|
||||||
|
that family: the existing driver's change-context evidence pass currently projects
|
||||||
|
taint questions, and a dominance-only scan with `change_context` reports
|
||||||
|
`context_projection_unavailable:contributor-scan`. Preserve that limitation in the
|
||||||
|
full replay integration at 5.1; any extension belongs to the contributor-scan
|
||||||
|
context contract, not a hook-specific graph or detector workaround.
|
||||||
|
|
||||||
|
Duplicate identical source lines also remain ambiguous to the snapshot adapter.
|
||||||
|
Controls with unique lines prove the narrow supported correspondence path; they
|
||||||
|
do not establish general identity across arbitrary refactoring. Changed source
|
||||||
|
text alone is insufficient to distinguish a new source from a benign identifier
|
||||||
|
rename. Those comparisons remain unknown. A completed comparable base with no flow
|
||||||
|
to an unchanged mapped sink supplies a supported new-connection control instead.
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
{
|
{
|
||||||
"feature_name": "pre-push-safety-net",
|
"feature_name": "pre-push-safety-net",
|
||||||
"created_at": "2026-09-12T15:35:00Z",
|
"created_at": "2026-09-12T15:35:00Z",
|
||||||
"updated_at": "2026-09-13T09:50:01.821464+00:00",
|
"updated_at": "2026-09-13T10:26:47.307154+00:00",
|
||||||
"language": "en",
|
"language": "en",
|
||||||
"phase": "implementation",
|
"phase": "implementation",
|
||||||
"approvals": {
|
"approvals": {
|
||||||
@@ -19,7 +19,7 @@
|
|||||||
}
|
}
|
||||||
},
|
},
|
||||||
"ready_for_implementation": true,
|
"ready_for_implementation": true,
|
||||||
"implementation_approval_note": "Groups 1\u20133 verified complete. Contributor-scan frontend-retention prerequisite 2.18 approved and verified. Remaining approved tasks require independent review and fresh verification.",
|
"implementation_approval_note": "Groups 1\u20133 verified complete. Maintainer explicitly authorized group 4 as a whole; each task requires independent review and fresh verification.",
|
||||||
"depends_on": [
|
"depends_on": [
|
||||||
"contributor-scan"
|
"contributor-scan"
|
||||||
],
|
],
|
||||||
@@ -55,6 +55,11 @@
|
|||||||
"at": "2026-09-13T09:17:01.788332+00:00",
|
"at": "2026-09-13T09:17:01.788332+00:00",
|
||||||
"source": "approved",
|
"source": "approved",
|
||||||
"action": "Approved contributor-scan frontend retention prerequisite; implement it and resume task 3.4 review and verification."
|
"action": "Approved contributor-scan frontend retention prerequisite; implement it and resume task 3.4 review and verification."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"at": "2026-09-13T10:26:47.307154+00:00",
|
||||||
|
"source": "move on to group 4 as a whole",
|
||||||
|
"action": "Authorized all of group 4 (4.1\u20134.3), sequential implementation, independent task review and fresh verification. Groups 5\u20138 remain pending."
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"task_review": {
|
"task_review": {
|
||||||
@@ -78,14 +83,16 @@
|
|||||||
"3.1",
|
"3.1",
|
||||||
"3.2",
|
"3.2",
|
||||||
"3.3",
|
"3.3",
|
||||||
"3.4"
|
"3.4",
|
||||||
|
"4.1"
|
||||||
],
|
],
|
||||||
"remaining_tasks": 16,
|
"remaining_tasks": 15,
|
||||||
"verification_record": "implementation-group-3.md",
|
"verification_record": "implementation-group-4.md",
|
||||||
"verification_records": [
|
"verification_records": [
|
||||||
"implementation-group-1.md",
|
"implementation-group-1.md",
|
||||||
"implementation-group-2.md",
|
"implementation-group-2.md",
|
||||||
"implementation-group-3.md"
|
"implementation-group-3.md",
|
||||||
|
"implementation-group-4.md"
|
||||||
],
|
],
|
||||||
"next_group": 4,
|
"next_group": 4,
|
||||||
"blocked_tasks": []
|
"blocked_tasks": []
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
# Implementation Plan: pre-push-safety-net
|
# Implementation Plan: pre-push-safety-net
|
||||||
|
|
||||||
Status: task plan approved by the maintainer's "approve." on 2026-09-12. Groups 1–3 (tasks 1.1–3.4)
|
Status: task plan approved by the maintainer's "approve." on 2026-09-12. Groups 1–3 (tasks 1.1–3.4)
|
||||||
are complete after independent review and fresh verification (11/27 subtasks). Groups 4–8 remain pending;
|
are complete after independent review and fresh verification (12/27 subtasks, including task 4.1). Tasks 4.2–4.3 and groups 5–8 remain pending;
|
||||||
the hook and capability admission are not yet implemented. See `implementation-group-1.md` and `implementation-group-2.md`.
|
the hook and capability admission are not yet implemented. See `implementation-group-1.md` and `implementation-group-2.md`.
|
||||||
|
|
||||||
## Execution contract
|
## Execution contract
|
||||||
@@ -113,10 +113,11 @@ the hook and capability admission are not yet implemented. See `implementation-g
|
|||||||
- _Verified 2026-09-13:_ independent review APPROVED; 1,151 passed, nine skipped; Ruff/format/mypy pass. Real PHP/JavaScript graphs retain tests/browser source, omit vendor nodes and share one ranker limit. Named census survives PHP aggregation; missing source names and unreadable discovery inputs remain explicit. Python exercises all three batch census schemas. Evidence: `benchmarks/measurements/2026-09-13-partition-scope-smoke.json`. No hook capability or latency claim.
|
- _Verified 2026-09-13:_ independent review APPROVED; 1,151 passed, nine skipped; Ruff/format/mypy pass. Real PHP/JavaScript graphs retain tests/browser source, omit vendor nodes and share one ranker limit. Named census survives PHP aggregation; missing source names and unreadable discovery inputs remain explicit. Python exercises all three batch census schemas. Evidence: `benchmarks/measurements/2026-09-13-partition-scope-smoke.json`. No hook capability or latency claim.
|
||||||
|
|
||||||
- [ ] 4. Implement change attribution and actionable output
|
- [ ] 4. Implement change attribution and actionable output
|
||||||
- [ ] 4.1 Compare targeted base/head evidence for defect novelty
|
- [x] 4.1 Compare targeted base/head evidence for defect novelty
|
||||||
- Establish stable mechanism/operation identity with rename/span mapping and collect comparable base evidence under the same semantics and remaining deadline.
|
- Establish stable mechanism/operation identity with rename/span mapping and collect comparable base evidence under the same semantics and remaining deadline.
|
||||||
- Distinguish new, worsened, unchanged and unknown; an absent finding from a failed/truncated base is unknown. Support removed guards and new source connections at unchanged sinks.
|
- Distinguish new, worsened, unchanged and unknown; an absent finding from a failed/truncated base is unknown. Support removed guards and new source connections at unchanged sinks.
|
||||||
- Done when benign movement and unchanged backlog do not become new defects, while a supported removed-guard case has a concrete change-to-witness relationship.
|
- Done when benign movement and unchanged backlog do not become new defects, while a supported removed-guard case has a concrete change-to-witness relationship.
|
||||||
|
- _Verified 2026-09-13:_ independent review APPROVED; 1,167 tests passed, nine skipped; real JavaScript guard/movement and PHP connection controls passed. Final policy rechecked against captured actual answers after malformed-schema repair. See `implementation-group-4.md`.
|
||||||
- _Boundary: Delta and Actionability Policy, engine comparison integration_
|
- _Boundary: Delta and Actionability Policy, engine comparison integration_
|
||||||
- _Depends: 2.3, 3.3, 3.4_
|
- _Depends: 2.3, 3.3, 3.4_
|
||||||
- _Requirements: 2.4, 3.1, 3.4, 4.1_
|
- _Requirements: 2.4, 3.1, 3.4, 4.1_
|
||||||
|
|||||||
@@ -1,10 +1,10 @@
|
|||||||
{
|
{
|
||||||
"counts": {
|
"counts": {
|
||||||
"load_bearing": 99,
|
"load_bearing": 100,
|
||||||
"standalone_capability": 9
|
"standalone_capability": 9
|
||||||
},
|
},
|
||||||
"modules": 108,
|
"modules": 109,
|
||||||
"note": "Refreshed after pre-push-safety-net task 3.4: first-party frontend partitions share the authoritative ranker and execution budget.",
|
"note": "Refreshed after pre-push-safety-net task 4.1: targeted base comparison reuses existing arbiters and shared execution budget; replay integration remains task 5.1.",
|
||||||
"orphaned": [],
|
"orphaned": [],
|
||||||
"rows": [
|
"rows": [
|
||||||
{
|
{
|
||||||
@@ -116,7 +116,8 @@
|
|||||||
"config",
|
"config",
|
||||||
"cpg.artifact",
|
"cpg.artifact",
|
||||||
"model.contracts",
|
"model.contracts",
|
||||||
"push.contracts"
|
"push.contracts",
|
||||||
|
"push.policy"
|
||||||
],
|
],
|
||||||
"module": "contracts",
|
"module": "contracts",
|
||||||
"reason": "pre-push-safety-net Req 7.3: strict shared manifest serialization, independent of push"
|
"reason": "pre-push-safety-net Req 7.3: strict shared manifest serialization, independent of push"
|
||||||
@@ -132,7 +133,8 @@
|
|||||||
"model.partitions",
|
"model.partitions",
|
||||||
"model.pipeline",
|
"model.pipeline",
|
||||||
"model.scan",
|
"model.scan",
|
||||||
"model.taint"
|
"model.taint",
|
||||||
|
"push.policy"
|
||||||
],
|
],
|
||||||
"module": "cpg",
|
"module": "cpg",
|
||||||
"reason": "model-grounded-detection Req 4: the CPG seam package"
|
"reason": "model-grounded-detection Req 4: the CPG seam package"
|
||||||
@@ -153,7 +155,8 @@
|
|||||||
"model.partitions",
|
"model.partitions",
|
||||||
"model.pipeline",
|
"model.pipeline",
|
||||||
"model.scan",
|
"model.scan",
|
||||||
"model.taint"
|
"model.taint",
|
||||||
|
"push.policy"
|
||||||
],
|
],
|
||||||
"module": "cpg.backend",
|
"module": "cpg.backend",
|
||||||
"reason": "model-grounded-detection Req 4.1/4.5: the single Joern subprocess boundary"
|
"reason": "model-grounded-detection Req 4.1/4.5: the single Joern subprocess boundary"
|
||||||
@@ -356,6 +359,7 @@
|
|||||||
"model.taint",
|
"model.taint",
|
||||||
"preprocess",
|
"preprocess",
|
||||||
"push.contracts",
|
"push.contracts",
|
||||||
|
"push.policy",
|
||||||
"push.snapshot"
|
"push.snapshot"
|
||||||
],
|
],
|
||||||
"module": "model",
|
"module": "model",
|
||||||
@@ -399,6 +403,7 @@
|
|||||||
"model.regions",
|
"model.regions",
|
||||||
"model.scan",
|
"model.scan",
|
||||||
"push.contracts",
|
"push.contracts",
|
||||||
|
"push.policy",
|
||||||
"push.snapshot"
|
"push.snapshot"
|
||||||
],
|
],
|
||||||
"module": "model.contracts",
|
"module": "model.contracts",
|
||||||
@@ -450,7 +455,8 @@
|
|||||||
"model.judge",
|
"model.judge",
|
||||||
"model.pipeline",
|
"model.pipeline",
|
||||||
"model.scan",
|
"model.scan",
|
||||||
"model.taint"
|
"model.taint",
|
||||||
|
"push.policy"
|
||||||
],
|
],
|
||||||
"module": "model.ladder",
|
"module": "model.ladder",
|
||||||
"reason": "model-grounded-detection Req 5: the evidence ladder every finding carries"
|
"reason": "model-grounded-detection Req 5: the evidence ladder every finding carries"
|
||||||
@@ -476,7 +482,8 @@
|
|||||||
{
|
{
|
||||||
"classification": "load_bearing",
|
"classification": "load_bearing",
|
||||||
"importers": [
|
"importers": [
|
||||||
"model.scan"
|
"model.scan",
|
||||||
|
"push.policy"
|
||||||
],
|
],
|
||||||
"module": "model.pipeline",
|
"module": "model.pipeline",
|
||||||
"reason": "model-grounded-detection Req 8: the enumerator proposes, the LLM answers, the model disposes"
|
"reason": "model-grounded-detection Req 8: the enumerator proposes, the LLM answers, the model disposes"
|
||||||
@@ -486,7 +493,8 @@
|
|||||||
"importers": [
|
"importers": [
|
||||||
"cli",
|
"cli",
|
||||||
"model.layout",
|
"model.layout",
|
||||||
"model.scan"
|
"model.scan",
|
||||||
|
"push.policy"
|
||||||
],
|
],
|
||||||
"module": "model.regions",
|
"module": "model.regions",
|
||||||
"reason": "contributor-scan Req 2: regions from entry points, with the families each admits"
|
"reason": "contributor-scan Req 2: regions from entry points, with the families each admits"
|
||||||
@@ -502,7 +510,8 @@
|
|||||||
{
|
{
|
||||||
"classification": "load_bearing",
|
"classification": "load_bearing",
|
||||||
"importers": [
|
"importers": [
|
||||||
"cli"
|
"cli",
|
||||||
|
"push.policy"
|
||||||
],
|
],
|
||||||
"module": "model.scan",
|
"module": "model.scan",
|
||||||
"reason": "contributor-scan Req 3: the repository driver -- one CPG, one budget, ranked spend"
|
"reason": "contributor-scan Req 3: the repository driver -- one CPG, one budget, ranked spend"
|
||||||
@@ -667,6 +676,12 @@
|
|||||||
"module": "push.contracts",
|
"module": "push.contracts",
|
||||||
"reason": "pre-push-safety-net Req 5.5/7.3: immutable comparisons and independent result statuses; runner pending"
|
"reason": "pre-push-safety-net Req 5.5/7.3: immutable comparisons and independent result statuses; runner pending"
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
"classification": "load_bearing",
|
||||||
|
"importers": [],
|
||||||
|
"module": "push.policy",
|
||||||
|
"reason": "pre-push-safety-net Req 2.4/3.1/3.4/4.1: targeted same-semantics base comparison; replay integration pending"
|
||||||
|
},
|
||||||
{
|
{
|
||||||
"classification": "load_bearing",
|
"classification": "load_bearing",
|
||||||
"importers": [],
|
"importers": [],
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
@@ -197,3 +197,20 @@ unshipped test-path records. Runtime origin remains unspecified without declarat
|
|||||||
readable Python graph and exercises the production taint, dominance and configuration
|
readable Python graph and exercises the production taint, dominance and configuration
|
||||||
batch queries. Each must return the actual source filename in its census. This
|
batch queries. Each must return the actual source filename in its census. This
|
||||||
checks query compilation and census transport, not security-property coverage.
|
checks query compilation and census transport, not security-property coverage.
|
||||||
|
|
||||||
|
## Targeted novelty comparison smoke
|
||||||
|
|
||||||
|
After rebuilding the image, run a declared JavaScript guard-removal comparison:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker run --rm -i --network none --entrypoint python openultrasast:dev - --case guard < ops/smoke_delta.py
|
||||||
|
```
|
||||||
|
|
||||||
|
Use `--case movement` for unchanged JavaScript backlog shifted by comments, or
|
||||||
|
`--case connection` for a PHP input newly reaching an unchanged operation. Every
|
||||||
|
case verifies immutable base/head bytes, runs the existing driver and targeted
|
||||||
|
comparison under one 900-second lab budget, and checks that dirty checkout/index
|
||||||
|
state survives. Output contains an evidence JSON object followed by a completion
|
||||||
|
JSON object; success requires exit zero and `verified: true` in that final object.
|
||||||
|
These declared regions exercise comparison mechanics; full replay, normal alert
|
||||||
|
admission and hook latency are separate evaluation work.
|
||||||
|
|||||||
@@ -0,0 +1,138 @@
|
|||||||
|
"""Real Git/Joern novelty controls; no hook admission or latency claim."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import subprocess
|
||||||
|
import tempfile
|
||||||
|
import time
|
||||||
|
from dataclasses import asdict
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from openultrasast.cpg.backend import JoernBackend
|
||||||
|
from openultrasast.model.contracts import ExecutionBudget
|
||||||
|
from openultrasast.model.regions import ScanRegion
|
||||||
|
from openultrasast.model.scan import ScanBudget, scan_repository
|
||||||
|
from openultrasast.push.policy import compare_targeted_base, semantics_digest
|
||||||
|
from openultrasast.push.snapshot import SnapshotAdapter
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument("--case", choices=("guard", "movement", "connection"), required=True)
|
||||||
|
args = parser.parse_args()
|
||||||
|
before = (
|
||||||
|
b"export function handler(req) {\n"
|
||||||
|
b" if (!authenticate(req)) return;\n"
|
||||||
|
b" deleteUser(req.params.id);\n} // handler\n"
|
||||||
|
b"export function sibling(other) {\n"
|
||||||
|
b" if (!authenticate(other)) return;\n"
|
||||||
|
b" deleteUser(other.params.id);\n} // sibling\n"
|
||||||
|
)
|
||||||
|
after = before.replace(b" if (!authenticate(req)) return;\n", b"", 1)
|
||||||
|
language, family, filename = "javascript", "access_control", "api.js"
|
||||||
|
expected = "worsened"
|
||||||
|
if args.case == "movement":
|
||||||
|
before = after
|
||||||
|
after = b"// documentation moved this existing operation\n\n" + before
|
||||||
|
expected = "unchanged"
|
||||||
|
elif args.case == "connection":
|
||||||
|
language, family, filename = "php", "injection", "api.php"
|
||||||
|
before = b'<?php\nfunction handler($req) {\n $value = "fixed";\n eval($value);\n}\n'
|
||||||
|
after = before.replace(b'$value = "fixed";', b"$value = $req;")
|
||||||
|
expected = "new"
|
||||||
|
started = time.monotonic()
|
||||||
|
budget = ExecutionBudget(started + 900, 2.0)
|
||||||
|
report: dict[str, object] = {
|
||||||
|
"case": args.case,
|
||||||
|
"purpose": (
|
||||||
|
"Real targeted comparison with declared regions and Git context; no automatic context projection, full replay or admission"
|
||||||
|
),
|
||||||
|
"shared_budget_seconds": 900,
|
||||||
|
}
|
||||||
|
with tempfile.TemporaryDirectory(prefix="ousast-delta-smoke-") as temporary:
|
||||||
|
repo = Path(temporary)
|
||||||
|
|
||||||
|
def git(*parts: str) -> bytes:
|
||||||
|
return subprocess.run(["git", "-C", str(repo), *parts], check=True, capture_output=True, timeout=10).stdout
|
||||||
|
|
||||||
|
git("init")
|
||||||
|
git("config", "user.name", "Delta smoke")
|
||||||
|
git("config", "user.email", "[email protected]")
|
||||||
|
git("config", "core.hooksPath", os.devnull)
|
||||||
|
source = repo / filename
|
||||||
|
source.write_bytes(before)
|
||||||
|
git("add", filename)
|
||||||
|
git("commit", "-m", "base")
|
||||||
|
base_oid = git("rev-parse", "HEAD").decode().strip()
|
||||||
|
source.write_bytes(after)
|
||||||
|
git("add", filename)
|
||||||
|
git("commit", "-m", args.case)
|
||||||
|
head_oid = git("rev-parse", "HEAD").decode().strip()
|
||||||
|
source.write_bytes(b"uncommitted content\n")
|
||||||
|
index = (repo / ".git/index").read_bytes()
|
||||||
|
status = git("status", "--porcelain=v1", "-z")
|
||||||
|
adapter = SnapshotAdapter(repo)
|
||||||
|
(comparison,) = adapter.resolve_updates(f"refs/heads/topic {head_oid} refs/heads/topic {base_oid}\n").comparisons
|
||||||
|
context = adapter.compare(comparison, declaration_paths=(), budget=budget)
|
||||||
|
regions = (ScanRegion(filename, "handler", language, (family,), 1.0, "entry_point"),)
|
||||||
|
backend = JoernBackend(heap_mb=1024)
|
||||||
|
semantics = semantics_digest(engine_identity="joern-4.0.625/frontend-retention-v1", options={"max_regions": 1})
|
||||||
|
with adapter.materialize(base_oid, budget=budget) as base_snapshot, adapter.materialize(head_oid, budget=budget) as head_snapshot:
|
||||||
|
assert (base_snapshot.root / filename).read_bytes() == before and before
|
||||||
|
assert (head_snapshot.root / filename).read_bytes() == after and after
|
||||||
|
head = scan_repository(
|
||||||
|
head_snapshot.root,
|
||||||
|
regions,
|
||||||
|
backend=backend,
|
||||||
|
budget=ScanBudget(max_regions=1),
|
||||||
|
execution_budget=budget,
|
||||||
|
ranking_mode="evidence",
|
||||||
|
unit="delta-smoke",
|
||||||
|
population_complete=True,
|
||||||
|
)
|
||||||
|
assert head.findings and head.scope and head.question_outcomes, head
|
||||||
|
assert not head.degradations and all(q.status == "completed" for q in head.question_outcomes), head
|
||||||
|
delta = compare_targeted_base(
|
||||||
|
base_snapshot.root,
|
||||||
|
head=head,
|
||||||
|
head_regions=regions,
|
||||||
|
base_regions=regions,
|
||||||
|
context=context,
|
||||||
|
backend=backend,
|
||||||
|
execution_budget=budget,
|
||||||
|
scan_budget=ScanBudget(max_regions=1),
|
||||||
|
head_semantics=semantics,
|
||||||
|
base_semantics=semantics,
|
||||||
|
)
|
||||||
|
report.update(
|
||||||
|
{
|
||||||
|
"comparison": comparison.to_payload(),
|
||||||
|
"context": context.to_payload(),
|
||||||
|
"base_input": {"bytes": len(before), "sha256": hashlib.sha256(before).hexdigest()},
|
||||||
|
"head_input": {"bytes": len(after), "sha256": hashlib.sha256(after).hexdigest()},
|
||||||
|
"head": asdict(head),
|
||||||
|
"delta": asdict(delta),
|
||||||
|
"expected_novelty": expected,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
# Emit evidence before assertions so a failed instrument cannot look like a clean zero.
|
||||||
|
print(json.dumps(report, indent=2), flush=True)
|
||||||
|
assert delta.base_scan and delta.base_scan.question_outcomes, delta
|
||||||
|
assert not delta.base_scan.degradations, delta
|
||||||
|
assert all(q.status == "completed" for q in delta.base_scan.question_outcomes), delta
|
||||||
|
assert delta.candidates and all(c.novelty == expected for c in delta.candidates), delta
|
||||||
|
assert not delta.coverage_reasons, delta
|
||||||
|
assert not base_snapshot.root.exists() and not head_snapshot.root.exists()
|
||||||
|
assert source.read_bytes() == b"uncommitted content\n"
|
||||||
|
assert (repo / ".git/index").read_bytes() == index
|
||||||
|
assert git("status", "--porcelain=v1", "-z") == status
|
||||||
|
# A separate completion record distinguishes all assertions passing from partial JSON.
|
||||||
|
print(json.dumps({"verified": True, "case": args.case, "elapsed_seconds": round(time.monotonic() - started, 2)}))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -28,6 +28,7 @@ ROOTS = ("gate", "map_gate", "pair_gate", "cli", "__main__")
|
|||||||
|
|
||||||
# Modules that stand alone: their own CLI subcommand or external protocol, deliberately off the scan path.
|
# Modules that stand alone: their own CLI subcommand or external protocol, deliberately off the scan path.
|
||||||
SPEC_OWNED = {
|
SPEC_OWNED = {
|
||||||
|
"push.policy": "pre-push-safety-net Req 2.4/3.1/3.4/4.1: targeted same-semantics base comparison; replay integration pending",
|
||||||
"push.snapshot": "pre-push-safety-net Req 1.1/1.3/1.4/7.1: local immutable ref resolution; runner integration pending",
|
"push.snapshot": "pre-push-safety-net Req 1.1/1.3/1.4/7.1: local immutable ref resolution; runner integration pending",
|
||||||
"contracts": "pre-push-safety-net Req 7.3: strict shared manifest serialization, independent of push",
|
"contracts": "pre-push-safety-net Req 7.3: strict shared manifest serialization, independent of push",
|
||||||
"model.contracts": "pre-push-safety-net Req 4.1/7.3: generic deadline, change and exact scope contracts; integration pending",
|
"model.contracts": "pre-push-safety-net Req 4.1/7.3: generic deadline, change and exact scope contracts; integration pending",
|
||||||
|
|||||||
@@ -0,0 +1,374 @@
|
|||||||
|
"""Compare arbiter-backed evidence from immutable revisions; this does not admit alerts."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import time
|
||||||
|
from collections.abc import Sequence
|
||||||
|
from dataclasses import dataclass, replace
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any, Literal
|
||||||
|
|
||||||
|
from openultrasast.contracts import Contract
|
||||||
|
from openultrasast.cpg.backend import CpgResult
|
||||||
|
from openultrasast.model.contracts import ChangeContext, ExecutionBudget, QuestionIdentity
|
||||||
|
from openultrasast.model.ladder import Rung
|
||||||
|
from openultrasast.model.pipeline import ModelFinding, _arbitrate_all
|
||||||
|
from openultrasast.model.regions import ScanRegion
|
||||||
|
from openultrasast.model.scan import ModelScanResult, ScanBudget, _prefetched, _spec_for, scan_repository
|
||||||
|
from openultrasast.model.specs import ConfigSpec, DominanceSpec, TaintSpec
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class EvidenceOperation(Contract):
|
||||||
|
question: QuestionIdentity
|
||||||
|
mechanism: Literal["taint", "dominance", "config"]
|
||||||
|
path: str
|
||||||
|
line: int
|
||||||
|
operation: str
|
||||||
|
source: str
|
||||||
|
discharged: bool
|
||||||
|
detail: str
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class CandidateDelta(Contract):
|
||||||
|
family: str
|
||||||
|
site: str
|
||||||
|
rung: str
|
||||||
|
witness: str | None
|
||||||
|
defect_id: str
|
||||||
|
novelty: Literal["new", "worsened", "unchanged", "unknown"]
|
||||||
|
reason: str
|
||||||
|
head_operation: EvidenceOperation | None
|
||||||
|
base_operations: tuple[EvidenceOperation, ...]
|
||||||
|
change_evidence: tuple[str, ...]
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class DeltaComparison:
|
||||||
|
candidates: tuple[CandidateDelta, ...]
|
||||||
|
base_scan: ModelScanResult | None
|
||||||
|
semantics: str
|
||||||
|
coverage_reasons: tuple[str, ...]
|
||||||
|
|
||||||
|
|
||||||
|
def semantics_digest(*, engine_identity: str, options: dict[str, object]) -> str:
|
||||||
|
"""Fingerprint shipped semantics plus caller-owned engine/options provenance.
|
||||||
|
|
||||||
|
Include frontend adaptation, declaration/exclusion policy and scan options in options.
|
||||||
|
This is an identity, never evidence of engine liveness or input readability.
|
||||||
|
"""
|
||||||
|
root = Path(__file__).resolve().parents[1]
|
||||||
|
files = sorted([*(root / "cpg" / "queries").glob("*.sc"), *(root / "ruleset").rglob("*.toml"), *(root / "model").glob("*.py")])
|
||||||
|
payload = {
|
||||||
|
"engine": engine_identity,
|
||||||
|
"options": options,
|
||||||
|
"files": [(str(p.relative_to(root)), hashlib.sha256(p.read_bytes()).hexdigest()) for p in files],
|
||||||
|
}
|
||||||
|
return hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def _operations(scan: ModelScanResult, budget: ExecutionBudget | None = None) -> tuple[EvidenceOperation, ...]:
|
||||||
|
result = []
|
||||||
|
for answer in scan.question_outcomes:
|
||||||
|
if budget is not None and time.monotonic() >= budget.deadline_monotonic:
|
||||||
|
break
|
||||||
|
if answer.status != "completed" or answer.raw_rows_json is None:
|
||||||
|
continue
|
||||||
|
spec = _spec_for(answer.identity.family, answer.identity.language)
|
||||||
|
expected = (
|
||||||
|
"taint"
|
||||||
|
if isinstance(spec, TaintSpec)
|
||||||
|
else "dominance"
|
||||||
|
if isinstance(spec, DominanceSpec)
|
||||||
|
else "config"
|
||||||
|
if isinstance(spec, ConfigSpec)
|
||||||
|
else None
|
||||||
|
)
|
||||||
|
for row in json.loads(answer.raw_rows_json):
|
||||||
|
if budget is not None and time.monotonic() >= budget.deadline_monotonic:
|
||||||
|
break
|
||||||
|
kind: Literal["taint", "dominance", "config"]
|
||||||
|
if not isinstance(row, dict):
|
||||||
|
continue
|
||||||
|
if "sink" in row:
|
||||||
|
kind, path, line, op = "taint", row.get("sinkFile"), row.get("sinkLine"), row.get("sink")
|
||||||
|
if not isinstance(row.get("source"), str) or not row["source"].strip():
|
||||||
|
continue
|
||||||
|
if type(row.get("sanitized")) is not bool or type(row.get("bounded")) is not bool:
|
||||||
|
continue
|
||||||
|
source = json.dumps([row.get("sourceKind", "unknown"), row.get("source")], separators=(",", ":"))
|
||||||
|
discharged = row["sanitized"] or row["bounded"]
|
||||||
|
elif "operation" in row:
|
||||||
|
kind, path, line, op = "dominance", row.get("opFile"), row.get("opLine"), row.get("operation")
|
||||||
|
source = "obligation"
|
||||||
|
if not isinstance(row.get("dominatingGuards"), list) or not all(isinstance(g, str) for g in row["dominatingGuards"]):
|
||||||
|
continue
|
||||||
|
discharged = bool(row["dominatingGuards"])
|
||||||
|
elif "setting" in row:
|
||||||
|
kind, path, line, op = "config", row.get("file"), row.get("line"), row.get("setting")
|
||||||
|
source = "configuration"
|
||||||
|
discharged = False
|
||||||
|
else:
|
||||||
|
continue
|
||||||
|
if (
|
||||||
|
kind != expected
|
||||||
|
or not isinstance(path, str)
|
||||||
|
or not path
|
||||||
|
or not isinstance(op, str)
|
||||||
|
or not op.strip()
|
||||||
|
or not str(line).isdigit()
|
||||||
|
or int(str(line)) < 1
|
||||||
|
):
|
||||||
|
continue
|
||||||
|
result.append(
|
||||||
|
EvidenceOperation(answer.identity, kind, path, int(str(line)), str(op), source, discharged, json.dumps(row, sort_keys=True))
|
||||||
|
)
|
||||||
|
return tuple(result)
|
||||||
|
|
||||||
|
|
||||||
|
def _base_location(op: EvidenceOperation, context: ChangeContext) -> tuple[str, int] | None:
|
||||||
|
matches = []
|
||||||
|
for anchor in context.line_correspondences:
|
||||||
|
size = anchor.base_end_line - anchor.base_start_line
|
||||||
|
if context.decode_path(anchor.head_path) == op.path and anchor.head_start_line <= op.line <= anchor.head_start_line + size:
|
||||||
|
matches.append((context.decode_path(anchor.base_path), anchor.base_start_line + op.line - anchor.head_start_line))
|
||||||
|
if matches:
|
||||||
|
return matches[0] if len(set(matches)) == 1 else None
|
||||||
|
changed = {context.decode_path(p) for p in context.changed_paths}
|
||||||
|
if op.path not in changed:
|
||||||
|
return op.path, op.line
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _change_evidence(op: EvidenceOperation, context: ChangeContext) -> tuple[str, ...]:
|
||||||
|
paths = {op.path, op.question.path}
|
||||||
|
for relation in context.relationships:
|
||||||
|
if relation.target == op.question and relation.evidence:
|
||||||
|
paths.add(relation.source.path)
|
||||||
|
paths.update(context.decode_path(r.base_path) for r in context.renames if context.decode_path(r.head_path) in paths)
|
||||||
|
return tuple(
|
||||||
|
f"{span.side}:{context.decode_path(span.path)}:{span.start_line}-{span.end_line}"
|
||||||
|
for span in context.spans
|
||||||
|
if context.decode_path(span.path) in paths
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _candidate(
|
||||||
|
finding: ModelFinding,
|
||||||
|
op: EvidenceOperation | None,
|
||||||
|
bases: tuple[EvidenceOperation, ...],
|
||||||
|
novelty: Literal["new", "worsened", "unchanged", "unknown"],
|
||||||
|
reason: str,
|
||||||
|
change: tuple[str, ...],
|
||||||
|
context: ChangeContext,
|
||||||
|
) -> CandidateDelta:
|
||||||
|
location = _base_location(op, context) if op else None
|
||||||
|
identity = [
|
||||||
|
finding.family,
|
||||||
|
op.mechanism if op else "unknown",
|
||||||
|
location or ((op.path, op.line) if op else finding.site),
|
||||||
|
op.operation if op else finding.site,
|
||||||
|
op.source if op else "unknown",
|
||||||
|
op.question.function if op else None,
|
||||||
|
]
|
||||||
|
key = hashlib.sha256(json.dumps(identity, sort_keys=True).encode()).hexdigest()
|
||||||
|
return CandidateDelta(
|
||||||
|
finding.family, finding.site, finding.rung.value, finding.witness or None, key, novelty, reason, op, bases, change
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _valid_answer(scan: ModelScanResult, question: QuestionIdentity, operations: tuple[EvidenceOperation, ...]) -> bool:
|
||||||
|
answer = next((q for q in scan.question_outcomes if q.identity == question), None)
|
||||||
|
if answer is None or answer.raw_rows_json is None:
|
||||||
|
return False
|
||||||
|
rows = json.loads(answer.raw_rows_json)
|
||||||
|
# Query bodies are operation rows. An unrecognized/malformed row is not an empty answer.
|
||||||
|
return len(rows) == sum(op.question == question for op in operations)
|
||||||
|
|
||||||
|
|
||||||
|
def _context_boundaries(scan: ModelScanResult) -> tuple[str, ...]:
|
||||||
|
# A missing scheduling vector is not missing arbiter evidence after its question completed.
|
||||||
|
return tuple(b for b in scan.scope.unresolved_boundaries if not b.startswith("evidence_unknown:")) if scan.scope else ("scope_missing",)
|
||||||
|
|
||||||
|
|
||||||
|
def _counterpart(question: QuestionIdentity, context: ChangeContext) -> QuestionIdentity:
|
||||||
|
renamed = {context.decode_path(r.head_path): context.decode_path(r.base_path) for r in context.renames}
|
||||||
|
return replace(question, path=renamed.get(question.path, question.path))
|
||||||
|
|
||||||
|
|
||||||
|
def _proven_finding(finding: ModelFinding, op: EvidenceOperation, head: ModelScanResult) -> bool:
|
||||||
|
"""Match the existing arbiter's exact witness, not just a coincident line number."""
|
||||||
|
answer = next((q for q in head.question_outcomes if q.identity == op.question), None)
|
||||||
|
spec = _spec_for(op.question.family, op.question.language)
|
||||||
|
if answer is None or answer.raw_rows_json is None or spec is None:
|
||||||
|
return False
|
||||||
|
rows = json.loads(answer.raw_rows_json)
|
||||||
|
verdicts = _arbitrate_all(
|
||||||
|
CpgResult(Path("prefetched"), _prefetched(rows)),
|
||||||
|
spec,
|
||||||
|
path=op.question.path,
|
||||||
|
function=op.question.function or "",
|
||||||
|
parameter_sources=False,
|
||||||
|
call_depth=0,
|
||||||
|
)
|
||||||
|
return any(v.witness == finding.witness and v.rung == finding.rung for v in verdicts)
|
||||||
|
|
||||||
|
|
||||||
|
def compare_evidence(
|
||||||
|
head: ModelScanResult,
|
||||||
|
base: ModelScanResult | None,
|
||||||
|
*,
|
||||||
|
context: ChangeContext,
|
||||||
|
head_semantics: str,
|
||||||
|
base_semantics: str,
|
||||||
|
execution_budget: ExecutionBudget | None = None,
|
||||||
|
) -> tuple[CandidateDelta, ...]:
|
||||||
|
"""Absence counts only in completed comparable scope, never from scan silence."""
|
||||||
|
head_ops, base_ops = _operations(head, execution_budget), _operations(base, execution_budget) if base else ()
|
||||||
|
results = []
|
||||||
|
for finding in head.findings:
|
||||||
|
if execution_budget is not None and time.monotonic() >= execution_budget.deadline_monotonic:
|
||||||
|
results.append(_candidate(finding, None, (), "unknown", "deadline_exhausted", (), context))
|
||||||
|
continue
|
||||||
|
matching = [
|
||||||
|
op
|
||||||
|
for op in head_ops
|
||||||
|
if op.question.family == finding.family
|
||||||
|
and finding.site.startswith(f"{op.path}:{op.line}:")
|
||||||
|
and _proven_finding(finding, op, head)
|
||||||
|
]
|
||||||
|
if len(matching) > 1:
|
||||||
|
matching = [
|
||||||
|
op
|
||||||
|
for op in matching
|
||||||
|
if op.mechanism != "taint" or finding.witness.startswith(str(json.loads(op.detail).get("source")) + " -> ")
|
||||||
|
]
|
||||||
|
op = matching[0] if len(matching) == 1 else None
|
||||||
|
novelty: Literal["new", "worsened", "unchanged", "unknown"] = "unknown"
|
||||||
|
reason = "witness_identity_unresolved"
|
||||||
|
old: tuple[EvidenceOperation, ...] = ()
|
||||||
|
change: tuple[str, ...] = ()
|
||||||
|
if op is not None:
|
||||||
|
location = _base_location(op, context)
|
||||||
|
counterpart = _counterpart(op.question, context)
|
||||||
|
contextual = tuple(b for b in base_ops if b.question == counterpart and b.mechanism == op.mechanism)
|
||||||
|
old = tuple(b for b in contextual if (b.path, b.line) == location and b.operation == op.operation)
|
||||||
|
change = _change_evidence(op, context)
|
||||||
|
reason = "base_incomplete"
|
||||||
|
complete = bool(
|
||||||
|
base
|
||||||
|
and base.scope
|
||||||
|
and base.scope.population_complete
|
||||||
|
and not base.scope.deferred
|
||||||
|
and not _context_boundaries(base)
|
||||||
|
and not base.degradations
|
||||||
|
and _valid_answer(base, counterpart, base_ops)
|
||||||
|
and counterpart in {q.identity for q in base.scope.selected}
|
||||||
|
and counterpart in {q.identity for q in base.question_outcomes if q.status == "completed"}
|
||||||
|
and {q.identity for q in base.scope.selected} == {q.identity for q in base.question_outcomes}
|
||||||
|
and all(q.status == "completed" for q in base.question_outcomes)
|
||||||
|
)
|
||||||
|
if head_semantics != base_semantics:
|
||||||
|
reason = "semantics_mismatch"
|
||||||
|
elif context.base_revision is None:
|
||||||
|
reason = "base_unavailable"
|
||||||
|
elif context.unresolved_boundaries:
|
||||||
|
reason = "change_context_unresolved"
|
||||||
|
elif (
|
||||||
|
not head.scope
|
||||||
|
or _context_boundaries(head)
|
||||||
|
or head.degradations
|
||||||
|
or not _valid_answer(head, op.question, head_ops)
|
||||||
|
or op.question not in {q.identity for q in head.scope.selected}
|
||||||
|
):
|
||||||
|
reason = "head_context_incomplete"
|
||||||
|
elif execution_budget is not None and time.monotonic() >= execution_budget.deadline_monotonic:
|
||||||
|
reason = "deadline_exhausted"
|
||||||
|
elif complete:
|
||||||
|
same = tuple(b for b in old if b.source == op.source)
|
||||||
|
reason = "operation_correspondence_unresolved"
|
||||||
|
if any(
|
||||||
|
b.discharged == op.discharged
|
||||||
|
and (op.mechanism != "config" or json.loads(b.detail).get("literalArgs") == json.loads(op.detail).get("literalArgs"))
|
||||||
|
for b in same
|
||||||
|
):
|
||||||
|
novelty, reason = "unchanged", "same_supported_mechanism"
|
||||||
|
elif old and change and finding.rung == Rung.ENTAILED and not op.discharged:
|
||||||
|
if same and all(b.discharged for b in same):
|
||||||
|
novelty, reason = "worsened", "discharge_removed"
|
||||||
|
elif not same and op.mechanism == "taint":
|
||||||
|
reason = "source_correspondence_unresolved"
|
||||||
|
elif (
|
||||||
|
location is not None
|
||||||
|
and not old
|
||||||
|
and change
|
||||||
|
and op.mechanism == "taint"
|
||||||
|
and finding.rung == Rung.ENTAILED
|
||||||
|
and not op.discharged
|
||||||
|
):
|
||||||
|
novelty, reason = "new", "source_connection_absent_from_comparable_base"
|
||||||
|
results.append(_candidate(finding, op, old, novelty, reason, change, context))
|
||||||
|
return tuple(results)
|
||||||
|
|
||||||
|
|
||||||
|
def compare_targeted_base(
|
||||||
|
base_root: Path,
|
||||||
|
*,
|
||||||
|
head: ModelScanResult,
|
||||||
|
head_regions: Sequence[ScanRegion],
|
||||||
|
base_regions: Sequence[ScanRegion],
|
||||||
|
context: ChangeContext,
|
||||||
|
backend: Any,
|
||||||
|
execution_budget: ExecutionBudget,
|
||||||
|
scan_budget: ScanBudget,
|
||||||
|
head_semantics: str,
|
||||||
|
base_semantics: str,
|
||||||
|
) -> DeltaComparison:
|
||||||
|
"""Re-run counterparts of the ranker's selected head questions through the existing driver.
|
||||||
|
|
||||||
|
The caller supplies immutable base regions, not a different selector. Unknown function
|
||||||
|
correspondence is left unknown rather than widening to every function in the file.
|
||||||
|
"""
|
||||||
|
selected = {q.identity for q in head.scope.selected} if head.scope else set()
|
||||||
|
renames = {context.decode_path(r.head_path): context.decode_path(r.base_path) for r in context.renames}
|
||||||
|
wanted = {(renames.get(q.path, q.path), q.function, q.family) for q in selected}
|
||||||
|
targets = tuple(
|
||||||
|
replace(r, families=tuple(f for f in r.families if (r.path, r.function, f) in wanted))
|
||||||
|
for r in base_regions
|
||||||
|
if any((r.path, r.function, family) in wanted for family in r.families)
|
||||||
|
)
|
||||||
|
# Preserve source/entry-point contracts and family depth; mismatches are not comparable.
|
||||||
|
source_contract = {(renames.get(r.path, r.path), r.function): r.source for r in head_regions}
|
||||||
|
compatible = all(source_contract.get((r.path, r.function)) == r.source for r in targets)
|
||||||
|
reasons = []
|
||||||
|
base = None
|
||||||
|
if context.base_revision is None:
|
||||||
|
reasons.append("base_unavailable")
|
||||||
|
elif head_semantics != base_semantics or not compatible:
|
||||||
|
reasons.append("semantics_mismatch")
|
||||||
|
elif time.monotonic() >= execution_budget.deadline_monotonic:
|
||||||
|
reasons.append("deadline_exhausted")
|
||||||
|
elif not targets:
|
||||||
|
reasons.append("base_counterpart_unresolved")
|
||||||
|
else:
|
||||||
|
try:
|
||||||
|
base = scan_repository(
|
||||||
|
base_root,
|
||||||
|
targets,
|
||||||
|
backend=backend,
|
||||||
|
budget=scan_budget,
|
||||||
|
execution_budget=execution_budget,
|
||||||
|
ranking_mode=head.scope.ranking_mode if head.scope else "static",
|
||||||
|
unit=next(iter(selected)).unit,
|
||||||
|
population_complete=True,
|
||||||
|
)
|
||||||
|
except Exception as error: # noqa: BLE001 -- comparison failure is coverage, not a new defect
|
||||||
|
reasons.append(f"base_scan_failed:{type(error).__name__}")
|
||||||
|
candidates = compare_evidence(
|
||||||
|
head, base, context=context, head_semantics=head_semantics, base_semantics=base_semantics, execution_budget=execution_budget
|
||||||
|
)
|
||||||
|
reasons.extend(c.reason for c in candidates if c.novelty == "unknown")
|
||||||
|
return DeltaComparison(candidates, base, head_semantics, tuple(dict.fromkeys(reasons)))
|
||||||
@@ -0,0 +1,208 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import time
|
||||||
|
from dataclasses import replace
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from openultrasast.cpg.backend import CpgResult
|
||||||
|
from openultrasast.model.contracts import (
|
||||||
|
ChangeContext,
|
||||||
|
ChangedSpan,
|
||||||
|
ExecutionBudget,
|
||||||
|
LineCorrespondence,
|
||||||
|
PathRename,
|
||||||
|
QuestionIdentity,
|
||||||
|
QuestionOutcome,
|
||||||
|
RankedQuestion,
|
||||||
|
ScopeDecision,
|
||||||
|
)
|
||||||
|
from openultrasast.model.ladder import Rung
|
||||||
|
from openultrasast.model.pipeline import ModelFinding
|
||||||
|
from openultrasast.model.regions import ScanRegion
|
||||||
|
from openultrasast.model.scan import ModelScanResult, ScanBudget
|
||||||
|
from openultrasast.push.policy import CandidateDelta, compare_evidence, compare_targeted_base
|
||||||
|
|
||||||
|
|
||||||
|
def scan(*, path="a.js", line=4, source="req.body", sanitized=False, findings=True, status="completed", rows=None):
|
||||||
|
q = QuestionIdentity("repository", "javascript", path, "handler", "injection")
|
||||||
|
row = {
|
||||||
|
"sink": "exec(cmd)",
|
||||||
|
"sinkFile": path,
|
||||||
|
"sinkLine": line,
|
||||||
|
"sinkMethod": "handler",
|
||||||
|
"source": source,
|
||||||
|
"sourceKind": "field",
|
||||||
|
"sanitized": sanitized,
|
||||||
|
"bounded": False,
|
||||||
|
"length": 2,
|
||||||
|
}
|
||||||
|
return ModelScanResult(
|
||||||
|
findings=(
|
||||||
|
ModelFinding(f"{path}:{line}:handler", "injection", Rung.ENTAILED, f"{source} -> exec(cmd) in {path} (line {line}, 2 steps)"),
|
||||||
|
)
|
||||||
|
if findings
|
||||||
|
else (),
|
||||||
|
scope=ScopeDecision("v1", "static", True, (RankedQuestion(q, 1.0, ()),), (), ()),
|
||||||
|
question_outcomes=(
|
||||||
|
QuestionOutcome(q, status, "test_answer", json.dumps([row] if rows is None else rows) if status != "unanswered" else None),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def context(*, rename=False):
|
||||||
|
return ChangeContext(
|
||||||
|
"base",
|
||||||
|
"head",
|
||||||
|
("a.js",),
|
||||||
|
(),
|
||||||
|
(ChangedSpan("old.js" if rename else "a.js", 2, 2, "base"),),
|
||||||
|
(PathRename("old.js", "a.js"),) if rename else (),
|
||||||
|
(),
|
||||||
|
(),
|
||||||
|
(),
|
||||||
|
line_correspondences=(LineCorrespondence("old.js" if rename else "a.js", "a.js", 5, 5, 4),),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def delta(head, base, ctx=None, **kwargs):
|
||||||
|
return compare_evidence(head, base, context=ctx or context(), head_semantics="same", base_semantics=kwargs.get("semantics", "same"))[0]
|
||||||
|
|
||||||
|
|
||||||
|
def test_removed_discharge_is_worsened_at_unchanged_sink():
|
||||||
|
result = delta(scan(), scan(line=5, sanitized=True, findings=False))
|
||||||
|
assert result.novelty == "worsened"
|
||||||
|
assert result.reason == "discharge_removed"
|
||||||
|
assert result.change_evidence == ("base:a.js:2-2",)
|
||||||
|
assert CandidateDelta.from_payload(result.to_payload()) == result
|
||||||
|
|
||||||
|
|
||||||
|
def test_unchanged_backlog_and_renamed_movement_stay_unchanged():
|
||||||
|
assert delta(scan(), scan(line=5)).novelty == "unchanged"
|
||||||
|
renamed = delta(scan(), scan(path="old.js", line=5), context(rename=True))
|
||||||
|
assert renamed.novelty == "unchanged"
|
||||||
|
baseline = ChangeContext("base", "head", (), (), (), (), (), (), ())
|
||||||
|
assert renamed.defect_id == delta(scan(path="old.js", line=5), scan(path="old.js", line=5), baseline).defect_id
|
||||||
|
|
||||||
|
|
||||||
|
def test_changed_source_text_needs_identity_proof_not_a_rename_guess():
|
||||||
|
assert delta(scan(), scan(line=5, source="req.query")).reason == "source_correspondence_unresolved"
|
||||||
|
assert delta(scan(source="request.body"), scan(line=5)).novelty == "unknown"
|
||||||
|
|
||||||
|
|
||||||
|
def test_failed_truncated_incompatible_base_remains_unknown():
|
||||||
|
assert delta(scan(), scan(status="unanswered")).novelty == "unknown"
|
||||||
|
assert delta(scan(), replace(scan(), degradations=({"reason": "files_unparsed"},))).novelty == "unknown"
|
||||||
|
assert delta(scan(), scan(line=5), semantics="different").reason == "semantics_mismatch"
|
||||||
|
assert delta(scan(), None).novelty == "unknown"
|
||||||
|
|
||||||
|
|
||||||
|
def test_unrelated_lexical_guard_removal_does_not_override_same_flow():
|
||||||
|
assert delta(scan(), scan(line=5)).reason == "same_supported_mechanism"
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_anchor_and_ambiguous_witness_do_not_invent_novelty():
|
||||||
|
assert delta(scan(), scan(line=5), replace(context(), line_correspondences=())).novelty == "unknown"
|
||||||
|
answer = scan().question_outcomes[0]
|
||||||
|
ambiguous = replace(scan(), question_outcomes=(replace(answer, raw_rows_json=json.dumps(json.loads(answer.raw_rows_json) * 2)),))
|
||||||
|
assert delta(ambiguous, scan()).novelty == "unknown"
|
||||||
|
|
||||||
|
|
||||||
|
def test_targeted_base_uses_real_driver_same_deadline(tmp_path):
|
||||||
|
(tmp_path / "a.js").write_text("function handler(req) { exec(req.body); }\n")
|
||||||
|
seen = []
|
||||||
|
|
||||||
|
class Backend:
|
||||||
|
def build(self, root, *, language, exclude, execution_budget):
|
||||||
|
assert (root / "a.js").read_bytes()
|
||||||
|
seen.append(execution_budget)
|
||||||
|
rows = json.loads(scan(line=5, sanitized=True).question_outcomes[0].raw_rows_json)
|
||||||
|
return CpgResult(Path("graph"), lambda kind, params: rows)
|
||||||
|
|
||||||
|
region = ScanRegion("a.js", "handler", "javascript", ("injection",), 1.0, "entry_point")
|
||||||
|
budget = ExecutionBudget(time.monotonic() + 30, 1.0)
|
||||||
|
result = compare_targeted_base(
|
||||||
|
tmp_path,
|
||||||
|
head=scan(),
|
||||||
|
head_regions=(region,),
|
||||||
|
base_regions=(region,),
|
||||||
|
context=context(),
|
||||||
|
backend=Backend(),
|
||||||
|
execution_budget=budget,
|
||||||
|
scan_budget=ScanBudget(tiering=False),
|
||||||
|
head_semantics="same",
|
||||||
|
base_semantics="same",
|
||||||
|
)
|
||||||
|
assert seen == [budget]
|
||||||
|
assert result.base_scan.question_outcomes[0].status == "completed"
|
||||||
|
assert result.candidates[0].novelty == "worsened"
|
||||||
|
|
||||||
|
|
||||||
|
def test_new_source_at_unchanged_operation_needs_complete_base_answer():
|
||||||
|
assert delta(scan(), scan(rows=[], findings=False)).novelty == "new"
|
||||||
|
assert delta(scan(), scan(rows=[], status="unanswered", findings=False)).novelty == "unknown"
|
||||||
|
|
||||||
|
|
||||||
|
def test_changed_new_operation_in_comparable_existing_file():
|
||||||
|
ctx = replace(context(), spans=(ChangedSpan("a.js", 4, 4, "head"),), line_correspondences=())
|
||||||
|
assert delta(scan(), scan(rows=[], findings=False), ctx).novelty == "unknown"
|
||||||
|
assert delta(scan(), scan(line=6), ctx).novelty == "unknown" # movement is not new
|
||||||
|
|
||||||
|
|
||||||
|
def test_other_completed_question_cannot_prove_absence_for_missing_counterpart():
|
||||||
|
assert delta(scan(), scan(path="other.js", findings=False)).novelty == "unknown"
|
||||||
|
assert delta(scan(), replace(scan(), scope=replace(scan().scope, selected=()))).novelty == "unknown"
|
||||||
|
|
||||||
|
|
||||||
|
def test_head_failure_or_unproven_witness_stays_unknown():
|
||||||
|
assert delta(replace(scan(), degradations=({"reason": "cpg_sharded"},)), scan(line=5)).novelty == "unknown"
|
||||||
|
finding = replace(scan().findings[0], witness="invented witness")
|
||||||
|
assert delta(replace(scan(), findings=(finding,)), scan(line=5)).novelty == "unknown"
|
||||||
|
|
||||||
|
|
||||||
|
def test_string_false_is_not_supported_discharge_evidence():
|
||||||
|
base = scan(line=5, sanitized=True)
|
||||||
|
row = json.loads(base.question_outcomes[0].raw_rows_json)[0]
|
||||||
|
row["sanitized"] = "false"
|
||||||
|
result = delta(scan(), scan(line=5, rows=[row]))
|
||||||
|
# A malformed returned row must not be reinterpreted as valid negative evidence.
|
||||||
|
assert result.novelty == "unknown"
|
||||||
|
|
||||||
|
|
||||||
|
def test_deadline_exhaustion_prevents_any_new_base_build(tmp_path):
|
||||||
|
region = ScanRegion("a.js", "handler", "javascript", ("injection",), 1.0, "entry_point")
|
||||||
|
|
||||||
|
class Backend:
|
||||||
|
def build(self, *args, **kwargs):
|
||||||
|
raise AssertionError("expired budget must prevent build")
|
||||||
|
|
||||||
|
result = compare_targeted_base(
|
||||||
|
tmp_path,
|
||||||
|
head=scan(),
|
||||||
|
head_regions=(region,),
|
||||||
|
base_regions=(region,),
|
||||||
|
context=context(),
|
||||||
|
backend=Backend(),
|
||||||
|
execution_budget=ExecutionBudget(0.0, 1.0),
|
||||||
|
scan_budget=ScanBudget(),
|
||||||
|
head_semantics="same",
|
||||||
|
base_semantics="same",
|
||||||
|
)
|
||||||
|
assert "deadline_exhausted" in result.coverage_reasons
|
||||||
|
assert result.candidates[0].novelty == "unknown"
|
||||||
|
|
||||||
|
|
||||||
|
def test_benign_identifier_rename_is_unknown_not_new():
|
||||||
|
ctx = replace(context(), spans=(ChangedSpan("a.js", 4, 4, "head"),), line_correspondences=())
|
||||||
|
assert delta(scan(source="request.body"), scan(line=4), ctx).novelty == "unknown"
|
||||||
|
|
||||||
|
|
||||||
|
def test_wrong_query_mechanism_cannot_prove_base_absence():
|
||||||
|
row = {"operation": "exec(cmd)", "opFile": "a.js", "opLine": 5, "dominatingGuards": []}
|
||||||
|
assert delta(scan(), scan(line=5, rows=[row])).novelty == "unknown"
|
||||||
|
|
||||||
|
|
||||||
|
def test_non_string_operation_cannot_prove_base_absence():
|
||||||
|
row = json.loads(scan(line=5).question_outcomes[0].raw_rows_json)[0]
|
||||||
|
row["sink"] = 42
|
||||||
|
assert delta(scan(), scan(line=5, rows=[row])).novelty == "unknown"
|
||||||
Reference in New Issue
Block a user