feat(pre-push-safety-net): compare targeted base and head evidence

This commit is contained in:
Marius Ciepluch
2026-09-13 12:42:15 +02:00
parent 6ba2b06353
commit eb320730b6
11 changed files with 2077 additions and 18 deletions
@@ -0,0 +1,41 @@
# Group 4 implementation and verification
Date: 2026-09-13. The maintainer authorized group 4 as a whole. Scope is tasks
4.1–4.3: targeted novelty comparison, actionability admission and compact reporting.
Groups 1–3 are verified at `6ba2b06`; full transaction replay remains task 5.1.
Status: implementation in progress. Task completion requires independent review
and fresh evidence. No production capability is enabled for normal hook alerts;
capability admission still requires task 8.3 evaluation.
## Verification boundary
Exercise production policy functions with immutable Git snapshots and actual engine
answers, plus deterministic error-path and equivalent-origin controls. Prove source
bytes were read before interpreting empty answers. Synthetic evaluated capabilities
exercise admission mechanics only and never establish real capability eligibility.
The established gates protect detector regressions; these controls do not establish
production hook latency, broad alert precision or feature-level readiness.
## Task 4.1 verification
VERIFIED (TASK scope), independent review APPROVED. Full suite: 1,167 passed,
nine skipped in 122.44 seconds; global Ruff/format (233 files), mypy (110 source
files), all three regression gates and diff checks passed. The first OFF control
failed on novelty behavior. Review regressions reproduced false NEW classifications
from wrong-mechanism base rows and non-string operations; both now stay unknown.
`benchmarks/measurements/2026-09-13-delta-comparison-smoke.json` records actual
JavaScript guard removal as worsened, unchanged backlog moved by comments as
unchanged, and a PHP source newly connected to a mapped unchanged sink as new.
All inputs were read and both sides' questions completed. Current reviewed policy
was independently rechecked against captured actual answers after the schema-only
repair; the artifact preserves exact source hashes and that narrower provenance.
Concurrent lab costs were 214.81, 214.52 and 291.01 seconds respectively under
900-second budgets. These are not hook latency measurements.
Automatic dominance-only change-context projection remains an explicit upstream
limitation for the later replay integration. These controls use declared questions
and Git context directly; no missing-coverage result was erased. Ambiguous source
and identifier changes remain unknown. Task 4.1 brings progress to 12/27 tasks;
admission and rendering remain pending within this authorized group.
@@ -178,3 +178,22 @@ incomplete coverage. A synthetic graph file must never compensate for an omitted
real source, and a wrapper must not discard evidence supplied by a frontend.
See `implementation-group-3.md` for final evidence. Eleven of 27 tasks are complete;
change attribution, actionability admission and compact output are next.
## Group 4 comparison controls (2026-09-13)
Real JavaScript dominance rows distinguish a guarded base operation from an
unguarded head operation while retaining a guarded sibling. The direct task 4.1
comparator can consume these answers with declared questions and immutable Git
span correspondence. This does not establish automatic context projection for
that family: the existing driver's change-context evidence pass currently projects
taint questions, and a dominance-only scan with `change_context` reports
`context_projection_unavailable:contributor-scan`. Preserve that limitation in the
full replay integration at 5.1; any extension belongs to the contributor-scan
context contract, not a hook-specific graph or detector workaround.
Duplicate identical source lines also remain ambiguous to the snapshot adapter.
Controls with unique lines prove the narrow supported correspondence path; they
do not establish general identity across arbitrary refactoring. Changed source
text alone is insufficient to distinguish a new source from a benign identifier
rename. Those comparisons remain unknown. A completed comparable base with no flow
to an unchanged mapped sink supplies a supported new-connection control instead.
+13 -6
View File
@@ -1,7 +1,7 @@
{
"feature_name": "pre-push-safety-net",
"created_at": "2026-09-12T15:35:00Z",
"updated_at": "2026-09-13T09:50:01.821464+00:00",
"updated_at": "2026-09-13T10:26:47.307154+00:00",
"language": "en",
"phase": "implementation",
"approvals": {
@@ -19,7 +19,7 @@
}
},
"ready_for_implementation": true,
"implementation_approval_note": "Groups 1\u20133 verified complete. Contributor-scan frontend-retention prerequisite 2.18 approved and verified. Remaining approved tasks require independent review and fresh verification.",
"implementation_approval_note": "Groups 1\u20133 verified complete. Maintainer explicitly authorized group 4 as a whole; each task requires independent review and fresh verification.",
"depends_on": [
"contributor-scan"
],
@@ -55,6 +55,11 @@
"at": "2026-09-13T09:17:01.788332+00:00",
"source": "approved",
"action": "Approved contributor-scan frontend retention prerequisite; implement it and resume task 3.4 review and verification."
},
{
"at": "2026-09-13T10:26:47.307154+00:00",
"source": "move on to group 4 as a whole",
"action": "Authorized all of group 4 (4.1\u20134.3), sequential implementation, independent task review and fresh verification. Groups 5\u20138 remain pending."
}
],
"task_review": {
@@ -78,14 +83,16 @@
"3.1",
"3.2",
"3.3",
"3.4"
"3.4",
"4.1"
],
"remaining_tasks": 16,
"verification_record": "implementation-group-3.md",
"remaining_tasks": 15,
"verification_record": "implementation-group-4.md",
"verification_records": [
"implementation-group-1.md",
"implementation-group-2.md",
"implementation-group-3.md"
"implementation-group-3.md",
"implementation-group-4.md"
],
"next_group": 4,
"blocked_tasks": []
+3 -2
View File
@@ -1,7 +1,7 @@
# Implementation Plan: pre-push-safety-net
Status: task plan approved by the maintainer's "approve." on 2026-09-12. Groups 1–3 (tasks 1.1–3.4)
are complete after independent review and fresh verification (11/27 subtasks). Groups 4–8 remain pending;
are complete after independent review and fresh verification (12/27 subtasks, including task 4.1). Tasks 4.2–4.3 and groups 5–8 remain pending;
the hook and capability admission are not yet implemented. See `implementation-group-1.md` and `implementation-group-2.md`.
## Execution contract
@@ -113,10 +113,11 @@ the hook and capability admission are not yet implemented. See `implementation-g
- _Verified 2026-09-13:_ independent review APPROVED; 1,151 passed, nine skipped; Ruff/format/mypy pass. Real PHP/JavaScript graphs retain tests/browser source, omit vendor nodes and share one ranker limit. Named census survives PHP aggregation; missing source names and unreadable discovery inputs remain explicit. Python exercises all three batch census schemas. Evidence: `benchmarks/measurements/2026-09-13-partition-scope-smoke.json`. No hook capability or latency claim.
- [ ] 4. Implement change attribution and actionable output
- [ ] 4.1 Compare targeted base/head evidence for defect novelty
- [x] 4.1 Compare targeted base/head evidence for defect novelty
- Establish stable mechanism/operation identity with rename/span mapping and collect comparable base evidence under the same semantics and remaining deadline.
- Distinguish new, worsened, unchanged and unknown; an absent finding from a failed/truncated base is unknown. Support removed guards and new source connections at unchanged sinks.
- Done when benign movement and unchanged backlog do not become new defects, while a supported removed-guard case has a concrete change-to-witness relationship.
- _Verified 2026-09-13:_ independent review APPROVED; 1,167 tests passed, nine skipped; real JavaScript guard/movement and PHP connection controls passed. Final policy rechecked against captured actual answers after malformed-schema repair. See `implementation-group-4.md`.
- _Boundary: Delta and Actionability Policy, engine comparison integration_
- _Depends: 2.3, 3.3, 3.4_
- _Requirements: 2.4, 3.1, 3.4, 4.1_
@@ -1,10 +1,10 @@
{
"counts": {
"load_bearing": 99,
"load_bearing": 100,
"standalone_capability": 9
},
"modules": 108,
"note": "Refreshed after pre-push-safety-net task 3.4: first-party frontend partitions share the authoritative ranker and execution budget.",
"modules": 109,
"note": "Refreshed after pre-push-safety-net task 4.1: targeted base comparison reuses existing arbiters and shared execution budget; replay integration remains task 5.1.",
"orphaned": [],
"rows": [
{
@@ -116,7 +116,8 @@
"config",
"cpg.artifact",
"model.contracts",
"push.contracts"
"push.contracts",
"push.policy"
],
"module": "contracts",
"reason": "pre-push-safety-net Req 7.3: strict shared manifest serialization, independent of push"
@@ -132,7 +133,8 @@
"model.partitions",
"model.pipeline",
"model.scan",
"model.taint"
"model.taint",
"push.policy"
],
"module": "cpg",
"reason": "model-grounded-detection Req 4: the CPG seam package"
@@ -153,7 +155,8 @@
"model.partitions",
"model.pipeline",
"model.scan",
"model.taint"
"model.taint",
"push.policy"
],
"module": "cpg.backend",
"reason": "model-grounded-detection Req 4.1/4.5: the single Joern subprocess boundary"
@@ -356,6 +359,7 @@
"model.taint",
"preprocess",
"push.contracts",
"push.policy",
"push.snapshot"
],
"module": "model",
@@ -399,6 +403,7 @@
"model.regions",
"model.scan",
"push.contracts",
"push.policy",
"push.snapshot"
],
"module": "model.contracts",
@@ -450,7 +455,8 @@
"model.judge",
"model.pipeline",
"model.scan",
"model.taint"
"model.taint",
"push.policy"
],
"module": "model.ladder",
"reason": "model-grounded-detection Req 5: the evidence ladder every finding carries"
@@ -476,7 +482,8 @@
{
"classification": "load_bearing",
"importers": [
"model.scan"
"model.scan",
"push.policy"
],
"module": "model.pipeline",
"reason": "model-grounded-detection Req 8: the enumerator proposes, the LLM answers, the model disposes"
@@ -486,7 +493,8 @@
"importers": [
"cli",
"model.layout",
"model.scan"
"model.scan",
"push.policy"
],
"module": "model.regions",
"reason": "contributor-scan Req 2: regions from entry points, with the families each admits"
@@ -502,7 +510,8 @@
{
"classification": "load_bearing",
"importers": [
"cli"
"cli",
"push.policy"
],
"module": "model.scan",
"reason": "contributor-scan Req 3: the repository driver -- one CPG, one budget, ranked spend"
@@ -667,6 +676,12 @@
"module": "push.contracts",
"reason": "pre-push-safety-net Req 5.5/7.3: immutable comparisons and independent result statuses; runner pending"
},
{
"classification": "load_bearing",
"importers": [],
"module": "push.policy",
"reason": "pre-push-safety-net Req 2.4/3.1/3.4/4.1: targeted same-semantics base comparison; replay integration pending"
},
{
"classification": "load_bearing",
"importers": [],
File diff suppressed because it is too large Load Diff
+17
View File
@@ -197,3 +197,20 @@ unshipped test-path records. Runtime origin remains unspecified without declarat
readable Python graph and exercises the production taint, dominance and configuration
batch queries. Each must return the actual source filename in its census. This
checks query compilation and census transport, not security-property coverage.
## Targeted novelty comparison smoke
After rebuilding the image, run a declared JavaScript guard-removal comparison:
```bash
docker run --rm -i --network none --entrypoint python openultrasast:dev - --case guard < ops/smoke_delta.py
```
Use `--case movement` for unchanged JavaScript backlog shifted by comments, or
`--case connection` for a PHP input newly reaching an unchanged operation. Every
case verifies immutable base/head bytes, runs the existing driver and targeted
comparison under one 900-second lab budget, and checks that dirty checkout/index
state survives. Output contains an evidence JSON object followed by a completion
JSON object; success requires exit zero and `verified: true` in that final object.
These declared regions exercise comparison mechanics; full replay, normal alert
admission and hook latency are separate evaluation work.
+138
View File
@@ -0,0 +1,138 @@
"""Real Git/Joern novelty controls; no hook admission or latency claim."""
from __future__ import annotations
import argparse
import hashlib
import json
import os
import subprocess
import tempfile
import time
from dataclasses import asdict
from pathlib import Path
from openultrasast.cpg.backend import JoernBackend
from openultrasast.model.contracts import ExecutionBudget
from openultrasast.model.regions import ScanRegion
from openultrasast.model.scan import ScanBudget, scan_repository
from openultrasast.push.policy import compare_targeted_base, semantics_digest
from openultrasast.push.snapshot import SnapshotAdapter
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--case", choices=("guard", "movement", "connection"), required=True)
args = parser.parse_args()
before = (
b"export function handler(req) {\n"
b" if (!authenticate(req)) return;\n"
b" deleteUser(req.params.id);\n} // handler\n"
b"export function sibling(other) {\n"
b" if (!authenticate(other)) return;\n"
b" deleteUser(other.params.id);\n} // sibling\n"
)
after = before.replace(b" if (!authenticate(req)) return;\n", b"", 1)
language, family, filename = "javascript", "access_control", "api.js"
expected = "worsened"
if args.case == "movement":
before = after
after = b"// documentation moved this existing operation\n\n" + before
expected = "unchanged"
elif args.case == "connection":
language, family, filename = "php", "injection", "api.php"
before = b'<?php\nfunction handler($req) {\n $value = "fixed";\n eval($value);\n}\n'
after = before.replace(b'$value = "fixed";', b"$value = $req;")
expected = "new"
started = time.monotonic()
budget = ExecutionBudget(started + 900, 2.0)
report: dict[str, object] = {
"case": args.case,
"purpose": (
"Real targeted comparison with declared regions and Git context; no automatic context projection, full replay or admission"
),
"shared_budget_seconds": 900,
}
with tempfile.TemporaryDirectory(prefix="ousast-delta-smoke-") as temporary:
repo = Path(temporary)
def git(*parts: str) -> bytes:
return subprocess.run(["git", "-C", str(repo), *parts], check=True, capture_output=True, timeout=10).stdout
git("init")
git("config", "user.name", "Delta smoke")
git("config", "user.email", "[email protected]")
git("config", "core.hooksPath", os.devnull)
source = repo / filename
source.write_bytes(before)
git("add", filename)
git("commit", "-m", "base")
base_oid = git("rev-parse", "HEAD").decode().strip()
source.write_bytes(after)
git("add", filename)
git("commit", "-m", args.case)
head_oid = git("rev-parse", "HEAD").decode().strip()
source.write_bytes(b"uncommitted content\n")
index = (repo / ".git/index").read_bytes()
status = git("status", "--porcelain=v1", "-z")
adapter = SnapshotAdapter(repo)
(comparison,) = adapter.resolve_updates(f"refs/heads/topic {head_oid} refs/heads/topic {base_oid}\n").comparisons
context = adapter.compare(comparison, declaration_paths=(), budget=budget)
regions = (ScanRegion(filename, "handler", language, (family,), 1.0, "entry_point"),)
backend = JoernBackend(heap_mb=1024)
semantics = semantics_digest(engine_identity="joern-4.0.625/frontend-retention-v1", options={"max_regions": 1})
with adapter.materialize(base_oid, budget=budget) as base_snapshot, adapter.materialize(head_oid, budget=budget) as head_snapshot:
assert (base_snapshot.root / filename).read_bytes() == before and before
assert (head_snapshot.root / filename).read_bytes() == after and after
head = scan_repository(
head_snapshot.root,
regions,
backend=backend,
budget=ScanBudget(max_regions=1),
execution_budget=budget,
ranking_mode="evidence",
unit="delta-smoke",
population_complete=True,
)
assert head.findings and head.scope and head.question_outcomes, head
assert not head.degradations and all(q.status == "completed" for q in head.question_outcomes), head
delta = compare_targeted_base(
base_snapshot.root,
head=head,
head_regions=regions,
base_regions=regions,
context=context,
backend=backend,
execution_budget=budget,
scan_budget=ScanBudget(max_regions=1),
head_semantics=semantics,
base_semantics=semantics,
)
report.update(
{
"comparison": comparison.to_payload(),
"context": context.to_payload(),
"base_input": {"bytes": len(before), "sha256": hashlib.sha256(before).hexdigest()},
"head_input": {"bytes": len(after), "sha256": hashlib.sha256(after).hexdigest()},
"head": asdict(head),
"delta": asdict(delta),
"expected_novelty": expected,
}
)
# Emit evidence before assertions so a failed instrument cannot look like a clean zero.
print(json.dumps(report, indent=2), flush=True)
assert delta.base_scan and delta.base_scan.question_outcomes, delta
assert not delta.base_scan.degradations, delta
assert all(q.status == "completed" for q in delta.base_scan.question_outcomes), delta
assert delta.candidates and all(c.novelty == expected for c in delta.candidates), delta
assert not delta.coverage_reasons, delta
assert not base_snapshot.root.exists() and not head_snapshot.root.exists()
assert source.read_bytes() == b"uncommitted content\n"
assert (repo / ".git/index").read_bytes() == index
assert git("status", "--porcelain=v1", "-z") == status
# A separate completion record distinguishes all assertions passing from partial JSON.
print(json.dumps({"verified": True, "case": args.case, "elapsed_seconds": round(time.monotonic() - started, 2)}))
if __name__ == "__main__":
main()
+1
View File
@@ -28,6 +28,7 @@ ROOTS = ("gate", "map_gate", "pair_gate", "cli", "__main__")
# Modules that stand alone: their own CLI subcommand or external protocol, deliberately off the scan path.
SPEC_OWNED = {
"push.policy": "pre-push-safety-net Req 2.4/3.1/3.4/4.1: targeted same-semantics base comparison; replay integration pending",
"push.snapshot": "pre-push-safety-net Req 1.1/1.3/1.4/7.1: local immutable ref resolution; runner integration pending",
"contracts": "pre-push-safety-net Req 7.3: strict shared manifest serialization, independent of push",
"model.contracts": "pre-push-safety-net Req 4.1/7.3: generic deadline, change and exact scope contracts; integration pending",
+374
View File
@@ -0,0 +1,374 @@
"""Compare arbiter-backed evidence from immutable revisions; this does not admit alerts."""
from __future__ import annotations
import hashlib
import json
import time
from collections.abc import Sequence
from dataclasses import dataclass, replace
from pathlib import Path
from typing import Any, Literal
from openultrasast.contracts import Contract
from openultrasast.cpg.backend import CpgResult
from openultrasast.model.contracts import ChangeContext, ExecutionBudget, QuestionIdentity
from openultrasast.model.ladder import Rung
from openultrasast.model.pipeline import ModelFinding, _arbitrate_all
from openultrasast.model.regions import ScanRegion
from openultrasast.model.scan import ModelScanResult, ScanBudget, _prefetched, _spec_for, scan_repository
from openultrasast.model.specs import ConfigSpec, DominanceSpec, TaintSpec
@dataclass(frozen=True)
class EvidenceOperation(Contract):
question: QuestionIdentity
mechanism: Literal["taint", "dominance", "config"]
path: str
line: int
operation: str
source: str
discharged: bool
detail: str
@dataclass(frozen=True)
class CandidateDelta(Contract):
family: str
site: str
rung: str
witness: str | None
defect_id: str
novelty: Literal["new", "worsened", "unchanged", "unknown"]
reason: str
head_operation: EvidenceOperation | None
base_operations: tuple[EvidenceOperation, ...]
change_evidence: tuple[str, ...]
@dataclass(frozen=True)
class DeltaComparison:
candidates: tuple[CandidateDelta, ...]
base_scan: ModelScanResult | None
semantics: str
coverage_reasons: tuple[str, ...]
def semantics_digest(*, engine_identity: str, options: dict[str, object]) -> str:
"""Fingerprint shipped semantics plus caller-owned engine/options provenance.
Include frontend adaptation, declaration/exclusion policy and scan options in options.
This is an identity, never evidence of engine liveness or input readability.
"""
root = Path(__file__).resolve().parents[1]
files = sorted([*(root / "cpg" / "queries").glob("*.sc"), *(root / "ruleset").rglob("*.toml"), *(root / "model").glob("*.py")])
payload = {
"engine": engine_identity,
"options": options,
"files": [(str(p.relative_to(root)), hashlib.sha256(p.read_bytes()).hexdigest()) for p in files],
}
return hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()
def _operations(scan: ModelScanResult, budget: ExecutionBudget | None = None) -> tuple[EvidenceOperation, ...]:
result = []
for answer in scan.question_outcomes:
if budget is not None and time.monotonic() >= budget.deadline_monotonic:
break
if answer.status != "completed" or answer.raw_rows_json is None:
continue
spec = _spec_for(answer.identity.family, answer.identity.language)
expected = (
"taint"
if isinstance(spec, TaintSpec)
else "dominance"
if isinstance(spec, DominanceSpec)
else "config"
if isinstance(spec, ConfigSpec)
else None
)
for row in json.loads(answer.raw_rows_json):
if budget is not None and time.monotonic() >= budget.deadline_monotonic:
break
kind: Literal["taint", "dominance", "config"]
if not isinstance(row, dict):
continue
if "sink" in row:
kind, path, line, op = "taint", row.get("sinkFile"), row.get("sinkLine"), row.get("sink")
if not isinstance(row.get("source"), str) or not row["source"].strip():
continue
if type(row.get("sanitized")) is not bool or type(row.get("bounded")) is not bool:
continue
source = json.dumps([row.get("sourceKind", "unknown"), row.get("source")], separators=(",", ":"))
discharged = row["sanitized"] or row["bounded"]
elif "operation" in row:
kind, path, line, op = "dominance", row.get("opFile"), row.get("opLine"), row.get("operation")
source = "obligation"
if not isinstance(row.get("dominatingGuards"), list) or not all(isinstance(g, str) for g in row["dominatingGuards"]):
continue
discharged = bool(row["dominatingGuards"])
elif "setting" in row:
kind, path, line, op = "config", row.get("file"), row.get("line"), row.get("setting")
source = "configuration"
discharged = False
else:
continue
if (
kind != expected
or not isinstance(path, str)
or not path
or not isinstance(op, str)
or not op.strip()
or not str(line).isdigit()
or int(str(line)) < 1
):
continue
result.append(
EvidenceOperation(answer.identity, kind, path, int(str(line)), str(op), source, discharged, json.dumps(row, sort_keys=True))
)
return tuple(result)
def _base_location(op: EvidenceOperation, context: ChangeContext) -> tuple[str, int] | None:
matches = []
for anchor in context.line_correspondences:
size = anchor.base_end_line - anchor.base_start_line
if context.decode_path(anchor.head_path) == op.path and anchor.head_start_line <= op.line <= anchor.head_start_line + size:
matches.append((context.decode_path(anchor.base_path), anchor.base_start_line + op.line - anchor.head_start_line))
if matches:
return matches[0] if len(set(matches)) == 1 else None
changed = {context.decode_path(p) for p in context.changed_paths}
if op.path not in changed:
return op.path, op.line
return None
def _change_evidence(op: EvidenceOperation, context: ChangeContext) -> tuple[str, ...]:
paths = {op.path, op.question.path}
for relation in context.relationships:
if relation.target == op.question and relation.evidence:
paths.add(relation.source.path)
paths.update(context.decode_path(r.base_path) for r in context.renames if context.decode_path(r.head_path) in paths)
return tuple(
f"{span.side}:{context.decode_path(span.path)}:{span.start_line}-{span.end_line}"
for span in context.spans
if context.decode_path(span.path) in paths
)
def _candidate(
finding: ModelFinding,
op: EvidenceOperation | None,
bases: tuple[EvidenceOperation, ...],
novelty: Literal["new", "worsened", "unchanged", "unknown"],
reason: str,
change: tuple[str, ...],
context: ChangeContext,
) -> CandidateDelta:
location = _base_location(op, context) if op else None
identity = [
finding.family,
op.mechanism if op else "unknown",
location or ((op.path, op.line) if op else finding.site),
op.operation if op else finding.site,
op.source if op else "unknown",
op.question.function if op else None,
]
key = hashlib.sha256(json.dumps(identity, sort_keys=True).encode()).hexdigest()
return CandidateDelta(
finding.family, finding.site, finding.rung.value, finding.witness or None, key, novelty, reason, op, bases, change
)
def _valid_answer(scan: ModelScanResult, question: QuestionIdentity, operations: tuple[EvidenceOperation, ...]) -> bool:
answer = next((q for q in scan.question_outcomes if q.identity == question), None)
if answer is None or answer.raw_rows_json is None:
return False
rows = json.loads(answer.raw_rows_json)
# Query bodies are operation rows. An unrecognized/malformed row is not an empty answer.
return len(rows) == sum(op.question == question for op in operations)
def _context_boundaries(scan: ModelScanResult) -> tuple[str, ...]:
# A missing scheduling vector is not missing arbiter evidence after its question completed.
return tuple(b for b in scan.scope.unresolved_boundaries if not b.startswith("evidence_unknown:")) if scan.scope else ("scope_missing",)
def _counterpart(question: QuestionIdentity, context: ChangeContext) -> QuestionIdentity:
renamed = {context.decode_path(r.head_path): context.decode_path(r.base_path) for r in context.renames}
return replace(question, path=renamed.get(question.path, question.path))
def _proven_finding(finding: ModelFinding, op: EvidenceOperation, head: ModelScanResult) -> bool:
"""Match the existing arbiter's exact witness, not just a coincident line number."""
answer = next((q for q in head.question_outcomes if q.identity == op.question), None)
spec = _spec_for(op.question.family, op.question.language)
if answer is None or answer.raw_rows_json is None or spec is None:
return False
rows = json.loads(answer.raw_rows_json)
verdicts = _arbitrate_all(
CpgResult(Path("prefetched"), _prefetched(rows)),
spec,
path=op.question.path,
function=op.question.function or "",
parameter_sources=False,
call_depth=0,
)
return any(v.witness == finding.witness and v.rung == finding.rung for v in verdicts)
def compare_evidence(
head: ModelScanResult,
base: ModelScanResult | None,
*,
context: ChangeContext,
head_semantics: str,
base_semantics: str,
execution_budget: ExecutionBudget | None = None,
) -> tuple[CandidateDelta, ...]:
"""Absence counts only in completed comparable scope, never from scan silence."""
head_ops, base_ops = _operations(head, execution_budget), _operations(base, execution_budget) if base else ()
results = []
for finding in head.findings:
if execution_budget is not None and time.monotonic() >= execution_budget.deadline_monotonic:
results.append(_candidate(finding, None, (), "unknown", "deadline_exhausted", (), context))
continue
matching = [
op
for op in head_ops
if op.question.family == finding.family
and finding.site.startswith(f"{op.path}:{op.line}:")
and _proven_finding(finding, op, head)
]
if len(matching) > 1:
matching = [
op
for op in matching
if op.mechanism != "taint" or finding.witness.startswith(str(json.loads(op.detail).get("source")) + " -> ")
]
op = matching[0] if len(matching) == 1 else None
novelty: Literal["new", "worsened", "unchanged", "unknown"] = "unknown"
reason = "witness_identity_unresolved"
old: tuple[EvidenceOperation, ...] = ()
change: tuple[str, ...] = ()
if op is not None:
location = _base_location(op, context)
counterpart = _counterpart(op.question, context)
contextual = tuple(b for b in base_ops if b.question == counterpart and b.mechanism == op.mechanism)
old = tuple(b for b in contextual if (b.path, b.line) == location and b.operation == op.operation)
change = _change_evidence(op, context)
reason = "base_incomplete"
complete = bool(
base
and base.scope
and base.scope.population_complete
and not base.scope.deferred
and not _context_boundaries(base)
and not base.degradations
and _valid_answer(base, counterpart, base_ops)
and counterpart in {q.identity for q in base.scope.selected}
and counterpart in {q.identity for q in base.question_outcomes if q.status == "completed"}
and {q.identity for q in base.scope.selected} == {q.identity for q in base.question_outcomes}
and all(q.status == "completed" for q in base.question_outcomes)
)
if head_semantics != base_semantics:
reason = "semantics_mismatch"
elif context.base_revision is None:
reason = "base_unavailable"
elif context.unresolved_boundaries:
reason = "change_context_unresolved"
elif (
not head.scope
or _context_boundaries(head)
or head.degradations
or not _valid_answer(head, op.question, head_ops)
or op.question not in {q.identity for q in head.scope.selected}
):
reason = "head_context_incomplete"
elif execution_budget is not None and time.monotonic() >= execution_budget.deadline_monotonic:
reason = "deadline_exhausted"
elif complete:
same = tuple(b for b in old if b.source == op.source)
reason = "operation_correspondence_unresolved"
if any(
b.discharged == op.discharged
and (op.mechanism != "config" or json.loads(b.detail).get("literalArgs") == json.loads(op.detail).get("literalArgs"))
for b in same
):
novelty, reason = "unchanged", "same_supported_mechanism"
elif old and change and finding.rung == Rung.ENTAILED and not op.discharged:
if same and all(b.discharged for b in same):
novelty, reason = "worsened", "discharge_removed"
elif not same and op.mechanism == "taint":
reason = "source_correspondence_unresolved"
elif (
location is not None
and not old
and change
and op.mechanism == "taint"
and finding.rung == Rung.ENTAILED
and not op.discharged
):
novelty, reason = "new", "source_connection_absent_from_comparable_base"
results.append(_candidate(finding, op, old, novelty, reason, change, context))
return tuple(results)
def compare_targeted_base(
base_root: Path,
*,
head: ModelScanResult,
head_regions: Sequence[ScanRegion],
base_regions: Sequence[ScanRegion],
context: ChangeContext,
backend: Any,
execution_budget: ExecutionBudget,
scan_budget: ScanBudget,
head_semantics: str,
base_semantics: str,
) -> DeltaComparison:
"""Re-run counterparts of the ranker's selected head questions through the existing driver.
The caller supplies immutable base regions, not a different selector. Unknown function
correspondence is left unknown rather than widening to every function in the file.
"""
selected = {q.identity for q in head.scope.selected} if head.scope else set()
renames = {context.decode_path(r.head_path): context.decode_path(r.base_path) for r in context.renames}
wanted = {(renames.get(q.path, q.path), q.function, q.family) for q in selected}
targets = tuple(
replace(r, families=tuple(f for f in r.families if (r.path, r.function, f) in wanted))
for r in base_regions
if any((r.path, r.function, family) in wanted for family in r.families)
)
# Preserve source/entry-point contracts and family depth; mismatches are not comparable.
source_contract = {(renames.get(r.path, r.path), r.function): r.source for r in head_regions}
compatible = all(source_contract.get((r.path, r.function)) == r.source for r in targets)
reasons = []
base = None
if context.base_revision is None:
reasons.append("base_unavailable")
elif head_semantics != base_semantics or not compatible:
reasons.append("semantics_mismatch")
elif time.monotonic() >= execution_budget.deadline_monotonic:
reasons.append("deadline_exhausted")
elif not targets:
reasons.append("base_counterpart_unresolved")
else:
try:
base = scan_repository(
base_root,
targets,
backend=backend,
budget=scan_budget,
execution_budget=execution_budget,
ranking_mode=head.scope.ranking_mode if head.scope else "static",
unit=next(iter(selected)).unit,
population_complete=True,
)
except Exception as error: # noqa: BLE001 -- comparison failure is coverage, not a new defect
reasons.append(f"base_scan_failed:{type(error).__name__}")
candidates = compare_evidence(
head, base, context=context, head_semantics=head_semantics, base_semantics=base_semantics, execution_budget=execution_budget
)
reasons.extend(c.reason for c in candidates if c.novelty == "unknown")
return DeltaComparison(candidates, base, head_semantics, tuple(dict.fromkeys(reasons)))
+208
View File
@@ -0,0 +1,208 @@
from __future__ import annotations
import json
import time
from dataclasses import replace
from pathlib import Path
from openultrasast.cpg.backend import CpgResult
from openultrasast.model.contracts import (
ChangeContext,
ChangedSpan,
ExecutionBudget,
LineCorrespondence,
PathRename,
QuestionIdentity,
QuestionOutcome,
RankedQuestion,
ScopeDecision,
)
from openultrasast.model.ladder import Rung
from openultrasast.model.pipeline import ModelFinding
from openultrasast.model.regions import ScanRegion
from openultrasast.model.scan import ModelScanResult, ScanBudget
from openultrasast.push.policy import CandidateDelta, compare_evidence, compare_targeted_base
def scan(*, path="a.js", line=4, source="req.body", sanitized=False, findings=True, status="completed", rows=None):
q = QuestionIdentity("repository", "javascript", path, "handler", "injection")
row = {
"sink": "exec(cmd)",
"sinkFile": path,
"sinkLine": line,
"sinkMethod": "handler",
"source": source,
"sourceKind": "field",
"sanitized": sanitized,
"bounded": False,
"length": 2,
}
return ModelScanResult(
findings=(
ModelFinding(f"{path}:{line}:handler", "injection", Rung.ENTAILED, f"{source} -> exec(cmd) in {path} (line {line}, 2 steps)"),
)
if findings
else (),
scope=ScopeDecision("v1", "static", True, (RankedQuestion(q, 1.0, ()),), (), ()),
question_outcomes=(
QuestionOutcome(q, status, "test_answer", json.dumps([row] if rows is None else rows) if status != "unanswered" else None),
),
)
def context(*, rename=False):
return ChangeContext(
"base",
"head",
("a.js",),
(),
(ChangedSpan("old.js" if rename else "a.js", 2, 2, "base"),),
(PathRename("old.js", "a.js"),) if rename else (),
(),
(),
(),
line_correspondences=(LineCorrespondence("old.js" if rename else "a.js", "a.js", 5, 5, 4),),
)
def delta(head, base, ctx=None, **kwargs):
return compare_evidence(head, base, context=ctx or context(), head_semantics="same", base_semantics=kwargs.get("semantics", "same"))[0]
def test_removed_discharge_is_worsened_at_unchanged_sink():
result = delta(scan(), scan(line=5, sanitized=True, findings=False))
assert result.novelty == "worsened"
assert result.reason == "discharge_removed"
assert result.change_evidence == ("base:a.js:2-2",)
assert CandidateDelta.from_payload(result.to_payload()) == result
def test_unchanged_backlog_and_renamed_movement_stay_unchanged():
assert delta(scan(), scan(line=5)).novelty == "unchanged"
renamed = delta(scan(), scan(path="old.js", line=5), context(rename=True))
assert renamed.novelty == "unchanged"
baseline = ChangeContext("base", "head", (), (), (), (), (), (), ())
assert renamed.defect_id == delta(scan(path="old.js", line=5), scan(path="old.js", line=5), baseline).defect_id
def test_changed_source_text_needs_identity_proof_not_a_rename_guess():
assert delta(scan(), scan(line=5, source="req.query")).reason == "source_correspondence_unresolved"
assert delta(scan(source="request.body"), scan(line=5)).novelty == "unknown"
def test_failed_truncated_incompatible_base_remains_unknown():
assert delta(scan(), scan(status="unanswered")).novelty == "unknown"
assert delta(scan(), replace(scan(), degradations=({"reason": "files_unparsed"},))).novelty == "unknown"
assert delta(scan(), scan(line=5), semantics="different").reason == "semantics_mismatch"
assert delta(scan(), None).novelty == "unknown"
def test_unrelated_lexical_guard_removal_does_not_override_same_flow():
assert delta(scan(), scan(line=5)).reason == "same_supported_mechanism"
def test_missing_anchor_and_ambiguous_witness_do_not_invent_novelty():
assert delta(scan(), scan(line=5), replace(context(), line_correspondences=())).novelty == "unknown"
answer = scan().question_outcomes[0]
ambiguous = replace(scan(), question_outcomes=(replace(answer, raw_rows_json=json.dumps(json.loads(answer.raw_rows_json) * 2)),))
assert delta(ambiguous, scan()).novelty == "unknown"
def test_targeted_base_uses_real_driver_same_deadline(tmp_path):
(tmp_path / "a.js").write_text("function handler(req) { exec(req.body); }\n")
seen = []
class Backend:
def build(self, root, *, language, exclude, execution_budget):
assert (root / "a.js").read_bytes()
seen.append(execution_budget)
rows = json.loads(scan(line=5, sanitized=True).question_outcomes[0].raw_rows_json)
return CpgResult(Path("graph"), lambda kind, params: rows)
region = ScanRegion("a.js", "handler", "javascript", ("injection",), 1.0, "entry_point")
budget = ExecutionBudget(time.monotonic() + 30, 1.0)
result = compare_targeted_base(
tmp_path,
head=scan(),
head_regions=(region,),
base_regions=(region,),
context=context(),
backend=Backend(),
execution_budget=budget,
scan_budget=ScanBudget(tiering=False),
head_semantics="same",
base_semantics="same",
)
assert seen == [budget]
assert result.base_scan.question_outcomes[0].status == "completed"
assert result.candidates[0].novelty == "worsened"
def test_new_source_at_unchanged_operation_needs_complete_base_answer():
assert delta(scan(), scan(rows=[], findings=False)).novelty == "new"
assert delta(scan(), scan(rows=[], status="unanswered", findings=False)).novelty == "unknown"
def test_changed_new_operation_in_comparable_existing_file():
ctx = replace(context(), spans=(ChangedSpan("a.js", 4, 4, "head"),), line_correspondences=())
assert delta(scan(), scan(rows=[], findings=False), ctx).novelty == "unknown"
assert delta(scan(), scan(line=6), ctx).novelty == "unknown" # movement is not new
def test_other_completed_question_cannot_prove_absence_for_missing_counterpart():
assert delta(scan(), scan(path="other.js", findings=False)).novelty == "unknown"
assert delta(scan(), replace(scan(), scope=replace(scan().scope, selected=()))).novelty == "unknown"
def test_head_failure_or_unproven_witness_stays_unknown():
assert delta(replace(scan(), degradations=({"reason": "cpg_sharded"},)), scan(line=5)).novelty == "unknown"
finding = replace(scan().findings[0], witness="invented witness")
assert delta(replace(scan(), findings=(finding,)), scan(line=5)).novelty == "unknown"
def test_string_false_is_not_supported_discharge_evidence():
base = scan(line=5, sanitized=True)
row = json.loads(base.question_outcomes[0].raw_rows_json)[0]
row["sanitized"] = "false"
result = delta(scan(), scan(line=5, rows=[row]))
# A malformed returned row must not be reinterpreted as valid negative evidence.
assert result.novelty == "unknown"
def test_deadline_exhaustion_prevents_any_new_base_build(tmp_path):
region = ScanRegion("a.js", "handler", "javascript", ("injection",), 1.0, "entry_point")
class Backend:
def build(self, *args, **kwargs):
raise AssertionError("expired budget must prevent build")
result = compare_targeted_base(
tmp_path,
head=scan(),
head_regions=(region,),
base_regions=(region,),
context=context(),
backend=Backend(),
execution_budget=ExecutionBudget(0.0, 1.0),
scan_budget=ScanBudget(),
head_semantics="same",
base_semantics="same",
)
assert "deadline_exhausted" in result.coverage_reasons
assert result.candidates[0].novelty == "unknown"
def test_benign_identifier_rename_is_unknown_not_new():
ctx = replace(context(), spans=(ChangedSpan("a.js", 4, 4, "head"),), line_correspondences=())
assert delta(scan(source="request.body"), scan(line=4), ctx).novelty == "unknown"
def test_wrong_query_mechanism_cannot_prove_base_absence():
row = {"operation": "exec(cmd)", "opFile": "a.js", "opLine": 5, "dominatingGuards": []}
assert delta(scan(), scan(line=5, rows=[row])).novelty == "unknown"
def test_non_string_operation_cannot_prove_base_absence():
row = json.loads(scan(line=5).question_outcomes[0].raw_rows_json)[0]
row["sink"] = 42
assert delta(scan(), scan(line=5, rows=[row])).novelty == "unknown"