feat(pre-push-safety-net): scope comparison completeness and ambiguity (M1b 10.2, 10.3)

Two vetoes M1a recorded were global where the evidence is local.

Comparison required every selected base question to complete, so one unsupported
family anywhere prevented an otherwise complete target comparison. Completeness
is now demanded over the required scope: the counterpart question plus the
dependencies the change context actually recorded for it. Every other gap stays
in aggregate coverage and is still reported, never discharged. An incomplete
counterpart, an incomplete recorded dependency, an unknown-ownership boundary
and a truncated base scan all still yield unknown.

Correspondence ambiguity was recorded file-wide and inherited by every question,
so repeated unchanged lines in one file invalidated every question in the
repository and blocked the whole transaction comparison. It is now decided per
operation by the correspondence classifier: an operation needing an ambiguous
anchor stays uncomparable, while a uniquely mapped or newly introduced operation
in the same file is unaffected. The mapping is still withheld and still reported.

On the NodeGoat case the ambiguity is on the target's own file, so file-level
scoping would not have been enough; only per-operation resolution is.

1,347 tests passed, nine skipped. No capability enabled; rollout stays NO-GO.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-09-16 15:45:57 +02:00
co-authored by Claude Opus 5
parent 7e0d01d9e8
commit 8ff1d9a638
5 changed files with 182 additions and 23 deletions
+20 -9
View File
@@ -267,45 +267,56 @@ Preserve all completed task evidence below.
- _Verified 2026-09-16:_ all three pinned revisions replayed through the production CLI under a named 900-second development budget; expectations frozen on disk first. Raw `<lambda>2` witnesses present at lines 32–34; fixed twin analyzed and quiet (target question answered with zero rows, no deadline reason, three base-only findings); unchanged revision analyzed and recorded `unchanged`. Five vetoes fired and were recorded; the remaining blocker is `operation_correspondence_unresolved` for an operation on a changed line. Zero admitted defects. Evidence: [M1a record](m1a-recorded-vetoes.md) and `benchmarks/measurements/2026-09-16-nodegoat-m1a-recorded-vetoes.json`. Diagnosis only; no capability enabled and no latency claim.
- [ ] 10. Repair only the vetoes M1a recorded (release milestone M1b)
- [ ] 10.1 Establish operation correspondence for an operation on a changed line
- [x] 10.1 Establish operation correspondence for an operation on a changed line
- M1a measured this as the blocker that survives every other repair: a newly introduced operation sits on a changed line, has no base line anchor, and therefore cannot be classified even when every completeness veto is lifted.
- Bind the required base/head correspondence to the specific operation identity rather than to a line anchor. A complete base answer for the counterpart question that contains no operation with the same operation and source establishes absence; a matching operation elsewhere in that answer is movement, not novelty.
- Keep the existing conservative rules: an incomplete or failed base stays unknown, benign movement and renames stay unchanged, and lexical alignment is never semantic proof.
- Add absent, renamed, removed-guard, moved-operation and repeated-identical-operation controls. Done when a newly introduced operation on a changed line is classified new, while movement, renames and an incomplete base remain unchanged or unknown.
- _Boundary: Delta and Actionability Policy_
- _Requirements: 2.4, 3.1, 3.4_
- [ ] 10.2 Scope correspondence ambiguity to the operations it actually affects
- _Verified 2026-09-16:_ 1,337 tests passed, nine skipped; Ruff/format/mypy passed. Correspondence now reports `anchored`, `ambiguous`, `changed_line`, `unmapped` or `unchanged_path`, and only `changed_line` supports an introduced operation. Movement keys on the operation spelling, which the existing benign-rename control proved necessary. One superseded control was updated with its reason; movement, ambiguity, unmapped operations, discharged or unentailed claims and an incomplete base all remain unknown.
- [x] 10.2 Establish the required comparison scope explicitly
- Comparison currently requires every selected base question to complete with no base degradation and no non-tier-zero deferred question, so one unsupported family anywhere prevents an otherwise complete target comparison. The owner research names this and requires the amendment to state the required scope rather than relax the bar.
- Define the comparison's required scope as the counterpart question plus the questions its evidence actually depends on, and require completeness over exactly that scope. Retain every other gap in aggregate coverage; a question outside the required scope stays reported and unresolved, never silently discharged.
- A missing raw finding in an incomplete base still cannot establish novelty. Unknown dependency remains inside the required scope.
- Done when an unrelated unsupported family no longer prevents a complete target comparison, while an incomplete counterpart, an incomplete dependency of it, or unknown dependency ownership still yields unknown.
- _Boundary: Delta and Actionability Policy_
- _Depends: 10.1_
- _Requirements: 2.2, 2.4, 3.4, 6.4_
- _Verified 2026-09-16:_ 1,342 tests passed, nine skipped; Ruff/format/mypy passed. Completeness is now demanded over the counterpart plus its recorded dependencies. Controls prove an unrelated unsupported family no longer blocks a complete target comparison, while an incomplete counterpart, an incomplete recorded dependency, an unknown-ownership boundary and a truncated base scan all still yield unknown. Every other gap stays in aggregate coverage.
- [x] 10.3 Scope correspondence ambiguity to the operations it actually affects
- `ambiguous_line_correspondence` is recorded file-wide and inherited by every question, so repeated unchanged lines invalidate distinct, uniquely mapped operations elsewhere.
- Retain ambiguous mappings as unavailable and record which regions are ambiguous, so only an operation whose required correspondence needs an ambiguous line is vetoed. Do not delete the ambiguity check.
- Done when a uniquely mapped operation is comparable in a file that also contains repeated unchanged lines, while an operation needing an ambiguous line stays unresolved.
- _Boundary: Snapshot Adapter, Delta and Actionability Policy_
- _Depends: 10.1_
- _Depends: 10.2_
- _Requirements: 2.4, 3.4_
- [ ] 10.3 Make the external and dynamic context boundary question-owned
- _Verified 2026-09-16:_ 1,347 tests passed, nine skipped; Ruff/format/mypy passed. Ambiguity is no longer inherited as a context gap by every question, and no longer blocks the transaction comparison; it is decided per operation by the correspondence classifier. Controls prove a uniquely mapped or newly introduced operation stays comparable in a file with repeated unchanged lines, an operation needing an ambiguous anchor stays unresolved, and any other transaction gap still blocks. The mapping is still withheld and still reported in aggregate coverage.
- [ ] 10.4 Make the external and dynamic context boundary question-owned
- M1a recorded `dynamic_external_or_depth_context_unresolved` on the target's own question. The flag is computed over every in-scope method, so one unresolved call anywhere vetoes every question, including those whose relevant methods are fully resolved.
- Report the boundary with the methods whose destinations are unresolved, and inherit it into a question only when such a method participates in that question's change relationships. Unknown ownership remains a blocker; absent evidence of independence is not proof of independence.
- Done when a question whose relevant methods resolve completely is not vetoed by an unrelated external call, while a question that depends on an unresolved destination stays unresolved.
- _Boundary: contributor-scan query/context evidence, Ranker Scope Contract_
- _Requirements: 2.1, 2.2, 8.1, 8.2_
- [ ] 10.4 Stop vendor exclusion acting as a global completion veto
- [ ] 10.5 Stop vendor exclusion acting as a global completion veto
- `vendor_semantics_unresolved` currently degrades every otherwise complete outcome to `graph_incomplete`, including on a complete 44-file first-party census with successful query receipts.
- Keep the exclusion physical and keep the coverage note. Let the question-owned dependency evidence from 10.3 carry whether a specific answer needed excluded semantics; a recorded dependency on an excluded path still blocks that question.
- Done when an unrelated vendor exclusion no longer invalidates a first-party answer, while a question with evidence of dependency on excluded code remains unresolved and the exclusion stays out of targets and graphs.
- _Boundary: contributor-scan graph/scope evidence, Cache and Engine Boundary_
- _Depends: 10.3_
- _Depends: 10.4_
- _Requirements: 2.2, 2.3, 8.2, 8.4_
- [ ] 10.5 Carry query-to-operation provenance into the reported evidence
- [ ] 10.6 Carry query-to-operation provenance into the reported evidence
- The target is selected through a file-fallback question with `function=None` while rows report method `<lambda>2`. M1a showed this did not block the comparison once the question completed, so it is provenance quality rather than the blocker.
- Record verified query-to-operation provenance on the evidence; do not inject benchmark function names into regions or count coincident sites.
- Done when a reported operation names the question that produced it and the method the engine reported, with unresolved provenance explicit.
- _Boundary: Delta and Actionability Policy, contributor-scan normalization_
- _Requirements: 3.1, 3.4, 6.3_
- [ ] 10.6 Rerun the three revisions through the production rules
- [ ] 10.7 Rerun the three revisions through the production rules
- Replay the same pinned vulnerable, fixed and unchanged revisions **without** the recorded-veto flag under the named development budget. Demonstrate a developer-readable evaluation finding with a supported consequence, exact location, witness, change attribution and repair direction.
- Verify the fixed twin is analyzed and produces no actionable defect and the unchanged revision is not reported as a new regression. Keep qualification eligibility disabled; experimental evaluation output cannot advertise an unqualified hook alert.
- Done when one replayable record states which vetoes were repaired and which remain, with completed required checks and readable source and census receipts.
- _Boundary: Evaluation harness, full-run integration_
- _Depends: 10.1, 10.2, 10.3, 10.4, 10.5_
- _Depends: 10.1, 10.2, 10.3, 10.4, 10.5, 10.6_
- _Requirements: 1.1, 2.1, 2.4, 3.1, 3.4, 6.3, 7.1, 7.3_
## Implementation Notes
+9 -1
View File
@@ -179,6 +179,9 @@ def _families(language: str, *, has_handler: bool) -> tuple[str, ...]:
return tuple(sorted(families))
AMBIGUOUS_CORRESPONDENCE = "ambiguous_line_correspondence"
def affected_context(
context: ChangeContext,
questions: Sequence[QuestionIdentity],
@@ -202,7 +205,12 @@ def affected_context(
gaps.append("configuration_dependency_projection_unavailable:contributor-scan")
if context.base_revision is None:
gaps.append("comparison_base_unavailable")
inherited_gap = bool(gaps)
# Line-correspondence ambiguity is not a context gap: it says which base lines a head
# line could map to, which only the later per-operation comparison consumes. Inheriting
# it here made one file with repeated unchanged lines invalidate every question in the
# repository. The mapping itself is still withheld, so an operation that needs an
# ambiguous anchor remains uncomparable.
inherited_gap = any(not gap.endswith(":" + AMBIGUOUS_CORRESPONDENCE) for gap in gaps)
renamed = {context.decode_path(r.base_path): context.decode_path(r.head_path) for r in context.renames}
for identity in questions:
if execution_budget is not None and time.monotonic() >= execution_budget.deadline_monotonic:
+43 -13
View File
@@ -17,7 +17,7 @@ from openultrasast.cpg.backend import CpgResult
from openultrasast.model.contracts import ChangeContext, ExecutionBudget, QuestionIdentity
from openultrasast.model.ladder import Rung
from openultrasast.model.pipeline import ModelFinding, _arbitrate_all
from openultrasast.model.regions import ScanRegion
from openultrasast.model.regions import AMBIGUOUS_CORRESPONDENCE, ScanRegion
from openultrasast.model.scan import ModelScanResult, ScanBudget, _prefetched, _spec_for, scan_repository
from openultrasast.model.specs import ConfigSpec, DominanceSpec, TaintSpec
from openultrasast.push.cache import ArtifactCache, SemanticKeys
@@ -211,7 +211,7 @@ def _valid_answer(scan: ModelScanResult, question: QuestionIdentity, operations:
return len(rows) == sum(op.question == question for op in operations)
def _context_boundaries(scan: ModelScanResult, question: QuestionIdentity | None = None) -> tuple[str, ...]:
def _context_boundaries(scan: ModelScanResult, *questions: QuestionIdentity) -> tuple[str, ...]:
# Aggregate coverage keeps every gap. A candidate may ignore only a recognized
# question-owned boundary attached to a different, recorded question. Unknown
# ownership and graph/declaration gaps remain global.
@@ -222,16 +222,43 @@ def _context_boundaries(scan: ModelScanResult, question: QuestionIdentity | None
"context_scope_empty:contributor-scan:",
"dynamic_external_or_depth_context_unresolved:contributor-scan:",
)
others = {q.identity.question_id for q in scan.scope.selected if q.identity != question}
others.update(q.identity.question_id for q in scan.scope.deferred if q.identity != question)
retained = set(questions)
others = {q.identity.question_id for q in scan.scope.selected if q.identity not in retained}
others.update(q.identity.question_id for q in scan.scope.deferred if q.identity not in retained)
return tuple(
boundary
for boundary in scan.scope.unresolved_boundaries
if not boundary.startswith("evidence_unknown:")
and not (question is not None and boundary.startswith(owned_prefixes) and boundary.rsplit(":", 1)[-1] in others)
and not (questions and boundary.startswith(owned_prefixes) and boundary.rsplit(":", 1)[-1] in others)
)
def _blocking_context(context: ChangeContext) -> tuple[str, ...]:
"""Transaction context gaps that block any comparison.
Correspondence ambiguity is excluded because it is now decided per operation: an
operation needing an ambiguous anchor is reported `ambiguous` and stays uncomparable,
while a uniquely mapped or newly introduced operation elsewhere in the same file is
unaffected. Every other gap still blocks and is still reported.
"""
return tuple(gap for gap in context.unresolved_boundaries if not gap.endswith(":" + AMBIGUOUS_CORRESPONDENCE))
def _required_scope(question: QuestionIdentity, context: ChangeContext) -> set[QuestionIdentity]:
"""Base questions this comparison requires: the counterpart and its recorded dependencies.
Completeness is demanded over exactly this scope. Every other gap stays in aggregate
coverage and is still reported; it is never discharged. An unrelated unsupported family
no longer prevents an otherwise complete target comparison, and unknown dependency
ownership remains inside the scope because it is not attributed to another question.
"""
required = {_counterpart(question, context)}
for relation in context.relationships:
if relation.target == question and relation.evidence:
required.add(_counterpart(relation.source, context))
return required
def _counterpart(question: QuestionIdentity, context: ChangeContext) -> QuestionIdentity:
renamed = {context.decode_path(r.head_path): context.decode_path(r.base_path) for r in context.renames}
return replace(question, path=renamed.get(question.path, question.path))
@@ -287,7 +314,7 @@ def compare_evidence(
and execution_budget is not None
and time.monotonic() < execution_budget.deadline_monotonic
and context.base_revision is not None
and not context.unresolved_boundaries
and not _blocking_context(context)
and head_semantics == base_semantics
and complete(head)
and complete(base)
@@ -387,24 +414,27 @@ def _compare_evidence(
identical = tuple(b for b in contextual if b.operation == op.operation)
change = _change_evidence(op, context)
reason = "base_incomplete"
required = _required_scope(op.question, context)
selected = {q.identity for q in base.scope.selected} if base is not None and base.scope is not None else set()
answered = {q.identity for q in base.question_outcomes if q.status == "completed"} if base is not None else set()
complete = bool(
base
and base.scope
and base.scope.population_complete
# A truncated base scan cannot establish absence, wherever it was truncated.
and all(question.reason == "tier_zero" for question in base.scope.deferred)
and not _context_boundaries(base)
and not _context_boundaries(base, *required)
and not base.degradations
and _valid_answer(base, counterpart, base_ops)
and counterpart in {q.identity for q in base.scope.selected}
and counterpart in {q.identity for q in base.question_outcomes if q.status == "completed"}
and {q.identity for q in base.scope.selected} == {q.identity for q in base.question_outcomes}
and all(q.status == "completed" for q in base.question_outcomes)
and required <= selected
and required <= answered
and {q.identity for q in base.question_outcomes} <= selected
and all(_valid_answer(base, question, base_ops) for question in required)
)
if head_semantics != base_semantics:
reason = "semantics_mismatch"
elif context.base_revision is None:
reason = "base_unavailable"
elif context.unresolved_boundaries:
elif _blocking_context(context):
reason = "change_context_unresolved"
elif (
not head.scope
+28
View File
@@ -169,3 +169,31 @@ def test_missing_base_without_adapter_diagnostic_stays_incomplete(tmp_path):
)
assert result.question_outcomes[0].status == "unresolved"
assert "comparison_base_unavailable" in result.scope.unresolved_boundaries
def test_correspondence_ambiguity_alone_does_not_make_every_question_incomplete(tmp_path):
"""M1b 10.3: repeated unchanged lines in one file no longer invalidate every question."""
(tmp_path / "source.php").write_text("readable\n")
ambiguous = replace(context(), unresolved_boundaries=(b"source.php".hex() + ":ambiguous_line_correspondence",))
result = scan_repository(
tmp_path,
[region("source.php", "php")],
backend=Backend({"source.php": vector() + [row("source.php"), {"kind": "context_summary"}]}),
change_context=ambiguous,
)
assert result.question_outcomes[0].status == "completed"
# The mapping itself is still withheld and still reported in aggregate coverage.
assert any("ambiguous_line_correspondence" in gap for gap in result.change_context.unresolved_boundaries)
def test_any_other_inherited_transaction_gap_still_marks_questions_incomplete(tmp_path):
(tmp_path / "source.php").write_text("readable\n")
blocked = replace(context(), unresolved_boundaries=("snapshot:lfs_blob_unavailable",))
result = scan_repository(
tmp_path,
[region("source.php", "php")],
backend=Backend({"source.php": vector() + [row("source.php"), {"kind": "context_summary"}]}),
change_context=blocked,
)
assert result.question_outcomes[0].status == "unresolved"
assert result.question_outcomes[0].reason == "change_context_incomplete"
+82
View File
@@ -342,3 +342,85 @@ def test_renamed_file_keeps_operation_identity_across_a_changed_line():
ctx = replace(changed_line_context(), renames=(PathRename("old.js", "a.js"),), spans=spans)
# Identical operation carried across the rename is movement, never novelty.
assert delta(scan(), scan(path="old.js"), ctx).novelty == "unknown"
def with_extra_question(result, *, status="completed", reason="answered", rows="[]", family="config_secrets", deferred=None):
"""Add an unrelated question to a scan, optionally unresolved or deferred."""
from openultrasast.model.contracts import DeferredQuestion
other = replace(result.scope.selected[0].identity, family=family)
scope = replace(
result.scope,
selected=(*result.scope.selected, RankedQuestion(other, 0.0, ())) if deferred is None else result.scope.selected,
deferred=(DeferredQuestion(other, deferred, ()),) if deferred is not None else result.scope.deferred,
)
added = QuestionOutcome(other, status, reason, rows)
outcomes = result.question_outcomes if deferred is not None else (*result.question_outcomes, added)
return replace(result, scope=scope, question_outcomes=outcomes), other
def test_unrelated_unsupported_family_no_longer_blocks_a_complete_target_comparison():
"""M1b 10.2: completeness is demanded over the required scope, not the whole repository."""
base, other = with_extra_question(scan(rows=[], findings=False), status="unresolved", reason="unsupported_family", rows="[]")
owned = "context_projection_unavailable:contributor-scan:" + other.question_id
base = replace(base, scope=replace(base.scope, unresolved_boundaries=(owned,)))
assert delta(scan(), base).novelty == "new"
def test_incomplete_counterpart_still_blocks():
base = scan(rows=[], status="unanswered", findings=False)
assert delta(scan(), base).novelty == "unknown"
own = scan(rows=[], findings=False)
owned = "context_projection_unavailable:contributor-scan:" + own.scope.selected[0].identity.question_id
own = replace(own, scope=replace(own.scope, unresolved_boundaries=(owned,)))
assert delta(scan(), own).novelty == "unknown"
def test_unknown_ownership_boundary_stays_inside_the_required_scope():
base, _ = with_extra_question(scan(rows=[], findings=False))
base = replace(base, scope=replace(base.scope, unresolved_boundaries=("graph_incomplete",)))
assert delta(scan(), base).novelty == "unknown"
def test_truncated_base_cannot_establish_absence_wherever_it_was_truncated():
base, _ = with_extra_question(scan(rows=[], findings=False), deferred="budget_exhausted")
assert delta(scan(), base).novelty == "unknown"
base, _ = with_extra_question(scan(rows=[], findings=False), deferred="tier_zero")
assert delta(scan(), base).novelty == "new"
def test_a_recorded_dependency_of_the_counterpart_must_also_be_complete():
from openultrasast.model.contracts import AffectedRelationship
head = scan()
target = head.scope.selected[0].identity
helper = replace(target, path="helper.js", function="carry")
ctx = replace(context(), relationships=(AffectedRelationship(helper, target, "callee", ("head_span:a.js:4-4",)),))
base = scan(rows=[], findings=False)
# The dependency is not answered in the base, so the required scope is incomplete.
assert delta(head, base, ctx).novelty == "unknown"
extended = replace(
base,
scope=replace(base.scope, selected=(*base.scope.selected, RankedQuestion(helper, 0.0, ()))),
question_outcomes=(*base.question_outcomes, QuestionOutcome(helper, "completed", "answered", "[]")),
)
assert delta(head, extended, ctx).novelty == "new"
def test_repeated_unchanged_lines_do_not_invalidate_a_distinct_operation():
"""M1b 10.3: file-wide ambiguity is scoped to the operations whose anchor needs it."""
ambiguous = "612e6a73:" + "ambiguous_line_correspondence"
ctx = replace(changed_line_context(), unresolved_boundaries=(ambiguous,))
assert delta(scan(), scan(rows=[], findings=False), ctx).novelty == "new"
def test_an_operation_needing_an_ambiguous_anchor_is_still_uncomparable():
ambiguous = "612e6a73:" + "ambiguous_line_correspondence"
ctx = replace(changed_line_context(ambiguous=True), unresolved_boundaries=(ambiguous,))
assert delta(scan(), scan(rows=[], findings=False), ctx).novelty == "unknown"
def test_any_other_transaction_context_gap_still_blocks():
ctx = replace(changed_line_context(), unresolved_boundaries=("snapshot:lfs_blob_unavailable",))
candidate = delta(scan(), scan(rows=[], findings=False), ctx)
assert (candidate.novelty, candidate.reason) == ("unknown", "change_context_unresolved")