mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 11:49:28 +00:00
feat(pre-push-safety-net): classify an operation the change introduced (M1b 10.1)
M1a measured that a newly introduced operation sits on a changed line, has no base line anchor, and therefore stayed unclassified even with every completeness veto lifted. Establish absence over the complete counterpart answer instead of over a line anchor. Correspondence now reports why a head operation does not map: `changed_line` inside a recorded head edit, `ambiguous` where several base lines could correspond, or `unmapped` where no anchor exists. Only the first supports reasoning about an introduced operation, and only when the base counterpart answer is complete and contains no operation with that spelling. Movement is matched on the operation spelling alone, so a benign identifier rename that changes only the source text reads as movement rather than a new flow. The existing rename control caught that; the key was too brittle before. One prior control is superseded and updated with its reason: a new operation on a changed line in a comparable existing file is now new rather than unknown. Movement, ambiguity, unmapped operations, discharged or unentailed claims and an incomplete or failed base all remain unknown. 1,337 tests passed, nine skipped. No capability enabled; rollout stays NO-GO. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
co-authored by
Claude Opus 5
parent
3224884d8f
commit
7e0d01d9e8
@@ -266,6 +266,48 @@ Preserve all completed task evidence below.
|
||||
- _Requirements: 1.1, 2.1, 2.4, 3.4, 6.3, 7.1, 7.3_
|
||||
- _Verified 2026-09-16:_ all three pinned revisions replayed through the production CLI under a named 900-second development budget; expectations frozen on disk first. Raw `<lambda>2` witnesses present at lines 32–34; fixed twin analyzed and quiet (target question answered with zero rows, no deadline reason, three base-only findings); unchanged revision analyzed and recorded `unchanged`. Five vetoes fired and were recorded; the remaining blocker is `operation_correspondence_unresolved` for an operation on a changed line. Zero admitted defects. Evidence: [M1a record](m1a-recorded-vetoes.md) and `benchmarks/measurements/2026-09-16-nodegoat-m1a-recorded-vetoes.json`. Diagnosis only; no capability enabled and no latency claim.
|
||||
|
||||
- [ ] 10. Repair only the vetoes M1a recorded (release milestone M1b)
|
||||
- [ ] 10.1 Establish operation correspondence for an operation on a changed line
|
||||
- M1a measured this as the blocker that survives every other repair: a newly introduced operation sits on a changed line, has no base line anchor, and therefore cannot be classified even when every completeness veto is lifted.
|
||||
- Bind the required base/head correspondence to the specific operation identity rather than to a line anchor. A complete base answer for the counterpart question that contains no operation with the same operation and source establishes absence; a matching operation elsewhere in that answer is movement, not novelty.
|
||||
- Keep the existing conservative rules: an incomplete or failed base stays unknown, benign movement and renames stay unchanged, and lexical alignment is never semantic proof.
|
||||
- Add absent, renamed, removed-guard, moved-operation and repeated-identical-operation controls. Done when a newly introduced operation on a changed line is classified new, while movement, renames and an incomplete base remain unchanged or unknown.
|
||||
- _Boundary: Delta and Actionability Policy_
|
||||
- _Requirements: 2.4, 3.1, 3.4_
|
||||
- [ ] 10.2 Scope correspondence ambiguity to the operations it actually affects
|
||||
- `ambiguous_line_correspondence` is recorded file-wide and inherited by every question, so repeated unchanged lines invalidate distinct, uniquely mapped operations elsewhere.
|
||||
- Retain ambiguous mappings as unavailable and record which regions are ambiguous, so only an operation whose required correspondence needs an ambiguous line is vetoed. Do not delete the ambiguity check.
|
||||
- Done when a uniquely mapped operation is comparable in a file that also contains repeated unchanged lines, while an operation needing an ambiguous line stays unresolved.
|
||||
- _Boundary: Snapshot Adapter, Delta and Actionability Policy_
|
||||
- _Depends: 10.1_
|
||||
- _Requirements: 2.4, 3.4_
|
||||
- [ ] 10.3 Make the external and dynamic context boundary question-owned
|
||||
- M1a recorded `dynamic_external_or_depth_context_unresolved` on the target's own question. The flag is computed over every in-scope method, so one unresolved call anywhere vetoes every question, including those whose relevant methods are fully resolved.
|
||||
- Report the boundary with the methods whose destinations are unresolved, and inherit it into a question only when such a method participates in that question's change relationships. Unknown ownership remains a blocker; absent evidence of independence is not proof of independence.
|
||||
- Done when a question whose relevant methods resolve completely is not vetoed by an unrelated external call, while a question that depends on an unresolved destination stays unresolved.
|
||||
- _Boundary: contributor-scan query/context evidence, Ranker Scope Contract_
|
||||
- _Requirements: 2.1, 2.2, 8.1, 8.2_
|
||||
- [ ] 10.4 Stop vendor exclusion acting as a global completion veto
|
||||
- `vendor_semantics_unresolved` currently degrades every otherwise complete outcome to `graph_incomplete`, including on a complete 44-file first-party census with successful query receipts.
|
||||
- Keep the exclusion physical and keep the coverage note. Let the question-owned dependency evidence from 10.3 carry whether a specific answer needed excluded semantics; a recorded dependency on an excluded path still blocks that question.
|
||||
- Done when an unrelated vendor exclusion no longer invalidates a first-party answer, while a question with evidence of dependency on excluded code remains unresolved and the exclusion stays out of targets and graphs.
|
||||
- _Boundary: contributor-scan graph/scope evidence, Cache and Engine Boundary_
|
||||
- _Depends: 10.3_
|
||||
- _Requirements: 2.2, 2.3, 8.2, 8.4_
|
||||
- [ ] 10.5 Carry query-to-operation provenance into the reported evidence
|
||||
- The target is selected through a file-fallback question with `function=None` while rows report method `<lambda>2`. M1a showed this did not block the comparison once the question completed, so it is provenance quality rather than the blocker.
|
||||
- Record verified query-to-operation provenance on the evidence; do not inject benchmark function names into regions or count coincident sites.
|
||||
- Done when a reported operation names the question that produced it and the method the engine reported, with unresolved provenance explicit.
|
||||
- _Boundary: Delta and Actionability Policy, contributor-scan normalization_
|
||||
- _Requirements: 3.1, 3.4, 6.3_
|
||||
- [ ] 10.6 Rerun the three revisions through the production rules
|
||||
- Replay the same pinned vulnerable, fixed and unchanged revisions **without** the recorded-veto flag under the named development budget. Demonstrate a developer-readable evaluation finding with a supported consequence, exact location, witness, change attribution and repair direction.
|
||||
- Verify the fixed twin is analyzed and produces no actionable defect and the unchanged revision is not reported as a new regression. Keep qualification eligibility disabled; experimental evaluation output cannot advertise an unqualified hook alert.
|
||||
- Done when one replayable record states which vetoes were repaired and which remain, with completed required checks and readable source and census receipts.
|
||||
- _Boundary: Evaluation harness, full-run integration_
|
||||
- _Depends: 10.1, 10.2, 10.3, 10.4, 10.5_
|
||||
- _Requirements: 1.1, 2.1, 2.4, 3.1, 3.4, 6.3, 7.1, 7.3_
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- Task5.2 measured PMPro timing out after both snapshots were materialized but before change context/graph analysis. Group6 must account for compatible snapshot/discovery reuse as well as graph reuse; identical-tip hits cannot substitute for the later representative changed-code latency gate. No new detector, scope heuristic or incremental CPG mutation is implied.
|
||||
|
||||
@@ -136,18 +136,33 @@ def _operations(scan: ModelScanResult, budget: ExecutionBudget | None = None) ->
|
||||
return tuple(result)
|
||||
|
||||
|
||||
def _base_location(op: EvidenceOperation, context: ChangeContext) -> tuple[str, int] | None:
|
||||
def _correspondence(op: EvidenceOperation, context: ChangeContext) -> tuple[tuple[str, int] | None, str]:
|
||||
"""Where this head operation maps in the base, and when it does not, why.
|
||||
|
||||
`changed_line` means the operation lies inside a recorded head edit, so no base line
|
||||
can correspond to it. That is a different fact from `ambiguous`, where several base
|
||||
lines could correspond, and from `unmapped`, where no anchor was recorded at all.
|
||||
Only the first supports reasoning about an operation the change introduced.
|
||||
"""
|
||||
matches = []
|
||||
for anchor in context.line_correspondences:
|
||||
size = anchor.base_end_line - anchor.base_start_line
|
||||
if context.decode_path(anchor.head_path) == op.path and anchor.head_start_line <= op.line <= anchor.head_start_line + size:
|
||||
matches.append((context.decode_path(anchor.base_path), anchor.base_start_line + op.line - anchor.head_start_line))
|
||||
if matches:
|
||||
return matches[0] if len(set(matches)) == 1 else None
|
||||
return (matches[0], "anchored") if len(set(matches)) == 1 else (None, "ambiguous")
|
||||
changed = {context.decode_path(p) for p in context.changed_paths}
|
||||
if op.path not in changed:
|
||||
return op.path, op.line
|
||||
return None
|
||||
return (op.path, op.line), "unchanged_path"
|
||||
introduced = any(
|
||||
span.side == "head" and context.decode_path(span.path) == op.path and span.start_line <= op.line <= span.end_line
|
||||
for span in context.spans
|
||||
)
|
||||
return None, ("changed_line" if introduced else "unmapped")
|
||||
|
||||
|
||||
def _base_location(op: EvidenceOperation, context: ChangeContext) -> tuple[str, int] | None:
|
||||
return _correspondence(op, context)[0]
|
||||
|
||||
|
||||
def _change_evidence(op: EvidenceOperation, context: ChangeContext) -> tuple[str, ...]:
|
||||
@@ -361,10 +376,15 @@ def _compare_evidence(
|
||||
old: tuple[EvidenceOperation, ...] = ()
|
||||
change: tuple[str, ...] = ()
|
||||
if op is not None:
|
||||
location = _base_location(op, context)
|
||||
location, correspondence = _correspondence(op, context)
|
||||
counterpart = _counterpart(op.question, context)
|
||||
contextual = tuple(b for b in base_ops if b.question == counterpart and b.mechanism == op.mechanism)
|
||||
old = tuple(b for b in contextual if (b.path, b.line) == location and b.operation == op.operation)
|
||||
# Identity, not position. An operation the change introduced has no base line to
|
||||
# map to, so absence must be established over the complete counterpart answer.
|
||||
# Match on the operation spelling alone: a benign identifier rename changes the
|
||||
# source text without introducing anything, and must read as movement.
|
||||
identical = tuple(b for b in contextual if b.operation == op.operation)
|
||||
change = _change_evidence(op, context)
|
||||
reason = "base_incomplete"
|
||||
complete = bool(
|
||||
@@ -419,6 +439,14 @@ def _compare_evidence(
|
||||
and not op.discharged
|
||||
):
|
||||
novelty, reason = "new", "source_connection_absent_from_comparable_base"
|
||||
elif correspondence == "changed_line" and change and op.mechanism == "taint":
|
||||
# The edit introduced these lines. A complete counterpart answer that
|
||||
# contains this operation elsewhere is movement, never novelty; only its
|
||||
# total absence establishes that the change introduced the flow.
|
||||
if identical:
|
||||
reason = "operation_moved_within_change"
|
||||
elif finding.rung == Rung.ENTAILED and not op.discharged:
|
||||
novelty, reason = "new", "operation_absent_from_comparable_base"
|
||||
results.append(_candidate(finding, op, old, novelty, reason, change, context))
|
||||
return tuple(
|
||||
replace(
|
||||
|
||||
@@ -151,8 +151,11 @@ def test_new_source_at_unchanged_operation_needs_complete_base_answer():
|
||||
|
||||
|
||||
def test_changed_new_operation_in_comparable_existing_file():
|
||||
# M1b 10.1 changed the first case: an operation the edit introduced has no base line to
|
||||
# map to, so a complete counterpart answer containing no such operation now establishes
|
||||
# absence. Before that repair this stayed unknown and no introduced flow could be seen.
|
||||
ctx = replace(context(), spans=(ChangedSpan("a.js", 4, 4, "head"),), line_correspondences=())
|
||||
assert delta(scan(), scan(rows=[], findings=False), ctx).novelty == "unknown"
|
||||
assert delta(scan(), scan(rows=[], findings=False), ctx).novelty == "new"
|
||||
assert delta(scan(), scan(line=6), ctx).novelty == "unknown" # movement is not new
|
||||
|
||||
|
||||
@@ -272,3 +275,70 @@ def test_comparison_cache_reuses_only_completed_compatible_evidence(tmp_path, mo
|
||||
hits = cache.hits
|
||||
policy.compare_evidence(scan(), scan(status="unanswered"), **args)
|
||||
assert cache.hits == hits
|
||||
|
||||
|
||||
def changed_line_context(*, ambiguous=False, head_line=4):
|
||||
"""The head edit covers the operation's line, so no base line can correspond to it."""
|
||||
return ChangeContext(
|
||||
"base",
|
||||
"head",
|
||||
("a.js",),
|
||||
(),
|
||||
(ChangedSpan("a.js", 2, 2, "base"), ChangedSpan("a.js", head_line, head_line, "head")),
|
||||
(),
|
||||
(),
|
||||
(),
|
||||
(),
|
||||
line_correspondences=(
|
||||
(LineCorrespondence("a.js", "a.js", 9, 9, head_line), LineCorrespondence("a.js", "a.js", 11, 11, head_line))
|
||||
if ambiguous
|
||||
else ()
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def test_operation_introduced_on_a_changed_line_is_new_against_a_complete_base():
|
||||
"""M1b 10.1: absence is established over the counterpart answer, not over a line anchor."""
|
||||
candidate = delta(scan(), scan(rows=[], findings=False), changed_line_context())
|
||||
assert (candidate.novelty, candidate.reason) == ("new", "operation_absent_from_comparable_base")
|
||||
|
||||
|
||||
def test_moved_identical_operation_on_a_changed_line_is_not_new():
|
||||
moved = scan(line=9) # same operation and source, different line in the base
|
||||
candidate = delta(scan(), moved, changed_line_context())
|
||||
assert candidate.novelty == "unknown" and candidate.reason == "operation_moved_within_change"
|
||||
|
||||
|
||||
def test_changed_line_novelty_still_requires_a_complete_base():
|
||||
empty = scan(rows=[], findings=False)
|
||||
for base in (scan(rows=[], status="unanswered", findings=False), replace(empty, degradations=({"reason": "cpg_empty"},))):
|
||||
assert delta(scan(), base, changed_line_context()).novelty == "unknown"
|
||||
assert delta(scan(), None, changed_line_context()).novelty == "unknown"
|
||||
|
||||
|
||||
def test_ambiguous_correspondence_on_a_changed_line_is_never_new():
|
||||
candidate = delta(scan(), scan(rows=[], findings=False), changed_line_context(ambiguous=True))
|
||||
assert candidate.novelty == "unknown" and candidate.reason != "operation_absent_from_comparable_base"
|
||||
|
||||
|
||||
def test_operation_outside_any_recorded_head_span_is_not_new():
|
||||
"""An unmapped operation in a changed file is unresolved, not introduced."""
|
||||
candidate = delta(scan(), scan(rows=[], findings=False), changed_line_context(head_line=99))
|
||||
assert candidate.novelty == "unknown" and candidate.reason == "operation_correspondence_unresolved"
|
||||
|
||||
|
||||
def test_discharged_or_unentailed_operation_on_a_changed_line_is_not_new():
|
||||
from openultrasast.model.pipeline import ModelFinding
|
||||
|
||||
assert delta(scan(sanitized=True), scan(rows=[], findings=False), changed_line_context()).novelty != "new"
|
||||
weak = scan()
|
||||
weak = replace(weak, findings=(replace(weak.findings[0], rung=Rung.CORROBORATED),))
|
||||
assert delta(weak, scan(rows=[], findings=False), changed_line_context()).novelty != "new"
|
||||
assert isinstance(weak.findings[0], ModelFinding)
|
||||
|
||||
|
||||
def test_renamed_file_keeps_operation_identity_across_a_changed_line():
|
||||
spans = (ChangedSpan("old.js", 2, 2, "base"), ChangedSpan("a.js", 4, 4, "head"))
|
||||
ctx = replace(changed_line_context(), renames=(PathRename("old.js", "a.js"),), spans=spans)
|
||||
# Identical operation carried across the rename is movement, never novelty.
|
||||
assert delta(scan(), scan(path="old.js"), ctx).novelty == "unknown"
|
||||
|
||||
Reference in New Issue
Block a user