mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-18 12:39:28 +00:00
feat(pre-push-safety-net): scope comparison completeness and ambiguity (M1b 10.2, 10.3)
Two vetoes M1a recorded were global where the evidence is local. Comparison required every selected base question to complete, so one unsupported family anywhere prevented an otherwise complete target comparison. Completeness is now demanded over the required scope: the counterpart question plus the dependencies the change context actually recorded for it. Every other gap stays in aggregate coverage and is still reported, never discharged. An incomplete counterpart, an incomplete recorded dependency, an unknown-ownership boundary and a truncated base scan all still yield unknown. Correspondence ambiguity was recorded file-wide and inherited by every question, so repeated unchanged lines in one file invalidated every question in the repository and blocked the whole transaction comparison. It is now decided per operation by the correspondence classifier: an operation needing an ambiguous anchor stays uncomparable, while a uniquely mapped or newly introduced operation in the same file is unaffected. The mapping is still withheld and still reported. On the NodeGoat case the ambiguity is on the target's own file, so file-level scoping would not have been enough; only per-operation resolution is. 1,347 tests passed, nine skipped. No capability enabled; rollout stays NO-GO. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
co-authored by
Claude Opus 5
parent
7e0d01d9e8
commit
8ff1d9a638
@@ -267,45 +267,56 @@ Preserve all completed task evidence below.
|
||||
- _Verified 2026-09-16:_ all three pinned revisions replayed through the production CLI under a named 900-second development budget; expectations frozen on disk first. Raw `<lambda>2` witnesses present at lines 32–34; fixed twin analyzed and quiet (target question answered with zero rows, no deadline reason, three base-only findings); unchanged revision analyzed and recorded `unchanged`. Five vetoes fired and were recorded; the remaining blocker is `operation_correspondence_unresolved` for an operation on a changed line. Zero admitted defects. Evidence: [M1a record](m1a-recorded-vetoes.md) and `benchmarks/measurements/2026-09-16-nodegoat-m1a-recorded-vetoes.json`. Diagnosis only; no capability enabled and no latency claim.
|
||||
|
||||
- [ ] 10. Repair only the vetoes M1a recorded (release milestone M1b)
|
||||
- [ ] 10.1 Establish operation correspondence for an operation on a changed line
|
||||
- [x] 10.1 Establish operation correspondence for an operation on a changed line
|
||||
- M1a measured this as the blocker that survives every other repair: a newly introduced operation sits on a changed line, has no base line anchor, and therefore cannot be classified even when every completeness veto is lifted.
|
||||
- Bind the required base/head correspondence to the specific operation identity rather than to a line anchor. A complete base answer for the counterpart question that contains no operation with the same operation and source establishes absence; a matching operation elsewhere in that answer is movement, not novelty.
|
||||
- Keep the existing conservative rules: an incomplete or failed base stays unknown, benign movement and renames stay unchanged, and lexical alignment is never semantic proof.
|
||||
- Add absent, renamed, removed-guard, moved-operation and repeated-identical-operation controls. Done when a newly introduced operation on a changed line is classified new, while movement, renames and an incomplete base remain unchanged or unknown.
|
||||
- _Boundary: Delta and Actionability Policy_
|
||||
- _Requirements: 2.4, 3.1, 3.4_
|
||||
- [ ] 10.2 Scope correspondence ambiguity to the operations it actually affects
|
||||
- _Verified 2026-09-16:_ 1,337 tests passed, nine skipped; Ruff/format/mypy passed. Correspondence now reports `anchored`, `ambiguous`, `changed_line`, `unmapped` or `unchanged_path`, and only `changed_line` supports an introduced operation. Movement keys on the operation spelling, which the existing benign-rename control proved necessary. One superseded control was updated with its reason; movement, ambiguity, unmapped operations, discharged or unentailed claims and an incomplete base all remain unknown.
|
||||
- [x] 10.2 Establish the required comparison scope explicitly
|
||||
- Comparison currently requires every selected base question to complete with no base degradation and no non-tier-zero deferred question, so one unsupported family anywhere prevents an otherwise complete target comparison. The owner research names this and requires the amendment to state the required scope rather than relax the bar.
|
||||
- Define the comparison's required scope as the counterpart question plus the questions its evidence actually depends on, and require completeness over exactly that scope. Retain every other gap in aggregate coverage; a question outside the required scope stays reported and unresolved, never silently discharged.
|
||||
- A missing raw finding in an incomplete base still cannot establish novelty. Unknown dependency remains inside the required scope.
|
||||
- Done when an unrelated unsupported family no longer prevents a complete target comparison, while an incomplete counterpart, an incomplete dependency of it, or unknown dependency ownership still yields unknown.
|
||||
- _Boundary: Delta and Actionability Policy_
|
||||
- _Depends: 10.1_
|
||||
- _Requirements: 2.2, 2.4, 3.4, 6.4_
|
||||
- _Verified 2026-09-16:_ 1,342 tests passed, nine skipped; Ruff/format/mypy passed. Completeness is now demanded over the counterpart plus its recorded dependencies. Controls prove an unrelated unsupported family no longer blocks a complete target comparison, while an incomplete counterpart, an incomplete recorded dependency, an unknown-ownership boundary and a truncated base scan all still yield unknown. Every other gap stays in aggregate coverage.
|
||||
- [x] 10.3 Scope correspondence ambiguity to the operations it actually affects
|
||||
- `ambiguous_line_correspondence` is recorded file-wide and inherited by every question, so repeated unchanged lines invalidate distinct, uniquely mapped operations elsewhere.
|
||||
- Retain ambiguous mappings as unavailable and record which regions are ambiguous, so only an operation whose required correspondence needs an ambiguous line is vetoed. Do not delete the ambiguity check.
|
||||
- Done when a uniquely mapped operation is comparable in a file that also contains repeated unchanged lines, while an operation needing an ambiguous line stays unresolved.
|
||||
- _Boundary: Snapshot Adapter, Delta and Actionability Policy_
|
||||
- _Depends: 10.1_
|
||||
- _Depends: 10.2_
|
||||
- _Requirements: 2.4, 3.4_
|
||||
- [ ] 10.3 Make the external and dynamic context boundary question-owned
|
||||
- _Verified 2026-09-16:_ 1,347 tests passed, nine skipped; Ruff/format/mypy passed. Ambiguity is no longer inherited as a context gap by every question, and no longer blocks the transaction comparison; it is decided per operation by the correspondence classifier. Controls prove a uniquely mapped or newly introduced operation stays comparable in a file with repeated unchanged lines, an operation needing an ambiguous anchor stays unresolved, and any other transaction gap still blocks. The mapping is still withheld and still reported in aggregate coverage.
|
||||
- [ ] 10.4 Make the external and dynamic context boundary question-owned
|
||||
- M1a recorded `dynamic_external_or_depth_context_unresolved` on the target's own question. The flag is computed over every in-scope method, so one unresolved call anywhere vetoes every question, including those whose relevant methods are fully resolved.
|
||||
- Report the boundary with the methods whose destinations are unresolved, and inherit it into a question only when such a method participates in that question's change relationships. Unknown ownership remains a blocker; absent evidence of independence is not proof of independence.
|
||||
- Done when a question whose relevant methods resolve completely is not vetoed by an unrelated external call, while a question that depends on an unresolved destination stays unresolved.
|
||||
- _Boundary: contributor-scan query/context evidence, Ranker Scope Contract_
|
||||
- _Requirements: 2.1, 2.2, 8.1, 8.2_
|
||||
- [ ] 10.4 Stop vendor exclusion acting as a global completion veto
|
||||
- [ ] 10.5 Stop vendor exclusion acting as a global completion veto
|
||||
- `vendor_semantics_unresolved` currently degrades every otherwise complete outcome to `graph_incomplete`, including on a complete 44-file first-party census with successful query receipts.
|
||||
- Keep the exclusion physical and keep the coverage note. Let the question-owned dependency evidence from 10.3 carry whether a specific answer needed excluded semantics; a recorded dependency on an excluded path still blocks that question.
|
||||
- Done when an unrelated vendor exclusion no longer invalidates a first-party answer, while a question with evidence of dependency on excluded code remains unresolved and the exclusion stays out of targets and graphs.
|
||||
- _Boundary: contributor-scan graph/scope evidence, Cache and Engine Boundary_
|
||||
- _Depends: 10.3_
|
||||
- _Depends: 10.4_
|
||||
- _Requirements: 2.2, 2.3, 8.2, 8.4_
|
||||
- [ ] 10.5 Carry query-to-operation provenance into the reported evidence
|
||||
- [ ] 10.6 Carry query-to-operation provenance into the reported evidence
|
||||
- The target is selected through a file-fallback question with `function=None` while rows report method `<lambda>2`. M1a showed this did not block the comparison once the question completed, so it is provenance quality rather than the blocker.
|
||||
- Record verified query-to-operation provenance on the evidence; do not inject benchmark function names into regions or count coincident sites.
|
||||
- Done when a reported operation names the question that produced it and the method the engine reported, with unresolved provenance explicit.
|
||||
- _Boundary: Delta and Actionability Policy, contributor-scan normalization_
|
||||
- _Requirements: 3.1, 3.4, 6.3_
|
||||
- [ ] 10.6 Rerun the three revisions through the production rules
|
||||
- [ ] 10.7 Rerun the three revisions through the production rules
|
||||
- Replay the same pinned vulnerable, fixed and unchanged revisions **without** the recorded-veto flag under the named development budget. Demonstrate a developer-readable evaluation finding with a supported consequence, exact location, witness, change attribution and repair direction.
|
||||
- Verify the fixed twin is analyzed and produces no actionable defect and the unchanged revision is not reported as a new regression. Keep qualification eligibility disabled; experimental evaluation output cannot advertise an unqualified hook alert.
|
||||
- Done when one replayable record states which vetoes were repaired and which remain, with completed required checks and readable source and census receipts.
|
||||
- _Boundary: Evaluation harness, full-run integration_
|
||||
- _Depends: 10.1, 10.2, 10.3, 10.4, 10.5_
|
||||
- _Depends: 10.1, 10.2, 10.3, 10.4, 10.5, 10.6_
|
||||
- _Requirements: 1.1, 2.1, 2.4, 3.1, 3.4, 6.3, 7.1, 7.3_
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
Reference in New Issue
Block a user