mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
feat(pre-push-safety-net): measure M1b step 1 and record the inventory prerequisite
Rerunning the three NodeGoat revisions after tasks 10.1-10.3 shows the recorded view now classifies the case: the vulnerable revision reaches new with operation_absent_from_comparable_base, and the unchanged revision reaches unchanged with its base location resolved to the same lines. Both previously stopped at operation_correspondence_unresolved and witness_identity_unresolved. The rerun also caught a defect in the repair itself: the new novelty reason was missing from the admission allowlist, so admission rejected the candidate as change_unsupported. Fixed with a control. The production path is still blocked by the last two vetoes, and the measurement shows neither can be scoped or relaxed as the plan described. A taint answer reports traced flows, so an absent row means either that the operation does not exist in the base or that no traced flow reached it. dynamic_external_or_depth_context_unresolved exists to stop the second being read as the first, and removing it would turn a wrong absence into a wrong new finding. It is also computed over every in-scope method, so on any real Express application it is always true and can never be discharged, which is why the tool reports nothing on real projects. The resolution is new evidence rather than a relaxed rule: export the operations the query actually enumerated, separately from the flows it traced, so absence is established from the inventory while a traced-flow claim still requires complete reachability. That is now task 10.4, ahead of the two vetoes it unblocks, and it adds no detector, alias resolver or second source IR. 1,348 tests passed, nine skipped. No capability enabled; rollout stays NO-GO. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
co-authored by
Claude Opus 5
Claude Opus 5
parent
8ff1d9a638
commit
86980d8aa0
@@ -114,3 +114,56 @@ data; it cannot qualify production Node support, and Node evidence cannot qualif
|
||||
The lifted view is a diagnostic reconstruction, not a comparison result: it shows what the
|
||||
existing evidence would support if those vetoes were repaired, and no repair has been made.
|
||||
Runtime, independent qualification and rollout all remain NO-GO.
|
||||
|
||||
## M1b step 1: what the first three repairs changed
|
||||
|
||||
Rerun 2026-09-16 after tasks 10.1–10.3, same instrument, same pins, same budget.
|
||||
Evidence: [summary](../../../benchmarks/measurements/2026-09-16-nodegoat-m1b-step1.json)
|
||||
and its [bundle](../../../benchmarks/measurements/2026-09-16-nodegoat-m1b-step1.json.gz).
|
||||
|
||||
The recorded view now classifies the case correctly. The vulnerable revision reaches
|
||||
`new` / `operation_absent_from_comparable_base`, and the unchanged revision reaches
|
||||
`unchanged` / `same_supported_mechanism` with its base location resolved to the same lines.
|
||||
Before the repairs both stopped at `operation_correspondence_unresolved` and
|
||||
`witness_identity_unresolved`.
|
||||
|
||||
| Revision | Before 10.1–10.3 | After 10.1–10.3 |
|
||||
|---|---|---|
|
||||
| vulnerable | unknown, `operation_correspondence_unresolved` | new, `operation_absent_from_comparable_base` |
|
||||
| unchanged | unknown, `witness_identity_unresolved` | unchanged, `same_supported_mechanism` |
|
||||
|
||||
The rerun also caught a defect in the repair itself: the new novelty reason was missing from
|
||||
the admission allowlist, so admission rejected the candidate as `change_unsupported`. Fixed
|
||||
with a control.
|
||||
|
||||
The production path is unchanged so far, because two vetoes still demote the target
|
||||
questions: `change_context_incomplete` on the head from the question's own
|
||||
`dynamic_external_or_depth_context_unresolved`, and `graph_incomplete` on the base from the
|
||||
vendor exclusion.
|
||||
|
||||
## Why the last two vetoes need new query evidence
|
||||
|
||||
Neither remaining veto can be scoped or relaxed soundly as the plan originally described,
|
||||
and the rerun is what showed it.
|
||||
|
||||
A taint answer reports traced flows. An absent row therefore means either that the operation
|
||||
does not exist in the base, or that it exists and no traced flow reached it.
|
||||
`dynamic_external_or_depth_context_unresolved` exists precisely to stop the second being read
|
||||
as the first. Removing or narrowing it would let an unreached operation look absent, which
|
||||
turns a wrong absence into a wrong new finding: the false-positive direction this project
|
||||
cannot afford.
|
||||
|
||||
At the same time the flag carries no discrimination. It is computed over every in-scope
|
||||
method, and on any real Express application ordinary framework and dependency calls have
|
||||
unresolved destinations, so it is always true and can never be discharged. A veto that is
|
||||
always on and can never be lifted is why the tool reports nothing on real projects.
|
||||
|
||||
The resolution is neither to keep it nor to drop it, but to supply the evidence that makes
|
||||
absence provable independently of reachability: export the operations the query actually
|
||||
enumerated in the question's scope, separately from the flows it traced. Absence is then
|
||||
established from the inventory, while a traced-flow claim still requires complete
|
||||
reachability. Vendor exclusion can then stop acting as a global completion veto for the same
|
||||
reason, since dependency on excluded code becomes visible in the evidence rather than assumed.
|
||||
|
||||
That prerequisite is now task 10.4, ahead of the two vetoes it unblocks. It adds no detector,
|
||||
alias resolver or second source IR.
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"feature_name": "pre-push-safety-net",
|
||||
"created_at": "2026-09-12T15:35:00Z",
|
||||
"updated_at": "2026-09-16T13:30:00+00:00",
|
||||
"updated_at": "2026-09-16T14:05:00+00:00",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"approvals": {
|
||||
@@ -125,7 +125,10 @@
|
||||
"8.2",
|
||||
"8.4",
|
||||
"9.1",
|
||||
"9.2"
|
||||
"9.2",
|
||||
"10.1",
|
||||
"10.2",
|
||||
"10.3"
|
||||
],
|
||||
"remaining_tasks": 1,
|
||||
"verification_record": "implementation-group-8.md",
|
||||
@@ -139,7 +142,7 @@
|
||||
"implementation-group-7.md",
|
||||
"implementation-group-8.md"
|
||||
],
|
||||
"next_group": 9,
|
||||
"next_group": 10,
|
||||
"blocked_tasks": [
|
||||
"8.3"
|
||||
],
|
||||
@@ -165,6 +168,17 @@
|
||||
"measurement": "../../../benchmarks/measurements/2026-09-16-nodegoat-m1a-recorded-vetoes.json",
|
||||
"kill_criterion_triggered": false,
|
||||
"result": "Raw <lambda>2 witnesses present at contributions.js 32-34; fixed twin analyzed and quiet (target question answered with zero rows, no deadline reason); unchanged revision recorded 'unchanged'. Five vetoes fired and were recorded. With all five lifted the detection stops at operation_correspondence_unresolved, a fifth veto not previously named. No defect admitted, no capability enabled."
|
||||
},
|
||||
"m1b": {
|
||||
"status": "IN_PROGRESS",
|
||||
"verified": [
|
||||
"10.1",
|
||||
"10.2",
|
||||
"10.3"
|
||||
],
|
||||
"blocked_on": "10.4",
|
||||
"measurement": "../../../benchmarks/measurements/2026-09-16-nodegoat-m1b-step1.json",
|
||||
"finding": "The recorded view now reaches new on the vulnerable revision and unchanged on the unchanged revision. The last two vetoes cannot be scoped or relaxed soundly: a taint answer reports traced flows, so an absent row means either the operation does not exist or no flow reached it. The veto preventing that confusion is computed over every in-scope method and is always true on a real Express application, so it can neither be discharged nor safely removed. Task 10.4 exports the enumerated operation inventory separately from traced flows so absence is provable, and now precedes the two vetoes it unblocks."
|
||||
}
|
||||
},
|
||||
"release": {
|
||||
@@ -192,6 +206,7 @@
|
||||
],
|
||||
"implementation_design_approval": "Existing approvals preserved; this records milestone order and exit criteria, not blanket approval of future design changes.",
|
||||
"m1_split_accepted": "2026-09-14: maintainer asked to split M1 into M1a/M1b after review",
|
||||
"m1a_outcome": "Met inside the one-week box. Corrected the owner diagnosis: all four named vetoes fire, and operation correspondence for an operation on a changed line is the blocker that survives every other repair."
|
||||
"m1a_outcome": "Met inside the one-week box. Corrected the owner diagnosis: all four named vetoes fire, and operation correspondence for an operation on a changed line is the blocker that survives every other repair.",
|
||||
"m1b_position": "10.1-10.3 verified and measured; 10.4 operation inventory is the new prerequisite for 10.5 and 10.6."
|
||||
}
|
||||
}
|
||||
|
||||
@@ -7,8 +7,16 @@ it does not approve normal hook capabilities or production enforcement.
|
||||
|
||||
## Next release milestone
|
||||
|
||||
**M1a is met. M1b is next: repair only the vetoes M1a showed to matter, then rerun the
|
||||
three revisions through the production rules.** The other four milestones remain unverified.
|
||||
**M1a is met. M1b is in progress: three of its repairs are verified, and the remaining two
|
||||
are blocked on one new piece of query evidence.** The other four milestones remain unverified.
|
||||
|
||||
M1b tasks 10.1 to 10.3 are done and measured. The recorded view of the NodeGoat case now
|
||||
reaches `new` on the vulnerable revision and `unchanged` on the unchanged revision. The
|
||||
production path is still blocked by the last two vetoes, and the rerun showed neither can be
|
||||
scoped or relaxed soundly: a taint answer reports flows, so an absent row means either that
|
||||
the operation does not exist or that no traced flow reached it. Establishing absence needs the
|
||||
enumerated operation inventory exported separately from traced flows, which is now task 10.4
|
||||
ahead of the two vetoes it unblocks. See [M1a evidence](m1a-recorded-vetoes.md).
|
||||
|
||||
M1a ran on 2026-09-16 within its one-week box. The raw `<lambda>2` witnesses are present at
|
||||
contributions.js lines 32–34, the fixed twin was genuinely analyzed and quiet, and the
|
||||
|
||||
@@ -292,31 +292,39 @@ Preserve all completed task evidence below.
|
||||
- _Depends: 10.2_
|
||||
- _Requirements: 2.4, 3.4_
|
||||
- _Verified 2026-09-16:_ 1,347 tests passed, nine skipped; Ruff/format/mypy passed. Ambiguity is no longer inherited as a context gap by every question, and no longer blocks the transaction comparison; it is decided per operation by the correspondence classifier. Controls prove a uniquely mapped or newly introduced operation stays comparable in a file with repeated unchanged lines, an operation needing an ambiguous anchor stays unresolved, and any other transaction gap still blocks. The mapping is still withheld and still reported in aggregate coverage.
|
||||
- [ ] 10.4 Make the external and dynamic context boundary question-owned
|
||||
- M1a recorded `dynamic_external_or_depth_context_unresolved` on the target's own question. The flag is computed over every in-scope method, so one unresolved call anywhere vetoes every question, including those whose relevant methods are fully resolved.
|
||||
- Report the boundary with the methods whose destinations are unresolved, and inherit it into a question only when such a method participates in that question's change relationships. Unknown ownership remains a blocker; absent evidence of independence is not proof of independence.
|
||||
- Done when a question whose relevant methods resolve completely is not vetoed by an unrelated external call, while a question that depends on an unresolved destination stays unresolved.
|
||||
- [ ] 10.4 Export the enumerated operation inventory so absence is provable
|
||||
- **Prerequisite discovered 2026-09-16, not in the original plan.** The 10.1-10.3 rerun showed the remaining two vetoes cannot be scoped or relaxed soundly as written. A taint answer reports flows, so an absent row means either that the operation does not exist in the base or that no traced flow reached it. `dynamic_external_or_depth_context_unresolved` exists precisely to stop the second being read as the first, and on a real Express application it is always true, because ordinary framework and dependency calls have unresolved destinations. It therefore carries no discrimination and can never be discharged, while removing it would make an unreached operation look absent.
|
||||
- Export the operations the query actually enumerated in the question's scope, separately from the flows it traced, so absence of an operation is established by the inventory rather than by the absence of a flow. Sanitizer and source facts are unchanged; this adds no detector, alias resolver or second IR.
|
||||
- Done when a base answer distinguishes "this operation does not exist here" from "no traced flow reached it", with an unenumerated scope remaining explicitly unresolved.
|
||||
- _Boundary: contributor-scan query/evidence contract, Delta and Actionability Policy_
|
||||
- _Depends: 10.1_
|
||||
- _Requirements: 2.1, 2.2, 3.4, 8.1, 8.2_
|
||||
- [ ] 10.5 Make the external and dynamic context boundary question-owned
|
||||
- M1a recorded `dynamic_external_or_depth_context_unresolved` on the target's own question. The flag is computed over every in-scope method, so one unresolved call anywhere vetoes every question.
|
||||
- With 10.4 supplying inventory evidence, bind the boundary to what it actually limits: reachability, not enumeration. A question establishing absence from the inventory no longer needs complete reachability; a question asserting a traced flow still does. Unknown ownership remains a blocker; absent evidence of independence is not proof of independence.
|
||||
- Done when an absence established from the inventory survives an unrelated unresolved destination, while a reachability claim under a bounded scope stays unresolved.
|
||||
- _Boundary: contributor-scan query/context evidence, Ranker Scope Contract_
|
||||
- _Depends: 10.4_
|
||||
- _Requirements: 2.1, 2.2, 8.1, 8.2_
|
||||
- [ ] 10.5 Stop vendor exclusion acting as a global completion veto
|
||||
- `vendor_semantics_unresolved` currently degrades every otherwise complete outcome to `graph_incomplete`, including on a complete 44-file first-party census with successful query receipts.
|
||||
- Keep the exclusion physical and keep the coverage note. Let the question-owned dependency evidence from 10.3 carry whether a specific answer needed excluded semantics; a recorded dependency on an excluded path still blocks that question.
|
||||
- [ ] 10.6 Stop vendor exclusion acting as a global completion veto
|
||||
- `vendor_semantics_unresolved` degrades every otherwise complete outcome to `graph_incomplete`, including on a complete 44-file first-party census with successful query receipts.
|
||||
- Keep the exclusion physical and keep the coverage note. Let the inventory and reachability evidence from 10.4 and 10.5 carry whether a specific answer needed excluded semantics; a recorded dependency on an excluded path still blocks that question.
|
||||
- Done when an unrelated vendor exclusion no longer invalidates a first-party answer, while a question with evidence of dependency on excluded code remains unresolved and the exclusion stays out of targets and graphs.
|
||||
- _Boundary: contributor-scan graph/scope evidence, Cache and Engine Boundary_
|
||||
- _Depends: 10.4_
|
||||
- _Depends: 10.5_
|
||||
- _Requirements: 2.2, 2.3, 8.2, 8.4_
|
||||
- [ ] 10.6 Carry query-to-operation provenance into the reported evidence
|
||||
- [ ] 10.7 Carry query-to-operation provenance into the reported evidence
|
||||
- The target is selected through a file-fallback question with `function=None` while rows report method `<lambda>2`. M1a showed this did not block the comparison once the question completed, so it is provenance quality rather than the blocker.
|
||||
- Record verified query-to-operation provenance on the evidence; do not inject benchmark function names into regions or count coincident sites.
|
||||
- Done when a reported operation names the question that produced it and the method the engine reported, with unresolved provenance explicit.
|
||||
- _Boundary: Delta and Actionability Policy, contributor-scan normalization_
|
||||
- _Requirements: 3.1, 3.4, 6.3_
|
||||
- [ ] 10.7 Rerun the three revisions through the production rules
|
||||
- [ ] 10.8 Rerun the three revisions through the production rules
|
||||
- Replay the same pinned vulnerable, fixed and unchanged revisions **without** the recorded-veto flag under the named development budget. Demonstrate a developer-readable evaluation finding with a supported consequence, exact location, witness, change attribution and repair direction.
|
||||
- Verify the fixed twin is analyzed and produces no actionable defect and the unchanged revision is not reported as a new regression. Keep qualification eligibility disabled; experimental evaluation output cannot advertise an unqualified hook alert.
|
||||
- Done when one replayable record states which vetoes were repaired and which remain, with completed required checks and readable source and census receipts.
|
||||
- _Boundary: Evaluation harness, full-run integration_
|
||||
- _Depends: 10.1, 10.2, 10.3, 10.4, 10.5, 10.6_
|
||||
- _Depends: 10.1, 10.2, 10.3, 10.4, 10.5, 10.6, 10.7_
|
||||
- _Requirements: 1.1, 2.1, 2.4, 3.1, 3.4, 6.3, 7.1, 7.3_
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
@@ -15,7 +15,7 @@ not a demonstrated useful safety net. Task 8.3 and rollout remain NO-GO.
|
||||
| Order | Milestone | Observable exit | Current state |
|
||||
|---|---|---|---|
|
||||
| M1a | Show the NodeGoat detection with every admission veto recorded, not applied | Raw finding with witness on the vulnerable revision; analyzed fixed twin and unchanged control quiet; per-finding veto list. One-week time box | MET 2026-09-16 |
|
||||
| M1b | Repair only the vetoes M1a showed to matter | Supported evaluation finding through production rules, exact change/base evidence and repair direction; fixed twin and unchanged control stay quiet | NEXT |
|
||||
| M1b | Repair only the vetoes M1a showed to matter | Supported evaluation finding through production rules, exact change/base evidence and repair direction; fixed twin and unchanged control stay quiet | IN PROGRESS: 10.1-10.3 verified; 10.4 operation inventory now blocks the rest |
|
||||
| M2 | Make that same analysis fast | Equivalent evidence; representative warm changed-code p95 <=30s, cancellation/report <=2s, >=95% supported completion | Pending M1 |
|
||||
| M3 | Prove the shared design transfers | PMPro/PHP and VAmPI/Python vulnerable/fixed evidence; unchanged core rules; Node replay after any PMPro core change | Pending M2 |
|
||||
| M4 | Qualify on independent cases | Reviewed untouched vulnerable/fixed/benign populations pass joint quality, coverage and latency gates per capability | Pending M3 |
|
||||
@@ -23,13 +23,14 @@ not a demonstrated useful safety net. Task 8.3 and rollout remain NO-GO.
|
||||
|
||||
### Immediate next action
|
||||
|
||||
Repair the five vetoes M1a recorded, then rerun the same three NodeGoat revisions without
|
||||
the flag through the production rules (M1b). Start with operation correspondence for an
|
||||
operation on a changed line: M1a measured that this one survives every other repair, and it
|
||||
was not in the earlier four-veto diagnosis. The others are per-family change context,
|
||||
question-owned vendor relevance, file-wide correspondence ambiguity scope and
|
||||
query-to-operation provenance. Vetoes that did not fire stay unchanged and their families
|
||||
stay experimental. Normal hook eligibility stays disabled.
|
||||
Export the enumerated operation inventory separately from traced flows (task 10.4), then
|
||||
finish the last two vetoes it unblocks and rerun the three NodeGoat revisions through the
|
||||
production rules. Operation correspondence, required comparison scope and ambiguity scoping
|
||||
are already verified and measured. The inventory is needed because a taint answer reports
|
||||
flows: an absent row means either that the operation does not exist or that no traced flow
|
||||
reached it, and the veto that stops those being confused is always true on a real Express
|
||||
application, so it can neither be discharged nor safely removed. Vetoes that did not fire
|
||||
stay unchanged and their families stay experimental. Normal hook eligibility stays disabled.
|
||||
Do not start another broad speed campaign before this outcome exists. See the
|
||||
[M1a evidence](../specs/pre-push-safety-net/m1a-recorded-vetoes.md) and the
|
||||
[owner diagnosis](../specs/contributor-scan/release-remediation-research.md).
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Binary file not shown.
@@ -696,6 +696,7 @@ def admit_candidates(candidates: Sequence[AdmissionCandidate], *, capabilities:
|
||||
reasons.append("unchanged")
|
||||
elif not delta.change_evidence or (delta.novelty, delta.reason) not in (
|
||||
("new", "source_connection_absent_from_comparable_base"),
|
||||
("new", "operation_absent_from_comparable_base"),
|
||||
("worsened", "discharge_removed"),
|
||||
):
|
||||
reasons.append("change_unsupported")
|
||||
|
||||
@@ -221,3 +221,13 @@ def test_evaluated_operation_matching_preserves_receiver_and_symbol_boundaries(o
|
||||
candidate, delta=replace(candidate.delta, head_operation=replace(candidate.delta.head_operation, operation=operation))
|
||||
)
|
||||
assert not admit_candidates((candidate,), capabilities=(capability,)).defects
|
||||
|
||||
|
||||
def test_an_operation_the_change_introduced_is_a_supported_change_basis():
|
||||
"""M1b 10.1: the new novelty reason must reach admission, not be rejected as unsupported."""
|
||||
candidate, capability, _ = control()
|
||||
introduced = replace(candidate, delta=replace(candidate.delta, reason="operation_absent_from_comparable_base"))
|
||||
assert admit_candidates((introduced,), capabilities=(capability,)).defects
|
||||
invented = replace(candidate, delta=replace(candidate.delta, reason="operation_moved_within_change"))
|
||||
disposition = admit_candidates((invented,), capabilities=(capability,)).dispositions[0]
|
||||
assert not disposition.admitted and "change_unsupported" in disposition.reasons
|
||||
|
||||
Reference in New Issue
Block a user