test(pre-push-safety-net): record the M1b production comparison on NodeGoat

Rerunning the three pinned revisions after tasks 10.1-10.6 shows the production comparison classifying all of them with no lifting, and agreeing exactly with the recorded view. That agreement is the point: the vetoes M1a recorded were repaired rather than bypassed. The vulnerable revision is new with operation_absent_from_comparable_base, and the three eval witnesses at contributions.js lines 32 to 34 are attributed to the change that introduced them. The fixed twin is analyzed and quiet with three base-only findings. The unchanged revision is unchanged. In M1a every one of the 436 questions was demoted on both sides and production reported witness_identity_unresolved on every finding. All three artifacts still report finding_status none with zero admitted defects. Admission rejects on capability_unavailable, the deliberate eligibility gate, and on dependency_unresolved, which the rerun exposed as coarse plumbing: the runner passes every coverage reason as every candidate's dependency gaps. That is now task 10.7, ahead of M4, since it would block admission forever once capabilities qualify. It is not a comparison failure and not an M1b exit blocker. Three comparisons on one inspected teaching repository under a development budget. No capability enabled, no alert emitted, no latency claim. Runtime, independent qualification and rollout all remain NO-GO. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-16 11:49:28 +00:00 · 2026-09-16 18:06:39 +02:00
parent bc68000e93
commit b8b46f1c19
8 changed files with 2327 additions and 31 deletions
@@ -167,3 +167,61 @@ reason, since dependency on excluded code becomes visible in the evidence rather

 That prerequisite is now task 10.4, ahead of the two vetoes it unblocks. It adds no detector,
 alias resolver or second source IR.
+
+## M1b outcome: the same case through the production rules
+
+Rerun 2026-09-16 after tasks 10.1 to 10.6, same instrument, same pins, same budget.
+Evidence: [summary](../../../benchmarks/measurements/2026-09-16-nodegoat-m1b-production.json)
+and its [bundle](../../../benchmarks/measurements/2026-09-16-nodegoat-m1b-production.json.gz).
+
+The production comparison now classifies all three revisions without any lifting, and its
+verdicts match the recorded view exactly. That match is the point: the vetoes M1a recorded
+were repaired rather than bypassed.
+
+| Revision | Production verdict | Recorded verdict |
+|---|---|---|
+| vulnerable | new, `operation_absent_from_comparable_base` | identical |
+| fixed twin | quiet, analyzed, three base-only findings | identical |
+| unchanged | unchanged, `same_supported_mechanism` | identical |
+
+Compared with M1a, where production reported `witness_identity_unresolved` on every finding
+and every one of the 436 questions was demoted on both sides.
+
+The head target questions now complete, the base scan completes all 436 questions, and the
+three `eval` witnesses at contributions.js lines 32 to 34 are attributed to the change that
+introduced them. Timings were 325.7 s, 158.9 s and 260.3 s under the named 900-second
+development budget, which measures nothing about the hook gate.
+
+All three artifacts still report `finding_status: none` with zero admitted defects. Admission
+rejects on `capability_unavailable`, which is the deliberate eligibility gate, and on
+`dependency_unresolved`, which is a coarse plumbing defect recorded as its own task: the runner
+passes every coverage reason as every candidate's dependency gaps rather than the gaps that
+pertain to that candidate. Neither is a comparison failure.
+
+### What each repair contributed, measured
+
+1. **10.1 operation correspondence.** Moved the vulnerable revision from
+   `operation_correspondence_unresolved` to a classifiable operation.
+2. **10.2 required comparison scope.** Stopped unrelated unsupported families from blocking a
+   complete target comparison.
+3. **10.3 ambiguity scoping.** Stopped one file's repeated unchanged lines invalidating every
+   question in the repository.
+4. **10.4 operation inventory.** Made absence provable from enumeration rather than from the
+   absence of a traced flow, which is what let 10.5 and 10.6 be done soundly.
+5. **10.5 reachability binding.** Let the head questions complete, since bounded reachability
+   cannot invalidate a flow that was traced.
+6. **10.6 declared exclusion.** Let the base scan's 436 answers stand, since excluding
+   dependencies is a scope choice rather than a failed read.
+
+A seventh change was needed after the first rerun: the same exclusion and ambiguity distinction
+was still missing inside the comparison's own degradation and boundary checks, so the verdict
+stayed at `head_context_incomplete` even once both scans completed. The rerun is what surfaced
+it, not the unit suite.
+
+### Limits
+
+Three comparisons on one inspected teaching repository, under a development budget. NodeGoat is
+development and transfer data: it cannot qualify production Node support, and Node evidence
+cannot qualify PHP or Python. No capability is enabled, no defect is admitted, no alert is
+emitted, and no latency claim is made. Runtime, independent qualification and rollout all
+remain NO-GO.
@@ -1,7 +1,7 @@
 {
  "feature_name": "pre-push-safety-net",
  "created_at": "2026-09-12T15:35:00Z",
-  "updated_at": "2026-09-16T14:05:00+00:00",
+  "updated_at": "2026-09-16T16:00:00+00:00",
  "language": "en",
  "phase": "implementation",
  "approvals": {
@@ -128,7 +128,10 @@
      "9.2",
      "10.1",
      "10.2",
-      "10.3"
+      "10.3",
+      "10.4",
+      "10.5",
+      "10.6"
    ],
    "remaining_tasks": 1,
    "verification_record": "implementation-group-8.md",
@@ -174,11 +177,18 @@
      "verified": [
        "10.1",
        "10.2",
-        "10.3"
+        "10.3",
+        "10.4",
+        "10.5",
+        "10.6"
      ],
-      "blocked_on": "10.4",
-      "measurement": "../../../benchmarks/measurements/2026-09-16-nodegoat-m1b-step1.json",
-      "finding": "The recorded view now reaches new on the vulnerable revision and unchanged on the unchanged revision. The last two vetoes cannot be scoped or relaxed soundly: a taint answer reports traced flows, so an absent row means either the operation does not exist or no flow reached it. The veto preventing that confusion is computed over every in-scope method and is always true on a real Express application, so it can neither be discharged nor safely removed. Task 10.4 exports the enumerated operation inventory separately from traced flows so absence is provable, and now precedes the two vetoes it unblocks."
+      "remaining": [
+        "10.7",
+        "10.8",
+        "10.9"
+      ],
+      "measurement": "../../../benchmarks/measurements/2026-09-16-nodegoat-m1b-production.json",
+      "result": "Every veto M1a recorded is repaired. The production comparison classifies all three pinned NodeGoat revisions with no lifting: vulnerable new with the three eval witnesses attributed to the change, fixed twin analyzed and quiet with three base-only findings, unchanged recorded unchanged. Production and the recorded view agree exactly. Zero admitted defects; admission rejects on capability_unavailable, the deliberate eligibility gate, and dependency_unresolved, recorded as task 10.7."
    }
  },
  "release": {
@@ -207,6 +217,6 @@
    "implementation_design_approval": "Existing approvals preserved; this records milestone order and exit criteria, not blanket approval of future design changes.",
    "m1_split_accepted": "2026-09-14: maintainer asked to split M1 into M1a/M1b after review",
    "m1a_outcome": "Met inside the one-week box. Corrected the owner diagnosis: all four named vetoes fire, and operation correspondence for an operation on a changed line is the blocker that survives every other repair.",
-    "m1b_position": "10.1-10.3 verified and measured; 10.4 operation inventory is the new prerequisite for 10.5 and 10.6."
+    "m1b_position": "Comparison verified through production rules on NodeGoat; 10.7 dependency-gap binding, 10.8 provenance and 10.9 the recorded rerun remain before M1b is verified."
  }
 }
@@ -7,16 +7,21 @@ it does not approve normal hook capabilities or production enforcement.

 ## Next release milestone

-**M1a is met. M1b is in progress: three of its repairs are verified, and the remaining two
-are blocked on one new piece of query evidence.** The other four milestones remain unverified.
+**M1a is met and the M1b comparison now works through the production rules.** The other four
+milestones remain unverified, and rollout remains NO-GO.

-M1b tasks 10.1 to 10.3 are done and measured. The recorded view of the NodeGoat case now
-reaches `new` on the vulnerable revision and `unchanged` on the unchanged revision. The
-production path is still blocked by the last two vetoes, and the rerun showed neither can be
-scoped or relaxed soundly: a taint answer reports flows, so an absent row means either that
-the operation does not exist or that no traced flow reached it. Establishing absence needs the
-enumerated operation inventory exported separately from traced flows, which is now task 10.4
-ahead of the two vetoes it unblocks. See [M1a evidence](m1a-recorded-vetoes.md).
+Every veto M1a recorded has been repaired. On 2026-09-16 the production comparison classified
+the pinned NodeGoat revisions without any lifting: the vulnerable revision is `new` with the
+three `eval` witnesses attributed to the change that introduced them, the fixed twin is
+analyzed and quiet, and the unchanged revision is `unchanged`. Production and the recorded view
+agree exactly, which is how we know the vetoes were repaired rather than bypassed. No capability
+is enabled, no defect is admitted and no alert is emitted. See
+[M1a and M1b evidence](m1a-recorded-vetoes.md).
+
+Two tasks remain before M1b is verified: binding candidate dependency gaps to the candidate,
+which the rerun exposed as coarse plumbing that would block admission forever after
+qualification, and carrying query-to-operation provenance. Then the run is repeated and
+recorded as the milestone's exit evidence.

 M1a ran on 2026-09-16 within its one-week box. The raw `<lambda>2` witnesses are present at
 contributions.js lines 32–34, the fixed twin was genuinely analyzed and quiet, and the
@@ -316,18 +316,26 @@ Preserve all completed task evidence below.
  - _Depends: 10.5_
  - _Requirements: 2.2, 2.3, 8.2, 8.4_
  - _Verified 2026-09-16:_ 1,365 tests passed, nine skipped; Ruff/format/mypy passed. Graph boundaries are split into integrity failures, which still demote a completed answer, and declared exclusions, which do not. Whether a particular answer needed the excluded semantics is carried by that question's own unresolved call destinations from 10.5, not assumed for every answer. Controls exercise the real partition logic rather than an injected reason: a vendored directory leaves the answer completed with the exclusion still reported, an unresolved symlink still demotes, and the two together still demote. The exclusion stays physical and out of targets and graphs. The NodeGoat rerun showed the same distinction was still missing inside comparison, where any degradation and any boundary blocked: comparison now filters declared exclusions and correspondence ambiguity on both sides, keeping integrity degradations and every other boundary blocking.
- [ ] 10.7 Carry query-to-operation provenance into the reported evidence
+- [ ] 10.7 Bind candidate dependency gaps to the candidate
+  - **Discovered 2026-09-16 in the M1b production rerun.** The runner passes every coverage reason as every candidate's `dependency_gaps`, so admission always answers `dependency_unresolved`. That is not "an unresolved dependency that undermines the claim"; it is every gap anywhere in the transaction.
+  - Supply the gaps that pertain to this candidate's required comparison scope, the same scope 10.2 established. Unknown ownership still counts against the candidate.
+  - Not an M1b exit blocker, since eligibility stays disabled and no alert is emitted either way. It would prevent admission from ever passing after qualification, so it belongs before M4.
+  - Done when an unrelated transaction gap no longer appears as this candidate's unresolved dependency, while a gap in its required scope still does.
+  - _Boundary: Push runner, Delta and Actionability Policy_
+  - _Depends: 10.2_
+  - _Requirements: 3.1, 3.3, 3.4_
+- [ ] 10.8 Carry query-to-operation provenance into the reported evidence
  - The target is selected through a file-fallback question with `function=None` while rows report method `<lambda>2`. M1a showed this did not block the comparison once the question completed, so it is provenance quality rather than the blocker.
  - Record verified query-to-operation provenance on the evidence; do not inject benchmark function names into regions or count coincident sites.
  - Done when a reported operation names the question that produced it and the method the engine reported, with unresolved provenance explicit.
  - _Boundary: Delta and Actionability Policy, contributor-scan normalization_
  - _Requirements: 3.1, 3.4, 6.3_
- [ ] 10.8 Rerun the three revisions through the production rules
+- [ ] 10.9 Rerun the three revisions through the production rules
  - Replay the same pinned vulnerable, fixed and unchanged revisions **without** the recorded-veto flag under the named development budget. Demonstrate a developer-readable evaluation finding with a supported consequence, exact location, witness, change attribution and repair direction.
  - Verify the fixed twin is analyzed and produces no actionable defect and the unchanged revision is not reported as a new regression. Keep qualification eligibility disabled; experimental evaluation output cannot advertise an unqualified hook alert.
  - Done when one replayable record states which vetoes were repaired and which remain, with completed required checks and readable source and census receipts.
  - _Boundary: Evaluation harness, full-run integration_
-  - _Depends: 10.1, 10.2, 10.3, 10.4, 10.5, 10.6, 10.7_
+  - _Depends: 10.1, 10.2, 10.3, 10.4, 10.5, 10.6, 10.7, 10.8_
  - _Requirements: 1.1, 2.1, 2.4, 3.1, 3.4, 6.3, 7.1, 7.3_

 ## Implementation Notes
@@ -15,7 +15,7 @@ not a demonstrated useful safety net. Task 8.3 and rollout remain NO-GO.
 | Order | Milestone | Observable exit | Current state |
 |---|---|---|---|
 | M1a | Show the NodeGoat detection with every admission veto recorded, not applied | Raw finding with witness on the vulnerable revision; analyzed fixed twin and unchanged control quiet; per-finding veto list. One-week time box | MET 2026-09-16 |
-| M1b | Repair only the vetoes M1a showed to matter | Supported evaluation finding through production rules, exact change/base evidence and repair direction; fixed twin and unchanged control stay quiet | IN PROGRESS: 10.1-10.3 verified; 10.4 operation inventory now blocks the rest |
+| M1b | Repair only the vetoes M1a showed to matter | Supported evaluation finding through production rules, exact change/base evidence and repair direction; fixed twin and unchanged control stay quiet | IN PROGRESS: every recorded veto repaired and the production comparison classifies all three revisions; 10.7-10.9 remain |
 | M2 | Make that same analysis fast | Equivalent evidence; representative warm changed-code p95 <=30s, cancellation/report <=2s, >=95% supported completion | Pending M1 |
 | M3 | Prove the shared design transfers | PMPro/PHP and VAmPI/Python vulnerable/fixed evidence; unchanged core rules; Node replay after any PMPro core change | Pending M2 |
 | M4 | Qualify on independent cases | Reviewed untouched vulnerable/fixed/benign populations pass joint quality, coverage and latency gates per capability | Pending M3 |
@@ -23,14 +23,13 @@ not a demonstrated useful safety net. Task 8.3 and rollout remain NO-GO.

 ### Immediate next action

-Export the enumerated operation inventory separately from traced flows (task 10.4), then
-finish the last two vetoes it unblocks and rerun the three NodeGoat revisions through the
-production rules. Operation correspondence, required comparison scope and ambiguity scoping
-are already verified and measured. The inventory is needed because a taint answer reports
-flows: an absent row means either that the operation does not exist or that no traced flow
-reached it, and the veto that stops those being confused is always true on a real Express
-application, so it can neither be discharged nor safely removed. Vetoes that did not fire
-stay unchanged and their families stay experimental. Normal hook eligibility stays disabled.
+Finish M1b: bind candidate dependency gaps to the candidate (10.7), carry query-to-operation
+provenance (10.8), then rerun and record the three NodeGoat revisions as the milestone's exit
+evidence (10.9). Every veto M1a recorded is repaired and the production comparison already
+classifies all three revisions correctly, with production and the recorded view agreeing. The
+dependency-gap defect is coarse plumbing rather than a comparison failure, but it would block
+admission forever once capabilities qualify, so it belongs before M4. Normal hook eligibility
+stays disabled and no capability is enabled by any of this.
 Do not start another broad speed campaign before this outcome exists. See the
 [M1a evidence](../specs/pre-push-safety-net/m1a-recorded-vetoes.md) and the
 [owner diagnosis](../specs/contributor-scan/release-remediation-research.md).
@@ -7,9 +7,11 @@ Release order accepted 2026-09-14: first prove one useful Node changed-code comp
 with an analyzed quiet fixed twin, then make the same analysis fast, verify PHP/Python
 transfer, qualify independently, and enable only passing advisory capabilities. See
 [the milestone spec](../specs/pre-push-safety-net/release-milestones.md). M1a is met
-(2026-09-16: witness shown, fixed twin analyzed and quiet, five vetoes recorded); we are at
-M1b, repairing only what fired. M1a added a fifth veto the earlier diagnosis missed:
-operation correspondence for an operation on a changed line.
+(2026-09-16: witness shown, fixed twin analyzed and quiet, five vetoes recorded), and the M1b
+comparison now classifies all three NodeGoat revisions through the production rules with no
+lifting. M1a added a fifth veto the earlier diagnosis missed, operation correspondence for an
+operation on a changed line, and M1b added a prerequisite it did not anticipate: absence must
+be provable from an enumerated operation inventory, not from the absence of a traced flow.
 Task counts and an experimental alpha tag do not establish release readiness. PMPro
 remains a core regression/development case; any PMPro core retune still requires frozen
 Node transfer evidence. This sequence supersedes older runtime-first next-step wording.