mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
test(pre-push-safety-net): record the M1b exit run (task 10.9)
All three pinned NodeGoat revisions replayed through the production rules with exact pins and reopened source receipts. The vulnerable revision reports all three sites new with operation_absent_from_comparable_base, and admission now rejects on capability_unavailable alone: dependency_unresolved is gone, which verifies 10.7 on real data since those gaps belonged to unrelated questions. The fixed twin is analyzed and quiet with three base-only findings. The unchanged revision reports all three sites unchanged with base locations resolved to the same lines. Every finding carries its question, the engine-reported method and its witness. The head census is the complete 44 files and all 436 base questions complete. Zero admitted defects on every revision. All five vetoes M1a recorded are repaired, plus two the repairs exposed. What remains is recorded rather than repaired: 231 head questions still report an incomplete change context from families whose queries export none, which did not block this case and whose families stay experimental, and the vendor exclusion and correspondence ambiguity remain reported as the limits they are. M1b is not yet verified. The milestone also asks for a developer-readable finding carrying a supported security consequence and a concrete repair direction, and both are owned by an evaluated capability declaration that is deliberately empty. Writing that text without a declaration is what admission exists to prevent, so task 10.10 renders it through an explicitly experimental declaration with eligibility still disabled. Runtime, independent qualification and rollout all remain NO-GO. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
co-authored by
Claude Opus 5
parent
541ee5813e
commit
57d6ddcbba
@@ -225,3 +225,67 @@ development and transfer data: it cannot qualify production Node support, and No
|
|||||||
cannot qualify PHP or Python. No capability is enabled, no defect is admitted, no alert is
|
cannot qualify PHP or Python. No capability is enabled, no defect is admitted, no alert is
|
||||||
emitted, and no latency claim is made. Runtime, independent qualification and rollout all
|
emitted, and no latency claim is made. Runtime, independent qualification and rollout all
|
||||||
remain NO-GO.
|
remain NO-GO.
|
||||||
|
|
||||||
|
## M1b exit run (tasks 10.1 to 10.8)
|
||||||
|
|
||||||
|
Rerun 2026-09-16 after every repair, same instrument, same pins, same 900-second development
|
||||||
|
budget. Evidence: [summary](../../../benchmarks/measurements/2026-09-16-nodegoat-m1b-exit.json)
|
||||||
|
and its [bundle](../../../benchmarks/measurements/2026-09-16-nodegoat-m1b-exit.json.gz).
|
||||||
|
|
||||||
|
| Revision | Verdict | Admission | Elapsed |
|
||||||
|
|---|---|---|---:|
|
||||||
|
| vulnerable | all three sites new, `operation_absent_from_comparable_base` | `capability_unavailable` only | 326.4 s |
|
||||||
|
| fixed twin | analyzed and quiet, three base-only findings | none to admit | 157.8 s |
|
||||||
|
| unchanged | all three sites `unchanged`, `same_supported_mechanism` | `capability_unavailable`, `unchanged` | 256.8 s |
|
||||||
|
|
||||||
|
The vulnerable revision's only remaining admission reason is the deliberate eligibility gate.
|
||||||
|
`dependency_unresolved` is gone, which verifies task 10.7 on real data: those gaps belonged to
|
||||||
|
unrelated questions.
|
||||||
|
|
||||||
|
Each finding now carries its provenance explicitly: question function `None`, engine method
|
||||||
|
`<lambda>2`, witness `eval(req.body.preTax) -> const preTax = eval(req.body.preTax)` at
|
||||||
|
contributions.js line 32, and likewise for lines 33 and 34. The unchanged revision resolves each
|
||||||
|
base location to the same line, which is why it reads `unchanged` rather than new.
|
||||||
|
|
||||||
|
Source receipts were reopened before the run, and the head census is the complete 44 files.
|
||||||
|
|
||||||
|
### Which vetoes were repaired and which remain
|
||||||
|
|
||||||
|
All five vetoes M1a recorded are repaired, plus two the repairs exposed:
|
||||||
|
|
||||||
|
| Veto | State |
|
||||||
|
|---|---|
|
||||||
|
| `change_context_incomplete` on the target question | repaired, question completes |
|
||||||
|
| `graph_incomplete` on the base from vendor exclusion | repaired, all 436 base questions complete |
|
||||||
|
| `ambiguous_line_correspondence` inherited repository-wide | repaired, decided per operation |
|
||||||
|
| `dynamic_external_or_depth_context_unresolved` as a blanket veto | repaired, bound to reachability claims |
|
||||||
|
| `operation_correspondence_unresolved` on a changed line | repaired via the enumerated inventory |
|
||||||
|
| comparison's own degradation and boundary checks | repaired after the first rerun exposed them |
|
||||||
|
| `dependency_unresolved` on every candidate | repaired, gaps attributed per candidate |
|
||||||
|
|
||||||
|
What remains, recorded rather than repaired, because each is outside this case or a later
|
||||||
|
milestone:
|
||||||
|
|
||||||
|
- 231 of 436 head questions still report `change_context_incomplete`, from
|
||||||
|
`context_projection_unavailable` and `context_scope_empty` on families whose queries export no
|
||||||
|
context. Those are dominance and configuration questions. They did not block this case and
|
||||||
|
their families stay experimental, so per the milestone they stay as they are.
|
||||||
|
- The vendor exclusion and the file's correspondence ambiguity remain reported in coverage. That
|
||||||
|
is intended: they are recorded limits, not repaired absences.
|
||||||
|
|
||||||
|
### What M1b has not yet demonstrated
|
||||||
|
|
||||||
|
The milestone also asks for a developer-readable evaluation finding carrying a supported
|
||||||
|
security consequence and a concrete repair direction. Those two come from an evaluated
|
||||||
|
capability declaration, and the registry is deliberately empty, so nothing renders them today.
|
||||||
|
Writing that text without a declaration is precisely what admission exists to prevent, so it is
|
||||||
|
not something to work around here. The comparison half of M1b is verified; the explanation half
|
||||||
|
waits on a declaration, which is M4's subject.
|
||||||
|
|
||||||
|
### Limits
|
||||||
|
|
||||||
|
Three comparisons on one inspected teaching repository under a development budget. NodeGoat is
|
||||||
|
development and transfer data: it cannot qualify production Node support, and Node evidence
|
||||||
|
cannot qualify PHP or Python. No capability is enabled, no defect is admitted, no alert is
|
||||||
|
emitted and no latency claim is made. Runtime, independent qualification and rollout all remain
|
||||||
|
NO-GO.
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
{
|
{
|
||||||
"feature_name": "pre-push-safety-net",
|
"feature_name": "pre-push-safety-net",
|
||||||
"created_at": "2026-09-12T15:35:00Z",
|
"created_at": "2026-09-12T15:35:00Z",
|
||||||
"updated_at": "2026-09-16T16:00:00+00:00",
|
"updated_at": "2026-09-16T17:15:00+00:00",
|
||||||
"language": "en",
|
"language": "en",
|
||||||
"phase": "implementation",
|
"phase": "implementation",
|
||||||
"approvals": {
|
"approvals": {
|
||||||
@@ -131,7 +131,10 @@
|
|||||||
"10.3",
|
"10.3",
|
||||||
"10.4",
|
"10.4",
|
||||||
"10.5",
|
"10.5",
|
||||||
"10.6"
|
"10.6",
|
||||||
|
"10.7",
|
||||||
|
"10.8",
|
||||||
|
"10.9"
|
||||||
],
|
],
|
||||||
"remaining_tasks": 1,
|
"remaining_tasks": 1,
|
||||||
"verification_record": "implementation-group-8.md",
|
"verification_record": "implementation-group-8.md",
|
||||||
@@ -180,15 +183,17 @@
|
|||||||
"10.3",
|
"10.3",
|
||||||
"10.4",
|
"10.4",
|
||||||
"10.5",
|
"10.5",
|
||||||
"10.6"
|
"10.6",
|
||||||
],
|
|
||||||
"remaining": [
|
|
||||||
"10.7",
|
"10.7",
|
||||||
"10.8",
|
"10.8",
|
||||||
"10.9"
|
"10.9"
|
||||||
],
|
],
|
||||||
"measurement": "../../../benchmarks/measurements/2026-09-16-nodegoat-m1b-production.json",
|
"remaining": [
|
||||||
"result": "Every veto M1a recorded is repaired. The production comparison classifies all three pinned NodeGoat revisions with no lifting: vulnerable new with the three eval witnesses attributed to the change, fixed twin analyzed and quiet with three base-only findings, unchanged recorded unchanged. Production and the recorded view agree exactly. Zero admitted defects; admission rejects on capability_unavailable, the deliberate eligibility gate, and dependency_unresolved, recorded as task 10.7."
|
"10.10"
|
||||||
|
],
|
||||||
|
"measurement": "../../../benchmarks/measurements/2026-09-16-nodegoat-m1b-exit.json",
|
||||||
|
"result": "Exit run recorded. All three pinned NodeGoat revisions classify through the production rules: vulnerable three sites new with operation_absent_from_comparable_base and admission rejecting on capability_unavailable alone; fixed twin analyzed and quiet with three base-only findings; unchanged recorded unchanged with base locations resolved. Every veto M1a listed is repaired, plus two the repairs exposed. Zero admitted defects.",
|
||||||
|
"remaining_gap": "The supported security consequence and concrete repair direction are not demonstrated. Both are owned by an evaluated capability declaration and the registry is deliberately empty. Task 10.10 renders them through an explicitly experimental declaration with eligibility still disabled; inventing the text is what admission exists to prevent."
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"release": {
|
"release": {
|
||||||
@@ -217,6 +222,6 @@
|
|||||||
"implementation_design_approval": "Existing approvals preserved; this records milestone order and exit criteria, not blanket approval of future design changes.",
|
"implementation_design_approval": "Existing approvals preserved; this records milestone order and exit criteria, not blanket approval of future design changes.",
|
||||||
"m1_split_accepted": "2026-09-14: maintainer asked to split M1 into M1a/M1b after review",
|
"m1_split_accepted": "2026-09-14: maintainer asked to split M1 into M1a/M1b after review",
|
||||||
"m1a_outcome": "Met inside the one-week box. Corrected the owner diagnosis: all four named vetoes fire, and operation correspondence for an operation on a changed line is the blocker that survives every other repair.",
|
"m1a_outcome": "Met inside the one-week box. Corrected the owner diagnosis: all four named vetoes fire, and operation correspondence for an operation on a changed line is the blocker that survives every other repair.",
|
||||||
"m1b_position": "Comparison verified through production rules on NodeGoat; 10.7 dependency-gap binding, 10.8 provenance and 10.9 the recorded rerun remain before M1b is verified."
|
"m1b_position": "Comparison half verified and recorded on NodeGoat; the consequence and repair rendering (10.10) remains before M1b is verified."
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -18,10 +18,17 @@ agree exactly, which is how we know the vetoes were repaired rather than bypasse
|
|||||||
is enabled, no defect is admitted and no alert is emitted. See
|
is enabled, no defect is admitted and no alert is emitted. See
|
||||||
[M1a and M1b evidence](m1a-recorded-vetoes.md).
|
[M1a and M1b evidence](m1a-recorded-vetoes.md).
|
||||||
|
|
||||||
Two tasks remain before M1b is verified: binding candidate dependency gaps to the candidate,
|
The exit run is recorded. Every veto M1a listed is repaired, plus two the repairs exposed: the
|
||||||
which the rerun exposed as coarse plumbing that would block admission forever after
|
comparison's own degradation and boundary checks, and the runner handing every candidate the
|
||||||
qualification, and carrying query-to-operation provenance. Then the run is repeated and
|
whole transaction's gaps. On the vulnerable revision admission now rejects on
|
||||||
recorded as the milestone's exit evidence.
|
`capability_unavailable` alone.
|
||||||
|
|
||||||
|
One half of M1b remains. The milestone also asks for a developer-readable finding carrying a
|
||||||
|
supported security consequence and a concrete repair direction. Both are owned by an evaluated
|
||||||
|
capability declaration and the registry is deliberately empty, so nothing renders them. Writing
|
||||||
|
that text without a declaration is what admission exists to prevent, so it is not worked around.
|
||||||
|
Task 10.10 renders it through an explicitly experimental declaration with eligibility still
|
||||||
|
disabled.
|
||||||
|
|
||||||
M1a ran on 2026-09-16 within its one-week box. The raw `<lambda>2` witnesses are present at
|
M1a ran on 2026-09-16 within its one-week box. The raw `<lambda>2` witnesses are present at
|
||||||
contributions.js lines 32–34, the fixed twin was genuinely analyzed and quiet, and the
|
contributions.js lines 32–34, the fixed twin was genuinely analyzed and quiet, and the
|
||||||
|
|||||||
@@ -332,13 +332,23 @@ Preserve all completed task evidence below.
|
|||||||
- _Boundary: Delta and Actionability Policy, contributor-scan normalization_
|
- _Boundary: Delta and Actionability Policy, contributor-scan normalization_
|
||||||
- _Requirements: 3.1, 3.4, 6.3_
|
- _Requirements: 3.1, 3.4, 6.3_
|
||||||
- _Verified 2026-09-16:_ 1,371 tests passed, nine skipped; Ruff/format/mypy passed. Evidence now carries the method the engine reported beside the question that produced it, so the link is inspectable rather than inferred or dug out of a detail blob. Provenance is recorded as `exact`, `file_scope` when the question named no function and the engine supplied one, `unresolved` when the two disagree, and `method_unreported` when the engine named none. Controls cover all four. No region is renamed, no benchmark function name is injected and no coincident site is counted; the state is recorded, not applied.
|
- _Verified 2026-09-16:_ 1,371 tests passed, nine skipped; Ruff/format/mypy passed. Evidence now carries the method the engine reported beside the question that produced it, so the link is inspectable rather than inferred or dug out of a detail blob. Provenance is recorded as `exact`, `file_scope` when the question named no function and the engine supplied one, `unresolved` when the two disagree, and `method_unreported` when the engine named none. Controls cover all four. No region is renamed, no benchmark function name is injected and no coincident site is counted; the state is recorded, not applied.
|
||||||
- [ ] 10.9 Rerun the three revisions through the production rules
|
- [x] 10.9 Rerun the three revisions through the production rules
|
||||||
- Replay the same pinned vulnerable, fixed and unchanged revisions **without** the recorded-veto flag under the named development budget. Demonstrate a developer-readable evaluation finding with a supported consequence, exact location, witness, change attribution and repair direction.
|
- Replay the same pinned vulnerable, fixed and unchanged revisions **without** the recorded-veto flag under the named development budget. Demonstrate a developer-readable evaluation finding with a supported consequence, exact location, witness, change attribution and repair direction.
|
||||||
- Verify the fixed twin is analyzed and produces no actionable defect and the unchanged revision is not reported as a new regression. Keep qualification eligibility disabled; experimental evaluation output cannot advertise an unqualified hook alert.
|
- Verify the fixed twin is analyzed and produces no actionable defect and the unchanged revision is not reported as a new regression. Keep qualification eligibility disabled; experimental evaluation output cannot advertise an unqualified hook alert.
|
||||||
- Done when one replayable record states which vetoes were repaired and which remain, with completed required checks and readable source and census receipts.
|
- Done when one replayable record states which vetoes were repaired and which remain, with completed required checks and readable source and census receipts.
|
||||||
- _Boundary: Evaluation harness, full-run integration_
|
- _Boundary: Evaluation harness, full-run integration_
|
||||||
- _Depends: 10.1, 10.2, 10.3, 10.4, 10.5, 10.6, 10.7, 10.8_
|
- _Depends: 10.1, 10.2, 10.3, 10.4, 10.5, 10.6, 10.7, 10.8_
|
||||||
- _Requirements: 1.1, 2.1, 2.4, 3.1, 3.4, 6.3, 7.1, 7.3_
|
- _Requirements: 1.1, 2.1, 2.4, 3.1, 3.4, 6.3, 7.1, 7.3_
|
||||||
|
- _Verified 2026-09-16 with one named exception:_ all three revisions replayed through the production rules with exact pins and reopened source receipts. Vulnerable: three sites new with `operation_absent_from_comparable_base`, admission rejecting on `capability_unavailable` alone. Fixed twin: analyzed and quiet with three base-only findings. Unchanged: three sites `unchanged` with base locations resolved to the same lines. Every finding carries its question, the engine-reported method and its witness; the head census is the complete 44 files; all 436 base questions complete. Zero admitted defects. The repaired and remaining veto lists are in [the M1b record](m1a-recorded-vetoes.md). **Exception:** the supported security consequence and concrete repair direction are not demonstrated, because both are owned by an evaluated capability declaration and the registry is deliberately empty. M1b's comparison half is verified; its explanation half waits on M4.
|
||||||
|
|
||||||
|
- [ ] 10.10 Render the experimental evaluation finding with its consequence and repair
|
||||||
|
- **The remaining half of M1b's exit, named by the 10.9 run.** The comparison is verified, but the milestone also asks for a developer-readable finding carrying a supported security consequence and a concrete repair direction. Both come from an evaluated capability declaration and the registry is deliberately empty, so nothing renders them.
|
||||||
|
- Generate an explicitly unreviewed, experimental declaration for this capability with the existing registry tool, and render the finding through the same grounded-template path the admitted case uses. Keep eligibility disabled: an experimental declaration is not enabled and not PASS, so admission must still reject it and no normal hook alert may be advertised.
|
||||||
|
- Do not write consequence or repair text that no declaration supports. Refusing generic advice without a supported context and operation is the rule admission exists to enforce, and the experimental path must not become a way around it.
|
||||||
|
- Done when the experimental output shows family, exact location, witness, change attribution, consequence and repair direction for the vulnerable revision, is labeled experimental throughout, and the production result still reports no actionable defect with zero admitted defects.
|
||||||
|
- _Boundary: Result and Hook Adapter, Delta and Actionability Policy, Evaluation harness_
|
||||||
|
- _Depends: 10.9_
|
||||||
|
- _Requirements: 3.1, 3.2, 3.3, 5.1, 5.5, 6.3_
|
||||||
|
|
||||||
## Implementation Notes
|
## Implementation Notes
|
||||||
|
|
||||||
|
|||||||
@@ -15,7 +15,7 @@ not a demonstrated useful safety net. Task 8.3 and rollout remain NO-GO.
|
|||||||
| Order | Milestone | Observable exit | Current state |
|
| Order | Milestone | Observable exit | Current state |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| M1a | Show the NodeGoat detection with every admission veto recorded, not applied | Raw finding with witness on the vulnerable revision; analyzed fixed twin and unchanged control quiet; per-finding veto list. One-week time box | MET 2026-09-16 |
|
| M1a | Show the NodeGoat detection with every admission veto recorded, not applied | Raw finding with witness on the vulnerable revision; analyzed fixed twin and unchanged control quiet; per-finding veto list. One-week time box | MET 2026-09-16 |
|
||||||
| M1b | Repair only the vetoes M1a showed to matter | Supported evaluation finding through production rules, exact change/base evidence and repair direction; fixed twin and unchanged control stay quiet | IN PROGRESS: every recorded veto repaired and the production comparison classifies all three revisions; 10.7-10.9 remain |
|
| M1b | Repair only the vetoes M1a showed to matter | Supported evaluation finding through production rules, exact change/base evidence and repair direction; fixed twin and unchanged control stay quiet | IN PROGRESS: comparison half verified on all three revisions; the consequence and repair rendering (10.10) remains |
|
||||||
| M2 | Make that same analysis fast | Equivalent evidence; representative warm changed-code p95 <=30s, cancellation/report <=2s, >=95% supported completion | Pending M1 |
|
| M2 | Make that same analysis fast | Equivalent evidence; representative warm changed-code p95 <=30s, cancellation/report <=2s, >=95% supported completion | Pending M1 |
|
||||||
| M3 | Prove the shared design transfers | PMPro/PHP and VAmPI/Python vulnerable/fixed evidence; unchanged core rules; Node replay after any PMPro core change | Pending M2 |
|
| M3 | Prove the shared design transfers | PMPro/PHP and VAmPI/Python vulnerable/fixed evidence; unchanged core rules; Node replay after any PMPro core change | Pending M2 |
|
||||||
| M4 | Qualify on independent cases | Reviewed untouched vulnerable/fixed/benign populations pass joint quality, coverage and latency gates per capability | Pending M3 |
|
| M4 | Qualify on independent cases | Reviewed untouched vulnerable/fixed/benign populations pass joint quality, coverage and latency gates per capability | Pending M3 |
|
||||||
@@ -23,13 +23,12 @@ not a demonstrated useful safety net. Task 8.3 and rollout remain NO-GO.
|
|||||||
|
|
||||||
### Immediate next action
|
### Immediate next action
|
||||||
|
|
||||||
Finish M1b: bind candidate dependency gaps to the candidate (10.7), carry query-to-operation
|
Finish M1b by rendering the experimental evaluation finding with its consequence and repair
|
||||||
provenance (10.8), then rerun and record the three NodeGoat revisions as the milestone's exit
|
direction (10.10), through an explicitly unreviewed declaration and with eligibility still
|
||||||
evidence (10.9). Every veto M1a recorded is repaired and the production comparison already
|
disabled. The comparison half is verified: the production rules classify the vulnerable, fixed
|
||||||
classifies all three revisions correctly, with production and the recorded view agreeing. The
|
and unchanged NodeGoat revisions correctly, and admission rejects the vulnerable one on the
|
||||||
dependency-gap defect is coarse plumbing rather than a comparison failure, but it would block
|
eligibility gate alone. What is missing is the explanation, which no declaration currently
|
||||||
admission forever once capabilities qualify, so it belongs before M4. Normal hook eligibility
|
supports, and inventing it is exactly what admission exists to prevent. Then M2 runtime.
|
||||||
stays disabled and no capability is enabled by any of this.
|
|
||||||
Do not start another broad speed campaign before this outcome exists. See the
|
Do not start another broad speed campaign before this outcome exists. See the
|
||||||
[M1a evidence](../specs/pre-push-safety-net/m1a-recorded-vetoes.md) and the
|
[M1a evidence](../specs/pre-push-safety-net/m1a-recorded-vetoes.md) and the
|
||||||
[owner diagnosis](../specs/contributor-scan/release-remediation-research.md).
|
[owner diagnosis](../specs/contributor-scan/release-remediation-research.md).
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
Binary file not shown.
Reference in New Issue
Block a user