mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
feat(pre-push-safety-net): record admission vetoes and meet M1a on NodeGoat
Add group 9 to the approved task plan as the bounded owner amendment release milestone M1a requires, implement the task-local recorded-veto evaluation flag, and run the three pinned NodeGoat revisions under it. The flag re-runs the production comparison and admission over a lifted copy of the same evidence and records each veto instead of applying it. Recognized completeness vetoes are lifted and logged; failures never are, because silence from a failure is not a quiet fixed twin. It is off by default, accepted only on explicit base/head replay, refused in Git stdin mode, labels every payload it produces, and admits no defect even when a lifted candidate otherwise would. M1a outcome, inside its one-week box and with expectations frozen on disk first: the three entailed <lambda>2 witnesses are present at contributions.js 32-34; the fixed twin was genuinely analyzed and quiet, answering its target question with zero rows and no deadline reason while the same findings appear as base-only; the unchanged revision was analyzed and recorded unchanged. The kill criterion did not trigger. Five vetoes fired and were recorded. M1a also corrected the diagnosis: with all five lifted the detection still stops at operation_correspondence_unresolved, because a newly introduced operation sits on a changed line and has no base anchor. The unchanged revision resolves its base location on the same lines and classifies as unchanged, which isolates correspondence as the mechanism. That fifth veto was not in the earlier four-veto diagnosis and is now M1b's central repair. Diagnosis only. No defect admitted, no capability enabled, no gate changed, no latency claim; the 900-second development budget cannot measure the hook gate. Rollout and task 8.3 remain NO-GO. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
co-authored by
Claude Opus 5
Claude Opus 5
parent
8b66d5cb75
commit
3224884d8f
@@ -0,0 +1,116 @@
|
||||
# M1a evidence: the NodeGoat detection with every veto recorded
|
||||
|
||||
Run date 2026-09-16. Milestone **M1a** of [release-milestones.md](release-milestones.md).
|
||||
Instrument `benchmarks/push/m1a_record_vetoes.py`; flag `--experimental-record-vetoes`.
|
||||
Evidence: [summary](../../../benchmarks/measurements/2026-09-16-nodegoat-m1a-recorded-vetoes.json)
|
||||
and its 11-file [bundle](../../../benchmarks/measurements/2026-09-16-nodegoat-m1a-recorded-vetoes.json.gz).
|
||||
|
||||
This record diagnoses. It admits no defect, enables no capability, qualifies no language and
|
||||
makes no latency claim. All three artifacts report `finding_status: none` with zero admitted
|
||||
defects; every recorded section carries its experimental label.
|
||||
|
||||
## What was declared before running
|
||||
|
||||
Expected base/head pins, expected operation and expected outcome for all three comparisons
|
||||
were written to `m1a-record.json` before the first scan. The declared target is
|
||||
`app/routes/contributions.js`, `handleContributionsUpdate`, family `injection`, lines 32–34.
|
||||
|
||||
| Case | Base | Head | Declared expectation |
|
||||
|---|---|---|---|
|
||||
| nodegoat-introduce | fixed `0b453d5a` | vulnerable `c5cb68a7` | raw finding at the three eval lines, recorded novelty new |
|
||||
| nodegoat-repair | vulnerable `c5cb68a7` | fixed `0b453d5a` | fixed twin analyzed and quiet; three base-only findings |
|
||||
| nodegoat-benign | vulnerable `c5cb68a7` | benign `f772c59e` | backlog unchanged; never new or worsened |
|
||||
|
||||
Each revision's target file was reopened before the run: 2,485, 2,491 and 2,498 bytes with
|
||||
recorded SHA-256 digests. Each head scan reported a complete 44-file JavaScript census.
|
||||
Installed engine, facts, query, policy, core and semantics provenance matched the frozen
|
||||
values in all three artifacts.
|
||||
|
||||
## Observed outcome
|
||||
|
||||
| Case | Elapsed | Target findings | Target question answered | Deadline reached |
|
||||
|---|---:|---:|---|---|
|
||||
| nodegoat-introduce | 349.8 s | 3 | yes, 9 rows | no |
|
||||
| nodegoat-repair | 153.8 s | 0 | yes, 0 rows | no |
|
||||
| nodegoat-benign | 261.4 s | 3 | yes | no |
|
||||
|
||||
All three ran under an explicitly named 900-second development budget. These durations are
|
||||
development measurements and cannot satisfy or measure the 30-second hook latency gate.
|
||||
|
||||
Verdict recorded by the instrument: raw witness present, fixed twin quiet, unchanged revision
|
||||
not reported as a new regression, recorded evaluation produced for all three. The milestone's
|
||||
**kill criterion did not trigger**: the fixed twin was genuinely analyzed, answering its target
|
||||
question with zero rows and no deadline reason, while the same three findings appear on its
|
||||
base side as base-only findings. Silence from failure was not accepted as a pass.
|
||||
|
||||
## The vetoes that actually fired
|
||||
|
||||
Identical for each of the three `eval` witnesses on the vulnerable revision. Five recognized
|
||||
vetoes fired and were recorded rather than applied:
|
||||
|
||||
| Stage | Veto | Evidence |
|
||||
|---|---|---|
|
||||
| head question | `change_context_incomplete` | the answered target question carried 9 rows and was then demoted |
|
||||
| head scan | `dynamic_external_or_depth_context_unresolved` | on the target's own question identity, not an unrelated one |
|
||||
| head scan | `vendor_semantics_unresolved` | recorded both as a degradation and as a scope boundary |
|
||||
| base scan | `graph_incomplete` | the base question answered with 0 rows and was then demoted |
|
||||
| change context | `ambiguous_line_correspondence` | file-level, on the target file |
|
||||
|
||||
The earlier owner diagnosis named four independent vetoes and expected query-to-operation
|
||||
provenance and question-owned vendor relevance to be the two that mattered. Both fired. The
|
||||
run also confirms the third and fourth, so all four named vetoes are real on this case.
|
||||
|
||||
## What M1a corrected in the diagnosis
|
||||
|
||||
**A fifth veto is the actual remaining blocker.** With all five recognized vetoes lifted, the
|
||||
detection does not become new. Its reason advances from `witness_identity_unresolved` to
|
||||
`operation_correspondence_unresolved`, because the head operation sits on a changed line and
|
||||
therefore has no base line anchor. The comparison can classify an operation only when it maps
|
||||
to a base location.
|
||||
|
||||
The unchanged revision proves this is the mechanism rather than a coincidence. Its comment-only
|
||||
edit leaves lines 32–34 unchanged, the base location resolves to the same lines, and the
|
||||
recorded novelty becomes `unchanged` / `same_supported_mechanism`. Same evidence, same policy,
|
||||
opposite correspondence outcome:
|
||||
|
||||
| Revision | Operation line | Base location | Recorded novelty |
|
||||
|---|---|---|---|
|
||||
| vulnerable | changed | unresolved | unknown, `operation_correspondence_unresolved` |
|
||||
| unchanged | unchanged | resolved to the same line | unchanged, `same_supported_mechanism` |
|
||||
|
||||
**Query identity differs from witness identity, as predicted.** The target is selected through a
|
||||
file-fallback question with `function=None`, while the rows report method `<lambda>2` at lines
|
||||
32–34. The declared benchmark function is `handleContributionsUpdate`. This is recorded, not
|
||||
applied: it is why production reported `witness_identity_unresolved` before any lifting, since
|
||||
an unresolved question yields no comparable operation at all.
|
||||
|
||||
**Admission reasons that no flag can remove.** `capability_unavailable` remains on every
|
||||
candidate because the registry is empty and this flag cannot enable a capability.
|
||||
`dependency_unresolved` remains because unlifted coverage reasons still reach admission.
|
||||
|
||||
## What M1b must repair, ordered by this evidence
|
||||
|
||||
1. **Authentic per-family change context.** `change_context_incomplete` demoted every answered
|
||||
question in all three cases. Dominance and configuration questions need their own context
|
||||
producers describing their actual query scope; taint context must not be copied into them.
|
||||
2. **Question-owned vendor relevance.** `vendor_semantics_unresolved` turned an otherwise
|
||||
complete 44-file first-party census into `graph_incomplete` on every base question. Keep the
|
||||
exclusion physical; establish which question actually requires excluded semantics.
|
||||
3. **Operation correspondence for changed lines.** The blocker M1a surfaced. Bind the required
|
||||
base/head correspondence to the specific operation rather than to a line anchor, with absent,
|
||||
renamed, removed-guard and ambiguous-operation controls. Do not delete the ambiguity check
|
||||
and do not treat lexical alignment as semantic proof.
|
||||
4. **Correspondence ambiguity scoped to its file region.** `ambiguous_line_correspondence` is
|
||||
recorded file-wide and inherited by every question, including uniquely mapped operations.
|
||||
5. **Query-to-operation provenance.** Carry verified provenance from the selected question to the
|
||||
reported operation. Do not inject benchmark function names into regions.
|
||||
|
||||
Vetoes that did not fire on this case stay as they are and their families stay experimental.
|
||||
|
||||
## Limits
|
||||
|
||||
Three comparisons on one inspected teaching repository. NodeGoat is development and transfer
|
||||
data; it cannot qualify production Node support, and Node evidence cannot qualify PHP or Python.
|
||||
The lifted view is a diagnostic reconstruction, not a comparison result: it shows what the
|
||||
existing evidence would support if those vetoes were repaired, and no repair has been made.
|
||||
Runtime, independent qualification and rollout all remain NO-GO.
|
||||
@@ -8,8 +8,8 @@ It does not approve an unreviewed implementation design or enable any capability
|
||||
|
||||
## Current position
|
||||
|
||||
**Next milestone: M1a. None of M1–M5 is verified yet.** The 26/27 completed executable
|
||||
tasks measure implementation progress, not release readiness. Task 8.3 remains blocked.
|
||||
**Next milestone: M1b. M1a is met; none of M1–M5 is verified yet.** The 28/29 completed
|
||||
executable tasks measure implementation progress, not release readiness. Task 8.3 remains blocked.
|
||||
The published `v1.2.0-alpha.1` is an experimental implementation release, not a qualified
|
||||
advisory rollout. See [current status](status.md) for measured evidence and limitations.
|
||||
|
||||
@@ -37,8 +37,14 @@ result is a full production pass, not one detection.
|
||||
|
||||
### M1a — Show the detection with every veto recorded
|
||||
|
||||
Status: **NEXT — not verified**. Time box: one week of maintainer effort from the first
|
||||
M1a run. Missing the box is a finding, not a reason to extend silently.
|
||||
Status: **MET 2026-09-16**, inside its one-week box. Evidence:
|
||||
[M1a record](m1a-recorded-vetoes.md) and
|
||||
[measurement](../../../benchmarks/measurements/2026-09-16-nodegoat-m1a-recorded-vetoes.json).
|
||||
Raw witnesses present at contributions.js lines 32–34; fixed twin analyzed and quiet with
|
||||
zero rows and no deadline reason; unchanged revision analyzed and recorded `unchanged`. The
|
||||
kill criterion did not trigger. Five vetoes fired and were recorded, and M1a corrected the
|
||||
diagnosis: with all five lifted the detection stops at `operation_correspondence_unresolved`,
|
||||
a fifth veto that was not previously named. No defect was admitted and no capability enabled.
|
||||
|
||||
- Use the pinned NodeGoat contribution injection case, its fixed twin and an unchanged
|
||||
vulnerable revision as development data. Record exact base/head revisions, expected
|
||||
@@ -63,7 +69,11 @@ diagnosed and recorded.
|
||||
|
||||
### M1b — Repair only the vetoes M1a showed to matter
|
||||
|
||||
Status: **PENDING M1a**.
|
||||
Status: **NEXT — not verified**. M1a supplied the veto list: per-family change context,
|
||||
question-owned vendor relevance, operation correspondence for an operation on a changed line,
|
||||
file-wide correspondence ambiguity scope, and query-to-operation provenance. The owner
|
||||
diagnosis expected the first and the last two; operation correspondence is the addition M1a
|
||||
measured and is the blocker that survives every other repair.
|
||||
|
||||
- From the M1a veto list, repair only the context/family/identity contracts this case
|
||||
actually needs. The owner diagnosis expects query-to-operation provenance and question-
|
||||
@@ -192,9 +202,10 @@ remaining blocker and next concrete action. Report implementation task counts se
|
||||
Change a milestone to verified only with linked evidence; do not infer progress from
|
||||
elapsed effort or a completed experimental release tag.
|
||||
|
||||
Immediate next action (M1a): add the task-local recorded-veto experimental flag, run the
|
||||
NodeGoat vulnerable, fixed and unchanged revisions under it, and produce the per-finding
|
||||
veto list alongside the `<lambda>2` witness. Then trace that witness into the declared
|
||||
contribution handler and turn the vetoes that actually fired into the smallest owner fix
|
||||
(M1b). Use [remediation research](../contributor-scan/release-remediation-research.md) as
|
||||
the starting record; do not begin another broad timing campaign before the M1 outcome exists.
|
||||
Immediate next action (M1b): repair the five vetoes M1a recorded, smallest first, starting
|
||||
with operation correspondence for an operation on a changed line, since that one survives
|
||||
every other repair. Then rerun the same three revisions without the flag through the
|
||||
production evidence and comparison rules. Use
|
||||
[M1a evidence](m1a-recorded-vetoes.md) and
|
||||
[remediation research](../contributor-scan/release-remediation-research.md) as the starting
|
||||
record; do not begin another broad timing campaign before the M1 outcome exists.
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"feature_name": "pre-push-safety-net",
|
||||
"created_at": "2026-09-12T15:35:00Z",
|
||||
"updated_at": "2026-09-14T11:34:19.788221+00:00",
|
||||
"updated_at": "2026-09-16T13:30:00+00:00",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"approvals": {
|
||||
@@ -39,17 +39,17 @@
|
||||
{
|
||||
"at": "2026-09-12T16:20:40.814596Z",
|
||||
"source": "approve.",
|
||||
"action": "Accepted the generated task plan and authorized starting group 1 (1.1\u20131.4). Implementation started; tasks require independent review and fresh verification before completion."
|
||||
"action": "Accepted the generated task plan and authorized starting group 1 (1.1–1.4). Implementation started; tasks require independent review and fresh verification before completion."
|
||||
},
|
||||
{
|
||||
"at": "2026-09-12T18:32:19.033933Z",
|
||||
"source": "ok, next kiro phase, approved; correct. we can continue with the spec",
|
||||
"action": "Authorized group 2 (2.1\u20132.3), resumed after local Claude Mem repair. Independent review and fresh verification remain required."
|
||||
"action": "Authorized group 2 (2.1–2.3), resumed after local Claude Mem repair. Independent review and fresh verification remain required."
|
||||
},
|
||||
{
|
||||
"at": "2026-09-13T07:48:02.155687Z",
|
||||
"source": "continue with the task",
|
||||
"action": "Authorized continuing group 3 (3.1\u20133.4) after the group 3 next-step summary. Independent task review and fresh verification remain required."
|
||||
"action": "Authorized continuing group 3 (3.1–3.4) after the group 3 next-step summary. Independent task review and fresh verification remain required."
|
||||
},
|
||||
{
|
||||
"at": "2026-09-13T09:17:01.788332+00:00",
|
||||
@@ -59,7 +59,7 @@
|
||||
{
|
||||
"at": "2026-09-13T10:26:47.307154+00:00",
|
||||
"source": "move on to group 4 as a whole",
|
||||
"action": "Authorized all of group 4 (4.1\u20134.3), sequential implementation, independent task review and fresh verification. Groups 5\u20138 remain pending."
|
||||
"action": "Authorized all of group 4 (4.1–4.3), sequential implementation, independent task review and fresh verification. Groups 5–8 remain pending."
|
||||
},
|
||||
{
|
||||
"at": "2026-09-13T12:40:57.961577+00:00",
|
||||
@@ -69,17 +69,22 @@
|
||||
{
|
||||
"at": "2026-09-13T13:58:52.336506+00:00",
|
||||
"source": "continue with group 6 as a whole. do we need to amend the spec to get a go for the rollout",
|
||||
"action": "Authorized group 6 (6.1\u20136.4). Clarified complete preparation/discovery reuse after measured group 5 bottleneck; rollout gates remain unchanged. Manual Kiro review fallback remains necessary after a fresh agent dispatch hit the thread limit."
|
||||
"action": "Authorized group 6 (6.1–6.4). Clarified complete preparation/discovery reuse after measured group 5 bottleneck; rollout gates remain unchanged. Manual Kiro review fallback remains necessary after a fresh agent dispatch hit the thread limit."
|
||||
},
|
||||
{
|
||||
"at": "2026-09-13T15:45:15.902603+00:00",
|
||||
"source": "ok, continue with group 7 as a whole",
|
||||
"action": "Authorized tasks 7.1\u20137.3; manual Kiro fallback after fresh dispatch hit thread limit. Group 8 and rollout remain pending."
|
||||
"action": "Authorized tasks 7.1–7.3; manual Kiro fallback after fresh dispatch hit thread limit. Group 8 and rollout remain pending."
|
||||
},
|
||||
{
|
||||
"at": "2026-09-13T16:41:21.889949+00:00",
|
||||
"source": "move on to group 8",
|
||||
"action": "Authorized tasks 8.1\u20138.4 and their qualification work; preserve joint gates and record NO-GO wherever prerequisites or measurements do not pass. Fresh dispatch hit thread limit; manual Kiro fallback."
|
||||
"action": "Authorized tasks 8.1–8.4 and their qualification work; preserve joint gates and record NO-GO wherever prerequisites or measurements do not pass. Fresh dispatch hit thread limit; manual Kiro fallback."
|
||||
},
|
||||
{
|
||||
"at": "2026-09-16T13:30:00+00:00",
|
||||
"source": "add the M1a task to tasks.md and start it; continue with the plan",
|
||||
"action": "Added group 9 (M1a) to the approved task plan as the bounded owner amendment the milestone requires, implemented the task-local recorded-veto flag (9.1) and ran the three NodeGoat revisions (9.2). Diagnosis only: no capability enabled, no gate changed, rollout remains NO-GO."
|
||||
}
|
||||
],
|
||||
"task_review": {
|
||||
@@ -118,7 +123,9 @@
|
||||
"7.3",
|
||||
"8.1",
|
||||
"8.2",
|
||||
"8.4"
|
||||
"8.4",
|
||||
"9.1",
|
||||
"9.2"
|
||||
],
|
||||
"remaining_tasks": 1,
|
||||
"verification_record": "implementation-group-8.md",
|
||||
@@ -132,11 +139,11 @@
|
||||
"implementation-group-7.md",
|
||||
"implementation-group-8.md"
|
||||
],
|
||||
"next_group": 8,
|
||||
"next_group": 9,
|
||||
"blocked_tasks": [
|
||||
"8.3"
|
||||
],
|
||||
"feasibility": "NO-GO: post-census profile measured all 21 cases; 18 timed out, three identical-tip replays completed in 18.521\u201318.874s with valid reuse; 0/24 target checks completed. Maximum cancellation/report overrun 0.874s. See 2026-09-14-nodegoat-post-census-feasibility.json.",
|
||||
"feasibility": "NO-GO: post-census profile measured all 21 cases; 18 timed out, three identical-tip replays completed in 18.521–18.874s with valid reuse; 0/24 target checks completed. Maximum cancellation/report overrun 0.874s. See 2026-09-14-nodegoat-post-census-feasibility.json.",
|
||||
"qualification": "NO-GO: eligibility implementation and evidence delivered; required independent populations and verified VAmPI transitive witness remain missing. Zero capabilities enabled.",
|
||||
"validation": {
|
||||
"decision": "NO-GO",
|
||||
@@ -150,7 +157,15 @@
|
||||
"record": "../contributor-scan/census-repair-verification.md",
|
||||
"effect": "Node 44/44 input census restored; partial/missing census remains explicitly unresolved. Existing eligibility stale; runtime/context/independent qualification remain NO-GO."
|
||||
},
|
||||
"remediation_research": "../contributor-scan/release-remediation-research.md"
|
||||
"remediation_research": "../contributor-scan/release-remediation-research.md",
|
||||
"m1a": {
|
||||
"status": "MET",
|
||||
"date": "2026-09-16",
|
||||
"record": "m1a-recorded-vetoes.md",
|
||||
"measurement": "../../../benchmarks/measurements/2026-09-16-nodegoat-m1a-recorded-vetoes.json",
|
||||
"kill_criterion_triggered": false,
|
||||
"result": "Raw <lambda>2 witnesses present at contributions.js 32-34; fixed twin analyzed and quiet (target question answered with zero rows, no deadline reason); unchanged revision recorded 'unchanged'. Five vetoes fired and were recorded. With all five lifted the detection stops at operation_correspondence_unresolved, a fifth veto not previously named. No defect admitted, no capability enabled."
|
||||
}
|
||||
},
|
||||
"release": {
|
||||
"tag": "v1.2.0-alpha.1",
|
||||
@@ -163,10 +178,12 @@
|
||||
"record": "release-milestones.md",
|
||||
"direction_approved": true,
|
||||
"approval_source": "I agree. please note this into the raodmap as a spec",
|
||||
"current": "M1a",
|
||||
"verified": [],
|
||||
"current": "M1b",
|
||||
"verified": [
|
||||
"M1a"
|
||||
],
|
||||
"sequence": [
|
||||
"M1a: NodeGoat detection with admission vetoes recorded, not applied (one-week box)",
|
||||
"M1a: NodeGoat detection with admission vetoes recorded, not applied (MET 2026-09-16)",
|
||||
"M1b: repair only the vetoes M1a showed to matter; rerun through production rules",
|
||||
"M2: equivalent bounded runtime",
|
||||
"M3: shared-language transfer",
|
||||
@@ -174,6 +191,7 @@
|
||||
"M5: narrow advisory rollout"
|
||||
],
|
||||
"implementation_design_approval": "Existing approvals preserved; this records milestone order and exit criteria, not blanket approval of future design changes.",
|
||||
"m1_split_accepted": "2026-09-14: maintainer asked to split M1 into M1a/M1b after review"
|
||||
"m1_split_accepted": "2026-09-14: maintainer asked to split M1 into M1a/M1b after review",
|
||||
"m1a_outcome": "Met inside the one-week box. Corrected the owner diagnosis: all four named vetoes fire, and operation correspondence for an operation on a changed line is the blocker that survives every other repair."
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,17 +1,24 @@
|
||||
# Pre-push safety net status — v1.2.0-alpha.1
|
||||
|
||||
Updated 2026-09-14. Phase: **implementation**. Requirements, design and tasks are
|
||||
approved. **26/27 executable tasks complete; task 8.3 remains blocked.**
|
||||
Updated 2026-09-16. Phase: **implementation**. Requirements, design and tasks are
|
||||
approved. **28/29 executable tasks complete; task 8.3 remains blocked.**
|
||||
Rollout: **NO-GO**. The alpha tag publishes experimental implementation and evidence;
|
||||
it does not approve normal hook capabilities or production enforcement.
|
||||
|
||||
## Next release milestone
|
||||
|
||||
**M1a: show the NodeGoat detection with every admission veto recorded rather than
|
||||
applied, fixed twin analyzed and quiet, one-week time box. Then M1b: repair only the
|
||||
vetoes that fired and rerun through production rules.** None of the five release
|
||||
milestones is verified yet.
|
||||
The 26/27 task count is implementation progress, not proximity to rollout. Follow the
|
||||
**M1a is met. M1b is next: repair only the vetoes M1a showed to matter, then rerun the
|
||||
three revisions through the production rules.** The other four milestones remain unverified.
|
||||
|
||||
M1a ran on 2026-09-16 within its one-week box. The raw `<lambda>2` witnesses are present at
|
||||
contributions.js lines 32–34, the fixed twin was genuinely analyzed and quiet, and the
|
||||
unchanged revision was not reported as a new regression, so the kill criterion did not
|
||||
trigger. Five vetoes fired and were recorded; with all five lifted the detection still stops
|
||||
at `operation_correspondence_unresolved`, because a newly introduced operation sits on a
|
||||
changed line and has no base anchor. That fifth veto was not in the earlier diagnosis and is
|
||||
now M1b's central repair. See [M1a evidence](m1a-recorded-vetoes.md).
|
||||
|
||||
The 28/29 task count is implementation progress, not proximity to rollout. Follow the
|
||||
accepted [release milestone spec](release-milestones.md): usefulness → speed → shared
|
||||
PHP/Python transfer → independent qualification → narrow advisory rollout.
|
||||
|
||||
@@ -37,7 +44,10 @@ gates and a packaged native PHP/JavaScript partition smoke. See
|
||||
Still 0/24 declared target checks completed. Maximum observed cancellation/report
|
||||
overrun was 0.874s, below the two-second allowance. After M1 establishes the useful
|
||||
comparison, amortize engine startup/loading and rerun the frozen seven-class profile; require warm p95 <=30s and >=95% completed supported checks.
|
||||
2. **Target and change-context evidence remains incomplete.** VAmPI preserves raw findings
|
||||
2. **Target and change-context evidence remains incomplete.** M1a measured exactly which
|
||||
contracts fail on the Node case: per-family change context, question-owned vendor
|
||||
relevance, operation correspondence for changed lines, file-wide correspondence ambiguity
|
||||
and query-to-operation provenance. VAmPI preserves raw findings
|
||||
at all three known sites, but the exact SQLI function question/transitive witness and
|
||||
supported authorization context are unresolved. Raw findings or rank positions cannot
|
||||
establish an actionable new regression. Repair this in the shared scan boundary and
|
||||
|
||||
@@ -1,14 +1,16 @@
|
||||
# Implementation Plan: pre-push-safety-net
|
||||
|
||||
Status: the existing task plan remains approved. Groups 1–7 and tasks 8.1, 8.2 and 8.4
|
||||
are verified (26/27 executable subtasks); task 8.3 and rollout remain NO-GO. Later work
|
||||
are verified (28/29 executable subtasks, including group 9); task 8.3 and rollout remain NO-GO. Later work
|
||||
used the documented manual review fallback where agent dispatch reached its limit.
|
||||
|
||||
The accepted [release milestone spec](release-milestones.md) now orders remaining work:
|
||||
M1 useful Node comparison → M2 speed → M3 shared-language transfer → M4 independent
|
||||
qualification → M5 narrow advisory rollout. All five milestones remain unverified.
|
||||
These are release outcomes, not additional checked implementation tasks. Prepare bounded
|
||||
owner design/task amendments as needed; preserve all completed task evidence below.
|
||||
M1a diagnose → M1b repair → M2 speed → M3 shared-language transfer → M4 independent
|
||||
qualification → M5 narrow advisory rollout. All milestones remain unverified.
|
||||
Milestones are release outcomes, not checked implementation tasks. Group 9 below is the
|
||||
bounded owner amendment M1a requires; it adds the task-local recorded-veto evaluation and
|
||||
its three-revision development run. It enables no capability and changes no gate.
|
||||
Preserve all completed task evidence below.
|
||||
|
||||
## Execution contract
|
||||
|
||||
@@ -245,9 +247,30 @@ owner design/task amendments as needed; preserve all completed task evidence bel
|
||||
- _Requirements: 3.1, 3.4, 5.1, 5.2, 5.3, 5.4, 5.5, 6.5, 7.1, 7.3, 7.4, 8.3, 8.4_
|
||||
|
||||
|
||||
- [x] 9. Diagnose the first useful detection with vetoes recorded (release milestone M1a)
|
||||
- [x] 9.1 Add the task-local recorded-veto evaluation flag
|
||||
- Add an explicit experimental evaluation path that **records** each comparison and admission veto per raw finding instead of applying it. Keep it task-local, off by default, and available only on explicit `--base`/`--head` replay; refuse it on the Git stdin path so it can never become a hook capability.
|
||||
- Lift only recognized completeness vetoes over a copy of the evidence: answered-then-demoted question outcomes, the vendor exclusion completion veto, and the diagnosed context/correspondence boundary kinds. Never lift failures: deadline exhaustion, unanswered questions, unreadable input, empty graphs and unparsed files stay applied, because silence from failure is not a quiet fixed twin.
|
||||
- Retain the production novelty, reason and admission reasons beside the recorded ones for every finding, plus the query-to-witness identity difference, base-only findings and readable census receipts. Every payload carries an explicit experimental label; the recorded path admits no defect and enables no capability.
|
||||
- Done when the flag is off by default, produces a labeled artifact section and terminal line under explicit replay, is refused in Git stdin mode, and unit controls prove recognized vetoes are recorded while failures and unrecognized boundaries remain applied.
|
||||
- _Boundary: Delta and Actionability Policy, Push runner, Result and Hook Adapter, CLI integration_
|
||||
- _Requirements: 3.1, 3.4, 5.5, 6.3, 7.3_
|
||||
- _Verified 2026-09-16:_ 1,330 tests passed, nine skipped; global Ruff/format and touched mypy passed. Flag off by default, refused in Git stdin mode, labeled artifact section and terminal line under explicit replay. Controls prove recognized vetoes are recorded while deadline exhaustion, unanswered questions, empty graphs and unrecognized boundaries stay applied, and that the recorded path emits no defect even when a lifted candidate would otherwise admit. Regenerated the committed module audit for the added module.
|
||||
- [x] 9.2 Run the three NodeGoat revisions and produce the per-finding veto list
|
||||
- Record exact base/head revisions, expected operation and expected outcome before running. Exercise the shipped replay path, existing ranker and physically vendor-free graph; do not substitute a hand-selected slice or hard-coded target function.
|
||||
- Replay the pinned vulnerable, fixed and unchanged revisions under the flag with an explicitly named development budget. That budget cannot satisfy or measure the hook latency gate.
|
||||
- Deliverable: the raw finding on the vulnerable revision with location and witness, the fixed twin genuinely analyzed and quiet, the unchanged revision not reported as a new regression, and for every finding the exact list of vetoes that fired and why, with source and census receipts.
|
||||
- Done when one replayable three-revision record states each expectation against its observed outcome and names the vetoes that actually fired. A fixed twin that timed out or was not analyzed fails M1a and blocks M1b until diagnosed.
|
||||
- _Boundary: Evaluation harness, full-run integration_
|
||||
- _Depends: 9.1_
|
||||
- _Requirements: 1.1, 2.1, 2.4, 3.4, 6.3, 7.1, 7.3_
|
||||
- _Verified 2026-09-16:_ all three pinned revisions replayed through the production CLI under a named 900-second development budget; expectations frozen on disk first. Raw `<lambda>2` witnesses present at lines 32–34; fixed twin analyzed and quiet (target question answered with zero rows, no deadline reason, three base-only findings); unchanged revision analyzed and recorded `unchanged`. Five vetoes fired and were recorded; the remaining blocker is `operation_correspondence_unresolved` for an operation on a changed line. Zero admitted defects. Evidence: [M1a record](m1a-recorded-vetoes.md) and `benchmarks/measurements/2026-09-16-nodegoat-m1a-recorded-vetoes.json`. Diagnosis only; no capability enabled and no latency claim.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- Task5.2 measured PMPro timing out after both snapshots were materialized but before change context/graph analysis. Group6 must account for compatible snapshot/discovery reuse as well as graph reuse; identical-tip hits cannot substitute for the later representative changed-code latency gate. No new detector, scope heuristic or incremental CPG mutation is implied.
|
||||
- Cancellation must cover semantic planning and reporting, not only engine subprocesses: `_collect` and `_scope_work` now consume the shared deadline, and replay instantiates the budget-aware backend once. Preserve the 400-question failure-census and real Node artifact-retention regressions.
|
||||
|
||||
- Group 9 (M1a) corrected the owner diagnosis: all four named vetoes fire on the NodeGoat case, and a fifth, `operation_correspondence_unresolved`, is the actual remaining blocker because a newly introduced operation sits on a changed line and has no base anchor. The unchanged revision resolves its base location on the same lines and is classified `unchanged`, which isolates correspondence as the mechanism. M1b repairs context producers, question-owned vendor relevance, operation correspondence, file-wide ambiguity scope and query-to-operation provenance; vetoes that did not fire stay unchanged.
|
||||
|
||||
- Group 6: complete discovery/graph/query reuse is explicit via `--cache-dir`; source/declaration edits invalidate affected units. Installed frontend/adaptation bytes now bind graph and admission provenance. A valid cached census cannot qualify a fresh answer that omitted its own census. Identical-revision timing is not the changed-code rollout gate.
|
||||
|
||||
@@ -8,14 +8,14 @@ fast, prove cross-language transfer, qualify independently, then enable narrow a
|
||||
rollout**. The executable outcome contract lives in the existing
|
||||
[pre-push-safety-net release milestone spec](../specs/pre-push-safety-net/release-milestones.md).
|
||||
|
||||
**We are at M1a. None of the five release milestones is verified yet.** The published
|
||||
`v1.2.0-alpha.1` and 26/27 completed implementation tasks describe delivered machinery,
|
||||
**We are at M1b. M1a is met; no other release milestone is verified yet.** The published
|
||||
`v1.2.0-alpha.1` and 28/29 completed implementation tasks describe delivered machinery,
|
||||
not a demonstrated useful safety net. Task 8.3 and rollout remain NO-GO.
|
||||
|
||||
| Order | Milestone | Observable exit | Current state |
|
||||
|---|---|---|---|
|
||||
| M1a | Show the NodeGoat detection with every admission veto recorded, not applied | Raw finding with witness on the vulnerable revision; analyzed fixed twin and unchanged control quiet; per-finding veto list. One-week time box | NEXT |
|
||||
| M1b | Repair only the vetoes M1a showed to matter | Supported evaluation finding through production rules, exact change/base evidence and repair direction; fixed twin and unchanged control stay quiet | Pending M1a |
|
||||
| M1a | Show the NodeGoat detection with every admission veto recorded, not applied | Raw finding with witness on the vulnerable revision; analyzed fixed twin and unchanged control quiet; per-finding veto list. One-week time box | MET 2026-09-16 |
|
||||
| M1b | Repair only the vetoes M1a showed to matter | Supported evaluation finding through production rules, exact change/base evidence and repair direction; fixed twin and unchanged control stay quiet | NEXT |
|
||||
| M2 | Make that same analysis fast | Equivalent evidence; representative warm changed-code p95 <=30s, cancellation/report <=2s, >=95% supported completion | Pending M1 |
|
||||
| M3 | Prove the shared design transfers | PMPro/PHP and VAmPI/Python vulnerable/fixed evidence; unchanged core rules; Node replay after any PMPro core change | Pending M2 |
|
||||
| M4 | Qualify on independent cases | Reviewed untouched vulnerable/fixed/benign populations pass joint quality, coverage and latency gates per capability | Pending M3 |
|
||||
@@ -23,12 +23,15 @@ not a demonstrated useful safety net. Task 8.3 and rollout remain NO-GO.
|
||||
|
||||
### Immediate next action
|
||||
|
||||
Add a task-local recorded-veto experimental flag and run the pinned NodeGoat
|
||||
vulnerable, fixed and unchanged revisions under it (M1a). The raw `<lambda>2` witness
|
||||
already exists; the deliverable is that witness plus the exact list of admission vetoes
|
||||
that discard it. Then repair only those vetoes and rerun through production rules (M1b).
|
||||
Normal hook eligibility stays disabled during this experiment.
|
||||
Repair the five vetoes M1a recorded, then rerun the same three NodeGoat revisions without
|
||||
the flag through the production rules (M1b). Start with operation correspondence for an
|
||||
operation on a changed line: M1a measured that this one survives every other repair, and it
|
||||
was not in the earlier four-veto diagnosis. The others are per-family change context,
|
||||
question-owned vendor relevance, file-wide correspondence ambiguity scope and
|
||||
query-to-operation provenance. Vetoes that did not fire stay unchanged and their families
|
||||
stay experimental. Normal hook eligibility stays disabled.
|
||||
Do not start another broad speed campaign before this outcome exists. See the
|
||||
[M1a evidence](../specs/pre-push-safety-net/m1a-recorded-vetoes.md) and the
|
||||
[owner diagnosis](../specs/contributor-scan/release-remediation-research.md).
|
||||
|
||||
### Approach and boundaries
|
||||
|
||||
@@ -6,9 +6,11 @@ WordPress-first release framing in older steering; historical measurements remai
|
||||
Release order accepted 2026-09-14: first prove one useful Node changed-code comparison
|
||||
with an analyzed quiet fixed twin, then make the same analysis fast, verify PHP/Python
|
||||
transfer, qualify independently, and enable only passing advisory capabilities. See
|
||||
[the milestone spec](../specs/pre-push-safety-net/release-milestones.md). We are at M1a
|
||||
(show the detection with vetoes recorded, one-week box), then M1b (repair only what fired);
|
||||
task counts and an experimental alpha tag do not establish release readiness. PMPro
|
||||
[the milestone spec](../specs/pre-push-safety-net/release-milestones.md). M1a is met
|
||||
(2026-09-16: witness shown, fixed twin analyzed and quiet, five vetoes recorded); we are at
|
||||
M1b, repairing only what fired. M1a added a fifth veto the earlier diagnosis missed:
|
||||
operation correspondence for an operation on a changed line.
|
||||
Task counts and an experimental alpha tag do not establish release readiness. PMPro
|
||||
remains a core regression/development case; any PMPro core retune still requires frozen
|
||||
Node transfer evidence. This sequence supersedes older runtime-first next-step wording.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user