feat(pre-push-safety-net): enforce evidence eligibility and record blocked qualification

This commit is contained in:
Marius Ciepluch
2026-09-13 19:32:43 +02:00
parent 2491ce497c
commit b506516d87
14 changed files with 7394 additions and 17 deletions
@@ -41,3 +41,26 @@ and its compressed full evidence bundle. All 21 deadline-bound transactions time
Every measured artifact matches the frozen installed semantics. No source, engine,
query, fact or ranking change was made during this run. The follow-up remains necessary
before rollout; the evaluation task does not authorize weakening its gates.
## VAmPI qualification follow-up, 2026-09-13
Task 8.3's packaged frozen regression (139,164 total snapshot bytes read per replay)
retains raw findings at `models/user_model.py:73:get_user`,
`api_views/users.py:187:update_password`, and `api_views/books.py:51:get_by_title`.
The SQLI target has no selected exact function question; the two authorization target
questions are selected at positions 93 and 63 but have `change_context_incomplete`.
Head scope retains 138 selected and 20 deferred questions. Raw findings do not prove
completed target questions or a transitive source-to-target witness.
The first lab run took 112.415s; same-snapshot cached replays took 6.816s and 6.684s,
all incomplete. This is diagnostic reuse, not changed-code performance qualification.
The 600-second budget is distinct from the hook acceptance profile. Full artifacts are
in `2026-09-13-push-qualification.json.gz` under the regression population.
Reconcile evidence normalization and the unsupported context projection at the shared
scan boundary before downstream admission: retain the actual query-origin/path witness
when available, resolve supported first-party context, and keep unresolved dynamic/vendor
boundaries explicit. Do not fabricate transitivity from a raw finding at the target.
Revalidate ordinary VAmPI scan, exact question accounting, push regressions and independent
qualification after an approved owner fix. This note records the prerequisite; it does
not silently approve an upstream implementation or change detector semantics.
@@ -106,3 +106,98 @@ retune the ranker or weaken the population/gates to compensate.
- CLAIM: Task 8.2 completed its prescribed measurement and owner routing; rollout remains NO-GO.
- EVIDENCE: Versioned scorecard and 116-file verified bundle; seven scorer controls and full regression suite pass.
- GAPS: No qualified warm graph path or independent capability eligibility established. These failed gates cannot authorize rollout.
## Task 8.3 brief
Requirements 3.1, 3.2, 6.1–6.5 and 8.1–8.4; evaluation and actionability integration.
Implement a trusted installed eligibility resource whose evidence is recomputed against
exact core/engine/runtime/facts/query/policy/semantic identities. Require reviewed
operation/advice scope, vulnerable/fixed/benign populations, nonzero actual positive
controls and reviewed alerts, untouched per-capability evaluation and all joint gates.
The normal runner consumes only eligible declarations; stale or inconsistent resources
admit nothing. This is integrity checking of trusted release evidence, not signing or
permission to load repository-authored eligibility.
Freeze the current packaged candidate before the six PMPro/Node development replays.
Run readable VAmPI regression snapshots under a separate 600-second laboratory profile;
keep their missing independent pairs unresolved. Execute libpng as an unsupported
30-second envelope control. Do not inspect or tune on Ghost before its reviewed
population exists. Missing required evidence leaves task 8.3 qualification blocked,
even when the reproducible NO-GO artifact and admission implementation are delivered.
### Review scope
Synthetic controls prove that a matching reviewed population can reach the real runner
admission seam, while stale facts/runtime, missing positive observations, absent reviews,
wrong untouched workloads and changed consequence/repair semantics cannot. They are
contract controls, not real capability qualification. The installed NO-GO resource must
recompute identically and enable zero capabilities. The source has no repository- or
language-specific rank changes, frontend adaptations or new abstract domains.
### Task 8.3 measured outcome
Packaged candidate `9464e229ccfdc873a02e3a0db7c1e83a6d6ec6d3e8234ae7b97677b4aed836fd`
executed offline as UID 1000 with a 3 GiB memory limit. Each profile was written before
its runs. All six development pushes saved artifacts, timed out in 30.559–30.678s and
retained all 11 declared cases. Source snapshot receipts read 57.9–58.1 MB per PMPro
comparison and 5.95 MB per Node comparison. These are snapshot bytes, not graph census.
The supported introducing-change recall is 0/2, Wilson 95% interval [0, 0.658]; actual
reviewed alert precision is undefined (0/0), never 100%. Neither fixed case has a
completed witnessed negative. The two benign pushes emitted no alerts; that silence
with incomplete coverage does not establish a useful safety net.
VAmPI's 600-second laboratory profile executed all three pinned regression targets,
reading 139,164 snapshot bytes per replay. First run 112.415s; subsequent identical
comparisons 6.816s and 6.684s, each with two graph hits and 403 query hits. Raw findings
retain the known SQLI and two authorization sites, but the SQLI exact function question
is unlocated and authorization questions are `change_context_incomplete`. All three
remain unresolved; no transitive witness or completed regression detection is credited.
There are 138 selected and 20 deferred head questions. Context/witness normalization
is routed to the shared scan owner. A registry-generation provenance probe overlapped
startup of the first lab run; these timings are diagnostic, not runtime-gate samples.
The initial instrument labeled all regression replays cold. The two proved identical
cache replays were relabeled `identical_tip` and rescored without changing measurements,
profile or population; original run/scorecard/instrument and correction are retained.
Future instrument output distinguishes these identical comparisons. None is a warm edit.
The libpng unsupported envelope saved a partial artifact after 30.590s. Ghost's checkout
and reviewed independent cases remain missing and were not scanned. Independent PHP
and Python admission populations also remain missing. No ranker, fact, adapter or query
retune occurred. The 59-file compressed evidence bundle is verified against originals:
`benchmarks/measurements/2026-09-13-push-qualification.json.gz`, SHA-256
`79226b4e54042559ba5cc2b94b69b1dcddb550872fd0783b2380368be72f90bd`.
The installed eligibility resource is reproducible NO-GO, digest
`2ddd94d4dc3fd6b84e7de38e5f211d2653fb993c64e4e43cedf7e06062c3265c`, with zero enabled
capabilities. Task 8.2 runtime evidence remains failed and predates the eligibility
policy addition; it is correctly rejected as admission evidence, not relabeled current.
## Status Report
- STATUS: BLOCKED
- TASK: 8.3
- RED_PHASE_OUTPUT: `/tmp/group8-3-red.log`: eight failures with eligibility disabled; ON passed, switch removed; final synthetic and installed-resource contracts verified separately.
- IMPLEMENTED: Versioned eligibility generation/loading, production admission integration, real offline replay and NO-GO artifact.
- BLOCKER: Required independent Node/PHP/Python reviewed populations and verified VAmPI transitive question witness are absent. Runtime/completion gates fail.
## Review Verdict
- VERDICT: REJECTED
- TASK: 8.3 completion claim
- MECHANICAL_RESULTS: Eligibility and runner implementation inspected; fixed semantic/advice scope, observed controls and exact provenance required. No language-specific ranker changes, placeholders or secrets. RED phase verified.
- FINDINGS: Implementation and reproducible NO-GO decision are concrete, but task 8.3's mandatory independent transfer and transitive-regression evidence is not established. Main-context fallback does not provide independent review.
- REMEDIATION: Repair the shared context/witness and census/runtime prerequisites; obtain reviewed untouched paired populations, refreeze and rerun all joint gates. Keep capabilities disabled meanwhile.
- SUMMARY: Retain the useful implementation, leave task 8.3 unchecked and feature rollout NO-GO.
## Verification Result
- STATUS: NOT_VERIFIED
- CLAIM_TYPE: TASK
- CLAIM: Full task 8.3 qualification complete.
- EVIDENCE: Real frozen replay and installed decision explicitly preserve failed gates and missing evidence.
- GAPS: Independent populations, completed positive/fixed controls, VAmPI transitive witness and representative runtime/coverage.
- NOTES: Proceed with authorized task 8.4's explicit NO-GO acceptance reconciliation; it does not depend on pretending qualification passed.
Task 8.3 implementation verification: full suite **1317 passed, 9 skipped in 115.80s**
(`/tmp/group8-3-complete-suite.log`, exit 0). After the measurement-label correction,
14 focused eligibility/replay-instrument tests passed in 0.95s. Ruff, format, mypy
(116 source files) and diff checks pass. These verify the delivered code and recorded
NO-GO behavior, not the rejected full qualification claim above.
+5 -2
View File
@@ -132,7 +132,10 @@
"implementation-group-8.md"
],
"next_group": 8,
"blocked_tasks": [],
"feasibility": "NO-GO: all 21 frozen NodeGoat transactions timed out; 0/24 declared target checks completed. Cancellation/reporting remained below 2 seconds. Priming did not qualify graph reuse; named census and engine lifecycle work routed to contributor-scan."
"blocked_tasks": [
"8.3"
],
"feasibility": "NO-GO: all 21 frozen NodeGoat transactions timed out; 0/24 declared target checks completed. Cancellation/reporting remained below 2 seconds. Priming did not qualify graph reuse; named census and engine lifecycle work routed to contributor-scan.",
"qualification": "NO-GO: eligibility implementation and evidence delivered; required independent populations and verified VAmPI transitive witness remain missing. Zero capabilities enabled."
}
}
+1
View File
@@ -234,6 +234,7 @@ independent capability qualification. Groups 7–8 remain pending; rollout remai
- _Boundary: Evaluation harness, Delta and Actionability Policy integration_
- _Depends: 1.2, 1.3, 5.2, 7.3, 8.2_
- _Requirements: 3.1, 3.2, 6.1, 6.2, 6.3, 6.4, 6.5, 8.1, 8.2, 8.3, 8.4_
- _Blocked 2026-09-13:_ Eligibility implementation and reproducible NO-GO artifact delivered; independent reviewed Node/PHP/Python populations and verified VAmPI transitive witness remain missing. Runtime/coverage gates fail. See `implementation-group-8.md` and contributor-scan owner follow-up.
- [ ] 8.4 Verify the packaged end-to-end outcome and acceptance coverage
- Run the required regression checks and a local packaged smoke over supported security/fixed/benign pushes; verify no-model startup, preservation of hook behavior and truthful partial results.
- Reconcile every acceptance criterion with fresh evidence and record unresolved capability/performance blockers separately from completed plumbing.