feat(pre-push-safety-net): pin labeled evaluation inputs

This commit is contained in:
Marius Ciepluch
2026-09-12 18:45:54 +02:00
parent 56cfc9da89
commit 8acada0d98
11 changed files with 1194 additions and 6 deletions
@@ -1,7 +1,7 @@
# Evaluation: core optimization and cross-language transfer
Status: evaluation contract accepted with requirements/design for task generation, 2026-09-12. Implements requirements 6.1–6.5 and 8.1–8.4.
No new repository has been pinned, scanned or admitted by this document.
Task 1.2 now pins development and reserved inputs in `benchmarks/push/inputs.json`; see its README for provenance and missing prerequisites. No new scanner result or capability admission is claimed.
## Workload roles
@@ -80,3 +80,7 @@ graph exclusion policy, deadline and admission contract. Add at least one benign
Produce one joint scorecard before any further core optimization. A missing Node frontend, broken
middleware mapping or wrong sanitizer model is an explicit finding of this experiment, not a reason
to remove Node from the evaluation or silently fall back to a PHP-only release claim.
## Task 1.2 input preparation
The versioned push manifest records upstream PMPro vulnerable/fixed revisions, NodeGoat with an explicitly authored eval-removal repair, and comment-only benign changes that retain vulnerable backlog. Ghost is reserved independently for production Node module/async API evaluation; its reviewed supported-family pair population remains missing. The validator verifies immutable Git blob identities and source/license byte counts offline, retains every unresolved case, and keeps all capabilities experimental. Independent PHP and Python admission populations remain prerequisites; VAmPI regressions and the libpng unsupported-property envelope retain their separate roles. See `benchmarks/push/README.md` for the reproducible command and provenance boundaries.
+2 -1
View File
@@ -29,7 +29,7 @@ started with group 1 (tasks 1.1–1.4); each task remains pending until independ
- _Boundary: Cache and Engine Boundary, packaging integration_
- _Requirements: 7.1, 7.3, 8.2_
- _Verified 2026-09-12:_ independent review APPROVED; 888 tests passed, 9 skipped. Actual local image PHP/JavaScript graph census and source witnesses passed, unreadable PHP explicitly refused. Parent verified current provenance hashes, image identity, non-root user and CLI/frontend liveness. Evidence: `benchmarks/measurements/2026-09-12-engine-runtime-smoke.json`. Runtime readiness only; no hook latency/detection claim. Existing mypy errors in `model/scan.py` at 299/302/310 remain unchanged.
- [ ] 1.2 Prepare labeled development and reserved evaluation changes
- [x] 1.2 Prepare labeled development and reserved evaluation changes
- Pin a readable PMPro change/fix and a supported Node/Express change/fix, plus benign controls; verify license and source bytes before accepting recipes.
- Use NodeGoat only if its selected mechanism fits a current arbiter; label authored repairs as authored. Reserve an independently selected Node API repository with modern module/async context and reviewable history before tuning on transfer results.
- Keep Python regression and libpng envelope recipes, explicit supported/unsupported populations, ref identities and development/holdout roles in machine-readable evaluation inputs.
@@ -37,6 +37,7 @@ started with group 1 (tasks 1.1–1.4); each task remains pending until independ
- Done when recipe validation resolves immutable inputs and labels or names an explicit missing prerequisite; no favorable scanner result is required to select a recipe.
- _Boundary: Evaluation harness_
- _Requirements: 6.1, 6.3, 8.3, 8.4_
- _Verified 2026-09-12:_ independent review APPROVED; 902 tests passed, 9 skipped, then 34 focused tests passed after a diagnostic-only correction. Fresh offline validation opened 506,502 bytes and retained all 11 cases/9 snapshots: 7 cases input-ready, 4 with named missing prerequisites. See `benchmarks/push/inputs.json` and `input-validation.json`. Ghost checkout/reviewed labels and independent per-capability PHP/Python populations remain missing; every capability stays experimental. No detector run or admission claim.
- [ ] 1.3 Make the baseline and push scorer fail honestly
- Retain unmatched targets and unanswered queries in the declared population; require a queried witness before labeling a target transitively detected.
- Emit actual vulnerable/fixed outcomes beside rank, selected mode, engine/facts/scope and elapsed stage costs. Freeze a versioned workload/profile before measuring.