Open_Ultra_SAST/.kiro/specs/pre-push-safety-net/evaluation.md

Evaluation: core optimization and cross-language transfer

Status: evaluation contract accepted with requirements/design for task generation, 2026-09-12. Implements requirements 6.1–6.5 and 8.1–8.4. Task 1.2 now pins development and reserved inputs in benchmarks/push/inputs.json; see its README for provenance and missing prerequisites. No new scanner result or capability admission is claimed.

Workload roles

Workload	Role	Required evidence
Existing pinned PMPro	Primary development and cost-attribution example	Known CVE retained; fixed/benign controls; total build/evidence/query cost
Existing WP Statistics and MW WP Form	PHP structural regressions	Hook/field flows preserved; vendor exclusion does not become a ranking trick
JavaScript/Express Node API	Required transfer and hook integration workload	Same frozen core; route/middleware/handler/service context; vulnerable/fixed/benign changes
Separate untouched Node API repository	Generalization evaluation	Predeclared labels/population; no tuning on its results; all misses and gaps retained
Existing pinned VAmPI	Python and authorization regression	Actual transitive SQLI witness plus obligation findings; no omitted position targets
Existing pinned libpng	C graph/cost and capability-limit control	Envelope measured; unsupported arithmetic/bounds explicitly reported
Future C or C++ positive control	Additional abstract-property transfer	Reviewed vuln/fix for a supported property; C++ frontend behavior verified independently

Recommended JavaScript starting point

OWASP NodeGoat is the proposed first Express wiring/teaching candidate. It is a deliberately vulnerable Node application with remediation teaching material, and its package manifest declares Express. It exercises server-side code without making an entire browser application the primary measurement unit. Project, package manifest.

This is candidate selection, not detection evidence. Before adding an executable recipe, pin an immutable commit, verify readable source/license bytes, identify the exact affected operation and its caller context, and review the vulnerable and fixed forms. An authored repair must be labeled as authored; do not present it as an upstream CVE fix. Do not assume the tutorial's vulnerability categories correspond to supported arbiters: a MongoDB operator-injection case is not automatically covered by SQL string-taint facts. Select a supported positive control and retain other cases as gaps.

NodeGoat is intentionally educational and will be inspected during integration. It cannot establish production generalization on its own. Its inspected manifest uses an older Express 4 dependency range; it is not evidence of coverage for current module/async/framework patterns. Require those patterns in the additional Node workload and record its exact versions. Reserve an independently selected Node API repository with reviewable security-fix history before the transfer result is used to tune the core. Selection must not depend on getting a favorable scanner score. No npm installation or project execution is needed to establish the static input; runtime controls, when needed, belong in a separate controlled harness.

Paired changes and negative controls

1. Request input crosses route -> imported handler -> service -> supported dangerous operation; a correct family-specific repair stops the finding on the fixed side.
2. A removed authorization/ownership guard exposes an unchanged operation. A guard that is present but does not govern the operation must not discharge it; correctly ordered governing middleware must.
3. Callback/async data transfer preserves a supported source-to-operation witness. Unresolved dynamic calls are counted as a context gap, not silently treated as no flow.
4. Input validation for an unrelated context does not cleanse SQL/command/path operations. Path normalization alone is not assumed to establish confinement to an allowed root. Test both the real repair and a plausible but insufficient repair.
5. Benign refactoring, renaming, formatting, parameter binding and unaffected backlog do not generate new hook alerts. Retain benign push counts so low noise cannot be inferred only from security cases.
6. node_modules and other declared vendor files are absent from both graph and targets, with first-party callers still represented. Missing external semantics remain visible coverage limits.

Do not force all these shapes into one repository. Keep authored cross-language contract tests separate from reviewed repository changes, and preserve their provenance in the scorecard.

Freeze, transfer, classify

1. Freeze core ranker features/weights, arbiter/query semantics, engine version, budget, and existing framework facts after the PMPro experiment. Run Node with available adapters first; save that result.
2. Classify each miss before changing anything: unread input/instrument failure, frontend normalization, missing framework fact, ranker scope/budget, wrong abstract semantics, or unsupported abstract domain.
3. If a framework fact or frontend adaptation is needed, review it separately and rerun both languages. Report adapted transfer separately from the unchanged-core result. A core change starts a new version and requires a fresh untouched evaluation; it is not hidden in a "JavaScript config" label.
4. Compare actual findings, fixed-side silence, actionable precision, benign interruptions, coverage, and elapsed cost together. Equivalent normalized evidence must produce identical core tier/score results; equivalent syntax across languages is not required and is not presumed.
5. Do not generalize a C envelope result to C/C++ detection. Select a positive control only after its property fits a declared arbiter; numerical bounds/overflow needs a suitable domain and separate validation. Do not expand taint sources or sanitizer names to simulate arithmetic reasoning.

First bounded implementation experiment

Replay one supported PMPro change/fix and one Node/Express change/fix through the same ranker, graph exclusion policy, deadline and admission contract. Add at least one benign change per workload. Produce one joint scorecard before any further core optimization. A missing Node frontend, broken middleware mapping or wrong sanitizer model is an explicit finding of this experiment, not a reason to remove Node from the evaluation or silently fall back to a PHP-only release claim.

Task 1.2 input preparation

The versioned push manifest records upstream PMPro vulnerable/fixed revisions, NodeGoat with an explicitly authored eval-removal repair, and comment-only benign changes that retain vulnerable backlog. Ghost is reserved independently for production Node module/async API evaluation; its reviewed supported-family pair population remains missing. The validator verifies immutable Git blob identities and source/license byte counts offline, retains every unresolved case, and keeps all capabilities experimental. Independent PHP and Python admission populations remain prerequisites; VAmPI regressions and the libpng unsupported-property envelope retain their separate roles. See benchmarks/push/README.md for the reproducible command and provenance boundaries.