feat(pre-push-safety-net): enforce evidence eligibility and record blocked qualification

This commit is contained in:
Marius Ciepluch
2026-09-13 19:32:43 +02:00
parent 2491ce497c
commit b506516d87
14 changed files with 7394 additions and 17 deletions
@@ -41,3 +41,26 @@ and its compressed full evidence bundle. All 21 deadline-bound transactions time
Every measured artifact matches the frozen installed semantics. No source, engine, Every measured artifact matches the frozen installed semantics. No source, engine,
query, fact or ranking change was made during this run. The follow-up remains necessary query, fact or ranking change was made during this run. The follow-up remains necessary
before rollout; the evaluation task does not authorize weakening its gates. before rollout; the evaluation task does not authorize weakening its gates.
## VAmPI qualification follow-up, 2026-09-13
Task 8.3's packaged frozen regression (139,164 total snapshot bytes read per replay)
retains raw findings at `models/user_model.py:73:get_user`,
`api_views/users.py:187:update_password`, and `api_views/books.py:51:get_by_title`.
The SQLI target has no selected exact function question; the two authorization target
questions are selected at positions 93 and 63 but have `change_context_incomplete`.
Head scope retains 138 selected and 20 deferred questions. Raw findings do not prove
completed target questions or a transitive source-to-target witness.
The first lab run took 112.415s; same-snapshot cached replays took 6.816s and 6.684s,
all incomplete. This is diagnostic reuse, not changed-code performance qualification.
The 600-second budget is distinct from the hook acceptance profile. Full artifacts are
in `2026-09-13-push-qualification.json.gz` under the regression population.
Reconcile evidence normalization and the unsupported context projection at the shared
scan boundary before downstream admission: retain the actual query-origin/path witness
when available, resolve supported first-party context, and keep unresolved dynamic/vendor
boundaries explicit. Do not fabricate transitivity from a raw finding at the target.
Revalidate ordinary VAmPI scan, exact question accounting, push regressions and independent
qualification after an approved owner fix. This note records the prerequisite; it does
not silently approve an upstream implementation or change detector semantics.
@@ -106,3 +106,98 @@ retune the ranker or weaken the population/gates to compensate.
- CLAIM: Task 8.2 completed its prescribed measurement and owner routing; rollout remains NO-GO. - CLAIM: Task 8.2 completed its prescribed measurement and owner routing; rollout remains NO-GO.
- EVIDENCE: Versioned scorecard and 116-file verified bundle; seven scorer controls and full regression suite pass. - EVIDENCE: Versioned scorecard and 116-file verified bundle; seven scorer controls and full regression suite pass.
- GAPS: No qualified warm graph path or independent capability eligibility established. These failed gates cannot authorize rollout. - GAPS: No qualified warm graph path or independent capability eligibility established. These failed gates cannot authorize rollout.
## Task 8.3 brief
Requirements 3.1, 3.2, 6.1–6.5 and 8.1–8.4; evaluation and actionability integration.
Implement a trusted installed eligibility resource whose evidence is recomputed against
exact core/engine/runtime/facts/query/policy/semantic identities. Require reviewed
operation/advice scope, vulnerable/fixed/benign populations, nonzero actual positive
controls and reviewed alerts, untouched per-capability evaluation and all joint gates.
The normal runner consumes only eligible declarations; stale or inconsistent resources
admit nothing. This is integrity checking of trusted release evidence, not signing or
permission to load repository-authored eligibility.
Freeze the current packaged candidate before the six PMPro/Node development replays.
Run readable VAmPI regression snapshots under a separate 600-second laboratory profile;
keep their missing independent pairs unresolved. Execute libpng as an unsupported
30-second envelope control. Do not inspect or tune on Ghost before its reviewed
population exists. Missing required evidence leaves task 8.3 qualification blocked,
even when the reproducible NO-GO artifact and admission implementation are delivered.
### Review scope
Synthetic controls prove that a matching reviewed population can reach the real runner
admission seam, while stale facts/runtime, missing positive observations, absent reviews,
wrong untouched workloads and changed consequence/repair semantics cannot. They are
contract controls, not real capability qualification. The installed NO-GO resource must
recompute identically and enable zero capabilities. The source has no repository- or
language-specific rank changes, frontend adaptations or new abstract domains.
### Task 8.3 measured outcome
Packaged candidate `9464e229ccfdc873a02e3a0db7c1e83a6d6ec6d3e8234ae7b97677b4aed836fd`
executed offline as UID 1000 with a 3 GiB memory limit. Each profile was written before
its runs. All six development pushes saved artifacts, timed out in 30.559–30.678s and
retained all 11 declared cases. Source snapshot receipts read 57.9–58.1 MB per PMPro
comparison and 5.95 MB per Node comparison. These are snapshot bytes, not graph census.
The supported introducing-change recall is 0/2, Wilson 95% interval [0, 0.658]; actual
reviewed alert precision is undefined (0/0), never 100%. Neither fixed case has a
completed witnessed negative. The two benign pushes emitted no alerts; that silence
with incomplete coverage does not establish a useful safety net.
VAmPI's 600-second laboratory profile executed all three pinned regression targets,
reading 139,164 snapshot bytes per replay. First run 112.415s; subsequent identical
comparisons 6.816s and 6.684s, each with two graph hits and 403 query hits. Raw findings
retain the known SQLI and two authorization sites, but the SQLI exact function question
is unlocated and authorization questions are `change_context_incomplete`. All three
remain unresolved; no transitive witness or completed regression detection is credited.
There are 138 selected and 20 deferred head questions. Context/witness normalization
is routed to the shared scan owner. A registry-generation provenance probe overlapped
startup of the first lab run; these timings are diagnostic, not runtime-gate samples.
The initial instrument labeled all regression replays cold. The two proved identical
cache replays were relabeled `identical_tip` and rescored without changing measurements,
profile or population; original run/scorecard/instrument and correction are retained.
Future instrument output distinguishes these identical comparisons. None is a warm edit.
The libpng unsupported envelope saved a partial artifact after 30.590s. Ghost's checkout
and reviewed independent cases remain missing and were not scanned. Independent PHP
and Python admission populations also remain missing. No ranker, fact, adapter or query
retune occurred. The 59-file compressed evidence bundle is verified against originals:
`benchmarks/measurements/2026-09-13-push-qualification.json.gz`, SHA-256
`79226b4e54042559ba5cc2b94b69b1dcddb550872fd0783b2380368be72f90bd`.
The installed eligibility resource is reproducible NO-GO, digest
`2ddd94d4dc3fd6b84e7de38e5f211d2653fb993c64e4e43cedf7e06062c3265c`, with zero enabled
capabilities. Task 8.2 runtime evidence remains failed and predates the eligibility
policy addition; it is correctly rejected as admission evidence, not relabeled current.
## Status Report
- STATUS: BLOCKED
- TASK: 8.3
- RED_PHASE_OUTPUT: `/tmp/group8-3-red.log`: eight failures with eligibility disabled; ON passed, switch removed; final synthetic and installed-resource contracts verified separately.
- IMPLEMENTED: Versioned eligibility generation/loading, production admission integration, real offline replay and NO-GO artifact.
- BLOCKER: Required independent Node/PHP/Python reviewed populations and verified VAmPI transitive question witness are absent. Runtime/completion gates fail.
## Review Verdict
- VERDICT: REJECTED
- TASK: 8.3 completion claim
- MECHANICAL_RESULTS: Eligibility and runner implementation inspected; fixed semantic/advice scope, observed controls and exact provenance required. No language-specific ranker changes, placeholders or secrets. RED phase verified.
- FINDINGS: Implementation and reproducible NO-GO decision are concrete, but task 8.3's mandatory independent transfer and transitive-regression evidence is not established. Main-context fallback does not provide independent review.
- REMEDIATION: Repair the shared context/witness and census/runtime prerequisites; obtain reviewed untouched paired populations, refreeze and rerun all joint gates. Keep capabilities disabled meanwhile.
- SUMMARY: Retain the useful implementation, leave task 8.3 unchecked and feature rollout NO-GO.
## Verification Result
- STATUS: NOT_VERIFIED
- CLAIM_TYPE: TASK
- CLAIM: Full task 8.3 qualification complete.
- EVIDENCE: Real frozen replay and installed decision explicitly preserve failed gates and missing evidence.
- GAPS: Independent populations, completed positive/fixed controls, VAmPI transitive witness and representative runtime/coverage.
- NOTES: Proceed with authorized task 8.4's explicit NO-GO acceptance reconciliation; it does not depend on pretending qualification passed.
Task 8.3 implementation verification: full suite **1317 passed, 9 skipped in 115.80s**
(`/tmp/group8-3-complete-suite.log`, exit 0). After the measurement-label correction,
14 focused eligibility/replay-instrument tests passed in 0.95s. Ruff, format, mypy
(116 source files) and diff checks pass. These verify the delivered code and recorded
NO-GO behavior, not the rejected full qualification claim above.
+5 -2
View File
@@ -132,7 +132,10 @@
"implementation-group-8.md" "implementation-group-8.md"
], ],
"next_group": 8, "next_group": 8,
"blocked_tasks": [], "blocked_tasks": [
"feasibility": "NO-GO: all 21 frozen NodeGoat transactions timed out; 0/24 declared target checks completed. Cancellation/reporting remained below 2 seconds. Priming did not qualify graph reuse; named census and engine lifecycle work routed to contributor-scan." "8.3"
],
"feasibility": "NO-GO: all 21 frozen NodeGoat transactions timed out; 0/24 declared target checks completed. Cancellation/reporting remained below 2 seconds. Priming did not qualify graph reuse; named census and engine lifecycle work routed to contributor-scan.",
"qualification": "NO-GO: eligibility implementation and evidence delivered; required independent populations and verified VAmPI transitive witness remain missing. Zero capabilities enabled."
} }
} }
+1
View File
@@ -234,6 +234,7 @@ independent capability qualification. Groups 7–8 remain pending; rollout remai
- _Boundary: Evaluation harness, Delta and Actionability Policy integration_ - _Boundary: Evaluation harness, Delta and Actionability Policy integration_
- _Depends: 1.2, 1.3, 5.2, 7.3, 8.2_ - _Depends: 1.2, 1.3, 5.2, 7.3, 8.2_
- _Requirements: 3.1, 3.2, 6.1, 6.2, 6.3, 6.4, 6.5, 8.1, 8.2, 8.3, 8.4_ - _Requirements: 3.1, 3.2, 6.1, 6.2, 6.3, 6.4, 6.5, 8.1, 8.2, 8.3, 8.4_
- _Blocked 2026-09-13:_ Eligibility implementation and reproducible NO-GO artifact delivered; independent reviewed Node/PHP/Python populations and verified VAmPI transitive witness remain missing. Runtime/coverage gates fail. See `implementation-group-8.md` and contributor-scan owner follow-up.
- [ ] 8.4 Verify the packaged end-to-end outcome and acceptance coverage - [ ] 8.4 Verify the packaged end-to-end outcome and acceptance coverage
- Run the required regression checks and a local packaged smoke over supported security/fixed/benign pushes; verify no-model startup, preservation of hook behavior and truthful partial results. - Run the required regression checks and a local packaged smoke over supported security/fixed/benign pushes; verify no-model startup, preservation of hook behavior and truthful partial results.
- Reconcile every acceptance criterion with fresh evidence and record unresolved capability/performance blockers separately from completed plumbing. - Reconcile every acceptance criterion with fresh evidence and record unresolved capability/performance blockers separately from completed plumbing.
@@ -1,10 +1,10 @@
{ {
"counts": { "counts": {
"load_bearing": 105, "load_bearing": 106,
"standalone_capability": 9 "standalone_capability": 9
}, },
"modules": 114, "modules": 115,
"note": "Refreshed for group 7: push runner reaches optional bounded witness selection; registry remains empty.", "note": "Group 8: runner reaches recomputed versioned eligibility and its frozen scorer.",
"orphaned": [], "orphaned": [],
"rows": [ "rows": [
{ {
@@ -140,6 +140,7 @@
"model.scan", "model.scan",
"model.taint", "model.taint",
"push.cache", "push.cache",
"push.eligibility",
"push.policy", "push.policy",
"push.reuse", "push.reuse",
"push.runner" "push.runner"
@@ -152,6 +153,7 @@
"importers": [ "importers": [
"cpg.backend", "cpg.backend",
"push.cache", "push.cache",
"push.eligibility",
"push.policy", "push.policy",
"push.reuse", "push.reuse",
"push.runner" "push.runner"
@@ -704,6 +706,7 @@
"importers": [ "importers": [
"cli", "cli",
"push.assistance", "push.assistance",
"push.eligibility",
"push.policy", "push.policy",
"push.report", "push.report",
"push.reuse", "push.reuse",
@@ -746,6 +749,15 @@
{ {
"classification": "load_bearing", "classification": "load_bearing",
"importers": [ "importers": [
"push.runner"
],
"module": "push.eligibility",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"push.eligibility",
"push.report", "push.report",
"push.runner" "push.runner"
], ],
@@ -794,7 +806,9 @@
}, },
{ {
"classification": "standalone_capability", "classification": "standalone_capability",
"importers": [], "importers": [
"push.eligibility"
],
"module": "push_scoring", "module": "push_scoring",
"reason": "pre-push-safety-net task 1.3: frozen-profile diagnostic outcome scorer; python -m openultrasast.push_scoring" "reason": "pre-push-safety-net task 1.3: frozen-profile diagnostic outcome scorer; python -m openultrasast.push_scoring"
}, },
File diff suppressed because it is too large Load Diff
+24
View File
@@ -188,3 +188,27 @@ reports nearest-rank p50/p95 with sample counts, and never treats no answers as
Exit 1 with a complete scorecard is measured NO-GO; a missing scorecard is an instrument Exit 1 with a complete scorecard is measured NO-GO; a missing scorecard is an instrument
failure. Three samples per workload on development NodeGoat cannot independently qualify failure. Three samples per workload on development NodeGoat cannot independently qualify
a production language/framework/family or establish broad multi-project performance. a production language/framework/family or establish broad multi-project performance.
### Capability qualification (task 8.3)
The same `measure.py` replay instrument accepts `--selection development` (the
six PMPro/NodeGoat changes), `--selection regression` (three VAmPI targets), or
`--selection envelope` (libpng). Each profile retains the full declared population.
Readable regression snapshots may execute while their missing independent evaluation
prerequisites remain unresolved. `--deadline 600 --cache-dir /results/lab-cache`
is a separately frozen laboratory experiment, never a replacement for the 30-second
hook profile. A ranked or directly detected VAmPI target does not prove a transitive path.
`qualify_registry.py --evidence DIR --runtime SCORECARD --out NEW_FILE` recomputes
eligibility from the frozen profile, validated inputs, actual run and runtime scorecard.
Without `--declarations` it creates explicitly unreviewed, experimental declarations.
A reviewed declaration must match its frozen capability key, operation symbols, advice,
review identity, untouched workload and exact installed analysis semantics; all quality,
completion and runtime gates must pass. Synthetic positive controls in the unit suite
verify this contract and do not qualify real capabilities.
The installed `openultrasast/push/eligibility.json` is a trusted release input, not
repository configuration. The runner recomputes its decision and rejects stale or
modified evidence. This integrity check is not a cryptographic reviewer signature.
The versioned NO-GO registry enables no capability. Regenerate evidence after changes
to core, engine, facts, queries, policy or semantics; do not relabel old measurements.
+31 -9
View File
@@ -163,31 +163,44 @@ def main() -> int:
parser.add_argument("--manifest", type=Path, required=True) parser.add_argument("--manifest", type=Path, required=True)
parser.add_argument("--cache", type=Path, required=True) parser.add_argument("--cache", type=Path, required=True)
parser.add_argument("--out", type=Path, required=True) parser.add_argument("--out", type=Path, required=True)
parser.add_argument("--selection", choices=("development", "regression", "envelope"), default="development")
parser.add_argument("--deadline", type=float, default=30)
parser.add_argument("--cache-dir", type=Path)
args = parser.parse_args() args = parser.parse_args()
if args.deadline <= 0:
parser.error("deadline must be positive")
args.out.mkdir(parents=True, exist_ok=False) args.out.mkdir(parents=True, exist_ok=False)
manifest = json.loads(args.manifest.read_bytes()) manifest = json.loads(args.manifest.read_bytes())
validation = validate_manifest(args.manifest, cache_root=args.cache, fetch=False) validation = validate_manifest(args.manifest, cache_root=args.cache, fetch=False)
write(args.out / "input-validation.json", validation) write(args.out / "input-validation.json", validation)
settings = ScanBudget(max_model_calls=0, max_regions=500, order_by_evidence=True) settings = ScanBudget(max_model_calls=0, max_regions=500, order_by_evidence=True)
initial = _provenance(args.manifest.parent, settings) initial = _provenance(args.manifest.parent, settings)
keys = ("ranking_mode", "engine", "facts", "queries", "policy", "core") keys = ("ranking_mode", "engine", "engine_runtime", "facts", "queries", "policy", "core", "semantics")
config = {key: initial[key] for key in keys} config = {key: initial[key] for key in keys}
config.update( config.update(
version="first-replay-v2-2026-09-13", version="qualification-replay-v1-" + args.selection,
scope="full tracked tree; declared first-party graph/target exclusions; evidence ranker", scope="full tracked tree; declared first-party graph/target exclusions; evidence ranker",
hardware=platform.platform(), hardware=platform.platform(),
budget_seconds=30, budget_seconds=args.deadline,
targets=json.loads((args.manifest.parent / "diagnostic-config-v1.json").read_text())["targets"], targets=json.loads((args.manifest.parent / "diagnostic-config-v1.json").read_text())["targets"],
) )
profile = freeze_profile(args.manifest.read_bytes(), config) profile = freeze_profile(args.manifest.read_bytes(), config)
write(args.out / "profile.json", profile) # Freeze on disk BEFORE any measurement. write(args.out / "profile.json", profile) # Freeze on disk BEFORE any measurement.
run = {"profile_sha256": profile["profile_sha256"], "context": dict(config), "records": []} run = {"profile_sha256": profile["profile_sha256"], "context": dict(config), "records": []}
evidence = {"cases": [], "core_changes": [], "adapter_changes": [], "freeze_after_pmpro": None} evidence = {"cases": [], "core_changes": [], "adapter_changes": [], "freeze_after_pmpro": None}
ready = {c["id"]: c["status"] == "ready" for c in validation["cases"]} valid_snapshots = {s["id"] for s in validation["snapshots"] if s["status"] == "ready"}
ready = {
c["id"]: c["status"] == "ready" or (c["label"] == "regression" and c["base"] in valid_snapshots and c["tip"] in valid_snapshots)
for c in validation["cases"]
}
snapshots = {s["id"]: s for s in manifest["snapshots"]} snapshots = {s["id"]: s for s in manifest["snapshots"]}
# Fixed declared order; reserved, regression and unsupported populations stay # Fixed declared order; reserved, regression and unsupported populations stay
# in the joint scorer without scanning or tuning on the holdout. # in the joint scorer without scanning or tuning on the holdout.
cases = [c for prefix in ("pmpro-", "nodegoat-") for c in manifest["cases"] if c["id"].startswith(prefix)] cases = [c for prefix in ("pmpro-", "nodegoat-") for c in manifest["cases"] if c["id"].startswith(prefix)]
if args.selection == "regression":
cases = [c for c in manifest["cases"] if c["label"] == "regression"]
elif args.selection == "envelope":
cases = [c for c in manifest["cases"] if c["label"] == "unsupported"]
with tempfile.TemporaryDirectory(prefix="ousast-experiment-") as temporary: with tempfile.TemporaryDirectory(prefix="ousast-experiment-") as temporary:
bare = Path(temporary) / "objects.git" bare = Path(temporary) / "objects.git"
bare.mkdir() bare.mkdir()
@@ -209,6 +222,7 @@ def main() -> int:
args.out / "replay-pins.json", args.out / "replay-pins.json",
{"snapshots": pins, "provenance": "Full upstream trees; declared authored edits in a private bare Git repository."}, {"snapshots": pins, "provenance": "Full upstream trees; declared authored edits in a private bare Git repository."},
) )
seen_comparisons = set()
for case in cases: for case in cases:
if not ready[case["id"]]: if not ready[case["id"]]:
evidence["cases"].append({"case_id": case["id"], "status": "missing_input_prerequisite"}) evidence["cases"].append({"case_id": case["id"], "status": "missing_input_prerequisite"})
@@ -231,14 +245,16 @@ def main() -> int:
"--artifact", "--artifact",
str(artifact_path), str(artifact_path),
"--deadline", "--deadline",
"30", str(args.deadline),
"--cancellation-allowance", "--cancellation-allowance",
"2", "2",
"--max-regions", "--max-regions",
"500", "500",
] ]
if args.cache_dir is not None:
command += ["--cache-dir", str(args.cache_dir)]
start = time.monotonic() start = time.monotonic()
process = subprocess.run(command, capture_output=True, text=True, timeout=62) process = subprocess.run(command, capture_output=True, text=True, timeout=args.deadline + 5)
elapsed = time.monotonic() - start elapsed = time.monotonic() - start
(args.out / (case["id"] + ".stdout")).write_text(process.stdout) (args.out / (case["id"] + ".stdout")).write_text(process.stdout)
(args.out / (case["id"] + ".stderr")).write_text(process.stderr) (args.out / (case["id"] + ".stderr")).write_text(process.stderr)
@@ -257,7 +273,11 @@ def main() -> int:
# Unmeasured provenance caused by preparation timeout is explicit; # Unmeasured provenance caused by preparation timeout is explicit;
# never replace it with a fictitious runtime measurement. # never replace it with a fictitious runtime measurement.
entry["runtime_provenance_available"] = all(artifact["provenance"].get(key) == initial[key] for key in keys) entry["runtime_provenance_available"] = all(artifact["provenance"].get(key) == initial[key] for key in keys)
record = execution_record(case["id"], artifact, elapsed=elapsed, target=config["targets"][case["id"]]) record = execution_record(case["id"], artifact, elapsed=elapsed, target=config["targets"].get(case["id"]))
comparison = (pins[case["base"]], pins[case["tip"]])
if args.cache_dir is not None and comparison in seen_comparisons and artifact["timings"].get("graph_hits", 0) >= 2:
record["cache_state"] = "identical_tip"
seen_comparisons.add(comparison)
entry["coverage_reasons"] = artifact["admission"]["coverage_reasons"] entry["coverage_reasons"] = artifact["admission"]["coverage_reasons"]
entry["stage_timings"] = artifact["timings"] entry["stage_timings"] = artifact["timings"]
entry["snapshot_bytes_read"] = sum(s["bytes_read"] for s in artifact["snapshots"]) entry["snapshot_bytes_read"] = sum(s["bytes_read"] for s in artifact["snapshots"])
@@ -291,13 +311,15 @@ def main() -> int:
"decision": "NO-GO", "decision": "NO-GO",
"capabilities_enabled": False, "capabilities_enabled": False,
"scorecard": "scorecard.json", "scorecard": "scorecard.json",
"reason": "First cold diagnostic experiment; independent eligibility and joint quality/warm-latency gates remain unmet.", "reason": "Qualification evidence run; independent eligibility and joint quality/warm-latency gates remain unmet.",
"selection": args.selection,
"analysis_deadline_seconds": args.deadline,
"recorded_cases": len(run["records"]), "recorded_cases": len(run["records"]),
"declared_population": len(manifest["cases"]), "declared_population": len(manifest["cases"]),
"core_unchanged": True, "core_unchanged": True,
"adapter_changes": [], "adapter_changes": [],
"limits": [ "limits": [
"No warm-cache measurements", "No representative warm changed-code measurements; identical-comparison reuse is reported separately",
"Untouched Ghost population not scanned", "Untouched Ghost population not scanned",
"PHP/Python admission populations missing", "PHP/Python admission populations missing",
"libpng retained as unsupported envelope; no C/C++ detection claim", "libpng retained as unsupported envelope; no C/C++ detection claim",
+73
View File
@@ -0,0 +1,73 @@
"""Recompute versioned eligibility from frozen results; default declarations are experimental."""
from __future__ import annotations
import argparse
import json
from pathlib import Path
from openultrasast.model.scan import ScanBudget
from openultrasast.push.eligibility import create_decision
from openultrasast.push.policy import CapabilityAdmission, CapabilityKey
from openultrasast.push.runner import _provenance
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--evidence", type=Path, required=True)
parser.add_argument("--runtime", type=Path, required=True)
parser.add_argument("--out", type=Path, required=True)
parser.add_argument("--declarations", type=Path)
args = parser.parse_args()
def read(name):
return json.loads((args.evidence / name).read_bytes())
profile, validation, run = read("profile.json"), read("input-validation.json"), read("run.json")
current = _provenance(args.evidence, ScanBudget(max_model_calls=0, max_regions=500, order_by_evidence=True))
if args.declarations:
declarations = tuple(CapabilityAdmission.from_payload(d) for d in json.loads(args.declarations.read_bytes()))
else:
declarations = []
for capability in sorted({c["capability"] for c in profile["cases"]}):
language, framework, family = capability.split("/")
key = CapabilityKey(
language,
"unspecified",
framework,
family,
"dominance" if family == "access_control" else "taint",
"unreviewed",
current["semantics"],
)
declarations.append(
CapabilityAdmission(
key,
"unqualified-population-v1",
str(args.evidence / "scorecard.json"),
"experimental",
"Unreviewed consequence for {operation} in {context}.",
"Unreviewed repair for {operation} in {context}.",
)
)
decision = create_decision(
profile, validation, run, current=current, runtime=json.loads(args.runtime.read_bytes()), declarations=declarations
)
args.out.parent.mkdir(parents=True, exist_ok=True)
with args.out.open("x") as stream:
json.dump(decision, stream, indent=2, allow_nan=False)
stream.write("\n")
print(
json.dumps(
{
"verdict": decision["verdict"],
"enabled": sum(e["declaration"]["enabled"] for e in decision["entries"]),
"artifact_sha256": decision["artifact_sha256"],
}
)
)
return 0 if decision["verdict"] == "PASS" else 1
if __name__ == "__main__":
raise SystemExit(main())
File diff suppressed because one or more lines are too long
+171
View File
@@ -0,0 +1,171 @@
"""Versioned, locally shipped capability eligibility from frozen evaluation evidence.
This verifies integrity and recomputes gates; it is not a signature verifier. Registry
files are trusted release inputs, never repository configuration or model responses.
"""
from __future__ import annotations
import json
import math
from collections.abc import Mapping, Sequence
from dataclasses import dataclass, replace
from pathlib import Path
from typing import Any
from openultrasast.cpg.artifact import digest_value
from openultrasast.push.policy import CapabilityAdmission
from openultrasast.push_scoring import _metrics, score
_BINDINGS = ("core", "engine", "engine_runtime", "facts", "queries", "policy", "semantics")
_WORKLOADS = ("identical-tip", "function-edit", "dependency-edit", "configuration-edit", "cold", "growth", "multi-ref")
_DEFAULT = Path(__file__).with_name("eligibility.json")
@dataclass(frozen=True)
class EligibilityLoad:
capabilities: tuple[CapabilityAdmission, ...]
status: str
reasons: tuple[str, ...]
def fingerprint() -> str:
try:
return digest_value(json.loads(_DEFAULT.read_bytes()))
except (OSError, ValueError):
return "eligibility_unavailable"
def _runtime_pass(runtime: Mapping[str, Any], current: Mapping[str, str]) -> bool:
if runtime.get("runtime_verdict") != "PASS" or any(runtime.get("provenance", {}).get(k) != current.get(k) for k in _BINDINGS):
return False
try:
for name in _WORKLOADS:
row = runtime["workloads"][name]
values = [
row[k]
for k in (
"population",
"observed",
"missing",
"latency_samples",
"p95_seconds",
"completion",
"complete_coverage_cases",
"cancellation_overrun_seconds",
"timeouts",
)
]
if any(type(v) not in (int, float) or not math.isfinite(v) or v < 0 for v in values):
return False
population = row["population"]
if (
population < 1
or row["observed"] != population
or row["latency_samples"] != population
or row["missing"]
or row["timeouts"]
or row["completion"] < 0.95
or row["complete_coverage_cases"] / population < 0.95
or row["cancellation_overrun_seconds"] > 2
or (name != "cold" and row["p95_seconds"] > 30)
):
return False
except (KeyError, TypeError, ZeroDivisionError):
return False
return True
def create_decision(
profile: dict[str, Any],
validation: dict[str, Any],
run: dict[str, Any],
*,
current: Mapping[str, str],
runtime: dict[str, Any],
declarations: Sequence[CapabilityAdmission],
) -> dict[str, Any]:
scored = score(profile, validation, run)
bindings = {k: current.get(k, "") for k in _BINDINGS}
common = []
if any(not bindings[k] or profile["config"].get(k) != bindings[k] for k in _BINDINGS):
common.append("evaluation_provenance_mismatch")
if not _runtime_pass(runtime, current):
common.append("runtime_gate_or_provenance_failed")
entries = []
for declaration in declarations:
key = declaration.key
capability = f"{key.language}/{key.framework}/{key.family}"
rows = [r for r in scored["cases"] if r["capability"] == capability and r["label"] != "regression"]
metrics = _metrics(rows)
reasons = list(common)
labels = {r["label"] for r in rows}
if not {"vulnerable", "fixed", "benign"} <= labels:
reasons.append("paired_positive_fixed_benign_population_missing")
if any(r["label"] in ("unsupported", "unreviewed") or r["input_status"] != "ready" or r.get("prerequisites") for r in rows):
reasons.append("unresolved_input_or_unsupported_population")
if not declaration.untouched_workload or any(
r.get("role") not in ("independent", "reserved") or r.get("workload") != declaration.untouched_workload for r in rows
):
reasons.append("independent_workload_missing")
semantic = profile["config"].get("capability_semantics", {}).get(digest_value(key.to_payload()), {})
if (
semantic.get("key") != key.to_payload()
or semantic.get("operation_symbols") != list(declaration.operation_symbols)
or semantic.get("review_id") != declaration.evaluation_id
or semantic.get("consequence_template") != declaration.consequence_template
or semantic.get("repair_template") != declaration.repair_template
or not declaration.operation_symbols
or key.semantics != bindings["semantics"]
or key.context in ("unreviewed", "unspecified", "")
or not declaration.calibration_artifact
):
reasons.append("semantic_scope_unreviewed_or_mismatched")
for name, threshold in (("precision", 0.95), ("actionable_recall", 0.90), ("completion", 0.95), ("query_completion", 0.95)):
if metrics[name]["value"] is None or metrics[name]["value"] < threshold:
reasons.append(name + "_gate_failed")
if metrics["fixed_side_silence"]["value"] != 1 or metrics["benign_interruption"]["value"] != 0:
reasons.append("fixed_or_benign_gate_failed")
warm = metrics["latency"]["warm_changed"]
if not warm["samples"] or warm["p95"] > 30:
reasons.append("independent_warm_latency_missing_or_failed")
positive = metrics["recall"]["numerator"]
reviewed = metrics["precision"]["numerator"]
if positive < 1 or reviewed < 1:
reasons.append("positive_controls_or_reviewed_alerts_missing")
accepted = not reasons
resolved = replace(
declaration, verdict="PASS" if accepted else "NO-GO", enabled=accepted, positive_controls=positive, reviewed_alerts=reviewed
)
entries.append(dict(declaration=resolved.to_payload(), metrics=metrics, reasons=sorted(set(reasons))))
result = dict(
schema_version=1,
bindings=bindings,
verdict="PASS" if any(not e["reasons"] for e in entries) else "NO-GO",
entries=entries,
inputs=dict(profile=profile, validation=validation, run=run, runtime=runtime, declarations=[d.to_payload() for d in declarations]),
)
result["artifact_sha256"] = digest_value(result)
return result
def load_registry(*, current: Mapping[str, str], path: Path | None = None) -> EligibilityLoad:
try:
with (path or _DEFAULT).open("rb") as stream:
content = stream.read(2 * 1024**2 + 1)
if len(content) > 2 * 1024**2:
raise ValueError("registry_size_limit")
data = json.loads(content)
if data.get("schema_version") != 1 or data.get("bindings") != {k: current.get(k, "") for k in _BINDINGS}:
return EligibilityLoad((), "stale", ("eligibility_provenance_mismatch",))
inputs = data["inputs"]
declarations = tuple(CapabilityAdmission.from_payload(d) for d in inputs["declarations"])
recomputed = create_decision(
inputs["profile"], inputs["validation"], inputs["run"], current=current, runtime=inputs["runtime"], declarations=declarations
)
if recomputed != data:
raise ValueError("registry_evidence_mismatch")
capabilities = tuple(CapabilityAdmission.from_payload(e["declaration"]) for e in data["entries"] if not e["reasons"])
return EligibilityLoad(capabilities, data["verdict"], tuple(sorted({reason for e in data["entries"] for reason in e["reasons"]})))
except (OSError, ValueError, TypeError, KeyError, AttributeError):
return EligibilityLoad((), "unavailable", ("eligibility_invalid_or_unavailable",))
+14 -2
View File
@@ -26,6 +26,7 @@ from openultrasast.model.shipped import declared_sources
from openultrasast.preprocess import build_file_target, enumerate_source_files from openultrasast.preprocess import build_file_target, enumerate_source_files
from openultrasast.push.cache import ArtifactCache, SemanticKeys from openultrasast.push.cache import ArtifactCache, SemanticKeys
from openultrasast.push.contracts import ComparisonAnalysis, PushComparison, PushResult, SnapshotManifest from openultrasast.push.contracts import ComparisonAnalysis, PushComparison, PushResult, SnapshotManifest
from openultrasast.push.eligibility import fingerprint, load_registry
from openultrasast.push.policy import ( from openultrasast.push.policy import (
ActionableDefect, ActionableDefect,
AdmissionCandidate, AdmissionCandidate,
@@ -151,11 +152,22 @@ def _provenance(repository: Path, scan_budget: ScanBudget) -> dict[str, str]:
), ),
"ranking_mode": "evidence", "ranking_mode": "evidence",
"model": "disabled", "model": "disabled",
"capabilities": "empty_registry_pending_task_8.3", "capabilities": fingerprint(),
"cache": "cold_no_reuse", "cache": "cold_no_reuse",
} }
def _admit(candidates: tuple[AdmissionCandidate, ...], provenance: dict[str, str]) -> AdmissionResult:
registry = load_registry(current=provenance)
result = admit_candidates(candidates, capabilities=registry.capabilities)
if candidates and registry.reasons:
result = replace(
result,
coverage_reasons=tuple(dict.fromkeys((*result.coverage_reasons, *("eligibility:" + reason for reason in registry.reasons)))),
)
return result
def _discovery_identity() -> str: def _discovery_identity() -> str:
root = Path(__file__).resolve().parents[1] root = Path(__file__).resolve().parents[1]
# Discovery invokes the shipped mapper, preprocessing, layout and family specs. # Discovery invokes the shipped mapper, preprocessing, layout and family specs.
@@ -326,7 +338,7 @@ def _analyze(
) )
for c in delta.candidates for c in delta.candidates
) )
admission = _prepare(lambda: admit_candidates(candidates), budget) admission = _prepare(lambda: _admit(candidates, provenance), budget)
timings[stage + "_seconds"] = time.monotonic() - stage_started timings[stage + "_seconds"] = time.monotonic() - stage_started
stage, stage_started = "cleanup", time.monotonic() stage, stage_started = "cleanup", time.monotonic()
timings[stage + "_seconds"] = time.monotonic() - stage_started timings[stage + "_seconds"] = time.monotonic() - stage_started
+163
View File
@@ -0,0 +1,163 @@
"""Synthetic eligibility contracts, not real capability qualification."""
import copy
import json
from dataclasses import replace
import pytest
from test_push_admission import control
from test_push_scoring import inputs, record
from openultrasast.push.eligibility import create_decision, load_registry
from openultrasast.push_scoring import freeze_profile
def fixture():
old, validation = inputs()
current = {key: "synthetic-" + key for key in ("core", "engine", "engine_runtime", "facts", "queries", "policy", "semantics")}
from openultrasast.cpg.artifact import digest_value
_, declaration, _ = control()
declaration = replace(declaration, key=replace(declaration.key, semantics=current["semantics"]), untouched_workload="untouched-control")
config = {
**old["config"],
**current,
"capability_semantics": {
digest_value(declaration.key.to_payload()): {
"key": declaration.key.to_payload(),
"operation_symbols": list(declaration.operation_symbols),
"review_id": declaration.evaluation_id,
"consequence_template": declaration.consequence_template,
"repair_template": declaration.repair_template,
}
},
}
cases = [{**row, "role": "independent", "workload": "untouched-control"} for row in old["cases"][:3]]
raw = json.dumps({"schema_version": 1, "cases": cases}).encode()
profile = freeze_profile(raw, config)
validation = {**validation, "manifest_sha256": profile["manifest_sha256"], "cases": [{**c, "status": "ready"} for c in cases]}
positive = record(
alerts=[dict(review="actionable_true", review_evidence="synthetic-review", question_id="q-handler", site="api.js:4")],
cache_state="warm_changed",
)
fixed = record(
"fixed",
queries=[
dict(
id="q-handler",
status="answered",
outcome="negative",
target=config["targets"]["fixed"],
witness="bound parameter discharges target",
)
],
delta="removed",
cache_state="warm_changed",
)
benign = record("benign", delta="unchanged", cache_state="warm_changed")
run = {"profile_sha256": profile["profile_sha256"], "context": config, "records": [positive, fixed, benign]}
_, declaration, _ = control()
declaration = replace(declaration, key=replace(declaration.key, semantics=current["semantics"]), untouched_workload="untouched-control")
runtime = {
"runtime_verdict": "PASS",
"provenance": current,
"workloads": {
name: dict(
population=3,
observed=3,
missing=0,
latency_samples=3,
p95_seconds=3,
completion=1,
complete_coverage_cases=3,
cancellation_overrun_seconds=0,
timeouts=0,
)
for name in ("identical-tip", "function-edit", "dependency-edit", "configuration-edit", "cold", "growth", "multi-ref")
},
}
return profile, validation, run, current, runtime, (declaration,)
def decision(parts):
profile, validation, run, current, runtime, declarations = parts
return create_decision(profile, validation, run, current=current, runtime=runtime, declarations=declarations)
def test_matching_reviewed_synthetic_population_can_be_loaded(tmp_path):
parts = fixture()
data = decision(parts)
assert data["verdict"] == "PASS"
path = tmp_path / "eligibility.json"
path.write_text(json.dumps(data))
loaded = load_registry(current=parts[3], path=path)
assert len(loaded.capabilities) == 1
assert loaded.capabilities[0].enabled
assert loaded.capabilities[0].positive_controls == 1
assert loaded.capabilities[0].reviewed_alerts == 1
@pytest.mark.parametrize("mutation", ["no_alert", "runtime", "stale_runtime", "no_independent", "no_positive", "wrong_semantics"])
def test_missing_joint_evidence_never_enables_advisory(mutation, tmp_path):
parts = list(copy.deepcopy(fixture()))
if mutation == "no_alert":
parts[2]["records"][0]["alerts"] = []
elif mutation == "runtime":
parts[4]["workloads"]["function-edit"]["p95_seconds"] = 31
elif mutation == "stale_runtime":
parts[4]["provenance"]["policy"] = "other"
elif mutation == "no_independent":
parts[5] = (replace(parts[5][0], untouched_workload="wrong-repository"),)
elif mutation == "no_positive":
parts[2]["records"] = parts[2]["records"][1:]
else:
parts[5] = (replace(parts[5][0], key=replace(parts[5][0].key, semantics="wrong")),)
data = decision(parts)
assert data["verdict"] == "NO-GO"
path = tmp_path / "eligibility.json"
path.write_text(json.dumps(data))
assert not load_registry(current=parts[3], path=path).capabilities
def test_stale_or_tampered_registry_is_rejected(tmp_path):
parts = fixture()
data = decision(parts)
path = tmp_path / "eligibility.json"
path.write_text(json.dumps(data))
stale = {**parts[3], "facts": "new-facts"}
assert not load_registry(current=stale, path=path).capabilities
data["entries"][0]["declaration"]["repair_template"] = "invented advice"
path.write_text(json.dumps(data))
assert not load_registry(current=parts[3], path=path).capabilities
def test_runner_consumes_only_matching_registry(tmp_path, monkeypatch):
import openultrasast.push.eligibility as registry
from openultrasast.push.runner import _admit
parts = fixture()
data = decision(parts)
path = tmp_path / "eligibility.json"
path.write_text(json.dumps(data))
monkeypatch.setattr(registry, "_DEFAULT", path)
candidate, _, _ = control()
candidate = replace(candidate, capability=parts[5][0].key, delta=replace(candidate.delta, semantics=parts[3]["semantics"]))
assert len(_admit((candidate,), parts[3]).defects) == 1
assert not _admit((candidate,), {**parts[3], "facts": "changed"}).defects
def test_changed_semantic_advice_needs_a_new_frozen_review():
parts = list(fixture())
parts[5] = (replace(parts[5][0], repair_template="Unreviewed advice at {operation} for {context}"),)
assert decision(parts)["verdict"] == "NO-GO"
def test_shipped_registry_is_reproducible_no_go():
from openultrasast.push.eligibility import _DEFAULT
data = json.loads(_DEFAULT.read_bytes())
loaded = load_registry(current=data["bindings"])
assert loaded.status == "NO-GO"
assert not loaded.capabilities
assert data["entries"]
assert all(not row["declaration"]["enabled"] for row in data["entries"])