mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
feat(pre-push-safety-net): enforce evidence eligibility and record blocked qualification
This commit is contained in:
@@ -41,3 +41,26 @@ and its compressed full evidence bundle. All 21 deadline-bound transactions time
|
||||
Every measured artifact matches the frozen installed semantics. No source, engine,
|
||||
query, fact or ranking change was made during this run. The follow-up remains necessary
|
||||
before rollout; the evaluation task does not authorize weakening its gates.
|
||||
|
||||
## VAmPI qualification follow-up, 2026-09-13
|
||||
|
||||
Task 8.3's packaged frozen regression (139,164 total snapshot bytes read per replay)
|
||||
retains raw findings at `models/user_model.py:73:get_user`,
|
||||
`api_views/users.py:187:update_password`, and `api_views/books.py:51:get_by_title`.
|
||||
The SQLI target has no selected exact function question; the two authorization target
|
||||
questions are selected at positions 93 and 63 but have `change_context_incomplete`.
|
||||
Head scope retains 138 selected and 20 deferred questions. Raw findings do not prove
|
||||
completed target questions or a transitive source-to-target witness.
|
||||
|
||||
The first lab run took 112.415s; same-snapshot cached replays took 6.816s and 6.684s,
|
||||
all incomplete. This is diagnostic reuse, not changed-code performance qualification.
|
||||
The 600-second budget is distinct from the hook acceptance profile. Full artifacts are
|
||||
in `2026-09-13-push-qualification.json.gz` under the regression population.
|
||||
|
||||
Reconcile evidence normalization and the unsupported context projection at the shared
|
||||
scan boundary before downstream admission: retain the actual query-origin/path witness
|
||||
when available, resolve supported first-party context, and keep unresolved dynamic/vendor
|
||||
boundaries explicit. Do not fabricate transitivity from a raw finding at the target.
|
||||
Revalidate ordinary VAmPI scan, exact question accounting, push regressions and independent
|
||||
qualification after an approved owner fix. This note records the prerequisite; it does
|
||||
not silently approve an upstream implementation or change detector semantics.
|
||||
|
||||
@@ -106,3 +106,98 @@ retune the ranker or weaken the population/gates to compensate.
|
||||
- CLAIM: Task 8.2 completed its prescribed measurement and owner routing; rollout remains NO-GO.
|
||||
- EVIDENCE: Versioned scorecard and 116-file verified bundle; seven scorer controls and full regression suite pass.
|
||||
- GAPS: No qualified warm graph path or independent capability eligibility established. These failed gates cannot authorize rollout.
|
||||
|
||||
## Task 8.3 brief
|
||||
|
||||
Requirements 3.1, 3.2, 6.1–6.5 and 8.1–8.4; evaluation and actionability integration.
|
||||
Implement a trusted installed eligibility resource whose evidence is recomputed against
|
||||
exact core/engine/runtime/facts/query/policy/semantic identities. Require reviewed
|
||||
operation/advice scope, vulnerable/fixed/benign populations, nonzero actual positive
|
||||
controls and reviewed alerts, untouched per-capability evaluation and all joint gates.
|
||||
The normal runner consumes only eligible declarations; stale or inconsistent resources
|
||||
admit nothing. This is integrity checking of trusted release evidence, not signing or
|
||||
permission to load repository-authored eligibility.
|
||||
|
||||
Freeze the current packaged candidate before the six PMPro/Node development replays.
|
||||
Run readable VAmPI regression snapshots under a separate 600-second laboratory profile;
|
||||
keep their missing independent pairs unresolved. Execute libpng as an unsupported
|
||||
30-second envelope control. Do not inspect or tune on Ghost before its reviewed
|
||||
population exists. Missing required evidence leaves task 8.3 qualification blocked,
|
||||
even when the reproducible NO-GO artifact and admission implementation are delivered.
|
||||
|
||||
### Review scope
|
||||
|
||||
Synthetic controls prove that a matching reviewed population can reach the real runner
|
||||
admission seam, while stale facts/runtime, missing positive observations, absent reviews,
|
||||
wrong untouched workloads and changed consequence/repair semantics cannot. They are
|
||||
contract controls, not real capability qualification. The installed NO-GO resource must
|
||||
recompute identically and enable zero capabilities. The source has no repository- or
|
||||
language-specific rank changes, frontend adaptations or new abstract domains.
|
||||
|
||||
### Task 8.3 measured outcome
|
||||
|
||||
Packaged candidate `9464e229ccfdc873a02e3a0db7c1e83a6d6ec6d3e8234ae7b97677b4aed836fd`
|
||||
executed offline as UID 1000 with a 3 GiB memory limit. Each profile was written before
|
||||
its runs. All six development pushes saved artifacts, timed out in 30.559–30.678s and
|
||||
retained all 11 declared cases. Source snapshot receipts read 57.9–58.1 MB per PMPro
|
||||
comparison and 5.95 MB per Node comparison. These are snapshot bytes, not graph census.
|
||||
The supported introducing-change recall is 0/2, Wilson 95% interval [0, 0.658]; actual
|
||||
reviewed alert precision is undefined (0/0), never 100%. Neither fixed case has a
|
||||
completed witnessed negative. The two benign pushes emitted no alerts; that silence
|
||||
with incomplete coverage does not establish a useful safety net.
|
||||
|
||||
VAmPI's 600-second laboratory profile executed all three pinned regression targets,
|
||||
reading 139,164 snapshot bytes per replay. First run 112.415s; subsequent identical
|
||||
comparisons 6.816s and 6.684s, each with two graph hits and 403 query hits. Raw findings
|
||||
retain the known SQLI and two authorization sites, but the SQLI exact function question
|
||||
is unlocated and authorization questions are `change_context_incomplete`. All three
|
||||
remain unresolved; no transitive witness or completed regression detection is credited.
|
||||
There are 138 selected and 20 deferred head questions. Context/witness normalization
|
||||
is routed to the shared scan owner. A registry-generation provenance probe overlapped
|
||||
startup of the first lab run; these timings are diagnostic, not runtime-gate samples.
|
||||
|
||||
The initial instrument labeled all regression replays cold. The two proved identical
|
||||
cache replays were relabeled `identical_tip` and rescored without changing measurements,
|
||||
profile or population; original run/scorecard/instrument and correction are retained.
|
||||
Future instrument output distinguishes these identical comparisons. None is a warm edit.
|
||||
|
||||
The libpng unsupported envelope saved a partial artifact after 30.590s. Ghost's checkout
|
||||
and reviewed independent cases remain missing and were not scanned. Independent PHP
|
||||
and Python admission populations also remain missing. No ranker, fact, adapter or query
|
||||
retune occurred. The 59-file compressed evidence bundle is verified against originals:
|
||||
`benchmarks/measurements/2026-09-13-push-qualification.json.gz`, SHA-256
|
||||
`79226b4e54042559ba5cc2b94b69b1dcddb550872fd0783b2380368be72f90bd`.
|
||||
|
||||
The installed eligibility resource is reproducible NO-GO, digest
|
||||
`2ddd94d4dc3fd6b84e7de38e5f211d2653fb993c64e4e43cedf7e06062c3265c`, with zero enabled
|
||||
capabilities. Task 8.2 runtime evidence remains failed and predates the eligibility
|
||||
policy addition; it is correctly rejected as admission evidence, not relabeled current.
|
||||
|
||||
## Status Report
|
||||
- STATUS: BLOCKED
|
||||
- TASK: 8.3
|
||||
- RED_PHASE_OUTPUT: `/tmp/group8-3-red.log`: eight failures with eligibility disabled; ON passed, switch removed; final synthetic and installed-resource contracts verified separately.
|
||||
- IMPLEMENTED: Versioned eligibility generation/loading, production admission integration, real offline replay and NO-GO artifact.
|
||||
- BLOCKER: Required independent Node/PHP/Python reviewed populations and verified VAmPI transitive question witness are absent. Runtime/completion gates fail.
|
||||
|
||||
## Review Verdict
|
||||
- VERDICT: REJECTED
|
||||
- TASK: 8.3 completion claim
|
||||
- MECHANICAL_RESULTS: Eligibility and runner implementation inspected; fixed semantic/advice scope, observed controls and exact provenance required. No language-specific ranker changes, placeholders or secrets. RED phase verified.
|
||||
- FINDINGS: Implementation and reproducible NO-GO decision are concrete, but task 8.3's mandatory independent transfer and transitive-regression evidence is not established. Main-context fallback does not provide independent review.
|
||||
- REMEDIATION: Repair the shared context/witness and census/runtime prerequisites; obtain reviewed untouched paired populations, refreeze and rerun all joint gates. Keep capabilities disabled meanwhile.
|
||||
- SUMMARY: Retain the useful implementation, leave task 8.3 unchecked and feature rollout NO-GO.
|
||||
|
||||
## Verification Result
|
||||
- STATUS: NOT_VERIFIED
|
||||
- CLAIM_TYPE: TASK
|
||||
- CLAIM: Full task 8.3 qualification complete.
|
||||
- EVIDENCE: Real frozen replay and installed decision explicitly preserve failed gates and missing evidence.
|
||||
- GAPS: Independent populations, completed positive/fixed controls, VAmPI transitive witness and representative runtime/coverage.
|
||||
- NOTES: Proceed with authorized task 8.4's explicit NO-GO acceptance reconciliation; it does not depend on pretending qualification passed.
|
||||
|
||||
Task 8.3 implementation verification: full suite **1317 passed, 9 skipped in 115.80s**
|
||||
(`/tmp/group8-3-complete-suite.log`, exit 0). After the measurement-label correction,
|
||||
14 focused eligibility/replay-instrument tests passed in 0.95s. Ruff, format, mypy
|
||||
(116 source files) and diff checks pass. These verify the delivered code and recorded
|
||||
NO-GO behavior, not the rejected full qualification claim above.
|
||||
|
||||
@@ -132,7 +132,10 @@
|
||||
"implementation-group-8.md"
|
||||
],
|
||||
"next_group": 8,
|
||||
"blocked_tasks": [],
|
||||
"feasibility": "NO-GO: all 21 frozen NodeGoat transactions timed out; 0/24 declared target checks completed. Cancellation/reporting remained below 2 seconds. Priming did not qualify graph reuse; named census and engine lifecycle work routed to contributor-scan."
|
||||
"blocked_tasks": [
|
||||
"8.3"
|
||||
],
|
||||
"feasibility": "NO-GO: all 21 frozen NodeGoat transactions timed out; 0/24 declared target checks completed. Cancellation/reporting remained below 2 seconds. Priming did not qualify graph reuse; named census and engine lifecycle work routed to contributor-scan.",
|
||||
"qualification": "NO-GO: eligibility implementation and evidence delivered; required independent populations and verified VAmPI transitive witness remain missing. Zero capabilities enabled."
|
||||
}
|
||||
}
|
||||
|
||||
@@ -234,6 +234,7 @@ independent capability qualification. Groups 7–8 remain pending; rollout remai
|
||||
- _Boundary: Evaluation harness, Delta and Actionability Policy integration_
|
||||
- _Depends: 1.2, 1.3, 5.2, 7.3, 8.2_
|
||||
- _Requirements: 3.1, 3.2, 6.1, 6.2, 6.3, 6.4, 6.5, 8.1, 8.2, 8.3, 8.4_
|
||||
- _Blocked 2026-09-13:_ Eligibility implementation and reproducible NO-GO artifact delivered; independent reviewed Node/PHP/Python populations and verified VAmPI transitive witness remain missing. Runtime/coverage gates fail. See `implementation-group-8.md` and contributor-scan owner follow-up.
|
||||
- [ ] 8.4 Verify the packaged end-to-end outcome and acceptance coverage
|
||||
- Run the required regression checks and a local packaged smoke over supported security/fixed/benign pushes; verify no-model startup, preservation of hook behavior and truthful partial results.
|
||||
- Reconcile every acceptance criterion with fresh evidence and record unresolved capability/performance blockers separately from completed plumbing.
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
{
|
||||
"counts": {
|
||||
"load_bearing": 105,
|
||||
"load_bearing": 106,
|
||||
"standalone_capability": 9
|
||||
},
|
||||
"modules": 114,
|
||||
"note": "Refreshed for group 7: push runner reaches optional bounded witness selection; registry remains empty.",
|
||||
"modules": 115,
|
||||
"note": "Group 8: runner reaches recomputed versioned eligibility and its frozen scorer.",
|
||||
"orphaned": [],
|
||||
"rows": [
|
||||
{
|
||||
@@ -140,6 +140,7 @@
|
||||
"model.scan",
|
||||
"model.taint",
|
||||
"push.cache",
|
||||
"push.eligibility",
|
||||
"push.policy",
|
||||
"push.reuse",
|
||||
"push.runner"
|
||||
@@ -152,6 +153,7 @@
|
||||
"importers": [
|
||||
"cpg.backend",
|
||||
"push.cache",
|
||||
"push.eligibility",
|
||||
"push.policy",
|
||||
"push.reuse",
|
||||
"push.runner"
|
||||
@@ -704,6 +706,7 @@
|
||||
"importers": [
|
||||
"cli",
|
||||
"push.assistance",
|
||||
"push.eligibility",
|
||||
"push.policy",
|
||||
"push.report",
|
||||
"push.reuse",
|
||||
@@ -746,6 +749,15 @@
|
||||
{
|
||||
"classification": "load_bearing",
|
||||
"importers": [
|
||||
"push.runner"
|
||||
],
|
||||
"module": "push.eligibility",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
|
||||
},
|
||||
{
|
||||
"classification": "load_bearing",
|
||||
"importers": [
|
||||
"push.eligibility",
|
||||
"push.report",
|
||||
"push.runner"
|
||||
],
|
||||
@@ -794,7 +806,9 @@
|
||||
},
|
||||
{
|
||||
"classification": "standalone_capability",
|
||||
"importers": [],
|
||||
"importers": [
|
||||
"push.eligibility"
|
||||
],
|
||||
"module": "push_scoring",
|
||||
"reason": "pre-push-safety-net task 1.3: frozen-profile diagnostic outcome scorer; python -m openultrasast.push_scoring"
|
||||
},
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Binary file not shown.
@@ -188,3 +188,27 @@ reports nearest-rank p50/p95 with sample counts, and never treats no answers as
|
||||
Exit 1 with a complete scorecard is measured NO-GO; a missing scorecard is an instrument
|
||||
failure. Three samples per workload on development NodeGoat cannot independently qualify
|
||||
a production language/framework/family or establish broad multi-project performance.
|
||||
|
||||
### Capability qualification (task 8.3)
|
||||
|
||||
The same `measure.py` replay instrument accepts `--selection development` (the
|
||||
six PMPro/NodeGoat changes), `--selection regression` (three VAmPI targets), or
|
||||
`--selection envelope` (libpng). Each profile retains the full declared population.
|
||||
Readable regression snapshots may execute while their missing independent evaluation
|
||||
prerequisites remain unresolved. `--deadline 600 --cache-dir /results/lab-cache`
|
||||
is a separately frozen laboratory experiment, never a replacement for the 30-second
|
||||
hook profile. A ranked or directly detected VAmPI target does not prove a transitive path.
|
||||
|
||||
`qualify_registry.py --evidence DIR --runtime SCORECARD --out NEW_FILE` recomputes
|
||||
eligibility from the frozen profile, validated inputs, actual run and runtime scorecard.
|
||||
Without `--declarations` it creates explicitly unreviewed, experimental declarations.
|
||||
A reviewed declaration must match its frozen capability key, operation symbols, advice,
|
||||
review identity, untouched workload and exact installed analysis semantics; all quality,
|
||||
completion and runtime gates must pass. Synthetic positive controls in the unit suite
|
||||
verify this contract and do not qualify real capabilities.
|
||||
|
||||
The installed `openultrasast/push/eligibility.json` is a trusted release input, not
|
||||
repository configuration. The runner recomputes its decision and rejects stale or
|
||||
modified evidence. This integrity check is not a cryptographic reviewer signature.
|
||||
The versioned NO-GO registry enables no capability. Regenerate evidence after changes
|
||||
to core, engine, facts, queries, policy or semantics; do not relabel old measurements.
|
||||
|
||||
@@ -163,31 +163,44 @@ def main() -> int:
|
||||
parser.add_argument("--manifest", type=Path, required=True)
|
||||
parser.add_argument("--cache", type=Path, required=True)
|
||||
parser.add_argument("--out", type=Path, required=True)
|
||||
parser.add_argument("--selection", choices=("development", "regression", "envelope"), default="development")
|
||||
parser.add_argument("--deadline", type=float, default=30)
|
||||
parser.add_argument("--cache-dir", type=Path)
|
||||
args = parser.parse_args()
|
||||
if args.deadline <= 0:
|
||||
parser.error("deadline must be positive")
|
||||
args.out.mkdir(parents=True, exist_ok=False)
|
||||
manifest = json.loads(args.manifest.read_bytes())
|
||||
validation = validate_manifest(args.manifest, cache_root=args.cache, fetch=False)
|
||||
write(args.out / "input-validation.json", validation)
|
||||
settings = ScanBudget(max_model_calls=0, max_regions=500, order_by_evidence=True)
|
||||
initial = _provenance(args.manifest.parent, settings)
|
||||
keys = ("ranking_mode", "engine", "facts", "queries", "policy", "core")
|
||||
keys = ("ranking_mode", "engine", "engine_runtime", "facts", "queries", "policy", "core", "semantics")
|
||||
config = {key: initial[key] for key in keys}
|
||||
config.update(
|
||||
version="first-replay-v2-2026-09-13",
|
||||
version="qualification-replay-v1-" + args.selection,
|
||||
scope="full tracked tree; declared first-party graph/target exclusions; evidence ranker",
|
||||
hardware=platform.platform(),
|
||||
budget_seconds=30,
|
||||
budget_seconds=args.deadline,
|
||||
targets=json.loads((args.manifest.parent / "diagnostic-config-v1.json").read_text())["targets"],
|
||||
)
|
||||
profile = freeze_profile(args.manifest.read_bytes(), config)
|
||||
write(args.out / "profile.json", profile) # Freeze on disk BEFORE any measurement.
|
||||
run = {"profile_sha256": profile["profile_sha256"], "context": dict(config), "records": []}
|
||||
evidence = {"cases": [], "core_changes": [], "adapter_changes": [], "freeze_after_pmpro": None}
|
||||
ready = {c["id"]: c["status"] == "ready" for c in validation["cases"]}
|
||||
valid_snapshots = {s["id"] for s in validation["snapshots"] if s["status"] == "ready"}
|
||||
ready = {
|
||||
c["id"]: c["status"] == "ready" or (c["label"] == "regression" and c["base"] in valid_snapshots and c["tip"] in valid_snapshots)
|
||||
for c in validation["cases"]
|
||||
}
|
||||
snapshots = {s["id"]: s for s in manifest["snapshots"]}
|
||||
# Fixed declared order; reserved, regression and unsupported populations stay
|
||||
# in the joint scorer without scanning or tuning on the holdout.
|
||||
cases = [c for prefix in ("pmpro-", "nodegoat-") for c in manifest["cases"] if c["id"].startswith(prefix)]
|
||||
if args.selection == "regression":
|
||||
cases = [c for c in manifest["cases"] if c["label"] == "regression"]
|
||||
elif args.selection == "envelope":
|
||||
cases = [c for c in manifest["cases"] if c["label"] == "unsupported"]
|
||||
with tempfile.TemporaryDirectory(prefix="ousast-experiment-") as temporary:
|
||||
bare = Path(temporary) / "objects.git"
|
||||
bare.mkdir()
|
||||
@@ -209,6 +222,7 @@ def main() -> int:
|
||||
args.out / "replay-pins.json",
|
||||
{"snapshots": pins, "provenance": "Full upstream trees; declared authored edits in a private bare Git repository."},
|
||||
)
|
||||
seen_comparisons = set()
|
||||
for case in cases:
|
||||
if not ready[case["id"]]:
|
||||
evidence["cases"].append({"case_id": case["id"], "status": "missing_input_prerequisite"})
|
||||
@@ -231,14 +245,16 @@ def main() -> int:
|
||||
"--artifact",
|
||||
str(artifact_path),
|
||||
"--deadline",
|
||||
"30",
|
||||
str(args.deadline),
|
||||
"--cancellation-allowance",
|
||||
"2",
|
||||
"--max-regions",
|
||||
"500",
|
||||
]
|
||||
if args.cache_dir is not None:
|
||||
command += ["--cache-dir", str(args.cache_dir)]
|
||||
start = time.monotonic()
|
||||
process = subprocess.run(command, capture_output=True, text=True, timeout=62)
|
||||
process = subprocess.run(command, capture_output=True, text=True, timeout=args.deadline + 5)
|
||||
elapsed = time.monotonic() - start
|
||||
(args.out / (case["id"] + ".stdout")).write_text(process.stdout)
|
||||
(args.out / (case["id"] + ".stderr")).write_text(process.stderr)
|
||||
@@ -257,7 +273,11 @@ def main() -> int:
|
||||
# Unmeasured provenance caused by preparation timeout is explicit;
|
||||
# never replace it with a fictitious runtime measurement.
|
||||
entry["runtime_provenance_available"] = all(artifact["provenance"].get(key) == initial[key] for key in keys)
|
||||
record = execution_record(case["id"], artifact, elapsed=elapsed, target=config["targets"][case["id"]])
|
||||
record = execution_record(case["id"], artifact, elapsed=elapsed, target=config["targets"].get(case["id"]))
|
||||
comparison = (pins[case["base"]], pins[case["tip"]])
|
||||
if args.cache_dir is not None and comparison in seen_comparisons and artifact["timings"].get("graph_hits", 0) >= 2:
|
||||
record["cache_state"] = "identical_tip"
|
||||
seen_comparisons.add(comparison)
|
||||
entry["coverage_reasons"] = artifact["admission"]["coverage_reasons"]
|
||||
entry["stage_timings"] = artifact["timings"]
|
||||
entry["snapshot_bytes_read"] = sum(s["bytes_read"] for s in artifact["snapshots"])
|
||||
@@ -291,13 +311,15 @@ def main() -> int:
|
||||
"decision": "NO-GO",
|
||||
"capabilities_enabled": False,
|
||||
"scorecard": "scorecard.json",
|
||||
"reason": "First cold diagnostic experiment; independent eligibility and joint quality/warm-latency gates remain unmet.",
|
||||
"reason": "Qualification evidence run; independent eligibility and joint quality/warm-latency gates remain unmet.",
|
||||
"selection": args.selection,
|
||||
"analysis_deadline_seconds": args.deadline,
|
||||
"recorded_cases": len(run["records"]),
|
||||
"declared_population": len(manifest["cases"]),
|
||||
"core_unchanged": True,
|
||||
"adapter_changes": [],
|
||||
"limits": [
|
||||
"No warm-cache measurements",
|
||||
"No representative warm changed-code measurements; identical-comparison reuse is reported separately",
|
||||
"Untouched Ghost population not scanned",
|
||||
"PHP/Python admission populations missing",
|
||||
"libpng retained as unsupported envelope; no C/C++ detection claim",
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
"""Recompute versioned eligibility from frozen results; default declarations are experimental."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from openultrasast.model.scan import ScanBudget
|
||||
from openultrasast.push.eligibility import create_decision
|
||||
from openultrasast.push.policy import CapabilityAdmission, CapabilityKey
|
||||
from openultrasast.push.runner import _provenance
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--evidence", type=Path, required=True)
|
||||
parser.add_argument("--runtime", type=Path, required=True)
|
||||
parser.add_argument("--out", type=Path, required=True)
|
||||
parser.add_argument("--declarations", type=Path)
|
||||
args = parser.parse_args()
|
||||
|
||||
def read(name):
|
||||
return json.loads((args.evidence / name).read_bytes())
|
||||
|
||||
profile, validation, run = read("profile.json"), read("input-validation.json"), read("run.json")
|
||||
current = _provenance(args.evidence, ScanBudget(max_model_calls=0, max_regions=500, order_by_evidence=True))
|
||||
if args.declarations:
|
||||
declarations = tuple(CapabilityAdmission.from_payload(d) for d in json.loads(args.declarations.read_bytes()))
|
||||
else:
|
||||
declarations = []
|
||||
for capability in sorted({c["capability"] for c in profile["cases"]}):
|
||||
language, framework, family = capability.split("/")
|
||||
key = CapabilityKey(
|
||||
language,
|
||||
"unspecified",
|
||||
framework,
|
||||
family,
|
||||
"dominance" if family == "access_control" else "taint",
|
||||
"unreviewed",
|
||||
current["semantics"],
|
||||
)
|
||||
declarations.append(
|
||||
CapabilityAdmission(
|
||||
key,
|
||||
"unqualified-population-v1",
|
||||
str(args.evidence / "scorecard.json"),
|
||||
"experimental",
|
||||
"Unreviewed consequence for {operation} in {context}.",
|
||||
"Unreviewed repair for {operation} in {context}.",
|
||||
)
|
||||
)
|
||||
decision = create_decision(
|
||||
profile, validation, run, current=current, runtime=json.loads(args.runtime.read_bytes()), declarations=declarations
|
||||
)
|
||||
args.out.parent.mkdir(parents=True, exist_ok=True)
|
||||
with args.out.open("x") as stream:
|
||||
json.dump(decision, stream, indent=2, allow_nan=False)
|
||||
stream.write("\n")
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"verdict": decision["verdict"],
|
||||
"enabled": sum(e["declaration"]["enabled"] for e in decision["entries"]),
|
||||
"artifact_sha256": decision["artifact_sha256"],
|
||||
}
|
||||
)
|
||||
)
|
||||
return 0 if decision["verdict"] == "PASS" else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,171 @@
|
||||
"""Versioned, locally shipped capability eligibility from frozen evaluation evidence.
|
||||
|
||||
This verifies integrity and recomputes gates; it is not a signature verifier. Registry
|
||||
files are trusted release inputs, never repository configuration or model responses.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import math
|
||||
from collections.abc import Mapping, Sequence
|
||||
from dataclasses import dataclass, replace
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from openultrasast.cpg.artifact import digest_value
|
||||
from openultrasast.push.policy import CapabilityAdmission
|
||||
from openultrasast.push_scoring import _metrics, score
|
||||
|
||||
_BINDINGS = ("core", "engine", "engine_runtime", "facts", "queries", "policy", "semantics")
|
||||
_WORKLOADS = ("identical-tip", "function-edit", "dependency-edit", "configuration-edit", "cold", "growth", "multi-ref")
|
||||
_DEFAULT = Path(__file__).with_name("eligibility.json")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EligibilityLoad:
|
||||
capabilities: tuple[CapabilityAdmission, ...]
|
||||
status: str
|
||||
reasons: tuple[str, ...]
|
||||
|
||||
|
||||
def fingerprint() -> str:
|
||||
try:
|
||||
return digest_value(json.loads(_DEFAULT.read_bytes()))
|
||||
except (OSError, ValueError):
|
||||
return "eligibility_unavailable"
|
||||
|
||||
|
||||
def _runtime_pass(runtime: Mapping[str, Any], current: Mapping[str, str]) -> bool:
|
||||
if runtime.get("runtime_verdict") != "PASS" or any(runtime.get("provenance", {}).get(k) != current.get(k) for k in _BINDINGS):
|
||||
return False
|
||||
try:
|
||||
for name in _WORKLOADS:
|
||||
row = runtime["workloads"][name]
|
||||
values = [
|
||||
row[k]
|
||||
for k in (
|
||||
"population",
|
||||
"observed",
|
||||
"missing",
|
||||
"latency_samples",
|
||||
"p95_seconds",
|
||||
"completion",
|
||||
"complete_coverage_cases",
|
||||
"cancellation_overrun_seconds",
|
||||
"timeouts",
|
||||
)
|
||||
]
|
||||
if any(type(v) not in (int, float) or not math.isfinite(v) or v < 0 for v in values):
|
||||
return False
|
||||
population = row["population"]
|
||||
if (
|
||||
population < 1
|
||||
or row["observed"] != population
|
||||
or row["latency_samples"] != population
|
||||
or row["missing"]
|
||||
or row["timeouts"]
|
||||
or row["completion"] < 0.95
|
||||
or row["complete_coverage_cases"] / population < 0.95
|
||||
or row["cancellation_overrun_seconds"] > 2
|
||||
or (name != "cold" and row["p95_seconds"] > 30)
|
||||
):
|
||||
return False
|
||||
except (KeyError, TypeError, ZeroDivisionError):
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def create_decision(
|
||||
profile: dict[str, Any],
|
||||
validation: dict[str, Any],
|
||||
run: dict[str, Any],
|
||||
*,
|
||||
current: Mapping[str, str],
|
||||
runtime: dict[str, Any],
|
||||
declarations: Sequence[CapabilityAdmission],
|
||||
) -> dict[str, Any]:
|
||||
scored = score(profile, validation, run)
|
||||
bindings = {k: current.get(k, "") for k in _BINDINGS}
|
||||
common = []
|
||||
if any(not bindings[k] or profile["config"].get(k) != bindings[k] for k in _BINDINGS):
|
||||
common.append("evaluation_provenance_mismatch")
|
||||
if not _runtime_pass(runtime, current):
|
||||
common.append("runtime_gate_or_provenance_failed")
|
||||
entries = []
|
||||
for declaration in declarations:
|
||||
key = declaration.key
|
||||
capability = f"{key.language}/{key.framework}/{key.family}"
|
||||
rows = [r for r in scored["cases"] if r["capability"] == capability and r["label"] != "regression"]
|
||||
metrics = _metrics(rows)
|
||||
reasons = list(common)
|
||||
labels = {r["label"] for r in rows}
|
||||
if not {"vulnerable", "fixed", "benign"} <= labels:
|
||||
reasons.append("paired_positive_fixed_benign_population_missing")
|
||||
if any(r["label"] in ("unsupported", "unreviewed") or r["input_status"] != "ready" or r.get("prerequisites") for r in rows):
|
||||
reasons.append("unresolved_input_or_unsupported_population")
|
||||
if not declaration.untouched_workload or any(
|
||||
r.get("role") not in ("independent", "reserved") or r.get("workload") != declaration.untouched_workload for r in rows
|
||||
):
|
||||
reasons.append("independent_workload_missing")
|
||||
semantic = profile["config"].get("capability_semantics", {}).get(digest_value(key.to_payload()), {})
|
||||
if (
|
||||
semantic.get("key") != key.to_payload()
|
||||
or semantic.get("operation_symbols") != list(declaration.operation_symbols)
|
||||
or semantic.get("review_id") != declaration.evaluation_id
|
||||
or semantic.get("consequence_template") != declaration.consequence_template
|
||||
or semantic.get("repair_template") != declaration.repair_template
|
||||
or not declaration.operation_symbols
|
||||
or key.semantics != bindings["semantics"]
|
||||
or key.context in ("unreviewed", "unspecified", "")
|
||||
or not declaration.calibration_artifact
|
||||
):
|
||||
reasons.append("semantic_scope_unreviewed_or_mismatched")
|
||||
for name, threshold in (("precision", 0.95), ("actionable_recall", 0.90), ("completion", 0.95), ("query_completion", 0.95)):
|
||||
if metrics[name]["value"] is None or metrics[name]["value"] < threshold:
|
||||
reasons.append(name + "_gate_failed")
|
||||
if metrics["fixed_side_silence"]["value"] != 1 or metrics["benign_interruption"]["value"] != 0:
|
||||
reasons.append("fixed_or_benign_gate_failed")
|
||||
warm = metrics["latency"]["warm_changed"]
|
||||
if not warm["samples"] or warm["p95"] > 30:
|
||||
reasons.append("independent_warm_latency_missing_or_failed")
|
||||
positive = metrics["recall"]["numerator"]
|
||||
reviewed = metrics["precision"]["numerator"]
|
||||
if positive < 1 or reviewed < 1:
|
||||
reasons.append("positive_controls_or_reviewed_alerts_missing")
|
||||
accepted = not reasons
|
||||
resolved = replace(
|
||||
declaration, verdict="PASS" if accepted else "NO-GO", enabled=accepted, positive_controls=positive, reviewed_alerts=reviewed
|
||||
)
|
||||
entries.append(dict(declaration=resolved.to_payload(), metrics=metrics, reasons=sorted(set(reasons))))
|
||||
result = dict(
|
||||
schema_version=1,
|
||||
bindings=bindings,
|
||||
verdict="PASS" if any(not e["reasons"] for e in entries) else "NO-GO",
|
||||
entries=entries,
|
||||
inputs=dict(profile=profile, validation=validation, run=run, runtime=runtime, declarations=[d.to_payload() for d in declarations]),
|
||||
)
|
||||
result["artifact_sha256"] = digest_value(result)
|
||||
return result
|
||||
|
||||
|
||||
def load_registry(*, current: Mapping[str, str], path: Path | None = None) -> EligibilityLoad:
|
||||
try:
|
||||
with (path or _DEFAULT).open("rb") as stream:
|
||||
content = stream.read(2 * 1024**2 + 1)
|
||||
if len(content) > 2 * 1024**2:
|
||||
raise ValueError("registry_size_limit")
|
||||
data = json.loads(content)
|
||||
if data.get("schema_version") != 1 or data.get("bindings") != {k: current.get(k, "") for k in _BINDINGS}:
|
||||
return EligibilityLoad((), "stale", ("eligibility_provenance_mismatch",))
|
||||
inputs = data["inputs"]
|
||||
declarations = tuple(CapabilityAdmission.from_payload(d) for d in inputs["declarations"])
|
||||
recomputed = create_decision(
|
||||
inputs["profile"], inputs["validation"], inputs["run"], current=current, runtime=inputs["runtime"], declarations=declarations
|
||||
)
|
||||
if recomputed != data:
|
||||
raise ValueError("registry_evidence_mismatch")
|
||||
capabilities = tuple(CapabilityAdmission.from_payload(e["declaration"]) for e in data["entries"] if not e["reasons"])
|
||||
return EligibilityLoad(capabilities, data["verdict"], tuple(sorted({reason for e in data["entries"] for reason in e["reasons"]})))
|
||||
except (OSError, ValueError, TypeError, KeyError, AttributeError):
|
||||
return EligibilityLoad((), "unavailable", ("eligibility_invalid_or_unavailable",))
|
||||
@@ -26,6 +26,7 @@ from openultrasast.model.shipped import declared_sources
|
||||
from openultrasast.preprocess import build_file_target, enumerate_source_files
|
||||
from openultrasast.push.cache import ArtifactCache, SemanticKeys
|
||||
from openultrasast.push.contracts import ComparisonAnalysis, PushComparison, PushResult, SnapshotManifest
|
||||
from openultrasast.push.eligibility import fingerprint, load_registry
|
||||
from openultrasast.push.policy import (
|
||||
ActionableDefect,
|
||||
AdmissionCandidate,
|
||||
@@ -151,11 +152,22 @@ def _provenance(repository: Path, scan_budget: ScanBudget) -> dict[str, str]:
|
||||
),
|
||||
"ranking_mode": "evidence",
|
||||
"model": "disabled",
|
||||
"capabilities": "empty_registry_pending_task_8.3",
|
||||
"capabilities": fingerprint(),
|
||||
"cache": "cold_no_reuse",
|
||||
}
|
||||
|
||||
|
||||
def _admit(candidates: tuple[AdmissionCandidate, ...], provenance: dict[str, str]) -> AdmissionResult:
|
||||
registry = load_registry(current=provenance)
|
||||
result = admit_candidates(candidates, capabilities=registry.capabilities)
|
||||
if candidates and registry.reasons:
|
||||
result = replace(
|
||||
result,
|
||||
coverage_reasons=tuple(dict.fromkeys((*result.coverage_reasons, *("eligibility:" + reason for reason in registry.reasons)))),
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def _discovery_identity() -> str:
|
||||
root = Path(__file__).resolve().parents[1]
|
||||
# Discovery invokes the shipped mapper, preprocessing, layout and family specs.
|
||||
@@ -326,7 +338,7 @@ def _analyze(
|
||||
)
|
||||
for c in delta.candidates
|
||||
)
|
||||
admission = _prepare(lambda: admit_candidates(candidates), budget)
|
||||
admission = _prepare(lambda: _admit(candidates, provenance), budget)
|
||||
timings[stage + "_seconds"] = time.monotonic() - stage_started
|
||||
stage, stage_started = "cleanup", time.monotonic()
|
||||
timings[stage + "_seconds"] = time.monotonic() - stage_started
|
||||
|
||||
@@ -0,0 +1,163 @@
|
||||
"""Synthetic eligibility contracts, not real capability qualification."""
|
||||
|
||||
import copy
|
||||
import json
|
||||
from dataclasses import replace
|
||||
|
||||
import pytest
|
||||
from test_push_admission import control
|
||||
from test_push_scoring import inputs, record
|
||||
|
||||
from openultrasast.push.eligibility import create_decision, load_registry
|
||||
from openultrasast.push_scoring import freeze_profile
|
||||
|
||||
|
||||
def fixture():
|
||||
old, validation = inputs()
|
||||
current = {key: "synthetic-" + key for key in ("core", "engine", "engine_runtime", "facts", "queries", "policy", "semantics")}
|
||||
from openultrasast.cpg.artifact import digest_value
|
||||
|
||||
_, declaration, _ = control()
|
||||
declaration = replace(declaration, key=replace(declaration.key, semantics=current["semantics"]), untouched_workload="untouched-control")
|
||||
config = {
|
||||
**old["config"],
|
||||
**current,
|
||||
"capability_semantics": {
|
||||
digest_value(declaration.key.to_payload()): {
|
||||
"key": declaration.key.to_payload(),
|
||||
"operation_symbols": list(declaration.operation_symbols),
|
||||
"review_id": declaration.evaluation_id,
|
||||
"consequence_template": declaration.consequence_template,
|
||||
"repair_template": declaration.repair_template,
|
||||
}
|
||||
},
|
||||
}
|
||||
cases = [{**row, "role": "independent", "workload": "untouched-control"} for row in old["cases"][:3]]
|
||||
raw = json.dumps({"schema_version": 1, "cases": cases}).encode()
|
||||
profile = freeze_profile(raw, config)
|
||||
validation = {**validation, "manifest_sha256": profile["manifest_sha256"], "cases": [{**c, "status": "ready"} for c in cases]}
|
||||
positive = record(
|
||||
alerts=[dict(review="actionable_true", review_evidence="synthetic-review", question_id="q-handler", site="api.js:4")],
|
||||
cache_state="warm_changed",
|
||||
)
|
||||
fixed = record(
|
||||
"fixed",
|
||||
queries=[
|
||||
dict(
|
||||
id="q-handler",
|
||||
status="answered",
|
||||
outcome="negative",
|
||||
target=config["targets"]["fixed"],
|
||||
witness="bound parameter discharges target",
|
||||
)
|
||||
],
|
||||
delta="removed",
|
||||
cache_state="warm_changed",
|
||||
)
|
||||
benign = record("benign", delta="unchanged", cache_state="warm_changed")
|
||||
run = {"profile_sha256": profile["profile_sha256"], "context": config, "records": [positive, fixed, benign]}
|
||||
_, declaration, _ = control()
|
||||
declaration = replace(declaration, key=replace(declaration.key, semantics=current["semantics"]), untouched_workload="untouched-control")
|
||||
runtime = {
|
||||
"runtime_verdict": "PASS",
|
||||
"provenance": current,
|
||||
"workloads": {
|
||||
name: dict(
|
||||
population=3,
|
||||
observed=3,
|
||||
missing=0,
|
||||
latency_samples=3,
|
||||
p95_seconds=3,
|
||||
completion=1,
|
||||
complete_coverage_cases=3,
|
||||
cancellation_overrun_seconds=0,
|
||||
timeouts=0,
|
||||
)
|
||||
for name in ("identical-tip", "function-edit", "dependency-edit", "configuration-edit", "cold", "growth", "multi-ref")
|
||||
},
|
||||
}
|
||||
return profile, validation, run, current, runtime, (declaration,)
|
||||
|
||||
|
||||
def decision(parts):
|
||||
profile, validation, run, current, runtime, declarations = parts
|
||||
return create_decision(profile, validation, run, current=current, runtime=runtime, declarations=declarations)
|
||||
|
||||
|
||||
def test_matching_reviewed_synthetic_population_can_be_loaded(tmp_path):
|
||||
parts = fixture()
|
||||
data = decision(parts)
|
||||
assert data["verdict"] == "PASS"
|
||||
path = tmp_path / "eligibility.json"
|
||||
path.write_text(json.dumps(data))
|
||||
loaded = load_registry(current=parts[3], path=path)
|
||||
assert len(loaded.capabilities) == 1
|
||||
assert loaded.capabilities[0].enabled
|
||||
assert loaded.capabilities[0].positive_controls == 1
|
||||
assert loaded.capabilities[0].reviewed_alerts == 1
|
||||
|
||||
|
||||
@pytest.mark.parametrize("mutation", ["no_alert", "runtime", "stale_runtime", "no_independent", "no_positive", "wrong_semantics"])
|
||||
def test_missing_joint_evidence_never_enables_advisory(mutation, tmp_path):
|
||||
parts = list(copy.deepcopy(fixture()))
|
||||
if mutation == "no_alert":
|
||||
parts[2]["records"][0]["alerts"] = []
|
||||
elif mutation == "runtime":
|
||||
parts[4]["workloads"]["function-edit"]["p95_seconds"] = 31
|
||||
elif mutation == "stale_runtime":
|
||||
parts[4]["provenance"]["policy"] = "other"
|
||||
elif mutation == "no_independent":
|
||||
parts[5] = (replace(parts[5][0], untouched_workload="wrong-repository"),)
|
||||
elif mutation == "no_positive":
|
||||
parts[2]["records"] = parts[2]["records"][1:]
|
||||
else:
|
||||
parts[5] = (replace(parts[5][0], key=replace(parts[5][0].key, semantics="wrong")),)
|
||||
data = decision(parts)
|
||||
assert data["verdict"] == "NO-GO"
|
||||
path = tmp_path / "eligibility.json"
|
||||
path.write_text(json.dumps(data))
|
||||
assert not load_registry(current=parts[3], path=path).capabilities
|
||||
|
||||
|
||||
def test_stale_or_tampered_registry_is_rejected(tmp_path):
|
||||
parts = fixture()
|
||||
data = decision(parts)
|
||||
path = tmp_path / "eligibility.json"
|
||||
path.write_text(json.dumps(data))
|
||||
stale = {**parts[3], "facts": "new-facts"}
|
||||
assert not load_registry(current=stale, path=path).capabilities
|
||||
data["entries"][0]["declaration"]["repair_template"] = "invented advice"
|
||||
path.write_text(json.dumps(data))
|
||||
assert not load_registry(current=parts[3], path=path).capabilities
|
||||
|
||||
|
||||
def test_runner_consumes_only_matching_registry(tmp_path, monkeypatch):
|
||||
import openultrasast.push.eligibility as registry
|
||||
from openultrasast.push.runner import _admit
|
||||
|
||||
parts = fixture()
|
||||
data = decision(parts)
|
||||
path = tmp_path / "eligibility.json"
|
||||
path.write_text(json.dumps(data))
|
||||
monkeypatch.setattr(registry, "_DEFAULT", path)
|
||||
candidate, _, _ = control()
|
||||
candidate = replace(candidate, capability=parts[5][0].key, delta=replace(candidate.delta, semantics=parts[3]["semantics"]))
|
||||
assert len(_admit((candidate,), parts[3]).defects) == 1
|
||||
assert not _admit((candidate,), {**parts[3], "facts": "changed"}).defects
|
||||
|
||||
|
||||
def test_changed_semantic_advice_needs_a_new_frozen_review():
|
||||
parts = list(fixture())
|
||||
parts[5] = (replace(parts[5][0], repair_template="Unreviewed advice at {operation} for {context}"),)
|
||||
assert decision(parts)["verdict"] == "NO-GO"
|
||||
|
||||
|
||||
def test_shipped_registry_is_reproducible_no_go():
|
||||
from openultrasast.push.eligibility import _DEFAULT
|
||||
|
||||
data = json.loads(_DEFAULT.read_bytes())
|
||||
loaded = load_registry(current=data["bindings"])
|
||||
assert loaded.status == "NO-GO"
|
||||
assert not loaded.capabilities
|
||||
assert data["entries"]
|
||||
assert all(not row["declaration"]["enabled"] for row in data["entries"])
|
||||
Reference in New Issue
Block a user