mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
feat(pre-push-safety-net): enforce evidence eligibility and record blocked qualification
This commit is contained in:
@@ -41,3 +41,26 @@ and its compressed full evidence bundle. All 21 deadline-bound transactions time
|
|||||||
Every measured artifact matches the frozen installed semantics. No source, engine,
|
Every measured artifact matches the frozen installed semantics. No source, engine,
|
||||||
query, fact or ranking change was made during this run. The follow-up remains necessary
|
query, fact or ranking change was made during this run. The follow-up remains necessary
|
||||||
before rollout; the evaluation task does not authorize weakening its gates.
|
before rollout; the evaluation task does not authorize weakening its gates.
|
||||||
|
|
||||||
|
## VAmPI qualification follow-up, 2026-09-13
|
||||||
|
|
||||||
|
Task 8.3's packaged frozen regression (139,164 total snapshot bytes read per replay)
|
||||||
|
retains raw findings at `models/user_model.py:73:get_user`,
|
||||||
|
`api_views/users.py:187:update_password`, and `api_views/books.py:51:get_by_title`.
|
||||||
|
The SQLI target has no selected exact function question; the two authorization target
|
||||||
|
questions are selected at positions 93 and 63 but have `change_context_incomplete`.
|
||||||
|
Head scope retains 138 selected and 20 deferred questions. Raw findings do not prove
|
||||||
|
completed target questions or a transitive source-to-target witness.
|
||||||
|
|
||||||
|
The first lab run took 112.415s; same-snapshot cached replays took 6.816s and 6.684s,
|
||||||
|
all incomplete. This is diagnostic reuse, not changed-code performance qualification.
|
||||||
|
The 600-second budget is distinct from the hook acceptance profile. Full artifacts are
|
||||||
|
in `2026-09-13-push-qualification.json.gz` under the regression population.
|
||||||
|
|
||||||
|
Reconcile evidence normalization and the unsupported context projection at the shared
|
||||||
|
scan boundary before downstream admission: retain the actual query-origin/path witness
|
||||||
|
when available, resolve supported first-party context, and keep unresolved dynamic/vendor
|
||||||
|
boundaries explicit. Do not fabricate transitivity from a raw finding at the target.
|
||||||
|
Revalidate ordinary VAmPI scan, exact question accounting, push regressions and independent
|
||||||
|
qualification after an approved owner fix. This note records the prerequisite; it does
|
||||||
|
not silently approve an upstream implementation or change detector semantics.
|
||||||
|
|||||||
@@ -106,3 +106,98 @@ retune the ranker or weaken the population/gates to compensate.
|
|||||||
- CLAIM: Task 8.2 completed its prescribed measurement and owner routing; rollout remains NO-GO.
|
- CLAIM: Task 8.2 completed its prescribed measurement and owner routing; rollout remains NO-GO.
|
||||||
- EVIDENCE: Versioned scorecard and 116-file verified bundle; seven scorer controls and full regression suite pass.
|
- EVIDENCE: Versioned scorecard and 116-file verified bundle; seven scorer controls and full regression suite pass.
|
||||||
- GAPS: No qualified warm graph path or independent capability eligibility established. These failed gates cannot authorize rollout.
|
- GAPS: No qualified warm graph path or independent capability eligibility established. These failed gates cannot authorize rollout.
|
||||||
|
|
||||||
|
## Task 8.3 brief
|
||||||
|
|
||||||
|
Requirements 3.1, 3.2, 6.1–6.5 and 8.1–8.4; evaluation and actionability integration.
|
||||||
|
Implement a trusted installed eligibility resource whose evidence is recomputed against
|
||||||
|
exact core/engine/runtime/facts/query/policy/semantic identities. Require reviewed
|
||||||
|
operation/advice scope, vulnerable/fixed/benign populations, nonzero actual positive
|
||||||
|
controls and reviewed alerts, untouched per-capability evaluation and all joint gates.
|
||||||
|
The normal runner consumes only eligible declarations; stale or inconsistent resources
|
||||||
|
admit nothing. This is integrity checking of trusted release evidence, not signing or
|
||||||
|
permission to load repository-authored eligibility.
|
||||||
|
|
||||||
|
Freeze the current packaged candidate before the six PMPro/Node development replays.
|
||||||
|
Run readable VAmPI regression snapshots under a separate 600-second laboratory profile;
|
||||||
|
keep their missing independent pairs unresolved. Execute libpng as an unsupported
|
||||||
|
30-second envelope control. Do not inspect or tune on Ghost before its reviewed
|
||||||
|
population exists. Missing required evidence leaves task 8.3 qualification blocked,
|
||||||
|
even when the reproducible NO-GO artifact and admission implementation are delivered.
|
||||||
|
|
||||||
|
### Review scope
|
||||||
|
|
||||||
|
Synthetic controls prove that a matching reviewed population can reach the real runner
|
||||||
|
admission seam, while stale facts/runtime, missing positive observations, absent reviews,
|
||||||
|
wrong untouched workloads and changed consequence/repair semantics cannot. They are
|
||||||
|
contract controls, not real capability qualification. The installed NO-GO resource must
|
||||||
|
recompute identically and enable zero capabilities. The source has no repository- or
|
||||||
|
language-specific rank changes, frontend adaptations or new abstract domains.
|
||||||
|
|
||||||
|
### Task 8.3 measured outcome
|
||||||
|
|
||||||
|
Packaged candidate `9464e229ccfdc873a02e3a0db7c1e83a6d6ec6d3e8234ae7b97677b4aed836fd`
|
||||||
|
executed offline as UID 1000 with a 3 GiB memory limit. Each profile was written before
|
||||||
|
its runs. All six development pushes saved artifacts, timed out in 30.559–30.678s and
|
||||||
|
retained all 11 declared cases. Source snapshot receipts read 57.9–58.1 MB per PMPro
|
||||||
|
comparison and 5.95 MB per Node comparison. These are snapshot bytes, not graph census.
|
||||||
|
The supported introducing-change recall is 0/2, Wilson 95% interval [0, 0.658]; actual
|
||||||
|
reviewed alert precision is undefined (0/0), never 100%. Neither fixed case has a
|
||||||
|
completed witnessed negative. The two benign pushes emitted no alerts; that silence
|
||||||
|
with incomplete coverage does not establish a useful safety net.
|
||||||
|
|
||||||
|
VAmPI's 600-second laboratory profile executed all three pinned regression targets,
|
||||||
|
reading 139,164 snapshot bytes per replay. First run 112.415s; subsequent identical
|
||||||
|
comparisons 6.816s and 6.684s, each with two graph hits and 403 query hits. Raw findings
|
||||||
|
retain the known SQLI and two authorization sites, but the SQLI exact function question
|
||||||
|
is unlocated and authorization questions are `change_context_incomplete`. All three
|
||||||
|
remain unresolved; no transitive witness or completed regression detection is credited.
|
||||||
|
There are 138 selected and 20 deferred head questions. Context/witness normalization
|
||||||
|
is routed to the shared scan owner. A registry-generation provenance probe overlapped
|
||||||
|
startup of the first lab run; these timings are diagnostic, not runtime-gate samples.
|
||||||
|
|
||||||
|
The initial instrument labeled all regression replays cold. The two proved identical
|
||||||
|
cache replays were relabeled `identical_tip` and rescored without changing measurements,
|
||||||
|
profile or population; original run/scorecard/instrument and correction are retained.
|
||||||
|
Future instrument output distinguishes these identical comparisons. None is a warm edit.
|
||||||
|
|
||||||
|
The libpng unsupported envelope saved a partial artifact after 30.590s. Ghost's checkout
|
||||||
|
and reviewed independent cases remain missing and were not scanned. Independent PHP
|
||||||
|
and Python admission populations also remain missing. No ranker, fact, adapter or query
|
||||||
|
retune occurred. The 59-file compressed evidence bundle is verified against originals:
|
||||||
|
`benchmarks/measurements/2026-09-13-push-qualification.json.gz`, SHA-256
|
||||||
|
`79226b4e54042559ba5cc2b94b69b1dcddb550872fd0783b2380368be72f90bd`.
|
||||||
|
|
||||||
|
The installed eligibility resource is reproducible NO-GO, digest
|
||||||
|
`2ddd94d4dc3fd6b84e7de38e5f211d2653fb993c64e4e43cedf7e06062c3265c`, with zero enabled
|
||||||
|
capabilities. Task 8.2 runtime evidence remains failed and predates the eligibility
|
||||||
|
policy addition; it is correctly rejected as admission evidence, not relabeled current.
|
||||||
|
|
||||||
|
## Status Report
|
||||||
|
- STATUS: BLOCKED
|
||||||
|
- TASK: 8.3
|
||||||
|
- RED_PHASE_OUTPUT: `/tmp/group8-3-red.log`: eight failures with eligibility disabled; ON passed, switch removed; final synthetic and installed-resource contracts verified separately.
|
||||||
|
- IMPLEMENTED: Versioned eligibility generation/loading, production admission integration, real offline replay and NO-GO artifact.
|
||||||
|
- BLOCKER: Required independent Node/PHP/Python reviewed populations and verified VAmPI transitive question witness are absent. Runtime/completion gates fail.
|
||||||
|
|
||||||
|
## Review Verdict
|
||||||
|
- VERDICT: REJECTED
|
||||||
|
- TASK: 8.3 completion claim
|
||||||
|
- MECHANICAL_RESULTS: Eligibility and runner implementation inspected; fixed semantic/advice scope, observed controls and exact provenance required. No language-specific ranker changes, placeholders or secrets. RED phase verified.
|
||||||
|
- FINDINGS: Implementation and reproducible NO-GO decision are concrete, but task 8.3's mandatory independent transfer and transitive-regression evidence is not established. Main-context fallback does not provide independent review.
|
||||||
|
- REMEDIATION: Repair the shared context/witness and census/runtime prerequisites; obtain reviewed untouched paired populations, refreeze and rerun all joint gates. Keep capabilities disabled meanwhile.
|
||||||
|
- SUMMARY: Retain the useful implementation, leave task 8.3 unchecked and feature rollout NO-GO.
|
||||||
|
|
||||||
|
## Verification Result
|
||||||
|
- STATUS: NOT_VERIFIED
|
||||||
|
- CLAIM_TYPE: TASK
|
||||||
|
- CLAIM: Full task 8.3 qualification complete.
|
||||||
|
- EVIDENCE: Real frozen replay and installed decision explicitly preserve failed gates and missing evidence.
|
||||||
|
- GAPS: Independent populations, completed positive/fixed controls, VAmPI transitive witness and representative runtime/coverage.
|
||||||
|
- NOTES: Proceed with authorized task 8.4's explicit NO-GO acceptance reconciliation; it does not depend on pretending qualification passed.
|
||||||
|
|
||||||
|
Task 8.3 implementation verification: full suite **1317 passed, 9 skipped in 115.80s**
|
||||||
|
(`/tmp/group8-3-complete-suite.log`, exit 0). After the measurement-label correction,
|
||||||
|
14 focused eligibility/replay-instrument tests passed in 0.95s. Ruff, format, mypy
|
||||||
|
(116 source files) and diff checks pass. These verify the delivered code and recorded
|
||||||
|
NO-GO behavior, not the rejected full qualification claim above.
|
||||||
|
|||||||
@@ -132,7 +132,10 @@
|
|||||||
"implementation-group-8.md"
|
"implementation-group-8.md"
|
||||||
],
|
],
|
||||||
"next_group": 8,
|
"next_group": 8,
|
||||||
"blocked_tasks": [],
|
"blocked_tasks": [
|
||||||
"feasibility": "NO-GO: all 21 frozen NodeGoat transactions timed out; 0/24 declared target checks completed. Cancellation/reporting remained below 2 seconds. Priming did not qualify graph reuse; named census and engine lifecycle work routed to contributor-scan."
|
"8.3"
|
||||||
|
],
|
||||||
|
"feasibility": "NO-GO: all 21 frozen NodeGoat transactions timed out; 0/24 declared target checks completed. Cancellation/reporting remained below 2 seconds. Priming did not qualify graph reuse; named census and engine lifecycle work routed to contributor-scan.",
|
||||||
|
"qualification": "NO-GO: eligibility implementation and evidence delivered; required independent populations and verified VAmPI transitive witness remain missing. Zero capabilities enabled."
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -234,6 +234,7 @@ independent capability qualification. Groups 7–8 remain pending; rollout remai
|
|||||||
- _Boundary: Evaluation harness, Delta and Actionability Policy integration_
|
- _Boundary: Evaluation harness, Delta and Actionability Policy integration_
|
||||||
- _Depends: 1.2, 1.3, 5.2, 7.3, 8.2_
|
- _Depends: 1.2, 1.3, 5.2, 7.3, 8.2_
|
||||||
- _Requirements: 3.1, 3.2, 6.1, 6.2, 6.3, 6.4, 6.5, 8.1, 8.2, 8.3, 8.4_
|
- _Requirements: 3.1, 3.2, 6.1, 6.2, 6.3, 6.4, 6.5, 8.1, 8.2, 8.3, 8.4_
|
||||||
|
- _Blocked 2026-09-13:_ Eligibility implementation and reproducible NO-GO artifact delivered; independent reviewed Node/PHP/Python populations and verified VAmPI transitive witness remain missing. Runtime/coverage gates fail. See `implementation-group-8.md` and contributor-scan owner follow-up.
|
||||||
- [ ] 8.4 Verify the packaged end-to-end outcome and acceptance coverage
|
- [ ] 8.4 Verify the packaged end-to-end outcome and acceptance coverage
|
||||||
- Run the required regression checks and a local packaged smoke over supported security/fixed/benign pushes; verify no-model startup, preservation of hook behavior and truthful partial results.
|
- Run the required regression checks and a local packaged smoke over supported security/fixed/benign pushes; verify no-model startup, preservation of hook behavior and truthful partial results.
|
||||||
- Reconcile every acceptance criterion with fresh evidence and record unresolved capability/performance blockers separately from completed plumbing.
|
- Reconcile every acceptance criterion with fresh evidence and record unresolved capability/performance blockers separately from completed plumbing.
|
||||||
|
|||||||
@@ -1,10 +1,10 @@
|
|||||||
{
|
{
|
||||||
"counts": {
|
"counts": {
|
||||||
"load_bearing": 105,
|
"load_bearing": 106,
|
||||||
"standalone_capability": 9
|
"standalone_capability": 9
|
||||||
},
|
},
|
||||||
"modules": 114,
|
"modules": 115,
|
||||||
"note": "Refreshed for group 7: push runner reaches optional bounded witness selection; registry remains empty.",
|
"note": "Group 8: runner reaches recomputed versioned eligibility and its frozen scorer.",
|
||||||
"orphaned": [],
|
"orphaned": [],
|
||||||
"rows": [
|
"rows": [
|
||||||
{
|
{
|
||||||
@@ -140,6 +140,7 @@
|
|||||||
"model.scan",
|
"model.scan",
|
||||||
"model.taint",
|
"model.taint",
|
||||||
"push.cache",
|
"push.cache",
|
||||||
|
"push.eligibility",
|
||||||
"push.policy",
|
"push.policy",
|
||||||
"push.reuse",
|
"push.reuse",
|
||||||
"push.runner"
|
"push.runner"
|
||||||
@@ -152,6 +153,7 @@
|
|||||||
"importers": [
|
"importers": [
|
||||||
"cpg.backend",
|
"cpg.backend",
|
||||||
"push.cache",
|
"push.cache",
|
||||||
|
"push.eligibility",
|
||||||
"push.policy",
|
"push.policy",
|
||||||
"push.reuse",
|
"push.reuse",
|
||||||
"push.runner"
|
"push.runner"
|
||||||
@@ -704,6 +706,7 @@
|
|||||||
"importers": [
|
"importers": [
|
||||||
"cli",
|
"cli",
|
||||||
"push.assistance",
|
"push.assistance",
|
||||||
|
"push.eligibility",
|
||||||
"push.policy",
|
"push.policy",
|
||||||
"push.report",
|
"push.report",
|
||||||
"push.reuse",
|
"push.reuse",
|
||||||
@@ -746,6 +749,15 @@
|
|||||||
{
|
{
|
||||||
"classification": "load_bearing",
|
"classification": "load_bearing",
|
||||||
"importers": [
|
"importers": [
|
||||||
|
"push.runner"
|
||||||
|
],
|
||||||
|
"module": "push.eligibility",
|
||||||
|
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"classification": "load_bearing",
|
||||||
|
"importers": [
|
||||||
|
"push.eligibility",
|
||||||
"push.report",
|
"push.report",
|
||||||
"push.runner"
|
"push.runner"
|
||||||
],
|
],
|
||||||
@@ -794,7 +806,9 @@
|
|||||||
},
|
},
|
||||||
{
|
{
|
||||||
"classification": "standalone_capability",
|
"classification": "standalone_capability",
|
||||||
"importers": [],
|
"importers": [
|
||||||
|
"push.eligibility"
|
||||||
|
],
|
||||||
"module": "push_scoring",
|
"module": "push_scoring",
|
||||||
"reason": "pre-push-safety-net task 1.3: frozen-profile diagnostic outcome scorer; python -m openultrasast.push_scoring"
|
"reason": "pre-push-safety-net task 1.3: frozen-profile diagnostic outcome scorer; python -m openultrasast.push_scoring"
|
||||||
},
|
},
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
Binary file not shown.
@@ -188,3 +188,27 @@ reports nearest-rank p50/p95 with sample counts, and never treats no answers as
|
|||||||
Exit 1 with a complete scorecard is measured NO-GO; a missing scorecard is an instrument
|
Exit 1 with a complete scorecard is measured NO-GO; a missing scorecard is an instrument
|
||||||
failure. Three samples per workload on development NodeGoat cannot independently qualify
|
failure. Three samples per workload on development NodeGoat cannot independently qualify
|
||||||
a production language/framework/family or establish broad multi-project performance.
|
a production language/framework/family or establish broad multi-project performance.
|
||||||
|
|
||||||
|
### Capability qualification (task 8.3)
|
||||||
|
|
||||||
|
The same `measure.py` replay instrument accepts `--selection development` (the
|
||||||
|
six PMPro/NodeGoat changes), `--selection regression` (three VAmPI targets), or
|
||||||
|
`--selection envelope` (libpng). Each profile retains the full declared population.
|
||||||
|
Readable regression snapshots may execute while their missing independent evaluation
|
||||||
|
prerequisites remain unresolved. `--deadline 600 --cache-dir /results/lab-cache`
|
||||||
|
is a separately frozen laboratory experiment, never a replacement for the 30-second
|
||||||
|
hook profile. A ranked or directly detected VAmPI target does not prove a transitive path.
|
||||||
|
|
||||||
|
`qualify_registry.py --evidence DIR --runtime SCORECARD --out NEW_FILE` recomputes
|
||||||
|
eligibility from the frozen profile, validated inputs, actual run and runtime scorecard.
|
||||||
|
Without `--declarations` it creates explicitly unreviewed, experimental declarations.
|
||||||
|
A reviewed declaration must match its frozen capability key, operation symbols, advice,
|
||||||
|
review identity, untouched workload and exact installed analysis semantics; all quality,
|
||||||
|
completion and runtime gates must pass. Synthetic positive controls in the unit suite
|
||||||
|
verify this contract and do not qualify real capabilities.
|
||||||
|
|
||||||
|
The installed `openultrasast/push/eligibility.json` is a trusted release input, not
|
||||||
|
repository configuration. The runner recomputes its decision and rejects stale or
|
||||||
|
modified evidence. This integrity check is not a cryptographic reviewer signature.
|
||||||
|
The versioned NO-GO registry enables no capability. Regenerate evidence after changes
|
||||||
|
to core, engine, facts, queries, policy or semantics; do not relabel old measurements.
|
||||||
|
|||||||
@@ -163,31 +163,44 @@ def main() -> int:
|
|||||||
parser.add_argument("--manifest", type=Path, required=True)
|
parser.add_argument("--manifest", type=Path, required=True)
|
||||||
parser.add_argument("--cache", type=Path, required=True)
|
parser.add_argument("--cache", type=Path, required=True)
|
||||||
parser.add_argument("--out", type=Path, required=True)
|
parser.add_argument("--out", type=Path, required=True)
|
||||||
|
parser.add_argument("--selection", choices=("development", "regression", "envelope"), default="development")
|
||||||
|
parser.add_argument("--deadline", type=float, default=30)
|
||||||
|
parser.add_argument("--cache-dir", type=Path)
|
||||||
args = parser.parse_args()
|
args = parser.parse_args()
|
||||||
|
if args.deadline <= 0:
|
||||||
|
parser.error("deadline must be positive")
|
||||||
args.out.mkdir(parents=True, exist_ok=False)
|
args.out.mkdir(parents=True, exist_ok=False)
|
||||||
manifest = json.loads(args.manifest.read_bytes())
|
manifest = json.loads(args.manifest.read_bytes())
|
||||||
validation = validate_manifest(args.manifest, cache_root=args.cache, fetch=False)
|
validation = validate_manifest(args.manifest, cache_root=args.cache, fetch=False)
|
||||||
write(args.out / "input-validation.json", validation)
|
write(args.out / "input-validation.json", validation)
|
||||||
settings = ScanBudget(max_model_calls=0, max_regions=500, order_by_evidence=True)
|
settings = ScanBudget(max_model_calls=0, max_regions=500, order_by_evidence=True)
|
||||||
initial = _provenance(args.manifest.parent, settings)
|
initial = _provenance(args.manifest.parent, settings)
|
||||||
keys = ("ranking_mode", "engine", "facts", "queries", "policy", "core")
|
keys = ("ranking_mode", "engine", "engine_runtime", "facts", "queries", "policy", "core", "semantics")
|
||||||
config = {key: initial[key] for key in keys}
|
config = {key: initial[key] for key in keys}
|
||||||
config.update(
|
config.update(
|
||||||
version="first-replay-v2-2026-09-13",
|
version="qualification-replay-v1-" + args.selection,
|
||||||
scope="full tracked tree; declared first-party graph/target exclusions; evidence ranker",
|
scope="full tracked tree; declared first-party graph/target exclusions; evidence ranker",
|
||||||
hardware=platform.platform(),
|
hardware=platform.platform(),
|
||||||
budget_seconds=30,
|
budget_seconds=args.deadline,
|
||||||
targets=json.loads((args.manifest.parent / "diagnostic-config-v1.json").read_text())["targets"],
|
targets=json.loads((args.manifest.parent / "diagnostic-config-v1.json").read_text())["targets"],
|
||||||
)
|
)
|
||||||
profile = freeze_profile(args.manifest.read_bytes(), config)
|
profile = freeze_profile(args.manifest.read_bytes(), config)
|
||||||
write(args.out / "profile.json", profile) # Freeze on disk BEFORE any measurement.
|
write(args.out / "profile.json", profile) # Freeze on disk BEFORE any measurement.
|
||||||
run = {"profile_sha256": profile["profile_sha256"], "context": dict(config), "records": []}
|
run = {"profile_sha256": profile["profile_sha256"], "context": dict(config), "records": []}
|
||||||
evidence = {"cases": [], "core_changes": [], "adapter_changes": [], "freeze_after_pmpro": None}
|
evidence = {"cases": [], "core_changes": [], "adapter_changes": [], "freeze_after_pmpro": None}
|
||||||
ready = {c["id"]: c["status"] == "ready" for c in validation["cases"]}
|
valid_snapshots = {s["id"] for s in validation["snapshots"] if s["status"] == "ready"}
|
||||||
|
ready = {
|
||||||
|
c["id"]: c["status"] == "ready" or (c["label"] == "regression" and c["base"] in valid_snapshots and c["tip"] in valid_snapshots)
|
||||||
|
for c in validation["cases"]
|
||||||
|
}
|
||||||
snapshots = {s["id"]: s for s in manifest["snapshots"]}
|
snapshots = {s["id"]: s for s in manifest["snapshots"]}
|
||||||
# Fixed declared order; reserved, regression and unsupported populations stay
|
# Fixed declared order; reserved, regression and unsupported populations stay
|
||||||
# in the joint scorer without scanning or tuning on the holdout.
|
# in the joint scorer without scanning or tuning on the holdout.
|
||||||
cases = [c for prefix in ("pmpro-", "nodegoat-") for c in manifest["cases"] if c["id"].startswith(prefix)]
|
cases = [c for prefix in ("pmpro-", "nodegoat-") for c in manifest["cases"] if c["id"].startswith(prefix)]
|
||||||
|
if args.selection == "regression":
|
||||||
|
cases = [c for c in manifest["cases"] if c["label"] == "regression"]
|
||||||
|
elif args.selection == "envelope":
|
||||||
|
cases = [c for c in manifest["cases"] if c["label"] == "unsupported"]
|
||||||
with tempfile.TemporaryDirectory(prefix="ousast-experiment-") as temporary:
|
with tempfile.TemporaryDirectory(prefix="ousast-experiment-") as temporary:
|
||||||
bare = Path(temporary) / "objects.git"
|
bare = Path(temporary) / "objects.git"
|
||||||
bare.mkdir()
|
bare.mkdir()
|
||||||
@@ -209,6 +222,7 @@ def main() -> int:
|
|||||||
args.out / "replay-pins.json",
|
args.out / "replay-pins.json",
|
||||||
{"snapshots": pins, "provenance": "Full upstream trees; declared authored edits in a private bare Git repository."},
|
{"snapshots": pins, "provenance": "Full upstream trees; declared authored edits in a private bare Git repository."},
|
||||||
)
|
)
|
||||||
|
seen_comparisons = set()
|
||||||
for case in cases:
|
for case in cases:
|
||||||
if not ready[case["id"]]:
|
if not ready[case["id"]]:
|
||||||
evidence["cases"].append({"case_id": case["id"], "status": "missing_input_prerequisite"})
|
evidence["cases"].append({"case_id": case["id"], "status": "missing_input_prerequisite"})
|
||||||
@@ -231,14 +245,16 @@ def main() -> int:
|
|||||||
"--artifact",
|
"--artifact",
|
||||||
str(artifact_path),
|
str(artifact_path),
|
||||||
"--deadline",
|
"--deadline",
|
||||||
"30",
|
str(args.deadline),
|
||||||
"--cancellation-allowance",
|
"--cancellation-allowance",
|
||||||
"2",
|
"2",
|
||||||
"--max-regions",
|
"--max-regions",
|
||||||
"500",
|
"500",
|
||||||
]
|
]
|
||||||
|
if args.cache_dir is not None:
|
||||||
|
command += ["--cache-dir", str(args.cache_dir)]
|
||||||
start = time.monotonic()
|
start = time.monotonic()
|
||||||
process = subprocess.run(command, capture_output=True, text=True, timeout=62)
|
process = subprocess.run(command, capture_output=True, text=True, timeout=args.deadline + 5)
|
||||||
elapsed = time.monotonic() - start
|
elapsed = time.monotonic() - start
|
||||||
(args.out / (case["id"] + ".stdout")).write_text(process.stdout)
|
(args.out / (case["id"] + ".stdout")).write_text(process.stdout)
|
||||||
(args.out / (case["id"] + ".stderr")).write_text(process.stderr)
|
(args.out / (case["id"] + ".stderr")).write_text(process.stderr)
|
||||||
@@ -257,7 +273,11 @@ def main() -> int:
|
|||||||
# Unmeasured provenance caused by preparation timeout is explicit;
|
# Unmeasured provenance caused by preparation timeout is explicit;
|
||||||
# never replace it with a fictitious runtime measurement.
|
# never replace it with a fictitious runtime measurement.
|
||||||
entry["runtime_provenance_available"] = all(artifact["provenance"].get(key) == initial[key] for key in keys)
|
entry["runtime_provenance_available"] = all(artifact["provenance"].get(key) == initial[key] for key in keys)
|
||||||
record = execution_record(case["id"], artifact, elapsed=elapsed, target=config["targets"][case["id"]])
|
record = execution_record(case["id"], artifact, elapsed=elapsed, target=config["targets"].get(case["id"]))
|
||||||
|
comparison = (pins[case["base"]], pins[case["tip"]])
|
||||||
|
if args.cache_dir is not None and comparison in seen_comparisons and artifact["timings"].get("graph_hits", 0) >= 2:
|
||||||
|
record["cache_state"] = "identical_tip"
|
||||||
|
seen_comparisons.add(comparison)
|
||||||
entry["coverage_reasons"] = artifact["admission"]["coverage_reasons"]
|
entry["coverage_reasons"] = artifact["admission"]["coverage_reasons"]
|
||||||
entry["stage_timings"] = artifact["timings"]
|
entry["stage_timings"] = artifact["timings"]
|
||||||
entry["snapshot_bytes_read"] = sum(s["bytes_read"] for s in artifact["snapshots"])
|
entry["snapshot_bytes_read"] = sum(s["bytes_read"] for s in artifact["snapshots"])
|
||||||
@@ -291,13 +311,15 @@ def main() -> int:
|
|||||||
"decision": "NO-GO",
|
"decision": "NO-GO",
|
||||||
"capabilities_enabled": False,
|
"capabilities_enabled": False,
|
||||||
"scorecard": "scorecard.json",
|
"scorecard": "scorecard.json",
|
||||||
"reason": "First cold diagnostic experiment; independent eligibility and joint quality/warm-latency gates remain unmet.",
|
"reason": "Qualification evidence run; independent eligibility and joint quality/warm-latency gates remain unmet.",
|
||||||
|
"selection": args.selection,
|
||||||
|
"analysis_deadline_seconds": args.deadline,
|
||||||
"recorded_cases": len(run["records"]),
|
"recorded_cases": len(run["records"]),
|
||||||
"declared_population": len(manifest["cases"]),
|
"declared_population": len(manifest["cases"]),
|
||||||
"core_unchanged": True,
|
"core_unchanged": True,
|
||||||
"adapter_changes": [],
|
"adapter_changes": [],
|
||||||
"limits": [
|
"limits": [
|
||||||
"No warm-cache measurements",
|
"No representative warm changed-code measurements; identical-comparison reuse is reported separately",
|
||||||
"Untouched Ghost population not scanned",
|
"Untouched Ghost population not scanned",
|
||||||
"PHP/Python admission populations missing",
|
"PHP/Python admission populations missing",
|
||||||
"libpng retained as unsupported envelope; no C/C++ detection claim",
|
"libpng retained as unsupported envelope; no C/C++ detection claim",
|
||||||
|
|||||||
@@ -0,0 +1,73 @@
|
|||||||
|
"""Recompute versioned eligibility from frozen results; default declarations are experimental."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from openultrasast.model.scan import ScanBudget
|
||||||
|
from openultrasast.push.eligibility import create_decision
|
||||||
|
from openultrasast.push.policy import CapabilityAdmission, CapabilityKey
|
||||||
|
from openultrasast.push.runner import _provenance
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument("--evidence", type=Path, required=True)
|
||||||
|
parser.add_argument("--runtime", type=Path, required=True)
|
||||||
|
parser.add_argument("--out", type=Path, required=True)
|
||||||
|
parser.add_argument("--declarations", type=Path)
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
def read(name):
|
||||||
|
return json.loads((args.evidence / name).read_bytes())
|
||||||
|
|
||||||
|
profile, validation, run = read("profile.json"), read("input-validation.json"), read("run.json")
|
||||||
|
current = _provenance(args.evidence, ScanBudget(max_model_calls=0, max_regions=500, order_by_evidence=True))
|
||||||
|
if args.declarations:
|
||||||
|
declarations = tuple(CapabilityAdmission.from_payload(d) for d in json.loads(args.declarations.read_bytes()))
|
||||||
|
else:
|
||||||
|
declarations = []
|
||||||
|
for capability in sorted({c["capability"] for c in profile["cases"]}):
|
||||||
|
language, framework, family = capability.split("/")
|
||||||
|
key = CapabilityKey(
|
||||||
|
language,
|
||||||
|
"unspecified",
|
||||||
|
framework,
|
||||||
|
family,
|
||||||
|
"dominance" if family == "access_control" else "taint",
|
||||||
|
"unreviewed",
|
||||||
|
current["semantics"],
|
||||||
|
)
|
||||||
|
declarations.append(
|
||||||
|
CapabilityAdmission(
|
||||||
|
key,
|
||||||
|
"unqualified-population-v1",
|
||||||
|
str(args.evidence / "scorecard.json"),
|
||||||
|
"experimental",
|
||||||
|
"Unreviewed consequence for {operation} in {context}.",
|
||||||
|
"Unreviewed repair for {operation} in {context}.",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
decision = create_decision(
|
||||||
|
profile, validation, run, current=current, runtime=json.loads(args.runtime.read_bytes()), declarations=declarations
|
||||||
|
)
|
||||||
|
args.out.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
with args.out.open("x") as stream:
|
||||||
|
json.dump(decision, stream, indent=2, allow_nan=False)
|
||||||
|
stream.write("\n")
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"verdict": decision["verdict"],
|
||||||
|
"enabled": sum(e["declaration"]["enabled"] for e in decision["entries"]),
|
||||||
|
"artifact_sha256": decision["artifact_sha256"],
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return 0 if decision["verdict"] == "PASS" else 1
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,171 @@
|
|||||||
|
"""Versioned, locally shipped capability eligibility from frozen evaluation evidence.
|
||||||
|
|
||||||
|
This verifies integrity and recomputes gates; it is not a signature verifier. Registry
|
||||||
|
files are trusted release inputs, never repository configuration or model responses.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import math
|
||||||
|
from collections.abc import Mapping, Sequence
|
||||||
|
from dataclasses import dataclass, replace
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from openultrasast.cpg.artifact import digest_value
|
||||||
|
from openultrasast.push.policy import CapabilityAdmission
|
||||||
|
from openultrasast.push_scoring import _metrics, score
|
||||||
|
|
||||||
|
_BINDINGS = ("core", "engine", "engine_runtime", "facts", "queries", "policy", "semantics")
|
||||||
|
_WORKLOADS = ("identical-tip", "function-edit", "dependency-edit", "configuration-edit", "cold", "growth", "multi-ref")
|
||||||
|
_DEFAULT = Path(__file__).with_name("eligibility.json")
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class EligibilityLoad:
|
||||||
|
capabilities: tuple[CapabilityAdmission, ...]
|
||||||
|
status: str
|
||||||
|
reasons: tuple[str, ...]
|
||||||
|
|
||||||
|
|
||||||
|
def fingerprint() -> str:
|
||||||
|
try:
|
||||||
|
return digest_value(json.loads(_DEFAULT.read_bytes()))
|
||||||
|
except (OSError, ValueError):
|
||||||
|
return "eligibility_unavailable"
|
||||||
|
|
||||||
|
|
||||||
|
def _runtime_pass(runtime: Mapping[str, Any], current: Mapping[str, str]) -> bool:
|
||||||
|
if runtime.get("runtime_verdict") != "PASS" or any(runtime.get("provenance", {}).get(k) != current.get(k) for k in _BINDINGS):
|
||||||
|
return False
|
||||||
|
try:
|
||||||
|
for name in _WORKLOADS:
|
||||||
|
row = runtime["workloads"][name]
|
||||||
|
values = [
|
||||||
|
row[k]
|
||||||
|
for k in (
|
||||||
|
"population",
|
||||||
|
"observed",
|
||||||
|
"missing",
|
||||||
|
"latency_samples",
|
||||||
|
"p95_seconds",
|
||||||
|
"completion",
|
||||||
|
"complete_coverage_cases",
|
||||||
|
"cancellation_overrun_seconds",
|
||||||
|
"timeouts",
|
||||||
|
)
|
||||||
|
]
|
||||||
|
if any(type(v) not in (int, float) or not math.isfinite(v) or v < 0 for v in values):
|
||||||
|
return False
|
||||||
|
population = row["population"]
|
||||||
|
if (
|
||||||
|
population < 1
|
||||||
|
or row["observed"] != population
|
||||||
|
or row["latency_samples"] != population
|
||||||
|
or row["missing"]
|
||||||
|
or row["timeouts"]
|
||||||
|
or row["completion"] < 0.95
|
||||||
|
or row["complete_coverage_cases"] / population < 0.95
|
||||||
|
or row["cancellation_overrun_seconds"] > 2
|
||||||
|
or (name != "cold" and row["p95_seconds"] > 30)
|
||||||
|
):
|
||||||
|
return False
|
||||||
|
except (KeyError, TypeError, ZeroDivisionError):
|
||||||
|
return False
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def create_decision(
|
||||||
|
profile: dict[str, Any],
|
||||||
|
validation: dict[str, Any],
|
||||||
|
run: dict[str, Any],
|
||||||
|
*,
|
||||||
|
current: Mapping[str, str],
|
||||||
|
runtime: dict[str, Any],
|
||||||
|
declarations: Sequence[CapabilityAdmission],
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
scored = score(profile, validation, run)
|
||||||
|
bindings = {k: current.get(k, "") for k in _BINDINGS}
|
||||||
|
common = []
|
||||||
|
if any(not bindings[k] or profile["config"].get(k) != bindings[k] for k in _BINDINGS):
|
||||||
|
common.append("evaluation_provenance_mismatch")
|
||||||
|
if not _runtime_pass(runtime, current):
|
||||||
|
common.append("runtime_gate_or_provenance_failed")
|
||||||
|
entries = []
|
||||||
|
for declaration in declarations:
|
||||||
|
key = declaration.key
|
||||||
|
capability = f"{key.language}/{key.framework}/{key.family}"
|
||||||
|
rows = [r for r in scored["cases"] if r["capability"] == capability and r["label"] != "regression"]
|
||||||
|
metrics = _metrics(rows)
|
||||||
|
reasons = list(common)
|
||||||
|
labels = {r["label"] for r in rows}
|
||||||
|
if not {"vulnerable", "fixed", "benign"} <= labels:
|
||||||
|
reasons.append("paired_positive_fixed_benign_population_missing")
|
||||||
|
if any(r["label"] in ("unsupported", "unreviewed") or r["input_status"] != "ready" or r.get("prerequisites") for r in rows):
|
||||||
|
reasons.append("unresolved_input_or_unsupported_population")
|
||||||
|
if not declaration.untouched_workload or any(
|
||||||
|
r.get("role") not in ("independent", "reserved") or r.get("workload") != declaration.untouched_workload for r in rows
|
||||||
|
):
|
||||||
|
reasons.append("independent_workload_missing")
|
||||||
|
semantic = profile["config"].get("capability_semantics", {}).get(digest_value(key.to_payload()), {})
|
||||||
|
if (
|
||||||
|
semantic.get("key") != key.to_payload()
|
||||||
|
or semantic.get("operation_symbols") != list(declaration.operation_symbols)
|
||||||
|
or semantic.get("review_id") != declaration.evaluation_id
|
||||||
|
or semantic.get("consequence_template") != declaration.consequence_template
|
||||||
|
or semantic.get("repair_template") != declaration.repair_template
|
||||||
|
or not declaration.operation_symbols
|
||||||
|
or key.semantics != bindings["semantics"]
|
||||||
|
or key.context in ("unreviewed", "unspecified", "")
|
||||||
|
or not declaration.calibration_artifact
|
||||||
|
):
|
||||||
|
reasons.append("semantic_scope_unreviewed_or_mismatched")
|
||||||
|
for name, threshold in (("precision", 0.95), ("actionable_recall", 0.90), ("completion", 0.95), ("query_completion", 0.95)):
|
||||||
|
if metrics[name]["value"] is None or metrics[name]["value"] < threshold:
|
||||||
|
reasons.append(name + "_gate_failed")
|
||||||
|
if metrics["fixed_side_silence"]["value"] != 1 or metrics["benign_interruption"]["value"] != 0:
|
||||||
|
reasons.append("fixed_or_benign_gate_failed")
|
||||||
|
warm = metrics["latency"]["warm_changed"]
|
||||||
|
if not warm["samples"] or warm["p95"] > 30:
|
||||||
|
reasons.append("independent_warm_latency_missing_or_failed")
|
||||||
|
positive = metrics["recall"]["numerator"]
|
||||||
|
reviewed = metrics["precision"]["numerator"]
|
||||||
|
if positive < 1 or reviewed < 1:
|
||||||
|
reasons.append("positive_controls_or_reviewed_alerts_missing")
|
||||||
|
accepted = not reasons
|
||||||
|
resolved = replace(
|
||||||
|
declaration, verdict="PASS" if accepted else "NO-GO", enabled=accepted, positive_controls=positive, reviewed_alerts=reviewed
|
||||||
|
)
|
||||||
|
entries.append(dict(declaration=resolved.to_payload(), metrics=metrics, reasons=sorted(set(reasons))))
|
||||||
|
result = dict(
|
||||||
|
schema_version=1,
|
||||||
|
bindings=bindings,
|
||||||
|
verdict="PASS" if any(not e["reasons"] for e in entries) else "NO-GO",
|
||||||
|
entries=entries,
|
||||||
|
inputs=dict(profile=profile, validation=validation, run=run, runtime=runtime, declarations=[d.to_payload() for d in declarations]),
|
||||||
|
)
|
||||||
|
result["artifact_sha256"] = digest_value(result)
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def load_registry(*, current: Mapping[str, str], path: Path | None = None) -> EligibilityLoad:
|
||||||
|
try:
|
||||||
|
with (path or _DEFAULT).open("rb") as stream:
|
||||||
|
content = stream.read(2 * 1024**2 + 1)
|
||||||
|
if len(content) > 2 * 1024**2:
|
||||||
|
raise ValueError("registry_size_limit")
|
||||||
|
data = json.loads(content)
|
||||||
|
if data.get("schema_version") != 1 or data.get("bindings") != {k: current.get(k, "") for k in _BINDINGS}:
|
||||||
|
return EligibilityLoad((), "stale", ("eligibility_provenance_mismatch",))
|
||||||
|
inputs = data["inputs"]
|
||||||
|
declarations = tuple(CapabilityAdmission.from_payload(d) for d in inputs["declarations"])
|
||||||
|
recomputed = create_decision(
|
||||||
|
inputs["profile"], inputs["validation"], inputs["run"], current=current, runtime=inputs["runtime"], declarations=declarations
|
||||||
|
)
|
||||||
|
if recomputed != data:
|
||||||
|
raise ValueError("registry_evidence_mismatch")
|
||||||
|
capabilities = tuple(CapabilityAdmission.from_payload(e["declaration"]) for e in data["entries"] if not e["reasons"])
|
||||||
|
return EligibilityLoad(capabilities, data["verdict"], tuple(sorted({reason for e in data["entries"] for reason in e["reasons"]})))
|
||||||
|
except (OSError, ValueError, TypeError, KeyError, AttributeError):
|
||||||
|
return EligibilityLoad((), "unavailable", ("eligibility_invalid_or_unavailable",))
|
||||||
@@ -26,6 +26,7 @@ from openultrasast.model.shipped import declared_sources
|
|||||||
from openultrasast.preprocess import build_file_target, enumerate_source_files
|
from openultrasast.preprocess import build_file_target, enumerate_source_files
|
||||||
from openultrasast.push.cache import ArtifactCache, SemanticKeys
|
from openultrasast.push.cache import ArtifactCache, SemanticKeys
|
||||||
from openultrasast.push.contracts import ComparisonAnalysis, PushComparison, PushResult, SnapshotManifest
|
from openultrasast.push.contracts import ComparisonAnalysis, PushComparison, PushResult, SnapshotManifest
|
||||||
|
from openultrasast.push.eligibility import fingerprint, load_registry
|
||||||
from openultrasast.push.policy import (
|
from openultrasast.push.policy import (
|
||||||
ActionableDefect,
|
ActionableDefect,
|
||||||
AdmissionCandidate,
|
AdmissionCandidate,
|
||||||
@@ -151,11 +152,22 @@ def _provenance(repository: Path, scan_budget: ScanBudget) -> dict[str, str]:
|
|||||||
),
|
),
|
||||||
"ranking_mode": "evidence",
|
"ranking_mode": "evidence",
|
||||||
"model": "disabled",
|
"model": "disabled",
|
||||||
"capabilities": "empty_registry_pending_task_8.3",
|
"capabilities": fingerprint(),
|
||||||
"cache": "cold_no_reuse",
|
"cache": "cold_no_reuse",
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _admit(candidates: tuple[AdmissionCandidate, ...], provenance: dict[str, str]) -> AdmissionResult:
|
||||||
|
registry = load_registry(current=provenance)
|
||||||
|
result = admit_candidates(candidates, capabilities=registry.capabilities)
|
||||||
|
if candidates and registry.reasons:
|
||||||
|
result = replace(
|
||||||
|
result,
|
||||||
|
coverage_reasons=tuple(dict.fromkeys((*result.coverage_reasons, *("eligibility:" + reason for reason in registry.reasons)))),
|
||||||
|
)
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
def _discovery_identity() -> str:
|
def _discovery_identity() -> str:
|
||||||
root = Path(__file__).resolve().parents[1]
|
root = Path(__file__).resolve().parents[1]
|
||||||
# Discovery invokes the shipped mapper, preprocessing, layout and family specs.
|
# Discovery invokes the shipped mapper, preprocessing, layout and family specs.
|
||||||
@@ -326,7 +338,7 @@ def _analyze(
|
|||||||
)
|
)
|
||||||
for c in delta.candidates
|
for c in delta.candidates
|
||||||
)
|
)
|
||||||
admission = _prepare(lambda: admit_candidates(candidates), budget)
|
admission = _prepare(lambda: _admit(candidates, provenance), budget)
|
||||||
timings[stage + "_seconds"] = time.monotonic() - stage_started
|
timings[stage + "_seconds"] = time.monotonic() - stage_started
|
||||||
stage, stage_started = "cleanup", time.monotonic()
|
stage, stage_started = "cleanup", time.monotonic()
|
||||||
timings[stage + "_seconds"] = time.monotonic() - stage_started
|
timings[stage + "_seconds"] = time.monotonic() - stage_started
|
||||||
|
|||||||
@@ -0,0 +1,163 @@
|
|||||||
|
"""Synthetic eligibility contracts, not real capability qualification."""
|
||||||
|
|
||||||
|
import copy
|
||||||
|
import json
|
||||||
|
from dataclasses import replace
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from test_push_admission import control
|
||||||
|
from test_push_scoring import inputs, record
|
||||||
|
|
||||||
|
from openultrasast.push.eligibility import create_decision, load_registry
|
||||||
|
from openultrasast.push_scoring import freeze_profile
|
||||||
|
|
||||||
|
|
||||||
|
def fixture():
|
||||||
|
old, validation = inputs()
|
||||||
|
current = {key: "synthetic-" + key for key in ("core", "engine", "engine_runtime", "facts", "queries", "policy", "semantics")}
|
||||||
|
from openultrasast.cpg.artifact import digest_value
|
||||||
|
|
||||||
|
_, declaration, _ = control()
|
||||||
|
declaration = replace(declaration, key=replace(declaration.key, semantics=current["semantics"]), untouched_workload="untouched-control")
|
||||||
|
config = {
|
||||||
|
**old["config"],
|
||||||
|
**current,
|
||||||
|
"capability_semantics": {
|
||||||
|
digest_value(declaration.key.to_payload()): {
|
||||||
|
"key": declaration.key.to_payload(),
|
||||||
|
"operation_symbols": list(declaration.operation_symbols),
|
||||||
|
"review_id": declaration.evaluation_id,
|
||||||
|
"consequence_template": declaration.consequence_template,
|
||||||
|
"repair_template": declaration.repair_template,
|
||||||
|
}
|
||||||
|
},
|
||||||
|
}
|
||||||
|
cases = [{**row, "role": "independent", "workload": "untouched-control"} for row in old["cases"][:3]]
|
||||||
|
raw = json.dumps({"schema_version": 1, "cases": cases}).encode()
|
||||||
|
profile = freeze_profile(raw, config)
|
||||||
|
validation = {**validation, "manifest_sha256": profile["manifest_sha256"], "cases": [{**c, "status": "ready"} for c in cases]}
|
||||||
|
positive = record(
|
||||||
|
alerts=[dict(review="actionable_true", review_evidence="synthetic-review", question_id="q-handler", site="api.js:4")],
|
||||||
|
cache_state="warm_changed",
|
||||||
|
)
|
||||||
|
fixed = record(
|
||||||
|
"fixed",
|
||||||
|
queries=[
|
||||||
|
dict(
|
||||||
|
id="q-handler",
|
||||||
|
status="answered",
|
||||||
|
outcome="negative",
|
||||||
|
target=config["targets"]["fixed"],
|
||||||
|
witness="bound parameter discharges target",
|
||||||
|
)
|
||||||
|
],
|
||||||
|
delta="removed",
|
||||||
|
cache_state="warm_changed",
|
||||||
|
)
|
||||||
|
benign = record("benign", delta="unchanged", cache_state="warm_changed")
|
||||||
|
run = {"profile_sha256": profile["profile_sha256"], "context": config, "records": [positive, fixed, benign]}
|
||||||
|
_, declaration, _ = control()
|
||||||
|
declaration = replace(declaration, key=replace(declaration.key, semantics=current["semantics"]), untouched_workload="untouched-control")
|
||||||
|
runtime = {
|
||||||
|
"runtime_verdict": "PASS",
|
||||||
|
"provenance": current,
|
||||||
|
"workloads": {
|
||||||
|
name: dict(
|
||||||
|
population=3,
|
||||||
|
observed=3,
|
||||||
|
missing=0,
|
||||||
|
latency_samples=3,
|
||||||
|
p95_seconds=3,
|
||||||
|
completion=1,
|
||||||
|
complete_coverage_cases=3,
|
||||||
|
cancellation_overrun_seconds=0,
|
||||||
|
timeouts=0,
|
||||||
|
)
|
||||||
|
for name in ("identical-tip", "function-edit", "dependency-edit", "configuration-edit", "cold", "growth", "multi-ref")
|
||||||
|
},
|
||||||
|
}
|
||||||
|
return profile, validation, run, current, runtime, (declaration,)
|
||||||
|
|
||||||
|
|
||||||
|
def decision(parts):
|
||||||
|
profile, validation, run, current, runtime, declarations = parts
|
||||||
|
return create_decision(profile, validation, run, current=current, runtime=runtime, declarations=declarations)
|
||||||
|
|
||||||
|
|
||||||
|
def test_matching_reviewed_synthetic_population_can_be_loaded(tmp_path):
|
||||||
|
parts = fixture()
|
||||||
|
data = decision(parts)
|
||||||
|
assert data["verdict"] == "PASS"
|
||||||
|
path = tmp_path / "eligibility.json"
|
||||||
|
path.write_text(json.dumps(data))
|
||||||
|
loaded = load_registry(current=parts[3], path=path)
|
||||||
|
assert len(loaded.capabilities) == 1
|
||||||
|
assert loaded.capabilities[0].enabled
|
||||||
|
assert loaded.capabilities[0].positive_controls == 1
|
||||||
|
assert loaded.capabilities[0].reviewed_alerts == 1
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("mutation", ["no_alert", "runtime", "stale_runtime", "no_independent", "no_positive", "wrong_semantics"])
|
||||||
|
def test_missing_joint_evidence_never_enables_advisory(mutation, tmp_path):
|
||||||
|
parts = list(copy.deepcopy(fixture()))
|
||||||
|
if mutation == "no_alert":
|
||||||
|
parts[2]["records"][0]["alerts"] = []
|
||||||
|
elif mutation == "runtime":
|
||||||
|
parts[4]["workloads"]["function-edit"]["p95_seconds"] = 31
|
||||||
|
elif mutation == "stale_runtime":
|
||||||
|
parts[4]["provenance"]["policy"] = "other"
|
||||||
|
elif mutation == "no_independent":
|
||||||
|
parts[5] = (replace(parts[5][0], untouched_workload="wrong-repository"),)
|
||||||
|
elif mutation == "no_positive":
|
||||||
|
parts[2]["records"] = parts[2]["records"][1:]
|
||||||
|
else:
|
||||||
|
parts[5] = (replace(parts[5][0], key=replace(parts[5][0].key, semantics="wrong")),)
|
||||||
|
data = decision(parts)
|
||||||
|
assert data["verdict"] == "NO-GO"
|
||||||
|
path = tmp_path / "eligibility.json"
|
||||||
|
path.write_text(json.dumps(data))
|
||||||
|
assert not load_registry(current=parts[3], path=path).capabilities
|
||||||
|
|
||||||
|
|
||||||
|
def test_stale_or_tampered_registry_is_rejected(tmp_path):
|
||||||
|
parts = fixture()
|
||||||
|
data = decision(parts)
|
||||||
|
path = tmp_path / "eligibility.json"
|
||||||
|
path.write_text(json.dumps(data))
|
||||||
|
stale = {**parts[3], "facts": "new-facts"}
|
||||||
|
assert not load_registry(current=stale, path=path).capabilities
|
||||||
|
data["entries"][0]["declaration"]["repair_template"] = "invented advice"
|
||||||
|
path.write_text(json.dumps(data))
|
||||||
|
assert not load_registry(current=parts[3], path=path).capabilities
|
||||||
|
|
||||||
|
|
||||||
|
def test_runner_consumes_only_matching_registry(tmp_path, monkeypatch):
|
||||||
|
import openultrasast.push.eligibility as registry
|
||||||
|
from openultrasast.push.runner import _admit
|
||||||
|
|
||||||
|
parts = fixture()
|
||||||
|
data = decision(parts)
|
||||||
|
path = tmp_path / "eligibility.json"
|
||||||
|
path.write_text(json.dumps(data))
|
||||||
|
monkeypatch.setattr(registry, "_DEFAULT", path)
|
||||||
|
candidate, _, _ = control()
|
||||||
|
candidate = replace(candidate, capability=parts[5][0].key, delta=replace(candidate.delta, semantics=parts[3]["semantics"]))
|
||||||
|
assert len(_admit((candidate,), parts[3]).defects) == 1
|
||||||
|
assert not _admit((candidate,), {**parts[3], "facts": "changed"}).defects
|
||||||
|
|
||||||
|
|
||||||
|
def test_changed_semantic_advice_needs_a_new_frozen_review():
|
||||||
|
parts = list(fixture())
|
||||||
|
parts[5] = (replace(parts[5][0], repair_template="Unreviewed advice at {operation} for {context}"),)
|
||||||
|
assert decision(parts)["verdict"] == "NO-GO"
|
||||||
|
|
||||||
|
|
||||||
|
def test_shipped_registry_is_reproducible_no_go():
|
||||||
|
from openultrasast.push.eligibility import _DEFAULT
|
||||||
|
|
||||||
|
data = json.loads(_DEFAULT.read_bytes())
|
||||||
|
loaded = load_registry(current=data["bindings"])
|
||||||
|
assert loaded.status == "NO-GO"
|
||||||
|
assert not loaded.capabilities
|
||||||
|
assert data["entries"]
|
||||||
|
assert all(not row["declaration"]["enabled"] for row in data["entries"])
|
||||||
Reference in New Issue
Block a user