# Design Document: detection-core ## Overview **Purpose:** detection-core provides the family-neutral, versioned contract between deterministic program analysis, vulnerability-family reasoning, bounded model programs, execution proof, measurement, and downstream consumers. It makes semantic incompleteness a first-class result and stops a family at the earliest failing readiness stage. **Users:** vulnerability-family implementers produce evidence through adapters; DSPy and proof adapters consume immutable evidence; benchmark workflows validate readiness and admission; pre-push and reporting consumers receive supported claims, bounded satisfaction, and bounded silence. **Impact:** the current authorization-shaped core becomes a temporary v1 compatibility surface. A neutral v2 write contract replaces it through a finite adapter migration. Authorization remains parked and supplies regression evidence only. Historical v1 artifacts remain readable but are never rewritten or used as a v2 trend baseline. Live v1 contracts and serialization first move behind one explicit, deletable `legacy_v1` namespace; cutover is then proved by syntax and capability removal rather than runtime alias inference. The evaluation-admission amendment adds a separate V3 root for exact-scope, multi-arm qualification under one shared population spend. It preserves every V2 admission artifact and replay path without conversion or reinterpretation. ### Goals - Define one typed, canonical v2 evidence record whose vocabulary is independent of any vulnerability family. - Let each family declare its security object, semantic roles, decisions, completeness rules, and model residuals without changing core code. - Expose instrument, recognition, transition, constraint, pair, development, admission, and integration failures before a later gate can run. - Restore the deterministic-evidence-to-DSPy loop through a dependency-free model handoff. - Preserve one execution-backed route to blocking without moving proof execution or rollout policy into the core. - End migration with one live write contract and two unlike reference-family conformance suites. - Make the v1 removal gate structural: every live v1 import and writer crosses an explicit compatibility namespace that is deleted at cutover. - Qualify every component and combined arm independently for each exact family/ecosystem scope while binding all arms to one reviewed plan, one atomic spend, and one complete replay. - Preserve V2 admission bytes, digests, decisions, and replay behavior as an immutable compatibility contract. ### Non-Goals - Implementing vulnerability-family algorithms, prompts, examples, or model optimization. - Expanding framework vocabulary or analyzing vendored dependency source. - Running DSPy, Joern, benchmark acquisition, proof sandboxes, or pre-push transactions inside `openultrasast.core`. - Reviving or re-admitting the parked authorization arm. - Enabling any capability for advisory or blocking rollout. - Providing cross-language canonical signatures; v2 is a Python-runtime contract. - Converting pooled V2 admission evidence into V3 scoped evidence or treating a V2 report as a V3 baseline. - Owning population lifecycle, durable CAS/object storage, observation access enforcement, detector isolation, retry orchestration, workflow presentation, or rollout decisions for the multi-arm benchmark. ## Boundary Commitments ### This Spec Owns - The v2 canonical JSON profile and digest rules for core records. - Typed semantic transitions, observations, completeness, analyzed scope, decisions, neutral outcomes, and evidence bases. - The versioned family contract and validation of declared ecosystems, roles, fields, decisions, outcome rules, and residual questions. - Coverage funnels, instrument refusal, measured-budget records, provenance and label lineage, fact-pack validation, readiness prerequisites, admission feasibility/scoring, and measurement schemas. - Dependency-free model request/result contracts and proof-attachment validation. - Push-consumer interfaces for defect claims, bounded satisfaction, and bounded silence. - Thin adjacent-owner translation seams that prove those public contracts integrate; they receive existing subsystem records and perform no analysis, model, proof, or persistence orchestration. - A finite v1 migration and read-only historical decoder. - The temporary `legacy_v1` compatibility namespace, frozen migration inventory, and syntax-only cutover boundary that make v1 write capability removable. - The V3 evaluation plan, exact arm/scope baseline and policy schemas, per-scope feasibility, sealed-observation authority validation, all-arm spend binding, V3 release, scoped reports, complete replay, and replay-only consumer authorization. ### Out of Boundary - Family analysis and abstract interpretation, including source/sink, dominance, sanitization, constraint, or configuration algorithms. - Target-specific facts, framework strategies, and fact acquisition. - DSPy execution, prompt construction, caching, optimization, and provider transport. - Population construction, label custody, readiness-artifact persistence, and run orchestration. - Proof search, sandbox execution, oracle semantics, and demonstration qualification. - Push scoping policy, alert rendering, capability rollout, and BLOCK enablement. - Production vulnerability-family contracts, target-specific authority bundles, structural matchers, and match-artifact persistence. Those belong to the downstream family spec. - Benchmark-owned population lifecycle, object-store/CAS adapters, detector execution, observation access enforcement, label joining, retries, workflow status, presentation, notification, and rollout policy. ### Allowed Dependencies - `openultrasast.core` depends only on Python 3.11 standard-library modules and lower core modules identified by the dependency direction below. - Core may consume immutable, typed records produced by benchmark, analyzer, model, and proof adapters; it never imports those packages. - Core may call an injected read-only `ContentResolver` protocol to verify referenced bytes and storage receipts; concrete stores import or implement the protocol and core never writes through it. - Core may read the current complete population-use chain through `PopulationCustodyPort`; the benchmark custodian remains the sole append-only writer. - Authorization, DSPy, search-proof, benchmark, and push adapters may import the public core contract. - Development and admission populations are read-only identities at the core boundary. - Downstream family adapters may inject typed contracts, packs, matches, policies, lineages, custody ports, content resolvers, and persisted content references. Core validates and applies those objects but never imports the target adapter or creates its authority. - No core module imports `openultrasast.semantic`, `openultrasast.hunt`, `openultrasast.search`, `openultrasast.tools`, `benchmarks`, or `dspy`. - V2 modules never import `openultrasast.core.legacy_v1` or the retained read-only `openultrasast.core.legacy` decoder. During migration, only compatibility shims and registered adjacent v1 consumers may import `legacy_v1`. - Production imports name a core submodule or symbol explicitly. Module-valued or dynamic access to `openultrasast.core` is outside the migration contract because it would hide legacy use from a deletion-backed gate. - V3 evaluation modules may reuse version-neutral core canonical, review, custody, readiness, evidence, proof, coverage, budget, and metric-arithmetic primitives. They do not import V2 aggregate roots; V2 modules do not import V3 modules. ### Revalidation Triggers - Changing the canonical profile, v2 schema, field-kind vocabulary, relation kinds, outcomes, or evidence bases revalidates every family adapter and consumer. - Changing a family contract, fact schema, model program identity, or outcome mapping invalidates dependent readiness receipts and starts a new measurement baseline. - Changing readiness stage order or prerequisite rules revalidates development, admission, and integration workflows. - Changing denominator policy, admission metrics, or threshold semantics requires new admission configuration and baseline identity. - Changing proof-attachment identity rules revalidates search-with-proof and every proof-consuming verdict path. - Adding a non-Python writer or cryptographic wire signature requires a canonicalization design review. - Adding a v1 compatibility export, migration-inventory entry, direct v1 serializer, or dynamic core import reopens the namespace-isolation design and blocks cutover. - Changing any V3 plan, observation-slot, retry/history, shared-spend, case-release, proof-candidate, combination, scoped metric, aggregate, report, replay, or consumer selection contract revalidates the benchmark and every V3 consumer. - Extracting or changing a version-neutral helper requires the frozen V2 canonical, decision, failure, and replay suite to pass byte-for-byte. ## Architecture ### Existing Architecture Analysis The existing `src/openultrasast/core/` package already contains deterministic serialization, budget derivation, coverage buckets, digest verification, fact-pack loading, admission manifest refusal, probes, and consumer records. Its public `TraceRecord` requires entry, operation, resource, authority, guard, and witness fields, while `Judgment` is exactly `unguarded`, `discharged`, or `indeterminate`. Tests pin those fields and values. The authorization producer converts directly from `semantic/obligations/trace.py`, and the benchmark admission schema scores authorization judgments. A zero-shot DSPy experiment exists in `benchmarks/valid/judge_pass.py`, but no core model handoff exists; one current test explicitly prohibits a public model path. Production push and proof paths do not yet consume the core end to end. The migration therefore preserves reusable mechanisms while replacing the public domain model. The implemented V2 admission root is intentionally retained but structurally limited: one arm owns one baseline, every metric rule receives the complete claimed-scope tuple, feasibility pools capacity across those scopes, and one unscoped metric set produces one decision. Exact per-arm/per-family/per-ecosystem qualification therefore requires an additive V3 aggregate graph rather than an in-place V2 extension. ### Architecture Pattern and Boundary Map The selected pattern is a **versioned ports-and-adapters contract with an explicit, deletable compatibility namespace**. Core contracts and pure validators form the domain. External systems implement adapters and retain execution ownership. The migration boundary is a module boundary, not a semantic-use analysis over Python execution. ```mermaid flowchart LR subgraph Producers[Evidence producers] Analyzer[Deterministic analyzer] Packs[Semantic fact packs] end subgraph Core[Detection core v2] Semantics[Semantic evidence] Family[Family contract] Readiness[Readiness validation] Decision[Decision and verdict] Admission[Measurement and admission] end subgraph Adapters[External adapters] ModelAdapter[Model adapter] ProofAdapter[Proof adapter] LegacyAdapter[Legacy adapter] end subgraph Compatibility[Temporary compatibility] LegacyV1[Explicit legacy v1 namespace] LegacyRead[Read only decoder] end subgraph Consumers[Consumers] Benchmark[Benchmark workflow] Push[Pre push consumer] end Analyzer --> Semantics Packs --> Semantics Semantics --> Family Family --> Readiness Family --> Decision Readiness --> Admission Decision --> Admission Semantics --> ModelAdapter ModelAdapter --> Decision ProofAdapter --> Decision LegacyAdapter --> Semantics LegacyV1 --> LegacyAdapter LegacyRead --> Benchmark Admission --> Benchmark Decision --> Push ``` **Dependency direction:** ```text canonical → semantics → provenance → evidence → family, coverage, budget → factpack, model → validation → readiness, verdict, probes → measurement → admission → consumer ``` Each core layer imports only layers to its left. `legacy_v1` imports only the canonical primitive needed for byte stability and is never imported by a v2 module. `legacy.py` validates historical bytes without importing `legacy_v1` or constructing v2 types. External adapters import the public core package and their respective execution subsystem. **Architecture integration:** - Existing frozen dataclasses, enums, SHA-256 identities, and canonical JSON patterns are retained and strengthened. - Family-specific complex structure lives in ordered transitions and observations; family payload fields remain shallow, closed, and typed. - Benchmark workflows produce and persist readiness artifacts; core validates their state and freshness. - The model and proof paths derive decisions tied to an unchanged deterministic packet digest; neither can rewrite analyzer evidence. - Historical v1 readers remain validation-only after migration; all new writes use v2. - Before consumer migration, every v1 type and write path is re-parented to `legacy_v1`. The old top-level names remain temporary explicit shims, while mixed v2 modules cease to define legacy symbols. This makes later zero checks syntactic and makes final deletion a proof of capability removal. ### Technology Stack | Layer | Choice and Version | Role | Notes | |---|---|---|---| | Core contracts | Python 3.11 or newer, frozen dataclasses, enums, protocols | Typed immutable domain and validation | Standard library only; no unrestricted `Any` in public contracts | | Serialization | UTF-8 JSON and SHA-256 | Canonical bytes, record IDs, baseline identities | Sorted keys, compact separators, sequence order retained, non-finite numbers refused | | Adjacent model runtime | Existing optional DSPy `>=3.4,<3.5` | Hunt-owned execution of family-declared residual questions | Never imported by core; results cross `hunt/model_contract.py` | | Artifact persistence | Existing benchmark JSON artifacts and stores | Persist readiness, development, and admission records | Core defines records but performs no I/O orchestration | | Tests | pytest 8 or newer | Contract, adapter, migration, and readiness validation | Two unlike test-only reference families | No new runtime dependency is introduced. ## File Structure Plan ### Directory Structure ```text src/openultrasast/core/ ├── __init__.py # Public v2 exports only after migration ├── legacy_v1/ # Temporary closed compatibility namespace; deleted at cutover │ ├── __init__.py # Explicit v1 contract exports; no dynamic discovery │ ├── contracts.py # Authorization-shaped v1 values moved out of mixed v2 modules │ └── writer.py # Only live v1 serialization capability during migration ├── canonical.py # Canonical bytes, identities, content references and read-only resolver port ├── semantics.py # Locations, nodes, transitions, observations, completeness ├── evidence.py # Scope, packet, decision, evidence record, neutral enums ├── family.py # Family contract, catalog, field and decision validation ├── coverage.py # Semantic stage funnel, relation buckets, instrument gate ├── budget.py # Change complexity, workload derivation and elapsed-time gate ├── provenance.py # Input/decision identities, lineage and population-use receipts ├── factpack.py # Pack review, structural match validation and fact application ├── model.py # Dependency-free model request, identity and result contracts ├── validation.py # Closed evidence context and full record revalidation seam ├── readiness.py # Ordered stages, receipts, freshness and promotion rules ├── verdict.py # Claims, satisfaction, silence, proof attachment and aggregation ├── probes.py # Reviewed semantic-equivalence probes and fitted-component evidence gates ├── measurement.py # Complete baseline identity, rates, funnel and report schema v2 ├── admission_case.py # Case results and authenticated post-spend release bindings ├── admission_metrics.py # Evidence-derived scoped and whole-population metric algebra ├── admission.py # Census, feasibility and fitted-component gate ├── admission_report.py # Evidence-backed gates, report assembly and replay validation ├── consumer.py # Scoped push request and verdict consumption └── legacy.py # Read-only v1 trace and report decoder and validator src/openultrasast/semantic/obligations/ └── core_adapter.py # Authorization trace to neutral v2 evidence src/openultrasast/hunt/ └── model_contract.py # Translate injected executor I/O; no DSPy execution ownership src/openultrasast/search/ └── core_adapter.py # Demonstrated verification record to ProofAttachment benchmarks/valid/ ├── readiness_contract.py # Construct and validate receipts; caller owns persistence/sequence ├── development_runner.py # Produce v2 development observations ├── admission_scoring.py # Delegate neutral scoring to core admission ├── admission_run.py # Consume v2 manifest and readiness prerequisites ├── run_manifests.py # Supply expanded v2 identities and lineage └── population_custody.py # Sole append-only writer for core-built use/spend receipts tests/fixtures/detection_core/ ├── obligation_family.json # Test-only control and constraint family contract ├── taint_family.json # Test-only flow and sanitization family contract ├── obligation_cases.json # Positive, negative and missing-evidence triplet ├── taint_cases.json # Positive, negative and missing-evidence triplet └── v1/ ├── trace.json # Byte-frozen historical trace ├── report.json # Byte-frozen historical report └── migration_inventory.json # Frozen registered v1 readers and writers tests/ ├── test_core_canonical.py ├── test_core_evidence.py ├── test_core_family.py ├── test_core_coverage.py ├── test_core_budget.py ├── test_core_readiness.py ├── test_core_model.py ├── test_core_validation.py ├── test_core_verdict.py ├── test_core_probes.py ├── test_core_factpack.py ├── test_core_provenance.py ├── test_core_measurement.py ├── test_core_admission.py ├── test_core_consumer.py ├── test_core_migration.py └── core_migration_boundary.py # Syntax-only import/writer and inventory-closure checks ``` Files below `hunt/`, `search/`, `semantic/`, and `benchmarks/` are adjacent-owner conformance edits, not ownership transfers. `model_contract.py` translates an already executed provider result and never selects or runs DSPy. `readiness_contract.py` constructs and validates canonical records while the benchmark workflow remains the sole owner of sequencing and persistence. The search and semantic adapters translate typed records but do not execute proof or family analysis. ### Modified and Removed Files - `src/openultrasast/core/trace.py` and `_legacy_factpack.py` — temporary v1 import shims during migration; removed after the syntactic legacy boundary reaches zero. They re-export from `legacy_v1` and own no contract or serializer. Historical decoding remains in `legacy.py`. - `src/openultrasast/core/verdict.py`, `consumer.py`, and `admission.py` — temporary explicit shims over `legacy_v1`; at cutover their v1 contents disappear and the public v2 modules/facades take these canonical paths atomically. - `src/openultrasast/core/coverage.py`, `budget.py`, `provenance.py`, `probes.py`, and `factpack.py` — retain only v2/shared definitions after legacy definitions move to `legacy_v1`; they do not re-export v1 names. - `src/openultrasast/core/__init__.py` — preserves the old named API by importing from `legacy_v1` during migration, then atomically replaces it with the public v2 surface. - `src/openultrasast/semantic/obligations/trace.py` — removes imports and conversion logic for core types; remains the authorization-family trace domain. - `src/openultrasast/hunt/prover.py` — sends authorization results through `semantic/obligations/core_adapter.py`; it does not define neutral core types. - Existing `tests/test_core_*.py` files — authorization assumptions move to adapter or legacy tests; neutral tests use the two reference contracts. - `tests/test_core_contract.py` — replaces the assertion that no model path exists with the assertion that core has no DSPy import and only exposes dependency-free model contracts. - `tests/core_migration_boundary.py` — parses import and call syntax without evaluating aliases; it rejects `legacy_v1`, temporary shim, named top-level v1, dynamic core access, and v1-writer references according to the migration phase. - `tests/fixtures/detection_core/v1/migration_inventory.json` — immutable inventory derived from the reviewed Task 1.1 census. Every entry must become migrated or retired; the inventory cannot grow to make a test pass. Every new module remains within the project cap of 400 logical lines; functions remain at or below 50 logical lines and cyclomatic complexity 10. ## System Flows ### Deterministic Evidence, Model Residual, and Proof ```mermaid sequenceDiagram participant Analyzer as Analyzer adapter participant Core as Detection core participant Family as Family contract participant Model as DSPy adapter participant Proof as Proof adapter participant Consumer as Push consumer Analyzer->>Core: Evidence packet and deterministic decision Core->>Family: Validate scope roles fields and prerequisites Family-->>Core: Valid record or indeterminate reasons alt Declared residual is ready Core->>Model: Model request with packet digest Model-->>Core: Model result with identities Core->>Family: Validate decision mapping end alt Demonstrated proof exists Proof-->>Core: Differential record control and verifier receipt Core->>Core: Resolve artifacts and validate family location and packet identity end Core-->>Consumer: Defect satisfaction or bounded silence ``` The deterministic packet is immutable. A model result creates a new decision referencing the packet digest. A proof attachment may independently establish execution-backed basis for a matching violation; it does not mark missing static relations complete. ### Readiness Promotion ```mermaid flowchart LR Instrument[Instrument] --> Recognition[Recognition] Recognition --> Transitions[Transitions] Transitions --> Constraints[Security object and constraints] Constraints --> Pairs[Vulnerable fixed pairs] Pairs --> Development[Development population] Development --> Admission[Fresh admission] Admission --> Integration[Consumer integration] Instrument --> Stopped[Stopped] Recognition --> Stopped Transitions --> Stopped Constraints --> Stopped Pairs --> Stopped Development --> Stopped Admission --> Stopped ``` Every forward edge requires a current `passed` receipt. A failed, absent, or digest-stale predecessor routes to `Stopped` with the failing layer and correction lever. Historical receipts are immutable; staleness is derived against current input identities. ### Admission Sequence ```mermaid sequenceDiagram participant Benchmark as Benchmark workflow participant Core as Detection core participant Arm as Frozen arm participant Custodian as Population custodian Benchmark->>Core: Reviewed manifest without exposed labels Core->>Core: Validate catalog context lineage readiness denominator and feasibility Core-->>Benchmark: Authorization plus head-bound label-release request Benchmark->>Custodian: Compare-and-swap append AVAILABLE to SPENT Custodian-->>Benchmark: Durable spend receipt then release labels Benchmark->>Arm: Run frozen analyzer facts and model Arm-->>Benchmark: Immutable case results Benchmark->>Custodian: Submit immutable case-result identities Custodian-->>Benchmark: Attested case release bound to spend receipt Benchmark->>Core: Preparation receipt case release results and instrument receipts Core->>Core: Revalidate release bindings canonical inputs and scoped scores Core-->>Benchmark: Report bound to authorization spend receipt and case release ``` Core never creates, edits, labels, or persists the admission population. The benchmark custodian performs durable append and label release as one guarded operation: a crash before the append releases no labels, while a crash after it leaves the population spent. Concurrent attempts race on the same custody head, and only the compare-and-swap winner receives label bytes. The case release is constructed only after that spend. It discloses the per-case role and pair mapping that the aggregate census deliberately withholds, and it binds those labels to the exact immutable case-result and evidence/input identities submitted for scoring. ## Requirements Traceability | Requirement Criteria | Summary | Components and Interfaces | Flows | |---|---|---|---| | 1.1, 1.2, 1.3, 1.4 | Canonical family-neutral evidence | Canonical Codec, Semantic Evidence, Evidence Aggregate, Family Contract | Deterministic evidence flow | | 2.1, 2.2, 2.3, 2.4, 2.5 | Neutral outcome, bounded scope and basis | Evidence Aggregate, Family Contract, Verdict | Evidence and decision validation | | 3.1, 3.2, 3.3, 3.4, 3.5 | Family conformance and unlike reference shapes | Family Contract, Family Catalog, reference fixtures | Family registration and conformance | | 4.1, 4.2, 4.3, 4.4 | Typed external semantic relations | Semantic Evidence, Fact Pack | Analyzer and pack ingestion | | 5.1, 5.2, 5.3, 5.4, 5.5 | Stage and instrument coverage | Coverage Funnel, Canonical evidence receipts | Readiness promotion | | 6.1, 6.2, 6.3, 6.4 | Measured budget and immutable change scope | Budget, Analyzed Scope, Consumer | Push consumption | | 7.1, 7.2, 7.3, 7.4 | Defect, satisfaction and bounded silence | Verdict, Evidence Aggregate, Consumer | Decision aggregation | | 8.1, 8.2, 8.3, 8.4, 8.5, 8.6 | Ordered readiness, fixtures and staleness | Readiness Validator, benchmark readiness contract | Readiness promotion | | 9.1, 9.2, 9.3, 9.4, 9.5, 9.6 | Bounded model and proof roles | Model Handoff, Family Contract, Verdict, Proof Attachment | Model residual and proof flow | | 10.1, 10.2, 10.3, 10.4 | Complete reproducibility identities | Provenance and Lineage, Fact Pack, Model Handoff | All record flows | | 11.1, 11.2, 11.3, 11.4, 11.5, 11.6 | Reviewed semantic packs and supported scope | Fact Pack, Semantic Evidence, Measurement | Pack ingestion and reports | | 12.1, 12.2, 12.3, 12.4, 12.5, 12.6 | Admission prerequisites, authenticated case release and scoped scoring | Admission, Readiness, Measurement | Admission sequence | | 13.1, 13.2, 13.3, 13.4 | Development and evaluation separation | Provenance and Lineage, Probes, Admission | Freeze and admission validation | | 14.1, 14.2, 14.3, 14.4, 14.5 | Shared consumer and family-spec contract | Consumer, Verdict, Family Capability Declaration, Legacy Compatibility Namespace, Legacy Decoder | Push and migration flows | | 15.1, 15.2, 15.3, 15.4, 15.5 | Fixed versioned measurement and baseline rules | Measurement, Coverage, Readiness | Development and admission reporting | ## Components and Interfaces | Component | Domain | Intent | Requirement Coverage | Key Dependencies | Contracts | |---|---|---|---|---|---| | Canonical Codec | Primitive | Produce deterministic bytes and identities | 1.4, 10.4 | Standard library | Service | | Semantic Evidence | Evidence | Represent typed relations and observations | 1.2, 4.1–4.4, 5.2–5.3 | Canonical Codec | State | | Evidence Aggregate | Evidence | Bind scope, deterministic packet and decision | 1.1–1.4, 2.1–2.5, 10.1 | Semantic Evidence, Provenance | State, Service | | Family Contract and Catalog | Family | Declare, catalog, resolve, and validate family semantics | 1.3, 2.1–2.4, 3.1–3.5, 9.1, 14.5 | Evidence Aggregate | State, Service | | Coverage Funnel | Operations | Account for every semantic stage and input | 5.1–5.5 | Semantic Evidence | State, Service | | Budget and Scope | Operations | Validate changed scope and measured deadline | 6.1–6.4 | Evidence Aggregate | State, Service | | Provenance and Lineage | Governance | Bind all inputs and label ancestry | 10.1–10.4, 13.1, 13.3–13.4 | Canonical Codec | State, Service | | Fact Pack | Extension | Validate reviewed semantic knowledge | 4.1–4.4, 11.1–11.6 | Family Contract, Provenance | State, Service | | Model Handoff | Extension | Validate dependency-free model requests/results | 9.1–9.5, 10.3, 13.1 | Evidence, Family, Provenance | Service, State | | Evidence Validation | Composition | Revalidate provenance, family, facts, and model derivation before use | 4.4, 10.1–10.4, 12.6, 14.1 | Family, Fact Pack, Model, Provenance | Service, State | | Readiness Validator | Governance | Gate ordered stage promotion and freshness | 8.1–8.6, 12.1, 15.5 | Coverage, Provenance | Service, State | | Verdict and Proof | Decision | Produce bounded outputs and attach proof | 7.1–7.4, 9.4, 9.6, 14.2, 14.4 | Evidence, Model | Service, State | | Probes | Governance | Test semantic shortcut resistance | 13.2 | Evidence, Family, Fact Pack review policy | Service | | Measurement | Evaluation | Create baseline identity and fixed report schema | 11.6, 15.1–15.5 | Coverage, Readiness | Service, State | | Admission | Evaluation | Validate manifests, feasibility, post-spend case release and scoped scores | 12.1–12.6, 13.4 | Readiness, Measurement, Evidence Validation, Verdict, Probes, Budget, Provenance | Service, Batch | | Consumer | Edge | Validate push requests and return bounded verdicts | 6.4, 14.1–14.5 | Budget, Evidence Validation, Verdict, Provenance, Canonical resolver | Service | | Legacy Compatibility Namespace | Migration | Isolate every live v1 contract and writer behind a deletable boundary | 14.3, 15.3 | Canonical Codec, frozen migration inventory | State, Service | | Legacy Decoder | Migration | Read and validate historical v1 bytes only | 14.3, 15.3 | Canonical Codec | Service | | External Adapters | Integration | Translate authorization, DSPy and proof records | 3.5, 9.1–9.6, 14.2–14.3 | Public core | Service | ### Evidence Domain #### Canonical Codec and Semantic Evidence | Field | Detail | |---|---| | Intent | Define the only v2 serialization profile and family-neutral semantic vocabulary | | Requirements | 1.2, 1.4, 4.1, 4.2, 4.3, 4.4, 5.2, 10.4 | **Responsibilities and constraints** - Accept only JSON-compatible scalar values, tuples, and string-keyed mappings. - Refuse non-finite numbers, unknown enums, duplicate logical identities, and unsupported values before hashing. - Sort mapping keys, preserve tuple order, use compact separators, and include schema version in every digestible aggregate. - Give every relation and observation a stable identity, role, source location, completeness, provenance, and optional semantic fact identity. - Represent registration, invocation, call, return, value transformation, operation effect, and constraint effect as a closed `RelationKind`. **Contracts:** Service, State. ```python JsonScalar = str | int | float | bool | None JsonValue = JsonScalar | tuple["JsonValue", ...] | Mapping[str, "JsonValue"] def canonical_bytes(value: JsonValue) -> bytes: ... def content_digest(value: JsonValue) -> str: ... class FieldKind(Enum): TEXT = "text" INTEGER = "integer" NUMBER = "number" BOOLEAN = "boolean" TEXT_TUPLE = "text_tuple" SOURCE_LOCATION = "source_location" @dataclass(frozen=True) class FieldValue: name: str kind: FieldKind value: JsonValue @dataclass(frozen=True) class ContentRef: artifact_id: str content_digest: str media_type: str byte_length: int retention_policy_digest: str storage_receipt_digest: str @dataclass(frozen=True) class StorageReceipt: store_identity: str content_digest: str byte_length: int retention_policy_digest: str receipt_digest: str @dataclass(frozen=True) class ResolvedContent: content: bytes receipt: StorageReceipt class ContentResolver(Protocol): def resolve(self, reference: ContentRef) -> ResolvedContent: ... @dataclass(frozen=True) class SourceLocation: path: str line: int column: int | None symbol: str | None @dataclass(frozen=True) class SemanticNode: node_id: str role: str location: SourceLocation class RelationKind(Enum): REGISTRATION = "registration" INVOCATION = "invocation" CALL = "call" RETURN = "return" VALUE_TRANSFORMATION = "value_transformation" OPERATION_EFFECT = "operation_effect" CONSTRAINT_EFFECT = "constraint_effect" class RelationStatus(Enum): COMPLETE = "complete" INCOMPLETE = "incomplete" UNRESOLVED = "unresolved" class ProvenanceKind(Enum): PRODUCT_CODE = "product_code" SEMANTIC_FACT = "semantic_fact" @dataclass(frozen=True) class EvidenceProvenance: kind: ProvenanceKind source_content_digest: str semantic_pack_digest: str | None semantic_fact_id: str | None fact_match: ContentRef | None @dataclass(frozen=True) class SemanticRelation: relation_id: str kind: RelationKind role: str source: SemanticNode target: SemanticNode location: SourceLocation status: RelationStatus conditions: tuple[str, ...] reasons: tuple[str, ...] provenance: EvidenceProvenance @dataclass(frozen=True) class Observation: observation_id: str role: str location: SourceLocation fields: tuple[FieldValue, ...] status: RelationStatus provenance: EvidenceProvenance ``` An unmet declared condition yields `INCOMPLETE`; conflicting complete candidates yield an `UNRESOLVED` ambiguity relation. Neither path silently selects a winner. Semantic-fact provenance requires the exact pack digest, stable fact ID, and resolved match reference. Product-code provenance forbids all three fields, preventing an unattributed relation from changing origin later. #### Evidence Aggregate | Field | Detail | |---|---| | Intent | Keep deterministic evidence immutable while decisions derive from its digest | | Requirements | 1.1–1.4, 2.1–2.5, 6.4, 7.4, 10.1–10.4 | **Responsibilities and constraints** - `EvidencePacket` owns the analyzer-produced data and never contains model text or proof execution output. - `Decision` owns the family decision, neutral outcome, evidence basis, bounded scope, and named indeterminate reasons. Its separate `DecisionProvenance` owns any model-result or proof-record identity; derived decisions never add those identities to the packet. - `EvidenceRecord` binds one decision to one packet digest. Derivation creates a new record; it does not mutate the packet. - `SATISFIED` requires an explicit bounded scope and complete family-required relations. - `INDETERMINATE` requires at least one named reason. - Deterministic basis forbids model/proof references; model-corroborated basis requires the complete model call and result identities; execution-backed basis requires the proof record and reviewed producer identities. Core validates these as a closed union. **Contracts:** Service, State. ```python class NeutralOutcome(Enum): VIOLATION = "violation" SATISFIED = "satisfied" INDETERMINATE = "indeterminate" class EvidenceBasis(Enum): DETERMINISTIC = "deterministic" MODEL_CORROBORATED = "model_corroborated" EXECUTION_BACKED = "execution_backed" @dataclass(frozen=True) class EvidencePacket: schema_version: str family_id: str family_contract_digest: str security_object_id: str ecosystem: str scope: AnalyzedScope transitions: tuple[SemanticRelation, ...] observations: tuple[Observation, ...] family_fields: tuple[FieldValue, ...] completeness_reasons: tuple[str, ...] provenance: ProvenanceStamp @dataclass(frozen=True) class Decision: family_code: str outcome: NeutralOutcome basis: EvidenceBasis bounded_scope: AnalyzedScope reasons: tuple[str, ...] packet_digest: str provenance: DecisionProvenance @dataclass(frozen=True) class EvidenceRecord: packet: EvidencePacket decision: Decision record_id: str ``` ### Family Domain #### Family Contract and Catalog | Field | Detail | |---|---| | Intent | Declare family semantics as immutable data and validate records without importing family algorithms | | Requirements | 1.3, 2.1–2.4, 3.1–3.5, 7.2, 9.1, 14.5 | **Responsibilities and constraints** - Declare family and schema versions, ecosystems, security-object kind and fields, observation roles and fields, decisions, neutral mappings, required roles and relation statuses, named indeterminate reasons, and residual questions. - Use a closed `FieldKind` vocabulary for text, integer, finite number, boolean, text tuple, and source location values. - Keep `FamilyCatalog` immutable; constructing it re-derives every contract digest, requires canonical order, and refuses duplicate family/version/schema identities or digests. - Refuse unsupported ecosystems, unknown roles/fields/decisions, undeclared reasons, and determinate outcomes whose required evidence is incomplete. - Declare, per role, whether product-code relations are sufficient or excluded-dependency behavior requires a reviewed semantic fact. A missing reviewed dependency relation is incomplete evidence, never an implicit product-code relation. - Validate test-only obligation and taint-flow contracts with the same code used by real family adapters. **Contracts:** Service, State. ```python @dataclass(frozen=True) class FieldSpec: name: str kind: FieldKind required: bool @dataclass(frozen=True) class ObjectSpec: kind: str fields: tuple[FieldSpec, ...] identity_fields: tuple[str, ...] class RelationOriginPolicy(Enum): PRODUCT_OR_REVIEWED_FACT = "product_or_reviewed_fact" EXCLUDED_DEPENDENCY_FACT_ONLY = "excluded_dependency_fact_only" @dataclass(frozen=True) class RoleSpec: role: str fields: tuple[FieldSpec, ...] allowed_relation_kinds: tuple[RelationKind, ...] origin_policy: RelationOriginPolicy @dataclass(frozen=True) class DecisionRule: family_code: str outcome: NeutralOutcome required_roles: tuple[str, ...] allowed_statuses: tuple[RelationStatus, ...] @dataclass(frozen=True) class ResidualQuestion: question_id: str required_roles: tuple[str, ...] allowed_decisions: tuple[str, ...] @dataclass(frozen=True) class FamilyContract: family_id: str family_version: str schema_version: str ecosystems: tuple[str, ...] security_object: ObjectSpec roles: tuple[RoleSpec, ...] decisions: tuple[DecisionRule, ...] residual_questions: tuple[ResidualQuestion, ...] indeterminate_reasons: tuple[str, ...] @dataclass(frozen=True) class FamilyCatalogEntry: contract: FamilyContract contract_digest: str @dataclass(frozen=True) class FamilyCatalog: entries: tuple[FamilyCatalogEntry, ...] catalog_digest: str @dataclass(frozen=True) class FamilyCapabilityDeclaration: arm_id: str family_contract_digest: str readiness_fixture_digest: str development_population_digest: str admission_boundary_digest: str def validate_catalog(catalog: FamilyCatalog) -> None: ... def resolve_contract( catalog: FamilyCatalog, family_id: str, contract_digest: str, ecosystem: str, ) -> FamilyContract: ... def validate_record(contract: FamilyContract, record: EvidenceRecord) -> None: ... def validate_capability( declaration: FamilyCapabilityDeclaration, catalog: FamilyCatalog, ) -> None: ... ``` `FamilyCapabilityDeclaration` references the family contract, readiness-fixture digest, development boundary, and admission boundary. It is a machine-validated artifact for family specs; the core does not parse Markdown specifications. `resolve_contract` requires exactly one catalog entry matching family, digest, and ecosystem; it never falls back to a different version. For a role declared `EXCLUDED_DEPENDENCY_FACT_ONLY`, `validate_record` refuses a `COMPLETE` product-code relation. The relation can become complete only through `SEMANTIC_FACT` provenance whose pack, fact, and structural match are revalidated. If that knowledge is absent, the adapter records `INCOMPLETE` with `missing_reviewed_dependency_relation`; a determinate rule that requires the role is refused. This is the explicit vendored-source boundary required by criterion 4.4. ### Extension Contracts #### Model Handoff | Field | Detail | |---|---| | Intent | Make the evidence-to-DSPy seam reproducible without importing DSPy into core | | Requirements | 9.1–9.6, 10.3, 13.1 | **Responsibilities and constraints** - Build requests only for residual questions declared by the matching family contract. - Present the canonical evidence record, family contract, residual-question definition, and per-role completeness snapshot in the request. Their redundant digests are validated before dispatch so an adapter cannot substitute content behind an identity. - Require provider, model, program, prompt, configuration, and adapter identities on every result, including failures. - Freeze those identities, the decision mapping, optional examples/fitted component, and their actual lineage manifest in a pre-execution `ModelProgramDeclaration`; a digest-only lineage assertion is not sufficient. - Reject a result for a different packet, family, question, or decision mapping. - Reject the same program identity as proposer and adjudicator on one claim path. - Preserve the normalized response plus either redacted inline response bytes or a content-addressed immutable response artifact. Preserve model failure as coverage and measurement evidence; never convert it to satisfaction or silence without a reason. - Permit model-corroborated outcomes only when every prerequisite of the residual question is complete. Model output can never set `EXECUTION_BACKED`. **Contracts:** Service, State. ```python class ModelStatus(Enum): SUCCEEDED = "succeeded" FAILED = "failed" @dataclass(frozen=True) class CompletenessItem: role: str status: RelationStatus reasons: tuple[str, ...] @dataclass(frozen=True) class ModelProgramDeclaration: family_contract_digest: str provider: str model: str program_digest: str prompt_digest: str example_set_digest: str | None fitted_component_digest: str | None configuration_digest: str adapter_revision: str decision_mapping_digest: str input_bundle_digest: str lineage_manifest_digest: str declaration_digest: str @dataclass(frozen=True) class ModelRequest: request_id: str record: EvidenceRecord contract: FamilyContract residual_question: ResidualQuestion completeness: tuple[CompletenessItem, ...] packet_digest: str family_contract_digest: str residual_question_id: str completeness_digest: str program: ModelProgramDeclaration lineage: LineageManifest custody_policy_digest: str proposal_program_digests: tuple[str, ...] trusted_response_store_identities: tuple[str, ...] maximum_response_bytes: int @dataclass(frozen=True) class PreservedModelResponse: schema_version: str normalized_fields: tuple[FieldValue, ...] inline_redacted_text: str | None artifact: ContentRef | None content_digest: str redaction_policy_digest: str @dataclass(frozen=True) class ModelResult: request_digest: str packet_digest: str identity: ModelCallIdentity status: ModelStatus response: PreservedModelResponse | None family_decision: str | None decision_mapping_digest: str | None failure_reason: str | None result_digest: str def apply_model_result( record: EvidenceRecord, contract: FamilyContract, request: ModelRequest, result: ModelResult, resolver: ContentResolver, custody: PopulationCustodyPort, custody_policy: PopulationCustodyPolicy, ) -> EvidenceRecord: ... ``` `PreservedModelResponse` requires exactly one of `inline_redacted_text` or `artifact`. The core re-derives the inline digest; for an artifact it requires `content_digest` to equal the reference digest and validates the storage-receipt, trusted-store, byte-limit, and retention-policy identities declared by the request. A `ContentRef` names an opaque artifact ID, media type, byte length, retention-policy digest, storage-receipt digest, and full SHA-256 digest; the adapter or benchmark store owns and verifies the bytes. The normalized fields used for the decision are always inline and canonical. This retains the response used for adjudication without making provider transport data or credentials part of the core. A successful result requires the response, family decision, and mapping digest; a failed result requires a named failure and may omit response fields only when no response was received. Before accepting either outcome, `apply_model_result` re-derives the program declaration, input-bundle, request, and lineage digests; requires the declared custody policy; and calls `validate_lineage` against the record population and the current complete custody chains. The caller-specific check maps `PROMPT`, `EXAMPLE`, `FITTED_MODEL`, and `DECISION_MAPPING` uses to required `LineageExpectation` values for the corresponding non-null declared artifact digests, with `input_bundle_digest` as the manifest subject. Every use must match the purpose, source population, influenced-artifact digest, and receipt embedded in the current chain. Omitted history, an old prefix, or use of the evaluated admission population's labels is refused. Result provider/model/program/prompt/configuration/adapter and decision-mapping identities must equal the frozen declaration. The derived `DecisionProvenance` retains both the declaration and validated lineage-manifest digests. The adjacent hunt subsystem owns prompt text, DSPy signatures, execution, caching, storage, and policy-driven redaction. `model_contract.py` only maps its structured input/output to the core records. Core validation treats the result as untrusted input and checks the response, decision-mapping, request, and packet identities. `apply_model_result` returns a new decision whose `DecisionProvenance` contains the complete `ModelCallIdentity` and result digest; the deterministic packet remains byte-identical. #### Fact Pack The v2 `FactPack` carries schema version, family-contract digest, family, ecosystem, language scope, stable fact IDs, declared roles, structural requirements, relation effects, normalized content digest, and review identity. A match becomes a complete semantic relation only when the adapter supplies every required structural match item. A bare name match can at most produce incomplete evidence. Pack insufficiency preserves partial records and named unresolved reasons. ```python class KnowledgeEvidenceKind(Enum): SOURCE_CODE = "source_code" OFFICIAL_DOCUMENTATION = "official_documentation" INDEPENDENT_BEHAVIOR = "independent_behavior" @dataclass(frozen=True) class IndependenceAttestation: subject_digest: str producer_identity: str reviewer_identity: str conflict_disclosure: ContentRef attestation_digest: str @dataclass(frozen=True) class ReviewIndependencePolicy: reviewed_reviewer_digests: tuple[str, ...] forbidden_identity_pairs: tuple[tuple[str, str], ...] trusted_store_identities: tuple[str, ...] maximum_evidence_bytes: int policy_digest: str @dataclass(frozen=True) class KnowledgeEvidence: kind: KnowledgeEvidenceKind content: ContentRef producer_identity: str reviewer_identity: str independence: IndependenceAttestation lineage_manifest_digest: str class StructuralRequirementKind(Enum): QUALIFIED_NAME = "qualified_name" AST_SHAPE = "ast_shape" RECEIVER_TYPE = "receiver_type" CALL_TARGET = "call_target" ARGUMENT_SHAPE = "argument_shape" CONFIGURATION = "configuration" @dataclass(frozen=True) class StructuralRequirement: requirement_id: str kind: StructuralRequirementKind field: str expected: JsonValue @dataclass(frozen=True) class RelationEffect: kind: RelationKind role: str source_role: str target_role: str conditions: tuple[str, ...] @dataclass(frozen=True) class StructuralObservation: requirement_id: str location: SourceLocation observed: JsonValue evidence: ContentRef @dataclass(frozen=True) class FactMatchEvidence: pack_digest: str fact_id: str product_location: SourceLocation observations: tuple[StructuralObservation, ...] matcher_revision: str matcher_behavior_digest: str matcher_lineage_manifest_digest: str match_digest: str @dataclass(frozen=True) class SemanticFact: fact_id: str role: str structural_requirements: tuple[StructuralRequirement, ...] relation_effects: tuple[RelationEffect, ...] @dataclass(frozen=True) class FactPack: schema_version: str family_contract_digest: str family_id: str ecosystem: str language: str facts: tuple[SemanticFact, ...] normalized_content_digest: str review_identity: str review_evidence: tuple[KnowledgeEvidence, ...] review_evidence_digest: str review_policy_digest: str def validate_fact_pack( pack: FactPack, contract: FamilyContract, protected_population_digest: str | None, lineages: Sequence[LineageManifest], custody: PopulationCustodyPort, custody_policy: PopulationCustodyPolicy, review_policy: ReviewIndependencePolicy, resolver: ContentResolver, ) -> None: ... def apply_fact( pack: FactPack, fact_id: str, match: FactMatchEvidence, match_reference: ContentRef, contract: FamilyContract, evaluation_population_digest: str, lineages: Sequence[LineageManifest], custody: PopulationCustodyPort, custody_policy: PopulationCustodyPolicy, review_policy: ReviewIndependencePolicy, resolver: ContentResolver, ) -> tuple[SemanticRelation, ...]: ... def validate_fact_relations( record: EvidenceRecord, packs: Sequence[FactPack], lineages: Sequence[LineageManifest], custody: PopulationCustodyPort, custody_policy: PopulationCustodyPolicy, review_policy: ReviewIndependencePolicy, resolver: ContentResolver, ) -> None: ... ``` The normalized digest covers schema, scope, roles, every structural predicate, every effect, and the review evidence. Validation refuses unknown fields and roles, duplicate fact IDs, scope mismatch, missing review evidence, and a digest that cannot be re-derived. Each knowledge item must be source code, official documentation, or independently reviewed behavioral evidence and must reference a label-isolated lineage manifest. For behavioral evidence, producer and reviewer must differ, the reviewer must be admitted by the `ReviewIndependencePolicy`, the attestation subject must equal the content digest, and the conflict disclosure must resolve and hash correctly. Vulnerability, fix-location, or ordinary-change label ancestry spends the population and refuses the pack. `validate_fact_pack` resolves every referenced manifest from the supplied lineage set, requires its subject to equal `normalized_content_digest`, declares a required `SEMANTIC_FACT` expectation and an optional `PROMOTION` expectation for that same digest, sets `forbid_label_access=True`, and validates its uses against current custody chains. `apply_fact` is the only route from a semantic fact to fact-derived relations. It resolves and validates the pack with the supplied review policy and evaluation-population boundary, then resolves each structural observation. It requires the observation IDs to equal the fact's complete requirement-ID set, checks values against the typed predicates, and constructs relations whose provenance contains the pack/fact identities and resolved match reference. Missing or failed predicates produce incomplete evidence; duplicates, fabricated IDs, or ambiguity produce unresolved evidence. A name observation cannot yield `COMPLETE` while any non-name structural requirement is absent. The match reference must resolve to canonical bytes whose digest is `match_digest`; record validation re-runs this join from the named pack and match. The match digest covers the matcher revision, behavior digest, and lineage-manifest digest. `apply_fact` and `validate_fact_relations` resolve that manifest, require subject equality with `matcher_behavior_digest`, supply a required `MATCHING_BEHAVIOR` expectation for that digest, and validate every receipt against the current custody chain while protecting the record's evaluation population. Matching behavior influenced by the admission population's labels therefore spends that population and cannot complete a relation in the same admission run. #### Downstream Fact-Adapter Seam A downstream vulnerability-family adapter owns its production `FamilyContract`, reviewed packs, structural matcher, and persistence of canonical `StructuralObservation` and `FactMatchEvidence` bytes. It supplies those typed objects, their content references, lineages, policies, custody port, and resolver to the core. Core only revalidates the pack, match, provenance, and relation application; it neither imports target modules nor mints missing trust or evidence. A legacy recognition projection may coexist during downstream migration, but it is non-authoritative and cannot construct v2 evidence, readiness, or admission by itself. #### Evidence Validation Composition | Field | Detail | |---|---| | Intent | Give admission and every consumer one non-bypassable composition seam for provenance, family, fact, and model validation | | Requirements | 4.4, 10.1–10.4, 12.6, 14.1, 14.3 | `validation.py` is later than both `factpack.py` and `model.py`; it composes their pure validators without moving family reasoning into core. Raw `EvidenceRecord` values are data, not proof that validation already occurred. Public scoring and consumption APIs accept the input envelope and context below and call `validate_evidence_input` before deriving a claim, satisfaction, silence, metric, or gate. ```python @dataclass(frozen=True) class EvidenceValidationContext: family_catalog: FamilyCatalog fact_packs: tuple[FactPack, ...] model_programs: tuple[ModelProgramDeclaration, ...] lineages: tuple[LineageManifest, ...] fact_review_policy: ReviewIndependencePolicy provenance_trust_policy: ProvenanceTrustPolicy custody_policy: PopulationCustodyPolicy context_digest: str @dataclass(frozen=True) class EvidenceValidationInput: record: EvidenceRecord named_inputs: NamedInputs model_request: ModelRequest | None model_result: ModelResult | None input_digest: str def validate_evidence_input( value: EvidenceValidationInput, context: EvidenceValidationContext, custody: PopulationCustodyPort, resolver: ContentResolver, ) -> EvidenceRecord: ... ``` Validation re-derives the input/context/catalog/pack/model/lineage digests, resolves exactly one contract, validates the seven named provenance inputs, calls `validate_record`, and replays every fact relation through `validate_fact_relations`. A deterministic record must omit model request/result values and model provenance. A model-corroborated record must carry both; its declaration must be an exact context member, `apply_model_result` is rerun from `model_request.record`, and the derived record must equal `value.record` canonically. Execution-backed basis is added only later by the proof validator from an already validated base record. No constructible “validated” marker is accepted as a shortcut. ### Governance and Evaluation #### Changed Scope and Measured Budget | Field | Detail | |---|---| | Intent | Bind every record to the push change and derive its deadline from disclosed measurements | | Requirements | 6.1, 6.2, 6.3, 6.4, 14.1 | **Responsibilities and constraints** - The push runner supplies immutable base/head revisions and normalized changed regions; the core validates containment but does not discover repository scope. - Complexity records the change identity, region count, transition count, and changed byte count alongside total reply time for every sample. - A budget is derived from a nonempty measured workload class and carries sample count, percentile, reference workload, reference hardware, derived duration, and derivation digest. There is no default or static fallback budget. - The elapsed-time gate covers preparation, deterministic analysis, optional model and proof handling, validation, and reporting. Failure names the change, elapsed duration, workload class, and derivation exceeded. **Contracts:** Service, State. `SourceRegion`, `AnalyzedScope`, and `validate_scope` are owned by `evidence.py`, because the evidence packet depends on them. The remaining records and functions in this subsection are owned by `budget.py`, which imports evidence types and never the reverse. ```python @dataclass(frozen=True) class SourceRegion: path: str start_line: int end_line: int @dataclass(frozen=True) class AnalyzedScope: base_revision: str head_revision: str change_digest: str regions: tuple[SourceRegion, ...] @dataclass(frozen=True) class ChangeComplexity: change_id: str workload_class: str changed_regions: int transition_count: int changed_bytes: int measurement: ContentRef @dataclass(frozen=True) class MeasuredSample: complexity: ChangeComplexity total_reply_ms: int @dataclass(frozen=True) class ReferenceWorkload: name: str description: str changed_region_range: tuple[int, int] transition_count_range: tuple[int, int] changed_bytes_range: tuple[int, int] sample_artifact: ContentRef record_digest: str @dataclass(frozen=True) class ReferenceHardware: cpu_model: str logical_cpu_count: int memory_bytes: int operating_system: str runtime_version: str record_digest: str @dataclass(frozen=True) class BudgetDerivation: workload_class: str sample_count: int percentile: float reference_workload: ReferenceWorkload reference_hardware: ReferenceHardware budget_ms: int derivation_digest: str def derive_budget( samples: Sequence[MeasuredSample], workload_class: str, percentile: float, reference_workload: ReferenceWorkload, reference_hardware: ReferenceHardware, ) -> BudgetDerivation: ... def validate_scope(packet: EvidencePacket, allowed: AnalyzedScope) -> None: ... def check_budget( change: ChangeComplexity, total_reply_ms: int, budget: BudgetDerivation, resolver: ContentResolver, ) -> None: ... ``` #### Coverage, Provenance, and Readiness | Field | Detail | |---|---| | Intent | Locate the first failing layer and prevent stale or failed evidence from reaching later gates | | Requirements | 5.1–5.5, 8.1–8.6, 10.1–10.4, 13.1–13.4, 15.5 | **Responsibilities and constraints** - Coverage records per-stage input/output counts for reads, questions, sites considered, sites recognized, transitions, security objects, constraints, determinate outcomes, and findings. - Every considered relation occupies exactly one complete, incomplete, or unresolved bucket. - Zero-byte or unreadable tracked input is an `InstrumentFailure` that invalidates the run. - A zero stage is diagnostic rather than automatically clean or failed; fixture expectations identify the first expected stage that produced zero. - Packet provenance covers analyzer, family contract, fact set, acquisition, configuration, population, change, and semantic facts; decision provenance separately covers model and proof identities. - Label lineage names all populations used for prompts, examples, thresholds, facts, matching behavior, mappings, and promotion. Core derives a canonical `PopulationUseReceipt`; label influence marks the population spent and refuses admission use. - Readiness receipts contain fixture roles, identity snapshots, predecessor digests, measurement digest, result, and correction lever. Staleness is derived without changing the stored receipt. **Contracts:** Service, State. ```python @dataclass(frozen=True) class ProvenanceStamp: analyzer_revision: str family_contract_digest: str fact_set_digest: str acquisition_record_digest: str configuration_digest: str population_digest: str change_input_digest: str @dataclass(frozen=True) class ModelCallIdentity: provider: str model: str program_digest: str prompt_digest: str configuration_digest: str adapter_revision: str @dataclass(frozen=True) class DecisionProvenance: model_identity: ModelCallIdentity | None model_result_digest: str | None model_declaration_digest: str | None proof_record_digest: str | None proof_producer_digest: str | None lineage_manifest_digest: str | None class LineagePurpose(Enum): PROMPT = "prompt" EXAMPLE = "example" THRESHOLD = "threshold" FITTED_MODEL = "fitted_model" SEMANTIC_FACT = "semantic_fact" MATCHING_BEHAVIOR = "matching_behavior" DECISION_MAPPING = "decision_mapping" PROMOTION = "promotion" ADMISSION_LABEL_OPEN = "admission_label_open" @dataclass(frozen=True) class LineageUse: purpose: LineagePurpose source_population_digest: str labels_accessed: bool influenced_artifact_digest: str population_use_receipt_digest: str @dataclass(frozen=True) class LineageManifest: arm_id: str subject_digest: str uses: tuple[LineageUse, ...] manifest_digest: str @dataclass(frozen=True) class LineageExpectation: purpose: LineagePurpose influenced_artifact_digest: str minimum_uses: int maximum_uses: int | None class PopulationStatus(Enum): AVAILABLE = "available" SPENT = "spent" @dataclass(frozen=True) class PopulationUseRequest: population_digest: str arm_id: str purpose: LineagePurpose labels_accessed: bool influenced_artifact_digest: str | None expected_log_digest: str @dataclass(frozen=True) class PopulationUseReceipt: request: PopulationUseRequest request_digest: str population_digest: str custodian_identity: str sequence: int predecessor_digest: str | None prior_status: PopulationStatus status: PopulationStatus reasons: tuple[str, ...] receipt_digest: str @dataclass(frozen=True) class PopulationCustodyPolicy: reviewed_custodian_digests: tuple[str, ...] policy_digest: str @dataclass(frozen=True) class PopulationCustodyLog: population_digest: str custodian_identity: str receipts: tuple[PopulationUseReceipt, ...] receipt_count: int head_receipt_digest: str | None status: PopulationStatus log_digest: str class PopulationCustodyPort(Protocol): def current_log(self, population_digest: str) -> PopulationCustodyLog: ... @dataclass(frozen=True) class LineageDecision: allowed: bool spent_population_digests: tuple[str, ...] reasons: tuple[str, ...] decision_digest: str class ProvenanceInputRole(Enum): ANALYZER = "analyzer" FAMILY_CONTRACT = "family_contract" FACT_SET = "fact_set" ACQUISITION = "acquisition" CONFIGURATION = "configuration" POPULATION = "population" CHANGE_INPUT = "change_input" @dataclass(frozen=True) class ProvenanceTrustPolicy: trusted_store_identities: tuple[str, ...] maximum_input_bytes: int policy_digest: str @dataclass(frozen=True) class NamedInput: role: ProvenanceInputRole identity: str content: ContentRef @dataclass(frozen=True) class NamedInputs: inputs: tuple[NamedInput, ...] manifest_digest: str def validate_provenance( stamp: ProvenanceStamp, named_inputs: NamedInputs, resolver: ContentResolver, trust: ProvenanceTrustPolicy, ) -> None: ... def record_population_use( request: PopulationUseRequest, prior: PopulationCustodyLog, custodian_identity: str, policy: PopulationCustodyPolicy, ) -> PopulationUseReceipt: ... def validate_lineage( lineage: LineageManifest, expected_subject_digest: str, expectations: tuple[LineageExpectation, ...], protected_population_digest: str | None, forbid_label_access: bool, custody: PopulationCustodyPort, policy: PopulationCustodyPolicy, ) -> LineageDecision: ... @dataclass(frozen=True) class InputManifest: tracked_input_ids: tuple[str, ...] manifest_digest: str @dataclass(frozen=True) class InputReadReceipt: input_id: str path: str opened: bool byte_count: int content_digest: str | None @dataclass(frozen=True) class RelationStatusCount: status: RelationStatus count: int @dataclass(frozen=True) class ReasonCount: reason: str count: int @dataclass(frozen=True) class CoverageFunnel: inputs_read: int questions_formed: int sites_considered: int sites_recognized: int transitions_materialized: int security_objects_constructed: int constraints_evaluated: int determinate_outcomes: int findings_emitted: int relation_status_counts: tuple[RelationStatusCount, ...] unresolved_by_reason: tuple[ReasonCount, ...] class CoverageStage(Enum): INPUTS = "inputs" QUESTIONS = "questions" SITES_CONSIDERED = "sites_considered" SITES_RECOGNIZED = "sites_recognized" TRANSITIONS = "transitions" SECURITY_OBJECTS = "security_objects" CONSTRAINTS = "constraints" OUTCOMES = "outcomes" FINDINGS = "findings" @dataclass(frozen=True) class StageExpectation: stage: CoverageStage expected_nonzero_input: bool expected_nonzero_output: bool @dataclass(frozen=True) class StageDiagnosis: last_verified_nonzero_stage: CoverageStage | None first_failed_output_stage: CoverageStage | None reasons: tuple[str, ...] def validate_instrument(manifest: InputManifest, receipts: Sequence[InputReadReceipt]) -> None: ... def diagnose_first_zero(funnel: CoverageFunnel, expected: Sequence[StageExpectation]) -> StageDiagnosis: ... class ReadinessStage(Enum): INSTRUMENT = "instrument" RECOGNITION = "recognition" TRANSITIONS = "transitions" CONSTRAINTS = "constraints" PAIRS = "pairs" DEVELOPMENT = "development" ADMISSION = "admission" INTEGRATION = "integration" class StageResult(Enum): PASSED = "passed" FAILED = "failed" class FixtureRole(Enum): POSITIVE = "positive" NEGATIVE = "negative" AMBIGUITY = "ambiguity" class ReadinessEvidenceUse(Enum): REGRESSION_FIXTURE = "regression_fixture" DEVELOPMENT_POPULATION = "development_population" REPOSITORY_HOLDOUT = "repository_holdout" FRESH_ADMISSION = "fresh_admission" CONSUMER_INTEGRATION = "consumer_integration" @dataclass(frozen=True) class FixtureResult: role: FixtureRole fixture_digest: str passed: bool reasons: tuple[str, ...] @dataclass(frozen=True) class AnalysisIdentity: core_schema_version: str semantic_vocabulary_digest: str analyzer_revision: str family_contract_digest: str fact_set_digest: str acquisition_record_digest: str configuration_digest: str model_program_digest: str | None outcome_mapping_digest: str @dataclass(frozen=True) class ReadinessResult: arm_id: str family_id: str ecosystem: str stage: ReadinessStage evidence_use: ReadinessEvidenceUse population_digest: str | None population_freeze_digest: str | None population_use_receipt_digests: tuple[str, ...] result: StageResult identity: AnalysisIdentity predecessor_digests: tuple[str, ...] fixture_results: tuple[FixtureResult, ...] measurement_digest: str correction_lever: str | None result_digest: str @dataclass(frozen=True) class StageAuthorization: target: ReadinessStage authorized: bool failing_stage: ReadinessStage | None reasons: tuple[str, ...] correction_lever: str | None def authorize_stage( target: ReadinessStage, receipts: Sequence[ReadinessResult], current_identity: AnalysisIdentity, ) -> StageAuthorization: ... ``` The benchmark population custodian is the single writer for population-use receipts: it reads the current log, invokes the pure core constructor with that head, and appends the returned bytes using compare-and-swap on `expected_log_digest` and `predecessor_digest`. Core reads the current full log through `PopulationCustodyPort`, accepts only a reviewed custodian, recomputes every embedded request and receipt digest, requires contiguous sequence/predecessor links, and requires the declared head/count/status to equal the derived chain. `SPENT` is monotonic: no later receipt or manifest can return it to `AVAILABLE`. A caller-selected prefix or an omitted earlier spend receipt therefore fails current-head validation. `validate_provenance` requires the closed seven-role input set exactly once, re-derives the named-input manifest, maps every role to the corresponding `ProvenanceStamp` identity, and resolves every `ContentRef`. The resolved bytes, digest, length, storage receipt, retention policy, trusted store, and byte limit must all validate. A stamp cannot therefore be accepted from identities alone when its named inputs are unreadable. `validate_lineage` re-derives the manifest and requires the expected subject and typed `LineageExpectation` values supplied by the caller. Every use must match an expected purpose and influenced-artifact digest, and each expectation must satisfy its declared minimum/maximum count. For each use the validator resolves the receipt from the current complete custody chain and requires exact equality of purpose, source population, label access, and influenced-artifact digest. `forbid_label_access` rejects any labelled ancestry; otherwise labelled use is allowed only when its source is not the protected evaluation population. For prompt, example, threshold, fitted-model, semantic-fact, matching-behavior, decision-mapping, or promotion use, `labels_accessed` together with an influenced artifact deterministically yields `SPENT`. A `MATCHING_BEHAVIOR` use must name the matcher behavior digest. Legitimate one-time admission label opening uses `ADMISSION_LABEL_OPEN`, is bound to the authorization digest, and is durably appended before label bytes are released. Benchmark workflows persist results. The core returns `stale` when any recorded identity or predecessor digest differs from current inputs, including core schema, semantic vocabulary, acquisition, or outcome mapping. Instrument through pair receipts use regression fixtures; development uses a development population; admission requires a separately frozen, available population and a scored admission report; integration uses consumer evidence. An evidence-use mismatch cannot satisfy a stage, so preserved regression fixtures can never stand in for fresh admission. #### Fitted-component Probes | Field | Detail | |---|---| | Intent | Refuse fitted components that exploit reversible surface shape instead of declared security semantics | | Requirements | 13.2 | A reversed-pair probe is valid only when an independent fixture review declares the two inputs security-semantically equivalent for the family contract while their surface/diff shape is deliberately reversed. The semantic projection includes the security object, required roles, relations, completeness, and expected family decision; it excludes line counts, deletion direction, filenames, and presentation order. If the projections differ, the probe is invalid and says nothing about shortcut learning. This replaces the current rule that treats any score difference between an arbitrary fix and its reverse as proof of a shortcut. ```python @dataclass(frozen=True) class RoleProjection: role: str fields: tuple[FieldValue, ...] relation_kinds: tuple[RelationKind, ...] statuses: tuple[RelationStatus, ...] @dataclass(frozen=True) class SemanticProjection: family_contract_digest: str security_object_fields: tuple[FieldValue, ...] roles: tuple[RoleProjection, ...] completeness_reasons: tuple[str, ...] family_decision: str projection_digest: str @dataclass(frozen=True) class SemanticEquivalenceReview: forward_projection_digest: str reversed_projection_digest: str independence: IndependenceAttestation review_digest: str @dataclass(frozen=True) class ReversedPairProbe: probe_id: str family_contract_digest: str forward_record: EvidenceRecord reversed_record: EvidenceRecord forward_surface: ContentRef reversed_surface: ContentRef equivalence: SemanticEquivalenceReview maximum_score_delta: float @dataclass(frozen=True) class ProbeObservation: probe_digest: str component_digest: str forward_decision: str reversed_decision: str forward_score: float | None reversed_score: float | None @dataclass(frozen=True) class ProbeEvaluationInput: probe: ReversedPairProbe observation: ProbeObservation @dataclass(frozen=True) class ProbeResult: probe_digest: str component_digest: str valid: bool passed: bool reasons: tuple[str, ...] result_digest: str @dataclass(frozen=True) class FittedComponentEvidence: component_digest: str probe_result_digests: tuple[str, ...] fitting_lineage_manifest_digest: str training_population_manifest: ContentRef training_repository_digests: tuple[str, ...] repository_holdout_report_digest: str admission_report_digest: str def project_semantics(record: EvidenceRecord, contract: FamilyContract) -> SemanticProjection: ... def evaluate_probe( probe: ReversedPairProbe, observation: ProbeObservation, contract: FamilyContract, review_policy: ReviewIndependencePolicy, resolver: ContentResolver, ) -> ProbeResult: ... ``` Core derives both projections from the embedded records and contract, resolves both surface artifacts, and verifies the equivalence review against the same independence policy used for facts. The attestation subject must equal the canonical digest of both derived projections, the family contract, and the two surface references. Validation requires distinct surface bytes, equal derived semantic projections, the same declared family decision on both sides, and a score delta no larger than the pre-registered finite tolerance. A caller-supplied digest cannot stand in for projection or review content. Fitted-component authorization additionally requires all probes to pass, a typed repository-held-out report whose repository manifest is disjoint from the resolved training manifest, a matching fitting-lineage decision, and an untouched admission `GO`. Probe construction and evaluation live in `probes.py`; fitted-component authorization lives in `admission.py`, where it accepts `ProbeEvaluationInput` values and calls `evaluate_probe` itself. Caller-supplied `ProbeResult` summaries never participate in authorization; the derived result digests must exactly equal the fitted-component evidence. #### Measurement and Admission | Field | Detail | |---|---| | Intent | Produce comparable, family-scoped evidence and refuse impossible admission configurations | | Requirements | 11.6, 12.1–12.6, 15.1–15.5 | **Responsibilities and constraints** - `BaselineIdentity` covers arm, population/freeze, family/ecosystem scope, denominator policy, threshold set, family contract, facts, analyzer, configuration, model program, and outcome mapping. - `MeasurementRecordV2` includes neutral outcomes and bases, rates with sample counts and Wilson intervals, the full funnel, model failures, runtime, instrument failures, and every prerequisite readiness result with its digest. - A zero or clean-looking measurement is invalid unless its instrument readiness reference passed over a complete tracked-input manifest and every earlier semantic-stage reference is present and passed. - Trend comparison requires identical baseline identities. - `AdmissionManifestV2` is externally produced, reviewed, label-isolated, immutable, and unspent. It names every frozen input, a reviewed label-blind population census, and the current readiness receipts. It binds a serializable validation context containing the actual catalog, packs, model declarations, lineages, and trust policies needed for revalidation; digests never substitute for those objects. - `DenominatorPolicy` maps each registered metric to a closed numerator and denominator definition over explicit family/ecosystem scope. Its interval method is fixed to Wilson 95%; a configurable confidence level is refused. Whole-population coverage is reported separately. - The census exposes only reviewed aggregate cardinalities, never case identities or per-case labels. It is bound to the same population digest as the manifest. - After the population is durably spent, the custodian produces an independently attested `AdmissionCaseRelease`. The release is bound to the population, manifest, authorization, preparation, exact spend receipt, and custodian identity. Its bindings associate each case identity with one scope, role, scoped pair identity where applicable, result digest, evidence-input digests, change-input digests, and input-manifest digest. - Before labels open, feasibility enumerates the arithmetically possible counts using pair, case, semantic-site, and model-call capacities and applies count, rate, per-100, or Wilson-bound statistics. Every threshold group must have an attainable joint count. An impossible group is a configuration refusal, not a measured no-go result. - Scoring is a pure function over immutable post-opening case results carrying the actual evidence, fact packs/matches, claims/proof attachments, coverage, instrument receipts, model results, complexity, budget, and timing. Counts and gates are derived; no opaque aggregate supplied by the runner is trusted. The scorer re-runs record/fact/verdict validation and refuses supplied claims/satisfaction/silence that differ from the derived result. Core never persists or relabels cases. - Scoring treats the post-spend case release as the sole authority for case roles and pair membership. It re-derives each case result and its operational projection, requires an exact one-to-one binding, and refuses missing, extra, duplicate, substituted, swapped, or cross-scope cases before producing any metric. - Scoring returns an immutable `AdmissionReplay`, not a bare report. The replay carries the exact authoritative objects needed to reproduce the report. A raw `AdmissionReportV2` is serializable result data, never an authorization token. - Admission authorization produces a custody-head-bound label-release request. The benchmark custodian must durably spend the population before releasing label bytes; scoring revalidates that exact current-chain receipt and never emits a post-scoring spend request. **Contracts:** Service, Batch, State. `measurement.py` owns `ScopeKey`, baseline identity, the closed metric algebra, interval and gate results, readiness references, `MeasurementRecordV2`, and `RepositoryHoldoutReport`. `admission.py` imports those types and owns census, metric rules, thresholds, manifests, case inputs, authorization, and fitted-component authorization. `admission_report.py` owns gate derivation, report assembly, the authoritative replay bundle, and replay validation. Measurement never imports admission modules. ```python @dataclass(frozen=True) class ScopeKey: family_id: str ecosystem: str @dataclass(frozen=True) class BaselineIdentity: arm_id: str population_digest: str freeze_digest: str claimed_scopes: tuple[ScopeKey, ...] family_contract_digests: tuple[str, ...] family_contract_set_digest: str fact_schema_digest: str fact_set_digest: str acquisition_record_digest: str analyzer_revision: str configuration_digest: str model_program_digests: tuple[str, ...] model_program_set_digest: str | None denominator_policy_digest: str threshold_set_digest: str outcome_mapping_digest: str @dataclass(frozen=True) class ScopeCensus: scope: ScopeKey total_cases: int pair_count: int vulnerable_side_count: int fixed_side_count: int ordinary_case_count: int semantic_site_capacity: int model_call_capacity: int @dataclass(frozen=True) class PopulationCensus: population_digest: str rows: tuple[ScopeCensus, ...] case_labels_exposed: bool census_payload_digest: str review: IndependenceAttestation census_digest: str class MetricId(Enum): PAIRED_CORRECTNESS = "paired_correctness" VULNERABLE_RECALL = "vulnerable_recall" FIXED_FALSE_ALARM_RATE = "fixed_false_alarm_rate" ORDINARY_FALSE_ALARMS_PER_100 = "ordinary_false_alarms_per_100" PRECISION = "precision" SEMANTIC_STAGE_COVERAGE = "semantic_stage_coverage" MODEL_SUCCESS_RATE = "model_success_rate" class NumeratorKind(Enum): CORRECT_PAIRS = "correct_pairs" VULNERABLE_VIOLATION_CASES = "vulnerable_violation_cases" FIXED_DEFECT_CASES = "fixed_defect_cases" ORDINARY_DEFECT_CASES = "ordinary_defect_cases" TRUE_DEFECT_CASES = "true_defect_cases" SITES_RECOGNIZED = "sites_recognized" SUCCESSFUL_MODEL_CALLS = "successful_model_calls" class DenominatorKind(Enum): PAIRS = "pairs" VULNERABLE_SIDES = "vulnerable_sides" FIXED_SIDES = "fixed_sides" ORDINARY_CASES = "ordinary_cases" ALL_DEFECT_CASES = "all_defect_cases" SITES_CONSIDERED = "sites_considered" MODEL_CALLS = "model_calls" @dataclass(frozen=True) class MetricRule: metric_id: MetricId scopes: tuple[ScopeKey, ...] numerator: NumeratorKind denominator: DenominatorKind class IntervalMethod(Enum): WILSON_95 = "wilson_95" @dataclass(frozen=True) class DenominatorPolicy: rules: tuple[MetricRule, ...] interval_method: IntervalMethod policy_digest: str class Statistic(Enum): COUNT = "count" RATE = "rate" PER_100 = "per_100" WILSON_LOWER = "wilson_lower" WILSON_UPPER = "wilson_upper" class Comparator(Enum): AT_LEAST = "at_least" AT_MOST = "at_most" @dataclass(frozen=True) class Threshold: metric_id: MetricId statistic: Statistic comparator: Comparator target: float @dataclass(frozen=True) class AdmissionValidationContext: evidence: EvidenceValidationContext admission_review_policy: ReviewIndependencePolicy proof_trust_policy: ProofTrustPolicy | None context_digest: str @dataclass(frozen=True) class AdmissionManifestV2: manifest_payload_digest: str review: IndependenceAttestation label_isolated: bool labels_opened: bool identity: BaselineIdentity census: PopulationCensus readiness_digests: tuple[str, ...] custody_log_digest: str denominator: DenominatorPolicy thresholds: tuple[Threshold, ...] threshold_set_digest: str threshold_lineage_manifest_digest: str validation_context_digest: str manifest_digest: str class AdmissionCaseRole(Enum): VULNERABLE = "vulnerable" FIXED = "fixed" ORDINARY = "ordinary" @dataclass(frozen=True) class AdmissionCaseResult: case_id_digest: str scope: ScopeKey role: AdmissionCaseRole pair_id_digest: str | None evidence_inputs: tuple[EvidenceValidationInput, ...] defect_claims: tuple[DefectClaim, ...] satisfactions: tuple[BoundedSatisfaction, ...] silence: tuple[BoundedSilence, ...] proof_attachments: tuple[ProofAttachment, ...] coverage: CoverageFunnel input_manifest: InputManifest input_receipts: tuple[InputReadReceipt, ...] complexity: ChangeComplexity budget: BudgetDerivation total_reply_ms: int result_digest: str @dataclass(frozen=True) class ReleasedCaseBinding: case_id_digest: str scope: ScopeKey role: AdmissionCaseRole pair_id_digest: str | None case_result_digest: str evidence_input_digests: tuple[str, ...] change_input_digests: tuple[str, ...] input_manifest_digest: str binding_digest: str @dataclass(frozen=True) class AdmissionCaseRelease: schema_version: str population_digest: str manifest_digest: str authorization_digest: str preparation_digest: str label_release_receipt_digest: str custodian_identity: str bindings: tuple[ReleasedCaseBinding, ...] release_payload_digest: str review: IndependenceAttestation release_digest: str @dataclass(frozen=True) class MetricResult: metric_id: MetricId numerator: int denominator: int value: float | None interval_method: IntervalMethod interval_lower: float | None interval_upper: float | None @dataclass(frozen=True) class OutcomeCount: outcome: NeutralOutcome count: int @dataclass(frozen=True) class BasisCount: basis: EvidenceBasis count: int class GateKind(Enum): INSTRUMENT = "instrument" BUDGET = "budget" QUALITY = "quality" READINESS = "readiness" @dataclass(frozen=True) class GateResult: kind: GateKind gate_id: str passed: bool reasons: tuple[str, ...] evidence_digests: tuple[str, ...] class AdmissionDecision(Enum): GO = "go" NO_GO = "no_go" @dataclass(frozen=True) class ReadinessReference: stage: ReadinessStage result: StageResult result_digest: str @dataclass(frozen=True) class MeasurementRecordV2: identity: BaselineIdentity family_decision_vocabulary: tuple[str, ...] family_decision_counts: tuple[ReasonCount, ...] neutral_outcome_counts: tuple[OutcomeCount, ...] evidence_basis_counts: tuple[BasisCount, ...] metrics: tuple[MetricResult, ...] stage_funnel: CoverageFunnel instrument_failures: tuple[str, ...] model_outcome_counts: tuple[ReasonCount, ...] model_failures: tuple[str, ...] runtime_ms: int prerequisite_readiness: tuple[ReadinessReference, ...] record_digest: str @dataclass(frozen=True) class RepositoryHoldoutReport: component_digest: str role: ReadinessEvidenceUse training_population_manifest: ContentRef holdout_population_manifest: ContentRef training_repository_digests: tuple[str, ...] holdout_repository_digests: tuple[str, ...] fitting_lineage_manifest_digest: str measurement: MeasurementRecordV2 report_digest: str @dataclass(frozen=True) class AdmissionReportV2: identity: BaselineIdentity census_digest: str validation_context_digest: str authorization_digest: str label_release_receipt_digest: str case_release_digest: str case_result_digests: tuple[str, ...] in_scope_case_count: int whole_population_case_count: int measurement: MeasurementRecordV2 instrument_results: tuple[GateResult, ...] budget_results: tuple[GateResult, ...] readiness_results: tuple[GateResult, ...] quality_results: tuple[GateResult, ...] decision: AdmissionDecision report_digest: str @dataclass(frozen=True) class AdmissionReplay: manifest: AdmissionManifestV2 denominator: DenominatorPolicy readiness: tuple[ReadinessResult, ...] preparation: AdmissionPreparation label_release_receipt: PopulationUseReceipt case_release: AdmissionCaseRelease cases: tuple[AdmissionCaseResult, ...] report: AdmissionReportV2 replay_digest: str @dataclass(frozen=True) class AdmissionAuthorization: manifest_digest: str validation_context_digest: str readiness_digests: tuple[str, ...] custody_log_digest: str feasibility_digest: str authorization_digest: str @dataclass(frozen=True) class AdmissionPreparation: authorization: AdmissionAuthorization label_release_request: PopulationUseRequest preparation_digest: str def authorize_admission( manifest: AdmissionManifestV2, readiness: Sequence[ReadinessResult], context: AdmissionValidationContext, custody: PopulationCustodyPort, resolver: ContentResolver, ) -> AdmissionPreparation: ... def score_admission( manifest: AdmissionManifestV2, denominator: DenominatorPolicy, readiness: Sequence[ReadinessResult], preparation: AdmissionPreparation, label_release_receipt: PopulationUseReceipt, case_release: AdmissionCaseRelease, cases: Sequence[AdmissionCaseResult], context: AdmissionValidationContext, custody: PopulationCustodyPort, resolver: ContentResolver, ) -> AdmissionReplay: ... def validate_admission_replay( replay: AdmissionReplay, context: AdmissionValidationContext, custody: PopulationCustodyPort, resolver: ContentResolver, ) -> AdmissionReportV2: ... def authorize_fitted_component( evidence: FittedComponentEvidence, probe_inputs: Sequence[ProbeEvaluationInput], context: AdmissionValidationContext, fitting_lineage: LineageManifest, repository_holdout: RepositoryHoldoutReport, admission: AdmissionReplay, custody: PopulationCustodyPort, resolver: ContentResolver, ) -> None: ... ``` `PopulationCensus.case_labels_exposed` and `AdmissionManifestV2.labels_opened` must both be false during validation. The census and manifest payload digests exclude their review attestations but include every substantive field; each `IndependenceAttestation.subject_digest` must equal its payload digest. Producer and reviewer identities must differ and satisfy the context's admission-review policy and forbidden-pair rules; the reviewer must be admitted, and the conflict disclosure must resolve under the policy's store and byte limits. A self-review, unknown reviewer, fabricated digest, or review copied from another payload is refused before feasibility. `AdmissionCaseRelease` is a post-spend contract, not part of the label-blind census or manifest. Its payload digest includes every substantive field except the review attestation; the release digest includes both. The review producer must equal both the declared custodian and the validated spend receipt's custodian, while the reviewer must be distinct and admitted by the context's admission-review policy. The attestation subject must equal the release payload digest, and its conflict disclosure is resolved under the same store and byte limits. The release population, manifest, authorization, preparation, and receipt digests must match the exact values being scored. The manifest identity, census population, claimed scopes, contract-set, facts, acquisition, analyzer, configuration, model-program set, denominator, threshold set, and outcome mapping must all match. `authorize_admission` re-derives the validation-context digest and validates the actual family catalog, fact packs, model declarations, lineages, and policies. The catalog's exact sorted contract-digest set—not a superset—must equal `BaselineIdentity.family_contract_digests`, and `catalog_digest` must equal `family_contract_set_digest`; every claimed scope resolves exactly once to a contract that declares the ecosystem. Fact and model sets are re-derived from the context and must equal the baseline identities. The baseline outcome-mapping digest is re-derived from the exact ordered decision rules in those contracts. The manifest's sorted thresholds must re-derive both its and the baseline's `threshold_set_digest`. The named threshold lineage must be present in the context, target that digest, satisfy a required `THRESHOLD` expectation for that same digest, and validate against current complete custody chains while protecting the admission population. All other referenced prompt, example, fitted-model, fact, matcher, mapping, and promotion lineages in `context.evidence` are similarly validated before label opening. This makes requirement 13.1 an object-level check rather than a claimed digest. The manifest custody-log digest must equal the current complete reviewed-custodian log and that log must leave the population available. `authorize_admission` also validates every named readiness receipt from instrument through development, performs label-blind threshold feasibility, and returns an `AdmissionPreparation`. Its `ADMISSION_LABEL_OPEN` request has `labels_accessed=True`, names the authorization digest as the influenced artifact, and commits to the authorization's exact custody-log head. The benchmark custodian durably compare-and-swap appends the derived `AVAILABLE` to `SPENT` receipt before releasing labels; it never restores availability. A crash before append releases no labels, a crash after append leaves the population spent, and concurrent losers receive no label bytes. `score_admission` requires that spend receipt to be the current-chain direct successor of the authorization head and to match the preparation byte for byte. It refuses a different manifest, context, readiness set, census, case population, receipt, or case release. It validates the case release only after the spend receipt has been revalidated. Release bindings are canonically ordered by case identity and unique; vulnerable and fixed cases require one pair identity qualified by their `ScopeKey`, ordinary cases prohibit one, and every scoped pair contains exactly one vulnerable and one fixed binding. For each supplied case, scoring re-derives `result_digest`, the sorted evidence-input digests, the distinct change-input digests from validated packet provenance, and the input-manifest digest and requires exact equality with its release binding. The supplied scope, role, and pair must also equal the binding. Missing, extra, duplicate, substituted, role-swapped, payload-swapped, or cross-scope bindings fail before metrics. For every case it calls `validate_evidence_input` for every `evidence_inputs` member, then performs instrument and budget validation, and verdict/proof validation using the context-bound policies. Proof attachments are applied only to those validated base records and the resulting claims must equal the supplied claims canonically. Every case model declaration must be an exact member of `context.evidence.model_programs`; every pack and lineage reference must resolve to the context-bound object rather than an embedded substitute. Claims, satisfaction, silence, decision vocabulary, metrics, and gates are then derived from those revalidated objects. A forged family decision, substituted contract, invalid fact review or match, contaminated lineage, or untrusted proof fails before any metric or gate is produced. The final report binds the validation context, authorization, and pre-label spend receipt, and authenticated post-spend case release; there is no post-scoring population-use request. `score_admission` freezes the denominator, readiness, and cases exactly once, derives the report, and returns them with every other authoritative scoring input in an `AdmissionReplay`. Its `replay_digest` protects transport integrity of the bundle but is not a validation token. `validate_admission_replay` revalidates every carried object against an independently supplied current context, custody port, and content resolver, re-runs report derivation, and requires exact report and replay equality. A caller cannot authorize from a raw report or from a coordinated set of rehashed report, threshold, readiness, or case objects. The report validates case counts back to the census, reports paired vulnerable/fixed behavior and ordinary false alarms through registered metrics, imports semantic-stage and model-call outcomes, and stratifies findings by evidence basis so model-corroborated and deterministic claims are not silently merged. Policy validation fixes the standard metric algebra: correct pairs are pairs with a vulnerable-side defect and no fixed-side defect; recall is vulnerable cases with a defect over vulnerable cases; fixed and ordinary false alarms are cases with at least one defect over their respective case denominators (ordinary is scaled per 100); precision is vulnerable defect cases over all defect cases; semantic coverage is recognized over considered sites; and model success is successful over attempted calls. Success thresholds are `AT_LEAST` for correctness, recall, precision, coverage, and model success, and `AT_MOST` for false alarms; reversed comparators are refused. Every rate publishes numerator, denominator, and Wilson 95% interval. A zero denominator is serialized with null value/bounds, never as zero or clean, and cannot pass a threshold. Fitted-component authorization requires `RepositoryHoldoutReport.role` to be exactly `REPOSITORY_HOLDOUT`, resolves both population manifests, proves their repository sets are disjoint, matches component/training/lineage/report digests, validates current custody for the fitting lineage with subject `component_digest` and a required `FITTED_MODEL` expectation for that digest, and requires the admission report to be a `GO` produced from a population that was untouched before its bound label-release spend. It first calls `validate_admission_replay`; a bare `AdmissionReportV2` or a replay whose carried inputs no longer validate cannot authorize a component. It re-derives `context.context_digest`, requires it to equal the validated report's validation-context digest, and requires exactly one baseline-listed declaration in `context.evidence.model_programs` whose `fitted_component_digest` equals `evidence.component_digest`. The selected declaration supplies the family-contract digest; each probe's two embedded records must agree on family and ecosystem, both probe and record contract digests must equal the declaration, and that exact contract is resolved from `context.evidence.family_catalog`. It also requires nonempty unique probe inputs, checks every probe and observation against that component and resolved contract, calls `evaluate_probe` with the actual records, surfaces, `context.evidence.fact_review_policy`, and resolver, and validates fitting lineage with `context.evidence.custody_policy`. Every derived result must be valid and passed. The derived digest tuple must equal `FittedComponentEvidence.probe_result_digests`; a fabricated passed summary cannot authorize the component. Every admission report must contain `PAIRED_CORRECTNESS`, `VULNERABLE_RECALL`, `FIXED_FALSE_ALARM_RATE`, `ORDINARY_FALSE_ALARMS_PER_100`, `PRECISION`, and `SEMANTIC_STAGE_COVERAGE` with nonzero pre-registered denominators, plus model success when applicable. Model success is applicable when the frozen baseline configures at least one model program **or** validated claimed-scope cases contain declared model calls. An applicable model with zero attempted calls, no model-success threshold, or any failed model outcome forces `NO_GO`; calls wholly outside the claimed scopes do not make the model applicable. Claimed-scope model calls without a baseline-listed declaration are invalid evidence and are refused before report production. The report must also contain evidence-backed instrument, budget, readiness, and quality gate results. `GO` is invalid if any mandatory metric or gate is absent, any instrument receipt failed, or any required gate did not pass. ### Decision and Consumer Edge #### Verdict, Proof Attachment, and Consumer | Field | Detail | |---|---| | Intent | Turn validated evidence into bounded outputs and attach independently demonstrated proof | | Requirements | 6.4, 7.1–7.4, 9.4, 9.6, 14.1–14.5 | **Responsibilities and constraints** - `DefectClaim` carries the complete evidence record, evidence basis, exact defect location, and a change anchor inside the analyzed scope. Its basis must equal the record decision's basis. - `BoundedSatisfaction` carries complete question IDs, supporting records, and its exact bounded scope. `BoundedSilence` carries affected questions, named reasons, record digests, and a coverage digest. - Incomplete paths remain silence unless another complete deterministic path or a matching demonstrated proof independently supports a claim. - A proof attachment contains a canonical differential proof record: reviewed producer, sandbox and oracle identities; typed affected and safe observations; and a content-addressed witness. Core re-derives its digest and requires the producer in the supplied immutable trust policy. Only `demonstrated` with affected-only observation may derive execution-backed basis. - Proof attachment does not enable blocking; it supplies the evidence basis consumed by downstream capability policy. - `PushRequest` supplies the immutable analyzed scope, workload class, measured budget, and measured `ChangeComplexity`, and request identity. Core validates region and transition counts against scope/records, resolves the changed-byte measurement, and validates every claim anchor and record location. The measurement bytes must bind the same base, head, change digest, regions, transition count, and changed size; core does not choose regions. - All work, including optional model and proof attachment processing, is accounted against the measured push budget supplied by the consumer. **Contracts:** Service, State. ```python @dataclass(frozen=True) class DefectClaim: record: EvidenceRecord basis: EvidenceBasis location: SourceLocation change_anchor: SourceRegion @dataclass(frozen=True) class BoundedSatisfaction: question_ids: tuple[str, ...] records: tuple[EvidenceRecord, ...] scope: AnalyzedScope @dataclass(frozen=True) class BoundedSilence: question_ids: tuple[str, ...] reasons: tuple[str, ...] evidence_record_digests: tuple[str, ...] coverage_digest: str @dataclass(frozen=True) class ProofProducerIdentity: producer_id: str verifier_revision: str sandbox_profile_digest: str oracle_contract_digest: str review_digest: str class ProofOutcome(Enum): DEMONSTRATED = "demonstrated" INCONCLUSIVE = "inconclusive" FAILED = "failed" @dataclass(frozen=True) class ProofSideResult: input_digest: str observed: bool observation_digest: str run_count: int result_digest: str @dataclass(frozen=True) class ControlBinding: affected_change_input_digest: str safe_input_digest: str control_manifest: ContentRef reviewer_identity: str binding_digest: str @dataclass(frozen=True) class DifferentialProofRecord: schema_version: str producer: ProofProducerIdentity affected: ProofSideResult safe: ProofSideResult control: ControlBinding witness: ContentRef outcome: ProofOutcome record_digest: str @dataclass(frozen=True) class ProofVerificationReceipt: producer_identity_digest: str proof_record_digest: str affected_result_digest: str safe_result_digest: str witness_content_digest: str attestation: ContentRef receipt_digest: str @dataclass(frozen=True) class ProofTrustPolicy: reviewed_producer_digests: tuple[str, ...] reviewed_control_reviewer_digests: tuple[str, ...] trusted_store_identities: tuple[str, ...] maximum_artifact_bytes: int policy_digest: str @dataclass(frozen=True) class ProofAttachment: proof_id: str family_id: str packet_digest: str location: SourceLocation proof: DifferentialProofRecord verification: ProofVerificationReceipt @dataclass(frozen=True) class PushRequest: request_id: str scope: AnalyzedScope workload_class: str complexity: ChangeComplexity budget: BudgetDerivation @dataclass(frozen=True) class PushVerdict: request_digest: str defects: tuple[DefectClaim, ...] satisfactions: tuple[BoundedSatisfaction, ...] silence: tuple[BoundedSilence, ...] input_record_digests: tuple[str, ...] complexity: ChangeComplexity total_reply_ms: int budget_derivation_digest: str def attach_proof( claim: DefectClaim, attachment: ProofAttachment, trust: ProofTrustPolicy, resolver: ContentResolver, ) -> DefectClaim: ... def consume_records( request: PushRequest, inputs: Sequence[EvidenceValidationInput], validation_context: EvidenceValidationContext, custody: PopulationCustodyPort, total_reply_ms: int, resolver: ContentResolver, attachments: Sequence[ProofAttachment] = (), proof_trust: ProofTrustPolicy | None = None, ) -> PushVerdict: ... ``` `attach_proof` validates the existing claim's packet, family, exact location, reviewed producer, control reviewer, canonical proof and verification-receipt digests, and the differential condition `affected.observed is True` and `safe.observed is False`. The affected input must equal the packet's `change_input_digest`; the safe input must equal the reviewed control binding. The resolver must return witness, control-manifest, attestation, and storage bytes whose hashes, lengths, receipts, retention policies, and trusted store identities match. An opaque reference or self-asserted producer is insufficient. The function returns a new immutable claim whose derived record carries proof and producer digests in `DecisionProvenance`, leaving the packet untouched. `consume_records` first calls `validate_evidence_input` for every raw input and creates base claims only from the returned records, then applies matching attachments through `attach_proof`; nonempty attachments require the trust policy, and the shared resolver also verifies the complexity measurement. It preserves every bounded-silence reason verbatim, rejects records outside the request scope, and applies `check_budget` before returning a verdict that retains the same complexity and derivation. A missing or non-positive elapsed or complexity measurement is an instrument failure, never an implied fast run. ### Migration Components #### Legacy Compatibility Namespace and Cutover Boundary | Field | Detail | |---|---| | Intent | Turn v1 dependency and write capability into explicit modules that can be counted and deleted without interpreting Python execution | | Requirements | 14.3, 15.3 | **Responsibilities and constraints** - `legacy_v1/contracts.py` temporarily owns every authorization-shaped v1 value moved out of mixed v2 modules. `legacy_v1/writer.py` owns the only live v1 serialization capability. - Existing top-level v1 names remain compatible during migration only through explicit imports from `legacy_v1`. Compatibility modules contain no independent implementation. - The frozen migration inventory names every reviewed reader and writer by path, symbol, artifact family, and disposition. An entry closes only as `v2_migrated` or `retired`; adding a new entry is a design regression, not remediation. - The boundary checker reads every scoped source file and proves it is nonzero before parsing. It classifies import statements and direct writer-facade references only. It never follows assignment, default-argument, exception, comprehension, dynamic-call, or runtime attribute flow. - Production module-valued imports of `openultrasast.core`, dynamic imports of the package, and dynamic attribute access to its compatibility names are prohibited. Explicit submodule or symbol imports are required and mechanically checkable. - Single-writer cutover requires four simultaneous facts: every inventory entry is closed; syntactic legacy imports and writer references are zero; the compatibility namespace and shim modules are absent; and the old top-level attributes are not importable. A plausible zero from an unread or unparsable file is an instrument failure. **Contracts:** Service, State. ```python class MigrationDisposition(Enum): OPEN = "open" V2_MIGRATED = "v2_migrated" RETIRED = "retired" @dataclass(frozen=True) class LegacyInventoryEntry: path: str symbol: str dependency_kind: str artifact_family: str disposition: MigrationDisposition @dataclass(frozen=True) class LegacyBoundaryReport: files_read: int bytes_read: int open_inventory_entries: tuple[LegacyInventoryEntry, ...] legacy_import_sites: tuple[str, ...] legacy_writer_sites: tuple[str, ...] forbidden_dynamic_sites: tuple[str, ...] def inspect_legacy_boundary(paths: Sequence[Path]) -> LegacyBoundaryReport: ... def require_cutover_ready(report: LegacyBoundaryReport) -> None: ... ``` `inspect_legacy_boundary` is a migration test instrument, not a runtime core service. Its schema is intentionally syntactic. Runtime use-flow analysis is neither required nor accepted as evidence for cutover. #### Historical Decoder and External Adapters `legacy.py` decodes and validates historical v1 traces and reports from exact built-in `bytes`. It never imports `legacy_v1`, constructs v2 determinate outcomes, or emits a new v1 artifact. The compatibility namespace is therefore deletable while historical artifacts remain readable. Authorization conversion moves to `semantic/obligations/core_adapter.py`, where old fields map to the authorization family contract. Missing per-hop completeness remains incomplete. The DSPy and search adapters similarly map external typed records without leaking their dependencies into core. ## Evaluation-Admission V3 Amendment Requirements 16–20 extend the implemented family-neutral core with a distinct exact-scope, multi-arm evaluation contract. This section is normative for V3; the earlier V2 measurement and admission design remains normative and unchanged for historical and compatibility use. ### Boundary Commitments Detection-core owns the V3 canonical schemas, exact-type validators, reviewed-plan and observation-authority contracts, per-arm/per-scope feasibility and metric algebra, shared spend binding, V3 case release, component/combined lineage validation, scoped reports, complete replay, and consumer authorization. It preserves every V2 schema, byte, digest, decision, validator, and replay path unchanged. Detection-core does not own population sourcing or lifecycle, durable object storage or CAS, observation access enforcement, detector execution or isolation, retry orchestration, label joining, workflow status, notification, presentation, or rollout policy. The benchmark produces immutable reviewed lifecycle, observation, custody, and release artifacts; core validates them and derives no authority from an unreviewed workflow state. Allowed dependencies are Python 3.11 standard-library modules and existing version-neutral core primitives: canonical content references/resolution, independence review, custody, readiness, evidence validation, proof attachment, coverage, budget, and Wilson arithmetic. V3 modules never import V2 aggregate roots. V2 modules never import V3 modules. Adjacent benchmark and consumer code imports the public V3 facade. Downstream revalidation is required by a V3 schema or canonical-profile change; a change to plan, baseline, observation-slot, disposition/retry, shared-spend, release, proof-candidate, combination, denominator, gate, aggregate, or replay semantics; or any shared-helper change that alters frozen V2 output. ### Architecture The selected pattern is parallel versioned aggregate roots with version-neutral internal helpers. V2 remains an independently replayable historical/live compatibility contract. V3 is a new baseline and has no conversion path from V2. The public root schema is exactly `openultrasast.detection-core.evaluation-admission.v3`. Strict decoding accepts only the registered V3 root schemas below; unknown versions, V2 roots, subclasses, and graphs whose children name a different version are refused without shape inference. ```mermaid flowchart LR Benchmark[Benchmark workflow] --> Plan[Reviewed plan] Plan --> Feasibility[Exact scope feasibility] Benchmark --> Observations[Sealed observations] Feasibility --> Authority[Admission authority] Observations --> Authority Authority --> Custody[Shared custody transition] Custody --> Release[V3 case release] Release --> Reports[Scoped arm reports] Reports --> Replay[Complete evaluation replay] Replay --> Consumer[Consumer authorization] ``` Dependency direction: ```text canonical, review, custody, readiness, evidence, proof, coverage, budget → evaluation_measurement → evaluation_plan → evaluation_feasibility → evaluation_observation → evaluation_authority → evaluation_release → evaluation_metrics → evaluation_report → evaluation_replay → evaluation_admission facade → admission_fitted V3 consumer ``` ### File Structure Plan New files: - `src/openultrasast/core/_metric_algebra.py` — version-neutral closed metric algebra and Wilson arithmetic, created only after V2 golden fixtures lock current behavior; contains no schema, serializer, aggregate root, or version dispatch. - `src/openultrasast/core/evaluation_measurement.py` — V3 arm/scope keys, arm configuration and baseline identities, exact denominator policies, scoped thresholds, sufficiency/precision rules, scoped metric results, and non-authorizing aggregates. - `src/openultrasast/core/evaluation_plan.py` — non-circular plan declaration, expected observation slots, component/combined arm plans, V3 census, and independently reviewed `EvaluationPlanManifestV3`. - `src/openultrasast/core/evaluation_feasibility.py` — per-arm/per-scope threshold, minimum-size, and interval-precision witnesses and shared `may_spend` derivation. - `src/openultrasast/core/evaluation_observation.py` — sealed observation references, reviewed resolver-backed attempt history, retry/disposition evidence, and `EvaluationAdmissionManifestV3`. - `src/openultrasast/core/evaluation_authority.py` — post-observation revalidation, `EvaluationAdmissionAuthorityV3`, shared preparation, and all-arm spend binding. - `src/openultrasast/core/evaluation_release.py` — arm-aware released-case bindings, proof-candidate continuity, and independently reviewed `AdmissionCaseReleaseV3`. - `src/openultrasast/core/evaluation_metrics.py` — exact-scope case validation and metric derivation, ordinary ecosystem projection, whole-push deduplication, and combination preservation. - `src/openultrasast/core/evaluation_report.py` — completed per-scope decisions, per-arm operational evidence, whole-population coverage, degradation reasons, and aggregates. - `src/openultrasast/core/evaluation_replay.py` — complete V3 graph construction and full authoritative replay validation. - `src/openultrasast/core/evaluation_admission.py` — sole public V3 facade for preflight, authorization, score/replay, strict decoding, and replay validation. Modified files: - `src/openultrasast/core/admission_fitted.py` — add a separate V3 consumer function that accepts only a complete replay plus one exact arm/scope key; keep the V2 function exact. - `src/openultrasast/core/__init__.py` — expose the V3 facade without changing V2 names. - `tests/fixtures/detection_core/admission_v2/` — frozen accepted bytes, digests, decisions, validation failures, and replay outputs. - `tests/test_core_admission_v2_compatibility.py` and `tests/test_core_evaluation_*.py` — V2 stability and V3 contract/conformance suites. ### System Flow ```mermaid sequenceDiagram participant Run as Benchmark workflow participant Core as Detection core V3 participant Store as Observation resolver participant Custody as Custody writer participant Reviewer as Independent reviewer Run->>Reviewer: Label blind plan payload Reviewer-->>Run: Plan attestation Run->>Core: EvaluationPlanManifestV3 Core-->>Run: EvaluationFeasibilityV3 Run->>Run: Execute every planned arm label blind Run->>Store: Seal every expected observation Run->>Reviewer: Plan observations and disposition payload Reviewer-->>Run: Admission attestation Run->>Core: EvaluationAdmissionManifestV3 Core->>Store: Resolve exact nonzero observations Core-->>Run: EvaluationPreparationV3 Run->>Custody: Compare and swap shared request Custody-->>Run: One spend receipt and labels Run->>Reviewer: Arm case release payload Reviewer-->>Run: V3 release attestation Run->>Core: Receipt release and every arm case Core-->>Run: EvaluationReplayV3 ``` Any insufficient arm/scope ends before execution/spend. Missing, changed, unreadable, or unsealed observations end before preparation. After spend, missing or invalid cases, denominator mismatch, or incomplete arms invalidate the evaluation and cannot be reported as insufficiency, GO, or NO-GO. The custody writer remains external and releases labels only to the exact CAS winner. ### Requirements Traceability | Requirement | Design realization | |---|---| | 16.1, 16.2, 16.3, 16.4 | Separate V3 facade, exact schema/type validators, no conversion, frozen V2 compatibility suite | | 16.5 | Plan declaration plus per-arm V3 baseline binds every listed identity and changes on any plan/configuration mutation | | 16.6 | Typed preflight/impairment/aggregate/report artifacts have no consumer-authority interface; only complete replay authorizes | | 17.1, 17.2 | Independently reviewed plan declaration, arm plans, component/combined lineage, and expected slots | | 17.3, 17.4 | Complete arm/scope feasibility product and shared `may_spend`; insufficiency prevents preparation | | 17.5, 17.6 | Independently reviewed admission manifest and exact resolved observation-slot equality | | 17.7, 17.8, 17.9 | One authority, preparation, CAS request/receipt, and all-arm spend binding; no subset construction | | 17.10, 17.11, 17.12 | Reviewed sealed/development/spent disposition and exact retry identity; only sealed unchanged retries validate | | 18.1, 18.2, 18.3, 18.4 | Full metric set for each `ArmScopeKey`; exact roles/scopes; ecosystem-only ordinary projection; exact precision | | 18.5 | `underlying_push_digest` deduplicates whole coverage independently from scoped projections | | 18.6, 18.7 | Scoped metric numerator/denominator/sample/Wilson fields and aggregate type without gate/decision | | 18.8 | Post-spend zero or census mismatch is validation failure before report construction | | 19.1, 19.2, 19.3 | Independently reviewed arm-aware V3 release bound to plan, observations, authority and shared spend; V2 type refused | | 19.4, 19.5 | Exact component observations/reports and deterministic combination lineage are mandatory and contradiction-checked | | 19.6, 19.7, 19.8, 19.9, 19.10 | Pre-release candidate is non-authoritative; post-release proof matches exact candidate/binding; no transfer or mismatch | | 19.11 | Planned-arm/report cardinality equality; no result can mask another arm | | 20.1, 20.2 | Completed result per arm/scope with GO/NO-GO; insufficiency, impairment, incomplete and invalid remain distinct | | 20.3 | Per-arm scoped metrics plus separate whole coverage, completed/deferred work, funnels, failures, runtime and degradations | | 20.4, 20.5, 20.6, 20.7 | Closed replay binds and rederives the full graph; missing components refuse completion; only replay authorizes | | 20.8 | Each arm/scope requires nonzero read receipts and required-stage execution evidence | ### Components and Interfaces #### V3 identity, plan, and feasibility `ArmScopeKey` is the natural key for every scoped policy, threshold, metric, gate, and decision. An arm configuration carries the actual immutable policy objects and the identities listed in 16.5. `EvaluationPlanDeclarationV3` is hashed before arm baselines are constructed, avoiding a digest cycle. Each baseline binds the declaration digest and one arm-configuration digest. ```python @dataclass(frozen=True, order=True) class ArmScopeKey: arm_id: str scope: ScopeKey class ArmKindV3(Enum): COMPONENT = "component" COMBINED = "combined" @dataclass(frozen=True) class ScopedMetricRuleV3: key: ArmScopeKey metric_id: MetricId numerator: NumeratorKind denominator: DenominatorKind rule_digest: str @dataclass(frozen=True) class ScopedThresholdV3: key: ArmScopeKey metric_id: MetricId statistic: Statistic comparator: Comparator target: float threshold_digest: str @dataclass(frozen=True) class IntervalPrecisionRuleV3: metric_id: MetricId maximum_wilson_width: float minimum_denominator: int rule_digest: str @dataclass(frozen=True) class DenominatorPolicyV3: key: ArmScopeKey rules: tuple[ScopedMetricRuleV3, ...] interval_method: IntervalMethod policy_digest: str @dataclass(frozen=True) class ScopeSufficiencyRuleV3: key: ArmScopeKey minimum_pairs: int minimum_ordinary_pushes: int precision_rules: tuple[IntervalPrecisionRuleV3, ...] rule_digest: str @dataclass(frozen=True) class ExactScopeCensusV3: scope: ScopeKey pair_count: int vulnerable_side_count: int fixed_side_count: int semantic_site_capacity: int model_call_capacity: int @dataclass(frozen=True) class EcosystemOrdinaryCensusV3: ecosystem: str ordinary_push_count: int @dataclass(frozen=True) class PopulationCensusV3: population_digest: str scope_rows: tuple[ExactScopeCensusV3, ...] ordinary_rows: tuple[EcosystemOrdinaryCensusV3, ...] whole_underlying_push_count: int labels_exposed: bool payload_digest: str review: IndependenceAttestation census_digest: str class CombinationOperatorV3(Enum): DECLARATIVE_TABLE = "declarative_table" class ProofCandidatePolicyV3(Enum): IDENTICAL_COMPONENT_ONLY = "identical_component_only" @dataclass(frozen=True) class CombinationDecisionRowV3: component_outcomes: tuple[NeutralOutcome, ...] combined_outcome: NeutralOutcome row_digest: str @dataclass(frozen=True) class CombinationRuleV3: combined_arm_id: str component_arm_ids: tuple[str, ...] operator: CombinationOperatorV3 decision_rows: tuple[CombinationDecisionRowV3, ...] candidate_policy: ProofCandidatePolicyV3 rule_digest: str @dataclass(frozen=True) class CombinationLineageV3: rule: CombinationRuleV3 component_observation_digests: tuple[str, ...] retained_proof_candidate_digests: tuple[str, ...] lineage_digest: str @dataclass(frozen=True) class ArmConfigurationV3: arm_id: str kind: ArmKindV3 exact_scopes: tuple[ScopeKey, ...] denominator_policies: tuple[DenominatorPolicyV3, ...] thresholds: tuple[ScopedThresholdV3, ...] sufficiency_rules: tuple[ScopeSufficiencyRuleV3, ...] judgment_vocabulary_digest: str analyzer_identity: str fact_set_digest: str configuration_digest: str registered_budget_digest: str execution_identity: str outcome_mapping_digest: str combination_rule: CombinationRuleV3 | None configuration_identity_digest: str @dataclass(frozen=True) class ExpectedObservationSlotV3: slot_id: str arm_id: str input_identity_digest: str execution_identity: str slot_digest: str @dataclass(frozen=True) class EvaluationPlanDeclarationV3: population_digest: str freeze_digest: str input_set_digest: str arm_configurations: tuple[ArmConfigurationV3, ...] expected_observation_slots: tuple[ExpectedObservationSlotV3, ...] declaration_digest: str @dataclass(frozen=True) class EvaluationBaselineIdentityV3: arm_id: str plan_declaration_digest: str population_digest: str freeze_digest: str input_set_digest: str arm_configuration_digest: str baseline_digest: str @dataclass(frozen=True) class EvaluationPlanManifestV3: declaration: EvaluationPlanDeclarationV3 baselines: tuple[EvaluationBaselineIdentityV3, ...] census: PopulationCensusV3 readiness_digests: tuple[str, ...] custody_log_digest: str payload_digest: str review: IndependenceAttestation manifest_digest: str class FeasibilityStatusV3(Enum): FEASIBLE = "feasible" INSUFFICIENT = "insufficient" @dataclass(frozen=True) class ThresholdFeasibilityV3: threshold_digest: str attainable: bool witness_numerator: int witness_denominator: int witness_digest: str @dataclass(frozen=True) class SufficiencyResultV3: rule_digest: str minimum_size_passed: bool interval_precision_passed: bool reasons: tuple[str, ...] result_digest: str @dataclass(frozen=True) class ScopeFeasibilityV3: key: ArmScopeKey threshold_witnesses: tuple[ThresholdFeasibilityV3, ...] sufficiency_results: tuple[SufficiencyResultV3, ...] status: FeasibilityStatusV3 reasons: tuple[str, ...] result_digest: str @dataclass(frozen=True) class EvaluationFeasibilityV3: plan_manifest_digest: str rows: tuple[ScopeFeasibilityV3, ...] may_spend: bool feasibility_digest: str ``` The feasibility rows equal the canonical product of every arm and its exact claimed scopes. Every row rederives minimum sizes, threshold reachability, and interval precision from the reviewed label-blind census. `may_spend` is true only when every row is feasible. #### Observation authority and atomic spend ```python class PreSpendDispositionV3(Enum): SEALED = "sealed" DEVELOPMENT_OPENED = "development_opened" SPENT = "spent" class PublicStatusFieldV3(Enum): STATUS_CODE = "status_code" COUNT = "count" DIGEST = "digest" @dataclass(frozen=True) class ObservationAccessPolicyV3: allowed_public_fields: tuple[PublicStatusFieldV3, ...] raw_observation_access_allowed: bool population_diagnostic_access_allowed: bool policy_digest: str @dataclass(frozen=True) class ObservationCaseRefV3: case_id_digest: str underlying_push_digest: str result_digest: str proof_candidate_digest: str | None @dataclass(frozen=True) class ArmObservationSetV3: slot_digest: str arm_id: str input_identity_digest: str execution_identity: str registered_budget_digest: str configuration_digest: str input_read_evidence_digest: str stage_execution_evidence_digest: str cases: tuple[ObservationCaseRefV3, ...] observation_digest: str @dataclass(frozen=True) class SealedObservationRefV3: slot_digest: str arm_id: str input_identity_digest: str execution_identity: str registered_budget_digest: str configuration_digest: str observation: ContentRef observation_digest: str read_receipt_digest: str ref_digest: str @dataclass(frozen=True) class AttemptIdentityV3: population_digest: str plan_declaration_digest: str baseline_digests: tuple[str, ...] input_set_digest: str configuration_digests: tuple[str, ...] registered_budget_digests: tuple[str, ...] execution_identities: tuple[str, ...] retry_identity_digest: str @dataclass(frozen=True) class AttemptEventV3: attempt_identity: AttemptIdentityV3 disposition: PreSpendDispositionV3 access_policy: ObservationAccessPolicyV3 raw_observation_access_count: int population_diagnostic_access_count: int public_status_projection: ContentRef predecessor_event_digest: str | None payload_digest: str review: IndependenceAttestation event_digest: str @dataclass(frozen=True) class AttemptEventRefV3: event: ContentRef event_digest: str ref_digest: str @dataclass(frozen=True) class AttemptHistoryV3: population_digest: str events: tuple[AttemptEventRefV3, ...] head_event_digest: str history_digest: str @dataclass(frozen=True) class EvaluationAdmissionManifestV3: plan_manifest: EvaluationPlanManifestV3 feasibility_digest: str observations: tuple[SealedObservationRefV3, ...] readiness_digests: tuple[str, ...] custody_log_digest: str attempt_history: AttemptHistoryV3 payload_digest: str review: IndependenceAttestation manifest_digest: str @dataclass(frozen=True) class EvaluationAdmissionAuthorityV3: admission_manifest_digest: str plan_manifest_digest: str baseline_digests: tuple[str, ...] feasibility_digest: str observation_set_digest: str readiness_set_digest: str validation_context_digest: str custody_log_digest: str attempt_history_head_digest: str admission_review_digest: str authority_digest: str @dataclass(frozen=True) class EvaluationPreparationV3: authority: EvaluationAdmissionAuthorityV3 label_release_request: PopulationUseRequest planned_baseline_digests: tuple[str, ...] preparation_digest: str @dataclass(frozen=True) class EvaluationSpendBindingV3: preparation_digest: str receipt_digest: str planned_baseline_digests: tuple[str, ...] binding_digest: str ``` `EvaluationAdmissionManifestV3` is the externally produced, independently reviewed admission-authority artifact required by 17.5: its review subject is the complete payload digest, its producer and reviewer differ, and the payload binds the feasible plan, exact observations, the complete attempt-history head, readiness, and current custody head. The derived `EvaluationAdmissionAuthorityV3` is core's validated projection of that reviewed artifact; it carries the exact review digest and cannot exist without it. The admission manifest must contain exactly one observation reference for every expected slot. Validation resolves each content reference, proves nonzero bytes, parses its canonical `ArmObservationSetV3`, and checks arm, input, execution, budget, configuration, slot, input-read evidence, stage-execution evidence, case references, and observation identity. Core resolves every `AttemptEventRefV3`, validates its independent review and nonzero canonical bytes, and requires an exact predecessor chain ending at the declared head. Every unspent predecessor must carry the same rederived `AttemptIdentityV3`; a changed population, plan, baseline set, input set, configuration, registered budget, or execution identity requires a terminal development-opened or spent event. The access policy must prohibit raw-observation and population-diagnostic access, both access counts must be zero, and the resolved public projection may contain only closed status codes, counts, and digests. Any nonzero access count likewise requires a terminal development-opened or spent event. Terminal dispositions are monotonic, so no later sealed event can restore admission eligibility. Only a complete all-sealed identical-identity chain can authorize. The derived authority and replay both bind the history head; replay re-resolves the full chain. The authority also binds the independent admission-review digest and cannot be produced from a self-reviewed or unattested payload. The underlying `PopulationUseRequest` names the canonical evaluation-plan identity in its legacy `arm_id` field and the V3 authority digest as its influenced artifact. The authority transitively binds every baseline; `EvaluationSpendBindingV3` makes the relationship to the one exact receipt explicit. Core never performs the append or releases labels. #### V3 case release, combinations, and proof continuity ```python @dataclass(frozen=True) class ProofCandidateIdentityV3: component_arm_id: str observation_digest: str family_id: str scope: ScopeKey finding_location_digest: str change_input_digest: str candidate_digest: str @dataclass(frozen=True) class ReleasedArmCaseBindingV3: key: ArmScopeKey baseline_digest: str observation_slot_digest: str observation_digest: str underlying_push_digest: str case_id_digest: str role: AdmissionCaseRole pair_id_digest: str | None case_result_digest: str evidence_input_digests: tuple[str, ...] change_input_digests: tuple[str, ...] input_manifest_digest: str proof_candidate_digest: str | None binding_digest: str @dataclass(frozen=True) class AdmissionCaseReleaseV3: plan_manifest_digest: str admission_manifest_digest: str authority_digest: str preparation_digest: str spend_binding_digest: str bindings: tuple[ReleasedArmCaseBindingV3, ...] payload_digest: str review: IndependenceAttestation release_digest: str @dataclass(frozen=True) class ArmAdmissionCaseResultV3: key: ArmScopeKey observation_slot_digest: str observation_digest: str underlying_push_digest: str case_id_digest: str role: AdmissionCaseRole pair_id_digest: str | None evidence_inputs: tuple[EvidenceValidationInput, ...] claims: tuple[DefectClaim, ...] satisfactions: tuple[BoundedSatisfaction, ...] silence: tuple[BoundedSilence, ...] proof_candidate: ProofCandidateIdentityV3 | None proof_attachments: tuple[ProofAttachment, ...] coverage: CoverageFunnel input_manifest: InputManifest input_receipts: tuple[InputReadReceipt, ...] complexity: ChangeComplexity budget: BudgetDerivation total_reply_ms: int result_digest: str @dataclass(frozen=True) class ArmAdmissionCasesV3: arm_id: str baseline_digest: str cases: tuple[ArmAdmissionCaseResultV3, ...] cases_digest: str ``` The release is the sole post-spend authority for arm, scope, role, pair, result, evidence, inputs, and proof-candidate association. It is an exact V3 type; V2 release values and subclasses are refused. Component arms retain their own cases, metrics, gates, and reports. A combined arm's `CombinationLineageV3` names every component arm, observation digest, combination rule, and retained candidate. The rule is a finite canonical decision table over normalized component observation outcomes with a fixed identical-candidate-only proof policy. Core replays the table; arbitrary combination code is not accepted. Missing, extra, replaced, or contradictory component material invalidates the combined evaluation. A proof candidate is non-authoritative before release. After release, the existing proof validator applies only when family, scope, pair roles, inputs, location, differential, control, producer, verification receipt, and trust policy match the released candidate. Absent or inconclusive proof preserves not-demonstrated status. A combined arm can retain only the identical candidate from a named component observation. #### Exact metrics, reports, and complete replay ```python @dataclass(frozen=True) class ScopedMetricResultV3: key: ArmScopeKey metric_id: MetricId numerator: int denominator: int sample_count: int value: float | None interval_method: IntervalMethod interval_lower: float | None interval_upper: float | None @dataclass(frozen=True) class ScopeAdmissionResultV3: key: ArmScopeKey metrics: tuple[ScopedMetricResultV3, ...] instrument_gate: GateResult budget_gate: GateResult readiness_gate: GateResult quality_gate: GateResult decision: AdmissionDecision result_digest: str class DegradationReasonV3(Enum): INSTRUMENT = "instrument" MODEL = "model" BUDGET = "budget" COVERAGE = "coverage" DEFERRED = "deferred" @dataclass(frozen=True) class ArmOperationalEvidenceV3: completed_count: int deferred_count: int completed_case_digests: tuple[str, ...] deferred_case_digests: tuple[str, ...] stage_funnel: CoverageFunnel instrument_failures: tuple[str, ...] model_failures: tuple[str, ...] runtime_ms: int degradation_reasons: tuple[DegradationReasonV3, ...] evidence_digest: str @dataclass(frozen=True) class WholePopulationCoverageV3: underlying_push_count: int completed_underlying_push_count: int deferred_underlying_push_count: int coverage_digest: str @dataclass(frozen=True) class ArmEvaluationReportV3: baseline: EvaluationBaselineIdentityV3 scoped_results: tuple[ScopeAdmissionResultV3, ...] operational_evidence: ArmOperationalEvidenceV3 combination_lineage: CombinationLineageV3 | None report_digest: str @dataclass(frozen=True) class AggregateWeightV3: key: ArmScopeKey numerator: int denominator: int weight_digest: str @dataclass(frozen=True) class EvaluationAggregateV3: constituents: tuple[ArmScopeKey, ...] sample_counts: tuple[int, ...] weights: tuple[AggregateWeightV3, ...] metrics: tuple[AggregateMetricV3, ...] aggregate_digest: str @dataclass(frozen=True) class AggregateMetricV3: metric_id: MetricId weighted_value: float constituent_metric_digests: tuple[str, ...] metric_digest: str @dataclass(frozen=True) class EvaluationReplayV3: plan_manifest: EvaluationPlanManifestV3 feasibility: EvaluationFeasibilityV3 admission_manifest: EvaluationAdmissionManifestV3 authority: EvaluationAdmissionAuthorityV3 preparation: EvaluationPreparationV3 receipt: PopulationUseReceipt spend_binding: EvaluationSpendBindingV3 case_release: AdmissionCaseReleaseV3 cases_by_arm: tuple[ArmAdmissionCasesV3, ...] reports: tuple[ArmEvaluationReportV3, ...] combination_lineages: tuple[CombinationLineageV3, ...] attempt_history_head_digest: str whole_population_coverage: WholePopulationCoverageV3 aggregate: EvaluationAggregateV3 | None replay_digest: str ``` Vulnerable/fixed recall and pair metrics use only independently verified cases from the exact `ArmScopeKey`. Precision uses only findings and cases assigned to the same key. An eligible ordinary push projects once into each family claimed by that arm in the push's reviewed ecosystem and never into another ecosystem. Whole coverage deduplicates by `underlying_push_digest`. Each standard metric carries numerator, denominator, sample count, value, and Wilson 95% bounds. A post-spend zero or census mismatch is invalid and prevents report construction. `EvaluationAggregateV3` deliberately has no gate or decision. Its weights are canonical non-negative rational values, sum exactly to one by integer cross-multiplication, and correspond one-to-one with its constituents and sample counts. A replay is constructible only when its report identities exactly equal all planned arms, every arm report contains all exact scopes, and every combined lineage preserves its components. Replay validation re-resolves current trust/custody state and rederives identities, counts, metrics, gates, decisions, reports, and replay equality. #### Public service interface ```python def preflight_evaluation( manifest: EvaluationPlanManifestV3, context: AdmissionValidationContext, resolver: ContentResolver, ) -> EvaluationFeasibilityV3: ... def authorize_evaluation( manifest: EvaluationAdmissionManifestV3, feasibility: EvaluationFeasibilityV3, context: AdmissionValidationContext, custody: PopulationCustodyPort, resolver: ContentResolver, ) -> EvaluationPreparationV3: ... def score_and_replay( admission_manifest: EvaluationAdmissionManifestV3, feasibility: EvaluationFeasibilityV3, preparation: EvaluationPreparationV3, receipt: PopulationUseReceipt, release: AdmissionCaseReleaseV3, cases_by_arm: tuple[ArmAdmissionCasesV3, ...], context: AdmissionValidationContext, custody: PopulationCustodyPort, resolver: ContentResolver, ) -> EvaluationReplayV3: ... def validate_evaluation_replay( replay: EvaluationReplayV3, context: AdmissionValidationContext, custody: PopulationCustodyPort, resolver: ContentResolver, ) -> tuple[ArmEvaluationReportV3, ...]: ... def authorize_fitted_component_v3( evidence: FittedComponentEvidence, probe_inputs: tuple[ProbeEvaluationInput, ...], fitting_lineage: LineageManifest, repository_holdout: RepositoryHoldoutReport, replay: EvaluationReplayV3, key: ArmScopeKey, context: AdmissionValidationContext, custody: PopulationCustodyPort, resolver: ContentResolver, ) -> None: ... ``` Preconditions require exact V3 root types and canonical ordering. The fitted consumer first validates the complete replay, selects exactly one planned arm/scope, requires its completed GO result, and refuses a raw arm report, aggregate, V2 replay, incomplete graph, or ambiguous selection. It then performs the unchanged fitted-component checks: it replays every probe, validates fitting lineage, proves the resolved training and repository-holdout populations are disjoint, binds the selected model declaration and component, and compares derived probe digests. V3 replaces only the admission evidence source; it does not weaken 13.2. ### Error and Result Semantics - Pre-spend insufficiency is returned as `EvaluationFeasibilityV3` with `may_spend=False`; no preparation can be constructed. - Unreadable observation bytes, unknown/mixed versions, stale custody, disposition change, or incomplete slots are validation errors before spend. - After spend, missing cases/reports, invalid release/proof, denominator zero/mismatch, skipped stages, or unread inputs invalidate completion. They do not become insufficiency or quality decisions. - GO/NO-GO exists only inside completed `ScopeAdmissionResultV3` objects. - Aggregate and per-arm report objects have no consumer-authorization method. ### Testing Strategy - V2 compatibility: freeze representative canonical bytes, digests, decisions, failures, and replay; prove helper extraction and V3 imports do not change them (`16.1–16.4`). - Version/identity: reject V2-as-V3, unknown schemas, subclasses, mixed children, and every single-field baseline mutation; require a new baseline (`16.3–16.6`). - Plan/feasibility: cover each arm/scope Cartesian row, threshold reachability, minimum-size and interval precision, one insufficient row blocking all spend, and no arm launch or label read (`17.1–17.4`). - Observation/authority: mutate, remove, swap, truncate, zero, or cross-bind every slot; reject self-review, stale custody, broken/forked attempt history, raw-access disposition, changed predecessor retry identity, and any sealed event after a terminal event (`17.5–17.12`). - Atomic spend: prove one request/receipt/spend binding covers all planned baselines, concurrent losers receive no labels, and no subset preparation is constructible (`17.7–17.9`). - Exact metrics: exercise multi-family fixes, same-ecosystem ordinary projection, cross-ecosystem refusal, exact precision, Wilson fields, whole-push deduplication, aggregate weights/no decision, and post-spend zero/mismatch invalidity (`18.1–18.8`). - Release/combinations/proof: reject V2 release, role/pair/scope/arm/input/result swaps, missing components, contradictory combined results, candidate fabrication/transfer, and invalid attachments; retain absent proof as not demonstrated (`19.1–19.11`). - Reports/replay: derive every scoped GO/NO-GO, preserve whole operational evidence, reject missing gate/arm/scope/custody/release/lineage, rederive all source values, and authorize consumers only from complete replay (`20.1–20.8`). - Synthetic end to end: two component arms plus one combined arm over at least two families and two ecosystems pass plan review, per-scope preflight, sealed observations, one shared spend, V3 release, scoped reporting, complete replay, and exact fitted selection. ### Migration Strategy 1. Freeze V2 accepted outputs and failure behavior. 2. Add V3 measurement identities, plan declaration, exact policies, and preflight. 3. Add sealed observations, reviewed admission manifest, authority, and spend binding. 4. Add V3 release, exact metrics, reports, and complete replay. 5. Add the separate fitted-component V3 consumer and strict schema router. 6. Run synthetic multi-arm conformance and independent validation. 7. Unblock valid-detection-benchmark task generation only after the predecessor schema `openultrasast.detection-core.evaluation-admission.v3` is exported and validated. There is no V2-to-V3 migration. Existing V2 artifacts remain replayable under V2; every V3 evaluation starts a new baseline. Rollback removes the additive V3 facade/modules and leaves the frozen V2 path unchanged. ## Data Models ### Domain Model and Invariants | Aggregate | Authoritative identity | Invariants | |---|---|---| | `EvidencePacket` | Digest of canonical deterministic evidence | No model/proof data; all roles declared; every relation has one status | | `Decision` | Packet digest plus family decision, basis and decision provenance | One neutral outcome; bounded satisfaction; named indeterminate reasons; model/proof identity retained outside packet | | `EvidenceRecord` | Digest of packet and decision | Decision packet digest matches; immutable derivation only | | `FamilyContract` | Family, version and canonical contract digest | Closed fields, roles, decisions, residuals and reasons | | `FamilyCatalog` | Canonically sorted exact contract set | Every digest re-derived; unique family/version/schema; exact ecosystem resolution | | `FactPack` | Contract digest plus normalized pack and review identity | Stable fact IDs; reviewed content; valid scope and structural requirements | | `FactMatchEvidence` | Pack, fact, exact structural observations, matcher behavior and lineage | Complete relation only from the full validated requirement set and current uncontaminated matcher lineage | | `ReadinessResult` | Stage, analysis identity and predecessor digests | Positive, negative and ambiguity evidence; exact evidence use; no skipped predecessor | | `ModelResult` | Request, packet, response and program identities | Preserved response; frozen program declaration and validated lineage; declared mapping; no evidence rewrite or role collision | | `EvidenceValidationInput` | Record, named inputs, optional model derivation and input digest | Full provenance/fact/model replay before any claim, metric, or gate | | `ReversedPairProbe` | Reviewed semantic projections and component digest | Equal security semantics, reversed surface shape and pre-registered tolerance | | `PopulationCustodyLog` | Embedded requests, contiguous receipts and current head | Complete current history; reviewed custodian; spent status is monotonic | | `PopulationCensus` | Population, scope cardinalities, payload and independent review attestation | Aggregate and label-blind; no case IDs; same population as manifest; no self-review | | `BaselineIdentity` | Arm, population/freeze, claimed scopes, contract set, fact schema/set, acquisition, analyzer, configuration, model set, denominator, threshold set and outcome mapping | Any changed field creates a new baseline | | `AdmissionPreparation` | Manifest/context authorization and expected custody head | Feasible pre-registration; label-release request spends before bytes open | | `AdmissionCaseRelease` | Population, manifest, authorization, preparation, spend receipt, custodian and attested case bindings | Post-spend only; exact one-to-one case/result/evidence/change/input binding; scoped roles and pairs | | `AdmissionReportV2` | Baseline, validation context, authorization, spend receipt, case release and case/result digests | Canonical result data; in-scope scoring; all gates reported; never an authorization token by itself | | `AdmissionReplay` | Canonical digest of the exact authoritative scoring inputs and derived report | Immutable transport bundle; independently replayed against current context, custody and content before trust | | `RepositoryHoldoutReport` | Component, disjoint repository manifests and measurement | Explicit holdout role; matches fitting lineage and component identity | | `DifferentialProofRecord` | Reviewed producer, control-bound sides and resolved witness/verification receipt | Affected change identity, reviewed safe control, trusted producer/store policy | | `PushVerdict` | Request and record digests | No unbounded clean result; proof basis does not imply rollout permission | ### Canonical Data Contract - Wire format is JSON object schema `openultrasast.detection-core.v2`. - Enum values use their declared lowercase strings. - Mappings serialize in key order; semantic sequences retain declared order. - Sets are prohibited at the wire boundary and normalized to sorted tuples before record construction. - Non-finite numbers, duplicate IDs, unknown fields, and unknown schema versions are refused. - SHA-256 digests use the full lowercase hexadecimal representation in v2. Legacy truncated identities remain confined to the v1 decoder. No database schema is introduced. Benchmark workflows choose storage and preserve canonical bytes and digests. ## Error Handling ### Error Strategy Invalid contracts or identities fail fast. Expected analysis uncertainty becomes a typed indeterminate record. Readiness and admission failures are state outcomes, not exceptions that can be mistaken for a clean scan. | Category | Examples | Core response | |---|---|---| | Instrument failure | Unreadable or zero-byte tracked input | Invalidate run; emit no clean or satisfied result | | Contract violation | Unknown field, role, decision, schema, or ecosystem | Refuse record with exact field and contract identity | | Semantic uncertainty | Missing relation, ambiguous match, unmet condition | Preserve partial packet; return indeterminate with named reasons | | Model failure | Timeout, parse error, identity mismatch | Preserve deterministic decision; record model failure and coverage | | Proof failure | Inconclusive, mismatched side, missing witness | Refuse attachment; preserve prior basis | | Readiness failure | Failed, missing, or stale predecessor | Stop promotion; report layer and correction lever | | Admission configuration | Impossible threshold or wrong denominator | Refuse before label opening; do not report no-go measurement | | Budget failure | Elapsed time exceeds derived budget | Name change, elapsed time and derivation; return bounded failure | | Provenance mismatch | Any digest cannot be re-derived | Refuse affected result and dependent later receipts | | Review mismatch | Self-review, unknown reviewer, copied attestation, or unreadable disclosure | Refuse before feasibility or semantic use | | Custody mismatch | Old/incomplete log, broken chain, or prior spend | Refuse use/admission; preserve monotonic spent status | | Case release mismatch | Pre-spend release, wrong custodian/receipt, missing/extra case, role/pair swap, or operational payload substitution | Refuse before metric production | | Migration-boundary failure | Unreadable source, open inventory entry, legacy/shim import, dynamic core access, or v1 writer reference | Refuse cutover; report exact path, symbol and dependency kind | ### Monitoring Measurement artifacts carry stage counts, missing reasons, model-call failures, instrument failures, runtime, and identity digests. They do not contain secrets, raw model credentials, exploit payloads, or repository contents. Operator notification wiring remains outside this spec and consumes only status, counts, and digests. ## Testing Strategy ### Unit Contract Tests - `test_core_canonical.py`: deterministic bytes under mapping-order variation; sequence order retention; non-finite number and unsupported-value refusal (`1.4`, `10.4`). - `test_core_evidence.py`: all relation kinds, per-relation status, product/fact provenance, packet-decision binding, and immutable derivation (`1.1–1.3`, `2.1–2.5`, `4.1–4.3`). - `test_core_family.py`: family registration matrix, canonical catalog and exact resolution, ecosystem/schema refusal, closed roles/fields/decisions, required evidence, residual prerequisites, and a complete `FamilyCapabilityDeclaration` whose contract, readiness fixture, development population, and admission boundary fields are each removed or corrupted parametrically and refused (`3.1–3.4`, `9.1`, `14.5`). - `test_core_coverage.py`: exact stage counts, relation-bucket exclusivity, nonzero read proof including a missing receipt, and first failed-output diagnosis at every stage (`5.1–5.5`). - `test_core_budget.py`: transition-based complexity, derivation metadata, deterministic class budget, resolved changed-byte measurement, absent/zero timing or complexity, scope escape, and named gate failure (`6.1–6.4`). - `test_core_verdict.py`: exact claim location, bounded satisfaction, reasoned silence, mixed complete/incomplete paths, proof trust/store/control binding, and internal attachment aggregation (`7.1–7.4`, `9.6`). - `test_core_readiness.py`: exact stage order, fixture triplets, stopped promotion, schema/vocabulary/outcome-mapping transitive staleness, and enforced regression/development/fresh-admission/integration evidence-use separation (`8.1–8.6`). - `test_core_model.py`: request/result identities, packet binding, actual program lineage, current custody-chain validation, allowed mappings, inline and content-addressed response retention, proposer/adjudicator separation, model failure, and basis limits (`9.1–9.5`, `10.3`, `13.1`). - `test_core_validation.py`: full context/input digest reconstruction, exact catalog and pack/model membership, resolver-backed provenance, deterministic/model derivation replay, and proof that admission and consumer entry points cannot bypass the same validator with a raw record (`4.4`, `10.1–10.4`, `12.6`, `14.1`). - `test_core_provenance.py`: complete stamp, field-by-field tamper refusal, lineage acceptance, reviewed custody, canonical use/spend receipts, contamination, and refusal of omitted/old-prefix spend history. A matcher-specific case proves that label-influenced matching behavior creates a chained `SPENT` receipt and a later claimed `AVAILABLE` state cannot re-admit the population; wrong influenced-artifact expectations are refused even when manifest subjects match (`10.1–10.4`, `13.1`, `13.3–13.4`). - `test_core_factpack.py`: schema/scope/review matrix, structural match, name-only negative control, exact requirement-ID observations, fabricated match refusal, source/documentation/independent-behavior review and lineage, ambiguity, and partial-pack behavior. For a dependency-fact-only role it refuses product provenance, preserves a missing fact as incomplete/indeterminate, and accepts the same effect only from a reviewed pack plus valid structural match. Direct `apply_fact` use with an unreviewed pack, wrong review policy, or protected-population matcher contamination is refused (`4.4`, `10.4`, `11.1–11.5`, `13.3–13.4`). - Downstream conformance fixture: inject a complete family-owned contract, reviewed pack, persisted structural match, lineages, policies, custody and resolver; prove generic core validation succeeds while name-only, partial, substituted, self-authored, contaminated, or legacy-projection-only inputs are refused (`10.2`, `11.1–11.5`, `14.3`). - `test_core_admission.py`: consume-only refusal order, readiness prerequisites, threshold feasibility over label-blind census capacities and Wilson bounds, the historical impossible-denominator regression, scoring derived from full case evidence, fixed precision/Wilson-95 schema, and refusal of missing standard metrics or gates. It also covers spend-before-label crash/concurrency order and refuses a forged family decision, substituted contract, invalid fact review, or contaminated model/threshold/matcher lineage before metrics or gates. Manifest and census self-review, unknown reviewers, copied attestations, and unresolved conflict disclosures fail before feasibility. A post-spend release matrix rejects a wrong custodian or receipt, missing/extra/duplicate bindings, cross-scope pairs, role swaps, result swaps, and rehashed evidence/change/input substitutions before metric production (`12.1–12.6`, `13.1`, `13.4`). - `test_core_admission_report.py`: evidence-backed gate derivation, exact report identity, `AdmissionReplay` construction and authoritative revalidation, configured-model zero-call refusal, claimed-scope undeclared-model refusal, out-of-scope-only model-call isolation, and coordinated report/readiness/threshold/case substitution refusal. Raw reports cannot authorize later consumers (`12.6`, `13.1–13.2`, `15.1–15.5`). - `test_core_measurement.py`: required schema, basis stratification, Wilson counts, baseline mutation matrix, zero receipts, and prerequisite stage result/digest references. A two-arm case over one population keeps family/ecosystem scope, census rows, denominator rules, and reports isolated and refuses cross-arm aggregation as one baseline (`11.6`, `15.1–15.5`). - `test_core_probes.py`: reject unequal semantic projections as invalid; catch a deletion-count component on core-derived, independently reviewed equal-semantics/reversed-surface pairs; reject digest-only review; accept a relation-based component; reject a fabricated passed summary by re-evaluating the canonical probe inputs; and require disjoint resolved repository holdout, fitting lineage, and untouched admission (`13.2`). ### Reference-Family Conformance Both test-only families run the same positive, negative, and missing-evidence contract: - Obligation family: operation plus required constraint; exercises control and constraint relations. - Taint-flow family: source, transformations, sink, and sanitizer state; exercises ordered value flow. Each family must produce violation, bounded satisfaction, and indeterminate records without adding a family-specific field to a core module (`3.5`, `8.2`). These fixtures are regression data only and do not establish product capability or admission. ### Adapter and Integration Tests - Authorization adapter converts existing traces conservatively; missing per-hop completeness remains indeterminate and no target vocabulary enters core (`14.3`). - Hunt contract adapter translates a fake executor result; core imports no DSPy and accepts only a matching, declared result. DSPy/provider execution remains hunt-owned and outside this spec (`9.1–9.5`). - Search adapter accepts a canonical demonstrated differential record from a reviewed producer and trusted resolver with matching change/control, side, receipt, store, and witness digests; rejects self-asserted, opaque, inconclusive, same-side, or mismatched records (`9.6`, `14.2`). - Push request scope through evidence and verdict preserves silence reasons and accounts for the measured deadline (`6.4`, `14.1`, `14.4`). - Readiness failure prevents development, admission, and integration calls; a semantic digest change makes every dependent receipt stale (`8.3–8.5`, `12.1`). - Benchmark custody integration proves the guarded operation durably appends the exact head-bound spend before releasing label bytes; injected crashes before/after append and two concurrent preparations release labels to at most the compare-and-swap winner (`12.2`, `13.4`). - Legacy and v2 validators coexist by explicit schema version; committed historical reports remain byte-unchanged and cannot be written through the v2 API (`14.3`, `15.3`). - Namespace-isolation tests move every legacy definition out of mixed v2 modules while the temporary top-level named API remains byte-compatible. Direct imports of moved symbols from mixed modules fail, proving there is one compatibility owner (`14.3`). - The migration-boundary test proves every scoped input is readable and nonzero, requires the frozen inventory to shrink without additions, and reports explicit legacy namespace, shim, top-level symbol, writer-facade, module-valued core, and dynamic core access sites. Adversarial assignment, `try`/`finally`, comprehension, and generic-call aliases do not affect the result because the gate never interprets use flow (`14.3`, `15.3`). - Cutover tests delete the compatibility package and shims in an isolated tree, verify old top-level and submodule imports fail, verify v2 imports and writes still succeed, and validate the frozen v1 bytes through `legacy.py` (`14.3`, `15.3`). ### End-to-End Readiness Test One small authored repository per reference family runs through: ```text nonzero input read → semantic recognition → typed transitions → security object and constraints → vulnerable and fixed pair → development report → synthetic admission manifest → guarded label-release spend → consumer verdict ``` Each run injects one expected failure at every stage and confirms later stages refuse to act. Synthetic admission manifests test orchestration without spending a real population. ### Performance and Load Tests - Canonicalization and validation measure elapsed time against relation and observation counts and publish the workload sample; no static latency claim is introduced. - Budget derivation uses recorded changed regions, transitions, bytes, total elapsed time, sample count, percentile, workload, and hardware. - Large transition sets verify linear validation behavior and bounded memory without loosening semantic checks. - Optional model calls and proof attachment processing remain inside the total push deadline. ## Security Considerations - Model outputs, fact packs, manifests, and proof attachments are untrusted until validated against their contract and identities. - A raw evidence record is likewise untrusted; admission and consumer paths both replay the shared evidence-validation composition before deriving output. - Content-addressed model-response, fact-match, review, complexity, control, and proof-witness references are accepted only when their full digest, byte length, media type, retention policy, storage receipt, and trusted producer/store policy match; an opaque location alone is never evidence. - Source text in an evidence packet is data, never an instruction channel; the hunt-owned DSPy runtime uses structured signature fields and preserves prompt/program identities. - No model response can create a semantic relation, claim execution-backed basis, or bypass readiness/admission. - Fact review binds normalized content, schema, scope, structural matches, independent reviewer attestation, and review evidence in one digest. - Label lineage is declarative and reviewable; the core can detect declared contamination but cannot discover undisclosed training data. External review remains the trust boundary. - Admission labels open only after analyzer, facts, matcher behavior, model, configuration, readiness, scope, denominator, thresholds, and the current complete custody head are frozen, independently reviewed where required, and validated. The custodian durably appends the head-bound spend before releasing labels; scoring accepts only that current-chain receipt. - Proof attachment verifies resolved identity, producer receipt, reviewed control binding, and differential outcome; trusted search infrastructure retains sandbox and oracle execution responsibility. - Migration never treats a successful alias analysis as proof that v1 is absent. Legacy write capability is isolated, inventory-closed, and deleted; module-valued/dynamic access to the compatibility surface is prohibited so a hidden dependency fails the boundary check or import after cutover. ## Performance and Scalability - Core validation is proportional to fields, transitions, observations, receipts, or cases supplied to the corresponding contract; no whole-repository traversal occurs in core. - Resolver-backed artifacts are length-checked before loading and bounded by the applicable provenance, response, review, or proof policy; oversized content is refused before parsing. - Change scoping is supplied by the push runner and recorded immutably. The core rejects evidence or claim anchors outside that scope. - Budget is derived per workload class from measurements and includes preparation, analysis, model, proof attachment, validation, and reporting time. - A changed canonical schema, complexity vocabulary, or workload-class assignment starts a new baseline. ## Migration Strategy ```mermaid flowchart LR V1[Mixed legacy surface] --> Isolate[Isolate legacy namespace] Isolate --> Inventory[Freeze migration inventory] Inventory --> V2[Add v2 contracts] V2 --> References[Reference conformance] References --> Adapters[Adjacent adapters migrate] Adapters --> Zero[Syntax and inventory zero] Zero --> RemoveV1[Delete v1 writer and shims] RemoveV1 --> PublishV2[Publish v2 top level] PublishV2 --> LegacyRead[Retain read only decoder] ``` 1. **Foundation:** add canonical, semantics, evidence, family, and reference-family contracts without changing v1 behavior. 2. **Namespace isolation:** move all v1 contracts and serialization behind `legacy_v1`, convert the old named API to explicit shims, remove legacy definitions from mixed v2 modules, and freeze the reviewed migration inventory. 3. **Adapter migration:** move authorization conversion out of engine trace code and emit v2 in development. Preserve conservative incomplete hops and existing family behavior. 4. **Governance migration:** add readiness, measurement, admission, lineage, and model v2; historical v1 validation remains separate. This creates a new baseline. 5. **Consumer conformance:** add DSPy and proof adapters and validate push consumption with synthetic/reference records. Downstream rollout remains disabled. 6. **Structural zero gate:** close every frozen inventory entry and require zero explicit compatibility imports, top-level legacy symbols, shim imports, dynamic core access, and v1 writer-facade references. The checker uses syntax only and fails on unreadable input. 7. **Single-writer cutover:** delete `legacy_v1`, the writer facade, and compatibility shims; prove old imports fail; then publish the dependency-free v2 top-level surface. 8. **Historical retention:** retain `legacy.py` read-only validation for committed v1 artifacts. It exposes no analyzer, contract constructor, converter, or writer API. Rollback is allowed before the single-writer cutover by retaining the isolated compatibility namespace and shims. After cutover, rollback means reverting the complete migration change; it must not restore parallel v1 writes. No migration phase re-runs authorization admission or changes its parked status. ## Supporting References - [Research and design decisions](research.md) - [Python JSON documentation](https://docs.python.org/3.11/library/json.html) - [DSPy signatures](https://dspy.ai/3.0.4/learn/programming/signatures/) - [DSPy modules](https://dspy.ai/3.0.4/learn/programming/modules/)