From 1eb9ef11b51b1c1921e4b051092536cb151c0bc7 Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Sat, 31 Oct 2026 12:35:42 +0100 Subject: [PATCH] docs(detection-core): approve v2 implementation plan --- .kiro/specs/detection-core/design.md | 2699 ++++++++++++++++++-- .kiro/specs/detection-core/requirements.md | 386 ++- .kiro/specs/detection-core/research.md | 472 ++++ .kiro/specs/detection-core/spec.json | 2 +- .kiro/specs/detection-core/tasks.md | 517 ++-- 5 files changed, 3650 insertions(+), 426 deletions(-) diff --git a/.kiro/specs/detection-core/design.md b/.kiro/specs/detection-core/design.md index 7d83b2c..f8379b9 100644 --- a/.kiro/specs/detection-core/design.md +++ b/.kiro/specs/detection-core/design.md @@ -2,227 +2,2600 @@ ## Overview -**Purpose:** detection-core specifies the single security object every detection arm reuses: -the canonical trace with its verdict, coverage, budget, provenance, and admission contracts. -It extracts what `access-control-prover` already implements for one family into a -family-agnostic contract, so arms, fact packs, and consumers stop maintaining parallel -runner contracts. +**Purpose:** detection-core provides the family-neutral, versioned contract between +deterministic program analysis, vulnerability-family reasoning, bounded model programs, +execution proof, measurement, and downstream consumers. It makes semantic incompleteness a +first-class result and stops a family at the earliest failing readiness stage. -**Users:** spec authors and implementers of detection arms (prover, hunter, push hook, -proof arm) consume the contract; the maintainer reads admission GO/NO-GO over it. +**Users:** vulnerability-family implementers produce evidence through adapters; DSPy and +proof adapters consume immutable evidence; benchmark workflows validate readiness and +admission; pre-push and reporting consumers receive supported claims, bounded satisfaction, +and bounded silence. -**Impact:** replaces per-arm envelopes with one contract; existing behavior stays -byte-identical per consumer during migration (the 2026-10-30 admission suite is the -regression gate). +**Impact:** the current authorization-shaped core becomes a temporary v1 compatibility +surface. A neutral v2 write contract replaces it through a finite adapter migration. +Authorization remains parked and supplies regression evidence only. Historical v1 artifacts +remain readable but are never rewritten or used as a v2 trend baseline. ### Goals -- One canonical trace schema and three-way verdict vocabulary shared by all arms (Req 1, 2, 5). -- Coverage accounting that makes every zero explainable (Req 3); instrument rule enforced at core level. -- A measured, per-workload-class time budget instead of a static constant (Req 4). -- Provenance digests on every analysis, with refusal on mismatch (Req 6). -- A generalized admission gate over untouched populations (Req 7), no fitting on labels with - reversed-pair probes (Req 8), models bounded to residuals (Req 9), families as fact packs - with scope disclosure (Req 10), consumer contracts (Req 11), fixed measurement schema (Req 12). + +- Define one typed, canonical v2 evidence record whose vocabulary is independent of any + vulnerability family. +- Let each family declare its security object, semantic roles, decisions, completeness + rules, and model residuals without changing core code. +- Expose instrument, recognition, transition, constraint, pair, development, admission, + and integration failures before a later gate can run. +- Restore the deterministic-evidence-to-DSPy loop through a dependency-free model handoff. +- Preserve one execution-backed route to blocking without moving proof execution or rollout + policy into the core. +- End migration with one live write contract and two unlike reference-family conformance + suites. ### Non-Goals -- New vulnerability families or their fact content. -- The pre-push transaction itself (pre-push-safety-net consumes this core). -- Fitted models anywhere in the core path. -- Engine/graph internals (contributor-scan). + +- Implementing vulnerability-family algorithms, prompts, examples, or model optimization. +- Expanding framework vocabulary or analyzing vendored dependency source. +- Running DSPy, Joern, benchmark acquisition, proof sandboxes, or pre-push transactions + inside `openultrasast.core`. +- Reviving or re-admitting the parked authorization arm. +- Enabling any capability for advisory or blocking rollout. +- Providing cross-language canonical signatures; v2 is a Python-runtime contract. ## Boundary Commitments ### This Spec Owns -- The canonical trace schema (entry, hops, operation, witnesses, guards, judgment, completeness, digests) and its deterministic serialization. -- The verdict contract: proven claim (witness attached) vs bounded silence (coverage reason attached); indeterminate never converts to clean. -- Coverage accounting schema (entry recognition, per-call resolution buckets, unchecked reasons) and the zero-byte instrument refusal. -- The measured-budget contract: complexity features recorded per change, per-workload-class budget derivation published with sample/percentile/hardware, configurable budget. -- Provenance stamping (analyzer/fact/acquisition/configuration/population/input digests) and refusal on digest mismatch. -- The generalized admission contract: consume-only manifest validation, paired scoring, gates, GO/NO-GO report schema, family-scope disclosure. -- The fact-pack admission contract (reviewed facts, provenance stamping, family tag). -- The consumer interface for the pre-push transaction and the proof layer. + +- The v2 canonical JSON profile and digest rules for core records. +- Typed semantic transitions, observations, completeness, analyzed scope, decisions, + neutral outcomes, and evidence bases. +- The versioned family contract and validation of declared ecosystems, roles, fields, + decisions, outcome rules, and residual questions. +- Coverage funnels, instrument refusal, measured-budget records, provenance and label + lineage, fact-pack validation, readiness prerequisites, admission feasibility/scoring, + and measurement schemas. +- Dependency-free model request/result contracts and proof-attachment validation. +- Push-consumer interfaces for defect claims, bounded satisfaction, and bounded silence. +- Thin adjacent-owner translation seams that prove those public contracts integrate; they + receive existing subsystem records and perform no analysis, model, proof, or persistence + orchestration. +- A finite v1 migration and read-only historical decoder. ### Out of Boundary -- Pre-push hook transaction, rollout policy, capability registry (pre-push-safety-net). -- Family fact content (fact-pack specs, starting from access-control-prover's rulesets). -- Hunter/proposer internals and proof-arm internals (they consume the contract). -- Engine/CPG internals (contributor-scan). + +- Family analysis and abstract interpretation, including source/sink, dominance, + sanitization, constraint, or configuration algorithms. +- Target-specific facts, framework strategies, and fact acquisition. +- DSPy execution, prompt construction, caching, optimization, and provider transport. +- Population construction, label custody, readiness-artifact persistence, and run + orchestration. +- Proof search, sandbox execution, oracle semantics, and demonstration qualification. +- Push scoping policy, alert rendering, capability rollout, and BLOCK enablement. ### Allowed Dependencies -- `valid-detection-benchmark` pools and freeze artifacts (read-only; never label-tuned). -- Existing prover implementation as the first contract implementation. -- Steering rules in `detection-quality.md` (no fitting on pool labels; instrument rule; stop conditions). + +- `openultrasast.core` depends only on Python 3.11 standard-library modules and lower core + modules identified by the dependency direction below. +- Core may consume immutable, typed records produced by benchmark, analyzer, model, and + proof adapters; it never imports those packages. +- Core may call an injected read-only `ContentResolver` protocol to verify referenced bytes + and storage receipts; concrete stores import or implement the protocol and core never + writes through it. +- Core may read the current complete population-use chain through + `PopulationCustodyPort`; the benchmark custodian remains the sole append-only writer. +- Authorization, DSPy, search-proof, benchmark, and push adapters may import the public core + contract. +- Development and admission populations are read-only identities at the core boundary. +- No core module imports `openultrasast.semantic`, `openultrasast.hunt`, + `openultrasast.search`, `openultrasast.tools`, `benchmarks`, or `dspy`. ### Revalidation Triggers -- Trace schema or verdict vocabulary changes → every arm and consumer re-validates. -- Admission report schema changes → every admission gate re-runs. -- Budget derivation method changes → new baseline under Req 12 (not a trend). -- Fact-pack admission contract changes → all fact packs re-reviewed. + +- Changing the canonical profile, v2 schema, field-kind vocabulary, relation kinds, + outcomes, or evidence bases revalidates every family adapter and consumer. +- Changing a family contract, fact schema, model program identity, or outcome mapping + invalidates dependent readiness receipts and starts a new measurement baseline. +- Changing readiness stage order or prerequisite rules revalidates development, admission, + and integration workflows. +- Changing denominator policy, admission metrics, or threshold semantics requires new + admission configuration and baseline identity. +- Changing proof-attachment identity rules revalidates search-with-proof and every + proof-consuming verdict path. +- Adding a non-Python writer or cryptographic wire signature requires a canonicalization + design review. ## Architecture ### Existing Architecture Analysis -Today each arm owns its envelope: `prover_arm.py` (records + summarize), `development_runner.py` -(bundle + funnel + gates), `arm_regression.py` (arm records), `admission_run.py` (manifest + -paired scoring). The trace itself (`Trace`, `ProverResult`) exists in the prover. The gap is -the missing shared contract layer, not missing analysis code. -### Architecture Pattern & Boundary Map +The existing `src/openultrasast/core/` package already contains deterministic serialization, +budget derivation, coverage buckets, digest verification, fact-pack loading, admission +manifest refusal, probes, and consumer records. Its public `TraceRecord` requires entry, +operation, resource, authority, guard, and witness fields, while `Judgment` is exactly +`unguarded`, `discharged`, or `indeterminate`. Tests pin those fields and values. -Pattern: **thin core contract package; existing implementations adopt it incrementally** -(hybrid option C from research.md). The core owns contracts only — no analysis logic. +The authorization producer converts directly from `semantic/obligations/trace.py`, and the +benchmark admission schema scores authorization judgments. A zero-shot DSPy experiment +exists in `benchmarks/valid/judge_pass.py`, but no core model handoff exists; one current +test explicitly prohibits a public model path. Production push and proof paths do not yet +consume the core end to end. The migration therefore preserves reusable mechanisms while +replacing the public domain model. + +### Architecture Pattern and Boundary Map + +The selected pattern is a **versioned ports-and-adapters contract with a finite compatibility +migration**. Core contracts and pure validators form the domain. External systems implement +adapters and retain execution ownership. ```mermaid flowchart LR - subgraph CORE[detection-core contract package] - T[trace schema + verdict] - C[coverage accounting] - B[measured budget] - P[provenance digests] - A[admission contract] - F[fact-pack contract] - end - subgraph ARMS[arms / fact packs] - PR[access-control-prover arm] - FP[family fact packs] - HU[hunter proposal lane] - end - subgraph CONSUMERS[consumers] - PP[pre-push-safety-net] - PR2[search-with-proof] - BM[valid-detection-benchmark] - end - ARMS --> CORE - CORE --> CONSUMERS - BM -. measures .-> CORE + subgraph Producers[Evidence producers] + Analyzer[Deterministic analyzer] + Packs[Semantic fact packs] + end + subgraph Core[Detection core v2] + Semantics[Semantic evidence] + Family[Family contract] + Readiness[Readiness validation] + Decision[Decision and verdict] + Admission[Measurement and admission] + end + subgraph Adapters[External adapters] + ModelAdapter[Model adapter] + ProofAdapter[Proof adapter] + LegacyAdapter[Legacy adapter] + end + subgraph Consumers[Consumers] + Benchmark[Benchmark workflow] + Push[Pre push consumer] + end + + Analyzer --> Semantics + Packs --> Semantics + Semantics --> Family + Family --> Readiness + Family --> Decision + Readiness --> Admission + Decision --> Admission + Semantics --> ModelAdapter + ModelAdapter --> Decision + ProofAdapter --> Decision + LegacyAdapter --> Semantics + Admission --> Benchmark + Decision --> Push ``` +**Dependency direction:** + +```text +canonical + → semantics + → provenance + → evidence + → family, coverage, budget + → factpack, model + → validation + → readiness, verdict, probes + → measurement + → admission + → consumer +``` + +Each core layer imports only layers to its left. `legacy.py` may import public v2 types to +decode historical input but no v2 module imports `legacy.py`. External adapters import the +public core package and their respective execution subsystem. + +**Architecture integration:** + +- Existing frozen dataclasses, enums, SHA-256 identities, and canonical JSON patterns are + retained and strengthened. +- Family-specific complex structure lives in ordered transitions and observations; family + payload fields remain shallow, closed, and typed. +- Benchmark workflows produce and persist readiness artifacts; core validates their state + and freshness. +- The model and proof paths derive decisions tied to an unchanged deterministic packet + digest; neither can rewrite analyzer evidence. +- Historical v1 readers remain validation-only after migration; all new writes use v2. + ### Technology Stack -| Layer | Choice | Role | Notes | +| Layer | Choice and Version | Role | Notes | |---|---|---|---| -| Core contract | Python, frozen dataclasses + canonical JSON | trace/verdict/coverage/budget/admission schemas | mirrors existing Trace.record() conventions | -| Runners | existing benchmark runners | adopt core contracts incrementally | behavior byte-identical per consumer | -| Tests | pytest | contract conformance per consumer | admission suite is the regression gate | +| Core contracts | Python 3.11 or newer, frozen dataclasses, enums, protocols | Typed immutable domain and validation | Standard library only; no unrestricted `Any` in public contracts | +| Serialization | UTF-8 JSON and SHA-256 | Canonical bytes, record IDs, baseline identities | Sorted keys, compact separators, sequence order retained, non-finite numbers refused | +| Adjacent model runtime | Existing optional DSPy `>=3.4,<3.5` | Hunt-owned execution of family-declared residual questions | Never imported by core; results cross `hunt/model_contract.py` | +| Artifact persistence | Existing benchmark JSON artifacts and stores | Persist readiness, development, and admission records | Core defines records but performs no I/O orchestration | +| Tests | pytest 8 or newer | Contract, adapter, migration, and readiness validation | Two unlike test-only reference families | + +No new runtime dependency is introduced. ## File Structure Plan ### Directory Structure -``` -src/openultrasast/core/ # NEW: the contract package (no analysis logic) -├── __init__.py # public contract exports -├── trace.py # canonical trace schema, deterministic serialization (Req 1, 2) -├── verdict.py # proven-claim vs bounded-silence contract (Req 5) -├── coverage.py # entry/resolution/unchecked accounting + zero-byte refusal (Req 3) -├── budget.py # measured budget: complexity features, per-class derivation (Req 4) -├── provenance.py # digest stamping + mismatch refusal (Req 6) -├── admission.py # manifest validation, paired scoring, gates, GO/NO-GO schema (Req 7, 10) -├── probes.py # reversed-pair / holdout probes for fitted components (Req 8) -└── factpack.py # family fact-pack admission contract (Req 10) -tests/test_core_trace.py # conformance tests per contract module -tests/test_core_admission.py -tests/test_core_budget.py + +```text +src/openultrasast/core/ +├── __init__.py # Public v2 exports only after migration +├── canonical.py # Canonical bytes, identities, content references and read-only resolver port +├── semantics.py # Locations, nodes, transitions, observations, completeness +├── evidence.py # Scope, packet, decision, evidence record, neutral enums +├── family.py # Family contract, catalog, field and decision validation +├── coverage.py # Semantic stage funnel, relation buckets, instrument gate +├── budget.py # Change complexity, workload derivation and elapsed-time gate +├── provenance.py # Input/decision identities, lineage and population-use receipts +├── factpack.py # Pack review, structural match validation and fact application +├── model.py # Dependency-free model request, identity and result contracts +├── validation.py # Closed evidence context and full record revalidation seam +├── readiness.py # Ordered stages, receipts, freshness and promotion rules +├── verdict.py # Claims, satisfaction, silence, proof attachment and aggregation +├── probes.py # Reviewed semantic-equivalence probes and fitted-component evidence gates +├── measurement.py # Complete baseline identity, rates, funnel and report schema v2 +├── admission.py # Census, feasibility, evidence-derived scoring and fitted-component gate +├── consumer.py # Scoped push request and verdict consumption +└── legacy.py # Read-only v1 trace and report decoder and validator + +src/openultrasast/semantic/obligations/ +└── core_adapter.py # Authorization trace to neutral v2 evidence + +src/openultrasast/hunt/ +└── model_contract.py # Translate injected executor I/O; no DSPy execution ownership + +src/openultrasast/search/ +└── core_adapter.py # Demonstrated verification record to ProofAttachment + +benchmarks/valid/ +├── readiness_contract.py # Construct and validate receipts; caller owns persistence/sequence +├── development_runner.py # Produce v2 development observations +├── admission_scoring.py # Delegate neutral scoring to core admission +├── admission_run.py # Consume v2 manifest and readiness prerequisites +├── run_manifests.py # Supply expanded v2 identities and lineage +└── population_custody.py # Sole append-only writer for core-built use/spend receipts + +tests/fixtures/detection_core/ +├── obligation_family.json # Test-only control and constraint family contract +├── taint_family.json # Test-only flow and sanitization family contract +├── obligation_cases.json # Positive, negative and missing-evidence triplet +└── taint_cases.json # Positive, negative and missing-evidence triplet + +tests/ +├── test_core_canonical.py +├── test_core_evidence.py +├── test_core_family.py +├── test_core_coverage.py +├── test_core_budget.py +├── test_core_readiness.py +├── test_core_model.py +├── test_core_validation.py +├── test_core_verdict.py +├── test_core_probes.py +├── test_core_factpack.py +├── test_core_provenance.py +├── test_core_measurement.py +├── test_core_admission.py +├── test_core_consumer.py +└── test_core_migration.py ``` -### Modified Files -- `src/openultrasast/hunt/prover.py`, `semantic/obligations/trace.py` — implement/re-export - the core trace contract for access_control (Req 1, 2, 5); behavior unchanged. -- `benchmarks/valid/prover_admission.py`, `admission_scoring.py`, `admission_run.py` — become - thin consumers of `core/admission.py` (Req 7); existing refusal order and report schema preserved. -- `benchmarks/valid/development_runner.py`, `arm_regression.py`, `prover_arm.py` — adopt - `core/coverage.py`, `core/budget.py`, `core/provenance.py` (Req 3, 4, 6, 11, 12). -- `benchmarks/valid/run_manifests.py` — bundle digests feed `core/provenance.py` (Req 6). +Files below `hunt/`, `search/`, `semantic/`, and `benchmarks/` are adjacent-owner +conformance edits, not ownership transfers. `model_contract.py` translates an already +executed provider result and never selects or runs DSPy. `readiness_contract.py` constructs +and validates canonical records while the benchmark workflow remains the sole owner of +sequencing and persistence. The search and semantic adapters translate typed records but do +not execute proof or family analysis. + +### Modified and Removed Files + +- `src/openultrasast/core/trace.py` — temporary v1 import shim during migration; removed + after repository import census reaches zero. Historical decoding moves to `legacy.py`. +- `src/openultrasast/semantic/obligations/trace.py` — removes imports and conversion logic + for core types; remains the authorization-family trace domain. +- `src/openultrasast/hunt/prover.py` — sends authorization results through + `semantic/obligations/core_adapter.py`; it does not define neutral core types. +- `src/openultrasast/mapping/entry_packs.py` — consumes versioned typed packs and supplies + structural match evidence. +- `src/openultrasast/ruleset/entries/express.json` — migrates metadata and fact rows to the + v2 pack schema without changing target behavior. +- Existing `tests/test_core_*.py` files — authorization assumptions move to adapter or + legacy tests; neutral tests use the two reference contracts. +- `tests/test_core_contract.py` — replaces the assertion that no model path exists with the + assertion that core has no DSPy import and only exposes dependency-free model contracts. + +Every new module remains within the project cap of 400 logical lines; functions remain at +or below 50 logical lines and cyclomatic complexity 10. ## System Flows +### Deterministic Evidence, Model Residual, and Proof + ```mermaid sequenceDiagram - participant PP as pre-push transaction - participant CORE as detection-core - participant ARM as arm (prover/fact pack) - participant BM as benchmark pools - PP->>CORE: push (base, head), budget class - CORE->>CORE: scope change regions (Req 4) - CORE->>ARM: analyze scoped regions - ARM-->>CORE: trace + coverage + complexity + seconds - CORE->>CORE: stamp digests, derive budget check (Req 4, 6) - CORE-->>PP: verdict (proven claims with witnesses | bounded silence) - BM-->>CORE: dev pool + untouched admission pool - CORE->>CORE: admission gate: manifest, pairs, criteria -> GO/NO-GO (Req 7) + participant Analyzer as Analyzer adapter + participant Core as Detection core + participant Family as Family contract + participant Model as DSPy adapter + participant Proof as Proof adapter + participant Consumer as Push consumer + + Analyzer->>Core: Evidence packet and deterministic decision + Core->>Family: Validate scope roles fields and prerequisites + Family-->>Core: Valid record or indeterminate reasons + alt Declared residual is ready + Core->>Model: Model request with packet digest + Model-->>Core: Model result with identities + Core->>Family: Validate decision mapping + end + alt Demonstrated proof exists + Proof-->>Core: Differential record control and verifier receipt + Core->>Core: Resolve artifacts and validate family location and packet identity + end + Core-->>Consumer: Defect satisfaction or bounded silence ``` +The deterministic packet is immutable. A model result creates a new decision referencing +the packet digest. A proof attachment may independently establish execution-backed basis +for a matching violation; it does not mark missing static relations complete. + +### Readiness Promotion + +```mermaid +flowchart LR + Instrument[Instrument] --> Recognition[Recognition] + Recognition --> Transitions[Transitions] + Transitions --> Constraints[Security object and constraints] + Constraints --> Pairs[Vulnerable fixed pairs] + Pairs --> Development[Development population] + Development --> Admission[Fresh admission] + Admission --> Integration[Consumer integration] + Instrument --> Stopped[Stopped] + Recognition --> Stopped + Transitions --> Stopped + Constraints --> Stopped + Pairs --> Stopped + Development --> Stopped + Admission --> Stopped +``` + +Every forward edge requires a current `passed` receipt. A failed, absent, or digest-stale +predecessor routes to `Stopped` with the failing layer and correction lever. Historical +receipts are immutable; staleness is derived against current input identities. + +### Admission Sequence + +```mermaid +sequenceDiagram + participant Benchmark as Benchmark workflow + participant Core as Detection core + participant Arm as Frozen arm + participant Custodian as Population custodian + + Benchmark->>Core: Reviewed manifest without exposed labels + Core->>Core: Validate catalog context lineage readiness denominator and feasibility + Core-->>Benchmark: Authorization plus head-bound label-release request + Benchmark->>Custodian: Compare-and-swap append AVAILABLE to SPENT + Custodian-->>Benchmark: Durable spend receipt then release labels + Benchmark->>Arm: Run frozen analyzer facts and model + Arm-->>Benchmark: Immutable case results + Benchmark->>Core: Preparation receipt case results and instrument receipts + Core->>Core: Revalidate canonical inputs and score scopes separately + Core-->>Benchmark: Report bound to authorization and spend receipt +``` + +Core never creates, edits, labels, or persists the admission population. The benchmark +custodian performs durable append and label release as one guarded operation: a crash before +the append releases no labels, while a crash after it leaves the population spent. Concurrent +attempts race on the same custody head, and only the compare-and-swap winner receives label +bytes. + ## Requirements Traceability -| Req | Summary | Components | Contracts | +| Requirement Criteria | Summary | Components and Interfaces | Flows | |---|---|---|---| -| 1 | canonical trace | core/trace.py | Trace schema, deterministic record() | -| 2 | three-way judgment | core/trace.py, core/verdict.py | Judgment enum; indeterminate-never-clean rule | -| 3 | coverage accounting | core/coverage.py | funnel + resolution buckets + zero-byte refusal | -| 4 | measured budget | core/budget.py | complexity features; per-class derivation; gate names derivation | -| 5 | verdict contract | core/verdict.py | proven claim w/ witness; bounded silence w/ reason | -| 6 | provenance | core/provenance.py | digest stamping; mismatch refusal | -| 7 | admission gate | core/admission.py | consume-only manifest; paired scoring; GO/NO-GO | -| 8 | no fitting on labels | core/probes.py + admission | reversed-pair probes; holdout; no pool-label tuning | -| 9 | bounded models | core/verdict.py contract | zero-shot judge over trace; BLOCK only via proof | -| 10 | fact packs | core/factpack.py + admission family tag | reviewed facts, provenance, family-scope in reports | -| 11 | consumer contract | core/__init__.py | push-consumer and proof-consumer interfaces | -| 12 | measurement schema | core/admission.py report schema | digests, vocabulary, rates+n+Wilson, coverage, instrument, runtime | +| 1.1, 1.2, 1.3, 1.4 | Canonical family-neutral evidence | Canonical Codec, Semantic Evidence, Evidence Aggregate, Family Contract | Deterministic evidence flow | +| 2.1, 2.2, 2.3, 2.4, 2.5 | Neutral outcome, bounded scope and basis | Evidence Aggregate, Family Contract, Verdict | Evidence and decision validation | +| 3.1, 3.2, 3.3, 3.4, 3.5 | Family conformance and unlike reference shapes | Family Contract, Family Catalog, reference fixtures | Family registration and conformance | +| 4.1, 4.2, 4.3, 4.4 | Typed external semantic relations | Semantic Evidence, Fact Pack | Analyzer and pack ingestion | +| 5.1, 5.2, 5.3, 5.4, 5.5 | Stage and instrument coverage | Coverage Funnel, Canonical evidence receipts | Readiness promotion | +| 6.1, 6.2, 6.3, 6.4 | Measured budget and immutable change scope | Budget, Analyzed Scope, Consumer | Push consumption | +| 7.1, 7.2, 7.3, 7.4 | Defect, satisfaction and bounded silence | Verdict, Evidence Aggregate, Consumer | Decision aggregation | +| 8.1, 8.2, 8.3, 8.4, 8.5, 8.6 | Ordered readiness, fixtures and staleness | Readiness Validator, benchmark readiness contract | Readiness promotion | +| 9.1, 9.2, 9.3, 9.4, 9.5, 9.6 | Bounded model and proof roles | Model Handoff, Family Contract, Verdict, Proof Attachment | Model residual and proof flow | +| 10.1, 10.2, 10.3, 10.4 | Complete reproducibility identities | Provenance and Lineage, Fact Pack, Model Handoff | All record flows | +| 11.1, 11.2, 11.3, 11.4, 11.5, 11.6 | Reviewed semantic packs and supported scope | Fact Pack, Semantic Evidence, Measurement | Pack ingestion and reports | +| 12.1, 12.2, 12.3, 12.4, 12.5, 12.6 | Admission prerequisites and scoped scoring | Admission, Readiness, Measurement | Admission sequence | +| 13.1, 13.2, 13.3, 13.4 | Development and evaluation separation | Provenance and Lineage, Probes, Admission | Freeze and admission validation | +| 14.1, 14.2, 14.3, 14.4, 14.5 | Shared consumer and family-spec contract | Consumer, Verdict, Family Capability Declaration, Legacy Adapter | Push and migration flows | +| 15.1, 15.2, 15.3, 15.4, 15.5 | Fixed versioned measurement and baseline rules | Measurement, Coverage, Readiness | Development and admission reporting | ## Components and Interfaces -| Component | Intent | Req | Key contracts | -|---|---|---|---| -| core/trace.py | canonical trace + judgment | 1, 2 | `TraceRecord.record() -> canonical JSON` | -| core/verdict.py | verdict contract | 5, 9 | `Verdict = ProvenClaim(witness, location) | BoundedSilence(reason)` | -| core/coverage.py | coverage accounting | 3 | `Coverage{entry_recognition, resolution_buckets, unchecked_reasons}` | -| core/budget.py | measured budget | 4 | `Complexity{regions, calls, changed_bytes}`; `derive_budget(samples, percentile, hardware) -> Budget` | -| core/provenance.py | digests | 6 | `stamp(bundle, population, change) -> digests`; mismatch raises | -| core/admission.py | admission gate | 7, 10, 12 | `validate_manifest` (consume-only), `admission_report(...) -> GO/NO-GO` | -| core/probes.py | anti-shortcut probes | 8 | `reversed_pair_probe(component, fix, reversed_fix)` | -| core/factpack.py | family packs | 10 | `admit_fact_pack(facts, review) -> stamped pack` | +| Component | Domain | Intent | Requirement Coverage | Key Dependencies | Contracts | +|---|---|---|---|---|---| +| Canonical Codec | Primitive | Produce deterministic bytes and identities | 1.4, 10.4 | Standard library | Service | +| Semantic Evidence | Evidence | Represent typed relations and observations | 1.2, 4.1–4.4, 5.2–5.3 | Canonical Codec | State | +| Evidence Aggregate | Evidence | Bind scope, deterministic packet and decision | 1.1–1.4, 2.1–2.5, 10.1 | Semantic Evidence, Provenance | State, Service | +| Family Contract and Catalog | Family | Declare, catalog, resolve, and validate family semantics | 1.3, 2.1–2.4, 3.1–3.5, 9.1, 14.5 | Evidence Aggregate | State, Service | +| Coverage Funnel | Operations | Account for every semantic stage and input | 5.1–5.5 | Semantic Evidence | State, Service | +| Budget and Scope | Operations | Validate changed scope and measured deadline | 6.1–6.4 | Evidence Aggregate | State, Service | +| Provenance and Lineage | Governance | Bind all inputs and label ancestry | 10.1–10.4, 13.1, 13.3–13.4 | Canonical Codec | State, Service | +| Fact Pack | Extension | Validate reviewed semantic knowledge | 4.1–4.4, 11.1–11.6 | Family Contract, Provenance | State, Service | +| Model Handoff | Extension | Validate dependency-free model requests/results | 9.1–9.5, 10.3, 13.1 | Evidence, Family, Provenance | Service, State | +| Evidence Validation | Composition | Revalidate provenance, family, facts, and model derivation before use | 4.4, 10.1–10.4, 12.6, 14.1 | Family, Fact Pack, Model, Provenance | Service, State | +| Readiness Validator | Governance | Gate ordered stage promotion and freshness | 8.1–8.6, 12.1, 15.5 | Coverage, Provenance | Service, State | +| Verdict and Proof | Decision | Produce bounded outputs and attach proof | 7.1–7.4, 9.4, 9.6, 14.2, 14.4 | Evidence, Model | Service, State | +| Probes | Governance | Test semantic shortcut resistance | 13.2 | Evidence, Family, Fact Pack review policy | Service | +| Measurement | Evaluation | Create baseline identity and fixed report schema | 11.6, 15.1–15.5 | Coverage, Readiness | Service, State | +| Admission | Evaluation | Validate manifests, feasibility and scoped scores | 12.1–12.6, 13.4 | Readiness, Measurement, Evidence Validation, Verdict, Probes, Budget, Provenance | Service, Batch | +| Consumer | Edge | Validate push requests and return bounded verdicts | 6.4, 14.1–14.5 | Budget, Evidence Validation, Verdict, Provenance, Canonical resolver | Service | +| Legacy Decoder | Migration | Read and validate historical v1 only | 14.3, 15.3 | Canonical Codec, Evidence | Service | +| External Adapters | Integration | Translate authorization, DSPy and proof records | 3.5, 9.1–9.6, 14.2–14.3 | Public core | Service | + +### Evidence Domain + +#### Canonical Codec and Semantic Evidence + +| Field | Detail | +|---|---| +| Intent | Define the only v2 serialization profile and family-neutral semantic vocabulary | +| Requirements | 1.2, 1.4, 4.1, 4.2, 4.3, 4.4, 5.2, 10.4 | + +**Responsibilities and constraints** + +- Accept only JSON-compatible scalar values, tuples, and string-keyed mappings. +- Refuse non-finite numbers, unknown enums, duplicate logical identities, and unsupported + values before hashing. +- Sort mapping keys, preserve tuple order, use compact separators, and include schema + version in every digestible aggregate. +- Give every relation and observation a stable identity, role, source location, + completeness, provenance, and optional semantic fact identity. +- Represent registration, invocation, call, return, value transformation, operation effect, + and constraint effect as a closed `RelationKind`. + +**Contracts:** Service, State. + +```python +JsonScalar = str | int | float | bool | None +JsonValue = JsonScalar | tuple["JsonValue", ...] | Mapping[str, "JsonValue"] + +def canonical_bytes(value: JsonValue) -> bytes: ... +def content_digest(value: JsonValue) -> str: ... + +class FieldKind(Enum): + TEXT = "text" + INTEGER = "integer" + NUMBER = "number" + BOOLEAN = "boolean" + TEXT_TUPLE = "text_tuple" + SOURCE_LOCATION = "source_location" + +@dataclass(frozen=True) +class FieldValue: + name: str + kind: FieldKind + value: JsonValue + +@dataclass(frozen=True) +class ContentRef: + artifact_id: str + content_digest: str + media_type: str + byte_length: int + retention_policy_digest: str + storage_receipt_digest: str + +@dataclass(frozen=True) +class StorageReceipt: + store_identity: str + content_digest: str + byte_length: int + retention_policy_digest: str + receipt_digest: str + +@dataclass(frozen=True) +class ResolvedContent: + content: bytes + receipt: StorageReceipt + +class ContentResolver(Protocol): + def resolve(self, reference: ContentRef) -> ResolvedContent: ... + +@dataclass(frozen=True) +class SourceLocation: + path: str + line: int + column: int | None + symbol: str | None + +@dataclass(frozen=True) +class SemanticNode: + node_id: str + role: str + location: SourceLocation + +class RelationKind(Enum): + REGISTRATION = "registration" + INVOCATION = "invocation" + CALL = "call" + RETURN = "return" + VALUE_TRANSFORMATION = "value_transformation" + OPERATION_EFFECT = "operation_effect" + CONSTRAINT_EFFECT = "constraint_effect" + +class RelationStatus(Enum): + COMPLETE = "complete" + INCOMPLETE = "incomplete" + UNRESOLVED = "unresolved" + +class ProvenanceKind(Enum): + PRODUCT_CODE = "product_code" + SEMANTIC_FACT = "semantic_fact" + +@dataclass(frozen=True) +class EvidenceProvenance: + kind: ProvenanceKind + source_content_digest: str + semantic_pack_digest: str | None + semantic_fact_id: str | None + fact_match: ContentRef | None + +@dataclass(frozen=True) +class SemanticRelation: + relation_id: str + kind: RelationKind + role: str + source: SemanticNode + target: SemanticNode + location: SourceLocation + status: RelationStatus + conditions: tuple[str, ...] + reasons: tuple[str, ...] + provenance: EvidenceProvenance + +@dataclass(frozen=True) +class Observation: + observation_id: str + role: str + location: SourceLocation + fields: tuple[FieldValue, ...] + status: RelationStatus + provenance: EvidenceProvenance +``` + +An unmet declared condition yields `INCOMPLETE`; conflicting complete candidates yield an +`UNRESOLVED` ambiguity relation. Neither path silently selects a winner. Semantic-fact +provenance requires the exact pack digest, stable fact ID, and resolved match reference. +Product-code provenance forbids all three fields, preventing an unattributed relation from +changing origin later. + +#### Evidence Aggregate + +| Field | Detail | +|---|---| +| Intent | Keep deterministic evidence immutable while decisions derive from its digest | +| Requirements | 1.1–1.4, 2.1–2.5, 6.4, 7.4, 10.1–10.4 | + +**Responsibilities and constraints** + +- `EvidencePacket` owns the analyzer-produced data and never contains model text or proof + execution output. +- `Decision` owns the family decision, neutral outcome, evidence basis, bounded scope, and + named indeterminate reasons. Its separate `DecisionProvenance` owns any model-result or + proof-record identity; derived decisions never add those identities to the packet. +- `EvidenceRecord` binds one decision to one packet digest. Derivation creates a new record; + it does not mutate the packet. +- `SATISFIED` requires an explicit bounded scope and complete family-required relations. +- `INDETERMINATE` requires at least one named reason. +- Deterministic basis forbids model/proof references; model-corroborated basis requires the + complete model call and result identities; execution-backed basis requires the proof + record and reviewed producer identities. Core validates these as a closed union. + +**Contracts:** Service, State. + +```python +class NeutralOutcome(Enum): + VIOLATION = "violation" + SATISFIED = "satisfied" + INDETERMINATE = "indeterminate" + +class EvidenceBasis(Enum): + DETERMINISTIC = "deterministic" + MODEL_CORROBORATED = "model_corroborated" + EXECUTION_BACKED = "execution_backed" + +@dataclass(frozen=True) +class EvidencePacket: + schema_version: str + family_id: str + family_contract_digest: str + security_object_id: str + ecosystem: str + scope: AnalyzedScope + transitions: tuple[SemanticRelation, ...] + observations: tuple[Observation, ...] + family_fields: tuple[FieldValue, ...] + completeness_reasons: tuple[str, ...] + provenance: ProvenanceStamp + +@dataclass(frozen=True) +class Decision: + family_code: str + outcome: NeutralOutcome + basis: EvidenceBasis + bounded_scope: AnalyzedScope + reasons: tuple[str, ...] + packet_digest: str + provenance: DecisionProvenance + +@dataclass(frozen=True) +class EvidenceRecord: + packet: EvidencePacket + decision: Decision + record_id: str +``` + +### Family Domain + +#### Family Contract and Catalog + +| Field | Detail | +|---|---| +| Intent | Declare family semantics as immutable data and validate records without importing family algorithms | +| Requirements | 1.3, 2.1–2.4, 3.1–3.5, 7.2, 9.1, 14.5 | + +**Responsibilities and constraints** + +- Declare family and schema versions, ecosystems, security-object kind and fields, + observation roles and fields, decisions, neutral mappings, required roles and relation + statuses, named indeterminate reasons, and residual questions. +- Use a closed `FieldKind` vocabulary for text, integer, finite number, boolean, text tuple, + and source location values. +- Keep `FamilyCatalog` immutable; constructing it re-derives every contract digest, requires + canonical order, and refuses duplicate family/version/schema identities or digests. +- Refuse unsupported ecosystems, unknown roles/fields/decisions, undeclared reasons, and + determinate outcomes whose required evidence is incomplete. +- Declare, per role, whether product-code relations are sufficient or excluded-dependency + behavior requires a reviewed semantic fact. A missing reviewed dependency relation is + incomplete evidence, never an implicit product-code relation. +- Validate test-only obligation and taint-flow contracts with the same code used by real + family adapters. + +**Contracts:** Service, State. + +```python +@dataclass(frozen=True) +class FieldSpec: + name: str + kind: FieldKind + required: bool + +@dataclass(frozen=True) +class ObjectSpec: + kind: str + fields: tuple[FieldSpec, ...] + identity_fields: tuple[str, ...] + +class RelationOriginPolicy(Enum): + PRODUCT_OR_REVIEWED_FACT = "product_or_reviewed_fact" + EXCLUDED_DEPENDENCY_FACT_ONLY = "excluded_dependency_fact_only" + +@dataclass(frozen=True) +class RoleSpec: + role: str + fields: tuple[FieldSpec, ...] + allowed_relation_kinds: tuple[RelationKind, ...] + origin_policy: RelationOriginPolicy + +@dataclass(frozen=True) +class DecisionRule: + family_code: str + outcome: NeutralOutcome + required_roles: tuple[str, ...] + allowed_statuses: tuple[RelationStatus, ...] + +@dataclass(frozen=True) +class ResidualQuestion: + question_id: str + required_roles: tuple[str, ...] + allowed_decisions: tuple[str, ...] + +@dataclass(frozen=True) +class FamilyContract: + family_id: str + family_version: str + schema_version: str + ecosystems: tuple[str, ...] + security_object: ObjectSpec + roles: tuple[RoleSpec, ...] + decisions: tuple[DecisionRule, ...] + residual_questions: tuple[ResidualQuestion, ...] + indeterminate_reasons: tuple[str, ...] + +@dataclass(frozen=True) +class FamilyCatalogEntry: + contract: FamilyContract + contract_digest: str + +@dataclass(frozen=True) +class FamilyCatalog: + entries: tuple[FamilyCatalogEntry, ...] + catalog_digest: str + +@dataclass(frozen=True) +class FamilyCapabilityDeclaration: + arm_id: str + family_contract_digest: str + readiness_fixture_digest: str + development_population_digest: str + admission_boundary_digest: str + +def validate_catalog(catalog: FamilyCatalog) -> None: ... +def resolve_contract( + catalog: FamilyCatalog, + family_id: str, + contract_digest: str, + ecosystem: str, +) -> FamilyContract: ... +def validate_record(contract: FamilyContract, record: EvidenceRecord) -> None: ... +def validate_capability( + declaration: FamilyCapabilityDeclaration, + catalog: FamilyCatalog, +) -> None: ... +``` + +`FamilyCapabilityDeclaration` references the family contract, readiness-fixture digest, +development boundary, and admission boundary. It is a machine-validated artifact for family +specs; the core does not parse Markdown specifications. + +`resolve_contract` requires exactly one catalog entry matching family, digest, and ecosystem; +it never falls back to a different version. For a role declared +`EXCLUDED_DEPENDENCY_FACT_ONLY`, `validate_record` refuses a `COMPLETE` product-code +relation. The relation can become complete only through `SEMANTIC_FACT` provenance whose +pack, fact, and structural match are revalidated. If that knowledge is absent, the adapter +records `INCOMPLETE` with `missing_reviewed_dependency_relation`; a determinate rule that +requires the role is refused. This is the explicit vendored-source boundary required by +criterion 4.4. + +### Extension Contracts + +#### Model Handoff + +| Field | Detail | +|---|---| +| Intent | Make the evidence-to-DSPy seam reproducible without importing DSPy into core | +| Requirements | 9.1–9.6, 10.3, 13.1 | + +**Responsibilities and constraints** + +- Build requests only for residual questions declared by the matching family contract. +- Present the canonical evidence record, family contract, residual-question definition, + and per-role completeness snapshot in the request. Their redundant digests are validated + before dispatch so an adapter cannot substitute content behind an identity. +- Require provider, model, program, prompt, configuration, and adapter identities on every + result, including failures. +- Freeze those identities, the decision mapping, optional examples/fitted component, and + their actual lineage manifest in a pre-execution `ModelProgramDeclaration`; a digest-only + lineage assertion is not sufficient. +- Reject a result for a different packet, family, question, or decision mapping. +- Reject the same program identity as proposer and adjudicator on one claim path. +- Preserve the normalized response plus either redacted inline response bytes or a + content-addressed immutable response artifact. Preserve model failure as coverage and + measurement evidence; never convert it to satisfaction or silence without a reason. +- Permit model-corroborated outcomes only when every prerequisite of the residual question + is complete. Model output can never set `EXECUTION_BACKED`. + +**Contracts:** Service, State. + +```python +class ModelStatus(Enum): + SUCCEEDED = "succeeded" + FAILED = "failed" + +@dataclass(frozen=True) +class CompletenessItem: + role: str + status: RelationStatus + reasons: tuple[str, ...] + +@dataclass(frozen=True) +class ModelProgramDeclaration: + family_contract_digest: str + provider: str + model: str + program_digest: str + prompt_digest: str + example_set_digest: str | None + fitted_component_digest: str | None + configuration_digest: str + adapter_revision: str + decision_mapping_digest: str + input_bundle_digest: str + lineage_manifest_digest: str + declaration_digest: str + +@dataclass(frozen=True) +class ModelRequest: + request_id: str + record: EvidenceRecord + contract: FamilyContract + residual_question: ResidualQuestion + completeness: tuple[CompletenessItem, ...] + packet_digest: str + family_contract_digest: str + residual_question_id: str + completeness_digest: str + program: ModelProgramDeclaration + lineage: LineageManifest + custody_policy_digest: str + proposal_program_digests: tuple[str, ...] + trusted_response_store_identities: tuple[str, ...] + maximum_response_bytes: int + +@dataclass(frozen=True) +class PreservedModelResponse: + schema_version: str + normalized_fields: tuple[FieldValue, ...] + inline_redacted_text: str | None + artifact: ContentRef | None + content_digest: str + redaction_policy_digest: str + +@dataclass(frozen=True) +class ModelResult: + request_digest: str + packet_digest: str + identity: ModelCallIdentity + status: ModelStatus + response: PreservedModelResponse | None + family_decision: str | None + decision_mapping_digest: str | None + failure_reason: str | None + result_digest: str + +def apply_model_result( + record: EvidenceRecord, + contract: FamilyContract, + request: ModelRequest, + result: ModelResult, + resolver: ContentResolver, + custody: PopulationCustodyPort, + custody_policy: PopulationCustodyPolicy, +) -> EvidenceRecord: ... +``` + +`PreservedModelResponse` requires exactly one of `inline_redacted_text` or `artifact`. The +core re-derives the inline digest; for an artifact it requires `content_digest` to equal the +reference digest and validates the storage-receipt, trusted-store, byte-limit, and +retention-policy identities declared by the request. A +`ContentRef` names an opaque artifact ID, media type, byte length, retention-policy digest, +storage-receipt digest, and full SHA-256 digest; the adapter or benchmark store owns and +verifies the bytes. The normalized fields used for the decision are always inline and +canonical. This retains the response used for adjudication without making +provider transport data or credentials part of the core. A successful result requires the +response, family decision, and mapping digest; a failed result requires a named failure and +may omit response fields only when no response was received. + +Before accepting either outcome, `apply_model_result` re-derives the program declaration, +input-bundle, request, and lineage digests; requires the declared custody policy; and calls +`validate_lineage` against the record population and the current complete custody chains. +The caller-specific check maps `PROMPT`, `EXAMPLE`, `FITTED_MODEL`, and +`DECISION_MAPPING` uses to required `LineageExpectation` values for the corresponding +non-null declared artifact digests, with `input_bundle_digest` as the manifest subject. +Every use must +match the purpose, source population, influenced-artifact digest, and receipt embedded in +the current chain. Omitted history, an old prefix, or use of the evaluated admission +population's labels is refused. Result provider/model/program/prompt/configuration/adapter +and decision-mapping identities must equal the frozen declaration. The derived +`DecisionProvenance` retains both the declaration and validated lineage-manifest digests. + +The adjacent hunt subsystem owns prompt text, DSPy signatures, execution, caching, storage, +and policy-driven redaction. `model_contract.py` only maps its structured input/output to +the core records. Core validation treats the result as untrusted input and checks the +response, decision-mapping, request, and packet identities. +`apply_model_result` returns a new decision whose `DecisionProvenance` contains the complete +`ModelCallIdentity` and result digest; the deterministic packet remains byte-identical. + +#### Fact Pack + +The v2 `FactPack` carries schema version, family-contract digest, family, ecosystem, +language scope, stable fact IDs, declared roles, structural requirements, relation effects, +normalized content digest, and review identity. A match becomes a complete semantic +relation only when the adapter supplies every required structural match item. A bare name +match can at most produce incomplete evidence. Pack insufficiency preserves partial records +and named unresolved reasons. + +```python +class KnowledgeEvidenceKind(Enum): + SOURCE_CODE = "source_code" + OFFICIAL_DOCUMENTATION = "official_documentation" + INDEPENDENT_BEHAVIOR = "independent_behavior" + +@dataclass(frozen=True) +class IndependenceAttestation: + subject_digest: str + producer_identity: str + reviewer_identity: str + conflict_disclosure: ContentRef + attestation_digest: str + +@dataclass(frozen=True) +class ReviewIndependencePolicy: + reviewed_reviewer_digests: tuple[str, ...] + forbidden_identity_pairs: tuple[tuple[str, str], ...] + trusted_store_identities: tuple[str, ...] + maximum_evidence_bytes: int + policy_digest: str + +@dataclass(frozen=True) +class KnowledgeEvidence: + kind: KnowledgeEvidenceKind + content: ContentRef + producer_identity: str + reviewer_identity: str + independence: IndependenceAttestation + lineage_manifest_digest: str + +class StructuralRequirementKind(Enum): + QUALIFIED_NAME = "qualified_name" + AST_SHAPE = "ast_shape" + RECEIVER_TYPE = "receiver_type" + CALL_TARGET = "call_target" + ARGUMENT_SHAPE = "argument_shape" + CONFIGURATION = "configuration" + +@dataclass(frozen=True) +class StructuralRequirement: + requirement_id: str + kind: StructuralRequirementKind + field: str + expected: JsonValue + +@dataclass(frozen=True) +class RelationEffect: + kind: RelationKind + role: str + source_role: str + target_role: str + conditions: tuple[str, ...] + +@dataclass(frozen=True) +class StructuralObservation: + requirement_id: str + location: SourceLocation + observed: JsonValue + evidence: ContentRef + +@dataclass(frozen=True) +class FactMatchEvidence: + pack_digest: str + fact_id: str + product_location: SourceLocation + observations: tuple[StructuralObservation, ...] + matcher_revision: str + matcher_behavior_digest: str + matcher_lineage_manifest_digest: str + match_digest: str + +@dataclass(frozen=True) +class SemanticFact: + fact_id: str + role: str + structural_requirements: tuple[StructuralRequirement, ...] + relation_effects: tuple[RelationEffect, ...] + +@dataclass(frozen=True) +class FactPack: + schema_version: str + family_contract_digest: str + family_id: str + ecosystem: str + language: str + facts: tuple[SemanticFact, ...] + normalized_content_digest: str + review_identity: str + review_evidence: tuple[KnowledgeEvidence, ...] + review_evidence_digest: str + review_policy_digest: str + +def validate_fact_pack( + pack: FactPack, + contract: FamilyContract, + protected_population_digest: str | None, + lineages: Sequence[LineageManifest], + custody: PopulationCustodyPort, + custody_policy: PopulationCustodyPolicy, + review_policy: ReviewIndependencePolicy, + resolver: ContentResolver, +) -> None: ... + +def apply_fact( + pack: FactPack, + fact_id: str, + match: FactMatchEvidence, + match_reference: ContentRef, + contract: FamilyContract, + evaluation_population_digest: str, + lineages: Sequence[LineageManifest], + custody: PopulationCustodyPort, + custody_policy: PopulationCustodyPolicy, + review_policy: ReviewIndependencePolicy, + resolver: ContentResolver, +) -> tuple[SemanticRelation, ...]: ... + +def validate_fact_relations( + record: EvidenceRecord, + packs: Sequence[FactPack], + lineages: Sequence[LineageManifest], + custody: PopulationCustodyPort, + custody_policy: PopulationCustodyPolicy, + review_policy: ReviewIndependencePolicy, + resolver: ContentResolver, +) -> None: ... +``` + +The normalized digest covers schema, scope, roles, every structural predicate, every +effect, and the review evidence. Validation refuses unknown fields and roles, duplicate +fact IDs, scope mismatch, missing review evidence, and a digest that cannot be re-derived. +Each knowledge item must be source code, official documentation, or independently reviewed +behavioral evidence and must reference a label-isolated lineage manifest. For behavioral +evidence, producer and reviewer must differ, the reviewer must be admitted by the +`ReviewIndependencePolicy`, the attestation subject must equal the content digest, and the +conflict disclosure must resolve and hash correctly. Vulnerability, fix-location, or +ordinary-change label ancestry spends the population and refuses the pack. +`validate_fact_pack` resolves every referenced manifest from the supplied lineage set, +requires its subject to equal `normalized_content_digest`, declares a required +`SEMANTIC_FACT` expectation and an optional `PROMOTION` expectation for that same digest, +sets `forbid_label_access=True`, and validates its uses against current custody chains. + +`apply_fact` is the only route from a semantic fact to fact-derived relations. It resolves +and validates the pack with the supplied review policy and evaluation-population boundary, +then resolves each structural observation. It requires the observation IDs to equal the fact's complete +requirement-ID set, checks values against the typed predicates, and constructs relations +whose provenance contains the pack/fact identities and resolved match reference. Missing or failed predicates +produce incomplete evidence; duplicates, fabricated IDs, or ambiguity produce unresolved +evidence. A name observation cannot yield `COMPLETE` while any non-name structural +requirement is absent. The match reference must resolve to canonical bytes whose digest is +`match_digest`; record validation re-runs this join from the named pack and match. +The match digest covers the matcher revision, behavior digest, and lineage-manifest digest. +`apply_fact` and `validate_fact_relations` resolve that manifest, require subject equality +with `matcher_behavior_digest`, supply a required `MATCHING_BEHAVIOR` expectation for that +digest, and validate every receipt against the current custody chain while protecting the +record's evaluation population. Matching behavior influenced by the admission +population's labels therefore spends that population and cannot complete a relation in the +same admission run. + +#### Evidence Validation Composition + +| Field | Detail | +|---|---| +| Intent | Give admission and every consumer one non-bypassable composition seam for provenance, family, fact, and model validation | +| Requirements | 4.4, 10.1–10.4, 12.6, 14.1, 14.3 | + +`validation.py` is later than both `factpack.py` and `model.py`; it composes their pure +validators without moving family reasoning into core. Raw `EvidenceRecord` values are data, +not proof that validation already occurred. Public scoring and consumption APIs accept the +input envelope and context below and call `validate_evidence_input` before deriving a claim, +satisfaction, silence, metric, or gate. + +```python +@dataclass(frozen=True) +class EvidenceValidationContext: + family_catalog: FamilyCatalog + fact_packs: tuple[FactPack, ...] + model_programs: tuple[ModelProgramDeclaration, ...] + lineages: tuple[LineageManifest, ...] + fact_review_policy: ReviewIndependencePolicy + provenance_trust_policy: ProvenanceTrustPolicy + custody_policy: PopulationCustodyPolicy + context_digest: str + +@dataclass(frozen=True) +class EvidenceValidationInput: + record: EvidenceRecord + named_inputs: NamedInputs + model_request: ModelRequest | None + model_result: ModelResult | None + input_digest: str + +def validate_evidence_input( + value: EvidenceValidationInput, + context: EvidenceValidationContext, + custody: PopulationCustodyPort, + resolver: ContentResolver, +) -> EvidenceRecord: ... +``` + +Validation re-derives the input/context/catalog/pack/model/lineage digests, resolves exactly +one contract, validates the seven named provenance inputs, calls `validate_record`, and +replays every fact relation through `validate_fact_relations`. A deterministic record must +omit model request/result values and model provenance. A model-corroborated record must +carry both; its declaration must be an exact context member, `apply_model_result` is rerun +from `model_request.record`, and the derived record must equal `value.record` canonically. +Execution-backed basis is added only later by the proof validator from an already validated +base record. No constructible “validated” marker is accepted as a shortcut. + +### Governance and Evaluation + +#### Changed Scope and Measured Budget + +| Field | Detail | +|---|---| +| Intent | Bind every record to the push change and derive its deadline from disclosed measurements | +| Requirements | 6.1, 6.2, 6.3, 6.4, 14.1 | + +**Responsibilities and constraints** + +- The push runner supplies immutable base/head revisions and normalized changed regions; + the core validates containment but does not discover repository scope. +- Complexity records the change identity, region count, transition count, and changed byte + count alongside total reply time for every sample. +- A budget is derived from a nonempty measured workload class and carries sample count, + percentile, reference workload, reference hardware, derived duration, and derivation + digest. There is no default or static fallback budget. +- The elapsed-time gate covers preparation, deterministic analysis, optional model and + proof handling, validation, and reporting. Failure names the change, elapsed duration, + workload class, and derivation exceeded. + +**Contracts:** Service, State. + +`SourceRegion`, `AnalyzedScope`, and `validate_scope` are owned by `evidence.py`, because the +evidence packet depends on them. The remaining records and functions in this subsection are +owned by `budget.py`, which imports evidence types and never the reverse. + +```python +@dataclass(frozen=True) +class SourceRegion: + path: str + start_line: int + end_line: int + +@dataclass(frozen=True) +class AnalyzedScope: + base_revision: str + head_revision: str + change_digest: str + regions: tuple[SourceRegion, ...] + +@dataclass(frozen=True) +class ChangeComplexity: + change_id: str + workload_class: str + changed_regions: int + transition_count: int + changed_bytes: int + measurement: ContentRef + +@dataclass(frozen=True) +class MeasuredSample: + complexity: ChangeComplexity + total_reply_ms: int + +@dataclass(frozen=True) +class ReferenceWorkload: + name: str + description: str + changed_region_range: tuple[int, int] + transition_count_range: tuple[int, int] + changed_bytes_range: tuple[int, int] + sample_artifact: ContentRef + record_digest: str + +@dataclass(frozen=True) +class ReferenceHardware: + cpu_model: str + logical_cpu_count: int + memory_bytes: int + operating_system: str + runtime_version: str + record_digest: str + +@dataclass(frozen=True) +class BudgetDerivation: + workload_class: str + sample_count: int + percentile: float + reference_workload: ReferenceWorkload + reference_hardware: ReferenceHardware + budget_ms: int + derivation_digest: str + +def derive_budget( + samples: Sequence[MeasuredSample], + workload_class: str, + percentile: float, + reference_workload: ReferenceWorkload, + reference_hardware: ReferenceHardware, +) -> BudgetDerivation: ... + +def validate_scope(packet: EvidencePacket, allowed: AnalyzedScope) -> None: ... +def check_budget( + change: ChangeComplexity, + total_reply_ms: int, + budget: BudgetDerivation, + resolver: ContentResolver, +) -> None: ... +``` + +#### Coverage, Provenance, and Readiness + +| Field | Detail | +|---|---| +| Intent | Locate the first failing layer and prevent stale or failed evidence from reaching later gates | +| Requirements | 5.1–5.5, 8.1–8.6, 10.1–10.4, 13.1–13.4, 15.5 | + +**Responsibilities and constraints** + +- Coverage records per-stage input/output counts for reads, questions, sites considered, + sites recognized, transitions, security objects, constraints, determinate outcomes, and + findings. +- Every considered relation occupies exactly one complete, incomplete, or unresolved bucket. +- Zero-byte or unreadable tracked input is an `InstrumentFailure` that invalidates the run. +- A zero stage is diagnostic rather than automatically clean or failed; fixture expectations + identify the first expected stage that produced zero. +- Packet provenance covers analyzer, family contract, fact set, acquisition, configuration, + population, change, and semantic facts; decision provenance separately covers model and + proof identities. +- Label lineage names all populations used for prompts, examples, thresholds, facts, + matching behavior, mappings, and promotion. Core derives a canonical + `PopulationUseReceipt`; label influence marks the population spent and refuses admission + use. +- Readiness receipts contain fixture roles, identity snapshots, predecessor digests, + measurement digest, result, and correction lever. Staleness is derived without changing + the stored receipt. + +**Contracts:** Service, State. + +```python +@dataclass(frozen=True) +class ProvenanceStamp: + analyzer_revision: str + family_contract_digest: str + fact_set_digest: str + acquisition_record_digest: str + configuration_digest: str + population_digest: str + change_input_digest: str + +@dataclass(frozen=True) +class ModelCallIdentity: + provider: str + model: str + program_digest: str + prompt_digest: str + configuration_digest: str + adapter_revision: str + +@dataclass(frozen=True) +class DecisionProvenance: + model_identity: ModelCallIdentity | None + model_result_digest: str | None + model_declaration_digest: str | None + proof_record_digest: str | None + proof_producer_digest: str | None + lineage_manifest_digest: str | None + +class LineagePurpose(Enum): + PROMPT = "prompt" + EXAMPLE = "example" + THRESHOLD = "threshold" + FITTED_MODEL = "fitted_model" + SEMANTIC_FACT = "semantic_fact" + MATCHING_BEHAVIOR = "matching_behavior" + DECISION_MAPPING = "decision_mapping" + PROMOTION = "promotion" + ADMISSION_LABEL_OPEN = "admission_label_open" + +@dataclass(frozen=True) +class LineageUse: + purpose: LineagePurpose + source_population_digest: str + labels_accessed: bool + influenced_artifact_digest: str + population_use_receipt_digest: str + +@dataclass(frozen=True) +class LineageManifest: + arm_id: str + subject_digest: str + uses: tuple[LineageUse, ...] + manifest_digest: str + +@dataclass(frozen=True) +class LineageExpectation: + purpose: LineagePurpose + influenced_artifact_digest: str + minimum_uses: int + maximum_uses: int | None + +class PopulationStatus(Enum): + AVAILABLE = "available" + SPENT = "spent" + +@dataclass(frozen=True) +class PopulationUseRequest: + population_digest: str + arm_id: str + purpose: LineagePurpose + labels_accessed: bool + influenced_artifact_digest: str | None + expected_log_digest: str + +@dataclass(frozen=True) +class PopulationUseReceipt: + request: PopulationUseRequest + request_digest: str + population_digest: str + custodian_identity: str + sequence: int + predecessor_digest: str | None + prior_status: PopulationStatus + status: PopulationStatus + reasons: tuple[str, ...] + receipt_digest: str + +@dataclass(frozen=True) +class PopulationCustodyPolicy: + reviewed_custodian_digests: tuple[str, ...] + policy_digest: str + +@dataclass(frozen=True) +class PopulationCustodyLog: + population_digest: str + custodian_identity: str + receipts: tuple[PopulationUseReceipt, ...] + receipt_count: int + head_receipt_digest: str | None + status: PopulationStatus + log_digest: str + +class PopulationCustodyPort(Protocol): + def current_log(self, population_digest: str) -> PopulationCustodyLog: ... + +@dataclass(frozen=True) +class LineageDecision: + allowed: bool + spent_population_digests: tuple[str, ...] + reasons: tuple[str, ...] + decision_digest: str + +class ProvenanceInputRole(Enum): + ANALYZER = "analyzer" + FAMILY_CONTRACT = "family_contract" + FACT_SET = "fact_set" + ACQUISITION = "acquisition" + CONFIGURATION = "configuration" + POPULATION = "population" + CHANGE_INPUT = "change_input" + +@dataclass(frozen=True) +class ProvenanceTrustPolicy: + trusted_store_identities: tuple[str, ...] + maximum_input_bytes: int + policy_digest: str + +@dataclass(frozen=True) +class NamedInput: + role: ProvenanceInputRole + identity: str + content: ContentRef + +@dataclass(frozen=True) +class NamedInputs: + inputs: tuple[NamedInput, ...] + manifest_digest: str + +def validate_provenance( + stamp: ProvenanceStamp, + named_inputs: NamedInputs, + resolver: ContentResolver, + trust: ProvenanceTrustPolicy, +) -> None: ... +def record_population_use( + request: PopulationUseRequest, + prior: PopulationCustodyLog, + custodian_identity: str, + policy: PopulationCustodyPolicy, +) -> PopulationUseReceipt: ... + +def validate_lineage( + lineage: LineageManifest, + expected_subject_digest: str, + expectations: tuple[LineageExpectation, ...], + protected_population_digest: str | None, + forbid_label_access: bool, + custody: PopulationCustodyPort, + policy: PopulationCustodyPolicy, +) -> LineageDecision: ... + +@dataclass(frozen=True) +class InputManifest: + tracked_input_ids: tuple[str, ...] + manifest_digest: str + +@dataclass(frozen=True) +class InputReadReceipt: + input_id: str + path: str + opened: bool + byte_count: int + content_digest: str | None + +@dataclass(frozen=True) +class RelationStatusCount: + status: RelationStatus + count: int + +@dataclass(frozen=True) +class ReasonCount: + reason: str + count: int + +@dataclass(frozen=True) +class CoverageFunnel: + inputs_read: int + questions_formed: int + sites_considered: int + sites_recognized: int + transitions_materialized: int + security_objects_constructed: int + constraints_evaluated: int + determinate_outcomes: int + findings_emitted: int + relation_status_counts: tuple[RelationStatusCount, ...] + unresolved_by_reason: tuple[ReasonCount, ...] + +class CoverageStage(Enum): + INPUTS = "inputs" + QUESTIONS = "questions" + SITES_CONSIDERED = "sites_considered" + SITES_RECOGNIZED = "sites_recognized" + TRANSITIONS = "transitions" + SECURITY_OBJECTS = "security_objects" + CONSTRAINTS = "constraints" + OUTCOMES = "outcomes" + FINDINGS = "findings" + +@dataclass(frozen=True) +class StageExpectation: + stage: CoverageStage + expected_nonzero_input: bool + expected_nonzero_output: bool + +@dataclass(frozen=True) +class StageDiagnosis: + last_verified_nonzero_stage: CoverageStage | None + first_failed_output_stage: CoverageStage | None + reasons: tuple[str, ...] + +def validate_instrument(manifest: InputManifest, receipts: Sequence[InputReadReceipt]) -> None: ... +def diagnose_first_zero(funnel: CoverageFunnel, expected: Sequence[StageExpectation]) -> StageDiagnosis: ... + +class ReadinessStage(Enum): + INSTRUMENT = "instrument" + RECOGNITION = "recognition" + TRANSITIONS = "transitions" + CONSTRAINTS = "constraints" + PAIRS = "pairs" + DEVELOPMENT = "development" + ADMISSION = "admission" + INTEGRATION = "integration" + +class StageResult(Enum): + PASSED = "passed" + FAILED = "failed" + +class FixtureRole(Enum): + POSITIVE = "positive" + NEGATIVE = "negative" + AMBIGUITY = "ambiguity" + +class ReadinessEvidenceUse(Enum): + REGRESSION_FIXTURE = "regression_fixture" + DEVELOPMENT_POPULATION = "development_population" + REPOSITORY_HOLDOUT = "repository_holdout" + FRESH_ADMISSION = "fresh_admission" + CONSUMER_INTEGRATION = "consumer_integration" + +@dataclass(frozen=True) +class FixtureResult: + role: FixtureRole + fixture_digest: str + passed: bool + reasons: tuple[str, ...] + +@dataclass(frozen=True) +class AnalysisIdentity: + core_schema_version: str + semantic_vocabulary_digest: str + analyzer_revision: str + family_contract_digest: str + fact_set_digest: str + acquisition_record_digest: str + configuration_digest: str + model_program_digest: str | None + outcome_mapping_digest: str + +@dataclass(frozen=True) +class ReadinessResult: + arm_id: str + family_id: str + ecosystem: str + stage: ReadinessStage + evidence_use: ReadinessEvidenceUse + population_digest: str | None + population_freeze_digest: str | None + population_use_receipt_digests: tuple[str, ...] + result: StageResult + identity: AnalysisIdentity + predecessor_digests: tuple[str, ...] + fixture_results: tuple[FixtureResult, ...] + measurement_digest: str + correction_lever: str | None + result_digest: str + +@dataclass(frozen=True) +class StageAuthorization: + target: ReadinessStage + authorized: bool + failing_stage: ReadinessStage | None + reasons: tuple[str, ...] + correction_lever: str | None + +def authorize_stage( + target: ReadinessStage, + receipts: Sequence[ReadinessResult], + current_identity: AnalysisIdentity, +) -> StageAuthorization: ... +``` + +The benchmark population custodian is the single writer for population-use receipts: it +reads the current log, invokes the pure core constructor with that head, and appends the +returned bytes using compare-and-swap on `expected_log_digest` and `predecessor_digest`. +Core reads the current full +log through `PopulationCustodyPort`, accepts only a reviewed custodian, recomputes every +embedded request and receipt digest, requires contiguous sequence/predecessor links, and +requires the declared head/count/status to equal the derived chain. `SPENT` is monotonic: +no later receipt or manifest can return it to `AVAILABLE`. A caller-selected prefix or an +omitted earlier spend receipt therefore fails current-head validation. + +`validate_provenance` requires the closed seven-role input set exactly once, re-derives the +named-input manifest, maps every role to the corresponding `ProvenanceStamp` identity, and +resolves every `ContentRef`. The resolved bytes, digest, length, storage receipt, retention +policy, trusted store, and byte limit must all validate. A stamp cannot therefore be +accepted from identities alone when its named inputs are unreadable. + +`validate_lineage` re-derives the manifest and requires the expected subject and typed +`LineageExpectation` values supplied by the caller. Every use must match an expected +purpose and influenced-artifact digest, and each expectation must satisfy its declared +minimum/maximum count. For each use the validator resolves the receipt from the current +complete custody chain and requires exact equality of purpose, source population, label +access, and influenced-artifact digest. `forbid_label_access` rejects any labelled ancestry; +otherwise labelled use is allowed only when its source is not the protected evaluation +population. For prompt, example, threshold, fitted-model, +semantic-fact, matching-behavior, decision-mapping, or promotion use, `labels_accessed` +together with an influenced artifact deterministically yields `SPENT`. A +`MATCHING_BEHAVIOR` use must name the matcher behavior digest. Legitimate one-time +admission label opening uses `ADMISSION_LABEL_OPEN`, is bound to the authorization digest, +and is durably appended before label bytes are released. + +Benchmark workflows persist results. The core returns `stale` when any recorded identity or +predecessor digest differs from current inputs, including core schema, semantic vocabulary, +acquisition, or outcome mapping. Instrument through pair receipts use regression fixtures; +development uses a development population; admission requires a separately frozen, +available population and a scored admission report; integration uses consumer evidence. +An evidence-use mismatch cannot satisfy a stage, so preserved regression fixtures can never +stand in for fresh admission. + +#### Fitted-component Probes + +| Field | Detail | +|---|---| +| Intent | Refuse fitted components that exploit reversible surface shape instead of declared security semantics | +| Requirements | 13.2 | + +A reversed-pair probe is valid only when an independent fixture review declares the two +inputs security-semantically equivalent for the family contract while their surface/diff +shape is deliberately reversed. The semantic projection includes the security object, +required roles, relations, completeness, and expected family decision; it excludes line +counts, deletion direction, filenames, and presentation order. If the projections differ, +the probe is invalid and says nothing about shortcut learning. This replaces the current +rule that treats any score difference between an arbitrary fix and its reverse as proof of +a shortcut. + +```python +@dataclass(frozen=True) +class RoleProjection: + role: str + fields: tuple[FieldValue, ...] + relation_kinds: tuple[RelationKind, ...] + statuses: tuple[RelationStatus, ...] + +@dataclass(frozen=True) +class SemanticProjection: + family_contract_digest: str + security_object_fields: tuple[FieldValue, ...] + roles: tuple[RoleProjection, ...] + completeness_reasons: tuple[str, ...] + family_decision: str + projection_digest: str + +@dataclass(frozen=True) +class SemanticEquivalenceReview: + forward_projection_digest: str + reversed_projection_digest: str + independence: IndependenceAttestation + review_digest: str + +@dataclass(frozen=True) +class ReversedPairProbe: + probe_id: str + family_contract_digest: str + forward_record: EvidenceRecord + reversed_record: EvidenceRecord + forward_surface: ContentRef + reversed_surface: ContentRef + equivalence: SemanticEquivalenceReview + maximum_score_delta: float + +@dataclass(frozen=True) +class ProbeObservation: + probe_digest: str + component_digest: str + forward_decision: str + reversed_decision: str + forward_score: float | None + reversed_score: float | None + +@dataclass(frozen=True) +class ProbeEvaluationInput: + probe: ReversedPairProbe + observation: ProbeObservation + +@dataclass(frozen=True) +class ProbeResult: + probe_digest: str + component_digest: str + valid: bool + passed: bool + reasons: tuple[str, ...] + result_digest: str + +@dataclass(frozen=True) +class FittedComponentEvidence: + component_digest: str + probe_result_digests: tuple[str, ...] + fitting_lineage_manifest_digest: str + training_population_manifest: ContentRef + training_repository_digests: tuple[str, ...] + repository_holdout_report_digest: str + admission_report_digest: str + +def project_semantics(record: EvidenceRecord, contract: FamilyContract) -> SemanticProjection: ... +def evaluate_probe( + probe: ReversedPairProbe, + observation: ProbeObservation, + contract: FamilyContract, + review_policy: ReviewIndependencePolicy, + resolver: ContentResolver, +) -> ProbeResult: ... +``` + +Core derives both projections from the embedded records and contract, resolves both surface +artifacts, and verifies the equivalence review against the same independence policy used for +facts. The attestation subject must equal the canonical digest of both derived projections, +the family contract, and the two surface references. Validation requires distinct surface bytes, equal derived semantic projections, the +same declared family decision on both sides, and a score delta no larger than the +pre-registered finite tolerance. A caller-supplied digest cannot stand in for projection or +review content. Fitted-component authorization additionally requires all probes to pass, a +typed repository-held-out report whose repository manifest is disjoint from the resolved +training manifest, a matching fitting-lineage decision, and an untouched admission `GO`. +Probe construction and evaluation live in `probes.py`; fitted-component authorization lives +in `admission.py`, where it accepts `ProbeEvaluationInput` values and calls +`evaluate_probe` itself. Caller-supplied `ProbeResult` summaries never participate in +authorization; the derived result digests must exactly equal the fitted-component evidence. + +#### Measurement and Admission + +| Field | Detail | +|---|---| +| Intent | Produce comparable, family-scoped evidence and refuse impossible admission configurations | +| Requirements | 11.6, 12.1–12.6, 15.1–15.5 | + +**Responsibilities and constraints** + +- `BaselineIdentity` covers arm, population/freeze, family/ecosystem scope, denominator + policy, threshold set, family contract, facts, analyzer, configuration, model program, + and outcome mapping. +- `MeasurementRecordV2` includes neutral outcomes and bases, rates with sample counts and + Wilson intervals, the full funnel, model failures, runtime, instrument failures, and + every prerequisite readiness result with its digest. +- A zero or clean-looking measurement is invalid unless its instrument readiness reference + passed over a complete tracked-input manifest and every earlier semantic-stage reference + is present and passed. +- Trend comparison requires identical baseline identities. +- `AdmissionManifestV2` is externally produced, reviewed, label-isolated, immutable, and + unspent. It names every frozen input, a reviewed label-blind population census, and the + current readiness receipts. It binds a serializable validation context containing the + actual catalog, packs, model declarations, lineages, and trust policies needed for + revalidation; digests never substitute for those objects. +- `DenominatorPolicy` maps each registered metric to a closed numerator and denominator + definition over explicit family/ecosystem scope. Its interval method is fixed to Wilson + 95%; a configurable confidence level is refused. Whole-population coverage is reported + separately. +- The census exposes only reviewed aggregate cardinalities, never case identities or + per-case labels. It is bound to the same population digest as the manifest. +- Before labels open, feasibility enumerates the arithmetically possible counts using pair, + case, semantic-site, and model-call capacities and applies count, rate, per-100, or + Wilson-bound statistics. Every threshold group must have an attainable joint count. An + impossible group is a configuration refusal, not a measured no-go result. +- Scoring is a pure function over immutable post-opening case results carrying the actual + evidence, fact packs/matches, claims/proof attachments, coverage, instrument receipts, model results, + complexity, budget, and timing. Counts and gates are derived; no opaque aggregate supplied by the runner is + trusted. The scorer re-runs record/fact/verdict validation and refuses supplied + claims/satisfaction/silence that differ from the derived result. Core never persists or + relabels cases. +- Admission authorization produces a custody-head-bound label-release request. The + benchmark custodian must durably spend the population before releasing label bytes; + scoring revalidates that exact current-chain receipt and never emits a post-scoring spend + request. + +**Contracts:** Service, Batch, State. + +`measurement.py` owns `ScopeKey`, baseline identity, the closed metric algebra, interval and +gate results, readiness references, `MeasurementRecordV2`, and +`RepositoryHoldoutReport`. `admission.py` imports those types and owns census, metric rules, +thresholds, manifests, case inputs, authorization, scoring, admission reports, and fitted +component authorization. Measurement never imports admission. + +```python +@dataclass(frozen=True) +class ScopeKey: + family_id: str + ecosystem: str + +@dataclass(frozen=True) +class BaselineIdentity: + arm_id: str + population_digest: str + freeze_digest: str + claimed_scopes: tuple[ScopeKey, ...] + family_contract_digests: tuple[str, ...] + family_contract_set_digest: str + fact_schema_digest: str + fact_set_digest: str + acquisition_record_digest: str + analyzer_revision: str + configuration_digest: str + model_program_digests: tuple[str, ...] + model_program_set_digest: str | None + denominator_policy_digest: str + threshold_set_digest: str + outcome_mapping_digest: str + +@dataclass(frozen=True) +class ScopeCensus: + scope: ScopeKey + total_cases: int + pair_count: int + vulnerable_side_count: int + fixed_side_count: int + ordinary_case_count: int + semantic_site_capacity: int + model_call_capacity: int + +@dataclass(frozen=True) +class PopulationCensus: + population_digest: str + rows: tuple[ScopeCensus, ...] + case_labels_exposed: bool + census_payload_digest: str + review: IndependenceAttestation + census_digest: str + +class MetricId(Enum): + PAIRED_CORRECTNESS = "paired_correctness" + VULNERABLE_RECALL = "vulnerable_recall" + FIXED_FALSE_ALARM_RATE = "fixed_false_alarm_rate" + ORDINARY_FALSE_ALARMS_PER_100 = "ordinary_false_alarms_per_100" + PRECISION = "precision" + SEMANTIC_STAGE_COVERAGE = "semantic_stage_coverage" + MODEL_SUCCESS_RATE = "model_success_rate" + +class NumeratorKind(Enum): + CORRECT_PAIRS = "correct_pairs" + VULNERABLE_VIOLATION_CASES = "vulnerable_violation_cases" + FIXED_DEFECT_CASES = "fixed_defect_cases" + ORDINARY_DEFECT_CASES = "ordinary_defect_cases" + TRUE_DEFECT_CASES = "true_defect_cases" + SITES_RECOGNIZED = "sites_recognized" + SUCCESSFUL_MODEL_CALLS = "successful_model_calls" + +class DenominatorKind(Enum): + PAIRS = "pairs" + VULNERABLE_SIDES = "vulnerable_sides" + FIXED_SIDES = "fixed_sides" + ORDINARY_CASES = "ordinary_cases" + ALL_DEFECT_CASES = "all_defect_cases" + SITES_CONSIDERED = "sites_considered" + MODEL_CALLS = "model_calls" + +@dataclass(frozen=True) +class MetricRule: + metric_id: MetricId + scopes: tuple[ScopeKey, ...] + numerator: NumeratorKind + denominator: DenominatorKind + +class IntervalMethod(Enum): + WILSON_95 = "wilson_95" + +@dataclass(frozen=True) +class DenominatorPolicy: + rules: tuple[MetricRule, ...] + interval_method: IntervalMethod + policy_digest: str + +class Statistic(Enum): + COUNT = "count" + RATE = "rate" + PER_100 = "per_100" + WILSON_LOWER = "wilson_lower" + WILSON_UPPER = "wilson_upper" + +class Comparator(Enum): + AT_LEAST = "at_least" + AT_MOST = "at_most" + +@dataclass(frozen=True) +class Threshold: + metric_id: MetricId + statistic: Statistic + comparator: Comparator + target: float + +@dataclass(frozen=True) +class AdmissionValidationContext: + evidence: EvidenceValidationContext + admission_review_policy: ReviewIndependencePolicy + proof_trust_policy: ProofTrustPolicy | None + context_digest: str + +@dataclass(frozen=True) +class AdmissionManifestV2: + manifest_payload_digest: str + review: IndependenceAttestation + label_isolated: bool + labels_opened: bool + identity: BaselineIdentity + census: PopulationCensus + readiness_digests: tuple[str, ...] + custody_log_digest: str + denominator: DenominatorPolicy + thresholds: tuple[Threshold, ...] + threshold_set_digest: str + threshold_lineage_manifest_digest: str + validation_context_digest: str + manifest_digest: str + +class AdmissionCaseRole(Enum): + VULNERABLE = "vulnerable" + FIXED = "fixed" + ORDINARY = "ordinary" + +@dataclass(frozen=True) +class AdmissionCaseResult: + case_id_digest: str + scope: ScopeKey + role: AdmissionCaseRole + pair_id_digest: str | None + evidence_inputs: tuple[EvidenceValidationInput, ...] + defect_claims: tuple[DefectClaim, ...] + satisfactions: tuple[BoundedSatisfaction, ...] + silence: tuple[BoundedSilence, ...] + proof_attachments: tuple[ProofAttachment, ...] + coverage: CoverageFunnel + input_manifest: InputManifest + input_receipts: tuple[InputReadReceipt, ...] + complexity: ChangeComplexity + budget: BudgetDerivation + total_reply_ms: int + result_digest: str + +@dataclass(frozen=True) +class MetricResult: + metric_id: MetricId + numerator: int + denominator: int + value: float | None + interval_method: IntervalMethod + interval_lower: float | None + interval_upper: float | None + +@dataclass(frozen=True) +class OutcomeCount: + outcome: NeutralOutcome + count: int + +@dataclass(frozen=True) +class BasisCount: + basis: EvidenceBasis + count: int + +class GateKind(Enum): + INSTRUMENT = "instrument" + BUDGET = "budget" + QUALITY = "quality" + READINESS = "readiness" + +@dataclass(frozen=True) +class GateResult: + kind: GateKind + gate_id: str + passed: bool + reasons: tuple[str, ...] + evidence_digests: tuple[str, ...] + +class AdmissionDecision(Enum): + GO = "go" + NO_GO = "no_go" + +@dataclass(frozen=True) +class ReadinessReference: + stage: ReadinessStage + result: StageResult + result_digest: str + +@dataclass(frozen=True) +class MeasurementRecordV2: + identity: BaselineIdentity + family_decision_vocabulary: tuple[str, ...] + family_decision_counts: tuple[ReasonCount, ...] + neutral_outcome_counts: tuple[OutcomeCount, ...] + evidence_basis_counts: tuple[BasisCount, ...] + metrics: tuple[MetricResult, ...] + stage_funnel: CoverageFunnel + instrument_failures: tuple[str, ...] + model_outcome_counts: tuple[ReasonCount, ...] + model_failures: tuple[str, ...] + runtime_ms: int + prerequisite_readiness: tuple[ReadinessReference, ...] + record_digest: str + +@dataclass(frozen=True) +class RepositoryHoldoutReport: + component_digest: str + role: ReadinessEvidenceUse + training_population_manifest: ContentRef + holdout_population_manifest: ContentRef + training_repository_digests: tuple[str, ...] + holdout_repository_digests: tuple[str, ...] + fitting_lineage_manifest_digest: str + measurement: MeasurementRecordV2 + report_digest: str + +@dataclass(frozen=True) +class AdmissionReportV2: + identity: BaselineIdentity + census_digest: str + validation_context_digest: str + authorization_digest: str + label_release_receipt_digest: str + case_result_digests: tuple[str, ...] + in_scope_case_count: int + whole_population_case_count: int + measurement: MeasurementRecordV2 + instrument_results: tuple[GateResult, ...] + budget_results: tuple[GateResult, ...] + readiness_results: tuple[GateResult, ...] + quality_results: tuple[GateResult, ...] + decision: AdmissionDecision + report_digest: str + +@dataclass(frozen=True) +class AdmissionAuthorization: + manifest_digest: str + validation_context_digest: str + readiness_digests: tuple[str, ...] + custody_log_digest: str + feasibility_digest: str + authorization_digest: str + +@dataclass(frozen=True) +class AdmissionPreparation: + authorization: AdmissionAuthorization + label_release_request: PopulationUseRequest + preparation_digest: str + +def authorize_admission( + manifest: AdmissionManifestV2, + readiness: Sequence[ReadinessResult], + context: AdmissionValidationContext, + custody: PopulationCustodyPort, + resolver: ContentResolver, +) -> AdmissionPreparation: ... + +def score_admission( + manifest: AdmissionManifestV2, + preparation: AdmissionPreparation, + label_release_receipt: PopulationUseReceipt, + cases: Sequence[AdmissionCaseResult], + context: AdmissionValidationContext, + custody: PopulationCustodyPort, + resolver: ContentResolver, +) -> AdmissionReportV2: ... + +def authorize_fitted_component( + evidence: FittedComponentEvidence, + probe_inputs: Sequence[ProbeEvaluationInput], + context: AdmissionValidationContext, + fitting_lineage: LineageManifest, + repository_holdout: RepositoryHoldoutReport, + admission: AdmissionReportV2, + custody: PopulationCustodyPort, + resolver: ContentResolver, +) -> None: ... +``` + +`PopulationCensus.case_labels_exposed` and `AdmissionManifestV2.labels_opened` must both be +false during validation. The census and manifest payload digests exclude their review +attestations but include every substantive field; each `IndependenceAttestation.subject_digest` +must equal its payload digest. Producer and reviewer identities must differ and satisfy the +context's admission-review policy and forbidden-pair rules; the reviewer must be admitted, +and the conflict disclosure must resolve under the policy's store and byte limits. A +self-review, unknown reviewer, fabricated digest, or review copied from another payload is +refused before feasibility. + +The manifest identity, census population, claimed scopes, +contract-set, facts, acquisition, analyzer, configuration, model-program set, denominator, +threshold set, and outcome mapping must all match. `authorize_admission` re-derives the +validation-context digest and validates the actual family catalog, fact packs, model +declarations, lineages, and policies. The catalog's exact sorted contract-digest set—not a +superset—must equal `BaselineIdentity.family_contract_digests`, and +`catalog_digest` must equal `family_contract_set_digest`; every +claimed scope resolves exactly once to a contract that declares the ecosystem. Fact and +model sets are re-derived from the context and must equal the baseline identities. The +baseline outcome-mapping digest is re-derived from the exact ordered decision rules in +those contracts. + +The manifest's sorted thresholds must re-derive both its and the baseline's +`threshold_set_digest`. The named threshold lineage must be present in the context, target +that digest, satisfy a required `THRESHOLD` expectation for that same digest, and validate +against current complete custody chains while protecting the admission population. +All other referenced prompt, example, fitted-model, fact, matcher, mapping, and promotion +lineages in `context.evidence` are similarly validated before label opening. This makes requirement 13.1 an +object-level check rather than a claimed digest. + +The manifest custody-log digest must equal the current complete reviewed-custodian log and +that log must leave the population available. `authorize_admission` also validates every +named readiness receipt from instrument through development, performs label-blind threshold +feasibility, and returns an `AdmissionPreparation`. Its `ADMISSION_LABEL_OPEN` request has +`labels_accessed=True`, names the authorization digest as the influenced artifact, and +commits to the authorization's exact custody-log head. The benchmark custodian durably +compare-and-swap appends the derived `AVAILABLE` to `SPENT` receipt before releasing labels; +it never restores availability. A crash before append releases no labels, a crash after +append leaves the population spent, and concurrent losers receive no label bytes. + +`score_admission` requires that spend receipt to be the current-chain direct successor of +the authorization head and to match the preparation byte for byte. It refuses a different +manifest, context, readiness set, census, case population, or receipt. For every case it +calls `validate_evidence_input` for every `evidence_inputs` member, then performs instrument +and budget validation, and verdict/proof validation using the context-bound policies. +Proof attachments are applied only to those validated base records and the resulting claims +must equal the supplied claims canonically. +Every case model declaration must be an exact member of `context.evidence.model_programs`; every +pack and lineage reference must resolve to the context-bound object rather than an embedded +substitute. +Claims, satisfaction, silence, decision vocabulary, metrics, and gates are then derived +from those revalidated objects. A forged family decision, substituted contract, invalid +fact review or match, contaminated lineage, or untrusted proof fails before any metric or +gate is produced. The final report binds the validation context, authorization, and +pre-label spend receipt; there is no post-scoring population-use request. + +The report validates case counts back to the census, reports paired vulnerable/fixed +behavior and ordinary false alarms through registered metrics, imports semantic-stage and +model-call outcomes, and stratifies findings by evidence basis so model-corroborated and +deterministic claims are not silently merged. Policy validation fixes the standard metric +algebra: correct pairs are pairs with a vulnerable-side defect and no fixed-side defect; +recall is vulnerable cases with a defect over vulnerable cases; fixed and ordinary false +alarms are cases with at least one defect over their respective case denominators (ordinary +is scaled per 100); precision is vulnerable defect cases over all defect cases; semantic +coverage is recognized over considered sites; and model success is successful over attempted +calls. Success thresholds are `AT_LEAST` for correctness, recall, precision, coverage, and +model success, and `AT_MOST` for false alarms; reversed comparators are refused. Every rate +publishes numerator, denominator, and Wilson 95% interval. A zero denominator is serialized +with null value/bounds, never as zero or clean, and cannot pass a threshold. + +Fitted-component authorization requires `RepositoryHoldoutReport.role` to be exactly +`REPOSITORY_HOLDOUT`, resolves both population manifests, proves their repository sets are +disjoint, matches component/training/lineage/report digests, validates current custody for +the fitting lineage with subject `component_digest` and a required `FITTED_MODEL` +expectation for that digest, and requires +the admission report to be a `GO` produced from a population that was untouched before its +bound label-release spend. It re-derives `context.context_digest`, requires it to equal the +report's validation-context digest, and requires exactly one baseline-listed declaration in +`context.evidence.model_programs` whose `fitted_component_digest` equals +`evidence.component_digest`. The selected declaration supplies the family-contract digest; +each probe's two embedded records must agree on family and ecosystem, both probe and record +contract digests must equal the declaration, and that exact contract is resolved from +`context.evidence.family_catalog`. It also requires nonempty unique probe inputs, checks every probe and +observation against that component and resolved contract, calls `evaluate_probe` with the actual +records, surfaces, `context.evidence.fact_review_policy`, and resolver, and validates fitting +lineage with `context.evidence.custody_policy`. Every derived result must be +valid and passed. The derived digest tuple must equal +`FittedComponentEvidence.probe_result_digests`; a fabricated passed summary cannot +authorize the component. + +Every admission report must contain `PAIRED_CORRECTNESS`, `VULNERABLE_RECALL`, +`FIXED_FALSE_ALARM_RATE`, `ORDINARY_FALSE_ALARMS_PER_100`, `PRECISION`, and +`SEMANTIC_STAGE_COVERAGE` with nonzero pre-registered denominators, plus model success when +applicable. It must also contain evidence-backed instrument, budget, readiness, and quality +gate results. `GO` is invalid if any mandatory metric or gate is absent, any instrument +receipt failed, or any required gate did not pass. + +### Decision and Consumer Edge + +#### Verdict, Proof Attachment, and Consumer + +| Field | Detail | +|---|---| +| Intent | Turn validated evidence into bounded outputs and attach independently demonstrated proof | +| Requirements | 6.4, 7.1–7.4, 9.4, 9.6, 14.1–14.5 | + +**Responsibilities and constraints** + +- `DefectClaim` carries the complete evidence record, evidence basis, exact defect location, + and a change anchor inside the analyzed scope. Its basis must equal the record decision's + basis. +- `BoundedSatisfaction` carries complete question IDs, supporting records, and its exact + bounded scope. `BoundedSilence` carries affected questions, named reasons, record digests, + and a coverage digest. +- Incomplete paths remain silence unless another complete deterministic path or a matching + demonstrated proof independently supports a claim. +- A proof attachment contains a canonical differential proof record: reviewed producer, + sandbox and oracle identities; typed affected and safe observations; and a + content-addressed witness. Core re-derives its digest and requires the producer in the + supplied immutable trust policy. Only `demonstrated` with affected-only observation may + derive execution-backed basis. +- Proof attachment does not enable blocking; it supplies the evidence basis consumed by + downstream capability policy. +- `PushRequest` supplies the immutable analyzed scope, workload class, measured budget, and + measured `ChangeComplexity`, and request identity. Core validates region and transition + counts against scope/records, resolves the changed-byte measurement, and validates every + claim anchor and record location. The measurement bytes must bind the same base, head, + change digest, regions, transition count, and changed size; core does not choose regions. +- All work, including optional model and proof attachment processing, is accounted against + the measured push budget supplied by the consumer. + +**Contracts:** Service, State. + +```python +@dataclass(frozen=True) +class DefectClaim: + record: EvidenceRecord + basis: EvidenceBasis + location: SourceLocation + change_anchor: SourceRegion + +@dataclass(frozen=True) +class BoundedSatisfaction: + question_ids: tuple[str, ...] + records: tuple[EvidenceRecord, ...] + scope: AnalyzedScope + +@dataclass(frozen=True) +class BoundedSilence: + question_ids: tuple[str, ...] + reasons: tuple[str, ...] + evidence_record_digests: tuple[str, ...] + coverage_digest: str + +@dataclass(frozen=True) +class ProofProducerIdentity: + producer_id: str + verifier_revision: str + sandbox_profile_digest: str + oracle_contract_digest: str + review_digest: str + +class ProofOutcome(Enum): + DEMONSTRATED = "demonstrated" + INCONCLUSIVE = "inconclusive" + FAILED = "failed" + +@dataclass(frozen=True) +class ProofSideResult: + input_digest: str + observed: bool + observation_digest: str + run_count: int + result_digest: str + +@dataclass(frozen=True) +class ControlBinding: + affected_change_input_digest: str + safe_input_digest: str + control_manifest: ContentRef + reviewer_identity: str + binding_digest: str + +@dataclass(frozen=True) +class DifferentialProofRecord: + schema_version: str + producer: ProofProducerIdentity + affected: ProofSideResult + safe: ProofSideResult + control: ControlBinding + witness: ContentRef + outcome: ProofOutcome + record_digest: str + +@dataclass(frozen=True) +class ProofVerificationReceipt: + producer_identity_digest: str + proof_record_digest: str + affected_result_digest: str + safe_result_digest: str + witness_content_digest: str + attestation: ContentRef + receipt_digest: str + +@dataclass(frozen=True) +class ProofTrustPolicy: + reviewed_producer_digests: tuple[str, ...] + reviewed_control_reviewer_digests: tuple[str, ...] + trusted_store_identities: tuple[str, ...] + maximum_artifact_bytes: int + policy_digest: str + +@dataclass(frozen=True) +class ProofAttachment: + proof_id: str + family_id: str + packet_digest: str + location: SourceLocation + proof: DifferentialProofRecord + verification: ProofVerificationReceipt + +@dataclass(frozen=True) +class PushRequest: + request_id: str + scope: AnalyzedScope + workload_class: str + complexity: ChangeComplexity + budget: BudgetDerivation + +@dataclass(frozen=True) +class PushVerdict: + request_digest: str + defects: tuple[DefectClaim, ...] + satisfactions: tuple[BoundedSatisfaction, ...] + silence: tuple[BoundedSilence, ...] + input_record_digests: tuple[str, ...] + complexity: ChangeComplexity + total_reply_ms: int + budget_derivation_digest: str + +def attach_proof( + claim: DefectClaim, + attachment: ProofAttachment, + trust: ProofTrustPolicy, + resolver: ContentResolver, +) -> DefectClaim: ... + +def consume_records( + request: PushRequest, + inputs: Sequence[EvidenceValidationInput], + validation_context: EvidenceValidationContext, + custody: PopulationCustodyPort, + total_reply_ms: int, + resolver: ContentResolver, + attachments: Sequence[ProofAttachment] = (), + proof_trust: ProofTrustPolicy | None = None, +) -> PushVerdict: ... +``` + +`attach_proof` validates the existing claim's packet, family, exact location, reviewed +producer, control reviewer, canonical proof and verification-receipt digests, and the +differential condition `affected.observed is True` and `safe.observed is False`. The +affected input must equal the packet's `change_input_digest`; the safe input must equal the +reviewed control binding. The resolver must return witness, control-manifest, attestation, +and storage bytes whose hashes, lengths, receipts, retention policies, and trusted store +identities match. An opaque reference or self-asserted producer is insufficient. The +function returns a new immutable claim whose derived record carries proof and producer +digests in `DecisionProvenance`, leaving the packet untouched. + +`consume_records` first calls `validate_evidence_input` for every raw input and creates base +claims only from the returned records, then applies matching attachments through +`attach_proof`; nonempty attachments require the trust policy, and the shared resolver also +verifies the complexity measurement. It preserves every bounded-silence +reason verbatim, rejects records outside the request scope, and applies `check_budget` +before returning a verdict that retains the same complexity and derivation. A missing or +non-positive elapsed or complexity measurement is an instrument failure, never an implied +fast run. + +### Migration Components + +`legacy.py` decodes and validates historical v1 traces and reports. It never constructs v2 +determinate outcomes without a family adapter and never emits a new v1 artifact. +Authorization conversion moves to `semantic/obligations/core_adapter.py`, where old fields +map to the authorization family contract. Missing per-hop completeness remains incomplete. +The DSPy and search adapters similarly map external typed records without leaking their +dependencies into core. ## Data Models -- **TraceRecord**: entry identity; hops (each with call-target completeness); operation - (kind, resource, authority); witnesses (provenance + match relations); guards (effects); - completeness reasons; judgment; digests. Canonical JSON, sorted keys (existing convention). -- **Verdict**: `ProvenClaim{trace_ref, witness, path, line, family}` or - `BoundedSilence{path, reason, coverage}`. Never a bare "clean". -- **Coverage**: entry points + frameworks recognized; per-call exactly one of - complete/incomplete/unresolved; unchecked operations with named reasons. -- **BudgetSample**: change complexity + measured seconds; **Budget**: per-class percentile - derivation with n and hardware disclosed; configurable, never a single constant. +### Domain Model and Invariants + +| Aggregate | Authoritative identity | Invariants | +|---|---|---| +| `EvidencePacket` | Digest of canonical deterministic evidence | No model/proof data; all roles declared; every relation has one status | +| `Decision` | Packet digest plus family decision, basis and decision provenance | One neutral outcome; bounded satisfaction; named indeterminate reasons; model/proof identity retained outside packet | +| `EvidenceRecord` | Digest of packet and decision | Decision packet digest matches; immutable derivation only | +| `FamilyContract` | Family, version and canonical contract digest | Closed fields, roles, decisions, residuals and reasons | +| `FamilyCatalog` | Canonically sorted exact contract set | Every digest re-derived; unique family/version/schema; exact ecosystem resolution | +| `FactPack` | Contract digest plus normalized pack and review identity | Stable fact IDs; reviewed content; valid scope and structural requirements | +| `FactMatchEvidence` | Pack, fact, exact structural observations, matcher behavior and lineage | Complete relation only from the full validated requirement set and current uncontaminated matcher lineage | +| `ReadinessResult` | Stage, analysis identity and predecessor digests | Positive, negative and ambiguity evidence; exact evidence use; no skipped predecessor | +| `ModelResult` | Request, packet, response and program identities | Preserved response; frozen program declaration and validated lineage; declared mapping; no evidence rewrite or role collision | +| `EvidenceValidationInput` | Record, named inputs, optional model derivation and input digest | Full provenance/fact/model replay before any claim, metric, or gate | +| `ReversedPairProbe` | Reviewed semantic projections and component digest | Equal security semantics, reversed surface shape and pre-registered tolerance | +| `PopulationCustodyLog` | Embedded requests, contiguous receipts and current head | Complete current history; reviewed custodian; spent status is monotonic | +| `PopulationCensus` | Population, scope cardinalities, payload and independent review attestation | Aggregate and label-blind; no case IDs; same population as manifest; no self-review | +| `BaselineIdentity` | Arm, population/freeze, claimed scopes, contract set, fact schema/set, acquisition, analyzer, configuration, model set, denominator, threshold set and outcome mapping | Any changed field creates a new baseline | +| `AdmissionPreparation` | Manifest/context authorization and expected custody head | Feasible pre-registration; label-release request spends before bytes open | +| `AdmissionReportV2` | Baseline, validation context, authorization, spend receipt and case/result digests | Canonical input revalidation; in-scope scoring; all gates reported | +| `RepositoryHoldoutReport` | Component, disjoint repository manifests and measurement | Explicit holdout role; matches fitting lineage and component identity | +| `DifferentialProofRecord` | Reviewed producer, control-bound sides and resolved witness/verification receipt | Affected change identity, reviewed safe control, trusted producer/store policy | +| `PushVerdict` | Request and record digests | No unbounded clean result; proof basis does not imply rollout permission | + +### Canonical Data Contract + +- Wire format is JSON object schema `openultrasast.detection-core.v2`. +- Enum values use their declared lowercase strings. +- Mappings serialize in key order; semantic sequences retain declared order. +- Sets are prohibited at the wire boundary and normalized to sorted tuples before record + construction. +- Non-finite numbers, duplicate IDs, unknown fields, and unknown schema versions are + refused. +- SHA-256 digests use the full lowercase hexadecimal representation in v2. Legacy truncated + identities remain confined to the v1 decoder. + +No database schema is introduced. Benchmark workflows choose storage and preserve canonical +bytes and digests. ## Error Handling -- Zero-byte or unreadable input → instrument failure; the run is invalid, never clean (Req 3). -- Digest mismatch → refuse the result (Req 6). -- Manifest unreviewed / label-exposing / acquisition-changed / spent → ordered refusal, no fallback (Req 7). -- Budget exceeded → gate failure naming change, elapsed time, and the derivation exceeded (Req 4). -- Indeterminate evidence → indeterminate verdict with reasons; never converted (Req 2). + +### Error Strategy + +Invalid contracts or identities fail fast. Expected analysis uncertainty becomes a typed +indeterminate record. Readiness and admission failures are state outcomes, not exceptions +that can be mistaken for a clean scan. + +| Category | Examples | Core response | +|---|---|---| +| Instrument failure | Unreadable or zero-byte tracked input | Invalidate run; emit no clean or satisfied result | +| Contract violation | Unknown field, role, decision, schema, or ecosystem | Refuse record with exact field and contract identity | +| Semantic uncertainty | Missing relation, ambiguous match, unmet condition | Preserve partial packet; return indeterminate with named reasons | +| Model failure | Timeout, parse error, identity mismatch | Preserve deterministic decision; record model failure and coverage | +| Proof failure | Inconclusive, mismatched side, missing witness | Refuse attachment; preserve prior basis | +| Readiness failure | Failed, missing, or stale predecessor | Stop promotion; report layer and correction lever | +| Admission configuration | Impossible threshold or wrong denominator | Refuse before label opening; do not report no-go measurement | +| Budget failure | Elapsed time exceeds derived budget | Name change, elapsed time and derivation; return bounded failure | +| Provenance mismatch | Any digest cannot be re-derived | Refuse affected result and dependent later receipts | +| Review mismatch | Self-review, unknown reviewer, copied attestation, or unreadable disclosure | Refuse before feasibility or semantic use | +| Custody mismatch | Old/incomplete log, broken chain, or prior spend | Refuse use/admission; preserve monotonic spent status | + +### Monitoring + +Measurement artifacts carry stage counts, missing reasons, model-call failures, instrument +failures, runtime, and identity digests. They do not contain secrets, raw model credentials, +exploit payloads, or repository contents. Operator notification wiring remains outside this +spec and consumes only status, counts, and digests. ## Testing Strategy -- Unit: trace serialization determinism; verdict rules; coverage bucket exclusivity; budget - derivation (fixed samples → fixed budget); digest refusal; manifest refusal order. -- Contract conformance: prover's existing Trace satisfies core/trace.py; admission suite - output byte-identical before/after adoption (regression gate). -- Probes: reversed-pair probe catches a shape-only component (deletion-count control). -- Measurement: admission suite re-run on the pilot pool must reproduce the 2026-10-30 - NO-GO numbers exactly (instrument check). -## Performance & Scalability -- Budget is measured (Req 4): fit the complexity→reply-time relation on the dev (483) and - pilot (264) record sets as measurement only; publish per-class budgets (n, percentile, - hardware). The static 120s remains as a provisional default until the first derivation - is published; changing it starts a new baseline (Req 12). +### Unit Contract Tests + +- `test_core_canonical.py`: deterministic bytes under mapping-order variation; sequence + order retention; non-finite number and unsupported-value refusal (`1.4`, `10.4`). +- `test_core_evidence.py`: all relation kinds, per-relation status, product/fact provenance, + packet-decision binding, and immutable derivation (`1.1–1.3`, `2.1–2.5`, `4.1–4.3`). +- `test_core_family.py`: family registration matrix, canonical catalog and exact resolution, + ecosystem/schema refusal, closed roles/fields/decisions, required evidence, residual + prerequisites, and a complete `FamilyCapabilityDeclaration` whose contract, readiness + fixture, development population, and admission boundary fields are each removed or + corrupted parametrically and refused (`3.1–3.4`, `9.1`, `14.5`). +- `test_core_coverage.py`: exact stage counts, relation-bucket exclusivity, nonzero read + proof including a missing receipt, and first failed-output diagnosis at every stage + (`5.1–5.5`). +- `test_core_budget.py`: transition-based complexity, derivation metadata, deterministic + class budget, resolved changed-byte measurement, absent/zero timing or complexity, scope + escape, and named gate failure (`6.1–6.4`). +- `test_core_verdict.py`: exact claim location, bounded satisfaction, reasoned silence, + mixed complete/incomplete paths, proof trust/store/control binding, and internal + attachment aggregation (`7.1–7.4`, `9.6`). +- `test_core_readiness.py`: exact stage order, fixture triplets, stopped promotion, + schema/vocabulary/outcome-mapping transitive staleness, and enforced + regression/development/fresh-admission/integration evidence-use separation (`8.1–8.6`). +- `test_core_model.py`: request/result identities, packet binding, actual program lineage, + current custody-chain validation, allowed mappings, + inline and content-addressed response retention, proposer/adjudicator separation, model + failure, and basis limits (`9.1–9.5`, `10.3`, `13.1`). +- `test_core_validation.py`: full context/input digest reconstruction, exact catalog and + pack/model membership, resolver-backed provenance, deterministic/model derivation replay, + and proof that admission and consumer entry points cannot bypass the same validator with + a raw record (`4.4`, `10.1–10.4`, `12.6`, `14.1`). +- `test_core_provenance.py`: complete stamp, field-by-field tamper refusal, lineage + acceptance, reviewed custody, canonical use/spend receipts, contamination, and + refusal of omitted/old-prefix spend history. A matcher-specific case proves that + label-influenced matching behavior creates a chained `SPENT` receipt and a later claimed + `AVAILABLE` state cannot re-admit the population; wrong influenced-artifact expectations + are refused even when manifest subjects match (`10.1–10.4`, `13.1`, `13.3–13.4`). +- `test_core_factpack.py`: schema/scope/review matrix, structural match, name-only negative + control, exact requirement-ID observations, fabricated match refusal, + source/documentation/independent-behavior review and lineage, ambiguity, and partial-pack + behavior. For a dependency-fact-only role it refuses product provenance, preserves a + missing fact as incomplete/indeterminate, and accepts the same effect only from a reviewed + pack plus valid structural match. Direct `apply_fact` use with an unreviewed pack, wrong + review policy, or protected-population matcher contamination is refused (`4.4`, `10.4`, + `11.1–11.5`, `13.3–13.4`). +- `test_core_admission.py`: consume-only refusal order, readiness prerequisites, threshold + feasibility over label-blind census capacities and Wilson bounds, the historical + impossible-denominator regression, scoring derived from full case evidence, fixed + precision/Wilson-95 schema, and refusal of missing standard metrics or gates. It also + covers spend-before-label crash/concurrency order and refuses a forged family decision, + substituted contract, invalid fact review, or contaminated model/threshold/matcher + lineage before metrics or gates. Manifest and census self-review, unknown reviewers, + copied attestations, and unresolved conflict disclosures fail before feasibility + (`12.1–12.6`, `13.1`, `13.4`). +- `test_core_measurement.py`: required schema, basis stratification, Wilson counts, baseline + mutation matrix, zero receipts, and prerequisite stage result/digest references. A + two-arm case over one population keeps family/ecosystem scope, census rows, denominator + rules, and reports isolated and refuses cross-arm aggregation as one baseline + (`11.6`, `15.1–15.5`). +- `test_core_probes.py`: reject unequal semantic projections as invalid; catch a + deletion-count component on core-derived, independently reviewed + equal-semantics/reversed-surface pairs; reject digest-only review; accept a relation-based + component; reject a fabricated passed summary by re-evaluating the canonical probe + inputs; and require disjoint resolved repository holdout, fitting lineage, and untouched + admission (`13.2`). + +### Reference-Family Conformance + +Both test-only families run the same positive, negative, and missing-evidence contract: + +- Obligation family: operation plus required constraint; exercises control and constraint + relations. +- Taint-flow family: source, transformations, sink, and sanitizer state; exercises ordered + value flow. + +Each family must produce violation, bounded satisfaction, and indeterminate records without +adding a family-specific field to a core module (`3.5`, `8.2`). These fixtures are regression +data only and do not establish product capability or admission. + +### Adapter and Integration Tests + +- Authorization adapter converts existing traces conservatively; missing per-hop + completeness remains indeterminate and no target vocabulary enters core (`14.3`). +- Hunt contract adapter translates a fake executor result; core imports no DSPy and accepts + only a matching, declared result. DSPy/provider execution remains hunt-owned and outside + this spec (`9.1–9.5`). +- Search adapter accepts a canonical demonstrated differential record from a reviewed + producer and trusted resolver with matching change/control, side, receipt, store, and + witness digests; rejects self-asserted, opaque, inconclusive, same-side, or mismatched + records (`9.6`, `14.2`). +- Push request scope through evidence and verdict preserves silence reasons and accounts for + the measured deadline (`6.4`, `14.1`, `14.4`). +- Readiness failure prevents development, admission, and integration calls; a semantic + digest change makes every dependent receipt stale (`8.3–8.5`, `12.1`). +- Benchmark custody integration proves the guarded operation durably appends the exact + head-bound spend before releasing label bytes; injected crashes before/after append and + two concurrent preparations release labels to at most the compare-and-swap winner + (`12.2`, `13.4`). +- Legacy and v2 validators coexist by explicit schema version; committed historical reports + remain byte-unchanged and cannot be written through the v2 API (`14.3`, `15.3`). + +### End-to-End Readiness Test + +One small authored repository per reference family runs through: + +```text +nonzero input read +→ semantic recognition +→ typed transitions +→ security object and constraints +→ vulnerable and fixed pair +→ development report +→ synthetic admission manifest +→ guarded label-release spend +→ consumer verdict +``` + +Each run injects one expected failure at every stage and confirms later stages refuse to +act. Synthetic admission manifests test orchestration without spending a real population. + +### Performance and Load Tests + +- Canonicalization and validation measure elapsed time against relation and observation + counts and publish the workload sample; no static latency claim is introduced. +- Budget derivation uses recorded changed regions, transitions, bytes, total elapsed time, + sample count, percentile, workload, and hardware. +- Large transition sets verify linear validation behavior and bounded memory without + loosening semantic checks. +- Optional model calls and proof attachment processing remain inside the total push deadline. ## Security Considerations -- Label isolation: the core never reads pool/admission labels except through the validated - admission path; forbidden markers enforced at manifest validation (Req 7, 8). -- Repository identity masked in shared summaries (existing convention preserved). + +- Model outputs, fact packs, manifests, and proof attachments are untrusted until validated + against their contract and identities. +- A raw evidence record is likewise untrusted; admission and consumer paths both replay the + shared evidence-validation composition before deriving output. +- Content-addressed model-response, fact-match, review, complexity, control, and + proof-witness references are accepted only when + their full digest, byte length, media type, retention policy, storage receipt, and + trusted producer/store policy match; an opaque location alone is never evidence. +- Source text in an evidence packet is data, never an instruction channel; the hunt-owned + DSPy runtime uses structured signature fields and preserves prompt/program identities. +- No model response can create a semantic relation, claim execution-backed basis, or bypass + readiness/admission. +- Fact review binds normalized content, schema, scope, structural matches, independent + reviewer attestation, and review evidence in one digest. +- Label lineage is declarative and reviewable; the core can detect declared contamination + but cannot discover undisclosed training data. External review remains the trust boundary. +- Admission labels open only after analyzer, facts, matcher behavior, model, configuration, + readiness, scope, denominator, thresholds, and the current complete custody head are + frozen, independently reviewed where required, and validated. The custodian durably appends the head-bound spend before releasing + labels; scoring accepts only that current-chain receipt. +- Proof attachment verifies resolved identity, producer receipt, reviewed control binding, + and differential outcome; trusted search infrastructure retains sandbox and oracle + execution responsibility. + +## Performance and Scalability + +- Core validation is proportional to fields, transitions, observations, receipts, or cases + supplied to the corresponding contract; no whole-repository traversal occurs in core. +- Resolver-backed artifacts are length-checked before loading and bounded by the applicable + provenance, response, review, or proof policy; oversized content is refused before parsing. +- Change scoping is supplied by the push runner and recorded immutably. The core rejects + evidence or claim anchors outside that scope. +- Budget is derived per workload class from measurements and includes preparation, analysis, + model, proof attachment, validation, and reporting time. +- A changed canonical schema, complexity vocabulary, or workload-class assignment starts a + new baseline. ## Migration Strategy -1. Create `core/` contract package mirroring existing shapes (no behavior change). -2. Prover re-exports core types; conformance tests prove equivalence. -3. Runners adopt coverage/budget/provenance one at a time; admission suite re-run after each. -4. Admission modules become thin consumers of core/admission.py; 2026-10-30 NO-GO reproduced. -5. Fact-pack contract adopted by ruleset loader; family-scope appears in reports. + +```mermaid +flowchart LR + V1[Legacy v1 writes] --> V2[Add v2 contracts] + V2 --> References[Reference conformance] + References --> AuthAdapter[Authorization adapter] + AuthAdapter --> Benchmarks[Readiness and measurement v2] + Benchmarks --> Consumers[Model proof and push conformance] + Consumers --> StopV1[Stop v1 writes] + StopV1 --> RemoveV1[Remove v1 exports] + RemoveV1 --> LegacyRead[Read only legacy decoder] +``` + +1. **Foundation:** add canonical, semantics, evidence, family, and reference-family + contracts without changing v1 consumers. +2. **Adapter migration:** move authorization conversion out of engine trace code and emit v2 + in development. Preserve conservative incomplete hops and existing family behavior. +3. **Governance migration:** add readiness, measurement, admission, lineage, and model v2; + historical v1 validation remains separate. This creates a new baseline. +4. **Consumer conformance:** add DSPy and proof adapters and validate push consumption with + synthetic/reference records. Downstream rollout remains disabled. +5. **Single-writer cutover:** stop v1 writes after repository import and artifact-writer + censuses reach zero. Remove authorization-shaped top-level exports and the `trace.py` + shim. +6. **Historical retention:** retain `legacy.py` read-only validation for committed v1 + artifacts. It exposes no analyzer or writer API. + +Rollback is allowed before the single-writer cutover by keeping existing consumers on v1. +After cutover, rollback means reverting the complete migration change; it must not restore +parallel v1 writes. No migration phase re-runs authorization admission or changes its parked +status. + +## Supporting References + +- [Research and design decisions](research.md) +- [Python JSON documentation](https://docs.python.org/3.11/library/json.html) +- [DSPy signatures](https://dspy.ai/3.0.4/learn/programming/signatures/) +- [DSPy modules](https://dspy.ai/3.0.4/learn/programming/modules/) diff --git a/.kiro/specs/detection-core/requirements.md b/.kiro/specs/detection-core/requirements.md index 5a446da..6d26bff 100644 --- a/.kiro/specs/detection-core/requirements.md +++ b/.kiro/specs/detection-core/requirements.md @@ -2,139 +2,301 @@ ## Project Description (Input) -**Who has the problem:** the maintainer of OpenUltraSAST. Every detection arm built so far -(prover, hunter, push hook, proof arm) is corrected link by link, and corrections do not -compose — v1→v2 retunes transferred 0/17 on untouched cases, and the 2026-10-30 admission -run on the untouched pilot population returned NO-GO (0/12 pairs, in-family 0/3, 9/12 -repositories with zero framework recognition). +**Who has the problem:** the maintainer of OpenUltraSAST. The intended detection loop is +to augment family-specific model programs with deterministic program-analysis evidence. +Today that loop cannot be extended safely because the shared core embeds authorization +vocabulary, semantic gaps surface only during repository admission, and a clean-looking +zero can still hide a failure before the model receives useful evidence. -**Current situation:** detection is specified as per-family, per-arm runners around an -implicit core. The ordered trace exists for one family (access-control-prover), learned -components fit pool shortcuts (deletion count separates reversed fixes at 0.90), and -misses concentrate in unmeasured links: entry recognition, call resolution (11.9% -complete), guard dominance, and the 120s pre-push budget. +**Current situation:** the instrument-verified 2026-10-31 authorization diagnosis showed +two different failures. Framework-mediated reachability stopped before protected +operations, and paths that survived still lacked the constraint semantics needed to +distinguish vulnerable and fixed code. These are useful limits of the deterministic +evidence producer, not a reason to require it to become a complete vulnerability prover. +The core must expose such limits early and give each vulnerability family a stable way to +interpret the evidence without translating its semantics into authorization terms. -**What should change:** one specified core object — the security trace — with a contract -every arm, fact pack, and consumer reuses: given a push (base, head) within a fixed budget, -output proven defect claims (each with its witness) plus explicit coverage of what was not -decided, on repositories never seen. Models are bounded to residuals; admission on an -untouched population gates the trace itself; no fitted component ever sees pool labels. +**What should change:** the detection core shall provide a family-neutral evidence and +verdict envelope. Deterministic analysis shall contribute typed transitions, effects, +provenance, and explicit completeness. Each vulnerability family shall define its security +object and decision vocabulary and may use a bounded model program to answer residual +questions over that evidence. A staged readiness contract shall stop a family at the first +failing semantic layer, before population-level admission is attempted. + +## Boundary Context + +- **In scope**: the family-neutral evidence, outcome, trace, coverage, provenance, budget, + staged-readiness, admission, measurement, fact-pack, model-handoff, and consumer + contracts; family conformance requirements; failure-visible semantic-stage accounting. +- **Out of scope**: vulnerability-family detection algorithms and prompt content; + framework-specific fact content; model optimization; analysis of vendored dependency + source; the pre-push transaction and rollout policy; proof execution; enabling a new + vulnerability family for users. +- **Adjacent expectations**: the existing deterministic analysis is an evidence producer, + not a required complete prover; family-specific DSPy programs may consume the evidence + packet under the model-handoff contract; proof-based blocking remains a separate + consumer. A parked arm remains parked unless a separately reviewed strategy creates a + new arm, baseline, and admission population. ## Requirements -### Requirement 1: Canonical trace object -The detection core shall define one canonical trace object per analyzed change as the -shared security object for all detection arms. -1. When a change is analyzed, the detection core shall produce a canonical trace carrying - entry identity, ordered hops with call-target completeness, operation kind with resource - and authority, witnesses with provenance and match relations, guard effects, completeness - reasons, a final judgment, and the input digests of the analysis. -2. The detection core shall serialize every trace deterministically, so that identical - inputs produce byte-identical traces. +### Requirement 1: Family-neutral evidence record +The detection core shall define one canonical evidence record for every examined security +question within an analyzed change. +1. When a security question is analyzed, the detection core shall produce a canonical + evidence record carrying the family identity, security-object identity, analyzed scope, + ordered semantic transitions, observations with provenance, completeness reasons, + family decision, neutral outcome, evidence basis, and input digests. +2. When a transition or observation is derived from product code or reviewed semantic + knowledge, the detection core shall record its semantic role, source location, + provenance, and completeness independently of every other transition or observation. +3. When a family supplies family-specific evidence fields, the detection core shall + preserve those fields under the declared family schema without requiring them to use + another family's vocabulary. +4. The detection core shall serialize every evidence record deterministically, so that + identical inputs produce byte-identical records. -### Requirement 2: Three-way judgment vocabulary -The detection core shall judge every analyzed operation as exactly one of unguarded, -discharged, or indeterminate. -1. The detection core shall never convert an indeterminate judgment into a clean - (discharged or unguarded-free) result. -2. When evidence for a judgment is incomplete, the detection core shall return - indeterminate with the named incompleteness reasons. +### Requirement 2: Neutral outcome and evidence basis +The detection core shall separate a family-specific decision from the neutral outcome and +the basis supporting that outcome. +1. When a family reaches a decision, the detection core shall record exactly one neutral + outcome of violation, satisfied, or indeterminate and shall retain the family's own + decision code separately. +2. When an outcome is satisfied, the detection core shall report the exact bounded scope + over which satisfaction was established and shall not present it as repository-wide + safety. +3. When evidence is incomplete for the family decision, the detection core shall return + indeterminate with every named incompleteness reason. +4. If a required semantic relation is unavailable or ambiguous, the detection core shall + prevent that missing relation from creating a violation or satisfied outcome. +5. When an outcome is reported, the detection core shall identify whether its basis is + deterministic analysis, model corroboration, or execution-backed proof. -### Requirement 3: Coverage accounting -The detection core shall account for every input it did or did not use, so a zero result is -never indistinguishable from an unread input. -1. When a change is analyzed, the detection core shall report entry-recognition coverage, - per-call resolution outcomes with exactly one outcome bucket per call, and named - reasons for every unchecked operation. -2. If any input file is read as zero bytes, the detection core shall fail the run as an - instrument failure rather than reporting a clean result. +### Requirement 3: Vulnerability-family conformance contract +The detection core shall require every vulnerability family to declare the meaning and +limits of the evidence it consumes and the decisions it produces. +1. When a family is registered, the detection core shall require its family and schema + versions, claimed ecosystems, security-object definition, accepted semantic roles, + decision vocabulary, neutral-outcome mapping, completeness conditions, and named + indeterminate reasons. +2. When a family claims a violation or satisfied outcome, the detection core shall require + the family contract to identify the minimum evidence and completeness conditions for + that outcome. +3. When a family receives evidence outside its declared ecosystem or schema scope, the + detection core shall report the question as unsupported rather than applying a + different family's interpretation. +4. If a family decision code, evidence field, or semantic role is not declared by the + family contract, the detection core shall refuse the result. +5. The detection core shall demonstrate conformance with at least two reference families + that use materially different security objects before the family-neutral contract is + declared ready for downstream family specifications. -### Requirement 4: Measured budget and change scoping +### Requirement 4: Typed semantic transitions and effects +The detection core shall preserve reviewed semantic knowledge needed to connect product +code across framework and dependency boundaries without analyzing vendored source. +1. When reviewed semantic knowledge supplies a registration, invocation, call, return, + value transformation, operation effect, or constraint effect, the detection core shall + record the typed relation and the supplying fact identity in the evidence record. +2. When a typed relation is complete only under stated conditions, the detection core shall + retain those conditions and shall mark the relation incomplete when they are not met. +3. If multiple reviewed relations conflict or match ambiguously, the detection core shall + report the ambiguity and shall not choose a complete relation silently. +4. When vendored source is excluded, the detection core shall require a reviewed semantic + relation for any dependency behavior needed by a determinate family decision. + +### Requirement 5: Coverage and instrument accounting +The detection core shall account for every input and semantic stage it did or did not use, +so a zero result is never indistinguishable from unread or unmodeled input. +1. When a change is analyzed, the detection core shall report counts for inputs read, + security questions formed, semantic sites recognized, transitions materialized, + security objects constructed, constraints evaluated, determinate outcomes, and findings + emitted. +2. When product-code or fact-derived relations are considered, the detection core shall + assign each relation exactly one complete, incomplete, or unresolved outcome. +3. If a semantic site, transition, security object, or constraint cannot be modeled, the + detection core shall report the specific missing relation rather than interpreting its + absence as safe. +4. If any tracked input file is read as zero bytes or cannot be opened, the detection core + shall fail the run as an instrument failure rather than reporting a clean result. +5. When a stage produces zero outputs, the detection core shall identify the last stage + with verified nonzero input and the first stage that failed to produce its expected + output. + +### Requirement 6: Measured budget and change scoping The detection core shall derive its time budget from measurement, not from a fixed constant. 1. When a change is analyzed, the detection core shall measure and record reply time and - change complexity (changed regions, call count, changed size) for every change. -2. The detection core shall derive the per-change budget from the measured distribution - (sample count, percentile, reference workload and hardware disclosed) and shall publish - that derivation under the fixed measurement schema; the budget shall be configurable per - workload class, never a single static constant. -3. When a change exceeds the measured budget, the detection core shall fail the budget gate - naming the change, the elapsed time, and the budget derivation exceeded. -4. While a push is analyzed, the detection core shall limit analysis to regions affected by - the change and shall record the scope it used. + change complexity, including changed regions, transition count, and changed size. +2. When a workload-class budget is published, the detection core shall disclose the sample + count, percentile, reference workload, reference hardware, and derived budget. +3. When a change exceeds its measured workload-class budget, the detection core shall fail + the budget gate naming the change, elapsed time, and budget derivation exceeded. +4. While a push is analyzed, the detection core shall limit analysis to regions affected + by the change and shall record the scope it used. -### Requirement 5: Verdict contract -The detection core shall distinguish proven defect claims from bounded silence. -1. When the detection core reports a defect, the claim shall carry its witness evidence and - an exact location tied to the change. -2. When the detection core cannot decide a site, it shall report that site as undecided - with its coverage reason, and shall never present an undecided site as a clean result. +### Requirement 7: Verdict contract +The detection core shall distinguish supported defect claims, bounded satisfaction, and +bounded silence. +1. When the detection core reports a defect, the claim shall carry its evidence basis, an + exact location tied to the change, and the complete evidence record supporting the + claim. +2. When the detection core reports satisfaction, the claim shall carry the completed + security questions and the bounded scope over which they were decided. +3. When the detection core cannot decide a question, it shall report that question as + indeterminate with its coverage reasons and shall never present it as clean. +4. If a claim depends on an incomplete or unresolved semantic relation, the detection core + shall expose the affected question as bounded silence unless another complete path + independently supports the claim. -### Requirement 6: Provenance and digests -The detection core shall stamp every trace and verdict with the identities of its inputs. -1. The detection core shall record, for every analysis, the analyzer revision, fact-set - digest, acquisition-record digest, configuration digest, population digest, and - per-change input digest. -2. If any recorded digest cannot be re-derived from the named inputs, the detection core - shall refuse the result rather than accept it. +### Requirement 8: Staged readiness and promotion +The detection core shall require a vulnerability-family arm to pass each semantic readiness +stage before it can advance to broader evaluation. +1. When an arm begins development, the detection core shall require separate readiness + results for instrument validity, semantic recognition, transition construction, + security-object and constraint evaluation, vulnerable/fixed pair behavior, development + population behavior, fresh admission, and consumer integration. +2. When a readiness stage is evaluated, the detection core shall require a positive case, + a negative case, and an ambiguity or missing-evidence case appropriate to that stage. +3. If a readiness stage fails, the detection core shall stop promotion at that stage and + shall report the failing semantic layer and named correction lever. +4. If any prerequisite readiness result is missing, stale, or failed, the detection core + shall refuse to run a later population or integration gate as evidence of readiness. +5. When a semantic change invalidates an earlier readiness result, the detection core shall + mark every dependent later result stale. +6. When a family passes a stage, the detection core shall preserve the fixtures and results + as regression evidence without treating them as fresh admission evidence. -### Requirement 7: Admission gate contract -The detection core shall gate any generalization claim on an untouched population. -1. When an arm is proposed for pre-push use, the detection core shall require a reviewed - manifest naming the frozen analyzer, facts, acquisition, configuration, and an untouched - admission population before any admission label is opened. +### Requirement 9: Model-augmentation contract +The detection core shall support bounded family-specific model reasoning over deterministic +evidence without allowing model output to rewrite that evidence. +1. Where a family uses a model program, the detection core shall present the canonical + evidence record, its completeness state, the family contract, and the residual question + to be answered. +2. When a model program returns a family decision, the detection core shall preserve the + model response, program and configuration identities, evidence-record digest, and + declared decision mapping. +3. If deterministic evidence is incomplete, the detection core shall prevent a model + response from converting the affected question into a satisfied outcome or a + proof-backed violation. +4. When model output supports an advisory claim, the detection core shall label the claim + as model-corroborated and shall keep it distinct from deterministic and execution-backed + claims. +5. The detection core shall not allow a model to propose and independently adjudicate the + same claim within one evidence path. +6. The detection core shall reserve blocking verdicts for execution-backed proof with a + differential witness. + +### Requirement 10: Provenance and digests +The detection core shall stamp every evidence record and verdict with the identities of its +inputs. +1. When an analysis is recorded, the detection core shall include the analyzer revision, + family-contract digest, fact-set digest, acquisition-record digest, configuration + digest, population digest, and per-change input digest. +2. When an evidence record uses reviewed semantic knowledge, the detection core shall + include the semantic pack and fact identities needed to reproduce each derived relation + or effect. +3. When a model program contributes a decision, the detection core shall include the model, + program, prompt, and configuration identities needed to reproduce the call. +4. If any recorded digest or semantic fact identity cannot be re-derived from the named + inputs, the detection core shall refuse the result rather than accept it. + +### Requirement 11: Reviewed semantic fact packs as the extension point +The detection core shall admit family and ecosystem knowledge only through reviewed, +versioned semantic fact packs over the family-neutral contract. +1. When a semantic fact pack is admitted, the detection core shall validate its schema + version, claimed family and ecosystem scope, stable fact identities, declared semantic + roles, and review evidence. +2. When a semantic fact matches analyzed product code, the detection core shall permit the + consuming family to record the corresponding transition or effect only with the matching + evidence and fact provenance attached. +3. The detection core shall not allow a name-only match without the fact's required + structural evidence to create a complete semantic relation or determinate outcome. +4. If a semantic fact has an undeclared role, unknown field, invalid scope, missing review + evidence, or inconsistent digest, the detection core shall refuse the pack. +5. When admitted knowledge is insufficient to resolve a semantic relation, the detection + core shall preserve the partial evidence record and report the relation as incomplete or + unresolved. +6. The detection core shall report, per arm, the vulnerability families and ecosystems the + arm claims, so admission and rollout numbers are never read across unsupported scopes. + +### Requirement 12: Admission gate contract +The detection core shall gate every generalization claim on completed readiness stages and +a reviewed untouched population whose denominator matches the arm's declared scope. +1. When an arm is proposed for consumer use, the detection core shall require a reviewed + manifest naming the frozen analyzer, family contract, model program where applicable, + facts, acquisition, configuration, claimed families, claimed ecosystems, denominator + policy, readiness results, and untouched admission population before any admission + label is opened. 2. The detection core shall consume admission manifests without producing, mutating, or - re-labeling them, and shall refuse manifests that are unreviewed, label-exposing, + re-labeling them and shall refuse manifests that are unreviewed, label-exposing, acquisition-changed, or spent. -3. When admission evidence is scored, the detection core shall report paired - vulnerable/fixed behavior, ordinary false alarms, and every gate (budget, instrument, - pre-registered criteria) as an evidence-backed GO or NO-GO. +3. When an admission manifest is validated before label opening, the detection core shall + verify that every pre-registered success threshold is arithmetically reachable under the + declared family, ecosystem, and denominator policy. +4. If a threshold cannot be reached even when every in-scope case succeeds, the detection + core shall refuse the admission configuration rather than run or report a misleading + no-go result. +5. When a population contains cases outside an arm's declared scope, the detection core + shall score quality gates on the pre-registered in-scope denominator and shall report + whole-population coverage separately. +6. When admission evidence is scored, the detection core shall report paired + vulnerable/fixed behavior, ordinary false alarms, semantic-stage coverage, model-call + outcomes where applicable, and every budget, instrument, and quality gate as an + evidence-backed go or no-go result. -### Requirement 8: No fitting on evaluation labels -The detection core shall not allow any fitted component to train on evaluation labels. -1. The detection core shall prohibit tuned prompts, thresholds, or fitted models whose - training data includes pool or admission labels. -2. When a fitted component is proposed, the detection core shall require it to pass - reversed-pair probes (a fix and its reversed fix shall not be separable by the - component's decision features), per-repository holdout, and admission on an untouched - population before any claim is made. +### Requirement 13: Separation of development and evaluation +The detection core shall prevent evaluation labels from influencing an arm before its +generalization claim is scored. +1. The detection core shall prohibit prompts, examples, thresholds, fitted models, semantic + facts, or decision mappings whose preparation data includes admission labels. +2. When a fitted component is proposed, the detection core shall require reversed-pair + probes, repository-held-out development evaluation, and admission on an untouched + population before any generalization claim is made. +3. When semantic knowledge is acquired, the detection core shall require source, + documentation, or independently reviewed behavioral evidence that is isolated from + vulnerability, fix-location, and ordinary-change labels. +4. If evaluation labels influence semantic fact content, matching behavior, prompt + content, examples, thresholds, or promotion, the detection core shall mark the affected + population spent and shall refuse it for admission. -### Requirement 9: Bounded role for models -The detection core shall bound every model to residual questions over the trace. -1. Where a model judges a trace, the detection core shall present the deterministic trace - and require the judgment to be zero-shot over that trace. -2. The detection core shall not allow a model to propose-and-adjudicate the same claim, and - shall reserve blocking verdicts for demonstrated proofs with a differential witness. - -### Requirement 10: Family fact packs as the extension point -The detection core shall admit new vulnerability families only as fact packs over the core -trace contract. -1. When a new family is added, the detection core shall require the family's entry, - operation, and guard knowledge as reviewed fact packs stamped into the trace's - provenance, without changing the trace schema or judgment vocabulary. -2. The detection core shall report, per arm, the vulnerability families the arm claims, so - admission and rollout numbers are never read across families the arm does not cover. - -### Requirement 11: Consumer and re-parenting contract -The detection core shall be the single contract that existing detection specs reuse. +### Requirement 14: Consumer and re-parenting contract +The detection core shall be the single contract that existing and future detection specs +reuse. 1. The detection core shall define the interface through which the pre-push transaction - consumes traces and verdicts within its deadline. -2. The detection core shall define the interface through which proof-based blocking evidence - attaches to a trace verdict. + consumes evidence records and verdicts within its deadline. +2. The detection core shall define the interface through which proof-based blocking + evidence attaches to an existing claim. 3. When an existing arm's behavior is described, the detection core shall require that - description to be expressed over the core contract rather than as a separate runner - contract. + description to be expressed over the family-neutral contract rather than as a separate + runner contract. +4. When a consumer receives bounded silence caused by missing semantic knowledge, the + detection core shall require the consumer to preserve the named coverage reasons. +5. When a family specification is created, the detection core shall require it to identify + its family contract, readiness fixtures, development population, and admission boundary. -### Requirement 12: Fixed measurement schema +### Requirement 15: Fixed measurement schema The detection core shall require one fixed measurement schema for every detection number. 1. When any detection result is reported, the detection core shall require population and - freeze digests, judgment vocabulary, every rate with its sample count and uncertainty - interval, coverage, instrument failures, and runtime. -2. When a measurement changes population or judgment vocabulary, the detection core shall - treat the result as a new baseline and shall not present it as a trend against the - previous baseline. + freeze digests, claimed family and ecosystem scope, denominator policy, family decision + vocabulary, neutral outcomes, evidence bases, every rate with its sample count and + uncertainty interval, coverage, instrument failures, model-call failures, and runtime. +2. When semantic relations participate in an arm, the detection core shall require the + report to include sites considered, sites recognized, transitions completed, security + objects constructed, constraints evaluated, determinate outcomes, and unresolved counts + by named reason. +3. When a measurement changes population, denominator policy, claimed scope, family + contract, fact schema, model program, or outcome mapping, the detection core shall treat + the result as a new baseline and shall not present it as a trend against the previous + baseline. +4. When a zero or clean-looking result is reported, the detection core shall require proof + that every tracked input was read and that every prerequisite semantic stage executed. +5. When a later-stage metric is reported, the detection core shall include the results and + digests of every prerequisite readiness stage. ## Out of Scope -- The pre-push hook transaction, rollout policy, and capability registry (pre-push-safety-net). -- New vulnerability families and their fact content (fact-pack specs). -- Fitted models of any kind in the core path. -- Engine and graph internals (contributor-scan). +- The pre-push hook transaction, rollout policy, and capability registry. +- Vulnerability-family algorithms, prompts, examples, and optimization programs. +- Framework- or repository-specific semantic fact content. +- Analysis or indexing of vendored dependency source. +- Proof execution and sandbox orchestration. +- Enabling a vulnerability family for advisory or blocking use. +- Analysis-engine, frontend, graph, and family-internal implementation details. diff --git a/.kiro/specs/detection-core/research.md b/.kiro/specs/detection-core/research.md index 46ec2eb..e0844e9 100644 --- a/.kiro/specs/detection-core/research.md +++ b/.kiro/specs/detection-core/research.md @@ -51,3 +51,475 @@ in `arm_regression.py`), and the admission workflow (`benchmarks/valid/prover_ad (dev 483 changes with seconds; pilot 264). Carry the 13.3/13.4 record sets as the fitting data — measurement only, never training a detector on them. - Reversed-pair probe harness for Req 8: fix/reversed-fix pairs exist in the pools. + +--- + +## Gap analysis: family-neutral evidence and model augmentation (2026-10-31) + +### Scope and approval state + +- **Feature type:** complex integration over an existing contract package. The core exists, + but its public schema, tests, producer adapter, and historical admission schema embody the + authorization family that supplied the original implementation. +- **Requirements state:** generated but not approved. This analysis may inform review; design + generation remains gated on explicit requirements approval. +- **Discovery method:** integration-focused source inspection. No external dependency research + was needed: the neutral core can remain Python 3.11 typed dataclasses and canonical JSON, + while DSPy remains the existing optional `hunt` dependency (`>=3.4,<3.5`) behind an adapter. +- **Instrument check:** representative requirements, core, authorization-adapter, model-runner, + and project-metadata inputs were opened with nonzero byte counts before interpreting gaps. + +### Current-state inventory + +The existing `src/openultrasast/core/` package is a compact contract layer with reusable +machinery: + +- `trace.py` supplies immutable records and deterministic JSON serialization. +- `verdict.py` supplies the authorization three-way judgment plus claim and bounded-silence + records. +- `coverage.py` supplies exact call-resolution buckets and zero-byte input refusal. +- `budget.py` supplies measured percentile budgets and a named budget gate. +- `provenance.py` supplies input-identity stamping and mismatch refusal. +- `factpack.py` supplies reviewed pack loading and content digests. +- `admission.py` supplies consume-only manifest validation and a fixed report schema. +- `probes.py` supplies reversed-pair and holdout/admission checks. +- `consumer.py` supplies a push request/verdict envelope. + +Current integration is narrow. `semantic/obligations/trace.py` converts the authorization +trace into the core `TraceRecord`; `hunt/prover.py` converts authorization coverage; +`mapping/entry_packs.py` consumes core fact packs; benchmark admission modules consume part +of the admission contract. The production push and proof paths do not yet consume the core +end to end. `benchmarks/valid/judge_pass.py` is the closest existing DSPy residual-judgment +adapter, but it reads authorization findings directly rather than a family-neutral evidence +record. + +### Requirement-to-asset map + +| Requirements | Existing assets | Gap classification and impact | +|---|---|---| +| 1–2 Evidence record and outcomes | Deterministic serialization, frozen dataclasses | **Missing / breaking:** `TraceRecord` requires authorization fields (`entry`, operation, resource, authority, guards) and `Judgment` is exactly unguarded/discharged/indeterminate. Missing family/security-object identity, analyzed scope, generic transitions and observations, family decision, neutral outcome, evidence basis, and bounded satisfaction. | +| 3 Family contract | Family strings exist in findings and packs | **Missing:** no versioned contract, ecosystem scope, security-object schema, accepted semantic roles, decision mapping, completeness policy, or two-family conformance fixtures. | +| 4 Typed semantics | Authorization hops/guards and recognition packs | **Missing:** no generic registration, invocation, call, return, transformation, operation-effect, or constraint-effect relation with conditions, fact identity, ambiguity handling, and per-relation completeness. | +| 5 Coverage and instrument | Zero-byte refusal, call buckets, authorization funnel | **Partial:** missing generic question → recognition → transition → security-object → constraint → outcome → finding counters and last-success/first-zero diagnosis. Existing coverage cannot distinguish all newly required failure layers. | +| 6 Budget and scope | `Complexity`, `Budget`, percentile derivation; change-scoped runners | **Constraint:** reusable budget logic is not bound to the neutral run, `calls` is not a generic transition count, and current runners may still pass a static 120-second value. Scope identity is absent from the evidence result. | +| 7 Verdict | `ProvenClaim`, `BoundedSilence`, consumer validation | **Partial / semantic mismatch:** no bounded-satisfaction record, evidence-basis tag, evidence-record identity, completed-question inventory, or multi-path aggregation. `ProvenClaim` currently implies proof-grade evidence for every claim. | +| 8 Readiness | Separate development and admission runners | **Missing:** no ordered readiness stages, prerequisite digests, staleness propagation, positive/negative/ambiguity fixture contract, or refusal of later gates after an earlier failure. | +| 9 Model augmentation | Existing DSPy program, cached calls, parsing/failure records, zero-shot judge experiment | **Missing / contradictory test:** no core handoff/result contract; a core test explicitly asserts that no model path is public. Model programs consume code/features or authorization findings, not canonical evidence records. The core must not import optional DSPy. | +| 10 Provenance | Six current digests and mismatch refusal | **Partial:** missing family-contract, semantic-pack/fact, model, program, prompt, and model-configuration identities. | +| 11 Fact packs | Review attestation, content digest, Express entry pack | **Partial / validation gap:** facts remain `dict[str, Any]`; the typed object lacks schema version, ecosystem/language scope, closed roles, structural match requirements, relation conditions, and conflict detection. The embedded review digest authenticates the review note, not the fact content itself. | +| 12 Admission | Consume-only manifests, pair scoring, Wilson intervals, GO/NO-GO reports | **Partial / authorization-shaped:** missing readiness prerequisites, ecosystem and denominator policy, family/model identities, arithmetic feasibility, in-scope scoring, neutral outcomes, semantic-stage funnel, and model-call results. | +| 13 Evaluation separation | Reversed-pair probe and coarse holdout/admission check | **Partial:** no lineage proving prompts, examples, facts, mappings, and promotion excluded admission labels. The current probe treats any differing score as a shortcut and is not a semantic audit by itself. | +| 14 Consumers | Push wrapper and authorization native conversion | **Partial:** the request is not used by `consume_verdict`; no proof-attachment interface, family readiness declaration, or production pre-push consumption exists. | +| 15 Measurement | Fixed report keys and benchmark metric helpers | **Partial / schema migration:** missing neutral outcomes, evidence bases, family/model identities, the generic semantic funnel, model failures, readiness digests, and enforceable baseline-reset metadata. Existing keys encode the legacy admission vocabulary. | + +### Compatibility and architecture constraints + +1. `TraceRecord` and `Judgment` are public exports and are pinned exactly by contract tests. + Replacing them in place breaks the authorization adapter immediately. +2. Historical admission reports are immutable evidence. A neutral schema must have a new + version and baseline; historical artifacts must not be rewritten to pass a new validator. +3. The authorization trace has no per-hop completeness and conservatively maps every hop to + incomplete. Migration must preserve that behavior until the family producer supplies + stronger evidence. +4. The core package must remain dependency-light. DSPy execution belongs in a hunt-side or + family-side adapter; the core owns only typed request, result, and provenance contracts. +5. Vendored source remains excluded. Framework behavior needed for a determinate decision + must arrive through reviewed semantic relations or remain indeterminate. +6. `core/admission.py` currently imports forbidden-label markers from `tools.fact_init`, an + upward dependency from a foundational contract into acquisition tooling. The design must + reverse or remove this dependency. +7. Fact-pack strengthening affects `mapping/entry_packs.py` and the shipped Express pack; + versioned loading or migration is required. +8. Core-unit success alone cannot establish integration because the production push and + proof consumers are not yet wired to the contract. + +### Implementation approach options + +#### Option A: Extend the existing types in place + +Add neutral fields and aliases while retaining authorization fields and judgment values. + +- **Strengths:** smallest initial patch; current producers and tests need fewer immediate + changes. +- **Risks:** creates dual sources of truth, permanently leaks authorization vocabulary into + every family, and cannot convincingly satisfy two-family neutrality. +- **Effort / risk:** L / High. + +#### Option B: Add an independent v2 core beside the current package + +Build a separate neutral package and migrate consumers later. + +- **Strengths:** clean schema and clear compatibility boundary. +- **Risks:** recreates the parallel-contract problem this spec exists to remove; downstream + specs can continue using v1 indefinitely; admission and measurement logic is duplicated. +- **Effort / risk:** XL / High. + +#### Option C: Hybrid versioned migration + +Add neutral typed primitives and family, readiness, and model contracts inside the existing +core boundary; retain reusable budget, provenance, and validation machinery; place explicit +authorization and DSPy adapters outside the neutral types; migrate consumers before removing +legacy exports. + +- **Strengths:** preserves measured machinery without preserving the wrong abstraction; + maintains a dependency-light core; provides a bounded migration path and a natural place + for two unlike reference-family conformance fixtures. +- **Risks:** requires deliberate schema versioning, temporary compatibility adapters, and a + new measurement baseline. +- **Effort / risk:** XL / Medium-High. +- **Design-phase recommendation:** prefer this option, subject to requirements approval and + resolution of the carried research items below. + +### Candidate responsibility and file boundaries + +The gap analysis identifies responsibilities, not final design choices: + +- `core/evidence.py` — family-neutral evidence record, typed transitions, observations, and + completeness. +- `core/family.py` — versioned family contract, declared schemas/roles, and decision mapping. +- `core/readiness.py` — stage results, prerequisites, invalidation, and promotion refusal. +- `core/model.py` — dependency-free model handoff/result and identities. +- `core/measurement.py` — neutral funnel and versioned measurement schema. +- Existing `verdict.py`, `coverage.py`, `factpack.py`, `provenance.py`, `admission.py`, + `consumer.py`, `probes.py`, `budget.py`, and `__init__.py` — extend reusable behavior and + phase out authorization-shaped exports. +- `semantic/obligations/trace.py` — authorization-family adapter rather than a core schema + definition. +- A hunt-side adapter — converts an evidence packet into the family DSPy input and converts + the model result into the core model-result contract without importing DSPy into core. +- Benchmark development/admission modules — produce readiness artifacts and score the neutral + schema while preserving historical report validation separately. +- Contract, migration, two-reference-family, model-handoff, readiness, and admission tests — + replace authorization-specific assumptions with explicit adapter conformance. + +### Design research carried forward + +1. Choose whether the two conformance families are synthetic authorization-plus-taint + fixtures or real adapters. They must prove different security-object shapes without + implying user-facing capability enablement. +2. Define the stable proof-attachment subset of demonstrated search results; proof execution + remains outside this spec. +3. Assign readiness-artifact persistence: the likely seam is that benchmark workflows + produce/store artifacts while the core validates prerequisites and staleness. +4. Define the wire migration and removal policy for legacy `TraceRecord`, `Judgment`, and + historical authorization report validation. +5. Pin the rule for model-corroborated outcomes: a model may answer only a family-declared + residual question. A missing relation declared necessary for the family outcome remains + indeterminate; model corroboration cannot fill it implicitly. +6. Decide whether budget workload-class assignment stays a runner responsibility or becomes + part of the neutral request contract; the core should validate, not infer, change scope. + +### Gap conclusion + +The existing package provides useful primitives but does not yet implement the amended +family-neutral contract. The dominant work is schema and lifecycle migration, not new graph +analysis. The authorization arm should remain a family adapter and regression source; it +must not define the new core vocabulary. Design generation should begin only after explicit +requirements approval. + +--- + +## Design discovery and synthesis (2026-10-31) + +### Summary + +- **Feature:** detection-core +- **Discovery scope:** complex integration and public-contract migration. +- **Key findings:** + - The reusable assets are canonical serialization, immutable records, budget derivation, + digest verification, manifest refusal, and metric helpers. Authorization field names and + outcomes are not reusable core concepts. + - The smallest viable architecture is a versioned ports-and-adapters contract: the core + owns typed evidence and validation; deterministic analyzers, DSPy, proof execution, + benchmarks, and push policy remain outside it. + - Readiness must be an immutable prerequisite graph validated by the core and persisted by + benchmark workflows. This makes a failed semantic stage stop later population gates. + - Migration must be finite: v2 is the only write contract after consumers migrate; v1 + remains a read-only historical decoder, not a second detection core. + +### Research log + +#### Existing integration and dependency direction + +- **Context:** the amended requirements generalize an already implemented authorization + contract without reopening the parked arm. +- **Sources consulted:** `src/openultrasast/core/`, authorization trace and prover adapters, + benchmark development/admission runners, push consumers, search verifier records, current + tests, and project steering. +- **Findings:** + - Core production imports are concentrated, so a staged adapter migration is feasible. + - `core/admission.py` currently imports acquisition-tool policy, which inverts the desired + dependency direction. + - Production push and proof paths are not yet end-to-end core consumers; unit conformance + alone cannot establish integration readiness. +- **Implications:** core modules import only the standard library and other core modules. + Analyzer, model, benchmark, proof, and push adapters import core, never the reverse. + +#### DSPy boundary + +- **Context:** the original product direction augments DSPy programs with deterministic + analysis evidence, while the core installation must remain independent of the optional + model stack. +- **Sources consulted:** existing `hunt/program.py` and `judge_pass.py`; official DSPy + documentation for signatures and modules. +- **Findings:** DSPy signatures declare structured input/output behavior, while modules may + contain learnable prompt or model parameters. A stable core handoff can therefore be a + typed request/result contract, but reproducibility requires program, prompt, model, and + configuration identities rather than a model name alone. +- **Implications:** DSPy execution, caching, parsing, and redaction remain in a hunt-side + adapter. The core receives a `ModelResult` tied to an immutable evidence digest and a + family-declared residual question. + +#### Canonical records and schema choice + +- **Context:** evidence records, facts, readiness receipts, and admission identities require + deterministic bytes and closed family extensions. +- **Sources consulted:** Python 3.11 `json` documentation, RFC 8785, JSON Schema Draft + 2020-12, and the existing core serializer. +- **Findings:** Python's sorted-key compact JSON is sufficient for repeatable records inside + the current Python boundary when non-finite numbers and unsupported objects are refused. + RFC 8785 is stricter and useful for cross-language signatures, which are not required here. + General JSON Schema validation would add a runtime dependency and permit a broader payload + language than the current families require. +- **Implications:** v2 defines a narrow canonical JSON profile and closed typed `FieldSpec` + values. Ordered transitions remain tuples; family payloads stay shallow and typed; complex + structure is represented by semantic transitions and observations. Cross-language signing + is a revalidation trigger rather than speculative scope. + +#### Proof attachment boundary + +- **Context:** execution-backed evidence may support blocking, but proof execution and + rollout policy are outside this spec. +- **Sources consulted:** `search/verify.py`, `search/coordinator.py`, search tests, and + `safety-net.md`. +- **Findings:** the verifier already produces a differential `demonstrated` outcome over + affected and safe sides. The core needs only the immutable result identity, matched family + and location, side digests, verifier revision, witness reference, and evidence-record + digest. +- **Implications:** the core validates and attaches a proof record; it does not run or + reinterpret the oracle. Execution-backed basis is necessary but does not by itself enable + downstream BLOCK policy. + +#### Readiness and admission ownership + +- **Context:** repository admission exposed semantic failures that should have stopped the + arm at recognition, transitions, or constraints. +- **Sources consulted:** development/admission runners, the 2026-10-31 handoff, and detection + quality steering. +- **Findings:** artifact production and storage already belong to benchmark workflows. + Contract validation, prerequisite freshness, denominator feasibility, pure scoring, and + report validation are reusable core responsibilities. +- **Implications:** benchmark workflows persist immutable `ReadinessResult` and case-result + artifacts. The core derives `passed`, `failed`, or `stale`, refuses later gates when + prerequisites do not pass, and never mutates historical artifacts. + +### Architecture pattern evaluation + +| Option | Description | Strengths | Risks and limitations | Decision | +|---|---|---|---|---| +| Extend v1 in place | Add neutral fields and aliases to authorization records | Small initial patch | Dual sources of truth and permanent family leakage | Rejected | +| Parallel v2 package | Build a second neutral package and migrate later | Clean initial schema | Recreates parallel contracts and duplicated admission | Rejected | +| Versioned ports and adapters | Add v2 neutral contracts to core, migrate through explicit adapters, retain v1 read-only decoding | Preserves reusable machinery and enforces boundaries | Requires controlled schema and baseline migration | Selected | + +### Design decisions + +#### Decision: split deterministic packet from decision + +- **Context:** model and proof adapters must not rewrite program-analysis evidence. +- **Selected approach:** `EvidencePacket` contains immutable deterministic semantics; + `Decision` contains family code, neutral outcome, basis, and bounded scope; + `EvidenceRecord` binds them by packet digest. +- **Rationale:** derived decisions can change while the underlying evidence identity remains + stable and auditable. +- **Trade-off:** one additional aggregate type, offset by a clear mutation boundary. + +#### Decision: family contracts are data, family reasoning stays outside core + +- **Context:** the core must validate family vocabulary without implementing every + vulnerability abstraction. +- **Selected approach:** an immutable `FamilyContract` declares ecosystems, shallow typed + fields, roles, decisions, mappings, required evidence, completeness rules, residual + questions, and named indeterminate reasons. Family adapters construct evidence and apply + their algorithms. +- **Rationale:** the core can refuse undeclared roles, fields, decisions, and scopes without + importing family logic. +- **Trade-off:** adding a new field kind changes the core schema and triggers revalidation; + this is preferred to unrestricted `Any` payloads. + +#### Decision: conformance uses two test-only semantic shapes + +- **Context:** authorization alone cannot demonstrate neutrality, and the parked arm must + not be revived through the core spec. +- **Selected approach:** test-only obligation and taint-flow contracts each supply positive, + negative, and missing-evidence fixtures. They make no user-facing capability claim. +- **Rationale:** the pair exercises control/constraint and flow/sanitization shapes without + target vocabulary or admission reuse. + +#### Decision: model corroboration answers declared residuals only + +- **Context:** Requirements 2.4 and 9.3–9.4 must not allow a model to fill missing required + deterministic relations. +- **Selected approach:** each residual question declares deterministic prerequisites and + allowed family decisions. A missing required prerequisite forces indeterminate. A valid + model result may produce model-corroborated advisory evidence but never execution-backed + basis. +- **Rationale:** this restores DSPy augmentation without treating plausible text as missing + static evidence. + +#### Decision: readiness persistence belongs to benchmarks + +- **Context:** core-owned files or storage would duplicate run orchestration. +- **Selected approach:** the core defines and validates receipts and freshness; benchmark + workflows write and load them. +- **Rationale:** separates pure contract logic from population custody and run persistence. + +#### Decision: versioned canonical JSON profile, not a new serializer dependency + +- **Context:** identical input must produce byte-identical records and stable digests. +- **Selected approach:** UTF-8 JSON with sorted keys, compact separators, preserved sequence + order, no non-finite numbers, and explicit enum values. The schema version is part of every + digestible aggregate. +- **Rationale:** uses the existing Python runtime and meets the current boundary without + claiming RFC 8785 interoperability. + +#### Decision: finite legacy migration + +- **Context:** historical v1 evidence must remain readable without allowing two live write + contracts. +- **Selected approach:** add v2, migrate authorization and benchmark adapters, stop v1 + writes, migrate downstream consumers, remove v1 public exports, and retain only a + read-only legacy decoder/validator. +- **Rationale:** preserves historical artifacts while restoring one canonical write path. + +### Risks and mitigations + +- **Permanent dual contracts:** milestone explicitly stops v1 writes and removes v1 exports + after an import census reaches zero. +- **Authorization vocabulary leaks into v2:** neutral-type review plus two unlike synthetic + family conformance suites. +- **Model fills semantic holes:** family-declared residual prerequisites and core-side result + validation. +- **Readiness becomes ceremonial:** later gates require current predecessor digests and + refuse stale, failed, or absent receipts. +- **Historical comparability is lost:** v1 stays immutable and v2 creates a new baseline + identity; cross-baseline trend comparison is refused. +- **Fact review attests prose but not content:** v2 review identity covers normalized facts, + schema, scope, and review evidence together. +- **Proof result silently enables blocking:** proof attachment changes evidence basis only; + rollout eligibility remains downstream. +- **Hidden training contamination:** the core validates declared lineage and population + digests; external review remains the trust boundary for undisclosed training data. + +### References + +- [DSPy signatures](https://dspy.ai/3.0.4/learn/programming/signatures/) — structured, + declarative model inputs and outputs. +- [DSPy modules](https://dspy.ai/3.0.4/learn/programming/modules/) — module composition and + learnable program state. +- [Python 3.11 JSON](https://docs.python.org/3.11/library/json.html) — sorted keys, compact + separators, and strict number handling. +- [RFC 8785](https://www.rfc-editor.org/rfc/rfc8785.html) — evaluated but not adopted for + the current single-runtime wire boundary. +- [JSON Schema Draft 2020-12](https://json-schema.org/draft/2020-12) — evaluated but not + adopted for the bounded family-field contract. + +--- + +## Review-gate contract closure (2026-10-31) + +### Scope of this pass + +- **Context:** the first merge draft covered all requirement IDs but several late validators + received only digests or caller-produced summaries, so their stated revalidation behavior + was not implementable from the interface. +- **Sources consulted:** the approved detection-core requirements, complete project steering, + the merge draft, current core/benchmark seams already recorded above, and independent + architecture, risk, and traceability reviews. +- **Decision:** keep the requirements and architecture boundary unchanged; repair the + interfaces that must carry canonical objects and fix the label-custody order before task + generation. + +### Decision: admission receives a closed validation context + +- `FamilyCatalog` is now a real immutable value with re-derived contract and catalog + digests and exact family/version/ecosystem resolution. +- `AdmissionValidationContext` carries the actual catalog, fact packs, model declarations, + lineages, and review/custody/provenance/proof policies. Authorization and scoring both + re-derive its digest and validate the objects; a set digest is not treated as the set. +- The threshold set is a baseline identity. Its actual lineage targets the threshold-set + digest and is validated before labels open. + +### Decision: excluded-dependency behavior is typed + +- A family role declares either product-or-reviewed-fact provenance or + excluded-dependency-fact-only provenance. +- A dependency-fact-only relation cannot be complete from product-code evidence. Missing + reviewed knowledge is an explicit incomplete reason; only a revalidated pack, fact, and + structural match can make it complete. +- This closes the vendored-source boundary without importing dependency source or family + algorithms into core. + +### Decision: lineage binds model, matcher, and threshold behavior + +- Lineage manifests name a subject; individual uses name the exact influenced artifact and + resolve to current custody receipts. +- Model requests embed a frozen program declaration and the actual lineage used for prompt, + examples, fitted components, and decision mappings. +- Fact matches bind matcher revision, matcher-behavior digest, and matcher lineage. + Admission-label influence on matching behavior spends the population just as prompt, + fact, threshold, or promotion influence does. + +### Decision: provenance validation must resolve bytes + +- Named provenance inputs use a closed role vocabulary and a manifest digest. +- Validation receives a resolver and trust policy and checks bytes, SHA-256, length, + storage receipt, retention policy, trusted store, and size bound before accepting the + stamp. Unreadable input cannot validate as a clean identity-only result. + +### Decision: spend before label release + +- Authorization returns a label-release request committed to the exact current custody + head and authorization digest. +- The benchmark custodian compare-and-swap appends `AVAILABLE` to `SPENT` durably and only + then releases label bytes. Scoring accepts that exact current-chain receipt and returns a + report bound to it; it does not request a later spend. +- This removes the crash and concurrency window in which labels could be observed while the + population still appeared available. + +### Decision: fitted-component authorization replays probes + +- Authorization receives probe plus observation pairs and calls the canonical evaluator. + It does not accept caller-produced `passed` summaries. +- Derived probe digests must equal the fitted-component evidence, alongside disjoint + repository holdout, fitting lineage, and the bound admission report. +- Contract, review policy, and custody policy are resolved from the same context bound into + the admission report. The authorization API does not accept caller-selected alternatives. + +### Decision: raw records share one validation composition seam + +- Admission and push consumption previously had unequal validation context: admission could + re-open facts and lineage while a consumer could still receive a raw `EvidenceRecord`. +- `validation.py` now owns `EvidenceValidationContext`, `EvidenceValidationInput`, and the + composition validator over catalog, named provenance inputs, facts, model derivation, + lineage, custody, and resolver trust. Both admission and consumer APIs call that seam; + there is no constructible validated-token shortcut. +- Per-case fact/model tuples were removed in favor of the single validation envelope, so + context membership and derivation replay have one source of truth. + +### Decision: admission review is independently attestable + +- The manifest and label-blind census each carry a payload digest and independence + attestation. Their reviewer cannot equal the producer and must satisfy a context-bound + review policy; the conflict disclosure is resolver-validated. +- A review flag or unattached digest is insufficient for criteria 12.1–12.2 and fails before + feasibility or label release. + +### Decision: lineage expectations bind the influenced artifact + +- A matching manifest subject is not enough because an embedded receipt could name a + different influenced artifact. +- Callers provide typed expectations containing purpose, artifact digest, and allowed use + counts. Generic lineage validation checks every current-chain receipt against those + expectations. Fact content forbids all labelled ancestry; model, threshold, matcher, and + fitted-component validation protects the current evaluation population while allowing + separately spent development populations. diff --git a/.kiro/specs/detection-core/spec.json b/.kiro/specs/detection-core/spec.json index c08b02b..ecdf6bf 100644 --- a/.kiro/specs/detection-core/spec.json +++ b/.kiro/specs/detection-core/spec.json @@ -1,7 +1,7 @@ { "feature_name": "detection-core", "created_at": "2026-10-30T13:55:30Z", - "updated_at": "2026-10-30T13:55:30Z", + "updated_at": "2026-10-31T10:59:35Z", "language": "en", "phase": "tasks-generated", "approvals": { diff --git a/.kiro/specs/detection-core/tasks.md b/.kiro/specs/detection-core/tasks.md index 1092ccb..7fc0680 100644 --- a/.kiro/specs/detection-core/tasks.md +++ b/.kiro/specs/detection-core/tasks.md @@ -1,162 +1,379 @@ # Implementation Plan -_Requirement IDs use N.M where N is the requirement number in requirements.md and M is its acceptance criterion._ +## Merge record + +| Previous v1 item | Disposition in this plan | +|---|---| +| Completed groups 1–7 and 8.1 | Retained as regression substrate; their former requirement mappings do not count toward v2 coverage | +| Previous task 8.2 | Permanently parked after admission NO-GO #3; no task below revives that arm | +| Previous tasks 8.3–8.4 | Superseded by v2 consumer and proof conformance work; no rollout or live capability enablement is included | ## Task constraints -- Complexity: McCabe at most 10 per touched function, at most 50 logical lines per function and 400 per new module. -- The core contract package owns contracts only; no analysis logic moves into it. -- Every migration step keeps existing behavior byte-identical; the 2026-10-30 admission suite is the regression gate. -- Instrument rule: prove nonzero input reads before interpreting any zero. -- Pool measures, never trains: no fitted component reads pool or admission labels. +- New and amended code keeps McCabe complexity at most 10, functions at most 50 logical lines, and new modules at most 400 logical lines. +- Core owns contracts and pure validation only; analyzers, model execution, proof execution, persistence, acquisition, and rollout remain with adjacent owners. +- Every zero-result test first proves nonzero readable input; an unread input is an instrument failure. +- Historical v1 artifacts remain byte-identical. V2 creates a new baseline and is never trended against v1. +- The parked authorization arm remains parked. Admission orchestration uses synthetic/reference inputs only and does not consume a live population. +- Existing target facts may be schema-migrated without expanding vocabulary or behavior. -- [x] 1. Foundation: core contract package skeleton -- [x] 1.1 (P) Canonical trace schema in the core package with deterministic serialization - - Define the trace record (entry, hops with call-target completeness, operation, witnesses, guards, completeness reasons, judgment, digests) as a frozen, family-agnostic contract. - - Done when a fixed trace serializes to byte-identical canonical output on repeat runs and a conformance test pins the schema. - - _Requirements: 1.1, 1.2_ - - _Boundary: core trace contract_ -- [x] 1.2 (P) Three-way verdict vocabulary and verdict contract - - Define unguarded/discharged/indeterminate, the proven-claim-with-witness vs bounded-silence verdict, and the rule that indeterminate never converts to clean. - - Done when unit tests cover all three judgments, the conversion refusal, and zero-shot-judge-over-trace as the only model role on verdicts. - - _Requirements: 2.1, 2.2, 5.1, 5.2, 9.1_ - - _Boundary: core verdict contract_ -- [x] 1.3 (P) Coverage accounting schema with instrument refusal - - Define entry-recognition coverage, exactly-one-bucket-per-call resolution outcomes, named unchecked reasons, and zero-byte/unreadable input as an instrument failure at core level. - - Done when a zero-byte read fails the run in a unit test and the bucket exclusivity is pinned. - - _Requirements: 3.1, 3.2_ - - _Boundary: core coverage contract_ -- [x] 1.4 (P) Provenance digest stamping with mismatch refusal - - Define the analyzer/fact/acquisition/configuration/population/input digest stamping and refusal when any digest cannot be re-derived. - - Done when a mismatched digest is refused in a unit test. - - _Requirements: 6.1, 6.2_ - - _Boundary: core provenance contract_ +- [ ] 1. Establish migration-safe foundations +- [ ] 1.1 Freeze the v1 compatibility boundary and enumerate active readers and writers + - Pin representative v1 trace and report bytes as immutable regression fixtures. + - Record every current import and artifact-writing path that still depends on the authorization-shaped contract. + - Preserve existing v1 behavior as migration input without treating it as v2 conformance. + - Done when fixture tests detect any byte change and the import/writer census identifies every cutover target. + - _Requirements: 14.3, 15.3_ +- [ ] 1.2 Build the strict canonical codec and resolver-backed artifact boundary + - Produce deterministic UTF-8 JSON with full SHA-256 identities, stable mapping order, and retained sequence order. + - Refuse non-finite numbers, unsupported values, unknown schemas, duplicate logical identities, and malformed digests. + - Validate resolved bytes, length, storage receipt, retention identity, trusted store, and configured size limits. + - Provide deterministic in-memory resolver and receipt fixtures for later contract tests. + - Done when identical values produce byte-identical output and each content or receipt mismatch fails before parsing. + - _Requirements: 1.4, 10.4_ -- [x] 2. Measured budget replaces the static gate -- [x] 2.1 Record per-change complexity and reply time - - Extend runner records with complexity features (changed regions, call count, changed bytes) alongside the already-recorded seconds. - - Done when dev-pool records carry all complexity fields and seconds. - - _Requirements: 4.1, 12.1_ - - _Boundary: runner records_ - - _Depends: 1.3_ -- [x] 2.2 Derive per-workload-class budgets from measurement - - Fit complexity to reply time on the dev (483-change) and pilot (264-change) record sets as measurement only; publish per-class budgets with n, percentile, and hardware; make the budget configurable per class, demoting the static constant to provisional default. - - Done when the derivation table (class, n, percentile, hardware, budget) is published under the fixed measurement schema and a new-baseline note records the change. - - _Requirements: 4.2, 12.1, 12.2_ - - _Boundary: core budget contract_ - - _Depends: 2.1_ -- [x] 2.3 Budget gate and scoping record - - Gate failures name the change, elapsed time, and the derivation exceeded; the analyzed scope (change regions) is recorded per run. - - Done when an over-budget fixture fails with all three fields and the scope is present in the record. - - _Requirements: 4.3, 4.4_ - - _Boundary: core budget contract_ - - _Depends: 2.2_ - -- [x] 3. Prover arm adopts the core contract -- [x] 3.1 Prover trace implements the core schema - - Re-export/implement the core trace contract in the access_control arm without behavior change. - - Done when a conformance test proves the prover's trace satisfies the core schema field-for-field. - - _Requirements: 1.1, 1.2, 2.1_ - - _Boundary: prover arm_ - - _Depends: 1.1, 1.2_ -- [x] 3.2 Prover verdict and coverage over the core contract - - Emit verdicts and coverage through the core contracts; existing strict-lane behavior stays byte-identical. - - Done when the keyword regression and prover suites pass unchanged after adoption. - - _Requirements: 3.1, 5.1, 5.2_ - - _Boundary: prover arm_ - - _Depends: 3.1, 1.3_ - -- [x] 4. Generalized admission contract -- [x] 4.1 (P) Consume-only manifest validation generalized into the core - - Move the ordered refusal chain (malformed, producer, unreviewed, label-isolated, label-exposure, acquisition-changed, spent) into the core admission contract; the benchmark validator becomes a thin wrapper. - - Done when all refusal branches pass unit tests through the core contract. - - _Requirements: 7.1, 7.2_ - - _Boundary: core admission contract_ - - _Depends: 1.4_ -- [x] 4.2 Paired scoring and GO/NO-GO report schema with family scope - - Generalize paired vulnerable/fixed scoring, ordinary false alarms, gates, and the report schema into the core; add the claimed-families field. - - Done when the report schema includes family scope and every rate carries n and Wilson 95. - - _Requirements: 7.3, 10.2, 12.1_ - - _Boundary: core admission contract_ - - _Depends: 1.2, 4.1_ -- [x] 4.3 Admission suite reproduces the 2026-10-30 NO-GO - - Re-run the pilot admission through the core contract and reproduce 0/12 pairs, 0.83 flagged/100, 0 instrument failures, NO-GO exactly. - - Done when the reproduced report matches the committed 2026-10-30 report field-for-field. - - _Requirements: 7.3, 12.1_ - - _Boundary: admission measurement_ - - _Depends: 4.2, 3.2_ - -- [x] 5. Anti-shortcut probes for any fitted component -- [x] 5.1 (P) Reversed-pair probe harness - - Build the fix/reversed-fix probe; the deletion-count shape feature must fail it (control test). - - Done when the control probe fails a shape-only component and passes a relation-based one. - - _Requirements: 8.1, 8.2_ - - _Boundary: core probes_ +- [ ] 2. Implement canonical semantic, provenance, and evidence contracts +- [ ] 2.1 (P) Represent family-neutral semantic relations and observations + - Cover registration, invocation, call, return, value transformation, operation effect, and constraint effect. + - Preserve role, location, conditions, provenance, and independent complete, incomplete, or unresolved status. + - Refuse conflicting complete candidates by producing explicit ambiguity rather than selecting one. + - Done when contract tests exercise every relation kind and all three relation statuses with canonical identities. + - _Requirements: 1.2, 4.1, 4.2, 4.3, 5.2_ + - _Boundary: Semantic Evidence_ - _Depends: 1.2_ -- [x] 5.2 Fitted-component gate - - Require per-repository holdout and untouched-population admission before any fitted component makes a claim; BLOCK reserved for demonstrated proofs. - - Done when a fitted component without admission evidence is refused by the gate in a test. - - _Requirements: 8.2, 9.2_ - - _Boundary: core probes + admission_ - - _Depends: 4.1, 5.1_ +- [ ] 2.2 (P) Validate complete input and decision provenance + - Stamp analyzer, family contract, fact set, acquisition, configuration, population, and change-input identities. + - Require the closed named-input role set and resolve every referenced input through the trusted artifact boundary. + - Keep model and proof identities outside the immutable deterministic packet. + - Done when every stamp field has a tamper test and unreadable or substituted named input is refused. + - _Requirements: 10.1, 10.4_ + - _Boundary: Provenance and Lineage_ + - _Depends: 1.2_ +- [ ] 2.3 Enforce append-only population custody + - Construct and validate contiguous reviewed-custodian receipt chains against the complete current log. + - Require compare-and-swap requests to name the expected log head and derive the next sequence and predecessor. + - Make spent status monotonic and reject old prefixes, omitted receipts, broken chains, and later availability claims. + - Provide deterministic in-memory custody-port and custodian fixtures for lineage and admission tests. + - Done when concurrent requests have one valid successor and no fixture can restore a spent population. + - _Requirements: 12.2, 13.1, 13.4_ +- [ ] 2.4 Bind lineage purposes to exact influenced artifacts + - Require each lineage use to name its purpose, influenced artifact, source population, label access, and current-chain receipt. + - Enforce required and optional use counts for prompts, examples, thresholds, fitted models, facts, matching behavior, mappings, and promotion. + - Distinguish fact ancestry that forbids all label access from development lineage that may use a different spent population. + - Refuse protected-population influence, redirected artifact identities, wrong purposes, and incomplete receipt history. + - Done when matching subjects cannot mask a wrong influenced artifact and contaminated populations are deterministically refused. + - _Requirements: 13.1, 13.3, 13.4_ +- [ ] 2.5 Bind immutable evidence packets to neutral decisions + - Preserve family identity, security object, analyzed scope, ordered semantics, observations, family fields, completeness, and provenance. + - Keep family decision codes separate from violation, satisfied, and indeterminate outcomes. + - Require bounded scope for satisfaction, named reasons for indeterminate outcomes, and a valid deterministic, model-corroborated, or execution-backed basis. + - Prevent incomplete or ambiguous required relations from producing violation or satisfaction. + - Done when positive, satisfied, and incomplete records round-trip canonically and every invalid packet/decision combination is refused. + - _Requirements: 1.1, 1.2, 1.3, 2.1, 2.2, 2.3, 2.4, 2.5, 10.1_ -- [x] 6. Fact packs and consumer interfaces -- [x] 6.1 Fact-pack admission contract - - Ruleset loader admits family fact packs only with review evidence and provenance stamping; packs carry their family tag. - - Done when an unreviewed pack is refused and a reviewed pack stamps provenance in a test. - - _Requirements: 10.1_ - - _Boundary: fact-pack contract_ - - _Depends: 1.4_ -- [x] 6.2 Consumer interfaces for push and proof - - Define the interfaces through which the pre-push transaction consumes traces/verdicts within its deadline and the proof layer attaches demonstrated evidence to a verdict; existing arms express behavior over the core contract. - - Done when both interfaces are exercised by conformance tests from a mock consumer. - - _Requirements: 9.2, 11.1, 11.2, 11.3_ - - _Boundary: core consumer contracts_ - - _Depends: 1.2, 3.2_ -- [x] 6.3 Runners adopt core coverage, budget, provenance incrementally - - Migrate development_runner, arm_regression, prover_arm, run_manifests onto the core contracts one at a time, re-running the regression suite after each. - - Done when all runners consume the core contracts and the keyword regression passes at the pre-migration count plus new tests. - - _Requirements: 3.1, 4.1, 4.3, 6.1, 11.3, 12.1_ - - _Boundary: benchmark runners_ - - _Depends: 2.3, 3.2, 6.2_ +- [ ] 3. Add vulnerability-family contracts and unlike reference families +- [ ] 3.1 Validate a closed vulnerability-family declaration + - Require family/schema versions, ecosystems, security-object fields, roles, decisions, outcome rules, completeness conditions, residual questions, and indeterminate reasons. + - Enforce typed family fields and exact ecosystem/schema scope. + - Declare whether each role accepts product evidence or requires reviewed excluded-dependency facts. + - Refuse undeclared fields, roles, decisions, reasons, or determinate outcomes without their minimum evidence. + - Done when the family registration matrix accepts only declared combinations and reports unsupported scope without cross-family fallback. + - _Requirements: 1.3, 2.1, 2.4, 3.1, 3.2, 3.3, 3.4, 4.4, 9.1_ +- [ ] 3.2 Build exact family catalog resolution and capability declarations + - Re-derive every contract and catalog digest and require canonical ordering and unique family/version/schema identities. + - Resolve a contract only when family, digest, and ecosystem identify exactly one entry. + - Validate that each family capability names its contract, readiness fixtures, development population, and admission boundary. + - Done when duplicate/substituted catalog entries and each missing or corrupted capability field are refused. + - _Requirements: 3.1, 3.3, 3.4, 14.5_ +- [ ] 3.3 Create obligation and taint-flow reference conformance fixtures + - Model an operation-plus-constraint family and an ordered source-to-sink family without target vocabulary. + - Provide positive, negative, and ambiguity or missing-evidence cases for both semantic shapes. + - Require each family to produce violation, bounded satisfaction, and indeterminate output through the same validators. + - Done when both fixture suites pass without adding a family-specific field to any shared core contract. + - _Requirements: 3.5, 8.2_ -- [x] 7. Validation -- [x] 7.1 Fixed measurement schema enforced in reports - - Every report carries population/freeze digests, judgment vocabulary, rates with n and Wilson 95, coverage, instrument failures, runtime; vocabulary/population change starts a new baseline. - - Done when a schema-validation test rejects a report missing any field and accepts the 2026-10-30 report. +- [ ] 4. Expose operational coverage, budget, and readiness gates +- [ ] 4.1 (P) Account for instrument reads and every semantic-stage outcome + - Count reads, questions, sites considered and recognized, transitions, security objects, constraints, determinate outcomes, and findings. + - Put every considered relation in exactly one complete, incomplete, or unresolved bucket. + - Fail unopened or zero-byte tracked input and diagnose the last verified nonzero stage and first missing expected output. + - Done when stage-by-stage fixtures distinguish a valid zero from unread input and identify every injected first failure. + - _Requirements: 5.1, 5.2, 5.3, 5.4, 5.5, 15.2, 15.4_ + - _Boundary: Coverage Funnel_ + - _Depends: 2.1, 2.5_ +- [ ] 4.2 (P) Derive and enforce measured workload budgets and change scope + - Record changed regions, semantic-transition count, changed bytes, and total reply time. + - Derive workload-class budgets with sample count, percentile, reference workload, reference hardware, and canonical identity. + - Resolve the changed-byte measurement and refuse missing, zero, non-finite, or incompatible complexity data. + - Reject evidence and claim anchors outside the supplied affected scope and name the exceeded derivation on budget failure. + - Done when class derivation is deterministic and scope escape or elapsed-budget excess yields a named gate failure. + - _Requirements: 6.1, 6.2, 6.3, 6.4_ + - _Boundary: Budget and Scope_ + - _Depends: 1.2, 2.5_ +- [ ] 4.3 Record ordered readiness evidence with explicit evidence-use roles + - Cover instrument, recognition, transitions, constraints, pairs, development, admission, and consumer integration in fixed order. + - Bind each result to arm/family/ecosystem, analysis identity, population use, fixtures, predecessors, measurement, and correction lever. + - Keep regression, development, repository-holdout, fresh-admission, and integration evidence uses non-interchangeable. + - Done when each stage accepts the correct positive/negative/ambiguity triplet and rejects the wrong evidence-use role. + - _Requirements: 8.1, 8.2, 8.6_ +- [ ] 4.4 Stop promotion on failed, absent, or stale prerequisites + - Authorize a stage only when every ordered predecessor is current and passed. + - Derive transitive staleness from core schema, semantic vocabulary, family, facts, acquisition, configuration, model, and outcome mapping. + - Return the first failing layer and named correction lever without mutating historical results. + - Done when one changed identity makes all dependent later results stale and later population/integration gates refuse to run. + - _Requirements: 8.3, 8.4, 8.5, 15.5_ + +- [ ] 5. Replace semantic extension points with reviewed fact packs +- [ ] 5.1 Validate versioned packs and independent review evidence + - Bind schema, family, ecosystem, language, stable facts, declared roles, normalized content, and review evidence. + - Accept source, official-documentation, or independently reviewed behavioral evidence under a trusted review policy. + - Resolve conflict disclosures and require producer/reviewer independence where behavioral evidence is used. + - Reject label-derived semantic-fact ancestry, unknown scope/roles, missing review, and inconsistent digests. + - Done when the schema/scope/review matrix accepts only fully reviewed label-isolated packs. + - _Requirements: 11.1, 11.4, 13.3_ +- [ ] 5.2 Validate complete structural observations and matcher lineage + - Match the declared requirement set, including qualified name, shape, receiver, target, argument, and configuration evidence as applicable. + - Resolve every observation artifact and require its identifier set to equal the fact’s complete structural requirements. + - Bind matcher revision, behavior digest, and current lineage to the exact evaluated product location. + - Prevent name-only, partial, fabricated, ambiguous, or protected-population-contaminated matches from becoming complete. + - Done when only a fully observed structural match with valid matcher lineage is eligible for effect application. + - _Requirements: 10.2, 11.2, 11.3, 13.4_ +- [ ] 5.3 Apply declared fact effects with reproducible provenance + - Materialize only relation kinds, roles, source/target roles, conditions, and effects declared by the matched fact and family. + - Attach exact pack, fact, product location, structural observation, matcher, and match-reference identities to every derived relation. + - Preserve unmet conditions as incomplete evidence and conflicting eligible matches as unresolved ambiguity. + - Mark label-influenced matching behavior or promotion as spent before the relation can participate in evaluation. + - Done when valid matches yield canonical effects while partial and ambiguous cases remain non-determinate with named reasons. + - _Requirements: 4.1, 4.2, 4.3, 10.2, 11.2, 11.3, 11.5, 13.4_ +- [ ] 5.4 Revalidate fact-derived relations and enforce the dependency boundary + - Re-open referenced packs, facts, match evidence, review policy, lineage, custody, and resolved bytes for every fact-derived relation. + - Reject undeclared roles, invalid scope, inconsistent content, and fact provenance that cannot be reproduced. + - Require reviewed semantic facts for determinate behavior attributed to excluded dependencies. + - Preserve missing dependency knowledge as named incomplete evidence rather than product-derived safety. + - Done when product-only dependency behavior is refused, a missing fact stays indeterminate, and the reviewed structurally matched equivalent is accepted. + - _Requirements: 2.4, 4.4, 10.4, 11.4, 11.5_ + +- [ ] 6. Add bounded model handoff and semantic shortcut probes +- [ ] 6.1 (P) Preserve reproducible model requests, results, and responses + - Present the canonical evidence, completeness snapshot, family declaration, and declared residual question without importing a model runtime. + - Freeze provider, model, program, prompt, examples, fitted component, configuration, adapter, decision mapping, and input-bundle identities before execution. + - Preserve normalized responses plus one bounded redacted inline value or trusted immutable artifact, including failures. + - Done when matching success/failure fixtures round-trip and substituted request, packet, response, program, or mapping identities are refused. + - _Requirements: 9.1, 9.2, 10.3_ + - _Boundary: Model Handoff_ + - _Depends: 2.4, 3.1_ +- [ ] 6.2 Apply model results without rewriting deterministic evidence + - Validate current prompt/example/fitted-model/mapping lineage against the evaluated population and declared artifacts. + - Reject proposer/adjudicator role collision and results for another packet, family, residual, or mapping. + - Allow model-corroborated output only when deterministic residual prerequisites are complete. + - Prevent model output from creating semantic relations, satisfaction over missing evidence, or execution-backed basis. + - Done when accepted derivation leaves the evidence packet byte-identical and every incomplete or contaminated path remains indeterminate. + - _Requirements: 2.4, 2.5, 9.3, 9.4, 9.5, 13.1_ +- [ ] 6.3 (P) Evaluate independently reviewed equal-semantics reversed-surface probes + - Derive semantic projections from records and family contracts rather than trusting caller summaries. + - Resolve distinct surface artifacts and independently reviewed semantic-equivalence evidence. + - Reject unequal semantics, digest-only review, non-finite tolerances, and fabricated passed summaries. + - Detect a surface-shape component while accepting a relation-based component within its pre-registered tolerance. + - Done when probe results are reproducible solely from canonical probe inputs and review policy. + - _Requirements: 13.2_ + - _Boundary: Probes_ + - _Depends: 3.3, 5.1_ + +- [ ] 7. Compose validated evidence into bounded decisions and consumer output +- [ ] 7.1 Revalidate deterministic and fact-derived records through one shared context + - Bind the exact catalog, packs, model declarations, lineage set, review policies, provenance trust, and custody policy. + - Resolve the family contract and all seven named provenance inputs for each raw record. + - Re-run family and fact validation and reject embedded pack, lineage, or named-input substitutes. + - Done when raw, forged, and context-mismatched records cannot reach admission or consumer decision code. + - _Requirements: 4.4, 10.1, 10.2, 10.4, 12.6, 14.1_ +- [ ] 7.2 Replay model derivation at the shared validation seam + - Require deterministic records to omit model derivation and model-corroborated records to include both request and result. + - Re-run model application from the pre-model record and compare the derived record canonically. + - Refuse constructible “validated” markers or model programs outside the context. + - Done when admission and consumer tests exercise the same validator and neither can bypass it with a raw record. + - _Requirements: 9.2, 9.3, 9.4, 9.5, 10.3, 10.4, 12.6, 14.1_ +- [ ] 7.3 (P) Derive defect claims, bounded satisfaction, and bounded silence + - Require exact change-tied locations and complete evidence for defect claims. + - Carry completed question identities and exact scope for satisfaction. + - Preserve named coverage reasons and record identities for every undecided question. + - Keep incomplete paths silent unless a separate complete path independently supports a claim. + - Done when mixed complete/incomplete fixtures produce supported claims plus bounded silence without any unbounded clean result. + - _Requirements: 2.2, 2.3, 7.1, 7.2, 7.3, 7.4, 14.4_ + - _Boundary: Verdict and Proof_ + - _Depends: 2.5, 3.1_ +- [ ] 7.4 Validate differential proof attachment without enabling rollout + - Re-derive reviewed producer, sandbox, oracle, affected/safe side, control, witness, and verification identities. + - Resolve all proof artifacts under the supplied producer, reviewer, store, and byte-limit trust policy. + - Upgrade basis only for demonstrated affected-only behavior tied to the same family, packet, location, and change/control. + - Reject opaque, self-asserted, inconclusive, same-side, or mismatched proof records. + - Done when only the valid differential fixture yields execution-backed basis and no attachment changes rollout policy. + - _Requirements: 2.5, 9.6, 14.2_ +- [ ] 7.5 Integrate validated records, proof, scope, and budget at the core consumer edge + - Accept raw validation envelopes and call the shared validator before creating any output. + - Preserve every bounded-silence reason and reject records or anchors outside the supplied change scope. + - Apply optional proof attachments through the proof validator and include all processing in total elapsed time. + - Return defects, satisfactions, silence, input identities, complexity, and measured budget identity. + - Done when a synthetic push returns a bounded verdict and scope, trust, or deadline violations fail explicitly. + - _Requirements: 6.4, 14.1, 14.4_ + - _Depends: 4.2, 7.2, 7.4_ + +- [ ] 8. Implement fixed measurement and leakage-safe admission +- [ ] 8.1 (P) Define comparable baseline identities and fixed metric algebra + - Bind arm, population/freeze, claimed scopes, exact contract/fact/model sets, acquisition, analyzer, configuration, denominator, threshold set, and outcome mapping. + - Define closed numerator/denominator rules for pair correctness, recall, fixed and ordinary false alarms, precision, semantic coverage, and model success. + - Require Wilson 95 intervals with numerator and denominator; serialize zero denominators as null and prevent them from passing. + - Treat any identity mutation as a new baseline and refuse cross-arm or cross-scope aggregation. + - Done when a baseline mutation matrix and two-arm isolation fixture distinguish every non-comparable result. + - _Requirements: 11.6, 15.1, 15.3_ + - _Boundary: Measurement_ + - _Depends: 4.1, 4.4_ +- [ ] 8.2 Build complete measurement records and typed repository-holdout reports + - Record family decision vocabulary, neutral outcomes, evidence bases, fixed metrics, semantic funnel, model outcomes/failures, instrument failures, runtime, and readiness ancestry. + - Require every later-stage record to carry current prerequisite readiness stage results and digests. + - Define a typed repository-holdout report with an explicit evidence-use role, training and holdout manifest references, repository identities, fitting lineage, and distinct measurement/report identities. + - Refuse zero or clean-looking results unless tracked input reads and prerequisite semantic stages passed. + - Done when missing metrics, counts, failure fields, readiness ancestry, or holdout-report identity fields invalidate the corresponding record. + - _Requirements: 13.2, 15.1, 15.2, 15.4, 15.5_ +- [ ] 8.3 Validate independently reviewed census and manifest payloads + - Canonically bind every substantive census and manifest field into separate review-subject digests. + - Require producer/reviewer separation, admitted reviewer identity, allowed identity pairing, and resolver-validated conflict disclosure. + - Reject self-review, copied attestations, unknown reviewers, malformed review digests, and unreadable disclosures. + - Preserve aggregate census cardinalities without exposing case identities or per-case labels. + - Done when only independently attested label-blind payloads can proceed to manifest validation. - _Requirements: 12.1, 12.2_ - - _Boundary: measurement schema_ - - _Depends: 4.2, 6.3_ -- [x] 7.2 End-to-end: one push through core to verdict to admission report - - Run one dev-pool push end-to-end: scope, recognize, resolve, trace, verdict, digests, admission report on the dev pool. - - Done when the E2E test produces a verdict with witness or bounded silence plus a complete report. - - _Requirements: 5.1, 5.2, 7.3, 11.1_ - - _Boundary: end-to-end integration_ - - _Depends: 6.3, 7.1_ +- [ ] 8.4 Validate exact admission context, readiness, scope, sets, and threshold lineage + - Bind the exact evidence-validation context, reviewed census, denominator, thresholds, readiness results, untouched population, and current custody head. + - Re-derive claimed scopes and exact contract, fact, model, threshold, and outcome-mapping sets from canonical objects. + - Require current instrument-through-development readiness with matching analysis identity and evidence-use roles. + - Validate prompt, example, fitted-model, fact, matcher, mapping, promotion, and threshold lineage before labels open. + - Reject label exposure, acquisition drift, context substitution, stale readiness, scope mismatch, or prior spend. + - Done when each refusal branch fails before feasibility and no digest-only set or lineage claim is accepted. + - _Requirements: 12.1, 12.2, 13.1_ +- [ ] 8.5 Refuse arithmetically impossible threshold configurations + - Enumerate reachable counts from label-blind pair, case, semantic-site, and model-call capacities. + - Evaluate count, rate, per-100, and Wilson-bound thresholds with the fixed comparator direction for each metric. + - Require every grouped threshold to have at least one jointly attainable count assignment. + - Done when the historical impossible-denominator case is a configuration refusal rather than a measured no-go. + - _Requirements: 12.3, 12.4_ +- [ ] 8.6 Validate the pre-label population-spend protocol inside core + - Construct a label-release request bound to the exact validated authorization and current custody-log head. + - Validate the returned receipt as the exact current-chain direct successor that changes the named population from available to spent. + - Reject missing, stale, mismatched, non-successor, or availability-restoring receipts before scoring authorization. + - Keep durable append, compare-and-swap execution, label release, persistence, crash handling, and concurrency outside core. + - Done when only the exact spent successor authorizes scoring and every other receipt shape is refused without an external side effect. + - _Requirements: 12.2, 13.4_ +- [ ] 8.7 Derive scoped admission metrics from canonical case evidence + - Re-run shared evidence validation, instrument, budget, verdict, and proof validation for every case. + - Match case roles and pairs back to the reviewed census and derive decision vocabulary and counts from the exact catalog. + - Score quality only over pre-registered family/ecosystem scopes while retaining whole-population coverage. + - Compute paired behavior, ordinary false alarms, semantic coverage, model outcomes, and all standard metrics without trusting runner aggregates. + - Done when forged decisions, substituted contracts, invalid fact review, contaminated lineage, or case-count drift fail before metric production. + - _Requirements: 12.5, 12.6, 15.1, 15.2_ +- [ ] 8.8 Assemble evidence-backed gates and the final admission report + - Emit instrument, budget, readiness, and quality gates with reasons and supporting identities. + - Bind the report to the validation context, authorization, pre-label spend receipt, census, and complete case-result set. + - Require every mandatory metric and gate, plus model success when applicable, before allowing GO. + - Keep model-corroborated, deterministic, and execution-backed counts separate. + - Done when any missing/failed mandatory gate forces NO-GO and a complete passing synthetic fixture yields a reproducible GO report. + - _Requirements: 12.6, 15.1, 15.2, 15.4, 15.5_ +- [ ] 8.9 Authorize fitted components only from replayed probes and disjoint evaluation + - Resolve the component’s unique model declaration, family contract, review policy, and custody policy from the admission-bound context. + - Re-evaluate every canonical probe input and require the derived result set to match the component evidence. + - Consume the typed repository-holdout report from task 8.2 and validate its evidence-use role, training/holdout references, report identity, and fitting lineage. + - Prove the report’s training and repository-held-out repository sets are disjoint. + - Require an admission GO that was untouched before its bound label-release spend. + - Done when fabricated summaries, caller-selected policy substitutes, overlapping repositories, contaminated lineage, or mismatched admission are refused. + - _Requirements: 13.1, 13.2, 13.4_ -- [ ] 8. Integration with existing specs (binding phase, roadmap 2026-10-30) -- [x] 8.1 Prover native conformance to the core trace contract - - access-control-prover's Trace implements core TraceRecord natively; hunt/core_adapter.py is removed once conformance tests pass; behavior byte-identical (admission suite regression gate). - - Done when test_core_adapter.py equivalents pass against native types and the adapter module is deleted. - - _Requirements: 1.1, 1.2, 11.3_ - - _Boundary: prover arm_ - - _Depends: 3.1_ -- [ ] 8.2 Fact-pack workstream from the recognition-coverage table - - valid-detection-benchmark task 7.1's manifest-derived table becomes reviewed fact packs admitted via core/factpack.py, family-tagged and digest-stamped; measured on pool-train, admission-gated on the untouched pilot. - - Done when one new ecosystem pack passes admission gates and appears with its family in the report. - - _Requirements: 10.1, 10.2, 7.3_ - - _Boundary: fact-pack contract + benchmark_ - - _Depends: 6.1, 4.2_ - - _Parked 2026-10-31: final gated round NO-GO #3 (rule 10 stop condition, maintainer strategy-review); done-when unreachable for this arm -- the unique-declaration member-call binding increment ships (dc28325, dev complete-share 11.9->19.2 with recall/FA unchanged) but the admission gap is entry recognition (4/12 repos unrecognised; coverage 8/12) and framework operation facts, not call resolution. See benchmarks/measurements/2026-10-31-access-control-admission-unique-binding/_ -- [ ] 8.3 Pre-push hook consumes the core consumer contract - - pre-push-safety-net builds PushRequest and consumes PushVerdict via core/consumer.py; budget from the measured per-class derivation; one real dev-pool push E2E. - - Done when an E2E hook run on a dev-pool repo emits a PushVerdict with witnesses/reasons and a measured-class budget. - - _Requirements: 4.2, 11.1, 11.3_ - - _Boundary: pre-push transaction_ - - _Depends: 6.2, 2.2_ -- [ ] 8.4 Proof layer attaches demonstrated evidence to verdicts - - search-with-proof attaches proof artifacts to PushVerdict claims; BLOCK only on proven claims; bounded-silence entries stay advisory. - - Done when a demonstrated-finding fixture blocks and an unproven claim stays advisory in a test. - - _Requirements: 9.2, 11.2_ - - _Boundary: proof layer_ - - _Depends: 6.2_ +- [ ] 9. Integrate adjacent producers and benchmark-owned orchestration +- [ ] 9.1 (P) Translate the parked authorization trace conservatively into v2 evidence + - Move conversion out of the family trace domain and keep authorization vocabulary outside shared core modules. + - Map missing per-hop knowledge to incomplete relations and indeterminate outcomes without behavior lifting. + - Preserve existing parked-arm regression behavior and avoid any admission rerun or reactivation. + - Done when adapter conformance matches the frozen v1 behavior while the authorization domain no longer imports v2 core types directly. + - _Requirements: 3.5, 14.3_ + - _Boundary: Authorization Adapter_ + - _Depends: 3.3, 7.2_ +- [ ] 9.2 (P) Translate hunt-owned model execution through the bounded handoff + - Convert structured executor inputs and outputs to the canonical request/result contracts. + - Keep prompt creation, provider calls, caching, parsing, and redaction in the hunt subsystem. + - Reject mismatched declared results through core validation without importing DSPy into core. + - Done when a fake executor success/failure pair crosses the adapter and core’s import census contains no model-runtime dependency. + - _Requirements: 9.1, 9.2, 9.3, 9.4, 9.5, 14.3_ + - _Boundary: Model Adapter_ + - _Depends: 6.2, 7.2_ +- [ ] 9.3 (P) Translate demonstrated search results into trusted proof attachments + - Map existing typed affected/safe verification records without moving sandbox or oracle execution into core. + - Preserve producer, control, witness, storage, and verification identities required by proof validation. + - Refuse inconclusive and mismatched fixtures at the adapter/core seam. + - Done when a canonical demonstrated fixture attaches successfully and every negative proof shape remains at its prior evidence basis. + - _Requirements: 9.6, 14.2, 14.3_ + - _Boundary: Proof Adapter_ + - _Depends: 7.4_ +- [ ] 9.4 (P) Migrate the existing entry-pack loader to the reviewed v2 schema + - Translate existing metadata and fact rows without adding framework vocabulary or changing target behavior. + - Supply complete structural match evidence and pack/fact/matcher provenance to the shared fact validator. + - Keep acquisition and review ownership outside core. + - Done when the migrated existing pack produces byte-stable behavior through v2 and partial/name-only matches remain incomplete. + - _Requirements: 10.2, 11.1, 11.2, 11.3, 11.4, 11.5, 14.3_ + - _Boundary: Fact Pack Adapter_ + - _Depends: 5.4_ +- [ ] 9.5 (P) Produce readiness and development records through benchmark-owned adapters + - Construct canonical readiness results while leaving sequencing and persistence with benchmark workflows. + - Emit development observations with v2 identities, evidence-use roles, instrument receipts, and prerequisite ancestry. + - Stop a runner before population work when readiness is failed, absent, or stale. + - Done when development fixtures pass through v2 and an injected semantic identity change prevents the later run. + - _Requirements: 8.1, 8.3, 8.4, 8.5, 8.6, 14.3, 15.5_ + - _Boundary: Benchmark Readiness Adapter_ + - _Depends: 4.4, 8.2_ +- [ ] 9.6 Integrate benchmark-owned admission scoring and guarded custody + - Delegate neutral scoring to core while retaining population persistence and run orchestration in the benchmark. + - Make the custodian the sole append-only writer and combine durable compare-and-swap spend plus label release as one guarded operation. + - Supply expanded manifest, lineage, resolver, and validation-context identities without exposing labels early. + - Exercise durable append, release, crash-before-append, crash-after-append, and concurrent-winner behavior at this external boundary. + - Done when synthetic admission completes end to end and injected crash/concurrency cases prove labels reach at most the successful compare-and-swap winner. + - _Requirements: 12.1, 12.2, 12.3, 12.4, 12.5, 12.6, 13.4_ + - _Depends: 8.8, 9.5_ + +- [ ] 10. Complete the single-writer v2 migration +- [ ] 10.1 Prepare schema-routed legacy reads and cutover tests without publishing v2 + - Add an internal read-only decoder that validates committed v1 traces and reports without constructing v2 determinate outcomes automatically. + - Route v1/v2 test reads by explicit schema version and prohibit all new writes through the legacy surface. + - Keep the top-level public API unchanged until the import and writer census reaches zero. + - Done when historical artifacts validate byte-unchanged, legacy write attempts fail, and cutover tests pass without publishing top-level v2 exports. + - _Requirements: 14.3, 15.3_ +- [ ] 10.2 Stop v1 writers and publish the single v2 public surface + - Re-run repository import and artifact-writer censuses after all adjacent adapters use v2. + - Stop v1 writes and remove the compatibility shim, authorization-shaped public exports, and family-domain conversion only when both censuses reach zero. + - Publish dependency-free v2 contracts through the top-level core API after the zero-census gate. + - Retain the read-only legacy decoder and refuse comparison between v1 and v2 baseline identities. + - Done when no production writer/import targets the v1 contract, historical fixtures remain unchanged, and all public writes use v2. + - _Requirements: 14.3, 15.3_ + +- [ ] 11. Validate readiness, failure visibility, and implementation boundaries +- [ ] 11.1 Run both reference families through the complete synthetic readiness path + - Start from a verified nonzero input and run recognition, transitions, security object/constraints, pair, development, reviewed synthetic admission, guarded spend, and consumer verdict. + - Produce violation, bounded satisfaction, and bounded silence for both unlike families. + - Require current prerequisite readiness identities at every later stage. + - Done when both authored repositories complete the path without target-specific core fields or a real admission population. + - _Requirements: 3.5, 8.1, 8.2, 8.4, 8.6, 12.1, 14.1, 15.5_ +- [ ] 11.2 Inject one failure at every semantic and governance stage + - Exercise unread/zero input, missing recognition, incomplete transition, missing constraint, pair mismatch, failed development, contaminated admission, and refused integration. + - Verify each run stops at the first failing stage with a named correction lever and no later metric or clean result. + - Mutate a semantic identity and verify transitive staleness through all dependent readiness and measurement records. + - Done when every injected failure is diagnosed at its source and no late admission-only surprise remains. + - _Requirements: 5.3, 5.4, 5.5, 8.3, 8.4, 8.5, 15.4, 15.5_ +- [ ] 11.3 Verify bounded validation cost and total consumer deadline accounting + - Measure canonicalization and validation across increasing relation, observation, receipt, and case counts. + - Exercise large transition sets for linear behavior and bounded memory without weakening semantic checks. + - Include optional model and proof processing in the same measured consumer budget. + - Publish sample count, percentile, workload, hardware, and derived budget through the fixed measurement contract. + - Done when performance fixtures satisfy the derived workload envelope and any excess fails the named budget gate. + - _Requirements: 1.4, 6.1, 6.2, 6.3, 6.4_ +- [ ] 11.4 Run the full migration, security, and regression gate + - Verify core imports only standard-library and lower core layers and contains no target, DSPy, benchmark, proof-execution, or analyzer dependencies. + - Run all unit, adapter, migration, reference-family, end-to-end, admission, and historical compatibility suites. + - Enforce function/module complexity caps and scan for remaining v1 imports or writers. + - Prove each test instrument read nonzero fixture bytes before interpreting zero findings or failures. + - Done when the complete suite passes, boundary/import censuses are clean, historical bytes are unchanged, and all declared adapter and consumer seams pass conformance. + - _Requirements: 1.4, 10.4, 14.3, 15.3, 15.4_