diff --git a/.kiro/specs/detection-core/design.md b/.kiro/specs/detection-core/design.md index f8379b9..389f012 100644 --- a/.kiro/specs/detection-core/design.md +++ b/.kiro/specs/detection-core/design.md @@ -229,7 +229,9 @@ src/openultrasast/core/ ├── verdict.py # Claims, satisfaction, silence, proof attachment and aggregation ├── probes.py # Reviewed semantic-equivalence probes and fitted-component evidence gates ├── measurement.py # Complete baseline identity, rates, funnel and report schema v2 -├── admission.py # Census, feasibility, evidence-derived scoring and fitted-component gate +├── admission_case.py # Case results and authenticated post-spend release bindings +├── admission_metrics.py # Evidence-derived scoped and whole-population metric algebra +├── admission.py # Census, feasibility, report assembly and fitted-component gate ├── consumer.py # Scoped push request and verdict consumption └── legacy.py # Read-only v1 trace and report decoder and validator @@ -374,16 +376,21 @@ sequenceDiagram Custodian-->>Benchmark: Durable spend receipt then release labels Benchmark->>Arm: Run frozen analyzer facts and model Arm-->>Benchmark: Immutable case results - Benchmark->>Core: Preparation receipt case results and instrument receipts - Core->>Core: Revalidate canonical inputs and score scopes separately - Core-->>Benchmark: Report bound to authorization and spend receipt + Benchmark->>Custodian: Submit immutable case-result identities + Custodian-->>Benchmark: Attested case release bound to spend receipt + Benchmark->>Core: Preparation receipt case release results and instrument receipts + Core->>Core: Revalidate release bindings canonical inputs and scoped scores + Core-->>Benchmark: Report bound to authorization spend receipt and case release ``` Core never creates, edits, labels, or persists the admission population. The benchmark custodian performs durable append and label release as one guarded operation: a crash before the append releases no labels, while a crash after it leaves the population spent. Concurrent attempts race on the same custody head, and only the compare-and-swap winner receives label -bytes. +bytes. The case release is constructed only after that spend. It discloses the per-case +role and pair mapping that the aggregate census deliberately withholds, and it binds those +labels to the exact immutable case-result and evidence/input identities submitted for +scoring. ## Requirements Traceability @@ -400,7 +407,7 @@ bytes. | 9.1, 9.2, 9.3, 9.4, 9.5, 9.6 | Bounded model and proof roles | Model Handoff, Family Contract, Verdict, Proof Attachment | Model residual and proof flow | | 10.1, 10.2, 10.3, 10.4 | Complete reproducibility identities | Provenance and Lineage, Fact Pack, Model Handoff | All record flows | | 11.1, 11.2, 11.3, 11.4, 11.5, 11.6 | Reviewed semantic packs and supported scope | Fact Pack, Semantic Evidence, Measurement | Pack ingestion and reports | -| 12.1, 12.2, 12.3, 12.4, 12.5, 12.6 | Admission prerequisites and scoped scoring | Admission, Readiness, Measurement | Admission sequence | +| 12.1, 12.2, 12.3, 12.4, 12.5, 12.6 | Admission prerequisites, authenticated case release and scoped scoring | Admission, Readiness, Measurement | Admission sequence | | 13.1, 13.2, 13.3, 13.4 | Development and evaluation separation | Provenance and Lineage, Probes, Admission | Freeze and admission validation | | 14.1, 14.2, 14.3, 14.4, 14.5 | Shared consumer and family-spec contract | Consumer, Verdict, Family Capability Declaration, Legacy Adapter | Push and migration flows | | 15.1, 15.2, 15.3, 15.4, 15.5 | Fixed versioned measurement and baseline rules | Measurement, Coverage, Readiness | Development and admission reporting | @@ -423,7 +430,7 @@ bytes. | Verdict and Proof | Decision | Produce bounded outputs and attach proof | 7.1–7.4, 9.4, 9.6, 14.2, 14.4 | Evidence, Model | Service, State | | Probes | Governance | Test semantic shortcut resistance | 13.2 | Evidence, Family, Fact Pack review policy | Service | | Measurement | Evaluation | Create baseline identity and fixed report schema | 11.6, 15.1–15.5 | Coverage, Readiness | Service, State | -| Admission | Evaluation | Validate manifests, feasibility and scoped scores | 12.1–12.6, 13.4 | Readiness, Measurement, Evidence Validation, Verdict, Probes, Budget, Provenance | Service, Batch | +| Admission | Evaluation | Validate manifests, feasibility, post-spend case release and scoped scores | 12.1–12.6, 13.4 | Readiness, Measurement, Evidence Validation, Verdict, Probes, Budget, Provenance | Service, Batch | | Consumer | Edge | Validate push requests and return bounded verdicts | 6.4, 14.1–14.5 | Budget, Evidence Validation, Verdict, Provenance, Canonical resolver | Service | | Legacy Decoder | Migration | Read and validate historical v1 only | 14.3, 15.3 | Canonical Codec, Evidence | Service | | External Adapters | Integration | Translate authorization, DSPy and proof records | 3.5, 9.1–9.6, 14.2–14.3 | Public core | Service | @@ -1721,6 +1728,11 @@ authorization; the derived result digests must exactly equal the fitted-componen separately. - The census exposes only reviewed aggregate cardinalities, never case identities or per-case labels. It is bound to the same population digest as the manifest. +- After the population is durably spent, the custodian produces an independently attested + `AdmissionCaseRelease`. The release is bound to the population, manifest, authorization, + preparation, exact spend receipt, and custodian identity. Its bindings associate each + case identity with one scope, role, scoped pair identity where applicable, result digest, + evidence-input digests, change-input digests, and input-manifest digest. - Before labels open, feasibility enumerates the arithmetically possible counts using pair, case, semantic-site, and model-call capacities and applies count, rate, per-100, or Wilson-bound statistics. Every threshold group must have an attainable joint count. An @@ -1731,6 +1743,10 @@ authorization; the derived result digests must exactly equal the fitted-componen trusted. The scorer re-runs record/fact/verdict validation and refuses supplied claims/satisfaction/silence that differ from the derived result. Core never persists or relabels cases. +- Scoring treats the post-spend case release as the sole authority for case roles and pair + membership. It re-derives each case result and its operational projection, requires an + exact one-to-one binding, and refuses missing, extra, duplicate, substituted, swapped, or + cross-scope cases before producing any metric. - Admission authorization produces a custody-head-bound label-release request. The benchmark custodian must durably spend the population before releasing label bytes; scoring revalidates that exact current-chain receipt and never emits a post-scoring spend @@ -1898,6 +1914,32 @@ class AdmissionCaseResult: total_reply_ms: int result_digest: str +@dataclass(frozen=True) +class ReleasedCaseBinding: + case_id_digest: str + scope: ScopeKey + role: AdmissionCaseRole + pair_id_digest: str | None + case_result_digest: str + evidence_input_digests: tuple[str, ...] + change_input_digests: tuple[str, ...] + input_manifest_digest: str + binding_digest: str + +@dataclass(frozen=True) +class AdmissionCaseRelease: + schema_version: str + population_digest: str + manifest_digest: str + authorization_digest: str + preparation_digest: str + label_release_receipt_digest: str + custodian_identity: str + bindings: tuple[ReleasedCaseBinding, ...] + release_payload_digest: str + review: IndependenceAttestation + release_digest: str + @dataclass(frozen=True) class MetricResult: metric_id: MetricId @@ -1977,6 +2019,7 @@ class AdmissionReportV2: validation_context_digest: str authorization_digest: str label_release_receipt_digest: str + case_release_digest: str case_result_digests: tuple[str, ...] in_scope_case_count: int whole_population_case_count: int @@ -2015,6 +2058,7 @@ def score_admission( manifest: AdmissionManifestV2, preparation: AdmissionPreparation, label_release_receipt: PopulationUseReceipt, + case_release: AdmissionCaseRelease, cases: Sequence[AdmissionCaseResult], context: AdmissionValidationContext, custody: PopulationCustodyPort, @@ -2042,6 +2086,15 @@ and the conflict disclosure must resolve under the policy's store and byte limit self-review, unknown reviewer, fabricated digest, or review copied from another payload is refused before feasibility. +`AdmissionCaseRelease` is a post-spend contract, not part of the label-blind census or +manifest. Its payload digest includes every substantive field except the review attestation; +the release digest includes both. The review producer must equal both the declared custodian +and the validated spend receipt's custodian, while the reviewer must be distinct and admitted +by the context's admission-review policy. The attestation subject must equal the release +payload digest, and its conflict disclosure is resolved under the same store and byte limits. +The release population, manifest, authorization, preparation, and receipt digests must match +the exact values being scored. + The manifest identity, census population, claimed scopes, contract-set, facts, acquisition, analyzer, configuration, model-program set, denominator, threshold set, and outcome mapping must all match. `authorize_admission` re-derives the @@ -2074,7 +2127,16 @@ append leaves the population spent, and concurrent losers receive no label bytes `score_admission` requires that spend receipt to be the current-chain direct successor of the authorization head and to match the preparation byte for byte. It refuses a different -manifest, context, readiness set, census, case population, or receipt. For every case it +manifest, context, readiness set, census, case population, receipt, or case release. It +validates the case release only after the spend receipt has been revalidated. Release +bindings are canonically ordered by case identity and unique; vulnerable and fixed cases +require one pair identity qualified by their `ScopeKey`, ordinary cases prohibit one, and +every scoped pair contains exactly one vulnerable and one fixed binding. For each supplied +case, scoring re-derives `result_digest`, the sorted evidence-input digests, the distinct +change-input digests from validated packet provenance, and the input-manifest digest and +requires exact equality with its release binding. The supplied scope, role, and pair must +also equal the binding. Missing, extra, duplicate, substituted, role-swapped, payload-swapped, +or cross-scope bindings fail before metrics. For every case it calls `validate_evidence_input` for every `evidence_inputs` member, then performs instrument and budget validation, and verdict/proof validation using the context-bound policies. Proof attachments are applied only to those validated base records and the resulting claims @@ -2086,7 +2148,8 @@ Claims, satisfaction, silence, decision vocabulary, metrics, and gates are then from those revalidated objects. A forged family decision, substituted contract, invalid fact review or match, contaminated lineage, or untrusted proof fails before any metric or gate is produced. The final report binds the validation context, authorization, and -pre-label spend receipt; there is no post-scoring population-use request. +pre-label spend receipt, and authenticated post-spend case release; there is no +post-scoring population-use request. The report validates case counts back to the census, reports paired vulnerable/fixed behavior and ordinary false alarms through registered metrics, imports semantic-stage and @@ -2340,7 +2403,8 @@ dependencies into core. | `PopulationCensus` | Population, scope cardinalities, payload and independent review attestation | Aggregate and label-blind; no case IDs; same population as manifest; no self-review | | `BaselineIdentity` | Arm, population/freeze, claimed scopes, contract set, fact schema/set, acquisition, analyzer, configuration, model set, denominator, threshold set and outcome mapping | Any changed field creates a new baseline | | `AdmissionPreparation` | Manifest/context authorization and expected custody head | Feasible pre-registration; label-release request spends before bytes open | -| `AdmissionReportV2` | Baseline, validation context, authorization, spend receipt and case/result digests | Canonical input revalidation; in-scope scoring; all gates reported | +| `AdmissionCaseRelease` | Population, manifest, authorization, preparation, spend receipt, custodian and attested case bindings | Post-spend only; exact one-to-one case/result/evidence/change/input binding; scoped roles and pairs | +| `AdmissionReportV2` | Baseline, validation context, authorization, spend receipt, case release and case/result digests | Canonical input revalidation; in-scope scoring; all gates reported | | `RepositoryHoldoutReport` | Component, disjoint repository manifests and measurement | Explicit holdout role; matches fitting lineage and component identity | | `DifferentialProofRecord` | Reviewed producer, control-bound sides and resolved witness/verification receipt | Affected change identity, reviewed safe control, trusted producer/store policy | | `PushVerdict` | Request and record digests | No unbounded clean result; proof basis does not imply rollout permission | @@ -2381,6 +2445,7 @@ that can be mistaken for a clean scan. | Provenance mismatch | Any digest cannot be re-derived | Refuse affected result and dependent later receipts | | Review mismatch | Self-review, unknown reviewer, copied attestation, or unreadable disclosure | Refuse before feasibility or semantic use | | Custody mismatch | Old/incomplete log, broken chain, or prior spend | Refuse use/admission; preserve monotonic spent status | +| Case release mismatch | Pre-spend release, wrong custodian/receipt, missing/extra case, role/pair swap, or operational payload substitution | Refuse before metric production | ### Monitoring @@ -2443,7 +2508,10 @@ spec and consumes only status, counts, and digests. covers spend-before-label crash/concurrency order and refuses a forged family decision, substituted contract, invalid fact review, or contaminated model/threshold/matcher lineage before metrics or gates. Manifest and census self-review, unknown reviewers, - copied attestations, and unresolved conflict disclosures fail before feasibility + copied attestations, and unresolved conflict disclosures fail before feasibility. A + post-spend release matrix rejects a wrong custodian or receipt, missing/extra/duplicate + bindings, cross-scope pairs, role swaps, result swaps, and rehashed evidence/change/input + substitutions before metric production (`12.1–12.6`, `13.1`, `13.4`). - `test_core_measurement.py`: required schema, basis stratification, Wilson counts, baseline mutation matrix, zero receipts, and prerequisite stage result/digest references. A diff --git a/.kiro/specs/detection-core/research.md b/.kiro/specs/detection-core/research.md index e0844e9..4ed5f4c 100644 --- a/.kiro/specs/detection-core/research.md +++ b/.kiro/specs/detection-core/research.md @@ -523,3 +523,44 @@ requirements approval. expectations. Fact content forbids all labelled ancestry; model, threshold, matcher, and fitted-component validation protects the current evaluation population while allowing separately spent development populations. + +--- + +## Post-spend case-release trust closure (2026-11-01) + +### Scope of this pass + +- **Context:** task 8.7 demonstrated that aggregate census cardinalities and a valid spend + receipt do not authenticate which immutable operational result belongs to which released + case role and pair. A caller could swap vulnerable/fixed payloads, recompute result + digests, and change admission metrics while preserving every approved pre-label object. +- **Sources consulted:** requirements 12.1–12.6 and 15.1–15.5, the admission design and + custody contracts, the task-8.7 implementation diff and adversarial review reproducer, + and project steering on label isolation and fixed measurement. +- **Finding:** this is a design-contract gap, not a metric-algebra bug. The pre-label census + correctly withholds per-case identities and labels; therefore it cannot also authenticate + the post-opening mapping used for scoring. + +### Decision: add an authenticated post-spend case release + +- **Alternatives considered:** put per-case commitments in the pre-label census; trust + caller-supplied case roles after a valid spend; or add a separate post-spend release. +- **Selected approach:** the custodian produces an independently attested + `AdmissionCaseRelease` only after the exact label-opening spend. Each binding joins a case + identity and released scope/role/pair to the exact result, evidence-input, change-input, + and input-manifest identities that core re-derives during scoring. +- **Rationale:** the release preserves aggregate-only pre-registration while giving the + scorer an authority for the association that determines every numerator and denominator. +- **Trade-offs:** admission orchestration gains one immutable artifact and review step. In + exchange, case substitution, role swaps, cross-scope pair collisions, and rehashed payload + swaps fail before any measurement is emitted. +- **Boundary:** core defines and validates the release; the benchmark custodian remains the + sole producer, durable writer, and label releaser. No cryptographic signing system or + population persistence moves into core. + +### Revalidation impact + +- Admission scoring and report assembly must consume and bind the release digest. +- Benchmark-owned guarded custody must produce the release after the winning spend and + prove crash/concurrency behavior at that external boundary. +- The pre-label census and feasibility contracts remain unchanged and label-blind. diff --git a/.kiro/specs/detection-core/spec.json b/.kiro/specs/detection-core/spec.json index ecdf6bf..9d403d5 100644 --- a/.kiro/specs/detection-core/spec.json +++ b/.kiro/specs/detection-core/spec.json @@ -1,7 +1,7 @@ { "feature_name": "detection-core", "created_at": "2026-10-30T13:55:30Z", - "updated_at": "2026-10-31T10:59:35Z", + "updated_at": "2026-11-01T09:03:42Z", "language": "en", "phase": "tasks-generated", "approvals": { diff --git a/.kiro/specs/detection-core/tasks.md b/.kiro/specs/detection-core/tasks.md index 0ffcb53..2a02657 100644 --- a/.kiro/specs/detection-core/tasks.md +++ b/.kiro/specs/detection-core/tasks.md @@ -263,21 +263,31 @@ - Keep durable append, compare-and-swap execution, label release, persistence, crash handling, and concurrency outside core. - Done when only the exact spent successor authorizes scoring and every other receipt shape is refused without an external side effect. - _Requirements: 12.2, 13.4_ -- [ ] 8.7 Derive scoped admission metrics from canonical case evidence +- [ ] 8.7 Validate the authenticated post-spend case release + - Define canonical released-case bindings over case identity, scope, role, scoped pair identity, exact result digest, evidence-input digests, change-input digests, and input-manifest digest. + - Bind the release to the population, manifest, authorization, preparation, exact current-chain spend receipt, custodian identity, and an independent review attestation. + - Validate the release only after the exact spend successor and require one canonical binding for every supplied case and no others. + - Reject missing, extra, duplicate, cross-scope, role-swapped, pair-swapped, result-swapped, or rehashed operational-payload substitutions before metric production. + - Keep case identities and labels out of the pre-label census and make the external custodian, not the runner, authoritative for their post-spend association. + - Done when the reproduced vulnerable/fixed payload swap and every release-field mutation fail before any admission metric is returned. + - _Requirements: 12.2, 12.5, 12.6, 13.4, 15.1_ + - _Boundary: Admission Case Release_ +- [ ] 8.8 Derive scoped admission metrics from canonical case evidence - Re-run shared evidence validation, instrument, budget, verdict, and proof validation for every case. - - Match case roles and pairs back to the reviewed census and derive decision vocabulary and counts from the exact catalog. + - Consume only cases that exactly match the authenticated post-spend release; use its roles and scoped pairs while reconciling aggregate counts to the reviewed census. - Score quality only over pre-registered family/ecosystem scopes while retaining whole-population coverage. - Compute paired behavior, ordinary false alarms, semantic coverage, model outcomes, and all standard metrics without trusting runner aggregates. - Done when forged decisions, substituted contracts, invalid fact review, contaminated lineage, or case-count drift fail before metric production. - _Requirements: 12.5, 12.6, 15.1, 15.2_ -- [ ] 8.8 Assemble evidence-backed gates and the final admission report + - _Depends: 8.7_ +- [ ] 8.9 Assemble evidence-backed gates and the final admission report - Emit instrument, budget, readiness, and quality gates with reasons and supporting identities. - - Bind the report to the validation context, authorization, pre-label spend receipt, census, and complete case-result set. + - Bind the report to the validation context, authorization, pre-label spend receipt, authenticated case release, census, and complete case-result set. - Require every mandatory metric and gate, plus model success when applicable, before allowing GO. - Keep model-corroborated, deterministic, and execution-backed counts separate. - Done when any missing/failed mandatory gate forces NO-GO and a complete passing synthetic fixture yields a reproducible GO report. - _Requirements: 12.6, 15.1, 15.2, 15.4, 15.5_ -- [ ] 8.9 Authorize fitted components only from replayed probes and disjoint evaluation +- [ ] 8.10 Authorize fitted components only from replayed probes and disjoint evaluation - Resolve the component’s unique model declaration, family contract, review policy, and custody policy from the admission-bound context. - Re-evaluate every canonical probe input and require the derived result set to match the component evidence. - Consume the typed repository-holdout report from task 8.2 and validate its evidence-use role, training/holdout references, report identity, and fitting lineage. @@ -330,11 +340,12 @@ - [ ] 9.6 Integrate benchmark-owned admission scoring and guarded custody - Delegate neutral scoring to core while retaining population persistence and run orchestration in the benchmark. - Make the custodian the sole append-only writer and combine durable compare-and-swap spend plus label release as one guarded operation. + - After the winning spend, produce and independently attest the canonical case release over the exact immutable result, evidence, change, and input identities supplied to scoring. - Supply expanded manifest, lineage, resolver, and validation-context identities without exposing labels early. - - Exercise durable append, release, crash-before-append, crash-after-append, and concurrent-winner behavior at this external boundary. - - Done when synthetic admission completes end to end and injected crash/concurrency cases prove labels reach at most the successful compare-and-swap winner. + - Exercise durable append, label release, case-release production, crash-before-append, crash-after-append, and concurrent-winner behavior at this external boundary. + - Done when synthetic admission completes end to end, injected crash/concurrency cases prove labels reach at most the successful compare-and-swap winner, and swapped operational payloads are refused by core. - _Requirements: 12.1, 12.2, 12.3, 12.4, 12.5, 12.6, 13.4_ - - _Depends: 8.8, 9.5_ + - _Depends: 8.9, 9.5_ - [ ] 10. Complete the single-writer v2 migration - [ ] 10.1 Prepare schema-routed legacy reads and cutover tests without publishing v2