From 6a1bc2023a605debc9ade6b923ecaa2664bfe738 Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Sun, 1 Nov 2026 11:16:38 +0100 Subject: [PATCH] docs(detection-core): add authoritative admission replay --- .kiro/specs/detection-core/design.md | 67 ++++++++++++++++++++++---- .kiro/specs/detection-core/research.md | 34 +++++++++++++ .kiro/specs/detection-core/spec.json | 2 +- .kiro/specs/detection-core/tasks.md | 9 +++- 4 files changed, 100 insertions(+), 12 deletions(-) diff --git a/.kiro/specs/detection-core/design.md b/.kiro/specs/detection-core/design.md index 389f012..d9a5ee7 100644 --- a/.kiro/specs/detection-core/design.md +++ b/.kiro/specs/detection-core/design.md @@ -231,7 +231,8 @@ src/openultrasast/core/ ├── measurement.py # Complete baseline identity, rates, funnel and report schema v2 ├── admission_case.py # Case results and authenticated post-spend release bindings ├── admission_metrics.py # Evidence-derived scoped and whole-population metric algebra -├── admission.py # Census, feasibility, report assembly and fitted-component gate +├── admission.py # Census, feasibility and fitted-component gate +├── admission_report.py # Evidence-backed gates, report assembly and replay validation ├── consumer.py # Scoped push request and verdict consumption └── legacy.py # Read-only v1 trace and report decoder and validator @@ -1747,6 +1748,9 @@ authorization; the derived result digests must exactly equal the fitted-componen membership. It re-derives each case result and its operational projection, requires an exact one-to-one binding, and refuses missing, extra, duplicate, substituted, swapped, or cross-scope cases before producing any metric. +- Scoring returns an immutable `AdmissionReplay`, not a bare report. The replay carries the + exact authoritative objects needed to reproduce the report. A raw `AdmissionReportV2` is + serializable result data, never an authorization token. - Admission authorization produces a custody-head-bound label-release request. The benchmark custodian must durably spend the population before releasing label bytes; scoring revalidates that exact current-chain receipt and never emits a post-scoring spend @@ -1757,8 +1761,9 @@ authorization; the derived result digests must exactly equal the fitted-componen `measurement.py` owns `ScopeKey`, baseline identity, the closed metric algebra, interval and gate results, readiness references, `MeasurementRecordV2`, and `RepositoryHoldoutReport`. `admission.py` imports those types and owns census, metric rules, -thresholds, manifests, case inputs, authorization, scoring, admission reports, and fitted -component authorization. Measurement never imports admission. +thresholds, manifests, case inputs, authorization, and fitted-component authorization. +`admission_report.py` owns gate derivation, report assembly, the authoritative replay bundle, +and replay validation. Measurement never imports admission modules. ```python @dataclass(frozen=True) @@ -2031,6 +2036,18 @@ class AdmissionReportV2: decision: AdmissionDecision report_digest: str +@dataclass(frozen=True) +class AdmissionReplay: + manifest: AdmissionManifestV2 + denominator: DenominatorPolicy + readiness: tuple[ReadinessResult, ...] + preparation: AdmissionPreparation + label_release_receipt: PopulationUseReceipt + case_release: AdmissionCaseRelease + cases: tuple[AdmissionCaseResult, ...] + report: AdmissionReportV2 + replay_digest: str + @dataclass(frozen=True) class AdmissionAuthorization: manifest_digest: str @@ -2056,6 +2073,8 @@ def authorize_admission( def score_admission( manifest: AdmissionManifestV2, + denominator: DenominatorPolicy, + readiness: Sequence[ReadinessResult], preparation: AdmissionPreparation, label_release_receipt: PopulationUseReceipt, case_release: AdmissionCaseRelease, @@ -2063,6 +2082,13 @@ def score_admission( context: AdmissionValidationContext, custody: PopulationCustodyPort, resolver: ContentResolver, +) -> AdmissionReplay: ... + +def validate_admission_replay( + replay: AdmissionReplay, + context: AdmissionValidationContext, + custody: PopulationCustodyPort, + resolver: ContentResolver, ) -> AdmissionReportV2: ... def authorize_fitted_component( @@ -2071,7 +2097,7 @@ def authorize_fitted_component( context: AdmissionValidationContext, fitting_lineage: LineageManifest, repository_holdout: RepositoryHoldoutReport, - admission: AdmissionReportV2, + admission: AdmissionReplay, custody: PopulationCustodyPort, resolver: ContentResolver, ) -> None: ... @@ -2151,6 +2177,15 @@ gate is produced. The final report binds the validation context, authorization, pre-label spend receipt, and authenticated post-spend case release; there is no post-scoring population-use request. +`score_admission` freezes the denominator, readiness, and cases exactly once, derives the +report, and returns them with every other authoritative scoring input in an +`AdmissionReplay`. Its `replay_digest` protects transport integrity of the bundle but is not +a validation token. `validate_admission_replay` revalidates every carried object against an +independently supplied current context, custody port, and content resolver, re-runs report +derivation, and requires exact report and replay equality. A caller cannot authorize from a +raw report or from a coordinated set of rehashed report, threshold, readiness, or case +objects. + The report validates case counts back to the census, reports paired vulnerable/fixed behavior and ordinary false alarms through registered metrics, imports semantic-stage and model-call outcomes, and stratifies findings by evidence basis so model-corroborated and @@ -2171,7 +2206,9 @@ disjoint, matches component/training/lineage/report digests, validates current c the fitting lineage with subject `component_digest` and a required `FITTED_MODEL` expectation for that digest, and requires the admission report to be a `GO` produced from a population that was untouched before its -bound label-release spend. It re-derives `context.context_digest`, requires it to equal the +bound label-release spend. It first calls `validate_admission_replay`; a bare +`AdmissionReportV2` or a replay whose carried inputs no longer validate cannot authorize a +component. It re-derives `context.context_digest`, requires it to equal the validated report's validation-context digest, and requires exactly one baseline-listed declaration in `context.evidence.model_programs` whose `fitted_component_digest` equals `evidence.component_digest`. The selected declaration supplies the family-contract digest; @@ -2188,9 +2225,15 @@ authorize the component. Every admission report must contain `PAIRED_CORRECTNESS`, `VULNERABLE_RECALL`, `FIXED_FALSE_ALARM_RATE`, `ORDINARY_FALSE_ALARMS_PER_100`, `PRECISION`, and `SEMANTIC_STAGE_COVERAGE` with nonzero pre-registered denominators, plus model success when -applicable. It must also contain evidence-backed instrument, budget, readiness, and quality -gate results. `GO` is invalid if any mandatory metric or gate is absent, any instrument -receipt failed, or any required gate did not pass. +applicable. Model success is applicable when the frozen baseline configures at least one +model program **or** validated claimed-scope cases contain declared model calls. An +applicable model with zero attempted calls, no model-success threshold, or any failed model +outcome forces `NO_GO`; calls wholly outside the claimed scopes do not make the model +applicable. Claimed-scope model calls without a baseline-listed declaration are invalid +evidence and are refused before report production. The report must also contain +evidence-backed instrument, budget, readiness, and quality gate results. `GO` is invalid if +any mandatory metric or gate is absent, any instrument receipt failed, or any required gate +did not pass. ### Decision and Consumer Edge @@ -2404,7 +2447,8 @@ dependencies into core. | `BaselineIdentity` | Arm, population/freeze, claimed scopes, contract set, fact schema/set, acquisition, analyzer, configuration, model set, denominator, threshold set and outcome mapping | Any changed field creates a new baseline | | `AdmissionPreparation` | Manifest/context authorization and expected custody head | Feasible pre-registration; label-release request spends before bytes open | | `AdmissionCaseRelease` | Population, manifest, authorization, preparation, spend receipt, custodian and attested case bindings | Post-spend only; exact one-to-one case/result/evidence/change/input binding; scoped roles and pairs | -| `AdmissionReportV2` | Baseline, validation context, authorization, spend receipt, case release and case/result digests | Canonical input revalidation; in-scope scoring; all gates reported | +| `AdmissionReportV2` | Baseline, validation context, authorization, spend receipt, case release and case/result digests | Canonical result data; in-scope scoring; all gates reported; never an authorization token by itself | +| `AdmissionReplay` | Canonical digest of the exact authoritative scoring inputs and derived report | Immutable transport bundle; independently replayed against current context, custody and content before trust | | `RepositoryHoldoutReport` | Component, disjoint repository manifests and measurement | Explicit holdout role; matches fitting lineage and component identity | | `DifferentialProofRecord` | Reviewed producer, control-bound sides and resolved witness/verification receipt | Affected change identity, reviewed safe control, trusted producer/store policy | | `PushVerdict` | Request and record digests | No unbounded clean result; proof basis does not imply rollout permission | @@ -2513,6 +2557,11 @@ spec and consumes only status, counts, and digests. bindings, cross-scope pairs, role swaps, result swaps, and rehashed evidence/change/input substitutions before metric production (`12.1–12.6`, `13.1`, `13.4`). +- `test_core_admission_report.py`: evidence-backed gate derivation, exact report identity, + `AdmissionReplay` construction and authoritative revalidation, configured-model zero-call + refusal, claimed-scope undeclared-model refusal, out-of-scope-only model-call isolation, + and coordinated report/readiness/threshold/case substitution refusal. Raw reports cannot + authorize later consumers (`12.6`, `13.1–13.2`, `15.1–15.5`). - `test_core_measurement.py`: required schema, basis stratification, Wilson counts, baseline mutation matrix, zero receipts, and prerequisite stage result/digest references. A two-arm case over one population keeps family/ecosystem scope, census rows, denominator diff --git a/.kiro/specs/detection-core/research.md b/.kiro/specs/detection-core/research.md index 4ed5f4c..add3d80 100644 --- a/.kiro/specs/detection-core/research.md +++ b/.kiro/specs/detection-core/research.md @@ -564,3 +564,37 @@ requirements approval. - Benchmark-owned guarded custody must produce the release after the winning spend and prove crash/concurrency behavior at that external boundary. - The pre-label census and feasibility contracts remain unchanged and label-blind. + +--- + +## Authoritative admission replay closure (2026-11-01) + +### Scope of this pass + +- **Context:** task 8.9 showed that a structurally valid, self-consistent report cannot be + authenticated later without the authoritative manifest, denominator, readiness, + preparation, spend receipt, case release, and case results from which it was derived. +- **Finding:** contextual report validation inside the scoring call fixed local gate forgery + but did not give task 8.10 a safe consumer contract. Accepting a bare report would either + trust its digest as a capability or require every consumer to recover unbound inputs. +- **Classification:** specification conflict at the 8.9/8.10 boundary, not an isolated code + defect. The report is result data; later authorization needs replayable evidence. + +### Decision: carry and independently replay authoritative inputs + +- **Alternatives considered:** treat `report_digest` as an authorization token; add more + nested evidence to the report; make every fitted-component caller pass the original inputs + separately; or return one immutable replay bundle from scoring. +- **Selected approach:** `score_admission` returns `AdmissionReplay`, containing the exact + frozen authoritative inputs and the derived report. `validate_admission_replay` accepts + only that bundle plus independently supplied current context, custody, and content + resolution, re-runs the full derivation, and compares exact output. +- **Rationale:** the bundle prevents accidental input loss without promoting a digest or + caller-provided aggregate into a capability. Current external trust state remains an + explicit validator input. +- **Model applicability closure:** model success is mandatory when a model is frozen in the + baseline or validated claimed-scope calls occurred. Configured zero-call, missing model + threshold, or failed declared calls force `NO_GO`; undeclared claimed-scope calls are + refused before report production, and out-of-scope-only calls do not alter the arm's gate. +- **Boundary:** replay production remains part of task 8.9. Task 8.10 consumes and validates + the replay before fitted-component checks and refuses raw reports. diff --git a/.kiro/specs/detection-core/spec.json b/.kiro/specs/detection-core/spec.json index 9d403d5..77313e6 100644 --- a/.kiro/specs/detection-core/spec.json +++ b/.kiro/specs/detection-core/spec.json @@ -1,7 +1,7 @@ { "feature_name": "detection-core", "created_at": "2026-10-30T13:55:30Z", - "updated_at": "2026-11-01T09:03:42Z", + "updated_at": "2026-11-01T10:15:05Z", "language": "en", "phase": "tasks-generated", "approvals": { diff --git a/.kiro/specs/detection-core/tasks.md b/.kiro/specs/detection-core/tasks.md index 336380d..55d714d 100644 --- a/.kiro/specs/detection-core/tasks.md +++ b/.kiro/specs/detection-core/tasks.md @@ -283,11 +283,15 @@ - [ ] 8.9 Assemble evidence-backed gates and the final admission report - Emit instrument, budget, readiness, and quality gates with reasons and supporting identities. - Bind the report to the validation context, authorization, pre-label spend receipt, authenticated case release, census, and complete case-result set. - - Require every mandatory metric and gate, plus model success when applicable, before allowing GO. + - Require every mandatory metric and gate before allowing GO; make model success mandatory when a model is baseline-configured or validated claimed-scope calls occur. Refuse undeclared claimed-scope calls before report production; make configured zero-call, missing-threshold, and failed declared-call cases NO-GO while ignoring out-of-scope-only calls. + - Return an immutable admission replay bundle carrying the exact manifest, denominator, readiness, preparation, spend receipt, case release, cases, and derived report; give the bundle a canonical transport digest without treating it as a validation token. + - Revalidate the replay against independently supplied current context, custody, and content resolution by re-running the complete derivation and requiring exact report and replay equality. - Keep model-corroborated, deterministic, and execution-backed counts separate. - - Done when any missing/failed mandatory gate forces NO-GO and a complete passing synthetic fixture yields a reproducible GO report. + - Done when any missing/failed mandatory gate forces NO-GO, coordinated report/readiness/threshold/case substitutions are refused, and a complete passing synthetic fixture yields a reproducible replayed GO report. - _Requirements: 12.6, 15.1, 15.2, 15.4, 15.5_ + - _Depends: 8.8_ - [ ] 8.10 Authorize fitted components only from replayed probes and disjoint evaluation + - Accept only the typed admission replay bundle from task 8.9, validate it first against current authoritative context/custody/content, and refuse raw admission reports or coordinated replay substitutions. - Resolve the component’s unique model declaration, family contract, review policy, and custody policy from the admission-bound context. - Re-evaluate every canonical probe input and require the derived result set to match the component evidence. - Consume the typed repository-holdout report from task 8.2 and validate its evidence-use role, training/holdout references, report identity, and fitting lineage. @@ -295,6 +299,7 @@ - Require an admission GO that was untouched before its bound label-release spend. - Done when fabricated summaries, caller-selected policy substitutes, overlapping repositories, contaminated lineage, or mismatched admission are refused. - _Requirements: 13.1, 13.2, 13.4_ + - _Depends: 8.2, 8.9_ - [ ] 9. Integrate adjacent producers and benchmark-owned orchestration - [ ] 9.1 (P) Translate the parked authorization trace conservatively into v2 evidence