marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-11-14 10:18:17 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(specs): approve scoped evaluation admission design

Browse Source
This commit is contained in:
Marius Ciepluch
2026-11-06 14:11:42 +01:00
parent 3c2f86bbc0
commit 501a214b25
9 changed files with 3103 additions and 186 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/specs/detection-core/design.md
+898
View File
@@ -18,6 +18,9 @@ Authorization remains parked and supplies regression evidence only. Historical v
remain readable but are never rewritten or used as a v2 trend baseline. Live v1 contracts
and serialization first move behind one explicit, deletable `legacy_v1` namespace; cutover
is then proved by syntax and capability removal rather than runtime alias inference.
The evaluation-admission amendment adds a separate V3 root for exact-scope, multi-arm
qualification under one shared population spend. It preserves every V2 admission artifact
and replay path without conversion or reinterpretation.
### Goals
@@ -34,6 +37,11 @@ is then proved by syntax and capability removal rather than runtime alias infere
suites.
- Make the v1 removal gate structural: every live v1 import and writer crosses an explicit
compatibility namespace that is deleted at cutover.
- Qualify every component and combined arm independently for each exact family/ecosystem
scope while binding all arms to one reviewed plan, one atomic spend, and one complete
replay.
- Preserve V2 admission bytes, digests, decisions, and replay behavior as an immutable
compatibility contract.
### Non-Goals
@@ -44,6 +52,11 @@ is then proved by syntax and capability removal rather than runtime alias infere
- Reviving or re-admitting the parked authorization arm.
- Enabling any capability for advisory or blocking rollout.
- Providing cross-language canonical signatures; v2 is a Python-runtime contract.
- Converting pooled V2 admission evidence into V3 scoped evidence or treating a V2 report
as a V3 baseline.
- Owning population lifecycle, durable CAS/object storage, observation access enforcement,
detector isolation, retry orchestration, workflow presentation, or rollout decisions for
the multi-arm benchmark.
## Boundary Commitments
@@ -65,6 +78,9 @@ is then proved by syntax and capability removal rather than runtime alias infere
- A finite v1 migration and read-only historical decoder.
- The temporary `legacy_v1` compatibility namespace, frozen migration inventory, and
syntax-only cutover boundary that make v1 write capability removable.
- The V3 evaluation plan, exact arm/scope baseline and policy schemas, per-scope
feasibility, sealed-observation authority validation, all-arm spend binding, V3 release,
scoped reports, complete replay, and replay-only consumer authorization.
### Out of Boundary
@@ -78,6 +94,9 @@ is then proved by syntax and capability removal rather than runtime alias infere
- Push scoping policy, alert rendering, capability rollout, and BLOCK enablement.
- Production vulnerability-family contracts, target-specific authority bundles, structural
matchers, and match-artifact persistence. Those belong to the downstream family spec.
- Benchmark-owned population lifecycle, object-store/CAS adapters, detector execution,
observation access enforcement, label joining, retries, workflow status, presentation,
notification, and rollout policy.
### Allowed Dependencies
@@ -104,6 +123,9 @@ is then proved by syntax and capability removal rather than runtime alias infere
- Production imports name a core submodule or symbol explicitly. Module-valued or dynamic
access to `openultrasast.core` is outside the migration contract because it would hide
legacy use from a deletion-backed gate.
- V3 evaluation modules may reuse version-neutral core canonical, review, custody,
readiness, evidence, proof, coverage, budget, and metric-arithmetic primitives. They do
not import V2 aggregate roots; V2 modules do not import V3 modules.
### Revalidation Triggers
@@ -121,6 +143,11 @@ is then proved by syntax and capability removal rather than runtime alias infere
design review.
- Adding a v1 compatibility export, migration-inventory entry, direct v1 serializer, or
dynamic core import reopens the namespace-isolation design and blocks cutover.
- Changing any V3 plan, observation-slot, retry/history, shared-spend, case-release,
proof-candidate, combination, scoped metric, aggregate, report, replay, or consumer
selection contract revalidates the benchmark and every V3 consumer.
- Extracting or changing a version-neutral helper requires the frozen V2 canonical,
decision, failure, and replay suite to pass byte-for-byte.
## Architecture
@@ -139,6 +166,12 @@ test explicitly prohibits a public model path. Production push and proof paths d
consume the core end to end. The migration therefore preserves reusable mechanisms while
replacing the public domain model.
The implemented V2 admission root is intentionally retained but structurally limited: one
arm owns one baseline, every metric rule receives the complete claimed-scope tuple,
feasibility pools capacity across those scopes, and one unscoped metric set produces one
decision. Exact per-arm/per-family/per-ecosystem qualification therefore requires an
additive V3 aggregate graph rather than an in-place V2 extension.
### Architecture Pattern and Boundary Map
The selected pattern is a **versioned ports-and-adapters contract with an explicit,
@@ -2555,6 +2588,871 @@ map to the authorization family contract. Missing per-hop completeness remains i
The DSPy and search adapters similarly map external typed records without leaking their
dependencies into core.
## Evaluation-Admission V3 Amendment
Requirements 16–20 extend the implemented family-neutral core with a distinct exact-scope,
multi-arm evaluation contract. This section is normative for V3; the earlier V2 measurement
and admission design remains normative and unchanged for historical and compatibility use.
### Boundary Commitments
Detection-core owns the V3 canonical schemas, exact-type validators, reviewed-plan and
observation-authority contracts, per-arm/per-scope feasibility and metric algebra, shared
spend binding, V3 case release, component/combined lineage validation, scoped reports,
complete replay, and consumer authorization. It preserves every V2 schema, byte, digest,
decision, validator, and replay path unchanged.
Detection-core does not own population sourcing or lifecycle, durable object storage or
CAS, observation access enforcement, detector execution or isolation, retry orchestration,
label joining, workflow status, notification, presentation, or rollout policy. The benchmark
produces immutable reviewed lifecycle, observation, custody, and release artifacts; core
validates them and derives no authority from an unreviewed workflow state.
Allowed dependencies are Python 3.11 standard-library modules and existing version-neutral
core primitives: canonical content references/resolution, independence review, custody,
readiness, evidence validation, proof attachment, coverage, budget, and Wilson arithmetic.
V3 modules never import V2 aggregate roots. V2 modules never import V3 modules. Adjacent
benchmark and consumer code imports the public V3 facade.
Downstream revalidation is required by a V3 schema or canonical-profile change; a change to
plan, baseline, observation-slot, disposition/retry, shared-spend, release, proof-candidate,
combination, denominator, gate, aggregate, or replay semantics; or any shared-helper change
that alters frozen V2 output.
### Architecture
The selected pattern is parallel versioned aggregate roots with version-neutral internal
helpers. V2 remains an independently replayable historical/live compatibility contract.
V3 is a new baseline and has no conversion path from V2.
The public root schema is exactly
`openultrasast.detection-core.evaluation-admission.v3`. Strict decoding accepts only the
registered V3 root schemas below; unknown versions, V2 roots, subclasses, and graphs whose
children name a different version are refused without shape inference.
```mermaid
flowchart LR
Benchmark[Benchmark workflow] --> Plan[Reviewed plan]
Plan --> Feasibility[Exact scope feasibility]
Benchmark --> Observations[Sealed observations]
Feasibility --> Authority[Admission authority]
Observations --> Authority
Authority --> Custody[Shared custody transition]
Custody --> Release[V3 case release]
Release --> Reports[Scoped arm reports]
Reports --> Replay[Complete evaluation replay]
Replay --> Consumer[Consumer authorization]
```
Dependency direction:
```text
canonical, review, custody, readiness, evidence, proof, coverage, budget
→ evaluation_measurement
→ evaluation_plan
→ evaluation_feasibility
→ evaluation_observation
→ evaluation_authority
→ evaluation_release
→ evaluation_metrics
→ evaluation_report
→ evaluation_replay
→ evaluation_admission facade
→ admission_fitted V3 consumer
```
### File Structure Plan
New files:
- `src/openultrasast/core/_metric_algebra.py` — version-neutral closed metric algebra
and Wilson arithmetic, created only after V2 golden fixtures lock current behavior;
contains no schema, serializer, aggregate root, or version dispatch.
- `src/openultrasast/core/evaluation_measurement.py` — V3 arm/scope keys, arm
configuration and baseline identities, exact denominator policies, scoped thresholds,
sufficiency/precision rules, scoped metric results, and non-authorizing aggregates.
- `src/openultrasast/core/evaluation_plan.py` — non-circular plan declaration, expected
observation slots, component/combined arm plans, V3 census, and independently reviewed
`EvaluationPlanManifestV3`.
- `src/openultrasast/core/evaluation_feasibility.py` — per-arm/per-scope threshold,
minimum-size, and interval-precision witnesses and shared `may_spend` derivation.
- `src/openultrasast/core/evaluation_observation.py` — sealed observation references,
reviewed resolver-backed attempt history, retry/disposition evidence, and
`EvaluationAdmissionManifestV3`.
- `src/openultrasast/core/evaluation_authority.py` — post-observation revalidation,
`EvaluationAdmissionAuthorityV3`, shared preparation, and all-arm spend binding.
- `src/openultrasast/core/evaluation_release.py` — arm-aware released-case bindings,
proof-candidate continuity, and independently reviewed `AdmissionCaseReleaseV3`.
- `src/openultrasast/core/evaluation_metrics.py` — exact-scope case validation and metric
derivation, ordinary ecosystem projection, whole-push deduplication, and combination
preservation.
- `src/openultrasast/core/evaluation_report.py` — completed per-scope decisions, per-arm
operational evidence, whole-population coverage, degradation reasons, and aggregates.
- `src/openultrasast/core/evaluation_replay.py` — complete V3 graph construction and full
authoritative replay validation.
- `src/openultrasast/core/evaluation_admission.py` — sole public V3 facade for preflight,
authorization, score/replay, strict decoding, and replay validation.
Modified files:
- `src/openultrasast/core/admission_fitted.py` — add a separate V3 consumer function that
accepts only a complete replay plus one exact arm/scope key; keep the V2 function exact.
- `src/openultrasast/core/__init__.py` — expose the V3 facade without changing V2 names.
- `tests/fixtures/detection_core/admission_v2/` — frozen accepted bytes, digests,
decisions, validation failures, and replay outputs.
- `tests/test_core_admission_v2_compatibility.py` and
`tests/test_core_evaluation_*.py` — V2 stability and V3 contract/conformance suites.
### System Flow
```mermaid
sequenceDiagram
participant Run as Benchmark workflow
participant Core as Detection core V3
participant Store as Observation resolver
participant Custody as Custody writer
participant Reviewer as Independent reviewer
Run->>Reviewer: Label blind plan payload
Reviewer-->>Run: Plan attestation
Run->>Core: EvaluationPlanManifestV3
Core-->>Run: EvaluationFeasibilityV3
Run->>Run: Execute every planned arm label blind
Run->>Store: Seal every expected observation
Run->>Reviewer: Plan observations and disposition payload
Reviewer-->>Run: Admission attestation
Run->>Core: EvaluationAdmissionManifestV3
Core->>Store: Resolve exact nonzero observations
Core-->>Run: EvaluationPreparationV3
Run->>Custody: Compare and swap shared request
Custody-->>Run: One spend receipt and labels
Run->>Reviewer: Arm case release payload
Reviewer-->>Run: V3 release attestation
Run->>Core: Receipt release and every arm case
Core-->>Run: EvaluationReplayV3
```
Any insufficient arm/scope ends before execution/spend. Missing, changed, unreadable, or
unsealed observations end before preparation. After spend, missing or invalid cases,
denominator mismatch, or incomplete arms invalidate the evaluation and cannot be reported
as insufficiency, GO, or NO-GO. The custody writer remains external and releases labels
only to the exact CAS winner.
### Requirements Traceability
| Requirement | Design realization |
|---|---|
| 16.1, 16.2, 16.3, 16.4 | Separate V3 facade, exact schema/type validators, no conversion, frozen V2 compatibility suite |
| 16.5 | Plan declaration plus per-arm V3 baseline binds every listed identity and changes on any plan/configuration mutation |
| 16.6 | Typed preflight/impairment/aggregate/report artifacts have no consumer-authority interface; only complete replay authorizes |
| 17.1, 17.2 | Independently reviewed plan declaration, arm plans, component/combined lineage, and expected slots |
| 17.3, 17.4 | Complete arm/scope feasibility product and shared `may_spend`; insufficiency prevents preparation |
| 17.5, 17.6 | Independently reviewed admission manifest and exact resolved observation-slot equality |
| 17.7, 17.8, 17.9 | One authority, preparation, CAS request/receipt, and all-arm spend binding; no subset construction |
| 17.10, 17.11, 17.12 | Reviewed sealed/development/spent disposition and exact retry identity; only sealed unchanged retries validate |
| 18.1, 18.2, 18.3, 18.4 | Full metric set for each `ArmScopeKey`; exact roles/scopes; ecosystem-only ordinary projection; exact precision |
| 18.5 | `underlying_push_digest` deduplicates whole coverage independently from scoped projections |
| 18.6, 18.7 | Scoped metric numerator/denominator/sample/Wilson fields and aggregate type without gate/decision |
| 18.8 | Post-spend zero or census mismatch is validation failure before report construction |
| 19.1, 19.2, 19.3 | Independently reviewed arm-aware V3 release bound to plan, observations, authority and shared spend; V2 type refused |
| 19.4, 19.5 | Exact component observations/reports and deterministic combination lineage are mandatory and contradiction-checked |
| 19.6, 19.7, 19.8, 19.9, 19.10 | Pre-release candidate is non-authoritative; post-release proof matches exact candidate/binding; no transfer or mismatch |
| 19.11 | Planned-arm/report cardinality equality; no result can mask another arm |
| 20.1, 20.2 | Completed result per arm/scope with GO/NO-GO; insufficiency, impairment, incomplete and invalid remain distinct |
| 20.3 | Per-arm scoped metrics plus separate whole coverage, completed/deferred work, funnels, failures, runtime and degradations |
| 20.4, 20.5, 20.6, 20.7 | Closed replay binds and rederives the full graph; missing components refuse completion; only replay authorizes |
| 20.8 | Each arm/scope requires nonzero read receipts and required-stage execution evidence |
### Components and Interfaces
#### V3 identity, plan, and feasibility
`ArmScopeKey` is the natural key for every scoped policy, threshold, metric, gate, and
decision. An arm configuration carries the actual immutable policy objects and the
identities listed in 16.5. `EvaluationPlanDeclarationV3` is hashed before arm baselines are
constructed, avoiding a digest cycle. Each baseline binds the declaration digest and one
arm-configuration digest.
```python
@dataclass(frozen=True, order=True)
class ArmScopeKey:
arm_id: str
scope: ScopeKey
class ArmKindV3(Enum):
COMPONENT = "component"
COMBINED = "combined"
@dataclass(frozen=True)
class ScopedMetricRuleV3:
key: ArmScopeKey
metric_id: MetricId
numerator: NumeratorKind
denominator: DenominatorKind
rule_digest: str
@dataclass(frozen=True)
class ScopedThresholdV3:
key: ArmScopeKey
metric_id: MetricId
statistic: Statistic
comparator: Comparator
target: float
threshold_digest: str
@dataclass(frozen=True)
class IntervalPrecisionRuleV3:
metric_id: MetricId
maximum_wilson_width: float
minimum_denominator: int
rule_digest: str
@dataclass(frozen=True)
class DenominatorPolicyV3:
key: ArmScopeKey
rules: tuple[ScopedMetricRuleV3, ...]
interval_method: IntervalMethod
policy_digest: str
@dataclass(frozen=True)
class ScopeSufficiencyRuleV3:
key: ArmScopeKey
minimum_pairs: int
minimum_ordinary_pushes: int
precision_rules: tuple[IntervalPrecisionRuleV3, ...]
rule_digest: str
@dataclass(frozen=True)
class ExactScopeCensusV3:
scope: ScopeKey
pair_count: int
vulnerable_side_count: int
fixed_side_count: int
semantic_site_capacity: int
model_call_capacity: int
@dataclass(frozen=True)
class EcosystemOrdinaryCensusV3:
ecosystem: str
ordinary_push_count: int
@dataclass(frozen=True)
class PopulationCensusV3:
population_digest: str
scope_rows: tuple[ExactScopeCensusV3, ...]
ordinary_rows: tuple[EcosystemOrdinaryCensusV3, ...]
whole_underlying_push_count: int
labels_exposed: bool
payload_digest: str
review: IndependenceAttestation
census_digest: str
class CombinationOperatorV3(Enum):
DECLARATIVE_TABLE = "declarative_table"
class ProofCandidatePolicyV3(Enum):
IDENTICAL_COMPONENT_ONLY = "identical_component_only"
@dataclass(frozen=True)
class CombinationDecisionRowV3:
component_outcomes: tuple[NeutralOutcome, ...]
combined_outcome: NeutralOutcome
row_digest: str
@dataclass(frozen=True)
class CombinationRuleV3:
combined_arm_id: str
component_arm_ids: tuple[str, ...]
operator: CombinationOperatorV3
decision_rows: tuple[CombinationDecisionRowV3, ...]
candidate_policy: ProofCandidatePolicyV3
rule_digest: str
@dataclass(frozen=True)
class CombinationLineageV3:
rule: CombinationRuleV3
component_observation_digests: tuple[str, ...]
retained_proof_candidate_digests: tuple[str, ...]
lineage_digest: str
@dataclass(frozen=True)
class ArmConfigurationV3:
arm_id: str
kind: ArmKindV3
exact_scopes: tuple[ScopeKey, ...]
denominator_policies: tuple[DenominatorPolicyV3, ...]
thresholds: tuple[ScopedThresholdV3, ...]
sufficiency_rules: tuple[ScopeSufficiencyRuleV3, ...]
judgment_vocabulary_digest: str
analyzer_identity: str
fact_set_digest: str
configuration_digest: str
registered_budget_digest: str
execution_identity: str
outcome_mapping_digest: str
combination_rule: CombinationRuleV3 | None
configuration_identity_digest: str
@dataclass(frozen=True)
class ExpectedObservationSlotV3:
slot_id: str
arm_id: str
input_identity_digest: str
execution_identity: str
slot_digest: str
@dataclass(frozen=True)
class EvaluationPlanDeclarationV3:
population_digest: str
freeze_digest: str
input_set_digest: str
arm_configurations: tuple[ArmConfigurationV3, ...]
expected_observation_slots: tuple[ExpectedObservationSlotV3, ...]
declaration_digest: str
@dataclass(frozen=True)
class EvaluationBaselineIdentityV3:
arm_id: str
plan_declaration_digest: str
population_digest: str
freeze_digest: str
input_set_digest: str
arm_configuration_digest: str
baseline_digest: str
@dataclass(frozen=True)
class EvaluationPlanManifestV3:
declaration: EvaluationPlanDeclarationV3
baselines: tuple[EvaluationBaselineIdentityV3, ...]
census: PopulationCensusV3
readiness_digests: tuple[str, ...]
custody_log_digest: str
payload_digest: str
review: IndependenceAttestation
manifest_digest: str
class FeasibilityStatusV3(Enum):
FEASIBLE = "feasible"
INSUFFICIENT = "insufficient"
@dataclass(frozen=True)
class ThresholdFeasibilityV3:
threshold_digest: str
attainable: bool
witness_numerator: int
witness_denominator: int
witness_digest: str
@dataclass(frozen=True)
class SufficiencyResultV3:
rule_digest: str
minimum_size_passed: bool
interval_precision_passed: bool
reasons: tuple[str, ...]
result_digest: str
@dataclass(frozen=True)
class ScopeFeasibilityV3:
key: ArmScopeKey
threshold_witnesses: tuple[ThresholdFeasibilityV3, ...]
sufficiency_results: tuple[SufficiencyResultV3, ...]
status: FeasibilityStatusV3
reasons: tuple[str, ...]
result_digest: str
@dataclass(frozen=True)
class EvaluationFeasibilityV3:
plan_manifest_digest: str
rows: tuple[ScopeFeasibilityV3, ...]
may_spend: bool
feasibility_digest: str
```
The feasibility rows equal the canonical product of every arm and its exact claimed scopes.
Every row rederives minimum sizes, threshold reachability, and interval precision from the
reviewed label-blind census. `may_spend` is true only when every row is feasible.
#### Observation authority and atomic spend
```python
class PreSpendDispositionV3(Enum):
SEALED = "sealed"
DEVELOPMENT_OPENED = "development_opened"
SPENT = "spent"
class PublicStatusFieldV3(Enum):
STATUS_CODE = "status_code"
COUNT = "count"
DIGEST = "digest"
@dataclass(frozen=True)
class ObservationAccessPolicyV3:
allowed_public_fields: tuple[PublicStatusFieldV3, ...]
raw_observation_access_allowed: bool
population_diagnostic_access_allowed: bool
policy_digest: str
@dataclass(frozen=True)
class ObservationCaseRefV3:
case_id_digest: str
underlying_push_digest: str
result_digest: str
proof_candidate_digest: str | None
@dataclass(frozen=True)
class ArmObservationSetV3:
slot_digest: str
arm_id: str
input_identity_digest: str
execution_identity: str
registered_budget_digest: str
configuration_digest: str
input_read_evidence_digest: str
stage_execution_evidence_digest: str
cases: tuple[ObservationCaseRefV3, ...]
observation_digest: str
@dataclass(frozen=True)
class SealedObservationRefV3:
slot_digest: str
arm_id: str
input_identity_digest: str
execution_identity: str
registered_budget_digest: str
configuration_digest: str
observation: ContentRef
observation_digest: str
read_receipt_digest: str
ref_digest: str
@dataclass(frozen=True)
class AttemptIdentityV3:
population_digest: str
plan_declaration_digest: str
baseline_digests: tuple[str, ...]
input_set_digest: str
configuration_digests: tuple[str, ...]
registered_budget_digests: tuple[str, ...]
execution_identities: tuple[str, ...]
retry_identity_digest: str
@dataclass(frozen=True)
class AttemptEventV3:
attempt_identity: AttemptIdentityV3
disposition: PreSpendDispositionV3
access_policy: ObservationAccessPolicyV3
raw_observation_access_count: int
population_diagnostic_access_count: int
public_status_projection: ContentRef
predecessor_event_digest: str | None
payload_digest: str
review: IndependenceAttestation
event_digest: str
@dataclass(frozen=True)
class AttemptEventRefV3:
event: ContentRef
event_digest: str
ref_digest: str
@dataclass(frozen=True)
class AttemptHistoryV3:
population_digest: str
events: tuple[AttemptEventRefV3, ...]
head_event_digest: str
history_digest: str
@dataclass(frozen=True)
class EvaluationAdmissionManifestV3:
plan_manifest: EvaluationPlanManifestV3
feasibility_digest: str
observations: tuple[SealedObservationRefV3, ...]
readiness_digests: tuple[str, ...]
custody_log_digest: str
attempt_history: AttemptHistoryV3
payload_digest: str
review: IndependenceAttestation
manifest_digest: str
@dataclass(frozen=True)
class EvaluationAdmissionAuthorityV3:
admission_manifest_digest: str
plan_manifest_digest: str
baseline_digests: tuple[str, ...]
feasibility_digest: str
observation_set_digest: str
readiness_set_digest: str
validation_context_digest: str
custody_log_digest: str
attempt_history_head_digest: str
admission_review_digest: str
authority_digest: str
@dataclass(frozen=True)
class EvaluationPreparationV3:
authority: EvaluationAdmissionAuthorityV3
label_release_request: PopulationUseRequest
planned_baseline_digests: tuple[str, ...]
preparation_digest: str
@dataclass(frozen=True)
class EvaluationSpendBindingV3:
preparation_digest: str
receipt_digest: str
planned_baseline_digests: tuple[str, ...]
binding_digest: str
```
`EvaluationAdmissionManifestV3` is the externally produced, independently reviewed
admission-authority artifact required by 17.5: its review subject is the complete payload
digest, its producer and reviewer differ, and the payload binds the feasible plan, exact
observations, the complete attempt-history head, readiness, and current custody head. The derived
`EvaluationAdmissionAuthorityV3` is core's validated projection of that reviewed artifact;
it carries the exact review digest and cannot exist without it.
The admission manifest must contain exactly one observation reference for every expected
slot. Validation resolves each content reference, proves nonzero bytes, parses its canonical
`ArmObservationSetV3`, and checks arm, input, execution, budget, configuration, slot,
input-read evidence, stage-execution evidence, case references, and observation
identity. Core resolves every `AttemptEventRefV3`, validates its independent review and
nonzero canonical bytes, and requires an exact predecessor chain ending at the declared
head. Every unspent predecessor must carry the same rederived `AttemptIdentityV3`; a changed
population, plan, baseline set, input set, configuration, registered budget, or execution
identity requires a terminal development-opened or spent event. The access policy must
prohibit raw-observation and population-diagnostic access, both access counts must be zero,
and the resolved public projection may contain only closed status codes, counts, and
digests. Any nonzero access count likewise requires a terminal development-opened or spent
event. Terminal dispositions are monotonic, so no later sealed event can restore admission
eligibility. Only a complete all-sealed identical-identity chain can authorize. The derived
authority and replay both bind the history head; replay re-resolves the full chain. The
authority also binds the independent admission-review digest and cannot be produced from a
self-reviewed or unattested payload.
The underlying `PopulationUseRequest` names the canonical evaluation-plan identity in its
legacy `arm_id` field and the V3 authority digest as its influenced artifact. The authority
transitively binds every baseline; `EvaluationSpendBindingV3` makes the relationship to the
one exact receipt explicit. Core never performs the append or releases labels.
#### V3 case release, combinations, and proof continuity
```python
@dataclass(frozen=True)
class ProofCandidateIdentityV3:
component_arm_id: str
observation_digest: str
family_id: str
scope: ScopeKey
finding_location_digest: str
change_input_digest: str
candidate_digest: str
@dataclass(frozen=True)
class ReleasedArmCaseBindingV3:
key: ArmScopeKey
baseline_digest: str
observation_slot_digest: str
observation_digest: str
underlying_push_digest: str
case_id_digest: str
role: AdmissionCaseRole
pair_id_digest: str | None
case_result_digest: str
evidence_input_digests: tuple[str, ...]
change_input_digests: tuple[str, ...]
input_manifest_digest: str
proof_candidate_digest: str | None
binding_digest: str
@dataclass(frozen=True)
class AdmissionCaseReleaseV3:
plan_manifest_digest: str
admission_manifest_digest: str
authority_digest: str
preparation_digest: str
spend_binding_digest: str
bindings: tuple[ReleasedArmCaseBindingV3, ...]
payload_digest: str
review: IndependenceAttestation
release_digest: str
@dataclass(frozen=True)
class ArmAdmissionCaseResultV3:
key: ArmScopeKey
observation_slot_digest: str
observation_digest: str
underlying_push_digest: str
case_id_digest: str
role: AdmissionCaseRole
pair_id_digest: str | None
evidence_inputs: tuple[EvidenceValidationInput, ...]
claims: tuple[DefectClaim, ...]
satisfactions: tuple[BoundedSatisfaction, ...]
silence: tuple[BoundedSilence, ...]
proof_candidate: ProofCandidateIdentityV3 | None
proof_attachments: tuple[ProofAttachment, ...]
coverage: CoverageFunnel
input_manifest: InputManifest
input_receipts: tuple[InputReadReceipt, ...]
complexity: ChangeComplexity
budget: BudgetDerivation
total_reply_ms: int
result_digest: str
@dataclass(frozen=True)
class ArmAdmissionCasesV3:
arm_id: str
baseline_digest: str
cases: tuple[ArmAdmissionCaseResultV3, ...]
cases_digest: str
```
The release is the sole post-spend authority for arm, scope, role, pair, result, evidence,
inputs, and proof-candidate association. It is an exact V3 type; V2 release values and
subclasses are refused. Component arms retain their own cases, metrics, gates, and reports.
A combined arm's `CombinationLineageV3` names every component arm, observation digest,
combination rule, and retained candidate. The rule is a finite canonical decision table
over normalized component observation outcomes with a fixed identical-candidate-only proof
policy. Core replays the table; arbitrary combination code is not accepted. Missing, extra,
replaced, or contradictory component material invalidates the combined evaluation.
A proof candidate is non-authoritative before release. After release, the existing proof
validator applies only when family, scope, pair roles, inputs, location, differential,
control, producer, verification receipt, and trust policy match the released candidate.
Absent or inconclusive proof preserves not-demonstrated status. A combined arm can retain
only the identical candidate from a named component observation.
#### Exact metrics, reports, and complete replay
```python
@dataclass(frozen=True)
class ScopedMetricResultV3:
key: ArmScopeKey
metric_id: MetricId
numerator: int
denominator: int
sample_count: int
value: float | None
interval_method: IntervalMethod
interval_lower: float | None
interval_upper: float | None
@dataclass(frozen=True)
class ScopeAdmissionResultV3:
key: ArmScopeKey
metrics: tuple[ScopedMetricResultV3, ...]
instrument_gate: GateResult
budget_gate: GateResult
readiness_gate: GateResult
quality_gate: GateResult
decision: AdmissionDecision
result_digest: str
class DegradationReasonV3(Enum):
INSTRUMENT = "instrument"
MODEL = "model"
BUDGET = "budget"
COVERAGE = "coverage"
DEFERRED = "deferred"
@dataclass(frozen=True)
class ArmOperationalEvidenceV3:
completed_count: int
deferred_count: int
completed_case_digests: tuple[str, ...]
deferred_case_digests: tuple[str, ...]
stage_funnel: CoverageFunnel
instrument_failures: tuple[str, ...]
model_failures: tuple[str, ...]
runtime_ms: int
degradation_reasons: tuple[DegradationReasonV3, ...]
evidence_digest: str
@dataclass(frozen=True)
class WholePopulationCoverageV3:
underlying_push_count: int
completed_underlying_push_count: int
deferred_underlying_push_count: int
coverage_digest: str
@dataclass(frozen=True)
class ArmEvaluationReportV3:
baseline: EvaluationBaselineIdentityV3
scoped_results: tuple[ScopeAdmissionResultV3, ...]
operational_evidence: ArmOperationalEvidenceV3
combination_lineage: CombinationLineageV3 | None
report_digest: str
@dataclass(frozen=True)
class AggregateWeightV3:
key: ArmScopeKey
numerator: int
denominator: int
weight_digest: str
@dataclass(frozen=True)
class EvaluationAggregateV3:
constituents: tuple[ArmScopeKey, ...]
sample_counts: tuple[int, ...]
weights: tuple[AggregateWeightV3, ...]
metrics: tuple[AggregateMetricV3, ...]
aggregate_digest: str
@dataclass(frozen=True)
class AggregateMetricV3:
metric_id: MetricId
weighted_value: float
constituent_metric_digests: tuple[str, ...]
metric_digest: str
@dataclass(frozen=True)
class EvaluationReplayV3:
plan_manifest: EvaluationPlanManifestV3
feasibility: EvaluationFeasibilityV3
admission_manifest: EvaluationAdmissionManifestV3
authority: EvaluationAdmissionAuthorityV3
preparation: EvaluationPreparationV3
receipt: PopulationUseReceipt
spend_binding: EvaluationSpendBindingV3
case_release: AdmissionCaseReleaseV3
cases_by_arm: tuple[ArmAdmissionCasesV3, ...]
reports: tuple[ArmEvaluationReportV3, ...]
combination_lineages: tuple[CombinationLineageV3, ...]
attempt_history_head_digest: str
whole_population_coverage: WholePopulationCoverageV3
aggregate: EvaluationAggregateV3 | None
replay_digest: str
```
Vulnerable/fixed recall and pair metrics use only independently verified cases from the
exact `ArmScopeKey`. Precision uses only findings and cases assigned to the same key.
An eligible ordinary push projects once into each family claimed by that arm in the push's
reviewed ecosystem and never into another ecosystem. Whole coverage deduplicates by
`underlying_push_digest`. Each standard metric carries numerator, denominator, sample count,
value, and Wilson 95% bounds. A post-spend zero or census mismatch is invalid and prevents
report construction.
`EvaluationAggregateV3` deliberately has no gate or decision. Its weights are canonical
non-negative rational values, sum exactly to one by integer cross-multiplication, and correspond one-to-one with its
constituents and sample counts. A replay is constructible
only when its report identities exactly equal all planned arms, every arm report contains
all exact scopes, and every combined lineage preserves its components. Replay validation
re-resolves current trust/custody state and rederives identities, counts, metrics, gates,
decisions, reports, and replay equality.
#### Public service interface
```python
def preflight_evaluation(
manifest: EvaluationPlanManifestV3,
context: AdmissionValidationContext,
resolver: ContentResolver,
) -> EvaluationFeasibilityV3: ...
def authorize_evaluation(
manifest: EvaluationAdmissionManifestV3,
feasibility: EvaluationFeasibilityV3,
context: AdmissionValidationContext,
custody: PopulationCustodyPort,
resolver: ContentResolver,
) -> EvaluationPreparationV3: ...
def score_and_replay(
admission_manifest: EvaluationAdmissionManifestV3,
feasibility: EvaluationFeasibilityV3,
preparation: EvaluationPreparationV3,
receipt: PopulationUseReceipt,
release: AdmissionCaseReleaseV3,
cases_by_arm: tuple[ArmAdmissionCasesV3, ...],
context: AdmissionValidationContext,
custody: PopulationCustodyPort,
resolver: ContentResolver,
) -> EvaluationReplayV3: ...
def validate_evaluation_replay(
replay: EvaluationReplayV3,
context: AdmissionValidationContext,
custody: PopulationCustodyPort,
resolver: ContentResolver,
) -> tuple[ArmEvaluationReportV3, ...]: ...
def authorize_fitted_component_v3(
evidence: FittedComponentEvidence,
probe_inputs: tuple[ProbeEvaluationInput, ...],
fitting_lineage: LineageManifest,
repository_holdout: RepositoryHoldoutReport,
replay: EvaluationReplayV3,
key: ArmScopeKey,
context: AdmissionValidationContext,
custody: PopulationCustodyPort,
resolver: ContentResolver,
) -> None: ...
```
Preconditions require exact V3 root types and canonical ordering. The fitted consumer first
validates the complete replay, selects exactly one planned arm/scope, requires its completed
GO result, and refuses a raw arm report, aggregate, V2 replay, incomplete graph, or ambiguous
selection. It then performs the unchanged fitted-component checks: it replays every probe,
validates fitting lineage, proves the resolved training and repository-holdout populations
are disjoint, binds the selected model declaration and component, and compares derived probe
digests. V3 replaces only the admission evidence source; it does not weaken 13.2.
### Error and Result Semantics
- Pre-spend insufficiency is returned as `EvaluationFeasibilityV3` with
`may_spend=False`; no preparation can be constructed.
- Unreadable observation bytes, unknown/mixed versions, stale custody, disposition change,
or incomplete slots are validation errors before spend.
- After spend, missing cases/reports, invalid release/proof, denominator zero/mismatch,
skipped stages, or unread inputs invalidate completion. They do not become insufficiency
or quality decisions.
- GO/NO-GO exists only inside completed `ScopeAdmissionResultV3` objects.
- Aggregate and per-arm report objects have no consumer-authorization method.
### Testing Strategy
- V2 compatibility: freeze representative canonical bytes, digests, decisions, failures,
and replay; prove helper extraction and V3 imports do not change them (`16.1–16.4`).
- Version/identity: reject V2-as-V3, unknown schemas, subclasses, mixed children, and every
single-field baseline mutation; require a new baseline (`16.3–16.6`).
- Plan/feasibility: cover each arm/scope Cartesian row, threshold reachability,
minimum-size and interval precision, one insufficient row blocking all spend, and no arm
launch or label read (`17.1–17.4`).
- Observation/authority: mutate, remove, swap, truncate, zero, or cross-bind every slot;
reject self-review, stale custody, broken/forked attempt history, raw-access disposition,
changed predecessor retry identity, and any sealed event after a terminal event
(`17.5–17.12`).
- Atomic spend: prove one request/receipt/spend binding covers all planned baselines,
concurrent losers receive no labels, and no subset preparation is constructible
(`17.7–17.9`).
- Exact metrics: exercise multi-family fixes, same-ecosystem ordinary projection,
cross-ecosystem refusal, exact precision, Wilson fields, whole-push deduplication,
aggregate weights/no decision, and post-spend zero/mismatch invalidity (`18.1–18.8`).
- Release/combinations/proof: reject V2 release, role/pair/scope/arm/input/result swaps,
missing components, contradictory combined results, candidate fabrication/transfer, and
invalid attachments; retain absent proof as not demonstrated (`19.1–19.11`).
- Reports/replay: derive every scoped GO/NO-GO, preserve whole operational evidence,
reject missing gate/arm/scope/custody/release/lineage, rederive all source values, and
authorize consumers only from complete replay (`20.1–20.8`).
- Synthetic end to end: two component arms plus one combined arm over at least two families
and two ecosystems pass plan review, per-scope preflight, sealed observations, one shared
spend, V3 release, scoped reporting, complete replay, and exact fitted selection.
### Migration Strategy
1. Freeze V2 accepted outputs and failure behavior.
2. Add V3 measurement identities, plan declaration, exact policies, and preflight.
3. Add sealed observations, reviewed admission manifest, authority, and spend binding.
4. Add V3 release, exact metrics, reports, and complete replay.
5. Add the separate fitted-component V3 consumer and strict schema router.
6. Run synthetic multi-arm conformance and independent validation.
7. Unblock valid-detection-benchmark task generation only after the predecessor schema
`openultrasast.detection-core.evaluation-admission.v3` is exported and validated.
There is no V2-to-V3 migration. Existing V2 artifacts remain replayable under V2; every V3
evaluation starts a new baseline. Rollback removes the additive V3 facade/modules and leaves
the frozen V2 path unchanged.
## Data Models
### Domain Model and Invariants
.kiro/specs/detection-core/requirements.md
+94 -4
View File
@@ -23,20 +23,32 @@ object and decision vocabulary and may use a bounded model program to answer res
questions over that evidence. A staged readiness contract shall stop a family at the first
failing semantic layer, before population-level admission is attempted.
**Evaluation-admission V3 amendment, 2026-11-06:** the validated version-two admission
contract pools all claimed scopes and represents one arm. The approved replacement benchmark
requires a new contract that evaluates multiple component and combined arms under one population
spend while preserving an independently interpretable result for every exact vulnerability-family
and ecosystem scope. Existing version-two evidence remains historical and unchanged.
## Boundary Context
- **In scope**: the family-neutral evidence, outcome, trace, coverage, provenance, budget,
staged-readiness, admission, measurement, fact-pack, model-handoff, and consumer
contracts; family conformance requirements; failure-visible semantic-stage accounting.
contracts; family conformance requirements; failure-visible semantic-stage accounting;
and the versioned exact-scope, multi-arm evaluation-admission validation and replay contract.
- **Out of scope**: vulnerability-family detection algorithms and prompt content;
framework-specific fact content; model optimization; analysis of vendored dependency
source; the pre-push transaction and rollout policy; proof execution; enabling a new
vulnerability family for users.
vulnerability family for users; population sourcing, curation, reservation, licensing,
and lifecycle; durable object-store or compare-and-swap adapters and bucket administration;
detector execution or isolation; observation storage, access enforcement, and retry
orchestration; workflow heartbeat and status presentation; and operator notifications.
- **Adjacent expectations**: the existing deterministic analysis is an evidence producer,
not a required complete prover; family-specific DSPy programs may consume the evidence
packet under the model-handoff contract; proof-based blocking remains a separate
consumer. A parked arm remains parked unless a separately reviewed strategy creates a
new arm, baseline, and admission population.
consumer. Benchmark-owned workflows supply the reviewed population, lifecycle, custody,
sealed-observation, access-control, and post-spend release evidence that the core validates;
the core does not create or operate those workflows. A parked arm remains parked unless a
separately reviewed strategy creates a new arm, baseline, and admission population.
## Requirements
@@ -292,6 +304,80 @@ The detection core shall require one fixed measurement schema for every detectio
5. When a later-stage metric is reported, the detection core shall include the results and
digests of every prerequisite readiness stage.
## Evaluation-Admission V3 Amendment
Requirements 16–20 extend Requirements 12 and 15 for a new evaluation-admission contract. Requirements 1–15 and completed version-two evidence retain their existing meaning, identities, and validation behavior; the amendment does not reinterpret or upgrade them.
### Requirement 16: Versioned evaluation-admission compatibility
The detection core shall provide a new evaluation-admission contract for exact-scope, multi-arm qualification without changing the meaning or validation of historical admission artifacts.
#### Acceptance Criteria
1. When a new evaluation admission is requested, the detection core shall validate it under a distinct contract version and shall keep every earlier admission contract independently replayable.
2. The detection core shall preserve the accepted bytes, digests, decisions, and replay behavior of existing version-two admission artifacts.
3. If pooled version-two evidence is presented as exact-scope version-three evidence, the detection core shall refuse the conversion and shall require a new baseline evaluated under the new contract.
4. If an admission artifact names an unknown, mixed, or incompatible contract version, the detection core shall refuse it rather than infer a compatible shape.
5. When a population, evaluation plan, arm identity, claimed scope, denominator policy, threshold set, judgment vocabulary, analyzer identity, fact identity, configuration identity, registered budget, execution identity, outcome mapping, or combination rule changes, the detection core shall require a new baseline identity.
6. The detection core shall prevent an insufficient result, impaired result, aggregate result, incomplete arm result, raw report, or other partial artifact from authorizing consumer use.
### Requirement 17: Reviewed multi-arm plan and atomic spend authority
The detection core shall bind every arm evaluated against one untouched population to a reviewed label-blind plan and one atomic admission spend.
#### Acceptance Criteria
1. When an evaluation plan is registered before label access, the detection core shall require a reviewed label-blind plan naming the population, input-set identity, every component arm, every combined arm, and every expected observation slot.
2. When an arm enters an evaluation plan, the detection core shall require its immutable baseline identity, exact claimed family-and-ecosystem scopes, denominator policy, thresholds, sufficiency rules, outcome mapping, and combination lineage where applicable.
3. When the label-blind plan is reviewed, the detection core shall evaluate threshold reachability and evidence sufficiency independently for every planned arm and exact claimed scope.
4. If any planned arm or claimed scope cannot meet its registered minimum-size, threshold-reachability, or interval-precision rule, the detection core shall report that arm and scope as insufficient and shall refuse the shared population spend.
5. When every planned observation is complete and before label access, the detection core shall require an admission authority reviewed independently from its producer that binds the feasible plan to the exact completed observation identities and the current population custody state.
6. If an expected observation is absent, unreadable, changed, associated with another arm, input, or plan, or inconsistent with the feasible plan, the detection core shall refuse admission authorization.
7. When shared labels are released, the detection core shall require one admission preparation, one exact monotonic population transition, and one resulting label-release receipt binding every planned arm.
8. If the custody head is stale, the transition is not the exact available-to-spent successor, the receipt is absent or mismatched, or availability is later restored, the detection core shall refuse label release and scoring.
9. The detection core shall not authorize a partial spend for a subset of planned arms or scopes.
10. While the population remains unspent, the detection core shall permit admission reuse only when raw observations and population-specific diagnostics remain unavailable to detector development and only closed process status, counts, and digests are exposed.
11. If raw observations or population-specific diagnostics are inspected, or an arm, plan, configuration, registered budget, or execution identity changes after an attempt, the detection core shall require a development or spent disposition and shall refuse that population for admission.
12. When a pre-spend attempt is retried, the detection core shall accept the retry only under the exact unchanged population, plan, arms, inputs, configuration, registered budgets, and execution identities.
### Requirement 18: Exact-scope denominators and whole-population coverage
The detection core shall derive qualification independently for each arm's exact vulnerability-family and ecosystem scope while preserving separate whole-population coverage.
#### Acceptance Criteria
1. When quality is evaluated, the detection core shall derive one complete standard metric set for every exact arm, vulnerability-family, and ecosystem scope declared in the reviewed plan.
2. When recall or pair correctness is derived for a scope, the detection core shall use only vulnerable and fixed cases independently verified for that exact family and ecosystem.
3. When ordinary false alarms are derived for an arm, the detection core shall count each eligible ordinary push once for every family claimed by that arm in the push's reviewed ecosystem and shall not project it into another ecosystem.
4. When precision is derived for a scope, the detection core shall use only findings and cases assigned to that exact arm, family, and ecosystem under the reviewed denominator policy.
5. When one underlying push contributes to multiple verified scoped denominators, the detection core shall count it once in whole-population coverage.
6. When a scoped metric is reported, the detection core shall include its numerator, denominator, value, sample count, and Wilson 95% interval under the fixed metric algebra.
7. Where an overall aggregate is reported, the detection core shall identify its constituent arms and scopes, sample counts, and weighting and shall not attach a gate or authorization decision to the aggregate.
8. If a post-spend scoped denominator is zero or differs from the reviewed label-blind census and feasible plan, the detection core shall refuse the evaluation as invalid rather than report insufficient, go, or no-go.
### Requirement 19: Version-three case release, arm results, and proof binding
The detection core shall validate post-spend evidence separately for every planned arm while preserving shared population custody and combination lineage.
#### Acceptance Criteria
1. When post-spend cases are released, the detection core shall require one version-three case-release graph reviewed independently from its producer and bound to the reviewed evaluation plan, shared spend, and exact completed observations.
2. When a case is released for an arm, the detection core shall bind its arm identity, exact scope, role, pair identity, result, evidence, change inputs, reviewed input manifest, and release review.
3. If a version-two case release is supplied to a version-three evaluation, the detection core shall refuse it rather than reinterpret it under the new contract.
4. When a combined arm is evaluated, the detection core shall preserve every component arm's observations, metrics, gates, and decision and shall report the registered combination lineage.
5. If a combined result omits, replaces, or contradicts a required component result, the detection core shall refuse the combined evaluation.
6. While ground-truth roles remain sealed, the detection core shall treat every label-blind proof candidate as non-authoritative for demonstrated status.
7. When a proof-backed status is derived after case release, the detection core shall require an exact validated proof attachment matching the released family, scope, pair roles, change inputs, finding location, differential record, control binding, producer, verification receipt, and trust policy.
8. When a combined arm retains proof evidence, the detection core shall require the identical validated proof candidate from the matching component observation and shall prevent the combined arm from creating or transferring proof evidence.
9. If proof is absent or inconclusive, the detection core shall retain the finding as not demonstrated.
10. If a proof attachment is role-swapped, cross-family, cross-scope, location-mismatched, or bound to unrelated inputs or controls, the detection core shall refuse the attachment as invalid and shall not count it as demonstrated.
11. The detection core shall not allow one planned arm's valid result to hide another planned arm's missing, invalid, or failed result.
### Requirement 20: Complete evaluation reports and replay authority
The detection core shall make the complete multi-arm evaluation replay the only admission artifact capable of authorizing downstream consumption.
#### Acceptance Criteria
1. When a planned arm completes evaluation, the detection core shall report one go or no-go decision for every exact claimed scope from that arm's validated instrument, budget, readiness, and quality gates.
2. The detection core shall reserve go and no-go for completed quality evaluations and shall keep pre-spend insufficiency, unread input, skipped stages, incomplete arms, and other instrument failures distinct from quality decisions.
3. When an evaluation report is produced, the detection core shall preserve per-arm scoped metrics separately from whole-population coverage, completed and deferred work, coverage funnels, instrument and model failures, runtime, and every degradation reason.
4. When every planned arm has a valid report, the detection core shall produce one complete evaluation replay binding the reviewed plans, feasibility results, admission authority, shared custody transition and receipt, the version-three case-release graph, independently valid per-arm scoped reports, and combination lineage.
5. If any planned arm report, mandatory gate, custody binding, case release, or replay component is missing or invalid, the detection core shall refuse evaluation completion.
6. When an evaluation replay is validated, the detection core shall rederive identities, counts, metrics, gates, and decisions from the canonical source artifacts rather than trust caller-supplied summaries.
7. When a consumer requests admission evidence, the detection core shall provide authorization only from a complete validated evaluation replay and shall not authorize from a per-arm report or aggregate alone.
8. When a zero or clean-looking arm result is included, the detection core shall require nonzero input-read evidence and execution evidence for every required stage before accepting the replay.
## Out of Scope
- The pre-push hook transaction, rollout policy, and capability registry.
- Vulnerability-family algorithms, prompts, examples, and optimization programs.
@@ -300,3 +386,7 @@ The detection core shall require one fixed measurement schema for every detectio
- Proof execution and sandbox orchestration.
- Enabling a vulnerability family for advisory or blocking use.
- Analysis-engine, frontend, graph, and family-internal implementation details.
- Population sourcing, human curation, permanent reservation, licensing, and lifecycle operations.
- Durable object-store or compare-and-swap adapter implementation, bucket administration, and retention configuration.
- Detector execution and isolation, observation storage and access enforcement, and retry orchestration.
- Workflow heartbeat and operator-status presentation, notification transport, and product rollout.
.kiro/specs/detection-core/research.md
+362
View File
@@ -711,3 +711,365 @@ requirements approval.
legacy imports and then proves those attributes are absent after cutover.
- **Historical artifacts become unreadable:** frozen trace/report bytes and hashes continue
through the independent read-only decoder before and after namespace deletion.
---
## Gap analysis: exact-scope multi-arm evaluation admission V3 (2026-11-06)
### Scope, approval state, and instrument check
- **Feature type:** complex brownfield contract extension. Requirements 16–20 add a new
evaluation-admission root beside the implemented V2 admission graph; they do not replace
the family-neutral evidence contracts in Requirements 1–15.
- **Approval warning:** Requirements 16–20 are generated but not approved. This gap analysis
is permitted to inform review, but `design.md` cannot be regenerated until the requirements
are approved or `/kiro-spec-design detection-core -y` is invoked.
- **Discovery method:** integration-focused source inspection of the admission contracts,
their tests, their only production authorization consumer, and the approved
`valid-detection-benchmark` consumer design. No external dependency research was needed:
the amendment is a standard-library Python contract, validation, and replay change.
- **Instrument proof:** the requirements (32,917 bytes), current design (130,524 bytes),
existing research (45,818 bytes), approved valid-benchmark design (72,402 bytes), ten
primary admission/core modules (166,630 bytes), and the relevant admission/measurement
tests (270,157 bytes) were opened as nonzero inputs before interpreting gaps.
### Current architecture and reusable assets
The implemented V2 admission path is a closed canonical graph:
`BaselineIdentity` → reviewed `AdmissionManifestV2` and census → context validation and
pooled feasibility → one `AdmissionPreparation` → exact `AVAILABLE`-to-`SPENT` custody
receipt → independently reviewed `AdmissionCaseRelease` → revalidated case results →
one `AdmissionReportV2` → one `AdmissionReplay`.
The following machinery is reusable without changing its meaning:
- canonical JSON, content digests, closed-graph checks, exact-dataclass checks, and strict
enum/type validation;
- `ScopeKey`, the fixed metric identifiers and algebra, Wilson 95% computation, readiness
validation, review-independence attestations, content resolution, and provenance checks;
- the append-only custody log and exact monotonic `AVAILABLE`-to-`SPENT` receipt validation;
- case evidence, input, budget, coverage, and proof-attachment validation after a V3 release
establishes the correct arm/scope association;
- V2 report and replay rederivation, which must remain independently replayable and
byte-compatible.
The reusable implementation is not the V2 aggregate root itself. V2 has one arm identity,
assigns every metric rule the full claimed-scope tuple, pools capacities and cases across
those scopes, derives one unscoped result per metric, and emits one arm-level decision. That
shape cannot express the exact per-arm, family, and ecosystem claims required by V3.
### Requirement-to-asset map
| Requirement | Existing asset | Gap classification and impact |
|---|---|---|
| 16.1–16.4 Version isolation | Canonical schemas, exact-type replay validation, historical decoder patterns | **Missing / compatibility constraint:** no V3 root or explicit version dispatcher exists. Extending V2 dataclasses changes accepted bytes and digests; inferred conversion would make pooled evidence appear exact. |
| 16.5 Baseline identity | `BaselineIdentity` binds one arm, population/freeze, scopes, facts, analyzer, configuration, denominator, thresholds, and outcomes | **Missing:** plan, input-set, sufficiency, registered budget, execution, observation-set, and combination-rule identities are absent. These must belong to a distinct V3 identity rather than changing V2. |
| 16.6 Non-authorizing partial artifacts | V2 replay, not a raw report, authorizes fitted-component use | **Partial:** there is no V3 completion root or closed status distinction for insufficiency, impairment, aggregates, incomplete arm results, and raw per-arm reports. |
| 17.1–17.4 Reviewed multi-arm preflight | Independent V2 manifest/census review and threshold feasibility | **Missing:** no reviewed plan containing every component/combined arm and observation slot; no per-arm/per-scope sufficiency rules; V2 feasibility sums capacities across selected scopes and therefore cannot veto the shared spend independently for each exact scope. |
| 17.5–17.6 Post-observation authority | V2 authorization revalidates context, feasibility, readiness, and custody | **Missing:** no second reviewed manifest binding the feasible plan to exact sealed observation identities, no observation-byte resolution, and no all-slot completeness check immediately before spend. |
| 17.7–17.9 Atomic shared spend | Exact V2 preparation and custody successor validation | **Constraint:** V2 `PopulationUseRequest` names one arm. No preparation/receipt binds every planned arm and scope, and no root rejects a partial subset while retaining one shared custody transition. |
| 17.10–17.12 Confidential retry/disposition | Append-only custody and lineage policies | **Missing / adjacent boundary:** core lacks typed retry identity and development/spent disposition validation for raw-observation access. The benchmark owns access enforcement and lifecycle events; core must validate their immutable authority and exact retry identity, not implement storage or access control. |
| 18.1–18.4 Exact scoped metrics | Fixed metric algebra, Wilson intervals, reviewed scope census, validated cases | **Missing:** `standard_metric_rules()` assigns all scopes to every rule; `MetricResult` and `AdmissionThreshold` have no scope key; context requires pooled rule scopes; feasibility and scoring derive once across the union. |
| 18.3, 18.5 Ordinary projection and whole coverage | Case roles and an intermediate whole-population funnel | **Missing:** ordinary cases lack a stable underlying-push identity that permits once-per-family projection inside the reviewed ecosystem while deduplicating the same push once for whole coverage. Whole-population evidence is not preserved in the final V2 report. |
| 18.6–18.8 Scoped reporting, aggregates, post-spend validity | Metric numerator/denominator/value/Wilson fields | **Partial:** metric arithmetic is reusable, but scope-qualified results and non-authorizing aggregate identities/weights do not exist. Post-spend zero or census mismatch must be invalid, not a zero-valued metric, insufficiency, GO, or NO-GO. |
| 19.1–19.3 V3 case release | Strong independent V2 release with exact role/pair/result/evidence/input binding | **Missing:** `ReleasedCaseBinding` has no arm identity, and the release binds one V2 manifest and authorization. Exact V2/V3 type separation is required. |
| 19.4–19.5 Combination lineage | No equivalent admission root | **Missing:** there is no graph preserving every component observation/report/gate while binding a combined arm to its registered deterministic lineage or detecting omission, replacement, and contradiction. |
| 19.6–19.10 Proof continuity | Strong `ProofAttachment` validation | **Partial:** the proof contract can be reused after V3 release, but no label-blind proof-candidate identity is carried from the component observation to prevent a combined arm from creating or transferring proof evidence. |
| 19.11 Arm independence | Exact V2 replay for one arm | **Missing:** no all-planned-arm cardinality and identity check prevents one valid report from hiding another arm's missing, invalid, or failed result. |
| 20.1–20.2 Scoped decisions and failure classes | V2 mandatory gates and GO/NO-GO derivation | **Missing:** V2 emits one pooled arm decision. V3 needs a completed decision per exact scope and separate non-quality states for insufficiency, unread inputs, skipped stages, impairment, and incomplete arms. |
| 20.3 Complete reports | `MeasurementRecordV2`; intermediate `AdmissionMetricEvidence` carries an extra whole funnel/runtime | **Partial:** completed/deferred work, whole coverage, degradation reasons, and complete instrument failures are not preserved in the final report; `_measurement()` currently supplies `instrument_failures=()`. |
| 20.4–20.7 Complete replay authority | `AdmissionReplay` rederives one V2 report from authoritative inputs | **Missing:** no multi-arm replay binds both reviewed stages, shared feasibility/authority/custody/release, all independently valid scoped reports, and combination lineage. `admission_fitted.py` accepts only V2 `AdmissionReplay` and derives authorization from its single report. |
| 20.8 Nonzero execution evidence | Coverage/input validation and clean-measurement checks | **Partial:** reusable case evidence exists, but V3 completion must require it independently for every arm and exact scope before accepting a clean-looking result. |
### Concrete integration surfaces
The V3 amendment affects these core responsibilities while retaining V2 definitions:
- `src/openultrasast/core/measurement.py`: reusable scope and metric algebra; new V3
baseline, scope-qualified metric/policy/threshold, aggregate, and sufficiency contracts.
- `src/openultrasast/core/admission_review.py`: new reviewed two-stage plan and admission
manifests with canonical per-arm plans and expected/actual observation bindings.
- `src/openultrasast/core/admission_feasibility.py`: independent feasibility witnesses for
every arm and exact scope plus one shared non-spend result.
- `src/openultrasast/core/admission_context.py`: V3 plan, identity, policy, readiness,
observation, and lineage validation without the V2 all-scopes-equal rule.
- `src/openultrasast/core/admission_spend.py`: a distinct V3 authority/preparation binding
every planned arm and the current custody head; V2 functions remain unchanged.
- `src/openultrasast/core/admission_case.py`: an arm-aware V3 case-release graph bound to
the plan, exact observations, authority, and shared receipt.
- `src/openultrasast/core/admission_metrics.py`: exact-scope derivation, ordinary-push
projection within ecosystem, whole-population deduplication, and proof-candidate
continuity.
- `src/openultrasast/core/admission_report.py`: independently valid per-arm/per-scope
reports, non-authorizing aggregates, and explicit non-quality failure classes.
- `src/openultrasast/core/_admission_replay.py`: one closed `EvaluationReplayV3` containing
the complete shared graph and all planned reports.
- `src/openultrasast/core/admission_fitted.py`: explicit version dispatch; V3 authorization
accepts only a complete validated evaluation replay and an exact arm/scope selection,
never a raw report or aggregate.
- `tests/test_core_admission_*.py`, `tests/test_core_measurement*.py`, and frozen V2
fixtures: compatibility, exact-scope algebra, all-arm completeness, mixed-version
refusal, atomic-spend, proof-continuity, and replay conformance.
The approved benchmark remains a downstream consumer. Its current
`benchmarks/valid/admission_run.py`, `admission_scoring.py`, and `population_custody.py`
are V2 single-arm paths and stay compatibility-only. Population lifecycle, durable object
storage/CAS, observation access, detector execution, retry orchestration, and operator
presentation remain benchmark-owned and are not implementation work for this amendment.
### Implementation approach options
#### Option A: extend V2 contracts in place
Add arm/scope/plan fields to existing V2 identities, policies, manifests, releases,
reports, and replay.
- **Strengths:** smallest initial module count and direct reuse of current call paths.
- **Risks / limitations:** changes canonical V2 bytes and digests, makes exact-type version
refusal ambiguous, overloads already dense single-arm modules, and risks accidental
projection of pooled evidence. It conflicts with 16.1–16.4.
- **Effort / risk:** L / High; not a viable compatibility path unless Requirement 16 is
weakened, which is not recommended.
#### Option B: create a fully parallel V3 stack and facade
Introduce an independent `evaluation_admission` facade and V3 modules for identity, plan,
feasibility, authority, release, metrics, reports, and replay.
- **Strengths:** clearest public version boundary, exact-type refusal, and protection of V2
bytes and behavior; components are testable in isolation.
- **Risks / limitations:** duplicates canonical validation, custody, readiness, proof, and
metric plumbing unless reuse is disciplined; two large stacks increase maintenance cost.
- **Effort / risk:** XL / Medium-High.
#### Option C: parallel V3 public roots with version-neutral internal helpers
Keep V2 and V3 aggregate roots, facades, schemas, and validators exact and separate. Reuse
or extract only helpers whose inputs and outputs have no serialized V2 identity, such as
Wilson arithmetic, review validation, custody-chain checks, proof validation, and canonical
collection validation.
- **Strengths:** preserves strict version boundaries and V2 replay while avoiding avoidable
duplication; aligns with the current closed-graph and ports-and-adapters pattern.
- **Risks / limitations:** helper extraction can still change V2 behavior if performed
before compatibility fixtures lock accepted bytes and errors; dependency direction and
exact-type tests must be explicit.
- **Effort / risk:** XL / High.
- **Design-phase candidate:** this offers the best balance, using Option B-style public
separation. It remains a candidate until the design review resolves the research items
below.
### Complexity and risk
- **Effort: XL (2+ weeks).** The amendment adds two-stage review, exact-scope algebra,
multi-arm completeness, one shared spend, a new release graph, per-arm reports, consumer
dispatch, and a closed replay root across multiple existing contract modules.
- **Risk: High.** The main failure modes are silent V2 behavior drift, pooled-to-scoped
misinterpretation, partial-arm authorization, stale observation/custody acceptance,
ordinary-push double counting, and an aggregate or raw report becoming an authorization
token.
### Design-phase recommendations and research needed
1. Freeze representative accepted V2 canonical bytes, digests, decisions, validation
failures, and replay outputs before extracting helpers. Treat any change as a regression.
2. Define one explicit V3 facade/schema family and refuse unknown, mixed, subclassed, or
substituted V2/V3 roots at every public boundary.
3. Resolve the exact V3 module split and aggregate roots: per-arm plan, reviewed plan
manifest, observation binding, admission authority/preparation, V3 release, scoped
report, and evaluation replay.
4. Define the observation-slot identity and content-resolution contract, including how an
observation proves arm, input, execution, budget, configuration, and plan identity
without exposing raw diagnostics before spend.
5. Define a stable underlying-push identity for ordinary projection and whole-population
deduplication, including multi-family fixes and ecosystem review authority.
6. Define sufficiency and interval-precision rules as reviewed per-arm/per-scope contracts,
separate from post-spend invalidity and completed GO/NO-GO gates.
7. Define the benchmark-supplied retry/development disposition evidence that core validates;
do not move durable lifecycle, access enforcement, or CAS implementation into core.
8. Define component-to-combined observation lineage and proof-candidate identity so a
combined arm can retain only the identical component candidate and cannot suppress a
component report.
9. Define V3 fitted-component consumption as complete-replay validation plus exact arm/scope
selection. Per-arm reports, aggregates, incomplete roots, and V2 artifacts cannot
authorize a V3 consumer.
10. Derive the V3 conformance suite from 16.1–20.8: frozen V2 compatibility; unknown/mixed
version refusal; per-arm/per-scope feasibility; missing observation and stale custody
refusal; one atomic spend; exact ordinary projection/dedup; post-spend zero invalidity;
proof continuity; all-arm completion; and nonzero input/stage execution evidence.
### Gap conclusion
The current implementation provides strong canonical validation, custody, evidence,
metric, proof, and replay primitives, but its V2 admission root is structurally unable to
represent an exact-scope multi-arm evaluation. The gap is therefore an additive versioned
contract and replay graph, not a correction to the V2 metric formula. The design should
retain V2 unchanged, introduce exact V3 roots, and reuse only version-neutral internals.
Design generation remains blocked on approval of Requirements 16–20.
---
## Design discovery and synthesis: evaluation-admission V3 (2026-11-06)
### Summary
- **Feature:** detection-core evaluation-admission V3 amendment.
- **Discovery scope:** complex integration and immutable-contract version extension.
- **Selected pattern:** parallel V3 public aggregate roots with reuse limited to
version-neutral canonical, review, custody, proof, readiness, and metric primitives.
- **Key findings:** V2 aggregate roots cannot be generalized without changing their bytes;
the V3 plan needs a non-circular declaration/baseline identity; and one complete replay,
rather than a report or aggregate, must be the only downstream authorization source.
### Research log
#### Version and canonicalization boundary
- **Context:** 16.1–16.4 require historical V2 bytes, digests, decisions, and replay to
remain unchanged while V3 refuses mixed or inferred shapes.
- **Sources consulted:** current V2 canonical and admission modules, frozen-dataclass and
exact-type patterns in the repository, Python 3.11 JSON documentation, and RFC 8785.
- **Finding:** the existing sorted-key compact JSON profile is sufficient for the current
single-Python-runtime contract. Adopting JCS would create a second canonical profile and
risk changing V2 bytes without satisfying an additional requirement.
- **Implication:** `evaluation_admission.py` is the only V3 facade. V3 validators require
exact dataclass types and exact V3 schema constants. A read router may select a validator
by schema but never converts V2 to V3.
#### Non-circular plan and baseline identity
- **Context:** every arm needs an immutable baseline, and a change to the evaluation plan
must start a new baseline without making the plan and baseline digests recursive.
- **Finding:** a reviewed plan declaration can be hashed before baseline construction. It
contains the population, input set, arm configurations, combination rules, and expected
observation slots. Each arm baseline then binds that declaration digest and its exact arm
configuration digest.
- **Implication:** any plan or arm mutation changes the declaration and therefore every arm
baseline, while the plan manifest can embed the derived baselines without a digest cycle.
#### Two-stage review and shared spend
- **Context:** arithmetic feasibility happens before detector execution; spend authority
requires every exact sealed observation and the current custody head afterwards.
- **Finding:** one manifest cannot safely serve both moments. The second authority must
resolve observation bytes and bind the earlier feasible plan rather than repeat or
summarize it.
- **Implication:** `EvaluationPlanManifestV3` supports label-blind preflight;
`EvaluationAdmissionManifestV3` binds exact sealed observations and retry/disposition
evidence. `EvaluationAdmissionAuthorityV3` is constructible only when both validate.
#### Existing custody reuse
- **Context:** 17.7–17.9 require one atomic population transition binding all arms, while
the existing custody graph already validates an exact monotonic transition.
- **Finding:** the existing request/receipt can remain unchanged if its influenced artifact
is the V3 authority digest, which transitively commits to every planned baseline. A V3
spend binding makes that transitive relationship explicit inside the replay.
- **Implication:** no second custody implementation is introduced. The benchmark remains
the CAS writer and label releaser; core builds and validates the authority, preparation,
receipt binding, and replay relationship.
#### Exact-scope projection and whole coverage
- **Context:** one underlying push may contribute to several verified family scopes while
whole-population coverage must count it once.
- **Finding:** case identity alone is insufficient because vulnerable and fixed sides and
several arm/scope projections can refer to the same push.
- **Implication:** V3 release bindings carry a separate `underlying_push_digest` and exact
`ArmScopeKey`. Scoped metrics project from arm/scope bindings; whole coverage deduplicates
only by the underlying push identity. Ordinary projection is limited to families claimed
by the arm in the push's reviewed ecosystem.
#### Combination and proof continuity
- **Context:** a combined arm must preserve component results and cannot manufacture or
transfer demonstrated proof.
- **Finding:** proof validation alone establishes attachment correctness but does not prove
candidate continuity from a component observation to a combined observation.
- **Implication:** label-blind observations and released bindings carry a stable
proof-candidate digest. Combination lineage names exact component observation digests;
the combined arm may retain only an identical candidate named by a component.
### Architecture pattern evaluation
| Option | Description | Strengths | Risks / limitations | Decision |
|---|---|---|---|---|
| Extend V2 roots | Add plan, arm and scope fields to current types | Smallest initial file count | Changes V2 bytes and permits pooled/scoped ambiguity | Rejected |
| Fully parallel V3 implementation | Duplicate the entire validation stack | Strong version separation | Duplicates custody, proof, review and metric machinery | Rejected |
| Parallel V3 roots plus neutral helpers | Separate public schemas and aggregates; share only identity-neutral primitives | Preserves V2 and avoids duplicate trusted logic | Helper extraction needs frozen V2 regression evidence | Selected |
### Design decisions
#### Decision: one V3 facade and exact roots
- `evaluation_admission.py` exports preflight, authorization, score/replay, and replay
validation. V2 admission entry points remain unchanged.
- V3 modules do not import V2 aggregate roots. `admission_fitted.py` exposes a separate V3
consumer function rather than broadening the V2 function's accepted graph.
#### Decision: plan declaration precedes baselines
- `EvaluationPlanDeclarationV3` owns population, freeze, input set, arm configurations,
expected observation slots, and combination declarations.
- `EvaluationBaselineIdentityV3` binds the declaration digest and one arm configuration.
The reviewed manifest embeds both and proves the exact canonical derivation.
#### Decision: insufficiency is a pre-spend result, not a quality decision
- `EvaluationFeasibilityV3` contains exactly one row per declared arm/scope and derives one
`may_spend` value. Any insufficient row makes it false and prevents preparation.
- GO and NO-GO occur only in completed post-spend scope reports. Instrument impairment,
unread inputs, skipped stages, incomplete arms, and invalid post-spend denominators never
map to those decisions.
#### Decision: reports are evidence; the complete replay is authority
- Per-arm scoped reports and optional aggregates are serializable evidence but cannot
authorize consumers.
- `EvaluationReplayV3` binds both reviewed manifests, feasibility, authority, preparation,
shared receipt, spend binding, V3 release, exact cases, all planned reports, and
combination lineage. Validation rederives the graph and exact planned-arm cardinality.
#### Decision: aggregate has no decision field
- An aggregate identifies constituents, sample counts, weights, and derived values. Its
type contains neither gates nor authorization decisions, preventing an aggregate from
being mistaken for an admission result.
### Synthesis
- **Generalization:** expected observation slots, arm/scope keys, and complete replay
cardinality are three applications of one closed-plan principle: later artifacts must be
an exact projection of the reviewed plan, never a caller-selected subset.
- **Build versus adopt:** reuse the existing Python canonical, custody, review, readiness,
proof, and Wilson primitives. Do not adopt a new serializer, workflow engine, object-store
client, or transaction library because persistence and orchestration stay downstream.
- **Simplification:** use the existing custody request/receipt transitively through a V3
authority and explicit spend binding; do not create a second custody state machine.
Keep lifecycle/access enforcement outside core and validate only immutable disposition
evidence supplied by the benchmark.
### Risks and mitigations
- **V2 drift during helper extraction:** freeze canonical V2 bytes, digests, decisions,
failures, and replay outputs before extraction; run them after every shared-helper change.
- **Partial-arm authorization:** derive report and replay cardinality from the plan and make
the complete replay the sole consumer input.
- **Observation substitution:** resolve nonzero bytes and require an exact one-to-one slot,
arm, input, execution, budget, configuration, and digest match before authority.
- **Ordinary-push double counting:** separate scoped projection keys from the deduplicated
underlying-push identity.
- **Combined proof fabrication:** require component observation lineage and identical
proof-candidate identity before accepting a retained attachment.
- **Core absorbing benchmark workflow:** keep storage, CAS, access enforcement, retries,
detector isolation, label joining, status, and presentation explicitly out of boundary.
### References
- [Python 3.11 JSON documentation](https://docs.python.org/3.11/library/json.html) —
deterministic sorted-key compact serialization controls used by the existing profile.
- [RFC 8785](https://www.rfc-editor.org/rfc/rfc8785.html) — evaluated but not adopted;
cross-runtime canonical signatures are outside the current boundary.
- [Approved valid-detection-benchmark design](../valid-detection-benchmark/design.md) —
downstream V3 consumer contract and ownership boundary.
.kiro/specs/detection-core/spec.json
+3 -2
View File
@@ -1,7 +1,7 @@
{
"feature_name": "detection-core",
"created_at": "2026-10-30T13:55:30Z",
"updated_at": "2026-11-02T07:51:51Z",
"updated_at": "2026-11-06T12:21:47Z",
"language": "en",
"phase": "tasks-generated",
"approvals": {
@@ -18,5 +18,6 @@
"approved": true
}
},
"ready_for_implementation": true
"ready_for_implementation": true,
"approval_note": "2026-11-06: Requirements and the merged V3 design were auto-approved by the maintainer's explicit -y request. Additive V3 task groups 12-18 passed the task-plan review and an independent graph sanity review; existing groups 1-11 remain historical V2 context, and blocked task 11.4 is not a V3 prerequisite. Tasks are approved for implementation."
}
.kiro/specs/detection-core/tasks.md
+268
View File
@@ -7,6 +7,7 @@
| Completed groups 1–7 and 8.1 | Retained as regression substrate; their former requirement mappings do not count toward v2 coverage |
| Previous task 8.2 | Permanently parked after admission NO-GO #3; no task below revives that arm |
| Previous tasks 8.3–8.4 | Superseded by v2 consumer and proof conformance work; no rollout or live capability enablement is included |
| Existing task 11.4 | Retained as blocked historical V2 validation; it is not a prerequisite for additive V3 tasks 12–18, which have explicit dependencies and their own compatibility gate |
## Task constraints
@@ -16,6 +17,9 @@
- Historical v1 artifacts remain byte-identical. V2 creates a new baseline and is never trended against v1.
- The parked authorization arm remains parked. Admission orchestration uses synthetic/reference inputs only and does not consume a live population.
- Existing target facts may be schema-migrated without expanding vocabulary or behavior.
- Tasks 12–18 are the current additive V3 implementation scope; completed groups 1–11 remain historical V2 implementation context.
- V2 aggregate roots and public routes remain frozen; V3 uses a distinct schema and adds no converter or inferred projection.
- Benchmark-owned storage, compare-and-swap, execution, access, retry, and workflow orchestration remain outside core and are exercised only through synthetic or in-memory boundaries.
- [ ] 1. Establish migration-safe foundations
- [x] 1.1 Freeze historical v1 artifacts and the reviewed migration inventory
@@ -437,6 +441,269 @@
- Done when the complete suite passes, boundary/import censuses are clean, historical bytes are unchanged, and all declared adapter and consumer seams pass conformance.
- _Requirements: 1.4, 10.4, 14.3, 15.3, 15.4_
- _Depends: 10.7_
- _Blocked: Benchmark population governance conflict: the permanent unseen-pool reservation guard and the later spent/retired training-only pilot policy disagree. The benchmark owner must approve the authoritative lifecycle and replacement/retirement plan before the full suite can pass._
## Evaluation-admission V3 amendment
Tasks 12–18 implement Requirements 16–20 as an additive predecessor for the approved
valid-detection-benchmark design. Existing groups 1–11 remain the historical V2
implementation record. The unresolved downstream-governance item 11.4 is not reopened and
is not a prerequisite for this V3 stream; V3 tasks depend explicitly on completed V2
contract work and use synthetic/in-memory workflow boundaries only.
- [ ] 12. Preserve V2 admission and establish V3 measurement foundations
- [ ] 12.1 (P) Freeze representative V2 admission behavior
- Pin accepted canonical bytes, digests, decisions, validation failures, report output, and authoritative replay output for representative V2 admissions.
- Prove every frozen fixture is readable and nonzero before treating a stable result as compatibility evidence.
- Keep pooled V2 evidence replayable only under V2 and add no converter or inferred V3 projection.
- Done when a byte, digest, decision, failure-order, or replay mutation fails the compatibility suite while the current V2 fixtures pass unchanged.
- _Requirements: 16.1, 16.2, 16.3, 16.4_
- _Boundary: V2 Admission Compatibility_
- _Depends: 8.9, 8.10_
- [ ] 12.2 Extract version-neutral metric arithmetic without changing V2
- Isolate only the closed standard-metric algebra and Wilson calculation; keep every serialized V2 identity and validator in its existing owner.
- Route V2 through the shared arithmetic and compare all frozen V2 outputs byte-for-byte.
- Expose the same pure arithmetic to V3 without accepting a V2 policy, result, or report as a V3 value.
- Done when both versions use one arithmetic implementation and the complete V2 compatibility fixture remains unchanged.
- _Requirements: 16.2, 18.6_
- _Boundary: Version-Neutral Metric Algebra_
- _Depends: 12.1_
- [ ] 12.3 Define exact arm-scope measurement and sufficiency contracts
- Represent one canonical arm-and-scope key for every policy, threshold, sufficiency rule, metric, gate, and decision.
- Bind exact denominator algebra, fixed Wilson intervals, sample counts, minimum sizes, interval precision, and rational aggregate weights.
- Include every arm identity input that must start a new baseline when changed, including judgment vocabulary, analyzer, facts, configuration, registered budget, execution, outcome mapping, and combination rule.
- Refuse duplicate, unsorted, unknown, cross-key, non-finite, or incomplete values through exact closed validation.
- Done when canonical contract tests accept one complete component and combined-arm configuration and reject every single-field identity or arm-scope mismatch.
- _Requirements: 16.1, 16.4, 16.5, 18.1, 18.6, 18.7_
- _Boundary: Evaluation Measurement_
- _Depends: 12.2_
- [ ] 12.4 Establish distinct V3 version-boundary primitives
- Reserve the exact evaluation-admission V3 schema identity and version-routing primitives without changing any V2 public name or route.
- Make V3 contracts exact-type only and prohibit V2-to-V3 conversion, subclass acceptance, and mixed-version child relationships as each later root is introduced.
- Define the shared non-authorizing marker for feasibility, insufficiency, raw reports, aggregates, incomplete arm results, and other partial values; leave complete root decoding to facade integration.
- Done when boundary tests preserve exact V2 routing, recognize only the reserved V3 identity on the V3 path, and prove that a representative partial V3 value has no authorization capability.
- _Requirements: 16.1, 16.3, 16.4, 16.6_
- _Boundary: Evaluation Version Compatibility_
- _Depends: 12.1, 12.3_
- [ ] 13. Register reviewed multi-arm plans and exact preflight
- [ ] 13.1 Define component, combined-arm, and declarative combination configurations
- Represent component and combined arm configurations with exact scopes, policies, thresholds, sufficiency rules, outcome mapping, and a closed declarative combination rule.
- Require component identities and a finite normalized decision table for every combined arm, with identical-component-candidate-only proof retention.
- Refuse missing components, duplicate arms, cross-key policies, and arbitrary executable combination behavior.
- Done when canonical component and combined configurations validate and every component, rule-row, policy, threshold, sufficiency, or outcome-mapping mutation changes the configuration identity.
- _Requirements: 16.5, 17.1, 17.2, 19.4_
- _Boundary: Evaluation Arm Configuration_
- _Depends: 12.3_
- [ ] 13.2 Build the non-circular plan declaration, baselines, and expected slots
- Derive the complete label-blind plan declaration before constructing per-arm baselines so the plan and baselines cannot form a digest cycle.
- Bind each baseline to the population, freeze, input set, declaration, and exact arm configuration so any listed change creates a new baseline.
- Define expected observation slots for every planned arm and input without including ground-truth roles or labels.
- Done when plan reordering is canonical, every substantive mutation changes the declaration and affected baselines, and a combined arm cannot name an absent component.
- _Requirements: 16.5, 17.1, 17.2_
- _Boundary: Evaluation Plan_
- _Depends: 13.1_
- [ ] 13.3 Validate the reviewed label-blind census and plan manifest
- Represent exact vulnerable/fixed scope capacities, ecosystem-specific ordinary-push capacities, semantic/model capacities, and deduplicated whole-population size without case labels.
- Require an independently reviewed plan payload that binds the population, input set, arm configurations, baselines, readiness, custody head, census, and expected slots.
- Resolve review evidence under the configured trust and byte limits and refuse self-review, copied attestations, exposed labels, missing scopes, or inconsistent population identities.
- Done when one reviewed multi-arm manifest validates with labels sealed and every review, census, scope, ordinary-ecosystem, or slot substitution is refused before feasibility.
- _Requirements: 17.1, 17.2, 17.3, 18.1, 18.3_
- _Boundary: Evaluation Plan Review_
- [ ] 13.4 Derive feasibility independently for every planned arm and exact scope
- Produce exactly one feasibility row for the canonical product of planned arms and their claimed scopes.
- Recompute minimum sample size, threshold reachability, and Wilson interval precision from the reviewed label-blind census for each row.
- Derive shared spend eligibility only when every row is feasible; keep insufficiency and its reasons distinct from GO and NO-GO.
- Refuse missing, extra, duplicated, or caller-summarized rows and prevent preparation when one row is insufficient.
- Done when a single insufficient scope blocks the shared spend without opening labels, while the complete feasible plan yields one reproducible preflight digest.
- _Requirements: 17.3, 17.4, 20.2_
- _Boundary: Evaluation Feasibility_
- [ ] 14. Seal planned observations and authorize one atomic spend
- [ ] 14.1 Validate exact nonzero observations against every expected slot
- Represent label-blind observation headers that bind slot, arm, input, execution, registered budget, configuration, input-read evidence, stage-execution evidence, cases, and proof candidates.
- Resolve every sealed observation through trusted content references and prove its bytes are nonzero before accepting its header.
- Require one-to-one equality between expected slots and sealed observations; refuse missing, extra, changed, unreadable, cross-arm, cross-input, or duplicate values.
- Done when the complete slot set validates and each single-slot removal, swap, truncation, zero-byte value, or identity mutation is refused before authority.
- _Requirements: 17.5, 17.6, 20.8_
- _Boundary: Evaluation Observation_
- _Depends: 13.3_
- [ ] 14.2 Validate reviewed attempt history, sealed access, and exact retries
- Bind each attempt event to the complete retry identity, closed access policy, raw-observation and diagnostic access counts, public status projection, disposition, review, and predecessor.
- Resolve the reviewed predecessor chain from its declared head and require every unspent attempt to have the identical population, plan, arms, inputs, configurations, budgets, and execution identities.
- Permit public pre-spend projection only for closed status codes, counts, and digests; any raw/diagnostic access or identity change produces a terminal development-opened or spent disposition.
- Enforce terminal monotonicity so a later sealed event cannot restore admission eligibility.
- Done when an unchanged all-sealed retry chain validates and broken, forked, changed, exposed, self-reviewed, or terminal-then-sealed histories cannot authorize.
- _Requirements: 17.10, 17.11, 17.12_
- _Boundary: Evaluation Attempt History_
- _Depends: 13.3_
- [ ] 14.3 Validate the independently reviewed admission authority
- Bind the feasible plan to the exact sealed observation set, attempt-history head, current readiness, validation context, and current complete custody head.
- Require the admission-authority payload to be reviewed independently from its producer and resolve its review evidence before authorization.
- Revalidate feasibility, all observation bytes, attempt history, readiness, and custody immediately before producing authority.
- Refuse stale or restored availability, changed observations, missing review, and any difference from the reviewed plan rather than returning a partial authority.
- Done when one canonical authority is derived only from the complete reviewed graph and every stale, substituted, or incomplete source fails before spend.
- _Requirements: 17.5, 17.6, 17.8, 17.10, 17.11, 17.12_
- _Boundary: Evaluation Authority_
- _Depends: 13.4, 14.1, 14.2_
- [ ] 14.4 Bind every planned arm to one preparation and custody receipt
- Construct one head-bound label-release request whose influenced artifact transitively commits to every planned baseline and exact authority.
- Validate only the exact monotonic available-to-spent successor and derive an explicit all-arm binding between preparation, receipt, and planned baselines.
- Reject subset preparations, missing or mismatched receipts, stale concurrent attempts, broken custody chains, and any later restoration of availability.
- Keep durable compare-and-swap and label release outside core while exercising the boundary with an in-memory custody port.
- Done when one receipt authorizes all and only the planned arms, concurrent/stale losers fail, and no partial spend is constructible.
- _Requirements: 17.7, 17.8, 17.9_
- _Boundary: Evaluation Shared Spend_
- _Depends: 14.3_
- [ ] 15. Authenticate V3 case release, combinations, and proof continuity
- [ ] 15.1 Build the arm-aware V3 case and independently reviewed release graph
- Bind each post-spend case to its arm-scope key, baseline, observation, underlying push, role, pair, result, evidence inputs, change inputs, input manifest, and proof-candidate identity.
- Bind one independently reviewed release to the plan, admission authority, preparation, exact shared spend, and complete canonical case set.
- Require exact planned-arm and released-case cardinality and exact V3 root types; refuse V2 releases and subclasses.
- Require nonzero read and required-stage execution evidence in every released case before it can enter scoring.
- Done when the complete V3 release validates and V2 substitution, missing arms, duplicate cases, wrong spend, or copied review fail before metric production.
- _Requirements: 19.1, 19.2, 19.3, 19.11, 20.8_
- _Boundary: Evaluation Release_
- _Depends: 14.4_
- [ ] 15.2 Revalidate exact role, pair, scope, result, and input associations
- Recompute every case and binding digest from canonical evidence rather than accepting caller-supplied summaries.
- Require vulnerable/fixed roles and scoped pair identities to form exact pairs; prohibit pair identities on ordinary cases.
- Preserve the separately reviewed underlying-push identity for scoped projection and later whole-population deduplication.
- Refuse arm, scope, role, pair, observation, result, evidence, location, input, or manifest swaps before any count is derived.
- Done when a canonical multi-scope release passes and a mutation matrix proves every association mismatch is invalid rather than a failed quality gate.
- _Requirements: 18.2, 18.5, 19.2, 19.10_
- _Boundary: Released Case Validation_
- [ ] 15.3 Replay combined-arm rules and preserve component observation lineage
- Recompute combined observation outcomes from the registered finite decision table and exact component observation digests.
- Preserve each component arm's observation identity and outcome independently of the combined result; defer completed metric, gate, decision, and report preservation to replay assembly.
- Refuse missing, extra, replaced, reordered, or contradictory component material and any combined arm whose lineage differs from the reviewed plan.
- Done when the registered combined arm replays deterministically from exact component observations and no component observation can hide another component's absence, contradiction, or lineage mismatch.
- _Requirements: 19.4, 19.5, 19.11_
- _Boundary: Combination Validation_
- _Depends: 15.1, 15.2_
- [ ] 15.4 Enforce proof-candidate identity and prevent proof transfer
- Keep every label-blind proof candidate non-authoritative until the V3 release establishes its exact arm, family, scope, roles, inputs, and location.
- Apply proof-backed status only through the existing trusted differential-proof validator and the identical released candidate.
- Permit a combined arm to retain only a candidate already present in its named component observation; prevent creation, substitution, and cross-arm transfer.
- Preserve absent or inconclusive proof as not demonstrated and refuse role-swapped, cross-family, cross-scope, location, input, control, producer, receipt, or trust-policy mismatches.
- Done when valid released proof attaches once, absent proof remains non-demonstrated, and the complete transfer/mismatch matrix is refused.
- _Requirements: 19.6, 19.7, 19.8, 19.9, 19.10_
- _Boundary: Proof Continuity_
- _Depends: 15.3_
- [ ] 16. Derive exact scoped metrics and completed reports
- [ ] 16.1 Partition quality evidence by exact arm, family, and ecosystem
- Select vulnerable and fixed cases only from the independently released exact scope and derive pair correctness without cross-scope reuse.
- Derive precision only from findings and cases assigned to the same arm-scope key under its reviewed denominator policy.
- Preserve family decision vocabulary, neutral outcomes, evidence bases, model outcomes, and semantic coverage independently for each key.
- Done when a multi-arm, multi-scope case set yields isolated partitions and any cross-arm, cross-family, or cross-ecosystem contribution is refused.
- _Requirements: 18.1, 18.2, 18.4_
- _Boundary: Evaluation Scoped Metrics_
- _Depends: 15.2, 15.4_
- [ ] 16.2 Project ordinary pushes by ecosystem and deduplicate whole coverage
- Count each eligible ordinary push once for every family claimed by the arm in the push's reviewed ecosystem and never in another ecosystem.
- Deduplicate one underlying push once in whole-population coverage even when it contributes to several verified scope denominators.
- Reconcile projected ordinary counts and global unique-push counts back to the reviewed census and released identities.
- Done when same-ecosystem multi-family projection produces the expected scoped counts, cross-ecosystem projection is absent, and whole coverage counts each push once.
- _Requirements: 18.3, 18.5_
- _Boundary: Evaluation Ordinary Projection_
- [ ] 16.3 Produce the complete scoped metric set and reject post-spend invalidity
- Derive every standard metric for every arm-scope key using the fixed algebra and Wilson 95% calculation.
- Publish numerator, denominator, sample count, value, and interval bounds for each scoped metric.
- Treat a post-spend zero denominator, census difference, missing standard metric, or inconsistent projection as invalid evaluation evidence rather than insufficiency, GO, or NO-GO.
- Done when complete nonzero scoped metrics match independent count fixtures and every zero/mismatch mutation fails before report construction.
- _Requirements: 18.1, 18.6, 18.8_
- _Boundary: Evaluation Metric Derivation_
- [ ] 16.4 Derive completed per-scope decisions and per-arm operational evidence
- Derive instrument, budget, readiness, and quality gates for every exact arm-scope result from canonical evidence.
- Emit GO or NO-GO only when the scope evaluation is complete; keep insufficiency, unread input, skipped stage, incomplete arm, and instrument impairment outside quality decisions.
- Preserve per-arm completed and deferred case identities, stage funnel, instrument/model failures, runtime, and closed degradation reasons separately from shared whole coverage.
- Require nonzero input reads and required-stage execution before accepting a clean-looking result.
- Done when every completed scope receives one reproducible decision and every incomplete or impaired fixture is refused without a quality decision.
- _Requirements: 20.1, 20.2, 20.3, 20.8_
- _Boundary: Evaluation Report_
- [ ] 16.5 Derive optional aggregates that can never authorize
- Bind aggregate constituents, sample counts, canonical rational weights, and derived metric values to exact arm-scope metric identities.
- Require weights to be non-negative, canonical, one-to-one with constituents, and sum exactly to one through integer arithmetic.
- Exclude gates, GO/NO-GO, proof promotion, and every consumer-authorization capability from the aggregate contract.
- Done when a valid aggregate reproduces from its constituents and raw, incomplete, gated, decision-bearing, or differently weighted aggregates are refused as authorization evidence.
- _Requirements: 16.6, 18.7, 20.7_
- _Boundary: Evaluation Aggregate_
- [ ] 17. Close complete replay and replay-only consumer authorization
- [ ] 17.1 Assemble one complete replay over the shared evaluation graph
- Bind the reviewed plan, exact feasibility, reviewed admission authority, preparation, receipt, all-arm spend binding, attempt-history head, V3 release, cases, per-arm reports, combination lineages, shared whole coverage, and optional aggregate.
- Require exact equality between planned arms/scopes and case/report identities, including every component required by a combined arm.
- Prevent replay construction when any arm, scope, mandatory gate, custody binding, release, case, component, or lineage is missing or invalid.
- Done when the complete graph produces one deterministic replay digest and every omitted or substituted component prevents construction.
- _Requirements: 19.4, 19.5, 19.11, 20.4, 20.5_
- _Boundary: Evaluation Replay_
- _Depends: 15.3, 16.4, 16.5_
- [ ] 17.2 Revalidate replay from canonical source artifacts and current trust state
- Resolve the complete attempt chain and observation/review content again, revalidate current custody, and rederive every identity, count, metric, gate, decision, report, lineage, and replay digest.
- Require exact equality with the carried values and refuse coordinated rehashing, stale external state, unknown/mixed versions, and partial roots.
- Make the validated complete replay the only value that may cross a V3 authorization boundary.
- Done when independent replay validation reproduces the original reports exactly and every source, summary, or trust-state mutation is refused.
- _Requirements: 16.6, 20.5, 20.6, 20.7, 20.8_
- _Boundary: Evaluation Replay Validation_
- [ ] 17.3 Publish the integrated V3 preflight, authorization, scoring, and replay facade
- Wire the exact plan preflight, reviewed authority, shared-spend validation, score/replay, strict decoding, and replay validation through one public V3 contract surface.
- Preserve every V2 facade and import unchanged and expose no benchmark storage, execution, lifecycle, or presentation implementation from core.
- Reject wrong-phase values and make each returned artifact usable only by the next declared phase.
- Done when a client can execute the complete synthetic contract through the public V3 schema and V2/unknown/mixed artifacts fail at the first boundary.
- _Requirements: 16.1, 16.4, 20.4, 20.6_
- _Boundary: Evaluation Admission Facade Integration_
- _Depends: 17.1, 17.2_
- [ ] 17.4 (P) Authorize fitted components only from one exact replayed arm-scope result
- Validate the complete V3 replay before selecting exactly one planned arm-scope result and require that completed result to be GO.
- Preserve existing reversed-pair probe replay, fitting lineage, exact model declaration, and disjoint resolved repository-holdout checks.
- Refuse V2 replay, raw arm report, aggregate, incomplete replay, missing/ambiguous key, NO-GO scope, or substituted fitted evidence.
- Done when a matching fitted component authorizes only from its exact GO scope and every partial, aggregate, cross-arm, or probe/holdout mutation fails.
- _Requirements: 13.2, 16.6, 20.7_
- _Boundary: Fitted Consumer V3_
- _Depends: 17.2_
- [ ] 18. Validate the V3 predecessor end to end
- [ ] 18.1 Run a synthetic multi-arm, multi-family evaluation through one shared spend
- Use at least two component arms and one combined arm across at least two families and two ecosystems with multi-scope fixes and representative ordinary pushes.
- Prove nonzero reads, reviewed plan, exact preflight, sealed observations, reviewed attempt history, one preparation/receipt, V3 release, scoped reports, complete replay, and exact fitted selection.
- Preserve component observations/results and verify ordinary projection plus shared whole-population deduplication in the completed replay.
- Done when the public V3 facade produces one validated replay with every planned arm/scope and no V2 or benchmark-owned implementation is used as a substitute.
- _Requirements: 16.1, 16.5, 17.1, 17.3, 17.5, 17.7, 18.1, 18.3, 18.5, 19.1, 19.4, 20.1, 20.4, 20.7, 20.8_
- _Boundary: V3 End-to-End Conformance_
- _Depends: 17.3, 17.4_
- [ ] 18.2 (P) Run the adversarial pre-spend and compatibility matrix
- Exercise V2 byte/digest/replay stability, V2-as-V3, unknown/mixed/subclass schemas, baseline mutations, impossible scopes, missing slots, zero/unreadable observations, self-review, forked attempt history, identity changes, raw access, stale custody, and subset spend.
- Prove every measurement instrument reads nonzero fixture bytes and fails loudly on unread input.
- Verify insufficiency leaves custody available and no refused case reaches detector execution or label release in the synthetic boundary.
- Done when every pre-spend mutation fails at its owning layer, V2 remains unchanged, and no failure produces a clean-looking result or quality decision.
- _Requirements: 16.1, 16.2, 16.3, 16.4, 16.5, 16.6, 17.2, 17.4, 17.6, 17.8, 17.9, 17.10, 17.11, 17.12, 20.2, 20.8_
- _Boundary: V3 Pre-Spend and Compatibility Validation_
- _Depends: 17.3_
- [ ] 18.3 (P) Run the adversarial post-spend, metric, combination, and replay matrix
- Exercise V2 release substitution, arm/scope/role/pair/result/input swaps, missing components, contradictory combinations, proof transfer, absent/inconclusive proof, cross-ecosystem ordinary projection, duplicate whole coverage, zero/mismatched denominators, missing gates, and incomplete reports/replay.
- Verify invalid post-spend evidence never becomes insufficiency, GO, NO-GO, aggregate authority, or consumer eligibility.
- Revalidate every complete replay from source artifacts and current custody rather than trusting carried summaries.
- Done when the valid control replay reproduces exactly and every post-spend mutation is refused before authorization without hiding another arm's status.
- _Requirements: 18.2, 18.4, 18.6, 18.7, 18.8, 19.2, 19.3, 19.5, 19.6, 19.7, 19.8, 19.9, 19.10, 19.11, 20.3, 20.5, 20.6, 20.7_
- _Boundary: V3 Post-Spend and Replay Validation_
- _Depends: 17.2, 17.4_
- [ ] 18.4 Verify dependency boundaries, complexity caps, and bounded validation growth
- Verify dependency direction, no V2/V3 aggregate imports across the version boundary, standard-library-only core runtime, and absence of benchmark, analyzer, model-runtime, or proof-execution ownership in V3 modules.
- Enforce function and module complexity caps and measure validation growth over increasing arms, scopes, observations, attempt events, cases, and reports.
- Fail each measurement fixture unless its scoped inputs are readable and nonzero before interpreting timing, count, or clean-result output.
- Done when import/boundary checks are clean, amended code stays within complexity caps, and validation cost remains bounded by supplied artifact counts without weakening checks.
- _Requirements: 16.2, 20.6, 20.8_
- _Boundary: V3 Boundary and Performance Validation_
- _Depends: 18.1, 18.2, 18.3_
- [ ] 18.5 Run the final compatibility, regression, and schema-export gate
- Run the V2 golden suite, every V3 unit/integration/adversarial suite, and the existing core regression suite without consuming a real admission population.
- Prove all fixtures are readable and nonzero and the exact `openultrasast.detection-core.evaluation-admission.v3` schema is publicly exported.
- Done when all gates pass, V2 outputs remain frozen, validation stays bounded by supplied artifacts, and the validated predecessor is ready for downstream benchmark task generation.
- _Requirements: 16.1, 16.2, 16.4, 20.6, 20.8_
- _Boundary: V3 Final Validation Integration_
- _Depends: 18.4_
## Implementation Notes
@@ -446,3 +713,4 @@
- Task 10.6 debug established that exact-byte successor authority must require canonical repository-relative paths and strict resolved-root containment, including rejection of outside-root symlink targets, before module digests can prove closure.
- Human review reopened Task 10.6 for one focused remediation cycle covering canonical path syntax, resolved-root containment/symlink refusal, and the task-local formatting gate.
- Human review reopened Task 10.5 for one focused remediation cycle covering exact source-closure evidence, collision-resistant create-only attempt persistence, and complexity-cap conformance.
- Task 11.4 debug established that the 73 reserved-repository references are a benchmark-governance specification conflict, not a detection-core or instrument defect; historical artifacts and the reservation guard must remain unchanged pending an owning-spec decision.