diff --git a/.kiro/specs/detection-core/design.md b/.kiro/specs/detection-core/design.md index 2418218..bbdbe5e 100644 --- a/.kiro/specs/detection-core/design.md +++ b/.kiro/specs/detection-core/design.md @@ -18,6 +18,9 @@ Authorization remains parked and supplies regression evidence only. Historical v remain readable but are never rewritten or used as a v2 trend baseline. Live v1 contracts and serialization first move behind one explicit, deletable `legacy_v1` namespace; cutover is then proved by syntax and capability removal rather than runtime alias inference. +The evaluation-admission amendment adds a separate V3 root for exact-scope, multi-arm +qualification under one shared population spend. It preserves every V2 admission artifact +and replay path without conversion or reinterpretation. ### Goals @@ -34,6 +37,11 @@ is then proved by syntax and capability removal rather than runtime alias infere suites. - Make the v1 removal gate structural: every live v1 import and writer crosses an explicit compatibility namespace that is deleted at cutover. +- Qualify every component and combined arm independently for each exact family/ecosystem + scope while binding all arms to one reviewed plan, one atomic spend, and one complete + replay. +- Preserve V2 admission bytes, digests, decisions, and replay behavior as an immutable + compatibility contract. ### Non-Goals @@ -44,6 +52,11 @@ is then proved by syntax and capability removal rather than runtime alias infere - Reviving or re-admitting the parked authorization arm. - Enabling any capability for advisory or blocking rollout. - Providing cross-language canonical signatures; v2 is a Python-runtime contract. +- Converting pooled V2 admission evidence into V3 scoped evidence or treating a V2 report + as a V3 baseline. +- Owning population lifecycle, durable CAS/object storage, observation access enforcement, + detector isolation, retry orchestration, workflow presentation, or rollout decisions for + the multi-arm benchmark. ## Boundary Commitments @@ -65,6 +78,9 @@ is then proved by syntax and capability removal rather than runtime alias infere - A finite v1 migration and read-only historical decoder. - The temporary `legacy_v1` compatibility namespace, frozen migration inventory, and syntax-only cutover boundary that make v1 write capability removable. +- The V3 evaluation plan, exact arm/scope baseline and policy schemas, per-scope + feasibility, sealed-observation authority validation, all-arm spend binding, V3 release, + scoped reports, complete replay, and replay-only consumer authorization. ### Out of Boundary @@ -78,6 +94,9 @@ is then proved by syntax and capability removal rather than runtime alias infere - Push scoping policy, alert rendering, capability rollout, and BLOCK enablement. - Production vulnerability-family contracts, target-specific authority bundles, structural matchers, and match-artifact persistence. Those belong to the downstream family spec. +- Benchmark-owned population lifecycle, object-store/CAS adapters, detector execution, + observation access enforcement, label joining, retries, workflow status, presentation, + notification, and rollout policy. ### Allowed Dependencies @@ -104,6 +123,9 @@ is then proved by syntax and capability removal rather than runtime alias infere - Production imports name a core submodule or symbol explicitly. Module-valued or dynamic access to `openultrasast.core` is outside the migration contract because it would hide legacy use from a deletion-backed gate. +- V3 evaluation modules may reuse version-neutral core canonical, review, custody, + readiness, evidence, proof, coverage, budget, and metric-arithmetic primitives. They do + not import V2 aggregate roots; V2 modules do not import V3 modules. ### Revalidation Triggers @@ -121,6 +143,11 @@ is then proved by syntax and capability removal rather than runtime alias infere design review. - Adding a v1 compatibility export, migration-inventory entry, direct v1 serializer, or dynamic core import reopens the namespace-isolation design and blocks cutover. +- Changing any V3 plan, observation-slot, retry/history, shared-spend, case-release, + proof-candidate, combination, scoped metric, aggregate, report, replay, or consumer + selection contract revalidates the benchmark and every V3 consumer. +- Extracting or changing a version-neutral helper requires the frozen V2 canonical, + decision, failure, and replay suite to pass byte-for-byte. ## Architecture @@ -139,6 +166,12 @@ test explicitly prohibits a public model path. Production push and proof paths d consume the core end to end. The migration therefore preserves reusable mechanisms while replacing the public domain model. +The implemented V2 admission root is intentionally retained but structurally limited: one +arm owns one baseline, every metric rule receives the complete claimed-scope tuple, +feasibility pools capacity across those scopes, and one unscoped metric set produces one +decision. Exact per-arm/per-family/per-ecosystem qualification therefore requires an +additive V3 aggregate graph rather than an in-place V2 extension. + ### Architecture Pattern and Boundary Map The selected pattern is a **versioned ports-and-adapters contract with an explicit, @@ -2555,6 +2588,871 @@ map to the authorization family contract. Missing per-hop completeness remains i The DSPy and search adapters similarly map external typed records without leaking their dependencies into core. +## Evaluation-Admission V3 Amendment + +Requirements 16–20 extend the implemented family-neutral core with a distinct exact-scope, +multi-arm evaluation contract. This section is normative for V3; the earlier V2 measurement +and admission design remains normative and unchanged for historical and compatibility use. + +### Boundary Commitments + +Detection-core owns the V3 canonical schemas, exact-type validators, reviewed-plan and +observation-authority contracts, per-arm/per-scope feasibility and metric algebra, shared +spend binding, V3 case release, component/combined lineage validation, scoped reports, +complete replay, and consumer authorization. It preserves every V2 schema, byte, digest, +decision, validator, and replay path unchanged. + +Detection-core does not own population sourcing or lifecycle, durable object storage or +CAS, observation access enforcement, detector execution or isolation, retry orchestration, +label joining, workflow status, notification, presentation, or rollout policy. The benchmark +produces immutable reviewed lifecycle, observation, custody, and release artifacts; core +validates them and derives no authority from an unreviewed workflow state. + +Allowed dependencies are Python 3.11 standard-library modules and existing version-neutral +core primitives: canonical content references/resolution, independence review, custody, +readiness, evidence validation, proof attachment, coverage, budget, and Wilson arithmetic. +V3 modules never import V2 aggregate roots. V2 modules never import V3 modules. Adjacent +benchmark and consumer code imports the public V3 facade. + +Downstream revalidation is required by a V3 schema or canonical-profile change; a change to +plan, baseline, observation-slot, disposition/retry, shared-spend, release, proof-candidate, +combination, denominator, gate, aggregate, or replay semantics; or any shared-helper change +that alters frozen V2 output. + +### Architecture + +The selected pattern is parallel versioned aggregate roots with version-neutral internal +helpers. V2 remains an independently replayable historical/live compatibility contract. +V3 is a new baseline and has no conversion path from V2. + +The public root schema is exactly +`openultrasast.detection-core.evaluation-admission.v3`. Strict decoding accepts only the +registered V3 root schemas below; unknown versions, V2 roots, subclasses, and graphs whose +children name a different version are refused without shape inference. + +```mermaid +flowchart LR + Benchmark[Benchmark workflow] --> Plan[Reviewed plan] + Plan --> Feasibility[Exact scope feasibility] + Benchmark --> Observations[Sealed observations] + Feasibility --> Authority[Admission authority] + Observations --> Authority + Authority --> Custody[Shared custody transition] + Custody --> Release[V3 case release] + Release --> Reports[Scoped arm reports] + Reports --> Replay[Complete evaluation replay] + Replay --> Consumer[Consumer authorization] +``` + +Dependency direction: + +```text +canonical, review, custody, readiness, evidence, proof, coverage, budget + → evaluation_measurement + → evaluation_plan + → evaluation_feasibility + → evaluation_observation + → evaluation_authority + → evaluation_release + → evaluation_metrics + → evaluation_report + → evaluation_replay + → evaluation_admission facade + → admission_fitted V3 consumer +``` + +### File Structure Plan + +New files: + +- `src/openultrasast/core/_metric_algebra.py` — version-neutral closed metric algebra + and Wilson arithmetic, created only after V2 golden fixtures lock current behavior; + contains no schema, serializer, aggregate root, or version dispatch. +- `src/openultrasast/core/evaluation_measurement.py` — V3 arm/scope keys, arm + configuration and baseline identities, exact denominator policies, scoped thresholds, + sufficiency/precision rules, scoped metric results, and non-authorizing aggregates. +- `src/openultrasast/core/evaluation_plan.py` — non-circular plan declaration, expected + observation slots, component/combined arm plans, V3 census, and independently reviewed + `EvaluationPlanManifestV3`. +- `src/openultrasast/core/evaluation_feasibility.py` — per-arm/per-scope threshold, + minimum-size, and interval-precision witnesses and shared `may_spend` derivation. +- `src/openultrasast/core/evaluation_observation.py` — sealed observation references, + reviewed resolver-backed attempt history, retry/disposition evidence, and + `EvaluationAdmissionManifestV3`. +- `src/openultrasast/core/evaluation_authority.py` — post-observation revalidation, + `EvaluationAdmissionAuthorityV3`, shared preparation, and all-arm spend binding. +- `src/openultrasast/core/evaluation_release.py` — arm-aware released-case bindings, + proof-candidate continuity, and independently reviewed `AdmissionCaseReleaseV3`. +- `src/openultrasast/core/evaluation_metrics.py` — exact-scope case validation and metric + derivation, ordinary ecosystem projection, whole-push deduplication, and combination + preservation. +- `src/openultrasast/core/evaluation_report.py` — completed per-scope decisions, per-arm + operational evidence, whole-population coverage, degradation reasons, and aggregates. +- `src/openultrasast/core/evaluation_replay.py` — complete V3 graph construction and full + authoritative replay validation. +- `src/openultrasast/core/evaluation_admission.py` — sole public V3 facade for preflight, + authorization, score/replay, strict decoding, and replay validation. + +Modified files: + +- `src/openultrasast/core/admission_fitted.py` — add a separate V3 consumer function that + accepts only a complete replay plus one exact arm/scope key; keep the V2 function exact. +- `src/openultrasast/core/__init__.py` — expose the V3 facade without changing V2 names. +- `tests/fixtures/detection_core/admission_v2/` — frozen accepted bytes, digests, + decisions, validation failures, and replay outputs. +- `tests/test_core_admission_v2_compatibility.py` and + `tests/test_core_evaluation_*.py` — V2 stability and V3 contract/conformance suites. + +### System Flow + +```mermaid +sequenceDiagram + participant Run as Benchmark workflow + participant Core as Detection core V3 + participant Store as Observation resolver + participant Custody as Custody writer + participant Reviewer as Independent reviewer + + Run->>Reviewer: Label blind plan payload + Reviewer-->>Run: Plan attestation + Run->>Core: EvaluationPlanManifestV3 + Core-->>Run: EvaluationFeasibilityV3 + Run->>Run: Execute every planned arm label blind + Run->>Store: Seal every expected observation + Run->>Reviewer: Plan observations and disposition payload + Reviewer-->>Run: Admission attestation + Run->>Core: EvaluationAdmissionManifestV3 + Core->>Store: Resolve exact nonzero observations + Core-->>Run: EvaluationPreparationV3 + Run->>Custody: Compare and swap shared request + Custody-->>Run: One spend receipt and labels + Run->>Reviewer: Arm case release payload + Reviewer-->>Run: V3 release attestation + Run->>Core: Receipt release and every arm case + Core-->>Run: EvaluationReplayV3 +``` + +Any insufficient arm/scope ends before execution/spend. Missing, changed, unreadable, or +unsealed observations end before preparation. After spend, missing or invalid cases, +denominator mismatch, or incomplete arms invalidate the evaluation and cannot be reported +as insufficiency, GO, or NO-GO. The custody writer remains external and releases labels +only to the exact CAS winner. + +### Requirements Traceability + +| Requirement | Design realization | +|---|---| +| 16.1, 16.2, 16.3, 16.4 | Separate V3 facade, exact schema/type validators, no conversion, frozen V2 compatibility suite | +| 16.5 | Plan declaration plus per-arm V3 baseline binds every listed identity and changes on any plan/configuration mutation | +| 16.6 | Typed preflight/impairment/aggregate/report artifacts have no consumer-authority interface; only complete replay authorizes | +| 17.1, 17.2 | Independently reviewed plan declaration, arm plans, component/combined lineage, and expected slots | +| 17.3, 17.4 | Complete arm/scope feasibility product and shared `may_spend`; insufficiency prevents preparation | +| 17.5, 17.6 | Independently reviewed admission manifest and exact resolved observation-slot equality | +| 17.7, 17.8, 17.9 | One authority, preparation, CAS request/receipt, and all-arm spend binding; no subset construction | +| 17.10, 17.11, 17.12 | Reviewed sealed/development/spent disposition and exact retry identity; only sealed unchanged retries validate | +| 18.1, 18.2, 18.3, 18.4 | Full metric set for each `ArmScopeKey`; exact roles/scopes; ecosystem-only ordinary projection; exact precision | +| 18.5 | `underlying_push_digest` deduplicates whole coverage independently from scoped projections | +| 18.6, 18.7 | Scoped metric numerator/denominator/sample/Wilson fields and aggregate type without gate/decision | +| 18.8 | Post-spend zero or census mismatch is validation failure before report construction | +| 19.1, 19.2, 19.3 | Independently reviewed arm-aware V3 release bound to plan, observations, authority and shared spend; V2 type refused | +| 19.4, 19.5 | Exact component observations/reports and deterministic combination lineage are mandatory and contradiction-checked | +| 19.6, 19.7, 19.8, 19.9, 19.10 | Pre-release candidate is non-authoritative; post-release proof matches exact candidate/binding; no transfer or mismatch | +| 19.11 | Planned-arm/report cardinality equality; no result can mask another arm | +| 20.1, 20.2 | Completed result per arm/scope with GO/NO-GO; insufficiency, impairment, incomplete and invalid remain distinct | +| 20.3 | Per-arm scoped metrics plus separate whole coverage, completed/deferred work, funnels, failures, runtime and degradations | +| 20.4, 20.5, 20.6, 20.7 | Closed replay binds and rederives the full graph; missing components refuse completion; only replay authorizes | +| 20.8 | Each arm/scope requires nonzero read receipts and required-stage execution evidence | + +### Components and Interfaces + +#### V3 identity, plan, and feasibility + +`ArmScopeKey` is the natural key for every scoped policy, threshold, metric, gate, and +decision. An arm configuration carries the actual immutable policy objects and the +identities listed in 16.5. `EvaluationPlanDeclarationV3` is hashed before arm baselines are +constructed, avoiding a digest cycle. Each baseline binds the declaration digest and one +arm-configuration digest. + +```python +@dataclass(frozen=True, order=True) +class ArmScopeKey: + arm_id: str + scope: ScopeKey + +class ArmKindV3(Enum): + COMPONENT = "component" + COMBINED = "combined" + +@dataclass(frozen=True) +class ScopedMetricRuleV3: + key: ArmScopeKey + metric_id: MetricId + numerator: NumeratorKind + denominator: DenominatorKind + rule_digest: str + +@dataclass(frozen=True) +class ScopedThresholdV3: + key: ArmScopeKey + metric_id: MetricId + statistic: Statistic + comparator: Comparator + target: float + threshold_digest: str + +@dataclass(frozen=True) +class IntervalPrecisionRuleV3: + metric_id: MetricId + maximum_wilson_width: float + minimum_denominator: int + rule_digest: str + +@dataclass(frozen=True) +class DenominatorPolicyV3: + key: ArmScopeKey + rules: tuple[ScopedMetricRuleV3, ...] + interval_method: IntervalMethod + policy_digest: str + +@dataclass(frozen=True) +class ScopeSufficiencyRuleV3: + key: ArmScopeKey + minimum_pairs: int + minimum_ordinary_pushes: int + precision_rules: tuple[IntervalPrecisionRuleV3, ...] + rule_digest: str + +@dataclass(frozen=True) +class ExactScopeCensusV3: + scope: ScopeKey + pair_count: int + vulnerable_side_count: int + fixed_side_count: int + semantic_site_capacity: int + model_call_capacity: int + +@dataclass(frozen=True) +class EcosystemOrdinaryCensusV3: + ecosystem: str + ordinary_push_count: int + +@dataclass(frozen=True) +class PopulationCensusV3: + population_digest: str + scope_rows: tuple[ExactScopeCensusV3, ...] + ordinary_rows: tuple[EcosystemOrdinaryCensusV3, ...] + whole_underlying_push_count: int + labels_exposed: bool + payload_digest: str + review: IndependenceAttestation + census_digest: str + +class CombinationOperatorV3(Enum): + DECLARATIVE_TABLE = "declarative_table" + +class ProofCandidatePolicyV3(Enum): + IDENTICAL_COMPONENT_ONLY = "identical_component_only" + +@dataclass(frozen=True) +class CombinationDecisionRowV3: + component_outcomes: tuple[NeutralOutcome, ...] + combined_outcome: NeutralOutcome + row_digest: str + +@dataclass(frozen=True) +class CombinationRuleV3: + combined_arm_id: str + component_arm_ids: tuple[str, ...] + operator: CombinationOperatorV3 + decision_rows: tuple[CombinationDecisionRowV3, ...] + candidate_policy: ProofCandidatePolicyV3 + rule_digest: str + +@dataclass(frozen=True) +class CombinationLineageV3: + rule: CombinationRuleV3 + component_observation_digests: tuple[str, ...] + retained_proof_candidate_digests: tuple[str, ...] + lineage_digest: str + +@dataclass(frozen=True) +class ArmConfigurationV3: + arm_id: str + kind: ArmKindV3 + exact_scopes: tuple[ScopeKey, ...] + denominator_policies: tuple[DenominatorPolicyV3, ...] + thresholds: tuple[ScopedThresholdV3, ...] + sufficiency_rules: tuple[ScopeSufficiencyRuleV3, ...] + judgment_vocabulary_digest: str + analyzer_identity: str + fact_set_digest: str + configuration_digest: str + registered_budget_digest: str + execution_identity: str + outcome_mapping_digest: str + combination_rule: CombinationRuleV3 | None + configuration_identity_digest: str + +@dataclass(frozen=True) +class ExpectedObservationSlotV3: + slot_id: str + arm_id: str + input_identity_digest: str + execution_identity: str + slot_digest: str + +@dataclass(frozen=True) +class EvaluationPlanDeclarationV3: + population_digest: str + freeze_digest: str + input_set_digest: str + arm_configurations: tuple[ArmConfigurationV3, ...] + expected_observation_slots: tuple[ExpectedObservationSlotV3, ...] + declaration_digest: str + +@dataclass(frozen=True) +class EvaluationBaselineIdentityV3: + arm_id: str + plan_declaration_digest: str + population_digest: str + freeze_digest: str + input_set_digest: str + arm_configuration_digest: str + baseline_digest: str + +@dataclass(frozen=True) +class EvaluationPlanManifestV3: + declaration: EvaluationPlanDeclarationV3 + baselines: tuple[EvaluationBaselineIdentityV3, ...] + census: PopulationCensusV3 + readiness_digests: tuple[str, ...] + custody_log_digest: str + payload_digest: str + review: IndependenceAttestation + manifest_digest: str + +class FeasibilityStatusV3(Enum): + FEASIBLE = "feasible" + INSUFFICIENT = "insufficient" + +@dataclass(frozen=True) +class ThresholdFeasibilityV3: + threshold_digest: str + attainable: bool + witness_numerator: int + witness_denominator: int + witness_digest: str + +@dataclass(frozen=True) +class SufficiencyResultV3: + rule_digest: str + minimum_size_passed: bool + interval_precision_passed: bool + reasons: tuple[str, ...] + result_digest: str + +@dataclass(frozen=True) +class ScopeFeasibilityV3: + key: ArmScopeKey + threshold_witnesses: tuple[ThresholdFeasibilityV3, ...] + sufficiency_results: tuple[SufficiencyResultV3, ...] + status: FeasibilityStatusV3 + reasons: tuple[str, ...] + result_digest: str + +@dataclass(frozen=True) +class EvaluationFeasibilityV3: + plan_manifest_digest: str + rows: tuple[ScopeFeasibilityV3, ...] + may_spend: bool + feasibility_digest: str +``` + +The feasibility rows equal the canonical product of every arm and its exact claimed scopes. +Every row rederives minimum sizes, threshold reachability, and interval precision from the +reviewed label-blind census. `may_spend` is true only when every row is feasible. + +#### Observation authority and atomic spend + +```python +class PreSpendDispositionV3(Enum): + SEALED = "sealed" + DEVELOPMENT_OPENED = "development_opened" + SPENT = "spent" + +class PublicStatusFieldV3(Enum): + STATUS_CODE = "status_code" + COUNT = "count" + DIGEST = "digest" + +@dataclass(frozen=True) +class ObservationAccessPolicyV3: + allowed_public_fields: tuple[PublicStatusFieldV3, ...] + raw_observation_access_allowed: bool + population_diagnostic_access_allowed: bool + policy_digest: str + +@dataclass(frozen=True) +class ObservationCaseRefV3: + case_id_digest: str + underlying_push_digest: str + result_digest: str + proof_candidate_digest: str | None + +@dataclass(frozen=True) +class ArmObservationSetV3: + slot_digest: str + arm_id: str + input_identity_digest: str + execution_identity: str + registered_budget_digest: str + configuration_digest: str + input_read_evidence_digest: str + stage_execution_evidence_digest: str + cases: tuple[ObservationCaseRefV3, ...] + observation_digest: str + +@dataclass(frozen=True) +class SealedObservationRefV3: + slot_digest: str + arm_id: str + input_identity_digest: str + execution_identity: str + registered_budget_digest: str + configuration_digest: str + observation: ContentRef + observation_digest: str + read_receipt_digest: str + ref_digest: str + +@dataclass(frozen=True) +class AttemptIdentityV3: + population_digest: str + plan_declaration_digest: str + baseline_digests: tuple[str, ...] + input_set_digest: str + configuration_digests: tuple[str, ...] + registered_budget_digests: tuple[str, ...] + execution_identities: tuple[str, ...] + retry_identity_digest: str + +@dataclass(frozen=True) +class AttemptEventV3: + attempt_identity: AttemptIdentityV3 + disposition: PreSpendDispositionV3 + access_policy: ObservationAccessPolicyV3 + raw_observation_access_count: int + population_diagnostic_access_count: int + public_status_projection: ContentRef + predecessor_event_digest: str | None + payload_digest: str + review: IndependenceAttestation + event_digest: str + +@dataclass(frozen=True) +class AttemptEventRefV3: + event: ContentRef + event_digest: str + ref_digest: str + +@dataclass(frozen=True) +class AttemptHistoryV3: + population_digest: str + events: tuple[AttemptEventRefV3, ...] + head_event_digest: str + history_digest: str + +@dataclass(frozen=True) +class EvaluationAdmissionManifestV3: + plan_manifest: EvaluationPlanManifestV3 + feasibility_digest: str + observations: tuple[SealedObservationRefV3, ...] + readiness_digests: tuple[str, ...] + custody_log_digest: str + attempt_history: AttemptHistoryV3 + payload_digest: str + review: IndependenceAttestation + manifest_digest: str + +@dataclass(frozen=True) +class EvaluationAdmissionAuthorityV3: + admission_manifest_digest: str + plan_manifest_digest: str + baseline_digests: tuple[str, ...] + feasibility_digest: str + observation_set_digest: str + readiness_set_digest: str + validation_context_digest: str + custody_log_digest: str + attempt_history_head_digest: str + admission_review_digest: str + authority_digest: str + +@dataclass(frozen=True) +class EvaluationPreparationV3: + authority: EvaluationAdmissionAuthorityV3 + label_release_request: PopulationUseRequest + planned_baseline_digests: tuple[str, ...] + preparation_digest: str + +@dataclass(frozen=True) +class EvaluationSpendBindingV3: + preparation_digest: str + receipt_digest: str + planned_baseline_digests: tuple[str, ...] + binding_digest: str +``` + +`EvaluationAdmissionManifestV3` is the externally produced, independently reviewed +admission-authority artifact required by 17.5: its review subject is the complete payload +digest, its producer and reviewer differ, and the payload binds the feasible plan, exact +observations, the complete attempt-history head, readiness, and current custody head. The derived +`EvaluationAdmissionAuthorityV3` is core's validated projection of that reviewed artifact; +it carries the exact review digest and cannot exist without it. + +The admission manifest must contain exactly one observation reference for every expected +slot. Validation resolves each content reference, proves nonzero bytes, parses its canonical +`ArmObservationSetV3`, and checks arm, input, execution, budget, configuration, slot, +input-read evidence, stage-execution evidence, case references, and observation +identity. Core resolves every `AttemptEventRefV3`, validates its independent review and +nonzero canonical bytes, and requires an exact predecessor chain ending at the declared +head. Every unspent predecessor must carry the same rederived `AttemptIdentityV3`; a changed +population, plan, baseline set, input set, configuration, registered budget, or execution +identity requires a terminal development-opened or spent event. The access policy must +prohibit raw-observation and population-diagnostic access, both access counts must be zero, +and the resolved public projection may contain only closed status codes, counts, and +digests. Any nonzero access count likewise requires a terminal development-opened or spent +event. Terminal dispositions are monotonic, so no later sealed event can restore admission +eligibility. Only a complete all-sealed identical-identity chain can authorize. The derived +authority and replay both bind the history head; replay re-resolves the full chain. The +authority also binds the independent admission-review digest and cannot be produced from a +self-reviewed or unattested payload. + +The underlying `PopulationUseRequest` names the canonical evaluation-plan identity in its +legacy `arm_id` field and the V3 authority digest as its influenced artifact. The authority +transitively binds every baseline; `EvaluationSpendBindingV3` makes the relationship to the +one exact receipt explicit. Core never performs the append or releases labels. + +#### V3 case release, combinations, and proof continuity + +```python +@dataclass(frozen=True) +class ProofCandidateIdentityV3: + component_arm_id: str + observation_digest: str + family_id: str + scope: ScopeKey + finding_location_digest: str + change_input_digest: str + candidate_digest: str + +@dataclass(frozen=True) +class ReleasedArmCaseBindingV3: + key: ArmScopeKey + baseline_digest: str + observation_slot_digest: str + observation_digest: str + underlying_push_digest: str + case_id_digest: str + role: AdmissionCaseRole + pair_id_digest: str | None + case_result_digest: str + evidence_input_digests: tuple[str, ...] + change_input_digests: tuple[str, ...] + input_manifest_digest: str + proof_candidate_digest: str | None + binding_digest: str + +@dataclass(frozen=True) +class AdmissionCaseReleaseV3: + plan_manifest_digest: str + admission_manifest_digest: str + authority_digest: str + preparation_digest: str + spend_binding_digest: str + bindings: tuple[ReleasedArmCaseBindingV3, ...] + payload_digest: str + review: IndependenceAttestation + release_digest: str + +@dataclass(frozen=True) +class ArmAdmissionCaseResultV3: + key: ArmScopeKey + observation_slot_digest: str + observation_digest: str + underlying_push_digest: str + case_id_digest: str + role: AdmissionCaseRole + pair_id_digest: str | None + evidence_inputs: tuple[EvidenceValidationInput, ...] + claims: tuple[DefectClaim, ...] + satisfactions: tuple[BoundedSatisfaction, ...] + silence: tuple[BoundedSilence, ...] + proof_candidate: ProofCandidateIdentityV3 | None + proof_attachments: tuple[ProofAttachment, ...] + coverage: CoverageFunnel + input_manifest: InputManifest + input_receipts: tuple[InputReadReceipt, ...] + complexity: ChangeComplexity + budget: BudgetDerivation + total_reply_ms: int + result_digest: str + +@dataclass(frozen=True) +class ArmAdmissionCasesV3: + arm_id: str + baseline_digest: str + cases: tuple[ArmAdmissionCaseResultV3, ...] + cases_digest: str +``` + +The release is the sole post-spend authority for arm, scope, role, pair, result, evidence, +inputs, and proof-candidate association. It is an exact V3 type; V2 release values and +subclasses are refused. Component arms retain their own cases, metrics, gates, and reports. +A combined arm's `CombinationLineageV3` names every component arm, observation digest, +combination rule, and retained candidate. The rule is a finite canonical decision table +over normalized component observation outcomes with a fixed identical-candidate-only proof +policy. Core replays the table; arbitrary combination code is not accepted. Missing, extra, +replaced, or contradictory component material invalidates the combined evaluation. + +A proof candidate is non-authoritative before release. After release, the existing proof +validator applies only when family, scope, pair roles, inputs, location, differential, +control, producer, verification receipt, and trust policy match the released candidate. +Absent or inconclusive proof preserves not-demonstrated status. A combined arm can retain +only the identical candidate from a named component observation. + +#### Exact metrics, reports, and complete replay + +```python +@dataclass(frozen=True) +class ScopedMetricResultV3: + key: ArmScopeKey + metric_id: MetricId + numerator: int + denominator: int + sample_count: int + value: float | None + interval_method: IntervalMethod + interval_lower: float | None + interval_upper: float | None + +@dataclass(frozen=True) +class ScopeAdmissionResultV3: + key: ArmScopeKey + metrics: tuple[ScopedMetricResultV3, ...] + instrument_gate: GateResult + budget_gate: GateResult + readiness_gate: GateResult + quality_gate: GateResult + decision: AdmissionDecision + result_digest: str + +class DegradationReasonV3(Enum): + INSTRUMENT = "instrument" + MODEL = "model" + BUDGET = "budget" + COVERAGE = "coverage" + DEFERRED = "deferred" + +@dataclass(frozen=True) +class ArmOperationalEvidenceV3: + completed_count: int + deferred_count: int + completed_case_digests: tuple[str, ...] + deferred_case_digests: tuple[str, ...] + stage_funnel: CoverageFunnel + instrument_failures: tuple[str, ...] + model_failures: tuple[str, ...] + runtime_ms: int + degradation_reasons: tuple[DegradationReasonV3, ...] + evidence_digest: str + +@dataclass(frozen=True) +class WholePopulationCoverageV3: + underlying_push_count: int + completed_underlying_push_count: int + deferred_underlying_push_count: int + coverage_digest: str + +@dataclass(frozen=True) +class ArmEvaluationReportV3: + baseline: EvaluationBaselineIdentityV3 + scoped_results: tuple[ScopeAdmissionResultV3, ...] + operational_evidence: ArmOperationalEvidenceV3 + combination_lineage: CombinationLineageV3 | None + report_digest: str + +@dataclass(frozen=True) +class AggregateWeightV3: + key: ArmScopeKey + numerator: int + denominator: int + weight_digest: str + +@dataclass(frozen=True) +class EvaluationAggregateV3: + constituents: tuple[ArmScopeKey, ...] + sample_counts: tuple[int, ...] + weights: tuple[AggregateWeightV3, ...] + metrics: tuple[AggregateMetricV3, ...] + aggregate_digest: str + +@dataclass(frozen=True) +class AggregateMetricV3: + metric_id: MetricId + weighted_value: float + constituent_metric_digests: tuple[str, ...] + metric_digest: str + +@dataclass(frozen=True) +class EvaluationReplayV3: + plan_manifest: EvaluationPlanManifestV3 + feasibility: EvaluationFeasibilityV3 + admission_manifest: EvaluationAdmissionManifestV3 + authority: EvaluationAdmissionAuthorityV3 + preparation: EvaluationPreparationV3 + receipt: PopulationUseReceipt + spend_binding: EvaluationSpendBindingV3 + case_release: AdmissionCaseReleaseV3 + cases_by_arm: tuple[ArmAdmissionCasesV3, ...] + reports: tuple[ArmEvaluationReportV3, ...] + combination_lineages: tuple[CombinationLineageV3, ...] + attempt_history_head_digest: str + whole_population_coverage: WholePopulationCoverageV3 + aggregate: EvaluationAggregateV3 | None + replay_digest: str +``` + +Vulnerable/fixed recall and pair metrics use only independently verified cases from the +exact `ArmScopeKey`. Precision uses only findings and cases assigned to the same key. +An eligible ordinary push projects once into each family claimed by that arm in the push's +reviewed ecosystem and never into another ecosystem. Whole coverage deduplicates by +`underlying_push_digest`. Each standard metric carries numerator, denominator, sample count, +value, and Wilson 95% bounds. A post-spend zero or census mismatch is invalid and prevents +report construction. + +`EvaluationAggregateV3` deliberately has no gate or decision. Its weights are canonical +non-negative rational values, sum exactly to one by integer cross-multiplication, and correspond one-to-one with its +constituents and sample counts. A replay is constructible +only when its report identities exactly equal all planned arms, every arm report contains +all exact scopes, and every combined lineage preserves its components. Replay validation +re-resolves current trust/custody state and rederives identities, counts, metrics, gates, +decisions, reports, and replay equality. + +#### Public service interface + +```python +def preflight_evaluation( + manifest: EvaluationPlanManifestV3, + context: AdmissionValidationContext, + resolver: ContentResolver, +) -> EvaluationFeasibilityV3: ... + +def authorize_evaluation( + manifest: EvaluationAdmissionManifestV3, + feasibility: EvaluationFeasibilityV3, + context: AdmissionValidationContext, + custody: PopulationCustodyPort, + resolver: ContentResolver, +) -> EvaluationPreparationV3: ... + +def score_and_replay( + admission_manifest: EvaluationAdmissionManifestV3, + feasibility: EvaluationFeasibilityV3, + preparation: EvaluationPreparationV3, + receipt: PopulationUseReceipt, + release: AdmissionCaseReleaseV3, + cases_by_arm: tuple[ArmAdmissionCasesV3, ...], + context: AdmissionValidationContext, + custody: PopulationCustodyPort, + resolver: ContentResolver, +) -> EvaluationReplayV3: ... + +def validate_evaluation_replay( + replay: EvaluationReplayV3, + context: AdmissionValidationContext, + custody: PopulationCustodyPort, + resolver: ContentResolver, +) -> tuple[ArmEvaluationReportV3, ...]: ... + +def authorize_fitted_component_v3( + evidence: FittedComponentEvidence, + probe_inputs: tuple[ProbeEvaluationInput, ...], + fitting_lineage: LineageManifest, + repository_holdout: RepositoryHoldoutReport, + replay: EvaluationReplayV3, + key: ArmScopeKey, + context: AdmissionValidationContext, + custody: PopulationCustodyPort, + resolver: ContentResolver, +) -> None: ... +``` + +Preconditions require exact V3 root types and canonical ordering. The fitted consumer first +validates the complete replay, selects exactly one planned arm/scope, requires its completed +GO result, and refuses a raw arm report, aggregate, V2 replay, incomplete graph, or ambiguous +selection. It then performs the unchanged fitted-component checks: it replays every probe, +validates fitting lineage, proves the resolved training and repository-holdout populations +are disjoint, binds the selected model declaration and component, and compares derived probe +digests. V3 replaces only the admission evidence source; it does not weaken 13.2. + +### Error and Result Semantics + +- Pre-spend insufficiency is returned as `EvaluationFeasibilityV3` with + `may_spend=False`; no preparation can be constructed. +- Unreadable observation bytes, unknown/mixed versions, stale custody, disposition change, + or incomplete slots are validation errors before spend. +- After spend, missing cases/reports, invalid release/proof, denominator zero/mismatch, + skipped stages, or unread inputs invalidate completion. They do not become insufficiency + or quality decisions. +- GO/NO-GO exists only inside completed `ScopeAdmissionResultV3` objects. +- Aggregate and per-arm report objects have no consumer-authorization method. + +### Testing Strategy + +- V2 compatibility: freeze representative canonical bytes, digests, decisions, failures, + and replay; prove helper extraction and V3 imports do not change them (`16.1–16.4`). +- Version/identity: reject V2-as-V3, unknown schemas, subclasses, mixed children, and every + single-field baseline mutation; require a new baseline (`16.3–16.6`). +- Plan/feasibility: cover each arm/scope Cartesian row, threshold reachability, + minimum-size and interval precision, one insufficient row blocking all spend, and no arm + launch or label read (`17.1–17.4`). +- Observation/authority: mutate, remove, swap, truncate, zero, or cross-bind every slot; + reject self-review, stale custody, broken/forked attempt history, raw-access disposition, + changed predecessor retry identity, and any sealed event after a terminal event + (`17.5–17.12`). +- Atomic spend: prove one request/receipt/spend binding covers all planned baselines, + concurrent losers receive no labels, and no subset preparation is constructible + (`17.7–17.9`). +- Exact metrics: exercise multi-family fixes, same-ecosystem ordinary projection, + cross-ecosystem refusal, exact precision, Wilson fields, whole-push deduplication, + aggregate weights/no decision, and post-spend zero/mismatch invalidity (`18.1–18.8`). +- Release/combinations/proof: reject V2 release, role/pair/scope/arm/input/result swaps, + missing components, contradictory combined results, candidate fabrication/transfer, and + invalid attachments; retain absent proof as not demonstrated (`19.1–19.11`). +- Reports/replay: derive every scoped GO/NO-GO, preserve whole operational evidence, + reject missing gate/arm/scope/custody/release/lineage, rederive all source values, and + authorize consumers only from complete replay (`20.1–20.8`). +- Synthetic end to end: two component arms plus one combined arm over at least two families + and two ecosystems pass plan review, per-scope preflight, sealed observations, one shared + spend, V3 release, scoped reporting, complete replay, and exact fitted selection. + +### Migration Strategy + +1. Freeze V2 accepted outputs and failure behavior. +2. Add V3 measurement identities, plan declaration, exact policies, and preflight. +3. Add sealed observations, reviewed admission manifest, authority, and spend binding. +4. Add V3 release, exact metrics, reports, and complete replay. +5. Add the separate fitted-component V3 consumer and strict schema router. +6. Run synthetic multi-arm conformance and independent validation. +7. Unblock valid-detection-benchmark task generation only after the predecessor schema + `openultrasast.detection-core.evaluation-admission.v3` is exported and validated. + +There is no V2-to-V3 migration. Existing V2 artifacts remain replayable under V2; every V3 +evaluation starts a new baseline. Rollback removes the additive V3 facade/modules and leaves +the frozen V2 path unchanged. + + ## Data Models ### Domain Model and Invariants diff --git a/.kiro/specs/detection-core/requirements.md b/.kiro/specs/detection-core/requirements.md index 6d26bff..53e9e0a 100644 --- a/.kiro/specs/detection-core/requirements.md +++ b/.kiro/specs/detection-core/requirements.md @@ -23,20 +23,32 @@ object and decision vocabulary and may use a bounded model program to answer res questions over that evidence. A staged readiness contract shall stop a family at the first failing semantic layer, before population-level admission is attempted. +**Evaluation-admission V3 amendment, 2026-11-06:** the validated version-two admission +contract pools all claimed scopes and represents one arm. The approved replacement benchmark +requires a new contract that evaluates multiple component and combined arms under one population +spend while preserving an independently interpretable result for every exact vulnerability-family +and ecosystem scope. Existing version-two evidence remains historical and unchanged. + ## Boundary Context - **In scope**: the family-neutral evidence, outcome, trace, coverage, provenance, budget, staged-readiness, admission, measurement, fact-pack, model-handoff, and consumer - contracts; family conformance requirements; failure-visible semantic-stage accounting. + contracts; family conformance requirements; failure-visible semantic-stage accounting; + and the versioned exact-scope, multi-arm evaluation-admission validation and replay contract. - **Out of scope**: vulnerability-family detection algorithms and prompt content; framework-specific fact content; model optimization; analysis of vendored dependency source; the pre-push transaction and rollout policy; proof execution; enabling a new - vulnerability family for users. + vulnerability family for users; population sourcing, curation, reservation, licensing, + and lifecycle; durable object-store or compare-and-swap adapters and bucket administration; + detector execution or isolation; observation storage, access enforcement, and retry + orchestration; workflow heartbeat and status presentation; and operator notifications. - **Adjacent expectations**: the existing deterministic analysis is an evidence producer, not a required complete prover; family-specific DSPy programs may consume the evidence packet under the model-handoff contract; proof-based blocking remains a separate - consumer. A parked arm remains parked unless a separately reviewed strategy creates a - new arm, baseline, and admission population. + consumer. Benchmark-owned workflows supply the reviewed population, lifecycle, custody, + sealed-observation, access-control, and post-spend release evidence that the core validates; + the core does not create or operate those workflows. A parked arm remains parked unless a + separately reviewed strategy creates a new arm, baseline, and admission population. ## Requirements @@ -292,6 +304,80 @@ The detection core shall require one fixed measurement schema for every detectio 5. When a later-stage metric is reported, the detection core shall include the results and digests of every prerequisite readiness stage. +## Evaluation-Admission V3 Amendment + +Requirements 16–20 extend Requirements 12 and 15 for a new evaluation-admission contract. Requirements 1–15 and completed version-two evidence retain their existing meaning, identities, and validation behavior; the amendment does not reinterpret or upgrade them. + +### Requirement 16: Versioned evaluation-admission compatibility +The detection core shall provide a new evaluation-admission contract for exact-scope, multi-arm qualification without changing the meaning or validation of historical admission artifacts. + +#### Acceptance Criteria +1. When a new evaluation admission is requested, the detection core shall validate it under a distinct contract version and shall keep every earlier admission contract independently replayable. +2. The detection core shall preserve the accepted bytes, digests, decisions, and replay behavior of existing version-two admission artifacts. +3. If pooled version-two evidence is presented as exact-scope version-three evidence, the detection core shall refuse the conversion and shall require a new baseline evaluated under the new contract. +4. If an admission artifact names an unknown, mixed, or incompatible contract version, the detection core shall refuse it rather than infer a compatible shape. +5. When a population, evaluation plan, arm identity, claimed scope, denominator policy, threshold set, judgment vocabulary, analyzer identity, fact identity, configuration identity, registered budget, execution identity, outcome mapping, or combination rule changes, the detection core shall require a new baseline identity. +6. The detection core shall prevent an insufficient result, impaired result, aggregate result, incomplete arm result, raw report, or other partial artifact from authorizing consumer use. + +### Requirement 17: Reviewed multi-arm plan and atomic spend authority +The detection core shall bind every arm evaluated against one untouched population to a reviewed label-blind plan and one atomic admission spend. + +#### Acceptance Criteria +1. When an evaluation plan is registered before label access, the detection core shall require a reviewed label-blind plan naming the population, input-set identity, every component arm, every combined arm, and every expected observation slot. +2. When an arm enters an evaluation plan, the detection core shall require its immutable baseline identity, exact claimed family-and-ecosystem scopes, denominator policy, thresholds, sufficiency rules, outcome mapping, and combination lineage where applicable. +3. When the label-blind plan is reviewed, the detection core shall evaluate threshold reachability and evidence sufficiency independently for every planned arm and exact claimed scope. +4. If any planned arm or claimed scope cannot meet its registered minimum-size, threshold-reachability, or interval-precision rule, the detection core shall report that arm and scope as insufficient and shall refuse the shared population spend. +5. When every planned observation is complete and before label access, the detection core shall require an admission authority reviewed independently from its producer that binds the feasible plan to the exact completed observation identities and the current population custody state. +6. If an expected observation is absent, unreadable, changed, associated with another arm, input, or plan, or inconsistent with the feasible plan, the detection core shall refuse admission authorization. +7. When shared labels are released, the detection core shall require one admission preparation, one exact monotonic population transition, and one resulting label-release receipt binding every planned arm. +8. If the custody head is stale, the transition is not the exact available-to-spent successor, the receipt is absent or mismatched, or availability is later restored, the detection core shall refuse label release and scoring. +9. The detection core shall not authorize a partial spend for a subset of planned arms or scopes. +10. While the population remains unspent, the detection core shall permit admission reuse only when raw observations and population-specific diagnostics remain unavailable to detector development and only closed process status, counts, and digests are exposed. +11. If raw observations or population-specific diagnostics are inspected, or an arm, plan, configuration, registered budget, or execution identity changes after an attempt, the detection core shall require a development or spent disposition and shall refuse that population for admission. +12. When a pre-spend attempt is retried, the detection core shall accept the retry only under the exact unchanged population, plan, arms, inputs, configuration, registered budgets, and execution identities. + +### Requirement 18: Exact-scope denominators and whole-population coverage +The detection core shall derive qualification independently for each arm's exact vulnerability-family and ecosystem scope while preserving separate whole-population coverage. + +#### Acceptance Criteria +1. When quality is evaluated, the detection core shall derive one complete standard metric set for every exact arm, vulnerability-family, and ecosystem scope declared in the reviewed plan. +2. When recall or pair correctness is derived for a scope, the detection core shall use only vulnerable and fixed cases independently verified for that exact family and ecosystem. +3. When ordinary false alarms are derived for an arm, the detection core shall count each eligible ordinary push once for every family claimed by that arm in the push's reviewed ecosystem and shall not project it into another ecosystem. +4. When precision is derived for a scope, the detection core shall use only findings and cases assigned to that exact arm, family, and ecosystem under the reviewed denominator policy. +5. When one underlying push contributes to multiple verified scoped denominators, the detection core shall count it once in whole-population coverage. +6. When a scoped metric is reported, the detection core shall include its numerator, denominator, value, sample count, and Wilson 95% interval under the fixed metric algebra. +7. Where an overall aggregate is reported, the detection core shall identify its constituent arms and scopes, sample counts, and weighting and shall not attach a gate or authorization decision to the aggregate. +8. If a post-spend scoped denominator is zero or differs from the reviewed label-blind census and feasible plan, the detection core shall refuse the evaluation as invalid rather than report insufficient, go, or no-go. + +### Requirement 19: Version-three case release, arm results, and proof binding +The detection core shall validate post-spend evidence separately for every planned arm while preserving shared population custody and combination lineage. + +#### Acceptance Criteria +1. When post-spend cases are released, the detection core shall require one version-three case-release graph reviewed independently from its producer and bound to the reviewed evaluation plan, shared spend, and exact completed observations. +2. When a case is released for an arm, the detection core shall bind its arm identity, exact scope, role, pair identity, result, evidence, change inputs, reviewed input manifest, and release review. +3. If a version-two case release is supplied to a version-three evaluation, the detection core shall refuse it rather than reinterpret it under the new contract. +4. When a combined arm is evaluated, the detection core shall preserve every component arm's observations, metrics, gates, and decision and shall report the registered combination lineage. +5. If a combined result omits, replaces, or contradicts a required component result, the detection core shall refuse the combined evaluation. +6. While ground-truth roles remain sealed, the detection core shall treat every label-blind proof candidate as non-authoritative for demonstrated status. +7. When a proof-backed status is derived after case release, the detection core shall require an exact validated proof attachment matching the released family, scope, pair roles, change inputs, finding location, differential record, control binding, producer, verification receipt, and trust policy. +8. When a combined arm retains proof evidence, the detection core shall require the identical validated proof candidate from the matching component observation and shall prevent the combined arm from creating or transferring proof evidence. +9. If proof is absent or inconclusive, the detection core shall retain the finding as not demonstrated. +10. If a proof attachment is role-swapped, cross-family, cross-scope, location-mismatched, or bound to unrelated inputs or controls, the detection core shall refuse the attachment as invalid and shall not count it as demonstrated. +11. The detection core shall not allow one planned arm's valid result to hide another planned arm's missing, invalid, or failed result. + +### Requirement 20: Complete evaluation reports and replay authority +The detection core shall make the complete multi-arm evaluation replay the only admission artifact capable of authorizing downstream consumption. + +#### Acceptance Criteria +1. When a planned arm completes evaluation, the detection core shall report one go or no-go decision for every exact claimed scope from that arm's validated instrument, budget, readiness, and quality gates. +2. The detection core shall reserve go and no-go for completed quality evaluations and shall keep pre-spend insufficiency, unread input, skipped stages, incomplete arms, and other instrument failures distinct from quality decisions. +3. When an evaluation report is produced, the detection core shall preserve per-arm scoped metrics separately from whole-population coverage, completed and deferred work, coverage funnels, instrument and model failures, runtime, and every degradation reason. +4. When every planned arm has a valid report, the detection core shall produce one complete evaluation replay binding the reviewed plans, feasibility results, admission authority, shared custody transition and receipt, the version-three case-release graph, independently valid per-arm scoped reports, and combination lineage. +5. If any planned arm report, mandatory gate, custody binding, case release, or replay component is missing or invalid, the detection core shall refuse evaluation completion. +6. When an evaluation replay is validated, the detection core shall rederive identities, counts, metrics, gates, and decisions from the canonical source artifacts rather than trust caller-supplied summaries. +7. When a consumer requests admission evidence, the detection core shall provide authorization only from a complete validated evaluation replay and shall not authorize from a per-arm report or aggregate alone. +8. When a zero or clean-looking arm result is included, the detection core shall require nonzero input-read evidence and execution evidence for every required stage before accepting the replay. + ## Out of Scope - The pre-push hook transaction, rollout policy, and capability registry. - Vulnerability-family algorithms, prompts, examples, and optimization programs. @@ -300,3 +386,7 @@ The detection core shall require one fixed measurement schema for every detectio - Proof execution and sandbox orchestration. - Enabling a vulnerability family for advisory or blocking use. - Analysis-engine, frontend, graph, and family-internal implementation details. +- Population sourcing, human curation, permanent reservation, licensing, and lifecycle operations. +- Durable object-store or compare-and-swap adapter implementation, bucket administration, and retention configuration. +- Detector execution and isolation, observation storage and access enforcement, and retry orchestration. +- Workflow heartbeat and operator-status presentation, notification transport, and product rollout. diff --git a/.kiro/specs/detection-core/research.md b/.kiro/specs/detection-core/research.md index c6a2254..c9096d2 100644 --- a/.kiro/specs/detection-core/research.md +++ b/.kiro/specs/detection-core/research.md @@ -711,3 +711,365 @@ requirements approval. legacy imports and then proves those attributes are absent after cutover. - **Historical artifacts become unreadable:** frozen trace/report bytes and hashes continue through the independent read-only decoder before and after namespace deletion. + +--- + +## Gap analysis: exact-scope multi-arm evaluation admission V3 (2026-11-06) + +### Scope, approval state, and instrument check + +- **Feature type:** complex brownfield contract extension. Requirements 16–20 add a new + evaluation-admission root beside the implemented V2 admission graph; they do not replace + the family-neutral evidence contracts in Requirements 1–15. +- **Approval warning:** Requirements 16–20 are generated but not approved. This gap analysis + is permitted to inform review, but `design.md` cannot be regenerated until the requirements + are approved or `/kiro-spec-design detection-core -y` is invoked. +- **Discovery method:** integration-focused source inspection of the admission contracts, + their tests, their only production authorization consumer, and the approved + `valid-detection-benchmark` consumer design. No external dependency research was needed: + the amendment is a standard-library Python contract, validation, and replay change. +- **Instrument proof:** the requirements (32,917 bytes), current design (130,524 bytes), + existing research (45,818 bytes), approved valid-benchmark design (72,402 bytes), ten + primary admission/core modules (166,630 bytes), and the relevant admission/measurement + tests (270,157 bytes) were opened as nonzero inputs before interpreting gaps. + +### Current architecture and reusable assets + +The implemented V2 admission path is a closed canonical graph: + +`BaselineIdentity` → reviewed `AdmissionManifestV2` and census → context validation and +pooled feasibility → one `AdmissionPreparation` → exact `AVAILABLE`-to-`SPENT` custody +receipt → independently reviewed `AdmissionCaseRelease` → revalidated case results → +one `AdmissionReportV2` → one `AdmissionReplay`. + +The following machinery is reusable without changing its meaning: + +- canonical JSON, content digests, closed-graph checks, exact-dataclass checks, and strict + enum/type validation; +- `ScopeKey`, the fixed metric identifiers and algebra, Wilson 95% computation, readiness + validation, review-independence attestations, content resolution, and provenance checks; +- the append-only custody log and exact monotonic `AVAILABLE`-to-`SPENT` receipt validation; +- case evidence, input, budget, coverage, and proof-attachment validation after a V3 release + establishes the correct arm/scope association; +- V2 report and replay rederivation, which must remain independently replayable and + byte-compatible. + +The reusable implementation is not the V2 aggregate root itself. V2 has one arm identity, +assigns every metric rule the full claimed-scope tuple, pools capacities and cases across +those scopes, derives one unscoped result per metric, and emits one arm-level decision. That +shape cannot express the exact per-arm, family, and ecosystem claims required by V3. + +### Requirement-to-asset map + +| Requirement | Existing asset | Gap classification and impact | +|---|---|---| +| 16.1–16.4 Version isolation | Canonical schemas, exact-type replay validation, historical decoder patterns | **Missing / compatibility constraint:** no V3 root or explicit version dispatcher exists. Extending V2 dataclasses changes accepted bytes and digests; inferred conversion would make pooled evidence appear exact. | +| 16.5 Baseline identity | `BaselineIdentity` binds one arm, population/freeze, scopes, facts, analyzer, configuration, denominator, thresholds, and outcomes | **Missing:** plan, input-set, sufficiency, registered budget, execution, observation-set, and combination-rule identities are absent. These must belong to a distinct V3 identity rather than changing V2. | +| 16.6 Non-authorizing partial artifacts | V2 replay, not a raw report, authorizes fitted-component use | **Partial:** there is no V3 completion root or closed status distinction for insufficiency, impairment, aggregates, incomplete arm results, and raw per-arm reports. | +| 17.1–17.4 Reviewed multi-arm preflight | Independent V2 manifest/census review and threshold feasibility | **Missing:** no reviewed plan containing every component/combined arm and observation slot; no per-arm/per-scope sufficiency rules; V2 feasibility sums capacities across selected scopes and therefore cannot veto the shared spend independently for each exact scope. | +| 17.5–17.6 Post-observation authority | V2 authorization revalidates context, feasibility, readiness, and custody | **Missing:** no second reviewed manifest binding the feasible plan to exact sealed observation identities, no observation-byte resolution, and no all-slot completeness check immediately before spend. | +| 17.7–17.9 Atomic shared spend | Exact V2 preparation and custody successor validation | **Constraint:** V2 `PopulationUseRequest` names one arm. No preparation/receipt binds every planned arm and scope, and no root rejects a partial subset while retaining one shared custody transition. | +| 17.10–17.12 Confidential retry/disposition | Append-only custody and lineage policies | **Missing / adjacent boundary:** core lacks typed retry identity and development/spent disposition validation for raw-observation access. The benchmark owns access enforcement and lifecycle events; core must validate their immutable authority and exact retry identity, not implement storage or access control. | +| 18.1–18.4 Exact scoped metrics | Fixed metric algebra, Wilson intervals, reviewed scope census, validated cases | **Missing:** `standard_metric_rules()` assigns all scopes to every rule; `MetricResult` and `AdmissionThreshold` have no scope key; context requires pooled rule scopes; feasibility and scoring derive once across the union. | +| 18.3, 18.5 Ordinary projection and whole coverage | Case roles and an intermediate whole-population funnel | **Missing:** ordinary cases lack a stable underlying-push identity that permits once-per-family projection inside the reviewed ecosystem while deduplicating the same push once for whole coverage. Whole-population evidence is not preserved in the final V2 report. | +| 18.6–18.8 Scoped reporting, aggregates, post-spend validity | Metric numerator/denominator/value/Wilson fields | **Partial:** metric arithmetic is reusable, but scope-qualified results and non-authorizing aggregate identities/weights do not exist. Post-spend zero or census mismatch must be invalid, not a zero-valued metric, insufficiency, GO, or NO-GO. | +| 19.1–19.3 V3 case release | Strong independent V2 release with exact role/pair/result/evidence/input binding | **Missing:** `ReleasedCaseBinding` has no arm identity, and the release binds one V2 manifest and authorization. Exact V2/V3 type separation is required. | +| 19.4–19.5 Combination lineage | No equivalent admission root | **Missing:** there is no graph preserving every component observation/report/gate while binding a combined arm to its registered deterministic lineage or detecting omission, replacement, and contradiction. | +| 19.6–19.10 Proof continuity | Strong `ProofAttachment` validation | **Partial:** the proof contract can be reused after V3 release, but no label-blind proof-candidate identity is carried from the component observation to prevent a combined arm from creating or transferring proof evidence. | +| 19.11 Arm independence | Exact V2 replay for one arm | **Missing:** no all-planned-arm cardinality and identity check prevents one valid report from hiding another arm's missing, invalid, or failed result. | +| 20.1–20.2 Scoped decisions and failure classes | V2 mandatory gates and GO/NO-GO derivation | **Missing:** V2 emits one pooled arm decision. V3 needs a completed decision per exact scope and separate non-quality states for insufficiency, unread inputs, skipped stages, impairment, and incomplete arms. | +| 20.3 Complete reports | `MeasurementRecordV2`; intermediate `AdmissionMetricEvidence` carries an extra whole funnel/runtime | **Partial:** completed/deferred work, whole coverage, degradation reasons, and complete instrument failures are not preserved in the final report; `_measurement()` currently supplies `instrument_failures=()`. | +| 20.4–20.7 Complete replay authority | `AdmissionReplay` rederives one V2 report from authoritative inputs | **Missing:** no multi-arm replay binds both reviewed stages, shared feasibility/authority/custody/release, all independently valid scoped reports, and combination lineage. `admission_fitted.py` accepts only V2 `AdmissionReplay` and derives authorization from its single report. | +| 20.8 Nonzero execution evidence | Coverage/input validation and clean-measurement checks | **Partial:** reusable case evidence exists, but V3 completion must require it independently for every arm and exact scope before accepting a clean-looking result. | + +### Concrete integration surfaces + +The V3 amendment affects these core responsibilities while retaining V2 definitions: + +- `src/openultrasast/core/measurement.py`: reusable scope and metric algebra; new V3 + baseline, scope-qualified metric/policy/threshold, aggregate, and sufficiency contracts. +- `src/openultrasast/core/admission_review.py`: new reviewed two-stage plan and admission + manifests with canonical per-arm plans and expected/actual observation bindings. +- `src/openultrasast/core/admission_feasibility.py`: independent feasibility witnesses for + every arm and exact scope plus one shared non-spend result. +- `src/openultrasast/core/admission_context.py`: V3 plan, identity, policy, readiness, + observation, and lineage validation without the V2 all-scopes-equal rule. +- `src/openultrasast/core/admission_spend.py`: a distinct V3 authority/preparation binding + every planned arm and the current custody head; V2 functions remain unchanged. +- `src/openultrasast/core/admission_case.py`: an arm-aware V3 case-release graph bound to + the plan, exact observations, authority, and shared receipt. +- `src/openultrasast/core/admission_metrics.py`: exact-scope derivation, ordinary-push + projection within ecosystem, whole-population deduplication, and proof-candidate + continuity. +- `src/openultrasast/core/admission_report.py`: independently valid per-arm/per-scope + reports, non-authorizing aggregates, and explicit non-quality failure classes. +- `src/openultrasast/core/_admission_replay.py`: one closed `EvaluationReplayV3` containing + the complete shared graph and all planned reports. +- `src/openultrasast/core/admission_fitted.py`: explicit version dispatch; V3 authorization + accepts only a complete validated evaluation replay and an exact arm/scope selection, + never a raw report or aggregate. +- `tests/test_core_admission_*.py`, `tests/test_core_measurement*.py`, and frozen V2 + fixtures: compatibility, exact-scope algebra, all-arm completeness, mixed-version + refusal, atomic-spend, proof-continuity, and replay conformance. + +The approved benchmark remains a downstream consumer. Its current +`benchmarks/valid/admission_run.py`, `admission_scoring.py`, and `population_custody.py` +are V2 single-arm paths and stay compatibility-only. Population lifecycle, durable object +storage/CAS, observation access, detector execution, retry orchestration, and operator +presentation remain benchmark-owned and are not implementation work for this amendment. + +### Implementation approach options + +#### Option A: extend V2 contracts in place + +Add arm/scope/plan fields to existing V2 identities, policies, manifests, releases, +reports, and replay. + +- **Strengths:** smallest initial module count and direct reuse of current call paths. +- **Risks / limitations:** changes canonical V2 bytes and digests, makes exact-type version + refusal ambiguous, overloads already dense single-arm modules, and risks accidental + projection of pooled evidence. It conflicts with 16.1–16.4. +- **Effort / risk:** L / High; not a viable compatibility path unless Requirement 16 is + weakened, which is not recommended. + +#### Option B: create a fully parallel V3 stack and facade + +Introduce an independent `evaluation_admission` facade and V3 modules for identity, plan, +feasibility, authority, release, metrics, reports, and replay. + +- **Strengths:** clearest public version boundary, exact-type refusal, and protection of V2 + bytes and behavior; components are testable in isolation. +- **Risks / limitations:** duplicates canonical validation, custody, readiness, proof, and + metric plumbing unless reuse is disciplined; two large stacks increase maintenance cost. +- **Effort / risk:** XL / Medium-High. + +#### Option C: parallel V3 public roots with version-neutral internal helpers + +Keep V2 and V3 aggregate roots, facades, schemas, and validators exact and separate. Reuse +or extract only helpers whose inputs and outputs have no serialized V2 identity, such as +Wilson arithmetic, review validation, custody-chain checks, proof validation, and canonical +collection validation. + +- **Strengths:** preserves strict version boundaries and V2 replay while avoiding avoidable + duplication; aligns with the current closed-graph and ports-and-adapters pattern. +- **Risks / limitations:** helper extraction can still change V2 behavior if performed + before compatibility fixtures lock accepted bytes and errors; dependency direction and + exact-type tests must be explicit. +- **Effort / risk:** XL / High. +- **Design-phase candidate:** this offers the best balance, using Option B-style public + separation. It remains a candidate until the design review resolves the research items + below. + +### Complexity and risk + +- **Effort: XL (2+ weeks).** The amendment adds two-stage review, exact-scope algebra, + multi-arm completeness, one shared spend, a new release graph, per-arm reports, consumer + dispatch, and a closed replay root across multiple existing contract modules. +- **Risk: High.** The main failure modes are silent V2 behavior drift, pooled-to-scoped + misinterpretation, partial-arm authorization, stale observation/custody acceptance, + ordinary-push double counting, and an aggregate or raw report becoming an authorization + token. + +### Design-phase recommendations and research needed + +1. Freeze representative accepted V2 canonical bytes, digests, decisions, validation + failures, and replay outputs before extracting helpers. Treat any change as a regression. +2. Define one explicit V3 facade/schema family and refuse unknown, mixed, subclassed, or + substituted V2/V3 roots at every public boundary. +3. Resolve the exact V3 module split and aggregate roots: per-arm plan, reviewed plan + manifest, observation binding, admission authority/preparation, V3 release, scoped + report, and evaluation replay. +4. Define the observation-slot identity and content-resolution contract, including how an + observation proves arm, input, execution, budget, configuration, and plan identity + without exposing raw diagnostics before spend. +5. Define a stable underlying-push identity for ordinary projection and whole-population + deduplication, including multi-family fixes and ecosystem review authority. +6. Define sufficiency and interval-precision rules as reviewed per-arm/per-scope contracts, + separate from post-spend invalidity and completed GO/NO-GO gates. +7. Define the benchmark-supplied retry/development disposition evidence that core validates; + do not move durable lifecycle, access enforcement, or CAS implementation into core. +8. Define component-to-combined observation lineage and proof-candidate identity so a + combined arm can retain only the identical component candidate and cannot suppress a + component report. +9. Define V3 fitted-component consumption as complete-replay validation plus exact arm/scope + selection. Per-arm reports, aggregates, incomplete roots, and V2 artifacts cannot + authorize a V3 consumer. +10. Derive the V3 conformance suite from 16.1–20.8: frozen V2 compatibility; unknown/mixed + version refusal; per-arm/per-scope feasibility; missing observation and stale custody + refusal; one atomic spend; exact ordinary projection/dedup; post-spend zero invalidity; + proof continuity; all-arm completion; and nonzero input/stage execution evidence. + +### Gap conclusion + +The current implementation provides strong canonical validation, custody, evidence, +metric, proof, and replay primitives, but its V2 admission root is structurally unable to +represent an exact-scope multi-arm evaluation. The gap is therefore an additive versioned +contract and replay graph, not a correction to the V2 metric formula. The design should +retain V2 unchanged, introduce exact V3 roots, and reuse only version-neutral internals. +Design generation remains blocked on approval of Requirements 16–20. + +--- + +## Design discovery and synthesis: evaluation-admission V3 (2026-11-06) + +### Summary + +- **Feature:** detection-core evaluation-admission V3 amendment. +- **Discovery scope:** complex integration and immutable-contract version extension. +- **Selected pattern:** parallel V3 public aggregate roots with reuse limited to + version-neutral canonical, review, custody, proof, readiness, and metric primitives. +- **Key findings:** V2 aggregate roots cannot be generalized without changing their bytes; + the V3 plan needs a non-circular declaration/baseline identity; and one complete replay, + rather than a report or aggregate, must be the only downstream authorization source. + +### Research log + +#### Version and canonicalization boundary + +- **Context:** 16.1–16.4 require historical V2 bytes, digests, decisions, and replay to + remain unchanged while V3 refuses mixed or inferred shapes. +- **Sources consulted:** current V2 canonical and admission modules, frozen-dataclass and + exact-type patterns in the repository, Python 3.11 JSON documentation, and RFC 8785. +- **Finding:** the existing sorted-key compact JSON profile is sufficient for the current + single-Python-runtime contract. Adopting JCS would create a second canonical profile and + risk changing V2 bytes without satisfying an additional requirement. +- **Implication:** `evaluation_admission.py` is the only V3 facade. V3 validators require + exact dataclass types and exact V3 schema constants. A read router may select a validator + by schema but never converts V2 to V3. + +#### Non-circular plan and baseline identity + +- **Context:** every arm needs an immutable baseline, and a change to the evaluation plan + must start a new baseline without making the plan and baseline digests recursive. +- **Finding:** a reviewed plan declaration can be hashed before baseline construction. It + contains the population, input set, arm configurations, combination rules, and expected + observation slots. Each arm baseline then binds that declaration digest and its exact arm + configuration digest. +- **Implication:** any plan or arm mutation changes the declaration and therefore every arm + baseline, while the plan manifest can embed the derived baselines without a digest cycle. + +#### Two-stage review and shared spend + +- **Context:** arithmetic feasibility happens before detector execution; spend authority + requires every exact sealed observation and the current custody head afterwards. +- **Finding:** one manifest cannot safely serve both moments. The second authority must + resolve observation bytes and bind the earlier feasible plan rather than repeat or + summarize it. +- **Implication:** `EvaluationPlanManifestV3` supports label-blind preflight; + `EvaluationAdmissionManifestV3` binds exact sealed observations and retry/disposition + evidence. `EvaluationAdmissionAuthorityV3` is constructible only when both validate. + +#### Existing custody reuse + +- **Context:** 17.7–17.9 require one atomic population transition binding all arms, while + the existing custody graph already validates an exact monotonic transition. +- **Finding:** the existing request/receipt can remain unchanged if its influenced artifact + is the V3 authority digest, which transitively commits to every planned baseline. A V3 + spend binding makes that transitive relationship explicit inside the replay. +- **Implication:** no second custody implementation is introduced. The benchmark remains + the CAS writer and label releaser; core builds and validates the authority, preparation, + receipt binding, and replay relationship. + +#### Exact-scope projection and whole coverage + +- **Context:** one underlying push may contribute to several verified family scopes while + whole-population coverage must count it once. +- **Finding:** case identity alone is insufficient because vulnerable and fixed sides and + several arm/scope projections can refer to the same push. +- **Implication:** V3 release bindings carry a separate `underlying_push_digest` and exact + `ArmScopeKey`. Scoped metrics project from arm/scope bindings; whole coverage deduplicates + only by the underlying push identity. Ordinary projection is limited to families claimed + by the arm in the push's reviewed ecosystem. + +#### Combination and proof continuity + +- **Context:** a combined arm must preserve component results and cannot manufacture or + transfer demonstrated proof. +- **Finding:** proof validation alone establishes attachment correctness but does not prove + candidate continuity from a component observation to a combined observation. +- **Implication:** label-blind observations and released bindings carry a stable + proof-candidate digest. Combination lineage names exact component observation digests; + the combined arm may retain only an identical candidate named by a component. + +### Architecture pattern evaluation + +| Option | Description | Strengths | Risks / limitations | Decision | +|---|---|---|---|---| +| Extend V2 roots | Add plan, arm and scope fields to current types | Smallest initial file count | Changes V2 bytes and permits pooled/scoped ambiguity | Rejected | +| Fully parallel V3 implementation | Duplicate the entire validation stack | Strong version separation | Duplicates custody, proof, review and metric machinery | Rejected | +| Parallel V3 roots plus neutral helpers | Separate public schemas and aggregates; share only identity-neutral primitives | Preserves V2 and avoids duplicate trusted logic | Helper extraction needs frozen V2 regression evidence | Selected | + +### Design decisions + +#### Decision: one V3 facade and exact roots + +- `evaluation_admission.py` exports preflight, authorization, score/replay, and replay + validation. V2 admission entry points remain unchanged. +- V3 modules do not import V2 aggregate roots. `admission_fitted.py` exposes a separate V3 + consumer function rather than broadening the V2 function's accepted graph. + +#### Decision: plan declaration precedes baselines + +- `EvaluationPlanDeclarationV3` owns population, freeze, input set, arm configurations, + expected observation slots, and combination declarations. +- `EvaluationBaselineIdentityV3` binds the declaration digest and one arm configuration. + The reviewed manifest embeds both and proves the exact canonical derivation. + +#### Decision: insufficiency is a pre-spend result, not a quality decision + +- `EvaluationFeasibilityV3` contains exactly one row per declared arm/scope and derives one + `may_spend` value. Any insufficient row makes it false and prevents preparation. +- GO and NO-GO occur only in completed post-spend scope reports. Instrument impairment, + unread inputs, skipped stages, incomplete arms, and invalid post-spend denominators never + map to those decisions. + +#### Decision: reports are evidence; the complete replay is authority + +- Per-arm scoped reports and optional aggregates are serializable evidence but cannot + authorize consumers. +- `EvaluationReplayV3` binds both reviewed manifests, feasibility, authority, preparation, + shared receipt, spend binding, V3 release, exact cases, all planned reports, and + combination lineage. Validation rederives the graph and exact planned-arm cardinality. + +#### Decision: aggregate has no decision field + +- An aggregate identifies constituents, sample counts, weights, and derived values. Its + type contains neither gates nor authorization decisions, preventing an aggregate from + being mistaken for an admission result. + +### Synthesis + +- **Generalization:** expected observation slots, arm/scope keys, and complete replay + cardinality are three applications of one closed-plan principle: later artifacts must be + an exact projection of the reviewed plan, never a caller-selected subset. +- **Build versus adopt:** reuse the existing Python canonical, custody, review, readiness, + proof, and Wilson primitives. Do not adopt a new serializer, workflow engine, object-store + client, or transaction library because persistence and orchestration stay downstream. +- **Simplification:** use the existing custody request/receipt transitively through a V3 + authority and explicit spend binding; do not create a second custody state machine. + Keep lifecycle/access enforcement outside core and validate only immutable disposition + evidence supplied by the benchmark. + +### Risks and mitigations + +- **V2 drift during helper extraction:** freeze canonical V2 bytes, digests, decisions, + failures, and replay outputs before extraction; run them after every shared-helper change. +- **Partial-arm authorization:** derive report and replay cardinality from the plan and make + the complete replay the sole consumer input. +- **Observation substitution:** resolve nonzero bytes and require an exact one-to-one slot, + arm, input, execution, budget, configuration, and digest match before authority. +- **Ordinary-push double counting:** separate scoped projection keys from the deduplicated + underlying-push identity. +- **Combined proof fabrication:** require component observation lineage and identical + proof-candidate identity before accepting a retained attachment. +- **Core absorbing benchmark workflow:** keep storage, CAS, access enforcement, retries, + detector isolation, label joining, status, and presentation explicitly out of boundary. + +### References + +- [Python 3.11 JSON documentation](https://docs.python.org/3.11/library/json.html) — + deterministic sorted-key compact serialization controls used by the existing profile. +- [RFC 8785](https://www.rfc-editor.org/rfc/rfc8785.html) — evaluated but not adopted; + cross-runtime canonical signatures are outside the current boundary. +- [Approved valid-detection-benchmark design](../valid-detection-benchmark/design.md) — + downstream V3 consumer contract and ownership boundary. diff --git a/.kiro/specs/detection-core/spec.json b/.kiro/specs/detection-core/spec.json index 6e93450..3d62478 100644 --- a/.kiro/specs/detection-core/spec.json +++ b/.kiro/specs/detection-core/spec.json @@ -1,7 +1,7 @@ { "feature_name": "detection-core", "created_at": "2026-10-30T13:55:30Z", - "updated_at": "2026-11-02T07:51:51Z", + "updated_at": "2026-11-06T12:21:47Z", "language": "en", "phase": "tasks-generated", "approvals": { @@ -18,5 +18,6 @@ "approved": true } }, - "ready_for_implementation": true + "ready_for_implementation": true, + "approval_note": "2026-11-06: Requirements and the merged V3 design were auto-approved by the maintainer's explicit -y request. Additive V3 task groups 12-18 passed the task-plan review and an independent graph sanity review; existing groups 1-11 remain historical V2 context, and blocked task 11.4 is not a V3 prerequisite. Tasks are approved for implementation." } diff --git a/.kiro/specs/detection-core/tasks.md b/.kiro/specs/detection-core/tasks.md index 79c7ea4..81eeb29 100644 --- a/.kiro/specs/detection-core/tasks.md +++ b/.kiro/specs/detection-core/tasks.md @@ -7,6 +7,7 @@ | Completed groups 1–7 and 8.1 | Retained as regression substrate; their former requirement mappings do not count toward v2 coverage | | Previous task 8.2 | Permanently parked after admission NO-GO #3; no task below revives that arm | | Previous tasks 8.3–8.4 | Superseded by v2 consumer and proof conformance work; no rollout or live capability enablement is included | +| Existing task 11.4 | Retained as blocked historical V2 validation; it is not a prerequisite for additive V3 tasks 12–18, which have explicit dependencies and their own compatibility gate | ## Task constraints @@ -16,6 +17,9 @@ - Historical v1 artifacts remain byte-identical. V2 creates a new baseline and is never trended against v1. - The parked authorization arm remains parked. Admission orchestration uses synthetic/reference inputs only and does not consume a live population. - Existing target facts may be schema-migrated without expanding vocabulary or behavior. +- Tasks 12–18 are the current additive V3 implementation scope; completed groups 1–11 remain historical V2 implementation context. +- V2 aggregate roots and public routes remain frozen; V3 uses a distinct schema and adds no converter or inferred projection. +- Benchmark-owned storage, compare-and-swap, execution, access, retry, and workflow orchestration remain outside core and are exercised only through synthetic or in-memory boundaries. - [ ] 1. Establish migration-safe foundations - [x] 1.1 Freeze historical v1 artifacts and the reviewed migration inventory @@ -437,6 +441,269 @@ - Done when the complete suite passes, boundary/import censuses are clean, historical bytes are unchanged, and all declared adapter and consumer seams pass conformance. - _Requirements: 1.4, 10.4, 14.3, 15.3, 15.4_ - _Depends: 10.7_ + - _Blocked: Benchmark population governance conflict: the permanent unseen-pool reservation guard and the later spent/retired training-only pilot policy disagree. The benchmark owner must approve the authoritative lifecycle and replacement/retirement plan before the full suite can pass._ + +## Evaluation-admission V3 amendment + +Tasks 12–18 implement Requirements 16–20 as an additive predecessor for the approved +valid-detection-benchmark design. Existing groups 1–11 remain the historical V2 +implementation record. The unresolved downstream-governance item 11.4 is not reopened and +is not a prerequisite for this V3 stream; V3 tasks depend explicitly on completed V2 +contract work and use synthetic/in-memory workflow boundaries only. + +- [ ] 12. Preserve V2 admission and establish V3 measurement foundations +- [ ] 12.1 (P) Freeze representative V2 admission behavior + - Pin accepted canonical bytes, digests, decisions, validation failures, report output, and authoritative replay output for representative V2 admissions. + - Prove every frozen fixture is readable and nonzero before treating a stable result as compatibility evidence. + - Keep pooled V2 evidence replayable only under V2 and add no converter or inferred V3 projection. + - Done when a byte, digest, decision, failure-order, or replay mutation fails the compatibility suite while the current V2 fixtures pass unchanged. + - _Requirements: 16.1, 16.2, 16.3, 16.4_ + - _Boundary: V2 Admission Compatibility_ + - _Depends: 8.9, 8.10_ +- [ ] 12.2 Extract version-neutral metric arithmetic without changing V2 + - Isolate only the closed standard-metric algebra and Wilson calculation; keep every serialized V2 identity and validator in its existing owner. + - Route V2 through the shared arithmetic and compare all frozen V2 outputs byte-for-byte. + - Expose the same pure arithmetic to V3 without accepting a V2 policy, result, or report as a V3 value. + - Done when both versions use one arithmetic implementation and the complete V2 compatibility fixture remains unchanged. + - _Requirements: 16.2, 18.6_ + - _Boundary: Version-Neutral Metric Algebra_ + - _Depends: 12.1_ +- [ ] 12.3 Define exact arm-scope measurement and sufficiency contracts + - Represent one canonical arm-and-scope key for every policy, threshold, sufficiency rule, metric, gate, and decision. + - Bind exact denominator algebra, fixed Wilson intervals, sample counts, minimum sizes, interval precision, and rational aggregate weights. + - Include every arm identity input that must start a new baseline when changed, including judgment vocabulary, analyzer, facts, configuration, registered budget, execution, outcome mapping, and combination rule. + - Refuse duplicate, unsorted, unknown, cross-key, non-finite, or incomplete values through exact closed validation. + - Done when canonical contract tests accept one complete component and combined-arm configuration and reject every single-field identity or arm-scope mismatch. + - _Requirements: 16.1, 16.4, 16.5, 18.1, 18.6, 18.7_ + - _Boundary: Evaluation Measurement_ + - _Depends: 12.2_ +- [ ] 12.4 Establish distinct V3 version-boundary primitives + - Reserve the exact evaluation-admission V3 schema identity and version-routing primitives without changing any V2 public name or route. + - Make V3 contracts exact-type only and prohibit V2-to-V3 conversion, subclass acceptance, and mixed-version child relationships as each later root is introduced. + - Define the shared non-authorizing marker for feasibility, insufficiency, raw reports, aggregates, incomplete arm results, and other partial values; leave complete root decoding to facade integration. + - Done when boundary tests preserve exact V2 routing, recognize only the reserved V3 identity on the V3 path, and prove that a representative partial V3 value has no authorization capability. + - _Requirements: 16.1, 16.3, 16.4, 16.6_ + - _Boundary: Evaluation Version Compatibility_ + - _Depends: 12.1, 12.3_ + +- [ ] 13. Register reviewed multi-arm plans and exact preflight +- [ ] 13.1 Define component, combined-arm, and declarative combination configurations + - Represent component and combined arm configurations with exact scopes, policies, thresholds, sufficiency rules, outcome mapping, and a closed declarative combination rule. + - Require component identities and a finite normalized decision table for every combined arm, with identical-component-candidate-only proof retention. + - Refuse missing components, duplicate arms, cross-key policies, and arbitrary executable combination behavior. + - Done when canonical component and combined configurations validate and every component, rule-row, policy, threshold, sufficiency, or outcome-mapping mutation changes the configuration identity. + - _Requirements: 16.5, 17.1, 17.2, 19.4_ + - _Boundary: Evaluation Arm Configuration_ + - _Depends: 12.3_ +- [ ] 13.2 Build the non-circular plan declaration, baselines, and expected slots + - Derive the complete label-blind plan declaration before constructing per-arm baselines so the plan and baselines cannot form a digest cycle. + - Bind each baseline to the population, freeze, input set, declaration, and exact arm configuration so any listed change creates a new baseline. + - Define expected observation slots for every planned arm and input without including ground-truth roles or labels. + - Done when plan reordering is canonical, every substantive mutation changes the declaration and affected baselines, and a combined arm cannot name an absent component. + - _Requirements: 16.5, 17.1, 17.2_ + - _Boundary: Evaluation Plan_ + - _Depends: 13.1_ +- [ ] 13.3 Validate the reviewed label-blind census and plan manifest + - Represent exact vulnerable/fixed scope capacities, ecosystem-specific ordinary-push capacities, semantic/model capacities, and deduplicated whole-population size without case labels. + - Require an independently reviewed plan payload that binds the population, input set, arm configurations, baselines, readiness, custody head, census, and expected slots. + - Resolve review evidence under the configured trust and byte limits and refuse self-review, copied attestations, exposed labels, missing scopes, or inconsistent population identities. + - Done when one reviewed multi-arm manifest validates with labels sealed and every review, census, scope, ordinary-ecosystem, or slot substitution is refused before feasibility. + - _Requirements: 17.1, 17.2, 17.3, 18.1, 18.3_ + - _Boundary: Evaluation Plan Review_ +- [ ] 13.4 Derive feasibility independently for every planned arm and exact scope + - Produce exactly one feasibility row for the canonical product of planned arms and their claimed scopes. + - Recompute minimum sample size, threshold reachability, and Wilson interval precision from the reviewed label-blind census for each row. + - Derive shared spend eligibility only when every row is feasible; keep insufficiency and its reasons distinct from GO and NO-GO. + - Refuse missing, extra, duplicated, or caller-summarized rows and prevent preparation when one row is insufficient. + - Done when a single insufficient scope blocks the shared spend without opening labels, while the complete feasible plan yields one reproducible preflight digest. + - _Requirements: 17.3, 17.4, 20.2_ + - _Boundary: Evaluation Feasibility_ + +- [ ] 14. Seal planned observations and authorize one atomic spend +- [ ] 14.1 Validate exact nonzero observations against every expected slot + - Represent label-blind observation headers that bind slot, arm, input, execution, registered budget, configuration, input-read evidence, stage-execution evidence, cases, and proof candidates. + - Resolve every sealed observation through trusted content references and prove its bytes are nonzero before accepting its header. + - Require one-to-one equality between expected slots and sealed observations; refuse missing, extra, changed, unreadable, cross-arm, cross-input, or duplicate values. + - Done when the complete slot set validates and each single-slot removal, swap, truncation, zero-byte value, or identity mutation is refused before authority. + - _Requirements: 17.5, 17.6, 20.8_ + - _Boundary: Evaluation Observation_ + - _Depends: 13.3_ +- [ ] 14.2 Validate reviewed attempt history, sealed access, and exact retries + - Bind each attempt event to the complete retry identity, closed access policy, raw-observation and diagnostic access counts, public status projection, disposition, review, and predecessor. + - Resolve the reviewed predecessor chain from its declared head and require every unspent attempt to have the identical population, plan, arms, inputs, configurations, budgets, and execution identities. + - Permit public pre-spend projection only for closed status codes, counts, and digests; any raw/diagnostic access or identity change produces a terminal development-opened or spent disposition. + - Enforce terminal monotonicity so a later sealed event cannot restore admission eligibility. + - Done when an unchanged all-sealed retry chain validates and broken, forked, changed, exposed, self-reviewed, or terminal-then-sealed histories cannot authorize. + - _Requirements: 17.10, 17.11, 17.12_ + - _Boundary: Evaluation Attempt History_ + - _Depends: 13.3_ +- [ ] 14.3 Validate the independently reviewed admission authority + - Bind the feasible plan to the exact sealed observation set, attempt-history head, current readiness, validation context, and current complete custody head. + - Require the admission-authority payload to be reviewed independently from its producer and resolve its review evidence before authorization. + - Revalidate feasibility, all observation bytes, attempt history, readiness, and custody immediately before producing authority. + - Refuse stale or restored availability, changed observations, missing review, and any difference from the reviewed plan rather than returning a partial authority. + - Done when one canonical authority is derived only from the complete reviewed graph and every stale, substituted, or incomplete source fails before spend. + - _Requirements: 17.5, 17.6, 17.8, 17.10, 17.11, 17.12_ + - _Boundary: Evaluation Authority_ + - _Depends: 13.4, 14.1, 14.2_ +- [ ] 14.4 Bind every planned arm to one preparation and custody receipt + - Construct one head-bound label-release request whose influenced artifact transitively commits to every planned baseline and exact authority. + - Validate only the exact monotonic available-to-spent successor and derive an explicit all-arm binding between preparation, receipt, and planned baselines. + - Reject subset preparations, missing or mismatched receipts, stale concurrent attempts, broken custody chains, and any later restoration of availability. + - Keep durable compare-and-swap and label release outside core while exercising the boundary with an in-memory custody port. + - Done when one receipt authorizes all and only the planned arms, concurrent/stale losers fail, and no partial spend is constructible. + - _Requirements: 17.7, 17.8, 17.9_ + - _Boundary: Evaluation Shared Spend_ + - _Depends: 14.3_ + +- [ ] 15. Authenticate V3 case release, combinations, and proof continuity +- [ ] 15.1 Build the arm-aware V3 case and independently reviewed release graph + - Bind each post-spend case to its arm-scope key, baseline, observation, underlying push, role, pair, result, evidence inputs, change inputs, input manifest, and proof-candidate identity. + - Bind one independently reviewed release to the plan, admission authority, preparation, exact shared spend, and complete canonical case set. + - Require exact planned-arm and released-case cardinality and exact V3 root types; refuse V2 releases and subclasses. + - Require nonzero read and required-stage execution evidence in every released case before it can enter scoring. + - Done when the complete V3 release validates and V2 substitution, missing arms, duplicate cases, wrong spend, or copied review fail before metric production. + - _Requirements: 19.1, 19.2, 19.3, 19.11, 20.8_ + - _Boundary: Evaluation Release_ + - _Depends: 14.4_ +- [ ] 15.2 Revalidate exact role, pair, scope, result, and input associations + - Recompute every case and binding digest from canonical evidence rather than accepting caller-supplied summaries. + - Require vulnerable/fixed roles and scoped pair identities to form exact pairs; prohibit pair identities on ordinary cases. + - Preserve the separately reviewed underlying-push identity for scoped projection and later whole-population deduplication. + - Refuse arm, scope, role, pair, observation, result, evidence, location, input, or manifest swaps before any count is derived. + - Done when a canonical multi-scope release passes and a mutation matrix proves every association mismatch is invalid rather than a failed quality gate. + - _Requirements: 18.2, 18.5, 19.2, 19.10_ + - _Boundary: Released Case Validation_ +- [ ] 15.3 Replay combined-arm rules and preserve component observation lineage + - Recompute combined observation outcomes from the registered finite decision table and exact component observation digests. + - Preserve each component arm's observation identity and outcome independently of the combined result; defer completed metric, gate, decision, and report preservation to replay assembly. + - Refuse missing, extra, replaced, reordered, or contradictory component material and any combined arm whose lineage differs from the reviewed plan. + - Done when the registered combined arm replays deterministically from exact component observations and no component observation can hide another component's absence, contradiction, or lineage mismatch. + - _Requirements: 19.4, 19.5, 19.11_ + - _Boundary: Combination Validation_ + - _Depends: 15.1, 15.2_ +- [ ] 15.4 Enforce proof-candidate identity and prevent proof transfer + - Keep every label-blind proof candidate non-authoritative until the V3 release establishes its exact arm, family, scope, roles, inputs, and location. + - Apply proof-backed status only through the existing trusted differential-proof validator and the identical released candidate. + - Permit a combined arm to retain only a candidate already present in its named component observation; prevent creation, substitution, and cross-arm transfer. + - Preserve absent or inconclusive proof as not demonstrated and refuse role-swapped, cross-family, cross-scope, location, input, control, producer, receipt, or trust-policy mismatches. + - Done when valid released proof attaches once, absent proof remains non-demonstrated, and the complete transfer/mismatch matrix is refused. + - _Requirements: 19.6, 19.7, 19.8, 19.9, 19.10_ + - _Boundary: Proof Continuity_ + - _Depends: 15.3_ + +- [ ] 16. Derive exact scoped metrics and completed reports +- [ ] 16.1 Partition quality evidence by exact arm, family, and ecosystem + - Select vulnerable and fixed cases only from the independently released exact scope and derive pair correctness without cross-scope reuse. + - Derive precision only from findings and cases assigned to the same arm-scope key under its reviewed denominator policy. + - Preserve family decision vocabulary, neutral outcomes, evidence bases, model outcomes, and semantic coverage independently for each key. + - Done when a multi-arm, multi-scope case set yields isolated partitions and any cross-arm, cross-family, or cross-ecosystem contribution is refused. + - _Requirements: 18.1, 18.2, 18.4_ + - _Boundary: Evaluation Scoped Metrics_ + - _Depends: 15.2, 15.4_ +- [ ] 16.2 Project ordinary pushes by ecosystem and deduplicate whole coverage + - Count each eligible ordinary push once for every family claimed by the arm in the push's reviewed ecosystem and never in another ecosystem. + - Deduplicate one underlying push once in whole-population coverage even when it contributes to several verified scope denominators. + - Reconcile projected ordinary counts and global unique-push counts back to the reviewed census and released identities. + - Done when same-ecosystem multi-family projection produces the expected scoped counts, cross-ecosystem projection is absent, and whole coverage counts each push once. + - _Requirements: 18.3, 18.5_ + - _Boundary: Evaluation Ordinary Projection_ +- [ ] 16.3 Produce the complete scoped metric set and reject post-spend invalidity + - Derive every standard metric for every arm-scope key using the fixed algebra and Wilson 95% calculation. + - Publish numerator, denominator, sample count, value, and interval bounds for each scoped metric. + - Treat a post-spend zero denominator, census difference, missing standard metric, or inconsistent projection as invalid evaluation evidence rather than insufficiency, GO, or NO-GO. + - Done when complete nonzero scoped metrics match independent count fixtures and every zero/mismatch mutation fails before report construction. + - _Requirements: 18.1, 18.6, 18.8_ + - _Boundary: Evaluation Metric Derivation_ +- [ ] 16.4 Derive completed per-scope decisions and per-arm operational evidence + - Derive instrument, budget, readiness, and quality gates for every exact arm-scope result from canonical evidence. + - Emit GO or NO-GO only when the scope evaluation is complete; keep insufficiency, unread input, skipped stage, incomplete arm, and instrument impairment outside quality decisions. + - Preserve per-arm completed and deferred case identities, stage funnel, instrument/model failures, runtime, and closed degradation reasons separately from shared whole coverage. + - Require nonzero input reads and required-stage execution before accepting a clean-looking result. + - Done when every completed scope receives one reproducible decision and every incomplete or impaired fixture is refused without a quality decision. + - _Requirements: 20.1, 20.2, 20.3, 20.8_ + - _Boundary: Evaluation Report_ +- [ ] 16.5 Derive optional aggregates that can never authorize + - Bind aggregate constituents, sample counts, canonical rational weights, and derived metric values to exact arm-scope metric identities. + - Require weights to be non-negative, canonical, one-to-one with constituents, and sum exactly to one through integer arithmetic. + - Exclude gates, GO/NO-GO, proof promotion, and every consumer-authorization capability from the aggregate contract. + - Done when a valid aggregate reproduces from its constituents and raw, incomplete, gated, decision-bearing, or differently weighted aggregates are refused as authorization evidence. + - _Requirements: 16.6, 18.7, 20.7_ + - _Boundary: Evaluation Aggregate_ + +- [ ] 17. Close complete replay and replay-only consumer authorization +- [ ] 17.1 Assemble one complete replay over the shared evaluation graph + - Bind the reviewed plan, exact feasibility, reviewed admission authority, preparation, receipt, all-arm spend binding, attempt-history head, V3 release, cases, per-arm reports, combination lineages, shared whole coverage, and optional aggregate. + - Require exact equality between planned arms/scopes and case/report identities, including every component required by a combined arm. + - Prevent replay construction when any arm, scope, mandatory gate, custody binding, release, case, component, or lineage is missing or invalid. + - Done when the complete graph produces one deterministic replay digest and every omitted or substituted component prevents construction. + - _Requirements: 19.4, 19.5, 19.11, 20.4, 20.5_ + - _Boundary: Evaluation Replay_ + - _Depends: 15.3, 16.4, 16.5_ +- [ ] 17.2 Revalidate replay from canonical source artifacts and current trust state + - Resolve the complete attempt chain and observation/review content again, revalidate current custody, and rederive every identity, count, metric, gate, decision, report, lineage, and replay digest. + - Require exact equality with the carried values and refuse coordinated rehashing, stale external state, unknown/mixed versions, and partial roots. + - Make the validated complete replay the only value that may cross a V3 authorization boundary. + - Done when independent replay validation reproduces the original reports exactly and every source, summary, or trust-state mutation is refused. + - _Requirements: 16.6, 20.5, 20.6, 20.7, 20.8_ + - _Boundary: Evaluation Replay Validation_ +- [ ] 17.3 Publish the integrated V3 preflight, authorization, scoring, and replay facade + - Wire the exact plan preflight, reviewed authority, shared-spend validation, score/replay, strict decoding, and replay validation through one public V3 contract surface. + - Preserve every V2 facade and import unchanged and expose no benchmark storage, execution, lifecycle, or presentation implementation from core. + - Reject wrong-phase values and make each returned artifact usable only by the next declared phase. + - Done when a client can execute the complete synthetic contract through the public V3 schema and V2/unknown/mixed artifacts fail at the first boundary. + - _Requirements: 16.1, 16.4, 20.4, 20.6_ + - _Boundary: Evaluation Admission Facade Integration_ + - _Depends: 17.1, 17.2_ +- [ ] 17.4 (P) Authorize fitted components only from one exact replayed arm-scope result + - Validate the complete V3 replay before selecting exactly one planned arm-scope result and require that completed result to be GO. + - Preserve existing reversed-pair probe replay, fitting lineage, exact model declaration, and disjoint resolved repository-holdout checks. + - Refuse V2 replay, raw arm report, aggregate, incomplete replay, missing/ambiguous key, NO-GO scope, or substituted fitted evidence. + - Done when a matching fitted component authorizes only from its exact GO scope and every partial, aggregate, cross-arm, or probe/holdout mutation fails. + - _Requirements: 13.2, 16.6, 20.7_ + - _Boundary: Fitted Consumer V3_ + - _Depends: 17.2_ + +- [ ] 18. Validate the V3 predecessor end to end +- [ ] 18.1 Run a synthetic multi-arm, multi-family evaluation through one shared spend + - Use at least two component arms and one combined arm across at least two families and two ecosystems with multi-scope fixes and representative ordinary pushes. + - Prove nonzero reads, reviewed plan, exact preflight, sealed observations, reviewed attempt history, one preparation/receipt, V3 release, scoped reports, complete replay, and exact fitted selection. + - Preserve component observations/results and verify ordinary projection plus shared whole-population deduplication in the completed replay. + - Done when the public V3 facade produces one validated replay with every planned arm/scope and no V2 or benchmark-owned implementation is used as a substitute. + - _Requirements: 16.1, 16.5, 17.1, 17.3, 17.5, 17.7, 18.1, 18.3, 18.5, 19.1, 19.4, 20.1, 20.4, 20.7, 20.8_ + - _Boundary: V3 End-to-End Conformance_ + - _Depends: 17.3, 17.4_ +- [ ] 18.2 (P) Run the adversarial pre-spend and compatibility matrix + - Exercise V2 byte/digest/replay stability, V2-as-V3, unknown/mixed/subclass schemas, baseline mutations, impossible scopes, missing slots, zero/unreadable observations, self-review, forked attempt history, identity changes, raw access, stale custody, and subset spend. + - Prove every measurement instrument reads nonzero fixture bytes and fails loudly on unread input. + - Verify insufficiency leaves custody available and no refused case reaches detector execution or label release in the synthetic boundary. + - Done when every pre-spend mutation fails at its owning layer, V2 remains unchanged, and no failure produces a clean-looking result or quality decision. + - _Requirements: 16.1, 16.2, 16.3, 16.4, 16.5, 16.6, 17.2, 17.4, 17.6, 17.8, 17.9, 17.10, 17.11, 17.12, 20.2, 20.8_ + - _Boundary: V3 Pre-Spend and Compatibility Validation_ + - _Depends: 17.3_ +- [ ] 18.3 (P) Run the adversarial post-spend, metric, combination, and replay matrix + - Exercise V2 release substitution, arm/scope/role/pair/result/input swaps, missing components, contradictory combinations, proof transfer, absent/inconclusive proof, cross-ecosystem ordinary projection, duplicate whole coverage, zero/mismatched denominators, missing gates, and incomplete reports/replay. + - Verify invalid post-spend evidence never becomes insufficiency, GO, NO-GO, aggregate authority, or consumer eligibility. + - Revalidate every complete replay from source artifacts and current custody rather than trusting carried summaries. + - Done when the valid control replay reproduces exactly and every post-spend mutation is refused before authorization without hiding another arm's status. + - _Requirements: 18.2, 18.4, 18.6, 18.7, 18.8, 19.2, 19.3, 19.5, 19.6, 19.7, 19.8, 19.9, 19.10, 19.11, 20.3, 20.5, 20.6, 20.7_ + - _Boundary: V3 Post-Spend and Replay Validation_ + - _Depends: 17.2, 17.4_ +- [ ] 18.4 Verify dependency boundaries, complexity caps, and bounded validation growth + - Verify dependency direction, no V2/V3 aggregate imports across the version boundary, standard-library-only core runtime, and absence of benchmark, analyzer, model-runtime, or proof-execution ownership in V3 modules. + - Enforce function and module complexity caps and measure validation growth over increasing arms, scopes, observations, attempt events, cases, and reports. + - Fail each measurement fixture unless its scoped inputs are readable and nonzero before interpreting timing, count, or clean-result output. + - Done when import/boundary checks are clean, amended code stays within complexity caps, and validation cost remains bounded by supplied artifact counts without weakening checks. + - _Requirements: 16.2, 20.6, 20.8_ + - _Boundary: V3 Boundary and Performance Validation_ + - _Depends: 18.1, 18.2, 18.3_ +- [ ] 18.5 Run the final compatibility, regression, and schema-export gate + - Run the V2 golden suite, every V3 unit/integration/adversarial suite, and the existing core regression suite without consuming a real admission population. + - Prove all fixtures are readable and nonzero and the exact `openultrasast.detection-core.evaluation-admission.v3` schema is publicly exported. + - Done when all gates pass, V2 outputs remain frozen, validation stays bounded by supplied artifacts, and the validated predecessor is ready for downstream benchmark task generation. + - _Requirements: 16.1, 16.2, 16.4, 20.6, 20.8_ + - _Boundary: V3 Final Validation Integration_ + - _Depends: 18.4_ ## Implementation Notes @@ -446,3 +713,4 @@ - Task 10.6 debug established that exact-byte successor authority must require canonical repository-relative paths and strict resolved-root containment, including rejection of outside-root symlink targets, before module digests can prove closure. - Human review reopened Task 10.6 for one focused remediation cycle covering canonical path syntax, resolved-root containment/symlink refusal, and the task-local formatting gate. - Human review reopened Task 10.5 for one focused remediation cycle covering exact source-closure evidence, collision-resistant create-only attempt persistence, and complexity-cap conformance. +- Task 11.4 debug established that the 73 reserved-repository references are a benchmark-governance specification conflict, not a detection-core or instrument defect; historical artifacts and the reservation guard must remain unchanged pending an owning-spec decision. diff --git a/.kiro/specs/valid-detection-benchmark/design.md b/.kiro/specs/valid-detection-benchmark/design.md index 1bbce1a..021e38e 100644 --- a/.kiro/specs/valid-detection-benchmark/design.md +++ b/.kiro/specs/valid-detection-benchmark/design.md @@ -1,129 +1,990 @@ -# Design: valid detection benchmark +# Technical Design: Valid Detection Benchmark -Builds the benchmark in R1-R8. New code lives under `benchmarks/valid/`. It reuses the sourcing, -labelling, negative-drawing, replay, and scoring that already exist, and drops the SZZ layer. +## Overview -## What is reused and what is new +This feature replaces the withdrawn 2026-10-12 pilot with a reproducible, independently reviewed benchmark for measuring detector behavior on unseen Python and JavaScript/TypeScript application repositories. It establishes fix-derived positives, representative ordinary pushes, permanent reservation, label-isolated admission, exact family-and-ecosystem metrics, differential-proof reporting, and durable operator outcomes. -Reused as-is: -- Sourcing advisories to repos and fix commits: `benchmarks/unseen/source.py`, `draft.py`, - `eligibility.py` (the used-set held-out guard). -- Fix-site labelling: `benchmarks/unseen/extract.py:fix_sites()` and the pair builder in - `benchmarks/pairs/advisory-fixes/` (`materialize.py`, `catalog.toml`, `verification.json`). -- Function cutting: `openultrasast.learn.excerpt.function_bounds`. -- Ordinary negatives: `extract.py:ordinary_history()` and `draw_ordinary()`. -- Replay: `benchmarks/unseen/replay.py` and `replay_entry.py` (whole tool via the push hook). -- Scoring base: `benchmarks/unseen/score.py` (catch, false alarm, Wilson, repo-clustered interval, - required_n, McNemar gate). +The design preserves detection-core as the only authority for admission algebra and replay. The benchmark owns the population and workflow around that authority. An approved and validated `detection-core` amendment exporting `openultrasast.detection-core.evaluation-admission.v3` is an explicit predecessor deliverable because the current V2 root pools all claimed scopes into one metric result and cannot safely authorize per-scope or multi-arm claims. Task generation for this spec is blocked until that predecessor exists; valid-benchmark tasks never edit core-owned files. -New: -1. `benchmarks/valid/catalog.py` - sourcing to a fix-site positive catalog, SZZ paths excluded. -2. `benchmarks/valid/verify.py` - the committed LLM verification driver (replaces the scratchpad - `label_audit.py`). One call per case confirms vulnerable / fix-addresses / reachable-in-repo. This is - also the app-shaped filter: reachability is answered here, not by a separate classifier. -3. `benchmarks/valid/metrics.py` - precision, false alarms per 100 pushes, and exploit-verified rate, - layered on `score.py` output. -4. `benchmarks/valid/assemble.py` - freezes the verified positives plus the negative draw into a - replay manifest with a digest and the held-out guard. +Maintainers and independent reviewers use the workflow to source, review, freeze, spend, execute, and report a population. Detector owners supply immutable arm registrations and label-blind adapters; product specifications consume eligible scoped outcomes but do not receive rollout authority from this feature. -## Data flow +### Goals -``` -advisories (OSV/GHSA) - -> source.py / draft.py resolve repo + fix commit + parent [reuse] - -> catalog.py fix-site label (no SZZ); emit candidate cases [new, wraps extract.fix_sites + pairs materialize] - -> verify.py LLM: vulnerable? fix addresses it? reachable in-repo? + confidence [new] - -> human spot-check record decision, reviewer, date [manual, recorded in verification.json] - -> draw_ordinary() negatives from the same repos [reuse] - -> assemble.py freeze positives+negatives -> manifest + digest, held-out guard [new, wraps eligibility.normalize] - -> replay.py (docker lane) arm (a): the push hook as shipped (quick + engine) [reuse] - -> hunter arm runner arm (b): the hunter over the same changes, off-cluster [new, group 6] - -> combine (a) and (b) joined per change id [new, group 6] - -> score.py + metrics.py recall, precision, FP/100, exploit-verified, intervals [reuse + new] +- Freeze a replacement population with at least 40 accepted positives across both initial ecosystems and at least two vulnerability families. +- Preserve one permanent-reservation authority, human-authoritative labels, and durable spend-before-read custody. +- Produce exact family-and-ecosystem quality decisions, separate whole-population coverage, and explicit insufficient or impaired outcomes. +- Evaluate shipped, experimental, and pre-registered combined arms over identical changes without modifying detector behavior. +- Make every result reproducible from digest-bound artifacts and replayable through detection-core. + +### Non-Goals + +- Adding or tuning detectors, facts, prompts, judges, thresholds, or framework vocabularies. +- Wiring an arm into the pre-push product, enabling rollout, or changing BLOCK policy. +- Reactivating or requalifying the parked access-control arm. +- Owning AX, Kubernetes, worker, network, or object-store administration. +- Migrating pooled V2 reports into per-scope V3 reports. +- Implementing Discord transport; this design produces a safe operator projection for a separately owned notifier. + +## Boundary Commitments + +### This Spec Owns + +- Repository influence events, permanent-reservation census projection, and exact-file historical authorizations used by benchmark population construction. +- Canonical repository ecosystem review, fix-derived ground-truth review, ordinary-change selection, exclusions, and population classification. +- Deterministic public/private population freeze artifacts, label-blind census construction, sealed-label references, and replacement lifecycle records. +- A durable S3-compatible compare-and-swap adapter for the existing benchmark custody protocol, including capability and recovery probes. +- A versioned content-addressed artifact repository for reservation, population, lifecycle, observation, workflow, and final-report artifacts. +- Immutable arm registrations, pre-registered combination plans, label-blind arm execution, case binding, and general benchmark orchestration. +- Benchmark workflow envelopes and redacted JSON/text operator reports. +- Consumption and conformance testing of the detection-core V3 admission contract. + +### Out of Boundary + +- Metric algebra, scope validation, feasibility algebra, quality gates, admission decisions, proof validation, and authoritative replay; detection-core owns these. +- Implementation of the V3 predecessor contract. This design specifies the consumer contract and readiness gate; the `detection-core` amendment owns delivery and validation. +- Detector implementation and detector-specific interpretation. The new workflow does not import access-control site-judgment helpers. +- Repository/framework recognition vocabularies; concrete targets are population evidence, not implementation tasks. +- Product eligibility installation, advisory rollout, BLOCK enablement, and pre-push UX. +- Infrastructure health and bucket creation, lifecycle mutation, credential provisioning, cluster scheduling, and retry policy for AX workers. +- Partial-scope or partial-arm custody. One evaluation plan spends one whole population only after every planned arm and claimed scope is feasible and every label-blind observation set is durably frozen. + +### Allowed Dependencies + +- The validated `openultrasast.detection-core.evaluation-admission.v3` predecessor for canonical identities, per-arm exact-scope policies, shared feasibility, one shared custody receipt, V3 case-release validation, per-arm scoped reports, and one multi-arm evaluation replay root. +- Existing V2 core contracts only for historical replay and compatibility tests; the replacement admission path does not score through V2. +- `benchmarks/unseen/` sourcing, fix-site extraction, ordinary-history draw, Git identity resolution, and bucket-first persistence patterns. +- `benchmarks/valid/artifact_contract.py` for canonical envelopes, sequence identities, atomic local materialization, and conflict refusal. +- boto3/botocore `>=1.35,<2` as already declared; the locked `1.43.107` model exposes conditional `PutObject` parameters. +- A versioned S3-compatible durable store whose exact conditional-write and lifecycle behavior passes the benchmark capability probe. +- Search-with-proof artifacts only through detection-core `ProofAttachment` validation. + +Dependency direction is fixed: + +`detection-core contracts → artifact repository and reservation/review/freeze artifacts → isolated arm execution → custody and admission orchestrator → workflow envelope → operator presenter` + +Core never imports benchmark modules. Presenters and notifications never influence admission. Detector adapters never receive label bytes. + +### Revalidation Triggers + +- Any detection-core V3 schema, digest, scope, feasibility, result, report, or replay contract change. +- A change to canonical ecosystem values, review policy, alias/fork resolution, influence projection, license policy, or historical authorization semantics. +- A change to population, freeze, sealed labels, claimed scopes, denominator policy, thresholds, analyzer, facts, vocabulary, combination rule, or outcome mapping. +- A change to conditional-write tokens, custody transition semantics, object-store version visibility, retention rules, or restart recovery. +- A change that lets an arm see labels, lets presentation affect scoring, or moves detector-specific knowledge into benchmark/core components. +- A change to the multi-arm evaluation-plan root, shared-spend binding, V3 case-release graph, isolated arm-execution boundary, or observation checkpoint identity. +- A change to proof attachment matching or the advisory/BLOCK interpretation consumed by downstream product specifications. + +## Architecture + +### Existing Architecture Analysis + +The current valid benchmark has reusable fix-site cataloging, model-assisted review, deterministic 20:1 negative draws, artifact envelopes, V2 custody, proof attachments, and V2 core scoring. Its admission runner and some metrics remain access-control-specific or legacy. The permanent reservation authority exists mainly as test logic, and the production custody store is only a protocol. + +Detection-core V2 validates claimed scopes but assigns every metric rule the complete scope tuple, sums capacities across scopes, emits one unscoped result per metric, and binds one arm identity to one replay. This design keeps all V2 schemas immutable and requires a separate upstream V3 evaluation root. No post-spend projection or legacy benchmark scorer may substitute for that root. + +### Architecture Pattern and Boundary Map + +The selected pattern is a versioned contract core with benchmark-owned ports and adapters. Append-only reviewed artifacts form the population control plane; the admission path is a fail-closed state machine around one irreversible custody transition. + +```mermaid +graph TB + Sources[Advisories and Git history] --> Reservation[Reservation authority] + Sources --> Catalog[Fix site catalog] + Catalog --> Review[Human review authority] + Reservation --> Freeze[Population freezer] + Review --> Freeze + Repository[Artifact repository] --> Freeze + Freeze --> Census[Label blind census] + Census --> CorePreflight[Detection core V3 preflight] + CorePreflight --> ArmRun[Isolated arm execution] + Arms[Registered arm adapters] --> ArmRun + ArmRun --> Repository + Repository --> Orchestrator[Admission orchestrator] + Orchestrator --> Custody[Durable custody adapter] + Proof[Proof attachments] --> Orchestrator + Orchestrator --> CoreScore[Detection core V3 score and replay] + CoreScore --> Outcome[Workflow outcome] + Outcome --> Presenter[Operator report] ``` -## Schemas +Key decisions: -### Positive case (catalog.py output, extends `advisory-fixes/catalog.toml`) -``` -[[case]] -id = "owner-repo-cve" -repo = "owner/repo" -url = "https://github.com/owner/repo" -advisory = "GHSA-xxxx / CVE-xxxx" -cwe = "CWE-89" -family = "sql_injection" -language = "python" -commit = "" # post-fix (safe) side -parent = "" # pre-fix (vulnerable) side -path = "app/db.py" -function = "get_member_by_code" -function_lines = [[40, 58]] -method = "fix_site" # never introducing/last_touch +- Reservation, review, and lifecycle state are append-only reviewed records with deterministic projections. +- The population freezer binds public material, opaque private references, review records, and a label-blind census without exposing label bytes to arms. +- Every planned arm and claimed scope must pass V3 feasibility before execution. Insufficiency leaves custody available. +- Every arm runs in an isolated label-blind process before spend. Its immutable observations are durably frozen and combined without label access. +- One V3 evaluation manifest binds all per-arm plans, frozen observation sets, one custody receipt, V3 case releases, independently valid per-arm reports, and one evaluation replay root. +- The custodian opens labels only after every planned observation is complete, commits the one spend, and then joins observations to ground truth. +- Completed quality decisions come only from detection-core V3. The benchmark envelope represents orchestration status, including failures that occur before a core report exists. + +### Technology Stack + +| Layer | Choice / Version | Role in Feature | Notes | +|---|---|---|---| +| Language | Python 3.11+ | Typed contracts, orchestration, and CLI | Frozen dataclasses, enums, protocols, and boundary validation | +| Admission authority | detection-core V3 | Exact-scope feasibility, metrics, gates, report, replay | Upstream prerequisite; V2 remains historical | +| Local artifacts | Canonical JSON and JSONL | Review, lifecycle, freeze, plans, outcomes | Reuses benchmark artifact envelopes | +| Durable storage | S3 API via boto3/botocore `>=1.35,<2` | Versioned population, custody, provenance, and transient runs | Conditional create/update required; capability-tested | +| Arm isolation | Existing subprocess or container lanes | Execute serialized label-blind bundles before spend | Allowlisted environment; no custody/private-label credentials | +| Execution | Existing Docker, local, or AX arm adapters | Runs immutable label-blind cases | Execution-plane health remains external | +| Presentation | Deterministic JSON and plain text | Operator status, counts, digests, gates | Safe input to future notifier transport | + +No new third-party dependency is introduced. + +## File Structure Plan + +### Directory Structure + +```text +benchmarks/ +├── unseen/ +│ ├── reservation.py # Reservation Authority and production guard API +│ └── reservations/ +│ ├── pilot-2026-10-12-withdrawn.json # Exact historical inventory and withdrawal event +│ └── pilot-2026-10-12-replacement.json # Later replacement binding event +└── valid/ + ├── review.py # Review Authority contracts and validation + ├── lifecycle.py # Baseline Lifecycle Authority and retirement projection + ├── population.py # Population Freezer and label-blind census builder + ├── object_store.py # Artifact Repository for durable content-addressed records + ├── custody_store.py # Durable Custody Store and capability probe + ├── arm_contract.py # Arm Registry contracts and label-blind adapter protocol + ├── arm_executor.py # Isolated Arm Executor and observation checkpoints + ├── arm_combination.py # Combination Planner over preserved observations + ├── admission_orchestrator.py # Admission Orchestrator state machine and case binding + ├── workflow_journal.py # CAS workflow head, leases, and causal heartbeat chain + ├── workflow_outcome.py # Workflow Outcome canonical envelope + ├── operator_report.py # Operator Presenter JSON and text projection + └── run.py # Benchmark-only operator entry point +tests/ +├── test_unseen_reservation.py # Reservation projection and exact authorization +├── test_valid_review.py # Ecosystem and ground-truth review authority +├── test_valid_lifecycle.py # Withdrawal, replacement, and retirement projections +├── test_valid_population.py # Freeze, classification, license split, and census +├── test_valid_object_store.py # Content-addressed persistence and scratch-loss recovery +├── test_valid_custody_store.py # Conditional writes, concurrency, restart, retention +├── test_valid_benchmark_arm_contract.py # Arm identity and combination plan +├── test_valid_benchmark_orchestrator.py # Spend ordering, isolation, resume, and scoring +├── test_valid_benchmark_workflow_outcome.py # Closed status/reason and redaction +├── test_valid_benchmark_workflow_journal.py # Lease ownership, CAS resume, and stalled projection +├── test_valid_benchmark_operator_report.py # Deterministic operator projection +├── test_valid_benchmark_proof_reporting.py # Differential proof binding +└── test_valid_benchmark_e2e.py # Synthetic two-ecosystem multi-family lifecycle ``` -### Verification record (verify.py output, extends `advisory-fixes/verification.json`) -``` -{ "id": "...", "vulnerable_pre_fix": true, "fix_addresses_it": true, - "reachable_in_repo": true, "entry_point": "flask route /member/", - "confidence": 0.86, "model": "deepseek-...", "advisory_cited": true, - "human_spot_check": {"decision": "accept", "reviewer": "", "date": "2026-10-13"}, - "verdict": "accepted" } # accepted only if all three true AND human accept +### Modified Files + +- `benchmarks/valid/catalog.py` — retain package ecosystem, language, license, repository identity, and source-evidence digests while building fix-site candidates. +- `benchmarks/valid/verify.py` — record model assistance separately and delegate authoritative acceptance to `review.py`. +- `benchmarks/valid/assemble.py` — remain a legacy/development facade; replacement admission delegates to `population.py` and cannot emit an authorizing legacy manifest. +- `benchmarks/valid/population_custody.py` — bind the production store adapter and V3 spend authority without weakening committed-reread-before-label ordering. +- `benchmarks/valid/admission_scoring.py` — expose a V3 delegation seam and mark legacy aggregate helpers non-authorizing for the replacement. +- `tests/test_independent_population.py` — call `reservation.py` rather than owning reservation exceptions in test code. +- `.env.example` — list benchmark S3 setting names blank if new names are required; no credentials or endpoints are committed. + +### Upstream Prerequisite Files + +These paths remain owned and changed only by the `detection-core` amendment that exports `openultrasast.detection-core.evaluation-admission.v3`. They appear here to make the predecessor contract explicit, not to widen this spec's implementation boundary. Valid-benchmark task generation remains blocked until that amendment is approved, implemented, and independently validated. + +- `src/openultrasast/core/measurement.py` — exact-scope `DenominatorPolicyV3` and scope-qualified thresholds. +- `src/openultrasast/core/admission_review.py` — V3 multi-arm evaluation manifest and per-arm plan bindings. +- `src/openultrasast/core/admission_feasibility.py` — per-scope sufficiency and atomic `may_spend`. +- `src/openultrasast/core/admission_context.py` — V3 identity and policy validation. +- `src/openultrasast/core/admission_metrics.py` — exact-scope derivation and unique-push whole coverage. +- `src/openultrasast/core/admission_case.py` — exact V3 per-arm case-release graph bound to the shared receipt. +- `src/openultrasast/core/admission_report.py` — independently valid per-arm scope results, non-authorizing aggregates, and reports. +- `src/openultrasast/core/_admission_replay.py` — one exact-schema V3 evaluation replay containing the shared spend and every planned arm report. +- `src/openultrasast/core/admission_fitted.py` — consumer dispatch for V2 and V3 roots. +- `tests/test_core_admission_*.py` — V3 algebra, compatibility, and replay conformance. + +Every named component maps to one of the files above. No component shares authority across benchmark and core paths. + +## System Flows + +### Population Lifecycle + +```mermaid +stateDiagram-v2 + [*] --> Sourcing + Sourcing --> Reviewed: candidate and independent review accepted + Sourcing --> Excluded: reserved rejected or unresolved + Reviewed --> DevelopmentPilot: frozen positives below forty + Reviewed --> Insufficient: required ecosystem or scope evidence missing + Reviewed --> AdmissionAvailable: full freeze and reviewed census + AdmissionAvailable --> AdmissionAvailable: gate preflight insufficient + AdmissionAvailable --> Observed: all label blind arm observations frozen + Observed --> Observed: automatic retry of exact sealed plan + Observed --> DevelopmentPilot: raw observations or diagnostics inspected + Observed --> Spent: custody append committed and reread + Spent --> Scoring: labels released to custodian + Scoring --> Completed: core evaluation replay validated + Scoring --> Impaired: timeout budget resource instrument or interruption + Impaired --> Scoring: resume same immutable run + Completed --> Retired: contaminated negative or identity invalidation ``` -### Replay manifest (assemble.py output) -A `pool.toml`-shaped file so `replay.py` consumes it unchanged: `[[repository]]` rows with `changes`, -each change `kind = "vulnerability"` (positives, label from fix) or `kind = "ordinary"` (negatives), -with `base`/`head`/`files`/`function`. A `freeze-valid.json` carries the freeze digest and the held-out -guard result, matching the SZZ pool's freeze format so the ledger and replay flow are identical. +`DevelopmentPilot` and `Insufficient` are non-authorizing classifications. A pre-spend failure permits only an automatic retry with the exact plan, arms, inputs, budgets, execution environment, and admission identity while raw observations and diagnostics remain custodian-sealed. A workflow identifier alone never authorizes reuse. Human inspection or any configuration/plan change appends `development_opened` plus influence events and permanently disqualifies the population from admission. `Impaired` after spend never restores `AdmissionAvailable`; recovery resumes the same digest-bound run or records a terminal impaired outcome. `Retired` is an append-only lifecycle projection, never a mutation of the freeze. -## Metrics (metrics.py) +### Admission Sequence -Input is the scored records from `score.py`. New outputs: -- **precision** = flagged-and-real / all-flagged, over positives and negatives together, with Wilson 95%. -- **false alarms per 100 pushes** = false-positive rate x 100, the presentation of `score.py`'s - `false_alarm` rate. This is a presentation layer, not a new measurement. -- **exploit-verified rate** = of flagged positives, the share the search-with-proof arm - (`src/openultrasast/search/`) demonstrates on the vulnerable side and not the fixed side. Joined by - case `id` from a proof-arm run; absent where the arm has not run (reported as "not yet proved", not as - a failure). +```mermaid +sequenceDiagram + participant Operator + participant Freeze as Population Freezer + participant Core as Detection core V3 + participant Store as Custody Store + participant Run as Admission Orchestrator + participant Arm as Arm Adapters + participant View as Operator Presenter -Headline string, built here: -> "On unseen app repos: recall X% (Wilson A-B), false alarms Y per 100 pushes, precision Z%; exploit-verified N of M." + Operator->>Freeze: select population and evaluation plan + Freeze->>Core: reviewed label blind manifest and policies + Core-->>Run: exact scope feasibility + alt any scope insufficient + Run->>View: insufficient envelope without spend + else every scope feasible + Run->>Arm: isolated identical label blind bundles + Arm-->>Run: frozen observation sets and read receipts + Run->>Run: validate complete plan and combination lineage + Run->>Core: seal admission manifest with exact observations + Core-->>Run: spend authority after lineage and custody head check + Run->>Store: conditional custody append + Store-->>Run: reread committed successor + Run->>Run: release labels inside custodian boundary + Run->>Run: bind labels observations scopes and reviews + Run->>Core: score every arm and validate evaluation replay + Core-->>Run: per arm scoped reports and one replay root + Run->>View: completed envelope and safe projection + end +``` -## Verification driver (verify.py) details +The arm call occurs before label release in an isolated process or container. Its serialized bundle carries no labels, review judgments, expected outcomes, custody credentials, private-label references, or label-store settings. Each observation set is content-addressed and immutable; resume reuses a matching completed set and never reruns it silently under the same plan identity. -- Model: DeepSeek via the shipped provider, off-cluster (cheap, ~$0.0002/call). The brain tier runs - locally in research; it does not need the kube engine. -- Prompt inputs per case: the advisory text, the pre-fix function, the post-fix function, and the file - context around the function (for the entry-point question). No repo-wide context in the first version. -- Output is the verification record above. Reachability uses the same entry-point and source concepts the - taint engine already models (`src/openultrasast/ruleset/frameworks.py`, framework routes/handlers), so - the LLM is asked a concrete question, not an open one. -- Instrument rule: the driver prints bytes read for each side's function before it calls the model, and - fails loudly on an empty read. A verdict on an unread case is rejected. (AGENTS.md instrument rule.) +## Requirements Traceability -## First increment (pilot) +| Requirement | Summary | Components | Interfaces | Flows | +|---|---|---|---|---| +| 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8 | General two-ecosystem multi-family population and insufficiency | Review Authority, Population Freezer, Scoped Admission V3 | `RepositoryEcosystemReviewV1`, `PopulationFreezeV3`, `AdmissionFeasibilityV3` | Population lifecycle | +| 2.1, 2.2, 2.3, 2.4, 2.5, 2.6 | Fix-derived, reachable, human-authoritative ground truth | Catalog, Review Authority, Population Freezer | `GroundTruthReviewV1`, `ModelAssistanceRef`, exact scope tuple | Sourcing to reviewed freeze | +| 3.1, 3.2, 3.3, 3.4, 3.5, 3.6 | Same-repository ordinary pushes and contamination handling | Population Freezer, Baseline Lifecycle Authority, Scoped Admission V3 | `OrdinarySelectionV1`, scoped policy, `PopulationLifecycleEventV1` | Population lifecycle | +| 4.1, 4.2, 4.3, 4.4, 4.5, 4.6, 4.7, 4.8, 4.9, 4.10 | Permanent reservation and withdrawn pilot | Reservation Authority, Baseline Lifecycle Authority | `InfluenceEventV1`, `ReservationCensusV1`, `PopulationLifecycleEventV1`, `HistoricalAuthorizationV1` | Reservation projection before freeze | +| 5.1, 5.2, 5.3, 5.4, 5.5, 5.6, 5.7, 5.8, 5.9 | Development/admission separation and spend-before-read | Population Freezer, Artifact Repository, Isolated Arm Executor, Durable Custody Store, Admission Orchestrator | reviewed evaluation manifest, frozen observations, custody CAS, V3 case release | Admission sequence | +| 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 6.7 | Exact scoped denominators, aggregates, and feasible gates | Arm Registry, Scoped Admission V3 | `DenominatorPolicyV3`, `AdmissionFeasibilityV3`, `ScopeAdmissionResultV3` | Preflight and core scoring | +| 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 7.7, 7.8 | Reproducible freeze and durable resumable custody | Population Freezer, Artifact Repository, Durable Custody Store, Workflow Outcome | content identities, create-only objects, custody CAS, retention profile | Lifecycle and restart recovery | +| 8.1, 8.2, 8.3, 8.4, 8.5, 8.6 | Complete measurement and failure visibility | Scoped Admission V3, Workflow Outcome, Operator Presenter | scope report, whole coverage, instrument receipt, baseline identity | Score or impaired outcome | +| 9.1, 9.2, 9.3, 9.4, 9.5, 9.6, 9.7 | Separate arms, combinations, and demonstrated evidence | Arm Registry, Isolated Arm Executor, Combination Planner, Admission Orchestrator, Scoped Admission V3 | `EvaluationPlanV1`, frozen observations, `ProofAttachment`, evaluation replay | Pre-spend arm execution and one shared spend | +| 10.1, 10.2, 10.3, 10.4, 10.5, 10.6 | Operator outcomes and ownership boundaries | Workflow Outcome, Operator Presenter, Admission Orchestrator | closed status/reason enums and redacted projection | Completion and impairment reporting | -1. Source Python and JS advisories with a clean fix link, held out from the tool's training/mining sets. -2. `catalog.py` builds candidate cases by fix site. Target ~25 candidates to land ~12 verified. -3. `verify.py` runs the LLM check; human spot-checks all of them (dozens, so full review is cheap). -4. `draw_ordinary()` draws >=20x negatives from the same repos. -5. `assemble.py` freezes the manifest + digest + held-out guard. -6. `replay.py` on the Docker lane runs the push hook as shipped (arm a). The hunter arm runner adds arm - (b) over the same changes, off-cluster, and the two are combined per change. -7. `score.py` + `metrics.py` produce the four metrics with intervals and the headline, for (a) and for - (a)+(b). (b) is the number the tool is held to; the difference is the product gap on pre-push. +## Components and Interfaces -Wiring the hunter into the pre-push hook itself is product work in `pre-push-safety-net`. This spec -measures the push path with and without it; it does not change the hook. +| Component | Domain | Intent | Requirements | Key Dependencies | Contracts | +|---|---|---|---|---|---| +| Scoped Admission V3 | Upstream core | Authoritative exact-scope feasibility, scoring, gates, and replay | 1.7, 3.3, 3.4, 6.1–6.7, 8.1–8.6, 9.3–9.7 | Core canonical and V2 contracts P0 | Service, State | +| Reservation Authority | Population governance | Project reviewed influence into permanent reservation and exact historical authorization | 4.1–4.10, 7.1, 7.7 | Git identity resolver P0, artifact contract P1 | Service, State | +| Review Authority | Curation | Decide ecosystem and ground truth independently of model assistance | 1.1–1.8, 2.1–2.6 | Catalog P0, core review policy P1 | Service, State | +| Baseline Lifecycle Authority | Population governance | Project withdrawal, replacement completeness, and immutable retirement | 3.6, 4.5–4.10, 7.7 | Reservation P0, artifact repository P0 | Service, State | +| Population Freezer | Population | Draw, classify, split, freeze, and census the replacement | 1.1–1.8, 3.1–3.6, 5.1–5.5, 7.1–7.7 | Reservation and review P0 | Service, Batch, State | +| Artifact Repository | Persistence | Persist immutable content-addressed authorities and recover after scratch loss | 4.6–4.9, 7.1–7.8, 10.1–10.3 | S3 versioning P0 | Service, State | +| Durable Custody Store | Persistence | Commit and recover monotonic custody transitions | 5.7, 7.3–7.5, 7.7, 7.8 | S3 conditional writes P0 | Service, State | +| Arm Registry and Combination Planner | Evaluation plan | Bind arm identities, scopes, and label-blind combination rules | 5.5, 6.1, 9.1, 9.2 | Core identities P0 | Service, State | +| Isolated Arm Executor | Execution adapter | Freeze complete label-blind observations before spend | 5.6, 8.4, 8.5, 9.1, 9.2 | Arm registry P0, execution lane P1 | Service, Batch, State | +| Admission Orchestrator | Workflow | Enforce preflight, spend, isolation, execution, release, score, and resume | 5.3–5.9, 8.4, 8.5, 9.1–9.7, 10.4–10.6 | Core V3 P0, custody P0, arms P0 | Service, Batch, State | +| Workflow Outcome | Operations | Persist completed, insufficient, and impaired workflow status | 8.3–8.6, 10.1–10.3 | Artifact contract P0 | State | +| Operator Presenter | Presentation | Render safe deterministic status without becoming authority | 8.1–8.3, 10.1–10.3, 10.6 | Workflow outcome P0 | Service | -The pilot proves the pipeline and gives the first real numbers. Then grow positives to ~40 (set exactly -by `required_n`) and, where the proof arm has run, fill in the exploit-verified rate. +### Upstream Core -## Risks -- Harvesting is rate-bound (GitHub 5k/hr) and infra is operator-owned; the sourcing step runs - deliberately, not in a tight loop, and reuses the banked resolver. -- The LLM verifier can be wrong; the human spot-check is the backstop, and the proof arm is the - label-free check above it. -- App-shaped is a judgement call; record the entry point the verifier names so a human can audit it. +#### Scoped Admission V3 + +**Responsibilities and constraints** + +- Own a two-stage root: `EvaluationPlanManifestV3` binds the reviewed label-blind population, a canonical tuple of `ArmAdmissionPlanV3` values, input-set identity, policies, thresholds, scopes, and expected observation slots; `EvaluationAdmissionManifestV3` later binds that plan manifest to the exact frozen observation-set references and combination lineage. Each arm or combined arm has its own baseline identity, exact scopes, denominator policy, thresholds, and combination lineage. +- Own the exact Cartesian product of every arm's claimed scopes and standard metrics. +- Evaluate minimum pairs, minimum ordinary pushes, and interval-precision rules for every planned arm before label access. +- Evaluate arithmetic feasibility from the plan manifest before execution. After observations freeze, revalidate plan lineage, completeness, readiness, current custody head, and observation bytes before deriving shared `may_spend` in `EvaluationAdmissionAuthorityV3`. +- Validate `AdmissionCaseReleaseV3`; V2 `AdmissionCaseRelease` is not substitutable. +- Emit one GO or NO-GO result per exact scope for each planned arm after the shared spend. +- Emit whole-population coverage deduplicated by canonical change-input identity. +- Permit an optional aggregate with declared constituents and weighting but no decision or gate field. +- Emit one `EvaluationReplayV3` containing the shared preparation/receipt/release graph and independently valid per-arm reports. Completion requires every arm in the plan. +- Preserve V2 validators and bytes; dispatch by exact schema root and never auto-upgrade a V2 release. + +**Dependencies** + +- Inbound: benchmark reviewed census, policies, released cases, and proof attachments (P0). +- Outbound: benchmark orchestrator and downstream eligibility consumers (P0). +- External: none. + +**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [x] + +```python +class ScopedAdmissionV3(Protocol): + def evaluate_feasibility( + self, + manifest: EvaluationPlanManifestV3, + ) -> EvaluationFeasibilityV3: ... + + def authorize_admission( + self, + manifest: EvaluationAdmissionManifestV3, + feasibility: EvaluationFeasibilityV3, + current_custody_log: PopulationCustodyLog, + ) -> EvaluationAdmissionAuthorityV3: ... + + def score_and_replay( + self, + authority: EvaluationAdmissionAuthorityV3, + release: AdmissionCaseReleaseV3, + cases_by_arm: tuple[ArmAdmissionCasesV3, ...], + ) -> EvaluationReplayV3: ... +``` + +`EvaluationAdmissionAuthorityV3` binds the admission-manifest digest, plan-manifest digest, every per-arm policy/threshold set, shared feasibility digest, validation context, exact sealed observation-set references, combination lineage, readiness, current custody-head digest, and preparation. The subsequent V3 case release binds the one committed receipt. Preconditions require exact reviewed identities and canonical arm/scope ordering. Postconditions rederive all counts from validated cases; aggregates cannot authorize; every per-arm report is independently replayable inside the closed evaluation root. A V3 consumer never infers per-scope values from V2. + +### Population Governance + +#### Reservation Authority + +**Responsibilities and constraints** + +- Resolve canonical repository, alias, former-name, redirect, and fork-network identities before accepting an event. +- Append reviewed `enumerated_only` or `influenced` events; only influenced identities enter the permanent census. +- Project legacy reservation sources and accepted events into one deterministic `ReservationCensusV1`. +- Validate historical audit material by normalized repository-relative path and current SHA-256 digest. +- Preserve withdrawal and replacement as append-only lifecycle events. + +**Dependencies** + +- Inbound: sourcing and development workflows (P1). +- Outbound: Population Freezer and permanent guard test (P0). +- External: Git provider identity API during sourcing only (P1). + +**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [x] + +```python +class InfluenceDisposition(Enum): + ENUMERATED_ONLY = "enumerated_only" + INFLUENCED = "influenced" + +class ReservationAuthority(Protocol): + def append_influence(self, event: InfluenceEventV1) -> ReservationCensusV1: ... + def project_census(self, events: tuple[InfluenceEventV1, ...]) -> ReservationCensusV1: ... + def authorize_historical_path( + self, path: PurePosixPath, content_digest: str, lifecycle: PopulationLifecycleProjectionV1 + ) -> HistoricalAuthorizationV1: ... +``` + +An authorization cannot match a directory, repository name, glob, changed byte sequence, or relocated path. + +#### Review Authority + +**Responsibilities and constraints** + +- Preserve package ecosystem, observed language, license, source, and fix evidence from cataloging. +- Accept canonical repository ecosystem values `python` or `javascript`; TypeScript remains an observed language under `javascript`. +- Require producer/reviewer separation, policy satisfaction, and conflict disclosure. +- Record explicit defect-present, fix-addresses, and in-repository-reachable judgments. +- Make model assistance optional and non-authoritative. +- Attach an explicit tuple of independently verified exact scopes to multi-label fixes. + +**Dependencies** + +- Inbound: fix-site Catalog and optional model assistance (P0/P2). +- Outbound: Population Freezer (P0). +- External: none. + +**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [x] + +```python +CanonicalEcosystem = Literal["python", "javascript"] +class ReviewAuthority(Protocol): + def review_ecosystem( + self, evidence: RepositoryEcosystemEvidence, attestation: IndependenceAttestation + ) -> RepositoryEcosystemReviewV1: ... + + def review_ground_truth( + self, evidence: FixGroundTruthEvidence, attestation: IndependenceAttestation + ) -> GroundTruthReviewV1: ... +``` + +Missing, conflicting, or rejected judgments produce an exclusion or insufficient record, never an accepted label. + +#### Baseline Lifecycle Authority + +**Responsibilities and constraints** + +- Project immutable population lifecycle events without rewriting a freeze. +- Mark the historical pilot `replacement_pending` after the reviewed withdrawal event and `replacement_complete` only after a reviewed event links its exact identities to the new population and freeze. +- Prevent replacement acceptance or spend until the historical replacement projection is complete. +- Retire a frozen baseline through an independently reviewed `retired` event bound to the population/freeze identities, contamination-evidence digest, reason code, and event digest. +- Record `development_opened` when any person receives population-specific raw observations or diagnostics before spend. The event also creates reviewed influence events for the affected repository identities. +- Prevent any retired or development-opened baseline from passing admission preflight while preserving its prior reports unchanged. +- Rebuild projections from the complete durable event set as one causal chain. Every event names its predecessor; duplicate equal events are idempotent, forks/conflicting events are refused, `replacement_bound` must name the exact reviewed withdrawal digest, and `retired` or `development_opened` is terminal for admission. + +**Dependencies** + +- Inbound: Reservation Authority, Population Freezer, and contamination review (P0). +- Outbound: Admission Orchestrator and permanent guard (P0). +- External: Artifact Repository (P0). + +**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [x] + +```python +class PopulationLifecycleKind(Enum): + WITHDRAWN = "withdrawn" + REPLACEMENT_BOUND = "replacement_bound" + RETIRED = "retired" + DEVELOPMENT_OPENED = "development_opened" +class BaselineLifecycleAuthority(Protocol): + def append(self, event: PopulationLifecycleEventV1) -> PopulationLifecycleProjectionV1: ... + def retire_baseline( + self, request: BaselineRetirementRequest, review: IndependenceAttestation + ) -> PopulationLifecycleProjectionV1: ... + def open_for_development( + self, request: DevelopmentOpeningRequest, review: IndependenceAttestation + ) -> PopulationLifecycleProjectionV1: ... + def require_admission_eligible( + self, population_digest: str, freeze_digest: str + ) -> PopulationLifecycleProjectionV1: ... +``` + +Accepted V1 lifecycle events are immutable and have no general correction transition. A rejected event never enters the projection; repairing its proposed metadata creates a different reviewed event from the same valid predecessor. No event can undo retirement/development opening or delete historical bytes, identities, reports, or authorization evidence. + +#### Population Freezer + +**Responsibilities and constraints** + +- Consume only accepted review records and the authoritative reservation census. +- Draw at least 20 eligible first-parent ordinary pushes per accepted positive and exclude the fix. +- Record each eligible ordinary push once with its reviewed canonical ecosystem. Detection-core V3 projects it once into every exact family claimed by each arm in that ecosystem and never across ecosystems; arm execution and whole coverage remain deduplicated once per underlying push. +- Reject known contaminated negatives before freeze and retire the baseline if contamination is established later. +- Count one fix push once in whole coverage and once per independently verified exact scope. +- Classify fewer than 40 positives as `development_pilot` or `insufficient`, never full replacement admission. +- Bind deterministic selection, draw, exclusions, reviews, reservation census, provenance, policies, public/private split, sealed labels, and label-blind census. +- Publish permissively licensed material only; private material is represented publicly by opaque digest-bound references. + +**Dependencies** + +- Inbound: Reservation Authority and Review Authority (P0). +- Outbound: Scoped Admission V3 and Admission Orchestrator (P0). +- External: Artifact Repository (P0). + +**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [x] / State [x] + +```python +class PopulationClass(Enum): + DEVELOPMENT_PILOT = "development_pilot" + INSUFFICIENT = "insufficient" + ADMISSION = "admission" +class PopulationFreezer(Protocol): + def freeze(self, request: PopulationFreezeRequest) -> PopulationFreezeV3: ... + def build_label_blind_population_manifest( + self, freeze: PopulationFreezeV3, review: IndependenceAttestation + ) -> LabelBlindPopulationManifestV3: ... +``` + +An unchanged request produces the same identity. An existing identity with different bytes is a conflict. Freeze writes are idempotent and never replace historical pilot bytes. + +### Persistence + +#### Artifact Repository + +**Responsibilities and constraints** + +- Persist canonical reservation, review, lifecycle, population, provenance, sealed-label, custodian-sealed observation, workflow, and final-report artifacts by content identity. +- Create an object only when its key is absent; an existing equal object is idempotent and an existing different object is a conflict. +- Read strictly by expected kind and digest, expose ordered projections, and prove nonzero bytes before decoding. +- Validate versioning and purpose-specific lifecycle policy without mutating administrator-owned configuration. +- Recover every authoritative projection from the durable store after local scratch loss. +- Enforce artifact access classes. Pre-spend observation sets and population-specific diagnostics are `custodian_sealed`; operator/developer reads return only counts, digests, and fixed reason codes. A reviewed development-opening transition is required before raw access and makes admission ineligible. +- Keep custody-head compare-and-swap in `DurableCustodyStore`; content-addressed create-only persistence is not a substitute for the mutable head protocol. + +**Dependencies** + +- Inbound: Reservation, Review, Lifecycle, Population, Arm Executor, Workflow Outcome, and Presenter (P0). +- Outbound: S3-compatible store (P0). +- External: boto3/botocore and target RustFS release (P0). + +**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [x] + +```python +class ArtifactRepository(Protocol): + def put_if_absent( + self, artifact: BenchmarkArtifact, access: ArtifactAccessClass + ) -> ContentRef: ... + def get( + self, reference: ContentRef, expected_kind: str, authority: ArtifactReadAuthority + ) -> BenchmarkArtifact: ... + def list_projection(self, prefix: ArtifactPrefix) -> tuple[ContentRef, ...]: ... + def verify_policy(self, profile: RetentionProfile) -> ArtifactStoreCapabilityReport: ... +``` + +The repository uses conditional create or immutable digest keys plus strict readback. Same-key/different-byte writes fail and never create a new accepted decision. + +#### Durable Custody Store + +**Responsibilities and constraints** + +- Implement the existing compare-and-swap custody seam using conditional object operations. +- Use canonical log digests as domain identity and ETag/version IDs only as storage concurrency tokens. +- Conditional-create the first head and conditional-update the current head; reread and validate the exact successor. +- Reject 409, 412, missing versions, stale heads, invalid content, and ambiguous server responses. +- Reconstruct current state from durable objects after process or scratch loss. +- Verify versioning and retention policy but never create buckets or mutate administrator-owned configuration. +- Separate durable pool/custody records from time-bounded transient run records. + +**Dependencies** + +- Inbound: `GuardedPopulationCustodian` (P0). +- Outbound: S3-compatible store (P0). +- External: boto3/botocore and target RustFS release (P0). + +**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [x] + +```python +class DurableCustodyStore(CompareAndSwapCustodyStore, Protocol): + def probe(self) -> CustodyCapabilityReport: ... + def current_log(self, population_digest: str) -> PopulationCustodyLog: ... + def compare_and_swap( + self, expected_log_digest: str, replacement: PopulationCustodyLog + ) -> bool: ... +``` + +The capability report covers conditional create, conditional update, two-writer conflict, version visibility, restart recovery, and current/noncurrent expiry. Real population writes require every mandatory probe to pass. + +### Evaluation and Workflow + +#### Arm Registry and Combination Planner + +**Responsibilities and constraints** + +- Bind each arm's immutable analyzer/configuration/fact/vocabulary/outcome identities and exact claimed scopes before label access. +- Preserve shipped, experimental, and combined identities separately. +- Bind each arm to lifecycle state `active` or `parked`. A parked arm requires a separately reviewed strategy-decision reference before it can enter a plan; this spec supplies no such reference for access control. +- Bind a combined arm to a closed deterministic rule over component observations and retain each component result. +- Bind one evaluation plan to the complete canonical tuple of per-arm baseline identities, policies, thresholds, sufficiency rules, and combination lineage. + +**Dependencies** + +- Inbound: detector-owned adapter and immutable identities (P0). +- Outbound: Admission Orchestrator and Scoped Admission V3 (P0). +- External: detector runtime (P1). + +**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [x] + +```python +class ArmLifecycleState(Enum): + ACTIVE = "active" + PARKED = "parked" +@dataclass(frozen=True) +class ArmRegistrationV1: + arm_identity_digest: str + lifecycle_state: ArmLifecycleState + strategy_decision_ref: ContentRef | None + claimed_scopes: tuple[ScopeKey, ...] + execution_identity: BaselineIdentity + +class CombinationRule(Enum): + UNION = "union" + INTERSECTION = "intersection" +class ArmPlanner(Protocol): + def register(self, plan: EvaluationPlanV1) -> RegisteredEvaluationPlan: ... + def combine( + self, rule: RegisteredCombinationRule, observations: tuple[ArmObservationSet, ...] + ) -> ArmObservationSet: ... +``` + +Only rules required by a pre-registered plan are implemented. A combination cannot manufacture or transfer proof: it may retain a proof reference only when family, packet/change input, and finding-location anchor are identical. Core remains the proof authority. Adding score fitting, threshold tuning, or family-specific fusion is outside the boundary. + +#### Isolated Arm Executor + +**Responsibilities and constraints** + +- Materialize one exact `LabelBlindExecutionBundleV1` per plan containing ordered case/input identities, code/change references required by the detector, arm registration, budgets, and output destination. +- Launch the arm in an isolated subprocess or container before custody spend with an allowlisted environment and no custody, private-label, sealed-label, review, or expected-outcome credentials/references. +- Prove nonzero input reads and required-stage execution in an `ExecutionReceiptV1`. +- Validate, canonicalize, and durably freeze one custodian-sealed `ArmObservationSetV1` per planned component arm; then derive combined sets without rerunning cases. +- Resume by reusing an exact digest-matching completed observation set. A missing or mismatched checkpoint cannot be treated as completed and a completed arm is not silently rerun under the same plan identity. +- Expose before spend only closed process reason codes, counts, and digests. Population-specific findings, paths, exception text, and diagnostics remain sealed. Any human opening of them triggers the lifecycle development-opening transition before access. + +**Dependencies** + +- Inbound: Arm Registry and label-blind population manifest (P0). +- Outbound: Admission Orchestrator and Artifact Repository (P0). +- External: existing local, Docker, or AX execution adapter (P1). + +**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [x] / State [x] + +```python +@dataclass(frozen=True) +class LabelBlindCaseInputV1: + case_input_digest: str + change_input_digest: str + input_manifest_ref: ContentRef +@dataclass(frozen=True) +class ArmObservationV1: + arm_identity_digest: str + case_input_digest: str + change_input_digest: str + claimed_scope: ScopeKey + finding_anchor: FindingAnchor | None + outcome: ArmOutcome + proof_candidates: tuple[LabelBlindProofCandidateV1, ...] + execution_receipt_digest: str + observation_digest: str + +@dataclass(frozen=True) +class LabelBlindProofCandidateV1: + arm_identity_digest: str + family_claim: str + finding_anchor: FindingAnchor + flagged_input_digest: str + control_input_digest: str + differential_record_ref: ContentRef + verification_receipt_ref: ContentRef + candidate_digest: str +@dataclass(frozen=True) +class ArmObservationSetV1: + plan_digest: str + arm_identity_digest: str + input_set_digest: str + observations: tuple[ArmObservationV1, ...] + execution_receipt: ExecutionReceiptV1 + observation_set_digest: str +class IsolatedArmExecutor(Protocol): + def execute( + self, registration: ArmRegistrationV1, bundle: LabelBlindExecutionBundleV1 + ) -> ArmObservationSetV1: ... +``` + +No label-bearing object exists in the executor's address space. An executor impairment occurs before spend and cannot produce an admission quality decision. Automatic retry is permitted only for the exact immutable admission identity; a changed arm, plan, input, budget, or environment requires a new population or development-opening disposition. + +#### Admission Orchestrator + +**Responsibilities and constraints** + +- Validate dependency schema and store capabilities before accepting an admission run. +- Require lifecycle eligibility, compose the reviewed V3 evaluation manifest, evaluate every arm/scope label-blind, and stop without spend if any arm or scope is insufficient. +- Execute and durably freeze all component and combined observation sets before spend; missing planned observations prevent admission. +- Re-run core authorization after observation sealing and immediately before CAS; changed, missing, unreadable, or unsealed observation bytes make `may_spend=false`. +- Commit and reread the custody transition before the custodian opens nonzero label bytes. +- Join frozen observations to ground truth only inside the custodian boundary and obtain an independent V3 release review. +- Promote a `LabelBlindProofCandidateV1` to core `ProofAttachment` only after released ground truth establishes the exact vulnerable/fixed roles, family/ecosystem scope, packet/change inputs, finding location, control binding, differential record, and verification receipt. Swapped, unrelated, or cross-scope candidates are refused; absence remains `not demonstrated`. +- Project one underlying push into exact-scope cases without rerunning it; whole-population coverage remains deduplicated by change-input identity. +- Validate proof attachments, submit every arm's exact-scope cases under the shared authority, and require the complete evaluation replay before completion. +- Persist state transitions so restart resumes the same immutable run. +- Never call legacy aggregate scoring or access-control-specific conversion helpers. + +**Dependencies** + +- Inbound: Lifecycle Authority, Population Freezer, Artifact Repository, Isolated Arm Executor, Durable Custody Store, and Arm Registry (P0). +- Outbound: Scoped Admission V3 and Workflow Outcome (P0). +- External: configured execution lanes through adapters (P1). + +**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [x] / State [x] + +```python +class AdmissionOrchestrator(Protocol): + def preflight(self, request: AdmissionRunRequestV1) -> AdmissionPreflightV1: ... + def run(self, request: AdmissionRunRequestV1) -> BenchmarkWorkflowOutcomeV1: ... + def resume(self, workflow_digest: str) -> BenchmarkWorkflowOutcomeV1: ... +``` + +`AdmissionRunRequestV1` binds the admission-class population/freeze, lifecycle projection, label-blind population manifest, registered evaluation plan, retention profiles, store capability reports, execution deadline/budgets, and workflow identity. `AdmissionPreflightV1` binds `EvaluationPlanManifestV3`, all per-arm feasibility results, lifecycle eligibility, input-set digest, and expected observation slots. After execution, the orchestrator creates `EvaluationAdmissionManifestV3` with exact sealed observation references and obtains `EvaluationAdmissionAuthorityV3` immediately before CAS. Preconditions require every planned arm/scope to be feasible and every expected observation to be durable and unchanged. Postcondition: either one validated multi-arm V3 evaluation replay is referenced by a completed envelope or an explicit insufficient/impaired envelope is durable. + +### Operations and Presentation + +#### Workflow Outcome + +**Responsibilities and constraints** + +- Persist workflow state even when no core report can be constructed. +- Persist append-only progress/heartbeat records so `run.py status` can distinguish moving work from an expired workflow without interpreting AX infrastructure health. +- Maintain one CAS-protected workflow head. Every journal entry names its sequence, predecessor, attempt, and lease owner; resume acquires a new lease only from an expired validated head, and a stale owner cannot append or renew. +- Use a closed top-level status and reason taxonomy. +- Bind status, phase, counts, safe degradations, runtime, population/freeze/plan/workflow digests, spend state, failed gate names, and optional core replay reference. +- Reject arbitrary exception text, paths, labels, repository contents, payloads, secrets, and customer content. +- Distinguish resumable interruption from terminal completion without undoing spend. + +**Contracts**: Service [ ] / API [ ] / Event [ ] / Batch [ ] / State [x] + +```python +class WorkflowStatus(Enum): + COMPLETED = "completed" + INSUFFICIENT = "insufficient" + IMPAIRED = "impaired" +class WorkflowReasonCode(Enum): + SCOPE_INSUFFICIENT = "scope_insufficient" + TIMED_OUT = "timed_out" + BUDGET_EXHAUSTED = "budget_exhausted" + RESOURCE_EXHAUSTED = "resource_exhausted" + INSTRUMENT_FAILURE = "instrument_failure" + INTERRUPTED = "interrupted" + STALLED = "stalled" +@dataclass(frozen=True) +class WorkflowCounts: + planned_arms: int + completed_arms: int + planned_inputs: int + completed_inputs: int + deferred_inputs: int +@dataclass(frozen=True) +class WorkflowJournalEntryV1: + workflow_digest: str + sequence: int + previous_entry_digest: str | None + attempt_id: str + lease_owner_id: str + phase: WorkflowPhase + heartbeat_at: str + lease_expires_at: str + counts: WorkflowCounts + entry_digest: str +@dataclass(frozen=True) +class WorkflowHeadV1: + workflow_digest: str + sequence: int + entry_digest: str + lease_owner_id: str + lease_expires_at: str + head_digest: str +class WorkflowJournalStore(Protocol): + def current_head(self, workflow_digest: str) -> WorkflowHeadV1: ... + def append_and_swap( + self, expected_head_digest: str, entry: WorkflowJournalEntryV1, head: WorkflowHeadV1 + ) -> bool: ... + def acquire_expired_lease( + self, expected_head_digest: str, attempt_id: str, lease_owner_id: str + ) -> WorkflowHeadV1: ... + +@dataclass(frozen=True) +class BenchmarkWorkflowOutcomeV1: + status: WorkflowStatus + reason_codes: tuple[WorkflowReasonCode, ...] + population_digest: str + freeze_digest: str + plan_digest: str + workflow_digest: str + labels_spent: bool + counts: WorkflowCounts + failed_gate_names: tuple[str, ...] + degradation_codes: tuple[DegradationCode, ...] + runtime_ms: int + evaluation_replay_ref: ContentRef | None + outcome_digest: str +``` + +`DegradationCode` and `WorkflowPhase` are closed enums defined in `workflow_outcome.py`; free-form messages are not serializable. `run.py status` validates the complete predecessor chain to the CAS head and ignores orphan/stale entries. An expired authoritative head derives a safe `IMPAIRED`/`STALLED` status projection. A completed outcome requires all planned arms in the referenced evaluation replay. Durable terminal outcomes never expire with the transient journal. + +#### Operator Presenter + +**Responsibilities and constraints** + +- Render deterministic JSON and one concise text summary from validated outcomes. +- For completion, resolve the validated evaluation replay and show each arm's exact-scope GO/NO-GO results, counts, digests, and named failed gates. +- For impairment, show the closed safe reason and whether labels were spent. +- Never reinterpret metrics, decide eligibility, or expose private fields. It reports validated demonstrated evidence; it does not enable BLOCK or rollout. +- Remain transport-neutral; stdout/file output may later be passed to a separately owned Discord notifier. + +**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ] + +```python +class OperatorPresenter(Protocol): + def as_json( + self, outcome: BenchmarkWorkflowOutcomeV1, replay: EvaluationReplayV3 | None + ) -> Mapping[str, JsonValue]: ... + def as_text( + self, outcome: BenchmarkWorkflowOutcomeV1, replay: EvaluationReplayV3 | None + ) -> str: ... +``` + +The presenter validates that a supplied replay matches `evaluation_replay_ref`, reports demonstrated status, and never emits BLOCK eligibility or rollout state. + +## Data Models + +### Domain Model + +```mermaid +erDiagram + INFLUENCE_EVENT }o--|| RESERVATION_CENSUS : projects + LIFECYCLE_EVENT }o--|| HISTORICAL_AUTHORIZATION : permits + ECOSYSTEM_REVIEW ||--o{ GROUND_TRUTH_REVIEW : scopes + GROUND_TRUTH_REVIEW }o--|| POPULATION_FREEZE : selected + RESERVATION_CENSUS ||--|| POPULATION_FREEZE : excludes + POPULATION_FREEZE ||--|| ADMISSION_MANIFEST : censuses + ADMISSION_MANIFEST ||--|| CUSTODY_LOG : spends + EVALUATION_PLAN ||--o{ ARM_OBSERVATION_SET : freezes + CUSTODY_LOG ||--|| CASE_RELEASE : authorizes + CASE_RELEASE ||--|| EVALUATION_REPLAY : scores + EVALUATION_REPLAY ||--|| WORKFLOW_OUTCOME : completes +``` + +Primary invariants: + +- A repository has one accepted canonical ecosystem or an insufficient record. +- A positive has explicit human judgments and one or more independently verified exact scopes. +- The permanent census contains reviewed influence, not enumeration alone. +- The public freeze binds private bytes by digest without publishing them. +- One underlying push appears once in whole coverage and once in each verified scoped denominator. +- One label-blind arm execution occurs per underlying push; the custodian projects it into exact verified scopes after spend. +- Every planned arm observation set exists before the shared spend, and every planned arm report exists before evaluation completion. +- One population gate is either wholly unspent or durably spent; there is no partial untouched state. +- A completed outcome references exactly one validated V3 replay. + +### Logical Data Model + +`RepositoryEcosystemReviewV1` binds canonical repository identity, selected ecosystem or insufficiency, source-evidence digest, producer identity, independent review, conflict disclosure, policy digest, and review digest. + +`GroundTruthReviewV1` binds case/fix/revision/location evidence, explicit judgments, exclusion reason when applicable, exact verified scopes, optional model-assistance reference, producer identity, independent review, and digest. + +`InfluenceEventV1` binds canonical network identity, event source, disposition, influence kind, subject digest, actor, independent review, policy, and digest. `ReservationCensusV1` is the sorted deterministic projection over accepted influence events and legacy sources. + +`PopulationLifecycleEventV1` is `withdrawn` with exact path/digest inventory and reason, `replacement_bound` with historical and replacement population/freeze identities, `retired` with baseline identities and contamination/invalidity evidence, or `development_opened` with the reviewed raw-observation/diagnostic access and derived influence events. Every event names its predecessor. The projection follows one complete causal chain; duplicates are idempotent, forks conflict, replacement requires the exact withdrawal, and retired/development-opened states are terminal for admission. Admission requires replacement-complete and active. + +`PopulationFreezeV3` binds population class, positive/ordinary selection digests, review-set digests, reservation census, exclusion counts, provenance, license partitions, public manifest, private-object digest, sealed-label digest, label-blind census, policies, and freeze digest. + +`EvaluationPlanV1` binds component arms, optional combined arms, exact scopes, per-arm identities, denominator policies, thresholds, sufficiency rules, outcome mappings, combination lineage, and plan digest before label access. `EvaluationPlanManifestV3` embeds the reviewed plan, label-blind population manifest, input set, and expected observation slots for arithmetic preflight. `EvaluationAdmissionManifestV3` subsequently binds the plan-manifest digest to exact custodian-sealed observation sets. `EvaluationReplayV3` binds one shared spend and V3 case release to a canonical tuple of independently valid per-arm reports. + +`BenchmarkWorkflowOutcomeV1` is the benchmark-owned root for workflow observability. It references, but does not duplicate, a completed V3 replay. + +### Physical Data Model + +Durable keys use content identity and purpose-specific prefixes: + +- `reservations/events/.json` +- `reservations/census/.json` +- `lifecycles/events/.json` +- `lifecycles/projections/.json` +- `populations//freeze/.json` +- `populations//public/` +- `populations//private/` +- `custody//head.json` +- `custody//events/.json` +- `observations///.json` +- `admissions//outcome/.json` +- `admissions//replay/.json` +- `admissions//reports/.json` +- `runs//head.json` +- `runs//journal/.json` +- `runs//temporary/` + +Population, provenance, lifecycle, custody, final observations, terminal outcomes, evaluation replays, and final reports are versioned and retained for audit. Only `runs/` journals, leases, queue state, and intermediate execution artifacts use a separately configured time-bounded policy with recorded current/noncurrent deadlines. Expiring transient data cannot make a validated replay or terminal spent outcome unreachable. Startup validates the required policies and reports impairment; application credentials never configure them. + +## Error Handling + +### Error Strategy + +- Validation and identity conflicts fail before writes or label access. +- Unsupported store capabilities, failed conditional writes, stale heads, and ambiguous rereads fail closed. +- Pre-spend inadequacy produces `INSUFFICIENT`, not NO-GO and not a clean result. +- Pre-spend arm execution or observation validation failure produces `IMPAIRED` and leaves custody unspent; it never consumes labels to diagnose the arm. +- Post-spend failures preserve `labels_spent=true` and a safe impaired outcome; retry resumes the same run. +- Unreadable, empty, skipped, or prematurely terminated required inputs produce `INSTRUMENT_FAILURE` or a bounded incomplete reason. +- A completed core GO/NO-GO exists only after replay validation. + +### Error Categories and Responses + +| Category | Examples | Outcome | +|---|---|---| +| Population invalid | reserved identity, review conflict, ambiguous ecosystem, license violation | exclusion or `INSUFFICIENT`; no spend | +| Identity conflict | same digest key with different bytes, stale policy, incompatible baseline | refuse write; new identity required | +| Custody conflict | 409, 412, stale ETag, invalid successor, missing version | `INSTRUMENT_FAILURE`; no label read unless spend already committed | +| Execution impaired | deadline, budget, disk, memory, interruption, expired heartbeat | `IMPAIRED` with a closed reason; spent flag reflects the durable custody state | +| Measurement invalid | unread input, missing stage, cross-scope case, proof mismatch | core refusal plus benchmark `INSTRUMENT_FAILURE` | +| Quality failure | valid scope metric misses a gate | core scope `NO_GO` in completed outcome | +| Insufficient evidence | population below 40 or scope precision/minimum impossible | pre-spend `INSUFFICIENT` | + +Monitoring is artifact-based: every phase appends a heartbeat/progress entry with status, counts, digests, runtime, degradation codes, and named gates. `run.py status` reports an expired workflow lease as `IMPAIRED`/`STALLED` without claiming an AX or cluster failure. Raw exception strings are retained only in local diagnostic logs subject to existing redaction and never enter shared outcomes. + +## Testing Strategy + +### Unit Tests + +- Reservation and Lifecycle Authorities: alias/fork resolution, `enumerated_only` exclusion, influenced projection, deterministic census, exact path/digest authorization, refusal of broad allowlists, causal predecessor/fork rules, replacement-pending/completeness, immutable retirement/development opening, and terminal-preflight refusal cover 3.6, 4.1–4.10, and 5.2. +- Review Authority: producer/reviewer separation, conflict disclosure, explicit ground-truth judgments, model non-authority, TypeScript-to-JavaScript mapping, polyglot insufficiency, exclusions, and multi-scope labels cover 1.5, 1.6, 2.1–2.6. +- Population Freezer: both ecosystems, two families, 39-versus-40 classification, 20:1 ordinary draws, ecosystem projection, deterministic identity, conflict refusal, license split, and post-freeze retirement cover 1.1–1.8, 3.1–3.6, 7.1–7.7. +- Arm contracts and executor: immutable identities, lifecycle state, exact scopes, sanitized execution bundle/environment, component preservation, deterministic combination, proof non-transfer, idempotent observation checkpoints, and refusal of a parked arm without a strategy artifact cover 5.6, 6.1, 8.4, 8.5, 9.1, 9.2, and 10.5. +- Workflow outcomes and presenter: closed statuses/reasons, heartbeat/lease expiry, digest validation, deterministic projection, gate names, spent state, replay resolution, and serialization rejection for secrets, paths, labels, payloads, repository content, and arbitrary exception text cover 8.3–8.5 and 10.1–10.3. + +### Core Conformance Tests + +- Exact Cartesian scope/metric rules and scope-qualified thresholds. +- Two-stage V3 validation: arithmetic feasibility accepts only the reviewed plan manifest; spend authority accepts only the matching admission manifest with every exact sealed observation byte and current custody head. +- Per-arm/per-scope feasibility witnesses and shared `may_spend=false` when any planned arm or scope is insufficient. +- Exact-scope metric derivation with no cross-scope substitution. +- Each ordinary push is stored and executed once and projected by core once per family claimed by each arm in its reviewed ecosystem, never across ecosystems. +- Multi-label push counted once in whole coverage and once per verified scope. +- Aggregate constituents and weighting preserved with no authorizing decision/gate. +- One shared preparation and custody receipt bind all planned arm reports; a missing planned report prevents evaluation replay completion. +- V3 case release validates against the V3 evaluation manifest and cannot substitute a V2 exact release type. +- V2 bytes and replay remain unchanged; pooled V2 cannot auto-upgrade to V3. +- Population, vocabulary, scope, denominator, analyzer, facts, thresholds, combination, or outcome-mapping changes require a new baseline. + +### Integration Tests + +- Artifact Repository tests cover create-only idempotency, same-key/different-byte refusal, strict kind/digest reads, custodian-sealed observation access, ordered projection, version/lifecycle validation, and full recovery of reservation/freeze/lifecycle/observation/outcome authorities after scratch loss. +- Custody adapter capability test covers conditional create/update, two concurrent writers with one winner, version visibility, committed successor reread, restart recovery, and both current/noncurrent transient expiry. Test input bytes are read and reported before interpreting results. +- Orchestrator test proves an insufficient preflight never launches arms or calls the label loader; every arm gets identical ordered label-blind case IDs; all observation sets freeze before CAS; a failed/missing arm prevents spend; a failed CAS never calls the label loader; and detector processes receive no label credentials, references, or bytes. +- Admission sealing tests mutate, remove, or swap an observation after feasibility and prove that V3 authorization refuses spend. +- Pre-spend confidentiality tests prove an impaired attempt exposes only fixed codes/counts/digests; raw observation or repository-specific diagnostic access first appends `development_opened` and influence events and blocks later spend. Automatic retry succeeds only for the exact immutable admission identity. +- Multi-arm integration proves one reviewed evaluation plan, preparation, and receipt bind every component/combined report; component observations remain preserved; missing one planned replay prevents completion. +- Crash-recovery tests cover arm failure before CAS, failure after committed spend but before label load, scoring interruption, and failure before report publication. Pre-spend failure remains unspent; post-spend failures remain spent and resume the same run. Completed arm checkpoints are not rerun under the same plan. +- Workflow journal tests cover causal sequence/predecessor validation, missing predecessors, CAS head conflicts, concurrent resume with one lease winner, stale-owner renewal refusal, and choosing a newer valid heartbeat over an orphan expired entry. Status derives `IMPAIRED`/`STALLED` only from the validated head without attributing infrastructure health. +- Case-release integration binds exact scope, role, pair, observation, evidence, change inputs, reviewed manifest, and independent release review. +- Proof integration promotes only exact label-blind candidates after ground-truth release; swapped vulnerable/fixed roles, unrelated controls, cross-family/location reuse, and mismatched pair/input are refused; missing/inconclusive proof is `not demonstrated`. The presenter reports demonstration but never BLOCK eligibility. + +### End-to-End Tests + +- A synthetic population includes Python and JavaScript/TypeScript, at least two families, multi-scope fixes, representative ordinary pushes, shipped/experimental/combined arms, and produces one evaluation replay with separate per-arm exact-scope results plus deduplicated whole coverage. +- A small freeze emits pilot/insufficient and does not spend; a full synthetic freeze reaches admission, persists custody, survives restart, and validates the V3 replay. +- A required stage that reads zero bytes, exits early, or fails produces an impaired outcome rather than a clean metric. +- A verified post-freeze contaminated ordinary push retires the baseline without rewriting the recorded result. +- Historical pilot bytes remain unchanged while the permanent guard passes only through reviewed exact-file authorizations and later replacement binding. +- A reviewed replacement remains unspendable while the historical lifecycle is replacement-pending, becomes eligible only after the exact replacement binding, and a later reviewed contamination event retires it without rewriting prior evidence. + +### Performance and Scale Tests + +- Stream freeze and replay artifacts so resident memory is bounded by one case/object batch rather than the whole population. +- Measure cold/warm workflow runtime and every arm runtime; no completion claim is made without nonzero input/read receipts. +- Exercise at least the full target shape of 40 positives and 800 ordinary pushes for manifest, policy, serialization, and report operations. +- Respect execution-lane back-pressure through adapters; benchmark correctness does not depend on cluster ownership or a particular lane. + +## Security Considerations + +- Admission labels and uncertain/non-permissive content remain in private digest-bound objects. Arms and presenters receive no label bytes. +- Pre-spend observations and population-specific diagnostics remain custodian-sealed. Human access requires a reviewed development-opening transition that disqualifies admission and reserves any newly influential repositories. +- Object-store credentials and endpoints come from environment settings, never artifacts, prompts, reports, or committed files. +- Conditional writes and reread validation prevent lost custody transitions; content digests remain the authority when ETags are not content hashes. +- Shared reports contain status, counts, digests, fixed reason/degradation codes, and named gates only. +- Differential proofs are referenced by validated digests; exploit payloads and request strings are not included in operator output. +- Independent review policy applies to ecosystem, ground truth, label-blind manifest, and post-spend case release. + +## Performance and Scalability + +Population sourcing is rate- and review-bound, not a synchronous product path. Durable writes occur incrementally and are resumable from the object store. The freezer avoids local durable accumulation, and transient run artifacts use time-bounded retention. Admission execution may use existing local, Docker, or AX lanes, but the workflow measures and reports their completion, runtime, budgets, and degradations rather than assuming infrastructure health. + +The minimum target produces at least 840 pushes before multi-scope projection. Exact-scope projections are deterministic linear passes over case metadata. Detector runtime dominates; the design does not add a second scan or proof execution merely to create metrics. + +## Migration Strategy + +```mermaid +flowchart LR + CoreV3[Approved detection core V3 predecessor] --> Reservation[Reservation and withdrawal] + Reservation --> Review[Review and population schemas] + Review --> Store[Artifact repository and lifecycle] + Store --> Freeze[Replacement freeze and census] + Freeze --> Custody[Durable custody probe and adapter] + Custody --> Orchestration[Isolated arms orchestration and outcomes] + Orchestration --> Source[Replacement sourcing and review] + Source --> Admission[One admission spend] +``` + +1. Amend, approve, implement, and independently validate `detection-core` until it exports `openultrasast.detection-core.evaluation-admission.v3`, its exact V3 release validator, and multi-arm evaluation replay while preserving all V2 schemas and replay tests. Valid-benchmark task generation waits at this predecessor gate. +2. Promote reservation semantics to production code, inventory the historical pilot exactly, and make the guard pass without broad exclusions. +3. Add the content-addressed Artifact Repository, lifecycle projections, ecosystem/ground-truth reviews, and V3 population artifacts; retain legacy pilot/train readers as non-authorizing compatibility paths. +4. Capability-test the configured durable store, then enable CAS custody and restart recovery. A failed probe blocks admission. +5. Add general arm registration, isolated pre-spend execution, observation checkpoints, multi-arm orchestration, workflow journals/outcomes, and operator presentation using synthetic populations only. +6. Source and independently review the replacement. Freeze sub-40 work as development pilots until all full-population and exact-scope preflight rules pass. +7. Bind the exact replacement lifecycle event, freeze every planned arm observation, and perform the one immutable admission spend. Downstream product work may consume only validated per-arm scoped results. + +There is no destructive rollback. Before spend, failed-attempt artifacts remain custodian-sealed and permit only an automatic retry of the exact admission identity; opening them for diagnosis records development use. After spend, no operation can restore untouched status. The run is resumed or retained as spent/impaired evidence. diff --git a/.kiro/specs/valid-detection-benchmark/requirements.md b/.kiro/specs/valid-detection-benchmark/requirements.md index f9a746f..602bb12 100644 --- a/.kiro/specs/valid-detection-benchmark/requirements.md +++ b/.kiro/specs/valid-detection-benchmark/requirements.md @@ -1,89 +1,158 @@ -# Requirements: valid detection benchmark +# Requirements Document -Turns `brief.md` into testable requirements. The brief supersedes the retired SZZ pool -(`unseen-repo-evaluation`). This spec builds a benchmark with sound ground truth and standard -security metrics. +## Introduction -## Scope decisions (resolving the brief's open questions) +This specification defines the valid detection benchmark used to determine whether a detector works on application repositories it has not seen. It replaces the 2026-10-12 pilot as admission evidence because that pilot reused repositories protected by the permanent reservation policy. The historical pilot remains preserved as withdrawn evidence, while a separately sourced population establishes a new baseline. -- **First-slice size.** Build a pilot of 10 to 12 verified positives end to end first, to prove the - pipeline, then grow to a target of about 40 verified positives. 40 gives a recall interval of roughly - ±0.10 at a true recall near 0.85 to 0.90; `score.py:required_n(p, width)` computes the exact number, - so the target is set from that, not guessed. -- **Negative-to-positive ratio.** At least 20 ordinary commits per positive, drawn from the same repos. - Negatives are free to collect, and false alarms decide adoption, so err large. Report false alarms as - a rate per 100 pushes. -- **Languages first.** Python and JavaScript/TypeScript web applications (Flask, Django, FastAPI, - Express, Koa). These are where the tool is strongest and where in-repo entry points are clear. PHP and - others come later, named explicitly as not yet covered. -- **Curation record.** Each case stores the advisory, repo, fix commit, parent, path, function, the - pre-fix and post-fix excerpts, the LLM verdict with confidence, and the human spot-check decision with - reviewer and date. The set is frozen with a digest, the same way `advisory-fixes/` freezes its pairs. -- **Lane.** Curate and score locally on the Docker lane (research is local execution). The kube `ax` - lane stays the scale target but is not required here and is still blocked on the gVisor checkpoint path. +The replacement is a general Python and JavaScript/TypeScript benchmark rather than an access-control-only population. It measures multiple vulnerability families and reports quality for each declared vulnerability-family and ecosystem scope. A whole-population result describes coverage only; it cannot substitute for a scoped quality denominator. + +## Boundary Context + +- **In scope**: population sourcing and review, fix-derived labels, ordinary-change selection, permanent reservation, development/admission separation, label isolation, lifecycle and custody, reproducible freezing, push-path replay, scoped denominators, standard metrics, failure visibility, and benchmark reporting. +- **Out of scope**: adding or tuning detection logic, facts, prompts, thresholds, or judges; wiring detector arms into the pre-push product; enabling rollout; reactivating a parked detector arm; and operating execution-plane infrastructure. +- **Adjacent expectations**: detector arms provide immutable declared identities and claimed scopes; the shared detection contracts validate admission and scoring; the proof workflow supplies demonstrated-exploit evidence; product specifications decide rollout from eligible scoped results. ## Requirements -### R1. Positives labelled from the fix, never from SZZ -- Take the vulnerability location from the security fix: the file and function the fix changed, read at - the parent commit (pre-fix). -- Do not run SZZ blame walk-back. The sourcing must not use `extract.py`'s `introducing` or `last_touch` - methods. Use the fix site directly (`fix_sites()`) and first-parent history only. -- Each positive is a pair: the vulnerable function at `parent`, and the same function at the fix commit. +### Requirement 1: General Multi-Family Population -### R2. App-shaped and reachable in-repo -- Keep a positive only if the vulnerability is reachable from an entry point inside the repo: a web - route, a request handler, or a CLI argument. -- Drop library-internal cases whose trigger depends on a caller outside the repo. A pre-push tool cannot - see that trust boundary, so scoring it there is unfair. +**Objective:** As a maintainer, I want one general benchmark spanning the initial supported languages and multiple vulnerability families, so that shared detection improvements and class-specific arms can be evaluated without building a new population for every arm. -### R3. Every positive is verified -- Verify each case with an LLM adjudication call and a human spot-check. The call must confirm: - (a) the pre-fix code is vulnerable, (b) the fix addresses it, (c) it is reachable from an in-repo entry - point. -- Record the verdict, the confidence, and the human decision. A case that fails any of (a), (b), (c), or - the human spot-check does not enter the benchmark. +#### Acceptance Criteria -### R4. Negatives are ordinary commits from the same repos -- Draw non-fix commits from the first-parent history of the same application repos, excluding the fix. -- The negative set is at least 20x the positive set. +1. The Valid Detection Benchmark shall include accepted application cases from both Python and JavaScript/TypeScript ecosystems. +2. The Valid Detection Benchmark shall include accepted positives from at least two distinct vulnerability families overall. +3. When the full replacement population is accepted, the Valid Detection Benchmark shall contain at least 40 accepted positives overall. +4. If a frozen population contains fewer than 40 accepted positives, the Valid Detection Benchmark shall classify it as a development pilot or insufficient population and shall not use it as the full replacement admission baseline. +5. When the benchmark records an ecosystem, the Valid Detection Benchmark shall use one reviewed canonical ecosystem identity for that repository. +6. If a repository's ecosystem classification remains unresolved, the Valid Detection Benchmark shall record insufficient scope evidence for that repository. +7. If a vulnerability-family and ecosystem scope lacks enough eligible cases for its pre-registered gate, the Valid Detection Benchmark shall report that scope as insufficient and shall not infer support from the whole-population count. +8. When concrete frameworks or repositories are cited, the Valid Detection Benchmark shall treat them as coverage evidence rather than as detector implementation requirements. -### R5. Held out -- No repository in the benchmark was used to build, train, tune, mine, or harvest the tool. Enforce this - with the existing used-set canonicalization (`eligibility.normalize`), the same guard the SZZ pool used. +### Requirement 2: Fix-Derived Positive Ground Truth -### R6. Measure the push path on unseen repos, with the LLM tier -- The tool a developer runs at push time must include the LLM detection tier (the hunter). That is a - product requirement on the pre-push hook, owned by `pre-push-safety-net`. The benchmark measures the - push path, not a scan mode. -- Today the shipped hook runs quick rules and the engine only. Its model path (`push/assistance.py`) - selects witnesses for already-admitted defects and cannot add a finding. Until the hunter is wired into - the hook, the benchmark reports two arms over the same changes and labels them: - - (a) the push hook as shipped (quick rules + engine), through the replay harness; - - (b) the push hook plus the hunter arm, the hunter run off-cluster over the same changes and the - records combined per change. - (b) is the number the tool is held to. (a) is what ships today. The gap between them is product work, - not a benchmark artefact, and is reported as such. -- Use the existing replay harness for (a) so the measured path is the shipped push hook. +**Objective:** As a benchmark reviewer, I want each positive anchored to a security fix and reachable application behavior, so that recall is measured against reviewed vulnerability locations rather than inferred history. -### R7. Standard metrics -- Report recall (true-positive rate), precision (positive predictive value), false-positive rate as - false alarms per 100 pushes, and exploit-verified rate (the share of flagged positives the proof arm - demonstrates with a working proof of concept). -- Report recall with a Wilson 95% interval and a repo-clustered interval, reusing `score.py`. -- The headline pairs recall and false alarms per 100 pushes in one plain sentence. +#### Acceptance Criteria -### R8. Reproducible and frozen -- The positive catalog, the negative draw, and the held-out guarantee are frozen with digests so a run is - reproducible. Re-running the freeze on an unchanged source yields the same digest. +1. When a positive candidate is sourced, the Valid Detection Benchmark shall derive its vulnerable location from the file and function changed by the security fix at the vulnerable parent revision. +2. The Valid Detection Benchmark shall not derive a positive location from a blame walk-back or another SZZ-style introducing-commit estimate. +3. When a positive candidate is reviewed, the Valid Detection Benchmark shall require a human reviewer whose recorded identity differs from the producer's identity, satisfies the pre-registered review policy, and includes a conflict disclosure to confirm that the vulnerable revision contains the defect, the fixed revision addresses it, and an entry point inside the repository can reach it. +4. If any required ground-truth or reachability judgment is rejected or missing, the Valid Detection Benchmark shall exclude the candidate and record a reason. +5. Where model-assisted adjudication is used during curation, the Valid Detection Benchmark shall record it as non-authoritative review assistance and shall not treat model agreement as a label. +6. When one fix has independently verified labels in multiple scopes, the Valid Detection Benchmark shall count the push once in whole-population coverage and once in each verified scoped denominator without granting credit to an unverified family or ecosystem. -## Non-goals -- No SZZ, no frozen-pool catch/false-alarm numbers (retired). -- No new detection capability in this spec. This measures the tool that exists. -- Not required to run on kube `ax` to be valid; the Docker lane is enough for the first numbers. +### Requirement 3: Representative Ordinary Changes -## Acceptance -- A frozen pilot of at least 10 verified positives plus their negatives, on held-out Python/JS repos, - run through the whole tool, producing recall, precision, false alarms per 100 pushes, and - exploit-verified rate, with intervals. Every positive has an LLM verdict and a human spot-check on - record. +**Objective:** As a developer, I want false alarms measured on ordinary changes from the same application environments, so that the reported interruption rate reflects the push experience. + +#### Acceptance Criteria + +1. When an accepted positive repository contributes negatives, the Valid Detection Benchmark shall draw ordinary non-security changes from that repository's first-parent history and shall exclude the security fix. +2. The Valid Detection Benchmark shall include at least 20 eligible ordinary pushes per accepted positive in each frozen population. +3. When calculating a scoped false-alarm denominator, the Valid Detection Benchmark shall include each eligible ordinary push from the matching ecosystem once for each vulnerability family claimed by the evaluated arm in that ecosystem. +4. The Valid Detection Benchmark shall not pool an ordinary push from one ecosystem into another ecosystem's denominator. +5. If an ordinary change is verified to contain a relevant security defect before freeze, the Valid Detection Benchmark shall exclude it from the population and record the reason. +6. If an ordinary change is verified to contain a relevant security defect after freeze, the Valid Detection Benchmark shall retire the affected baseline rather than silently rewriting its recorded result. + +### Requirement 4: Permanent Reservation and Historical Withdrawal + +**Objective:** As a benchmark custodian, I want prior repository use and withdrawn populations handled explicitly, so that development knowledge cannot leak into new admission evidence. + +#### Acceptance Criteria + +1. The Valid Detection Benchmark shall exclude every repository identity present in the authoritative permanent-reservation census. +2. When repository aliases, former names, redirects, or fork relationships resolve to a reserved identity, the Valid Detection Benchmark shall treat the candidate as reserved. +3. When repository code, labels, or metadata influence population selection, detector development, training, tuning, mining, fact promotion, prompt selection, or threshold selection, the Valid Detection Benchmark shall add that repository identity to the permanent-reservation census. +4. If a repository was only enumerated and none of its code, labels, or metadata influenced a decision, the Valid Detection Benchmark shall not classify it as used unless another reservation source includes it. +5. The Valid Detection Benchmark shall classify the 2026-10-12 pilot as withdrawn and non-admissible and shall not present its results as current qualification evidence. +6. When the historical pilot is withdrawn, the Valid Detection Benchmark shall preserve its bytes and digests. +7. When the historical pilot is withdrawn, the Valid Detection Benchmark shall add a reviewed lifecycle record containing the withdrawal reason and replacement relationship. +8. If historical identity-bearing material remains accessible for audit, the Valid Detection Benchmark shall authorize it only by exact path and digest and shall not create a blanket reservation-guard exclusion. +9. When the replacement population is frozen, the Valid Detection Benchmark shall use new population and freeze identities and shall not replace or rewrite the historical pilot in place. +10. When a replacement population is about to be accepted, the Valid Detection Benchmark shall pass the permanent-reservation guard while permitting only reviewed exact-path-and-digest historical authorizations and without weakening reservation semantics. + +### Requirement 5: Development and Admission Separation + +**Objective:** As a release decision maker, I want detector development separated from untouched admission, so that measured qualification is not influenced by the labels it evaluates. + +#### Acceptance Criteria + +1. The Valid Detection Benchmark shall maintain disjoint development and admission populations. +2. When a population or slice is inspected to diagnose, tune, select, or promote detector behavior, the Valid Detection Benchmark shall classify it as development or spent and shall not use it for admission. +3. When admission is prepared and before labels are opened, the Valid Detection Benchmark shall freeze a label-blind manifest containing only scope census information and digests. +4. When a label-blind manifest is reviewed, the Valid Detection Benchmark shall require a reviewer whose recorded identity differs from the producer's identity, satisfies the pre-registered review policy, and includes a conflict disclosure. +5. When admission is prepared and before labels are opened, the Valid Detection Benchmark shall bind the population, analyzer, configuration, fact set, judgment vocabulary, outcome mapping, denominator policy, thresholds, and claimed scopes to immutable identities. +6. If admission labels influence facts, strategies, matchers, prompts, examples, thresholds, judges, decision mappings, or promotions, the Valid Detection Benchmark shall mark the population spent and refuse an admission result from it. +7. When admission labels are released, the Valid Detection Benchmark shall durably record the spent transition before exposing any label bytes. +8. When a post-spend case is released for scoring, the Valid Detection Benchmark shall bind the case to its scope, role, pair identity, result, evidence, change inputs, and reviewed input manifest. +9. When a post-spend case release is reviewed, the Valid Detection Benchmark shall require a reviewer whose recorded identity differs from the producer's identity, satisfies the pre-registered review policy, and includes a conflict disclosure. + +### Requirement 6: Scoped Denominators and Feasible Gates + +**Objective:** As a detector owner, I want quality evaluated only on the families and ecosystems an arm claims, so that out-of-scope cases neither dilute nor inflate its result. + +#### Acceptance Criteria + +1. When an arm is registered and before label access, the Valid Detection Benchmark shall require the arm to declare every vulnerability-family and ecosystem scope it claims. +2. When recall or pair correctness is calculated for a scope, the Valid Detection Benchmark shall use only vulnerable and fixed cases independently verified for that exact family and ecosystem. +3. When false alarms and precision are calculated for a scope, the Valid Detection Benchmark shall use only findings and ordinary pushes assigned to that exact family and ecosystem under the pre-registered denominator policy. +4. The Valid Detection Benchmark shall report whole-population coverage separately from scoped quality metrics. +5. Where an overall aggregate is provided, the Valid Detection Benchmark shall identify its constituent scopes, sample counts, and weighting and shall not use that aggregate to authorize a capability or ecosystem claim. +6. When a gate is registered and before label access, the Valid Detection Benchmark shall prove from the label-blind census that every pre-registered threshold is arithmetically reachable for its declared denominator. +7. If a threshold is impossible or a scoped denominator fails its pre-registered minimum size or interval-precision rule, the Valid Detection Benchmark shall refuse to spend the population for that gate and shall report the scope as insufficient. + +### Requirement 7: Reproducible Population and Durable Custody + +**Objective:** As an operator, I want population state and provenance to survive restart or local-disk loss, so that a long sourcing or admission workflow remains reproducible and auditable. + +#### Acceptance Criteria + +1. When a population is frozen, the Valid Detection Benchmark shall bind its positive selection, ordinary-change draw, reservation census, review records, exclusion counts, and provenance to content digests. +2. When unchanged source material is frozen again under the same policy, the Valid Detection Benchmark shall produce the same population identity. +3. If an existing frozen identity resolves to different content, the Valid Detection Benchmark shall refuse the write and preserve the prior artifact. +4. The Valid Detection Benchmark shall keep durable population state, custody transitions, provenance, and resumable sourcing state in a versioned durable store rather than relying on local scratch files. +5. If local scratch state is lost or execution restarts, the Valid Detection Benchmark shall resume from durable custody without changing accepted decisions or identities. +6. Where material is publicly distributed, the Valid Detection Benchmark shall include only permissively licensed content; non-permissive or uncertain material shall remain private behind opaque digest-bound references. +7. The Valid Detection Benchmark shall retain durable population, provenance, lifecycle, and custody records in an auditable form. +8. When transient run records are created, the Valid Detection Benchmark shall apply a pre-registered time-bounded retention policy that expires both current and noncurrent versions by their recorded deadlines. + +### Requirement 8: Standard Measurement and Failure Visibility + +**Objective:** As a reviewer, I want every benchmark result to carry the same complete measurement record, so that a clean result, an unsupported result, and a failed instrument cannot be confused. + +#### Acceptance Criteria + +1. When a benchmark result is published, the Valid Detection Benchmark shall identify the population and freeze digests, judgment vocabulary, claimed scopes, denominator policy, analyzer and fact identities, and threshold set. +2. When a scoped result is published, the Valid Detection Benchmark shall report recall, false alarms per 100 ordinary pushes, precision, pair correctness, sample counts, and Wilson 95% intervals. +3. When a benchmark result is published, the Valid Detection Benchmark shall report whole-population coverage, completed and deferred work, instrument and model failures, runtime, and every degradation with its reason. +4. When a zero or clean-looking result is about to be interpreted, the Valid Detection Benchmark shall prove nonzero input reads and execution of every stage required by the declared arm. +5. If an input is unreadable, empty, skipped, prematurely terminated, or processed by a failed required stage, the Valid Detection Benchmark shall report an instrument failure or bounded incomplete result and shall not count it as clean. +6. When the population, vocabulary, denominator, claimed scope, analyzer identity, fact identity, threshold set, or outcome mapping changes, the Valid Detection Benchmark shall start a new baseline and shall not describe the new result as a regression trend against an incompatible baseline. + +### Requirement 9: Push-Path Arms and Demonstrated Evidence + +**Objective:** As a product owner, I want benchmark arms and proof strength reported distinctly, so that experimental assistance is not mistaken for shipped behavior or demonstrated exploitation. + +#### Acceptance Criteria + +1. When the shipped push path and an experimental detector arm are evaluated, the Valid Detection Benchmark shall report their results separately over the same eligible changes. +2. Where a pre-registered combined arm is evaluated, the Valid Detection Benchmark shall identify its combination rule before label access and shall preserve each contributing arm's result. +3. When an exploit-demonstrated rate is reported, the Valid Detection Benchmark shall require a matching differential proof artifact showing the owned effect on the vulnerable revision and its absence on the fixed revision. +4. If a flagged positive lacks a matching differential proof artifact, the Valid Detection Benchmark shall report it as not demonstrated rather than as a failed exploit or an exploit-verified finding. +5. The Valid Detection Benchmark shall not treat scanner verification, model confidence, or model agreement as demonstrated-exploit evidence. +6. When reporting advisory eligibility, the Valid Detection Benchmark shall scope the result per capability and ecosystem. +7. When reporting a blocking claim, the Valid Detection Benchmark shall require demonstrated proof. + +### Requirement 10: Operator Outcomes and Feature Boundary + +**Objective:** As an operator, I want benchmark completion and impairment states to be explicit while detector and infrastructure decisions remain with their owning workflows. + +#### Acceptance Criteria + +1. When a benchmark workflow completes, the Valid Detection Benchmark shall expose its status, counts, digests, scoped GO, NO-GO, or insufficient outcomes, and named failed gates. +2. If a benchmark workflow stalls, times out, exhausts a budget or resource, produces an insufficient result, or encounters an instrument failure, the Valid Detection Benchmark shall expose the impaired state and its reason. +3. The Valid Detection Benchmark shall not include secrets, exploit payloads, private labels, repository contents, or customer content in shared reports or operator notifications. +4. The Valid Detection Benchmark shall not modify detector logic, facts, prompts, judges, thresholds, or product rollout state while constructing or scoring a population. +5. The Valid Detection Benchmark shall not reactivate or requalify the parked access-control arm without a separate maintainer strategy decision. +6. Where an execution plane is used, the Valid Detection Benchmark shall report benchmark process outcomes without assuming ownership of worker, cluster, or network infrastructure health. diff --git a/.kiro/specs/valid-detection-benchmark/research.md b/.kiro/specs/valid-detection-benchmark/research.md new file mode 100644 index 0000000..bdcdc66 --- /dev/null +++ b/.kiro/specs/valid-detection-benchmark/research.md @@ -0,0 +1,337 @@ +# Research and Gap Analysis: valid-detection-benchmark + +## Summary + +- **Feature**: `valid-detection-benchmark` +- **Analysis date**: 2026-11-05 +- **Discovery scope**: Complex brownfield integration; gap analysis only +- **Requirements state**: Generated but unapproved. The existing `design.md` and `tasks.md` predate the reopened requirements and are stale. +- **Overall finding**: The repository has strong reusable fix-site, ordinary-history, artifact, admission, custody, proof, and measurement foundations. It does not yet have a valid replacement population, authoritative reservation/lifecycle workflow, durable custody store, canonical ecosystem review, or exact per-family/per-ecosystem quality reports. +- **Leading design candidate**: A hybrid approach that minimally extends detection-core's authoritative scoped result contract and adds benchmark-owned population governance, persistence, adapters, and operator reporting. +- **Estimated effort**: XL, more than two weeks of implementation plus separately scheduled sourcing and human review. +- **Risk**: High overall. The largest risks are cross-spec metric authority, irreversible admission spend, durable compare-and-swap semantics, and the human/data work needed for at least 40 accepted positives. + +## Analysis Context + +The selected direction is a general Python and JavaScript/TypeScript benchmark with multiple vulnerability families and quality denominators scoped by exact family and ecosystem. It supersedes the access-control-specific use of the 2026-10-12 pilot. Permanent reservation remains authoritative, the historical pilot remains byte-preserved but non-admissible, and the parked access-control arm is not reactivated by this work. + +No external dependency research was necessary for this pass. The main gaps are internal contracts and workflows. External capability research is deferred only for the deployed durable store's conditional-write semantics. + +## Current Architecture and Reusable Assets + +### Population construction + +- `benchmarks/valid/catalog.py` builds fix-site cases and already recognizes Python, JavaScript, and TypeScript files. It records vulnerability family and CWE but drops source ecosystem metadata. +- `benchmarks/valid/verify.py` reads both revisions with nonzero-byte instrumentation, records model assistance, and accepts a human spot check. Its current verdict still makes the model booleans authoritative prerequisites. +- `benchmarks/valid/assemble.py` filters accepted fix-site cases, draws at least 20 ordinary first-parent commits per distinct vulnerability push, collapses multi-function fixes into one push, freezes deterministic TOML/JSON artifacts, and refuses conflicting in-place rewrites. +- `benchmarks/unseen/eligibility.py` gathers repository identities from development sources and resolves aliases, former names, redirects, and fork networks. +- `tests/test_independent_population.py` is the authoritative permanent-reservation guard. + +### Artifact, admission, and custody contracts + +- `benchmarks/valid/artifact_contract.py` supplies canonical digest envelopes, ordered collection identities, atomic writes, idempotency, and conflict refusal. +- `src/openultrasast/core/admission_review.py` supplies reviewed label-blind `PopulationCensus`, `ScopeCensus`, `AdmissionManifestV2`, and thresholds. +- `src/openultrasast/core/admission_feasibility.py` proves threshold arithmetic before label access. +- `src/openultrasast/core/admission_case.py` binds post-spend cases to exact scope, role, pair, result, evidence, change inputs, and reviewed input manifest. +- `src/openultrasast/core/admission_spend.py` and `benchmarks/valid/population_custody.py` enforce available-to-spent custody before label bytes are released. +- `src/openultrasast/core/admission_metrics.py` rederives metrics from validated evidence and namespaces pair identity by family and ecosystem. +- `src/openultrasast/core/admission_report.py` produces authoritative GO/NO-GO reports and replay validation. +- `src/openultrasast/core/proof.py` and the search adapter provide the differential proof authority required for demonstrated claims. + +### Operational persistence patterns + +- `benchmarks/unseen/bank.py` and `src/openultrasast/plane/memory.py` provide reusable S3-compatible persistence and lifecycle checks. +- Those writers are unconditional. No production implementation of `CompareAndSwapCustodyStore` exists; only test-local in-memory implementations exercise the custody protocol. + +## Requirement-to-Asset Gap Map + +| Requirement | Existing assets | Status | Principal gap | +|---|---|---|---| +| 1. General multi-family population | `catalog.py`, source family/ecosystem metadata, multi-target freeze tests | Partial / Constraint | No replacement population; no 40-positive acceptance; source ecosystem is dropped; no canonical ecosystem review; core pools claimed scopes into one metric result | +| 2. Fix-derived positive ground truth | `catalog.build_cases`, `extract.fix_sites`, `verify_case`, `spot_check` | Partial / Constraint | Human independence policy and conflict disclosure are absent; model booleans currently control acceptance; exclusions lack durable reasons; multi-scope labels are not reviewed | +| 3. Representative ordinary changes | `ordinary_history`, `draw_ordinary`, `assemble.negatives`, legacy mislabeled-defect refusal | Partial / Constraint / Missing | No ecosystem on ordinary rows; no reviewed projection across claimed family scopes; no reviewed negative adjudication or post-freeze retirement transition | +| 4. Permanent reservation and withdrawal | `eligibility.py`, permanent guard, immutable historical files | Missing / Constraint | No authoritative reservation census consumed by freeze; no influence events; no enumerated-only disposition; no withdrawal/inventory record; no exact-file historical authorization; guard currently fails | +| 5. Development/admission separation | Core reviewed manifest, readiness, custody, spend, case release | Partial | Benchmark cannot author the label-blind manifest; development inspection is not wired to custody/lineage; durable CAS store is missing | +| 6. Scoped denominators and gates | `ScopeKey`, `DenominatorPolicy`, `ScopeCensus`, feasibility and metric validation | Partial / Constraint | One policy rule per metric currently pools all claimed scopes; no per-scope result container, minimum-size/interval rule, or `insufficient` outcome | +| 7. Reproducible population and custody | deterministic freeze, artifact contract, custody protocol, object-store patterns | Partial / Missing / Unknown | Population freeze uses legacy artifacts; no production CAS, resumable replacement workflow, license boundary, withdrawal lifecycle, or transient-record retention enforcement | +| 8. Standard measurement and failure visibility | `MeasurementRecordV2`, coverage validation, admission report/replay | Partial / Constraint | Final admission report drops some whole-population/runtime detail, records no general degradations, and cannot persist pre-report failures as bounded workflow outcomes | +| 9. Push-path arms and demonstrated evidence | separate runners, legacy hunter combine, proof contracts and adapters | Partial / Missing | No manifest-bound combined-arm rule, component report preservation, scoped demonstrated rate, or general multi-arm admission report | +| 10. Operator outcomes and boundary | core GO/NO-GO gates and steering notification policy | Partial / Missing | No general benchmark CLI/presenter, `insufficient` or failed workflow envelope, safe report projection, or notifier integration; current runner is access-control-specific | + +## Detailed Gap Findings + +### Population shape and ecosystem identity + +The existing candidate sources often carry package ecosystems such as `pip` and `npm`, while existing detection-core examples use language-like ecosystem values such as `python` and `javascript`. The benchmark needs one reviewed vocabulary and a rule for repositories containing more than one framework or ecosystem. This is a design decision; the implementation must not silently infer or coerce the value. + +`assemble.freeze()` accepts any nonempty retained population. It does not enforce the 40-positive full-population boundary or classify smaller freezes as development pilots or insufficient populations. It also freezes labels and identities in a legacy manifest rather than producing the label-blind reviewed census required before admission. + +### Curation authority + +Fix-site extraction is already the correct foundation and has strong offline tests. The human-review record is too weak for the reopened requirements: it contains only decision, reviewer text, and date. It does not bind producer identity, review policy, conflict disclosure, or individual ground-truth judgments. + +The current `verify.verdict()` accepts a case only when the model returned all three positive booleans. That makes model assistance part of label authority. The design must retain the model output as review evidence while making a policy-compliant human attestation authoritative. + +### Permanent reservation and historical pilot + +The guard was run with readable inputs: unseen TOML inputs totaled 3,475,696 bytes and the guard test file totaled 9,146 bytes. The focused test exited 1 after 33.84 seconds with 73 reserved references, all under `benchmarks`; `src`, `tests`, and `plane` contributed zero. This is a real policy conflict, not an unread-input zero. + +The pilot payloads must remain unchanged. A new reservation-owned withdrawal inventory must bind each retained historical path and SHA-256 digest, record its non-admissible status and replacement relationship, and authorize only those exact files. The guard may consume reviewed exact-file authorizations but must not gain a directory exclusion or repository-name allowlist. + +`eligibility.build_used()` is a broad discovery collector, not an authoritative event ledger. The replacement workflow needs reviewed influence dispositions so an enumerated-only repository can be distinguished from one whose code, labels, or metadata informed a decision. + +### Scoped metric authority + +Detection-core already validates exact `ScopeKey` values, rejects cross-scope pair substitution, preserves whole-population coverage alongside in-scope cases, and proves threshold feasibility. However, one `DenominatorPolicy` rule currently names the complete claimed scope tuple and `derive_admission_metrics()` pools those scopes into one `MetricResult` per metric. `MetricResult` carries no scope or weighting metadata. + +Therefore the requirement for independently interpretable family/ecosystem results cannot be implemented safely only in `benchmarks/valid/metrics.py`. That would create a second metric authority. The design must choose a versioned detection-core extension, such as one admission manifest producing a canonical tuple of exact-scope reports plus a non-authorizing aggregate. Separate post-spend reports must not independently reinterpret one label release. + +The core also needs an explicit `insufficient` outcome or equivalent non-authorizing status for scopes whose minimum sample or interval rule fails. GO/NO-GO alone cannot represent a pre-spend denominator that is validly too small. + +### Durable custody and recovery + +The custody protocol correctly requires a successful compare-and-swap append and reread before label access. Its production storage seam is unimplemented. Existing object-store adapters provide versioning and lifecycle inspection but not conditional compare-and-swap. + +Before selecting an adapter, the design phase must capability-test the deployed store for conditional create/update, concurrent-writer behavior, version visibility, restart recovery, and current/noncurrent expiry. If the store cannot provide the required semantics, the design needs a different durable serialization authority; ordinary overwriting `put_object` calls are not sufficient. + +### Measurement, proof, and operator outcomes + +The v2 core measurement path is substantially stronger than the legacy `benchmarks/valid/metrics.py` path. It revalidates evidence, metrics, gates, and replay identities. The benchmark should adapt to it rather than extend the legacy aggregate scorer as an admission authority. + +The current v2 report does not carry a durable workflow outcome for exceptions that happen before report construction. Instrument failures fail closed, but operators cannot distinguish a bounded incomplete run from an absent report. A benchmark-owned workflow envelope can record safe reason codes and digests without weakening the authoritative core report. + +Proof authority already exists. The missing work is to bind proof-derived scoped rates and any combined arm rule into the pre-label baseline. Legacy shape checking in `metrics.demonstrated()` is useful compatibility code but cannot authorize blocking. + +## Architecture Pattern Evaluation + +| Option | Description | Strengths | Risks / Limitations | Effort / Risk | +|---|---|---|---|---| +| A. Extend existing components | Expand current `assemble.py`, `metrics.py`, admission contracts, and guard in place | Few new modules; maximum reuse; short path for simple criteria | Bloating of already mixed responsibilities; legacy and v2 authority remain easy to confuse; strict core schemas make broad in-place edits risky | L–XL / High | +| B. New benchmark-only layer | Leave detection-core unchanged and implement scoped projections, lifecycle, custody, and reports entirely under `benchmarks/valid/` | Clear benchmark ownership; limited core migration | Duplicates or bypasses authoritative metric algebra; unsafe after one irreversible spend; creates competing report authorities | L / High integrity risk | +| C. Hybrid versioned extension | Add the minimum scoped-result and workflow-status contracts upstream; add benchmark-owned population, reservation, custody adapter, case adapters, and presenter | Preserves one metric authority and existing ownership split; migration can be staged; most current assets remain reusable | Cross-spec coordination and artifact migration; more components and conformance tests | XL / High overall, Medium per component | + +Option C is the leading design candidate because it is the only approach that satisfies scoped denominators without duplicating admission authority. It remains a candidate until the core schema shape and durable-store capability are resolved in design. + +## Compatibility and Migration Boundaries + +- Historical pilot and measurement payloads are immutable audit inputs, never migration targets. +- Legacy `metrics.py` and the parked access-control helpers remain readable historical/development paths but do not authorize the replacement admission baseline. +- New population, lifecycle, census, manifest, custody, run, and report artifacts require versioned kinds and new identities. +- A population, vocabulary, scope, denominator, analyzer, fact, threshold, or outcome-mapping change creates a new baseline. +- Detection-core owns metric algebra, scope validation, spend validation, and authoritative replay. The benchmark owns sourcing, review records, reservation/lifecycle state, durable label custody, arm execution adapters, and operator presentation. +- The replacement workflow must not call or requalify the parked access-control arm unless a separate maintainer decision changes that policy. + +## Complexity and Risk Assessment + +### Effort + +- **Core scoped-result extension**: L. It changes strict dataclasses, canonical digests, report/replay validation, and conformance tests. +- **Reservation withdrawal and exact-file authorization**: M. The code is bounded, but migration evidence and immutable inventory require care. +- **Ecosystem, review, and population freeze schemas**: L. Existing catalog and assembly code are reusable, but responsibilities must be separated. +- **Durable CAS custody adapter and recovery tests**: M–L, depending on store capability. +- **General arm orchestration, workflow outcomes, and presentation**: L. +- **Replacement sourcing and independent review**: XL elapsed work; rate limits, license decisions, repository availability, and human review dominate. +- **Overall**: XL, more than two weeks, with operational curation continuing beyond pure code implementation. + +### Risk + +- **Overall risk: High.** Admission spend is irreversible, the existing guard is red, and one wrong scope projection can authorize a false claim. +- **Schema migration risk: High.** Core artifacts are closed, digest-bound graphs; changing fields requires full replay and compatibility tests. +- **Persistence risk: High until measured.** Durable conditional-write semantics are unverified on the deployed store. +- **Population availability risk: High.** Forty accepted, reachable, permissively distributable positives across two language groups and multiple families may require many more candidates. +- **Detection implementation risk: Low within this spec.** Detector logic is explicitly out of scope and must remain unchanged. + +## Design-Phase Recommendations + +1. Define a versioned core scoped-result container: exact `ScopeKey` result rows, one non-authorizing aggregate, minimum-size/interval sufficiency, and a workflow-safe `insufficient` status. +2. Keep core changes in the detection-core ownership boundary. The benchmark design consumes that contract and must not reproduce its metric algebra. +3. Add an authoritative reservation/influence ledger and a digest-bound withdrawal inventory before sourcing the replacement. +4. Define one canonical ecosystem vocabulary and independent review record before changing catalog or freeze formats. +5. Replace model-controlled acceptance with human-authoritative reviewed ground truth while retaining model output as non-authoritative evidence. +6. Author label-blind census/manifest artifacts from the new freeze and integrate them with existing readiness, spend, case release, and replay contracts. +7. Capability-test the durable store before committing to a CAS adapter. Preserve the custody protocol's spend-before-read ordering. +8. Add a benchmark workflow envelope for completion, insufficient, timeout, resource, and instrument-failure outcomes using fixed redacted reason codes. +9. Migrate in this order: core scoped contract; reservation/withdrawal; ecosystem and review schemas; population freeze; durable custody; arm adapters; scoped reporting; replacement sourcing and review. + +## Research Needed for Design + +1. **Scoped result shape**: one manifest with a tuple of per-scope reports versus another versioned container; define aggregate weighting and replay rules. +2. **Canonical ecosystem taxonomy**: package ecosystem, runtime/framework ecosystem, or a reviewed composite; specify multi-ecosystem repositories. +3. **Durable CAS capability**: conditional write semantics, concurrent writers, version visibility, crash recovery, and lifecycle enforcement on the deployed store. +4. **Population sizing**: candidate yield needed to reach 40 accepted positives and the pre-registered minimum/interval rule for each claimed scope. +5. **Historical authorization format**: review subject, exact path/digest inventory, relocation rules, and guard integration. +6. **Safe workflow reason taxonomy**: stable operator-facing codes that reveal no private labels, paths, payloads, or exception text. +7. **License custody**: approved permissive-license vocabulary and opaque private-reference workflow for uncertain or non-permissive material. + +## Validation Evidence + +- The requirement headings and acceptance criteria passed the EARS mechanical gate before this analysis. +- Focused reservation guard command: + + ```text + PYTHONPATH=src:tests .venv/bin/python -m pytest -q -p no:cacheprovider \ + tests/test_independent_population.py::test_no_reserved_repository_is_referenced_by_the_development_tree + ``` + +- Result: `1 failed in 33.84s`, exit code `1`, with `73` benchmark references and zero references from `src`, `tests`, or `plane`. +- No full test suite was run for this read-only analysis. + +## References + +- `.kiro/specs/valid-detection-benchmark/requirements.md` +- `.kiro/specs/valid-detection-benchmark/design.md` — stale reference only +- `.kiro/steering/detection-quality.md` +- `.kiro/steering/storage-streaming.md` +- `.kiro/steering/code-standards.md` +- `benchmarks/valid/` +- `benchmarks/unseen/eligibility.py` +- `tests/test_independent_population.py` +- `src/openultrasast/core/admission_*.py` +- `src/openultrasast/core/measurement.py` +- `src/openultrasast/core/proof.py` + +--- + +## Design Discovery and Synthesis (2026-11-06) + +### Discovery Summary + +- **Discovery type**: Complex integration, full discovery. +- **Selected pattern**: Hybrid versioned extension. Detection-core remains the sole authority for scope validation, metric algebra, threshold evaluation, and replay. The benchmark owns population governance, reservation state, review, freezing, custody adapters, arm execution, workflow outcomes, and operator presentation. +- **Critical prerequisite**: The replacement admission baseline requires a detection-core V3 admission root. V2 necessarily pools all claimed scopes and cannot be projected into authoritative per-scope results after label release. +- **Compatibility rule**: V2 artifacts remain byte-preserved and replayable. A pooled V2 result is never auto-upgraded; the V3 population starts a new baseline. + +### Research Log + +#### Detection-core scoped-result seam + +- **Context**: Requirements 6 and 8 require exact family-and-ecosystem metrics, per-scope gates, whole-population coverage, and non-authorizing aggregates. +- **Sources consulted**: `measurement.py`, `admission_context.py`, `admission_feasibility.py`, `admission_metrics.py`, `admission_report.py`, `_admission_replay.py`, and their core admission tests. +- **Findings**: + - V2 assigns every standard metric rule the full claimed-scope tuple, sums census capacity across the tuple, derives one metric result per metric, and stores no scope on `MetricResult`. + - The smallest safe change is a new V3 root with exact-scope metric rules and thresholds, pre-spend scope sufficiency, one canonical result per scope, separate deduplicated whole-population coverage, and an optional aggregate type with no gate or decision field. + - `insufficient` belongs to pre-spend feasibility. Because custody releases one population, any insufficient claimed scope makes `may_spend` false. Partial-scope label release is a different custody model and is out of scope. + - Multi-scope whole-population coverage must deduplicate by canonical change-input identity; summing scoped rows can count one multi-label push more than once. +- **Implications**: This spec consumes the V3 contract as an upstream prerequisite. It does not duplicate metric algebra under `benchmarks/valid/` and does not modify detection-core-owned files. + +#### Canonical ecosystem and human review authority + +- **Context**: Existing source records carry package ecosystem and license evidence, while the valid catalog drops it and current model booleans control acceptance. +- **Sources consulted**: `benchmarks/unseen/source.py`, `benchmarks/valid/catalog.py`, `verify.py`, `assemble.py`, and review contracts under `src/openultrasast/core/`. +- **Findings**: + - The V1 benchmark vocabulary is `python` and `javascript`; TypeScript maps to the JavaScript ecosystem while remaining visible as an observed language. Package ecosystems such as `pip` and `npm` remain source evidence, not canonical scope values. + - One independent repository review selects exactly one canonical ecosystem. A polyglot or conflicting case is `insufficient`, never silently coerced. + - Ground truth is a human attestation over defect presence, fix effect, in-repository reachability, and exact verified scopes. Model output is retained only as a digest-bound assistance reference and never participates in acceptance algebra. +- **Implications**: Cataloging preserves source evidence; a new review component owns authoritative decisions; freezing consumes only accepted reviewed records. + +#### Permanent reservation and historical withdrawal + +- **Context**: The old pilot is permanently reserved but must remain accessible for audit without creating a broad guard exclusion. +- **Sources consulted**: `benchmarks/unseen/eligibility.py`, `tests/test_independent_population.py`, the current pilot artifacts, and the focused guard evidence recorded above. +- **Findings**: + - Broad discovery of repository names is not an influence ledger. A reviewed append-only event must distinguish `enumerated_only` from `influenced` and resolve aliases/forks before projection into the permanent census. + - Historical authorization is an append-only lifecycle: a `withdrawn` event binds exact repository-relative paths and SHA-256 digests; a later `replacement_bound` event links the new population and freeze identities. + - The production guard permits only exact path-and-current-digest matches. Relocation or byte changes require a new reviewed event; repository-name or directory allowlists are forbidden. +- **Implications**: Reservation semantics move from test-only logic into `benchmarks/unseen/reservation.py`; the existing test becomes a consumer of that authority. + +#### Durable compare-and-swap custody + +- **Context**: The core requires a committed and reread custody transition before exposing label bytes, but current S3 writers are unconditional. +- **Sources consulted**: installed boto3/botocore service model, AWS S3 conditional-write documentation, RustFS S3 compatibility documentation, `population_custody.py`, `bank.py`, and `plane/memory.py`. +- **Findings**: + - The locked boto3/botocore `1.43.107` service model exposes `IfMatch` and `IfNoneMatch` on `PutObject`. + - Amazon S3 documents conditional create/update using `If-None-Match: *` and `If-Match` ETags, with 412 for failed preconditions and possible 409 conflicts under concurrency. + - RustFS documents common conditional requests, versioning, and lifecycle, but explicitly requires applications to validate exact advanced operations against the target release. Its published compatibility matrix does not establish this project's complete CAS protocol. + - The adapter must therefore perform a target-store capability probe for conditional create, conditional update, two-writer conflict, version visibility, restart recovery, and lifecycle behavior. Unsupported or ambiguous behavior fails closed before any real population is stored. +- **Implications**: A benchmark-owned S3 CAS adapter uses content digests as domain identity and ETag/version only as the storage concurrency token. Existing unconditional `CorpusObjects` is not reused as custody authority. +- **Official references**: + - [Amazon S3 PutObject API](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html) + - [Amazon S3 conditional writes](https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes.html) + - [RustFS S3 protocol and compatibility scope](https://docs.rustfs.com/en/administration/protocols/s3) + - [RustFS S3 compatibility matrix](https://docs.rustfs.com/en/reference/s3-compatibility) + +#### General arm orchestration and operator outcomes + +- **Context**: `admission_run.py` is access-control-specific and the parked arm must remain parked. Completed core reports cannot represent failures before report construction. +- **Sources consulted**: `benchmarks/valid/admission_run.py`, `admission_scoring.py`, `metrics.py`, core case/proof/report contracts, and `operator-notifications.md`. +- **Findings**: + - The reusable ordering is preflight, authorize, durable spend, label release, label-blind arm execution, reviewed case binding, core score, and replay validation. + - Arms receive immutable label-blind case inputs. The custodian alone joins detector observations to released ground truth. Combined-arm rules are digest-bound before label access and preserve every component result. + - Early or impaired runs need a benchmark-owned envelope with a closed redacted reason taxonomy. It may reference a completed core replay but cannot fabricate a core report. + - The spec owns deterministic JSON/text presentation. Discord transport remains outside the boundary because no shared notifier implementation exists and steering permits separate wiring. +- **Implications**: The replacement runner is general and cannot import access-control-specific site judgment helpers. + +### Architecture Pattern Evaluation Update + +| Option | Decision | Rationale | +|---|---|---| +| Extend V2 and legacy benchmark modules | Rejected | Pooled V2 metrics cannot become exact-scope authority; mixed modules would retain competing admission paths. | +| Benchmark-only V3 projection | Rejected | It would duplicate detection-core algebra after irreversible label release. | +| Hybrid versioned extension | Selected | One authoritative core result graph, bounded benchmark ownership, preserved V2 compatibility, and independently testable persistence/adapters. | + +### Design Decisions + +#### Decision: Treat detection-core V3 as an explicit upstream prerequisite + +- **Selected approach**: Detection-core owns `DenominatorPolicyV3`, exact-scope feasibility, scoped results, deduplicated whole-population coverage, non-authorizing aggregate, report, and replay. This spec consumes those contracts. +- **Rationale**: Scope and metric authority already belong to detection-core; moving them into benchmark code would create hidden shared ownership. +- **Trade-off**: Implementation cannot begin admission scoring until the dependency amendment lands, but population governance and non-label-bearing work can proceed in parallel. + +#### Decision: Make population spend atomic across all claimed scopes + +- **Selected approach**: Every claimed scope must be feasible before the one population label release. Any insufficiency emits a pre-spend outcome and leaves the population unspent. +- **Rationale**: The current custody object releases the population as a unit. Partial release would expose labels for all scopes while pretending some remained untouched. +- **Trade-off**: Sparse scopes delay admission, but the result remains interpretable and reusable. + +#### Decision: Use reviewed repository-level ecosystem identity + +- **Selected approach**: Canonical values are `python` and `javascript`; TypeScript is an observed language within `javascript`. One independent review chooses the repository ecosystem or records `insufficient`. +- **Rationale**: This matches the existing core examples, gives stable denominators, and prevents framework-specific inventories from becoming architecture. +- **Trade-off**: Ambiguous polyglot repositories are excluded from admission until their benchmark role is clear. + +#### Decision: Build reservation state as an append-only reviewed projection + +- **Selected approach**: Influence and lifecycle events are immutable; deterministic censuses and exact-file authorizations are projections. +- **Rationale**: It distinguishes enumeration from influence, preserves historical bytes, and makes every exception auditable without weakening the guard. +- **Trade-off**: Accepted V1 lifecycle events have no general correction transition. Invalid proposals are rejected before projection; terminal development/retirement states cannot be undone. + +#### Decision: Adopt S3 conditional writes only after a capability probe + +- **Selected approach**: Use existing boto3 and S3 conditional headers through a new CAS adapter; validate the deployed RustFS release and bucket policy before real use. +- **Rationale**: The SDK supports the API, but compatibility claims are not a substitute for testing the exact target store and concurrency protocol. +- **Trade-off**: A failed probe blocks custody implementation; there is no unsafe unconditional-write fallback. + +#### Decision: Keep workflow status outside the authoritative core report + +- **Selected approach**: A benchmark envelope records completed, insufficient, timeout, budget, resource, instrument, and interruption outcomes. Completed entries reference the core V3 replay. +- **Rationale**: Pre-report failures need durable visibility but must not be made to look like a quality decision. +- **Trade-off**: Operators consume two nested authorities: workflow completion from the benchmark and quality decisions from detection-core. + +### Synthesis + +- **Generalization**: Population review, arm registration, and outcome reporting are horizontal contracts. Concrete repositories, frameworks, and vulnerability vocabularies remain data/evidence. +- **Build versus adopt**: Reuse core canonicalization, case/proof/admission contracts, benchmark artifact envelopes, Git identity resolution, boto3, S3 conditional requests, and RustFS versioning/lifecycle. Build only the missing governance, CAS adapter, general orchestration, and presentation seams. +- **Simplification**: No partial-scope custody, no V2 migration, no benchmark metric engine, no notifier transport, no detector plugin framework, and no access-control compatibility adapter in the new path. + +### Updated Risks and Mitigations + +- **Upstream V3 drift** — pin exact schema identifiers and add consumer conformance tests before population spend. +- **Sparse scope blocks the whole gate** — run label-blind feasibility while sourcing and classify small freezes as pilots; never open labels to learn sample adequacy. +- **RustFS CAS mismatch** — run the exact capability probe against the configured target and fail closed. +- **Polyglot ambiguity** — require independent ecosystem review; unresolved rows are insufficient. +- **Model influence on labels** — store assistance separately and derive acceptance only from human attestations. +- **Historical authorization broadens silently** — normalize paths, verify current bytes, and accept only reviewed exact path/digest inventory rows. +- **Post-spend interruption** — persist the spent transition and workflow envelope; resume the same immutable run without restoring untouched status. + +### Design Review Gate Refinements + +The automatic gate required two bounded repair passes. The resulting refinements are architectural commitments, not implementation preferences: + +- **Two-stage V3 authorization**: `EvaluationPlanManifestV3` supports label-blind arithmetic feasibility before arm execution. After every custodian-sealed observation set exists, `EvaluationAdmissionManifestV3` binds their exact bytes and core derives the spend authority against the current custody head. +- **One shared multi-arm root**: one reviewed plan and one custody receipt bind independent per-arm scoped reports inside one `EvaluationReplayV3`. Missing any planned arm prevents completion. +- **Execute before spend, inspect only after disqualification**: isolated arms run before labels open. Observation sets remain custodian-sealed; only closed counts/digests/reason codes are visible. Human access or any plan/configuration change records `development_opened`, adds influence events, and makes the population admission-ineligible. +- **Core-owned ordinary projection**: the freeze stores and executes an ordinary push once with its reviewed ecosystem. Detection-core projects it per claimed family in that ecosystem and deduplicates whole-population coverage. +- **Typed proof promotion**: pre-spend proof candidates carry no ground-truth role. The custodian promotes one to `ProofAttachment` only after exact family, pair, role, input, location, differential record, control, and receipt binding succeeds. +- **Separate durable authorities**: a content-addressed artifact repository owns immutable events, freezes, sealed observations, terminal outcomes, reports, and replays; custody and workflow heads use stronger conditional-update protocols. Expiring run journals cannot make terminal evidence unreachable. +- **Causal lifecycle and workflow chains**: lifecycle events name predecessors and terminal development/retirement cannot be undone. Workflow heartbeats use a CAS head, sequence, predecessor, attempt, and lease owner so concurrent resume and stale renewal are deterministic. +- **Explicit predecessor gate**: implementation task generation waits for an approved and validated `detection-core` amendment exporting `openultrasast.detection-core.evaluation-admission.v3`; this spec does not absorb upstream core changes. diff --git a/.kiro/specs/valid-detection-benchmark/spec.json b/.kiro/specs/valid-detection-benchmark/spec.json new file mode 100644 index 0000000..25ef5b4 --- /dev/null +++ b/.kiro/specs/valid-detection-benchmark/spec.json @@ -0,0 +1,31 @@ +{ + "feature_name": "valid-detection-benchmark", + "created_at": "2026-10-12T14:38:18Z", + "updated_at": "2026-11-06T08:15:41Z", + "language": "en", + "phase": "design-generated", + "approvals": { + "requirements": { + "generated": true, + "approved": true + }, + "design": { + "generated": true, + "approved": true + }, + "tasks": { + "generated": false, + "approved": false + } + }, + "ready_for_implementation": false, + "depends_on": [ + "detection-core" + ], + "related_specs": [ + "access-control-prover", + "pre-push-safety-net", + "search-with-proof" + ], + "approval_note": "2026-11-06: requirements auto-approved by explicit -y request; the replacement design passed the bounded automatic review gate and the maintainer explicitly approved it. Implementation task generation remains gated on an approved, implemented, and independently validated detection-core amendment exporting openultrasast.detection-core.evaluation-admission.v3; valid-detection-benchmark does not own upstream core edits." +}