feat(detection-core): foundation contract package (trace, verdict, coverage, provenance)

2026-11-14 10:18:17 +00:00 · 2026-10-30 16:04:10 +01:00
parent 839d58e3d3
commit 0b56ad5514
12 changed files with 1077 additions and 0 deletions
@@ -0,0 +1,228 @@
+# Design Document: detection-core
+
+## Overview
+
+**Purpose:** detection-core specifies the single security object every detection arm reuses:
+the canonical trace with its verdict, coverage, budget, provenance, and admission contracts.
+It extracts what `access-control-prover` already implements for one family into a
+family-agnostic contract, so arms, fact packs, and consumers stop maintaining parallel
+runner contracts.
+
+**Users:** spec authors and implementers of detection arms (prover, hunter, push hook,
+proof arm) consume the contract; the maintainer reads admission GO/NO-GO over it.
+
+**Impact:** replaces per-arm envelopes with one contract; existing behavior stays
+byte-identical per consumer during migration (the 2026-10-30 admission suite is the
+regression gate).
+
+### Goals
+- One canonical trace schema and three-way verdict vocabulary shared by all arms (Req 1, 2, 5).
+- Coverage accounting that makes every zero explainable (Req 3); instrument rule enforced at core level.
+- A measured, per-workload-class time budget instead of a static constant (Req 4).
+- Provenance digests on every analysis, with refusal on mismatch (Req 6).
+- A generalized admission gate over untouched populations (Req 7), no fitting on labels with
+  reversed-pair probes (Req 8), models bounded to residuals (Req 9), families as fact packs
+  with scope disclosure (Req 10), consumer contracts (Req 11), fixed measurement schema (Req 12).
+
+### Non-Goals
+- New vulnerability families or their fact content.
+- The pre-push transaction itself (pre-push-safety-net consumes this core).
+- Fitted models anywhere in the core path.
+- Engine/graph internals (contributor-scan).
+
+## Boundary Commitments
+
+### This Spec Owns
+- The canonical trace schema (entry, hops, operation, witnesses, guards, judgment, completeness, digests) and its deterministic serialization.
+- The verdict contract: proven claim (witness attached) vs bounded silence (coverage reason attached); indeterminate never converts to clean.
+- Coverage accounting schema (entry recognition, per-call resolution buckets, unchecked reasons) and the zero-byte instrument refusal.
+- The measured-budget contract: complexity features recorded per change, per-workload-class budget derivation published with sample/percentile/hardware, configurable budget.
+- Provenance stamping (analyzer/fact/acquisition/configuration/population/input digests) and refusal on digest mismatch.
+- The generalized admission contract: consume-only manifest validation, paired scoring, gates, GO/NO-GO report schema, family-scope disclosure.
+- The fact-pack admission contract (reviewed facts, provenance stamping, family tag).
+- The consumer interface for the pre-push transaction and the proof layer.
+
+### Out of Boundary
+- Pre-push hook transaction, rollout policy, capability registry (pre-push-safety-net).
+- Family fact content (fact-pack specs, starting from access-control-prover's rulesets).
+- Hunter/proposer internals and proof-arm internals (they consume the contract).
+- Engine/CPG internals (contributor-scan).
+
+### Allowed Dependencies
+- `valid-detection-benchmark` pools and freeze artifacts (read-only; never label-tuned).
+- Existing prover implementation as the first contract implementation.
+- Steering rules in `detection-quality.md` (no fitting on pool labels; instrument rule; stop conditions).
+
+### Revalidation Triggers
+- Trace schema or verdict vocabulary changes → every arm and consumer re-validates.
+- Admission report schema changes → every admission gate re-runs.
+- Budget derivation method changes → new baseline under Req 12 (not a trend).
+- Fact-pack admission contract changes → all fact packs re-reviewed.
+
+## Architecture
+
+### Existing Architecture Analysis
+Today each arm owns its envelope: `prover_arm.py` (records + summarize), `development_runner.py`
+(bundle + funnel + gates), `arm_regression.py` (arm records), `admission_run.py` (manifest +
+paired scoring). The trace itself (`Trace`, `ProverResult`) exists in the prover. The gap is
+the missing shared contract layer, not missing analysis code.
+
+### Architecture Pattern & Boundary Map
+
+Pattern: **thin core contract package; existing implementations adopt it incrementally**
+(hybrid option C from research.md). The core owns contracts only — no analysis logic.
+
+```mermaid
+flowchart LR
+  subgraph CORE[detection-core contract package]
+    T[trace schema + verdict]
+    C[coverage accounting]
+    B[measured budget]
+    P[provenance digests]
+    A[admission contract]
+    F[fact-pack contract]
+  end
+  subgraph ARMS[arms / fact packs]
+    PR[access-control-prover arm]
+    FP[family fact packs]
+    HU[hunter proposal lane]
+  end
+  subgraph CONSUMERS[consumers]
+    PP[pre-push-safety-net]
+    PR2[search-with-proof]
+    BM[valid-detection-benchmark]
+  end
+  ARMS --> CORE
+  CORE --> CONSUMERS
+  BM -. measures .-> CORE
+```
+
+### Technology Stack
+
+| Layer | Choice | Role | Notes |
+|---|---|---|---|
+| Core contract | Python, frozen dataclasses + canonical JSON | trace/verdict/coverage/budget/admission schemas | mirrors existing Trace.record() conventions |
+| Runners | existing benchmark runners | adopt core contracts incrementally | behavior byte-identical per consumer |
+| Tests | pytest | contract conformance per consumer | admission suite is the regression gate |
+
+## File Structure Plan
+
+### Directory Structure
+```
+src/openultrasast/core/          # NEW: the contract package (no analysis logic)
+├── __init__.py                  # public contract exports
+├── trace.py                     # canonical trace schema, deterministic serialization (Req 1, 2)
+├── verdict.py                   # proven-claim vs bounded-silence contract (Req 5)
+├── coverage.py                  # entry/resolution/unchecked accounting + zero-byte refusal (Req 3)
+├── budget.py                    # measured budget: complexity features, per-class derivation (Req 4)
+├── provenance.py                # digest stamping + mismatch refusal (Req 6)
+├── admission.py                 # manifest validation, paired scoring, gates, GO/NO-GO schema (Req 7, 10)
+├── probes.py                    # reversed-pair / holdout probes for fitted components (Req 8)
+└── factpack.py                  # family fact-pack admission contract (Req 10)
+tests/test_core_trace.py         # conformance tests per contract module
+tests/test_core_admission.py
+tests/test_core_budget.py
+```
+
+### Modified Files
+- `src/openultrasast/hunt/prover.py`, `semantic/obligations/trace.py` — implement/re-export
+  the core trace contract for access_control (Req 1, 2, 5); behavior unchanged.
+- `benchmarks/valid/prover_admission.py`, `admission_scoring.py`, `admission_run.py` — become
+  thin consumers of `core/admission.py` (Req 7); existing refusal order and report schema preserved.
+- `benchmarks/valid/development_runner.py`, `arm_regression.py`, `prover_arm.py` — adopt
+  `core/coverage.py`, `core/budget.py`, `core/provenance.py` (Req 3, 4, 6, 11, 12).
+- `benchmarks/valid/run_manifests.py` — bundle digests feed `core/provenance.py` (Req 6).
+
+## System Flows
+
+```mermaid
+sequenceDiagram
+  participant PP as pre-push transaction
+  participant CORE as detection-core
+  participant ARM as arm (prover/fact pack)
+  participant BM as benchmark pools
+  PP->>CORE: push (base, head), budget class
+  CORE->>CORE: scope change regions (Req 4)
+  CORE->>ARM: analyze scoped regions
+  ARM-->>CORE: trace + coverage + complexity + seconds
+  CORE->>CORE: stamp digests, derive budget check (Req 4, 6)
+  CORE-->>PP: verdict (proven claims with witnesses | bounded silence)
+  BM-->>CORE: dev pool + untouched admission pool
+  CORE->>CORE: admission gate: manifest, pairs, criteria -> GO/NO-GO (Req 7)
+```
+
+## Requirements Traceability
+
+| Req | Summary | Components | Contracts |
+|---|---|---|---|
+| 1 | canonical trace | core/trace.py | Trace schema, deterministic record() |
+| 2 | three-way judgment | core/trace.py, core/verdict.py | Judgment enum; indeterminate-never-clean rule |
+| 3 | coverage accounting | core/coverage.py | funnel + resolution buckets + zero-byte refusal |
+| 4 | measured budget | core/budget.py | complexity features; per-class derivation; gate names derivation |
+| 5 | verdict contract | core/verdict.py | proven claim w/ witness; bounded silence w/ reason |
+| 6 | provenance | core/provenance.py | digest stamping; mismatch refusal |
+| 7 | admission gate | core/admission.py | consume-only manifest; paired scoring; GO/NO-GO |
+| 8 | no fitting on labels | core/probes.py + admission | reversed-pair probes; holdout; no pool-label tuning |
+| 9 | bounded models | core/verdict.py contract | zero-shot judge over trace; BLOCK only via proof |
+| 10 | fact packs | core/factpack.py + admission family tag | reviewed facts, provenance, family-scope in reports |
+| 11 | consumer contract | core/__init__.py | push-consumer and proof-consumer interfaces |
+| 12 | measurement schema | core/admission.py report schema | digests, vocabulary, rates+n+Wilson, coverage, instrument, runtime |
+
+## Components and Interfaces
+
+| Component | Intent | Req | Key contracts |
+|---|---|---|---|
+| core/trace.py | canonical trace + judgment | 1, 2 | `TraceRecord.record() -> canonical JSON` |
+| core/verdict.py | verdict contract | 5, 9 | `Verdict = ProvenClaim(witness, location) | BoundedSilence(reason)` |
+| core/coverage.py | coverage accounting | 3 | `Coverage{entry_recognition, resolution_buckets, unchecked_reasons}` |
+| core/budget.py | measured budget | 4 | `Complexity{regions, calls, changed_bytes}`; `derive_budget(samples, percentile, hardware) -> Budget` |
+| core/provenance.py | digests | 6 | `stamp(bundle, population, change) -> digests`; mismatch raises |
+| core/admission.py | admission gate | 7, 10, 12 | `validate_manifest` (consume-only), `admission_report(...) -> GO/NO-GO` |
+| core/probes.py | anti-shortcut probes | 8 | `reversed_pair_probe(component, fix, reversed_fix)` |
+| core/factpack.py | family packs | 10 | `admit_fact_pack(facts, review) -> stamped pack` |
+
+## Data Models
+
+- **TraceRecord**: entry identity; hops (each with call-target completeness); operation
+  (kind, resource, authority); witnesses (provenance + match relations); guards (effects);
+  completeness reasons; judgment; digests. Canonical JSON, sorted keys (existing convention).
+- **Verdict**: `ProvenClaim{trace_ref, witness, path, line, family}` or
+  `BoundedSilence{path, reason, coverage}`. Never a bare "clean".
+- **Coverage**: entry points + frameworks recognized; per-call exactly one of
+  complete/incomplete/unresolved; unchecked operations with named reasons.
+- **BudgetSample**: change complexity + measured seconds; **Budget**: per-class percentile
+  derivation with n and hardware disclosed; configurable, never a single constant.
+
+## Error Handling
+- Zero-byte or unreadable input → instrument failure; the run is invalid, never clean (Req 3).
+- Digest mismatch → refuse the result (Req 6).
+- Manifest unreviewed / label-exposing / acquisition-changed / spent → ordered refusal, no fallback (Req 7).
+- Budget exceeded → gate failure naming change, elapsed time, and the derivation exceeded (Req 4).
+- Indeterminate evidence → indeterminate verdict with reasons; never converted (Req 2).
+
+## Testing Strategy
+- Unit: trace serialization determinism; verdict rules; coverage bucket exclusivity; budget
+  derivation (fixed samples → fixed budget); digest refusal; manifest refusal order.
+- Contract conformance: prover's existing Trace satisfies core/trace.py; admission suite
+  output byte-identical before/after adoption (regression gate).
+- Probes: reversed-pair probe catches a shape-only component (deletion-count control).
+- Measurement: admission suite re-run on the pilot pool must reproduce the 2026-10-30
+  NO-GO numbers exactly (instrument check).
+
+## Performance & Scalability
+- Budget is measured (Req 4): fit the complexity→reply-time relation on the dev (483) and
+  pilot (264) record sets as measurement only; publish per-class budgets (n, percentile,
+  hardware). The static 120s remains as a provisional default until the first derivation
+  is published; changing it starts a new baseline (Req 12).
+
+## Security Considerations
+- Label isolation: the core never reads pool/admission labels except through the validated
+  admission path; forbidden markers enforced at manifest validation (Req 7, 8).
+- Repository identity masked in shared summaries (existing convention preserved).
+
+## Migration Strategy
+1. Create `core/` contract package mirroring existing shapes (no behavior change).
+2. Prover re-exports core types; conformance tests prove equivalence.
+3. Runners adopt coverage/budget/provenance one at a time; admission suite re-run after each.
+4. Admission modules become thin consumers of core/admission.py; 2026-10-30 NO-GO reproduced.
+5. Fact-pack contract adopted by ruleset loader; family-scope appears in reports.