mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-11-13 01:48:17 +00:00
feat(detection-core): foundation contract package (trace, verdict, coverage, provenance)
This commit is contained in:
@@ -0,0 +1,95 @@
|
||||
# Brief: detection-core
|
||||
|
||||
## Problem
|
||||
|
||||
Detection in this project is specified as a set of per-family, per-arm runners (prover,
|
||||
hunter, hook, proof arm). Corrections patch one arm and do not compose: v1→v2 retunes
|
||||
transferred 0/17 on untouched cases, and the 2026-10-30 admission run returned NO-GO
|
||||
(0/12 pairs, in-family 0/3, 9/12 pilot repositories with zero framework recognition).
|
||||
Learned components have nothing constraining them, so DSPy fitted a shape shortcut
|
||||
(deletion count separates reversed fixes at 0.90). The root cause: the core security
|
||||
object — one deterministic, checkable artifact per change that makes a defect claim
|
||||
provable (with witness) and a non-claim bounded (with coverage) — was implicit, never
|
||||
specified.
|
||||
|
||||
## Current State
|
||||
|
||||
- The ordered trace exists for one family: `access-control-prover` implements entry →
|
||||
hops → operation → witnesses → dominance → three-way judgment, with digests and the
|
||||
Req 12 admission workflow (dev pool + untouched admission pool). 13.3's four-arm run
|
||||
proved attribution works around the trace (analyzer_only −246 unguarded).
|
||||
- Measured ceilings of the chain: entry recognition (9/12 pilot repos unrecognized),
|
||||
call resolution (11.9% complete), guard dominance coarseness, 120s pre-push budget
|
||||
(whole-repo CPGs do not fit: libpng 446s, 637-file PMPro does not load).
|
||||
- `valid-detection-benchmark` provides the measurement instrument; `detection-quality.md`
|
||||
(steering, 2026-10-30) holds standing rules and pre-registered stop conditions.
|
||||
|
||||
## Desired Outcome
|
||||
|
||||
One specified core object — the security trace — with a contract that every detection arm,
|
||||
fact pack, and consumer reuses: given a push (base, head) within a fixed budget, output
|
||||
proven defect claims (each with its witness) plus explicit coverage of what was not
|
||||
decided, on repositories never seen. No fitted component ever sees pool labels.
|
||||
|
||||
## Approach
|
||||
|
||||
Specify the trace as the core: scope (change regions, hard budget) → recognize (counted
|
||||
entry/operation coverage) → resolve (every call in one completeness bucket) → trace
|
||||
(entry → hops → operation → witnesses → dominance → judgment → digests) → verdict (proven
|
||||
claim | bounded silence). Models are bounded to residuals: zero-shot judge on the trace,
|
||||
hunter proposes trust boundaries, proof arm demonstrates (only BLOCK evidence). Admission
|
||||
on an untouched population gates the trace itself.
|
||||
|
||||
## Scope
|
||||
|
||||
- **In**: trace schema and witness algebra; three-way judgment; budget/scoping contract;
|
||||
coverage accounting (funnel, resolution buckets, named incompleteness reasons); digest
|
||||
and provenance fields; admission-gate contract (Req 12 workflow generalized); the
|
||||
anti-shortcut rule for any fitted component (reversed-pair probes, per-repo holdout,
|
||||
untouched-population admission); re-parenting map of existing specs onto the core.
|
||||
- **Out**: new vulnerability families; per-framework fact content (that is fact-pack work,
|
||||
e.g. recognition-coverage table task 7.1 in valid-detection-benchmark); the pre-push
|
||||
transaction itself (pre-push-safety-net consumes the core); engine internals.
|
||||
|
||||
## Boundary Candidates
|
||||
|
||||
- Core trace object and verdict contract (this spec).
|
||||
- Fact packs: per-family recognition/operation/guard facts, admission-gated (extension
|
||||
point, starting with access_control from access-control-prover).
|
||||
- Consumers: pre-push-safety-net (push transaction), search-with-proof (BLOCK evidence),
|
||||
hunter-learning-loop (proposal lane only).
|
||||
|
||||
## Out of Boundary
|
||||
|
||||
- The pre-push hook transaction and rollout policy.
|
||||
- New detection families beyond what the trace contract enables.
|
||||
- Fitted models of any kind in the core path.
|
||||
- Engine/CPG internals (contributor-scan owns those).
|
||||
|
||||
## Upstream / Downstream
|
||||
|
||||
- **Upstream**: `valid-detection-benchmark` (measurement instrument, dev + admission
|
||||
pools), `access-control-prover` (trace implemented for access_control), steering
|
||||
`detection-quality.md` (standing rules and stop conditions).
|
||||
- **Downstream**: `pre-push-safety-net` (M-series milestones), `search-with-proof`,
|
||||
`hunter-learning-loop`, future fact-pack specs per family/ecosystem.
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
|
||||
- **Extends**: `access-control-prover` — its trace schema, judgment vocabulary and
|
||||
admission workflow become the core contract, generalized across families.
|
||||
- **Adjacent**: `pre-push-safety-net`, `search-with-proof`, `hunter-learning-loop`,
|
||||
`valid-detection-benchmark`, `contributor-scan` — consumers/instruments; must not
|
||||
duplicate the trace contract.
|
||||
|
||||
## Constraints
|
||||
|
||||
- Pool measures, never trains: no fitted prompt/threshold/model on pool labels
|
||||
(detection-quality.md rules 4-5, 13-proposal).
|
||||
- Fixed measurement schema: population + freeze digests, judgment vocabulary, rates with
|
||||
n and Wilson 95%, coverage, instrument failures, runtime.
|
||||
- Budget is measured, not static: per-change reply time and complexity are recorded, and
|
||||
the budget is derived from the measured distribution per workload class (sample, percentile,
|
||||
hardware disclosed); scoping to the change is part of the core contract.
|
||||
- Instrument rule: prove nonzero input reads before interpreting any zero.
|
||||
- Plain English in all documents; standard metrics only.
|
||||
@@ -0,0 +1,228 @@
|
||||
# Design Document: detection-core
|
||||
|
||||
## Overview
|
||||
|
||||
**Purpose:** detection-core specifies the single security object every detection arm reuses:
|
||||
the canonical trace with its verdict, coverage, budget, provenance, and admission contracts.
|
||||
It extracts what `access-control-prover` already implements for one family into a
|
||||
family-agnostic contract, so arms, fact packs, and consumers stop maintaining parallel
|
||||
runner contracts.
|
||||
|
||||
**Users:** spec authors and implementers of detection arms (prover, hunter, push hook,
|
||||
proof arm) consume the contract; the maintainer reads admission GO/NO-GO over it.
|
||||
|
||||
**Impact:** replaces per-arm envelopes with one contract; existing behavior stays
|
||||
byte-identical per consumer during migration (the 2026-10-30 admission suite is the
|
||||
regression gate).
|
||||
|
||||
### Goals
|
||||
- One canonical trace schema and three-way verdict vocabulary shared by all arms (Req 1, 2, 5).
|
||||
- Coverage accounting that makes every zero explainable (Req 3); instrument rule enforced at core level.
|
||||
- A measured, per-workload-class time budget instead of a static constant (Req 4).
|
||||
- Provenance digests on every analysis, with refusal on mismatch (Req 6).
|
||||
- A generalized admission gate over untouched populations (Req 7), no fitting on labels with
|
||||
reversed-pair probes (Req 8), models bounded to residuals (Req 9), families as fact packs
|
||||
with scope disclosure (Req 10), consumer contracts (Req 11), fixed measurement schema (Req 12).
|
||||
|
||||
### Non-Goals
|
||||
- New vulnerability families or their fact content.
|
||||
- The pre-push transaction itself (pre-push-safety-net consumes this core).
|
||||
- Fitted models anywhere in the core path.
|
||||
- Engine/graph internals (contributor-scan).
|
||||
|
||||
## Boundary Commitments
|
||||
|
||||
### This Spec Owns
|
||||
- The canonical trace schema (entry, hops, operation, witnesses, guards, judgment, completeness, digests) and its deterministic serialization.
|
||||
- The verdict contract: proven claim (witness attached) vs bounded silence (coverage reason attached); indeterminate never converts to clean.
|
||||
- Coverage accounting schema (entry recognition, per-call resolution buckets, unchecked reasons) and the zero-byte instrument refusal.
|
||||
- The measured-budget contract: complexity features recorded per change, per-workload-class budget derivation published with sample/percentile/hardware, configurable budget.
|
||||
- Provenance stamping (analyzer/fact/acquisition/configuration/population/input digests) and refusal on digest mismatch.
|
||||
- The generalized admission contract: consume-only manifest validation, paired scoring, gates, GO/NO-GO report schema, family-scope disclosure.
|
||||
- The fact-pack admission contract (reviewed facts, provenance stamping, family tag).
|
||||
- The consumer interface for the pre-push transaction and the proof layer.
|
||||
|
||||
### Out of Boundary
|
||||
- Pre-push hook transaction, rollout policy, capability registry (pre-push-safety-net).
|
||||
- Family fact content (fact-pack specs, starting from access-control-prover's rulesets).
|
||||
- Hunter/proposer internals and proof-arm internals (they consume the contract).
|
||||
- Engine/CPG internals (contributor-scan).
|
||||
|
||||
### Allowed Dependencies
|
||||
- `valid-detection-benchmark` pools and freeze artifacts (read-only; never label-tuned).
|
||||
- Existing prover implementation as the first contract implementation.
|
||||
- Steering rules in `detection-quality.md` (no fitting on pool labels; instrument rule; stop conditions).
|
||||
|
||||
### Revalidation Triggers
|
||||
- Trace schema or verdict vocabulary changes → every arm and consumer re-validates.
|
||||
- Admission report schema changes → every admission gate re-runs.
|
||||
- Budget derivation method changes → new baseline under Req 12 (not a trend).
|
||||
- Fact-pack admission contract changes → all fact packs re-reviewed.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Existing Architecture Analysis
|
||||
Today each arm owns its envelope: `prover_arm.py` (records + summarize), `development_runner.py`
|
||||
(bundle + funnel + gates), `arm_regression.py` (arm records), `admission_run.py` (manifest +
|
||||
paired scoring). The trace itself (`Trace`, `ProverResult`) exists in the prover. The gap is
|
||||
the missing shared contract layer, not missing analysis code.
|
||||
|
||||
### Architecture Pattern & Boundary Map
|
||||
|
||||
Pattern: **thin core contract package; existing implementations adopt it incrementally**
|
||||
(hybrid option C from research.md). The core owns contracts only — no analysis logic.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph CORE[detection-core contract package]
|
||||
T[trace schema + verdict]
|
||||
C[coverage accounting]
|
||||
B[measured budget]
|
||||
P[provenance digests]
|
||||
A[admission contract]
|
||||
F[fact-pack contract]
|
||||
end
|
||||
subgraph ARMS[arms / fact packs]
|
||||
PR[access-control-prover arm]
|
||||
FP[family fact packs]
|
||||
HU[hunter proposal lane]
|
||||
end
|
||||
subgraph CONSUMERS[consumers]
|
||||
PP[pre-push-safety-net]
|
||||
PR2[search-with-proof]
|
||||
BM[valid-detection-benchmark]
|
||||
end
|
||||
ARMS --> CORE
|
||||
CORE --> CONSUMERS
|
||||
BM -. measures .-> CORE
|
||||
```
|
||||
|
||||
### Technology Stack
|
||||
|
||||
| Layer | Choice | Role | Notes |
|
||||
|---|---|---|---|
|
||||
| Core contract | Python, frozen dataclasses + canonical JSON | trace/verdict/coverage/budget/admission schemas | mirrors existing Trace.record() conventions |
|
||||
| Runners | existing benchmark runners | adopt core contracts incrementally | behavior byte-identical per consumer |
|
||||
| Tests | pytest | contract conformance per consumer | admission suite is the regression gate |
|
||||
|
||||
## File Structure Plan
|
||||
|
||||
### Directory Structure
|
||||
```
|
||||
src/openultrasast/core/ # NEW: the contract package (no analysis logic)
|
||||
├── __init__.py # public contract exports
|
||||
├── trace.py # canonical trace schema, deterministic serialization (Req 1, 2)
|
||||
├── verdict.py # proven-claim vs bounded-silence contract (Req 5)
|
||||
├── coverage.py # entry/resolution/unchecked accounting + zero-byte refusal (Req 3)
|
||||
├── budget.py # measured budget: complexity features, per-class derivation (Req 4)
|
||||
├── provenance.py # digest stamping + mismatch refusal (Req 6)
|
||||
├── admission.py # manifest validation, paired scoring, gates, GO/NO-GO schema (Req 7, 10)
|
||||
├── probes.py # reversed-pair / holdout probes for fitted components (Req 8)
|
||||
└── factpack.py # family fact-pack admission contract (Req 10)
|
||||
tests/test_core_trace.py # conformance tests per contract module
|
||||
tests/test_core_admission.py
|
||||
tests/test_core_budget.py
|
||||
```
|
||||
|
||||
### Modified Files
|
||||
- `src/openultrasast/hunt/prover.py`, `semantic/obligations/trace.py` — implement/re-export
|
||||
the core trace contract for access_control (Req 1, 2, 5); behavior unchanged.
|
||||
- `benchmarks/valid/prover_admission.py`, `admission_scoring.py`, `admission_run.py` — become
|
||||
thin consumers of `core/admission.py` (Req 7); existing refusal order and report schema preserved.
|
||||
- `benchmarks/valid/development_runner.py`, `arm_regression.py`, `prover_arm.py` — adopt
|
||||
`core/coverage.py`, `core/budget.py`, `core/provenance.py` (Req 3, 4, 6, 11, 12).
|
||||
- `benchmarks/valid/run_manifests.py` — bundle digests feed `core/provenance.py` (Req 6).
|
||||
|
||||
## System Flows
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant PP as pre-push transaction
|
||||
participant CORE as detection-core
|
||||
participant ARM as arm (prover/fact pack)
|
||||
participant BM as benchmark pools
|
||||
PP->>CORE: push (base, head), budget class
|
||||
CORE->>CORE: scope change regions (Req 4)
|
||||
CORE->>ARM: analyze scoped regions
|
||||
ARM-->>CORE: trace + coverage + complexity + seconds
|
||||
CORE->>CORE: stamp digests, derive budget check (Req 4, 6)
|
||||
CORE-->>PP: verdict (proven claims with witnesses | bounded silence)
|
||||
BM-->>CORE: dev pool + untouched admission pool
|
||||
CORE->>CORE: admission gate: manifest, pairs, criteria -> GO/NO-GO (Req 7)
|
||||
```
|
||||
|
||||
## Requirements Traceability
|
||||
|
||||
| Req | Summary | Components | Contracts |
|
||||
|---|---|---|---|
|
||||
| 1 | canonical trace | core/trace.py | Trace schema, deterministic record() |
|
||||
| 2 | three-way judgment | core/trace.py, core/verdict.py | Judgment enum; indeterminate-never-clean rule |
|
||||
| 3 | coverage accounting | core/coverage.py | funnel + resolution buckets + zero-byte refusal |
|
||||
| 4 | measured budget | core/budget.py | complexity features; per-class derivation; gate names derivation |
|
||||
| 5 | verdict contract | core/verdict.py | proven claim w/ witness; bounded silence w/ reason |
|
||||
| 6 | provenance | core/provenance.py | digest stamping; mismatch refusal |
|
||||
| 7 | admission gate | core/admission.py | consume-only manifest; paired scoring; GO/NO-GO |
|
||||
| 8 | no fitting on labels | core/probes.py + admission | reversed-pair probes; holdout; no pool-label tuning |
|
||||
| 9 | bounded models | core/verdict.py contract | zero-shot judge over trace; BLOCK only via proof |
|
||||
| 10 | fact packs | core/factpack.py + admission family tag | reviewed facts, provenance, family-scope in reports |
|
||||
| 11 | consumer contract | core/__init__.py | push-consumer and proof-consumer interfaces |
|
||||
| 12 | measurement schema | core/admission.py report schema | digests, vocabulary, rates+n+Wilson, coverage, instrument, runtime |
|
||||
|
||||
## Components and Interfaces
|
||||
|
||||
| Component | Intent | Req | Key contracts |
|
||||
|---|---|---|---|
|
||||
| core/trace.py | canonical trace + judgment | 1, 2 | `TraceRecord.record() -> canonical JSON` |
|
||||
| core/verdict.py | verdict contract | 5, 9 | `Verdict = ProvenClaim(witness, location) | BoundedSilence(reason)` |
|
||||
| core/coverage.py | coverage accounting | 3 | `Coverage{entry_recognition, resolution_buckets, unchecked_reasons}` |
|
||||
| core/budget.py | measured budget | 4 | `Complexity{regions, calls, changed_bytes}`; `derive_budget(samples, percentile, hardware) -> Budget` |
|
||||
| core/provenance.py | digests | 6 | `stamp(bundle, population, change) -> digests`; mismatch raises |
|
||||
| core/admission.py | admission gate | 7, 10, 12 | `validate_manifest` (consume-only), `admission_report(...) -> GO/NO-GO` |
|
||||
| core/probes.py | anti-shortcut probes | 8 | `reversed_pair_probe(component, fix, reversed_fix)` |
|
||||
| core/factpack.py | family packs | 10 | `admit_fact_pack(facts, review) -> stamped pack` |
|
||||
|
||||
## Data Models
|
||||
|
||||
- **TraceRecord**: entry identity; hops (each with call-target completeness); operation
|
||||
(kind, resource, authority); witnesses (provenance + match relations); guards (effects);
|
||||
completeness reasons; judgment; digests. Canonical JSON, sorted keys (existing convention).
|
||||
- **Verdict**: `ProvenClaim{trace_ref, witness, path, line, family}` or
|
||||
`BoundedSilence{path, reason, coverage}`. Never a bare "clean".
|
||||
- **Coverage**: entry points + frameworks recognized; per-call exactly one of
|
||||
complete/incomplete/unresolved; unchecked operations with named reasons.
|
||||
- **BudgetSample**: change complexity + measured seconds; **Budget**: per-class percentile
|
||||
derivation with n and hardware disclosed; configurable, never a single constant.
|
||||
|
||||
## Error Handling
|
||||
- Zero-byte or unreadable input → instrument failure; the run is invalid, never clean (Req 3).
|
||||
- Digest mismatch → refuse the result (Req 6).
|
||||
- Manifest unreviewed / label-exposing / acquisition-changed / spent → ordered refusal, no fallback (Req 7).
|
||||
- Budget exceeded → gate failure naming change, elapsed time, and the derivation exceeded (Req 4).
|
||||
- Indeterminate evidence → indeterminate verdict with reasons; never converted (Req 2).
|
||||
|
||||
## Testing Strategy
|
||||
- Unit: trace serialization determinism; verdict rules; coverage bucket exclusivity; budget
|
||||
derivation (fixed samples → fixed budget); digest refusal; manifest refusal order.
|
||||
- Contract conformance: prover's existing Trace satisfies core/trace.py; admission suite
|
||||
output byte-identical before/after adoption (regression gate).
|
||||
- Probes: reversed-pair probe catches a shape-only component (deletion-count control).
|
||||
- Measurement: admission suite re-run on the pilot pool must reproduce the 2026-10-30
|
||||
NO-GO numbers exactly (instrument check).
|
||||
|
||||
## Performance & Scalability
|
||||
- Budget is measured (Req 4): fit the complexity→reply-time relation on the dev (483) and
|
||||
pilot (264) record sets as measurement only; publish per-class budgets (n, percentile,
|
||||
hardware). The static 120s remains as a provisional default until the first derivation
|
||||
is published; changing it starts a new baseline (Req 12).
|
||||
|
||||
## Security Considerations
|
||||
- Label isolation: the core never reads pool/admission labels except through the validated
|
||||
admission path; forbidden markers enforced at manifest validation (Req 7, 8).
|
||||
- Repository identity masked in shared summaries (existing convention preserved).
|
||||
|
||||
## Migration Strategy
|
||||
1. Create `core/` contract package mirroring existing shapes (no behavior change).
|
||||
2. Prover re-exports core types; conformance tests prove equivalence.
|
||||
3. Runners adopt coverage/budget/provenance one at a time; admission suite re-run after each.
|
||||
4. Admission modules become thin consumers of core/admission.py; 2026-10-30 NO-GO reproduced.
|
||||
5. Fact-pack contract adopted by ruleset loader; family-scope appears in reports.
|
||||
@@ -0,0 +1,140 @@
|
||||
# Requirements Document
|
||||
|
||||
## Project Description (Input)
|
||||
|
||||
**Who has the problem:** the maintainer of OpenUltraSAST. Every detection arm built so far
|
||||
(prover, hunter, push hook, proof arm) is corrected link by link, and corrections do not
|
||||
compose — v1→v2 retunes transferred 0/17 on untouched cases, and the 2026-10-30 admission
|
||||
run on the untouched pilot population returned NO-GO (0/12 pairs, in-family 0/3, 9/12
|
||||
repositories with zero framework recognition).
|
||||
|
||||
**Current situation:** detection is specified as per-family, per-arm runners around an
|
||||
implicit core. The ordered trace exists for one family (access-control-prover), learned
|
||||
components fit pool shortcuts (deletion count separates reversed fixes at 0.90), and
|
||||
misses concentrate in unmeasured links: entry recognition, call resolution (11.9%
|
||||
complete), guard dominance, and the 120s pre-push budget.
|
||||
|
||||
**What should change:** one specified core object — the security trace — with a contract
|
||||
every arm, fact pack, and consumer reuses: given a push (base, head) within a fixed budget,
|
||||
output proven defect claims (each with its witness) plus explicit coverage of what was not
|
||||
decided, on repositories never seen. Models are bounded to residuals; admission on an
|
||||
untouched population gates the trace itself; no fitted component ever sees pool labels.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement 1: Canonical trace object
|
||||
The detection core shall define one canonical trace object per analyzed change as the
|
||||
shared security object for all detection arms.
|
||||
1. When a change is analyzed, the detection core shall produce a canonical trace carrying
|
||||
entry identity, ordered hops with call-target completeness, operation kind with resource
|
||||
and authority, witnesses with provenance and match relations, guard effects, completeness
|
||||
reasons, a final judgment, and the input digests of the analysis.
|
||||
2. The detection core shall serialize every trace deterministically, so that identical
|
||||
inputs produce byte-identical traces.
|
||||
|
||||
### Requirement 2: Three-way judgment vocabulary
|
||||
The detection core shall judge every analyzed operation as exactly one of unguarded,
|
||||
discharged, or indeterminate.
|
||||
1. The detection core shall never convert an indeterminate judgment into a clean
|
||||
(discharged or unguarded-free) result.
|
||||
2. When evidence for a judgment is incomplete, the detection core shall return
|
||||
indeterminate with the named incompleteness reasons.
|
||||
|
||||
### Requirement 3: Coverage accounting
|
||||
The detection core shall account for every input it did or did not use, so a zero result is
|
||||
never indistinguishable from an unread input.
|
||||
1. When a change is analyzed, the detection core shall report entry-recognition coverage,
|
||||
per-call resolution outcomes with exactly one outcome bucket per call, and named
|
||||
reasons for every unchecked operation.
|
||||
2. If any input file is read as zero bytes, the detection core shall fail the run as an
|
||||
instrument failure rather than reporting a clean result.
|
||||
|
||||
### Requirement 4: Measured budget and change scoping
|
||||
The detection core shall derive its time budget from measurement, not from a fixed constant.
|
||||
1. When a change is analyzed, the detection core shall measure and record reply time and
|
||||
change complexity (changed regions, call count, changed size) for every change.
|
||||
2. The detection core shall derive the per-change budget from the measured distribution
|
||||
(sample count, percentile, reference workload and hardware disclosed) and shall publish
|
||||
that derivation under the fixed measurement schema; the budget shall be configurable per
|
||||
workload class, never a single static constant.
|
||||
3. When a change exceeds the measured budget, the detection core shall fail the budget gate
|
||||
naming the change, the elapsed time, and the budget derivation exceeded.
|
||||
4. While a push is analyzed, the detection core shall limit analysis to regions affected by
|
||||
the change and shall record the scope it used.
|
||||
|
||||
### Requirement 5: Verdict contract
|
||||
The detection core shall distinguish proven defect claims from bounded silence.
|
||||
1. When the detection core reports a defect, the claim shall carry its witness evidence and
|
||||
an exact location tied to the change.
|
||||
2. When the detection core cannot decide a site, it shall report that site as undecided
|
||||
with its coverage reason, and shall never present an undecided site as a clean result.
|
||||
|
||||
### Requirement 6: Provenance and digests
|
||||
The detection core shall stamp every trace and verdict with the identities of its inputs.
|
||||
1. The detection core shall record, for every analysis, the analyzer revision, fact-set
|
||||
digest, acquisition-record digest, configuration digest, population digest, and
|
||||
per-change input digest.
|
||||
2. If any recorded digest cannot be re-derived from the named inputs, the detection core
|
||||
shall refuse the result rather than accept it.
|
||||
|
||||
### Requirement 7: Admission gate contract
|
||||
The detection core shall gate any generalization claim on an untouched population.
|
||||
1. When an arm is proposed for pre-push use, the detection core shall require a reviewed
|
||||
manifest naming the frozen analyzer, facts, acquisition, configuration, and an untouched
|
||||
admission population before any admission label is opened.
|
||||
2. The detection core shall consume admission manifests without producing, mutating, or
|
||||
re-labeling them, and shall refuse manifests that are unreviewed, label-exposing,
|
||||
acquisition-changed, or spent.
|
||||
3. When admission evidence is scored, the detection core shall report paired
|
||||
vulnerable/fixed behavior, ordinary false alarms, and every gate (budget, instrument,
|
||||
pre-registered criteria) as an evidence-backed GO or NO-GO.
|
||||
|
||||
### Requirement 8: No fitting on evaluation labels
|
||||
The detection core shall not allow any fitted component to train on evaluation labels.
|
||||
1. The detection core shall prohibit tuned prompts, thresholds, or fitted models whose
|
||||
training data includes pool or admission labels.
|
||||
2. When a fitted component is proposed, the detection core shall require it to pass
|
||||
reversed-pair probes (a fix and its reversed fix shall not be separable by the
|
||||
component's decision features), per-repository holdout, and admission on an untouched
|
||||
population before any claim is made.
|
||||
|
||||
### Requirement 9: Bounded role for models
|
||||
The detection core shall bound every model to residual questions over the trace.
|
||||
1. Where a model judges a trace, the detection core shall present the deterministic trace
|
||||
and require the judgment to be zero-shot over that trace.
|
||||
2. The detection core shall not allow a model to propose-and-adjudicate the same claim, and
|
||||
shall reserve blocking verdicts for demonstrated proofs with a differential witness.
|
||||
|
||||
### Requirement 10: Family fact packs as the extension point
|
||||
The detection core shall admit new vulnerability families only as fact packs over the core
|
||||
trace contract.
|
||||
1. When a new family is added, the detection core shall require the family's entry,
|
||||
operation, and guard knowledge as reviewed fact packs stamped into the trace's
|
||||
provenance, without changing the trace schema or judgment vocabulary.
|
||||
2. The detection core shall report, per arm, the vulnerability families the arm claims, so
|
||||
admission and rollout numbers are never read across families the arm does not cover.
|
||||
|
||||
### Requirement 11: Consumer and re-parenting contract
|
||||
The detection core shall be the single contract that existing detection specs reuse.
|
||||
1. The detection core shall define the interface through which the pre-push transaction
|
||||
consumes traces and verdicts within its deadline.
|
||||
2. The detection core shall define the interface through which proof-based blocking evidence
|
||||
attaches to a trace verdict.
|
||||
3. When an existing arm's behavior is described, the detection core shall require that
|
||||
description to be expressed over the core contract rather than as a separate runner
|
||||
contract.
|
||||
|
||||
### Requirement 12: Fixed measurement schema
|
||||
The detection core shall require one fixed measurement schema for every detection number.
|
||||
1. When any detection result is reported, the detection core shall require population and
|
||||
freeze digests, judgment vocabulary, every rate with its sample count and uncertainty
|
||||
interval, coverage, instrument failures, and runtime.
|
||||
2. When a measurement changes population or judgment vocabulary, the detection core shall
|
||||
treat the result as a new baseline and shall not present it as a trend against the
|
||||
previous baseline.
|
||||
|
||||
## Out of Scope
|
||||
- The pre-push hook transaction, rollout policy, and capability registry (pre-push-safety-net).
|
||||
- New vulnerability families and their fact content (fact-pack specs).
|
||||
- Fitted models of any kind in the core path.
|
||||
- Engine and graph internals (contributor-scan).
|
||||
@@ -0,0 +1,53 @@
|
||||
# Research Log: detection-core
|
||||
|
||||
## Gap analysis (2026-10-30)
|
||||
|
||||
### Current-state inventory (from codebase exploration)
|
||||
|
||||
The security trace already exists for one family (access_control) in the prover:
|
||||
`Trace`/`PathRecord`/`ProverResult` (`src/openultrasast/hunt/prover.py`,
|
||||
`semantic/obligations/trace.py`), three-way judgment (`PairJudgment`,
|
||||
`semantic/obligations/judgment.py`), coverage counters (`Coverage`, `stage_funnel`,
|
||||
resolution buckets in `semantic/obligations/resolution.py`), digest stamping
|
||||
(`benchmarks/valid/run_manifests.py` `AnalyzerBundle`, `_input_digest`/`_population_digest`
|
||||
in `arm_regression.py`), and the admission workflow (`benchmarks/valid/prover_admission.py`,
|
||||
`admission_scoring.py`, `admission_run.py`).
|
||||
|
||||
### Requirement-to-asset map
|
||||
|
||||
| Req | Existing asset | Status |
|
||||
|---|---|---|
|
||||
| 1 trace object | `Trace`/`ProverResult` in prover | Exists, access_control-only; needs family-agnostic core contract |
|
||||
| 2 three-way judgment | `PairJudgment`, `judgment.py` | Exists; promote to core vocabulary |
|
||||
| 3 coverage accounting | `Coverage`, `stage_funnel`, resolution buckets | Exists; zero-byte refusal lives only in runners |
|
||||
| 4 measured budget | static `BUDGET_SECONDS = 120.0` in runners; per-change `seconds` recorded | **Gap**: budget is a constant; complexity features not recorded; no per-class derivation. Evidence: dev pool 20/483 overruns (max 240s) vs pilot max 7.7s |
|
||||
| 5 verdict contract | findings carry witness; indeterminate handling in prover | Exists; needs core-level contract (proven vs bounded silence) |
|
||||
| 6 digests/provenance | `AnalyzerBundle`, input/population digests | Exists; stamping rule must become core-level |
|
||||
| 7 admission gate | `prover_admission.py`/`admission_scoring.py`/`admission_run.py` | Exists for one arm; needs family-agnostic generalization |
|
||||
| 8 no fitting on labels | steering rule only (detection-quality.md) | **Gap**: no reversed-pair probe harness exists |
|
||||
| 9 bounded model role | zero-shot judge measured 2026-10-28 (nearly inert, safe) | Policy; core must state the boundary contract |
|
||||
| 10 fact packs | `ruleset/obligations/*.toml`, `fact_init.py` | Exists per language; needs family-pack admission + scope tagging |
|
||||
| 11 consumer contract | each runner has its own envelope (`prover_arm.run/summarize`, push.admission.defects) | **Gap**: no shared core interface |
|
||||
| 12 measurement schema | `score.rate`, `summarize_*` | Partial; vocabulary/digest fields not unified |
|
||||
|
||||
### Implementation approach options
|
||||
|
||||
- **A. Extend in place**: grow the prover modules into the core. Minimal new files, but
|
||||
bloats family-specific code and keeps the "one arm = one contract" problem.
|
||||
- **B. New core module, prover reimplemented**: clean contract, but discards measured,
|
||||
reviewer-approved code (risk of regression in the 34.8%-recall strict lane).
|
||||
- **C. Hybrid (recommended)**: new thin core-contract package that the prover's existing
|
||||
types implement, and the runners consume incrementally. Contract extracted, behavior
|
||||
preserved; migration is per-consumer, each step reviewable.
|
||||
|
||||
### Effort / risk
|
||||
- Effort: M-L (contract extraction across ~6 runner modules plus core package).
|
||||
- Risk: Medium — contract extraction touches measured code paths; mitigated by keeping
|
||||
existing behavior byte-identical per consumer during migration and re-running the
|
||||
admission suite as the regression gate.
|
||||
|
||||
### Research needed (carried to design)
|
||||
- Measured-budget model: which complexity features predict reply time on the two pools
|
||||
(dev 483 changes with seconds; pilot 264). Carry the 13.3/13.4 record sets as the fitting
|
||||
data — measurement only, never training a detector on them.
|
||||
- Reversed-pair probe harness for Req 8: fix/reversed-fix pairs exist in the pools.
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"feature_name": "detection-core",
|
||||
"created_at": "2026-10-30T13:55:30Z",
|
||||
"updated_at": "2026-10-30T13:55:30Z",
|
||||
"language": "en",
|
||||
"phase": "tasks-generated",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"design": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"tasks": {
|
||||
"generated": true,
|
||||
"approved": false
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": false
|
||||
}
|
||||
@@ -0,0 +1,135 @@
|
||||
# Implementation Plan
|
||||
|
||||
_Requirement IDs use N.M where N is the requirement number in requirements.md and M is its acceptance criterion._
|
||||
|
||||
## Task constraints
|
||||
|
||||
- Complexity: McCabe at most 10 per touched function, at most 50 logical lines per function and 400 per new module.
|
||||
- The core contract package owns contracts only; no analysis logic moves into it.
|
||||
- Every migration step keeps existing behavior byte-identical; the 2026-10-30 admission suite is the regression gate.
|
||||
- Instrument rule: prove nonzero input reads before interpreting any zero.
|
||||
- Pool measures, never trains: no fitted component reads pool or admission labels.
|
||||
|
||||
- [ ] 1. Foundation: core contract package skeleton
|
||||
- [x] 1.1 (P) Canonical trace schema in the core package with deterministic serialization
|
||||
- Define the trace record (entry, hops with call-target completeness, operation, witnesses, guards, completeness reasons, judgment, digests) as a frozen, family-agnostic contract.
|
||||
- Done when a fixed trace serializes to byte-identical canonical output on repeat runs and a conformance test pins the schema.
|
||||
- _Requirements: 1.1, 1.2_
|
||||
- _Boundary: core trace contract_
|
||||
- [x] 1.2 (P) Three-way verdict vocabulary and verdict contract
|
||||
- Define unguarded/discharged/indeterminate, the proven-claim-with-witness vs bounded-silence verdict, and the rule that indeterminate never converts to clean.
|
||||
- Done when unit tests cover all three judgments, the conversion refusal, and zero-shot-judge-over-trace as the only model role on verdicts.
|
||||
- _Requirements: 2.1, 2.2, 5.1, 5.2, 9.1_
|
||||
- _Boundary: core verdict contract_
|
||||
- [x] 1.3 (P) Coverage accounting schema with instrument refusal
|
||||
- Define entry-recognition coverage, exactly-one-bucket-per-call resolution outcomes, named unchecked reasons, and zero-byte/unreadable input as an instrument failure at core level.
|
||||
- Done when a zero-byte read fails the run in a unit test and the bucket exclusivity is pinned.
|
||||
- _Requirements: 3.1, 3.2_
|
||||
- _Boundary: core coverage contract_
|
||||
- [x] 1.4 (P) Provenance digest stamping with mismatch refusal
|
||||
- Define the analyzer/fact/acquisition/configuration/population/input digest stamping and refusal when any digest cannot be re-derived.
|
||||
- Done when a mismatched digest is refused in a unit test.
|
||||
- _Requirements: 6.1, 6.2_
|
||||
- _Boundary: core provenance contract_
|
||||
|
||||
- [ ] 2. Measured budget replaces the static gate
|
||||
- [ ] 2.1 Record per-change complexity and reply time
|
||||
- Extend runner records with complexity features (changed regions, call count, changed bytes) alongside the already-recorded seconds.
|
||||
- Done when dev-pool records carry all complexity fields and seconds.
|
||||
- _Requirements: 4.1, 12.1_
|
||||
- _Boundary: runner records_
|
||||
- _Depends: 1.3_
|
||||
- [ ] 2.2 Derive per-workload-class budgets from measurement
|
||||
- Fit complexity to reply time on the dev (483-change) and pilot (264-change) record sets as measurement only; publish per-class budgets with n, percentile, and hardware; make the budget configurable per class, demoting the static constant to provisional default.
|
||||
- Done when the derivation table (class, n, percentile, hardware, budget) is published under the fixed measurement schema and a new-baseline note records the change.
|
||||
- _Requirements: 4.2, 12.1, 12.2_
|
||||
- _Boundary: core budget contract_
|
||||
- _Depends: 2.1_
|
||||
- [ ] 2.3 Budget gate and scoping record
|
||||
- Gate failures name the change, elapsed time, and the derivation exceeded; the analyzed scope (change regions) is recorded per run.
|
||||
- Done when an over-budget fixture fails with all three fields and the scope is present in the record.
|
||||
- _Requirements: 4.3, 4.4_
|
||||
- _Boundary: core budget contract_
|
||||
- _Depends: 2.2_
|
||||
|
||||
- [ ] 3. Prover arm adopts the core contract
|
||||
- [ ] 3.1 Prover trace implements the core schema
|
||||
- Re-export/implement the core trace contract in the access_control arm without behavior change.
|
||||
- Done when a conformance test proves the prover's trace satisfies the core schema field-for-field.
|
||||
- _Requirements: 1.1, 1.2, 2.1_
|
||||
- _Boundary: prover arm_
|
||||
- _Depends: 1.1, 1.2_
|
||||
- [ ] 3.2 Prover verdict and coverage over the core contract
|
||||
- Emit verdicts and coverage through the core contracts; existing strict-lane behavior stays byte-identical.
|
||||
- Done when the keyword regression and prover suites pass unchanged after adoption.
|
||||
- _Requirements: 3.1, 5.1, 5.2_
|
||||
- _Boundary: prover arm_
|
||||
- _Depends: 3.1, 1.3_
|
||||
|
||||
- [ ] 4. Generalized admission contract
|
||||
- [ ] 4.1 (P) Consume-only manifest validation generalized into the core
|
||||
- Move the ordered refusal chain (malformed, producer, unreviewed, label-isolated, label-exposure, acquisition-changed, spent) into the core admission contract; the benchmark validator becomes a thin wrapper.
|
||||
- Done when all refusal branches pass unit tests through the core contract.
|
||||
- _Requirements: 7.1, 7.2_
|
||||
- _Boundary: core admission contract_
|
||||
- _Depends: 1.4_
|
||||
- [ ] 4.2 Paired scoring and GO/NO-GO report schema with family scope
|
||||
- Generalize paired vulnerable/fixed scoring, ordinary false alarms, gates, and the report schema into the core; add the claimed-families field.
|
||||
- Done when the report schema includes family scope and every rate carries n and Wilson 95.
|
||||
- _Requirements: 7.3, 10.2, 12.1_
|
||||
- _Boundary: core admission contract_
|
||||
- _Depends: 1.2, 4.1_
|
||||
- [ ] 4.3 Admission suite reproduces the 2026-10-30 NO-GO
|
||||
- Re-run the pilot admission through the core contract and reproduce 0/12 pairs, 0.83 flagged/100, 0 instrument failures, NO-GO exactly.
|
||||
- Done when the reproduced report matches the committed 2026-10-30 report field-for-field.
|
||||
- _Requirements: 7.3, 12.1_
|
||||
- _Boundary: admission measurement_
|
||||
- _Depends: 4.2, 3.2_
|
||||
|
||||
- [ ] 5. Anti-shortcut probes for any fitted component
|
||||
- [ ] 5.1 (P) Reversed-pair probe harness
|
||||
- Build the fix/reversed-fix probe; the deletion-count shape feature must fail it (control test).
|
||||
- Done when the control probe fails a shape-only component and passes a relation-based one.
|
||||
- _Requirements: 8.1, 8.2_
|
||||
- _Boundary: core probes_
|
||||
- _Depends: 1.2_
|
||||
- [ ] 5.2 Fitted-component gate
|
||||
- Require per-repository holdout and untouched-population admission before any fitted component makes a claim; BLOCK reserved for demonstrated proofs.
|
||||
- Done when a fitted component without admission evidence is refused by the gate in a test.
|
||||
- _Requirements: 8.2, 9.2_
|
||||
- _Boundary: core probes + admission_
|
||||
- _Depends: 4.1, 5.1_
|
||||
|
||||
- [ ] 6. Fact packs and consumer interfaces
|
||||
- [ ] 6.1 Fact-pack admission contract
|
||||
- Ruleset loader admits family fact packs only with review evidence and provenance stamping; packs carry their family tag.
|
||||
- Done when an unreviewed pack is refused and a reviewed pack stamps provenance in a test.
|
||||
- _Requirements: 10.1_
|
||||
- _Boundary: fact-pack contract_
|
||||
- _Depends: 1.4_
|
||||
- [ ] 6.2 Consumer interfaces for push and proof
|
||||
- Define the interfaces through which the pre-push transaction consumes traces/verdicts within its deadline and the proof layer attaches demonstrated evidence to a verdict; existing arms express behavior over the core contract.
|
||||
- Done when both interfaces are exercised by conformance tests from a mock consumer.
|
||||
- _Requirements: 9.2, 11.1, 11.2, 11.3_
|
||||
- _Boundary: core consumer contracts_
|
||||
- _Depends: 1.2, 3.2_
|
||||
- [ ] 6.3 Runners adopt core coverage, budget, provenance incrementally
|
||||
- Migrate development_runner, arm_regression, prover_arm, run_manifests onto the core contracts one at a time, re-running the regression suite after each.
|
||||
- Done when all runners consume the core contracts and the keyword regression passes at the pre-migration count plus new tests.
|
||||
- _Requirements: 3.1, 4.1, 4.3, 6.1, 11.3, 12.1_
|
||||
- _Boundary: benchmark runners_
|
||||
- _Depends: 2.3, 3.2, 6.2_
|
||||
|
||||
- [ ] 7. Validation
|
||||
- [ ] 7.1 Fixed measurement schema enforced in reports
|
||||
- Every report carries population/freeze digests, judgment vocabulary, rates with n and Wilson 95, coverage, instrument failures, runtime; vocabulary/population change starts a new baseline.
|
||||
- Done when a schema-validation test rejects a report missing any field and accepts the 2026-10-30 report.
|
||||
- _Requirements: 12.1, 12.2_
|
||||
- _Boundary: measurement schema_
|
||||
- _Depends: 4.2, 6.3_
|
||||
- [ ] 7.2 End-to-end: one push through core to verdict to admission report
|
||||
- Run one dev-pool push end-to-end: scope, recognize, resolve, trace, verdict, digests, admission report on the dev pool.
|
||||
- Done when the E2E test produces a verdict with witness or bounded silence plus a complete report.
|
||||
- _Requirements: 5.1, 5.2, 7.3, 11.1_
|
||||
- _Boundary: end-to-end integration_
|
||||
- _Depends: 6.3, 7.1_
|
||||
@@ -0,0 +1,28 @@
|
||||
"""Detection core: the shared security-trace contract package.
|
||||
|
||||
Owns contracts only -- no analysis logic. Arms implement the trace contract;
|
||||
runners and consumers adopt coverage, budget, provenance, and admission contracts.
|
||||
"""
|
||||
|
||||
from openultrasast.core.coverage import Coverage, InstrumentFailure, ResolutionOutcome, check_reads
|
||||
from openultrasast.core.provenance import ProvenanceMismatch, stamp, verify_stamp
|
||||
from openultrasast.core.trace import Guard, Hop, TraceRecord, Witness
|
||||
from openultrasast.core.verdict import BoundedSilence, Judgment, ProvenClaim, as_clean
|
||||
|
||||
__all__ = [
|
||||
"BoundedSilence",
|
||||
"Coverage",
|
||||
"Guard",
|
||||
"Hop",
|
||||
"InstrumentFailure",
|
||||
"Judgment",
|
||||
"ProvenClaim",
|
||||
"ProvenanceMismatch",
|
||||
"ResolutionOutcome",
|
||||
"TraceRecord",
|
||||
"Witness",
|
||||
"as_clean",
|
||||
"check_reads",
|
||||
"stamp",
|
||||
"verify_stamp",
|
||||
]
|
||||
@@ -0,0 +1,57 @@
|
||||
"""Coverage accounting for the detection core (Req 3).
|
||||
|
||||
Every input is accounted for: entry recognition, exactly one resolution bucket per
|
||||
call, named reasons for unchecked operations. A zero-byte or unreadable input is an
|
||||
instrument failure -- a zero from an unread input is a failed run, never a clean one.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import enum
|
||||
from collections.abc import Mapping
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
|
||||
class InstrumentFailure(ValueError):
|
||||
"""The run's input was not read; any result from it is invalid."""
|
||||
|
||||
|
||||
class ResolutionOutcome(enum.Enum):
|
||||
"""Exactly one bucket per call (Req 3.1)."""
|
||||
|
||||
COMPLETE = "complete"
|
||||
INCOMPLETE = "incomplete"
|
||||
UNRESOLVED = "unresolved"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Coverage:
|
||||
"""Coverage of one analyzed change."""
|
||||
|
||||
entry_points: int
|
||||
frameworks_recognised: dict[str, int] = field(default_factory=dict)
|
||||
calls: int = 0
|
||||
complete: int = 0
|
||||
incomplete: int = 0
|
||||
unresolved: int = 0
|
||||
unchecked_reasons: dict[str, int] = field(default_factory=dict)
|
||||
|
||||
def validate(self) -> None:
|
||||
"""Refuse coverage where calls lack exactly one resolution bucket."""
|
||||
buckets = (self.complete, self.incomplete, self.unresolved)
|
||||
if min(buckets) < 0:
|
||||
raise ValueError(f"negative resolution bucket in {buckets}")
|
||||
if sum(buckets) != self.calls:
|
||||
raise ValueError(
|
||||
f"resolution buckets ({self.complete}+{self.incomplete}+{self.unresolved}) must equal the call count ({self.calls})"
|
||||
)
|
||||
|
||||
|
||||
def check_reads(reads: Mapping[str, int]) -> None:
|
||||
"""Fail the run when any tracked input was read as zero bytes (Req 3.2)."""
|
||||
zero = sorted(path for path, size in reads.items() if size <= 0)
|
||||
if zero:
|
||||
raise InstrumentFailure(f"zero-byte read of tracked input: {', '.join(zero)}")
|
||||
|
||||
|
||||
__all__ = ["Coverage", "InstrumentFailure", "ResolutionOutcome", "check_reads"]
|
||||
@@ -0,0 +1,62 @@
|
||||
"""Provenance digest stamping for the detection core (Req 6).
|
||||
|
||||
Every analysis is stamped with the analyzer revision, fact-set, acquisition-record,
|
||||
configuration, population, and per-change input digests. A result whose digests
|
||||
cannot be re-derived from the named inputs is refused, never accepted.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
from collections.abc import Mapping
|
||||
from typing import Any
|
||||
|
||||
|
||||
class ProvenanceMismatch(ValueError):
|
||||
"""A recorded digest does not match the re-derived input identity."""
|
||||
|
||||
|
||||
_DIGEST_FIELDS = (
|
||||
"analyzer_revision",
|
||||
"fact_set_digest",
|
||||
"acquisition_record_digest",
|
||||
"configuration_digest",
|
||||
"population_digest",
|
||||
"input_digest",
|
||||
)
|
||||
|
||||
|
||||
def _input_digest(change_id: str, population_digest: str) -> str:
|
||||
payload = json.dumps([population_digest, change_id], sort_keys=True, separators=(",", ":"))
|
||||
return hashlib.sha256(payload.encode()).hexdigest()[:12]
|
||||
|
||||
|
||||
def stamp(
|
||||
*,
|
||||
analyzer_revision: str,
|
||||
fact_set_digest: str,
|
||||
acquisition_record_digest: str,
|
||||
configuration_digest: str,
|
||||
population_digest: str,
|
||||
change_id: str,
|
||||
) -> dict[str, str]:
|
||||
"""The provenance stamp of one analysis (Req 6.1)."""
|
||||
return {
|
||||
"analyzer_revision": analyzer_revision,
|
||||
"fact_set_digest": fact_set_digest,
|
||||
"acquisition_record_digest": acquisition_record_digest,
|
||||
"configuration_digest": configuration_digest,
|
||||
"population_digest": population_digest,
|
||||
"input_digest": _input_digest(change_id, population_digest),
|
||||
}
|
||||
|
||||
|
||||
def verify_stamp(recorded: Mapping[str, Any], rederived: Mapping[str, Any]) -> None:
|
||||
"""Refuse when any digest cannot be re-derived (Req 6.2)."""
|
||||
for field_name in _DIGEST_FIELDS:
|
||||
if recorded.get(field_name) != rederived.get(field_name):
|
||||
raise ProvenanceMismatch(f"{field_name}: recorded {recorded.get(field_name)!r} != rederived {rederived.get(field_name)!r}")
|
||||
|
||||
|
||||
__all__ = ["ProvenanceMismatch", "stamp", "verify_stamp"]
|
||||
@@ -0,0 +1,74 @@
|
||||
"""Canonical trace schema for the detection core (Req 1).
|
||||
|
||||
One family-agnostic trace record per analyzed change, serialized deterministically:
|
||||
identical inputs produce byte-identical canonical output. The access-control
|
||||
prover's trace is the first implementation of this contract.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from typing import Any
|
||||
|
||||
from openultrasast.core.verdict import Judgment
|
||||
|
||||
|
||||
def _canonical_bytes(payload: dict[str, Any]) -> bytes:
|
||||
return json.dumps(payload, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode()
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Hop:
|
||||
"""One ordered hop with its call-target completeness."""
|
||||
|
||||
function: str
|
||||
call_line: int | None
|
||||
complete: bool
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Witness:
|
||||
"""Evidence for or against a guard, with provenance and match relation."""
|
||||
|
||||
kind: str
|
||||
provenance: str
|
||||
relation: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Guard:
|
||||
"""A guard effect observed on a path (kind plus effect)."""
|
||||
|
||||
kind: str
|
||||
effect: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TraceRecord:
|
||||
"""The canonical security trace: entry, hops, operation, witnesses, guards, judgment, digests."""
|
||||
|
||||
entry: str
|
||||
hops: tuple[Hop, ...]
|
||||
operation_kind: str
|
||||
resource: str
|
||||
authority: str
|
||||
witnesses: tuple[Witness, ...] = ()
|
||||
guards: tuple[Guard, ...] = ()
|
||||
completeness_reasons: tuple[str, ...] = ()
|
||||
judgment: Judgment = Judgment.INDETERMINATE
|
||||
digests: dict[str, str] = field(default_factory=dict)
|
||||
|
||||
def record(self) -> dict[str, Any]:
|
||||
"""JSON-ready dict with sorted keys; enums encode as their value strings."""
|
||||
payload: dict[str, Any] = asdict(self)
|
||||
payload["judgment"] = self.judgment.value
|
||||
record: dict[str, Any] = json.loads(json.dumps(payload, sort_keys=True))
|
||||
return record
|
||||
|
||||
def canonical(self) -> bytes:
|
||||
"""Deterministic byte serialization (Req 1.2)."""
|
||||
return _canonical_bytes(self.record())
|
||||
|
||||
|
||||
__all__ = ["Hop", "TraceRecord", "Witness"]
|
||||
@@ -0,0 +1,51 @@
|
||||
"""Verdict vocabulary and verdict contract for the detection core (Req 2, 5, 9).
|
||||
|
||||
Every analyzed operation is judged as exactly one of unguarded, discharged, or
|
||||
indeterminate. A verdict is either a proven claim (witness attached, exact
|
||||
location) or a bounded silence (coverage reason attached). Indeterminate never
|
||||
converts to clean, and models may only judge residuals zero-shot over a trace.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import enum
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
|
||||
class Judgment(enum.Enum):
|
||||
"""The three-way judgment vocabulary; value strings are the wire format."""
|
||||
|
||||
UNGUARDED = "unguarded"
|
||||
DISCHARGED = "discharged"
|
||||
INDETERMINATE = "indeterminate"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ProvenClaim:
|
||||
"""A defect claim carrying its witness and exact location (Req 5.1)."""
|
||||
|
||||
trace_id: str
|
||||
witness: str
|
||||
path: str
|
||||
line: int
|
||||
family: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class BoundedSilence:
|
||||
"""An undecided site with its coverage reason; never a clean result (Req 5.2)."""
|
||||
|
||||
path: str
|
||||
reason: str
|
||||
coverage: dict[str, Any]
|
||||
|
||||
|
||||
def as_clean(judgment: Judgment) -> Judgment:
|
||||
"""Return a clean judgment, refusing to convert indeterminate (Req 2.1)."""
|
||||
if judgment is Judgment.INDETERMINATE:
|
||||
raise ValueError("indeterminate never converts to a clean judgment")
|
||||
return judgment
|
||||
|
||||
|
||||
__all__ = ["BoundedSilence", "Judgment", "ProvenClaim", "as_clean"]
|
||||
@@ -0,0 +1,132 @@
|
||||
"""Core contract conformance tests for detection-core tasks 1.1-1.4 (Reqs 1, 2, 3, 5, 6, 9)."""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
|
||||
from openultrasast.core.coverage import Coverage, InstrumentFailure, check_reads # noqa: E402
|
||||
from openultrasast.core.provenance import ProvenanceMismatch, stamp, verify_stamp # noqa: E402
|
||||
from openultrasast.core.trace import Guard, Hop, TraceRecord, Witness # noqa: E402
|
||||
from openultrasast.core.verdict import BoundedSilence, Judgment, ProvenClaim, as_clean # noqa: E402
|
||||
|
||||
|
||||
def _trace() -> TraceRecord:
|
||||
return TraceRecord(
|
||||
entry="src/app.py::handler",
|
||||
hops=(Hop(function="handler", call_line=10, complete=True),),
|
||||
operation_kind="protected_read",
|
||||
resource="credentials",
|
||||
authority="admin",
|
||||
witnesses=(Witness(kind="guard", provenance="fact:checkPermission", relation="all_paths"),),
|
||||
guards=(Guard(kind="permission_check", effect="branches"),),
|
||||
completeness_reasons=(),
|
||||
judgment=Judgment.UNGUARDED,
|
||||
digests={"input": "abc123"},
|
||||
)
|
||||
|
||||
|
||||
def test_trace_serializes_byte_identical_on_repeat() -> None:
|
||||
first = _trace().canonical()
|
||||
second = _trace().canonical()
|
||||
assert first == second
|
||||
assert isinstance(first, bytes)
|
||||
|
||||
|
||||
def test_trace_record_pins_schema_fields() -> None:
|
||||
record = _trace().record()
|
||||
for field in (
|
||||
"entry",
|
||||
"hops",
|
||||
"operation_kind",
|
||||
"resource",
|
||||
"authority",
|
||||
"witnesses",
|
||||
"guards",
|
||||
"completeness_reasons",
|
||||
"judgment",
|
||||
"digests",
|
||||
):
|
||||
assert field in record
|
||||
assert record["judgment"] == "unguarded"
|
||||
|
||||
|
||||
def test_judgment_vocabulary_is_exactly_three_way() -> None:
|
||||
assert {j.value for j in Judgment} == {"unguarded", "discharged", "indeterminate"}
|
||||
|
||||
|
||||
def test_indeterminate_never_converts_to_clean() -> None:
|
||||
assert as_clean(Judgment.DISCHARGED) == Judgment.DISCHARGED
|
||||
try:
|
||||
as_clean(Judgment.INDETERMINATE)
|
||||
raise AssertionError("indeterminate must never convert to clean")
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
|
||||
def test_proven_claim_requires_witness_and_location() -> None:
|
||||
claim = ProvenClaim(trace_id="t1", witness="w1", path="src/app.py", line=10, family="access_control")
|
||||
assert claim.family == "access_control"
|
||||
silence = BoundedSilence(path="src/app.py", reason="unreached", coverage={"entry_points": 0})
|
||||
assert silence.reason == "unreached"
|
||||
|
||||
|
||||
def test_coverage_requires_exactly_one_bucket_per_call() -> None:
|
||||
good = Coverage(entry_points=2, frameworks_recognised={"jenkins": 1}, calls=3, complete=1, incomplete=1, unresolved=1)
|
||||
good.validate()
|
||||
bad = Coverage(entry_points=1, frameworks_recognised={}, calls=3, complete=2, incomplete=1, unresolved=1)
|
||||
try:
|
||||
bad.validate()
|
||||
raise AssertionError("two buckets for one call must be refused")
|
||||
except ValueError:
|
||||
pass
|
||||
negative = Coverage(entry_points=1, frameworks_recognised={}, calls=3, complete=-1, incomplete=2, unresolved=2)
|
||||
try:
|
||||
negative.validate()
|
||||
raise AssertionError("negative bucket counts must be refused")
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
|
||||
def test_verdict_contract_exposes_no_model_role_path() -> None:
|
||||
import openultrasast.core.verdict as verdict_module
|
||||
|
||||
public = verdict_module.__all__
|
||||
assert not any("model" in name.lower() or "judge" in name.lower() for name in public), (
|
||||
"the verdict contract must expose no model/judge path; models judge residuals zero-shot over a trace only"
|
||||
)
|
||||
|
||||
|
||||
def test_zero_byte_read_is_instrument_failure() -> None:
|
||||
try:
|
||||
check_reads({"src/app.py": 0, "src/lib.py": 12})
|
||||
raise AssertionError("zero-byte read must fail the run")
|
||||
except InstrumentFailure:
|
||||
pass
|
||||
check_reads({"src/app.py": 5})
|
||||
|
||||
|
||||
def test_provenance_stamp_and_mismatch_refusal() -> None:
|
||||
record = stamp(
|
||||
analyzer_revision="d063167",
|
||||
fact_set_digest="3eb5bf699434",
|
||||
acquisition_record_digest="none",
|
||||
configuration_digest="095b5a4905ad",
|
||||
population_digest="c866c0e812c0",
|
||||
change_id="case-1",
|
||||
)
|
||||
assert set(record) == {
|
||||
"analyzer_revision",
|
||||
"fact_set_digest",
|
||||
"acquisition_record_digest",
|
||||
"configuration_digest",
|
||||
"population_digest",
|
||||
"input_digest",
|
||||
}
|
||||
verify_stamp(record, record.copy())
|
||||
tampered = {**record, "fact_set_digest": "ffffffffffff"}
|
||||
try:
|
||||
verify_stamp(tampered, record)
|
||||
raise AssertionError("mismatched digest must be refused")
|
||||
except ProvenanceMismatch:
|
||||
pass
|
||||
Reference in New Issue
Block a user