docs(detection-core): approve deletion-backed migration plan

This commit is contained in:
Marius Ciepluch
2026-11-02 09:38:45 +01:00
parent 7c0745247e
commit 1b9018644e
4 changed files with 323 additions and 58 deletions
+175 -35
View File
@@ -15,7 +15,9 @@ and bounded silence.
**Impact:** the current authorization-shaped core becomes a temporary v1 compatibility **Impact:** the current authorization-shaped core becomes a temporary v1 compatibility
surface. A neutral v2 write contract replaces it through a finite adapter migration. surface. A neutral v2 write contract replaces it through a finite adapter migration.
Authorization remains parked and supplies regression evidence only. Historical v1 artifacts Authorization remains parked and supplies regression evidence only. Historical v1 artifacts
remain readable but are never rewritten or used as a v2 trend baseline. remain readable but are never rewritten or used as a v2 trend baseline. Live v1 contracts
and serialization first move behind one explicit, deletable `legacy_v1` namespace; cutover
is then proved by syntax and capability removal rather than runtime alias inference.
### Goals ### Goals
@@ -30,6 +32,8 @@ remain readable but are never rewritten or used as a v2 trend baseline.
policy into the core. policy into the core.
- End migration with one live write contract and two unlike reference-family conformance - End migration with one live write contract and two unlike reference-family conformance
suites. suites.
- Make the v1 removal gate structural: every live v1 import and writer crosses an explicit
compatibility namespace that is deleted at cutover.
### Non-Goals ### Non-Goals
@@ -59,6 +63,8 @@ remain readable but are never rewritten or used as a v2 trend baseline.
receive existing subsystem records and perform no analysis, model, proof, or persistence receive existing subsystem records and perform no analysis, model, proof, or persistence
orchestration. orchestration.
- A finite v1 migration and read-only historical decoder. - A finite v1 migration and read-only historical decoder.
- The temporary `legacy_v1` compatibility namespace, frozen migration inventory, and
syntax-only cutover boundary that make v1 write capability removable.
### Out of Boundary ### Out of Boundary
@@ -92,6 +98,12 @@ remain readable but are never rewritten or used as a v2 trend baseline.
applies those objects but never imports the target adapter or creates its authority. applies those objects but never imports the target adapter or creates its authority.
- No core module imports `openultrasast.semantic`, `openultrasast.hunt`, - No core module imports `openultrasast.semantic`, `openultrasast.hunt`,
`openultrasast.search`, `openultrasast.tools`, `benchmarks`, or `dspy`. `openultrasast.search`, `openultrasast.tools`, `benchmarks`, or `dspy`.
- V2 modules never import `openultrasast.core.legacy_v1` or the retained read-only
`openultrasast.core.legacy` decoder. During migration, only compatibility shims and
registered adjacent v1 consumers may import `legacy_v1`.
- Production imports name a core submodule or symbol explicitly. Module-valued or dynamic
access to `openultrasast.core` is outside the migration contract because it would hide
legacy use from a deletion-backed gate.
### Revalidation Triggers ### Revalidation Triggers
@@ -107,6 +119,8 @@ remain readable but are never rewritten or used as a v2 trend baseline.
proof-consuming verdict path. proof-consuming verdict path.
- Adding a non-Python writer or cryptographic wire signature requires a canonicalization - Adding a non-Python writer or cryptographic wire signature requires a canonicalization
design review. design review.
- Adding a v1 compatibility export, migration-inventory entry, direct v1 serializer, or
dynamic core import reopens the namespace-isolation design and blocks cutover.
## Architecture ## Architecture
@@ -127,9 +141,10 @@ replacing the public domain model.
### Architecture Pattern and Boundary Map ### Architecture Pattern and Boundary Map
The selected pattern is a **versioned ports-and-adapters contract with a finite compatibility The selected pattern is a **versioned ports-and-adapters contract with an explicit,
migration**. Core contracts and pure validators form the domain. External systems implement deletable compatibility namespace**. Core contracts and pure validators form the domain.
adapters and retain execution ownership. External systems implement adapters and retain execution ownership. The migration boundary
is a module boundary, not a semantic-use analysis over Python execution.
```mermaid ```mermaid
flowchart LR flowchart LR
@@ -149,6 +164,10 @@ flowchart LR
ProofAdapter[Proof adapter] ProofAdapter[Proof adapter]
LegacyAdapter[Legacy adapter] LegacyAdapter[Legacy adapter]
end end
subgraph Compatibility[Temporary compatibility]
LegacyV1[Explicit legacy v1 namespace]
LegacyRead[Read only decoder]
end
subgraph Consumers[Consumers] subgraph Consumers[Consumers]
Benchmark[Benchmark workflow] Benchmark[Benchmark workflow]
Push[Pre push consumer] Push[Pre push consumer]
@@ -165,6 +184,8 @@ flowchart LR
ModelAdapter --> Decision ModelAdapter --> Decision
ProofAdapter --> Decision ProofAdapter --> Decision
LegacyAdapter --> Semantics LegacyAdapter --> Semantics
LegacyV1 --> LegacyAdapter
LegacyRead --> Benchmark
Admission --> Benchmark Admission --> Benchmark
Decision --> Push Decision --> Push
``` ```
@@ -185,9 +206,10 @@ canonical
→ consumer → consumer
``` ```
Each core layer imports only layers to its left. `legacy.py` may import public v2 types to Each core layer imports only layers to its left. `legacy_v1` imports only the canonical
decode historical input but no v2 module imports `legacy.py`. External adapters import the primitive needed for byte stability and is never imported by a v2 module. `legacy.py`
public core package and their respective execution subsystem. validates historical bytes without importing `legacy_v1` or constructing v2 types. External
adapters import the public core package and their respective execution subsystem.
**Architecture integration:** **Architecture integration:**
@@ -200,6 +222,10 @@ public core package and their respective execution subsystem.
- The model and proof paths derive decisions tied to an unchanged deterministic packet - The model and proof paths derive decisions tied to an unchanged deterministic packet
digest; neither can rewrite analyzer evidence. digest; neither can rewrite analyzer evidence.
- Historical v1 readers remain validation-only after migration; all new writes use v2. - Historical v1 readers remain validation-only after migration; all new writes use v2.
- Before consumer migration, every v1 type and write path is re-parented to `legacy_v1`.
The old top-level names remain temporary explicit shims, while mixed v2 modules cease to
define legacy symbols. This makes later zero checks syntactic and makes final deletion a
proof of capability removal.
### Technology Stack ### Technology Stack
@@ -220,6 +246,10 @@ No new runtime dependency is introduced.
```text ```text
src/openultrasast/core/ src/openultrasast/core/
├── __init__.py # Public v2 exports only after migration ├── __init__.py # Public v2 exports only after migration
├── legacy_v1/ # Temporary closed compatibility namespace; deleted at cutover
│ ├── __init__.py # Explicit v1 contract exports; no dynamic discovery
│ ├── contracts.py # Authorization-shaped v1 values moved out of mixed v2 modules
│ └── writer.py # Only live v1 serialization capability during migration
├── canonical.py # Canonical bytes, identities, content references and read-only resolver port ├── canonical.py # Canonical bytes, identities, content references and read-only resolver port
├── semantics.py # Locations, nodes, transitions, observations, completeness ├── semantics.py # Locations, nodes, transitions, observations, completeness
├── evidence.py # Scope, packet, decision, evidence record, neutral enums ├── evidence.py # Scope, packet, decision, evidence record, neutral enums
@@ -262,7 +292,11 @@ tests/fixtures/detection_core/
├── obligation_family.json # Test-only control and constraint family contract ├── obligation_family.json # Test-only control and constraint family contract
├── taint_family.json # Test-only flow and sanitization family contract ├── taint_family.json # Test-only flow and sanitization family contract
├── obligation_cases.json # Positive, negative and missing-evidence triplet ├── obligation_cases.json # Positive, negative and missing-evidence triplet
└── taint_cases.json # Positive, negative and missing-evidence triplet ├── taint_cases.json # Positive, negative and missing-evidence triplet
└── v1/
├── trace.json # Byte-frozen historical trace
├── report.json # Byte-frozen historical report
└── migration_inventory.json # Frozen registered v1 readers and writers
tests/ tests/
├── test_core_canonical.py ├── test_core_canonical.py
@@ -280,7 +314,8 @@ tests/
├── test_core_measurement.py ├── test_core_measurement.py
├── test_core_admission.py ├── test_core_admission.py
├── test_core_consumer.py ├── test_core_consumer.py
└── test_core_migration.py ├── test_core_migration.py
└── core_migration_boundary.py # Syntax-only import/writer and inventory-closure checks
``` ```
Files below `hunt/`, `search/`, `semantic/`, and `benchmarks/` are adjacent-owner Files below `hunt/`, `search/`, `semantic/`, and `benchmarks/` are adjacent-owner
@@ -292,8 +327,18 @@ not execute proof or family analysis.
### Modified and Removed Files ### Modified and Removed Files
- `src/openultrasast/core/trace.py` — temporary v1 import shim during migration; removed - `src/openultrasast/core/trace.py` and `_legacy_factpack.py` — temporary v1 import shims
after repository import census reaches zero. Historical decoding moves to `legacy.py`. during migration; removed after the syntactic legacy boundary reaches zero. They
re-export from `legacy_v1` and own no contract or serializer. Historical decoding remains
in `legacy.py`.
- `src/openultrasast/core/verdict.py`, `consumer.py`, and `admission.py` — temporary
explicit shims over `legacy_v1`; at cutover their v1 contents disappear and the public v2
modules/facades take these canonical paths atomically.
- `src/openultrasast/core/coverage.py`, `budget.py`, `provenance.py`, `probes.py`, and
`factpack.py` — retain only v2/shared definitions after legacy definitions move to
`legacy_v1`; they do not re-export v1 names.
- `src/openultrasast/core/__init__.py` — preserves the old named API by importing from
`legacy_v1` during migration, then atomically replaces it with the public v2 surface.
- `src/openultrasast/semantic/obligations/trace.py` — removes imports and conversion logic - `src/openultrasast/semantic/obligations/trace.py` — removes imports and conversion logic
for core types; remains the authorization-family trace domain. for core types; remains the authorization-family trace domain.
- `src/openultrasast/hunt/prover.py` — sends authorization results through - `src/openultrasast/hunt/prover.py` — sends authorization results through
@@ -302,6 +347,12 @@ not execute proof or family analysis.
legacy tests; neutral tests use the two reference contracts. legacy tests; neutral tests use the two reference contracts.
- `tests/test_core_contract.py` — replaces the assertion that no model path exists with the - `tests/test_core_contract.py` — replaces the assertion that no model path exists with the
assertion that core has no DSPy import and only exposes dependency-free model contracts. assertion that core has no DSPy import and only exposes dependency-free model contracts.
- `tests/core_migration_boundary.py` — parses import and call syntax without evaluating
aliases; it rejects `legacy_v1`, temporary shim, named top-level v1, dynamic core access,
and v1-writer references according to the migration phase.
- `tests/fixtures/detection_core/v1/migration_inventory.json` — immutable inventory derived
from the reviewed Task 1.1 census. Every entry must become migrated or retired; the
inventory cannot grow to make a test pass.
Every new module remains within the project cap of 400 logical lines; functions remain at Every new module remains within the project cap of 400 logical lines; functions remain at
or below 50 logical lines and cyclomatic complexity 10. or below 50 logical lines and cyclomatic complexity 10.
@@ -411,7 +462,7 @@ scoring.
| 11.1, 11.2, 11.3, 11.4, 11.5, 11.6 | Reviewed semantic packs and supported scope | Fact Pack, Semantic Evidence, Measurement | Pack ingestion and reports | | 11.1, 11.2, 11.3, 11.4, 11.5, 11.6 | Reviewed semantic packs and supported scope | Fact Pack, Semantic Evidence, Measurement | Pack ingestion and reports |
| 12.1, 12.2, 12.3, 12.4, 12.5, 12.6 | Admission prerequisites, authenticated case release and scoped scoring | Admission, Readiness, Measurement | Admission sequence | | 12.1, 12.2, 12.3, 12.4, 12.5, 12.6 | Admission prerequisites, authenticated case release and scoped scoring | Admission, Readiness, Measurement | Admission sequence |
| 13.1, 13.2, 13.3, 13.4 | Development and evaluation separation | Provenance and Lineage, Probes, Admission | Freeze and admission validation | | 13.1, 13.2, 13.3, 13.4 | Development and evaluation separation | Provenance and Lineage, Probes, Admission | Freeze and admission validation |
| 14.1, 14.2, 14.3, 14.4, 14.5 | Shared consumer and family-spec contract | Consumer, Verdict, Family Capability Declaration, Legacy Adapter | Push and migration flows | | 14.1, 14.2, 14.3, 14.4, 14.5 | Shared consumer and family-spec contract | Consumer, Verdict, Family Capability Declaration, Legacy Compatibility Namespace, Legacy Decoder | Push and migration flows |
| 15.1, 15.2, 15.3, 15.4, 15.5 | Fixed versioned measurement and baseline rules | Measurement, Coverage, Readiness | Development and admission reporting | | 15.1, 15.2, 15.3, 15.4, 15.5 | Fixed versioned measurement and baseline rules | Measurement, Coverage, Readiness | Development and admission reporting |
## Components and Interfaces ## Components and Interfaces
@@ -434,7 +485,8 @@ scoring.
| Measurement | Evaluation | Create baseline identity and fixed report schema | 11.6, 15.1–15.5 | Coverage, Readiness | Service, State | | Measurement | Evaluation | Create baseline identity and fixed report schema | 11.6, 15.1–15.5 | Coverage, Readiness | Service, State |
| Admission | Evaluation | Validate manifests, feasibility, post-spend case release and scoped scores | 12.1–12.6, 13.4 | Readiness, Measurement, Evidence Validation, Verdict, Probes, Budget, Provenance | Service, Batch | | Admission | Evaluation | Validate manifests, feasibility, post-spend case release and scoped scores | 12.1–12.6, 13.4 | Readiness, Measurement, Evidence Validation, Verdict, Probes, Budget, Provenance | Service, Batch |
| Consumer | Edge | Validate push requests and return bounded verdicts | 6.4, 14.1–14.5 | Budget, Evidence Validation, Verdict, Provenance, Canonical resolver | Service | | Consumer | Edge | Validate push requests and return bounded verdicts | 6.4, 14.1–14.5 | Budget, Evidence Validation, Verdict, Provenance, Canonical resolver | Service |
| Legacy Decoder | Migration | Read and validate historical v1 only | 14.3, 15.3 | Canonical Codec, Evidence | Service | | Legacy Compatibility Namespace | Migration | Isolate every live v1 contract and writer behind a deletable boundary | 14.3, 15.3 | Canonical Codec, frozen migration inventory | State, Service |
| Legacy Decoder | Migration | Read and validate historical v1 bytes only | 14.3, 15.3 | Canonical Codec | Service |
| External Adapters | Integration | Translate authorization, DSPy and proof records | 3.5, 9.1–9.6, 14.2–14.3 | Public core | Service | | External Adapters | Integration | Translate authorization, DSPy and proof records | 3.5, 9.1–9.6, 14.2–14.3 | Public core | Service |
### Evidence Domain ### Evidence Domain
@@ -2430,8 +2482,74 @@ fast run.
### Migration Components ### Migration Components
`legacy.py` decodes and validates historical v1 traces and reports. It never constructs v2 #### Legacy Compatibility Namespace and Cutover Boundary
determinate outcomes without a family adapter and never emits a new v1 artifact.
| Field | Detail |
|---|---|
| Intent | Turn v1 dependency and write capability into explicit modules that can be counted and deleted without interpreting Python execution |
| Requirements | 14.3, 15.3 |
**Responsibilities and constraints**
- `legacy_v1/contracts.py` temporarily owns every authorization-shaped v1 value moved out
of mixed v2 modules. `legacy_v1/writer.py` owns the only live v1 serialization capability.
- Existing top-level v1 names remain compatible during migration only through explicit
imports from `legacy_v1`. Compatibility modules contain no independent implementation.
- The frozen migration inventory names every reviewed reader and writer by path, symbol,
artifact family, and disposition. An entry closes only as `v2_migrated` or `retired`;
adding a new entry is a design regression, not remediation.
- The boundary checker reads every scoped source file and proves it is nonzero before
parsing. It classifies import statements and direct writer-facade references only. It
never follows assignment, default-argument, exception, comprehension, dynamic-call, or
runtime attribute flow.
- Production module-valued imports of `openultrasast.core`, dynamic imports of the package,
and dynamic attribute access to its compatibility names are prohibited. Explicit
submodule or symbol imports are required and mechanically checkable.
- Single-writer cutover requires four simultaneous facts: every inventory entry is closed;
syntactic legacy imports and writer references are zero; the compatibility namespace and
shim modules are absent; and the old top-level attributes are not importable. A plausible
zero from an unread or unparsable file is an instrument failure.
**Contracts:** Service, State.
```python
class MigrationDisposition(Enum):
OPEN = "open"
V2_MIGRATED = "v2_migrated"
RETIRED = "retired"
@dataclass(frozen=True)
class LegacyInventoryEntry:
path: str
symbol: str
dependency_kind: str
artifact_family: str
disposition: MigrationDisposition
@dataclass(frozen=True)
class LegacyBoundaryReport:
files_read: int
bytes_read: int
open_inventory_entries: tuple[LegacyInventoryEntry, ...]
legacy_import_sites: tuple[str, ...]
legacy_writer_sites: tuple[str, ...]
forbidden_dynamic_sites: tuple[str, ...]
def inspect_legacy_boundary(paths: Sequence[Path]) -> LegacyBoundaryReport: ...
def require_cutover_ready(report: LegacyBoundaryReport) -> None: ...
```
`inspect_legacy_boundary` is a migration test instrument, not a runtime core service. Its
schema is intentionally syntactic. Runtime use-flow analysis is neither required nor
accepted as evidence for cutover.
#### Historical Decoder and External Adapters
`legacy.py` decodes and validates historical v1 traces and reports from exact built-in
`bytes`. It never imports `legacy_v1`, constructs v2 determinate outcomes, or emits a new v1
artifact. The compatibility namespace is therefore deletable while historical artifacts
remain readable.
Authorization conversion moves to `semantic/obligations/core_adapter.py`, where old fields Authorization conversion moves to `semantic/obligations/core_adapter.py`, where old fields
map to the authorization family contract. Missing per-hop completeness remains incomplete. map to the authorization family contract. Missing per-hop completeness remains incomplete.
The DSPy and search adapters similarly map external typed records without leaking their The DSPy and search adapters similarly map external typed records without leaking their
@@ -2502,6 +2620,7 @@ that can be mistaken for a clean scan.
| Review mismatch | Self-review, unknown reviewer, copied attestation, or unreadable disclosure | Refuse before feasibility or semantic use | | Review mismatch | Self-review, unknown reviewer, copied attestation, or unreadable disclosure | Refuse before feasibility or semantic use |
| Custody mismatch | Old/incomplete log, broken chain, or prior spend | Refuse use/admission; preserve monotonic spent status | | Custody mismatch | Old/incomplete log, broken chain, or prior spend | Refuse use/admission; preserve monotonic spent status |
| Case release mismatch | Pre-spend release, wrong custodian/receipt, missing/extra case, role/pair swap, or operational payload substitution | Refuse before metric production | | Case release mismatch | Pre-spend release, wrong custodian/receipt, missing/extra case, role/pair swap, or operational payload substitution | Refuse before metric production |
| Migration-boundary failure | Unreadable source, open inventory entry, legacy/shim import, dynamic core access, or v1 writer reference | Refuse cutover; report exact path, symbol and dependency kind |
### Monitoring ### Monitoring
@@ -2624,6 +2743,17 @@ data only and do not establish product capability or admission.
(`12.2`, `13.4`). (`12.2`, `13.4`).
- Legacy and v2 validators coexist by explicit schema version; committed historical reports - Legacy and v2 validators coexist by explicit schema version; committed historical reports
remain byte-unchanged and cannot be written through the v2 API (`14.3`, `15.3`). remain byte-unchanged and cannot be written through the v2 API (`14.3`, `15.3`).
- Namespace-isolation tests move every legacy definition out of mixed v2 modules while the
temporary top-level named API remains byte-compatible. Direct imports of moved symbols
from mixed modules fail, proving there is one compatibility owner (`14.3`).
- The migration-boundary test proves every scoped input is readable and nonzero, requires
the frozen inventory to shrink without additions, and reports explicit legacy namespace,
shim, top-level symbol, writer-facade, module-valued core, and dynamic core access sites.
Adversarial assignment, `try`/`finally`, comprehension, and generic-call aliases do not
affect the result because the gate never interprets use flow (`14.3`, `15.3`).
- Cutover tests delete the compatibility package and shims in an isolated tree, verify old
top-level and submodule imports fail, verify v2 imports and writes still succeed, and
validate the frozen v1 bytes through `legacy.py` (`14.3`, `15.3`).
### End-to-End Readiness Test ### End-to-End Readiness Test
@@ -2679,6 +2809,10 @@ act. Synthetic admission manifests test orchestration without spending a real po
- Proof attachment verifies resolved identity, producer receipt, reviewed control binding, - Proof attachment verifies resolved identity, producer receipt, reviewed control binding,
and differential outcome; trusted search infrastructure retains sandbox and oracle and differential outcome; trusted search infrastructure retains sandbox and oracle
execution responsibility. execution responsibility.
- Migration never treats a successful alias analysis as proof that v1 is absent. Legacy
write capability is isolated, inventory-closed, and deleted; module-valued/dynamic access
to the compatibility surface is prohibited so a hidden dependency fails the boundary
check or import after cutover.
## Performance and Scalability ## Performance and Scalability
@@ -2697,34 +2831,40 @@ act. Synthetic admission manifests test orchestration without spending a real po
```mermaid ```mermaid
flowchart LR flowchart LR
V1[Legacy v1 writes] --> V2[Add v2 contracts] V1[Mixed legacy surface] --> Isolate[Isolate legacy namespace]
Isolate --> Inventory[Freeze migration inventory]
Inventory --> V2[Add v2 contracts]
V2 --> References[Reference conformance] V2 --> References[Reference conformance]
References --> AuthAdapter[Authorization adapter] References --> Adapters[Adjacent adapters migrate]
AuthAdapter --> Benchmarks[Readiness and measurement v2] Adapters --> Zero[Syntax and inventory zero]
Benchmarks --> Consumers[Model proof and push conformance] Zero --> RemoveV1[Delete v1 writer and shims]
Consumers --> StopV1[Stop v1 writes] RemoveV1 --> PublishV2[Publish v2 top level]
StopV1 --> RemoveV1[Remove v1 exports] PublishV2 --> LegacyRead[Retain read only decoder]
RemoveV1 --> LegacyRead[Read only legacy decoder]
``` ```
1. **Foundation:** add canonical, semantics, evidence, family, and reference-family 1. **Foundation:** add canonical, semantics, evidence, family, and reference-family
contracts without changing v1 consumers. contracts without changing v1 behavior.
2. **Adapter migration:** move authorization conversion out of engine trace code and emit v2 2. **Namespace isolation:** move all v1 contracts and serialization behind `legacy_v1`,
convert the old named API to explicit shims, remove legacy definitions from mixed v2
modules, and freeze the reviewed migration inventory.
3. **Adapter migration:** move authorization conversion out of engine trace code and emit v2
in development. Preserve conservative incomplete hops and existing family behavior. in development. Preserve conservative incomplete hops and existing family behavior.
3. **Governance migration:** add readiness, measurement, admission, lineage, and model v2; 4. **Governance migration:** add readiness, measurement, admission, lineage, and model v2;
historical v1 validation remains separate. This creates a new baseline. historical v1 validation remains separate. This creates a new baseline.
4. **Consumer conformance:** add DSPy and proof adapters and validate push consumption with 5. **Consumer conformance:** add DSPy and proof adapters and validate push consumption with
synthetic/reference records. Downstream rollout remains disabled. synthetic/reference records. Downstream rollout remains disabled.
5. **Single-writer cutover:** stop v1 writes after repository import and artifact-writer 6. **Structural zero gate:** close every frozen inventory entry and require zero explicit
censuses reach zero. Remove authorization-shaped top-level exports and the `trace.py` compatibility imports, top-level legacy symbols, shim imports, dynamic core access, and
shim. v1 writer-facade references. The checker uses syntax only and fails on unreadable input.
6. **Historical retention:** retain `legacy.py` read-only validation for committed v1 7. **Single-writer cutover:** delete `legacy_v1`, the writer facade, and compatibility shims;
artifacts. It exposes no analyzer or writer API. prove old imports fail; then publish the dependency-free v2 top-level surface.
8. **Historical retention:** retain `legacy.py` read-only validation for committed v1
artifacts. It exposes no analyzer, contract constructor, converter, or writer API.
Rollback is allowed before the single-writer cutover by keeping existing consumers on v1. Rollback is allowed before the single-writer cutover by retaining the isolated compatibility
After cutover, rollback means reverting the complete migration change; it must not restore namespace and shims. After cutover, rollback means reverting the complete migration change;
parallel v1 writes. No migration phase re-runs authorization admission or changes its parked it must not restore parallel v1 writes. No migration phase re-runs authorization admission
status. or changes its parked status.
## Supporting References ## Supporting References
+85
View File
@@ -626,3 +626,88 @@ requirements approval.
- The downstream spec must first deliver authority, reviewed packs, persistence, and the - The downstream spec must first deliver authority, reviewed packs, persistence, and the
structural adapter. Core conformance then proves those objects satisfy the generic structural adapter. Core conformance then proves those objects satisfy the generic
contract before readiness, admission, or single-writer cutover can depend on them. contract before readiness, admission, or single-writer cutover can depend on them.
---
## Legacy namespace isolation and cutover-gate correction (2026-11-02)
### Scope and instrument evidence
- **Trigger:** task 10.1 could validate frozen v1 bytes and enumerate known writers, but
repeated adversarial review showed that an exact semantic-use census over arbitrary
Python aliases is equivalent to a partial Python interpreter. The bounded analyzer
produced false zeroes for partial `try`/`finally` paths, generic alias escape, and
module-valued `ImportFrom` bindings.
- **Instrument proof:** the final review opened 803 nonzero Python inputs totaling
8,193,901 bytes with a 29-byte minimum; the focused migration suite passed 85 tests and
the core suite passed 916. The failure is therefore in the census model, not unread
input, stale fixtures, or a broken decoder.
- **Existing assets retained:** the exact-byte v1 trace/report decoder, hostile-byte and
report-invariant tests, the frozen historical fixtures, the reviewed initial
import/writer inventory, and the v2 contracts remain valid inputs.
### Architecture pattern evaluation
| Option | Description | Strengths | Risks and limitations | Decision |
|---|---|---|---|---|
| Continue alias interpretation | Extend the custom AST flow model until it recognizes more Python execution forms | Preserves mixed modules | Unbounded semantics; another construct can create a plausible zero | Rejected |
| Count every mixed-module import | Treat imports of shared modules such as `provenance` as v1 dependencies | Simple and conservative | V2-only users can never reach zero; gate measures import style rather than v1 dependency | Rejected |
| Explicit legacy namespace | Move v1 contracts and temporary write capability behind `openultrasast.core.legacy_v1`; keep only explicit compatibility shims until cutover | Syntactic, deletion-backed gate; no use-flow inference; preserves historical bytes | Requires one bounded namespace-isolation migration before consumer cutover | Selected |
### Decision: make legacy capability explicit and deletable
- `openultrasast.core.legacy_v1` temporarily owns all live v1 contract values and the only
v1 serialization capability. Mixed v2 modules stop defining or re-exporting legacy
symbols.
- The existing top-level v1 API remains byte- and name-compatible during migration, but its
implementation is an explicit shim importing from `legacy_v1`. Production code may not
use module-valued `openultrasast.core` imports or dynamic import/getattr to reach that
shim; imports must name a submodule or symbol.
- The previously reviewed census becomes a frozen migration inventory. Every registered
reader/writer is migrated to v2 or retired; new inventory entries are not a path to
completion.
- The cutover checker parses import and call syntax only. It reports imports of the legacy
namespace, temporary shim modules, top-level legacy symbols, and the explicit v1 writer
facade. It does not infer runtime alias values.
- The zero gate additionally proves that the v1 writer facade and compatibility shim
modules are deleted and no legacy symbol is importable from the top-level package. A
surviving dependency therefore fails import instead of disappearing from a semantic-use
census.
- `openultrasast.core.legacy` remains a separate read-only historical byte decoder. It has
no contract constructors, serializer, converter, or v2 outcome mapping and survives the
cutover.
### Boundary and task impact
- Detection-core owns namespace isolation, the frozen migration inventory, the syntactic
boundary checker, top-level shim removal, and the retained decoder.
- Adjacent owners migrate their registered imports and writers; they do not extend the
compatibility namespace.
- Task 10.1 must be re-planned as namespace isolation plus schema-routed historical reads.
Task 10.2 may publish the v2 top-level only after inventory closure, syntactic zero,
deletion of the v1 write/shim capability, and unchanged historical fixture hashes.
### Synthesis
- **Generalization:** absence is proved by removing a capability boundary, not by predicting
every way Python can transport an alias. The same rule covers readers, writers, top-level
compatibility names, and future adjacent consumers.
- **Build versus adopt:** Python's standard `ast` is sufficient for the intentionally
syntactic boundary check. A new import-lint or abstract-interpretation dependency would
either duplicate the project rule or reintroduce semantic-use analysis and is not
adopted.
- **Simplification:** discard the custom lexical/control-flow census and its migration-only
flow engine. Retain the reviewed inventory as evidence, then rely on explicit modules,
forbidden dynamic access, import-failure tests, and deletion of the writer capability.
### Risks and mitigations
- **Hidden dynamic access:** production dynamic import/getattr of `openultrasast.core` is
prohibited and checked syntactically; no runtime dataflow inference is attempted.
- **New unregistered v1 writer:** the compatibility namespace is closed, and only its
explicit writer facade can serialize a v1 contract. Review and tests reject new direct
compatibility exports or inventory growth.
- **Top-level compatibility masks incomplete migration:** the gate counts named top-level
legacy imports and then proves those attributes are absent after cutover.
- **Historical artifacts become unreadable:** frozen trace/report bytes and hashes continue
through the independent read-only decoder before and after namespace deletion.
+1 -1
View File
@@ -1,7 +1,7 @@
{ {
"feature_name": "detection-core", "feature_name": "detection-core",
"created_at": "2026-10-30T13:55:30Z", "created_at": "2026-10-30T13:55:30Z",
"updated_at": "2026-11-01T15:44:59Z", "updated_at": "2026-11-02T07:51:51Z",
"language": "en", "language": "en",
"phase": "tasks-generated", "phase": "tasks-generated",
"approvals": { "approvals": {
+62 -22
View File
@@ -18,13 +18,14 @@
- Existing target facts may be schema-migrated without expanding vocabulary or behavior. - Existing target facts may be schema-migrated without expanding vocabulary or behavior.
- [ ] 1. Establish migration-safe foundations - [ ] 1. Establish migration-safe foundations
- [ ] 1.1 Freeze the v1 compatibility boundary and enumerate active readers and writers - [ ] 1.1 Freeze historical v1 artifacts and the reviewed migration inventory
- Pin representative v1 trace and report bytes as immutable regression fixtures. - Pin representative v1 trace and report bytes and hashes as immutable regression fixtures.
- Record every current import and artifact-writing path that still depends on the authorization-shaped contract. - Provide an independent schema-routed read-only decoder that validates committed v1 artifacts without importing the compatibility namespace, constructing v2 determinate outcomes, or exposing a writer or converter.
- Preserve existing v1 behavior as migration input without treating it as v2 conformance. - Freeze the reviewed initial set of v1 readers and writers with stable entry identities, paths, symbols or capabilities, owner boundaries, and open migration status.
- Done when fixture tests detect any byte change and the import/writer census identifies every cutover target. - Preserve every inventory identity permanently and allow only monotone transitions from open to `v2_migrated` or `retired`; fail unreadable or zero-byte sources, additions, deletions, and rekeying.
- Preserve v1 as migration input only and never treat it as v2 conformance or a comparable trend baseline.
- Done when fixture tests detect any byte or hash change, the decoder validates both historical schemas read-only, every inventory entry resolves to a nonzero source and valid target, and schema, status, or addition mutations fail.
- _Requirements: 14.3, 15.3_ - _Requirements: 14.3, 15.3_
- _Blocked: debug attempted twice, still failing — a bounded AST census with exact provenance registries still omits active hunter and Semgrep scorer-record writer families, so “every cutover target” is not yet mechanically demonstrated._
- [x] 1.2 Build the strict canonical codec and resolver-backed artifact boundary - [x] 1.2 Build the strict canonical codec and resolver-backed artifact boundary
- Produce deterministic UTF-8 JSON with full SHA-256 identities, stable mapping order, and retained sequence order. - Produce deterministic UTF-8 JSON with full SHA-256 identities, stable mapping order, and retained sequence order.
- Refuse non-finite numbers, unsupported values, unknown schemas, duplicate logical identities, and malformed digests. - Refuse non-finite numbers, unsupported values, unknown schemas, duplicate logical identities, and malformed digests.
@@ -334,7 +335,6 @@
- _Requirements: 10.2, 10.4, 11.1, 11.2, 11.3, 11.4, 11.5, 13.4, 14.3_ - _Requirements: 10.2, 10.4, 11.1, 11.2, 11.3, 11.4, 11.5, 13.4, 14.3_
- _Boundary: Fact Pack Adapter Conformance_ - _Boundary: Fact Pack Adapter Conformance_
- _Depends: 5.4_ - _Depends: 5.4_
- _Blocked: consume the handoff only after access-control-prover tasks 14.1–14.8 produce the downstream artifacts and immutable conformance manifest_
- [ ] 9.5 (P) Produce readiness and development records through benchmark-owned adapters - [ ] 9.5 (P) Produce readiness and development records through benchmark-owned adapters
- Construct canonical readiness results while leaving sequencing and persistence with benchmark workflows. - Construct canonical readiness results while leaving sequencing and persistence with benchmark workflows.
- Emit development observations with v2 identities, evidence-use roles, instrument receipts, and prerequisite ancestry. - Emit development observations with v2 identities, evidence-use roles, instrument receipts, and prerequisite ancestry.
@@ -354,21 +354,60 @@
- _Depends: 8.9, 9.5_ - _Depends: 8.9, 9.5_
- [ ] 10. Complete the single-writer v2 migration - [ ] 10. Complete the single-writer v2 migration
- [ ] 10.1 Prepare schema-routed legacy reads and cutover tests without publishing v2 - [ ] 10.1 Isolate every live v1 contract and writer behind the closed compatibility namespace
- Add an internal read-only decoder that validates committed v1 traces and reports without constructing v2 determinate outcomes automatically. - Move all authorization-shaped v1 values into one compatibility contract owner and all live v1 serialization into its sole writer facade.
- Route v1/v2 test reads by explicit schema version and prohibit all new writes through the legacy surface. - Limit the compatibility namespace to the canonical primitive needed for byte stability and prevent every v2 module from importing it.
- Keep the top-level public API unchanged until the import and writer census reaches zero. - Preserve the frozen v1 wire representation while eliminating independent contract or serializer implementations elsewhere.
- Done when historical artifacts validate byte-unchanged, legacy write attempts fail, and cutover tests pass without publishing top-level v2 exports. - Done when old writes remain byte-identical, the compatibility namespace is the only live v1 contract and write owner, and ownership/import tests refuse any duplicate implementation.
- _Requirements: 14.3, 15.3_ - _Requirements: 14.3, 15.3_
- [ ] 10.2 Stop v1 writers and publish the single v2 public surface - _Depends: 1.1_
- Re-run repository import and artifact-writer censuses after all adjacent adapters use v2. - [ ] 10.2 Convert the temporary public and module compatibility surfaces into explicit shims
- Stop v1 writes and remove the compatibility shim, authorization-shaped public exports, and family-domain conversion only when both censuses reach zero. - Keep the old named top-level API and designated compatibility modules as explicit imports from the isolated namespace during migration.
- Publish dependency-free v2 contracts through the top-level core API after the zero-census gate. - Remove legacy definitions and re-exports from mixed v2 modules so each compatibility name has exactly one owner.
- Retain the read-only legacy decoder and refuse comparison between v1 and v2 baseline identities. - Keep the canonical v2 paths private until the final atomic cutover.
- Done when no production writer/import targets the v1 contract, historical fixtures remain unchanged, and all public writes use v2. - Done when the old named API remains compatible before cutover, direct imports of moved legacy symbols from mixed modules fail, and no shim owns an independent implementation.
- _Requirements: 14.3, 15.3_ - _Requirements: 14.3, 15.3_
- _Depends: 9.4_ - [ ] 10.3 Build the syntax-only migration-boundary instrument
- _Blocked: access-control-prover task 14.9 must migrate the production caller and prove a zero-bypass census after task 9.4 accepts the handoff_ - Prove every scoped source file is readable and nonzero before interpreting a zero result.
- Report explicit compatibility-namespace imports, designated shim imports, named top-level v1 symbols, writer-facade references, and module-valued or dynamic core access without evaluating alias or data flow.
- Validate immutable inventory identities and their monotone open, `v2_migrated`, or `retired` status transitions.
- During migration, distinguish the designated compatibility implementation from the production-consumer scope; reserve repository-wide zero for post-deletion validation.
- Done when an adversarial syntax matrix reports the exact forbidden sites and `try`/`finally`, comprehension, assignment, and generic-call alias shapes cannot turn an explicit forbidden reference into a false zero.
- _Requirements: 14.3, 15.3_
- [ ] 10.4 Migrate or retire remaining registered production source-tree consumers
- Move each registered non-benchmark production reader and writer to the canonical v2 contract or retire it within its owning boundary.
- Update the matching stable inventory entry to `v2_migrated` or `retired` in the same change without removing or rekeying it.
- Keep authorization analysis, model execution, proof execution, and other adjacent behavior in their existing owner boundaries while changing only their contract seam.
- Done when every source-tree entry owned by core or an adjacent adapter is terminal and the verified nonzero production-consumer scope outside the designated compatibility implementation has zero forbidden references.
- _Requirements: 14.1, 14.3, 15.3_
- _Boundary: Core and Adjacent Adapter Migration_
- _Depends: 10.3_
- [ ] 10.5 Migrate benchmark readiness and admission consumers
- Move every registered benchmark reader and writer to v2 while retaining readiness sequencing, persistence, population custody, and admission orchestration in the benchmark boundary.
- Update each matching stable inventory entry to `v2_migrated` or `retired` without changing its identity.
- Preserve guarded spend, authenticated case release, fixed measurement, and bounded model behavior through the migration.
- Done when benchmark production scope has zero forbidden consumer references outside the designated compatibility implementation, every benchmark entry is terminal, and synthetic v2 readiness and admission complete end to end.
- _Requirements: 8.1, 12.6, 14.3, 15.3_
- _Boundary: Benchmark Readiness and Admission Adapters_
- _Depends: 9.5, 9.6, 10.3_
- [ ] 10.6 Migrate or retire registered test and operational-tool consumers
- Move authorization assumptions into legacy or adapter tests, keep family-neutral contract tests on v2, and retain frozen v1 bytes only for decoder regression.
- Move or retire registered operational-tool consumers without adding a second compatibility or write path.
- Mark the corresponding stable inventory entries `v2_migrated` or `retired` and require every original entry to reach a terminal status.
- Done when the complete original inventory is terminal and the boundary instrument proves a nonzero repository scope with no forbidden consumer reference outside the designated compatibility implementation.
- _Requirements: 14.3, 15.3_
- _Boundary: Migration Tests and Operational Tooling_
- _Depends: 10.4, 10.5_
- [ ] 10.7 Delete v1 write capability and publish the single v2 surface atomically
- Require every inventory entry to be terminal and the migration-time production-consumer boundary to be zero before deletion begins.
- Delete the compatibility namespace, v1 writer facade, and temporary shims while publishing the canonical v2 modules and top-level exports in the same integration change.
- Prove old top-level and submodule imports and writes fail and that no v1 serializer or converter remains.
- Retain the independent read-only decoder for frozen historical bytes and refuse comparison between v1 and v2 baseline identities.
- After deletion, require repository-wide zero with no compatibility exclusion.
- Done when isolated-tree and repository tests show old imports and writes fail, v2 imports and writes succeed, historical bytes still validate, and the final nonzero boundary scan remains zero.
- _Requirements: 14.3, 15.3_
- _Boundary: Single-Writer Cutover Integration_
- _Depends: 10.6_
- [ ] 11. Validate readiness, failure visibility, and implementation boundaries - [ ] 11. Validate readiness, failure visibility, and implementation boundaries
- [ ] 11.1 Run both reference families through the complete synthetic readiness path - [ ] 11.1 Run both reference families through the complete synthetic readiness path
@@ -397,8 +436,9 @@
- Prove each test instrument read nonzero fixture bytes before interpreting zero findings or failures. - Prove each test instrument read nonzero fixture bytes before interpreting zero findings or failures.
- Done when the complete suite passes, boundary/import censuses are clean, historical bytes are unchanged, and all declared adapter and consumer seams pass conformance. - Done when the complete suite passes, boundary/import censuses are clean, historical bytes are unchanged, and all declared adapter and consumer seams pass conformance.
- _Requirements: 1.4, 10.4, 14.3, 15.3, 15.4_ - _Requirements: 1.4, 10.4, 14.3, 15.3, 15.4_
- _Depends: 9.4_ - _Depends: 10.7_
## Implementation Notes ## Implementation Notes
- Task 1.1: repository-wide traversal alone is insufficient for a complete writer census; active writers require bounded provenance across direct schema construction, local loader/producer flow, imported producers, and scorer-envelope aggregation. Two debug rounds closed loader and imported-prover gaps, but hunter/Semgrep scorer writers remain unresolved. - The frozen migration inventory is a reviewed worklist with stable identities, not a semantic-use census. It may only close entries as `v2_migrated` or `retired`; it never grows, deletes entries, or proves absence by interpreting Python flow.
- Cutover is proved by explicit syntax checks plus deletion of the compatibility and v1 write capability. No task extends the discarded alias/data-flow interpreter.