mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-11-13 01:48:17 +00:00
docs(detection-core): add authoritative admission replay
This commit is contained in:
@@ -231,7 +231,8 @@ src/openultrasast/core/
|
||||
├── measurement.py # Complete baseline identity, rates, funnel and report schema v2
|
||||
├── admission_case.py # Case results and authenticated post-spend release bindings
|
||||
├── admission_metrics.py # Evidence-derived scoped and whole-population metric algebra
|
||||
├── admission.py # Census, feasibility, report assembly and fitted-component gate
|
||||
├── admission.py # Census, feasibility and fitted-component gate
|
||||
├── admission_report.py # Evidence-backed gates, report assembly and replay validation
|
||||
├── consumer.py # Scoped push request and verdict consumption
|
||||
└── legacy.py # Read-only v1 trace and report decoder and validator
|
||||
|
||||
@@ -1747,6 +1748,9 @@ authorization; the derived result digests must exactly equal the fitted-componen
|
||||
membership. It re-derives each case result and its operational projection, requires an
|
||||
exact one-to-one binding, and refuses missing, extra, duplicate, substituted, swapped, or
|
||||
cross-scope cases before producing any metric.
|
||||
- Scoring returns an immutable `AdmissionReplay`, not a bare report. The replay carries the
|
||||
exact authoritative objects needed to reproduce the report. A raw `AdmissionReportV2` is
|
||||
serializable result data, never an authorization token.
|
||||
- Admission authorization produces a custody-head-bound label-release request. The
|
||||
benchmark custodian must durably spend the population before releasing label bytes;
|
||||
scoring revalidates that exact current-chain receipt and never emits a post-scoring spend
|
||||
@@ -1757,8 +1761,9 @@ authorization; the derived result digests must exactly equal the fitted-componen
|
||||
`measurement.py` owns `ScopeKey`, baseline identity, the closed metric algebra, interval and
|
||||
gate results, readiness references, `MeasurementRecordV2`, and
|
||||
`RepositoryHoldoutReport`. `admission.py` imports those types and owns census, metric rules,
|
||||
thresholds, manifests, case inputs, authorization, scoring, admission reports, and fitted
|
||||
component authorization. Measurement never imports admission.
|
||||
thresholds, manifests, case inputs, authorization, and fitted-component authorization.
|
||||
`admission_report.py` owns gate derivation, report assembly, the authoritative replay bundle,
|
||||
and replay validation. Measurement never imports admission modules.
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
@@ -2031,6 +2036,18 @@ class AdmissionReportV2:
|
||||
decision: AdmissionDecision
|
||||
report_digest: str
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AdmissionReplay:
|
||||
manifest: AdmissionManifestV2
|
||||
denominator: DenominatorPolicy
|
||||
readiness: tuple[ReadinessResult, ...]
|
||||
preparation: AdmissionPreparation
|
||||
label_release_receipt: PopulationUseReceipt
|
||||
case_release: AdmissionCaseRelease
|
||||
cases: tuple[AdmissionCaseResult, ...]
|
||||
report: AdmissionReportV2
|
||||
replay_digest: str
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AdmissionAuthorization:
|
||||
manifest_digest: str
|
||||
@@ -2056,6 +2073,8 @@ def authorize_admission(
|
||||
|
||||
def score_admission(
|
||||
manifest: AdmissionManifestV2,
|
||||
denominator: DenominatorPolicy,
|
||||
readiness: Sequence[ReadinessResult],
|
||||
preparation: AdmissionPreparation,
|
||||
label_release_receipt: PopulationUseReceipt,
|
||||
case_release: AdmissionCaseRelease,
|
||||
@@ -2063,6 +2082,13 @@ def score_admission(
|
||||
context: AdmissionValidationContext,
|
||||
custody: PopulationCustodyPort,
|
||||
resolver: ContentResolver,
|
||||
) -> AdmissionReplay: ...
|
||||
|
||||
def validate_admission_replay(
|
||||
replay: AdmissionReplay,
|
||||
context: AdmissionValidationContext,
|
||||
custody: PopulationCustodyPort,
|
||||
resolver: ContentResolver,
|
||||
) -> AdmissionReportV2: ...
|
||||
|
||||
def authorize_fitted_component(
|
||||
@@ -2071,7 +2097,7 @@ def authorize_fitted_component(
|
||||
context: AdmissionValidationContext,
|
||||
fitting_lineage: LineageManifest,
|
||||
repository_holdout: RepositoryHoldoutReport,
|
||||
admission: AdmissionReportV2,
|
||||
admission: AdmissionReplay,
|
||||
custody: PopulationCustodyPort,
|
||||
resolver: ContentResolver,
|
||||
) -> None: ...
|
||||
@@ -2151,6 +2177,15 @@ gate is produced. The final report binds the validation context, authorization,
|
||||
pre-label spend receipt, and authenticated post-spend case release; there is no
|
||||
post-scoring population-use request.
|
||||
|
||||
`score_admission` freezes the denominator, readiness, and cases exactly once, derives the
|
||||
report, and returns them with every other authoritative scoring input in an
|
||||
`AdmissionReplay`. Its `replay_digest` protects transport integrity of the bundle but is not
|
||||
a validation token. `validate_admission_replay` revalidates every carried object against an
|
||||
independently supplied current context, custody port, and content resolver, re-runs report
|
||||
derivation, and requires exact report and replay equality. A caller cannot authorize from a
|
||||
raw report or from a coordinated set of rehashed report, threshold, readiness, or case
|
||||
objects.
|
||||
|
||||
The report validates case counts back to the census, reports paired vulnerable/fixed
|
||||
behavior and ordinary false alarms through registered metrics, imports semantic-stage and
|
||||
model-call outcomes, and stratifies findings by evidence basis so model-corroborated and
|
||||
@@ -2171,7 +2206,9 @@ disjoint, matches component/training/lineage/report digests, validates current c
|
||||
the fitting lineage with subject `component_digest` and a required `FITTED_MODEL`
|
||||
expectation for that digest, and requires
|
||||
the admission report to be a `GO` produced from a population that was untouched before its
|
||||
bound label-release spend. It re-derives `context.context_digest`, requires it to equal the
|
||||
bound label-release spend. It first calls `validate_admission_replay`; a bare
|
||||
`AdmissionReportV2` or a replay whose carried inputs no longer validate cannot authorize a
|
||||
component. It re-derives `context.context_digest`, requires it to equal the validated
|
||||
report's validation-context digest, and requires exactly one baseline-listed declaration in
|
||||
`context.evidence.model_programs` whose `fitted_component_digest` equals
|
||||
`evidence.component_digest`. The selected declaration supplies the family-contract digest;
|
||||
@@ -2188,9 +2225,15 @@ authorize the component.
|
||||
Every admission report must contain `PAIRED_CORRECTNESS`, `VULNERABLE_RECALL`,
|
||||
`FIXED_FALSE_ALARM_RATE`, `ORDINARY_FALSE_ALARMS_PER_100`, `PRECISION`, and
|
||||
`SEMANTIC_STAGE_COVERAGE` with nonzero pre-registered denominators, plus model success when
|
||||
applicable. It must also contain evidence-backed instrument, budget, readiness, and quality
|
||||
gate results. `GO` is invalid if any mandatory metric or gate is absent, any instrument
|
||||
receipt failed, or any required gate did not pass.
|
||||
applicable. Model success is applicable when the frozen baseline configures at least one
|
||||
model program **or** validated claimed-scope cases contain declared model calls. An
|
||||
applicable model with zero attempted calls, no model-success threshold, or any failed model
|
||||
outcome forces `NO_GO`; calls wholly outside the claimed scopes do not make the model
|
||||
applicable. Claimed-scope model calls without a baseline-listed declaration are invalid
|
||||
evidence and are refused before report production. The report must also contain
|
||||
evidence-backed instrument, budget, readiness, and quality gate results. `GO` is invalid if
|
||||
any mandatory metric or gate is absent, any instrument receipt failed, or any required gate
|
||||
did not pass.
|
||||
|
||||
### Decision and Consumer Edge
|
||||
|
||||
@@ -2404,7 +2447,8 @@ dependencies into core.
|
||||
| `BaselineIdentity` | Arm, population/freeze, claimed scopes, contract set, fact schema/set, acquisition, analyzer, configuration, model set, denominator, threshold set and outcome mapping | Any changed field creates a new baseline |
|
||||
| `AdmissionPreparation` | Manifest/context authorization and expected custody head | Feasible pre-registration; label-release request spends before bytes open |
|
||||
| `AdmissionCaseRelease` | Population, manifest, authorization, preparation, spend receipt, custodian and attested case bindings | Post-spend only; exact one-to-one case/result/evidence/change/input binding; scoped roles and pairs |
|
||||
| `AdmissionReportV2` | Baseline, validation context, authorization, spend receipt, case release and case/result digests | Canonical input revalidation; in-scope scoring; all gates reported |
|
||||
| `AdmissionReportV2` | Baseline, validation context, authorization, spend receipt, case release and case/result digests | Canonical result data; in-scope scoring; all gates reported; never an authorization token by itself |
|
||||
| `AdmissionReplay` | Canonical digest of the exact authoritative scoring inputs and derived report | Immutable transport bundle; independently replayed against current context, custody and content before trust |
|
||||
| `RepositoryHoldoutReport` | Component, disjoint repository manifests and measurement | Explicit holdout role; matches fitting lineage and component identity |
|
||||
| `DifferentialProofRecord` | Reviewed producer, control-bound sides and resolved witness/verification receipt | Affected change identity, reviewed safe control, trusted producer/store policy |
|
||||
| `PushVerdict` | Request and record digests | No unbounded clean result; proof basis does not imply rollout permission |
|
||||
@@ -2513,6 +2557,11 @@ spec and consumes only status, counts, and digests.
|
||||
bindings, cross-scope pairs, role swaps, result swaps, and rehashed evidence/change/input
|
||||
substitutions before metric production
|
||||
(`12.1–12.6`, `13.1`, `13.4`).
|
||||
- `test_core_admission_report.py`: evidence-backed gate derivation, exact report identity,
|
||||
`AdmissionReplay` construction and authoritative revalidation, configured-model zero-call
|
||||
refusal, claimed-scope undeclared-model refusal, out-of-scope-only model-call isolation,
|
||||
and coordinated report/readiness/threshold/case substitution refusal. Raw reports cannot
|
||||
authorize later consumers (`12.6`, `13.1–13.2`, `15.1–15.5`).
|
||||
- `test_core_measurement.py`: required schema, basis stratification, Wilson counts, baseline
|
||||
mutation matrix, zero receipts, and prerequisite stage result/digest references. A
|
||||
two-arm case over one population keeps family/ecosystem scope, census rows, denominator
|
||||
|
||||
@@ -564,3 +564,37 @@ requirements approval.
|
||||
- Benchmark-owned guarded custody must produce the release after the winning spend and
|
||||
prove crash/concurrency behavior at that external boundary.
|
||||
- The pre-label census and feasibility contracts remain unchanged and label-blind.
|
||||
|
||||
---
|
||||
|
||||
## Authoritative admission replay closure (2026-11-01)
|
||||
|
||||
### Scope of this pass
|
||||
|
||||
- **Context:** task 8.9 showed that a structurally valid, self-consistent report cannot be
|
||||
authenticated later without the authoritative manifest, denominator, readiness,
|
||||
preparation, spend receipt, case release, and case results from which it was derived.
|
||||
- **Finding:** contextual report validation inside the scoring call fixed local gate forgery
|
||||
but did not give task 8.10 a safe consumer contract. Accepting a bare report would either
|
||||
trust its digest as a capability or require every consumer to recover unbound inputs.
|
||||
- **Classification:** specification conflict at the 8.9/8.10 boundary, not an isolated code
|
||||
defect. The report is result data; later authorization needs replayable evidence.
|
||||
|
||||
### Decision: carry and independently replay authoritative inputs
|
||||
|
||||
- **Alternatives considered:** treat `report_digest` as an authorization token; add more
|
||||
nested evidence to the report; make every fitted-component caller pass the original inputs
|
||||
separately; or return one immutable replay bundle from scoring.
|
||||
- **Selected approach:** `score_admission` returns `AdmissionReplay`, containing the exact
|
||||
frozen authoritative inputs and the derived report. `validate_admission_replay` accepts
|
||||
only that bundle plus independently supplied current context, custody, and content
|
||||
resolution, re-runs the full derivation, and compares exact output.
|
||||
- **Rationale:** the bundle prevents accidental input loss without promoting a digest or
|
||||
caller-provided aggregate into a capability. Current external trust state remains an
|
||||
explicit validator input.
|
||||
- **Model applicability closure:** model success is mandatory when a model is frozen in the
|
||||
baseline or validated claimed-scope calls occurred. Configured zero-call, missing model
|
||||
threshold, or failed declared calls force `NO_GO`; undeclared claimed-scope calls are
|
||||
refused before report production, and out-of-scope-only calls do not alter the arm's gate.
|
||||
- **Boundary:** replay production remains part of task 8.9. Task 8.10 consumes and validates
|
||||
the replay before fitted-component checks and refuses raw reports.
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"feature_name": "detection-core",
|
||||
"created_at": "2026-10-30T13:55:30Z",
|
||||
"updated_at": "2026-11-01T09:03:42Z",
|
||||
"updated_at": "2026-11-01T10:15:05Z",
|
||||
"language": "en",
|
||||
"phase": "tasks-generated",
|
||||
"approvals": {
|
||||
|
||||
@@ -283,11 +283,15 @@
|
||||
- [ ] 8.9 Assemble evidence-backed gates and the final admission report
|
||||
- Emit instrument, budget, readiness, and quality gates with reasons and supporting identities.
|
||||
- Bind the report to the validation context, authorization, pre-label spend receipt, authenticated case release, census, and complete case-result set.
|
||||
- Require every mandatory metric and gate, plus model success when applicable, before allowing GO.
|
||||
- Require every mandatory metric and gate before allowing GO; make model success mandatory when a model is baseline-configured or validated claimed-scope calls occur. Refuse undeclared claimed-scope calls before report production; make configured zero-call, missing-threshold, and failed declared-call cases NO-GO while ignoring out-of-scope-only calls.
|
||||
- Return an immutable admission replay bundle carrying the exact manifest, denominator, readiness, preparation, spend receipt, case release, cases, and derived report; give the bundle a canonical transport digest without treating it as a validation token.
|
||||
- Revalidate the replay against independently supplied current context, custody, and content resolution by re-running the complete derivation and requiring exact report and replay equality.
|
||||
- Keep model-corroborated, deterministic, and execution-backed counts separate.
|
||||
- Done when any missing/failed mandatory gate forces NO-GO and a complete passing synthetic fixture yields a reproducible GO report.
|
||||
- Done when any missing/failed mandatory gate forces NO-GO, coordinated report/readiness/threshold/case substitutions are refused, and a complete passing synthetic fixture yields a reproducible replayed GO report.
|
||||
- _Requirements: 12.6, 15.1, 15.2, 15.4, 15.5_
|
||||
- _Depends: 8.8_
|
||||
- [ ] 8.10 Authorize fitted components only from replayed probes and disjoint evaluation
|
||||
- Accept only the typed admission replay bundle from task 8.9, validate it first against current authoritative context/custody/content, and refuse raw admission reports or coordinated replay substitutions.
|
||||
- Resolve the component’s unique model declaration, family contract, review policy, and custody policy from the admission-bound context.
|
||||
- Re-evaluate every canonical probe input and require the derived result set to match the component evidence.
|
||||
- Consume the typed repository-holdout report from task 8.2 and validate its evidence-use role, training/holdout references, report identity, and fitting lineage.
|
||||
@@ -295,6 +299,7 @@
|
||||
- Require an admission GO that was untouched before its bound label-release spend.
|
||||
- Done when fabricated summaries, caller-selected policy substitutes, overlapping repositories, contaminated lineage, or mismatched admission are refused.
|
||||
- _Requirements: 13.1, 13.2, 13.4_
|
||||
- _Depends: 8.2, 8.9_
|
||||
|
||||
- [ ] 9. Integrate adjacent producers and benchmark-owned orchestration
|
||||
- [ ] 9.1 (P) Translate the parked authorization trace conservatively into v2 evidence
|
||||
|
||||
Reference in New Issue
Block a user