mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-11-14 10:18:17 +00:00
docs(specs): approve scoped evaluation admission design
This commit is contained in:
Marius Ciepluch
File diff suppressed because it is too large
Load Diff
@@ -1,89 +1,158 @@
|
||||
# Requirements: valid detection benchmark
|
||||
# Requirements Document
|
||||
|
||||
Turns `brief.md` into testable requirements. The brief supersedes the retired SZZ pool
|
||||
(`unseen-repo-evaluation`). This spec builds a benchmark with sound ground truth and standard
|
||||
security metrics.
|
||||
## Introduction
|
||||
|
||||
## Scope decisions (resolving the brief's open questions)
|
||||
This specification defines the valid detection benchmark used to determine whether a detector works on application repositories it has not seen. It replaces the 2026-10-12 pilot as admission evidence because that pilot reused repositories protected by the permanent reservation policy. The historical pilot remains preserved as withdrawn evidence, while a separately sourced population establishes a new baseline.
|
||||
|
||||
- **First-slice size.** Build a pilot of 10 to 12 verified positives end to end first, to prove the
|
||||
pipeline, then grow to a target of about 40 verified positives. 40 gives a recall interval of roughly
|
||||
±0.10 at a true recall near 0.85 to 0.90; `score.py:required_n(p, width)` computes the exact number,
|
||||
so the target is set from that, not guessed.
|
||||
- **Negative-to-positive ratio.** At least 20 ordinary commits per positive, drawn from the same repos.
|
||||
Negatives are free to collect, and false alarms decide adoption, so err large. Report false alarms as
|
||||
a rate per 100 pushes.
|
||||
- **Languages first.** Python and JavaScript/TypeScript web applications (Flask, Django, FastAPI,
|
||||
Express, Koa). These are where the tool is strongest and where in-repo entry points are clear. PHP and
|
||||
others come later, named explicitly as not yet covered.
|
||||
- **Curation record.** Each case stores the advisory, repo, fix commit, parent, path, function, the
|
||||
pre-fix and post-fix excerpts, the LLM verdict with confidence, and the human spot-check decision with
|
||||
reviewer and date. The set is frozen with a digest, the same way `advisory-fixes/` freezes its pairs.
|
||||
- **Lane.** Curate and score locally on the Docker lane (research is local execution). The kube `ax`
|
||||
lane stays the scale target but is not required here and is still blocked on the gVisor checkpoint path.
|
||||
The replacement is a general Python and JavaScript/TypeScript benchmark rather than an access-control-only population. It measures multiple vulnerability families and reports quality for each declared vulnerability-family and ecosystem scope. A whole-population result describes coverage only; it cannot substitute for a scoped quality denominator.
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: population sourcing and review, fix-derived labels, ordinary-change selection, permanent reservation, development/admission separation, label isolation, lifecycle and custody, reproducible freezing, push-path replay, scoped denominators, standard metrics, failure visibility, and benchmark reporting.
|
||||
- **Out of scope**: adding or tuning detection logic, facts, prompts, thresholds, or judges; wiring detector arms into the pre-push product; enabling rollout; reactivating a parked detector arm; and operating execution-plane infrastructure.
|
||||
- **Adjacent expectations**: detector arms provide immutable declared identities and claimed scopes; the shared detection contracts validate admission and scoring; the proof workflow supplies demonstrated-exploit evidence; product specifications decide rollout from eligible scoped results.
|
||||
|
||||
## Requirements
|
||||
|
||||
### R1. Positives labelled from the fix, never from SZZ
|
||||
- Take the vulnerability location from the security fix: the file and function the fix changed, read at
|
||||
the parent commit (pre-fix).
|
||||
- Do not run SZZ blame walk-back. The sourcing must not use `extract.py`'s `introducing` or `last_touch`
|
||||
methods. Use the fix site directly (`fix_sites()`) and first-parent history only.
|
||||
- Each positive is a pair: the vulnerable function at `parent`, and the same function at the fix commit.
|
||||
### Requirement 1: General Multi-Family Population
|
||||
|
||||
### R2. App-shaped and reachable in-repo
|
||||
- Keep a positive only if the vulnerability is reachable from an entry point inside the repo: a web
|
||||
route, a request handler, or a CLI argument.
|
||||
- Drop library-internal cases whose trigger depends on a caller outside the repo. A pre-push tool cannot
|
||||
see that trust boundary, so scoring it there is unfair.
|
||||
**Objective:** As a maintainer, I want one general benchmark spanning the initial supported languages and multiple vulnerability families, so that shared detection improvements and class-specific arms can be evaluated without building a new population for every arm.
|
||||
|
||||
### R3. Every positive is verified
|
||||
- Verify each case with an LLM adjudication call and a human spot-check. The call must confirm:
|
||||
(a) the pre-fix code is vulnerable, (b) the fix addresses it, (c) it is reachable from an in-repo entry
|
||||
point.
|
||||
- Record the verdict, the confidence, and the human decision. A case that fails any of (a), (b), (c), or
|
||||
the human spot-check does not enter the benchmark.
|
||||
#### Acceptance Criteria
|
||||
|
||||
### R4. Negatives are ordinary commits from the same repos
|
||||
- Draw non-fix commits from the first-parent history of the same application repos, excluding the fix.
|
||||
- The negative set is at least 20x the positive set.
|
||||
1. The Valid Detection Benchmark shall include accepted application cases from both Python and JavaScript/TypeScript ecosystems.
|
||||
2. The Valid Detection Benchmark shall include accepted positives from at least two distinct vulnerability families overall.
|
||||
3. When the full replacement population is accepted, the Valid Detection Benchmark shall contain at least 40 accepted positives overall.
|
||||
4. If a frozen population contains fewer than 40 accepted positives, the Valid Detection Benchmark shall classify it as a development pilot or insufficient population and shall not use it as the full replacement admission baseline.
|
||||
5. When the benchmark records an ecosystem, the Valid Detection Benchmark shall use one reviewed canonical ecosystem identity for that repository.
|
||||
6. If a repository's ecosystem classification remains unresolved, the Valid Detection Benchmark shall record insufficient scope evidence for that repository.
|
||||
7. If a vulnerability-family and ecosystem scope lacks enough eligible cases for its pre-registered gate, the Valid Detection Benchmark shall report that scope as insufficient and shall not infer support from the whole-population count.
|
||||
8. When concrete frameworks or repositories are cited, the Valid Detection Benchmark shall treat them as coverage evidence rather than as detector implementation requirements.
|
||||
|
||||
### R5. Held out
|
||||
- No repository in the benchmark was used to build, train, tune, mine, or harvest the tool. Enforce this
|
||||
with the existing used-set canonicalization (`eligibility.normalize`), the same guard the SZZ pool used.
|
||||
### Requirement 2: Fix-Derived Positive Ground Truth
|
||||
|
||||
### R6. Measure the push path on unseen repos, with the LLM tier
|
||||
- The tool a developer runs at push time must include the LLM detection tier (the hunter). That is a
|
||||
product requirement on the pre-push hook, owned by `pre-push-safety-net`. The benchmark measures the
|
||||
push path, not a scan mode.
|
||||
- Today the shipped hook runs quick rules and the engine only. Its model path (`push/assistance.py`)
|
||||
selects witnesses for already-admitted defects and cannot add a finding. Until the hunter is wired into
|
||||
the hook, the benchmark reports two arms over the same changes and labels them:
|
||||
- (a) the push hook as shipped (quick rules + engine), through the replay harness;
|
||||
- (b) the push hook plus the hunter arm, the hunter run off-cluster over the same changes and the
|
||||
records combined per change.
|
||||
(b) is the number the tool is held to. (a) is what ships today. The gap between them is product work,
|
||||
not a benchmark artefact, and is reported as such.
|
||||
- Use the existing replay harness for (a) so the measured path is the shipped push hook.
|
||||
**Objective:** As a benchmark reviewer, I want each positive anchored to a security fix and reachable application behavior, so that recall is measured against reviewed vulnerability locations rather than inferred history.
|
||||
|
||||
### R7. Standard metrics
|
||||
- Report recall (true-positive rate), precision (positive predictive value), false-positive rate as
|
||||
false alarms per 100 pushes, and exploit-verified rate (the share of flagged positives the proof arm
|
||||
demonstrates with a working proof of concept).
|
||||
- Report recall with a Wilson 95% interval and a repo-clustered interval, reusing `score.py`.
|
||||
- The headline pairs recall and false alarms per 100 pushes in one plain sentence.
|
||||
#### Acceptance Criteria
|
||||
|
||||
### R8. Reproducible and frozen
|
||||
- The positive catalog, the negative draw, and the held-out guarantee are frozen with digests so a run is
|
||||
reproducible. Re-running the freeze on an unchanged source yields the same digest.
|
||||
1. When a positive candidate is sourced, the Valid Detection Benchmark shall derive its vulnerable location from the file and function changed by the security fix at the vulnerable parent revision.
|
||||
2. The Valid Detection Benchmark shall not derive a positive location from a blame walk-back or another SZZ-style introducing-commit estimate.
|
||||
3. When a positive candidate is reviewed, the Valid Detection Benchmark shall require a human reviewer whose recorded identity differs from the producer's identity, satisfies the pre-registered review policy, and includes a conflict disclosure to confirm that the vulnerable revision contains the defect, the fixed revision addresses it, and an entry point inside the repository can reach it.
|
||||
4. If any required ground-truth or reachability judgment is rejected or missing, the Valid Detection Benchmark shall exclude the candidate and record a reason.
|
||||
5. Where model-assisted adjudication is used during curation, the Valid Detection Benchmark shall record it as non-authoritative review assistance and shall not treat model agreement as a label.
|
||||
6. When one fix has independently verified labels in multiple scopes, the Valid Detection Benchmark shall count the push once in whole-population coverage and once in each verified scoped denominator without granting credit to an unverified family or ecosystem.
|
||||
|
||||
## Non-goals
|
||||
- No SZZ, no frozen-pool catch/false-alarm numbers (retired).
|
||||
- No new detection capability in this spec. This measures the tool that exists.
|
||||
- Not required to run on kube `ax` to be valid; the Docker lane is enough for the first numbers.
|
||||
### Requirement 3: Representative Ordinary Changes
|
||||
|
||||
## Acceptance
|
||||
- A frozen pilot of at least 10 verified positives plus their negatives, on held-out Python/JS repos,
|
||||
run through the whole tool, producing recall, precision, false alarms per 100 pushes, and
|
||||
exploit-verified rate, with intervals. Every positive has an LLM verdict and a human spot-check on
|
||||
record.
|
||||
**Objective:** As a developer, I want false alarms measured on ordinary changes from the same application environments, so that the reported interruption rate reflects the push experience.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When an accepted positive repository contributes negatives, the Valid Detection Benchmark shall draw ordinary non-security changes from that repository's first-parent history and shall exclude the security fix.
|
||||
2. The Valid Detection Benchmark shall include at least 20 eligible ordinary pushes per accepted positive in each frozen population.
|
||||
3. When calculating a scoped false-alarm denominator, the Valid Detection Benchmark shall include each eligible ordinary push from the matching ecosystem once for each vulnerability family claimed by the evaluated arm in that ecosystem.
|
||||
4. The Valid Detection Benchmark shall not pool an ordinary push from one ecosystem into another ecosystem's denominator.
|
||||
5. If an ordinary change is verified to contain a relevant security defect before freeze, the Valid Detection Benchmark shall exclude it from the population and record the reason.
|
||||
6. If an ordinary change is verified to contain a relevant security defect after freeze, the Valid Detection Benchmark shall retire the affected baseline rather than silently rewriting its recorded result.
|
||||
|
||||
### Requirement 4: Permanent Reservation and Historical Withdrawal
|
||||
|
||||
**Objective:** As a benchmark custodian, I want prior repository use and withdrawn populations handled explicitly, so that development knowledge cannot leak into new admission evidence.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The Valid Detection Benchmark shall exclude every repository identity present in the authoritative permanent-reservation census.
|
||||
2. When repository aliases, former names, redirects, or fork relationships resolve to a reserved identity, the Valid Detection Benchmark shall treat the candidate as reserved.
|
||||
3. When repository code, labels, or metadata influence population selection, detector development, training, tuning, mining, fact promotion, prompt selection, or threshold selection, the Valid Detection Benchmark shall add that repository identity to the permanent-reservation census.
|
||||
4. If a repository was only enumerated and none of its code, labels, or metadata influenced a decision, the Valid Detection Benchmark shall not classify it as used unless another reservation source includes it.
|
||||
5. The Valid Detection Benchmark shall classify the 2026-10-12 pilot as withdrawn and non-admissible and shall not present its results as current qualification evidence.
|
||||
6. When the historical pilot is withdrawn, the Valid Detection Benchmark shall preserve its bytes and digests.
|
||||
7. When the historical pilot is withdrawn, the Valid Detection Benchmark shall add a reviewed lifecycle record containing the withdrawal reason and replacement relationship.
|
||||
8. If historical identity-bearing material remains accessible for audit, the Valid Detection Benchmark shall authorize it only by exact path and digest and shall not create a blanket reservation-guard exclusion.
|
||||
9. When the replacement population is frozen, the Valid Detection Benchmark shall use new population and freeze identities and shall not replace or rewrite the historical pilot in place.
|
||||
10. When a replacement population is about to be accepted, the Valid Detection Benchmark shall pass the permanent-reservation guard while permitting only reviewed exact-path-and-digest historical authorizations and without weakening reservation semantics.
|
||||
|
||||
### Requirement 5: Development and Admission Separation
|
||||
|
||||
**Objective:** As a release decision maker, I want detector development separated from untouched admission, so that measured qualification is not influenced by the labels it evaluates.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The Valid Detection Benchmark shall maintain disjoint development and admission populations.
|
||||
2. When a population or slice is inspected to diagnose, tune, select, or promote detector behavior, the Valid Detection Benchmark shall classify it as development or spent and shall not use it for admission.
|
||||
3. When admission is prepared and before labels are opened, the Valid Detection Benchmark shall freeze a label-blind manifest containing only scope census information and digests.
|
||||
4. When a label-blind manifest is reviewed, the Valid Detection Benchmark shall require a reviewer whose recorded identity differs from the producer's identity, satisfies the pre-registered review policy, and includes a conflict disclosure.
|
||||
5. When admission is prepared and before labels are opened, the Valid Detection Benchmark shall bind the population, analyzer, configuration, fact set, judgment vocabulary, outcome mapping, denominator policy, thresholds, and claimed scopes to immutable identities.
|
||||
6. If admission labels influence facts, strategies, matchers, prompts, examples, thresholds, judges, decision mappings, or promotions, the Valid Detection Benchmark shall mark the population spent and refuse an admission result from it.
|
||||
7. When admission labels are released, the Valid Detection Benchmark shall durably record the spent transition before exposing any label bytes.
|
||||
8. When a post-spend case is released for scoring, the Valid Detection Benchmark shall bind the case to its scope, role, pair identity, result, evidence, change inputs, and reviewed input manifest.
|
||||
9. When a post-spend case release is reviewed, the Valid Detection Benchmark shall require a reviewer whose recorded identity differs from the producer's identity, satisfies the pre-registered review policy, and includes a conflict disclosure.
|
||||
|
||||
### Requirement 6: Scoped Denominators and Feasible Gates
|
||||
|
||||
**Objective:** As a detector owner, I want quality evaluated only on the families and ecosystems an arm claims, so that out-of-scope cases neither dilute nor inflate its result.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When an arm is registered and before label access, the Valid Detection Benchmark shall require the arm to declare every vulnerability-family and ecosystem scope it claims.
|
||||
2. When recall or pair correctness is calculated for a scope, the Valid Detection Benchmark shall use only vulnerable and fixed cases independently verified for that exact family and ecosystem.
|
||||
3. When false alarms and precision are calculated for a scope, the Valid Detection Benchmark shall use only findings and ordinary pushes assigned to that exact family and ecosystem under the pre-registered denominator policy.
|
||||
4. The Valid Detection Benchmark shall report whole-population coverage separately from scoped quality metrics.
|
||||
5. Where an overall aggregate is provided, the Valid Detection Benchmark shall identify its constituent scopes, sample counts, and weighting and shall not use that aggregate to authorize a capability or ecosystem claim.
|
||||
6. When a gate is registered and before label access, the Valid Detection Benchmark shall prove from the label-blind census that every pre-registered threshold is arithmetically reachable for its declared denominator.
|
||||
7. If a threshold is impossible or a scoped denominator fails its pre-registered minimum size or interval-precision rule, the Valid Detection Benchmark shall refuse to spend the population for that gate and shall report the scope as insufficient.
|
||||
|
||||
### Requirement 7: Reproducible Population and Durable Custody
|
||||
|
||||
**Objective:** As an operator, I want population state and provenance to survive restart or local-disk loss, so that a long sourcing or admission workflow remains reproducible and auditable.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When a population is frozen, the Valid Detection Benchmark shall bind its positive selection, ordinary-change draw, reservation census, review records, exclusion counts, and provenance to content digests.
|
||||
2. When unchanged source material is frozen again under the same policy, the Valid Detection Benchmark shall produce the same population identity.
|
||||
3. If an existing frozen identity resolves to different content, the Valid Detection Benchmark shall refuse the write and preserve the prior artifact.
|
||||
4. The Valid Detection Benchmark shall keep durable population state, custody transitions, provenance, and resumable sourcing state in a versioned durable store rather than relying on local scratch files.
|
||||
5. If local scratch state is lost or execution restarts, the Valid Detection Benchmark shall resume from durable custody without changing accepted decisions or identities.
|
||||
6. Where material is publicly distributed, the Valid Detection Benchmark shall include only permissively licensed content; non-permissive or uncertain material shall remain private behind opaque digest-bound references.
|
||||
7. The Valid Detection Benchmark shall retain durable population, provenance, lifecycle, and custody records in an auditable form.
|
||||
8. When transient run records are created, the Valid Detection Benchmark shall apply a pre-registered time-bounded retention policy that expires both current and noncurrent versions by their recorded deadlines.
|
||||
|
||||
### Requirement 8: Standard Measurement and Failure Visibility
|
||||
|
||||
**Objective:** As a reviewer, I want every benchmark result to carry the same complete measurement record, so that a clean result, an unsupported result, and a failed instrument cannot be confused.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When a benchmark result is published, the Valid Detection Benchmark shall identify the population and freeze digests, judgment vocabulary, claimed scopes, denominator policy, analyzer and fact identities, and threshold set.
|
||||
2. When a scoped result is published, the Valid Detection Benchmark shall report recall, false alarms per 100 ordinary pushes, precision, pair correctness, sample counts, and Wilson 95% intervals.
|
||||
3. When a benchmark result is published, the Valid Detection Benchmark shall report whole-population coverage, completed and deferred work, instrument and model failures, runtime, and every degradation with its reason.
|
||||
4. When a zero or clean-looking result is about to be interpreted, the Valid Detection Benchmark shall prove nonzero input reads and execution of every stage required by the declared arm.
|
||||
5. If an input is unreadable, empty, skipped, prematurely terminated, or processed by a failed required stage, the Valid Detection Benchmark shall report an instrument failure or bounded incomplete result and shall not count it as clean.
|
||||
6. When the population, vocabulary, denominator, claimed scope, analyzer identity, fact identity, threshold set, or outcome mapping changes, the Valid Detection Benchmark shall start a new baseline and shall not describe the new result as a regression trend against an incompatible baseline.
|
||||
|
||||
### Requirement 9: Push-Path Arms and Demonstrated Evidence
|
||||
|
||||
**Objective:** As a product owner, I want benchmark arms and proof strength reported distinctly, so that experimental assistance is not mistaken for shipped behavior or demonstrated exploitation.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When the shipped push path and an experimental detector arm are evaluated, the Valid Detection Benchmark shall report their results separately over the same eligible changes.
|
||||
2. Where a pre-registered combined arm is evaluated, the Valid Detection Benchmark shall identify its combination rule before label access and shall preserve each contributing arm's result.
|
||||
3. When an exploit-demonstrated rate is reported, the Valid Detection Benchmark shall require a matching differential proof artifact showing the owned effect on the vulnerable revision and its absence on the fixed revision.
|
||||
4. If a flagged positive lacks a matching differential proof artifact, the Valid Detection Benchmark shall report it as not demonstrated rather than as a failed exploit or an exploit-verified finding.
|
||||
5. The Valid Detection Benchmark shall not treat scanner verification, model confidence, or model agreement as demonstrated-exploit evidence.
|
||||
6. When reporting advisory eligibility, the Valid Detection Benchmark shall scope the result per capability and ecosystem.
|
||||
7. When reporting a blocking claim, the Valid Detection Benchmark shall require demonstrated proof.
|
||||
|
||||
### Requirement 10: Operator Outcomes and Feature Boundary
|
||||
|
||||
**Objective:** As an operator, I want benchmark completion and impairment states to be explicit while detector and infrastructure decisions remain with their owning workflows.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When a benchmark workflow completes, the Valid Detection Benchmark shall expose its status, counts, digests, scoped GO, NO-GO, or insufficient outcomes, and named failed gates.
|
||||
2. If a benchmark workflow stalls, times out, exhausts a budget or resource, produces an insufficient result, or encounters an instrument failure, the Valid Detection Benchmark shall expose the impaired state and its reason.
|
||||
3. The Valid Detection Benchmark shall not include secrets, exploit payloads, private labels, repository contents, or customer content in shared reports or operator notifications.
|
||||
4. The Valid Detection Benchmark shall not modify detector logic, facts, prompts, judges, thresholds, or product rollout state while constructing or scoring a population.
|
||||
5. The Valid Detection Benchmark shall not reactivate or requalify the parked access-control arm without a separate maintainer strategy decision.
|
||||
6. Where an execution plane is used, the Valid Detection Benchmark shall report benchmark process outcomes without assuming ownership of worker, cluster, or network infrastructure health.
|
||||
|
||||
@@ -0,0 +1,337 @@
|
||||
# Research and Gap Analysis: valid-detection-benchmark
|
||||
|
||||
## Summary
|
||||
|
||||
- **Feature**: `valid-detection-benchmark`
|
||||
- **Analysis date**: 2026-11-05
|
||||
- **Discovery scope**: Complex brownfield integration; gap analysis only
|
||||
- **Requirements state**: Generated but unapproved. The existing `design.md` and `tasks.md` predate the reopened requirements and are stale.
|
||||
- **Overall finding**: The repository has strong reusable fix-site, ordinary-history, artifact, admission, custody, proof, and measurement foundations. It does not yet have a valid replacement population, authoritative reservation/lifecycle workflow, durable custody store, canonical ecosystem review, or exact per-family/per-ecosystem quality reports.
|
||||
- **Leading design candidate**: A hybrid approach that minimally extends detection-core's authoritative scoped result contract and adds benchmark-owned population governance, persistence, adapters, and operator reporting.
|
||||
- **Estimated effort**: XL, more than two weeks of implementation plus separately scheduled sourcing and human review.
|
||||
- **Risk**: High overall. The largest risks are cross-spec metric authority, irreversible admission spend, durable compare-and-swap semantics, and the human/data work needed for at least 40 accepted positives.
|
||||
|
||||
## Analysis Context
|
||||
|
||||
The selected direction is a general Python and JavaScript/TypeScript benchmark with multiple vulnerability families and quality denominators scoped by exact family and ecosystem. It supersedes the access-control-specific use of the 2026-10-12 pilot. Permanent reservation remains authoritative, the historical pilot remains byte-preserved but non-admissible, and the parked access-control arm is not reactivated by this work.
|
||||
|
||||
No external dependency research was necessary for this pass. The main gaps are internal contracts and workflows. External capability research is deferred only for the deployed durable store's conditional-write semantics.
|
||||
|
||||
## Current Architecture and Reusable Assets
|
||||
|
||||
### Population construction
|
||||
|
||||
- `benchmarks/valid/catalog.py` builds fix-site cases and already recognizes Python, JavaScript, and TypeScript files. It records vulnerability family and CWE but drops source ecosystem metadata.
|
||||
- `benchmarks/valid/verify.py` reads both revisions with nonzero-byte instrumentation, records model assistance, and accepts a human spot check. Its current verdict still makes the model booleans authoritative prerequisites.
|
||||
- `benchmarks/valid/assemble.py` filters accepted fix-site cases, draws at least 20 ordinary first-parent commits per distinct vulnerability push, collapses multi-function fixes into one push, freezes deterministic TOML/JSON artifacts, and refuses conflicting in-place rewrites.
|
||||
- `benchmarks/unseen/eligibility.py` gathers repository identities from development sources and resolves aliases, former names, redirects, and fork networks.
|
||||
- `tests/test_independent_population.py` is the authoritative permanent-reservation guard.
|
||||
|
||||
### Artifact, admission, and custody contracts
|
||||
|
||||
- `benchmarks/valid/artifact_contract.py` supplies canonical digest envelopes, ordered collection identities, atomic writes, idempotency, and conflict refusal.
|
||||
- `src/openultrasast/core/admission_review.py` supplies reviewed label-blind `PopulationCensus`, `ScopeCensus`, `AdmissionManifestV2`, and thresholds.
|
||||
- `src/openultrasast/core/admission_feasibility.py` proves threshold arithmetic before label access.
|
||||
- `src/openultrasast/core/admission_case.py` binds post-spend cases to exact scope, role, pair, result, evidence, change inputs, and reviewed input manifest.
|
||||
- `src/openultrasast/core/admission_spend.py` and `benchmarks/valid/population_custody.py` enforce available-to-spent custody before label bytes are released.
|
||||
- `src/openultrasast/core/admission_metrics.py` rederives metrics from validated evidence and namespaces pair identity by family and ecosystem.
|
||||
- `src/openultrasast/core/admission_report.py` produces authoritative GO/NO-GO reports and replay validation.
|
||||
- `src/openultrasast/core/proof.py` and the search adapter provide the differential proof authority required for demonstrated claims.
|
||||
|
||||
### Operational persistence patterns
|
||||
|
||||
- `benchmarks/unseen/bank.py` and `src/openultrasast/plane/memory.py` provide reusable S3-compatible persistence and lifecycle checks.
|
||||
- Those writers are unconditional. No production implementation of `CompareAndSwapCustodyStore` exists; only test-local in-memory implementations exercise the custody protocol.
|
||||
|
||||
## Requirement-to-Asset Gap Map
|
||||
|
||||
| Requirement | Existing assets | Status | Principal gap |
|
||||
|---|---|---|---|
|
||||
| 1. General multi-family population | `catalog.py`, source family/ecosystem metadata, multi-target freeze tests | Partial / Constraint | No replacement population; no 40-positive acceptance; source ecosystem is dropped; no canonical ecosystem review; core pools claimed scopes into one metric result |
|
||||
| 2. Fix-derived positive ground truth | `catalog.build_cases`, `extract.fix_sites`, `verify_case`, `spot_check` | Partial / Constraint | Human independence policy and conflict disclosure are absent; model booleans currently control acceptance; exclusions lack durable reasons; multi-scope labels are not reviewed |
|
||||
| 3. Representative ordinary changes | `ordinary_history`, `draw_ordinary`, `assemble.negatives`, legacy mislabeled-defect refusal | Partial / Constraint / Missing | No ecosystem on ordinary rows; no reviewed projection across claimed family scopes; no reviewed negative adjudication or post-freeze retirement transition |
|
||||
| 4. Permanent reservation and withdrawal | `eligibility.py`, permanent guard, immutable historical files | Missing / Constraint | No authoritative reservation census consumed by freeze; no influence events; no enumerated-only disposition; no withdrawal/inventory record; no exact-file historical authorization; guard currently fails |
|
||||
| 5. Development/admission separation | Core reviewed manifest, readiness, custody, spend, case release | Partial | Benchmark cannot author the label-blind manifest; development inspection is not wired to custody/lineage; durable CAS store is missing |
|
||||
| 6. Scoped denominators and gates | `ScopeKey`, `DenominatorPolicy`, `ScopeCensus`, feasibility and metric validation | Partial / Constraint | One policy rule per metric currently pools all claimed scopes; no per-scope result container, minimum-size/interval rule, or `insufficient` outcome |
|
||||
| 7. Reproducible population and custody | deterministic freeze, artifact contract, custody protocol, object-store patterns | Partial / Missing / Unknown | Population freeze uses legacy artifacts; no production CAS, resumable replacement workflow, license boundary, withdrawal lifecycle, or transient-record retention enforcement |
|
||||
| 8. Standard measurement and failure visibility | `MeasurementRecordV2`, coverage validation, admission report/replay | Partial / Constraint | Final admission report drops some whole-population/runtime detail, records no general degradations, and cannot persist pre-report failures as bounded workflow outcomes |
|
||||
| 9. Push-path arms and demonstrated evidence | separate runners, legacy hunter combine, proof contracts and adapters | Partial / Missing | No manifest-bound combined-arm rule, component report preservation, scoped demonstrated rate, or general multi-arm admission report |
|
||||
| 10. Operator outcomes and boundary | core GO/NO-GO gates and steering notification policy | Partial / Missing | No general benchmark CLI/presenter, `insufficient` or failed workflow envelope, safe report projection, or notifier integration; current runner is access-control-specific |
|
||||
|
||||
## Detailed Gap Findings
|
||||
|
||||
### Population shape and ecosystem identity
|
||||
|
||||
The existing candidate sources often carry package ecosystems such as `pip` and `npm`, while existing detection-core examples use language-like ecosystem values such as `python` and `javascript`. The benchmark needs one reviewed vocabulary and a rule for repositories containing more than one framework or ecosystem. This is a design decision; the implementation must not silently infer or coerce the value.
|
||||
|
||||
`assemble.freeze()` accepts any nonempty retained population. It does not enforce the 40-positive full-population boundary or classify smaller freezes as development pilots or insufficient populations. It also freezes labels and identities in a legacy manifest rather than producing the label-blind reviewed census required before admission.
|
||||
|
||||
### Curation authority
|
||||
|
||||
Fix-site extraction is already the correct foundation and has strong offline tests. The human-review record is too weak for the reopened requirements: it contains only decision, reviewer text, and date. It does not bind producer identity, review policy, conflict disclosure, or individual ground-truth judgments.
|
||||
|
||||
The current `verify.verdict()` accepts a case only when the model returned all three positive booleans. That makes model assistance part of label authority. The design must retain the model output as review evidence while making a policy-compliant human attestation authoritative.
|
||||
|
||||
### Permanent reservation and historical pilot
|
||||
|
||||
The guard was run with readable inputs: unseen TOML inputs totaled 3,475,696 bytes and the guard test file totaled 9,146 bytes. The focused test exited 1 after 33.84 seconds with 73 reserved references, all under `benchmarks`; `src`, `tests`, and `plane` contributed zero. This is a real policy conflict, not an unread-input zero.
|
||||
|
||||
The pilot payloads must remain unchanged. A new reservation-owned withdrawal inventory must bind each retained historical path and SHA-256 digest, record its non-admissible status and replacement relationship, and authorize only those exact files. The guard may consume reviewed exact-file authorizations but must not gain a directory exclusion or repository-name allowlist.
|
||||
|
||||
`eligibility.build_used()` is a broad discovery collector, not an authoritative event ledger. The replacement workflow needs reviewed influence dispositions so an enumerated-only repository can be distinguished from one whose code, labels, or metadata informed a decision.
|
||||
|
||||
### Scoped metric authority
|
||||
|
||||
Detection-core already validates exact `ScopeKey` values, rejects cross-scope pair substitution, preserves whole-population coverage alongside in-scope cases, and proves threshold feasibility. However, one `DenominatorPolicy` rule currently names the complete claimed scope tuple and `derive_admission_metrics()` pools those scopes into one `MetricResult` per metric. `MetricResult` carries no scope or weighting metadata.
|
||||
|
||||
Therefore the requirement for independently interpretable family/ecosystem results cannot be implemented safely only in `benchmarks/valid/metrics.py`. That would create a second metric authority. The design must choose a versioned detection-core extension, such as one admission manifest producing a canonical tuple of exact-scope reports plus a non-authorizing aggregate. Separate post-spend reports must not independently reinterpret one label release.
|
||||
|
||||
The core also needs an explicit `insufficient` outcome or equivalent non-authorizing status for scopes whose minimum sample or interval rule fails. GO/NO-GO alone cannot represent a pre-spend denominator that is validly too small.
|
||||
|
||||
### Durable custody and recovery
|
||||
|
||||
The custody protocol correctly requires a successful compare-and-swap append and reread before label access. Its production storage seam is unimplemented. Existing object-store adapters provide versioning and lifecycle inspection but not conditional compare-and-swap.
|
||||
|
||||
Before selecting an adapter, the design phase must capability-test the deployed store for conditional create/update, concurrent-writer behavior, version visibility, restart recovery, and current/noncurrent expiry. If the store cannot provide the required semantics, the design needs a different durable serialization authority; ordinary overwriting `put_object` calls are not sufficient.
|
||||
|
||||
### Measurement, proof, and operator outcomes
|
||||
|
||||
The v2 core measurement path is substantially stronger than the legacy `benchmarks/valid/metrics.py` path. It revalidates evidence, metrics, gates, and replay identities. The benchmark should adapt to it rather than extend the legacy aggregate scorer as an admission authority.
|
||||
|
||||
The current v2 report does not carry a durable workflow outcome for exceptions that happen before report construction. Instrument failures fail closed, but operators cannot distinguish a bounded incomplete run from an absent report. A benchmark-owned workflow envelope can record safe reason codes and digests without weakening the authoritative core report.
|
||||
|
||||
Proof authority already exists. The missing work is to bind proof-derived scoped rates and any combined arm rule into the pre-label baseline. Legacy shape checking in `metrics.demonstrated()` is useful compatibility code but cannot authorize blocking.
|
||||
|
||||
## Architecture Pattern Evaluation
|
||||
|
||||
| Option | Description | Strengths | Risks / Limitations | Effort / Risk |
|
||||
|---|---|---|---|---|
|
||||
| A. Extend existing components | Expand current `assemble.py`, `metrics.py`, admission contracts, and guard in place | Few new modules; maximum reuse; short path for simple criteria | Bloating of already mixed responsibilities; legacy and v2 authority remain easy to confuse; strict core schemas make broad in-place edits risky | L–XL / High |
|
||||
| B. New benchmark-only layer | Leave detection-core unchanged and implement scoped projections, lifecycle, custody, and reports entirely under `benchmarks/valid/` | Clear benchmark ownership; limited core migration | Duplicates or bypasses authoritative metric algebra; unsafe after one irreversible spend; creates competing report authorities | L / High integrity risk |
|
||||
| C. Hybrid versioned extension | Add the minimum scoped-result and workflow-status contracts upstream; add benchmark-owned population, reservation, custody adapter, case adapters, and presenter | Preserves one metric authority and existing ownership split; migration can be staged; most current assets remain reusable | Cross-spec coordination and artifact migration; more components and conformance tests | XL / High overall, Medium per component |
|
||||
|
||||
Option C is the leading design candidate because it is the only approach that satisfies scoped denominators without duplicating admission authority. It remains a candidate until the core schema shape and durable-store capability are resolved in design.
|
||||
|
||||
## Compatibility and Migration Boundaries
|
||||
|
||||
- Historical pilot and measurement payloads are immutable audit inputs, never migration targets.
|
||||
- Legacy `metrics.py` and the parked access-control helpers remain readable historical/development paths but do not authorize the replacement admission baseline.
|
||||
- New population, lifecycle, census, manifest, custody, run, and report artifacts require versioned kinds and new identities.
|
||||
- A population, vocabulary, scope, denominator, analyzer, fact, threshold, or outcome-mapping change creates a new baseline.
|
||||
- Detection-core owns metric algebra, scope validation, spend validation, and authoritative replay. The benchmark owns sourcing, review records, reservation/lifecycle state, durable label custody, arm execution adapters, and operator presentation.
|
||||
- The replacement workflow must not call or requalify the parked access-control arm unless a separate maintainer decision changes that policy.
|
||||
|
||||
## Complexity and Risk Assessment
|
||||
|
||||
### Effort
|
||||
|
||||
- **Core scoped-result extension**: L. It changes strict dataclasses, canonical digests, report/replay validation, and conformance tests.
|
||||
- **Reservation withdrawal and exact-file authorization**: M. The code is bounded, but migration evidence and immutable inventory require care.
|
||||
- **Ecosystem, review, and population freeze schemas**: L. Existing catalog and assembly code are reusable, but responsibilities must be separated.
|
||||
- **Durable CAS custody adapter and recovery tests**: M–L, depending on store capability.
|
||||
- **General arm orchestration, workflow outcomes, and presentation**: L.
|
||||
- **Replacement sourcing and independent review**: XL elapsed work; rate limits, license decisions, repository availability, and human review dominate.
|
||||
- **Overall**: XL, more than two weeks, with operational curation continuing beyond pure code implementation.
|
||||
|
||||
### Risk
|
||||
|
||||
- **Overall risk: High.** Admission spend is irreversible, the existing guard is red, and one wrong scope projection can authorize a false claim.
|
||||
- **Schema migration risk: High.** Core artifacts are closed, digest-bound graphs; changing fields requires full replay and compatibility tests.
|
||||
- **Persistence risk: High until measured.** Durable conditional-write semantics are unverified on the deployed store.
|
||||
- **Population availability risk: High.** Forty accepted, reachable, permissively distributable positives across two language groups and multiple families may require many more candidates.
|
||||
- **Detection implementation risk: Low within this spec.** Detector logic is explicitly out of scope and must remain unchanged.
|
||||
|
||||
## Design-Phase Recommendations
|
||||
|
||||
1. Define a versioned core scoped-result container: exact `ScopeKey` result rows, one non-authorizing aggregate, minimum-size/interval sufficiency, and a workflow-safe `insufficient` status.
|
||||
2. Keep core changes in the detection-core ownership boundary. The benchmark design consumes that contract and must not reproduce its metric algebra.
|
||||
3. Add an authoritative reservation/influence ledger and a digest-bound withdrawal inventory before sourcing the replacement.
|
||||
4. Define one canonical ecosystem vocabulary and independent review record before changing catalog or freeze formats.
|
||||
5. Replace model-controlled acceptance with human-authoritative reviewed ground truth while retaining model output as non-authoritative evidence.
|
||||
6. Author label-blind census/manifest artifacts from the new freeze and integrate them with existing readiness, spend, case release, and replay contracts.
|
||||
7. Capability-test the durable store before committing to a CAS adapter. Preserve the custody protocol's spend-before-read ordering.
|
||||
8. Add a benchmark workflow envelope for completion, insufficient, timeout, resource, and instrument-failure outcomes using fixed redacted reason codes.
|
||||
9. Migrate in this order: core scoped contract; reservation/withdrawal; ecosystem and review schemas; population freeze; durable custody; arm adapters; scoped reporting; replacement sourcing and review.
|
||||
|
||||
## Research Needed for Design
|
||||
|
||||
1. **Scoped result shape**: one manifest with a tuple of per-scope reports versus another versioned container; define aggregate weighting and replay rules.
|
||||
2. **Canonical ecosystem taxonomy**: package ecosystem, runtime/framework ecosystem, or a reviewed composite; specify multi-ecosystem repositories.
|
||||
3. **Durable CAS capability**: conditional write semantics, concurrent writers, version visibility, crash recovery, and lifecycle enforcement on the deployed store.
|
||||
4. **Population sizing**: candidate yield needed to reach 40 accepted positives and the pre-registered minimum/interval rule for each claimed scope.
|
||||
5. **Historical authorization format**: review subject, exact path/digest inventory, relocation rules, and guard integration.
|
||||
6. **Safe workflow reason taxonomy**: stable operator-facing codes that reveal no private labels, paths, payloads, or exception text.
|
||||
7. **License custody**: approved permissive-license vocabulary and opaque private-reference workflow for uncertain or non-permissive material.
|
||||
|
||||
## Validation Evidence
|
||||
|
||||
- The requirement headings and acceptance criteria passed the EARS mechanical gate before this analysis.
|
||||
- Focused reservation guard command:
|
||||
|
||||
```text
|
||||
PYTHONPATH=src:tests .venv/bin/python -m pytest -q -p no:cacheprovider \
|
||||
tests/test_independent_population.py::test_no_reserved_repository_is_referenced_by_the_development_tree
|
||||
```
|
||||
|
||||
- Result: `1 failed in 33.84s`, exit code `1`, with `73` benchmark references and zero references from `src`, `tests`, or `plane`.
|
||||
- No full test suite was run for this read-only analysis.
|
||||
|
||||
## References
|
||||
|
||||
- `.kiro/specs/valid-detection-benchmark/requirements.md`
|
||||
- `.kiro/specs/valid-detection-benchmark/design.md` — stale reference only
|
||||
- `.kiro/steering/detection-quality.md`
|
||||
- `.kiro/steering/storage-streaming.md`
|
||||
- `.kiro/steering/code-standards.md`
|
||||
- `benchmarks/valid/`
|
||||
- `benchmarks/unseen/eligibility.py`
|
||||
- `tests/test_independent_population.py`
|
||||
- `src/openultrasast/core/admission_*.py`
|
||||
- `src/openultrasast/core/measurement.py`
|
||||
- `src/openultrasast/core/proof.py`
|
||||
|
||||
---
|
||||
|
||||
## Design Discovery and Synthesis (2026-11-06)
|
||||
|
||||
### Discovery Summary
|
||||
|
||||
- **Discovery type**: Complex integration, full discovery.
|
||||
- **Selected pattern**: Hybrid versioned extension. Detection-core remains the sole authority for scope validation, metric algebra, threshold evaluation, and replay. The benchmark owns population governance, reservation state, review, freezing, custody adapters, arm execution, workflow outcomes, and operator presentation.
|
||||
- **Critical prerequisite**: The replacement admission baseline requires a detection-core V3 admission root. V2 necessarily pools all claimed scopes and cannot be projected into authoritative per-scope results after label release.
|
||||
- **Compatibility rule**: V2 artifacts remain byte-preserved and replayable. A pooled V2 result is never auto-upgraded; the V3 population starts a new baseline.
|
||||
|
||||
### Research Log
|
||||
|
||||
#### Detection-core scoped-result seam
|
||||
|
||||
- **Context**: Requirements 6 and 8 require exact family-and-ecosystem metrics, per-scope gates, whole-population coverage, and non-authorizing aggregates.
|
||||
- **Sources consulted**: `measurement.py`, `admission_context.py`, `admission_feasibility.py`, `admission_metrics.py`, `admission_report.py`, `_admission_replay.py`, and their core admission tests.
|
||||
- **Findings**:
|
||||
- V2 assigns every standard metric rule the full claimed-scope tuple, sums census capacity across the tuple, derives one metric result per metric, and stores no scope on `MetricResult`.
|
||||
- The smallest safe change is a new V3 root with exact-scope metric rules and thresholds, pre-spend scope sufficiency, one canonical result per scope, separate deduplicated whole-population coverage, and an optional aggregate type with no gate or decision field.
|
||||
- `insufficient` belongs to pre-spend feasibility. Because custody releases one population, any insufficient claimed scope makes `may_spend` false. Partial-scope label release is a different custody model and is out of scope.
|
||||
- Multi-scope whole-population coverage must deduplicate by canonical change-input identity; summing scoped rows can count one multi-label push more than once.
|
||||
- **Implications**: This spec consumes the V3 contract as an upstream prerequisite. It does not duplicate metric algebra under `benchmarks/valid/` and does not modify detection-core-owned files.
|
||||
|
||||
#### Canonical ecosystem and human review authority
|
||||
|
||||
- **Context**: Existing source records carry package ecosystem and license evidence, while the valid catalog drops it and current model booleans control acceptance.
|
||||
- **Sources consulted**: `benchmarks/unseen/source.py`, `benchmarks/valid/catalog.py`, `verify.py`, `assemble.py`, and review contracts under `src/openultrasast/core/`.
|
||||
- **Findings**:
|
||||
- The V1 benchmark vocabulary is `python` and `javascript`; TypeScript maps to the JavaScript ecosystem while remaining visible as an observed language. Package ecosystems such as `pip` and `npm` remain source evidence, not canonical scope values.
|
||||
- One independent repository review selects exactly one canonical ecosystem. A polyglot or conflicting case is `insufficient`, never silently coerced.
|
||||
- Ground truth is a human attestation over defect presence, fix effect, in-repository reachability, and exact verified scopes. Model output is retained only as a digest-bound assistance reference and never participates in acceptance algebra.
|
||||
- **Implications**: Cataloging preserves source evidence; a new review component owns authoritative decisions; freezing consumes only accepted reviewed records.
|
||||
|
||||
#### Permanent reservation and historical withdrawal
|
||||
|
||||
- **Context**: The old pilot is permanently reserved but must remain accessible for audit without creating a broad guard exclusion.
|
||||
- **Sources consulted**: `benchmarks/unseen/eligibility.py`, `tests/test_independent_population.py`, the current pilot artifacts, and the focused guard evidence recorded above.
|
||||
- **Findings**:
|
||||
- Broad discovery of repository names is not an influence ledger. A reviewed append-only event must distinguish `enumerated_only` from `influenced` and resolve aliases/forks before projection into the permanent census.
|
||||
- Historical authorization is an append-only lifecycle: a `withdrawn` event binds exact repository-relative paths and SHA-256 digests; a later `replacement_bound` event links the new population and freeze identities.
|
||||
- The production guard permits only exact path-and-current-digest matches. Relocation or byte changes require a new reviewed event; repository-name or directory allowlists are forbidden.
|
||||
- **Implications**: Reservation semantics move from test-only logic into `benchmarks/unseen/reservation.py`; the existing test becomes a consumer of that authority.
|
||||
|
||||
#### Durable compare-and-swap custody
|
||||
|
||||
- **Context**: The core requires a committed and reread custody transition before exposing label bytes, but current S3 writers are unconditional.
|
||||
- **Sources consulted**: installed boto3/botocore service model, AWS S3 conditional-write documentation, RustFS S3 compatibility documentation, `population_custody.py`, `bank.py`, and `plane/memory.py`.
|
||||
- **Findings**:
|
||||
- The locked boto3/botocore `1.43.107` service model exposes `IfMatch` and `IfNoneMatch` on `PutObject`.
|
||||
- Amazon S3 documents conditional create/update using `If-None-Match: *` and `If-Match` ETags, with 412 for failed preconditions and possible 409 conflicts under concurrency.
|
||||
- RustFS documents common conditional requests, versioning, and lifecycle, but explicitly requires applications to validate exact advanced operations against the target release. Its published compatibility matrix does not establish this project's complete CAS protocol.
|
||||
- The adapter must therefore perform a target-store capability probe for conditional create, conditional update, two-writer conflict, version visibility, restart recovery, and lifecycle behavior. Unsupported or ambiguous behavior fails closed before any real population is stored.
|
||||
- **Implications**: A benchmark-owned S3 CAS adapter uses content digests as domain identity and ETag/version only as the storage concurrency token. Existing unconditional `CorpusObjects` is not reused as custody authority.
|
||||
- **Official references**:
|
||||
- [Amazon S3 PutObject API](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html)
|
||||
- [Amazon S3 conditional writes](https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes.html)
|
||||
- [RustFS S3 protocol and compatibility scope](https://docs.rustfs.com/en/administration/protocols/s3)
|
||||
- [RustFS S3 compatibility matrix](https://docs.rustfs.com/en/reference/s3-compatibility)
|
||||
|
||||
#### General arm orchestration and operator outcomes
|
||||
|
||||
- **Context**: `admission_run.py` is access-control-specific and the parked arm must remain parked. Completed core reports cannot represent failures before report construction.
|
||||
- **Sources consulted**: `benchmarks/valid/admission_run.py`, `admission_scoring.py`, `metrics.py`, core case/proof/report contracts, and `operator-notifications.md`.
|
||||
- **Findings**:
|
||||
- The reusable ordering is preflight, authorize, durable spend, label release, label-blind arm execution, reviewed case binding, core score, and replay validation.
|
||||
- Arms receive immutable label-blind case inputs. The custodian alone joins detector observations to released ground truth. Combined-arm rules are digest-bound before label access and preserve every component result.
|
||||
- Early or impaired runs need a benchmark-owned envelope with a closed redacted reason taxonomy. It may reference a completed core replay but cannot fabricate a core report.
|
||||
- The spec owns deterministic JSON/text presentation. Discord transport remains outside the boundary because no shared notifier implementation exists and steering permits separate wiring.
|
||||
- **Implications**: The replacement runner is general and cannot import access-control-specific site judgment helpers.
|
||||
|
||||
### Architecture Pattern Evaluation Update
|
||||
|
||||
| Option | Decision | Rationale |
|
||||
|---|---|---|
|
||||
| Extend V2 and legacy benchmark modules | Rejected | Pooled V2 metrics cannot become exact-scope authority; mixed modules would retain competing admission paths. |
|
||||
| Benchmark-only V3 projection | Rejected | It would duplicate detection-core algebra after irreversible label release. |
|
||||
| Hybrid versioned extension | Selected | One authoritative core result graph, bounded benchmark ownership, preserved V2 compatibility, and independently testable persistence/adapters. |
|
||||
|
||||
### Design Decisions
|
||||
|
||||
#### Decision: Treat detection-core V3 as an explicit upstream prerequisite
|
||||
|
||||
- **Selected approach**: Detection-core owns `DenominatorPolicyV3`, exact-scope feasibility, scoped results, deduplicated whole-population coverage, non-authorizing aggregate, report, and replay. This spec consumes those contracts.
|
||||
- **Rationale**: Scope and metric authority already belong to detection-core; moving them into benchmark code would create hidden shared ownership.
|
||||
- **Trade-off**: Implementation cannot begin admission scoring until the dependency amendment lands, but population governance and non-label-bearing work can proceed in parallel.
|
||||
|
||||
#### Decision: Make population spend atomic across all claimed scopes
|
||||
|
||||
- **Selected approach**: Every claimed scope must be feasible before the one population label release. Any insufficiency emits a pre-spend outcome and leaves the population unspent.
|
||||
- **Rationale**: The current custody object releases the population as a unit. Partial release would expose labels for all scopes while pretending some remained untouched.
|
||||
- **Trade-off**: Sparse scopes delay admission, but the result remains interpretable and reusable.
|
||||
|
||||
#### Decision: Use reviewed repository-level ecosystem identity
|
||||
|
||||
- **Selected approach**: Canonical values are `python` and `javascript`; TypeScript is an observed language within `javascript`. One independent review chooses the repository ecosystem or records `insufficient`.
|
||||
- **Rationale**: This matches the existing core examples, gives stable denominators, and prevents framework-specific inventories from becoming architecture.
|
||||
- **Trade-off**: Ambiguous polyglot repositories are excluded from admission until their benchmark role is clear.
|
||||
|
||||
#### Decision: Build reservation state as an append-only reviewed projection
|
||||
|
||||
- **Selected approach**: Influence and lifecycle events are immutable; deterministic censuses and exact-file authorizations are projections.
|
||||
- **Rationale**: It distinguishes enumeration from influence, preserves historical bytes, and makes every exception auditable without weakening the guard.
|
||||
- **Trade-off**: Accepted V1 lifecycle events have no general correction transition. Invalid proposals are rejected before projection; terminal development/retirement states cannot be undone.
|
||||
|
||||
#### Decision: Adopt S3 conditional writes only after a capability probe
|
||||
|
||||
- **Selected approach**: Use existing boto3 and S3 conditional headers through a new CAS adapter; validate the deployed RustFS release and bucket policy before real use.
|
||||
- **Rationale**: The SDK supports the API, but compatibility claims are not a substitute for testing the exact target store and concurrency protocol.
|
||||
- **Trade-off**: A failed probe blocks custody implementation; there is no unsafe unconditional-write fallback.
|
||||
|
||||
#### Decision: Keep workflow status outside the authoritative core report
|
||||
|
||||
- **Selected approach**: A benchmark envelope records completed, insufficient, timeout, budget, resource, instrument, and interruption outcomes. Completed entries reference the core V3 replay.
|
||||
- **Rationale**: Pre-report failures need durable visibility but must not be made to look like a quality decision.
|
||||
- **Trade-off**: Operators consume two nested authorities: workflow completion from the benchmark and quality decisions from detection-core.
|
||||
|
||||
### Synthesis
|
||||
|
||||
- **Generalization**: Population review, arm registration, and outcome reporting are horizontal contracts. Concrete repositories, frameworks, and vulnerability vocabularies remain data/evidence.
|
||||
- **Build versus adopt**: Reuse core canonicalization, case/proof/admission contracts, benchmark artifact envelopes, Git identity resolution, boto3, S3 conditional requests, and RustFS versioning/lifecycle. Build only the missing governance, CAS adapter, general orchestration, and presentation seams.
|
||||
- **Simplification**: No partial-scope custody, no V2 migration, no benchmark metric engine, no notifier transport, no detector plugin framework, and no access-control compatibility adapter in the new path.
|
||||
|
||||
### Updated Risks and Mitigations
|
||||
|
||||
- **Upstream V3 drift** — pin exact schema identifiers and add consumer conformance tests before population spend.
|
||||
- **Sparse scope blocks the whole gate** — run label-blind feasibility while sourcing and classify small freezes as pilots; never open labels to learn sample adequacy.
|
||||
- **RustFS CAS mismatch** — run the exact capability probe against the configured target and fail closed.
|
||||
- **Polyglot ambiguity** — require independent ecosystem review; unresolved rows are insufficient.
|
||||
- **Model influence on labels** — store assistance separately and derive acceptance only from human attestations.
|
||||
- **Historical authorization broadens silently** — normalize paths, verify current bytes, and accept only reviewed exact path/digest inventory rows.
|
||||
- **Post-spend interruption** — persist the spent transition and workflow envelope; resume the same immutable run without restoring untouched status.
|
||||
|
||||
### Design Review Gate Refinements
|
||||
|
||||
The automatic gate required two bounded repair passes. The resulting refinements are architectural commitments, not implementation preferences:
|
||||
|
||||
- **Two-stage V3 authorization**: `EvaluationPlanManifestV3` supports label-blind arithmetic feasibility before arm execution. After every custodian-sealed observation set exists, `EvaluationAdmissionManifestV3` binds their exact bytes and core derives the spend authority against the current custody head.
|
||||
- **One shared multi-arm root**: one reviewed plan and one custody receipt bind independent per-arm scoped reports inside one `EvaluationReplayV3`. Missing any planned arm prevents completion.
|
||||
- **Execute before spend, inspect only after disqualification**: isolated arms run before labels open. Observation sets remain custodian-sealed; only closed counts/digests/reason codes are visible. Human access or any plan/configuration change records `development_opened`, adds influence events, and makes the population admission-ineligible.
|
||||
- **Core-owned ordinary projection**: the freeze stores and executes an ordinary push once with its reviewed ecosystem. Detection-core projects it per claimed family in that ecosystem and deduplicates whole-population coverage.
|
||||
- **Typed proof promotion**: pre-spend proof candidates carry no ground-truth role. The custodian promotes one to `ProofAttachment` only after exact family, pair, role, input, location, differential record, control, and receipt binding succeeds.
|
||||
- **Separate durable authorities**: a content-addressed artifact repository owns immutable events, freezes, sealed observations, terminal outcomes, reports, and replays; custody and workflow heads use stronger conditional-update protocols. Expiring run journals cannot make terminal evidence unreachable.
|
||||
- **Causal lifecycle and workflow chains**: lifecycle events name predecessors and terminal development/retirement cannot be undone. Workflow heartbeats use a CAS head, sequence, predecessor, attempt, and lease owner so concurrent resume and stale renewal are deterministic.
|
||||
- **Explicit predecessor gate**: implementation task generation waits for an approved and validated `detection-core` amendment exporting `openultrasast.detection-core.evaluation-admission.v3`; this spec does not absorb upstream core changes.
|
||||
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"feature_name": "valid-detection-benchmark",
|
||||
"created_at": "2026-10-12T14:38:18Z",
|
||||
"updated_at": "2026-11-06T08:15:41Z",
|
||||
"language": "en",
|
||||
"phase": "design-generated",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"design": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"tasks": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": false,
|
||||
"depends_on": [
|
||||
"detection-core"
|
||||
],
|
||||
"related_specs": [
|
||||
"access-control-prover",
|
||||
"pre-push-safety-net",
|
||||
"search-with-proof"
|
||||
],
|
||||
"approval_note": "2026-11-06: requirements auto-approved by explicit -y request; the replacement design passed the bounded automatic review gate and the maintainer explicitly approved it. Implementation task generation remains gated on an approved, implemented, and independently validated detection-core amendment exporting openultrasast.detection-core.evaluation-admission.v3; valid-detection-benchmark does not own upstream core edits."
|
||||
}
|
||||
Reference in New Issue
Block a user