docs(specs): approve scoped evaluation admission design

.kiro/specs/valid-detection-benchmark/requirements.md

@@ -1,89 +1,158 @@
-# Requirements: valid detection benchmark
+# Requirements Document
 
-Turns `brief.md` into testable requirements. The brief supersedes the retired SZZ pool
-(`unseen-repo-evaluation`). This spec builds a benchmark with sound ground truth and standard
-security metrics.
+## Introduction
 
-## Scope decisions (resolving the brief's open questions)
+This specification defines the valid detection benchmark used to determine whether a detector works on application repositories it has not seen. It replaces the 2026-10-12 pilot as admission evidence because that pilot reused repositories protected by the permanent reservation policy. The historical pilot remains preserved as withdrawn evidence, while a separately sourced population establishes a new baseline.
 
-- **First-slice size.** Build a pilot of 10 to 12 verified positives end to end first, to prove the
-  pipeline, then grow to a target of about 40 verified positives. 40 gives a recall interval of roughly
-  ±0.10 at a true recall near 0.85 to 0.90; `score.py:required_n(p, width)` computes the exact number,
-  so the target is set from that, not guessed.
-- **Negative-to-positive ratio.** At least 20 ordinary commits per positive, drawn from the same repos.
-  Negatives are free to collect, and false alarms decide adoption, so err large. Report false alarms as
-  a rate per 100 pushes.
-- **Languages first.** Python and JavaScript/TypeScript web applications (Flask, Django, FastAPI,
-  Express, Koa). These are where the tool is strongest and where in-repo entry points are clear. PHP and
-  others come later, named explicitly as not yet covered.
-- **Curation record.** Each case stores the advisory, repo, fix commit, parent, path, function, the
-  pre-fix and post-fix excerpts, the LLM verdict with confidence, and the human spot-check decision with
-  reviewer and date. The set is frozen with a digest, the same way `advisory-fixes/` freezes its pairs.
-- **Lane.** Curate and score locally on the Docker lane (research is local execution). The kube `ax`
-  lane stays the scale target but is not required here and is still blocked on the gVisor checkpoint path.
+The replacement is a general Python and JavaScript/TypeScript benchmark rather than an access-control-only population. It measures multiple vulnerability families and reports quality for each declared vulnerability-family and ecosystem scope. A whole-population result describes coverage only; it cannot substitute for a scoped quality denominator.
+
+## Boundary Context
+
+- **In scope**: population sourcing and review, fix-derived labels, ordinary-change selection, permanent reservation, development/admission separation, label isolation, lifecycle and custody, reproducible freezing, push-path replay, scoped denominators, standard metrics, failure visibility, and benchmark reporting.
+- **Out of scope**: adding or tuning detection logic, facts, prompts, thresholds, or judges; wiring detector arms into the pre-push product; enabling rollout; reactivating a parked detector arm; and operating execution-plane infrastructure.
+- **Adjacent expectations**: detector arms provide immutable declared identities and claimed scopes; the shared detection contracts validate admission and scoring; the proof workflow supplies demonstrated-exploit evidence; product specifications decide rollout from eligible scoped results.
 
 ## Requirements
 
-### R1. Positives labelled from the fix, never from SZZ
-- Take the vulnerability location from the security fix: the file and function the fix changed, read at
-  the parent commit (pre-fix).
-- Do not run SZZ blame walk-back. The sourcing must not use `extract.py`'s `introducing` or `last_touch`
-  methods. Use the fix site directly (`fix_sites()`) and first-parent history only.
-- Each positive is a pair: the vulnerable function at `parent`, and the same function at the fix commit.
+### Requirement 1: General Multi-Family Population
 
-### R2. App-shaped and reachable in-repo
-- Keep a positive only if the vulnerability is reachable from an entry point inside the repo: a web
-  route, a request handler, or a CLI argument.
-- Drop library-internal cases whose trigger depends on a caller outside the repo. A pre-push tool cannot
-  see that trust boundary, so scoring it there is unfair.
+**Objective:** As a maintainer, I want one general benchmark spanning the initial supported languages and multiple vulnerability families, so that shared detection improvements and class-specific arms can be evaluated without building a new population for every arm.
 
-### R3. Every positive is verified
-- Verify each case with an LLM adjudication call and a human spot-check. The call must confirm:
-  (a) the pre-fix code is vulnerable, (b) the fix addresses it, (c) it is reachable from an in-repo entry
-  point.
-- Record the verdict, the confidence, and the human decision. A case that fails any of (a), (b), (c), or
-  the human spot-check does not enter the benchmark.
+#### Acceptance Criteria
 
-### R4. Negatives are ordinary commits from the same repos
-- Draw non-fix commits from the first-parent history of the same application repos, excluding the fix.
-- The negative set is at least 20x the positive set.
+1. The Valid Detection Benchmark shall include accepted application cases from both Python and JavaScript/TypeScript ecosystems.
+2. The Valid Detection Benchmark shall include accepted positives from at least two distinct vulnerability families overall.
+3. When the full replacement population is accepted, the Valid Detection Benchmark shall contain at least 40 accepted positives overall.
+4. If a frozen population contains fewer than 40 accepted positives, the Valid Detection Benchmark shall classify it as a development pilot or insufficient population and shall not use it as the full replacement admission baseline.
+5. When the benchmark records an ecosystem, the Valid Detection Benchmark shall use one reviewed canonical ecosystem identity for that repository.
+6. If a repository's ecosystem classification remains unresolved, the Valid Detection Benchmark shall record insufficient scope evidence for that repository.
+7. If a vulnerability-family and ecosystem scope lacks enough eligible cases for its pre-registered gate, the Valid Detection Benchmark shall report that scope as insufficient and shall not infer support from the whole-population count.
+8. When concrete frameworks or repositories are cited, the Valid Detection Benchmark shall treat them as coverage evidence rather than as detector implementation requirements.
 
-### R5. Held out
-- No repository in the benchmark was used to build, train, tune, mine, or harvest the tool. Enforce this
-  with the existing used-set canonicalization (`eligibility.normalize`), the same guard the SZZ pool used.
+### Requirement 2: Fix-Derived Positive Ground Truth
 
-### R6. Measure the push path on unseen repos, with the LLM tier
-- The tool a developer runs at push time must include the LLM detection tier (the hunter). That is a
-  product requirement on the pre-push hook, owned by `pre-push-safety-net`. The benchmark measures the
-  push path, not a scan mode.
-- Today the shipped hook runs quick rules and the engine only. Its model path (`push/assistance.py`)
-  selects witnesses for already-admitted defects and cannot add a finding. Until the hunter is wired into
-  the hook, the benchmark reports two arms over the same changes and labels them:
-  - (a) the push hook as shipped (quick rules + engine), through the replay harness;
-  - (b) the push hook plus the hunter arm, the hunter run off-cluster over the same changes and the
-    records combined per change.
-  (b) is the number the tool is held to. (a) is what ships today. The gap between them is product work,
-  not a benchmark artefact, and is reported as such.
-- Use the existing replay harness for (a) so the measured path is the shipped push hook.
+**Objective:** As a benchmark reviewer, I want each positive anchored to a security fix and reachable application behavior, so that recall is measured against reviewed vulnerability locations rather than inferred history.
 
-### R7. Standard metrics
-- Report recall (true-positive rate), precision (positive predictive value), false-positive rate as
-  false alarms per 100 pushes, and exploit-verified rate (the share of flagged positives the proof arm
-  demonstrates with a working proof of concept).
-- Report recall with a Wilson 95% interval and a repo-clustered interval, reusing `score.py`.
-- The headline pairs recall and false alarms per 100 pushes in one plain sentence.
+#### Acceptance Criteria
 
-### R8. Reproducible and frozen
-- The positive catalog, the negative draw, and the held-out guarantee are frozen with digests so a run is
-  reproducible. Re-running the freeze on an unchanged source yields the same digest.
+1. When a positive candidate is sourced, the Valid Detection Benchmark shall derive its vulnerable location from the file and function changed by the security fix at the vulnerable parent revision.
+2. The Valid Detection Benchmark shall not derive a positive location from a blame walk-back or another SZZ-style introducing-commit estimate.
+3. When a positive candidate is reviewed, the Valid Detection Benchmark shall require a human reviewer whose recorded identity differs from the producer's identity, satisfies the pre-registered review policy, and includes a conflict disclosure to confirm that the vulnerable revision contains the defect, the fixed revision addresses it, and an entry point inside the repository can reach it.
+4. If any required ground-truth or reachability judgment is rejected or missing, the Valid Detection Benchmark shall exclude the candidate and record a reason.
+5. Where model-assisted adjudication is used during curation, the Valid Detection Benchmark shall record it as non-authoritative review assistance and shall not treat model agreement as a label.
+6. When one fix has independently verified labels in multiple scopes, the Valid Detection Benchmark shall count the push once in whole-population coverage and once in each verified scoped denominator without granting credit to an unverified family or ecosystem.
 
-## Non-goals
-- No SZZ, no frozen-pool catch/false-alarm numbers (retired).
-- No new detection capability in this spec. This measures the tool that exists.
-- Not required to run on kube `ax` to be valid; the Docker lane is enough for the first numbers.
+### Requirement 3: Representative Ordinary Changes
 
-## Acceptance
-- A frozen pilot of at least 10 verified positives plus their negatives, on held-out Python/JS repos,
-  run through the whole tool, producing recall, precision, false alarms per 100 pushes, and
-  exploit-verified rate, with intervals. Every positive has an LLM verdict and a human spot-check on
-  record.
+**Objective:** As a developer, I want false alarms measured on ordinary changes from the same application environments, so that the reported interruption rate reflects the push experience.
+
+#### Acceptance Criteria
+
+1. When an accepted positive repository contributes negatives, the Valid Detection Benchmark shall draw ordinary non-security changes from that repository's first-parent history and shall exclude the security fix.
+2. The Valid Detection Benchmark shall include at least 20 eligible ordinary pushes per accepted positive in each frozen population.
+3. When calculating a scoped false-alarm denominator, the Valid Detection Benchmark shall include each eligible ordinary push from the matching ecosystem once for each vulnerability family claimed by the evaluated arm in that ecosystem.
+4. The Valid Detection Benchmark shall not pool an ordinary push from one ecosystem into another ecosystem's denominator.
+5. If an ordinary change is verified to contain a relevant security defect before freeze, the Valid Detection Benchmark shall exclude it from the population and record the reason.
+6. If an ordinary change is verified to contain a relevant security defect after freeze, the Valid Detection Benchmark shall retire the affected baseline rather than silently rewriting its recorded result.
+
+### Requirement 4: Permanent Reservation and Historical Withdrawal
+
+**Objective:** As a benchmark custodian, I want prior repository use and withdrawn populations handled explicitly, so that development knowledge cannot leak into new admission evidence.
+
+#### Acceptance Criteria
+
+1. The Valid Detection Benchmark shall exclude every repository identity present in the authoritative permanent-reservation census.
+2. When repository aliases, former names, redirects, or fork relationships resolve to a reserved identity, the Valid Detection Benchmark shall treat the candidate as reserved.
+3. When repository code, labels, or metadata influence population selection, detector development, training, tuning, mining, fact promotion, prompt selection, or threshold selection, the Valid Detection Benchmark shall add that repository identity to the permanent-reservation census.
+4. If a repository was only enumerated and none of its code, labels, or metadata influenced a decision, the Valid Detection Benchmark shall not classify it as used unless another reservation source includes it.
+5. The Valid Detection Benchmark shall classify the 2026-10-12 pilot as withdrawn and non-admissible and shall not present its results as current qualification evidence.
+6. When the historical pilot is withdrawn, the Valid Detection Benchmark shall preserve its bytes and digests.
+7. When the historical pilot is withdrawn, the Valid Detection Benchmark shall add a reviewed lifecycle record containing the withdrawal reason and replacement relationship.
+8. If historical identity-bearing material remains accessible for audit, the Valid Detection Benchmark shall authorize it only by exact path and digest and shall not create a blanket reservation-guard exclusion.
+9. When the replacement population is frozen, the Valid Detection Benchmark shall use new population and freeze identities and shall not replace or rewrite the historical pilot in place.
+10. When a replacement population is about to be accepted, the Valid Detection Benchmark shall pass the permanent-reservation guard while permitting only reviewed exact-path-and-digest historical authorizations and without weakening reservation semantics.
+
+### Requirement 5: Development and Admission Separation
+
+**Objective:** As a release decision maker, I want detector development separated from untouched admission, so that measured qualification is not influenced by the labels it evaluates.
+
+#### Acceptance Criteria
+
+1. The Valid Detection Benchmark shall maintain disjoint development and admission populations.
+2. When a population or slice is inspected to diagnose, tune, select, or promote detector behavior, the Valid Detection Benchmark shall classify it as development or spent and shall not use it for admission.
+3. When admission is prepared and before labels are opened, the Valid Detection Benchmark shall freeze a label-blind manifest containing only scope census information and digests.
+4. When a label-blind manifest is reviewed, the Valid Detection Benchmark shall require a reviewer whose recorded identity differs from the producer's identity, satisfies the pre-registered review policy, and includes a conflict disclosure.
+5. When admission is prepared and before labels are opened, the Valid Detection Benchmark shall bind the population, analyzer, configuration, fact set, judgment vocabulary, outcome mapping, denominator policy, thresholds, and claimed scopes to immutable identities.
+6. If admission labels influence facts, strategies, matchers, prompts, examples, thresholds, judges, decision mappings, or promotions, the Valid Detection Benchmark shall mark the population spent and refuse an admission result from it.
+7. When admission labels are released, the Valid Detection Benchmark shall durably record the spent transition before exposing any label bytes.
+8. When a post-spend case is released for scoring, the Valid Detection Benchmark shall bind the case to its scope, role, pair identity, result, evidence, change inputs, and reviewed input manifest.
+9. When a post-spend case release is reviewed, the Valid Detection Benchmark shall require a reviewer whose recorded identity differs from the producer's identity, satisfies the pre-registered review policy, and includes a conflict disclosure.
+
+### Requirement 6: Scoped Denominators and Feasible Gates
+
+**Objective:** As a detector owner, I want quality evaluated only on the families and ecosystems an arm claims, so that out-of-scope cases neither dilute nor inflate its result.
+
+#### Acceptance Criteria
+
+1. When an arm is registered and before label access, the Valid Detection Benchmark shall require the arm to declare every vulnerability-family and ecosystem scope it claims.
+2. When recall or pair correctness is calculated for a scope, the Valid Detection Benchmark shall use only vulnerable and fixed cases independently verified for that exact family and ecosystem.
+3. When false alarms and precision are calculated for a scope, the Valid Detection Benchmark shall use only findings and ordinary pushes assigned to that exact family and ecosystem under the pre-registered denominator policy.
+4. The Valid Detection Benchmark shall report whole-population coverage separately from scoped quality metrics.
+5. Where an overall aggregate is provided, the Valid Detection Benchmark shall identify its constituent scopes, sample counts, and weighting and shall not use that aggregate to authorize a capability or ecosystem claim.
+6. When a gate is registered and before label access, the Valid Detection Benchmark shall prove from the label-blind census that every pre-registered threshold is arithmetically reachable for its declared denominator.
+7. If a threshold is impossible or a scoped denominator fails its pre-registered minimum size or interval-precision rule, the Valid Detection Benchmark shall refuse to spend the population for that gate and shall report the scope as insufficient.
+
+### Requirement 7: Reproducible Population and Durable Custody
+
+**Objective:** As an operator, I want population state and provenance to survive restart or local-disk loss, so that a long sourcing or admission workflow remains reproducible and auditable.
+
+#### Acceptance Criteria
+
+1. When a population is frozen, the Valid Detection Benchmark shall bind its positive selection, ordinary-change draw, reservation census, review records, exclusion counts, and provenance to content digests.
+2. When unchanged source material is frozen again under the same policy, the Valid Detection Benchmark shall produce the same population identity.
+3. If an existing frozen identity resolves to different content, the Valid Detection Benchmark shall refuse the write and preserve the prior artifact.
+4. The Valid Detection Benchmark shall keep durable population state, custody transitions, provenance, and resumable sourcing state in a versioned durable store rather than relying on local scratch files.
+5. If local scratch state is lost or execution restarts, the Valid Detection Benchmark shall resume from durable custody without changing accepted decisions or identities.
+6. Where material is publicly distributed, the Valid Detection Benchmark shall include only permissively licensed content; non-permissive or uncertain material shall remain private behind opaque digest-bound references.
+7. The Valid Detection Benchmark shall retain durable population, provenance, lifecycle, and custody records in an auditable form.
+8. When transient run records are created, the Valid Detection Benchmark shall apply a pre-registered time-bounded retention policy that expires both current and noncurrent versions by their recorded deadlines.
+
+### Requirement 8: Standard Measurement and Failure Visibility
+
+**Objective:** As a reviewer, I want every benchmark result to carry the same complete measurement record, so that a clean result, an unsupported result, and a failed instrument cannot be confused.
+
+#### Acceptance Criteria
+
+1. When a benchmark result is published, the Valid Detection Benchmark shall identify the population and freeze digests, judgment vocabulary, claimed scopes, denominator policy, analyzer and fact identities, and threshold set.
+2. When a scoped result is published, the Valid Detection Benchmark shall report recall, false alarms per 100 ordinary pushes, precision, pair correctness, sample counts, and Wilson 95% intervals.
+3. When a benchmark result is published, the Valid Detection Benchmark shall report whole-population coverage, completed and deferred work, instrument and model failures, runtime, and every degradation with its reason.
+4. When a zero or clean-looking result is about to be interpreted, the Valid Detection Benchmark shall prove nonzero input reads and execution of every stage required by the declared arm.
+5. If an input is unreadable, empty, skipped, prematurely terminated, or processed by a failed required stage, the Valid Detection Benchmark shall report an instrument failure or bounded incomplete result and shall not count it as clean.
+6. When the population, vocabulary, denominator, claimed scope, analyzer identity, fact identity, threshold set, or outcome mapping changes, the Valid Detection Benchmark shall start a new baseline and shall not describe the new result as a regression trend against an incompatible baseline.
+
+### Requirement 9: Push-Path Arms and Demonstrated Evidence
+
+**Objective:** As a product owner, I want benchmark arms and proof strength reported distinctly, so that experimental assistance is not mistaken for shipped behavior or demonstrated exploitation.
+
+#### Acceptance Criteria
+
+1. When the shipped push path and an experimental detector arm are evaluated, the Valid Detection Benchmark shall report their results separately over the same eligible changes.
+2. Where a pre-registered combined arm is evaluated, the Valid Detection Benchmark shall identify its combination rule before label access and shall preserve each contributing arm's result.
+3. When an exploit-demonstrated rate is reported, the Valid Detection Benchmark shall require a matching differential proof artifact showing the owned effect on the vulnerable revision and its absence on the fixed revision.
+4. If a flagged positive lacks a matching differential proof artifact, the Valid Detection Benchmark shall report it as not demonstrated rather than as a failed exploit or an exploit-verified finding.
+5. The Valid Detection Benchmark shall not treat scanner verification, model confidence, or model agreement as demonstrated-exploit evidence.
+6. When reporting advisory eligibility, the Valid Detection Benchmark shall scope the result per capability and ecosystem.
+7. When reporting a blocking claim, the Valid Detection Benchmark shall require demonstrated proof.
+
+### Requirement 10: Operator Outcomes and Feature Boundary
+
+**Objective:** As an operator, I want benchmark completion and impairment states to be explicit while detector and infrastructure decisions remain with their owning workflows.
+
+#### Acceptance Criteria
+
+1. When a benchmark workflow completes, the Valid Detection Benchmark shall expose its status, counts, digests, scoped GO, NO-GO, or insufficient outcomes, and named failed gates.
+2. If a benchmark workflow stalls, times out, exhausts a budget or resource, produces an insufficient result, or encounters an instrument failure, the Valid Detection Benchmark shall expose the impaired state and its reason.
+3. The Valid Detection Benchmark shall not include secrets, exploit payloads, private labels, repository contents, or customer content in shared reports or operator notifications.
+4. The Valid Detection Benchmark shall not modify detector logic, facts, prompts, judges, thresholds, or product rollout state while constructing or scoring a population.
+5. The Valid Detection Benchmark shall not reactivate or requalify the parked access-control arm without a separate maintainer strategy decision.
+6. Where an execution plane is used, the Valid Detection Benchmark shall report benchmark process outcomes without assuming ownership of worker, cluster, or network infrastructure health.