Open_Ultra_SAST/.kiro/steering/detection-quality.md

Detection quality: standing rules (2026-10-30)

Detection on unseen repositories is the binding constraint of this project. The rules below extend the 2026-10-28 gate-5 decision and the Req 12 admission workflow to every spec that touches detection. Where this file disagrees with older spec text, this file wins for detection claims.

Ordering

1. Recognition, then resolution, then judgment. No new fact families, no judge-prompt tuning, and no per-framework patching cycles until entry recognition and call resolution are measured on the valid pool's ecosystems.
2. The injection-family spec stays deferred: its gate-5 condition (the post-fix pilot holds recall with false alarms moving toward the gate) is not met. The 2026-10-30 admission run on the untouched pilot population returned NO-GO: 0/12 pairs, in-family 0/3, 9 of 12 repositories with zero framework recognition (benchmarks/measurements/2026-10-30-access-control-admission/).

Comparability

3. Every detection measurement publishes one fixed schema: population digest and freeze digest, judgment vocabulary, recall / false alarms per 100 / precision / pair correctness each with n and Wilson 95%, coverage, instrument failures, runtime. A change of population or vocabulary starts a new baseline; "worse than last turn" claims across vocabularies are invalid. The 2026-10-28 baseline used a binary vocabulary and the 2026-10-30 amended prover uses the strict three-way lane: 65.2% and 34.8% recall are different instruments, not a regression line.

Admission before generalization

4. Any detection arm that will feed a pre-push claim must have a development population and a separate untouched admission population, consumed through the Req 12.1-12.9 workflow (freeze before labels, manifest validation, paired scoring, gates, GO/NO-GO).
5. No tuning of facts, strategies, thresholds, prompts or judges on admission labels. Observing labels during acquisition marks the population spent (Req 12.3).
6. Admission and rollout reports state the vulnerability families the arm claims (the prover claims access_control only; the pilot population holds 3 of 12 in-family).
7. Retired and spent populations stay retired: no number from the SZZ pool or the independent v1/v2 populations is quoted anywhere.

Instrument and honesty

8. A zero from an unread input is a failed run. Every measurement artifact proves nonzero input reads (hook bytes, record counts) before any clean or zero result is interpreted.

Rollout shape

9. Rollout is per capability, per ecosystem, advisory first (M5 already permits this). BLOCK only on a demonstrated proof (search-with-proof), never on a confidence score.

Failure modes and stop conditions

A self-correcting plan can still converge to failure; correction without a stopping rule is slower failure. These conditions are pre-registered now, before the next workstream starts:

10. Two consecutive admission NO-GOs on the same arm against the same pre-registered criteria park that arm. Parked arms are not patched incrementally; they go to a strategy review with the maintainer (the 2026-09-28 strategy-review pattern).
11. If the entry-recognition + resolution workstream does not move admission past the pre-registered criteria on the untouched pilot, the fallback is not another fact round: it is a narrowed product (one ecosystem, advisory-only -- option C of the 2026-09-28 strategy review) plus hunter-with-proof for BLOCK-grade findings.
12. Every correction turn names its lever and its measurement before it starts -- recognition / resolution / judgment / facts / judge, and the number that will show the lever moved. A turn that cannot name both is not started.