docs(steering): detection-quality rules and stop conditions after admission NO-GO

This commit is contained in:
Marius Ciepluch
2026-10-30 14:48:40 +01:00
parent 1edc4790af
commit 839d58e3d3
8 changed files with 143 additions and 0 deletions
@@ -274,3 +274,13 @@ misses and then remeasured the same population, which is valid regression work b
promoted facts and include the fact identifier in every match.
- **Admission scarcity**: the current pool is already development data. Source and review a separate population
through `valid-detection-benchmark`; do not relabel the current pool as untouched.
## Admission outcome (2026-10-30)
The first admission run on the untouched pilot population returned **NO-GO** (0/12 pairs,
in-family 0/3; 9/12 repositories with zero framework recognition; 0 instrument failures;
determinism 4/4). See `benchmarks/measurements/2026-10-30-access-control-admission/`. The
next lever is entry-recognition and call-resolution breadth for the valid pool's
ecosystems, gated by the same admission workflow. The pilot pool remains unspent: no
acquisition observed its labels; only admission scoring read them. Standing rules:
`steering/detection-quality.md`.
@@ -84,3 +84,15 @@ to remove Node from the evaluation or silently fall back to a PHP-only release c
## Task 1.2 input preparation
The versioned push manifest records upstream PMPro vulnerable/fixed revisions, NodeGoat with an explicitly authored eval-removal repair, and comment-only benign changes that retain vulnerable backlog. Ghost is reserved independently for production Node module/async API evaluation; its reviewed supported-family pair population remains missing. The validator verifies immutable Git blob identities and source/license byte counts offline, retains every unresolved case, and keeps all capabilities experimental. Independent PHP and Python admission populations remain prerequisites; VAmPI regressions and the libpng unsupported-property envelope retain their separate roles. See `benchmarks/push/README.md` for the reproducible command and provenance boundaries.
## Measurement schema and admission separation (amended 2026-10-30)
- Every detection measurement on this workload matrix publishes one fixed schema: population
and freeze digests, judgment vocabulary, recall / false alarms per 100 / precision / pair
correctness each with sample counts and Wilson 95%, coverage, instrument failures, and
runtime. A change of population or vocabulary starts a new baseline; cross-vocabulary
trend claims are invalid.
- M4 qualification reports per family scope (which families the capability claims) and per
untouched population, through the Req 12 admission workflow. No tuning on admission labels.
- Stop conditions per `steering/detection-quality.md`: two consecutive admission NO-GOs on
the same capability park it for a strategy review; no incremental patching of parked arms.
@@ -255,3 +255,10 @@ arbitration, so a scan could spend everything on queries and report none of what
On the 500-region population that previously completed zero questions at every budget tried: 271 of
406 completed, 213 findings, 2,950 seconds. The three declared comparisons are running at that
budget; detection is not transfer, and only the comparison decides M3.
## Detection is the release blocker (2026-10-30)
Plumbing is implemented; detection on unseen repositories is not demonstrated by any arm
(hook 0/12, hunter 5-6/12 at 54 FA/100 on the valid pilot; prover admission NO-GO 0/12 on the
untouched pilot). M1b/M2 work continues, but no rollout claim may be made until some family
passes admission. Standing rules and stop conditions: `steering/detection-quality.md`.
@@ -68,3 +68,10 @@ Order is dependency order. Each task is small enough to review on its own. New c
- Running on kube `ax` (blocked on gVisor; Docker lane is enough here).
- Any new detection capability, including wiring the hunter into the pre-push hook. That is product
work in `pre-push-safety-net`; this spec measures the push path with and without it.
## Group 7 - recognition-coverage target table (2026-10-30)
- [ ] 7.1 Enumerate each pilot-pool repository's entry framework from its manifest
(package.json / pyproject.toml / composer.json, read-only) and record which of the
prover's registered entry frameworks match: the recognition-coverage target table for the
next detection workstream. No pool labels are read; manifests only.
+64
View File
@@ -0,0 +1,64 @@
# Detection quality: standing rules (2026-10-30)
Detection on unseen repositories is the binding constraint of this project. The rules below
extend the 2026-10-28 gate-5 decision and the Req 12 admission workflow to every spec that
touches detection. Where this file disagrees with older spec text, this file wins for
detection claims.
## Ordering
1. Recognition, then resolution, then judgment. No new fact families, no judge-prompt
tuning, and no per-framework patching cycles until entry recognition and call resolution
are measured on the valid pool's ecosystems.
2. The injection-family spec stays deferred: its gate-5 condition (the post-fix pilot holds
recall with false alarms moving toward the gate) is **not met**. The 2026-10-30 admission
run on the untouched pilot population returned NO-GO: 0/12 pairs, in-family 0/3, 9 of 12
repositories with zero framework recognition (`benchmarks/measurements/2026-10-30-access-control-admission/`).
## Comparability
3. Every detection measurement publishes one fixed schema: population digest and freeze
digest, judgment vocabulary, recall / false alarms per 100 / precision / pair
correctness each with n and Wilson 95%, coverage, instrument failures, runtime. A change
of population or vocabulary starts a **new baseline**; "worse than last turn" claims
across vocabularies are invalid. The 2026-10-28 baseline used a binary vocabulary and the
2026-10-30 amended prover uses the strict three-way lane: 65.2% and 34.8% recall are
different instruments, not a regression line.
## Admission before generalization
4. Any detection arm that will feed a pre-push claim must have a development population and a
separate untouched admission population, consumed through the Req 12.1-12.9 workflow
(freeze before labels, manifest validation, paired scoring, gates, GO/NO-GO).
5. No tuning of facts, strategies, thresholds, prompts or judges on admission labels.
Observing labels during acquisition marks the population spent (Req 12.3).
6. Admission and rollout reports state the vulnerability families the arm claims (the prover
claims access_control only; the pilot population holds 3 of 12 in-family).
7. Retired and spent populations stay retired: no number from the SZZ pool or the
independent v1/v2 populations is quoted anywhere.
## Instrument and honesty
8. A zero from an unread input is a failed run. Every measurement artifact proves nonzero
input reads (hook bytes, record counts) before any clean or zero result is interpreted.
## Rollout shape
9. Rollout is per capability, per ecosystem, advisory first (M5 already permits this). BLOCK
only on a demonstrated proof (search-with-proof), never on a confidence score.
## Failure modes and stop conditions
A self-correcting plan can still converge to failure; correction without a stopping rule is
slower failure. These conditions are pre-registered now, before the next workstream starts:
10. **Two consecutive admission NO-GOs on the same arm against the same pre-registered
criteria park that arm.** Parked arms are not patched incrementally; they go to a
strategy review with the maintainer (the 2026-09-28 strategy-review pattern).
11. **If the entry-recognition + resolution workstream does not move admission past the
pre-registered criteria on the untouched pilot, the fallback is not another fact
round:** it is a narrowed product (one ecosystem, advisory-only -- option C of the
2026-09-28 strategy review) plus hunter-with-proof for BLOCK-grade findings.
12. **Every correction turn names its lever and its measurement before it starts** --
recognition / resolution / judgment / facts / judge, and the number that will show the
lever moved. A turn that cannot name both is not started.
+17
View File
@@ -1,5 +1,22 @@
# Overview — what this is, and what is actually true of it
## Current state, 2026-10-30
Everything below this section is dated history; where it disagrees with this section, this section wins.
- **Detection on unseen repositories is still the open problem, and now measured honestly.**
The valid benchmark pilot (2026-10-12): push hook 0/12, hunter 5-6/12 at 54 FA/100. The
access-control prover: 15/23 on the development pool (2026-10-28, binary vocabulary),
8/23 on the strict three-way lane (2026-10-30, FA 15.0 -> 5.2/100), and **admission NO-GO
0/12 on the untouched pilot** (in-family 0/3; 9/12 repos zero framework recognition). No
arm meets the gate; rollout stays NO-GO.
- **Standing detection rules** (comparability schema, admission-before-generalization,
family-scope disclosure, stop conditions) are in `detection-quality.md`, which wins for
detection claims.
- **The turns are comparable only within a fixed population and vocabulary.** The apparent
recall decline 65.2% -> 34.8% mixes a binary baseline with a strict three-way lane; both
numbers are real instruments, neither is a regression line against the other.
## Current state, 2026-10-13
Everything below this section is dated history; where it disagrees with this section, this section wins.
+12
View File
@@ -1,5 +1,17 @@
# Roadmap
## Roadmap -- admission read, stop conditions (2026-10-30)
This section supersedes the ordering below it for detection claims. The 2026-10-30 admission
run (`benchmarks/measurements/2026-10-30-access-control-admission/`, committed 1edc479)
returned **NO-GO** on the untouched pilot population: 0/12 pairs, in-family 0/3, 9/12
repositories with zero framework recognition, 0 instrument failures. The standing rules,
including pre-registered stop conditions (two consecutive admission NO-GOs park an arm;
fallback is a narrowed one-ecosystem advisory product plus proof-based BLOCK), live in
`detection-quality.md` and win over older sections here. Next lever: entry recognition and
call resolution breadth for the valid pool's ecosystems, measured on pool-train and gated on
the pilot pool. The injection spec stays deferred; its gate-5 condition is unmet.
## Roadmap -- measure the trace first (2026-10-28)
This section supersedes the ordering below it. The older sections stay as history. It builds on the
+14
View File
@@ -87,6 +87,20 @@ alert admission per capability, and add scheduling sophistication only when it i
developer outcome. Do not rebuild a custom IR, add a second engine, or assume incremental Joern
updates as a shortcut to an unmeasured latency promise.
## Detection-quality amendments (2026-10-30)
- Every detection measurement publishes one fixed schema (population and freeze digests,
judgment vocabulary, recall / FA per 100 / precision / pair correctness with n and
Wilson 95%, coverage, instrument failures, runtime). A change of population or vocabulary
starts a new baseline, not a regression line.
- No detection arm feeds a pre-push claim without a separate untouched admission population
consumed through the Req 12 workflow; no tuning on admission labels (observing them marks
the population spent).
- Admission and rollout reports state the vulnerability families the arm claims.
- Pre-registered stop conditions (two consecutive admission NO-GOs park an arm; fallback is
a narrowed one-ecosystem advisory product plus proof-based BLOCK) are in
`detection-quality.md`.
## Approval state
The product direction above is user-requested. The amended requirements, design and initial