mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-11-13 17:58:18 +00:00
docs(steering): detection-quality rules and stop conditions after admission NO-GO
This commit is contained in:
@@ -274,3 +274,13 @@ misses and then remeasured the same population, which is valid regression work b
|
|||||||
promoted facts and include the fact identifier in every match.
|
promoted facts and include the fact identifier in every match.
|
||||||
- **Admission scarcity**: the current pool is already development data. Source and review a separate population
|
- **Admission scarcity**: the current pool is already development data. Source and review a separate population
|
||||||
through `valid-detection-benchmark`; do not relabel the current pool as untouched.
|
through `valid-detection-benchmark`; do not relabel the current pool as untouched.
|
||||||
|
|
||||||
|
## Admission outcome (2026-10-30)
|
||||||
|
|
||||||
|
The first admission run on the untouched pilot population returned **NO-GO** (0/12 pairs,
|
||||||
|
in-family 0/3; 9/12 repositories with zero framework recognition; 0 instrument failures;
|
||||||
|
determinism 4/4). See `benchmarks/measurements/2026-10-30-access-control-admission/`. The
|
||||||
|
next lever is entry-recognition and call-resolution breadth for the valid pool's
|
||||||
|
ecosystems, gated by the same admission workflow. The pilot pool remains unspent: no
|
||||||
|
acquisition observed its labels; only admission scoring read them. Standing rules:
|
||||||
|
`steering/detection-quality.md`.
|
||||||
|
|||||||
@@ -84,3 +84,15 @@ to remove Node from the evaluation or silently fall back to a PHP-only release c
|
|||||||
## Task 1.2 input preparation
|
## Task 1.2 input preparation
|
||||||
|
|
||||||
The versioned push manifest records upstream PMPro vulnerable/fixed revisions, NodeGoat with an explicitly authored eval-removal repair, and comment-only benign changes that retain vulnerable backlog. Ghost is reserved independently for production Node module/async API evaluation; its reviewed supported-family pair population remains missing. The validator verifies immutable Git blob identities and source/license byte counts offline, retains every unresolved case, and keeps all capabilities experimental. Independent PHP and Python admission populations remain prerequisites; VAmPI regressions and the libpng unsupported-property envelope retain their separate roles. See `benchmarks/push/README.md` for the reproducible command and provenance boundaries.
|
The versioned push manifest records upstream PMPro vulnerable/fixed revisions, NodeGoat with an explicitly authored eval-removal repair, and comment-only benign changes that retain vulnerable backlog. Ghost is reserved independently for production Node module/async API evaluation; its reviewed supported-family pair population remains missing. The validator verifies immutable Git blob identities and source/license byte counts offline, retains every unresolved case, and keeps all capabilities experimental. Independent PHP and Python admission populations remain prerequisites; VAmPI regressions and the libpng unsupported-property envelope retain their separate roles. See `benchmarks/push/README.md` for the reproducible command and provenance boundaries.
|
||||||
|
|
||||||
|
## Measurement schema and admission separation (amended 2026-10-30)
|
||||||
|
|
||||||
|
- Every detection measurement on this workload matrix publishes one fixed schema: population
|
||||||
|
and freeze digests, judgment vocabulary, recall / false alarms per 100 / precision / pair
|
||||||
|
correctness each with sample counts and Wilson 95%, coverage, instrument failures, and
|
||||||
|
runtime. A change of population or vocabulary starts a new baseline; cross-vocabulary
|
||||||
|
trend claims are invalid.
|
||||||
|
- M4 qualification reports per family scope (which families the capability claims) and per
|
||||||
|
untouched population, through the Req 12 admission workflow. No tuning on admission labels.
|
||||||
|
- Stop conditions per `steering/detection-quality.md`: two consecutive admission NO-GOs on
|
||||||
|
the same capability park it for a strategy review; no incremental patching of parked arms.
|
||||||
|
|||||||
@@ -255,3 +255,10 @@ arbitration, so a scan could spend everything on queries and report none of what
|
|||||||
On the 500-region population that previously completed zero questions at every budget tried: 271 of
|
On the 500-region population that previously completed zero questions at every budget tried: 271 of
|
||||||
406 completed, 213 findings, 2,950 seconds. The three declared comparisons are running at that
|
406 completed, 213 findings, 2,950 seconds. The three declared comparisons are running at that
|
||||||
budget; detection is not transfer, and only the comparison decides M3.
|
budget; detection is not transfer, and only the comparison decides M3.
|
||||||
|
|
||||||
|
## Detection is the release blocker (2026-10-30)
|
||||||
|
|
||||||
|
Plumbing is implemented; detection on unseen repositories is not demonstrated by any arm
|
||||||
|
(hook 0/12, hunter 5-6/12 at 54 FA/100 on the valid pilot; prover admission NO-GO 0/12 on the
|
||||||
|
untouched pilot). M1b/M2 work continues, but no rollout claim may be made until some family
|
||||||
|
passes admission. Standing rules and stop conditions: `steering/detection-quality.md`.
|
||||||
|
|||||||
@@ -68,3 +68,10 @@ Order is dependency order. Each task is small enough to review on its own. New c
|
|||||||
- Running on kube `ax` (blocked on gVisor; Docker lane is enough here).
|
- Running on kube `ax` (blocked on gVisor; Docker lane is enough here).
|
||||||
- Any new detection capability, including wiring the hunter into the pre-push hook. That is product
|
- Any new detection capability, including wiring the hunter into the pre-push hook. That is product
|
||||||
work in `pre-push-safety-net`; this spec measures the push path with and without it.
|
work in `pre-push-safety-net`; this spec measures the push path with and without it.
|
||||||
|
|
||||||
|
## Group 7 - recognition-coverage target table (2026-10-30)
|
||||||
|
|
||||||
|
- [ ] 7.1 Enumerate each pilot-pool repository's entry framework from its manifest
|
||||||
|
(package.json / pyproject.toml / composer.json, read-only) and record which of the
|
||||||
|
prover's registered entry frameworks match: the recognition-coverage target table for the
|
||||||
|
next detection workstream. No pool labels are read; manifests only.
|
||||||
|
|||||||
@@ -0,0 +1,64 @@
|
|||||||
|
# Detection quality: standing rules (2026-10-30)
|
||||||
|
|
||||||
|
Detection on unseen repositories is the binding constraint of this project. The rules below
|
||||||
|
extend the 2026-10-28 gate-5 decision and the Req 12 admission workflow to every spec that
|
||||||
|
touches detection. Where this file disagrees with older spec text, this file wins for
|
||||||
|
detection claims.
|
||||||
|
|
||||||
|
## Ordering
|
||||||
|
|
||||||
|
1. Recognition, then resolution, then judgment. No new fact families, no judge-prompt
|
||||||
|
tuning, and no per-framework patching cycles until entry recognition and call resolution
|
||||||
|
are measured on the valid pool's ecosystems.
|
||||||
|
2. The injection-family spec stays deferred: its gate-5 condition (the post-fix pilot holds
|
||||||
|
recall with false alarms moving toward the gate) is **not met**. The 2026-10-30 admission
|
||||||
|
run on the untouched pilot population returned NO-GO: 0/12 pairs, in-family 0/3, 9 of 12
|
||||||
|
repositories with zero framework recognition (`benchmarks/measurements/2026-10-30-access-control-admission/`).
|
||||||
|
|
||||||
|
## Comparability
|
||||||
|
|
||||||
|
3. Every detection measurement publishes one fixed schema: population digest and freeze
|
||||||
|
digest, judgment vocabulary, recall / false alarms per 100 / precision / pair
|
||||||
|
correctness each with n and Wilson 95%, coverage, instrument failures, runtime. A change
|
||||||
|
of population or vocabulary starts a **new baseline**; "worse than last turn" claims
|
||||||
|
across vocabularies are invalid. The 2026-10-28 baseline used a binary vocabulary and the
|
||||||
|
2026-10-30 amended prover uses the strict three-way lane: 65.2% and 34.8% recall are
|
||||||
|
different instruments, not a regression line.
|
||||||
|
|
||||||
|
## Admission before generalization
|
||||||
|
|
||||||
|
4. Any detection arm that will feed a pre-push claim must have a development population and a
|
||||||
|
separate untouched admission population, consumed through the Req 12.1-12.9 workflow
|
||||||
|
(freeze before labels, manifest validation, paired scoring, gates, GO/NO-GO).
|
||||||
|
5. No tuning of facts, strategies, thresholds, prompts or judges on admission labels.
|
||||||
|
Observing labels during acquisition marks the population spent (Req 12.3).
|
||||||
|
6. Admission and rollout reports state the vulnerability families the arm claims (the prover
|
||||||
|
claims access_control only; the pilot population holds 3 of 12 in-family).
|
||||||
|
7. Retired and spent populations stay retired: no number from the SZZ pool or the
|
||||||
|
independent v1/v2 populations is quoted anywhere.
|
||||||
|
|
||||||
|
## Instrument and honesty
|
||||||
|
|
||||||
|
8. A zero from an unread input is a failed run. Every measurement artifact proves nonzero
|
||||||
|
input reads (hook bytes, record counts) before any clean or zero result is interpreted.
|
||||||
|
|
||||||
|
## Rollout shape
|
||||||
|
|
||||||
|
9. Rollout is per capability, per ecosystem, advisory first (M5 already permits this). BLOCK
|
||||||
|
only on a demonstrated proof (search-with-proof), never on a confidence score.
|
||||||
|
|
||||||
|
## Failure modes and stop conditions
|
||||||
|
|
||||||
|
A self-correcting plan can still converge to failure; correction without a stopping rule is
|
||||||
|
slower failure. These conditions are pre-registered now, before the next workstream starts:
|
||||||
|
|
||||||
|
10. **Two consecutive admission NO-GOs on the same arm against the same pre-registered
|
||||||
|
criteria park that arm.** Parked arms are not patched incrementally; they go to a
|
||||||
|
strategy review with the maintainer (the 2026-09-28 strategy-review pattern).
|
||||||
|
11. **If the entry-recognition + resolution workstream does not move admission past the
|
||||||
|
pre-registered criteria on the untouched pilot, the fallback is not another fact
|
||||||
|
round:** it is a narrowed product (one ecosystem, advisory-only -- option C of the
|
||||||
|
2026-09-28 strategy review) plus hunter-with-proof for BLOCK-grade findings.
|
||||||
|
12. **Every correction turn names its lever and its measurement before it starts** --
|
||||||
|
recognition / resolution / judgment / facts / judge, and the number that will show the
|
||||||
|
lever moved. A turn that cannot name both is not started.
|
||||||
@@ -1,5 +1,22 @@
|
|||||||
# Overview — what this is, and what is actually true of it
|
# Overview — what this is, and what is actually true of it
|
||||||
|
|
||||||
|
## Current state, 2026-10-30
|
||||||
|
|
||||||
|
Everything below this section is dated history; where it disagrees with this section, this section wins.
|
||||||
|
|
||||||
|
- **Detection on unseen repositories is still the open problem, and now measured honestly.**
|
||||||
|
The valid benchmark pilot (2026-10-12): push hook 0/12, hunter 5-6/12 at 54 FA/100. The
|
||||||
|
access-control prover: 15/23 on the development pool (2026-10-28, binary vocabulary),
|
||||||
|
8/23 on the strict three-way lane (2026-10-30, FA 15.0 -> 5.2/100), and **admission NO-GO
|
||||||
|
0/12 on the untouched pilot** (in-family 0/3; 9/12 repos zero framework recognition). No
|
||||||
|
arm meets the gate; rollout stays NO-GO.
|
||||||
|
- **Standing detection rules** (comparability schema, admission-before-generalization,
|
||||||
|
family-scope disclosure, stop conditions) are in `detection-quality.md`, which wins for
|
||||||
|
detection claims.
|
||||||
|
- **The turns are comparable only within a fixed population and vocabulary.** The apparent
|
||||||
|
recall decline 65.2% -> 34.8% mixes a binary baseline with a strict three-way lane; both
|
||||||
|
numbers are real instruments, neither is a regression line against the other.
|
||||||
|
|
||||||
## Current state, 2026-10-13
|
## Current state, 2026-10-13
|
||||||
|
|
||||||
Everything below this section is dated history; where it disagrees with this section, this section wins.
|
Everything below this section is dated history; where it disagrees with this section, this section wins.
|
||||||
|
|||||||
@@ -1,5 +1,17 @@
|
|||||||
# Roadmap
|
# Roadmap
|
||||||
|
|
||||||
|
## Roadmap -- admission read, stop conditions (2026-10-30)
|
||||||
|
|
||||||
|
This section supersedes the ordering below it for detection claims. The 2026-10-30 admission
|
||||||
|
run (`benchmarks/measurements/2026-10-30-access-control-admission/`, committed 1edc479)
|
||||||
|
returned **NO-GO** on the untouched pilot population: 0/12 pairs, in-family 0/3, 9/12
|
||||||
|
repositories with zero framework recognition, 0 instrument failures. The standing rules,
|
||||||
|
including pre-registered stop conditions (two consecutive admission NO-GOs park an arm;
|
||||||
|
fallback is a narrowed one-ecosystem advisory product plus proof-based BLOCK), live in
|
||||||
|
`detection-quality.md` and win over older sections here. Next lever: entry recognition and
|
||||||
|
call resolution breadth for the valid pool's ecosystems, measured on pool-train and gated on
|
||||||
|
the pilot pool. The injection spec stays deferred; its gate-5 condition is unmet.
|
||||||
|
|
||||||
## Roadmap -- measure the trace first (2026-10-28)
|
## Roadmap -- measure the trace first (2026-10-28)
|
||||||
|
|
||||||
This section supersedes the ordering below it. The older sections stay as history. It builds on the
|
This section supersedes the ordering below it. The older sections stay as history. It builds on the
|
||||||
|
|||||||
@@ -87,6 +87,20 @@ alert admission per capability, and add scheduling sophistication only when it i
|
|||||||
developer outcome. Do not rebuild a custom IR, add a second engine, or assume incremental Joern
|
developer outcome. Do not rebuild a custom IR, add a second engine, or assume incremental Joern
|
||||||
updates as a shortcut to an unmeasured latency promise.
|
updates as a shortcut to an unmeasured latency promise.
|
||||||
|
|
||||||
|
## Detection-quality amendments (2026-10-30)
|
||||||
|
|
||||||
|
- Every detection measurement publishes one fixed schema (population and freeze digests,
|
||||||
|
judgment vocabulary, recall / FA per 100 / precision / pair correctness with n and
|
||||||
|
Wilson 95%, coverage, instrument failures, runtime). A change of population or vocabulary
|
||||||
|
starts a new baseline, not a regression line.
|
||||||
|
- No detection arm feeds a pre-push claim without a separate untouched admission population
|
||||||
|
consumed through the Req 12 workflow; no tuning on admission labels (observing them marks
|
||||||
|
the population spent).
|
||||||
|
- Admission and rollout reports state the vulnerability families the arm claims.
|
||||||
|
- Pre-registered stop conditions (two consecutive admission NO-GOs park an arm; fallback is
|
||||||
|
a narrowed one-ecosystem advisory product plus proof-based BLOCK) are in
|
||||||
|
`detection-quality.md`.
|
||||||
|
|
||||||
## Approval state
|
## Approval state
|
||||||
|
|
||||||
The product direction above is user-requested. The amended requirements, design and initial
|
The product direction above is user-requested. The amended requirements, design and initial
|
||||||
|
|||||||
Reference in New Issue
Block a user