mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-11-13 17:58:18 +00:00
docs(steering): detection-quality rules and stop conditions after admission NO-GO
This commit is contained in:
@@ -274,3 +274,13 @@ misses and then remeasured the same population, which is valid regression work b
|
||||
promoted facts and include the fact identifier in every match.
|
||||
- **Admission scarcity**: the current pool is already development data. Source and review a separate population
|
||||
through `valid-detection-benchmark`; do not relabel the current pool as untouched.
|
||||
|
||||
## Admission outcome (2026-10-30)
|
||||
|
||||
The first admission run on the untouched pilot population returned **NO-GO** (0/12 pairs,
|
||||
in-family 0/3; 9/12 repositories with zero framework recognition; 0 instrument failures;
|
||||
determinism 4/4). See `benchmarks/measurements/2026-10-30-access-control-admission/`. The
|
||||
next lever is entry-recognition and call-resolution breadth for the valid pool's
|
||||
ecosystems, gated by the same admission workflow. The pilot pool remains unspent: no
|
||||
acquisition observed its labels; only admission scoring read them. Standing rules:
|
||||
`steering/detection-quality.md`.
|
||||
|
||||
@@ -84,3 +84,15 @@ to remove Node from the evaluation or silently fall back to a PHP-only release c
|
||||
## Task 1.2 input preparation
|
||||
|
||||
The versioned push manifest records upstream PMPro vulnerable/fixed revisions, NodeGoat with an explicitly authored eval-removal repair, and comment-only benign changes that retain vulnerable backlog. Ghost is reserved independently for production Node module/async API evaluation; its reviewed supported-family pair population remains missing. The validator verifies immutable Git blob identities and source/license byte counts offline, retains every unresolved case, and keeps all capabilities experimental. Independent PHP and Python admission populations remain prerequisites; VAmPI regressions and the libpng unsupported-property envelope retain their separate roles. See `benchmarks/push/README.md` for the reproducible command and provenance boundaries.
|
||||
|
||||
## Measurement schema and admission separation (amended 2026-10-30)
|
||||
|
||||
- Every detection measurement on this workload matrix publishes one fixed schema: population
|
||||
and freeze digests, judgment vocabulary, recall / false alarms per 100 / precision / pair
|
||||
correctness each with sample counts and Wilson 95%, coverage, instrument failures, and
|
||||
runtime. A change of population or vocabulary starts a new baseline; cross-vocabulary
|
||||
trend claims are invalid.
|
||||
- M4 qualification reports per family scope (which families the capability claims) and per
|
||||
untouched population, through the Req 12 admission workflow. No tuning on admission labels.
|
||||
- Stop conditions per `steering/detection-quality.md`: two consecutive admission NO-GOs on
|
||||
the same capability park it for a strategy review; no incremental patching of parked arms.
|
||||
|
||||
@@ -255,3 +255,10 @@ arbitration, so a scan could spend everything on queries and report none of what
|
||||
On the 500-region population that previously completed zero questions at every budget tried: 271 of
|
||||
406 completed, 213 findings, 2,950 seconds. The three declared comparisons are running at that
|
||||
budget; detection is not transfer, and only the comparison decides M3.
|
||||
|
||||
## Detection is the release blocker (2026-10-30)
|
||||
|
||||
Plumbing is implemented; detection on unseen repositories is not demonstrated by any arm
|
||||
(hook 0/12, hunter 5-6/12 at 54 FA/100 on the valid pilot; prover admission NO-GO 0/12 on the
|
||||
untouched pilot). M1b/M2 work continues, but no rollout claim may be made until some family
|
||||
passes admission. Standing rules and stop conditions: `steering/detection-quality.md`.
|
||||
|
||||
@@ -68,3 +68,10 @@ Order is dependency order. Each task is small enough to review on its own. New c
|
||||
- Running on kube `ax` (blocked on gVisor; Docker lane is enough here).
|
||||
- Any new detection capability, including wiring the hunter into the pre-push hook. That is product
|
||||
work in `pre-push-safety-net`; this spec measures the push path with and without it.
|
||||
|
||||
## Group 7 - recognition-coverage target table (2026-10-30)
|
||||
|
||||
- [ ] 7.1 Enumerate each pilot-pool repository's entry framework from its manifest
|
||||
(package.json / pyproject.toml / composer.json, read-only) and record which of the
|
||||
prover's registered entry frameworks match: the recognition-coverage target table for the
|
||||
next detection workstream. No pool labels are read; manifests only.
|
||||
|
||||
Reference in New Issue
Block a user