mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-11-13 17:58:18 +00:00
Marius Ciepluch
2c594b2c27
Lever: unique-declaration member-call binding (dc28325). Measurement 1: zero movement on in-family sites (0 unique-name member calls exist there; unresolved mass is framework callee names). Screen: all 264 admission records byte-identical atdc28325. Measurement 2 (dev pool, single-lever isolation): complete-share 11.9->19.2 (+360478, exact conservation), recall/location/FA identical, 5 ordinary none->indeterminate. Measurement 3 (ac-admission-3, fresh freeze, criteria byte-identical): NO-GO #3, same four gates, 0/12 pairs, 264/264 fresh dc28325-stamped records. Stop condition fired: access-control arm parked permanently (rule-11 fallback). Remaining gap: entry recognition (4/12 zero-recognition; coverage 8/12) and framework operation facts, not call resolution.
91 lines
5.5 KiB
Markdown
91 lines
5.5 KiB
Markdown
# Detection quality: standing rules (2026-10-30)
|
|
|
|
Detection on unseen repositories is the binding constraint of this project. The rules below
|
|
extend the 2026-10-28 gate-5 decision and the Req 12 admission workflow to every spec that
|
|
touches detection. Where this file disagrees with older spec text, this file wins for
|
|
detection claims.
|
|
|
|
## Ordering
|
|
|
|
1. Recognition, then resolution, then judgment. No new fact families, no judge-prompt
|
|
tuning, and no per-framework patching cycles until entry recognition and call resolution
|
|
are measured on the valid pool's ecosystems.
|
|
2. The injection-family spec stays deferred: its gate-5 condition (the post-fix pilot holds
|
|
recall with false alarms moving toward the gate) is **not met**. The 2026-10-30 admission
|
|
run on the untouched pilot population returned NO-GO: 0/12 pairs, in-family 0/3, 9 of 12
|
|
repositories with zero framework recognition (`benchmarks/measurements/2026-10-30-access-control-admission/`).
|
|
|
|
## Comparability
|
|
|
|
3. Every detection measurement publishes one fixed schema: population digest and freeze
|
|
digest, judgment vocabulary, recall / false alarms per 100 / precision / pair
|
|
correctness each with n and Wilson 95%, coverage, instrument failures, runtime. A change
|
|
of population or vocabulary starts a **new baseline**; "worse than last turn" claims
|
|
across vocabularies are invalid. The 2026-10-28 baseline used a binary vocabulary and the
|
|
2026-10-30 amended prover uses the strict three-way lane: 65.2% and 34.8% recall are
|
|
different instruments, not a regression line.
|
|
|
|
## Admission before generalization
|
|
|
|
4. Any detection arm that will feed a pre-push claim must have a development population and a
|
|
separate untouched admission population, consumed through the Req 12.1-12.9 workflow
|
|
(freeze before labels, manifest validation, paired scoring, gates, GO/NO-GO).
|
|
5. No tuning of facts, strategies, thresholds, prompts or judges on admission labels.
|
|
Observing labels during acquisition marks the population spent (Req 12.3).
|
|
6. Admission and rollout reports state the vulnerability families the arm claims (the prover
|
|
claims access_control only; the pilot population holds 3 of 12 in-family).
|
|
7. Retired and spent populations stay retired: no number from the SZZ pool or the
|
|
independent v1/v2 populations is quoted anywhere.
|
|
|
|
## Instrument and honesty
|
|
|
|
8. A zero from an unread input is a failed run. Every measurement artifact proves nonzero
|
|
input reads (hook bytes, record counts) before any clean or zero result is interpreted.
|
|
|
|
## Rollout shape
|
|
|
|
9. Rollout is per capability, per ecosystem, advisory first (M5 already permits this). BLOCK
|
|
only on a demonstrated proof (search-with-proof), never on a confidence score.
|
|
|
|
## Failure modes and stop conditions
|
|
|
|
A self-correcting plan can still converge to failure; correction without a stopping rule is
|
|
slower failure. These conditions are pre-registered now, before the next workstream starts:
|
|
|
|
10. **Two consecutive admission NO-GOs on the same arm against the same pre-registered
|
|
criteria park that arm.** Parked arms are not patched incrementally; they go to a
|
|
strategy review with the maintainer (the 2026-09-28 strategy-review pattern).
|
|
11. **If the entry-recognition + resolution workstream does not move admission past the
|
|
pre-registered criteria on the untouched pilot, the fallback is not another fact
|
|
round:** it is a narrowed product (one ecosystem, advisory-only -- option C of the
|
|
2026-09-28 strategy review) plus hunter-with-proof for BLOCK-grade findings.
|
|
12. **Every correction turn names its lever and its measurement before it starts** --
|
|
recognition / resolution / judgment / facts / judge, and the number that will show the
|
|
lever moved. A turn that cannot name both is not started.
|
|
|
|
### Strategy-review outcome (maintainer decision, 2026-10-31)
|
|
|
|
Rule 10 fired for the access-control arm on 2026-10-30 (second consecutive admission
|
|
NO-GO, `benchmarks/measurements/2026-10-30-access-control-admission-express-pack/`).
|
|
The maintainer reviewed the two diagnoses -- NO-GO #1 recognition starvation (fixed:
|
|
coverage 4/12 -> 8/12), NO-GO #2 operation-facts/resolution starvation (new lever) --
|
|
and authorized **exactly one further gated round on operation facts and call resolution
|
|
for the recognized entry shapes**. This is the final incremental round for this arm:
|
|
a third admission NO-GO parks it permanently regardless of diagnosis, and the rule-11
|
|
fallback (narrowed one-ecosystem advisory product plus proof-based BLOCK) applies.
|
|
|
|
### Final gated round outcome (2026-10-31): ARM PARKED
|
|
|
|
The final round ran its full course (`benchmarks/measurements/2026-10-31-access-control-admission-unique-binding/`):
|
|
the authorized resolution lever (unique-declaration member-call binding, dc28325) moved the
|
|
dev-pool funnel (complete-calls share 11.9% -> 19.2%) with dev recall and false alarms exactly
|
|
unchanged, but is disconnected from the admission population -- zero unique-name member calls
|
|
exist across all 12 admission repositories. Admission run ac-admission-3 reproduced run 2
|
|
exactly: 0/12 pairs, third NO-GO on the same pre-registered criteria. The stop condition
|
|
fired: **the access-control arm is parked permanently**; the rule-11 fallback (narrowed
|
|
one-ecosystem advisory product plus proof-based BLOCK) applies. The binding itself ships --
|
|
it is a strict improvement where repository-defined names exist. Diagnosis for any future
|
|
strategy review: the remaining admission gap is entry recognition (4/12 repositories with
|
|
zero frameworks recognised; coverage 8/12) and operation facts for framework-defined
|
|
operations, not call resolution.
|