From e4de54610af0da116980d68e261f49fa8e232b9c Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Thu, 24 Sep 2026 16:44:32 +0200 Subject: [PATCH] spec(pre-push-safety-net): pre-register the evaluation protocol for the independent population Committed before any case is scanned: what is under test (the analyzer at the runner's commit, per-case family only, four scans per case), how a finding MATCHES a case (a fix-changed file, the case family, the named function or within 10 lines of a fix hunk), detection / fixed-side / benign false alerts, unanswered and instrument failures, the precision sampling rule (every third pooled vulnerable-pin finding, or all if 15 or fewer), what is reported against the M4 gates, and that no retune may be informed by the results. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo --- benchmarks/independent/protocol-v1.md | 61 +++++++++++++++++++++++++++ 1 file changed, 61 insertions(+) create mode 100644 benchmarks/independent/protocol-v1.md diff --git a/benchmarks/independent/protocol-v1.md b/benchmarks/independent/protocol-v1.md new file mode 100644 index 0000000..a39c33e --- /dev/null +++ b/benchmarks/independent/protocol-v1.md @@ -0,0 +1,61 @@ +# Evaluation protocol v1 for the independent population (pre-registered) + +Written and committed on 2026-09-24, BEFORE the first scan of any case in `population-v1.toml`. It fixes what is +measured, how a result is scored and what is reported, so that no rule of the scoring can be chosen after seeing +an outcome. Changing it after the first scan makes the run exploratory, and it must then be reported as such. + +## What is under test + +- The analyzer at the commit that introduces the evaluation runner, exported with + `benchmarks/push/freeze_source.sh`; its hash is recorded in the result. +- Per case, only the case's own family (`injection`, `untrusted_destination` or `config_secrets`), through the + production discovery and scan (`benchmarks/push/finding_dump.py`), unbounded region count (5,000), a 2,400 s + deadline per scan, one engine container at a time. +- Four scans per case: the vulnerable pin, the fixed pin, the benign base and the benign tip. Each checkout is a + `git archive` of the pin, with no project tooling run on it. + +## Scoring, per case + +A finding MATCHES the case when its file is one the fix changes (`git diff --name-only vulnerable fixed`), its +family is the case family, and either its function is one the record names for the sink or fix site, or its line +lies within 10 lines of a hunk the fix changes on that side. + +- **Detected** (recall): at least one matching finding at the vulnerable pin. +- **Fixed-side false alert**: a matching finding at the fixed pin. +- **Benign false alert**: any finding of the case family at the benign tip whose (file, function) has no finding + at the benign base. +- **Unanswered**: the scan completed no question covering the sink file's function. It counts as NOT detected, + and the reason is reported. A scan with no questions at all, or unreadable input, is an instrument failure: it + is rerun once, and if it fails again it is reported as a failure, never as a clean result. + +## Precision + +All findings of the case family at the VULNERABLE pin, across the population, are pooled and sorted by +(case, site). Every third one, starting from the first, is adjudicated against the source, as are all of them if +the pool holds 15 or fewer. The same standard as the PMPro adjudications applies: + +- **True**: attacker-controlled data reaches the sink without a sufficient guard. +- **True, privileged**: the same, reachable only with a privileged role. +- **False, guarded**: a guard makes the flow safe. +- **False, not attacker**: the value does not come from an attacker. + +The matched case findings are part of the pool and are adjudicated like any other. + +## What is reported + +Per case: detected / unanswered, fixed-side and benign false alerts, completion (questions completed / total) +for each of the four scans, and seconds. Overall: + +- recall with a Wilson 95% interval; +- sampled precision with a Wilson 95% interval; +- total fixed-side and benign false alerts. + +These are reported against the M4 profile: at least 95% precision, at least 90% recall, at least 95% completion, +and zero fixed or benign false alerts. A gate that is not met is reported as not met. + +## What may not happen + +- No rule, fact, query or scoring change informed by these results before the result is committed. +- Any change made after it is a retune. A retune can only be qualified on a NEW untouched population, as + release-milestones M4 requires. +- The OpenCVE case is not published until its licence is checked.