feat(access-control-prover): 13.4 admission measurement

2026-11-13 17:58:18 +00:00 · 2026-10-30 13:25:34 +01:00
parent d063167b28
commit 1edc4790af
10 changed files with 960 additions and 1 deletions
@@ -0,0 +1,119 @@
+# Access-control prover admission measurement (Task 13.4)
+
+First admission run of the amended access-control prover arm on the untouched pilot
+population, consumed through the reviewed `valid-detection-benchmark` manifest. Result:
+**NO-GO** — the development score does not generalize to the admission population.
+
+## What ran
+
+| Input | Value |
+|---|---|
+| Admission manifest | `manifest.json` (this directory), consumed and validated by `benchmarks/valid/admission_run.py` |
+| Analyzer revision | `d063167b28ed7db2335737fdb0dae0555225ea85` (HEAD at freeze) |
+| Fact set | 18 ruleset TOMLs, manifest digest `e7db4031d3d5` (unchanged since the 2026-10-30 four-arm run), bundle artifact digest `3eb5bf699434` |
+| Configuration | `(arm=access-control-admission, judgment_mode=strict)`, digest `095b5a4905ad` |
+| Population | `benchmarks/valid/pilot/pool-valid.toml`, freeze digest `c866c0e812c0`, held-out guard 0, labels 31 confirmed / 0 unchecked |
+| Population contents | 12 repositories, 252 changes (12 vulnerability twins + 240 reviewed ordinary) |
+| Isolation | No pilot repository referenced by any fact artifact (verified at freeze); facts trace to pool-train clones and approved documentation only |
+| Criteria | Pre-registered in `criteria.json` BEFORE the run: min_fix_recall 0.5, min_pair_correctness 0.5, min_verified_location_recall 0.5, min_precision 0.25, max_flagged_ordinary_per_100 10.0, max_indeterminate_rate 0.5 |
+
+The freeze script verified facts-on-disk == facts-at-HEAD, re-derived the pool freeze
+digest, checked the held-out guard, proved isolation, materialised the bundle and wrote
+the manifest and criteria before any prover execution (Req 12.1 ordering).
+
+## Result
+
+| Metric | Value |
+|---|---|
+| Decision | **NO-GO** |
+| Failed gates | `min_fix_recall: 0.0 < 0.5`, `min_pair_correctness: 0.0 < 0.5`, `min_precision: 0.0 < 0.25`, `min_verified_location_recall: not measured (n=0)` |
+| Pairs | n=12, correct=0 |
+| fix_recall | 0/12 (Wilson95 0.0-0.243) |
+| flagged_ordinary_per_100 | 2/240 = 0.83 (Wilson95 0.23-2.99) |
+| precision | 0/14 |
+| indeterminate_rate | 3/264 sides = 1.1% (Wilson95 0.4-3.3) — the three fixed-side locator misses below |
+| coverage | 4/12 repositories (0.333) |
+| runtime | mean 2.8s, max 7.1s per pair; max ordinary change 7.7s (every change budget-gated at 120s, gate passed) |
+| instrument failures | 0 (all 264 records prove nonzero input reads) |
+
+Fixed-side twin protocol: 9 of 12 labelled functions were located by name at the fix
+revision (e.g. dbhub `scanTokenMySQL` shifts from lines 135-143 vulnerable to 158-165
+fixed); the remaining 3 are **locator misses** — the function name is still present at
+the fix but the declaration scan cannot name it (md-editor `fenceGroup`, i18next
+`extendOptionsWithDefaults`, webpack `chartData`, as const-arrows/getters/import-moved
+syntax). Locator misses score `indeterminate` on the fixed side and are never converted
+to clean results (design.md:371); a function genuinely absent from the fixed revision
+would count as quiet ("removed"). No pair was affected: all 3 fixed records carry empty
+evidence and the vulnerable sides were not detected.
+
+## Reading
+
+The zero is a **recognition failure, not a judgment failure**. On 9 of 12 repositories
+the prover recognized no framework at the vulnerable sites, and every vulnerable-side
+judgment funnel came back all-zero: no obligation pair was ever formed, so no guard
+evidence and no unguarded evidence existed. Under the pair vocabulary these sites score
+as completed-quiet ("discharged" per the documented quiet-or-discharged mapping, Req
+12.5); the indeterminate rate of 1.1% counts exactly the three fixed-side locator
+misses — nothing on the vulnerable side was incomplete; the engine simply had no
+knowledge to apply. The obligation facts were promoted from pool-train
+clones (Jenkins/Stapler-shaped Java, plus Spring/Servlet documentation); the admission
+population is TypeScript/JavaScript/Python MCP-and-web ecosystems. Four repositories
+had any recognition at all (fastapi-users 117, malla 161, jupyterlab-lsp 10 framework
+hits, gitlab-mcp 2 on ordinary changes only); none produced a site finding.
+
+The false-alarm side improved against development (0.83 vs 5.2 flagged/100) and the
+budget gate passed everywhere (max 7.1s vs 120s), but with zero recall the arm cannot
+enter pre-push admission. This is exactly the generalisation check Requirement 12
+exists to force: the development evidence (recall 34.8% on pool-train) does not
+transfer to untouched repositories.
+
+## Verification
+
+- Instrument: 264/264 records carry nonzero `hook_bytes_read`; zero zero-byte reads;
+  0 instrument failures; `status=prover_covered` everywhere.
+- Determinism: 4/4 spot re-proofs (2 vulnerable sites + 2 ordinary) byte-identical on
+  findings, funnel, status and read counts; the independent reviewer re-proved 4/4
+  more, including both flagged ordinary records.
+- Unit tests: 24/24 (`tests/test_admission_run.py`). Statics: ruff check, format,
+  C901 and mypy (6-file anchor set) all exit 0.
+- Keyword regression (handover-documented expression): 682 passed + 2 skipped, exit 0.
+  The full suite still fails `test_no_reserved_repository_is_referenced_by_the_development_tree`
+  — a pre-existing clean-HEAD failure from committed 2026-10-12 artifacts, documented
+  at the 2026-10-30 four-arm run and outside this task's boundary.
+- Boundary: no fact, strategy or threshold file was modified (`git status` shows only
+  this measurement directory plus the runner and its tests).
+
+## Corrections (round-1 review)
+
+The first review round rejected the task on two points, both remediated:
+
+1. The fixed-side protocol mapped every locator miss to `fixed_operation="removed"`
+   and thence to a clean `discharged` judgment — an inflationary path that contradicts
+   design.md:371. Remediated by the three-state protocol (located / missed / removed,
+   `classify_fixed_site`), with locator misses scored `indeterminate`. All 12 fixed
+   sides were re-proved under the new stamps (19s); the three misses are disclosed
+   above and the report was re-assembled from the resumable records.
+2. Ordinary changes are now individually budget-gated (design.md:370), and the
+   regression count was re-measured under the documented keyword expression.
+
+No measurement number changed except `indeterminate_rate` (0 -> 3/264, the honest
+locator-miss accounting); the decision remains NO-GO on the same four gates.
+
+## Reproduction
+
+Records: `~/ousast-results/prover/ac-admission-1/{vulnerable,fixed,ordinary}/<change-id>.json`
+(resumable by input digest; `population_digest` on every record is the authoritative
+freeze digest `c866c0e812c0`).
+
+```sh
+.venv/bin/python /tmp/opencode/freeze_134.py        # verify + author manifest + criteria
+.venv/bin/python /tmp/opencode/run_134.py           # prove population, write report.json
+.venv/bin/python /tmp/opencode/determinism_134.py   # 4/4 reproducibility spot-check
+```
+
+The run is resumable: re-running `run_134.py` after the round-1 protocol fix reused
+the 252 unchanged records and re-proved only the 12 fixed sides.
+
+Smoke trail: `/tmp/opencode/smoke_134.py` proved one twin pair end-to-end before the
+full run (fixed-side function located at shifted lines by name, e.g. dbhub
+`scanTokenMySQL` 135-143 vulnerable vs 158-165 fixed).
@@ -0,0 +1,20 @@
+{
+  "c/rules.toml": "88ed02da9fef6f170b912f601df12d3040b990d962b28ce4021f3f335442f713",
+  "families.toml": "5b4c0487c755e8a280e33bbcfb1f497dc40d8703b5961c6f9322a17a3291881a",
+  "frameworks.toml": "cbb9f791e25d73cc215315b2dbfe9654d2ccd5747852e2a0d11ee45ea3012642",
+  "groovy/rules.toml": "ed0ab48e2da2ad9d084a2e5f55ac929e94ea381000d674c160843f79c352396a",
+  "java/rules.toml": "5934e10f595f907f37629d3b05cb56744f70d3c5cb534231b4b3f13a6510c234",
+  "javascript/rules.toml": "075ea7439d65082ecacdf9d9b98397ee278ed93c11ebf552687603d8fb171ce3",
+  "obligations/java.toml": "9125cb707312a72ac766cd3a10e1a09063ac4a2861487ebcb342eadb620e9108",
+  "obligations/javascript.toml": "14670c32a3622b213eb553256a7ffb38c40ffb3398012eb6f02a1e4da04d9a33",
+  "obligations/php.toml": "a7bcbd9bedace740567c5900db91acaa1d0aeb2049dea3101f518acb934827ce",
+  "obligations/python.toml": "f65cb000dbf9e777dce0f2888b0dca39b531b1624dcc3026cc2d55461647b983",
+  "obligations/typescript.toml": "5b9a9bf1bb4c1846a518f724db161b4b5aa533451bc32aaf84ecde851d824bdf",
+  "php/rules.toml": "e839846d760b57c7d65e4fcad6c206e1f1aa852d0ec4bbb9b0b459e761467115",
+  "python/rules.toml": "e323456fd46b5c3e4092a92463695458a620830b6d9b5f05a4ade1e14d78a5c9",
+  "semantic/c.toml": "2d4bbc842596fc17a70e51e528b57af5961ef05515259a57a203faa18018fa4e",
+  "semantic/java.toml": "e7f5b73ffc0fe71d7a9096bdf705ee8cc8e5659e2715cdb1705bd59f6c31dced",
+  "semantic/javascript.toml": "91c6aa895021aff755c4bf4b4ed4f4c7d8a991d260f1f9d9363be68f2134dfd5",
+  "semantic/php.toml": "8c7585c1241bd904558389f66decf88c27554107fc7c6e3fa1b0855797e8ea0c",
+  "semantic/python.toml": "97fd7cfd79ef90a3d51255d951c9f4e1ccabe09d5b6b2574f4ff788556977ab0"
+}
@@ -0,0 +1,8 @@
+{
+  "acquisition_artifact": null,
+  "acquisition_record_digest": "none",
+  "analyzer_revision": "d063167b28ed7db2335737fdb0dae0555225ea85",
+  "configuration_digest": "095b5a4905ad",
+  "fact_artifact": "facts-admission.json",
+  "fact_set_digest": "3eb5bf699434"
+}
@@ -0,0 +1,8 @@
+{
+  "max_flagged_ordinary_per_100": 10.0,
+  "max_indeterminate_rate": 0.5,
+  "min_fix_recall": 0.5,
+  "min_pair_correctness": 0.5,
+  "min_precision": 0.25,
+  "min_verified_location_recall": 0.5
+}
@@ -0,0 +1,20 @@
+{
+  "c/rules.toml": "88ed02da9fef6f170b912f601df12d3040b990d962b28ce4021f3f335442f713",
+  "families.toml": "5b4c0487c755e8a280e33bbcfb1f497dc40d8703b5961c6f9322a17a3291881a",
+  "frameworks.toml": "cbb9f791e25d73cc215315b2dbfe9654d2ccd5747852e2a0d11ee45ea3012642",
+  "groovy/rules.toml": "ed0ab48e2da2ad9d084a2e5f55ac929e94ea381000d674c160843f79c352396a",
+  "java/rules.toml": "5934e10f595f907f37629d3b05cb56744f70d3c5cb534231b4b3f13a6510c234",
+  "javascript/rules.toml": "075ea7439d65082ecacdf9d9b98397ee278ed93c11ebf552687603d8fb171ce3",
+  "obligations/java.toml": "9125cb707312a72ac766cd3a10e1a09063ac4a2861487ebcb342eadb620e9108",
+  "obligations/javascript.toml": "14670c32a3622b213eb553256a7ffb38c40ffb3398012eb6f02a1e4da04d9a33",
+  "obligations/php.toml": "a7bcbd9bedace740567c5900db91acaa1d0aeb2049dea3101f518acb934827ce",
+  "obligations/python.toml": "f65cb000dbf9e777dce0f2888b0dca39b531b1624dcc3026cc2d55461647b983",
+  "obligations/typescript.toml": "5b9a9bf1bb4c1846a518f724db161b4b5aa533451bc32aaf84ecde851d824bdf",
+  "php/rules.toml": "e839846d760b57c7d65e4fcad6c206e1f1aa852d0ec4bbb9b0b459e761467115",
+  "python/rules.toml": "e323456fd46b5c3e4092a92463695458a620830b6d9b5f05a4ade1e14d78a5c9",
+  "semantic/c.toml": "2d4bbc842596fc17a70e51e528b57af5961ef05515259a57a203faa18018fa4e",
+  "semantic/java.toml": "e7f5b73ffc0fe71d7a9096bdf705ee8cc8e5659e2715cdb1705bd59f6c31dced",
+  "semantic/javascript.toml": "91c6aa895021aff755c4bf4b4ed4f4c7d8a991d260f1f9d9363be68f2134dfd5",
+  "semantic/php.toml": "8c7585c1241bd904558389f66decf88c27554107fc7c6e3fa1b0855797e8ea0c",
+  "semantic/python.toml": "97fd7cfd79ef90a3d51255d951c9f4e1ccabe09d5b6b2574f4ff788556977ab0"
+}
@@ -0,0 +1,13 @@
+{
+  "acquisition_record_digest": "none",
+  "analyzer_revision": "d063167b28ed7db2335737fdb0dae0555225ea85",
+  "configuration_digest": "095b5a4905ad",
+  "fact_set_digest": "3eb5bf699434",
+  "label_isolated": true,
+  "labels": [],
+  "population_digest": "c866c0e812c0b54476c7168fdb0e4a20652de0d30485270afecbe51e5c35e424",
+  "population_role": "admission",
+  "produced_by": "valid-detection-benchmark",
+  "reviewed": true,
+  "spent": false
+}
@@ -0,0 +1,90 @@
+{
+  "coverage": 0.3333333333333333,
+  "criteria_pre_registered": {
+    "max_flagged_ordinary_per_100": 10.0,
+    "max_indeterminate_rate": 0.5,
+    "min_fix_recall": 0.5,
+    "min_pair_correctness": 0.5,
+    "min_precision": 0.25,
+    "min_verified_location_recall": 0.5
+  },
+  "decision": "NO-GO",
+  "failed_gates": [
+    "min_fix_recall: 0.0 < 0.5",
+    "min_pair_correctness: 0.0 < 0.5",
+    "min_precision: 0.0 < 0.25",
+    "min_verified_location_recall: not measured (n=0) vs 0.5"
+  ],
+  "fix_recall": {
+    "count": 0,
+    "n": 12,
+    "rate": 0.0,
+    "wilson95": [
+      0,
+      0.2425007574992425
+    ]
+  },
+  "flagged_ordinary_per_100": {
+    "count": 2,
+    "n": 240,
+    "rate": 0.8333333333333334,
+    "wilson95": [
+      0.2288235363161974,
+      2.9870346273441455
+    ]
+  },
+  "indeterminate_rate": {
+    "count": 3,
+    "n": 264,
+    "rate": 0.011363636363636364,
+    "wilson95": [
+      0.003871976570054633,
+      0.03287212890123884
+    ]
+  },
+  "instrument_failures": 0,
+  "manifest": {
+    "acquisition_record_digest": "none",
+    "analyzer_revision": "d063167b28ed7db2335737fdb0dae0555225ea85",
+    "configuration_digest": "095b5a4905ad",
+    "fact_set_digest": "3eb5bf699434",
+    "population_digest": "c866c0e812c0b54476c7168fdb0e4a20652de0d30485270afecbe51e5c35e424",
+    "population_role": "admission"
+  },
+  "pair_correctness": {
+    "count": 0,
+    "n": 12,
+    "rate": 0.0,
+    "wilson95": [
+      0,
+      0.2425007574992425
+    ]
+  },
+  "pairs": {
+    "correct": 0,
+    "n": 12
+  },
+  "precision": {
+    "count": 0,
+    "n": 14,
+    "rate": 0.0,
+    "wilson95": [
+      0,
+      0.21531701192718142
+    ]
+  },
+  "runtime": {
+    "max_seconds": 7.112766742007807,
+    "mean_seconds": 2.8931797610857757,
+    "n": 12
+  },
+  "verified_location_recall": {
+    "count": 0,
+    "n": 0,
+    "rate": null,
+    "wilson95": [
+      null,
+      null
+    ]
+  }
+}