measure(prover): gate 1 on the 23-repo training pool: recall 65.2% (Wilson 44.9-81.2), 15.0 FA/100, 0 instrument failures; 4/5 on the hunt overlap incl. trigger.dev

.kiro/specs/access-control-prover/tasks.md

@@ -107,10 +107,14 @@
   - _Depends: 4.1_
 
 - [ ] 6. Measurement (maintainer, not Codex)
-- [ ] 6.1 Run the arm on `benchmarks/valid/train/pool-train.toml` over the training clones; write
+- [x] 6.1 Run the arm on `benchmarks/valid/train/pool-train.toml` over the training clones; write
   `benchmarks/measurements/<date>-access-control-prover/` with NOTES, summary, coverage table; compare with the
   DSPy and hunter arms of the same pool
   - _Requirements: 5.2, 5.3, 7.2_
+  - Done 2026-10-28: `benchmarks/measurements/2026-10-28-access-control-prover/` -- 23-repo
+    pool (483 changes, 93 locations), recall 65.2% (Wilson 44.9-81.2), 15.0 FA/100, 0
+    instrument failures; on the 5 repos shared with the 2026-10-14 hunt arms, prover 4/5 vs
+    DSPy 2/5 and hunter 2/5 (union 4/5), including trigger.dev which neither arm found.
 
 
 ## Implementation Notes

.kiro/steering/roadmap.md

@@ -1,5 +1,55 @@
 # Roadmap
 
+## Roadmap -- measure the trace first (2026-10-28)
+
+This section supersedes the ordering below it. The older sections stay as history. It builds on the
+2026-10-14 decision-strategy correction (steering `overview.md`): the trace decides, a judge reads it
+zero-shot, the pool measures and never trains.
+
+### Decision (2026-10-28, maintainer session)
+
+The DSPy annotation loop was built and measured; its premise -- that shape annotations teach the hunter --
+was falsified by its own instruments (`benchmarks/measurements/2026-10-13-access-control-screen/`,
+`2026-10-13-hunt-repeats/`, `2026-10-14-access-control-hunt/`). The surviving direction inverts the
+augmentation: compute the deterministic security object (the ordered trace) first, and let the LLM judge
+the residual question on it. Before anything generalizes, that direction gets its number. No new spec
+starts until the gates below are read.
+
+### The sequence, each step a gate
+
+1. `access-control-prover` task 6.1: run the prover arm over `benchmarks/valid/train/pool-train.toml` --
+   the first full-repository recall and false-alarm numbers for the trace. No model calls. Record the
+   provider error code for any `LMInvalidRequestError`-class failure (the 2026-10-14 run's failures sat on
+   38 of 48 locations), and read the summary's `uncovered` repositories before calling any miss a logic
+   failure rather than a fact-coverage gap.
+   - **Read 2026-10-28** (`benchmarks/measurements/2026-10-28-access-control-prover/`): 23-repo pool,
+     483 changes, 93 locations -- recall 65.2% (Wilson 44.9-81.2), 15.0 FA/100, 0 instrument failures.
+     On the 5 repos shared with the 2026-10-14 arms: prover 4/5 vs DSPy 2/5, hunter 2/5 (union 4/5),
+     including trigger.dev which neither arm found. Misses: 2 fact-coverage (seatunnel, byobu -- no
+     entry points), 6 covered-but-unflagged for gate 3. Gate 1 holds; proceed to gate 2.
+2. Zero-shot judge pass over those traces ("is this entry meant to be public"): does it drop false alarms
+   without killing recall? The pool reports how often the judge is right (correction, rule 3).
+3. Trace-component ablation (the amended `hunter-learning-loop` role -- selection, not fitting): which
+   trace components the judge needs to see, with intervals.
+4. One pilot run under the at-most-once rule (`hunter-learning-loop` req 8.6): trace + judge over the 12
+   verified positives and the ordinary pushes. Read the gate distance off the curve (recall at 10 false
+   alarms per 100 pushes, false alarms at 90% recall), per family -- the first such number for the new
+   direction.
+5. Decision:
+   - access_control holds recall with false alarms moving toward the gate -> formalize the injection
+     security object (`engine-change-scoped-analysis` brief -> spec): the taint flow with argument
+     provenance as the deterministic object, the fact-proposal lane included.
+   - it fails -> fix the trace, do not generalize. A failure traced to fact coverage grows the fact
+     tables first; a logic failure revisits the prover.
+
+### What is deferred, explicitly
+
+- The injection-family spec waits for gate 5. Its fact-proposal lane absorbs the disciplined DSPy steer:
+   the LLM proposes trust boundaries and missing sinks, the engine verifies, and repetition promotes them
+   to closed facts -- never a prompt fitted on pool labels.
+- `hunter-learning-loop` task 4.1 (training-set sourcing) stays parked: gates 1-4 need no fitted model.
+- `pre-push-safety-net` wiring follows a search that clears the bar, unchanged from the 2026-10-13 order.
+
 ## Roadmap to detection with the decision engine on kube-ax (2026-10-04)
 
 This section supersedes the ordering below it. The older sections stay as history.

benchmarks/measurements/2026-10-28-access-control-prover/NOTES.md

@@ -0,0 +1,114 @@
+# Access-control prover arm -- training pool (gate 1)
+
+Date: 2026-10-28. Roadmap: `.kiro/steering/roadmap.md`, "measure the trace first", gate 1.
+
+## Setup
+
+- Arm: `benchmarks/valid/prover_arm.py`, deterministic trace + dominance check. No model
+  calls, no API key, no fitted components.
+- Pool: `benchmarks/valid/train/pool-train.toml` (frozen 2026-10-14 09:31, freeze digest
+  e6a67487) -- 23 repositories x (1 vulnerability fix push + 20 ordinary pushes) = 483
+  changes, 93 confirmed vulnerable locations.
+- Clones: `~/ousast-results/valid/train/hist/<owner__name>`; repos mapping
+  `~/ousast-results/valid/train/repos_hist_pool23.json` -- rebuilt for this run. The
+  previous mapping (`repos_hist.json`, 29 entries from the 5-repo pool era) lacked 18 of
+  the 23 pool repositories and produced 378/441 `KeyError` instrument failures on a first
+  launch; those records were deleted and never summarized. `prover_arm._prove` records
+  only the exception type, so the missing key was invisible in records; it was recovered
+  from the run log. Follow-up candidate: record the exception message in instrument
+  failures.
+- Command: `PYTHONPATH=$PWD .venv/bin/python -m benchmarks.valid.prover_arm run
+  benchmarks/valid/train/pool-train.toml --repos repos_hist_pool23.json --out
+  ~/ousast-results/prover/ac-train-1 --only all`, then `summarize` on the same out dir.
+- Runtime: mean 27.9 s/change, median 0.9 s (ordinary pushes are fast; first touch of a
+  repo pays tree walk + framework scan, max 1063 s). 483/483 records,
+  **0 instrument failures**, no zero-byte read refusals.
+
+## Result (whole pool, 23 repos)
+
+Headline (from `summary.json`): on the training pool, recall 65.2% (Wilson 44.9-81.2),
+false alarms 15.0 per 100 pushes, precision 17.9%; exploit-verified 0 of 78 (78 not yet
+proved -- runtime proof is the async lane, out of scope here).
+
+| metric | value |
+|---|---|
+| recall (fix pushes) | 15/23 = 65.2% (Wilson 44.9-81.2; repo-clustered 95% 43.5-82.6) |
+| location recall | 55/93 = 59.1% (Wilson 49.0-68.6) |
+| false alarms | 69 on 460 ordinary pushes = 15.0 per 100 pushes |
+| precision (suspicion level) | 15/84 = 17.9% |
+| instrument failures | 0 |
+
+## Comparison with the DSPy and hunter arms (5 overlapping repos only)
+
+The 2026-10-14 hunt arms ran the old 5-repo pool (dependency-track, job-import, p4,
+shopper, trigger.dev; digest 53f9f364, 48 locations, 100 ordinary pushes). The prover ran
+the 23-repo pool; restrict to those 5 repositories for a like-for-like recall comparison.
+
+| arm | recall on 5 repos | notes |
+|---|---|---|
+| DSPy (2026-10-14) | 2/5 | 4/48 locations, 15.6 FA/100, 11 LMInvalidRequestError failures |
+| hunter (2026-10-14) | 2/5 | different repos; union of the two arms 4/5, 99/100 ordinary pushes flagged |
+| **prover (this run)** | **4/5** | misses p4 only; includes trigger.dev, which neither arm found |
+
+The prover alone matches the union of both learned arms on their own training pool, at
+DSPy-level false-alarm rate (15.0 vs 15.6 FA/100) and with zero instrument failures and
+zero model cost. False-alarm counts come from different ordinary-push sets (460 vs 90),
+so treat the FA comparison as indicative, not paired.
+
+## Misses (8 fix pushes)
+
+Zero coverage (no entry points found -- framework facts absent, fix site never reached):
+- apache/seatunnel (Java, JettyService) -- 0 entry points.
+- friendsofflarum/byobu (Flarum JS) -- 0 entry points.
+
+Covered but not flagged:
+- jenkinsci/analysis-core -- 172 entry points, 0 operations checked.
+- jenkinsci/crx-content-package-deployer -- 12 checked, 2 discharged.
+- jenkinsci/git -- 15 checked, 11 discharged.
+- jenkinsci/global-build-stats -- 1 checked, 1 discharged.
+- jenkinsci/p4 -- 38 checked, 3 discharged (the only overlap-repo miss).
+- kimai/kimai -- 80 checked, 78 discharged; the fix's protected operation is dominated by
+  an existing check on every traced path, or the fix site is not entry-reachable under
+  current facts.
+
+Reading: 2 of 8 misses are fact-coverage failures (grow the entry/fact tables, gate 5's
+"fact-coverage failure grows fact tables first"), 6 are trace/logic failures for gate 3
+ablation (what the checked-but-unflagged traces lack: entry reachability of the fix site,
+missing operation kinds, or real dominance by an existing check).
+
+## Per-repository coverage
+
+| repository | recall | entry points | reached | ops checked | discharged |
+|---|---|---|---|---|---|
+| apache/seatunnel | 0/1 | 0 | 0 | 0 | 0 |
+| friendsofflarum/byobu | 0/1 | 0 | 0 | 0 | 0 |
+| jenkinsci/analysis-core | 0/1 | 172 | 7 | 0 | 0 |
+| jenkinsci/azure-credentials | 1/1 | 100 | 29 | 6 | 0 |
+| jenkinsci/azure-keyvault | 1/1 | 133 | 6 | 7 | 4 |
+| jenkinsci/blueocean | 1/1 | 475 | 18 | 35 | 2 |
+| jenkinsci/compuware-scm-downloader | 1/1 | 297 | 6 | 6 | 0 |
+| jenkinsci/crx-content-package-deployer | 0/1 | 168 | 67 | 12 | 2 |
+| jenkinsci/database | 1/1 | 93 | 38 | 8 | 5 |
+| jenkinsci/dependency-track | 1/1 | 159 | 7 | 15 | 0 |
+| jenkinsci/ec2 | 1/1 | 742 | 24 | 29 | 10 |
+| jenkinsci/git | 0/1 | 584 | 24 | 15 | 11 |
+| jenkinsci/global-build-stats | 0/1 | 448 | 32 | 1 | 1 |
+| jenkinsci/hashicorp-vault | 1/1 | 179 | 15 | 17 | 2 |
+| jenkinsci/jiratestresultreporter | 1/1 | 187 | 13 | 6 | 0 |
+| jenkinsci/job-import | 1/1 | 103 | 43 | 17 | 9 |
+| jenkinsci/opentelemetry | 1/1 | 157 | 9 | 5 | 2 |
+| jenkinsci/p4 | 0/1 | 1018 | 35 | 38 | 3 |
+| jenkinsci/rocketchatnotifier | 1/1 | 57 | 10 | 6 | 0 |
+| kimai/kimai | 0/1 | 1951 | 78 | 80 | 78 |
+| mage-ai/mage-ai | 1/1 | 214 | 1 | 63 | 0 |
+| shopperlabs/shopper | 1/1 | 2645 | 53 | 115 | 1 |
+| triggerdotdev/trigger.dev | 1/1 | 7164 | 20 | 14 | 0 |
+
+Raw records: `~/ousast-results/prover/ac-train-1/` (483 records + `summary.json`, also
+copied here). Watcher log: `~/ousast-results/prover/ac-train-1.log.watch`.
+
+## Gate decision
+
+Gate 1 passes its purpose: the trace-based prover holds recall the learned arms only
+reached as a union, at comparable false alarms, with a clean instrument. Next gate (2):
+zero-shot judge pass over these same traces where judgment remains.