Open_Ultra_SAST/benchmarks/measurements/2026-10-28-access-control-judge/NOTES.md

Zero-shot judge over prover traces -- training pool (gate 2)

Date: 2026-10-28. Roadmap: .kiro/steering/roadmap.md, "measure the trace first", gate 2. Input: gate 1 prover findings (~/ousast-results/prover/ac-train-1/, measurements in ../2026-10-28-access-control-prover/).

Setup

• Judge: benchmarks/valid/judge_pass.py (run + summarize). One zero-shot judgment per prover finding (340 findings on 84 flagged pushes: 15 true + 69 false).
• Question, fixed before any answer was seen: "is this entry meant to be public?" with a strict JSON answer {public: bool, confidence, reason} and the statement that public: true means the flag is a false alarm. Payload per judgment: repository, entry metadata, the embedded trace, evidence lines, operation, missing obligation. No examples, no tuning -- the pool measures, never trains (roadmap rule 3).
• Model: openai/deepseek-flash, temperature 0, per-run dspy cache. 340/340 judgments, 0 failures, 34.4 min (~7 s per judgment), no provider errors (the hunt arm's LMInvalidRequestError mode did not appear).
• Rollup rule: a push is dropped only when ALL its findings are judged public. Raw judgments: ~/ousast-results/prover/ac-judge-1/judgments.jsonl; summary and run metadata copied here.

Instrument notes

Two checks before believing the numbers. First, a 2-judgment dry run on one vuln and one ordinary finding: parse worked, reasons were real. Second, the harness fired a "run complete" notification while the process was still writing (340 claimed, 319-333 on disk); a summarize in that window produced partial numbers and was discarded. Final numbers below are from the completed run only: 340 lines, 340 unique keys, run.json judged: 340, failures: 0.

Result

Headline (from summary.json): judge over prover findings: keeps 15/15 true flags, drops 5/69 false flags (recall 65.2%, FA 13.9/100).

metric	before judge (gate 1)	after judge
recall (fix pushes)	15/23 = 65.2%	15/23 = 65.2% (no true flag lost)
false alarms per 100 pushes	15.0 (69/460)	13.9 (64/460)
precision (push level)	15/84 = 17.9%	15/79 = 19.0%

Finding-level verdicts (340 judgments): on the 15 true pushes the judge voted protected 90 times and public once -- it almost never argues with the prover on a real vulnerability. On the 69 false pushes it voted protected 216 times and public 33 times: it calls only about 14% of the false-alarm findings "meant to be public".

Reading

Lane 2 is safe but nearly inert on this pool. It costs no recall, but the public-entry question does not separate the prover's false alarms: 64 of 69 false pushes survive because at least one of their findings looks genuinely protected (internal endpoints, admin-adjacent code -- exactly the code the prover was built to flag). The judge cannot see why the flag is wrong; the wrongness lives in the trace (an unchecked operation that is in fact guarded by a mechanism the trace does not model, or an entry that is not attacker-reachable), which is lane 1 territory.

Decision input for gate 5: do not spend effort tuning this judge prompt. The FA lever is trace and fact coverage -- measure it in gate 3 (ablation over the 8 prover misses and a sample of the 64 judge-kept false pushes) before considering any judge changes.