measure(independent): known-answer check of the verifier -- 11/15 sites, unstable verdicts, escalation costs more

Declared vulnerable sites: 11 of the 15 that were candidates are confirmed (8 at 3
steps, 3 more at 6). Missed: winml (a CORS config; the reachability question does
not fit), FUXA, Budibase, Froxlor.

Restart-based escalation is strictly worse than a straight 6-step budget: 88% of
pgAdmin candidates used every step, so they paid 3 + 6 ($0.0201 per candidate vs
$0.0139 baseline). Defaults are now 6 steps, no escalation. The cache order change
did not lower the per-call cost ($0.002): the conversation prefix was already
cached; 56% of prompt tokens are served from cache.

The larger finding: on 180 pgAdmin candidates verified twice, 15 and 13 were
flagged with only 2 in common. Verdicts on borderline candidates flip run to run,
while the declared sites were found consistently. Precision needs agreement
across runs or a second judgement, not a single pass.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo

benchmarks/independent/verifier-check-2026-09-29.json

@@ -0,0 +1,46 @@
+{
+ "measurement": "known-answer check of the sink verifier after 041d871 (context-first order, 3->6 restart escalation)",
+ "declared_sites": {
+  "found": 11,
+  "of_cases_with_a_candidate_site": 15,
+  "found_at_3_steps": [
+   "pgadmin",
+   "lmdeploy",
+   "reactpress",
+   "unleash",
+   "medplum",
+   "cct-studio x2",
+   "gopay",
+   "directorist"
+  ],
+  "found_only_at_6": [
+   "qwed",
+   "contextforge",
+   "groundhogg"
+  ],
+  "missed": [
+   "winml (config, question shape)",
+   "fuxa",
+   "budibase",
+   "froxlor"
+  ],
+  "usd": 0.283
+ },
+ "pgadmin_180_overlap": {
+  "baseline_findings": 15,
+  "check_findings": 13,
+  "in_common": 2,
+  "note": "verdicts on borderline candidates flip run to run; declared sites were stable"
+ },
+ "cost": {
+  "baseline_usd_per_candidate": 0.0139,
+  "check_usd_per_candidate": 0.0201,
+  "escalated_share": 0.88,
+  "non_escalated_usd": 0.0038,
+  "escalated_usd": 0.0222,
+  "cache_share_of_prompt_tokens": 0.56,
+  "per_call_usd": 0.002,
+  "conclusion": "restart escalation is strictly worse than a straight 6-step budget; the cache was already effective within a conversation"
+ },
+ "decision": "defaults set to --max-steps 6 --escalate-steps 0"
+}

benchmarks/independent/verify_sinks.py

@@ -131,8 +131,8 @@ def main() -> int:
     parser.add_argument("--population", type=Path, default=HERE / "population-v2.toml")
     parser.add_argument("--only", default="")
     parser.add_argument("--model", default=DEFAULT_DETECTOR_MODEL)
-    parser.add_argument("--max-steps", type=int, default=3)
+    parser.add_argument("--max-steps", type=int, default=6)
-    parser.add_argument("--escalate-steps", type=int, default=6, help="rerun a hunt that used every step without a verdict; 0 disables")
+    parser.add_argument("--escalate-steps", type=int, default=0, help="rerun a hunt that used every step without a verdict; 0 disables (measured: 88% escalate, +45% cost)")
     parser.add_argument("--workers", type=int, default=4)
     parser.add_argument("--budget-usd", type=float, default=40.0)
     parser.add_argument("--fixed", action="store_true", help="recheck detected sites at the fixed pin")