Marius CiepluchandClaude Opus 5.5 c5e6ed3221 bench(plane): first increment measured on ax -- gate not met (14/20 agreed vs 16) ...

validation-46 ran done on google/ax: 60 tasks, 357 DeepSeek calls, $0.83 with
recorded triage ($0.0194 per candidate, reference $0.022: cost met). Declared
sites agreed 14/20 against the reference's 16/20: recall criterion not met.
Both runs find 19/20 sites in at least one pass; the two-pass agreement
flicker moved three sites. Design revisit before task 9.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo

2026-09-29 22:19:38 +02:00

..

evaluate.py

measure(independent): the unchanged tool hunter on population v2 -- recall 1/17, and why

2026-09-28 11:36:32 +02:00

freeze-v1.json

spec(pre-push-safety-net): freeze the independent population v1 (15.1)

2026-09-24 16:43:32 +02:00

freeze-v2.json

spec(pre-push-safety-net): freeze the independent population v2

2026-09-26 18:02:03 +02:00

freeze.py

spec(pre-push-safety-net): freeze the independent population v2

2026-09-26 18:02:03 +02:00

hunt.py

spec(pre-push-safety-net): pre-register the hunter experiment on population v2

2026-09-28 09:10:02 +02:00

hunter-protocol-v2.md

spec(pre-push-safety-net): pre-register the hunter experiment on population v2

2026-09-28 09:10:02 +02:00

model_sinks.py

measure(independent): model-classified sinks diagnostic (source only, no vocabulary)

2026-09-28 12:43:03 +02:00

plane-increment-1.json

bench(plane): first increment measured on ax -- gate not met (14/20 agreed vs 16)

2026-09-29 22:19:38 +02:00

population-v1.toml

spec(pre-push-safety-net): freeze the independent population v1 (15.1)

2026-09-24 16:43:32 +02:00

population-v2.toml

spec(pre-push-safety-net): freeze the independent population v2

2026-09-26 18:02:03 +02:00

protocol-v1.md

spec(pre-push-safety-net): pre-register the evaluation protocol for the independent population

2026-09-24 16:44:32 +02:00

protocol-v2.md

spec(pre-push-safety-net): pre-register evaluation protocol v2 and its runner

2026-09-26 18:02:03 +02:00

research-2026-09-24.md

spec(pre-push-safety-net): reserve the independent population v1 (15.1), unscanned

2026-09-24 16:12:22 +02:00

research-2026-09-26.md

spec(pre-push-safety-net): reserve the independent population v2 (17 cases, unscanned)

2026-09-26 17:53:51 +02:00

results-v1.json

measure(independent): first evaluation on the frozen population -- fails every M4 gate

2026-09-25 09:56:26 +02:00

results-v2-hunter.json

measure(independent): the unchanged tool hunter on population v2 -- recall 1/17, and why

2026-09-28 11:36:32 +02:00

results-v2-model-pipeline.json

docs(pre-push-safety-net): record the model-driven pipeline result on v2 and stop

2026-09-29 11:29:06 +02:00

results-v2.json

measure(independent): evaluation of population v2 under protocol v2 -- fails every M4 gate

2026-09-28 08:52:53 +02:00

score_batched.py

feat(independent): fixed-pin recheck and an exploratory scorer for the batched verifier

2026-09-29 10:25:08 +02:00

selection-v2-model-sinks.json

measure(independent): model-classified sinks put the vulnerable function in scope in 15/17

2026-09-28 13:21:41 +02:00

sink_candidates.py

measure(independent): sink-first selection from source -- the vocabulary misses most real sinks

2026-09-28 12:36:43 +02:00

verifier-batched-check-2026-09-29.json

measure(independent): triage + per-file batching + two-pass agreement -- 16/20 sites, flicker cut 26 -> 9, 34% cheaper

2026-09-29 10:22:11 +02:00

verifier-check-2026-09-29.json

measure(independent): known-answer check of the verifier -- 11/15 sites, unstable verdicts, escalation costs more

2026-09-29 10:10:33 +02:00

verify_batched.py

feat(independent): fixed-pin recheck and an exploratory scorer for the batched verifier

2026-09-29 10:25:08 +02:00

verify_sinks.py

measure(independent): known-answer check of the verifier -- 11/15 sites, unstable verdicts, escalation costs more

2026-09-29 10:10:33 +02:00