From 3692c85d76f700252d10586b170de52c83304d50 Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Tue, 13 Oct 2026 13:16:31 +0200 Subject: [PATCH] measure: hunt cache replay verified (zero tokens on rerun) Co-Authored-By: Claude Opus 5.5 --- benchmarks/measurements/2026-10-13-hunt-baseline/NOTES.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/benchmarks/measurements/2026-10-13-hunt-baseline/NOTES.md b/benchmarks/measurements/2026-10-13-hunt-baseline/NOTES.md index d54e07d..1ca1f2c 100644 --- a/benchmarks/measurements/2026-10-13-hunt-baseline/NOTES.md +++ b/benchmarks/measurements/2026-10-13-hunt-baseline/NOTES.md @@ -31,3 +31,10 @@ What this says: Next: verify cache replay on one change (two runs, identical output and zero new tokens), then k = 5 runs of the baseline and of the pilot hunter's prompt on the 12 positives, paired by change. + +## Cache replay, verified after the fix (2026-10-13) + +One change (nvbn/thefuck), program without tools, three runs into one per-run cache configured through +`dspy.configure_cache`: runs 1 and 2 gave the same answer, run 3 used zero tokens and the same answer. Replay now +works; the earlier failure was the cache directory being set after DSPy had chosen its default. Repeated runs for +variance therefore have to use separate cache directories (or `cache=False`), or they measure nothing.