marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

measure: hunt cache replay verified (zero tokens on rerun)

Browse Source 
Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-10-13 13:16:31 +02:00
co-authored by Claude Opus 5.5
parent c2bfcef3df
commit 3692c85d76
1 changed files with 7 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benchmarks/measurements/2026-10-13-hunt-baseline/NOTES.md
+7
View File
@@ -31,3 +31,10 @@ What this says:
Next: verify cache replay on one change (two runs, identical output and zero new tokens), then k = 5 runs of the
baseline and of the pilot hunter's prompt on the 12 positives, paired by change.
## Cache replay, verified after the fix (2026-10-13)
One change (nvbn/thefuck), program without tools, three runs into one per-run cache configured through
`dspy.configure_cache`: runs 1 and 2 gave the same answer, run 3 used zero tokens and the same answer. Replay now
works; the earlier failure was the cache directory being set after DSPy had chosen its default. Repeated runs for
variance therefore have to use separate cache directories (or `cache=False`), or they measure nothing.