marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

measure: hunt cache replay verified (zero tokens on rerun)

Browse Source 
Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-10-13 13:16:31 +02:00
co-authored by Claude Opus 5.5
parent c2bfcef3df
commit 3692c85d76
1 changed files with 7 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benchmarks/measurements/2026-10-13-hunt-baseline/NOTES.md
+7
View File
@@ -31,3 +31,10 @@ What this says:
Next: verify cache replay on one change (two runs, identical output and zero new tokens), then k = 5 runs of the Next: verify cache replay on one change (two runs, identical output and zero new tokens), then k = 5 runs of the
baseline and of the pilot hunter's prompt on the 12 positives, paired by change. baseline and of the pilot hunter's prompt on the 12 positives, paired by change.
## Cache replay, verified after the fix (2026-10-13)
One change (nvbn/thefuck), program without tools, three runs into one per-run cache configured through
`dspy.configure_cache`: runs 1 and 2 gave the same answer, run 3 used zero tokens and the same answer. Replay now
works; the earlier failure was the cache directory being set after DSPy had chosen its default. Repeated runs for
variance therefore have to use separate cache directories (or `cache=False`), or they measure nothing.