marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

measure: hunt baseline on the 12 pilot positives

Browse Source 
3 of 12 with the fixed answer check against the pilot hunter's 6 of 12, one
run each, not significant. The intended zero-cost replay was not a replay:
answers at temperature 0 differ between runs and the cache did not serve the
second run. Programs must be compared over repeated runs, paired by change.
About $1.10 for both runs.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-10-13 13:15:22 +02:00
co-authored by Claude Opus 5.5
parent 602ec23e03
commit c2bfcef3df
1 changed files with 33 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benchmarks/measurements/2026-10-13-hunt-baseline/NOTES.md
+33
View File
@@ -0,0 +1,33 @@
# Hunt baseline on the 12 pilot positives (task 3.2), 2026-10-13
The first DSPy hunt program (today's hunter prompt plus the changed hunks and the four producers' annotations,
ReAct over read_file, grep_repo and find_refs, at most 8 steps, deepseek-flash at temperature 0) run over the 12
verified pilot positives as a report run. Records: `~/ousast-results/hunt/baseline-positives/records{,-v2}/`.
Scorer: `benchmarks/unseen/score.py`, the same as the pilot.
| Run | Answer check | Search recall | Parse failures | Cost (tokens at our prices) |
|---|---|---|---|---|
| Pilot hunter (2026-10-12, hand-written prompt, no hunks) | none | 6 of 12 | - | - |
| Baseline run 1 | counted only the rendered function as shown (bug) | 1 of 12 | 2 | about $0.65 |
| Baseline run 1, as reported by the model | (family as the model gave it) | thefuck x2, jupyterlab-lsp, contentful at the right family | - | - |
| Baseline run 2 (fixed check: tool-read lines count as shown) | fixed | 3 of 12 (jupyterlab-lsp, ckan-mcp-server, malla) | 2 | about $0.44 |
What this says:
- **The instrument bug was real and is fixed** (commit 602ec23): 7 of 8 suspicions in run 1 cited lines the model
had read through `read_file` and were wrongly scored as `unknown`.
- **Run 2 was meant to replay run 1 from the response cache and did not.** The answers and per-change times
differ (thefuck: two path findings in run 1, none in run 2; md-editor: a hit in the pilot, a parse failure in
run 2). At temperature 0 the provider's answers are not stable from run to run, and the DSPy cache did not
serve the second run. The per-run cache is now configured through `dspy.configure_cache` (602ec23); whether it
replays has to be checked before any run is called a replay.
- **One run per program is not a measurement.** On 12 positives a single run moves by several hits between
identical programs. Any comparison between programs or masks has to repeat each arm (k runs, paired by change)
and report the paired bootstrap and McNemar from `openultrasast.stats`. The acceptance rule (requirements 6.2,
6.5) already demands intervals; this run shows why the variance has to be inside them.
- The two programs are not yet equal on the same footing: the baseline adds hunks and annotations to the pilot
hunter's prompt and runs a different loop. 3 of 12 against 6 of 12 is not a significant difference on this n
(one run each) and is not read as one.
Next: verify cache replay on one change (two runs, identical output and zero new tokens), then k = 5 runs of the
baseline and of the pilot hunter's prompt on the 12 positives, paired by change.