mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
perf(hunter): cacheable message order; adaptive step budget in the sink verifier
Token efficiency, measured on 180 verified pgAdmin candidates: stage 2 costs $0.0139 per candidate, 90% of the spend, because the conversation is re-sent on every turn and grows with each tool result. - Context files now precede the question in the hunter's first message. DeepSeek bills a cached prefix at 1/30 of a fresh one, by prefix; with the question first, every candidate asked about the same file (about 5 per file) broke the prefix on its first line and paid for the file again. A test pins the order. - The verifier runs each candidate at 3 steps and repeats it once at 6 only when the hunt spent every step on tool calls without a verdict. On the seven declared v2 sites, 3 steps found 5 and 8 steps found 6; most candidates conclude in 3. - Per-candidate token usage (prompt, cached, completion, calls) and escalation are recorded, and each case reports its cache share, so the next efficiency decision is made on numbers. Validation on the 180 pgAdmin candidates plus the 22 known sites (about $1) waits for account credit: DeepSeek returns 402 at the moment. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
66fa08cdd0
commit
041d87199a
@@ -365,3 +365,14 @@ def test_the_final_answer_is_asked_plainly_and_a_wrapped_array_is_read(tmp_path)
|
||||
found = tool_hunter.run_tool_hunter(tmp_path, [spot], client=Client(), model="m", max_steps=3)
|
||||
assert [(f.path, f.line) for f in found] == [("app.py", 2)]
|
||||
assert seen[-1] == {"tools": [], "json_object": False}, seen[-1]
|
||||
|
||||
|
||||
def test_context_files_precede_the_question_so_the_prefix_caches(tmp_path) -> None:
|
||||
"""DeepSeek bills a cached prefix at 1/30 of a fresh one, by prefix. With the question first, every candidate
|
||||
asked about the same file broke the prefix on its first line and paid for the file again."""
|
||||
from openultrasast import tool_hunter
|
||||
|
||||
(tmp_path / "app.py").write_text("def run(q):\n return eval(q)\n")
|
||||
message = tool_hunter._user_message(tmp_path, [], "Judge the eval in run.", ["app.py"], 4000)
|
||||
assert message.index("--- app.py") < message.index("def run(q)") < message.index("Judge the eval in run.")
|
||||
assert message.endswith("Judge the eval in run.")
|
||||
|
||||
Reference in New Issue
Block a user