Marius CiepluchandClaude Opus 5.5
6bfb05f952
This is the deciding experiment of the strategy review, fixed before any v2 pin is hunted. - Hunter: the unchanged tool hunter (default prompt, tools and 4 steps) on deepseek-flash. - Scope: the ranker's top 20 regions for the case family, in four hunts. The model never sees the fix diff or the declared sites. - Families: a fixed keyword table maps the model's family to the protocol's three. - Scoring: exactly protocol v2's scorer. - Cost: a hard $10 ceiling. hunt.py runs it and scores it (--score). A pilot on a v1 development case (mongo-express) found the vulnerability at lib/bson.js:60, at $0.031 per pin, alongside a lot of speculative noise. Transport fix found by the pilot: the DeepSeek endpoint the hunter builds was unpriced, so the cost meter read $0.00 for any number of calls. It is now priced. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
3.0 KiB
Hunter experiment on population v2 (pre-registered)
Written and committed on 2026-09-28, BEFORE the tool hunter is run on any v2 pin. It is the deciding experiment
of the strategy review (.kiro/specs/pre-push-safety-net/strategy-review-2026-09-28.md): does a model with
repository tools find what the static engine did not (v2: recall 0/17, precision 0/17)?
What is under test
tool_hunter.run_tool_hunteras it stands: default system prompt, default tools (read_file,grep_repo,find_refs),DEFAULT_MAX_STEPS(4), the final-answer turn. Its prompt, tools and parsing are unchanged since the 2026-09-06 measurement; the only changes today are transport (cf57995and this commit: the client reaches DeepSeek, and the endpoint is priced so the budget can see spending).- Model:
deepseek-flash(DeepSeek, the project's chat provider), throughresolve_hunter_client(). - Runner:
benchmarks/independent/hunt.pyat the commit that introduces this file.
Scope handed to the model, per pin
The regions the static scan's discovery and ranker produce for the case's family (regions_for over
analyze_entry_points, the same as finding_dump.py), sorted by rank, top 20, as hunter hotspots in four groups
of five: one bounded hunt per group. Nothing else: not the fix diff, not the advisory, not the case's declared
sites, not the case id.
Pins
Protocol v2's six per case: vulnerable a/b, fixed a/b, benign base and tip. Each is a git archive of the pin.
Turning a hunter answer into a scoreable finding
site=path:line:function, with the function the model reported, or empty.family= the model's own family and title mapped by the fixed keyword table inhunt.py(FAMILY_WORDS, checked in that order: untrusted_destination, config_secrets, injection; anything else isother).
Scoring
Exactly protocol v2 (protocol-v2.md), via evaluate.score_v2 over the hunter's results: the same matching
(v1's changed-line rule or a declared path::function site), detection in BOTH vulnerable runs, a fixed-side
alert in EITHER fixed run, benign alerts, stability, and the precision sample (every third stable
vulnerable-pin finding) adjudicated to v1's standard. A finding with no function can match only by the
changed-line rule. A pin whose four hunts all fail is an instrument failure, reported as such.
"Questions" for the hunter are its hunts (4 per pin), so completion is hunts completed / hunts run; it is not comparable to the engine's question completion and is reported only as an instrument check.
Cost
Hard ceiling $10 of recorded model spend, enforced by the runner; a pilot on a v1 development case (mongo-express, 4 hunts) cost $0.031 and 57 s. If the ceiling stops the run, the unscanned pins are reported as not run.
What may not happen
- No change to the hunter's prompt, tools, step budget, parsing, scope rule or family table before this result is committed.
- Whatever is learned here is a finding about the hunter as it is; tuning it on v2 afterwards is a retune and needs a new population to qualify.