measure(independent): the unchanged tool hunter on population v2 -- recall 1/17, and why

The run follows hunter-protocol-v2.md: deepseek-flash, the ranker's top 20
regions per pin, protocol v2 scoring, $8.88 spent.
- Recall: 1/17 (Wilson 0.01-0.27).
- Precision: 2/14 sampled (both true positives need a privileged role, one of
  them GoPay's CVE).
- Fixed-side alerts: 1. Benign alerts: 16.
- Stability: 73% of findings differ between two identical runs.

The deciding observation: the ranker's top 20 contained the vulnerable file in 1
of 17 cases, and in that case (GoPay) the hunter detected it. With 4 tool steps
per hunt it cannot search its way to code it is not handed. The result therefore
says less about the model than about scope selection, which sits upstream of the
engine too.

Instrument failures, reported as failures: every FUXA pin and three Directorist
pins (DeepSeek HTTP 400 on every hunt; neither offered a region of its family).

Scorer fix: protocol v2 defines benign alerts and the precision pool over the
CASE FAMILY, but score_v2 pooled every family. The engine's dump emits only the
case family, so its recorded result (21adc40) is unchanged by this; rescoring
reproduces 3 benign alerts and 49 pooled findings.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo

benchmarks/independent/evaluate.py

@@ -270,7 +270,13 @@ def score_v2() -> int:
         hit_b = {f["site"] for f in vb if matches_v2(f, case, old)}
         fixed_alerts = sorted({f["site"] for f in fa + fb if matches_v2(f, case, new)})
         base_regions = {tuple(f["site"].split(":")[0::2]) for f in scans["benign_base"].get("findings", [])}
-        benign_alerts = [f["site"] for f in scans["benign_tip"].get("findings", []) if tuple(f["site"].split(":")[0::2]) not in base_regions]
+        # protocol-v2.md: benign alerts and the precision pool are findings OF THE CASE FAMILY. The engine's dump
+        # emits only that family, so this was invisible until the hunter, whose findings span every family.
+        benign_alerts = [
+            f["site"]
+            for f in scans["benign_tip"].get("findings", [])
+            if f.get("family") == case["family"] and tuple(f["site"].split(":")[0::2]) not in base_regions
+        ]
         site_files = {site.partition("::")[0] for site in case.get("sites", [])}
         covered = set(old) | site_files
         answered = any(r.split(":")[0] in covered for label in ("vulnerable_a", "vulnerable_b") for r in scans[label].get("completed_regions", []))
@@ -295,9 +301,9 @@ def score_v2() -> int:
                 "seconds": {label: s.get("seconds") for label, s in scans.items()},
             }
         )  # fmt: skip
-        stable = {f["site"] for f in vb}
-        pool += [{"case": case["id"], **f} for f in va if f["site"] in stable]
-        unstable_pool += len({f["site"] for f in va} ^ stable)
+        stable = {f["site"] for f in vb if f.get("family") == case["family"]}
+        pool += [{"case": case["id"], **f} for f in va if f.get("family") == case["family"] and f["site"] in stable]
+        unstable_pool += len({f["site"] for f in va if f.get("family") == case["family"]} ^ stable)
     pool.sort(key=lambda f: (f["case"], f["site"]))
     sample = pool if len(pool) <= 15 else pool[0::3]
     scored = [c for c in per_case if c.get("status") != "incomplete"]