measure(g2): model hunter on sound targets -- first LLM catch the static tiers missed (1/2)

ousast scan --mode deep (DeepSeek deepseek-flash, model layer + tool_hunter, 60-call bound) on
bedrock + graphiti, the two sound pilot targets the static engine missed:
- graphiti: HIT -- falkordb_driver.py:207, tag family:cypher-injection; rationale traces
  build_fulltext_query concatenating user input into a Cypher string at line 278 (inside the
  labelled function, location-matches). The static engine produced zero findings here.
- bedrock: miss -- 12 suspicions on adjacent concerns (network-PUBLIC, download_file), never
  named the install_packages -> pip command injection.
Cost is a non-issue: ~$0.007/repo, 64 model calls for a whole-repo deep scan. BUT everything
stopped at suspicion rung -- the proof arm did not elevate to proven; release-grade BLOCK needs
proven findings.

Direction (maintainer): prompt evolution is a core lever -- a DSPy-based loop (GEPA/MIPRO,
pluggable; evolving field) whose objective is to learn what hints/annotations steer the model
to the real sink, sourced from the framework knowledge already shipped (no new user
annotation), with LLM-proposed boundaries graduating into the facts.

Next: measure proof-arm lift on graphiti; stand up the DSPy hint loop + re-run bedrock; widen
the hunter to the 23 sound targets for a catch rate; then a change-scoped + proven run.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

benchmarks/measurements/2026-10-12-hunter-sound-targets/NOTES.md

@@ -0,0 +1,43 @@
+# Model hunter on the two sound pilot targets — 2026-10-12
+
+## Why
+The engine-on pilot showed the static tiers (regex + Joern dataflow) catch 0/5, and the LLM adjudication tier was
+disabled in the push path. Of the 5 pilot repos, the label audit found only **bedrock + graphiti (injection)** are
+sound (change-located, change-introduced) fair targets — and the static engine missed both. We ran the shipped
+**model hunter** (`ousast scan --mode deep`, DeepSeek `deepseek-flash`, model layer + tool_hunter enabled,
+unbounded by a push deadline, bounded to 60 model calls / 8 hotspots) on those two to test whether the LLM arm
+catches what the static tiers miss.
+
+## Result: hunter 1/2 (static engine 0/2), cheap, but both at suspicion rung
+| repo | vuln | static engine | **hunter** | model calls | cost |
+|---|---|---|---|---|---|
+| getzep/graphiti | Cypher injection in `build_fulltext_query` | miss | **HIT (suspicion)** | 64 | $0.0071 |
+| aws/bedrock-agentcore | command injection in `install_packages` | miss | **miss (suspicions on adjacent)** | 64 | $0.0076 |
+
+- **graphiti — correct catch.** `falkordb_driver.py:207` finding, `tags:[family:cypher-injection]`, severity medium,
+  confidence low, rung **suspicion**. Rationale traces it exactly: *"build_fulltext_query concatenates the sanitized
+  user query into a Cypher string: full_query = group_filter + ' (' + sanitized_query + ')' (line 278)"*. Line 278
+  is inside the labelled `function_lines [249,280]`; head_lines include [206],[249-251],[280] → location-matches.
+  The static engine produced zero findings here.
+- **bedrock — miss.** 12 findings, all suspicion; the 3 on the vuln file were adjacent concerns (network-PUBLIC
+  default @114, caller client_token @188, download_file @583) — it never named the `install_packages` → `pip
+  install` command injection. A prompting/targeting miss, not cost (64 calls, $0.0076) or capability.
+
+## Reads
+1. **The LLM arm lifts catch over the static tiers** — graphiti is the first unseen vuln caught that regex+dataflow
+   missed. Cost is a non-issue (~$0.007/repo, 64 calls for a whole-repo deep scan).
+2. **Everything stops at `suspicion`.** Neither reached corroborated/entailed/execution_confirmed — the proof arm
+   (sandbox verification) did not elevate them. For release-grade BLOCK the findings must be *proven*, not just
+   suspected. The proof arm + verification are the next depth to measure.
+3. **bedrock motivates prompt evolution.** The hunter has the capability but its current prompts did not steer it to
+   the real sink. Directive: a **DSPy**-based prompt-optimisation loop (GEPA/MIPRO; keep pluggable) whose objective
+   is to learn **what hints/annotations** the model needs (bedrock: "install_packages runs pip with caller-supplied
+   names → command injection") — sourced from the **framework knowledge already shipped** (`ruleset/semantic/*.toml`,
+   `frameworks.toml`), no new user annotation; LLM-proposed boundaries graduate into the facts (#2 feedback loop).
+
+## Caveats
+- Whole-repo deep scan (not change-scoped), so this measures detection CAPABILITY, not pre-push latency. A
+  change-scoped + proven run is the release-relevant shape.
+- Only 2 sound targets here; widen to the 23 sound public targets next for a rate.
+- graphiti's finding is suspicion/low-confidence; a human/proof confirmation is still needed before it is a
+  release-grade catch. Findings JSON: `bedrock-findings.json`, `graphiti-findings.json`.