measure(g2): banks fixed-search MVP -- "no wrong targets, only wrong searching" demonstrated

Same CPG, only the search changes. Baseline (shipped path spec) resolves 0
prompt->write_text flows on banks; injecting the two annotations an LLM steer
would supply (parameter_sources=True + write_text/read_text path sinks) makes the
engine resolve prompt -> path-join -> prompt_file.write_text(prompt.raw). First
measured instance of "#2 guided by #1" closing on an unseen target; fix needed
exactly the param-source + file-write-sink annotations.

Files the instrument bug the first MVP hit: single-query cpg.run("taint",
request_params(...)) is dead -- request_params emits 5 keys (guards/fixedOrigin/
originAnchors/quotedSanitizers/fieldParameterSources) absent from taint.sc's exec
signature; both single-query sub-routes reject them (silent None). Batch path
(run_batch->requestsFile JSON) unaffected; push/scan route uses batch.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

.kiro/specs/engine-change-scoped-analysis/brief.md

@@ -462,3 +462,30 @@ suspicion. So:
 - **Bet #3 (the proof arm) is the ASYNC BLOCK lane**: run `search/verify` on the plane/CI for the suspicions worth
   proving, with authored (eventually DSPy-authored) demos. Worth an empirical run on one tractable target
   (mcp-remote) to confirm the arm demonstrates on a real unseen vuln — a bounded, separate effort.
+
+## BANKS FIXED-SEARCH MVP (2026-10-12) — "no wrong targets, only wrong searching" demonstrated
+
+Record: `benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/`. Target: banks (the "library" that gave
+the engine ZERO path findings in the pilot). Same target, same CPG — only the search changes.
+
+- Injected the two annotations a per-target LLM steer would supply: **parameter-as-source**
+  (`parameter_sources=True`, so `prompt.name` counts as untrusted) + the **missing file-write sink**
+  (`write_text`/`read_text` added to path sinks).
+- **Baseline (shipped search): 0 resolved `prompt → write_text` flows** — its 5 flow rows are all census/
+  evidence (`sink=None`), matching the pilot's zero path findings.
+- **Fixed search: the engine resolves `prompt → path-join → prompt_file.write_text(prompt.raw)` in
+  `from_prompt_path`** (`source=prompt`), plus a second `write_text` sink in `_scan`. These appear ONLY with
+  the annotations.
+- **Thesis demonstrated on a concrete case:** banks was not a wrong target; the search was unannotated. The
+  fix needed exactly **two** inputs — the param-source and the file-write sink — which are precisely what the
+  per-target LLM steer (DSPy-optimised, framework-seeded) has to learn to supply. This is the first measured
+  instance of the "#2 guided by #1" loop closing on an unseen target.
+- Caveats: (1) one row shows `source=path` not `prompt` — the param-source fires broadly, a precision knob for
+  the steering loop; (2) this is a TRACE, not the runtime differential proof (proof stays the async BLOCK lane).
+
+**Instrument note / filed bug:** the first MVP used the single-query `cpg.run("taint", …)` path and returned
+0/0 — an INSTRUMENT FAILURE, not a result (`cpg query taint failed` on every query). Root cause filed in
+`bugs/taint-sc-single-query-signature-skew.md`: `request_params()` emits 5 keys (`guards`, `fixedOrigin`,
+`originAnchors`, `quotedSanitizers`, `fieldParameterSources`) that `taint.sc`'s `exec` signature does not
+declare, so both single-query sub-routes (CLI `--param` and session named-arg) reject them. The batch path
+(`run_batch` → `requestsFile` JSON) tolerates them, which is why the pilot scans ran and this MVP used it.

.kiro/specs/engine-change-scoped-analysis/bugs/taint-sc-single-query-signature-skew.md

@@ -0,0 +1,86 @@
+# BUG: `taint.sc` single-query path broken by exec-signature skew
+
+Filed: 2026-10-12 · Repo HEAD at filing: `7ab5c2a` · Severity: **high (silent zero)** · Status: OPEN
+
+## Summary
+
+Any call on the **single-query** taint path — `cpg.run("taint", request_params(...))` — fails on **both**
+of its sub-routes, returns `None`, and is read upstream as "no flows / clean". `request_params()` emits five
+keys that `taint.sc`'s `@main def exec(...)` does not declare, and the single-query dispatch passes every
+param straight to the script. The **batch** path (`run_batch` → `requestsFile` JSON) is unaffected, which is
+why the engine-on scans ran while the single-query MVP did not.
+
+This is exactly the failure shape AGENTS.md warns about: a tool that could not read/execute its input reports
+a plausible zero that is indistinguishable from a clean result.
+
+## Mechanism
+
+`request_params()` (`src/openultrasast/model/taint.py`) always emits, among others:
+
+- `fieldParameterSources` (line 54)
+- `fixedOrigin` (line 65)
+- `originAnchors` (line 66)
+- `quotedSanitizers` (line 67)
+- `guards` (line 68)
+
+`taint.sc`'s signature (`src/openultrasast/cpg/queries/taint.sc:41` `@main def exec(...)`) declares **none**
+of those five. Its 19 params are: `cpgFile, sources, sinks, sanitizers, function, file, callDepth,
+boundedSinks, parameterSources, fieldSources, evidenceOnly, hookCallbacks, dispatchApply, dispatchValue,
+contextFilter, contextPaths, requests, requestsFile, trace`.
+
+`backend.query()` (`src/openultrasast/cpg/backend.py:1266`) dispatches a single query two ways, **both** of
+which choke on the undeclared keys:
+
+- **Disposable / CLI route** (lines 1276–1278): appends `--param {key}={value}` for every param. joern errors
+  `Unknown arguments: --fixedOrigin --guards --quotedSanitizers --originAnchors --fieldParameterSources`
+  (and, as a knock-on, `Missing argument: --cpgFile`). Observed verbatim via `banks_debug.py` on 2026-10-12.
+- **Session route** (lines 1321–1326): defines `def ousast_taint(...)` by renaming `@main def exec(` and then
+  calls `ousast_taint(key = value, ...)` with named args. The same five keys are not parameters of that
+  method → the Scala named-arg call fails to compile. (By construction; the rename preserves the exact
+  `exec` parameter list.)
+
+Why the batch path survives: `run_batch` writes the per-request params **inside** the `requestsFile` JSON
+object, and `taint.sc` parses that JSON itself (`requestsJson`, taint.sc:67). The extra fields ride as JSON
+keys, never as `exec` arguments, so they are tolerated.
+
+## Blast radius
+
+Broken (single-query taint):
+
+- `taint._usable_flows` (`taint.py:124`) → `verdicts()` (`taint.py:131`) → `verdict()` (`taint.py:99`).
+- `pipeline.scan_region` dispatches taint families to `taint_verdict` (`pipeline.py:121`) → so **per-region
+  single-query taint via `scan_region` is dead** whenever the disposable or session route is taken.
+
+Unaffected:
+
+- The push/scan **batch** path: `scan.py` uses `run_batch("taint", …)` (scan.py:1259, 1264, 1387). This is
+  why the engine-on pilot produced results.
+- `config` and `dominance` single-queries (`config_value.py:46`, `dominance.py:54`) — their `.sc` signatures
+  are presumed to match what their specs emit (confirm separately; not exercised here).
+
+Open question to close: enumerate every production path that reaches `scan_region`/`verdict()` taint rather
+than the batch route, to state whether any shipped push/replay flow silently loses taint flows today, or
+whether the single-query path is only hit by pair measurements and ad-hoc harnesses.
+
+## Evidence
+
+- `benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/` — the batch path works (11 flow rows,
+  `write_text` sinks); the v1 single-query attempt returned 0/0 on `cpg query taint failed`.
+- Raw joern error captured by `banks_debug.py` (scratchpad): `Missing argument: --cpgFile; Unknown
+  arguments: --fixedOrigin --guards --quotedSanitizers --originAnchors --fieldParameterSources`.
+
+## Suggested fix (not yet applied)
+
+Pick one, lowest-risk first:
+
+1. **Make `backend.query()` filter params to the script's declared signature** before building `--param` /
+   named-arg lists (parse the `@main def exec(` arg names once, intersect). Fixes every current and future
+   signature skew for all single-query scripts, symmetrically with how the batch path already tolerates
+   extras. Preferred.
+2. Add the five params to `taint.sc`'s `exec` signature as ignored passthroughs (mirrors batch semantics but
+   duplicates the JSON parsing already in the script; narrower).
+3. Route `_usable_flows` through `run_batch` with a one-element request (makes single-query taint share the
+   tolerant path; changes the call shape upstream).
+
+Add a regression test under the joern-gated suite: `cpg.run("taint", request_params(path_spec, …))` must
+return a parsed payload (not `None`) on a fixture CPG, asserting the instrument actually executed.

.kiro/steering/roadmap.md

@@ -132,6 +132,14 @@ against a realistic pre-push window (≤120 s target, 300 s hard):
     5 pilot regions in LLM-guided-source mode; measure source naming, traced finding, catch lift vs regex 1/5, and
     false-alarm on ordinary changes; stage 5–10 → ~10–20 → full on lift. #3 change-anchoring deprioritised until
     tiers yield findings.
+  - **BANKS FIXED-SEARCH MVP DONE (2026-10-12) — "#2 guided by #1" closes on an unseen target.** See brief
+    "BANKS FIXED-SEARCH MVP" + `benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/`. On banks (the
+    library that gave the engine zero), injecting the two annotations an LLM steer would supply (param-as-source
+    + `write_text`/`read_text` sink) makes the engine resolve `prompt → path-join → write_text` where the
+    shipped search resolved nothing. Proves the governing principle "no wrong targets, only wrong searching" and
+    names the exact inputs the DSPy-optimised steer must learn. Filed instrument bug:
+    `.kiro/specs/engine-change-scoped-analysis/bugs/taint-sc-single-query-signature-skew.md` (single-query taint
+    path dead on an exec-signature skew; batch path unaffected).
   - LLM: L1 model latency and calls per push; L2 the recall LIFT of the LLM tier over regex+dataflow on the pool.
   - Deliverable: a barrier table (budget × tier combo → fits? measured recall/FP) → the minimal releasable config.
 

benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/NOTES.md

@@ -0,0 +1,74 @@
+# Banks fixed-search MVP — "no wrong targets, only wrong searching"
+
+Date: 2026-10-12 · Host: clean (local docker, replay-task image) · Repo HEAD: `7ab5c2a`
+Target: `masci/banks` @ `5e7093c9b3c2bbe2e0bc281f5a3602bbdabf00b9` (the "library" that gave the static engine ZERO path findings in the engine-on pilot).
+
+## Thesis under test
+
+The governing principle (maintainer, 2026-10-12): we have a **search-with-proof** architecture, so there
+are **no wrong targets, only wrong searching**. Banks produced zero path-family findings in the engine-on
+pilot not because it is a wrong target but because the SEARCH was not annotated for it:
+
+1. the **source** — `prompt.name`, a function parameter — is not recognised (`entry:false`, so
+   `_parameters_are_input` returns False and parameters are not treated as untrusted); and
+2. the **sink** — `Path.write_text` / `Path.read_text` at a joined path — is not in the shipped
+   path-family sink list.
+
+Inject exactly those two annotations (what a per-target LLM steer would supply) and check whether the engine
+now traces `prompt → path-join → write_text` where the shipped search produced nothing.
+
+## Instrument (verified before believing the number)
+
+Per AGENTS.md "verify the instrument": the first MVP attempt (single-query `cpg.run("taint", …)` path)
+returned 0/0, but that was an **instrument failure**, not a result — every query died on
+`cpg query taint failed` (see the filed bug `taint-sc-single-query-signature-skew.md`). This record uses the
+**batch path** (`cpg.run_batch("taint", {requestsFile JSON})`), the same invocation the real engine-on scans
+use. Confirmed it ran: CPG built, `run_batch` available, and the fixed search returned **concrete sink
+strings** (`prompt_file.write_text(prompt.raw)`) — the opposite of the v1 failure's empty rows. Harness:
+`banks_mvp2.py`. Raw output: `results.out`.
+
+## Setup
+
+Same target, same CPG; only the search changes.
+
+- Functions probed: `from_prompt_path`, `set`, `_load`, `_scan` in `src/banks/registries/directory.py`.
+- BASELINE = shipped `path` taint spec, `parameter_sources=False`.
+- FIXED = shipped spec + `write_text`/`read_text` added to path sinks, `parameter_sources=True`.
+
+## Result — flow-row counts
+
+| Function          | Baseline (shipped)   | Fixed search (annotation ON)                                  |
+|-------------------|----------------------|---------------------------------------------------------------|
+| `from_prompt_path`| 1 row (`sink=None`)  | **4 rows** — incl. `sink=prompt_file.write_text(prompt.raw)  source=prompt` |
+| `set`             | 1 row (`sink=None`)  | 1 row (`sink=None`)                                           |
+| `_load`           | 1 row (`sink=None`)  | 2 rows (`sink=None`)                                          |
+| `_scan`           | 2 rows (`sink=None`) | **4 rows** — incl. `sink=self._index_path.write_text(...)`    |
+| **Total**         | **5**                | **11**                                                        |
+
+### What the counts actually mean (honest read)
+
+The raw totals (5 vs 11) **overstate** the effect. All 5 baseline rows have `sink=None, source=None` — they
+are census/evidence rows, **not** resolved source→sink path flows. That is consistent with the engine-on
+pilot producing zero path-family findings on banks. The result that carries the thesis is the
+**resolved-flow delta**:
+
+- **Baseline: 0 resolved `prompt → write_text` flows.**
+- **Fixed search: the engine resolves `prompt → path-join → prompt_file.write_text(prompt.raw)` in
+  `from_prompt_path`** (`source=prompt`), plus a second `write_text` sink in `_scan`. These appear **only**
+  with the two injected annotations.
+
+## Verdict
+
+On the exact target that gave the static engine zero, the engine traces the real flow once the search is
+corrected. It needed exactly **two** annotations: the parameter-as-source and the missing file-write sink.
+This demonstrates **"no wrong targets, only wrong searching"** on a concrete case, and names the precise
+inputs the per-target LLM steer (DSPy-optimised, framework-seeded) must learn to supply.
+
+## Caveats (not papered over)
+
+1. One `from_prompt_path` row shows `source=path` rather than `prompt` — the parameter-source annotation
+   fires broadly, not yet pinned to the untrusted parameter. A precision knob for the steering loop, not a
+   blocker.
+2. This shows the engine *traces* the flow; it is **not** the runtime differential proof. Proving
+   attacker-reachability remains the async BLOCK lane (search-with-proof verifier).
+3. The single-query taint path is broken (filed separately); this record uses the batch path deliberately.

benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/banks_mvp2.py

@@ -0,0 +1,45 @@
+"""Banks FIXED-SEARCH MVP v2 (batch path -- the working invocation; the single cpg.run path is broken by a
+taint.sc exec-signature skew). Build banks CPG, run the taint BATCH on the vuln functions with the annotation
+OFF (shipped) vs ON (parameter_sources=True + write_text/read_text added to path sinks). A non-empty flow row =
+the engine traced source->sink. Thesis: same target, only the search changes."""
+import dataclasses, json, sys
+from pathlib import Path
+sys.path.insert(0, "/venv/lib/python3.12/site-packages")
+from openultrasast.cpg.backend import resolve_cpg_backend
+from openultrasast.model.specs import taint_specs
+from openultrasast.model.taint import request_params
+
+ROOT = Path("/work/banks"); VP = "src/banks/registries/directory.py"
+FUNCS = ["from_prompt_path", "set", "_load", "_scan"]
+
+backend = resolve_cpg_backend()
+cpg = backend.build(ROOT, language="python")
+if cpg is None or not callable(getattr(cpg, "run_batch", None)):
+    print("CPG build failed or no run_batch"); raise SystemExit(1)
+print("CPG built; run_batch available.", flush=True)
+
+base = taint_specs(language="python")["path"]
+annot = dataclasses.replace(base, sinks=tuple(sorted(set(base.sinks) | {"write_text", "read_text"})))
+print("baseline sinks:", base.sinks)
+print("annotated sinks:", annot.sinks, flush=True)
+
+def run(label, spec, fn, psrc):
+    reqs = {"r1": request_params(spec, function=fn, file=VP, parameter_sources=psrc)}
+    try:
+        out = cpg.run_batch("taint", reqs)
+    except Exception as e:
+        print(f"  [{label}] {fn}: ERROR {type(e).__name__}: {str(e)[:150]}"); return 0
+    rows = (out or {}).get("r1") or []
+    n = len([r for r in rows if isinstance(r, (dict,)) ])
+    print(f"  [{label}] {fn} psrc={psrc}: {len(rows)} flow row(s)")
+    for r in rows[:3]:
+        if isinstance(r, dict):
+            print(f"      sink={r.get('sink')} source={r.get('source')} line={r.get('line')} evidence={str(r.get('evidence'))[:60]}")
+    return len(rows)
+
+print("\n=== BASELINE (shipped search) ===", flush=True)
+b = sum(run("baseline", base, fn, False) for fn in FUNCS)
+print("\n=== FIXED SEARCH (annotation: param-source + write_text/read_text sink) ===", flush=True)
+f = sum(run("fixed", annot, fn, True) for fn in FUNCS)
+print(f"\n=== RESULT: baseline flow rows={b}  fixed-search flow rows={f} ===")
+print("FIXED SEARCH TRACES BANKS (search was wrong, not the target)" if f > b else "no change", flush=True)

benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/results.out

@@ -0,0 +1,32 @@
+CPG built; run_batch available.
+baseline sinks: ('FileResponse', 'open', 'os.remove', 'os.unlink', 'send_file', 'send_from_directory', 'shutil.copy', 'shutil.move', 'shutil.rmtree')
+annotated sinks: ('FileResponse', 'open', 'os.remove', 'os.unlink', 'read_text', 'send_file', 'send_from_directory', 'shutil.copy', 'shutil.move', 'shutil.rmtree', 'write_text')
+
+=== BASELINE (shipped search) ===
+  [baseline] from_prompt_path psrc=False: 1 flow row(s)
+      sink=None source=None line=None evidence=None
+  [baseline] set psrc=False: 1 flow row(s)
+      sink=None source=None line=None evidence=None
+  [baseline] _load psrc=False: 1 flow row(s)
+      sink=None source=None line=None evidence=None
+  [baseline] _scan psrc=False: 2 flow row(s)
+      sink=None source=None line=None evidence=None
+      sink=None source=None line=None evidence=None
+
+=== FIXED SEARCH (annotation: param-source + write_text/read_text sink) ===
+  [fixed] from_prompt_path psrc=True: 4 flow row(s)
+      sink=prompt_file.write_text(prompt.raw) source=path line=None evidence=None
+      sink=prompt_file.write_text(prompt.raw) source=prompt line=None evidence=None
+      sink=None source=None line=None evidence=None
+  [fixed] set psrc=True: 1 flow row(s)
+      sink=None source=None line=None evidence=None
+  [fixed] _load psrc=True: 2 flow row(s)
+      sink=None source=None line=None evidence=None
+      sink=None source=None line=None evidence=None
+  [fixed] _scan psrc=True: 4 flow row(s)
+      sink=self._index_path.write_text(self._index.model_dump_json()) source=self line=None evidence=None
+      sink=None source=None line=None evidence=None
+      sink=None source=None line=None evidence=None
+
+=== RESULT: baseline flow rows=5  fixed-search flow rows=11 ===
+FIXED SEARCH TRACES BANKS (search was wrong, not the target)