measure(g2): banks fixed-search MVP -- "no wrong targets, only wrong searching" demonstrated

Same CPG, only the search changes. Baseline (shipped path spec) resolves 0 prompt->write_text flows on banks; injecting the two annotations an LLM steer would supply (parameter_sources=True + write_text/read_text path sinks) makes the engine resolve prompt -> path-join -> prompt_file.write_text(prompt.raw). First measured instance of "#2 guided by #1" closing on an unseen target; fix needed exactly the param-source + file-write-sink annotations. Files the instrument bug the first MVP hit: single-query cpg.run("taint", request_params(...)) is dead -- request_params emits 5 keys (guards/fixedOrigin/ originAnchors/quotedSanitizers/fieldParameterSources) absent from taint.sc's exec signature; both single-query sub-routes reject them (silent None). Batch path (run_batch->requestsFile JSON) unaffected; push/scan route uses batch. Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-16 03:49:26 +00:00 · 2026-10-12 14:02:41 +02:00
parent 7ab5c2ab2d
commit 8f7fc4ae83
6 changed files with 272 additions and 0 deletions
@@ -462,3 +462,30 @@ suspicion. So:
 - **Bet #3 (the proof arm) is the ASYNC BLOCK lane**: run `search/verify` on the plane/CI for the suspicions worth
  proving, with authored (eventually DSPy-authored) demos. Worth an empirical run on one tractable target
  (mcp-remote) to confirm the arm demonstrates on a real unseen vuln — a bounded, separate effort.
 ## BANKS FIXED-SEARCH MVP (2026-10-12) — "no wrong targets, only wrong searching" demonstrated
 Record: `benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/`. Target: banks (the "library" that gave
 the engine ZERO path findings in the pilot). Same target, same CPG — only the search changes.
 - Injected the two annotations a per-target LLM steer would supply: **parameter-as-source**
  (`parameter_sources=True`, so `prompt.name` counts as untrusted) + the **missing file-write sink**
  (`write_text`/`read_text` added to path sinks).
 - **Baseline (shipped search): 0 resolved `prompt → write_text` flows** — its 5 flow rows are all census/
  evidence (`sink=None`), matching the pilot's zero path findings.
 - **Fixed search: the engine resolves `prompt → path-join → prompt_file.write_text(prompt.raw)` in
  `from_prompt_path`** (`source=prompt`), plus a second `write_text` sink in `_scan`. These appear ONLY with
  the annotations.
 - **Thesis demonstrated on a concrete case:** banks was not a wrong target; the search was unannotated. The
  fix needed exactly **two** inputs — the param-source and the file-write sink — which are precisely what the
  per-target LLM steer (DSPy-optimised, framework-seeded) has to learn to supply. This is the first measured
  instance of the "#2 guided by #1" loop closing on an unseen target.
 - Caveats: (1) one row shows `source=path` not `prompt` — the param-source fires broadly, a precision knob for
  the steering loop; (2) this is a TRACE, not the runtime differential proof (proof stays the async BLOCK lane).
 **Instrument note / filed bug:** the first MVP used the single-query `cpg.run("taint", …)` path and returned
 0/0 — an INSTRUMENT FAILURE, not a result (`cpg query taint failed` on every query). Root cause filed in
 `bugs/taint-sc-single-query-signature-skew.md`: `request_params()` emits 5 keys (`guards`, `fixedOrigin`,
 `originAnchors`, `quotedSanitizers`, `fieldParameterSources`) that `taint.sc`'s `exec` signature does not
 declare, so both single-query sub-routes (CLI `--param` and session named-arg) reject them. The batch path
 (`run_batch` → `requestsFile` JSON) tolerates them, which is why the pilot scans ran and this MVP used it.
@@ -0,0 +1,86 @@
 # BUG: `taint.sc` single-query path broken by exec-signature skew
 Filed: 2026-10-12 · Repo HEAD at filing: `7ab5c2a` · Severity: **high (silent zero)** · Status: OPEN
 ## Summary
 Any call on the **single-query** taint path — `cpg.run("taint", request_params(...))` — fails on **both**
 of its sub-routes, returns `None`, and is read upstream as "no flows / clean". `request_params()` emits five
 keys that `taint.sc`'s `@main def exec(...)` does not declare, and the single-query dispatch passes every
 param straight to the script. The **batch** path (`run_batch` → `requestsFile` JSON) is unaffected, which is
 why the engine-on scans ran while the single-query MVP did not.
 This is exactly the failure shape AGENTS.md warns about: a tool that could not read/execute its input reports
 a plausible zero that is indistinguishable from a clean result.
 ## Mechanism
 `request_params()` (`src/openultrasast/model/taint.py`) always emits, among others:
 - `fieldParameterSources` (line 54)
 - `fixedOrigin` (line 65)
 - `originAnchors` (line 66)
 - `quotedSanitizers` (line 67)
 - `guards` (line 68)
 `taint.sc`'s signature (`src/openultrasast/cpg/queries/taint.sc:41` `@main def exec(...)`) declares **none**
 of those five. Its 19 params are: `cpgFile, sources, sinks, sanitizers, function, file, callDepth,
 boundedSinks, parameterSources, fieldSources, evidenceOnly, hookCallbacks, dispatchApply, dispatchValue,
 contextFilter, contextPaths, requests, requestsFile, trace`.
 `backend.query()` (`src/openultrasast/cpg/backend.py:1266`) dispatches a single query two ways, **both** of
 which choke on the undeclared keys:
 - **Disposable / CLI route** (lines 1276–1278): appends `--param {key}={value}` for every param. joern errors
  `Unknown arguments: --fixedOrigin --guards --quotedSanitizers --originAnchors --fieldParameterSources`
  (and, as a knock-on, `Missing argument: --cpgFile`). Observed verbatim via `banks_debug.py` on 2026-10-12.
 - **Session route** (lines 1321–1326): defines `def ousast_taint(...)` by renaming `@main def exec(` and then
  calls `ousast_taint(key = value, ...)` with named args. The same five keys are not parameters of that
  method → the Scala named-arg call fails to compile. (By construction; the rename preserves the exact
  `exec` parameter list.)
 Why the batch path survives: `run_batch` writes the per-request params **inside** the `requestsFile` JSON
 object, and `taint.sc` parses that JSON itself (`requestsJson`, taint.sc:67). The extra fields ride as JSON
 keys, never as `exec` arguments, so they are tolerated.
 ## Blast radius
 Broken (single-query taint):
 - `taint._usable_flows` (`taint.py:124`) → `verdicts()` (`taint.py:131`) → `verdict()` (`taint.py:99`).
 - `pipeline.scan_region` dispatches taint families to `taint_verdict` (`pipeline.py:121`) → so **per-region
  single-query taint via `scan_region` is dead** whenever the disposable or session route is taken.
 Unaffected:
 - The push/scan **batch** path: `scan.py` uses `run_batch("taint", …)` (scan.py:1259, 1264, 1387). This is
  why the engine-on pilot produced results.
 - `config` and `dominance` single-queries (`config_value.py:46`, `dominance.py:54`) — their `.sc` signatures
  are presumed to match what their specs emit (confirm separately; not exercised here).
 Open question to close: enumerate every production path that reaches `scan_region`/`verdict()` taint rather
 than the batch route, to state whether any shipped push/replay flow silently loses taint flows today, or
 whether the single-query path is only hit by pair measurements and ad-hoc harnesses.
 ## Evidence
 - `benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/` — the batch path works (11 flow rows,
  `write_text` sinks); the v1 single-query attempt returned 0/0 on `cpg query taint failed`.
 - Raw joern error captured by `banks_debug.py` (scratchpad): `Missing argument: --cpgFile; Unknown
  arguments: --fixedOrigin --guards --quotedSanitizers --originAnchors --fieldParameterSources`.
 ## Suggested fix (not yet applied)
 Pick one, lowest-risk first:
 1. **Make `backend.query()` filter params to the script's declared signature** before building `--param` /
   named-arg lists (parse the `@main def exec(` arg names once, intersect). Fixes every current and future
   signature skew for all single-query scripts, symmetrically with how the batch path already tolerates
   extras. Preferred.
 2. Add the five params to `taint.sc`'s `exec` signature as ignored passthroughs (mirrors batch semantics but
   duplicates the JSON parsing already in the script; narrower).
 3. Route `_usable_flows` through `run_batch` with a one-element request (makes single-query taint share the
   tolerant path; changes the call shape upstream).
 Add a regression test under the joern-gated suite: `cpg.run("taint", request_params(path_spec, …))` must
 return a parsed payload (not `None`) on a fixture CPG, asserting the instrument actually executed.
@@ -132,6 +132,14 @@ against a realistic pre-push window (≤120 s target, 300 s hard):
    5 pilot regions in LLM-guided-source mode; measure source naming, traced finding, catch lift vs regex 1/5, and
    false-alarm on ordinary changes; stage 5–10 → ~10–20 → full on lift. #3 change-anchoring deprioritised until
    tiers yield findings.
  - **BANKS FIXED-SEARCH MVP DONE (2026-10-12) — "#2 guided by #1" closes on an unseen target.** See brief
    "BANKS FIXED-SEARCH MVP" + `benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/`. On banks (the
    library that gave the engine zero), injecting the two annotations an LLM steer would supply (param-as-source
    + `write_text`/`read_text` sink) makes the engine resolve `prompt → path-join → write_text` where the
    shipped search resolved nothing. Proves the governing principle "no wrong targets, only wrong searching" and
    names the exact inputs the DSPy-optimised steer must learn. Filed instrument bug:
    `.kiro/specs/engine-change-scoped-analysis/bugs/taint-sc-single-query-signature-skew.md` (single-query taint
    path dead on an exec-signature skew; batch path unaffected).
  - LLM: L1 model latency and calls per push; L2 the recall LIFT of the LLM tier over regex+dataflow on the pool.
  - Deliverable: a barrier table (budget × tier combo → fits? measured recall/FP) → the minimal releasable config.
@@ -0,0 +1,74 @@
 # Banks fixed-search MVP — "no wrong targets, only wrong searching"
 Date: 2026-10-12 · Host: clean (local docker, replay-task image) · Repo HEAD: `7ab5c2a`
 Target: `masci/banks` @ `5e7093c9b3c2bbe2e0bc281f5a3602bbdabf00b9` (the "library" that gave the static engine ZERO path findings in the engine-on pilot).
 ## Thesis under test
 The governing principle (maintainer, 2026-10-12): we have a **search-with-proof** architecture, so there
 are **no wrong targets, only wrong searching**. Banks produced zero path-family findings in the engine-on
 pilot not because it is a wrong target but because the SEARCH was not annotated for it:
 1. the **source** — `prompt.name`, a function parameter — is not recognised (`entry:false`, so
   `_parameters_are_input` returns False and parameters are not treated as untrusted); and
 2. the **sink** — `Path.write_text` / `Path.read_text` at a joined path — is not in the shipped
   path-family sink list.
 Inject exactly those two annotations (what a per-target LLM steer would supply) and check whether the engine
 now traces `prompt → path-join → write_text` where the shipped search produced nothing.
 ## Instrument (verified before believing the number)
 Per AGENTS.md "verify the instrument": the first MVP attempt (single-query `cpg.run("taint", …)` path)
 returned 0/0, but that was an **instrument failure**, not a result — every query died on
 `cpg query taint failed` (see the filed bug `taint-sc-single-query-signature-skew.md`). This record uses the
 **batch path** (`cpg.run_batch("taint", {requestsFile JSON})`), the same invocation the real engine-on scans
 use. Confirmed it ran: CPG built, `run_batch` available, and the fixed search returned **concrete sink
 strings** (`prompt_file.write_text(prompt.raw)`) — the opposite of the v1 failure's empty rows. Harness:
 `banks_mvp2.py`. Raw output: `results.out`.
 ## Setup
 Same target, same CPG; only the search changes.
 - Functions probed: `from_prompt_path`, `set`, `_load`, `_scan` in `src/banks/registries/directory.py`.
 - BASELINE = shipped `path` taint spec, `parameter_sources=False`.
 - FIXED = shipped spec + `write_text`/`read_text` added to path sinks, `parameter_sources=True`.
 ## Result — flow-row counts
 | Function          | Baseline (shipped)   | Fixed search (annotation ON)                                  |
 |-------------------|----------------------|---------------------------------------------------------------|
 | `from_prompt_path`| 1 row (`sink=None`)  | **4 rows** — incl. `sink=prompt_file.write_text(prompt.raw)  source=prompt` |
 | `set`             | 1 row (`sink=None`)  | 1 row (`sink=None`)                                           |
 | `_load`           | 1 row (`sink=None`)  | 2 rows (`sink=None`)                                          |
 | `_scan`           | 2 rows (`sink=None`) | **4 rows** — incl. `sink=self._index_path.write_text(...)`    |
 | **Total**         | **5**                | **11**                                                        |
 ### What the counts actually mean (honest read)
 The raw totals (5 vs 11) **overstate** the effect. All 5 baseline rows have `sink=None, source=None` — they
 are census/evidence rows, **not** resolved source→sink path flows. That is consistent with the engine-on
 pilot producing zero path-family findings on banks. The result that carries the thesis is the
 **resolved-flow delta**:
 - **Baseline: 0 resolved `prompt → write_text` flows.**
 - **Fixed search: the engine resolves `prompt → path-join → prompt_file.write_text(prompt.raw)` in
  `from_prompt_path`** (`source=prompt`), plus a second `write_text` sink in `_scan`. These appear **only**
  with the two injected annotations.
 ## Verdict
 On the exact target that gave the static engine zero, the engine traces the real flow once the search is
 corrected. It needed exactly **two** annotations: the parameter-as-source and the missing file-write sink.
 This demonstrates **"no wrong targets, only wrong searching"** on a concrete case, and names the precise
 inputs the per-target LLM steer (DSPy-optimised, framework-seeded) must learn to supply.
 ## Caveats (not papered over)
 1. One `from_prompt_path` row shows `source=path` rather than `prompt` — the parameter-source annotation
   fires broadly, not yet pinned to the untrusted parameter. A precision knob for the steering loop, not a
   blocker.
 2. This shows the engine *traces* the flow; it is **not** the runtime differential proof. Proving
   attacker-reachability remains the async BLOCK lane (search-with-proof verifier).
 3. The single-query taint path is broken (filed separately); this record uses the batch path deliberately.
@@ -0,0 +1,45 @@
 """Banks FIXED-SEARCH MVP v2 (batch path -- the working invocation; the single cpg.run path is broken by a
 taint.sc exec-signature skew). Build banks CPG, run the taint BATCH on the vuln functions with the annotation
 OFF (shipped) vs ON (parameter_sources=True + write_text/read_text added to path sinks). A non-empty flow row =
 the engine traced source->sink. Thesis: same target, only the search changes."""
 import dataclasses, json, sys
 from pathlib import Path
 sys.path.insert(0, "/venv/lib/python3.12/site-packages")
 from openultrasast.cpg.backend import resolve_cpg_backend
 from openultrasast.model.specs import taint_specs
 from openultrasast.model.taint import request_params
 ROOT = Path("/work/banks"); VP = "src/banks/registries/directory.py"
 FUNCS = ["from_prompt_path", "set", "_load", "_scan"]
 backend = resolve_cpg_backend()
 cpg = backend.build(ROOT, language="python")
 if cpg is None or not callable(getattr(cpg, "run_batch", None)):
    print("CPG build failed or no run_batch"); raise SystemExit(1)
 print("CPG built; run_batch available.", flush=True)
 base = taint_specs(language="python")["path"]
 annot = dataclasses.replace(base, sinks=tuple(sorted(set(base.sinks) | {"write_text", "read_text"})))
 print("baseline sinks:", base.sinks)
 print("annotated sinks:", annot.sinks, flush=True)
 def run(label, spec, fn, psrc):
    reqs = {"r1": request_params(spec, function=fn, file=VP, parameter_sources=psrc)}
    try:
        out = cpg.run_batch("taint", reqs)
    except Exception as e:
        print(f"  [{label}] {fn}: ERROR {type(e).__name__}: {str(e)[:150]}"); return 0
    rows = (out or {}).get("r1") or []
    n = len([r for r in rows if isinstance(r, (dict,)) ])
    print(f"  [{label}] {fn} psrc={psrc}: {len(rows)} flow row(s)")
    for r in rows[:3]:
        if isinstance(r, dict):
            print(f"      sink={r.get('sink')} source={r.get('source')} line={r.get('line')} evidence={str(r.get('evidence'))[:60]}")
    return len(rows)
 print("\n=== BASELINE (shipped search) ===", flush=True)
 b = sum(run("baseline", base, fn, False) for fn in FUNCS)
 print("\n=== FIXED SEARCH (annotation: param-source + write_text/read_text sink) ===", flush=True)
 f = sum(run("fixed", annot, fn, True) for fn in FUNCS)
 print(f"\n=== RESULT: baseline flow rows={b}  fixed-search flow rows={f} ===")
 print("FIXED SEARCH TRACES BANKS (search was wrong, not the target)" if f > b else "no change", flush=True)
@@ -0,0 +1,32 @@
 CPG built; run_batch available.
 baseline sinks: ('FileResponse', 'open', 'os.remove', 'os.unlink', 'send_file', 'send_from_directory', 'shutil.copy', 'shutil.move', 'shutil.rmtree')
 annotated sinks: ('FileResponse', 'open', 'os.remove', 'os.unlink', 'read_text', 'send_file', 'send_from_directory', 'shutil.copy', 'shutil.move', 'shutil.rmtree', 'write_text')
 === BASELINE (shipped search) ===
  [baseline] from_prompt_path psrc=False: 1 flow row(s)
      sink=None source=None line=None evidence=None
  [baseline] set psrc=False: 1 flow row(s)
      sink=None source=None line=None evidence=None
  [baseline] _load psrc=False: 1 flow row(s)
      sink=None source=None line=None evidence=None
  [baseline] _scan psrc=False: 2 flow row(s)
      sink=None source=None line=None evidence=None
      sink=None source=None line=None evidence=None
 === FIXED SEARCH (annotation: param-source + write_text/read_text sink) ===
  [fixed] from_prompt_path psrc=True: 4 flow row(s)
      sink=prompt_file.write_text(prompt.raw) source=path line=None evidence=None
      sink=prompt_file.write_text(prompt.raw) source=prompt line=None evidence=None
      sink=None source=None line=None evidence=None
  [fixed] set psrc=True: 1 flow row(s)
      sink=None source=None line=None evidence=None
  [fixed] _load psrc=True: 2 flow row(s)
      sink=None source=None line=None evidence=None
      sink=None source=None line=None evidence=None
  [fixed] _scan psrc=True: 4 flow row(s)
      sink=self._index_path.write_text(self._index.model_dump_json()) source=self line=None evidence=None
      sink=None source=None line=None evidence=None
      sink=None source=None line=None evidence=None
 === RESULT: baseline flow rows=5  fixed-search flow rows=11 ===
 FIXED SEARCH TRACES BANKS (search was wrong, not the target)