diff --git a/.kiro/specs/engine-change-scoped-analysis/brief.md b/.kiro/specs/engine-change-scoped-analysis/brief.md index d754098..fbd3f73 100644 --- a/.kiro/specs/engine-change-scoped-analysis/brief.md +++ b/.kiro/specs/engine-change-scoped-analysis/brief.md @@ -462,3 +462,30 @@ suspicion. So: - **Bet #3 (the proof arm) is the ASYNC BLOCK lane**: run `search/verify` on the plane/CI for the suspicions worth proving, with authored (eventually DSPy-authored) demos. Worth an empirical run on one tractable target (mcp-remote) to confirm the arm demonstrates on a real unseen vuln — a bounded, separate effort. + +## BANKS FIXED-SEARCH MVP (2026-10-12) — "no wrong targets, only wrong searching" demonstrated + +Record: `benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/`. Target: banks (the "library" that gave +the engine ZERO path findings in the pilot). Same target, same CPG — only the search changes. + +- Injected the two annotations a per-target LLM steer would supply: **parameter-as-source** + (`parameter_sources=True`, so `prompt.name` counts as untrusted) + the **missing file-write sink** + (`write_text`/`read_text` added to path sinks). +- **Baseline (shipped search): 0 resolved `prompt → write_text` flows** — its 5 flow rows are all census/ + evidence (`sink=None`), matching the pilot's zero path findings. +- **Fixed search: the engine resolves `prompt → path-join → prompt_file.write_text(prompt.raw)` in + `from_prompt_path`** (`source=prompt`), plus a second `write_text` sink in `_scan`. These appear ONLY with + the annotations. +- **Thesis demonstrated on a concrete case:** banks was not a wrong target; the search was unannotated. The + fix needed exactly **two** inputs — the param-source and the file-write sink — which are precisely what the + per-target LLM steer (DSPy-optimised, framework-seeded) has to learn to supply. This is the first measured + instance of the "#2 guided by #1" loop closing on an unseen target. +- Caveats: (1) one row shows `source=path` not `prompt` — the param-source fires broadly, a precision knob for + the steering loop; (2) this is a TRACE, not the runtime differential proof (proof stays the async BLOCK lane). + +**Instrument note / filed bug:** the first MVP used the single-query `cpg.run("taint", …)` path and returned +0/0 — an INSTRUMENT FAILURE, not a result (`cpg query taint failed` on every query). Root cause filed in +`bugs/taint-sc-single-query-signature-skew.md`: `request_params()` emits 5 keys (`guards`, `fixedOrigin`, +`originAnchors`, `quotedSanitizers`, `fieldParameterSources`) that `taint.sc`'s `exec` signature does not +declare, so both single-query sub-routes (CLI `--param` and session named-arg) reject them. The batch path +(`run_batch` → `requestsFile` JSON) tolerates them, which is why the pilot scans ran and this MVP used it. diff --git a/.kiro/specs/engine-change-scoped-analysis/bugs/taint-sc-single-query-signature-skew.md b/.kiro/specs/engine-change-scoped-analysis/bugs/taint-sc-single-query-signature-skew.md new file mode 100644 index 0000000..e3ae6f6 --- /dev/null +++ b/.kiro/specs/engine-change-scoped-analysis/bugs/taint-sc-single-query-signature-skew.md @@ -0,0 +1,86 @@ +# BUG: `taint.sc` single-query path broken by exec-signature skew + +Filed: 2026-10-12 · Repo HEAD at filing: `7ab5c2a` · Severity: **high (silent zero)** · Status: OPEN + +## Summary + +Any call on the **single-query** taint path — `cpg.run("taint", request_params(...))` — fails on **both** +of its sub-routes, returns `None`, and is read upstream as "no flows / clean". `request_params()` emits five +keys that `taint.sc`'s `@main def exec(...)` does not declare, and the single-query dispatch passes every +param straight to the script. The **batch** path (`run_batch` → `requestsFile` JSON) is unaffected, which is +why the engine-on scans ran while the single-query MVP did not. + +This is exactly the failure shape AGENTS.md warns about: a tool that could not read/execute its input reports +a plausible zero that is indistinguishable from a clean result. + +## Mechanism + +`request_params()` (`src/openultrasast/model/taint.py`) always emits, among others: + +- `fieldParameterSources` (line 54) +- `fixedOrigin` (line 65) +- `originAnchors` (line 66) +- `quotedSanitizers` (line 67) +- `guards` (line 68) + +`taint.sc`'s signature (`src/openultrasast/cpg/queries/taint.sc:41` `@main def exec(...)`) declares **none** +of those five. Its 19 params are: `cpgFile, sources, sinks, sanitizers, function, file, callDepth, +boundedSinks, parameterSources, fieldSources, evidenceOnly, hookCallbacks, dispatchApply, dispatchValue, +contextFilter, contextPaths, requests, requestsFile, trace`. + +`backend.query()` (`src/openultrasast/cpg/backend.py:1266`) dispatches a single query two ways, **both** of +which choke on the undeclared keys: + +- **Disposable / CLI route** (lines 1276–1278): appends `--param {key}={value}` for every param. joern errors + `Unknown arguments: --fixedOrigin --guards --quotedSanitizers --originAnchors --fieldParameterSources` + (and, as a knock-on, `Missing argument: --cpgFile`). Observed verbatim via `banks_debug.py` on 2026-10-12. +- **Session route** (lines 1321–1326): defines `def ousast_taint(...)` by renaming `@main def exec(` and then + calls `ousast_taint(key = value, ...)` with named args. The same five keys are not parameters of that + method → the Scala named-arg call fails to compile. (By construction; the rename preserves the exact + `exec` parameter list.) + +Why the batch path survives: `run_batch` writes the per-request params **inside** the `requestsFile` JSON +object, and `taint.sc` parses that JSON itself (`requestsJson`, taint.sc:67). The extra fields ride as JSON +keys, never as `exec` arguments, so they are tolerated. + +## Blast radius + +Broken (single-query taint): + +- `taint._usable_flows` (`taint.py:124`) → `verdicts()` (`taint.py:131`) → `verdict()` (`taint.py:99`). +- `pipeline.scan_region` dispatches taint families to `taint_verdict` (`pipeline.py:121`) → so **per-region + single-query taint via `scan_region` is dead** whenever the disposable or session route is taken. + +Unaffected: + +- The push/scan **batch** path: `scan.py` uses `run_batch("taint", …)` (scan.py:1259, 1264, 1387). This is + why the engine-on pilot produced results. +- `config` and `dominance` single-queries (`config_value.py:46`, `dominance.py:54`) — their `.sc` signatures + are presumed to match what their specs emit (confirm separately; not exercised here). + +Open question to close: enumerate every production path that reaches `scan_region`/`verdict()` taint rather +than the batch route, to state whether any shipped push/replay flow silently loses taint flows today, or +whether the single-query path is only hit by pair measurements and ad-hoc harnesses. + +## Evidence + +- `benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/` — the batch path works (11 flow rows, + `write_text` sinks); the v1 single-query attempt returned 0/0 on `cpg query taint failed`. +- Raw joern error captured by `banks_debug.py` (scratchpad): `Missing argument: --cpgFile; Unknown + arguments: --fixedOrigin --guards --quotedSanitizers --originAnchors --fieldParameterSources`. + +## Suggested fix (not yet applied) + +Pick one, lowest-risk first: + +1. **Make `backend.query()` filter params to the script's declared signature** before building `--param` / + named-arg lists (parse the `@main def exec(` arg names once, intersect). Fixes every current and future + signature skew for all single-query scripts, symmetrically with how the batch path already tolerates + extras. Preferred. +2. Add the five params to `taint.sc`'s `exec` signature as ignored passthroughs (mirrors batch semantics but + duplicates the JSON parsing already in the script; narrower). +3. Route `_usable_flows` through `run_batch` with a one-element request (makes single-query taint share the + tolerant path; changes the call shape upstream). + +Add a regression test under the joern-gated suite: `cpg.run("taint", request_params(path_spec, …))` must +return a parsed payload (not `None`) on a fixture CPG, asserting the instrument actually executed. diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index 57fee6b..dfb56ae 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -132,6 +132,14 @@ against a realistic pre-push window (≤120 s target, 300 s hard): 5 pilot regions in LLM-guided-source mode; measure source naming, traced finding, catch lift vs regex 1/5, and false-alarm on ordinary changes; stage 5–10 → ~10–20 → full on lift. #3 change-anchoring deprioritised until tiers yield findings. + - **BANKS FIXED-SEARCH MVP DONE (2026-10-12) — "#2 guided by #1" closes on an unseen target.** See brief + "BANKS FIXED-SEARCH MVP" + `benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/`. On banks (the + library that gave the engine zero), injecting the two annotations an LLM steer would supply (param-as-source + + `write_text`/`read_text` sink) makes the engine resolve `prompt → path-join → write_text` where the + shipped search resolved nothing. Proves the governing principle "no wrong targets, only wrong searching" and + names the exact inputs the DSPy-optimised steer must learn. Filed instrument bug: + `.kiro/specs/engine-change-scoped-analysis/bugs/taint-sc-single-query-signature-skew.md` (single-query taint + path dead on an exec-signature skew; batch path unaffected). - LLM: L1 model latency and calls per push; L2 the recall LIFT of the LLM tier over regex+dataflow on the pool. - Deliverable: a barrier table (budget × tier combo → fits? measured recall/FP) → the minimal releasable config. diff --git a/benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/NOTES.md b/benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/NOTES.md new file mode 100644 index 0000000..07bc04f --- /dev/null +++ b/benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/NOTES.md @@ -0,0 +1,74 @@ +# Banks fixed-search MVP — "no wrong targets, only wrong searching" + +Date: 2026-10-12 · Host: clean (local docker, replay-task image) · Repo HEAD: `7ab5c2a` +Target: `masci/banks` @ `5e7093c9b3c2bbe2e0bc281f5a3602bbdabf00b9` (the "library" that gave the static engine ZERO path findings in the engine-on pilot). + +## Thesis under test + +The governing principle (maintainer, 2026-10-12): we have a **search-with-proof** architecture, so there +are **no wrong targets, only wrong searching**. Banks produced zero path-family findings in the engine-on +pilot not because it is a wrong target but because the SEARCH was not annotated for it: + +1. the **source** — `prompt.name`, a function parameter — is not recognised (`entry:false`, so + `_parameters_are_input` returns False and parameters are not treated as untrusted); and +2. the **sink** — `Path.write_text` / `Path.read_text` at a joined path — is not in the shipped + path-family sink list. + +Inject exactly those two annotations (what a per-target LLM steer would supply) and check whether the engine +now traces `prompt → path-join → write_text` where the shipped search produced nothing. + +## Instrument (verified before believing the number) + +Per AGENTS.md "verify the instrument": the first MVP attempt (single-query `cpg.run("taint", …)` path) +returned 0/0, but that was an **instrument failure**, not a result — every query died on +`cpg query taint failed` (see the filed bug `taint-sc-single-query-signature-skew.md`). This record uses the +**batch path** (`cpg.run_batch("taint", {requestsFile JSON})`), the same invocation the real engine-on scans +use. Confirmed it ran: CPG built, `run_batch` available, and the fixed search returned **concrete sink +strings** (`prompt_file.write_text(prompt.raw)`) — the opposite of the v1 failure's empty rows. Harness: +`banks_mvp2.py`. Raw output: `results.out`. + +## Setup + +Same target, same CPG; only the search changes. + +- Functions probed: `from_prompt_path`, `set`, `_load`, `_scan` in `src/banks/registries/directory.py`. +- BASELINE = shipped `path` taint spec, `parameter_sources=False`. +- FIXED = shipped spec + `write_text`/`read_text` added to path sinks, `parameter_sources=True`. + +## Result — flow-row counts + +| Function | Baseline (shipped) | Fixed search (annotation ON) | +|-------------------|----------------------|---------------------------------------------------------------| +| `from_prompt_path`| 1 row (`sink=None`) | **4 rows** — incl. `sink=prompt_file.write_text(prompt.raw) source=prompt` | +| `set` | 1 row (`sink=None`) | 1 row (`sink=None`) | +| `_load` | 1 row (`sink=None`) | 2 rows (`sink=None`) | +| `_scan` | 2 rows (`sink=None`) | **4 rows** — incl. `sink=self._index_path.write_text(...)` | +| **Total** | **5** | **11** | + +### What the counts actually mean (honest read) + +The raw totals (5 vs 11) **overstate** the effect. All 5 baseline rows have `sink=None, source=None` — they +are census/evidence rows, **not** resolved source→sink path flows. That is consistent with the engine-on +pilot producing zero path-family findings on banks. The result that carries the thesis is the +**resolved-flow delta**: + +- **Baseline: 0 resolved `prompt → write_text` flows.** +- **Fixed search: the engine resolves `prompt → path-join → prompt_file.write_text(prompt.raw)` in + `from_prompt_path`** (`source=prompt`), plus a second `write_text` sink in `_scan`. These appear **only** + with the two injected annotations. + +## Verdict + +On the exact target that gave the static engine zero, the engine traces the real flow once the search is +corrected. It needed exactly **two** annotations: the parameter-as-source and the missing file-write sink. +This demonstrates **"no wrong targets, only wrong searching"** on a concrete case, and names the precise +inputs the per-target LLM steer (DSPy-optimised, framework-seeded) must learn to supply. + +## Caveats (not papered over) + +1. One `from_prompt_path` row shows `source=path` rather than `prompt` — the parameter-source annotation + fires broadly, not yet pinned to the untrusted parameter. A precision knob for the steering loop, not a + blocker. +2. This shows the engine *traces* the flow; it is **not** the runtime differential proof. Proving + attacker-reachability remains the async BLOCK lane (search-with-proof verifier). +3. The single-query taint path is broken (filed separately); this record uses the batch path deliberately. diff --git a/benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/banks_mvp2.py b/benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/banks_mvp2.py new file mode 100644 index 0000000..fcc8409 --- /dev/null +++ b/benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/banks_mvp2.py @@ -0,0 +1,45 @@ +"""Banks FIXED-SEARCH MVP v2 (batch path -- the working invocation; the single cpg.run path is broken by a +taint.sc exec-signature skew). Build banks CPG, run the taint BATCH on the vuln functions with the annotation +OFF (shipped) vs ON (parameter_sources=True + write_text/read_text added to path sinks). A non-empty flow row = +the engine traced source->sink. Thesis: same target, only the search changes.""" +import dataclasses, json, sys +from pathlib import Path +sys.path.insert(0, "/venv/lib/python3.12/site-packages") +from openultrasast.cpg.backend import resolve_cpg_backend +from openultrasast.model.specs import taint_specs +from openultrasast.model.taint import request_params + +ROOT = Path("/work/banks"); VP = "src/banks/registries/directory.py" +FUNCS = ["from_prompt_path", "set", "_load", "_scan"] + +backend = resolve_cpg_backend() +cpg = backend.build(ROOT, language="python") +if cpg is None or not callable(getattr(cpg, "run_batch", None)): + print("CPG build failed or no run_batch"); raise SystemExit(1) +print("CPG built; run_batch available.", flush=True) + +base = taint_specs(language="python")["path"] +annot = dataclasses.replace(base, sinks=tuple(sorted(set(base.sinks) | {"write_text", "read_text"}))) +print("baseline sinks:", base.sinks) +print("annotated sinks:", annot.sinks, flush=True) + +def run(label, spec, fn, psrc): + reqs = {"r1": request_params(spec, function=fn, file=VP, parameter_sources=psrc)} + try: + out = cpg.run_batch("taint", reqs) + except Exception as e: + print(f" [{label}] {fn}: ERROR {type(e).__name__}: {str(e)[:150]}"); return 0 + rows = (out or {}).get("r1") or [] + n = len([r for r in rows if isinstance(r, (dict,)) ]) + print(f" [{label}] {fn} psrc={psrc}: {len(rows)} flow row(s)") + for r in rows[:3]: + if isinstance(r, dict): + print(f" sink={r.get('sink')} source={r.get('source')} line={r.get('line')} evidence={str(r.get('evidence'))[:60]}") + return len(rows) + +print("\n=== BASELINE (shipped search) ===", flush=True) +b = sum(run("baseline", base, fn, False) for fn in FUNCS) +print("\n=== FIXED SEARCH (annotation: param-source + write_text/read_text sink) ===", flush=True) +f = sum(run("fixed", annot, fn, True) for fn in FUNCS) +print(f"\n=== RESULT: baseline flow rows={b} fixed-search flow rows={f} ===") +print("FIXED SEARCH TRACES BANKS (search was wrong, not the target)" if f > b else "no change", flush=True) diff --git a/benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/results.out b/benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/results.out new file mode 100644 index 0000000..1dfad1c --- /dev/null +++ b/benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/results.out @@ -0,0 +1,32 @@ +CPG built; run_batch available. +baseline sinks: ('FileResponse', 'open', 'os.remove', 'os.unlink', 'send_file', 'send_from_directory', 'shutil.copy', 'shutil.move', 'shutil.rmtree') +annotated sinks: ('FileResponse', 'open', 'os.remove', 'os.unlink', 'read_text', 'send_file', 'send_from_directory', 'shutil.copy', 'shutil.move', 'shutil.rmtree', 'write_text') + +=== BASELINE (shipped search) === + [baseline] from_prompt_path psrc=False: 1 flow row(s) + sink=None source=None line=None evidence=None + [baseline] set psrc=False: 1 flow row(s) + sink=None source=None line=None evidence=None + [baseline] _load psrc=False: 1 flow row(s) + sink=None source=None line=None evidence=None + [baseline] _scan psrc=False: 2 flow row(s) + sink=None source=None line=None evidence=None + sink=None source=None line=None evidence=None + +=== FIXED SEARCH (annotation: param-source + write_text/read_text sink) === + [fixed] from_prompt_path psrc=True: 4 flow row(s) + sink=prompt_file.write_text(prompt.raw) source=path line=None evidence=None + sink=prompt_file.write_text(prompt.raw) source=prompt line=None evidence=None + sink=None source=None line=None evidence=None + [fixed] set psrc=True: 1 flow row(s) + sink=None source=None line=None evidence=None + [fixed] _load psrc=True: 2 flow row(s) + sink=None source=None line=None evidence=None + sink=None source=None line=None evidence=None + [fixed] _scan psrc=True: 4 flow row(s) + sink=self._index_path.write_text(self._index.model_dump_json()) source=self line=None evidence=None + sink=None source=None line=None evidence=None + sink=None source=None line=None evidence=None + +=== RESULT: baseline flow rows=5 fixed-search flow rows=11 === +FIXED SEARCH TRACES BANKS (search was wrong, not the target)