marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

measure(g2): banks fixed-search MVP -- "no wrong targets, only wrong searching" demonstrated

Browse Source 
Same CPG, only the search changes. Baseline (shipped path spec) resolves 0
prompt->write_text flows on banks; injecting the two annotations an LLM steer
would supply (parameter_sources=True + write_text/read_text path sinks) makes the
engine resolve prompt -> path-join -> prompt_file.write_text(prompt.raw). First
measured instance of "#2 guided by #1" closing on an unseen target; fix needed
exactly the param-source + file-write-sink annotations.

Files the instrument bug the first MVP hit: single-query cpg.run("taint",
request_params(...)) is dead -- request_params emits 5 keys (guards/fixedOrigin/
originAnchors/quotedSanitizers/fieldParameterSources) absent from taint.sc's exec
signature; both single-query sub-routes reject them (silent None). Batch path
(run_batch->requestsFile JSON) unaffected; push/scan route uses batch.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-10-12 14:02:41 +02:00
co-authored by Claude Opus 4.8
parent 7ab5c2ab2d
commit 8f7fc4ae83
6 changed files with 272 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/NOTES.md
+74
View File
@@ -0,0 +1,74 @@
# Banks fixed-search MVP — "no wrong targets, only wrong searching"
Date: 2026-10-12 · Host: clean (local docker, replay-task image) · Repo HEAD: `7ab5c2a`
Target: `masci/banks` @ `5e7093c9b3c2bbe2e0bc281f5a3602bbdabf00b9` (the "library" that gave the static engine ZERO path findings in the engine-on pilot).
## Thesis under test
The governing principle (maintainer, 2026-10-12): we have a **search-with-proof** architecture, so there
are **no wrong targets, only wrong searching**. Banks produced zero path-family findings in the engine-on
pilot not because it is a wrong target but because the SEARCH was not annotated for it:
1. the **source** — `prompt.name`, a function parameter — is not recognised (`entry:false`, so
`_parameters_are_input` returns False and parameters are not treated as untrusted); and
2. the **sink** — `Path.write_text` / `Path.read_text` at a joined path — is not in the shipped
path-family sink list.
Inject exactly those two annotations (what a per-target LLM steer would supply) and check whether the engine
now traces `prompt → path-join → write_text` where the shipped search produced nothing.
## Instrument (verified before believing the number)
Per AGENTS.md "verify the instrument": the first MVP attempt (single-query `cpg.run("taint", …)` path)
returned 0/0, but that was an **instrument failure**, not a result — every query died on
`cpg query taint failed` (see the filed bug `taint-sc-single-query-signature-skew.md`). This record uses the
**batch path** (`cpg.run_batch("taint", {requestsFile JSON})`), the same invocation the real engine-on scans
use. Confirmed it ran: CPG built, `run_batch` available, and the fixed search returned **concrete sink
strings** (`prompt_file.write_text(prompt.raw)`) — the opposite of the v1 failure's empty rows. Harness:
`banks_mvp2.py`. Raw output: `results.out`.
## Setup
Same target, same CPG; only the search changes.
- Functions probed: `from_prompt_path`, `set`, `_load`, `_scan` in `src/banks/registries/directory.py`.
- BASELINE = shipped `path` taint spec, `parameter_sources=False`.
- FIXED = shipped spec + `write_text`/`read_text` added to path sinks, `parameter_sources=True`.
## Result — flow-row counts
| Function | Baseline (shipped) | Fixed search (annotation ON) |
|-------------------|----------------------|---------------------------------------------------------------|
| `from_prompt_path`| 1 row (`sink=None`) | **4 rows** — incl. `sink=prompt_file.write_text(prompt.raw) source=prompt` |
| `set` | 1 row (`sink=None`) | 1 row (`sink=None`) |
| `_load` | 1 row (`sink=None`) | 2 rows (`sink=None`) |
| `_scan` | 2 rows (`sink=None`) | **4 rows** — incl. `sink=self._index_path.write_text(...)` |
| **Total** | **5** | **11** |
### What the counts actually mean (honest read)
The raw totals (5 vs 11) **overstate** the effect. All 5 baseline rows have `sink=None, source=None` — they
are census/evidence rows, **not** resolved source→sink path flows. That is consistent with the engine-on
pilot producing zero path-family findings on banks. The result that carries the thesis is the
**resolved-flow delta**:
- **Baseline: 0 resolved `prompt → write_text` flows.**
- **Fixed search: the engine resolves `prompt → path-join → prompt_file.write_text(prompt.raw)` in
`from_prompt_path`** (`source=prompt`), plus a second `write_text` sink in `_scan`. These appear **only**
with the two injected annotations.
## Verdict
On the exact target that gave the static engine zero, the engine traces the real flow once the search is
corrected. It needed exactly **two** annotations: the parameter-as-source and the missing file-write sink.
This demonstrates **"no wrong targets, only wrong searching"** on a concrete case, and names the precise
inputs the per-target LLM steer (DSPy-optimised, framework-seeded) must learn to supply.
## Caveats (not papered over)
1. One `from_prompt_path` row shows `source=path` rather than `prompt` — the parameter-source annotation
fires broadly, not yet pinned to the untrusted parameter. A precision knob for the steering loop, not a
blocker.
2. This shows the engine *traces* the flow; it is **not** the runtime differential proof. Proving
attacker-reachability remains the async BLOCK lane (search-with-proof verifier).
3. The single-query taint path is broken (filed separately); this record uses the batch path deliberately.
benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/banks_mvp2.py
+45
View File
@@ -0,0 +1,45 @@
"""Banks FIXED-SEARCH MVP v2 (batch path -- the working invocation; the single cpg.run path is broken by a
taint.sc exec-signature skew). Build banks CPG, run the taint BATCH on the vuln functions with the annotation
OFF (shipped) vs ON (parameter_sources=True + write_text/read_text added to path sinks). A non-empty flow row =
the engine traced source->sink. Thesis: same target, only the search changes."""
import dataclasses, json, sys
from pathlib import Path
sys.path.insert(0, "/venv/lib/python3.12/site-packages")
from openultrasast.cpg.backend import resolve_cpg_backend
from openultrasast.model.specs import taint_specs
from openultrasast.model.taint import request_params
ROOT = Path("/work/banks"); VP = "src/banks/registries/directory.py"
FUNCS = ["from_prompt_path", "set", "_load", "_scan"]
backend = resolve_cpg_backend()
cpg = backend.build(ROOT, language="python")
if cpg is None or not callable(getattr(cpg, "run_batch", None)):
print("CPG build failed or no run_batch"); raise SystemExit(1)
print("CPG built; run_batch available.", flush=True)
base = taint_specs(language="python")["path"]
annot = dataclasses.replace(base, sinks=tuple(sorted(set(base.sinks) | {"write_text", "read_text"})))
print("baseline sinks:", base.sinks)
print("annotated sinks:", annot.sinks, flush=True)
def run(label, spec, fn, psrc):
reqs = {"r1": request_params(spec, function=fn, file=VP, parameter_sources=psrc)}
try:
out = cpg.run_batch("taint", reqs)
except Exception as e:
print(f" [{label}] {fn}: ERROR {type(e).__name__}: {str(e)[:150]}"); return 0
rows = (out or {}).get("r1") or []
n = len([r for r in rows if isinstance(r, (dict,)) ])
print(f" [{label}] {fn} psrc={psrc}: {len(rows)} flow row(s)")
for r in rows[:3]:
if isinstance(r, dict):
print(f" sink={r.get('sink')} source={r.get('source')} line={r.get('line')} evidence={str(r.get('evidence'))[:60]}")
return len(rows)
print("\n=== BASELINE (shipped search) ===", flush=True)
b = sum(run("baseline", base, fn, False) for fn in FUNCS)
print("\n=== FIXED SEARCH (annotation: param-source + write_text/read_text sink) ===", flush=True)
f = sum(run("fixed", annot, fn, True) for fn in FUNCS)
print(f"\n=== RESULT: baseline flow rows={b} fixed-search flow rows={f} ===")
print("FIXED SEARCH TRACES BANKS (search was wrong, not the target)" if f > b else "no change", flush=True)
benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/results.out
+32
View File
@@ -0,0 +1,32 @@
CPG built; run_batch available.
baseline sinks: ('FileResponse', 'open', 'os.remove', 'os.unlink', 'send_file', 'send_from_directory', 'shutil.copy', 'shutil.move', 'shutil.rmtree')
annotated sinks: ('FileResponse', 'open', 'os.remove', 'os.unlink', 'read_text', 'send_file', 'send_from_directory', 'shutil.copy', 'shutil.move', 'shutil.rmtree', 'write_text')
=== BASELINE (shipped search) ===
[baseline] from_prompt_path psrc=False: 1 flow row(s)
sink=None source=None line=None evidence=None
[baseline] set psrc=False: 1 flow row(s)
sink=None source=None line=None evidence=None
[baseline] _load psrc=False: 1 flow row(s)
sink=None source=None line=None evidence=None
[baseline] _scan psrc=False: 2 flow row(s)
sink=None source=None line=None evidence=None
sink=None source=None line=None evidence=None
=== FIXED SEARCH (annotation: param-source + write_text/read_text sink) ===
[fixed] from_prompt_path psrc=True: 4 flow row(s)
sink=prompt_file.write_text(prompt.raw) source=path line=None evidence=None
sink=prompt_file.write_text(prompt.raw) source=prompt line=None evidence=None
sink=None source=None line=None evidence=None
[fixed] set psrc=True: 1 flow row(s)
sink=None source=None line=None evidence=None
[fixed] _load psrc=True: 2 flow row(s)
sink=None source=None line=None evidence=None
sink=None source=None line=None evidence=None
[fixed] _scan psrc=True: 4 flow row(s)
sink=self._index_path.write_text(self._index.model_dump_json()) source=self line=None evidence=None
sink=None source=None line=None evidence=None
sink=None source=None line=None evidence=None
=== RESULT: baseline flow rows=5 fixed-search flow rows=11 ===
FIXED SEARCH TRACES BANKS (search was wrong, not the target)