mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
Marius CiepluchandClaude Opus 4.8
27a7c12ca1
ousast scan --mode deep (DeepSeek deepseek-flash, model layer + tool_hunter, 60-call bound) on bedrock + graphiti, the two sound pilot targets the static engine missed: - graphiti: HIT -- falkordb_driver.py:207, tag family:cypher-injection; rationale traces build_fulltext_query concatenating user input into a Cypher string at line 278 (inside the labelled function, location-matches). The static engine produced zero findings here. - bedrock: miss -- 12 suspicions on adjacent concerns (network-PUBLIC, download_file), never named the install_packages -> pip command injection. Cost is a non-issue: ~$0.007/repo, 64 model calls for a whole-repo deep scan. BUT everything stopped at suspicion rung -- the proof arm did not elevate to proven; release-grade BLOCK needs proven findings. Direction (maintainer): prompt evolution is a core lever -- a DSPy-based loop (GEPA/MIPRO, pluggable; evolving field) whose objective is to learn what hints/annotations steer the model to the real sink, sourced from the framework knowledge already shipped (no new user annotation), with LLM-proposed boundaries graduating into the facts. Next: measure proof-arm lift on graphiti; stand up the DSPy hint loop + re-run bedrock; widen the hunter to the 23 sound targets for a catch rate; then a change-scoped + proven run. Co-Authored-By: Claude Opus 4.8 <[email protected]>
3.4 KiB
3.4 KiB
Model hunter on the two sound pilot targets — 2026-10-12
Why
The engine-on pilot showed the static tiers (regex + Joern dataflow) catch 0/5, and the LLM adjudication tier was
disabled in the push path. Of the 5 pilot repos, the label audit found only bedrock + graphiti (injection) are
sound (change-located, change-introduced) fair targets — and the static engine missed both. We ran the shipped
model hunter (ousast scan --mode deep, DeepSeek deepseek-flash, model layer + tool_hunter enabled,
unbounded by a push deadline, bounded to 60 model calls / 8 hotspots) on those two to test whether the LLM arm
catches what the static tiers miss.
Result: hunter 1/2 (static engine 0/2), cheap, but both at suspicion rung
| repo | vuln | static engine | hunter | model calls | cost |
|---|---|---|---|---|---|
| getzep/graphiti | Cypher injection in build_fulltext_query |
miss | HIT (suspicion) | 64 | $0.0071 |
| aws/bedrock-agentcore | command injection in install_packages |
miss | miss (suspicions on adjacent) | 64 | $0.0076 |
- graphiti — correct catch.
falkordb_driver.py:207finding,tags:[family:cypher-injection], severity medium, confidence low, rung suspicion. Rationale traces it exactly: "build_fulltext_query concatenates the sanitized user query into a Cypher string: full_query = group_filter + ' (' + sanitized_query + ')' (line 278)". Line 278 is inside the labelledfunction_lines [249,280]; head_lines include [206],[249-251],[280] → location-matches. The static engine produced zero findings here. - bedrock — miss. 12 findings, all suspicion; the 3 on the vuln file were adjacent concerns (network-PUBLIC
default @114, caller client_token @188, download_file @583) — it never named the
install_packages→pip installcommand injection. A prompting/targeting miss, not cost (64 calls, $0.0076) or capability.
Reads
- The LLM arm lifts catch over the static tiers — graphiti is the first unseen vuln caught that regex+dataflow missed. Cost is a non-issue (~$0.007/repo, 64 calls for a whole-repo deep scan).
- Everything stops at
suspicion. Neither reached corroborated/entailed/execution_confirmed — the proof arm (sandbox verification) did not elevate them. For release-grade BLOCK the findings must be proven, not just suspected. The proof arm + verification are the next depth to measure. - bedrock motivates prompt evolution. The hunter has the capability but its current prompts did not steer it to
the real sink. Directive: a DSPy-based prompt-optimisation loop (GEPA/MIPRO; keep pluggable) whose objective
is to learn what hints/annotations the model needs (bedrock: "install_packages runs pip with caller-supplied
names → command injection") — sourced from the framework knowledge already shipped (
ruleset/semantic/*.toml,frameworks.toml), no new user annotation; LLM-proposed boundaries graduate into the facts (#2 feedback loop).
Caveats
- Whole-repo deep scan (not change-scoped), so this measures detection CAPABILITY, not pre-push latency. A change-scoped + proven run is the release-relevant shape.
- Only 2 sound targets here; widen to the 23 sound public targets next for a rate.
- graphiti's finding is suspicion/low-confidence; a human/proof confirmation is still needed before it is a
release-grade catch. Findings JSON:
bedrock-findings.json,graphiti-findings.json.