Marius CiepluchandClaude Opus 4.8
04667c815e
Ran ousast scan --mode deep on the 5 app-shaped + sound pool targets (the fair change-scoped eval set from the shape classification). Detection 3/5: graphiti (hunter, Cypher injection), mcp-remote (hunter, url-scheme injection), dcmcp (ENGINE overlay -- 'outbound.request reached from parameter' on readFileFromUrl; parameter-as-source fired because an entry point is present). 2/5 missed: malla (DOM/template XSS) and dms (Jinja SSTI), both in families families.toml documents as limited. This validates the strategic pivot: the near-zero catch was substantially a TARGET MISMATCH -- on app-shaped repos the engine itself detects (0 -> 3/5). But all hits stop at suspicion/ static_corroboration; release-grade BLOCK still needs proof-arm elevation + family coverage + the DSPy hint loop. Cost ~$0.007/repo. So bet #1 (fix eval target) is necessary and material but not sufficient. Next: recompute a floor on an app-shaped sound subset / real app corpus (VAmPI, NodeGoat, DjanGoat, vibe-py); measure proof-arm suspicion->proven lift. Co-Authored-By: Claude Opus 4.8 <[email protected]>
2.8 KiB
Deep scan (regex/engine/hunter) on the 5 app-shaped + sound "fair" targets — 2026-10-12
Why
Strategic pivot "fix the evaluation target first". The shape classification left 5 pool targets that are BOTH
label-sound AND application-shaped (reachability in-repo) — the fair change-scoped eval set. Ran ousast scan --mode deep (DeepSeek, model layer + tool_hunter, 60-call bound) on all 5 to get a per-tier catch rate on the
corrected target set (vs ~0 on the library-heavy pilot). graphiti from the earlier run; the other 4 here.
Result: detection 3/5 on fair targets (vs ~0 library-heavy), all at suspicion rung
| target | family | detected | by | rung | model calls / cost |
|---|---|---|---|---|---|
| getzep/graphiti | injection (Cypher) | YES | hunter (family:cypher-injection, build_fulltext_query L278) | suspicion | 64 / $0.0071 |
| punkpeye/mcp-remote | injection (url-scheme) | YES | hunter (family:url-scheme-injection, sanitizeUrl L708) | suspicion | 65 / $0.0075 |
| sorlen008/desktopcommandermcp | untrusted_destination (SSRF) | YES | engine overlay ("outbound.request reached from parameter", readFileFromUrl L174) | suspicion | 60 / $0.0067 |
| zenitram/malla | output_encoding (DOM XSS) | no | — (no finding on the vuln JS file) | — | 65 / $0.0072 |
| adfinis/document-merge-service | injection (Jinja SSTI) | no | — (no finding on jinja.py get_jinja_env) | — | 64 / $0.0075 |
Reads
- The target mismatch was real and material: detection 0 → 3/5 when the target is app-shaped + sound. On dcmcp (an MCP server) the ENGINE's parameter-as-source mode fired (entry point present) and flagged the SSRF — the exact thing it could NOT do on library code. On graphiti/mcpremote the hunter caught injection with a family tag. The near-zero pilot catch was substantially a target artifact.
- The 2 misses are in families the tool DOCUMENTS as limited — malla is DOM/template XSS (output_encoding's
limitsin families.toml: cannot decide template-mediated escaping) and dms is Jinja SSTI/autoescape-config. Expected coverage gaps, not reachability failures. - Everything still stops at
suspicion/static_corroboration— none reached corroborated/entailed/ execution_confirmed. Release-grade BLOCK needs the proof arm to elevate these. - Cost remains a non-issue: ~$0.007/repo, ~60-65 model calls for a whole-repo deep scan.
Conclusion for the strategy
Fixing the evaluation target (bet #1) is validated as necessary and material but NOT sufficient: on fair targets detection appears (3/5) but stays at suspicion. The remaining work is (a) proof-arm elevation (suspicion→proven) for BLOCK, and (b) family coverage for the documented-limited families — and the DSPy hint loop for the misses. This is also the honest basis to recompute record-07 on a sound+app-shaped subset / a real app corpus (VAmPI, NodeGoat, DjanGoat, vibe-py).