feat(detection): language-scoped rules + multi-language benchmark corpora

Rebuild the quick-scan detection engine and ground-truth benchmarks to hit >=90% recall / <10% false positives for C/C++, Python, JavaScript, and Java. Engine (findings.py, preprocess.py): - Scope each PatternRule by language (fixes cross-language false positives such as the Python eval/exec rule firing on JS .exec()). - Add JavaScript/TypeScript rules (command exec, eval/Function, reflected XSS, SQLi, path traversal, SSRF, weak hash, deserialization); execFile/spawn-argv excluded as safe alternatives. - Add Java rules (Runtime exec, SQL string concatenation, weak hash, readObject) and a Groovy server-side template rule (yieldUnescaped of a bound variable). - Broaden C/C++ and Python coverage; precision guards for format strings and parameterised vs concatenated SQL; #-comment skipping; CWE in rationale. - Register .java/.groovy/.tpl source extensions. Benchmarks: - Add realistic corpora modeled on Damn Vulnerable C/C++, PyGoat/DVPWA, and NodeGoat, plus the real kiview/damn-vulnerable-spring-boot-app vendored verbatim (MIT, attribution in LICENSE/NOTICE.md). Each manifest carries complete, independently-labeled ground truth including genuinely hard-to-detect vulns (integer overflow, second-order SQLi, prototype pollution) so recall is measured honestly, plus safe-API files for precision. - Fix pre-existing smoke-manifest ground-truth errors (Python/JS line offsets, missing C/C++ strcpy entry, Java SQLi line). benchmark.py: - Fix sink matching to normalise the expected sink consistently so sinks with underscores (e.g. render_template_string) can match. Tests: - Add tests/test_detection_benchmarks.py enforcing the >=90% recall / <10% FP goal per language and overall. Measured: C/C++ 93.3%, Python 93.3%, JavaScript 92.3%, Java 100%; overall 93.6% recall, 0.0% false-positive rate. Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-17 03:59:28 +00:00 · 2026-06-21 11:49:31 +02:00
parent be65aed96e
commit cf9f945992
38 changed files with 2420 additions and 15 deletions
@@ -0,0 +1,10 @@
+#include <cstdlib>
+#include <cstring>
+
+void copy_user(char *dst, char *src) { memcpy(dst, src, 128); }
+
+void run_user(char *arg) {
+  char cmd[256];
+  std::strcpy(cmd, arg);
+  system(cmd);
+}
@@ -0,0 +1,38 @@
+// Memory-safety vulnerabilities modeled on Damn Vulnerable C/C++ Program.
+#include <cstdio>
+#include <cstdlib>
+#include <cstring>
+
+void copy_name(char *src) {
+  char dst[16];
+  strcpy(dst, src); // CWE-120 unbounded strcpy into fixed buffer
+}
+
+void greet(char *src) {
+  char dst[16];
+  strcat(dst, src); // CWE-120 unbounded strcat into fixed buffer
+}
+
+void read_line() {
+  char buf[32];
+  gets(buf); // CWE-120 gets has no bounds checking
+}
+
+void render(char *user) {
+  char out[64];
+  sprintf(out, "hello %s", user); // CWE-120 sprintf overflow
+}
+
+void copy_blob(char *dst, char *src, int len) {
+  memcpy(dst, src, len); // CWE-120 memcpy with attacker length
+}
+
+void parse_int() {
+  char name[8];
+  scanf("%s", name); // CWE-120 unbounded scanf %s
+}
+
+char *grow(int len, int count) {
+  // CWE-190 integer overflow in allocation size (no direct dangerous sink token)
+  return (char *)malloc(len * count);
+}
@@ -0,0 +1,23 @@
+// Command-execution and format-string vulnerabilities.
+#include <cstdio>
+#include <cstdlib>
+#include <unistd.h>
+
+void run_cmd(char *arg) {
+  char cmd[256];
+  strcpy(cmd, arg);
+  system(cmd); // CWE-78 system runs attacker-controlled command
+}
+
+void read_pipe(char *arg) {
+  FILE *fp = popen(arg, "r"); // CWE-78 popen runs attacker-controlled command
+  if (fp) pclose(fp);
+}
+
+void launch(char *arg) {
+  execlp("sh", "sh", "-c", arg, (char *)NULL); // CWE-78 execlp runs attacker input
+}
+
+void log_msg(char *user) {
+  printf(user); // CWE-134 format string controlled by attacker
+}
@@ -0,0 +1,23 @@
+// Safe equivalents. None of these lines should produce a finding.
+#include <cstdio>
+#include <cstdlib>
+#include <cstring>
+
+void copy_name_safe(const char *src) {
+  char dst[16];
+  strncpy(dst, src, sizeof(dst) - 1); // bounded copy, not flagged
+  dst[sizeof(dst) - 1] = '\0';
+}
+
+void render_safe(const char *user) {
+  char out[64];
+  snprintf(out, sizeof(out), "hello %s", user); // bounded, not flagged
+}
+
+void log_safe(const char *user) {
+  printf("%s", user); // constant format string, not a format-string bug
+}
+
+size_t length_safe(const char *src) {
+  return strnlen(src, 64); // bounded length, no dangerous sink
+}
@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2017 GDATA Advanced Analytics GmbH
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
@@ -0,0 +1 @@
+Vendored verbatim from https://github.com/kiview/damn-vulnerable-spring-boot-app (MIT, (c) 2017 GDATA Advanced Analytics GmbH) as a benchmark corpus for OpenUltraSAST. Source files unmodified; line numbers match upstream.
@@ -0,0 +1,13 @@
+package com.groovycoder.dvsba;
+
+import org.springframework.boot.SpringApplication;
+import org.springframework.boot.autoconfigure.SpringBootApplication;
+
+@SpringBootApplication
+public class DvsbaApplication {
+
+    public static void main(String[] args) {
+        SpringApplication.run(DvsbaApplication.class, args);
+    }
+
+}
@@ -0,0 +1,14 @@
+package com.groovycoder.dvsba;
+
+import org.springframework.stereotype.Controller;
+import org.springframework.web.bind.annotation.GetMapping;
+import org.springframework.web.servlet.ModelAndView;
+
+@Controller
+public class HomeController {
+
+    @GetMapping("/")
+    public ModelAndView home() {
+        return new ModelAndView("views/home");
+    }
+}
@@ -0,0 +1,13 @@
+package com.groovycoder.dvsba.books;
+
+public class Book {
+    Long id;
+    String name;
+    String author;
+
+    public Book(Long id, String name, String author) {
+        this.id = id;
+        this.name = name;
+        this.author = author;
+    }
+}
@@ -0,0 +1,88 @@
+package com.groovycoder.dvsba.books;
+
+import org.springframework.jdbc.core.JdbcTemplate;
+import org.springframework.jdbc.core.ResultSetExtractor;
+import org.springframework.stereotype.Controller;
+import org.springframework.web.bind.annotation.RequestMapping;
+import org.springframework.web.bind.annotation.RequestParam;
+import org.springframework.web.servlet.ModelAndView;
+
+import javax.annotation.PostConstruct;
+import java.util.*;
+
+@RequestMapping("/books")
+@Controller
+class BookController {
+
+    private final JdbcTemplate jdbcTemplate;
+
+    public BookController(JdbcTemplate jdbcTemplate) {
+        this.jdbcTemplate = jdbcTemplate;
+    }
+
+    @RequestMapping("/")
+    public ModelAndView home() {
+        List<Book> books = loadBooks();
+
+        Map<String, Object> model = new HashMap<>();
+        model.put("books", books);
+
+        return new ModelAndView("views/sql/home", model);
+    }
+
+    @RequestMapping("/detail")
+    public ModelAndView detail(@RequestParam(value = "id") String id) {
+        String sql = "SELECT * FROM books WHERE id=" + id;
+        final Book[] book = new Book[1];
+        jdbcTemplate.query(sql, (ResultSetExtractor) rs -> {
+            if (rs.next())
+                book[0] = new Book(rs.getLong(1), rs.getString(2), rs.getString(3));
+
+            return null;
+
+        });
+
+        Map<String, Object> model = new HashMap<>();
+        model.put("book", book[0]);
+
+        return new ModelAndView("views/sql/detail", model);
+    }
+
+    @PostConstruct
+    private void bootstrap() {
+        initDb();
+
+        List<Book> books = Arrays.asList(
+                new Book(1L, "Moby Dick", "Herman Melville"),
+                new Book(2L, "Unsichtbare Spuren", "Andreas Franz"),
+                new Book(3L, "Das Paket", "Sebastian Fitzek")
+        );
+        saveBooks(books);
+    }
+
+    private void saveBooks(List<Book> books) {
+        books.forEach(book ->
+                jdbcTemplate.update("INSERT INTO books (id, `name`, author) values (?, ?, ?)",
+                        book.id, book.name, book.author)
+        );
+    }
+
+    private List<Book> loadBooks() {
+
+        return jdbcTemplate.query("SELECT * FROM books", rs -> {
+            List<Book> books = new LinkedList<>();
+
+            while (rs.next()) {
+                books.add(new Book(rs.getLong(1), rs.getString(2), rs.getString(3)));
+            }
+
+            return books;
+        });
+
+    }
+
+    private void initDb() {
+        jdbcTemplate.execute("CREATE TABLE books (id NUMBER, name VARCHAR(255), author VARCHAR(255))");
+    }
+
+}
@@ -0,0 +1,47 @@
+package com.groovycoder.dvsba.comments;
+
+import org.springframework.stereotype.Controller;
+import org.springframework.web.bind.annotation.GetMapping;
+import org.springframework.web.bind.annotation.PostMapping;
+import org.springframework.web.bind.annotation.RequestMapping;
+import org.springframework.web.bind.annotation.RequestParam;
+import org.springframework.web.servlet.ModelAndView;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+@Controller
+@RequestMapping("/comments")
+public class CommentController {
+
+    private List<String> comments = new ArrayList<>(Arrays.asList(
+            "Hi super Webseite leider keine Mobile App deswegen nur einen Stern.",
+            "Ich muss darauf hinweisen, dass eine Webseite ohne Impressum zu betreiben !!!strafbar!!! ist!!!",
+            "Hallo, wir sinds! Wolltest du nicht die Fotos von Samstag hochladen? Grüße!"));
+
+    @GetMapping(path = "/")
+    public ModelAndView listComments() {
+        Map<String, Object> model = new HashMap<>();
+        model.put("comments", comments);
+
+        return new ModelAndView(
+                "views/comment/comment",
+                model);
+    }
+
+    @PostMapping("/")
+    public ModelAndView postComment(@RequestParam String input) {
+        comments.add(input);
+
+        Map<String, Object> model = new HashMap<>();
+        model.put("comments", comments);
+
+        return new ModelAndView(
+                "views/comment/comment",
+                model);
+    }
+
+}
@@ -0,0 +1,25 @@
+yieldUnescaped '<!DOCTYPE html>'
+html {
+    head {
+        title('Spring Boot - Groovy templates example')
+        link(rel: 'stylesheet', href: '/css/bootstrap.min.css')
+    }
+    body {
+        form(action: "/comments/", method: "POST") {
+            input(name: "input") {
+
+            }
+        }
+
+        comments.each { comment ->
+            div {
+                h3(style: "border-bottom: 1px solid black;") {
+                    yield "Comment by Anonymous"
+                }
+                div(style: "border-bottom: 1px solid black;") {
+                    yieldUnescaped comment
+                }
+            }
+        }
+    }
+}
@@ -0,0 +1,13 @@
+yieldUnescaped '<!DOCTYPE html>'
+html {
+    head {
+        title('Damn Vulnerable Spring Boot App - Home')
+    }
+    body {
+        h1 {
+            yield 'Hello and Welcome to the DVSBA!'
+        }
+        p ("Take a look around! We have a ${$a(href:'/books/', "catalouge")} of our favourite books which you can browse. If you like it, leave a comment in the ${ $a(href: '/comments/', "comment section")}!")
+
+    }
+}
@@ -0,0 +1,16 @@
+yieldUnescaped '<!DOCTYPE html>'
+html {
+    head {
+        title('Damn Vulnerable Spring Boot App')
+    }
+    body {
+        div(class: 'container') {
+            a(class: 'brand',
+                href: '/books/',
+                'List')
+            br()
+            br()
+            p "ID: ${book.id}, Name: ${book.name}, Author: ${book.author}"
+        }
+    }
+}
@@ -0,0 +1,38 @@
+yieldUnescaped '<!DOCTYPE html>'
+html {
+    head {
+        title('Damn Vulnerable Spring Boot App')
+    }
+    body {
+        div(class: 'container') {
+            form(method: "GET", action: "/books/detail") {
+                input(name: "id", placeholder: "Search for a book id!")
+            }
+            br
+            table(border:1) {
+                thead {
+                    tr {
+                        td 'ID'
+                        td 'Name'
+                        td 'Author'
+                      }
+                    }
+                tbody {
+                    books.each { book ->
+                        tr {
+                            td {
+                                a(href: "/books/detail?id=${book.id}", "${book.id}")
+                            }
+                            td {
+                                a(href: "/books/detail?id=${book.id}", "${book.name}")
+                            }
+                            td {
+                                a(href: "/books/detail?id=${book.id}", "${book.author}")
+                            }
+                        }
+                    }
+                }
+            }
+        }
+    }
+}
@@ -0,0 +1,15 @@
+package example;
+
+import jakarta.servlet.http.HttpServletRequest;
+import java.sql.Connection;
+
+class VulnerableServlet {
+  void ping(HttpServletRequest request) throws Exception {
+    Runtime.getRuntime().exec("ping -c 1 " + request.getParameter("host"));
+  }
+
+  void lookup(HttpServletRequest request, Connection connection) throws Exception {
+    String sql = "select * from users where name = '" + request.getParameter("name") + "'";
+    connection.createStatement().executeQuery(sql);
+  }
+}
@@ -0,0 +1,13 @@
+const express = require("express");
+const childProcess = require("child_process");
+const app = express();
+
+app.get("/ping", (req, res) => {
+  childProcess.exec("ping -c 1 " + req.query.host, (error, stdout) => res.send(stdout));
+});
+
+app.get("/hello", (req, res) => {
+  res.send("<h1>Hello " + req.query.name + "</h1>");
+});
+
+module.exports = app;
@@ -0,0 +1,59 @@
+// Vulnerable Express app modeled on NodeGoat / vulnerable JavaScript samples.
+const express = require("express");
+const childProcess = require("child_process");
+const fs = require("fs");
+const crypto = require("crypto");
+const axios = require("axios");
+const app = express();
+
+app.get("/ping", (req, res) => {
+  childProcess.exec("ping -c 1 " + req.query.host, (e, out) => res.send(out)); // CWE-78 command injection
+});
+
+app.get("/build", (req, res) => {
+  childProcess.execSync("make " + req.query.target); // CWE-78 command injection via execSync
+});
+
+app.get("/calc", (req, res) => {
+  const result = eval(req.query.expr); // CWE-95 eval of user input
+  res.send(String(result));
+});
+
+app.get("/fn", (req, res) => {
+  const f = new Function("x", req.query.body); // CWE-95 Function constructor injection
+  res.send(String(f(1)));
+});
+
+app.get("/hello", (req, res) => {
+  res.send("<h1>Hello " + req.query.name + "</h1>"); // CWE-79 reflected XSS
+});
+
+app.get("/user", (req, res) => {
+  db.query("select * from users where name = '" + req.query.name + "'"); // CWE-89 SQL injection
+});
+
+app.get("/report", (req, res) => {
+  db.query(`select * from reports where id = ${req.query.id}`); // CWE-89 SQL injection template literal
+});
+
+app.get("/read", (req, res) => {
+  const data = fs.readFileSync(req.query.path); // CWE-22 path traversal
+  res.send(data);
+});
+
+app.get("/fetch", (req, res) => {
+  axios.get(req.query.url).then((r) => res.send(r.data)); // CWE-918 SSRF
+});
+
+app.get("/hash", (req, res) => {
+  const digest = crypto.createHash("md5").update(req.query.pw).digest("hex"); // CWE-327 weak hash
+  res.send(digest);
+});
+
+app.get("/merge", (req, res) => {
+  const target = {};
+  for (const k in req.query) target[k] = req.query[k]; // CWE-1321 prototype pollution (regex blind spot)
+  res.json(target);
+});
+
+module.exports = app;
@@ -0,0 +1,24 @@
+// Safe equivalents. None of these lines should produce a finding.
+const { execFile } = require("child_process");
+const crypto = require("crypto");
+
+function pingSafe(host, res) {
+  // execFile with an argument array does not invoke a shell with concatenation
+  execFile("ping", ["-c", "1", host], (e, out) => res.send(out));
+}
+
+function userSafe(db, req) {
+  // parameterised query, no concatenation or template interpolation
+  db.query("select * from users where name = ?", [req.query.name]);
+}
+
+function helloSafe(req, res) {
+  res.send("<h1>Hello world</h1>"); // constant response, no reflection
+}
+
+function hashSafe(pw) {
+  return crypto.createHash("sha256").update(pw).digest("hex"); // strong hash
+}
+
+// eval(req.query.expr) would be dangerous but this is a comment
+module.exports = { pingSafe, userSafe, helloSafe, hashSafe };
@@ -0,0 +1,81 @@
+"""Vulnerable Flask app modeled on PyGoat / DVPWA."""
+import hashlib
+import os
+import pickle
+import subprocess
+
+import requests
+import yaml
+from flask import Flask, render_template_string, request
+
+app = Flask(__name__)
+
+
+@app.get("/ping")
+def ping():
+    host = request.args["host"]
+    return os.system("ping -c 1 " + host)  # CWE-78 os.system command injection
+
+
+@app.get("/run")
+def run():
+    cmd = request.args["cmd"]
+    return subprocess.check_output(cmd, shell=True)  # CWE-78 subprocess shell=True
+
+
+@app.get("/calc")
+def calc():
+    return str(eval(request.args["expr"]))  # CWE-95 eval of user input
+
+
+@app.get("/load")
+def load():
+    return pickle.loads(request.data)  # CWE-502 pickle deserialization of request body
+
+
+@app.get("/config")
+def config():
+    return yaml.load(request.data)  # CWE-502 yaml.load of untrusted input
+
+
+@app.get("/user")
+def user(db):
+    name = request.args["name"]
+    return db.execute("select * from users where name = '" + name + "'")  # CWE-89 SQL injection
+
+
+@app.get("/report")
+def report(db):
+    uid = request.args["id"]
+    return db.execute("select * from reports where id = %s" % uid)  # CWE-89 SQL injection via %
+
+
+@app.get("/read")
+def read():
+    return open(request.args["path"]).read()  # CWE-22 path traversal from request
+
+
+@app.get("/fetch")
+def fetch():
+    return requests.get(request.args["url"]).text  # CWE-918 SSRF from request url
+
+
+@app.get("/token")
+def token():
+    return hashlib.md5(request.args["pw"].encode()).hexdigest()  # CWE-327 weak hash
+
+
+@app.get("/render")
+def render():
+    return render_template_string("Hello " + request.args["name"])  # CWE-94 SSTI
+
+
+@app.get("/search")
+def search(db):
+    term = request.args["q"]
+    query = "select * from items where title like '%" + term + "%'"
+    return db.execute(query)  # CWE-89 second-order: query built on prior line (regex blind spot)
+
+
+if __name__ == "__main__":
+    app.run(debug=True)  # CWE-489 debug mode in production
@@ -0,0 +1,30 @@
+"""Safe equivalents. None of these lines should produce a finding."""
+import hashlib
+import subprocess
+
+import yaml
+from flask import request
+
+
+def ping_safe(host):
+    # list-form subprocess without shell is not command injection
+    return subprocess.check_output(["ping", "-c", "1", host])
+
+
+def user_safe(db):
+    name = request.args["name"]
+    # parameterised query: %s here is a bound placeholder, not %-formatting
+    return db.execute("select * from users where name = %s", (name,))
+
+
+def config_safe(data):
+    return yaml.safe_load(data)  # safe_load is not the unsafe yaml.load sink
+
+
+def token_safe(pw):
+    return hashlib.sha256(pw.encode()).hexdigest()  # strong hash
+
+
+def comment_only():
+    # os.system("rm -rf /") would be dangerous but this is a comment
+    return "ok"
@@ -0,0 +1,13 @@
+import subprocess
+
+from flask import Flask, request
+
+app = Flask(__name__)
+
+@app.get("/ping")
+def ping():
+    return subprocess.check_output("ping -c 1 " + request.args["host"], shell=True)
+
+@app.get("/calc")
+def calc():
+    return str(eval(request.args["expr"]))
				`@@ -0,0 +1 @@`
				`Vendored verbatim from https://github.com/kiview/damn-vulnerable-spring-boot-app (MIT, (c) 2017 GDATA Advanced Analytics GmbH) as a benchmark corpus for OpenUltraSAST. Source files unmodified; line numbers match upstream.`