feat(detection): language-scoped rules + multi-language benchmark corpora

Rebuild the quick-scan detection engine and ground-truth benchmarks to hit
>=90% recall / <10% false positives for C/C++, Python, JavaScript, and Java.

Engine (findings.py, preprocess.py):
- Scope each PatternRule by language (fixes cross-language false positives such
  as the Python eval/exec rule firing on JS .exec()).
- Add JavaScript/TypeScript rules (command exec, eval/Function, reflected XSS,
  SQLi, path traversal, SSRF, weak hash, deserialization); execFile/spawn-argv
  excluded as safe alternatives.
- Add Java rules (Runtime exec, SQL string concatenation, weak hash, readObject)
  and a Groovy server-side template rule (yieldUnescaped of a bound variable).
- Broaden C/C++ and Python coverage; precision guards for format strings and
  parameterised vs concatenated SQL; #-comment skipping; CWE in rationale.
- Register .java/.groovy/.tpl source extensions.

Benchmarks:
- Add realistic corpora modeled on Damn Vulnerable C/C++, PyGoat/DVPWA, and
  NodeGoat, plus the real kiview/damn-vulnerable-spring-boot-app vendored
  verbatim (MIT, attribution in LICENSE/NOTICE.md). Each manifest carries
  complete, independently-labeled ground truth including genuinely
  hard-to-detect vulns (integer overflow, second-order SQLi, prototype
  pollution) so recall is measured honestly, plus safe-API files for precision.
- Fix pre-existing smoke-manifest ground-truth errors (Python/JS line offsets,
  missing C/C++ strcpy entry, Java SQLi line).

benchmark.py:
- Fix sink matching to normalise the expected sink consistently so sinks with
  underscores (e.g. render_template_string) can match.

Tests:
- Add tests/test_detection_benchmarks.py enforcing the >=90% recall / <10% FP
  goal per language and overall.

Measured: C/C++ 93.3%, Python 93.3%, JavaScript 92.3%, Java 100%;
overall 93.6% recall, 0.0% false-positive rate.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-06-21 11:49:31 +02:00
co-authored by Claude Opus 4.8
parent be65aed96e
commit cf9f945992
38 changed files with 2420 additions and 15 deletions
+21
View File
@@ -23,6 +23,27 @@ def test_quick_scan_findings_emit_static_corroboration(tmp_path: Path) -> None:
assert findings[0].line == 1
def test_quick_scan_findings_emit_common_cpp_hazards(tmp_path: Path) -> None:
repo = tmp_path / "repo"
repo.mkdir()
(repo / "vuln.cpp").write_text(
"#include <cstring>\n"
"#include <cstdlib>\n"
"// system(cmd) in a comment is not executable evidence.\n"
"void copy(char *src) { char dst[4]; memcpy(dst, src, 16); }\n"
"void run(char *cmd) { system(cmd); }\n"
"void fmt(char *format, va_list args) { char dst[8]; vsprintf(dst, format, args); }\n"
)
_, targets = preprocess_repository(repo)
finding_ids = {finding.finding_id for finding in quick_scan_findings(repo, targets, rank_targets(targets))}
assert "c-shell-exec:vuln.cpp:3" not in finding_ids
assert "c-unsafe-memory-copy:vuln.cpp:4" in finding_ids
assert "c-shell-exec:vuln.cpp:5" in finding_ids
assert "c-unsafe-format:vuln.cpp:6" in finding_ids
def test_findings_and_report_artifacts(tmp_path: Path) -> None:
repo = tmp_path / "repo"
repo.mkdir()