diff --git a/benchmarks/measurements/2026-10-13-semgrep-arm/NOTES.md b/benchmarks/measurements/2026-10-13-semgrep-arm/NOTES.md new file mode 100644 index 0000000..50cee39 --- /dev/null +++ b/benchmarks/measurements/2026-10-13-semgrep-arm/NOTES.md @@ -0,0 +1,160 @@ +# Semgrep CE as a third arm on the valid pilot (2026-10-13) + +The question from `../2026-10-13-semgrep-brief/BRIEF.md` (option C): does Semgrep, running our own +rules and facts, catch more on the valid pilot than our engine does? It does not. It caught none of the +12 positives and raised false alarms on 12.6 of every 100 ordinary pushes. + +Instrument: semgrep 1.179.0 CE (linuxbrew), `-j 1`, `--timeout 60 --max-memory 1500`, offline flags. +Rules: `rules/` here (81 rules, from `python -m benchmarks.valid.semgrep_arm rules`). Records: +`~/ousast-results/valid/pilot/semgrep/` (252 records, outside the repo). Same frozen manifest +(`benchmarks/valid/pilot/pool-valid.toml`, slice 1) and same scorer as the other arms. + +## Headline + +| Arm | Recall (12 positives) | False alarms per 100 pushes (Wilson 95%) | Precision | +|---|---|---|---| +| (a) hook as shipped, 163 done | 0/12, 0.0% (0.0-24.3) | 0.7 (0.1-3.7), n=149 | 0.0% (0/1) | +| (b) hunter, raw | 6/12, 50.0% (25.4-74.6) | 92.5 (88.4-95.2), n=239 | 2.6% | +| (b') hunter, classifier-filtered | 5/12, 41.7% (19.3-68.0) | 54.4 | 3.6% | +| **(c) Semgrep, our rules + facts** | **0/12, 0.0% (0.0-24.3)** | **12.6 (8.9-17.4), n=239**; cluster 3.8-23.4, ICC 0.28 | **0.0% (0/33)** | + +`metrics_semgrep.json`: "recall 0.0% (Wilson 0.0-24.3), false alarms 12.6 per 100 pushes, precision +0.0%; exploit-verified 0 of 5 (5 not yet proved)". Exploit-verified is not applicable: the 5 flagged +positives are all wrong-family hits, none was a catch. + +The (c) false-alarm rate is not like-for-like with (a). The hook keeps a quick hit only on a changed +line or inside a changed function (`push/quick.py:122-130`); this arm counts every result anywhere in a +changed file, as the run instructions asked. Restricting the same records afterwards to results on a +changed line gives 5/239 = 2.1 per 100 (Wilson 0.9-4.8); within 10 lines of a change, 9/239 = 3.8 +(2.0-7.0). Scoped either way it is still a higher rate than (a), and recall stays at zero. + +## Per positive + +| Repo | Family | (a) hook | (b) raw | (c) Semgrep | What Semgrep reported | +|---|---|---|---|---|---| +| bytebase/dbhub | access_control | miss | miss | miss | `js-command-exec` at `sql-parser.ts:87`: `dollarQuoteOpenRegex.exec(...)`, a RegExp call | +| contentful/contentful-mcp-server | untrusted_destination | miss | miss | miss | nothing | +| fastapi-users/fastapi-users | access_control | miss | miss | miss | nothing | +| i18next/i18next-http-middleware | output_encoding | miss | hit | miss | nothing | +| imzbf/md-editor-v3 | output_encoding | miss | hit | miss | `js-command-exec` at `code/index.ts:37`: `re.exec(info)`, a RegExp call | +| jupyter-lsp/jupyterlab-lsp | path | miss | hit | miss | nothing | +| matrix-org/matrix-appservice-bridge | access_control | miss | miss | miss | nothing | +| nvbn/thefuck | path | miss | hit | miss | nothing | +| ondata/ckan-mcp-server | untrusted_destination | miss | hit | miss | nothing | +| webpack/webpack-bundle-analyzer | output_encoding | miss | miss | miss | nothing | +| zenitram/malla | output_encoding | miss | hit | miss | nothing | +| zereight/gitlab-mcp | path | miss | miss | miss | `js-command-exec` + taint `sink:javascript:exec` at `index.ts:12429`: `/^Bearer.../i.exec(authHeader)` | + +No rule origin caught anything. Why the positives are silent: three are access_control, which only the +obligations express, and none of those convert. The four output_encoding positives are a value reaching +`innerHTML` from a function argument or a markdown renderer, and the taint rules' sources are request +objects (`req.query`, `request.args`). The intra-function CE taint never sees an argument as a source. +The path and destination positives take their input from tool arguments or archive members, which are +also not modelled sources. Our hook misses these 12 for the same reason. Swapping the engine does not +change the vocabulary. + +## False alarms (239 ordinary pushes, 30 flagged) + +Pushes flagged, by rule origin: `js-command-exec` 16, `python-sql-injection` 8, `sink:javascript:exec` +8, `python-unsafe-eval` 3, `sink:javascript:html.response` 2, `python-os-command` 1, +`sink:javascript:prototype.merge` 1. The results are 95 in total: 90 injection, 3 prototype, 2 +output_encoding. Only 18 carry a taint trace; the other 77 are regex. + +The flags come from five repos: malla 10, gitlab-mcp 10, webpack-bundle-analyzer 4, dbhub 3, +matrix-appservice-bridge 2, thefuck 1. An ICC of 0.28 means a handful of files drive the rate. What the +lines are: + +- `js-command-exec` (`\b(exec|execSync)\s*\(`) and the taint sink `$OBJ.exec(...)` match `RegExp.exec`. + In gitlab-mcp the request header is a taint source, so `/regex/.exec(authHeader)` reads as command + injection with a dataflow trace. The rest are `childProcess.execSync` in test files. +- `python-sql-injection` matches multi-line `cursor.execute(` calls built from f-strings or constants in + malla's repositories. The regex cannot tell a constant from input. +- `python-unsafe-eval` matched `eval()` inside a comment. The quick engine drops matches in comments; + `pattern-regex` does not (see "not carried" below). +- `sink:javascript:prototype.merge` matched `Map.set(...)`. lodash's `set` converts to `$OBJ.set(...)`. +- `sink:javascript:html.response` matched `res.send({token, userId})`, an object Express sends as JSON. + +## Runtime + +251 Semgrep runs: mean 8.5 s per change, median 7.7 s, min 5.6 s, max 37.3 s (gitlab-mcp, a 453 KB +`index.ts`). Total 35.6 min of Semgrep wall time; the whole run (with checkouts) took 36 min, +12:31 to 13:07. 846 target files and 24.8 MB were read before the runs. Most of the per-change cost is +Semgrep's fixed startup, about 5 s (the brief measured 4.7 s). The in-process regex tier runs in under a +second. + +## Instrument + +- Every run read its targets and printed bytes per file first. No run returned faster than 1 s + (`suspect_fast` 0). Every Semgrep exit was 0. For each change, `paths.scanned` is recorded next to + the bytes read. +- 1 instrument failure (0.4%, under the scorer's 5% limit): a matrix-appservice-bridge ordinary change + whose four changed files were all deleted at head (`no_readable_targets`). +- 30 changed files were missing at head and skipped. +- 103 changes had targets Semgrep did not scan because no rule covers that language: `.md` 94, `.json` + 72, no extension 16, `.html` 13, `.yml` 11, `.css` 10, and others. Our hook does not read these + either. +- 2 PartialParsing errors (TypeScript `import("...")` types, a regex literal). Both files were still + scanned. +- Smoke test before the run: the converted rules fired with traces on hand-written Python, JS/TSX, Java, + C and PHP samples (`os.system`, `innerHTML`, `dangerouslySetInnerHTML`, `executeQuery`, `strcpy`, + `echo $x;`, `include $_GET`). So the zero recall is not a dead ruleset. + +## Conversion report (`rules/conversion_report.json`) + +- Regex rules: 41 of 52 converted to `pattern-regex`. 11 were skipped: 5 shadow (`php-sql-call-composition`, + `php-code-eval`, `php-unserialize`, `php-maybe-unserialize`, `php-xss-echo-input`) and 6 + `*-language` fallbacks (inactive while their tagged rule is active). No rule targets non-code + files, and none was dropped by an emit floor (the arm uses the hook's default floor, 0.0; `--min-precision` + sets one). Groovy has no parser and becomes `languages: [generic]` with `paths.include: [*.groovy, *.tpl]`. +- One rewrite: `php-sql-sprintf-unescaped` uses `(?:...){0,2000}?`, which PCRE2 rejects as "regular + expression is too large". It became `*?`, so the window is now the whole statement instead of 2000 + characters. +- Facts: 40 taint/search rules (JS 10, PHP 10, Python 10, Java 7, C 3), one per (language, sink id, + CWE). Sources come from every source fact of that language; sanitizers come from cleansing sanitizer + facts. `weak_literals` sinks (hashlib, MessageDigest) become search rules with a `metavariable-regex`. + Format sinks focus their format argument, and parameterized calls (`execute`, `query`) focus a + single-argument call. +- Not convertible, skipped and named: + - `guard` sanitizers (`php:allowlist` = `in_array`, `python:url_allowlist`): safety depends on + dominance. + - `parameterized` sanitizer facts (4): a call shape, expressed as the focused sink form instead. + - `literal_format_arg` (C): expressed by focusing the format argument. + - `dispatch:php:wordpress_hooks`: string-keyed registry routing. + - `layout` facts (2): scope policy, not rules. + - `v-html`: a template directive. + - All 37 obligation tables: an absence has no pattern. +- Lossy (converted, but with weaker semantics): + - `quoted_only` `sql_escape` became a full sanitizer. This silences the CVE-2024-1071 shape the project + keeps. + - `prefix_fixes_origin` on 4 destination sinks (JS/Python outbound requests and redirects) is + dropped. A fixed-origin prefix no longer clears the value; Django's `reverse` anchors became + sanitizers. +- Not carried at all: comment skipping, framework priors as a switch, `min_evidence_level`, ranking, the + rung. + +## What this says + +Running the same vocabulary through Semgrep's engine adds no catches on these 12 positives. It adds +false alarms, from rules that the quick tier keeps quiet with comment skipping and changed-line scoping. +The misses come from the vocabulary (sources limited to request objects, no access_control without +obligations), not from the engine. This is the measured reason the brief asked for, and it points +against option A/B: the steering rule ("do not ... add a second engine") stands. If the arm is reused, +fix the cheap false alarms first: exclude `RegExp.exec`, skip comments, and scope results to changed +lines. + +## Reproduce + +```bash +export PYTHONPATH=$PWD +.venv/bin/python -m benchmarks.valid.semgrep_arm rules --out ~/ousast-results/valid/pilot/semgrep-rules/ +for only in vulnerability ordinary; do + .venv/bin/python -m benchmarks.valid.semgrep_arm run benchmarks/valid/pilot/pool-valid.toml --slice 1 \ + --repos ~/ousast-results/valid/pilot/repos_hist.json --rules ~/ousast-results/valid/pilot/semgrep-rules \ + --out ~/ousast-results/valid/pilot/semgrep --only $only +done +.venv/bin/python -m benchmarks.unseen.score ~/ousast-results/valid/pilot/semgrep --out ~/ousast-results/valid/pilot/scored_semgrep.json --slice-id 1 +.venv/bin/python -m benchmarks.valid.metrics ~/ousast-results/valid/pilot/semgrep --scored ~/ousast-results/valid/pilot/scored_semgrep.json --out ~/ousast-results/valid/pilot/metrics_semgrep.json +``` + +The ordinary pass exits 2 because of the one instrument failure; the run still finishes and resumes +past done records. diff --git a/benchmarks/measurements/2026-10-13-semgrep-arm/metrics_semgrep.json b/benchmarks/measurements/2026-10-13-semgrep-arm/metrics_semgrep.json new file mode 100644 index 0000000..e4a52a9 --- /dev/null +++ b/benchmarks/measurements/2026-10-13-semgrep-arm/metrics_semgrep.json @@ -0,0 +1,58 @@ +{ + "exploit_verified": { + "count": 0, + "flagged_positive_pushes": 3, + "flagged_positives": 5, + "n": 0, + "not_yet_proved": [ + "bytebase-dbhub-f2624a2ead8d31ca", + "imzbf-md-editor-v3-14658213915dbde5", + "imzbf-md-editor-v3-b9c833a3524a57a9", + "zereight-gitlab-mcp-2367b459f8c3802b", + "zereight-gitlab-mcp-768c7d5a32ccc95b" + ], + "rate": null, + "status": "not yet proved", + "wilson95": [ + null, + null + ] + }, + "false_alarms_per_100_pushes": { + "cluster95": [ + 3.75, + 23.430962343096233 + ], + "count": 30, + "icc": 0.27735846730536884, + "n": 239, + "rate": 12.552301255230125, + "wilson95": [ + 8.93573620529521, + 17.35366396337441 + ] + }, + "headline": "On unseen app repos: recall 0.0% (Wilson 0.0-24.3), false alarms 12.6 per 100 pushes, precision 0.0%; exploit-verified 0 of 5 (5 not yet proved).", + "precision": { + "count": 0, + "n": 33, + "rate": 0.0, + "wilson95": [ + 0, + 0.10427343003561192 + ] + }, + "recall": { + "cluster95": [ + 0.0, + 0.0 + ], + "count": 0, + "n": 12, + "rate": 0.0, + "wilson95": [ + 0, + 0.2425007574992425 + ] + } +} diff --git a/benchmarks/measurements/2026-10-13-semgrep-arm/rules/c.yaml b/benchmarks/measurements/2026-10-13-semgrep-arm/rules/c.yaml new file mode 100644 index 0000000..8327139 --- /dev/null +++ b/benchmarks/measurements/2026-10-13-semgrep-arm/rules/c.yaml @@ -0,0 +1,216 @@ +rules: +- id: ousast.c.regex.c-exec-family + languages: + - c + - cpp + severity: WARNING + message: C exec command execution needs review + metadata: + family: injection + cwe: CWE-78 + origin: c-exec-family + precision_estimate: '0.0' + kind: regex + tags: + - syscall_entry + status: enabled + version: '1' + pattern-regex: \b(execl|execlp|execle|execv|execvp|execvpe)\s*\( +- id: ousast.c.regex.c-shell-exec + languages: + - c + - cpp + severity: WARNING + message: C shell execution needs review + metadata: + family: injection + cwe: CWE-78 + origin: c-shell-exec + precision_estimate: '0.0' + kind: regex + tags: + - syscall_entry + status: enabled + version: '1' + pattern-regex: \b(system|popen)\s*\( +- id: ousast.c.regex.c-unsafe-copy + languages: + - c + - cpp + severity: WARNING + message: Unsafe C string copy needs review + metadata: + family: memory + cwe: CWE-121 + origin: c-unsafe-copy + precision_estimate: '0.0' + kind: regex + tags: + - memory_unsafe + status: enabled + version: '1' + pattern-regex: \b(gets|strcpy|strcat|sprintf)\s*\( +- id: ousast.c.regex.c-unsafe-format + languages: + - c + - cpp + severity: WARNING + message: Unsafe C format string needs review + metadata: + family: memory + cwe: CWE-134 + origin: c-unsafe-format + precision_estimate: '0.0' + kind: regex + tags: + - memory_unsafe + status: enabled + version: '1' + pattern-regex: \b(printf|vprintf|vsprintf|vsnprintf|vfprintf|syslog)\s*\(\s*[A-Za-z_] +- id: ousast.c.regex.c-unsafe-memory-copy + languages: + - c + - cpp + severity: WARNING + message: Unsafe C memory copy needs review + metadata: + family: memory + cwe: CWE-121 + origin: c-unsafe-memory-copy + precision_estimate: '0.0' + kind: regex + tags: + - memory_unsafe + status: enabled + version: '1' + pattern-regex: \b(memcpy|memmove|bcopy)\s*\( +- id: ousast.c.regex.c-unsafe-scanf + languages: + - c + - cpp + severity: WARNING + message: Unbounded C scanf needs review + metadata: + family: memory + cwe: CWE-121 + origin: c-unsafe-scanf + precision_estimate: '0.0' + kind: regex + tags: + - memory_unsafe + status: enabled + version: '1' + pattern-regex: \b(scanf|fscanf|sscanf)\s*\([^\n]*%s +- id: ousast.c.taint.printf + mode: taint + languages: &id001 + - c + - cpp + severity: ERROR + message: 'printf: untrusted input reaches fprintf, printf, sprintf, syslog, vprintf, vsprintf' + metadata: + family: memory + cwe: CWE-134 + origin: sink:c:printf + precision_estimate: unmeasured + kind: taint + calls: + - fprintf + - printf + - sprintf + - syslog + - vprintf + - vsprintf + pattern-sources: + - pattern-either: &id002 + - pattern: argv + - pattern: argv(...) + - pattern: getenv + - pattern: getenv(...) + - pattern: fgets + - pattern: fgets(...) + - pattern: gets + - pattern: gets(...) + - pattern: scanf + - pattern: scanf(...) + - pattern: fscanf + - pattern: fscanf(...) + pattern-sinks: + - pattern-either: + - patterns: + - pattern: fprintf($FORMAT, ...) + - focus-metavariable: $FORMAT + - patterns: + - pattern: printf($FORMAT, ...) + - focus-metavariable: $FORMAT + - patterns: + - pattern: sprintf($FORMAT, ...) + - focus-metavariable: $FORMAT + - patterns: + - pattern: syslog($FORMAT, ...) + - focus-metavariable: $FORMAT + - patterns: + - pattern: vprintf($FORMAT, ...) + - focus-metavariable: $FORMAT + - patterns: + - pattern: vsprintf($FORMAT, ...) + - focus-metavariable: $FORMAT +- id: ousast.c.taint.strcpy + mode: taint + languages: *id001 + severity: ERROR + message: 'strcpy: untrusted input reaches gets, sprintf, strcat, strcpy' + metadata: + family: memory + cwe: CWE-121 + origin: sink:c:strcpy + precision_estimate: unmeasured + kind: taint + calls: + - gets + - sprintf + - strcat + - strcpy + pattern-sources: + - pattern-either: *id002 + pattern-sinks: + - pattern-either: + - pattern: gets(...) + - pattern: $OBJ.gets(...) + - pattern: sprintf(...) + - pattern: $OBJ.sprintf(...) + - pattern: strcat(...) + - pattern: $OBJ.strcat(...) + - pattern: strcpy(...) + - pattern: $OBJ.strcpy(...) +- id: ousast.c.taint.system + mode: taint + languages: *id001 + severity: ERROR + message: 'system: untrusted input reaches execl, execv, execvp, popen, system' + metadata: + family: injection + cwe: CWE-78 + origin: sink:c:system + precision_estimate: unmeasured + kind: taint + calls: + - execl + - execv + - execvp + - popen + - system + pattern-sources: + - pattern-either: *id002 + pattern-sinks: + - pattern-either: + - pattern: execl(...) + - pattern: $OBJ.execl(...) + - pattern: execv(...) + - pattern: $OBJ.execv(...) + - pattern: execvp(...) + - pattern: $OBJ.execvp(...) + - pattern: popen(...) + - pattern: $OBJ.popen(...) + - pattern: system(...) + - pattern: $OBJ.system(...) diff --git a/benchmarks/measurements/2026-10-13-semgrep-arm/rules/conversion_report.json b/benchmarks/measurements/2026-10-13-semgrep-arm/rules/conversion_report.json new file mode 100644 index 0000000..a0deaf1 --- /dev/null +++ b/benchmarks/measurements/2026-10-13-semgrep-arm/rules/conversion_report.json @@ -0,0 +1,233 @@ +{ + "facts": { + "converted": [ + "sink:c:printf", + "sink:c:strcpy", + "sink:c:system", + "sink:java:MessageDigest", + "sink:java:exec", + "sink:java:executeQuery", + "sink:java:file.open", + "sink:java:outbound.request", + "sink:java:readObject", + "sink:java:response.write", + "sink:javascript:eval", + "sink:javascript:exec", + "sink:javascript:html.response", + "sink:javascript:html.unescaped", + "sink:javascript:outbound.request", + "sink:javascript:path.open", + "sink:javascript:prototype.merge", + "sink:javascript:query", + "sink:javascript:redirect", + "sink:javascript:vm.eval", + "sink:php:code_eval", + "sink:php:command", + "sink:php:echo", + "sink:php:file_inclusion", + "sink:php:filesystem", + "sink:php:outbound_request", + "sink:php:redirect", + "sink:php:sql", + "sink:php:unserialize", + "sink:php:wpdb", + "sink:python:eval", + "sink:python:execute", + "sink:python:hashlib", + "sink:python:html.unescaped", + "sink:python:orm.raw", + "sink:python:os.system", + "sink:python:outbound.request", + "sink:python:path.open", + "sink:python:pickle.loads", + "sink:python:redirect" + ], + "lossy": [ + { + "loss": "prefix_fixes_origin: a value appended after a fixed origin still counts as tainted", + "origin": "sink:javascript:outbound.request" + }, + { + "loss": "prefix_fixes_origin: a value appended after a fixed origin still counts as tainted", + "origin": "sink:javascript:redirect" + }, + { + "loss": "quoted_only escape treated as a full sanitizer (silences the CVE-2024-1071 shape)", + "origin": "sanitizer:php:sql_escape" + }, + { + "loss": "prefix_fixes_origin: a value appended after a fixed origin still counts as tainted", + "origin": "sink:python:outbound.request" + }, + { + "loss": "prefix_fixes_origin: a value appended after a fixed origin still counts as tainted", + "origin": "sink:python:redirect" + } + ], + "skipped": [ + { + "origin": "sanitizer:c:literal_format", + "reason": "literal_format_arg: a shape; a literal format is never tainted at the focused sink argument" + }, + { + "origin": "sanitizer:java:parameterized_statement", + "reason": "parameterized: a call shape, not a cleansing; expressed as the single-argument sink form" + }, + { + "origin": "sanitizer:javascript:parameterized_query", + "reason": "parameterized: a call shape, not a cleansing; expressed as the single-argument sink form" + }, + { + "origin": "layout:javascript:javascript_first_party_tests", + "reason": "layout: scope policy (vendored, tests), not a rule" + }, + { + "origin": "sink:javascript:html.unescaped:v-html", + "reason": "template directive, not a call" + }, + { + "origin": "sanitizer:php:allowlist", + "reason": "guard: a check whose safety depends on dominance; Semgrep has no equivalent" + }, + { + "origin": "sanitizer:php:prepared_statement", + "reason": "parameterized: a call shape, not a cleansing; expressed as the single-argument sink form" + }, + { + "origin": "dispatch:php:wordpress_hooks", + "reason": "dispatch: string-keyed hook registry routing; no Semgrep equivalent" + }, + { + "origin": "layout:php:php_layout", + "reason": "layout: scope policy (vendored, tests), not a rule" + }, + { + "origin": "sanitizer:python:parameterized_execute", + "reason": "parameterized: a call shape, not a cleansing; expressed as the single-argument sink form" + }, + { + "origin": "sanitizer:python:url_allowlist", + "reason": "guard: a check whose safety depends on dominance; Semgrep has no equivalent" + } + ] + }, + "files": { + "c.yaml": 9, + "groovy.yaml": 1, + "java.yaml": 11, + "javascript.yaml": 21, + "php.yaml": 19, + "python.yaml": 20 + }, + "not_carried": [ + "comment skipping: the quick engine drops a regex match inside a comment; pattern-regex does not", + "framework priors and fallback rules: the tagged rule is emitted, its language-level fallback is not", + "min_evidence_level, ranking priority, the rung: post-match policy on our side" + ], + "obligations": { + "reason": "an obligation finds a missing guard (an absence), which no pattern expresses", + "skipped_tables": 37 + }, + "regex": { + "converted": [ + "c-exec-family", + "c-shell-exec", + "c-unsafe-copy", + "c-unsafe-format", + "c-unsafe-memory-copy", + "c-unsafe-scanf", + "groovy-xss-unescaped", + "java-command-exec", + "java-deserialization", + "java-sql-injection", + "java-weak-hash", + "js-command-exec", + "js-deserialize", + "js-dom-xss", + "js-eval", + "js-function-constructor", + "js-path-traversal", + "js-spawn-shell", + "js-sql-injection", + "js-ssrf", + "js-weak-hash", + "js-xss-response", + "php-wpdb-sql-composition", + "php-sql-unquoted-concat", + "php-sql-unquoted-interpolation", + "php-sql-sprintf-unescaped", + "php-command-injection", + "php-file-inclusion-input", + "php-path-input", + "php-ssrf-input", + "php-open-redirect", + "python-flask-debug", + "python-os-command", + "python-path-traversal", + "python-shell-true", + "python-sql-injection", + "python-ssrf", + "python-ssti", + "python-unsafe-deserialization", + "python-unsafe-eval", + "python-weak-hash" + ], + "rewritten": [ + { + "origin": "php-sql-sprintf-unescaped", + "rewrite": "bounded lazy group repeat {0,N}? widened to *? (PCRE2 size limit)" + } + ], + "skipped": [ + { + "origin": "js-ssrf-language", + "reason": "fallback for js-ssrf: the tagged rule is active" + }, + { + "origin": "php-sql-call-composition", + "reason": "shadow status (precision_estimate 0.25); not admitted by the hook" + }, + { + "origin": "php-sql-unquoted-concat-language", + "reason": "fallback for php-sql-unquoted-concat: the tagged rule is active" + }, + { + "origin": "php-sql-unquoted-interpolation-language", + "reason": "fallback for php-sql-unquoted-interpolation: the tagged rule is active" + }, + { + "origin": "php-sql-sprintf-unescaped-language", + "reason": "fallback for php-sql-sprintf-unescaped: the tagged rule is active" + }, + { + "origin": "php-code-eval", + "reason": "shadow status (precision_estimate 0.333); not admitted by the hook" + }, + { + "origin": "php-unserialize", + "reason": "shadow status (precision_estimate 0.25); not admitted by the hook" + }, + { + "origin": "php-maybe-unserialize", + "reason": "shadow status (precision_estimate 0.25); not admitted by the hook" + }, + { + "origin": "php-xss-echo-input", + "reason": "shadow status (precision_estimate 0.333); not admitted by the hook" + }, + { + "origin": "php-xss-echo-input-language", + "reason": "fallback for php-xss-echo-input: the tagged rule is active" + }, + { + "origin": "python-unsafe-deserialization-language", + "reason": "fallback for python-unsafe-deserialization: the tagged rule is active" + } + ] + }, + "validation": { + "exit": 0, + "stderr": "Configuration is valid - found 0 configuration error(s), and 81 rule(s).\n", + "stdout": "" + } +} diff --git a/benchmarks/measurements/2026-10-13-semgrep-arm/rules/groovy.yaml b/benchmarks/measurements/2026-10-13-semgrep-arm/rules/groovy.yaml new file mode 100644 index 0000000..813595e --- /dev/null +++ b/benchmarks/measurements/2026-10-13-semgrep-arm/rules/groovy.yaml @@ -0,0 +1,21 @@ +rules: +- id: ousast.groovy.regex.groovy-xss-unescaped + languages: + - generic + severity: WARNING + message: Groovy template unescaped output needs review + metadata: + family: output_encoding + cwe: CWE-79 + origin: groovy-xss-unescaped + precision_estimate: '0.0' + kind: regex + tags: + - injection + status: enabled + version: '1' + pattern-regex: (yieldUnescaped)\s+[A-Za-z_] + paths: + include: + - '*.groovy' + - '*.tpl' diff --git a/benchmarks/measurements/2026-10-13-semgrep-arm/rules/java.yaml b/benchmarks/measurements/2026-10-13-semgrep-arm/rules/java.yaml new file mode 100644 index 0000000..c3d3892 --- /dev/null +++ b/benchmarks/measurements/2026-10-13-semgrep-arm/rules/java.yaml @@ -0,0 +1,362 @@ +rules: +- id: ousast.java.regex.java-command-exec + languages: + - java + severity: WARNING + message: Java runtime command execution needs review + metadata: + family: injection + cwe: CWE-78 + origin: java-command-exec + precision_estimate: '0.0' + kind: regex + tags: + - syscall_entry + status: enabled + version: '1' + pattern-regex: \b(exec)\s*\( +- id: ousast.java.regex.java-deserialization + languages: + - java + severity: WARNING + message: Java unsafe deserialization needs review + metadata: + family: deserialization + cwe: CWE-502 + origin: java-deserialization + precision_estimate: '0.0' + kind: regex + tags: + - deserialization + status: enabled + version: '1' + pattern-regex: \.(readObject)\s*\( +- id: ousast.java.regex.java-sql-injection + languages: + - java + severity: WARNING + message: Java SQL string composition needs review + metadata: + family: injection + cwe: CWE-89 + origin: java-sql-injection + precision_estimate: '0.0' + kind: regex + tags: + - injection + status: enabled + version: '1' + pattern-regex: (?i)\"\s*(select|insert|update|delete)\b[^\"]*\"\s*\+ +- id: ousast.java.regex.java-weak-hash + languages: + - java + severity: WARNING + message: Java weak hash needs review + metadata: + family: config_secrets + cwe: CWE-327 + origin: java-weak-hash + precision_estimate: '0.0' + kind: regex + tags: + - crypto + status: enabled + version: '1' + pattern-regex: MessageDigest\.getInstance\s*\(\s*\"(MD5|SHA-1)\" +- id: ousast.java.weak.MessageDigest + languages: &id001 + - java + severity: WARNING + message: 'MessageDigest: a weak literal reaches getInstance' + metadata: + family: config_secrets + cwe: CWE-327 + origin: sink:java:MessageDigest + precision_estimate: unmeasured + kind: weak_literal + weak_literals: + - MD5 + - SHA-1 + - SHA1 + pattern-either: + - patterns: + - pattern: getInstance($ALGORITHM, ...) + - metavariable-regex: + metavariable: $ALGORITHM + regex: ^["']?(MD5|SHA\-1|SHA1)["']?$ + - patterns: + - pattern: $OBJ.getInstance($ALGORITHM, ...) + - metavariable-regex: + metavariable: $ALGORITHM + regex: ^["']?(MD5|SHA\-1|SHA1)["']?$ +- id: ousast.java.taint.exec + mode: taint + languages: *id001 + severity: ERROR + message: 'exec: untrusted input reaches exec' + metadata: + family: injection + cwe: CWE-78 + origin: sink:java:exec + precision_estimate: unmeasured + kind: taint + calls: + - exec + pattern-sources: + - pattern-either: &id002 + - pattern: getParameter + - pattern: $OBJ.getParameter + - pattern: getParameter(...) + - pattern: $OBJ.getParameter(...) + - pattern: getHeader + - pattern: $OBJ.getHeader + - pattern: getHeader(...) + - pattern: $OBJ.getHeader(...) + - pattern: getCookies + - pattern: $OBJ.getCookies + - pattern: getCookies(...) + - pattern: $OBJ.getCookies(...) + - pattern: getQueryString + - pattern: $OBJ.getQueryString + - pattern: getQueryString(...) + - pattern: $OBJ.getQueryString(...) + - pattern: readLine + - pattern: $OBJ.readLine + - pattern: readLine(...) + - pattern: $OBJ.readLine(...) + pattern-sinks: + - pattern-either: + - pattern: exec(...) + - pattern: $OBJ.exec(...) + pattern-sanitizers: + - pattern-either: + - &id003 + pattern: getCanonicalPath(...) + - &id004 + pattern: $OBJ.getCanonicalPath(...) + - &id005 + pattern: normalize(...) + - &id006 + pattern: $OBJ.normalize(...) + - &id007 + pattern: URLEncoder.encode(...) + - &id008 + pattern: ESAPI.encoder(...) + - &id009 + pattern: encodeForHTML(...) + - &id010 + pattern: $OBJ.encodeForHTML(...) + - &id011 + pattern: encodeForURL(...) + - &id012 + pattern: $OBJ.encodeForURL(...) + - &id013 + pattern: StringEscapeUtils.escapeHtml4(...) +- id: ousast.java.taint.executeQuery + mode: taint + languages: *id001 + severity: ERROR + message: 'executeQuery: untrusted input reaches execute, executeQuery, prepareCall' + metadata: + family: injection + cwe: CWE-89 + origin: sink:java:executeQuery + precision_estimate: unmeasured + kind: taint + calls: + - execute + - executeQuery + - prepareCall + pattern-sources: + - pattern-either: *id002 + pattern-sinks: + - pattern-either: + - pattern: execute(...) + - pattern: $OBJ.execute(...) + - pattern: executeQuery(...) + - pattern: $OBJ.executeQuery(...) + - pattern: prepareCall(...) + - pattern: $OBJ.prepareCall(...) + pattern-sanitizers: + - pattern-either: + - *id003 + - *id004 + - *id005 + - *id006 + - *id007 + - *id008 + - *id009 + - *id010 + - *id011 + - *id012 + - *id013 +- id: ousast.java.taint.file-open + mode: taint + languages: *id001 + severity: ERROR + message: 'file.open: untrusted input reaches File, FileInputStream, FileOutputStream, FileReader, FileWriter, Files.newInputStream, Files.readAllBytes, Paths.get, RandomAccessFile' + metadata: + family: path + cwe: CWE-22 + origin: sink:java:file.open + precision_estimate: unmeasured + kind: taint + calls: + - File + - FileInputStream + - FileOutputStream + - FileReader + - FileWriter + - Files.newInputStream + - Files.readAllBytes + - Paths.get + - RandomAccessFile + pattern-sources: + - pattern-either: *id002 + pattern-sinks: + - pattern-either: + - pattern: new File(...) + - pattern: new FileInputStream(...) + - pattern: new FileOutputStream(...) + - pattern: new FileReader(...) + - pattern: new FileWriter(...) + - pattern: Files.newInputStream(...) + - pattern: Files.readAllBytes(...) + - pattern: Paths.get(...) + - pattern: new RandomAccessFile(...) + pattern-sanitizers: + - pattern-either: + - *id003 + - *id004 + - *id005 + - *id006 + - *id007 + - *id008 + - *id009 + - *id010 + - *id011 + - *id012 + - *id013 +- id: ousast.java.taint.outbound-request + mode: taint + languages: *id001 + severity: ERROR + message: 'outbound.request: untrusted input reaches exchange, getForObject, openConnection, openStream, sendRedirect' + metadata: + family: untrusted_destination + cwe: CWE-918 + origin: sink:java:outbound.request + precision_estimate: unmeasured + kind: taint + calls: + - exchange + - getForObject + - openConnection + - openStream + - sendRedirect + pattern-sources: + - pattern-either: *id002 + pattern-sinks: + - pattern-either: + - pattern: exchange(...) + - pattern: $OBJ.exchange(...) + - pattern: getForObject(...) + - pattern: $OBJ.getForObject(...) + - pattern: openConnection(...) + - pattern: $OBJ.openConnection(...) + - pattern: openStream(...) + - pattern: $OBJ.openStream(...) + - pattern: sendRedirect(...) + - pattern: $OBJ.sendRedirect(...) + pattern-sanitizers: + - pattern-either: + - *id003 + - *id004 + - *id005 + - *id006 + - *id007 + - *id008 + - *id009 + - *id010 + - *id011 + - *id012 + - *id013 +- id: ousast.java.taint.readObject + mode: taint + languages: *id001 + severity: ERROR + message: 'readObject: untrusted input reaches readObject' + metadata: + family: deserialization + cwe: CWE-502 + origin: sink:java:readObject + precision_estimate: unmeasured + kind: taint + calls: + - readObject + pattern-sources: + - pattern-either: *id002 + pattern-sinks: + - pattern-either: + - pattern: readObject(...) + - pattern: $OBJ.readObject(...) + pattern-sanitizers: + - pattern-either: + - *id003 + - *id004 + - *id005 + - *id006 + - *id007 + - *id008 + - *id009 + - *id010 + - *id011 + - *id012 + - *id013 +- id: ousast.java.taint.response-write + mode: taint + languages: *id001 + severity: ERROR + message: 'response.write: untrusted input reaches addHeader, getWriter, print, println, setHeader, write' + metadata: + family: output_encoding + cwe: CWE-79 + origin: sink:java:response.write + precision_estimate: unmeasured + kind: taint + calls: + - addHeader + - getWriter + - print + - println + - setHeader + - write + pattern-sources: + - pattern-either: *id002 + pattern-sinks: + - pattern-either: + - pattern: addHeader(...) + - pattern: $OBJ.addHeader(...) + - pattern: getWriter(...) + - pattern: $OBJ.getWriter(...) + - pattern: print(...) + - pattern: $OBJ.print(...) + - pattern: println(...) + - pattern: $OBJ.println(...) + - pattern: setHeader(...) + - pattern: $OBJ.setHeader(...) + - pattern: write(...) + - pattern: $OBJ.write(...) + pattern-sanitizers: + - pattern-either: + - *id003 + - *id004 + - *id005 + - *id006 + - *id007 + - *id008 + - *id009 + - *id010 + - *id011 + - *id012 + - *id013 diff --git a/benchmarks/measurements/2026-10-13-semgrep-arm/rules/javascript.yaml b/benchmarks/measurements/2026-10-13-semgrep-arm/rules/javascript.yaml new file mode 100644 index 0000000..fe98118 --- /dev/null +++ b/benchmarks/measurements/2026-10-13-semgrep-arm/rules/javascript.yaml @@ -0,0 +1,679 @@ +rules: +- id: ousast.javascript.regex.js-command-exec + languages: + - javascript + - typescript + severity: WARNING + message: Node child_process command execution needs review + metadata: + family: injection + cwe: CWE-78 + origin: js-command-exec + precision_estimate: '0.0' + kind: regex + tags: + - syscall_entry + status: enabled + version: '1' + pattern-regex: \b(exec|execSync)\s*\( +- id: ousast.javascript.regex.js-deserialize + languages: + - javascript + - typescript + severity: WARNING + message: JavaScript unsafe deserialization needs review + metadata: + family: deserialization + cwe: CWE-502 + origin: js-deserialize + precision_estimate: '0.0' + kind: regex + tags: + - deserialization + status: enabled + version: '1' + pattern-regex: \b(unserialize)\s*\( +- id: ousast.javascript.regex.js-dom-xss + languages: + - javascript + - typescript + severity: WARNING + message: DOM sink with untrusted input needs review + metadata: + family: output_encoding + cwe: CWE-79 + origin: js-dom-xss + precision_estimate: '0.0' + kind: regex + tags: + - injection + status: enabled + version: '1' + pattern-regex: \.(innerHTML|outerHTML)\s*=\s*[^\n]*\b(?:req|request|location|document\.location) +- id: ousast.javascript.regex.js-eval + languages: + - javascript + - typescript + severity: WARNING + message: Dynamic JavaScript evaluation needs review + metadata: + family: injection + cwe: CWE-95 + origin: js-eval + precision_estimate: '0.0' + kind: regex + tags: + - syscall_entry + status: enabled + version: '1' + pattern-regex: \b(eval)\s*\( +- id: ousast.javascript.regex.js-function-constructor + languages: + - javascript + - typescript + severity: WARNING + message: JavaScript Function constructor needs review + metadata: + family: injection + cwe: CWE-95 + origin: js-function-constructor + precision_estimate: '0.0' + kind: regex + tags: + - syscall_entry + status: enabled + version: '1' + pattern-regex: new\s+(Function)\s*\( +- id: ousast.javascript.regex.js-path-traversal + languages: + - javascript + - typescript + severity: WARNING + message: Node filesystem access with external input needs review + metadata: + family: path + cwe: CWE-22 + origin: js-path-traversal + precision_estimate: '0.0' + kind: regex + tags: + - filesystem_entry + status: enabled + version: '1' + pattern-regex: \bfs\.(readFile|readFileSync|writeFile|writeFileSync|createReadStream|unlink|unlinkSync)\s*\(\s*[^\n]*\b(?:req|request)\. +- id: ousast.javascript.regex.js-spawn-shell + languages: + - javascript + - typescript + severity: WARNING + message: Node spawn with shell needs review + metadata: + family: injection + cwe: CWE-78 + origin: js-spawn-shell + precision_estimate: '0.0' + kind: regex + tags: + - syscall_entry + status: enabled + version: '1' + pattern-regex: \b(spawn|spawnSync)\s*\([^\n]*shell\s*:\s*true +- id: ousast.javascript.regex.js-sql-injection + languages: + - javascript + - typescript + severity: WARNING + message: JavaScript SQL string composition needs review + metadata: + family: injection + cwe: CWE-89 + origin: js-sql-injection + precision_estimate: '0.0' + kind: regex + tags: + - injection + status: enabled + version: '1' + pattern-regex: \.(query|execute)\s*\(\s*[^\n]*(?:\$\{|\+\s*[A-Za-z_$]) +- id: ousast.javascript.regex.js-ssrf + languages: + - javascript + - typescript + severity: WARNING + message: JavaScript outbound request with external input needs review + metadata: + family: untrusted_destination + cwe: CWE-918 + origin: js-ssrf + precision_estimate: '0.0' + kind: regex + tags: + - network_entry + status: enabled + version: '1' + pattern-regex: \b(axios|fetch|got)\s*[.(][^\n]*\b(?:req|request)\. +- id: ousast.javascript.regex.js-weak-hash + languages: + - javascript + - typescript + severity: WARNING + message: JavaScript weak hash needs review + metadata: + family: config_secrets + cwe: CWE-327 + origin: js-weak-hash + precision_estimate: '0.0' + kind: regex + tags: + - crypto + status: enabled + version: '1' + pattern-regex: createHash\s*\(\s*[\"'](md5|sha1)[\"'] +- id: ousast.javascript.regex.js-xss-response + languages: + - javascript + - typescript + severity: WARNING + message: Unsanitized HTTP response reflection needs review + metadata: + family: output_encoding + cwe: CWE-79 + origin: js-xss-response + precision_estimate: '0.0' + kind: regex + tags: + - injection + status: enabled + version: '1' + pattern-regex: \.(send|write|end)\s*\(\s*[^\n]*\b(?:req|request)\.(?:query|params|body) +- id: ousast.javascript.taint.eval + mode: taint + languages: &id001 + - javascript + - typescript + severity: ERROR + message: 'eval: untrusted input reaches eval' + metadata: + family: injection + cwe: CWE-95 + origin: sink:javascript:eval + precision_estimate: unmeasured + kind: taint + calls: + - eval + pattern-sources: + - pattern-either: &id002 + - pattern: req.query + - pattern: req.query(...) + - pattern: req.body + - pattern: req.body(...) + - pattern: req.params + - pattern: req.params(...) + - pattern: req.cookies + - pattern: req.cookies(...) + - pattern: req.headers + - pattern: req.headers(...) + - pattern: request.query + - pattern: request.query(...) + - pattern: req.url + - pattern: req.url(...) + - pattern: request.url + - pattern: request.url(...) + - pattern: req.rawHeaders + - pattern: req.rawHeaders(...) + - pattern: url.parse + - pattern: url.parse(...) + - pattern: parsedUrl + - pattern: $OBJ.parsedUrl + - pattern: parsedUrl(...) + - pattern: $OBJ.parsedUrl(...) + pattern-sinks: + - pattern-either: + - pattern: eval(...) + - pattern: $OBJ.eval(...) + pattern-sanitizers: + - pattern-either: + - &id003 + pattern: path.normalize(...) + - &id004 + pattern: path.resolve(...) + - &id005 + pattern: encodeURIComponent(...) + - &id006 + pattern: $OBJ.encodeURIComponent(...) + - &id007 + pattern: encodeURI(...) + - &id008 + pattern: $OBJ.encodeURI(...) + - &id009 + pattern: escape(...) + - &id010 + pattern: $OBJ.escape(...) + - &id011 + pattern: escapeHtml(...) + - &id012 + pattern: $OBJ.escapeHtml(...) + - &id013 + pattern: sanitizeHtml(...) + - &id014 + pattern: $OBJ.sanitizeHtml(...) + - &id015 + pattern: DOMPurify.sanitize(...) +- id: ousast.javascript.taint.exec + mode: taint + languages: *id001 + severity: ERROR + message: 'exec: untrusted input reaches exec, execSync' + metadata: + family: injection + cwe: CWE-78 + origin: sink:javascript:exec + precision_estimate: unmeasured + kind: taint + calls: + - exec + - execSync + pattern-sources: + - pattern-either: *id002 + pattern-sinks: + - pattern-either: + - pattern: exec(...) + - pattern: $OBJ.exec(...) + - pattern: execSync(...) + - pattern: $OBJ.execSync(...) + pattern-sanitizers: + - pattern-either: + - *id003 + - *id004 + - *id005 + - *id006 + - *id007 + - *id008 + - *id009 + - *id010 + - *id011 + - *id012 + - *id013 + - *id014 + - *id015 +- id: ousast.javascript.taint.html-response + mode: taint + languages: *id001 + severity: ERROR + message: 'html.response: untrusted input reaches res.end, res.send, res.write, response.end, response.send, response.write' + metadata: + family: output_encoding + cwe: CWE-79 + origin: sink:javascript:html.response + precision_estimate: unmeasured + kind: taint + calls: + - res.end + - res.send + - res.write + - response.end + - response.send + - response.write + pattern-sources: + - pattern-either: *id002 + pattern-sinks: + - pattern-either: + - pattern: res.end(...) + - pattern: res.send(...) + - pattern: res.write(...) + - pattern: response.end(...) + - pattern: response.send(...) + - pattern: response.write(...) + pattern-sanitizers: + - pattern-either: + - *id003 + - *id004 + - *id005 + - *id006 + - *id007 + - *id008 + - *id009 + - *id010 + - *id011 + - *id012 + - *id013 + - *id014 + - *id015 +- id: ousast.javascript.taint.html-unescaped + mode: taint + languages: *id001 + severity: ERROR + message: 'html.unescaped: untrusted input reaches dangerouslySetInnerHTML, document.write, innerHTML, insertAdjacentHTML, outerHTML, v-html' + metadata: + family: output_encoding + cwe: CWE-79 + origin: sink:javascript:html.unescaped + precision_estimate: unmeasured + kind: taint + calls: + - dangerouslySetInnerHTML + - document.write + - innerHTML + - insertAdjacentHTML + - outerHTML + - v-html + pattern-sources: + - pattern-either: *id002 + pattern-sinks: + - pattern-either: + - patterns: + - pattern: '{__html: $VALUE}' + - focus-metavariable: $VALUE + - pattern: document.write(...) + - patterns: + - pattern: $OBJ.innerHTML = $VALUE + - focus-metavariable: $VALUE + - pattern: insertAdjacentHTML(...) + - pattern: $OBJ.insertAdjacentHTML(...) + - patterns: + - pattern: $OBJ.outerHTML = $VALUE + - focus-metavariable: $VALUE + pattern-sanitizers: + - pattern-either: + - *id003 + - *id004 + - *id005 + - *id006 + - *id007 + - *id008 + - *id009 + - *id010 + - *id011 + - *id012 + - *id013 + - *id014 + - *id015 +- id: ousast.javascript.taint.outbound-request + mode: taint + languages: *id001 + severity: ERROR + message: 'outbound.request: untrusted input reaches axios, fetch, got, http.get, http.request, https.get, https.request, needle, request, superagent, undici, unirest' + metadata: + family: untrusted_destination + cwe: CWE-918 + origin: sink:javascript:outbound.request + precision_estimate: unmeasured + kind: taint + calls: + - axios + - fetch + - got + - http.get + - http.request + - https.get + - https.request + - needle + - request + - superagent + - undici + - unirest + pattern-sources: + - pattern-either: *id002 + pattern-sinks: + - pattern-either: + - pattern: axios(...) + - pattern: $OBJ.axios(...) + - pattern: fetch(...) + - pattern: $OBJ.fetch(...) + - pattern: got(...) + - pattern: $OBJ.got(...) + - pattern: http.get(...) + - pattern: http.request(...) + - pattern: https.get(...) + - pattern: https.request(...) + - pattern: needle(...) + - pattern: $OBJ.needle(...) + - pattern: request(...) + - pattern: $OBJ.request(...) + - pattern: superagent(...) + - pattern: $OBJ.superagent(...) + - pattern: undici(...) + - pattern: $OBJ.undici(...) + - pattern: unirest(...) + - pattern: $OBJ.unirest(...) + pattern-sanitizers: + - pattern-either: + - *id003 + - *id004 + - *id005 + - *id006 + - *id007 + - *id008 + - *id009 + - *id010 + - *id011 + - *id012 + - *id013 + - *id014 + - *id015 +- id: ousast.javascript.taint.path-open + mode: taint + languages: *id001 + severity: ERROR + message: 'path.open: untrusted input reaches createReadStream, readFile, readFileSync, sendFile, unlink, unlinkSync' + metadata: + family: path + cwe: CWE-22 + origin: sink:javascript:path.open + precision_estimate: unmeasured + kind: taint + calls: + - createReadStream + - readFile + - readFileSync + - sendFile + - unlink + - unlinkSync + pattern-sources: + - pattern-either: *id002 + pattern-sinks: + - pattern-either: + - pattern: createReadStream(...) + - pattern: $OBJ.createReadStream(...) + - pattern: readFile(...) + - pattern: $OBJ.readFile(...) + - pattern: readFileSync(...) + - pattern: $OBJ.readFileSync(...) + - pattern: sendFile(...) + - pattern: $OBJ.sendFile(...) + - pattern: unlink(...) + - pattern: $OBJ.unlink(...) + - pattern: unlinkSync(...) + - pattern: $OBJ.unlinkSync(...) + pattern-sanitizers: + - pattern-either: + - *id003 + - *id004 + - *id005 + - *id006 + - *id007 + - *id008 + - *id009 + - *id010 + - *id011 + - *id012 + - *id013 + - *id014 + - *id015 +- id: ousast.javascript.taint.prototype-merge + mode: taint + languages: *id001 + severity: ERROR + message: 'prototype.merge: untrusted input reaches Object.assign, defaultsDeep, extend, merge, mergeDeep, set, setWith' + metadata: + family: prototype + cwe: CWE-1321 + origin: sink:javascript:prototype.merge + precision_estimate: unmeasured + kind: taint + calls: + - Object.assign + - defaultsDeep + - extend + - merge + - mergeDeep + - set + - setWith + pattern-sources: + - pattern-either: *id002 + pattern-sinks: + - pattern-either: + - pattern: Object.assign(...) + - pattern: defaultsDeep(...) + - pattern: $OBJ.defaultsDeep(...) + - pattern: extend(...) + - pattern: $OBJ.extend(...) + - pattern: merge(...) + - pattern: $OBJ.merge(...) + - pattern: mergeDeep(...) + - pattern: $OBJ.mergeDeep(...) + - pattern: set(...) + - pattern: $OBJ.set(...) + - pattern: setWith(...) + - pattern: $OBJ.setWith(...) + pattern-sanitizers: + - pattern-either: + - *id003 + - *id004 + - *id005 + - *id006 + - *id007 + - *id008 + - *id009 + - *id010 + - *id011 + - *id012 + - *id013 + - *id014 + - *id015 +- id: ousast.javascript.taint.query + mode: taint + languages: *id001 + severity: ERROR + message: 'query: untrusted input reaches execute, query' + metadata: + family: injection + cwe: CWE-89 + origin: sink:javascript:query + precision_estimate: unmeasured + kind: taint + calls: + - execute + - query + pattern-sources: + - pattern-either: *id002 + pattern-sinks: + - pattern-either: + - patterns: + - pattern: execute($QUERY) + - focus-metavariable: $QUERY + - patterns: + - pattern: $OBJ.execute($QUERY) + - focus-metavariable: $QUERY + - patterns: + - pattern: query($QUERY) + - focus-metavariable: $QUERY + - patterns: + - pattern: $OBJ.query($QUERY) + - focus-metavariable: $QUERY + pattern-sanitizers: + - pattern-either: + - *id003 + - *id004 + - *id005 + - *id006 + - *id007 + - *id008 + - *id009 + - *id010 + - *id011 + - *id012 + - *id013 + - *id014 + - *id015 +- id: ousast.javascript.taint.redirect + mode: taint + languages: *id001 + severity: ERROR + message: 'redirect: untrusted input reaches location.assign, location.replace, redirect' + metadata: + family: untrusted_destination + cwe: CWE-601 + origin: sink:javascript:redirect + precision_estimate: unmeasured + kind: taint + calls: + - location.assign + - location.replace + - redirect + pattern-sources: + - pattern-either: *id002 + pattern-sinks: + - pattern-either: + - pattern: location.assign(...) + - pattern: location.replace(...) + - pattern: redirect(...) + - pattern: $OBJ.redirect(...) + pattern-sanitizers: + - pattern-either: + - *id003 + - *id004 + - *id005 + - *id006 + - *id007 + - *id008 + - *id009 + - *id010 + - *id011 + - *id012 + - *id013 + - *id014 + - *id015 +- id: ousast.javascript.taint.vm-eval + mode: taint + languages: *id001 + severity: ERROR + message: 'vm.eval: untrusted input reaches compileFunction, runInContext, runInNewContext, runInThisContext' + metadata: + family: injection + cwe: CWE-95 + origin: sink:javascript:vm.eval + precision_estimate: unmeasured + kind: taint + calls: + - compileFunction + - runInContext + - runInNewContext + - runInThisContext + pattern-sources: + - pattern-either: *id002 + pattern-sinks: + - pattern-either: + - pattern: compileFunction(...) + - pattern: $OBJ.compileFunction(...) + - pattern: runInContext(...) + - pattern: $OBJ.runInContext(...) + - pattern: runInNewContext(...) + - pattern: $OBJ.runInNewContext(...) + - pattern: runInThisContext(...) + - pattern: $OBJ.runInThisContext(...) + pattern-sanitizers: + - pattern-either: + - *id003 + - *id004 + - *id005 + - *id006 + - *id007 + - *id008 + - *id009 + - *id010 + - *id011 + - *id012 + - *id013 + - *id014 + - *id015 diff --git a/benchmarks/measurements/2026-10-13-semgrep-arm/rules/php.yaml b/benchmarks/measurements/2026-10-13-semgrep-arm/rules/php.yaml new file mode 100644 index 0000000..2b033b4 --- /dev/null +++ b/benchmarks/measurements/2026-10-13-semgrep-arm/rules/php.yaml @@ -0,0 +1,647 @@ +rules: +- id: ousast.php.regex.php-wpdb-sql-composition + languages: + - php + severity: WARNING + message: WordPress $wpdb query composed from a variable without $wpdb->prepare + metadata: + family: injection + cwe: CWE-89 + origin: php-wpdb-sql-composition + precision_estimate: '0.032' + kind: regex + tags: + - injection + status: enabled + version: '1' + pattern-regex: \$wpdb->(get_var|get_results|get_row|get_col|query|replace)\s*\(\s*(?!\$wpdb->prepare)(?:"[^"\n]*\{?\$(?!wpdb->)[A-Za-z_]|[^\n;]*?["']\s*\.\s*\$(?!wpdb->)[A-Za-z_]|[^\n;]*?\$_(?:GET|POST|REQUEST|COOKIE)\b) +- id: ousast.php.regex.php-sql-unquoted-concat + languages: + - php + severity: WARNING + message: PHP SQL literal concatenates a variable into an unquoted position + metadata: + family: injection + cwe: CWE-89 + origin: php-sql-unquoted-concat + precision_estimate: '0.034' + kind: regex + tags: + - injection + status: enabled + version: '1' + pattern-regex: (?])(?=(VALUES|IN|ORDER|GROUP|LIMIT|OFFSET|WHERE|AND|OR|SET|HAVING)\b)(?:(?:VALUES|IN)\s*\(|(?:ORDER|GROUP)\s+BY\b[^"'\n]*|(?:LIMIT|OFFSET)\s+(?:\d+\s*,\s*)?|(?:WHERE|AND|OR|SET|HAVING)\s+[\w.`]+\s*(?:=|<>|!=|<=|>=|<|>|LIKE)\s*)["']\s*\.\s*\$(?!wpdb->)(?!\w*(?:format|placeholder))[A-Za-z_] +- id: ousast.php.regex.php-sql-unquoted-interpolation + languages: + - php + severity: WARNING + message: PHP SQL string interpolates a variable into an unquoted position + metadata: + family: injection + cwe: CWE-89 + origin: php-sql-unquoted-interpolation + precision_estimate: '0.129' + kind: regex + tags: + - injection + status: enabled + version: '1' + pattern-regex: (?])(?=(VALUES|IN|ORDER|GROUP|LIMIT|OFFSET|WHERE|AND|OR|SET|HAVING)\b)(?:(?:VALUES|IN)\s*\(\s*|(?:ORDER|GROUP)\s+BY\b[^"'\n;]*?|(?:LIMIT|OFFSET)\s+(?:\d+\s*,\s*)?|(?:WHERE|AND|OR|SET|HAVING)\s+[\w.`]+\s*(?:=|<>|!=|<=|>=|<|>|LIKE)\s*)\{?\$(?!wpdb->)(?!\w*(?:format|placeholder))[A-Za-z_] +- id: ousast.php.regex.php-sql-sprintf-unescaped + languages: + - php + severity: WARNING + message: PHP sprintf-built SQL with an unescaped %s value + metadata: + family: injection + cwe: CWE-89 + origin: php-sql-sprintf-unescaped + precision_estimate: '0.109' + kind: regex + tags: + - injection + status: enabled + version: '1' + pattern-regex: (sprintf)\(\s*["']\s*(?:SELECT|INSERT|UPDATE|DELETE|REPLACE)\b(?=(?:(?!\);)[\s\S])*?'%s')(?=(?:(?![Ee]scape|quote\(|intval\(|absint\(|\(int\)|esc_sql\()[\s\S])*?\);) +- id: ousast.php.regex.php-command-injection + languages: + - php + severity: WARNING + message: PHP shell command built from a variable without escapeshellarg + metadata: + family: injection + cwe: CWE-78 + origin: php-command-injection + precision_estimate: '1.0' + kind: regex + tags: + - syscall_entry + status: enabled + version: '1' + pattern-regex: (?:$\\])(?:-])(include|include_once|require|require_once)\b(?!['"\]])\s*\(?[^\n;]*\$_(?:GET|POST|REQUEST|COOKIE)\b +- id: ousast.php.regex.php-path-input + languages: + - php + severity: WARNING + message: PHP filesystem call on a path from request input + metadata: + family: path + cwe: CWE-22 + origin: php-path-input + precision_estimate: '1.0' + kind: regex + tags: + - filesystem_entry + status: enabled + version: '1' + pattern-regex: (?m)^(?![^\n]*\b(?:basename|realpath)\s*\()[^\n]*?(?:$\\])(fopen|file_get_contents|file_put_contents|readfile|unlink|copy|rename|scandir|opendir|mkdir|rmdir|move_uploaded_file)\s*\([^\n;]*\$_(?:GET|POST|REQUEST|COOKIE)\b +- id: ousast.php.regex.php-ssrf-input + languages: + - php + severity: WARNING + message: PHP outbound request to a URL from request input + metadata: + family: untrusted_destination + cwe: CWE-918 + origin: php-ssrf-input + precision_estimate: '1.0' + kind: regex + tags: + - network_entry + status: enabled + version: '1' + pattern-regex: (?:$\\])(curl_init|curl_setopt|fsockopen|stream_socket_client|get_headers)\s*\([^\n;]*\$_(?:GET|POST|REQUEST|COOKIE)\b +- id: ousast.php.regex.php-open-redirect + languages: + - php + severity: WARNING + message: PHP Location header from request input + metadata: + family: untrusted_destination + cwe: CWE-601 + origin: php-open-redirect + precision_estimate: '1.0' + kind: regex + tags: + - injection + status: enabled + version: '1' + pattern-regex: (?:$\\])(header)\s*\(\s*["']Location:[^\n;]*(?:\$_(?:GET|POST|REQUEST|COOKIE)\b|HTTP_REFERER) +- id: ousast.php.taint.code_eval + mode: taint + languages: &id001 + - php + severity: ERROR + message: 'code_eval: untrusted input reaches assert, create_function, eval' + metadata: + family: injection + cwe: CWE-95 + origin: sink:php:code_eval + precision_estimate: unmeasured + kind: taint + calls: + - assert + - create_function + - eval + pattern-sources: + - pattern-either: &id002 + - pattern: $_GET + - pattern: $_GET[...] + - pattern: $_POST + - pattern: $_POST[...] + - pattern: $_REQUEST + - pattern: $_REQUEST[...] + - pattern: $_COOKIE + - pattern: $_COOKIE[...] + - pattern: $_FILES + - pattern: $_FILES[...] + - pattern: $_SERVER + - pattern: $_SERVER[...] + - pattern: file_get_contents("php://input") + pattern-sinks: + - pattern-either: + - pattern: assert(...) + - pattern: create_function(...) + - pattern: eval(...) + pattern-sanitizers: + - pattern-either: + - &id003 + pattern: htmlspecialchars(...) + - &id004 + pattern: htmlentities(...) + - &id005 + pattern: html_entity_decode_safe(...) + - &id006 + pattern: mysqli_real_escape_string(...) + - &id007 + pattern: mysql_real_escape_string(...) + - &id008 + pattern: pg_escape_string(...) + - &id009 + pattern: esc_sql(...) + - &id010 + pattern: pg_escape_literal(...) + - &id011 + pattern: $wpdb->prepare(...) + - &id012 + pattern: escapeshellarg(...) + - &id013 + pattern: escapeshellcmd(...) + - &id014 + pattern: basename(...) + - &id015 + pattern: realpath(...) + - &id016 + pattern: intval(...) + - &id017 + pattern: floatval(...) + - &id018 + pattern: filter_var(...) + - &id019 + pattern: settype(...) + - &id020 + pattern: absint(...) +- id: ousast.php.taint.command + mode: taint + languages: *id001 + severity: ERROR + message: 'command: untrusted input reaches exec, passthru, pcntl_exec, popen, proc_open, shell_exec, system' + metadata: + family: injection + cwe: CWE-78 + origin: sink:php:command + precision_estimate: unmeasured + kind: taint + calls: + - exec + - passthru + - pcntl_exec + - popen + - proc_open + - shell_exec + - system + pattern-sources: + - pattern-either: *id002 + pattern-sinks: + - pattern-either: + - pattern: exec(...) + - pattern: passthru(...) + - pattern: pcntl_exec(...) + - pattern: popen(...) + - pattern: proc_open(...) + - pattern: shell_exec(...) + - pattern: system(...) + pattern-sanitizers: + - pattern-either: + - *id003 + - *id004 + - *id005 + - *id006 + - *id007 + - *id008 + - *id009 + - *id010 + - *id011 + - *id012 + - *id013 + - *id014 + - *id015 + - *id016 + - *id017 + - *id018 + - *id019 + - *id020 +- id: ousast.php.taint.echo + mode: taint + languages: *id001 + severity: ERROR + message: 'echo: untrusted input reaches echo, fwrite, print, print_r, printf, var_dump, vprintf' + metadata: + family: output_encoding + cwe: CWE-79 + origin: sink:php:echo + precision_estimate: unmeasured + kind: taint + calls: + - echo + - fwrite + - print + - print_r + - printf + - var_dump + - vprintf + pattern-sources: + - pattern-either: *id002 + pattern-sinks: + - pattern-either: + - pattern: echo $VALUE; + - pattern: fwrite(...) + - pattern: print $VALUE; + - pattern: print_r(...) + - pattern: printf(...) + - pattern: var_dump(...) + - pattern: vprintf(...) + pattern-sanitizers: + - pattern-either: + - *id003 + - *id004 + - *id005 + - *id006 + - *id007 + - *id008 + - *id009 + - *id010 + - *id011 + - *id012 + - *id013 + - *id014 + - *id015 + - *id016 + - *id017 + - *id018 + - *id019 + - *id020 +- id: ousast.php.taint.file_inclusion + mode: taint + languages: *id001 + severity: ERROR + message: 'file_inclusion: untrusted input reaches include, include_once, require, require_once' + metadata: + family: path + cwe: CWE-98 + origin: sink:php:file_inclusion + precision_estimate: unmeasured + kind: taint + calls: + - include + - include_once + - require + - require_once + pattern-sources: + - pattern-either: *id002 + pattern-sinks: + - pattern-either: + - pattern: include ... + - pattern: include_once ... + - pattern: require ... + - pattern: require_once ... + pattern-sanitizers: + - pattern-either: + - *id003 + - *id004 + - *id005 + - *id006 + - *id007 + - *id008 + - *id009 + - *id010 + - *id011 + - *id012 + - *id013 + - *id014 + - *id015 + - *id016 + - *id017 + - *id018 + - *id019 + - *id020 +- id: ousast.php.taint.filesystem + mode: taint + languages: *id001 + severity: ERROR + message: 'filesystem: untrusted input reaches copy, file_get_contents, file_put_contents, fopen, mkdir, move_uploaded_file, opendir, readfile, rename, rmdir, scandir, unlink' + metadata: + family: path + cwe: CWE-22 + origin: sink:php:filesystem + precision_estimate: unmeasured + kind: taint + calls: + - copy + - file_get_contents + - file_put_contents + - fopen + - mkdir + - move_uploaded_file + - opendir + - readfile + - rename + - rmdir + - scandir + - unlink + pattern-sources: + - pattern-either: *id002 + pattern-sinks: + - pattern-either: + - pattern: copy(...) + - pattern: file_get_contents(...) + - pattern: file_put_contents(...) + - pattern: fopen(...) + - pattern: mkdir(...) + - pattern: move_uploaded_file(...) + - pattern: opendir(...) + - pattern: readfile(...) + - pattern: rename(...) + - pattern: rmdir(...) + - pattern: scandir(...) + - pattern: unlink(...) + pattern-sanitizers: + - pattern-either: + - *id003 + - *id004 + - *id005 + - *id006 + - *id007 + - *id008 + - *id009 + - *id010 + - *id011 + - *id012 + - *id013 + - *id014 + - *id015 + - *id016 + - *id017 + - *id018 + - *id019 + - *id020 +- id: ousast.php.taint.outbound_request + mode: taint + languages: *id001 + severity: ERROR + message: 'outbound_request: untrusted input reaches curl_exec, curl_setopt, fsockopen, get_headers, stream_socket_client' + metadata: + family: untrusted_destination + cwe: CWE-918 + origin: sink:php:outbound_request + precision_estimate: unmeasured + kind: taint + calls: + - curl_exec + - curl_setopt + - fsockopen + - get_headers + - stream_socket_client + pattern-sources: + - pattern-either: *id002 + pattern-sinks: + - pattern-either: + - pattern: curl_exec(...) + - pattern: curl_setopt(...) + - pattern: fsockopen(...) + - pattern: get_headers(...) + - pattern: stream_socket_client(...) + pattern-sanitizers: + - pattern-either: + - *id003 + - *id004 + - *id005 + - *id006 + - *id007 + - *id008 + - *id009 + - *id010 + - *id011 + - *id012 + - *id013 + - *id014 + - *id015 + - *id016 + - *id017 + - *id018 + - *id019 + - *id020 +- id: ousast.php.taint.redirect + mode: taint + languages: *id001 + severity: ERROR + message: 'redirect: untrusted input reaches header' + metadata: + family: untrusted_destination + cwe: CWE-601 + origin: sink:php:redirect + precision_estimate: unmeasured + kind: taint + calls: + - header + pattern-sources: + - pattern-either: *id002 + pattern-sinks: + - pattern-either: + - pattern: header(...) + pattern-sanitizers: + - pattern-either: + - *id003 + - *id004 + - *id005 + - *id006 + - *id007 + - *id008 + - *id009 + - *id010 + - *id011 + - *id012 + - *id013 + - *id014 + - *id015 + - *id016 + - *id017 + - *id018 + - *id019 + - *id020 +- id: ousast.php.taint.sql + mode: taint + languages: *id001 + severity: ERROR + message: 'sql: untrusted input reaches mysql_query, mysqli_multi_query, mysqli_query, mysqli_real_query, pg_query, pg_send_query, sqlite_query, sqlite_unbuffered_query' + metadata: + family: injection + cwe: CWE-89 + origin: sink:php:sql + precision_estimate: unmeasured + kind: taint + calls: + - mysql_query + - mysqli_multi_query + - mysqli_query + - mysqli_real_query + - pg_query + - pg_send_query + - sqlite_query + - sqlite_unbuffered_query + pattern-sources: + - pattern-either: *id002 + pattern-sinks: + - pattern-either: + - pattern: mysql_query(...) + - pattern: mysqli_multi_query(...) + - pattern: mysqli_query(...) + - pattern: mysqli_real_query(...) + - pattern: pg_query(...) + - pattern: pg_send_query(...) + - pattern: sqlite_query(...) + - pattern: sqlite_unbuffered_query(...) + pattern-sanitizers: + - pattern-either: + - *id003 + - *id004 + - *id005 + - *id006 + - *id007 + - *id008 + - *id009 + - *id010 + - *id011 + - *id012 + - *id013 + - *id014 + - *id015 + - *id016 + - *id017 + - *id018 + - *id019 + - *id020 +- id: ousast.php.taint.unserialize + mode: taint + languages: *id001 + severity: ERROR + message: 'unserialize: untrusted input reaches maybe_unserialize, unserialize' + metadata: + family: deserialization + cwe: CWE-502 + origin: sink:php:unserialize + precision_estimate: unmeasured + kind: taint + calls: + - maybe_unserialize + - unserialize + pattern-sources: + - pattern-either: *id002 + pattern-sinks: + - pattern-either: + - pattern: maybe_unserialize(...) + - pattern: unserialize(...) + pattern-sanitizers: + - pattern-either: + - *id003 + - *id004 + - *id005 + - *id006 + - *id007 + - *id008 + - *id009 + - *id010 + - *id011 + - *id012 + - *id013 + - *id014 + - *id015 + - *id016 + - *id017 + - *id018 + - *id019 + - *id020 +- id: ousast.php.taint.wpdb + mode: taint + languages: *id001 + severity: ERROR + message: 'wpdb: untrusted input reaches $wpdb->get_col, $wpdb->get_results, $wpdb->get_row, $wpdb->get_var, $wpdb->query, $wpdb->replace' + metadata: + family: injection + cwe: CWE-89 + origin: sink:php:wpdb + precision_estimate: unmeasured + kind: taint + calls: + - $wpdb->get_col + - $wpdb->get_results + - $wpdb->get_row + - $wpdb->get_var + - $wpdb->query + - $wpdb->replace + pattern-sources: + - pattern-either: *id002 + pattern-sinks: + - pattern-either: + - pattern: $wpdb->get_col(...) + - pattern: $wpdb->get_results(...) + - pattern: $wpdb->get_row(...) + - pattern: $wpdb->get_var(...) + - pattern: $wpdb->query(...) + - pattern: $wpdb->replace(...) + pattern-sanitizers: + - pattern-either: + - *id003 + - *id004 + - *id005 + - *id006 + - *id007 + - *id008 + - *id009 + - *id010 + - *id011 + - *id012 + - *id013 + - *id014 + - *id015 + - *id016 + - *id017 + - *id018 + - *id019 + - *id020 diff --git a/benchmarks/measurements/2026-10-13-semgrep-arm/rules/python.yaml b/benchmarks/measurements/2026-10-13-semgrep-arm/rules/python.yaml new file mode 100644 index 0000000..4a442eb --- /dev/null +++ b/benchmarks/measurements/2026-10-13-semgrep-arm/rules/python.yaml @@ -0,0 +1,558 @@ +rules: +- id: ousast.python.regex.python-flask-debug + languages: + - python + severity: WARNING + message: Flask debug mode enabled needs review + metadata: + family: config_secrets + cwe: CWE-489 + origin: python-flask-debug + precision_estimate: '0.0' + kind: regex + tags: + - insecure_default + status: enabled + version: '1' + pattern-regex: \.run\s*\([^\n]*(debug)\s*=\s*True +- id: ousast.python.regex.python-os-command + languages: + - python + severity: WARNING + message: Python OS command execution needs review + metadata: + family: injection + cwe: CWE-78 + origin: python-os-command + precision_estimate: '0.0' + kind: regex + tags: + - syscall_entry + status: enabled + version: '1' + pattern-regex: \b(os\.system|os\.popen|os\.execl|os\.execlp|os\.execv|os\.execvp)\s*\( +- id: ousast.python.regex.python-path-traversal + languages: + - python + severity: WARNING + message: Python file open with external input needs review + metadata: + family: path + cwe: CWE-22 + origin: python-path-traversal + precision_estimate: '0.0' + kind: regex + tags: + - filesystem_entry + status: enabled + version: '1' + pattern-regex: \b(open)\s*\(\s*[^\n]*(?:request\.|req\.|argv|input\s*\(|params\[|GET\[|POST\[) +- id: ousast.python.regex.python-shell-true + languages: + - python + severity: WARNING + message: Subprocess shell execution needs review + metadata: + family: injection + cwe: CWE-78 + origin: python-shell-true + precision_estimate: '0.0' + kind: regex + tags: + - syscall_entry + status: enabled + version: '1' + pattern-regex: subprocess\.[a-zA-Z_]+\s*\([^\n]*shell\s*=\s*True +- id: ousast.python.regex.python-sql-injection + languages: + - python + severity: WARNING + message: Python SQL string composition needs review + metadata: + family: injection + cwe: CWE-89 + origin: python-sql-injection + precision_estimate: '0.0' + kind: regex + tags: + - injection + status: enabled + version: '1' + pattern-regex: (\.execute)\s*\(\s*[^\n]*(?:[\"']\s*%\s*[(A-Za-z_]|\.format\s*\(|[\"']\s*\+|\bf[\"']) +- id: ousast.python.regex.python-ssrf + languages: + - python + severity: WARNING + message: Python outbound request with external input needs review + metadata: + family: untrusted_destination + cwe: CWE-918 + origin: python-ssrf + precision_estimate: '0.0' + kind: regex + tags: + - network_entry + status: enabled + version: '1' + pattern-regex: \b(requests\.(?:get|post|put|delete|head|patch)|urllib\.request\.urlopen|urlopen)\s*\(\s*[^\n]*(?:request\.|req\.|argv|input\s*\() +- id: ousast.python.regex.python-ssti + languages: + - python + severity: WARNING + message: Python server-side template injection needs review + metadata: + family: injection + cwe: CWE-94 + origin: python-ssti + precision_estimate: '0.0' + kind: regex + tags: + - injection + status: enabled + version: '1' + pattern-regex: \b(render_template_string)\s*\( +- id: ousast.python.regex.python-unsafe-deserialization + languages: + - python + severity: WARNING + message: Unsafe Python deserialization needs review + metadata: + family: deserialization + cwe: CWE-502 + origin: python-unsafe-deserialization + precision_estimate: '0.0' + kind: regex + tags: + - deserialization + status: enabled + version: '1' + pattern-regex: \b(pickle\.loads|pickle\.load|cPickle\.loads|yaml\.load|marshal\.loads)\s*\( +- id: ousast.python.regex.python-unsafe-eval + languages: + - python + severity: WARNING + message: Dynamic Python execution needs review + metadata: + family: injection + cwe: CWE-95 + origin: python-unsafe-eval + precision_estimate: '0.0' + kind: regex + tags: + - syscall_entry + status: enabled + version: '1' + pattern-regex: \b(eval|exec)\s*\( +- id: ousast.python.regex.python-weak-hash + languages: + - python + severity: WARNING + message: Python weak hash needs review + metadata: + family: config_secrets + cwe: CWE-327 + origin: python-weak-hash + precision_estimate: '0.0' + kind: regex + tags: + - crypto + status: enabled + version: '1' + pattern-regex: \bhashlib\.(md5|sha1)\s*\( +- id: ousast.python.taint.eval + mode: taint + languages: &id001 + - python + severity: ERROR + message: 'eval: untrusted input reaches eval, exec' + metadata: + family: injection + cwe: CWE-95 + origin: sink:python:eval + precision_estimate: unmeasured + kind: taint + calls: + - eval + - exec + pattern-sources: + - pattern-either: &id002 + - pattern: request.args + - pattern: request.args(...) + - pattern: request.form + - pattern: request.form(...) + - pattern: request.data + - pattern: request.data(...) + - pattern: request.json + - pattern: request.json(...) + - pattern: request.cookies + - pattern: request.cookies(...) + - pattern: request.values + - pattern: request.values(...) + - pattern: request.get_json + - pattern: request.get_json(...) + - pattern: request.headers + - pattern: request.headers(...) + - pattern: request.files + - pattern: request.files(...) + - pattern: request.get + - pattern: request.get(...) + - pattern: request.GET + - pattern: request.GET(...) + - pattern: request.POST + - pattern: request.POST(...) + - pattern: request.body + - pattern: request.body(...) + - pattern: request.COOKIES + - pattern: request.COOKIES(...) + - pattern: request.FILES + - pattern: request.FILES(...) + - pattern: request.META + - pattern: request.META(...) + - pattern: request.query_params + - pattern: request.query_params(...) + - pattern: cleaned_data + - pattern: $OBJ.cleaned_data + - pattern: cleaned_data(...) + - pattern: $OBJ.cleaned_data(...) + - pattern: sys.argv + - pattern: sys.argv(...) + - pattern: sys.stdin + - pattern: sys.stdin(...) + - pattern: stdin.read + - pattern: stdin.read(...) + - pattern: input(...) + pattern-sinks: + - pattern-either: + - pattern: eval(...) + - pattern: $OBJ.eval(...) + - pattern: exec(...) + - pattern: $OBJ.exec(...) + pattern-sanitizers: + - pattern-either: + - &id003 + pattern: ast.literal_eval(...) + - &id004 + pattern: yaml.safe_load(...) + - &id005 + pattern: safe_load(...) + - &id006 + pattern: $OBJ.safe_load(...) +- id: ousast.python.taint.execute + mode: taint + languages: *id001 + severity: ERROR + message: 'execute: untrusted input reaches execute, executemany' + metadata: + family: injection + cwe: CWE-89 + origin: sink:python:execute + precision_estimate: unmeasured + kind: taint + calls: + - execute + - executemany + pattern-sources: + - pattern-either: *id002 + pattern-sinks: + - pattern-either: + - patterns: + - pattern: execute($QUERY) + - focus-metavariable: $QUERY + - patterns: + - pattern: $OBJ.execute($QUERY) + - focus-metavariable: $QUERY + - patterns: + - pattern: executemany($QUERY) + - focus-metavariable: $QUERY + - patterns: + - pattern: $OBJ.executemany($QUERY) + - focus-metavariable: $QUERY + pattern-sanitizers: + - pattern-either: + - *id003 + - *id004 + - *id005 + - *id006 +- id: ousast.python.weak.hashlib + languages: *id001 + severity: WARNING + message: 'hashlib: a weak literal reaches hashlib.md5, hashlib.new, hashlib.sha1' + metadata: + family: config_secrets + cwe: CWE-327 + origin: sink:python:hashlib + precision_estimate: unmeasured + kind: weak_literal + weak_literals: + - MD5 + - SHA1 + - md5 + - sha-1 + - sha1 + pattern-either: + - pattern: hashlib.md5(...) + - patterns: + - pattern: hashlib.new($ALGORITHM, ...) + - metavariable-regex: + metavariable: $ALGORITHM + regex: ^["']?(MD5|SHA1|md5|sha\-1|sha1)["']?$ + - pattern: hashlib.sha1(...) +- id: ousast.python.taint.html-unescaped + mode: taint + languages: *id001 + severity: ERROR + message: 'html.unescaped: untrusted input reaches HttpResponse, Markup, format_html, mark_safe, render_template_string' + metadata: + family: output_encoding + cwe: CWE-79 + origin: sink:python:html.unescaped + precision_estimate: unmeasured + kind: taint + calls: + - HttpResponse + - Markup + - format_html + - mark_safe + - render_template_string + pattern-sources: + - pattern-either: *id002 + pattern-sinks: + - pattern-either: + - pattern: HttpResponse(...) + - pattern: $OBJ.HttpResponse(...) + - pattern: Markup(...) + - pattern: $OBJ.Markup(...) + - pattern: format_html(...) + - pattern: $OBJ.format_html(...) + - pattern: mark_safe(...) + - pattern: $OBJ.mark_safe(...) + - pattern: render_template_string(...) + - pattern: $OBJ.render_template_string(...) + pattern-sanitizers: + - pattern-either: + - *id003 + - *id004 + - *id005 + - *id006 +- id: ousast.python.taint.orm-raw + mode: taint + languages: *id001 + severity: ERROR + message: 'orm.raw: untrusted input reaches RawSQL, extra, from_statement, objects.raw, raw, text' + metadata: + family: injection + cwe: CWE-89 + origin: sink:python:orm.raw + precision_estimate: unmeasured + kind: taint + calls: + - RawSQL + - extra + - from_statement + - objects.raw + - raw + - text + pattern-sources: + - pattern-either: *id002 + pattern-sinks: + - pattern-either: + - pattern: RawSQL(...) + - pattern: $OBJ.RawSQL(...) + - pattern: extra(...) + - pattern: $OBJ.extra(...) + - pattern: from_statement(...) + - pattern: $OBJ.from_statement(...) + - pattern: objects.raw(...) + - pattern: raw(...) + - pattern: $OBJ.raw(...) + - pattern: text(...) + - pattern: $OBJ.text(...) + pattern-sanitizers: + - pattern-either: + - *id003 + - *id004 + - *id005 + - *id006 +- id: ousast.python.taint.os-system + mode: taint + languages: *id001 + severity: ERROR + message: 'os.system: untrusted input reaches os.popen, os.system, subprocess.Popen, subprocess.call, subprocess.check_output, subprocess.run' + metadata: + family: injection + cwe: CWE-78 + origin: sink:python:os.system + precision_estimate: unmeasured + kind: taint + calls: + - os.popen + - os.system + - subprocess.Popen + - subprocess.call + - subprocess.check_output + - subprocess.run + pattern-sources: + - pattern-either: *id002 + pattern-sinks: + - pattern-either: + - pattern: os.popen(...) + - pattern: os.system(...) + - pattern: subprocess.Popen(...) + - pattern: subprocess.call(...) + - pattern: subprocess.check_output(...) + - pattern: subprocess.run(...) + pattern-sanitizers: + - pattern-either: + - *id003 + - *id004 + - *id005 + - *id006 +- id: ousast.python.taint.outbound-request + mode: taint + languages: *id001 + severity: ERROR + message: 'outbound.request: untrusted input reaches httpx.get, httpx.post, requests.get, requests.post, requests.request, urlopen, urlretrieve' + metadata: + family: untrusted_destination + cwe: CWE-918 + origin: sink:python:outbound.request + precision_estimate: unmeasured + kind: taint + calls: + - httpx.get + - httpx.post + - requests.get + - requests.post + - requests.request + - urlopen + - urlretrieve + pattern-sources: + - pattern-either: *id002 + pattern-sinks: + - pattern-either: + - pattern: httpx.get(...) + - pattern: httpx.post(...) + - pattern: requests.get(...) + - pattern: requests.post(...) + - pattern: requests.request(...) + - pattern: urlopen(...) + - pattern: $OBJ.urlopen(...) + - pattern: urlretrieve(...) + - pattern: $OBJ.urlretrieve(...) + pattern-sanitizers: + - pattern-either: + - *id003 + - *id004 + - *id005 + - *id006 +- id: ousast.python.taint.path-open + mode: taint + languages: *id001 + severity: ERROR + message: 'path.open: untrusted input reaches FileResponse, open, os.remove, os.unlink, send_file, send_from_directory, shutil.copy, shutil.move, shutil.rmtree' + metadata: + family: path + cwe: CWE-22 + origin: sink:python:path.open + precision_estimate: unmeasured + kind: taint + calls: + - FileResponse + - open + - os.remove + - os.unlink + - send_file + - send_from_directory + - shutil.copy + - shutil.move + - shutil.rmtree + pattern-sources: + - pattern-either: *id002 + pattern-sinks: + - pattern-either: + - pattern: FileResponse(...) + - pattern: $OBJ.FileResponse(...) + - pattern: open(...) + - pattern: $OBJ.open(...) + - pattern: os.remove(...) + - pattern: os.unlink(...) + - pattern: send_file(...) + - pattern: $OBJ.send_file(...) + - pattern: send_from_directory(...) + - pattern: $OBJ.send_from_directory(...) + - pattern: shutil.copy(...) + - pattern: shutil.move(...) + - pattern: shutil.rmtree(...) + pattern-sanitizers: + - pattern-either: + - *id003 + - *id004 + - *id005 + - *id006 +- id: ousast.python.taint.pickle-loads + mode: taint + languages: *id001 + severity: ERROR + message: 'pickle.loads: untrusted input reaches cPickle.loads, marshal.loads, pickle.load, pickle.loads, yaml.load' + metadata: + family: deserialization + cwe: CWE-502 + origin: sink:python:pickle.loads + precision_estimate: unmeasured + kind: taint + calls: + - cPickle.loads + - marshal.loads + - pickle.load + - pickle.loads + - yaml.load + pattern-sources: + - pattern-either: *id002 + pattern-sinks: + - pattern-either: + - pattern: cPickle.loads(...) + - pattern: marshal.loads(...) + - pattern: pickle.load(...) + - pattern: pickle.loads(...) + - pattern: yaml.load(...) + pattern-sanitizers: + - pattern-either: + - *id003 + - *id004 + - *id005 + - *id006 +- id: ousast.python.taint.redirect + mode: taint + languages: *id001 + severity: ERROR + message: 'redirect: untrusted input reaches HttpResponseRedirect, redirect, url_for' + metadata: + family: untrusted_destination + cwe: CWE-601 + origin: sink:python:redirect + precision_estimate: unmeasured + kind: taint + calls: + - HttpResponseRedirect + - redirect + - url_for + pattern-sources: + - pattern-either: *id002 + pattern-sinks: + - pattern-either: + - pattern: HttpResponseRedirect(...) + - pattern: $OBJ.HttpResponseRedirect(...) + - pattern: redirect(...) + - pattern: $OBJ.redirect(...) + - pattern: url_for(...) + - pattern: $OBJ.url_for(...) + pattern-sanitizers: + - pattern-either: + - *id003 + - *id004 + - *id005 + - *id006 + - pattern: reverse(...) + - pattern: $OBJ.reverse(...) + - pattern: reverse_lazy(...) + - pattern: $OBJ.reverse_lazy(...) diff --git a/benchmarks/measurements/2026-10-13-semgrep-arm/scored_semgrep.json b/benchmarks/measurements/2026-10-13-semgrep-arm/scored_semgrep.json new file mode 100644 index 0000000..f61ddc9 --- /dev/null +++ b/benchmarks/measurements/2026-10-13-semgrep-arm/scored_semgrep.json @@ -0,0 +1,500 @@ +{ + "alerts_only": { + "count": 30, + "n": 239, + "rate": 0.12552301255230125, + "wilson95": [ + 0.08935736205295211, + 0.17353663963374408 + ] + }, + "breakdowns": { + "coverage": { + "engine_covered": { + "alerts_only": { + "count": 0, + "n": 0, + "rate": null, + "wilson95": [ + null, + null + ] + }, + "catch": { + "count": 0, + "n": 0, + "rate": null, + "wilson95": [ + null, + null + ] + }, + "false_alarm": { + "cluster95": [ + null, + null + ], + "count": 0, + "icc": null, + "n": 0, + "rate": null, + "wilson95": [ + null, + null + ] + } + }, + "quick_only": { + "alerts_only": { + "count": 30, + "n": 239, + "rate": 0.12552301255230125, + "wilson95": [ + 0.08935736205295211, + 0.17353663963374408 + ] + }, + "catch": { + "count": 0, + "n": 12, + "rate": 0.0, + "wilson95": [ + 0, + 0.2425007574992425 + ] + }, + "false_alarm": { + "cluster95": [ + 0.0375, + 0.23430962343096234 + ], + "count": 30, + "icc": 0.27735846730536884, + "n": 239, + "rate": 0.12552301255230125, + "wilson95": [ + 0.08935736205295211, + 0.17353663963374408 + ] + } + }, + "unanalysable": { + "alerts_only": { + "count": 0, + "n": 0, + "rate": null, + "wilson95": [ + null, + null + ] + }, + "catch": { + "count": 0, + "n": 0, + "rate": null, + "wilson95": [ + null, + null + ] + }, + "false_alarm": { + "cluster95": [ + null, + null + ], + "count": 0, + "icc": null, + "n": 0, + "rate": null, + "wilson95": [ + null, + null + ] + } + } + }, + "family": { + "access_control": { + "alerts_only": { + "count": 0, + "n": 0, + "rate": null, + "wilson95": [ + null, + null + ] + }, + "catch": { + "count": 0, + "n": 3, + "rate": 0.0, + "wilson95": [ + 0, + 0.5615060804490177 + ] + }, + "false_alarm": { + "cluster95": [ + null, + null + ], + "count": 0, + "icc": null, + "n": 0, + "rate": null, + "wilson95": [ + null, + null + ] + } + }, + "output_encoding": { + "alerts_only": { + "count": 0, + "n": 0, + "rate": null, + "wilson95": [ + null, + null + ] + }, + "catch": { + "count": 0, + "n": 4, + "rate": 0.0, + "wilson95": [ + 0, + 0.48990002040399916 + ] + }, + "false_alarm": { + "cluster95": [ + null, + null + ], + "count": 0, + "icc": null, + "n": 0, + "rate": null, + "wilson95": [ + null, + null + ] + } + }, + "path": { + "alerts_only": { + "count": 0, + "n": 0, + "rate": null, + "wilson95": [ + null, + null + ] + }, + "catch": { + "count": 0, + "n": 3, + "rate": 0.0, + "wilson95": [ + 0, + 0.5615060804490177 + ] + }, + "false_alarm": { + "cluster95": [ + null, + null + ], + "count": 0, + "icc": null, + "n": 0, + "rate": null, + "wilson95": [ + null, + null + ] + } + }, + "unknown": { + "alerts_only": { + "count": 30, + "n": 239, + "rate": 0.12552301255230125, + "wilson95": [ + 0.08935736205295211, + 0.17353663963374408 + ] + }, + "catch": { + "count": 0, + "n": 0, + "rate": null, + "wilson95": [ + null, + null + ] + }, + "false_alarm": { + "cluster95": [ + 0.0375, + 0.23430962343096234 + ], + "count": 30, + "icc": 0.27735846730536884, + "n": 239, + "rate": 0.12552301255230125, + "wilson95": [ + 0.08935736205295211, + 0.17353663963374408 + ] + } + }, + "untrusted_destination": { + "alerts_only": { + "count": 0, + "n": 0, + "rate": null, + "wilson95": [ + null, + null + ] + }, + "catch": { + "count": 0, + "n": 2, + "rate": 0.0, + "wilson95": [ + 0, + 0.6576280471103807 + ] + }, + "false_alarm": { + "cluster95": [ + null, + null + ], + "count": 0, + "icc": null, + "n": 0, + "rate": null, + "wilson95": [ + null, + null + ] + } + } + }, + "method": { + "first_parent": { + "alerts_only": { + "count": 30, + "n": 239, + "rate": 0.12552301255230125, + "wilson95": [ + 0.08935736205295211, + 0.17353663963374408 + ] + }, + "catch": { + "count": 0, + "n": 0, + "rate": null, + "wilson95": [ + null, + null + ] + }, + "false_alarm": { + "cluster95": [ + 0.0375, + 0.23430962343096234 + ], + "count": 30, + "icc": 0.27735846730536884, + "n": 239, + "rate": 0.12552301255230125, + "wilson95": [ + 0.08935736205295211, + 0.17353663963374408 + ] + } + }, + "fix_site": { + "alerts_only": { + "count": 0, + "n": 0, + "rate": null, + "wilson95": [ + null, + null + ] + }, + "catch": { + "count": 0, + "n": 12, + "rate": 0.0, + "wilson95": [ + 0, + 0.2425007574992425 + ] + }, + "false_alarm": { + "cluster95": [ + null, + null + ], + "count": 0, + "icc": null, + "n": 0, + "rate": null, + "wilson95": [ + null, + null + ] + } + } + }, + "post_cutoff": { + "unknown": { + "alerts_only": { + "count": 30, + "n": 239, + "rate": 0.12552301255230125, + "wilson95": [ + 0.08935736205295211, + 0.17353663963374408 + ] + }, + "catch": { + "count": 0, + "n": 12, + "rate": 0.0, + "wilson95": [ + 0, + 0.2425007574992425 + ] + }, + "false_alarm": { + "cluster95": [ + 0.0375, + 0.23430962343096234 + ], + "count": 30, + "icc": 0.27735846730536884, + "n": 239, + "rate": 0.12552301255230125, + "wilson95": [ + 0.08935736205295211, + 0.17353663963374408 + ] + } + } + }, + "stream": { + "advisory": { + "count": 0, + "n": 12, + "rate": 0.0, + "wilson95": [ + 0, + 0.2425007574992425 + ] + }, + "alerts": { + "count": 0, + "n": 12, + "rate": 0.0, + "wilson95": [ + 0, + 0.2425007574992425 + ] + }, + "quick": { + "count": 0, + "n": 12, + "rate": 0.0, + "wilson95": [ + 0, + 0.2425007574992425 + ] + } + } + }, + "catch": { + "count": 0, + "n": 12, + "rate": 0.0, + "wilson95": [ + 0, + 0.2425007574992425 + ] + }, + "confirmed": { + "alerts_only": { + "count": 0, + "n": 0, + "rate": null, + "wilson95": [ + null, + null + ] + }, + "catch": { + "count": 0, + "n": 12, + "rate": 0.0, + "wilson95": [ + 0, + 0.2425007574992425 + ] + }, + "false_alarm": { + "cluster95": [ + null, + null + ], + "count": 0, + "icc": null, + "n": 0, + "rate": null, + "wilson95": [ + null, + null + ] + } + }, + "coverage": { + "engine_covered": 0, + "instrument_failure": 1, + "quick_only": 251, + "unanalysable": 0 + }, + "deadlines": [ + { + "applied_deadline_scale": null, + "count": 252, + "deadline_scale": null, + "deadline_seconds": null, + "lane": "unknown" + } + ], + "false_alarm": { + "cluster95": [ + 0.0375, + 0.23430962343096234 + ], + "count": 30, + "icc": 0.27735846730536884, + "n": 239, + "rate": 0.12552301255230125, + "wilson95": [ + 0.08935736205295211, + 0.17353663963374408 + ] + }, + "total": 252, + "unseen": { + "gate": "not_run" + } +} diff --git a/benchmarks/valid/semgrep_arm.py b/benchmarks/valid/semgrep_arm.py new file mode 100644 index 0000000..8775b1e --- /dev/null +++ b/benchmarks/valid/semgrep_arm.py @@ -0,0 +1,634 @@ +"""Semgrep CE as a benchmark arm: our regex rules and taint facts converted, run per frozen change. + +Two subcommands. ``rules`` converts every enabled quick rule to a ``pattern-regex`` rule and every +source/sink/sanitizer fact triple to a ``mode: taint`` rule, one YAML file per language, and writes a +conversion report naming what was skipped and why. ``run`` checks out each frozen change's head into a +temporary worktree, runs Semgrep over the changed files only, proves the targets were read (bytes per +file), and writes one scorer-shaped record per change, resumable exactly like ``hunter_arm``. + +Nothing here ships: the rules are ours, the engine is a measurement instrument. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import re +import shutil +import subprocess +import sys +import tempfile +import time +import tomllib +from pathlib import Path + +import yaml + +from openultrasast.hunter_tools import clamp_repo_path +from openultrasast.model.taxonomy import load_families +from openultrasast.ruleset import DEFAULT_RULESET_DIR, load_ruleset +from openultrasast.semantic.facts import DEFAULT_FACTS_DIR, load_facts + +if __package__ in {None, ""}: + sys.path.insert(0, str(Path(__file__).resolve().parents[2])) + +from benchmarks.unseen.extract import SHA, InstrumentFailure +from benchmarks.unseen.replay import canonical, load_slice +from benchmarks.valid.catalog import local_git + +# Our language ids to Semgrep's. Groovy has no Semgrep parser: `generic` plus a path filter. +SEMGREP_LANGUAGES = { + "c": "c", + "cpp": "cpp", + "python": "python", + "javascript": "javascript", + "typescript": "typescript", + "java": "java", + "php": "php", +} +GENERIC_PATHS = {"groovy": ["*.groovy", "*.tpl"]} +# The rule file a rule lands in: one per language family. +FILE_OF_LANGUAGE = {"cpp": "c", "typescript": "javascript"} +# Fact languages whose Semgrep rules need a second dialect. +FACT_DIALECTS = {"c": ["c", "cpp"], "javascript": ["javascript", "typescript"]} +# PHP language constructs are not calls: `echo(...)` would parse as a call expression, and Semgrep's PHP +# pattern parser takes `echo` only as a whole statement. +PHP_CONSTRUCTS = { + "echo": "echo $VALUE;", + "print": "print $VALUE;", + "include": "include ...", + "include_once": "include_once ...", + "require": "require ...", + "require_once": "require_once ...", +} +# DOM properties assigned, not called. +ASSIGNED_SINKS = {"innerHTML", "outerHTML"} +SEMGREP_FLAGS = ( + "--json", + "--metrics=off", + "--disable-version-check", + "--no-git-ignore", + "--timeout", + "60", + "--max-memory", + "1500", + "-j", + "1", +) +# A Semgrep launch that returns faster than its own startup never ran the engine. +MIN_PLAUSIBLE_SECONDS = 1.0 +# PCRE2 compiles a bounded repeat of a group by copying it: `(?:...){0,2000}?` is "too large". The lazy +# unbounded form keeps the match and widens the window from 2000 characters to the statement. +BOUNDED_GROUP_REPEAT = re.compile(r"\{0,\d+\}\?") + + +def _family_of(cwe): + family = load_families().family_of_cwe(cwe or "") + return family.id if family else "unknown" + + +def _ident(value): + return re.sub(r"[^A-Za-z0-9_-]+", "-", value).strip("-") + + +# --- regex rules ------------------------------------------------------------------------------------------- + + +def _raw_rules(ruleset_dir): + rows = [] + for path in sorted(Path(ruleset_dir).rglob("*.toml")): + if path.parent.name in {"semantic", "obligations"} or path.parent == Path(ruleset_dir): + continue + for item in tomllib.loads(path.read_text()).get("rule", []): + rows.append(item) + return rows + + +def convert_regex_rules(ruleset_dir=DEFAULT_RULESET_DIR, *, min_precision=0.0): + """Every enabled quick rule as a Semgrep ``pattern-regex`` rule; the rest named with a reason.""" + active = {rule.rule_id: rule for rule in load_ruleset(Path(ruleset_dir))} + converted, skipped, rewrites = {}, [], [] + for item in _raw_rules(ruleset_dir): + rule_id = str(item["rule_id"]) + rule = active.get(rule_id) + if rule is None: + skipped.append({"origin": rule_id, "reason": f"fallback for {item.get('fallback_for')}: the tagged rule is active"}) + continue + if rule.status == "disabled": + skipped.append({"origin": rule_id, "reason": "disabled"}) + continue + if rule.status == "shadow": + skipped.append( + {"origin": rule_id, "reason": f"shadow status (precision_estimate {rule.precision_estimate}); not admitted by the hook"} + ) + continue + if rule.precision_estimate < min_precision: + skipped.append({"origin": rule_id, "reason": f"emit floor: precision_estimate {rule.precision_estimate} < {min_precision}"}) + continue + if not rule.languages: + skipped.append({"origin": rule_id, "reason": "non-code file targets: the rule applies to every file"}) + continue + languages = [SEMGREP_LANGUAGES[language] for language in rule.languages if language in SEMGREP_LANGUAGES] + generic = [language for language in rule.languages if language in GENERIC_PATHS] + unsupported = [language for language in rule.languages if language not in SEMGREP_LANGUAGES and language not in GENERIC_PATHS] + if unsupported: + skipped.append({"origin": rule_id, "reason": f"no Semgrep language for {unsupported}"}) + continue + family = _family_of(rule.cwe) + pattern = BOUNDED_GROUP_REPEAT.sub("*?", rule.pattern) + if pattern != rule.pattern: + rewrites.append({"origin": rule_id, "rewrite": "bounded lazy group repeat {0,N}? widened to *? (PCRE2 size limit)"}) + entry = { + "id": f"ousast.{FILE_OF_LANGUAGE.get(rule.languages[0], rule.languages[0])}.regex.{_ident(rule_id)}", + "languages": languages or ["generic"], + "severity": "WARNING", + "message": rule.title, + "metadata": { + "family": family, + "cwe": rule.cwe, + "origin": rule_id, + "precision_estimate": str(rule.precision_estimate), + "kind": "regex", + "tags": list(rule.tags), + "status": rule.status, + "version": rule.version, + }, + "pattern-regex": pattern, + } + if not languages: + entry["paths"] = {"include": [glob for language in generic for glob in GENERIC_PATHS[language]]} + key = FILE_OF_LANGUAGE.get(rule.languages[0], rule.languages[0]) + converted.setdefault(key, []).append(entry) + return converted, skipped, rewrites + + +# --- taint facts ------------------------------------------------------------------------------------------- + + +def _call_patterns(call, language): + """The Semgrep call shapes of one fact token (a function, a method, a construct).""" + if language == "php": + if call in PHP_CONSTRUCTS: + return [PHP_CONSTRUCTS[call]] + return [f"{call}(...)"] + if "." in call or "->" in call: + return [f"{call}(...)"] + if language == "java" and call[:1].isupper(): + return [f"new {call}(...)"] + return [f"{call}(...)", f"$OBJ.{call}(...)"] + + +def _source_patterns(token, language): + if language == "php": + if token == "php://input": + return ['file_get_contents("php://input")'] + return [token, f"{token}[...]"] + if token.endswith("("): + return [f"{token}...)"] + if "." in token: + return [token, f"{token}(...)"] + if language == "c": + return [token, f"{token}(...)"] + return [token, f"$OBJ.{token}", f"{token}(...)", f"$OBJ.{token}(...)"] + + +def _sink_entries(call, language, *, format_arg=None, single_argument=False): + """Sink patterns for one call; a focused argument where the fact names one.""" + if call in ASSIGNED_SINKS: + return [{"patterns": [{"pattern": f"$OBJ.{call} = $VALUE"}, {"focus-metavariable": "$VALUE"}]}] + if call == "dangerouslySetInnerHTML": + return [{"patterns": [{"pattern": "{__html: $VALUE}"}, {"focus-metavariable": "$VALUE"}]}] + if format_arg is not None: + arguments = ", ".join([*(f"$ARG{index}" for index in range(format_arg)), "$FORMAT", "..."]) + heads = [call] if "." in call or language == "c" else [call, f"$OBJ.{call}"] + return [{"patterns": [{"pattern": f"{head}({arguments})"}, {"focus-metavariable": "$FORMAT"}]} for head in heads] + if single_argument: + heads = [call] if "." in call or "->" in call else [call, f"$OBJ.{call}"] + return [{"patterns": [{"pattern": f"{head}($QUERY)"}, {"focus-metavariable": "$QUERY"}]} for head in heads] + return [{"pattern": pattern} for pattern in _call_patterns(call, language)] + + +def _weak_literal_rule(language, sink_id, cwe, calls, literals, dialects): + """A `weak_literals` sink is a search, not a flow: the call with a named weak literal, or a call named after one.""" + alternatives = [] + lowered = {literal.lower() for literal in literals} + for call in sorted(calls): + for head in [call] if "." in call else [call, f"$OBJ.{call}"]: + if call.rsplit(".", 1)[-1].lower() in lowered: + alternatives.append({"pattern": f"{head}(...)"}) + else: + alternatives.append( + { + "patterns": [ + {"pattern": f"{head}($ALGORITHM, ...)"}, + { + "metavariable-regex": { + "metavariable": "$ALGORITHM", + "regex": "^[\"']?(" + "|".join(re.escape(literal) for literal in sorted(literals)) + ")[\"']?$", + } + }, + ] + } + ) + return { + "id": f"ousast.{language}.weak.{_ident(sink_id)}", + "languages": dialects, + "severity": "WARNING", + "message": f"{sink_id}: a weak literal reaches {', '.join(sorted(calls))}", + "metadata": { + "family": _family_of(cwe), + "cwe": cwe, + "origin": f"sink:{language}:{sink_id}", + "precision_estimate": "unmeasured", + "kind": "weak_literal", + "weak_literals": sorted(literals), + }, + "pattern-either": alternatives, + } + + +def convert_facts(facts_dir=DEFAULT_FACTS_DIR): + """One taint rule per (language, sink id, CWE); the facts Semgrep has no word for are named, not dropped silently.""" + facts = load_facts(Path(facts_dir)) + converted, skipped, lossy = {}, [], [] + for language in sorted({fact.language for fact in facts.sinks}): + scoped = facts.for_language(language) + dialects = FACT_DIALECTS.get(language, [SEMGREP_LANGUAGES[language]]) + sources = [ + {"pattern": pattern} for fact in scoped.sources for token in fact.patterns for pattern in _source_patterns(token, language) + ] + parameterized = set() + sanitizers = [] + for fact in scoped.sanitizers: + origin = f"sanitizer:{language}:{fact.id}" + if fact.guard: + skipped.append({"origin": origin, "reason": "guard: a check whose safety depends on dominance; Semgrep has no equivalent"}) + continue + if fact.parameterized: + parameterized.update(fact.calls) + skipped.append( + {"origin": origin, "reason": "parameterized: a call shape, not a cleansing; expressed as the single-argument sink form"} + ) + continue + if fact.literal_format_arg is not None: + skipped.append( + { + "origin": origin, + "reason": "literal_format_arg: a shape; a literal format is never tainted at the focused sink argument", + } + ) + continue + if fact.quoted_only: + loss = {"origin": origin, "loss": "quoted_only escape treated as a full sanitizer (silences the CVE-2024-1071 shape)"} + if loss not in lossy: + lossy.append(loss) + sanitizers.extend({"pattern": pattern} for call in fact.calls for pattern in _call_patterns(call, language)) + for fact in scoped.dispatches: + skipped.append( + { + "origin": f"dispatch:{language}:{fact.id}", + "reason": "dispatch: string-keyed hook registry routing; no Semgrep equivalent", + } + ) + for fact in scoped.layouts: + skipped.append({"origin": f"layout:{language}:{fact.id}", "reason": "layout: scope policy (vendored, tests), not a rule"}) + grouped = {} + for fact in scoped.sinks: + grouped.setdefault((fact.id, fact.cwe), []).append(fact) + for (sink_id, cwe), rows in sorted(grouped.items()): + origin = f"sink:{language}:{sink_id}" + calls = sorted({call for row in rows for call in row.calls}) + literals = {literal for row in rows for literal in row.weak_literals} + if literals: + converted.setdefault(language, []).append(_weak_literal_rule(language, sink_id, cwe, calls, literals, dialects)) + continue + entries, extra_sanitizers = [], [] + format_arg = next((row.format_arg for row in rows if row.format_arg is not None), None) + for call in calls: + if call == "v-html": + skipped.append({"origin": f"{origin}:{call}", "reason": "template directive, not a call"}) + continue + entries.extend(_sink_entries(call, language, format_arg=format_arg, single_argument=call in parameterized)) + for row in rows: + if row.prefix_fixes_origin: + lossy.append( + {"origin": origin, "loss": "prefix_fixes_origin: a value appended after a fixed origin still counts as tainted"} + ) + break + for anchor in sorted({anchor for row in rows for anchor in row.origin_anchors}): + extra_sanitizers.extend({"pattern": pattern} for pattern in _call_patterns(anchor, language)) + if not entries: + skipped.append({"origin": origin, "reason": "no call form survives conversion"}) + continue + rule = { + "id": f"ousast.{language}.taint.{_ident(sink_id)}", + "mode": "taint", + "languages": dialects, + "severity": "ERROR", + "message": f"{sink_id}: untrusted input reaches {', '.join(calls)}", + "metadata": { + "family": _family_of(cwe), + "cwe": cwe, + "origin": origin, + "precision_estimate": "unmeasured", + "kind": "taint", + "calls": calls, + }, + "pattern-sources": [{"pattern-either": sources}], + "pattern-sinks": [{"pattern-either": entries}], + } + if sanitizers or extra_sanitizers: + rule["pattern-sanitizers"] = [{"pattern-either": [*sanitizers, *extra_sanitizers]}] + converted.setdefault(language, []).append(rule) + return converted, skipped, lossy + + +def _obligation_tables(ruleset_dir): + count = 0 + for path in sorted((Path(ruleset_dir) / "obligations").glob("*.toml")): + payload = tomllib.loads(path.read_text()) + count += sum(len(value) for value in payload.values() if isinstance(value, list)) + return count + + +def write_rules(out, *, ruleset_dir=DEFAULT_RULESET_DIR, facts_dir=DEFAULT_FACTS_DIR, min_precision=0.0, semgrep=None): + """Write one YAML per language under ``out`` plus ``conversion_report.json``; validate with Semgrep when present.""" + out = Path(out) + out.mkdir(parents=True, exist_ok=True) + regex_rules, regex_skipped, rewrites = convert_regex_rules(ruleset_dir, min_precision=min_precision) + taint_rules, fact_skipped, lossy = convert_facts(facts_dir) + files = {} + for language in sorted(set(regex_rules) | set(taint_rules)): + rules = [*regex_rules.get(language, []), *taint_rules.get(language, [])] + path = out / f"{language}.yaml" + path.write_text(yaml.safe_dump({"rules": rules}, sort_keys=False, width=200, allow_unicode=True)) + files[path.name] = len(rules) + obligations = _obligation_tables(ruleset_dir) + report = { + "regex": { + "converted": sorted(rule["metadata"]["origin"] for rules in regex_rules.values() for rule in rules), + "skipped": regex_skipped, + "rewritten": rewrites, + }, + "facts": { + "converted": sorted(rule["metadata"]["origin"] for rules in taint_rules.values() for rule in rules), + "skipped": fact_skipped, + "lossy": lossy, + }, + "obligations": { + "skipped_tables": obligations, + "reason": "an obligation finds a missing guard (an absence), which no pattern expresses", + }, + "not_carried": [ + "comment skipping: the quick engine drops a regex match inside a comment; pattern-regex does not", + "framework priors and fallback rules: the tagged rule is emitted, its language-level fallback is not", + "min_evidence_level, ranking priority, the rung: post-match policy on our side", + ], + "files": files, + "validation": None, + } + binary = semgrep or shutil.which("semgrep") + if binary: + try: + completed = subprocess.run( + [binary, "--validate", "--config", str(out), "--metrics=off", "--disable-version-check"], capture_output=True, text=True + ) + except OSError as error: + report["validation"] = {"exit": None, "error": f"semgrep not found or not runnable: {type(error).__name__}"} + else: + report["validation"] = {"exit": completed.returncode, "stderr": completed.stderr[-4000:], "stdout": completed.stdout[-4000:]} + (out / "conversion_report.json").write_text(json.dumps(report, indent=2, sort_keys=True) + "\n") + return report + + +# --- run --------------------------------------------------------------------------------------------------- + + +def result_defect(result): + """One Semgrep result as a scorer defect; the family and CWE come from the rule's own metadata.""" + extra = result.get("extra", {}) + metadata = extra.get("metadata", {}) + path = result["path"] + line = result.get("start", {}).get("line", 0) + origin = metadata.get("origin", result["check_id"]) + return { + "defect_id": f"{origin}:{path}:{line}", + "family": metadata.get("family", "unknown"), + "cwe": metadata.get("cwe"), + "locations": [[path, line]], + "source": "semgrep", + "rule_id": result["check_id"], + "origin": origin, + "title": extra.get("message", ""), + "rationale": extra.get("message", ""), + "confidence": extra.get("severity"), + "dataflow": bool(extra.get("dataflow_trace")), + } + + +def _write_json(path, value): + path.parent.mkdir(parents=True, exist_ok=True) + temporary = path.with_suffix(".json.tmp") + temporary.write_text(json.dumps(value, sort_keys=True, indent=2) + "\n") + temporary.replace(path) + + +def _record_path(out, identity): + return out / (hashlib.sha256(identity.encode()).hexdigest() + ".json") + + +def _inputs(out): + path = out / "inputs.json" + entries = json.loads(path.read_text()) if path.exists() else [] + for key in (lambda entry: entry["item_id"], lambda entry: entry["change"]["id"]): + if len({key(entry) for entry in entries}) != len(entries): + raise ValueError(f"duplicate ids in {path}") + return entries + + +def run_semgrep(binary, rules, root, targets, *, timeout=900): + """Semgrep over explicit targets inside ``root``; paths in the output stay repository-relative.""" + argv = [binary, "scan", "--config", str(rules), *SEMGREP_FLAGS, *targets] + started = time.monotonic() + completed = subprocess.run(argv, cwd=root, capture_output=True, text=True, timeout=timeout) + return completed, time.monotonic() - started + + +def _scan_change(change, repos, rules, *, binary): + instrument = { + "hook_exit": 0, + "hook_bytes_read": 0, + "changed_bytes": {}, + "head_verified": False, + "base_verified": False, + "failure": None, + "suspect_fast": False, + "semgrep": {"exit": None, "wall_seconds": None, "scanned": [], "errors": [], "results": 0}, + } + record = { + "id": change["id"], + "done": True, + "status": "semgrep_covered", + "instrument": instrument, + "push": {"admission": {"defects": [], "dispositions": []}, "quick_tier": []}, + "skipped": [], + } + try: + if not all(SHA.fullmatch(change[side]) for side in ("base", "head")): + raise InstrumentFailure("invalid_revision") + git = local_git(Path(repos[change["repo"]]).expanduser().resolve()) + git.run("cat-file", "-e", change["base"] + "^{commit}") + instrument["base_verified"] = True + with tempfile.TemporaryDirectory(prefix="ousast-semgrep-") as temporary: + root = Path(temporary) / "head" + added = False + try: + git.run("-c", "core.hooksPath=/dev/null", "worktree", "add", "--detach", str(root), change["head"]) + added = True + if local_git(root).run("rev-parse", "HEAD").strip() != change["head"]: + raise InstrumentFailure("head_mismatch") + instrument["head_verified"] = True + targets = [] + for relative in dict.fromkeys(file["path"] for file in change.get("files", [])): + path = clamp_repo_path(root, relative) + try: + path.stat() + except FileNotFoundError: + record["skipped"].append({"path": relative, "reason": "missing_at_head"}) + print(f"{change['id']} {relative}: skipped (missing_at_head)", flush=True) + continue + try: + data = path.read_bytes() + except OSError: + print(f"{change['id']} {relative}: 0 bytes read", flush=True) + raise InstrumentFailure(f"unreadable_target:{relative}") from None + size = len(data) + print(f"{change['id']} {relative}: {size} bytes read", flush=True) + if not size: + record["skipped"].append({"path": relative, "reason": "empty_at_head"}) + continue + instrument["changed_bytes"][relative] = size + instrument["hook_bytes_read"] += size + targets.append(relative) + if not targets: + raise InstrumentFailure("no_readable_targets") + try: + completed, wall = run_semgrep(binary, rules, root, targets) + except subprocess.TimeoutExpired: + raise InstrumentFailure("semgrep_timeout") from None + semgrep = instrument["semgrep"] + semgrep.update(exit=completed.returncode, wall_seconds=round(wall, 3)) + instrument["hook_exit"] = completed.returncode + if wall < MIN_PLAUSIBLE_SECONDS: + instrument["suspect_fast"] = True + if completed.returncode not in (0, 1): + semgrep["stderr"] = completed.stderr[-2000:] + raise InstrumentFailure(f"semgrep_exit_{completed.returncode}") + try: + payload = json.loads(completed.stdout) + except json.JSONDecodeError: + semgrep["stderr"] = completed.stderr[-2000:] + raise InstrumentFailure("semgrep_output_unreadable") from None + semgrep["scanned"] = sorted(payload.get("paths", {}).get("scanned", [])) + semgrep["errors"] = [ + {"type": str(error.get("type")), "path": error.get("path"), "message": str(error.get("message", ""))[:300]} + for error in payload.get("errors", []) + ] + results = payload.get("results", []) + semgrep["results"] = len(results) + record["push"]["admission"]["defects"] = [result_defect(result) for result in results] + print(f"{change['id']}: semgrep exit {completed.returncode}, {len(results)} results, {wall:.1f}s", flush=True) + finally: + if added: + git.run("worktree", "remove", "--force", str(root)) + except Exception as error: # noqa: BLE001 + reason = str(error) if isinstance(error, InstrumentFailure) else type(error).__name__ + instrument.update(hook_exit=2, failure=reason) + record["status"] = "instrument_failure" + record["push"]["admission"]["defects"] = [] + print(f"{change['id']}: instrument_failure ({reason})", file=sys.stderr, flush=True) + return record + + +def run(manifest, repos_path, rules, out, *, slice_id="1", only="all", semgrep=None): + binary = semgrep or shutil.which("semgrep") + if binary is None: + print("No semgrep binary found on PATH.", file=sys.stderr) + return 2 + rules = Path(rules).expanduser().resolve() + if not rules.is_dir(): + raise ValueError(f"rules directory missing: {rules}") + changes = load_slice(manifest, slice_id) + if len({change["id"] for change in changes}) != len(changes): + raise ValueError("duplicate change ids") + changes = [change for change in changes if only == "all" or change["kind"] == only] + entries = _inputs(out) + indexed = {entry["change"]["id"]: entry for entry in entries} + pending = [] + for change in changes: + path = _record_path(out, change["id"]) + previous = json.loads(path.read_text()) if path.exists() else {} + entry = indexed.get(change["id"]) + if entry is None: + entry = { + "item_id": change["id"], + "change": change, + "input_digest": previous.get("input_digest") or hashlib.sha256(canonical(change)).hexdigest(), + } + entries.append(entry) + elif entry["change"] != change or entry["item_id"] != change["id"]: + raise ValueError("existing semgrep inputs do not match frozen change") + if previous.get("done") is True: + print(f"{change['id']}: skipped (done)", flush=True) + continue + pending.append(entry) + _write_json(out / "inputs.json", entries) + if not pending: + return 0 + repos = json.loads(Path(repos_path).expanduser().read_text()) + failed = False + for entry in pending: + record = _scan_change(entry["change"], repos, rules, binary=binary) + record["input_digest"] = entry["input_digest"] + _write_json(_record_path(out, entry["item_id"]), record) + failed |= record["status"] == "instrument_failure" + return 2 if failed else 0 + + +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__) + commands = parser.add_subparsers(dest="command", required=True) + rules = commands.add_parser("rules", help="convert the ruleset and the facts into Semgrep rules") + rules.add_argument("--out", type=Path, required=True) + rules.add_argument("--ruleset", type=Path, default=DEFAULT_RULESET_DIR) + rules.add_argument("--facts", type=Path, default=DEFAULT_FACTS_DIR) + rules.add_argument("--min-precision", type=float, default=0.0, help="the hook's min_emit_precision floor") + rules.add_argument("--semgrep", help="semgrep binary (default: PATH)") + scan = commands.add_parser("run", help="run Semgrep over each frozen change's changed files") + scan.add_argument("manifest", type=Path) + scan.add_argument("--slice", default="1") + scan.add_argument("--repos", type=Path, required=True) + scan.add_argument("--rules", type=Path, required=True) + scan.add_argument("--out", type=Path, required=True) + scan.add_argument("--only", choices=("vulnerability", "ordinary", "all"), default="all") + scan.add_argument("--semgrep", help="semgrep binary (default: PATH)") + args = parser.parse_args(argv) + if args.command == "rules": + report = write_rules( + args.out, ruleset_dir=args.ruleset, facts_dir=args.facts, min_precision=args.min_precision, semgrep=args.semgrep + ) + validation = report["validation"] + print( + f"regex converted {len(report['regex']['converted'])}, skipped {len(report['regex']['skipped'])}; " + f"facts converted {len(report['facts']['converted'])}, skipped {len(report['facts']['skipped'])}, " + f"lossy {len(report['facts']['lossy'])}; " + f"obligations skipped {report['obligations']['skipped_tables']}; " + f"validation {'not run' if validation is None else 'exit ' + str(validation['exit'])}" + ) + return 0 if validation is None or validation["exit"] == 0 else 2 + return run(args.manifest, args.repos, args.rules, args.out.expanduser(), slice_id=args.slice, only=args.only, semgrep=args.semgrep) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/test_semgrep_arm.py b/tests/test_semgrep_arm.py new file mode 100644 index 0000000..28f5e83 --- /dev/null +++ b/tests/test_semgrep_arm.py @@ -0,0 +1,397 @@ +"""Offline coverage of the Semgrep benchmark arm: rule conversion, result mapping, record shape.""" + +import hashlib +import importlib +import json +import shutil +import sys +from pathlib import Path + +import pytest +import yaml + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +semgrep_arm = importlib.import_module("benchmarks.valid.semgrep_arm") +catalog = importlib.import_module("benchmarks.valid.catalog") +score = importlib.import_module("benchmarks.unseen.score") + +RULES_TOML = """ +[[rule]] +rule_id = "python-shell-true" +title = "Subprocess shell execution needs review" +languages = ["python"] +cwe = "CWE-78" +tags = ["syscall_entry"] +status = "enabled" +precision_estimate = 0.0 +version = "1" +pattern = '''subprocess\\.[a-zA-Z_]+\\s*\\([^\\n]*shell\\s*=\\s*True''' + +[[rule]] +rule_id = "php-sql-shadow" +title = "PHP SQL call composed from a variable" +languages = ["php"] +cwe = "CWE-89" +tags = ["injection"] +status = "shadow" +precision_estimate = 0.25 +version = "1" +pattern = '''mysqli_query\\s*\\([^\\n]*\\$''' + +[[rule]] +rule_id = "php-sql-bounded" +title = "PHP sprintf-built SQL" +languages = ["php"] +cwe = "CWE-89" +tags = ["injection"] +status = "enabled" +precision_estimate = 0.1 +version = "1" +pattern = '''(sprintf)\\((?=(?:(?!\\);)[\\s\\S]){0,2000}?'%s')''' + +[[rule]] +rule_id = "groovy-xss-unescaped" +title = "Groovy template unescaped output needs review" +languages = ["groovy"] +cwe = "CWE-79" +tags = ["injection"] +status = "enabled" +precision_estimate = 0.0 +version = "1" +pattern = '''(yieldUnescaped)\\s+[A-Za-z_]''' +""" + +FACTS_TOML = """ +version = "1" +language = "python" + +[[source]] +id = "request" +patterns = ["request.args", "request.get_json", "input("] + +[[sink]] +id = "os.system" +cwe = "CWE-78" +calls = ["os.system", "subprocess.run"] +rule_ids = ["python-os-command"] + +[[sink]] +id = "execute" +cwe = "CWE-89" +calls = ["execute"] +rule_ids = [] + +[[sink]] +id = "hashlib" +cwe = "CWE-327" +calls = ["hashlib.md5", "hashlib.new"] +rule_ids = [] +weak_literals = ["md5", "sha1"] + +[[sanitizer]] +id = "shlex.quote" +calls = ["shlex.quote"] + +[[sanitizer]] +id = "parameterized_execute" +calls = ["execute"] +parameterized = true + +[[sanitizer]] +id = "url_allowlist" +calls = ["url_has_allowed_host_and_scheme"] +guard = true +""" + +SEMGREP_SAMPLE = { + "version": "1.179.0", + "results": [ + { + "check_id": "rules.ousast.python.taint.os.system", + "path": "app/views.py", + "start": {"line": 12, "col": 5, "offset": 300}, + "end": {"line": 12, "col": 30, "offset": 325}, + "extra": { + "message": "os.system: untrusted input reaches os.system, subprocess.run", + "metadata": {"family": "injection", "cwe": "CWE-78", "origin": "sink:python:os.system", "precision_estimate": "unmeasured"}, + "severity": "ERROR", + "lines": ' os.system("ls " + c)', + "dataflow_trace": {"taint_source": ["CliLoc", [{"path": "app/views.py", "start": {"line": 11}}, "request.args"]]}, + }, + }, + { + "check_id": "rules.ousast.python.regex.python-os-command", + "path": "app/views.py", + "start": {"line": 12, "col": 5, "offset": 300}, + "end": {"line": 12, "col": 15, "offset": 310}, + "extra": { + "message": "Python OS command execution needs review", + "metadata": {"family": "injection", "cwe": "CWE-78", "origin": "python-os-command", "precision_estimate": "0.0"}, + "severity": "WARNING", + "lines": ' os.system("ls " + c)', + }, + }, + ], + "errors": [], + "paths": {"scanned": ["app/views.py"]}, +} + + +@pytest.fixture +def ruleset(tmp_path): + root = tmp_path / "ruleset" + (root / "python").mkdir(parents=True) + (root / "python" / "rules.toml").write_text(RULES_TOML) + (root / "obligations").mkdir() + (root / "obligations" / "python.toml").write_text('[[operation]]\nid = "x"\n[[discharger]]\nid = "y"\n') + facts = root / "semantic" + facts.mkdir() + (facts / "python.toml").write_text(FACTS_TOML) + return root + + +def test_regex_rules_convert_with_metadata_and_skip_reasons(ruleset): + converted, skipped, rewrites = semgrep_arm.convert_regex_rules(ruleset) + by_origin = {rule["metadata"]["origin"]: rule for rules in converted.values() for rule in rules} + shell = by_origin["python-shell-true"] + assert shell["languages"] == ["python"] + assert shell["pattern-regex"] == r"subprocess\.[a-zA-Z_]+\s*\([^\n]*shell\s*=\s*True" + assert shell["metadata"]["family"] == "injection" + assert shell["metadata"]["cwe"] == "CWE-78" + assert shell["metadata"]["precision_estimate"] == "0.0" + assert shell["id"].startswith("ousast.python.regex.") + groovy = by_origin["groovy-xss-unescaped"] + assert groovy["languages"] == ["generic"] + assert groovy["paths"] == {"include": ["*.groovy", "*.tpl"]} + assert groovy["metadata"]["family"] == "output_encoding" + assert by_origin["php-sql-bounded"]["pattern-regex"] == r"(sprintf)\((?=(?:(?!\);)[\s\S])*?'%s')" + assert [rewrite["origin"] for rewrite in rewrites] == ["php-sql-bounded"] + assert [entry["origin"] for entry in skipped] == ["php-sql-shadow"] + assert "shadow" in skipped[0]["reason"] + + +def test_emit_floor_skips_rules_below_precision(ruleset): + _, skipped, _ = semgrep_arm.convert_regex_rules(ruleset, min_precision=0.05) + reasons = {entry["origin"]: entry["reason"] for entry in skipped} + assert reasons["python-shell-true"].startswith("emit floor") + assert reasons["groovy-xss-unescaped"].startswith("emit floor") + assert "php-sql-bounded" not in reasons + + +def test_fact_triple_becomes_a_taint_rule(ruleset): + converted, skipped, lossy = semgrep_arm.convert_facts(ruleset / "semantic") + rules = {rule["metadata"]["origin"]: rule for rule in converted["python"]} + taint = rules["sink:python:os.system"] + assert taint["mode"] == "taint" + assert taint["languages"] == ["python"] + assert taint["metadata"] == { + "family": "injection", + "cwe": "CWE-78", + "origin": "sink:python:os.system", + "precision_estimate": "unmeasured", + "kind": "taint", + "calls": ["os.system", "subprocess.run"], + } + sources = [entry["pattern"] for entry in taint["pattern-sources"][0]["pattern-either"]] + assert {"request.args", "request.get_json(...)", "input(...)"} <= set(sources) + sinks = [entry["pattern"] for entry in taint["pattern-sinks"][0]["pattern-either"]] + assert sinks == ["os.system(...)", "subprocess.run(...)"] + sanitizers = [entry["pattern"] for entry in taint["pattern-sanitizers"][0]["pattern-either"]] + assert sanitizers == ["shlex.quote(...)"] + execute = rules["sink:python:execute"]["pattern-sinks"][0]["pattern-either"] + assert execute[0] == {"patterns": [{"pattern": "execute($QUERY)"}, {"focus-metavariable": "$QUERY"}]} + weak = rules["sink:python:hashlib"] + assert "mode" not in weak and weak["metadata"]["kind"] == "weak_literal" and weak["metadata"]["family"] == "config_secrets" + reasons = {entry["origin"]: entry["reason"] for entry in skipped} + assert reasons["sanitizer:python:url_allowlist"].startswith("guard") + assert reasons["sanitizer:python:parameterized_execute"].startswith("parameterized") + assert lossy == [] + + +def test_write_rules_emits_one_yaml_per_language_and_a_report(ruleset, tmp_path): + out = tmp_path / "rules" + report = semgrep_arm.write_rules(out, ruleset_dir=ruleset, facts_dir=ruleset / "semantic", semgrep="/nonexistent/semgrep") + assert sorted(path.name for path in out.glob("*.yaml")) == ["groovy.yaml", "php.yaml", "python.yaml"] + assert report["files"] == {"groovy.yaml": 1, "php.yaml": 1, "python.yaml": 4} + loaded = yaml.safe_load((out / "python.yaml").read_text()) + ids = [rule["id"] for rule in loaded["rules"]] + assert len(ids) == len(set(ids)) == 4 + for rule in loaded["rules"]: + assert set(rule) >= {"id", "languages", "severity", "message", "metadata"} + assert {"family", "cwe", "origin", "precision_estimate"} <= set(rule["metadata"]) + assert isinstance(rule["metadata"]["precision_estimate"], str) + assert ("pattern-regex" in rule) or ("pattern-either" in rule) or (rule.get("mode") == "taint") + saved = json.loads((out / "conversion_report.json").read_text()) + assert saved["obligations"]["skipped_tables"] == 2 + assert saved["regex"]["converted"] == ["groovy-xss-unescaped", "php-sql-bounded", "python-shell-true"] + assert saved["facts"]["converted"] == ["sink:python:execute", "sink:python:hashlib", "sink:python:os.system"] + # A missing binary is recorded, never reported as a successful validation. + assert report["validation"]["exit"] is None and "not found" in report["validation"]["error"] + + +def test_result_to_defect_mapping(): + defects = [semgrep_arm.result_defect(result) for result in SEMGREP_SAMPLE["results"]] + assert defects[0] == { + "defect_id": "sink:python:os.system:app/views.py:12", + "family": "injection", + "cwe": "CWE-78", + "locations": [["app/views.py", 12]], + "source": "semgrep", + "rule_id": "rules.ousast.python.taint.os.system", + "origin": "sink:python:os.system", + "title": "os.system: untrusted input reaches os.system, subprocess.run", + "rationale": "os.system: untrusted input reaches os.system, subprocess.run", + "confidence": "ERROR", + "dataflow": True, + } + assert defects[1]["dataflow"] is False + assert defects[1]["confidence"] == "WARNING" + assert defects[1]["origin"] == "python-os-command" + + +def _change(**overrides): + change = dict( + id="fixture-positive", + repo="fixture/app", + repository_url="https://example.invalid/app", + kind="vulnerability", + base="b" * 40, + head="a" * 40, + family="injection", + cwe="CWE-78", + path="app/views.py", + function="handler", + function_lines=[10, 14], + files=[{"path": "app/views.py", "head_lines": [[10, 14]]}], + method="fix_site", + slice="1", + ) + change.update(overrides) + return change + + +def test_record_shape_is_accepted_by_the_scorer(tmp_path): + out = tmp_path / "semgrep" + change = _change() + digest = hashlib.sha256(json.dumps(change, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode()).hexdigest() + semgrep_arm._write_json(out / "inputs.json", [{"item_id": change["id"], "change": change, "input_digest": digest}]) + record = { + "id": change["id"], + "done": True, + "status": "semgrep_covered", + "input_digest": digest, + "instrument": { + "hook_exit": 0, + "hook_bytes_read": 412, + "changed_bytes": {"app/views.py": 412}, + "head_verified": True, + "base_verified": True, + "failure": None, + "suspect_fast": False, + "semgrep": {"exit": 0, "wall_seconds": 4.9, "scanned": ["app/views.py"], "errors": [], "results": 2}, + }, + "push": { + "admission": {"defects": [semgrep_arm.result_defect(result) for result in SEMGREP_SAMPLE["results"]], "dispositions": []}, + "quick_tier": [], + }, + "skipped": [], + } + semgrep_arm._write_json(semgrep_arm._record_path(out, change["id"]), record) + rows = score.load_records(out) + assert len(rows) == 1 and rows[0]["status"] == "semgrep_covered" + assert score.coverage(rows[0]) == "quick_only" + assert score.catches(rows[0]) == ["alerts"] + assert score.false_alarm(rows[0]) + # A family mismatch is not a catch, exactly as the hunter arm is scored. + rows = score.load_records(out) + rows[0]["family"] = "path" + assert score.catches(rows[0]) == [] + scored = score.score(rows) + assert scored["coverage"]["quick_only"] == 1 + + +def test_instrument_failure_record_is_not_covered(tmp_path): + out = tmp_path / "semgrep" + change = _change() + digest = "0" * 64 + semgrep_arm._write_json(out / "inputs.json", [{"item_id": change["id"], "change": change, "input_digest": digest}]) + record = { + "id": change["id"], + "done": True, + "status": "instrument_failure", + "input_digest": digest, + "instrument": { + "hook_exit": 2, + "hook_bytes_read": 0, + "changed_bytes": {}, + "head_verified": True, + "base_verified": True, + "failure": "semgrep_exit_2", + "suspect_fast": False, + }, + "push": {"admission": {"defects": [], "dispositions": []}, "quick_tier": []}, + "skipped": [], + } + semgrep_arm._write_json(semgrep_arm._record_path(out, change["id"]), record) + rows = score.load_records(out) + assert score.coverage(rows[0]) == "instrument_failure" + assert not score.false_alarm(rows[0]) + + +@pytest.fixture +def fixture(tmp_path): + root = tmp_path / "repo" + root.mkdir() + git = catalog.local_git(root) + git.run("init", "-q") + git.run("config", "user.name", "Fixture") + git.run("config", "user.email", "fixture@example.invalid") + heads = [] + for text in ( + "import os\nfrom flask import request\n\n\ndef handler():\n c = request.args.get('c')\n os.system('ls ' + c)\n", + "# Later checkout\n", + ): + (root / "app.py").write_text(text) + git.run("add", "app.py") + git.run("-c", "core.hooksPath=/dev/null", "commit", "-qm", "fixture") + heads.append(git.run("rev-parse", "HEAD").strip()) + change = _change( + base=heads[1], + head=heads[0], + path="app.py", + function="handler", + function_lines=[5, 7], + files=[{"path": "app.py", "head_lines": [[5, 7]]}, {"path": "gone.py", "head_lines": [[1, 1]]}], + ) + frozen = tmp_path / "frozen.json" + frozen.write_text( + json.dumps({"changes": [change], "sha256": hashlib.sha256(json.dumps([change], sort_keys=True).encode()).hexdigest()}) + ) + repos = tmp_path / "repos.json" + repos.write_text(json.dumps({change["repo"]: str(root)})) + return {"root": root, "git": git, "change": change, "frozen": frozen, "repos": repos} + + +@pytest.mark.skipif(shutil.which("semgrep") is None, reason="semgrep binary not installed") +def test_run_scans_changed_files_and_cleans_up(fixture, tmp_path): + rules = tmp_path / "rules" + semgrep_arm.write_rules(rules, semgrep="/nonexistent/semgrep") + out = tmp_path / "out" + assert semgrep_arm.run(fixture["frozen"], fixture["repos"], rules, out, only="vulnerability") == 0 + rows = score.load_records(out) + assert len(rows) == 1 + record = rows[0] + assert record["status"] == "semgrep_covered" + assert record["instrument"]["changed_bytes"] == {"app.py": record["instrument"]["hook_bytes_read"]} + assert record["instrument"]["hook_bytes_read"] > 0 + assert record["instrument"]["semgrep"]["exit"] == 0 and not record["instrument"]["suspect_fast"] + assert record["skipped"] == [{"path": "gone.py", "reason": "missing_at_head"}] + assert record["instrument"]["semgrep"]["scanned"] == ["app.py"] + origins = {defect["origin"] for defect in record["push"]["admission"]["defects"]} + assert {"python-os-command", "sink:python:os.system"} <= origins + assert any(defect["dataflow"] for defect in record["push"]["admission"]["defects"]) + assert score.coverage(record) == "quick_only" + assert score.catches(record) == ["alerts"] + assert fixture["git"].run("worktree", "list").strip().count("\n") == 0 + # A second run resumes without touching the record. + assert semgrep_arm.run(fixture["frozen"], fixture["repos"], rules, out, only="vulnerability") == 0