measure: Semgrep arm on the valid pilot, our rules and facts converted

0 of 12 positives; 12.6 false alarms per 100 ordinary pushes (2.1 on changed
lines only). The misses come from what the rules and facts express (access
control lives in obligations, which do not convert; sources cover request
objects only), not from the engine. Comparison arm only: a Semgrep miss is
never evidence against a hunter finding.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

benchmarks/measurements/2026-10-13-semgrep-arm/NOTES.md

@@ -0,0 +1,160 @@
+# Semgrep CE as a third arm on the valid pilot (2026-10-13)
+
+The question from `../2026-10-13-semgrep-brief/BRIEF.md` (option C): does Semgrep, running our own
+rules and facts, catch more on the valid pilot than our engine does? It does not. It caught none of the
+12 positives and raised false alarms on 12.6 of every 100 ordinary pushes.
+
+Instrument: semgrep 1.179.0 CE (linuxbrew), `-j 1`, `--timeout 60 --max-memory 1500`, offline flags.
+Rules: `rules/` here (81 rules, from `python -m benchmarks.valid.semgrep_arm rules`). Records:
+`~/ousast-results/valid/pilot/semgrep/` (252 records, outside the repo). Same frozen manifest
+(`benchmarks/valid/pilot/pool-valid.toml`, slice 1) and same scorer as the other arms.
+
+## Headline
+
+| Arm | Recall (12 positives) | False alarms per 100 pushes (Wilson 95%) | Precision |
+|---|---|---|---|
+| (a) hook as shipped, 163 done | 0/12, 0.0% (0.0-24.3) | 0.7 (0.1-3.7), n=149 | 0.0% (0/1) |
+| (b) hunter, raw | 6/12, 50.0% (25.4-74.6) | 92.5 (88.4-95.2), n=239 | 2.6% |
+| (b') hunter, classifier-filtered | 5/12, 41.7% (19.3-68.0) | 54.4 | 3.6% |
+| **(c) Semgrep, our rules + facts** | **0/12, 0.0% (0.0-24.3)** | **12.6 (8.9-17.4), n=239**; cluster 3.8-23.4, ICC 0.28 | **0.0% (0/33)** |
+
+`metrics_semgrep.json`: "recall 0.0% (Wilson 0.0-24.3), false alarms 12.6 per 100 pushes, precision
+0.0%; exploit-verified 0 of 5 (5 not yet proved)". Exploit-verified is not applicable: the 5 flagged
+positives are all wrong-family hits, none was a catch.
+
+The (c) false-alarm rate is not like-for-like with (a). The hook keeps a quick hit only on a changed
+line or inside a changed function (`push/quick.py:122-130`); this arm counts every result anywhere in a
+changed file, as the run instructions asked. Restricting the same records afterwards to results on a
+changed line gives 5/239 = 2.1 per 100 (Wilson 0.9-4.8); within 10 lines of a change, 9/239 = 3.8
+(2.0-7.0). Scoped either way it is still a higher rate than (a), and recall stays at zero.
+
+## Per positive
+
+| Repo | Family | (a) hook | (b) raw | (c) Semgrep | What Semgrep reported |
+|---|---|---|---|---|---|
+| bytebase/dbhub | access_control | miss | miss | miss | `js-command-exec` at `sql-parser.ts:87`: `dollarQuoteOpenRegex.exec(...)`, a RegExp call |
+| contentful/contentful-mcp-server | untrusted_destination | miss | miss | miss | nothing |
+| fastapi-users/fastapi-users | access_control | miss | miss | miss | nothing |
+| i18next/i18next-http-middleware | output_encoding | miss | hit | miss | nothing |
+| imzbf/md-editor-v3 | output_encoding | miss | hit | miss | `js-command-exec` at `code/index.ts:37`: `re.exec(info)`, a RegExp call |
+| jupyter-lsp/jupyterlab-lsp | path | miss | hit | miss | nothing |
+| matrix-org/matrix-appservice-bridge | access_control | miss | miss | miss | nothing |
+| nvbn/thefuck | path | miss | hit | miss | nothing |
+| ondata/ckan-mcp-server | untrusted_destination | miss | hit | miss | nothing |
+| webpack/webpack-bundle-analyzer | output_encoding | miss | miss | miss | nothing |
+| zenitram/malla | output_encoding | miss | hit | miss | nothing |
+| zereight/gitlab-mcp | path | miss | miss | miss | `js-command-exec` + taint `sink:javascript:exec` at `index.ts:12429`: `/^Bearer.../i.exec(authHeader)` |
+
+No rule origin caught anything. Why the positives are silent: three are access_control, which only the
+obligations express, and none of those convert. The four output_encoding positives are a value reaching
+`innerHTML` from a function argument or a markdown renderer, and the taint rules' sources are request
+objects (`req.query`, `request.args`). The intra-function CE taint never sees an argument as a source.
+The path and destination positives take their input from tool arguments or archive members, which are
+also not modelled sources. Our hook misses these 12 for the same reason. Swapping the engine does not
+change the vocabulary.
+
+## False alarms (239 ordinary pushes, 30 flagged)
+
+Pushes flagged, by rule origin: `js-command-exec` 16, `python-sql-injection` 8, `sink:javascript:exec`
+8, `python-unsafe-eval` 3, `sink:javascript:html.response` 2, `python-os-command` 1,
+`sink:javascript:prototype.merge` 1. The results are 95 in total: 90 injection, 3 prototype, 2
+output_encoding. Only 18 carry a taint trace; the other 77 are regex.
+
+The flags come from five repos: malla 10, gitlab-mcp 10, webpack-bundle-analyzer 4, dbhub 3,
+matrix-appservice-bridge 2, thefuck 1. An ICC of 0.28 means a handful of files drive the rate. What the
+lines are:
+
+- `js-command-exec` (`\b(exec|execSync)\s*\(`) and the taint sink `$OBJ.exec(...)` match `RegExp.exec`.
+  In gitlab-mcp the request header is a taint source, so `/regex/.exec(authHeader)` reads as command
+  injection with a dataflow trace. The rest are `childProcess.execSync` in test files.
+- `python-sql-injection` matches multi-line `cursor.execute(` calls built from f-strings or constants in
+  malla's repositories. The regex cannot tell a constant from input.
+- `python-unsafe-eval` matched `eval()` inside a comment. The quick engine drops matches in comments;
+  `pattern-regex` does not (see "not carried" below).
+- `sink:javascript:prototype.merge` matched `Map.set(...)`. lodash's `set` converts to `$OBJ.set(...)`.
+- `sink:javascript:html.response` matched `res.send({token, userId})`, an object Express sends as JSON.
+
+## Runtime
+
+251 Semgrep runs: mean 8.5 s per change, median 7.7 s, min 5.6 s, max 37.3 s (gitlab-mcp, a 453 KB
+`index.ts`). Total 35.6 min of Semgrep wall time; the whole run (with checkouts) took 36 min,
+12:31 to 13:07. 846 target files and 24.8 MB were read before the runs. Most of the per-change cost is
+Semgrep's fixed startup, about 5 s (the brief measured 4.7 s). The in-process regex tier runs in under a
+second.
+
+## Instrument
+
+- Every run read its targets and printed bytes per file first. No run returned faster than 1 s
+  (`suspect_fast` 0). Every Semgrep exit was 0. For each change, `paths.scanned` is recorded next to
+  the bytes read.
+- 1 instrument failure (0.4%, under the scorer's 5% limit): a matrix-appservice-bridge ordinary change
+  whose four changed files were all deleted at head (`no_readable_targets`).
+- 30 changed files were missing at head and skipped.
+- 103 changes had targets Semgrep did not scan because no rule covers that language: `.md` 94, `.json`
+  72, no extension 16, `.html` 13, `.yml` 11, `.css` 10, and others. Our hook does not read these
+  either.
+- 2 PartialParsing errors (TypeScript `import("...")` types, a regex literal). Both files were still
+  scanned.
+- Smoke test before the run: the converted rules fired with traces on hand-written Python, JS/TSX, Java,
+  C and PHP samples (`os.system`, `innerHTML`, `dangerouslySetInnerHTML`, `executeQuery`, `strcpy`,
+  `echo $x;`, `include $_GET`). So the zero recall is not a dead ruleset.
+
+## Conversion report (`rules/conversion_report.json`)
+
+- Regex rules: 41 of 52 converted to `pattern-regex`. 11 were skipped: 5 shadow (`php-sql-call-composition`,
+  `php-code-eval`, `php-unserialize`, `php-maybe-unserialize`, `php-xss-echo-input`) and 6
+  `*-language` fallbacks (inactive while their tagged rule is active). No rule targets non-code
+  files, and none was dropped by an emit floor (the arm uses the hook's default floor, 0.0; `--min-precision`
+  sets one). Groovy has no parser and becomes `languages: [generic]` with `paths.include: [*.groovy, *.tpl]`.
+- One rewrite: `php-sql-sprintf-unescaped` uses `(?:...){0,2000}?`, which PCRE2 rejects as "regular
+  expression is too large". It became `*?`, so the window is now the whole statement instead of 2000
+  characters.
+- Facts: 40 taint/search rules (JS 10, PHP 10, Python 10, Java 7, C 3), one per (language, sink id,
+  CWE). Sources come from every source fact of that language; sanitizers come from cleansing sanitizer
+  facts. `weak_literals` sinks (hashlib, MessageDigest) become search rules with a `metavariable-regex`.
+  Format sinks focus their format argument, and parameterized calls (`execute`, `query`) focus a
+  single-argument call.
+- Not convertible, skipped and named:
+  - `guard` sanitizers (`php:allowlist` = `in_array`, `python:url_allowlist`): safety depends on
+    dominance.
+  - `parameterized` sanitizer facts (4): a call shape, expressed as the focused sink form instead.
+  - `literal_format_arg` (C): expressed by focusing the format argument.
+  - `dispatch:php:wordpress_hooks`: string-keyed registry routing.
+  - `layout` facts (2): scope policy, not rules.
+  - `v-html`: a template directive.
+  - All 37 obligation tables: an absence has no pattern.
+- Lossy (converted, but with weaker semantics):
+  - `quoted_only` `sql_escape` became a full sanitizer. This silences the CVE-2024-1071 shape the project
+    keeps.
+  - `prefix_fixes_origin` on 4 destination sinks (JS/Python outbound requests and redirects) is
+    dropped. A fixed-origin prefix no longer clears the value; Django's `reverse` anchors became
+    sanitizers.
+- Not carried at all: comment skipping, framework priors as a switch, `min_evidence_level`, ranking, the
+  rung.
+
+## What this says
+
+Running the same vocabulary through Semgrep's engine adds no catches on these 12 positives. It adds
+false alarms, from rules that the quick tier keeps quiet with comment skipping and changed-line scoping.
+The misses come from the vocabulary (sources limited to request objects, no access_control without
+obligations), not from the engine. This is the measured reason the brief asked for, and it points
+against option A/B: the steering rule ("do not ... add a second engine") stands. If the arm is reused,
+fix the cheap false alarms first: exclude `RegExp.exec`, skip comments, and scope results to changed
+lines.
+
+## Reproduce
+
+```bash
+export PYTHONPATH=$PWD
+.venv/bin/python -m benchmarks.valid.semgrep_arm rules --out ~/ousast-results/valid/pilot/semgrep-rules/
+for only in vulnerability ordinary; do
+  .venv/bin/python -m benchmarks.valid.semgrep_arm run benchmarks/valid/pilot/pool-valid.toml --slice 1 \
+    --repos ~/ousast-results/valid/pilot/repos_hist.json --rules ~/ousast-results/valid/pilot/semgrep-rules \
+    --out ~/ousast-results/valid/pilot/semgrep --only $only
+done
+.venv/bin/python -m benchmarks.unseen.score ~/ousast-results/valid/pilot/semgrep --out ~/ousast-results/valid/pilot/scored_semgrep.json --slice-id 1
+.venv/bin/python -m benchmarks.valid.metrics ~/ousast-results/valid/pilot/semgrep --scored ~/ousast-results/valid/pilot/scored_semgrep.json --out ~/ousast-results/valid/pilot/metrics_semgrep.json
+```
+
+The ordinary pass exits 2 because of the one instrument failure; the run still finishes and resumes
+past done records.

benchmarks/measurements/2026-10-13-semgrep-arm/metrics_semgrep.json

@@ -0,0 +1,58 @@
+{
+  "exploit_verified": {
+    "count": 0,
+    "flagged_positive_pushes": 3,
+    "flagged_positives": 5,
+    "n": 0,
+    "not_yet_proved": [
+      "bytebase-dbhub-f2624a2ead8d31ca",
+      "imzbf-md-editor-v3-14658213915dbde5",
+      "imzbf-md-editor-v3-b9c833a3524a57a9",
+      "zereight-gitlab-mcp-2367b459f8c3802b",
+      "zereight-gitlab-mcp-768c7d5a32ccc95b"
+    ],
+    "rate": null,
+    "status": "not yet proved",
+    "wilson95": [
+      null,
+      null
+    ]
+  },
+  "false_alarms_per_100_pushes": {
+    "cluster95": [
+      3.75,
+      23.430962343096233
+    ],
+    "count": 30,
+    "icc": 0.27735846730536884,
+    "n": 239,
+    "rate": 12.552301255230125,
+    "wilson95": [
+      8.93573620529521,
+      17.35366396337441
+    ]
+  },
+  "headline": "On unseen app repos: recall 0.0% (Wilson 0.0-24.3), false alarms 12.6 per 100 pushes, precision 0.0%; exploit-verified 0 of 5 (5 not yet proved).",
+  "precision": {
+    "count": 0,
+    "n": 33,
+    "rate": 0.0,
+    "wilson95": [
+      0,
+      0.10427343003561192
+    ]
+  },
+  "recall": {
+    "cluster95": [
+      0.0,
+      0.0
+    ],
+    "count": 0,
+    "n": 12,
+    "rate": 0.0,
+    "wilson95": [
+      0,
+      0.2425007574992425
+    ]
+  }
+}

benchmarks/measurements/2026-10-13-semgrep-arm/rules/c.yaml

@@ -0,0 +1,216 @@
+rules:
+- id: ousast.c.regex.c-exec-family
+  languages:
+  - c
+  - cpp
+  severity: WARNING
+  message: C exec command execution needs review
+  metadata:
+    family: injection
+    cwe: CWE-78
+    origin: c-exec-family
+    precision_estimate: '0.0'
+    kind: regex
+    tags:
+    - syscall_entry
+    status: enabled
+    version: '1'
+  pattern-regex: \b(execl|execlp|execle|execv|execvp|execvpe)\s*\(
+- id: ousast.c.regex.c-shell-exec
+  languages:
+  - c
+  - cpp
+  severity: WARNING
+  message: C shell execution needs review
+  metadata:
+    family: injection
+    cwe: CWE-78
+    origin: c-shell-exec
+    precision_estimate: '0.0'
+    kind: regex
+    tags:
+    - syscall_entry
+    status: enabled
+    version: '1'
+  pattern-regex: \b(system|popen)\s*\(
+- id: ousast.c.regex.c-unsafe-copy
+  languages:
+  - c
+  - cpp
+  severity: WARNING
+  message: Unsafe C string copy needs review
+  metadata:
+    family: memory
+    cwe: CWE-121
+    origin: c-unsafe-copy
+    precision_estimate: '0.0'
+    kind: regex
+    tags:
+    - memory_unsafe
+    status: enabled
+    version: '1'
+  pattern-regex: \b(gets|strcpy|strcat|sprintf)\s*\(
+- id: ousast.c.regex.c-unsafe-format
+  languages:
+  - c
+  - cpp
+  severity: WARNING
+  message: Unsafe C format string needs review
+  metadata:
+    family: memory
+    cwe: CWE-134
+    origin: c-unsafe-format
+    precision_estimate: '0.0'
+    kind: regex
+    tags:
+    - memory_unsafe
+    status: enabled
+    version: '1'
+  pattern-regex: \b(printf|vprintf|vsprintf|vsnprintf|vfprintf|syslog)\s*\(\s*[A-Za-z_]
+- id: ousast.c.regex.c-unsafe-memory-copy
+  languages:
+  - c
+  - cpp
+  severity: WARNING
+  message: Unsafe C memory copy needs review
+  metadata:
+    family: memory
+    cwe: CWE-121
+    origin: c-unsafe-memory-copy
+    precision_estimate: '0.0'
+    kind: regex
+    tags:
+    - memory_unsafe
+    status: enabled
+    version: '1'
+  pattern-regex: \b(memcpy|memmove|bcopy)\s*\(
+- id: ousast.c.regex.c-unsafe-scanf
+  languages:
+  - c
+  - cpp
+  severity: WARNING
+  message: Unbounded C scanf needs review
+  metadata:
+    family: memory
+    cwe: CWE-121
+    origin: c-unsafe-scanf
+    precision_estimate: '0.0'
+    kind: regex
+    tags:
+    - memory_unsafe
+    status: enabled
+    version: '1'
+  pattern-regex: \b(scanf|fscanf|sscanf)\s*\([^\n]*%s
+- id: ousast.c.taint.printf
+  mode: taint
+  languages: &id001
+  - c
+  - cpp
+  severity: ERROR
+  message: 'printf: untrusted input reaches fprintf, printf, sprintf, syslog, vprintf, vsprintf'
+  metadata:
+    family: memory
+    cwe: CWE-134
+    origin: sink:c:printf
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - fprintf
+    - printf
+    - sprintf
+    - syslog
+    - vprintf
+    - vsprintf
+  pattern-sources:
+  - pattern-either: &id002
+    - pattern: argv
+    - pattern: argv(...)
+    - pattern: getenv
+    - pattern: getenv(...)
+    - pattern: fgets
+    - pattern: fgets(...)
+    - pattern: gets
+    - pattern: gets(...)
+    - pattern: scanf
+    - pattern: scanf(...)
+    - pattern: fscanf
+    - pattern: fscanf(...)
+  pattern-sinks:
+  - pattern-either:
+    - patterns:
+      - pattern: fprintf($FORMAT, ...)
+      - focus-metavariable: $FORMAT
+    - patterns:
+      - pattern: printf($FORMAT, ...)
+      - focus-metavariable: $FORMAT
+    - patterns:
+      - pattern: sprintf($FORMAT, ...)
+      - focus-metavariable: $FORMAT
+    - patterns:
+      - pattern: syslog($FORMAT, ...)
+      - focus-metavariable: $FORMAT
+    - patterns:
+      - pattern: vprintf($FORMAT, ...)
+      - focus-metavariable: $FORMAT
+    - patterns:
+      - pattern: vsprintf($FORMAT, ...)
+      - focus-metavariable: $FORMAT
+- id: ousast.c.taint.strcpy
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'strcpy: untrusted input reaches gets, sprintf, strcat, strcpy'
+  metadata:
+    family: memory
+    cwe: CWE-121
+    origin: sink:c:strcpy
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - gets
+    - sprintf
+    - strcat
+    - strcpy
+  pattern-sources:
+  - pattern-either: *id002
+  pattern-sinks:
+  - pattern-either:
+    - pattern: gets(...)
+    - pattern: $OBJ.gets(...)
+    - pattern: sprintf(...)
+    - pattern: $OBJ.sprintf(...)
+    - pattern: strcat(...)
+    - pattern: $OBJ.strcat(...)
+    - pattern: strcpy(...)
+    - pattern: $OBJ.strcpy(...)
+- id: ousast.c.taint.system
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'system: untrusted input reaches execl, execv, execvp, popen, system'
+  metadata:
+    family: injection
+    cwe: CWE-78
+    origin: sink:c:system
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - execl
+    - execv
+    - execvp
+    - popen
+    - system
+  pattern-sources:
+  - pattern-either: *id002
+  pattern-sinks:
+  - pattern-either:
+    - pattern: execl(...)
+    - pattern: $OBJ.execl(...)
+    - pattern: execv(...)
+    - pattern: $OBJ.execv(...)
+    - pattern: execvp(...)
+    - pattern: $OBJ.execvp(...)
+    - pattern: popen(...)
+    - pattern: $OBJ.popen(...)
+    - pattern: system(...)
+    - pattern: $OBJ.system(...)

benchmarks/measurements/2026-10-13-semgrep-arm/rules/conversion_report.json

@@ -0,0 +1,233 @@
+{
+  "facts": {
+    "converted": [
+      "sink:c:printf",
+      "sink:c:strcpy",
+      "sink:c:system",
+      "sink:java:MessageDigest",
+      "sink:java:exec",
+      "sink:java:executeQuery",
+      "sink:java:file.open",
+      "sink:java:outbound.request",
+      "sink:java:readObject",
+      "sink:java:response.write",
+      "sink:javascript:eval",
+      "sink:javascript:exec",
+      "sink:javascript:html.response",
+      "sink:javascript:html.unescaped",
+      "sink:javascript:outbound.request",
+      "sink:javascript:path.open",
+      "sink:javascript:prototype.merge",
+      "sink:javascript:query",
+      "sink:javascript:redirect",
+      "sink:javascript:vm.eval",
+      "sink:php:code_eval",
+      "sink:php:command",
+      "sink:php:echo",
+      "sink:php:file_inclusion",
+      "sink:php:filesystem",
+      "sink:php:outbound_request",
+      "sink:php:redirect",
+      "sink:php:sql",
+      "sink:php:unserialize",
+      "sink:php:wpdb",
+      "sink:python:eval",
+      "sink:python:execute",
+      "sink:python:hashlib",
+      "sink:python:html.unescaped",
+      "sink:python:orm.raw",
+      "sink:python:os.system",
+      "sink:python:outbound.request",
+      "sink:python:path.open",
+      "sink:python:pickle.loads",
+      "sink:python:redirect"
+    ],
+    "lossy": [
+      {
+        "loss": "prefix_fixes_origin: a value appended after a fixed origin still counts as tainted",
+        "origin": "sink:javascript:outbound.request"
+      },
+      {
+        "loss": "prefix_fixes_origin: a value appended after a fixed origin still counts as tainted",
+        "origin": "sink:javascript:redirect"
+      },
+      {
+        "loss": "quoted_only escape treated as a full sanitizer (silences the CVE-2024-1071 shape)",
+        "origin": "sanitizer:php:sql_escape"
+      },
+      {
+        "loss": "prefix_fixes_origin: a value appended after a fixed origin still counts as tainted",
+        "origin": "sink:python:outbound.request"
+      },
+      {
+        "loss": "prefix_fixes_origin: a value appended after a fixed origin still counts as tainted",
+        "origin": "sink:python:redirect"
+      }
+    ],
+    "skipped": [
+      {
+        "origin": "sanitizer:c:literal_format",
+        "reason": "literal_format_arg: a shape; a literal format is never tainted at the focused sink argument"
+      },
+      {
+        "origin": "sanitizer:java:parameterized_statement",
+        "reason": "parameterized: a call shape, not a cleansing; expressed as the single-argument sink form"
+      },
+      {
+        "origin": "sanitizer:javascript:parameterized_query",
+        "reason": "parameterized: a call shape, not a cleansing; expressed as the single-argument sink form"
+      },
+      {
+        "origin": "layout:javascript:javascript_first_party_tests",
+        "reason": "layout: scope policy (vendored, tests), not a rule"
+      },
+      {
+        "origin": "sink:javascript:html.unescaped:v-html",
+        "reason": "template directive, not a call"
+      },
+      {
+        "origin": "sanitizer:php:allowlist",
+        "reason": "guard: a check whose safety depends on dominance; Semgrep has no equivalent"
+      },
+      {
+        "origin": "sanitizer:php:prepared_statement",
+        "reason": "parameterized: a call shape, not a cleansing; expressed as the single-argument sink form"
+      },
+      {
+        "origin": "dispatch:php:wordpress_hooks",
+        "reason": "dispatch: string-keyed hook registry routing; no Semgrep equivalent"
+      },
+      {
+        "origin": "layout:php:php_layout",
+        "reason": "layout: scope policy (vendored, tests), not a rule"
+      },
+      {
+        "origin": "sanitizer:python:parameterized_execute",
+        "reason": "parameterized: a call shape, not a cleansing; expressed as the single-argument sink form"
+      },
+      {
+        "origin": "sanitizer:python:url_allowlist",
+        "reason": "guard: a check whose safety depends on dominance; Semgrep has no equivalent"
+      }
+    ]
+  },
+  "files": {
+    "c.yaml": 9,
+    "groovy.yaml": 1,
+    "java.yaml": 11,
+    "javascript.yaml": 21,
+    "php.yaml": 19,
+    "python.yaml": 20
+  },
+  "not_carried": [
+    "comment skipping: the quick engine drops a regex match inside a comment; pattern-regex does not",
+    "framework priors and fallback rules: the tagged rule is emitted, its language-level fallback is not",
+    "min_evidence_level, ranking priority, the rung: post-match policy on our side"
+  ],
+  "obligations": {
+    "reason": "an obligation finds a missing guard (an absence), which no pattern expresses",
+    "skipped_tables": 37
+  },
+  "regex": {
+    "converted": [
+      "c-exec-family",
+      "c-shell-exec",
+      "c-unsafe-copy",
+      "c-unsafe-format",
+      "c-unsafe-memory-copy",
+      "c-unsafe-scanf",
+      "groovy-xss-unescaped",
+      "java-command-exec",
+      "java-deserialization",
+      "java-sql-injection",
+      "java-weak-hash",
+      "js-command-exec",
+      "js-deserialize",
+      "js-dom-xss",
+      "js-eval",
+      "js-function-constructor",
+      "js-path-traversal",
+      "js-spawn-shell",
+      "js-sql-injection",
+      "js-ssrf",
+      "js-weak-hash",
+      "js-xss-response",
+      "php-wpdb-sql-composition",
+      "php-sql-unquoted-concat",
+      "php-sql-unquoted-interpolation",
+      "php-sql-sprintf-unescaped",
+      "php-command-injection",
+      "php-file-inclusion-input",
+      "php-path-input",
+      "php-ssrf-input",
+      "php-open-redirect",
+      "python-flask-debug",
+      "python-os-command",
+      "python-path-traversal",
+      "python-shell-true",
+      "python-sql-injection",
+      "python-ssrf",
+      "python-ssti",
+      "python-unsafe-deserialization",
+      "python-unsafe-eval",
+      "python-weak-hash"
+    ],
+    "rewritten": [
+      {
+        "origin": "php-sql-sprintf-unescaped",
+        "rewrite": "bounded lazy group repeat {0,N}? widened to *? (PCRE2 size limit)"
+      }
+    ],
+    "skipped": [
+      {
+        "origin": "js-ssrf-language",
+        "reason": "fallback for js-ssrf: the tagged rule is active"
+      },
+      {
+        "origin": "php-sql-call-composition",
+        "reason": "shadow status (precision_estimate 0.25); not admitted by the hook"
+      },
+      {
+        "origin": "php-sql-unquoted-concat-language",
+        "reason": "fallback for php-sql-unquoted-concat: the tagged rule is active"
+      },
+      {
+        "origin": "php-sql-unquoted-interpolation-language",
+        "reason": "fallback for php-sql-unquoted-interpolation: the tagged rule is active"
+      },
+      {
+        "origin": "php-sql-sprintf-unescaped-language",
+        "reason": "fallback for php-sql-sprintf-unescaped: the tagged rule is active"
+      },
+      {
+        "origin": "php-code-eval",
+        "reason": "shadow status (precision_estimate 0.333); not admitted by the hook"
+      },
+      {
+        "origin": "php-unserialize",
+        "reason": "shadow status (precision_estimate 0.25); not admitted by the hook"
+      },
+      {
+        "origin": "php-maybe-unserialize",
+        "reason": "shadow status (precision_estimate 0.25); not admitted by the hook"
+      },
+      {
+        "origin": "php-xss-echo-input",
+        "reason": "shadow status (precision_estimate 0.333); not admitted by the hook"
+      },
+      {
+        "origin": "php-xss-echo-input-language",
+        "reason": "fallback for php-xss-echo-input: the tagged rule is active"
+      },
+      {
+        "origin": "python-unsafe-deserialization-language",
+        "reason": "fallback for python-unsafe-deserialization: the tagged rule is active"
+      }
+    ]
+  },
+  "validation": {
+    "exit": 0,
+    "stderr": "Configuration is valid - found 0 configuration error(s), and 81 rule(s).\n",
+    "stdout": ""
+  }
+}

benchmarks/measurements/2026-10-13-semgrep-arm/rules/groovy.yaml

@@ -0,0 +1,21 @@
+rules:
+- id: ousast.groovy.regex.groovy-xss-unescaped
+  languages:
+  - generic
+  severity: WARNING
+  message: Groovy template unescaped output needs review
+  metadata:
+    family: output_encoding
+    cwe: CWE-79
+    origin: groovy-xss-unescaped
+    precision_estimate: '0.0'
+    kind: regex
+    tags:
+    - injection
+    status: enabled
+    version: '1'
+  pattern-regex: (yieldUnescaped)\s+[A-Za-z_]
+  paths:
+    include:
+    - '*.groovy'
+    - '*.tpl'

benchmarks/measurements/2026-10-13-semgrep-arm/rules/java.yaml

@@ -0,0 +1,362 @@
+rules:
+- id: ousast.java.regex.java-command-exec
+  languages:
+  - java
+  severity: WARNING
+  message: Java runtime command execution needs review
+  metadata:
+    family: injection
+    cwe: CWE-78
+    origin: java-command-exec
+    precision_estimate: '0.0'
+    kind: regex
+    tags:
+    - syscall_entry
+    status: enabled
+    version: '1'
+  pattern-regex: \b(exec)\s*\(
+- id: ousast.java.regex.java-deserialization
+  languages:
+  - java
+  severity: WARNING
+  message: Java unsafe deserialization needs review
+  metadata:
+    family: deserialization
+    cwe: CWE-502
+    origin: java-deserialization
+    precision_estimate: '0.0'
+    kind: regex
+    tags:
+    - deserialization
+    status: enabled
+    version: '1'
+  pattern-regex: \.(readObject)\s*\(
+- id: ousast.java.regex.java-sql-injection
+  languages:
+  - java
+  severity: WARNING
+  message: Java SQL string composition needs review
+  metadata:
+    family: injection
+    cwe: CWE-89
+    origin: java-sql-injection
+    precision_estimate: '0.0'
+    kind: regex
+    tags:
+    - injection
+    status: enabled
+    version: '1'
+  pattern-regex: (?i)\"\s*(select|insert|update|delete)\b[^\"]*\"\s*\+
+- id: ousast.java.regex.java-weak-hash
+  languages:
+  - java
+  severity: WARNING
+  message: Java weak hash needs review
+  metadata:
+    family: config_secrets
+    cwe: CWE-327
+    origin: java-weak-hash
+    precision_estimate: '0.0'
+    kind: regex
+    tags:
+    - crypto
+    status: enabled
+    version: '1'
+  pattern-regex: MessageDigest\.getInstance\s*\(\s*\"(MD5|SHA-1)\"
+- id: ousast.java.weak.MessageDigest
+  languages: &id001
+  - java
+  severity: WARNING
+  message: 'MessageDigest: a weak literal reaches getInstance'
+  metadata:
+    family: config_secrets
+    cwe: CWE-327
+    origin: sink:java:MessageDigest
+    precision_estimate: unmeasured
+    kind: weak_literal
+    weak_literals:
+    - MD5
+    - SHA-1
+    - SHA1
+  pattern-either:
+  - patterns:
+    - pattern: getInstance($ALGORITHM, ...)
+    - metavariable-regex:
+        metavariable: $ALGORITHM
+        regex: ^["']?(MD5|SHA\-1|SHA1)["']?$
+  - patterns:
+    - pattern: $OBJ.getInstance($ALGORITHM, ...)
+    - metavariable-regex:
+        metavariable: $ALGORITHM
+        regex: ^["']?(MD5|SHA\-1|SHA1)["']?$
+- id: ousast.java.taint.exec
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'exec: untrusted input reaches exec'
+  metadata:
+    family: injection
+    cwe: CWE-78
+    origin: sink:java:exec
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - exec
+  pattern-sources:
+  - pattern-either: &id002
+    - pattern: getParameter
+    - pattern: $OBJ.getParameter
+    - pattern: getParameter(...)
+    - pattern: $OBJ.getParameter(...)
+    - pattern: getHeader
+    - pattern: $OBJ.getHeader
+    - pattern: getHeader(...)
+    - pattern: $OBJ.getHeader(...)
+    - pattern: getCookies
+    - pattern: $OBJ.getCookies
+    - pattern: getCookies(...)
+    - pattern: $OBJ.getCookies(...)
+    - pattern: getQueryString
+    - pattern: $OBJ.getQueryString
+    - pattern: getQueryString(...)
+    - pattern: $OBJ.getQueryString(...)
+    - pattern: readLine
+    - pattern: $OBJ.readLine
+    - pattern: readLine(...)
+    - pattern: $OBJ.readLine(...)
+  pattern-sinks:
+  - pattern-either:
+    - pattern: exec(...)
+    - pattern: $OBJ.exec(...)
+  pattern-sanitizers:
+  - pattern-either:
+    - &id003
+      pattern: getCanonicalPath(...)
+    - &id004
+      pattern: $OBJ.getCanonicalPath(...)
+    - &id005
+      pattern: normalize(...)
+    - &id006
+      pattern: $OBJ.normalize(...)
+    - &id007
+      pattern: URLEncoder.encode(...)
+    - &id008
+      pattern: ESAPI.encoder(...)
+    - &id009
+      pattern: encodeForHTML(...)
+    - &id010
+      pattern: $OBJ.encodeForHTML(...)
+    - &id011
+      pattern: encodeForURL(...)
+    - &id012
+      pattern: $OBJ.encodeForURL(...)
+    - &id013
+      pattern: StringEscapeUtils.escapeHtml4(...)
+- id: ousast.java.taint.executeQuery
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'executeQuery: untrusted input reaches execute, executeQuery, prepareCall'
+  metadata:
+    family: injection
+    cwe: CWE-89
+    origin: sink:java:executeQuery
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - execute
+    - executeQuery
+    - prepareCall
+  pattern-sources:
+  - pattern-either: *id002
+  pattern-sinks:
+  - pattern-either:
+    - pattern: execute(...)
+    - pattern: $OBJ.execute(...)
+    - pattern: executeQuery(...)
+    - pattern: $OBJ.executeQuery(...)
+    - pattern: prepareCall(...)
+    - pattern: $OBJ.prepareCall(...)
+  pattern-sanitizers:
+  - pattern-either:
+    - *id003
+    - *id004
+    - *id005
+    - *id006
+    - *id007
+    - *id008
+    - *id009
+    - *id010
+    - *id011
+    - *id012
+    - *id013
+- id: ousast.java.taint.file-open
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'file.open: untrusted input reaches File, FileInputStream, FileOutputStream, FileReader, FileWriter, Files.newInputStream, Files.readAllBytes, Paths.get, RandomAccessFile'
+  metadata:
+    family: path
+    cwe: CWE-22
+    origin: sink:java:file.open
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - File
+    - FileInputStream
+    - FileOutputStream
+    - FileReader
+    - FileWriter
+    - Files.newInputStream
+    - Files.readAllBytes
+    - Paths.get
+    - RandomAccessFile
+  pattern-sources:
+  - pattern-either: *id002
+  pattern-sinks:
+  - pattern-either:
+    - pattern: new File(...)
+    - pattern: new FileInputStream(...)
+    - pattern: new FileOutputStream(...)
+    - pattern: new FileReader(...)
+    - pattern: new FileWriter(...)
+    - pattern: Files.newInputStream(...)
+    - pattern: Files.readAllBytes(...)
+    - pattern: Paths.get(...)
+    - pattern: new RandomAccessFile(...)
+  pattern-sanitizers:
+  - pattern-either:
+    - *id003
+    - *id004
+    - *id005
+    - *id006
+    - *id007
+    - *id008
+    - *id009
+    - *id010
+    - *id011
+    - *id012
+    - *id013
+- id: ousast.java.taint.outbound-request
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'outbound.request: untrusted input reaches exchange, getForObject, openConnection, openStream, sendRedirect'
+  metadata:
+    family: untrusted_destination
+    cwe: CWE-918
+    origin: sink:java:outbound.request
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - exchange
+    - getForObject
+    - openConnection
+    - openStream
+    - sendRedirect
+  pattern-sources:
+  - pattern-either: *id002
+  pattern-sinks:
+  - pattern-either:
+    - pattern: exchange(...)
+    - pattern: $OBJ.exchange(...)
+    - pattern: getForObject(...)
+    - pattern: $OBJ.getForObject(...)
+    - pattern: openConnection(...)
+    - pattern: $OBJ.openConnection(...)
+    - pattern: openStream(...)
+    - pattern: $OBJ.openStream(...)
+    - pattern: sendRedirect(...)
+    - pattern: $OBJ.sendRedirect(...)
+  pattern-sanitizers:
+  - pattern-either:
+    - *id003
+    - *id004
+    - *id005
+    - *id006
+    - *id007
+    - *id008
+    - *id009
+    - *id010
+    - *id011
+    - *id012
+    - *id013
+- id: ousast.java.taint.readObject
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'readObject: untrusted input reaches readObject'
+  metadata:
+    family: deserialization
+    cwe: CWE-502
+    origin: sink:java:readObject
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - readObject
+  pattern-sources:
+  - pattern-either: *id002
+  pattern-sinks:
+  - pattern-either:
+    - pattern: readObject(...)
+    - pattern: $OBJ.readObject(...)
+  pattern-sanitizers:
+  - pattern-either:
+    - *id003
+    - *id004
+    - *id005
+    - *id006
+    - *id007
+    - *id008
+    - *id009
+    - *id010
+    - *id011
+    - *id012
+    - *id013
+- id: ousast.java.taint.response-write
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'response.write: untrusted input reaches addHeader, getWriter, print, println, setHeader, write'
+  metadata:
+    family: output_encoding
+    cwe: CWE-79
+    origin: sink:java:response.write
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - addHeader
+    - getWriter
+    - print
+    - println
+    - setHeader
+    - write
+  pattern-sources:
+  - pattern-either: *id002
+  pattern-sinks:
+  - pattern-either:
+    - pattern: addHeader(...)
+    - pattern: $OBJ.addHeader(...)
+    - pattern: getWriter(...)
+    - pattern: $OBJ.getWriter(...)
+    - pattern: print(...)
+    - pattern: $OBJ.print(...)
+    - pattern: println(...)
+    - pattern: $OBJ.println(...)
+    - pattern: setHeader(...)
+    - pattern: $OBJ.setHeader(...)
+    - pattern: write(...)
+    - pattern: $OBJ.write(...)
+  pattern-sanitizers:
+  - pattern-either:
+    - *id003
+    - *id004
+    - *id005
+    - *id006
+    - *id007
+    - *id008
+    - *id009
+    - *id010
+    - *id011
+    - *id012
+    - *id013

benchmarks/measurements/2026-10-13-semgrep-arm/rules/javascript.yaml

@@ -0,0 +1,679 @@
+rules:
+- id: ousast.javascript.regex.js-command-exec
+  languages:
+  - javascript
+  - typescript
+  severity: WARNING
+  message: Node child_process command execution needs review
+  metadata:
+    family: injection
+    cwe: CWE-78
+    origin: js-command-exec
+    precision_estimate: '0.0'
+    kind: regex
+    tags:
+    - syscall_entry
+    status: enabled
+    version: '1'
+  pattern-regex: \b(exec|execSync)\s*\(
+- id: ousast.javascript.regex.js-deserialize
+  languages:
+  - javascript
+  - typescript
+  severity: WARNING
+  message: JavaScript unsafe deserialization needs review
+  metadata:
+    family: deserialization
+    cwe: CWE-502
+    origin: js-deserialize
+    precision_estimate: '0.0'
+    kind: regex
+    tags:
+    - deserialization
+    status: enabled
+    version: '1'
+  pattern-regex: \b(unserialize)\s*\(
+- id: ousast.javascript.regex.js-dom-xss
+  languages:
+  - javascript
+  - typescript
+  severity: WARNING
+  message: DOM sink with untrusted input needs review
+  metadata:
+    family: output_encoding
+    cwe: CWE-79
+    origin: js-dom-xss
+    precision_estimate: '0.0'
+    kind: regex
+    tags:
+    - injection
+    status: enabled
+    version: '1'
+  pattern-regex: \.(innerHTML|outerHTML)\s*=\s*[^\n]*\b(?:req|request|location|document\.location)
+- id: ousast.javascript.regex.js-eval
+  languages:
+  - javascript
+  - typescript
+  severity: WARNING
+  message: Dynamic JavaScript evaluation needs review
+  metadata:
+    family: injection
+    cwe: CWE-95
+    origin: js-eval
+    precision_estimate: '0.0'
+    kind: regex
+    tags:
+    - syscall_entry
+    status: enabled
+    version: '1'
+  pattern-regex: \b(eval)\s*\(
+- id: ousast.javascript.regex.js-function-constructor
+  languages:
+  - javascript
+  - typescript
+  severity: WARNING
+  message: JavaScript Function constructor needs review
+  metadata:
+    family: injection
+    cwe: CWE-95
+    origin: js-function-constructor
+    precision_estimate: '0.0'
+    kind: regex
+    tags:
+    - syscall_entry
+    status: enabled
+    version: '1'
+  pattern-regex: new\s+(Function)\s*\(
+- id: ousast.javascript.regex.js-path-traversal
+  languages:
+  - javascript
+  - typescript
+  severity: WARNING
+  message: Node filesystem access with external input needs review
+  metadata:
+    family: path
+    cwe: CWE-22
+    origin: js-path-traversal
+    precision_estimate: '0.0'
+    kind: regex
+    tags:
+    - filesystem_entry
+    status: enabled
+    version: '1'
+  pattern-regex: \bfs\.(readFile|readFileSync|writeFile|writeFileSync|createReadStream|unlink|unlinkSync)\s*\(\s*[^\n]*\b(?:req|request)\.
+- id: ousast.javascript.regex.js-spawn-shell
+  languages:
+  - javascript
+  - typescript
+  severity: WARNING
+  message: Node spawn with shell needs review
+  metadata:
+    family: injection
+    cwe: CWE-78
+    origin: js-spawn-shell
+    precision_estimate: '0.0'
+    kind: regex
+    tags:
+    - syscall_entry
+    status: enabled
+    version: '1'
+  pattern-regex: \b(spawn|spawnSync)\s*\([^\n]*shell\s*:\s*true
+- id: ousast.javascript.regex.js-sql-injection
+  languages:
+  - javascript
+  - typescript
+  severity: WARNING
+  message: JavaScript SQL string composition needs review
+  metadata:
+    family: injection
+    cwe: CWE-89
+    origin: js-sql-injection
+    precision_estimate: '0.0'
+    kind: regex
+    tags:
+    - injection
+    status: enabled
+    version: '1'
+  pattern-regex: \.(query|execute)\s*\(\s*[^\n]*(?:\$\{|\+\s*[A-Za-z_$])
+- id: ousast.javascript.regex.js-ssrf
+  languages:
+  - javascript
+  - typescript
+  severity: WARNING
+  message: JavaScript outbound request with external input needs review
+  metadata:
+    family: untrusted_destination
+    cwe: CWE-918
+    origin: js-ssrf
+    precision_estimate: '0.0'
+    kind: regex
+    tags:
+    - network_entry
+    status: enabled
+    version: '1'
+  pattern-regex: \b(axios|fetch|got)\s*[.(][^\n]*\b(?:req|request)\.
+- id: ousast.javascript.regex.js-weak-hash
+  languages:
+  - javascript
+  - typescript
+  severity: WARNING
+  message: JavaScript weak hash needs review
+  metadata:
+    family: config_secrets
+    cwe: CWE-327
+    origin: js-weak-hash
+    precision_estimate: '0.0'
+    kind: regex
+    tags:
+    - crypto
+    status: enabled
+    version: '1'
+  pattern-regex: createHash\s*\(\s*[\"'](md5|sha1)[\"']
+- id: ousast.javascript.regex.js-xss-response
+  languages:
+  - javascript
+  - typescript
+  severity: WARNING
+  message: Unsanitized HTTP response reflection needs review
+  metadata:
+    family: output_encoding
+    cwe: CWE-79
+    origin: js-xss-response
+    precision_estimate: '0.0'
+    kind: regex
+    tags:
+    - injection
+    status: enabled
+    version: '1'
+  pattern-regex: \.(send|write|end)\s*\(\s*[^\n]*\b(?:req|request)\.(?:query|params|body)
+- id: ousast.javascript.taint.eval
+  mode: taint
+  languages: &id001
+  - javascript
+  - typescript
+  severity: ERROR
+  message: 'eval: untrusted input reaches eval'
+  metadata:
+    family: injection
+    cwe: CWE-95
+    origin: sink:javascript:eval
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - eval
+  pattern-sources:
+  - pattern-either: &id002
+    - pattern: req.query
+    - pattern: req.query(...)
+    - pattern: req.body
+    - pattern: req.body(...)
+    - pattern: req.params
+    - pattern: req.params(...)
+    - pattern: req.cookies
+    - pattern: req.cookies(...)
+    - pattern: req.headers
+    - pattern: req.headers(...)
+    - pattern: request.query
+    - pattern: request.query(...)
+    - pattern: req.url
+    - pattern: req.url(...)
+    - pattern: request.url
+    - pattern: request.url(...)
+    - pattern: req.rawHeaders
+    - pattern: req.rawHeaders(...)
+    - pattern: url.parse
+    - pattern: url.parse(...)
+    - pattern: parsedUrl
+    - pattern: $OBJ.parsedUrl
+    - pattern: parsedUrl(...)
+    - pattern: $OBJ.parsedUrl(...)
+  pattern-sinks:
+  - pattern-either:
+    - pattern: eval(...)
+    - pattern: $OBJ.eval(...)
+  pattern-sanitizers:
+  - pattern-either:
+    - &id003
+      pattern: path.normalize(...)
+    - &id004
+      pattern: path.resolve(...)
+    - &id005
+      pattern: encodeURIComponent(...)
+    - &id006
+      pattern: $OBJ.encodeURIComponent(...)
+    - &id007
+      pattern: encodeURI(...)
+    - &id008
+      pattern: $OBJ.encodeURI(...)
+    - &id009
+      pattern: escape(...)
+    - &id010
+      pattern: $OBJ.escape(...)
+    - &id011
+      pattern: escapeHtml(...)
+    - &id012
+      pattern: $OBJ.escapeHtml(...)
+    - &id013
+      pattern: sanitizeHtml(...)
+    - &id014
+      pattern: $OBJ.sanitizeHtml(...)
+    - &id015
+      pattern: DOMPurify.sanitize(...)
+- id: ousast.javascript.taint.exec
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'exec: untrusted input reaches exec, execSync'
+  metadata:
+    family: injection
+    cwe: CWE-78
+    origin: sink:javascript:exec
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - exec
+    - execSync
+  pattern-sources:
+  - pattern-either: *id002
+  pattern-sinks:
+  - pattern-either:
+    - pattern: exec(...)
+    - pattern: $OBJ.exec(...)
+    - pattern: execSync(...)
+    - pattern: $OBJ.execSync(...)
+  pattern-sanitizers:
+  - pattern-either:
+    - *id003
+    - *id004
+    - *id005
+    - *id006
+    - *id007
+    - *id008
+    - *id009
+    - *id010
+    - *id011
+    - *id012
+    - *id013
+    - *id014
+    - *id015
+- id: ousast.javascript.taint.html-response
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'html.response: untrusted input reaches res.end, res.send, res.write, response.end, response.send, response.write'
+  metadata:
+    family: output_encoding
+    cwe: CWE-79
+    origin: sink:javascript:html.response
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - res.end
+    - res.send
+    - res.write
+    - response.end
+    - response.send
+    - response.write
+  pattern-sources:
+  - pattern-either: *id002
+  pattern-sinks:
+  - pattern-either:
+    - pattern: res.end(...)
+    - pattern: res.send(...)
+    - pattern: res.write(...)
+    - pattern: response.end(...)
+    - pattern: response.send(...)
+    - pattern: response.write(...)
+  pattern-sanitizers:
+  - pattern-either:
+    - *id003
+    - *id004
+    - *id005
+    - *id006
+    - *id007
+    - *id008
+    - *id009
+    - *id010
+    - *id011
+    - *id012
+    - *id013
+    - *id014
+    - *id015
+- id: ousast.javascript.taint.html-unescaped
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'html.unescaped: untrusted input reaches dangerouslySetInnerHTML, document.write, innerHTML, insertAdjacentHTML, outerHTML, v-html'
+  metadata:
+    family: output_encoding
+    cwe: CWE-79
+    origin: sink:javascript:html.unescaped
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - dangerouslySetInnerHTML
+    - document.write
+    - innerHTML
+    - insertAdjacentHTML
+    - outerHTML
+    - v-html
+  pattern-sources:
+  - pattern-either: *id002
+  pattern-sinks:
+  - pattern-either:
+    - patterns:
+      - pattern: '{__html: $VALUE}'
+      - focus-metavariable: $VALUE
+    - pattern: document.write(...)
+    - patterns:
+      - pattern: $OBJ.innerHTML = $VALUE
+      - focus-metavariable: $VALUE
+    - pattern: insertAdjacentHTML(...)
+    - pattern: $OBJ.insertAdjacentHTML(...)
+    - patterns:
+      - pattern: $OBJ.outerHTML = $VALUE
+      - focus-metavariable: $VALUE
+  pattern-sanitizers:
+  - pattern-either:
+    - *id003
+    - *id004
+    - *id005
+    - *id006
+    - *id007
+    - *id008
+    - *id009
+    - *id010
+    - *id011
+    - *id012
+    - *id013
+    - *id014
+    - *id015
+- id: ousast.javascript.taint.outbound-request
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'outbound.request: untrusted input reaches axios, fetch, got, http.get, http.request, https.get, https.request, needle, request, superagent, undici, unirest'
+  metadata:
+    family: untrusted_destination
+    cwe: CWE-918
+    origin: sink:javascript:outbound.request
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - axios
+    - fetch
+    - got
+    - http.get
+    - http.request
+    - https.get
+    - https.request
+    - needle
+    - request
+    - superagent
+    - undici
+    - unirest
+  pattern-sources:
+  - pattern-either: *id002
+  pattern-sinks:
+  - pattern-either:
+    - pattern: axios(...)
+    - pattern: $OBJ.axios(...)
+    - pattern: fetch(...)
+    - pattern: $OBJ.fetch(...)
+    - pattern: got(...)
+    - pattern: $OBJ.got(...)
+    - pattern: http.get(...)
+    - pattern: http.request(...)
+    - pattern: https.get(...)
+    - pattern: https.request(...)
+    - pattern: needle(...)
+    - pattern: $OBJ.needle(...)
+    - pattern: request(...)
+    - pattern: $OBJ.request(...)
+    - pattern: superagent(...)
+    - pattern: $OBJ.superagent(...)
+    - pattern: undici(...)
+    - pattern: $OBJ.undici(...)
+    - pattern: unirest(...)
+    - pattern: $OBJ.unirest(...)
+  pattern-sanitizers:
+  - pattern-either:
+    - *id003
+    - *id004
+    - *id005
+    - *id006
+    - *id007
+    - *id008
+    - *id009
+    - *id010
+    - *id011
+    - *id012
+    - *id013
+    - *id014
+    - *id015
+- id: ousast.javascript.taint.path-open
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'path.open: untrusted input reaches createReadStream, readFile, readFileSync, sendFile, unlink, unlinkSync'
+  metadata:
+    family: path
+    cwe: CWE-22
+    origin: sink:javascript:path.open
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - createReadStream
+    - readFile
+    - readFileSync
+    - sendFile
+    - unlink
+    - unlinkSync
+  pattern-sources:
+  - pattern-either: *id002
+  pattern-sinks:
+  - pattern-either:
+    - pattern: createReadStream(...)
+    - pattern: $OBJ.createReadStream(...)
+    - pattern: readFile(...)
+    - pattern: $OBJ.readFile(...)
+    - pattern: readFileSync(...)
+    - pattern: $OBJ.readFileSync(...)
+    - pattern: sendFile(...)
+    - pattern: $OBJ.sendFile(...)
+    - pattern: unlink(...)
+    - pattern: $OBJ.unlink(...)
+    - pattern: unlinkSync(...)
+    - pattern: $OBJ.unlinkSync(...)
+  pattern-sanitizers:
+  - pattern-either:
+    - *id003
+    - *id004
+    - *id005
+    - *id006
+    - *id007
+    - *id008
+    - *id009
+    - *id010
+    - *id011
+    - *id012
+    - *id013
+    - *id014
+    - *id015
+- id: ousast.javascript.taint.prototype-merge
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'prototype.merge: untrusted input reaches Object.assign, defaultsDeep, extend, merge, mergeDeep, set, setWith'
+  metadata:
+    family: prototype
+    cwe: CWE-1321
+    origin: sink:javascript:prototype.merge
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - Object.assign
+    - defaultsDeep
+    - extend
+    - merge
+    - mergeDeep
+    - set
+    - setWith
+  pattern-sources:
+  - pattern-either: *id002
+  pattern-sinks:
+  - pattern-either:
+    - pattern: Object.assign(...)
+    - pattern: defaultsDeep(...)
+    - pattern: $OBJ.defaultsDeep(...)
+    - pattern: extend(...)
+    - pattern: $OBJ.extend(...)
+    - pattern: merge(...)
+    - pattern: $OBJ.merge(...)
+    - pattern: mergeDeep(...)
+    - pattern: $OBJ.mergeDeep(...)
+    - pattern: set(...)
+    - pattern: $OBJ.set(...)
+    - pattern: setWith(...)
+    - pattern: $OBJ.setWith(...)
+  pattern-sanitizers:
+  - pattern-either:
+    - *id003
+    - *id004
+    - *id005
+    - *id006
+    - *id007
+    - *id008
+    - *id009
+    - *id010
+    - *id011
+    - *id012
+    - *id013
+    - *id014
+    - *id015
+- id: ousast.javascript.taint.query
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'query: untrusted input reaches execute, query'
+  metadata:
+    family: injection
+    cwe: CWE-89
+    origin: sink:javascript:query
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - execute
+    - query
+  pattern-sources:
+  - pattern-either: *id002
+  pattern-sinks:
+  - pattern-either:
+    - patterns:
+      - pattern: execute($QUERY)
+      - focus-metavariable: $QUERY
+    - patterns:
+      - pattern: $OBJ.execute($QUERY)
+      - focus-metavariable: $QUERY
+    - patterns:
+      - pattern: query($QUERY)
+      - focus-metavariable: $QUERY
+    - patterns:
+      - pattern: $OBJ.query($QUERY)
+      - focus-metavariable: $QUERY
+  pattern-sanitizers:
+  - pattern-either:
+    - *id003
+    - *id004
+    - *id005
+    - *id006
+    - *id007
+    - *id008
+    - *id009
+    - *id010
+    - *id011
+    - *id012
+    - *id013
+    - *id014
+    - *id015
+- id: ousast.javascript.taint.redirect
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'redirect: untrusted input reaches location.assign, location.replace, redirect'
+  metadata:
+    family: untrusted_destination
+    cwe: CWE-601
+    origin: sink:javascript:redirect
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - location.assign
+    - location.replace
+    - redirect
+  pattern-sources:
+  - pattern-either: *id002
+  pattern-sinks:
+  - pattern-either:
+    - pattern: location.assign(...)
+    - pattern: location.replace(...)
+    - pattern: redirect(...)
+    - pattern: $OBJ.redirect(...)
+  pattern-sanitizers:
+  - pattern-either:
+    - *id003
+    - *id004
+    - *id005
+    - *id006
+    - *id007
+    - *id008
+    - *id009
+    - *id010
+    - *id011
+    - *id012
+    - *id013
+    - *id014
+    - *id015
+- id: ousast.javascript.taint.vm-eval
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'vm.eval: untrusted input reaches compileFunction, runInContext, runInNewContext, runInThisContext'
+  metadata:
+    family: injection
+    cwe: CWE-95
+    origin: sink:javascript:vm.eval
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - compileFunction
+    - runInContext
+    - runInNewContext
+    - runInThisContext
+  pattern-sources:
+  - pattern-either: *id002
+  pattern-sinks:
+  - pattern-either:
+    - pattern: compileFunction(...)
+    - pattern: $OBJ.compileFunction(...)
+    - pattern: runInContext(...)
+    - pattern: $OBJ.runInContext(...)
+    - pattern: runInNewContext(...)
+    - pattern: $OBJ.runInNewContext(...)
+    - pattern: runInThisContext(...)
+    - pattern: $OBJ.runInThisContext(...)
+  pattern-sanitizers:
+  - pattern-either:
+    - *id003
+    - *id004
+    - *id005
+    - *id006
+    - *id007
+    - *id008
+    - *id009
+    - *id010
+    - *id011
+    - *id012
+    - *id013
+    - *id014
+    - *id015

benchmarks/measurements/2026-10-13-semgrep-arm/rules/php.yaml

@@ -0,0 +1,647 @@
+rules:
+- id: ousast.php.regex.php-wpdb-sql-composition
+  languages:
+  - php
+  severity: WARNING
+  message: WordPress $wpdb query composed from a variable without $wpdb->prepare
+  metadata:
+    family: injection
+    cwe: CWE-89
+    origin: php-wpdb-sql-composition
+    precision_estimate: '0.032'
+    kind: regex
+    tags:
+    - injection
+    status: enabled
+    version: '1'
+  pattern-regex: \$wpdb->(get_var|get_results|get_row|get_col|query|replace)\s*\(\s*(?!\$wpdb->prepare)(?:"[^"\n]*\{?\$(?!wpdb->)[A-Za-z_]|[^\n;]*?["']\s*\.\s*\$(?!wpdb->)[A-Za-z_]|[^\n;]*?\$_(?:GET|POST|REQUEST|COOKIE)\b)
+- id: ousast.php.regex.php-sql-unquoted-concat
+  languages:
+  - php
+  severity: WARNING
+  message: PHP SQL literal concatenates a variable into an unquoted position
+  metadata:
+    family: injection
+    cwe: CWE-89
+    origin: php-sql-unquoted-concat
+    precision_estimate: '0.034'
+    kind: regex
+    tags:
+    - injection
+    status: enabled
+    version: '1'
+  pattern-regex: (?<![$\w>])(?=(VALUES|IN|ORDER|GROUP|LIMIT|OFFSET|WHERE|AND|OR|SET|HAVING)\b)(?:(?:VALUES|IN)\s*\(|(?:ORDER|GROUP)\s+BY\b[^"'\n]*|(?:LIMIT|OFFSET)\s+(?:\d+\s*,\s*)?|(?:WHERE|AND|OR|SET|HAVING)\s+[\w.`]+\s*(?:=|<>|!=|<=|>=|<|>|LIKE)\s*)["']\s*\.\s*\$(?!wpdb->)(?!\w*(?:format|placeholder))[A-Za-z_]
+- id: ousast.php.regex.php-sql-unquoted-interpolation
+  languages:
+  - php
+  severity: WARNING
+  message: PHP SQL string interpolates a variable into an unquoted position
+  metadata:
+    family: injection
+    cwe: CWE-89
+    origin: php-sql-unquoted-interpolation
+    precision_estimate: '0.129'
+    kind: regex
+    tags:
+    - injection
+    status: enabled
+    version: '1'
+  pattern-regex: (?<![$\w>])(?=(VALUES|IN|ORDER|GROUP|LIMIT|OFFSET|WHERE|AND|OR|SET|HAVING)\b)(?:(?:VALUES|IN)\s*\(\s*|(?:ORDER|GROUP)\s+BY\b[^"'\n;]*?|(?:LIMIT|OFFSET)\s+(?:\d+\s*,\s*)?|(?:WHERE|AND|OR|SET|HAVING)\s+[\w.`]+\s*(?:=|<>|!=|<=|>=|<|>|LIKE)\s*)\{?\$(?!wpdb->)(?!\w*(?:format|placeholder))[A-Za-z_]
+- id: ousast.php.regex.php-sql-sprintf-unescaped
+  languages:
+  - php
+  severity: WARNING
+  message: PHP sprintf-built SQL with an unescaped %s value
+  metadata:
+    family: injection
+    cwe: CWE-89
+    origin: php-sql-sprintf-unescaped
+    precision_estimate: '0.109'
+    kind: regex
+    tags:
+    - injection
+    status: enabled
+    version: '1'
+  pattern-regex: (sprintf)\(\s*["']\s*(?:SELECT|INSERT|UPDATE|DELETE|REPLACE)\b(?=(?:(?!\);)[\s\S])*?'%s')(?=(?:(?![Ee]scape|quote\(|intval\(|absint\(|\(int\)|esc_sql\()[\s\S])*?\);)
+- id: ousast.php.regex.php-command-injection
+  languages:
+  - php
+  severity: WARNING
+  message: PHP shell command built from a variable without escapeshellarg
+  metadata:
+    family: injection
+    cwe: CWE-78
+    origin: php-command-injection
+    precision_estimate: '1.0'
+    kind: regex
+    tags:
+    - syscall_entry
+    status: enabled
+    version: '1'
+  pattern-regex: (?<![\w>:$\\])(?<!function )(system|exec|shell_exec|passthru|popen|proc_open|pcntl_exec)\s*\((?![^\n]*escapeshell(?:arg|cmd)\s*\()[^,\n;]*\$[A-Za-z_]
+- id: ousast.php.regex.php-file-inclusion-input
+  languages:
+  - php
+  severity: WARNING
+  message: PHP include/require of a path from request input
+  metadata:
+    family: path
+    cwe: CWE-98
+    origin: php-file-inclusion-input
+    precision_estimate: '0.5'
+    kind: regex
+    tags:
+    - filesystem_entry
+    status: enabled
+    version: '1'
+  pattern-regex: (?m)^(?![^\n]*\b(?:basename|realpath)\s*\()[^\n]*?(?<![\w'"$>:-])(include|include_once|require|require_once)\b(?!['"\]])\s*\(?[^\n;]*\$_(?:GET|POST|REQUEST|COOKIE)\b
+- id: ousast.php.regex.php-path-input
+  languages:
+  - php
+  severity: WARNING
+  message: PHP filesystem call on a path from request input
+  metadata:
+    family: path
+    cwe: CWE-22
+    origin: php-path-input
+    precision_estimate: '1.0'
+    kind: regex
+    tags:
+    - filesystem_entry
+    status: enabled
+    version: '1'
+  pattern-regex: (?m)^(?![^\n]*\b(?:basename|realpath)\s*\()[^\n]*?(?<![\w>:$\\])(fopen|file_get_contents|file_put_contents|readfile|unlink|copy|rename|scandir|opendir|mkdir|rmdir|move_uploaded_file)\s*\([^\n;]*\$_(?:GET|POST|REQUEST|COOKIE)\b
+- id: ousast.php.regex.php-ssrf-input
+  languages:
+  - php
+  severity: WARNING
+  message: PHP outbound request to a URL from request input
+  metadata:
+    family: untrusted_destination
+    cwe: CWE-918
+    origin: php-ssrf-input
+    precision_estimate: '1.0'
+    kind: regex
+    tags:
+    - network_entry
+    status: enabled
+    version: '1'
+  pattern-regex: (?<![\w>:$\\])(curl_init|curl_setopt|fsockopen|stream_socket_client|get_headers)\s*\([^\n;]*\$_(?:GET|POST|REQUEST|COOKIE)\b
+- id: ousast.php.regex.php-open-redirect
+  languages:
+  - php
+  severity: WARNING
+  message: PHP Location header from request input
+  metadata:
+    family: untrusted_destination
+    cwe: CWE-601
+    origin: php-open-redirect
+    precision_estimate: '1.0'
+    kind: regex
+    tags:
+    - injection
+    status: enabled
+    version: '1'
+  pattern-regex: (?<![\w>:$\\])(header)\s*\(\s*["']Location:[^\n;]*(?:\$_(?:GET|POST|REQUEST|COOKIE)\b|HTTP_REFERER)
+- id: ousast.php.taint.code_eval
+  mode: taint
+  languages: &id001
+  - php
+  severity: ERROR
+  message: 'code_eval: untrusted input reaches assert, create_function, eval'
+  metadata:
+    family: injection
+    cwe: CWE-95
+    origin: sink:php:code_eval
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - assert
+    - create_function
+    - eval
+  pattern-sources:
+  - pattern-either: &id002
+    - pattern: $_GET
+    - pattern: $_GET[...]
+    - pattern: $_POST
+    - pattern: $_POST[...]
+    - pattern: $_REQUEST
+    - pattern: $_REQUEST[...]
+    - pattern: $_COOKIE
+    - pattern: $_COOKIE[...]
+    - pattern: $_FILES
+    - pattern: $_FILES[...]
+    - pattern: $_SERVER
+    - pattern: $_SERVER[...]
+    - pattern: file_get_contents("php://input")
+  pattern-sinks:
+  - pattern-either:
+    - pattern: assert(...)
+    - pattern: create_function(...)
+    - pattern: eval(...)
+  pattern-sanitizers:
+  - pattern-either:
+    - &id003
+      pattern: htmlspecialchars(...)
+    - &id004
+      pattern: htmlentities(...)
+    - &id005
+      pattern: html_entity_decode_safe(...)
+    - &id006
+      pattern: mysqli_real_escape_string(...)
+    - &id007
+      pattern: mysql_real_escape_string(...)
+    - &id008
+      pattern: pg_escape_string(...)
+    - &id009
+      pattern: esc_sql(...)
+    - &id010
+      pattern: pg_escape_literal(...)
+    - &id011
+      pattern: $wpdb->prepare(...)
+    - &id012
+      pattern: escapeshellarg(...)
+    - &id013
+      pattern: escapeshellcmd(...)
+    - &id014
+      pattern: basename(...)
+    - &id015
+      pattern: realpath(...)
+    - &id016
+      pattern: intval(...)
+    - &id017
+      pattern: floatval(...)
+    - &id018
+      pattern: filter_var(...)
+    - &id019
+      pattern: settype(...)
+    - &id020
+      pattern: absint(...)
+- id: ousast.php.taint.command
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'command: untrusted input reaches exec, passthru, pcntl_exec, popen, proc_open, shell_exec, system'
+  metadata:
+    family: injection
+    cwe: CWE-78
+    origin: sink:php:command
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - exec
+    - passthru
+    - pcntl_exec
+    - popen
+    - proc_open
+    - shell_exec
+    - system
+  pattern-sources:
+  - pattern-either: *id002
+  pattern-sinks:
+  - pattern-either:
+    - pattern: exec(...)
+    - pattern: passthru(...)
+    - pattern: pcntl_exec(...)
+    - pattern: popen(...)
+    - pattern: proc_open(...)
+    - pattern: shell_exec(...)
+    - pattern: system(...)
+  pattern-sanitizers:
+  - pattern-either:
+    - *id003
+    - *id004
+    - *id005
+    - *id006
+    - *id007
+    - *id008
+    - *id009
+    - *id010
+    - *id011
+    - *id012
+    - *id013
+    - *id014
+    - *id015
+    - *id016
+    - *id017
+    - *id018
+    - *id019
+    - *id020
+- id: ousast.php.taint.echo
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'echo: untrusted input reaches echo, fwrite, print, print_r, printf, var_dump, vprintf'
+  metadata:
+    family: output_encoding
+    cwe: CWE-79
+    origin: sink:php:echo
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - echo
+    - fwrite
+    - print
+    - print_r
+    - printf
+    - var_dump
+    - vprintf
+  pattern-sources:
+  - pattern-either: *id002
+  pattern-sinks:
+  - pattern-either:
+    - pattern: echo $VALUE;
+    - pattern: fwrite(...)
+    - pattern: print $VALUE;
+    - pattern: print_r(...)
+    - pattern: printf(...)
+    - pattern: var_dump(...)
+    - pattern: vprintf(...)
+  pattern-sanitizers:
+  - pattern-either:
+    - *id003
+    - *id004
+    - *id005
+    - *id006
+    - *id007
+    - *id008
+    - *id009
+    - *id010
+    - *id011
+    - *id012
+    - *id013
+    - *id014
+    - *id015
+    - *id016
+    - *id017
+    - *id018
+    - *id019
+    - *id020
+- id: ousast.php.taint.file_inclusion
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'file_inclusion: untrusted input reaches include, include_once, require, require_once'
+  metadata:
+    family: path
+    cwe: CWE-98
+    origin: sink:php:file_inclusion
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - include
+    - include_once
+    - require
+    - require_once
+  pattern-sources:
+  - pattern-either: *id002
+  pattern-sinks:
+  - pattern-either:
+    - pattern: include ...
+    - pattern: include_once ...
+    - pattern: require ...
+    - pattern: require_once ...
+  pattern-sanitizers:
+  - pattern-either:
+    - *id003
+    - *id004
+    - *id005
+    - *id006
+    - *id007
+    - *id008
+    - *id009
+    - *id010
+    - *id011
+    - *id012
+    - *id013
+    - *id014
+    - *id015
+    - *id016
+    - *id017
+    - *id018
+    - *id019
+    - *id020
+- id: ousast.php.taint.filesystem
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'filesystem: untrusted input reaches copy, file_get_contents, file_put_contents, fopen, mkdir, move_uploaded_file, opendir, readfile, rename, rmdir, scandir, unlink'
+  metadata:
+    family: path
+    cwe: CWE-22
+    origin: sink:php:filesystem
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - copy
+    - file_get_contents
+    - file_put_contents
+    - fopen
+    - mkdir
+    - move_uploaded_file
+    - opendir
+    - readfile
+    - rename
+    - rmdir
+    - scandir
+    - unlink
+  pattern-sources:
+  - pattern-either: *id002
+  pattern-sinks:
+  - pattern-either:
+    - pattern: copy(...)
+    - pattern: file_get_contents(...)
+    - pattern: file_put_contents(...)
+    - pattern: fopen(...)
+    - pattern: mkdir(...)
+    - pattern: move_uploaded_file(...)
+    - pattern: opendir(...)
+    - pattern: readfile(...)
+    - pattern: rename(...)
+    - pattern: rmdir(...)
+    - pattern: scandir(...)
+    - pattern: unlink(...)
+  pattern-sanitizers:
+  - pattern-either:
+    - *id003
+    - *id004
+    - *id005
+    - *id006
+    - *id007
+    - *id008
+    - *id009
+    - *id010
+    - *id011
+    - *id012
+    - *id013
+    - *id014
+    - *id015
+    - *id016
+    - *id017
+    - *id018
+    - *id019
+    - *id020
+- id: ousast.php.taint.outbound_request
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'outbound_request: untrusted input reaches curl_exec, curl_setopt, fsockopen, get_headers, stream_socket_client'
+  metadata:
+    family: untrusted_destination
+    cwe: CWE-918
+    origin: sink:php:outbound_request
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - curl_exec
+    - curl_setopt
+    - fsockopen
+    - get_headers
+    - stream_socket_client
+  pattern-sources:
+  - pattern-either: *id002
+  pattern-sinks:
+  - pattern-either:
+    - pattern: curl_exec(...)
+    - pattern: curl_setopt(...)
+    - pattern: fsockopen(...)
+    - pattern: get_headers(...)
+    - pattern: stream_socket_client(...)
+  pattern-sanitizers:
+  - pattern-either:
+    - *id003
+    - *id004
+    - *id005
+    - *id006
+    - *id007
+    - *id008
+    - *id009
+    - *id010
+    - *id011
+    - *id012
+    - *id013
+    - *id014
+    - *id015
+    - *id016
+    - *id017
+    - *id018
+    - *id019
+    - *id020
+- id: ousast.php.taint.redirect
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'redirect: untrusted input reaches header'
+  metadata:
+    family: untrusted_destination
+    cwe: CWE-601
+    origin: sink:php:redirect
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - header
+  pattern-sources:
+  - pattern-either: *id002
+  pattern-sinks:
+  - pattern-either:
+    - pattern: header(...)
+  pattern-sanitizers:
+  - pattern-either:
+    - *id003
+    - *id004
+    - *id005
+    - *id006
+    - *id007
+    - *id008
+    - *id009
+    - *id010
+    - *id011
+    - *id012
+    - *id013
+    - *id014
+    - *id015
+    - *id016
+    - *id017
+    - *id018
+    - *id019
+    - *id020
+- id: ousast.php.taint.sql
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'sql: untrusted input reaches mysql_query, mysqli_multi_query, mysqli_query, mysqli_real_query, pg_query, pg_send_query, sqlite_query, sqlite_unbuffered_query'
+  metadata:
+    family: injection
+    cwe: CWE-89
+    origin: sink:php:sql
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - mysql_query
+    - mysqli_multi_query
+    - mysqli_query
+    - mysqli_real_query
+    - pg_query
+    - pg_send_query
+    - sqlite_query
+    - sqlite_unbuffered_query
+  pattern-sources:
+  - pattern-either: *id002
+  pattern-sinks:
+  - pattern-either:
+    - pattern: mysql_query(...)
+    - pattern: mysqli_multi_query(...)
+    - pattern: mysqli_query(...)
+    - pattern: mysqli_real_query(...)
+    - pattern: pg_query(...)
+    - pattern: pg_send_query(...)
+    - pattern: sqlite_query(...)
+    - pattern: sqlite_unbuffered_query(...)
+  pattern-sanitizers:
+  - pattern-either:
+    - *id003
+    - *id004
+    - *id005
+    - *id006
+    - *id007
+    - *id008
+    - *id009
+    - *id010
+    - *id011
+    - *id012
+    - *id013
+    - *id014
+    - *id015
+    - *id016
+    - *id017
+    - *id018
+    - *id019
+    - *id020
+- id: ousast.php.taint.unserialize
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'unserialize: untrusted input reaches maybe_unserialize, unserialize'
+  metadata:
+    family: deserialization
+    cwe: CWE-502
+    origin: sink:php:unserialize
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - maybe_unserialize
+    - unserialize
+  pattern-sources:
+  - pattern-either: *id002
+  pattern-sinks:
+  - pattern-either:
+    - pattern: maybe_unserialize(...)
+    - pattern: unserialize(...)
+  pattern-sanitizers:
+  - pattern-either:
+    - *id003
+    - *id004
+    - *id005
+    - *id006
+    - *id007
+    - *id008
+    - *id009
+    - *id010
+    - *id011
+    - *id012
+    - *id013
+    - *id014
+    - *id015
+    - *id016
+    - *id017
+    - *id018
+    - *id019
+    - *id020
+- id: ousast.php.taint.wpdb
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'wpdb: untrusted input reaches $wpdb->get_col, $wpdb->get_results, $wpdb->get_row, $wpdb->get_var, $wpdb->query, $wpdb->replace'
+  metadata:
+    family: injection
+    cwe: CWE-89
+    origin: sink:php:wpdb
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - $wpdb->get_col
+    - $wpdb->get_results
+    - $wpdb->get_row
+    - $wpdb->get_var
+    - $wpdb->query
+    - $wpdb->replace
+  pattern-sources:
+  - pattern-either: *id002
+  pattern-sinks:
+  - pattern-either:
+    - pattern: $wpdb->get_col(...)
+    - pattern: $wpdb->get_results(...)
+    - pattern: $wpdb->get_row(...)
+    - pattern: $wpdb->get_var(...)
+    - pattern: $wpdb->query(...)
+    - pattern: $wpdb->replace(...)
+  pattern-sanitizers:
+  - pattern-either:
+    - *id003
+    - *id004
+    - *id005
+    - *id006
+    - *id007
+    - *id008
+    - *id009
+    - *id010
+    - *id011
+    - *id012
+    - *id013
+    - *id014
+    - *id015
+    - *id016
+    - *id017
+    - *id018
+    - *id019
+    - *id020

benchmarks/measurements/2026-10-13-semgrep-arm/rules/python.yaml

@@ -0,0 +1,558 @@
+rules:
+- id: ousast.python.regex.python-flask-debug
+  languages:
+  - python
+  severity: WARNING
+  message: Flask debug mode enabled needs review
+  metadata:
+    family: config_secrets
+    cwe: CWE-489
+    origin: python-flask-debug
+    precision_estimate: '0.0'
+    kind: regex
+    tags:
+    - insecure_default
+    status: enabled
+    version: '1'
+  pattern-regex: \.run\s*\([^\n]*(debug)\s*=\s*True
+- id: ousast.python.regex.python-os-command
+  languages:
+  - python
+  severity: WARNING
+  message: Python OS command execution needs review
+  metadata:
+    family: injection
+    cwe: CWE-78
+    origin: python-os-command
+    precision_estimate: '0.0'
+    kind: regex
+    tags:
+    - syscall_entry
+    status: enabled
+    version: '1'
+  pattern-regex: \b(os\.system|os\.popen|os\.execl|os\.execlp|os\.execv|os\.execvp)\s*\(
+- id: ousast.python.regex.python-path-traversal
+  languages:
+  - python
+  severity: WARNING
+  message: Python file open with external input needs review
+  metadata:
+    family: path
+    cwe: CWE-22
+    origin: python-path-traversal
+    precision_estimate: '0.0'
+    kind: regex
+    tags:
+    - filesystem_entry
+    status: enabled
+    version: '1'
+  pattern-regex: \b(open)\s*\(\s*[^\n]*(?:request\.|req\.|argv|input\s*\(|params\[|GET\[|POST\[)
+- id: ousast.python.regex.python-shell-true
+  languages:
+  - python
+  severity: WARNING
+  message: Subprocess shell execution needs review
+  metadata:
+    family: injection
+    cwe: CWE-78
+    origin: python-shell-true
+    precision_estimate: '0.0'
+    kind: regex
+    tags:
+    - syscall_entry
+    status: enabled
+    version: '1'
+  pattern-regex: subprocess\.[a-zA-Z_]+\s*\([^\n]*shell\s*=\s*True
+- id: ousast.python.regex.python-sql-injection
+  languages:
+  - python
+  severity: WARNING
+  message: Python SQL string composition needs review
+  metadata:
+    family: injection
+    cwe: CWE-89
+    origin: python-sql-injection
+    precision_estimate: '0.0'
+    kind: regex
+    tags:
+    - injection
+    status: enabled
+    version: '1'
+  pattern-regex: (\.execute)\s*\(\s*[^\n]*(?:[\"']\s*%\s*[(A-Za-z_]|\.format\s*\(|[\"']\s*\+|\bf[\"'])
+- id: ousast.python.regex.python-ssrf
+  languages:
+  - python
+  severity: WARNING
+  message: Python outbound request with external input needs review
+  metadata:
+    family: untrusted_destination
+    cwe: CWE-918
+    origin: python-ssrf
+    precision_estimate: '0.0'
+    kind: regex
+    tags:
+    - network_entry
+    status: enabled
+    version: '1'
+  pattern-regex: \b(requests\.(?:get|post|put|delete|head|patch)|urllib\.request\.urlopen|urlopen)\s*\(\s*[^\n]*(?:request\.|req\.|argv|input\s*\()
+- id: ousast.python.regex.python-ssti
+  languages:
+  - python
+  severity: WARNING
+  message: Python server-side template injection needs review
+  metadata:
+    family: injection
+    cwe: CWE-94
+    origin: python-ssti
+    precision_estimate: '0.0'
+    kind: regex
+    tags:
+    - injection
+    status: enabled
+    version: '1'
+  pattern-regex: \b(render_template_string)\s*\(
+- id: ousast.python.regex.python-unsafe-deserialization
+  languages:
+  - python
+  severity: WARNING
+  message: Unsafe Python deserialization needs review
+  metadata:
+    family: deserialization
+    cwe: CWE-502
+    origin: python-unsafe-deserialization
+    precision_estimate: '0.0'
+    kind: regex
+    tags:
+    - deserialization
+    status: enabled
+    version: '1'
+  pattern-regex: \b(pickle\.loads|pickle\.load|cPickle\.loads|yaml\.load|marshal\.loads)\s*\(
+- id: ousast.python.regex.python-unsafe-eval
+  languages:
+  - python
+  severity: WARNING
+  message: Dynamic Python execution needs review
+  metadata:
+    family: injection
+    cwe: CWE-95
+    origin: python-unsafe-eval
+    precision_estimate: '0.0'
+    kind: regex
+    tags:
+    - syscall_entry
+    status: enabled
+    version: '1'
+  pattern-regex: \b(eval|exec)\s*\(
+- id: ousast.python.regex.python-weak-hash
+  languages:
+  - python
+  severity: WARNING
+  message: Python weak hash needs review
+  metadata:
+    family: config_secrets
+    cwe: CWE-327
+    origin: python-weak-hash
+    precision_estimate: '0.0'
+    kind: regex
+    tags:
+    - crypto
+    status: enabled
+    version: '1'
+  pattern-regex: \bhashlib\.(md5|sha1)\s*\(
+- id: ousast.python.taint.eval
+  mode: taint
+  languages: &id001
+  - python
+  severity: ERROR
+  message: 'eval: untrusted input reaches eval, exec'
+  metadata:
+    family: injection
+    cwe: CWE-95
+    origin: sink:python:eval
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - eval
+    - exec
+  pattern-sources:
+  - pattern-either: &id002
+    - pattern: request.args
+    - pattern: request.args(...)
+    - pattern: request.form
+    - pattern: request.form(...)
+    - pattern: request.data
+    - pattern: request.data(...)
+    - pattern: request.json
+    - pattern: request.json(...)
+    - pattern: request.cookies
+    - pattern: request.cookies(...)
+    - pattern: request.values
+    - pattern: request.values(...)
+    - pattern: request.get_json
+    - pattern: request.get_json(...)
+    - pattern: request.headers
+    - pattern: request.headers(...)
+    - pattern: request.files
+    - pattern: request.files(...)
+    - pattern: request.get
+    - pattern: request.get(...)
+    - pattern: request.GET
+    - pattern: request.GET(...)
+    - pattern: request.POST
+    - pattern: request.POST(...)
+    - pattern: request.body
+    - pattern: request.body(...)
+    - pattern: request.COOKIES
+    - pattern: request.COOKIES(...)
+    - pattern: request.FILES
+    - pattern: request.FILES(...)
+    - pattern: request.META
+    - pattern: request.META(...)
+    - pattern: request.query_params
+    - pattern: request.query_params(...)
+    - pattern: cleaned_data
+    - pattern: $OBJ.cleaned_data
+    - pattern: cleaned_data(...)
+    - pattern: $OBJ.cleaned_data(...)
+    - pattern: sys.argv
+    - pattern: sys.argv(...)
+    - pattern: sys.stdin
+    - pattern: sys.stdin(...)
+    - pattern: stdin.read
+    - pattern: stdin.read(...)
+    - pattern: input(...)
+  pattern-sinks:
+  - pattern-either:
+    - pattern: eval(...)
+    - pattern: $OBJ.eval(...)
+    - pattern: exec(...)
+    - pattern: $OBJ.exec(...)
+  pattern-sanitizers:
+  - pattern-either:
+    - &id003
+      pattern: ast.literal_eval(...)
+    - &id004
+      pattern: yaml.safe_load(...)
+    - &id005
+      pattern: safe_load(...)
+    - &id006
+      pattern: $OBJ.safe_load(...)
+- id: ousast.python.taint.execute
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'execute: untrusted input reaches execute, executemany'
+  metadata:
+    family: injection
+    cwe: CWE-89
+    origin: sink:python:execute
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - execute
+    - executemany
+  pattern-sources:
+  - pattern-either: *id002
+  pattern-sinks:
+  - pattern-either:
+    - patterns:
+      - pattern: execute($QUERY)
+      - focus-metavariable: $QUERY
+    - patterns:
+      - pattern: $OBJ.execute($QUERY)
+      - focus-metavariable: $QUERY
+    - patterns:
+      - pattern: executemany($QUERY)
+      - focus-metavariable: $QUERY
+    - patterns:
+      - pattern: $OBJ.executemany($QUERY)
+      - focus-metavariable: $QUERY
+  pattern-sanitizers:
+  - pattern-either:
+    - *id003
+    - *id004
+    - *id005
+    - *id006
+- id: ousast.python.weak.hashlib
+  languages: *id001
+  severity: WARNING
+  message: 'hashlib: a weak literal reaches hashlib.md5, hashlib.new, hashlib.sha1'
+  metadata:
+    family: config_secrets
+    cwe: CWE-327
+    origin: sink:python:hashlib
+    precision_estimate: unmeasured
+    kind: weak_literal
+    weak_literals:
+    - MD5
+    - SHA1
+    - md5
+    - sha-1
+    - sha1
+  pattern-either:
+  - pattern: hashlib.md5(...)
+  - patterns:
+    - pattern: hashlib.new($ALGORITHM, ...)
+    - metavariable-regex:
+        metavariable: $ALGORITHM
+        regex: ^["']?(MD5|SHA1|md5|sha\-1|sha1)["']?$
+  - pattern: hashlib.sha1(...)
+- id: ousast.python.taint.html-unescaped
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'html.unescaped: untrusted input reaches HttpResponse, Markup, format_html, mark_safe, render_template_string'
+  metadata:
+    family: output_encoding
+    cwe: CWE-79
+    origin: sink:python:html.unescaped
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - HttpResponse
+    - Markup
+    - format_html
+    - mark_safe
+    - render_template_string
+  pattern-sources:
+  - pattern-either: *id002
+  pattern-sinks:
+  - pattern-either:
+    - pattern: HttpResponse(...)
+    - pattern: $OBJ.HttpResponse(...)
+    - pattern: Markup(...)
+    - pattern: $OBJ.Markup(...)
+    - pattern: format_html(...)
+    - pattern: $OBJ.format_html(...)
+    - pattern: mark_safe(...)
+    - pattern: $OBJ.mark_safe(...)
+    - pattern: render_template_string(...)
+    - pattern: $OBJ.render_template_string(...)
+  pattern-sanitizers:
+  - pattern-either:
+    - *id003
+    - *id004
+    - *id005
+    - *id006
+- id: ousast.python.taint.orm-raw
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'orm.raw: untrusted input reaches RawSQL, extra, from_statement, objects.raw, raw, text'
+  metadata:
+    family: injection
+    cwe: CWE-89
+    origin: sink:python:orm.raw
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - RawSQL
+    - extra
+    - from_statement
+    - objects.raw
+    - raw
+    - text
+  pattern-sources:
+  - pattern-either: *id002
+  pattern-sinks:
+  - pattern-either:
+    - pattern: RawSQL(...)
+    - pattern: $OBJ.RawSQL(...)
+    - pattern: extra(...)
+    - pattern: $OBJ.extra(...)
+    - pattern: from_statement(...)
+    - pattern: $OBJ.from_statement(...)
+    - pattern: objects.raw(...)
+    - pattern: raw(...)
+    - pattern: $OBJ.raw(...)
+    - pattern: text(...)
+    - pattern: $OBJ.text(...)
+  pattern-sanitizers:
+  - pattern-either:
+    - *id003
+    - *id004
+    - *id005
+    - *id006
+- id: ousast.python.taint.os-system
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'os.system: untrusted input reaches os.popen, os.system, subprocess.Popen, subprocess.call, subprocess.check_output, subprocess.run'
+  metadata:
+    family: injection
+    cwe: CWE-78
+    origin: sink:python:os.system
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - os.popen
+    - os.system
+    - subprocess.Popen
+    - subprocess.call
+    - subprocess.check_output
+    - subprocess.run
+  pattern-sources:
+  - pattern-either: *id002
+  pattern-sinks:
+  - pattern-either:
+    - pattern: os.popen(...)
+    - pattern: os.system(...)
+    - pattern: subprocess.Popen(...)
+    - pattern: subprocess.call(...)
+    - pattern: subprocess.check_output(...)
+    - pattern: subprocess.run(...)
+  pattern-sanitizers:
+  - pattern-either:
+    - *id003
+    - *id004
+    - *id005
+    - *id006
+- id: ousast.python.taint.outbound-request
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'outbound.request: untrusted input reaches httpx.get, httpx.post, requests.get, requests.post, requests.request, urlopen, urlretrieve'
+  metadata:
+    family: untrusted_destination
+    cwe: CWE-918
+    origin: sink:python:outbound.request
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - httpx.get
+    - httpx.post
+    - requests.get
+    - requests.post
+    - requests.request
+    - urlopen
+    - urlretrieve
+  pattern-sources:
+  - pattern-either: *id002
+  pattern-sinks:
+  - pattern-either:
+    - pattern: httpx.get(...)
+    - pattern: httpx.post(...)
+    - pattern: requests.get(...)
+    - pattern: requests.post(...)
+    - pattern: requests.request(...)
+    - pattern: urlopen(...)
+    - pattern: $OBJ.urlopen(...)
+    - pattern: urlretrieve(...)
+    - pattern: $OBJ.urlretrieve(...)
+  pattern-sanitizers:
+  - pattern-either:
+    - *id003
+    - *id004
+    - *id005
+    - *id006
+- id: ousast.python.taint.path-open
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'path.open: untrusted input reaches FileResponse, open, os.remove, os.unlink, send_file, send_from_directory, shutil.copy, shutil.move, shutil.rmtree'
+  metadata:
+    family: path
+    cwe: CWE-22
+    origin: sink:python:path.open
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - FileResponse
+    - open
+    - os.remove
+    - os.unlink
+    - send_file
+    - send_from_directory
+    - shutil.copy
+    - shutil.move
+    - shutil.rmtree
+  pattern-sources:
+  - pattern-either: *id002
+  pattern-sinks:
+  - pattern-either:
+    - pattern: FileResponse(...)
+    - pattern: $OBJ.FileResponse(...)
+    - pattern: open(...)
+    - pattern: $OBJ.open(...)
+    - pattern: os.remove(...)
+    - pattern: os.unlink(...)
+    - pattern: send_file(...)
+    - pattern: $OBJ.send_file(...)
+    - pattern: send_from_directory(...)
+    - pattern: $OBJ.send_from_directory(...)
+    - pattern: shutil.copy(...)
+    - pattern: shutil.move(...)
+    - pattern: shutil.rmtree(...)
+  pattern-sanitizers:
+  - pattern-either:
+    - *id003
+    - *id004
+    - *id005
+    - *id006
+- id: ousast.python.taint.pickle-loads
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'pickle.loads: untrusted input reaches cPickle.loads, marshal.loads, pickle.load, pickle.loads, yaml.load'
+  metadata:
+    family: deserialization
+    cwe: CWE-502
+    origin: sink:python:pickle.loads
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - cPickle.loads
+    - marshal.loads
+    - pickle.load
+    - pickle.loads
+    - yaml.load
+  pattern-sources:
+  - pattern-either: *id002
+  pattern-sinks:
+  - pattern-either:
+    - pattern: cPickle.loads(...)
+    - pattern: marshal.loads(...)
+    - pattern: pickle.load(...)
+    - pattern: pickle.loads(...)
+    - pattern: yaml.load(...)
+  pattern-sanitizers:
+  - pattern-either:
+    - *id003
+    - *id004
+    - *id005
+    - *id006
+- id: ousast.python.taint.redirect
+  mode: taint
+  languages: *id001
+  severity: ERROR
+  message: 'redirect: untrusted input reaches HttpResponseRedirect, redirect, url_for'
+  metadata:
+    family: untrusted_destination
+    cwe: CWE-601
+    origin: sink:python:redirect
+    precision_estimate: unmeasured
+    kind: taint
+    calls:
+    - HttpResponseRedirect
+    - redirect
+    - url_for
+  pattern-sources:
+  - pattern-either: *id002
+  pattern-sinks:
+  - pattern-either:
+    - pattern: HttpResponseRedirect(...)
+    - pattern: $OBJ.HttpResponseRedirect(...)
+    - pattern: redirect(...)
+    - pattern: $OBJ.redirect(...)
+    - pattern: url_for(...)
+    - pattern: $OBJ.url_for(...)
+  pattern-sanitizers:
+  - pattern-either:
+    - *id003
+    - *id004
+    - *id005
+    - *id006
+    - pattern: reverse(...)
+    - pattern: $OBJ.reverse(...)
+    - pattern: reverse_lazy(...)
+    - pattern: $OBJ.reverse_lazy(...)

benchmarks/measurements/2026-10-13-semgrep-arm/scored_semgrep.json

@@ -0,0 +1,500 @@
+{
+  "alerts_only": {
+    "count": 30,
+    "n": 239,
+    "rate": 0.12552301255230125,
+    "wilson95": [
+      0.08935736205295211,
+      0.17353663963374408
+    ]
+  },
+  "breakdowns": {
+    "coverage": {
+      "engine_covered": {
+        "alerts_only": {
+          "count": 0,
+          "n": 0,
+          "rate": null,
+          "wilson95": [
+            null,
+            null
+          ]
+        },
+        "catch": {
+          "count": 0,
+          "n": 0,
+          "rate": null,
+          "wilson95": [
+            null,
+            null
+          ]
+        },
+        "false_alarm": {
+          "cluster95": [
+            null,
+            null
+          ],
+          "count": 0,
+          "icc": null,
+          "n": 0,
+          "rate": null,
+          "wilson95": [
+            null,
+            null
+          ]
+        }
+      },
+      "quick_only": {
+        "alerts_only": {
+          "count": 30,
+          "n": 239,
+          "rate": 0.12552301255230125,
+          "wilson95": [
+            0.08935736205295211,
+            0.17353663963374408
+          ]
+        },
+        "catch": {
+          "count": 0,
+          "n": 12,
+          "rate": 0.0,
+          "wilson95": [
+            0,
+            0.2425007574992425
+          ]
+        },
+        "false_alarm": {
+          "cluster95": [
+            0.0375,
+            0.23430962343096234
+          ],
+          "count": 30,
+          "icc": 0.27735846730536884,
+          "n": 239,
+          "rate": 0.12552301255230125,
+          "wilson95": [
+            0.08935736205295211,
+            0.17353663963374408
+          ]
+        }
+      },
+      "unanalysable": {
+        "alerts_only": {
+          "count": 0,
+          "n": 0,
+          "rate": null,
+          "wilson95": [
+            null,
+            null
+          ]
+        },
+        "catch": {
+          "count": 0,
+          "n": 0,
+          "rate": null,
+          "wilson95": [
+            null,
+            null
+          ]
+        },
+        "false_alarm": {
+          "cluster95": [
+            null,
+            null
+          ],
+          "count": 0,
+          "icc": null,
+          "n": 0,
+          "rate": null,
+          "wilson95": [
+            null,
+            null
+          ]
+        }
+      }
+    },
+    "family": {
+      "access_control": {
+        "alerts_only": {
+          "count": 0,
+          "n": 0,
+          "rate": null,
+          "wilson95": [
+            null,
+            null
+          ]
+        },
+        "catch": {
+          "count": 0,
+          "n": 3,
+          "rate": 0.0,
+          "wilson95": [
+            0,
+            0.5615060804490177
+          ]
+        },
+        "false_alarm": {
+          "cluster95": [
+            null,
+            null
+          ],
+          "count": 0,
+          "icc": null,
+          "n": 0,
+          "rate": null,
+          "wilson95": [
+            null,
+            null
+          ]
+        }
+      },
+      "output_encoding": {
+        "alerts_only": {
+          "count": 0,
+          "n": 0,
+          "rate": null,
+          "wilson95": [
+            null,
+            null
+          ]
+        },
+        "catch": {
+          "count": 0,
+          "n": 4,
+          "rate": 0.0,
+          "wilson95": [
+            0,
+            0.48990002040399916
+          ]
+        },
+        "false_alarm": {
+          "cluster95": [
+            null,
+            null
+          ],
+          "count": 0,
+          "icc": null,
+          "n": 0,
+          "rate": null,
+          "wilson95": [
+            null,
+            null
+          ]
+        }
+      },
+      "path": {
+        "alerts_only": {
+          "count": 0,
+          "n": 0,
+          "rate": null,
+          "wilson95": [
+            null,
+            null
+          ]
+        },
+        "catch": {
+          "count": 0,
+          "n": 3,
+          "rate": 0.0,
+          "wilson95": [
+            0,
+            0.5615060804490177
+          ]
+        },
+        "false_alarm": {
+          "cluster95": [
+            null,
+            null
+          ],
+          "count": 0,
+          "icc": null,
+          "n": 0,
+          "rate": null,
+          "wilson95": [
+            null,
+            null
+          ]
+        }
+      },
+      "unknown": {
+        "alerts_only": {
+          "count": 30,
+          "n": 239,
+          "rate": 0.12552301255230125,
+          "wilson95": [
+            0.08935736205295211,
+            0.17353663963374408
+          ]
+        },
+        "catch": {
+          "count": 0,
+          "n": 0,
+          "rate": null,
+          "wilson95": [
+            null,
+            null
+          ]
+        },
+        "false_alarm": {
+          "cluster95": [
+            0.0375,
+            0.23430962343096234
+          ],
+          "count": 30,
+          "icc": 0.27735846730536884,
+          "n": 239,
+          "rate": 0.12552301255230125,
+          "wilson95": [
+            0.08935736205295211,
+            0.17353663963374408
+          ]
+        }
+      },
+      "untrusted_destination": {
+        "alerts_only": {
+          "count": 0,
+          "n": 0,
+          "rate": null,
+          "wilson95": [
+            null,
+            null
+          ]
+        },
+        "catch": {
+          "count": 0,
+          "n": 2,
+          "rate": 0.0,
+          "wilson95": [
+            0,
+            0.6576280471103807
+          ]
+        },
+        "false_alarm": {
+          "cluster95": [
+            null,
+            null
+          ],
+          "count": 0,
+          "icc": null,
+          "n": 0,
+          "rate": null,
+          "wilson95": [
+            null,
+            null
+          ]
+        }
+      }
+    },
+    "method": {
+      "first_parent": {
+        "alerts_only": {
+          "count": 30,
+          "n": 239,
+          "rate": 0.12552301255230125,
+          "wilson95": [
+            0.08935736205295211,
+            0.17353663963374408
+          ]
+        },
+        "catch": {
+          "count": 0,
+          "n": 0,
+          "rate": null,
+          "wilson95": [
+            null,
+            null
+          ]
+        },
+        "false_alarm": {
+          "cluster95": [
+            0.0375,
+            0.23430962343096234
+          ],
+          "count": 30,
+          "icc": 0.27735846730536884,
+          "n": 239,
+          "rate": 0.12552301255230125,
+          "wilson95": [
+            0.08935736205295211,
+            0.17353663963374408
+          ]
+        }
+      },
+      "fix_site": {
+        "alerts_only": {
+          "count": 0,
+          "n": 0,
+          "rate": null,
+          "wilson95": [
+            null,
+            null
+          ]
+        },
+        "catch": {
+          "count": 0,
+          "n": 12,
+          "rate": 0.0,
+          "wilson95": [
+            0,
+            0.2425007574992425
+          ]
+        },
+        "false_alarm": {
+          "cluster95": [
+            null,
+            null
+          ],
+          "count": 0,
+          "icc": null,
+          "n": 0,
+          "rate": null,
+          "wilson95": [
+            null,
+            null
+          ]
+        }
+      }
+    },
+    "post_cutoff": {
+      "unknown": {
+        "alerts_only": {
+          "count": 30,
+          "n": 239,
+          "rate": 0.12552301255230125,
+          "wilson95": [
+            0.08935736205295211,
+            0.17353663963374408
+          ]
+        },
+        "catch": {
+          "count": 0,
+          "n": 12,
+          "rate": 0.0,
+          "wilson95": [
+            0,
+            0.2425007574992425
+          ]
+        },
+        "false_alarm": {
+          "cluster95": [
+            0.0375,
+            0.23430962343096234
+          ],
+          "count": 30,
+          "icc": 0.27735846730536884,
+          "n": 239,
+          "rate": 0.12552301255230125,
+          "wilson95": [
+            0.08935736205295211,
+            0.17353663963374408
+          ]
+        }
+      }
+    },
+    "stream": {
+      "advisory": {
+        "count": 0,
+        "n": 12,
+        "rate": 0.0,
+        "wilson95": [
+          0,
+          0.2425007574992425
+        ]
+      },
+      "alerts": {
+        "count": 0,
+        "n": 12,
+        "rate": 0.0,
+        "wilson95": [
+          0,
+          0.2425007574992425
+        ]
+      },
+      "quick": {
+        "count": 0,
+        "n": 12,
+        "rate": 0.0,
+        "wilson95": [
+          0,
+          0.2425007574992425
+        ]
+      }
+    }
+  },
+  "catch": {
+    "count": 0,
+    "n": 12,
+    "rate": 0.0,
+    "wilson95": [
+      0,
+      0.2425007574992425
+    ]
+  },
+  "confirmed": {
+    "alerts_only": {
+      "count": 0,
+      "n": 0,
+      "rate": null,
+      "wilson95": [
+        null,
+        null
+      ]
+    },
+    "catch": {
+      "count": 0,
+      "n": 12,
+      "rate": 0.0,
+      "wilson95": [
+        0,
+        0.2425007574992425
+      ]
+    },
+    "false_alarm": {
+      "cluster95": [
+        null,
+        null
+      ],
+      "count": 0,
+      "icc": null,
+      "n": 0,
+      "rate": null,
+      "wilson95": [
+        null,
+        null
+      ]
+    }
+  },
+  "coverage": {
+    "engine_covered": 0,
+    "instrument_failure": 1,
+    "quick_only": 251,
+    "unanalysable": 0
+  },
+  "deadlines": [
+    {
+      "applied_deadline_scale": null,
+      "count": 252,
+      "deadline_scale": null,
+      "deadline_seconds": null,
+      "lane": "unknown"
+    }
+  ],
+  "false_alarm": {
+    "cluster95": [
+      0.0375,
+      0.23430962343096234
+    ],
+    "count": 30,
+    "icc": 0.27735846730536884,
+    "n": 239,
+    "rate": 0.12552301255230125,
+    "wilson95": [
+      0.08935736205295211,
+      0.17353663963374408
+    ]
+  },
+  "total": 252,
+  "unseen": {
+    "gate": "not_run"
+  }
+}

benchmarks/valid/semgrep_arm.py

@@ -0,0 +1,634 @@
+"""Semgrep CE as a benchmark arm: our regex rules and taint facts converted, run per frozen change.
+
+Two subcommands. ``rules`` converts every enabled quick rule to a ``pattern-regex`` rule and every
+source/sink/sanitizer fact triple to a ``mode: taint`` rule, one YAML file per language, and writes a
+conversion report naming what was skipped and why. ``run`` checks out each frozen change's head into a
+temporary worktree, runs Semgrep over the changed files only, proves the targets were read (bytes per
+file), and writes one scorer-shaped record per change, resumable exactly like ``hunter_arm``.
+
+Nothing here ships: the rules are ours, the engine is a measurement instrument.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import re
+import shutil
+import subprocess
+import sys
+import tempfile
+import time
+import tomllib
+from pathlib import Path
+
+import yaml
+
+from openultrasast.hunter_tools import clamp_repo_path
+from openultrasast.model.taxonomy import load_families
+from openultrasast.ruleset import DEFAULT_RULESET_DIR, load_ruleset
+from openultrasast.semantic.facts import DEFAULT_FACTS_DIR, load_facts
+
+if __package__ in {None, ""}:
+    sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+
+from benchmarks.unseen.extract import SHA, InstrumentFailure
+from benchmarks.unseen.replay import canonical, load_slice
+from benchmarks.valid.catalog import local_git
+
+# Our language ids to Semgrep's. Groovy has no Semgrep parser: `generic` plus a path filter.
+SEMGREP_LANGUAGES = {
+    "c": "c",
+    "cpp": "cpp",
+    "python": "python",
+    "javascript": "javascript",
+    "typescript": "typescript",
+    "java": "java",
+    "php": "php",
+}
+GENERIC_PATHS = {"groovy": ["*.groovy", "*.tpl"]}
+# The rule file a rule lands in: one per language family.
+FILE_OF_LANGUAGE = {"cpp": "c", "typescript": "javascript"}
+# Fact languages whose Semgrep rules need a second dialect.
+FACT_DIALECTS = {"c": ["c", "cpp"], "javascript": ["javascript", "typescript"]}
+# PHP language constructs are not calls: `echo(...)` would parse as a call expression, and Semgrep's PHP
+# pattern parser takes `echo` only as a whole statement.
+PHP_CONSTRUCTS = {
+    "echo": "echo $VALUE;",
+    "print": "print $VALUE;",
+    "include": "include ...",
+    "include_once": "include_once ...",
+    "require": "require ...",
+    "require_once": "require_once ...",
+}
+# DOM properties assigned, not called.
+ASSIGNED_SINKS = {"innerHTML", "outerHTML"}
+SEMGREP_FLAGS = (
+    "--json",
+    "--metrics=off",
+    "--disable-version-check",
+    "--no-git-ignore",
+    "--timeout",
+    "60",
+    "--max-memory",
+    "1500",
+    "-j",
+    "1",
+)
+# A Semgrep launch that returns faster than its own startup never ran the engine.
+MIN_PLAUSIBLE_SECONDS = 1.0
+# PCRE2 compiles a bounded repeat of a group by copying it: `(?:...){0,2000}?` is "too large". The lazy
+# unbounded form keeps the match and widens the window from 2000 characters to the statement.
+BOUNDED_GROUP_REPEAT = re.compile(r"\{0,\d+\}\?")
+
+
+def _family_of(cwe):
+    family = load_families().family_of_cwe(cwe or "")
+    return family.id if family else "unknown"
+
+
+def _ident(value):
+    return re.sub(r"[^A-Za-z0-9_-]+", "-", value).strip("-")
+
+
+# --- regex rules -------------------------------------------------------------------------------------------
+
+
+def _raw_rules(ruleset_dir):
+    rows = []
+    for path in sorted(Path(ruleset_dir).rglob("*.toml")):
+        if path.parent.name in {"semantic", "obligations"} or path.parent == Path(ruleset_dir):
+            continue
+        for item in tomllib.loads(path.read_text()).get("rule", []):
+            rows.append(item)
+    return rows
+
+
+def convert_regex_rules(ruleset_dir=DEFAULT_RULESET_DIR, *, min_precision=0.0):
+    """Every enabled quick rule as a Semgrep ``pattern-regex`` rule; the rest named with a reason."""
+    active = {rule.rule_id: rule for rule in load_ruleset(Path(ruleset_dir))}
+    converted, skipped, rewrites = {}, [], []
+    for item in _raw_rules(ruleset_dir):
+        rule_id = str(item["rule_id"])
+        rule = active.get(rule_id)
+        if rule is None:
+            skipped.append({"origin": rule_id, "reason": f"fallback for {item.get('fallback_for')}: the tagged rule is active"})
+            continue
+        if rule.status == "disabled":
+            skipped.append({"origin": rule_id, "reason": "disabled"})
+            continue
+        if rule.status == "shadow":
+            skipped.append(
+                {"origin": rule_id, "reason": f"shadow status (precision_estimate {rule.precision_estimate}); not admitted by the hook"}
+            )
+            continue
+        if rule.precision_estimate < min_precision:
+            skipped.append({"origin": rule_id, "reason": f"emit floor: precision_estimate {rule.precision_estimate} < {min_precision}"})
+            continue
+        if not rule.languages:
+            skipped.append({"origin": rule_id, "reason": "non-code file targets: the rule applies to every file"})
+            continue
+        languages = [SEMGREP_LANGUAGES[language] for language in rule.languages if language in SEMGREP_LANGUAGES]
+        generic = [language for language in rule.languages if language in GENERIC_PATHS]
+        unsupported = [language for language in rule.languages if language not in SEMGREP_LANGUAGES and language not in GENERIC_PATHS]
+        if unsupported:
+            skipped.append({"origin": rule_id, "reason": f"no Semgrep language for {unsupported}"})
+            continue
+        family = _family_of(rule.cwe)
+        pattern = BOUNDED_GROUP_REPEAT.sub("*?", rule.pattern)
+        if pattern != rule.pattern:
+            rewrites.append({"origin": rule_id, "rewrite": "bounded lazy group repeat {0,N}? widened to *? (PCRE2 size limit)"})
+        entry = {
+            "id": f"ousast.{FILE_OF_LANGUAGE.get(rule.languages[0], rule.languages[0])}.regex.{_ident(rule_id)}",
+            "languages": languages or ["generic"],
+            "severity": "WARNING",
+            "message": rule.title,
+            "metadata": {
+                "family": family,
+                "cwe": rule.cwe,
+                "origin": rule_id,
+                "precision_estimate": str(rule.precision_estimate),
+                "kind": "regex",
+                "tags": list(rule.tags),
+                "status": rule.status,
+                "version": rule.version,
+            },
+            "pattern-regex": pattern,
+        }
+        if not languages:
+            entry["paths"] = {"include": [glob for language in generic for glob in GENERIC_PATHS[language]]}
+        key = FILE_OF_LANGUAGE.get(rule.languages[0], rule.languages[0])
+        converted.setdefault(key, []).append(entry)
+    return converted, skipped, rewrites
+
+
+# --- taint facts -------------------------------------------------------------------------------------------
+
+
+def _call_patterns(call, language):
+    """The Semgrep call shapes of one fact token (a function, a method, a construct)."""
+    if language == "php":
+        if call in PHP_CONSTRUCTS:
+            return [PHP_CONSTRUCTS[call]]
+        return [f"{call}(...)"]
+    if "." in call or "->" in call:
+        return [f"{call}(...)"]
+    if language == "java" and call[:1].isupper():
+        return [f"new {call}(...)"]
+    return [f"{call}(...)", f"$OBJ.{call}(...)"]
+
+
+def _source_patterns(token, language):
+    if language == "php":
+        if token == "php://input":
+            return ['file_get_contents("php://input")']
+        return [token, f"{token}[...]"]
+    if token.endswith("("):
+        return [f"{token}...)"]
+    if "." in token:
+        return [token, f"{token}(...)"]
+    if language == "c":
+        return [token, f"{token}(...)"]
+    return [token, f"$OBJ.{token}", f"{token}(...)", f"$OBJ.{token}(...)"]
+
+
+def _sink_entries(call, language, *, format_arg=None, single_argument=False):
+    """Sink patterns for one call; a focused argument where the fact names one."""
+    if call in ASSIGNED_SINKS:
+        return [{"patterns": [{"pattern": f"$OBJ.{call} = $VALUE"}, {"focus-metavariable": "$VALUE"}]}]
+    if call == "dangerouslySetInnerHTML":
+        return [{"patterns": [{"pattern": "{__html: $VALUE}"}, {"focus-metavariable": "$VALUE"}]}]
+    if format_arg is not None:
+        arguments = ", ".join([*(f"$ARG{index}" for index in range(format_arg)), "$FORMAT", "..."])
+        heads = [call] if "." in call or language == "c" else [call, f"$OBJ.{call}"]
+        return [{"patterns": [{"pattern": f"{head}({arguments})"}, {"focus-metavariable": "$FORMAT"}]} for head in heads]
+    if single_argument:
+        heads = [call] if "." in call or "->" in call else [call, f"$OBJ.{call}"]
+        return [{"patterns": [{"pattern": f"{head}($QUERY)"}, {"focus-metavariable": "$QUERY"}]} for head in heads]
+    return [{"pattern": pattern} for pattern in _call_patterns(call, language)]
+
+
+def _weak_literal_rule(language, sink_id, cwe, calls, literals, dialects):
+    """A `weak_literals` sink is a search, not a flow: the call with a named weak literal, or a call named after one."""
+    alternatives = []
+    lowered = {literal.lower() for literal in literals}
+    for call in sorted(calls):
+        for head in [call] if "." in call else [call, f"$OBJ.{call}"]:
+            if call.rsplit(".", 1)[-1].lower() in lowered:
+                alternatives.append({"pattern": f"{head}(...)"})
+            else:
+                alternatives.append(
+                    {
+                        "patterns": [
+                            {"pattern": f"{head}($ALGORITHM, ...)"},
+                            {
+                                "metavariable-regex": {
+                                    "metavariable": "$ALGORITHM",
+                                    "regex": "^[\"']?(" + "|".join(re.escape(literal) for literal in sorted(literals)) + ")[\"']?$",
+                                }
+                            },
+                        ]
+                    }
+                )
+    return {
+        "id": f"ousast.{language}.weak.{_ident(sink_id)}",
+        "languages": dialects,
+        "severity": "WARNING",
+        "message": f"{sink_id}: a weak literal reaches {', '.join(sorted(calls))}",
+        "metadata": {
+            "family": _family_of(cwe),
+            "cwe": cwe,
+            "origin": f"sink:{language}:{sink_id}",
+            "precision_estimate": "unmeasured",
+            "kind": "weak_literal",
+            "weak_literals": sorted(literals),
+        },
+        "pattern-either": alternatives,
+    }
+
+
+def convert_facts(facts_dir=DEFAULT_FACTS_DIR):
+    """One taint rule per (language, sink id, CWE); the facts Semgrep has no word for are named, not dropped silently."""
+    facts = load_facts(Path(facts_dir))
+    converted, skipped, lossy = {}, [], []
+    for language in sorted({fact.language for fact in facts.sinks}):
+        scoped = facts.for_language(language)
+        dialects = FACT_DIALECTS.get(language, [SEMGREP_LANGUAGES[language]])
+        sources = [
+            {"pattern": pattern} for fact in scoped.sources for token in fact.patterns for pattern in _source_patterns(token, language)
+        ]
+        parameterized = set()
+        sanitizers = []
+        for fact in scoped.sanitizers:
+            origin = f"sanitizer:{language}:{fact.id}"
+            if fact.guard:
+                skipped.append({"origin": origin, "reason": "guard: a check whose safety depends on dominance; Semgrep has no equivalent"})
+                continue
+            if fact.parameterized:
+                parameterized.update(fact.calls)
+                skipped.append(
+                    {"origin": origin, "reason": "parameterized: a call shape, not a cleansing; expressed as the single-argument sink form"}
+                )
+                continue
+            if fact.literal_format_arg is not None:
+                skipped.append(
+                    {
+                        "origin": origin,
+                        "reason": "literal_format_arg: a shape; a literal format is never tainted at the focused sink argument",
+                    }
+                )
+                continue
+            if fact.quoted_only:
+                loss = {"origin": origin, "loss": "quoted_only escape treated as a full sanitizer (silences the CVE-2024-1071 shape)"}
+                if loss not in lossy:
+                    lossy.append(loss)
+            sanitizers.extend({"pattern": pattern} for call in fact.calls for pattern in _call_patterns(call, language))
+        for fact in scoped.dispatches:
+            skipped.append(
+                {
+                    "origin": f"dispatch:{language}:{fact.id}",
+                    "reason": "dispatch: string-keyed hook registry routing; no Semgrep equivalent",
+                }
+            )
+        for fact in scoped.layouts:
+            skipped.append({"origin": f"layout:{language}:{fact.id}", "reason": "layout: scope policy (vendored, tests), not a rule"})
+        grouped = {}
+        for fact in scoped.sinks:
+            grouped.setdefault((fact.id, fact.cwe), []).append(fact)
+        for (sink_id, cwe), rows in sorted(grouped.items()):
+            origin = f"sink:{language}:{sink_id}"
+            calls = sorted({call for row in rows for call in row.calls})
+            literals = {literal for row in rows for literal in row.weak_literals}
+            if literals:
+                converted.setdefault(language, []).append(_weak_literal_rule(language, sink_id, cwe, calls, literals, dialects))
+                continue
+            entries, extra_sanitizers = [], []
+            format_arg = next((row.format_arg for row in rows if row.format_arg is not None), None)
+            for call in calls:
+                if call == "v-html":
+                    skipped.append({"origin": f"{origin}:{call}", "reason": "template directive, not a call"})
+                    continue
+                entries.extend(_sink_entries(call, language, format_arg=format_arg, single_argument=call in parameterized))
+            for row in rows:
+                if row.prefix_fixes_origin:
+                    lossy.append(
+                        {"origin": origin, "loss": "prefix_fixes_origin: a value appended after a fixed origin still counts as tainted"}
+                    )
+                    break
+            for anchor in sorted({anchor for row in rows for anchor in row.origin_anchors}):
+                extra_sanitizers.extend({"pattern": pattern} for pattern in _call_patterns(anchor, language))
+            if not entries:
+                skipped.append({"origin": origin, "reason": "no call form survives conversion"})
+                continue
+            rule = {
+                "id": f"ousast.{language}.taint.{_ident(sink_id)}",
+                "mode": "taint",
+                "languages": dialects,
+                "severity": "ERROR",
+                "message": f"{sink_id}: untrusted input reaches {', '.join(calls)}",
+                "metadata": {
+                    "family": _family_of(cwe),
+                    "cwe": cwe,
+                    "origin": origin,
+                    "precision_estimate": "unmeasured",
+                    "kind": "taint",
+                    "calls": calls,
+                },
+                "pattern-sources": [{"pattern-either": sources}],
+                "pattern-sinks": [{"pattern-either": entries}],
+            }
+            if sanitizers or extra_sanitizers:
+                rule["pattern-sanitizers"] = [{"pattern-either": [*sanitizers, *extra_sanitizers]}]
+            converted.setdefault(language, []).append(rule)
+    return converted, skipped, lossy
+
+
+def _obligation_tables(ruleset_dir):
+    count = 0
+    for path in sorted((Path(ruleset_dir) / "obligations").glob("*.toml")):
+        payload = tomllib.loads(path.read_text())
+        count += sum(len(value) for value in payload.values() if isinstance(value, list))
+    return count
+
+
+def write_rules(out, *, ruleset_dir=DEFAULT_RULESET_DIR, facts_dir=DEFAULT_FACTS_DIR, min_precision=0.0, semgrep=None):
+    """Write one YAML per language under ``out`` plus ``conversion_report.json``; validate with Semgrep when present."""
+    out = Path(out)
+    out.mkdir(parents=True, exist_ok=True)
+    regex_rules, regex_skipped, rewrites = convert_regex_rules(ruleset_dir, min_precision=min_precision)
+    taint_rules, fact_skipped, lossy = convert_facts(facts_dir)
+    files = {}
+    for language in sorted(set(regex_rules) | set(taint_rules)):
+        rules = [*regex_rules.get(language, []), *taint_rules.get(language, [])]
+        path = out / f"{language}.yaml"
+        path.write_text(yaml.safe_dump({"rules": rules}, sort_keys=False, width=200, allow_unicode=True))
+        files[path.name] = len(rules)
+    obligations = _obligation_tables(ruleset_dir)
+    report = {
+        "regex": {
+            "converted": sorted(rule["metadata"]["origin"] for rules in regex_rules.values() for rule in rules),
+            "skipped": regex_skipped,
+            "rewritten": rewrites,
+        },
+        "facts": {
+            "converted": sorted(rule["metadata"]["origin"] for rules in taint_rules.values() for rule in rules),
+            "skipped": fact_skipped,
+            "lossy": lossy,
+        },
+        "obligations": {
+            "skipped_tables": obligations,
+            "reason": "an obligation finds a missing guard (an absence), which no pattern expresses",
+        },
+        "not_carried": [
+            "comment skipping: the quick engine drops a regex match inside a comment; pattern-regex does not",
+            "framework priors and fallback rules: the tagged rule is emitted, its language-level fallback is not",
+            "min_evidence_level, ranking priority, the rung: post-match policy on our side",
+        ],
+        "files": files,
+        "validation": None,
+    }
+    binary = semgrep or shutil.which("semgrep")
+    if binary:
+        try:
+            completed = subprocess.run(
+                [binary, "--validate", "--config", str(out), "--metrics=off", "--disable-version-check"], capture_output=True, text=True
+            )
+        except OSError as error:
+            report["validation"] = {"exit": None, "error": f"semgrep not found or not runnable: {type(error).__name__}"}
+        else:
+            report["validation"] = {"exit": completed.returncode, "stderr": completed.stderr[-4000:], "stdout": completed.stdout[-4000:]}
+    (out / "conversion_report.json").write_text(json.dumps(report, indent=2, sort_keys=True) + "\n")
+    return report
+
+
+# --- run ---------------------------------------------------------------------------------------------------
+
+
+def result_defect(result):
+    """One Semgrep result as a scorer defect; the family and CWE come from the rule's own metadata."""
+    extra = result.get("extra", {})
+    metadata = extra.get("metadata", {})
+    path = result["path"]
+    line = result.get("start", {}).get("line", 0)
+    origin = metadata.get("origin", result["check_id"])
+    return {
+        "defect_id": f"{origin}:{path}:{line}",
+        "family": metadata.get("family", "unknown"),
+        "cwe": metadata.get("cwe"),
+        "locations": [[path, line]],
+        "source": "semgrep",
+        "rule_id": result["check_id"],
+        "origin": origin,
+        "title": extra.get("message", ""),
+        "rationale": extra.get("message", ""),
+        "confidence": extra.get("severity"),
+        "dataflow": bool(extra.get("dataflow_trace")),
+    }
+
+
+def _write_json(path, value):
+    path.parent.mkdir(parents=True, exist_ok=True)
+    temporary = path.with_suffix(".json.tmp")
+    temporary.write_text(json.dumps(value, sort_keys=True, indent=2) + "\n")
+    temporary.replace(path)
+
+
+def _record_path(out, identity):
+    return out / (hashlib.sha256(identity.encode()).hexdigest() + ".json")
+
+
+def _inputs(out):
+    path = out / "inputs.json"
+    entries = json.loads(path.read_text()) if path.exists() else []
+    for key in (lambda entry: entry["item_id"], lambda entry: entry["change"]["id"]):
+        if len({key(entry) for entry in entries}) != len(entries):
+            raise ValueError(f"duplicate ids in {path}")
+    return entries
+
+
+def run_semgrep(binary, rules, root, targets, *, timeout=900):
+    """Semgrep over explicit targets inside ``root``; paths in the output stay repository-relative."""
+    argv = [binary, "scan", "--config", str(rules), *SEMGREP_FLAGS, *targets]
+    started = time.monotonic()
+    completed = subprocess.run(argv, cwd=root, capture_output=True, text=True, timeout=timeout)
+    return completed, time.monotonic() - started
+
+
+def _scan_change(change, repos, rules, *, binary):
+    instrument = {
+        "hook_exit": 0,
+        "hook_bytes_read": 0,
+        "changed_bytes": {},
+        "head_verified": False,
+        "base_verified": False,
+        "failure": None,
+        "suspect_fast": False,
+        "semgrep": {"exit": None, "wall_seconds": None, "scanned": [], "errors": [], "results": 0},
+    }
+    record = {
+        "id": change["id"],
+        "done": True,
+        "status": "semgrep_covered",
+        "instrument": instrument,
+        "push": {"admission": {"defects": [], "dispositions": []}, "quick_tier": []},
+        "skipped": [],
+    }
+    try:
+        if not all(SHA.fullmatch(change[side]) for side in ("base", "head")):
+            raise InstrumentFailure("invalid_revision")
+        git = local_git(Path(repos[change["repo"]]).expanduser().resolve())
+        git.run("cat-file", "-e", change["base"] + "^{commit}")
+        instrument["base_verified"] = True
+        with tempfile.TemporaryDirectory(prefix="ousast-semgrep-") as temporary:
+            root = Path(temporary) / "head"
+            added = False
+            try:
+                git.run("-c", "core.hooksPath=/dev/null", "worktree", "add", "--detach", str(root), change["head"])
+                added = True
+                if local_git(root).run("rev-parse", "HEAD").strip() != change["head"]:
+                    raise InstrumentFailure("head_mismatch")
+                instrument["head_verified"] = True
+                targets = []
+                for relative in dict.fromkeys(file["path"] for file in change.get("files", [])):
+                    path = clamp_repo_path(root, relative)
+                    try:
+                        path.stat()
+                    except FileNotFoundError:
+                        record["skipped"].append({"path": relative, "reason": "missing_at_head"})
+                        print(f"{change['id']} {relative}: skipped (missing_at_head)", flush=True)
+                        continue
+                    try:
+                        data = path.read_bytes()
+                    except OSError:
+                        print(f"{change['id']} {relative}: 0 bytes read", flush=True)
+                        raise InstrumentFailure(f"unreadable_target:{relative}") from None
+                    size = len(data)
+                    print(f"{change['id']} {relative}: {size} bytes read", flush=True)
+                    if not size:
+                        record["skipped"].append({"path": relative, "reason": "empty_at_head"})
+                        continue
+                    instrument["changed_bytes"][relative] = size
+                    instrument["hook_bytes_read"] += size
+                    targets.append(relative)
+                if not targets:
+                    raise InstrumentFailure("no_readable_targets")
+                try:
+                    completed, wall = run_semgrep(binary, rules, root, targets)
+                except subprocess.TimeoutExpired:
+                    raise InstrumentFailure("semgrep_timeout") from None
+                semgrep = instrument["semgrep"]
+                semgrep.update(exit=completed.returncode, wall_seconds=round(wall, 3))
+                instrument["hook_exit"] = completed.returncode
+                if wall < MIN_PLAUSIBLE_SECONDS:
+                    instrument["suspect_fast"] = True
+                if completed.returncode not in (0, 1):
+                    semgrep["stderr"] = completed.stderr[-2000:]
+                    raise InstrumentFailure(f"semgrep_exit_{completed.returncode}")
+                try:
+                    payload = json.loads(completed.stdout)
+                except json.JSONDecodeError:
+                    semgrep["stderr"] = completed.stderr[-2000:]
+                    raise InstrumentFailure("semgrep_output_unreadable") from None
+                semgrep["scanned"] = sorted(payload.get("paths", {}).get("scanned", []))
+                semgrep["errors"] = [
+                    {"type": str(error.get("type")), "path": error.get("path"), "message": str(error.get("message", ""))[:300]}
+                    for error in payload.get("errors", [])
+                ]
+                results = payload.get("results", [])
+                semgrep["results"] = len(results)
+                record["push"]["admission"]["defects"] = [result_defect(result) for result in results]
+                print(f"{change['id']}: semgrep exit {completed.returncode}, {len(results)} results, {wall:.1f}s", flush=True)
+            finally:
+                if added:
+                    git.run("worktree", "remove", "--force", str(root))
+    except Exception as error:  # noqa: BLE001
+        reason = str(error) if isinstance(error, InstrumentFailure) else type(error).__name__
+        instrument.update(hook_exit=2, failure=reason)
+        record["status"] = "instrument_failure"
+        record["push"]["admission"]["defects"] = []
+        print(f"{change['id']}: instrument_failure ({reason})", file=sys.stderr, flush=True)
+    return record
+
+
+def run(manifest, repos_path, rules, out, *, slice_id="1", only="all", semgrep=None):
+    binary = semgrep or shutil.which("semgrep")
+    if binary is None:
+        print("No semgrep binary found on PATH.", file=sys.stderr)
+        return 2
+    rules = Path(rules).expanduser().resolve()
+    if not rules.is_dir():
+        raise ValueError(f"rules directory missing: {rules}")
+    changes = load_slice(manifest, slice_id)
+    if len({change["id"] for change in changes}) != len(changes):
+        raise ValueError("duplicate change ids")
+    changes = [change for change in changes if only == "all" or change["kind"] == only]
+    entries = _inputs(out)
+    indexed = {entry["change"]["id"]: entry for entry in entries}
+    pending = []
+    for change in changes:
+        path = _record_path(out, change["id"])
+        previous = json.loads(path.read_text()) if path.exists() else {}
+        entry = indexed.get(change["id"])
+        if entry is None:
+            entry = {
+                "item_id": change["id"],
+                "change": change,
+                "input_digest": previous.get("input_digest") or hashlib.sha256(canonical(change)).hexdigest(),
+            }
+            entries.append(entry)
+        elif entry["change"] != change or entry["item_id"] != change["id"]:
+            raise ValueError("existing semgrep inputs do not match frozen change")
+        if previous.get("done") is True:
+            print(f"{change['id']}: skipped (done)", flush=True)
+            continue
+        pending.append(entry)
+    _write_json(out / "inputs.json", entries)
+    if not pending:
+        return 0
+    repos = json.loads(Path(repos_path).expanduser().read_text())
+    failed = False
+    for entry in pending:
+        record = _scan_change(entry["change"], repos, rules, binary=binary)
+        record["input_digest"] = entry["input_digest"]
+        _write_json(_record_path(out, entry["item_id"]), record)
+        failed |= record["status"] == "instrument_failure"
+    return 2 if failed else 0
+
+
+def main(argv=None):
+    parser = argparse.ArgumentParser(description=__doc__)
+    commands = parser.add_subparsers(dest="command", required=True)
+    rules = commands.add_parser("rules", help="convert the ruleset and the facts into Semgrep rules")
+    rules.add_argument("--out", type=Path, required=True)
+    rules.add_argument("--ruleset", type=Path, default=DEFAULT_RULESET_DIR)
+    rules.add_argument("--facts", type=Path, default=DEFAULT_FACTS_DIR)
+    rules.add_argument("--min-precision", type=float, default=0.0, help="the hook's min_emit_precision floor")
+    rules.add_argument("--semgrep", help="semgrep binary (default: PATH)")
+    scan = commands.add_parser("run", help="run Semgrep over each frozen change's changed files")
+    scan.add_argument("manifest", type=Path)
+    scan.add_argument("--slice", default="1")
+    scan.add_argument("--repos", type=Path, required=True)
+    scan.add_argument("--rules", type=Path, required=True)
+    scan.add_argument("--out", type=Path, required=True)
+    scan.add_argument("--only", choices=("vulnerability", "ordinary", "all"), default="all")
+    scan.add_argument("--semgrep", help="semgrep binary (default: PATH)")
+    args = parser.parse_args(argv)
+    if args.command == "rules":
+        report = write_rules(
+            args.out, ruleset_dir=args.ruleset, facts_dir=args.facts, min_precision=args.min_precision, semgrep=args.semgrep
+        )
+        validation = report["validation"]
+        print(
+            f"regex converted {len(report['regex']['converted'])}, skipped {len(report['regex']['skipped'])}; "
+            f"facts converted {len(report['facts']['converted'])}, skipped {len(report['facts']['skipped'])}, "
+            f"lossy {len(report['facts']['lossy'])}; "
+            f"obligations skipped {report['obligations']['skipped_tables']}; "
+            f"validation {'not run' if validation is None else 'exit ' + str(validation['exit'])}"
+        )
+        return 0 if validation is None or validation["exit"] == 0 else 2
+    return run(args.manifest, args.repos, args.rules, args.out.expanduser(), slice_id=args.slice, only=args.only, semgrep=args.semgrep)
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

tests/test_semgrep_arm.py

@@ -0,0 +1,397 @@
+"""Offline coverage of the Semgrep benchmark arm: rule conversion, result mapping, record shape."""
+
+import hashlib
+import importlib
+import json
+import shutil
+import sys
+from pathlib import Path
+
+import pytest
+import yaml
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+semgrep_arm = importlib.import_module("benchmarks.valid.semgrep_arm")
+catalog = importlib.import_module("benchmarks.valid.catalog")
+score = importlib.import_module("benchmarks.unseen.score")
+
+RULES_TOML = """
+[[rule]]
+rule_id = "python-shell-true"
+title = "Subprocess shell execution needs review"
+languages = ["python"]
+cwe = "CWE-78"
+tags = ["syscall_entry"]
+status = "enabled"
+precision_estimate = 0.0
+version = "1"
+pattern = '''subprocess\\.[a-zA-Z_]+\\s*\\([^\\n]*shell\\s*=\\s*True'''
+
+[[rule]]
+rule_id = "php-sql-shadow"
+title = "PHP SQL call composed from a variable"
+languages = ["php"]
+cwe = "CWE-89"
+tags = ["injection"]
+status = "shadow"
+precision_estimate = 0.25
+version = "1"
+pattern = '''mysqli_query\\s*\\([^\\n]*\\$'''
+
+[[rule]]
+rule_id = "php-sql-bounded"
+title = "PHP sprintf-built SQL"
+languages = ["php"]
+cwe = "CWE-89"
+tags = ["injection"]
+status = "enabled"
+precision_estimate = 0.1
+version = "1"
+pattern = '''(sprintf)\\((?=(?:(?!\\);)[\\s\\S]){0,2000}?'%s')'''
+
+[[rule]]
+rule_id = "groovy-xss-unescaped"
+title = "Groovy template unescaped output needs review"
+languages = ["groovy"]
+cwe = "CWE-79"
+tags = ["injection"]
+status = "enabled"
+precision_estimate = 0.0
+version = "1"
+pattern = '''(yieldUnescaped)\\s+[A-Za-z_]'''
+"""
+
+FACTS_TOML = """
+version = "1"
+language = "python"
+
+[[source]]
+id = "request"
+patterns = ["request.args", "request.get_json", "input("]
+
+[[sink]]
+id = "os.system"
+cwe = "CWE-78"
+calls = ["os.system", "subprocess.run"]
+rule_ids = ["python-os-command"]
+
+[[sink]]
+id = "execute"
+cwe = "CWE-89"
+calls = ["execute"]
+rule_ids = []
+
+[[sink]]
+id = "hashlib"
+cwe = "CWE-327"
+calls = ["hashlib.md5", "hashlib.new"]
+rule_ids = []
+weak_literals = ["md5", "sha1"]
+
+[[sanitizer]]
+id = "shlex.quote"
+calls = ["shlex.quote"]
+
+[[sanitizer]]
+id = "parameterized_execute"
+calls = ["execute"]
+parameterized = true
+
+[[sanitizer]]
+id = "url_allowlist"
+calls = ["url_has_allowed_host_and_scheme"]
+guard = true
+"""
+
+SEMGREP_SAMPLE = {
+    "version": "1.179.0",
+    "results": [
+        {
+            "check_id": "rules.ousast.python.taint.os.system",
+            "path": "app/views.py",
+            "start": {"line": 12, "col": 5, "offset": 300},
+            "end": {"line": 12, "col": 30, "offset": 325},
+            "extra": {
+                "message": "os.system: untrusted input reaches os.system, subprocess.run",
+                "metadata": {"family": "injection", "cwe": "CWE-78", "origin": "sink:python:os.system", "precision_estimate": "unmeasured"},
+                "severity": "ERROR",
+                "lines": '    os.system("ls " + c)',
+                "dataflow_trace": {"taint_source": ["CliLoc", [{"path": "app/views.py", "start": {"line": 11}}, "request.args"]]},
+            },
+        },
+        {
+            "check_id": "rules.ousast.python.regex.python-os-command",
+            "path": "app/views.py",
+            "start": {"line": 12, "col": 5, "offset": 300},
+            "end": {"line": 12, "col": 15, "offset": 310},
+            "extra": {
+                "message": "Python OS command execution needs review",
+                "metadata": {"family": "injection", "cwe": "CWE-78", "origin": "python-os-command", "precision_estimate": "0.0"},
+                "severity": "WARNING",
+                "lines": '    os.system("ls " + c)',
+            },
+        },
+    ],
+    "errors": [],
+    "paths": {"scanned": ["app/views.py"]},
+}
+
+
+@pytest.fixture
+def ruleset(tmp_path):
+    root = tmp_path / "ruleset"
+    (root / "python").mkdir(parents=True)
+    (root / "python" / "rules.toml").write_text(RULES_TOML)
+    (root / "obligations").mkdir()
+    (root / "obligations" / "python.toml").write_text('[[operation]]\nid = "x"\n[[discharger]]\nid = "y"\n')
+    facts = root / "semantic"
+    facts.mkdir()
+    (facts / "python.toml").write_text(FACTS_TOML)
+    return root
+
+
+def test_regex_rules_convert_with_metadata_and_skip_reasons(ruleset):
+    converted, skipped, rewrites = semgrep_arm.convert_regex_rules(ruleset)
+    by_origin = {rule["metadata"]["origin"]: rule for rules in converted.values() for rule in rules}
+    shell = by_origin["python-shell-true"]
+    assert shell["languages"] == ["python"]
+    assert shell["pattern-regex"] == r"subprocess\.[a-zA-Z_]+\s*\([^\n]*shell\s*=\s*True"
+    assert shell["metadata"]["family"] == "injection"
+    assert shell["metadata"]["cwe"] == "CWE-78"
+    assert shell["metadata"]["precision_estimate"] == "0.0"
+    assert shell["id"].startswith("ousast.python.regex.")
+    groovy = by_origin["groovy-xss-unescaped"]
+    assert groovy["languages"] == ["generic"]
+    assert groovy["paths"] == {"include": ["*.groovy", "*.tpl"]}
+    assert groovy["metadata"]["family"] == "output_encoding"
+    assert by_origin["php-sql-bounded"]["pattern-regex"] == r"(sprintf)\((?=(?:(?!\);)[\s\S])*?'%s')"
+    assert [rewrite["origin"] for rewrite in rewrites] == ["php-sql-bounded"]
+    assert [entry["origin"] for entry in skipped] == ["php-sql-shadow"]
+    assert "shadow" in skipped[0]["reason"]
+
+
+def test_emit_floor_skips_rules_below_precision(ruleset):
+    _, skipped, _ = semgrep_arm.convert_regex_rules(ruleset, min_precision=0.05)
+    reasons = {entry["origin"]: entry["reason"] for entry in skipped}
+    assert reasons["python-shell-true"].startswith("emit floor")
+    assert reasons["groovy-xss-unescaped"].startswith("emit floor")
+    assert "php-sql-bounded" not in reasons
+
+
+def test_fact_triple_becomes_a_taint_rule(ruleset):
+    converted, skipped, lossy = semgrep_arm.convert_facts(ruleset / "semantic")
+    rules = {rule["metadata"]["origin"]: rule for rule in converted["python"]}
+    taint = rules["sink:python:os.system"]
+    assert taint["mode"] == "taint"
+    assert taint["languages"] == ["python"]
+    assert taint["metadata"] == {
+        "family": "injection",
+        "cwe": "CWE-78",
+        "origin": "sink:python:os.system",
+        "precision_estimate": "unmeasured",
+        "kind": "taint",
+        "calls": ["os.system", "subprocess.run"],
+    }
+    sources = [entry["pattern"] for entry in taint["pattern-sources"][0]["pattern-either"]]
+    assert {"request.args", "request.get_json(...)", "input(...)"} <= set(sources)
+    sinks = [entry["pattern"] for entry in taint["pattern-sinks"][0]["pattern-either"]]
+    assert sinks == ["os.system(...)", "subprocess.run(...)"]
+    sanitizers = [entry["pattern"] for entry in taint["pattern-sanitizers"][0]["pattern-either"]]
+    assert sanitizers == ["shlex.quote(...)"]
+    execute = rules["sink:python:execute"]["pattern-sinks"][0]["pattern-either"]
+    assert execute[0] == {"patterns": [{"pattern": "execute($QUERY)"}, {"focus-metavariable": "$QUERY"}]}
+    weak = rules["sink:python:hashlib"]
+    assert "mode" not in weak and weak["metadata"]["kind"] == "weak_literal" and weak["metadata"]["family"] == "config_secrets"
+    reasons = {entry["origin"]: entry["reason"] for entry in skipped}
+    assert reasons["sanitizer:python:url_allowlist"].startswith("guard")
+    assert reasons["sanitizer:python:parameterized_execute"].startswith("parameterized")
+    assert lossy == []
+
+
+def test_write_rules_emits_one_yaml_per_language_and_a_report(ruleset, tmp_path):
+    out = tmp_path / "rules"
+    report = semgrep_arm.write_rules(out, ruleset_dir=ruleset, facts_dir=ruleset / "semantic", semgrep="/nonexistent/semgrep")
+    assert sorted(path.name for path in out.glob("*.yaml")) == ["groovy.yaml", "php.yaml", "python.yaml"]
+    assert report["files"] == {"groovy.yaml": 1, "php.yaml": 1, "python.yaml": 4}
+    loaded = yaml.safe_load((out / "python.yaml").read_text())
+    ids = [rule["id"] for rule in loaded["rules"]]
+    assert len(ids) == len(set(ids)) == 4
+    for rule in loaded["rules"]:
+        assert set(rule) >= {"id", "languages", "severity", "message", "metadata"}
+        assert {"family", "cwe", "origin", "precision_estimate"} <= set(rule["metadata"])
+        assert isinstance(rule["metadata"]["precision_estimate"], str)
+        assert ("pattern-regex" in rule) or ("pattern-either" in rule) or (rule.get("mode") == "taint")
+    saved = json.loads((out / "conversion_report.json").read_text())
+    assert saved["obligations"]["skipped_tables"] == 2
+    assert saved["regex"]["converted"] == ["groovy-xss-unescaped", "php-sql-bounded", "python-shell-true"]
+    assert saved["facts"]["converted"] == ["sink:python:execute", "sink:python:hashlib", "sink:python:os.system"]
+    # A missing binary is recorded, never reported as a successful validation.
+    assert report["validation"]["exit"] is None and "not found" in report["validation"]["error"]
+
+
+def test_result_to_defect_mapping():
+    defects = [semgrep_arm.result_defect(result) for result in SEMGREP_SAMPLE["results"]]
+    assert defects[0] == {
+        "defect_id": "sink:python:os.system:app/views.py:12",
+        "family": "injection",
+        "cwe": "CWE-78",
+        "locations": [["app/views.py", 12]],
+        "source": "semgrep",
+        "rule_id": "rules.ousast.python.taint.os.system",
+        "origin": "sink:python:os.system",
+        "title": "os.system: untrusted input reaches os.system, subprocess.run",
+        "rationale": "os.system: untrusted input reaches os.system, subprocess.run",
+        "confidence": "ERROR",
+        "dataflow": True,
+    }
+    assert defects[1]["dataflow"] is False
+    assert defects[1]["confidence"] == "WARNING"
+    assert defects[1]["origin"] == "python-os-command"
+
+
+def _change(**overrides):
+    change = dict(
+        id="fixture-positive",
+        repo="fixture/app",
+        repository_url="https://example.invalid/app",
+        kind="vulnerability",
+        base="b" * 40,
+        head="a" * 40,
+        family="injection",
+        cwe="CWE-78",
+        path="app/views.py",
+        function="handler",
+        function_lines=[10, 14],
+        files=[{"path": "app/views.py", "head_lines": [[10, 14]]}],
+        method="fix_site",
+        slice="1",
+    )
+    change.update(overrides)
+    return change
+
+
+def test_record_shape_is_accepted_by_the_scorer(tmp_path):
+    out = tmp_path / "semgrep"
+    change = _change()
+    digest = hashlib.sha256(json.dumps(change, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode()).hexdigest()
+    semgrep_arm._write_json(out / "inputs.json", [{"item_id": change["id"], "change": change, "input_digest": digest}])
+    record = {
+        "id": change["id"],
+        "done": True,
+        "status": "semgrep_covered",
+        "input_digest": digest,
+        "instrument": {
+            "hook_exit": 0,
+            "hook_bytes_read": 412,
+            "changed_bytes": {"app/views.py": 412},
+            "head_verified": True,
+            "base_verified": True,
+            "failure": None,
+            "suspect_fast": False,
+            "semgrep": {"exit": 0, "wall_seconds": 4.9, "scanned": ["app/views.py"], "errors": [], "results": 2},
+        },
+        "push": {
+            "admission": {"defects": [semgrep_arm.result_defect(result) for result in SEMGREP_SAMPLE["results"]], "dispositions": []},
+            "quick_tier": [],
+        },
+        "skipped": [],
+    }
+    semgrep_arm._write_json(semgrep_arm._record_path(out, change["id"]), record)
+    rows = score.load_records(out)
+    assert len(rows) == 1 and rows[0]["status"] == "semgrep_covered"
+    assert score.coverage(rows[0]) == "quick_only"
+    assert score.catches(rows[0]) == ["alerts"]
+    assert score.false_alarm(rows[0])
+    # A family mismatch is not a catch, exactly as the hunter arm is scored.
+    rows = score.load_records(out)
+    rows[0]["family"] = "path"
+    assert score.catches(rows[0]) == []
+    scored = score.score(rows)
+    assert scored["coverage"]["quick_only"] == 1
+
+
+def test_instrument_failure_record_is_not_covered(tmp_path):
+    out = tmp_path / "semgrep"
+    change = _change()
+    digest = "0" * 64
+    semgrep_arm._write_json(out / "inputs.json", [{"item_id": change["id"], "change": change, "input_digest": digest}])
+    record = {
+        "id": change["id"],
+        "done": True,
+        "status": "instrument_failure",
+        "input_digest": digest,
+        "instrument": {
+            "hook_exit": 2,
+            "hook_bytes_read": 0,
+            "changed_bytes": {},
+            "head_verified": True,
+            "base_verified": True,
+            "failure": "semgrep_exit_2",
+            "suspect_fast": False,
+        },
+        "push": {"admission": {"defects": [], "dispositions": []}, "quick_tier": []},
+        "skipped": [],
+    }
+    semgrep_arm._write_json(semgrep_arm._record_path(out, change["id"]), record)
+    rows = score.load_records(out)
+    assert score.coverage(rows[0]) == "instrument_failure"
+    assert not score.false_alarm(rows[0])
+
+
+@pytest.fixture
+def fixture(tmp_path):
+    root = tmp_path / "repo"
+    root.mkdir()
+    git = catalog.local_git(root)
+    git.run("init", "-q")
+    git.run("config", "user.name", "Fixture")
+    git.run("config", "user.email", "[email protected]")
+    heads = []
+    for text in (
+        "import os\nfrom flask import request\n\n\ndef handler():\n    c = request.args.get('c')\n    os.system('ls ' + c)\n",
+        "# Later checkout\n",
+    ):
+        (root / "app.py").write_text(text)
+        git.run("add", "app.py")
+        git.run("-c", "core.hooksPath=/dev/null", "commit", "-qm", "fixture")
+        heads.append(git.run("rev-parse", "HEAD").strip())
+    change = _change(
+        base=heads[1],
+        head=heads[0],
+        path="app.py",
+        function="handler",
+        function_lines=[5, 7],
+        files=[{"path": "app.py", "head_lines": [[5, 7]]}, {"path": "gone.py", "head_lines": [[1, 1]]}],
+    )
+    frozen = tmp_path / "frozen.json"
+    frozen.write_text(
+        json.dumps({"changes": [change], "sha256": hashlib.sha256(json.dumps([change], sort_keys=True).encode()).hexdigest()})
+    )
+    repos = tmp_path / "repos.json"
+    repos.write_text(json.dumps({change["repo"]: str(root)}))
+    return {"root": root, "git": git, "change": change, "frozen": frozen, "repos": repos}
+
+
+@pytest.mark.skipif(shutil.which("semgrep") is None, reason="semgrep binary not installed")
+def test_run_scans_changed_files_and_cleans_up(fixture, tmp_path):
+    rules = tmp_path / "rules"
+    semgrep_arm.write_rules(rules, semgrep="/nonexistent/semgrep")
+    out = tmp_path / "out"
+    assert semgrep_arm.run(fixture["frozen"], fixture["repos"], rules, out, only="vulnerability") == 0
+    rows = score.load_records(out)
+    assert len(rows) == 1
+    record = rows[0]
+    assert record["status"] == "semgrep_covered"
+    assert record["instrument"]["changed_bytes"] == {"app.py": record["instrument"]["hook_bytes_read"]}
+    assert record["instrument"]["hook_bytes_read"] > 0
+    assert record["instrument"]["semgrep"]["exit"] == 0 and not record["instrument"]["suspect_fast"]
+    assert record["skipped"] == [{"path": "gone.py", "reason": "missing_at_head"}]
+    assert record["instrument"]["semgrep"]["scanned"] == ["app.py"]
+    origins = {defect["origin"] for defect in record["push"]["admission"]["defects"]}
+    assert {"python-os-command", "sink:python:os.system"} <= origins
+    assert any(defect["dataflow"] for defect in record["push"]["admission"]["defects"])
+    assert score.coverage(record) == "quick_only"
+    assert score.catches(record) == ["alerts"]
+    assert fixture["git"].run("worktree", "list").strip().count("\n") == 0
+    # A second run resumes without touching the record.
+    assert semgrep_arm.run(fixture["frozen"], fixture["repos"], rules, out, only="vulnerability") == 0