research: Semgrep CE and Opengrep as the quick tier, measured

Engine LGPL-2.1, shelling out carries no obligation; the registry rules are
under the Semgrep Rules License since 2024-12-13 and cannot ship, only our
own converted rules can. Measured 4.7 s startup and 5 to 10 ms per file;
converted regex and taint rules fire for Python, PHP and JavaScript; C
propagation and Groovy do not; one sanitizer semantic we refuse on purpose
is accepted by Semgrep. Recommendation: a Semgrep arm on the valid pilot
first. Recorded in the hunter-learning-loop research note and breadth.toml.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

.kiro/breadth.toml

@@ -252,3 +252,10 @@ status = "review"
 modules = ["push_scoring.py"]   # push_inputs.py moved to benchmarks/push 2026-10-13; push_scoring stays (eligibility.py:18)
 lines = 462
 closes = "U3: move under benchmarks/push or delete if only tests use them"
+
+[capability.quick-rules-and-overlay]
+status = "review"
+modules = ["findings.py", "ruleset", "semantic (taint, overlay, prove_filter)"]
+note = "maintainer 2026-10-13: 'semgrep oss is a good idea'; brief in benchmarks/measurements/2026-10-13-semgrep-brief/ says registry rules are not shippable, our converted rules are; 4.7 s startup floor"
+closes = "run the Semgrep arm on the valid pilot (benchmarks/valid/semgrep_arm.py); if it scores at least as well, retire findings.py, ruleset/*.py and the taint overlay (~1,250 lines) and amend safety-net.md's second-engine rule with the measurement"
+

.kiro/specs/hunter-learning-loop/research.md

@@ -75,8 +75,18 @@
   and Perl grammars that ours handles by regex today. Adopt it as the grammar packaging in `semantic/extra.py`;
   map `.c` to the c grammar first (20 of the 56 rejections clear); cache the parser. `tree-sitter-languages` is
   ABI-broken under tree-sitter 0.26; universal-ctags has no pip build.
-- **Pending**: the Semgrep CE and Opengrep brief (`~/ousast-results/fossil/semgrep-brief.md`). Producers land behind
-  the same contract; the ablation decides whether their columns stay.
+- **Result, Semgrep CE and Opengrep (benchmarks/measurements/2026-10-13-semgrep-brief/BRIEF.md)**: the engine is
+  LGPL-2.1 and shelling out carries no obligation, but every Semgrep-maintained registry rule has been under the
+  Semgrep Rules License since 2024-12-13 (internal use only, no distribution, not in competing products), so
+  nothing but our own converted rules could ship. Measured here: 4.7 s startup floor and about 5 to 10 ms per
+  file after it (the regex tier runs in-process under a second); converted regex and taint rules fire with
+  traces for Python, PHP and JavaScript; C propagation through `strcpy` does not (propagators are Pro); Groovy has
+  no parser; a Semgrep sanitizer silenced the `ORDER BY` shape our `quoted_only` flag refuses on purpose; guards,
+  dispatch facts, obligations, shadow status and emit floors have no equivalent. A swap would retire about 1,250
+  lines. Recommendation taken from the brief: option C first, a Semgrep arm on the valid pilot
+  (`benchmarks/valid/semgrep_arm.py`, our rules converted, scored by the same scorer) before any decision on the
+  quick tier; adopting it later means amending the "no second engine" rule in safety-net.md with that measurement.
+  For this spec, Semgrep findings are a candidate feature column like any other producer.
 - **Research needed**: Semgrep registry rule licence terms for an open-source tool.
 
 ### Training data apart from the pilot

benchmarks/measurements/2026-10-13-semgrep-brief/BRIEF.md

@@ -0,0 +1,179 @@
+# Could Semgrep CE or Opengrep replace the quick-rule engine and the tree-sitter taint overlay?
+
+Date: 2026-10-13. Read-only research; measurements ran on this host (semgrep 1.179.0, 4 cores). Short
+answer: as an engine, yes for most quick rules and the intra-function taint overlay; as a shipped rule
+corpus, no; obligations, sanitizer semantics and measured floors do not carry over. Recommend C first.
+
+## 1. What we have
+
+**Quick rules.** `src/openultrasast/findings.py:45-75` compiles every enabled TOML rule to a Python
+regex, filters by `rule.languages`, skips matches in comments (`:197-218`), and emits a
+`StaticFinding` (`:15-36`: `finding_id, path, title, severity, confidence, evidence_level, rationale,
+line, function_name, reachability_status/evidence/conditions, tags, ranking_priority, status, rung`).
+Severity comes from `policy/CWE_Score.tsv` (168 rows: Flaw Category, CWE ID, Name, Flaw Severity
+0-5, Static, Dynamic) via `resolve_severity` (`findings.py:121`). Floors `min_emit_priority` /
+`min_emit_precision` (`:52-60`) and the ledger overlay (`ruleset/store.py:44-74`) keep low-precision
+rules shadowed. Rule schema, `ruleset/store.py:20-42`: `rule_id, title, languages, cwe, tags,
+pattern, status (enabled|shadow|disabled), min_evidence_level, precision_estimate, version,
+framework, library, fallback_for`. Counts: C 6, Groovy 1, Java 4, JavaScript 12, PHP 18 (6 shadow),
+Python 11 = 52 rules.
+
+Example 1 (`ruleset/python/rules.toml`):
+```toml
+[[rule]]  rule_id = "python-shell-true"   cwe = "CWE-78"   tags = ["syscall_entry"]   languages = ["python"]
+status = "enabled"   precision_estimate = 0.0   version = "1"
+pattern = '''subprocess\.[a-zA-Z_]+\s*\([^\n]*shell\s*=\s*True'''
+```
+Example 2, `ruleset/php/rules.toml:48-59`: `php-sql-unquoted-concat`, `cwe = "CWE-89"`, `framework =
+"wordpress"`, `precision_estimate = 0.034`, a 300-character lookbehind regex over `ORDER BY ... " . $x`.
+
+**Taint overlay.** Facts live in `ruleset/semantic/*.toml` (101 tables: Python 5 sources / 18 sinks /
+4 sanitizers; PHP 3/11/11; JavaScript 2/17/6; Java 2/9/4; C 2/3/1). The dataclasses
+(`semantic/facts.py:27-70`) carry semantics Semgrep has no word for: `SanitizerFact.quoted_only` (an
+escape that only protects a quoted literal), `guard` (a check, safe only where it dominates),
+`parameterized`, `literal_format_arg`; `SinkFact.prefix_fixes_origin`, `origin_anchors`; plus
+`DispatchFact` (string-keyed hook registries) and `LayoutFact`. `semantic/taint.py:1` is "intra-file
+taint on FileIR": per function, parameters start tainted (`:84`), sources match by substring
+(`:222-227`), and the result is promote / demote / unadjudicated / coverage
+(`semantic/overlay.py:32-50`), never a bare match. Obligation checks (`semantic/obligations/`, 1,635
+lines; `ruleset/obligations/*.toml`, 37 tables) find a missing guard, an absence, not a pattern. The
+facts tables also feed the Joern model layer (`model/candidates.py:216`, `model/layout.py:14`).
+
+**Languages, gate, scorers, hook.** `preprocess.py:11-36` maps 15 languages (c, cpp, python,
+javascript, typescript, go, rust, java, groovy, kotlin, ruby, php, csharp, swift, solidity);
+`README.md:167-177` names the three scan modes. `gate.py:37-38` requires recall >= 0.90 and FP < 0.10
+on the fixture manifests (`:30-35`), scored on `quick_scan_findings` (`:80-84`). The pair scorer
+(`pairs.py:1089-1111`) matches an overlay record by `rule_id` in `proposal_id`, by sink id, or by CWE
+inside the named function; obligations only by an `obligation:<kind>` tag (`:1114-1126`). The push
+scorer (`benchmarks/unseen/score.py:49-91`) needs `cwe` (mapped to a family), `path`, `line`,
+`function` on each quick finding. `push/quick.py:60-119` runs the rules over the head side of changed
+files and keeps a hit only on a changed line or in a changed function (`:122-130`); advisory (`:35`).
+
+**Steering.** `.kiro/steering/safety-net.md:85-88`: "Start with the existing detector, earn alert
+admission per capability, and add scheduling sophistication only when it improves the measured
+developer outcome. Do not rebuild a custom IR, add a second engine, or assume incremental Joern
+updates as a shortcut to an unmeasured latency promise."
+
+## 2. Licensing in 2026
+
+- Engine: Semgrep CE is LGPL-2.1 (https://github.com/semgrep/semgrep/blob/develop/LICENSE); CE is
+  "single-file analysis with single-function taint tracking" (https://docs.semgrep.dev/faq/comparisons/opengrep-1).
+- Rules: everything Semgrep maintains in `semgrep-rules` (so `p/default`, `p/python`, `r/...` rows
+  from that repo) moved on 2024-12-13 to the Semgrep Rules License v1.0
+  (https://semgrep.dev/legal/rules-license/): "You may use the rules only for your own internal
+  business purposes." "This license does not allow you to distribute the rules, or to make them
+  available to others as a service." Semgrep's summary: "Vendors may not use them in competing
+  products or SaaS offerings." Shipping them inside this Apache-2.0 tool (`pyproject.toml:10`) is
+  distribution; not allowed. Third-party registry rules keep their own licences. Before the change
+  the rules were LGPL-2.1 + Commons Clause (https://joshcgrossman.com/2025/01/28/whats-going-on-with-sem-open-grep/).
+- Opengrep (https://github.com/opengrep/opengrep): LGPL-2.1 fork of Semgrep v1.100.0 backed by
+  Aikido, Amplify, Endor Labs, Kodem, Orca. Releases 1.30.0/1.30.1/1.30.2 (Sep 7 to Oct 7 2026) plus
+  `v2.0.0-nopython-interfile` alphas (https://github.com/opengrep/opengrep/releases). Adds
+  `--taint-intrafile` (cross-function in one file), fingerprints, per-rule timeouts, signed single
+  binaries, no pip (https://docs.semgrep.dev/faq/comparisons/opengrep). `opengrep-rules` is a
+  2024-12-13 snapshot of semgrep-rules under LGPL-2.1 + Commons Clause, "for research, testing &
+  benchmarking", archived 2025-11-28 (https://github.com/opengrep/opengrep-rules).
+
+## 3. Install and runtime
+
+- `pip install semgrep` 1.180.0 (2026-10-07): Python >= 3.10; Linux x86_64 wheel 70.6 MB, unpacking
+  to a 256 MB static OCaml `semgrep-core` plus libssl (https://pypi.org/project/semgrep/; wheel
+  listing checked locally). A release every 5-13 days (https://github.com/semgrep/semgrep/releases).
+- Offline: `--config <dir> --metrics=off --disable-version-check`; "metrics are never sent" with `off`
+  (https://docs.semgrep.dev/metrics, https://docs.semgrep.dev/cli-reference). No air-gap guarantee is documented.
+- Measured here: startup on an empty tree 4.7 s, 136 MB RSS. 12 pilot checkouts
+  (`~/ousast-results/valid/pilot/hist`, 407 Python files, 3 rules): 8.6 s at `-j 1`, 6.8 s default,
+  so 5-10 ms per file after the fixed 4.7 s; a 1,000-file repo is roughly 10-15 s. Semgrep claims
+  3x from its multicore engine (https://semgrep.dev/blog/2025/benchmarking-semgrep-performance-improvements).
+  The fixed 4.7 s matters for the push hook, whose regex tier runs in-process in under a second.
+
+## 4. Capability
+
+- CE taint: `mode: taint` with `pattern-sources/sinks/sanitizers`, intraprocedural only ("can only
+  analyze interactions within a single function", https://docs.semgrep.dev/semgrep-code/semgrep-pro-engine-intro);
+  `pattern-propagators` and `--pro-intrafile` are Pro (https://docs.semgrep.dev/writing-rules/data-flow/taint-mode).
+  Verified locally: PHP, JavaScript and Python taint rules fire with a `dataflow_trace`; a C rule
+  `argv -> system()` did not fire through `strcpy` (needs a propagator). No Groovy parser: `generic` + regex only.
+- JSON observed on 1.179.0 CE (docs list some as platform-only): `check_id, path,
+  start/end.{line,col,offset}, extra.{message, metadata, severity, lines, fingerprint, metavars,
+  dataflow_trace, engine_kind, is_ignored}`; SARIF 2.1.0 with `fingerprints`
+  (https://docs.semgrep.dev/semgrep-appsec-platform/json-and-sarif). `metadata` is free-form but
+  rejects a float ("expected a list, dict, tree, int or str"): `precision_estimate` becomes a string.
+- Changed-line scoping: `--baseline-commit` reports findings absent at a commit and "aborts run if not
+  currently in a git directory, there are unstaged changes"; per file, so `push/quick.py:122-130` stays on top.
+
+## 5. Rule conversion
+
+Regex rule `python-shell-true` as a syntax pattern or a verbatim PCRE2 regex:
+```yaml
+- id: python-shell-true
+  languages: [python]   severity: WARNING   message: Subprocess shell execution needs review
+  metadata: {cwe: CWE-78, family: injection, tags: [syscall_entry], precision_estimate: "0.0", status: enabled}
+  pattern: subprocess.$F(..., shell=True, ...)   # or pattern-regex: 'subprocess\.[a-zA-Z_]+\s*\([^\n]*shell\s*=\s*True'
+```
+Fact triple (`semantic/python.toml`: source `request`, sink `os.system`, a sanitizer) becomes:
+```yaml
+- id: python-os-command-taint
+  mode: taint   languages: [python]   severity: ERROR   message: request data reaches an OS command sink
+  metadata: {cwe: CWE-78, family: injection, sink: os.system, source: request}
+  pattern-sources: [{pattern-either: [{pattern: request.args}, {pattern: request.form}, {pattern: request.get_json(...)}, {pattern: sys.argv}]}]
+  pattern-sanitizers: [{pattern: shlex.quote(...)}]
+  pattern-sinks: [{pattern-either: [{pattern: os.system(...)}, {pattern: subprocess.run(...)}, {pattern: subprocess.Popen(...)}]}]
+```
+Both validated and fired on a test file (two findings, with traces). Not expressible: `quoted_only`
+(Semgrep's PHP rule silenced `ORDER BY " . mysqli_real_escape_string(...)`, the CVE-2024-1071 shape
+the project keeps), `guard` dominance, `prefix_fixes_origin`, `DispatchFact` routing, every
+obligation, `shadow` status and emit floors (a post-filter on our side), framework priors and
+`fallback_for` (two rule files plus a selector), `uncovered_languages` (`findings.py:78-99`). Regex
+rules on non-code files map to `languages: [generic]` + `pattern-regex`.
+
+## 6. Risks
+
+- Steering forbids "a second engine". Semgrep beside Joern is one; adopting it means amending
+  `safety-net.md` with a measured reason, the rule's own test.
+- LGPL: running `semgrep-core` as a subprocess and reading JSON is "communicate at arm's length"; the
+  GNU FAQ says fork-and-exec with pipes or command-line arguments keeps two programs separate
+  (https://www.gnu.org/licenses/gpl-faq.html#GPLPlugins, #MereAggregation; gnu.org rate-limited
+  today, quoted via mirrors). No obligation on our Apache-2.0 code; shipping the binary in the Docker
+  image obliges only passing on the LGPL text and a source offer.
+- Supply chain: weekly releases, pin version and hash. Opengrep: signed binaries, 2.0 in alpha.
+- Detection quality is unknown: the 52 rules carry measured floors and six shadow rules learned on
+  PHP; gate and pair numbers restart from zero, and Semgrep's rules cannot be shipped to close the gap.
+
+## 7. Comparison, options, recommendation
+
+| | Ours | Semgrep CE | Opengrep |
+|---|---|---|---|
+| Licence | Apache-2.0 | engine LGPL-2.1; rules not open | engine LGPL-2.1; no own rules |
+| Languages with taint | C, Java, JS, PHP, Python (facts) | all parsed languages, no Groovy | same, plus Apex/VB/Crystal |
+| Changed-line scoping | spans + changed function | `--baseline-commit`, per file | same |
+| Offline | yes | yes, three flags | yes |
+| Install size | 0 extra (tree-sitter extra) | 70 MB wheel / 256 MB binary | one binary, similar |
+| JSON / SARIF | own JSON + SARIF | both, with traces | both, with fingerprints |
+| Interprocedural | no (Joern does that) | no (Pro) | within one file |
+
+Lines a swap retires: `findings.py` 223 + `ruleset/*.py` 328 + `semantic/{taint,overlay,engines,
+prove_filter}.py` 695 = about 1,250. Kept: `semantic/ir.py`, `cst.py`, `facts.py` (used by `source.py`,
+`model/candidates.py`, `learn/slice.py`), obligations (1,635), policy, scorers.
+
+- A. Semgrep CE, our rules converted, no registry. Effort L (TOML-to-YAML emitter, adapter to
+  `StaticFinding`, re-measure gate and pairs, hook latency). Risk high: +4.7 s per hook run,
+  sanitizer semantics lost, steering amended.
+- B. Opengrep with our rules. Effort L plus binary packaging. Risk high: smaller project, 2.0 in
+  alpha; gains in-file cross-function taint.
+- C. Keep ours; Semgrep as a benchmark arm. Effort S. Risk low: nothing ships; it answers the one
+  open question (does the engine catch more on the valid pilot?).
+
+**Recommendation.** Do C. The engine is cheap and licence-clean, but this project's value is in the
+facts and the policy around a match (quoted-only escapes, guards, shadow rules, obligations, the
+rung), and none of that moves with the engine. A swap buys a slower hook and an unmeasured gate for
+about 1,250 lines removed. Only a measured catch gap on unseen application repos justifies amending
+the steering rule; C produces that number.
+
+**First increment.** `benchmarks/valid/semgrep_arm.py`: emit `ours.yaml` from the 52 TOML rules and
+five facts files (regex -> `pattern-regex`, facts -> `mode: taint`); run `semgrep scan --config
+ours.yaml --metrics=off --disable-version-check --json` on each head checkout under
+`~/ousast-results/valid/pilot/hist/`; map results to the `quick_tier` shape (`rule, cwe, path, line,
+function, scope` via `head_spans`); write a second replay record; score both with
+`benchmarks/unseen/score.py` (catches per family, false alarms, McNemar). Instrument rule: print
+`paths.scanned` and bytes read per repo before believing a zero.