feat(tree-sitter-overlay-extra): parse JS/C/Java overlay via optional extra

Add openultrasast[semantic] with official grammar wheels and one CST walker
into FileIR. quick stays extra-free. Host CLI is not overlay-ready. Default CI
stays extra-free; a semantic job runs pytest -m semantic.

.kiro/specs/tree-sitter-overlay-extra/brief.md

@@ -0,0 +1,45 @@
+# Brief: tree-sitter-overlay-extra
+
+## Problem
+
+AppSec scanning mixed working trees gets `language_unsupported` on JS/C/Java, and Python-only `ast` cannot follow the patterns those languages actually use. The overlay spec promised tree-sitter as the primary engine; the CLI probe never builds `FileIR`. Maintaining hand-rolled parsers would explode the codebase.
+
+## Current State
+
+`parse_tree_sitter_cli` returns `None` even on a successful parse. Python falls back to stdlib `ast`. JS/C/Java stay unadjudicated. `quick` is correctly zero-dep. SAST overlay Youden on the 11-pair textbook slice is 0% vs +9% inventory — not because regex got worse, but because Python overlay fail-closes on ConfigParser-style heap and C/Java never parse.
+
+## Desired Outcome
+
+With the extra installed, tree-sitter grammars produce `FileIR` for Python, JavaScript, C/C++, and Java. Existing taint/facts/dispositions consume that IR unchanged. Without the extra, behavior stays fail-closed unadjudicated. `quick` still has no parser wheels.
+
+## Approach
+
+Add optional extra `openultrasast[semantic]` (`tree-sitter` + official language packages). One CST walker extracts binds and calls into the current `FileIR`. Delete the dead CLI sexp stub. Do not add a fourth language-specific AST visitor.
+
+## Scope
+
+- **In**: extra, probe that prefers the Python API when importable, CST → FileIR, tests with and without the extra.
+- **Out**: Joern queries, inter-file taint, CWE-121 buffer semantics, new pair corpora (owned by `real-world-vfc-slice`).
+
+## Boundary Candidates
+
+- Packaging extra vs core install
+- CST walker vs taint engine (taint stays in `semantic/taint.py`)
+
+## Out of Boundary
+
+- Pattern rules, CWE policy, sandbox, mechanism JSONL schema, pair catalog harvest
+
+## Upstream / Downstream
+
+- **Upstream**: `propose-adjudicate-prove` overlay records and taint
+- **Downstream**: `real-world-vfc-slice` overlay scoring on C/JS/Java pairs
+
+## Existing Spec Touchpoints
+
+- **Extends**: none (new spec). Does not reopen `propose-adjudicate-prove` tasks.
+- **Adjacent**: `three-stage-scan` modes; `quick` must remain zero-dep.
+
+## Constraints
+
+No new required dependency. Missing extra → `language_unsupported` or Python ast, never crash. No LLM parse.

.kiro/specs/tree-sitter-overlay-extra/design.md

@@ -0,0 +1,272 @@
+# Design Document
+
+## Overview
+
+**Purpose.** This feature turns on a real structured parse for overlay adjudication by installing an optional extra. AppSec scans of JavaScript, C/C++, Java, and Python then get `FileIR` from official tree-sitter grammars instead of `language_unsupported`.
+
+**Users.** Default/CI installs keep `quick` and extra-free pytest. Operators who install `openultrasast[semantic]` get overlay promote/demote on the in-scope languages.
+
+**Impact.** `parse_file` prefers extra grammars when they load. The dead CLI parse-success path is removed. Taint, facts, dispositions, sandbox, and the detection gate do not change.
+
+### Goals
+
+- Optional extra with official grammar wheels; core `dependencies` stay empty.
+- One CST walker into existing `FileIR` (`Bind`, `CallSite`, `FunctionIR`).
+- Fail closed: missing extra, missing grammar, or error tree → `unadjudicated`.
+- Dual test matrix: extra-free always; extra-on JS plus C or Java.
+
+### Non-Goals
+
+- Joern queries, inter-file taint, CWE-121 buffer semantics, pair-catalog harvest, LLM parse.
+- New overlay dispositions or evidence rungs.
+- Requiring the extra for `quick` or for importing `openultrasast`.
+
+## Boundary Commitments
+
+### This Spec Owns
+
+- Packaging extra `semantic` and lazy grammar load.
+- CST → `FileIR` projection and `parse_file` dispatch order.
+- Probe semantics: extra/grammar vs host CLI.
+- pytest `semantic` marker and CI extra-on job.
+
+### Out of Boundary
+
+- Overlay dispositions, fact TOML, taint worklist (`propose-adjudicate-prove`).
+- Stage plan, sandbox, hunter suspicion (`three-stage-scan`).
+- Pair catalog slices (`real-world-vfc-slice`).
+- CWE policy, pattern rules, MCP `deep`.
+
+### Allowed Dependencies
+
+- Existing `FileIR` types and `parse_file` callers (`taint.py`, `overlay.py`).
+- `harness_ext` capability-guard pattern (find_spec, no import at module load).
+- External: PyPI `tree-sitter` and grammar packages listed in Technology Stack (extra only).
+
+### Revalidation Triggers
+
+- Changing `FileIR` / `Bind` / `CallSite` fields.
+- Importing `tree_sitter` at module load of `cli` or `semantic/__init__`.
+- Adding core (non-extra) parser dependencies.
+- Using host `tree-sitter` CLI output as overlay IR.
+
+## Architecture
+
+### Existing Architecture Analysis
+
+Overlay already calls `parse_file` → taint → dispositions. Python `ast` works. `parse_tree_sitter_cli` always returns `None`. `tree_sitter_available()` is true if a CLI binary exists, which does not imply a grammar.
+
+### Architecture Pattern and Boundary Map
+
+Selected: **hybrid extra guard + CST walker + existing dispatch.** Same packaging seam as HarnessX. `parse_file` remains the only overlay parse entry.
+
+```mermaid
+flowchart TD
+  overlay[overlay adjudicate]
+  parse[parse_file]
+  extra[has_semantic_extra]
+  gram[grammar_for language]
+  cst[cst walker]
+  ast[python ast]
+  fail[FileIR language_unsupported or parse_failed]
+  taint[taint on FileIR]
+  overlay --> parse
+  parse --> extra
+  extra -->|yes| gram
+  gram -->|loaded| cst
+  cst -->|has_error| fail
+  cst -->|ok| taint
+  extra -->|no| ast
+  gram -->|missing| ast
+  ast -->|python ok| taint
+  ast -->|not python| fail
+```
+
+**Key decisions**
+
+- Extra grammars are the product parse. Host CLI is not a success path.
+- Taint does not import tree-sitter.
+- Dependency direction: extra guard → cst walker → `parse_file` → taint → overlay. No reverse imports.
+
+### Technology Stack
+
+| Layer | Choice / Version | Role in Feature | Notes |
+|-------|------------------|-----------------|-------|
+| Packaging | extra `semantic` | Optional wheels | Core `dependencies` stay `[]` |
+| Parser runtime | `tree-sitter>=0.23,<0.27` | `Language` + `Parser` | Constructor API, not `build_library` |
+| Grammars | `tree-sitter-python`, `tree-sitter-javascript`, `tree-sitter-c`, `tree-sitter-cpp`, `tree-sitter-java` | Per-language CST | `cpp` for `.cpp`; `c` for `.c` |
+| Tests | pytest marker `semantic` | Skip extra-on tests | Default CI extra-free |
+| Types | mypy override `tree_sitter*` | Extra absent in typecheck env | Same as `harnessx.*` |
+
+License of adopted wheels: MIT. Pins are verified at implement time against the selected minor.
+
+## File Structure Plan
+
+```
+src/openultrasast/semantic/
+  extra.py      # has_semantic_extra, grammar_for; no tree_sitter import at module load
+  cst.py        # Node walk → FileIR; node-kind map
+  ir.py         # parse_file dispatch; keep python ast; remove CLI success path
+  engines.py    # stop treating host CLI as overlay-ready
+pyproject.toml
+tests/test_semantic_extra.py
+tests/test_semantic_cst.py
+.github/workflows/ci.yml
+```
+
+### Modified Files
+
+- `pyproject.toml` — extra `semantic`; pytest marker; mypy overrides.
+- `src/openultrasast/semantic/ir.py` — dispatch; delete `parse_tree_sitter_cli` as success path.
+- `src/openultrasast/semantic/engines.py` — `tree_sitter_available` must not mean CLI-only; extra/grammar helpers live in `extra.py`.
+- `src/openultrasast/semantic/__init__.py` — export `has_semantic_extra` only if needed; do not import cst at package load.
+- `tests/test_semantic_overlay.py` — extra-free JS stays `language_unsupported` without monkeypatching a fake IR as “tree-sitter success”.
+- `.github/workflows/ci.yml` — second job with `--extra semantic` running `@pytest.mark.semantic`.
+- `dagger/ci.py` — keep extra-free unless the same extra-on command is cheap to add; default is GHA second job only.
+
+Do not modify `taint.py`, `overlay.py`, `facts.py`, `gate.py`, `pairs.py`.
+
+## System Flows
+
+```mermaid
+flowchart TD
+  start[parse_file]
+  g{grammar_for language}
+  err{root has_error}
+  py{language python}
+  start --> g
+  g -->|yes| err
+  err -->|yes| failed[parse_failed]
+  err -->|no| ir[FileIR engine tree-sitter]
+  g -->|no| py
+  py -->|yes| ast[python-ast FileIR]
+  py -->|no| uns[language_unsupported]
+```
+
+Gating: grammar load failure is missing extra/grammar, not a scan abort. `has_error` is `parse_failed`.
+
+## Requirements Traceability
+
+| Requirement | Summary | Components | Flows |
+|-------------|---------|------------|-------|
+| 1.1–1.3 | Extra-free quick and import | extra.py, pyproject | no grammar load on quick |
+| 2.1–2.4 | Structured parse; JS promote; C/Java demote; same dispositions | extra.py, cst.py, ir.py, existing taint/overlay | parse_file → taint |
+| 3.1–3.5 | Fail closed; Python ast last resort; missing extra does not fail standard | ir.py, extra.py | unsupported / parse_failed |
+| 4.1–4.3 | One walker; no JS/C visitors; CLI not success | cst.py, ir.py | grammar_for only |
+| 5.1–5.5 | No LLM; Joern not required; corroboration; prove/gate unchanged | none new | existing overlay |
+| 6.1–6.3 | Dual tests | markers, CI job | skip vs extra-on job |
+
+## Components and Interfaces
+
+| Component | Domain | Intent | Req Coverage | Key Dependencies | Contracts |
+|-----------|--------|--------|--------------|------------------|-----------|
+| SemanticExtra | Packaging | Probe extra and load grammar | 1.1–1.3, 3.1–3.2, 4.3 | PyPI wheels P0 | Service |
+| CstWalker | Parse | CST to FileIR | 2.1–2.3, 3.4, 4.1 | SemanticExtra P0, FileIR P0 | Service |
+| ParseDispatch | Parse | parse_file order | 2.1, 3.3–3.5, 4.3 | CstWalker P0, python ast P0 | Service |
+| ExtraTestMatrix | Validation | skip vs extra-on CI | 6.1–6.3 | pytest P0 | Batch |
+
+### Parse layer
+
+#### SemanticExtra
+
+| Field | Detail |
+|-------|--------|
+| Intent | Capability guard and per-language grammar load |
+| Requirements | 1.2, 1.3, 3.1, 3.2, 4.3 |
+
+**Contracts**: Service
+
+```python
+def has_semantic_extra() -> bool:
+    """True iff tree_sitter and at least one in-scope grammar module are importable. No eager import of wheels at semantic package load."""
+
+def grammar_for(language: str) -> object | None:
+    """Return a tree_sitter.Language for python, javascript, c, cpp, java, or None. Lazy importlib.import_module. Never raises into the scan."""
+```
+
+- Preconditions: `language` is a preprocess language id.
+- Postconditions: `None` means overlay must not claim tree-sitter IR.
+- Invariants: host CLI path does not make this return non-None.
+
+#### CstWalker
+
+| Field | Detail |
+|-------|--------|
+| Intent | One walk from a tree-sitter tree to FileIR |
+| Requirements | 2.1–2.3, 3.4, 4.1 |
+
+**Contracts**: Service
+
+```python
+def file_ir_from_tree(*, path: str, text: str, language: str, tree: object) -> FileIR:
+    """Project named nodes into Bind, CallSite, FunctionIR. If root.has_error, parse_ok False reason parse_failed. engine is tree-sitter when parse_ok."""
+```
+
+Node-kind map (names are grammar types; walker is shared):
+
+| Kind | python | javascript | c / cpp | java |
+|------|--------|------------|---------|------|
+| function | function_definition | function_declaration, method_definition, arrow_function | function_definition | method_declaration, constructor_declaration |
+| assign | assignment, augmented_assignment | assignment_expression | assignment_expression | assignment_expression |
+| call | call | call_expression | call_expression | method_invocation |
+| name | identifier | identifier | identifier | identifier |
+| string | string | string | string_literal | string_literal |
+
+**Implementation Notes**
+
+- Integration: only `parse_file` calls this after `grammar_for`.
+- Validation: JS `eval(req.query.x)` and C `printf("%s\\n", data)` fixtures with extra on.
+- Risks: node-type aliases (JS `assignment_pattern`); table is the extension point, not a new visitor module per language.
+
+#### ParseDispatch
+
+| Field | Detail |
+|-------|--------|
+| Intent | Existing `parse_file` with extra-first order |
+| Requirements | 2.1, 3.3–3.5, 4.2, 4.3 |
+
+Order: grammar_for → cst; else python ast if `language == python`; else `language_unsupported`. Do not call host CLI for overlay IR.
+
+## Data Models
+
+`FileIR`, `Bind`, `CallSite`, `FunctionIR` stay as in `ir.py`. No new disposition. `engine` values: `tree-sitter` | `python-ast` | `none`.
+
+## Error Handling
+
+| Condition | Overlay | Scan |
+|-----------|---------|------|
+| Extra not installed | language_unsupported for non-Python | standard succeeds |
+| Grammar missing for language | language_unsupported or Python ast | standard succeeds |
+| root.has_error | parse_failed | standard succeeds |
+| Wheel import error | treat as extra absent | do not raise |
+
+## Testing Strategy
+
+- Unit extra-free: import `openultrasast.cli` does not import `tree_sitter`; JS `parse_file` → `language_unsupported`; Python ast still promotes `eval(request.args)`; `quick` scan unchanged (1.1–1.3, 3.1, 3.3, 6.1).
+- Unit extra-on (`pytest.mark.semantic`): JS `eval(req.query.x)` overlay `promote` (2.2); C `printf("%s\\n", data)` overlay `demote` with dominating fact (2.3); syntax-broken JS → `parse_failed` (3.4); `engine == tree-sitter` on success (2.1).
+- Skip: if `not has_semantic_extra()`, marked tests skip, extra-free suite green (6.3).
+- CI: default job extra-free; second job installs extra and runs `-m semantic` (6.2).
+- Do not assert detection-gate Youden (5.5).
+
+## Security Considerations
+
+Parser wheels run in-process on operator source. Same trust as reading the tree today. Extra does not execute target code. Overlay evidence stays `static_corroboration` or below (5.3).
+
+## Performance
+
+Parse is per-file MAP, same file budget as rank. No whole-program index.
+
+## Migration Strategy
+
+Default install behavior unchanged. Operators who want JS/C/Java overlay run `uv sync --extra semantic`. No artifact format migration.
+
+## Supporting References
+
+Grammar load sketch (not runtime-imported in core):
+
+```python
+from tree_sitter import Language, Parser
+import tree_sitter_javascript as tsjs
+parser = Parser(Language(tsjs.language()))
+tree = parser.parse(text.encode())
+```

.kiro/specs/tree-sitter-overlay-extra/requirements.md

@@ -0,0 +1,78 @@
+# Requirements Document
+
+## Introduction
+
+AppSec engineers scanning mixed working trees need overlay adjudication on Python, JavaScript, C/C++, and Java, not only on files a Python stdlib parse can read. Today a `standard` or `deep` scan leaves JS/C/Java inventory as `language_unsupported` even when a host parser CLI is present, so those proposals never promote or demote. This spec makes an **optional semantic extra** the way to turn on a real structured parse for those languages, while `quick` stays usable with no extra packages and a missing extra stays fail-closed.
+
+## Boundary Context
+
+- **In scope**: Optional extra install; overlay using a structured parse for Python, JavaScript, C/C++, and Java when that extra provides a grammar; fail-closed `unadjudicated` when the extra or grammar is missing; `quick` unchanged and extra-free; existing overlay dispositions unchanged.
+- **Out of scope**: New pair corpora or Firefox/VFC harvest (`real-world-vfc-slice`); Joern CPG queries; inter-file or whole-program taint; buffer-size / CWE-121 semantics; language-model parse or taint; replacing pattern inventory.
+- **Adjacent expectations**: `propose-adjudicate-prove` still owns dispositions, facts, taint rules, and prove-budget. `three-stage-scan` still owns `quick` / `standard` / `deep` and the sandbox. This spec only supplies the structured parse those stages already asked for.
+
+## Requirements
+
+### Requirement 1: Core install and quick stay extra-free
+
+**Objective:** As a developer or CI job, I want `quick` inventory with no optional packages, so that a default install never pulls parser wheels.
+
+#### Acceptance Criteria
+
+1. When OpenUltraSAST runs a `quick` scan, the system shall emit inventory proposals without requiring the semantic extra.
+2. The system shall not require the semantic extra to be installed in order to import or run the default OpenUltraSAST command.
+3. If the semantic extra is absent, then a `quick` scan shall still complete and shall not abort because a parser package is missing.
+
+### Requirement 2: Extra enables structured parse on in-scope languages
+
+**Objective:** As an AppSec engineer with the semantic extra installed, I want overlay to read JavaScript, C/C++, Java, and Python as structured source, so that those files can be promoted or demoted instead of left unsupported.
+
+#### Acceptance Criteria
+
+1. Where the semantic extra is installed and a grammar for the file’s language is available, when OpenUltraSAST runs a `standard` or `deep` scan, the system shall use that structured parse as the primary intra-file input to overlay adjudication for Python, JavaScript, C, C++, and Java.
+2. When a JavaScript file contains `eval` of a request query (or the language-equivalent untrusted source to a known sink) and the extra provides a JavaScript grammar, the system shall be able to emit a `promote` overlay disposition for that proposal.
+3. When a C or Java file contains a sink fed only by a constant or a dominating sanitizer (for example a literal format string to `printf`) and the extra provides a grammar for that language, the system shall `demote` that proposal rather than leave it `language_unsupported`.
+4. The system shall feed the structured parse into the existing overlay dispositions (`promote`, `demote`, `unadjudicated`, `coverage`) and shall not introduce a new disposition.
+
+### Requirement 3: Missing extra or grammar fails closed
+
+**Objective:** As an operator without the extra or without a grammar for one language, I want honest `unadjudicated` outcomes, so that silence is not reported as safety.
+
+#### Acceptance Criteria
+
+1. If the semantic extra is not installed, then when OpenUltraSAST adjudicates a JavaScript, C, C++, or Java file, the system shall leave matching inventory proposals `unadjudicated` with `language_unsupported` or `parse_failed`.
+2. If the extra is installed but no grammar is available for that file’s language, then the system shall leave those proposals `unadjudicated` with `language_unsupported` or `parse_failed` and shall not crash the scan.
+3. When the extra is absent, the system shall still try a stdlib parse only for Python and shall otherwise follow the fail-closed rule in criterion 1.
+4. If a file cannot be parsed even with a grammar present (syntax error or incomplete source), then the system shall leave matching proposals `unadjudicated` with `parse_failed`.
+5. The system shall not treat a missing extra as a failed `standard` scan by itself.
+
+### Requirement 4: One parse path, no hand-rolled language visitors
+
+**Objective:** As a maintainer, I want a single structured-parse path for all in-scope languages, so that we do not grow a new visitor per language.
+
+#### Acceptance Criteria
+
+1. When the extra supplies grammars for more than one in-scope language, the system shall extract overlay parse facts (assignments, calls, constants) through one shared structured-parse path rather than a separate visitor per language.
+2. The system shall not add a new language-specific stdlib or regex AST as the product parse engine for JavaScript, C, C++, or Java.
+3. Where a host parser CLI is present but the semantic extra is not, the system shall not claim a successful structured parse for overlay unless the extra’s grammar actually produced one.
+
+### Requirement 5: Overlay and stages stay in their existing contracts
+
+**Objective:** As an operator, I want the extra to change who can be adjudicated, not what a disposition or stage means, so that prove-budget and fail-on rules stay honest.
+
+#### Acceptance Criteria
+
+1. The system shall not use a language model as the parse or taint engine.
+2. The system shall not require Joern for the semantic extra to succeed.
+3. When overlay promotes or demotes from a structured parse, the system shall keep that result at `static_corroboration` or below.
+4. The system shall not select sandbox candidates from demoted or unadjudicated findings solely because a structured parse ran.
+5. The system shall not fold overlay Youden or extra-install status into the stage-1 detection gate (`python -m openultrasast.gate`).
+
+### Requirement 6: Extra present and extra absent are both testable
+
+**Objective:** As a maintainer, I want CI to prove both the extra-free core and the extra-enabled overlay, so that default install and AppSec install cannot silently diverge.
+
+#### Acceptance Criteria
+
+1. When the test suite runs without the semantic extra, the system shall still pass the extra-free `quick` and fail-closed overlay checks in Requirements 1 and 3.
+2. Where the semantic extra is installed in a test environment, the system shall exercise Requirement 2 for at least one JavaScript file and at least one C or Java file.
+3. If a test environment lacks the extra, then the suite shall skip extra-enabled checks rather than fail the extra-free core.

.kiro/specs/tree-sitter-overlay-extra/research.md

@@ -0,0 +1,171 @@
+# Gap Analysis: tree-sitter-overlay-extra
+
+Requirements are **generated but not approved**. Analysis proceeds to inform design; it does not treat requirements as locked.
+
+Steering `product.md` / `tech.md` / `structure.md` are absent. Context is the overlay package, `harness_ext` extra pattern, `pyproject.toml`, CI, and PyPI tree-sitter bindings.
+
+---
+
+## 1. Current State
+
+### Assets that already exist
+
+| Asset | Role today |
+|-------|------------|
+| `src/openultrasast/semantic/ir.py` | `FileIR` / `Bind` / `CallSite`; Python `ast` extractor (~170 lines); `parse_tree_sitter_cli` **always returns `None`** |
+| `src/openultrasast/semantic/engines.py` | Probe: env flag, `importlib.util.find_spec("tree_sitter")`, or `tree-sitter` CLI on PATH |
+| `src/openultrasast/semantic/taint.py` | Language-agnostic taint on `FileIR` (names, constants, fact tables) |
+| `src/openultrasast/semantic/overlay.py` | Dispositions; already consumes `parse_file` |
+| `src/openultrasast/ruleset/semantic/*.toml` | JS/C/Java facts already written |
+| `src/openultrasast/harness_ext.py` | Pattern for optional extra: `find_spec`, lazy import, mypy override, cold-import tests |
+| `pyproject.toml` | `dependencies = []`; only extra is `harnessx` |
+| `tests/test_semantic_overlay.py` | JS unsupported without probe; JS IR only via **monkeypatch**, not a real grammar |
+
+### Conventions
+
+- Optional native/heavy deps stay off the default install; capability probe must not import at module load (`harness_ext`, Docker sandbox probe).
+- mypy `ignore_missing_imports` for extras (`harnessx.*`).
+- pytest marker already exists for `docker`; extra-enabled tests would follow that skip pattern.
+- CI `checks` job: `uv sync --group dev` then pytest — **no optional extras**.
+
+### Integration surfaces
+
+- `parse_file(path, text, language) -> FileIR` is the only seam taint/overlay need.
+- `tree_sitter_available()` is a **presence** probe, not “grammar produced IR”. Host CLI on PATH makes it True even with no PyPI wheels (this environment has `/home/linuxbrew/.../tree-sitter` and **no languages**).
+- Pair eval already falls back to inventory when a language never promotes/demotes.
+
+## 2. Requirements Feasibility
+
+| Req | Need | Gap |
+|-----|------|-----|
+| 1 Extra-free `quick` | Empty core deps; no import of parser wheels at load | **Constraint met.** Missing extra cannot currently break `quick` because there is no extra. Must keep it that way. |
+| 2 Structured parse for Py/JS/C/C++/Java | Real CST → `FileIR`; JS promote; C/Java demote on constant/sanitizer | **Missing.** CLI parse never builds `FileIR`. No `[semantic]` extra. No CST walker. |
+| 3 Fail closed | `language_unsupported` / `parse_failed`; Python ast last resort | **Mostly present** via `parse_file` fallback. Syntax-error Python already `parse_failed`. JS/C/Java without IR already `language_unsupported`. |
+| 4 One shared extract path | Not a visitor per language; CLI without extra must not fake success | **Missing walker.** CLI stub already returns `None` (does not fake success). Probe can still report “available” without IR — **constraint** for Req 4.3 wording vs `tree_sitter_available()`. |
+| 5 Overlay contracts | No LLM parse; Joern not required; corroboration; promotions-only sandbox; gate inventory-only | **Present** in overlay/prove/gate. Do not reopen. |
+| 6 Dual test matrix | Extra-free always; extra-on JS + C/Java; skip if extra absent | **Partial.** Extra-free tests exist. Extra-on tests are mocked. No pytest marker. CI does not install a semantic extra. |
+
+**Complexity signal:** external native-wheel extra + one algorithmic CST walk + pytest/CI matrix. Not CRUD.
+
+**Research needed (design phase, not blockers):**
+
+1. Pin compatible `tree-sitter` + `tree-sitter-{python,javascript,c,cpp,java}` wheel ABIs (`Language(mod.language())` vs older `build_library`). PyPI: `tree-sitter` 0.26 (2026-06-30), `tree-sitter-javascript` 0.25.
+2. Per-language CST node names for assignment, call, identifier, string, function (JS `assignment_expression` vs C `assignment_expression` vs Java `assignment_expression` vs Python `assignment`). Shared walker needs a small node-kind table, not four visitors.
+3. C++: separate `tree-sitter-cpp` vs C grammar; `preprocess` maps `.cpp` to `cpp`.
+4. Whether default CI should add a second job `uv sync --extra semantic` or only skip-marked tests on the extra-free job.
+
+## 3. Implementation Approach Options
+
+### Option A: Extend `ir.py` / `engines.py` only
+
+Put extra pins in `pyproject.toml`, implement API parse + CST walk inside `ir.py`, tighten `tree_sitter_available()`.
+
+- **Pros:** Few files; `parse_file` stays the seam.
+- **Cons:** `ir.py` is already ~277 lines (Python ast + dead CLI). A walker + language table will bloat it and mix packaging probe with extract.
+
+### Option B: New components only
+
+New `semantic/cst.py` + `semantic/extra.py`; leave `ir.py` as Python ast; overlay calls a new entrypoint.
+
+- **Pros:** Clean extra guard.
+- **Cons:** Two parse entrypoints; easy to fork `parse_file` consumers.
+
+### Option C: Hybrid (recommended for design)
+
+- **New:** `semantic/cst.py` — one walker: tree-sitter `Node` → `Bind` / `CallSite` / `FunctionIR` using a node-kind map.
+- **New or thin:** extra guard modeled on `harness_ext` (`has_semantic_extra`, lazy grammar load, never import wheels at module load).
+- **Extend:** `pyproject.toml` `[project.optional-dependencies] semantic = [...]`; mypy override `tree_sitter*`; pytest marker `semantic`; CI extra-free job unchanged; optional second job or skip.
+- **Extend:** `parse_file` prefers extra CST when a grammar loads; else Python ast; else `language_unsupported`. **Delete or stop calling** `parse_tree_sitter_cli` as a success path (Req 4.3).
+- **Do not extend:** `taint.py`, `overlay.py`, facts, gate, pair catalog.
+
+**Trade-offs:** Extra files, but `ir.py` shrinks (drop dead CLI), taint stays language-agnostic, extra follows an existing packaging seam.
+
+## 4. Effort and Risk
+
+- **Effort: M (3–7 days).** Extra pins + walker + dual tests. Overlay/taint reuse. Not a new adjudicator.
+- **Risk: Medium.** Native wheels and grammar ABI pins; CST node-name drift across languages; CI without extra must stay green. Mitigate with skip markers and a version pin table in design.
+
+## 5. Design-phase recommendations
+
+- Prefer **Option C**.
+- Treat `FileIR` as frozen contract; do not add dispositions.
+- Decide probe semantics: `tree_sitter_available` should mean “can produce FileIR for this language”, not “CLI binary exists”.
+- Carry research items 1–4 into design (ABI pins, node-kind map, cpp package, CI job vs skip).
+- Do not pull Joern, pair harvest, or CWE-121 into this spec.
+
+---
+
+## Summary (design discovery)
+
+- **Feature**: tree-sitter-overlay-extra
+- **Discovery Scope**: Extension of `parse_file` / overlay IR; optional extra packaging
+- **Key Findings**:
+  - Official PyPI wheels: `tree-sitter` (0.26), `tree-sitter-python`, `tree-sitter-javascript` (0.25), `tree-sitter-c` (0.24.x), `tree-sitter-cpp` (0.23.4), `tree-sitter-java`. API is `Parser(Language(mod.language()))`; `build_library` is gone.
+  - `FileIR` is the only overlay seam; taint does not need language visitors.
+  - Host CLI presence must not count as a successful overlay parse (Req 4.3).
+
+## Research Log
+
+### PyPI ABI and load API
+
+- **Context**: Pin wheels that work on Python 3.11+ without compiling grammars in CI.
+- **Sources Consulted**: https://pypi.org/project/tree-sitter/ ; https://pypi.org/project/tree-sitter-javascript/ ; https://pypi.org/project/tree-sitter-cpp/ ; py-tree-sitter 0.23 release notes (Language capsule, Parser constructor).
+- **Findings**: Grammars ship abi3 wheels. Load with `Language(tree_sitter_<lang>.language())` then `Parser(language)`. Do not use `Language.build_library`.
+- **Implications**: Extra lists core + five grammar packages. mypy ignores `tree_sitter*`.
+
+### Probe vs grammar
+
+- **Context**: `tree_sitter_available()` is true when CLI is on PATH with zero grammars.
+- **Findings**: CLI parse cannot populate `FileIR` without a walker. Extra Python API is the product parse.
+- **Implications**: Split `has_semantic_extra()` from per-language `grammar_for`. Remove CLI as a success path.
+
+## Architecture Pattern Evaluation
+
+| Option | Description | Strengths | Risks | Notes |
+|--------|-------------|-----------|-------|-------|
+| A Extend ir.py only | Walker + extra inside ir.py | Few files | Bloats 277-line module | Rejected |
+| B New entrypoint | Overlay calls cst.py directly | Clean extra | Forks parse_file | Rejected |
+| C Hybrid | Extra guard + cst walker; parse_file dispatch | Matches harness_ext; FileIR frozen | Node-kind table | **Selected** |
+
+## Design Decisions
+
+### Decision: Adopt official grammar wheels, build one walker
+
+- **Context**: Avoid per-language AST visitors (Req 4).
+- **Alternatives**: tree-sitter CLI sexp; py-tree-sitter-languages (unmaintained); hand-rolled JS/C parsers.
+- **Selected**: `openultrasast[semantic]` = `tree-sitter` plus python/javascript/c/cpp/java grammar packages. One CST walk keyed by a node-kind map.
+- **Rationale**: Wheels are MIT, maintained, match in-scope languages. Taint stays language-agnostic.
+- **Trade-offs**: Native wheels in the extra; ABI pins. Core install stays empty.
+- **Follow-up**: Verify Parser constructor against the pinned `tree-sitter` minor at implement time.
+
+### Decision: Extra-free CI stays default; extra-on is a second job plus skip marker
+
+- **Context**: Req 6 dual matrix.
+- **Selected**: pytest `semantic` marker skips when extra absent. GitHub Actions adds a job `uv sync --extra semantic --group dev` that runs marked tests. Default `checks` job unchanged.
+- **Rationale**: Default CI must not fail without wheels. Extra-on tests must actually run somewhere.
+- **Trade-offs**: Two pytest invocations in CI.
+
+### Decision: has_error implies parse_failed
+
+- **Context**: tree-sitter is error-tolerant; Req 3.4 wants fail-closed on syntax errors.
+- **Selected**: If `tree.root_node.has_error` (or ERROR children), return `FileIR(parse_ok=False, reason=parse_failed)`.
+- **Rationale**: Partial IR would invite false demote.
+
+## Synthesis
+
+- **Generalization**: One `parse_file` dispatch; extra is a grammar source, not a second overlay.
+- **Build vs adopt**: Adopt tree-sitter wheels; build only the FileIR projection (binds/calls).
+- **Simplification**: No CLI sexp walker; no second parse API; no taint/overlay changes.
+
+## Risks & Mitigations
+
+- Grammar ABI mismatch — pin extras together; extra-on CI job catches import/parse failure.
+- Node-type drift — table in cst.py; tests assert JS eval and C printf on real trees.
+- CLI still on PATH — must not set parse_ok.
+
+## References
+
+- https://pypi.org/project/tree-sitter/
+- https://github.com/tree-sitter/py-tree-sitter/releases
+- `src/openultrasast/harness_ext.py` extra guard pattern
+

.kiro/specs/tree-sitter-overlay-extra/spec.json

@@ -0,0 +1,22 @@
+{
+  "feature_name": "tree-sitter-overlay-extra",
+  "created_at": "2026-09-04T14:00:00Z",
+  "updated_at": "2026-09-04T15:00:00Z",
+  "language": "en",
+  "phase": "tasks-generated",
+  "approvals": {
+    "requirements": {
+      "generated": true,
+      "approved": true
+    },
+    "design": {
+      "generated": true,
+      "approved": true
+    },
+    "tasks": {
+      "generated": true,
+      "approved": true
+    }
+  },
+  "ready_for_implementation": true
+}

.kiro/specs/tree-sitter-overlay-extra/tasks.md

@@ -0,0 +1,93 @@
+# Implementation Plan
+
+- [x] 1. Foundation: optional extra and capability probe
+- [x] 1.1 Declare the semantic extra without adding core dependencies
+  - Core install dependencies stay empty.
+  - Optional extra named semantic lists the parser runtime and grammars for Python, JavaScript, C, C++, and Java.
+  - Typecheck ignores the extra modules the same way it ignores the existing optional agentic extra.
+  - Pytest registers a semantic marker for extra-on tests.
+  - Observable: a default sync still has an empty core dependency list; the extra is listed only as optional.
+  - _Requirements: 1.1, 1.2, 6.3_
+  - _Boundary: SemanticExtra_
+
+- [x] 1.2 Probe the extra and load a grammar without importing wheels at package load
+  - Capability check uses importability, not a host CLI binary.
+  - Grammar load is lazy per language and returns nothing when the extra or that grammar is missing.
+  - Importing the overlay package or CLI does not import parser wheels.
+  - Host CLI on PATH does not make a grammar available.
+  - Observable: with the extra absent, the probe is false and a JavaScript language request yields no grammar; importing the CLI in a fresh interpreter does not load tree_sitter.
+  - _Requirements: 1.2, 1.3, 3.1, 3.2, 4.3_
+  - _Boundary: SemanticExtra_
+  - _Depends: 1.1_
+
+- [x] 1.3 Stop treating a host parser CLI as overlay-ready
+  - Overlay-ready means a grammar can be loaded for that language, not that a CLI exists.
+  - Existing env override for tests can still force the probe off.
+  - Observable: a machine with a parser CLI and no extra still reports no overlay grammar for JavaScript.
+  - _Requirements: 4.3, 3.1_
+  - _Boundary: SemanticExtra_
+  - _Depends: 1.2_
+
+- [x] 2. Shared structured parse into existing overlay IR
+- [x] 2.1 Walk one syntax tree into assignments, calls, and functions
+  - One walker serves Python, JavaScript, C, C++, and Java via a node-kind table, not a visitor per language.
+  - Successful trees set engine to tree-sitter and fill the existing IR records overlay taint already consumes.
+  - No new language-specific stdlib or regex AST is added for JavaScript, C, C++, or Java.
+  - Observable: with a JavaScript grammar loaded, `eval(req.query.x)` produces a call record for eval with the query expression as an argument.
+  - _Requirements: 2.1, 4.1, 4.2_
+  - _Boundary: CstWalker_
+  - _Depends: 1.2_
+
+- [x] 2.2 Treat error trees as parse failure
+  - If the tree reports an error, the IR is not parse_ok and the reason is parse_failed.
+  - A syntax-broken file does not yield a fake clean tree for overlay.
+  - Observable: incomplete JavaScript source returns parse_failed rather than a promote or demote.
+  - _Requirements: 3.4_
+  - _Boundary: CstWalker_
+  - _Depends: 2.1_
+
+- [x] 3. Integration: parse dispatch
+- [x] 3.1 Prefer extra grammar, then Python stdlib, then unsupported
+  - Parse order is grammar walk, else Python stdlib parse for Python only, else language_unsupported.
+  - Host CLI output is never used as a successful overlay parse.
+  - Missing extra does not fail a standard scan by itself.
+  - Overlay still emits only promote, demote, unadjudicated, and coverage.
+  - Observable: extra-free JavaScript inventory is unadjudicated language_unsupported; extra-free Python `eval(request.args)` still promotes via stdlib parse; extra-on JavaScript uses engine tree-sitter.
+  - _Requirements: 2.1, 2.4, 3.3, 3.5, 4.2, 4.3_
+  - _Boundary: ParseDispatch_
+  - _Depends: 1.2, 2.1, 2.2_
+
+- [x] 4. Validation: extra-free core, extra-on overlay, CI matrix
+- [x] 4.1 Prove extra-free quick and fail-closed overlay
+  - Quick scan completes without the extra and still emits inventory.
+  - JavaScript, C, and Java stay unadjudicated language_unsupported or parse_failed without the extra.
+  - Python stdlib parse remains the last resort when the extra is absent.
+  - Observable: extra-free pytest covering quick plus JS unsupported is green; detection gate still ignores overlay Youden.
+  - _Requirements: 1.1, 1.3, 3.1, 3.3, 5.5, 6.1_
+  - _Boundary: ExtraTestMatrix_
+  - _Depends: 3.1_
+
+- [x] 4.2 (P) Prove extra-on JavaScript promote and C or Java demote
+  - Marked tests skip when the extra is absent instead of failing the extra-free suite.
+  - With the extra, JavaScript eval of request query promotes.
+  - With the extra, a C or Java sink fed only by a constant or dominating sanitizer demotes rather than language_unsupported.
+  - Evidence stays at static_corroboration or below; sandbox selection is still promotions only.
+  - Observable: extra-on tests pass when wheels are installed and skip otherwise; JS promote and C or Java demote assertions fail if the walker is removed.
+  - _Requirements: 2.2, 2.3, 3.4, 5.3, 5.4, 6.2, 6.3_
+  - _Boundary: ExtraTestMatrix_
+  - _Depends: 3.1_
+
+- [x] 4.3 Run extra-on checks in CI without breaking extra-free CI
+  - Default CI job stays extra-free and must remain green.
+  - A second job installs the semantic extra and runs the semantic-marked tests.
+  - Joern is not required; no language model is used as parse or taint.
+  - Observable: extra-free job does not install parser wheels; extra-on job executes the marked JS and C or Java tests.
+  - _Requirements: 5.1, 5.2, 6.1, 6.2, 6.3_
+  - _Boundary: ExtraTestMatrix_
+  - _Depends: 4.1, 4.2_
+
+## Implementation Notes
+
+- Extra wheels: tree-sitter 0.26 with grammar packages python/javascript/c/cpp/java. `OPENULTRASAST_TREE_SITTER_PROBE=0` simulates extra-absent even when wheels are installed locally.
+- Host `tree-sitter` CLI is not overlay-ready; only lazy `grammar_for` produces FileIR.
+- Default CI stays extra-free; job `semantic` runs `uv sync --extra semantic` then `pytest -m semantic`.

.kiro/steering/roadmap.md

@@ -0,0 +1,47 @@
+# Roadmap
+
+## Overview
+
+OpenUltraSAST can propose sinks with regex and prove a few of them in a sandbox, but adjudication is still Python-`ast` plus a tree-sitter CLI stub. Textbook SAST pairs (OWASP/Juliet) did not improve after the overlay: Youden went from **+9.09% inventory to 0% overlay** on the same 11-pair slice. Real operator trees and browser/C/C++ history are not in the catalog. The next work is to put a real tree-sitter extra under the existing overlay, then grow an offline function-level vuln-vs-fix slice from public VFC datasets (including Firefox/gecko-dev where commits exist) without vendoring whole megarepos.
+
+## Approach Decision
+
+- **Chosen**: Optional tree-sitter extra first, then a harvested VFC slice scored like `github` (honesty dashboard, not smoke gate). Isolated functions only. Firefox is one project filter among Chromium, OpenSSL, curl — not a special-case full mozilla-central clone.
+- **Why**: The overlay already has dispositions, facts, and taint. JS/C/Java stay `language_unsupported` because we refused a parser library. Textbook Youden is the wrong north star; large-project VFCs are the right one. Adopting tree-sitter deletes the need to hand-write language parsers.
+- **Rejected alternatives**:
+  - Whole Firefox tree in CI — multi-GB, memory-safety heavy, security-bug commit messages are deliberately scrubbed.
+  - FixFox dataset — embargoed until 2030.
+  - Joern-as-required or inter-file taint in this phase — deferred by `propose-adjudicate-prove`.
+  - Chasing OWASP Youden with more regex — leaks on goodG2B/false files.
+
+## Scope
+
+- **In**: tree-sitter + grammars as `openultrasast[semantic]`; CST walker into existing FileIR/taint; harvested function-level pairs from CVEfixes/MoreFixes/CWE-Bench-Java/SVEN with provenance; overlay scoring on that slice; keep `quick` zero-dep.
+- **Out**: Cloning mozilla-central/Chromium into the repo; making overlay Youden a merge gate; buffer-size/CWE-121 semantics; inter-file taint; auto-facts from improve; LLM as parser.
+
+## Constraints
+
+- Core `dependencies = []`. Parser wheels live only in an extra.
+- Pair fixtures stay out of `LANGUAGE_MANIFESTS`.
+- Do not vendor 16 GB VFC dumps. Pointers in `datasets.toml`; a small reviewed catalog of isolated functions is what CI runs.
+- DiverseVul-style auto labels (~60% accurate) are not ground truth without review.
+
+## Boundary Strategy
+
+- **Why this split**: Engine (how we parse) is independent of corpus (what we score). The extra can ship without any new pairs; the corpus can score inventory-only until the extra lands, then overlay.
+- **Shared seams to watch**: `FileIR` / `parse_file`; pair catalog `slice` values; overlay vs inventory scoring in `pairs.py`.
+
+## Existing Spec Updates
+
+- [x] propose-adjudicate-prove -- overlay roles, facts, prove-budget, sast overlay scoring (landed). Residual: tree-sitter path returns no IR.
+- [ ] propose-adjudicate-prove -- do not reopen for parsers or corpora; those are the new specs below.
+
+## Direct Implementation Candidates
+
+- [ ] Record inventory vs overlay Youden side-by-side in `ousast pairs --slice sast` JSON (small report field, not a new spec).
+- [ ] Add Firefox/gecko-dev, Chromium, OpenSSL pointers to `benchmarks/pairs/datasets.toml`.
+
+## Specs (dependency order)
+
+- [ ] tree-sitter-overlay-extra -- Optional extra: tree-sitter + grammars, one CST walker, `quick` stays zero-dep. Dependencies: none
+- [ ] real-world-vfc-slice -- Harvest reviewed function-level vuln/fix pairs from large projects (Firefox/gecko-dev, Chromium, OpenSSL, …) as an honesty slice. Dependencies: none