Open_Ultra_SAST/.kiro/specs/tree-sitter-overlay-extra/research.md

Gap Analysis: tree-sitter-overlay-extra

Requirements are generated but not approved. Analysis proceeds to inform design; it does not treat requirements as locked.

Steering product.md / tech.md / structure.md are absent. Context is the overlay package, harness_ext extra pattern, pyproject.toml, CI, and PyPI tree-sitter bindings.

---

1. Current State

Assets that already exist

Asset	Role today
src/openultrasast/semantic/ir.py	FileIR / Bind / CallSite; Python ast extractor (~170 lines); parse_tree_sitter_cli always returns None
src/openultrasast/semantic/engines.py	Probe: env flag, importlib.util.find_spec("tree_sitter"), or tree-sitter CLI on PATH
src/openultrasast/semantic/taint.py	Language-agnostic taint on FileIR (names, constants, fact tables)
src/openultrasast/semantic/overlay.py	Dispositions; already consumes parse_file
src/openultrasast/ruleset/semantic/*.toml	JS/C/Java facts already written
src/openultrasast/harness_ext.py	Pattern for optional extra: find_spec, lazy import, mypy override, cold-import tests
pyproject.toml	dependencies = []; only extra is harnessx
tests/test_semantic_overlay.py	JS unsupported without probe; JS IR only via monkeypatch, not a real grammar

Conventions

• Optional native/heavy deps stay off the default install; capability probe must not import at module load (harness_ext, Docker sandbox probe).
• mypy ignore_missing_imports for extras (harnessx.*).
• pytest marker already exists for docker; extra-enabled tests would follow that skip pattern.
• CI checks job: uv sync --group dev then pytest, no optional extras.

Integration surfaces

• parse_file(path, text, language) -> FileIR is the only seam taint/overlay need.
• tree_sitter_available() is a presence probe, not "grammar produced IR". Host CLI on PATH makes it True even with no PyPI wheels (this environment has /home/linuxbrew/.../tree-sitter and no languages).
• Pair eval already falls back to inventory when a language never promotes/demotes.

2. Requirements Feasibility

Req	Need	Gap
1 Extra-free quick	Empty core deps; no import of parser wheels at load	Constraint met. Missing extra cannot currently break quick because there is no extra. Must keep it that way.
2 Structured parse for Py/JS/C/C++/Java	Real CST → FileIR; JS promote; C/Java demote on constant/sanitizer	Missing. CLI parse never builds FileIR. No [semantic] extra. No CST walker.
3 Fail closed	language_unsupported / parse_failed; Python ast last resort	Mostly present via parse_file fallback. Syntax-error Python already parse_failed. JS/C/Java without IR already language_unsupported.
4 One shared extract path	Not a visitor per language; CLI without extra must not fake success	Missing walker. CLI stub already returns None (does not fake success). Probe can still report “available” without IR — constraint for Req 4.3 wording vs tree_sitter_available().
5 Overlay contracts	No LLM parse; Joern not required; corroboration; promotions-only sandbox; gate inventory-only	Present in overlay/prove/gate. Do not reopen.
6 Dual test matrix	Extra-free always; extra-on JS + C/Java; skip if extra absent	Partial. Extra-free tests exist. Extra-on tests are mocked. No pytest marker. CI does not install a semantic extra.

Complexity signal: external native-wheel extra + one algorithmic CST walk + pytest/CI matrix. Not CRUD.

Research needed (design phase, not blockers):

1. Pin compatible tree-sitter + tree-sitter-{python,javascript,c,cpp,java} wheel ABIs (Language(mod.language()) vs older build_library). PyPI: tree-sitter 0.26 (2026-06-30), tree-sitter-javascript 0.25.
2. Per-language CST node names for assignment, call, identifier, string, function (JS assignment_expression vs C assignment_expression vs Java assignment_expression vs Python assignment). Shared walker needs a small node-kind table, not four visitors.
3. C++: separate tree-sitter-cpp vs C grammar; preprocess maps .cpp to cpp.
4. Whether default CI should add a second job uv sync --extra semantic or only skip-marked tests on the extra-free job.

3. Implementation Approach Options

Option A: Extend ir.py / engines.py only

Put extra pins in pyproject.toml, implement API parse + CST walk inside ir.py, tighten tree_sitter_available().

• Pros: Few files; parse_file stays the seam.
• Cons: ir.py is already ~277 lines (Python ast + dead CLI). A walker + language table will bloat it and mix packaging probe with extract.

Option B: New components only

New semantic/cst.py + semantic/extra.py; leave ir.py as Python ast; overlay calls a new entrypoint.

• Pros: Clean extra guard.
• Cons: Two parse entrypoints; easy to fork parse_file consumers.

Option C: Hybrid (recommended for design)

• New: semantic/cst.py: one walker: tree-sitter Node → Bind / CallSite / FunctionIR using a node-kind map.
• New or thin: extra guard modeled on harness_ext (has_semantic_extra, lazy grammar load, never import wheels at module load).
• Extend: pyproject.toml [project.optional-dependencies] semantic = [...]; mypy override tree_sitter*; pytest marker semantic; CI extra-free job unchanged; optional second job or skip.
• Extend: parse_file prefers extra CST when a grammar loads; else Python ast; else language_unsupported. Delete or stop calling parse_tree_sitter_cli as a success path (Req 4.3).
• Do not extend: taint.py, overlay.py, facts, gate, pair catalog.

Trade-offs: Extra files, but ir.py shrinks (drop dead CLI), taint stays language-agnostic, extra follows an existing packaging seam.

4. Effort and Risk

• Effort: M (3–7 days). Extra pins + walker + dual tests. Overlay/taint reuse. Not a new adjudicator.
• Risk: Medium. Native wheels and grammar ABI pins; CST node-name drift across languages; CI without extra must stay green. Mitigate with skip markers and a version pin table in design.

5. Design-phase recommendations

• Prefer Option C.
• Treat FileIR as frozen contract; do not add dispositions.
• Decide probe semantics: tree_sitter_available should mean "can produce FileIR for this language", not "CLI binary exists".
• Carry research items 1–4 into design (ABI pins, node-kind map, cpp package, CI job vs skip).
• Do not pull Joern, pair harvest, or CWE-121 into this spec.

---

Summary (design discovery)

• Feature: tree-sitter-overlay-extra
• Discovery Scope: Extension of parse_file / overlay IR; optional extra packaging
• Key Findings:
  • Official PyPI wheels: tree-sitter (0.26), tree-sitter-python, tree-sitter-javascript (0.25), tree-sitter-c (0.24.x), tree-sitter-cpp (0.23.4), tree-sitter-java. API is Parser(Language(mod.language())); build_library is gone.
  • FileIR is the only overlay seam; taint does not need language visitors.
  • Host CLI presence must not count as a successful overlay parse (Req 4.3).

Research Log

PyPI ABI and load API

• Context: Pin wheels that work on Python 3.11+ without compiling grammars in CI.
• Sources Consulted: https://pypi.org/project/tree-sitter/ ; https://pypi.org/project/tree-sitter-javascript/ ; https://pypi.org/project/tree-sitter-cpp/ ; py-tree-sitter 0.23 release notes (Language capsule, Parser constructor).
• Findings: Grammars ship abi3 wheels. Load with Language(tree_sitter_<lang>.language()) then Parser(language). Do not use Language.build_library.
• Implications: Extra lists core + five grammar packages. mypy ignores tree_sitter*.

Probe vs grammar

• Context: tree_sitter_available() is true when CLI is on PATH with zero grammars.
• Findings: CLI parse cannot populate FileIR without a walker. Extra Python API is the product parse.
• Implications: Split has_semantic_extra() from per-language grammar_for. Remove CLI as a success path.

Architecture Pattern Evaluation

Option	Description	Strengths	Risks	Notes
A Extend ir.py only	Walker + extra inside ir.py	Few files	Bloats 277-line module	Rejected
B New entrypoint	Overlay calls cst.py directly	Clean extra	Forks parse_file	Rejected
C Hybrid	Extra guard + cst walker; parse_file dispatch	Matches harness_ext; FileIR frozen	Node-kind table	Selected

Design Decisions

Decision: Adopt official grammar wheels, build one walker

• Context: Avoid per-language AST visitors (Req 4).
• Alternatives: tree-sitter CLI sexp; py-tree-sitter-languages (unmaintained); hand-rolled JS/C parsers.
• Selected: openultrasast[semantic] = tree-sitter plus python/javascript/c/cpp/java grammar packages. One CST walk keyed by a node-kind map.
• Rationale: Wheels are MIT, maintained, match in-scope languages. Taint stays language-agnostic.
• Trade-offs: Native wheels in the extra; ABI pins. Core install stays empty.
• Follow-up: Verify Parser constructor against the pinned tree-sitter minor at implement time.

Decision: Extra-free CI stays default; extra-on is a second job plus skip marker

• Context: Req 6 dual matrix.
• Selected: pytest semantic marker skips when extra absent. GitHub Actions adds a job uv sync --extra semantic --group dev that runs marked tests. Default checks job unchanged.
• Rationale: Default CI must not fail without wheels. Extra-on tests must actually run somewhere.
• Trade-offs: Two pytest invocations in CI.

Decision: has_error implies parse_failed

• Context: tree-sitter is error-tolerant; Req 3.4 wants fail-closed on syntax errors.
• Selected: If tree.root_node.has_error (or ERROR children), return FileIR(parse_ok=False, reason=parse_failed).
• Rationale: Partial IR would invite false demote.

Synthesis

• Generalization: One parse_file dispatch; extra is a grammar source, not a second overlay.
• Build vs adopt: Adopt tree-sitter wheels; build only the FileIR projection (binds/calls).
• Simplification: No CLI sexp walker; no second parse API; no taint/overlay changes.

Risks & Mitigations

• Grammar ABI mismatch: pin extras together; extra-on CI job catches import/parse failure.
• Node-type drift: table in cst.py; tests assert JS eval and C printf on real trees.
• CLI still on PATH: must not set parse_ok.

References

• https://pypi.org/project/tree-sitter/
• https://github.com/tree-sitter/py-tree-sitter/releases
• src/openultrasast/harness_ext.py extra guard pattern