diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8184317..e5a74ee 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -32,6 +32,19 @@ jobs: run: uv run python -m openultrasast.pair_gate - run: uv run python -m compileall src tests + semantic: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v5 + - uses: astral-sh/setup-uv@v7 + with: + enable-cache: true + - uses: actions/setup-python@v6 + with: + python-version-file: .python-version + - run: uv sync --group dev --extra semantic + - run: uv run pytest -m semantic + dagger: runs-on: ubuntu-latest timeout-minutes: 20 diff --git a/.kiro/specs/tree-sitter-overlay-extra/brief.md b/.kiro/specs/tree-sitter-overlay-extra/brief.md new file mode 100644 index 0000000..9b8d075 --- /dev/null +++ b/.kiro/specs/tree-sitter-overlay-extra/brief.md @@ -0,0 +1,45 @@ +# Brief: tree-sitter-overlay-extra + +## Problem + +AppSec scanning mixed working trees gets `language_unsupported` on JS/C/Java, and Python-only `ast` cannot follow the patterns those languages actually use. The overlay spec promised tree-sitter as the primary engine; the CLI probe never builds `FileIR`. Maintaining hand-rolled parsers would explode the codebase. + +## Current State + +`parse_tree_sitter_cli` returns `None` even on a successful parse. Python falls back to stdlib `ast`. JS/C/Java stay unadjudicated. `quick` is correctly zero-dep. SAST overlay Youden on the 11-pair textbook slice is 0% vs +9% inventory — not because regex got worse, but because Python overlay fail-closes on ConfigParser-style heap and C/Java never parse. + +## Desired Outcome + +With the extra installed, tree-sitter grammars produce `FileIR` for Python, JavaScript, C/C++, and Java. Existing taint/facts/dispositions consume that IR unchanged. Without the extra, behavior stays fail-closed unadjudicated. `quick` still has no parser wheels. + +## Approach + +Add optional extra `openultrasast[semantic]` (`tree-sitter` + official language packages). One CST walker extracts binds and calls into the current `FileIR`. Delete the dead CLI sexp stub. Do not add a fourth language-specific AST visitor. + +## Scope + +- **In**: extra, probe that prefers the Python API when importable, CST → FileIR, tests with and without the extra. +- **Out**: Joern queries, inter-file taint, CWE-121 buffer semantics, new pair corpora (owned by `real-world-vfc-slice`). + +## Boundary Candidates + +- Packaging extra vs core install +- CST walker vs taint engine (taint stays in `semantic/taint.py`) + +## Out of Boundary + +- Pattern rules, CWE policy, sandbox, mechanism JSONL schema, pair catalog harvest + +## Upstream / Downstream + +- **Upstream**: `propose-adjudicate-prove` overlay records and taint +- **Downstream**: `real-world-vfc-slice` overlay scoring on C/JS/Java pairs + +## Existing Spec Touchpoints + +- **Extends**: none (new spec). Does not reopen `propose-adjudicate-prove` tasks. +- **Adjacent**: `three-stage-scan` modes; `quick` must remain zero-dep. + +## Constraints + +No new required dependency. Missing extra → `language_unsupported` or Python ast, never crash. No LLM parse. diff --git a/.kiro/specs/tree-sitter-overlay-extra/design.md b/.kiro/specs/tree-sitter-overlay-extra/design.md new file mode 100644 index 0000000..6887f8e --- /dev/null +++ b/.kiro/specs/tree-sitter-overlay-extra/design.md @@ -0,0 +1,272 @@ +# Design Document + +## Overview + +**Purpose.** This feature turns on a real structured parse for overlay adjudication by installing an optional extra. AppSec scans of JavaScript, C/C++, Java, and Python then get `FileIR` from official tree-sitter grammars instead of `language_unsupported`. + +**Users.** Default/CI installs keep `quick` and extra-free pytest. Operators who install `openultrasast[semantic]` get overlay promote/demote on the in-scope languages. + +**Impact.** `parse_file` prefers extra grammars when they load. The dead CLI parse-success path is removed. Taint, facts, dispositions, sandbox, and the detection gate do not change. + +### Goals + +- Optional extra with official grammar wheels; core `dependencies` stay empty. +- One CST walker into existing `FileIR` (`Bind`, `CallSite`, `FunctionIR`). +- Fail closed: missing extra, missing grammar, or error tree → `unadjudicated`. +- Dual test matrix: extra-free always; extra-on JS plus C or Java. + +### Non-Goals + +- Joern queries, inter-file taint, CWE-121 buffer semantics, pair-catalog harvest, LLM parse. +- New overlay dispositions or evidence rungs. +- Requiring the extra for `quick` or for importing `openultrasast`. + +## Boundary Commitments + +### This Spec Owns + +- Packaging extra `semantic` and lazy grammar load. +- CST → `FileIR` projection and `parse_file` dispatch order. +- Probe semantics: extra/grammar vs host CLI. +- pytest `semantic` marker and CI extra-on job. + +### Out of Boundary + +- Overlay dispositions, fact TOML, taint worklist (`propose-adjudicate-prove`). +- Stage plan, sandbox, hunter suspicion (`three-stage-scan`). +- Pair catalog slices (`real-world-vfc-slice`). +- CWE policy, pattern rules, MCP `deep`. + +### Allowed Dependencies + +- Existing `FileIR` types and `parse_file` callers (`taint.py`, `overlay.py`). +- `harness_ext` capability-guard pattern (find_spec, no import at module load). +- External: PyPI `tree-sitter` and grammar packages listed in Technology Stack (extra only). + +### Revalidation Triggers + +- Changing `FileIR` / `Bind` / `CallSite` fields. +- Importing `tree_sitter` at module load of `cli` or `semantic/__init__`. +- Adding core (non-extra) parser dependencies. +- Using host `tree-sitter` CLI output as overlay IR. + +## Architecture + +### Existing Architecture Analysis + +Overlay already calls `parse_file` → taint → dispositions. Python `ast` works. `parse_tree_sitter_cli` always returns `None`. `tree_sitter_available()` is true if a CLI binary exists, which does not imply a grammar. + +### Architecture Pattern and Boundary Map + +Selected: **hybrid extra guard + CST walker + existing dispatch.** Same packaging seam as HarnessX. `parse_file` remains the only overlay parse entry. + +```mermaid +flowchart TD + overlay[overlay adjudicate] + parse[parse_file] + extra[has_semantic_extra] + gram[grammar_for language] + cst[cst walker] + ast[python ast] + fail[FileIR language_unsupported or parse_failed] + taint[taint on FileIR] + overlay --> parse + parse --> extra + extra -->|yes| gram + gram -->|loaded| cst + cst -->|has_error| fail + cst -->|ok| taint + extra -->|no| ast + gram -->|missing| ast + ast -->|python ok| taint + ast -->|not python| fail +``` + +**Key decisions** + +- Extra grammars are the product parse. Host CLI is not a success path. +- Taint does not import tree-sitter. +- Dependency direction: extra guard → cst walker → `parse_file` → taint → overlay. No reverse imports. + +### Technology Stack + +| Layer | Choice / Version | Role in Feature | Notes | +|-------|------------------|-----------------|-------| +| Packaging | extra `semantic` | Optional wheels | Core `dependencies` stay `[]` | +| Parser runtime | `tree-sitter>=0.23,<0.27` | `Language` + `Parser` | Constructor API, not `build_library` | +| Grammars | `tree-sitter-python`, `tree-sitter-javascript`, `tree-sitter-c`, `tree-sitter-cpp`, `tree-sitter-java` | Per-language CST | `cpp` for `.cpp`; `c` for `.c` | +| Tests | pytest marker `semantic` | Skip extra-on tests | Default CI extra-free | +| Types | mypy override `tree_sitter*` | Extra absent in typecheck env | Same as `harnessx.*` | + +License of adopted wheels: MIT. Pins are verified at implement time against the selected minor. + +## File Structure Plan + +``` +src/openultrasast/semantic/ + extra.py # has_semantic_extra, grammar_for; no tree_sitter import at module load + cst.py # Node walk → FileIR; node-kind map + ir.py # parse_file dispatch; keep python ast; remove CLI success path + engines.py # stop treating host CLI as overlay-ready +pyproject.toml +tests/test_semantic_extra.py +tests/test_semantic_cst.py +.github/workflows/ci.yml +``` + +### Modified Files + +- `pyproject.toml` — extra `semantic`; pytest marker; mypy overrides. +- `src/openultrasast/semantic/ir.py` — dispatch; delete `parse_tree_sitter_cli` as success path. +- `src/openultrasast/semantic/engines.py` — `tree_sitter_available` must not mean CLI-only; extra/grammar helpers live in `extra.py`. +- `src/openultrasast/semantic/__init__.py` — export `has_semantic_extra` only if needed; do not import cst at package load. +- `tests/test_semantic_overlay.py` — extra-free JS stays `language_unsupported` without monkeypatching a fake IR as “tree-sitter success”. +- `.github/workflows/ci.yml` — second job with `--extra semantic` running `@pytest.mark.semantic`. +- `dagger/ci.py` — keep extra-free unless the same extra-on command is cheap to add; default is GHA second job only. + +Do not modify `taint.py`, `overlay.py`, `facts.py`, `gate.py`, `pairs.py`. + +## System Flows + +```mermaid +flowchart TD + start[parse_file] + g{grammar_for language} + err{root has_error} + py{language python} + start --> g + g -->|yes| err + err -->|yes| failed[parse_failed] + err -->|no| ir[FileIR engine tree-sitter] + g -->|no| py + py -->|yes| ast[python-ast FileIR] + py -->|no| uns[language_unsupported] +``` + +Gating: grammar load failure is missing extra/grammar, not a scan abort. `has_error` is `parse_failed`. + +## Requirements Traceability + +| Requirement | Summary | Components | Flows | +|-------------|---------|------------|-------| +| 1.1–1.3 | Extra-free quick and import | extra.py, pyproject | no grammar load on quick | +| 2.1–2.4 | Structured parse; JS promote; C/Java demote; same dispositions | extra.py, cst.py, ir.py, existing taint/overlay | parse_file → taint | +| 3.1–3.5 | Fail closed; Python ast last resort; missing extra does not fail standard | ir.py, extra.py | unsupported / parse_failed | +| 4.1–4.3 | One walker; no JS/C visitors; CLI not success | cst.py, ir.py | grammar_for only | +| 5.1–5.5 | No LLM; Joern not required; corroboration; prove/gate unchanged | none new | existing overlay | +| 6.1–6.3 | Dual tests | markers, CI job | skip vs extra-on job | + +## Components and Interfaces + +| Component | Domain | Intent | Req Coverage | Key Dependencies | Contracts | +|-----------|--------|--------|--------------|------------------|-----------| +| SemanticExtra | Packaging | Probe extra and load grammar | 1.1–1.3, 3.1–3.2, 4.3 | PyPI wheels P0 | Service | +| CstWalker | Parse | CST to FileIR | 2.1–2.3, 3.4, 4.1 | SemanticExtra P0, FileIR P0 | Service | +| ParseDispatch | Parse | parse_file order | 2.1, 3.3–3.5, 4.3 | CstWalker P0, python ast P0 | Service | +| ExtraTestMatrix | Validation | skip vs extra-on CI | 6.1–6.3 | pytest P0 | Batch | + +### Parse layer + +#### SemanticExtra + +| Field | Detail | +|-------|--------| +| Intent | Capability guard and per-language grammar load | +| Requirements | 1.2, 1.3, 3.1, 3.2, 4.3 | + +**Contracts**: Service + +```python +def has_semantic_extra() -> bool: + """True iff tree_sitter and at least one in-scope grammar module are importable. No eager import of wheels at semantic package load.""" + +def grammar_for(language: str) -> object | None: + """Return a tree_sitter.Language for python, javascript, c, cpp, java, or None. Lazy importlib.import_module. Never raises into the scan.""" +``` + +- Preconditions: `language` is a preprocess language id. +- Postconditions: `None` means overlay must not claim tree-sitter IR. +- Invariants: host CLI path does not make this return non-None. + +#### CstWalker + +| Field | Detail | +|-------|--------| +| Intent | One walk from a tree-sitter tree to FileIR | +| Requirements | 2.1–2.3, 3.4, 4.1 | + +**Contracts**: Service + +```python +def file_ir_from_tree(*, path: str, text: str, language: str, tree: object) -> FileIR: + """Project named nodes into Bind, CallSite, FunctionIR. If root.has_error, parse_ok False reason parse_failed. engine is tree-sitter when parse_ok.""" +``` + +Node-kind map (names are grammar types; walker is shared): + +| Kind | python | javascript | c / cpp | java | +|------|--------|------------|---------|------| +| function | function_definition | function_declaration, method_definition, arrow_function | function_definition | method_declaration, constructor_declaration | +| assign | assignment, augmented_assignment | assignment_expression | assignment_expression | assignment_expression | +| call | call | call_expression | call_expression | method_invocation | +| name | identifier | identifier | identifier | identifier | +| string | string | string | string_literal | string_literal | + +**Implementation Notes** + +- Integration: only `parse_file` calls this after `grammar_for`. +- Validation: JS `eval(req.query.x)` and C `printf("%s\\n", data)` fixtures with extra on. +- Risks: node-type aliases (JS `assignment_pattern`); table is the extension point, not a new visitor module per language. + +#### ParseDispatch + +| Field | Detail | +|-------|--------| +| Intent | Existing `parse_file` with extra-first order | +| Requirements | 2.1, 3.3–3.5, 4.2, 4.3 | + +Order: grammar_for → cst; else python ast if `language == python`; else `language_unsupported`. Do not call host CLI for overlay IR. + +## Data Models + +`FileIR`, `Bind`, `CallSite`, `FunctionIR` stay as in `ir.py`. No new disposition. `engine` values: `tree-sitter` | `python-ast` | `none`. + +## Error Handling + +| Condition | Overlay | Scan | +|-----------|---------|------| +| Extra not installed | language_unsupported for non-Python | standard succeeds | +| Grammar missing for language | language_unsupported or Python ast | standard succeeds | +| root.has_error | parse_failed | standard succeeds | +| Wheel import error | treat as extra absent | do not raise | + +## Testing Strategy + +- Unit extra-free: import `openultrasast.cli` does not import `tree_sitter`; JS `parse_file` → `language_unsupported`; Python ast still promotes `eval(request.args)`; `quick` scan unchanged (1.1–1.3, 3.1, 3.3, 6.1). +- Unit extra-on (`pytest.mark.semantic`): JS `eval(req.query.x)` overlay `promote` (2.2); C `printf("%s\\n", data)` overlay `demote` with dominating fact (2.3); syntax-broken JS → `parse_failed` (3.4); `engine == tree-sitter` on success (2.1). +- Skip: if `not has_semantic_extra()`, marked tests skip, extra-free suite green (6.3). +- CI: default job extra-free; second job installs extra and runs `-m semantic` (6.2). +- Do not assert detection-gate Youden (5.5). + +## Security Considerations + +Parser wheels run in-process on operator source. Same trust as reading the tree today. Extra does not execute target code. Overlay evidence stays `static_corroboration` or below (5.3). + +## Performance + +Parse is per-file MAP, same file budget as rank. No whole-program index. + +## Migration Strategy + +Default install behavior unchanged. Operators who want JS/C/Java overlay run `uv sync --extra semantic`. No artifact format migration. + +## Supporting References + +Grammar load sketch (not runtime-imported in core): + +```python +from tree_sitter import Language, Parser +import tree_sitter_javascript as tsjs +parser = Parser(Language(tsjs.language())) +tree = parser.parse(text.encode()) +``` diff --git a/.kiro/specs/tree-sitter-overlay-extra/requirements.md b/.kiro/specs/tree-sitter-overlay-extra/requirements.md new file mode 100644 index 0000000..7030dd0 --- /dev/null +++ b/.kiro/specs/tree-sitter-overlay-extra/requirements.md @@ -0,0 +1,78 @@ +# Requirements Document + +## Introduction + +AppSec engineers scanning mixed working trees need overlay adjudication on Python, JavaScript, C/C++, and Java, not only on files a Python stdlib parse can read. Today a `standard` or `deep` scan leaves JS/C/Java inventory as `language_unsupported` even when a host parser CLI is present, so those proposals never promote or demote. This spec makes an **optional semantic extra** the way to turn on a real structured parse for those languages, while `quick` stays usable with no extra packages and a missing extra stays fail-closed. + +## Boundary Context + +- **In scope**: Optional extra install; overlay using a structured parse for Python, JavaScript, C/C++, and Java when that extra provides a grammar; fail-closed `unadjudicated` when the extra or grammar is missing; `quick` unchanged and extra-free; existing overlay dispositions unchanged. +- **Out of scope**: New pair corpora or Firefox/VFC harvest (`real-world-vfc-slice`); Joern CPG queries; inter-file or whole-program taint; buffer-size / CWE-121 semantics; language-model parse or taint; replacing pattern inventory. +- **Adjacent expectations**: `propose-adjudicate-prove` still owns dispositions, facts, taint rules, and prove-budget. `three-stage-scan` still owns `quick` / `standard` / `deep` and the sandbox. This spec only supplies the structured parse those stages already asked for. + +## Requirements + +### Requirement 1: Core install and quick stay extra-free + +**Objective:** As a developer or CI job, I want `quick` inventory with no optional packages, so that a default install never pulls parser wheels. + +#### Acceptance Criteria + +1. When OpenUltraSAST runs a `quick` scan, the system shall emit inventory proposals without requiring the semantic extra. +2. The system shall not require the semantic extra to be installed in order to import or run the default OpenUltraSAST command. +3. If the semantic extra is absent, then a `quick` scan shall still complete and shall not abort because a parser package is missing. + +### Requirement 2: Extra enables structured parse on in-scope languages + +**Objective:** As an AppSec engineer with the semantic extra installed, I want overlay to read JavaScript, C/C++, Java, and Python as structured source, so that those files can be promoted or demoted instead of left unsupported. + +#### Acceptance Criteria + +1. Where the semantic extra is installed and a grammar for the file’s language is available, when OpenUltraSAST runs a `standard` or `deep` scan, the system shall use that structured parse as the primary intra-file input to overlay adjudication for Python, JavaScript, C, C++, and Java. +2. When a JavaScript file contains `eval` of a request query (or the language-equivalent untrusted source to a known sink) and the extra provides a JavaScript grammar, the system shall be able to emit a `promote` overlay disposition for that proposal. +3. When a C or Java file contains a sink fed only by a constant or a dominating sanitizer (for example a literal format string to `printf`) and the extra provides a grammar for that language, the system shall `demote` that proposal rather than leave it `language_unsupported`. +4. The system shall feed the structured parse into the existing overlay dispositions (`promote`, `demote`, `unadjudicated`, `coverage`) and shall not introduce a new disposition. + +### Requirement 3: Missing extra or grammar fails closed + +**Objective:** As an operator without the extra or without a grammar for one language, I want honest `unadjudicated` outcomes, so that silence is not reported as safety. + +#### Acceptance Criteria + +1. If the semantic extra is not installed, then when OpenUltraSAST adjudicates a JavaScript, C, C++, or Java file, the system shall leave matching inventory proposals `unadjudicated` with `language_unsupported` or `parse_failed`. +2. If the extra is installed but no grammar is available for that file’s language, then the system shall leave those proposals `unadjudicated` with `language_unsupported` or `parse_failed` and shall not crash the scan. +3. When the extra is absent, the system shall still try a stdlib parse only for Python and shall otherwise follow the fail-closed rule in criterion 1. +4. If a file cannot be parsed even with a grammar present (syntax error or incomplete source), then the system shall leave matching proposals `unadjudicated` with `parse_failed`. +5. The system shall not treat a missing extra as a failed `standard` scan by itself. + +### Requirement 4: One parse path, no hand-rolled language visitors + +**Objective:** As a maintainer, I want a single structured-parse path for all in-scope languages, so that we do not grow a new visitor per language. + +#### Acceptance Criteria + +1. When the extra supplies grammars for more than one in-scope language, the system shall extract overlay parse facts (assignments, calls, constants) through one shared structured-parse path rather than a separate visitor per language. +2. The system shall not add a new language-specific stdlib or regex AST as the product parse engine for JavaScript, C, C++, or Java. +3. Where a host parser CLI is present but the semantic extra is not, the system shall not claim a successful structured parse for overlay unless the extra’s grammar actually produced one. + +### Requirement 5: Overlay and stages stay in their existing contracts + +**Objective:** As an operator, I want the extra to change who can be adjudicated, not what a disposition or stage means, so that prove-budget and fail-on rules stay honest. + +#### Acceptance Criteria + +1. The system shall not use a language model as the parse or taint engine. +2. The system shall not require Joern for the semantic extra to succeed. +3. When overlay promotes or demotes from a structured parse, the system shall keep that result at `static_corroboration` or below. +4. The system shall not select sandbox candidates from demoted or unadjudicated findings solely because a structured parse ran. +5. The system shall not fold overlay Youden or extra-install status into the stage-1 detection gate (`python -m openultrasast.gate`). + +### Requirement 6: Extra present and extra absent are both testable + +**Objective:** As a maintainer, I want CI to prove both the extra-free core and the extra-enabled overlay, so that default install and AppSec install cannot silently diverge. + +#### Acceptance Criteria + +1. When the test suite runs without the semantic extra, the system shall still pass the extra-free `quick` and fail-closed overlay checks in Requirements 1 and 3. +2. Where the semantic extra is installed in a test environment, the system shall exercise Requirement 2 for at least one JavaScript file and at least one C or Java file. +3. If a test environment lacks the extra, then the suite shall skip extra-enabled checks rather than fail the extra-free core. diff --git a/.kiro/specs/tree-sitter-overlay-extra/research.md b/.kiro/specs/tree-sitter-overlay-extra/research.md new file mode 100644 index 0000000..65e9586 --- /dev/null +++ b/.kiro/specs/tree-sitter-overlay-extra/research.md @@ -0,0 +1,171 @@ +# Gap Analysis: tree-sitter-overlay-extra + +Requirements are **generated but not approved**. Analysis proceeds to inform design; it does not treat requirements as locked. + +Steering `product.md` / `tech.md` / `structure.md` are absent. Context is the overlay package, `harness_ext` extra pattern, `pyproject.toml`, CI, and PyPI tree-sitter bindings. + +--- + +## 1. Current State + +### Assets that already exist + +| Asset | Role today | +|-------|------------| +| `src/openultrasast/semantic/ir.py` | `FileIR` / `Bind` / `CallSite`; Python `ast` extractor (~170 lines); `parse_tree_sitter_cli` **always returns `None`** | +| `src/openultrasast/semantic/engines.py` | Probe: env flag, `importlib.util.find_spec("tree_sitter")`, or `tree-sitter` CLI on PATH | +| `src/openultrasast/semantic/taint.py` | Language-agnostic taint on `FileIR` (names, constants, fact tables) | +| `src/openultrasast/semantic/overlay.py` | Dispositions; already consumes `parse_file` | +| `src/openultrasast/ruleset/semantic/*.toml` | JS/C/Java facts already written | +| `src/openultrasast/harness_ext.py` | Pattern for optional extra: `find_spec`, lazy import, mypy override, cold-import tests | +| `pyproject.toml` | `dependencies = []`; only extra is `harnessx` | +| `tests/test_semantic_overlay.py` | JS unsupported without probe; JS IR only via **monkeypatch**, not a real grammar | + +### Conventions + +- Optional native/heavy deps stay off the default install; capability probe must not import at module load (`harness_ext`, Docker sandbox probe). +- mypy `ignore_missing_imports` for extras (`harnessx.*`). +- pytest marker already exists for `docker`; extra-enabled tests would follow that skip pattern. +- CI `checks` job: `uv sync --group dev` then pytest — **no optional extras**. + +### Integration surfaces + +- `parse_file(path, text, language) -> FileIR` is the only seam taint/overlay need. +- `tree_sitter_available()` is a **presence** probe, not “grammar produced IR”. Host CLI on PATH makes it True even with no PyPI wheels (this environment has `/home/linuxbrew/.../tree-sitter` and **no languages**). +- Pair eval already falls back to inventory when a language never promotes/demotes. + +## 2. Requirements Feasibility + +| Req | Need | Gap | +|-----|------|-----| +| 1 Extra-free `quick` | Empty core deps; no import of parser wheels at load | **Constraint met.** Missing extra cannot currently break `quick` because there is no extra. Must keep it that way. | +| 2 Structured parse for Py/JS/C/C++/Java | Real CST → `FileIR`; JS promote; C/Java demote on constant/sanitizer | **Missing.** CLI parse never builds `FileIR`. No `[semantic]` extra. No CST walker. | +| 3 Fail closed | `language_unsupported` / `parse_failed`; Python ast last resort | **Mostly present** via `parse_file` fallback. Syntax-error Python already `parse_failed`. JS/C/Java without IR already `language_unsupported`. | +| 4 One shared extract path | Not a visitor per language; CLI without extra must not fake success | **Missing walker.** CLI stub already returns `None` (does not fake success). Probe can still report “available” without IR — **constraint** for Req 4.3 wording vs `tree_sitter_available()`. | +| 5 Overlay contracts | No LLM parse; Joern not required; corroboration; promotions-only sandbox; gate inventory-only | **Present** in overlay/prove/gate. Do not reopen. | +| 6 Dual test matrix | Extra-free always; extra-on JS + C/Java; skip if extra absent | **Partial.** Extra-free tests exist. Extra-on tests are mocked. No pytest marker. CI does not install a semantic extra. | + +**Complexity signal:** external native-wheel extra + one algorithmic CST walk + pytest/CI matrix. Not CRUD. + +**Research needed (design phase, not blockers):** + +1. Pin compatible `tree-sitter` + `tree-sitter-{python,javascript,c,cpp,java}` wheel ABIs (`Language(mod.language())` vs older `build_library`). PyPI: `tree-sitter` 0.26 (2026-06-30), `tree-sitter-javascript` 0.25. +2. Per-language CST node names for assignment, call, identifier, string, function (JS `assignment_expression` vs C `assignment_expression` vs Java `assignment_expression` vs Python `assignment`). Shared walker needs a small node-kind table, not four visitors. +3. C++: separate `tree-sitter-cpp` vs C grammar; `preprocess` maps `.cpp` to `cpp`. +4. Whether default CI should add a second job `uv sync --extra semantic` or only skip-marked tests on the extra-free job. + +## 3. Implementation Approach Options + +### Option A: Extend `ir.py` / `engines.py` only + +Put extra pins in `pyproject.toml`, implement API parse + CST walk inside `ir.py`, tighten `tree_sitter_available()`. + +- **Pros:** Few files; `parse_file` stays the seam. +- **Cons:** `ir.py` is already ~277 lines (Python ast + dead CLI). A walker + language table will bloat it and mix packaging probe with extract. + +### Option B: New components only + +New `semantic/cst.py` + `semantic/extra.py`; leave `ir.py` as Python ast; overlay calls a new entrypoint. + +- **Pros:** Clean extra guard. +- **Cons:** Two parse entrypoints; easy to fork `parse_file` consumers. + +### Option C: Hybrid (recommended for design) + +- **New:** `semantic/cst.py` — one walker: tree-sitter `Node` → `Bind` / `CallSite` / `FunctionIR` using a node-kind map. +- **New or thin:** extra guard modeled on `harness_ext` (`has_semantic_extra`, lazy grammar load, never import wheels at module load). +- **Extend:** `pyproject.toml` `[project.optional-dependencies] semantic = [...]`; mypy override `tree_sitter*`; pytest marker `semantic`; CI extra-free job unchanged; optional second job or skip. +- **Extend:** `parse_file` prefers extra CST when a grammar loads; else Python ast; else `language_unsupported`. **Delete or stop calling** `parse_tree_sitter_cli` as a success path (Req 4.3). +- **Do not extend:** `taint.py`, `overlay.py`, facts, gate, pair catalog. + +**Trade-offs:** Extra files, but `ir.py` shrinks (drop dead CLI), taint stays language-agnostic, extra follows an existing packaging seam. + +## 4. Effort and Risk + +- **Effort: M (3–7 days).** Extra pins + walker + dual tests. Overlay/taint reuse. Not a new adjudicator. +- **Risk: Medium.** Native wheels and grammar ABI pins; CST node-name drift across languages; CI without extra must stay green. Mitigate with skip markers and a version pin table in design. + +## 5. Design-phase recommendations + +- Prefer **Option C**. +- Treat `FileIR` as frozen contract; do not add dispositions. +- Decide probe semantics: `tree_sitter_available` should mean “can produce FileIR for this language”, not “CLI binary exists”. +- Carry research items 1–4 into design (ABI pins, node-kind map, cpp package, CI job vs skip). +- Do not pull Joern, pair harvest, or CWE-121 into this spec. + +--- + +## Summary (design discovery) + +- **Feature**: tree-sitter-overlay-extra +- **Discovery Scope**: Extension of `parse_file` / overlay IR; optional extra packaging +- **Key Findings**: + - Official PyPI wheels: `tree-sitter` (0.26), `tree-sitter-python`, `tree-sitter-javascript` (0.25), `tree-sitter-c` (0.24.x), `tree-sitter-cpp` (0.23.4), `tree-sitter-java`. API is `Parser(Language(mod.language()))`; `build_library` is gone. + - `FileIR` is the only overlay seam; taint does not need language visitors. + - Host CLI presence must not count as a successful overlay parse (Req 4.3). + +## Research Log + +### PyPI ABI and load API + +- **Context**: Pin wheels that work on Python 3.11+ without compiling grammars in CI. +- **Sources Consulted**: https://pypi.org/project/tree-sitter/ ; https://pypi.org/project/tree-sitter-javascript/ ; https://pypi.org/project/tree-sitter-cpp/ ; py-tree-sitter 0.23 release notes (Language capsule, Parser constructor). +- **Findings**: Grammars ship abi3 wheels. Load with `Language(tree_sitter_.language())` then `Parser(language)`. Do not use `Language.build_library`. +- **Implications**: Extra lists core + five grammar packages. mypy ignores `tree_sitter*`. + +### Probe vs grammar + +- **Context**: `tree_sitter_available()` is true when CLI is on PATH with zero grammars. +- **Findings**: CLI parse cannot populate `FileIR` without a walker. Extra Python API is the product parse. +- **Implications**: Split `has_semantic_extra()` from per-language `grammar_for`. Remove CLI as a success path. + +## Architecture Pattern Evaluation + +| Option | Description | Strengths | Risks | Notes | +|--------|-------------|-----------|-------|-------| +| A Extend ir.py only | Walker + extra inside ir.py | Few files | Bloats 277-line module | Rejected | +| B New entrypoint | Overlay calls cst.py directly | Clean extra | Forks parse_file | Rejected | +| C Hybrid | Extra guard + cst walker; parse_file dispatch | Matches harness_ext; FileIR frozen | Node-kind table | **Selected** | + +## Design Decisions + +### Decision: Adopt official grammar wheels, build one walker + +- **Context**: Avoid per-language AST visitors (Req 4). +- **Alternatives**: tree-sitter CLI sexp; py-tree-sitter-languages (unmaintained); hand-rolled JS/C parsers. +- **Selected**: `openultrasast[semantic]` = `tree-sitter` plus python/javascript/c/cpp/java grammar packages. One CST walk keyed by a node-kind map. +- **Rationale**: Wheels are MIT, maintained, match in-scope languages. Taint stays language-agnostic. +- **Trade-offs**: Native wheels in the extra; ABI pins. Core install stays empty. +- **Follow-up**: Verify Parser constructor against the pinned `tree-sitter` minor at implement time. + +### Decision: Extra-free CI stays default; extra-on is a second job plus skip marker + +- **Context**: Req 6 dual matrix. +- **Selected**: pytest `semantic` marker skips when extra absent. GitHub Actions adds a job `uv sync --extra semantic --group dev` that runs marked tests. Default `checks` job unchanged. +- **Rationale**: Default CI must not fail without wheels. Extra-on tests must actually run somewhere. +- **Trade-offs**: Two pytest invocations in CI. + +### Decision: has_error implies parse_failed + +- **Context**: tree-sitter is error-tolerant; Req 3.4 wants fail-closed on syntax errors. +- **Selected**: If `tree.root_node.has_error` (or ERROR children), return `FileIR(parse_ok=False, reason=parse_failed)`. +- **Rationale**: Partial IR would invite false demote. + +## Synthesis + +- **Generalization**: One `parse_file` dispatch; extra is a grammar source, not a second overlay. +- **Build vs adopt**: Adopt tree-sitter wheels; build only the FileIR projection (binds/calls). +- **Simplification**: No CLI sexp walker; no second parse API; no taint/overlay changes. + +## Risks & Mitigations + +- Grammar ABI mismatch — pin extras together; extra-on CI job catches import/parse failure. +- Node-type drift — table in cst.py; tests assert JS eval and C printf on real trees. +- CLI still on PATH — must not set parse_ok. + +## References + +- https://pypi.org/project/tree-sitter/ +- https://github.com/tree-sitter/py-tree-sitter/releases +- `src/openultrasast/harness_ext.py` extra guard pattern + diff --git a/.kiro/specs/tree-sitter-overlay-extra/spec.json b/.kiro/specs/tree-sitter-overlay-extra/spec.json new file mode 100644 index 0000000..fe3f2c5 --- /dev/null +++ b/.kiro/specs/tree-sitter-overlay-extra/spec.json @@ -0,0 +1,22 @@ +{ + "feature_name": "tree-sitter-overlay-extra", + "created_at": "2026-09-04T14:00:00Z", + "updated_at": "2026-09-04T15:00:00Z", + "language": "en", + "phase": "tasks-generated", + "approvals": { + "requirements": { + "generated": true, + "approved": true + }, + "design": { + "generated": true, + "approved": true + }, + "tasks": { + "generated": true, + "approved": true + } + }, + "ready_for_implementation": true +} diff --git a/.kiro/specs/tree-sitter-overlay-extra/tasks.md b/.kiro/specs/tree-sitter-overlay-extra/tasks.md new file mode 100644 index 0000000..e944d9d --- /dev/null +++ b/.kiro/specs/tree-sitter-overlay-extra/tasks.md @@ -0,0 +1,93 @@ +# Implementation Plan + +- [x] 1. Foundation: optional extra and capability probe +- [x] 1.1 Declare the semantic extra without adding core dependencies + - Core install dependencies stay empty. + - Optional extra named semantic lists the parser runtime and grammars for Python, JavaScript, C, C++, and Java. + - Typecheck ignores the extra modules the same way it ignores the existing optional agentic extra. + - Pytest registers a semantic marker for extra-on tests. + - Observable: a default sync still has an empty core dependency list; the extra is listed only as optional. + - _Requirements: 1.1, 1.2, 6.3_ + - _Boundary: SemanticExtra_ + +- [x] 1.2 Probe the extra and load a grammar without importing wheels at package load + - Capability check uses importability, not a host CLI binary. + - Grammar load is lazy per language and returns nothing when the extra or that grammar is missing. + - Importing the overlay package or CLI does not import parser wheels. + - Host CLI on PATH does not make a grammar available. + - Observable: with the extra absent, the probe is false and a JavaScript language request yields no grammar; importing the CLI in a fresh interpreter does not load tree_sitter. + - _Requirements: 1.2, 1.3, 3.1, 3.2, 4.3_ + - _Boundary: SemanticExtra_ + - _Depends: 1.1_ + +- [x] 1.3 Stop treating a host parser CLI as overlay-ready + - Overlay-ready means a grammar can be loaded for that language, not that a CLI exists. + - Existing env override for tests can still force the probe off. + - Observable: a machine with a parser CLI and no extra still reports no overlay grammar for JavaScript. + - _Requirements: 4.3, 3.1_ + - _Boundary: SemanticExtra_ + - _Depends: 1.2_ + +- [x] 2. Shared structured parse into existing overlay IR +- [x] 2.1 Walk one syntax tree into assignments, calls, and functions + - One walker serves Python, JavaScript, C, C++, and Java via a node-kind table, not a visitor per language. + - Successful trees set engine to tree-sitter and fill the existing IR records overlay taint already consumes. + - No new language-specific stdlib or regex AST is added for JavaScript, C, C++, or Java. + - Observable: with a JavaScript grammar loaded, `eval(req.query.x)` produces a call record for eval with the query expression as an argument. + - _Requirements: 2.1, 4.1, 4.2_ + - _Boundary: CstWalker_ + - _Depends: 1.2_ + +- [x] 2.2 Treat error trees as parse failure + - If the tree reports an error, the IR is not parse_ok and the reason is parse_failed. + - A syntax-broken file does not yield a fake clean tree for overlay. + - Observable: incomplete JavaScript source returns parse_failed rather than a promote or demote. + - _Requirements: 3.4_ + - _Boundary: CstWalker_ + - _Depends: 2.1_ + +- [x] 3. Integration: parse dispatch +- [x] 3.1 Prefer extra grammar, then Python stdlib, then unsupported + - Parse order is grammar walk, else Python stdlib parse for Python only, else language_unsupported. + - Host CLI output is never used as a successful overlay parse. + - Missing extra does not fail a standard scan by itself. + - Overlay still emits only promote, demote, unadjudicated, and coverage. + - Observable: extra-free JavaScript inventory is unadjudicated language_unsupported; extra-free Python `eval(request.args)` still promotes via stdlib parse; extra-on JavaScript uses engine tree-sitter. + - _Requirements: 2.1, 2.4, 3.3, 3.5, 4.2, 4.3_ + - _Boundary: ParseDispatch_ + - _Depends: 1.2, 2.1, 2.2_ + +- [x] 4. Validation: extra-free core, extra-on overlay, CI matrix +- [x] 4.1 Prove extra-free quick and fail-closed overlay + - Quick scan completes without the extra and still emits inventory. + - JavaScript, C, and Java stay unadjudicated language_unsupported or parse_failed without the extra. + - Python stdlib parse remains the last resort when the extra is absent. + - Observable: extra-free pytest covering quick plus JS unsupported is green; detection gate still ignores overlay Youden. + - _Requirements: 1.1, 1.3, 3.1, 3.3, 5.5, 6.1_ + - _Boundary: ExtraTestMatrix_ + - _Depends: 3.1_ + +- [x] 4.2 (P) Prove extra-on JavaScript promote and C or Java demote + - Marked tests skip when the extra is absent instead of failing the extra-free suite. + - With the extra, JavaScript eval of request query promotes. + - With the extra, a C or Java sink fed only by a constant or dominating sanitizer demotes rather than language_unsupported. + - Evidence stays at static_corroboration or below; sandbox selection is still promotions only. + - Observable: extra-on tests pass when wheels are installed and skip otherwise; JS promote and C or Java demote assertions fail if the walker is removed. + - _Requirements: 2.2, 2.3, 3.4, 5.3, 5.4, 6.2, 6.3_ + - _Boundary: ExtraTestMatrix_ + - _Depends: 3.1_ + +- [x] 4.3 Run extra-on checks in CI without breaking extra-free CI + - Default CI job stays extra-free and must remain green. + - A second job installs the semantic extra and runs the semantic-marked tests. + - Joern is not required; no language model is used as parse or taint. + - Observable: extra-free job does not install parser wheels; extra-on job executes the marked JS and C or Java tests. + - _Requirements: 5.1, 5.2, 6.1, 6.2, 6.3_ + - _Boundary: ExtraTestMatrix_ + - _Depends: 4.1, 4.2_ + +## Implementation Notes + +- Extra wheels: tree-sitter 0.26 with grammar packages python/javascript/c/cpp/java. `OPENULTRASAST_TREE_SITTER_PROBE=0` simulates extra-absent even when wheels are installed locally. +- Host `tree-sitter` CLI is not overlay-ready; only lazy `grammar_for` produces FileIR. +- Default CI stays extra-free; job `semantic` runs `uv sync --extra semantic` then `pytest -m semantic`. diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md new file mode 100644 index 0000000..ce657d9 --- /dev/null +++ b/.kiro/steering/roadmap.md @@ -0,0 +1,47 @@ +# Roadmap + +## Overview + +OpenUltraSAST can propose sinks with regex and prove a few of them in a sandbox, but adjudication is still Python-`ast` plus a tree-sitter CLI stub. Textbook SAST pairs (OWASP/Juliet) did not improve after the overlay: Youden went from **+9.09% inventory to 0% overlay** on the same 11-pair slice. Real operator trees and browser/C/C++ history are not in the catalog. The next work is to put a real tree-sitter extra under the existing overlay, then grow an offline function-level vuln-vs-fix slice from public VFC datasets (including Firefox/gecko-dev where commits exist) without vendoring whole megarepos. + +## Approach Decision + +- **Chosen**: Optional tree-sitter extra first, then a harvested VFC slice scored like `github` (honesty dashboard, not smoke gate). Isolated functions only. Firefox is one project filter among Chromium, OpenSSL, curl — not a special-case full mozilla-central clone. +- **Why**: The overlay already has dispositions, facts, and taint. JS/C/Java stay `language_unsupported` because we refused a parser library. Textbook Youden is the wrong north star; large-project VFCs are the right one. Adopting tree-sitter deletes the need to hand-write language parsers. +- **Rejected alternatives**: + - Whole Firefox tree in CI — multi-GB, memory-safety heavy, security-bug commit messages are deliberately scrubbed. + - FixFox dataset — embargoed until 2030. + - Joern-as-required or inter-file taint in this phase — deferred by `propose-adjudicate-prove`. + - Chasing OWASP Youden with more regex — leaks on goodG2B/false files. + +## Scope + +- **In**: tree-sitter + grammars as `openultrasast[semantic]`; CST walker into existing FileIR/taint; harvested function-level pairs from CVEfixes/MoreFixes/CWE-Bench-Java/SVEN with provenance; overlay scoring on that slice; keep `quick` zero-dep. +- **Out**: Cloning mozilla-central/Chromium into the repo; making overlay Youden a merge gate; buffer-size/CWE-121 semantics; inter-file taint; auto-facts from improve; LLM as parser. + +## Constraints + +- Core `dependencies = []`. Parser wheels live only in an extra. +- Pair fixtures stay out of `LANGUAGE_MANIFESTS`. +- Do not vendor 16 GB VFC dumps. Pointers in `datasets.toml`; a small reviewed catalog of isolated functions is what CI runs. +- DiverseVul-style auto labels (~60% accurate) are not ground truth without review. + +## Boundary Strategy + +- **Why this split**: Engine (how we parse) is independent of corpus (what we score). The extra can ship without any new pairs; the corpus can score inventory-only until the extra lands, then overlay. +- **Shared seams to watch**: `FileIR` / `parse_file`; pair catalog `slice` values; overlay vs inventory scoring in `pairs.py`. + +## Existing Spec Updates + +- [x] propose-adjudicate-prove -- overlay roles, facts, prove-budget, sast overlay scoring (landed). Residual: tree-sitter path returns no IR. +- [ ] propose-adjudicate-prove -- do not reopen for parsers or corpora; those are the new specs below. + +## Direct Implementation Candidates + +- [ ] Record inventory vs overlay Youden side-by-side in `ousast pairs --slice sast` JSON (small report field, not a new spec). +- [ ] Add Firefox/gecko-dev, Chromium, OpenSSL pointers to `benchmarks/pairs/datasets.toml`. + +## Specs (dependency order) + +- [ ] tree-sitter-overlay-extra -- Optional extra: tree-sitter + grammars, one CST walker, `quick` stays zero-dep. Dependencies: none +- [ ] real-world-vfc-slice -- Harvest reviewed function-level vuln/fix pairs from large projects (Firefox/gecko-dev, Chromium, OpenSSL, …) as an honesty slice. Dependencies: none diff --git a/pyproject.toml b/pyproject.toml index 402cdfb..9cf9c28 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -20,6 +20,16 @@ dependencies = [] harnessx = [ "harnessx @ git+https://github.com/Darwin-Agent/HarnessX.git@274c512eea67c64701f992d3b6fe06f0fd590ef0", ] +# Structured parse for overlay (JS/C/C++/Java/Python). Core install stays empty. +# Wheels are MIT; import is lazy via openultrasast.semantic.extra. +semantic = [ + "tree-sitter>=0.23,<0.27", + "tree-sitter-python>=0.23", + "tree-sitter-javascript>=0.23", + "tree-sitter-c>=0.23", + "tree-sitter-cpp>=0.23", + "tree-sitter-java>=0.23", +] [project.scripts] ousast = "openultrasast.cli:main" @@ -27,7 +37,10 @@ ousast = "openultrasast.cli:main" [tool.pytest.ini_options] testpaths = ["tests"] pythonpath = ["src"] -markers = ["docker: requires a docker daemon"] +markers = [ + "docker: requires a docker daemon", + "semantic: requires openultrasast[semantic] tree-sitter extra", +] [tool.ruff] line-length = 140 @@ -55,6 +68,10 @@ warn_unused_configs = true module = "harnessx.*" ignore_missing_imports = true +[[tool.mypy.overrides]] +module = "tree_sitter" +ignore_missing_imports = true + [dependency-groups] dev = [ "dagger-io>=0.21.7", diff --git a/src/openultrasast/semantic/__init__.py b/src/openultrasast/semantic/__init__.py index 45e6a38..ae339f1 100644 --- a/src/openultrasast/semantic/__init__.py +++ b/src/openultrasast/semantic/__init__.py @@ -1,6 +1,7 @@ """Propose/adjudicate/prove overlay: regex proposes, semantics adjudicates.""" from .engines import joern_available, tree_sitter_available +from .extra import grammar_for, has_semantic_extra from .facts import FactLoadError, SemanticFacts, load_facts from .mechanisms import Mechanism, MechanismStore, append_mechanism, order_promotions from .overlay import OverlayError, OverlayRecord, adjudicate, finding_from_coverage, write_overlay @@ -17,6 +18,8 @@ __all__ = [ "append_mechanism", "filter_promoted_hotspots", "finding_from_coverage", + "grammar_for", + "has_semantic_extra", "joern_available", "load_facts", "order_promotions", diff --git a/src/openultrasast/semantic/cst.py b/src/openultrasast/semantic/cst.py new file mode 100644 index 0000000..04e64f9 --- /dev/null +++ b/src/openultrasast/semantic/cst.py @@ -0,0 +1,258 @@ +"""One CST walker: tree-sitter nodes into FileIR. No per-language visitors.""" + +from __future__ import annotations + +from typing import Any + +from .ir import Bind, CallSite, FileIR, FunctionIR + +_KIND = { + "python": { + "function": frozenset({"function_definition"}), + "assign": frozenset({"assignment", "augmented_assignment"}), + "call": frozenset({"call"}), + "args": frozenset({"argument_list", "arguments"}), + "string": frozenset({"string"}), + "params": frozenset({"parameters"}), + }, + "javascript": { + "function": frozenset({"function_declaration", "method_definition", "arrow_function", "function_expression"}), + "assign": frozenset({"assignment_expression", "augmented_assignment_expression"}), + "call": frozenset({"call_expression"}), + "args": frozenset({"arguments"}), + "string": frozenset({"string"}), + "params": frozenset({"formal_parameters"}), + }, + "c": { + "function": frozenset({"function_definition"}), + "assign": frozenset({"assignment_expression"}), + "call": frozenset({"call_expression"}), + "args": frozenset({"argument_list"}), + "string": frozenset({"string_literal"}), + "params": frozenset({"parameter_list"}), + }, + "java": { + "function": frozenset({"method_declaration", "constructor_declaration"}), + "assign": frozenset({"assignment_expression", "variable_declarator"}), + "call": frozenset({"method_invocation"}), + "args": frozenset({"argument_list"}), + "string": frozenset({"string_literal"}), + "params": frozenset({"formal_parameters"}), + }, +} +_KIND["cpp"] = _KIND["c"] +_KIND["typescript"] = _KIND["javascript"] + +_CONST = frozenset( + { + "string", + "string_literal", + "integer", + "number", + "float", + "true", + "false", + "null", + "nil", + "character_literal", + "escape_sequence", + } +) +_SEQUENCE = frozenset({"list", "tuple", "array", "array_creation_expression", "initializer_list"}) + + +def parse_with_cst(path: str, text: str, language: str) -> FileIR | None: + from .extra import grammar_for + + grammar = grammar_for(language) + if grammar is None: + return None + try: + from tree_sitter import Parser + + parser = Parser(grammar) + tree = parser.parse(text.encode()) + except Exception: + return FileIR(path=path, language=language, engine="tree-sitter", functions=(), parse_ok=False, reason="parse_failed") + return file_ir_from_tree(path=path, text=text, language=language, tree=tree) + + +def file_ir_from_tree(*, path: str, text: str, language: str, tree: Any) -> FileIR: + del text + root = tree.root_node + if bool(getattr(root, "has_error", False)): + return FileIR(path=path, language=language, engine="tree-sitter", functions=(), parse_ok=False, reason="parse_failed") + kinds = _KIND.get(language) + if kinds is None: + return FileIR(path=path, language=language, engine="none", functions=(), parse_ok=False, reason="language_unsupported") + functions: list[FunctionIR] = [] + module_binds, module_calls = _collect(root, kinds, skip=kinds["function"]) + functions.append( + FunctionIR( + name="", + params=(), + start_line=_line(root), + end_line=_end_line(root), + binds=tuple(module_binds), + calls=tuple(module_calls), + ) + ) + for node in _walk(root): + if node.type not in kinds["function"]: + continue + binds, calls = _collect(node, kinds, skip=kinds["function"]) + functions.append( + FunctionIR( + name=_function_name(node), + params=_params(node, kinds), + start_line=_line(node), + end_line=_end_line(node), + binds=tuple(binds), + calls=tuple(calls), + ) + ) + return FileIR(path=path, language=language, engine="tree-sitter", functions=tuple(functions), parse_ok=True) + + +def _collect(node: Any, kinds: dict[str, frozenset[str]], *, skip: frozenset[str]) -> tuple[list[Bind], list[CallSite]]: + binds: list[Bind] = [] + calls: list[CallSite] = [] + for child in _walk(node, skip=skip): + if child.type in kinds["assign"]: + bind = _bind(child, kinds) + if bind is not None: + binds.append(bind) + if child.type in kinds["call"]: + calls.append(_call(child, kinds)) + return binds, calls + + +def _walk(node: Any, *, skip: frozenset[str] | None = None) -> list[Any]: + found: list[Any] = [node] + for child in getattr(node, "named_children", ()) or (): + if skip is not None and child.type in skip: + continue + found.extend(_walk(child, skip=skip)) + return found + + +def _bind(node: Any, kinds: dict[str, frozenset[str]]) -> Bind | None: + children = list(getattr(node, "named_children", ()) or ()) + if not children: + return None + name_node = children[0] + value_node = children[-1] if len(children) > 1 else None + name = _left_name(name_node) + if not name: + return None + value_text = _text(value_node) if value_node is not None else "" + return Bind( + name=name, + line=_line(node), + value_text=value_text, + is_constant=_is_constant(value_node, kinds) if value_node is not None else False, + names=_identifiers(value_node) if value_node is not None else (), + call_name=_call_name(value_node, kinds) if value_node is not None and value_node.type in kinds["call"] else None, + ) + + +def _call(node: Any, kinds: dict[str, frozenset[str]]) -> CallSite: + args_node = next((child for child in getattr(node, "named_children", ()) or () if child.type in kinds["args"]), None) + arg_nodes = list(getattr(args_node, "named_children", ()) or ()) if args_node is not None else [] + extra = len(arg_nodes) >= 2 and arg_nodes[1].type in _SEQUENCE + return CallSite( + name=_callee_name(node, args_node), + line=_line(node), + arg_texts=tuple(_text(arg) for arg in arg_nodes), + arg_is_constant=tuple(_is_constant(arg, kinds) for arg in arg_nodes), + arg_names=tuple(_identifiers(arg) for arg in arg_nodes), + extra_arg_is_sequence=extra, + ) + + +def _callee_name(node: Any, args_node: Any) -> str: + idents = [ + _text(child) + for child in getattr(node, "named_children", ()) or () + if child is not args_node and child.type in {"identifier", "property_identifier"} + ] + if idents: + return idents[-1] + for child in getattr(node, "named_children", ()) or (): + if child is not args_node: + text = _text(child) + if text: + return text.split("(")[0] + return _text(node).split("(")[0] + + +def _call_name(node: Any, kinds: dict[str, frozenset[str]]) -> str | None: + if node.type in kinds["call"]: + args_node = next((child for child in getattr(node, "named_children", ()) or () if child.type in kinds["args"]), None) + return _callee_name(node, args_node) + return None + + +def _function_name(node: Any) -> str: + for child in getattr(node, "named_children", ()) or (): + if child.type == "identifier": + return _text(child) + if child.type == "function_declarator": + for grandchild in getattr(child, "named_children", ()) or (): + if grandchild.type == "identifier": + return _text(grandchild) + return "" + + +def _params(node: Any, kinds: dict[str, frozenset[str]]) -> tuple[str, ...]: + for child in _walk(node, skip=frozenset()): + if child.type in kinds["params"]: + return _identifiers(child) + if child.type == "function_declarator": + for grandchild in getattr(child, "named_children", ()) or (): + if grandchild.type in kinds["params"]: + return _identifiers(grandchild) + return () + + +def _left_name(node: Any) -> str: + if node.type in {"identifier", "property_identifier"}: + return _text(node) + idents = _identifiers(node) + return idents[0] if idents else "" + + +def _identifiers(node: Any) -> tuple[str, ...]: + names: list[str] = [] + for child in _walk(node): + if child.type == "identifier": + names.append(_text(child)) + return tuple(names) + + +def _is_constant(node: Any, kinds: dict[str, frozenset[str]]) -> bool: + if node.type in _CONST or node.type in kinds["string"]: + return True + if node.type not in {"binary_expression", "concatenated_string"}: + return False + children = list(getattr(node, "named_children", ()) or ()) + return bool(children) and all(_is_constant(child, kinds) for child in children) + + +def _text(node: Any) -> str: + raw = getattr(node, "text", None) + if raw is None: + return "" + if isinstance(raw, bytes): + return raw.decode(errors="ignore") + return str(raw) + + +def _line(node: Any) -> int: + point = getattr(node, "start_point", (0, 0)) + return int(point[0]) + 1 + + +def _end_line(node: Any) -> int: + point = getattr(node, "end_point", getattr(node, "start_point", (0, 0))) + return int(point[0]) + 1 diff --git a/src/openultrasast/semantic/engines.py b/src/openultrasast/semantic/engines.py index 8fa7ec8..8eccb6b 100644 --- a/src/openultrasast/semantic/engines.py +++ b/src/openultrasast/semantic/engines.py @@ -1,15 +1,15 @@ -"""Probe tree-sitter and Joern like the Docker sandbox probe. Never required for inventory.""" +"""Probe semantic extra and Joern. Host CLI alone is not overlay-ready.""" from __future__ import annotations -import importlib.util import os import shutil import subprocess from collections.abc import Callable from subprocess import CompletedProcess -TREE_SITTER_PROBE_ENV = "OPENULTRASAST_TREE_SITTER_PROBE" +from .extra import has_semantic_extra + JOERN_PROBE_ENV = "OPENULTRASAST_JOERN_PROBE" PROBE_TIMEOUT_SECONDS = 5 _PROBE_OFF = frozenset({"0", "off", "false", "unavailable", "no"}) @@ -17,22 +17,9 @@ _PROBE_ON = frozenset({"1", "on", "true", "available", "yes"}) def tree_sitter_available(*, runner: Callable[..., CompletedProcess[str]] | None = None) -> bool: - flag = os.environ.get(TREE_SITTER_PROBE_ENV, "").strip().lower() - if flag in _PROBE_OFF: - return False - if flag in _PROBE_ON: - return True - if _python_tree_sitter_present(): - return True - binary = shutil.which("tree-sitter") - if binary is None: - return False - run = runner if runner is not None else subprocess.run - try: - result = run([binary, "--version"], capture_output=True, text=True, timeout=PROBE_TIMEOUT_SECONDS, check=False) - return result.returncode == 0 - except Exception: - return False + """True when a grammar extra can produce overlay IR. Host CLI is not sufficient.""" + del runner + return has_semantic_extra() def joern_available(*, runner: Callable[..., CompletedProcess[str]] | None = None) -> bool: @@ -58,7 +45,3 @@ def joern_flow(path: str, text: str, language: str) -> tuple[object, ...]: if not joern_available(): return () return () - - -def _python_tree_sitter_present() -> bool: - return importlib.util.find_spec("tree_sitter") is not None diff --git a/src/openultrasast/semantic/extra.py b/src/openultrasast/semantic/extra.py new file mode 100644 index 0000000..30a3f12 --- /dev/null +++ b/src/openultrasast/semantic/extra.py @@ -0,0 +1,48 @@ +"""Optional semantic extra: lazy grammar load. Never import wheels at package load.""" + +from __future__ import annotations + +import importlib +import importlib.util +import os +from typing import Any + +_GRAMMAR_MODULES = { + "python": "tree_sitter_python", + "javascript": "tree_sitter_javascript", + "c": "tree_sitter_c", + "cpp": "tree_sitter_cpp", + "java": "tree_sitter_java", +} +_PROBE_ENV = "OPENULTRASAST_TREE_SITTER_PROBE" +_PROBE_OFF = frozenset({"0", "off", "false", "unavailable", "no"}) +_PROBE_ON = frozenset({"1", "on", "true", "available", "yes"}) + + +def has_semantic_extra() -> bool: + """True when the parser runtime and at least one in-scope grammar are importable.""" + flag = os.environ.get(_PROBE_ENV, "").strip().lower() + if flag in _PROBE_OFF: + return False + if importlib.util.find_spec("tree_sitter") is None: + return False + return any(importlib.util.find_spec(module) is not None for module in _GRAMMAR_MODULES.values()) + + +def grammar_for(language: str) -> Any | None: + """Return a tree_sitter.Language for an in-scope language, or None. Never raises into the scan.""" + flag = os.environ.get(_PROBE_ENV, "").strip().lower() + if flag in _PROBE_OFF: + return None + module_name = _GRAMMAR_MODULES.get(language) + if module_name is None or importlib.util.find_spec("tree_sitter") is None: + return None + if importlib.util.find_spec(module_name) is None: + return None + try: + from tree_sitter import Language + + module = importlib.import_module(module_name) + return Language(module.language()) + except Exception: + return None diff --git a/src/openultrasast/semantic/ir.py b/src/openultrasast/semantic/ir.py index 1a4358d..ffe30b3 100644 --- a/src/openultrasast/semantic/ir.py +++ b/src/openultrasast/semantic/ir.py @@ -3,22 +3,7 @@ from __future__ import annotations import ast -import shutil -import subprocess -import tempfile from dataclasses import dataclass -from pathlib import Path - -from .engines import tree_sitter_available - -_EXTENSION = { - "python": ".py", - "javascript": ".js", - "typescript": ".ts", - "c": ".c", - "cpp": ".cpp", - "java": ".java", -} @dataclass(frozen=True) @@ -62,10 +47,11 @@ class FileIR: def parse_file(path: str, text: str, language: str) -> FileIR: - if tree_sitter_available(): - parsed = parse_tree_sitter_cli(path, text, language) - if parsed is not None and parsed.parse_ok: - return parsed + from .cst import parse_with_cst + + parsed = parse_with_cst(path, text, language) + if parsed is not None: + return parsed if language == "python": return parse_python_ast(path, text) return FileIR(path=path, language=language, engine="none", functions=(), parse_ok=False, reason="language_unsupported") @@ -80,37 +66,6 @@ def parse_python_ast(path: str, text: str) -> FileIR: return FileIR(path=path, language="python", engine="python-ast", functions=tuple(functions), parse_ok=True) -def parse_tree_sitter_cli(path: str, text: str, language: str) -> FileIR | None: - binary = shutil.which("tree-sitter") - if binary is None: - return None - suffix = _EXTENSION.get(language, ".txt") - tmp_path: str | None = None - try: - with tempfile.NamedTemporaryFile(suffix=suffix, delete=False) as handle: - handle.write(text.encode()) - tmp_path = handle.name - result = subprocess.run( - [binary, "parse", tmp_path], - capture_output=True, - text=True, - timeout=5, - check=False, - ) - except Exception: - return None - finally: - if tmp_path: - Path(tmp_path).unlink(missing_ok=True) - combined = f"{result.stdout or ''}{result.stderr or ''}" - if result.returncode != 0 or "No language found" in combined or "Failed to load language" in combined: - return None - # A successful parse without an extractor still cannot adjudicate JS/C/Java. - # Return a parse_ok IR only when we actually built functions; otherwise None - # so Python can fall back to ast and other languages stay unsupported. - return None - - def _python_functions(tree: ast.AST) -> list[FunctionIR]: functions: list[FunctionIR] = [] module_binds, module_calls = _statements(getattr(tree, "body", [])) diff --git a/tests/test_semantic_cst.py b/tests/test_semantic_cst.py new file mode 100644 index 0000000..7c9e4f1 --- /dev/null +++ b/tests/test_semantic_cst.py @@ -0,0 +1,70 @@ +"""Extra-on CST walker and overlay. Skips when wheels are absent.""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from openultrasast.findings import quick_scan_findings +from openultrasast.preprocess import preprocess_repository +from openultrasast.rank import rank_targets +from openultrasast.semantic import adjudicate, has_semantic_extra +from openultrasast.semantic.ir import parse_file + +pytestmark = pytest.mark.semantic + + +def _require_extra() -> None: + if not has_semantic_extra(): + pytest.skip("requires openultrasast[semantic]") + + +def test_javascript_eval_query_is_tree_sitter_call() -> None: + _require_extra() + ir = parse_file("app.js", "eval(req.query.x)\n", "javascript") + assert ir.parse_ok is True + assert ir.engine == "tree-sitter" + calls = [call for function in ir.functions for call in function.calls] + eval_calls = [call for call in calls if call.name == "eval"] + assert eval_calls + assert any("req.query" in arg for arg in eval_calls[0].arg_texts) + + +def test_broken_javascript_is_parse_failed() -> None: + _require_extra() + ir = parse_file("app.js", "eval(req.query.x\n", "javascript") + assert ir.parse_ok is False + assert ir.reason == "parse_failed" + + +def test_javascript_eval_promotes(tmp_path: Path) -> None: + _require_extra() + root = tmp_path / "repo" + root.mkdir() + (root / "app.js").write_text("eval(req.query.x)\n") + _, targets = preprocess_repository(root) + findings = quick_scan_findings(root, targets, rank_targets(targets)) + records = adjudicate(root=root, targets=targets, findings=findings) + eval_records = [record for record in records if "js-eval" in record.proposal_id] + assert eval_records + assert eval_records[0].disposition == "promote" + assert eval_records[0].evidence_level == "static_corroboration" + assert eval_records[0].engine == "tree-sitter" + + +def test_c_constant_system_demotes(tmp_path: Path) -> None: + _require_extra() + root = tmp_path / "repo" + root.mkdir() + (root / "run.c").write_text('void f(void) { system("ls"); }\n') + _, targets = preprocess_repository(root) + findings = quick_scan_findings(root, targets, rank_targets(targets)) + records = adjudicate(root=root, targets=targets, findings=findings) + hits = [record for record in records if record.path == "run.c"] + assert hits + assert all(record.disposition != "unadjudicated" or record.reason != "language_unsupported" for record in hits) + demoted = [record for record in hits if record.disposition == "demote"] + assert demoted + assert demoted[0].dominating_fact == "constant" + assert demoted[0].evidence_level == "static_corroboration" diff --git a/tests/test_semantic_extra.py b/tests/test_semantic_extra.py new file mode 100644 index 0000000..102de50 --- /dev/null +++ b/tests/test_semantic_extra.py @@ -0,0 +1,71 @@ +"""Semantic extra packaging and extra-free overlay parse.""" + +from __future__ import annotations + +import json +import tomllib +from pathlib import Path + +import pytest + +from openultrasast.cli import main +from openultrasast.semantic.extra import grammar_for, has_semantic_extra +from openultrasast.semantic.ir import parse_file + + +def test_core_dependencies_stay_empty_and_semantic_is_optional() -> None: + payload = tomllib.loads(Path("pyproject.toml").read_text()) + assert payload["project"]["dependencies"] == [] + extra = payload["project"]["optional-dependencies"]["semantic"] + joined = " ".join(extra) + assert "tree-sitter" in joined + assert "tree-sitter-javascript" in joined + assert "tree-sitter-c" in joined + assert "tree-sitter-java" in joined + assert "semantic:" in " ".join(payload["tool"]["pytest"]["ini_options"]["markers"]) + + +def test_cli_import_does_not_load_tree_sitter(assert_cold_of_harnessx) -> None: # type: ignore[no-untyped-def] + proc_code = ( + "import sys\n" + "import openultrasast.cli\n" + "leaked = sorted(m for m in sys.modules if m == 'tree_sitter' or m.startswith('tree_sitter_'))\n" + "assert not leaked, leaked\n" + ) + assert_cold_of_harnessx(proc_code) + + +def test_probe_off_yields_no_javascript_grammar(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setenv("OPENULTRASAST_TREE_SITTER_PROBE", "0") + assert has_semantic_extra() is False + assert grammar_for("javascript") is None + ir = parse_file("app.js", "eval(req.query.x)\n", "javascript") + assert ir.parse_ok is False + assert ir.reason == "language_unsupported" + + +def test_standard_javascript_is_unadjudicated_without_extra(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setenv("OPENULTRASAST_TREE_SITTER_PROBE", "0") + monkeypatch.setenv("OPENULTRASAST_RUNS_DIR", ".runs") + repo = tmp_path / "repo" + repo.mkdir() + (repo / "app.js").write_text("eval(req.query.x)\n") + assert main(["scan", str(repo), "--mode", "standard"]) == 0 + run_dir = sorted((repo / ".runs").iterdir())[-1] + overlay = json.loads((run_dir / "overlay.json").read_text())["records"] + assert overlay + assert all(item["disposition"] == "unadjudicated" for item in overlay) + assert all(item["reason"] in {"language_unsupported", "parse_failed"} for item in overlay) + + +def test_quick_scan_without_extra_still_inventories(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setenv("OPENULTRASAST_TREE_SITTER_PROBE", "0") + monkeypatch.setenv("OPENULTRASAST_RUNS_DIR", ".runs") + repo = tmp_path / "repo" + repo.mkdir() + (repo / "app.py").write_text("def run():\n return eval(request.args)\n") + assert main(["scan", str(repo), "--mode", "quick"]) == 0 + run_dir = sorted((repo / ".runs").iterdir())[-1] + findings = json.loads((run_dir / "findings.json").read_text())["findings"] + assert findings + assert not (run_dir / "overlay.json").exists() diff --git a/tests/test_semantic_overlay.py b/tests/test_semantic_overlay.py index 960c9e4..8501104 100644 --- a/tests/test_semantic_overlay.py +++ b/tests/test_semantic_overlay.py @@ -13,7 +13,7 @@ from openultrasast.rank import rank_targets from openultrasast.semantic import OverlayError, OverlayRecord, adjudicate, load_facts from openultrasast.semantic.engines import joern_available, tree_sitter_available from openultrasast.semantic.facts import FactLoadError -from openultrasast.semantic.ir import FileIR, FunctionIR, parse_file +from openultrasast.semantic.ir import parse_file from openultrasast.semantic.taint import TaintPath, analyze_file @@ -186,29 +186,13 @@ def test_overlay_length_matches_proposals_plus_coverage_only(tmp_path: Path) -> assert all(record.evidence_level == "static_corroboration" for record in records if record.disposition == "coverage") -def test_js_language_unsupported_without_tree_sitter(monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setattr("openultrasast.semantic.ir.tree_sitter_available", lambda: False) - monkeypatch.setattr("openultrasast.semantic.engines.tree_sitter_available", lambda: False) +def test_js_language_unsupported_without_semantic_extra(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setenv("OPENULTRASAST_TREE_SITTER_PROBE", "0") ir = parse_file("app.js", "eval(req.query.x)\n", "javascript") assert ir.parse_ok is False assert ir.reason == "language_unsupported" -def test_js_yields_ir_when_tree_sitter_available(monkeypatch: pytest.MonkeyPatch) -> None: - fake = FileIR( - path="app.js", - language="javascript", - engine="tree-sitter", - functions=(FunctionIR(name="run", params=("x",), start_line=1, end_line=1, binds=(), calls=()),), - parse_ok=True, - ) - monkeypatch.setattr("openultrasast.semantic.ir.tree_sitter_available", lambda: True) - monkeypatch.setattr("openultrasast.semantic.ir.parse_tree_sitter_cli", lambda *args, **kwargs: fake) - ir = parse_file("app.js", "eval(req.query.x)\n", "javascript") - assert ir.parse_ok is True - assert ir.engine == "tree-sitter" - - def test_joern_absence_does_not_block_overlay(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setattr("openultrasast.semantic.overlay.joern_available", lambda: False) root = tmp_path / "repo" diff --git a/uv.lock b/uv.lock index b2a30bc..fb3a8b8 100644 --- a/uv.lock +++ b/uv.lock @@ -2145,6 +2145,14 @@ source = { editable = "." } harnessx = [ { name = "harnessx" }, ] +semantic = [ + { name = "tree-sitter" }, + { name = "tree-sitter-c" }, + { name = "tree-sitter-cpp" }, + { name = "tree-sitter-java" }, + { name = "tree-sitter-javascript" }, + { name = "tree-sitter-python" }, +] [package.dev-dependencies] dev = [ @@ -2156,8 +2164,16 @@ dev = [ ] [package.metadata] -requires-dist = [{ name = "harnessx", marker = "extra == 'harnessx'", git = "https://github.com/Darwin-Agent/HarnessX.git?rev=274c512eea67c64701f992d3b6fe06f0fd590ef0" }] -provides-extras = ["harnessx"] +requires-dist = [ + { name = "harnessx", marker = "extra == 'harnessx'", git = "https://github.com/Darwin-Agent/HarnessX.git?rev=274c512eea67c64701f992d3b6fe06f0fd590ef0" }, + { name = "tree-sitter", marker = "extra == 'semantic'", specifier = ">=0.23,<0.27" }, + { name = "tree-sitter-c", marker = "extra == 'semantic'", specifier = ">=0.23" }, + { name = "tree-sitter-cpp", marker = "extra == 'semantic'", specifier = ">=0.23" }, + { name = "tree-sitter-java", marker = "extra == 'semantic'", specifier = ">=0.23" }, + { name = "tree-sitter-javascript", marker = "extra == 'semantic'", specifier = ">=0.23" }, + { name = "tree-sitter-python", marker = "extra == 'semantic'", specifier = ">=0.23" }, +] +provides-extras = ["harnessx", "semantic"] [package.metadata.requires-dev] dev = [ @@ -3335,6 +3351,124 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/d8/8e/bb97bb0c71802080bfc8952937d174e49cfc50de5c951dd47b2496f0dcdb/tqdm-4.68.3-py3-none-any.whl", hash = "sha256:39832cc2def2789a6f29df83f172db7416cea70052c0907a57801c5f2fdccb03", size = 78337, upload-time = "2026-06-17T07:36:50.132Z" }, ] +[[package]] +name = "tree-sitter" +version = "0.26.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f7/03/5600b84aff2e6c4fe80cfebb4063fe2f50299521befe5f6092ab8c082f4a/tree_sitter-0.26.0.tar.gz", hash = "sha256:b40c219edccc4564530c96f8f1556f6202b37cda964d1cbd7bd2b7e68b40a245", size = 191423, upload-time = "2026-06-30T12:14:27.933Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/41/18/78aae7e4b5a36daaebb0276e4b07d084d45298758000787838e89329e11f/tree_sitter-0.26.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:1d6fe0e8fb4df77b5ee816228e2c4475a63d8cc1d4d3a7ffd7097b2b87fc3e95", size = 148679, upload-time = "2026-06-30T12:13:52.27Z" }, + { url = "https://files.pythonhosted.org/packages/24/e4/b371b9553b0e47d130fc2073e56cab94fecc868be04666bf5bbd1fcd1cc9/tree_sitter-0.26.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:514a9bf8993e5210e7970736aaf6020d1759b670e195ef17b1c48f586aa30736", size = 140759, upload-time = "2026-06-30T12:13:53.221Z" }, + { url = "https://files.pythonhosted.org/packages/22/7d/266fb0f2c41e6fb00b0f40e7a3338cdf99651e6a6511ca72bc78fc697636/tree_sitter-0.26.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:10f0d4eb94aa7242dcb7f554bcd24dd7ba1c114f00d58759ba08c7a46c8ec51a", size = 637206, upload-time = "2026-06-30T12:13:54.334Z" }, + { url = "https://files.pythonhosted.org/packages/40/9f/47cf22febb47132d5b3a507a27bb99ef89fe5c8ec420a13c6daa9b64f782/tree_sitter-0.26.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:335294ce0504fcefde5245dff596778ffaf820205b98ae0b549c72e48855f1d8", size = 664758, upload-time = "2026-06-30T12:13:55.42Z" }, + { url = "https://files.pythonhosted.org/packages/4c/4d/8d144ca3beb46a62a5102b6deac76bb0da55235c2c7840faf3b12f2e9d97/tree_sitter-0.26.0-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:f9997ba61368c48ed54e715676afadf703947a1542464e39d047764fb3624b01", size = 647438, upload-time = "2026-06-30T12:13:56.523Z" }, + { url = "https://files.pythonhosted.org/packages/4d/ed/ed1d6e78520c4fb64ed52fec3f2947bf8c1fbad7bc24e282c56193c9ba42/tree_sitter-0.26.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:c56581ad256c4195a21bfe449fed5d44a02fe83a4a7d6e70e6ec302c881191c7", size = 661944, upload-time = "2026-06-30T12:13:57.82Z" }, + { url = "https://files.pythonhosted.org/packages/10/83/45f5bd43db1b8248d2fd08ef6cbe43e2725c539e09a2cfb8bc2818646788/tree_sitter-0.26.0-cp311-cp311-win_amd64.whl", hash = "sha256:0f8793fd18ad7eec276ed4b51c097b4bf2002b357259b66b0d75db1f3f41c754", size = 129496, upload-time = "2026-06-30T12:13:59.216Z" }, + { url = "https://files.pythonhosted.org/packages/f1/8d/be68e6c04563eb54145424cc83fe0aa8b0ba6c90d8989cf8a032671b5f16/tree_sitter-0.26.0-cp311-cp311-win_arm64.whl", hash = "sha256:dea4b4e27d49e9ec5b785d4f994da000e6726882fcc6ad05ec98478500c71aef", size = 116484, upload-time = "2026-06-30T12:14:00.147Z" }, + { url = "https://files.pythonhosted.org/packages/87/ca/565702c44815393e3a973552ad546db4e5ca081ca8698640b4e93d809f51/tree_sitter-0.26.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:6cb2bd20efb2544c19ac54486ab7cb8ec7b36f913bbe1ce95df84acb96743d9c", size = 148934, upload-time = "2026-06-30T12:14:01.188Z" }, + { url = "https://files.pythonhosted.org/packages/54/6f/8bb61957f16ec1b1d92410a006cdc84a952b6352a7313b2ad299f2d21484/tree_sitter-0.26.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:918d89529786873f0982a0f59c2a303cd065fbfd1b903d71a8e4e1584f67b42e", size = 140820, upload-time = "2026-06-30T12:14:02.087Z" }, + { url = "https://files.pythonhosted.org/packages/78/0a/8a6f08559182643a814a4ab559948ae817b2851890fd9b995a4fff6541ce/tree_sitter-0.26.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:30a88be89ff1f2755297f81e8080d88b795dd98720c3f9fa2acf93873182cc95", size = 638844, upload-time = "2026-06-30T12:14:03.428Z" }, + { url = "https://files.pythonhosted.org/packages/8a/2f/6e6781b31677231366cb3cf27bc8269157f6d4b03c9032865a4f5f2bbe7e/tree_sitter-0.26.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5a6b333b0282d8bb0af741f9b018bd2523d4eecb2686bf6717066a625fecfaa4", size = 667487, upload-time = "2026-06-30T12:14:04.669Z" }, + { url = "https://files.pythonhosted.org/packages/02/0b/0483078c8567445557a7015b0e5b187f6d7d4fda73464df9c4bdea7f7f3c/tree_sitter-0.26.0-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:3f3c44339dd34fe8eb2b8d5aa7610660499a795f70376b130bbee7a437337280", size = 647975, upload-time = "2026-06-30T12:14:05.797Z" }, + { url = "https://files.pythonhosted.org/packages/27/68/da83ca72c984e96ab4eb3bee0db1a6ffb5de1c8c455f92bd9f420cde7f0e/tree_sitter-0.26.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:94550e13b6ae576969da40246f4c4abb206380b5375ad43f26dd9151d55438e3", size = 665018, upload-time = "2026-06-30T12:14:07.278Z" }, + { url = "https://files.pythonhosted.org/packages/d1/36/4d67927fd47b89af4a00f65f55a7370e28778cd50e972c2430487e3ecc27/tree_sitter-0.26.0-cp312-cp312-win_amd64.whl", hash = "sha256:ca89e361a276dbc934b28a43dd881199e25d34ff5493ee0ce45f3c52a6124a37", size = 129619, upload-time = "2026-06-30T12:14:08.373Z" }, + { url = "https://files.pythonhosted.org/packages/ed/72/cdefad523eb78710679c6da6a79e3d90f5afd32b1c6aa5a17bac7eef99f6/tree_sitter-0.26.0-cp312-cp312-win_arm64.whl", hash = "sha256:bc6cb01d5ee75c85424aa1f1c72a82d8f07fd52539a0f3c4a6ed3e8721079b84", size = 116545, upload-time = "2026-06-30T12:14:09.273Z" }, + { url = "https://files.pythonhosted.org/packages/cb/b0/465257cf8f972ad9f9812ec1cbaa8ec210ebebb601ade9a15881aa2436b4/tree_sitter-0.26.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:ed0889dbed843ce45ede9f5169c0b2dea2222f12685844a03fadb81f12705867", size = 148893, upload-time = "2026-06-30T12:14:10.541Z" }, + { url = "https://files.pythonhosted.org/packages/a1/ec/19d093e854b45e807fecfdd26105c266f43aeecc39c4dc97992a7074ad5a/tree_sitter-0.26.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:6189c6c340c7384357711e3d92645e96bfb79f7a502f86de1ebdb23eb43f7dab", size = 140829, upload-time = "2026-06-30T12:14:11.626Z" }, + { url = "https://files.pythonhosted.org/packages/9b/ee/87e74671ed63a837e7a1f17ab94aa3913871e033b27523d8e7b83d6f7ad0/tree_sitter-0.26.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8ff2e0750b7daa722302838356d7b65e303829b7eb73c915df127ddba115e1d1", size = 639334, upload-time = "2026-06-30T12:14:12.836Z" }, + { url = "https://files.pythonhosted.org/packages/66/e7/f7e04cd9dff6b6ac0adf23922796fbc76accd4cf4bcda50542748d485679/tree_sitter-0.26.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7075ef857ef86f327dbb72d1e2574dda78db5754b3a1fca6506acd7fe5d561a7", size = 668102, upload-time = "2026-06-30T12:14:14.035Z" }, + { url = "https://files.pythonhosted.org/packages/d3/90/0bfb16b7894fea728c774a89d5af421a9368a2f913bbd4e8dcab7caaecfb/tree_sitter-0.26.0-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:26c996c1edfee86e977bb3f5462e74fcec0d0b0db1e85a3c475875763caa03be", size = 648560, upload-time = "2026-06-30T12:14:15.302Z" }, + { url = "https://files.pythonhosted.org/packages/cd/e6/0fe05ba396e9623b0ae40ccf34171336b8701ec8d7bd0ee9f5224d638665/tree_sitter-0.26.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:00289bfe7978f3e0dc0ce69813a20fa9f44ea4c100b3ec62043e5eb74ccfc3a2", size = 665121, upload-time = "2026-06-30T12:14:16.403Z" }, + { url = "https://files.pythonhosted.org/packages/eb/d2/a944b1ca35bed6068dc84a9967aaf3049d8cc0b7a36179eea8787270a6ab/tree_sitter-0.26.0-cp313-cp313-win_amd64.whl", hash = "sha256:93e220cab7e6a823efeb2046c49171427de92ef71c7c681c01820d14d8d3721f", size = 129615, upload-time = "2026-06-30T12:14:17.463Z" }, + { url = "https://files.pythonhosted.org/packages/09/ef/c7ca48293580d2249f36940c4eed5b4ddeb9ce75baf9a4ef30621987e0c7/tree_sitter-0.26.0-cp313-cp313-win_arm64.whl", hash = "sha256:b31a8195d2f224224c530ac814632d98c1dcc123d227442c07c736e86b70d564", size = 116525, upload-time = "2026-06-30T12:14:18.53Z" }, + { url = "https://files.pythonhosted.org/packages/c5/7a/4d84e6f6ae2c3e757490dd84de251712c31e293dfe31f28da1ec019cefa2/tree_sitter-0.26.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:5a3c93a352b7e6f70f73e121bbfa2d0117ba7478bd51114ed35c91b0b78814fa", size = 148901, upload-time = "2026-06-30T12:14:19.452Z" }, + { url = "https://files.pythonhosted.org/packages/b0/d9/efe62ec65dc9d096e834d27b8c058127e2146e42ff3380b822a233f016a6/tree_sitter-0.26.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:5fc2f41bf246ff2f70a9cc3690be35ec7580a4923151873d898c8bcb1a4503d3", size = 140805, upload-time = "2026-06-30T12:14:20.478Z" }, + { url = "https://files.pythonhosted.org/packages/c4/2c/c82326b7b97e3c485c18679883b16f89e5e913c639d3b219d3da70c9e67e/tree_sitter-0.26.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b8ea92a255c91671a7ec4625aba3ab7bb5220c423630ffbf83c45d7312abe084", size = 640586, upload-time = "2026-06-30T12:14:21.527Z" }, + { url = "https://files.pythonhosted.org/packages/e2/7a/f56e7d8282859452611024c7cbc623bfba5b24b8cb9b8f8bc88c5219fe9a/tree_sitter-0.26.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f665510f0fcf4636fb9696f1f7853bed7a3bd764b7bb0cb8494e619c14ed5a0c", size = 668300, upload-time = "2026-06-30T12:14:22.728Z" }, + { url = "https://files.pythonhosted.org/packages/91/51/240ee81b9d5e9ca0a6cb1528e8605ffa70ab58c89ce126631be96d3e4bae/tree_sitter-0.26.0-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:253df7ab82cc0a9d311cd65f06e9f99fb3eac55996ae9fc94da22f123a861b90", size = 649627, upload-time = "2026-06-30T12:14:23.819Z" }, + { url = "https://files.pythonhosted.org/packages/6a/54/760035cefedf9eb44f0f84c4ac22f1322e73155853e272576ee876336312/tree_sitter-0.26.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:ff80d4833d330a73184a3ac5132abe93c575d2dea31975c6f15c0d21fef238aa", size = 664885, upload-time = "2026-06-30T12:14:25.064Z" }, + { url = "https://files.pythonhosted.org/packages/c9/1b/0b36fe2a984ecedc4ce6aefd5d56447a6626a8e9b595c4e48658510ce8f8/tree_sitter-0.26.0-cp314-cp314-win_amd64.whl", hash = "sha256:a4033fecc8f606c7f2e8b8014d0057b74668a7f0152763606f7bc25c5f9ec64c", size = 132688, upload-time = "2026-06-30T12:14:26.106Z" }, + { url = "https://files.pythonhosted.org/packages/4d/74/ebc041a13fbf40144afdb0d4b447e48e0b4012ca866c63de8b48f801f0c1/tree_sitter-0.26.0-cp314-cp314-win_arm64.whl", hash = "sha256:823251c4b6725a7c03ed497a339135ede7ae4bdde75bb8be7ef5e305aeb4ff52", size = 120287, upload-time = "2026-06-30T12:14:26.991Z" }, +] + +[[package]] +name = "tree-sitter-c" +version = "0.24.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/a6/c9/3834f3d9278251aea7312274971bc4c45b17aec2490fd4b884d93bd7019a/tree_sitter_c-0.24.2.tar.gz", hash = "sha256:1628584df0299b5a340aa63f8e67b6c97c91517f52fa7e7a4c557e40adb330a9", size = 228397, upload-time = "2026-04-22T08:06:14.491Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/28/c1/26ed17730ec2c17bedc1b673349e5e0a466c578e3eb0327c3b73cf52bf97/tree_sitter_c-0.24.2-cp310-abi3-macosx_10_9_x86_64.whl", hash = "sha256:4d4579a8b54f0a442f903d88d3304cab77cd5c2031d4015baa4f2f8e15d6dcb7", size = 81016, upload-time = "2026-04-22T08:06:07.208Z" }, + { url = "https://files.pythonhosted.org/packages/c1/1c/1140db75e7e375cda3c68792a33826c4fd40b5b98c3259d93c75f6c8368f/tree_sitter_c-0.24.2-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:97bc80a224d48215d4e6e6376bf30d114f4c317b8145ff1b02afe785d4ba7bdd", size = 86213, upload-time = "2026-04-22T08:06:08.136Z" }, + { url = "https://files.pythonhosted.org/packages/e9/8c/0dfb88d726f8821d1c4c36042f092be974a800afd734307a595b8604190c/tree_sitter_c-0.24.2-cp310-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:5041ef67eb68ce6bc8bb0b1f8ef3a5585ce523dae0c7eec109ab0627dd75aede", size = 94264, upload-time = "2026-04-22T08:06:08.918Z" }, + { url = "https://files.pythonhosted.org/packages/87/78/47dc570e7aee6b0a1ecc2520b30639cc2b06003154c9ab0672d86bf720d5/tree_sitter_c-0.24.2-cp310-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c098bedcd5ac86ff93fa734d51d1dd86aed40fd5ed7d634c7af11380a0469969", size = 94560, upload-time = "2026-04-22T08:06:09.852Z" }, + { url = "https://files.pythonhosted.org/packages/29/37/75d59d3f74f4cfc00f04472917e933d8a9c9fdc6eff980ef9552e010e6aa/tree_sitter_c-0.24.2-cp310-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:82842c5a5f2acd93f4de10038c33ac179c8979defc39376f990348d6289e933b", size = 94023, upload-time = "2026-04-22T08:06:10.682Z" }, + { url = "https://files.pythonhosted.org/packages/64/57/8fc655d5a446a70a637e92b98bd2fdaab88bf5bb5b36076ac4add544808d/tree_sitter_c-0.24.2-cp310-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:e2b42e8e22202c251f8629306f9321233542e07a6e01611b5fe83489272143eb", size = 94160, upload-time = "2026-04-22T08:06:11.497Z" }, + { url = "https://files.pythonhosted.org/packages/c1/f7/72a1d6b42dd31fd37e03ff67e7dc5ee572301499e6b216002b8dd42a1714/tree_sitter_c-0.24.2-cp310-abi3-win_amd64.whl", hash = "sha256:abb549225091f7b25df2dd3a0143ece6e208f7055d8bcb4700b41ee79b9ef1e1", size = 84669, upload-time = "2026-04-22T08:06:12.347Z" }, + { url = "https://files.pythonhosted.org/packages/e2/9d/7475d9ae8ef679aa36c7dfe6c903ab78e573651c68b6ef9862d6a3f994db/tree_sitter_c-0.24.2-cp310-abi3-win_arm64.whl", hash = "sha256:4a2f4371cd816cc3153458f69062135ebb2ea5f275ddd90494e5c823d778204a", size = 82956, upload-time = "2026-04-22T08:06:13.364Z" }, +] + +[[package]] +name = "tree-sitter-cpp" +version = "0.23.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/20/2c/4dd63d705a8933543cad9b92ff31be849b164fec91a6eb63475ebc9ce668/tree_sitter_cpp-0.23.4.tar.gz", hash = "sha256:6a59c4cebb1ad1dc2e8d586cf8a72b39d21b8108b7b139d089719e81a339e41d", size = 940358, upload-time = "2024-11-11T06:59:24.934Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b6/ac/11d56670f7b048362db872ca866fd00ba2002a322ab179f047b7c0fb2910/tree_sitter_cpp-0.23.4-cp39-abi3-macosx_10_9_x86_64.whl", hash = "sha256:aacb1759f0efd9dbc25bd8ee88184a340483018869f75412d9c3bc32c039a520", size = 287861, upload-time = "2024-11-11T06:59:15.005Z" }, + { url = "https://files.pythonhosted.org/packages/12/1c/0337c016bdc00a77a3326d12f10ee836401dd28f27db6fd5b7734bfb21ed/tree_sitter_cpp-0.23.4-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:bc3c404d9f0cbd87951213a85440afbf4c31e718f8d907fa9ee12bea4b8d276f", size = 315513, upload-time = "2024-11-11T06:59:16.679Z" }, + { url = "https://files.pythonhosted.org/packages/b3/7b/dd38c049b10ed7fda118b903a1d28a8b55a36b98c30606ef90e8f374c6de/tree_sitter_cpp-0.23.4-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ccc43ddf1279d5d5a4ef190373f4cb16522801bec4492bcd4754edf2aeba2b7b", size = 334813, upload-time = "2024-11-11T06:59:18.253Z" }, + { url = "https://files.pythonhosted.org/packages/6a/4d/23e390234d2acd351f5563b1079c515d7c1fe13ddb7392cee543be74dda3/tree_sitter_cpp-0.23.4-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:773d2cafc08bbc0f998687fa33f42f378c1a371cdb582870c4d13abb06092706", size = 316110, upload-time = "2024-11-11T06:59:19.823Z" }, + { url = "https://files.pythonhosted.org/packages/32/c7/b94a7e0e803af9d3bd4608fb4f0cfb2e9e233abaf0a38c928bfb0b1a025d/tree_sitter_cpp-0.23.4-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:247d127f0eb6574b0f6b30c0151e0bd0774e2e7acf9c558bdf9fbb8adc2e80c0", size = 308242, upload-time = "2024-11-11T06:59:21.466Z" }, + { url = "https://files.pythonhosted.org/packages/37/7e/909e52b3dec09c475140b0e175511e275d0d00ba2dbd7c68102d377ae0f6/tree_sitter_cpp-0.23.4-cp39-abi3-win_amd64.whl", hash = "sha256:68606a45bea92669d155399e1239f771a7767d8683cd8f8e30e7d813107030ca", size = 290997, upload-time = "2024-11-11T06:59:22.432Z" }, + { url = "https://files.pythonhosted.org/packages/d4/6a/65435d4d1f4c735be7ffe52d7c2e7b8a7f7c2790343a2719c60c548611c8/tree_sitter_cpp-0.23.4-cp39-abi3-win_arm64.whl", hash = "sha256:712f84f18be94cbe2a148fa4fdf40fcf4a8c25a8f7670efb9f8a47ddec2fc281", size = 288203, upload-time = "2024-11-11T06:59:23.404Z" }, +] + +[[package]] +name = "tree-sitter-java" +version = "0.23.5" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/fa/dc/eb9c8f96304e5d8ae1663126d89967a622a80937ad2909903569ccb7ec8f/tree_sitter_java-0.23.5.tar.gz", hash = "sha256:f5cd57b8f1270a7f0438878750d02ccc79421d45cca65ff284f1527e9ef02e38", size = 138121, upload-time = "2024-12-21T18:24:26.936Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/67/21/b3399780b440e1567a11d384d0ebb1aea9b642d0d98becf30fa55c0e3a3b/tree_sitter_java-0.23.5-cp39-abi3-macosx_10_9_x86_64.whl", hash = "sha256:355ce0308672d6f7013ec913dee4a0613666f4cda9044a7824240d17f38209df", size = 58926, upload-time = "2024-12-21T18:24:12.53Z" }, + { url = "https://files.pythonhosted.org/packages/57/ef/6406b444e2a93bc72a04e802f4107e9ecf04b8de4a5528830726d210599c/tree_sitter_java-0.23.5-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:24acd59c4720dedad80d548fe4237e43ef2b7a4e94c8549b0ca6e4c4d7bf6e69", size = 62288, upload-time = "2024-12-21T18:24:14.634Z" }, + { url = "https://files.pythonhosted.org/packages/4e/6c/74b1c150d4f69c291ab0b78d5dd1b59712559bbe7e7daf6d8466d483463f/tree_sitter_java-0.23.5-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:9401e7271f0b333df39fc8a8336a0caf1b891d9a2b89ddee99fae66b794fc5b7", size = 85533, upload-time = "2024-12-21T18:24:16.695Z" }, + { url = "https://files.pythonhosted.org/packages/29/09/e0d08f5c212062fd046db35c1015a2621c2631bc8b4aae5740d7adb276ad/tree_sitter_java-0.23.5-cp39-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:370b204b9500b847f6d0c5ad584045831cee69e9a3e4d878535d39e4a7e4c4f1", size = 84033, upload-time = "2024-12-21T18:24:18.758Z" }, + { url = "https://files.pythonhosted.org/packages/43/56/7d06b23ddd09bde816a131aa504ee11a1bbe87c6b62ab9b2ed23849a3382/tree_sitter_java-0.23.5-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:aae84449e330363b55b14a2af0585e4e0dae75eb64ea509b7e5b0e1de536846a", size = 82564, upload-time = "2024-12-21T18:24:20.493Z" }, + { url = "https://files.pythonhosted.org/packages/da/d6/0528c7e1e88a18221dbd8ccee3825bf274b1fa300f745fd74eb343878043/tree_sitter_java-0.23.5-cp39-abi3-win_amd64.whl", hash = "sha256:1ee45e790f8d31d416bc84a09dac2e2c6bc343e89b8a2e1d550513498eedfde7", size = 60650, upload-time = "2024-12-21T18:24:22.902Z" }, + { url = "https://files.pythonhosted.org/packages/72/57/5bab54d23179350356515526fff3cc0f3ac23bfbc1a1d518a15978d4880e/tree_sitter_java-0.23.5-cp39-abi3-win_arm64.whl", hash = "sha256:402efe136104c5603b429dc26c7e75ae14faaca54cfd319ecc41c8f2534750f4", size = 59059, upload-time = "2024-12-21T18:24:24.934Z" }, +] + +[[package]] +name = "tree-sitter-javascript" +version = "0.25.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/59/e0/e63103c72a9d3dfd89a31e02e660263ad84b7438e5f44ee82e443e65bbde/tree_sitter_javascript-0.25.0.tar.gz", hash = "sha256:329b5414874f0588a98f1c291f1b28138286617aa907746ffe55adfdcf963f38", size = 132338, upload-time = "2025-09-01T07:13:44.792Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2c/df/5106ac250cd03661ebc3cc75da6b3d9f6800a3606393a0122eca58038104/tree_sitter_javascript-0.25.0-cp310-abi3-macosx_10_9_x86_64.whl", hash = "sha256:b70f887fb269d6e58c349d683f59fa647140c410cfe2bee44a883b20ec92e3dc", size = 64052, upload-time = "2025-09-01T07:13:36.865Z" }, + { url = "https://files.pythonhosted.org/packages/b1/8f/6b4b2bc90d8ab3955856ce852cc9d1e82c81d7ab9646385f0e75ffd5b5d3/tree_sitter_javascript-0.25.0-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:8264a996b8845cfce06965152a013b5d9cbb7d199bc3503e12b5682e62bb1de1", size = 66440, upload-time = "2025-09-01T07:13:37.962Z" }, + { url = "https://files.pythonhosted.org/packages/5f/c4/7da74ecdcd8a398f88bd003a87c65403b5fe0e958cdd43fbd5fd4a398fcf/tree_sitter_javascript-0.25.0-cp310-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:9dc04ba91fc8583344e57c1f1ed5b2c97ecaaf47480011b92fbeab8dda96db75", size = 99728, upload-time = "2025-09-01T07:13:38.755Z" }, + { url = "https://files.pythonhosted.org/packages/96/c8/97da3af4796495e46421e9344738addb3602fa6426ea695be3fcbadbee37/tree_sitter_javascript-0.25.0-cp310-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:199d09985190852e0912da2b8d26c932159be314bc04952cf917ed0e4c633e6b", size = 106072, upload-time = "2025-09-01T07:13:39.798Z" }, + { url = "https://files.pythonhosted.org/packages/13/be/c964e8130be08cc9bd6627d845f0e4460945b158429d39510953bbcb8fcc/tree_sitter_javascript-0.25.0-cp310-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:dfcf789064c58dc13c0a4edb550acacfc6f0f280577f1e7a00de3e89fc7f8ddc", size = 104388, upload-time = "2025-09-01T07:13:40.866Z" }, + { url = "https://files.pythonhosted.org/packages/ee/89/9b773dee0f8961d1bb8d7baf0a204ab587618df19897c1ef260916f318ec/tree_sitter_javascript-0.25.0-cp310-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:1b852d3aee8a36186dbcc32c798b11b4869f9b5041743b63b65c2ef793db7a54", size = 98377, upload-time = "2025-09-01T07:13:41.838Z" }, + { url = "https://files.pythonhosted.org/packages/3b/dc/d90cb1790f8cec9b4878d278ad9faf7c8f893189ce0f855304fd704fc274/tree_sitter_javascript-0.25.0-cp310-abi3-win_amd64.whl", hash = "sha256:e5ed840f5bd4a3f0272e441d19429b26eedc257abe5574c8546da6b556865e3c", size = 62975, upload-time = "2025-09-01T07:13:42.828Z" }, + { url = "https://files.pythonhosted.org/packages/2e/1f/f9eba1038b7d4394410f3c0a6ec2122b590cd7acb03f196e52fa57ebbe72/tree_sitter_javascript-0.25.0-cp310-abi3-win_arm64.whl", hash = "sha256:622a69d677aa7f6ee2931d8c77c981a33f0ebb6d275aa9d43d3397c879a9bb0b", size = 61668, upload-time = "2025-09-01T07:13:43.803Z" }, +] + +[[package]] +name = "tree-sitter-python" +version = "0.25.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/b8/8b/c992ff0e768cb6768d5c96234579bf8842b3a633db641455d86dd30d5dac/tree_sitter_python-0.25.0.tar.gz", hash = "sha256:b13e090f725f5b9c86aa455a268553c65cadf325471ad5b65cd29cac8a1a68ac", size = 159845, upload-time = "2025-09-11T06:47:58.159Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/cf/64/a4e503c78a4eb3ac46d8e72a29c1b1237fa85238d8e972b063e0751f5a94/tree_sitter_python-0.25.0-cp310-abi3-macosx_10_9_x86_64.whl", hash = "sha256:14a79a47ddef72f987d5a2c122d148a812169d7484ff5c75a3db9609d419f361", size = 73790, upload-time = "2025-09-11T06:47:47.652Z" }, + { url = "https://files.pythonhosted.org/packages/e6/1d/60d8c2a0cc63d6ec4ba4e99ce61b802d2e39ef9db799bdf2a8f932a6cd4b/tree_sitter_python-0.25.0-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:480c21dbd995b7fe44813e741d71fed10ba695e7caab627fb034e3828469d762", size = 76691, upload-time = "2025-09-11T06:47:49.038Z" }, + { url = "https://files.pythonhosted.org/packages/aa/cb/d9b0b67d037922d60cbe0359e0c86457c2da721bc714381a63e2c8e35eba/tree_sitter_python-0.25.0-cp310-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:86f118e5eecad616ecdb81d171a36dde9bef5a0b21ed71ea9c3e390813c3baf5", size = 108133, upload-time = "2025-09-11T06:47:50.499Z" }, + { url = "https://files.pythonhosted.org/packages/40/bd/bf4787f57e6b2860f3f1c8c62f045b39fb32d6bac4b53d7a9e66de968440/tree_sitter_python-0.25.0-cp310-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:be71650ca2b93b6e9649e5d65c6811aad87a7614c8c1003246b303f6b150f61b", size = 110603, upload-time = "2025-09-11T06:47:51.985Z" }, + { url = "https://files.pythonhosted.org/packages/5d/25/feff09f5c2f32484fbce15db8b49455c7572346ce61a699a41972dea7318/tree_sitter_python-0.25.0-cp310-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:e6d5b5799628cc0f24691ab2a172a8e676f668fe90dc60468bee14084a35c16d", size = 108998, upload-time = "2025-09-11T06:47:53.046Z" }, + { url = "https://files.pythonhosted.org/packages/75/69/4946da3d6c0df316ccb938316ce007fb565d08f89d02d854f2d308f0309f/tree_sitter_python-0.25.0-cp310-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:71959832fc5d9642e52c11f2f7d79ae520b461e63334927e93ca46cd61cd9683", size = 107268, upload-time = "2025-09-11T06:47:54.388Z" }, + { url = "https://files.pythonhosted.org/packages/ed/a2/996fc2dfa1076dc460d3e2f3c75974ea4b8f02f6bc925383aaae519920e8/tree_sitter_python-0.25.0-cp310-abi3-win_amd64.whl", hash = "sha256:9bcde33f18792de54ee579b00e1b4fe186b7926825444766f849bf7181793a76", size = 76073, upload-time = "2025-09-11T06:47:55.773Z" }, + { url = "https://files.pythonhosted.org/packages/07/19/4b5569d9b1ebebb5907d11554a96ef3fa09364a30fcfabeff587495b512f/tree_sitter_python-0.25.0-cp310-abi3-win_arm64.whl", hash = "sha256:0fbf6a3774ad7e89ee891851204c2e2c47e12b63a5edbe2e9156997731c128bb", size = 74169, upload-time = "2025-09-11T06:47:56.747Z" }, +] + [[package]] name = "typer" version = "0.25.1"