mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
feat(tree-sitter-overlay-extra): parse JS/C/Java overlay via optional extra
Add openultrasast[semantic] with official grammar wheels and one CST walker into FileIR. quick stays extra-free. Host CLI is not overlay-ready. Default CI stays extra-free; a semantic job runs pytest -m semantic.
This commit is contained in:
Marius Ciepluch
@@ -32,6 +32,19 @@ jobs:
|
||||
run: uv run python -m openultrasast.pair_gate
|
||||
- run: uv run python -m compileall src tests
|
||||
|
||||
semantic:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
- uses: astral-sh/setup-uv@v7
|
||||
with:
|
||||
enable-cache: true
|
||||
- uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version-file: .python-version
|
||||
- run: uv sync --group dev --extra semantic
|
||||
- run: uv run pytest -m semantic
|
||||
|
||||
dagger:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 20
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
# Brief: tree-sitter-overlay-extra
|
||||
|
||||
## Problem
|
||||
|
||||
AppSec scanning mixed working trees gets `language_unsupported` on JS/C/Java, and Python-only `ast` cannot follow the patterns those languages actually use. The overlay spec promised tree-sitter as the primary engine; the CLI probe never builds `FileIR`. Maintaining hand-rolled parsers would explode the codebase.
|
||||
|
||||
## Current State
|
||||
|
||||
`parse_tree_sitter_cli` returns `None` even on a successful parse. Python falls back to stdlib `ast`. JS/C/Java stay unadjudicated. `quick` is correctly zero-dep. SAST overlay Youden on the 11-pair textbook slice is 0% vs +9% inventory — not because regex got worse, but because Python overlay fail-closes on ConfigParser-style heap and C/Java never parse.
|
||||
|
||||
## Desired Outcome
|
||||
|
||||
With the extra installed, tree-sitter grammars produce `FileIR` for Python, JavaScript, C/C++, and Java. Existing taint/facts/dispositions consume that IR unchanged. Without the extra, behavior stays fail-closed unadjudicated. `quick` still has no parser wheels.
|
||||
|
||||
## Approach
|
||||
|
||||
Add optional extra `openultrasast[semantic]` (`tree-sitter` + official language packages). One CST walker extracts binds and calls into the current `FileIR`. Delete the dead CLI sexp stub. Do not add a fourth language-specific AST visitor.
|
||||
|
||||
## Scope
|
||||
|
||||
- **In**: extra, probe that prefers the Python API when importable, CST → FileIR, tests with and without the extra.
|
||||
- **Out**: Joern queries, inter-file taint, CWE-121 buffer semantics, new pair corpora (owned by `real-world-vfc-slice`).
|
||||
|
||||
## Boundary Candidates
|
||||
|
||||
- Packaging extra vs core install
|
||||
- CST walker vs taint engine (taint stays in `semantic/taint.py`)
|
||||
|
||||
## Out of Boundary
|
||||
|
||||
- Pattern rules, CWE policy, sandbox, mechanism JSONL schema, pair catalog harvest
|
||||
|
||||
## Upstream / Downstream
|
||||
|
||||
- **Upstream**: `propose-adjudicate-prove` overlay records and taint
|
||||
- **Downstream**: `real-world-vfc-slice` overlay scoring on C/JS/Java pairs
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
|
||||
- **Extends**: none (new spec). Does not reopen `propose-adjudicate-prove` tasks.
|
||||
- **Adjacent**: `three-stage-scan` modes; `quick` must remain zero-dep.
|
||||
|
||||
## Constraints
|
||||
|
||||
No new required dependency. Missing extra → `language_unsupported` or Python ast, never crash. No LLM parse.
|
||||
@@ -0,0 +1,272 @@
|
||||
# Design Document
|
||||
|
||||
## Overview
|
||||
|
||||
**Purpose.** This feature turns on a real structured parse for overlay adjudication by installing an optional extra. AppSec scans of JavaScript, C/C++, Java, and Python then get `FileIR` from official tree-sitter grammars instead of `language_unsupported`.
|
||||
|
||||
**Users.** Default/CI installs keep `quick` and extra-free pytest. Operators who install `openultrasast[semantic]` get overlay promote/demote on the in-scope languages.
|
||||
|
||||
**Impact.** `parse_file` prefers extra grammars when they load. The dead CLI parse-success path is removed. Taint, facts, dispositions, sandbox, and the detection gate do not change.
|
||||
|
||||
### Goals
|
||||
|
||||
- Optional extra with official grammar wheels; core `dependencies` stay empty.
|
||||
- One CST walker into existing `FileIR` (`Bind`, `CallSite`, `FunctionIR`).
|
||||
- Fail closed: missing extra, missing grammar, or error tree → `unadjudicated`.
|
||||
- Dual test matrix: extra-free always; extra-on JS plus C or Java.
|
||||
|
||||
### Non-Goals
|
||||
|
||||
- Joern queries, inter-file taint, CWE-121 buffer semantics, pair-catalog harvest, LLM parse.
|
||||
- New overlay dispositions or evidence rungs.
|
||||
- Requiring the extra for `quick` or for importing `openultrasast`.
|
||||
|
||||
## Boundary Commitments
|
||||
|
||||
### This Spec Owns
|
||||
|
||||
- Packaging extra `semantic` and lazy grammar load.
|
||||
- CST → `FileIR` projection and `parse_file` dispatch order.
|
||||
- Probe semantics: extra/grammar vs host CLI.
|
||||
- pytest `semantic` marker and CI extra-on job.
|
||||
|
||||
### Out of Boundary
|
||||
|
||||
- Overlay dispositions, fact TOML, taint worklist (`propose-adjudicate-prove`).
|
||||
- Stage plan, sandbox, hunter suspicion (`three-stage-scan`).
|
||||
- Pair catalog slices (`real-world-vfc-slice`).
|
||||
- CWE policy, pattern rules, MCP `deep`.
|
||||
|
||||
### Allowed Dependencies
|
||||
|
||||
- Existing `FileIR` types and `parse_file` callers (`taint.py`, `overlay.py`).
|
||||
- `harness_ext` capability-guard pattern (find_spec, no import at module load).
|
||||
- External: PyPI `tree-sitter` and grammar packages listed in Technology Stack (extra only).
|
||||
|
||||
### Revalidation Triggers
|
||||
|
||||
- Changing `FileIR` / `Bind` / `CallSite` fields.
|
||||
- Importing `tree_sitter` at module load of `cli` or `semantic/__init__`.
|
||||
- Adding core (non-extra) parser dependencies.
|
||||
- Using host `tree-sitter` CLI output as overlay IR.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Existing Architecture Analysis
|
||||
|
||||
Overlay already calls `parse_file` → taint → dispositions. Python `ast` works. `parse_tree_sitter_cli` always returns `None`. `tree_sitter_available()` is true if a CLI binary exists, which does not imply a grammar.
|
||||
|
||||
### Architecture Pattern and Boundary Map
|
||||
|
||||
Selected: **hybrid extra guard + CST walker + existing dispatch.** Same packaging seam as HarnessX. `parse_file` remains the only overlay parse entry.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
overlay[overlay adjudicate]
|
||||
parse[parse_file]
|
||||
extra[has_semantic_extra]
|
||||
gram[grammar_for language]
|
||||
cst[cst walker]
|
||||
ast[python ast]
|
||||
fail[FileIR language_unsupported or parse_failed]
|
||||
taint[taint on FileIR]
|
||||
overlay --> parse
|
||||
parse --> extra
|
||||
extra -->|yes| gram
|
||||
gram -->|loaded| cst
|
||||
cst -->|has_error| fail
|
||||
cst -->|ok| taint
|
||||
extra -->|no| ast
|
||||
gram -->|missing| ast
|
||||
ast -->|python ok| taint
|
||||
ast -->|not python| fail
|
||||
```
|
||||
|
||||
**Key decisions**
|
||||
|
||||
- Extra grammars are the product parse. Host CLI is not a success path.
|
||||
- Taint does not import tree-sitter.
|
||||
- Dependency direction: extra guard → cst walker → `parse_file` → taint → overlay. No reverse imports.
|
||||
|
||||
### Technology Stack
|
||||
|
||||
| Layer | Choice / Version | Role in Feature | Notes |
|
||||
|-------|------------------|-----------------|-------|
|
||||
| Packaging | extra `semantic` | Optional wheels | Core `dependencies` stay `[]` |
|
||||
| Parser runtime | `tree-sitter>=0.23,<0.27` | `Language` + `Parser` | Constructor API, not `build_library` |
|
||||
| Grammars | `tree-sitter-python`, `tree-sitter-javascript`, `tree-sitter-c`, `tree-sitter-cpp`, `tree-sitter-java` | Per-language CST | `cpp` for `.cpp`; `c` for `.c` |
|
||||
| Tests | pytest marker `semantic` | Skip extra-on tests | Default CI extra-free |
|
||||
| Types | mypy override `tree_sitter*` | Extra absent in typecheck env | Same as `harnessx.*` |
|
||||
|
||||
License of adopted wheels: MIT. Pins are verified at implement time against the selected minor.
|
||||
|
||||
## File Structure Plan
|
||||
|
||||
```
|
||||
src/openultrasast/semantic/
|
||||
extra.py # has_semantic_extra, grammar_for; no tree_sitter import at module load
|
||||
cst.py # Node walk → FileIR; node-kind map
|
||||
ir.py # parse_file dispatch; keep python ast; remove CLI success path
|
||||
engines.py # stop treating host CLI as overlay-ready
|
||||
pyproject.toml
|
||||
tests/test_semantic_extra.py
|
||||
tests/test_semantic_cst.py
|
||||
.github/workflows/ci.yml
|
||||
```
|
||||
|
||||
### Modified Files
|
||||
|
||||
- `pyproject.toml` — extra `semantic`; pytest marker; mypy overrides.
|
||||
- `src/openultrasast/semantic/ir.py` — dispatch; delete `parse_tree_sitter_cli` as success path.
|
||||
- `src/openultrasast/semantic/engines.py` — `tree_sitter_available` must not mean CLI-only; extra/grammar helpers live in `extra.py`.
|
||||
- `src/openultrasast/semantic/__init__.py` — export `has_semantic_extra` only if needed; do not import cst at package load.
|
||||
- `tests/test_semantic_overlay.py` — extra-free JS stays `language_unsupported` without monkeypatching a fake IR as “tree-sitter success”.
|
||||
- `.github/workflows/ci.yml` — second job with `--extra semantic` running `@pytest.mark.semantic`.
|
||||
- `dagger/ci.py` — keep extra-free unless the same extra-on command is cheap to add; default is GHA second job only.
|
||||
|
||||
Do not modify `taint.py`, `overlay.py`, `facts.py`, `gate.py`, `pairs.py`.
|
||||
|
||||
## System Flows
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
start[parse_file]
|
||||
g{grammar_for language}
|
||||
err{root has_error}
|
||||
py{language python}
|
||||
start --> g
|
||||
g -->|yes| err
|
||||
err -->|yes| failed[parse_failed]
|
||||
err -->|no| ir[FileIR engine tree-sitter]
|
||||
g -->|no| py
|
||||
py -->|yes| ast[python-ast FileIR]
|
||||
py -->|no| uns[language_unsupported]
|
||||
```
|
||||
|
||||
Gating: grammar load failure is missing extra/grammar, not a scan abort. `has_error` is `parse_failed`.
|
||||
|
||||
## Requirements Traceability
|
||||
|
||||
| Requirement | Summary | Components | Flows |
|
||||
|-------------|---------|------------|-------|
|
||||
| 1.1–1.3 | Extra-free quick and import | extra.py, pyproject | no grammar load on quick |
|
||||
| 2.1–2.4 | Structured parse; JS promote; C/Java demote; same dispositions | extra.py, cst.py, ir.py, existing taint/overlay | parse_file → taint |
|
||||
| 3.1–3.5 | Fail closed; Python ast last resort; missing extra does not fail standard | ir.py, extra.py | unsupported / parse_failed |
|
||||
| 4.1–4.3 | One walker; no JS/C visitors; CLI not success | cst.py, ir.py | grammar_for only |
|
||||
| 5.1–5.5 | No LLM; Joern not required; corroboration; prove/gate unchanged | none new | existing overlay |
|
||||
| 6.1–6.3 | Dual tests | markers, CI job | skip vs extra-on job |
|
||||
|
||||
## Components and Interfaces
|
||||
|
||||
| Component | Domain | Intent | Req Coverage | Key Dependencies | Contracts |
|
||||
|-----------|--------|--------|--------------|------------------|-----------|
|
||||
| SemanticExtra | Packaging | Probe extra and load grammar | 1.1–1.3, 3.1–3.2, 4.3 | PyPI wheels P0 | Service |
|
||||
| CstWalker | Parse | CST to FileIR | 2.1–2.3, 3.4, 4.1 | SemanticExtra P0, FileIR P0 | Service |
|
||||
| ParseDispatch | Parse | parse_file order | 2.1, 3.3–3.5, 4.3 | CstWalker P0, python ast P0 | Service |
|
||||
| ExtraTestMatrix | Validation | skip vs extra-on CI | 6.1–6.3 | pytest P0 | Batch |
|
||||
|
||||
### Parse layer
|
||||
|
||||
#### SemanticExtra
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Capability guard and per-language grammar load |
|
||||
| Requirements | 1.2, 1.3, 3.1, 3.2, 4.3 |
|
||||
|
||||
**Contracts**: Service
|
||||
|
||||
```python
|
||||
def has_semantic_extra() -> bool:
|
||||
"""True iff tree_sitter and at least one in-scope grammar module are importable. No eager import of wheels at semantic package load."""
|
||||
|
||||
def grammar_for(language: str) -> object | None:
|
||||
"""Return a tree_sitter.Language for python, javascript, c, cpp, java, or None. Lazy importlib.import_module. Never raises into the scan."""
|
||||
```
|
||||
|
||||
- Preconditions: `language` is a preprocess language id.
|
||||
- Postconditions: `None` means overlay must not claim tree-sitter IR.
|
||||
- Invariants: host CLI path does not make this return non-None.
|
||||
|
||||
#### CstWalker
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | One walk from a tree-sitter tree to FileIR |
|
||||
| Requirements | 2.1–2.3, 3.4, 4.1 |
|
||||
|
||||
**Contracts**: Service
|
||||
|
||||
```python
|
||||
def file_ir_from_tree(*, path: str, text: str, language: str, tree: object) -> FileIR:
|
||||
"""Project named nodes into Bind, CallSite, FunctionIR. If root.has_error, parse_ok False reason parse_failed. engine is tree-sitter when parse_ok."""
|
||||
```
|
||||
|
||||
Node-kind map (names are grammar types; walker is shared):
|
||||
|
||||
| Kind | python | javascript | c / cpp | java |
|
||||
|------|--------|------------|---------|------|
|
||||
| function | function_definition | function_declaration, method_definition, arrow_function | function_definition | method_declaration, constructor_declaration |
|
||||
| assign | assignment, augmented_assignment | assignment_expression | assignment_expression | assignment_expression |
|
||||
| call | call | call_expression | call_expression | method_invocation |
|
||||
| name | identifier | identifier | identifier | identifier |
|
||||
| string | string | string | string_literal | string_literal |
|
||||
|
||||
**Implementation Notes**
|
||||
|
||||
- Integration: only `parse_file` calls this after `grammar_for`.
|
||||
- Validation: JS `eval(req.query.x)` and C `printf("%s\\n", data)` fixtures with extra on.
|
||||
- Risks: node-type aliases (JS `assignment_pattern`); table is the extension point, not a new visitor module per language.
|
||||
|
||||
#### ParseDispatch
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Existing `parse_file` with extra-first order |
|
||||
| Requirements | 2.1, 3.3–3.5, 4.2, 4.3 |
|
||||
|
||||
Order: grammar_for → cst; else python ast if `language == python`; else `language_unsupported`. Do not call host CLI for overlay IR.
|
||||
|
||||
## Data Models
|
||||
|
||||
`FileIR`, `Bind`, `CallSite`, `FunctionIR` stay as in `ir.py`. No new disposition. `engine` values: `tree-sitter` | `python-ast` | `none`.
|
||||
|
||||
## Error Handling
|
||||
|
||||
| Condition | Overlay | Scan |
|
||||
|-----------|---------|------|
|
||||
| Extra not installed | language_unsupported for non-Python | standard succeeds |
|
||||
| Grammar missing for language | language_unsupported or Python ast | standard succeeds |
|
||||
| root.has_error | parse_failed | standard succeeds |
|
||||
| Wheel import error | treat as extra absent | do not raise |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- Unit extra-free: import `openultrasast.cli` does not import `tree_sitter`; JS `parse_file` → `language_unsupported`; Python ast still promotes `eval(request.args)`; `quick` scan unchanged (1.1–1.3, 3.1, 3.3, 6.1).
|
||||
- Unit extra-on (`pytest.mark.semantic`): JS `eval(req.query.x)` overlay `promote` (2.2); C `printf("%s\\n", data)` overlay `demote` with dominating fact (2.3); syntax-broken JS → `parse_failed` (3.4); `engine == tree-sitter` on success (2.1).
|
||||
- Skip: if `not has_semantic_extra()`, marked tests skip, extra-free suite green (6.3).
|
||||
- CI: default job extra-free; second job installs extra and runs `-m semantic` (6.2).
|
||||
- Do not assert detection-gate Youden (5.5).
|
||||
|
||||
## Security Considerations
|
||||
|
||||
Parser wheels run in-process on operator source. Same trust as reading the tree today. Extra does not execute target code. Overlay evidence stays `static_corroboration` or below (5.3).
|
||||
|
||||
## Performance
|
||||
|
||||
Parse is per-file MAP, same file budget as rank. No whole-program index.
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
Default install behavior unchanged. Operators who want JS/C/Java overlay run `uv sync --extra semantic`. No artifact format migration.
|
||||
|
||||
## Supporting References
|
||||
|
||||
Grammar load sketch (not runtime-imported in core):
|
||||
|
||||
```python
|
||||
from tree_sitter import Language, Parser
|
||||
import tree_sitter_javascript as tsjs
|
||||
parser = Parser(Language(tsjs.language()))
|
||||
tree = parser.parse(text.encode())
|
||||
```
|
||||
@@ -0,0 +1,78 @@
|
||||
# Requirements Document
|
||||
|
||||
## Introduction
|
||||
|
||||
AppSec engineers scanning mixed working trees need overlay adjudication on Python, JavaScript, C/C++, and Java, not only on files a Python stdlib parse can read. Today a `standard` or `deep` scan leaves JS/C/Java inventory as `language_unsupported` even when a host parser CLI is present, so those proposals never promote or demote. This spec makes an **optional semantic extra** the way to turn on a real structured parse for those languages, while `quick` stays usable with no extra packages and a missing extra stays fail-closed.
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: Optional extra install; overlay using a structured parse for Python, JavaScript, C/C++, and Java when that extra provides a grammar; fail-closed `unadjudicated` when the extra or grammar is missing; `quick` unchanged and extra-free; existing overlay dispositions unchanged.
|
||||
- **Out of scope**: New pair corpora or Firefox/VFC harvest (`real-world-vfc-slice`); Joern CPG queries; inter-file or whole-program taint; buffer-size / CWE-121 semantics; language-model parse or taint; replacing pattern inventory.
|
||||
- **Adjacent expectations**: `propose-adjudicate-prove` still owns dispositions, facts, taint rules, and prove-budget. `three-stage-scan` still owns `quick` / `standard` / `deep` and the sandbox. This spec only supplies the structured parse those stages already asked for.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement 1: Core install and quick stay extra-free
|
||||
|
||||
**Objective:** As a developer or CI job, I want `quick` inventory with no optional packages, so that a default install never pulls parser wheels.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When OpenUltraSAST runs a `quick` scan, the system shall emit inventory proposals without requiring the semantic extra.
|
||||
2. The system shall not require the semantic extra to be installed in order to import or run the default OpenUltraSAST command.
|
||||
3. If the semantic extra is absent, then a `quick` scan shall still complete and shall not abort because a parser package is missing.
|
||||
|
||||
### Requirement 2: Extra enables structured parse on in-scope languages
|
||||
|
||||
**Objective:** As an AppSec engineer with the semantic extra installed, I want overlay to read JavaScript, C/C++, Java, and Python as structured source, so that those files can be promoted or demoted instead of left unsupported.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. Where the semantic extra is installed and a grammar for the file’s language is available, when OpenUltraSAST runs a `standard` or `deep` scan, the system shall use that structured parse as the primary intra-file input to overlay adjudication for Python, JavaScript, C, C++, and Java.
|
||||
2. When a JavaScript file contains `eval` of a request query (or the language-equivalent untrusted source to a known sink) and the extra provides a JavaScript grammar, the system shall be able to emit a `promote` overlay disposition for that proposal.
|
||||
3. When a C or Java file contains a sink fed only by a constant or a dominating sanitizer (for example a literal format string to `printf`) and the extra provides a grammar for that language, the system shall `demote` that proposal rather than leave it `language_unsupported`.
|
||||
4. The system shall feed the structured parse into the existing overlay dispositions (`promote`, `demote`, `unadjudicated`, `coverage`) and shall not introduce a new disposition.
|
||||
|
||||
### Requirement 3: Missing extra or grammar fails closed
|
||||
|
||||
**Objective:** As an operator without the extra or without a grammar for one language, I want honest `unadjudicated` outcomes, so that silence is not reported as safety.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. If the semantic extra is not installed, then when OpenUltraSAST adjudicates a JavaScript, C, C++, or Java file, the system shall leave matching inventory proposals `unadjudicated` with `language_unsupported` or `parse_failed`.
|
||||
2. If the extra is installed but no grammar is available for that file’s language, then the system shall leave those proposals `unadjudicated` with `language_unsupported` or `parse_failed` and shall not crash the scan.
|
||||
3. When the extra is absent, the system shall still try a stdlib parse only for Python and shall otherwise follow the fail-closed rule in criterion 1.
|
||||
4. If a file cannot be parsed even with a grammar present (syntax error or incomplete source), then the system shall leave matching proposals `unadjudicated` with `parse_failed`.
|
||||
5. The system shall not treat a missing extra as a failed `standard` scan by itself.
|
||||
|
||||
### Requirement 4: One parse path, no hand-rolled language visitors
|
||||
|
||||
**Objective:** As a maintainer, I want a single structured-parse path for all in-scope languages, so that we do not grow a new visitor per language.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When the extra supplies grammars for more than one in-scope language, the system shall extract overlay parse facts (assignments, calls, constants) through one shared structured-parse path rather than a separate visitor per language.
|
||||
2. The system shall not add a new language-specific stdlib or regex AST as the product parse engine for JavaScript, C, C++, or Java.
|
||||
3. Where a host parser CLI is present but the semantic extra is not, the system shall not claim a successful structured parse for overlay unless the extra’s grammar actually produced one.
|
||||
|
||||
### Requirement 5: Overlay and stages stay in their existing contracts
|
||||
|
||||
**Objective:** As an operator, I want the extra to change who can be adjudicated, not what a disposition or stage means, so that prove-budget and fail-on rules stay honest.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The system shall not use a language model as the parse or taint engine.
|
||||
2. The system shall not require Joern for the semantic extra to succeed.
|
||||
3. When overlay promotes or demotes from a structured parse, the system shall keep that result at `static_corroboration` or below.
|
||||
4. The system shall not select sandbox candidates from demoted or unadjudicated findings solely because a structured parse ran.
|
||||
5. The system shall not fold overlay Youden or extra-install status into the stage-1 detection gate (`python -m openultrasast.gate`).
|
||||
|
||||
### Requirement 6: Extra present and extra absent are both testable
|
||||
|
||||
**Objective:** As a maintainer, I want CI to prove both the extra-free core and the extra-enabled overlay, so that default install and AppSec install cannot silently diverge.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When the test suite runs without the semantic extra, the system shall still pass the extra-free `quick` and fail-closed overlay checks in Requirements 1 and 3.
|
||||
2. Where the semantic extra is installed in a test environment, the system shall exercise Requirement 2 for at least one JavaScript file and at least one C or Java file.
|
||||
3. If a test environment lacks the extra, then the suite shall skip extra-enabled checks rather than fail the extra-free core.
|
||||
@@ -0,0 +1,171 @@
|
||||
# Gap Analysis: tree-sitter-overlay-extra
|
||||
|
||||
Requirements are **generated but not approved**. Analysis proceeds to inform design; it does not treat requirements as locked.
|
||||
|
||||
Steering `product.md` / `tech.md` / `structure.md` are absent. Context is the overlay package, `harness_ext` extra pattern, `pyproject.toml`, CI, and PyPI tree-sitter bindings.
|
||||
|
||||
---
|
||||
|
||||
## 1. Current State
|
||||
|
||||
### Assets that already exist
|
||||
|
||||
| Asset | Role today |
|
||||
|-------|------------|
|
||||
| `src/openultrasast/semantic/ir.py` | `FileIR` / `Bind` / `CallSite`; Python `ast` extractor (~170 lines); `parse_tree_sitter_cli` **always returns `None`** |
|
||||
| `src/openultrasast/semantic/engines.py` | Probe: env flag, `importlib.util.find_spec("tree_sitter")`, or `tree-sitter` CLI on PATH |
|
||||
| `src/openultrasast/semantic/taint.py` | Language-agnostic taint on `FileIR` (names, constants, fact tables) |
|
||||
| `src/openultrasast/semantic/overlay.py` | Dispositions; already consumes `parse_file` |
|
||||
| `src/openultrasast/ruleset/semantic/*.toml` | JS/C/Java facts already written |
|
||||
| `src/openultrasast/harness_ext.py` | Pattern for optional extra: `find_spec`, lazy import, mypy override, cold-import tests |
|
||||
| `pyproject.toml` | `dependencies = []`; only extra is `harnessx` |
|
||||
| `tests/test_semantic_overlay.py` | JS unsupported without probe; JS IR only via **monkeypatch**, not a real grammar |
|
||||
|
||||
### Conventions
|
||||
|
||||
- Optional native/heavy deps stay off the default install; capability probe must not import at module load (`harness_ext`, Docker sandbox probe).
|
||||
- mypy `ignore_missing_imports` for extras (`harnessx.*`).
|
||||
- pytest marker already exists for `docker`; extra-enabled tests would follow that skip pattern.
|
||||
- CI `checks` job: `uv sync --group dev` then pytest — **no optional extras**.
|
||||
|
||||
### Integration surfaces
|
||||
|
||||
- `parse_file(path, text, language) -> FileIR` is the only seam taint/overlay need.
|
||||
- `tree_sitter_available()` is a **presence** probe, not “grammar produced IR”. Host CLI on PATH makes it True even with no PyPI wheels (this environment has `/home/linuxbrew/.../tree-sitter` and **no languages**).
|
||||
- Pair eval already falls back to inventory when a language never promotes/demotes.
|
||||
|
||||
## 2. Requirements Feasibility
|
||||
|
||||
| Req | Need | Gap |
|
||||
|-----|------|-----|
|
||||
| 1 Extra-free `quick` | Empty core deps; no import of parser wheels at load | **Constraint met.** Missing extra cannot currently break `quick` because there is no extra. Must keep it that way. |
|
||||
| 2 Structured parse for Py/JS/C/C++/Java | Real CST → `FileIR`; JS promote; C/Java demote on constant/sanitizer | **Missing.** CLI parse never builds `FileIR`. No `[semantic]` extra. No CST walker. |
|
||||
| 3 Fail closed | `language_unsupported` / `parse_failed`; Python ast last resort | **Mostly present** via `parse_file` fallback. Syntax-error Python already `parse_failed`. JS/C/Java without IR already `language_unsupported`. |
|
||||
| 4 One shared extract path | Not a visitor per language; CLI without extra must not fake success | **Missing walker.** CLI stub already returns `None` (does not fake success). Probe can still report “available” without IR — **constraint** for Req 4.3 wording vs `tree_sitter_available()`. |
|
||||
| 5 Overlay contracts | No LLM parse; Joern not required; corroboration; promotions-only sandbox; gate inventory-only | **Present** in overlay/prove/gate. Do not reopen. |
|
||||
| 6 Dual test matrix | Extra-free always; extra-on JS + C/Java; skip if extra absent | **Partial.** Extra-free tests exist. Extra-on tests are mocked. No pytest marker. CI does not install a semantic extra. |
|
||||
|
||||
**Complexity signal:** external native-wheel extra + one algorithmic CST walk + pytest/CI matrix. Not CRUD.
|
||||
|
||||
**Research needed (design phase, not blockers):**
|
||||
|
||||
1. Pin compatible `tree-sitter` + `tree-sitter-{python,javascript,c,cpp,java}` wheel ABIs (`Language(mod.language())` vs older `build_library`). PyPI: `tree-sitter` 0.26 (2026-06-30), `tree-sitter-javascript` 0.25.
|
||||
2. Per-language CST node names for assignment, call, identifier, string, function (JS `assignment_expression` vs C `assignment_expression` vs Java `assignment_expression` vs Python `assignment`). Shared walker needs a small node-kind table, not four visitors.
|
||||
3. C++: separate `tree-sitter-cpp` vs C grammar; `preprocess` maps `.cpp` to `cpp`.
|
||||
4. Whether default CI should add a second job `uv sync --extra semantic` or only skip-marked tests on the extra-free job.
|
||||
|
||||
## 3. Implementation Approach Options
|
||||
|
||||
### Option A: Extend `ir.py` / `engines.py` only
|
||||
|
||||
Put extra pins in `pyproject.toml`, implement API parse + CST walk inside `ir.py`, tighten `tree_sitter_available()`.
|
||||
|
||||
- **Pros:** Few files; `parse_file` stays the seam.
|
||||
- **Cons:** `ir.py` is already ~277 lines (Python ast + dead CLI). A walker + language table will bloat it and mix packaging probe with extract.
|
||||
|
||||
### Option B: New components only
|
||||
|
||||
New `semantic/cst.py` + `semantic/extra.py`; leave `ir.py` as Python ast; overlay calls a new entrypoint.
|
||||
|
||||
- **Pros:** Clean extra guard.
|
||||
- **Cons:** Two parse entrypoints; easy to fork `parse_file` consumers.
|
||||
|
||||
### Option C: Hybrid (recommended for design)
|
||||
|
||||
- **New:** `semantic/cst.py` — one walker: tree-sitter `Node` → `Bind` / `CallSite` / `FunctionIR` using a node-kind map.
|
||||
- **New or thin:** extra guard modeled on `harness_ext` (`has_semantic_extra`, lazy grammar load, never import wheels at module load).
|
||||
- **Extend:** `pyproject.toml` `[project.optional-dependencies] semantic = [...]`; mypy override `tree_sitter*`; pytest marker `semantic`; CI extra-free job unchanged; optional second job or skip.
|
||||
- **Extend:** `parse_file` prefers extra CST when a grammar loads; else Python ast; else `language_unsupported`. **Delete or stop calling** `parse_tree_sitter_cli` as a success path (Req 4.3).
|
||||
- **Do not extend:** `taint.py`, `overlay.py`, facts, gate, pair catalog.
|
||||
|
||||
**Trade-offs:** Extra files, but `ir.py` shrinks (drop dead CLI), taint stays language-agnostic, extra follows an existing packaging seam.
|
||||
|
||||
## 4. Effort and Risk
|
||||
|
||||
- **Effort: M (3–7 days).** Extra pins + walker + dual tests. Overlay/taint reuse. Not a new adjudicator.
|
||||
- **Risk: Medium.** Native wheels and grammar ABI pins; CST node-name drift across languages; CI without extra must stay green. Mitigate with skip markers and a version pin table in design.
|
||||
|
||||
## 5. Design-phase recommendations
|
||||
|
||||
- Prefer **Option C**.
|
||||
- Treat `FileIR` as frozen contract; do not add dispositions.
|
||||
- Decide probe semantics: `tree_sitter_available` should mean “can produce FileIR for this language”, not “CLI binary exists”.
|
||||
- Carry research items 1–4 into design (ABI pins, node-kind map, cpp package, CI job vs skip).
|
||||
- Do not pull Joern, pair harvest, or CWE-121 into this spec.
|
||||
|
||||
---
|
||||
|
||||
## Summary (design discovery)
|
||||
|
||||
- **Feature**: tree-sitter-overlay-extra
|
||||
- **Discovery Scope**: Extension of `parse_file` / overlay IR; optional extra packaging
|
||||
- **Key Findings**:
|
||||
- Official PyPI wheels: `tree-sitter` (0.26), `tree-sitter-python`, `tree-sitter-javascript` (0.25), `tree-sitter-c` (0.24.x), `tree-sitter-cpp` (0.23.4), `tree-sitter-java`. API is `Parser(Language(mod.language()))`; `build_library` is gone.
|
||||
- `FileIR` is the only overlay seam; taint does not need language visitors.
|
||||
- Host CLI presence must not count as a successful overlay parse (Req 4.3).
|
||||
|
||||
## Research Log
|
||||
|
||||
### PyPI ABI and load API
|
||||
|
||||
- **Context**: Pin wheels that work on Python 3.11+ without compiling grammars in CI.
|
||||
- **Sources Consulted**: https://pypi.org/project/tree-sitter/ ; https://pypi.org/project/tree-sitter-javascript/ ; https://pypi.org/project/tree-sitter-cpp/ ; py-tree-sitter 0.23 release notes (Language capsule, Parser constructor).
|
||||
- **Findings**: Grammars ship abi3 wheels. Load with `Language(tree_sitter_<lang>.language())` then `Parser(language)`. Do not use `Language.build_library`.
|
||||
- **Implications**: Extra lists core + five grammar packages. mypy ignores `tree_sitter*`.
|
||||
|
||||
### Probe vs grammar
|
||||
|
||||
- **Context**: `tree_sitter_available()` is true when CLI is on PATH with zero grammars.
|
||||
- **Findings**: CLI parse cannot populate `FileIR` without a walker. Extra Python API is the product parse.
|
||||
- **Implications**: Split `has_semantic_extra()` from per-language `grammar_for`. Remove CLI as a success path.
|
||||
|
||||
## Architecture Pattern Evaluation
|
||||
|
||||
| Option | Description | Strengths | Risks | Notes |
|
||||
|--------|-------------|-----------|-------|-------|
|
||||
| A Extend ir.py only | Walker + extra inside ir.py | Few files | Bloats 277-line module | Rejected |
|
||||
| B New entrypoint | Overlay calls cst.py directly | Clean extra | Forks parse_file | Rejected |
|
||||
| C Hybrid | Extra guard + cst walker; parse_file dispatch | Matches harness_ext; FileIR frozen | Node-kind table | **Selected** |
|
||||
|
||||
## Design Decisions
|
||||
|
||||
### Decision: Adopt official grammar wheels, build one walker
|
||||
|
||||
- **Context**: Avoid per-language AST visitors (Req 4).
|
||||
- **Alternatives**: tree-sitter CLI sexp; py-tree-sitter-languages (unmaintained); hand-rolled JS/C parsers.
|
||||
- **Selected**: `openultrasast[semantic]` = `tree-sitter` plus python/javascript/c/cpp/java grammar packages. One CST walk keyed by a node-kind map.
|
||||
- **Rationale**: Wheels are MIT, maintained, match in-scope languages. Taint stays language-agnostic.
|
||||
- **Trade-offs**: Native wheels in the extra; ABI pins. Core install stays empty.
|
||||
- **Follow-up**: Verify Parser constructor against the pinned `tree-sitter` minor at implement time.
|
||||
|
||||
### Decision: Extra-free CI stays default; extra-on is a second job plus skip marker
|
||||
|
||||
- **Context**: Req 6 dual matrix.
|
||||
- **Selected**: pytest `semantic` marker skips when extra absent. GitHub Actions adds a job `uv sync --extra semantic --group dev` that runs marked tests. Default `checks` job unchanged.
|
||||
- **Rationale**: Default CI must not fail without wheels. Extra-on tests must actually run somewhere.
|
||||
- **Trade-offs**: Two pytest invocations in CI.
|
||||
|
||||
### Decision: has_error implies parse_failed
|
||||
|
||||
- **Context**: tree-sitter is error-tolerant; Req 3.4 wants fail-closed on syntax errors.
|
||||
- **Selected**: If `tree.root_node.has_error` (or ERROR children), return `FileIR(parse_ok=False, reason=parse_failed)`.
|
||||
- **Rationale**: Partial IR would invite false demote.
|
||||
|
||||
## Synthesis
|
||||
|
||||
- **Generalization**: One `parse_file` dispatch; extra is a grammar source, not a second overlay.
|
||||
- **Build vs adopt**: Adopt tree-sitter wheels; build only the FileIR projection (binds/calls).
|
||||
- **Simplification**: No CLI sexp walker; no second parse API; no taint/overlay changes.
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
- Grammar ABI mismatch — pin extras together; extra-on CI job catches import/parse failure.
|
||||
- Node-type drift — table in cst.py; tests assert JS eval and C printf on real trees.
|
||||
- CLI still on PATH — must not set parse_ok.
|
||||
|
||||
## References
|
||||
|
||||
- https://pypi.org/project/tree-sitter/
|
||||
- https://github.com/tree-sitter/py-tree-sitter/releases
|
||||
- `src/openultrasast/harness_ext.py` extra guard pattern
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"feature_name": "tree-sitter-overlay-extra",
|
||||
"created_at": "2026-09-04T14:00:00Z",
|
||||
"updated_at": "2026-09-04T15:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "tasks-generated",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"design": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"tasks": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": true
|
||||
}
|
||||
@@ -0,0 +1,93 @@
|
||||
# Implementation Plan
|
||||
|
||||
- [x] 1. Foundation: optional extra and capability probe
|
||||
- [x] 1.1 Declare the semantic extra without adding core dependencies
|
||||
- Core install dependencies stay empty.
|
||||
- Optional extra named semantic lists the parser runtime and grammars for Python, JavaScript, C, C++, and Java.
|
||||
- Typecheck ignores the extra modules the same way it ignores the existing optional agentic extra.
|
||||
- Pytest registers a semantic marker for extra-on tests.
|
||||
- Observable: a default sync still has an empty core dependency list; the extra is listed only as optional.
|
||||
- _Requirements: 1.1, 1.2, 6.3_
|
||||
- _Boundary: SemanticExtra_
|
||||
|
||||
- [x] 1.2 Probe the extra and load a grammar without importing wheels at package load
|
||||
- Capability check uses importability, not a host CLI binary.
|
||||
- Grammar load is lazy per language and returns nothing when the extra or that grammar is missing.
|
||||
- Importing the overlay package or CLI does not import parser wheels.
|
||||
- Host CLI on PATH does not make a grammar available.
|
||||
- Observable: with the extra absent, the probe is false and a JavaScript language request yields no grammar; importing the CLI in a fresh interpreter does not load tree_sitter.
|
||||
- _Requirements: 1.2, 1.3, 3.1, 3.2, 4.3_
|
||||
- _Boundary: SemanticExtra_
|
||||
- _Depends: 1.1_
|
||||
|
||||
- [x] 1.3 Stop treating a host parser CLI as overlay-ready
|
||||
- Overlay-ready means a grammar can be loaded for that language, not that a CLI exists.
|
||||
- Existing env override for tests can still force the probe off.
|
||||
- Observable: a machine with a parser CLI and no extra still reports no overlay grammar for JavaScript.
|
||||
- _Requirements: 4.3, 3.1_
|
||||
- _Boundary: SemanticExtra_
|
||||
- _Depends: 1.2_
|
||||
|
||||
- [x] 2. Shared structured parse into existing overlay IR
|
||||
- [x] 2.1 Walk one syntax tree into assignments, calls, and functions
|
||||
- One walker serves Python, JavaScript, C, C++, and Java via a node-kind table, not a visitor per language.
|
||||
- Successful trees set engine to tree-sitter and fill the existing IR records overlay taint already consumes.
|
||||
- No new language-specific stdlib or regex AST is added for JavaScript, C, C++, or Java.
|
||||
- Observable: with a JavaScript grammar loaded, `eval(req.query.x)` produces a call record for eval with the query expression as an argument.
|
||||
- _Requirements: 2.1, 4.1, 4.2_
|
||||
- _Boundary: CstWalker_
|
||||
- _Depends: 1.2_
|
||||
|
||||
- [x] 2.2 Treat error trees as parse failure
|
||||
- If the tree reports an error, the IR is not parse_ok and the reason is parse_failed.
|
||||
- A syntax-broken file does not yield a fake clean tree for overlay.
|
||||
- Observable: incomplete JavaScript source returns parse_failed rather than a promote or demote.
|
||||
- _Requirements: 3.4_
|
||||
- _Boundary: CstWalker_
|
||||
- _Depends: 2.1_
|
||||
|
||||
- [x] 3. Integration: parse dispatch
|
||||
- [x] 3.1 Prefer extra grammar, then Python stdlib, then unsupported
|
||||
- Parse order is grammar walk, else Python stdlib parse for Python only, else language_unsupported.
|
||||
- Host CLI output is never used as a successful overlay parse.
|
||||
- Missing extra does not fail a standard scan by itself.
|
||||
- Overlay still emits only promote, demote, unadjudicated, and coverage.
|
||||
- Observable: extra-free JavaScript inventory is unadjudicated language_unsupported; extra-free Python `eval(request.args)` still promotes via stdlib parse; extra-on JavaScript uses engine tree-sitter.
|
||||
- _Requirements: 2.1, 2.4, 3.3, 3.5, 4.2, 4.3_
|
||||
- _Boundary: ParseDispatch_
|
||||
- _Depends: 1.2, 2.1, 2.2_
|
||||
|
||||
- [x] 4. Validation: extra-free core, extra-on overlay, CI matrix
|
||||
- [x] 4.1 Prove extra-free quick and fail-closed overlay
|
||||
- Quick scan completes without the extra and still emits inventory.
|
||||
- JavaScript, C, and Java stay unadjudicated language_unsupported or parse_failed without the extra.
|
||||
- Python stdlib parse remains the last resort when the extra is absent.
|
||||
- Observable: extra-free pytest covering quick plus JS unsupported is green; detection gate still ignores overlay Youden.
|
||||
- _Requirements: 1.1, 1.3, 3.1, 3.3, 5.5, 6.1_
|
||||
- _Boundary: ExtraTestMatrix_
|
||||
- _Depends: 3.1_
|
||||
|
||||
- [x] 4.2 (P) Prove extra-on JavaScript promote and C or Java demote
|
||||
- Marked tests skip when the extra is absent instead of failing the extra-free suite.
|
||||
- With the extra, JavaScript eval of request query promotes.
|
||||
- With the extra, a C or Java sink fed only by a constant or dominating sanitizer demotes rather than language_unsupported.
|
||||
- Evidence stays at static_corroboration or below; sandbox selection is still promotions only.
|
||||
- Observable: extra-on tests pass when wheels are installed and skip otherwise; JS promote and C or Java demote assertions fail if the walker is removed.
|
||||
- _Requirements: 2.2, 2.3, 3.4, 5.3, 5.4, 6.2, 6.3_
|
||||
- _Boundary: ExtraTestMatrix_
|
||||
- _Depends: 3.1_
|
||||
|
||||
- [x] 4.3 Run extra-on checks in CI without breaking extra-free CI
|
||||
- Default CI job stays extra-free and must remain green.
|
||||
- A second job installs the semantic extra and runs the semantic-marked tests.
|
||||
- Joern is not required; no language model is used as parse or taint.
|
||||
- Observable: extra-free job does not install parser wheels; extra-on job executes the marked JS and C or Java tests.
|
||||
- _Requirements: 5.1, 5.2, 6.1, 6.2, 6.3_
|
||||
- _Boundary: ExtraTestMatrix_
|
||||
- _Depends: 4.1, 4.2_
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- Extra wheels: tree-sitter 0.26 with grammar packages python/javascript/c/cpp/java. `OPENULTRASAST_TREE_SITTER_PROBE=0` simulates extra-absent even when wheels are installed locally.
|
||||
- Host `tree-sitter` CLI is not overlay-ready; only lazy `grammar_for` produces FileIR.
|
||||
- Default CI stays extra-free; job `semantic` runs `uv sync --extra semantic` then `pytest -m semantic`.
|
||||
@@ -0,0 +1,47 @@
|
||||
# Roadmap
|
||||
|
||||
## Overview
|
||||
|
||||
OpenUltraSAST can propose sinks with regex and prove a few of them in a sandbox, but adjudication is still Python-`ast` plus a tree-sitter CLI stub. Textbook SAST pairs (OWASP/Juliet) did not improve after the overlay: Youden went from **+9.09% inventory to 0% overlay** on the same 11-pair slice. Real operator trees and browser/C/C++ history are not in the catalog. The next work is to put a real tree-sitter extra under the existing overlay, then grow an offline function-level vuln-vs-fix slice from public VFC datasets (including Firefox/gecko-dev where commits exist) without vendoring whole megarepos.
|
||||
|
||||
## Approach Decision
|
||||
|
||||
- **Chosen**: Optional tree-sitter extra first, then a harvested VFC slice scored like `github` (honesty dashboard, not smoke gate). Isolated functions only. Firefox is one project filter among Chromium, OpenSSL, curl — not a special-case full mozilla-central clone.
|
||||
- **Why**: The overlay already has dispositions, facts, and taint. JS/C/Java stay `language_unsupported` because we refused a parser library. Textbook Youden is the wrong north star; large-project VFCs are the right one. Adopting tree-sitter deletes the need to hand-write language parsers.
|
||||
- **Rejected alternatives**:
|
||||
- Whole Firefox tree in CI — multi-GB, memory-safety heavy, security-bug commit messages are deliberately scrubbed.
|
||||
- FixFox dataset — embargoed until 2030.
|
||||
- Joern-as-required or inter-file taint in this phase — deferred by `propose-adjudicate-prove`.
|
||||
- Chasing OWASP Youden with more regex — leaks on goodG2B/false files.
|
||||
|
||||
## Scope
|
||||
|
||||
- **In**: tree-sitter + grammars as `openultrasast[semantic]`; CST walker into existing FileIR/taint; harvested function-level pairs from CVEfixes/MoreFixes/CWE-Bench-Java/SVEN with provenance; overlay scoring on that slice; keep `quick` zero-dep.
|
||||
- **Out**: Cloning mozilla-central/Chromium into the repo; making overlay Youden a merge gate; buffer-size/CWE-121 semantics; inter-file taint; auto-facts from improve; LLM as parser.
|
||||
|
||||
## Constraints
|
||||
|
||||
- Core `dependencies = []`. Parser wheels live only in an extra.
|
||||
- Pair fixtures stay out of `LANGUAGE_MANIFESTS`.
|
||||
- Do not vendor 16 GB VFC dumps. Pointers in `datasets.toml`; a small reviewed catalog of isolated functions is what CI runs.
|
||||
- DiverseVul-style auto labels (~60% accurate) are not ground truth without review.
|
||||
|
||||
## Boundary Strategy
|
||||
|
||||
- **Why this split**: Engine (how we parse) is independent of corpus (what we score). The extra can ship without any new pairs; the corpus can score inventory-only until the extra lands, then overlay.
|
||||
- **Shared seams to watch**: `FileIR` / `parse_file`; pair catalog `slice` values; overlay vs inventory scoring in `pairs.py`.
|
||||
|
||||
## Existing Spec Updates
|
||||
|
||||
- [x] propose-adjudicate-prove -- overlay roles, facts, prove-budget, sast overlay scoring (landed). Residual: tree-sitter path returns no IR.
|
||||
- [ ] propose-adjudicate-prove -- do not reopen for parsers or corpora; those are the new specs below.
|
||||
|
||||
## Direct Implementation Candidates
|
||||
|
||||
- [ ] Record inventory vs overlay Youden side-by-side in `ousast pairs --slice sast` JSON (small report field, not a new spec).
|
||||
- [ ] Add Firefox/gecko-dev, Chromium, OpenSSL pointers to `benchmarks/pairs/datasets.toml`.
|
||||
|
||||
## Specs (dependency order)
|
||||
|
||||
- [ ] tree-sitter-overlay-extra -- Optional extra: tree-sitter + grammars, one CST walker, `quick` stays zero-dep. Dependencies: none
|
||||
- [ ] real-world-vfc-slice -- Harvest reviewed function-level vuln/fix pairs from large projects (Firefox/gecko-dev, Chromium, OpenSSL, …) as an honesty slice. Dependencies: none
|
||||
+18
-1
@@ -20,6 +20,16 @@ dependencies = []
|
||||
harnessx = [
|
||||
"harnessx @ git+https://github.com/Darwin-Agent/HarnessX.git@274c512eea67c64701f992d3b6fe06f0fd590ef0",
|
||||
]
|
||||
# Structured parse for overlay (JS/C/C++/Java/Python). Core install stays empty.
|
||||
# Wheels are MIT; import is lazy via openultrasast.semantic.extra.
|
||||
semantic = [
|
||||
"tree-sitter>=0.23,<0.27",
|
||||
"tree-sitter-python>=0.23",
|
||||
"tree-sitter-javascript>=0.23",
|
||||
"tree-sitter-c>=0.23",
|
||||
"tree-sitter-cpp>=0.23",
|
||||
"tree-sitter-java>=0.23",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
ousast = "openultrasast.cli:main"
|
||||
@@ -27,7 +37,10 @@ ousast = "openultrasast.cli:main"
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
pythonpath = ["src"]
|
||||
markers = ["docker: requires a docker daemon"]
|
||||
markers = [
|
||||
"docker: requires a docker daemon",
|
||||
"semantic: requires openultrasast[semantic] tree-sitter extra",
|
||||
]
|
||||
|
||||
[tool.ruff]
|
||||
line-length = 140
|
||||
@@ -55,6 +68,10 @@ warn_unused_configs = true
|
||||
module = "harnessx.*"
|
||||
ignore_missing_imports = true
|
||||
|
||||
[[tool.mypy.overrides]]
|
||||
module = "tree_sitter"
|
||||
ignore_missing_imports = true
|
||||
|
||||
[dependency-groups]
|
||||
dev = [
|
||||
"dagger-io>=0.21.7",
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
"""Propose/adjudicate/prove overlay: regex proposes, semantics adjudicates."""
|
||||
|
||||
from .engines import joern_available, tree_sitter_available
|
||||
from .extra import grammar_for, has_semantic_extra
|
||||
from .facts import FactLoadError, SemanticFacts, load_facts
|
||||
from .mechanisms import Mechanism, MechanismStore, append_mechanism, order_promotions
|
||||
from .overlay import OverlayError, OverlayRecord, adjudicate, finding_from_coverage, write_overlay
|
||||
@@ -17,6 +18,8 @@ __all__ = [
|
||||
"append_mechanism",
|
||||
"filter_promoted_hotspots",
|
||||
"finding_from_coverage",
|
||||
"grammar_for",
|
||||
"has_semantic_extra",
|
||||
"joern_available",
|
||||
"load_facts",
|
||||
"order_promotions",
|
||||
|
||||
@@ -0,0 +1,258 @@
|
||||
"""One CST walker: tree-sitter nodes into FileIR. No per-language visitors."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from .ir import Bind, CallSite, FileIR, FunctionIR
|
||||
|
||||
_KIND = {
|
||||
"python": {
|
||||
"function": frozenset({"function_definition"}),
|
||||
"assign": frozenset({"assignment", "augmented_assignment"}),
|
||||
"call": frozenset({"call"}),
|
||||
"args": frozenset({"argument_list", "arguments"}),
|
||||
"string": frozenset({"string"}),
|
||||
"params": frozenset({"parameters"}),
|
||||
},
|
||||
"javascript": {
|
||||
"function": frozenset({"function_declaration", "method_definition", "arrow_function", "function_expression"}),
|
||||
"assign": frozenset({"assignment_expression", "augmented_assignment_expression"}),
|
||||
"call": frozenset({"call_expression"}),
|
||||
"args": frozenset({"arguments"}),
|
||||
"string": frozenset({"string"}),
|
||||
"params": frozenset({"formal_parameters"}),
|
||||
},
|
||||
"c": {
|
||||
"function": frozenset({"function_definition"}),
|
||||
"assign": frozenset({"assignment_expression"}),
|
||||
"call": frozenset({"call_expression"}),
|
||||
"args": frozenset({"argument_list"}),
|
||||
"string": frozenset({"string_literal"}),
|
||||
"params": frozenset({"parameter_list"}),
|
||||
},
|
||||
"java": {
|
||||
"function": frozenset({"method_declaration", "constructor_declaration"}),
|
||||
"assign": frozenset({"assignment_expression", "variable_declarator"}),
|
||||
"call": frozenset({"method_invocation"}),
|
||||
"args": frozenset({"argument_list"}),
|
||||
"string": frozenset({"string_literal"}),
|
||||
"params": frozenset({"formal_parameters"}),
|
||||
},
|
||||
}
|
||||
_KIND["cpp"] = _KIND["c"]
|
||||
_KIND["typescript"] = _KIND["javascript"]
|
||||
|
||||
_CONST = frozenset(
|
||||
{
|
||||
"string",
|
||||
"string_literal",
|
||||
"integer",
|
||||
"number",
|
||||
"float",
|
||||
"true",
|
||||
"false",
|
||||
"null",
|
||||
"nil",
|
||||
"character_literal",
|
||||
"escape_sequence",
|
||||
}
|
||||
)
|
||||
_SEQUENCE = frozenset({"list", "tuple", "array", "array_creation_expression", "initializer_list"})
|
||||
|
||||
|
||||
def parse_with_cst(path: str, text: str, language: str) -> FileIR | None:
|
||||
from .extra import grammar_for
|
||||
|
||||
grammar = grammar_for(language)
|
||||
if grammar is None:
|
||||
return None
|
||||
try:
|
||||
from tree_sitter import Parser
|
||||
|
||||
parser = Parser(grammar)
|
||||
tree = parser.parse(text.encode())
|
||||
except Exception:
|
||||
return FileIR(path=path, language=language, engine="tree-sitter", functions=(), parse_ok=False, reason="parse_failed")
|
||||
return file_ir_from_tree(path=path, text=text, language=language, tree=tree)
|
||||
|
||||
|
||||
def file_ir_from_tree(*, path: str, text: str, language: str, tree: Any) -> FileIR:
|
||||
del text
|
||||
root = tree.root_node
|
||||
if bool(getattr(root, "has_error", False)):
|
||||
return FileIR(path=path, language=language, engine="tree-sitter", functions=(), parse_ok=False, reason="parse_failed")
|
||||
kinds = _KIND.get(language)
|
||||
if kinds is None:
|
||||
return FileIR(path=path, language=language, engine="none", functions=(), parse_ok=False, reason="language_unsupported")
|
||||
functions: list[FunctionIR] = []
|
||||
module_binds, module_calls = _collect(root, kinds, skip=kinds["function"])
|
||||
functions.append(
|
||||
FunctionIR(
|
||||
name="<module>",
|
||||
params=(),
|
||||
start_line=_line(root),
|
||||
end_line=_end_line(root),
|
||||
binds=tuple(module_binds),
|
||||
calls=tuple(module_calls),
|
||||
)
|
||||
)
|
||||
for node in _walk(root):
|
||||
if node.type not in kinds["function"]:
|
||||
continue
|
||||
binds, calls = _collect(node, kinds, skip=kinds["function"])
|
||||
functions.append(
|
||||
FunctionIR(
|
||||
name=_function_name(node),
|
||||
params=_params(node, kinds),
|
||||
start_line=_line(node),
|
||||
end_line=_end_line(node),
|
||||
binds=tuple(binds),
|
||||
calls=tuple(calls),
|
||||
)
|
||||
)
|
||||
return FileIR(path=path, language=language, engine="tree-sitter", functions=tuple(functions), parse_ok=True)
|
||||
|
||||
|
||||
def _collect(node: Any, kinds: dict[str, frozenset[str]], *, skip: frozenset[str]) -> tuple[list[Bind], list[CallSite]]:
|
||||
binds: list[Bind] = []
|
||||
calls: list[CallSite] = []
|
||||
for child in _walk(node, skip=skip):
|
||||
if child.type in kinds["assign"]:
|
||||
bind = _bind(child, kinds)
|
||||
if bind is not None:
|
||||
binds.append(bind)
|
||||
if child.type in kinds["call"]:
|
||||
calls.append(_call(child, kinds))
|
||||
return binds, calls
|
||||
|
||||
|
||||
def _walk(node: Any, *, skip: frozenset[str] | None = None) -> list[Any]:
|
||||
found: list[Any] = [node]
|
||||
for child in getattr(node, "named_children", ()) or ():
|
||||
if skip is not None and child.type in skip:
|
||||
continue
|
||||
found.extend(_walk(child, skip=skip))
|
||||
return found
|
||||
|
||||
|
||||
def _bind(node: Any, kinds: dict[str, frozenset[str]]) -> Bind | None:
|
||||
children = list(getattr(node, "named_children", ()) or ())
|
||||
if not children:
|
||||
return None
|
||||
name_node = children[0]
|
||||
value_node = children[-1] if len(children) > 1 else None
|
||||
name = _left_name(name_node)
|
||||
if not name:
|
||||
return None
|
||||
value_text = _text(value_node) if value_node is not None else ""
|
||||
return Bind(
|
||||
name=name,
|
||||
line=_line(node),
|
||||
value_text=value_text,
|
||||
is_constant=_is_constant(value_node, kinds) if value_node is not None else False,
|
||||
names=_identifiers(value_node) if value_node is not None else (),
|
||||
call_name=_call_name(value_node, kinds) if value_node is not None and value_node.type in kinds["call"] else None,
|
||||
)
|
||||
|
||||
|
||||
def _call(node: Any, kinds: dict[str, frozenset[str]]) -> CallSite:
|
||||
args_node = next((child for child in getattr(node, "named_children", ()) or () if child.type in kinds["args"]), None)
|
||||
arg_nodes = list(getattr(args_node, "named_children", ()) or ()) if args_node is not None else []
|
||||
extra = len(arg_nodes) >= 2 and arg_nodes[1].type in _SEQUENCE
|
||||
return CallSite(
|
||||
name=_callee_name(node, args_node),
|
||||
line=_line(node),
|
||||
arg_texts=tuple(_text(arg) for arg in arg_nodes),
|
||||
arg_is_constant=tuple(_is_constant(arg, kinds) for arg in arg_nodes),
|
||||
arg_names=tuple(_identifiers(arg) for arg in arg_nodes),
|
||||
extra_arg_is_sequence=extra,
|
||||
)
|
||||
|
||||
|
||||
def _callee_name(node: Any, args_node: Any) -> str:
|
||||
idents = [
|
||||
_text(child)
|
||||
for child in getattr(node, "named_children", ()) or ()
|
||||
if child is not args_node and child.type in {"identifier", "property_identifier"}
|
||||
]
|
||||
if idents:
|
||||
return idents[-1]
|
||||
for child in getattr(node, "named_children", ()) or ():
|
||||
if child is not args_node:
|
||||
text = _text(child)
|
||||
if text:
|
||||
return text.split("(")[0]
|
||||
return _text(node).split("(")[0]
|
||||
|
||||
|
||||
def _call_name(node: Any, kinds: dict[str, frozenset[str]]) -> str | None:
|
||||
if node.type in kinds["call"]:
|
||||
args_node = next((child for child in getattr(node, "named_children", ()) or () if child.type in kinds["args"]), None)
|
||||
return _callee_name(node, args_node)
|
||||
return None
|
||||
|
||||
|
||||
def _function_name(node: Any) -> str:
|
||||
for child in getattr(node, "named_children", ()) or ():
|
||||
if child.type == "identifier":
|
||||
return _text(child)
|
||||
if child.type == "function_declarator":
|
||||
for grandchild in getattr(child, "named_children", ()) or ():
|
||||
if grandchild.type == "identifier":
|
||||
return _text(grandchild)
|
||||
return "<anon>"
|
||||
|
||||
|
||||
def _params(node: Any, kinds: dict[str, frozenset[str]]) -> tuple[str, ...]:
|
||||
for child in _walk(node, skip=frozenset()):
|
||||
if child.type in kinds["params"]:
|
||||
return _identifiers(child)
|
||||
if child.type == "function_declarator":
|
||||
for grandchild in getattr(child, "named_children", ()) or ():
|
||||
if grandchild.type in kinds["params"]:
|
||||
return _identifiers(grandchild)
|
||||
return ()
|
||||
|
||||
|
||||
def _left_name(node: Any) -> str:
|
||||
if node.type in {"identifier", "property_identifier"}:
|
||||
return _text(node)
|
||||
idents = _identifiers(node)
|
||||
return idents[0] if idents else ""
|
||||
|
||||
|
||||
def _identifiers(node: Any) -> tuple[str, ...]:
|
||||
names: list[str] = []
|
||||
for child in _walk(node):
|
||||
if child.type == "identifier":
|
||||
names.append(_text(child))
|
||||
return tuple(names)
|
||||
|
||||
|
||||
def _is_constant(node: Any, kinds: dict[str, frozenset[str]]) -> bool:
|
||||
if node.type in _CONST or node.type in kinds["string"]:
|
||||
return True
|
||||
if node.type not in {"binary_expression", "concatenated_string"}:
|
||||
return False
|
||||
children = list(getattr(node, "named_children", ()) or ())
|
||||
return bool(children) and all(_is_constant(child, kinds) for child in children)
|
||||
|
||||
|
||||
def _text(node: Any) -> str:
|
||||
raw = getattr(node, "text", None)
|
||||
if raw is None:
|
||||
return ""
|
||||
if isinstance(raw, bytes):
|
||||
return raw.decode(errors="ignore")
|
||||
return str(raw)
|
||||
|
||||
|
||||
def _line(node: Any) -> int:
|
||||
point = getattr(node, "start_point", (0, 0))
|
||||
return int(point[0]) + 1
|
||||
|
||||
|
||||
def _end_line(node: Any) -> int:
|
||||
point = getattr(node, "end_point", getattr(node, "start_point", (0, 0)))
|
||||
return int(point[0]) + 1
|
||||
@@ -1,15 +1,15 @@
|
||||
"""Probe tree-sitter and Joern like the Docker sandbox probe. Never required for inventory."""
|
||||
"""Probe semantic extra and Joern. Host CLI alone is not overlay-ready."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
from collections.abc import Callable
|
||||
from subprocess import CompletedProcess
|
||||
|
||||
TREE_SITTER_PROBE_ENV = "OPENULTRASAST_TREE_SITTER_PROBE"
|
||||
from .extra import has_semantic_extra
|
||||
|
||||
JOERN_PROBE_ENV = "OPENULTRASAST_JOERN_PROBE"
|
||||
PROBE_TIMEOUT_SECONDS = 5
|
||||
_PROBE_OFF = frozenset({"0", "off", "false", "unavailable", "no"})
|
||||
@@ -17,22 +17,9 @@ _PROBE_ON = frozenset({"1", "on", "true", "available", "yes"})
|
||||
|
||||
|
||||
def tree_sitter_available(*, runner: Callable[..., CompletedProcess[str]] | None = None) -> bool:
|
||||
flag = os.environ.get(TREE_SITTER_PROBE_ENV, "").strip().lower()
|
||||
if flag in _PROBE_OFF:
|
||||
return False
|
||||
if flag in _PROBE_ON:
|
||||
return True
|
||||
if _python_tree_sitter_present():
|
||||
return True
|
||||
binary = shutil.which("tree-sitter")
|
||||
if binary is None:
|
||||
return False
|
||||
run = runner if runner is not None else subprocess.run
|
||||
try:
|
||||
result = run([binary, "--version"], capture_output=True, text=True, timeout=PROBE_TIMEOUT_SECONDS, check=False)
|
||||
return result.returncode == 0
|
||||
except Exception:
|
||||
return False
|
||||
"""True when a grammar extra can produce overlay IR. Host CLI is not sufficient."""
|
||||
del runner
|
||||
return has_semantic_extra()
|
||||
|
||||
|
||||
def joern_available(*, runner: Callable[..., CompletedProcess[str]] | None = None) -> bool:
|
||||
@@ -58,7 +45,3 @@ def joern_flow(path: str, text: str, language: str) -> tuple[object, ...]:
|
||||
if not joern_available():
|
||||
return ()
|
||||
return ()
|
||||
|
||||
|
||||
def _python_tree_sitter_present() -> bool:
|
||||
return importlib.util.find_spec("tree_sitter") is not None
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
"""Optional semantic extra: lazy grammar load. Never import wheels at package load."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import importlib.util
|
||||
import os
|
||||
from typing import Any
|
||||
|
||||
_GRAMMAR_MODULES = {
|
||||
"python": "tree_sitter_python",
|
||||
"javascript": "tree_sitter_javascript",
|
||||
"c": "tree_sitter_c",
|
||||
"cpp": "tree_sitter_cpp",
|
||||
"java": "tree_sitter_java",
|
||||
}
|
||||
_PROBE_ENV = "OPENULTRASAST_TREE_SITTER_PROBE"
|
||||
_PROBE_OFF = frozenset({"0", "off", "false", "unavailable", "no"})
|
||||
_PROBE_ON = frozenset({"1", "on", "true", "available", "yes"})
|
||||
|
||||
|
||||
def has_semantic_extra() -> bool:
|
||||
"""True when the parser runtime and at least one in-scope grammar are importable."""
|
||||
flag = os.environ.get(_PROBE_ENV, "").strip().lower()
|
||||
if flag in _PROBE_OFF:
|
||||
return False
|
||||
if importlib.util.find_spec("tree_sitter") is None:
|
||||
return False
|
||||
return any(importlib.util.find_spec(module) is not None for module in _GRAMMAR_MODULES.values())
|
||||
|
||||
|
||||
def grammar_for(language: str) -> Any | None:
|
||||
"""Return a tree_sitter.Language for an in-scope language, or None. Never raises into the scan."""
|
||||
flag = os.environ.get(_PROBE_ENV, "").strip().lower()
|
||||
if flag in _PROBE_OFF:
|
||||
return None
|
||||
module_name = _GRAMMAR_MODULES.get(language)
|
||||
if module_name is None or importlib.util.find_spec("tree_sitter") is None:
|
||||
return None
|
||||
if importlib.util.find_spec(module_name) is None:
|
||||
return None
|
||||
try:
|
||||
from tree_sitter import Language
|
||||
|
||||
module = importlib.import_module(module_name)
|
||||
return Language(module.language())
|
||||
except Exception:
|
||||
return None
|
||||
@@ -3,22 +3,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import ast
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
from .engines import tree_sitter_available
|
||||
|
||||
_EXTENSION = {
|
||||
"python": ".py",
|
||||
"javascript": ".js",
|
||||
"typescript": ".ts",
|
||||
"c": ".c",
|
||||
"cpp": ".cpp",
|
||||
"java": ".java",
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -62,10 +47,11 @@ class FileIR:
|
||||
|
||||
|
||||
def parse_file(path: str, text: str, language: str) -> FileIR:
|
||||
if tree_sitter_available():
|
||||
parsed = parse_tree_sitter_cli(path, text, language)
|
||||
if parsed is not None and parsed.parse_ok:
|
||||
return parsed
|
||||
from .cst import parse_with_cst
|
||||
|
||||
parsed = parse_with_cst(path, text, language)
|
||||
if parsed is not None:
|
||||
return parsed
|
||||
if language == "python":
|
||||
return parse_python_ast(path, text)
|
||||
return FileIR(path=path, language=language, engine="none", functions=(), parse_ok=False, reason="language_unsupported")
|
||||
@@ -80,37 +66,6 @@ def parse_python_ast(path: str, text: str) -> FileIR:
|
||||
return FileIR(path=path, language="python", engine="python-ast", functions=tuple(functions), parse_ok=True)
|
||||
|
||||
|
||||
def parse_tree_sitter_cli(path: str, text: str, language: str) -> FileIR | None:
|
||||
binary = shutil.which("tree-sitter")
|
||||
if binary is None:
|
||||
return None
|
||||
suffix = _EXTENSION.get(language, ".txt")
|
||||
tmp_path: str | None = None
|
||||
try:
|
||||
with tempfile.NamedTemporaryFile(suffix=suffix, delete=False) as handle:
|
||||
handle.write(text.encode())
|
||||
tmp_path = handle.name
|
||||
result = subprocess.run(
|
||||
[binary, "parse", tmp_path],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=5,
|
||||
check=False,
|
||||
)
|
||||
except Exception:
|
||||
return None
|
||||
finally:
|
||||
if tmp_path:
|
||||
Path(tmp_path).unlink(missing_ok=True)
|
||||
combined = f"{result.stdout or ''}{result.stderr or ''}"
|
||||
if result.returncode != 0 or "No language found" in combined or "Failed to load language" in combined:
|
||||
return None
|
||||
# A successful parse without an extractor still cannot adjudicate JS/C/Java.
|
||||
# Return a parse_ok IR only when we actually built functions; otherwise None
|
||||
# so Python can fall back to ast and other languages stay unsupported.
|
||||
return None
|
||||
|
||||
|
||||
def _python_functions(tree: ast.AST) -> list[FunctionIR]:
|
||||
functions: list[FunctionIR] = []
|
||||
module_binds, module_calls = _statements(getattr(tree, "body", []))
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
"""Extra-on CST walker and overlay. Skips when wheels are absent."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from openultrasast.findings import quick_scan_findings
|
||||
from openultrasast.preprocess import preprocess_repository
|
||||
from openultrasast.rank import rank_targets
|
||||
from openultrasast.semantic import adjudicate, has_semantic_extra
|
||||
from openultrasast.semantic.ir import parse_file
|
||||
|
||||
pytestmark = pytest.mark.semantic
|
||||
|
||||
|
||||
def _require_extra() -> None:
|
||||
if not has_semantic_extra():
|
||||
pytest.skip("requires openultrasast[semantic]")
|
||||
|
||||
|
||||
def test_javascript_eval_query_is_tree_sitter_call() -> None:
|
||||
_require_extra()
|
||||
ir = parse_file("app.js", "eval(req.query.x)\n", "javascript")
|
||||
assert ir.parse_ok is True
|
||||
assert ir.engine == "tree-sitter"
|
||||
calls = [call for function in ir.functions for call in function.calls]
|
||||
eval_calls = [call for call in calls if call.name == "eval"]
|
||||
assert eval_calls
|
||||
assert any("req.query" in arg for arg in eval_calls[0].arg_texts)
|
||||
|
||||
|
||||
def test_broken_javascript_is_parse_failed() -> None:
|
||||
_require_extra()
|
||||
ir = parse_file("app.js", "eval(req.query.x\n", "javascript")
|
||||
assert ir.parse_ok is False
|
||||
assert ir.reason == "parse_failed"
|
||||
|
||||
|
||||
def test_javascript_eval_promotes(tmp_path: Path) -> None:
|
||||
_require_extra()
|
||||
root = tmp_path / "repo"
|
||||
root.mkdir()
|
||||
(root / "app.js").write_text("eval(req.query.x)\n")
|
||||
_, targets = preprocess_repository(root)
|
||||
findings = quick_scan_findings(root, targets, rank_targets(targets))
|
||||
records = adjudicate(root=root, targets=targets, findings=findings)
|
||||
eval_records = [record for record in records if "js-eval" in record.proposal_id]
|
||||
assert eval_records
|
||||
assert eval_records[0].disposition == "promote"
|
||||
assert eval_records[0].evidence_level == "static_corroboration"
|
||||
assert eval_records[0].engine == "tree-sitter"
|
||||
|
||||
|
||||
def test_c_constant_system_demotes(tmp_path: Path) -> None:
|
||||
_require_extra()
|
||||
root = tmp_path / "repo"
|
||||
root.mkdir()
|
||||
(root / "run.c").write_text('void f(void) { system("ls"); }\n')
|
||||
_, targets = preprocess_repository(root)
|
||||
findings = quick_scan_findings(root, targets, rank_targets(targets))
|
||||
records = adjudicate(root=root, targets=targets, findings=findings)
|
||||
hits = [record for record in records if record.path == "run.c"]
|
||||
assert hits
|
||||
assert all(record.disposition != "unadjudicated" or record.reason != "language_unsupported" for record in hits)
|
||||
demoted = [record for record in hits if record.disposition == "demote"]
|
||||
assert demoted
|
||||
assert demoted[0].dominating_fact == "constant"
|
||||
assert demoted[0].evidence_level == "static_corroboration"
|
||||
@@ -0,0 +1,71 @@
|
||||
"""Semantic extra packaging and extra-free overlay parse."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import tomllib
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from openultrasast.cli import main
|
||||
from openultrasast.semantic.extra import grammar_for, has_semantic_extra
|
||||
from openultrasast.semantic.ir import parse_file
|
||||
|
||||
|
||||
def test_core_dependencies_stay_empty_and_semantic_is_optional() -> None:
|
||||
payload = tomllib.loads(Path("pyproject.toml").read_text())
|
||||
assert payload["project"]["dependencies"] == []
|
||||
extra = payload["project"]["optional-dependencies"]["semantic"]
|
||||
joined = " ".join(extra)
|
||||
assert "tree-sitter" in joined
|
||||
assert "tree-sitter-javascript" in joined
|
||||
assert "tree-sitter-c" in joined
|
||||
assert "tree-sitter-java" in joined
|
||||
assert "semantic:" in " ".join(payload["tool"]["pytest"]["ini_options"]["markers"])
|
||||
|
||||
|
||||
def test_cli_import_does_not_load_tree_sitter(assert_cold_of_harnessx) -> None: # type: ignore[no-untyped-def]
|
||||
proc_code = (
|
||||
"import sys\n"
|
||||
"import openultrasast.cli\n"
|
||||
"leaked = sorted(m for m in sys.modules if m == 'tree_sitter' or m.startswith('tree_sitter_'))\n"
|
||||
"assert not leaked, leaked\n"
|
||||
)
|
||||
assert_cold_of_harnessx(proc_code)
|
||||
|
||||
|
||||
def test_probe_off_yields_no_javascript_grammar(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setenv("OPENULTRASAST_TREE_SITTER_PROBE", "0")
|
||||
assert has_semantic_extra() is False
|
||||
assert grammar_for("javascript") is None
|
||||
ir = parse_file("app.js", "eval(req.query.x)\n", "javascript")
|
||||
assert ir.parse_ok is False
|
||||
assert ir.reason == "language_unsupported"
|
||||
|
||||
|
||||
def test_standard_javascript_is_unadjudicated_without_extra(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setenv("OPENULTRASAST_TREE_SITTER_PROBE", "0")
|
||||
monkeypatch.setenv("OPENULTRASAST_RUNS_DIR", ".runs")
|
||||
repo = tmp_path / "repo"
|
||||
repo.mkdir()
|
||||
(repo / "app.js").write_text("eval(req.query.x)\n")
|
||||
assert main(["scan", str(repo), "--mode", "standard"]) == 0
|
||||
run_dir = sorted((repo / ".runs").iterdir())[-1]
|
||||
overlay = json.loads((run_dir / "overlay.json").read_text())["records"]
|
||||
assert overlay
|
||||
assert all(item["disposition"] == "unadjudicated" for item in overlay)
|
||||
assert all(item["reason"] in {"language_unsupported", "parse_failed"} for item in overlay)
|
||||
|
||||
|
||||
def test_quick_scan_without_extra_still_inventories(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setenv("OPENULTRASAST_TREE_SITTER_PROBE", "0")
|
||||
monkeypatch.setenv("OPENULTRASAST_RUNS_DIR", ".runs")
|
||||
repo = tmp_path / "repo"
|
||||
repo.mkdir()
|
||||
(repo / "app.py").write_text("def run():\n return eval(request.args)\n")
|
||||
assert main(["scan", str(repo), "--mode", "quick"]) == 0
|
||||
run_dir = sorted((repo / ".runs").iterdir())[-1]
|
||||
findings = json.loads((run_dir / "findings.json").read_text())["findings"]
|
||||
assert findings
|
||||
assert not (run_dir / "overlay.json").exists()
|
||||
@@ -13,7 +13,7 @@ from openultrasast.rank import rank_targets
|
||||
from openultrasast.semantic import OverlayError, OverlayRecord, adjudicate, load_facts
|
||||
from openultrasast.semantic.engines import joern_available, tree_sitter_available
|
||||
from openultrasast.semantic.facts import FactLoadError
|
||||
from openultrasast.semantic.ir import FileIR, FunctionIR, parse_file
|
||||
from openultrasast.semantic.ir import parse_file
|
||||
from openultrasast.semantic.taint import TaintPath, analyze_file
|
||||
|
||||
|
||||
@@ -186,29 +186,13 @@ def test_overlay_length_matches_proposals_plus_coverage_only(tmp_path: Path) ->
|
||||
assert all(record.evidence_level == "static_corroboration" for record in records if record.disposition == "coverage")
|
||||
|
||||
|
||||
def test_js_language_unsupported_without_tree_sitter(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setattr("openultrasast.semantic.ir.tree_sitter_available", lambda: False)
|
||||
monkeypatch.setattr("openultrasast.semantic.engines.tree_sitter_available", lambda: False)
|
||||
def test_js_language_unsupported_without_semantic_extra(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setenv("OPENULTRASAST_TREE_SITTER_PROBE", "0")
|
||||
ir = parse_file("app.js", "eval(req.query.x)\n", "javascript")
|
||||
assert ir.parse_ok is False
|
||||
assert ir.reason == "language_unsupported"
|
||||
|
||||
|
||||
def test_js_yields_ir_when_tree_sitter_available(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
fake = FileIR(
|
||||
path="app.js",
|
||||
language="javascript",
|
||||
engine="tree-sitter",
|
||||
functions=(FunctionIR(name="run", params=("x",), start_line=1, end_line=1, binds=(), calls=()),),
|
||||
parse_ok=True,
|
||||
)
|
||||
monkeypatch.setattr("openultrasast.semantic.ir.tree_sitter_available", lambda: True)
|
||||
monkeypatch.setattr("openultrasast.semantic.ir.parse_tree_sitter_cli", lambda *args, **kwargs: fake)
|
||||
ir = parse_file("app.js", "eval(req.query.x)\n", "javascript")
|
||||
assert ir.parse_ok is True
|
||||
assert ir.engine == "tree-sitter"
|
||||
|
||||
|
||||
def test_joern_absence_does_not_block_overlay(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setattr("openultrasast.semantic.overlay.joern_available", lambda: False)
|
||||
root = tmp_path / "repo"
|
||||
|
||||
@@ -2145,6 +2145,14 @@ source = { editable = "." }
|
||||
harnessx = [
|
||||
{ name = "harnessx" },
|
||||
]
|
||||
semantic = [
|
||||
{ name = "tree-sitter" },
|
||||
{ name = "tree-sitter-c" },
|
||||
{ name = "tree-sitter-cpp" },
|
||||
{ name = "tree-sitter-java" },
|
||||
{ name = "tree-sitter-javascript" },
|
||||
{ name = "tree-sitter-python" },
|
||||
]
|
||||
|
||||
[package.dev-dependencies]
|
||||
dev = [
|
||||
@@ -2156,8 +2164,16 @@ dev = [
|
||||
]
|
||||
|
||||
[package.metadata]
|
||||
requires-dist = [{ name = "harnessx", marker = "extra == 'harnessx'", git = "https://github.com/Darwin-Agent/HarnessX.git?rev=274c512eea67c64701f992d3b6fe06f0fd590ef0" }]
|
||||
provides-extras = ["harnessx"]
|
||||
requires-dist = [
|
||||
{ name = "harnessx", marker = "extra == 'harnessx'", git = "https://github.com/Darwin-Agent/HarnessX.git?rev=274c512eea67c64701f992d3b6fe06f0fd590ef0" },
|
||||
{ name = "tree-sitter", marker = "extra == 'semantic'", specifier = ">=0.23,<0.27" },
|
||||
{ name = "tree-sitter-c", marker = "extra == 'semantic'", specifier = ">=0.23" },
|
||||
{ name = "tree-sitter-cpp", marker = "extra == 'semantic'", specifier = ">=0.23" },
|
||||
{ name = "tree-sitter-java", marker = "extra == 'semantic'", specifier = ">=0.23" },
|
||||
{ name = "tree-sitter-javascript", marker = "extra == 'semantic'", specifier = ">=0.23" },
|
||||
{ name = "tree-sitter-python", marker = "extra == 'semantic'", specifier = ">=0.23" },
|
||||
]
|
||||
provides-extras = ["harnessx", "semantic"]
|
||||
|
||||
[package.metadata.requires-dev]
|
||||
dev = [
|
||||
@@ -3335,6 +3351,124 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/d8/8e/bb97bb0c71802080bfc8952937d174e49cfc50de5c951dd47b2496f0dcdb/tqdm-4.68.3-py3-none-any.whl", hash = "sha256:39832cc2def2789a6f29df83f172db7416cea70052c0907a57801c5f2fdccb03", size = 78337, upload-time = "2026-06-17T07:36:50.132Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tree-sitter"
|
||||
version = "0.26.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/f7/03/5600b84aff2e6c4fe80cfebb4063fe2f50299521befe5f6092ab8c082f4a/tree_sitter-0.26.0.tar.gz", hash = "sha256:b40c219edccc4564530c96f8f1556f6202b37cda964d1cbd7bd2b7e68b40a245", size = 191423, upload-time = "2026-06-30T12:14:27.933Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/41/18/78aae7e4b5a36daaebb0276e4b07d084d45298758000787838e89329e11f/tree_sitter-0.26.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:1d6fe0e8fb4df77b5ee816228e2c4475a63d8cc1d4d3a7ffd7097b2b87fc3e95", size = 148679, upload-time = "2026-06-30T12:13:52.27Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/24/e4/b371b9553b0e47d130fc2073e56cab94fecc868be04666bf5bbd1fcd1cc9/tree_sitter-0.26.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:514a9bf8993e5210e7970736aaf6020d1759b670e195ef17b1c48f586aa30736", size = 140759, upload-time = "2026-06-30T12:13:53.221Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/22/7d/266fb0f2c41e6fb00b0f40e7a3338cdf99651e6a6511ca72bc78fc697636/tree_sitter-0.26.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:10f0d4eb94aa7242dcb7f554bcd24dd7ba1c114f00d58759ba08c7a46c8ec51a", size = 637206, upload-time = "2026-06-30T12:13:54.334Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/40/9f/47cf22febb47132d5b3a507a27bb99ef89fe5c8ec420a13c6daa9b64f782/tree_sitter-0.26.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:335294ce0504fcefde5245dff596778ffaf820205b98ae0b549c72e48855f1d8", size = 664758, upload-time = "2026-06-30T12:13:55.42Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/4c/4d/8d144ca3beb46a62a5102b6deac76bb0da55235c2c7840faf3b12f2e9d97/tree_sitter-0.26.0-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:f9997ba61368c48ed54e715676afadf703947a1542464e39d047764fb3624b01", size = 647438, upload-time = "2026-06-30T12:13:56.523Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/4d/ed/ed1d6e78520c4fb64ed52fec3f2947bf8c1fbad7bc24e282c56193c9ba42/tree_sitter-0.26.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:c56581ad256c4195a21bfe449fed5d44a02fe83a4a7d6e70e6ec302c881191c7", size = 661944, upload-time = "2026-06-30T12:13:57.82Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/10/83/45f5bd43db1b8248d2fd08ef6cbe43e2725c539e09a2cfb8bc2818646788/tree_sitter-0.26.0-cp311-cp311-win_amd64.whl", hash = "sha256:0f8793fd18ad7eec276ed4b51c097b4bf2002b357259b66b0d75db1f3f41c754", size = 129496, upload-time = "2026-06-30T12:13:59.216Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/f1/8d/be68e6c04563eb54145424cc83fe0aa8b0ba6c90d8989cf8a032671b5f16/tree_sitter-0.26.0-cp311-cp311-win_arm64.whl", hash = "sha256:dea4b4e27d49e9ec5b785d4f994da000e6726882fcc6ad05ec98478500c71aef", size = 116484, upload-time = "2026-06-30T12:14:00.147Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/87/ca/565702c44815393e3a973552ad546db4e5ca081ca8698640b4e93d809f51/tree_sitter-0.26.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:6cb2bd20efb2544c19ac54486ab7cb8ec7b36f913bbe1ce95df84acb96743d9c", size = 148934, upload-time = "2026-06-30T12:14:01.188Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/54/6f/8bb61957f16ec1b1d92410a006cdc84a952b6352a7313b2ad299f2d21484/tree_sitter-0.26.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:918d89529786873f0982a0f59c2a303cd065fbfd1b903d71a8e4e1584f67b42e", size = 140820, upload-time = "2026-06-30T12:14:02.087Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/78/0a/8a6f08559182643a814a4ab559948ae817b2851890fd9b995a4fff6541ce/tree_sitter-0.26.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:30a88be89ff1f2755297f81e8080d88b795dd98720c3f9fa2acf93873182cc95", size = 638844, upload-time = "2026-06-30T12:14:03.428Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/8a/2f/6e6781b31677231366cb3cf27bc8269157f6d4b03c9032865a4f5f2bbe7e/tree_sitter-0.26.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5a6b333b0282d8bb0af741f9b018bd2523d4eecb2686bf6717066a625fecfaa4", size = 667487, upload-time = "2026-06-30T12:14:04.669Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/02/0b/0483078c8567445557a7015b0e5b187f6d7d4fda73464df9c4bdea7f7f3c/tree_sitter-0.26.0-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:3f3c44339dd34fe8eb2b8d5aa7610660499a795f70376b130bbee7a437337280", size = 647975, upload-time = "2026-06-30T12:14:05.797Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/27/68/da83ca72c984e96ab4eb3bee0db1a6ffb5de1c8c455f92bd9f420cde7f0e/tree_sitter-0.26.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:94550e13b6ae576969da40246f4c4abb206380b5375ad43f26dd9151d55438e3", size = 665018, upload-time = "2026-06-30T12:14:07.278Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/d1/36/4d67927fd47b89af4a00f65f55a7370e28778cd50e972c2430487e3ecc27/tree_sitter-0.26.0-cp312-cp312-win_amd64.whl", hash = "sha256:ca89e361a276dbc934b28a43dd881199e25d34ff5493ee0ce45f3c52a6124a37", size = 129619, upload-time = "2026-06-30T12:14:08.373Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/ed/72/cdefad523eb78710679c6da6a79e3d90f5afd32b1c6aa5a17bac7eef99f6/tree_sitter-0.26.0-cp312-cp312-win_arm64.whl", hash = "sha256:bc6cb01d5ee75c85424aa1f1c72a82d8f07fd52539a0f3c4a6ed3e8721079b84", size = 116545, upload-time = "2026-06-30T12:14:09.273Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/cb/b0/465257cf8f972ad9f9812ec1cbaa8ec210ebebb601ade9a15881aa2436b4/tree_sitter-0.26.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:ed0889dbed843ce45ede9f5169c0b2dea2222f12685844a03fadb81f12705867", size = 148893, upload-time = "2026-06-30T12:14:10.541Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/a1/ec/19d093e854b45e807fecfdd26105c266f43aeecc39c4dc97992a7074ad5a/tree_sitter-0.26.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:6189c6c340c7384357711e3d92645e96bfb79f7a502f86de1ebdb23eb43f7dab", size = 140829, upload-time = "2026-06-30T12:14:11.626Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/9b/ee/87e74671ed63a837e7a1f17ab94aa3913871e033b27523d8e7b83d6f7ad0/tree_sitter-0.26.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8ff2e0750b7daa722302838356d7b65e303829b7eb73c915df127ddba115e1d1", size = 639334, upload-time = "2026-06-30T12:14:12.836Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/66/e7/f7e04cd9dff6b6ac0adf23922796fbc76accd4cf4bcda50542748d485679/tree_sitter-0.26.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7075ef857ef86f327dbb72d1e2574dda78db5754b3a1fca6506acd7fe5d561a7", size = 668102, upload-time = "2026-06-30T12:14:14.035Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/d3/90/0bfb16b7894fea728c774a89d5af421a9368a2f913bbd4e8dcab7caaecfb/tree_sitter-0.26.0-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:26c996c1edfee86e977bb3f5462e74fcec0d0b0db1e85a3c475875763caa03be", size = 648560, upload-time = "2026-06-30T12:14:15.302Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/cd/e6/0fe05ba396e9623b0ae40ccf34171336b8701ec8d7bd0ee9f5224d638665/tree_sitter-0.26.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:00289bfe7978f3e0dc0ce69813a20fa9f44ea4c100b3ec62043e5eb74ccfc3a2", size = 665121, upload-time = "2026-06-30T12:14:16.403Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/eb/d2/a944b1ca35bed6068dc84a9967aaf3049d8cc0b7a36179eea8787270a6ab/tree_sitter-0.26.0-cp313-cp313-win_amd64.whl", hash = "sha256:93e220cab7e6a823efeb2046c49171427de92ef71c7c681c01820d14d8d3721f", size = 129615, upload-time = "2026-06-30T12:14:17.463Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/09/ef/c7ca48293580d2249f36940c4eed5b4ddeb9ce75baf9a4ef30621987e0c7/tree_sitter-0.26.0-cp313-cp313-win_arm64.whl", hash = "sha256:b31a8195d2f224224c530ac814632d98c1dcc123d227442c07c736e86b70d564", size = 116525, upload-time = "2026-06-30T12:14:18.53Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/c5/7a/4d84e6f6ae2c3e757490dd84de251712c31e293dfe31f28da1ec019cefa2/tree_sitter-0.26.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:5a3c93a352b7e6f70f73e121bbfa2d0117ba7478bd51114ed35c91b0b78814fa", size = 148901, upload-time = "2026-06-30T12:14:19.452Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/b0/d9/efe62ec65dc9d096e834d27b8c058127e2146e42ff3380b822a233f016a6/tree_sitter-0.26.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:5fc2f41bf246ff2f70a9cc3690be35ec7580a4923151873d898c8bcb1a4503d3", size = 140805, upload-time = "2026-06-30T12:14:20.478Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/c4/2c/c82326b7b97e3c485c18679883b16f89e5e913c639d3b219d3da70c9e67e/tree_sitter-0.26.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b8ea92a255c91671a7ec4625aba3ab7bb5220c423630ffbf83c45d7312abe084", size = 640586, upload-time = "2026-06-30T12:14:21.527Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/e2/7a/f56e7d8282859452611024c7cbc623bfba5b24b8cb9b8f8bc88c5219fe9a/tree_sitter-0.26.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f665510f0fcf4636fb9696f1f7853bed7a3bd764b7bb0cb8494e619c14ed5a0c", size = 668300, upload-time = "2026-06-30T12:14:22.728Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/91/51/240ee81b9d5e9ca0a6cb1528e8605ffa70ab58c89ce126631be96d3e4bae/tree_sitter-0.26.0-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:253df7ab82cc0a9d311cd65f06e9f99fb3eac55996ae9fc94da22f123a861b90", size = 649627, upload-time = "2026-06-30T12:14:23.819Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/6a/54/760035cefedf9eb44f0f84c4ac22f1322e73155853e272576ee876336312/tree_sitter-0.26.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:ff80d4833d330a73184a3ac5132abe93c575d2dea31975c6f15c0d21fef238aa", size = 664885, upload-time = "2026-06-30T12:14:25.064Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/c9/1b/0b36fe2a984ecedc4ce6aefd5d56447a6626a8e9b595c4e48658510ce8f8/tree_sitter-0.26.0-cp314-cp314-win_amd64.whl", hash = "sha256:a4033fecc8f606c7f2e8b8014d0057b74668a7f0152763606f7bc25c5f9ec64c", size = 132688, upload-time = "2026-06-30T12:14:26.106Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/4d/74/ebc041a13fbf40144afdb0d4b447e48e0b4012ca866c63de8b48f801f0c1/tree_sitter-0.26.0-cp314-cp314-win_arm64.whl", hash = "sha256:823251c4b6725a7c03ed497a339135ede7ae4bdde75bb8be7ef5e305aeb4ff52", size = 120287, upload-time = "2026-06-30T12:14:26.991Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tree-sitter-c"
|
||||
version = "0.24.2"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/a6/c9/3834f3d9278251aea7312274971bc4c45b17aec2490fd4b884d93bd7019a/tree_sitter_c-0.24.2.tar.gz", hash = "sha256:1628584df0299b5a340aa63f8e67b6c97c91517f52fa7e7a4c557e40adb330a9", size = 228397, upload-time = "2026-04-22T08:06:14.491Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/28/c1/26ed17730ec2c17bedc1b673349e5e0a466c578e3eb0327c3b73cf52bf97/tree_sitter_c-0.24.2-cp310-abi3-macosx_10_9_x86_64.whl", hash = "sha256:4d4579a8b54f0a442f903d88d3304cab77cd5c2031d4015baa4f2f8e15d6dcb7", size = 81016, upload-time = "2026-04-22T08:06:07.208Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/c1/1c/1140db75e7e375cda3c68792a33826c4fd40b5b98c3259d93c75f6c8368f/tree_sitter_c-0.24.2-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:97bc80a224d48215d4e6e6376bf30d114f4c317b8145ff1b02afe785d4ba7bdd", size = 86213, upload-time = "2026-04-22T08:06:08.136Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/e9/8c/0dfb88d726f8821d1c4c36042f092be974a800afd734307a595b8604190c/tree_sitter_c-0.24.2-cp310-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:5041ef67eb68ce6bc8bb0b1f8ef3a5585ce523dae0c7eec109ab0627dd75aede", size = 94264, upload-time = "2026-04-22T08:06:08.918Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/87/78/47dc570e7aee6b0a1ecc2520b30639cc2b06003154c9ab0672d86bf720d5/tree_sitter_c-0.24.2-cp310-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c098bedcd5ac86ff93fa734d51d1dd86aed40fd5ed7d634c7af11380a0469969", size = 94560, upload-time = "2026-04-22T08:06:09.852Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/29/37/75d59d3f74f4cfc00f04472917e933d8a9c9fdc6eff980ef9552e010e6aa/tree_sitter_c-0.24.2-cp310-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:82842c5a5f2acd93f4de10038c33ac179c8979defc39376f990348d6289e933b", size = 94023, upload-time = "2026-04-22T08:06:10.682Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/64/57/8fc655d5a446a70a637e92b98bd2fdaab88bf5bb5b36076ac4add544808d/tree_sitter_c-0.24.2-cp310-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:e2b42e8e22202c251f8629306f9321233542e07a6e01611b5fe83489272143eb", size = 94160, upload-time = "2026-04-22T08:06:11.497Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/c1/f7/72a1d6b42dd31fd37e03ff67e7dc5ee572301499e6b216002b8dd42a1714/tree_sitter_c-0.24.2-cp310-abi3-win_amd64.whl", hash = "sha256:abb549225091f7b25df2dd3a0143ece6e208f7055d8bcb4700b41ee79b9ef1e1", size = 84669, upload-time = "2026-04-22T08:06:12.347Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/e2/9d/7475d9ae8ef679aa36c7dfe6c903ab78e573651c68b6ef9862d6a3f994db/tree_sitter_c-0.24.2-cp310-abi3-win_arm64.whl", hash = "sha256:4a2f4371cd816cc3153458f69062135ebb2ea5f275ddd90494e5c823d778204a", size = 82956, upload-time = "2026-04-22T08:06:13.364Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tree-sitter-cpp"
|
||||
version = "0.23.4"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/20/2c/4dd63d705a8933543cad9b92ff31be849b164fec91a6eb63475ebc9ce668/tree_sitter_cpp-0.23.4.tar.gz", hash = "sha256:6a59c4cebb1ad1dc2e8d586cf8a72b39d21b8108b7b139d089719e81a339e41d", size = 940358, upload-time = "2024-11-11T06:59:24.934Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/b6/ac/11d56670f7b048362db872ca866fd00ba2002a322ab179f047b7c0fb2910/tree_sitter_cpp-0.23.4-cp39-abi3-macosx_10_9_x86_64.whl", hash = "sha256:aacb1759f0efd9dbc25bd8ee88184a340483018869f75412d9c3bc32c039a520", size = 287861, upload-time = "2024-11-11T06:59:15.005Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/12/1c/0337c016bdc00a77a3326d12f10ee836401dd28f27db6fd5b7734bfb21ed/tree_sitter_cpp-0.23.4-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:bc3c404d9f0cbd87951213a85440afbf4c31e718f8d907fa9ee12bea4b8d276f", size = 315513, upload-time = "2024-11-11T06:59:16.679Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/b3/7b/dd38c049b10ed7fda118b903a1d28a8b55a36b98c30606ef90e8f374c6de/tree_sitter_cpp-0.23.4-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ccc43ddf1279d5d5a4ef190373f4cb16522801bec4492bcd4754edf2aeba2b7b", size = 334813, upload-time = "2024-11-11T06:59:18.253Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/6a/4d/23e390234d2acd351f5563b1079c515d7c1fe13ddb7392cee543be74dda3/tree_sitter_cpp-0.23.4-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:773d2cafc08bbc0f998687fa33f42f378c1a371cdb582870c4d13abb06092706", size = 316110, upload-time = "2024-11-11T06:59:19.823Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/32/c7/b94a7e0e803af9d3bd4608fb4f0cfb2e9e233abaf0a38c928bfb0b1a025d/tree_sitter_cpp-0.23.4-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:247d127f0eb6574b0f6b30c0151e0bd0774e2e7acf9c558bdf9fbb8adc2e80c0", size = 308242, upload-time = "2024-11-11T06:59:21.466Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/37/7e/909e52b3dec09c475140b0e175511e275d0d00ba2dbd7c68102d377ae0f6/tree_sitter_cpp-0.23.4-cp39-abi3-win_amd64.whl", hash = "sha256:68606a45bea92669d155399e1239f771a7767d8683cd8f8e30e7d813107030ca", size = 290997, upload-time = "2024-11-11T06:59:22.432Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/d4/6a/65435d4d1f4c735be7ffe52d7c2e7b8a7f7c2790343a2719c60c548611c8/tree_sitter_cpp-0.23.4-cp39-abi3-win_arm64.whl", hash = "sha256:712f84f18be94cbe2a148fa4fdf40fcf4a8c25a8f7670efb9f8a47ddec2fc281", size = 288203, upload-time = "2024-11-11T06:59:23.404Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tree-sitter-java"
|
||||
version = "0.23.5"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/fa/dc/eb9c8f96304e5d8ae1663126d89967a622a80937ad2909903569ccb7ec8f/tree_sitter_java-0.23.5.tar.gz", hash = "sha256:f5cd57b8f1270a7f0438878750d02ccc79421d45cca65ff284f1527e9ef02e38", size = 138121, upload-time = "2024-12-21T18:24:26.936Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/67/21/b3399780b440e1567a11d384d0ebb1aea9b642d0d98becf30fa55c0e3a3b/tree_sitter_java-0.23.5-cp39-abi3-macosx_10_9_x86_64.whl", hash = "sha256:355ce0308672d6f7013ec913dee4a0613666f4cda9044a7824240d17f38209df", size = 58926, upload-time = "2024-12-21T18:24:12.53Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/57/ef/6406b444e2a93bc72a04e802f4107e9ecf04b8de4a5528830726d210599c/tree_sitter_java-0.23.5-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:24acd59c4720dedad80d548fe4237e43ef2b7a4e94c8549b0ca6e4c4d7bf6e69", size = 62288, upload-time = "2024-12-21T18:24:14.634Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/4e/6c/74b1c150d4f69c291ab0b78d5dd1b59712559bbe7e7daf6d8466d483463f/tree_sitter_java-0.23.5-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:9401e7271f0b333df39fc8a8336a0caf1b891d9a2b89ddee99fae66b794fc5b7", size = 85533, upload-time = "2024-12-21T18:24:16.695Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/29/09/e0d08f5c212062fd046db35c1015a2621c2631bc8b4aae5740d7adb276ad/tree_sitter_java-0.23.5-cp39-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:370b204b9500b847f6d0c5ad584045831cee69e9a3e4d878535d39e4a7e4c4f1", size = 84033, upload-time = "2024-12-21T18:24:18.758Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/43/56/7d06b23ddd09bde816a131aa504ee11a1bbe87c6b62ab9b2ed23849a3382/tree_sitter_java-0.23.5-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:aae84449e330363b55b14a2af0585e4e0dae75eb64ea509b7e5b0e1de536846a", size = 82564, upload-time = "2024-12-21T18:24:20.493Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/da/d6/0528c7e1e88a18221dbd8ccee3825bf274b1fa300f745fd74eb343878043/tree_sitter_java-0.23.5-cp39-abi3-win_amd64.whl", hash = "sha256:1ee45e790f8d31d416bc84a09dac2e2c6bc343e89b8a2e1d550513498eedfde7", size = 60650, upload-time = "2024-12-21T18:24:22.902Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/72/57/5bab54d23179350356515526fff3cc0f3ac23bfbc1a1d518a15978d4880e/tree_sitter_java-0.23.5-cp39-abi3-win_arm64.whl", hash = "sha256:402efe136104c5603b429dc26c7e75ae14faaca54cfd319ecc41c8f2534750f4", size = 59059, upload-time = "2024-12-21T18:24:24.934Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tree-sitter-javascript"
|
||||
version = "0.25.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/59/e0/e63103c72a9d3dfd89a31e02e660263ad84b7438e5f44ee82e443e65bbde/tree_sitter_javascript-0.25.0.tar.gz", hash = "sha256:329b5414874f0588a98f1c291f1b28138286617aa907746ffe55adfdcf963f38", size = 132338, upload-time = "2025-09-01T07:13:44.792Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/2c/df/5106ac250cd03661ebc3cc75da6b3d9f6800a3606393a0122eca58038104/tree_sitter_javascript-0.25.0-cp310-abi3-macosx_10_9_x86_64.whl", hash = "sha256:b70f887fb269d6e58c349d683f59fa647140c410cfe2bee44a883b20ec92e3dc", size = 64052, upload-time = "2025-09-01T07:13:36.865Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/b1/8f/6b4b2bc90d8ab3955856ce852cc9d1e82c81d7ab9646385f0e75ffd5b5d3/tree_sitter_javascript-0.25.0-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:8264a996b8845cfce06965152a013b5d9cbb7d199bc3503e12b5682e62bb1de1", size = 66440, upload-time = "2025-09-01T07:13:37.962Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/5f/c4/7da74ecdcd8a398f88bd003a87c65403b5fe0e958cdd43fbd5fd4a398fcf/tree_sitter_javascript-0.25.0-cp310-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:9dc04ba91fc8583344e57c1f1ed5b2c97ecaaf47480011b92fbeab8dda96db75", size = 99728, upload-time = "2025-09-01T07:13:38.755Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/96/c8/97da3af4796495e46421e9344738addb3602fa6426ea695be3fcbadbee37/tree_sitter_javascript-0.25.0-cp310-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:199d09985190852e0912da2b8d26c932159be314bc04952cf917ed0e4c633e6b", size = 106072, upload-time = "2025-09-01T07:13:39.798Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/13/be/c964e8130be08cc9bd6627d845f0e4460945b158429d39510953bbcb8fcc/tree_sitter_javascript-0.25.0-cp310-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:dfcf789064c58dc13c0a4edb550acacfc6f0f280577f1e7a00de3e89fc7f8ddc", size = 104388, upload-time = "2025-09-01T07:13:40.866Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/ee/89/9b773dee0f8961d1bb8d7baf0a204ab587618df19897c1ef260916f318ec/tree_sitter_javascript-0.25.0-cp310-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:1b852d3aee8a36186dbcc32c798b11b4869f9b5041743b63b65c2ef793db7a54", size = 98377, upload-time = "2025-09-01T07:13:41.838Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/3b/dc/d90cb1790f8cec9b4878d278ad9faf7c8f893189ce0f855304fd704fc274/tree_sitter_javascript-0.25.0-cp310-abi3-win_amd64.whl", hash = "sha256:e5ed840f5bd4a3f0272e441d19429b26eedc257abe5574c8546da6b556865e3c", size = 62975, upload-time = "2025-09-01T07:13:42.828Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/2e/1f/f9eba1038b7d4394410f3c0a6ec2122b590cd7acb03f196e52fa57ebbe72/tree_sitter_javascript-0.25.0-cp310-abi3-win_arm64.whl", hash = "sha256:622a69d677aa7f6ee2931d8c77c981a33f0ebb6d275aa9d43d3397c879a9bb0b", size = 61668, upload-time = "2025-09-01T07:13:43.803Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tree-sitter-python"
|
||||
version = "0.25.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/b8/8b/c992ff0e768cb6768d5c96234579bf8842b3a633db641455d86dd30d5dac/tree_sitter_python-0.25.0.tar.gz", hash = "sha256:b13e090f725f5b9c86aa455a268553c65cadf325471ad5b65cd29cac8a1a68ac", size = 159845, upload-time = "2025-09-11T06:47:58.159Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/cf/64/a4e503c78a4eb3ac46d8e72a29c1b1237fa85238d8e972b063e0751f5a94/tree_sitter_python-0.25.0-cp310-abi3-macosx_10_9_x86_64.whl", hash = "sha256:14a79a47ddef72f987d5a2c122d148a812169d7484ff5c75a3db9609d419f361", size = 73790, upload-time = "2025-09-11T06:47:47.652Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/e6/1d/60d8c2a0cc63d6ec4ba4e99ce61b802d2e39ef9db799bdf2a8f932a6cd4b/tree_sitter_python-0.25.0-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:480c21dbd995b7fe44813e741d71fed10ba695e7caab627fb034e3828469d762", size = 76691, upload-time = "2025-09-11T06:47:49.038Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/aa/cb/d9b0b67d037922d60cbe0359e0c86457c2da721bc714381a63e2c8e35eba/tree_sitter_python-0.25.0-cp310-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:86f118e5eecad616ecdb81d171a36dde9bef5a0b21ed71ea9c3e390813c3baf5", size = 108133, upload-time = "2025-09-11T06:47:50.499Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/40/bd/bf4787f57e6b2860f3f1c8c62f045b39fb32d6bac4b53d7a9e66de968440/tree_sitter_python-0.25.0-cp310-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:be71650ca2b93b6e9649e5d65c6811aad87a7614c8c1003246b303f6b150f61b", size = 110603, upload-time = "2025-09-11T06:47:51.985Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/5d/25/feff09f5c2f32484fbce15db8b49455c7572346ce61a699a41972dea7318/tree_sitter_python-0.25.0-cp310-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:e6d5b5799628cc0f24691ab2a172a8e676f668fe90dc60468bee14084a35c16d", size = 108998, upload-time = "2025-09-11T06:47:53.046Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/75/69/4946da3d6c0df316ccb938316ce007fb565d08f89d02d854f2d308f0309f/tree_sitter_python-0.25.0-cp310-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:71959832fc5d9642e52c11f2f7d79ae520b461e63334927e93ca46cd61cd9683", size = 107268, upload-time = "2025-09-11T06:47:54.388Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/ed/a2/996fc2dfa1076dc460d3e2f3c75974ea4b8f02f6bc925383aaae519920e8/tree_sitter_python-0.25.0-cp310-abi3-win_amd64.whl", hash = "sha256:9bcde33f18792de54ee579b00e1b4fe186b7926825444766f849bf7181793a76", size = 76073, upload-time = "2025-09-11T06:47:55.773Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/07/19/4b5569d9b1ebebb5907d11554a96ef3fa09364a30fcfabeff587495b512f/tree_sitter_python-0.25.0-cp310-abi3-win_arm64.whl", hash = "sha256:0fbf6a3774ad7e89ee891851204c2e2c47e12b63a5edbe2e9156997731c128bb", size = 74169, upload-time = "2025-09-11T06:47:56.747Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "typer"
|
||||
version = "0.25.1"
|
||||
|
||||
Reference in New Issue
Block a user