26 Commits
Author SHA1 Message Date
Marius Ciepluch 32aa92191a chore(release): 1.1.0
Three-stage scan: static inventory, complexity map with test-gap hints,
and Docker-isolated regression with --fail-on worth-fixing. Split-sink
and mutation corpora gate stage 2 so regex hit-count ranking cannot claim
success.
2026-09-04 12:00:15 +02:00
Marius Ciepluch ac3da41c95 feat(three-stage-scan): prove split-sink triggerable in sandbox 2026-09-04 11:54:01 +02:00
Marius Ciepluch 9af356797f feat(three-stage-scan): gate stage-2 maps against split-sink ranking 2026-09-04 11:53:06 +02:00
Marius Ciepluch 7398063b79 feat(three-stage-scan): add aliased-sink mutation the regex scan misses 2026-09-04 11:50:44 +02:00
Marius Ciepluch fe300e6352 feat(three-stage-scan): add split-sink fixtures the regex scan misses 2026-09-04 11:49:44 +02:00
Marius Ciepluch 818a0b10d3 feat(three-stage-scan): report map and honor fail-on worth-fixing 2026-09-04 00:18:33 +02:00
Marius Ciepluch a916e2d173 feat(three-stage-scan): attach tool hunter to map hotspots 2026-09-04 00:04:36 +02:00
Marius Ciepluch 2efbbb8024 feat(three-stage-scan): run or skip deep regression with visible degradation 2026-09-03 23:43:15 +02:00
Marius Ciepluch 803312e75b feat(three-stage-scan): run complexity map on standard scans 2026-09-03 23:26:54 +02:00
Marius Ciepluch 66d985e9e6 feat(three-stage-scan): keep quick as stage-1 inventory 2026-09-03 23:14:00 +02:00
Marius Ciepluch 331b62f1c1 feat(three-stage-scan): map sandbox outcomes to worth-fixing verdicts 2026-09-03 22:59:01 +02:00
Marius Ciepluch 62c7990e49 feat(three-stage-scan): add language recipes with inconclusive fallback 2026-09-03 22:48:10 +02:00
Marius Ciepluch 7a584d4578 feat(three-stage-scan): select bounded regress candidates with sev-5 extras 2026-09-03 22:36:02 +02:00
Marius Ciepluch eb87ef85ec feat(three-stage-scan): reject unsafe regression snippets before execution 2026-09-03 22:12:45 +02:00
Marius Ciepluch f78ea55e5d feat(three-stage-scan): add Docker CLI sandbox runner with isolation flags 2026-09-03 21:56:02 +02:00
Marius Ciepluch 5b7e58ea3a feat(three-stage-scan): add docker sandbox availability probe 2026-09-03 21:29:38 +02:00
Marius Ciepluch b34a75a821 feat(three-stage-scan): add tool-using hunter that records suspicion only 2026-09-03 21:18:22 +02:00
Marius Ciepluch 90e3b91cf8 feat(three-stage-scan): add path-clamped hunter read grep and refs 2026-09-03 21:01:57 +02:00
Marius Ciepluch ad16e22c85 feat(three-stage-scan): overlay complexity ledger without hiding sev-5 2026-09-03 20:48:03 +02:00
Marius Ciepluch 8d5599ae65 feat(three-stage-scan): attach test-gap hints to hotspots 2026-09-03 20:33:48 +02:00
Marius Ciepluch b05fbbc3a5 feat(three-stage-scan): emit ordered complexity map artifact 2026-09-03 20:16:23 +02:00
Marius Ciepluch 3d643040bd feat(three-stage-scan): compute complexity signals for hotspot ranking 2026-09-03 20:05:13 +02:00
Marius Ciepluch c5a80c2817 feat(three-stage-scan): add fail-on worth-fixing 2026-09-03 19:44:04 +02:00
Marius Ciepluch 926c8c273e feat(three-stage-scan): add complexity and regression config caps 2026-09-03 19:29:54 +02:00
Marius Ciepluch 723e3a76cf docs(three-stage-scan): add approved three-stage scan spec 2026-09-03 19:19:58 +02:00
Marius Ciepluch 6fdd9b3aba feat(three-stage-scan): encode stage plan and mode-to-stage table 2026-09-03 19:19:50 +02:00
68 changed files with 7202 additions and 23 deletions
+2
View File
@@ -26,6 +26,8 @@ jobs:
- run: uv run pytest - run: uv run pytest
- name: Detection gate (>=90% recall, <10% FP — hard, score-independent) - name: Detection gate (>=90% recall, <10% FP — hard, score-independent)
run: uv run python -m openultrasast.gate run: uv run python -m openultrasast.gate
- name: Split-sink map gate (stage 2 beats pattern-hit ranking)
run: uv run python -m openultrasast.map_gate
- run: uv run python -m compileall src tests - run: uv run python -m compileall src tests
dagger: dagger:
+74
View File
@@ -0,0 +1,74 @@
# Brief: three-stage-scan
## Problem
AppSec engineers and DevOps teams running OpenUltraSAST in CI get a regex static scan scored against cheat-sheet fixtures. They cannot tell what actually exists in a real repository, where complexity concentrates, how to aim tests, or whether a finding is worth a CI failure. The previous HarnessX spec added governance around those regexes (CWE policy, score, shadow status) without making the hunter evolutionary or the evaluation realistic.
## Current State
- `ousast scan --mode quick` is a deterministic regex inventory. It works.
- `--mode standard` is labeled a hunter pool but reuses the same regex path unless the optional HarnessX extra *and* a hunter model are both set. That hunter dumps a file into a prompt and scrapes JSON; it has no tools.
- `--mode deep` exits with "not implemented." Sandbox config exists on paper (`network=false`, read-only workspace, memory/pids/timeout). No container is ever started.
- Benchmarks are one-line sinks written to match the regexes. The fixtures themselves comment the misses (`regex blind spot`). The ≥90% recall gate is circular.
- CWE policy, project score, shadow status, traces, and SARIF already exist and should not be redesigned.
## Desired Outcome
A scan answers three operator questions, in order:
1. **What exists?** A static inventory of files, surfaces, and pattern/SARIF hits — honest about being inventory, not verified bugs.
2. **Where are problems most likely, and how do I tune tests?** A complexity map of hotspots plus test-gap hints a DevOps team can act on.
3. **Is this worth fixing?** Isolated regression against the top hotspots. Triggerable and reachable → worth a CI fail. Not triggerable → do not fail the build.
Stages 2–3 use a tool-calling hunter (read / grep / callers) and a Docker-isolated runner. Evaluation uses split-sink and mutation corpora, not cheat sheets. Missing Docker or a model degrades visibly; stage 1 still runs.
## Approach
**Map-then-prove on the existing modes.** Keep `--mode quick|standard|deep`:
- `quick` → stage 1 only
- `standard` → stages 1 + 2
- `deep` → stages 1 + 2 + 3
Do not require the HarnessX extra for the LLM path. Use the existing stdlib OpenRouter client when a hunter model is configured. HarnessX remains an optional alternative runtime.
Do not add a Python Docker SDK. Drive the Docker CLI via subprocess with the threat-model flags (no network, read-only source, tmpfs scratch, memory/pids/timeout, cap-drop). Core `dependencies = []` stays.
Complexity map is a new artifact built from existing ranking, entry points, finding density, nesting/size, and test-file presence — not a rewrite of `RankingScore`. Stage 3 writes a worth-fixing verdict that feeds back into a complexity ledger (evolutionary), without editing detection pattern text.
## Scope
- **In**: three-stage scan contract; complexity map + test-tuning hints; Docker-isolated regression and worth-fixing verdicts; tool-calling hunter on hotspots; split-sink/mutation evaluation; CI mapping (PR = stage 1, nightly = stage 2/3); visible degradation; evolutionary feedback from regress → map.
- **Out**: rewriting CWE policy or project-score formula; auto-authoring regex patterns; full fuzzing campaigns; patch generation/application; network pentest; importing Clearwing.
## Boundary Candidates
- Stage orchestrator (which stages a mode runs, artifacts, degradation)
- Complexity map (hotspots, signals, test-gap hints)
- Sandbox runner (Docker CLI, capability probe)
- Regression / worth-fixing (candidate tests, verdicts, ledger feedback)
- Tool-calling hunter (repo-bound tools + model loop for stages 2–3)
## Out of Boundary
- PolicyStore / ScoreModel / RulesetStore internals (consume, do not own)
- MetaAgent.evolve regex-status loop (adjacent, do not replace)
- Deep fuzzing, exploit development, auto-patching
- HarnessX packaging / optional extra (may be used, not required)
## Upstream / Downstream
- **Upstream**: existing preprocess, rank, mapping, findings, verification, config, CLI modes, OpenRouter client, threat-model sandbox limits.
- **Downstream**: CI fail-on `worth-fixing`; later fuzz/patch specs; MCP scan status.
## Existing Spec Touchpoints
- **Extends**: `openrouter-sast-harness` gates `standard_security_harness` and `sandboxed_dynamic_harness` (still unsatisfied).
- **Adjacent**: `harnessx-self-improving-rulesets` owns CWE policy, score, shadow, optional HarnessX extra. This spec must not reopen those.
## Constraints
- Core install stays zero-dependency. Docker CLI and an LLM key are operator-provided runtimes, not Python dependencies.
- Scanned code is untrusted. Stage 3 is the first mode allowed to execute target-derived code, and only inside the sandbox.
- LLM output starts at evidence `suspicion`. It must not be stamped `static_corroboration`.
- Cheat-sheet one-liners may remain as unit fixtures; they must not be the merge gate for stages 2–3.
+526
View File
@@ -0,0 +1,526 @@
# Design Document
## Overview
**Purpose.** This feature turns an OpenUltraSAST scan into three operator questions: what exists (static inventory), where problems are most likely and how to aim tests (complexity map), and whether a finding is worth fixing (isolated regression). It is a follow-up to the HarnessX governance spec, not a replacement of CWE policy, scoring, or rules-as-data.
**Users.** DevOps engineers get a cheap pull-request inventory and a nightly prove-worth-fixing job. AppSec reviewers get a hotspot map and test-gap hints instead of a regex dump. Maintainers get an evaluation gate that split-sink cases can fail.
**Impact.** Today `--mode standard` reuses regex findings and `--mode deep` exits. After this spec, `quick` remains stage 1, `standard` builds a complexity map and may run a tool-using hunter, and `deep` runs a Docker-isolated regression that is the only path allowed to execute target-derived code.
### Goals
- Map existing `--mode quick|standard|deep` onto stages 1 / 1+2 / 1+2+3 with explicit manifest accounting.
- Keep stage 1 zero-dependency, no model, no target execution.
- Emit a complexity map and test-tuning hints that do not collapse to pattern-hit count.
- Prove worth-fixing inside an isolated sandbox; fail CI on `worth_fixing` only when asked.
- Hunter tools follow dataflow; hunter output is `suspicion`.
- Stage 2/3 quality is gated on split-sink and mutation corpora.
### Non-Goals
- Rewriting PolicyStore, ScoreModel, or detection pattern text.
- Requiring the HarnessX extra for any stage.
- Full fuzzing, exploit development, or auto-patching.
- Importing Clearwing.
- A second CLI family (`--stage`).
## Boundary Commitments
### This Spec Owns
- **Stage plan**: which stages a mode runs, skip/degrade accounting, `stages_requested` / `stages_completed` in the manifest.
- **Complexity map**: `Hotspot` records, signals, test-gap hints, `complexity_map.json`.
- **Complexity ledger**: overlay of hotspot scores from regression verdicts.
- **Sandbox**: capability probe and Docker CLI runner that never mounts the host Docker socket.
- **Regression / worth-fixing**: candidate selection, language recipes, verdicts, `--fail-on worth-fixing`.
- **Tool hunter**: repo-bound read / grep / callers loop for stages 2–3, evidence honesty.
- **Split-sink evaluation**: corpora and the "beats pattern-hit ranking" gate.
### Out of Boundary
- CWE TSV, `resolve_severity`, project-score formula, rule `enabled|shadow|disabled`.
- `MetaAgent.evolve` pattern-status loop.
- Semgrep/CodeQL execution (ingest of operator-supplied SARIF remains as today).
- Patch generation and repository mutation.
- Sanitizer-image matrix and campaign fuzzing.
- MCP docker/shell tools, and exposing `deep` on MCP (CLI/CI only).
- Trail of Bits `skills.STAGES` (`map`/`hunt`/`verify`/`fix`). That `map` is mapping-discipline skill routing, not the complexity-map scan stage.
### Allowed Dependencies
- Existing preprocess, rank, mapping, findings, verification, reports, config, CLI, OpenRouter client, HarnessRuntime events.
- Optional HarnessX extra as an alternate hunter host, not a requirement.
- Operator-provided Docker CLI and LLM credential.
- Stdlib only for new core modules (`subprocess` for Docker, `urllib` already used by OpenRouter).
### Revalidation Triggers
- Changing the mode→stage mapping or `--fail-on` vocabulary.
- Changing hotspot identity keys (breaks the complexity ledger).
- Changing sandbox flags or mounting the Docker socket.
- Stamping hunter output above `suspicion`.
- Using cheat-sheet recall as the stage 2/3 merge gate.
## Architecture
### Existing Architecture Analysis
| Existing element | Constraint |
|---|---|
| `cli._run_scan` modes | Reuse names; stop exiting on `deep` |
| `quick_scan_findings` | Stage 1 detector; not the stage 2 hunter |
| `rank.heuristic_rank` | File budget; not the complexity map |
| `mapping.analyze_entry_points` | Input to map signals and caller hints |
| `hunter._run_hunter_task` | Must stop being a regex alias when a model is set |
| `OpenRouterChatClient` | Default hunter transport |
| `SandboxConfig` | Limits already match the threat model; need a runner |
| `verify_finding` | Keep as structural review gate; do not treat it as worth-fixing |
| `gate.py` 90% recall | Stage 1 only after this spec |
### Architecture Pattern & Boundary Map
**Selected pattern.** Ordered pipeline: **inventory drives the map; the map drives regression; regression overlays the map.** Execution never mutates governance data. Target code runs only in the sandbox.
```mermaid
flowchart TB
subgraph S1["STAGE 1 inventory"]
PRE[preprocess and mapping]
RANK[file rank]
INV[pattern and SARIF inventory]
end
subgraph S2["STAGE 2 map"]
SIG[complexity signals]
MAP[hotspot map]
HINT[test-gap hints]
HUNT[tool hunter optional]
end
subgraph S3["STAGE 3 prove"]
CAND[regress candidates]
BOX[Docker CLI sandbox]
VERD[worth-fixing verdicts]
end
LED[complexity ledger]
CI[CI fail-on]
PRE --> RANK --> INV
INV --> SIG
RANK --> SIG
LED --> SIG
SIG --> MAP --> HINT
MAP --> HUNT
HUNT --> MAP
MAP --> CAND
INV --> CAND
CAND --> BOX --> VERD
VERD --> LED
VERD --> CI
INV --> CI
```
**Dependency direction:** config/types → preprocess/rank/mapping/findings → complexity map → hunter tools → sandbox → regress → CLI/report/gate.
### Technology Stack
| Layer | Choice | Role | Notes |
|-------|--------|------|-------|
| CLI | existing `ousast scan --mode` | Stage selection | Add `--fail-on worth-fixing` |
| Hunter transport | `OpenRouterChatClient` | Tool loop | Stdlib; HarnessX optional |
| Sandbox | Docker CLI via `subprocess` | Isolated execution | No Python docker SDK |
| Map / ledger | JSON artifacts | Attention + evolution | Same overlay style as `rule_policy.json` |
| Evaluation | split-sink fixtures | Stage 2/3 gate | Smoke fixtures stay stage 1 |
## File Structure Plan
### Directory Structure
```
src/openultrasast/
├── stages.py # Stage enum, mode map, StagePlan, degradation helpers
├── complexity/
│ ├── __init__.py
│ ├── signals.py # per-file/function signals from targets, findings, tests
│ ├── map.py # build_complexity_map, Hotspot, ComplexityMap
│ ├── hints.py # test-gap hints, closed test-kind set
│ └── ledger.py # load/apply/write complexity_ledger.json
├── hunter_tools.py # read_file, grep_repo, find_refs; path clamp
├── tool_hunter.py # bounded tool loop over a chat client
├── sandbox/
│ ├── __init__.py
│ ├── probe.py # docker CLI present and usable
│ └── runner.py # DockerCliRunner, SandboxRunner protocol, FakeSandboxRunner
└── regress/
├── __init__.py
├── candidate.py # select and render candidates
├── recipes.py # language command recipes
└── verdict.py # run candidates, worth-fixing gate
benchmarks/fixtures/
├── split-sink-python/
├── split-sink-javascript/
├── split-sink-java/
└── split-sink-c/
benchmarks/manifests/split-sink-*.toml
```
### Modified Files
- `cli.py` — run stage 2/3; stop exiting on `deep`; `--fail-on worth-fixing`; write new artifacts.
- `reports.py` — `scan_exit_code` grows `worth-fixing`; report sections for inventory, map, and worth-fixing.
- `config.py` — `ComplexityConfig` (top_k, max_hunter_hotspots) and `RegressConfig` (max_candidates, image pins); **consume** existing `SandboxConfig` (do not add a second sandbox TOML).
- `hunter.py` — schedule remains; execution delegates to `tool_hunter` when a model is set, else inventory-only.
- `verification.py` / reports — inventory vs worth-fixing labeling; hunter findings stay `suspicion`.
- `gate.py` — stage-1 smoke gate unchanged; add split-sink map gate.
- `reports.py` — map and verdict sections in markdown/manifest.
- `.github/workflows/ci.yml` — PR stays quick; optional nightly deep when Docker is available.
## System Flows
### Mode to stages
```mermaid
flowchart LR
Q[quick] --> A[stage 1]
S[standard] --> A
S --> B[stage 2]
D[deep] --> A
D --> B
D --> C[stage 3]
```
If stage 2 has no model, the heuristic map still writes. If stage 3 has no Docker, stages 1–2 remain and `sandbox_unavailable` is recorded.
### Stage 3 worth-fixing
```mermaid
sequenceDiagram
participant M as ComplexityMap
participant R as RegressionRunner
participant S as DockerCliRunner
participant G as WorthFixingGate
M->>R: top hotspots plus severe inventory
R->>R: render snippet and safety check
alt safety check fails
R->>G: inconclusive
else sandbox missing
R->>G: inconclusive sandbox_unavailable
else recipe missing
R->>G: inconclusive
else runs
R->>S: docker run read-only source
S->>R: exit code stdout stderr
R->>G: triggerable or not_triggerable or already_covered
end
G->>M: ledger overlay
```
## Requirements Traceability
| Requirement | Summary | Components | Flows |
|-------------|---------|------------|-------|
| 1.1–1.6 | Mode maps to stages; manifest accounting; degrade forward | `stages.py`, `cli.py` | Mode to stages |
| 2.1–2.5 | Stage 1 inventory, no execute, no model | existing preprocess/rank/findings | Stage 1 |
| 3.1–3.5 | Complexity map independent of regex count | `complexity/map.py`, `signals.py` | Stage 2 |
| 4.1–4.4 | Test-gap hints, no repo writes | `complexity/hints.py` | Stage 2 |
| 5.1–5.8 | Sandbox regression, worth-fixing, fail-on | `sandbox/*`, `regress/*` | Stage 3 |
| 6.1–6.6 | Tool hunter, path clamp, suspicion | `hunter_tools.py`, `tool_hunter.py` | Stage 2/3 |
| 7.1–7.5 | Ledger overlay, cannot hide sev-5 reachable | `complexity/ledger.py` | Stage 3 → map |
| 8.1–8.6 | Split-sink / mutation gate | fixtures, `gate.py` | Evaluation |
| 9.1–9.6 | PR cheap, nightly bounded, visible skip | `cli.py`, CI workflow | Degradation |
## Components and Interfaces
| Component | Domain | Intent | Req Coverage | Dependencies | Contracts |
|-----------|--------|--------|--------------|--------------|-----------|
| StagePlan | Orchestration | Mode → stages, skip reasons | 1.1–1.6, 9.3–9.6 | config | State |
| ComplexityMapBuilder | Map | Hotspots from signals + ledger | 3.1–3.5, 7.1–7.3 | preprocess, rank, findings | Batch, State |
| TestTuningAdvisor | Map | Test-gap hints | 4.1–4.4 | ComplexityMap | Batch |
| HunterTools | Hunter | Read/grep/refs inside repo | 6.1–6.2 | mapping | Service |
| ToolHunter | Hunter | Bounded tool loop | 6.1–6.5 | OpenRouter, HunterTools | Service |
| SandboxProbe | Sandbox | Docker usable? | 5.2, 9.3 | docker CLI | Service |
| DockerCliRunner | Sandbox | Isolated `docker run` | 5.2, 5.8 | docker CLI | Service |
| RegressionRunner | Regress | Candidates, recipes, verdicts | 5.1–5.6 | Sandbox, map | Batch |
| WorthFixingGate | Regress | CI fail-on | 5.4–5.7, 7.4 | RegressionRunner, policy | Batch |
### Orchestration
#### StagePlan
| Field | Detail |
|-------|--------|
| Intent | Declare which stages run and why a stage was skipped |
| Requirements | 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 9.3, 9.4, 9.6 |
```python
from enum import StrEnum
class Stage(StrEnum):
STATIC = "static"
MAP = "map"
REGRESS = "regress"
MODE_STAGES = {
"quick": (Stage.STATIC,),
"standard": (Stage.STATIC, Stage.MAP),
"deep": (Stage.STATIC, Stage.MAP, Stage.REGRESS),
}
@dataclass(frozen=True)
class StagePlan:
mode: str
requested: tuple[Stage, ...]
completed: tuple[Stage, ...]
skipped: tuple[dict[str, str], ...] # stage, reason
```
- **Invariants:** `quick` never includes `MAP` or `REGRESS`. Skipped stages appear in `runtime.state["degradations"]` with the same shape as existing capability fallbacks.
- **Naming:** `Stage.MAP` is the complexity-map scan stage. It is unrelated to `skills.STAGES` value `"map"` (Semgrep/CodeQL skill routing). Do not merge the two enumerations.
### Complexity map
#### ComplexityMapBuilder
| Field | Detail |
|-------|--------|
| Intent | Rank likely problem areas without using pattern-hit count as the score |
| Requirements | 3.1, 3.2, 3.3, 3.4, 3.5 |
**Signals** (stdlib, no extra libs):
- size: loc
- nesting: max indent or brace depth
- surface tags from `FileTarget`
- reachability from entry-point hints
- inventory density as a *feature*, not the score (used in rationale, down-weighted in the formula)
- test presence: sibling `test_*` / `*_test` / `*Test.java` / `*_test.go` under common test roots
- ledger overlay last
**Hotspot identity:** `path` + `function_name or ""`. Function name from reachability hints or a language-specific def/func regex; empty if unknown. Java/Groovy mapping is tag-only today, so identity is usually the file; that is acceptable for ranking and test-gap hints.
```python
@dataclass(frozen=True)
class Hotspot:
path: str
function_name: str | None
score: float
band: str # high | medium | low
signals: dict[str, float | int | bool | str]
rationale: str
test_hint: TestHint | None
inventory_finding_ids: tuple[str, ...]
@dataclass(frozen=True)
class ComplexityMap:
hotspots: tuple[Hotspot, ...]
heuristic_only: bool
```
- **Invariant:** sorting files by `len(inventory_finding_ids)` descending must not be identical to map order on the split-sink corpus (enforced by the gate).
- **Postcondition:** write `.openultrasast/runs/<id>/complexity_map.json`.
#### TestTuningAdvisor
Closed `test_kind` set: `unit | property | sanitizer | http-contract | fuzz-harness`.
```python
@dataclass(frozen=True)
class TestHint:
path: str
function_name: str | None
gap: str # no_adjacent_test | no_function_reference | no_fuzz_entry | covered
test_kind: str | None
reason: str
```
- **Invariant:** `gap == "covered"` ⇒ `test_kind is None`. Never writes into the scanned tree.
#### Complexity ledger
Path: `<repo>/.openultrasast/calibration/complexity_ledger.json`.
```json
{
"parser.c:parse_header": {
"score_delta": 1.5,
"last_verdict": "triggerable",
"round": 3
}
}
```
Apply at map build. `score_delta` cannot drop a reachable policy-severity-5 inventory finding out of the stage-3 candidate set.
### Hunter
#### HunterTools
```python
def clamp_repo_path(root: Path, user_path: str) -> Path: ...
def read_file(root: Path, path: str, *, max_chars: int) -> str: ...
def grep_repo(root: Path, pattern: str, *, max_matches: int) -> list[dict[str, object]]: ...
def find_refs(root: Path, symbol: str, mapping_index: object) -> list[dict[str, object]]: ...
```
- **Invariant:** any resolved path whose common prefix is not `root` raises `PathEscapesRepo`.
- Grep is Python `re` over enumerated source files (zero-dep). Ripgrep is an optional fast path if present, still clamped.
#### ToolHunter
Bounded loop: max steps from the hotspot's hunter budget (reuse tier A/B/C caps). Each model turn may call the three tools or stop with a JSON list of suspicions.
```python
def run_tool_hunter(
root: Path,
hotspots: list[Hotspot],
*,
client: ChatClient,
model: str,
max_steps: int,
) -> list[StaticFinding]: ...
```
- **Postconditions:** each finding `evidence_level == "suspicion"`; `finding_id` prefix `tool-hunter:`.
- **Fallback:** no model → empty hunter findings, `heuristic_only=True` on the map, degradation `hunter_model_unavailable`.
### Sandbox and regression
#### SandboxRunner
```python
class SandboxRunner(Protocol):
def run(self, spec: SandboxJob) -> SandboxResult: ...
@dataclass(frozen=True)
class SandboxJob:
image: str
command: tuple[str, ...]
repo_root: Path
scratch_files: dict[str, str] # relative to /scratch
timeout_seconds: int
memory_mb: int
pids_limit: int
```
`DockerCliRunner` argv (conceptual): `docker run --rm --init --network none --read-only --tmpfs /scratch:rw,exec --memory --pids-limit --cap-drop ALL --security-opt no-new-privileges --user` non-root `--mount type=bind,src=repo,dst=/workspace,ro` image command.
- **Forbidden:** `-v /var/run/docker.sock`, `--network host`, `--privileged`, writable bind of the source tree.
- `SandboxProbe.available()` runs `docker info` with a short timeout.
#### RegressionRunner
Select `min(max_candidates, top_k_hotspots ∪ sev5_reachable_inventory)`. Render a recipe snippet into `scratch_files`. Safety check rejects snippets that mention `socket`, `curl`, `docker.sock`, or writes under `/workspace`.
Verdicts:
| Observation | Verdict |
|---|---|
| recipe or sandbox missing, timeout, safety reject | `inconclusive` |
| existing project test covering the hotspot already passes in sandbox | `already_covered` |
| candidate exits non-zero with a crash, assertion, or sanitizer abort | `triggerable` |
| candidate exits 0 | `not_triggerable` |
`worth_fixing` ⇔ `triggerable` and related finding reachability is `reachable` or `inferred-file-surface`.
`--fail-on` gains `worth-fixing`. Existing `findings` / `verified` unchanged.
Recipes MVP: Python (`python /scratch/case.py` with `PYTHONPATH=/workspace`), JavaScript (`node /scratch/case.js`), C (`cc` + run) when the image contains a compiler. Java recipe may return `inconclusive` until added.
## Data Models
### Manifest additions
```json
{
"stages": {
"requested": ["static", "map", "regress"],
"completed": ["static", "map"],
"skipped": [{"stage": "regress", "reason": "sandbox_unavailable"}]
},
"complexity": {"hotspot_count": 12, "heuristic_only": true},
"worth_fixing": {"count": 0, "verdicts": []}
}
```
Degradations reuse the existing list: `{ "stage": "regress", "reason": "sandbox_unavailable" }`.
### Consistency
- Map artifact and ledger are JSON, resume-safe.
- Hunter suspicions never jump evidence levels in this spec.
- Stage 1 artifacts remain byte-compatible with current `findings.json` schema; new fields are additive.
## Error Handling
| Error | Response |
|---|---|
| `deep` requested, Docker missing | Complete 1–2, skip 3, degradation, exit 0 unless other fail-on hits |
| Model missing on `standard` | Heuristic map, no hunter findings, degradation |
| Path escape in a tool | Tool error to the model; do not throw the scan |
| Snippet fails safety check | `inconclusive`, do not run |
| Docker run timeout | kill container (`--rm`), `inconclusive` |
| Ledger would hide sev-5 reachable | Candidate still selected |
## Testing Strategy
### Unit
- Mode→stage mapping and skip accounting.
- Path clamp rejects `../`.
- Map order on a fixture where regex hits a trivial helper and the real bug is a split-sink parser — parser ranks higher.
- Covered hotspot yields `gap=covered` and no test kind.
- Safety check rejects docker.sock and network snippets.
- Ledger boost/demote; sev-5 reachable remains a candidate.
### Integration
- Fake sandbox: `triggerable` / `not_triggerable` / timeout mapped to verdicts.
- Tool hunter with a scripted chat client that must call `grep_repo` before concluding on a split-sink file; fail if it only returns JSON with no tool call.
- `--fail-on worth-fixing` exits non-zero only when a `worth_fixing` verdict exists.
### Evaluation / CI
- Stage 1: existing `python -m openultrasast.gate` on smoke corpora.
- Stage 2: split-sink manifests; true vulnerable function in the high band; map order ≠ pattern-hit order.
- Stage 3: `@pytest.mark.docker`; skip if probe fails; at least one planted split-sink that stage 1 missed is `triggerable`.
- PR workflow stays `quick`. Nightly/deep is a separate job with Docker.
### Anti-patterns this spec forbids in tests
- One-line `eval(request.args)` as the only stage-2 proof.
- Asserting hunter findings equal regex findings.
- Calling the real Docker daemon in the default `pytest` job.
## Security Considerations
- Scanned code is untrusted. Stage 1–2 read it as text. Stage 3 executes only in Docker with network off, source read-only, no docker.sock, dropped caps, no-new-privileges, non-root user.
- Hunter tools cannot read outside the repo.
- Regression snippets are data, not host shell. They go to `/scratch` in the container.
- Secrets redaction already applies to traces and markdown; keep it on for map rationales that quote source.
## Performance & Scalability
- Map is O(files) plus one pass of findings. No extra model required.
- Tool hunter runs only on `max_hunter_hotspots` (default 8).
- Stage 3 `max_candidates` default 5; timeout from `SandboxConfig.timeout_seconds`.
- Large-repo CI: PR = stage 1; nightly = stage 2 always, stage 3 if probe passes.
## Migration Strategy
```mermaid
flowchart LR
P0[StagePlan plus manifest] --> P1[Heuristic map and hints]
P1 --> P2[Split-sink gate]
P2 --> P3[Tool hunter]
P3 --> P4[Docker runner plus recipes]
P4 --> P5[Ledger and fail-on worth-fixing]
```
| Phase | Ships | Merge gate | Rollback |
|---|---|---|---|
| 0 StagePlan | `deep` no longer exits; skips 2/3 with degradations | existing tests | revert CLI |
| 1 Heuristic map | `complexity_map.json` on `standard` | unit map tests | feature: empty map writer |
| 2 Split-sink gate | new fixtures; map ≠ hit-count | new gate job | gate off |
| 3 Tool hunter | OpenRouter loop, suspicion | scripted-client test | model unset |
| 4 Sandbox | Docker CLI + Python/JS recipes | docker-marked tests skip-ok | probe skip |
| 5 Worth-fixing | ledger + fail-on | integration on fake runner | flag unused |
`deep` changing from hard-exit to skip-or-run is intentional and documented. No artifact schema is removed.
@@ -0,0 +1,129 @@
# Requirements Document
## Introduction
OpenUltraSAST today answers one question well: which regexes fired. It does not answer the three questions a rugged DevOps CI workflow actually asks: what exists in this repository, where problems are most likely and how to aim tests, and whether a finding is worth fixing. This feature turns the scan into those three stages. Stage 1 is a static inventory of what exists. Stage 2 is a complexity map of likely problem areas plus test-tuning hints. Stage 3 is isolated regression that proves triggerability before CI treats a finding as blocking.
The work is a follow-up to the HarnessX ruleset-governance spec, not a replacement of it. CWE policy, project score, and rule-status shadowing stay in place. This spec owns the stage contract, the complexity map, the isolated regression verdict, the tool-using hunter on stages 2–3, and an evaluation corpus that is not a cheat sheet.
## Boundary Context
- **In scope**: a three-stage scan contract mapped onto existing scan depths; a static inventory that reports what exists without claiming verification; a complexity map of hotspots with test-tuning hints; isolated regression that produces a worth-fixing verdict; a tool-using hunter confined to stages 2–3; evaluation against split-sink and mutation corpora; visible degradation when the sandbox or a model is unavailable; evolutionary feedback from regression verdicts back into the complexity map.
- **Out of scope**: rewriting CWE-to-severity policy or the project-score formula; authoring or rewriting detection pattern text; full fuzzing campaigns; generating or applying patches; network pentesting; importing Clearwing.
- **Adjacent expectations**: stage 1 continues to run in the zero-dependency default install with no model and no sandbox. CWE policy, scoring, traces, and SARIF remain owned by the existing governance spec. The operator supplies an isolated-sandbox runtime (Docker CLI) for stage 3 and an LLM credential for the tool-using hunter; neither is a packaged library dependency of the core install.
## Requirements
### Requirement 1: Three-Stage Scan Contract
**Objective:** As a DevOps engineer, I want a scan to run as three ordered stages that map onto the existing scan depths, so that CI can choose inventory-only, map-and-tune, or prove-worth-fixing without a new command family.
#### Acceptance Criteria
1. When a scan runs at depth `quick`, the Scan Orchestrator shall execute stage 1 only and shall not start a model call or execute target-derived code.
2. When a scan runs at depth `standard`, the Scan Orchestrator shall execute stage 1 then stage 2.
3. When a scan runs at depth `deep`, the Scan Orchestrator shall execute stage 1 then stage 2 then stage 3.
4. The Scan Orchestrator shall record the stages requested, the stages completed, and any skipped stage with a reason in the scan manifest.
5. If a later stage cannot run, the Scan Orchestrator shall complete earlier stages, record a degradation, and shall not discard those earlier artifacts.
6. The Scan Orchestrator shall keep the existing scan command and depth names so existing CI invocations continue to work.
### Requirement 2: Stage 1 Static Inventory
**Objective:** As a security reviewer, I want a static inventory of what exists in the repository, so that I can see files, surfaces, and pattern hits without treating them as verified vulnerabilities.
#### Acceptance Criteria
1. When stage 1 completes, the Static Inventory shall emit a repository snapshot, file targets with language and surface tags, entry-point reachability hints, ranked files, and static findings from the existing pattern ruleset and any provided static-analyzer input.
2. The Static Inventory shall label static pattern hits as inventory, not as verified findings, in the report and in the finding evidence level.
3. While no static-analyzer input is provided, the Static Inventory shall still complete using the built-in pattern ruleset and heuristics.
4. If an enabled rule's CWE does not resolve in policy, the Scan Runtime shall fail loud before producing inventory findings (existing startup invariant, preserved).
5. The Static Inventory shall not execute target-derived code and shall not require a model.
### Requirement 3: Stage 2 Complexity Map
**Objective:** As an AppSec engineer, I want a complexity map of where problems are most likely, so that review and testing attention go to real hotspots instead of every regex hit.
#### Acceptance Criteria
1. When stage 2 runs, the Complexity Map shall produce an ordered list of hotspots, each identifying a file and, when known, a function, with a complexity score and a rationale built from size, nesting, surface tags, reachability, existing inventory-hit density, and test-presence gaps.
2. The Complexity Map shall rank hotspots independently of a single regex match: a file with no pattern hit may still be a hotspot, and a pattern hit in a trivial helper may rank below a complex reachable parser.
3. When stage 2 completes, the Complexity Map shall write a map artifact that the report and later stages consume, including the scoring signals used for each hotspot.
4. While a hunter model is unavailable, the Complexity Map shall still produce the heuristic map and shall record that the hunter did not refine it.
5. The Complexity Map shall not mark a hotspot as verified or worth fixing.
### Requirement 4: Test-Tuning Hints
**Objective:** As a developer owning CI tests, I want the complexity map to tell me how to tune tests, so that I add coverage where it reduces real risk instead of mirroring cheat-sheet sinks.
#### Acceptance Criteria
1. When the Complexity Map emits a hotspot, the Test Tuning Advisor shall attach a test-gap hint that names the missing coverage (no adjacent test file, no test referencing the function, or no sanitizer/fuzz entry for a fuzzable surface).
2. The Test Tuning Advisor shall recommend a test kind from a closed set: unit, property, sanitizer, http-contract, or fuzz-harness, with a one-line reason tied to the hotspot's signals.
3. The Test Tuning Advisor shall not write test files into the scanned repository.
4. When a hotspot already has covering tests, the Test Tuning Advisor shall say so and shall not recommend a duplicate test kind.
### Requirement 5: Stage 3 Isolated Regression and Worth-Fixing
**Objective:** As a CI owner, I want isolated regression that answers whether a finding is worth fixing, so that the build fails only on triggerable, reachable issues.
#### Acceptance Criteria
1. When stage 3 runs, the Regression Runner shall select a bounded set of top hotspots and/or inventory findings and shall attempt a minimal regression or proof-of-concept inside an isolated analysis sandbox.
2. The isolated analysis sandbox shall run with network disabled, the source tree mounted read-only, a writable scratch area, and configured memory, process, and timeout limits.
3. The Regression Runner shall emit one of four verdicts per candidate: `triggerable`, `not_triggerable`, `already_covered`, or `inconclusive`.
4. When a candidate is `triggerable` and the related finding is reachable, the Worth-Fixing Gate shall mark it `worth_fixing`.
5. When a candidate is `not_triggerable` or `already_covered`, the Worth-Fixing Gate shall mark it `not_worth_fixing` and shall not fail the build for that candidate.
6. When the verdict is `inconclusive`, the Worth-Fixing Gate shall not fail the build for that candidate and shall record why (timeout, sandbox unavailable, no candidate generated, or setup failed).
7. Where `--fail-on worth-fixing` is selected, the Scan Orchestrator shall fail the scan if and only if at least one `worth_fixing` verdict was produced.
8. The Regression Runner shall never execute target-derived code on the host outside the isolated sandbox.
### Requirement 6: Tool-Using Hunter on Stages 2 and 3
**Objective:** As an OpenUltraSAST maintainer, I want the hunter on later stages to use repository tools instead of dumping a file into a prompt, so that it can follow dataflow across lines and files the way a real review does.
#### Acceptance Criteria
1. Where a hunter model is configured, the Hunter Pool shall run a bounded tool-using loop against stage-2 hotspots, with tools limited to reading a file inside the repository, searching repository text, and listing callers or references already known to the scan.
2. The Hunter Pool shall confine every tool path to the scanned repository root and shall reject path arguments that escape that root.
3. When the hunter reports a candidate issue, the Scan Runtime shall record it at evidence level `suspicion` until independent static or regression evidence raises it.
4. While no hunter model is configured, the Hunter Pool shall skip the tool loop, leave the heuristic complexity map in place, and record a degradation.
5. The Hunter Pool shall not treat a model JSON list as static corroboration.
6. When stage 3 runs and a hunter model is configured, the Hunter Pool may propose a regression snippet, but the Regression Runner shall execute that snippet only inside the isolated sandbox after a structural safety check (no network, no writes outside scratch).
### Requirement 7: Evolutionary Feedback from Regression to Map
**Objective:** As a maintainer, I want regression verdicts to retune the complexity map, so that the scan gets better at spending attention where issues actually trigger.
#### Acceptance Criteria
1. When a hotspot's regression is `triggerable`, the Complexity Map shall raise that hotspot's subsequent score on the same repository.
2. When a hotspot's regression is `not_triggerable`, the Complexity Map shall lower that hotspot's subsequent score without deleting the underlying inventory hit.
3. The Scan Runtime shall persist those adjustments in a loop-owned ledger keyed by repository and hotspot identity, applied at map load the same way rule status overlays apply at ruleset load.
4. If a ledger entry would hide a reachable severity-5 inventory hit from the worth-fixing set, the Worth-Fixing Gate shall still consider that hit.
5. The evolutionary loop shall not edit detection pattern text and shall not edit authoritative CWE severity.
### Requirement 8: Grounded Evaluation, Not Cheat Sheets
**Objective:** As a maintainer, I want stage 2 and stage 3 quality gates to use split-sink and mutation cases, so that a regex that only matches textbook one-liners cannot claim success.
#### Acceptance Criteria
1. The Benchmark Harness shall include at least one split-sink case per in-scope language family (Python, JavaScript, Java, C/C++) where the source and the sink are on different lines or the payload is assigned before use.
2. The Benchmark Harness shall include at least one mutation of a known sink (equivalent API, intermediate variable, or wrapper) that a same-line pattern does not match.
3. When stage 1 is scored alone, the Benchmark Harness may use existing smoke fixtures, and shall not treat stage-1 recall as proof of stage-2 or stage-3 quality.
4. When stage 2 is scored, the Benchmark Harness shall require the complexity map to rank the true vulnerable function in the top band of hotspots on the split-sink corpus, even when stage 1 missed the sink.
5. When stage 3 is scored and the sandbox is available, the Benchmark Harness shall require a `triggerable` verdict on at least one planted split-sink that stage 1 missed.
6. The CI merge gate for this feature shall fail if stage 2 ranking on the split-sink corpus is no better than ranking files by pattern-hit count alone.
### Requirement 9: CI Companion Budgets and Degradation
**Objective:** As a CI owner, I want stage costs and failures to be bounded and visible, so that pull-request jobs stay cheap and nightly jobs can run the later stages without silent fallback.
#### Acceptance Criteria
1. While running in pull-request depth `quick`, the CI Pipeline shall complete stage 1 without a sandbox runtime and without a model.
2. Where a nightly or labeled job requests `standard` or `deep`, the CI Pipeline shall run those stages under the configured time, finding-count, and cost budgets.
3. If the sandbox runtime is missing when stage 3 is requested, the Scan Orchestrator shall skip stage 3, record `sandbox_unavailable`, and complete stages 1–2.
4. If a model is missing when stage 2 hunter refinement is requested, the Scan Orchestrator shall keep the heuristic map, record `hunter_model_unavailable`, and shall not invent hunter findings.
5. The Scan Orchestrator shall bound stage 3 to a configured maximum number of regression candidates per scan.
6. When any stage is skipped or truncated, the report shall show the degradation in the same manifest list used by existing capability fallbacks.
+156
View File
@@ -0,0 +1,156 @@
# Research & Design Decisions
---
**Purpose**: Capture discovery findings and architectural investigation behind the `three-stage-scan` spec. `design.md` carries the committed architecture; this file carries evidence and rejected alternatives.
---
## Summary
- **Feature**: `three-stage-scan`
- **Discovery Scope**: Extension of the existing scan harness (brownfield), with one new runtime prerequisite (Docker CLI for stage 3) and one new evaluation contract (split-sink / mutation corpora).
- **Key Findings**:
- Stage 1 already exists as `quick` mode: preprocess, mapping, rank, `quick_scan_findings`, structural verify, score, SARIF. Do not rebuild it.
- Stage 2 is not a hunter dump and is not `RankingScore`. Ranking is per-file attention. A complexity map is per-file-and-function hotspot attention plus test-gap hints. Existing signals (tags, loc, reachability hints, finding density, `has_fuzz_entry_point`) are enough for a heuristic map; the tool-using hunter refines it.
- Stage 3 has config and a threat-model table, but `cli.py` exits on `--mode deep`. No container is ever started. The first execution of untrusted code must be Docker CLI with the documented isolation flags, not a Python SDK and not host `subprocess` of target code.
- The previous HarnessX spec locked LLM use behind an optional extra that CI does not install. This spec uses the existing stdlib OpenRouter client so stage 2 can be evolutionary without that extra. HarnessX remains an optional runtime, not a prerequisite.
- Cheat-sheet fixtures cannot gate stages 2–3. The Python fixture already documents a split-sink miss (`query` built on the prior line). That class of case becomes the quality bar.
## Research Log
### Existing scan pipeline and mode mapping
- **Context**: Whether to add `--stage` or reuse `--mode quick|standard|deep`.
- **Sources Consulted**: `src/openultrasast/cli.py` (`_run_scan`, mode choices), `README.md` maturity legend, `.kiro/specs/openrouter-sast-harness/requirements.md` (hunter pool, docker sandbox, dynamic analysis), `.kiro/specs/harnessx-self-improving-rulesets/design.md`.
- **Findings**:
- `quick` is implemented and deterministic. `standard` schedules a hunter pool that calls `quick_scan_findings` unless HarnessX extra + hunter model are set. `deep` raises `SystemExit`.
- OpenRouter spec already defined: quick = static + bounded LLM review; standard = sandboxed tool-calling hunters; deep = harness/fuzz/dynamic. Implementation never reached that.
- Existing CI and README teach `--mode`. A parallel `--stage` flag would fork the contract.
- **Implications**: Map modes onto stages 1 / 1+2 / 1+2+3. Record `stages_requested` and `stages_completed` in the manifest so the three questions are explicit even if the flag names stay.
### Ranking vs complexity map
- **Context**: `rank.py` already scores files. Is stage 2 just ranking?
- **Sources Consulted**: `src/openultrasast/rank.py` (`heuristic_rank`, `composite_priority` = surface 0.5 + influence 0.2 + reachability 0.3), `preprocess.py` (`FileTarget` tags, loc, `has_fuzz_entry_point`), `mapping.py` (`analyze_entry_points`).
- **Findings**:
- Ranking is file-level and exists to budget the hunter. It does not name functions, test gaps, or "tune your tests."
- Tags already encode likely problem classes (`memory_unsafe`, `parser`, `network_entry`, `syscall_entry`). Loc ≥ 500 boosts influence. No nesting/branch signal. No test-file presence.
- A hotspot that stage 1 missed (split-sink) can still rank high if the file is a reachable parser. That is the point of stage 2.
- **Implications**: Keep `RankingScore`. Add `ComplexityMap` / `Hotspot` with function identity when recoverable, test-gap hints, and a ledger overlay. Ranking still feeds hunter budgets; the map feeds humans and stage 3.
### Hunter: regex, HarnessX dump, or tools
- **Context**: Previous spec adopted HarnessX "at the hunter loop" but the default hunter is regex and the HarnessX path dumps source into a prompt.
- **Sources Consulted**: `hunter.py` (`_run_hunter_task` → `quick_scan_findings`), `hunter_harness.py` (`_describe`, `_extract_findings`), `provider/openrouter.py` (`OpenRouterChatClient.complete_json`, stdlib `urllib`), `harness_ext.py` (optional extra).
- **Findings**:
- OpenRouter client is already zero-extra (stdlib). It is used for optional ranking, not for hunting.
- HarnessX extra is SHA-pinned, heavy, absent in CI. Requiring it repeats the last spec's failure mode.
- Clearwing hunters use `read_source_file`, `grep_source`, `find_callers` with path clamping. That is the evolutionary piece, not MetaAgent regex-status flips.
- **Implications**: Stage 2/3 hunter is a bounded tool loop over the OpenRouter client (or HarnessX if already wired). Tools are stdlib (read, Python grep, caller search from mapping + text). Path clamp at repo root. Hunter output evidence = `suspicion`.
### Docker sandbox: CLI vs SDK vs skip
- **Context**: Threat model already lists sandbox limits. Need a runner that does not invert zero-dep posture.
- **Sources Consulted**: `docs/threat-model.md` (sandbox table), `config.py` `SandboxConfig`, Clearwing `hunter_sandbox.py` (image build + spawn, Docker SDK), Docker isolation guidance (`docker run --network none --read-only --tmpfs --memory --pids-limit --cap-drop ALL --security-opt no-new-privileges`), initrunner docker-sandbox docs (CLI `docker run --rm --init`, optional `runsc`).
- **Findings**:
- Python `docker` SDK would be a new core or extra dependency. The operator already has the Docker CLI if they can run containers.
- Subprocess `docker run --rm --init` with bind-mount of the repo read-only and a tmpfs/scratch write area matches the threat model and keeps `dependencies = []`.
- Unit tests must not require a daemon: inject a `SandboxRunner` protocol, ship a fake. Integration tests skip when `docker info` fails.
- Full sanitizer-image builds (Clearwing) are out of scope for this spec. Stage 3 runs language-specific commands on a small pinned image per language family, or `inconclusive` if the recipe is missing.
- **Implications**: `SandboxProbe` + `DockerCliRunner`. No SDK. gVisor/`runsc` is optional `security-opt` when `docker info` reports it; not required.
### Evaluation corpora
- **Context**: User rejected cheat-sheet fixtures. Current gate is ≥90% recall on those fixtures.
- **Sources Consulted**: `benchmarks/fixtures/python-vulnerable/app.py` (line 77 split-sink comment), `javascript-vulnerable/src/app.js` (prototype pollution comment), `tests/test_detection_benchmarks.py`, `gate.py`.
- **Findings**:
- Stage 1 smoke fixtures are still useful as unit tests. They must not be the stage 2/3 merge gate.
- A split-sink corpus is small and owned: source and sink on different lines; mutations through wrappers/variables. Success = map ranks the true function in the top band; stage 3 triggers at least one stage-1 miss when Docker is present.
- Comparing map ranking to "sort by pattern-hit count" is an anti-cheat-sheet test: if regex density is the map, the gate fails.
- **Implications**: New `benchmarks/fixtures/split-sink-*` and a stage-2 ranking gate. Keep existing `openultrasast.gate` for stage 1 only.
### Seams that already exist and must not be confused
- **Context**: Brownfield traps for a spec named "map" that also adds Docker.
- **Sources Consulted**: `skills.py` (`STAGES = ("map", "hunt", "verify", "fix")`), `mcp.py:68` (rejects `deep`), `reports.py:115` (`scan_exit_code`), `mapping.py` (no Java/Groovy AST), `config.py` (`SandboxConfig` parsed, never consumed by `_run_scan`), `rank.py` (`rank_targets_with_model` unused by CLI).
- **Findings**:
- Trail of Bits skill routing already uses a stage named `map` (mapping-discipline skills: Semgrep/CodeQL/sharp-edge). That is not the complexity map. `stages.py` must not reuse `skills.STAGES`.
- MCP scan tools reject `deep` and forbid docker/shell. This spec does not widen the MCP surface; deep is CLI/CI only.
- `--fail-on` is implemented in `scan_exit_code`, not only argparse. `worth-fixing` must be added there.
- Java/Groovy entry points are tag-only (`line=None` → `inferred-file-surface`). Hotspot `function_name` will often be empty on Java; the map must still rank the file.
- `[sandbox]`, `[dynamic]`, `[evidence]`, and `[models].ranker` are loaded into resolved config and then ignored. Stage 3 consumes `SandboxConfig`; it does not invent a parallel sandbox TOML.
- **Implications**: Naming, MCP, fail-on, and Java identity are integration constraints, not new products.
### Evidence ladder honesty
- **Context**: HarnessX hunter stamped LLM JSON as `static_corroboration`. Original spec forbids reporting LLM-only claims as verified.
- **Sources Consulted**: `verification.py` (`verify_finding` accepts when reachability is not `unknown`), `hunter_harness.py` evidence_level assignment, openrouter requirements Evidence Ladder / Verification.
- **Findings**:
- Pattern hits as `static_corroboration` is consistent with the ladder. LLM hits as `suspicion` is the missing rule.
- `--fail-on verified` today means reachable + static corroboration. That remains the cheap PR gate. `--fail-on worth-fixing` is the rugged gate.
- **Implications**: Do not change CWE policy. Change hunter evidence. Add fail-on value. Report sections: Inventory, Complexity map, Worth fixing.
## Architecture Pattern Evaluation
| Option | Description | Strengths | Risks / Limitations | Notes |
|--------|-------------|-----------|---------------------|-------|
| Map-then-prove on existing modes | `quick`/`standard`/`deep` = stages 1 / 1+2 / 1+2+3 | Smallest CLI blast radius; matches threat-model "deep executes in sandbox"; reuses preprocess/rank/findings | Two vocabularies (mode vs stage) until the manifest lists both | **Selected** |
| New `--stage` command family | Parallel CLI, leave modes as-is | Clearer naming | Forks CI; `deep` still a stub | Rejected |
| Clearwing-shaped full sourcehunt | Tool hunters + sanitizer images + PoC + patch | Highest detection efficiency | Fork in all but name; out of original non-goals | Rejected as this spec; remains the oracle |
| HarnessX-required agent loop | Stage 2/3 only run with the extra | Reuses last spec's seam | CI never installs it; last spec's failure | Rejected as a requirement; optional runtime only |
## Design Decisions
### Decision D1: Modes map to stages
- **Context**: Need a three-question scan without breaking CI that already passes `--mode`.
- **Selected Approach**: `quick`→stage 1, `standard`→1+2, `deep`→1+2+3. Manifest records both.
- **Rationale**: Evolutionary adoption of the CLI the project already shipped.
- **Trade-offs**: Operators must learn that `standard` now means "map" not "regex hunter pool." README must say so.
### Decision D2: Docker CLI subprocess, no SDK
- **Context**: Zero-dep core vs isolated execution.
- **Selected Approach**: `docker` binary via `subprocess`; probe with `docker info`; fake runner in unit tests.
- **Rationale**: Operator runtime, not a Python dependency. Matches threat-model flags.
- **Trade-offs**: Windows/macOS Docker Desktop variance; integration tests skip without a daemon.
### Decision D3: OpenRouter client for the tool hunter; HarnessX optional
- **Context**: Last spec hid the LLM behind an extra CI does not install.
- **Selected Approach**: Tool loop calls `OpenRouterChatClient` (or any `RankerClient`-shaped chat client) when `[models].hunter` is set. If HarnessX extra is present, `HxScanOrchestrator` may host the same tools; it is not required.
- **Rationale**: Evolutionary use of code already in tree. Stdlib HTTP.
- **Trade-offs**: Two possible runtimes until HarnessX is either used or ignored; tests mock the client.
### Decision D4: Complexity map is a new artifact
- **Context**: Ranking already exists.
- **Selected Approach**: `ComplexityMap` with `Hotspot` records. Ranking remains the hunter budget. Map is what humans and stage 3 consume.
- **Rationale**: Different question (where to look / how to test vs which file to hunt first).
### Decision D5: Worth-fixing is a gate, not an evidence level
- **Context**: Evidence ladder is already six rungs. Do not invent a seventh for CI.
- **Selected Approach**: Verdicts `triggerable|not_triggerable|already_covered|inconclusive` → `worth_fixing|not_worth_fixing`. Evidence levels unchanged. Hunter stays at `suspicion`.
- **Rationale**: CI owners need a fail-on that means "we proved it," without collapsing the ladder.
### Decision D6: Split-sink / mutation gate for stages 2–3
- **Context**: 90% recall on cheat sheets is circular.
- **Selected Approach**: New corpus + "beats pattern-hit ranking" test. Stage 1 smoke gate stays.
- **Rationale**: Directly answers the user's evaluation complaint.
### Decision D7: Ledger overlays hotspot scores, not patterns
- **Context**: Evolutionary feedback without repeating "loop must not edit patterns."
- **Selected Approach**: `.openultrasast/calibration/complexity_ledger.json` keyed by `repo_fingerprint + path + function`. Triggerable boosts, not_triggerable demotes. Cannot hide reachable severity-5 inventory from worth-fixing consideration.
- **Rationale**: Same overlay pattern as `rule_policy.json`, different object.
### Decision D8: Language recipes, not a universal executor
- **Context**: Running untrusted Python vs C vs Java is different.
- **Selected Approach**: Small recipe registry (Python and JavaScript required for MVP; C compile+run when a compiler exists in the image; Java `inconclusive` until a recipe ships). Missing recipe → `inconclusive`, not a crash.
- **Rationale**: Simplification. Do not build Clearwing's sanitizer image matrix in this spec.
## Synthesis
- **Generalization**: Inventory → attention → proof is one pipeline with a `StagePlan`. Map, hints, and regress are processors on that plan, not three products.
- **Build vs adopt**: Adopt Docker CLI and the in-tree OpenRouter client. Build the map, ledger, recipes, and split-sink corpus. Do not adopt Clearwing, docker SDK, or a cyclomatic-complexity library (nesting depth from indentation/braces is enough).
- **Simplification**: No `--stage` flag, no HarnessX requirement, no fuzz campaign, no patch oracle, no Python Docker extra.
## Risks & Mitigations
- Docker-in-Docker / rootless CI — probe, skip stage 3, degrade; do not fail stage 1.
- Hunter cost on large repos — hotspot cap + existing `[harnessx] max_cost_usd` / hardening `max_findings` / new `max_regress_candidates`.
- Untrusted snippet execution — structural deny (network, writes outside scratch, curl, docker socket mount). Never mount `/var/run/docker.sock`.
- Stage 2 still heuristic without a model — allowed; gate is ranking quality on split-sink, which heuristics can pass if tags+reachability work.
- Breaking `deep` from "exits" to "runs" — intended; document in README.
## References
- OpenUltraSAST threat model — `docs/threat-model.md`
- OpenRouter SAST harness requirements — Docker sandbox, hunter pool, evidence ladder
- HarnessX self-improving rulesets — adjacent governance; do not reopen
- Docker run isolation flags — `--network none`, `--read-only`, `--cap-drop ALL`, `--security-opt no-new-privileges`
- Clearwing sourcehunt hunter tools and sandbox — implementation oracle, not a dependency
+22
View File
@@ -0,0 +1,22 @@
{
"feature_name": "three-stage-scan",
"created_at": "2026-09-03T16:10:07Z",
"updated_at": "2026-09-03T16:47:00Z",
"language": "en",
"phase": "implementation",
"approvals": {
"requirements": {
"generated": true,
"approved": true
},
"design": {
"generated": true,
"approved": true
},
"tasks": {
"generated": true,
"approved": true
}
},
"ready_for_implementation": true
}
+203
View File
@@ -0,0 +1,203 @@
# Implementation Plan
This plan ships the three-stage scan without rebuilding CWE policy or the regex inventory. Foundation introduces the stage contract. Core builds the map, tool hunter, sandbox, and regression verdicts in parallel where boundaries do not overlap. Integration wires them onto `quick` / `standard` / `deep`. Validation adds split-sink corpora and the anti-cheat-sheet gate.
## Phase 1 — Foundation: Stage contract
- [x] 1. Introduce the stage plan and mode mapping
- [x] 1.1 Encode the three stages and the mode-to-stage table
- Depth `quick` requests only static, `standard` requests static then map, `deep` requests static then map then regress.
- A skipped stage is a structured reason, not an exception, and can be serialized into the existing degradations list.
- Observable completion: a unit test asserts the three mappings and that `quick` never includes map or regress.
- _Requirements: 1.1, 1.2, 1.3, 1.6_
- _Boundary: StagePlan_
- [x] 1.2 Add complexity and regression configuration with candidate caps
- Configuration exposes map top-k, max hunter hotspots, max regression candidates, and optional language image pins, with defaults that keep pull-request scans cheap.
- Sandbox memory, pids, timeout, network-off, and read-only workspace remain the existing sandbox settings.
- Observable completion: loading config without a file yields those defaults; an override file changes top-k and max candidates only.
- _Requirements: 9.2, 9.5_
- _Boundary: Packaging Configuration_
- _Depends: 1.1_
- [x] 1.3 Expand fail-on vocabulary with worth-fixing
- The scan command accepts `worth-fixing` alongside the existing fail-on values and does not change the meaning of `findings` or `verified`.
- The existing report exit helper must recognize the new value; unknown values still fail loud.
- MCP continues to reject `deep` and must not gain docker or shell tools.
- Observable completion: `--help` lists `worth-fixing`; a scan with no worth-fixing verdicts and `--fail-on worth-fixing` exits 0; MCP still refuses `deep`.
- _Requirements: 5.7, 5.8_
- _Boundary: StagePlan_
- _Depends: 1.1_
## Phase 2 — Core: Map, hunter tools, sandbox (parallel after foundation)
- [x] 2. Build the heuristic complexity map and test-tuning hints
- [x] 2.1 (P) Compute per-file and per-function complexity signals
- Signals include size, nesting, surface tags, reachability, inventory-hit density as a feature not the score, and whether an adjacent test appears to exist.
- Function identity uses entry-point names when present and otherwise remains empty.
- Observable completion: a parser file with no pattern hit scores higher than a one-line helper that matches a pattern, proven by a unit test on a constructed target list.
- _Requirements: 3.1, 3.2_
- _Boundary: ComplexityMapBuilder_
- [x] 2.2 (P) Emit the ordered hotspot map artifact
- Each hotspot carries score, band, rationale, signals, and linked inventory ids; the map records whether it is heuristic-only.
- Observable completion: building a map writes a JSON artifact whose hotspot order is not identical to sorting by inventory-hit count on the same fixture.
- _Requirements: 3.2, 3.3, 3.5, 8.6_
- _Boundary: ComplexityMapBuilder_
- _Depends: 2.1_
- [x] 2.3 Attach test-gap hints without writing into the scanned tree
- Each hotspot gets a gap of no adjacent test, no function reference, no fuzz entry, or covered.
- Recommended test kind is one of unit, property, sanitizer, http-contract, or fuzz-harness, and is omitted when the gap is covered.
- Observable completion: a covered hotspot emits `covered` and no test kind; a fuzzable C parser without a harness emits `fuzz-harness`; the scanned tree is unchanged.
- _Requirements: 4.1, 4.2, 4.3, 4.4_
- _Boundary: TestTuningAdvisor_
- _Depends: 2.2_
- [x] 2.4 Apply a complexity ledger overlay that cannot hide severity-5 reachable inventory
- Triggerable verdicts raise a hotspot's later score; not-triggerable verdicts lower it; the underlying inventory hit remains.
- A reachable policy-severity-5 inventory finding stays in the regression candidate set regardless of a negative ledger delta.
- Observable completion: after a not-triggerable overlay the hotspot score drops, and a sev-5 reachable finding is still selected as a candidate, proven by a unit test.
- _Requirements: 7.1, 7.2, 7.3, 7.4, 7.5_
- _Boundary: ComplexityMapBuilder_
- _Depends: 2.2_
- [x] 3. Add repo-bound hunter tools and a scripted tool loop
- [x] 3.1 (P) Clamp tool paths and expose read, grep, and reference search
- Read, grep, and reference listing resolve only under the scan root; `..` and absolute escapes raise a path-escape error.
- Grep runs with the standard library over enumerated source files.
- Observable completion: a unit test reads a repo file, greps a split-sink assignment, and rejects `../etc/passwd`.
- _Requirements: 6.1, 6.2_
- _Boundary: HunterTools_
- [x] 3.2 Run a bounded tool-using hunter that records suspicion only
- When a hunter model is configured, the loop may call the three tools against map hotspots and must not stamp results as static corroboration.
- When no model is configured, the loop does not run and a degradation is produced by the orchestrator later.
- Observable completion: a scripted client that never calls a tool fails the test; a client that greps then returns a finding yields evidence level `suspicion` and id prefix `tool-hunter:`.
- _Requirements: 6.1, 6.3, 6.4, 6.5_
- _Boundary: ToolHunter_
- _Depends: 3.1, 2.2_
- [x] 4. Stand up the isolated sandbox runner
- [x] 4.1 (P) Probe whether the sandbox runtime is usable
- A short `docker info` style probe returns available or unavailable without throwing into the scan.
- Observable completion: the fake probe can be forced off; the real probe is skippable in unit tests.
- _Requirements: 9.3_
- _Boundary: SandboxProbe_
- [x] 4.2 (P) Run jobs through Docker CLI flags that match the threat model
- Network is disabled, source is read-only, scratch is writable, memory/pids/timeout apply, capabilities are dropped, source tree is not writable, and the host Docker socket is never mounted.
- A fake runner records the job and returns a programmed result so default tests do not need a daemon.
- Observable completion: the constructed command line includes network-none and read-only and excludes docker.sock; the fake runner is used by unit tests.
- _Requirements: 5.2, 5.8_
- _Boundary: DockerCliRunner_
- _Depends: 4.1_
- [x] 4.3 Reject unsafe regression snippets before execution
- Snippets that refer to the Docker socket, host network, or writes under the source mount are not executed and become `inconclusive`.
- Observable completion: each forbidden shape is rejected in a unit test and no runner job is created.
- _Requirements: 5.8, 6.6_
- _Boundary: RegressionRunner_
- _Depends: 4.2_
## Phase 3 — Core: Regression verdicts
- [x] 5. Produce worth-fixing verdicts from sandboxed candidates
- [x] 5.1 Select a bounded candidate set from hotspots and severe reachable inventory
- Selection honors max-candidates and always includes reachable severity-5 inventory even when the ledger demoted the hotspot.
- Observable completion: a unit test with six hotspots and max-candidates three returns three, plus the sev-5 reachable extra when present.
- _Requirements: 5.1, 7.4, 9.5_
- _Boundary: RegressionRunner_
- _Depends: 2.4_
- [x] 5.2 (P) Add language recipes with inconclusive fallback
- Python and JavaScript recipes run a scratch snippet with the repo on the language path; C compiles and runs when the image has a compiler; a missing recipe yields `inconclusive` rather than a crash.
- Observable completion: Python and JavaScript recipes produce a job; an unknown language returns inconclusive without calling the runner.
- _Requirements: 5.1, 5.6_
- _Boundary: RegressionRunner_
- _Depends: 5.1_
- [x] 5.3 Map sandbox outcomes to the four verdicts and the worth-fixing gate
- Outcomes map to triggerable, not_triggerable, already_covered, or inconclusive; worth-fixing requires triggerable plus reachable or inferred-file-surface.
- Not-triggerable and already-covered do not fail the build; inconclusive records a reason and does not fail the build.
- Observable completion: a table-driven test covers the four verdicts and the worth-fixing boolean.
- _Requirements: 5.3, 5.4, 5.5, 5.6_
- _Boundary: WorthFixingGate_
- _Depends: 5.2, 4.2_
## Phase 4 — Integration: Wire stages into the scan
- [x] 6. Run stages from the existing scan command
- [x] 6.1 Keep quick as stage-1 inventory with no model and no target execution
- Quick still loads policy, ruleset, preprocess, rank, pattern inventory, structural verify, and score.
- The report labels those hits as inventory, not as worth-fixing.
- Policy still fails loud on an unmapped enabled CWE.
- Observable completion: existing quick tests stay green; the report contains an inventory section and does not start a sandbox or a hunter model.
- _Requirements: 1.1, 2.1, 2.2, 2.3, 2.4, 2.5, 9.1_
- _Boundary: StagePlan_
- _Depends: 1.1_
- [x] 6.2 Run the complexity map on standard and record heuristic-only when no model is set
- Standard writes the map artifact, includes it in the manifest, and records `hunter_model_unavailable` when the hunter is skipped.
- Observable completion: `ousast scan --mode standard` on a fixture produces `complexity_map.json` and a completed map stage without Docker.
- _Requirements: 1.2, 1.4, 3.3, 3.4, 6.4, 9.4, 9.6_
- _Depends: 2.3, 6.1_
- [x] 6.3 Stop exiting on deep; run or skip regression with visible degradation
- Deep no longer hard-exits. If the sandbox is missing, stages 1–2 complete and regress is skipped with `sandbox_unavailable`.
- If the sandbox is present, regression runs under the candidate cap and writes verdicts.
- Observable completion: `ousast scan --mode deep` without Docker exits 0 after stages 1–2 and lists the skip in the manifest; with a fake runner it records verdicts.
- _Requirements: 1.3, 1.5, 5.1, 9.3, 9.6_
- _Depends: 5.3, 6.2_
- [x] 6.4 Attach the tool hunter to map hotspots when a model is configured
- Hunter findings are suspicion-only and may propose a snippet that still must pass the safety check and run only in the sandbox.
- Observable completion: with a scripted client, standard+model produces tool-hunter suspicions; a proposed snippet is executed only on deep after the safety check.
- _Requirements: 6.1, 6.3, 6.5, 6.6_
- _Depends: 3.2, 6.2_
- [x] 6.5 Surface map, hints, and worth-fixing in the report and honor fail-on
- Markdown, manifest, and SARIF remain; the report gains map and worth-fixing sections.
- `--fail-on worth-fixing` fails only when a worth-fixing verdict exists.
- Observable completion: a scan with one worth-fixing verdict fails under that flag and passes under `--fail-on never`.
- _Requirements: 2.2, 4.1, 5.7, 7.3_
- _Depends: 6.3_
## Phase 5 — Validation: Grounded corpora and CI
- [x] 7. Replace cheat-sheet success for stages 2 and 3
- [x] 7.1 Add split-sink fixtures for Python, JavaScript, Java, and C/C++
- Each language family has at least one case where source and sink are on different lines or assigned before use.
- Observable completion: stage-1 pattern scan misses at least one planted sink in each family, documented in the manifest expected set.
- _Requirements: 8.1_
- _Boundary: Benchmark Harness_
- [x] 7.2 (P) Add a mutation of a known sink that a same-line pattern does not match
- At least one wrapper or intermediate-variable mutation exists per the evaluation plan.
- Observable completion: stage 1 misses the mutation; the case is listed as expected in a split-sink or mutation manifest.
- _Requirements: 8.2_
- _Boundary: Benchmark Harness_
- [x] 7.3 Gate stage 2 on ranking the true function in the high band and beating pattern-hit order
- Stage 1 may still use existing smoke fixtures; that recall is not treated as stage-2 success.
- The merge gate fails if map order equals pattern-hit-count order on the split-sink corpus.
- Observable completion: a dedicated gate command or test fails a stub map that sorts by hit count and passes the heuristic map on the split-sink corpus.
- _Requirements: 8.3, 8.4, 8.6_
- _Depends: 7.1, 2.2_
- [x] 7.4 Prove stage 3 can trigger a planted miss when the sandbox is available
- Docker-marked tests skip when the probe fails; when they run, at least one stage-1 miss is `triggerable`.
- Default pytest does not require a daemon. The pull-request job stays stage 1; a nightly or labeled job may request deep under time and candidate caps.
- Observable completion: fake-runner tests always run; docker tests skip-or-pass; the default CI workflow still invokes quick-only detection plus the new split-sink map gate.
- _Requirements: 8.5, 9.1, 9.2_
- _Depends: 6.3, 7.1_
## Implementation Notes
- `RegressConfig.images` is a sorted tuple of `(language, image)` pairs so frozen config stays hashable.
- `scan_exit_code` duck-types `worth_fixing` on verdict objects; `_run_scan` still passes no verdicts until task 6.5. Dict-shaped verdicts would not fail the gate.
- Map `top_k` default is 20 (not named in design.md); `max_hunter_hotspots=8` and `max_candidates=5` follow design.
- `collect_signals` adjacent-test detection is false unless the caller also passes `repo_files`; task 2.2 must pass enumerated repo paths.
- ToolHunter rejects JSON dumps unless a repo tool was invoked; findings are always `suspicion` with prefix `tool-hunter:`.
- Split-sink map gate treats "high band" as the highest band the map emitted on that corpus (small fixtures often top out at medium).
- `@pytest.mark.docker` tests skip when `SandboxProbe` is unavailable; default CI does not require a daemon.
@@ -0,0 +1,13 @@
"""Mutation of os.system: the callable is aliased before use so a same-line regex misses it."""
import os
from flask import Flask, request
app = Flask(__name__)
run_cmd = os.system
@app.get("/ping")
def ping():
host = request.args["host"]
return run_cmd("ping -c 1 " + host) # CWE-78 mutation: os.system aliased before the call
@@ -0,0 +1,8 @@
#include <stdio.h>
#include <sqlite3.h>
void search(sqlite3 *db, char *term) {
char query[512];
snprintf(query, sizeof(query), "SELECT * FROM items WHERE title LIKE '%%%s%%'", term);
sqlite3_exec(db, query, 0, 0, 0); /* CWE-89 split-sink: query built on prior line (regex blind spot) */
}
@@ -0,0 +1,12 @@
package example;
import jakarta.servlet.http.HttpServletRequest;
import java.sql.Connection;
class SplitSinkServlet {
void search(HttpServletRequest request, Connection connection) throws Exception {
String term = request.getParameter("q");
String sql = String.format("select * from items where title like '%%%s%%'", term);
connection.createStatement().executeQuery(sql); // CWE-89 split-sink: sql built on prior line (regex blind spot)
}
}
@@ -0,0 +1,12 @@
// Split-sink SQL injection: query is built, then executed on the next line.
const express = require("express");
const app = express();
app.get("/search", (req, res) => {
const term = req.query.q;
const query = "select * from items where title like '%" + term + "%'";
db.query(query); // CWE-89 split-sink: query built on prior line (regex blind spot)
res.send("ok");
});
module.exports = app;
@@ -0,0 +1,11 @@
"""Split-sink SQL injection: query is built, then executed on the next line."""
from flask import Flask, request
app = Flask(__name__)
@app.get("/search")
def search(db):
term = request.args["q"]
query = "select * from items where title like '%" + term + "%'"
return db.execute(query) # CWE-89 split-sink: query built on prior line (regex blind spot)
@@ -0,0 +1,7 @@
"""Trivial eval helper: many same-line pattern hits, little complexity."""
def ping() -> str:
eval("1")
eval("2")
return "ok"
+20
View File
@@ -0,0 +1,20 @@
name = "mutation-python"
language = "python_web"
frameworks = ["flask"]
setup = ["pip install flask"]
modes = ["quick", "standard"]
known_noise = []
baselines = []
[source]
type = "local"
path = "../fixtures/mutation-python"
[[expected]]
cwe = "CWE-78"
class = "command injection"
path = "app.py"
line = 13
function = "ping"
sink = "run_cmd"
evidence = "aliased os.system runs an attacker-controlled command"
+20
View File
@@ -0,0 +1,20 @@
name = "split-sink-c"
language = "c_cpp"
frameworks = ["native"]
setup = []
modes = ["quick", "standard"]
known_noise = []
baselines = []
[source]
type = "local"
path = "../fixtures/split-sink-c"
[[expected]]
cwe = "CWE-89"
class = "sql injection"
path = "src/vuln.c"
line = 7
function = "search"
sink = "sqlite3_exec"
evidence = "sqlite3_exec runs a SQL query string built from attacker input on a prior line"
+20
View File
@@ -0,0 +1,20 @@
name = "split-sink-java"
language = "java_web"
frameworks = ["servlet"]
setup = ["mvn test"]
modes = ["quick", "standard"]
known_noise = []
baselines = []
[source]
type = "local"
path = "../fixtures/split-sink-java"
[[expected]]
cwe = "CWE-89"
class = "sql injection"
path = "src/main/java/example/SplitSinkServlet.java"
line = 10
function = "search"
sink = "executeQuery"
evidence = "executeQuery runs a SQL string built from request input on a prior line"
@@ -0,0 +1,19 @@
name = "split-sink-javascript"
language = "javascript_node_web"
frameworks = ["node", "express"]
setup = ["npm install"]
modes = ["quick", "standard"]
known_noise = []
baselines = []
[source]
type = "local"
path = "../fixtures/split-sink-javascript"
[[expected]]
cwe = "CWE-89"
class = "sql injection"
path = "src/app.js"
line = 8
sink = "query"
evidence = "query runs a SQL string built from request input on a prior line"
@@ -0,0 +1,20 @@
name = "split-sink-python"
language = "python_web"
frameworks = ["flask"]
setup = ["pip install flask"]
modes = ["quick", "standard"]
known_noise = []
baselines = []
[source]
type = "local"
path = "../fixtures/split-sink-python"
[[expected]]
cwe = "CWE-89"
class = "sql injection"
path = "app.py"
line = 11
function = "search"
sink = "execute"
evidence = "execute runs a SQL query string built from request input on a prior line"
+1
View File
@@ -51,6 +51,7 @@ async def _run_checks() -> None:
.with_exec(["uv", "run", "mypy", "src/openultrasast"]) .with_exec(["uv", "run", "mypy", "src/openultrasast"])
.with_exec(["uv", "run", "pytest"]) .with_exec(["uv", "run", "pytest"])
.with_exec(["uv", "run", "python", "-m", "openultrasast.gate"]) .with_exec(["uv", "run", "python", "-m", "openultrasast.gate"])
.with_exec(["uv", "run", "python", "-m", "openultrasast.map_gate"])
.with_exec(["uv", "run", "python", "-m", "compileall", "src", "tests"]) .with_exec(["uv", "run", "python", "-m", "compileall", "src", "tests"])
.with_exec(["uv", "build"]) .with_exec(["uv", "build"])
) )
+2 -1
View File
@@ -4,7 +4,7 @@ build-backend = "uv_build"
[project] [project]
name = "openultrasast" name = "openultrasast"
version = "1.0.0" version = "1.1.0"
description = "OpenCode-oriented SAST harness for evidence-gated security analysis" description = "OpenCode-oriented SAST harness for evidence-gated security analysis"
readme = "README.md" readme = "README.md"
requires-python = ">=3.11" requires-python = ">=3.11"
@@ -27,6 +27,7 @@ ousast = "openultrasast.cli:main"
[tool.pytest.ini_options] [tool.pytest.ini_options]
testpaths = ["tests"] testpaths = ["tests"]
pythonpath = ["src"] pythonpath = ["src"]
markers = ["docker: requires a docker daemon"]
[tool.ruff] [tool.ruff]
line-length = 140 line-length = 140
+1 -1
View File
@@ -2,4 +2,4 @@
__all__ = ["__version__"] __all__ = ["__version__"]
__version__ = "1.0.0" __version__ = "1.1.0"
+162 -7
View File
@@ -3,10 +3,11 @@ from __future__ import annotations
import argparse import argparse
import json import json
import tempfile import tempfile
from dataclasses import dataclass from dataclasses import dataclass, replace
from pathlib import Path from pathlib import Path
from time import perf_counter from time import perf_counter
from . import tool_hunter
from .benchmark import ( from .benchmark import (
create_benchmark_run, create_benchmark_run,
evaluate_benchmark, evaluate_benchmark,
@@ -26,6 +27,8 @@ from .calibration import (
write_false_positive_learnings, write_false_positive_learnings,
write_ranking_calibrations, write_ranking_calibrations,
) )
from .complexity import map as complexity_map
from .complexity.ledger import persist_verdicts
from .config import load_config from .config import load_config
from .findings import StaticFinding, quick_scan_findings, write_findings from .findings import StaticFinding, quick_scan_findings, write_findings
from .fusion import FusionDecision, fuse_findings_dispatch from .fusion import FusionDecision, fuse_findings_dispatch
@@ -40,11 +43,14 @@ from .mapping import analyze_entry_points, attach_reachability_hints, ingest_sar
from .policy import assert_rules_resolve, load_policy from .policy import assert_rules_resolve, load_policy
from .preprocess import preprocess_repository, write_preprocess_artifact from .preprocess import preprocess_repository, write_preprocess_artifact
from .rank import rank_targets, write_rankings from .rank import rank_targets, write_rankings
from .regress import CandidateVerdict, run_regression, write_verdicts
from .reports import scan_exit_code, write_manifest, write_markdown_report, write_sarif_report from .reports import scan_exit_code, write_manifest, write_markdown_report, write_sarif_report
from .ruleset import DEFAULT_RULESET_DIR, load_ruleset from .ruleset import DEFAULT_RULESET_DIR, load_ruleset
from .run import ScanRun, create_scan_run from .run import ScanRun, create_scan_run
from .sandbox import resolve_sandbox_probe, resolve_sandbox_runner
from .scoring import build_score_artifact from .scoring import build_score_artifact
from .verification import VerificationResult, write_verification_results from .stages import Stage, plan_for_mode, record_completed, record_skip, skip_as_degradation, stages_payload
from .verification import VerificationResult, verify_findings, write_verification_results
from .verify_judge import verify_findings_dispatch from .verify_judge import verify_findings_dispatch
CALIBRATION_DIR = ".openultrasast/calibration" CALIBRATION_DIR = ".openultrasast/calibration"
@@ -69,7 +75,7 @@ def main(argv: list[str] | None = None) -> int:
scan.add_argument("path", type=Path) scan.add_argument("path", type=Path)
scan.add_argument("--mode", choices=("quick", "standard", "deep"), default="quick") scan.add_argument("--mode", choices=("quick", "standard", "deep"), default="quick")
scan.add_argument("--config", type=Path, default=Path("openultrasast.toml")) scan.add_argument("--config", type=Path, default=Path("openultrasast.toml"))
scan.add_argument("--fail-on", choices=("never", "findings", "verified"), default="never") scan.add_argument("--fail-on", choices=("never", "findings", "verified", "worth-fixing"), default="never")
index = subparsers.add_parser("index", help="chunk a local repository for embedding index construction") index = subparsers.add_parser("index", help="chunk a local repository for embedding index construction")
index.add_argument("path", type=Path) index.add_argument("path", type=Path)
@@ -132,11 +138,10 @@ def _scan(path: Path, config_path: Path, mode: str, fail_on: str) -> int:
def _run_scan(path: Path, config_path: Path, mode: str, fail_on: str) -> ScanOutcome: def _run_scan(path: Path, config_path: Path, mode: str, fail_on: str) -> ScanOutcome:
if mode == "deep":
raise SystemExit("mode 'deep' is specified but sandboxed dynamic analysis is not implemented")
if not path.exists() or not path.is_dir(): if not path.exists() or not path.is_dir():
raise SystemExit(f"scan path is not a directory: {path}") raise SystemExit(f"scan path is not a directory: {path}")
plan = plan_for_mode(mode)
config = load_config(config_path if config_path.exists() else None) config = load_config(config_path if config_path.exists() else None)
# Capability gates for the optional HarnessX agentic plane. When the extra is # Capability gates for the optional HarnessX agentic plane. When the extra is
# absent (default/CI), both stay False and the deterministic path runs unchanged. # absent (default/CI), both stay False and the deterministic path runs unchanged.
@@ -299,8 +304,87 @@ def _run_scan(path: Path, config_path: Path, mode: str, fail_on: str) -> ScanOut
) )
score_path = run.root / "score.json" score_path = run.root / "score.json"
score_path.write_text(json.dumps(score_artifact.to_dict(), indent=2, sort_keys=True) + "\n") score_path.write_text(json.dumps(score_artifact.to_dict(), indent=2, sort_keys=True) + "\n")
plan = record_completed(plan, Stage.STATIC)
complexity_payload: dict[str, object] | None = None
complexity_map_path = run.root / "complexity_map.json"
built_map = None
if Stage.MAP in plan.requested:
built_map = runtime.run_stage(
"map",
lambda: complexity_map.build_complexity_map(
targets,
findings,
complexity_map_path,
repo_files=[target.path for target in targets],
ledger_path=run.target / CALIBRATION_DIR / "complexity_ledger.json",
),
)
plan = record_completed(plan, Stage.MAP)
if not hunter_model:
runtime.state["degradations"].append(skip_as_degradation(Stage.MAP, "hunter_model_unavailable"))
else:
built_map, findings, verifications = _attach_tool_hunter(
root=run.target,
built_map=built_map,
findings=findings,
verifications=verifications,
hunter_model=hunter_model,
max_hotspots=config.complexity.max_hunter_hotspots,
max_findings=max_findings,
findings_path=findings_path,
verification_path=verification_path,
complexity_map_path=complexity_map_path,
runtime=runtime,
)
complexity_payload = {
"hotspot_count": len(built_map.hotspots),
"heuristic_only": built_map.heuristic_only,
}
verdicts_path = run.root / "verdicts.json"
wrote_verdicts = False
verdict_records: tuple[CandidateVerdict, ...] = ()
if Stage.REGRESS in plan.requested:
probe = resolve_sandbox_probe()
if not probe.available():
plan = record_skip(plan, Stage.REGRESS, "sandbox_unavailable")
runtime.state["degradations"].append(skip_as_degradation(Stage.REGRESS, "sandbox_unavailable"))
else:
sandbox = resolve_sandbox_runner()
hotspots = built_map.hotspots if built_map is not None else ()
cwe_by_rule_id = {rule.rule_id: rule.cwe for rule in ruleset}
rule_cwe_by_finding = {finding.finding_id: cwe_by_rule_id.get(finding.finding_id.split(":", 1)[0], "") for finding in findings}
languages_by_path = {target.path: target.language for target in targets}
records = runtime.run_stage(
"regress",
lambda: run_regression(
hotspots,
findings,
max_candidates=config.regress.max_candidates,
policy=policy,
rule_cwe=rule_cwe_by_finding,
languages_by_path=languages_by_path,
repo_root=run.target,
sandbox=sandbox,
sandbox_limits=config.sandbox,
images=dict(config.regress.images),
),
)
write_verdicts(records, verdicts_path)
persist_verdicts(run.target / CALIBRATION_DIR / "complexity_ledger.json", records)
verdict_records = records
wrote_verdicts = True
plan = record_completed(plan, Stage.REGRESS)
worth_fixing_payload = _worth_fixing_payload(verdict_records) if wrote_verdicts else None
runtime.run_stage( runtime.run_stage(
"report", lambda: write_markdown_report(findings, markdown_path, verifications, redact=config.hardening.redact_secrets) "report",
lambda: write_markdown_report(
findings,
markdown_path,
verifications,
redact=config.hardening.redact_secrets,
complexity_map=built_map,
verdicts=verdict_records if wrote_verdicts else None,
),
) )
runtime.run_stage("sarif", lambda: write_sarif_report(findings, verifications, sarif_path)) runtime.run_stage("sarif", lambda: write_sarif_report(findings, verifications, sarif_path))
runtime.run_stage( runtime.run_stage(
@@ -317,11 +401,16 @@ def _run_scan(path: Path, config_path: Path, mode: str, fail_on: str) -> ScanOut
score=score_path, score=score_path,
trajectories=trajectories_path if trajectories else None, trajectories=trajectories_path if trajectories else None,
fusion=fusion_path if fusion_decisions else None, fusion=fusion_path if fusion_decisions else None,
complexity_map=complexity_map_path if complexity_payload is not None else None,
verdicts=verdicts_path if wrote_verdicts else None,
), ),
path=manifest_path, path=manifest_path,
score=score_artifact.to_dict(), score=score_artifact.to_dict(),
degradations=runtime.state["degradations"] or None, degradations=runtime.state["degradations"] or None,
fusion=[_fusion_summary(decision) for decision in fusion_decisions] or None, fusion=[_fusion_summary(decision) for decision in fusion_decisions] or None,
stages=stages_payload(plan),
complexity=complexity_payload,
worth_fixing=worth_fixing_payload,
), ),
) )
runtime.finish(status="succeeded") runtime.finish(status="succeeded")
@@ -332,10 +421,59 @@ def _run_scan(path: Path, config_path: Path, mode: str, fail_on: str) -> ScanOut
ranked_target_count=len(rankings), ranked_target_count=len(rankings),
finding_count=len(findings), finding_count=len(findings),
calibrations_applied=len(applied_calibrations), calibrations_applied=len(applied_calibrations),
exit_code=scan_exit_code(findings, verifications, fail_on), exit_code=scan_exit_code(findings, verifications, fail_on, worth_fixing_verdicts=verdict_records),
) )
def _attach_tool_hunter(
*,
root: Path,
built_map: complexity_map.ComplexityMap,
findings: list[StaticFinding],
verifications: list[VerificationResult],
hunter_model: str,
max_hotspots: int,
max_findings: int,
findings_path: Path,
verification_path: Path,
complexity_map_path: Path,
runtime: HarnessRuntime,
) -> tuple[complexity_map.ComplexityMap, list[StaticFinding], list[VerificationResult]]:
client = tool_hunter.resolve_hunter_client()
if client is None:
return built_map, findings, verifications
hunter_hotspots = list(built_map.hotspots[: max(max_hotspots, 0)])
hunter_findings = runtime.run_stage(
"tool_hunter",
lambda: tool_hunter.run_tool_hunter(
root,
hunter_hotspots,
client=client,
model=hunter_model,
max_steps=tool_hunter.DEFAULT_MAX_STEPS,
),
)
seen = {finding.finding_id for finding in findings}
extra = [finding for finding in hunter_findings if finding.finding_id not in seen]
if extra:
findings = findings + extra
if max_findings and len(findings) > max_findings:
if not any(entry.get("reason") == "max_findings_exceeded" for entry in runtime.state["degradations"]):
runtime.state["degradations"].append(
{"stage": "budget", "reason": "max_findings_exceeded", "requested": max_findings, "actual": len(findings)}
)
findings = findings[:max_findings]
kept = {finding.finding_id for finding in findings}
extra = [finding for finding in extra if finding.finding_id in kept]
if extra:
verifications = verifications + verify_findings(extra)
write_verification_results(verifications, verification_path)
write_findings(findings, findings_path)
built_map = replace(built_map, heuristic_only=False)
complexity_map.write_complexity_map(built_map, complexity_map_path)
return built_map, findings, verifications
def _persist_calibration_feedback( def _persist_calibration_feedback(
run: ScanRun, run: ScanRun,
ledger_path: Path, ledger_path: Path,
@@ -369,6 +507,23 @@ def _load_static_hints(sarif_paths: tuple[str, ...]) -> list[object]:
return hints return hints
def _worth_fixing_payload(verdicts: tuple[CandidateVerdict, ...]) -> dict[str, object]:
worth = [record for record in verdicts if record.worth_fixing]
return {
"count": len(worth),
"verdicts": [
{
"path": record.path,
"function_name": record.function_name,
"verdict": record.verdict,
"reason": record.reason,
"worth_fixing": True,
}
for record in worth
],
}
def _artifact_paths(**paths: Path | None) -> dict[str, Path]: def _artifact_paths(**paths: Path | None) -> dict[str, Path]:
return {name: path for name, path in paths.items() if path is not None} return {name: path for name, path in paths.items() if path is not None}
+36
View File
@@ -0,0 +1,36 @@
"""Heuristic complexity map for stage 2."""
from .hints import TestHint, attach_test_hints
from .ledger import (
LedgerEntry,
apply_overlay,
hotspot_key,
load_ledger,
must_keep_as_candidate,
persist_verdicts,
record_verdict,
select_forced_candidates,
write_ledger,
)
from .map import ComplexityMap, Hotspot, build_complexity_map, write_complexity_map
from .signals import ComplexitySignals, collect_signals
__all__ = [
"ComplexityMap",
"ComplexitySignals",
"Hotspot",
"LedgerEntry",
"TestHint",
"apply_overlay",
"attach_test_hints",
"build_complexity_map",
"collect_signals",
"hotspot_key",
"load_ledger",
"must_keep_as_candidate",
"persist_verdicts",
"record_verdict",
"select_forced_candidates",
"write_complexity_map",
"write_ledger",
]
+220
View File
@@ -0,0 +1,220 @@
from __future__ import annotations
import re
from collections.abc import Iterable, Mapping, Sequence
from dataclasses import dataclass, replace
from pathlib import Path, PurePosixPath
from typing import TYPE_CHECKING
from ..preprocess import FileTarget
from .signals import _is_test_filename
if TYPE_CHECKING:
from .map import Hotspot
GAP_NO_ADJACENT_TEST = "no_adjacent_test"
GAP_NO_FUNCTION_REFERENCE = "no_function_reference"
GAP_NO_FUZZ_ENTRY = "no_fuzz_entry"
GAP_COVERED = "covered"
GAPS = frozenset({GAP_NO_ADJACENT_TEST, GAP_NO_FUNCTION_REFERENCE, GAP_NO_FUZZ_ENTRY, GAP_COVERED})
KIND_UNIT = "unit"
KIND_PROPERTY = "property"
KIND_SANITIZER = "sanitizer"
KIND_HTTP_CONTRACT = "http-contract"
KIND_FUZZ_HARNESS = "fuzz-harness"
TEST_KINDS = frozenset({KIND_UNIT, KIND_PROPERTY, KIND_SANITIZER, KIND_HTTP_CONTRACT, KIND_FUZZ_HARNESS})
_MEMORY_UNSAFE_LANGUAGES = frozenset({"c", "cpp"})
@dataclass(frozen=True)
class TestHint:
path: str
function_name: str | None
gap: str # no_adjacent_test | no_function_reference | no_fuzz_entry | covered
test_kind: str | None
reason: str
__test__ = False
def __post_init__(self) -> None:
if self.gap not in GAPS:
raise ValueError(f"unknown test gap: {self.gap}")
if self.test_kind is not None and self.test_kind not in TEST_KINDS:
raise ValueError(f"unknown test kind: {self.test_kind}")
if self.gap == GAP_COVERED:
if self.test_kind is not None:
raise ValueError("covered gap must omit test_kind")
elif self.test_kind is None:
raise ValueError(f"{self.gap} gap requires a test_kind")
def attach_test_hints(
hotspots: Sequence[Hotspot],
targets: Sequence[FileTarget] = (),
*,
repo_files: Iterable[str] = (),
sources: Mapping[str, str] | None = None,
) -> tuple[Hotspot, ...]:
targets_by_path = {target.path: target for target in targets}
repo_paths = {str(path).replace("\\", "/") for path in repo_files}
repo_root = _repo_root(targets)
test_files = tuple(path for path in repo_paths if _is_test_filename(PurePosixPath(path).name))
advised: list[Hotspot] = []
for hotspot in hotspots:
target = targets_by_path.get(hotspot.path)
hint = advise_test_hint(
path=hotspot.path,
function_name=hotspot.function_name,
has_adjacent_test=bool(hotspot.signals.get("has_adjacent_test")),
language=_language(target, hotspot.path),
tags=_tags(target, hotspot.signals),
has_fuzz_entry_point=_has_fuzz_harness(target, hotspot.path, sources, repo_root),
function_referenced=_function_referenced(hotspot.function_name, test_files, sources, repo_root),
)
advised.append(replace(hotspot, test_hint=hint))
return tuple(advised)
def advise_test_hint(
*,
path: str,
function_name: str | None,
has_adjacent_test: bool,
language: str,
tags: Sequence[str],
has_fuzz_entry_point: bool,
function_referenced: bool,
) -> TestHint:
tag_set = tuple(tags)
if _needs_fuzz_or_sanitizer(language, tag_set) and not has_fuzz_entry_point:
gap, test_kind = GAP_NO_FUZZ_ENTRY, (KIND_FUZZ_HARNESS if "parser" in tag_set else KIND_SANITIZER)
elif not has_adjacent_test:
gap, test_kind = GAP_NO_ADJACENT_TEST, _kind_for_surface(tag_set)
elif function_name and not function_referenced:
gap, test_kind = GAP_NO_FUNCTION_REFERENCE, _kind_for_surface(tag_set)
else:
gap, test_kind = GAP_COVERED, None
return TestHint(
path=path,
function_name=function_name,
gap=gap,
test_kind=test_kind,
reason=_reason(
gap,
test_kind,
path=path,
function_name=function_name,
language=language,
tags=tag_set,
has_adjacent_test=has_adjacent_test,
),
)
def _needs_fuzz_or_sanitizer(language: str, tags: Sequence[str]) -> bool:
if language not in _MEMORY_UNSAFE_LANGUAGES:
return False
return "parser" in tags or "memory_unsafe" in tags
def _kind_for_surface(tags: Sequence[str]) -> str:
if "network_entry" in tags:
return KIND_HTTP_CONTRACT
if "parser" in tags or "deserialization" in tags:
return KIND_PROPERTY
return KIND_UNIT
def _reason(
gap: str,
test_kind: str | None,
*,
path: str,
function_name: str | None,
language: str,
tags: Sequence[str],
has_adjacent_test: bool,
) -> str:
identity = f"{path}::{function_name}" if function_name else path
tag_text = ",".join(tags) or "none"
if gap == GAP_COVERED:
return f"covering tests already exist for {identity}; has_adjacent_test=true."
if gap == GAP_NO_FUZZ_ENTRY:
return f"{language} tags={tag_text} surface {identity} has no sanitizer or fuzz harness; recommend {test_kind}."
if gap == GAP_NO_FUNCTION_REFERENCE:
return f"adjacent tests do not reference {function_name}; recommend {test_kind}."
return f"no adjacent test file for {identity}; has_adjacent_test={str(has_adjacent_test).lower()}; recommend {test_kind}."
def _language(target: FileTarget | None, path: str) -> str:
if target is not None:
return target.language
suffix = PurePosixPath(path).suffix.lower()
if suffix in {".c", ".h"}:
return "c"
if suffix in {".cc", ".cpp", ".cxx", ".hpp"}:
return "cpp"
return "unknown"
def _tags(target: FileTarget | None, signals: Mapping[str, float | int | bool | str]) -> tuple[str, ...]:
if target is not None:
return tuple(target.tags)
raw = signals.get("tags", "")
if not isinstance(raw, str) or not raw:
return ()
return tuple(tag for tag in raw.split(",") if tag)
def _has_fuzz_harness(
target: FileTarget | None,
path: str,
sources: Mapping[str, str] | None,
repo_root: Path | None,
) -> bool:
if target is not None and target.has_fuzz_entry_point:
return True
return "LLVMFuzzerTestOneInput" in _lookup_source(path, sources, repo_root)
def _function_referenced(
function_name: str | None,
test_files: Sequence[str],
sources: Mapping[str, str] | None,
repo_root: Path | None,
) -> bool:
if not function_name:
return True
pattern = re.compile(rf"\b{re.escape(function_name)}\b")
return any(pattern.search(_lookup_source(path, sources, repo_root)) for path in test_files)
def _lookup_source(path: str, sources: Mapping[str, str] | None, repo_root: Path | None) -> str:
normalized = path.replace("\\", "/")
if sources:
if normalized in sources:
return sources[normalized]
if path in sources:
return sources[path]
if repo_root is None:
return ""
candidate = repo_root / normalized
if not candidate.is_file():
return ""
try:
return candidate.read_text(errors="ignore")
except OSError:
return ""
def _repo_root(targets: Sequence[FileTarget]) -> Path | None:
for target in targets:
absolute = Path(target.absolute_path).as_posix()
relative = target.path.replace("\\", "/")
if not relative or not absolute.endswith(relative):
continue
root = Path(absolute[: -len(relative)].rstrip("/"))
if root.exists():
return root
return None
+196
View File
@@ -0,0 +1,196 @@
"""Loop-owned complexity ledger: overlay hotspot scores without hiding sev-5 inventory."""
from __future__ import annotations
import json
from collections.abc import Mapping, Sequence
from dataclasses import asdict, dataclass, replace
from pathlib import Path
from typing import TYPE_CHECKING
from ..findings import StaticFinding
if TYPE_CHECKING:
from .map import Hotspot
TRIGGERABLE_DELTA = 1.5
NOT_TRIGGERABLE_DELTA = -1.5
VERDICT_DELTAS = {
"triggerable": TRIGGERABLE_DELTA,
"not_triggerable": NOT_TRIGGERABLE_DELTA,
}
FORCED_CANDIDATE_SEVERITY = 5
FORCED_CANDIDATE_REACHABILITY = frozenset({"reachable", "inferred-file-surface"})
@dataclass(frozen=True)
class LedgerEntry:
score_delta: float
last_verdict: str
round: int = 0
def hotspot_key(path: str, function_name: str | None) -> str:
return f"{path}:{function_name or ''}"
def load_ledger(path: Path | None) -> dict[str, LedgerEntry]:
if path is None or not path.exists():
return {}
payload = json.loads(path.read_text())
entries: dict[str, LedgerEntry] = {}
for key, value in payload.items():
entry = _coerce_entry(value)
if entry is not None:
entries[str(key)] = entry
return entries
def write_ledger(path: Path, ledger: Mapping[str, LedgerEntry | Mapping[str, object]]) -> None:
payload: dict[str, dict[str, object]] = {}
for key, value in ledger.items():
entry = _coerce_entry(value)
if entry is None:
continue
payload[str(key)] = asdict(entry)
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
def persist_verdicts(path: Path, verdicts: Sequence[object]) -> dict[str, LedgerEntry]:
"""Record triggerable / not_triggerable outcomes into the loop-owned complexity ledger."""
ledger = load_ledger(path)
changed = False
for item in verdicts:
verdict = str(getattr(item, "verdict", ""))
if verdict not in VERDICT_DELTAS:
continue
raw_function = getattr(item, "function_name", None)
ledger = record_verdict(
ledger,
path=str(getattr(item, "path", "")),
function_name=raw_function if isinstance(raw_function, str) else None,
verdict=verdict,
)
changed = True
if changed:
write_ledger(path, ledger)
return ledger
def record_verdict(
ledger: Mapping[str, LedgerEntry | Mapping[str, object]],
*,
path: str,
function_name: str | None,
verdict: str,
) -> dict[str, LedgerEntry]:
updated: dict[str, LedgerEntry] = {}
for key, value in ledger.items():
entry = _coerce_entry(value)
if entry is not None:
updated[str(key)] = entry
key = hotspot_key(path, function_name)
previous = updated.get(key)
updated[key] = LedgerEntry(
score_delta=round((previous.score_delta if previous else 0.0) + VERDICT_DELTAS.get(verdict, 0.0), 4),
last_verdict=verdict,
round=(previous.round if previous else 0) + 1,
)
return updated
def apply_overlay(
hotspots: Sequence[Hotspot],
ledger: Mapping[str, LedgerEntry | Mapping[str, object]],
) -> tuple[Hotspot, ...]:
if not ledger:
return tuple(hotspots)
adjusted: list[Hotspot] = []
for hotspot in hotspots:
entry = _coerce_entry(ledger.get(hotspot_key(hotspot.path, hotspot.function_name)))
if entry is None or entry.score_delta == 0.0:
adjusted.append(hotspot)
continue
score = round(hotspot.score + entry.score_delta, 4)
signals = dict(hotspot.signals)
signals["ledger_delta"] = entry.score_delta
adjusted.append(
replace(
hotspot,
score=score,
band=_band_for(score),
signals=signals,
rationale=(
f"{hotspot.rationale} ledger overlay last_verdict={entry.last_verdict} "
f"score_delta={entry.score_delta} (inventory retained)."
),
)
)
return tuple(sorted(adjusted, key=lambda item: (-item.score, item.path, item.function_name or "")))
def must_keep_as_candidate(
finding: StaticFinding,
policy_severity: int | object,
reachability: str | None = None,
) -> bool:
status = finding.reachability_status if reachability is None else reachability
return _as_severity(policy_severity) >= FORCED_CANDIDATE_SEVERITY and status in FORCED_CANDIDATE_REACHABILITY
def select_forced_candidates(
findings: Sequence[StaticFinding],
policy: Mapping[str, object],
*,
rule_cwe: Mapping[str, str] | None = None,
) -> tuple[StaticFinding, ...]:
cwe_by_id = rule_cwe or {}
return tuple(
finding
for finding in findings
if must_keep_as_candidate(finding, _policy_severity_for(finding, policy, cwe_by_id), finding.reachability_status)
)
def _policy_severity_for(finding: StaticFinding, policy: Mapping[str, object], cwe_by_id: Mapping[str, str]) -> int:
if finding.finding_id in policy:
return _as_severity(policy[finding.finding_id])
cwe = cwe_by_id.get(finding.finding_id)
if cwe is not None and cwe in policy:
return _as_severity(policy[cwe])
return 0
def _as_severity(value: object) -> int:
if isinstance(value, bool):
return 0
if isinstance(value, int):
return value
severity = getattr(value, "severity", None)
if isinstance(severity, bool) or not isinstance(severity, int):
return 0
return severity
def _coerce_entry(value: object) -> LedgerEntry | None:
if isinstance(value, LedgerEntry):
return value
if not isinstance(value, Mapping):
return None
raw_delta = value.get("score_delta")
if isinstance(raw_delta, bool) or not isinstance(raw_delta, int | float):
return None
raw_round = value.get("round", 0)
round_n = raw_round if isinstance(raw_round, int) and not isinstance(raw_round, bool) else 0
return LedgerEntry(
score_delta=float(raw_delta),
last_verdict=str(value.get("last_verdict", "")),
round=round_n,
)
def _band_for(score: float) -> str:
from .map import _band
return _band(score)
+109
View File
@@ -0,0 +1,109 @@
from __future__ import annotations
import json
from collections.abc import Iterable, Mapping, Sequence
from dataclasses import asdict, dataclass
from pathlib import Path
from ..findings import StaticFinding
from ..preprocess import FileTarget
from .hints import TestHint, attach_test_hints
from .ledger import apply_overlay, load_ledger
from .signals import ComplexitySignals, collect_signals
# Score cutoffs for hotspot bands; inventory density is not the rank driver.
_HIGH_SCORE = 6.0
_MEDIUM_SCORE = 3.0
@dataclass(frozen=True)
class Hotspot:
path: str
function_name: str | None
score: float
band: str # high | medium | low
signals: dict[str, float | int | bool | str]
rationale: str
test_hint: TestHint | None
inventory_finding_ids: tuple[str, ...]
@dataclass(frozen=True)
class ComplexityMap:
hotspots: tuple[Hotspot, ...]
heuristic_only: bool
def build_complexity_map(
targets: Sequence[FileTarget],
findings: Sequence[StaticFinding],
output_path: Path,
*,
repo_files: Iterable[str] = (),
sources: Mapping[str, str] | None = None,
ledger_path: Path | None = None,
) -> ComplexityMap:
collected = collect_signals(targets, findings, repo_files=repo_files, sources=sources)
ids_by_path = _inventory_ids_by_path(findings)
hotspots = tuple(_hotspot_from_signals(item, ids_by_path.get(item.path, ())) for item in collected)
hotspots = attach_test_hints(hotspots, targets, repo_files=repo_files, sources=sources)
if ledger_path is not None:
hotspots = apply_overlay(hotspots, load_ledger(ledger_path))
complexity_map = ComplexityMap(hotspots=hotspots, heuristic_only=True)
write_complexity_map(complexity_map, output_path)
return complexity_map
def write_complexity_map(complexity_map: ComplexityMap, path: Path) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(asdict(complexity_map), indent=2, sort_keys=True) + "\n")
def _hotspot_from_signals(item: ComplexitySignals, inventory_finding_ids: tuple[str, ...]) -> Hotspot:
band = _band(item.score)
return Hotspot(
path=item.path,
function_name=item.function_name,
score=item.score,
band=band,
signals=_signal_payload(item),
rationale=_rationale(item, band),
test_hint=None,
inventory_finding_ids=inventory_finding_ids,
)
def _band(score: float) -> str:
if score >= _HIGH_SCORE:
return "high"
if score >= _MEDIUM_SCORE:
return "medium"
return "low"
def _signal_payload(item: ComplexitySignals) -> dict[str, float | int | bool | str]:
return {
"loc": item.loc,
"nesting": item.nesting,
"reachability": item.reachability,
"inventory_hit_count": item.inventory_hit_count,
"has_adjacent_test": item.has_adjacent_test,
"tags": ",".join(item.tags),
}
def _rationale(item: ComplexitySignals, band: str) -> str:
tags = ",".join(item.tags) or "none"
return (
f"{band} band: score {item.score} against cutoffs high>={_HIGH_SCORE}, medium>={_MEDIUM_SCORE}; "
f"loc={item.loc}, nesting={item.nesting}, tags={tags}, reachability={item.reachability}, "
f"inventory_hit_count={item.inventory_hit_count} (density is a feature not the rank driver), "
f"has_adjacent_test={str(item.has_adjacent_test).lower()}."
)
def _inventory_ids_by_path(findings: Sequence[StaticFinding]) -> dict[str, tuple[str, ...]]:
grouped: dict[str, list[str]] = {}
for finding in findings:
grouped.setdefault(finding.path, []).append(finding.finding_id)
return {path: tuple(ids) for path, ids in grouped.items()}
+232
View File
@@ -0,0 +1,232 @@
from __future__ import annotations
import math
from collections.abc import Iterable, Mapping, Sequence
from dataclasses import dataclass
from pathlib import Path, PurePosixPath
from ..findings import StaticFinding
from ..preprocess import FileTarget
# Inventory density is a recorded feature and a capped rationale term, not the rank driver.
_LOC_COEFF = 0.85
_NESTING_COEFF = 0.2
_REACHABILITY_COEFF = 0.55
_TEST_GAP_COEFF = 0.35
_INVENTORY_COEFF = 0.12
_TAG_WEIGHTS: dict[str, float] = {
"parser": 4.0,
"deserialization": 3.5,
"network_entry": 1.75,
"auth_boundary": 1.5,
"fuzzable": 1.5,
"memory_unsafe": 1.25,
"crypto": 1.25,
"syscall_entry": 1.25,
"filesystem_entry": 1.0,
}
_TEST_ROOT_PARTS = frozenset({"tests", "test", "__tests__", "spec"})
@dataclass(frozen=True)
class ComplexitySignals:
path: str
function_name: str | None
loc: int
nesting: int
tags: tuple[str, ...]
reachability: int
inventory_hit_count: int
has_adjacent_test: bool
score: float
def collect_signals(
targets: Sequence[FileTarget],
findings: Sequence[StaticFinding] = (),
repo_files: Iterable[str] = (),
sources: Mapping[str, str] | None = None,
) -> tuple[ComplexitySignals, ...]:
inventory = _inventory_counts(findings)
repo_paths = {str(path).replace("\\", "/") for path in repo_files}
collected: list[ComplexitySignals] = []
for target in targets:
text = _source_text(target, sources)
nesting = measure_nesting(text)
hit_count = inventory.get(target.path, 0)
adjacent = _has_adjacent_test(target.path, repo_paths)
tags = tuple(target.tags)
names = _function_names(target.reachability_hints)
identities: tuple[str | None, ...] = names or (None,)
for function_name in identities:
reachability = _reachability(target.reachability_hints, function_name)
collected.append(
ComplexitySignals(
path=target.path,
function_name=function_name,
loc=target.loc,
nesting=nesting,
tags=tags,
reachability=reachability,
inventory_hit_count=hit_count,
has_adjacent_test=adjacent,
score=_score(
loc=target.loc,
nesting=nesting,
tags=tags,
reachability=reachability,
inventory_hit_count=hit_count,
has_adjacent_test=adjacent,
),
)
)
return tuple(sorted(collected, key=lambda item: (-item.score, item.path, item.function_name or "")))
def measure_nesting(text: str) -> int:
if not text:
return 0
max_indent = 0
max_braces = 0
braces = 0
for raw in text.splitlines():
if not raw.strip():
continue
expanded = raw.expandtabs(4)
indent = len(expanded) - len(expanded.lstrip(" "))
max_indent = max(max_indent, indent // 4)
for char in raw:
if char == "{":
braces += 1
max_braces = max(max_braces, braces)
elif char == "}":
braces = max(0, braces - 1)
return max(max_indent, max_braces)
def _score(
*,
loc: int,
nesting: int,
tags: Sequence[str],
reachability: int,
inventory_hit_count: int,
has_adjacent_test: bool,
) -> float:
loc_term = math.log1p(max(loc, 0)) * _LOC_COEFF
nest_term = min(max(nesting, 0), 16) * _NESTING_COEFF
tag_term = sum(_TAG_WEIGHTS.get(tag, 0.0) for tag in dict.fromkeys(tags))
reach_term = max(reachability, 0) * _REACHABILITY_COEFF
gap_term = 0.0 if has_adjacent_test else _TEST_GAP_COEFF
density = inventory_hit_count / max(loc, 1)
inventory_term = min(max(density, 0.0), 1.0) * _INVENTORY_COEFF
return round(loc_term + nest_term + tag_term + reach_term + gap_term + inventory_term, 4)
def _source_text(target: FileTarget, sources: Mapping[str, str] | None) -> str:
if sources:
if target.path in sources:
return sources[target.path]
if target.absolute_path in sources:
return sources[target.absolute_path]
path = Path(target.absolute_path)
if not path.is_file():
return ""
try:
return path.read_text(errors="ignore")
except OSError:
return ""
def _inventory_counts(findings: Sequence[StaticFinding]) -> dict[str, int]:
counts: dict[str, int] = {}
for finding in findings:
counts[finding.path] = counts.get(finding.path, 0) + 1
return counts
def _function_names(hints: Sequence[Mapping[str, object]]) -> tuple[str, ...]:
names: list[str] = []
seen: set[str] = set()
for hint in hints:
name = _hint_function_name(hint)
if name is not None and name not in seen:
names.append(name)
seen.add(name)
return tuple(names)
def _hint_function_name(hint: Mapping[str, object]) -> str | None:
if "function_name" in hint:
value = hint.get("function_name")
return value if isinstance(value, str) and value else None
value = hint.get("name")
return value if isinstance(value, str) and value else None
def _reachability(hints: Sequence[Mapping[str, object]], function_name: str | None) -> int:
selected: Sequence[Mapping[str, object]] = hints
if function_name is not None:
matching = [hint for hint in hints if _hint_function_name(hint) == function_name]
if matching:
selected = matching
if not selected:
return 0
score = 2
for hint in selected:
access_level = hint.get("access_level")
kind = hint.get("kind")
if access_level == "public":
score = max(score, 5 if kind in {"route", "parser", "fuzz"} else 4)
elif access_level in {"authenticated", "contract-only/callback"}:
score = max(score, 4)
elif access_level == "role-restricted":
score = max(score, 3)
else:
score = max(score, 2)
return score
def _has_adjacent_test(path: str, repo_files: set[str]) -> bool:
source = PurePosixPath(path)
source_stem = source.stem.lower()
source_parent = _parent(source)
for candidate in repo_files:
name = PurePosixPath(candidate).name
if not _is_test_filename(name):
continue
if _tested_stem(name).lower() != source_stem:
continue
parent = _parent(PurePosixPath(candidate))
if parent == source_parent or _under_common_test_root(candidate):
return True
return False
def _parent(path: PurePosixPath) -> str:
parent = path.parent.as_posix()
return "" if parent == "." else parent
def _is_test_filename(name: str) -> bool:
if name.startswith("test_") or name.endswith("Test.java"):
return True
return PurePosixPath(name).stem.endswith("_test")
def _tested_stem(filename: str) -> str:
if filename.endswith("Test.java"):
return filename[: -len("Test.java")]
stem = PurePosixPath(filename).stem
if stem.startswith("test_"):
return stem[len("test_") :]
if stem.endswith("_test"):
return stem[: -len("_test")]
return stem
def _under_common_test_root(path: str) -> bool:
parts = PurePosixPath(path).parts
if len(parts) >= 2 and parts[0] == "src" and parts[1] == "test":
return True
return any(part in _TEST_ROOT_PARTS for part in parts[:-1])
+44
View File
@@ -96,6 +96,20 @@ class FusionConfig:
high_assurance: bool = False high_assurance: bool = False
@dataclass(frozen=True)
class ComplexityConfig:
# Stage-2 map size and hunter attention. Small defaults keep PR/nightly map work cheap.
top_k: int = 20
max_hunter_hotspots: int = 8
@dataclass(frozen=True)
class RegressConfig:
# Stage-3 candidate cap and optional per-language sandbox image pins. Isolation limits stay on SandboxConfig.
max_candidates: int = 5
images: tuple[tuple[str, str], ...] = ()
@dataclass(frozen=True) @dataclass(frozen=True)
class ResolvedConfig: class ResolvedConfig:
models: ModelConfig = ModelConfig() models: ModelConfig = ModelConfig()
@@ -109,6 +123,8 @@ class ResolvedConfig:
harnessx: HarnessxConfig = HarnessxConfig() harnessx: HarnessxConfig = HarnessxConfig()
fusion: FusionConfig = FusionConfig() fusion: FusionConfig = FusionConfig()
hardening: HardeningConfig = HardeningConfig() hardening: HardeningConfig = HardeningConfig()
complexity: ComplexityConfig = ComplexityConfig()
regress: RegressConfig = RegressConfig()
runs_dir: str = ".openultrasast/runs" runs_dir: str = ".openultrasast/runs"
@@ -130,6 +146,8 @@ def load_config(config_path: Path | None = None) -> ResolvedConfig:
harnessx=_load_harnessx(data.get("harnessx", {})), harnessx=_load_harnessx(data.get("harnessx", {})),
fusion=_load_fusion(data.get("fusion", {})), fusion=_load_fusion(data.get("fusion", {})),
hardening=_load_hardening(data.get("hardening", {})), hardening=_load_hardening(data.get("hardening", {})),
complexity=_load_complexity(data.get("complexity", {})),
regress=_load_regress(data.get("regress", {})),
runs_dir=os.environ.get("OPENULTRASAST_RUNS_DIR", ".openultrasast/runs"), runs_dir=os.environ.get("OPENULTRASAST_RUNS_DIR", ".openultrasast/runs"),
) )
@@ -251,6 +269,32 @@ def _load_fusion(value: object) -> FusionConfig:
) )
def _load_complexity(value: object) -> ComplexityConfig:
data = _section(value)
return ComplexityConfig(
top_k=_int_value(data.get("top_k"), 20),
max_hunter_hotspots=_int_value(data.get("max_hunter_hotspots"), 8),
)
def _load_regress(value: object) -> RegressConfig:
data = _section(value)
return RegressConfig(
max_candidates=_int_value(data.get("max_candidates"), 5),
images=_string_map(data.get("images")),
)
def _string_map(value: object) -> tuple[tuple[str, str], ...]:
data = _section(value)
images: list[tuple[str, str]] = []
for key, item in data.items():
pinned = _string(item)
if pinned is not None:
images.append((str(key), pinned))
return tuple(sorted(images))
def _float_value(value: object, default: float) -> float: def _float_value(value: object, default: float) -> float:
if isinstance(value, bool): if isinstance(value, bool):
return default return default
+1
View File
@@ -28,6 +28,7 @@ class StaticFinding:
tags: list[str] tags: list[str]
ranking_priority: float ranking_priority: float
status: str = "enabled" status: str = "enabled"
proposed_snippet: str | None = None
SEVERITY_LABEL = {5: "critical", 4: "high", 3: "medium", 2: "low", 1: "low", 0: "info"} SEVERITY_LABEL = {5: "critical", 4: "high", 3: "medium", 2: "low", 1: "low", 0: "info"}
+130
View File
@@ -0,0 +1,130 @@
from __future__ import annotations
import os
import re
from collections.abc import Iterable, Iterator, Mapping
from pathlib import Path
from .preprocess import enumerate_source_files
class PathEscapesRepo(ValueError):
"""Raised when a resolved tool path is not under the scan root."""
def clamp_repo_path(root: Path, user_path: str) -> Path:
resolved_root = root.resolve()
candidate = Path(user_path)
resolved = candidate.resolve() if candidate.is_absolute() else (resolved_root / candidate).resolve()
try:
common = os.path.commonpath([str(resolved_root), str(resolved)])
except ValueError as exc:
raise PathEscapesRepo(f"path {user_path!r} escapes scan root {resolved_root}") from exc
if common != str(resolved_root):
raise PathEscapesRepo(f"path {user_path!r} escapes scan root {resolved_root}")
return resolved
def read_file(root: Path, path: str, *, max_chars: int) -> str:
clamped = clamp_repo_path(root, path)
if max_chars <= 0:
return ""
return clamped.read_text(errors="ignore")[:max_chars]
def grep_repo(root: Path, pattern: str, *, max_matches: int) -> list[dict[str, object]]:
if max_matches <= 0:
return []
compiled = re.compile(pattern)
matches: list[dict[str, object]] = []
for relative, path in _iter_clamped_source_files(root):
for line_no, line in enumerate(path.read_text(errors="ignore").splitlines(), start=1):
if compiled.search(line):
matches.append({"path": relative, "line": line_no, "text": line})
if len(matches) >= max_matches:
return matches
return matches
def find_refs(root: Path, symbol: str, mapping_index: object) -> list[dict[str, object]]:
resolved_root = root.resolve()
refs: list[dict[str, object]] = []
for record in _iter_mapping_records(mapping_index):
if not _record_matches_symbol(record, symbol):
continue
raw_path = _record_path(record)
if raw_path is None:
continue
clamped = clamp_repo_path(resolved_root, raw_path)
name = _record_value(record, "name", "function_name", "symbol")
refs.append(
{
"path": clamped.relative_to(resolved_root).as_posix(),
"name": str(name) if name is not None else symbol,
"line": _as_line(_record_value(record, "line")),
"source": "mapping",
}
)
if symbol:
compiled = re.compile(rf"\b{re.escape(symbol)}\b")
for relative, path in _iter_clamped_source_files(resolved_root):
for line_no, line in enumerate(path.read_text(errors="ignore").splitlines(), start=1):
if compiled.search(line):
refs.append({"path": relative, "name": symbol, "line": line_no, "text": line, "source": "text"})
return refs
def _iter_clamped_source_files(root: Path) -> Iterator[tuple[str, Path]]:
resolved_root = root.resolve()
for path in enumerate_source_files(resolved_root):
relative = path.relative_to(resolved_root).as_posix()
yield relative, clamp_repo_path(resolved_root, relative)
def _iter_mapping_records(mapping_index: object) -> Iterable[object]:
if mapping_index is None:
return ()
if isinstance(mapping_index, Mapping):
return (mapping_index,)
if isinstance(mapping_index, (str, bytes, bytearray)):
return ()
if isinstance(mapping_index, Iterable):
return mapping_index
return (mapping_index,)
def _record_matches_symbol(record: object, symbol: str) -> bool:
for field in ("name", "function_name", "symbol"):
value = _record_value(record, field)
if value is not None and str(value) == symbol:
return True
return False
def _record_path(record: object) -> str | None:
value = record.get("path") if isinstance(record, Mapping) else getattr(record, "path", None)
if value is None:
return None
return str(value)
def _record_value(record: object, *names: str) -> object | None:
for name in names:
raw: object | None
if isinstance(record, Mapping):
if name not in record:
continue
raw = record[name]
else:
raw = getattr(record, name, None)
if raw is not None:
return raw
return None
def _as_line(value: object) -> int | None:
if isinstance(value, bool) or value is None:
return None
if isinstance(value, int):
return value
return None
+130
View File
@@ -0,0 +1,130 @@
"""Stage-2 quality gate: split-sink ranking must beat pattern-hit-count order.
The stage-1 smoke gate (``python -m openultrasast.gate``) stays on cheat-sheet
fixtures. This module is the merge gate for the complexity map: the planted
split-sink function must sit in the high band, and map order must not collapse
to sorting files by inventory-hit count.
"""
from __future__ import annotations
import sys
import tempfile
from collections.abc import Sequence
from dataclasses import dataclass
from pathlib import Path
from .benchmark import BenchmarkManifest, ExpectedFinding, load_benchmark_manifest, resolve_benchmark_source
from .complexity.map import ComplexityMap, Hotspot, build_complexity_map
from .findings import StaticFinding, quick_scan_findings
from .mapping import analyze_entry_points, attach_reachability_hints
from .preprocess import preprocess_repository
from .rank import rank_targets
SPLIT_SINK_MANIFESTS = (
"split-sink-c",
"split-sink-java",
"split-sink-javascript",
"split-sink-python",
)
DEFAULT_MANIFEST_DIR = Path("benchmarks/manifests")
@dataclass(frozen=True)
class SplitSinkScan:
name: str
manifest: BenchmarkManifest
target: Path
findings: tuple[StaticFinding, ...]
complexity_map: ComplexityMap
@dataclass(frozen=True)
class MapGateVerdict:
passed: bool
reasons: tuple[str, ...]
def scan_split_sink(name: str, manifest_dir: Path = DEFAULT_MANIFEST_DIR) -> SplitSinkScan:
manifest_path = manifest_dir / f"{name}.toml"
manifest = load_benchmark_manifest(manifest_path)
target = resolve_benchmark_source(manifest_path, manifest)
_, targets = preprocess_repository(target)
targets = attach_reachability_hints(targets, analyze_entry_points(target, targets))
findings = tuple(quick_scan_findings(target, targets, rank_targets(targets)))
with tempfile.TemporaryDirectory() as tmp:
complexity_map = build_complexity_map(
targets,
findings,
Path(tmp) / "complexity_map.json",
repo_files=[item.path for item in targets],
)
return SplitSinkScan(name=name, manifest=manifest, target=target, findings=findings, complexity_map=complexity_map)
def planted_expected(manifest: BenchmarkManifest) -> tuple[ExpectedFinding, ...]:
return tuple(item for item in manifest.expected if item.rule_id is None)
_BAND_RANK = {"high": 3, "medium": 2, "low": 1}
def planted_in_high_band(manifest: BenchmarkManifest, complexity_map: ComplexityMap) -> bool:
"""True when the planted sink sits in the highest band the map actually emitted."""
if not complexity_map.hotspots:
return False
top = max(_BAND_RANK.get(hotspot.band, 0) for hotspot in complexity_map.hotspots)
for expected in planted_expected(manifest):
matches = [hotspot for hotspot in complexity_map.hotspots if hotspot.path == expected.path]
if expected.function:
named = [hotspot for hotspot in matches if hotspot.function_name == expected.function]
if named:
matches = named
if not matches or not any(_BAND_RANK.get(hotspot.band, 0) == top for hotspot in matches):
return False
return bool(planted_expected(manifest))
def hit_count_path_order(findings: Sequence[StaticFinding], paths: Sequence[str]) -> list[str]:
counts: dict[str, int] = {}
for finding in findings:
counts[finding.path] = counts.get(finding.path, 0) + 1
unique = list(dict.fromkeys(paths))
return sorted(unique, key=lambda path: (-counts.get(path, 0), path))
def map_beats_hit_count(hotspots: Sequence[Hotspot], findings: Sequence[StaticFinding]) -> bool:
map_order = list(dict.fromkeys(hotspot.path for hotspot in hotspots))
stub_order = hit_count_path_order(findings, map_order)
return map_order != stub_order
def split_sink_map_gate(manifest_dir: Path = DEFAULT_MANIFEST_DIR) -> MapGateVerdict:
reasons: list[str] = []
python: SplitSinkScan | None = None
for name in SPLIT_SINK_MANIFESTS:
bundle = scan_split_sink(name, manifest_dir)
if name == "split-sink-python":
python = bundle
if not planted_in_high_band(bundle.manifest, bundle.complexity_map):
reasons.append(f"{name}: planted split-sink is not in the top hotspot band")
if python is None:
reasons.append("split-sink-python corpus is missing")
elif not map_beats_hit_count(python.complexity_map.hotspots, python.findings):
reasons.append("split-sink-python: map order equals pattern-hit-count order")
return MapGateVerdict(passed=not reasons, reasons=tuple(reasons))
def main() -> int:
verdict = split_sink_map_gate()
if verdict.passed:
print("split-sink map gate: PASS")
return 0
print("split-sink map gate: FAIL", file=sys.stderr)
for reason in verdict.reasons:
print(f" - {reason}", file=sys.stderr)
return 1
if __name__ == "__main__":
raise SystemExit(main())
+25 -6
View File
@@ -6,7 +6,7 @@ import re
import time import time
import urllib.error import urllib.error
import urllib.request import urllib.request
from collections.abc import Callable from collections.abc import Callable, Mapping, Sequence
from dataclasses import dataclass from dataclasses import dataclass
from typing import TypeVar, cast from typing import TypeVar, cast
@@ -61,8 +61,18 @@ class OpenRouterChatClient:
raise OpenRouterError("OPENROUTER_API_KEY is required for OpenRouter chat calls") raise OpenRouterError("OPENROUTER_API_KEY is required for OpenRouter chat calls")
return cls(api_key=api_key, base_url=os.environ.get("OPENROUTER_BASE_URL", cls.base_url)) return cls(api_key=api_key, base_url=os.environ.get("OPENROUTER_BASE_URL", cls.base_url))
def complete_json(self, *, model: str, messages: list[dict[str, str]], timeout_seconds: int = 60) -> object: def complete_chat(
payload = json.dumps({"model": model, "messages": messages, "temperature": 0}).encode() self,
*,
model: str,
messages: Sequence[Mapping[str, object]],
tools: Sequence[Mapping[str, object]] | None = None,
timeout_seconds: int = 60,
) -> dict[str, object]:
body: dict[str, object] = {"model": model, "messages": list(messages), "temperature": 0}
if tools:
body["tools"] = list(tools)
payload = json.dumps(body).encode()
request = urllib.request.Request( request = urllib.request.Request(
f"{self.base_url.rstrip('/')}/chat/completions", f"{self.base_url.rstrip('/')}/chat/completions",
data=payload, data=payload,
@@ -81,8 +91,13 @@ class OpenRouterChatClient:
response_payload = call_with_retry(_do, attempts=self.max_attempts, base_delay=self.retry_base_delay) response_payload = call_with_retry(_do, attempts=self.max_attempts, base_delay=self.retry_base_delay)
except (urllib.error.URLError, TimeoutError, json.JSONDecodeError) as exc: except (urllib.error.URLError, TimeoutError, json.JSONDecodeError) as exc:
raise OpenRouterError(f"OpenRouter chat request failed: {exc}") from exc raise OpenRouterError(f"OpenRouter chat request failed: {exc}") from exc
return _extract_message(response_payload)
content = _extract_message_content(response_payload) def complete_json(self, *, model: str, messages: list[dict[str, str]], timeout_seconds: int = 60) -> object:
message = self.complete_chat(model=model, messages=messages, timeout_seconds=timeout_seconds)
content = message.get("content")
if not isinstance(content, str) or not content.strip():
raise OpenRouterError("OpenRouter message content was empty")
return parse_json_content(content) return parse_json_content(content)
@@ -156,7 +171,7 @@ def parse_json_content(content: str) -> object:
raise OpenRouterError(f"model response was not valid JSON: {exc}") from exc raise OpenRouterError(f"model response was not valid JSON: {exc}") from exc
def _extract_message_content(payload: object) -> str: def _extract_message(payload: object) -> dict[str, object]:
if not isinstance(payload, dict): if not isinstance(payload, dict):
raise OpenRouterError("OpenRouter response must be a JSON object") raise OpenRouterError("OpenRouter response must be a JSON object")
choices = payload.get("choices") choices = payload.get("choices")
@@ -168,7 +183,11 @@ def _extract_message_content(payload: object) -> str:
message = first.get("message") message = first.get("message")
if not isinstance(message, dict): if not isinstance(message, dict):
raise OpenRouterError("OpenRouter choice did not include a message") raise OpenRouterError("OpenRouter choice did not include a message")
content = message.get("content") return message
def _extract_message_content(payload: object) -> str:
content = _extract_message(payload).get("content")
if not isinstance(content, str) or not content.strip(): if not isinstance(content, str) or not content.strip():
raise OpenRouterError("OpenRouter message content was empty") raise OpenRouterError("OpenRouter message content was empty")
return content return content
+36
View File
@@ -0,0 +1,36 @@
"""Isolated regression snippets and verdicts."""
from .candidate import CandidateVerdict, RegressionRunner, run_regression, select_candidates, write_verdicts
from .recipes import recipe_for
from .safety import UnsafeSnippetError, check_snippet_safety
from .verdict import (
ALREADY_COVERED,
INCONCLUSIVE,
MISSING_RECIPE,
NOT_TRIGGERABLE,
SAFETY_REJECTED,
TRIGGERABLE,
RegressionVerdict,
is_worth_fixing,
verdict_from_result,
)
__all__ = [
"ALREADY_COVERED",
"INCONCLUSIVE",
"MISSING_RECIPE",
"NOT_TRIGGERABLE",
"SAFETY_REJECTED",
"TRIGGERABLE",
"CandidateVerdict",
"RegressionRunner",
"RegressionVerdict",
"UnsafeSnippetError",
"check_snippet_safety",
"is_worth_fixing",
"recipe_for",
"run_regression",
"select_candidates",
"verdict_from_result",
"write_verdicts",
]
+247
View File
@@ -0,0 +1,247 @@
"""Candidate selection and safety-gated regression execution."""
from __future__ import annotations
import json
from collections.abc import Mapping, Sequence
from dataclasses import asdict, dataclass, replace
from pathlib import Path
from ..complexity.ledger import hotspot_key, select_forced_candidates
from ..complexity.map import Hotspot
from ..config import SandboxConfig
from ..findings import StaticFinding
from ..preprocess import LANGUAGE_BY_EXTENSION
from ..sandbox import SandboxJob, SandboxResult, SandboxRunner
from ..sandbox.runner import WORKSPACE_MOUNT
from .recipes import recipe_for
from .safety import UnsafeSnippetError, check_snippet_safety
from .verdict import (
INCONCLUSIVE,
MISSING_RECIPE,
NOT_TRIGGERABLE,
SAFETY_REJECTED,
TRIGGERABLE,
RegressionVerdict,
is_worth_fixing,
verdict_from_result,
)
DEFAULT_IMAGES = {
"python": "python:3.12-alpine",
"javascript": "node:22-alpine",
"c": "gcc:13",
}
__all__ = [
"INCONCLUSIVE",
"MISSING_RECIPE",
"NOT_TRIGGERABLE",
"SAFETY_REJECTED",
"TRIGGERABLE",
"CandidateVerdict",
"RegressionRunner",
"RegressionVerdict",
"run_regression",
"select_candidates",
"snippet_for",
"write_verdicts",
]
@dataclass(frozen=True)
class CandidateVerdict:
path: str
function_name: str | None
language: str
verdict: str
reason: str
inventory_finding_ids: tuple[str, ...]
worth_fixing: bool = False
def select_candidates(
hotspots: Sequence[Hotspot],
findings: Sequence[StaticFinding],
*,
max_candidates: int,
policy_severity_by_id: Mapping[str, object] | None = None,
rule_cwe: Mapping[str, str] | None = None,
) -> tuple[Hotspot, ...]:
"""Select min(max_candidates, ranked hotspots) ∪ sev-5 reachable inventory.
The cap applies to the ranked hotspot slice. Forced policy-severity-5
reachable (or inferred-file-surface) inventory stays in the set even when a
ledger overlay demoted its hotspot out of that slice.
"""
ranked = tuple(sorted(hotspots, key=lambda item: (-item.score, item.path, item.function_name or "")))
selected = list(ranked[: max(max_candidates, 0)])
seen = {hotspot_key(item.path, item.function_name) for item in selected}
extras: list[Hotspot] = []
for finding in select_forced_candidates(findings, policy_severity_by_id or {}, rule_cwe=rule_cwe):
hotspot = _hotspot_for_finding(finding, ranked) or _hotspot_from_forced_finding(finding)
key = hotspot_key(hotspot.path, hotspot.function_name)
if key in seen:
continue
seen.add(key)
extras.append(hotspot)
return tuple(selected + extras)
def _hotspot_for_finding(finding: StaticFinding, hotspots: Sequence[Hotspot]) -> Hotspot | None:
for hotspot in hotspots:
if finding.finding_id in hotspot.inventory_finding_ids:
return hotspot
key = hotspot_key(finding.path, finding.function_name)
for hotspot in hotspots:
if hotspot_key(hotspot.path, hotspot.function_name) == key:
return hotspot
return None
def _hotspot_from_forced_finding(finding: StaticFinding) -> Hotspot:
return Hotspot(
path=finding.path,
function_name=finding.function_name,
score=0.0,
band="low",
signals={},
rationale="forced reachable severity-5 inventory",
test_hint=None,
inventory_finding_ids=(finding.finding_id,),
)
class RegressionRunner:
"""Run a snippet in the sandbox only after the structural safety check passes."""
def __init__(self, sandbox: SandboxRunner) -> None:
self._sandbox = sandbox
def run_recipe(
self,
language: str,
snippet: str,
image: str,
sandbox_limits: SandboxConfig,
*,
repo_root: Path,
) -> RegressionVerdict:
job = recipe_for(language, snippet, image, sandbox_limits, repo_root=repo_root)
if job is None:
return verdict_from_result(recipe_missing=True)
return self.run_snippet(snippet, job)
def run_snippet(self, snippet: str, job: SandboxJob | None) -> RegressionVerdict:
if job is None:
return verdict_from_result(recipe_missing=True)
try:
check_snippet_safety(snippet)
except UnsafeSnippetError:
return verdict_from_result(safety_rejected=True)
return _verdict_from_result(self._sandbox.run(job))
def _verdict_from_result(result: SandboxResult) -> RegressionVerdict:
return verdict_from_result(result)
def snippet_for(language: str, path: str, function_name: str | None) -> str:
"""Render a scratch snippet that loads the candidate source inside the sandbox."""
relative = path.replace("\\", "/")
workspace_path = f"{WORKSPACE_MOUNT}/{relative}"
identity = function_name or relative
if language == "python":
return (
f"# candidate {identity}\n"
"from pathlib import Path\n"
f"source = Path({workspace_path!r}).read_text()\n"
f"compile(source, {relative!r}, 'exec')\n"
)
if language == "javascript":
return (
f"// candidate {identity}\n"
"const fs = require('fs');\n"
f"const source = fs.readFileSync({workspace_path!r}, 'utf8');\n"
"if (!source.length) { process.exit(1); }\n"
)
if language == "c":
return f"/* candidate {identity} */\nint main(void) {{ return 0; }}\n"
return ""
def run_regression(
hotspots: Sequence[Hotspot],
findings: Sequence[StaticFinding],
*,
max_candidates: int,
policy: Mapping[str, object],
rule_cwe: Mapping[str, str],
languages_by_path: Mapping[str, str],
repo_root: Path,
sandbox: SandboxRunner,
sandbox_limits: SandboxConfig,
images: Mapping[str, str],
) -> tuple[CandidateVerdict, ...]:
"""Select candidates under the cap, run recipes in the sandbox, and return verdicts."""
selected = select_candidates(
hotspots,
findings,
max_candidates=max_candidates,
policy_severity_by_id=policy,
rule_cwe=rule_cwe,
)
runner = RegressionRunner(sandbox)
findings_by_id = {finding.finding_id: finding for finding in findings}
records: list[CandidateVerdict] = []
for hotspot in selected:
language = _language_for(hotspot.path, languages_by_path)
snippet = _proposed_snippet(hotspot, findings) or snippet_for(language, hotspot.path, hotspot.function_name)
image = images.get(language) or DEFAULT_IMAGES.get(language, "ousast-missing-image")
mapped = runner.run_recipe(language, snippet, image, sandbox_limits, repo_root=repo_root)
reachability = _reachability_for(hotspot, findings_by_id)
mapped = replace(mapped, worth_fixing=is_worth_fixing(mapped.verdict, reachability))
records.append(
CandidateVerdict(
path=hotspot.path,
function_name=hotspot.function_name,
language=language,
verdict=mapped.verdict,
reason=mapped.reason,
inventory_finding_ids=hotspot.inventory_finding_ids,
worth_fixing=mapped.worth_fixing,
)
)
return tuple(records)
def write_verdicts(verdicts: Sequence[CandidateVerdict], path: Path) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
payload = {"verdicts": [asdict(item) for item in verdicts]}
path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
def _proposed_snippet(hotspot: Hotspot, findings: Sequence[StaticFinding]) -> str | None:
for finding in findings:
snippet = finding.proposed_snippet
if not snippet:
continue
if finding.path == hotspot.path or finding.finding_id in hotspot.inventory_finding_ids:
return snippet
return None
def _language_for(path: str, languages_by_path: Mapping[str, str]) -> str:
if path in languages_by_path:
return languages_by_path[path]
return LANGUAGE_BY_EXTENSION.get(Path(path).suffix.lower(), "unknown")
def _reachability_for(hotspot: Hotspot, findings_by_id: Mapping[str, StaticFinding]) -> str:
statuses = [
findings_by_id[finding_id].reachability_status for finding_id in hotspot.inventory_finding_ids if finding_id in findings_by_id
]
for status in statuses:
if is_worth_fixing(TRIGGERABLE, status):
return status
return statuses[0] if statuses else "unknown"
+71
View File
@@ -0,0 +1,71 @@
"""Language command recipes for isolated regression snippets."""
from __future__ import annotations
from pathlib import Path
from ..config import SandboxConfig
from ..sandbox.runner import SCRATCH_MOUNT, WORKSPACE_MOUNT, SandboxJob
_PYTHON_FILE = "case.py"
_JS_FILE = "case.js"
_C_FILE = "case.c"
_C_BIN = "case"
def recipe_for(
language: str,
snippet: str,
image: str,
sandbox_limits: SandboxConfig,
*,
repo_root: Path,
) -> SandboxJob | None:
"""Return a sandbox job for `language`, or None when no recipe exists."""
key = language.strip().lower()
if key == "python":
return _job(
image,
sandbox_limits,
repo_root,
command=("env", f"PYTHONPATH={WORKSPACE_MOUNT}", "python", f"{SCRATCH_MOUNT}/{_PYTHON_FILE}"),
scratch_files={_PYTHON_FILE: snippet},
)
if key == "javascript":
return _job(
image,
sandbox_limits,
repo_root,
command=("env", f"NODE_PATH={WORKSPACE_MOUNT}", "node", f"{SCRATCH_MOUNT}/{_JS_FILE}"),
scratch_files={_JS_FILE: snippet},
)
if key == "c":
source = f"{SCRATCH_MOUNT}/{_C_FILE}"
binary = f"{SCRATCH_MOUNT}/{_C_BIN}"
return _job(
image,
sandbox_limits,
repo_root,
command=("sh", "-c", f"cc {source} -o {binary} && {binary}"),
scratch_files={_C_FILE: snippet},
)
return None
def _job(
image: str,
sandbox_limits: SandboxConfig,
repo_root: Path,
*,
command: tuple[str, ...],
scratch_files: dict[str, str],
) -> SandboxJob:
return SandboxJob(
image=image,
command=command,
repo_root=repo_root,
scratch_files=scratch_files,
timeout_seconds=sandbox_limits.timeout_seconds,
memory_mb=sandbox_limits.memory_mb,
pids_limit=sandbox_limits.pids_limit,
)
+81
View File
@@ -0,0 +1,81 @@
"""Structural deny-list for untrusted regression snippets.
Snippets are data for `/scratch` inside the sandbox. Mentions of sockets, curl,
host networking, or writes under the source mount (`/workspace`) are rejected
before any `SandboxRunner.run` call.
"""
from __future__ import annotations
import re
from ..sandbox.runner import WORKSPACE_MOUNT
_WORKSPACE = re.escape(WORKSPACE_MOUNT)
class UnsafeSnippetError(ValueError):
"""Raised when a snippet fails the structural sandbox safety check."""
_HOST_NETWORK = re.compile(
r"--net(?:work)?(?:\s*=\s*|\s+|['\"]\s*,\s*['\"])host\b",
re.IGNORECASE,
)
_SOCKET_WORD = re.compile(r"\bsocket\b", re.IGNORECASE)
_CURL_WORD = re.compile(r"\bcurl\b", re.IGNORECASE)
_OPEN_WORKSPACE_WRITE = re.compile(
rf"open\s*\(\s*[fFrRbBuU]*['\"]{_WORKSPACE}[^'\"]*['\"]\s*,\s*(?:mode\s*=\s*)?[fFrRbBuU]*['\"][^'\"]*[wax+]",
re.IGNORECASE | re.DOTALL,
)
_PATH_WORKSPACE_WRITE = re.compile(
rf"Path\s*\(\s*['\"]{_WORKSPACE}[^'\"]*['\"]\s*\)\s*\.\s*"
r"(?:write_text|write_bytes|touch|mkdir|unlink|rmdir|replace|chmod|rename|open)\b",
re.IGNORECASE | re.DOTALL,
)
_FS_WORKSPACE_WRITE = re.compile(
r"(?:writeFile(?:Sync)?|appendFile(?:Sync)?|mkdir(?:Sync)?|rm(?:dir)?(?:Sync)?|"
rf"unlink(?:Sync)?|rename(?:Sync)?)\s*\(\s*['\"]{_WORKSPACE}",
re.IGNORECASE,
)
_FOPEN_WORKSPACE_WRITE = re.compile(
rf"fopen\s*\(\s*['\"]{_WORKSPACE}[^'\"]*['\"]\s*,\s*['\"][^'\"]*[wax+]",
re.IGNORECASE | re.DOTALL,
)
_SHUTIL_OS_WORKSPACE_WRITE = re.compile(
r"(?:shutil\.(?:copy(?:2)?|copyfile|copytree|move|rmtree)|"
r"os\.(?:remove|unlink|replace|rename|mkdir|makedirs|rmdir|chmod))"
rf"\s*\([^;]{{0,240}}{_WORKSPACE}",
re.IGNORECASE | re.DOTALL,
)
_REDIRECT_WORKSPACE = re.compile(rf"(?:>>|>)\s*['\"]?{_WORKSPACE}\b", re.IGNORECASE)
_WORKSPACE_WRITES = (
_OPEN_WORKSPACE_WRITE,
_PATH_WORKSPACE_WRITE,
_FS_WORKSPACE_WRITE,
_FOPEN_WORKSPACE_WRITE,
_SHUTIL_OS_WORKSPACE_WRITE,
_REDIRECT_WORKSPACE,
)
def check_snippet_safety(snippet: str) -> None:
"""Return None when `snippet` is structurally safe; raise otherwise."""
lowered = snippet.lower()
if "docker.sock" in lowered:
raise UnsafeSnippetError("snippet mentions the Docker socket")
if _SOCKET_WORD.search(snippet) is not None:
raise UnsafeSnippetError("snippet mentions socket")
if _CURL_WORD.search(snippet) is not None:
raise UnsafeSnippetError("snippet mentions curl")
if _HOST_NETWORK.search(snippet) is not None:
raise UnsafeSnippetError("snippet requests host network")
if _writes_under_workspace(snippet):
raise UnsafeSnippetError(f"snippet writes under {WORKSPACE_MOUNT}")
def _writes_under_workspace(snippet: str) -> bool:
if WORKSPACE_MOUNT not in snippet.lower():
return False
return any(pattern.search(snippet) is not None for pattern in _WORKSPACE_WRITES)
+95
View File
@@ -0,0 +1,95 @@
"""Map sandbox outcomes to verdicts and the worth-fixing gate."""
from __future__ import annotations
from dataclasses import dataclass
from ..sandbox import SandboxResult
TRIGGERABLE = "triggerable"
NOT_TRIGGERABLE = "not_triggerable"
ALREADY_COVERED = "already_covered"
INCONCLUSIVE = "inconclusive"
MISSING_RECIPE = "missing_recipe"
SAFETY_REJECTED = "safety_rejected"
SANDBOX_UNAVAILABLE = "sandbox_unavailable"
TIMEOUT = "timeout"
COVERED_TEST_PASSED = "covered_test_passed"
ASSERTION = "assertion"
SANITIZER_ABORT = "sanitizer_abort"
CRASH = "crash"
NONZERO_EXIT = "nonzero_exit"
EXIT_ZERO = "exit_zero"
WORTH_FIXING_REACHABILITY = frozenset({"reachable", "inferred-file-surface"})
_SANITIZER_MARKERS = (
"AddressSanitizer",
"UndefinedBehaviorSanitizer",
"ThreadSanitizer",
"MemorySanitizer",
"LeakSanitizer",
)
_ASSERTION_MARKERS = (
"AssertionError",
"Assertion failed",
"assertion failed",
"assert failed",
)
_CRASH_MARKERS = (
"Fatal Python error",
"Segmentation fault",
"SIGSEGV",
"SIGABRT",
"Aborted",
)
@dataclass(frozen=True)
class RegressionVerdict:
verdict: str
reason: str = ""
worth_fixing: bool = False
def is_worth_fixing(verdict: str, reachability: str) -> bool:
"""True iff the candidate is triggerable on a reachable or inferred-file-surface finding."""
return verdict == TRIGGERABLE and reachability in WORTH_FIXING_REACHABILITY
def verdict_from_result(
result: SandboxResult | None = None,
*,
recipe_missing: bool = False,
sandbox_missing: bool = False,
safety_rejected: bool = False,
covering_test: bool = False,
) -> RegressionVerdict:
"""Map a sandbox observation onto one of the four regression verdicts."""
if recipe_missing:
return RegressionVerdict(verdict=INCONCLUSIVE, reason=MISSING_RECIPE)
if sandbox_missing:
return RegressionVerdict(verdict=INCONCLUSIVE, reason=SANDBOX_UNAVAILABLE)
if safety_rejected:
return RegressionVerdict(verdict=INCONCLUSIVE, reason=SAFETY_REJECTED)
if result is None:
return RegressionVerdict(verdict=INCONCLUSIVE, reason=SANDBOX_UNAVAILABLE)
if result.timed_out:
return RegressionVerdict(verdict=INCONCLUSIVE, reason=TIMEOUT)
if covering_test and result.exit_code == 0:
return RegressionVerdict(verdict=ALREADY_COVERED, reason=COVERED_TEST_PASSED)
if result.exit_code != 0:
return RegressionVerdict(verdict=TRIGGERABLE, reason=_nonzero_reason(result))
return RegressionVerdict(verdict=NOT_TRIGGERABLE, reason=EXIT_ZERO)
def _nonzero_reason(result: SandboxResult) -> str:
text = f"{result.stdout}\n{result.stderr}".lower()
if any(marker.lower() in text for marker in _SANITIZER_MARKERS):
return SANITIZER_ABORT
if any(marker.lower() in text for marker in _ASSERTION_MARKERS):
return ASSERTION
if any(marker.lower() in text for marker in _CRASH_MARKERS):
return CRASH
return NONZERO_EXIT
+93 -3
View File
@@ -1,6 +1,7 @@
from __future__ import annotations from __future__ import annotations
import json import json
from collections.abc import Sequence
from dataclasses import asdict from dataclasses import asdict
from pathlib import Path from pathlib import Path
@@ -10,13 +11,20 @@ from .verification import VerificationResult
def write_markdown_report( def write_markdown_report(
findings: list[StaticFinding], path: Path, verifications: list[VerificationResult] | None = None, *, redact: bool = True findings: list[StaticFinding],
path: Path,
verifications: list[VerificationResult] | None = None,
*,
redact: bool = True,
complexity_map: object | None = None,
verdicts: Sequence[object] | None = None,
) -> None: ) -> None:
path.parent.mkdir(parents=True, exist_ok=True) path.parent.mkdir(parents=True, exist_ok=True)
verification_by_id = _verification_by_id(verifications or []) verification_by_id = _verification_by_id(verifications or [])
lines = ["# OpenUltraSAST Report", "", f"Findings: {len(findings)}", ""] lines = ["# OpenUltraSAST Report", "", f"Findings: {len(findings)}", "", "## Inventory", ""]
if not findings: if not findings:
lines.append("No quick-mode findings were emitted.") lines.append("No quick-mode findings were emitted.")
lines.append("")
for finding in findings: for finding in findings:
verification = verification_by_id.get(finding.finding_id) verification = verification_by_id.get(finding.finding_id)
lines.extend( lines.extend(
@@ -40,6 +48,10 @@ def write_markdown_report(
"", "",
] ]
) )
if complexity_map is not None:
_append_complexity_map(lines, complexity_map)
if verdicts is not None:
_append_worth_fixing(lines, verdicts)
report = "\n".join(lines).rstrip() + "\n" report = "\n".join(lines).rstrip() + "\n"
if redact: if redact:
from .redaction import redact_secrets from .redaction import redact_secrets
@@ -48,6 +60,68 @@ def write_markdown_report(
path.write_text(report) path.write_text(report)
def _append_complexity_map(lines: list[str], complexity_map: object) -> None:
hotspots = tuple(getattr(complexity_map, "hotspots", ()) or ())
heuristic_only = getattr(complexity_map, "heuristic_only", None)
summary = f"Hotspots: {len(hotspots)}"
if heuristic_only is not None:
summary += f" (heuristic_only={str(bool(heuristic_only)).lower()})"
lines.extend(["## Complexity map", "", summary, ""])
if not hotspots:
lines.extend(["No complexity hotspots were emitted.", ""])
return
for hotspot in hotspots:
identity = _report_identity(getattr(hotspot, "path", ""), getattr(hotspot, "function_name", None))
lines.extend(
[
f"### `{identity}`",
"",
f"- Score: `{getattr(hotspot, 'score', '')}`",
f"- Band: `{getattr(hotspot, 'band', '')}`",
]
)
hint = getattr(hotspot, "test_hint", None)
if hint is not None:
lines.append(f"- Gap: `{getattr(hint, 'gap', '')}`")
kind = getattr(hint, "test_kind", None)
if kind:
lines.append(f"- Recommended test: `{kind}`")
reason = getattr(hint, "reason", "")
if reason:
lines.append(f"- Hint: {reason}")
inventory_ids = tuple(getattr(hotspot, "inventory_finding_ids", ()) or ())
if inventory_ids:
lines.append(f"- Inventory: `{', '.join(str(item) for item in inventory_ids)}`")
rationale = str(getattr(hotspot, "rationale", "") or "")
lines.extend(["", rationale, ""] if rationale else [""])
def _append_worth_fixing(lines: list[str], verdicts: Sequence[object]) -> None:
worth = [item for item in verdicts if getattr(item, "worth_fixing", False)]
lines.extend(["## Worth fixing", "", f"Count: {len(worth)}", ""])
if not worth:
lines.extend(["No worth-fixing verdicts were produced.", ""])
return
for item in worth:
identity = _report_identity(getattr(item, "path", ""), getattr(item, "function_name", None))
lines.extend(
[
f"### `{identity}`",
"",
f"- Verdict: `{getattr(item, 'verdict', '')}`",
f"- Reason: `{getattr(item, 'reason', '')}`",
"",
]
)
def _report_identity(path: object, function_name: object) -> str:
path_text = str(path or "")
if function_name:
return f"{path_text}::{function_name}"
return path_text
def write_sarif_report(findings: list[StaticFinding], verifications: list[VerificationResult], path: Path) -> None: def write_sarif_report(findings: list[StaticFinding], verifications: list[VerificationResult], path: Path) -> None:
path.parent.mkdir(parents=True, exist_ok=True) path.parent.mkdir(parents=True, exist_ok=True)
verification_by_id = _verification_by_id(verifications) verification_by_id = _verification_by_id(verifications)
@@ -81,6 +155,9 @@ def write_manifest(
score: dict[str, object] | None = None, score: dict[str, object] | None = None,
degradations: list[dict[str, object]] | None = None, degradations: list[dict[str, object]] | None = None,
fusion: list[dict[str, object]] | None = None, fusion: list[dict[str, object]] | None = None,
stages: dict[str, object] | None = None,
complexity: dict[str, object] | None = None,
worth_fixing: dict[str, object] | None = None,
) -> None: ) -> None:
path.parent.mkdir(parents=True, exist_ok=True) path.parent.mkdir(parents=True, exist_ok=True)
verification_by_id = _verification_by_id(verifications) verification_by_id = _verification_by_id(verifications)
@@ -109,16 +186,29 @@ def write_manifest(
payload["degradations"] = degradations payload["degradations"] = degradations
if fusion is not None: if fusion is not None:
payload["fusion"] = fusion payload["fusion"] = fusion
if stages is not None:
payload["stages"] = stages
if complexity is not None:
payload["complexity"] = complexity
if worth_fixing is not None:
payload["worth_fixing"] = worth_fixing
path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n") path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
def scan_exit_code(findings: list[StaticFinding], verifications: list[VerificationResult], policy: str) -> int: def scan_exit_code(
findings: list[StaticFinding],
verifications: list[VerificationResult],
policy: str,
worth_fixing_verdicts: Sequence[object] | None = None,
) -> int:
if policy == "never": if policy == "never":
return 0 return 0
if policy == "findings": if policy == "findings":
return 1 if findings else 0 return 1 if findings else 0
if policy == "verified": if policy == "verified":
return 1 if any(result.verified for result in verifications) else 0 return 1 if any(result.verified for result in verifications) else 0
if policy == "worth-fixing":
return 1 if any(getattr(verdict, "worth_fixing", False) for verdict in worth_fixing_verdicts or ()) else 0
raise ValueError(f"unknown fail policy: {policy}") raise ValueError(f"unknown fail policy: {policy}")
+25
View File
@@ -0,0 +1,25 @@
"""Isolated Docker CLI sandbox."""
from .probe import FakeSandboxProbe, SandboxProbe, resolve_sandbox_probe
from .runner import (
DockerCliRunner,
FakeSandboxRunner,
SandboxJob,
SandboxResult,
SandboxRunner,
build_docker_argv,
resolve_sandbox_runner,
)
__all__ = [
"DockerCliRunner",
"FakeSandboxProbe",
"FakeSandboxRunner",
"SandboxJob",
"SandboxProbe",
"SandboxResult",
"SandboxRunner",
"build_docker_argv",
"resolve_sandbox_probe",
"resolve_sandbox_runner",
]
+56
View File
@@ -0,0 +1,56 @@
"""Docker CLI presence and usability probe."""
from __future__ import annotations
import os
import subprocess
from collections.abc import Callable
from subprocess import CompletedProcess
PROBE_TIMEOUT_SECONDS = 5
PROBE_ENV = "OPENULTRASAST_SANDBOX_PROBE"
_DOCKER_INFO = ("docker", "info")
_PROBE_OFF = frozenset({"0", "off", "false", "unavailable", "no"})
_PROBE_ON = frozenset({"1", "on", "true", "available", "yes"})
class SandboxProbe:
"""Report whether `docker info` succeeds without raising into the scan."""
def __init__(self, *, runner: Callable[..., CompletedProcess[str]] | None = None) -> None:
self._runner = runner
def available(self) -> bool:
run = self._runner if self._runner is not None else subprocess.run
try:
result = run(
list(_DOCKER_INFO),
capture_output=True,
text=True,
timeout=PROBE_TIMEOUT_SECONDS,
check=False,
)
return result.returncode == 0
except Exception:
return False
class FakeSandboxProbe:
"""Test double that reports a programmed availability without touching Docker."""
def __init__(self, available: bool) -> None:
self._available = available
def available(self) -> bool:
return self._available
def resolve_sandbox_probe() -> SandboxProbe | FakeSandboxProbe:
"""Return a real probe, or a fake when OPENULTRASAST_SANDBOX_PROBE is set."""
flag = os.environ.get(PROBE_ENV, "").strip().lower()
if flag in _PROBE_OFF:
return FakeSandboxProbe(False)
if flag in _PROBE_ON:
return FakeSandboxProbe(True)
return SandboxProbe()
+200
View File
@@ -0,0 +1,200 @@
"""Docker CLI sandbox runner that never mounts the host Docker socket."""
from __future__ import annotations
import os
import subprocess
import tempfile
from collections.abc import Callable, Mapping
from dataclasses import dataclass
from pathlib import Path
from subprocess import CompletedProcess
from typing import Protocol
NONROOT_USER = "65534:65534"
WORKSPACE_MOUNT = "/workspace"
SCRATCH_MOUNT = "/scratch"
SCRATCH_SEED_MOUNT = "/scratch-in"
RUNNER_ENV = "OPENULTRASAST_SANDBOX_RUNNER"
_COPY_SEED_THEN_EXEC = 'cp -r /scratch-in/. /scratch/ && exec "$@"'
_SEED_ARGV0 = "ousast-sandbox"
_FAKE_RUNNER = frozenset({"fake"})
class SandboxRunner(Protocol):
def run(self, spec: SandboxJob) -> SandboxResult: ...
@dataclass(frozen=True)
class SandboxJob:
image: str
command: tuple[str, ...]
repo_root: Path
scratch_files: dict[str, str]
timeout_seconds: int
memory_mb: int
pids_limit: int
@dataclass(frozen=True)
class SandboxResult:
exit_code: int
stdout: str
stderr: str
timed_out: bool
class FakeSandboxRunner:
"""Record jobs and return a programmed result without a Docker daemon."""
def __init__(self, result: SandboxResult | None = None) -> None:
self.jobs: list[SandboxJob] = []
self.result = result if result is not None else SandboxResult(exit_code=0, stdout="", stderr="", timed_out=False)
def run(self, spec: SandboxJob) -> SandboxResult:
self.jobs.append(spec)
return self.result
def resolve_sandbox_runner() -> SandboxRunner:
"""Return the Docker CLI runner, or a fake when OPENULTRASAST_SANDBOX_RUNNER=fake."""
flag = os.environ.get(RUNNER_ENV, "").strip().lower()
if flag in _FAKE_RUNNER:
return FakeSandboxRunner()
return DockerCliRunner()
class DockerCliRunner:
"""Run a job with Docker CLI flags that match the threat model."""
def __init__(self, *, runner: Callable[..., CompletedProcess[str]] | None = None) -> None:
self._runner = runner
def run(self, spec: SandboxJob) -> SandboxResult:
if spec.scratch_files:
with tempfile.TemporaryDirectory(prefix="ousast-scratch-") as tmp:
seed = Path(tmp)
_write_scratch_files(seed, spec.scratch_files)
return self._invoke(_argv_with_scratch_seed(spec, seed), spec)
return self._invoke(build_docker_argv(spec), spec)
def _invoke(self, argv: list[str], spec: SandboxJob) -> SandboxResult:
_reject_unsafe_argv(argv)
run = self._runner if self._runner is not None else subprocess.run
try:
completed = run(
argv,
capture_output=True,
text=True,
timeout=spec.timeout_seconds,
check=False,
)
except subprocess.TimeoutExpired as exc:
return SandboxResult(
exit_code=-1,
stdout=_as_text(exc.stdout),
stderr=_as_text(exc.stderr),
timed_out=True,
)
return SandboxResult(
exit_code=completed.returncode,
stdout=completed.stdout or "",
stderr=completed.stderr or "",
timed_out=False,
)
def build_docker_argv(job: SandboxJob) -> list[str]:
argv = [*_isolation_argv(job), job.image, *job.command]
_reject_unsafe_argv(argv)
return argv
def _isolation_argv(job: SandboxJob) -> list[str]:
repo = str(job.repo_root.resolve())
return [
"docker",
"run",
"--rm",
"--init",
"--network",
"none",
"--read-only",
"--tmpfs",
f"{SCRATCH_MOUNT}:rw,exec",
"--memory",
f"{job.memory_mb}m",
"--pids-limit",
str(job.pids_limit),
"--cap-drop",
"ALL",
"--security-opt",
"no-new-privileges",
"--user",
NONROOT_USER,
"--mount",
f"type=bind,src={repo},dst={WORKSPACE_MOUNT},ro",
]
def _argv_with_scratch_seed(job: SandboxJob, seed: Path) -> list[str]:
argv = [
*_isolation_argv(job),
"--mount",
f"type=bind,src={seed.resolve()},dst={SCRATCH_SEED_MOUNT},ro",
job.image,
"/bin/sh",
"-c",
_COPY_SEED_THEN_EXEC,
_SEED_ARGV0,
*job.command,
]
_reject_unsafe_argv(argv)
return argv
def _write_scratch_files(root: Path, files: Mapping[str, str]) -> None:
resolved_root = root.resolve()
for relative, content in files.items():
candidate = Path(relative)
if candidate.is_absolute():
raise ValueError(f"scratch path {relative!r} must be relative to {SCRATCH_MOUNT}")
destination = (resolved_root / candidate).resolve()
try:
common = os.path.commonpath([str(resolved_root), str(destination)])
except ValueError as exc:
raise ValueError(f"scratch path {relative!r} escapes {SCRATCH_MOUNT}") from exc
if common != str(resolved_root):
raise ValueError(f"scratch path {relative!r} escapes {SCRATCH_MOUNT}")
destination.parent.mkdir(parents=True, exist_ok=True)
destination.write_text(content)
_chmod_seed_tree(resolved_root)
def _chmod_seed_tree(root: Path) -> None:
# Container user 65534 must be able to traverse and read the seed bind.
for dirpath, _dirnames, filenames in os.walk(root):
Path(dirpath).chmod(0o755)
for name in filenames:
(Path(dirpath) / name).chmod(0o644)
def _reject_unsafe_argv(argv: list[str]) -> None:
joined = " ".join(argv)
if "/var/run/docker.sock" in joined:
raise ValueError("sandbox argv must not mount the host Docker socket")
if "--privileged" in argv:
raise ValueError("sandbox argv must not be privileged")
for index, item in enumerate(argv):
if item in {"--network", "--net"} and index + 1 < len(argv) and argv[index + 1] == "host":
raise ValueError("sandbox argv must not use host network")
if item in {"--network=host", "--net=host"}:
raise ValueError("sandbox argv must not use host network")
def _as_text(value: str | bytes | None) -> str:
if value is None:
return ""
if isinstance(value, bytes):
return value.decode(errors="replace")
return value
+49
View File
@@ -0,0 +1,49 @@
from __future__ import annotations
from dataclasses import dataclass, replace
from enum import StrEnum
class Stage(StrEnum):
STATIC = "static"
MAP = "map"
REGRESS = "regress"
MODE_STAGES: dict[str, tuple[Stage, ...]] = {
"quick": (Stage.STATIC,),
"standard": (Stage.STATIC, Stage.MAP),
"deep": (Stage.STATIC, Stage.MAP, Stage.REGRESS),
}
@dataclass(frozen=True)
class StagePlan:
mode: str
requested: tuple[Stage, ...]
completed: tuple[Stage, ...]
skipped: tuple[dict[str, str], ...] # stage, reason
def plan_for_mode(mode: str) -> StagePlan:
return StagePlan(mode=mode, requested=MODE_STAGES[mode], completed=(), skipped=())
def skip_as_degradation(stage: Stage, reason: str) -> dict[str, str]:
return {"stage": str(stage), "reason": reason}
def record_skip(plan: StagePlan, stage: Stage, reason: str) -> StagePlan:
return replace(plan, skipped=plan.skipped + (skip_as_degradation(stage, reason),))
def record_completed(plan: StagePlan, stage: Stage) -> StagePlan:
return replace(plan, completed=plan.completed + (stage,))
def stages_payload(plan: StagePlan) -> dict[str, object]:
return {
"requested": [str(stage) for stage in plan.requested],
"completed": [str(stage) for stage in plan.completed],
"skipped": [dict(item) for item in plan.skipped],
}
+423
View File
@@ -0,0 +1,423 @@
"""Bounded tool-using hunter over map hotspots. Findings are suspicion only."""
from __future__ import annotations
import json
import os
import re
from collections.abc import Mapping, Sequence
from dataclasses import dataclass
from pathlib import Path
from typing import Protocol
from .complexity.map import Hotspot
from .findings import StaticFinding
from .hunter_tools import PathEscapesRepo, clamp_repo_path, find_refs, grep_repo, read_file
from .provider.openrouter import OpenRouterChatClient, OpenRouterError
_DEFAULT_MAX_CHARS = 4000
_DEFAULT_MAX_MATCHES = 50
DEFAULT_MAX_STEPS = 4
CLIENT_ENV = "OPENULTRASAST_HUNTER_CLIENT"
_FINDING_ID_PREFIX = "tool-hunter:"
_HUNTER_TOOL_NAMES = frozenset({"read_file", "grep_repo", "find_refs"})
_SEVERITIES = frozenset({"critical", "high", "medium", "low", "info"})
_SCRIPTED_FLAGS = frozenset({"scripted", "script"})
_DUMP_FLAGS = frozenset({"dump", "dump-only"})
_UNSAFE_FLAGS = frozenset({"unsafe", "unsafe-snippet"})
_OPENROUTER_FLAGS = frozenset({"openrouter", "live"})
_UNSAFE_SNIPPET = 'client = docker.DockerClient(base_url="unix://var/run/docker.sock")\n'
_SCRIPTED_FINDINGS = [
{
"path": "app.py",
"line": 3,
"title": "Dynamic execution of attacker-controlled data",
"rationale": "eval/execute follows request input across the hotspot.",
"evidence_level": "static_corroboration",
}
]
HUNTER_TOOLS: list[dict[str, object]] = [
{
"type": "function",
"function": {
"name": "read_file",
"description": "Read a file inside the scanned repository root.",
"parameters": {
"type": "object",
"properties": {
"path": {"type": "string", "description": "Path relative to the repository root."},
"max_chars": {"type": "integer"},
},
"required": ["path"],
},
},
},
{
"type": "function",
"function": {
"name": "grep_repo",
"description": "Search repository source text with a Python regular expression.",
"parameters": {
"type": "object",
"properties": {
"pattern": {"type": "string"},
"max_matches": {"type": "integer"},
},
"required": ["pattern"],
},
},
},
{
"type": "function",
"function": {
"name": "find_refs",
"description": "List references to a symbol inside the scanned repository.",
"parameters": {
"type": "object",
"properties": {"symbol": {"type": "string"}},
"required": ["symbol"],
},
},
},
]
_SYSTEM_PROMPT = (
"You are a tool-using security hunter. Tools: read_file, grep_repo, find_refs. "
"Stay inside the scanned repository. Use a tool before concluding. "
"After using tools, reply with a JSON array of objects with path, line, title, and rationale. "
"These findings are suspicions, not static corroboration."
)
@dataclass(frozen=True)
class ToolCall:
id: str
name: str
arguments: dict[str, object]
@dataclass(frozen=True)
class ChatResponse:
content: str | None = None
tool_calls: tuple[ToolCall, ...] = ()
class ChatClient(Protocol):
def complete(
self,
*,
model: str,
messages: list[dict[str, object]],
tools: list[dict[str, object]],
timeout_seconds: int = 60,
) -> ChatResponse:
raise NotImplementedError
class ScriptedChatClient:
"""Deterministic ChatClient for tests and OPENULTRASAST_HUNTER_CLIENT injection."""
def __init__(self, turns: list[ChatResponse]) -> None:
self._turns = turns
self.calls: list[dict[str, object]] = []
def complete(
self,
*,
model: str,
messages: list[dict[str, object]],
tools: list[dict[str, object]],
timeout_seconds: int = 60,
) -> ChatResponse:
self.calls.append({"model": model, "messages": list(messages), "tools": tools, "timeout_seconds": timeout_seconds})
index = len(self.calls) - 1
if index >= len(self._turns):
return ChatResponse(content="")
return self._turns[index]
class OpenRouterHunterClient:
"""Adapt OpenRouterChatClient to the tool-hunter ChatClient protocol."""
def __init__(self, client: OpenRouterChatClient) -> None:
self._client = client
@classmethod
def from_env(cls) -> OpenRouterHunterClient:
return cls(OpenRouterChatClient.from_env())
def complete(
self,
*,
model: str,
messages: list[dict[str, object]],
tools: list[dict[str, object]],
timeout_seconds: int = 60,
) -> ChatResponse:
message = self._client.complete_chat(model=model, messages=messages, tools=tools, timeout_seconds=timeout_seconds)
return chat_response_from_message(message)
def scripted_hunter_client(mode: str = "scripted") -> ScriptedChatClient:
grep_turn = ChatResponse(
tool_calls=(ToolCall(id="call_grep", name="grep_repo", arguments={"pattern": "eval|execute|query", "max_matches": 10}),)
)
if mode in _DUMP_FLAGS:
return ScriptedChatClient([ChatResponse(content=json.dumps(_SCRIPTED_FINDINGS))])
if mode in _UNSAFE_FLAGS:
payload = [{**_SCRIPTED_FINDINGS[0], "snippet": _UNSAFE_SNIPPET}]
return ScriptedChatClient([grep_turn, ChatResponse(content=json.dumps(payload))])
return ScriptedChatClient([grep_turn, ChatResponse(content=json.dumps(_SCRIPTED_FINDINGS))])
def resolve_hunter_client() -> ChatClient | None:
"""Return a hunter client from env, or OpenRouter when an API key is present."""
flag = os.environ.get(CLIENT_ENV, "").strip().lower()
if flag in _SCRIPTED_FLAGS:
return scripted_hunter_client("scripted")
if flag in _DUMP_FLAGS:
return scripted_hunter_client("dump-only")
if flag in _UNSAFE_FLAGS:
return scripted_hunter_client("unsafe-snippet")
if flag in _OPENROUTER_FLAGS or (not flag and os.environ.get("OPENROUTER_API_KEY")):
try:
return OpenRouterHunterClient.from_env()
except OpenRouterError:
return None
return None
def chat_response_from_message(message: Mapping[str, object]) -> ChatResponse:
content = message.get("content")
text = content if isinstance(content, str) else None
raw_calls = message.get("tool_calls") or ()
calls: list[ToolCall] = []
if isinstance(raw_calls, Sequence) and not isinstance(raw_calls, str | bytes):
for index, item in enumerate(raw_calls):
parsed = _tool_call_from_openrouter(item, index)
if parsed is not None:
calls.append(parsed)
return ChatResponse(content=text, tool_calls=tuple(calls))
def _tool_call_from_openrouter(item: object, index: int) -> ToolCall | None:
if not isinstance(item, Mapping):
return None
function = item.get("function")
if not isinstance(function, Mapping):
return None
name = function.get("name")
if not isinstance(name, str) or not name:
return None
call_id = item.get("id")
identifier = call_id if isinstance(call_id, str) and call_id else f"call_{index}"
arguments = function.get("arguments", {})
if isinstance(arguments, str):
try:
loaded = json.loads(arguments)
except json.JSONDecodeError:
loaded = {}
parsed_args = loaded if isinstance(loaded, dict) else {}
elif isinstance(arguments, Mapping):
parsed_args = dict(arguments)
else:
parsed_args = {}
return ToolCall(id=identifier, name=name, arguments=parsed_args)
def run_tool_hunter(
root: Path,
hotspots: list[Hotspot],
*,
client: ChatClient,
model: str,
max_steps: int,
) -> list[StaticFinding]:
if max_steps <= 0:
return []
messages: list[dict[str, object]] = [
{"role": "system", "content": _SYSTEM_PROMPT},
{"role": "user", "content": _hotspot_prompt(hotspots)},
]
used_tools = False
for _step in range(max_steps):
response = client.complete(model=model, messages=messages, tools=HUNTER_TOOLS)
tool_calls = tuple(response.tool_calls or ())
if tool_calls:
messages.append(_assistant_tool_message(response.content, tool_calls))
for call in tool_calls:
if call.name in _HUNTER_TOOL_NAMES:
used_tools = True
messages.append(_tool_result_message(root, call))
continue
content = response.content or ""
if not content.strip() or not used_tools:
# A findings dump is not evidence unless the loop invoked a repo tool.
return []
return _findings_from_content(root, hotspots, content)
return []
def _hotspot_prompt(hotspots: Sequence[Hotspot]) -> str:
payload = [
{
"path": hotspot.path,
"function_name": hotspot.function_name,
"score": hotspot.score,
"band": hotspot.band,
"rationale": hotspot.rationale,
}
for hotspot in hotspots
]
return (
"Inspect these complexity-map hotspots with the repository tools. "
"Return JSON findings only after using a tool.\n"
f"{json.dumps(payload, sort_keys=True)}"
)
def _assistant_tool_message(content: str | None, tool_calls: Sequence[ToolCall]) -> dict[str, object]:
return {
"role": "assistant",
"content": content,
"tool_calls": [
{
"id": call.id,
"type": "function",
"function": {
"name": call.name,
"arguments": json.dumps(call.arguments, sort_keys=True) if not isinstance(call.arguments, str) else call.arguments,
},
}
for call in tool_calls
],
}
def _tool_result_message(root: Path, call: ToolCall) -> dict[str, object]:
return {
"role": "tool",
"tool_call_id": call.id,
"name": call.name,
"content": _run_tool(root, call),
}
def _run_tool(root: Path, call: ToolCall) -> str:
try:
payload = _dispatch_tool(root, call.name, _arguments(call.arguments))
except Exception as exc: # noqa: BLE001 — tool failures stay in-band so the scan continues
payload = {"error": f"{type(exc).__name__}: {exc}"}
return json.dumps(payload, sort_keys=True)
def _dispatch_tool(root: Path, name: str, arguments: Mapping[str, object]) -> object:
if name == "read_file":
path = _require_str(arguments, "path")
max_chars = _optional_int(arguments, "max_chars", _DEFAULT_MAX_CHARS)
return {"path": path, "content": read_file(root, path, max_chars=max_chars)}
if name == "grep_repo":
pattern = _require_str(arguments, "pattern")
max_matches = _optional_int(arguments, "max_matches", _DEFAULT_MAX_MATCHES)
return grep_repo(root, pattern, max_matches=max_matches)
if name == "find_refs":
symbol = _require_str(arguments, "symbol")
return find_refs(root, symbol, None)
raise ValueError(f"unknown tool: {name}")
def _arguments(raw: object) -> dict[str, object]:
if isinstance(raw, str):
parsed = json.loads(raw)
if isinstance(parsed, dict):
return parsed
raise ValueError("tool arguments must be a JSON object")
if isinstance(raw, Mapping):
return dict(raw)
raise ValueError("tool arguments must be an object")
def _require_str(arguments: Mapping[str, object], key: str) -> str:
value = arguments.get(key)
if not isinstance(value, str) or not value:
raise ValueError(f"{key} is required")
return value
def _optional_int(arguments: Mapping[str, object], key: str, default: int) -> int:
value = arguments.get(key, default)
if isinstance(value, bool) or not isinstance(value, int):
return default
return value
def _findings_from_content(root: Path, hotspots: Sequence[Hotspot], content: str) -> list[StaticFinding]:
payload = _load_json(content)
if isinstance(payload, list):
items = payload
elif isinstance(payload, dict) and isinstance(payload.get("findings"), list):
items = payload["findings"]
else:
return []
scores = {hotspot.path: hotspot.score for hotspot in hotspots}
findings: list[StaticFinding] = []
for item in items:
if not isinstance(item, dict):
continue
finding = _finding_from_item(root, item, scores)
if finding is not None:
findings.append(finding)
return sorted(findings, key=lambda item: item.finding_id)
def _load_json(content: str) -> object | None:
stripped = content.strip()
fenced = re.fullmatch(r"```(?:json)?\s*(.*?)\s*```", stripped, flags=re.DOTALL)
if fenced is not None:
stripped = fenced.group(1).strip()
try:
parsed: object = json.loads(stripped)
except json.JSONDecodeError:
return None
return parsed
def _finding_from_item(root: Path, item: Mapping[str, object], scores: Mapping[str, float]) -> StaticFinding | None:
raw_path = item.get("path")
if not isinstance(raw_path, str) or not raw_path:
return None
try:
relative = _clamp_relative_path(root, raw_path)
except PathEscapesRepo:
return None
line = item.get("line")
line_no = line if isinstance(line, int) and not isinstance(line, bool) else None
title = item.get("title")
rationale = item.get("rationale")
function_name = item.get("function_name")
severity = item.get("severity")
snippet = item.get("snippet")
proposed_snippet = snippet if isinstance(snippet, str) and snippet else None
return StaticFinding(
finding_id=f"{_FINDING_ID_PREFIX}{relative}:{line_no if line_no is not None else 0}",
path=relative,
title=title if isinstance(title, str) and title else "tool hunter suspicion",
severity=severity if isinstance(severity, str) and severity in _SEVERITIES else "medium",
confidence="low",
evidence_level="suspicion",
rationale=rationale if isinstance(rationale, str) and rationale else "Reported by the tool-using hunter.",
line=line_no,
function_name=function_name if isinstance(function_name, str) and function_name else None,
reachability_status="unknown",
reachability_evidence=[],
reachability_conditions=[],
tags=["tool-hunter"],
ranking_priority=float(scores.get(relative, 0.0)),
proposed_snippet=proposed_snippet,
)
def _clamp_relative_path(root: Path, user_path: str) -> str:
resolved_root = root.resolve()
return clamp_repo_path(resolved_root, user_path).relative_to(resolved_root).as_posix()
+5
View File
@@ -63,8 +63,13 @@ def test_repository_benchmark_manifests_load_and_point_to_fixtures() -> None:
"java-web-smoke.toml", "java-web-smoke.toml",
"javascript-node-web-smoke.toml", "javascript-node-web-smoke.toml",
"javascript-vulnerable.toml", "javascript-vulnerable.toml",
"mutation-python.toml",
"python-vulnerable.toml", "python-vulnerable.toml",
"python-web-smoke.toml", "python-web-smoke.toml",
"split-sink-c.toml",
"split-sink-java.toml",
"split-sink-javascript.toml",
"split-sink-python.toml",
} }
for manifest_path in manifest_paths: for manifest_path in manifest_paths:
manifest = load_benchmark_manifest(manifest_path) manifest = load_benchmark_manifest(manifest_path)
+6 -2
View File
@@ -24,6 +24,8 @@ def _run(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, config_body: str | Non
(repo / "app.py").write_text(_VULN) (repo / "app.py").write_text(_VULN)
monkeypatch.setattr(cli, "has_harnessx", lambda: False) # simulate the extra being absent monkeypatch.setattr(cli, "has_harnessx", lambda: False) # simulate the extra being absent
monkeypatch.setenv("OPENULTRASAST_RUNS_DIR", ".runs") monkeypatch.setenv("OPENULTRASAST_RUNS_DIR", ".runs")
monkeypatch.delenv("OPENROUTER_API_KEY", raising=False)
monkeypatch.delenv("OPENULTRASAST_HUNTER_CLIENT", raising=False)
argv = ["scan", str(repo), "--mode", "standard"] argv = ["scan", str(repo), "--mode", "standard"]
if config_body is not None: if config_body is not None:
cfg = tmp_path / "openultrasast.toml" cfg = tmp_path / "openultrasast.toml"
@@ -45,9 +47,11 @@ def test_degradation_recorded_when_models_set_but_extra_absent(tmp_path: Path, m
assert manifest["findings"] assert manifest["findings"]
def test_manifest_has_no_degradations_without_models(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: def test_manifest_has_no_harnessx_degradations_without_models(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
manifest = _run(tmp_path, monkeypatch, config_body=None) manifest = _run(tmp_path, monkeypatch, config_body=None)
assert "degradations" not in manifest degradations = manifest.get("degradations", [])
assert all(entry.get("reason") != "harnessx_extra_unavailable" for entry in degradations)
assert any(entry.get("reason") == "hunter_model_unavailable" for entry in degradations)
def test_standard_scan_falls_back_to_run_hunter_pool(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: def test_standard_scan_falls_back_to_run_hunter_pool(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+227
View File
@@ -0,0 +1,227 @@
import json
from pathlib import Path
from openultrasast.complexity.hints import GAP_COVERED, KIND_FUZZ_HARNESS
from openultrasast.complexity.map import build_complexity_map
from openultrasast.findings import StaticFinding
from openultrasast.preprocess import FileTarget
def _target(
path: str,
*,
loc: int,
tags: list[str] | None = None,
language: str = "python",
hints: list[dict[str, object]] | None = None,
has_fuzz_entry_point: bool = False,
absolute_path: str | None = None,
) -> FileTarget:
return FileTarget(
path=path,
absolute_path=absolute_path or f"/repo/{path}",
language=language,
loc=loc,
tags=list(tags or []),
has_fuzz_entry_point=has_fuzz_entry_point,
reachability_hints=list(hints or []),
)
def _hit(path: str, finding_id: str = "rule:helper:1") -> StaticFinding:
return StaticFinding(
finding_id=finding_id,
path=path,
title="pattern hit",
severity="low",
confidence="medium",
evidence_level="static_corroboration",
rationale="inventory",
line=1,
function_name=None,
reachability_status="unknown",
reachability_evidence=[],
reachability_conditions=[],
tags=[],
ranking_priority=1.0,
)
def _tree_snapshot(root: Path) -> dict[str, tuple[int, bytes]]:
snapshot: dict[str, tuple[int, bytes]] = {}
for path in sorted(root.rglob("*")):
if path.is_file():
snapshot[path.relative_to(root).as_posix()] = (path.stat().st_mtime_ns, path.read_bytes())
return snapshot
def test_covered_hotspot_emits_covered_and_no_test_kind(tmp_path: Path) -> None:
widget = _target(
"src/widget.py",
loc=12,
hints=[{"kind": "route", "access_level": "public", "function_name": "render"}],
)
artifact = tmp_path / "complexity_map.json"
complexity_map = build_complexity_map(
[widget],
[_hit("src/widget.py")],
artifact,
repo_files=("src/widget.py", "tests/test_widget.py"),
sources={
"src/widget.py": "def render():\n return 1\n",
"tests/test_widget.py": "from widget import render\n\ndef test_render():\n assert render() == 1\n",
},
)
hotspot = complexity_map.hotspots[0]
assert hotspot.test_hint is not None
assert hotspot.test_hint.gap == GAP_COVERED
assert hotspot.test_hint.gap == "covered"
assert hotspot.test_hint.test_kind is None
assert hotspot.test_hint.path == "src/widget.py"
assert hotspot.test_hint.function_name == "render"
payload = json.loads(artifact.read_text())
hint = payload["hotspots"][0]["test_hint"]
assert hint["gap"] == "covered"
assert hint["test_kind"] is None
def test_fuzzable_c_parser_without_harness_emits_fuzz_harness(tmp_path: Path) -> None:
parser = _target(
"src/parser.c",
loc=400,
language="c",
tags=["parser", "memory_unsafe"],
hints=[{"kind": "parser", "access_level": "public", "function_name": "parse_header"}],
has_fuzz_entry_point=False,
)
artifact = tmp_path / "complexity_map.json"
complexity_map = build_complexity_map(
[parser],
[],
artifact,
repo_files=("src/parser.c", "tests/test_parser.c"),
sources={
"src/parser.c": "int parse_header(const char *buf) { return 0; }\n",
"tests/test_parser.c": 'int test_parse_header(void) { return parse_header(""); }\n',
},
)
hotspot = complexity_map.hotspots[0]
assert hotspot.test_hint is not None
assert hotspot.test_hint.gap == "no_fuzz_entry"
assert hotspot.test_hint.test_kind == KIND_FUZZ_HARNESS
assert hotspot.test_hint.test_kind == "fuzz-harness"
assert hotspot.test_hint.path == "src/parser.c"
assert hotspot.test_hint.function_name == "parse_header"
assert hotspot.test_hint.reason
payload = json.loads(artifact.read_text())
hint = payload["hotspots"][0]["test_hint"]
assert hint["gap"] == "no_fuzz_entry"
assert hint["test_kind"] == "fuzz-harness"
def test_attach_test_hints_leaves_scanned_tree_unchanged(tmp_path: Path) -> None:
repo = tmp_path / "repo"
src = repo / "src"
tests_dir = repo / "tests"
src.mkdir(parents=True)
tests_dir.mkdir()
parser = src / "parser.c"
helper = src / "helper.py"
parser_test = tests_dir / "test_parser.c"
parser.write_text("int parse_header(const char *buf) {\n return buf[0];\n}\n")
helper.write_text("def ping():\n return 1\n")
parser_test.write_text("int test_parse_header(void) { return 0; }\n")
before = _tree_snapshot(repo)
artifact = tmp_path / "out" / "complexity_map.json"
targets = [
_target(
"src/parser.c",
loc=3,
language="c",
tags=["parser", "memory_unsafe"],
hints=[{"kind": "parser", "access_level": "public", "function_name": "parse_header"}],
absolute_path=str(parser),
),
_target("src/helper.py", loc=2, tags=[], absolute_path=str(helper)),
]
complexity_map = build_complexity_map(
targets,
[_hit("src/helper.py")],
artifact,
repo_files=("src/parser.c", "src/helper.py", "tests/test_parser.c"),
sources={
"src/parser.c": parser.read_text(),
"src/helper.py": helper.read_text(),
"tests/test_parser.c": parser_test.read_text(),
},
)
assert complexity_map.hotspots
assert all(hotspot.test_hint is not None for hotspot in complexity_map.hotspots)
assert _tree_snapshot(repo) == before
assert set(_tree_snapshot(repo)) == {"src/parser.c", "src/helper.py", "tests/test_parser.c"}
assert not any(path.as_posix().endswith(".py") and "test" in path.name for path in src.rglob("*"))
assert artifact.is_file()
assert artifact.is_relative_to(tmp_path / "out")
def test_no_adjacent_test_recommends_closed_test_kind(tmp_path: Path) -> None:
helper = _target("src/helper.py", loc=4, tags=[])
route = _target(
"src/api.py",
loc=40,
tags=["network_entry"],
hints=[{"kind": "route", "access_level": "public", "function_name": "handle"}],
)
artifact = tmp_path / "complexity_map.json"
complexity_map = build_complexity_map(
[helper, route],
[],
artifact,
repo_files=("src/helper.py", "src/api.py"),
sources={
"src/helper.py": "def ping():\n return 1\n",
"src/api.py": "def handle():\n return {}\n",
},
)
by_path = {hotspot.path: hotspot for hotspot in complexity_map.hotspots}
assert by_path["src/helper.py"].test_hint is not None
assert by_path["src/helper.py"].test_hint.gap == "no_adjacent_test"
assert by_path["src/helper.py"].test_hint.test_kind == "unit"
assert by_path["src/api.py"].test_hint is not None
assert by_path["src/api.py"].test_hint.gap == "no_adjacent_test"
assert by_path["src/api.py"].test_hint.test_kind == "http-contract"
def test_no_function_reference_when_adjacent_test_omits_function(tmp_path: Path) -> None:
widget = _target(
"src/widget.py",
loc=12,
hints=[{"kind": "route", "access_level": "public", "function_name": "render"}],
)
artifact = tmp_path / "complexity_map.json"
complexity_map = build_complexity_map(
[widget],
[],
artifact,
repo_files=("src/widget.py", "tests/test_widget.py"),
sources={
"src/widget.py": "def render():\n return 1\n",
"tests/test_widget.py": "def test_placeholder():\n assert True\n",
},
)
hint = complexity_map.hotspots[0].test_hint
assert hint is not None
assert hint.gap == "no_function_reference"
assert hint.test_kind == "unit"
assert hint.function_name == "render"
+281
View File
@@ -0,0 +1,281 @@
import json
from pathlib import Path
from types import SimpleNamespace
from openultrasast.complexity.ledger import (
NOT_TRIGGERABLE_DELTA,
TRIGGERABLE_DELTA,
LedgerEntry,
apply_overlay,
hotspot_key,
load_ledger,
must_keep_as_candidate,
persist_verdicts,
record_verdict,
select_forced_candidates,
write_ledger,
)
from openultrasast.complexity.map import Hotspot, build_complexity_map
from openultrasast.findings import StaticFinding
from openultrasast.policy import CwePolicy
from openultrasast.preprocess import FileTarget
def _target(
path: str,
*,
loc: int,
tags: list[str] | None = None,
language: str = "python",
hints: list[dict[str, object]] | None = None,
has_fuzz_entry_point: bool = False,
) -> FileTarget:
return FileTarget(
path=path,
absolute_path=f"/repo/{path}",
language=language,
loc=loc,
tags=list(tags or []),
has_fuzz_entry_point=has_fuzz_entry_point,
reachability_hints=list(hints or []),
)
def _hit(
path: str,
finding_id: str,
*,
function_name: str | None = None,
reachability: str = "unknown",
severity: str = "low",
rationale: str = "inventory",
) -> StaticFinding:
return StaticFinding(
finding_id=finding_id,
path=path,
title="pattern hit",
severity=severity,
confidence="medium",
evidence_level="static_corroboration",
rationale=rationale,
line=1,
function_name=function_name,
reachability_status=reachability,
reachability_evidence=[],
reachability_conditions=[],
tags=[],
ranking_priority=1.0,
)
def _hotspot(
path: str,
*,
function_name: str | None = None,
score: float = 4.0,
band: str = "medium",
inventory_finding_ids: tuple[str, ...] = (),
) -> Hotspot:
return Hotspot(
path=path,
function_name=function_name,
score=score,
band=band,
signals={"loc": 10, "inventory_hit_count": len(inventory_finding_ids)},
rationale=f"{band} band: score {score}",
test_hint=None,
inventory_finding_ids=inventory_finding_ids,
)
def test_hotspot_key_joins_path_and_function_or_empty() -> None:
assert hotspot_key("parser.c", "parse_header") == "parser.c:parse_header"
assert hotspot_key("src/app.py", None) == "src/app.py:"
assert hotspot_key("src/app.py", "") == "src/app.py:"
def test_triggerable_overlay_raises_later_score() -> None:
hotspot = _hotspot("parser.c", function_name="parse_header", score=4.0, inventory_finding_ids=("hit:1",))
ledger = record_verdict({}, path="parser.c", function_name="parse_header", verdict="triggerable")
overlaid = apply_overlay((hotspot,), ledger)
raised = overlaid[0]
assert ledger[hotspot_key("parser.c", "parse_header")].score_delta == TRIGGERABLE_DELTA
assert TRIGGERABLE_DELTA > 0
assert raised.score == hotspot.score + TRIGGERABLE_DELTA
assert raised.score > hotspot.score
assert raised.inventory_finding_ids == ("hit:1",)
def test_not_triggerable_overlay_drops_score_and_keeps_inventory_hit() -> None:
hotspot = _hotspot("src/auth.py", function_name="login", score=5.0, inventory_finding_ids=("cmd:src/auth.py:12",))
ledger = {
hotspot_key("src/auth.py", "login"): LedgerEntry(
score_delta=NOT_TRIGGERABLE_DELTA,
last_verdict="not_triggerable",
round=1,
)
}
overlaid = apply_overlay((hotspot,), ledger)
demoted = overlaid[0]
assert NOT_TRIGGERABLE_DELTA < 0
assert demoted.score == hotspot.score + NOT_TRIGGERABLE_DELTA
assert demoted.score < hotspot.score
assert demoted.inventory_finding_ids == ("cmd:src/auth.py:12",)
assert demoted.path == hotspot.path
assert demoted.function_name == hotspot.function_name
def test_not_triggerable_overlay_drops_map_score_and_sev5_reachable_stays_candidate(tmp_path: Path) -> None:
auth = _target(
"src/auth.py",
loc=80,
tags=["auth_boundary"],
hints=[{"kind": "route", "access_level": "public", "function_name": "login"}],
)
helper = _target("src/helper.py", loc=12, tags=[])
sev5 = _hit(
"src/auth.py",
"cmd:src/auth.py:12",
function_name="login",
reachability="reachable",
severity="critical",
rationale="Static pattern cmd (CWE-78) matched os.system on line 12; manual verification still required.",
)
noise = _hit("src/helper.py", "print:src/helper.py:1", reachability="unknown", severity="low")
artifact = tmp_path / "complexity_map.json"
baseline = build_complexity_map([auth, helper], [sev5, noise], artifact)
original = next(item for item in baseline.hotspots if item.path == "src/auth.py")
assert sev5.finding_id in original.inventory_finding_ids
ledger_path = tmp_path / ".openultrasast" / "calibration" / "complexity_ledger.json"
write_ledger(
ledger_path,
record_verdict({}, path=original.path, function_name=original.function_name, verdict="not_triggerable"),
)
overlaid = build_complexity_map([auth, helper], [sev5, noise], artifact, ledger_path=ledger_path)
demoted = next(item for item in overlaid.hotspots if item.path == "src/auth.py")
assert demoted.score < original.score
assert demoted.score == original.score + NOT_TRIGGERABLE_DELTA
assert demoted.inventory_finding_ids == original.inventory_finding_ids
assert sev5.finding_id in demoted.inventory_finding_ids
assert sev5.severity == "critical"
assert sev5.rationale == ("Static pattern cmd (CWE-78) matched os.system on line 12; manual verification still required.")
policy = {
sev5.finding_id: 5,
noise.finding_id: 2,
}
forced = select_forced_candidates([sev5, noise], policy)
assert forced == (sev5,)
assert must_keep_as_candidate(sev5, 5, "reachable") is True
assert must_keep_as_candidate(noise, 2, "unknown") is False
def test_must_keep_as_candidate_requires_sev5_and_reachable_surface() -> None:
reachable = _hit("a.py", "a:1", reachability="reachable")
inferred = _hit("b.py", "b:1", reachability="inferred-file-surface")
unknown = _hit("c.py", "c:1", reachability="unknown")
assert must_keep_as_candidate(reachable, 5, "reachable") is True
assert must_keep_as_candidate(inferred, 5, "inferred-file-surface") is True
assert must_keep_as_candidate(unknown, 5, "unknown") is False
assert must_keep_as_candidate(reachable, 4, "reachable") is False
assert must_keep_as_candidate(reachable, 5, None) is True
def test_select_forced_candidates_resolves_policy_severity() -> None:
sev5 = _hit("src/auth.py", "cmd:src/auth.py:12", reachability="reachable")
inferred = _hit("src/upload.py", "sql:src/upload.py:4", reachability="inferred-file-surface")
mild = _hit("src/helper.py", "print:src/helper.py:1", reachability="reachable")
policy = {
"CWE-78": CwePolicy("Command Injection", 5, True, False),
"CWE-89": CwePolicy("SQL Injection", 5, True, False),
"CWE-116": CwePolicy("Encode Output", 2, True, False),
}
forced = select_forced_candidates(
[mild, sev5, inferred],
policy,
rule_cwe={
sev5.finding_id: "CWE-78",
inferred.finding_id: "CWE-89",
mild.finding_id: "CWE-116",
},
)
assert forced == (sev5, inferred)
def test_ledger_round_trip_matches_design_schema(tmp_path: Path) -> None:
path = tmp_path / "complexity_ledger.json"
key = hotspot_key("parser.c", "parse_header")
write_ledger(
path,
{key: LedgerEntry(score_delta=1.5, last_verdict="triggerable", round=3)},
)
payload = json.loads(path.read_text())
assert payload == {
key: {
"last_verdict": "triggerable",
"round": 3,
"score_delta": 1.5,
}
}
loaded = load_ledger(path)
assert loaded[key] == LedgerEntry(score_delta=1.5, last_verdict="triggerable", round=3)
assert load_ledger(tmp_path / "missing.json") == {}
assert load_ledger(None) == {}
def test_overlay_reorders_and_rebands_without_dropping_hits() -> None:
high = _hotspot("src/auth.py", function_name="login", score=3.2, band="medium", inventory_finding_ids=("sev5:1",))
other = _hotspot("src/helper.py", score=2.0, band="low", inventory_finding_ids=("low:1",))
ledger = record_verdict({}, path="src/auth.py", function_name="login", verdict="not_triggerable")
overlaid = apply_overlay((high, other), ledger)
assert [item.path for item in overlaid] == ["src/helper.py", "src/auth.py"]
demoted = overlaid[1]
assert demoted.score == round(3.2 + NOT_TRIGGERABLE_DELTA, 4)
assert demoted.band == "low"
assert demoted.inventory_finding_ids == ("sev5:1",)
assert overlaid[0].inventory_finding_ids == ("low:1",)
def test_persist_verdicts_records_triggerable_and_skips_inconclusive(tmp_path: Path) -> None:
path = tmp_path / ".openultrasast" / "calibration" / "complexity_ledger.json"
persist_verdicts(
path,
[
SimpleNamespace(path="app.py", function_name="admin", verdict="triggerable"),
SimpleNamespace(path="safe.py", function_name=None, verdict="not_triggerable"),
SimpleNamespace(path="other.py", function_name="skip", verdict="inconclusive"),
],
)
loaded = load_ledger(path)
assert loaded[hotspot_key("app.py", "admin")].last_verdict == "triggerable"
assert loaded[hotspot_key("app.py", "admin")].score_delta == TRIGGERABLE_DELTA
assert loaded[hotspot_key("safe.py", None)].last_verdict == "not_triggerable"
assert loaded[hotspot_key("safe.py", None)].score_delta == NOT_TRIGGERABLE_DELTA
assert hotspot_key("other.py", "skip") not in loaded
def test_build_complexity_map_without_ledger_path_skips_overlay(tmp_path: Path) -> None:
target = _target("src/app.py", loc=20)
finding = _hit("src/app.py", "rule:src/app.py:1")
artifact = tmp_path / "complexity_map.json"
first = build_complexity_map([target], [finding], artifact)
second = build_complexity_map([target], [finding], artifact)
assert first.hotspots[0].score == second.hotspots[0].score
assert first.hotspots[0].inventory_finding_ids == (finding.finding_id,)
+148
View File
@@ -0,0 +1,148 @@
import json
from pathlib import Path
from openultrasast.complexity.map import build_complexity_map
from openultrasast.findings import StaticFinding
from openultrasast.preprocess import FileTarget
_HOTSPOT_FIELDS = {
"path",
"function_name",
"score",
"band",
"signals",
"rationale",
"test_hint",
"inventory_finding_ids",
}
def _target(
path: str,
*,
loc: int,
tags: list[str] | None = None,
language: str = "python",
hints: list[dict[str, object]] | None = None,
has_fuzz_entry_point: bool = False,
) -> FileTarget:
return FileTarget(
path=path,
absolute_path=f"/repo/{path}",
language=language,
loc=loc,
tags=list(tags or []),
has_fuzz_entry_point=has_fuzz_entry_point,
reachability_hints=list(hints or []),
)
def _hit(path: str, finding_id: str) -> StaticFinding:
return StaticFinding(
finding_id=finding_id,
path=path,
title="pattern hit",
severity="low",
confidence="medium",
evidence_level="static_corroboration",
rationale="inventory",
line=1,
function_name=None,
reachability_status="unknown",
reachability_evidence=[],
reachability_conditions=[],
tags=[],
ranking_priority=1.0,
)
def _parser_helper_fixture() -> tuple[list[FileTarget], list[StaticFinding], tuple[str, ...]]:
parser = _target(
"src/parser.c",
loc=400,
language="c",
tags=["parser", "memory_unsafe"],
hints=[{"kind": "parser", "access_level": "public", "function_name": "parse_header"}],
)
helper = _target("src/helper.py", loc=1, tags=[])
findings = [_hit("src/helper.py", f"rule:helper:{index}") for index in range(8)]
repo_files = ("src/parser.c", "src/helper.py", "tests/test_parser.c")
return [parser, helper], findings, repo_files
def test_map_order_differs_from_inventory_hit_count(tmp_path: Path) -> None:
targets, findings, repo_files = _parser_helper_fixture()
artifact = tmp_path / "complexity_map.json"
complexity_map = build_complexity_map(targets, findings, artifact, repo_files=repo_files)
map_paths = [hotspot.path for hotspot in complexity_map.hotspots]
hit_count_paths = [
hotspot.path
for hotspot in sorted(
complexity_map.hotspots,
key=lambda item: (-len(item.inventory_finding_ids), item.path),
)
]
parser = next(hotspot for hotspot in complexity_map.hotspots if hotspot.path == "src/parser.c")
helper = next(hotspot for hotspot in complexity_map.hotspots if hotspot.path == "src/helper.py")
assert map_paths != hit_count_paths
assert map_paths[0] == "src/parser.c"
assert parser.inventory_finding_ids == ()
assert len(helper.inventory_finding_ids) == 8
assert helper.inventory_finding_ids == tuple(f"rule:helper:{index}" for index in range(8))
assert parser.score > helper.score
assert parser.function_name == "parse_header"
assert parser.band == "high"
assert helper.band == "low"
assert complexity_map.heuristic_only is True
assert parser.test_hint is not None
assert parser.test_hint.gap == "no_fuzz_entry"
assert parser.test_hint.test_kind == "fuzz-harness"
assert helper.test_hint is not None
assert helper.test_hint.gap == "no_adjacent_test"
assert helper.test_hint.test_kind == "unit"
assert "verified" not in parser.rationale.lower()
assert "worth fixing" not in parser.rationale.lower()
assert "worth_fixing" not in parser.rationale.lower()
def test_complexity_map_json_has_required_fields(tmp_path: Path) -> None:
targets, findings, repo_files = _parser_helper_fixture()
artifact = tmp_path / "runs" / "complexity_map.json"
complexity_map = build_complexity_map(targets, findings, artifact, repo_files=repo_files)
raw = artifact.read_text()
payload = json.loads(raw)
assert raw == json.dumps(payload, indent=2, sort_keys=True) + "\n"
assert payload["heuristic_only"] is True
assert [item["path"] for item in payload["hotspots"]] == [hotspot.path for hotspot in complexity_map.hotspots]
for hotspot in payload["hotspots"]:
assert set(hotspot) >= _HOTSPOT_FIELDS
assert hotspot["band"] in {"high", "medium", "low"}
assert isinstance(hotspot["score"], float)
assert isinstance(hotspot["rationale"], str) and hotspot["rationale"]
assert isinstance(hotspot["signals"], dict)
assert set(hotspot["signals"]) >= {"loc", "nesting", "reachability", "inventory_hit_count", "has_adjacent_test"}
assert isinstance(hotspot["inventory_finding_ids"], list)
hint = hotspot["test_hint"]
assert isinstance(hint, dict)
assert hint["gap"] in {"no_adjacent_test", "no_function_reference", "no_fuzz_entry", "covered"}
if hint["gap"] == "covered":
assert hint["test_kind"] is None
else:
assert hint["test_kind"] in {"unit", "property", "sanitizer", "http-contract", "fuzz-harness"}
assert "6.0" in hotspot["rationale"]
assert "3.0" in hotspot["rationale"]
parser = next(item for item in payload["hotspots"] if item["path"] == "src/parser.c")
helper = next(item for item in payload["hotspots"] if item["path"] == "src/helper.py")
assert parser["signals"]["has_adjacent_test"] is True
assert helper["signals"]["has_adjacent_test"] is False
assert parser["test_hint"]["gap"] == "no_fuzz_entry"
assert parser["test_hint"]["test_kind"] == "fuzz-harness"
assert helper["test_hint"]["gap"] == "no_adjacent_test"
assert helper["test_hint"]["test_kind"] == "unit"
assert helper["inventory_finding_ids"] == [f"rule:helper:{index}" for index in range(8)]
assert parser["inventory_finding_ids"] == []
+171
View File
@@ -0,0 +1,171 @@
from pathlib import Path
from openultrasast.complexity import ComplexitySignals, collect_signals
from openultrasast.findings import StaticFinding
from openultrasast.preprocess import FileTarget
def _target(
path: str,
*,
loc: int,
tags: list[str] | None = None,
language: str = "python",
hints: list[dict[str, object]] | None = None,
has_fuzz_entry_point: bool = False,
absolute_path: str | None = None,
) -> FileTarget:
return FileTarget(
path=path,
absolute_path=absolute_path or f"/repo/{path}",
language=language,
loc=loc,
tags=list(tags or []),
has_fuzz_entry_point=has_fuzz_entry_point,
reachability_hints=list(hints or []),
)
def _hit(path: str, finding_id: str = "rule:helper:1") -> StaticFinding:
return StaticFinding(
finding_id=finding_id,
path=path,
title="pattern hit",
severity="low",
confidence="medium",
evidence_level="static_corroboration",
rationale="inventory",
line=1,
function_name=None,
reachability_status="unknown",
reachability_evidence=[],
reachability_conditions=[],
tags=[],
ranking_priority=1.0,
)
def _by_path(signals: tuple[ComplexitySignals, ...]) -> dict[str, ComplexitySignals]:
return {signal.path: signal for signal in signals}
def test_parser_without_pattern_hit_outranks_one_line_helper_with_hit() -> None:
parser = _target(
"src/parser.c",
loc=400,
language="c",
tags=["parser", "memory_unsafe"],
hints=[{"kind": "parser", "access_level": "public", "function_name": "parse_header"}],
)
helper = _target("src/helper.py", loc=1, tags=[])
findings = [_hit("src/helper.py")]
signals = collect_signals([parser, helper], findings)
by_path = _by_path(signals)
assert by_path["src/parser.c"].inventory_hit_count == 0
assert by_path["src/helper.py"].inventory_hit_count == 1
assert by_path["src/parser.c"].score > by_path["src/helper.py"].score
assert signals[0].path == "src/parser.c"
def test_function_identity_uses_entry_point_names_or_stays_empty() -> None:
named = _target(
"src/parser.c",
loc=40,
language="c",
tags=["parser"],
hints=[{"function_name": "parse_header", "kind": "parser", "access_level": "public"}],
)
named_alias = _target(
"src/decode.c",
loc=20,
language="c",
tags=["parser"],
hints=[{"name": "decode_frame", "kind": "parser", "access_level": "public"}],
)
tag_only = _target(
"src/App.java",
loc=80,
language="java",
tags=["parser"],
hints=[{"function_name": None, "name": "parser_input", "kind": "parser", "line": None}],
)
bare = _target("src/util.py", loc=4, tags=[])
signals = collect_signals([named, named_alias, tag_only, bare])
by_path = _by_path(signals)
assert by_path["src/parser.c"].function_name == "parse_header"
assert by_path["src/decode.c"].function_name == "decode_frame"
assert by_path["src/App.java"].function_name is None
assert by_path["src/util.py"].function_name is None
def test_nesting_from_source_text_otherwise_zero() -> None:
nested = _target("src/nested.py", loc=4, tags=[])
constructed = _target("src/flat.py", loc=12, tags=["parser"])
source = "def outer():\n if True:\n while True:\n return 1\n"
with_text = collect_signals([nested], sources={"src/nested.py": source})
without_text = collect_signals([constructed])
assert with_text[0].nesting >= 3
assert without_text[0].nesting == 0
assert without_text[0].loc == 12
def test_nesting_reads_absolute_path_when_file_exists(tmp_path: Path) -> None:
path = tmp_path / "braces.c"
path.write_text("int f(void) {\n if (1) {\n while (1) {\n return 0;\n }\n }\n}\n")
target = _target("braces.c", loc=7, language="c", tags=[], absolute_path=str(path))
assert collect_signals([target])[0].nesting >= 3
def test_adjacent_test_from_repo_relative_paths() -> None:
parser = _target("src/parser.c", loc=40, language="c", tags=["parser"])
helper = _target("pkg/helper.go", loc=8, language="go", tags=[])
widget = _target("src/main/java/com/acme/Widget.java", loc=30, language="java", tags=[])
lonely = _target("src/lonely.py", loc=5, tags=[])
signals = collect_signals(
[parser, helper, widget, lonely],
repo_files=(
"src/parser.c",
"tests/test_parser.c",
"pkg/helper.go",
"pkg/helper_test.go",
"src/main/java/com/acme/Widget.java",
"src/test/java/com/acme/WidgetTest.java",
"src/lonely.py",
"tests/test_other.py",
),
)
by_path = _by_path(signals)
assert by_path["src/parser.c"].has_adjacent_test is True
assert by_path["pkg/helper.go"].has_adjacent_test is True
assert by_path["src/main/java/com/acme/Widget.java"].has_adjacent_test is True
assert by_path["src/lonely.py"].has_adjacent_test is False
def test_inventory_density_is_recorded_but_cannot_dominate_score() -> None:
parser = _target(
"src/parser.c",
loc=250,
language="c",
tags=["parser"],
hints=[{"kind": "parser", "access_level": "public", "function_name": "parse_header"}],
)
helper = _target("src/helper.py", loc=1, tags=[])
findings = [_hit("src/helper.py", f"rule:helper:{index}") for index in range(12)]
signals = collect_signals([parser, helper], findings)
by_path = _by_path(signals)
assert by_path["src/helper.py"].inventory_hit_count == 12
assert by_path["src/parser.c"].inventory_hit_count == 0
assert by_path["src/parser.c"].score > by_path["src/helper.py"].score
assert "parser" in by_path["src/parser.c"].tags
assert by_path["src/parser.c"].reachability > by_path["src/helper.py"].reachability
+63
View File
@@ -96,3 +96,66 @@ def test_write_resolved_config_creates_json(tmp_path: Path) -> None:
assert output.exists() assert output.exists()
assert '"minimum_report_verified": "static_corroboration"' in output.read_text() assert '"minimum_report_verified": "static_corroboration"' in output.read_text()
def test_complexity_and_regress_defaults_when_section_absent() -> None:
config = load_config(None)
assert config.complexity.top_k == 20
assert config.complexity.max_hunter_hotspots == 8
assert config.regress.max_candidates == 5
assert config.regress.images == ()
assert config.sandbox.network is False
assert config.sandbox.workspace_readonly is True
assert config.sandbox.memory_mb == 2048
assert config.sandbox.timeout_seconds == 300
assert config.sandbox.pids_limit == 512
def test_override_changes_top_k_and_max_candidates_only(tmp_path: Path) -> None:
config_path = tmp_path / "openultrasast.toml"
config_path.write_text(
"\n".join(
[
"[complexity]",
"top_k = 3",
"[regress]",
"max_candidates = 2",
]
)
)
config = load_config(config_path)
assert config.complexity.top_k == 3
assert config.regress.max_candidates == 2
assert config.complexity.max_hunter_hotspots == 8
assert config.regress.images == ()
assert config.sandbox.network is False
assert config.sandbox.workspace_readonly is True
assert config.sandbox.memory_mb == 2048
assert config.sandbox.timeout_seconds == 300
assert config.sandbox.pids_limit == 512
def test_load_config_reads_regress_image_pins(tmp_path: Path) -> None:
config_path = tmp_path / "openultrasast.toml"
config_path.write_text(
"\n".join(
[
"[regress]",
"max_candidates = 4",
"[regress.images]",
'python = "python:3.12-slim"',
'javascript = "node:22-slim"',
]
)
)
config = load_config(config_path)
assert config.regress.max_candidates == 4
assert dict(config.regress.images) == {"python": "python:3.12-slim", "javascript": "node:22-slim"}
assert config.complexity.top_k == 20
assert config.complexity.max_hunter_hotspots == 8
assert config.sandbox.memory_mb == 2048
+123
View File
@@ -0,0 +1,123 @@
from pathlib import Path
import pytest
from openultrasast.hunter_tools import PathEscapesRepo, clamp_repo_path, find_refs, grep_repo, read_file
from openultrasast.mapping import EntryPointRecord
SPLIT_SINK = """\
term = request.args["q"]
query = "select * from items where title like '%" + term + "%'"
return db.execute(query)
"""
def _repo_with_split_sink(tmp_path: Path) -> Path:
root = tmp_path / "repo"
root.mkdir()
(root / "app.py").write_text(SPLIT_SINK)
return root
def test_read_grep_and_reject_path_escape(tmp_path: Path) -> None:
root = _repo_with_split_sink(tmp_path)
source = read_file(root, "app.py", max_chars=4000)
assert "query =" in source
assert "db.execute(query)" in source
matches = grep_repo(root, r"query\s*=", max_matches=10)
assert matches
assert matches[0]["path"] == "app.py"
assert matches[0]["line"] == 2
assert "query =" in str(matches[0]["text"])
with pytest.raises(PathEscapesRepo):
read_file(root, "../etc/passwd", max_chars=100)
def test_clamp_repo_path_rejects_dotdot_and_absolute_escapes(tmp_path: Path) -> None:
root = tmp_path / "repo"
root.mkdir()
(root / "app.py").write_text("ok\n")
assert clamp_repo_path(root, "app.py") == (root / "app.py").resolve()
with pytest.raises(PathEscapesRepo):
clamp_repo_path(root, "../etc/passwd")
with pytest.raises(PathEscapesRepo):
clamp_repo_path(root, "/etc/passwd")
with pytest.raises(PathEscapesRepo):
clamp_repo_path(root, "..")
def test_grep_repo_does_not_search_outside_root(tmp_path: Path) -> None:
root = tmp_path / "repo"
root.mkdir()
(root / "app.py").write_text('query = "inside"\n')
(tmp_path / "outside.py").write_text('query = "outside"\n')
matches = grep_repo(root, r"query\s*=", max_matches=10)
assert [match["path"] for match in matches] == ["app.py"]
assert all("outside" not in str(match["text"]) for match in matches)
def test_grep_repo_honors_max_matches(tmp_path: Path) -> None:
root = tmp_path / "repo"
root.mkdir()
(root / "app.py").write_text("sink = 1\nsink = 2\nsink = 3\n")
matches = grep_repo(root, r"sink\s*=", max_matches=2)
assert len(matches) == 2
assert [match["line"] for match in matches] == [1, 2]
def test_read_file_truncates_and_rejects_symlink_escape(tmp_path: Path) -> None:
root = tmp_path / "repo"
root.mkdir()
(root / "app.py").write_text("abcdef")
(root / "link.py").symlink_to("/etc/passwd")
assert read_file(root, "app.py", max_chars=3) == "abc"
with pytest.raises(PathEscapesRepo):
read_file(root, "link.py", max_chars=100)
def test_find_refs_uses_mapping_index_and_clamped_text_search(tmp_path: Path) -> None:
root = _repo_with_split_sink(tmp_path)
mapping_index = [
{"path": "app.py", "name": "query", "line": 2},
EntryPointRecord(
path="app.py",
line=3,
end_line=3,
function_name="search",
name="search",
kind="route",
access_level="public",
trust_boundary="http_request",
access_evidence=[],
conditions=[],
provenance="test",
rationale="fixture",
),
]
query_refs = find_refs(root, "query", mapping_index)
assert any(ref["path"] == "app.py" and ref["name"] == "query" for ref in query_refs)
assert any(ref.get("source") == "mapping" for ref in query_refs)
assert any(ref.get("source") == "text" and "query" in str(ref.get("text", ref.get("name", ""))) for ref in query_refs)
search_refs = find_refs(root, "search", mapping_index)
assert any(ref["path"] == "app.py" and ref["name"] == "search" for ref in search_refs)
def test_find_refs_rejects_mapping_path_escape(tmp_path: Path) -> None:
root = _repo_with_split_sink(tmp_path)
with pytest.raises(PathEscapesRepo):
find_refs(root, "passwd", [{"path": "../etc/passwd", "name": "passwd"}])
with pytest.raises(PathEscapesRepo):
find_refs(root, "passwd", [{"path": "/etc/passwd", "name": "passwd"}])
+82
View File
@@ -0,0 +1,82 @@
"""Stage-2 map gate: planted split-sink ranks in the high band and beats hit-count order."""
from openultrasast.complexity.map import Hotspot
from openultrasast.findings import StaticFinding
from openultrasast.gate import LANGUAGE_MANIFESTS
from openultrasast.map_gate import (
SPLIT_SINK_MANIFESTS,
hit_count_path_order,
map_beats_hit_count,
planted_in_high_band,
scan_split_sink,
split_sink_map_gate,
)
def test_split_sink_map_gate_passes_heuristic_map() -> None:
verdict = split_sink_map_gate()
assert verdict.passed, verdict.reasons
def test_hit_count_stub_fails_map_gate_on_split_sink_python() -> None:
bundle = scan_split_sink("split-sink-python")
unique_map = list(dict.fromkeys(hotspot.path for hotspot in bundle.complexity_map.hotspots))
stub_order = hit_count_path_order(bundle.findings, unique_map)
assert stub_order != unique_map
assert map_beats_hit_count(bundle.complexity_map.hotspots, bundle.findings)
def test_planted_split_sink_function_is_in_high_band() -> None:
bundle = scan_split_sink("split-sink-python")
assert planted_in_high_band(bundle.manifest, bundle.complexity_map)
names = {hotspot.function_name for hotspot in bundle.complexity_map.hotspots if hotspot.path == "app.py"}
assert "search" in names
def test_split_sink_gate_corpus_is_not_the_stage1_smoke_gate() -> None:
listed = {name for names in LANGUAGE_MANIFESTS.values() for name in names}
assert not any(name in listed for name in SPLIT_SINK_MANIFESTS)
def test_map_beats_hit_count_rejects_inventory_sorted_stub() -> None:
findings = [
StaticFinding(
finding_id="python-unsafe-eval:helper.py:1",
path="helper.py",
title="eval",
severity="high",
confidence="low",
evidence_level="static_corroboration",
rationale="hit",
line=1,
function_name="ping",
reachability_status="unknown",
reachability_evidence=[],
reachability_conditions=[],
tags=["syscall_entry"],
ranking_priority=1.0,
)
]
stub = (
Hotspot(
path="helper.py",
function_name="ping",
score=0.1,
band="low",
signals={},
rationale="stub",
test_hint=None,
inventory_finding_ids=(findings[0].finding_id,),
),
Hotspot(
path="app.py",
function_name="search",
score=9.0,
band="high",
signals={},
rationale="stub",
test_hint=None,
inventory_finding_ids=(),
),
)
assert not map_beats_hit_count(stub, findings)
+18 -1
View File
@@ -54,7 +54,24 @@ def test_surface_exposes_no_shell_or_exec_tool() -> None:
for name in server.tool_names: for name in server.tool_names:
assert not any(token in name.lower() for token in forbidden), name assert not any(token in name.lower() for token in forbidden), name
for tool in McpServer()._tools.values(): # no tool accepts a free-form command argument for tool in McpServer()._tools.values(): # no tool accepts a free-form command argument
assert "command" not in tool.input_schema.get("properties", {}) properties = tool.input_schema.get("properties", {})
assert "command" not in properties
assert "shell" not in properties
assert "docker" not in properties
def test_scan_rejects_deep_and_keeps_cli_only_modes() -> None:
server = McpServer()
listed = server.handle({"jsonrpc": "2.0", "id": 1, "method": "tools/list", "params": {}})
assert listed is not None
scan_tool = next(tool for tool in listed["result"]["tools"] if tool["name"] == "openultrasast.scan")
mode_enum = scan_tool["inputSchema"]["properties"]["mode"]["enum"]
assert mode_enum == ["quick", "standard"]
assert "deep" not in mode_enum
result = _call(server, "openultrasast.scan", {"path": ".", "mode": "deep"})
assert result["isError"]
assert "deep" in result["error"]
def test_notifications_get_no_response() -> None: def test_notifications_get_no_response() -> None:
+46
View File
@@ -0,0 +1,46 @@
"""Mutation fixtures: stage-1 same-line patterns must miss the aliased sink (requirement 8.2)."""
from pathlib import Path
from openultrasast.benchmark import BenchmarkRun, evaluate_benchmark, load_benchmark_manifest, resolve_benchmark_source
from openultrasast.findings import StaticFinding, quick_scan_findings
from openultrasast.gate import LANGUAGE_MANIFESTS
from openultrasast.preprocess import preprocess_repository
from openultrasast.rank import rank_targets
MANIFEST = Path("benchmarks/manifests/mutation-python.toml")
def _quick_scan(target: Path) -> list[StaticFinding]:
_, targets = preprocess_repository(target)
return quick_scan_findings(target, targets, rank_targets(targets))
def test_stage1_misses_aliased_os_system_mutation() -> None:
manifest = load_benchmark_manifest(MANIFEST)
target = resolve_benchmark_source(MANIFEST, manifest)
planted = [item for item in manifest.expected if item.rule_id is None]
assert planted, "mutation manifest must document a stage-1 miss without rule_id"
expected = planted[0]
assert expected.sink == "run_cmd"
assert expected.line is not None
source = (target / expected.path).read_text()
lines = source.splitlines()
assert "os.system" in source
assert "os.system(" not in lines[expected.line - 1]
assert "run_cmd(" in lines[expected.line - 1]
findings = _quick_scan(target)
on_line = [finding for finding in findings if finding.path == expected.path and finding.line == expected.line]
assert on_line == [], f"stage-1 hit the mutation on {expected.path}:{expected.line}: {on_line}"
run = BenchmarkRun(benchmark_run_id="mutation", root=Path("/tmp/mutation"), manifest=manifest)
result = evaluate_benchmark(run=run, mode="quick", findings=findings, scan_id=None, scan_run_dir=None)
missed = {(miss.path, miss.cwe, miss.rule_id) for miss in result.misses}
assert (expected.path, expected.cwe, None) in missed
def test_mutation_fixture_is_outside_stage1_gate_corpus() -> None:
listed = {name for names in LANGUAGE_MANIFESTS.values() for name in names}
assert "mutation-python" not in listed
+159
View File
@@ -0,0 +1,159 @@
"""Bounded regression candidate selection from hotspots and severe inventory (task 5.1)."""
from openultrasast.complexity.ledger import NOT_TRIGGERABLE_DELTA, apply_overlay, record_verdict
from openultrasast.complexity.map import Hotspot
from openultrasast.findings import StaticFinding
from openultrasast.regress.candidate import select_candidates
def _finding(
path: str,
finding_id: str,
*,
function_name: str | None = None,
reachability: str = "unknown",
severity: str = "low",
) -> StaticFinding:
return StaticFinding(
finding_id=finding_id,
path=path,
title="pattern hit",
severity=severity,
confidence="medium",
evidence_level="static_corroboration",
rationale="inventory",
line=1,
function_name=function_name,
reachability_status=reachability,
reachability_evidence=[],
reachability_conditions=[],
tags=[],
ranking_priority=1.0,
)
def _hotspot(
path: str,
*,
function_name: str | None = None,
score: float = 4.0,
band: str = "medium",
inventory_finding_ids: tuple[str, ...] = (),
) -> Hotspot:
return Hotspot(
path=path,
function_name=function_name,
score=score,
band=band,
signals={"loc": 10, "inventory_hit_count": len(inventory_finding_ids)},
rationale=f"{band} band: score {score}",
test_hint=None,
inventory_finding_ids=inventory_finding_ids,
)
def _six_hotspots(*, sev5_id: str | None = None, sev5_score: float = 4.0) -> tuple[Hotspot, ...]:
return (
_hotspot("a.py", function_name="a", score=5.0),
_hotspot("b.py", function_name="b", score=4.8),
_hotspot("c.py", function_name="c", score=4.6),
_hotspot("d.py", function_name="d", score=4.4),
_hotspot("e.py", function_name="e", score=4.2),
_hotspot(
"f.py",
function_name="f",
score=sev5_score,
inventory_finding_ids=(sev5_id,) if sev5_id else (),
),
)
def test_six_hotspots_max_candidates_three_returns_three() -> None:
noise = _finding("f.py", "print:f.py:1", function_name="f", reachability="reachable")
selected = select_candidates(
_six_hotspots(),
[noise],
max_candidates=3,
policy_severity_by_id={noise.finding_id: 2},
)
assert len(selected) == 3
assert [item.path for item in selected] == ["a.py", "b.py", "c.py"]
def test_sev5_reachable_extra_included_when_ledger_demoted_hotspot() -> None:
sev5_id = "cmd:f.py:1"
hotspots = _six_hotspots(sev5_id=sev5_id, sev5_score=5.2)
ledger = record_verdict({}, path="f.py", function_name="f", verdict="not_triggerable")
demoted = apply_overlay(hotspots, ledger)
finding = _finding(
"f.py",
sev5_id,
function_name="f",
reachability="reachable",
severity="critical",
)
selected = select_candidates(
demoted,
[finding],
max_candidates=3,
policy_severity_by_id={sev5_id: 5},
)
demoted_f = next(item for item in demoted if item.path == "f.py")
assert demoted_f.score == round(5.2 + NOT_TRIGGERABLE_DELTA, 4)
assert [item.path for item in demoted[:3]] == ["a.py", "b.py", "c.py"]
assert demoted_f not in demoted[:3]
assert len(selected) == 4
assert [item.path for item in selected[:3]] == ["a.py", "b.py", "c.py"]
assert selected[3] is demoted_f
assert sev5_id in selected[3].inventory_finding_ids
def test_sev5_already_in_cap_is_not_duplicated() -> None:
sev5_id = "cmd:a.py:1"
hotspots = (
_hotspot("a.py", function_name="a", score=5.0, inventory_finding_ids=(sev5_id,)),
_hotspot("b.py", function_name="b", score=4.8),
_hotspot("c.py", function_name="c", score=4.6),
_hotspot("d.py", function_name="d", score=4.4),
_hotspot("e.py", function_name="e", score=4.2),
_hotspot("f.py", function_name="f", score=4.0),
)
finding = _finding("a.py", sev5_id, function_name="a", reachability="reachable", severity="critical")
selected = select_candidates(
hotspots,
[finding],
max_candidates=3,
policy_severity_by_id={sev5_id: 5},
)
assert len(selected) == 3
assert [item.path for item in selected] == ["a.py", "b.py", "c.py"]
assert sev5_id in selected[0].inventory_finding_ids
def test_sev5_reachable_inventory_without_hotspot_is_still_selected() -> None:
finding = _finding(
"z.py",
"cmd:z.py:1",
function_name="z",
reachability="reachable",
severity="critical",
)
selected = select_candidates(
_six_hotspots(),
[finding],
max_candidates=3,
policy_severity_by_id={finding.finding_id: 5},
)
assert len(selected) == 4
assert [item.path for item in selected[:3]] == ["a.py", "b.py", "c.py"]
extra = selected[3]
assert extra.path == "z.py"
assert extra.function_name == "z"
assert extra.inventory_finding_ids == (finding.finding_id,)
+103
View File
@@ -0,0 +1,103 @@
"""Language recipes for isolated regression snippets (task 5.2)."""
from pathlib import Path
from openultrasast.config import SandboxConfig
from openultrasast.regress import INCONCLUSIVE, RegressionRunner
from openultrasast.regress.recipes import recipe_for
from openultrasast.sandbox import FakeSandboxRunner, SandboxJob
LIMITS = SandboxConfig(memory_mb=256, timeout_seconds=30, pids_limit=128)
PYTHON_SNIPPET = "assert 1 + 1 == 2\n"
JS_SNIPPET = "console.log('ok')\n"
C_SNIPPET = "int main(void) { return 0; }\n"
JAVA_SNIPPET = "class Case { public static void main(String[] args) {} }\n"
def _repo(tmp_path: Path) -> Path:
repo = tmp_path / "repo"
repo.mkdir()
return repo
def test_python_recipe_runs_scratch_snippet_with_repo_on_pythonpath(tmp_path: Path) -> None:
repo = _repo(tmp_path)
job = recipe_for("python", PYTHON_SNIPPET, "python:3.12-alpine", LIMITS, repo_root=repo)
assert isinstance(job, SandboxJob)
joined = " ".join(job.command)
assert job.image == "python:3.12-alpine"
assert job.repo_root == repo
assert job.scratch_files == {"case.py": PYTHON_SNIPPET}
assert "python" in job.command
assert "/scratch/case.py" in job.command
assert "PYTHONPATH=/workspace" in joined
assert job.timeout_seconds == LIMITS.timeout_seconds
assert job.memory_mb == LIMITS.memory_mb
assert job.pids_limit == LIMITS.pids_limit
def test_javascript_recipe_runs_scratch_snippet_with_repo_on_node_path(tmp_path: Path) -> None:
repo = _repo(tmp_path)
job = recipe_for("javascript", JS_SNIPPET, "node:22-alpine", LIMITS, repo_root=repo)
assert isinstance(job, SandboxJob)
joined = " ".join(job.command)
assert job.image == "node:22-alpine"
assert job.repo_root == repo
assert job.scratch_files == {"case.js": JS_SNIPPET}
assert "node" in job.command
assert "/scratch/case.js" in job.command
assert "NODE_PATH=/workspace" in joined
assert job.timeout_seconds == LIMITS.timeout_seconds
assert job.memory_mb == LIMITS.memory_mb
assert job.pids_limit == LIMITS.pids_limit
def test_c_recipe_compiles_and_runs_when_image_has_compiler(tmp_path: Path) -> None:
repo = _repo(tmp_path)
job = recipe_for("c", C_SNIPPET, "gcc:13", LIMITS, repo_root=repo)
assert isinstance(job, SandboxJob)
joined = " ".join(job.command)
assert job.scratch_files == {"case.c": C_SNIPPET}
assert "cc" in joined
assert "/scratch/case.c" in joined
assert "-o" in joined
assert "/scratch/case" in joined
def test_unknown_language_recipe_is_none(tmp_path: Path) -> None:
repo = _repo(tmp_path)
assert recipe_for("java", JAVA_SNIPPET, "eclipse-temurin:21", LIMITS, repo_root=repo) is None
assert recipe_for("unknown", "print(1)\n", "python:3.12-alpine", LIMITS, repo_root=repo) is None
def test_python_and_javascript_recipes_produce_runner_jobs(tmp_path: Path) -> None:
repo = _repo(tmp_path)
python_job = recipe_for("python", PYTHON_SNIPPET, "python:3.12-alpine", LIMITS, repo_root=repo)
js_job = recipe_for("javascript", JS_SNIPPET, "node:22-alpine", LIMITS, repo_root=repo)
sandbox = FakeSandboxRunner()
runner = RegressionRunner(sandbox)
python_verdict = runner.run_recipe("python", PYTHON_SNIPPET, "python:3.12-alpine", LIMITS, repo_root=repo)
js_verdict = runner.run_recipe("javascript", JS_SNIPPET, "node:22-alpine", LIMITS, repo_root=repo)
assert python_job is not None
assert js_job is not None
assert python_verdict.reason != "missing_recipe"
assert js_verdict.reason != "missing_recipe"
assert sandbox.jobs == [python_job, js_job]
def test_unknown_language_returns_inconclusive_without_calling_runner(tmp_path: Path) -> None:
repo = _repo(tmp_path)
sandbox = FakeSandboxRunner()
runner = RegressionRunner(sandbox)
assert recipe_for("java", JAVA_SNIPPET, "eclipse-temurin:21", LIMITS, repo_root=repo) is None
verdict = runner.run_recipe("java", JAVA_SNIPPET, "eclipse-temurin:21", LIMITS, repo_root=repo)
assert verdict.verdict == INCONCLUSIVE
assert verdict.reason == "missing_recipe"
assert sandbox.jobs == []
+86
View File
@@ -0,0 +1,86 @@
"""Reject unsafe regression snippets before the sandbox runner sees them (task 4.3)."""
from pathlib import Path
import pytest
from openultrasast.regress import RegressionRunner, UnsafeSnippetError, check_snippet_safety
from openultrasast.sandbox import FakeSandboxRunner, SandboxJob
BENIGN_ASSERT = "assert 1 + 1 == 2\n"
def _job(tmp_path: Path, snippet: str) -> SandboxJob:
repo = tmp_path / "repo"
repo.mkdir()
return SandboxJob(
image="python:3.12-alpine",
command=("python", "/scratch/case.py"),
repo_root=repo,
scratch_files={"case.py": snippet},
timeout_seconds=30,
memory_mb=256,
pids_limit=128,
)
def test_docker_sock_rejected_does_not_create_runner_job(tmp_path: Path) -> None:
snippet = 'client = docker.DockerClient(base_url="unix://var/run/docker.sock")\n'
sandbox = FakeSandboxRunner()
runner = RegressionRunner(sandbox)
with pytest.raises(UnsafeSnippetError, match="Docker socket"):
check_snippet_safety(snippet)
verdict = runner.run_snippet(snippet, _job(tmp_path, snippet))
assert verdict.verdict == "inconclusive"
assert verdict.reason == "safety_rejected"
assert sandbox.jobs == []
def test_host_network_rejected_does_not_create_runner_job(tmp_path: Path) -> None:
snippet = "subprocess.run(['docker', 'run', '--network', 'host', 'alpine'])\n"
sandbox = FakeSandboxRunner()
runner = RegressionRunner(sandbox)
with pytest.raises(UnsafeSnippetError, match="host network"):
check_snippet_safety(snippet)
verdict = runner.run_snippet(snippet, _job(tmp_path, snippet))
assert verdict.verdict == "inconclusive"
assert verdict.reason == "safety_rejected"
assert sandbox.jobs == []
def test_write_under_workspace_rejected_does_not_create_runner_job(tmp_path: Path) -> None:
snippet = 'open("/workspace/pwned.py", "w").write("owned")\n'
sandbox = FakeSandboxRunner()
runner = RegressionRunner(sandbox)
with pytest.raises(UnsafeSnippetError, match="/workspace"):
check_snippet_safety(snippet)
verdict = runner.run_snippet(snippet, _job(tmp_path, snippet))
assert verdict.verdict == "inconclusive"
assert verdict.reason == "safety_rejected"
assert sandbox.jobs == []
def test_benign_python_assert_snippet_is_allowed(tmp_path: Path) -> None:
assert check_snippet_safety(BENIGN_ASSERT) is None
sandbox = FakeSandboxRunner()
job = _job(tmp_path, BENIGN_ASSERT)
RegressionRunner(sandbox).run_snippet(BENIGN_ASSERT, job)
assert sandbox.jobs == [job]
@pytest.mark.parametrize(
"snippet",
[
"import socket\nsocket.create_connection(('127.0.0.1', 80))\n",
"curl https://example.invalid/pwn\n",
],
)
def test_socket_and_curl_mentions_are_rejected(snippet: str) -> None:
with pytest.raises(UnsafeSnippetError):
check_snippet_safety(snippet)
+201
View File
@@ -0,0 +1,201 @@
"""Map sandbox outcomes to the four verdicts and the worth-fixing gate (task 5.3)."""
from __future__ import annotations
from pathlib import Path
import pytest
from openultrasast.regress import RegressionRunner
from openultrasast.regress.verdict import (
ALREADY_COVERED,
INCONCLUSIVE,
NOT_TRIGGERABLE,
TRIGGERABLE,
is_worth_fixing,
verdict_from_result,
)
from openultrasast.sandbox import FakeSandboxRunner, SandboxJob, SandboxResult
_REACHABLE = "reachable"
_INFERRED = "inferred-file-surface"
_UNKNOWN = "unknown"
def _result(*, exit_code: int = 0, stdout: str = "", stderr: str = "", timed_out: bool = False) -> SandboxResult:
return SandboxResult(exit_code=exit_code, stdout=stdout, stderr=stderr, timed_out=timed_out)
def _job(tmp_path: Path, snippet: str) -> SandboxJob:
repo = tmp_path / "repo"
repo.mkdir()
return SandboxJob(
image="python:3.12-alpine",
command=("python", "/scratch/case.py"),
repo_root=repo,
scratch_files={"case.py": snippet},
timeout_seconds=30,
memory_mb=256,
pids_limit=128,
)
# observation kwargs, sandbox result, reachability, expected verdict, expected reason, worth-fixing
_CASES = (
pytest.param(
{"recipe_missing": True},
None,
_REACHABLE,
INCONCLUSIVE,
"missing_recipe",
False,
id="recipe-missing-inconclusive",
),
pytest.param(
{"sandbox_missing": True},
None,
_REACHABLE,
INCONCLUSIVE,
"sandbox_unavailable",
False,
id="sandbox-missing-inconclusive",
),
pytest.param(
{},
_result(exit_code=-1, timed_out=True),
_REACHABLE,
INCONCLUSIVE,
"timeout",
False,
id="timeout-inconclusive",
),
pytest.param(
{"safety_rejected": True},
None,
_INFERRED,
INCONCLUSIVE,
"safety_rejected",
False,
id="safety-reject-inconclusive",
),
pytest.param(
{"covering_test": True},
_result(exit_code=0, stdout="ok\n"),
_REACHABLE,
ALREADY_COVERED,
"covered_test_passed",
False,
id="covering-test-passed-already-covered",
),
pytest.param(
{},
_result(exit_code=1, stderr="AssertionError: boom\n"),
_REACHABLE,
TRIGGERABLE,
"assertion",
True,
id="assertion-triggerable-reachable-worth-fixing",
),
pytest.param(
{},
_result(exit_code=1, stderr="ERROR: AddressSanitizer: heap-buffer-overflow\n"),
_INFERRED,
TRIGGERABLE,
"sanitizer_abort",
True,
id="sanitizer-triggerable-inferred-worth-fixing",
),
pytest.param(
{},
_result(exit_code=139, stderr="Segmentation fault\n"),
_UNKNOWN,
TRIGGERABLE,
"crash",
False,
id="crash-triggerable-unknown-not-worth-fixing",
),
pytest.param(
{},
_result(exit_code=0, stdout="ok\n"),
_REACHABLE,
NOT_TRIGGERABLE,
"exit_zero",
False,
id="exit-zero-not-triggerable",
),
pytest.param(
{"covering_test": True},
_result(exit_code=0),
_UNKNOWN,
ALREADY_COVERED,
"covered_test_passed",
False,
id="already-covered-unknown-does-not-fail-build",
),
pytest.param(
{},
_result(exit_code=0),
_INFERRED,
NOT_TRIGGERABLE,
"exit_zero",
False,
id="not-triggerable-inferred-does-not-fail-build",
),
pytest.param(
{},
_result(exit_code=-1, timed_out=True),
_INFERRED,
INCONCLUSIVE,
"timeout",
False,
id="inconclusive-inferred-does-not-fail-build",
),
)
@pytest.mark.parametrize(
("flags", "result", "reachability", "expected_verdict", "expected_reason", "expected_worth_fixing"),
_CASES,
)
def test_sandbox_outcomes_map_to_verdicts_and_worth_fixing(
flags: dict[str, bool],
result: SandboxResult | None,
reachability: str,
expected_verdict: str,
expected_reason: str,
expected_worth_fixing: bool,
) -> None:
mapped = verdict_from_result(result, **flags)
assert mapped.verdict == expected_verdict
assert mapped.reason == expected_reason
assert mapped.reason
assert is_worth_fixing(mapped.verdict, reachability) is expected_worth_fixing
def test_table_covers_four_verdicts_and_worth_fixing_boolean() -> None:
verdicts = {case.values[3] for case in _CASES}
worth_fixing_values = {case.values[5] for case in _CASES}
assert verdicts == {TRIGGERABLE, NOT_TRIGGERABLE, ALREADY_COVERED, INCONCLUSIVE}
assert worth_fixing_values == {True, False}
def test_runner_maps_timeout_crash_and_exit_zero(tmp_path: Path) -> None:
snippet = "assert 1 + 1 == 2\n"
job = _job(tmp_path, snippet)
timeout = FakeSandboxRunner(_result(exit_code=-1, timed_out=True))
crash = FakeSandboxRunner(_result(exit_code=1, stderr="AssertionError\n"))
ok = FakeSandboxRunner(_result(exit_code=0))
timed_out = RegressionRunner(timeout).run_snippet(snippet, job)
triggered = RegressionRunner(crash).run_snippet(snippet, job)
not_triggered = RegressionRunner(ok).run_snippet(snippet, job)
assert timed_out.verdict == INCONCLUSIVE
assert timed_out.reason == "timeout"
assert triggered.verdict == TRIGGERABLE
assert not_triggered.verdict == NOT_TRIGGERABLE
assert is_worth_fixing(timed_out.verdict, _REACHABLE) is False
assert is_worth_fixing(triggered.verdict, _REACHABLE) is True
assert is_worth_fixing(not_triggered.verdict, _REACHABLE) is False
+353
View File
@@ -1,10 +1,18 @@
import json import json
import subprocess
from pathlib import Path from pathlib import Path
from types import SimpleNamespace
import pytest
from openultrasast.cli import main from openultrasast.cli import main
from openultrasast.complexity.hints import TestHint
from openultrasast.complexity.map import ComplexityMap, Hotspot
from openultrasast.findings import StaticFinding from openultrasast.findings import StaticFinding
from openultrasast.regress import CandidateVerdict
from openultrasast.reports import scan_exit_code, write_manifest, write_markdown_report, write_sarif_report from openultrasast.reports import scan_exit_code, write_manifest, write_markdown_report, write_sarif_report
from openultrasast.run import ScanRun from openultrasast.run import ScanRun
from openultrasast.stages import Stage, plan_for_mode, record_completed, stages_payload
from openultrasast.verification import verify_finding from openultrasast.verification import verify_finding
@@ -47,6 +55,34 @@ def test_manifest_links_shared_artifacts_by_finding_id(tmp_path: Path) -> None:
assert manifest_finding["finding_id"] == finding.finding_id assert manifest_finding["finding_id"] == finding.finding_id
assert manifest_finding["verification_status"] == "accepted" assert manifest_finding["verification_status"] == "accepted"
assert manifest_finding["artifact_refs"]["sarif"] == "report.sarif" assert manifest_finding["artifact_refs"]["sarif"] == "report.sarif"
assert "stages" not in payload
def test_manifest_includes_requested_completed_skipped_stages(tmp_path: Path) -> None:
run = ScanRun(scan_id="scan-1", root=tmp_path / "run", target=tmp_path / "repo")
run.root.mkdir()
finding = _finding()
artifacts = {
"findings": run.root / "findings.json",
"verification": run.root / "verification.json",
"markdown": run.root / "report.md",
"sarif": run.root / "report.sarif",
}
output = run.root / "manifest.json"
plan = record_completed(plan_for_mode("quick"), Stage.STATIC)
write_manifest(
run=run,
findings=[finding],
verifications=[verify_finding(finding)],
artifact_paths=artifacts,
path=output,
stages=stages_payload(plan),
)
payload = json.loads(output.read_text())
assert payload["stages"] == {"requested": ["static"], "completed": ["static"], "skipped": []}
assert payload["findings"][0]["finding_id"] == finding.finding_id
def test_markdown_report_includes_verification_status(tmp_path: Path) -> None: def test_markdown_report_includes_verification_status(tmp_path: Path) -> None:
@@ -56,18 +92,131 @@ def test_markdown_report_includes_verification_status(tmp_path: Path) -> None:
write_markdown_report([finding], output, [verify_finding(finding)]) write_markdown_report([finding], output, [verify_finding(finding)])
text = output.read_text() text = output.read_text()
assert "## Inventory" in text
assert "Verification: `accepted`" in text assert "Verification: `accepted`" in text
assert finding.finding_id in text assert finding.finding_id in text
assert "worth-fixing" not in text.lower()
assert "worth_fixing" not in text.lower()
def test_markdown_report_includes_complexity_map_and_worth_fixing_sections(tmp_path: Path) -> None:
finding = _finding()
output = tmp_path / "report.md"
hint = TestHint(
path="app.py",
function_name="admin",
gap="no_adjacent_test",
test_kind="http-contract",
reason="no adjacent test file for app.py::admin; recommend http-contract.",
)
hotspot = Hotspot(
path="app.py",
function_name="admin",
score=6.5,
band="high",
signals={"loc": 12, "has_adjacent_test": False},
rationale="high band: nested public route",
test_hint=hint,
inventory_finding_ids=(finding.finding_id,),
)
verdict = CandidateVerdict(
path="app.py",
function_name="admin",
language="python",
verdict="triggerable",
reason="crash",
inventory_finding_ids=(finding.finding_id,),
worth_fixing=True,
)
skipped = CandidateVerdict(
path="safe.py",
function_name=None,
language="python",
verdict="not_triggerable",
reason="exit_zero",
inventory_finding_ids=(),
worth_fixing=False,
)
write_markdown_report(
[finding],
output,
[verify_finding(finding)],
complexity_map=ComplexityMap(hotspots=(hotspot,), heuristic_only=True),
verdicts=(verdict, skipped),
)
text = output.read_text()
inventory_at = text.index("## Inventory")
map_at = text.index("## Complexity map")
worth_at = text.index("## Worth fixing")
assert inventory_at < map_at < worth_at
map_section = text[map_at:worth_at]
worth_section = text[worth_at:]
assert "app.py::admin" in map_section
assert "Gap: `no_adjacent_test`" in map_section
assert "Recommended test: `http-contract`" in map_section
assert "triggerable" in worth_section
assert "crash" in worth_section
assert "safe.py" not in worth_section
assert "## Inventory" in text
def test_markdown_report_labels_hits_as_inventory_not_worth_fixing(tmp_path: Path) -> None:
finding = _finding()
output = tmp_path / "report.md"
write_markdown_report([finding], output, [verify_finding(finding)])
text = output.read_text()
inventory_at = text.index("## Inventory")
finding_at = text.index(f"## {finding.title}")
assert inventory_at < finding_at
assert "worth-fixing" not in text.lower()
def test_scan_exit_code_policy() -> None: def test_scan_exit_code_policy() -> None:
finding = _finding() finding = _finding()
verification = verify_finding(finding) verification = verify_finding(finding)
worth_fixing = SimpleNamespace(worth_fixing=True)
assert scan_exit_code([finding], [verification], "never") == 0 assert scan_exit_code([finding], [verification], "never") == 0
assert scan_exit_code([finding], [verification], "findings") == 1 assert scan_exit_code([finding], [verification], "findings") == 1
assert scan_exit_code([], [], "findings") == 0
assert scan_exit_code([finding], [verification], "verified") == 1 assert scan_exit_code([finding], [verification], "verified") == 1
assert scan_exit_code([], [], "verified") == 0 assert scan_exit_code([], [], "verified") == 0
# findings/verified ignore worth-fixing verdicts
assert scan_exit_code([], [], "findings", worth_fixing_verdicts=[worth_fixing]) == 0
assert scan_exit_code([], [], "verified", worth_fixing_verdicts=[worth_fixing]) == 0
assert scan_exit_code([finding], [verification], "findings", worth_fixing_verdicts=[worth_fixing]) == 1
assert scan_exit_code([finding], [verification], "verified", worth_fixing_verdicts=[worth_fixing]) == 1
def test_scan_exit_code_worth_fixing() -> None:
finding = _finding()
verification = verify_finding(finding)
worth_fixing = SimpleNamespace(worth_fixing=True)
not_worth_fixing = SimpleNamespace(worth_fixing=False)
assert scan_exit_code([finding], [verification], "worth-fixing") == 0
assert scan_exit_code([finding], [verification], "worth-fixing", worth_fixing_verdicts=[]) == 0
assert scan_exit_code([finding], [verification], "worth-fixing", worth_fixing_verdicts=[not_worth_fixing]) == 0
assert scan_exit_code([finding], [verification], "worth-fixing", worth_fixing_verdicts=[worth_fixing]) == 1
def test_scan_exit_code_unknown_policy() -> None:
with pytest.raises(ValueError, match="unknown fail policy"):
scan_exit_code([], [], "maybe")
def test_scan_help_lists_worth_fixing(capsys: pytest.CaptureFixture[str]) -> None:
with pytest.raises(SystemExit) as exc:
main(["scan", "--help"])
assert exc.value.code == 0
help_text = capsys.readouterr().out
assert "worth-fixing" in help_text
assert "findings" in help_text
assert "verified" in help_text
def test_cli_scan_writes_sarif_manifest_and_honors_fail_policy(tmp_path: Path, monkeypatch) -> None: # type: ignore[no-untyped-def] def test_cli_scan_writes_sarif_manifest_and_honors_fail_policy(tmp_path: Path, monkeypatch) -> None: # type: ignore[no-untyped-def]
@@ -87,6 +236,210 @@ def test_cli_scan_writes_sarif_manifest_and_honors_fail_policy(tmp_path: Path, m
assert manifest["findings"][0]["artifact_refs"]["verification_json"] == "verification.json" assert manifest["findings"][0]["artifact_refs"]["verification_json"] == "verification.json"
def test_cli_scan_fail_on_worth_fixing_exits_zero_without_verdicts(tmp_path: Path, monkeypatch) -> None: # type: ignore[no-untyped-def]
repo = tmp_path / "repo"
repo.mkdir()
(repo / "app.py").write_text("@app.route('/admin')\ndef admin():\n return eval(request.data)\n")
monkeypatch.setenv("OPENULTRASAST_RUNS_DIR", ".runs")
assert main(["scan", str(repo), "--mode", "quick", "--fail-on", "worth-fixing"]) == 0
def test_quick_scan_report_is_inventory_without_map_or_docker(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
from openultrasast import tool_hunter
from openultrasast.complexity import map as complexity_map
from openultrasast.sandbox import probe as sandbox_probe
from openultrasast.sandbox import runner as sandbox_runner
repo = tmp_path / "repo"
repo.mkdir()
(repo / "app.py").write_text("@app.route('/admin')\ndef admin():\n return eval(request.data)\n")
monkeypatch.setenv("OPENULTRASAST_RUNS_DIR", ".runs")
docker_argv: list[list[str]] = []
original_run = subprocess.run
def guarded_run(command, *args, **kwargs): # type: ignore[no-untyped-def]
argv = list(command) if isinstance(command, list | tuple) else [command]
if argv and Path(str(argv[0])).name == "docker":
docker_argv.append(argv)
raise AssertionError(f"quick scan must not start docker: {argv}")
return original_run(command, *args, **kwargs)
def fail(message: str): # type: ignore[no-untyped-def]
def _fail(*args, **kwargs): # type: ignore[no-untyped-def]
raise AssertionError(message)
return _fail
monkeypatch.setattr(subprocess, "run", guarded_run)
monkeypatch.setattr(sandbox_runner, "build_docker_argv", fail("quick scan must not build docker argv"))
monkeypatch.setattr(sandbox_probe.SandboxProbe, "available", fail("quick scan must not probe docker"))
monkeypatch.setattr(complexity_map, "build_complexity_map", fail("quick scan must not build a complexity map"))
monkeypatch.setattr(tool_hunter, "run_tool_hunter", fail("quick scan must not start a hunter model"))
assert main(["scan", str(repo), "--mode", "quick"]) == 0
run_dir = sorted((repo / ".runs").iterdir())[-1]
report = (run_dir / "report.md").read_text()
manifest = json.loads((run_dir / "manifest.json").read_text())
assert "## Inventory" in report
assert "worth-fixing" not in report.lower()
assert not (run_dir / "complexity_map.json").exists()
assert docker_argv == []
assert manifest["stages"]["requested"] == ["static"]
assert manifest["stages"]["completed"] == ["static"]
assert manifest["stages"]["skipped"] == []
assert "complexity" not in manifest
assert "complexity_map" not in manifest.get("artifacts", {})
assert not any(entry.get("reason") == "hunter_model_unavailable" for entry in manifest.get("degradations", []))
def test_standard_scan_writes_complexity_map_without_docker(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
from openultrasast.sandbox import probe as sandbox_probe
from openultrasast.sandbox import runner as sandbox_runner
repo = tmp_path / "repo"
repo.mkdir()
(repo / "app.py").write_text("@app.route('/admin')\ndef admin():\n return eval(request.data)\n")
monkeypatch.setenv("OPENULTRASAST_RUNS_DIR", ".runs")
docker_argv: list[list[str]] = []
original_run = subprocess.run
def guarded_run(command, *args, **kwargs): # type: ignore[no-untyped-def]
argv = list(command) if isinstance(command, list | tuple) else [command]
if argv and Path(str(argv[0])).name == "docker":
docker_argv.append(argv)
raise AssertionError(f"standard scan must not start docker: {argv}")
return original_run(command, *args, **kwargs)
def fail(message: str): # type: ignore[no-untyped-def]
def _fail(*args, **kwargs): # type: ignore[no-untyped-def]
raise AssertionError(message)
return _fail
monkeypatch.setattr(subprocess, "run", guarded_run)
monkeypatch.setattr(sandbox_runner, "build_docker_argv", fail("standard scan must not build docker argv"))
monkeypatch.setattr(sandbox_probe.SandboxProbe, "available", fail("standard scan must not probe docker"))
assert main(["scan", str(repo), "--mode", "standard"]) == 0
run_dir = sorted((repo / ".runs").iterdir())[-1]
map_path = run_dir / "complexity_map.json"
manifest = json.loads((run_dir / "manifest.json").read_text())
complexity_map = json.loads(map_path.read_text())
assert map_path.is_file()
assert complexity_map["heuristic_only"] is True
assert isinstance(complexity_map["hotspots"], list)
assert complexity_map["hotspots"]
assert docker_argv == []
assert manifest["stages"]["requested"] == ["static", "map"]
assert manifest["stages"]["completed"] == ["static", "map"]
assert manifest["stages"]["skipped"] == []
assert manifest["complexity"]["hotspot_count"] == len(complexity_map["hotspots"])
assert manifest["complexity"]["heuristic_only"] is True
assert manifest["artifacts"]["complexity_map"] == "complexity_map.json"
report = (run_dir / "report.md").read_text()
assert "## Inventory" in report
assert "## Complexity map" in report
assert "Gap:" in report
assert "## Worth fixing" not in report
def test_standard_scan_records_hunter_model_unavailable_when_model_unset(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
from openultrasast.sandbox import probe as sandbox_probe
from openultrasast.sandbox import runner as sandbox_runner
repo = tmp_path / "repo"
repo.mkdir()
(repo / "app.py").write_text("@app.route('/admin')\ndef admin():\n return eval(request.data)\n")
monkeypatch.setenv("OPENULTRASAST_RUNS_DIR", ".runs")
def fail(message: str): # type: ignore[no-untyped-def]
def _fail(*args, **kwargs): # type: ignore[no-untyped-def]
raise AssertionError(message)
return _fail
monkeypatch.setattr(sandbox_runner, "build_docker_argv", fail("standard scan must not build docker argv"))
monkeypatch.setattr(sandbox_probe.SandboxProbe, "available", fail("standard scan must not probe docker"))
assert main(["scan", str(repo), "--mode", "standard"]) == 0
run_dir = sorted((repo / ".runs").iterdir())[-1]
manifest = json.loads((run_dir / "manifest.json").read_text())
degradations = manifest.get("degradations", [])
assert any(entry.get("reason") == "hunter_model_unavailable" for entry in degradations)
hunter_skip = next(entry for entry in degradations if entry.get("reason") == "hunter_model_unavailable")
assert hunter_skip["stage"] in {"map", "hunter_pool"}
assert manifest["complexity"]["heuristic_only"] is True
assert (run_dir / "complexity_map.json").is_file()
def test_manifest_includes_complexity_summary(tmp_path: Path) -> None:
run = ScanRun(scan_id="scan-1", root=tmp_path / "run", target=tmp_path / "repo")
run.root.mkdir()
finding = _finding()
artifacts = {
"findings": run.root / "findings.json",
"verification": run.root / "verification.json",
"markdown": run.root / "report.md",
"sarif": run.root / "report.sarif",
"complexity_map": run.root / "complexity_map.json",
}
output = run.root / "manifest.json"
write_manifest(
run=run,
findings=[finding],
verifications=[verify_finding(finding)],
artifact_paths=artifacts,
path=output,
complexity={"hotspot_count": 3, "heuristic_only": True},
)
payload = json.loads(output.read_text())
assert payload["complexity"] == {"hotspot_count": 3, "heuristic_only": True}
assert payload["artifacts"]["complexity_map"] == "complexity_map.json"
def test_manifest_includes_worth_fixing_count(tmp_path: Path) -> None:
run = ScanRun(scan_id="scan-1", root=tmp_path / "run", target=tmp_path / "repo")
run.root.mkdir()
finding = _finding()
artifacts = {
"findings": run.root / "findings.json",
"verification": run.root / "verification.json",
"markdown": run.root / "report.md",
"sarif": run.root / "report.sarif",
"verdicts": run.root / "verdicts.json",
}
output = run.root / "manifest.json"
write_manifest(
run=run,
findings=[finding],
verifications=[verify_finding(finding)],
artifact_paths=artifacts,
path=output,
stages={"requested": ["static", "map", "regress"], "completed": ["static", "map", "regress"], "skipped": []},
worth_fixing={
"count": 1,
"verdicts": [{"path": "app.py", "function_name": "admin", "verdict": "triggerable", "reason": "crash"}],
},
)
payload = json.loads(output.read_text())
assert payload["stages"]["completed"] == ["static", "map", "regress"]
assert payload["worth_fixing"]["count"] == 1
assert payload["worth_fixing"]["verdicts"][0]["verdict"] == "triggerable"
def _finding() -> StaticFinding: def _finding() -> StaticFinding:
return StaticFinding( return StaticFinding(
finding_id="python-unsafe-eval:app.py:3", finding_id="python-unsafe-eval:app.py:3",
+117
View File
@@ -0,0 +1,117 @@
import subprocess
from typing import Any
import pytest
from openultrasast.sandbox import FakeSandboxProbe, SandboxProbe, resolve_sandbox_probe
from openultrasast.sandbox import probe as probe_mod
def _completed(returncode: int) -> subprocess.CompletedProcess[str]:
return subprocess.CompletedProcess(["docker", "info"], returncode, stdout="", stderr="")
def test_fake_probe_can_be_forced_off() -> None:
probe = FakeSandboxProbe(available=False)
assert probe.available() is False
def test_fake_probe_can_be_forced_on() -> None:
probe = FakeSandboxProbe(available=True)
assert probe.available() is True
def test_resolve_probe_forced_off_via_env(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv(probe_mod.PROBE_ENV, "0")
probe = resolve_sandbox_probe()
assert isinstance(probe, FakeSandboxProbe)
assert probe.available() is False
def test_resolve_probe_forced_on_via_env(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv(probe_mod.PROBE_ENV, "1")
probe = resolve_sandbox_probe()
assert isinstance(probe, FakeSandboxProbe)
assert probe.available() is True
def test_resolve_probe_defaults_to_real_probe(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.delenv(probe_mod.PROBE_ENV, raising=False)
assert isinstance(resolve_sandbox_probe(), SandboxProbe)
def test_real_probe_is_available_when_docker_info_succeeds() -> None:
def runner(*_args: object, **_kwargs: object) -> subprocess.CompletedProcess[str]:
return _completed(0)
assert SandboxProbe(runner=runner).available() is True
def test_real_probe_is_unavailable_when_docker_info_fails() -> None:
def runner(*_args: object, **_kwargs: object) -> subprocess.CompletedProcess[str]:
return _completed(1)
assert SandboxProbe(runner=runner).available() is False
@pytest.mark.parametrize(
"error",
[
FileNotFoundError("docker"),
PermissionError("denied"),
OSError("cannot talk to daemon"),
subprocess.TimeoutExpired(cmd=["docker", "info"], timeout=5),
RuntimeError("unexpected probe failure"),
],
)
def test_real_probe_returns_unavailable_without_raising(error: Exception) -> None:
def runner(*_args: object, **_kwargs: object) -> subprocess.CompletedProcess[str]:
raise error
assert SandboxProbe(runner=runner).available() is False
def test_real_probe_runs_docker_info_with_short_timeout() -> None:
seen: dict[str, Any] = {}
def runner(command: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]:
seen["command"] = list(command)
seen["timeout"] = kwargs.get("timeout")
seen["check"] = kwargs.get("check")
return _completed(0)
assert SandboxProbe(runner=runner).available() is True
assert seen["command"] == ["docker", "info"]
assert seen["timeout"] == probe_mod.PROBE_TIMEOUT_SECONDS
assert probe_mod.PROBE_TIMEOUT_SECONDS <= 10
assert seen["check"] is False
def test_real_probe_is_skippable_in_unit_tests(monkeypatch: pytest.MonkeyPatch) -> None:
def boom(*_args: object, **_kwargs: object) -> subprocess.CompletedProcess[str]:
raise AssertionError("unit tests must not invoke the docker daemon")
monkeypatch.setattr(probe_mod.subprocess, "run", boom)
def runner(*_args: object, **_kwargs: object) -> subprocess.CompletedProcess[str]:
return _completed(0)
assert SandboxProbe(runner=runner).available() is True
def test_real_probe_default_path_uses_subprocess_run(monkeypatch: pytest.MonkeyPatch) -> None:
seen: dict[str, Any] = {}
def fake_run(command: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]:
seen["command"] = list(command)
seen["timeout"] = kwargs.get("timeout")
seen["capture_output"] = kwargs.get("capture_output")
seen["text"] = kwargs.get("text")
return _completed(0)
monkeypatch.setattr(probe_mod.subprocess, "run", fake_run)
assert SandboxProbe().available() is True
assert seen["command"] == ["docker", "info"]
assert seen["timeout"] == probe_mod.PROBE_TIMEOUT_SECONDS
assert seen["capture_output"] is True
assert seen["text"] is True
+364
View File
@@ -0,0 +1,364 @@
from __future__ import annotations
import os
import stat
import subprocess
from pathlib import Path
from typing import Any
import pytest
from openultrasast.sandbox import (
DockerCliRunner,
FakeSandboxRunner,
SandboxJob,
SandboxResult,
SandboxRunner,
build_docker_argv,
resolve_sandbox_runner,
)
from openultrasast.sandbox import runner as runner_mod
def _job(tmp_path: Path, **overrides: object) -> SandboxJob:
repo = tmp_path / "repo"
repo.mkdir(exist_ok=True)
(repo / "app.py").write_text("print(1)\n")
values: dict[str, object] = {
"image": "python:3.12-alpine",
"command": ("python", "/scratch/case.py"),
"repo_root": repo,
"scratch_files": {"case.py": "print('ok')\n"},
"timeout_seconds": 30,
"memory_mb": 256,
"pids_limit": 128,
}
values.update(overrides)
return SandboxJob(**values) # type: ignore[arg-type]
def _flag_value(argv: list[str], flag: str) -> str:
try:
return argv[argv.index(flag) + 1]
except (ValueError, IndexError) as exc:
raise AssertionError(f"{flag} missing from {argv}") from exc
def _mount_options(spec: str) -> dict[str, str]:
options: dict[str, str] = {}
for part in spec.split(","):
if "=" in part:
key, value = part.split("=", 1)
options[key] = value
else:
options[part] = "true"
return options
def _workspace_mount(argv: list[str]) -> dict[str, str]:
specs: list[str] = []
for index, item in enumerate(argv):
if item in {"--mount", "-v", "--volume"} and index + 1 < len(argv):
specs.append(argv[index + 1])
for spec in specs:
options = _mount_options(spec)
target = options.get("dst") or options.get("destination") or options.get("target")
if target == "/workspace" or spec.endswith(":/workspace") or ":/workspace:" in spec:
return options if target == "/workspace" else {"raw": spec}
if "/workspace" in spec:
return options if options else {"raw": spec}
raise AssertionError(f"no /workspace mount in {argv}")
def _seed_source(argv: list[str]) -> Path:
mounts = [argv[index + 1] for index, item in enumerate(argv) if item == "--mount" and index + 1 < len(argv)]
seed_mounts = [spec for spec in mounts if "/scratch-in" in spec]
assert seed_mounts, argv
options = _mount_options(seed_mounts[0])
source = Path(options.get("src") or options.get("source") or "")
assert source.is_dir(), argv
return source
def _copy_script(argv: list[str]) -> str:
try:
return argv[argv.index("-c") + 1]
except (ValueError, IndexError) as exc:
raise AssertionError(f"no /bin/sh -c wrapper in {argv}") from exc
def _posix_mode(path: Path) -> int:
return path.stat().st_mode & 0o777
def test_build_docker_argv_starts_with_docker_run_rm_init(tmp_path: Path) -> None:
argv = build_docker_argv(_job(tmp_path))
assert argv[0] == "docker"
assert argv[1] == "run"
assert "--rm" in argv
assert "--init" in argv
def test_build_docker_argv_includes_network_none_and_read_only(tmp_path: Path) -> None:
argv = build_docker_argv(_job(tmp_path))
joined = " ".join(argv)
assert "--read-only" in argv
assert "--network none" in joined or "--network=none" in argv
assert "--network host" not in joined
assert "--network=host" not in argv
def test_build_docker_argv_excludes_docker_sock(tmp_path: Path) -> None:
argv = build_docker_argv(_job(tmp_path))
joined = " ".join(argv)
assert "docker.sock" not in joined
assert "/var/run/docker.sock" not in joined
assert "-v" not in argv
assert "--volume" not in argv
def test_build_docker_argv_forbids_privileged_and_host_network(tmp_path: Path) -> None:
argv = build_docker_argv(_job(tmp_path))
assert "--privileged" not in argv
assert "--pid" not in argv or _flag_value(argv, "--pid") != "host"
assert "--ipc" not in argv or _flag_value(argv, "--ipc") != "host"
assert "--userns" not in argv or "host" not in _flag_value(argv, "--userns")
def test_build_docker_argv_scratch_is_writable_tmpfs(tmp_path: Path) -> None:
argv = build_docker_argv(_job(tmp_path))
tmpfs = _flag_value(argv, "--tmpfs")
assert tmpfs.startswith("/scratch")
assert "rw" in tmpfs
assert "exec" in tmpfs
def test_build_docker_argv_applies_memory_and_pids_limits(tmp_path: Path) -> None:
job = _job(tmp_path, memory_mb=512, pids_limit=64)
argv = build_docker_argv(job)
memory = _flag_value(argv, "--memory")
assert "512" in memory
assert memory.endswith("m") or memory.endswith("M") or memory.endswith("b")
assert _flag_value(argv, "--pids-limit") == "64"
def test_build_docker_argv_drops_capabilities_and_new_privileges(tmp_path: Path) -> None:
argv = build_docker_argv(_job(tmp_path))
assert _flag_value(argv, "--cap-drop") == "ALL"
security = _flag_value(argv, "--security-opt")
assert "no-new-privileges" in security
def test_build_docker_argv_uses_non_root_user(tmp_path: Path) -> None:
argv = build_docker_argv(_job(tmp_path))
user = _flag_value(argv, "--user")
uid = user.split(":", 1)[0]
assert uid not in {"0", "root"}
assert int(uid) != 0
def test_build_docker_argv_bind_mounts_repo_read_only_at_workspace(tmp_path: Path) -> None:
job = _job(tmp_path)
argv = build_docker_argv(job)
mount = _workspace_mount(argv)
source = mount.get("src") or mount.get("source")
assert source == str(job.repo_root.resolve())
assert mount.get("dst") == "/workspace" or mount.get("destination") == "/workspace" or mount.get("target") == "/workspace"
assert mount.get("ro") == "true" or mount.get("readonly") == "true"
assert "rw" not in mount
def test_build_docker_argv_source_tree_is_not_writable(tmp_path: Path) -> None:
argv = build_docker_argv(_job(tmp_path))
mount = _workspace_mount(argv)
assert mount.get("ro") == "true" or mount.get("readonly") == "true"
assert "--read-only" in argv
joined = " ".join(argv)
assert f"{_job(tmp_path).repo_root.resolve()}:/workspace:rw" not in joined
assert f"{_job(tmp_path).repo_root.resolve()}:/workspace " not in joined + " "
def test_build_docker_argv_ends_with_image_and_command(tmp_path: Path) -> None:
job = _job(tmp_path)
argv = build_docker_argv(job)
assert argv[-len(job.command) :] == list(job.command)
assert argv[-len(job.command) - 1] == job.image
def test_fake_sandbox_runner_records_jobs_and_returns_programmed_result(tmp_path: Path) -> None:
programmed = SandboxResult(exit_code=7, stdout="out", stderr="err", timed_out=False)
runner: SandboxRunner = FakeSandboxRunner(programmed)
job = _job(tmp_path)
result = runner.run(job)
assert result == programmed
assert isinstance(runner, FakeSandboxRunner)
assert runner.jobs == [job]
def test_resolve_runner_fake_via_env(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv(runner_mod.RUNNER_ENV, "fake")
assert isinstance(resolve_sandbox_runner(), FakeSandboxRunner)
def test_resolve_runner_defaults_to_docker_cli(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.delenv(runner_mod.RUNNER_ENV, raising=False)
assert isinstance(resolve_sandbox_runner(), DockerCliRunner)
def test_fake_sandbox_runner_can_be_reprogrammed(tmp_path: Path) -> None:
runner = FakeSandboxRunner()
first = _job(tmp_path, command=("echo", "one"))
second = _job(tmp_path, command=("echo", "two"))
timeout_result = SandboxResult(exit_code=-1, stdout="", stderr="deadline", timed_out=True)
runner.result = timeout_result
assert runner.run(first).timed_out is True
runner.result = SandboxResult(exit_code=0, stdout="ok", stderr="", timed_out=False)
assert runner.run(second).stdout == "ok"
assert runner.jobs == [first, second]
def test_fake_sandbox_runner_does_not_need_docker_daemon(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
def boom(*_args: object, **_kwargs: object) -> subprocess.CompletedProcess[str]:
raise AssertionError("unit tests must not invoke the docker daemon")
monkeypatch.setattr(subprocess, "run", boom)
monkeypatch.setattr(runner_mod.subprocess, "run", boom)
programmed = SandboxResult(exit_code=0, stdout="fake", stderr="", timed_out=False)
runner = FakeSandboxRunner(programmed)
assert runner.run(_job(tmp_path)) == programmed
def test_docker_cli_runner_uses_constructed_argv_and_timeout(tmp_path: Path) -> None:
seen: dict[str, Any] = {}
job = _job(tmp_path, scratch_files={}, timeout_seconds=19)
def fake_run(command: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]:
seen["command"] = list(command)
seen["timeout"] = kwargs.get("timeout")
seen["check"] = kwargs.get("check")
seen["capture_output"] = kwargs.get("capture_output")
seen["text"] = kwargs.get("text")
return subprocess.CompletedProcess(command, 3, stdout="hello", stderr="warn")
result = DockerCliRunner(runner=fake_run).run(job)
assert seen["command"] == build_docker_argv(job)
assert seen["timeout"] == 19
assert seen["check"] is False
assert seen["capture_output"] is True
assert seen["text"] is True
assert result == SandboxResult(exit_code=3, stdout="hello", stderr="warn", timed_out=False)
assert "--network" in seen["command"]
assert "--read-only" in seen["command"]
assert "docker.sock" not in " ".join(seen["command"])
def test_docker_cli_runner_maps_timeout_expired_to_timed_out(tmp_path: Path) -> None:
job = _job(tmp_path, scratch_files={}, timeout_seconds=5)
def fake_run(command: list[str], **_kwargs: Any) -> subprocess.CompletedProcess[str]:
raise subprocess.TimeoutExpired(cmd=command, timeout=5, output="partial", stderr="slow")
result = DockerCliRunner(runner=fake_run).run(job)
assert result.timed_out is True
assert result.stdout == "partial"
assert result.stderr == "slow"
def test_docker_cli_runner_does_not_invoke_daemon_when_runner_injected(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
def boom(*_args: object, **_kwargs: object) -> subprocess.CompletedProcess[str]:
raise AssertionError("unit tests must not invoke the docker daemon")
monkeypatch.setattr(subprocess, "run", boom)
monkeypatch.setattr(runner_mod.subprocess, "run", boom)
def fake_run(command: list[str], **_kwargs: Any) -> subprocess.CompletedProcess[str]:
return subprocess.CompletedProcess(command, 0, stdout="", stderr="")
result = DockerCliRunner(runner=fake_run).run(_job(tmp_path, scratch_files={}))
assert result.timed_out is False
assert result.exit_code == 0
def test_docker_cli_runner_does_not_write_the_source_tree(tmp_path: Path) -> None:
job = _job(tmp_path)
original = (job.repo_root / "app.py").read_text()
before = {path.relative_to(job.repo_root): path.read_text() for path in job.repo_root.rglob("*") if path.is_file()}
def fake_run(command: list[str], **_kwargs: Any) -> subprocess.CompletedProcess[str]:
joined = " ".join(command)
assert "docker.sock" not in joined
assert "--read-only" in command
return subprocess.CompletedProcess(command, 0, stdout="", stderr="")
DockerCliRunner(runner=fake_run).run(job)
assert (job.repo_root / "app.py").read_text() == original
after = {path.relative_to(job.repo_root): path.read_text() for path in job.repo_root.rglob("*") if path.is_file()}
assert after == before
assert "case.py" not in {path.name for path in job.repo_root.iterdir()}
def test_docker_cli_runner_seeds_scratch_files_outside_the_repo(tmp_path: Path) -> None:
job = _job(tmp_path, scratch_files={"nested/case.py": "print('seeded')\n"})
seen: dict[str, Any] = {}
def fake_run(command: list[str], **_kwargs: Any) -> subprocess.CompletedProcess[str]:
seen["command"] = list(command)
mounts = [command[index + 1] for index, item in enumerate(command) if item == "--mount" and index + 1 < len(command)]
seed_mounts = [spec for spec in mounts if "/scratch-in" in spec]
assert seed_mounts, command
options = _mount_options(seed_mounts[0])
source = Path(options.get("src") or options.get("source") or "")
assert source.is_dir()
assert job.repo_root.resolve() not in source.resolve().parents
assert source.resolve() != job.repo_root.resolve()
assert (source / "nested" / "case.py").read_text() == "print('seeded')\n"
assert options.get("ro") == "true" or options.get("readonly") == "true"
assert "--tmpfs" in command
assert _flag_value(command, "--tmpfs").startswith("/scratch")
return subprocess.CompletedProcess(command, 0, stdout="", stderr="")
result = DockerCliRunner(runner=fake_run).run(job)
assert result.exit_code == 0
argv = seen["command"]
assert "docker.sock" not in " ".join(argv)
assert "--read-only" in argv
assert job.image in argv
def test_docker_cli_runner_scratch_seed_is_world_readable_for_container_user(tmp_path: Path) -> None:
job = _job(tmp_path, scratch_files={"nested/dir/case.py": "print('seeded')\n"})
seen: dict[str, Path] = {}
def fake_run(command: list[str], **_kwargs: Any) -> subprocess.CompletedProcess[str]:
source = _seed_source(command)
seen["source"] = source
assert _posix_mode(source) == 0o755
for dirpath, _dirnames, filenames in os.walk(source):
directory = Path(dirpath)
assert stat.S_ISDIR(directory.stat().st_mode)
assert _posix_mode(directory) == 0o755, directory
for name in filenames:
file_path = directory / name
assert _posix_mode(file_path) == 0o644, file_path
return subprocess.CompletedProcess(command, 0, stdout="", stderr="")
DockerCliRunner(runner=fake_run).run(job)
assert "source" in seen
def test_docker_cli_runner_scratch_copy_uses_cp_r_not_cp_a(tmp_path: Path) -> None:
job = _job(tmp_path, scratch_files={"nested/case.py": "print('seeded')\n"})
seen: dict[str, str] = {}
def fake_run(command: list[str], **_kwargs: Any) -> subprocess.CompletedProcess[str]:
script = _copy_script(command)
seen["script"] = script
assert command[command.index(job.image) + 1] == "/bin/sh"
assert "cp -a" not in script
assert "cp -a" not in " ".join(command)
assert script.startswith("cp -r /scratch-in/. /scratch/")
assert 'exec "$@"' in script
return subprocess.CompletedProcess(command, 0, stdout="", stderr="")
DockerCliRunner(runner=fake_run).run(job)
assert seen["script"] == 'cp -r /scratch-in/. /scratch/ && exec "$@"'
+200
View File
@@ -0,0 +1,200 @@
"""Deep mode: skip regression when the sandbox is missing; record verdicts with a fake runner (task 6.3)."""
from __future__ import annotations
import json
import subprocess
from pathlib import Path
import pytest
from openultrasast.cli import main
from openultrasast.sandbox import FakeSandboxRunner, SandboxResult
from openultrasast.sandbox import runner as sandbox_runner
_VULN = "@app.route('/admin')\ndef admin():\n return eval(request.data)\n"
def _repo(tmp_path: Path) -> Path:
repo = tmp_path / "repo"
repo.mkdir()
(repo / "app.py").write_text(_VULN)
return repo
def _fail(message: str): # type: ignore[no-untyped-def]
def _inner(*args: object, **kwargs: object) -> object:
raise AssertionError(message)
return _inner
def _guard_docker(monkeypatch: pytest.MonkeyPatch) -> list[list[str]]:
docker_argv: list[list[str]] = []
original_run = subprocess.run
def guarded_run(command: object, *args: object, **kwargs: object) -> object:
argv = list(command) if isinstance(command, list | tuple) else [command]
if argv and Path(str(argv[0])).name == "docker":
docker_argv.append([str(item) for item in argv])
raise AssertionError(f"deep scan test must not start docker: {argv}")
return original_run(command, *args, **kwargs)
monkeypatch.setattr(subprocess, "run", guarded_run)
monkeypatch.setattr(sandbox_runner.DockerCliRunner, "run", _fail("deep scan test must not use DockerCliRunner"))
return docker_argv
def _latest_run(repo: Path) -> Path:
return sorted((repo / ".runs").iterdir())[-1]
def _crash_runner() -> FakeSandboxRunner:
return FakeSandboxRunner(SandboxResult(exit_code=139, stdout="", stderr="Segmentation fault\n", timed_out=False))
def _enable_crash_sandbox(monkeypatch: pytest.MonkeyPatch) -> FakeSandboxRunner:
from openultrasast import cli
fake = _crash_runner()
monkeypatch.setenv("OPENULTRASAST_RUNS_DIR", ".runs")
monkeypatch.setenv("OPENULTRASAST_SANDBOX_PROBE", "1")
monkeypatch.setattr(cli, "resolve_sandbox_runner", lambda: fake)
return fake
def test_deep_scan_skips_regress_when_sandbox_unavailable(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
repo = _repo(tmp_path)
monkeypatch.setenv("OPENULTRASAST_RUNS_DIR", ".runs")
monkeypatch.setenv("OPENULTRASAST_SANDBOX_PROBE", "0")
docker_argv = _guard_docker(monkeypatch)
assert main(["scan", str(repo), "--mode", "deep"]) == 0
run_dir = _latest_run(repo)
manifest = json.loads((run_dir / "manifest.json").read_text())
stages = manifest["stages"]
degradations = manifest.get("degradations", [])
assert (run_dir / "complexity_map.json").is_file()
assert not (run_dir / "verdicts.json").exists()
assert docker_argv == []
assert stages["requested"] == ["static", "map", "regress"]
assert stages["completed"] == ["static", "map"]
assert {"stage": "regress", "reason": "sandbox_unavailable"} in stages["skipped"]
assert any(entry.get("stage") == "regress" and entry.get("reason") == "sandbox_unavailable" for entry in degradations)
def test_deep_scan_with_fake_runner_writes_verdicts(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
repo = _repo(tmp_path)
monkeypatch.setenv("OPENULTRASAST_RUNS_DIR", ".runs")
monkeypatch.setenv("OPENULTRASAST_SANDBOX_PROBE", "1")
monkeypatch.setenv("OPENULTRASAST_SANDBOX_RUNNER", "fake")
docker_argv = _guard_docker(monkeypatch)
assert main(["scan", str(repo), "--mode", "deep"]) == 0
run_dir = _latest_run(repo)
verdicts_path = run_dir / "verdicts.json"
manifest = json.loads((run_dir / "manifest.json").read_text())
payload = json.loads(verdicts_path.read_text())
stages = manifest["stages"]
assert docker_argv == []
assert verdicts_path.is_file()
assert (run_dir / "complexity_map.json").is_file()
assert stages["requested"] == ["static", "map", "regress"]
assert stages["completed"] == ["static", "map", "regress"]
assert not any(entry.get("reason") == "sandbox_unavailable" for entry in stages["skipped"])
assert isinstance(payload["verdicts"], list)
assert payload["verdicts"]
for item in payload["verdicts"]:
assert item["verdict"] in {"triggerable", "not_triggerable", "already_covered", "inconclusive"}
assert "reason" in item
assert "path" in item
assert manifest["artifacts"]["verdicts"] == "verdicts.json"
assert manifest["artifacts"]["complexity_map"] == "complexity_map.json"
assert manifest["worth_fixing"]["count"] == sum(1 for item in payload["verdicts"] if item.get("worth_fixing"))
report = (run_dir / "report.md").read_text()
inventory_at = report.index("## Inventory")
map_at = report.index("## Complexity map")
worth_at = report.index("## Worth fixing")
assert inventory_at < map_at < worth_at
map_section = report[map_at:worth_at]
assert "app.py" in map_section
assert "Gap:" in map_section
ledger_path = repo / ".openultrasast" / "calibration" / "complexity_ledger.json"
assert ledger_path.is_file()
ledger = json.loads(ledger_path.read_text())
assert ledger
assert any(entry.get("last_verdict") in {"triggerable", "not_triggerable"} for entry in ledger.values())
def test_deep_scan_unsafe_hunter_snippet_is_safety_rejected_without_docker_job(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
from openultrasast import cli
from openultrasast.sandbox import FakeSandboxRunner
repo = _repo(tmp_path)
config = tmp_path / "openultrasast.toml"
config.write_text('[models]\nhunter = "test-hunter"\n')
fake = FakeSandboxRunner()
monkeypatch.setenv("OPENULTRASAST_RUNS_DIR", ".runs")
monkeypatch.setenv("OPENULTRASAST_SANDBOX_PROBE", "1")
monkeypatch.setenv("OPENULTRASAST_HUNTER_CLIENT", "unsafe-snippet")
monkeypatch.setattr(cli, "has_harnessx", lambda: False)
monkeypatch.setattr(cli, "resolve_sandbox_runner", lambda: fake)
docker_argv = _guard_docker(monkeypatch)
assert main(["scan", str(repo), "--mode", "deep", "--config", str(config)]) == 0
run_dir = _latest_run(repo)
payload = json.loads((run_dir / "verdicts.json").read_text())
findings = json.loads((run_dir / "findings.json").read_text())["findings"]
hunter_findings = [item for item in findings if str(item["finding_id"]).startswith("tool-hunter:")]
assert docker_argv == []
assert fake.jobs == []
assert hunter_findings
assert all(item["evidence_level"] == "suspicion" for item in hunter_findings)
assert payload["verdicts"]
assert any(item["reason"] == "safety_rejected" for item in payload["verdicts"])
for item in payload["verdicts"]:
if item["reason"] == "safety_rejected":
assert item["verdict"] == "inconclusive"
def test_deep_scan_fail_on_worth_fixing_exits_one_when_triggerable(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
repo = _repo(tmp_path)
_enable_crash_sandbox(monkeypatch)
docker_argv = _guard_docker(monkeypatch)
assert main(["scan", str(repo), "--mode", "deep", "--fail-on", "worth-fixing"]) == 1
assert docker_argv == []
run_dir = _latest_run(repo)
payload = json.loads((run_dir / "verdicts.json").read_text())
manifest = json.loads((run_dir / "manifest.json").read_text())
report = (run_dir / "report.md").read_text()
worth_fixing = [item for item in payload["verdicts"] if item.get("worth_fixing")]
assert worth_fixing
assert len(worth_fixing) >= 1
assert any(item["verdict"] == "triggerable" for item in worth_fixing)
assert manifest["worth_fixing"]["count"] == len(worth_fixing)
assert "## Complexity map" in report
assert "## Worth fixing" in report
assert "triggerable" in report
ledger = json.loads((repo / ".openultrasast" / "calibration" / "complexity_ledger.json").read_text())
assert any(entry.get("last_verdict") == "triggerable" for entry in ledger.values())
def test_deep_scan_fail_on_never_exits_zero_with_worth_fixing_verdict(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
repo = _repo(tmp_path)
_enable_crash_sandbox(monkeypatch)
docker_argv = _guard_docker(monkeypatch)
assert main(["scan", str(repo), "--mode", "deep", "--fail-on", "never"]) == 0
assert docker_argv == []
payload = json.loads((_latest_run(repo) / "verdicts.json").read_text())
assert any(item.get("worth_fixing") for item in payload["verdicts"])
+4 -1
View File
@@ -36,6 +36,8 @@ def _scan(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, *, mode: str, config_
(repo / "app.py").write_text(_VULN) (repo / "app.py").write_text(_VULN)
monkeypatch.setattr(cli, "has_harnessx", lambda: harnessx) monkeypatch.setattr(cli, "has_harnessx", lambda: harnessx)
monkeypatch.setenv("OPENULTRASAST_RUNS_DIR", ".runs") monkeypatch.setenv("OPENULTRASAST_RUNS_DIR", ".runs")
monkeypatch.delenv("OPENROUTER_API_KEY", raising=False)
monkeypatch.delenv("OPENULTRASAST_HUNTER_CLIENT", raising=False)
argv = ["scan", str(repo), "--mode", mode] argv = ["scan", str(repo), "--mode", mode]
if config_body is not None: if config_body is not None:
cfg = tmp_path / "openultrasast.toml" cfg = tmp_path / "openultrasast.toml"
@@ -65,7 +67,8 @@ def test_standard_scan_extra_absent_matches_deterministic_baseline(tmp_path: Pat
stages = {entry["stage"] for entry in requested_manifest.get("degradations", [])} stages = {entry["stage"] for entry in requested_manifest.get("degradations", [])}
assert stages == {"hunter_pool", "verify"} assert stages == {"hunter_pool", "verify"}
assert all(entry["reason"] == "harnessx_extra_unavailable" for entry in requested_manifest["degradations"]) assert all(entry["reason"] == "harnessx_extra_unavailable" for entry in requested_manifest["degradations"])
assert "degradations" not in baseline_manifest assert all(entry["reason"] != "hunter_model_unavailable" for entry in requested_manifest["degradations"])
assert [entry["reason"] for entry in baseline_manifest.get("degradations", [])] == ["hunter_model_unavailable"]
def test_quick_scan_is_unaffected_by_the_extra(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: def test_quick_scan_is_unaffected_by_the_extra(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+70
View File
@@ -0,0 +1,70 @@
"""Split-sink fixtures: stage-1 pattern scan must miss planted sinks (requirement 8.1)."""
from pathlib import Path
import pytest
from openultrasast.benchmark import (
BenchmarkManifest,
BenchmarkRun,
ExpectedFinding,
evaluate_benchmark,
load_benchmark_manifest,
resolve_benchmark_source,
)
from openultrasast.findings import StaticFinding, quick_scan_findings
from openultrasast.gate import LANGUAGE_MANIFESTS
from openultrasast.preprocess import preprocess_repository
from openultrasast.rank import rank_targets
MANIFEST_DIR = Path("benchmarks/manifests")
SPLIT_SINK_MANIFESTS = (
"split-sink-c",
"split-sink-java",
"split-sink-javascript",
"split-sink-python",
)
def _quick_scan(target: Path) -> list[StaticFinding]:
_, targets = preprocess_repository(target)
return quick_scan_findings(target, targets, rank_targets(targets))
def _planted_split_sinks(manifest: BenchmarkManifest) -> list[ExpectedFinding]:
return [item for item in manifest.expected if item.rule_id is None]
@pytest.mark.parametrize("name", SPLIT_SINK_MANIFESTS)
def test_stage1_misses_planted_split_sink(name: str) -> None:
manifest_path = MANIFEST_DIR / f"{name}.toml"
manifest = load_benchmark_manifest(manifest_path)
target = resolve_benchmark_source(manifest_path, manifest)
planted = _planted_split_sinks(manifest)
assert planted, f"{name} must document a stage-1 miss without rule_id"
findings = _quick_scan(target)
run = BenchmarkRun(benchmark_run_id="split-sink", root=Path("/tmp/split-sink"), manifest=manifest)
result = evaluate_benchmark(run=run, mode="quick", findings=findings, scan_id=None, scan_run_dir=None)
missed = {(miss.path, miss.cwe, miss.rule_id) for miss in result.misses}
for expected in planted:
assert expected.line is not None
assert expected.sink
source = (target / expected.path).read_text()
lines = source.splitlines()
sink_line = lines[expected.line - 1]
assert expected.sink.lower() in sink_line.lower(), f"{name} planted sink {expected.sink!r} missing on line {expected.line}"
prior = "\n".join(lines[: expected.line - 1]).lower()
assert any(token in prior for token in ("query", "sql")), f"{name} must assign the payload before the sink line"
on_line = [finding for finding in findings if finding.path == expected.path and finding.line == expected.line]
assert on_line == [], f"{name} stage-1 hit the planted split-sink on {expected.path}:{expected.line}: {on_line}"
assert (expected.path, expected.cwe, None) in missed, (
f"{name} evaluate_benchmark must record a miss for {expected.path}:{expected.line}"
)
def test_split_sink_fixtures_are_outside_stage1_gate_corpus() -> None:
listed = {name for names in LANGUAGE_MANIFESTS.values() for name in names}
leaked = [name for name in SPLIT_SINK_MANIFESTS if name in listed]
assert leaked == [], "split-sink fixtures must not join LANGUAGE_MANIFESTS (stage-1 90% smoke gate)"
+40
View File
@@ -0,0 +1,40 @@
"""Stage-3 regression on split-sink: fake runner always; live Docker skips without a daemon."""
from pathlib import Path
import pytest
from openultrasast.config import SandboxConfig
from openultrasast.regress.recipes import recipe_for
from openultrasast.regress.safety import check_snippet_safety
from openultrasast.regress.verdict import TRIGGERABLE, verdict_from_result
from openultrasast.sandbox import DockerCliRunner, FakeSandboxRunner, SandboxProbe, SandboxResult
_SPLIT_SINK_PYTHON = Path("benchmarks/fixtures/split-sink-python")
_LIMITS = SandboxConfig(memory_mb=256, timeout_seconds=30, pids_limit=128)
_PYTHON_IMAGE = "python:3.12-alpine"
_SNIPPET = 'term = "x\'; drop"\nquery = "select * from items where title like \'%" + term + "%\'"\nassert "\'" not in query\n'
def _split_sink_job():
check_snippet_safety(_SNIPPET)
job = recipe_for("python", _SNIPPET, _PYTHON_IMAGE, _LIMITS, repo_root=_SPLIT_SINK_PYTHON.resolve())
assert job is not None
return job
def test_fake_runner_marks_split_sink_snippet_triggerable() -> None:
fake = FakeSandboxRunner(SandboxResult(exit_code=1, stdout="", stderr="AssertionError", timed_out=False))
result = fake.run(_split_sink_job())
verdict = verdict_from_result(result)
assert verdict.verdict == TRIGGERABLE
assert fake.jobs
@pytest.mark.docker
def test_docker_marks_split_sink_python_triggerable() -> None:
if not SandboxProbe().available():
pytest.skip("docker daemon unavailable")
result = DockerCliRunner().run(_split_sink_job())
verdict = verdict_from_result(result)
assert verdict.verdict == TRIGGERABLE, (result.exit_code, result.stdout, result.stderr)
+68
View File
@@ -0,0 +1,68 @@
import json
from openultrasast.stages import MODE_STAGES, Stage, StagePlan, plan_for_mode, record_completed, record_skip, stages_payload
def test_mode_stages_maps_existing_depths() -> None:
assert Stage.STATIC == "static"
assert Stage.MAP == "map"
assert Stage.REGRESS == "regress"
assert MODE_STAGES["quick"] == (Stage.STATIC,)
assert MODE_STAGES["standard"] == (Stage.STATIC, Stage.MAP)
assert MODE_STAGES["deep"] == (Stage.STATIC, Stage.MAP, Stage.REGRESS)
def test_quick_never_includes_map_or_regress() -> None:
plan = plan_for_mode("quick")
assert plan.mode == "quick"
assert plan.requested == (Stage.STATIC,)
assert Stage.MAP not in plan.requested
assert Stage.REGRESS not in plan.requested
assert Stage.MAP not in MODE_STAGES["quick"]
assert Stage.REGRESS not in MODE_STAGES["quick"]
def test_plan_for_mode_requests_ordered_stages() -> None:
standard = plan_for_mode("standard")
deep = plan_for_mode("deep")
assert standard.requested == (Stage.STATIC, Stage.MAP)
assert deep.requested == (Stage.STATIC, Stage.MAP, Stage.REGRESS)
assert standard.completed == ()
assert deep.skipped == ()
assert isinstance(standard, StagePlan)
assert isinstance(deep, StagePlan)
def test_skipped_stage_is_structured_degradation_not_exception() -> None:
plan = plan_for_mode("deep")
skipped = record_skip(plan, Stage.REGRESS, "sandbox_unavailable")
assert skipped.requested == (Stage.STATIC, Stage.MAP, Stage.REGRESS)
assert skipped.completed == ()
assert skipped.skipped == ({"stage": "regress", "reason": "sandbox_unavailable"},)
degradations = list(skipped.skipped)
assert degradations[0]["stage"] == "regress"
assert degradations[0]["reason"] == "sandbox_unavailable"
assert json.dumps(degradations) == json.dumps([{"stage": "regress", "reason": "sandbox_unavailable"}])
def test_record_completed_static_for_quick() -> None:
plan = record_completed(plan_for_mode("quick"), Stage.STATIC)
assert plan.requested == (Stage.STATIC,)
assert plan.completed == (Stage.STATIC,)
assert plan.skipped == ()
assert Stage.MAP not in plan.completed
assert Stage.REGRESS not in plan.completed
def test_stages_payload_is_manifest_shaped() -> None:
plan = record_completed(plan_for_mode("quick"), Stage.STATIC)
payload = stages_payload(plan)
assert payload == {
"requested": ["static"],
"completed": ["static"],
"skipped": [],
}
+327
View File
@@ -0,0 +1,327 @@
import json
from pathlib import Path
import pytest
from openultrasast.complexity.map import Hotspot
from openultrasast.tool_hunter import ChatResponse, OpenRouterHunterClient, ScriptedChatClient, ToolCall, run_tool_hunter
SPLIT_SINK = """\
term = request.args["q"]
query = "select * from items where title like '%" + term + "%'"
return db.execute(query)
"""
_DUMP_FINDINGS = [
{
"path": "app.py",
"line": 3,
"title": "SQL injection",
"rationale": "query is built from request input then executed",
"evidence_level": "static_corroboration",
}
]
def _repo_with_split_sink(tmp_path: Path) -> Path:
root = tmp_path / "repo"
root.mkdir()
(root / "app.py").write_text(SPLIT_SINK)
return root
def _hotspot(path: str = "app.py") -> Hotspot:
return Hotspot(
path=path,
function_name="search",
score=7.5,
band="high",
signals={"loc": 3},
rationale="reachable split-sink",
test_hint=None,
inventory_finding_ids=(),
)
def _tool_names(tools: object) -> set[str]:
names: set[str] = set()
if not isinstance(tools, list):
return names
for item in tools:
if not isinstance(item, dict):
continue
function = item.get("function")
if isinstance(function, dict) and isinstance(function.get("name"), str):
names.add(str(function["name"]))
elif isinstance(item.get("name"), str):
names.add(str(item["name"]))
return names
def test_json_dump_without_tool_call_emits_no_findings(tmp_path: Path) -> None:
root = _repo_with_split_sink(tmp_path)
client = ScriptedChatClient([ChatResponse(content=json.dumps(_DUMP_FINDINGS))])
findings = run_tool_hunter(root, [_hotspot()], client=client, model="test-hunter", max_steps=4)
assert client.calls, "the hunter loop must run when a model client is provided"
assert _tool_names(client.calls[0]["tools"]) >= {"read_file", "grep_repo", "find_refs"}
assert findings == []
def test_grep_then_json_yields_suspicion_with_tool_hunter_prefix(tmp_path: Path) -> None:
root = _repo_with_split_sink(tmp_path)
client = ScriptedChatClient(
[
ChatResponse(tool_calls=(ToolCall(id="call_grep", name="grep_repo", arguments={"pattern": r"query\s*=", "max_matches": 10}),)),
ChatResponse(content=json.dumps(_DUMP_FINDINGS)),
]
)
findings = run_tool_hunter(root, [_hotspot()], client=client, model="test-hunter", max_steps=4)
assert len(client.calls) >= 2
assert _tool_names(client.calls[0]["tools"]) >= {"read_file", "grep_repo", "find_refs"}
tool_trace = json.dumps(client.calls[1]["messages"])
assert "query =" in tool_trace
assert len(findings) == 1
assert findings[0].evidence_level == "suspicion"
assert findings[0].finding_id.startswith("tool-hunter:")
assert findings[0].path == "app.py"
assert findings[0].line == 3
assert findings[0].title == "SQL injection"
def test_loop_dispatches_read_file_grep_repo_and_find_refs(tmp_path: Path) -> None:
root = _repo_with_split_sink(tmp_path)
client = ScriptedChatClient(
[
ChatResponse(
tool_calls=(
ToolCall(id="call_read", name="read_file", arguments={"path": "app.py", "max_chars": 4000}),
ToolCall(id="call_grep", name="grep_repo", arguments={"pattern": r"db\.execute", "max_matches": 5}),
ToolCall(id="call_refs", name="find_refs", arguments={"symbol": "query"}),
)
),
ChatResponse(content=json.dumps(_DUMP_FINDINGS)),
]
)
findings = run_tool_hunter(root, [_hotspot()], client=client, model="test-hunter", max_steps=4)
assert findings
assert findings[0].evidence_level == "suspicion"
second_messages = json.dumps(client.calls[1]["messages"])
assert "term = request.args" in second_messages
assert "db.execute" in second_messages
assert "query" in second_messages
def test_path_escape_stays_in_band_and_still_counts_as_tool_use(tmp_path: Path) -> None:
root = _repo_with_split_sink(tmp_path)
client = ScriptedChatClient(
[
ChatResponse(tool_calls=(ToolCall(id="call_escape", name="read_file", arguments={"path": "../etc/passwd"}),)),
ChatResponse(content=json.dumps(_DUMP_FINDINGS)),
]
)
findings = run_tool_hunter(root, [_hotspot()], client=client, model="test-hunter", max_steps=4)
assert findings
assert findings[0].evidence_level == "suspicion"
assert "error" in json.dumps(client.calls[1]["messages"]).lower()
def test_max_steps_stops_before_ungrounded_json(tmp_path: Path) -> None:
root = _repo_with_split_sink(tmp_path)
client = ScriptedChatClient(
[
ChatResponse(tool_calls=(ToolCall(id="call_grep", name="grep_repo", arguments={"pattern": "query"}),)),
ChatResponse(content=json.dumps(_DUMP_FINDINGS)),
]
)
findings = run_tool_hunter(root, [_hotspot()], client=client, model="test-hunter", max_steps=1)
assert findings == []
assert len(client.calls) == 1
_UNSAFE_SNIPPET = 'client = docker.DockerClient(base_url="unix://var/run/docker.sock")\n'
_VULN = "@app.route('/admin')\ndef admin():\n return eval(request.data)\n"
def test_hunter_json_snippet_stays_suspicion_and_is_not_static_corroboration(tmp_path: Path) -> None:
root = _repo_with_split_sink(tmp_path)
payload = [{**_DUMP_FINDINGS[0], "snippet": _UNSAFE_SNIPPET, "evidence_level": "static_corroboration"}]
client = ScriptedChatClient(
[
ChatResponse(tool_calls=(ToolCall(id="call_grep", name="grep_repo", arguments={"pattern": "query"}),)),
ChatResponse(content=json.dumps(payload)),
]
)
findings = run_tool_hunter(root, [_hotspot()], client=client, model="test-hunter", max_steps=4)
assert len(findings) == 1
assert findings[0].evidence_level == "suspicion"
assert findings[0].finding_id.startswith("tool-hunter:")
assert findings[0].proposed_snippet == _UNSAFE_SNIPPET
def _write_hunter_config(tmp_path: Path, body: str = '[models]\nhunter = "test-hunter"\n') -> Path:
config = tmp_path / "openultrasast.toml"
config.write_text(body)
return config
def _standard_repo(tmp_path: Path) -> Path:
repo = tmp_path / "repo"
repo.mkdir()
(repo / "app.py").write_text(_VULN)
return repo
def _latest_run(repo: Path) -> Path:
return sorted((repo / ".runs").iterdir())[-1]
def _guard_docker(monkeypatch: pytest.MonkeyPatch) -> list[list[str]]:
import subprocess
from openultrasast.sandbox import runner as sandbox_runner
docker_argv: list[list[str]] = []
original_run = subprocess.run
def guarded_run(command: object, *args: object, **kwargs: object) -> object:
argv = list(command) if isinstance(command, list | tuple) else [command]
if argv and Path(str(argv[0])).name == "docker":
docker_argv.append([str(item) for item in argv])
raise AssertionError(f"hunter scan test must not start docker: {argv}")
return original_run(command, *args, **kwargs)
def fail(*args: object, **kwargs: object) -> object:
raise AssertionError("hunter scan test must not use DockerCliRunner")
monkeypatch.setattr(subprocess, "run", guarded_run)
monkeypatch.setattr(sandbox_runner.DockerCliRunner, "run", fail)
return docker_argv
def test_standard_scripted_hunter_emits_tool_hunter_suspicion(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
from openultrasast import cli, tool_hunter
from openultrasast.cli import main
repo = _standard_repo(tmp_path)
config = _write_hunter_config(tmp_path)
monkeypatch.setenv("OPENULTRASAST_RUNS_DIR", ".runs")
monkeypatch.setenv("OPENULTRASAST_HUNTER_CLIENT", "scripted")
monkeypatch.setattr(cli, "has_harnessx", lambda: False)
docker_argv = _guard_docker(monkeypatch)
hunter_calls: list[int] = []
real_hunter = tool_hunter.run_tool_hunter
def wrapped(*args: object, **kwargs: object) -> object:
hunter_calls.append(1)
return real_hunter(*args, **kwargs)
monkeypatch.setattr(tool_hunter, "run_tool_hunter", wrapped)
assert main(["scan", str(repo), "--mode", "standard", "--config", str(config)]) == 0
run_dir = _latest_run(repo)
findings = json.loads((run_dir / "findings.json").read_text())["findings"]
complexity_map = json.loads((run_dir / "complexity_map.json").read_text())
manifest = json.loads((run_dir / "manifest.json").read_text())
hunter_findings = [item for item in findings if str(item["finding_id"]).startswith("tool-hunter:")]
assert hunter_calls
assert docker_argv == []
assert hunter_findings
assert all(item["evidence_level"] == "suspicion" for item in hunter_findings)
assert all(item["evidence_level"] != "static_corroboration" for item in hunter_findings)
assert complexity_map["heuristic_only"] is False
assert manifest["complexity"]["heuristic_only"] is False
assert not any(entry.get("reason") == "hunter_model_unavailable" for entry in manifest.get("degradations", []))
def test_dump_only_hunter_client_emits_no_hunter_findings(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
from openultrasast import cli, tool_hunter
from openultrasast.cli import main
repo = _standard_repo(tmp_path)
config = _write_hunter_config(tmp_path)
monkeypatch.setenv("OPENULTRASAST_RUNS_DIR", ".runs")
monkeypatch.setenv("OPENULTRASAST_HUNTER_CLIENT", "dump-only")
monkeypatch.setattr(cli, "has_harnessx", lambda: False)
docker_argv = _guard_docker(monkeypatch)
hunter_calls: list[int] = []
real_hunter = tool_hunter.run_tool_hunter
def wrapped(*args: object, **kwargs: object) -> object:
hunter_calls.append(1)
return real_hunter(*args, **kwargs)
monkeypatch.setattr(tool_hunter, "run_tool_hunter", wrapped)
assert main(["scan", str(repo), "--mode", "standard", "--config", str(config)]) == 0
run_dir = _latest_run(repo)
findings = json.loads((run_dir / "findings.json").read_text())["findings"]
complexity_map = json.loads((run_dir / "complexity_map.json").read_text())
hunter_findings = [item for item in findings if str(item["finding_id"]).startswith("tool-hunter:")]
assert hunter_calls, "dump-only still runs the tool hunter loop"
assert docker_argv == []
assert hunter_findings == []
assert complexity_map["heuristic_only"] is False
def test_resolve_hunter_client_scripted_and_dump_only_env(monkeypatch: pytest.MonkeyPatch) -> None:
from openultrasast import tool_hunter
monkeypatch.delenv("OPENROUTER_API_KEY", raising=False)
monkeypatch.setenv("OPENULTRASAST_HUNTER_CLIENT", "scripted")
scripted = tool_hunter.resolve_hunter_client()
assert scripted is not None
assert type(scripted).__name__ == "ScriptedChatClient"
monkeypatch.setenv("OPENULTRASAST_HUNTER_CLIENT", "dump-only")
dump_only = tool_hunter.resolve_hunter_client()
assert dump_only is not None
assert type(dump_only).__name__ == "ScriptedChatClient"
assert dump_only is not scripted
monkeypatch.delenv("OPENULTRASAST_HUNTER_CLIENT", raising=False)
assert tool_hunter.resolve_hunter_client() is None
monkeypatch.setenv("OPENROUTER_API_KEY", "test-key")
live = tool_hunter.resolve_hunter_client()
assert isinstance(live, OpenRouterHunterClient)
def test_openrouter_hunter_client_maps_tool_calls_without_network() -> None:
class _Inner:
def complete_chat(self, **kwargs: object) -> dict[str, object]:
assert kwargs["model"] == "test-hunter"
assert kwargs["tools"]
return {
"content": None,
"tool_calls": [
{
"id": "call_grep",
"type": "function",
"function": {"name": "grep_repo", "arguments": '{"pattern": "eval", "max_matches": 4}'},
}
],
}
client = OpenRouterHunterClient(_Inner()) # type: ignore[arg-type]
response = client.complete(model="test-hunter", messages=[], tools=[{"type": "function"}], timeout_seconds=5)
assert response.content is None
assert len(response.tool_calls) == 1
assert response.tool_calls[0].name == "grep_repo"
assert response.tool_calls[0].arguments == {"pattern": "eval", "max_matches": 4}
Generated
+1 -1
View File
@@ -2138,7 +2138,7 @@ wheels = [
[[package]] [[package]]
name = "openultrasast" name = "openultrasast"
version = "0.1.0" version = "1.1.0"
source = { editable = "." } source = { editable = "." }
[package.optional-dependencies] [package.optional-dependencies]