mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
32aa92191a | ||
|
|
ac3da41c95 | ||
|
|
9af356797f | ||
|
|
7398063b79 | ||
|
|
fe300e6352 | ||
|
|
818a0b10d3 | ||
|
|
a916e2d173 | ||
|
|
2efbbb8024 | ||
|
|
803312e75b | ||
|
|
66d985e9e6 | ||
|
|
331b62f1c1 | ||
|
|
62c7990e49 | ||
|
|
7a584d4578 | ||
|
|
eb87ef85ec | ||
|
|
f78ea55e5d | ||
|
|
5b7e58ea3a | ||
|
|
b34a75a821 | ||
|
|
90e3b91cf8 | ||
|
|
ad16e22c85 | ||
|
|
8d5599ae65 | ||
|
|
b05fbbc3a5 | ||
|
|
3d643040bd | ||
|
|
c5a80c2817 | ||
|
|
926c8c273e | ||
|
|
723e3a76cf | ||
|
|
6fdd9b3aba |
@@ -26,6 +26,8 @@ jobs:
|
||||
- run: uv run pytest
|
||||
- name: Detection gate (>=90% recall, <10% FP — hard, score-independent)
|
||||
run: uv run python -m openultrasast.gate
|
||||
- name: Split-sink map gate (stage 2 beats pattern-hit ranking)
|
||||
run: uv run python -m openultrasast.map_gate
|
||||
- run: uv run python -m compileall src tests
|
||||
|
||||
dagger:
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
# Brief: three-stage-scan
|
||||
|
||||
## Problem
|
||||
|
||||
AppSec engineers and DevOps teams running OpenUltraSAST in CI get a regex static scan scored against cheat-sheet fixtures. They cannot tell what actually exists in a real repository, where complexity concentrates, how to aim tests, or whether a finding is worth a CI failure. The previous HarnessX spec added governance around those regexes (CWE policy, score, shadow status) without making the hunter evolutionary or the evaluation realistic.
|
||||
|
||||
## Current State
|
||||
|
||||
- `ousast scan --mode quick` is a deterministic regex inventory. It works.
|
||||
- `--mode standard` is labeled a hunter pool but reuses the same regex path unless the optional HarnessX extra *and* a hunter model are both set. That hunter dumps a file into a prompt and scrapes JSON; it has no tools.
|
||||
- `--mode deep` exits with "not implemented." Sandbox config exists on paper (`network=false`, read-only workspace, memory/pids/timeout). No container is ever started.
|
||||
- Benchmarks are one-line sinks written to match the regexes. The fixtures themselves comment the misses (`regex blind spot`). The ≥90% recall gate is circular.
|
||||
- CWE policy, project score, shadow status, traces, and SARIF already exist and should not be redesigned.
|
||||
|
||||
## Desired Outcome
|
||||
|
||||
A scan answers three operator questions, in order:
|
||||
|
||||
1. **What exists?** A static inventory of files, surfaces, and pattern/SARIF hits — honest about being inventory, not verified bugs.
|
||||
2. **Where are problems most likely, and how do I tune tests?** A complexity map of hotspots plus test-gap hints a DevOps team can act on.
|
||||
3. **Is this worth fixing?** Isolated regression against the top hotspots. Triggerable and reachable → worth a CI fail. Not triggerable → do not fail the build.
|
||||
|
||||
Stages 2–3 use a tool-calling hunter (read / grep / callers) and a Docker-isolated runner. Evaluation uses split-sink and mutation corpora, not cheat sheets. Missing Docker or a model degrades visibly; stage 1 still runs.
|
||||
|
||||
## Approach
|
||||
|
||||
**Map-then-prove on the existing modes.** Keep `--mode quick|standard|deep`:
|
||||
|
||||
- `quick` → stage 1 only
|
||||
- `standard` → stages 1 + 2
|
||||
- `deep` → stages 1 + 2 + 3
|
||||
|
||||
Do not require the HarnessX extra for the LLM path. Use the existing stdlib OpenRouter client when a hunter model is configured. HarnessX remains an optional alternative runtime.
|
||||
|
||||
Do not add a Python Docker SDK. Drive the Docker CLI via subprocess with the threat-model flags (no network, read-only source, tmpfs scratch, memory/pids/timeout, cap-drop). Core `dependencies = []` stays.
|
||||
|
||||
Complexity map is a new artifact built from existing ranking, entry points, finding density, nesting/size, and test-file presence — not a rewrite of `RankingScore`. Stage 3 writes a worth-fixing verdict that feeds back into a complexity ledger (evolutionary), without editing detection pattern text.
|
||||
|
||||
## Scope
|
||||
|
||||
- **In**: three-stage scan contract; complexity map + test-tuning hints; Docker-isolated regression and worth-fixing verdicts; tool-calling hunter on hotspots; split-sink/mutation evaluation; CI mapping (PR = stage 1, nightly = stage 2/3); visible degradation; evolutionary feedback from regress → map.
|
||||
- **Out**: rewriting CWE policy or project-score formula; auto-authoring regex patterns; full fuzzing campaigns; patch generation/application; network pentest; importing Clearwing.
|
||||
|
||||
## Boundary Candidates
|
||||
|
||||
- Stage orchestrator (which stages a mode runs, artifacts, degradation)
|
||||
- Complexity map (hotspots, signals, test-gap hints)
|
||||
- Sandbox runner (Docker CLI, capability probe)
|
||||
- Regression / worth-fixing (candidate tests, verdicts, ledger feedback)
|
||||
- Tool-calling hunter (repo-bound tools + model loop for stages 2–3)
|
||||
|
||||
## Out of Boundary
|
||||
|
||||
- PolicyStore / ScoreModel / RulesetStore internals (consume, do not own)
|
||||
- MetaAgent.evolve regex-status loop (adjacent, do not replace)
|
||||
- Deep fuzzing, exploit development, auto-patching
|
||||
- HarnessX packaging / optional extra (may be used, not required)
|
||||
|
||||
## Upstream / Downstream
|
||||
|
||||
- **Upstream**: existing preprocess, rank, mapping, findings, verification, config, CLI modes, OpenRouter client, threat-model sandbox limits.
|
||||
- **Downstream**: CI fail-on `worth-fixing`; later fuzz/patch specs; MCP scan status.
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
|
||||
- **Extends**: `openrouter-sast-harness` gates `standard_security_harness` and `sandboxed_dynamic_harness` (still unsatisfied).
|
||||
- **Adjacent**: `harnessx-self-improving-rulesets` owns CWE policy, score, shadow, optional HarnessX extra. This spec must not reopen those.
|
||||
|
||||
## Constraints
|
||||
|
||||
- Core install stays zero-dependency. Docker CLI and an LLM key are operator-provided runtimes, not Python dependencies.
|
||||
- Scanned code is untrusted. Stage 3 is the first mode allowed to execute target-derived code, and only inside the sandbox.
|
||||
- LLM output starts at evidence `suspicion`. It must not be stamped `static_corroboration`.
|
||||
- Cheat-sheet one-liners may remain as unit fixtures; they must not be the merge gate for stages 2–3.
|
||||
@@ -0,0 +1,526 @@
|
||||
# Design Document
|
||||
|
||||
## Overview
|
||||
|
||||
**Purpose.** This feature turns an OpenUltraSAST scan into three operator questions: what exists (static inventory), where problems are most likely and how to aim tests (complexity map), and whether a finding is worth fixing (isolated regression). It is a follow-up to the HarnessX governance spec, not a replacement of CWE policy, scoring, or rules-as-data.
|
||||
|
||||
**Users.** DevOps engineers get a cheap pull-request inventory and a nightly prove-worth-fixing job. AppSec reviewers get a hotspot map and test-gap hints instead of a regex dump. Maintainers get an evaluation gate that split-sink cases can fail.
|
||||
|
||||
**Impact.** Today `--mode standard` reuses regex findings and `--mode deep` exits. After this spec, `quick` remains stage 1, `standard` builds a complexity map and may run a tool-using hunter, and `deep` runs a Docker-isolated regression that is the only path allowed to execute target-derived code.
|
||||
|
||||
### Goals
|
||||
|
||||
- Map existing `--mode quick|standard|deep` onto stages 1 / 1+2 / 1+2+3 with explicit manifest accounting.
|
||||
- Keep stage 1 zero-dependency, no model, no target execution.
|
||||
- Emit a complexity map and test-tuning hints that do not collapse to pattern-hit count.
|
||||
- Prove worth-fixing inside an isolated sandbox; fail CI on `worth_fixing` only when asked.
|
||||
- Hunter tools follow dataflow; hunter output is `suspicion`.
|
||||
- Stage 2/3 quality is gated on split-sink and mutation corpora.
|
||||
|
||||
### Non-Goals
|
||||
|
||||
- Rewriting PolicyStore, ScoreModel, or detection pattern text.
|
||||
- Requiring the HarnessX extra for any stage.
|
||||
- Full fuzzing, exploit development, or auto-patching.
|
||||
- Importing Clearwing.
|
||||
- A second CLI family (`--stage`).
|
||||
|
||||
## Boundary Commitments
|
||||
|
||||
### This Spec Owns
|
||||
|
||||
- **Stage plan**: which stages a mode runs, skip/degrade accounting, `stages_requested` / `stages_completed` in the manifest.
|
||||
- **Complexity map**: `Hotspot` records, signals, test-gap hints, `complexity_map.json`.
|
||||
- **Complexity ledger**: overlay of hotspot scores from regression verdicts.
|
||||
- **Sandbox**: capability probe and Docker CLI runner that never mounts the host Docker socket.
|
||||
- **Regression / worth-fixing**: candidate selection, language recipes, verdicts, `--fail-on worth-fixing`.
|
||||
- **Tool hunter**: repo-bound read / grep / callers loop for stages 2–3, evidence honesty.
|
||||
- **Split-sink evaluation**: corpora and the "beats pattern-hit ranking" gate.
|
||||
|
||||
### Out of Boundary
|
||||
|
||||
- CWE TSV, `resolve_severity`, project-score formula, rule `enabled|shadow|disabled`.
|
||||
- `MetaAgent.evolve` pattern-status loop.
|
||||
- Semgrep/CodeQL execution (ingest of operator-supplied SARIF remains as today).
|
||||
- Patch generation and repository mutation.
|
||||
- Sanitizer-image matrix and campaign fuzzing.
|
||||
- MCP docker/shell tools, and exposing `deep` on MCP (CLI/CI only).
|
||||
- Trail of Bits `skills.STAGES` (`map`/`hunt`/`verify`/`fix`). That `map` is mapping-discipline skill routing, not the complexity-map scan stage.
|
||||
|
||||
### Allowed Dependencies
|
||||
|
||||
- Existing preprocess, rank, mapping, findings, verification, reports, config, CLI, OpenRouter client, HarnessRuntime events.
|
||||
- Optional HarnessX extra as an alternate hunter host, not a requirement.
|
||||
- Operator-provided Docker CLI and LLM credential.
|
||||
- Stdlib only for new core modules (`subprocess` for Docker, `urllib` already used by OpenRouter).
|
||||
|
||||
### Revalidation Triggers
|
||||
|
||||
- Changing the mode→stage mapping or `--fail-on` vocabulary.
|
||||
- Changing hotspot identity keys (breaks the complexity ledger).
|
||||
- Changing sandbox flags or mounting the Docker socket.
|
||||
- Stamping hunter output above `suspicion`.
|
||||
- Using cheat-sheet recall as the stage 2/3 merge gate.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Existing Architecture Analysis
|
||||
|
||||
| Existing element | Constraint |
|
||||
|---|---|
|
||||
| `cli._run_scan` modes | Reuse names; stop exiting on `deep` |
|
||||
| `quick_scan_findings` | Stage 1 detector; not the stage 2 hunter |
|
||||
| `rank.heuristic_rank` | File budget; not the complexity map |
|
||||
| `mapping.analyze_entry_points` | Input to map signals and caller hints |
|
||||
| `hunter._run_hunter_task` | Must stop being a regex alias when a model is set |
|
||||
| `OpenRouterChatClient` | Default hunter transport |
|
||||
| `SandboxConfig` | Limits already match the threat model; need a runner |
|
||||
| `verify_finding` | Keep as structural review gate; do not treat it as worth-fixing |
|
||||
| `gate.py` 90% recall | Stage 1 only after this spec |
|
||||
|
||||
### Architecture Pattern & Boundary Map
|
||||
|
||||
**Selected pattern.** Ordered pipeline: **inventory drives the map; the map drives regression; regression overlays the map.** Execution never mutates governance data. Target code runs only in the sandbox.
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph S1["STAGE 1 inventory"]
|
||||
PRE[preprocess and mapping]
|
||||
RANK[file rank]
|
||||
INV[pattern and SARIF inventory]
|
||||
end
|
||||
subgraph S2["STAGE 2 map"]
|
||||
SIG[complexity signals]
|
||||
MAP[hotspot map]
|
||||
HINT[test-gap hints]
|
||||
HUNT[tool hunter optional]
|
||||
end
|
||||
subgraph S3["STAGE 3 prove"]
|
||||
CAND[regress candidates]
|
||||
BOX[Docker CLI sandbox]
|
||||
VERD[worth-fixing verdicts]
|
||||
end
|
||||
LED[complexity ledger]
|
||||
CI[CI fail-on]
|
||||
PRE --> RANK --> INV
|
||||
INV --> SIG
|
||||
RANK --> SIG
|
||||
LED --> SIG
|
||||
SIG --> MAP --> HINT
|
||||
MAP --> HUNT
|
||||
HUNT --> MAP
|
||||
MAP --> CAND
|
||||
INV --> CAND
|
||||
CAND --> BOX --> VERD
|
||||
VERD --> LED
|
||||
VERD --> CI
|
||||
INV --> CI
|
||||
```
|
||||
|
||||
**Dependency direction:** config/types → preprocess/rank/mapping/findings → complexity map → hunter tools → sandbox → regress → CLI/report/gate.
|
||||
|
||||
### Technology Stack
|
||||
|
||||
| Layer | Choice | Role | Notes |
|
||||
|-------|--------|------|-------|
|
||||
| CLI | existing `ousast scan --mode` | Stage selection | Add `--fail-on worth-fixing` |
|
||||
| Hunter transport | `OpenRouterChatClient` | Tool loop | Stdlib; HarnessX optional |
|
||||
| Sandbox | Docker CLI via `subprocess` | Isolated execution | No Python docker SDK |
|
||||
| Map / ledger | JSON artifacts | Attention + evolution | Same overlay style as `rule_policy.json` |
|
||||
| Evaluation | split-sink fixtures | Stage 2/3 gate | Smoke fixtures stay stage 1 |
|
||||
|
||||
## File Structure Plan
|
||||
|
||||
### Directory Structure
|
||||
|
||||
```
|
||||
src/openultrasast/
|
||||
├── stages.py # Stage enum, mode map, StagePlan, degradation helpers
|
||||
├── complexity/
|
||||
│ ├── __init__.py
|
||||
│ ├── signals.py # per-file/function signals from targets, findings, tests
|
||||
│ ├── map.py # build_complexity_map, Hotspot, ComplexityMap
|
||||
│ ├── hints.py # test-gap hints, closed test-kind set
|
||||
│ └── ledger.py # load/apply/write complexity_ledger.json
|
||||
├── hunter_tools.py # read_file, grep_repo, find_refs; path clamp
|
||||
├── tool_hunter.py # bounded tool loop over a chat client
|
||||
├── sandbox/
|
||||
│ ├── __init__.py
|
||||
│ ├── probe.py # docker CLI present and usable
|
||||
│ └── runner.py # DockerCliRunner, SandboxRunner protocol, FakeSandboxRunner
|
||||
└── regress/
|
||||
├── __init__.py
|
||||
├── candidate.py # select and render candidates
|
||||
├── recipes.py # language command recipes
|
||||
└── verdict.py # run candidates, worth-fixing gate
|
||||
|
||||
benchmarks/fixtures/
|
||||
├── split-sink-python/
|
||||
├── split-sink-javascript/
|
||||
├── split-sink-java/
|
||||
└── split-sink-c/
|
||||
benchmarks/manifests/split-sink-*.toml
|
||||
```
|
||||
|
||||
### Modified Files
|
||||
|
||||
- `cli.py` — run stage 2/3; stop exiting on `deep`; `--fail-on worth-fixing`; write new artifacts.
|
||||
- `reports.py` — `scan_exit_code` grows `worth-fixing`; report sections for inventory, map, and worth-fixing.
|
||||
- `config.py` — `ComplexityConfig` (top_k, max_hunter_hotspots) and `RegressConfig` (max_candidates, image pins); **consume** existing `SandboxConfig` (do not add a second sandbox TOML).
|
||||
- `hunter.py` — schedule remains; execution delegates to `tool_hunter` when a model is set, else inventory-only.
|
||||
- `verification.py` / reports — inventory vs worth-fixing labeling; hunter findings stay `suspicion`.
|
||||
- `gate.py` — stage-1 smoke gate unchanged; add split-sink map gate.
|
||||
- `reports.py` — map and verdict sections in markdown/manifest.
|
||||
- `.github/workflows/ci.yml` — PR stays quick; optional nightly deep when Docker is available.
|
||||
|
||||
## System Flows
|
||||
|
||||
### Mode to stages
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Q[quick] --> A[stage 1]
|
||||
S[standard] --> A
|
||||
S --> B[stage 2]
|
||||
D[deep] --> A
|
||||
D --> B
|
||||
D --> C[stage 3]
|
||||
```
|
||||
|
||||
If stage 2 has no model, the heuristic map still writes. If stage 3 has no Docker, stages 1–2 remain and `sandbox_unavailable` is recorded.
|
||||
|
||||
### Stage 3 worth-fixing
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant M as ComplexityMap
|
||||
participant R as RegressionRunner
|
||||
participant S as DockerCliRunner
|
||||
participant G as WorthFixingGate
|
||||
M->>R: top hotspots plus severe inventory
|
||||
R->>R: render snippet and safety check
|
||||
alt safety check fails
|
||||
R->>G: inconclusive
|
||||
else sandbox missing
|
||||
R->>G: inconclusive sandbox_unavailable
|
||||
else recipe missing
|
||||
R->>G: inconclusive
|
||||
else runs
|
||||
R->>S: docker run read-only source
|
||||
S->>R: exit code stdout stderr
|
||||
R->>G: triggerable or not_triggerable or already_covered
|
||||
end
|
||||
G->>M: ledger overlay
|
||||
```
|
||||
|
||||
## Requirements Traceability
|
||||
|
||||
| Requirement | Summary | Components | Flows |
|
||||
|-------------|---------|------------|-------|
|
||||
| 1.1–1.6 | Mode maps to stages; manifest accounting; degrade forward | `stages.py`, `cli.py` | Mode to stages |
|
||||
| 2.1–2.5 | Stage 1 inventory, no execute, no model | existing preprocess/rank/findings | Stage 1 |
|
||||
| 3.1–3.5 | Complexity map independent of regex count | `complexity/map.py`, `signals.py` | Stage 2 |
|
||||
| 4.1–4.4 | Test-gap hints, no repo writes | `complexity/hints.py` | Stage 2 |
|
||||
| 5.1–5.8 | Sandbox regression, worth-fixing, fail-on | `sandbox/*`, `regress/*` | Stage 3 |
|
||||
| 6.1–6.6 | Tool hunter, path clamp, suspicion | `hunter_tools.py`, `tool_hunter.py` | Stage 2/3 |
|
||||
| 7.1–7.5 | Ledger overlay, cannot hide sev-5 reachable | `complexity/ledger.py` | Stage 3 → map |
|
||||
| 8.1–8.6 | Split-sink / mutation gate | fixtures, `gate.py` | Evaluation |
|
||||
| 9.1–9.6 | PR cheap, nightly bounded, visible skip | `cli.py`, CI workflow | Degradation |
|
||||
|
||||
## Components and Interfaces
|
||||
|
||||
| Component | Domain | Intent | Req Coverage | Dependencies | Contracts |
|
||||
|-----------|--------|--------|--------------|--------------|-----------|
|
||||
| StagePlan | Orchestration | Mode → stages, skip reasons | 1.1–1.6, 9.3–9.6 | config | State |
|
||||
| ComplexityMapBuilder | Map | Hotspots from signals + ledger | 3.1–3.5, 7.1–7.3 | preprocess, rank, findings | Batch, State |
|
||||
| TestTuningAdvisor | Map | Test-gap hints | 4.1–4.4 | ComplexityMap | Batch |
|
||||
| HunterTools | Hunter | Read/grep/refs inside repo | 6.1–6.2 | mapping | Service |
|
||||
| ToolHunter | Hunter | Bounded tool loop | 6.1–6.5 | OpenRouter, HunterTools | Service |
|
||||
| SandboxProbe | Sandbox | Docker usable? | 5.2, 9.3 | docker CLI | Service |
|
||||
| DockerCliRunner | Sandbox | Isolated `docker run` | 5.2, 5.8 | docker CLI | Service |
|
||||
| RegressionRunner | Regress | Candidates, recipes, verdicts | 5.1–5.6 | Sandbox, map | Batch |
|
||||
| WorthFixingGate | Regress | CI fail-on | 5.4–5.7, 7.4 | RegressionRunner, policy | Batch |
|
||||
|
||||
### Orchestration
|
||||
|
||||
#### StagePlan
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Declare which stages run and why a stage was skipped |
|
||||
| Requirements | 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 9.3, 9.4, 9.6 |
|
||||
|
||||
```python
|
||||
from enum import StrEnum
|
||||
|
||||
class Stage(StrEnum):
|
||||
STATIC = "static"
|
||||
MAP = "map"
|
||||
REGRESS = "regress"
|
||||
|
||||
MODE_STAGES = {
|
||||
"quick": (Stage.STATIC,),
|
||||
"standard": (Stage.STATIC, Stage.MAP),
|
||||
"deep": (Stage.STATIC, Stage.MAP, Stage.REGRESS),
|
||||
}
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class StagePlan:
|
||||
mode: str
|
||||
requested: tuple[Stage, ...]
|
||||
completed: tuple[Stage, ...]
|
||||
skipped: tuple[dict[str, str], ...] # stage, reason
|
||||
```
|
||||
|
||||
- **Invariants:** `quick` never includes `MAP` or `REGRESS`. Skipped stages appear in `runtime.state["degradations"]` with the same shape as existing capability fallbacks.
|
||||
- **Naming:** `Stage.MAP` is the complexity-map scan stage. It is unrelated to `skills.STAGES` value `"map"` (Semgrep/CodeQL skill routing). Do not merge the two enumerations.
|
||||
|
||||
### Complexity map
|
||||
|
||||
#### ComplexityMapBuilder
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Rank likely problem areas without using pattern-hit count as the score |
|
||||
| Requirements | 3.1, 3.2, 3.3, 3.4, 3.5 |
|
||||
|
||||
**Signals** (stdlib, no extra libs):
|
||||
|
||||
- size: loc
|
||||
- nesting: max indent or brace depth
|
||||
- surface tags from `FileTarget`
|
||||
- reachability from entry-point hints
|
||||
- inventory density as a *feature*, not the score (used in rationale, down-weighted in the formula)
|
||||
- test presence: sibling `test_*` / `*_test` / `*Test.java` / `*_test.go` under common test roots
|
||||
- ledger overlay last
|
||||
|
||||
**Hotspot identity:** `path` + `function_name or ""`. Function name from reachability hints or a language-specific def/func regex; empty if unknown. Java/Groovy mapping is tag-only today, so identity is usually the file; that is acceptable for ranking and test-gap hints.
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class Hotspot:
|
||||
path: str
|
||||
function_name: str | None
|
||||
score: float
|
||||
band: str # high | medium | low
|
||||
signals: dict[str, float | int | bool | str]
|
||||
rationale: str
|
||||
test_hint: TestHint | None
|
||||
inventory_finding_ids: tuple[str, ...]
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ComplexityMap:
|
||||
hotspots: tuple[Hotspot, ...]
|
||||
heuristic_only: bool
|
||||
```
|
||||
|
||||
- **Invariant:** sorting files by `len(inventory_finding_ids)` descending must not be identical to map order on the split-sink corpus (enforced by the gate).
|
||||
- **Postcondition:** write `.openultrasast/runs/<id>/complexity_map.json`.
|
||||
|
||||
#### TestTuningAdvisor
|
||||
|
||||
Closed `test_kind` set: `unit | property | sanitizer | http-contract | fuzz-harness`.
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class TestHint:
|
||||
path: str
|
||||
function_name: str | None
|
||||
gap: str # no_adjacent_test | no_function_reference | no_fuzz_entry | covered
|
||||
test_kind: str | None
|
||||
reason: str
|
||||
```
|
||||
|
||||
- **Invariant:** `gap == "covered"` ⇒ `test_kind is None`. Never writes into the scanned tree.
|
||||
|
||||
#### Complexity ledger
|
||||
|
||||
Path: `<repo>/.openultrasast/calibration/complexity_ledger.json`.
|
||||
|
||||
```json
|
||||
{
|
||||
"parser.c:parse_header": {
|
||||
"score_delta": 1.5,
|
||||
"last_verdict": "triggerable",
|
||||
"round": 3
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Apply at map build. `score_delta` cannot drop a reachable policy-severity-5 inventory finding out of the stage-3 candidate set.
|
||||
|
||||
### Hunter
|
||||
|
||||
#### HunterTools
|
||||
|
||||
```python
|
||||
def clamp_repo_path(root: Path, user_path: str) -> Path: ...
|
||||
def read_file(root: Path, path: str, *, max_chars: int) -> str: ...
|
||||
def grep_repo(root: Path, pattern: str, *, max_matches: int) -> list[dict[str, object]]: ...
|
||||
def find_refs(root: Path, symbol: str, mapping_index: object) -> list[dict[str, object]]: ...
|
||||
```
|
||||
|
||||
- **Invariant:** any resolved path whose common prefix is not `root` raises `PathEscapesRepo`.
|
||||
- Grep is Python `re` over enumerated source files (zero-dep). Ripgrep is an optional fast path if present, still clamped.
|
||||
|
||||
#### ToolHunter
|
||||
|
||||
Bounded loop: max steps from the hotspot's hunter budget (reuse tier A/B/C caps). Each model turn may call the three tools or stop with a JSON list of suspicions.
|
||||
|
||||
```python
|
||||
def run_tool_hunter(
|
||||
root: Path,
|
||||
hotspots: list[Hotspot],
|
||||
*,
|
||||
client: ChatClient,
|
||||
model: str,
|
||||
max_steps: int,
|
||||
) -> list[StaticFinding]: ...
|
||||
```
|
||||
|
||||
- **Postconditions:** each finding `evidence_level == "suspicion"`; `finding_id` prefix `tool-hunter:`.
|
||||
- **Fallback:** no model → empty hunter findings, `heuristic_only=True` on the map, degradation `hunter_model_unavailable`.
|
||||
|
||||
### Sandbox and regression
|
||||
|
||||
#### SandboxRunner
|
||||
|
||||
```python
|
||||
class SandboxRunner(Protocol):
|
||||
def run(self, spec: SandboxJob) -> SandboxResult: ...
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SandboxJob:
|
||||
image: str
|
||||
command: tuple[str, ...]
|
||||
repo_root: Path
|
||||
scratch_files: dict[str, str] # relative to /scratch
|
||||
timeout_seconds: int
|
||||
memory_mb: int
|
||||
pids_limit: int
|
||||
```
|
||||
|
||||
`DockerCliRunner` argv (conceptual): `docker run --rm --init --network none --read-only --tmpfs /scratch:rw,exec --memory --pids-limit --cap-drop ALL --security-opt no-new-privileges --user` non-root `--mount type=bind,src=repo,dst=/workspace,ro` image command.
|
||||
|
||||
- **Forbidden:** `-v /var/run/docker.sock`, `--network host`, `--privileged`, writable bind of the source tree.
|
||||
- `SandboxProbe.available()` runs `docker info` with a short timeout.
|
||||
|
||||
#### RegressionRunner
|
||||
|
||||
Select `min(max_candidates, top_k_hotspots ∪ sev5_reachable_inventory)`. Render a recipe snippet into `scratch_files`. Safety check rejects snippets that mention `socket`, `curl`, `docker.sock`, or writes under `/workspace`.
|
||||
|
||||
Verdicts:
|
||||
|
||||
| Observation | Verdict |
|
||||
|---|---|
|
||||
| recipe or sandbox missing, timeout, safety reject | `inconclusive` |
|
||||
| existing project test covering the hotspot already passes in sandbox | `already_covered` |
|
||||
| candidate exits non-zero with a crash, assertion, or sanitizer abort | `triggerable` |
|
||||
| candidate exits 0 | `not_triggerable` |
|
||||
|
||||
`worth_fixing` ⇔ `triggerable` and related finding reachability is `reachable` or `inferred-file-surface`.
|
||||
|
||||
`--fail-on` gains `worth-fixing`. Existing `findings` / `verified` unchanged.
|
||||
|
||||
Recipes MVP: Python (`python /scratch/case.py` with `PYTHONPATH=/workspace`), JavaScript (`node /scratch/case.js`), C (`cc` + run) when the image contains a compiler. Java recipe may return `inconclusive` until added.
|
||||
|
||||
## Data Models
|
||||
|
||||
### Manifest additions
|
||||
|
||||
```json
|
||||
{
|
||||
"stages": {
|
||||
"requested": ["static", "map", "regress"],
|
||||
"completed": ["static", "map"],
|
||||
"skipped": [{"stage": "regress", "reason": "sandbox_unavailable"}]
|
||||
},
|
||||
"complexity": {"hotspot_count": 12, "heuristic_only": true},
|
||||
"worth_fixing": {"count": 0, "verdicts": []}
|
||||
}
|
||||
```
|
||||
|
||||
Degradations reuse the existing list: `{ "stage": "regress", "reason": "sandbox_unavailable" }`.
|
||||
|
||||
### Consistency
|
||||
|
||||
- Map artifact and ledger are JSON, resume-safe.
|
||||
- Hunter suspicions never jump evidence levels in this spec.
|
||||
- Stage 1 artifacts remain byte-compatible with current `findings.json` schema; new fields are additive.
|
||||
|
||||
## Error Handling
|
||||
|
||||
| Error | Response |
|
||||
|---|---|
|
||||
| `deep` requested, Docker missing | Complete 1–2, skip 3, degradation, exit 0 unless other fail-on hits |
|
||||
| Model missing on `standard` | Heuristic map, no hunter findings, degradation |
|
||||
| Path escape in a tool | Tool error to the model; do not throw the scan |
|
||||
| Snippet fails safety check | `inconclusive`, do not run |
|
||||
| Docker run timeout | kill container (`--rm`), `inconclusive` |
|
||||
| Ledger would hide sev-5 reachable | Candidate still selected |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit
|
||||
|
||||
- Mode→stage mapping and skip accounting.
|
||||
- Path clamp rejects `../`.
|
||||
- Map order on a fixture where regex hits a trivial helper and the real bug is a split-sink parser — parser ranks higher.
|
||||
- Covered hotspot yields `gap=covered` and no test kind.
|
||||
- Safety check rejects docker.sock and network snippets.
|
||||
- Ledger boost/demote; sev-5 reachable remains a candidate.
|
||||
|
||||
### Integration
|
||||
|
||||
- Fake sandbox: `triggerable` / `not_triggerable` / timeout mapped to verdicts.
|
||||
- Tool hunter with a scripted chat client that must call `grep_repo` before concluding on a split-sink file; fail if it only returns JSON with no tool call.
|
||||
- `--fail-on worth-fixing` exits non-zero only when a `worth_fixing` verdict exists.
|
||||
|
||||
### Evaluation / CI
|
||||
|
||||
- Stage 1: existing `python -m openultrasast.gate` on smoke corpora.
|
||||
- Stage 2: split-sink manifests; true vulnerable function in the high band; map order ≠ pattern-hit order.
|
||||
- Stage 3: `@pytest.mark.docker`; skip if probe fails; at least one planted split-sink that stage 1 missed is `triggerable`.
|
||||
- PR workflow stays `quick`. Nightly/deep is a separate job with Docker.
|
||||
|
||||
### Anti-patterns this spec forbids in tests
|
||||
|
||||
- One-line `eval(request.args)` as the only stage-2 proof.
|
||||
- Asserting hunter findings equal regex findings.
|
||||
- Calling the real Docker daemon in the default `pytest` job.
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- Scanned code is untrusted. Stage 1–2 read it as text. Stage 3 executes only in Docker with network off, source read-only, no docker.sock, dropped caps, no-new-privileges, non-root user.
|
||||
- Hunter tools cannot read outside the repo.
|
||||
- Regression snippets are data, not host shell. They go to `/scratch` in the container.
|
||||
- Secrets redaction already applies to traces and markdown; keep it on for map rationales that quote source.
|
||||
|
||||
## Performance & Scalability
|
||||
|
||||
- Map is O(files) plus one pass of findings. No extra model required.
|
||||
- Tool hunter runs only on `max_hunter_hotspots` (default 8).
|
||||
- Stage 3 `max_candidates` default 5; timeout from `SandboxConfig.timeout_seconds`.
|
||||
- Large-repo CI: PR = stage 1; nightly = stage 2 always, stage 3 if probe passes.
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
P0[StagePlan plus manifest] --> P1[Heuristic map and hints]
|
||||
P1 --> P2[Split-sink gate]
|
||||
P2 --> P3[Tool hunter]
|
||||
P3 --> P4[Docker runner plus recipes]
|
||||
P4 --> P5[Ledger and fail-on worth-fixing]
|
||||
```
|
||||
|
||||
| Phase | Ships | Merge gate | Rollback |
|
||||
|---|---|---|---|
|
||||
| 0 StagePlan | `deep` no longer exits; skips 2/3 with degradations | existing tests | revert CLI |
|
||||
| 1 Heuristic map | `complexity_map.json` on `standard` | unit map tests | feature: empty map writer |
|
||||
| 2 Split-sink gate | new fixtures; map ≠ hit-count | new gate job | gate off |
|
||||
| 3 Tool hunter | OpenRouter loop, suspicion | scripted-client test | model unset |
|
||||
| 4 Sandbox | Docker CLI + Python/JS recipes | docker-marked tests skip-ok | probe skip |
|
||||
| 5 Worth-fixing | ledger + fail-on | integration on fake runner | flag unused |
|
||||
|
||||
`deep` changing from hard-exit to skip-or-run is intentional and documented. No artifact schema is removed.
|
||||
@@ -0,0 +1,129 @@
|
||||
# Requirements Document
|
||||
|
||||
## Introduction
|
||||
|
||||
OpenUltraSAST today answers one question well: which regexes fired. It does not answer the three questions a rugged DevOps CI workflow actually asks: what exists in this repository, where problems are most likely and how to aim tests, and whether a finding is worth fixing. This feature turns the scan into those three stages. Stage 1 is a static inventory of what exists. Stage 2 is a complexity map of likely problem areas plus test-tuning hints. Stage 3 is isolated regression that proves triggerability before CI treats a finding as blocking.
|
||||
|
||||
The work is a follow-up to the HarnessX ruleset-governance spec, not a replacement of it. CWE policy, project score, and rule-status shadowing stay in place. This spec owns the stage contract, the complexity map, the isolated regression verdict, the tool-using hunter on stages 2–3, and an evaluation corpus that is not a cheat sheet.
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: a three-stage scan contract mapped onto existing scan depths; a static inventory that reports what exists without claiming verification; a complexity map of hotspots with test-tuning hints; isolated regression that produces a worth-fixing verdict; a tool-using hunter confined to stages 2–3; evaluation against split-sink and mutation corpora; visible degradation when the sandbox or a model is unavailable; evolutionary feedback from regression verdicts back into the complexity map.
|
||||
- **Out of scope**: rewriting CWE-to-severity policy or the project-score formula; authoring or rewriting detection pattern text; full fuzzing campaigns; generating or applying patches; network pentesting; importing Clearwing.
|
||||
- **Adjacent expectations**: stage 1 continues to run in the zero-dependency default install with no model and no sandbox. CWE policy, scoring, traces, and SARIF remain owned by the existing governance spec. The operator supplies an isolated-sandbox runtime (Docker CLI) for stage 3 and an LLM credential for the tool-using hunter; neither is a packaged library dependency of the core install.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement 1: Three-Stage Scan Contract
|
||||
|
||||
**Objective:** As a DevOps engineer, I want a scan to run as three ordered stages that map onto the existing scan depths, so that CI can choose inventory-only, map-and-tune, or prove-worth-fixing without a new command family.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When a scan runs at depth `quick`, the Scan Orchestrator shall execute stage 1 only and shall not start a model call or execute target-derived code.
|
||||
2. When a scan runs at depth `standard`, the Scan Orchestrator shall execute stage 1 then stage 2.
|
||||
3. When a scan runs at depth `deep`, the Scan Orchestrator shall execute stage 1 then stage 2 then stage 3.
|
||||
4. The Scan Orchestrator shall record the stages requested, the stages completed, and any skipped stage with a reason in the scan manifest.
|
||||
5. If a later stage cannot run, the Scan Orchestrator shall complete earlier stages, record a degradation, and shall not discard those earlier artifacts.
|
||||
6. The Scan Orchestrator shall keep the existing scan command and depth names so existing CI invocations continue to work.
|
||||
|
||||
### Requirement 2: Stage 1 Static Inventory
|
||||
|
||||
**Objective:** As a security reviewer, I want a static inventory of what exists in the repository, so that I can see files, surfaces, and pattern hits without treating them as verified vulnerabilities.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When stage 1 completes, the Static Inventory shall emit a repository snapshot, file targets with language and surface tags, entry-point reachability hints, ranked files, and static findings from the existing pattern ruleset and any provided static-analyzer input.
|
||||
2. The Static Inventory shall label static pattern hits as inventory, not as verified findings, in the report and in the finding evidence level.
|
||||
3. While no static-analyzer input is provided, the Static Inventory shall still complete using the built-in pattern ruleset and heuristics.
|
||||
4. If an enabled rule's CWE does not resolve in policy, the Scan Runtime shall fail loud before producing inventory findings (existing startup invariant, preserved).
|
||||
5. The Static Inventory shall not execute target-derived code and shall not require a model.
|
||||
|
||||
### Requirement 3: Stage 2 Complexity Map
|
||||
|
||||
**Objective:** As an AppSec engineer, I want a complexity map of where problems are most likely, so that review and testing attention go to real hotspots instead of every regex hit.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When stage 2 runs, the Complexity Map shall produce an ordered list of hotspots, each identifying a file and, when known, a function, with a complexity score and a rationale built from size, nesting, surface tags, reachability, existing inventory-hit density, and test-presence gaps.
|
||||
2. The Complexity Map shall rank hotspots independently of a single regex match: a file with no pattern hit may still be a hotspot, and a pattern hit in a trivial helper may rank below a complex reachable parser.
|
||||
3. When stage 2 completes, the Complexity Map shall write a map artifact that the report and later stages consume, including the scoring signals used for each hotspot.
|
||||
4. While a hunter model is unavailable, the Complexity Map shall still produce the heuristic map and shall record that the hunter did not refine it.
|
||||
5. The Complexity Map shall not mark a hotspot as verified or worth fixing.
|
||||
|
||||
### Requirement 4: Test-Tuning Hints
|
||||
|
||||
**Objective:** As a developer owning CI tests, I want the complexity map to tell me how to tune tests, so that I add coverage where it reduces real risk instead of mirroring cheat-sheet sinks.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When the Complexity Map emits a hotspot, the Test Tuning Advisor shall attach a test-gap hint that names the missing coverage (no adjacent test file, no test referencing the function, or no sanitizer/fuzz entry for a fuzzable surface).
|
||||
2. The Test Tuning Advisor shall recommend a test kind from a closed set: unit, property, sanitizer, http-contract, or fuzz-harness, with a one-line reason tied to the hotspot's signals.
|
||||
3. The Test Tuning Advisor shall not write test files into the scanned repository.
|
||||
4. When a hotspot already has covering tests, the Test Tuning Advisor shall say so and shall not recommend a duplicate test kind.
|
||||
|
||||
### Requirement 5: Stage 3 Isolated Regression and Worth-Fixing
|
||||
|
||||
**Objective:** As a CI owner, I want isolated regression that answers whether a finding is worth fixing, so that the build fails only on triggerable, reachable issues.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When stage 3 runs, the Regression Runner shall select a bounded set of top hotspots and/or inventory findings and shall attempt a minimal regression or proof-of-concept inside an isolated analysis sandbox.
|
||||
2. The isolated analysis sandbox shall run with network disabled, the source tree mounted read-only, a writable scratch area, and configured memory, process, and timeout limits.
|
||||
3. The Regression Runner shall emit one of four verdicts per candidate: `triggerable`, `not_triggerable`, `already_covered`, or `inconclusive`.
|
||||
4. When a candidate is `triggerable` and the related finding is reachable, the Worth-Fixing Gate shall mark it `worth_fixing`.
|
||||
5. When a candidate is `not_triggerable` or `already_covered`, the Worth-Fixing Gate shall mark it `not_worth_fixing` and shall not fail the build for that candidate.
|
||||
6. When the verdict is `inconclusive`, the Worth-Fixing Gate shall not fail the build for that candidate and shall record why (timeout, sandbox unavailable, no candidate generated, or setup failed).
|
||||
7. Where `--fail-on worth-fixing` is selected, the Scan Orchestrator shall fail the scan if and only if at least one `worth_fixing` verdict was produced.
|
||||
8. The Regression Runner shall never execute target-derived code on the host outside the isolated sandbox.
|
||||
|
||||
### Requirement 6: Tool-Using Hunter on Stages 2 and 3
|
||||
|
||||
**Objective:** As an OpenUltraSAST maintainer, I want the hunter on later stages to use repository tools instead of dumping a file into a prompt, so that it can follow dataflow across lines and files the way a real review does.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. Where a hunter model is configured, the Hunter Pool shall run a bounded tool-using loop against stage-2 hotspots, with tools limited to reading a file inside the repository, searching repository text, and listing callers or references already known to the scan.
|
||||
2. The Hunter Pool shall confine every tool path to the scanned repository root and shall reject path arguments that escape that root.
|
||||
3. When the hunter reports a candidate issue, the Scan Runtime shall record it at evidence level `suspicion` until independent static or regression evidence raises it.
|
||||
4. While no hunter model is configured, the Hunter Pool shall skip the tool loop, leave the heuristic complexity map in place, and record a degradation.
|
||||
5. The Hunter Pool shall not treat a model JSON list as static corroboration.
|
||||
6. When stage 3 runs and a hunter model is configured, the Hunter Pool may propose a regression snippet, but the Regression Runner shall execute that snippet only inside the isolated sandbox after a structural safety check (no network, no writes outside scratch).
|
||||
|
||||
### Requirement 7: Evolutionary Feedback from Regression to Map
|
||||
|
||||
**Objective:** As a maintainer, I want regression verdicts to retune the complexity map, so that the scan gets better at spending attention where issues actually trigger.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When a hotspot's regression is `triggerable`, the Complexity Map shall raise that hotspot's subsequent score on the same repository.
|
||||
2. When a hotspot's regression is `not_triggerable`, the Complexity Map shall lower that hotspot's subsequent score without deleting the underlying inventory hit.
|
||||
3. The Scan Runtime shall persist those adjustments in a loop-owned ledger keyed by repository and hotspot identity, applied at map load the same way rule status overlays apply at ruleset load.
|
||||
4. If a ledger entry would hide a reachable severity-5 inventory hit from the worth-fixing set, the Worth-Fixing Gate shall still consider that hit.
|
||||
5. The evolutionary loop shall not edit detection pattern text and shall not edit authoritative CWE severity.
|
||||
|
||||
### Requirement 8: Grounded Evaluation, Not Cheat Sheets
|
||||
|
||||
**Objective:** As a maintainer, I want stage 2 and stage 3 quality gates to use split-sink and mutation cases, so that a regex that only matches textbook one-liners cannot claim success.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The Benchmark Harness shall include at least one split-sink case per in-scope language family (Python, JavaScript, Java, C/C++) where the source and the sink are on different lines or the payload is assigned before use.
|
||||
2. The Benchmark Harness shall include at least one mutation of a known sink (equivalent API, intermediate variable, or wrapper) that a same-line pattern does not match.
|
||||
3. When stage 1 is scored alone, the Benchmark Harness may use existing smoke fixtures, and shall not treat stage-1 recall as proof of stage-2 or stage-3 quality.
|
||||
4. When stage 2 is scored, the Benchmark Harness shall require the complexity map to rank the true vulnerable function in the top band of hotspots on the split-sink corpus, even when stage 1 missed the sink.
|
||||
5. When stage 3 is scored and the sandbox is available, the Benchmark Harness shall require a `triggerable` verdict on at least one planted split-sink that stage 1 missed.
|
||||
6. The CI merge gate for this feature shall fail if stage 2 ranking on the split-sink corpus is no better than ranking files by pattern-hit count alone.
|
||||
|
||||
### Requirement 9: CI Companion Budgets and Degradation
|
||||
|
||||
**Objective:** As a CI owner, I want stage costs and failures to be bounded and visible, so that pull-request jobs stay cheap and nightly jobs can run the later stages without silent fallback.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. While running in pull-request depth `quick`, the CI Pipeline shall complete stage 1 without a sandbox runtime and without a model.
|
||||
2. Where a nightly or labeled job requests `standard` or `deep`, the CI Pipeline shall run those stages under the configured time, finding-count, and cost budgets.
|
||||
3. If the sandbox runtime is missing when stage 3 is requested, the Scan Orchestrator shall skip stage 3, record `sandbox_unavailable`, and complete stages 1–2.
|
||||
4. If a model is missing when stage 2 hunter refinement is requested, the Scan Orchestrator shall keep the heuristic map, record `hunter_model_unavailable`, and shall not invent hunter findings.
|
||||
5. The Scan Orchestrator shall bound stage 3 to a configured maximum number of regression candidates per scan.
|
||||
6. When any stage is skipped or truncated, the report shall show the degradation in the same manifest list used by existing capability fallbacks.
|
||||
@@ -0,0 +1,156 @@
|
||||
# Research & Design Decisions
|
||||
|
||||
---
|
||||
**Purpose**: Capture discovery findings and architectural investigation behind the `three-stage-scan` spec. `design.md` carries the committed architecture; this file carries evidence and rejected alternatives.
|
||||
---
|
||||
|
||||
## Summary
|
||||
- **Feature**: `three-stage-scan`
|
||||
- **Discovery Scope**: Extension of the existing scan harness (brownfield), with one new runtime prerequisite (Docker CLI for stage 3) and one new evaluation contract (split-sink / mutation corpora).
|
||||
- **Key Findings**:
|
||||
- Stage 1 already exists as `quick` mode: preprocess, mapping, rank, `quick_scan_findings`, structural verify, score, SARIF. Do not rebuild it.
|
||||
- Stage 2 is not a hunter dump and is not `RankingScore`. Ranking is per-file attention. A complexity map is per-file-and-function hotspot attention plus test-gap hints. Existing signals (tags, loc, reachability hints, finding density, `has_fuzz_entry_point`) are enough for a heuristic map; the tool-using hunter refines it.
|
||||
- Stage 3 has config and a threat-model table, but `cli.py` exits on `--mode deep`. No container is ever started. The first execution of untrusted code must be Docker CLI with the documented isolation flags, not a Python SDK and not host `subprocess` of target code.
|
||||
- The previous HarnessX spec locked LLM use behind an optional extra that CI does not install. This spec uses the existing stdlib OpenRouter client so stage 2 can be evolutionary without that extra. HarnessX remains an optional runtime, not a prerequisite.
|
||||
- Cheat-sheet fixtures cannot gate stages 2–3. The Python fixture already documents a split-sink miss (`query` built on the prior line). That class of case becomes the quality bar.
|
||||
|
||||
## Research Log
|
||||
|
||||
### Existing scan pipeline and mode mapping
|
||||
- **Context**: Whether to add `--stage` or reuse `--mode quick|standard|deep`.
|
||||
- **Sources Consulted**: `src/openultrasast/cli.py` (`_run_scan`, mode choices), `README.md` maturity legend, `.kiro/specs/openrouter-sast-harness/requirements.md` (hunter pool, docker sandbox, dynamic analysis), `.kiro/specs/harnessx-self-improving-rulesets/design.md`.
|
||||
- **Findings**:
|
||||
- `quick` is implemented and deterministic. `standard` schedules a hunter pool that calls `quick_scan_findings` unless HarnessX extra + hunter model are set. `deep` raises `SystemExit`.
|
||||
- OpenRouter spec already defined: quick = static + bounded LLM review; standard = sandboxed tool-calling hunters; deep = harness/fuzz/dynamic. Implementation never reached that.
|
||||
- Existing CI and README teach `--mode`. A parallel `--stage` flag would fork the contract.
|
||||
- **Implications**: Map modes onto stages 1 / 1+2 / 1+2+3. Record `stages_requested` and `stages_completed` in the manifest so the three questions are explicit even if the flag names stay.
|
||||
|
||||
### Ranking vs complexity map
|
||||
- **Context**: `rank.py` already scores files. Is stage 2 just ranking?
|
||||
- **Sources Consulted**: `src/openultrasast/rank.py` (`heuristic_rank`, `composite_priority` = surface 0.5 + influence 0.2 + reachability 0.3), `preprocess.py` (`FileTarget` tags, loc, `has_fuzz_entry_point`), `mapping.py` (`analyze_entry_points`).
|
||||
- **Findings**:
|
||||
- Ranking is file-level and exists to budget the hunter. It does not name functions, test gaps, or "tune your tests."
|
||||
- Tags already encode likely problem classes (`memory_unsafe`, `parser`, `network_entry`, `syscall_entry`). Loc ≥ 500 boosts influence. No nesting/branch signal. No test-file presence.
|
||||
- A hotspot that stage 1 missed (split-sink) can still rank high if the file is a reachable parser. That is the point of stage 2.
|
||||
- **Implications**: Keep `RankingScore`. Add `ComplexityMap` / `Hotspot` with function identity when recoverable, test-gap hints, and a ledger overlay. Ranking still feeds hunter budgets; the map feeds humans and stage 3.
|
||||
|
||||
### Hunter: regex, HarnessX dump, or tools
|
||||
- **Context**: Previous spec adopted HarnessX "at the hunter loop" but the default hunter is regex and the HarnessX path dumps source into a prompt.
|
||||
- **Sources Consulted**: `hunter.py` (`_run_hunter_task` → `quick_scan_findings`), `hunter_harness.py` (`_describe`, `_extract_findings`), `provider/openrouter.py` (`OpenRouterChatClient.complete_json`, stdlib `urllib`), `harness_ext.py` (optional extra).
|
||||
- **Findings**:
|
||||
- OpenRouter client is already zero-extra (stdlib). It is used for optional ranking, not for hunting.
|
||||
- HarnessX extra is SHA-pinned, heavy, absent in CI. Requiring it repeats the last spec's failure mode.
|
||||
- Clearwing hunters use `read_source_file`, `grep_source`, `find_callers` with path clamping. That is the evolutionary piece, not MetaAgent regex-status flips.
|
||||
- **Implications**: Stage 2/3 hunter is a bounded tool loop over the OpenRouter client (or HarnessX if already wired). Tools are stdlib (read, Python grep, caller search from mapping + text). Path clamp at repo root. Hunter output evidence = `suspicion`.
|
||||
|
||||
### Docker sandbox: CLI vs SDK vs skip
|
||||
- **Context**: Threat model already lists sandbox limits. Need a runner that does not invert zero-dep posture.
|
||||
- **Sources Consulted**: `docs/threat-model.md` (sandbox table), `config.py` `SandboxConfig`, Clearwing `hunter_sandbox.py` (image build + spawn, Docker SDK), Docker isolation guidance (`docker run --network none --read-only --tmpfs --memory --pids-limit --cap-drop ALL --security-opt no-new-privileges`), initrunner docker-sandbox docs (CLI `docker run --rm --init`, optional `runsc`).
|
||||
- **Findings**:
|
||||
- Python `docker` SDK would be a new core or extra dependency. The operator already has the Docker CLI if they can run containers.
|
||||
- Subprocess `docker run --rm --init` with bind-mount of the repo read-only and a tmpfs/scratch write area matches the threat model and keeps `dependencies = []`.
|
||||
- Unit tests must not require a daemon: inject a `SandboxRunner` protocol, ship a fake. Integration tests skip when `docker info` fails.
|
||||
- Full sanitizer-image builds (Clearwing) are out of scope for this spec. Stage 3 runs language-specific commands on a small pinned image per language family, or `inconclusive` if the recipe is missing.
|
||||
- **Implications**: `SandboxProbe` + `DockerCliRunner`. No SDK. gVisor/`runsc` is optional `security-opt` when `docker info` reports it; not required.
|
||||
|
||||
### Evaluation corpora
|
||||
- **Context**: User rejected cheat-sheet fixtures. Current gate is ≥90% recall on those fixtures.
|
||||
- **Sources Consulted**: `benchmarks/fixtures/python-vulnerable/app.py` (line 77 split-sink comment), `javascript-vulnerable/src/app.js` (prototype pollution comment), `tests/test_detection_benchmarks.py`, `gate.py`.
|
||||
- **Findings**:
|
||||
- Stage 1 smoke fixtures are still useful as unit tests. They must not be the stage 2/3 merge gate.
|
||||
- A split-sink corpus is small and owned: source and sink on different lines; mutations through wrappers/variables. Success = map ranks the true function in the top band; stage 3 triggers at least one stage-1 miss when Docker is present.
|
||||
- Comparing map ranking to "sort by pattern-hit count" is an anti-cheat-sheet test: if regex density is the map, the gate fails.
|
||||
- **Implications**: New `benchmarks/fixtures/split-sink-*` and a stage-2 ranking gate. Keep existing `openultrasast.gate` for stage 1 only.
|
||||
|
||||
### Seams that already exist and must not be confused
|
||||
- **Context**: Brownfield traps for a spec named "map" that also adds Docker.
|
||||
- **Sources Consulted**: `skills.py` (`STAGES = ("map", "hunt", "verify", "fix")`), `mcp.py:68` (rejects `deep`), `reports.py:115` (`scan_exit_code`), `mapping.py` (no Java/Groovy AST), `config.py` (`SandboxConfig` parsed, never consumed by `_run_scan`), `rank.py` (`rank_targets_with_model` unused by CLI).
|
||||
- **Findings**:
|
||||
- Trail of Bits skill routing already uses a stage named `map` (mapping-discipline skills: Semgrep/CodeQL/sharp-edge). That is not the complexity map. `stages.py` must not reuse `skills.STAGES`.
|
||||
- MCP scan tools reject `deep` and forbid docker/shell. This spec does not widen the MCP surface; deep is CLI/CI only.
|
||||
- `--fail-on` is implemented in `scan_exit_code`, not only argparse. `worth-fixing` must be added there.
|
||||
- Java/Groovy entry points are tag-only (`line=None` → `inferred-file-surface`). Hotspot `function_name` will often be empty on Java; the map must still rank the file.
|
||||
- `[sandbox]`, `[dynamic]`, `[evidence]`, and `[models].ranker` are loaded into resolved config and then ignored. Stage 3 consumes `SandboxConfig`; it does not invent a parallel sandbox TOML.
|
||||
- **Implications**: Naming, MCP, fail-on, and Java identity are integration constraints, not new products.
|
||||
|
||||
### Evidence ladder honesty
|
||||
- **Context**: HarnessX hunter stamped LLM JSON as `static_corroboration`. Original spec forbids reporting LLM-only claims as verified.
|
||||
- **Sources Consulted**: `verification.py` (`verify_finding` accepts when reachability is not `unknown`), `hunter_harness.py` evidence_level assignment, openrouter requirements Evidence Ladder / Verification.
|
||||
- **Findings**:
|
||||
- Pattern hits as `static_corroboration` is consistent with the ladder. LLM hits as `suspicion` is the missing rule.
|
||||
- `--fail-on verified` today means reachable + static corroboration. That remains the cheap PR gate. `--fail-on worth-fixing` is the rugged gate.
|
||||
- **Implications**: Do not change CWE policy. Change hunter evidence. Add fail-on value. Report sections: Inventory, Complexity map, Worth fixing.
|
||||
|
||||
## Architecture Pattern Evaluation
|
||||
|
||||
| Option | Description | Strengths | Risks / Limitations | Notes |
|
||||
|--------|-------------|-----------|---------------------|-------|
|
||||
| Map-then-prove on existing modes | `quick`/`standard`/`deep` = stages 1 / 1+2 / 1+2+3 | Smallest CLI blast radius; matches threat-model "deep executes in sandbox"; reuses preprocess/rank/findings | Two vocabularies (mode vs stage) until the manifest lists both | **Selected** |
|
||||
| New `--stage` command family | Parallel CLI, leave modes as-is | Clearer naming | Forks CI; `deep` still a stub | Rejected |
|
||||
| Clearwing-shaped full sourcehunt | Tool hunters + sanitizer images + PoC + patch | Highest detection efficiency | Fork in all but name; out of original non-goals | Rejected as this spec; remains the oracle |
|
||||
| HarnessX-required agent loop | Stage 2/3 only run with the extra | Reuses last spec's seam | CI never installs it; last spec's failure | Rejected as a requirement; optional runtime only |
|
||||
|
||||
## Design Decisions
|
||||
|
||||
### Decision D1: Modes map to stages
|
||||
- **Context**: Need a three-question scan without breaking CI that already passes `--mode`.
|
||||
- **Selected Approach**: `quick`→stage 1, `standard`→1+2, `deep`→1+2+3. Manifest records both.
|
||||
- **Rationale**: Evolutionary adoption of the CLI the project already shipped.
|
||||
- **Trade-offs**: Operators must learn that `standard` now means "map" not "regex hunter pool." README must say so.
|
||||
|
||||
### Decision D2: Docker CLI subprocess, no SDK
|
||||
- **Context**: Zero-dep core vs isolated execution.
|
||||
- **Selected Approach**: `docker` binary via `subprocess`; probe with `docker info`; fake runner in unit tests.
|
||||
- **Rationale**: Operator runtime, not a Python dependency. Matches threat-model flags.
|
||||
- **Trade-offs**: Windows/macOS Docker Desktop variance; integration tests skip without a daemon.
|
||||
|
||||
### Decision D3: OpenRouter client for the tool hunter; HarnessX optional
|
||||
- **Context**: Last spec hid the LLM behind an extra CI does not install.
|
||||
- **Selected Approach**: Tool loop calls `OpenRouterChatClient` (or any `RankerClient`-shaped chat client) when `[models].hunter` is set. If HarnessX extra is present, `HxScanOrchestrator` may host the same tools; it is not required.
|
||||
- **Rationale**: Evolutionary use of code already in tree. Stdlib HTTP.
|
||||
- **Trade-offs**: Two possible runtimes until HarnessX is either used or ignored; tests mock the client.
|
||||
|
||||
### Decision D4: Complexity map is a new artifact
|
||||
- **Context**: Ranking already exists.
|
||||
- **Selected Approach**: `ComplexityMap` with `Hotspot` records. Ranking remains the hunter budget. Map is what humans and stage 3 consume.
|
||||
- **Rationale**: Different question (where to look / how to test vs which file to hunt first).
|
||||
|
||||
### Decision D5: Worth-fixing is a gate, not an evidence level
|
||||
- **Context**: Evidence ladder is already six rungs. Do not invent a seventh for CI.
|
||||
- **Selected Approach**: Verdicts `triggerable|not_triggerable|already_covered|inconclusive` → `worth_fixing|not_worth_fixing`. Evidence levels unchanged. Hunter stays at `suspicion`.
|
||||
- **Rationale**: CI owners need a fail-on that means "we proved it," without collapsing the ladder.
|
||||
|
||||
### Decision D6: Split-sink / mutation gate for stages 2–3
|
||||
- **Context**: 90% recall on cheat sheets is circular.
|
||||
- **Selected Approach**: New corpus + "beats pattern-hit ranking" test. Stage 1 smoke gate stays.
|
||||
- **Rationale**: Directly answers the user's evaluation complaint.
|
||||
|
||||
### Decision D7: Ledger overlays hotspot scores, not patterns
|
||||
- **Context**: Evolutionary feedback without repeating "loop must not edit patterns."
|
||||
- **Selected Approach**: `.openultrasast/calibration/complexity_ledger.json` keyed by `repo_fingerprint + path + function`. Triggerable boosts, not_triggerable demotes. Cannot hide reachable severity-5 inventory from worth-fixing consideration.
|
||||
- **Rationale**: Same overlay pattern as `rule_policy.json`, different object.
|
||||
|
||||
### Decision D8: Language recipes, not a universal executor
|
||||
- **Context**: Running untrusted Python vs C vs Java is different.
|
||||
- **Selected Approach**: Small recipe registry (Python and JavaScript required for MVP; C compile+run when a compiler exists in the image; Java `inconclusive` until a recipe ships). Missing recipe → `inconclusive`, not a crash.
|
||||
- **Rationale**: Simplification. Do not build Clearwing's sanitizer image matrix in this spec.
|
||||
|
||||
## Synthesis
|
||||
|
||||
- **Generalization**: Inventory → attention → proof is one pipeline with a `StagePlan`. Map, hints, and regress are processors on that plan, not three products.
|
||||
- **Build vs adopt**: Adopt Docker CLI and the in-tree OpenRouter client. Build the map, ledger, recipes, and split-sink corpus. Do not adopt Clearwing, docker SDK, or a cyclomatic-complexity library (nesting depth from indentation/braces is enough).
|
||||
- **Simplification**: No `--stage` flag, no HarnessX requirement, no fuzz campaign, no patch oracle, no Python Docker extra.
|
||||
|
||||
## Risks & Mitigations
|
||||
- Docker-in-Docker / rootless CI — probe, skip stage 3, degrade; do not fail stage 1.
|
||||
- Hunter cost on large repos — hotspot cap + existing `[harnessx] max_cost_usd` / hardening `max_findings` / new `max_regress_candidates`.
|
||||
- Untrusted snippet execution — structural deny (network, writes outside scratch, curl, docker socket mount). Never mount `/var/run/docker.sock`.
|
||||
- Stage 2 still heuristic without a model — allowed; gate is ranking quality on split-sink, which heuristics can pass if tags+reachability work.
|
||||
- Breaking `deep` from "exits" to "runs" — intended; document in README.
|
||||
|
||||
## References
|
||||
- OpenUltraSAST threat model — `docs/threat-model.md`
|
||||
- OpenRouter SAST harness requirements — Docker sandbox, hunter pool, evidence ladder
|
||||
- HarnessX self-improving rulesets — adjacent governance; do not reopen
|
||||
- Docker run isolation flags — `--network none`, `--read-only`, `--cap-drop ALL`, `--security-opt no-new-privileges`
|
||||
- Clearwing sourcehunt hunter tools and sandbox — implementation oracle, not a dependency
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"feature_name": "three-stage-scan",
|
||||
"created_at": "2026-09-03T16:10:07Z",
|
||||
"updated_at": "2026-09-03T16:47:00Z",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"design": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"tasks": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": true
|
||||
}
|
||||
@@ -0,0 +1,203 @@
|
||||
# Implementation Plan
|
||||
|
||||
This plan ships the three-stage scan without rebuilding CWE policy or the regex inventory. Foundation introduces the stage contract. Core builds the map, tool hunter, sandbox, and regression verdicts in parallel where boundaries do not overlap. Integration wires them onto `quick` / `standard` / `deep`. Validation adds split-sink corpora and the anti-cheat-sheet gate.
|
||||
|
||||
## Phase 1 — Foundation: Stage contract
|
||||
|
||||
- [x] 1. Introduce the stage plan and mode mapping
|
||||
- [x] 1.1 Encode the three stages and the mode-to-stage table
|
||||
- Depth `quick` requests only static, `standard` requests static then map, `deep` requests static then map then regress.
|
||||
- A skipped stage is a structured reason, not an exception, and can be serialized into the existing degradations list.
|
||||
- Observable completion: a unit test asserts the three mappings and that `quick` never includes map or regress.
|
||||
- _Requirements: 1.1, 1.2, 1.3, 1.6_
|
||||
- _Boundary: StagePlan_
|
||||
|
||||
- [x] 1.2 Add complexity and regression configuration with candidate caps
|
||||
- Configuration exposes map top-k, max hunter hotspots, max regression candidates, and optional language image pins, with defaults that keep pull-request scans cheap.
|
||||
- Sandbox memory, pids, timeout, network-off, and read-only workspace remain the existing sandbox settings.
|
||||
- Observable completion: loading config without a file yields those defaults; an override file changes top-k and max candidates only.
|
||||
- _Requirements: 9.2, 9.5_
|
||||
- _Boundary: Packaging Configuration_
|
||||
- _Depends: 1.1_
|
||||
|
||||
- [x] 1.3 Expand fail-on vocabulary with worth-fixing
|
||||
- The scan command accepts `worth-fixing` alongside the existing fail-on values and does not change the meaning of `findings` or `verified`.
|
||||
- The existing report exit helper must recognize the new value; unknown values still fail loud.
|
||||
- MCP continues to reject `deep` and must not gain docker or shell tools.
|
||||
- Observable completion: `--help` lists `worth-fixing`; a scan with no worth-fixing verdicts and `--fail-on worth-fixing` exits 0; MCP still refuses `deep`.
|
||||
- _Requirements: 5.7, 5.8_
|
||||
- _Boundary: StagePlan_
|
||||
- _Depends: 1.1_
|
||||
|
||||
## Phase 2 — Core: Map, hunter tools, sandbox (parallel after foundation)
|
||||
|
||||
- [x] 2. Build the heuristic complexity map and test-tuning hints
|
||||
- [x] 2.1 (P) Compute per-file and per-function complexity signals
|
||||
- Signals include size, nesting, surface tags, reachability, inventory-hit density as a feature not the score, and whether an adjacent test appears to exist.
|
||||
- Function identity uses entry-point names when present and otherwise remains empty.
|
||||
- Observable completion: a parser file with no pattern hit scores higher than a one-line helper that matches a pattern, proven by a unit test on a constructed target list.
|
||||
- _Requirements: 3.1, 3.2_
|
||||
- _Boundary: ComplexityMapBuilder_
|
||||
|
||||
- [x] 2.2 (P) Emit the ordered hotspot map artifact
|
||||
- Each hotspot carries score, band, rationale, signals, and linked inventory ids; the map records whether it is heuristic-only.
|
||||
- Observable completion: building a map writes a JSON artifact whose hotspot order is not identical to sorting by inventory-hit count on the same fixture.
|
||||
- _Requirements: 3.2, 3.3, 3.5, 8.6_
|
||||
- _Boundary: ComplexityMapBuilder_
|
||||
- _Depends: 2.1_
|
||||
|
||||
- [x] 2.3 Attach test-gap hints without writing into the scanned tree
|
||||
- Each hotspot gets a gap of no adjacent test, no function reference, no fuzz entry, or covered.
|
||||
- Recommended test kind is one of unit, property, sanitizer, http-contract, or fuzz-harness, and is omitted when the gap is covered.
|
||||
- Observable completion: a covered hotspot emits `covered` and no test kind; a fuzzable C parser without a harness emits `fuzz-harness`; the scanned tree is unchanged.
|
||||
- _Requirements: 4.1, 4.2, 4.3, 4.4_
|
||||
- _Boundary: TestTuningAdvisor_
|
||||
- _Depends: 2.2_
|
||||
|
||||
- [x] 2.4 Apply a complexity ledger overlay that cannot hide severity-5 reachable inventory
|
||||
- Triggerable verdicts raise a hotspot's later score; not-triggerable verdicts lower it; the underlying inventory hit remains.
|
||||
- A reachable policy-severity-5 inventory finding stays in the regression candidate set regardless of a negative ledger delta.
|
||||
- Observable completion: after a not-triggerable overlay the hotspot score drops, and a sev-5 reachable finding is still selected as a candidate, proven by a unit test.
|
||||
- _Requirements: 7.1, 7.2, 7.3, 7.4, 7.5_
|
||||
- _Boundary: ComplexityMapBuilder_
|
||||
- _Depends: 2.2_
|
||||
|
||||
- [x] 3. Add repo-bound hunter tools and a scripted tool loop
|
||||
- [x] 3.1 (P) Clamp tool paths and expose read, grep, and reference search
|
||||
- Read, grep, and reference listing resolve only under the scan root; `..` and absolute escapes raise a path-escape error.
|
||||
- Grep runs with the standard library over enumerated source files.
|
||||
- Observable completion: a unit test reads a repo file, greps a split-sink assignment, and rejects `../etc/passwd`.
|
||||
- _Requirements: 6.1, 6.2_
|
||||
- _Boundary: HunterTools_
|
||||
|
||||
- [x] 3.2 Run a bounded tool-using hunter that records suspicion only
|
||||
- When a hunter model is configured, the loop may call the three tools against map hotspots and must not stamp results as static corroboration.
|
||||
- When no model is configured, the loop does not run and a degradation is produced by the orchestrator later.
|
||||
- Observable completion: a scripted client that never calls a tool fails the test; a client that greps then returns a finding yields evidence level `suspicion` and id prefix `tool-hunter:`.
|
||||
- _Requirements: 6.1, 6.3, 6.4, 6.5_
|
||||
- _Boundary: ToolHunter_
|
||||
- _Depends: 3.1, 2.2_
|
||||
|
||||
- [x] 4. Stand up the isolated sandbox runner
|
||||
- [x] 4.1 (P) Probe whether the sandbox runtime is usable
|
||||
- A short `docker info` style probe returns available or unavailable without throwing into the scan.
|
||||
- Observable completion: the fake probe can be forced off; the real probe is skippable in unit tests.
|
||||
- _Requirements: 9.3_
|
||||
- _Boundary: SandboxProbe_
|
||||
|
||||
- [x] 4.2 (P) Run jobs through Docker CLI flags that match the threat model
|
||||
- Network is disabled, source is read-only, scratch is writable, memory/pids/timeout apply, capabilities are dropped, source tree is not writable, and the host Docker socket is never mounted.
|
||||
- A fake runner records the job and returns a programmed result so default tests do not need a daemon.
|
||||
- Observable completion: the constructed command line includes network-none and read-only and excludes docker.sock; the fake runner is used by unit tests.
|
||||
- _Requirements: 5.2, 5.8_
|
||||
- _Boundary: DockerCliRunner_
|
||||
- _Depends: 4.1_
|
||||
|
||||
- [x] 4.3 Reject unsafe regression snippets before execution
|
||||
- Snippets that refer to the Docker socket, host network, or writes under the source mount are not executed and become `inconclusive`.
|
||||
- Observable completion: each forbidden shape is rejected in a unit test and no runner job is created.
|
||||
- _Requirements: 5.8, 6.6_
|
||||
- _Boundary: RegressionRunner_
|
||||
- _Depends: 4.2_
|
||||
|
||||
## Phase 3 — Core: Regression verdicts
|
||||
|
||||
- [x] 5. Produce worth-fixing verdicts from sandboxed candidates
|
||||
- [x] 5.1 Select a bounded candidate set from hotspots and severe reachable inventory
|
||||
- Selection honors max-candidates and always includes reachable severity-5 inventory even when the ledger demoted the hotspot.
|
||||
- Observable completion: a unit test with six hotspots and max-candidates three returns three, plus the sev-5 reachable extra when present.
|
||||
- _Requirements: 5.1, 7.4, 9.5_
|
||||
- _Boundary: RegressionRunner_
|
||||
- _Depends: 2.4_
|
||||
|
||||
- [x] 5.2 (P) Add language recipes with inconclusive fallback
|
||||
- Python and JavaScript recipes run a scratch snippet with the repo on the language path; C compiles and runs when the image has a compiler; a missing recipe yields `inconclusive` rather than a crash.
|
||||
- Observable completion: Python and JavaScript recipes produce a job; an unknown language returns inconclusive without calling the runner.
|
||||
- _Requirements: 5.1, 5.6_
|
||||
- _Boundary: RegressionRunner_
|
||||
- _Depends: 5.1_
|
||||
|
||||
- [x] 5.3 Map sandbox outcomes to the four verdicts and the worth-fixing gate
|
||||
- Outcomes map to triggerable, not_triggerable, already_covered, or inconclusive; worth-fixing requires triggerable plus reachable or inferred-file-surface.
|
||||
- Not-triggerable and already-covered do not fail the build; inconclusive records a reason and does not fail the build.
|
||||
- Observable completion: a table-driven test covers the four verdicts and the worth-fixing boolean.
|
||||
- _Requirements: 5.3, 5.4, 5.5, 5.6_
|
||||
- _Boundary: WorthFixingGate_
|
||||
- _Depends: 5.2, 4.2_
|
||||
|
||||
## Phase 4 — Integration: Wire stages into the scan
|
||||
|
||||
- [x] 6. Run stages from the existing scan command
|
||||
- [x] 6.1 Keep quick as stage-1 inventory with no model and no target execution
|
||||
- Quick still loads policy, ruleset, preprocess, rank, pattern inventory, structural verify, and score.
|
||||
- The report labels those hits as inventory, not as worth-fixing.
|
||||
- Policy still fails loud on an unmapped enabled CWE.
|
||||
- Observable completion: existing quick tests stay green; the report contains an inventory section and does not start a sandbox or a hunter model.
|
||||
- _Requirements: 1.1, 2.1, 2.2, 2.3, 2.4, 2.5, 9.1_
|
||||
- _Boundary: StagePlan_
|
||||
- _Depends: 1.1_
|
||||
|
||||
- [x] 6.2 Run the complexity map on standard and record heuristic-only when no model is set
|
||||
- Standard writes the map artifact, includes it in the manifest, and records `hunter_model_unavailable` when the hunter is skipped.
|
||||
- Observable completion: `ousast scan --mode standard` on a fixture produces `complexity_map.json` and a completed map stage without Docker.
|
||||
- _Requirements: 1.2, 1.4, 3.3, 3.4, 6.4, 9.4, 9.6_
|
||||
- _Depends: 2.3, 6.1_
|
||||
|
||||
- [x] 6.3 Stop exiting on deep; run or skip regression with visible degradation
|
||||
- Deep no longer hard-exits. If the sandbox is missing, stages 1–2 complete and regress is skipped with `sandbox_unavailable`.
|
||||
- If the sandbox is present, regression runs under the candidate cap and writes verdicts.
|
||||
- Observable completion: `ousast scan --mode deep` without Docker exits 0 after stages 1–2 and lists the skip in the manifest; with a fake runner it records verdicts.
|
||||
- _Requirements: 1.3, 1.5, 5.1, 9.3, 9.6_
|
||||
- _Depends: 5.3, 6.2_
|
||||
|
||||
- [x] 6.4 Attach the tool hunter to map hotspots when a model is configured
|
||||
- Hunter findings are suspicion-only and may propose a snippet that still must pass the safety check and run only in the sandbox.
|
||||
- Observable completion: with a scripted client, standard+model produces tool-hunter suspicions; a proposed snippet is executed only on deep after the safety check.
|
||||
- _Requirements: 6.1, 6.3, 6.5, 6.6_
|
||||
- _Depends: 3.2, 6.2_
|
||||
|
||||
- [x] 6.5 Surface map, hints, and worth-fixing in the report and honor fail-on
|
||||
- Markdown, manifest, and SARIF remain; the report gains map and worth-fixing sections.
|
||||
- `--fail-on worth-fixing` fails only when a worth-fixing verdict exists.
|
||||
- Observable completion: a scan with one worth-fixing verdict fails under that flag and passes under `--fail-on never`.
|
||||
- _Requirements: 2.2, 4.1, 5.7, 7.3_
|
||||
- _Depends: 6.3_
|
||||
|
||||
## Phase 5 — Validation: Grounded corpora and CI
|
||||
|
||||
- [x] 7. Replace cheat-sheet success for stages 2 and 3
|
||||
- [x] 7.1 Add split-sink fixtures for Python, JavaScript, Java, and C/C++
|
||||
- Each language family has at least one case where source and sink are on different lines or assigned before use.
|
||||
- Observable completion: stage-1 pattern scan misses at least one planted sink in each family, documented in the manifest expected set.
|
||||
- _Requirements: 8.1_
|
||||
- _Boundary: Benchmark Harness_
|
||||
|
||||
- [x] 7.2 (P) Add a mutation of a known sink that a same-line pattern does not match
|
||||
- At least one wrapper or intermediate-variable mutation exists per the evaluation plan.
|
||||
- Observable completion: stage 1 misses the mutation; the case is listed as expected in a split-sink or mutation manifest.
|
||||
- _Requirements: 8.2_
|
||||
- _Boundary: Benchmark Harness_
|
||||
|
||||
- [x] 7.3 Gate stage 2 on ranking the true function in the high band and beating pattern-hit order
|
||||
- Stage 1 may still use existing smoke fixtures; that recall is not treated as stage-2 success.
|
||||
- The merge gate fails if map order equals pattern-hit-count order on the split-sink corpus.
|
||||
- Observable completion: a dedicated gate command or test fails a stub map that sorts by hit count and passes the heuristic map on the split-sink corpus.
|
||||
- _Requirements: 8.3, 8.4, 8.6_
|
||||
- _Depends: 7.1, 2.2_
|
||||
|
||||
- [x] 7.4 Prove stage 3 can trigger a planted miss when the sandbox is available
|
||||
- Docker-marked tests skip when the probe fails; when they run, at least one stage-1 miss is `triggerable`.
|
||||
- Default pytest does not require a daemon. The pull-request job stays stage 1; a nightly or labeled job may request deep under time and candidate caps.
|
||||
- Observable completion: fake-runner tests always run; docker tests skip-or-pass; the default CI workflow still invokes quick-only detection plus the new split-sink map gate.
|
||||
- _Requirements: 8.5, 9.1, 9.2_
|
||||
- _Depends: 6.3, 7.1_
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- `RegressConfig.images` is a sorted tuple of `(language, image)` pairs so frozen config stays hashable.
|
||||
- `scan_exit_code` duck-types `worth_fixing` on verdict objects; `_run_scan` still passes no verdicts until task 6.5. Dict-shaped verdicts would not fail the gate.
|
||||
- Map `top_k` default is 20 (not named in design.md); `max_hunter_hotspots=8` and `max_candidates=5` follow design.
|
||||
- `collect_signals` adjacent-test detection is false unless the caller also passes `repo_files`; task 2.2 must pass enumerated repo paths.
|
||||
- ToolHunter rejects JSON dumps unless a repo tool was invoked; findings are always `suspicion` with prefix `tool-hunter:`.
|
||||
- Split-sink map gate treats "high band" as the highest band the map emitted on that corpus (small fixtures often top out at medium).
|
||||
- `@pytest.mark.docker` tests skip when `SandboxProbe` is unavailable; default CI does not require a daemon.
|
||||
@@ -0,0 +1,13 @@
|
||||
"""Mutation of os.system: the callable is aliased before use so a same-line regex misses it."""
|
||||
import os
|
||||
|
||||
from flask import Flask, request
|
||||
|
||||
app = Flask(__name__)
|
||||
run_cmd = os.system
|
||||
|
||||
|
||||
@app.get("/ping")
|
||||
def ping():
|
||||
host = request.args["host"]
|
||||
return run_cmd("ping -c 1 " + host) # CWE-78 mutation: os.system aliased before the call
|
||||
@@ -0,0 +1,8 @@
|
||||
#include <stdio.h>
|
||||
#include <sqlite3.h>
|
||||
|
||||
void search(sqlite3 *db, char *term) {
|
||||
char query[512];
|
||||
snprintf(query, sizeof(query), "SELECT * FROM items WHERE title LIKE '%%%s%%'", term);
|
||||
sqlite3_exec(db, query, 0, 0, 0); /* CWE-89 split-sink: query built on prior line (regex blind spot) */
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
package example;
|
||||
|
||||
import jakarta.servlet.http.HttpServletRequest;
|
||||
import java.sql.Connection;
|
||||
|
||||
class SplitSinkServlet {
|
||||
void search(HttpServletRequest request, Connection connection) throws Exception {
|
||||
String term = request.getParameter("q");
|
||||
String sql = String.format("select * from items where title like '%%%s%%'", term);
|
||||
connection.createStatement().executeQuery(sql); // CWE-89 split-sink: sql built on prior line (regex blind spot)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
// Split-sink SQL injection: query is built, then executed on the next line.
|
||||
const express = require("express");
|
||||
const app = express();
|
||||
|
||||
app.get("/search", (req, res) => {
|
||||
const term = req.query.q;
|
||||
const query = "select * from items where title like '%" + term + "%'";
|
||||
db.query(query); // CWE-89 split-sink: query built on prior line (regex blind spot)
|
||||
res.send("ok");
|
||||
});
|
||||
|
||||
module.exports = app;
|
||||
@@ -0,0 +1,11 @@
|
||||
"""Split-sink SQL injection: query is built, then executed on the next line."""
|
||||
from flask import Flask, request
|
||||
|
||||
app = Flask(__name__)
|
||||
|
||||
|
||||
@app.get("/search")
|
||||
def search(db):
|
||||
term = request.args["q"]
|
||||
query = "select * from items where title like '%" + term + "%'"
|
||||
return db.execute(query) # CWE-89 split-sink: query built on prior line (regex blind spot)
|
||||
@@ -0,0 +1,7 @@
|
||||
"""Trivial eval helper: many same-line pattern hits, little complexity."""
|
||||
|
||||
|
||||
def ping() -> str:
|
||||
eval("1")
|
||||
eval("2")
|
||||
return "ok"
|
||||
@@ -0,0 +1,20 @@
|
||||
name = "mutation-python"
|
||||
language = "python_web"
|
||||
frameworks = ["flask"]
|
||||
setup = ["pip install flask"]
|
||||
modes = ["quick", "standard"]
|
||||
known_noise = []
|
||||
baselines = []
|
||||
|
||||
[source]
|
||||
type = "local"
|
||||
path = "../fixtures/mutation-python"
|
||||
|
||||
[[expected]]
|
||||
cwe = "CWE-78"
|
||||
class = "command injection"
|
||||
path = "app.py"
|
||||
line = 13
|
||||
function = "ping"
|
||||
sink = "run_cmd"
|
||||
evidence = "aliased os.system runs an attacker-controlled command"
|
||||
@@ -0,0 +1,20 @@
|
||||
name = "split-sink-c"
|
||||
language = "c_cpp"
|
||||
frameworks = ["native"]
|
||||
setup = []
|
||||
modes = ["quick", "standard"]
|
||||
known_noise = []
|
||||
baselines = []
|
||||
|
||||
[source]
|
||||
type = "local"
|
||||
path = "../fixtures/split-sink-c"
|
||||
|
||||
[[expected]]
|
||||
cwe = "CWE-89"
|
||||
class = "sql injection"
|
||||
path = "src/vuln.c"
|
||||
line = 7
|
||||
function = "search"
|
||||
sink = "sqlite3_exec"
|
||||
evidence = "sqlite3_exec runs a SQL query string built from attacker input on a prior line"
|
||||
@@ -0,0 +1,20 @@
|
||||
name = "split-sink-java"
|
||||
language = "java_web"
|
||||
frameworks = ["servlet"]
|
||||
setup = ["mvn test"]
|
||||
modes = ["quick", "standard"]
|
||||
known_noise = []
|
||||
baselines = []
|
||||
|
||||
[source]
|
||||
type = "local"
|
||||
path = "../fixtures/split-sink-java"
|
||||
|
||||
[[expected]]
|
||||
cwe = "CWE-89"
|
||||
class = "sql injection"
|
||||
path = "src/main/java/example/SplitSinkServlet.java"
|
||||
line = 10
|
||||
function = "search"
|
||||
sink = "executeQuery"
|
||||
evidence = "executeQuery runs a SQL string built from request input on a prior line"
|
||||
@@ -0,0 +1,19 @@
|
||||
name = "split-sink-javascript"
|
||||
language = "javascript_node_web"
|
||||
frameworks = ["node", "express"]
|
||||
setup = ["npm install"]
|
||||
modes = ["quick", "standard"]
|
||||
known_noise = []
|
||||
baselines = []
|
||||
|
||||
[source]
|
||||
type = "local"
|
||||
path = "../fixtures/split-sink-javascript"
|
||||
|
||||
[[expected]]
|
||||
cwe = "CWE-89"
|
||||
class = "sql injection"
|
||||
path = "src/app.js"
|
||||
line = 8
|
||||
sink = "query"
|
||||
evidence = "query runs a SQL string built from request input on a prior line"
|
||||
@@ -0,0 +1,20 @@
|
||||
name = "split-sink-python"
|
||||
language = "python_web"
|
||||
frameworks = ["flask"]
|
||||
setup = ["pip install flask"]
|
||||
modes = ["quick", "standard"]
|
||||
known_noise = []
|
||||
baselines = []
|
||||
|
||||
[source]
|
||||
type = "local"
|
||||
path = "../fixtures/split-sink-python"
|
||||
|
||||
[[expected]]
|
||||
cwe = "CWE-89"
|
||||
class = "sql injection"
|
||||
path = "app.py"
|
||||
line = 11
|
||||
function = "search"
|
||||
sink = "execute"
|
||||
evidence = "execute runs a SQL query string built from request input on a prior line"
|
||||
@@ -51,6 +51,7 @@ async def _run_checks() -> None:
|
||||
.with_exec(["uv", "run", "mypy", "src/openultrasast"])
|
||||
.with_exec(["uv", "run", "pytest"])
|
||||
.with_exec(["uv", "run", "python", "-m", "openultrasast.gate"])
|
||||
.with_exec(["uv", "run", "python", "-m", "openultrasast.map_gate"])
|
||||
.with_exec(["uv", "run", "python", "-m", "compileall", "src", "tests"])
|
||||
.with_exec(["uv", "build"])
|
||||
)
|
||||
|
||||
+2
-1
@@ -4,7 +4,7 @@ build-backend = "uv_build"
|
||||
|
||||
[project]
|
||||
name = "openultrasast"
|
||||
version = "1.0.0"
|
||||
version = "1.1.0"
|
||||
description = "OpenCode-oriented SAST harness for evidence-gated security analysis"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.11"
|
||||
@@ -27,6 +27,7 @@ ousast = "openultrasast.cli:main"
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
pythonpath = ["src"]
|
||||
markers = ["docker: requires a docker daemon"]
|
||||
|
||||
[tool.ruff]
|
||||
line-length = 140
|
||||
|
||||
@@ -2,4 +2,4 @@
|
||||
|
||||
__all__ = ["__version__"]
|
||||
|
||||
__version__ = "1.0.0"
|
||||
__version__ = "1.1.0"
|
||||
|
||||
+162
-7
@@ -3,10 +3,11 @@ from __future__ import annotations
|
||||
import argparse
|
||||
import json
|
||||
import tempfile
|
||||
from dataclasses import dataclass
|
||||
from dataclasses import dataclass, replace
|
||||
from pathlib import Path
|
||||
from time import perf_counter
|
||||
|
||||
from . import tool_hunter
|
||||
from .benchmark import (
|
||||
create_benchmark_run,
|
||||
evaluate_benchmark,
|
||||
@@ -26,6 +27,8 @@ from .calibration import (
|
||||
write_false_positive_learnings,
|
||||
write_ranking_calibrations,
|
||||
)
|
||||
from .complexity import map as complexity_map
|
||||
from .complexity.ledger import persist_verdicts
|
||||
from .config import load_config
|
||||
from .findings import StaticFinding, quick_scan_findings, write_findings
|
||||
from .fusion import FusionDecision, fuse_findings_dispatch
|
||||
@@ -40,11 +43,14 @@ from .mapping import analyze_entry_points, attach_reachability_hints, ingest_sar
|
||||
from .policy import assert_rules_resolve, load_policy
|
||||
from .preprocess import preprocess_repository, write_preprocess_artifact
|
||||
from .rank import rank_targets, write_rankings
|
||||
from .regress import CandidateVerdict, run_regression, write_verdicts
|
||||
from .reports import scan_exit_code, write_manifest, write_markdown_report, write_sarif_report
|
||||
from .ruleset import DEFAULT_RULESET_DIR, load_ruleset
|
||||
from .run import ScanRun, create_scan_run
|
||||
from .sandbox import resolve_sandbox_probe, resolve_sandbox_runner
|
||||
from .scoring import build_score_artifact
|
||||
from .verification import VerificationResult, write_verification_results
|
||||
from .stages import Stage, plan_for_mode, record_completed, record_skip, skip_as_degradation, stages_payload
|
||||
from .verification import VerificationResult, verify_findings, write_verification_results
|
||||
from .verify_judge import verify_findings_dispatch
|
||||
|
||||
CALIBRATION_DIR = ".openultrasast/calibration"
|
||||
@@ -69,7 +75,7 @@ def main(argv: list[str] | None = None) -> int:
|
||||
scan.add_argument("path", type=Path)
|
||||
scan.add_argument("--mode", choices=("quick", "standard", "deep"), default="quick")
|
||||
scan.add_argument("--config", type=Path, default=Path("openultrasast.toml"))
|
||||
scan.add_argument("--fail-on", choices=("never", "findings", "verified"), default="never")
|
||||
scan.add_argument("--fail-on", choices=("never", "findings", "verified", "worth-fixing"), default="never")
|
||||
|
||||
index = subparsers.add_parser("index", help="chunk a local repository for embedding index construction")
|
||||
index.add_argument("path", type=Path)
|
||||
@@ -132,11 +138,10 @@ def _scan(path: Path, config_path: Path, mode: str, fail_on: str) -> int:
|
||||
|
||||
|
||||
def _run_scan(path: Path, config_path: Path, mode: str, fail_on: str) -> ScanOutcome:
|
||||
if mode == "deep":
|
||||
raise SystemExit("mode 'deep' is specified but sandboxed dynamic analysis is not implemented")
|
||||
if not path.exists() or not path.is_dir():
|
||||
raise SystemExit(f"scan path is not a directory: {path}")
|
||||
|
||||
plan = plan_for_mode(mode)
|
||||
config = load_config(config_path if config_path.exists() else None)
|
||||
# Capability gates for the optional HarnessX agentic plane. When the extra is
|
||||
# absent (default/CI), both stay False and the deterministic path runs unchanged.
|
||||
@@ -299,8 +304,87 @@ def _run_scan(path: Path, config_path: Path, mode: str, fail_on: str) -> ScanOut
|
||||
)
|
||||
score_path = run.root / "score.json"
|
||||
score_path.write_text(json.dumps(score_artifact.to_dict(), indent=2, sort_keys=True) + "\n")
|
||||
plan = record_completed(plan, Stage.STATIC)
|
||||
complexity_payload: dict[str, object] | None = None
|
||||
complexity_map_path = run.root / "complexity_map.json"
|
||||
built_map = None
|
||||
if Stage.MAP in plan.requested:
|
||||
built_map = runtime.run_stage(
|
||||
"map",
|
||||
lambda: complexity_map.build_complexity_map(
|
||||
targets,
|
||||
findings,
|
||||
complexity_map_path,
|
||||
repo_files=[target.path for target in targets],
|
||||
ledger_path=run.target / CALIBRATION_DIR / "complexity_ledger.json",
|
||||
),
|
||||
)
|
||||
plan = record_completed(plan, Stage.MAP)
|
||||
if not hunter_model:
|
||||
runtime.state["degradations"].append(skip_as_degradation(Stage.MAP, "hunter_model_unavailable"))
|
||||
else:
|
||||
built_map, findings, verifications = _attach_tool_hunter(
|
||||
root=run.target,
|
||||
built_map=built_map,
|
||||
findings=findings,
|
||||
verifications=verifications,
|
||||
hunter_model=hunter_model,
|
||||
max_hotspots=config.complexity.max_hunter_hotspots,
|
||||
max_findings=max_findings,
|
||||
findings_path=findings_path,
|
||||
verification_path=verification_path,
|
||||
complexity_map_path=complexity_map_path,
|
||||
runtime=runtime,
|
||||
)
|
||||
complexity_payload = {
|
||||
"hotspot_count": len(built_map.hotspots),
|
||||
"heuristic_only": built_map.heuristic_only,
|
||||
}
|
||||
verdicts_path = run.root / "verdicts.json"
|
||||
wrote_verdicts = False
|
||||
verdict_records: tuple[CandidateVerdict, ...] = ()
|
||||
if Stage.REGRESS in plan.requested:
|
||||
probe = resolve_sandbox_probe()
|
||||
if not probe.available():
|
||||
plan = record_skip(plan, Stage.REGRESS, "sandbox_unavailable")
|
||||
runtime.state["degradations"].append(skip_as_degradation(Stage.REGRESS, "sandbox_unavailable"))
|
||||
else:
|
||||
sandbox = resolve_sandbox_runner()
|
||||
hotspots = built_map.hotspots if built_map is not None else ()
|
||||
cwe_by_rule_id = {rule.rule_id: rule.cwe for rule in ruleset}
|
||||
rule_cwe_by_finding = {finding.finding_id: cwe_by_rule_id.get(finding.finding_id.split(":", 1)[0], "") for finding in findings}
|
||||
languages_by_path = {target.path: target.language for target in targets}
|
||||
records = runtime.run_stage(
|
||||
"regress",
|
||||
lambda: run_regression(
|
||||
hotspots,
|
||||
findings,
|
||||
max_candidates=config.regress.max_candidates,
|
||||
policy=policy,
|
||||
rule_cwe=rule_cwe_by_finding,
|
||||
languages_by_path=languages_by_path,
|
||||
repo_root=run.target,
|
||||
sandbox=sandbox,
|
||||
sandbox_limits=config.sandbox,
|
||||
images=dict(config.regress.images),
|
||||
),
|
||||
)
|
||||
write_verdicts(records, verdicts_path)
|
||||
persist_verdicts(run.target / CALIBRATION_DIR / "complexity_ledger.json", records)
|
||||
verdict_records = records
|
||||
wrote_verdicts = True
|
||||
plan = record_completed(plan, Stage.REGRESS)
|
||||
worth_fixing_payload = _worth_fixing_payload(verdict_records) if wrote_verdicts else None
|
||||
runtime.run_stage(
|
||||
"report", lambda: write_markdown_report(findings, markdown_path, verifications, redact=config.hardening.redact_secrets)
|
||||
"report",
|
||||
lambda: write_markdown_report(
|
||||
findings,
|
||||
markdown_path,
|
||||
verifications,
|
||||
redact=config.hardening.redact_secrets,
|
||||
complexity_map=built_map,
|
||||
verdicts=verdict_records if wrote_verdicts else None,
|
||||
),
|
||||
)
|
||||
runtime.run_stage("sarif", lambda: write_sarif_report(findings, verifications, sarif_path))
|
||||
runtime.run_stage(
|
||||
@@ -317,11 +401,16 @@ def _run_scan(path: Path, config_path: Path, mode: str, fail_on: str) -> ScanOut
|
||||
score=score_path,
|
||||
trajectories=trajectories_path if trajectories else None,
|
||||
fusion=fusion_path if fusion_decisions else None,
|
||||
complexity_map=complexity_map_path if complexity_payload is not None else None,
|
||||
verdicts=verdicts_path if wrote_verdicts else None,
|
||||
),
|
||||
path=manifest_path,
|
||||
score=score_artifact.to_dict(),
|
||||
degradations=runtime.state["degradations"] or None,
|
||||
fusion=[_fusion_summary(decision) for decision in fusion_decisions] or None,
|
||||
stages=stages_payload(plan),
|
||||
complexity=complexity_payload,
|
||||
worth_fixing=worth_fixing_payload,
|
||||
),
|
||||
)
|
||||
runtime.finish(status="succeeded")
|
||||
@@ -332,10 +421,59 @@ def _run_scan(path: Path, config_path: Path, mode: str, fail_on: str) -> ScanOut
|
||||
ranked_target_count=len(rankings),
|
||||
finding_count=len(findings),
|
||||
calibrations_applied=len(applied_calibrations),
|
||||
exit_code=scan_exit_code(findings, verifications, fail_on),
|
||||
exit_code=scan_exit_code(findings, verifications, fail_on, worth_fixing_verdicts=verdict_records),
|
||||
)
|
||||
|
||||
|
||||
def _attach_tool_hunter(
|
||||
*,
|
||||
root: Path,
|
||||
built_map: complexity_map.ComplexityMap,
|
||||
findings: list[StaticFinding],
|
||||
verifications: list[VerificationResult],
|
||||
hunter_model: str,
|
||||
max_hotspots: int,
|
||||
max_findings: int,
|
||||
findings_path: Path,
|
||||
verification_path: Path,
|
||||
complexity_map_path: Path,
|
||||
runtime: HarnessRuntime,
|
||||
) -> tuple[complexity_map.ComplexityMap, list[StaticFinding], list[VerificationResult]]:
|
||||
client = tool_hunter.resolve_hunter_client()
|
||||
if client is None:
|
||||
return built_map, findings, verifications
|
||||
hunter_hotspots = list(built_map.hotspots[: max(max_hotspots, 0)])
|
||||
hunter_findings = runtime.run_stage(
|
||||
"tool_hunter",
|
||||
lambda: tool_hunter.run_tool_hunter(
|
||||
root,
|
||||
hunter_hotspots,
|
||||
client=client,
|
||||
model=hunter_model,
|
||||
max_steps=tool_hunter.DEFAULT_MAX_STEPS,
|
||||
),
|
||||
)
|
||||
seen = {finding.finding_id for finding in findings}
|
||||
extra = [finding for finding in hunter_findings if finding.finding_id not in seen]
|
||||
if extra:
|
||||
findings = findings + extra
|
||||
if max_findings and len(findings) > max_findings:
|
||||
if not any(entry.get("reason") == "max_findings_exceeded" for entry in runtime.state["degradations"]):
|
||||
runtime.state["degradations"].append(
|
||||
{"stage": "budget", "reason": "max_findings_exceeded", "requested": max_findings, "actual": len(findings)}
|
||||
)
|
||||
findings = findings[:max_findings]
|
||||
kept = {finding.finding_id for finding in findings}
|
||||
extra = [finding for finding in extra if finding.finding_id in kept]
|
||||
if extra:
|
||||
verifications = verifications + verify_findings(extra)
|
||||
write_verification_results(verifications, verification_path)
|
||||
write_findings(findings, findings_path)
|
||||
built_map = replace(built_map, heuristic_only=False)
|
||||
complexity_map.write_complexity_map(built_map, complexity_map_path)
|
||||
return built_map, findings, verifications
|
||||
|
||||
|
||||
def _persist_calibration_feedback(
|
||||
run: ScanRun,
|
||||
ledger_path: Path,
|
||||
@@ -369,6 +507,23 @@ def _load_static_hints(sarif_paths: tuple[str, ...]) -> list[object]:
|
||||
return hints
|
||||
|
||||
|
||||
def _worth_fixing_payload(verdicts: tuple[CandidateVerdict, ...]) -> dict[str, object]:
|
||||
worth = [record for record in verdicts if record.worth_fixing]
|
||||
return {
|
||||
"count": len(worth),
|
||||
"verdicts": [
|
||||
{
|
||||
"path": record.path,
|
||||
"function_name": record.function_name,
|
||||
"verdict": record.verdict,
|
||||
"reason": record.reason,
|
||||
"worth_fixing": True,
|
||||
}
|
||||
for record in worth
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _artifact_paths(**paths: Path | None) -> dict[str, Path]:
|
||||
return {name: path for name, path in paths.items() if path is not None}
|
||||
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
"""Heuristic complexity map for stage 2."""
|
||||
|
||||
from .hints import TestHint, attach_test_hints
|
||||
from .ledger import (
|
||||
LedgerEntry,
|
||||
apply_overlay,
|
||||
hotspot_key,
|
||||
load_ledger,
|
||||
must_keep_as_candidate,
|
||||
persist_verdicts,
|
||||
record_verdict,
|
||||
select_forced_candidates,
|
||||
write_ledger,
|
||||
)
|
||||
from .map import ComplexityMap, Hotspot, build_complexity_map, write_complexity_map
|
||||
from .signals import ComplexitySignals, collect_signals
|
||||
|
||||
__all__ = [
|
||||
"ComplexityMap",
|
||||
"ComplexitySignals",
|
||||
"Hotspot",
|
||||
"LedgerEntry",
|
||||
"TestHint",
|
||||
"apply_overlay",
|
||||
"attach_test_hints",
|
||||
"build_complexity_map",
|
||||
"collect_signals",
|
||||
"hotspot_key",
|
||||
"load_ledger",
|
||||
"must_keep_as_candidate",
|
||||
"persist_verdicts",
|
||||
"record_verdict",
|
||||
"select_forced_candidates",
|
||||
"write_complexity_map",
|
||||
"write_ledger",
|
||||
]
|
||||
@@ -0,0 +1,220 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections.abc import Iterable, Mapping, Sequence
|
||||
from dataclasses import dataclass, replace
|
||||
from pathlib import Path, PurePosixPath
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from ..preprocess import FileTarget
|
||||
from .signals import _is_test_filename
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from .map import Hotspot
|
||||
|
||||
GAP_NO_ADJACENT_TEST = "no_adjacent_test"
|
||||
GAP_NO_FUNCTION_REFERENCE = "no_function_reference"
|
||||
GAP_NO_FUZZ_ENTRY = "no_fuzz_entry"
|
||||
GAP_COVERED = "covered"
|
||||
GAPS = frozenset({GAP_NO_ADJACENT_TEST, GAP_NO_FUNCTION_REFERENCE, GAP_NO_FUZZ_ENTRY, GAP_COVERED})
|
||||
|
||||
KIND_UNIT = "unit"
|
||||
KIND_PROPERTY = "property"
|
||||
KIND_SANITIZER = "sanitizer"
|
||||
KIND_HTTP_CONTRACT = "http-contract"
|
||||
KIND_FUZZ_HARNESS = "fuzz-harness"
|
||||
TEST_KINDS = frozenset({KIND_UNIT, KIND_PROPERTY, KIND_SANITIZER, KIND_HTTP_CONTRACT, KIND_FUZZ_HARNESS})
|
||||
|
||||
_MEMORY_UNSAFE_LANGUAGES = frozenset({"c", "cpp"})
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TestHint:
|
||||
path: str
|
||||
function_name: str | None
|
||||
gap: str # no_adjacent_test | no_function_reference | no_fuzz_entry | covered
|
||||
test_kind: str | None
|
||||
reason: str
|
||||
__test__ = False
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if self.gap not in GAPS:
|
||||
raise ValueError(f"unknown test gap: {self.gap}")
|
||||
if self.test_kind is not None and self.test_kind not in TEST_KINDS:
|
||||
raise ValueError(f"unknown test kind: {self.test_kind}")
|
||||
if self.gap == GAP_COVERED:
|
||||
if self.test_kind is not None:
|
||||
raise ValueError("covered gap must omit test_kind")
|
||||
elif self.test_kind is None:
|
||||
raise ValueError(f"{self.gap} gap requires a test_kind")
|
||||
|
||||
|
||||
def attach_test_hints(
|
||||
hotspots: Sequence[Hotspot],
|
||||
targets: Sequence[FileTarget] = (),
|
||||
*,
|
||||
repo_files: Iterable[str] = (),
|
||||
sources: Mapping[str, str] | None = None,
|
||||
) -> tuple[Hotspot, ...]:
|
||||
targets_by_path = {target.path: target for target in targets}
|
||||
repo_paths = {str(path).replace("\\", "/") for path in repo_files}
|
||||
repo_root = _repo_root(targets)
|
||||
test_files = tuple(path for path in repo_paths if _is_test_filename(PurePosixPath(path).name))
|
||||
advised: list[Hotspot] = []
|
||||
for hotspot in hotspots:
|
||||
target = targets_by_path.get(hotspot.path)
|
||||
hint = advise_test_hint(
|
||||
path=hotspot.path,
|
||||
function_name=hotspot.function_name,
|
||||
has_adjacent_test=bool(hotspot.signals.get("has_adjacent_test")),
|
||||
language=_language(target, hotspot.path),
|
||||
tags=_tags(target, hotspot.signals),
|
||||
has_fuzz_entry_point=_has_fuzz_harness(target, hotspot.path, sources, repo_root),
|
||||
function_referenced=_function_referenced(hotspot.function_name, test_files, sources, repo_root),
|
||||
)
|
||||
advised.append(replace(hotspot, test_hint=hint))
|
||||
return tuple(advised)
|
||||
|
||||
|
||||
def advise_test_hint(
|
||||
*,
|
||||
path: str,
|
||||
function_name: str | None,
|
||||
has_adjacent_test: bool,
|
||||
language: str,
|
||||
tags: Sequence[str],
|
||||
has_fuzz_entry_point: bool,
|
||||
function_referenced: bool,
|
||||
) -> TestHint:
|
||||
tag_set = tuple(tags)
|
||||
if _needs_fuzz_or_sanitizer(language, tag_set) and not has_fuzz_entry_point:
|
||||
gap, test_kind = GAP_NO_FUZZ_ENTRY, (KIND_FUZZ_HARNESS if "parser" in tag_set else KIND_SANITIZER)
|
||||
elif not has_adjacent_test:
|
||||
gap, test_kind = GAP_NO_ADJACENT_TEST, _kind_for_surface(tag_set)
|
||||
elif function_name and not function_referenced:
|
||||
gap, test_kind = GAP_NO_FUNCTION_REFERENCE, _kind_for_surface(tag_set)
|
||||
else:
|
||||
gap, test_kind = GAP_COVERED, None
|
||||
return TestHint(
|
||||
path=path,
|
||||
function_name=function_name,
|
||||
gap=gap,
|
||||
test_kind=test_kind,
|
||||
reason=_reason(
|
||||
gap,
|
||||
test_kind,
|
||||
path=path,
|
||||
function_name=function_name,
|
||||
language=language,
|
||||
tags=tag_set,
|
||||
has_adjacent_test=has_adjacent_test,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _needs_fuzz_or_sanitizer(language: str, tags: Sequence[str]) -> bool:
|
||||
if language not in _MEMORY_UNSAFE_LANGUAGES:
|
||||
return False
|
||||
return "parser" in tags or "memory_unsafe" in tags
|
||||
|
||||
|
||||
def _kind_for_surface(tags: Sequence[str]) -> str:
|
||||
if "network_entry" in tags:
|
||||
return KIND_HTTP_CONTRACT
|
||||
if "parser" in tags or "deserialization" in tags:
|
||||
return KIND_PROPERTY
|
||||
return KIND_UNIT
|
||||
|
||||
|
||||
def _reason(
|
||||
gap: str,
|
||||
test_kind: str | None,
|
||||
*,
|
||||
path: str,
|
||||
function_name: str | None,
|
||||
language: str,
|
||||
tags: Sequence[str],
|
||||
has_adjacent_test: bool,
|
||||
) -> str:
|
||||
identity = f"{path}::{function_name}" if function_name else path
|
||||
tag_text = ",".join(tags) or "none"
|
||||
if gap == GAP_COVERED:
|
||||
return f"covering tests already exist for {identity}; has_adjacent_test=true."
|
||||
if gap == GAP_NO_FUZZ_ENTRY:
|
||||
return f"{language} tags={tag_text} surface {identity} has no sanitizer or fuzz harness; recommend {test_kind}."
|
||||
if gap == GAP_NO_FUNCTION_REFERENCE:
|
||||
return f"adjacent tests do not reference {function_name}; recommend {test_kind}."
|
||||
return f"no adjacent test file for {identity}; has_adjacent_test={str(has_adjacent_test).lower()}; recommend {test_kind}."
|
||||
|
||||
|
||||
def _language(target: FileTarget | None, path: str) -> str:
|
||||
if target is not None:
|
||||
return target.language
|
||||
suffix = PurePosixPath(path).suffix.lower()
|
||||
if suffix in {".c", ".h"}:
|
||||
return "c"
|
||||
if suffix in {".cc", ".cpp", ".cxx", ".hpp"}:
|
||||
return "cpp"
|
||||
return "unknown"
|
||||
|
||||
|
||||
def _tags(target: FileTarget | None, signals: Mapping[str, float | int | bool | str]) -> tuple[str, ...]:
|
||||
if target is not None:
|
||||
return tuple(target.tags)
|
||||
raw = signals.get("tags", "")
|
||||
if not isinstance(raw, str) or not raw:
|
||||
return ()
|
||||
return tuple(tag for tag in raw.split(",") if tag)
|
||||
|
||||
|
||||
def _has_fuzz_harness(
|
||||
target: FileTarget | None,
|
||||
path: str,
|
||||
sources: Mapping[str, str] | None,
|
||||
repo_root: Path | None,
|
||||
) -> bool:
|
||||
if target is not None and target.has_fuzz_entry_point:
|
||||
return True
|
||||
return "LLVMFuzzerTestOneInput" in _lookup_source(path, sources, repo_root)
|
||||
|
||||
|
||||
def _function_referenced(
|
||||
function_name: str | None,
|
||||
test_files: Sequence[str],
|
||||
sources: Mapping[str, str] | None,
|
||||
repo_root: Path | None,
|
||||
) -> bool:
|
||||
if not function_name:
|
||||
return True
|
||||
pattern = re.compile(rf"\b{re.escape(function_name)}\b")
|
||||
return any(pattern.search(_lookup_source(path, sources, repo_root)) for path in test_files)
|
||||
|
||||
|
||||
def _lookup_source(path: str, sources: Mapping[str, str] | None, repo_root: Path | None) -> str:
|
||||
normalized = path.replace("\\", "/")
|
||||
if sources:
|
||||
if normalized in sources:
|
||||
return sources[normalized]
|
||||
if path in sources:
|
||||
return sources[path]
|
||||
if repo_root is None:
|
||||
return ""
|
||||
candidate = repo_root / normalized
|
||||
if not candidate.is_file():
|
||||
return ""
|
||||
try:
|
||||
return candidate.read_text(errors="ignore")
|
||||
except OSError:
|
||||
return ""
|
||||
|
||||
|
||||
def _repo_root(targets: Sequence[FileTarget]) -> Path | None:
|
||||
for target in targets:
|
||||
absolute = Path(target.absolute_path).as_posix()
|
||||
relative = target.path.replace("\\", "/")
|
||||
if not relative or not absolute.endswith(relative):
|
||||
continue
|
||||
root = Path(absolute[: -len(relative)].rstrip("/"))
|
||||
if root.exists():
|
||||
return root
|
||||
return None
|
||||
@@ -0,0 +1,196 @@
|
||||
"""Loop-owned complexity ledger: overlay hotspot scores without hiding sev-5 inventory."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections.abc import Mapping, Sequence
|
||||
from dataclasses import asdict, dataclass, replace
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from ..findings import StaticFinding
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from .map import Hotspot
|
||||
|
||||
TRIGGERABLE_DELTA = 1.5
|
||||
NOT_TRIGGERABLE_DELTA = -1.5
|
||||
VERDICT_DELTAS = {
|
||||
"triggerable": TRIGGERABLE_DELTA,
|
||||
"not_triggerable": NOT_TRIGGERABLE_DELTA,
|
||||
}
|
||||
FORCED_CANDIDATE_SEVERITY = 5
|
||||
FORCED_CANDIDATE_REACHABILITY = frozenset({"reachable", "inferred-file-surface"})
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class LedgerEntry:
|
||||
score_delta: float
|
||||
last_verdict: str
|
||||
round: int = 0
|
||||
|
||||
|
||||
def hotspot_key(path: str, function_name: str | None) -> str:
|
||||
return f"{path}:{function_name or ''}"
|
||||
|
||||
|
||||
def load_ledger(path: Path | None) -> dict[str, LedgerEntry]:
|
||||
if path is None or not path.exists():
|
||||
return {}
|
||||
payload = json.loads(path.read_text())
|
||||
entries: dict[str, LedgerEntry] = {}
|
||||
for key, value in payload.items():
|
||||
entry = _coerce_entry(value)
|
||||
if entry is not None:
|
||||
entries[str(key)] = entry
|
||||
return entries
|
||||
|
||||
|
||||
def write_ledger(path: Path, ledger: Mapping[str, LedgerEntry | Mapping[str, object]]) -> None:
|
||||
payload: dict[str, dict[str, object]] = {}
|
||||
for key, value in ledger.items():
|
||||
entry = _coerce_entry(value)
|
||||
if entry is None:
|
||||
continue
|
||||
payload[str(key)] = asdict(entry)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
|
||||
|
||||
|
||||
def persist_verdicts(path: Path, verdicts: Sequence[object]) -> dict[str, LedgerEntry]:
|
||||
"""Record triggerable / not_triggerable outcomes into the loop-owned complexity ledger."""
|
||||
ledger = load_ledger(path)
|
||||
changed = False
|
||||
for item in verdicts:
|
||||
verdict = str(getattr(item, "verdict", ""))
|
||||
if verdict not in VERDICT_DELTAS:
|
||||
continue
|
||||
raw_function = getattr(item, "function_name", None)
|
||||
ledger = record_verdict(
|
||||
ledger,
|
||||
path=str(getattr(item, "path", "")),
|
||||
function_name=raw_function if isinstance(raw_function, str) else None,
|
||||
verdict=verdict,
|
||||
)
|
||||
changed = True
|
||||
if changed:
|
||||
write_ledger(path, ledger)
|
||||
return ledger
|
||||
|
||||
|
||||
def record_verdict(
|
||||
ledger: Mapping[str, LedgerEntry | Mapping[str, object]],
|
||||
*,
|
||||
path: str,
|
||||
function_name: str | None,
|
||||
verdict: str,
|
||||
) -> dict[str, LedgerEntry]:
|
||||
updated: dict[str, LedgerEntry] = {}
|
||||
for key, value in ledger.items():
|
||||
entry = _coerce_entry(value)
|
||||
if entry is not None:
|
||||
updated[str(key)] = entry
|
||||
key = hotspot_key(path, function_name)
|
||||
previous = updated.get(key)
|
||||
updated[key] = LedgerEntry(
|
||||
score_delta=round((previous.score_delta if previous else 0.0) + VERDICT_DELTAS.get(verdict, 0.0), 4),
|
||||
last_verdict=verdict,
|
||||
round=(previous.round if previous else 0) + 1,
|
||||
)
|
||||
return updated
|
||||
|
||||
|
||||
def apply_overlay(
|
||||
hotspots: Sequence[Hotspot],
|
||||
ledger: Mapping[str, LedgerEntry | Mapping[str, object]],
|
||||
) -> tuple[Hotspot, ...]:
|
||||
if not ledger:
|
||||
return tuple(hotspots)
|
||||
adjusted: list[Hotspot] = []
|
||||
for hotspot in hotspots:
|
||||
entry = _coerce_entry(ledger.get(hotspot_key(hotspot.path, hotspot.function_name)))
|
||||
if entry is None or entry.score_delta == 0.0:
|
||||
adjusted.append(hotspot)
|
||||
continue
|
||||
score = round(hotspot.score + entry.score_delta, 4)
|
||||
signals = dict(hotspot.signals)
|
||||
signals["ledger_delta"] = entry.score_delta
|
||||
adjusted.append(
|
||||
replace(
|
||||
hotspot,
|
||||
score=score,
|
||||
band=_band_for(score),
|
||||
signals=signals,
|
||||
rationale=(
|
||||
f"{hotspot.rationale} ledger overlay last_verdict={entry.last_verdict} "
|
||||
f"score_delta={entry.score_delta} (inventory retained)."
|
||||
),
|
||||
)
|
||||
)
|
||||
return tuple(sorted(adjusted, key=lambda item: (-item.score, item.path, item.function_name or "")))
|
||||
|
||||
|
||||
def must_keep_as_candidate(
|
||||
finding: StaticFinding,
|
||||
policy_severity: int | object,
|
||||
reachability: str | None = None,
|
||||
) -> bool:
|
||||
status = finding.reachability_status if reachability is None else reachability
|
||||
return _as_severity(policy_severity) >= FORCED_CANDIDATE_SEVERITY and status in FORCED_CANDIDATE_REACHABILITY
|
||||
|
||||
|
||||
def select_forced_candidates(
|
||||
findings: Sequence[StaticFinding],
|
||||
policy: Mapping[str, object],
|
||||
*,
|
||||
rule_cwe: Mapping[str, str] | None = None,
|
||||
) -> tuple[StaticFinding, ...]:
|
||||
cwe_by_id = rule_cwe or {}
|
||||
return tuple(
|
||||
finding
|
||||
for finding in findings
|
||||
if must_keep_as_candidate(finding, _policy_severity_for(finding, policy, cwe_by_id), finding.reachability_status)
|
||||
)
|
||||
|
||||
|
||||
def _policy_severity_for(finding: StaticFinding, policy: Mapping[str, object], cwe_by_id: Mapping[str, str]) -> int:
|
||||
if finding.finding_id in policy:
|
||||
return _as_severity(policy[finding.finding_id])
|
||||
cwe = cwe_by_id.get(finding.finding_id)
|
||||
if cwe is not None and cwe in policy:
|
||||
return _as_severity(policy[cwe])
|
||||
return 0
|
||||
|
||||
|
||||
def _as_severity(value: object) -> int:
|
||||
if isinstance(value, bool):
|
||||
return 0
|
||||
if isinstance(value, int):
|
||||
return value
|
||||
severity = getattr(value, "severity", None)
|
||||
if isinstance(severity, bool) or not isinstance(severity, int):
|
||||
return 0
|
||||
return severity
|
||||
|
||||
|
||||
def _coerce_entry(value: object) -> LedgerEntry | None:
|
||||
if isinstance(value, LedgerEntry):
|
||||
return value
|
||||
if not isinstance(value, Mapping):
|
||||
return None
|
||||
raw_delta = value.get("score_delta")
|
||||
if isinstance(raw_delta, bool) or not isinstance(raw_delta, int | float):
|
||||
return None
|
||||
raw_round = value.get("round", 0)
|
||||
round_n = raw_round if isinstance(raw_round, int) and not isinstance(raw_round, bool) else 0
|
||||
return LedgerEntry(
|
||||
score_delta=float(raw_delta),
|
||||
last_verdict=str(value.get("last_verdict", "")),
|
||||
round=round_n,
|
||||
)
|
||||
|
||||
|
||||
def _band_for(score: float) -> str:
|
||||
from .map import _band
|
||||
|
||||
return _band(score)
|
||||
@@ -0,0 +1,109 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections.abc import Iterable, Mapping, Sequence
|
||||
from dataclasses import asdict, dataclass
|
||||
from pathlib import Path
|
||||
|
||||
from ..findings import StaticFinding
|
||||
from ..preprocess import FileTarget
|
||||
from .hints import TestHint, attach_test_hints
|
||||
from .ledger import apply_overlay, load_ledger
|
||||
from .signals import ComplexitySignals, collect_signals
|
||||
|
||||
# Score cutoffs for hotspot bands; inventory density is not the rank driver.
|
||||
_HIGH_SCORE = 6.0
|
||||
_MEDIUM_SCORE = 3.0
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Hotspot:
|
||||
path: str
|
||||
function_name: str | None
|
||||
score: float
|
||||
band: str # high | medium | low
|
||||
signals: dict[str, float | int | bool | str]
|
||||
rationale: str
|
||||
test_hint: TestHint | None
|
||||
inventory_finding_ids: tuple[str, ...]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ComplexityMap:
|
||||
hotspots: tuple[Hotspot, ...]
|
||||
heuristic_only: bool
|
||||
|
||||
|
||||
def build_complexity_map(
|
||||
targets: Sequence[FileTarget],
|
||||
findings: Sequence[StaticFinding],
|
||||
output_path: Path,
|
||||
*,
|
||||
repo_files: Iterable[str] = (),
|
||||
sources: Mapping[str, str] | None = None,
|
||||
ledger_path: Path | None = None,
|
||||
) -> ComplexityMap:
|
||||
collected = collect_signals(targets, findings, repo_files=repo_files, sources=sources)
|
||||
ids_by_path = _inventory_ids_by_path(findings)
|
||||
hotspots = tuple(_hotspot_from_signals(item, ids_by_path.get(item.path, ())) for item in collected)
|
||||
hotspots = attach_test_hints(hotspots, targets, repo_files=repo_files, sources=sources)
|
||||
if ledger_path is not None:
|
||||
hotspots = apply_overlay(hotspots, load_ledger(ledger_path))
|
||||
complexity_map = ComplexityMap(hotspots=hotspots, heuristic_only=True)
|
||||
write_complexity_map(complexity_map, output_path)
|
||||
return complexity_map
|
||||
|
||||
|
||||
def write_complexity_map(complexity_map: ComplexityMap, path: Path) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(asdict(complexity_map), indent=2, sort_keys=True) + "\n")
|
||||
|
||||
|
||||
def _hotspot_from_signals(item: ComplexitySignals, inventory_finding_ids: tuple[str, ...]) -> Hotspot:
|
||||
band = _band(item.score)
|
||||
return Hotspot(
|
||||
path=item.path,
|
||||
function_name=item.function_name,
|
||||
score=item.score,
|
||||
band=band,
|
||||
signals=_signal_payload(item),
|
||||
rationale=_rationale(item, band),
|
||||
test_hint=None,
|
||||
inventory_finding_ids=inventory_finding_ids,
|
||||
)
|
||||
|
||||
|
||||
def _band(score: float) -> str:
|
||||
if score >= _HIGH_SCORE:
|
||||
return "high"
|
||||
if score >= _MEDIUM_SCORE:
|
||||
return "medium"
|
||||
return "low"
|
||||
|
||||
|
||||
def _signal_payload(item: ComplexitySignals) -> dict[str, float | int | bool | str]:
|
||||
return {
|
||||
"loc": item.loc,
|
||||
"nesting": item.nesting,
|
||||
"reachability": item.reachability,
|
||||
"inventory_hit_count": item.inventory_hit_count,
|
||||
"has_adjacent_test": item.has_adjacent_test,
|
||||
"tags": ",".join(item.tags),
|
||||
}
|
||||
|
||||
|
||||
def _rationale(item: ComplexitySignals, band: str) -> str:
|
||||
tags = ",".join(item.tags) or "none"
|
||||
return (
|
||||
f"{band} band: score {item.score} against cutoffs high>={_HIGH_SCORE}, medium>={_MEDIUM_SCORE}; "
|
||||
f"loc={item.loc}, nesting={item.nesting}, tags={tags}, reachability={item.reachability}, "
|
||||
f"inventory_hit_count={item.inventory_hit_count} (density is a feature not the rank driver), "
|
||||
f"has_adjacent_test={str(item.has_adjacent_test).lower()}."
|
||||
)
|
||||
|
||||
|
||||
def _inventory_ids_by_path(findings: Sequence[StaticFinding]) -> dict[str, tuple[str, ...]]:
|
||||
grouped: dict[str, list[str]] = {}
|
||||
for finding in findings:
|
||||
grouped.setdefault(finding.path, []).append(finding.finding_id)
|
||||
return {path: tuple(ids) for path, ids in grouped.items()}
|
||||
@@ -0,0 +1,232 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from collections.abc import Iterable, Mapping, Sequence
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path, PurePosixPath
|
||||
|
||||
from ..findings import StaticFinding
|
||||
from ..preprocess import FileTarget
|
||||
|
||||
# Inventory density is a recorded feature and a capped rationale term, not the rank driver.
|
||||
_LOC_COEFF = 0.85
|
||||
_NESTING_COEFF = 0.2
|
||||
_REACHABILITY_COEFF = 0.55
|
||||
_TEST_GAP_COEFF = 0.35
|
||||
_INVENTORY_COEFF = 0.12
|
||||
_TAG_WEIGHTS: dict[str, float] = {
|
||||
"parser": 4.0,
|
||||
"deserialization": 3.5,
|
||||
"network_entry": 1.75,
|
||||
"auth_boundary": 1.5,
|
||||
"fuzzable": 1.5,
|
||||
"memory_unsafe": 1.25,
|
||||
"crypto": 1.25,
|
||||
"syscall_entry": 1.25,
|
||||
"filesystem_entry": 1.0,
|
||||
}
|
||||
_TEST_ROOT_PARTS = frozenset({"tests", "test", "__tests__", "spec"})
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ComplexitySignals:
|
||||
path: str
|
||||
function_name: str | None
|
||||
loc: int
|
||||
nesting: int
|
||||
tags: tuple[str, ...]
|
||||
reachability: int
|
||||
inventory_hit_count: int
|
||||
has_adjacent_test: bool
|
||||
score: float
|
||||
|
||||
|
||||
def collect_signals(
|
||||
targets: Sequence[FileTarget],
|
||||
findings: Sequence[StaticFinding] = (),
|
||||
repo_files: Iterable[str] = (),
|
||||
sources: Mapping[str, str] | None = None,
|
||||
) -> tuple[ComplexitySignals, ...]:
|
||||
inventory = _inventory_counts(findings)
|
||||
repo_paths = {str(path).replace("\\", "/") for path in repo_files}
|
||||
collected: list[ComplexitySignals] = []
|
||||
for target in targets:
|
||||
text = _source_text(target, sources)
|
||||
nesting = measure_nesting(text)
|
||||
hit_count = inventory.get(target.path, 0)
|
||||
adjacent = _has_adjacent_test(target.path, repo_paths)
|
||||
tags = tuple(target.tags)
|
||||
names = _function_names(target.reachability_hints)
|
||||
identities: tuple[str | None, ...] = names or (None,)
|
||||
for function_name in identities:
|
||||
reachability = _reachability(target.reachability_hints, function_name)
|
||||
collected.append(
|
||||
ComplexitySignals(
|
||||
path=target.path,
|
||||
function_name=function_name,
|
||||
loc=target.loc,
|
||||
nesting=nesting,
|
||||
tags=tags,
|
||||
reachability=reachability,
|
||||
inventory_hit_count=hit_count,
|
||||
has_adjacent_test=adjacent,
|
||||
score=_score(
|
||||
loc=target.loc,
|
||||
nesting=nesting,
|
||||
tags=tags,
|
||||
reachability=reachability,
|
||||
inventory_hit_count=hit_count,
|
||||
has_adjacent_test=adjacent,
|
||||
),
|
||||
)
|
||||
)
|
||||
return tuple(sorted(collected, key=lambda item: (-item.score, item.path, item.function_name or "")))
|
||||
|
||||
|
||||
def measure_nesting(text: str) -> int:
|
||||
if not text:
|
||||
return 0
|
||||
max_indent = 0
|
||||
max_braces = 0
|
||||
braces = 0
|
||||
for raw in text.splitlines():
|
||||
if not raw.strip():
|
||||
continue
|
||||
expanded = raw.expandtabs(4)
|
||||
indent = len(expanded) - len(expanded.lstrip(" "))
|
||||
max_indent = max(max_indent, indent // 4)
|
||||
for char in raw:
|
||||
if char == "{":
|
||||
braces += 1
|
||||
max_braces = max(max_braces, braces)
|
||||
elif char == "}":
|
||||
braces = max(0, braces - 1)
|
||||
return max(max_indent, max_braces)
|
||||
|
||||
|
||||
def _score(
|
||||
*,
|
||||
loc: int,
|
||||
nesting: int,
|
||||
tags: Sequence[str],
|
||||
reachability: int,
|
||||
inventory_hit_count: int,
|
||||
has_adjacent_test: bool,
|
||||
) -> float:
|
||||
loc_term = math.log1p(max(loc, 0)) * _LOC_COEFF
|
||||
nest_term = min(max(nesting, 0), 16) * _NESTING_COEFF
|
||||
tag_term = sum(_TAG_WEIGHTS.get(tag, 0.0) for tag in dict.fromkeys(tags))
|
||||
reach_term = max(reachability, 0) * _REACHABILITY_COEFF
|
||||
gap_term = 0.0 if has_adjacent_test else _TEST_GAP_COEFF
|
||||
density = inventory_hit_count / max(loc, 1)
|
||||
inventory_term = min(max(density, 0.0), 1.0) * _INVENTORY_COEFF
|
||||
return round(loc_term + nest_term + tag_term + reach_term + gap_term + inventory_term, 4)
|
||||
|
||||
|
||||
def _source_text(target: FileTarget, sources: Mapping[str, str] | None) -> str:
|
||||
if sources:
|
||||
if target.path in sources:
|
||||
return sources[target.path]
|
||||
if target.absolute_path in sources:
|
||||
return sources[target.absolute_path]
|
||||
path = Path(target.absolute_path)
|
||||
if not path.is_file():
|
||||
return ""
|
||||
try:
|
||||
return path.read_text(errors="ignore")
|
||||
except OSError:
|
||||
return ""
|
||||
|
||||
|
||||
def _inventory_counts(findings: Sequence[StaticFinding]) -> dict[str, int]:
|
||||
counts: dict[str, int] = {}
|
||||
for finding in findings:
|
||||
counts[finding.path] = counts.get(finding.path, 0) + 1
|
||||
return counts
|
||||
|
||||
|
||||
def _function_names(hints: Sequence[Mapping[str, object]]) -> tuple[str, ...]:
|
||||
names: list[str] = []
|
||||
seen: set[str] = set()
|
||||
for hint in hints:
|
||||
name = _hint_function_name(hint)
|
||||
if name is not None and name not in seen:
|
||||
names.append(name)
|
||||
seen.add(name)
|
||||
return tuple(names)
|
||||
|
||||
|
||||
def _hint_function_name(hint: Mapping[str, object]) -> str | None:
|
||||
if "function_name" in hint:
|
||||
value = hint.get("function_name")
|
||||
return value if isinstance(value, str) and value else None
|
||||
value = hint.get("name")
|
||||
return value if isinstance(value, str) and value else None
|
||||
|
||||
|
||||
def _reachability(hints: Sequence[Mapping[str, object]], function_name: str | None) -> int:
|
||||
selected: Sequence[Mapping[str, object]] = hints
|
||||
if function_name is not None:
|
||||
matching = [hint for hint in hints if _hint_function_name(hint) == function_name]
|
||||
if matching:
|
||||
selected = matching
|
||||
if not selected:
|
||||
return 0
|
||||
score = 2
|
||||
for hint in selected:
|
||||
access_level = hint.get("access_level")
|
||||
kind = hint.get("kind")
|
||||
if access_level == "public":
|
||||
score = max(score, 5 if kind in {"route", "parser", "fuzz"} else 4)
|
||||
elif access_level in {"authenticated", "contract-only/callback"}:
|
||||
score = max(score, 4)
|
||||
elif access_level == "role-restricted":
|
||||
score = max(score, 3)
|
||||
else:
|
||||
score = max(score, 2)
|
||||
return score
|
||||
|
||||
|
||||
def _has_adjacent_test(path: str, repo_files: set[str]) -> bool:
|
||||
source = PurePosixPath(path)
|
||||
source_stem = source.stem.lower()
|
||||
source_parent = _parent(source)
|
||||
for candidate in repo_files:
|
||||
name = PurePosixPath(candidate).name
|
||||
if not _is_test_filename(name):
|
||||
continue
|
||||
if _tested_stem(name).lower() != source_stem:
|
||||
continue
|
||||
parent = _parent(PurePosixPath(candidate))
|
||||
if parent == source_parent or _under_common_test_root(candidate):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _parent(path: PurePosixPath) -> str:
|
||||
parent = path.parent.as_posix()
|
||||
return "" if parent == "." else parent
|
||||
|
||||
|
||||
def _is_test_filename(name: str) -> bool:
|
||||
if name.startswith("test_") or name.endswith("Test.java"):
|
||||
return True
|
||||
return PurePosixPath(name).stem.endswith("_test")
|
||||
|
||||
|
||||
def _tested_stem(filename: str) -> str:
|
||||
if filename.endswith("Test.java"):
|
||||
return filename[: -len("Test.java")]
|
||||
stem = PurePosixPath(filename).stem
|
||||
if stem.startswith("test_"):
|
||||
return stem[len("test_") :]
|
||||
if stem.endswith("_test"):
|
||||
return stem[: -len("_test")]
|
||||
return stem
|
||||
|
||||
|
||||
def _under_common_test_root(path: str) -> bool:
|
||||
parts = PurePosixPath(path).parts
|
||||
if len(parts) >= 2 and parts[0] == "src" and parts[1] == "test":
|
||||
return True
|
||||
return any(part in _TEST_ROOT_PARTS for part in parts[:-1])
|
||||
@@ -96,6 +96,20 @@ class FusionConfig:
|
||||
high_assurance: bool = False
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ComplexityConfig:
|
||||
# Stage-2 map size and hunter attention. Small defaults keep PR/nightly map work cheap.
|
||||
top_k: int = 20
|
||||
max_hunter_hotspots: int = 8
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RegressConfig:
|
||||
# Stage-3 candidate cap and optional per-language sandbox image pins. Isolation limits stay on SandboxConfig.
|
||||
max_candidates: int = 5
|
||||
images: tuple[tuple[str, str], ...] = ()
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ResolvedConfig:
|
||||
models: ModelConfig = ModelConfig()
|
||||
@@ -109,6 +123,8 @@ class ResolvedConfig:
|
||||
harnessx: HarnessxConfig = HarnessxConfig()
|
||||
fusion: FusionConfig = FusionConfig()
|
||||
hardening: HardeningConfig = HardeningConfig()
|
||||
complexity: ComplexityConfig = ComplexityConfig()
|
||||
regress: RegressConfig = RegressConfig()
|
||||
runs_dir: str = ".openultrasast/runs"
|
||||
|
||||
|
||||
@@ -130,6 +146,8 @@ def load_config(config_path: Path | None = None) -> ResolvedConfig:
|
||||
harnessx=_load_harnessx(data.get("harnessx", {})),
|
||||
fusion=_load_fusion(data.get("fusion", {})),
|
||||
hardening=_load_hardening(data.get("hardening", {})),
|
||||
complexity=_load_complexity(data.get("complexity", {})),
|
||||
regress=_load_regress(data.get("regress", {})),
|
||||
runs_dir=os.environ.get("OPENULTRASAST_RUNS_DIR", ".openultrasast/runs"),
|
||||
)
|
||||
|
||||
@@ -251,6 +269,32 @@ def _load_fusion(value: object) -> FusionConfig:
|
||||
)
|
||||
|
||||
|
||||
def _load_complexity(value: object) -> ComplexityConfig:
|
||||
data = _section(value)
|
||||
return ComplexityConfig(
|
||||
top_k=_int_value(data.get("top_k"), 20),
|
||||
max_hunter_hotspots=_int_value(data.get("max_hunter_hotspots"), 8),
|
||||
)
|
||||
|
||||
|
||||
def _load_regress(value: object) -> RegressConfig:
|
||||
data = _section(value)
|
||||
return RegressConfig(
|
||||
max_candidates=_int_value(data.get("max_candidates"), 5),
|
||||
images=_string_map(data.get("images")),
|
||||
)
|
||||
|
||||
|
||||
def _string_map(value: object) -> tuple[tuple[str, str], ...]:
|
||||
data = _section(value)
|
||||
images: list[tuple[str, str]] = []
|
||||
for key, item in data.items():
|
||||
pinned = _string(item)
|
||||
if pinned is not None:
|
||||
images.append((str(key), pinned))
|
||||
return tuple(sorted(images))
|
||||
|
||||
|
||||
def _float_value(value: object, default: float) -> float:
|
||||
if isinstance(value, bool):
|
||||
return default
|
||||
|
||||
@@ -28,6 +28,7 @@ class StaticFinding:
|
||||
tags: list[str]
|
||||
ranking_priority: float
|
||||
status: str = "enabled"
|
||||
proposed_snippet: str | None = None
|
||||
|
||||
|
||||
SEVERITY_LABEL = {5: "critical", 4: "high", 3: "medium", 2: "low", 1: "low", 0: "info"}
|
||||
|
||||
@@ -0,0 +1,130 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
from collections.abc import Iterable, Iterator, Mapping
|
||||
from pathlib import Path
|
||||
|
||||
from .preprocess import enumerate_source_files
|
||||
|
||||
|
||||
class PathEscapesRepo(ValueError):
|
||||
"""Raised when a resolved tool path is not under the scan root."""
|
||||
|
||||
|
||||
def clamp_repo_path(root: Path, user_path: str) -> Path:
|
||||
resolved_root = root.resolve()
|
||||
candidate = Path(user_path)
|
||||
resolved = candidate.resolve() if candidate.is_absolute() else (resolved_root / candidate).resolve()
|
||||
try:
|
||||
common = os.path.commonpath([str(resolved_root), str(resolved)])
|
||||
except ValueError as exc:
|
||||
raise PathEscapesRepo(f"path {user_path!r} escapes scan root {resolved_root}") from exc
|
||||
if common != str(resolved_root):
|
||||
raise PathEscapesRepo(f"path {user_path!r} escapes scan root {resolved_root}")
|
||||
return resolved
|
||||
|
||||
|
||||
def read_file(root: Path, path: str, *, max_chars: int) -> str:
|
||||
clamped = clamp_repo_path(root, path)
|
||||
if max_chars <= 0:
|
||||
return ""
|
||||
return clamped.read_text(errors="ignore")[:max_chars]
|
||||
|
||||
|
||||
def grep_repo(root: Path, pattern: str, *, max_matches: int) -> list[dict[str, object]]:
|
||||
if max_matches <= 0:
|
||||
return []
|
||||
compiled = re.compile(pattern)
|
||||
matches: list[dict[str, object]] = []
|
||||
for relative, path in _iter_clamped_source_files(root):
|
||||
for line_no, line in enumerate(path.read_text(errors="ignore").splitlines(), start=1):
|
||||
if compiled.search(line):
|
||||
matches.append({"path": relative, "line": line_no, "text": line})
|
||||
if len(matches) >= max_matches:
|
||||
return matches
|
||||
return matches
|
||||
|
||||
|
||||
def find_refs(root: Path, symbol: str, mapping_index: object) -> list[dict[str, object]]:
|
||||
resolved_root = root.resolve()
|
||||
refs: list[dict[str, object]] = []
|
||||
for record in _iter_mapping_records(mapping_index):
|
||||
if not _record_matches_symbol(record, symbol):
|
||||
continue
|
||||
raw_path = _record_path(record)
|
||||
if raw_path is None:
|
||||
continue
|
||||
clamped = clamp_repo_path(resolved_root, raw_path)
|
||||
name = _record_value(record, "name", "function_name", "symbol")
|
||||
refs.append(
|
||||
{
|
||||
"path": clamped.relative_to(resolved_root).as_posix(),
|
||||
"name": str(name) if name is not None else symbol,
|
||||
"line": _as_line(_record_value(record, "line")),
|
||||
"source": "mapping",
|
||||
}
|
||||
)
|
||||
if symbol:
|
||||
compiled = re.compile(rf"\b{re.escape(symbol)}\b")
|
||||
for relative, path in _iter_clamped_source_files(resolved_root):
|
||||
for line_no, line in enumerate(path.read_text(errors="ignore").splitlines(), start=1):
|
||||
if compiled.search(line):
|
||||
refs.append({"path": relative, "name": symbol, "line": line_no, "text": line, "source": "text"})
|
||||
return refs
|
||||
|
||||
|
||||
def _iter_clamped_source_files(root: Path) -> Iterator[tuple[str, Path]]:
|
||||
resolved_root = root.resolve()
|
||||
for path in enumerate_source_files(resolved_root):
|
||||
relative = path.relative_to(resolved_root).as_posix()
|
||||
yield relative, clamp_repo_path(resolved_root, relative)
|
||||
|
||||
|
||||
def _iter_mapping_records(mapping_index: object) -> Iterable[object]:
|
||||
if mapping_index is None:
|
||||
return ()
|
||||
if isinstance(mapping_index, Mapping):
|
||||
return (mapping_index,)
|
||||
if isinstance(mapping_index, (str, bytes, bytearray)):
|
||||
return ()
|
||||
if isinstance(mapping_index, Iterable):
|
||||
return mapping_index
|
||||
return (mapping_index,)
|
||||
|
||||
|
||||
def _record_matches_symbol(record: object, symbol: str) -> bool:
|
||||
for field in ("name", "function_name", "symbol"):
|
||||
value = _record_value(record, field)
|
||||
if value is not None and str(value) == symbol:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _record_path(record: object) -> str | None:
|
||||
value = record.get("path") if isinstance(record, Mapping) else getattr(record, "path", None)
|
||||
if value is None:
|
||||
return None
|
||||
return str(value)
|
||||
|
||||
|
||||
def _record_value(record: object, *names: str) -> object | None:
|
||||
for name in names:
|
||||
raw: object | None
|
||||
if isinstance(record, Mapping):
|
||||
if name not in record:
|
||||
continue
|
||||
raw = record[name]
|
||||
else:
|
||||
raw = getattr(record, name, None)
|
||||
if raw is not None:
|
||||
return raw
|
||||
return None
|
||||
|
||||
|
||||
def _as_line(value: object) -> int | None:
|
||||
if isinstance(value, bool) or value is None:
|
||||
return None
|
||||
if isinstance(value, int):
|
||||
return value
|
||||
return None
|
||||
@@ -0,0 +1,130 @@
|
||||
"""Stage-2 quality gate: split-sink ranking must beat pattern-hit-count order.
|
||||
|
||||
The stage-1 smoke gate (``python -m openultrasast.gate``) stays on cheat-sheet
|
||||
fixtures. This module is the merge gate for the complexity map: the planted
|
||||
split-sink function must sit in the high band, and map order must not collapse
|
||||
to sorting files by inventory-hit count.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
import tempfile
|
||||
from collections.abc import Sequence
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
from .benchmark import BenchmarkManifest, ExpectedFinding, load_benchmark_manifest, resolve_benchmark_source
|
||||
from .complexity.map import ComplexityMap, Hotspot, build_complexity_map
|
||||
from .findings import StaticFinding, quick_scan_findings
|
||||
from .mapping import analyze_entry_points, attach_reachability_hints
|
||||
from .preprocess import preprocess_repository
|
||||
from .rank import rank_targets
|
||||
|
||||
SPLIT_SINK_MANIFESTS = (
|
||||
"split-sink-c",
|
||||
"split-sink-java",
|
||||
"split-sink-javascript",
|
||||
"split-sink-python",
|
||||
)
|
||||
DEFAULT_MANIFEST_DIR = Path("benchmarks/manifests")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SplitSinkScan:
|
||||
name: str
|
||||
manifest: BenchmarkManifest
|
||||
target: Path
|
||||
findings: tuple[StaticFinding, ...]
|
||||
complexity_map: ComplexityMap
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class MapGateVerdict:
|
||||
passed: bool
|
||||
reasons: tuple[str, ...]
|
||||
|
||||
|
||||
def scan_split_sink(name: str, manifest_dir: Path = DEFAULT_MANIFEST_DIR) -> SplitSinkScan:
|
||||
manifest_path = manifest_dir / f"{name}.toml"
|
||||
manifest = load_benchmark_manifest(manifest_path)
|
||||
target = resolve_benchmark_source(manifest_path, manifest)
|
||||
_, targets = preprocess_repository(target)
|
||||
targets = attach_reachability_hints(targets, analyze_entry_points(target, targets))
|
||||
findings = tuple(quick_scan_findings(target, targets, rank_targets(targets)))
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
complexity_map = build_complexity_map(
|
||||
targets,
|
||||
findings,
|
||||
Path(tmp) / "complexity_map.json",
|
||||
repo_files=[item.path for item in targets],
|
||||
)
|
||||
return SplitSinkScan(name=name, manifest=manifest, target=target, findings=findings, complexity_map=complexity_map)
|
||||
|
||||
|
||||
def planted_expected(manifest: BenchmarkManifest) -> tuple[ExpectedFinding, ...]:
|
||||
return tuple(item for item in manifest.expected if item.rule_id is None)
|
||||
|
||||
|
||||
_BAND_RANK = {"high": 3, "medium": 2, "low": 1}
|
||||
|
||||
|
||||
def planted_in_high_band(manifest: BenchmarkManifest, complexity_map: ComplexityMap) -> bool:
|
||||
"""True when the planted sink sits in the highest band the map actually emitted."""
|
||||
if not complexity_map.hotspots:
|
||||
return False
|
||||
top = max(_BAND_RANK.get(hotspot.band, 0) for hotspot in complexity_map.hotspots)
|
||||
for expected in planted_expected(manifest):
|
||||
matches = [hotspot for hotspot in complexity_map.hotspots if hotspot.path == expected.path]
|
||||
if expected.function:
|
||||
named = [hotspot for hotspot in matches if hotspot.function_name == expected.function]
|
||||
if named:
|
||||
matches = named
|
||||
if not matches or not any(_BAND_RANK.get(hotspot.band, 0) == top for hotspot in matches):
|
||||
return False
|
||||
return bool(planted_expected(manifest))
|
||||
|
||||
|
||||
def hit_count_path_order(findings: Sequence[StaticFinding], paths: Sequence[str]) -> list[str]:
|
||||
counts: dict[str, int] = {}
|
||||
for finding in findings:
|
||||
counts[finding.path] = counts.get(finding.path, 0) + 1
|
||||
unique = list(dict.fromkeys(paths))
|
||||
return sorted(unique, key=lambda path: (-counts.get(path, 0), path))
|
||||
|
||||
|
||||
def map_beats_hit_count(hotspots: Sequence[Hotspot], findings: Sequence[StaticFinding]) -> bool:
|
||||
map_order = list(dict.fromkeys(hotspot.path for hotspot in hotspots))
|
||||
stub_order = hit_count_path_order(findings, map_order)
|
||||
return map_order != stub_order
|
||||
|
||||
|
||||
def split_sink_map_gate(manifest_dir: Path = DEFAULT_MANIFEST_DIR) -> MapGateVerdict:
|
||||
reasons: list[str] = []
|
||||
python: SplitSinkScan | None = None
|
||||
for name in SPLIT_SINK_MANIFESTS:
|
||||
bundle = scan_split_sink(name, manifest_dir)
|
||||
if name == "split-sink-python":
|
||||
python = bundle
|
||||
if not planted_in_high_band(bundle.manifest, bundle.complexity_map):
|
||||
reasons.append(f"{name}: planted split-sink is not in the top hotspot band")
|
||||
if python is None:
|
||||
reasons.append("split-sink-python corpus is missing")
|
||||
elif not map_beats_hit_count(python.complexity_map.hotspots, python.findings):
|
||||
reasons.append("split-sink-python: map order equals pattern-hit-count order")
|
||||
return MapGateVerdict(passed=not reasons, reasons=tuple(reasons))
|
||||
|
||||
|
||||
def main() -> int:
|
||||
verdict = split_sink_map_gate()
|
||||
if verdict.passed:
|
||||
print("split-sink map gate: PASS")
|
||||
return 0
|
||||
print("split-sink map gate: FAIL", file=sys.stderr)
|
||||
for reason in verdict.reasons:
|
||||
print(f" - {reason}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -6,7 +6,7 @@ import re
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from collections.abc import Callable
|
||||
from collections.abc import Callable, Mapping, Sequence
|
||||
from dataclasses import dataclass
|
||||
from typing import TypeVar, cast
|
||||
|
||||
@@ -61,8 +61,18 @@ class OpenRouterChatClient:
|
||||
raise OpenRouterError("OPENROUTER_API_KEY is required for OpenRouter chat calls")
|
||||
return cls(api_key=api_key, base_url=os.environ.get("OPENROUTER_BASE_URL", cls.base_url))
|
||||
|
||||
def complete_json(self, *, model: str, messages: list[dict[str, str]], timeout_seconds: int = 60) -> object:
|
||||
payload = json.dumps({"model": model, "messages": messages, "temperature": 0}).encode()
|
||||
def complete_chat(
|
||||
self,
|
||||
*,
|
||||
model: str,
|
||||
messages: Sequence[Mapping[str, object]],
|
||||
tools: Sequence[Mapping[str, object]] | None = None,
|
||||
timeout_seconds: int = 60,
|
||||
) -> dict[str, object]:
|
||||
body: dict[str, object] = {"model": model, "messages": list(messages), "temperature": 0}
|
||||
if tools:
|
||||
body["tools"] = list(tools)
|
||||
payload = json.dumps(body).encode()
|
||||
request = urllib.request.Request(
|
||||
f"{self.base_url.rstrip('/')}/chat/completions",
|
||||
data=payload,
|
||||
@@ -81,8 +91,13 @@ class OpenRouterChatClient:
|
||||
response_payload = call_with_retry(_do, attempts=self.max_attempts, base_delay=self.retry_base_delay)
|
||||
except (urllib.error.URLError, TimeoutError, json.JSONDecodeError) as exc:
|
||||
raise OpenRouterError(f"OpenRouter chat request failed: {exc}") from exc
|
||||
return _extract_message(response_payload)
|
||||
|
||||
content = _extract_message_content(response_payload)
|
||||
def complete_json(self, *, model: str, messages: list[dict[str, str]], timeout_seconds: int = 60) -> object:
|
||||
message = self.complete_chat(model=model, messages=messages, timeout_seconds=timeout_seconds)
|
||||
content = message.get("content")
|
||||
if not isinstance(content, str) or not content.strip():
|
||||
raise OpenRouterError("OpenRouter message content was empty")
|
||||
return parse_json_content(content)
|
||||
|
||||
|
||||
@@ -156,7 +171,7 @@ def parse_json_content(content: str) -> object:
|
||||
raise OpenRouterError(f"model response was not valid JSON: {exc}") from exc
|
||||
|
||||
|
||||
def _extract_message_content(payload: object) -> str:
|
||||
def _extract_message(payload: object) -> dict[str, object]:
|
||||
if not isinstance(payload, dict):
|
||||
raise OpenRouterError("OpenRouter response must be a JSON object")
|
||||
choices = payload.get("choices")
|
||||
@@ -168,7 +183,11 @@ def _extract_message_content(payload: object) -> str:
|
||||
message = first.get("message")
|
||||
if not isinstance(message, dict):
|
||||
raise OpenRouterError("OpenRouter choice did not include a message")
|
||||
content = message.get("content")
|
||||
return message
|
||||
|
||||
|
||||
def _extract_message_content(payload: object) -> str:
|
||||
content = _extract_message(payload).get("content")
|
||||
if not isinstance(content, str) or not content.strip():
|
||||
raise OpenRouterError("OpenRouter message content was empty")
|
||||
return content
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
"""Isolated regression snippets and verdicts."""
|
||||
|
||||
from .candidate import CandidateVerdict, RegressionRunner, run_regression, select_candidates, write_verdicts
|
||||
from .recipes import recipe_for
|
||||
from .safety import UnsafeSnippetError, check_snippet_safety
|
||||
from .verdict import (
|
||||
ALREADY_COVERED,
|
||||
INCONCLUSIVE,
|
||||
MISSING_RECIPE,
|
||||
NOT_TRIGGERABLE,
|
||||
SAFETY_REJECTED,
|
||||
TRIGGERABLE,
|
||||
RegressionVerdict,
|
||||
is_worth_fixing,
|
||||
verdict_from_result,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"ALREADY_COVERED",
|
||||
"INCONCLUSIVE",
|
||||
"MISSING_RECIPE",
|
||||
"NOT_TRIGGERABLE",
|
||||
"SAFETY_REJECTED",
|
||||
"TRIGGERABLE",
|
||||
"CandidateVerdict",
|
||||
"RegressionRunner",
|
||||
"RegressionVerdict",
|
||||
"UnsafeSnippetError",
|
||||
"check_snippet_safety",
|
||||
"is_worth_fixing",
|
||||
"recipe_for",
|
||||
"run_regression",
|
||||
"select_candidates",
|
||||
"verdict_from_result",
|
||||
"write_verdicts",
|
||||
]
|
||||
@@ -0,0 +1,247 @@
|
||||
"""Candidate selection and safety-gated regression execution."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections.abc import Mapping, Sequence
|
||||
from dataclasses import asdict, dataclass, replace
|
||||
from pathlib import Path
|
||||
|
||||
from ..complexity.ledger import hotspot_key, select_forced_candidates
|
||||
from ..complexity.map import Hotspot
|
||||
from ..config import SandboxConfig
|
||||
from ..findings import StaticFinding
|
||||
from ..preprocess import LANGUAGE_BY_EXTENSION
|
||||
from ..sandbox import SandboxJob, SandboxResult, SandboxRunner
|
||||
from ..sandbox.runner import WORKSPACE_MOUNT
|
||||
from .recipes import recipe_for
|
||||
from .safety import UnsafeSnippetError, check_snippet_safety
|
||||
from .verdict import (
|
||||
INCONCLUSIVE,
|
||||
MISSING_RECIPE,
|
||||
NOT_TRIGGERABLE,
|
||||
SAFETY_REJECTED,
|
||||
TRIGGERABLE,
|
||||
RegressionVerdict,
|
||||
is_worth_fixing,
|
||||
verdict_from_result,
|
||||
)
|
||||
|
||||
DEFAULT_IMAGES = {
|
||||
"python": "python:3.12-alpine",
|
||||
"javascript": "node:22-alpine",
|
||||
"c": "gcc:13",
|
||||
}
|
||||
|
||||
__all__ = [
|
||||
"INCONCLUSIVE",
|
||||
"MISSING_RECIPE",
|
||||
"NOT_TRIGGERABLE",
|
||||
"SAFETY_REJECTED",
|
||||
"TRIGGERABLE",
|
||||
"CandidateVerdict",
|
||||
"RegressionRunner",
|
||||
"RegressionVerdict",
|
||||
"run_regression",
|
||||
"select_candidates",
|
||||
"snippet_for",
|
||||
"write_verdicts",
|
||||
]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CandidateVerdict:
|
||||
path: str
|
||||
function_name: str | None
|
||||
language: str
|
||||
verdict: str
|
||||
reason: str
|
||||
inventory_finding_ids: tuple[str, ...]
|
||||
worth_fixing: bool = False
|
||||
|
||||
|
||||
def select_candidates(
|
||||
hotspots: Sequence[Hotspot],
|
||||
findings: Sequence[StaticFinding],
|
||||
*,
|
||||
max_candidates: int,
|
||||
policy_severity_by_id: Mapping[str, object] | None = None,
|
||||
rule_cwe: Mapping[str, str] | None = None,
|
||||
) -> tuple[Hotspot, ...]:
|
||||
"""Select min(max_candidates, ranked hotspots) ∪ sev-5 reachable inventory.
|
||||
|
||||
The cap applies to the ranked hotspot slice. Forced policy-severity-5
|
||||
reachable (or inferred-file-surface) inventory stays in the set even when a
|
||||
ledger overlay demoted its hotspot out of that slice.
|
||||
"""
|
||||
ranked = tuple(sorted(hotspots, key=lambda item: (-item.score, item.path, item.function_name or "")))
|
||||
selected = list(ranked[: max(max_candidates, 0)])
|
||||
seen = {hotspot_key(item.path, item.function_name) for item in selected}
|
||||
extras: list[Hotspot] = []
|
||||
for finding in select_forced_candidates(findings, policy_severity_by_id or {}, rule_cwe=rule_cwe):
|
||||
hotspot = _hotspot_for_finding(finding, ranked) or _hotspot_from_forced_finding(finding)
|
||||
key = hotspot_key(hotspot.path, hotspot.function_name)
|
||||
if key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
extras.append(hotspot)
|
||||
return tuple(selected + extras)
|
||||
|
||||
|
||||
def _hotspot_for_finding(finding: StaticFinding, hotspots: Sequence[Hotspot]) -> Hotspot | None:
|
||||
for hotspot in hotspots:
|
||||
if finding.finding_id in hotspot.inventory_finding_ids:
|
||||
return hotspot
|
||||
key = hotspot_key(finding.path, finding.function_name)
|
||||
for hotspot in hotspots:
|
||||
if hotspot_key(hotspot.path, hotspot.function_name) == key:
|
||||
return hotspot
|
||||
return None
|
||||
|
||||
|
||||
def _hotspot_from_forced_finding(finding: StaticFinding) -> Hotspot:
|
||||
return Hotspot(
|
||||
path=finding.path,
|
||||
function_name=finding.function_name,
|
||||
score=0.0,
|
||||
band="low",
|
||||
signals={},
|
||||
rationale="forced reachable severity-5 inventory",
|
||||
test_hint=None,
|
||||
inventory_finding_ids=(finding.finding_id,),
|
||||
)
|
||||
|
||||
|
||||
class RegressionRunner:
|
||||
"""Run a snippet in the sandbox only after the structural safety check passes."""
|
||||
|
||||
def __init__(self, sandbox: SandboxRunner) -> None:
|
||||
self._sandbox = sandbox
|
||||
|
||||
def run_recipe(
|
||||
self,
|
||||
language: str,
|
||||
snippet: str,
|
||||
image: str,
|
||||
sandbox_limits: SandboxConfig,
|
||||
*,
|
||||
repo_root: Path,
|
||||
) -> RegressionVerdict:
|
||||
job = recipe_for(language, snippet, image, sandbox_limits, repo_root=repo_root)
|
||||
if job is None:
|
||||
return verdict_from_result(recipe_missing=True)
|
||||
return self.run_snippet(snippet, job)
|
||||
|
||||
def run_snippet(self, snippet: str, job: SandboxJob | None) -> RegressionVerdict:
|
||||
if job is None:
|
||||
return verdict_from_result(recipe_missing=True)
|
||||
try:
|
||||
check_snippet_safety(snippet)
|
||||
except UnsafeSnippetError:
|
||||
return verdict_from_result(safety_rejected=True)
|
||||
return _verdict_from_result(self._sandbox.run(job))
|
||||
|
||||
|
||||
def _verdict_from_result(result: SandboxResult) -> RegressionVerdict:
|
||||
return verdict_from_result(result)
|
||||
|
||||
|
||||
def snippet_for(language: str, path: str, function_name: str | None) -> str:
|
||||
"""Render a scratch snippet that loads the candidate source inside the sandbox."""
|
||||
relative = path.replace("\\", "/")
|
||||
workspace_path = f"{WORKSPACE_MOUNT}/{relative}"
|
||||
identity = function_name or relative
|
||||
if language == "python":
|
||||
return (
|
||||
f"# candidate {identity}\n"
|
||||
"from pathlib import Path\n"
|
||||
f"source = Path({workspace_path!r}).read_text()\n"
|
||||
f"compile(source, {relative!r}, 'exec')\n"
|
||||
)
|
||||
if language == "javascript":
|
||||
return (
|
||||
f"// candidate {identity}\n"
|
||||
"const fs = require('fs');\n"
|
||||
f"const source = fs.readFileSync({workspace_path!r}, 'utf8');\n"
|
||||
"if (!source.length) { process.exit(1); }\n"
|
||||
)
|
||||
if language == "c":
|
||||
return f"/* candidate {identity} */\nint main(void) {{ return 0; }}\n"
|
||||
return ""
|
||||
|
||||
|
||||
def run_regression(
|
||||
hotspots: Sequence[Hotspot],
|
||||
findings: Sequence[StaticFinding],
|
||||
*,
|
||||
max_candidates: int,
|
||||
policy: Mapping[str, object],
|
||||
rule_cwe: Mapping[str, str],
|
||||
languages_by_path: Mapping[str, str],
|
||||
repo_root: Path,
|
||||
sandbox: SandboxRunner,
|
||||
sandbox_limits: SandboxConfig,
|
||||
images: Mapping[str, str],
|
||||
) -> tuple[CandidateVerdict, ...]:
|
||||
"""Select candidates under the cap, run recipes in the sandbox, and return verdicts."""
|
||||
selected = select_candidates(
|
||||
hotspots,
|
||||
findings,
|
||||
max_candidates=max_candidates,
|
||||
policy_severity_by_id=policy,
|
||||
rule_cwe=rule_cwe,
|
||||
)
|
||||
runner = RegressionRunner(sandbox)
|
||||
findings_by_id = {finding.finding_id: finding for finding in findings}
|
||||
records: list[CandidateVerdict] = []
|
||||
for hotspot in selected:
|
||||
language = _language_for(hotspot.path, languages_by_path)
|
||||
snippet = _proposed_snippet(hotspot, findings) or snippet_for(language, hotspot.path, hotspot.function_name)
|
||||
image = images.get(language) or DEFAULT_IMAGES.get(language, "ousast-missing-image")
|
||||
mapped = runner.run_recipe(language, snippet, image, sandbox_limits, repo_root=repo_root)
|
||||
reachability = _reachability_for(hotspot, findings_by_id)
|
||||
mapped = replace(mapped, worth_fixing=is_worth_fixing(mapped.verdict, reachability))
|
||||
records.append(
|
||||
CandidateVerdict(
|
||||
path=hotspot.path,
|
||||
function_name=hotspot.function_name,
|
||||
language=language,
|
||||
verdict=mapped.verdict,
|
||||
reason=mapped.reason,
|
||||
inventory_finding_ids=hotspot.inventory_finding_ids,
|
||||
worth_fixing=mapped.worth_fixing,
|
||||
)
|
||||
)
|
||||
return tuple(records)
|
||||
|
||||
|
||||
def write_verdicts(verdicts: Sequence[CandidateVerdict], path: Path) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
payload = {"verdicts": [asdict(item) for item in verdicts]}
|
||||
path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
|
||||
|
||||
|
||||
def _proposed_snippet(hotspot: Hotspot, findings: Sequence[StaticFinding]) -> str | None:
|
||||
for finding in findings:
|
||||
snippet = finding.proposed_snippet
|
||||
if not snippet:
|
||||
continue
|
||||
if finding.path == hotspot.path or finding.finding_id in hotspot.inventory_finding_ids:
|
||||
return snippet
|
||||
return None
|
||||
|
||||
|
||||
def _language_for(path: str, languages_by_path: Mapping[str, str]) -> str:
|
||||
if path in languages_by_path:
|
||||
return languages_by_path[path]
|
||||
return LANGUAGE_BY_EXTENSION.get(Path(path).suffix.lower(), "unknown")
|
||||
|
||||
|
||||
def _reachability_for(hotspot: Hotspot, findings_by_id: Mapping[str, StaticFinding]) -> str:
|
||||
statuses = [
|
||||
findings_by_id[finding_id].reachability_status for finding_id in hotspot.inventory_finding_ids if finding_id in findings_by_id
|
||||
]
|
||||
for status in statuses:
|
||||
if is_worth_fixing(TRIGGERABLE, status):
|
||||
return status
|
||||
return statuses[0] if statuses else "unknown"
|
||||
@@ -0,0 +1,71 @@
|
||||
"""Language command recipes for isolated regression snippets."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from ..config import SandboxConfig
|
||||
from ..sandbox.runner import SCRATCH_MOUNT, WORKSPACE_MOUNT, SandboxJob
|
||||
|
||||
_PYTHON_FILE = "case.py"
|
||||
_JS_FILE = "case.js"
|
||||
_C_FILE = "case.c"
|
||||
_C_BIN = "case"
|
||||
|
||||
|
||||
def recipe_for(
|
||||
language: str,
|
||||
snippet: str,
|
||||
image: str,
|
||||
sandbox_limits: SandboxConfig,
|
||||
*,
|
||||
repo_root: Path,
|
||||
) -> SandboxJob | None:
|
||||
"""Return a sandbox job for `language`, or None when no recipe exists."""
|
||||
key = language.strip().lower()
|
||||
if key == "python":
|
||||
return _job(
|
||||
image,
|
||||
sandbox_limits,
|
||||
repo_root,
|
||||
command=("env", f"PYTHONPATH={WORKSPACE_MOUNT}", "python", f"{SCRATCH_MOUNT}/{_PYTHON_FILE}"),
|
||||
scratch_files={_PYTHON_FILE: snippet},
|
||||
)
|
||||
if key == "javascript":
|
||||
return _job(
|
||||
image,
|
||||
sandbox_limits,
|
||||
repo_root,
|
||||
command=("env", f"NODE_PATH={WORKSPACE_MOUNT}", "node", f"{SCRATCH_MOUNT}/{_JS_FILE}"),
|
||||
scratch_files={_JS_FILE: snippet},
|
||||
)
|
||||
if key == "c":
|
||||
source = f"{SCRATCH_MOUNT}/{_C_FILE}"
|
||||
binary = f"{SCRATCH_MOUNT}/{_C_BIN}"
|
||||
return _job(
|
||||
image,
|
||||
sandbox_limits,
|
||||
repo_root,
|
||||
command=("sh", "-c", f"cc {source} -o {binary} && {binary}"),
|
||||
scratch_files={_C_FILE: snippet},
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
def _job(
|
||||
image: str,
|
||||
sandbox_limits: SandboxConfig,
|
||||
repo_root: Path,
|
||||
*,
|
||||
command: tuple[str, ...],
|
||||
scratch_files: dict[str, str],
|
||||
) -> SandboxJob:
|
||||
return SandboxJob(
|
||||
image=image,
|
||||
command=command,
|
||||
repo_root=repo_root,
|
||||
scratch_files=scratch_files,
|
||||
timeout_seconds=sandbox_limits.timeout_seconds,
|
||||
memory_mb=sandbox_limits.memory_mb,
|
||||
pids_limit=sandbox_limits.pids_limit,
|
||||
)
|
||||
@@ -0,0 +1,81 @@
|
||||
"""Structural deny-list for untrusted regression snippets.
|
||||
|
||||
Snippets are data for `/scratch` inside the sandbox. Mentions of sockets, curl,
|
||||
host networking, or writes under the source mount (`/workspace`) are rejected
|
||||
before any `SandboxRunner.run` call.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
from ..sandbox.runner import WORKSPACE_MOUNT
|
||||
|
||||
_WORKSPACE = re.escape(WORKSPACE_MOUNT)
|
||||
|
||||
|
||||
class UnsafeSnippetError(ValueError):
|
||||
"""Raised when a snippet fails the structural sandbox safety check."""
|
||||
|
||||
|
||||
_HOST_NETWORK = re.compile(
|
||||
r"--net(?:work)?(?:\s*=\s*|\s+|['\"]\s*,\s*['\"])host\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_SOCKET_WORD = re.compile(r"\bsocket\b", re.IGNORECASE)
|
||||
_CURL_WORD = re.compile(r"\bcurl\b", re.IGNORECASE)
|
||||
_OPEN_WORKSPACE_WRITE = re.compile(
|
||||
rf"open\s*\(\s*[fFrRbBuU]*['\"]{_WORKSPACE}[^'\"]*['\"]\s*,\s*(?:mode\s*=\s*)?[fFrRbBuU]*['\"][^'\"]*[wax+]",
|
||||
re.IGNORECASE | re.DOTALL,
|
||||
)
|
||||
_PATH_WORKSPACE_WRITE = re.compile(
|
||||
rf"Path\s*\(\s*['\"]{_WORKSPACE}[^'\"]*['\"]\s*\)\s*\.\s*"
|
||||
r"(?:write_text|write_bytes|touch|mkdir|unlink|rmdir|replace|chmod|rename|open)\b",
|
||||
re.IGNORECASE | re.DOTALL,
|
||||
)
|
||||
_FS_WORKSPACE_WRITE = re.compile(
|
||||
r"(?:writeFile(?:Sync)?|appendFile(?:Sync)?|mkdir(?:Sync)?|rm(?:dir)?(?:Sync)?|"
|
||||
rf"unlink(?:Sync)?|rename(?:Sync)?)\s*\(\s*['\"]{_WORKSPACE}",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_FOPEN_WORKSPACE_WRITE = re.compile(
|
||||
rf"fopen\s*\(\s*['\"]{_WORKSPACE}[^'\"]*['\"]\s*,\s*['\"][^'\"]*[wax+]",
|
||||
re.IGNORECASE | re.DOTALL,
|
||||
)
|
||||
_SHUTIL_OS_WORKSPACE_WRITE = re.compile(
|
||||
r"(?:shutil\.(?:copy(?:2)?|copyfile|copytree|move|rmtree)|"
|
||||
r"os\.(?:remove|unlink|replace|rename|mkdir|makedirs|rmdir|chmod))"
|
||||
rf"\s*\([^;]{{0,240}}{_WORKSPACE}",
|
||||
re.IGNORECASE | re.DOTALL,
|
||||
)
|
||||
_REDIRECT_WORKSPACE = re.compile(rf"(?:>>|>)\s*['\"]?{_WORKSPACE}\b", re.IGNORECASE)
|
||||
|
||||
_WORKSPACE_WRITES = (
|
||||
_OPEN_WORKSPACE_WRITE,
|
||||
_PATH_WORKSPACE_WRITE,
|
||||
_FS_WORKSPACE_WRITE,
|
||||
_FOPEN_WORKSPACE_WRITE,
|
||||
_SHUTIL_OS_WORKSPACE_WRITE,
|
||||
_REDIRECT_WORKSPACE,
|
||||
)
|
||||
|
||||
|
||||
def check_snippet_safety(snippet: str) -> None:
|
||||
"""Return None when `snippet` is structurally safe; raise otherwise."""
|
||||
lowered = snippet.lower()
|
||||
if "docker.sock" in lowered:
|
||||
raise UnsafeSnippetError("snippet mentions the Docker socket")
|
||||
if _SOCKET_WORD.search(snippet) is not None:
|
||||
raise UnsafeSnippetError("snippet mentions socket")
|
||||
if _CURL_WORD.search(snippet) is not None:
|
||||
raise UnsafeSnippetError("snippet mentions curl")
|
||||
if _HOST_NETWORK.search(snippet) is not None:
|
||||
raise UnsafeSnippetError("snippet requests host network")
|
||||
if _writes_under_workspace(snippet):
|
||||
raise UnsafeSnippetError(f"snippet writes under {WORKSPACE_MOUNT}")
|
||||
|
||||
|
||||
def _writes_under_workspace(snippet: str) -> bool:
|
||||
if WORKSPACE_MOUNT not in snippet.lower():
|
||||
return False
|
||||
return any(pattern.search(snippet) is not None for pattern in _WORKSPACE_WRITES)
|
||||
@@ -0,0 +1,95 @@
|
||||
"""Map sandbox outcomes to verdicts and the worth-fixing gate."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
from ..sandbox import SandboxResult
|
||||
|
||||
TRIGGERABLE = "triggerable"
|
||||
NOT_TRIGGERABLE = "not_triggerable"
|
||||
ALREADY_COVERED = "already_covered"
|
||||
INCONCLUSIVE = "inconclusive"
|
||||
|
||||
MISSING_RECIPE = "missing_recipe"
|
||||
SAFETY_REJECTED = "safety_rejected"
|
||||
SANDBOX_UNAVAILABLE = "sandbox_unavailable"
|
||||
TIMEOUT = "timeout"
|
||||
COVERED_TEST_PASSED = "covered_test_passed"
|
||||
ASSERTION = "assertion"
|
||||
SANITIZER_ABORT = "sanitizer_abort"
|
||||
CRASH = "crash"
|
||||
NONZERO_EXIT = "nonzero_exit"
|
||||
EXIT_ZERO = "exit_zero"
|
||||
|
||||
WORTH_FIXING_REACHABILITY = frozenset({"reachable", "inferred-file-surface"})
|
||||
|
||||
_SANITIZER_MARKERS = (
|
||||
"AddressSanitizer",
|
||||
"UndefinedBehaviorSanitizer",
|
||||
"ThreadSanitizer",
|
||||
"MemorySanitizer",
|
||||
"LeakSanitizer",
|
||||
)
|
||||
_ASSERTION_MARKERS = (
|
||||
"AssertionError",
|
||||
"Assertion failed",
|
||||
"assertion failed",
|
||||
"assert failed",
|
||||
)
|
||||
_CRASH_MARKERS = (
|
||||
"Fatal Python error",
|
||||
"Segmentation fault",
|
||||
"SIGSEGV",
|
||||
"SIGABRT",
|
||||
"Aborted",
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RegressionVerdict:
|
||||
verdict: str
|
||||
reason: str = ""
|
||||
worth_fixing: bool = False
|
||||
|
||||
|
||||
def is_worth_fixing(verdict: str, reachability: str) -> bool:
|
||||
"""True iff the candidate is triggerable on a reachable or inferred-file-surface finding."""
|
||||
return verdict == TRIGGERABLE and reachability in WORTH_FIXING_REACHABILITY
|
||||
|
||||
|
||||
def verdict_from_result(
|
||||
result: SandboxResult | None = None,
|
||||
*,
|
||||
recipe_missing: bool = False,
|
||||
sandbox_missing: bool = False,
|
||||
safety_rejected: bool = False,
|
||||
covering_test: bool = False,
|
||||
) -> RegressionVerdict:
|
||||
"""Map a sandbox observation onto one of the four regression verdicts."""
|
||||
if recipe_missing:
|
||||
return RegressionVerdict(verdict=INCONCLUSIVE, reason=MISSING_RECIPE)
|
||||
if sandbox_missing:
|
||||
return RegressionVerdict(verdict=INCONCLUSIVE, reason=SANDBOX_UNAVAILABLE)
|
||||
if safety_rejected:
|
||||
return RegressionVerdict(verdict=INCONCLUSIVE, reason=SAFETY_REJECTED)
|
||||
if result is None:
|
||||
return RegressionVerdict(verdict=INCONCLUSIVE, reason=SANDBOX_UNAVAILABLE)
|
||||
if result.timed_out:
|
||||
return RegressionVerdict(verdict=INCONCLUSIVE, reason=TIMEOUT)
|
||||
if covering_test and result.exit_code == 0:
|
||||
return RegressionVerdict(verdict=ALREADY_COVERED, reason=COVERED_TEST_PASSED)
|
||||
if result.exit_code != 0:
|
||||
return RegressionVerdict(verdict=TRIGGERABLE, reason=_nonzero_reason(result))
|
||||
return RegressionVerdict(verdict=NOT_TRIGGERABLE, reason=EXIT_ZERO)
|
||||
|
||||
|
||||
def _nonzero_reason(result: SandboxResult) -> str:
|
||||
text = f"{result.stdout}\n{result.stderr}".lower()
|
||||
if any(marker.lower() in text for marker in _SANITIZER_MARKERS):
|
||||
return SANITIZER_ABORT
|
||||
if any(marker.lower() in text for marker in _ASSERTION_MARKERS):
|
||||
return ASSERTION
|
||||
if any(marker.lower() in text for marker in _CRASH_MARKERS):
|
||||
return CRASH
|
||||
return NONZERO_EXIT
|
||||
@@ -1,6 +1,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections.abc import Sequence
|
||||
from dataclasses import asdict
|
||||
from pathlib import Path
|
||||
|
||||
@@ -10,13 +11,20 @@ from .verification import VerificationResult
|
||||
|
||||
|
||||
def write_markdown_report(
|
||||
findings: list[StaticFinding], path: Path, verifications: list[VerificationResult] | None = None, *, redact: bool = True
|
||||
findings: list[StaticFinding],
|
||||
path: Path,
|
||||
verifications: list[VerificationResult] | None = None,
|
||||
*,
|
||||
redact: bool = True,
|
||||
complexity_map: object | None = None,
|
||||
verdicts: Sequence[object] | None = None,
|
||||
) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
verification_by_id = _verification_by_id(verifications or [])
|
||||
lines = ["# OpenUltraSAST Report", "", f"Findings: {len(findings)}", ""]
|
||||
lines = ["# OpenUltraSAST Report", "", f"Findings: {len(findings)}", "", "## Inventory", ""]
|
||||
if not findings:
|
||||
lines.append("No quick-mode findings were emitted.")
|
||||
lines.append("")
|
||||
for finding in findings:
|
||||
verification = verification_by_id.get(finding.finding_id)
|
||||
lines.extend(
|
||||
@@ -40,6 +48,10 @@ def write_markdown_report(
|
||||
"",
|
||||
]
|
||||
)
|
||||
if complexity_map is not None:
|
||||
_append_complexity_map(lines, complexity_map)
|
||||
if verdicts is not None:
|
||||
_append_worth_fixing(lines, verdicts)
|
||||
report = "\n".join(lines).rstrip() + "\n"
|
||||
if redact:
|
||||
from .redaction import redact_secrets
|
||||
@@ -48,6 +60,68 @@ def write_markdown_report(
|
||||
path.write_text(report)
|
||||
|
||||
|
||||
def _append_complexity_map(lines: list[str], complexity_map: object) -> None:
|
||||
hotspots = tuple(getattr(complexity_map, "hotspots", ()) or ())
|
||||
heuristic_only = getattr(complexity_map, "heuristic_only", None)
|
||||
summary = f"Hotspots: {len(hotspots)}"
|
||||
if heuristic_only is not None:
|
||||
summary += f" (heuristic_only={str(bool(heuristic_only)).lower()})"
|
||||
lines.extend(["## Complexity map", "", summary, ""])
|
||||
if not hotspots:
|
||||
lines.extend(["No complexity hotspots were emitted.", ""])
|
||||
return
|
||||
for hotspot in hotspots:
|
||||
identity = _report_identity(getattr(hotspot, "path", ""), getattr(hotspot, "function_name", None))
|
||||
lines.extend(
|
||||
[
|
||||
f"### `{identity}`",
|
||||
"",
|
||||
f"- Score: `{getattr(hotspot, 'score', '')}`",
|
||||
f"- Band: `{getattr(hotspot, 'band', '')}`",
|
||||
]
|
||||
)
|
||||
hint = getattr(hotspot, "test_hint", None)
|
||||
if hint is not None:
|
||||
lines.append(f"- Gap: `{getattr(hint, 'gap', '')}`")
|
||||
kind = getattr(hint, "test_kind", None)
|
||||
if kind:
|
||||
lines.append(f"- Recommended test: `{kind}`")
|
||||
reason = getattr(hint, "reason", "")
|
||||
if reason:
|
||||
lines.append(f"- Hint: {reason}")
|
||||
inventory_ids = tuple(getattr(hotspot, "inventory_finding_ids", ()) or ())
|
||||
if inventory_ids:
|
||||
lines.append(f"- Inventory: `{', '.join(str(item) for item in inventory_ids)}`")
|
||||
rationale = str(getattr(hotspot, "rationale", "") or "")
|
||||
lines.extend(["", rationale, ""] if rationale else [""])
|
||||
|
||||
|
||||
def _append_worth_fixing(lines: list[str], verdicts: Sequence[object]) -> None:
|
||||
worth = [item for item in verdicts if getattr(item, "worth_fixing", False)]
|
||||
lines.extend(["## Worth fixing", "", f"Count: {len(worth)}", ""])
|
||||
if not worth:
|
||||
lines.extend(["No worth-fixing verdicts were produced.", ""])
|
||||
return
|
||||
for item in worth:
|
||||
identity = _report_identity(getattr(item, "path", ""), getattr(item, "function_name", None))
|
||||
lines.extend(
|
||||
[
|
||||
f"### `{identity}`",
|
||||
"",
|
||||
f"- Verdict: `{getattr(item, 'verdict', '')}`",
|
||||
f"- Reason: `{getattr(item, 'reason', '')}`",
|
||||
"",
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def _report_identity(path: object, function_name: object) -> str:
|
||||
path_text = str(path or "")
|
||||
if function_name:
|
||||
return f"{path_text}::{function_name}"
|
||||
return path_text
|
||||
|
||||
|
||||
def write_sarif_report(findings: list[StaticFinding], verifications: list[VerificationResult], path: Path) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
verification_by_id = _verification_by_id(verifications)
|
||||
@@ -81,6 +155,9 @@ def write_manifest(
|
||||
score: dict[str, object] | None = None,
|
||||
degradations: list[dict[str, object]] | None = None,
|
||||
fusion: list[dict[str, object]] | None = None,
|
||||
stages: dict[str, object] | None = None,
|
||||
complexity: dict[str, object] | None = None,
|
||||
worth_fixing: dict[str, object] | None = None,
|
||||
) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
verification_by_id = _verification_by_id(verifications)
|
||||
@@ -109,16 +186,29 @@ def write_manifest(
|
||||
payload["degradations"] = degradations
|
||||
if fusion is not None:
|
||||
payload["fusion"] = fusion
|
||||
if stages is not None:
|
||||
payload["stages"] = stages
|
||||
if complexity is not None:
|
||||
payload["complexity"] = complexity
|
||||
if worth_fixing is not None:
|
||||
payload["worth_fixing"] = worth_fixing
|
||||
path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
|
||||
|
||||
|
||||
def scan_exit_code(findings: list[StaticFinding], verifications: list[VerificationResult], policy: str) -> int:
|
||||
def scan_exit_code(
|
||||
findings: list[StaticFinding],
|
||||
verifications: list[VerificationResult],
|
||||
policy: str,
|
||||
worth_fixing_verdicts: Sequence[object] | None = None,
|
||||
) -> int:
|
||||
if policy == "never":
|
||||
return 0
|
||||
if policy == "findings":
|
||||
return 1 if findings else 0
|
||||
if policy == "verified":
|
||||
return 1 if any(result.verified for result in verifications) else 0
|
||||
if policy == "worth-fixing":
|
||||
return 1 if any(getattr(verdict, "worth_fixing", False) for verdict in worth_fixing_verdicts or ()) else 0
|
||||
raise ValueError(f"unknown fail policy: {policy}")
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
"""Isolated Docker CLI sandbox."""
|
||||
|
||||
from .probe import FakeSandboxProbe, SandboxProbe, resolve_sandbox_probe
|
||||
from .runner import (
|
||||
DockerCliRunner,
|
||||
FakeSandboxRunner,
|
||||
SandboxJob,
|
||||
SandboxResult,
|
||||
SandboxRunner,
|
||||
build_docker_argv,
|
||||
resolve_sandbox_runner,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"DockerCliRunner",
|
||||
"FakeSandboxProbe",
|
||||
"FakeSandboxRunner",
|
||||
"SandboxJob",
|
||||
"SandboxProbe",
|
||||
"SandboxResult",
|
||||
"SandboxRunner",
|
||||
"build_docker_argv",
|
||||
"resolve_sandbox_probe",
|
||||
"resolve_sandbox_runner",
|
||||
]
|
||||
@@ -0,0 +1,56 @@
|
||||
"""Docker CLI presence and usability probe."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
from collections.abc import Callable
|
||||
from subprocess import CompletedProcess
|
||||
|
||||
PROBE_TIMEOUT_SECONDS = 5
|
||||
PROBE_ENV = "OPENULTRASAST_SANDBOX_PROBE"
|
||||
|
||||
_DOCKER_INFO = ("docker", "info")
|
||||
_PROBE_OFF = frozenset({"0", "off", "false", "unavailable", "no"})
|
||||
_PROBE_ON = frozenset({"1", "on", "true", "available", "yes"})
|
||||
|
||||
|
||||
class SandboxProbe:
|
||||
"""Report whether `docker info` succeeds without raising into the scan."""
|
||||
|
||||
def __init__(self, *, runner: Callable[..., CompletedProcess[str]] | None = None) -> None:
|
||||
self._runner = runner
|
||||
|
||||
def available(self) -> bool:
|
||||
run = self._runner if self._runner is not None else subprocess.run
|
||||
try:
|
||||
result = run(
|
||||
list(_DOCKER_INFO),
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=PROBE_TIMEOUT_SECONDS,
|
||||
check=False,
|
||||
)
|
||||
return result.returncode == 0
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
class FakeSandboxProbe:
|
||||
"""Test double that reports a programmed availability without touching Docker."""
|
||||
|
||||
def __init__(self, available: bool) -> None:
|
||||
self._available = available
|
||||
|
||||
def available(self) -> bool:
|
||||
return self._available
|
||||
|
||||
|
||||
def resolve_sandbox_probe() -> SandboxProbe | FakeSandboxProbe:
|
||||
"""Return a real probe, or a fake when OPENULTRASAST_SANDBOX_PROBE is set."""
|
||||
flag = os.environ.get(PROBE_ENV, "").strip().lower()
|
||||
if flag in _PROBE_OFF:
|
||||
return FakeSandboxProbe(False)
|
||||
if flag in _PROBE_ON:
|
||||
return FakeSandboxProbe(True)
|
||||
return SandboxProbe()
|
||||
@@ -0,0 +1,200 @@
|
||||
"""Docker CLI sandbox runner that never mounts the host Docker socket."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
import tempfile
|
||||
from collections.abc import Callable, Mapping
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from subprocess import CompletedProcess
|
||||
from typing import Protocol
|
||||
|
||||
NONROOT_USER = "65534:65534"
|
||||
WORKSPACE_MOUNT = "/workspace"
|
||||
SCRATCH_MOUNT = "/scratch"
|
||||
SCRATCH_SEED_MOUNT = "/scratch-in"
|
||||
RUNNER_ENV = "OPENULTRASAST_SANDBOX_RUNNER"
|
||||
_COPY_SEED_THEN_EXEC = 'cp -r /scratch-in/. /scratch/ && exec "$@"'
|
||||
_SEED_ARGV0 = "ousast-sandbox"
|
||||
_FAKE_RUNNER = frozenset({"fake"})
|
||||
|
||||
|
||||
class SandboxRunner(Protocol):
|
||||
def run(self, spec: SandboxJob) -> SandboxResult: ...
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SandboxJob:
|
||||
image: str
|
||||
command: tuple[str, ...]
|
||||
repo_root: Path
|
||||
scratch_files: dict[str, str]
|
||||
timeout_seconds: int
|
||||
memory_mb: int
|
||||
pids_limit: int
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SandboxResult:
|
||||
exit_code: int
|
||||
stdout: str
|
||||
stderr: str
|
||||
timed_out: bool
|
||||
|
||||
|
||||
class FakeSandboxRunner:
|
||||
"""Record jobs and return a programmed result without a Docker daemon."""
|
||||
|
||||
def __init__(self, result: SandboxResult | None = None) -> None:
|
||||
self.jobs: list[SandboxJob] = []
|
||||
self.result = result if result is not None else SandboxResult(exit_code=0, stdout="", stderr="", timed_out=False)
|
||||
|
||||
def run(self, spec: SandboxJob) -> SandboxResult:
|
||||
self.jobs.append(spec)
|
||||
return self.result
|
||||
|
||||
|
||||
def resolve_sandbox_runner() -> SandboxRunner:
|
||||
"""Return the Docker CLI runner, or a fake when OPENULTRASAST_SANDBOX_RUNNER=fake."""
|
||||
flag = os.environ.get(RUNNER_ENV, "").strip().lower()
|
||||
if flag in _FAKE_RUNNER:
|
||||
return FakeSandboxRunner()
|
||||
return DockerCliRunner()
|
||||
|
||||
|
||||
class DockerCliRunner:
|
||||
"""Run a job with Docker CLI flags that match the threat model."""
|
||||
|
||||
def __init__(self, *, runner: Callable[..., CompletedProcess[str]] | None = None) -> None:
|
||||
self._runner = runner
|
||||
|
||||
def run(self, spec: SandboxJob) -> SandboxResult:
|
||||
if spec.scratch_files:
|
||||
with tempfile.TemporaryDirectory(prefix="ousast-scratch-") as tmp:
|
||||
seed = Path(tmp)
|
||||
_write_scratch_files(seed, spec.scratch_files)
|
||||
return self._invoke(_argv_with_scratch_seed(spec, seed), spec)
|
||||
return self._invoke(build_docker_argv(spec), spec)
|
||||
|
||||
def _invoke(self, argv: list[str], spec: SandboxJob) -> SandboxResult:
|
||||
_reject_unsafe_argv(argv)
|
||||
run = self._runner if self._runner is not None else subprocess.run
|
||||
try:
|
||||
completed = run(
|
||||
argv,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=spec.timeout_seconds,
|
||||
check=False,
|
||||
)
|
||||
except subprocess.TimeoutExpired as exc:
|
||||
return SandboxResult(
|
||||
exit_code=-1,
|
||||
stdout=_as_text(exc.stdout),
|
||||
stderr=_as_text(exc.stderr),
|
||||
timed_out=True,
|
||||
)
|
||||
return SandboxResult(
|
||||
exit_code=completed.returncode,
|
||||
stdout=completed.stdout or "",
|
||||
stderr=completed.stderr or "",
|
||||
timed_out=False,
|
||||
)
|
||||
|
||||
|
||||
def build_docker_argv(job: SandboxJob) -> list[str]:
|
||||
argv = [*_isolation_argv(job), job.image, *job.command]
|
||||
_reject_unsafe_argv(argv)
|
||||
return argv
|
||||
|
||||
|
||||
def _isolation_argv(job: SandboxJob) -> list[str]:
|
||||
repo = str(job.repo_root.resolve())
|
||||
return [
|
||||
"docker",
|
||||
"run",
|
||||
"--rm",
|
||||
"--init",
|
||||
"--network",
|
||||
"none",
|
||||
"--read-only",
|
||||
"--tmpfs",
|
||||
f"{SCRATCH_MOUNT}:rw,exec",
|
||||
"--memory",
|
||||
f"{job.memory_mb}m",
|
||||
"--pids-limit",
|
||||
str(job.pids_limit),
|
||||
"--cap-drop",
|
||||
"ALL",
|
||||
"--security-opt",
|
||||
"no-new-privileges",
|
||||
"--user",
|
||||
NONROOT_USER,
|
||||
"--mount",
|
||||
f"type=bind,src={repo},dst={WORKSPACE_MOUNT},ro",
|
||||
]
|
||||
|
||||
|
||||
def _argv_with_scratch_seed(job: SandboxJob, seed: Path) -> list[str]:
|
||||
argv = [
|
||||
*_isolation_argv(job),
|
||||
"--mount",
|
||||
f"type=bind,src={seed.resolve()},dst={SCRATCH_SEED_MOUNT},ro",
|
||||
job.image,
|
||||
"/bin/sh",
|
||||
"-c",
|
||||
_COPY_SEED_THEN_EXEC,
|
||||
_SEED_ARGV0,
|
||||
*job.command,
|
||||
]
|
||||
_reject_unsafe_argv(argv)
|
||||
return argv
|
||||
|
||||
|
||||
def _write_scratch_files(root: Path, files: Mapping[str, str]) -> None:
|
||||
resolved_root = root.resolve()
|
||||
for relative, content in files.items():
|
||||
candidate = Path(relative)
|
||||
if candidate.is_absolute():
|
||||
raise ValueError(f"scratch path {relative!r} must be relative to {SCRATCH_MOUNT}")
|
||||
destination = (resolved_root / candidate).resolve()
|
||||
try:
|
||||
common = os.path.commonpath([str(resolved_root), str(destination)])
|
||||
except ValueError as exc:
|
||||
raise ValueError(f"scratch path {relative!r} escapes {SCRATCH_MOUNT}") from exc
|
||||
if common != str(resolved_root):
|
||||
raise ValueError(f"scratch path {relative!r} escapes {SCRATCH_MOUNT}")
|
||||
destination.parent.mkdir(parents=True, exist_ok=True)
|
||||
destination.write_text(content)
|
||||
_chmod_seed_tree(resolved_root)
|
||||
|
||||
|
||||
def _chmod_seed_tree(root: Path) -> None:
|
||||
# Container user 65534 must be able to traverse and read the seed bind.
|
||||
for dirpath, _dirnames, filenames in os.walk(root):
|
||||
Path(dirpath).chmod(0o755)
|
||||
for name in filenames:
|
||||
(Path(dirpath) / name).chmod(0o644)
|
||||
|
||||
|
||||
def _reject_unsafe_argv(argv: list[str]) -> None:
|
||||
joined = " ".join(argv)
|
||||
if "/var/run/docker.sock" in joined:
|
||||
raise ValueError("sandbox argv must not mount the host Docker socket")
|
||||
if "--privileged" in argv:
|
||||
raise ValueError("sandbox argv must not be privileged")
|
||||
for index, item in enumerate(argv):
|
||||
if item in {"--network", "--net"} and index + 1 < len(argv) and argv[index + 1] == "host":
|
||||
raise ValueError("sandbox argv must not use host network")
|
||||
if item in {"--network=host", "--net=host"}:
|
||||
raise ValueError("sandbox argv must not use host network")
|
||||
|
||||
|
||||
def _as_text(value: str | bytes | None) -> str:
|
||||
if value is None:
|
||||
return ""
|
||||
if isinstance(value, bytes):
|
||||
return value.decode(errors="replace")
|
||||
return value
|
||||
@@ -0,0 +1,49 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, replace
|
||||
from enum import StrEnum
|
||||
|
||||
|
||||
class Stage(StrEnum):
|
||||
STATIC = "static"
|
||||
MAP = "map"
|
||||
REGRESS = "regress"
|
||||
|
||||
|
||||
MODE_STAGES: dict[str, tuple[Stage, ...]] = {
|
||||
"quick": (Stage.STATIC,),
|
||||
"standard": (Stage.STATIC, Stage.MAP),
|
||||
"deep": (Stage.STATIC, Stage.MAP, Stage.REGRESS),
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class StagePlan:
|
||||
mode: str
|
||||
requested: tuple[Stage, ...]
|
||||
completed: tuple[Stage, ...]
|
||||
skipped: tuple[dict[str, str], ...] # stage, reason
|
||||
|
||||
|
||||
def plan_for_mode(mode: str) -> StagePlan:
|
||||
return StagePlan(mode=mode, requested=MODE_STAGES[mode], completed=(), skipped=())
|
||||
|
||||
|
||||
def skip_as_degradation(stage: Stage, reason: str) -> dict[str, str]:
|
||||
return {"stage": str(stage), "reason": reason}
|
||||
|
||||
|
||||
def record_skip(plan: StagePlan, stage: Stage, reason: str) -> StagePlan:
|
||||
return replace(plan, skipped=plan.skipped + (skip_as_degradation(stage, reason),))
|
||||
|
||||
|
||||
def record_completed(plan: StagePlan, stage: Stage) -> StagePlan:
|
||||
return replace(plan, completed=plan.completed + (stage,))
|
||||
|
||||
|
||||
def stages_payload(plan: StagePlan) -> dict[str, object]:
|
||||
return {
|
||||
"requested": [str(stage) for stage in plan.requested],
|
||||
"completed": [str(stage) for stage in plan.completed],
|
||||
"skipped": [dict(item) for item in plan.skipped],
|
||||
}
|
||||
@@ -0,0 +1,423 @@
|
||||
"""Bounded tool-using hunter over map hotspots. Findings are suspicion only."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
from collections.abc import Mapping, Sequence
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Protocol
|
||||
|
||||
from .complexity.map import Hotspot
|
||||
from .findings import StaticFinding
|
||||
from .hunter_tools import PathEscapesRepo, clamp_repo_path, find_refs, grep_repo, read_file
|
||||
from .provider.openrouter import OpenRouterChatClient, OpenRouterError
|
||||
|
||||
_DEFAULT_MAX_CHARS = 4000
|
||||
_DEFAULT_MAX_MATCHES = 50
|
||||
DEFAULT_MAX_STEPS = 4
|
||||
CLIENT_ENV = "OPENULTRASAST_HUNTER_CLIENT"
|
||||
_FINDING_ID_PREFIX = "tool-hunter:"
|
||||
_HUNTER_TOOL_NAMES = frozenset({"read_file", "grep_repo", "find_refs"})
|
||||
_SEVERITIES = frozenset({"critical", "high", "medium", "low", "info"})
|
||||
_SCRIPTED_FLAGS = frozenset({"scripted", "script"})
|
||||
_DUMP_FLAGS = frozenset({"dump", "dump-only"})
|
||||
_UNSAFE_FLAGS = frozenset({"unsafe", "unsafe-snippet"})
|
||||
_OPENROUTER_FLAGS = frozenset({"openrouter", "live"})
|
||||
_UNSAFE_SNIPPET = 'client = docker.DockerClient(base_url="unix://var/run/docker.sock")\n'
|
||||
_SCRIPTED_FINDINGS = [
|
||||
{
|
||||
"path": "app.py",
|
||||
"line": 3,
|
||||
"title": "Dynamic execution of attacker-controlled data",
|
||||
"rationale": "eval/execute follows request input across the hotspot.",
|
||||
"evidence_level": "static_corroboration",
|
||||
}
|
||||
]
|
||||
|
||||
HUNTER_TOOLS: list[dict[str, object]] = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "read_file",
|
||||
"description": "Read a file inside the scanned repository root.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"path": {"type": "string", "description": "Path relative to the repository root."},
|
||||
"max_chars": {"type": "integer"},
|
||||
},
|
||||
"required": ["path"],
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "grep_repo",
|
||||
"description": "Search repository source text with a Python regular expression.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"pattern": {"type": "string"},
|
||||
"max_matches": {"type": "integer"},
|
||||
},
|
||||
"required": ["pattern"],
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "find_refs",
|
||||
"description": "List references to a symbol inside the scanned repository.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {"symbol": {"type": "string"}},
|
||||
"required": ["symbol"],
|
||||
},
|
||||
},
|
||||
},
|
||||
]
|
||||
|
||||
_SYSTEM_PROMPT = (
|
||||
"You are a tool-using security hunter. Tools: read_file, grep_repo, find_refs. "
|
||||
"Stay inside the scanned repository. Use a tool before concluding. "
|
||||
"After using tools, reply with a JSON array of objects with path, line, title, and rationale. "
|
||||
"These findings are suspicions, not static corroboration."
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ToolCall:
|
||||
id: str
|
||||
name: str
|
||||
arguments: dict[str, object]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ChatResponse:
|
||||
content: str | None = None
|
||||
tool_calls: tuple[ToolCall, ...] = ()
|
||||
|
||||
|
||||
class ChatClient(Protocol):
|
||||
def complete(
|
||||
self,
|
||||
*,
|
||||
model: str,
|
||||
messages: list[dict[str, object]],
|
||||
tools: list[dict[str, object]],
|
||||
timeout_seconds: int = 60,
|
||||
) -> ChatResponse:
|
||||
raise NotImplementedError
|
||||
|
||||
|
||||
class ScriptedChatClient:
|
||||
"""Deterministic ChatClient for tests and OPENULTRASAST_HUNTER_CLIENT injection."""
|
||||
|
||||
def __init__(self, turns: list[ChatResponse]) -> None:
|
||||
self._turns = turns
|
||||
self.calls: list[dict[str, object]] = []
|
||||
|
||||
def complete(
|
||||
self,
|
||||
*,
|
||||
model: str,
|
||||
messages: list[dict[str, object]],
|
||||
tools: list[dict[str, object]],
|
||||
timeout_seconds: int = 60,
|
||||
) -> ChatResponse:
|
||||
self.calls.append({"model": model, "messages": list(messages), "tools": tools, "timeout_seconds": timeout_seconds})
|
||||
index = len(self.calls) - 1
|
||||
if index >= len(self._turns):
|
||||
return ChatResponse(content="")
|
||||
return self._turns[index]
|
||||
|
||||
|
||||
class OpenRouterHunterClient:
|
||||
"""Adapt OpenRouterChatClient to the tool-hunter ChatClient protocol."""
|
||||
|
||||
def __init__(self, client: OpenRouterChatClient) -> None:
|
||||
self._client = client
|
||||
|
||||
@classmethod
|
||||
def from_env(cls) -> OpenRouterHunterClient:
|
||||
return cls(OpenRouterChatClient.from_env())
|
||||
|
||||
def complete(
|
||||
self,
|
||||
*,
|
||||
model: str,
|
||||
messages: list[dict[str, object]],
|
||||
tools: list[dict[str, object]],
|
||||
timeout_seconds: int = 60,
|
||||
) -> ChatResponse:
|
||||
message = self._client.complete_chat(model=model, messages=messages, tools=tools, timeout_seconds=timeout_seconds)
|
||||
return chat_response_from_message(message)
|
||||
|
||||
|
||||
def scripted_hunter_client(mode: str = "scripted") -> ScriptedChatClient:
|
||||
grep_turn = ChatResponse(
|
||||
tool_calls=(ToolCall(id="call_grep", name="grep_repo", arguments={"pattern": "eval|execute|query", "max_matches": 10}),)
|
||||
)
|
||||
if mode in _DUMP_FLAGS:
|
||||
return ScriptedChatClient([ChatResponse(content=json.dumps(_SCRIPTED_FINDINGS))])
|
||||
if mode in _UNSAFE_FLAGS:
|
||||
payload = [{**_SCRIPTED_FINDINGS[0], "snippet": _UNSAFE_SNIPPET}]
|
||||
return ScriptedChatClient([grep_turn, ChatResponse(content=json.dumps(payload))])
|
||||
return ScriptedChatClient([grep_turn, ChatResponse(content=json.dumps(_SCRIPTED_FINDINGS))])
|
||||
|
||||
|
||||
def resolve_hunter_client() -> ChatClient | None:
|
||||
"""Return a hunter client from env, or OpenRouter when an API key is present."""
|
||||
flag = os.environ.get(CLIENT_ENV, "").strip().lower()
|
||||
if flag in _SCRIPTED_FLAGS:
|
||||
return scripted_hunter_client("scripted")
|
||||
if flag in _DUMP_FLAGS:
|
||||
return scripted_hunter_client("dump-only")
|
||||
if flag in _UNSAFE_FLAGS:
|
||||
return scripted_hunter_client("unsafe-snippet")
|
||||
if flag in _OPENROUTER_FLAGS or (not flag and os.environ.get("OPENROUTER_API_KEY")):
|
||||
try:
|
||||
return OpenRouterHunterClient.from_env()
|
||||
except OpenRouterError:
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def chat_response_from_message(message: Mapping[str, object]) -> ChatResponse:
|
||||
content = message.get("content")
|
||||
text = content if isinstance(content, str) else None
|
||||
raw_calls = message.get("tool_calls") or ()
|
||||
calls: list[ToolCall] = []
|
||||
if isinstance(raw_calls, Sequence) and not isinstance(raw_calls, str | bytes):
|
||||
for index, item in enumerate(raw_calls):
|
||||
parsed = _tool_call_from_openrouter(item, index)
|
||||
if parsed is not None:
|
||||
calls.append(parsed)
|
||||
return ChatResponse(content=text, tool_calls=tuple(calls))
|
||||
|
||||
|
||||
def _tool_call_from_openrouter(item: object, index: int) -> ToolCall | None:
|
||||
if not isinstance(item, Mapping):
|
||||
return None
|
||||
function = item.get("function")
|
||||
if not isinstance(function, Mapping):
|
||||
return None
|
||||
name = function.get("name")
|
||||
if not isinstance(name, str) or not name:
|
||||
return None
|
||||
call_id = item.get("id")
|
||||
identifier = call_id if isinstance(call_id, str) and call_id else f"call_{index}"
|
||||
arguments = function.get("arguments", {})
|
||||
if isinstance(arguments, str):
|
||||
try:
|
||||
loaded = json.loads(arguments)
|
||||
except json.JSONDecodeError:
|
||||
loaded = {}
|
||||
parsed_args = loaded if isinstance(loaded, dict) else {}
|
||||
elif isinstance(arguments, Mapping):
|
||||
parsed_args = dict(arguments)
|
||||
else:
|
||||
parsed_args = {}
|
||||
return ToolCall(id=identifier, name=name, arguments=parsed_args)
|
||||
|
||||
|
||||
def run_tool_hunter(
|
||||
root: Path,
|
||||
hotspots: list[Hotspot],
|
||||
*,
|
||||
client: ChatClient,
|
||||
model: str,
|
||||
max_steps: int,
|
||||
) -> list[StaticFinding]:
|
||||
if max_steps <= 0:
|
||||
return []
|
||||
messages: list[dict[str, object]] = [
|
||||
{"role": "system", "content": _SYSTEM_PROMPT},
|
||||
{"role": "user", "content": _hotspot_prompt(hotspots)},
|
||||
]
|
||||
used_tools = False
|
||||
for _step in range(max_steps):
|
||||
response = client.complete(model=model, messages=messages, tools=HUNTER_TOOLS)
|
||||
tool_calls = tuple(response.tool_calls or ())
|
||||
if tool_calls:
|
||||
messages.append(_assistant_tool_message(response.content, tool_calls))
|
||||
for call in tool_calls:
|
||||
if call.name in _HUNTER_TOOL_NAMES:
|
||||
used_tools = True
|
||||
messages.append(_tool_result_message(root, call))
|
||||
continue
|
||||
content = response.content or ""
|
||||
if not content.strip() or not used_tools:
|
||||
# A findings dump is not evidence unless the loop invoked a repo tool.
|
||||
return []
|
||||
return _findings_from_content(root, hotspots, content)
|
||||
return []
|
||||
|
||||
|
||||
def _hotspot_prompt(hotspots: Sequence[Hotspot]) -> str:
|
||||
payload = [
|
||||
{
|
||||
"path": hotspot.path,
|
||||
"function_name": hotspot.function_name,
|
||||
"score": hotspot.score,
|
||||
"band": hotspot.band,
|
||||
"rationale": hotspot.rationale,
|
||||
}
|
||||
for hotspot in hotspots
|
||||
]
|
||||
return (
|
||||
"Inspect these complexity-map hotspots with the repository tools. "
|
||||
"Return JSON findings only after using a tool.\n"
|
||||
f"{json.dumps(payload, sort_keys=True)}"
|
||||
)
|
||||
|
||||
|
||||
def _assistant_tool_message(content: str | None, tool_calls: Sequence[ToolCall]) -> dict[str, object]:
|
||||
return {
|
||||
"role": "assistant",
|
||||
"content": content,
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": call.id,
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": call.name,
|
||||
"arguments": json.dumps(call.arguments, sort_keys=True) if not isinstance(call.arguments, str) else call.arguments,
|
||||
},
|
||||
}
|
||||
for call in tool_calls
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _tool_result_message(root: Path, call: ToolCall) -> dict[str, object]:
|
||||
return {
|
||||
"role": "tool",
|
||||
"tool_call_id": call.id,
|
||||
"name": call.name,
|
||||
"content": _run_tool(root, call),
|
||||
}
|
||||
|
||||
|
||||
def _run_tool(root: Path, call: ToolCall) -> str:
|
||||
try:
|
||||
payload = _dispatch_tool(root, call.name, _arguments(call.arguments))
|
||||
except Exception as exc: # noqa: BLE001 — tool failures stay in-band so the scan continues
|
||||
payload = {"error": f"{type(exc).__name__}: {exc}"}
|
||||
return json.dumps(payload, sort_keys=True)
|
||||
|
||||
|
||||
def _dispatch_tool(root: Path, name: str, arguments: Mapping[str, object]) -> object:
|
||||
if name == "read_file":
|
||||
path = _require_str(arguments, "path")
|
||||
max_chars = _optional_int(arguments, "max_chars", _DEFAULT_MAX_CHARS)
|
||||
return {"path": path, "content": read_file(root, path, max_chars=max_chars)}
|
||||
if name == "grep_repo":
|
||||
pattern = _require_str(arguments, "pattern")
|
||||
max_matches = _optional_int(arguments, "max_matches", _DEFAULT_MAX_MATCHES)
|
||||
return grep_repo(root, pattern, max_matches=max_matches)
|
||||
if name == "find_refs":
|
||||
symbol = _require_str(arguments, "symbol")
|
||||
return find_refs(root, symbol, None)
|
||||
raise ValueError(f"unknown tool: {name}")
|
||||
|
||||
|
||||
def _arguments(raw: object) -> dict[str, object]:
|
||||
if isinstance(raw, str):
|
||||
parsed = json.loads(raw)
|
||||
if isinstance(parsed, dict):
|
||||
return parsed
|
||||
raise ValueError("tool arguments must be a JSON object")
|
||||
if isinstance(raw, Mapping):
|
||||
return dict(raw)
|
||||
raise ValueError("tool arguments must be an object")
|
||||
|
||||
|
||||
def _require_str(arguments: Mapping[str, object], key: str) -> str:
|
||||
value = arguments.get(key)
|
||||
if not isinstance(value, str) or not value:
|
||||
raise ValueError(f"{key} is required")
|
||||
return value
|
||||
|
||||
|
||||
def _optional_int(arguments: Mapping[str, object], key: str, default: int) -> int:
|
||||
value = arguments.get(key, default)
|
||||
if isinstance(value, bool) or not isinstance(value, int):
|
||||
return default
|
||||
return value
|
||||
|
||||
|
||||
def _findings_from_content(root: Path, hotspots: Sequence[Hotspot], content: str) -> list[StaticFinding]:
|
||||
payload = _load_json(content)
|
||||
if isinstance(payload, list):
|
||||
items = payload
|
||||
elif isinstance(payload, dict) and isinstance(payload.get("findings"), list):
|
||||
items = payload["findings"]
|
||||
else:
|
||||
return []
|
||||
scores = {hotspot.path: hotspot.score for hotspot in hotspots}
|
||||
findings: list[StaticFinding] = []
|
||||
for item in items:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
finding = _finding_from_item(root, item, scores)
|
||||
if finding is not None:
|
||||
findings.append(finding)
|
||||
return sorted(findings, key=lambda item: item.finding_id)
|
||||
|
||||
|
||||
def _load_json(content: str) -> object | None:
|
||||
stripped = content.strip()
|
||||
fenced = re.fullmatch(r"```(?:json)?\s*(.*?)\s*```", stripped, flags=re.DOTALL)
|
||||
if fenced is not None:
|
||||
stripped = fenced.group(1).strip()
|
||||
try:
|
||||
parsed: object = json.loads(stripped)
|
||||
except json.JSONDecodeError:
|
||||
return None
|
||||
return parsed
|
||||
|
||||
|
||||
def _finding_from_item(root: Path, item: Mapping[str, object], scores: Mapping[str, float]) -> StaticFinding | None:
|
||||
raw_path = item.get("path")
|
||||
if not isinstance(raw_path, str) or not raw_path:
|
||||
return None
|
||||
try:
|
||||
relative = _clamp_relative_path(root, raw_path)
|
||||
except PathEscapesRepo:
|
||||
return None
|
||||
line = item.get("line")
|
||||
line_no = line if isinstance(line, int) and not isinstance(line, bool) else None
|
||||
title = item.get("title")
|
||||
rationale = item.get("rationale")
|
||||
function_name = item.get("function_name")
|
||||
severity = item.get("severity")
|
||||
snippet = item.get("snippet")
|
||||
proposed_snippet = snippet if isinstance(snippet, str) and snippet else None
|
||||
return StaticFinding(
|
||||
finding_id=f"{_FINDING_ID_PREFIX}{relative}:{line_no if line_no is not None else 0}",
|
||||
path=relative,
|
||||
title=title if isinstance(title, str) and title else "tool hunter suspicion",
|
||||
severity=severity if isinstance(severity, str) and severity in _SEVERITIES else "medium",
|
||||
confidence="low",
|
||||
evidence_level="suspicion",
|
||||
rationale=rationale if isinstance(rationale, str) and rationale else "Reported by the tool-using hunter.",
|
||||
line=line_no,
|
||||
function_name=function_name if isinstance(function_name, str) and function_name else None,
|
||||
reachability_status="unknown",
|
||||
reachability_evidence=[],
|
||||
reachability_conditions=[],
|
||||
tags=["tool-hunter"],
|
||||
ranking_priority=float(scores.get(relative, 0.0)),
|
||||
proposed_snippet=proposed_snippet,
|
||||
)
|
||||
|
||||
|
||||
def _clamp_relative_path(root: Path, user_path: str) -> str:
|
||||
resolved_root = root.resolve()
|
||||
return clamp_repo_path(resolved_root, user_path).relative_to(resolved_root).as_posix()
|
||||
@@ -63,8 +63,13 @@ def test_repository_benchmark_manifests_load_and_point_to_fixtures() -> None:
|
||||
"java-web-smoke.toml",
|
||||
"javascript-node-web-smoke.toml",
|
||||
"javascript-vulnerable.toml",
|
||||
"mutation-python.toml",
|
||||
"python-vulnerable.toml",
|
||||
"python-web-smoke.toml",
|
||||
"split-sink-c.toml",
|
||||
"split-sink-java.toml",
|
||||
"split-sink-javascript.toml",
|
||||
"split-sink-python.toml",
|
||||
}
|
||||
for manifest_path in manifest_paths:
|
||||
manifest = load_benchmark_manifest(manifest_path)
|
||||
|
||||
@@ -24,6 +24,8 @@ def _run(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, config_body: str | Non
|
||||
(repo / "app.py").write_text(_VULN)
|
||||
monkeypatch.setattr(cli, "has_harnessx", lambda: False) # simulate the extra being absent
|
||||
monkeypatch.setenv("OPENULTRASAST_RUNS_DIR", ".runs")
|
||||
monkeypatch.delenv("OPENROUTER_API_KEY", raising=False)
|
||||
monkeypatch.delenv("OPENULTRASAST_HUNTER_CLIENT", raising=False)
|
||||
argv = ["scan", str(repo), "--mode", "standard"]
|
||||
if config_body is not None:
|
||||
cfg = tmp_path / "openultrasast.toml"
|
||||
@@ -45,9 +47,11 @@ def test_degradation_recorded_when_models_set_but_extra_absent(tmp_path: Path, m
|
||||
assert manifest["findings"]
|
||||
|
||||
|
||||
def test_manifest_has_no_degradations_without_models(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
def test_manifest_has_no_harnessx_degradations_without_models(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
manifest = _run(tmp_path, monkeypatch, config_body=None)
|
||||
assert "degradations" not in manifest
|
||||
degradations = manifest.get("degradations", [])
|
||||
assert all(entry.get("reason") != "harnessx_extra_unavailable" for entry in degradations)
|
||||
assert any(entry.get("reason") == "hunter_model_unavailable" for entry in degradations)
|
||||
|
||||
|
||||
def test_standard_scan_falls_back_to_run_hunter_pool(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
|
||||
@@ -0,0 +1,227 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from openultrasast.complexity.hints import GAP_COVERED, KIND_FUZZ_HARNESS
|
||||
from openultrasast.complexity.map import build_complexity_map
|
||||
from openultrasast.findings import StaticFinding
|
||||
from openultrasast.preprocess import FileTarget
|
||||
|
||||
|
||||
def _target(
|
||||
path: str,
|
||||
*,
|
||||
loc: int,
|
||||
tags: list[str] | None = None,
|
||||
language: str = "python",
|
||||
hints: list[dict[str, object]] | None = None,
|
||||
has_fuzz_entry_point: bool = False,
|
||||
absolute_path: str | None = None,
|
||||
) -> FileTarget:
|
||||
return FileTarget(
|
||||
path=path,
|
||||
absolute_path=absolute_path or f"/repo/{path}",
|
||||
language=language,
|
||||
loc=loc,
|
||||
tags=list(tags or []),
|
||||
has_fuzz_entry_point=has_fuzz_entry_point,
|
||||
reachability_hints=list(hints or []),
|
||||
)
|
||||
|
||||
|
||||
def _hit(path: str, finding_id: str = "rule:helper:1") -> StaticFinding:
|
||||
return StaticFinding(
|
||||
finding_id=finding_id,
|
||||
path=path,
|
||||
title="pattern hit",
|
||||
severity="low",
|
||||
confidence="medium",
|
||||
evidence_level="static_corroboration",
|
||||
rationale="inventory",
|
||||
line=1,
|
||||
function_name=None,
|
||||
reachability_status="unknown",
|
||||
reachability_evidence=[],
|
||||
reachability_conditions=[],
|
||||
tags=[],
|
||||
ranking_priority=1.0,
|
||||
)
|
||||
|
||||
|
||||
def _tree_snapshot(root: Path) -> dict[str, tuple[int, bytes]]:
|
||||
snapshot: dict[str, tuple[int, bytes]] = {}
|
||||
for path in sorted(root.rglob("*")):
|
||||
if path.is_file():
|
||||
snapshot[path.relative_to(root).as_posix()] = (path.stat().st_mtime_ns, path.read_bytes())
|
||||
return snapshot
|
||||
|
||||
|
||||
def test_covered_hotspot_emits_covered_and_no_test_kind(tmp_path: Path) -> None:
|
||||
widget = _target(
|
||||
"src/widget.py",
|
||||
loc=12,
|
||||
hints=[{"kind": "route", "access_level": "public", "function_name": "render"}],
|
||||
)
|
||||
artifact = tmp_path / "complexity_map.json"
|
||||
|
||||
complexity_map = build_complexity_map(
|
||||
[widget],
|
||||
[_hit("src/widget.py")],
|
||||
artifact,
|
||||
repo_files=("src/widget.py", "tests/test_widget.py"),
|
||||
sources={
|
||||
"src/widget.py": "def render():\n return 1\n",
|
||||
"tests/test_widget.py": "from widget import render\n\ndef test_render():\n assert render() == 1\n",
|
||||
},
|
||||
)
|
||||
|
||||
hotspot = complexity_map.hotspots[0]
|
||||
assert hotspot.test_hint is not None
|
||||
assert hotspot.test_hint.gap == GAP_COVERED
|
||||
assert hotspot.test_hint.gap == "covered"
|
||||
assert hotspot.test_hint.test_kind is None
|
||||
assert hotspot.test_hint.path == "src/widget.py"
|
||||
assert hotspot.test_hint.function_name == "render"
|
||||
payload = json.loads(artifact.read_text())
|
||||
hint = payload["hotspots"][0]["test_hint"]
|
||||
assert hint["gap"] == "covered"
|
||||
assert hint["test_kind"] is None
|
||||
|
||||
|
||||
def test_fuzzable_c_parser_without_harness_emits_fuzz_harness(tmp_path: Path) -> None:
|
||||
parser = _target(
|
||||
"src/parser.c",
|
||||
loc=400,
|
||||
language="c",
|
||||
tags=["parser", "memory_unsafe"],
|
||||
hints=[{"kind": "parser", "access_level": "public", "function_name": "parse_header"}],
|
||||
has_fuzz_entry_point=False,
|
||||
)
|
||||
artifact = tmp_path / "complexity_map.json"
|
||||
|
||||
complexity_map = build_complexity_map(
|
||||
[parser],
|
||||
[],
|
||||
artifact,
|
||||
repo_files=("src/parser.c", "tests/test_parser.c"),
|
||||
sources={
|
||||
"src/parser.c": "int parse_header(const char *buf) { return 0; }\n",
|
||||
"tests/test_parser.c": 'int test_parse_header(void) { return parse_header(""); }\n',
|
||||
},
|
||||
)
|
||||
|
||||
hotspot = complexity_map.hotspots[0]
|
||||
assert hotspot.test_hint is not None
|
||||
assert hotspot.test_hint.gap == "no_fuzz_entry"
|
||||
assert hotspot.test_hint.test_kind == KIND_FUZZ_HARNESS
|
||||
assert hotspot.test_hint.test_kind == "fuzz-harness"
|
||||
assert hotspot.test_hint.path == "src/parser.c"
|
||||
assert hotspot.test_hint.function_name == "parse_header"
|
||||
assert hotspot.test_hint.reason
|
||||
payload = json.loads(artifact.read_text())
|
||||
hint = payload["hotspots"][0]["test_hint"]
|
||||
assert hint["gap"] == "no_fuzz_entry"
|
||||
assert hint["test_kind"] == "fuzz-harness"
|
||||
|
||||
|
||||
def test_attach_test_hints_leaves_scanned_tree_unchanged(tmp_path: Path) -> None:
|
||||
repo = tmp_path / "repo"
|
||||
src = repo / "src"
|
||||
tests_dir = repo / "tests"
|
||||
src.mkdir(parents=True)
|
||||
tests_dir.mkdir()
|
||||
parser = src / "parser.c"
|
||||
helper = src / "helper.py"
|
||||
parser_test = tests_dir / "test_parser.c"
|
||||
parser.write_text("int parse_header(const char *buf) {\n return buf[0];\n}\n")
|
||||
helper.write_text("def ping():\n return 1\n")
|
||||
parser_test.write_text("int test_parse_header(void) { return 0; }\n")
|
||||
before = _tree_snapshot(repo)
|
||||
|
||||
artifact = tmp_path / "out" / "complexity_map.json"
|
||||
targets = [
|
||||
_target(
|
||||
"src/parser.c",
|
||||
loc=3,
|
||||
language="c",
|
||||
tags=["parser", "memory_unsafe"],
|
||||
hints=[{"kind": "parser", "access_level": "public", "function_name": "parse_header"}],
|
||||
absolute_path=str(parser),
|
||||
),
|
||||
_target("src/helper.py", loc=2, tags=[], absolute_path=str(helper)),
|
||||
]
|
||||
|
||||
complexity_map = build_complexity_map(
|
||||
targets,
|
||||
[_hit("src/helper.py")],
|
||||
artifact,
|
||||
repo_files=("src/parser.c", "src/helper.py", "tests/test_parser.c"),
|
||||
sources={
|
||||
"src/parser.c": parser.read_text(),
|
||||
"src/helper.py": helper.read_text(),
|
||||
"tests/test_parser.c": parser_test.read_text(),
|
||||
},
|
||||
)
|
||||
|
||||
assert complexity_map.hotspots
|
||||
assert all(hotspot.test_hint is not None for hotspot in complexity_map.hotspots)
|
||||
assert _tree_snapshot(repo) == before
|
||||
assert set(_tree_snapshot(repo)) == {"src/parser.c", "src/helper.py", "tests/test_parser.c"}
|
||||
assert not any(path.as_posix().endswith(".py") and "test" in path.name for path in src.rglob("*"))
|
||||
assert artifact.is_file()
|
||||
assert artifact.is_relative_to(tmp_path / "out")
|
||||
|
||||
|
||||
def test_no_adjacent_test_recommends_closed_test_kind(tmp_path: Path) -> None:
|
||||
helper = _target("src/helper.py", loc=4, tags=[])
|
||||
route = _target(
|
||||
"src/api.py",
|
||||
loc=40,
|
||||
tags=["network_entry"],
|
||||
hints=[{"kind": "route", "access_level": "public", "function_name": "handle"}],
|
||||
)
|
||||
artifact = tmp_path / "complexity_map.json"
|
||||
|
||||
complexity_map = build_complexity_map(
|
||||
[helper, route],
|
||||
[],
|
||||
artifact,
|
||||
repo_files=("src/helper.py", "src/api.py"),
|
||||
sources={
|
||||
"src/helper.py": "def ping():\n return 1\n",
|
||||
"src/api.py": "def handle():\n return {}\n",
|
||||
},
|
||||
)
|
||||
by_path = {hotspot.path: hotspot for hotspot in complexity_map.hotspots}
|
||||
|
||||
assert by_path["src/helper.py"].test_hint is not None
|
||||
assert by_path["src/helper.py"].test_hint.gap == "no_adjacent_test"
|
||||
assert by_path["src/helper.py"].test_hint.test_kind == "unit"
|
||||
assert by_path["src/api.py"].test_hint is not None
|
||||
assert by_path["src/api.py"].test_hint.gap == "no_adjacent_test"
|
||||
assert by_path["src/api.py"].test_hint.test_kind == "http-contract"
|
||||
|
||||
|
||||
def test_no_function_reference_when_adjacent_test_omits_function(tmp_path: Path) -> None:
|
||||
widget = _target(
|
||||
"src/widget.py",
|
||||
loc=12,
|
||||
hints=[{"kind": "route", "access_level": "public", "function_name": "render"}],
|
||||
)
|
||||
artifact = tmp_path / "complexity_map.json"
|
||||
|
||||
complexity_map = build_complexity_map(
|
||||
[widget],
|
||||
[],
|
||||
artifact,
|
||||
repo_files=("src/widget.py", "tests/test_widget.py"),
|
||||
sources={
|
||||
"src/widget.py": "def render():\n return 1\n",
|
||||
"tests/test_widget.py": "def test_placeholder():\n assert True\n",
|
||||
},
|
||||
)
|
||||
|
||||
hint = complexity_map.hotspots[0].test_hint
|
||||
assert hint is not None
|
||||
assert hint.gap == "no_function_reference"
|
||||
assert hint.test_kind == "unit"
|
||||
assert hint.function_name == "render"
|
||||
@@ -0,0 +1,281 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
from openultrasast.complexity.ledger import (
|
||||
NOT_TRIGGERABLE_DELTA,
|
||||
TRIGGERABLE_DELTA,
|
||||
LedgerEntry,
|
||||
apply_overlay,
|
||||
hotspot_key,
|
||||
load_ledger,
|
||||
must_keep_as_candidate,
|
||||
persist_verdicts,
|
||||
record_verdict,
|
||||
select_forced_candidates,
|
||||
write_ledger,
|
||||
)
|
||||
from openultrasast.complexity.map import Hotspot, build_complexity_map
|
||||
from openultrasast.findings import StaticFinding
|
||||
from openultrasast.policy import CwePolicy
|
||||
from openultrasast.preprocess import FileTarget
|
||||
|
||||
|
||||
def _target(
|
||||
path: str,
|
||||
*,
|
||||
loc: int,
|
||||
tags: list[str] | None = None,
|
||||
language: str = "python",
|
||||
hints: list[dict[str, object]] | None = None,
|
||||
has_fuzz_entry_point: bool = False,
|
||||
) -> FileTarget:
|
||||
return FileTarget(
|
||||
path=path,
|
||||
absolute_path=f"/repo/{path}",
|
||||
language=language,
|
||||
loc=loc,
|
||||
tags=list(tags or []),
|
||||
has_fuzz_entry_point=has_fuzz_entry_point,
|
||||
reachability_hints=list(hints or []),
|
||||
)
|
||||
|
||||
|
||||
def _hit(
|
||||
path: str,
|
||||
finding_id: str,
|
||||
*,
|
||||
function_name: str | None = None,
|
||||
reachability: str = "unknown",
|
||||
severity: str = "low",
|
||||
rationale: str = "inventory",
|
||||
) -> StaticFinding:
|
||||
return StaticFinding(
|
||||
finding_id=finding_id,
|
||||
path=path,
|
||||
title="pattern hit",
|
||||
severity=severity,
|
||||
confidence="medium",
|
||||
evidence_level="static_corroboration",
|
||||
rationale=rationale,
|
||||
line=1,
|
||||
function_name=function_name,
|
||||
reachability_status=reachability,
|
||||
reachability_evidence=[],
|
||||
reachability_conditions=[],
|
||||
tags=[],
|
||||
ranking_priority=1.0,
|
||||
)
|
||||
|
||||
|
||||
def _hotspot(
|
||||
path: str,
|
||||
*,
|
||||
function_name: str | None = None,
|
||||
score: float = 4.0,
|
||||
band: str = "medium",
|
||||
inventory_finding_ids: tuple[str, ...] = (),
|
||||
) -> Hotspot:
|
||||
return Hotspot(
|
||||
path=path,
|
||||
function_name=function_name,
|
||||
score=score,
|
||||
band=band,
|
||||
signals={"loc": 10, "inventory_hit_count": len(inventory_finding_ids)},
|
||||
rationale=f"{band} band: score {score}",
|
||||
test_hint=None,
|
||||
inventory_finding_ids=inventory_finding_ids,
|
||||
)
|
||||
|
||||
|
||||
def test_hotspot_key_joins_path_and_function_or_empty() -> None:
|
||||
assert hotspot_key("parser.c", "parse_header") == "parser.c:parse_header"
|
||||
assert hotspot_key("src/app.py", None) == "src/app.py:"
|
||||
assert hotspot_key("src/app.py", "") == "src/app.py:"
|
||||
|
||||
|
||||
def test_triggerable_overlay_raises_later_score() -> None:
|
||||
hotspot = _hotspot("parser.c", function_name="parse_header", score=4.0, inventory_finding_ids=("hit:1",))
|
||||
ledger = record_verdict({}, path="parser.c", function_name="parse_header", verdict="triggerable")
|
||||
|
||||
overlaid = apply_overlay((hotspot,), ledger)
|
||||
raised = overlaid[0]
|
||||
|
||||
assert ledger[hotspot_key("parser.c", "parse_header")].score_delta == TRIGGERABLE_DELTA
|
||||
assert TRIGGERABLE_DELTA > 0
|
||||
assert raised.score == hotspot.score + TRIGGERABLE_DELTA
|
||||
assert raised.score > hotspot.score
|
||||
assert raised.inventory_finding_ids == ("hit:1",)
|
||||
|
||||
|
||||
def test_not_triggerable_overlay_drops_score_and_keeps_inventory_hit() -> None:
|
||||
hotspot = _hotspot("src/auth.py", function_name="login", score=5.0, inventory_finding_ids=("cmd:src/auth.py:12",))
|
||||
ledger = {
|
||||
hotspot_key("src/auth.py", "login"): LedgerEntry(
|
||||
score_delta=NOT_TRIGGERABLE_DELTA,
|
||||
last_verdict="not_triggerable",
|
||||
round=1,
|
||||
)
|
||||
}
|
||||
|
||||
overlaid = apply_overlay((hotspot,), ledger)
|
||||
demoted = overlaid[0]
|
||||
|
||||
assert NOT_TRIGGERABLE_DELTA < 0
|
||||
assert demoted.score == hotspot.score + NOT_TRIGGERABLE_DELTA
|
||||
assert demoted.score < hotspot.score
|
||||
assert demoted.inventory_finding_ids == ("cmd:src/auth.py:12",)
|
||||
assert demoted.path == hotspot.path
|
||||
assert demoted.function_name == hotspot.function_name
|
||||
|
||||
|
||||
def test_not_triggerable_overlay_drops_map_score_and_sev5_reachable_stays_candidate(tmp_path: Path) -> None:
|
||||
auth = _target(
|
||||
"src/auth.py",
|
||||
loc=80,
|
||||
tags=["auth_boundary"],
|
||||
hints=[{"kind": "route", "access_level": "public", "function_name": "login"}],
|
||||
)
|
||||
helper = _target("src/helper.py", loc=12, tags=[])
|
||||
sev5 = _hit(
|
||||
"src/auth.py",
|
||||
"cmd:src/auth.py:12",
|
||||
function_name="login",
|
||||
reachability="reachable",
|
||||
severity="critical",
|
||||
rationale="Static pattern cmd (CWE-78) matched os.system on line 12; manual verification still required.",
|
||||
)
|
||||
noise = _hit("src/helper.py", "print:src/helper.py:1", reachability="unknown", severity="low")
|
||||
artifact = tmp_path / "complexity_map.json"
|
||||
|
||||
baseline = build_complexity_map([auth, helper], [sev5, noise], artifact)
|
||||
original = next(item for item in baseline.hotspots if item.path == "src/auth.py")
|
||||
assert sev5.finding_id in original.inventory_finding_ids
|
||||
|
||||
ledger_path = tmp_path / ".openultrasast" / "calibration" / "complexity_ledger.json"
|
||||
write_ledger(
|
||||
ledger_path,
|
||||
record_verdict({}, path=original.path, function_name=original.function_name, verdict="not_triggerable"),
|
||||
)
|
||||
|
||||
overlaid = build_complexity_map([auth, helper], [sev5, noise], artifact, ledger_path=ledger_path)
|
||||
demoted = next(item for item in overlaid.hotspots if item.path == "src/auth.py")
|
||||
|
||||
assert demoted.score < original.score
|
||||
assert demoted.score == original.score + NOT_TRIGGERABLE_DELTA
|
||||
assert demoted.inventory_finding_ids == original.inventory_finding_ids
|
||||
assert sev5.finding_id in demoted.inventory_finding_ids
|
||||
assert sev5.severity == "critical"
|
||||
assert sev5.rationale == ("Static pattern cmd (CWE-78) matched os.system on line 12; manual verification still required.")
|
||||
|
||||
policy = {
|
||||
sev5.finding_id: 5,
|
||||
noise.finding_id: 2,
|
||||
}
|
||||
forced = select_forced_candidates([sev5, noise], policy)
|
||||
assert forced == (sev5,)
|
||||
assert must_keep_as_candidate(sev5, 5, "reachable") is True
|
||||
assert must_keep_as_candidate(noise, 2, "unknown") is False
|
||||
|
||||
|
||||
def test_must_keep_as_candidate_requires_sev5_and_reachable_surface() -> None:
|
||||
reachable = _hit("a.py", "a:1", reachability="reachable")
|
||||
inferred = _hit("b.py", "b:1", reachability="inferred-file-surface")
|
||||
unknown = _hit("c.py", "c:1", reachability="unknown")
|
||||
|
||||
assert must_keep_as_candidate(reachable, 5, "reachable") is True
|
||||
assert must_keep_as_candidate(inferred, 5, "inferred-file-surface") is True
|
||||
assert must_keep_as_candidate(unknown, 5, "unknown") is False
|
||||
assert must_keep_as_candidate(reachable, 4, "reachable") is False
|
||||
assert must_keep_as_candidate(reachable, 5, None) is True
|
||||
|
||||
|
||||
def test_select_forced_candidates_resolves_policy_severity() -> None:
|
||||
sev5 = _hit("src/auth.py", "cmd:src/auth.py:12", reachability="reachable")
|
||||
inferred = _hit("src/upload.py", "sql:src/upload.py:4", reachability="inferred-file-surface")
|
||||
mild = _hit("src/helper.py", "print:src/helper.py:1", reachability="reachable")
|
||||
policy = {
|
||||
"CWE-78": CwePolicy("Command Injection", 5, True, False),
|
||||
"CWE-89": CwePolicy("SQL Injection", 5, True, False),
|
||||
"CWE-116": CwePolicy("Encode Output", 2, True, False),
|
||||
}
|
||||
|
||||
forced = select_forced_candidates(
|
||||
[mild, sev5, inferred],
|
||||
policy,
|
||||
rule_cwe={
|
||||
sev5.finding_id: "CWE-78",
|
||||
inferred.finding_id: "CWE-89",
|
||||
mild.finding_id: "CWE-116",
|
||||
},
|
||||
)
|
||||
|
||||
assert forced == (sev5, inferred)
|
||||
|
||||
|
||||
def test_ledger_round_trip_matches_design_schema(tmp_path: Path) -> None:
|
||||
path = tmp_path / "complexity_ledger.json"
|
||||
key = hotspot_key("parser.c", "parse_header")
|
||||
write_ledger(
|
||||
path,
|
||||
{key: LedgerEntry(score_delta=1.5, last_verdict="triggerable", round=3)},
|
||||
)
|
||||
payload = json.loads(path.read_text())
|
||||
|
||||
assert payload == {
|
||||
key: {
|
||||
"last_verdict": "triggerable",
|
||||
"round": 3,
|
||||
"score_delta": 1.5,
|
||||
}
|
||||
}
|
||||
loaded = load_ledger(path)
|
||||
assert loaded[key] == LedgerEntry(score_delta=1.5, last_verdict="triggerable", round=3)
|
||||
assert load_ledger(tmp_path / "missing.json") == {}
|
||||
assert load_ledger(None) == {}
|
||||
|
||||
|
||||
def test_overlay_reorders_and_rebands_without_dropping_hits() -> None:
|
||||
high = _hotspot("src/auth.py", function_name="login", score=3.2, band="medium", inventory_finding_ids=("sev5:1",))
|
||||
other = _hotspot("src/helper.py", score=2.0, band="low", inventory_finding_ids=("low:1",))
|
||||
ledger = record_verdict({}, path="src/auth.py", function_name="login", verdict="not_triggerable")
|
||||
|
||||
overlaid = apply_overlay((high, other), ledger)
|
||||
|
||||
assert [item.path for item in overlaid] == ["src/helper.py", "src/auth.py"]
|
||||
demoted = overlaid[1]
|
||||
assert demoted.score == round(3.2 + NOT_TRIGGERABLE_DELTA, 4)
|
||||
assert demoted.band == "low"
|
||||
assert demoted.inventory_finding_ids == ("sev5:1",)
|
||||
assert overlaid[0].inventory_finding_ids == ("low:1",)
|
||||
|
||||
|
||||
def test_persist_verdicts_records_triggerable_and_skips_inconclusive(tmp_path: Path) -> None:
|
||||
path = tmp_path / ".openultrasast" / "calibration" / "complexity_ledger.json"
|
||||
persist_verdicts(
|
||||
path,
|
||||
[
|
||||
SimpleNamespace(path="app.py", function_name="admin", verdict="triggerable"),
|
||||
SimpleNamespace(path="safe.py", function_name=None, verdict="not_triggerable"),
|
||||
SimpleNamespace(path="other.py", function_name="skip", verdict="inconclusive"),
|
||||
],
|
||||
)
|
||||
loaded = load_ledger(path)
|
||||
|
||||
assert loaded[hotspot_key("app.py", "admin")].last_verdict == "triggerable"
|
||||
assert loaded[hotspot_key("app.py", "admin")].score_delta == TRIGGERABLE_DELTA
|
||||
assert loaded[hotspot_key("safe.py", None)].last_verdict == "not_triggerable"
|
||||
assert loaded[hotspot_key("safe.py", None)].score_delta == NOT_TRIGGERABLE_DELTA
|
||||
assert hotspot_key("other.py", "skip") not in loaded
|
||||
|
||||
|
||||
def test_build_complexity_map_without_ledger_path_skips_overlay(tmp_path: Path) -> None:
|
||||
target = _target("src/app.py", loc=20)
|
||||
finding = _hit("src/app.py", "rule:src/app.py:1")
|
||||
artifact = tmp_path / "complexity_map.json"
|
||||
|
||||
first = build_complexity_map([target], [finding], artifact)
|
||||
second = build_complexity_map([target], [finding], artifact)
|
||||
|
||||
assert first.hotspots[0].score == second.hotspots[0].score
|
||||
assert first.hotspots[0].inventory_finding_ids == (finding.finding_id,)
|
||||
@@ -0,0 +1,148 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from openultrasast.complexity.map import build_complexity_map
|
||||
from openultrasast.findings import StaticFinding
|
||||
from openultrasast.preprocess import FileTarget
|
||||
|
||||
_HOTSPOT_FIELDS = {
|
||||
"path",
|
||||
"function_name",
|
||||
"score",
|
||||
"band",
|
||||
"signals",
|
||||
"rationale",
|
||||
"test_hint",
|
||||
"inventory_finding_ids",
|
||||
}
|
||||
|
||||
|
||||
def _target(
|
||||
path: str,
|
||||
*,
|
||||
loc: int,
|
||||
tags: list[str] | None = None,
|
||||
language: str = "python",
|
||||
hints: list[dict[str, object]] | None = None,
|
||||
has_fuzz_entry_point: bool = False,
|
||||
) -> FileTarget:
|
||||
return FileTarget(
|
||||
path=path,
|
||||
absolute_path=f"/repo/{path}",
|
||||
language=language,
|
||||
loc=loc,
|
||||
tags=list(tags or []),
|
||||
has_fuzz_entry_point=has_fuzz_entry_point,
|
||||
reachability_hints=list(hints or []),
|
||||
)
|
||||
|
||||
|
||||
def _hit(path: str, finding_id: str) -> StaticFinding:
|
||||
return StaticFinding(
|
||||
finding_id=finding_id,
|
||||
path=path,
|
||||
title="pattern hit",
|
||||
severity="low",
|
||||
confidence="medium",
|
||||
evidence_level="static_corroboration",
|
||||
rationale="inventory",
|
||||
line=1,
|
||||
function_name=None,
|
||||
reachability_status="unknown",
|
||||
reachability_evidence=[],
|
||||
reachability_conditions=[],
|
||||
tags=[],
|
||||
ranking_priority=1.0,
|
||||
)
|
||||
|
||||
|
||||
def _parser_helper_fixture() -> tuple[list[FileTarget], list[StaticFinding], tuple[str, ...]]:
|
||||
parser = _target(
|
||||
"src/parser.c",
|
||||
loc=400,
|
||||
language="c",
|
||||
tags=["parser", "memory_unsafe"],
|
||||
hints=[{"kind": "parser", "access_level": "public", "function_name": "parse_header"}],
|
||||
)
|
||||
helper = _target("src/helper.py", loc=1, tags=[])
|
||||
findings = [_hit("src/helper.py", f"rule:helper:{index}") for index in range(8)]
|
||||
repo_files = ("src/parser.c", "src/helper.py", "tests/test_parser.c")
|
||||
return [parser, helper], findings, repo_files
|
||||
|
||||
|
||||
def test_map_order_differs_from_inventory_hit_count(tmp_path: Path) -> None:
|
||||
targets, findings, repo_files = _parser_helper_fixture()
|
||||
artifact = tmp_path / "complexity_map.json"
|
||||
|
||||
complexity_map = build_complexity_map(targets, findings, artifact, repo_files=repo_files)
|
||||
|
||||
map_paths = [hotspot.path for hotspot in complexity_map.hotspots]
|
||||
hit_count_paths = [
|
||||
hotspot.path
|
||||
for hotspot in sorted(
|
||||
complexity_map.hotspots,
|
||||
key=lambda item: (-len(item.inventory_finding_ids), item.path),
|
||||
)
|
||||
]
|
||||
parser = next(hotspot for hotspot in complexity_map.hotspots if hotspot.path == "src/parser.c")
|
||||
helper = next(hotspot for hotspot in complexity_map.hotspots if hotspot.path == "src/helper.py")
|
||||
|
||||
assert map_paths != hit_count_paths
|
||||
assert map_paths[0] == "src/parser.c"
|
||||
assert parser.inventory_finding_ids == ()
|
||||
assert len(helper.inventory_finding_ids) == 8
|
||||
assert helper.inventory_finding_ids == tuple(f"rule:helper:{index}" for index in range(8))
|
||||
assert parser.score > helper.score
|
||||
assert parser.function_name == "parse_header"
|
||||
assert parser.band == "high"
|
||||
assert helper.band == "low"
|
||||
assert complexity_map.heuristic_only is True
|
||||
assert parser.test_hint is not None
|
||||
assert parser.test_hint.gap == "no_fuzz_entry"
|
||||
assert parser.test_hint.test_kind == "fuzz-harness"
|
||||
assert helper.test_hint is not None
|
||||
assert helper.test_hint.gap == "no_adjacent_test"
|
||||
assert helper.test_hint.test_kind == "unit"
|
||||
assert "verified" not in parser.rationale.lower()
|
||||
assert "worth fixing" not in parser.rationale.lower()
|
||||
assert "worth_fixing" not in parser.rationale.lower()
|
||||
|
||||
|
||||
def test_complexity_map_json_has_required_fields(tmp_path: Path) -> None:
|
||||
targets, findings, repo_files = _parser_helper_fixture()
|
||||
artifact = tmp_path / "runs" / "complexity_map.json"
|
||||
|
||||
complexity_map = build_complexity_map(targets, findings, artifact, repo_files=repo_files)
|
||||
raw = artifact.read_text()
|
||||
payload = json.loads(raw)
|
||||
|
||||
assert raw == json.dumps(payload, indent=2, sort_keys=True) + "\n"
|
||||
assert payload["heuristic_only"] is True
|
||||
assert [item["path"] for item in payload["hotspots"]] == [hotspot.path for hotspot in complexity_map.hotspots]
|
||||
for hotspot in payload["hotspots"]:
|
||||
assert set(hotspot) >= _HOTSPOT_FIELDS
|
||||
assert hotspot["band"] in {"high", "medium", "low"}
|
||||
assert isinstance(hotspot["score"], float)
|
||||
assert isinstance(hotspot["rationale"], str) and hotspot["rationale"]
|
||||
assert isinstance(hotspot["signals"], dict)
|
||||
assert set(hotspot["signals"]) >= {"loc", "nesting", "reachability", "inventory_hit_count", "has_adjacent_test"}
|
||||
assert isinstance(hotspot["inventory_finding_ids"], list)
|
||||
hint = hotspot["test_hint"]
|
||||
assert isinstance(hint, dict)
|
||||
assert hint["gap"] in {"no_adjacent_test", "no_function_reference", "no_fuzz_entry", "covered"}
|
||||
if hint["gap"] == "covered":
|
||||
assert hint["test_kind"] is None
|
||||
else:
|
||||
assert hint["test_kind"] in {"unit", "property", "sanitizer", "http-contract", "fuzz-harness"}
|
||||
assert "6.0" in hotspot["rationale"]
|
||||
assert "3.0" in hotspot["rationale"]
|
||||
parser = next(item for item in payload["hotspots"] if item["path"] == "src/parser.c")
|
||||
helper = next(item for item in payload["hotspots"] if item["path"] == "src/helper.py")
|
||||
assert parser["signals"]["has_adjacent_test"] is True
|
||||
assert helper["signals"]["has_adjacent_test"] is False
|
||||
assert parser["test_hint"]["gap"] == "no_fuzz_entry"
|
||||
assert parser["test_hint"]["test_kind"] == "fuzz-harness"
|
||||
assert helper["test_hint"]["gap"] == "no_adjacent_test"
|
||||
assert helper["test_hint"]["test_kind"] == "unit"
|
||||
assert helper["inventory_finding_ids"] == [f"rule:helper:{index}" for index in range(8)]
|
||||
assert parser["inventory_finding_ids"] == []
|
||||
@@ -0,0 +1,171 @@
|
||||
from pathlib import Path
|
||||
|
||||
from openultrasast.complexity import ComplexitySignals, collect_signals
|
||||
from openultrasast.findings import StaticFinding
|
||||
from openultrasast.preprocess import FileTarget
|
||||
|
||||
|
||||
def _target(
|
||||
path: str,
|
||||
*,
|
||||
loc: int,
|
||||
tags: list[str] | None = None,
|
||||
language: str = "python",
|
||||
hints: list[dict[str, object]] | None = None,
|
||||
has_fuzz_entry_point: bool = False,
|
||||
absolute_path: str | None = None,
|
||||
) -> FileTarget:
|
||||
return FileTarget(
|
||||
path=path,
|
||||
absolute_path=absolute_path or f"/repo/{path}",
|
||||
language=language,
|
||||
loc=loc,
|
||||
tags=list(tags or []),
|
||||
has_fuzz_entry_point=has_fuzz_entry_point,
|
||||
reachability_hints=list(hints or []),
|
||||
)
|
||||
|
||||
|
||||
def _hit(path: str, finding_id: str = "rule:helper:1") -> StaticFinding:
|
||||
return StaticFinding(
|
||||
finding_id=finding_id,
|
||||
path=path,
|
||||
title="pattern hit",
|
||||
severity="low",
|
||||
confidence="medium",
|
||||
evidence_level="static_corroboration",
|
||||
rationale="inventory",
|
||||
line=1,
|
||||
function_name=None,
|
||||
reachability_status="unknown",
|
||||
reachability_evidence=[],
|
||||
reachability_conditions=[],
|
||||
tags=[],
|
||||
ranking_priority=1.0,
|
||||
)
|
||||
|
||||
|
||||
def _by_path(signals: tuple[ComplexitySignals, ...]) -> dict[str, ComplexitySignals]:
|
||||
return {signal.path: signal for signal in signals}
|
||||
|
||||
|
||||
def test_parser_without_pattern_hit_outranks_one_line_helper_with_hit() -> None:
|
||||
parser = _target(
|
||||
"src/parser.c",
|
||||
loc=400,
|
||||
language="c",
|
||||
tags=["parser", "memory_unsafe"],
|
||||
hints=[{"kind": "parser", "access_level": "public", "function_name": "parse_header"}],
|
||||
)
|
||||
helper = _target("src/helper.py", loc=1, tags=[])
|
||||
findings = [_hit("src/helper.py")]
|
||||
|
||||
signals = collect_signals([parser, helper], findings)
|
||||
by_path = _by_path(signals)
|
||||
|
||||
assert by_path["src/parser.c"].inventory_hit_count == 0
|
||||
assert by_path["src/helper.py"].inventory_hit_count == 1
|
||||
assert by_path["src/parser.c"].score > by_path["src/helper.py"].score
|
||||
assert signals[0].path == "src/parser.c"
|
||||
|
||||
|
||||
def test_function_identity_uses_entry_point_names_or_stays_empty() -> None:
|
||||
named = _target(
|
||||
"src/parser.c",
|
||||
loc=40,
|
||||
language="c",
|
||||
tags=["parser"],
|
||||
hints=[{"function_name": "parse_header", "kind": "parser", "access_level": "public"}],
|
||||
)
|
||||
named_alias = _target(
|
||||
"src/decode.c",
|
||||
loc=20,
|
||||
language="c",
|
||||
tags=["parser"],
|
||||
hints=[{"name": "decode_frame", "kind": "parser", "access_level": "public"}],
|
||||
)
|
||||
tag_only = _target(
|
||||
"src/App.java",
|
||||
loc=80,
|
||||
language="java",
|
||||
tags=["parser"],
|
||||
hints=[{"function_name": None, "name": "parser_input", "kind": "parser", "line": None}],
|
||||
)
|
||||
bare = _target("src/util.py", loc=4, tags=[])
|
||||
|
||||
signals = collect_signals([named, named_alias, tag_only, bare])
|
||||
by_path = _by_path(signals)
|
||||
|
||||
assert by_path["src/parser.c"].function_name == "parse_header"
|
||||
assert by_path["src/decode.c"].function_name == "decode_frame"
|
||||
assert by_path["src/App.java"].function_name is None
|
||||
assert by_path["src/util.py"].function_name is None
|
||||
|
||||
|
||||
def test_nesting_from_source_text_otherwise_zero() -> None:
|
||||
nested = _target("src/nested.py", loc=4, tags=[])
|
||||
constructed = _target("src/flat.py", loc=12, tags=["parser"])
|
||||
source = "def outer():\n if True:\n while True:\n return 1\n"
|
||||
|
||||
with_text = collect_signals([nested], sources={"src/nested.py": source})
|
||||
without_text = collect_signals([constructed])
|
||||
|
||||
assert with_text[0].nesting >= 3
|
||||
assert without_text[0].nesting == 0
|
||||
assert without_text[0].loc == 12
|
||||
|
||||
|
||||
def test_nesting_reads_absolute_path_when_file_exists(tmp_path: Path) -> None:
|
||||
path = tmp_path / "braces.c"
|
||||
path.write_text("int f(void) {\n if (1) {\n while (1) {\n return 0;\n }\n }\n}\n")
|
||||
target = _target("braces.c", loc=7, language="c", tags=[], absolute_path=str(path))
|
||||
|
||||
assert collect_signals([target])[0].nesting >= 3
|
||||
|
||||
|
||||
def test_adjacent_test_from_repo_relative_paths() -> None:
|
||||
parser = _target("src/parser.c", loc=40, language="c", tags=["parser"])
|
||||
helper = _target("pkg/helper.go", loc=8, language="go", tags=[])
|
||||
widget = _target("src/main/java/com/acme/Widget.java", loc=30, language="java", tags=[])
|
||||
lonely = _target("src/lonely.py", loc=5, tags=[])
|
||||
|
||||
signals = collect_signals(
|
||||
[parser, helper, widget, lonely],
|
||||
repo_files=(
|
||||
"src/parser.c",
|
||||
"tests/test_parser.c",
|
||||
"pkg/helper.go",
|
||||
"pkg/helper_test.go",
|
||||
"src/main/java/com/acme/Widget.java",
|
||||
"src/test/java/com/acme/WidgetTest.java",
|
||||
"src/lonely.py",
|
||||
"tests/test_other.py",
|
||||
),
|
||||
)
|
||||
by_path = _by_path(signals)
|
||||
|
||||
assert by_path["src/parser.c"].has_adjacent_test is True
|
||||
assert by_path["pkg/helper.go"].has_adjacent_test is True
|
||||
assert by_path["src/main/java/com/acme/Widget.java"].has_adjacent_test is True
|
||||
assert by_path["src/lonely.py"].has_adjacent_test is False
|
||||
|
||||
|
||||
def test_inventory_density_is_recorded_but_cannot_dominate_score() -> None:
|
||||
parser = _target(
|
||||
"src/parser.c",
|
||||
loc=250,
|
||||
language="c",
|
||||
tags=["parser"],
|
||||
hints=[{"kind": "parser", "access_level": "public", "function_name": "parse_header"}],
|
||||
)
|
||||
helper = _target("src/helper.py", loc=1, tags=[])
|
||||
findings = [_hit("src/helper.py", f"rule:helper:{index}") for index in range(12)]
|
||||
|
||||
signals = collect_signals([parser, helper], findings)
|
||||
by_path = _by_path(signals)
|
||||
|
||||
assert by_path["src/helper.py"].inventory_hit_count == 12
|
||||
assert by_path["src/parser.c"].inventory_hit_count == 0
|
||||
assert by_path["src/parser.c"].score > by_path["src/helper.py"].score
|
||||
assert "parser" in by_path["src/parser.c"].tags
|
||||
assert by_path["src/parser.c"].reachability > by_path["src/helper.py"].reachability
|
||||
@@ -96,3 +96,66 @@ def test_write_resolved_config_creates_json(tmp_path: Path) -> None:
|
||||
|
||||
assert output.exists()
|
||||
assert '"minimum_report_verified": "static_corroboration"' in output.read_text()
|
||||
|
||||
|
||||
def test_complexity_and_regress_defaults_when_section_absent() -> None:
|
||||
config = load_config(None)
|
||||
|
||||
assert config.complexity.top_k == 20
|
||||
assert config.complexity.max_hunter_hotspots == 8
|
||||
assert config.regress.max_candidates == 5
|
||||
assert config.regress.images == ()
|
||||
assert config.sandbox.network is False
|
||||
assert config.sandbox.workspace_readonly is True
|
||||
assert config.sandbox.memory_mb == 2048
|
||||
assert config.sandbox.timeout_seconds == 300
|
||||
assert config.sandbox.pids_limit == 512
|
||||
|
||||
|
||||
def test_override_changes_top_k_and_max_candidates_only(tmp_path: Path) -> None:
|
||||
config_path = tmp_path / "openultrasast.toml"
|
||||
config_path.write_text(
|
||||
"\n".join(
|
||||
[
|
||||
"[complexity]",
|
||||
"top_k = 3",
|
||||
"[regress]",
|
||||
"max_candidates = 2",
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
config = load_config(config_path)
|
||||
|
||||
assert config.complexity.top_k == 3
|
||||
assert config.regress.max_candidates == 2
|
||||
assert config.complexity.max_hunter_hotspots == 8
|
||||
assert config.regress.images == ()
|
||||
assert config.sandbox.network is False
|
||||
assert config.sandbox.workspace_readonly is True
|
||||
assert config.sandbox.memory_mb == 2048
|
||||
assert config.sandbox.timeout_seconds == 300
|
||||
assert config.sandbox.pids_limit == 512
|
||||
|
||||
|
||||
def test_load_config_reads_regress_image_pins(tmp_path: Path) -> None:
|
||||
config_path = tmp_path / "openultrasast.toml"
|
||||
config_path.write_text(
|
||||
"\n".join(
|
||||
[
|
||||
"[regress]",
|
||||
"max_candidates = 4",
|
||||
"[regress.images]",
|
||||
'python = "python:3.12-slim"',
|
||||
'javascript = "node:22-slim"',
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
config = load_config(config_path)
|
||||
|
||||
assert config.regress.max_candidates == 4
|
||||
assert dict(config.regress.images) == {"python": "python:3.12-slim", "javascript": "node:22-slim"}
|
||||
assert config.complexity.top_k == 20
|
||||
assert config.complexity.max_hunter_hotspots == 8
|
||||
assert config.sandbox.memory_mb == 2048
|
||||
|
||||
@@ -0,0 +1,123 @@
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from openultrasast.hunter_tools import PathEscapesRepo, clamp_repo_path, find_refs, grep_repo, read_file
|
||||
from openultrasast.mapping import EntryPointRecord
|
||||
|
||||
SPLIT_SINK = """\
|
||||
term = request.args["q"]
|
||||
query = "select * from items where title like '%" + term + "%'"
|
||||
return db.execute(query)
|
||||
"""
|
||||
|
||||
|
||||
def _repo_with_split_sink(tmp_path: Path) -> Path:
|
||||
root = tmp_path / "repo"
|
||||
root.mkdir()
|
||||
(root / "app.py").write_text(SPLIT_SINK)
|
||||
return root
|
||||
|
||||
|
||||
def test_read_grep_and_reject_path_escape(tmp_path: Path) -> None:
|
||||
root = _repo_with_split_sink(tmp_path)
|
||||
|
||||
source = read_file(root, "app.py", max_chars=4000)
|
||||
assert "query =" in source
|
||||
assert "db.execute(query)" in source
|
||||
|
||||
matches = grep_repo(root, r"query\s*=", max_matches=10)
|
||||
assert matches
|
||||
assert matches[0]["path"] == "app.py"
|
||||
assert matches[0]["line"] == 2
|
||||
assert "query =" in str(matches[0]["text"])
|
||||
|
||||
with pytest.raises(PathEscapesRepo):
|
||||
read_file(root, "../etc/passwd", max_chars=100)
|
||||
|
||||
|
||||
def test_clamp_repo_path_rejects_dotdot_and_absolute_escapes(tmp_path: Path) -> None:
|
||||
root = tmp_path / "repo"
|
||||
root.mkdir()
|
||||
(root / "app.py").write_text("ok\n")
|
||||
|
||||
assert clamp_repo_path(root, "app.py") == (root / "app.py").resolve()
|
||||
|
||||
with pytest.raises(PathEscapesRepo):
|
||||
clamp_repo_path(root, "../etc/passwd")
|
||||
with pytest.raises(PathEscapesRepo):
|
||||
clamp_repo_path(root, "/etc/passwd")
|
||||
with pytest.raises(PathEscapesRepo):
|
||||
clamp_repo_path(root, "..")
|
||||
|
||||
|
||||
def test_grep_repo_does_not_search_outside_root(tmp_path: Path) -> None:
|
||||
root = tmp_path / "repo"
|
||||
root.mkdir()
|
||||
(root / "app.py").write_text('query = "inside"\n')
|
||||
(tmp_path / "outside.py").write_text('query = "outside"\n')
|
||||
|
||||
matches = grep_repo(root, r"query\s*=", max_matches=10)
|
||||
|
||||
assert [match["path"] for match in matches] == ["app.py"]
|
||||
assert all("outside" not in str(match["text"]) for match in matches)
|
||||
|
||||
|
||||
def test_grep_repo_honors_max_matches(tmp_path: Path) -> None:
|
||||
root = tmp_path / "repo"
|
||||
root.mkdir()
|
||||
(root / "app.py").write_text("sink = 1\nsink = 2\nsink = 3\n")
|
||||
|
||||
matches = grep_repo(root, r"sink\s*=", max_matches=2)
|
||||
|
||||
assert len(matches) == 2
|
||||
assert [match["line"] for match in matches] == [1, 2]
|
||||
|
||||
|
||||
def test_read_file_truncates_and_rejects_symlink_escape(tmp_path: Path) -> None:
|
||||
root = tmp_path / "repo"
|
||||
root.mkdir()
|
||||
(root / "app.py").write_text("abcdef")
|
||||
(root / "link.py").symlink_to("/etc/passwd")
|
||||
|
||||
assert read_file(root, "app.py", max_chars=3) == "abc"
|
||||
with pytest.raises(PathEscapesRepo):
|
||||
read_file(root, "link.py", max_chars=100)
|
||||
|
||||
|
||||
def test_find_refs_uses_mapping_index_and_clamped_text_search(tmp_path: Path) -> None:
|
||||
root = _repo_with_split_sink(tmp_path)
|
||||
mapping_index = [
|
||||
{"path": "app.py", "name": "query", "line": 2},
|
||||
EntryPointRecord(
|
||||
path="app.py",
|
||||
line=3,
|
||||
end_line=3,
|
||||
function_name="search",
|
||||
name="search",
|
||||
kind="route",
|
||||
access_level="public",
|
||||
trust_boundary="http_request",
|
||||
access_evidence=[],
|
||||
conditions=[],
|
||||
provenance="test",
|
||||
rationale="fixture",
|
||||
),
|
||||
]
|
||||
|
||||
query_refs = find_refs(root, "query", mapping_index)
|
||||
assert any(ref["path"] == "app.py" and ref["name"] == "query" for ref in query_refs)
|
||||
assert any(ref.get("source") == "mapping" for ref in query_refs)
|
||||
assert any(ref.get("source") == "text" and "query" in str(ref.get("text", ref.get("name", ""))) for ref in query_refs)
|
||||
|
||||
search_refs = find_refs(root, "search", mapping_index)
|
||||
assert any(ref["path"] == "app.py" and ref["name"] == "search" for ref in search_refs)
|
||||
|
||||
|
||||
def test_find_refs_rejects_mapping_path_escape(tmp_path: Path) -> None:
|
||||
root = _repo_with_split_sink(tmp_path)
|
||||
|
||||
with pytest.raises(PathEscapesRepo):
|
||||
find_refs(root, "passwd", [{"path": "../etc/passwd", "name": "passwd"}])
|
||||
with pytest.raises(PathEscapesRepo):
|
||||
find_refs(root, "passwd", [{"path": "/etc/passwd", "name": "passwd"}])
|
||||
@@ -0,0 +1,82 @@
|
||||
"""Stage-2 map gate: planted split-sink ranks in the high band and beats hit-count order."""
|
||||
|
||||
from openultrasast.complexity.map import Hotspot
|
||||
from openultrasast.findings import StaticFinding
|
||||
from openultrasast.gate import LANGUAGE_MANIFESTS
|
||||
from openultrasast.map_gate import (
|
||||
SPLIT_SINK_MANIFESTS,
|
||||
hit_count_path_order,
|
||||
map_beats_hit_count,
|
||||
planted_in_high_band,
|
||||
scan_split_sink,
|
||||
split_sink_map_gate,
|
||||
)
|
||||
|
||||
|
||||
def test_split_sink_map_gate_passes_heuristic_map() -> None:
|
||||
verdict = split_sink_map_gate()
|
||||
assert verdict.passed, verdict.reasons
|
||||
|
||||
|
||||
def test_hit_count_stub_fails_map_gate_on_split_sink_python() -> None:
|
||||
bundle = scan_split_sink("split-sink-python")
|
||||
unique_map = list(dict.fromkeys(hotspot.path for hotspot in bundle.complexity_map.hotspots))
|
||||
stub_order = hit_count_path_order(bundle.findings, unique_map)
|
||||
assert stub_order != unique_map
|
||||
assert map_beats_hit_count(bundle.complexity_map.hotspots, bundle.findings)
|
||||
|
||||
|
||||
def test_planted_split_sink_function_is_in_high_band() -> None:
|
||||
bundle = scan_split_sink("split-sink-python")
|
||||
assert planted_in_high_band(bundle.manifest, bundle.complexity_map)
|
||||
names = {hotspot.function_name for hotspot in bundle.complexity_map.hotspots if hotspot.path == "app.py"}
|
||||
assert "search" in names
|
||||
|
||||
|
||||
def test_split_sink_gate_corpus_is_not_the_stage1_smoke_gate() -> None:
|
||||
listed = {name for names in LANGUAGE_MANIFESTS.values() for name in names}
|
||||
assert not any(name in listed for name in SPLIT_SINK_MANIFESTS)
|
||||
|
||||
|
||||
def test_map_beats_hit_count_rejects_inventory_sorted_stub() -> None:
|
||||
findings = [
|
||||
StaticFinding(
|
||||
finding_id="python-unsafe-eval:helper.py:1",
|
||||
path="helper.py",
|
||||
title="eval",
|
||||
severity="high",
|
||||
confidence="low",
|
||||
evidence_level="static_corroboration",
|
||||
rationale="hit",
|
||||
line=1,
|
||||
function_name="ping",
|
||||
reachability_status="unknown",
|
||||
reachability_evidence=[],
|
||||
reachability_conditions=[],
|
||||
tags=["syscall_entry"],
|
||||
ranking_priority=1.0,
|
||||
)
|
||||
]
|
||||
stub = (
|
||||
Hotspot(
|
||||
path="helper.py",
|
||||
function_name="ping",
|
||||
score=0.1,
|
||||
band="low",
|
||||
signals={},
|
||||
rationale="stub",
|
||||
test_hint=None,
|
||||
inventory_finding_ids=(findings[0].finding_id,),
|
||||
),
|
||||
Hotspot(
|
||||
path="app.py",
|
||||
function_name="search",
|
||||
score=9.0,
|
||||
band="high",
|
||||
signals={},
|
||||
rationale="stub",
|
||||
test_hint=None,
|
||||
inventory_finding_ids=(),
|
||||
),
|
||||
)
|
||||
assert not map_beats_hit_count(stub, findings)
|
||||
+18
-1
@@ -54,7 +54,24 @@ def test_surface_exposes_no_shell_or_exec_tool() -> None:
|
||||
for name in server.tool_names:
|
||||
assert not any(token in name.lower() for token in forbidden), name
|
||||
for tool in McpServer()._tools.values(): # no tool accepts a free-form command argument
|
||||
assert "command" not in tool.input_schema.get("properties", {})
|
||||
properties = tool.input_schema.get("properties", {})
|
||||
assert "command" not in properties
|
||||
assert "shell" not in properties
|
||||
assert "docker" not in properties
|
||||
|
||||
|
||||
def test_scan_rejects_deep_and_keeps_cli_only_modes() -> None:
|
||||
server = McpServer()
|
||||
listed = server.handle({"jsonrpc": "2.0", "id": 1, "method": "tools/list", "params": {}})
|
||||
assert listed is not None
|
||||
scan_tool = next(tool for tool in listed["result"]["tools"] if tool["name"] == "openultrasast.scan")
|
||||
mode_enum = scan_tool["inputSchema"]["properties"]["mode"]["enum"]
|
||||
assert mode_enum == ["quick", "standard"]
|
||||
assert "deep" not in mode_enum
|
||||
|
||||
result = _call(server, "openultrasast.scan", {"path": ".", "mode": "deep"})
|
||||
assert result["isError"]
|
||||
assert "deep" in result["error"]
|
||||
|
||||
|
||||
def test_notifications_get_no_response() -> None:
|
||||
|
||||
@@ -0,0 +1,46 @@
|
||||
"""Mutation fixtures: stage-1 same-line patterns must miss the aliased sink (requirement 8.2)."""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from openultrasast.benchmark import BenchmarkRun, evaluate_benchmark, load_benchmark_manifest, resolve_benchmark_source
|
||||
from openultrasast.findings import StaticFinding, quick_scan_findings
|
||||
from openultrasast.gate import LANGUAGE_MANIFESTS
|
||||
from openultrasast.preprocess import preprocess_repository
|
||||
from openultrasast.rank import rank_targets
|
||||
|
||||
MANIFEST = Path("benchmarks/manifests/mutation-python.toml")
|
||||
|
||||
|
||||
def _quick_scan(target: Path) -> list[StaticFinding]:
|
||||
_, targets = preprocess_repository(target)
|
||||
return quick_scan_findings(target, targets, rank_targets(targets))
|
||||
|
||||
|
||||
def test_stage1_misses_aliased_os_system_mutation() -> None:
|
||||
manifest = load_benchmark_manifest(MANIFEST)
|
||||
target = resolve_benchmark_source(MANIFEST, manifest)
|
||||
planted = [item for item in manifest.expected if item.rule_id is None]
|
||||
assert planted, "mutation manifest must document a stage-1 miss without rule_id"
|
||||
expected = planted[0]
|
||||
assert expected.sink == "run_cmd"
|
||||
assert expected.line is not None
|
||||
|
||||
source = (target / expected.path).read_text()
|
||||
lines = source.splitlines()
|
||||
assert "os.system" in source
|
||||
assert "os.system(" not in lines[expected.line - 1]
|
||||
assert "run_cmd(" in lines[expected.line - 1]
|
||||
|
||||
findings = _quick_scan(target)
|
||||
on_line = [finding for finding in findings if finding.path == expected.path and finding.line == expected.line]
|
||||
assert on_line == [], f"stage-1 hit the mutation on {expected.path}:{expected.line}: {on_line}"
|
||||
|
||||
run = BenchmarkRun(benchmark_run_id="mutation", root=Path("/tmp/mutation"), manifest=manifest)
|
||||
result = evaluate_benchmark(run=run, mode="quick", findings=findings, scan_id=None, scan_run_dir=None)
|
||||
missed = {(miss.path, miss.cwe, miss.rule_id) for miss in result.misses}
|
||||
assert (expected.path, expected.cwe, None) in missed
|
||||
|
||||
|
||||
def test_mutation_fixture_is_outside_stage1_gate_corpus() -> None:
|
||||
listed = {name for names in LANGUAGE_MANIFESTS.values() for name in names}
|
||||
assert "mutation-python" not in listed
|
||||
@@ -0,0 +1,159 @@
|
||||
"""Bounded regression candidate selection from hotspots and severe inventory (task 5.1)."""
|
||||
|
||||
from openultrasast.complexity.ledger import NOT_TRIGGERABLE_DELTA, apply_overlay, record_verdict
|
||||
from openultrasast.complexity.map import Hotspot
|
||||
from openultrasast.findings import StaticFinding
|
||||
from openultrasast.regress.candidate import select_candidates
|
||||
|
||||
|
||||
def _finding(
|
||||
path: str,
|
||||
finding_id: str,
|
||||
*,
|
||||
function_name: str | None = None,
|
||||
reachability: str = "unknown",
|
||||
severity: str = "low",
|
||||
) -> StaticFinding:
|
||||
return StaticFinding(
|
||||
finding_id=finding_id,
|
||||
path=path,
|
||||
title="pattern hit",
|
||||
severity=severity,
|
||||
confidence="medium",
|
||||
evidence_level="static_corroboration",
|
||||
rationale="inventory",
|
||||
line=1,
|
||||
function_name=function_name,
|
||||
reachability_status=reachability,
|
||||
reachability_evidence=[],
|
||||
reachability_conditions=[],
|
||||
tags=[],
|
||||
ranking_priority=1.0,
|
||||
)
|
||||
|
||||
|
||||
def _hotspot(
|
||||
path: str,
|
||||
*,
|
||||
function_name: str | None = None,
|
||||
score: float = 4.0,
|
||||
band: str = "medium",
|
||||
inventory_finding_ids: tuple[str, ...] = (),
|
||||
) -> Hotspot:
|
||||
return Hotspot(
|
||||
path=path,
|
||||
function_name=function_name,
|
||||
score=score,
|
||||
band=band,
|
||||
signals={"loc": 10, "inventory_hit_count": len(inventory_finding_ids)},
|
||||
rationale=f"{band} band: score {score}",
|
||||
test_hint=None,
|
||||
inventory_finding_ids=inventory_finding_ids,
|
||||
)
|
||||
|
||||
|
||||
def _six_hotspots(*, sev5_id: str | None = None, sev5_score: float = 4.0) -> tuple[Hotspot, ...]:
|
||||
return (
|
||||
_hotspot("a.py", function_name="a", score=5.0),
|
||||
_hotspot("b.py", function_name="b", score=4.8),
|
||||
_hotspot("c.py", function_name="c", score=4.6),
|
||||
_hotspot("d.py", function_name="d", score=4.4),
|
||||
_hotspot("e.py", function_name="e", score=4.2),
|
||||
_hotspot(
|
||||
"f.py",
|
||||
function_name="f",
|
||||
score=sev5_score,
|
||||
inventory_finding_ids=(sev5_id,) if sev5_id else (),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def test_six_hotspots_max_candidates_three_returns_three() -> None:
|
||||
noise = _finding("f.py", "print:f.py:1", function_name="f", reachability="reachable")
|
||||
selected = select_candidates(
|
||||
_six_hotspots(),
|
||||
[noise],
|
||||
max_candidates=3,
|
||||
policy_severity_by_id={noise.finding_id: 2},
|
||||
)
|
||||
|
||||
assert len(selected) == 3
|
||||
assert [item.path for item in selected] == ["a.py", "b.py", "c.py"]
|
||||
|
||||
|
||||
def test_sev5_reachable_extra_included_when_ledger_demoted_hotspot() -> None:
|
||||
sev5_id = "cmd:f.py:1"
|
||||
hotspots = _six_hotspots(sev5_id=sev5_id, sev5_score=5.2)
|
||||
ledger = record_verdict({}, path="f.py", function_name="f", verdict="not_triggerable")
|
||||
demoted = apply_overlay(hotspots, ledger)
|
||||
finding = _finding(
|
||||
"f.py",
|
||||
sev5_id,
|
||||
function_name="f",
|
||||
reachability="reachable",
|
||||
severity="critical",
|
||||
)
|
||||
|
||||
selected = select_candidates(
|
||||
demoted,
|
||||
[finding],
|
||||
max_candidates=3,
|
||||
policy_severity_by_id={sev5_id: 5},
|
||||
)
|
||||
|
||||
demoted_f = next(item for item in demoted if item.path == "f.py")
|
||||
assert demoted_f.score == round(5.2 + NOT_TRIGGERABLE_DELTA, 4)
|
||||
assert [item.path for item in demoted[:3]] == ["a.py", "b.py", "c.py"]
|
||||
assert demoted_f not in demoted[:3]
|
||||
assert len(selected) == 4
|
||||
assert [item.path for item in selected[:3]] == ["a.py", "b.py", "c.py"]
|
||||
assert selected[3] is demoted_f
|
||||
assert sev5_id in selected[3].inventory_finding_ids
|
||||
|
||||
|
||||
def test_sev5_already_in_cap_is_not_duplicated() -> None:
|
||||
sev5_id = "cmd:a.py:1"
|
||||
hotspots = (
|
||||
_hotspot("a.py", function_name="a", score=5.0, inventory_finding_ids=(sev5_id,)),
|
||||
_hotspot("b.py", function_name="b", score=4.8),
|
||||
_hotspot("c.py", function_name="c", score=4.6),
|
||||
_hotspot("d.py", function_name="d", score=4.4),
|
||||
_hotspot("e.py", function_name="e", score=4.2),
|
||||
_hotspot("f.py", function_name="f", score=4.0),
|
||||
)
|
||||
finding = _finding("a.py", sev5_id, function_name="a", reachability="reachable", severity="critical")
|
||||
|
||||
selected = select_candidates(
|
||||
hotspots,
|
||||
[finding],
|
||||
max_candidates=3,
|
||||
policy_severity_by_id={sev5_id: 5},
|
||||
)
|
||||
|
||||
assert len(selected) == 3
|
||||
assert [item.path for item in selected] == ["a.py", "b.py", "c.py"]
|
||||
assert sev5_id in selected[0].inventory_finding_ids
|
||||
|
||||
|
||||
def test_sev5_reachable_inventory_without_hotspot_is_still_selected() -> None:
|
||||
finding = _finding(
|
||||
"z.py",
|
||||
"cmd:z.py:1",
|
||||
function_name="z",
|
||||
reachability="reachable",
|
||||
severity="critical",
|
||||
)
|
||||
|
||||
selected = select_candidates(
|
||||
_six_hotspots(),
|
||||
[finding],
|
||||
max_candidates=3,
|
||||
policy_severity_by_id={finding.finding_id: 5},
|
||||
)
|
||||
|
||||
assert len(selected) == 4
|
||||
assert [item.path for item in selected[:3]] == ["a.py", "b.py", "c.py"]
|
||||
extra = selected[3]
|
||||
assert extra.path == "z.py"
|
||||
assert extra.function_name == "z"
|
||||
assert extra.inventory_finding_ids == (finding.finding_id,)
|
||||
@@ -0,0 +1,103 @@
|
||||
"""Language recipes for isolated regression snippets (task 5.2)."""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from openultrasast.config import SandboxConfig
|
||||
from openultrasast.regress import INCONCLUSIVE, RegressionRunner
|
||||
from openultrasast.regress.recipes import recipe_for
|
||||
from openultrasast.sandbox import FakeSandboxRunner, SandboxJob
|
||||
|
||||
LIMITS = SandboxConfig(memory_mb=256, timeout_seconds=30, pids_limit=128)
|
||||
PYTHON_SNIPPET = "assert 1 + 1 == 2\n"
|
||||
JS_SNIPPET = "console.log('ok')\n"
|
||||
C_SNIPPET = "int main(void) { return 0; }\n"
|
||||
JAVA_SNIPPET = "class Case { public static void main(String[] args) {} }\n"
|
||||
|
||||
|
||||
def _repo(tmp_path: Path) -> Path:
|
||||
repo = tmp_path / "repo"
|
||||
repo.mkdir()
|
||||
return repo
|
||||
|
||||
|
||||
def test_python_recipe_runs_scratch_snippet_with_repo_on_pythonpath(tmp_path: Path) -> None:
|
||||
repo = _repo(tmp_path)
|
||||
job = recipe_for("python", PYTHON_SNIPPET, "python:3.12-alpine", LIMITS, repo_root=repo)
|
||||
|
||||
assert isinstance(job, SandboxJob)
|
||||
joined = " ".join(job.command)
|
||||
assert job.image == "python:3.12-alpine"
|
||||
assert job.repo_root == repo
|
||||
assert job.scratch_files == {"case.py": PYTHON_SNIPPET}
|
||||
assert "python" in job.command
|
||||
assert "/scratch/case.py" in job.command
|
||||
assert "PYTHONPATH=/workspace" in joined
|
||||
assert job.timeout_seconds == LIMITS.timeout_seconds
|
||||
assert job.memory_mb == LIMITS.memory_mb
|
||||
assert job.pids_limit == LIMITS.pids_limit
|
||||
|
||||
|
||||
def test_javascript_recipe_runs_scratch_snippet_with_repo_on_node_path(tmp_path: Path) -> None:
|
||||
repo = _repo(tmp_path)
|
||||
job = recipe_for("javascript", JS_SNIPPET, "node:22-alpine", LIMITS, repo_root=repo)
|
||||
|
||||
assert isinstance(job, SandboxJob)
|
||||
joined = " ".join(job.command)
|
||||
assert job.image == "node:22-alpine"
|
||||
assert job.repo_root == repo
|
||||
assert job.scratch_files == {"case.js": JS_SNIPPET}
|
||||
assert "node" in job.command
|
||||
assert "/scratch/case.js" in job.command
|
||||
assert "NODE_PATH=/workspace" in joined
|
||||
assert job.timeout_seconds == LIMITS.timeout_seconds
|
||||
assert job.memory_mb == LIMITS.memory_mb
|
||||
assert job.pids_limit == LIMITS.pids_limit
|
||||
|
||||
|
||||
def test_c_recipe_compiles_and_runs_when_image_has_compiler(tmp_path: Path) -> None:
|
||||
repo = _repo(tmp_path)
|
||||
job = recipe_for("c", C_SNIPPET, "gcc:13", LIMITS, repo_root=repo)
|
||||
|
||||
assert isinstance(job, SandboxJob)
|
||||
joined = " ".join(job.command)
|
||||
assert job.scratch_files == {"case.c": C_SNIPPET}
|
||||
assert "cc" in joined
|
||||
assert "/scratch/case.c" in joined
|
||||
assert "-o" in joined
|
||||
assert "/scratch/case" in joined
|
||||
|
||||
|
||||
def test_unknown_language_recipe_is_none(tmp_path: Path) -> None:
|
||||
repo = _repo(tmp_path)
|
||||
assert recipe_for("java", JAVA_SNIPPET, "eclipse-temurin:21", LIMITS, repo_root=repo) is None
|
||||
assert recipe_for("unknown", "print(1)\n", "python:3.12-alpine", LIMITS, repo_root=repo) is None
|
||||
|
||||
|
||||
def test_python_and_javascript_recipes_produce_runner_jobs(tmp_path: Path) -> None:
|
||||
repo = _repo(tmp_path)
|
||||
python_job = recipe_for("python", PYTHON_SNIPPET, "python:3.12-alpine", LIMITS, repo_root=repo)
|
||||
js_job = recipe_for("javascript", JS_SNIPPET, "node:22-alpine", LIMITS, repo_root=repo)
|
||||
sandbox = FakeSandboxRunner()
|
||||
runner = RegressionRunner(sandbox)
|
||||
|
||||
python_verdict = runner.run_recipe("python", PYTHON_SNIPPET, "python:3.12-alpine", LIMITS, repo_root=repo)
|
||||
js_verdict = runner.run_recipe("javascript", JS_SNIPPET, "node:22-alpine", LIMITS, repo_root=repo)
|
||||
|
||||
assert python_job is not None
|
||||
assert js_job is not None
|
||||
assert python_verdict.reason != "missing_recipe"
|
||||
assert js_verdict.reason != "missing_recipe"
|
||||
assert sandbox.jobs == [python_job, js_job]
|
||||
|
||||
|
||||
def test_unknown_language_returns_inconclusive_without_calling_runner(tmp_path: Path) -> None:
|
||||
repo = _repo(tmp_path)
|
||||
sandbox = FakeSandboxRunner()
|
||||
runner = RegressionRunner(sandbox)
|
||||
|
||||
assert recipe_for("java", JAVA_SNIPPET, "eclipse-temurin:21", LIMITS, repo_root=repo) is None
|
||||
verdict = runner.run_recipe("java", JAVA_SNIPPET, "eclipse-temurin:21", LIMITS, repo_root=repo)
|
||||
|
||||
assert verdict.verdict == INCONCLUSIVE
|
||||
assert verdict.reason == "missing_recipe"
|
||||
assert sandbox.jobs == []
|
||||
@@ -0,0 +1,86 @@
|
||||
"""Reject unsafe regression snippets before the sandbox runner sees them (task 4.3)."""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from openultrasast.regress import RegressionRunner, UnsafeSnippetError, check_snippet_safety
|
||||
from openultrasast.sandbox import FakeSandboxRunner, SandboxJob
|
||||
|
||||
BENIGN_ASSERT = "assert 1 + 1 == 2\n"
|
||||
|
||||
|
||||
def _job(tmp_path: Path, snippet: str) -> SandboxJob:
|
||||
repo = tmp_path / "repo"
|
||||
repo.mkdir()
|
||||
return SandboxJob(
|
||||
image="python:3.12-alpine",
|
||||
command=("python", "/scratch/case.py"),
|
||||
repo_root=repo,
|
||||
scratch_files={"case.py": snippet},
|
||||
timeout_seconds=30,
|
||||
memory_mb=256,
|
||||
pids_limit=128,
|
||||
)
|
||||
|
||||
|
||||
def test_docker_sock_rejected_does_not_create_runner_job(tmp_path: Path) -> None:
|
||||
snippet = 'client = docker.DockerClient(base_url="unix://var/run/docker.sock")\n'
|
||||
sandbox = FakeSandboxRunner()
|
||||
runner = RegressionRunner(sandbox)
|
||||
|
||||
with pytest.raises(UnsafeSnippetError, match="Docker socket"):
|
||||
check_snippet_safety(snippet)
|
||||
verdict = runner.run_snippet(snippet, _job(tmp_path, snippet))
|
||||
|
||||
assert verdict.verdict == "inconclusive"
|
||||
assert verdict.reason == "safety_rejected"
|
||||
assert sandbox.jobs == []
|
||||
|
||||
|
||||
def test_host_network_rejected_does_not_create_runner_job(tmp_path: Path) -> None:
|
||||
snippet = "subprocess.run(['docker', 'run', '--network', 'host', 'alpine'])\n"
|
||||
sandbox = FakeSandboxRunner()
|
||||
runner = RegressionRunner(sandbox)
|
||||
|
||||
with pytest.raises(UnsafeSnippetError, match="host network"):
|
||||
check_snippet_safety(snippet)
|
||||
verdict = runner.run_snippet(snippet, _job(tmp_path, snippet))
|
||||
|
||||
assert verdict.verdict == "inconclusive"
|
||||
assert verdict.reason == "safety_rejected"
|
||||
assert sandbox.jobs == []
|
||||
|
||||
|
||||
def test_write_under_workspace_rejected_does_not_create_runner_job(tmp_path: Path) -> None:
|
||||
snippet = 'open("/workspace/pwned.py", "w").write("owned")\n'
|
||||
sandbox = FakeSandboxRunner()
|
||||
runner = RegressionRunner(sandbox)
|
||||
|
||||
with pytest.raises(UnsafeSnippetError, match="/workspace"):
|
||||
check_snippet_safety(snippet)
|
||||
verdict = runner.run_snippet(snippet, _job(tmp_path, snippet))
|
||||
|
||||
assert verdict.verdict == "inconclusive"
|
||||
assert verdict.reason == "safety_rejected"
|
||||
assert sandbox.jobs == []
|
||||
|
||||
|
||||
def test_benign_python_assert_snippet_is_allowed(tmp_path: Path) -> None:
|
||||
assert check_snippet_safety(BENIGN_ASSERT) is None
|
||||
sandbox = FakeSandboxRunner()
|
||||
job = _job(tmp_path, BENIGN_ASSERT)
|
||||
RegressionRunner(sandbox).run_snippet(BENIGN_ASSERT, job)
|
||||
assert sandbox.jobs == [job]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"snippet",
|
||||
[
|
||||
"import socket\nsocket.create_connection(('127.0.0.1', 80))\n",
|
||||
"curl https://example.invalid/pwn\n",
|
||||
],
|
||||
)
|
||||
def test_socket_and_curl_mentions_are_rejected(snippet: str) -> None:
|
||||
with pytest.raises(UnsafeSnippetError):
|
||||
check_snippet_safety(snippet)
|
||||
@@ -0,0 +1,201 @@
|
||||
"""Map sandbox outcomes to the four verdicts and the worth-fixing gate (task 5.3)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from openultrasast.regress import RegressionRunner
|
||||
from openultrasast.regress.verdict import (
|
||||
ALREADY_COVERED,
|
||||
INCONCLUSIVE,
|
||||
NOT_TRIGGERABLE,
|
||||
TRIGGERABLE,
|
||||
is_worth_fixing,
|
||||
verdict_from_result,
|
||||
)
|
||||
from openultrasast.sandbox import FakeSandboxRunner, SandboxJob, SandboxResult
|
||||
|
||||
_REACHABLE = "reachable"
|
||||
_INFERRED = "inferred-file-surface"
|
||||
_UNKNOWN = "unknown"
|
||||
|
||||
|
||||
def _result(*, exit_code: int = 0, stdout: str = "", stderr: str = "", timed_out: bool = False) -> SandboxResult:
|
||||
return SandboxResult(exit_code=exit_code, stdout=stdout, stderr=stderr, timed_out=timed_out)
|
||||
|
||||
|
||||
def _job(tmp_path: Path, snippet: str) -> SandboxJob:
|
||||
repo = tmp_path / "repo"
|
||||
repo.mkdir()
|
||||
return SandboxJob(
|
||||
image="python:3.12-alpine",
|
||||
command=("python", "/scratch/case.py"),
|
||||
repo_root=repo,
|
||||
scratch_files={"case.py": snippet},
|
||||
timeout_seconds=30,
|
||||
memory_mb=256,
|
||||
pids_limit=128,
|
||||
)
|
||||
|
||||
|
||||
# observation kwargs, sandbox result, reachability, expected verdict, expected reason, worth-fixing
|
||||
_CASES = (
|
||||
pytest.param(
|
||||
{"recipe_missing": True},
|
||||
None,
|
||||
_REACHABLE,
|
||||
INCONCLUSIVE,
|
||||
"missing_recipe",
|
||||
False,
|
||||
id="recipe-missing-inconclusive",
|
||||
),
|
||||
pytest.param(
|
||||
{"sandbox_missing": True},
|
||||
None,
|
||||
_REACHABLE,
|
||||
INCONCLUSIVE,
|
||||
"sandbox_unavailable",
|
||||
False,
|
||||
id="sandbox-missing-inconclusive",
|
||||
),
|
||||
pytest.param(
|
||||
{},
|
||||
_result(exit_code=-1, timed_out=True),
|
||||
_REACHABLE,
|
||||
INCONCLUSIVE,
|
||||
"timeout",
|
||||
False,
|
||||
id="timeout-inconclusive",
|
||||
),
|
||||
pytest.param(
|
||||
{"safety_rejected": True},
|
||||
None,
|
||||
_INFERRED,
|
||||
INCONCLUSIVE,
|
||||
"safety_rejected",
|
||||
False,
|
||||
id="safety-reject-inconclusive",
|
||||
),
|
||||
pytest.param(
|
||||
{"covering_test": True},
|
||||
_result(exit_code=0, stdout="ok\n"),
|
||||
_REACHABLE,
|
||||
ALREADY_COVERED,
|
||||
"covered_test_passed",
|
||||
False,
|
||||
id="covering-test-passed-already-covered",
|
||||
),
|
||||
pytest.param(
|
||||
{},
|
||||
_result(exit_code=1, stderr="AssertionError: boom\n"),
|
||||
_REACHABLE,
|
||||
TRIGGERABLE,
|
||||
"assertion",
|
||||
True,
|
||||
id="assertion-triggerable-reachable-worth-fixing",
|
||||
),
|
||||
pytest.param(
|
||||
{},
|
||||
_result(exit_code=1, stderr="ERROR: AddressSanitizer: heap-buffer-overflow\n"),
|
||||
_INFERRED,
|
||||
TRIGGERABLE,
|
||||
"sanitizer_abort",
|
||||
True,
|
||||
id="sanitizer-triggerable-inferred-worth-fixing",
|
||||
),
|
||||
pytest.param(
|
||||
{},
|
||||
_result(exit_code=139, stderr="Segmentation fault\n"),
|
||||
_UNKNOWN,
|
||||
TRIGGERABLE,
|
||||
"crash",
|
||||
False,
|
||||
id="crash-triggerable-unknown-not-worth-fixing",
|
||||
),
|
||||
pytest.param(
|
||||
{},
|
||||
_result(exit_code=0, stdout="ok\n"),
|
||||
_REACHABLE,
|
||||
NOT_TRIGGERABLE,
|
||||
"exit_zero",
|
||||
False,
|
||||
id="exit-zero-not-triggerable",
|
||||
),
|
||||
pytest.param(
|
||||
{"covering_test": True},
|
||||
_result(exit_code=0),
|
||||
_UNKNOWN,
|
||||
ALREADY_COVERED,
|
||||
"covered_test_passed",
|
||||
False,
|
||||
id="already-covered-unknown-does-not-fail-build",
|
||||
),
|
||||
pytest.param(
|
||||
{},
|
||||
_result(exit_code=0),
|
||||
_INFERRED,
|
||||
NOT_TRIGGERABLE,
|
||||
"exit_zero",
|
||||
False,
|
||||
id="not-triggerable-inferred-does-not-fail-build",
|
||||
),
|
||||
pytest.param(
|
||||
{},
|
||||
_result(exit_code=-1, timed_out=True),
|
||||
_INFERRED,
|
||||
INCONCLUSIVE,
|
||||
"timeout",
|
||||
False,
|
||||
id="inconclusive-inferred-does-not-fail-build",
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("flags", "result", "reachability", "expected_verdict", "expected_reason", "expected_worth_fixing"),
|
||||
_CASES,
|
||||
)
|
||||
def test_sandbox_outcomes_map_to_verdicts_and_worth_fixing(
|
||||
flags: dict[str, bool],
|
||||
result: SandboxResult | None,
|
||||
reachability: str,
|
||||
expected_verdict: str,
|
||||
expected_reason: str,
|
||||
expected_worth_fixing: bool,
|
||||
) -> None:
|
||||
mapped = verdict_from_result(result, **flags)
|
||||
|
||||
assert mapped.verdict == expected_verdict
|
||||
assert mapped.reason == expected_reason
|
||||
assert mapped.reason
|
||||
assert is_worth_fixing(mapped.verdict, reachability) is expected_worth_fixing
|
||||
|
||||
|
||||
def test_table_covers_four_verdicts_and_worth_fixing_boolean() -> None:
|
||||
verdicts = {case.values[3] for case in _CASES}
|
||||
worth_fixing_values = {case.values[5] for case in _CASES}
|
||||
|
||||
assert verdicts == {TRIGGERABLE, NOT_TRIGGERABLE, ALREADY_COVERED, INCONCLUSIVE}
|
||||
assert worth_fixing_values == {True, False}
|
||||
|
||||
|
||||
def test_runner_maps_timeout_crash_and_exit_zero(tmp_path: Path) -> None:
|
||||
snippet = "assert 1 + 1 == 2\n"
|
||||
job = _job(tmp_path, snippet)
|
||||
timeout = FakeSandboxRunner(_result(exit_code=-1, timed_out=True))
|
||||
crash = FakeSandboxRunner(_result(exit_code=1, stderr="AssertionError\n"))
|
||||
ok = FakeSandboxRunner(_result(exit_code=0))
|
||||
|
||||
timed_out = RegressionRunner(timeout).run_snippet(snippet, job)
|
||||
triggered = RegressionRunner(crash).run_snippet(snippet, job)
|
||||
not_triggered = RegressionRunner(ok).run_snippet(snippet, job)
|
||||
|
||||
assert timed_out.verdict == INCONCLUSIVE
|
||||
assert timed_out.reason == "timeout"
|
||||
assert triggered.verdict == TRIGGERABLE
|
||||
assert not_triggered.verdict == NOT_TRIGGERABLE
|
||||
assert is_worth_fixing(timed_out.verdict, _REACHABLE) is False
|
||||
assert is_worth_fixing(triggered.verdict, _REACHABLE) is True
|
||||
assert is_worth_fixing(not_triggered.verdict, _REACHABLE) is False
|
||||
@@ -1,10 +1,18 @@
|
||||
import json
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
|
||||
from openultrasast.cli import main
|
||||
from openultrasast.complexity.hints import TestHint
|
||||
from openultrasast.complexity.map import ComplexityMap, Hotspot
|
||||
from openultrasast.findings import StaticFinding
|
||||
from openultrasast.regress import CandidateVerdict
|
||||
from openultrasast.reports import scan_exit_code, write_manifest, write_markdown_report, write_sarif_report
|
||||
from openultrasast.run import ScanRun
|
||||
from openultrasast.stages import Stage, plan_for_mode, record_completed, stages_payload
|
||||
from openultrasast.verification import verify_finding
|
||||
|
||||
|
||||
@@ -47,6 +55,34 @@ def test_manifest_links_shared_artifacts_by_finding_id(tmp_path: Path) -> None:
|
||||
assert manifest_finding["finding_id"] == finding.finding_id
|
||||
assert manifest_finding["verification_status"] == "accepted"
|
||||
assert manifest_finding["artifact_refs"]["sarif"] == "report.sarif"
|
||||
assert "stages" not in payload
|
||||
|
||||
|
||||
def test_manifest_includes_requested_completed_skipped_stages(tmp_path: Path) -> None:
|
||||
run = ScanRun(scan_id="scan-1", root=tmp_path / "run", target=tmp_path / "repo")
|
||||
run.root.mkdir()
|
||||
finding = _finding()
|
||||
artifacts = {
|
||||
"findings": run.root / "findings.json",
|
||||
"verification": run.root / "verification.json",
|
||||
"markdown": run.root / "report.md",
|
||||
"sarif": run.root / "report.sarif",
|
||||
}
|
||||
output = run.root / "manifest.json"
|
||||
plan = record_completed(plan_for_mode("quick"), Stage.STATIC)
|
||||
|
||||
write_manifest(
|
||||
run=run,
|
||||
findings=[finding],
|
||||
verifications=[verify_finding(finding)],
|
||||
artifact_paths=artifacts,
|
||||
path=output,
|
||||
stages=stages_payload(plan),
|
||||
)
|
||||
|
||||
payload = json.loads(output.read_text())
|
||||
assert payload["stages"] == {"requested": ["static"], "completed": ["static"], "skipped": []}
|
||||
assert payload["findings"][0]["finding_id"] == finding.finding_id
|
||||
|
||||
|
||||
def test_markdown_report_includes_verification_status(tmp_path: Path) -> None:
|
||||
@@ -56,18 +92,131 @@ def test_markdown_report_includes_verification_status(tmp_path: Path) -> None:
|
||||
write_markdown_report([finding], output, [verify_finding(finding)])
|
||||
|
||||
text = output.read_text()
|
||||
assert "## Inventory" in text
|
||||
assert "Verification: `accepted`" in text
|
||||
assert finding.finding_id in text
|
||||
assert "worth-fixing" not in text.lower()
|
||||
assert "worth_fixing" not in text.lower()
|
||||
|
||||
|
||||
def test_markdown_report_includes_complexity_map_and_worth_fixing_sections(tmp_path: Path) -> None:
|
||||
finding = _finding()
|
||||
output = tmp_path / "report.md"
|
||||
hint = TestHint(
|
||||
path="app.py",
|
||||
function_name="admin",
|
||||
gap="no_adjacent_test",
|
||||
test_kind="http-contract",
|
||||
reason="no adjacent test file for app.py::admin; recommend http-contract.",
|
||||
)
|
||||
hotspot = Hotspot(
|
||||
path="app.py",
|
||||
function_name="admin",
|
||||
score=6.5,
|
||||
band="high",
|
||||
signals={"loc": 12, "has_adjacent_test": False},
|
||||
rationale="high band: nested public route",
|
||||
test_hint=hint,
|
||||
inventory_finding_ids=(finding.finding_id,),
|
||||
)
|
||||
verdict = CandidateVerdict(
|
||||
path="app.py",
|
||||
function_name="admin",
|
||||
language="python",
|
||||
verdict="triggerable",
|
||||
reason="crash",
|
||||
inventory_finding_ids=(finding.finding_id,),
|
||||
worth_fixing=True,
|
||||
)
|
||||
skipped = CandidateVerdict(
|
||||
path="safe.py",
|
||||
function_name=None,
|
||||
language="python",
|
||||
verdict="not_triggerable",
|
||||
reason="exit_zero",
|
||||
inventory_finding_ids=(),
|
||||
worth_fixing=False,
|
||||
)
|
||||
|
||||
write_markdown_report(
|
||||
[finding],
|
||||
output,
|
||||
[verify_finding(finding)],
|
||||
complexity_map=ComplexityMap(hotspots=(hotspot,), heuristic_only=True),
|
||||
verdicts=(verdict, skipped),
|
||||
)
|
||||
|
||||
text = output.read_text()
|
||||
inventory_at = text.index("## Inventory")
|
||||
map_at = text.index("## Complexity map")
|
||||
worth_at = text.index("## Worth fixing")
|
||||
assert inventory_at < map_at < worth_at
|
||||
map_section = text[map_at:worth_at]
|
||||
worth_section = text[worth_at:]
|
||||
assert "app.py::admin" in map_section
|
||||
assert "Gap: `no_adjacent_test`" in map_section
|
||||
assert "Recommended test: `http-contract`" in map_section
|
||||
assert "triggerable" in worth_section
|
||||
assert "crash" in worth_section
|
||||
assert "safe.py" not in worth_section
|
||||
assert "## Inventory" in text
|
||||
|
||||
|
||||
def test_markdown_report_labels_hits_as_inventory_not_worth_fixing(tmp_path: Path) -> None:
|
||||
finding = _finding()
|
||||
output = tmp_path / "report.md"
|
||||
|
||||
write_markdown_report([finding], output, [verify_finding(finding)])
|
||||
|
||||
text = output.read_text()
|
||||
inventory_at = text.index("## Inventory")
|
||||
finding_at = text.index(f"## {finding.title}")
|
||||
assert inventory_at < finding_at
|
||||
assert "worth-fixing" not in text.lower()
|
||||
|
||||
|
||||
def test_scan_exit_code_policy() -> None:
|
||||
finding = _finding()
|
||||
verification = verify_finding(finding)
|
||||
worth_fixing = SimpleNamespace(worth_fixing=True)
|
||||
|
||||
assert scan_exit_code([finding], [verification], "never") == 0
|
||||
assert scan_exit_code([finding], [verification], "findings") == 1
|
||||
assert scan_exit_code([], [], "findings") == 0
|
||||
assert scan_exit_code([finding], [verification], "verified") == 1
|
||||
assert scan_exit_code([], [], "verified") == 0
|
||||
# findings/verified ignore worth-fixing verdicts
|
||||
assert scan_exit_code([], [], "findings", worth_fixing_verdicts=[worth_fixing]) == 0
|
||||
assert scan_exit_code([], [], "verified", worth_fixing_verdicts=[worth_fixing]) == 0
|
||||
assert scan_exit_code([finding], [verification], "findings", worth_fixing_verdicts=[worth_fixing]) == 1
|
||||
assert scan_exit_code([finding], [verification], "verified", worth_fixing_verdicts=[worth_fixing]) == 1
|
||||
|
||||
|
||||
def test_scan_exit_code_worth_fixing() -> None:
|
||||
finding = _finding()
|
||||
verification = verify_finding(finding)
|
||||
worth_fixing = SimpleNamespace(worth_fixing=True)
|
||||
not_worth_fixing = SimpleNamespace(worth_fixing=False)
|
||||
|
||||
assert scan_exit_code([finding], [verification], "worth-fixing") == 0
|
||||
assert scan_exit_code([finding], [verification], "worth-fixing", worth_fixing_verdicts=[]) == 0
|
||||
assert scan_exit_code([finding], [verification], "worth-fixing", worth_fixing_verdicts=[not_worth_fixing]) == 0
|
||||
assert scan_exit_code([finding], [verification], "worth-fixing", worth_fixing_verdicts=[worth_fixing]) == 1
|
||||
|
||||
|
||||
def test_scan_exit_code_unknown_policy() -> None:
|
||||
with pytest.raises(ValueError, match="unknown fail policy"):
|
||||
scan_exit_code([], [], "maybe")
|
||||
|
||||
|
||||
def test_scan_help_lists_worth_fixing(capsys: pytest.CaptureFixture[str]) -> None:
|
||||
with pytest.raises(SystemExit) as exc:
|
||||
main(["scan", "--help"])
|
||||
assert exc.value.code == 0
|
||||
help_text = capsys.readouterr().out
|
||||
assert "worth-fixing" in help_text
|
||||
assert "findings" in help_text
|
||||
assert "verified" in help_text
|
||||
|
||||
|
||||
def test_cli_scan_writes_sarif_manifest_and_honors_fail_policy(tmp_path: Path, monkeypatch) -> None: # type: ignore[no-untyped-def]
|
||||
@@ -87,6 +236,210 @@ def test_cli_scan_writes_sarif_manifest_and_honors_fail_policy(tmp_path: Path, m
|
||||
assert manifest["findings"][0]["artifact_refs"]["verification_json"] == "verification.json"
|
||||
|
||||
|
||||
def test_cli_scan_fail_on_worth_fixing_exits_zero_without_verdicts(tmp_path: Path, monkeypatch) -> None: # type: ignore[no-untyped-def]
|
||||
repo = tmp_path / "repo"
|
||||
repo.mkdir()
|
||||
(repo / "app.py").write_text("@app.route('/admin')\ndef admin():\n return eval(request.data)\n")
|
||||
monkeypatch.setenv("OPENULTRASAST_RUNS_DIR", ".runs")
|
||||
|
||||
assert main(["scan", str(repo), "--mode", "quick", "--fail-on", "worth-fixing"]) == 0
|
||||
|
||||
|
||||
def test_quick_scan_report_is_inventory_without_map_or_docker(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
from openultrasast import tool_hunter
|
||||
from openultrasast.complexity import map as complexity_map
|
||||
from openultrasast.sandbox import probe as sandbox_probe
|
||||
from openultrasast.sandbox import runner as sandbox_runner
|
||||
|
||||
repo = tmp_path / "repo"
|
||||
repo.mkdir()
|
||||
(repo / "app.py").write_text("@app.route('/admin')\ndef admin():\n return eval(request.data)\n")
|
||||
monkeypatch.setenv("OPENULTRASAST_RUNS_DIR", ".runs")
|
||||
|
||||
docker_argv: list[list[str]] = []
|
||||
original_run = subprocess.run
|
||||
|
||||
def guarded_run(command, *args, **kwargs): # type: ignore[no-untyped-def]
|
||||
argv = list(command) if isinstance(command, list | tuple) else [command]
|
||||
if argv and Path(str(argv[0])).name == "docker":
|
||||
docker_argv.append(argv)
|
||||
raise AssertionError(f"quick scan must not start docker: {argv}")
|
||||
return original_run(command, *args, **kwargs)
|
||||
|
||||
def fail(message: str): # type: ignore[no-untyped-def]
|
||||
def _fail(*args, **kwargs): # type: ignore[no-untyped-def]
|
||||
raise AssertionError(message)
|
||||
|
||||
return _fail
|
||||
|
||||
monkeypatch.setattr(subprocess, "run", guarded_run)
|
||||
monkeypatch.setattr(sandbox_runner, "build_docker_argv", fail("quick scan must not build docker argv"))
|
||||
monkeypatch.setattr(sandbox_probe.SandboxProbe, "available", fail("quick scan must not probe docker"))
|
||||
monkeypatch.setattr(complexity_map, "build_complexity_map", fail("quick scan must not build a complexity map"))
|
||||
monkeypatch.setattr(tool_hunter, "run_tool_hunter", fail("quick scan must not start a hunter model"))
|
||||
|
||||
assert main(["scan", str(repo), "--mode", "quick"]) == 0
|
||||
|
||||
run_dir = sorted((repo / ".runs").iterdir())[-1]
|
||||
report = (run_dir / "report.md").read_text()
|
||||
manifest = json.loads((run_dir / "manifest.json").read_text())
|
||||
|
||||
assert "## Inventory" in report
|
||||
assert "worth-fixing" not in report.lower()
|
||||
assert not (run_dir / "complexity_map.json").exists()
|
||||
assert docker_argv == []
|
||||
assert manifest["stages"]["requested"] == ["static"]
|
||||
assert manifest["stages"]["completed"] == ["static"]
|
||||
assert manifest["stages"]["skipped"] == []
|
||||
assert "complexity" not in manifest
|
||||
assert "complexity_map" not in manifest.get("artifacts", {})
|
||||
assert not any(entry.get("reason") == "hunter_model_unavailable" for entry in manifest.get("degradations", []))
|
||||
|
||||
|
||||
def test_standard_scan_writes_complexity_map_without_docker(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
from openultrasast.sandbox import probe as sandbox_probe
|
||||
from openultrasast.sandbox import runner as sandbox_runner
|
||||
|
||||
repo = tmp_path / "repo"
|
||||
repo.mkdir()
|
||||
(repo / "app.py").write_text("@app.route('/admin')\ndef admin():\n return eval(request.data)\n")
|
||||
monkeypatch.setenv("OPENULTRASAST_RUNS_DIR", ".runs")
|
||||
|
||||
docker_argv: list[list[str]] = []
|
||||
original_run = subprocess.run
|
||||
|
||||
def guarded_run(command, *args, **kwargs): # type: ignore[no-untyped-def]
|
||||
argv = list(command) if isinstance(command, list | tuple) else [command]
|
||||
if argv and Path(str(argv[0])).name == "docker":
|
||||
docker_argv.append(argv)
|
||||
raise AssertionError(f"standard scan must not start docker: {argv}")
|
||||
return original_run(command, *args, **kwargs)
|
||||
|
||||
def fail(message: str): # type: ignore[no-untyped-def]
|
||||
def _fail(*args, **kwargs): # type: ignore[no-untyped-def]
|
||||
raise AssertionError(message)
|
||||
|
||||
return _fail
|
||||
|
||||
monkeypatch.setattr(subprocess, "run", guarded_run)
|
||||
monkeypatch.setattr(sandbox_runner, "build_docker_argv", fail("standard scan must not build docker argv"))
|
||||
monkeypatch.setattr(sandbox_probe.SandboxProbe, "available", fail("standard scan must not probe docker"))
|
||||
|
||||
assert main(["scan", str(repo), "--mode", "standard"]) == 0
|
||||
|
||||
run_dir = sorted((repo / ".runs").iterdir())[-1]
|
||||
map_path = run_dir / "complexity_map.json"
|
||||
manifest = json.loads((run_dir / "manifest.json").read_text())
|
||||
complexity_map = json.loads(map_path.read_text())
|
||||
|
||||
assert map_path.is_file()
|
||||
assert complexity_map["heuristic_only"] is True
|
||||
assert isinstance(complexity_map["hotspots"], list)
|
||||
assert complexity_map["hotspots"]
|
||||
assert docker_argv == []
|
||||
assert manifest["stages"]["requested"] == ["static", "map"]
|
||||
assert manifest["stages"]["completed"] == ["static", "map"]
|
||||
assert manifest["stages"]["skipped"] == []
|
||||
assert manifest["complexity"]["hotspot_count"] == len(complexity_map["hotspots"])
|
||||
assert manifest["complexity"]["heuristic_only"] is True
|
||||
assert manifest["artifacts"]["complexity_map"] == "complexity_map.json"
|
||||
report = (run_dir / "report.md").read_text()
|
||||
assert "## Inventory" in report
|
||||
assert "## Complexity map" in report
|
||||
assert "Gap:" in report
|
||||
assert "## Worth fixing" not in report
|
||||
|
||||
|
||||
def test_standard_scan_records_hunter_model_unavailable_when_model_unset(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
from openultrasast.sandbox import probe as sandbox_probe
|
||||
from openultrasast.sandbox import runner as sandbox_runner
|
||||
|
||||
repo = tmp_path / "repo"
|
||||
repo.mkdir()
|
||||
(repo / "app.py").write_text("@app.route('/admin')\ndef admin():\n return eval(request.data)\n")
|
||||
monkeypatch.setenv("OPENULTRASAST_RUNS_DIR", ".runs")
|
||||
|
||||
def fail(message: str): # type: ignore[no-untyped-def]
|
||||
def _fail(*args, **kwargs): # type: ignore[no-untyped-def]
|
||||
raise AssertionError(message)
|
||||
|
||||
return _fail
|
||||
|
||||
monkeypatch.setattr(sandbox_runner, "build_docker_argv", fail("standard scan must not build docker argv"))
|
||||
monkeypatch.setattr(sandbox_probe.SandboxProbe, "available", fail("standard scan must not probe docker"))
|
||||
|
||||
assert main(["scan", str(repo), "--mode", "standard"]) == 0
|
||||
|
||||
run_dir = sorted((repo / ".runs").iterdir())[-1]
|
||||
manifest = json.loads((run_dir / "manifest.json").read_text())
|
||||
degradations = manifest.get("degradations", [])
|
||||
|
||||
assert any(entry.get("reason") == "hunter_model_unavailable" for entry in degradations)
|
||||
hunter_skip = next(entry for entry in degradations if entry.get("reason") == "hunter_model_unavailable")
|
||||
assert hunter_skip["stage"] in {"map", "hunter_pool"}
|
||||
assert manifest["complexity"]["heuristic_only"] is True
|
||||
assert (run_dir / "complexity_map.json").is_file()
|
||||
|
||||
|
||||
def test_manifest_includes_complexity_summary(tmp_path: Path) -> None:
|
||||
run = ScanRun(scan_id="scan-1", root=tmp_path / "run", target=tmp_path / "repo")
|
||||
run.root.mkdir()
|
||||
finding = _finding()
|
||||
artifacts = {
|
||||
"findings": run.root / "findings.json",
|
||||
"verification": run.root / "verification.json",
|
||||
"markdown": run.root / "report.md",
|
||||
"sarif": run.root / "report.sarif",
|
||||
"complexity_map": run.root / "complexity_map.json",
|
||||
}
|
||||
output = run.root / "manifest.json"
|
||||
|
||||
write_manifest(
|
||||
run=run,
|
||||
findings=[finding],
|
||||
verifications=[verify_finding(finding)],
|
||||
artifact_paths=artifacts,
|
||||
path=output,
|
||||
complexity={"hotspot_count": 3, "heuristic_only": True},
|
||||
)
|
||||
|
||||
payload = json.loads(output.read_text())
|
||||
assert payload["complexity"] == {"hotspot_count": 3, "heuristic_only": True}
|
||||
assert payload["artifacts"]["complexity_map"] == "complexity_map.json"
|
||||
|
||||
|
||||
def test_manifest_includes_worth_fixing_count(tmp_path: Path) -> None:
|
||||
run = ScanRun(scan_id="scan-1", root=tmp_path / "run", target=tmp_path / "repo")
|
||||
run.root.mkdir()
|
||||
finding = _finding()
|
||||
artifacts = {
|
||||
"findings": run.root / "findings.json",
|
||||
"verification": run.root / "verification.json",
|
||||
"markdown": run.root / "report.md",
|
||||
"sarif": run.root / "report.sarif",
|
||||
"verdicts": run.root / "verdicts.json",
|
||||
}
|
||||
output = run.root / "manifest.json"
|
||||
|
||||
write_manifest(
|
||||
run=run,
|
||||
findings=[finding],
|
||||
verifications=[verify_finding(finding)],
|
||||
artifact_paths=artifacts,
|
||||
path=output,
|
||||
stages={"requested": ["static", "map", "regress"], "completed": ["static", "map", "regress"], "skipped": []},
|
||||
worth_fixing={
|
||||
"count": 1,
|
||||
"verdicts": [{"path": "app.py", "function_name": "admin", "verdict": "triggerable", "reason": "crash"}],
|
||||
},
|
||||
)
|
||||
|
||||
payload = json.loads(output.read_text())
|
||||
assert payload["stages"]["completed"] == ["static", "map", "regress"]
|
||||
assert payload["worth_fixing"]["count"] == 1
|
||||
assert payload["worth_fixing"]["verdicts"][0]["verdict"] == "triggerable"
|
||||
|
||||
|
||||
def _finding() -> StaticFinding:
|
||||
return StaticFinding(
|
||||
finding_id="python-unsafe-eval:app.py:3",
|
||||
|
||||
@@ -0,0 +1,117 @@
|
||||
import subprocess
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
from openultrasast.sandbox import FakeSandboxProbe, SandboxProbe, resolve_sandbox_probe
|
||||
from openultrasast.sandbox import probe as probe_mod
|
||||
|
||||
|
||||
def _completed(returncode: int) -> subprocess.CompletedProcess[str]:
|
||||
return subprocess.CompletedProcess(["docker", "info"], returncode, stdout="", stderr="")
|
||||
|
||||
|
||||
def test_fake_probe_can_be_forced_off() -> None:
|
||||
probe = FakeSandboxProbe(available=False)
|
||||
assert probe.available() is False
|
||||
|
||||
|
||||
def test_fake_probe_can_be_forced_on() -> None:
|
||||
probe = FakeSandboxProbe(available=True)
|
||||
assert probe.available() is True
|
||||
|
||||
|
||||
def test_resolve_probe_forced_off_via_env(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setenv(probe_mod.PROBE_ENV, "0")
|
||||
probe = resolve_sandbox_probe()
|
||||
assert isinstance(probe, FakeSandboxProbe)
|
||||
assert probe.available() is False
|
||||
|
||||
|
||||
def test_resolve_probe_forced_on_via_env(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setenv(probe_mod.PROBE_ENV, "1")
|
||||
probe = resolve_sandbox_probe()
|
||||
assert isinstance(probe, FakeSandboxProbe)
|
||||
assert probe.available() is True
|
||||
|
||||
|
||||
def test_resolve_probe_defaults_to_real_probe(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.delenv(probe_mod.PROBE_ENV, raising=False)
|
||||
assert isinstance(resolve_sandbox_probe(), SandboxProbe)
|
||||
|
||||
|
||||
def test_real_probe_is_available_when_docker_info_succeeds() -> None:
|
||||
def runner(*_args: object, **_kwargs: object) -> subprocess.CompletedProcess[str]:
|
||||
return _completed(0)
|
||||
|
||||
assert SandboxProbe(runner=runner).available() is True
|
||||
|
||||
|
||||
def test_real_probe_is_unavailable_when_docker_info_fails() -> None:
|
||||
def runner(*_args: object, **_kwargs: object) -> subprocess.CompletedProcess[str]:
|
||||
return _completed(1)
|
||||
|
||||
assert SandboxProbe(runner=runner).available() is False
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"error",
|
||||
[
|
||||
FileNotFoundError("docker"),
|
||||
PermissionError("denied"),
|
||||
OSError("cannot talk to daemon"),
|
||||
subprocess.TimeoutExpired(cmd=["docker", "info"], timeout=5),
|
||||
RuntimeError("unexpected probe failure"),
|
||||
],
|
||||
)
|
||||
def test_real_probe_returns_unavailable_without_raising(error: Exception) -> None:
|
||||
def runner(*_args: object, **_kwargs: object) -> subprocess.CompletedProcess[str]:
|
||||
raise error
|
||||
|
||||
assert SandboxProbe(runner=runner).available() is False
|
||||
|
||||
|
||||
def test_real_probe_runs_docker_info_with_short_timeout() -> None:
|
||||
seen: dict[str, Any] = {}
|
||||
|
||||
def runner(command: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]:
|
||||
seen["command"] = list(command)
|
||||
seen["timeout"] = kwargs.get("timeout")
|
||||
seen["check"] = kwargs.get("check")
|
||||
return _completed(0)
|
||||
|
||||
assert SandboxProbe(runner=runner).available() is True
|
||||
assert seen["command"] == ["docker", "info"]
|
||||
assert seen["timeout"] == probe_mod.PROBE_TIMEOUT_SECONDS
|
||||
assert probe_mod.PROBE_TIMEOUT_SECONDS <= 10
|
||||
assert seen["check"] is False
|
||||
|
||||
|
||||
def test_real_probe_is_skippable_in_unit_tests(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
def boom(*_args: object, **_kwargs: object) -> subprocess.CompletedProcess[str]:
|
||||
raise AssertionError("unit tests must not invoke the docker daemon")
|
||||
|
||||
monkeypatch.setattr(probe_mod.subprocess, "run", boom)
|
||||
|
||||
def runner(*_args: object, **_kwargs: object) -> subprocess.CompletedProcess[str]:
|
||||
return _completed(0)
|
||||
|
||||
assert SandboxProbe(runner=runner).available() is True
|
||||
|
||||
|
||||
def test_real_probe_default_path_uses_subprocess_run(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
seen: dict[str, Any] = {}
|
||||
|
||||
def fake_run(command: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]:
|
||||
seen["command"] = list(command)
|
||||
seen["timeout"] = kwargs.get("timeout")
|
||||
seen["capture_output"] = kwargs.get("capture_output")
|
||||
seen["text"] = kwargs.get("text")
|
||||
return _completed(0)
|
||||
|
||||
monkeypatch.setattr(probe_mod.subprocess, "run", fake_run)
|
||||
assert SandboxProbe().available() is True
|
||||
assert seen["command"] == ["docker", "info"]
|
||||
assert seen["timeout"] == probe_mod.PROBE_TIMEOUT_SECONDS
|
||||
assert seen["capture_output"] is True
|
||||
assert seen["text"] is True
|
||||
@@ -0,0 +1,364 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import stat
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
from openultrasast.sandbox import (
|
||||
DockerCliRunner,
|
||||
FakeSandboxRunner,
|
||||
SandboxJob,
|
||||
SandboxResult,
|
||||
SandboxRunner,
|
||||
build_docker_argv,
|
||||
resolve_sandbox_runner,
|
||||
)
|
||||
from openultrasast.sandbox import runner as runner_mod
|
||||
|
||||
|
||||
def _job(tmp_path: Path, **overrides: object) -> SandboxJob:
|
||||
repo = tmp_path / "repo"
|
||||
repo.mkdir(exist_ok=True)
|
||||
(repo / "app.py").write_text("print(1)\n")
|
||||
values: dict[str, object] = {
|
||||
"image": "python:3.12-alpine",
|
||||
"command": ("python", "/scratch/case.py"),
|
||||
"repo_root": repo,
|
||||
"scratch_files": {"case.py": "print('ok')\n"},
|
||||
"timeout_seconds": 30,
|
||||
"memory_mb": 256,
|
||||
"pids_limit": 128,
|
||||
}
|
||||
values.update(overrides)
|
||||
return SandboxJob(**values) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def _flag_value(argv: list[str], flag: str) -> str:
|
||||
try:
|
||||
return argv[argv.index(flag) + 1]
|
||||
except (ValueError, IndexError) as exc:
|
||||
raise AssertionError(f"{flag} missing from {argv}") from exc
|
||||
|
||||
|
||||
def _mount_options(spec: str) -> dict[str, str]:
|
||||
options: dict[str, str] = {}
|
||||
for part in spec.split(","):
|
||||
if "=" in part:
|
||||
key, value = part.split("=", 1)
|
||||
options[key] = value
|
||||
else:
|
||||
options[part] = "true"
|
||||
return options
|
||||
|
||||
|
||||
def _workspace_mount(argv: list[str]) -> dict[str, str]:
|
||||
specs: list[str] = []
|
||||
for index, item in enumerate(argv):
|
||||
if item in {"--mount", "-v", "--volume"} and index + 1 < len(argv):
|
||||
specs.append(argv[index + 1])
|
||||
for spec in specs:
|
||||
options = _mount_options(spec)
|
||||
target = options.get("dst") or options.get("destination") or options.get("target")
|
||||
if target == "/workspace" or spec.endswith(":/workspace") or ":/workspace:" in spec:
|
||||
return options if target == "/workspace" else {"raw": spec}
|
||||
if "/workspace" in spec:
|
||||
return options if options else {"raw": spec}
|
||||
raise AssertionError(f"no /workspace mount in {argv}")
|
||||
|
||||
|
||||
def _seed_source(argv: list[str]) -> Path:
|
||||
mounts = [argv[index + 1] for index, item in enumerate(argv) if item == "--mount" and index + 1 < len(argv)]
|
||||
seed_mounts = [spec for spec in mounts if "/scratch-in" in spec]
|
||||
assert seed_mounts, argv
|
||||
options = _mount_options(seed_mounts[0])
|
||||
source = Path(options.get("src") or options.get("source") or "")
|
||||
assert source.is_dir(), argv
|
||||
return source
|
||||
|
||||
|
||||
def _copy_script(argv: list[str]) -> str:
|
||||
try:
|
||||
return argv[argv.index("-c") + 1]
|
||||
except (ValueError, IndexError) as exc:
|
||||
raise AssertionError(f"no /bin/sh -c wrapper in {argv}") from exc
|
||||
|
||||
|
||||
def _posix_mode(path: Path) -> int:
|
||||
return path.stat().st_mode & 0o777
|
||||
|
||||
|
||||
def test_build_docker_argv_starts_with_docker_run_rm_init(tmp_path: Path) -> None:
|
||||
argv = build_docker_argv(_job(tmp_path))
|
||||
assert argv[0] == "docker"
|
||||
assert argv[1] == "run"
|
||||
assert "--rm" in argv
|
||||
assert "--init" in argv
|
||||
|
||||
|
||||
def test_build_docker_argv_includes_network_none_and_read_only(tmp_path: Path) -> None:
|
||||
argv = build_docker_argv(_job(tmp_path))
|
||||
joined = " ".join(argv)
|
||||
assert "--read-only" in argv
|
||||
assert "--network none" in joined or "--network=none" in argv
|
||||
assert "--network host" not in joined
|
||||
assert "--network=host" not in argv
|
||||
|
||||
|
||||
def test_build_docker_argv_excludes_docker_sock(tmp_path: Path) -> None:
|
||||
argv = build_docker_argv(_job(tmp_path))
|
||||
joined = " ".join(argv)
|
||||
assert "docker.sock" not in joined
|
||||
assert "/var/run/docker.sock" not in joined
|
||||
assert "-v" not in argv
|
||||
assert "--volume" not in argv
|
||||
|
||||
|
||||
def test_build_docker_argv_forbids_privileged_and_host_network(tmp_path: Path) -> None:
|
||||
argv = build_docker_argv(_job(tmp_path))
|
||||
assert "--privileged" not in argv
|
||||
assert "--pid" not in argv or _flag_value(argv, "--pid") != "host"
|
||||
assert "--ipc" not in argv or _flag_value(argv, "--ipc") != "host"
|
||||
assert "--userns" not in argv or "host" not in _flag_value(argv, "--userns")
|
||||
|
||||
|
||||
def test_build_docker_argv_scratch_is_writable_tmpfs(tmp_path: Path) -> None:
|
||||
argv = build_docker_argv(_job(tmp_path))
|
||||
tmpfs = _flag_value(argv, "--tmpfs")
|
||||
assert tmpfs.startswith("/scratch")
|
||||
assert "rw" in tmpfs
|
||||
assert "exec" in tmpfs
|
||||
|
||||
|
||||
def test_build_docker_argv_applies_memory_and_pids_limits(tmp_path: Path) -> None:
|
||||
job = _job(tmp_path, memory_mb=512, pids_limit=64)
|
||||
argv = build_docker_argv(job)
|
||||
memory = _flag_value(argv, "--memory")
|
||||
assert "512" in memory
|
||||
assert memory.endswith("m") or memory.endswith("M") or memory.endswith("b")
|
||||
assert _flag_value(argv, "--pids-limit") == "64"
|
||||
|
||||
|
||||
def test_build_docker_argv_drops_capabilities_and_new_privileges(tmp_path: Path) -> None:
|
||||
argv = build_docker_argv(_job(tmp_path))
|
||||
assert _flag_value(argv, "--cap-drop") == "ALL"
|
||||
security = _flag_value(argv, "--security-opt")
|
||||
assert "no-new-privileges" in security
|
||||
|
||||
|
||||
def test_build_docker_argv_uses_non_root_user(tmp_path: Path) -> None:
|
||||
argv = build_docker_argv(_job(tmp_path))
|
||||
user = _flag_value(argv, "--user")
|
||||
uid = user.split(":", 1)[0]
|
||||
assert uid not in {"0", "root"}
|
||||
assert int(uid) != 0
|
||||
|
||||
|
||||
def test_build_docker_argv_bind_mounts_repo_read_only_at_workspace(tmp_path: Path) -> None:
|
||||
job = _job(tmp_path)
|
||||
argv = build_docker_argv(job)
|
||||
mount = _workspace_mount(argv)
|
||||
source = mount.get("src") or mount.get("source")
|
||||
assert source == str(job.repo_root.resolve())
|
||||
assert mount.get("dst") == "/workspace" or mount.get("destination") == "/workspace" or mount.get("target") == "/workspace"
|
||||
assert mount.get("ro") == "true" or mount.get("readonly") == "true"
|
||||
assert "rw" not in mount
|
||||
|
||||
|
||||
def test_build_docker_argv_source_tree_is_not_writable(tmp_path: Path) -> None:
|
||||
argv = build_docker_argv(_job(tmp_path))
|
||||
mount = _workspace_mount(argv)
|
||||
assert mount.get("ro") == "true" or mount.get("readonly") == "true"
|
||||
assert "--read-only" in argv
|
||||
joined = " ".join(argv)
|
||||
assert f"{_job(tmp_path).repo_root.resolve()}:/workspace:rw" not in joined
|
||||
assert f"{_job(tmp_path).repo_root.resolve()}:/workspace " not in joined + " "
|
||||
|
||||
|
||||
def test_build_docker_argv_ends_with_image_and_command(tmp_path: Path) -> None:
|
||||
job = _job(tmp_path)
|
||||
argv = build_docker_argv(job)
|
||||
assert argv[-len(job.command) :] == list(job.command)
|
||||
assert argv[-len(job.command) - 1] == job.image
|
||||
|
||||
|
||||
def test_fake_sandbox_runner_records_jobs_and_returns_programmed_result(tmp_path: Path) -> None:
|
||||
programmed = SandboxResult(exit_code=7, stdout="out", stderr="err", timed_out=False)
|
||||
runner: SandboxRunner = FakeSandboxRunner(programmed)
|
||||
job = _job(tmp_path)
|
||||
result = runner.run(job)
|
||||
assert result == programmed
|
||||
assert isinstance(runner, FakeSandboxRunner)
|
||||
assert runner.jobs == [job]
|
||||
|
||||
|
||||
def test_resolve_runner_fake_via_env(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setenv(runner_mod.RUNNER_ENV, "fake")
|
||||
assert isinstance(resolve_sandbox_runner(), FakeSandboxRunner)
|
||||
|
||||
|
||||
def test_resolve_runner_defaults_to_docker_cli(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.delenv(runner_mod.RUNNER_ENV, raising=False)
|
||||
assert isinstance(resolve_sandbox_runner(), DockerCliRunner)
|
||||
|
||||
|
||||
def test_fake_sandbox_runner_can_be_reprogrammed(tmp_path: Path) -> None:
|
||||
runner = FakeSandboxRunner()
|
||||
first = _job(tmp_path, command=("echo", "one"))
|
||||
second = _job(tmp_path, command=("echo", "two"))
|
||||
timeout_result = SandboxResult(exit_code=-1, stdout="", stderr="deadline", timed_out=True)
|
||||
runner.result = timeout_result
|
||||
assert runner.run(first).timed_out is True
|
||||
runner.result = SandboxResult(exit_code=0, stdout="ok", stderr="", timed_out=False)
|
||||
assert runner.run(second).stdout == "ok"
|
||||
assert runner.jobs == [first, second]
|
||||
|
||||
|
||||
def test_fake_sandbox_runner_does_not_need_docker_daemon(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
def boom(*_args: object, **_kwargs: object) -> subprocess.CompletedProcess[str]:
|
||||
raise AssertionError("unit tests must not invoke the docker daemon")
|
||||
|
||||
monkeypatch.setattr(subprocess, "run", boom)
|
||||
monkeypatch.setattr(runner_mod.subprocess, "run", boom)
|
||||
programmed = SandboxResult(exit_code=0, stdout="fake", stderr="", timed_out=False)
|
||||
runner = FakeSandboxRunner(programmed)
|
||||
assert runner.run(_job(tmp_path)) == programmed
|
||||
|
||||
|
||||
def test_docker_cli_runner_uses_constructed_argv_and_timeout(tmp_path: Path) -> None:
|
||||
seen: dict[str, Any] = {}
|
||||
job = _job(tmp_path, scratch_files={}, timeout_seconds=19)
|
||||
|
||||
def fake_run(command: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]:
|
||||
seen["command"] = list(command)
|
||||
seen["timeout"] = kwargs.get("timeout")
|
||||
seen["check"] = kwargs.get("check")
|
||||
seen["capture_output"] = kwargs.get("capture_output")
|
||||
seen["text"] = kwargs.get("text")
|
||||
return subprocess.CompletedProcess(command, 3, stdout="hello", stderr="warn")
|
||||
|
||||
result = DockerCliRunner(runner=fake_run).run(job)
|
||||
assert seen["command"] == build_docker_argv(job)
|
||||
assert seen["timeout"] == 19
|
||||
assert seen["check"] is False
|
||||
assert seen["capture_output"] is True
|
||||
assert seen["text"] is True
|
||||
assert result == SandboxResult(exit_code=3, stdout="hello", stderr="warn", timed_out=False)
|
||||
assert "--network" in seen["command"]
|
||||
assert "--read-only" in seen["command"]
|
||||
assert "docker.sock" not in " ".join(seen["command"])
|
||||
|
||||
|
||||
def test_docker_cli_runner_maps_timeout_expired_to_timed_out(tmp_path: Path) -> None:
|
||||
job = _job(tmp_path, scratch_files={}, timeout_seconds=5)
|
||||
|
||||
def fake_run(command: list[str], **_kwargs: Any) -> subprocess.CompletedProcess[str]:
|
||||
raise subprocess.TimeoutExpired(cmd=command, timeout=5, output="partial", stderr="slow")
|
||||
|
||||
result = DockerCliRunner(runner=fake_run).run(job)
|
||||
assert result.timed_out is True
|
||||
assert result.stdout == "partial"
|
||||
assert result.stderr == "slow"
|
||||
|
||||
|
||||
def test_docker_cli_runner_does_not_invoke_daemon_when_runner_injected(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
def boom(*_args: object, **_kwargs: object) -> subprocess.CompletedProcess[str]:
|
||||
raise AssertionError("unit tests must not invoke the docker daemon")
|
||||
|
||||
monkeypatch.setattr(subprocess, "run", boom)
|
||||
monkeypatch.setattr(runner_mod.subprocess, "run", boom)
|
||||
|
||||
def fake_run(command: list[str], **_kwargs: Any) -> subprocess.CompletedProcess[str]:
|
||||
return subprocess.CompletedProcess(command, 0, stdout="", stderr="")
|
||||
|
||||
result = DockerCliRunner(runner=fake_run).run(_job(tmp_path, scratch_files={}))
|
||||
assert result.timed_out is False
|
||||
assert result.exit_code == 0
|
||||
|
||||
|
||||
def test_docker_cli_runner_does_not_write_the_source_tree(tmp_path: Path) -> None:
|
||||
job = _job(tmp_path)
|
||||
original = (job.repo_root / "app.py").read_text()
|
||||
before = {path.relative_to(job.repo_root): path.read_text() for path in job.repo_root.rglob("*") if path.is_file()}
|
||||
|
||||
def fake_run(command: list[str], **_kwargs: Any) -> subprocess.CompletedProcess[str]:
|
||||
joined = " ".join(command)
|
||||
assert "docker.sock" not in joined
|
||||
assert "--read-only" in command
|
||||
return subprocess.CompletedProcess(command, 0, stdout="", stderr="")
|
||||
|
||||
DockerCliRunner(runner=fake_run).run(job)
|
||||
assert (job.repo_root / "app.py").read_text() == original
|
||||
after = {path.relative_to(job.repo_root): path.read_text() for path in job.repo_root.rglob("*") if path.is_file()}
|
||||
assert after == before
|
||||
assert "case.py" not in {path.name for path in job.repo_root.iterdir()}
|
||||
|
||||
|
||||
def test_docker_cli_runner_seeds_scratch_files_outside_the_repo(tmp_path: Path) -> None:
|
||||
job = _job(tmp_path, scratch_files={"nested/case.py": "print('seeded')\n"})
|
||||
seen: dict[str, Any] = {}
|
||||
|
||||
def fake_run(command: list[str], **_kwargs: Any) -> subprocess.CompletedProcess[str]:
|
||||
seen["command"] = list(command)
|
||||
mounts = [command[index + 1] for index, item in enumerate(command) if item == "--mount" and index + 1 < len(command)]
|
||||
seed_mounts = [spec for spec in mounts if "/scratch-in" in spec]
|
||||
assert seed_mounts, command
|
||||
options = _mount_options(seed_mounts[0])
|
||||
source = Path(options.get("src") or options.get("source") or "")
|
||||
assert source.is_dir()
|
||||
assert job.repo_root.resolve() not in source.resolve().parents
|
||||
assert source.resolve() != job.repo_root.resolve()
|
||||
assert (source / "nested" / "case.py").read_text() == "print('seeded')\n"
|
||||
assert options.get("ro") == "true" or options.get("readonly") == "true"
|
||||
assert "--tmpfs" in command
|
||||
assert _flag_value(command, "--tmpfs").startswith("/scratch")
|
||||
return subprocess.CompletedProcess(command, 0, stdout="", stderr="")
|
||||
|
||||
result = DockerCliRunner(runner=fake_run).run(job)
|
||||
assert result.exit_code == 0
|
||||
argv = seen["command"]
|
||||
assert "docker.sock" not in " ".join(argv)
|
||||
assert "--read-only" in argv
|
||||
assert job.image in argv
|
||||
|
||||
|
||||
def test_docker_cli_runner_scratch_seed_is_world_readable_for_container_user(tmp_path: Path) -> None:
|
||||
job = _job(tmp_path, scratch_files={"nested/dir/case.py": "print('seeded')\n"})
|
||||
seen: dict[str, Path] = {}
|
||||
|
||||
def fake_run(command: list[str], **_kwargs: Any) -> subprocess.CompletedProcess[str]:
|
||||
source = _seed_source(command)
|
||||
seen["source"] = source
|
||||
assert _posix_mode(source) == 0o755
|
||||
for dirpath, _dirnames, filenames in os.walk(source):
|
||||
directory = Path(dirpath)
|
||||
assert stat.S_ISDIR(directory.stat().st_mode)
|
||||
assert _posix_mode(directory) == 0o755, directory
|
||||
for name in filenames:
|
||||
file_path = directory / name
|
||||
assert _posix_mode(file_path) == 0o644, file_path
|
||||
return subprocess.CompletedProcess(command, 0, stdout="", stderr="")
|
||||
|
||||
DockerCliRunner(runner=fake_run).run(job)
|
||||
assert "source" in seen
|
||||
|
||||
|
||||
def test_docker_cli_runner_scratch_copy_uses_cp_r_not_cp_a(tmp_path: Path) -> None:
|
||||
job = _job(tmp_path, scratch_files={"nested/case.py": "print('seeded')\n"})
|
||||
seen: dict[str, str] = {}
|
||||
|
||||
def fake_run(command: list[str], **_kwargs: Any) -> subprocess.CompletedProcess[str]:
|
||||
script = _copy_script(command)
|
||||
seen["script"] = script
|
||||
assert command[command.index(job.image) + 1] == "/bin/sh"
|
||||
assert "cp -a" not in script
|
||||
assert "cp -a" not in " ".join(command)
|
||||
assert script.startswith("cp -r /scratch-in/. /scratch/")
|
||||
assert 'exec "$@"' in script
|
||||
return subprocess.CompletedProcess(command, 0, stdout="", stderr="")
|
||||
|
||||
DockerCliRunner(runner=fake_run).run(job)
|
||||
assert seen["script"] == 'cp -r /scratch-in/. /scratch/ && exec "$@"'
|
||||
@@ -0,0 +1,200 @@
|
||||
"""Deep mode: skip regression when the sandbox is missing; record verdicts with a fake runner (task 6.3)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from openultrasast.cli import main
|
||||
from openultrasast.sandbox import FakeSandboxRunner, SandboxResult
|
||||
from openultrasast.sandbox import runner as sandbox_runner
|
||||
|
||||
_VULN = "@app.route('/admin')\ndef admin():\n return eval(request.data)\n"
|
||||
|
||||
|
||||
def _repo(tmp_path: Path) -> Path:
|
||||
repo = tmp_path / "repo"
|
||||
repo.mkdir()
|
||||
(repo / "app.py").write_text(_VULN)
|
||||
return repo
|
||||
|
||||
|
||||
def _fail(message: str): # type: ignore[no-untyped-def]
|
||||
def _inner(*args: object, **kwargs: object) -> object:
|
||||
raise AssertionError(message)
|
||||
|
||||
return _inner
|
||||
|
||||
|
||||
def _guard_docker(monkeypatch: pytest.MonkeyPatch) -> list[list[str]]:
|
||||
docker_argv: list[list[str]] = []
|
||||
original_run = subprocess.run
|
||||
|
||||
def guarded_run(command: object, *args: object, **kwargs: object) -> object:
|
||||
argv = list(command) if isinstance(command, list | tuple) else [command]
|
||||
if argv and Path(str(argv[0])).name == "docker":
|
||||
docker_argv.append([str(item) for item in argv])
|
||||
raise AssertionError(f"deep scan test must not start docker: {argv}")
|
||||
return original_run(command, *args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(subprocess, "run", guarded_run)
|
||||
monkeypatch.setattr(sandbox_runner.DockerCliRunner, "run", _fail("deep scan test must not use DockerCliRunner"))
|
||||
return docker_argv
|
||||
|
||||
|
||||
def _latest_run(repo: Path) -> Path:
|
||||
return sorted((repo / ".runs").iterdir())[-1]
|
||||
|
||||
|
||||
def _crash_runner() -> FakeSandboxRunner:
|
||||
return FakeSandboxRunner(SandboxResult(exit_code=139, stdout="", stderr="Segmentation fault\n", timed_out=False))
|
||||
|
||||
|
||||
def _enable_crash_sandbox(monkeypatch: pytest.MonkeyPatch) -> FakeSandboxRunner:
|
||||
from openultrasast import cli
|
||||
|
||||
fake = _crash_runner()
|
||||
monkeypatch.setenv("OPENULTRASAST_RUNS_DIR", ".runs")
|
||||
monkeypatch.setenv("OPENULTRASAST_SANDBOX_PROBE", "1")
|
||||
monkeypatch.setattr(cli, "resolve_sandbox_runner", lambda: fake)
|
||||
return fake
|
||||
|
||||
|
||||
def test_deep_scan_skips_regress_when_sandbox_unavailable(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
repo = _repo(tmp_path)
|
||||
monkeypatch.setenv("OPENULTRASAST_RUNS_DIR", ".runs")
|
||||
monkeypatch.setenv("OPENULTRASAST_SANDBOX_PROBE", "0")
|
||||
docker_argv = _guard_docker(monkeypatch)
|
||||
|
||||
assert main(["scan", str(repo), "--mode", "deep"]) == 0
|
||||
|
||||
run_dir = _latest_run(repo)
|
||||
manifest = json.loads((run_dir / "manifest.json").read_text())
|
||||
stages = manifest["stages"]
|
||||
degradations = manifest.get("degradations", [])
|
||||
|
||||
assert (run_dir / "complexity_map.json").is_file()
|
||||
assert not (run_dir / "verdicts.json").exists()
|
||||
assert docker_argv == []
|
||||
assert stages["requested"] == ["static", "map", "regress"]
|
||||
assert stages["completed"] == ["static", "map"]
|
||||
assert {"stage": "regress", "reason": "sandbox_unavailable"} in stages["skipped"]
|
||||
assert any(entry.get("stage") == "regress" and entry.get("reason") == "sandbox_unavailable" for entry in degradations)
|
||||
|
||||
|
||||
def test_deep_scan_with_fake_runner_writes_verdicts(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
repo = _repo(tmp_path)
|
||||
monkeypatch.setenv("OPENULTRASAST_RUNS_DIR", ".runs")
|
||||
monkeypatch.setenv("OPENULTRASAST_SANDBOX_PROBE", "1")
|
||||
monkeypatch.setenv("OPENULTRASAST_SANDBOX_RUNNER", "fake")
|
||||
docker_argv = _guard_docker(monkeypatch)
|
||||
|
||||
assert main(["scan", str(repo), "--mode", "deep"]) == 0
|
||||
|
||||
run_dir = _latest_run(repo)
|
||||
verdicts_path = run_dir / "verdicts.json"
|
||||
manifest = json.loads((run_dir / "manifest.json").read_text())
|
||||
payload = json.loads(verdicts_path.read_text())
|
||||
stages = manifest["stages"]
|
||||
|
||||
assert docker_argv == []
|
||||
assert verdicts_path.is_file()
|
||||
assert (run_dir / "complexity_map.json").is_file()
|
||||
assert stages["requested"] == ["static", "map", "regress"]
|
||||
assert stages["completed"] == ["static", "map", "regress"]
|
||||
assert not any(entry.get("reason") == "sandbox_unavailable" for entry in stages["skipped"])
|
||||
assert isinstance(payload["verdicts"], list)
|
||||
assert payload["verdicts"]
|
||||
for item in payload["verdicts"]:
|
||||
assert item["verdict"] in {"triggerable", "not_triggerable", "already_covered", "inconclusive"}
|
||||
assert "reason" in item
|
||||
assert "path" in item
|
||||
assert manifest["artifacts"]["verdicts"] == "verdicts.json"
|
||||
assert manifest["artifacts"]["complexity_map"] == "complexity_map.json"
|
||||
assert manifest["worth_fixing"]["count"] == sum(1 for item in payload["verdicts"] if item.get("worth_fixing"))
|
||||
report = (run_dir / "report.md").read_text()
|
||||
inventory_at = report.index("## Inventory")
|
||||
map_at = report.index("## Complexity map")
|
||||
worth_at = report.index("## Worth fixing")
|
||||
assert inventory_at < map_at < worth_at
|
||||
map_section = report[map_at:worth_at]
|
||||
assert "app.py" in map_section
|
||||
assert "Gap:" in map_section
|
||||
ledger_path = repo / ".openultrasast" / "calibration" / "complexity_ledger.json"
|
||||
assert ledger_path.is_file()
|
||||
ledger = json.loads(ledger_path.read_text())
|
||||
assert ledger
|
||||
assert any(entry.get("last_verdict") in {"triggerable", "not_triggerable"} for entry in ledger.values())
|
||||
|
||||
|
||||
def test_deep_scan_unsafe_hunter_snippet_is_safety_rejected_without_docker_job(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
from openultrasast import cli
|
||||
from openultrasast.sandbox import FakeSandboxRunner
|
||||
|
||||
repo = _repo(tmp_path)
|
||||
config = tmp_path / "openultrasast.toml"
|
||||
config.write_text('[models]\nhunter = "test-hunter"\n')
|
||||
fake = FakeSandboxRunner()
|
||||
monkeypatch.setenv("OPENULTRASAST_RUNS_DIR", ".runs")
|
||||
monkeypatch.setenv("OPENULTRASAST_SANDBOX_PROBE", "1")
|
||||
monkeypatch.setenv("OPENULTRASAST_HUNTER_CLIENT", "unsafe-snippet")
|
||||
monkeypatch.setattr(cli, "has_harnessx", lambda: False)
|
||||
monkeypatch.setattr(cli, "resolve_sandbox_runner", lambda: fake)
|
||||
docker_argv = _guard_docker(monkeypatch)
|
||||
|
||||
assert main(["scan", str(repo), "--mode", "deep", "--config", str(config)]) == 0
|
||||
|
||||
run_dir = _latest_run(repo)
|
||||
payload = json.loads((run_dir / "verdicts.json").read_text())
|
||||
findings = json.loads((run_dir / "findings.json").read_text())["findings"]
|
||||
hunter_findings = [item for item in findings if str(item["finding_id"]).startswith("tool-hunter:")]
|
||||
|
||||
assert docker_argv == []
|
||||
assert fake.jobs == []
|
||||
assert hunter_findings
|
||||
assert all(item["evidence_level"] == "suspicion" for item in hunter_findings)
|
||||
assert payload["verdicts"]
|
||||
assert any(item["reason"] == "safety_rejected" for item in payload["verdicts"])
|
||||
for item in payload["verdicts"]:
|
||||
if item["reason"] == "safety_rejected":
|
||||
assert item["verdict"] == "inconclusive"
|
||||
|
||||
|
||||
def test_deep_scan_fail_on_worth_fixing_exits_one_when_triggerable(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
repo = _repo(tmp_path)
|
||||
_enable_crash_sandbox(monkeypatch)
|
||||
docker_argv = _guard_docker(monkeypatch)
|
||||
|
||||
assert main(["scan", str(repo), "--mode", "deep", "--fail-on", "worth-fixing"]) == 1
|
||||
assert docker_argv == []
|
||||
|
||||
run_dir = _latest_run(repo)
|
||||
payload = json.loads((run_dir / "verdicts.json").read_text())
|
||||
manifest = json.loads((run_dir / "manifest.json").read_text())
|
||||
report = (run_dir / "report.md").read_text()
|
||||
worth_fixing = [item for item in payload["verdicts"] if item.get("worth_fixing")]
|
||||
|
||||
assert worth_fixing
|
||||
assert len(worth_fixing) >= 1
|
||||
assert any(item["verdict"] == "triggerable" for item in worth_fixing)
|
||||
assert manifest["worth_fixing"]["count"] == len(worth_fixing)
|
||||
assert "## Complexity map" in report
|
||||
assert "## Worth fixing" in report
|
||||
assert "triggerable" in report
|
||||
ledger = json.loads((repo / ".openultrasast" / "calibration" / "complexity_ledger.json").read_text())
|
||||
assert any(entry.get("last_verdict") == "triggerable" for entry in ledger.values())
|
||||
|
||||
|
||||
def test_deep_scan_fail_on_never_exits_zero_with_worth_fixing_verdict(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
repo = _repo(tmp_path)
|
||||
_enable_crash_sandbox(monkeypatch)
|
||||
docker_argv = _guard_docker(monkeypatch)
|
||||
|
||||
assert main(["scan", str(repo), "--mode", "deep", "--fail-on", "never"]) == 0
|
||||
assert docker_argv == []
|
||||
|
||||
payload = json.loads((_latest_run(repo) / "verdicts.json").read_text())
|
||||
assert any(item.get("worth_fixing") for item in payload["verdicts"])
|
||||
@@ -36,6 +36,8 @@ def _scan(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, *, mode: str, config_
|
||||
(repo / "app.py").write_text(_VULN)
|
||||
monkeypatch.setattr(cli, "has_harnessx", lambda: harnessx)
|
||||
monkeypatch.setenv("OPENULTRASAST_RUNS_DIR", ".runs")
|
||||
monkeypatch.delenv("OPENROUTER_API_KEY", raising=False)
|
||||
monkeypatch.delenv("OPENULTRASAST_HUNTER_CLIENT", raising=False)
|
||||
argv = ["scan", str(repo), "--mode", mode]
|
||||
if config_body is not None:
|
||||
cfg = tmp_path / "openultrasast.toml"
|
||||
@@ -65,7 +67,8 @@ def test_standard_scan_extra_absent_matches_deterministic_baseline(tmp_path: Pat
|
||||
stages = {entry["stage"] for entry in requested_manifest.get("degradations", [])}
|
||||
assert stages == {"hunter_pool", "verify"}
|
||||
assert all(entry["reason"] == "harnessx_extra_unavailable" for entry in requested_manifest["degradations"])
|
||||
assert "degradations" not in baseline_manifest
|
||||
assert all(entry["reason"] != "hunter_model_unavailable" for entry in requested_manifest["degradations"])
|
||||
assert [entry["reason"] for entry in baseline_manifest.get("degradations", [])] == ["hunter_model_unavailable"]
|
||||
|
||||
|
||||
def test_quick_scan_is_unaffected_by_the_extra(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
"""Split-sink fixtures: stage-1 pattern scan must miss planted sinks (requirement 8.1)."""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from openultrasast.benchmark import (
|
||||
BenchmarkManifest,
|
||||
BenchmarkRun,
|
||||
ExpectedFinding,
|
||||
evaluate_benchmark,
|
||||
load_benchmark_manifest,
|
||||
resolve_benchmark_source,
|
||||
)
|
||||
from openultrasast.findings import StaticFinding, quick_scan_findings
|
||||
from openultrasast.gate import LANGUAGE_MANIFESTS
|
||||
from openultrasast.preprocess import preprocess_repository
|
||||
from openultrasast.rank import rank_targets
|
||||
|
||||
MANIFEST_DIR = Path("benchmarks/manifests")
|
||||
SPLIT_SINK_MANIFESTS = (
|
||||
"split-sink-c",
|
||||
"split-sink-java",
|
||||
"split-sink-javascript",
|
||||
"split-sink-python",
|
||||
)
|
||||
|
||||
|
||||
def _quick_scan(target: Path) -> list[StaticFinding]:
|
||||
_, targets = preprocess_repository(target)
|
||||
return quick_scan_findings(target, targets, rank_targets(targets))
|
||||
|
||||
|
||||
def _planted_split_sinks(manifest: BenchmarkManifest) -> list[ExpectedFinding]:
|
||||
return [item for item in manifest.expected if item.rule_id is None]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("name", SPLIT_SINK_MANIFESTS)
|
||||
def test_stage1_misses_planted_split_sink(name: str) -> None:
|
||||
manifest_path = MANIFEST_DIR / f"{name}.toml"
|
||||
manifest = load_benchmark_manifest(manifest_path)
|
||||
target = resolve_benchmark_source(manifest_path, manifest)
|
||||
planted = _planted_split_sinks(manifest)
|
||||
assert planted, f"{name} must document a stage-1 miss without rule_id"
|
||||
|
||||
findings = _quick_scan(target)
|
||||
run = BenchmarkRun(benchmark_run_id="split-sink", root=Path("/tmp/split-sink"), manifest=manifest)
|
||||
result = evaluate_benchmark(run=run, mode="quick", findings=findings, scan_id=None, scan_run_dir=None)
|
||||
missed = {(miss.path, miss.cwe, miss.rule_id) for miss in result.misses}
|
||||
|
||||
for expected in planted:
|
||||
assert expected.line is not None
|
||||
assert expected.sink
|
||||
source = (target / expected.path).read_text()
|
||||
lines = source.splitlines()
|
||||
sink_line = lines[expected.line - 1]
|
||||
assert expected.sink.lower() in sink_line.lower(), f"{name} planted sink {expected.sink!r} missing on line {expected.line}"
|
||||
prior = "\n".join(lines[: expected.line - 1]).lower()
|
||||
assert any(token in prior for token in ("query", "sql")), f"{name} must assign the payload before the sink line"
|
||||
on_line = [finding for finding in findings if finding.path == expected.path and finding.line == expected.line]
|
||||
assert on_line == [], f"{name} stage-1 hit the planted split-sink on {expected.path}:{expected.line}: {on_line}"
|
||||
assert (expected.path, expected.cwe, None) in missed, (
|
||||
f"{name} evaluate_benchmark must record a miss for {expected.path}:{expected.line}"
|
||||
)
|
||||
|
||||
|
||||
def test_split_sink_fixtures_are_outside_stage1_gate_corpus() -> None:
|
||||
listed = {name for names in LANGUAGE_MANIFESTS.values() for name in names}
|
||||
leaked = [name for name in SPLIT_SINK_MANIFESTS if name in listed]
|
||||
assert leaked == [], "split-sink fixtures must not join LANGUAGE_MANIFESTS (stage-1 90% smoke gate)"
|
||||
@@ -0,0 +1,40 @@
|
||||
"""Stage-3 regression on split-sink: fake runner always; live Docker skips without a daemon."""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from openultrasast.config import SandboxConfig
|
||||
from openultrasast.regress.recipes import recipe_for
|
||||
from openultrasast.regress.safety import check_snippet_safety
|
||||
from openultrasast.regress.verdict import TRIGGERABLE, verdict_from_result
|
||||
from openultrasast.sandbox import DockerCliRunner, FakeSandboxRunner, SandboxProbe, SandboxResult
|
||||
|
||||
_SPLIT_SINK_PYTHON = Path("benchmarks/fixtures/split-sink-python")
|
||||
_LIMITS = SandboxConfig(memory_mb=256, timeout_seconds=30, pids_limit=128)
|
||||
_PYTHON_IMAGE = "python:3.12-alpine"
|
||||
_SNIPPET = 'term = "x\'; drop"\nquery = "select * from items where title like \'%" + term + "%\'"\nassert "\'" not in query\n'
|
||||
|
||||
|
||||
def _split_sink_job():
|
||||
check_snippet_safety(_SNIPPET)
|
||||
job = recipe_for("python", _SNIPPET, _PYTHON_IMAGE, _LIMITS, repo_root=_SPLIT_SINK_PYTHON.resolve())
|
||||
assert job is not None
|
||||
return job
|
||||
|
||||
|
||||
def test_fake_runner_marks_split_sink_snippet_triggerable() -> None:
|
||||
fake = FakeSandboxRunner(SandboxResult(exit_code=1, stdout="", stderr="AssertionError", timed_out=False))
|
||||
result = fake.run(_split_sink_job())
|
||||
verdict = verdict_from_result(result)
|
||||
assert verdict.verdict == TRIGGERABLE
|
||||
assert fake.jobs
|
||||
|
||||
|
||||
@pytest.mark.docker
|
||||
def test_docker_marks_split_sink_python_triggerable() -> None:
|
||||
if not SandboxProbe().available():
|
||||
pytest.skip("docker daemon unavailable")
|
||||
result = DockerCliRunner().run(_split_sink_job())
|
||||
verdict = verdict_from_result(result)
|
||||
assert verdict.verdict == TRIGGERABLE, (result.exit_code, result.stdout, result.stderr)
|
||||
@@ -0,0 +1,68 @@
|
||||
import json
|
||||
|
||||
from openultrasast.stages import MODE_STAGES, Stage, StagePlan, plan_for_mode, record_completed, record_skip, stages_payload
|
||||
|
||||
|
||||
def test_mode_stages_maps_existing_depths() -> None:
|
||||
assert Stage.STATIC == "static"
|
||||
assert Stage.MAP == "map"
|
||||
assert Stage.REGRESS == "regress"
|
||||
assert MODE_STAGES["quick"] == (Stage.STATIC,)
|
||||
assert MODE_STAGES["standard"] == (Stage.STATIC, Stage.MAP)
|
||||
assert MODE_STAGES["deep"] == (Stage.STATIC, Stage.MAP, Stage.REGRESS)
|
||||
|
||||
|
||||
def test_quick_never_includes_map_or_regress() -> None:
|
||||
plan = plan_for_mode("quick")
|
||||
assert plan.mode == "quick"
|
||||
assert plan.requested == (Stage.STATIC,)
|
||||
assert Stage.MAP not in plan.requested
|
||||
assert Stage.REGRESS not in plan.requested
|
||||
assert Stage.MAP not in MODE_STAGES["quick"]
|
||||
assert Stage.REGRESS not in MODE_STAGES["quick"]
|
||||
|
||||
|
||||
def test_plan_for_mode_requests_ordered_stages() -> None:
|
||||
standard = plan_for_mode("standard")
|
||||
deep = plan_for_mode("deep")
|
||||
|
||||
assert standard.requested == (Stage.STATIC, Stage.MAP)
|
||||
assert deep.requested == (Stage.STATIC, Stage.MAP, Stage.REGRESS)
|
||||
assert standard.completed == ()
|
||||
assert deep.skipped == ()
|
||||
assert isinstance(standard, StagePlan)
|
||||
assert isinstance(deep, StagePlan)
|
||||
|
||||
|
||||
def test_skipped_stage_is_structured_degradation_not_exception() -> None:
|
||||
plan = plan_for_mode("deep")
|
||||
skipped = record_skip(plan, Stage.REGRESS, "sandbox_unavailable")
|
||||
|
||||
assert skipped.requested == (Stage.STATIC, Stage.MAP, Stage.REGRESS)
|
||||
assert skipped.completed == ()
|
||||
assert skipped.skipped == ({"stage": "regress", "reason": "sandbox_unavailable"},)
|
||||
degradations = list(skipped.skipped)
|
||||
assert degradations[0]["stage"] == "regress"
|
||||
assert degradations[0]["reason"] == "sandbox_unavailable"
|
||||
assert json.dumps(degradations) == json.dumps([{"stage": "regress", "reason": "sandbox_unavailable"}])
|
||||
|
||||
|
||||
def test_record_completed_static_for_quick() -> None:
|
||||
plan = record_completed(plan_for_mode("quick"), Stage.STATIC)
|
||||
|
||||
assert plan.requested == (Stage.STATIC,)
|
||||
assert plan.completed == (Stage.STATIC,)
|
||||
assert plan.skipped == ()
|
||||
assert Stage.MAP not in plan.completed
|
||||
assert Stage.REGRESS not in plan.completed
|
||||
|
||||
|
||||
def test_stages_payload_is_manifest_shaped() -> None:
|
||||
plan = record_completed(plan_for_mode("quick"), Stage.STATIC)
|
||||
payload = stages_payload(plan)
|
||||
|
||||
assert payload == {
|
||||
"requested": ["static"],
|
||||
"completed": ["static"],
|
||||
"skipped": [],
|
||||
}
|
||||
@@ -0,0 +1,327 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from openultrasast.complexity.map import Hotspot
|
||||
from openultrasast.tool_hunter import ChatResponse, OpenRouterHunterClient, ScriptedChatClient, ToolCall, run_tool_hunter
|
||||
|
||||
SPLIT_SINK = """\
|
||||
term = request.args["q"]
|
||||
query = "select * from items where title like '%" + term + "%'"
|
||||
return db.execute(query)
|
||||
"""
|
||||
|
||||
_DUMP_FINDINGS = [
|
||||
{
|
||||
"path": "app.py",
|
||||
"line": 3,
|
||||
"title": "SQL injection",
|
||||
"rationale": "query is built from request input then executed",
|
||||
"evidence_level": "static_corroboration",
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
def _repo_with_split_sink(tmp_path: Path) -> Path:
|
||||
root = tmp_path / "repo"
|
||||
root.mkdir()
|
||||
(root / "app.py").write_text(SPLIT_SINK)
|
||||
return root
|
||||
|
||||
|
||||
def _hotspot(path: str = "app.py") -> Hotspot:
|
||||
return Hotspot(
|
||||
path=path,
|
||||
function_name="search",
|
||||
score=7.5,
|
||||
band="high",
|
||||
signals={"loc": 3},
|
||||
rationale="reachable split-sink",
|
||||
test_hint=None,
|
||||
inventory_finding_ids=(),
|
||||
)
|
||||
|
||||
|
||||
def _tool_names(tools: object) -> set[str]:
|
||||
names: set[str] = set()
|
||||
if not isinstance(tools, list):
|
||||
return names
|
||||
for item in tools:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
function = item.get("function")
|
||||
if isinstance(function, dict) and isinstance(function.get("name"), str):
|
||||
names.add(str(function["name"]))
|
||||
elif isinstance(item.get("name"), str):
|
||||
names.add(str(item["name"]))
|
||||
return names
|
||||
|
||||
|
||||
def test_json_dump_without_tool_call_emits_no_findings(tmp_path: Path) -> None:
|
||||
root = _repo_with_split_sink(tmp_path)
|
||||
client = ScriptedChatClient([ChatResponse(content=json.dumps(_DUMP_FINDINGS))])
|
||||
|
||||
findings = run_tool_hunter(root, [_hotspot()], client=client, model="test-hunter", max_steps=4)
|
||||
|
||||
assert client.calls, "the hunter loop must run when a model client is provided"
|
||||
assert _tool_names(client.calls[0]["tools"]) >= {"read_file", "grep_repo", "find_refs"}
|
||||
assert findings == []
|
||||
|
||||
|
||||
def test_grep_then_json_yields_suspicion_with_tool_hunter_prefix(tmp_path: Path) -> None:
|
||||
root = _repo_with_split_sink(tmp_path)
|
||||
client = ScriptedChatClient(
|
||||
[
|
||||
ChatResponse(tool_calls=(ToolCall(id="call_grep", name="grep_repo", arguments={"pattern": r"query\s*=", "max_matches": 10}),)),
|
||||
ChatResponse(content=json.dumps(_DUMP_FINDINGS)),
|
||||
]
|
||||
)
|
||||
|
||||
findings = run_tool_hunter(root, [_hotspot()], client=client, model="test-hunter", max_steps=4)
|
||||
|
||||
assert len(client.calls) >= 2
|
||||
assert _tool_names(client.calls[0]["tools"]) >= {"read_file", "grep_repo", "find_refs"}
|
||||
tool_trace = json.dumps(client.calls[1]["messages"])
|
||||
assert "query =" in tool_trace
|
||||
assert len(findings) == 1
|
||||
assert findings[0].evidence_level == "suspicion"
|
||||
assert findings[0].finding_id.startswith("tool-hunter:")
|
||||
assert findings[0].path == "app.py"
|
||||
assert findings[0].line == 3
|
||||
assert findings[0].title == "SQL injection"
|
||||
|
||||
|
||||
def test_loop_dispatches_read_file_grep_repo_and_find_refs(tmp_path: Path) -> None:
|
||||
root = _repo_with_split_sink(tmp_path)
|
||||
client = ScriptedChatClient(
|
||||
[
|
||||
ChatResponse(
|
||||
tool_calls=(
|
||||
ToolCall(id="call_read", name="read_file", arguments={"path": "app.py", "max_chars": 4000}),
|
||||
ToolCall(id="call_grep", name="grep_repo", arguments={"pattern": r"db\.execute", "max_matches": 5}),
|
||||
ToolCall(id="call_refs", name="find_refs", arguments={"symbol": "query"}),
|
||||
)
|
||||
),
|
||||
ChatResponse(content=json.dumps(_DUMP_FINDINGS)),
|
||||
]
|
||||
)
|
||||
|
||||
findings = run_tool_hunter(root, [_hotspot()], client=client, model="test-hunter", max_steps=4)
|
||||
|
||||
assert findings
|
||||
assert findings[0].evidence_level == "suspicion"
|
||||
second_messages = json.dumps(client.calls[1]["messages"])
|
||||
assert "term = request.args" in second_messages
|
||||
assert "db.execute" in second_messages
|
||||
assert "query" in second_messages
|
||||
|
||||
|
||||
def test_path_escape_stays_in_band_and_still_counts_as_tool_use(tmp_path: Path) -> None:
|
||||
root = _repo_with_split_sink(tmp_path)
|
||||
client = ScriptedChatClient(
|
||||
[
|
||||
ChatResponse(tool_calls=(ToolCall(id="call_escape", name="read_file", arguments={"path": "../etc/passwd"}),)),
|
||||
ChatResponse(content=json.dumps(_DUMP_FINDINGS)),
|
||||
]
|
||||
)
|
||||
|
||||
findings = run_tool_hunter(root, [_hotspot()], client=client, model="test-hunter", max_steps=4)
|
||||
|
||||
assert findings
|
||||
assert findings[0].evidence_level == "suspicion"
|
||||
assert "error" in json.dumps(client.calls[1]["messages"]).lower()
|
||||
|
||||
|
||||
def test_max_steps_stops_before_ungrounded_json(tmp_path: Path) -> None:
|
||||
root = _repo_with_split_sink(tmp_path)
|
||||
client = ScriptedChatClient(
|
||||
[
|
||||
ChatResponse(tool_calls=(ToolCall(id="call_grep", name="grep_repo", arguments={"pattern": "query"}),)),
|
||||
ChatResponse(content=json.dumps(_DUMP_FINDINGS)),
|
||||
]
|
||||
)
|
||||
|
||||
findings = run_tool_hunter(root, [_hotspot()], client=client, model="test-hunter", max_steps=1)
|
||||
|
||||
assert findings == []
|
||||
assert len(client.calls) == 1
|
||||
|
||||
|
||||
_UNSAFE_SNIPPET = 'client = docker.DockerClient(base_url="unix://var/run/docker.sock")\n'
|
||||
_VULN = "@app.route('/admin')\ndef admin():\n return eval(request.data)\n"
|
||||
|
||||
|
||||
def test_hunter_json_snippet_stays_suspicion_and_is_not_static_corroboration(tmp_path: Path) -> None:
|
||||
root = _repo_with_split_sink(tmp_path)
|
||||
payload = [{**_DUMP_FINDINGS[0], "snippet": _UNSAFE_SNIPPET, "evidence_level": "static_corroboration"}]
|
||||
client = ScriptedChatClient(
|
||||
[
|
||||
ChatResponse(tool_calls=(ToolCall(id="call_grep", name="grep_repo", arguments={"pattern": "query"}),)),
|
||||
ChatResponse(content=json.dumps(payload)),
|
||||
]
|
||||
)
|
||||
|
||||
findings = run_tool_hunter(root, [_hotspot()], client=client, model="test-hunter", max_steps=4)
|
||||
|
||||
assert len(findings) == 1
|
||||
assert findings[0].evidence_level == "suspicion"
|
||||
assert findings[0].finding_id.startswith("tool-hunter:")
|
||||
assert findings[0].proposed_snippet == _UNSAFE_SNIPPET
|
||||
|
||||
|
||||
def _write_hunter_config(tmp_path: Path, body: str = '[models]\nhunter = "test-hunter"\n') -> Path:
|
||||
config = tmp_path / "openultrasast.toml"
|
||||
config.write_text(body)
|
||||
return config
|
||||
|
||||
|
||||
def _standard_repo(tmp_path: Path) -> Path:
|
||||
repo = tmp_path / "repo"
|
||||
repo.mkdir()
|
||||
(repo / "app.py").write_text(_VULN)
|
||||
return repo
|
||||
|
||||
|
||||
def _latest_run(repo: Path) -> Path:
|
||||
return sorted((repo / ".runs").iterdir())[-1]
|
||||
|
||||
|
||||
def _guard_docker(monkeypatch: pytest.MonkeyPatch) -> list[list[str]]:
|
||||
import subprocess
|
||||
|
||||
from openultrasast.sandbox import runner as sandbox_runner
|
||||
|
||||
docker_argv: list[list[str]] = []
|
||||
original_run = subprocess.run
|
||||
|
||||
def guarded_run(command: object, *args: object, **kwargs: object) -> object:
|
||||
argv = list(command) if isinstance(command, list | tuple) else [command]
|
||||
if argv and Path(str(argv[0])).name == "docker":
|
||||
docker_argv.append([str(item) for item in argv])
|
||||
raise AssertionError(f"hunter scan test must not start docker: {argv}")
|
||||
return original_run(command, *args, **kwargs)
|
||||
|
||||
def fail(*args: object, **kwargs: object) -> object:
|
||||
raise AssertionError("hunter scan test must not use DockerCliRunner")
|
||||
|
||||
monkeypatch.setattr(subprocess, "run", guarded_run)
|
||||
monkeypatch.setattr(sandbox_runner.DockerCliRunner, "run", fail)
|
||||
return docker_argv
|
||||
|
||||
|
||||
def test_standard_scripted_hunter_emits_tool_hunter_suspicion(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
from openultrasast import cli, tool_hunter
|
||||
from openultrasast.cli import main
|
||||
|
||||
repo = _standard_repo(tmp_path)
|
||||
config = _write_hunter_config(tmp_path)
|
||||
monkeypatch.setenv("OPENULTRASAST_RUNS_DIR", ".runs")
|
||||
monkeypatch.setenv("OPENULTRASAST_HUNTER_CLIENT", "scripted")
|
||||
monkeypatch.setattr(cli, "has_harnessx", lambda: False)
|
||||
docker_argv = _guard_docker(monkeypatch)
|
||||
hunter_calls: list[int] = []
|
||||
real_hunter = tool_hunter.run_tool_hunter
|
||||
|
||||
def wrapped(*args: object, **kwargs: object) -> object:
|
||||
hunter_calls.append(1)
|
||||
return real_hunter(*args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(tool_hunter, "run_tool_hunter", wrapped)
|
||||
|
||||
assert main(["scan", str(repo), "--mode", "standard", "--config", str(config)]) == 0
|
||||
|
||||
run_dir = _latest_run(repo)
|
||||
findings = json.loads((run_dir / "findings.json").read_text())["findings"]
|
||||
complexity_map = json.loads((run_dir / "complexity_map.json").read_text())
|
||||
manifest = json.loads((run_dir / "manifest.json").read_text())
|
||||
hunter_findings = [item for item in findings if str(item["finding_id"]).startswith("tool-hunter:")]
|
||||
|
||||
assert hunter_calls
|
||||
assert docker_argv == []
|
||||
assert hunter_findings
|
||||
assert all(item["evidence_level"] == "suspicion" for item in hunter_findings)
|
||||
assert all(item["evidence_level"] != "static_corroboration" for item in hunter_findings)
|
||||
assert complexity_map["heuristic_only"] is False
|
||||
assert manifest["complexity"]["heuristic_only"] is False
|
||||
assert not any(entry.get("reason") == "hunter_model_unavailable" for entry in manifest.get("degradations", []))
|
||||
|
||||
|
||||
def test_dump_only_hunter_client_emits_no_hunter_findings(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
from openultrasast import cli, tool_hunter
|
||||
from openultrasast.cli import main
|
||||
|
||||
repo = _standard_repo(tmp_path)
|
||||
config = _write_hunter_config(tmp_path)
|
||||
monkeypatch.setenv("OPENULTRASAST_RUNS_DIR", ".runs")
|
||||
monkeypatch.setenv("OPENULTRASAST_HUNTER_CLIENT", "dump-only")
|
||||
monkeypatch.setattr(cli, "has_harnessx", lambda: False)
|
||||
docker_argv = _guard_docker(monkeypatch)
|
||||
hunter_calls: list[int] = []
|
||||
real_hunter = tool_hunter.run_tool_hunter
|
||||
|
||||
def wrapped(*args: object, **kwargs: object) -> object:
|
||||
hunter_calls.append(1)
|
||||
return real_hunter(*args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(tool_hunter, "run_tool_hunter", wrapped)
|
||||
|
||||
assert main(["scan", str(repo), "--mode", "standard", "--config", str(config)]) == 0
|
||||
|
||||
run_dir = _latest_run(repo)
|
||||
findings = json.loads((run_dir / "findings.json").read_text())["findings"]
|
||||
complexity_map = json.loads((run_dir / "complexity_map.json").read_text())
|
||||
hunter_findings = [item for item in findings if str(item["finding_id"]).startswith("tool-hunter:")]
|
||||
|
||||
assert hunter_calls, "dump-only still runs the tool hunter loop"
|
||||
assert docker_argv == []
|
||||
assert hunter_findings == []
|
||||
assert complexity_map["heuristic_only"] is False
|
||||
|
||||
|
||||
def test_resolve_hunter_client_scripted_and_dump_only_env(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
from openultrasast import tool_hunter
|
||||
|
||||
monkeypatch.delenv("OPENROUTER_API_KEY", raising=False)
|
||||
monkeypatch.setenv("OPENULTRASAST_HUNTER_CLIENT", "scripted")
|
||||
scripted = tool_hunter.resolve_hunter_client()
|
||||
assert scripted is not None
|
||||
assert type(scripted).__name__ == "ScriptedChatClient"
|
||||
|
||||
monkeypatch.setenv("OPENULTRASAST_HUNTER_CLIENT", "dump-only")
|
||||
dump_only = tool_hunter.resolve_hunter_client()
|
||||
assert dump_only is not None
|
||||
assert type(dump_only).__name__ == "ScriptedChatClient"
|
||||
assert dump_only is not scripted
|
||||
|
||||
monkeypatch.delenv("OPENULTRASAST_HUNTER_CLIENT", raising=False)
|
||||
assert tool_hunter.resolve_hunter_client() is None
|
||||
|
||||
monkeypatch.setenv("OPENROUTER_API_KEY", "test-key")
|
||||
live = tool_hunter.resolve_hunter_client()
|
||||
assert isinstance(live, OpenRouterHunterClient)
|
||||
|
||||
|
||||
def test_openrouter_hunter_client_maps_tool_calls_without_network() -> None:
|
||||
class _Inner:
|
||||
def complete_chat(self, **kwargs: object) -> dict[str, object]:
|
||||
assert kwargs["model"] == "test-hunter"
|
||||
assert kwargs["tools"]
|
||||
return {
|
||||
"content": None,
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": "call_grep",
|
||||
"type": "function",
|
||||
"function": {"name": "grep_repo", "arguments": '{"pattern": "eval", "max_matches": 4}'},
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
client = OpenRouterHunterClient(_Inner()) # type: ignore[arg-type]
|
||||
response = client.complete(model="test-hunter", messages=[], tools=[{"type": "function"}], timeout_seconds=5)
|
||||
|
||||
assert response.content is None
|
||||
assert len(response.tool_calls) == 1
|
||||
assert response.tool_calls[0].name == "grep_repo"
|
||||
assert response.tool_calls[0].arguments == {"pattern": "eval", "max_matches": 4}
|
||||
Reference in New Issue
Block a user