mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
docs(three-stage-scan): add approved three-stage scan spec
This commit is contained in:
Marius Ciepluch
@@ -0,0 +1,74 @@
|
||||
# Brief: three-stage-scan
|
||||
|
||||
## Problem
|
||||
|
||||
AppSec engineers and DevOps teams running OpenUltraSAST in CI get a regex static scan scored against cheat-sheet fixtures. They cannot tell what actually exists in a real repository, where complexity concentrates, how to aim tests, or whether a finding is worth a CI failure. The previous HarnessX spec added governance around those regexes (CWE policy, score, shadow status) without making the hunter evolutionary or the evaluation realistic.
|
||||
|
||||
## Current State
|
||||
|
||||
- `ousast scan --mode quick` is a deterministic regex inventory. It works.
|
||||
- `--mode standard` is labeled a hunter pool but reuses the same regex path unless the optional HarnessX extra *and* a hunter model are both set. That hunter dumps a file into a prompt and scrapes JSON; it has no tools.
|
||||
- `--mode deep` exits with "not implemented." Sandbox config exists on paper (`network=false`, read-only workspace, memory/pids/timeout). No container is ever started.
|
||||
- Benchmarks are one-line sinks written to match the regexes. The fixtures themselves comment the misses (`regex blind spot`). The ≥90% recall gate is circular.
|
||||
- CWE policy, project score, shadow status, traces, and SARIF already exist and should not be redesigned.
|
||||
|
||||
## Desired Outcome
|
||||
|
||||
A scan answers three operator questions, in order:
|
||||
|
||||
1. **What exists?** A static inventory of files, surfaces, and pattern/SARIF hits — honest about being inventory, not verified bugs.
|
||||
2. **Where are problems most likely, and how do I tune tests?** A complexity map of hotspots plus test-gap hints a DevOps team can act on.
|
||||
3. **Is this worth fixing?** Isolated regression against the top hotspots. Triggerable and reachable → worth a CI fail. Not triggerable → do not fail the build.
|
||||
|
||||
Stages 2–3 use a tool-calling hunter (read / grep / callers) and a Docker-isolated runner. Evaluation uses split-sink and mutation corpora, not cheat sheets. Missing Docker or a model degrades visibly; stage 1 still runs.
|
||||
|
||||
## Approach
|
||||
|
||||
**Map-then-prove on the existing modes.** Keep `--mode quick|standard|deep`:
|
||||
|
||||
- `quick` → stage 1 only
|
||||
- `standard` → stages 1 + 2
|
||||
- `deep` → stages 1 + 2 + 3
|
||||
|
||||
Do not require the HarnessX extra for the LLM path. Use the existing stdlib OpenRouter client when a hunter model is configured. HarnessX remains an optional alternative runtime.
|
||||
|
||||
Do not add a Python Docker SDK. Drive the Docker CLI via subprocess with the threat-model flags (no network, read-only source, tmpfs scratch, memory/pids/timeout, cap-drop). Core `dependencies = []` stays.
|
||||
|
||||
Complexity map is a new artifact built from existing ranking, entry points, finding density, nesting/size, and test-file presence — not a rewrite of `RankingScore`. Stage 3 writes a worth-fixing verdict that feeds back into a complexity ledger (evolutionary), without editing detection pattern text.
|
||||
|
||||
## Scope
|
||||
|
||||
- **In**: three-stage scan contract; complexity map + test-tuning hints; Docker-isolated regression and worth-fixing verdicts; tool-calling hunter on hotspots; split-sink/mutation evaluation; CI mapping (PR = stage 1, nightly = stage 2/3); visible degradation; evolutionary feedback from regress → map.
|
||||
- **Out**: rewriting CWE policy or project-score formula; auto-authoring regex patterns; full fuzzing campaigns; patch generation/application; network pentest; importing Clearwing.
|
||||
|
||||
## Boundary Candidates
|
||||
|
||||
- Stage orchestrator (which stages a mode runs, artifacts, degradation)
|
||||
- Complexity map (hotspots, signals, test-gap hints)
|
||||
- Sandbox runner (Docker CLI, capability probe)
|
||||
- Regression / worth-fixing (candidate tests, verdicts, ledger feedback)
|
||||
- Tool-calling hunter (repo-bound tools + model loop for stages 2–3)
|
||||
|
||||
## Out of Boundary
|
||||
|
||||
- PolicyStore / ScoreModel / RulesetStore internals (consume, do not own)
|
||||
- MetaAgent.evolve regex-status loop (adjacent, do not replace)
|
||||
- Deep fuzzing, exploit development, auto-patching
|
||||
- HarnessX packaging / optional extra (may be used, not required)
|
||||
|
||||
## Upstream / Downstream
|
||||
|
||||
- **Upstream**: existing preprocess, rank, mapping, findings, verification, config, CLI modes, OpenRouter client, threat-model sandbox limits.
|
||||
- **Downstream**: CI fail-on `worth-fixing`; later fuzz/patch specs; MCP scan status.
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
|
||||
- **Extends**: `openrouter-sast-harness` gates `standard_security_harness` and `sandboxed_dynamic_harness` (still unsatisfied).
|
||||
- **Adjacent**: `harnessx-self-improving-rulesets` owns CWE policy, score, shadow, optional HarnessX extra. This spec must not reopen those.
|
||||
|
||||
## Constraints
|
||||
|
||||
- Core install stays zero-dependency. Docker CLI and an LLM key are operator-provided runtimes, not Python dependencies.
|
||||
- Scanned code is untrusted. Stage 3 is the first mode allowed to execute target-derived code, and only inside the sandbox.
|
||||
- LLM output starts at evidence `suspicion`. It must not be stamped `static_corroboration`.
|
||||
- Cheat-sheet one-liners may remain as unit fixtures; they must not be the merge gate for stages 2–3.
|
||||
@@ -0,0 +1,526 @@
|
||||
# Design Document
|
||||
|
||||
## Overview
|
||||
|
||||
**Purpose.** This feature turns an OpenUltraSAST scan into three operator questions: what exists (static inventory), where problems are most likely and how to aim tests (complexity map), and whether a finding is worth fixing (isolated regression). It is a follow-up to the HarnessX governance spec, not a replacement of CWE policy, scoring, or rules-as-data.
|
||||
|
||||
**Users.** DevOps engineers get a cheap pull-request inventory and a nightly prove-worth-fixing job. AppSec reviewers get a hotspot map and test-gap hints instead of a regex dump. Maintainers get an evaluation gate that split-sink cases can fail.
|
||||
|
||||
**Impact.** Today `--mode standard` reuses regex findings and `--mode deep` exits. After this spec, `quick` remains stage 1, `standard` builds a complexity map and may run a tool-using hunter, and `deep` runs a Docker-isolated regression that is the only path allowed to execute target-derived code.
|
||||
|
||||
### Goals
|
||||
|
||||
- Map existing `--mode quick|standard|deep` onto stages 1 / 1+2 / 1+2+3 with explicit manifest accounting.
|
||||
- Keep stage 1 zero-dependency, no model, no target execution.
|
||||
- Emit a complexity map and test-tuning hints that do not collapse to pattern-hit count.
|
||||
- Prove worth-fixing inside an isolated sandbox; fail CI on `worth_fixing` only when asked.
|
||||
- Hunter tools follow dataflow; hunter output is `suspicion`.
|
||||
- Stage 2/3 quality is gated on split-sink and mutation corpora.
|
||||
|
||||
### Non-Goals
|
||||
|
||||
- Rewriting PolicyStore, ScoreModel, or detection pattern text.
|
||||
- Requiring the HarnessX extra for any stage.
|
||||
- Full fuzzing, exploit development, or auto-patching.
|
||||
- Importing Clearwing.
|
||||
- A second CLI family (`--stage`).
|
||||
|
||||
## Boundary Commitments
|
||||
|
||||
### This Spec Owns
|
||||
|
||||
- **Stage plan**: which stages a mode runs, skip/degrade accounting, `stages_requested` / `stages_completed` in the manifest.
|
||||
- **Complexity map**: `Hotspot` records, signals, test-gap hints, `complexity_map.json`.
|
||||
- **Complexity ledger**: overlay of hotspot scores from regression verdicts.
|
||||
- **Sandbox**: capability probe and Docker CLI runner that never mounts the host Docker socket.
|
||||
- **Regression / worth-fixing**: candidate selection, language recipes, verdicts, `--fail-on worth-fixing`.
|
||||
- **Tool hunter**: repo-bound read / grep / callers loop for stages 2–3, evidence honesty.
|
||||
- **Split-sink evaluation**: corpora and the "beats pattern-hit ranking" gate.
|
||||
|
||||
### Out of Boundary
|
||||
|
||||
- CWE TSV, `resolve_severity`, project-score formula, rule `enabled|shadow|disabled`.
|
||||
- `MetaAgent.evolve` pattern-status loop.
|
||||
- Semgrep/CodeQL execution (ingest of operator-supplied SARIF remains as today).
|
||||
- Patch generation and repository mutation.
|
||||
- Sanitizer-image matrix and campaign fuzzing.
|
||||
- MCP docker/shell tools, and exposing `deep` on MCP (CLI/CI only).
|
||||
- Trail of Bits `skills.STAGES` (`map`/`hunt`/`verify`/`fix`). That `map` is mapping-discipline skill routing, not the complexity-map scan stage.
|
||||
|
||||
### Allowed Dependencies
|
||||
|
||||
- Existing preprocess, rank, mapping, findings, verification, reports, config, CLI, OpenRouter client, HarnessRuntime events.
|
||||
- Optional HarnessX extra as an alternate hunter host, not a requirement.
|
||||
- Operator-provided Docker CLI and LLM credential.
|
||||
- Stdlib only for new core modules (`subprocess` for Docker, `urllib` already used by OpenRouter).
|
||||
|
||||
### Revalidation Triggers
|
||||
|
||||
- Changing the mode→stage mapping or `--fail-on` vocabulary.
|
||||
- Changing hotspot identity keys (breaks the complexity ledger).
|
||||
- Changing sandbox flags or mounting the Docker socket.
|
||||
- Stamping hunter output above `suspicion`.
|
||||
- Using cheat-sheet recall as the stage 2/3 merge gate.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Existing Architecture Analysis
|
||||
|
||||
| Existing element | Constraint |
|
||||
|---|---|
|
||||
| `cli._run_scan` modes | Reuse names; stop exiting on `deep` |
|
||||
| `quick_scan_findings` | Stage 1 detector; not the stage 2 hunter |
|
||||
| `rank.heuristic_rank` | File budget; not the complexity map |
|
||||
| `mapping.analyze_entry_points` | Input to map signals and caller hints |
|
||||
| `hunter._run_hunter_task` | Must stop being a regex alias when a model is set |
|
||||
| `OpenRouterChatClient` | Default hunter transport |
|
||||
| `SandboxConfig` | Limits already match the threat model; need a runner |
|
||||
| `verify_finding` | Keep as structural review gate; do not treat it as worth-fixing |
|
||||
| `gate.py` 90% recall | Stage 1 only after this spec |
|
||||
|
||||
### Architecture Pattern & Boundary Map
|
||||
|
||||
**Selected pattern.** Ordered pipeline: **inventory drives the map; the map drives regression; regression overlays the map.** Execution never mutates governance data. Target code runs only in the sandbox.
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph S1["STAGE 1 inventory"]
|
||||
PRE[preprocess and mapping]
|
||||
RANK[file rank]
|
||||
INV[pattern and SARIF inventory]
|
||||
end
|
||||
subgraph S2["STAGE 2 map"]
|
||||
SIG[complexity signals]
|
||||
MAP[hotspot map]
|
||||
HINT[test-gap hints]
|
||||
HUNT[tool hunter optional]
|
||||
end
|
||||
subgraph S3["STAGE 3 prove"]
|
||||
CAND[regress candidates]
|
||||
BOX[Docker CLI sandbox]
|
||||
VERD[worth-fixing verdicts]
|
||||
end
|
||||
LED[complexity ledger]
|
||||
CI[CI fail-on]
|
||||
PRE --> RANK --> INV
|
||||
INV --> SIG
|
||||
RANK --> SIG
|
||||
LED --> SIG
|
||||
SIG --> MAP --> HINT
|
||||
MAP --> HUNT
|
||||
HUNT --> MAP
|
||||
MAP --> CAND
|
||||
INV --> CAND
|
||||
CAND --> BOX --> VERD
|
||||
VERD --> LED
|
||||
VERD --> CI
|
||||
INV --> CI
|
||||
```
|
||||
|
||||
**Dependency direction:** config/types → preprocess/rank/mapping/findings → complexity map → hunter tools → sandbox → regress → CLI/report/gate.
|
||||
|
||||
### Technology Stack
|
||||
|
||||
| Layer | Choice | Role | Notes |
|
||||
|-------|--------|------|-------|
|
||||
| CLI | existing `ousast scan --mode` | Stage selection | Add `--fail-on worth-fixing` |
|
||||
| Hunter transport | `OpenRouterChatClient` | Tool loop | Stdlib; HarnessX optional |
|
||||
| Sandbox | Docker CLI via `subprocess` | Isolated execution | No Python docker SDK |
|
||||
| Map / ledger | JSON artifacts | Attention + evolution | Same overlay style as `rule_policy.json` |
|
||||
| Evaluation | split-sink fixtures | Stage 2/3 gate | Smoke fixtures stay stage 1 |
|
||||
|
||||
## File Structure Plan
|
||||
|
||||
### Directory Structure
|
||||
|
||||
```
|
||||
src/openultrasast/
|
||||
├── stages.py # Stage enum, mode map, StagePlan, degradation helpers
|
||||
├── complexity/
|
||||
│ ├── __init__.py
|
||||
│ ├── signals.py # per-file/function signals from targets, findings, tests
|
||||
│ ├── map.py # build_complexity_map, Hotspot, ComplexityMap
|
||||
│ ├── hints.py # test-gap hints, closed test-kind set
|
||||
│ └── ledger.py # load/apply/write complexity_ledger.json
|
||||
├── hunter_tools.py # read_file, grep_repo, find_refs; path clamp
|
||||
├── tool_hunter.py # bounded tool loop over a chat client
|
||||
├── sandbox/
|
||||
│ ├── __init__.py
|
||||
│ ├── probe.py # docker CLI present and usable
|
||||
│ └── runner.py # DockerCliRunner, SandboxRunner protocol, FakeSandboxRunner
|
||||
└── regress/
|
||||
├── __init__.py
|
||||
├── candidate.py # select and render candidates
|
||||
├── recipes.py # language command recipes
|
||||
└── verdict.py # run candidates, worth-fixing gate
|
||||
|
||||
benchmarks/fixtures/
|
||||
├── split-sink-python/
|
||||
├── split-sink-javascript/
|
||||
├── split-sink-java/
|
||||
└── split-sink-c/
|
||||
benchmarks/manifests/split-sink-*.toml
|
||||
```
|
||||
|
||||
### Modified Files
|
||||
|
||||
- `cli.py` — run stage 2/3; stop exiting on `deep`; `--fail-on worth-fixing`; write new artifacts.
|
||||
- `reports.py` — `scan_exit_code` grows `worth-fixing`; report sections for inventory, map, and worth-fixing.
|
||||
- `config.py` — `ComplexityConfig` (top_k, max_hunter_hotspots) and `RegressConfig` (max_candidates, image pins); **consume** existing `SandboxConfig` (do not add a second sandbox TOML).
|
||||
- `hunter.py` — schedule remains; execution delegates to `tool_hunter` when a model is set, else inventory-only.
|
||||
- `verification.py` / reports — inventory vs worth-fixing labeling; hunter findings stay `suspicion`.
|
||||
- `gate.py` — stage-1 smoke gate unchanged; add split-sink map gate.
|
||||
- `reports.py` — map and verdict sections in markdown/manifest.
|
||||
- `.github/workflows/ci.yml` — PR stays quick; optional nightly deep when Docker is available.
|
||||
|
||||
## System Flows
|
||||
|
||||
### Mode to stages
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Q[quick] --> A[stage 1]
|
||||
S[standard] --> A
|
||||
S --> B[stage 2]
|
||||
D[deep] --> A
|
||||
D --> B
|
||||
D --> C[stage 3]
|
||||
```
|
||||
|
||||
If stage 2 has no model, the heuristic map still writes. If stage 3 has no Docker, stages 1–2 remain and `sandbox_unavailable` is recorded.
|
||||
|
||||
### Stage 3 worth-fixing
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant M as ComplexityMap
|
||||
participant R as RegressionRunner
|
||||
participant S as DockerCliRunner
|
||||
participant G as WorthFixingGate
|
||||
M->>R: top hotspots plus severe inventory
|
||||
R->>R: render snippet and safety check
|
||||
alt safety check fails
|
||||
R->>G: inconclusive
|
||||
else sandbox missing
|
||||
R->>G: inconclusive sandbox_unavailable
|
||||
else recipe missing
|
||||
R->>G: inconclusive
|
||||
else runs
|
||||
R->>S: docker run read-only source
|
||||
S->>R: exit code stdout stderr
|
||||
R->>G: triggerable or not_triggerable or already_covered
|
||||
end
|
||||
G->>M: ledger overlay
|
||||
```
|
||||
|
||||
## Requirements Traceability
|
||||
|
||||
| Requirement | Summary | Components | Flows |
|
||||
|-------------|---------|------------|-------|
|
||||
| 1.1–1.6 | Mode maps to stages; manifest accounting; degrade forward | `stages.py`, `cli.py` | Mode to stages |
|
||||
| 2.1–2.5 | Stage 1 inventory, no execute, no model | existing preprocess/rank/findings | Stage 1 |
|
||||
| 3.1–3.5 | Complexity map independent of regex count | `complexity/map.py`, `signals.py` | Stage 2 |
|
||||
| 4.1–4.4 | Test-gap hints, no repo writes | `complexity/hints.py` | Stage 2 |
|
||||
| 5.1–5.8 | Sandbox regression, worth-fixing, fail-on | `sandbox/*`, `regress/*` | Stage 3 |
|
||||
| 6.1–6.6 | Tool hunter, path clamp, suspicion | `hunter_tools.py`, `tool_hunter.py` | Stage 2/3 |
|
||||
| 7.1–7.5 | Ledger overlay, cannot hide sev-5 reachable | `complexity/ledger.py` | Stage 3 → map |
|
||||
| 8.1–8.6 | Split-sink / mutation gate | fixtures, `gate.py` | Evaluation |
|
||||
| 9.1–9.6 | PR cheap, nightly bounded, visible skip | `cli.py`, CI workflow | Degradation |
|
||||
|
||||
## Components and Interfaces
|
||||
|
||||
| Component | Domain | Intent | Req Coverage | Dependencies | Contracts |
|
||||
|-----------|--------|--------|--------------|--------------|-----------|
|
||||
| StagePlan | Orchestration | Mode → stages, skip reasons | 1.1–1.6, 9.3–9.6 | config | State |
|
||||
| ComplexityMapBuilder | Map | Hotspots from signals + ledger | 3.1–3.5, 7.1–7.3 | preprocess, rank, findings | Batch, State |
|
||||
| TestTuningAdvisor | Map | Test-gap hints | 4.1–4.4 | ComplexityMap | Batch |
|
||||
| HunterTools | Hunter | Read/grep/refs inside repo | 6.1–6.2 | mapping | Service |
|
||||
| ToolHunter | Hunter | Bounded tool loop | 6.1–6.5 | OpenRouter, HunterTools | Service |
|
||||
| SandboxProbe | Sandbox | Docker usable? | 5.2, 9.3 | docker CLI | Service |
|
||||
| DockerCliRunner | Sandbox | Isolated `docker run` | 5.2, 5.8 | docker CLI | Service |
|
||||
| RegressionRunner | Regress | Candidates, recipes, verdicts | 5.1–5.6 | Sandbox, map | Batch |
|
||||
| WorthFixingGate | Regress | CI fail-on | 5.4–5.7, 7.4 | RegressionRunner, policy | Batch |
|
||||
|
||||
### Orchestration
|
||||
|
||||
#### StagePlan
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Declare which stages run and why a stage was skipped |
|
||||
| Requirements | 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 9.3, 9.4, 9.6 |
|
||||
|
||||
```python
|
||||
from enum import StrEnum
|
||||
|
||||
class Stage(StrEnum):
|
||||
STATIC = "static"
|
||||
MAP = "map"
|
||||
REGRESS = "regress"
|
||||
|
||||
MODE_STAGES = {
|
||||
"quick": (Stage.STATIC,),
|
||||
"standard": (Stage.STATIC, Stage.MAP),
|
||||
"deep": (Stage.STATIC, Stage.MAP, Stage.REGRESS),
|
||||
}
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class StagePlan:
|
||||
mode: str
|
||||
requested: tuple[Stage, ...]
|
||||
completed: tuple[Stage, ...]
|
||||
skipped: tuple[dict[str, str], ...] # stage, reason
|
||||
```
|
||||
|
||||
- **Invariants:** `quick` never includes `MAP` or `REGRESS`. Skipped stages appear in `runtime.state["degradations"]` with the same shape as existing capability fallbacks.
|
||||
- **Naming:** `Stage.MAP` is the complexity-map scan stage. It is unrelated to `skills.STAGES` value `"map"` (Semgrep/CodeQL skill routing). Do not merge the two enumerations.
|
||||
|
||||
### Complexity map
|
||||
|
||||
#### ComplexityMapBuilder
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Rank likely problem areas without using pattern-hit count as the score |
|
||||
| Requirements | 3.1, 3.2, 3.3, 3.4, 3.5 |
|
||||
|
||||
**Signals** (stdlib, no extra libs):
|
||||
|
||||
- size: loc
|
||||
- nesting: max indent or brace depth
|
||||
- surface tags from `FileTarget`
|
||||
- reachability from entry-point hints
|
||||
- inventory density as a *feature*, not the score (used in rationale, down-weighted in the formula)
|
||||
- test presence: sibling `test_*` / `*_test` / `*Test.java` / `*_test.go` under common test roots
|
||||
- ledger overlay last
|
||||
|
||||
**Hotspot identity:** `path` + `function_name or ""`. Function name from reachability hints or a language-specific def/func regex; empty if unknown. Java/Groovy mapping is tag-only today, so identity is usually the file; that is acceptable for ranking and test-gap hints.
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class Hotspot:
|
||||
path: str
|
||||
function_name: str | None
|
||||
score: float
|
||||
band: str # high | medium | low
|
||||
signals: dict[str, float | int | bool | str]
|
||||
rationale: str
|
||||
test_hint: TestHint | None
|
||||
inventory_finding_ids: tuple[str, ...]
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ComplexityMap:
|
||||
hotspots: tuple[Hotspot, ...]
|
||||
heuristic_only: bool
|
||||
```
|
||||
|
||||
- **Invariant:** sorting files by `len(inventory_finding_ids)` descending must not be identical to map order on the split-sink corpus (enforced by the gate).
|
||||
- **Postcondition:** write `.openultrasast/runs/<id>/complexity_map.json`.
|
||||
|
||||
#### TestTuningAdvisor
|
||||
|
||||
Closed `test_kind` set: `unit | property | sanitizer | http-contract | fuzz-harness`.
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class TestHint:
|
||||
path: str
|
||||
function_name: str | None
|
||||
gap: str # no_adjacent_test | no_function_reference | no_fuzz_entry | covered
|
||||
test_kind: str | None
|
||||
reason: str
|
||||
```
|
||||
|
||||
- **Invariant:** `gap == "covered"` ⇒ `test_kind is None`. Never writes into the scanned tree.
|
||||
|
||||
#### Complexity ledger
|
||||
|
||||
Path: `<repo>/.openultrasast/calibration/complexity_ledger.json`.
|
||||
|
||||
```json
|
||||
{
|
||||
"parser.c:parse_header": {
|
||||
"score_delta": 1.5,
|
||||
"last_verdict": "triggerable",
|
||||
"round": 3
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Apply at map build. `score_delta` cannot drop a reachable policy-severity-5 inventory finding out of the stage-3 candidate set.
|
||||
|
||||
### Hunter
|
||||
|
||||
#### HunterTools
|
||||
|
||||
```python
|
||||
def clamp_repo_path(root: Path, user_path: str) -> Path: ...
|
||||
def read_file(root: Path, path: str, *, max_chars: int) -> str: ...
|
||||
def grep_repo(root: Path, pattern: str, *, max_matches: int) -> list[dict[str, object]]: ...
|
||||
def find_refs(root: Path, symbol: str, mapping_index: object) -> list[dict[str, object]]: ...
|
||||
```
|
||||
|
||||
- **Invariant:** any resolved path whose common prefix is not `root` raises `PathEscapesRepo`.
|
||||
- Grep is Python `re` over enumerated source files (zero-dep). Ripgrep is an optional fast path if present, still clamped.
|
||||
|
||||
#### ToolHunter
|
||||
|
||||
Bounded loop: max steps from the hotspot's hunter budget (reuse tier A/B/C caps). Each model turn may call the three tools or stop with a JSON list of suspicions.
|
||||
|
||||
```python
|
||||
def run_tool_hunter(
|
||||
root: Path,
|
||||
hotspots: list[Hotspot],
|
||||
*,
|
||||
client: ChatClient,
|
||||
model: str,
|
||||
max_steps: int,
|
||||
) -> list[StaticFinding]: ...
|
||||
```
|
||||
|
||||
- **Postconditions:** each finding `evidence_level == "suspicion"`; `finding_id` prefix `tool-hunter:`.
|
||||
- **Fallback:** no model → empty hunter findings, `heuristic_only=True` on the map, degradation `hunter_model_unavailable`.
|
||||
|
||||
### Sandbox and regression
|
||||
|
||||
#### SandboxRunner
|
||||
|
||||
```python
|
||||
class SandboxRunner(Protocol):
|
||||
def run(self, spec: SandboxJob) -> SandboxResult: ...
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SandboxJob:
|
||||
image: str
|
||||
command: tuple[str, ...]
|
||||
repo_root: Path
|
||||
scratch_files: dict[str, str] # relative to /scratch
|
||||
timeout_seconds: int
|
||||
memory_mb: int
|
||||
pids_limit: int
|
||||
```
|
||||
|
||||
`DockerCliRunner` argv (conceptual): `docker run --rm --init --network none --read-only --tmpfs /scratch:rw,exec --memory --pids-limit --cap-drop ALL --security-opt no-new-privileges --user` non-root `--mount type=bind,src=repo,dst=/workspace,ro` image command.
|
||||
|
||||
- **Forbidden:** `-v /var/run/docker.sock`, `--network host`, `--privileged`, writable bind of the source tree.
|
||||
- `SandboxProbe.available()` runs `docker info` with a short timeout.
|
||||
|
||||
#### RegressionRunner
|
||||
|
||||
Select `min(max_candidates, top_k_hotspots ∪ sev5_reachable_inventory)`. Render a recipe snippet into `scratch_files`. Safety check rejects snippets that mention `socket`, `curl`, `docker.sock`, or writes under `/workspace`.
|
||||
|
||||
Verdicts:
|
||||
|
||||
| Observation | Verdict |
|
||||
|---|---|
|
||||
| recipe or sandbox missing, timeout, safety reject | `inconclusive` |
|
||||
| existing project test covering the hotspot already passes in sandbox | `already_covered` |
|
||||
| candidate exits non-zero with a crash, assertion, or sanitizer abort | `triggerable` |
|
||||
| candidate exits 0 | `not_triggerable` |
|
||||
|
||||
`worth_fixing` ⇔ `triggerable` and related finding reachability is `reachable` or `inferred-file-surface`.
|
||||
|
||||
`--fail-on` gains `worth-fixing`. Existing `findings` / `verified` unchanged.
|
||||
|
||||
Recipes MVP: Python (`python /scratch/case.py` with `PYTHONPATH=/workspace`), JavaScript (`node /scratch/case.js`), C (`cc` + run) when the image contains a compiler. Java recipe may return `inconclusive` until added.
|
||||
|
||||
## Data Models
|
||||
|
||||
### Manifest additions
|
||||
|
||||
```json
|
||||
{
|
||||
"stages": {
|
||||
"requested": ["static", "map", "regress"],
|
||||
"completed": ["static", "map"],
|
||||
"skipped": [{"stage": "regress", "reason": "sandbox_unavailable"}]
|
||||
},
|
||||
"complexity": {"hotspot_count": 12, "heuristic_only": true},
|
||||
"worth_fixing": {"count": 0, "verdicts": []}
|
||||
}
|
||||
```
|
||||
|
||||
Degradations reuse the existing list: `{ "stage": "regress", "reason": "sandbox_unavailable" }`.
|
||||
|
||||
### Consistency
|
||||
|
||||
- Map artifact and ledger are JSON, resume-safe.
|
||||
- Hunter suspicions never jump evidence levels in this spec.
|
||||
- Stage 1 artifacts remain byte-compatible with current `findings.json` schema; new fields are additive.
|
||||
|
||||
## Error Handling
|
||||
|
||||
| Error | Response |
|
||||
|---|---|
|
||||
| `deep` requested, Docker missing | Complete 1–2, skip 3, degradation, exit 0 unless other fail-on hits |
|
||||
| Model missing on `standard` | Heuristic map, no hunter findings, degradation |
|
||||
| Path escape in a tool | Tool error to the model; do not throw the scan |
|
||||
| Snippet fails safety check | `inconclusive`, do not run |
|
||||
| Docker run timeout | kill container (`--rm`), `inconclusive` |
|
||||
| Ledger would hide sev-5 reachable | Candidate still selected |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit
|
||||
|
||||
- Mode→stage mapping and skip accounting.
|
||||
- Path clamp rejects `../`.
|
||||
- Map order on a fixture where regex hits a trivial helper and the real bug is a split-sink parser — parser ranks higher.
|
||||
- Covered hotspot yields `gap=covered` and no test kind.
|
||||
- Safety check rejects docker.sock and network snippets.
|
||||
- Ledger boost/demote; sev-5 reachable remains a candidate.
|
||||
|
||||
### Integration
|
||||
|
||||
- Fake sandbox: `triggerable` / `not_triggerable` / timeout mapped to verdicts.
|
||||
- Tool hunter with a scripted chat client that must call `grep_repo` before concluding on a split-sink file; fail if it only returns JSON with no tool call.
|
||||
- `--fail-on worth-fixing` exits non-zero only when a `worth_fixing` verdict exists.
|
||||
|
||||
### Evaluation / CI
|
||||
|
||||
- Stage 1: existing `python -m openultrasast.gate` on smoke corpora.
|
||||
- Stage 2: split-sink manifests; true vulnerable function in the high band; map order ≠ pattern-hit order.
|
||||
- Stage 3: `@pytest.mark.docker`; skip if probe fails; at least one planted split-sink that stage 1 missed is `triggerable`.
|
||||
- PR workflow stays `quick`. Nightly/deep is a separate job with Docker.
|
||||
|
||||
### Anti-patterns this spec forbids in tests
|
||||
|
||||
- One-line `eval(request.args)` as the only stage-2 proof.
|
||||
- Asserting hunter findings equal regex findings.
|
||||
- Calling the real Docker daemon in the default `pytest` job.
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- Scanned code is untrusted. Stage 1–2 read it as text. Stage 3 executes only in Docker with network off, source read-only, no docker.sock, dropped caps, no-new-privileges, non-root user.
|
||||
- Hunter tools cannot read outside the repo.
|
||||
- Regression snippets are data, not host shell. They go to `/scratch` in the container.
|
||||
- Secrets redaction already applies to traces and markdown; keep it on for map rationales that quote source.
|
||||
|
||||
## Performance & Scalability
|
||||
|
||||
- Map is O(files) plus one pass of findings. No extra model required.
|
||||
- Tool hunter runs only on `max_hunter_hotspots` (default 8).
|
||||
- Stage 3 `max_candidates` default 5; timeout from `SandboxConfig.timeout_seconds`.
|
||||
- Large-repo CI: PR = stage 1; nightly = stage 2 always, stage 3 if probe passes.
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
P0[StagePlan plus manifest] --> P1[Heuristic map and hints]
|
||||
P1 --> P2[Split-sink gate]
|
||||
P2 --> P3[Tool hunter]
|
||||
P3 --> P4[Docker runner plus recipes]
|
||||
P4 --> P5[Ledger and fail-on worth-fixing]
|
||||
```
|
||||
|
||||
| Phase | Ships | Merge gate | Rollback |
|
||||
|---|---|---|---|
|
||||
| 0 StagePlan | `deep` no longer exits; skips 2/3 with degradations | existing tests | revert CLI |
|
||||
| 1 Heuristic map | `complexity_map.json` on `standard` | unit map tests | feature: empty map writer |
|
||||
| 2 Split-sink gate | new fixtures; map ≠ hit-count | new gate job | gate off |
|
||||
| 3 Tool hunter | OpenRouter loop, suspicion | scripted-client test | model unset |
|
||||
| 4 Sandbox | Docker CLI + Python/JS recipes | docker-marked tests skip-ok | probe skip |
|
||||
| 5 Worth-fixing | ledger + fail-on | integration on fake runner | flag unused |
|
||||
|
||||
`deep` changing from hard-exit to skip-or-run is intentional and documented. No artifact schema is removed.
|
||||
@@ -0,0 +1,129 @@
|
||||
# Requirements Document
|
||||
|
||||
## Introduction
|
||||
|
||||
OpenUltraSAST today answers one question well: which regexes fired. It does not answer the three questions a rugged DevOps CI workflow actually asks: what exists in this repository, where problems are most likely and how to aim tests, and whether a finding is worth fixing. This feature turns the scan into those three stages. Stage 1 is a static inventory of what exists. Stage 2 is a complexity map of likely problem areas plus test-tuning hints. Stage 3 is isolated regression that proves triggerability before CI treats a finding as blocking.
|
||||
|
||||
The work is a follow-up to the HarnessX ruleset-governance spec, not a replacement of it. CWE policy, project score, and rule-status shadowing stay in place. This spec owns the stage contract, the complexity map, the isolated regression verdict, the tool-using hunter on stages 2–3, and an evaluation corpus that is not a cheat sheet.
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: a three-stage scan contract mapped onto existing scan depths; a static inventory that reports what exists without claiming verification; a complexity map of hotspots with test-tuning hints; isolated regression that produces a worth-fixing verdict; a tool-using hunter confined to stages 2–3; evaluation against split-sink and mutation corpora; visible degradation when the sandbox or a model is unavailable; evolutionary feedback from regression verdicts back into the complexity map.
|
||||
- **Out of scope**: rewriting CWE-to-severity policy or the project-score formula; authoring or rewriting detection pattern text; full fuzzing campaigns; generating or applying patches; network pentesting; importing Clearwing.
|
||||
- **Adjacent expectations**: stage 1 continues to run in the zero-dependency default install with no model and no sandbox. CWE policy, scoring, traces, and SARIF remain owned by the existing governance spec. The operator supplies an isolated-sandbox runtime (Docker CLI) for stage 3 and an LLM credential for the tool-using hunter; neither is a packaged library dependency of the core install.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement 1: Three-Stage Scan Contract
|
||||
|
||||
**Objective:** As a DevOps engineer, I want a scan to run as three ordered stages that map onto the existing scan depths, so that CI can choose inventory-only, map-and-tune, or prove-worth-fixing without a new command family.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When a scan runs at depth `quick`, the Scan Orchestrator shall execute stage 1 only and shall not start a model call or execute target-derived code.
|
||||
2. When a scan runs at depth `standard`, the Scan Orchestrator shall execute stage 1 then stage 2.
|
||||
3. When a scan runs at depth `deep`, the Scan Orchestrator shall execute stage 1 then stage 2 then stage 3.
|
||||
4. The Scan Orchestrator shall record the stages requested, the stages completed, and any skipped stage with a reason in the scan manifest.
|
||||
5. If a later stage cannot run, the Scan Orchestrator shall complete earlier stages, record a degradation, and shall not discard those earlier artifacts.
|
||||
6. The Scan Orchestrator shall keep the existing scan command and depth names so existing CI invocations continue to work.
|
||||
|
||||
### Requirement 2: Stage 1 Static Inventory
|
||||
|
||||
**Objective:** As a security reviewer, I want a static inventory of what exists in the repository, so that I can see files, surfaces, and pattern hits without treating them as verified vulnerabilities.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When stage 1 completes, the Static Inventory shall emit a repository snapshot, file targets with language and surface tags, entry-point reachability hints, ranked files, and static findings from the existing pattern ruleset and any provided static-analyzer input.
|
||||
2. The Static Inventory shall label static pattern hits as inventory, not as verified findings, in the report and in the finding evidence level.
|
||||
3. While no static-analyzer input is provided, the Static Inventory shall still complete using the built-in pattern ruleset and heuristics.
|
||||
4. If an enabled rule's CWE does not resolve in policy, the Scan Runtime shall fail loud before producing inventory findings (existing startup invariant, preserved).
|
||||
5. The Static Inventory shall not execute target-derived code and shall not require a model.
|
||||
|
||||
### Requirement 3: Stage 2 Complexity Map
|
||||
|
||||
**Objective:** As an AppSec engineer, I want a complexity map of where problems are most likely, so that review and testing attention go to real hotspots instead of every regex hit.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When stage 2 runs, the Complexity Map shall produce an ordered list of hotspots, each identifying a file and, when known, a function, with a complexity score and a rationale built from size, nesting, surface tags, reachability, existing inventory-hit density, and test-presence gaps.
|
||||
2. The Complexity Map shall rank hotspots independently of a single regex match: a file with no pattern hit may still be a hotspot, and a pattern hit in a trivial helper may rank below a complex reachable parser.
|
||||
3. When stage 2 completes, the Complexity Map shall write a map artifact that the report and later stages consume, including the scoring signals used for each hotspot.
|
||||
4. While a hunter model is unavailable, the Complexity Map shall still produce the heuristic map and shall record that the hunter did not refine it.
|
||||
5. The Complexity Map shall not mark a hotspot as verified or worth fixing.
|
||||
|
||||
### Requirement 4: Test-Tuning Hints
|
||||
|
||||
**Objective:** As a developer owning CI tests, I want the complexity map to tell me how to tune tests, so that I add coverage where it reduces real risk instead of mirroring cheat-sheet sinks.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When the Complexity Map emits a hotspot, the Test Tuning Advisor shall attach a test-gap hint that names the missing coverage (no adjacent test file, no test referencing the function, or no sanitizer/fuzz entry for a fuzzable surface).
|
||||
2. The Test Tuning Advisor shall recommend a test kind from a closed set: unit, property, sanitizer, http-contract, or fuzz-harness, with a one-line reason tied to the hotspot's signals.
|
||||
3. The Test Tuning Advisor shall not write test files into the scanned repository.
|
||||
4. When a hotspot already has covering tests, the Test Tuning Advisor shall say so and shall not recommend a duplicate test kind.
|
||||
|
||||
### Requirement 5: Stage 3 Isolated Regression and Worth-Fixing
|
||||
|
||||
**Objective:** As a CI owner, I want isolated regression that answers whether a finding is worth fixing, so that the build fails only on triggerable, reachable issues.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When stage 3 runs, the Regression Runner shall select a bounded set of top hotspots and/or inventory findings and shall attempt a minimal regression or proof-of-concept inside an isolated analysis sandbox.
|
||||
2. The isolated analysis sandbox shall run with network disabled, the source tree mounted read-only, a writable scratch area, and configured memory, process, and timeout limits.
|
||||
3. The Regression Runner shall emit one of four verdicts per candidate: `triggerable`, `not_triggerable`, `already_covered`, or `inconclusive`.
|
||||
4. When a candidate is `triggerable` and the related finding is reachable, the Worth-Fixing Gate shall mark it `worth_fixing`.
|
||||
5. When a candidate is `not_triggerable` or `already_covered`, the Worth-Fixing Gate shall mark it `not_worth_fixing` and shall not fail the build for that candidate.
|
||||
6. When the verdict is `inconclusive`, the Worth-Fixing Gate shall not fail the build for that candidate and shall record why (timeout, sandbox unavailable, no candidate generated, or setup failed).
|
||||
7. Where `--fail-on worth-fixing` is selected, the Scan Orchestrator shall fail the scan if and only if at least one `worth_fixing` verdict was produced.
|
||||
8. The Regression Runner shall never execute target-derived code on the host outside the isolated sandbox.
|
||||
|
||||
### Requirement 6: Tool-Using Hunter on Stages 2 and 3
|
||||
|
||||
**Objective:** As an OpenUltraSAST maintainer, I want the hunter on later stages to use repository tools instead of dumping a file into a prompt, so that it can follow dataflow across lines and files the way a real review does.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. Where a hunter model is configured, the Hunter Pool shall run a bounded tool-using loop against stage-2 hotspots, with tools limited to reading a file inside the repository, searching repository text, and listing callers or references already known to the scan.
|
||||
2. The Hunter Pool shall confine every tool path to the scanned repository root and shall reject path arguments that escape that root.
|
||||
3. When the hunter reports a candidate issue, the Scan Runtime shall record it at evidence level `suspicion` until independent static or regression evidence raises it.
|
||||
4. While no hunter model is configured, the Hunter Pool shall skip the tool loop, leave the heuristic complexity map in place, and record a degradation.
|
||||
5. The Hunter Pool shall not treat a model JSON list as static corroboration.
|
||||
6. When stage 3 runs and a hunter model is configured, the Hunter Pool may propose a regression snippet, but the Regression Runner shall execute that snippet only inside the isolated sandbox after a structural safety check (no network, no writes outside scratch).
|
||||
|
||||
### Requirement 7: Evolutionary Feedback from Regression to Map
|
||||
|
||||
**Objective:** As a maintainer, I want regression verdicts to retune the complexity map, so that the scan gets better at spending attention where issues actually trigger.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When a hotspot's regression is `triggerable`, the Complexity Map shall raise that hotspot's subsequent score on the same repository.
|
||||
2. When a hotspot's regression is `not_triggerable`, the Complexity Map shall lower that hotspot's subsequent score without deleting the underlying inventory hit.
|
||||
3. The Scan Runtime shall persist those adjustments in a loop-owned ledger keyed by repository and hotspot identity, applied at map load the same way rule status overlays apply at ruleset load.
|
||||
4. If a ledger entry would hide a reachable severity-5 inventory hit from the worth-fixing set, the Worth-Fixing Gate shall still consider that hit.
|
||||
5. The evolutionary loop shall not edit detection pattern text and shall not edit authoritative CWE severity.
|
||||
|
||||
### Requirement 8: Grounded Evaluation, Not Cheat Sheets
|
||||
|
||||
**Objective:** As a maintainer, I want stage 2 and stage 3 quality gates to use split-sink and mutation cases, so that a regex that only matches textbook one-liners cannot claim success.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The Benchmark Harness shall include at least one split-sink case per in-scope language family (Python, JavaScript, Java, C/C++) where the source and the sink are on different lines or the payload is assigned before use.
|
||||
2. The Benchmark Harness shall include at least one mutation of a known sink (equivalent API, intermediate variable, or wrapper) that a same-line pattern does not match.
|
||||
3. When stage 1 is scored alone, the Benchmark Harness may use existing smoke fixtures, and shall not treat stage-1 recall as proof of stage-2 or stage-3 quality.
|
||||
4. When stage 2 is scored, the Benchmark Harness shall require the complexity map to rank the true vulnerable function in the top band of hotspots on the split-sink corpus, even when stage 1 missed the sink.
|
||||
5. When stage 3 is scored and the sandbox is available, the Benchmark Harness shall require a `triggerable` verdict on at least one planted split-sink that stage 1 missed.
|
||||
6. The CI merge gate for this feature shall fail if stage 2 ranking on the split-sink corpus is no better than ranking files by pattern-hit count alone.
|
||||
|
||||
### Requirement 9: CI Companion Budgets and Degradation
|
||||
|
||||
**Objective:** As a CI owner, I want stage costs and failures to be bounded and visible, so that pull-request jobs stay cheap and nightly jobs can run the later stages without silent fallback.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. While running in pull-request depth `quick`, the CI Pipeline shall complete stage 1 without a sandbox runtime and without a model.
|
||||
2. Where a nightly or labeled job requests `standard` or `deep`, the CI Pipeline shall run those stages under the configured time, finding-count, and cost budgets.
|
||||
3. If the sandbox runtime is missing when stage 3 is requested, the Scan Orchestrator shall skip stage 3, record `sandbox_unavailable`, and complete stages 1–2.
|
||||
4. If a model is missing when stage 2 hunter refinement is requested, the Scan Orchestrator shall keep the heuristic map, record `hunter_model_unavailable`, and shall not invent hunter findings.
|
||||
5. The Scan Orchestrator shall bound stage 3 to a configured maximum number of regression candidates per scan.
|
||||
6. When any stage is skipped or truncated, the report shall show the degradation in the same manifest list used by existing capability fallbacks.
|
||||
@@ -0,0 +1,156 @@
|
||||
# Research & Design Decisions
|
||||
|
||||
---
|
||||
**Purpose**: Capture discovery findings and architectural investigation behind the `three-stage-scan` spec. `design.md` carries the committed architecture; this file carries evidence and rejected alternatives.
|
||||
---
|
||||
|
||||
## Summary
|
||||
- **Feature**: `three-stage-scan`
|
||||
- **Discovery Scope**: Extension of the existing scan harness (brownfield), with one new runtime prerequisite (Docker CLI for stage 3) and one new evaluation contract (split-sink / mutation corpora).
|
||||
- **Key Findings**:
|
||||
- Stage 1 already exists as `quick` mode: preprocess, mapping, rank, `quick_scan_findings`, structural verify, score, SARIF. Do not rebuild it.
|
||||
- Stage 2 is not a hunter dump and is not `RankingScore`. Ranking is per-file attention. A complexity map is per-file-and-function hotspot attention plus test-gap hints. Existing signals (tags, loc, reachability hints, finding density, `has_fuzz_entry_point`) are enough for a heuristic map; the tool-using hunter refines it.
|
||||
- Stage 3 has config and a threat-model table, but `cli.py` exits on `--mode deep`. No container is ever started. The first execution of untrusted code must be Docker CLI with the documented isolation flags, not a Python SDK and not host `subprocess` of target code.
|
||||
- The previous HarnessX spec locked LLM use behind an optional extra that CI does not install. This spec uses the existing stdlib OpenRouter client so stage 2 can be evolutionary without that extra. HarnessX remains an optional runtime, not a prerequisite.
|
||||
- Cheat-sheet fixtures cannot gate stages 2–3. The Python fixture already documents a split-sink miss (`query` built on the prior line). That class of case becomes the quality bar.
|
||||
|
||||
## Research Log
|
||||
|
||||
### Existing scan pipeline and mode mapping
|
||||
- **Context**: Whether to add `--stage` or reuse `--mode quick|standard|deep`.
|
||||
- **Sources Consulted**: `src/openultrasast/cli.py` (`_run_scan`, mode choices), `README.md` maturity legend, `.kiro/specs/openrouter-sast-harness/requirements.md` (hunter pool, docker sandbox, dynamic analysis), `.kiro/specs/harnessx-self-improving-rulesets/design.md`.
|
||||
- **Findings**:
|
||||
- `quick` is implemented and deterministic. `standard` schedules a hunter pool that calls `quick_scan_findings` unless HarnessX extra + hunter model are set. `deep` raises `SystemExit`.
|
||||
- OpenRouter spec already defined: quick = static + bounded LLM review; standard = sandboxed tool-calling hunters; deep = harness/fuzz/dynamic. Implementation never reached that.
|
||||
- Existing CI and README teach `--mode`. A parallel `--stage` flag would fork the contract.
|
||||
- **Implications**: Map modes onto stages 1 / 1+2 / 1+2+3. Record `stages_requested` and `stages_completed` in the manifest so the three questions are explicit even if the flag names stay.
|
||||
|
||||
### Ranking vs complexity map
|
||||
- **Context**: `rank.py` already scores files. Is stage 2 just ranking?
|
||||
- **Sources Consulted**: `src/openultrasast/rank.py` (`heuristic_rank`, `composite_priority` = surface 0.5 + influence 0.2 + reachability 0.3), `preprocess.py` (`FileTarget` tags, loc, `has_fuzz_entry_point`), `mapping.py` (`analyze_entry_points`).
|
||||
- **Findings**:
|
||||
- Ranking is file-level and exists to budget the hunter. It does not name functions, test gaps, or "tune your tests."
|
||||
- Tags already encode likely problem classes (`memory_unsafe`, `parser`, `network_entry`, `syscall_entry`). Loc ≥ 500 boosts influence. No nesting/branch signal. No test-file presence.
|
||||
- A hotspot that stage 1 missed (split-sink) can still rank high if the file is a reachable parser. That is the point of stage 2.
|
||||
- **Implications**: Keep `RankingScore`. Add `ComplexityMap` / `Hotspot` with function identity when recoverable, test-gap hints, and a ledger overlay. Ranking still feeds hunter budgets; the map feeds humans and stage 3.
|
||||
|
||||
### Hunter: regex, HarnessX dump, or tools
|
||||
- **Context**: Previous spec adopted HarnessX "at the hunter loop" but the default hunter is regex and the HarnessX path dumps source into a prompt.
|
||||
- **Sources Consulted**: `hunter.py` (`_run_hunter_task` → `quick_scan_findings`), `hunter_harness.py` (`_describe`, `_extract_findings`), `provider/openrouter.py` (`OpenRouterChatClient.complete_json`, stdlib `urllib`), `harness_ext.py` (optional extra).
|
||||
- **Findings**:
|
||||
- OpenRouter client is already zero-extra (stdlib). It is used for optional ranking, not for hunting.
|
||||
- HarnessX extra is SHA-pinned, heavy, absent in CI. Requiring it repeats the last spec's failure mode.
|
||||
- Clearwing hunters use `read_source_file`, `grep_source`, `find_callers` with path clamping. That is the evolutionary piece, not MetaAgent regex-status flips.
|
||||
- **Implications**: Stage 2/3 hunter is a bounded tool loop over the OpenRouter client (or HarnessX if already wired). Tools are stdlib (read, Python grep, caller search from mapping + text). Path clamp at repo root. Hunter output evidence = `suspicion`.
|
||||
|
||||
### Docker sandbox: CLI vs SDK vs skip
|
||||
- **Context**: Threat model already lists sandbox limits. Need a runner that does not invert zero-dep posture.
|
||||
- **Sources Consulted**: `docs/threat-model.md` (sandbox table), `config.py` `SandboxConfig`, Clearwing `hunter_sandbox.py` (image build + spawn, Docker SDK), Docker isolation guidance (`docker run --network none --read-only --tmpfs --memory --pids-limit --cap-drop ALL --security-opt no-new-privileges`), initrunner docker-sandbox docs (CLI `docker run --rm --init`, optional `runsc`).
|
||||
- **Findings**:
|
||||
- Python `docker` SDK would be a new core or extra dependency. The operator already has the Docker CLI if they can run containers.
|
||||
- Subprocess `docker run --rm --init` with bind-mount of the repo read-only and a tmpfs/scratch write area matches the threat model and keeps `dependencies = []`.
|
||||
- Unit tests must not require a daemon: inject a `SandboxRunner` protocol, ship a fake. Integration tests skip when `docker info` fails.
|
||||
- Full sanitizer-image builds (Clearwing) are out of scope for this spec. Stage 3 runs language-specific commands on a small pinned image per language family, or `inconclusive` if the recipe is missing.
|
||||
- **Implications**: `SandboxProbe` + `DockerCliRunner`. No SDK. gVisor/`runsc` is optional `security-opt` when `docker info` reports it; not required.
|
||||
|
||||
### Evaluation corpora
|
||||
- **Context**: User rejected cheat-sheet fixtures. Current gate is ≥90% recall on those fixtures.
|
||||
- **Sources Consulted**: `benchmarks/fixtures/python-vulnerable/app.py` (line 77 split-sink comment), `javascript-vulnerable/src/app.js` (prototype pollution comment), `tests/test_detection_benchmarks.py`, `gate.py`.
|
||||
- **Findings**:
|
||||
- Stage 1 smoke fixtures are still useful as unit tests. They must not be the stage 2/3 merge gate.
|
||||
- A split-sink corpus is small and owned: source and sink on different lines; mutations through wrappers/variables. Success = map ranks the true function in the top band; stage 3 triggers at least one stage-1 miss when Docker is present.
|
||||
- Comparing map ranking to "sort by pattern-hit count" is an anti-cheat-sheet test: if regex density is the map, the gate fails.
|
||||
- **Implications**: New `benchmarks/fixtures/split-sink-*` and a stage-2 ranking gate. Keep existing `openultrasast.gate` for stage 1 only.
|
||||
|
||||
### Seams that already exist and must not be confused
|
||||
- **Context**: Brownfield traps for a spec named "map" that also adds Docker.
|
||||
- **Sources Consulted**: `skills.py` (`STAGES = ("map", "hunt", "verify", "fix")`), `mcp.py:68` (rejects `deep`), `reports.py:115` (`scan_exit_code`), `mapping.py` (no Java/Groovy AST), `config.py` (`SandboxConfig` parsed, never consumed by `_run_scan`), `rank.py` (`rank_targets_with_model` unused by CLI).
|
||||
- **Findings**:
|
||||
- Trail of Bits skill routing already uses a stage named `map` (mapping-discipline skills: Semgrep/CodeQL/sharp-edge). That is not the complexity map. `stages.py` must not reuse `skills.STAGES`.
|
||||
- MCP scan tools reject `deep` and forbid docker/shell. This spec does not widen the MCP surface; deep is CLI/CI only.
|
||||
- `--fail-on` is implemented in `scan_exit_code`, not only argparse. `worth-fixing` must be added there.
|
||||
- Java/Groovy entry points are tag-only (`line=None` → `inferred-file-surface`). Hotspot `function_name` will often be empty on Java; the map must still rank the file.
|
||||
- `[sandbox]`, `[dynamic]`, `[evidence]`, and `[models].ranker` are loaded into resolved config and then ignored. Stage 3 consumes `SandboxConfig`; it does not invent a parallel sandbox TOML.
|
||||
- **Implications**: Naming, MCP, fail-on, and Java identity are integration constraints, not new products.
|
||||
|
||||
### Evidence ladder honesty
|
||||
- **Context**: HarnessX hunter stamped LLM JSON as `static_corroboration`. Original spec forbids reporting LLM-only claims as verified.
|
||||
- **Sources Consulted**: `verification.py` (`verify_finding` accepts when reachability is not `unknown`), `hunter_harness.py` evidence_level assignment, openrouter requirements Evidence Ladder / Verification.
|
||||
- **Findings**:
|
||||
- Pattern hits as `static_corroboration` is consistent with the ladder. LLM hits as `suspicion` is the missing rule.
|
||||
- `--fail-on verified` today means reachable + static corroboration. That remains the cheap PR gate. `--fail-on worth-fixing` is the rugged gate.
|
||||
- **Implications**: Do not change CWE policy. Change hunter evidence. Add fail-on value. Report sections: Inventory, Complexity map, Worth fixing.
|
||||
|
||||
## Architecture Pattern Evaluation
|
||||
|
||||
| Option | Description | Strengths | Risks / Limitations | Notes |
|
||||
|--------|-------------|-----------|---------------------|-------|
|
||||
| Map-then-prove on existing modes | `quick`/`standard`/`deep` = stages 1 / 1+2 / 1+2+3 | Smallest CLI blast radius; matches threat-model "deep executes in sandbox"; reuses preprocess/rank/findings | Two vocabularies (mode vs stage) until the manifest lists both | **Selected** |
|
||||
| New `--stage` command family | Parallel CLI, leave modes as-is | Clearer naming | Forks CI; `deep` still a stub | Rejected |
|
||||
| Clearwing-shaped full sourcehunt | Tool hunters + sanitizer images + PoC + patch | Highest detection efficiency | Fork in all but name; out of original non-goals | Rejected as this spec; remains the oracle |
|
||||
| HarnessX-required agent loop | Stage 2/3 only run with the extra | Reuses last spec's seam | CI never installs it; last spec's failure | Rejected as a requirement; optional runtime only |
|
||||
|
||||
## Design Decisions
|
||||
|
||||
### Decision D1: Modes map to stages
|
||||
- **Context**: Need a three-question scan without breaking CI that already passes `--mode`.
|
||||
- **Selected Approach**: `quick`→stage 1, `standard`→1+2, `deep`→1+2+3. Manifest records both.
|
||||
- **Rationale**: Evolutionary adoption of the CLI the project already shipped.
|
||||
- **Trade-offs**: Operators must learn that `standard` now means "map" not "regex hunter pool." README must say so.
|
||||
|
||||
### Decision D2: Docker CLI subprocess, no SDK
|
||||
- **Context**: Zero-dep core vs isolated execution.
|
||||
- **Selected Approach**: `docker` binary via `subprocess`; probe with `docker info`; fake runner in unit tests.
|
||||
- **Rationale**: Operator runtime, not a Python dependency. Matches threat-model flags.
|
||||
- **Trade-offs**: Windows/macOS Docker Desktop variance; integration tests skip without a daemon.
|
||||
|
||||
### Decision D3: OpenRouter client for the tool hunter; HarnessX optional
|
||||
- **Context**: Last spec hid the LLM behind an extra CI does not install.
|
||||
- **Selected Approach**: Tool loop calls `OpenRouterChatClient` (or any `RankerClient`-shaped chat client) when `[models].hunter` is set. If HarnessX extra is present, `HxScanOrchestrator` may host the same tools; it is not required.
|
||||
- **Rationale**: Evolutionary use of code already in tree. Stdlib HTTP.
|
||||
- **Trade-offs**: Two possible runtimes until HarnessX is either used or ignored; tests mock the client.
|
||||
|
||||
### Decision D4: Complexity map is a new artifact
|
||||
- **Context**: Ranking already exists.
|
||||
- **Selected Approach**: `ComplexityMap` with `Hotspot` records. Ranking remains the hunter budget. Map is what humans and stage 3 consume.
|
||||
- **Rationale**: Different question (where to look / how to test vs which file to hunt first).
|
||||
|
||||
### Decision D5: Worth-fixing is a gate, not an evidence level
|
||||
- **Context**: Evidence ladder is already six rungs. Do not invent a seventh for CI.
|
||||
- **Selected Approach**: Verdicts `triggerable|not_triggerable|already_covered|inconclusive` → `worth_fixing|not_worth_fixing`. Evidence levels unchanged. Hunter stays at `suspicion`.
|
||||
- **Rationale**: CI owners need a fail-on that means "we proved it," without collapsing the ladder.
|
||||
|
||||
### Decision D6: Split-sink / mutation gate for stages 2–3
|
||||
- **Context**: 90% recall on cheat sheets is circular.
|
||||
- **Selected Approach**: New corpus + "beats pattern-hit ranking" test. Stage 1 smoke gate stays.
|
||||
- **Rationale**: Directly answers the user's evaluation complaint.
|
||||
|
||||
### Decision D7: Ledger overlays hotspot scores, not patterns
|
||||
- **Context**: Evolutionary feedback without repeating "loop must not edit patterns."
|
||||
- **Selected Approach**: `.openultrasast/calibration/complexity_ledger.json` keyed by `repo_fingerprint + path + function`. Triggerable boosts, not_triggerable demotes. Cannot hide reachable severity-5 inventory from worth-fixing consideration.
|
||||
- **Rationale**: Same overlay pattern as `rule_policy.json`, different object.
|
||||
|
||||
### Decision D8: Language recipes, not a universal executor
|
||||
- **Context**: Running untrusted Python vs C vs Java is different.
|
||||
- **Selected Approach**: Small recipe registry (Python and JavaScript required for MVP; C compile+run when a compiler exists in the image; Java `inconclusive` until a recipe ships). Missing recipe → `inconclusive`, not a crash.
|
||||
- **Rationale**: Simplification. Do not build Clearwing's sanitizer image matrix in this spec.
|
||||
|
||||
## Synthesis
|
||||
|
||||
- **Generalization**: Inventory → attention → proof is one pipeline with a `StagePlan`. Map, hints, and regress are processors on that plan, not three products.
|
||||
- **Build vs adopt**: Adopt Docker CLI and the in-tree OpenRouter client. Build the map, ledger, recipes, and split-sink corpus. Do not adopt Clearwing, docker SDK, or a cyclomatic-complexity library (nesting depth from indentation/braces is enough).
|
||||
- **Simplification**: No `--stage` flag, no HarnessX requirement, no fuzz campaign, no patch oracle, no Python Docker extra.
|
||||
|
||||
## Risks & Mitigations
|
||||
- Docker-in-Docker / rootless CI — probe, skip stage 3, degrade; do not fail stage 1.
|
||||
- Hunter cost on large repos — hotspot cap + existing `[harnessx] max_cost_usd` / hardening `max_findings` / new `max_regress_candidates`.
|
||||
- Untrusted snippet execution — structural deny (network, writes outside scratch, curl, docker socket mount). Never mount `/var/run/docker.sock`.
|
||||
- Stage 2 still heuristic without a model — allowed; gate is ranking quality on split-sink, which heuristics can pass if tags+reachability work.
|
||||
- Breaking `deep` from "exits" to "runs" — intended; document in README.
|
||||
|
||||
## References
|
||||
- OpenUltraSAST threat model — `docs/threat-model.md`
|
||||
- OpenRouter SAST harness requirements — Docker sandbox, hunter pool, evidence ladder
|
||||
- HarnessX self-improving rulesets — adjacent governance; do not reopen
|
||||
- Docker run isolation flags — `--network none`, `--read-only`, `--cap-drop ALL`, `--security-opt no-new-privileges`
|
||||
- Clearwing sourcehunt hunter tools and sandbox — implementation oracle, not a dependency
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"feature_name": "three-stage-scan",
|
||||
"created_at": "2026-09-03T16:10:07Z",
|
||||
"updated_at": "2026-09-03T16:47:00Z",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"design": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"tasks": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": true
|
||||
}
|
||||
Reference in New Issue
Block a user