From 723e3a76cf5e867bed4668d1aa3eb627a75f2f2b Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Thu, 3 Sep 2026 19:19:58 +0200 Subject: [PATCH] docs(three-stage-scan): add approved three-stage scan spec --- .kiro/specs/three-stage-scan/brief.md | 74 +++ .kiro/specs/three-stage-scan/design.md | 526 +++++++++++++++++++ .kiro/specs/three-stage-scan/requirements.md | 129 +++++ .kiro/specs/three-stage-scan/research.md | 156 ++++++ .kiro/specs/three-stage-scan/spec.json | 22 + 5 files changed, 907 insertions(+) create mode 100644 .kiro/specs/three-stage-scan/brief.md create mode 100644 .kiro/specs/three-stage-scan/design.md create mode 100644 .kiro/specs/three-stage-scan/requirements.md create mode 100644 .kiro/specs/three-stage-scan/research.md create mode 100644 .kiro/specs/three-stage-scan/spec.json diff --git a/.kiro/specs/three-stage-scan/brief.md b/.kiro/specs/three-stage-scan/brief.md new file mode 100644 index 0000000..882e018 --- /dev/null +++ b/.kiro/specs/three-stage-scan/brief.md @@ -0,0 +1,74 @@ +# Brief: three-stage-scan + +## Problem + +AppSec engineers and DevOps teams running OpenUltraSAST in CI get a regex static scan scored against cheat-sheet fixtures. They cannot tell what actually exists in a real repository, where complexity concentrates, how to aim tests, or whether a finding is worth a CI failure. The previous HarnessX spec added governance around those regexes (CWE policy, score, shadow status) without making the hunter evolutionary or the evaluation realistic. + +## Current State + +- `ousast scan --mode quick` is a deterministic regex inventory. It works. +- `--mode standard` is labeled a hunter pool but reuses the same regex path unless the optional HarnessX extra *and* a hunter model are both set. That hunter dumps a file into a prompt and scrapes JSON; it has no tools. +- `--mode deep` exits with "not implemented." Sandbox config exists on paper (`network=false`, read-only workspace, memory/pids/timeout). No container is ever started. +- Benchmarks are one-line sinks written to match the regexes. The fixtures themselves comment the misses (`regex blind spot`). The ≥90% recall gate is circular. +- CWE policy, project score, shadow status, traces, and SARIF already exist and should not be redesigned. + +## Desired Outcome + +A scan answers three operator questions, in order: + +1. **What exists?** A static inventory of files, surfaces, and pattern/SARIF hits — honest about being inventory, not verified bugs. +2. **Where are problems most likely, and how do I tune tests?** A complexity map of hotspots plus test-gap hints a DevOps team can act on. +3. **Is this worth fixing?** Isolated regression against the top hotspots. Triggerable and reachable → worth a CI fail. Not triggerable → do not fail the build. + +Stages 2–3 use a tool-calling hunter (read / grep / callers) and a Docker-isolated runner. Evaluation uses split-sink and mutation corpora, not cheat sheets. Missing Docker or a model degrades visibly; stage 1 still runs. + +## Approach + +**Map-then-prove on the existing modes.** Keep `--mode quick|standard|deep`: + +- `quick` → stage 1 only +- `standard` → stages 1 + 2 +- `deep` → stages 1 + 2 + 3 + +Do not require the HarnessX extra for the LLM path. Use the existing stdlib OpenRouter client when a hunter model is configured. HarnessX remains an optional alternative runtime. + +Do not add a Python Docker SDK. Drive the Docker CLI via subprocess with the threat-model flags (no network, read-only source, tmpfs scratch, memory/pids/timeout, cap-drop). Core `dependencies = []` stays. + +Complexity map is a new artifact built from existing ranking, entry points, finding density, nesting/size, and test-file presence — not a rewrite of `RankingScore`. Stage 3 writes a worth-fixing verdict that feeds back into a complexity ledger (evolutionary), without editing detection pattern text. + +## Scope + +- **In**: three-stage scan contract; complexity map + test-tuning hints; Docker-isolated regression and worth-fixing verdicts; tool-calling hunter on hotspots; split-sink/mutation evaluation; CI mapping (PR = stage 1, nightly = stage 2/3); visible degradation; evolutionary feedback from regress → map. +- **Out**: rewriting CWE policy or project-score formula; auto-authoring regex patterns; full fuzzing campaigns; patch generation/application; network pentest; importing Clearwing. + +## Boundary Candidates + +- Stage orchestrator (which stages a mode runs, artifacts, degradation) +- Complexity map (hotspots, signals, test-gap hints) +- Sandbox runner (Docker CLI, capability probe) +- Regression / worth-fixing (candidate tests, verdicts, ledger feedback) +- Tool-calling hunter (repo-bound tools + model loop for stages 2–3) + +## Out of Boundary + +- PolicyStore / ScoreModel / RulesetStore internals (consume, do not own) +- MetaAgent.evolve regex-status loop (adjacent, do not replace) +- Deep fuzzing, exploit development, auto-patching +- HarnessX packaging / optional extra (may be used, not required) + +## Upstream / Downstream + +- **Upstream**: existing preprocess, rank, mapping, findings, verification, config, CLI modes, OpenRouter client, threat-model sandbox limits. +- **Downstream**: CI fail-on `worth-fixing`; later fuzz/patch specs; MCP scan status. + +## Existing Spec Touchpoints + +- **Extends**: `openrouter-sast-harness` gates `standard_security_harness` and `sandboxed_dynamic_harness` (still unsatisfied). +- **Adjacent**: `harnessx-self-improving-rulesets` owns CWE policy, score, shadow, optional HarnessX extra. This spec must not reopen those. + +## Constraints + +- Core install stays zero-dependency. Docker CLI and an LLM key are operator-provided runtimes, not Python dependencies. +- Scanned code is untrusted. Stage 3 is the first mode allowed to execute target-derived code, and only inside the sandbox. +- LLM output starts at evidence `suspicion`. It must not be stamped `static_corroboration`. +- Cheat-sheet one-liners may remain as unit fixtures; they must not be the merge gate for stages 2–3. diff --git a/.kiro/specs/three-stage-scan/design.md b/.kiro/specs/three-stage-scan/design.md new file mode 100644 index 0000000..c75451e --- /dev/null +++ b/.kiro/specs/three-stage-scan/design.md @@ -0,0 +1,526 @@ +# Design Document + +## Overview + +**Purpose.** This feature turns an OpenUltraSAST scan into three operator questions: what exists (static inventory), where problems are most likely and how to aim tests (complexity map), and whether a finding is worth fixing (isolated regression). It is a follow-up to the HarnessX governance spec, not a replacement of CWE policy, scoring, or rules-as-data. + +**Users.** DevOps engineers get a cheap pull-request inventory and a nightly prove-worth-fixing job. AppSec reviewers get a hotspot map and test-gap hints instead of a regex dump. Maintainers get an evaluation gate that split-sink cases can fail. + +**Impact.** Today `--mode standard` reuses regex findings and `--mode deep` exits. After this spec, `quick` remains stage 1, `standard` builds a complexity map and may run a tool-using hunter, and `deep` runs a Docker-isolated regression that is the only path allowed to execute target-derived code. + +### Goals + +- Map existing `--mode quick|standard|deep` onto stages 1 / 1+2 / 1+2+3 with explicit manifest accounting. +- Keep stage 1 zero-dependency, no model, no target execution. +- Emit a complexity map and test-tuning hints that do not collapse to pattern-hit count. +- Prove worth-fixing inside an isolated sandbox; fail CI on `worth_fixing` only when asked. +- Hunter tools follow dataflow; hunter output is `suspicion`. +- Stage 2/3 quality is gated on split-sink and mutation corpora. + +### Non-Goals + +- Rewriting PolicyStore, ScoreModel, or detection pattern text. +- Requiring the HarnessX extra for any stage. +- Full fuzzing, exploit development, or auto-patching. +- Importing Clearwing. +- A second CLI family (`--stage`). + +## Boundary Commitments + +### This Spec Owns + +- **Stage plan**: which stages a mode runs, skip/degrade accounting, `stages_requested` / `stages_completed` in the manifest. +- **Complexity map**: `Hotspot` records, signals, test-gap hints, `complexity_map.json`. +- **Complexity ledger**: overlay of hotspot scores from regression verdicts. +- **Sandbox**: capability probe and Docker CLI runner that never mounts the host Docker socket. +- **Regression / worth-fixing**: candidate selection, language recipes, verdicts, `--fail-on worth-fixing`. +- **Tool hunter**: repo-bound read / grep / callers loop for stages 2–3, evidence honesty. +- **Split-sink evaluation**: corpora and the "beats pattern-hit ranking" gate. + +### Out of Boundary + +- CWE TSV, `resolve_severity`, project-score formula, rule `enabled|shadow|disabled`. +- `MetaAgent.evolve` pattern-status loop. +- Semgrep/CodeQL execution (ingest of operator-supplied SARIF remains as today). +- Patch generation and repository mutation. +- Sanitizer-image matrix and campaign fuzzing. +- MCP docker/shell tools, and exposing `deep` on MCP (CLI/CI only). +- Trail of Bits `skills.STAGES` (`map`/`hunt`/`verify`/`fix`). That `map` is mapping-discipline skill routing, not the complexity-map scan stage. + +### Allowed Dependencies + +- Existing preprocess, rank, mapping, findings, verification, reports, config, CLI, OpenRouter client, HarnessRuntime events. +- Optional HarnessX extra as an alternate hunter host, not a requirement. +- Operator-provided Docker CLI and LLM credential. +- Stdlib only for new core modules (`subprocess` for Docker, `urllib` already used by OpenRouter). + +### Revalidation Triggers + +- Changing the mode→stage mapping or `--fail-on` vocabulary. +- Changing hotspot identity keys (breaks the complexity ledger). +- Changing sandbox flags or mounting the Docker socket. +- Stamping hunter output above `suspicion`. +- Using cheat-sheet recall as the stage 2/3 merge gate. + +## Architecture + +### Existing Architecture Analysis + +| Existing element | Constraint | +|---|---| +| `cli._run_scan` modes | Reuse names; stop exiting on `deep` | +| `quick_scan_findings` | Stage 1 detector; not the stage 2 hunter | +| `rank.heuristic_rank` | File budget; not the complexity map | +| `mapping.analyze_entry_points` | Input to map signals and caller hints | +| `hunter._run_hunter_task` | Must stop being a regex alias when a model is set | +| `OpenRouterChatClient` | Default hunter transport | +| `SandboxConfig` | Limits already match the threat model; need a runner | +| `verify_finding` | Keep as structural review gate; do not treat it as worth-fixing | +| `gate.py` 90% recall | Stage 1 only after this spec | + +### Architecture Pattern & Boundary Map + +**Selected pattern.** Ordered pipeline: **inventory drives the map; the map drives regression; regression overlays the map.** Execution never mutates governance data. Target code runs only in the sandbox. + +```mermaid +flowchart TB + subgraph S1["STAGE 1 inventory"] + PRE[preprocess and mapping] + RANK[file rank] + INV[pattern and SARIF inventory] + end + subgraph S2["STAGE 2 map"] + SIG[complexity signals] + MAP[hotspot map] + HINT[test-gap hints] + HUNT[tool hunter optional] + end + subgraph S3["STAGE 3 prove"] + CAND[regress candidates] + BOX[Docker CLI sandbox] + VERD[worth-fixing verdicts] + end + LED[complexity ledger] + CI[CI fail-on] + PRE --> RANK --> INV + INV --> SIG + RANK --> SIG + LED --> SIG + SIG --> MAP --> HINT + MAP --> HUNT + HUNT --> MAP + MAP --> CAND + INV --> CAND + CAND --> BOX --> VERD + VERD --> LED + VERD --> CI + INV --> CI +``` + +**Dependency direction:** config/types → preprocess/rank/mapping/findings → complexity map → hunter tools → sandbox → regress → CLI/report/gate. + +### Technology Stack + +| Layer | Choice | Role | Notes | +|-------|--------|------|-------| +| CLI | existing `ousast scan --mode` | Stage selection | Add `--fail-on worth-fixing` | +| Hunter transport | `OpenRouterChatClient` | Tool loop | Stdlib; HarnessX optional | +| Sandbox | Docker CLI via `subprocess` | Isolated execution | No Python docker SDK | +| Map / ledger | JSON artifacts | Attention + evolution | Same overlay style as `rule_policy.json` | +| Evaluation | split-sink fixtures | Stage 2/3 gate | Smoke fixtures stay stage 1 | + +## File Structure Plan + +### Directory Structure + +``` +src/openultrasast/ +├── stages.py # Stage enum, mode map, StagePlan, degradation helpers +├── complexity/ +│ ├── __init__.py +│ ├── signals.py # per-file/function signals from targets, findings, tests +│ ├── map.py # build_complexity_map, Hotspot, ComplexityMap +│ ├── hints.py # test-gap hints, closed test-kind set +│ └── ledger.py # load/apply/write complexity_ledger.json +├── hunter_tools.py # read_file, grep_repo, find_refs; path clamp +├── tool_hunter.py # bounded tool loop over a chat client +├── sandbox/ +│ ├── __init__.py +│ ├── probe.py # docker CLI present and usable +│ └── runner.py # DockerCliRunner, SandboxRunner protocol, FakeSandboxRunner +└── regress/ + ├── __init__.py + ├── candidate.py # select and render candidates + ├── recipes.py # language command recipes + └── verdict.py # run candidates, worth-fixing gate + +benchmarks/fixtures/ +├── split-sink-python/ +├── split-sink-javascript/ +├── split-sink-java/ +└── split-sink-c/ +benchmarks/manifests/split-sink-*.toml +``` + +### Modified Files + +- `cli.py` — run stage 2/3; stop exiting on `deep`; `--fail-on worth-fixing`; write new artifacts. +- `reports.py` — `scan_exit_code` grows `worth-fixing`; report sections for inventory, map, and worth-fixing. +- `config.py` — `ComplexityConfig` (top_k, max_hunter_hotspots) and `RegressConfig` (max_candidates, image pins); **consume** existing `SandboxConfig` (do not add a second sandbox TOML). +- `hunter.py` — schedule remains; execution delegates to `tool_hunter` when a model is set, else inventory-only. +- `verification.py` / reports — inventory vs worth-fixing labeling; hunter findings stay `suspicion`. +- `gate.py` — stage-1 smoke gate unchanged; add split-sink map gate. +- `reports.py` — map and verdict sections in markdown/manifest. +- `.github/workflows/ci.yml` — PR stays quick; optional nightly deep when Docker is available. + +## System Flows + +### Mode to stages + +```mermaid +flowchart LR + Q[quick] --> A[stage 1] + S[standard] --> A + S --> B[stage 2] + D[deep] --> A + D --> B + D --> C[stage 3] +``` + +If stage 2 has no model, the heuristic map still writes. If stage 3 has no Docker, stages 1–2 remain and `sandbox_unavailable` is recorded. + +### Stage 3 worth-fixing + +```mermaid +sequenceDiagram + participant M as ComplexityMap + participant R as RegressionRunner + participant S as DockerCliRunner + participant G as WorthFixingGate + M->>R: top hotspots plus severe inventory + R->>R: render snippet and safety check + alt safety check fails + R->>G: inconclusive + else sandbox missing + R->>G: inconclusive sandbox_unavailable + else recipe missing + R->>G: inconclusive + else runs + R->>S: docker run read-only source + S->>R: exit code stdout stderr + R->>G: triggerable or not_triggerable or already_covered + end + G->>M: ledger overlay +``` + +## Requirements Traceability + +| Requirement | Summary | Components | Flows | +|-------------|---------|------------|-------| +| 1.1–1.6 | Mode maps to stages; manifest accounting; degrade forward | `stages.py`, `cli.py` | Mode to stages | +| 2.1–2.5 | Stage 1 inventory, no execute, no model | existing preprocess/rank/findings | Stage 1 | +| 3.1–3.5 | Complexity map independent of regex count | `complexity/map.py`, `signals.py` | Stage 2 | +| 4.1–4.4 | Test-gap hints, no repo writes | `complexity/hints.py` | Stage 2 | +| 5.1–5.8 | Sandbox regression, worth-fixing, fail-on | `sandbox/*`, `regress/*` | Stage 3 | +| 6.1–6.6 | Tool hunter, path clamp, suspicion | `hunter_tools.py`, `tool_hunter.py` | Stage 2/3 | +| 7.1–7.5 | Ledger overlay, cannot hide sev-5 reachable | `complexity/ledger.py` | Stage 3 → map | +| 8.1–8.6 | Split-sink / mutation gate | fixtures, `gate.py` | Evaluation | +| 9.1–9.6 | PR cheap, nightly bounded, visible skip | `cli.py`, CI workflow | Degradation | + +## Components and Interfaces + +| Component | Domain | Intent | Req Coverage | Dependencies | Contracts | +|-----------|--------|--------|--------------|--------------|-----------| +| StagePlan | Orchestration | Mode → stages, skip reasons | 1.1–1.6, 9.3–9.6 | config | State | +| ComplexityMapBuilder | Map | Hotspots from signals + ledger | 3.1–3.5, 7.1–7.3 | preprocess, rank, findings | Batch, State | +| TestTuningAdvisor | Map | Test-gap hints | 4.1–4.4 | ComplexityMap | Batch | +| HunterTools | Hunter | Read/grep/refs inside repo | 6.1–6.2 | mapping | Service | +| ToolHunter | Hunter | Bounded tool loop | 6.1–6.5 | OpenRouter, HunterTools | Service | +| SandboxProbe | Sandbox | Docker usable? | 5.2, 9.3 | docker CLI | Service | +| DockerCliRunner | Sandbox | Isolated `docker run` | 5.2, 5.8 | docker CLI | Service | +| RegressionRunner | Regress | Candidates, recipes, verdicts | 5.1–5.6 | Sandbox, map | Batch | +| WorthFixingGate | Regress | CI fail-on | 5.4–5.7, 7.4 | RegressionRunner, policy | Batch | + +### Orchestration + +#### StagePlan + +| Field | Detail | +|-------|--------| +| Intent | Declare which stages run and why a stage was skipped | +| Requirements | 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 9.3, 9.4, 9.6 | + +```python +from enum import StrEnum + +class Stage(StrEnum): + STATIC = "static" + MAP = "map" + REGRESS = "regress" + +MODE_STAGES = { + "quick": (Stage.STATIC,), + "standard": (Stage.STATIC, Stage.MAP), + "deep": (Stage.STATIC, Stage.MAP, Stage.REGRESS), +} + +@dataclass(frozen=True) +class StagePlan: + mode: str + requested: tuple[Stage, ...] + completed: tuple[Stage, ...] + skipped: tuple[dict[str, str], ...] # stage, reason +``` + +- **Invariants:** `quick` never includes `MAP` or `REGRESS`. Skipped stages appear in `runtime.state["degradations"]` with the same shape as existing capability fallbacks. +- **Naming:** `Stage.MAP` is the complexity-map scan stage. It is unrelated to `skills.STAGES` value `"map"` (Semgrep/CodeQL skill routing). Do not merge the two enumerations. + +### Complexity map + +#### ComplexityMapBuilder + +| Field | Detail | +|-------|--------| +| Intent | Rank likely problem areas without using pattern-hit count as the score | +| Requirements | 3.1, 3.2, 3.3, 3.4, 3.5 | + +**Signals** (stdlib, no extra libs): + +- size: loc +- nesting: max indent or brace depth +- surface tags from `FileTarget` +- reachability from entry-point hints +- inventory density as a *feature*, not the score (used in rationale, down-weighted in the formula) +- test presence: sibling `test_*` / `*_test` / `*Test.java` / `*_test.go` under common test roots +- ledger overlay last + +**Hotspot identity:** `path` + `function_name or ""`. Function name from reachability hints or a language-specific def/func regex; empty if unknown. Java/Groovy mapping is tag-only today, so identity is usually the file; that is acceptable for ranking and test-gap hints. + +```python +@dataclass(frozen=True) +class Hotspot: + path: str + function_name: str | None + score: float + band: str # high | medium | low + signals: dict[str, float | int | bool | str] + rationale: str + test_hint: TestHint | None + inventory_finding_ids: tuple[str, ...] + +@dataclass(frozen=True) +class ComplexityMap: + hotspots: tuple[Hotspot, ...] + heuristic_only: bool +``` + +- **Invariant:** sorting files by `len(inventory_finding_ids)` descending must not be identical to map order on the split-sink corpus (enforced by the gate). +- **Postcondition:** write `.openultrasast/runs//complexity_map.json`. + +#### TestTuningAdvisor + +Closed `test_kind` set: `unit | property | sanitizer | http-contract | fuzz-harness`. + +```python +@dataclass(frozen=True) +class TestHint: + path: str + function_name: str | None + gap: str # no_adjacent_test | no_function_reference | no_fuzz_entry | covered + test_kind: str | None + reason: str +``` + +- **Invariant:** `gap == "covered"` ⇒ `test_kind is None`. Never writes into the scanned tree. + +#### Complexity ledger + +Path: `/.openultrasast/calibration/complexity_ledger.json`. + +```json +{ + "parser.c:parse_header": { + "score_delta": 1.5, + "last_verdict": "triggerable", + "round": 3 + } +} +``` + +Apply at map build. `score_delta` cannot drop a reachable policy-severity-5 inventory finding out of the stage-3 candidate set. + +### Hunter + +#### HunterTools + +```python +def clamp_repo_path(root: Path, user_path: str) -> Path: ... +def read_file(root: Path, path: str, *, max_chars: int) -> str: ... +def grep_repo(root: Path, pattern: str, *, max_matches: int) -> list[dict[str, object]]: ... +def find_refs(root: Path, symbol: str, mapping_index: object) -> list[dict[str, object]]: ... +``` + +- **Invariant:** any resolved path whose common prefix is not `root` raises `PathEscapesRepo`. +- Grep is Python `re` over enumerated source files (zero-dep). Ripgrep is an optional fast path if present, still clamped. + +#### ToolHunter + +Bounded loop: max steps from the hotspot's hunter budget (reuse tier A/B/C caps). Each model turn may call the three tools or stop with a JSON list of suspicions. + +```python +def run_tool_hunter( + root: Path, + hotspots: list[Hotspot], + *, + client: ChatClient, + model: str, + max_steps: int, +) -> list[StaticFinding]: ... +``` + +- **Postconditions:** each finding `evidence_level == "suspicion"`; `finding_id` prefix `tool-hunter:`. +- **Fallback:** no model → empty hunter findings, `heuristic_only=True` on the map, degradation `hunter_model_unavailable`. + +### Sandbox and regression + +#### SandboxRunner + +```python +class SandboxRunner(Protocol): + def run(self, spec: SandboxJob) -> SandboxResult: ... + +@dataclass(frozen=True) +class SandboxJob: + image: str + command: tuple[str, ...] + repo_root: Path + scratch_files: dict[str, str] # relative to /scratch + timeout_seconds: int + memory_mb: int + pids_limit: int +``` + +`DockerCliRunner` argv (conceptual): `docker run --rm --init --network none --read-only --tmpfs /scratch:rw,exec --memory --pids-limit --cap-drop ALL --security-opt no-new-privileges --user` non-root `--mount type=bind,src=repo,dst=/workspace,ro` image command. + +- **Forbidden:** `-v /var/run/docker.sock`, `--network host`, `--privileged`, writable bind of the source tree. +- `SandboxProbe.available()` runs `docker info` with a short timeout. + +#### RegressionRunner + +Select `min(max_candidates, top_k_hotspots ∪ sev5_reachable_inventory)`. Render a recipe snippet into `scratch_files`. Safety check rejects snippets that mention `socket`, `curl`, `docker.sock`, or writes under `/workspace`. + +Verdicts: + +| Observation | Verdict | +|---|---| +| recipe or sandbox missing, timeout, safety reject | `inconclusive` | +| existing project test covering the hotspot already passes in sandbox | `already_covered` | +| candidate exits non-zero with a crash, assertion, or sanitizer abort | `triggerable` | +| candidate exits 0 | `not_triggerable` | + +`worth_fixing` ⇔ `triggerable` and related finding reachability is `reachable` or `inferred-file-surface`. + +`--fail-on` gains `worth-fixing`. Existing `findings` / `verified` unchanged. + +Recipes MVP: Python (`python /scratch/case.py` with `PYTHONPATH=/workspace`), JavaScript (`node /scratch/case.js`), C (`cc` + run) when the image contains a compiler. Java recipe may return `inconclusive` until added. + +## Data Models + +### Manifest additions + +```json +{ + "stages": { + "requested": ["static", "map", "regress"], + "completed": ["static", "map"], + "skipped": [{"stage": "regress", "reason": "sandbox_unavailable"}] + }, + "complexity": {"hotspot_count": 12, "heuristic_only": true}, + "worth_fixing": {"count": 0, "verdicts": []} +} +``` + +Degradations reuse the existing list: `{ "stage": "regress", "reason": "sandbox_unavailable" }`. + +### Consistency + +- Map artifact and ledger are JSON, resume-safe. +- Hunter suspicions never jump evidence levels in this spec. +- Stage 1 artifacts remain byte-compatible with current `findings.json` schema; new fields are additive. + +## Error Handling + +| Error | Response | +|---|---| +| `deep` requested, Docker missing | Complete 1–2, skip 3, degradation, exit 0 unless other fail-on hits | +| Model missing on `standard` | Heuristic map, no hunter findings, degradation | +| Path escape in a tool | Tool error to the model; do not throw the scan | +| Snippet fails safety check | `inconclusive`, do not run | +| Docker run timeout | kill container (`--rm`), `inconclusive` | +| Ledger would hide sev-5 reachable | Candidate still selected | + +## Testing Strategy + +### Unit + +- Mode→stage mapping and skip accounting. +- Path clamp rejects `../`. +- Map order on a fixture where regex hits a trivial helper and the real bug is a split-sink parser — parser ranks higher. +- Covered hotspot yields `gap=covered` and no test kind. +- Safety check rejects docker.sock and network snippets. +- Ledger boost/demote; sev-5 reachable remains a candidate. + +### Integration + +- Fake sandbox: `triggerable` / `not_triggerable` / timeout mapped to verdicts. +- Tool hunter with a scripted chat client that must call `grep_repo` before concluding on a split-sink file; fail if it only returns JSON with no tool call. +- `--fail-on worth-fixing` exits non-zero only when a `worth_fixing` verdict exists. + +### Evaluation / CI + +- Stage 1: existing `python -m openultrasast.gate` on smoke corpora. +- Stage 2: split-sink manifests; true vulnerable function in the high band; map order ≠ pattern-hit order. +- Stage 3: `@pytest.mark.docker`; skip if probe fails; at least one planted split-sink that stage 1 missed is `triggerable`. +- PR workflow stays `quick`. Nightly/deep is a separate job with Docker. + +### Anti-patterns this spec forbids in tests + +- One-line `eval(request.args)` as the only stage-2 proof. +- Asserting hunter findings equal regex findings. +- Calling the real Docker daemon in the default `pytest` job. + +## Security Considerations + +- Scanned code is untrusted. Stage 1–2 read it as text. Stage 3 executes only in Docker with network off, source read-only, no docker.sock, dropped caps, no-new-privileges, non-root user. +- Hunter tools cannot read outside the repo. +- Regression snippets are data, not host shell. They go to `/scratch` in the container. +- Secrets redaction already applies to traces and markdown; keep it on for map rationales that quote source. + +## Performance & Scalability + +- Map is O(files) plus one pass of findings. No extra model required. +- Tool hunter runs only on `max_hunter_hotspots` (default 8). +- Stage 3 `max_candidates` default 5; timeout from `SandboxConfig.timeout_seconds`. +- Large-repo CI: PR = stage 1; nightly = stage 2 always, stage 3 if probe passes. + +## Migration Strategy + +```mermaid +flowchart LR + P0[StagePlan plus manifest] --> P1[Heuristic map and hints] + P1 --> P2[Split-sink gate] + P2 --> P3[Tool hunter] + P3 --> P4[Docker runner plus recipes] + P4 --> P5[Ledger and fail-on worth-fixing] +``` + +| Phase | Ships | Merge gate | Rollback | +|---|---|---|---| +| 0 StagePlan | `deep` no longer exits; skips 2/3 with degradations | existing tests | revert CLI | +| 1 Heuristic map | `complexity_map.json` on `standard` | unit map tests | feature: empty map writer | +| 2 Split-sink gate | new fixtures; map ≠ hit-count | new gate job | gate off | +| 3 Tool hunter | OpenRouter loop, suspicion | scripted-client test | model unset | +| 4 Sandbox | Docker CLI + Python/JS recipes | docker-marked tests skip-ok | probe skip | +| 5 Worth-fixing | ledger + fail-on | integration on fake runner | flag unused | + +`deep` changing from hard-exit to skip-or-run is intentional and documented. No artifact schema is removed. diff --git a/.kiro/specs/three-stage-scan/requirements.md b/.kiro/specs/three-stage-scan/requirements.md new file mode 100644 index 0000000..67708a7 --- /dev/null +++ b/.kiro/specs/three-stage-scan/requirements.md @@ -0,0 +1,129 @@ +# Requirements Document + +## Introduction + +OpenUltraSAST today answers one question well: which regexes fired. It does not answer the three questions a rugged DevOps CI workflow actually asks: what exists in this repository, where problems are most likely and how to aim tests, and whether a finding is worth fixing. This feature turns the scan into those three stages. Stage 1 is a static inventory of what exists. Stage 2 is a complexity map of likely problem areas plus test-tuning hints. Stage 3 is isolated regression that proves triggerability before CI treats a finding as blocking. + +The work is a follow-up to the HarnessX ruleset-governance spec, not a replacement of it. CWE policy, project score, and rule-status shadowing stay in place. This spec owns the stage contract, the complexity map, the isolated regression verdict, the tool-using hunter on stages 2–3, and an evaluation corpus that is not a cheat sheet. + +## Boundary Context + +- **In scope**: a three-stage scan contract mapped onto existing scan depths; a static inventory that reports what exists without claiming verification; a complexity map of hotspots with test-tuning hints; isolated regression that produces a worth-fixing verdict; a tool-using hunter confined to stages 2–3; evaluation against split-sink and mutation corpora; visible degradation when the sandbox or a model is unavailable; evolutionary feedback from regression verdicts back into the complexity map. +- **Out of scope**: rewriting CWE-to-severity policy or the project-score formula; authoring or rewriting detection pattern text; full fuzzing campaigns; generating or applying patches; network pentesting; importing Clearwing. +- **Adjacent expectations**: stage 1 continues to run in the zero-dependency default install with no model and no sandbox. CWE policy, scoring, traces, and SARIF remain owned by the existing governance spec. The operator supplies an isolated-sandbox runtime (Docker CLI) for stage 3 and an LLM credential for the tool-using hunter; neither is a packaged library dependency of the core install. + +## Requirements + +### Requirement 1: Three-Stage Scan Contract + +**Objective:** As a DevOps engineer, I want a scan to run as three ordered stages that map onto the existing scan depths, so that CI can choose inventory-only, map-and-tune, or prove-worth-fixing without a new command family. + +#### Acceptance Criteria + +1. When a scan runs at depth `quick`, the Scan Orchestrator shall execute stage 1 only and shall not start a model call or execute target-derived code. +2. When a scan runs at depth `standard`, the Scan Orchestrator shall execute stage 1 then stage 2. +3. When a scan runs at depth `deep`, the Scan Orchestrator shall execute stage 1 then stage 2 then stage 3. +4. The Scan Orchestrator shall record the stages requested, the stages completed, and any skipped stage with a reason in the scan manifest. +5. If a later stage cannot run, the Scan Orchestrator shall complete earlier stages, record a degradation, and shall not discard those earlier artifacts. +6. The Scan Orchestrator shall keep the existing scan command and depth names so existing CI invocations continue to work. + +### Requirement 2: Stage 1 Static Inventory + +**Objective:** As a security reviewer, I want a static inventory of what exists in the repository, so that I can see files, surfaces, and pattern hits without treating them as verified vulnerabilities. + +#### Acceptance Criteria + +1. When stage 1 completes, the Static Inventory shall emit a repository snapshot, file targets with language and surface tags, entry-point reachability hints, ranked files, and static findings from the existing pattern ruleset and any provided static-analyzer input. +2. The Static Inventory shall label static pattern hits as inventory, not as verified findings, in the report and in the finding evidence level. +3. While no static-analyzer input is provided, the Static Inventory shall still complete using the built-in pattern ruleset and heuristics. +4. If an enabled rule's CWE does not resolve in policy, the Scan Runtime shall fail loud before producing inventory findings (existing startup invariant, preserved). +5. The Static Inventory shall not execute target-derived code and shall not require a model. + +### Requirement 3: Stage 2 Complexity Map + +**Objective:** As an AppSec engineer, I want a complexity map of where problems are most likely, so that review and testing attention go to real hotspots instead of every regex hit. + +#### Acceptance Criteria + +1. When stage 2 runs, the Complexity Map shall produce an ordered list of hotspots, each identifying a file and, when known, a function, with a complexity score and a rationale built from size, nesting, surface tags, reachability, existing inventory-hit density, and test-presence gaps. +2. The Complexity Map shall rank hotspots independently of a single regex match: a file with no pattern hit may still be a hotspot, and a pattern hit in a trivial helper may rank below a complex reachable parser. +3. When stage 2 completes, the Complexity Map shall write a map artifact that the report and later stages consume, including the scoring signals used for each hotspot. +4. While a hunter model is unavailable, the Complexity Map shall still produce the heuristic map and shall record that the hunter did not refine it. +5. The Complexity Map shall not mark a hotspot as verified or worth fixing. + +### Requirement 4: Test-Tuning Hints + +**Objective:** As a developer owning CI tests, I want the complexity map to tell me how to tune tests, so that I add coverage where it reduces real risk instead of mirroring cheat-sheet sinks. + +#### Acceptance Criteria + +1. When the Complexity Map emits a hotspot, the Test Tuning Advisor shall attach a test-gap hint that names the missing coverage (no adjacent test file, no test referencing the function, or no sanitizer/fuzz entry for a fuzzable surface). +2. The Test Tuning Advisor shall recommend a test kind from a closed set: unit, property, sanitizer, http-contract, or fuzz-harness, with a one-line reason tied to the hotspot's signals. +3. The Test Tuning Advisor shall not write test files into the scanned repository. +4. When a hotspot already has covering tests, the Test Tuning Advisor shall say so and shall not recommend a duplicate test kind. + +### Requirement 5: Stage 3 Isolated Regression and Worth-Fixing + +**Objective:** As a CI owner, I want isolated regression that answers whether a finding is worth fixing, so that the build fails only on triggerable, reachable issues. + +#### Acceptance Criteria + +1. When stage 3 runs, the Regression Runner shall select a bounded set of top hotspots and/or inventory findings and shall attempt a minimal regression or proof-of-concept inside an isolated analysis sandbox. +2. The isolated analysis sandbox shall run with network disabled, the source tree mounted read-only, a writable scratch area, and configured memory, process, and timeout limits. +3. The Regression Runner shall emit one of four verdicts per candidate: `triggerable`, `not_triggerable`, `already_covered`, or `inconclusive`. +4. When a candidate is `triggerable` and the related finding is reachable, the Worth-Fixing Gate shall mark it `worth_fixing`. +5. When a candidate is `not_triggerable` or `already_covered`, the Worth-Fixing Gate shall mark it `not_worth_fixing` and shall not fail the build for that candidate. +6. When the verdict is `inconclusive`, the Worth-Fixing Gate shall not fail the build for that candidate and shall record why (timeout, sandbox unavailable, no candidate generated, or setup failed). +7. Where `--fail-on worth-fixing` is selected, the Scan Orchestrator shall fail the scan if and only if at least one `worth_fixing` verdict was produced. +8. The Regression Runner shall never execute target-derived code on the host outside the isolated sandbox. + +### Requirement 6: Tool-Using Hunter on Stages 2 and 3 + +**Objective:** As an OpenUltraSAST maintainer, I want the hunter on later stages to use repository tools instead of dumping a file into a prompt, so that it can follow dataflow across lines and files the way a real review does. + +#### Acceptance Criteria + +1. Where a hunter model is configured, the Hunter Pool shall run a bounded tool-using loop against stage-2 hotspots, with tools limited to reading a file inside the repository, searching repository text, and listing callers or references already known to the scan. +2. The Hunter Pool shall confine every tool path to the scanned repository root and shall reject path arguments that escape that root. +3. When the hunter reports a candidate issue, the Scan Runtime shall record it at evidence level `suspicion` until independent static or regression evidence raises it. +4. While no hunter model is configured, the Hunter Pool shall skip the tool loop, leave the heuristic complexity map in place, and record a degradation. +5. The Hunter Pool shall not treat a model JSON list as static corroboration. +6. When stage 3 runs and a hunter model is configured, the Hunter Pool may propose a regression snippet, but the Regression Runner shall execute that snippet only inside the isolated sandbox after a structural safety check (no network, no writes outside scratch). + +### Requirement 7: Evolutionary Feedback from Regression to Map + +**Objective:** As a maintainer, I want regression verdicts to retune the complexity map, so that the scan gets better at spending attention where issues actually trigger. + +#### Acceptance Criteria + +1. When a hotspot's regression is `triggerable`, the Complexity Map shall raise that hotspot's subsequent score on the same repository. +2. When a hotspot's regression is `not_triggerable`, the Complexity Map shall lower that hotspot's subsequent score without deleting the underlying inventory hit. +3. The Scan Runtime shall persist those adjustments in a loop-owned ledger keyed by repository and hotspot identity, applied at map load the same way rule status overlays apply at ruleset load. +4. If a ledger entry would hide a reachable severity-5 inventory hit from the worth-fixing set, the Worth-Fixing Gate shall still consider that hit. +5. The evolutionary loop shall not edit detection pattern text and shall not edit authoritative CWE severity. + +### Requirement 8: Grounded Evaluation, Not Cheat Sheets + +**Objective:** As a maintainer, I want stage 2 and stage 3 quality gates to use split-sink and mutation cases, so that a regex that only matches textbook one-liners cannot claim success. + +#### Acceptance Criteria + +1. The Benchmark Harness shall include at least one split-sink case per in-scope language family (Python, JavaScript, Java, C/C++) where the source and the sink are on different lines or the payload is assigned before use. +2. The Benchmark Harness shall include at least one mutation of a known sink (equivalent API, intermediate variable, or wrapper) that a same-line pattern does not match. +3. When stage 1 is scored alone, the Benchmark Harness may use existing smoke fixtures, and shall not treat stage-1 recall as proof of stage-2 or stage-3 quality. +4. When stage 2 is scored, the Benchmark Harness shall require the complexity map to rank the true vulnerable function in the top band of hotspots on the split-sink corpus, even when stage 1 missed the sink. +5. When stage 3 is scored and the sandbox is available, the Benchmark Harness shall require a `triggerable` verdict on at least one planted split-sink that stage 1 missed. +6. The CI merge gate for this feature shall fail if stage 2 ranking on the split-sink corpus is no better than ranking files by pattern-hit count alone. + +### Requirement 9: CI Companion Budgets and Degradation + +**Objective:** As a CI owner, I want stage costs and failures to be bounded and visible, so that pull-request jobs stay cheap and nightly jobs can run the later stages without silent fallback. + +#### Acceptance Criteria + +1. While running in pull-request depth `quick`, the CI Pipeline shall complete stage 1 without a sandbox runtime and without a model. +2. Where a nightly or labeled job requests `standard` or `deep`, the CI Pipeline shall run those stages under the configured time, finding-count, and cost budgets. +3. If the sandbox runtime is missing when stage 3 is requested, the Scan Orchestrator shall skip stage 3, record `sandbox_unavailable`, and complete stages 1–2. +4. If a model is missing when stage 2 hunter refinement is requested, the Scan Orchestrator shall keep the heuristic map, record `hunter_model_unavailable`, and shall not invent hunter findings. +5. The Scan Orchestrator shall bound stage 3 to a configured maximum number of regression candidates per scan. +6. When any stage is skipped or truncated, the report shall show the degradation in the same manifest list used by existing capability fallbacks. diff --git a/.kiro/specs/three-stage-scan/research.md b/.kiro/specs/three-stage-scan/research.md new file mode 100644 index 0000000..1e4676d --- /dev/null +++ b/.kiro/specs/three-stage-scan/research.md @@ -0,0 +1,156 @@ +# Research & Design Decisions + +--- +**Purpose**: Capture discovery findings and architectural investigation behind the `three-stage-scan` spec. `design.md` carries the committed architecture; this file carries evidence and rejected alternatives. +--- + +## Summary +- **Feature**: `three-stage-scan` +- **Discovery Scope**: Extension of the existing scan harness (brownfield), with one new runtime prerequisite (Docker CLI for stage 3) and one new evaluation contract (split-sink / mutation corpora). +- **Key Findings**: + - Stage 1 already exists as `quick` mode: preprocess, mapping, rank, `quick_scan_findings`, structural verify, score, SARIF. Do not rebuild it. + - Stage 2 is not a hunter dump and is not `RankingScore`. Ranking is per-file attention. A complexity map is per-file-and-function hotspot attention plus test-gap hints. Existing signals (tags, loc, reachability hints, finding density, `has_fuzz_entry_point`) are enough for a heuristic map; the tool-using hunter refines it. + - Stage 3 has config and a threat-model table, but `cli.py` exits on `--mode deep`. No container is ever started. The first execution of untrusted code must be Docker CLI with the documented isolation flags, not a Python SDK and not host `subprocess` of target code. + - The previous HarnessX spec locked LLM use behind an optional extra that CI does not install. This spec uses the existing stdlib OpenRouter client so stage 2 can be evolutionary without that extra. HarnessX remains an optional runtime, not a prerequisite. + - Cheat-sheet fixtures cannot gate stages 2–3. The Python fixture already documents a split-sink miss (`query` built on the prior line). That class of case becomes the quality bar. + +## Research Log + +### Existing scan pipeline and mode mapping +- **Context**: Whether to add `--stage` or reuse `--mode quick|standard|deep`. +- **Sources Consulted**: `src/openultrasast/cli.py` (`_run_scan`, mode choices), `README.md` maturity legend, `.kiro/specs/openrouter-sast-harness/requirements.md` (hunter pool, docker sandbox, dynamic analysis), `.kiro/specs/harnessx-self-improving-rulesets/design.md`. +- **Findings**: + - `quick` is implemented and deterministic. `standard` schedules a hunter pool that calls `quick_scan_findings` unless HarnessX extra + hunter model are set. `deep` raises `SystemExit`. + - OpenRouter spec already defined: quick = static + bounded LLM review; standard = sandboxed tool-calling hunters; deep = harness/fuzz/dynamic. Implementation never reached that. + - Existing CI and README teach `--mode`. A parallel `--stage` flag would fork the contract. +- **Implications**: Map modes onto stages 1 / 1+2 / 1+2+3. Record `stages_requested` and `stages_completed` in the manifest so the three questions are explicit even if the flag names stay. + +### Ranking vs complexity map +- **Context**: `rank.py` already scores files. Is stage 2 just ranking? +- **Sources Consulted**: `src/openultrasast/rank.py` (`heuristic_rank`, `composite_priority` = surface 0.5 + influence 0.2 + reachability 0.3), `preprocess.py` (`FileTarget` tags, loc, `has_fuzz_entry_point`), `mapping.py` (`analyze_entry_points`). +- **Findings**: + - Ranking is file-level and exists to budget the hunter. It does not name functions, test gaps, or "tune your tests." + - Tags already encode likely problem classes (`memory_unsafe`, `parser`, `network_entry`, `syscall_entry`). Loc ≥ 500 boosts influence. No nesting/branch signal. No test-file presence. + - A hotspot that stage 1 missed (split-sink) can still rank high if the file is a reachable parser. That is the point of stage 2. +- **Implications**: Keep `RankingScore`. Add `ComplexityMap` / `Hotspot` with function identity when recoverable, test-gap hints, and a ledger overlay. Ranking still feeds hunter budgets; the map feeds humans and stage 3. + +### Hunter: regex, HarnessX dump, or tools +- **Context**: Previous spec adopted HarnessX "at the hunter loop" but the default hunter is regex and the HarnessX path dumps source into a prompt. +- **Sources Consulted**: `hunter.py` (`_run_hunter_task` → `quick_scan_findings`), `hunter_harness.py` (`_describe`, `_extract_findings`), `provider/openrouter.py` (`OpenRouterChatClient.complete_json`, stdlib `urllib`), `harness_ext.py` (optional extra). +- **Findings**: + - OpenRouter client is already zero-extra (stdlib). It is used for optional ranking, not for hunting. + - HarnessX extra is SHA-pinned, heavy, absent in CI. Requiring it repeats the last spec's failure mode. + - Clearwing hunters use `read_source_file`, `grep_source`, `find_callers` with path clamping. That is the evolutionary piece, not MetaAgent regex-status flips. +- **Implications**: Stage 2/3 hunter is a bounded tool loop over the OpenRouter client (or HarnessX if already wired). Tools are stdlib (read, Python grep, caller search from mapping + text). Path clamp at repo root. Hunter output evidence = `suspicion`. + +### Docker sandbox: CLI vs SDK vs skip +- **Context**: Threat model already lists sandbox limits. Need a runner that does not invert zero-dep posture. +- **Sources Consulted**: `docs/threat-model.md` (sandbox table), `config.py` `SandboxConfig`, Clearwing `hunter_sandbox.py` (image build + spawn, Docker SDK), Docker isolation guidance (`docker run --network none --read-only --tmpfs --memory --pids-limit --cap-drop ALL --security-opt no-new-privileges`), initrunner docker-sandbox docs (CLI `docker run --rm --init`, optional `runsc`). +- **Findings**: + - Python `docker` SDK would be a new core or extra dependency. The operator already has the Docker CLI if they can run containers. + - Subprocess `docker run --rm --init` with bind-mount of the repo read-only and a tmpfs/scratch write area matches the threat model and keeps `dependencies = []`. + - Unit tests must not require a daemon: inject a `SandboxRunner` protocol, ship a fake. Integration tests skip when `docker info` fails. + - Full sanitizer-image builds (Clearwing) are out of scope for this spec. Stage 3 runs language-specific commands on a small pinned image per language family, or `inconclusive` if the recipe is missing. +- **Implications**: `SandboxProbe` + `DockerCliRunner`. No SDK. gVisor/`runsc` is optional `security-opt` when `docker info` reports it; not required. + +### Evaluation corpora +- **Context**: User rejected cheat-sheet fixtures. Current gate is ≥90% recall on those fixtures. +- **Sources Consulted**: `benchmarks/fixtures/python-vulnerable/app.py` (line 77 split-sink comment), `javascript-vulnerable/src/app.js` (prototype pollution comment), `tests/test_detection_benchmarks.py`, `gate.py`. +- **Findings**: + - Stage 1 smoke fixtures are still useful as unit tests. They must not be the stage 2/3 merge gate. + - A split-sink corpus is small and owned: source and sink on different lines; mutations through wrappers/variables. Success = map ranks the true function in the top band; stage 3 triggers at least one stage-1 miss when Docker is present. + - Comparing map ranking to "sort by pattern-hit count" is an anti-cheat-sheet test: if regex density is the map, the gate fails. +- **Implications**: New `benchmarks/fixtures/split-sink-*` and a stage-2 ranking gate. Keep existing `openultrasast.gate` for stage 1 only. + +### Seams that already exist and must not be confused +- **Context**: Brownfield traps for a spec named "map" that also adds Docker. +- **Sources Consulted**: `skills.py` (`STAGES = ("map", "hunt", "verify", "fix")`), `mcp.py:68` (rejects `deep`), `reports.py:115` (`scan_exit_code`), `mapping.py` (no Java/Groovy AST), `config.py` (`SandboxConfig` parsed, never consumed by `_run_scan`), `rank.py` (`rank_targets_with_model` unused by CLI). +- **Findings**: + - Trail of Bits skill routing already uses a stage named `map` (mapping-discipline skills: Semgrep/CodeQL/sharp-edge). That is not the complexity map. `stages.py` must not reuse `skills.STAGES`. + - MCP scan tools reject `deep` and forbid docker/shell. This spec does not widen the MCP surface; deep is CLI/CI only. + - `--fail-on` is implemented in `scan_exit_code`, not only argparse. `worth-fixing` must be added there. + - Java/Groovy entry points are tag-only (`line=None` → `inferred-file-surface`). Hotspot `function_name` will often be empty on Java; the map must still rank the file. + - `[sandbox]`, `[dynamic]`, `[evidence]`, and `[models].ranker` are loaded into resolved config and then ignored. Stage 3 consumes `SandboxConfig`; it does not invent a parallel sandbox TOML. +- **Implications**: Naming, MCP, fail-on, and Java identity are integration constraints, not new products. + +### Evidence ladder honesty +- **Context**: HarnessX hunter stamped LLM JSON as `static_corroboration`. Original spec forbids reporting LLM-only claims as verified. +- **Sources Consulted**: `verification.py` (`verify_finding` accepts when reachability is not `unknown`), `hunter_harness.py` evidence_level assignment, openrouter requirements Evidence Ladder / Verification. +- **Findings**: + - Pattern hits as `static_corroboration` is consistent with the ladder. LLM hits as `suspicion` is the missing rule. + - `--fail-on verified` today means reachable + static corroboration. That remains the cheap PR gate. `--fail-on worth-fixing` is the rugged gate. +- **Implications**: Do not change CWE policy. Change hunter evidence. Add fail-on value. Report sections: Inventory, Complexity map, Worth fixing. + +## Architecture Pattern Evaluation + +| Option | Description | Strengths | Risks / Limitations | Notes | +|--------|-------------|-----------|---------------------|-------| +| Map-then-prove on existing modes | `quick`/`standard`/`deep` = stages 1 / 1+2 / 1+2+3 | Smallest CLI blast radius; matches threat-model "deep executes in sandbox"; reuses preprocess/rank/findings | Two vocabularies (mode vs stage) until the manifest lists both | **Selected** | +| New `--stage` command family | Parallel CLI, leave modes as-is | Clearer naming | Forks CI; `deep` still a stub | Rejected | +| Clearwing-shaped full sourcehunt | Tool hunters + sanitizer images + PoC + patch | Highest detection efficiency | Fork in all but name; out of original non-goals | Rejected as this spec; remains the oracle | +| HarnessX-required agent loop | Stage 2/3 only run with the extra | Reuses last spec's seam | CI never installs it; last spec's failure | Rejected as a requirement; optional runtime only | + +## Design Decisions + +### Decision D1: Modes map to stages +- **Context**: Need a three-question scan without breaking CI that already passes `--mode`. +- **Selected Approach**: `quick`→stage 1, `standard`→1+2, `deep`→1+2+3. Manifest records both. +- **Rationale**: Evolutionary adoption of the CLI the project already shipped. +- **Trade-offs**: Operators must learn that `standard` now means "map" not "regex hunter pool." README must say so. + +### Decision D2: Docker CLI subprocess, no SDK +- **Context**: Zero-dep core vs isolated execution. +- **Selected Approach**: `docker` binary via `subprocess`; probe with `docker info`; fake runner in unit tests. +- **Rationale**: Operator runtime, not a Python dependency. Matches threat-model flags. +- **Trade-offs**: Windows/macOS Docker Desktop variance; integration tests skip without a daemon. + +### Decision D3: OpenRouter client for the tool hunter; HarnessX optional +- **Context**: Last spec hid the LLM behind an extra CI does not install. +- **Selected Approach**: Tool loop calls `OpenRouterChatClient` (or any `RankerClient`-shaped chat client) when `[models].hunter` is set. If HarnessX extra is present, `HxScanOrchestrator` may host the same tools; it is not required. +- **Rationale**: Evolutionary use of code already in tree. Stdlib HTTP. +- **Trade-offs**: Two possible runtimes until HarnessX is either used or ignored; tests mock the client. + +### Decision D4: Complexity map is a new artifact +- **Context**: Ranking already exists. +- **Selected Approach**: `ComplexityMap` with `Hotspot` records. Ranking remains the hunter budget. Map is what humans and stage 3 consume. +- **Rationale**: Different question (where to look / how to test vs which file to hunt first). + +### Decision D5: Worth-fixing is a gate, not an evidence level +- **Context**: Evidence ladder is already six rungs. Do not invent a seventh for CI. +- **Selected Approach**: Verdicts `triggerable|not_triggerable|already_covered|inconclusive` → `worth_fixing|not_worth_fixing`. Evidence levels unchanged. Hunter stays at `suspicion`. +- **Rationale**: CI owners need a fail-on that means "we proved it," without collapsing the ladder. + +### Decision D6: Split-sink / mutation gate for stages 2–3 +- **Context**: 90% recall on cheat sheets is circular. +- **Selected Approach**: New corpus + "beats pattern-hit ranking" test. Stage 1 smoke gate stays. +- **Rationale**: Directly answers the user's evaluation complaint. + +### Decision D7: Ledger overlays hotspot scores, not patterns +- **Context**: Evolutionary feedback without repeating "loop must not edit patterns." +- **Selected Approach**: `.openultrasast/calibration/complexity_ledger.json` keyed by `repo_fingerprint + path + function`. Triggerable boosts, not_triggerable demotes. Cannot hide reachable severity-5 inventory from worth-fixing consideration. +- **Rationale**: Same overlay pattern as `rule_policy.json`, different object. + +### Decision D8: Language recipes, not a universal executor +- **Context**: Running untrusted Python vs C vs Java is different. +- **Selected Approach**: Small recipe registry (Python and JavaScript required for MVP; C compile+run when a compiler exists in the image; Java `inconclusive` until a recipe ships). Missing recipe → `inconclusive`, not a crash. +- **Rationale**: Simplification. Do not build Clearwing's sanitizer image matrix in this spec. + +## Synthesis + +- **Generalization**: Inventory → attention → proof is one pipeline with a `StagePlan`. Map, hints, and regress are processors on that plan, not three products. +- **Build vs adopt**: Adopt Docker CLI and the in-tree OpenRouter client. Build the map, ledger, recipes, and split-sink corpus. Do not adopt Clearwing, docker SDK, or a cyclomatic-complexity library (nesting depth from indentation/braces is enough). +- **Simplification**: No `--stage` flag, no HarnessX requirement, no fuzz campaign, no patch oracle, no Python Docker extra. + +## Risks & Mitigations +- Docker-in-Docker / rootless CI — probe, skip stage 3, degrade; do not fail stage 1. +- Hunter cost on large repos — hotspot cap + existing `[harnessx] max_cost_usd` / hardening `max_findings` / new `max_regress_candidates`. +- Untrusted snippet execution — structural deny (network, writes outside scratch, curl, docker socket mount). Never mount `/var/run/docker.sock`. +- Stage 2 still heuristic without a model — allowed; gate is ranking quality on split-sink, which heuristics can pass if tags+reachability work. +- Breaking `deep` from "exits" to "runs" — intended; document in README. + +## References +- OpenUltraSAST threat model — `docs/threat-model.md` +- OpenRouter SAST harness requirements — Docker sandbox, hunter pool, evidence ladder +- HarnessX self-improving rulesets — adjacent governance; do not reopen +- Docker run isolation flags — `--network none`, `--read-only`, `--cap-drop ALL`, `--security-opt no-new-privileges` +- Clearwing sourcehunt hunter tools and sandbox — implementation oracle, not a dependency diff --git a/.kiro/specs/three-stage-scan/spec.json b/.kiro/specs/three-stage-scan/spec.json new file mode 100644 index 0000000..52ef1e5 --- /dev/null +++ b/.kiro/specs/three-stage-scan/spec.json @@ -0,0 +1,22 @@ +{ + "feature_name": "three-stage-scan", + "created_at": "2026-09-03T16:10:07Z", + "updated_at": "2026-09-03T16:47:00Z", + "language": "en", + "phase": "implementation", + "approvals": { + "requirements": { + "generated": true, + "approved": true + }, + "design": { + "generated": true, + "approved": true + }, + "tasks": { + "generated": true, + "approved": true + } + }, + "ready_for_implementation": true +}