mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-19 21:19:28 +00:00
docs(specs): kick off corpus-seeded-mechanisms and reachability-flow-model (unapproved)
Two engine specs drafted after the pair-corpus baseline, awaiting owner approval before implementation: - corpus-seeded-mechanisms: structural shapes derived from trusted pairs, export, variant search merged with overlay flows, leave-one-out measurement, and a closed `mechanisms` lever for the improve loop. - reachability-flow-model: call graph over FunctionIR, reachability from entry points to sinks, hunter client preferring HarnessX (task 4.1b). Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
co-authored by
Claude Fable 5.1
Claude Fable 5.1
parent
3eba6e10a0
commit
7d238f2400
@@ -0,0 +1,56 @@
|
||||
# Brief: reachability-flow-model
|
||||
|
||||
## Problem
|
||||
|
||||
The tool's stated principle is to steer effort where risk is actually reduced. Today it cannot, because it has no model of how data or control reaches a sink:
|
||||
|
||||
- "Reachable" means the finding's line sits inside a route or CLI handler (`mapping.analyze_entry_points`). There is no call graph. On real code, sinks live in helpers several calls below the handler, so nearly everything is `unknown`, and `worth_fixing` (triggerable plus reachable) rarely fires.
|
||||
- The overlay's taint is intra-file and one hop. A flow from a request parameter to a memcpy three functions away is invisible, and `unadjudicated` carries no partial path.
|
||||
- Every stage handoff discards information. LLM hunter findings are capped at suspicion and cannot become sandbox candidates (`cli.py:404`, `prove_filter.py`); prove sees only regex-proposed, overlay-promoted ids.
|
||||
- The hunter reads whole files and returns JSON. That is the most expensive and least verifiable use of a model.
|
||||
- The improve loop's only levers are rule status and policy constants. It cannot add a sink, a sanitizer, a nullable callee, or a transformer, which are the only data that move recall.
|
||||
|
||||
Codebase shape confirms it: detection logic is about 16 percent of the source; harness plumbing, governance, and ranking are the rest, built around an engine that cannot express a guard or a call edge.
|
||||
|
||||
## Desired Outcome
|
||||
|
||||
A path model: entry point, hops, sink, guards seen, summaries used. Reachability becomes graph distance with a concrete path. The hunter works on paths and asks one checkable question per hole the static engine cannot fill. Answers become facts, gated on the pair corpus, reused at zero token cost. The report's first section is fix points: the few places whose fix collapses the most reachable risk.
|
||||
|
||||
True concolic execution is out of scope. The sandbox is the concrete half; the hunter proposing an input for a path and the sandbox running it is the loop this spec designs for, without a solver.
|
||||
|
||||
## Approach
|
||||
|
||||
Build a name-resolved call graph from the `FunctionIR` records tree-sitter already produces. Compute per-function taint summaries with the existing intra-file engine and compose them along the graph to a bounded depth. Attach graph reachability to findings, rank, map, and worth-fixing. Give the hunter a `path_context` tool and let a hunter finding with a locatable sink become a sandbox candidate after the safety check. Add a fact oracle: closed question kinds the static engine asks a model about a callee, cached as candidate facts with provenance, promoted only through the improve loop's new `facts` lever under the per-profile holdout gate from `pair-corpus-honesty`. Compute fix points as dominators on the path graph and report risk removed per fix.
|
||||
|
||||
## Scope
|
||||
|
||||
- **In**: call graph, summaries, path records, reachability attachment, rank and map integration, hunter on paths, hunter-to-candidate path, fact oracle with cache, `facts` lever with validator, fix-point ranking and report section, measurement on split-sink, vibe-py, vfc slices, offline tests.
|
||||
- **Out**: guard dominance semantics (`guard-dominance-regime` consumes path records), CST walker fixes (`overlay-ir-completeness`), a whole-program points-to or symbolic engine, concolic execution, executing target code outside the sandbox, new dispositions, LLM as parser, any merge gate change.
|
||||
|
||||
## Boundary Candidates
|
||||
|
||||
- Call graph and summaries (semantic) vs entry points (mapping) vs rank and map consumers
|
||||
- Fact oracle (asks and caches) vs facts lever (accepts) vs fact loader (consumes)
|
||||
- Hunter path context vs regression candidate selection
|
||||
|
||||
## Out of Boundary
|
||||
|
||||
- Overlay dispositions and evidence rungs (`propose-adjudicate-prove`)
|
||||
- CWE policy, project score, rule status lever (`harnessx-self-improving-rulesets`)
|
||||
- Sandbox flags, recipes, verdict vocabulary (`three-stage-scan`)
|
||||
- Pair scorer and label schema (`pair-corpus-honesty`)
|
||||
|
||||
## Upstream / Downstream
|
||||
|
||||
- **Upstream**: `FileIR`/`FunctionIR`, `analyze_entry_points`, `analyze_ir`, `load_facts`, `Hotspot`, `run_tool_hunter`, `select_candidates`, `run_improvement`, pair holdout metrics per profile.
|
||||
- **Downstream**: `guard-dominance-regime` attaches guards to path records; later web-authorization detection is a path whose hops include no auth guard; reports and MCP read fix points.
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
|
||||
- **Extends**: `three-stage-scan` (hunter tools, candidate selection, worth-fixing input), `propose-adjudicate-prove` (facts consumed unchanged; one new fact source), `harnessx-self-improving-rulesets` (one new lever under the same gate).
|
||||
- **Depends**: `pair-corpus-honesty` for per-profile holdout metrics and the hunter scorer.
|
||||
- **Adjacent**: `overlay-ir-completeness` improves the records this spec walks; neither blocks the other.
|
||||
|
||||
## Constraints
|
||||
|
||||
Core `dependencies = []`. Graph depth and fan-out bounded by config. Missing edges yield `unknown`, never `safe`. Model answers never bypass the gate; hand-written facts always win over candidate facts. No target code runs outside the sandbox. Youden on non-local slices stays off the merge gate.
|
||||
@@ -0,0 +1,412 @@
|
||||
# Design Document
|
||||
|
||||
## Overview
|
||||
|
||||
**Purpose.** This feature gives OpenUltraSAST a path model: how an entry point reaches a sink, through which functions, with which summaries. It turns reachability into graph distance, puts the hunter on paths, lets model answers become gated facts, and ranks fix points by reachable risk removed.
|
||||
|
||||
**Users.** AppSec engineers read paths and fix points. DevOps gets worth-fixing on real reachability. Maintainers get an improve loop that can accept facts. The hunter gets a path and one question instead of a file.
|
||||
|
||||
**Impact.** New `semantic/callgraph.py`, `semantic/summaries.py`, `semantic/oracle.py`, `risk/fixpoints.py`. `mapping` attaches graph reachability. `rank` and `complexity` consume distance and path count. `tool_hunter` gains `path_context`. `regress/candidate.py` accepts hunter findings with a locatable sink. `improve` gains a `facts` lever. Reports gain a fix-first section. No disposition, gate, or sandbox flag changes.
|
||||
|
||||
### Goals
|
||||
|
||||
- Path records from entry to sink across files, bounded, over-approximate, never inferring safety.
|
||||
- Reachability, rank, map, and worth-fixing use graph distance.
|
||||
- Hunter on paths; hunter findings can reach the sandbox; evidence rises on checks.
|
||||
- Model answers become candidate facts, accepted only through the gated lever.
|
||||
- Fix points with risk removed, reported first.
|
||||
|
||||
### Non-Goals
|
||||
|
||||
- Guard semantics (next spec), CST fixes (adjacent spec), alias or symbolic analysis, concolic execution, LLM parsing, evidence stamp changes for existing sources, new gates.
|
||||
|
||||
## Boundary Commitments
|
||||
|
||||
### This Spec Owns
|
||||
|
||||
- Call graph, summaries, path records, and their artifacts.
|
||||
- Graph reachability attachment and the rank and map signal additions.
|
||||
- `path_context` hunter tool and hunter-candidate eligibility.
|
||||
- Fact oracle, candidate and accepted fact stores, loader precedence, `facts` lever and its validator rules.
|
||||
- Fix-point computation and report section.
|
||||
|
||||
### Out of Boundary
|
||||
|
||||
- `OverlayRecord` fields, dispositions, evidence rungs.
|
||||
- Rule status and policy levers, CWE policy, project score.
|
||||
- Sandbox runner, recipes, verdict vocabulary, safety check.
|
||||
- Pair scorer, catalog schema.
|
||||
|
||||
### Allowed Dependencies
|
||||
|
||||
- `FileIR`, `FunctionIR`, `CallSite` plus one additive `imports` field on `FileIR`.
|
||||
- `analyze_ir`, `load_facts`, `SemanticFacts`.
|
||||
- `EntryPointRecord`, `attach_reachability_hints`, `heuristic_rank`, `build_complexity_map`, `Hotspot`.
|
||||
- `run_tool_hunter`, `HunterTools`, `select_candidates`, `_hotspot_from_forced_finding`, safety check.
|
||||
- `run_improvement`, validator, journal, per-profile holdout metrics.
|
||||
- `OpenRouterChatClient` or HarnessX provider through existing seams; `redaction`.
|
||||
|
||||
### Revalidation Triggers
|
||||
|
||||
- Changing `FunctionIR` or `CallSite` fields beyond additive.
|
||||
- Any consumer treating `unknown` distance as unreachable or safe.
|
||||
- Loading candidate facts without acceptance.
|
||||
- Adding a question kind.
|
||||
- Fix points influencing any gate.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Existing Architecture Analysis
|
||||
|
||||
| Existing element | Today | After |
|
||||
|---|---|---|
|
||||
| `mapping.attach_reachability_hints` | line-in-handler hints | plus graph distance and witness path |
|
||||
| `rank.heuristic_rank` reachability term | tags and hints integer | graph distance when available |
|
||||
| `complexity.build_complexity_map` | size, nesting, tags, density, tests | plus path count, min distance |
|
||||
| `tool_hunter.run_tool_hunter` | read, grep, find_refs over a hotspot file | plus `path_context`; path-scoped prompt |
|
||||
| `regress.select_candidates` | hotspots with promoted inventory ids, sev-5 forced | plus hunter findings with locatable sink |
|
||||
| `improve` levers | rule, policy | plus facts |
|
||||
| `load_facts` | one directory | hand-written, then accepted candidate facts |
|
||||
| reports | inventory, map, worth-fixing | plus fix first |
|
||||
|
||||
### Architecture Pattern & Boundary Map
|
||||
|
||||
Selected: **graph over existing IR, summaries from existing taint, closed oracle behind the existing gate.**
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
IR[FileIR per file + imports] --> CG[call graph]
|
||||
EP[entry points] --> RE[reachability: distance, witness path]
|
||||
CG --> RE
|
||||
IR --> SUM[function summaries via analyze_ir]
|
||||
FACTS[hand facts + accepted facts] --> SUM
|
||||
SUM --> PATHS[composed path records]
|
||||
CG --> PATHS
|
||||
RE --> FIND[findings reachability]
|
||||
RE --> RANK[rank]
|
||||
PATHS --> MAP[complexity map signals]
|
||||
PATHS --> HUNT[hunter path_context]
|
||||
PATHS --> Q[open questions]
|
||||
Q --> ORACLE[fact oracle]
|
||||
ORACLE --> CAND[candidate_facts.toml]
|
||||
CAND --> LEVER[improve facts lever + gate]
|
||||
LEVER --> ACC[accepted_facts.toml]
|
||||
ACC --> FACTS
|
||||
HUNT --> CANDS[regress candidates]
|
||||
PATHS --> FIX[fix points]
|
||||
FIX --> REPORT[report: fix first]
|
||||
```
|
||||
|
||||
**Key decisions**
|
||||
|
||||
- Edges over-approximate; distance `unknown` is never safe.
|
||||
- Summaries are pure functions of IR and facts, cached by hash.
|
||||
- Evidence for hunter findings rises only on path agreement or sandbox trigger.
|
||||
- Candidate facts never load until accepted; hand facts win.
|
||||
- Fix points are reported, never gated.
|
||||
|
||||
### Technology Stack
|
||||
|
||||
| Layer | Choice / Version | Role | Notes |
|
||||
|-------|------------------|------|-------|
|
||||
| Graph | stdlib dicts, BFS | call graph, distance | no networkx |
|
||||
| Summaries | existing `analyze_ir` | per-function facts | cached by file hash |
|
||||
| Oracle | existing chat client seams | closed questions | optional, offline-skippable |
|
||||
| Stores | TOML in `.openultrasast/calibration/` | candidate and accepted facts | same dir as ledgers |
|
||||
| Reports | existing markdown, manifest, SARIF | fix first section | additive |
|
||||
|
||||
## File Structure Plan
|
||||
|
||||
```
|
||||
src/openultrasast/
|
||||
├── semantic/
|
||||
│ ├── ir.py # FileIR.imports: tuple[str, ...] = () (additive); cst/ast fill it
|
||||
│ ├── callgraph.py # build_call_graph(irs) -> CallGraph; resolve; reachability(entry_points, depth)
|
||||
│ ├── summaries.py # summarize(function, facts) -> Summary; compose(graph, summaries, depth) -> paths
|
||||
│ ├── paths.py # PathRecord, write_paths, load_paths
|
||||
│ ├── oracle.py # open_questions(paths, graph) ; ask(questions, client) ; candidate store io
|
||||
│ └── facts.py # load_facts(dirs...) precedence: hand > accepted
|
||||
├── risk/
|
||||
│ └── fixpoints.py # dominators over path graph; risk removed; artifact
|
||||
├── mapping.py # attach_graph_reachability(targets, graph, distances)
|
||||
├── rank.py # reachability from distance when present
|
||||
├── complexity/signals.py# path_count, min_distance signals
|
||||
├── tool_hunter.py # path_context tool; path-scoped prompt when paths exist
|
||||
├── regress/candidate.py # hunter findings with locatable sink -> forced hotspot after safety check
|
||||
├── improve/validator.py # FactEdit lever "facts"; closed kinds; source must be candidate cache or benchmark signal
|
||||
├── improve/evolve.py # facts lever in run_round; same gate + per-profile clause
|
||||
├── reports.py # fix-first section; manifest.paths, manifest.open_questions
|
||||
├── config.py # [reach] depth=4, max_fanout=32, oracle_enabled, max_questions
|
||||
└── cli.py # wiring in MAP; artifacts callgraph.json, paths.json, fixpoints.json
|
||||
tests/
|
||||
├── test_callgraph.py, test_summaries.py, test_paths.py, test_oracle.py, test_fixpoints.py
|
||||
├── test_tool_hunter.py (path_context), test_regress_candidates.py (hunter eligibility), test_improve.py (facts lever)
|
||||
└── test_map_gate.py (path assertion)
|
||||
```
|
||||
|
||||
### Modified Files
|
||||
|
||||
- `semantic/cst.py`, `semantic/ir.py` — collect import statements into `FileIR.imports`; no other IR change.
|
||||
- `mapping.py` — new function; `EntryPointRecord` unchanged.
|
||||
- `stage_processors.py` — MAP stage runs graph, summaries, paths before overlay; overlay unchanged.
|
||||
- `map_gate.py` — path assertion per language family.
|
||||
|
||||
## System Flows
|
||||
|
||||
### Path construction
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A[parse files -> FileIR] --> B[build_call_graph]
|
||||
B --> C[reachability from entry points, depth bound]
|
||||
A --> D[summarize each function]
|
||||
D --> E[compose along graph]
|
||||
E --> F[PathRecord list]
|
||||
F --> G[open questions at unresolved hops]
|
||||
```
|
||||
|
||||
Composition rule: for a caller hop with tainted argument i to callee c, if `summary(c).param_to_sink[i]` is non-empty, extend the path; if `summary(c).param_to_return[i]`, the call result is tainted in the caller; if c is unresolved or beyond depth, mark `incomplete_at` and stop. No demotion is ever produced by composition.
|
||||
|
||||
### Hunter on a path
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant H as tool hunter
|
||||
participant P as path_context
|
||||
participant S as sandbox
|
||||
H->>P: path id
|
||||
P-->>H: entry, hops, sink, summaries, guards, open questions
|
||||
H->>H: verdict + proposed input (suspicion)
|
||||
H->>S: snippet (after safety check, deep only)
|
||||
S-->>H: triggerable / not
|
||||
Note over H: evidence rises on trigger or on path agreement
|
||||
```
|
||||
|
||||
### Facts lever
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Q[open questions] --> O[oracle asks model]
|
||||
O --> C[candidate_facts.toml]
|
||||
C --> L[improve: FactEdit accept]
|
||||
L --> G{existing gate + per-profile holdout}
|
||||
G -->|pass| A[accepted_facts.toml]
|
||||
G -->|fail| R[revert byte for byte]
|
||||
A --> F[load_facts precedence: hand > accepted]
|
||||
```
|
||||
|
||||
## Requirements Traceability
|
||||
|
||||
| Requirement | Summary | Components | Flows |
|
||||
|-------------|---------|------------|-------|
|
||||
| 1.1–1.6 | call graph, unresolved, distance, over-approx, extra-free, artifact | CallGraph | path construction |
|
||||
| 2.1–2.5 | summaries, composition, depth, unresolved hop, path records | Summaries, Paths | path construction |
|
||||
| 3.1–3.6 | reachability, rank, map, worth-fixing, map gate | ReachabilityAttach, rank, complexity | — |
|
||||
| 4.1–4.5 | path_context, path prompt, candidate eligibility, evidence, budget | HunterPaths, candidate | hunter on a path |
|
||||
| 5.1–5.6 | question kinds, when asked, cache, precedence, no exec, no model | FactOracle, facts loader | facts lever |
|
||||
| 6.1–6.5 | facts lever, validator, gate, journal, hand facts win | FactsLever | facts lever |
|
||||
| 7.1–7.5 | fix points, risk removed, report, sev-5 invariant, artifact | FixPoints, reports | — |
|
||||
| 8.1–8.5 | split-sink path check, labeled-function distance, gates unchanged, offline | map_gate, pairs payload, tests | — |
|
||||
|
||||
## Components and Interfaces
|
||||
|
||||
| Component | Domain | Intent | Req | Dependencies | Contracts |
|
||||
|-----------|--------|--------|-----|--------------|-----------|
|
||||
| CallGraph | semantic | resolve calls, distances | 1 | FileIR P0, entry points P0 | Service, State |
|
||||
| Summaries | semantic | per-function taint facts, composition | 2 | analyze_ir P0, facts P0 | Service |
|
||||
| Paths | semantic | path records artifact | 2.5 | Summaries P0 | State |
|
||||
| ReachabilityAttach | mapping | distance on findings, rank, map | 3 | CallGraph P0 | Service |
|
||||
| HunterPaths | hunter | path_context, path prompt, eligibility | 4 | Paths P0, tool_hunter P0, candidate P0 | Service |
|
||||
| FactOracle | semantic | closed questions, candidate cache | 5 | chat client P1, redaction P0 | Batch, State |
|
||||
| FactsLever | improve | accept or reject candidate facts | 6 | validator P0, evolve P0, holdout metrics P0 | Batch |
|
||||
| FixPoints | risk | dominators, risk removed, report | 7 | Paths P0, policy P0 | Batch, State |
|
||||
|
||||
### Semantic
|
||||
|
||||
#### CallGraph
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class CallEdge:
|
||||
caller: str # "path::function"
|
||||
callee: str | None # resolved node id or None
|
||||
call_line: int
|
||||
resolution: str # exact | import | ambiguous | unresolved
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CallGraph:
|
||||
nodes: tuple[str, ...]
|
||||
edges: tuple[CallEdge, ...]
|
||||
unresolved: int
|
||||
|
||||
def build_call_graph(irs: Sequence[FileIR]) -> CallGraph: ...
|
||||
def reachability(graph: CallGraph, entry_nodes: Sequence[str], *, max_depth: int) -> dict[str, tuple[int, tuple[str, ...]]]: ...
|
||||
```
|
||||
|
||||
Resolution order: same file definition; definition in an imported module (by `FileIR.imports` and path stem); any definition with that trailing name (ambiguous, one edge each, capped by `max_fanout`); else unresolved. Method calls resolve by trailing name. C resolves by bare name across the tree.
|
||||
|
||||
Invariants: never drops a call; `unresolved` counted; distance absent means `unknown`.
|
||||
|
||||
#### Summaries and Paths
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class Summary:
|
||||
node: str
|
||||
param_to_sink: dict[int, tuple[tuple[str, int], ...]] # param index -> (sink id, line)
|
||||
param_to_return: tuple[int, ...]
|
||||
constant_params: tuple[int, ...]
|
||||
facts_version: str
|
||||
source_hash: str
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PathRecord:
|
||||
path_id: str
|
||||
entry: str
|
||||
hops: tuple[tuple[str, int], ...] # (node, call line)
|
||||
sink: str
|
||||
sink_line: int
|
||||
sink_node: str
|
||||
cwe: str
|
||||
incomplete_at: str | None
|
||||
summaries_used: tuple[str, ...]
|
||||
open_questions: tuple[str, ...]
|
||||
|
||||
def summarize(function: FunctionIR, node: str, facts: SemanticFacts) -> Summary: ...
|
||||
def compose(graph: CallGraph, summaries: Mapping[str, Summary], entry_nodes: Sequence[str], *, max_depth: int) -> tuple[PathRecord, ...]: ...
|
||||
```
|
||||
|
||||
Preconditions: summaries for every node in the graph, or `incomplete_at` on missing.
|
||||
Postconditions: no `PathRecord` implies safety; incomplete paths are kept.
|
||||
|
||||
### Mapping and ranking
|
||||
|
||||
#### ReachabilityAttach
|
||||
|
||||
`attach_graph_reachability(targets, graph, distances)` adds to each target's reachability hints an entry `{"kind": "graph", "function_name", "distance", "witness"}`. `findings._reachability_for_line` treats a graph hint whose function contains the line as `reachable`. Rank: `reachability = max(existing, 5 - min(distance, 4))`, rationale records `graph_distance`. Map signals: `path_count`, `min_distance`.
|
||||
|
||||
### Hunter
|
||||
|
||||
#### HunterPaths
|
||||
|
||||
Tool `path_context(path_id) -> {entry, hops, sink, sink_line, summaries, guards: [], open_questions}`. Guards list is empty until `guard-dominance-regime`. When paths exist for a hotspot, the prompt names one path and asks for: verdict, reasoning tied to hops, proposed input. Findings carry `path_id` in tags. Eligibility: a hunter finding whose `line` matches a `PathRecord.sink_line` or an overlay sink line becomes a forced hotspot via `_hotspot_from_forced_finding`, ranked after overlay promotions, subject to `max_candidates` and the safety check. Evidence stays `suspicion`; a sandbox trigger sets the verdict as today; a path agreement is recorded in `reachability_evidence`.
|
||||
|
||||
### Oracle and facts
|
||||
|
||||
#### FactOracle
|
||||
|
||||
```python
|
||||
QUESTION_KINDS = ("nullable_return", "bounded_argument", "transformer_set_get", "returns_parameter_taint", "sanitizes_parameter")
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Question:
|
||||
kind: str
|
||||
callee: str
|
||||
language: str
|
||||
context_hash: str
|
||||
context: str # redacted callee text or signature only
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CandidateFact:
|
||||
kind: str
|
||||
callee: str
|
||||
language: str
|
||||
answer: dict[str, object] # structured per kind
|
||||
model: str
|
||||
context_hash: str
|
||||
asked_at: str
|
||||
```
|
||||
|
||||
Batch contract: trigger after composition when a model is configured; input open questions capped by `max_questions`; output appended to `candidate_facts.toml`; idempotent by `(kind, callee, context_hash)`. Never executes code; context passes through `redaction`.
|
||||
|
||||
`load_facts(directory, accepted: Path | None)` merges accepted facts after hand-written facts; on id collision the hand-written row wins and a degradation is recorded.
|
||||
|
||||
Answer shapes: `nullable_return: {nullable: bool}`, `bounded_argument: {arg: int, bounded_by: int | "constant"}`, `transformer_set_get: {set: str, get: str, key_args: [int], value_arg: int}`, `returns_parameter_taint: {params: [int]}`, `sanitizes_parameter: {params: [int], cwe: str}`.
|
||||
|
||||
### Improve
|
||||
|
||||
#### FactsLever
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class FactEdit:
|
||||
lever: str = "facts"
|
||||
action: str # accept | retract
|
||||
fact_key: str # kind:callee:context_hash
|
||||
source: str # candidate_cache | benchmark_signal
|
||||
rationale: str = ""
|
||||
```
|
||||
|
||||
Validator: kind in `QUESTION_KINDS`; `fact_key` present in the candidate cache; `retract` only for accepted candidate facts, never hand-written; no pattern text. `run_round` proposes accepts for candidate facts that touch open questions on missed pairs; the existing gate and the per-profile holdout clause decide; revert restores `accepted_facts.toml` byte for byte; journal records before and after metrics.
|
||||
|
||||
### Risk
|
||||
|
||||
#### FixPoints
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class FixPoint:
|
||||
node: str
|
||||
kind: str # function | entry
|
||||
paths_collapsed: int
|
||||
risk_removed: float
|
||||
witness_paths: tuple[str, ...]
|
||||
|
||||
def compute_fix_points(paths: Sequence[PathRecord], severity_by_cwe: Mapping[str, int], evidence_weight: Mapping[str, float]) -> tuple[FixPoint, ...]: ...
|
||||
```
|
||||
|
||||
A node dominates a path if every route from that path's entry to its sink passes through the node. Risk removed sums `severity × weight(evidence)` over dominated paths; weight defaults: sandbox-proven 1.0, path-complete 0.6, incomplete 0.3. Sorted by risk removed. Reported in markdown and manifest; the worth-fixing candidate set is unchanged.
|
||||
|
||||
## Data Models
|
||||
|
||||
Artifacts under `.openultrasast/runs/<id>/`: `callgraph.json`, `paths.json`, `fixpoints.json`. Manifest additions:
|
||||
|
||||
```json
|
||||
{
|
||||
"reach": {"nodes": 412, "edges": 1180, "unresolved": 96, "max_depth": 4, "entry_nodes": 9},
|
||||
"paths": {"complete": 14, "incomplete": 31},
|
||||
"open_questions": 12,
|
||||
"fix_first": [{"node": "api/runs.py::create_run", "paths_collapsed": 6, "risk_removed": 24.0}]
|
||||
}
|
||||
```
|
||||
|
||||
Calibration stores: `.openultrasast/calibration/candidate_facts.toml`, `accepted_facts.toml`. Both `[[fact]]` tables with the `CandidateFact` fields plus `accepted_at` on the latter.
|
||||
|
||||
## Error Handling
|
||||
|
||||
| Condition | Response |
|
||||
|-----------|----------|
|
||||
| Extra absent, non-Python file | node exists with no calls; edges from it unresolved; `graph_unavailable` degradation per language |
|
||||
| Parse failed file | node absent; callers' edges unresolved |
|
||||
| Depth or fan-out exceeded | path `incomplete_at` with reason `depth` or `fanout` |
|
||||
| No entry points | distances empty; findings keep today's hints; fix points empty |
|
||||
| Model absent | open questions recorded; oracle skipped with degradation |
|
||||
| Model answer malformed | dropped; question stays open |
|
||||
| Accepted fact collides with hand fact | hand fact wins; degradation `fact_shadowed` |
|
||||
| Hunter snippet fails safety check | inconclusive as today |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- Unit: graph resolution exact, import, ambiguous, unresolved; distance and witness on a small tree; summary of a function with param-to-sink and param-to-return; composition across two files; depth stop is incomplete not safe; fix points on a diamond graph; oracle idempotency; validator rejects pattern text and hand-fact retraction; loader precedence.
|
||||
- Integration: split-sink Python fixture yields a complete path from the route to the planted sink; map gate path assertion passes; a scripted hunter using `path_context` produces a finding that becomes a forced candidate and runs under the fake sandbox; `run_improvement` with a scripted candidate fact accepts when holdout improves and reverts when a profile regresses.
|
||||
- Gates: detection gate, local pair gate, existing split-sink assertions unchanged; extra-free suite green with graph assertions skipping.
|
||||
- No default test needs a model, sandbox, or network.
|
||||
|
||||
## Security Considerations
|
||||
|
||||
Graph and summaries are static reads. Oracle context is redacted and limited to the callee signature and body; no secrets, no execution. Hunter snippets run only in the sandbox after the existing safety check. Candidate facts are data and cannot introduce pattern text.
|
||||
|
||||
## Performance & Scalability
|
||||
|
||||
Graph is O(functions + calls). Summaries cached by file hash. Composition bounded by depth 4 and fan-out 32 by default. Oracle bounded by `max_questions` per run. Fix points run over path records, not statements.
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
1. Graph and distances as artifacts only; nothing consumes them. Baseline recorded.
|
||||
2. Reachability attachment and rank term; verify split-sink map gate still passes and now asserts a path.
|
||||
3. Summaries and path records; pairs payload reports labeled-function distance.
|
||||
4. Hunter path context and candidate eligibility behind config; default on for standard when a model is set.
|
||||
5. Oracle and facts lever; default `oracle_enabled = false` until the first accepted facts are reviewed.
|
||||
6. Fix points and report section.
|
||||
|
||||
Rollback per phase is config off; artifacts are additive.
|
||||
@@ -0,0 +1,112 @@
|
||||
# Requirements Document
|
||||
|
||||
## Introduction
|
||||
|
||||
OpenUltraSAST cannot yet steer effort where risk is reduced because it has no model of how an entry point reaches a sink. Reachability today means "inside a handler," taint is one hop inside one file, the LLM hunter cannot create a sandbox candidate, and the improve loop cannot add the facts that would raise recall. This spec adds a call graph and composed taint summaries over the existing `FunctionIR` records, attaches path-based reachability to findings and ranking, puts the hunter on paths with one checkable question per gap, turns model answers into gated facts, and ranks fix points by reachable risk removed. Concolic execution is out of scope; the sandbox remains the only place target code runs.
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: call graph, taint summaries, path records, reachability attachment to findings, rank, complexity map and worth-fixing; hunter path context; hunter findings as sandbox candidates; fact oracle and candidate-fact cache; `facts` improve lever; fix-point ranking and report section; offline tests and measurement on existing slices.
|
||||
- **Out of scope**: guard semantics; CST walker fixes; whole-program alias analysis; symbolic or concolic execution; new dispositions or evidence rungs; LLM as parser; any merge gate change; profile-specific fact tables beyond the gate defined by `pair-corpus-honesty`.
|
||||
- **Adjacent expectations**: `FileIR` shape is frozen except additive fields; `load_facts` merges candidate facts below hand-written facts; the improve loop's existing accept gate and per-profile holdout clause apply unchanged to the new lever; sandbox safety checks apply to hunter snippets as today.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement 1: Call graph and graph reachability
|
||||
|
||||
**Objective:** As an AppSec engineer, I want to know whether an entry point actually reaches a function, so that a finding's reachability is a path and not a line-in-handler guess.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When a tree is mapped, the system shall build a call graph whose nodes are functions from `FileIR` records and whose edges are resolved call sites, across files, using name and import resolution.
|
||||
2. When a call site cannot be resolved, the system shall record it as an unresolved edge and shall not treat the callee as unreachable or safe.
|
||||
3. When entry points exist, the system shall compute for each function the shortest distance from any entry point and one witness path, bounded by a configured maximum depth.
|
||||
4. The system shall prefer over-approximation: an ambiguous name resolving to several definitions yields edges to each, marked ambiguous.
|
||||
5. If the semantic extra is absent, then the system shall build the graph for Python from the stdlib parse and shall mark other languages `graph_unavailable` without failing the scan.
|
||||
6. The system shall write the graph and distances as a run artifact without any target code execution.
|
||||
|
||||
### Requirement 2: Taint summaries composed along the graph
|
||||
|
||||
**Objective:** As an engineer, I want flows that cross function and file boundaries to appear as one path, so that a request parameter reaching a sink three calls down is visible.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When a function is analyzed, the system shall produce a summary stating which parameters reach which sinks, which parameters reach the return value, and which parameters are fully constant-dominated, using the existing intra-file taint engine.
|
||||
2. When a caller passes a tainted argument to a callee whose summary maps that parameter to a sink, the system shall record a path from the caller's source through the callee to the sink.
|
||||
3. The system shall compose summaries to a configured depth and shall stop with `flow_incomplete` beyond it, carrying the partial path in the record.
|
||||
4. When a hop's callee is unresolved, the system shall record the path as incomplete at that hop, never as safe.
|
||||
5. The system shall emit path records with entry, ordered hops, sink, sink line, and the summary used at each hop.
|
||||
|
||||
### Requirement 3: Reachability drives rank, map, and worth-fixing
|
||||
|
||||
**Objective:** As a DevOps engineer, I want ranking and worth-fixing to use real reachability, so that budget and CI failures go to reachable risk.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When a finding lies in a function with a finite graph distance from an entry point, the system shall mark it `reachable` and attach the distance and witness path to its reachability evidence.
|
||||
2. When a finding lies in a function with no path within the depth bound, the system shall mark it `unknown`, not `unreachable`.
|
||||
3. The rank formula's reachability term shall use graph distance when available, falling back to today's hints otherwise, with the change recorded in the rationale.
|
||||
4. The complexity map shall add path count and minimum distance as signals and shall not use them as the sole score.
|
||||
5. The worth-fixing gate shall accept graph-reachable findings as reachable; the existing `inferred-file-surface` case stays.
|
||||
6. The system shall keep the split-sink map gate passing and shall extend it with a path check defined in Requirement 8.
|
||||
|
||||
### Requirement 4: Hunter on paths
|
||||
|
||||
**Objective:** As an engineer, I want the hunter to work on a path and ask one checkable question, so that model spend goes to what static analysis cannot do.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When a hunter runs on a hotspot that has path records, the system shall provide a `path_context` tool returning the entry, hops, sink, summaries, and known guards for a chosen path.
|
||||
2. The hunter prompt shall ask for a verdict on a specific path and a proposed input, not a whole-file review, when path records exist.
|
||||
3. When a hunter finding names a sink line that the overlay or the graph can locate, the system shall make it eligible as a sandbox candidate after the existing safety check, without requiring an inventory promotion.
|
||||
4. The system shall keep hunter findings at `suspicion` until an independent check agrees: a sandbox trigger, a composed path to that sink, or an accepted fact; evidence rises on that check, not on origin.
|
||||
5. The hunter shall stay within the existing step and cost budgets; path context counts toward them.
|
||||
|
||||
### Requirement 5: Fact oracle with gated candidate facts
|
||||
|
||||
**Objective:** As a maintainer, I want the model to answer one question about a callee and have the answer reused as data, so that recall rises without hand-writing every fact and without paying tokens on every scan.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The system shall define a closed set of question kinds: `nullable_return`, `bounded_argument`, `transformer_set_get`, `returns_parameter_taint`, `sanitizes_parameter`.
|
||||
2. When the static engine stops at an unresolved or unsummarized callee on a path, the system shall record a question for that callee; questions shall be asked only when a hunter or verifier model is configured.
|
||||
3. When a model answers, the system shall store a candidate fact with the question, the answer, the model id, the source hash, and a timestamp, in a cache next to the calibration ledgers.
|
||||
4. The system shall not load candidate facts into adjudication unless they have been accepted by the improve loop's `facts` lever; hand-written facts always take precedence over candidate facts.
|
||||
5. The system shall not execute target code to answer a question and shall not send secrets; redaction applies to question context.
|
||||
6. If no model is configured, then the system shall record the open questions in the manifest and proceed.
|
||||
|
||||
### Requirement 6: Facts as an improve-loop lever
|
||||
|
||||
**Objective:** As a maintainer, I want the improve loop to be able to accept or reject facts, so that the evolutionary machinery can move detection and not only rule status.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The improve loop shall gain a `facts` lever whose edits add or remove a candidate fact from the accepted set.
|
||||
2. The validator shall accept only edits whose fact kind is in the closed set and whose source is the candidate cache or a benchmark signal; free-form pattern text is rejected.
|
||||
3. When a `facts` edit is evaluated, the existing accept gate and the per-profile holdout clause shall apply unchanged; rejection reverts byte for byte.
|
||||
4. The journal shall record each fact edit with its provenance and the metrics before and after.
|
||||
5. The system shall never remove or override a hand-written fact through this lever.
|
||||
|
||||
### Requirement 7: Fix points ranked by reachable risk removed
|
||||
|
||||
**Objective:** As an operator, I want to know which few fixes remove the most reachable risk, so that I fix the worst things first.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When path records exist, the system shall compute fix points as functions or guard sites that dominate multiple paths in the path graph.
|
||||
2. Each fix point shall carry an estimate of risk removed: the sum over dominated paths of policy severity weighted by evidence, and the number of paths collapsed.
|
||||
3. The report shall gain a "fix first" section listing fix points in order of risk removed, each with its witness paths.
|
||||
4. The system shall not let fix-point ranking hide a reachable severity-5 finding from the worth-fixing candidate set.
|
||||
5. The fix-point list shall be written as a run artifact and exposed to reports without executing target code.
|
||||
|
||||
### Requirement 8: Measurement and unchanged gates
|
||||
|
||||
**Objective:** As a maintainer, I want the path model measured on the corpora we have, so that the shift is grounded.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. On the split-sink fixtures, the system shall report whether a path exists from a planted entry point to each planted function, and the map gate shall assert at least one such path per language family.
|
||||
2. On slices whose labels name a function, the pair payload shall report the fraction of labeled functions that have a finite graph distance.
|
||||
3. The stage-1 detection gate, the local pair gate, and the existing split-sink gate assertions shall be unchanged.
|
||||
4. Default tests shall not require a model, a sandbox, or network; oracle and hunter tests use scripted clients.
|
||||
5. The extra-free suite shall stay green with graph-dependent assertions skipping when the extra is absent.
|
||||
@@ -0,0 +1,113 @@
|
||||
# Research & Design Decisions
|
||||
|
||||
## Summary
|
||||
|
||||
- **Feature**: `reachability-flow-model`
|
||||
- **Discovery Scope**: Complex Integration across semantic, mapping, rank, complexity, hunter, regress, improve
|
||||
- **Key Findings**:
|
||||
- `FunctionIR` already carries every call site per function for Python, JS, TS, C, and Java when the extra is present. A name-resolved call graph is a few hundred lines over data that exists.
|
||||
- Reachability today is "line inside an entry-point handler" (`mapping.attach_reachability_hints`); `rank` bumps an integer from tags and hints; `worth_fixing` needs `reachable` or `inferred-file-surface`. No call edge exists anywhere.
|
||||
- Sandbox candidates come only from hotspots filtered to overlay-promoted inventory ids (`cli.py:404`, `prove_filter.filter_promoted_hotspots`); a hunter snippet is used only for an already promoted hotspot (`candidate.py:199`). The LLM plane cannot create a candidate.
|
||||
- Evidence is stamped by origin: regex hit `static_corroboration` (`findings.py:110`), hunter `suspicion` (`tool_hunter.py:408`).
|
||||
- Improve levers are `rule` and `policy` only (`improve/validator.py`); facts are not a lever. `propose-adjudicate-prove` excluded auto-authored facts.
|
||||
- Code shape: detection logic 1,910 lines of 12,128; harness plumbing 2,885; governance and benchmarks 3,178.
|
||||
- Clearwing fills reachability from a tree-sitter callgraph and lets taint paths boost surface; the embeddings retrieve mechanisms, not decide taint. That split is the reference.
|
||||
- Literature: RealVuln shows LLM-based scanners at 51.7 to 73 F3 on real Python web code versus Semgrep at 17.7; CVE-Bench shows agents exploit at most 13 percent of real web CVEs. Recall on real code comes from models with tools; precision and reproducibility come from static checks and execution.
|
||||
|
||||
## Research Log
|
||||
|
||||
### Where a call graph can come from
|
||||
|
||||
- **Context**: Avoid a new parser or a whole-program engine.
|
||||
- **Sources Consulted**: `semantic/ir.py`, `semantic/cst.py`, `mapping.py`, `preprocess.py`, `hunter_tools.find_refs`.
|
||||
- **Findings**: `FileIR.functions[*].calls[*].name` is the callee text. Python stdlib parse yields the same records. Imports are not in the IR; the walker can add an `imports` tuple per file cheaply. `find_refs` is a word-boundary regex over files and is the only cross-file lookup today.
|
||||
- **Implications**: `semantic/callgraph.py` consumes `FileIR` plus an additive `imports` field. Resolution order: same-file definition, imported module definition, any definition with that name (ambiguous edges). Method calls resolve by trailing name. C resolves by bare name across the tree.
|
||||
|
||||
### Summaries versus whole-program taint
|
||||
|
||||
- **Context**: `propose-adjudicate-prove` deferred inter-file taint as too large and risky for false demotes.
|
||||
- **Sources Consulted**: `semantic/taint.py` (`_analyze_function` seeds parameters as tainted, tracks names, emits `TaintPath` and `Demotion`), overlay invariants.
|
||||
- **Findings**: Because parameters are already seeded tainted, a per-function run already answers "which parameter reaches which sink." Adding "which parameter reaches the return" is a small extension. Composition is then a graph walk with a depth bound. Nothing needs alias analysis to produce a useful path; anything unresolved stays `flow_incomplete`.
|
||||
- **Implications**: Summaries are a pure function of `FunctionIR` plus facts, cacheable by file hash. Composition never demotes; it only adds paths or marks incompleteness.
|
||||
|
||||
### Hunter to candidate
|
||||
|
||||
- **Context**: The LLM plane is the recall lever the field data supports, but it is walled off from prove.
|
||||
- **Sources Consulted**: `tool_hunter.py`, `regress/candidate.py`, `three-stage-scan` design (hunter snippets run only in the sandbox after safety check), `verification.py`.
|
||||
- **Findings**: The safety check and sandbox are origin-agnostic. The wall is `filter_promoted_hotspots` and the candidate selector's reliance on inventory ids. A hunter finding with a locatable sink line can be given a synthetic hotspot the same way `_hotspot_from_forced_finding` already does for severity-5 inventory.
|
||||
- **Implications**: Add eligibility for hunter findings that name a sink line resolvable to a path record or an overlay sink. Evidence stays suspicion until sandbox trigger or path agreement.
|
||||
|
||||
### Facts from a model, safely
|
||||
|
||||
- **Context**: The engine lacks callee knowledge: nullable returns (`tls1_lookup_sigalg`), bounded arguments, transformer set/get (`ConfigParser`), macro out-params (`n2s`).
|
||||
- **Sources Consulted**: `semantic/facts.py` loader, `improve/validator.py`, `improve/journal.py`, `calibration.py` ledger patterns, `redaction.py`.
|
||||
- **Findings**: Facts are versioned TOML loaded by `load_facts`. The loader can merge a second directory at lower precedence. The improve loop has a strict validator and byte-for-byte revert. Redaction exists for prompts.
|
||||
- **Implications**: Model answers are candidate facts in `.openultrasast/calibration/candidate_facts.toml`; accepted facts in `accepted_facts.toml`; loader precedence hand-written > accepted > none. The `facts` lever moves rows between candidate and accepted under the same gate. No pattern text is ever model-authored; question kinds are closed.
|
||||
|
||||
### Fix points
|
||||
|
||||
- **Context**: The principle is risk reduced per fix, not findings per file.
|
||||
- **Sources Consulted**: `complexity/map.py`, `complexity/ledger.py`, `regress/verdict.py`, `scoring/project_score.py`.
|
||||
- **Findings**: The path graph is small (functions, not statements). Dominators over a DAG of entry-to-sink paths are cheap. Severity is available per CWE from policy.
|
||||
- **Implications**: `risk/fixpoints.py` computes dominator sets per sink path and aggregates risk removed. It is a report and artifact, not a gate; it cannot hide severity-5 reachable findings.
|
||||
|
||||
## Architecture Pattern Evaluation
|
||||
|
||||
| Option | Description | Strengths | Risks / Limitations | Notes |
|
||||
|--------|-------------|-----------|---------------------|-------|
|
||||
| Joern CPG as required engine | Full inter-procedural dataflow | Real dataflow | Scala runtime, not zero-dep, slow on PRs | Rejected; stays optional sidecar |
|
||||
| CodeQL databases | Strong inter-procedural | Mature | Build step, license limits, heavy | Rejected as required; SARIF ingest remains |
|
||||
| Whole-program taint in Python | Own engine | Full control | Months of work, alias analysis, false demotes | Rejected |
|
||||
| Name-resolved call graph plus composed summaries | Graph over existing IR; summaries from existing taint | Days to build; over-approximate by design; feeds hunter and fix points | Ambiguous names, dynamic dispatch | **Selected** |
|
||||
| LLM as reachability oracle | Ask the model who calls what | No code | Unverifiable, expensive per scan | Rejected; model answers closed callee questions only |
|
||||
|
||||
## Design Decisions
|
||||
|
||||
### Decision: Over-approximate edges, never infer safety
|
||||
|
||||
- **Context**: Missing edges silently hide risk; false edges cost hunter budget.
|
||||
- **Selected Approach**: Ambiguous names produce edges to every definition, flagged. Unresolved calls produce an unresolved edge. Distance is infinite only within the depth bound and is reported as `unknown`.
|
||||
- **Rationale**: Consistent with the fail-closed rule in the overlay.
|
||||
- **Trade-offs**: More hunter work on large trees; bounded by existing hotspot caps.
|
||||
|
||||
### Decision: Summaries are pure and cached
|
||||
|
||||
- **Context**: Cost on large repos.
|
||||
- **Selected Approach**: Summary keyed by file hash plus facts version; recomputed only when either changes.
|
||||
- **Trade-offs**: Cache invalidation on facts version bump; acceptable.
|
||||
|
||||
### Decision: Evidence rises on checks, not on origin
|
||||
|
||||
- **Context**: Regex stamped corroboration, hunter stamped suspicion.
|
||||
- **Selected Approach**: This spec does not change existing stamps. It adds the checks that raise hunter evidence: composed path agreement or sandbox trigger. A follow-up may lower regex-only hits to inventory; recorded as a revalidation trigger for `propose-adjudicate-prove`.
|
||||
- **Rationale**: Keep this spec additive; changing existing stamps changes gate baselines.
|
||||
|
||||
### Decision: Closed question kinds, gated acceptance
|
||||
|
||||
- **Context**: Auto-facts were excluded for good reason: unbounded model-authored patterns.
|
||||
- **Selected Approach**: Five question kinds, structured answers, candidate cache with provenance, acceptance only via the `facts` lever under the per-profile holdout gate.
|
||||
- **Trade-offs**: Slower than trusting answers directly; the gate is the point.
|
||||
|
||||
### Decision: Fix points are dominators, reported not gated
|
||||
|
||||
- **Context**: Do not add a seventh evidence rung or a new CI condition.
|
||||
- **Selected Approach**: Compute and report; the severity-5 reachable invariant from `three-stage-scan` stays.
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
- Dynamic dispatch and callbacks in JS and Python produce unresolved edges — report unresolved counts; hunter `find_refs` remains available as a fallback.
|
||||
- Graph blow-up on large trees — depth and fan-out bounds in config; hotspot caps unchanged.
|
||||
- Model answers wrong — never used unless accepted by the gate; provenance recorded; hand facts win.
|
||||
- Hunter candidates increase sandbox load — existing `max_candidates` cap; hunter candidates ranked after overlay promotions unless a path agrees.
|
||||
- Fix-point estimates misread as guarantees — label as estimates; show witness paths.
|
||||
|
||||
## References
|
||||
|
||||
- Clearwing sourcehunt callgraph, taint, ranker (reference architecture; not a dependency)
|
||||
- [RealVuln](https://arxiv.org/abs/2604.13764) — rule-based versus LLM scanners on real Python web code
|
||||
- [CVE-Bench](https://arxiv.org/abs/2503.17332) — agent exploitation ceiling on real web CVEs
|
||||
- `.kiro/specs/three-stage-scan/design.md` — hunter tools, candidate selection, worth-fixing
|
||||
- `.kiro/specs/propose-adjudicate-prove/design.md` — facts as data, dispositions, evidence honesty
|
||||
- `.kiro/specs/harnessx-self-improving-rulesets/design.md` — bounded levers, accept gate, revert
|
||||
- `.kiro/specs/pair-corpus-honesty/` — per-profile holdout gate, hunter scorer
|
||||
- Memory note `overlay-gaps-diagnosis-2026-09`
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"feature_name": "reachability-flow-model",
|
||||
"created_at": "2026-09-04T16:27:33Z",
|
||||
"updated_at": "2026-09-04T16:27:33Z",
|
||||
"language": "en",
|
||||
"phase": "tasks-generated",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
"approved": false
|
||||
},
|
||||
"design": {
|
||||
"generated": true,
|
||||
"approved": false
|
||||
},
|
||||
"tasks": {
|
||||
"generated": true,
|
||||
"approved": false
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": false
|
||||
}
|
||||
@@ -0,0 +1,125 @@
|
||||
# Implementation Plan
|
||||
|
||||
- [ ] 1. Foundation: call graph and reachability
|
||||
- [ ] 1.1 Collect imports into FileIR and build a name-resolved call graph
|
||||
- `FileIR` gains an additive `imports` field filled by both the CST walker and the stdlib Python parse.
|
||||
- Graph nodes are functions; edges resolve same-file, imported-module, ambiguous (one edge per definition, capped), or unresolved; no call is dropped.
|
||||
- Observable: a three-file fixture yields exact, import, ambiguous, and unresolved edges with the expected counts; extra-free Python still builds a graph.
|
||||
- _Requirements: 1.1, 1.2, 1.4, 1.5_
|
||||
- _Boundary: CallGraph_
|
||||
|
||||
- [ ] 1.2 Compute bounded distance and witness paths from entry points
|
||||
- BFS from entry-point functions to a configured depth; each reached node gets distance and one witness path; unreached nodes have no distance and are reported as unknown.
|
||||
- Graph, distances, and unresolved count are written as a run artifact in MAP without executing target code.
|
||||
- Observable: on the split-sink Python fixture the planted function has a finite distance and a witness path through the route; `callgraph.json` exists after a standard scan.
|
||||
- _Requirements: 1.3, 1.6, 3.6_
|
||||
- _Boundary: CallGraph_
|
||||
- _Depends: 1.1_
|
||||
|
||||
- [ ] 2. Core: summaries and path records
|
||||
- [ ] 2.1 Summarize each function with the existing taint engine
|
||||
- Summary states parameter-to-sink, parameter-to-return, and constant-dominated parameters; keyed by source hash and facts version; cached.
|
||||
- Observable: a function that forwards its parameter into `execute` reports param 0 to that sink; a function returning its parameter reports param 0 in param_to_return; a second run reuses the cache.
|
||||
- _Requirements: 2.1_
|
||||
- _Boundary: Summaries_
|
||||
- _Depends: 1.1_
|
||||
|
||||
- [ ] 2.2 Compose summaries along the graph into path records
|
||||
- Tainted arguments extend a path into a callee whose summary reaches a sink; return taint flows back to the caller; unresolved callees or depth exhaustion mark the path incomplete at that hop; composition never demotes.
|
||||
- Path records carry entry, hops, sink, sink line, summaries used, and open questions; written as `paths.json`.
|
||||
- Observable: a route in one file calling a helper in another that executes a query yields one complete path; cutting the helper's definition yields an incomplete path, not silence.
|
||||
- _Requirements: 2.2, 2.3, 2.4, 2.5_
|
||||
- _Boundary: Summaries, Paths_
|
||||
- _Depends: 2.1, 1.2_
|
||||
|
||||
- [ ] 3. Integration: reachability consumers
|
||||
- [ ] 3.1 Attach graph reachability to findings, rank, and the complexity map
|
||||
- A finding inside a function with finite distance is reachable with distance and witness in its evidence; no distance stays unknown; rank uses distance when present and records it; map gains path count and minimum distance signals without using them as the sole score.
|
||||
- Observable: a sink in a helper two calls below a route becomes reachable; rank rationale shows graph_distance; the map order on split-sink still differs from hit-count order.
|
||||
- _Requirements: 3.1, 3.2, 3.3, 3.4_
|
||||
- _Boundary: ReachabilityAttach_
|
||||
- _Depends: 1.2_
|
||||
|
||||
- [ ] 3.2 Feed worth-fixing and extend the map gate with a path check
|
||||
- Worth-fixing accepts graph-reachable findings; the split-sink map gate additionally asserts a path from a planted entry to a planted function per language family.
|
||||
- Observable: a triggerable fake-sandbox verdict on a graph-reachable helper is worth fixing; map gate passes with the new assertion; existing assertions unchanged.
|
||||
- _Requirements: 3.5, 3.6, 8.1, 8.3_
|
||||
- _Depends: 3.1, 2.2_
|
||||
|
||||
- [ ] 4. Integration: hunter on paths
|
||||
- [ ] 4.1 (P) Add the path_context tool and a path-scoped prompt
|
||||
- The hunter can request one path by id and receives entry, hops, sink, summaries, guards (empty for now), and open questions; when paths exist the prompt asks for a verdict and a proposed input on that path.
|
||||
- Path context counts toward the existing step and cost budgets.
|
||||
- Observable: a scripted client that calls path_context and returns a finding yields a suspicion finding tagged with the path id; a client that ignores the path still works as today.
|
||||
- _Requirements: 4.1, 4.2, 4.5_
|
||||
- _Boundary: HunterPaths_
|
||||
- _Depends: 2.2_
|
||||
|
||||
- [ ] 4.1b (P) Resolve the hunter client through HarnessX when the extra and `[harnessx]` provider are present
|
||||
- `resolve_hunter_client(prefer_harnessx=..., harnessx_provider=..., model=...)` prefers a HarnessX provider adapter (its own credentials) and falls back to the OpenAI-compatible endpoint (`OPENROUTER_API_KEY` + `OPENROUTER_BASE_URL`, e.g. DeepSeek); the adapter must not use `asyncio.run` on non-coroutine awaitables.
|
||||
- Observable: with the extra absent the resolver returns the OpenRouter client; with a scripted provider object the adapter yields a ChatResponse; the pairs `--hunter` scorer uses the same resolver.
|
||||
- _Requirements: 4.5_
|
||||
- _Boundary: HunterPaths_
|
||||
|
||||
- [ ] 4.2 Make hunter findings with a locatable sink eligible as sandbox candidates
|
||||
- A hunter finding whose line matches a path sink line or an overlay sink line becomes a forced hotspot after the safety check, ranked after overlay promotions, within the candidate cap; evidence stays suspicion until a sandbox trigger or a path agreement is recorded.
|
||||
- Observable: with the fake sandbox, a scripted hunter finding on a path sink runs and its verdict is recorded; a hunter finding with no locatable sink is not a candidate; evidence level is unchanged by eligibility alone.
|
||||
- _Requirements: 4.3, 4.4_
|
||||
- _Boundary: HunterPaths_
|
||||
- _Depends: 4.1_
|
||||
|
||||
- [ ] 5. Integration: fact oracle and facts lever
|
||||
- [ ] 5.1 (P) Record open questions and ask a model only when configured
|
||||
- Composition records closed-kind questions at unresolved or unsummarized hops; when a model is configured the oracle asks with redacted callee context and appends candidate facts with model id, context hash, and timestamp; idempotent per key; no model means questions are listed in the manifest.
|
||||
- Observable: a scripted client answering nullable_return produces one candidate fact; a second run does not duplicate it; with no client the manifest lists the open question; no target code runs.
|
||||
- _Requirements: 5.1, 5.2, 5.3, 5.5, 5.6_
|
||||
- _Boundary: FactOracle_
|
||||
- _Depends: 2.2_
|
||||
|
||||
- [ ] 5.2 Load accepted facts below hand-written facts
|
||||
- The fact loader merges an accepted-facts file after the hand-written directory; collisions keep the hand-written row and record a degradation; candidate facts are never loaded.
|
||||
- Observable: an accepted nullable fact is visible to adjudication; the same id in a hand-written file wins; a candidate-only fact is invisible.
|
||||
- _Requirements: 5.4, 6.5_
|
||||
- _Boundary: FactOracle_
|
||||
- _Depends: 5.1_
|
||||
|
||||
- [ ] 5.3 Add the facts lever to the improve loop under the existing gate
|
||||
- A facts edit accepts or retracts a candidate fact; the validator allows only closed kinds sourced from the cache or a benchmark signal and never retracts hand-written facts; the existing accept gate and the per-profile holdout clause decide; rejection reverts the accepted file byte for byte; the journal records the edit and metrics.
|
||||
- Observable: a scripted candidate fact that turns a missed pair into a pass on holdout is accepted and persisted; one that regresses a profile is rejected and the file is unchanged; a pattern-text edit is rejected by the validator.
|
||||
- _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5_
|
||||
- _Boundary: FactsLever_
|
||||
- _Depends: 5.2_
|
||||
|
||||
- [ ] 6. Integration: fix points
|
||||
- [ ] 6.1 Compute fix points as dominators and rank by risk removed
|
||||
- A node dominates a path when every route from its entry to its sink passes through it; risk removed sums policy severity weighted by evidence over dominated paths; written as `fixpoints.json`.
|
||||
- Observable: a diamond fixture where one helper sits on four paths ranks that helper first with four paths collapsed; a reachable severity-5 finding stays in the worth-fixing candidate set regardless of fix-point order.
|
||||
- _Requirements: 7.1, 7.2, 7.4, 7.5_
|
||||
- _Boundary: FixPoints_
|
||||
- _Depends: 2.2_
|
||||
|
||||
- [ ] 6.2 Report fix first
|
||||
- Markdown and manifest gain a fix-first section listing fix points with witness paths; labelled as estimates.
|
||||
- Observable: a standard scan report on the split-sink fixture contains the section; `--fail-on` behavior is unchanged.
|
||||
- _Requirements: 7.3_
|
||||
- _Depends: 6.1_
|
||||
|
||||
- [ ] 7. Validation
|
||||
- [ ] 7.1 Measure labeled-function reachability on pair slices
|
||||
- For slices whose labels name a function, the pairs payload reports the fraction with finite distance; recorded in the roadmap next to the pair-corpus baseline.
|
||||
- Observable: `pairs --slice vibe-py --json` includes the fraction; the roadmap has the number.
|
||||
- _Requirements: 8.2_
|
||||
- _Depends: 3.1_
|
||||
|
||||
- [ ] 7.2 Prove gates unchanged and default tests offline
|
||||
- Detection gate, local pair gate, and existing split-sink assertions produce the same verdicts; extra-free suite green with graph assertions skipping; no default test uses a model, sandbox, or network.
|
||||
- Observable: gate entrypoints unchanged; pytest green in both matrices.
|
||||
- _Requirements: 8.3, 8.4, 8.5_
|
||||
- _Depends: 3.2, 4.2, 5.3, 6.2_
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- Guards on path records stay empty until `guard-dominance-regime`; the tool and record shapes reserve the field so that spec is additive.
|
||||
- `oracle_enabled` defaults to false until the first accepted facts have been reviewed by a maintainer; open questions are still recorded.
|
||||
- Do not change evidence stamps of existing sources in this spec; log the regex-as-corroboration question as a revalidation trigger for `propose-adjudicate-prove`.
|
||||
- Expect more sandbox candidates once hunter eligibility lands; the existing `max_candidates` cap and heuristic order hold cost.
|
||||
Reference in New Issue
Block a user