Open_Ultra_SAST/.kiro/specs/reachability-flow-model/research.md

Research & Design Decisions

Summary

• Feature: reachability-flow-model
• Discovery Scope: Complex Integration across semantic, mapping, rank, complexity, hunter, regress, improve
• Key Findings:
  • FunctionIR already carries every call site per function for Python, JS, TS, C, and Java when the extra is present. A name-resolved call graph is a few hundred lines over data that exists.
  • Reachability today is "line inside an entry-point handler" (mapping.attach_reachability_hints); rank bumps an integer from tags and hints; worth_fixing needs reachable or inferred-file-surface. No call edge exists anywhere.
  • Sandbox candidates come only from hotspots filtered to overlay-promoted inventory ids (cli.py:404, prove_filter.filter_promoted_hotspots); a hunter snippet is used only for an already promoted hotspot (candidate.py:199). The LLM plane cannot create a candidate.
  • Evidence is stamped by origin: regex hit static_corroboration (findings.py:110), hunter suspicion (tool_hunter.py:408).
  • Improve levers are rule and policy only (improve/validator.py); facts are not a lever. propose-adjudicate-prove excluded auto-authored facts.
  • Code shape: detection logic 1,910 lines of 12,128; harness plumbing 2,885; governance and benchmarks 3,178.
  • Clearwing fills reachability from a tree-sitter callgraph and lets taint paths boost surface; the embeddings retrieve mechanisms, not decide taint. That split is the reference.
  • Literature: RealVuln shows LLM-based scanners at 51.7 to 73 F3 on real Python web code versus Semgrep at 17.7; CVE-Bench shows agents exploit at most 13 percent of real web CVEs. Recall on real code comes from models with tools; precision and reproducibility come from static checks and execution.

Research Log

Where a call graph can come from

• Context: Avoid a new parser or a whole-program engine.
• Sources Consulted: semantic/ir.py, semantic/cst.py, mapping.py, preprocess.py, hunter_tools.find_refs.
• Findings: FileIR.functions[*].calls[*].name is the callee text. Python stdlib parse yields the same records. Imports are not in the IR; the walker can add an imports tuple per file cheaply. find_refs is a word-boundary regex over files and is the only cross-file lookup today.
• Implications: semantic/callgraph.py consumes FileIR plus an additive imports field. Resolution order: same-file definition, imported module definition, any definition with that name (ambiguous edges). Method calls resolve by trailing name. C resolves by bare name across the tree.

Summaries versus whole-program taint

• Context: propose-adjudicate-prove deferred inter-file taint as too large and risky for false demotes.
• Sources Consulted: semantic/taint.py (_analyze_function seeds parameters as tainted, tracks names, emits TaintPath and Demotion), overlay invariants.
• Findings: Because parameters are already seeded tainted, a per-function run already answers "which parameter reaches which sink." Adding "which parameter reaches the return" is a small extension. Composition is then a graph walk with a depth bound. Nothing needs alias analysis to produce a useful path; anything unresolved stays flow_incomplete.
• Implications: Summaries are a pure function of FunctionIR plus facts, cacheable by file hash. Composition never demotes; it only adds paths or marks incompleteness.

Hunter to candidate

• Context: The LLM plane is the recall lever the field data supports, but it is walled off from prove.
• Sources Consulted: tool_hunter.py, regress/candidate.py, three-stage-scan design (hunter snippets run only in the sandbox after safety check), verification.py.
• Findings: The safety check and sandbox are origin-agnostic. The wall is filter_promoted_hotspots and the candidate selector's reliance on inventory ids. A hunter finding with a locatable sink line can be given a synthetic hotspot the same way _hotspot_from_forced_finding already does for severity-5 inventory.
• Implications: Add eligibility for hunter findings that name a sink line resolvable to a path record or an overlay sink. Evidence stays suspicion until sandbox trigger or path agreement.

Facts from a model, safely

• Context: The engine lacks callee knowledge: nullable returns (tls1_lookup_sigalg), bounded arguments, transformer set/get (ConfigParser), macro out-params (n2s).
• Sources Consulted: semantic/facts.py loader, improve/validator.py, improve/journal.py, calibration.py ledger patterns, redaction.py.
• Findings: Facts are versioned TOML loaded by load_facts. The loader can merge a second directory at lower precedence. The improve loop has a strict validator and byte-for-byte revert. Redaction exists for prompts.
• Implications: Model answers are candidate facts in .openultrasast/calibration/candidate_facts.toml; accepted facts in accepted_facts.toml; loader precedence hand-written > accepted > none. The facts lever moves rows between candidate and accepted under the same gate. No pattern text is ever model-authored; question kinds are closed.

Fix points

• Context: The principle is risk reduced per fix, not findings per file.
• Sources Consulted: complexity/map.py, complexity/ledger.py, regress/verdict.py, scoring/project_score.py.
• Findings: The path graph is small (functions, not statements). Dominators over a DAG of entry-to-sink paths are cheap. Severity is available per CWE from policy.
• Implications: risk/fixpoints.py computes dominator sets per sink path and aggregates risk removed. It is a report and artifact, not a gate; it cannot hide severity-5 reachable findings.

Architecture Pattern Evaluation

Option	Description	Strengths	Risks / Limitations	Notes
Joern CPG as required engine	Full inter-procedural dataflow	Real dataflow	Scala runtime, not zero-dep, slow on PRs	Rejected; stays optional sidecar
CodeQL databases	Strong inter-procedural	Mature	Build step, license limits, heavy	Rejected as required; SARIF ingest remains
Whole-program taint in Python	Own engine	Full control	Months of work, alias analysis, false demotes	Rejected
Name-resolved call graph plus composed summaries	Graph over existing IR; summaries from existing taint	Days to build; over-approximate by design; feeds hunter and fix points	Ambiguous names, dynamic dispatch	Selected
LLM as reachability oracle	Ask the model who calls what	No code	Unverifiable, expensive per scan	Rejected; model answers closed callee questions only

Design Decisions

Decision: Over-approximate edges, never infer safety

• Context: Missing edges silently hide risk; false edges cost hunter budget.
• Selected Approach: Ambiguous names produce edges to every definition, flagged. Unresolved calls produce an unresolved edge. Distance is infinite only within the depth bound and is reported as unknown.
• Rationale: Consistent with the fail-closed rule in the overlay.
• Trade-offs: More hunter work on large trees; bounded by existing hotspot caps.

Decision: Summaries are pure and cached

• Context: Cost on large repos.
• Selected Approach: Summary keyed by file hash plus facts version; recomputed only when either changes.
• Trade-offs: Cache invalidation on facts version bump; acceptable.

Decision: Evidence rises on checks, not on origin

• Context: Regex stamped corroboration, hunter stamped suspicion.
• Selected Approach: This spec does not change existing stamps. It adds the checks that raise hunter evidence: composed path agreement or sandbox trigger. A follow-up may lower regex-only hits to inventory; recorded as a revalidation trigger for propose-adjudicate-prove.
• Rationale: Keep this spec additive; changing existing stamps changes gate baselines.

Decision: Closed question kinds, gated acceptance

• Context: Auto-facts were excluded for good reason: unbounded model-authored patterns.
• Selected Approach: Five question kinds, structured answers, candidate cache with provenance, acceptance only via the facts lever under the per-profile holdout gate.
• Trade-offs: Slower than trusting answers directly; the gate is the point.

Decision: Fix points are dominators, reported not gated

• Context: Do not add a seventh evidence rung or a new CI condition.
• Selected Approach: Compute and report; the severity-5 reachable invariant from three-stage-scan stays.

Risks & Mitigations

• Dynamic dispatch and callbacks in JS and Python produce unresolved edges — report unresolved counts; hunter find_refs remains available as a fallback.
• Graph blow-up on large trees — depth and fan-out bounds in config; hotspot caps unchanged.
• Model answers wrong — never used unless accepted by the gate; provenance recorded; hand facts win.
• Hunter candidates increase sandbox load — existing max_candidates cap; hunter candidates ranked after overlay promotions unless a path agrees.
• Fix-point estimates misread as guarantees — label as estimates; show witness paths.

References

• Clearwing sourcehunt callgraph, taint, ranker (reference architecture; not a dependency)
• RealVuln — rule-based versus LLM scanners on real Python web code
• CVE-Bench — agent exploitation ceiling on real web CVEs
• .kiro/specs/three-stage-scan/design.md — hunter tools, candidate selection, worth-fixing
• .kiro/specs/propose-adjudicate-prove/design.md — facts as data, dispositions, evidence honesty
• .kiro/specs/harnessx-self-improving-rulesets/design.md — bounded levers, accept gate, revert
• .kiro/specs/pair-corpus-honesty/ — per-profile holdout gate, hunter scorer
• Memory note overlay-gaps-diagnosis-2026-09