Open_Ultra_SAST/.kiro/specs/propose-adjudicate-prove/design.md

Design Document

Overview

Purpose. This feature makes the three-stage scan mean three jobs: pattern inventory proposes sinks; a flow overlay adjudicates those proposals on real files; the sandbox proves only what was promoted. Operators scanning a working tree they are writing get honest unadjudicated instead of fake cleanliness.

Users. Developers get quick as cheap inventory on WIP code. AppSec gets standard dispositions (promote / demote / unadjudicated / coverage). DevOps gets deep budget spent on promoted reachable sinks.

Impact. Today quick_scan_findings is the de facto decision. After this spec, inventory is necessary but not sufficient. MAP writes overlay artifacts. REGRESS candidate selection reads promotions, not demotions.

Goals

• Map every current scan module onto propose, adjudicate, budget, or prove (rank/embed is not a verdict).
• Treat messy unlabeled trees as the default; labeled corpora as calibration.
• Fail closed: cannot parse or cannot follow flow → unadjudicated, not demote.
• Prioritize the semantic engine now: tree-sitter IR for in-scope languages; Joern optional CPG sidecar; stdlib Python parse only as last resort.
• Tunable embeddings + existing ranker order prove candidates.
• SAST pair slice scores the overlay; smoke gate stays inventory-only.

Non-Goals

• Replacing pattern rules or LANGUAGE_MANIFESTS.
• LLM-as-adjudicator or LLM-as-parser.
• Requiring Joern or an embedding API for quick.
• Using retrieval similarity to drop promotions.
• Auto-authoring source/sink/sanitizer facts from ousast improve.
• Buffer-size / memory-safety semantics (Juliet CWE-121 goodG2B stays a known leak until a later spec).

Boundary Commitments

This Spec Owns

• Overlay dispositions and flow-fact records.
• Source / sink / sanitizer facts as data (not CWE severity).
• Tree-sitter probe/runner (CLI-first, like Docker) and last-resort Python stdlib parse.
• Optional Joern probe/runner (CLI sidecar).
• Intra-file taint on the tree-sitter IR.
• Prove-budget: heuristic rank + OpenRouter embeddings over the mechanism log.
• Calibration scoring of overlay on the sast pair slice; unlabeled-scan honesty.

Out of Boundary

• Pattern text, rule enabled|shadow|disabled, CWE TSV, project score formula (three-stage-scan / ruleset / policy).
• Complexity-map identity keys, Docker sandbox flags, hunter suspicion contract (three-stage-scan).
• Pair catalog schema except scoring which findings count (overlay vs inventory).
• External analyzer execution (SARIF ingest remains a proposal input).
• MCP exposing deep.

Allowed Dependencies

• Preprocess, mapping, rank (heuristic_rank 0.5/0.2/0.3), quick_scan_findings, complexity map, ousast index / json-local vectorstore, tool hunter, sandbox runner, pair eval, reports.
• Host binaries when present: tree-sitter, joern / joern-parse (probed, never required for inventory).
• Optional embedding model from existing [embeddings] config.
• Clearwing as an oracle for tree-sitter callgraph + taint-pattern tables + mechanism retrieval, not a dependency.

Revalidation Triggers

• Adding a fourth disposition that implies "safe."
• Demoting when flow is incomplete.
• Stamping overlay above static_corroboration.
• Selecting sandbox candidates from demoted or unadjudicated sets.
• Demoting because retrieval score is low.
• Folding overlay Youden into python -m openultrasast.gate.
• Making tree-sitter or Joern a hard dependency of quick.

Architecture

Existing Architecture Analysis

Concrete mapping of what exists today onto the three roles:

Role	Stage / mode	Module today	What it actually does	After this spec
Propose	STATIC / quick	findings.quick_scan_findings	Pattern rules decide the report	Proposal list only
Propose	STATIC	mapping.ingest_sarif	Operator SARIF	Still proposals
Propose	STATIC	mapping.analyze_entry_points	Routes / CLI	Sources for overlay
Budget	STATIC	rank.heuristic_rank 0.5/0.2/0.3	File budget	Same formula; later prove order
Budget	ousast index	vectorstore json-local	Hunter retrieval	Prove retrieval on promotions
Adjudicate?	MAP	complexity.map	Hotspot band	Attention only
Adjudicate	MAP	new tree-sitter IR + taint	—	Primary semantic engine
Adjudicate	MAP	new Joern CLI sidecar	—	Optional CPG facts; never required
Adjudicate fallback	MAP	stdlib ast	Routes only today	Python last resort if tree-sitter missing
Suspicion	MAP	tool_hunter	LLM + grep	May query overlay; never adjudicate
Prove	REGRESS / deep	regress + sandbox	Recipes on ranked findings	Recipes on promoted then budgeted findings
Calibrate	CI	gate.py / pairs	Cheat-sheet vs Youden	Inventory vs overlay, unchanged split

Selected pattern. Propose → adjudicate → budget → prove. Clearwing's tree-sitter callgraph + taint tables + mechanism memory are the oracle, not a library import. Embeddings never demote.

flowchart TB
  subgraph propose["PROPOSE inventory"]
    PRE[preprocess]
    EP[entry points as sources]
    RX[pattern inventory]
    SARIF[optional SARIF]
  end
  subgraph adjudicate["ADJUDICATE overlay"]
    TS[tree-sitter IR]
    JOERN[optional Joern CPG]
    FALLBACK[Python stdlib parse last resort]
    FACTS[source sink sanitizer facts]
    TAINT[intra-file taint]
    DISP[promote demote unadjudicated coverage]
  end
  subgraph budget["BUDGET not a verdict"]
    RANK[heuristic and optional model ranker]
    VS[json-local embeddings]
  end
  subgraph prove["PROVE sandbox"]
    CAND[promoted reachable then budgeted]
    BOX[existing Docker runner]
    VERD[triggerable or not]
  end
  PRE --> EP
  PRE --> RX
  RX --> DISP
  SARIF --> DISP
  EP --> TAINT
  TS --> TAINT
  JOERN --> TAINT
  FALLBACK --> TAINT
  FACTS --> TAINT
  TAINT --> DISP
  DISP --> RANK
  DISP --> VS
  RANK --> CAND
  VS --> CAND
  CAND --> BOX
  BOX --> VERD

Technology Stack

Layer	Choice	Role
CLI / modes	existing quick standard deep	Role gating
Propose	pattern rules + SARIF	Inventory
Semantic IR	tree-sitter CLI or API extra; probe like Docker	Primary adjudicator parse
Semantic CPG	Joern CLI when present	Optional inter-file facts
Fallback parse	stdlib ast	Python only, last resort
Facts	TOML next to the ruleset	Sources sinks sanitizers
Budget	existing ranker + json-local vectorstore	Prove order, tunable
Prove	existing Docker CLI sandbox	Unchanged runner

quick stays zero-dep. standard/deep degrade to unadjudicated when parsers are missing.

File Structure Plan

src/openultrasast/semantic/
  __init__.py
  facts.py              # source sink sanitizer facts
  engines.py            # probe tree-sitter, joern, ast fallback
  ir.py                 # FileIR from the first engine that works
  taint.py              # worklist; incomplete -> Unadjudicated
  overlay.py            # dispositions
  prove_budget.py       # rank + optional retrieval over promotions
src/openultrasast/ruleset/semantic/
  python.toml javascript.toml c.toml java.toml

Modified Files

• stages.py / cli.py, overlay in MAP; prove-budget before regress.
• findings.py, inventory unchanged.
• regress/candidate.py, promote only, then budget order.
• rank.py / index.py / vectorstore.py, reuse; do not invent a second store.
• reports.py, disposition columns.
• pairs.py, sast slice uses overlay.
• mcp.py, still no deep.

System Flows

stateDiagram-v2
  [*] --> Proposed
  Proposed --> Unadjudicated: parse fail or flow incomplete or language unsupported
  Proposed --> Promoted: source reaches sink without sanitizer
  Proposed --> Demoted: constant or sanitizer dominates
  Proposed --> Coverage: flow to uninventoried sink
  Promoted --> Proven: sandbox triggerable
  Promoted --> Unproven: sandbox skip or unavailable
  Unadjudicated --> [*]
  Demoted --> [*]
  Coverage --> Promoted: same flow rule

Key decisions

• Demote only with a positive reason (constant or sanitizer on that path).
• Unadjudicated is the default for messy real files.
• Coverage findings enter the same state machine as proposals.

Unlabeled working-tree flow

No expected.toml. Success is: every inventory item has a disposition; unadjudicated reasons are explicit; deep does not execute demoted sinks; --fail-on worth-fixing ignores unadjudicated.

Requirements Traceability

Requirement	Summary	Components	Flows
1.1–1.5	Role split by mode	Stage plan, overlay, prove budget, regress	propose → adjudicate → budget → prove
8.1–8.6	Semantic engines	engines.py, ir.py, taint.py	tree-sitter then Joern then ast
9.1–9.5	Prove budget	prove_budget.py, rank, vectorstore	retrieval never demotes
2.1–2.5	Messy trees, unadjudicated	IR fail, taint incomplete	unlabeled scan
3.1–3.5	Promote demote coverage	taint, overlay	state diagram
4.1–4.5	Calibration vs product	gate.py, pairs.py	sast overlay score
5.1–5.5	Evidence honesty	overlay evidence, hunter	ladder
6.1–6.5	Prove promoted only	candidate filter, sandbox	deep
7.1–7.5	Artifacts and fail-on	reports, overlay.json	operator

Components and Interfaces

Component	Domain	Intent	Req Coverage	Dependencies	Contracts
InventoryProposer	Propose	Existing pattern + SARIF list	1.1	findings, mapping	State
FactStore	Adjudicate	Sources sinks sanitizers as data	3, 5.5	ruleset/semantic	State
FileIR	Adjudicate	Parse one file or fail	2.1	ast	Service
TaintEngine	Adjudicate	Intra-file paths or incomplete	2.2, 3	FileIR, FactStore	Service
Overlay	Adjudicate	Dispositions on proposals	1.2, 3, 5	TaintEngine	State
ProveFilter	Prove	Candidates from promote	1.3, 6	Overlay, regress	Service
OverlayPairEval	Calibrate	SAST slice uses dispositions	4	Overlay, pairs	Batch

Adjudicate layer

OverlayRecord

Field	Detail
Intent	One disposition per proposal or coverage item
Requirements	3.5, 7.1

State contract

proposal_id: str
path: str
line: int | None
disposition: promote | demote | unadjudicated | coverage
reason: str
cwe: str
sources: list[str]
sinks: list[str]
sanitizers: list[str]
evidence_level: static_corroboration | suspicion

Invariants

• demote requires a recorded constant or sanitizer.
• unadjudicated requires a reason in {parse_failed, language_unsupported, flow_incomplete, facts_unavailable}.
• promote and coverage do not exceed static_corroboration.

TaintEngine service

analyze_file(path, text, language, facts) -> FileFlow | Unadjudicated
adjudicate(proposals, flows, facts) -> list[OverlayRecord]

• Engine order: tree-sitter grammar → Joern file export if probed and useful → Python stdlib parse → unadjudicated.
• Joern may add paths; it must not demote a tree-sitter promote, and a Joern miss must not demote.
• Postconditions: every inventory proposal appears exactly once; extras are coverage only.

Prove layer

ProveFilter

Inbound: overlay + reachability from mapping. Outbound: existing regress candidates.

• Drop demote.
• Do not sandbox unadjudicated as a negative proof.
• Keep promote that are reachable or inferred-file-surface.
• Order survivors with heuristic rank, then mechanism recall: hard filter (language, CWE, tags) then OpenRouter dense cosine on the mechanism text. Low similarity never drops a promotion.
• Write mechanisms only after prove/triggerable. JSONL log is authoritative; json-local mechanisms namespace is a rebuildable embedding cache via OpenRouterEmbeddingClient.

Data Models

Facts (data, versioned with the ruleset)

• source: identifier (e.g. request parameter, header, stdin, argv)
• sink: call shape + CWE (aligned to existing rule ids when possible)
• sanitizer: call shape that stops taint (parameterized execute, printf with literal format, safe_load, list-form subprocess)

FileFlow

• assignments and call arguments in one function
• one-hop name copies (bar = param, sql = f"...{bar}")
• no types, no inter-file, no may-alias heap in v1

Mechanism (Clearwing reference architecture)

Identity is the pattern, not a file slice:

Field	Role
id	stable uuid
summary	one sentence, no path/line
cwe, language	filters
tags, keywords	hard filters, not a ranker
what_made_it_exploitable	why a near-miss became a bug
source_finding_id, source_repo	provenance only

Persistence: append-only JSONL (portable). Recall is not Clearwing's TF-IDF ladder.

1. Filter, language, CWE, tags (exact overlap). This is a hard prefilter, not a ranker.
2. Rank, OpenRouter /embeddings (OpenRouterEmbeddingClient, existing [embeddings].model) over summary + what_made_it_exploitable + tags. Cosine against the promotion's overlay text. Cache vectors in json-local mechanisms namespace, keyed by mechanism id + model id.
3. Degrade, no key / no model / embed failure → heuristic ranker only, degradation recorded. Never bag-of-words cosine.

Query for a promotion: embed the overlay rationale (sources, sink, CWE). Result is a boost to prove order, never a demote.

Repo-code chunks (ousast index 80-line slices) stay for hunter/verifier retrieval. They are not the prove-budget store. Textbook fixtures must not fill the mechanism log unless they were actually proven in the sandbox.

Error Handling

Condition	Disposition	Operator sees
Syntax error / parse fail	unadjudicated parse_failed	proposal kept, marked unresolved
No tree-sitter grammar and not Python	unadjudicated language_unsupported	same
Joern missing	ignore sidecar	standard still overlays via tree-sitter
Embedding model missing	heuristic prove order	deep still runs
Callee in another file	unadjudicated flow_incomplete	same
Fact file missing	unadjudicated facts_unavailable; quick still emits inventory	degradation in manifest
Sandbox missing	promoted remains unproven	sandbox_unavailable

Testing Strategy

• Unit: parse fail → unadjudicated; constant exec("ls") → demote; eval(request.args) → promote; sql = f"…{x}"; execute(sql) → coverage or promote.
• Pair: --slice sast uses overlay; juliet-c-cwe134-printf stays PASS; owasp-python-deser stays PASS; Juliet goodG2B exec/strcpy must demote or unadjudicated, not inventory-leak counted as pair fail for overlay.
• Unlabeled: tmp tree with a syntax-error file next to a clean eval(request.args). Error file unadjudicated, clean file promoted; no expected.toml required.
• Gate: python -m openultrasast.gate still inventory-only.
• Deep: candidate list excludes demoted.

Security

• Overlay does not execute target code.
• Facts cannot override CWE severity.
• Hunter still cannot stamp corroboration.

Performance

• Overlay is per-file, MAP stage, same file budget as rank.
• Skip files already unreadable at preprocess.

Migration

• Default standard runs overlay when facts load; if facts missing, degrade as Requirement 7.5.
• Existing quick CI unchanged.
• Pair gate local 100% remains inventory-or-overlay-equivalent on fixture counterparts (those files are still "nice"; unlabeled tests are additional).