Open_Ultra_SAST/.kiro/specs/propose-adjudicate-prove/research.md

# Research & Design Decisions

## Summary

- **Feature**: `propose-adjudicate-prove`
- **Discovery Scope**: Extension of `three-stage-scan` (MAP grows an overlay; STATIC and REGRESS stay)
- **Key Findings**:
  - Inventory already decides; MAP ranks; REGRESS can run. Nothing adjudicates flow.
  - Labeled SAST slice Youden is about +9%. Leaks are OWASP-false / Juliet `goodG2B` that still contain the sink. Misses are one-hop assignments and sink aliases.
  - Clearwing already does the split we want: tree-sitter callgraph + pattern taint tables, then ranker (surface/influence/reachability 0.5/0.2/0.3), then mechanism-memory embeddings. OpenUltraSAST already copied the rank formula and has `json-local` vectors; they are not wired as prove-budget.
- Tree-sitter CLI/API covers C/C++/Python/JS/Go/Rust (Clearwing grammars); Joern is the CPG sidecar when present. Stdlib `ast` is last-resort Python only. LLM hunter remains suspicion-only.
  - Real working trees will often fail parse or cross a file boundary. Those cases must stay **unadjudicated**, not demoted.

## Research Log

### Labeled vs unlabeled code

- **Context**: Operator: known-vuln corpora are “nice”; written code will not be.
- **Sources Consulted**: In-tree `LANGUAGE_MANIFESTS`, `benchmarks/pairs/`, OWASP Benchmark true/false files, Juliet `bad`/`goodG2B`/`goodB2G`, CWE-Bench-Java catalog.
- **Findings**:
  - Cheat-sheets: one CWE, named sink, often same-line source.
  - OWASP: true and false files both often contain `exec` / `eval`; the difference is source constancy or infeasible branches.
  - Juliet `goodG2B`: hardcoded source, **same sink**.
  - Juliet `goodB2G`: same source, **sanitized sink** (`printf("%s")`).
  - Working trees: syntax errors, generated bundles, wrappers (`helpers.db_sqlite.execute`), no labels.
- **Implications**: Overlay scored on sast slice; unlabeled scans must not pretend demote. Unadjudicated is a first-class disposition.

### Engine choice

- **Context**: Operator prefers tree-sitter + Joern look, embeddings before prove, still v1-focused.
- **Sources Consulted**: `clearwing/sourcehunt/callgraph.py`, `taint.py`, `ranker.py`, `mechanism_memory.py`; `openultrasast/rank.py`, `vectorstore.py`, `index.py`.
- **Findings**:
  - Clearwing tree-sitter is optional: missing grammar disables callgraph, does not crash.
  - Clearwing taint is intra-procedural, pattern tables for sources/sinks — same shape as our fact TOML.
  - Clearwing ranker fills reachability from the callgraph; taint_paths boost surface. Embeddings retrieve *mechanisms*, they do not decide taint.
  - Our `heuristic_rank` already uses surface·0.5 + influence·0.2 + reachability·0.3. Vector store bakeoff selected `json-local`. `VALID_NAMESPACES` already includes `mechanisms` but nothing writes Clearwing-style records into it.

### Clearwing vector-store reference (real-world bugs)

- **Context**: Operator: Clearwing’s store is effective on real bugs, not textbook files.
- **Sources Consulted**: `clearwing/sourcehunt/mechanism_memory.py`, `data/memory/semantic_memory.py`, `docs/architecture.md` steps 7–8; `openultrasast/index.py` namespaces.
- **Findings**:
  - Store **mechanisms**, not findings and not 80-line file windows. Example: “length field trusted before alloc; 16-bit value widened to size_t.”
  - JSONL is the source of truth (`~/.clearwing/sourcehunt/mechanisms.jsonl`). Chroma/sqlite embeddings are ephemeral caches rebuilt from JSONL.
  - Write path is **verified findings only** (after adversarial verify / crash). Suspicion never seeds the store.
  - Clearwing’s JSONL + verified-write + metadata filters are the reference. Its TF-IDF default is **not**: bag-of-words will not match paraphrased real bugs (`length field trusted before alloc` vs `user-controlled size_t in malloc`).
  - OpenUltraSAST already has `OpenRouterEmbeddingClient` and json-local cosine. Use that as the only semantic ranker; CWE/language/tags are hard filters. No TF-IDF, no Chroma, no local MiniLM.
  - Query is language + file tags + optional free text; same-language boost +0.5. Top-N injects hunter context and variant-grep seeds — it does not decide taint.
  - Variant loop then greps the *messy* tree for structural matches of that mechanism. That is how it leaves Juliet behind.
- **Implications**: Copy JSONL + verified-write + metadata filters. Rank with OpenRouter embeddings only. Do not copy TF-IDF. Do not embed 80-line file windows as the prove store. Seed from sandbox-proven mechanisms, not OWASP `BenchmarkTest00007`.
  - Joern CLI (`joern-parse` / scripts) is the only practical CPG without a Scala runtime in-process.
- **Implications**: Probe tree-sitter like Docker. Joern sidecar cannot demote. Embeddings order prove only.

### Decision: Python intra-file first

- **Context**: Zero-dep core vs multi-language trees.
- **Selected**: **Superseded.** Tree-sitter is the prio engine; stdlib `ast` is Python last resort; `quick` stays zero-dep.
- **Rationale**: Operator working trees are mixed-language; Python-only adjudication would leave JS/C/Java as perpetual unadjudicated.
- **Trade-offs**: `standard` needs a grammar or CLI on PATH; missing tools → unadjudicated, not skip the scan.

### Evidence ladder

- **Context**: Three-stage-scan forbids hunter JSON as corroboration.
- **Sources Consulted**: `verification.EvidenceLevel`, `tool_hunter` suspicion rule, `worth_fixing` gate.
- **Findings**: Overlay can justify promote at `static_corroboration`. It cannot skip to crash or exploit. Worth-fixing stays sandbox + reachability.
- **Implications**: Disposition is not an evidence rung. Do not add a seventh ladder value.

## Architecture Pattern Evaluation

| Option | Description | Strengths | Risks | Notes |
|--------|-------------|-----------|-------|-------|
| Replace regex with CodeQL | External engine is the detector | Real dataflow | Breaks zero-dep, smoke gate, ledger shadows | Rejected |
| LLM hunter as adjudicator | Model reads file and votes | Flexible on messy code | Unstable Youden; violates suspicion rule | Rejected |
| Overlay on inventory | Regex proposes; tree-sitter taint adjudicates; embed ranks prove | Fits stages; multi-lang | Parser must be probed | **Selected** |
| Whole-program taint v1 | Inter-file | Closes wrappers | Too large; false demote risk | Defer; stay unadjudicated |

## Design Decisions

### Decision: Unadjudicated is not demote

- **Context**: Messy files and cross-file flow.
- **Alternatives**: Silent drop; treat as safe; force demote.
- **Selected**: Explicit `unadjudicated` with reason.
- **Rationale**: Operator writing code must not read silence as “clean.”
- **Trade-offs**: Reports stay noisy until languages and inter-file land.
- **Follow-up**: Count unadjudicated rate on unlabeled trees as an honesty metric.

### Decision: Score overlay on sast slice, inventory on smoke gate

- **Context**: +9% Youden vs 93% cheat-sheet recall.
- **Selected**: Two gates stay separate; overlay Youden cannot unblock smoke failure.
- **Rationale**: Same as detection-gate vs score in the improve loop.

## Risks & Mitigations

- False demote on real bugs (infeasible-branch OWASP tricks, wrappers) — never demote when flow is incomplete; only demote when the engine *has* a dominating constant or sanitizer.
- Youden chase via extra sinks — each coverage rule needs a false/good counterpart that must stay demoted or silent.
- Parse failures on operator WIP — unadjudicated, `quick` still works.

## References

- `.kiro/specs/three-stage-scan/design.md` — stage plan, sandbox, hunter suspicion
- `benchmarks/pairs/sast/README.md` — OWASP/Juliet slice
- NIST Juliet `goodG2B` vs `goodB2G` templates
- OWASP Benchmark expectedresults true/false protocol