Open_Ultra_SAST/.kiro/specs/corpus-seeded-mechanisms/research.md

# Research & Design Decisions

## Summary

- **Feature**: `corpus-seeded-mechanisms`
- **Discovery Scope**: Extension of mechanism memory, MAP stage, pair eval, and the improve loop
- **Key Findings**:
  - `semantic/mechanisms.py` already has `Mechanism`, `MechanismStore` (append-only JSONL), `append_mechanism`, embedding cache, hard filters by language/CWE/tags, and `order_promotions`. It is written only by `cli._append_proven_mechanisms` after a sandbox trigger, so it is empty on every real tree.
  - Clearwing's recall lever is the variant loop: a proven mechanism becomes a structural search over the messy tree. OpenUltraSAST copied the store and the retrieval, not the loop.
  - `FileIR` call sites already carry callee name, argument texts, argument identifier lists, and constant flags; binds carry source text and names. That is enough for a shape without new parsing.
  - The pair scorer (`pair-corpus-honesty`) gives labeled function, mechanism id, provenance, and soon a review tier; the fixed excerpt gives the guard the fix added by diffing statement kinds.
  - The improve loop has bounded levers with a validator, journal, byte-for-byte revert, and now a per-profile holdout clause; a `mechanisms` lever fits without changing the gate.
  - Measured 2026-09-05: vibe-py 13/35, vfc-js 1/16, sast 4/10 achievable. Web classes repeat structurally (request-derived string into `exec`/`execute`, identity from the body, `os.path.join` with a parameter, permissive defaults), which is what a shape search catches.

## Research Log

### Mechanism store today

- **Context**: Can the existing store carry corpus-derived records?
- **Sources Consulted**: `semantic/mechanisms.py`, `cli.py` `_append_proven_mechanisms`, `propose-adjudicate-prove` design §Mechanism.
- **Findings**: `Mechanism` fields are summary, cwe, language, tags, keywords, what_made_it_exploitable, source_finding_id, source_repo. Identity is the pattern. JSONL is authoritative; vectors are a rebuildable cache keyed by id and model.
- **Implications**: add `origin` (`sandbox` | `corpus`), `review_tier`, `pairs`, `shape` (structured), `guard` as additive fields with defaults so old rows load. A second writer (`append_from_pair`) beside the sandbox writer.

### Shape derivation from FileIR

- **Context**: Avoid regex and LLM authoring.
- **Sources Consulted**: `semantic/ir.py` (`CallSite.arg_names`, `arg_is_constant`, `Bind.names`, `FunctionIR.params`), `semantic/taint.py` (`_source_in`, parameter seeding).
- **Findings**: for the labeled function, the sink call site is the call whose trailing name equals the label's sink or whose line equals the label's line; source positions are argument indexes whose identifiers trace by binds to a fact source, a parameter, or a container read. Guard kind comes from statements on the fixed side absent on the vulnerable side: `if` with `== NULL`/`is None`/`!x` → `null_test`; comparison with `<`, `>`, `len(` → `bounds_test`; membership in a literal collection or `startswith`/regex → `allowlist_test`; identity read from an auth context (`request.state`, `req.user`, `session`) → `auth_check`; parameterized second argument → `parameterized_call`; changed declared type → `type_change`.
- **Implications**: `semantic/variants.py` implements `derive_shape(pair_ir_vuln, pair_ir_fixed, label)` and `match_shape(file_ir, shape)`. Guard classification is heuristic and closed; unknown → `none`.

### Where variant search sits

- **Context**: Propose, adjudicate, prove roles must stay separate.
- **Sources Consulted**: `three-stage-scan` design (MAP stage, hunter suspicion), `propose-adjudicate-prove` (overlay records, coverage), `regress/candidate.py`.
- **Findings**: MAP already runs overlay and the tool hunter; variant search is a third proposer that reads `FileIR` and the store. Findings at `suspicion`, origin `variant`; when the overlay has a flow at that call site, merge (attach mechanism id to the overlay record). Candidate eligibility follows the hunter-eligibility path being added in `reachability-flow-model` or the sev-5 forced-hotspot path today.
- **Implications**: no disposition change; one new finding origin; manifest counters.

### Leave-one-out

- **Context**: Prove that corpus growth raises recall, honestly.
- **Findings**: pair eval materializes each side into a temp tree already; a leave-one-out runner seeds a temp store from the other trusted pairs and runs variant search on both sides, scoring with the pair scorer's detection and leak rules.
- **Implications**: `pairs.py` gains a read-only consumer path (`evaluate_loo`) or a sibling module `semantic/loo.py` to keep the scorer's boundary; report per slice, profile, mechanism; artifact only.

## Architecture Pattern Evaluation

| Option | Description | Strengths | Risks / Limitations | Notes |
|--------|-------------|-----------|---------------------|-------|
| Embedding retrieval only | Rank existing promotions by mechanism similarity | Already exists | Cannot propose anything new; store empty | Rejected as recall lever |
| LLM writes Semgrep rules from pairs | Model authors patterns | Flexible | Unbounded pattern text, unverifiable, violates no-model-authored-patterns | Rejected |
| Structural shapes from FileIR plus variant search | Closed shape vocabulary, deterministic match | Zero-dep, bounded, testable leave-one-out | Shapes coarse; guard heuristics | **Selected** |
| Fine-tuned classifier on pairs | ML detector | Learns anything | Opaque, no fix guidance, data too small | Rejected |

## Design Decisions

### Decision: Seed only from seeded and reviewed tiers

- **Context**: `title` pairs are unreviewed guesses; `advisory` pairs have correct fixes but unlabeled mechanisms.
- **Selected Approach**: export reads `review_tier`; `advisory`/`title` may be scored leave-one-out as held-out targets but never seed.
- **Trade-offs**: fewer seeds at first; honest about what the detector was taught.

### Decision: Variant is a proposer at suspicion

- **Context**: Evidence honesty.
- **Selected Approach**: origin `variant`, `suspicion`; merge with overlay flow when present; sandbox may raise.
- **Trade-offs**: variant-only hits do not fail CI; correct.

### Decision: Lever admits records, not shapes

- **Context**: Keep the validator closed.
- **Selected Approach**: `mechanisms` lever edits reference record ids produced by the exporter; free-form shapes are rejected.

## Risks & Mitigations

- Coarse shapes over-match (`exec(x)` everywhere) — source-position requirement plus fact/parameter/container check; report leak rate on fixed sides leave-one-out; the per-profile gate rejects admissions that leak.
- Guard heuristics misclassify — closed set with `none`; reported, not used for demotion.
- Store growth — dedupe by shape; one record per shape with pair provenance list.
- Leave-one-out cost — one temp store per pair over tens of pairs; fine; cache parsed IR per excerpt.

## References

- `semantic/mechanisms.py`, `semantic/ir.py`, `semantic/taint.py`
- `.kiro/specs/propose-adjudicate-prove/design.md` §Mechanism (Clearwing reference architecture)
- `.kiro/specs/pair-corpus-honesty/` (review tiers, pointer slices, scorer rules)
- `.kiro/specs/harnessx-self-improving-rulesets/design.md` (bounded levers, accept gate)
- Clearwing sourcehunt mechanism memory and variant loop (reference, not a dependency)