marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(specs): kick off corpus-seeded-mechanisms and reachability-flow-model (unapproved)

Browse Source 
Two engine specs drafted after the pair-corpus baseline, awaiting owner
approval before implementation:

- corpus-seeded-mechanisms: structural shapes derived from trusted pairs,
  export, variant search merged with overlay flows, leave-one-out
  measurement, and a closed `mechanisms` lever for the improve loop.
- reachability-flow-model: call graph over FunctionIR, reachability from
  entry points to sinks, hunter client preferring HarnessX (task 4.1b).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-05 10:45:43 +02:00
co-authored by Claude Fable 5.1
parent 3eba6e10a0
commit 7d238f2400
12 changed files with 1427 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/specs/corpus-seeded-mechanisms/brief.md
+48
View File
@@ -0,0 +1,48 @@
# Brief: corpus-seeded-mechanisms
## Problem
The pair corpus is only a ruler. `pair-corpus-honesty` made it honest and is growing it past the license and review walls, but nothing in the tool learns from a labeled pair: the detector on an operator's tree is the same regex inventory plus intra-file taint whether the corpus holds 11 pairs or 1,100. The mechanism store that Clearwing uses as a library (`semantic/mechanisms.py`, JSONL plus an embedding cache) exists here but is filled only from sandbox-proven findings, so it is empty in practice and only reorders prove candidates.
Measured 2026-09-05: vibe-py 13/35 pair-correct, vfc-js 1/16, sast 4/10 achievable, vfc 0/176. Recall on real web code is where the tool is weakest, and web bug classes repeat structurally across projects (identity from the request body, `exec` of a request-derived string, path joined from a parameter, RLS or CORS left open).
## Desired Outcome
Every `seeded` or `reviewed` pair becomes a mechanism record: the source shape, the sink statement shape, and the guard the fix added. A structural variant search runs those shapes over any scanned tree and proposes matches, which the overlay and the sandbox then verify. Leave-one-out recall on the corpus is the corpus's own detection rate and rises as the corpus grows. The improve loop gains a lever that admits mechanisms, so it can raise recall instead of only defending it.
Principle written into the roadmap: **recall first under a false-positive ceiling, per provenance profile, and every reviewed pair becomes a searchable mechanism.**
## Approach
Derive a mechanism record from a pair by parsing the vulnerable excerpt with the existing `FileIR`, taking the sink call site the label names, the identifiers that carry the source, and the guard statements that appear only on the fixed side. Normalize into a language-scoped shape: sink callee name, argument arity, which argument positions carry source-derived identifiers, and the guard kind. Variant search walks a scanned tree's `FileIR`, finds call sites with the same shape whose arguments carry a fact source or a parameter, and emits `variant` findings at `suspicion` carrying the mechanism id; when the overlay already has a flow to that sink, the record is `coverage` with the mechanism id attached. Leave-one-out evaluation seeds from all pairs but one and scores the held-out pair. The improve loop proposes admitting a candidate mechanism when it recovers a missed holdout pair; the existing accept gate and the per-profile holdout clause decide.
## Scope
- **In**: mechanism export from pairs; shape normalization; variant search as a proposer in MAP; leave-one-out measurement command and payload; `mechanisms` improve lever with validator rules; store hygiene (tier, provenance, dedupe, version); offline tests.
- **Out**: LLM-authored shapes; changing dispositions or evidence rungs; embedding retrieval changes (existing prove-order reuse stays); inter-file flow (reachability spec); any merge gate; vendoring more pairs (corpus spec).
## Boundary Candidates
- Shape extraction (reads FileIR) vs overlay dispositions (unchanged)
- Mechanism store schema (additive fields) vs prove-order retrieval (unchanged consumer)
- Improve lever (admit/retract mechanisms) vs rule/policy levers (unchanged)
## Out of Boundary
- Taint semantics, facts, CST walker (`overlay-ir-completeness`, `guard-dominance-regime`)
- Call graph, summaries, hunter (`reachability-flow-model`)
- Pair scorer and catalogs (`pair-corpus-honesty`), except one read-only consumer of `PairCase`
## Upstream / Downstream
- **Upstream**: `PairCase`/`load_pair_catalog` with `review_tier`; `FileIR`; `MechanismStore`; `adjudicate`; `run_improvement` gate with per-profile holdout.
- **Downstream**: `reachability-flow-model` attaches paths to variant findings; `guard-dominance-regime` consumes the guard shapes as sanitizer candidates; reports show mechanism ids on findings.
## Existing Spec Touchpoints
- **Extends**: `propose-adjudicate-prove` mechanism memory (additive fields, second write path), `three-stage-scan` MAP stage (one more proposer), `harnessx-self-improving-rulesets` improve loop (one more bounded lever).
- **Depends**: `pair-corpus-honesty` Req 9 (tiers) for which pairs may seed.
## Constraints
Core `dependencies = []`. Variant findings never exceed `suspicion` without overlay or sandbox agreement. Mechanisms seed only from `seeded` or `reviewed` pairs. JSONL stays authoritative; caches are rebuildable. Leave-one-out is reported, never gated. No target code executes outside the sandbox.
.kiro/specs/corpus-seeded-mechanisms/design.md
+255
View File
@@ -0,0 +1,255 @@
# Design Document
## Overview
**Purpose.** Turn trusted pairs into mechanism records, search scanned trees for structural variants, measure the corpus's own detection rate leave-one-out, and give the improve loop a recall-raising lever.
**Users.** Maintainers export and measure. AppSec engineers get variant findings that name the known mechanism and its known fix. The improve loop admits or retracts mechanisms under the existing gate.
**Impact.** `semantic/mechanisms.py` gains additive fields and a second writer. New `semantic/variants.py` (shapes and matching) and `semantic/loo.py` (leave-one-out). MAP runs variant search after overlay. `improve` gains the `mechanisms` lever. Reports show mechanism ids. No disposition, gate, or sandbox change.
### Goals
- One mechanism record per distinct shape, seeded from `seeded` and `reviewed` pairs.
- Variant findings on real trees at `suspicion`, merged with overlay flows when present.
- Leave-one-out recall, silence, and Youden per slice, profile, and mechanism.
- `mechanisms` lever under the existing accept gate and per-profile holdout.
### Non-Goals
- LLM-authored shapes, regex growth, disposition or ladder changes, inter-file flow, embedding changes, new gates.
## Boundary Commitments
### This Spec Owns
- Shape derivation and matching (`variants.py`).
- Additive mechanism fields, corpus writer, dedupe (`mechanisms.py`).
- Variant proposer in MAP and manifest counters.
- Leave-one-out runner and artifact (`loo.py`, `pairs --loo`).
- `mechanisms` lever, its validator rules, proposer, journal fields.
- Report rendering of mechanism ids.
### Out of Boundary
- `OverlayRecord` fields beyond an optional `mechanism_id`; dispositions; evidence rungs.
- Catalog schema, scorer rules, harvest (`pair-corpus-honesty`).
- Taint, facts, walker; call graph; hunter.
- Rule and policy levers; ledger format.
### Allowed Dependencies
- `FileIR`, `parse_file`, `SemanticFacts`/`load_facts` (read), `adjudicate` output, `MechanismStore`, `PairCase`/`load_pair_catalog`/`select_slice`, pair scorer matching helpers (read), `run_round` gate, `EvolveValidator`, journal, reports.
### Revalidation Triggers
- Variant findings stamped above `suspicion` without overlay or sandbox agreement.
- Seeding from `advisory` or `title` pairs.
- Free-form shape text accepted by the validator.
- Leave-one-out used as a gate.
## Architecture
```mermaid
flowchart TB
PAIRS[trusted pairs: seeded | reviewed] --> EXPORT[derive_shape per pair]
EXPORT --> STORE[mechanisms.jsonl origin=corpus]
STORE --> VS[variant search in MAP]
IR[FileIR of scanned tree] --> VS
FACTS[facts: sources] --> VS
VS --> VF[variant findings: suspicion, mechanism_id]
OV[overlay records] --> MERGE[merge when same call site has a flow]
VF --> MERGE
MERGE --> CAND[regress candidates after promotions]
PAIRS --> LOO[leave-one-out runner]
STORE -. temp store per held-out pair .-> LOO
LOO --> ART[loo.json + roadmap table]
LOO --> PROP[improve proposer: admit/retract]
PROP --> GATE[accept gate + per-profile holdout]
GATE --> STORE
```
**Key decisions**
- Shapes are closed structures derived from `FileIR`, never text patterns.
- Variant search proposes at `suspicion`; overlay or sandbox raises.
- Leave-one-out is the corpus's detection rate and is reported only.
- The lever edits record ids from the exporter; validator rejects anything else.
### Technology Stack
| Layer | Choice | Role | Notes |
|-------|--------|------|-------|
| Shapes | stdlib dataclasses over `FileIR` | derive, match | no new parser |
| Store | existing JSONL + cache | corpus and sandbox records | additive fields |
| Search | MAP stage | proposer | file budget as rank |
| LOO | temp dirs + existing pair materialization | measurement | artifact only |
| Improve | existing lever machinery | admit/retract | same gate |
## File Structure Plan
```
src/openultrasast/
├── semantic/
│ ├── mechanisms.py # Mechanism += origin, review_tier, pairs, shape, guard; append_from_pair; dedupe_by_shape
│ ├── variants.py # Shape, GuardKind, derive_shape(vuln_ir, fixed_ir, label, facts) -> Shape|None; match_shapes(file_ir, shapes, facts) -> list[VariantHit]
│ └── loo.py # evaluate_loo(cases, *, store_factory) -> LooResult; payload
├── findings.py # origin "variant" via tags/finding_id prefix "variant:" (no schema change)
├── cli.py # MAP: variant search after overlay; manifest counters; `pairs --loo`; `mechanisms export --slice`
├── improve/validator.py# MechanismEdit lever "mechanisms"
├── improve/evolve.py # proposer from loo misses/leaks; gate unchanged
└── reports.py # mechanism id, guard, provenance on findings
tests/
├── test_variants.py, test_loo.py, test_mechanism_export.py, test_improve.py (lever)
```
### Modified Files
- `semantic/overlay.py` — `OverlayRecord.mechanism_id: str | None = None` (additive).
- `regress/candidate.py` — variant findings eligible after overlay promotions (reuses forced-hotspot path).
- `config.py` — `[variants] enabled=true, max_mechanisms=500`.
## System Flows
### Export
```mermaid
flowchart LR
A[load trusted pairs] --> B[parse vuln + fixed excerpts]
B --> C[find labeled sink call site]
C --> D[source positions via binds]
B --> E[guard kind: statements only on fixed side]
D --> F[Shape]
E --> F
F --> G[dedupe by shape; append with pairs provenance]
```
### Variant search in MAP
For each parsed file within the MAP budget: for each call site whose trailing callee name equals a shape's sink name and whose arity matches, check the shape's source positions: the argument carries a fact source, a function parameter, or a container read (bind whose value is a subscript or `.get(`). On match emit a `variant:<mechanism_id>:<path>:<line>` finding at `suspicion`. If an overlay record exists at that path and line with a flow, set its `mechanism_id` and do not emit a separate finding.
### Leave-one-out
For each trusted pair P in the slice: temp store from all trusted pairs except P; materialize P's sides; run variant search on both; detection if a variant hit lies inside P's labeled function and names P's mechanism id or matches P's sink; leak if any hit on the fixed side. Aggregate per slice, profile, mechanism; record which mechanism found P.
## Requirements Traceability
| Requirement | Summary | Components | Flows |
|-------------|---------|------------|-------|
| 1.1–1.5 | seed from trusted pairs, dedupe, origin corpus, maintainer command | Export, Store | export |
| 2.1–2.5 | structural shapes, closed guard kinds, no text, extra-absent degradation | Shape | export |
| 3.1–3.6 | proposer at suspicion, merge, candidates, budget, absent from quick | VariantSearch | MAP |
| 4.1–4.4 | leave-one-out per slice/profile/mechanism, artifact, not a gate | LOO | leave-one-out |
| 5.1–5.5 | lever, validator, proposer, gate unchanged, journal | Lever | improve |
| 6.1–6.3 | reports, manifest counters, suspicion label | reports | — |
| 7.1–7.3 | offline tests, gates unchanged, extra-free green | tests | — |
## Components and Interfaces
| Component | Domain | Intent | Req | Dependencies | Contracts |
|-----------|--------|--------|-----|--------------|-----------|
| Shape | semantic | derive and compare shapes | 2 | FileIR, facts | Service |
| Export | maintainer | pairs → records | 1 | PairCase, Shape, Store | Batch |
| Store | semantic | additive fields, dedupe | 1.3, 1.4 | JSONL | State |
| VariantSearch | MAP | propose at suspicion | 3 | Shape, Store, overlay | Service |
| LOO | maintainer | detection rate of the corpus | 4 | Export, VariantSearch, pair rules | Batch |
| Lever | improve | admit/retract records | 5 | validator, gate | Batch |
### Shape
```python
GUARD_KINDS = ("null_test", "bounds_test", "allowlist_test", "auth_check", "parameterized_call", "type_change", "none")
SOURCE_KINDS = ("fact_source", "parameter", "container_read")
@dataclass(frozen=True)
class Shape:
language: str
sink_name: str # trailing callee segment
arity: int
source_positions: tuple[int, ...]
source_kinds: tuple[str, ...]
guard: str # GUARD_KINDS
mechanism: str # closed vocabulary id from the pair label
def key(self) -> str: ... # stable, text-free
def derive_shape(vuln: FileIR, fixed: FileIR, *, function: str, sink: str | None, line: int | None, mechanism: str, facts: SemanticFacts) -> Shape | None: ...
def match_shapes(ir: FileIR, shapes: Sequence[Shape], facts: SemanticFacts) -> list[VariantHit]: ...
```
Invariants: shapes carry no path, line, or literal beyond identifiers; `derive_shape` returns None when the labeled sink call site is not found.
### Store additions
`Mechanism` gains `origin: str = "sandbox"`, `review_tier: str = ""`, `pairs: tuple[str, ...] = ()`, `shape: dict[str, object] | None = None`, `guard: str = "none"`. `append_from_pair(store, shape, *, summary, cwe, pair, provenance, tier)` dedupes on `shape.key()` and extends `pairs`. Old rows load with defaults.
### VariantSearch
```python
@dataclass(frozen=True)
class VariantHit:
path: str
line: int
mechanism_id: str
sink_name: str
source_kind: str
def search_tree(root: Path, targets: Sequence[FileTarget], store: MechanismStore, facts: SemanticFacts, *, max_mechanisms: int) -> list[VariantHit]: ...
def hits_to_findings(hits, records: Sequence[OverlayRecord]) -> tuple[list[StaticFinding], list[OverlayRecord]]: ... # merge with overlay flows
```
Finding: id `variant:<mechanism_id>:<path>:<line>`, evidence `suspicion`, tags `["variant", "mechanism:<id>"]`, rationale names the mechanism summary and pair provenance.
### LOO
```python
@dataclass(frozen=True)
class LooOutcome:
pair: str
detected: bool
silent: bool
found_by: tuple[str, ...] # mechanism ids (and their pairs) that hit inside the labeled function
def evaluate_loo(cases: Sequence[PairCase], *, facts: SemanticFacts) -> LooResult: ... # per slice/profile/mechanism metrics
```
Only `seeded | reviewed` pairs seed; every trusted pair is held out once. `known_limit` pairs are excluded.
### Lever
`MechanismEdit(lever="mechanisms", action="admit"|"retract", mechanism_id, source="loo"|"export", rationale)`. Validator: id must exist in the exporter's candidate set; shape guard in `GUARD_KINDS`; no free text fields beyond rationale. Proposer: admit candidates whose leave-one-out hit recovers a currently missed holdout pair in some profile; retract admitted records that leak on holdout fixed sides. Gate unchanged; revert restores `mechanisms.jsonl` byte for byte.
## Data Models
Store row additions: `origin`, `review_tier`, `pairs`, `shape`, `guard`. Manifest: `"variants": {"mechanisms_searched": n, "files_searched": m, "findings": k, "merged_into_overlay": j}`. LOO artifact `loo.json`: `per_slice`, `per_profile`, `per_mechanism`, `outcomes[]` with `found_by`.
## Error Handling
| Condition | Response |
|-----------|----------|
| Extra absent for a language | shape derivation and search skipped for that language; degradation `variants_language_unsupported` |
| Labeled sink not found in vuln IR | pair skipped from export with a reason in the export report |
| Store row lacks `shape` (sandbox origin) | not used for search; still used for prove ordering |
| Admission leaks on holdout | round rejected, store reverted |
## Testing Strategy
- Unit: derive_shape on a Python pair (request → `os.system`, fix adds allowlist) yields sink `system`, arity 1, source position 0, guard `allowlist_test`; match_shapes finds a variant in another file and ignores constant arguments; shape has no path or literal; `<anon>` sinks resolve to enclosing named function for LOO scoring.
- Integration: export over the sast slice writes deduped rows with `origin=corpus`; variant search on the split-sink Python fixture emits a `variant` finding merged into the overlay coverage record; LOO on a three-pair toy slice reports which pair taught which.
- Improve: admitting a record that recovers a holdout pair is accepted; one that leaks is rejected with byte-identical store.
- Gates: detection, map, local pair gates unchanged; extra-free green with semantic tests skipping.
## Security Considerations
Shapes carry no code text; store rows are data. Variant findings never execute anything; sandbox candidates follow the existing safety check.
## Performance
Search is O(call sites × shapes) per file with a name index on `sink_name`; bounded by MAP budget and `max_mechanisms`. LOO parses each excerpt once and reuses IR.
## Migration Strategy
1. Store fields and exporter; export the sast and vibe-py trusted pairs; record counts.
2. Variant search behind `[variants].enabled` (default true in standard); manifest counters.
3. LOO command and roadmap table.
4. Lever and proposer.
.kiro/specs/corpus-seeded-mechanisms/requirements.md
+93
View File
@@ -0,0 +1,93 @@
# Requirements Document
## Introduction
OpenUltraSAST's labeled pair corpus measures the detector but does not improve it, and its mechanism memory is empty because only sandbox-proven findings may write to it. This spec turns every trusted pair into a mechanism record, adds a structural variant search that proposes matches on any scanned tree, measures the corpus's own detection rate leave-one-out, and gives the improve loop a lever that can raise recall under the existing false-positive ceiling and per-profile holdout gate.
## Boundary Context
- **In scope**: mechanism export from pairs; shape normalization per language; variant search proposer in MAP; leave-one-out evaluation; `mechanisms` improve lever; store schema additions; reports showing mechanism ids; offline tests.
- **Out of scope**: taint, facts, walker, dispositions, evidence rungs; embedding retrieval changes; inter-file flow; LLM-authored shapes; catalogs and scorer changes beyond reading `PairCase`; any merge gate.
- **Adjacent expectations**: `Mechanism` gains additive fields only; `order_promotions` keeps working on old records; the improve loop's existing clauses and the per-profile holdout clause apply unchanged; `local` remains the only slice that can fail CI.
## Requirements
### Requirement 1: Mechanisms seeded from trusted pairs
**Objective:** As a maintainer, I want each reviewed pair to become a mechanism record, so that the corpus is a library the detector can search, not only a ruler.
#### Acceptance Criteria
1. When a pair has `review_tier` `seeded` or `reviewed` and its vulnerable excerpt parses, the system shall derive one mechanism record carrying mechanism id, CWE, language, provenance, review tier, pair name, the sink shape, the source shape, and the guard shape the fixed side added.
2. The system shall not seed a mechanism from `advisory` or `title` pairs, from `known_limit` pairs, or from pairs whose vulnerable excerpt is `parse_failed`.
3. When the same shape is derived from several pairs, the system shall store one record and list every pair in its provenance.
4. Seeded records shall be written to the same append-only JSONL store as sandbox-proven mechanisms, tagged with `origin = "corpus"`, and the existing embedding cache shall remain rebuildable from it.
5. The system shall expose the export as a maintainer command that runs offline on the vendored catalog.
### Requirement 2: Shapes are structural and language-scoped
**Objective:** As an engineer, I want mechanism shapes that match code structure rather than text, so that variants across projects are found without regex growth.
#### Acceptance Criteria
1. A sink shape shall consist of the callee name (trailing segment), the argument count, and the argument positions that carry source-derived identifiers, derived from `FileIR` call sites.
2. A source shape shall be either a fact source id, a function parameter, or a container read, derived from the vulnerable excerpt's binds.
3. A guard shape shall be one of a closed set: `null_test`, `bounds_test`, `allowlist_test`, `auth_check`, `parameterized_call`, `type_change`, `none`, derived from statements present on the fixed side and absent on the vulnerable side.
4. Shapes shall never contain literal source text longer than an identifier; paths, line numbers, and secrets shall not appear in a shape.
5. When the semantic extra is absent for a language, the system shall skip shape derivation for that language with a recorded degradation and shall not fall back to regex.
### Requirement 3: Variant search proposes, it does not decide
**Objective:** As an AppSec engineer, I want a scanned tree searched for structural variants of known mechanisms, so that recall rises with the corpus while evidence stays honest.
#### Acceptance Criteria
1. When a standard scan runs and the mechanism store holds records for the file's language, the system shall search each parsed file's call sites for sink shapes whose source-position arguments carry a fact source, a parameter, or a container read.
2. A match shall produce a finding with origin `variant`, the mechanism id, evidence level `suspicion`, and the pair provenance in its rationale.
3. When the overlay already records a source-to-sink flow at the same call site, the overlay record shall carry the mechanism id and the variant finding shall be merged into it rather than duplicated.
4. Variant findings shall be eligible for the sandbox candidate set under the existing safety check and caps, ranked after overlay promotions.
5. The system shall bound variant search by the existing MAP file budget and shall record how many files and mechanisms were searched in the manifest.
6. Variant search shall never demote, never raise evidence above `suspicion` on its own, and shall be absent from `quick`.
### Requirement 4: Leave-one-out is the corpus's detection rate
**Objective:** As a maintainer, I want to know whether adding pairs raises recall, so that corpus growth is measured by what it teaches the detector.
#### Acceptance Criteria
1. When the maintainer runs the leave-one-out command on a slice, the system shall, for each trusted pair, seed a temporary store from every other trusted pair, run variant search on the held-out vulnerable and fixed excerpts, and score the pair with the pair scorer's rules.
2. The command shall report leave-one-out recall, silence on the fixed side, and Youden per slice, per provenance profile, and per mechanism id.
3. The report shall list, per held-out pair, which mechanism record found it, so that a maintainer can see which pairs teach and which only test.
4. Leave-one-out results shall be written as a JSON artifact and recorded in the roadmap table; they shall never be a merge condition.
### Requirement 5: A recall-raising lever in the improve loop
**Objective:** As a maintainer, I want the improve loop to be able to admit mechanisms, so that it can raise recall and not only shadow noisy rules.
#### Acceptance Criteria
1. The improve loop shall gain a `mechanisms` lever whose edits admit or retract a candidate mechanism record into the scan-time store.
2. The validator shall accept only records derived by the shape exporter from trusted pairs, with a closed guard kind and no literal text beyond identifiers; free-form patterns shall be rejected.
3. The proposer shall suggest admitting a candidate mechanism when it recovers a currently missed holdout pair in a profile, and retracting one when it leaks on holdout fixed sides.
4. When a `mechanisms` edit is evaluated, the existing accept gate and the per-profile holdout clause shall apply unchanged; rejection shall revert the store byte for byte.
5. The journal shall record each mechanism edit with its pair provenance and the metrics before and after.
### Requirement 6: Reports name the mechanism
**Objective:** As an operator, I want a finding to say which known mechanism it resembles and where that mechanism was learned, so that a fix can follow the known fix.
#### Acceptance Criteria
1. When a finding carries a mechanism id, the markdown and SARIF reports shall show the mechanism summary, the guard shape the known fix used, and the pair provenance.
2. The manifest shall record the number of variant findings and the mechanisms searched.
3. Reports shall label variant findings as suspicions unless the overlay or the sandbox raised them.
### Requirement 7: Offline tests and unchanged gates
**Objective:** As a maintainer, I want the feature proven without network, models, or a sandbox, so that it lands under the existing CI.
#### Acceptance Criteria
1. Shape derivation, variant matching, leave-one-out, and the lever shall be unit-tested on local fixtures; the semantic-marked tests shall skip without the extra.
2. The stage-1 detection gate, the split-sink map gate, and the local pair gate shall be unchanged.
3. The extra-free suite shall stay green.
.kiro/specs/corpus-seeded-mechanisms/research.md
+85
View File
@@ -0,0 +1,85 @@
# Research & Design Decisions
## Summary
- **Feature**: `corpus-seeded-mechanisms`
- **Discovery Scope**: Extension of mechanism memory, MAP stage, pair eval, and the improve loop
- **Key Findings**:
- `semantic/mechanisms.py` already has `Mechanism`, `MechanismStore` (append-only JSONL), `append_mechanism`, embedding cache, hard filters by language/CWE/tags, and `order_promotions`. It is written only by `cli._append_proven_mechanisms` after a sandbox trigger, so it is empty on every real tree.
- Clearwing's recall lever is the variant loop: a proven mechanism becomes a structural search over the messy tree. OpenUltraSAST copied the store and the retrieval, not the loop.
- `FileIR` call sites already carry callee name, argument texts, argument identifier lists, and constant flags; binds carry source text and names. That is enough for a shape without new parsing.
- The pair scorer (`pair-corpus-honesty`) gives labeled function, mechanism id, provenance, and soon a review tier; the fixed excerpt gives the guard the fix added by diffing statement kinds.
- The improve loop has bounded levers with a validator, journal, byte-for-byte revert, and now a per-profile holdout clause; a `mechanisms` lever fits without changing the gate.
- Measured 2026-09-05: vibe-py 13/35, vfc-js 1/16, sast 4/10 achievable. Web classes repeat structurally (request-derived string into `exec`/`execute`, identity from the body, `os.path.join` with a parameter, permissive defaults), which is what a shape search catches.
## Research Log
### Mechanism store today
- **Context**: Can the existing store carry corpus-derived records?
- **Sources Consulted**: `semantic/mechanisms.py`, `cli.py` `_append_proven_mechanisms`, `propose-adjudicate-prove` design §Mechanism.
- **Findings**: `Mechanism` fields are summary, cwe, language, tags, keywords, what_made_it_exploitable, source_finding_id, source_repo. Identity is the pattern. JSONL is authoritative; vectors are a rebuildable cache keyed by id and model.
- **Implications**: add `origin` (`sandbox` | `corpus`), `review_tier`, `pairs`, `shape` (structured), `guard` as additive fields with defaults so old rows load. A second writer (`append_from_pair`) beside the sandbox writer.
### Shape derivation from FileIR
- **Context**: Avoid regex and LLM authoring.
- **Sources Consulted**: `semantic/ir.py` (`CallSite.arg_names`, `arg_is_constant`, `Bind.names`, `FunctionIR.params`), `semantic/taint.py` (`_source_in`, parameter seeding).
- **Findings**: for the labeled function, the sink call site is the call whose trailing name equals the label's sink or whose line equals the label's line; source positions are argument indexes whose identifiers trace by binds to a fact source, a parameter, or a container read. Guard kind comes from statements on the fixed side absent on the vulnerable side: `if` with `== NULL`/`is None`/`!x` → `null_test`; comparison with `<`, `>`, `len(` → `bounds_test`; membership in a literal collection or `startswith`/regex → `allowlist_test`; identity read from an auth context (`request.state`, `req.user`, `session`) → `auth_check`; parameterized second argument → `parameterized_call`; changed declared type → `type_change`.
- **Implications**: `semantic/variants.py` implements `derive_shape(pair_ir_vuln, pair_ir_fixed, label)` and `match_shape(file_ir, shape)`. Guard classification is heuristic and closed; unknown → `none`.
### Where variant search sits
- **Context**: Propose, adjudicate, prove roles must stay separate.
- **Sources Consulted**: `three-stage-scan` design (MAP stage, hunter suspicion), `propose-adjudicate-prove` (overlay records, coverage), `regress/candidate.py`.
- **Findings**: MAP already runs overlay and the tool hunter; variant search is a third proposer that reads `FileIR` and the store. Findings at `suspicion`, origin `variant`; when the overlay has a flow at that call site, merge (attach mechanism id to the overlay record). Candidate eligibility follows the hunter-eligibility path being added in `reachability-flow-model` or the sev-5 forced-hotspot path today.
- **Implications**: no disposition change; one new finding origin; manifest counters.
### Leave-one-out
- **Context**: Prove that corpus growth raises recall, honestly.
- **Findings**: pair eval materializes each side into a temp tree already; a leave-one-out runner seeds a temp store from the other trusted pairs and runs variant search on both sides, scoring with the pair scorer's detection and leak rules.
- **Implications**: `pairs.py` gains a read-only consumer path (`evaluate_loo`) or a sibling module `semantic/loo.py` to keep the scorer's boundary; report per slice, profile, mechanism; artifact only.
## Architecture Pattern Evaluation
| Option | Description | Strengths | Risks / Limitations | Notes |
|--------|-------------|-----------|---------------------|-------|
| Embedding retrieval only | Rank existing promotions by mechanism similarity | Already exists | Cannot propose anything new; store empty | Rejected as recall lever |
| LLM writes Semgrep rules from pairs | Model authors patterns | Flexible | Unbounded pattern text, unverifiable, violates no-model-authored-patterns | Rejected |
| Structural shapes from FileIR plus variant search | Closed shape vocabulary, deterministic match | Zero-dep, bounded, testable leave-one-out | Shapes coarse; guard heuristics | **Selected** |
| Fine-tuned classifier on pairs | ML detector | Learns anything | Opaque, no fix guidance, data too small | Rejected |
## Design Decisions
### Decision: Seed only from seeded and reviewed tiers
- **Context**: `title` pairs are unreviewed guesses; `advisory` pairs have correct fixes but unlabeled mechanisms.
- **Selected Approach**: export reads `review_tier`; `advisory`/`title` may be scored leave-one-out as held-out targets but never seed.
- **Trade-offs**: fewer seeds at first; honest about what the detector was taught.
### Decision: Variant is a proposer at suspicion
- **Context**: Evidence honesty.
- **Selected Approach**: origin `variant`, `suspicion`; merge with overlay flow when present; sandbox may raise.
- **Trade-offs**: variant-only hits do not fail CI; correct.
### Decision: Lever admits records, not shapes
- **Context**: Keep the validator closed.
- **Selected Approach**: `mechanisms` lever edits reference record ids produced by the exporter; free-form shapes are rejected.
## Risks & Mitigations
- Coarse shapes over-match (`exec(x)` everywhere) — source-position requirement plus fact/parameter/container check; report leak rate on fixed sides leave-one-out; the per-profile gate rejects admissions that leak.
- Guard heuristics misclassify — closed set with `none`; reported, not used for demotion.
- Store growth — dedupe by shape; one record per shape with pair provenance list.
- Leave-one-out cost — one temp store per pair over tens of pairs; fine; cache parsed IR per excerpt.
## References
- `semantic/mechanisms.py`, `semantic/ir.py`, `semantic/taint.py`
- `.kiro/specs/propose-adjudicate-prove/design.md` §Mechanism (Clearwing reference architecture)
- `.kiro/specs/pair-corpus-honesty/` (review tiers, pointer slices, scorer rules)
- `.kiro/specs/harnessx-self-improving-rulesets/design.md` (bounded levers, accept gate)
- Clearwing sourcehunt mechanism memory and variant loop (reference, not a dependency)
.kiro/specs/corpus-seeded-mechanisms/spec.json
+22
View File
@@ -0,0 +1,22 @@
{
"feature_name": "corpus-seeded-mechanisms",
"created_at": "2026-09-05T06:18:31Z",
"updated_at": "2026-09-05T06:18:31Z",
"language": "en",
"phase": "tasks-generated",
"approvals": {
"requirements": {
"generated": true,
"approved": false
},
"design": {
"generated": true,
"approved": false
},
"tasks": {
"generated": true,
"approved": false
}
},
"ready_for_implementation": false
}
.kiro/specs/corpus-seeded-mechanisms/tasks.md
+84
View File
@@ -0,0 +1,84 @@
# Implementation Plan
- [ ] 1. Foundation: shapes and store fields
- [ ] 1.1 Define structural shapes and derive them from a pair
- `Shape` with language, sink name, arity, source positions and kinds, closed guard kind, mechanism id; `derive_shape` finds the labeled sink call site in the vulnerable `FileIR`, traces source positions through binds to a fact source, parameter, or container read, and classifies the guard from statements present only on the fixed side.
- Shapes contain no path, line, or literal beyond identifiers.
- Observable: a Python pair (request input into `os.system`, fix adds an allowlist) yields sink `system`, arity 1, source position 0, guard `allowlist_test`; an unlabeled sink returns None; the shape key is stable across runs.
- _Requirements: 2.1, 2.2, 2.3, 2.4_
- _Boundary: Shape_
- [ ] 1.2 Add additive mechanism fields and a corpus writer with dedupe
- `Mechanism` gains origin, review_tier, pairs, shape, guard with defaults; `append_from_pair` dedupes by shape key and extends provenance; old JSONL rows load unchanged.
- Observable: appending the same shape from two pairs yields one row listing both pairs; a pre-existing sandbox row loads with `origin = "sandbox"`.
- _Requirements: 1.3, 1.4_
- _Boundary: Store_
- _Depends: 1.1_
- [ ] 2. Core: export and search
- [ ] 2.1 Export mechanisms from trusted pairs as a maintainer command
- Only `seeded` and `reviewed`, non-known-limit, parse-ok pairs seed; the command runs offline over the vendored catalog and reports skipped pairs with reasons; extra-absent languages are skipped with a degradation.
- Observable: `ousast mechanisms export --slice sast` writes corpus rows and reports the skipped `title`/`advisory`/known-limit pairs.
- _Requirements: 1.1, 1.2, 1.5, 2.5_
- _Boundary: Export_
- _Depends: 1.2_
- [ ] 2.2 Search a tree for variants and merge with overlay flows
- For each parsed file within the MAP budget, call sites matching a shape's sink name, arity, and source positions (fact source, parameter, container read) produce `variant` findings at `suspicion` with the mechanism id; when the overlay has a flow at the same call site, its record gains `mechanism_id` and no duplicate finding is emitted.
- Observable: the split-sink Python fixture yields a variant finding merged into the overlay coverage record; a constant-argument call does not match; `quick` emits no variants.
- _Requirements: 3.1, 3.2, 3.3, 3.6_
- _Boundary: VariantSearch_
- _Depends: 2.1_
- [ ] 2.3 Make variant findings sandbox-eligible and count them in the manifest
- Variant findings enter the candidate set after overlay promotions under the existing safety check and caps; the manifest records mechanisms searched, files searched, findings, and merges.
- Observable: with the fake sandbox a variant finding runs; the manifest `variants` block is present on standard scans and absent on quick.
- _Requirements: 3.4, 3.5, 6.2_
- _Depends: 2.2_
- [ ] 3. Measurement: leave-one-out
- [ ] 3.1 Evaluate the corpus leave-one-out
- For each trusted pair, seed a temporary store from the others, search both excerpts, and score detection and silence with the pair rules; report per slice, profile, mechanism, and per pair which mechanism found it; write `loo.json`.
- Observable: `ousast pairs --slice vibe-py --loo --json` reports recall, silence, Youden per slice/profile/mechanism and `found_by` per pair; the run is offline.
- _Requirements: 4.1, 4.2, 4.3, 4.4_
- _Boundary: LOO_
- _Depends: 2.2_
- [ ] 3.2 Record the leave-one-out baseline
- Roadmap gains a leave-one-out table per slice next to the pair baselines; not a gate.
- Observable: roadmap table present; gates unchanged.
- _Requirements: 4.4, 7.2_
- _Depends: 3.1_
- [ ] 4. Integration: improve lever
- [ ] 4.1 Add the mechanisms lever with a closed validator
- `MechanismEdit` admit/retract by record id from the exporter's candidate set; validator rejects free-form shapes and unknown guard kinds; journal records pair provenance and metrics.
- Observable: a free-form edit is rejected; an admit edit of an exported record passes validation and appears in the journal.
- _Requirements: 5.1, 5.2, 5.5_
- _Boundary: Lever_
- _Depends: 2.1_
- [ ] 4.2 Propose admissions from leave-one-out misses and retractions from leaks under the existing gate
- Admit candidates that recover a currently missed holdout pair; retract admitted records that leak on holdout fixed sides; existing accept gate and per-profile clause decide; rejection reverts the store byte for byte.
- Observable: a scripted candidate that recovers a holdout pair is accepted and persisted; one that leaks on a fixed side is rejected and the store is unchanged.
- _Requirements: 5.3, 5.4_
- _Depends: 4.1, 3.1_
- [ ] 5. Integration: reports
- [ ] 5.1 Show mechanism id, known fix guard, and provenance on findings
- Markdown and SARIF show mechanism summary, guard shape, and pair provenance for findings carrying a mechanism id; variant findings are labeled suspicions unless raised.
- Observable: a standard scan report on the split-sink fixture names the mechanism and its guard; SARIF properties carry the id.
- _Requirements: 6.1, 6.3_
- _Depends: 2.2_
- [ ] 6. Validation
- [ ] 6.1 Prove offline tests and unchanged gates
- Unit and integration tests run without network, model, or sandbox; semantic-marked tests skip without the extra; detection, map, and local pair gates unchanged; extra-free suite green.
- Observable: both pytest matrices green; gate entrypoints unchanged.
- _Requirements: 7.1, 7.2, 7.3_
- _Depends: 2.3, 3.1, 4.2, 5.1_
## Implementation Notes
- Seeding reads `PairCase.review_tier` from `pair-corpus-honesty` task 8.1; until it lands, sast and vibe-py rows default to `advisory`/`seeded` per that design and only vibe-py seeds.
- Guard classification is heuristic and closed; `none` is allowed and reported, never used to demote.
.kiro/specs/reachability-flow-model/brief.md
+56
View File
@@ -0,0 +1,56 @@
# Brief: reachability-flow-model
## Problem
The tool's stated principle is to steer effort where risk is actually reduced. Today it cannot, because it has no model of how data or control reaches a sink:
- "Reachable" means the finding's line sits inside a route or CLI handler (`mapping.analyze_entry_points`). There is no call graph. On real code, sinks live in helpers several calls below the handler, so nearly everything is `unknown`, and `worth_fixing` (triggerable plus reachable) rarely fires.
- The overlay's taint is intra-file and one hop. A flow from a request parameter to a memcpy three functions away is invisible, and `unadjudicated` carries no partial path.
- Every stage handoff discards information. LLM hunter findings are capped at suspicion and cannot become sandbox candidates (`cli.py:404`, `prove_filter.py`); prove sees only regex-proposed, overlay-promoted ids.
- The hunter reads whole files and returns JSON. That is the most expensive and least verifiable use of a model.
- The improve loop's only levers are rule status and policy constants. It cannot add a sink, a sanitizer, a nullable callee, or a transformer, which are the only data that move recall.
Codebase shape confirms it: detection logic is about 16 percent of the source; harness plumbing, governance, and ranking are the rest, built around an engine that cannot express a guard or a call edge.
## Desired Outcome
A path model: entry point, hops, sink, guards seen, summaries used. Reachability becomes graph distance with a concrete path. The hunter works on paths and asks one checkable question per hole the static engine cannot fill. Answers become facts, gated on the pair corpus, reused at zero token cost. The report's first section is fix points: the few places whose fix collapses the most reachable risk.
True concolic execution is out of scope. The sandbox is the concrete half; the hunter proposing an input for a path and the sandbox running it is the loop this spec designs for, without a solver.
## Approach
Build a name-resolved call graph from the `FunctionIR` records tree-sitter already produces. Compute per-function taint summaries with the existing intra-file engine and compose them along the graph to a bounded depth. Attach graph reachability to findings, rank, map, and worth-fixing. Give the hunter a `path_context` tool and let a hunter finding with a locatable sink become a sandbox candidate after the safety check. Add a fact oracle: closed question kinds the static engine asks a model about a callee, cached as candidate facts with provenance, promoted only through the improve loop's new `facts` lever under the per-profile holdout gate from `pair-corpus-honesty`. Compute fix points as dominators on the path graph and report risk removed per fix.
## Scope
- **In**: call graph, summaries, path records, reachability attachment, rank and map integration, hunter on paths, hunter-to-candidate path, fact oracle with cache, `facts` lever with validator, fix-point ranking and report section, measurement on split-sink, vibe-py, vfc slices, offline tests.
- **Out**: guard dominance semantics (`guard-dominance-regime` consumes path records), CST walker fixes (`overlay-ir-completeness`), a whole-program points-to or symbolic engine, concolic execution, executing target code outside the sandbox, new dispositions, LLM as parser, any merge gate change.
## Boundary Candidates
- Call graph and summaries (semantic) vs entry points (mapping) vs rank and map consumers
- Fact oracle (asks and caches) vs facts lever (accepts) vs fact loader (consumes)
- Hunter path context vs regression candidate selection
## Out of Boundary
- Overlay dispositions and evidence rungs (`propose-adjudicate-prove`)
- CWE policy, project score, rule status lever (`harnessx-self-improving-rulesets`)
- Sandbox flags, recipes, verdict vocabulary (`three-stage-scan`)
- Pair scorer and label schema (`pair-corpus-honesty`)
## Upstream / Downstream
- **Upstream**: `FileIR`/`FunctionIR`, `analyze_entry_points`, `analyze_ir`, `load_facts`, `Hotspot`, `run_tool_hunter`, `select_candidates`, `run_improvement`, pair holdout metrics per profile.
- **Downstream**: `guard-dominance-regime` attaches guards to path records; later web-authorization detection is a path whose hops include no auth guard; reports and MCP read fix points.
## Existing Spec Touchpoints
- **Extends**: `three-stage-scan` (hunter tools, candidate selection, worth-fixing input), `propose-adjudicate-prove` (facts consumed unchanged; one new fact source), `harnessx-self-improving-rulesets` (one new lever under the same gate).
- **Depends**: `pair-corpus-honesty` for per-profile holdout metrics and the hunter scorer.
- **Adjacent**: `overlay-ir-completeness` improves the records this spec walks; neither blocks the other.
## Constraints
Core `dependencies = []`. Graph depth and fan-out bounded by config. Missing edges yield `unknown`, never `safe`. Model answers never bypass the gate; hand-written facts always win over candidate facts. No target code runs outside the sandbox. Youden on non-local slices stays off the merge gate.
.kiro/specs/reachability-flow-model/design.md
+412
View File
@@ -0,0 +1,412 @@
# Design Document
## Overview
**Purpose.** This feature gives OpenUltraSAST a path model: how an entry point reaches a sink, through which functions, with which summaries. It turns reachability into graph distance, puts the hunter on paths, lets model answers become gated facts, and ranks fix points by reachable risk removed.
**Users.** AppSec engineers read paths and fix points. DevOps gets worth-fixing on real reachability. Maintainers get an improve loop that can accept facts. The hunter gets a path and one question instead of a file.
**Impact.** New `semantic/callgraph.py`, `semantic/summaries.py`, `semantic/oracle.py`, `risk/fixpoints.py`. `mapping` attaches graph reachability. `rank` and `complexity` consume distance and path count. `tool_hunter` gains `path_context`. `regress/candidate.py` accepts hunter findings with a locatable sink. `improve` gains a `facts` lever. Reports gain a fix-first section. No disposition, gate, or sandbox flag changes.
### Goals
- Path records from entry to sink across files, bounded, over-approximate, never inferring safety.
- Reachability, rank, map, and worth-fixing use graph distance.
- Hunter on paths; hunter findings can reach the sandbox; evidence rises on checks.
- Model answers become candidate facts, accepted only through the gated lever.
- Fix points with risk removed, reported first.
### Non-Goals
- Guard semantics (next spec), CST fixes (adjacent spec), alias or symbolic analysis, concolic execution, LLM parsing, evidence stamp changes for existing sources, new gates.
## Boundary Commitments
### This Spec Owns
- Call graph, summaries, path records, and their artifacts.
- Graph reachability attachment and the rank and map signal additions.
- `path_context` hunter tool and hunter-candidate eligibility.
- Fact oracle, candidate and accepted fact stores, loader precedence, `facts` lever and its validator rules.
- Fix-point computation and report section.
### Out of Boundary
- `OverlayRecord` fields, dispositions, evidence rungs.
- Rule status and policy levers, CWE policy, project score.
- Sandbox runner, recipes, verdict vocabulary, safety check.
- Pair scorer, catalog schema.
### Allowed Dependencies
- `FileIR`, `FunctionIR`, `CallSite` plus one additive `imports` field on `FileIR`.
- `analyze_ir`, `load_facts`, `SemanticFacts`.
- `EntryPointRecord`, `attach_reachability_hints`, `heuristic_rank`, `build_complexity_map`, `Hotspot`.
- `run_tool_hunter`, `HunterTools`, `select_candidates`, `_hotspot_from_forced_finding`, safety check.
- `run_improvement`, validator, journal, per-profile holdout metrics.
- `OpenRouterChatClient` or HarnessX provider through existing seams; `redaction`.
### Revalidation Triggers
- Changing `FunctionIR` or `CallSite` fields beyond additive.
- Any consumer treating `unknown` distance as unreachable or safe.
- Loading candidate facts without acceptance.
- Adding a question kind.
- Fix points influencing any gate.
## Architecture
### Existing Architecture Analysis
| Existing element | Today | After |
|---|---|---|
| `mapping.attach_reachability_hints` | line-in-handler hints | plus graph distance and witness path |
| `rank.heuristic_rank` reachability term | tags and hints integer | graph distance when available |
| `complexity.build_complexity_map` | size, nesting, tags, density, tests | plus path count, min distance |
| `tool_hunter.run_tool_hunter` | read, grep, find_refs over a hotspot file | plus `path_context`; path-scoped prompt |
| `regress.select_candidates` | hotspots with promoted inventory ids, sev-5 forced | plus hunter findings with locatable sink |
| `improve` levers | rule, policy | plus facts |
| `load_facts` | one directory | hand-written, then accepted candidate facts |
| reports | inventory, map, worth-fixing | plus fix first |
### Architecture Pattern & Boundary Map
Selected: **graph over existing IR, summaries from existing taint, closed oracle behind the existing gate.**
```mermaid
flowchart TB
IR[FileIR per file + imports] --> CG[call graph]
EP[entry points] --> RE[reachability: distance, witness path]
CG --> RE
IR --> SUM[function summaries via analyze_ir]
FACTS[hand facts + accepted facts] --> SUM
SUM --> PATHS[composed path records]
CG --> PATHS
RE --> FIND[findings reachability]
RE --> RANK[rank]
PATHS --> MAP[complexity map signals]
PATHS --> HUNT[hunter path_context]
PATHS --> Q[open questions]
Q --> ORACLE[fact oracle]
ORACLE --> CAND[candidate_facts.toml]
CAND --> LEVER[improve facts lever + gate]
LEVER --> ACC[accepted_facts.toml]
ACC --> FACTS
HUNT --> CANDS[regress candidates]
PATHS --> FIX[fix points]
FIX --> REPORT[report: fix first]
```
**Key decisions**
- Edges over-approximate; distance `unknown` is never safe.
- Summaries are pure functions of IR and facts, cached by hash.
- Evidence for hunter findings rises only on path agreement or sandbox trigger.
- Candidate facts never load until accepted; hand facts win.
- Fix points are reported, never gated.
### Technology Stack
| Layer | Choice / Version | Role | Notes |
|-------|------------------|------|-------|
| Graph | stdlib dicts, BFS | call graph, distance | no networkx |
| Summaries | existing `analyze_ir` | per-function facts | cached by file hash |
| Oracle | existing chat client seams | closed questions | optional, offline-skippable |
| Stores | TOML in `.openultrasast/calibration/` | candidate and accepted facts | same dir as ledgers |
| Reports | existing markdown, manifest, SARIF | fix first section | additive |
## File Structure Plan
```
src/openultrasast/
├── semantic/
│ ├── ir.py # FileIR.imports: tuple[str, ...] = () (additive); cst/ast fill it
│ ├── callgraph.py # build_call_graph(irs) -> CallGraph; resolve; reachability(entry_points, depth)
│ ├── summaries.py # summarize(function, facts) -> Summary; compose(graph, summaries, depth) -> paths
│ ├── paths.py # PathRecord, write_paths, load_paths
│ ├── oracle.py # open_questions(paths, graph) ; ask(questions, client) ; candidate store io
│ └── facts.py # load_facts(dirs...) precedence: hand > accepted
├── risk/
│ └── fixpoints.py # dominators over path graph; risk removed; artifact
├── mapping.py # attach_graph_reachability(targets, graph, distances)
├── rank.py # reachability from distance when present
├── complexity/signals.py# path_count, min_distance signals
├── tool_hunter.py # path_context tool; path-scoped prompt when paths exist
├── regress/candidate.py # hunter findings with locatable sink -> forced hotspot after safety check
├── improve/validator.py # FactEdit lever "facts"; closed kinds; source must be candidate cache or benchmark signal
├── improve/evolve.py # facts lever in run_round; same gate + per-profile clause
├── reports.py # fix-first section; manifest.paths, manifest.open_questions
├── config.py # [reach] depth=4, max_fanout=32, oracle_enabled, max_questions
└── cli.py # wiring in MAP; artifacts callgraph.json, paths.json, fixpoints.json
tests/
├── test_callgraph.py, test_summaries.py, test_paths.py, test_oracle.py, test_fixpoints.py
├── test_tool_hunter.py (path_context), test_regress_candidates.py (hunter eligibility), test_improve.py (facts lever)
└── test_map_gate.py (path assertion)
```
### Modified Files
- `semantic/cst.py`, `semantic/ir.py` — collect import statements into `FileIR.imports`; no other IR change.
- `mapping.py` — new function; `EntryPointRecord` unchanged.
- `stage_processors.py` — MAP stage runs graph, summaries, paths before overlay; overlay unchanged.
- `map_gate.py` — path assertion per language family.
## System Flows
### Path construction
```mermaid
flowchart LR
A[parse files -> FileIR] --> B[build_call_graph]
B --> C[reachability from entry points, depth bound]
A --> D[summarize each function]
D --> E[compose along graph]
E --> F[PathRecord list]
F --> G[open questions at unresolved hops]
```
Composition rule: for a caller hop with tainted argument i to callee c, if `summary(c).param_to_sink[i]` is non-empty, extend the path; if `summary(c).param_to_return[i]`, the call result is tainted in the caller; if c is unresolved or beyond depth, mark `incomplete_at` and stop. No demotion is ever produced by composition.
### Hunter on a path
```mermaid
sequenceDiagram
participant H as tool hunter
participant P as path_context
participant S as sandbox
H->>P: path id
P-->>H: entry, hops, sink, summaries, guards, open questions
H->>H: verdict + proposed input (suspicion)
H->>S: snippet (after safety check, deep only)
S-->>H: triggerable / not
Note over H: evidence rises on trigger or on path agreement
```
### Facts lever
```mermaid
flowchart LR
Q[open questions] --> O[oracle asks model]
O --> C[candidate_facts.toml]
C --> L[improve: FactEdit accept]
L --> G{existing gate + per-profile holdout}
G -->|pass| A[accepted_facts.toml]
G -->|fail| R[revert byte for byte]
A --> F[load_facts precedence: hand > accepted]
```
## Requirements Traceability
| Requirement | Summary | Components | Flows |
|-------------|---------|------------|-------|
| 1.1–1.6 | call graph, unresolved, distance, over-approx, extra-free, artifact | CallGraph | path construction |
| 2.1–2.5 | summaries, composition, depth, unresolved hop, path records | Summaries, Paths | path construction |
| 3.1–3.6 | reachability, rank, map, worth-fixing, map gate | ReachabilityAttach, rank, complexity | — |
| 4.1–4.5 | path_context, path prompt, candidate eligibility, evidence, budget | HunterPaths, candidate | hunter on a path |
| 5.1–5.6 | question kinds, when asked, cache, precedence, no exec, no model | FactOracle, facts loader | facts lever |
| 6.1–6.5 | facts lever, validator, gate, journal, hand facts win | FactsLever | facts lever |
| 7.1–7.5 | fix points, risk removed, report, sev-5 invariant, artifact | FixPoints, reports | — |
| 8.1–8.5 | split-sink path check, labeled-function distance, gates unchanged, offline | map_gate, pairs payload, tests | — |
## Components and Interfaces
| Component | Domain | Intent | Req | Dependencies | Contracts |
|-----------|--------|--------|-----|--------------|-----------|
| CallGraph | semantic | resolve calls, distances | 1 | FileIR P0, entry points P0 | Service, State |
| Summaries | semantic | per-function taint facts, composition | 2 | analyze_ir P0, facts P0 | Service |
| Paths | semantic | path records artifact | 2.5 | Summaries P0 | State |
| ReachabilityAttach | mapping | distance on findings, rank, map | 3 | CallGraph P0 | Service |
| HunterPaths | hunter | path_context, path prompt, eligibility | 4 | Paths P0, tool_hunter P0, candidate P0 | Service |
| FactOracle | semantic | closed questions, candidate cache | 5 | chat client P1, redaction P0 | Batch, State |
| FactsLever | improve | accept or reject candidate facts | 6 | validator P0, evolve P0, holdout metrics P0 | Batch |
| FixPoints | risk | dominators, risk removed, report | 7 | Paths P0, policy P0 | Batch, State |
### Semantic
#### CallGraph
```python
@dataclass(frozen=True)
class CallEdge:
caller: str # "path::function"
callee: str | None # resolved node id or None
call_line: int
resolution: str # exact | import | ambiguous | unresolved
@dataclass(frozen=True)
class CallGraph:
nodes: tuple[str, ...]
edges: tuple[CallEdge, ...]
unresolved: int
def build_call_graph(irs: Sequence[FileIR]) -> CallGraph: ...
def reachability(graph: CallGraph, entry_nodes: Sequence[str], *, max_depth: int) -> dict[str, tuple[int, tuple[str, ...]]]: ...
```
Resolution order: same file definition; definition in an imported module (by `FileIR.imports` and path stem); any definition with that trailing name (ambiguous, one edge each, capped by `max_fanout`); else unresolved. Method calls resolve by trailing name. C resolves by bare name across the tree.
Invariants: never drops a call; `unresolved` counted; distance absent means `unknown`.
#### Summaries and Paths
```python
@dataclass(frozen=True)
class Summary:
node: str
param_to_sink: dict[int, tuple[tuple[str, int], ...]] # param index -> (sink id, line)
param_to_return: tuple[int, ...]
constant_params: tuple[int, ...]
facts_version: str
source_hash: str
@dataclass(frozen=True)
class PathRecord:
path_id: str
entry: str
hops: tuple[tuple[str, int], ...] # (node, call line)
sink: str
sink_line: int
sink_node: str
cwe: str
incomplete_at: str | None
summaries_used: tuple[str, ...]
open_questions: tuple[str, ...]
def summarize(function: FunctionIR, node: str, facts: SemanticFacts) -> Summary: ...
def compose(graph: CallGraph, summaries: Mapping[str, Summary], entry_nodes: Sequence[str], *, max_depth: int) -> tuple[PathRecord, ...]: ...
```
Preconditions: summaries for every node in the graph, or `incomplete_at` on missing.
Postconditions: no `PathRecord` implies safety; incomplete paths are kept.
### Mapping and ranking
#### ReachabilityAttach
`attach_graph_reachability(targets, graph, distances)` adds to each target's reachability hints an entry `{"kind": "graph", "function_name", "distance", "witness"}`. `findings._reachability_for_line` treats a graph hint whose function contains the line as `reachable`. Rank: `reachability = max(existing, 5 - min(distance, 4))`, rationale records `graph_distance`. Map signals: `path_count`, `min_distance`.
### Hunter
#### HunterPaths
Tool `path_context(path_id) -> {entry, hops, sink, sink_line, summaries, guards: [], open_questions}`. Guards list is empty until `guard-dominance-regime`. When paths exist for a hotspot, the prompt names one path and asks for: verdict, reasoning tied to hops, proposed input. Findings carry `path_id` in tags. Eligibility: a hunter finding whose `line` matches a `PathRecord.sink_line` or an overlay sink line becomes a forced hotspot via `_hotspot_from_forced_finding`, ranked after overlay promotions, subject to `max_candidates` and the safety check. Evidence stays `suspicion`; a sandbox trigger sets the verdict as today; a path agreement is recorded in `reachability_evidence`.
### Oracle and facts
#### FactOracle
```python
QUESTION_KINDS = ("nullable_return", "bounded_argument", "transformer_set_get", "returns_parameter_taint", "sanitizes_parameter")
@dataclass(frozen=True)
class Question:
kind: str
callee: str
language: str
context_hash: str
context: str # redacted callee text or signature only
@dataclass(frozen=True)
class CandidateFact:
kind: str
callee: str
language: str
answer: dict[str, object] # structured per kind
model: str
context_hash: str
asked_at: str
```
Batch contract: trigger after composition when a model is configured; input open questions capped by `max_questions`; output appended to `candidate_facts.toml`; idempotent by `(kind, callee, context_hash)`. Never executes code; context passes through `redaction`.
`load_facts(directory, accepted: Path | None)` merges accepted facts after hand-written facts; on id collision the hand-written row wins and a degradation is recorded.
Answer shapes: `nullable_return: {nullable: bool}`, `bounded_argument: {arg: int, bounded_by: int | "constant"}`, `transformer_set_get: {set: str, get: str, key_args: [int], value_arg: int}`, `returns_parameter_taint: {params: [int]}`, `sanitizes_parameter: {params: [int], cwe: str}`.
### Improve
#### FactsLever
```python
@dataclass(frozen=True)
class FactEdit:
lever: str = "facts"
action: str # accept | retract
fact_key: str # kind:callee:context_hash
source: str # candidate_cache | benchmark_signal
rationale: str = ""
```
Validator: kind in `QUESTION_KINDS`; `fact_key` present in the candidate cache; `retract` only for accepted candidate facts, never hand-written; no pattern text. `run_round` proposes accepts for candidate facts that touch open questions on missed pairs; the existing gate and the per-profile holdout clause decide; revert restores `accepted_facts.toml` byte for byte; journal records before and after metrics.
### Risk
#### FixPoints
```python
@dataclass(frozen=True)
class FixPoint:
node: str
kind: str # function | entry
paths_collapsed: int
risk_removed: float
witness_paths: tuple[str, ...]
def compute_fix_points(paths: Sequence[PathRecord], severity_by_cwe: Mapping[str, int], evidence_weight: Mapping[str, float]) -> tuple[FixPoint, ...]: ...
```
A node dominates a path if every route from that path's entry to its sink passes through the node. Risk removed sums `severity × weight(evidence)` over dominated paths; weight defaults: sandbox-proven 1.0, path-complete 0.6, incomplete 0.3. Sorted by risk removed. Reported in markdown and manifest; the worth-fixing candidate set is unchanged.
## Data Models
Artifacts under `.openultrasast/runs/<id>/`: `callgraph.json`, `paths.json`, `fixpoints.json`. Manifest additions:
```json
{
"reach": {"nodes": 412, "edges": 1180, "unresolved": 96, "max_depth": 4, "entry_nodes": 9},
"paths": {"complete": 14, "incomplete": 31},
"open_questions": 12,
"fix_first": [{"node": "api/runs.py::create_run", "paths_collapsed": 6, "risk_removed": 24.0}]
}
```
Calibration stores: `.openultrasast/calibration/candidate_facts.toml`, `accepted_facts.toml`. Both `[[fact]]` tables with the `CandidateFact` fields plus `accepted_at` on the latter.
## Error Handling
| Condition | Response |
|-----------|----------|
| Extra absent, non-Python file | node exists with no calls; edges from it unresolved; `graph_unavailable` degradation per language |
| Parse failed file | node absent; callers' edges unresolved |
| Depth or fan-out exceeded | path `incomplete_at` with reason `depth` or `fanout` |
| No entry points | distances empty; findings keep today's hints; fix points empty |
| Model absent | open questions recorded; oracle skipped with degradation |
| Model answer malformed | dropped; question stays open |
| Accepted fact collides with hand fact | hand fact wins; degradation `fact_shadowed` |
| Hunter snippet fails safety check | inconclusive as today |
## Testing Strategy
- Unit: graph resolution exact, import, ambiguous, unresolved; distance and witness on a small tree; summary of a function with param-to-sink and param-to-return; composition across two files; depth stop is incomplete not safe; fix points on a diamond graph; oracle idempotency; validator rejects pattern text and hand-fact retraction; loader precedence.
- Integration: split-sink Python fixture yields a complete path from the route to the planted sink; map gate path assertion passes; a scripted hunter using `path_context` produces a finding that becomes a forced candidate and runs under the fake sandbox; `run_improvement` with a scripted candidate fact accepts when holdout improves and reverts when a profile regresses.
- Gates: detection gate, local pair gate, existing split-sink assertions unchanged; extra-free suite green with graph assertions skipping.
- No default test needs a model, sandbox, or network.
## Security Considerations
Graph and summaries are static reads. Oracle context is redacted and limited to the callee signature and body; no secrets, no execution. Hunter snippets run only in the sandbox after the existing safety check. Candidate facts are data and cannot introduce pattern text.
## Performance & Scalability
Graph is O(functions + calls). Summaries cached by file hash. Composition bounded by depth 4 and fan-out 32 by default. Oracle bounded by `max_questions` per run. Fix points run over path records, not statements.
## Migration Strategy
1. Graph and distances as artifacts only; nothing consumes them. Baseline recorded.
2. Reachability attachment and rank term; verify split-sink map gate still passes and now asserts a path.
3. Summaries and path records; pairs payload reports labeled-function distance.
4. Hunter path context and candidate eligibility behind config; default on for standard when a model is set.
5. Oracle and facts lever; default `oracle_enabled = false` until the first accepted facts are reviewed.
6. Fix points and report section.
Rollback per phase is config off; artifacts are additive.
.kiro/specs/reachability-flow-model/requirements.md
+112
View File
@@ -0,0 +1,112 @@
# Requirements Document
## Introduction
OpenUltraSAST cannot yet steer effort where risk is reduced because it has no model of how an entry point reaches a sink. Reachability today means "inside a handler," taint is one hop inside one file, the LLM hunter cannot create a sandbox candidate, and the improve loop cannot add the facts that would raise recall. This spec adds a call graph and composed taint summaries over the existing `FunctionIR` records, attaches path-based reachability to findings and ranking, puts the hunter on paths with one checkable question per gap, turns model answers into gated facts, and ranks fix points by reachable risk removed. Concolic execution is out of scope; the sandbox remains the only place target code runs.
## Boundary Context
- **In scope**: call graph, taint summaries, path records, reachability attachment to findings, rank, complexity map and worth-fixing; hunter path context; hunter findings as sandbox candidates; fact oracle and candidate-fact cache; `facts` improve lever; fix-point ranking and report section; offline tests and measurement on existing slices.
- **Out of scope**: guard semantics; CST walker fixes; whole-program alias analysis; symbolic or concolic execution; new dispositions or evidence rungs; LLM as parser; any merge gate change; profile-specific fact tables beyond the gate defined by `pair-corpus-honesty`.
- **Adjacent expectations**: `FileIR` shape is frozen except additive fields; `load_facts` merges candidate facts below hand-written facts; the improve loop's existing accept gate and per-profile holdout clause apply unchanged to the new lever; sandbox safety checks apply to hunter snippets as today.
## Requirements
### Requirement 1: Call graph and graph reachability
**Objective:** As an AppSec engineer, I want to know whether an entry point actually reaches a function, so that a finding's reachability is a path and not a line-in-handler guess.
#### Acceptance Criteria
1. When a tree is mapped, the system shall build a call graph whose nodes are functions from `FileIR` records and whose edges are resolved call sites, across files, using name and import resolution.
2. When a call site cannot be resolved, the system shall record it as an unresolved edge and shall not treat the callee as unreachable or safe.
3. When entry points exist, the system shall compute for each function the shortest distance from any entry point and one witness path, bounded by a configured maximum depth.
4. The system shall prefer over-approximation: an ambiguous name resolving to several definitions yields edges to each, marked ambiguous.
5. If the semantic extra is absent, then the system shall build the graph for Python from the stdlib parse and shall mark other languages `graph_unavailable` without failing the scan.
6. The system shall write the graph and distances as a run artifact without any target code execution.
### Requirement 2: Taint summaries composed along the graph
**Objective:** As an engineer, I want flows that cross function and file boundaries to appear as one path, so that a request parameter reaching a sink three calls down is visible.
#### Acceptance Criteria
1. When a function is analyzed, the system shall produce a summary stating which parameters reach which sinks, which parameters reach the return value, and which parameters are fully constant-dominated, using the existing intra-file taint engine.
2. When a caller passes a tainted argument to a callee whose summary maps that parameter to a sink, the system shall record a path from the caller's source through the callee to the sink.
3. The system shall compose summaries to a configured depth and shall stop with `flow_incomplete` beyond it, carrying the partial path in the record.
4. When a hop's callee is unresolved, the system shall record the path as incomplete at that hop, never as safe.
5. The system shall emit path records with entry, ordered hops, sink, sink line, and the summary used at each hop.
### Requirement 3: Reachability drives rank, map, and worth-fixing
**Objective:** As a DevOps engineer, I want ranking and worth-fixing to use real reachability, so that budget and CI failures go to reachable risk.
#### Acceptance Criteria
1. When a finding lies in a function with a finite graph distance from an entry point, the system shall mark it `reachable` and attach the distance and witness path to its reachability evidence.
2. When a finding lies in a function with no path within the depth bound, the system shall mark it `unknown`, not `unreachable`.
3. The rank formula's reachability term shall use graph distance when available, falling back to today's hints otherwise, with the change recorded in the rationale.
4. The complexity map shall add path count and minimum distance as signals and shall not use them as the sole score.
5. The worth-fixing gate shall accept graph-reachable findings as reachable; the existing `inferred-file-surface` case stays.
6. The system shall keep the split-sink map gate passing and shall extend it with a path check defined in Requirement 8.
### Requirement 4: Hunter on paths
**Objective:** As an engineer, I want the hunter to work on a path and ask one checkable question, so that model spend goes to what static analysis cannot do.
#### Acceptance Criteria
1. When a hunter runs on a hotspot that has path records, the system shall provide a `path_context` tool returning the entry, hops, sink, summaries, and known guards for a chosen path.
2. The hunter prompt shall ask for a verdict on a specific path and a proposed input, not a whole-file review, when path records exist.
3. When a hunter finding names a sink line that the overlay or the graph can locate, the system shall make it eligible as a sandbox candidate after the existing safety check, without requiring an inventory promotion.
4. The system shall keep hunter findings at `suspicion` until an independent check agrees: a sandbox trigger, a composed path to that sink, or an accepted fact; evidence rises on that check, not on origin.
5. The hunter shall stay within the existing step and cost budgets; path context counts toward them.
### Requirement 5: Fact oracle with gated candidate facts
**Objective:** As a maintainer, I want the model to answer one question about a callee and have the answer reused as data, so that recall rises without hand-writing every fact and without paying tokens on every scan.
#### Acceptance Criteria
1. The system shall define a closed set of question kinds: `nullable_return`, `bounded_argument`, `transformer_set_get`, `returns_parameter_taint`, `sanitizes_parameter`.
2. When the static engine stops at an unresolved or unsummarized callee on a path, the system shall record a question for that callee; questions shall be asked only when a hunter or verifier model is configured.
3. When a model answers, the system shall store a candidate fact with the question, the answer, the model id, the source hash, and a timestamp, in a cache next to the calibration ledgers.
4. The system shall not load candidate facts into adjudication unless they have been accepted by the improve loop's `facts` lever; hand-written facts always take precedence over candidate facts.
5. The system shall not execute target code to answer a question and shall not send secrets; redaction applies to question context.
6. If no model is configured, then the system shall record the open questions in the manifest and proceed.
### Requirement 6: Facts as an improve-loop lever
**Objective:** As a maintainer, I want the improve loop to be able to accept or reject facts, so that the evolutionary machinery can move detection and not only rule status.
#### Acceptance Criteria
1. The improve loop shall gain a `facts` lever whose edits add or remove a candidate fact from the accepted set.
2. The validator shall accept only edits whose fact kind is in the closed set and whose source is the candidate cache or a benchmark signal; free-form pattern text is rejected.
3. When a `facts` edit is evaluated, the existing accept gate and the per-profile holdout clause shall apply unchanged; rejection reverts byte for byte.
4. The journal shall record each fact edit with its provenance and the metrics before and after.
5. The system shall never remove or override a hand-written fact through this lever.
### Requirement 7: Fix points ranked by reachable risk removed
**Objective:** As an operator, I want to know which few fixes remove the most reachable risk, so that I fix the worst things first.
#### Acceptance Criteria
1. When path records exist, the system shall compute fix points as functions or guard sites that dominate multiple paths in the path graph.
2. Each fix point shall carry an estimate of risk removed: the sum over dominated paths of policy severity weighted by evidence, and the number of paths collapsed.
3. The report shall gain a "fix first" section listing fix points in order of risk removed, each with its witness paths.
4. The system shall not let fix-point ranking hide a reachable severity-5 finding from the worth-fixing candidate set.
5. The fix-point list shall be written as a run artifact and exposed to reports without executing target code.
### Requirement 8: Measurement and unchanged gates
**Objective:** As a maintainer, I want the path model measured on the corpora we have, so that the shift is grounded.
#### Acceptance Criteria
1. On the split-sink fixtures, the system shall report whether a path exists from a planted entry point to each planted function, and the map gate shall assert at least one such path per language family.
2. On slices whose labels name a function, the pair payload shall report the fraction of labeled functions that have a finite graph distance.
3. The stage-1 detection gate, the local pair gate, and the existing split-sink gate assertions shall be unchanged.
4. Default tests shall not require a model, a sandbox, or network; oracle and hunter tests use scripted clients.
5. The extra-free suite shall stay green with graph-dependent assertions skipping when the extra is absent.
.kiro/specs/reachability-flow-model/research.md
+113
View File
@@ -0,0 +1,113 @@
# Research & Design Decisions
## Summary
- **Feature**: `reachability-flow-model`
- **Discovery Scope**: Complex Integration across semantic, mapping, rank, complexity, hunter, regress, improve
- **Key Findings**:
- `FunctionIR` already carries every call site per function for Python, JS, TS, C, and Java when the extra is present. A name-resolved call graph is a few hundred lines over data that exists.
- Reachability today is "line inside an entry-point handler" (`mapping.attach_reachability_hints`); `rank` bumps an integer from tags and hints; `worth_fixing` needs `reachable` or `inferred-file-surface`. No call edge exists anywhere.
- Sandbox candidates come only from hotspots filtered to overlay-promoted inventory ids (`cli.py:404`, `prove_filter.filter_promoted_hotspots`); a hunter snippet is used only for an already promoted hotspot (`candidate.py:199`). The LLM plane cannot create a candidate.
- Evidence is stamped by origin: regex hit `static_corroboration` (`findings.py:110`), hunter `suspicion` (`tool_hunter.py:408`).
- Improve levers are `rule` and `policy` only (`improve/validator.py`); facts are not a lever. `propose-adjudicate-prove` excluded auto-authored facts.
- Code shape: detection logic 1,910 lines of 12,128; harness plumbing 2,885; governance and benchmarks 3,178.
- Clearwing fills reachability from a tree-sitter callgraph and lets taint paths boost surface; the embeddings retrieve mechanisms, not decide taint. That split is the reference.
- Literature: RealVuln shows LLM-based scanners at 51.7 to 73 F3 on real Python web code versus Semgrep at 17.7; CVE-Bench shows agents exploit at most 13 percent of real web CVEs. Recall on real code comes from models with tools; precision and reproducibility come from static checks and execution.
## Research Log
### Where a call graph can come from
- **Context**: Avoid a new parser or a whole-program engine.
- **Sources Consulted**: `semantic/ir.py`, `semantic/cst.py`, `mapping.py`, `preprocess.py`, `hunter_tools.find_refs`.
- **Findings**: `FileIR.functions[*].calls[*].name` is the callee text. Python stdlib parse yields the same records. Imports are not in the IR; the walker can add an `imports` tuple per file cheaply. `find_refs` is a word-boundary regex over files and is the only cross-file lookup today.
- **Implications**: `semantic/callgraph.py` consumes `FileIR` plus an additive `imports` field. Resolution order: same-file definition, imported module definition, any definition with that name (ambiguous edges). Method calls resolve by trailing name. C resolves by bare name across the tree.
### Summaries versus whole-program taint
- **Context**: `propose-adjudicate-prove` deferred inter-file taint as too large and risky for false demotes.
- **Sources Consulted**: `semantic/taint.py` (`_analyze_function` seeds parameters as tainted, tracks names, emits `TaintPath` and `Demotion`), overlay invariants.
- **Findings**: Because parameters are already seeded tainted, a per-function run already answers "which parameter reaches which sink." Adding "which parameter reaches the return" is a small extension. Composition is then a graph walk with a depth bound. Nothing needs alias analysis to produce a useful path; anything unresolved stays `flow_incomplete`.
- **Implications**: Summaries are a pure function of `FunctionIR` plus facts, cacheable by file hash. Composition never demotes; it only adds paths or marks incompleteness.
### Hunter to candidate
- **Context**: The LLM plane is the recall lever the field data supports, but it is walled off from prove.
- **Sources Consulted**: `tool_hunter.py`, `regress/candidate.py`, `three-stage-scan` design (hunter snippets run only in the sandbox after safety check), `verification.py`.
- **Findings**: The safety check and sandbox are origin-agnostic. The wall is `filter_promoted_hotspots` and the candidate selector's reliance on inventory ids. A hunter finding with a locatable sink line can be given a synthetic hotspot the same way `_hotspot_from_forced_finding` already does for severity-5 inventory.
- **Implications**: Add eligibility for hunter findings that name a sink line resolvable to a path record or an overlay sink. Evidence stays suspicion until sandbox trigger or path agreement.
### Facts from a model, safely
- **Context**: The engine lacks callee knowledge: nullable returns (`tls1_lookup_sigalg`), bounded arguments, transformer set/get (`ConfigParser`), macro out-params (`n2s`).
- **Sources Consulted**: `semantic/facts.py` loader, `improve/validator.py`, `improve/journal.py`, `calibration.py` ledger patterns, `redaction.py`.
- **Findings**: Facts are versioned TOML loaded by `load_facts`. The loader can merge a second directory at lower precedence. The improve loop has a strict validator and byte-for-byte revert. Redaction exists for prompts.
- **Implications**: Model answers are candidate facts in `.openultrasast/calibration/candidate_facts.toml`; accepted facts in `accepted_facts.toml`; loader precedence hand-written > accepted > none. The `facts` lever moves rows between candidate and accepted under the same gate. No pattern text is ever model-authored; question kinds are closed.
### Fix points
- **Context**: The principle is risk reduced per fix, not findings per file.
- **Sources Consulted**: `complexity/map.py`, `complexity/ledger.py`, `regress/verdict.py`, `scoring/project_score.py`.
- **Findings**: The path graph is small (functions, not statements). Dominators over a DAG of entry-to-sink paths are cheap. Severity is available per CWE from policy.
- **Implications**: `risk/fixpoints.py` computes dominator sets per sink path and aggregates risk removed. It is a report and artifact, not a gate; it cannot hide severity-5 reachable findings.
## Architecture Pattern Evaluation
| Option | Description | Strengths | Risks / Limitations | Notes |
|--------|-------------|-----------|---------------------|-------|
| Joern CPG as required engine | Full inter-procedural dataflow | Real dataflow | Scala runtime, not zero-dep, slow on PRs | Rejected; stays optional sidecar |
| CodeQL databases | Strong inter-procedural | Mature | Build step, license limits, heavy | Rejected as required; SARIF ingest remains |
| Whole-program taint in Python | Own engine | Full control | Months of work, alias analysis, false demotes | Rejected |
| Name-resolved call graph plus composed summaries | Graph over existing IR; summaries from existing taint | Days to build; over-approximate by design; feeds hunter and fix points | Ambiguous names, dynamic dispatch | **Selected** |
| LLM as reachability oracle | Ask the model who calls what | No code | Unverifiable, expensive per scan | Rejected; model answers closed callee questions only |
## Design Decisions
### Decision: Over-approximate edges, never infer safety
- **Context**: Missing edges silently hide risk; false edges cost hunter budget.
- **Selected Approach**: Ambiguous names produce edges to every definition, flagged. Unresolved calls produce an unresolved edge. Distance is infinite only within the depth bound and is reported as `unknown`.
- **Rationale**: Consistent with the fail-closed rule in the overlay.
- **Trade-offs**: More hunter work on large trees; bounded by existing hotspot caps.
### Decision: Summaries are pure and cached
- **Context**: Cost on large repos.
- **Selected Approach**: Summary keyed by file hash plus facts version; recomputed only when either changes.
- **Trade-offs**: Cache invalidation on facts version bump; acceptable.
### Decision: Evidence rises on checks, not on origin
- **Context**: Regex stamped corroboration, hunter stamped suspicion.
- **Selected Approach**: This spec does not change existing stamps. It adds the checks that raise hunter evidence: composed path agreement or sandbox trigger. A follow-up may lower regex-only hits to inventory; recorded as a revalidation trigger for `propose-adjudicate-prove`.
- **Rationale**: Keep this spec additive; changing existing stamps changes gate baselines.
### Decision: Closed question kinds, gated acceptance
- **Context**: Auto-facts were excluded for good reason: unbounded model-authored patterns.
- **Selected Approach**: Five question kinds, structured answers, candidate cache with provenance, acceptance only via the `facts` lever under the per-profile holdout gate.
- **Trade-offs**: Slower than trusting answers directly; the gate is the point.
### Decision: Fix points are dominators, reported not gated
- **Context**: Do not add a seventh evidence rung or a new CI condition.
- **Selected Approach**: Compute and report; the severity-5 reachable invariant from `three-stage-scan` stays.
## Risks & Mitigations
- Dynamic dispatch and callbacks in JS and Python produce unresolved edges — report unresolved counts; hunter `find_refs` remains available as a fallback.
- Graph blow-up on large trees — depth and fan-out bounds in config; hotspot caps unchanged.
- Model answers wrong — never used unless accepted by the gate; provenance recorded; hand facts win.
- Hunter candidates increase sandbox load — existing `max_candidates` cap; hunter candidates ranked after overlay promotions unless a path agrees.
- Fix-point estimates misread as guarantees — label as estimates; show witness paths.
## References
- Clearwing sourcehunt callgraph, taint, ranker (reference architecture; not a dependency)
- [RealVuln](https://arxiv.org/abs/2604.13764) — rule-based versus LLM scanners on real Python web code
- [CVE-Bench](https://arxiv.org/abs/2503.17332) — agent exploitation ceiling on real web CVEs
- `.kiro/specs/three-stage-scan/design.md` — hunter tools, candidate selection, worth-fixing
- `.kiro/specs/propose-adjudicate-prove/design.md` — facts as data, dispositions, evidence honesty
- `.kiro/specs/harnessx-self-improving-rulesets/design.md` — bounded levers, accept gate, revert
- `.kiro/specs/pair-corpus-honesty/` — per-profile holdout gate, hunter scorer
- Memory note `overlay-gaps-diagnosis-2026-09`
.kiro/specs/reachability-flow-model/spec.json
+22
View File
@@ -0,0 +1,22 @@
{
"feature_name": "reachability-flow-model",
"created_at": "2026-09-04T16:27:33Z",
"updated_at": "2026-09-04T16:27:33Z",
"language": "en",
"phase": "tasks-generated",
"approvals": {
"requirements": {
"generated": true,
"approved": false
},
"design": {
"generated": true,
"approved": false
},
"tasks": {
"generated": true,
"approved": false
}
},
"ready_for_implementation": false
}
.kiro/specs/reachability-flow-model/tasks.md
+125
View File
@@ -0,0 +1,125 @@
# Implementation Plan
- [ ] 1. Foundation: call graph and reachability
- [ ] 1.1 Collect imports into FileIR and build a name-resolved call graph
- `FileIR` gains an additive `imports` field filled by both the CST walker and the stdlib Python parse.
- Graph nodes are functions; edges resolve same-file, imported-module, ambiguous (one edge per definition, capped), or unresolved; no call is dropped.
- Observable: a three-file fixture yields exact, import, ambiguous, and unresolved edges with the expected counts; extra-free Python still builds a graph.
- _Requirements: 1.1, 1.2, 1.4, 1.5_
- _Boundary: CallGraph_
- [ ] 1.2 Compute bounded distance and witness paths from entry points
- BFS from entry-point functions to a configured depth; each reached node gets distance and one witness path; unreached nodes have no distance and are reported as unknown.
- Graph, distances, and unresolved count are written as a run artifact in MAP without executing target code.
- Observable: on the split-sink Python fixture the planted function has a finite distance and a witness path through the route; `callgraph.json` exists after a standard scan.
- _Requirements: 1.3, 1.6, 3.6_
- _Boundary: CallGraph_
- _Depends: 1.1_
- [ ] 2. Core: summaries and path records
- [ ] 2.1 Summarize each function with the existing taint engine
- Summary states parameter-to-sink, parameter-to-return, and constant-dominated parameters; keyed by source hash and facts version; cached.
- Observable: a function that forwards its parameter into `execute` reports param 0 to that sink; a function returning its parameter reports param 0 in param_to_return; a second run reuses the cache.
- _Requirements: 2.1_
- _Boundary: Summaries_
- _Depends: 1.1_
- [ ] 2.2 Compose summaries along the graph into path records
- Tainted arguments extend a path into a callee whose summary reaches a sink; return taint flows back to the caller; unresolved callees or depth exhaustion mark the path incomplete at that hop; composition never demotes.
- Path records carry entry, hops, sink, sink line, summaries used, and open questions; written as `paths.json`.
- Observable: a route in one file calling a helper in another that executes a query yields one complete path; cutting the helper's definition yields an incomplete path, not silence.
- _Requirements: 2.2, 2.3, 2.4, 2.5_
- _Boundary: Summaries, Paths_
- _Depends: 2.1, 1.2_
- [ ] 3. Integration: reachability consumers
- [ ] 3.1 Attach graph reachability to findings, rank, and the complexity map
- A finding inside a function with finite distance is reachable with distance and witness in its evidence; no distance stays unknown; rank uses distance when present and records it; map gains path count and minimum distance signals without using them as the sole score.
- Observable: a sink in a helper two calls below a route becomes reachable; rank rationale shows graph_distance; the map order on split-sink still differs from hit-count order.
- _Requirements: 3.1, 3.2, 3.3, 3.4_
- _Boundary: ReachabilityAttach_
- _Depends: 1.2_
- [ ] 3.2 Feed worth-fixing and extend the map gate with a path check
- Worth-fixing accepts graph-reachable findings; the split-sink map gate additionally asserts a path from a planted entry to a planted function per language family.
- Observable: a triggerable fake-sandbox verdict on a graph-reachable helper is worth fixing; map gate passes with the new assertion; existing assertions unchanged.
- _Requirements: 3.5, 3.6, 8.1, 8.3_
- _Depends: 3.1, 2.2_
- [ ] 4. Integration: hunter on paths
- [ ] 4.1 (P) Add the path_context tool and a path-scoped prompt
- The hunter can request one path by id and receives entry, hops, sink, summaries, guards (empty for now), and open questions; when paths exist the prompt asks for a verdict and a proposed input on that path.
- Path context counts toward the existing step and cost budgets.
- Observable: a scripted client that calls path_context and returns a finding yields a suspicion finding tagged with the path id; a client that ignores the path still works as today.
- _Requirements: 4.1, 4.2, 4.5_
- _Boundary: HunterPaths_
- _Depends: 2.2_
- [ ] 4.1b (P) Resolve the hunter client through HarnessX when the extra and `[harnessx]` provider are present
- `resolve_hunter_client(prefer_harnessx=..., harnessx_provider=..., model=...)` prefers a HarnessX provider adapter (its own credentials) and falls back to the OpenAI-compatible endpoint (`OPENROUTER_API_KEY` + `OPENROUTER_BASE_URL`, e.g. DeepSeek); the adapter must not use `asyncio.run` on non-coroutine awaitables.
- Observable: with the extra absent the resolver returns the OpenRouter client; with a scripted provider object the adapter yields a ChatResponse; the pairs `--hunter` scorer uses the same resolver.
- _Requirements: 4.5_
- _Boundary: HunterPaths_
- [ ] 4.2 Make hunter findings with a locatable sink eligible as sandbox candidates
- A hunter finding whose line matches a path sink line or an overlay sink line becomes a forced hotspot after the safety check, ranked after overlay promotions, within the candidate cap; evidence stays suspicion until a sandbox trigger or a path agreement is recorded.
- Observable: with the fake sandbox, a scripted hunter finding on a path sink runs and its verdict is recorded; a hunter finding with no locatable sink is not a candidate; evidence level is unchanged by eligibility alone.
- _Requirements: 4.3, 4.4_
- _Boundary: HunterPaths_
- _Depends: 4.1_
- [ ] 5. Integration: fact oracle and facts lever
- [ ] 5.1 (P) Record open questions and ask a model only when configured
- Composition records closed-kind questions at unresolved or unsummarized hops; when a model is configured the oracle asks with redacted callee context and appends candidate facts with model id, context hash, and timestamp; idempotent per key; no model means questions are listed in the manifest.
- Observable: a scripted client answering nullable_return produces one candidate fact; a second run does not duplicate it; with no client the manifest lists the open question; no target code runs.
- _Requirements: 5.1, 5.2, 5.3, 5.5, 5.6_
- _Boundary: FactOracle_
- _Depends: 2.2_
- [ ] 5.2 Load accepted facts below hand-written facts
- The fact loader merges an accepted-facts file after the hand-written directory; collisions keep the hand-written row and record a degradation; candidate facts are never loaded.
- Observable: an accepted nullable fact is visible to adjudication; the same id in a hand-written file wins; a candidate-only fact is invisible.
- _Requirements: 5.4, 6.5_
- _Boundary: FactOracle_
- _Depends: 5.1_
- [ ] 5.3 Add the facts lever to the improve loop under the existing gate
- A facts edit accepts or retracts a candidate fact; the validator allows only closed kinds sourced from the cache or a benchmark signal and never retracts hand-written facts; the existing accept gate and the per-profile holdout clause decide; rejection reverts the accepted file byte for byte; the journal records the edit and metrics.
- Observable: a scripted candidate fact that turns a missed pair into a pass on holdout is accepted and persisted; one that regresses a profile is rejected and the file is unchanged; a pattern-text edit is rejected by the validator.
- _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5_
- _Boundary: FactsLever_
- _Depends: 5.2_
- [ ] 6. Integration: fix points
- [ ] 6.1 Compute fix points as dominators and rank by risk removed
- A node dominates a path when every route from its entry to its sink passes through it; risk removed sums policy severity weighted by evidence over dominated paths; written as `fixpoints.json`.
- Observable: a diamond fixture where one helper sits on four paths ranks that helper first with four paths collapsed; a reachable severity-5 finding stays in the worth-fixing candidate set regardless of fix-point order.
- _Requirements: 7.1, 7.2, 7.4, 7.5_
- _Boundary: FixPoints_
- _Depends: 2.2_
- [ ] 6.2 Report fix first
- Markdown and manifest gain a fix-first section listing fix points with witness paths; labelled as estimates.
- Observable: a standard scan report on the split-sink fixture contains the section; `--fail-on` behavior is unchanged.
- _Requirements: 7.3_
- _Depends: 6.1_
- [ ] 7. Validation
- [ ] 7.1 Measure labeled-function reachability on pair slices
- For slices whose labels name a function, the pairs payload reports the fraction with finite distance; recorded in the roadmap next to the pair-corpus baseline.
- Observable: `pairs --slice vibe-py --json` includes the fraction; the roadmap has the number.
- _Requirements: 8.2_
- _Depends: 3.1_
- [ ] 7.2 Prove gates unchanged and default tests offline
- Detection gate, local pair gate, and existing split-sink assertions produce the same verdicts; extra-free suite green with graph assertions skipping; no default test uses a model, sandbox, or network.
- Observable: gate entrypoints unchanged; pytest green in both matrices.
- _Requirements: 8.3, 8.4, 8.5_
- _Depends: 3.2, 4.2, 5.3, 6.2_
## Implementation Notes
- Guards on path records stay empty until `guard-dominance-regime`; the tool and record shapes reserve the field so that spec is additive.
- `oracle_enabled` defaults to false until the first accepted facts have been reviewed by a maintainer; open questions are still recorded.
- Do not change evidence stamps of existing sources in this spec; log the regex-as-corroboration question as a revalidation trigger for `propose-adjudicate-prove`.
- Expect more sandbox candidates once hunter eligibility lands; the existing `max_candidates` cap and heuristic order hold cost.