mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-22 06:29:29 +00:00
docs(specs): kick off corpus-seeded-mechanisms and reachability-flow-model (unapproved)
Two engine specs drafted after the pair-corpus baseline, awaiting owner approval before implementation: - corpus-seeded-mechanisms: structural shapes derived from trusted pairs, export, variant search merged with overlay flows, leave-one-out measurement, and a closed `mechanisms` lever for the improve loop. - reachability-flow-model: call graph over FunctionIR, reachability from entry points to sinks, hunter client preferring HarnessX (task 4.1b). Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
co-authored by
Claude Fable 5.1
Claude Fable 5.1
parent
3eba6e10a0
commit
7d238f2400
@@ -0,0 +1,255 @@
|
||||
# Design Document
|
||||
|
||||
## Overview
|
||||
|
||||
**Purpose.** Turn trusted pairs into mechanism records, search scanned trees for structural variants, measure the corpus's own detection rate leave-one-out, and give the improve loop a recall-raising lever.
|
||||
|
||||
**Users.** Maintainers export and measure. AppSec engineers get variant findings that name the known mechanism and its known fix. The improve loop admits or retracts mechanisms under the existing gate.
|
||||
|
||||
**Impact.** `semantic/mechanisms.py` gains additive fields and a second writer. New `semantic/variants.py` (shapes and matching) and `semantic/loo.py` (leave-one-out). MAP runs variant search after overlay. `improve` gains the `mechanisms` lever. Reports show mechanism ids. No disposition, gate, or sandbox change.
|
||||
|
||||
### Goals
|
||||
|
||||
- One mechanism record per distinct shape, seeded from `seeded` and `reviewed` pairs.
|
||||
- Variant findings on real trees at `suspicion`, merged with overlay flows when present.
|
||||
- Leave-one-out recall, silence, and Youden per slice, profile, and mechanism.
|
||||
- `mechanisms` lever under the existing accept gate and per-profile holdout.
|
||||
|
||||
### Non-Goals
|
||||
|
||||
- LLM-authored shapes, regex growth, disposition or ladder changes, inter-file flow, embedding changes, new gates.
|
||||
|
||||
## Boundary Commitments
|
||||
|
||||
### This Spec Owns
|
||||
|
||||
- Shape derivation and matching (`variants.py`).
|
||||
- Additive mechanism fields, corpus writer, dedupe (`mechanisms.py`).
|
||||
- Variant proposer in MAP and manifest counters.
|
||||
- Leave-one-out runner and artifact (`loo.py`, `pairs --loo`).
|
||||
- `mechanisms` lever, its validator rules, proposer, journal fields.
|
||||
- Report rendering of mechanism ids.
|
||||
|
||||
### Out of Boundary
|
||||
|
||||
- `OverlayRecord` fields beyond an optional `mechanism_id`; dispositions; evidence rungs.
|
||||
- Catalog schema, scorer rules, harvest (`pair-corpus-honesty`).
|
||||
- Taint, facts, walker; call graph; hunter.
|
||||
- Rule and policy levers; ledger format.
|
||||
|
||||
### Allowed Dependencies
|
||||
|
||||
- `FileIR`, `parse_file`, `SemanticFacts`/`load_facts` (read), `adjudicate` output, `MechanismStore`, `PairCase`/`load_pair_catalog`/`select_slice`, pair scorer matching helpers (read), `run_round` gate, `EvolveValidator`, journal, reports.
|
||||
|
||||
### Revalidation Triggers
|
||||
|
||||
- Variant findings stamped above `suspicion` without overlay or sandbox agreement.
|
||||
- Seeding from `advisory` or `title` pairs.
|
||||
- Free-form shape text accepted by the validator.
|
||||
- Leave-one-out used as a gate.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
PAIRS[trusted pairs: seeded | reviewed] --> EXPORT[derive_shape per pair]
|
||||
EXPORT --> STORE[mechanisms.jsonl origin=corpus]
|
||||
STORE --> VS[variant search in MAP]
|
||||
IR[FileIR of scanned tree] --> VS
|
||||
FACTS[facts: sources] --> VS
|
||||
VS --> VF[variant findings: suspicion, mechanism_id]
|
||||
OV[overlay records] --> MERGE[merge when same call site has a flow]
|
||||
VF --> MERGE
|
||||
MERGE --> CAND[regress candidates after promotions]
|
||||
PAIRS --> LOO[leave-one-out runner]
|
||||
STORE -. temp store per held-out pair .-> LOO
|
||||
LOO --> ART[loo.json + roadmap table]
|
||||
LOO --> PROP[improve proposer: admit/retract]
|
||||
PROP --> GATE[accept gate + per-profile holdout]
|
||||
GATE --> STORE
|
||||
```
|
||||
|
||||
**Key decisions**
|
||||
|
||||
- Shapes are closed structures derived from `FileIR`, never text patterns.
|
||||
- Variant search proposes at `suspicion`; overlay or sandbox raises.
|
||||
- Leave-one-out is the corpus's detection rate and is reported only.
|
||||
- The lever edits record ids from the exporter; validator rejects anything else.
|
||||
|
||||
### Technology Stack
|
||||
|
||||
| Layer | Choice | Role | Notes |
|
||||
|-------|--------|------|-------|
|
||||
| Shapes | stdlib dataclasses over `FileIR` | derive, match | no new parser |
|
||||
| Store | existing JSONL + cache | corpus and sandbox records | additive fields |
|
||||
| Search | MAP stage | proposer | file budget as rank |
|
||||
| LOO | temp dirs + existing pair materialization | measurement | artifact only |
|
||||
| Improve | existing lever machinery | admit/retract | same gate |
|
||||
|
||||
## File Structure Plan
|
||||
|
||||
```
|
||||
src/openultrasast/
|
||||
├── semantic/
|
||||
│ ├── mechanisms.py # Mechanism += origin, review_tier, pairs, shape, guard; append_from_pair; dedupe_by_shape
|
||||
│ ├── variants.py # Shape, GuardKind, derive_shape(vuln_ir, fixed_ir, label, facts) -> Shape|None; match_shapes(file_ir, shapes, facts) -> list[VariantHit]
|
||||
│ └── loo.py # evaluate_loo(cases, *, store_factory) -> LooResult; payload
|
||||
├── findings.py # origin "variant" via tags/finding_id prefix "variant:" (no schema change)
|
||||
├── cli.py # MAP: variant search after overlay; manifest counters; `pairs --loo`; `mechanisms export --slice`
|
||||
├── improve/validator.py# MechanismEdit lever "mechanisms"
|
||||
├── improve/evolve.py # proposer from loo misses/leaks; gate unchanged
|
||||
└── reports.py # mechanism id, guard, provenance on findings
|
||||
tests/
|
||||
├── test_variants.py, test_loo.py, test_mechanism_export.py, test_improve.py (lever)
|
||||
```
|
||||
|
||||
### Modified Files
|
||||
|
||||
- `semantic/overlay.py` — `OverlayRecord.mechanism_id: str | None = None` (additive).
|
||||
- `regress/candidate.py` — variant findings eligible after overlay promotions (reuses forced-hotspot path).
|
||||
- `config.py` — `[variants] enabled=true, max_mechanisms=500`.
|
||||
|
||||
## System Flows
|
||||
|
||||
### Export
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A[load trusted pairs] --> B[parse vuln + fixed excerpts]
|
||||
B --> C[find labeled sink call site]
|
||||
C --> D[source positions via binds]
|
||||
B --> E[guard kind: statements only on fixed side]
|
||||
D --> F[Shape]
|
||||
E --> F
|
||||
F --> G[dedupe by shape; append with pairs provenance]
|
||||
```
|
||||
|
||||
### Variant search in MAP
|
||||
|
||||
For each parsed file within the MAP budget: for each call site whose trailing callee name equals a shape's sink name and whose arity matches, check the shape's source positions: the argument carries a fact source, a function parameter, or a container read (bind whose value is a subscript or `.get(`). On match emit a `variant:<mechanism_id>:<path>:<line>` finding at `suspicion`. If an overlay record exists at that path and line with a flow, set its `mechanism_id` and do not emit a separate finding.
|
||||
|
||||
### Leave-one-out
|
||||
|
||||
For each trusted pair P in the slice: temp store from all trusted pairs except P; materialize P's sides; run variant search on both; detection if a variant hit lies inside P's labeled function and names P's mechanism id or matches P's sink; leak if any hit on the fixed side. Aggregate per slice, profile, mechanism; record which mechanism found P.
|
||||
|
||||
## Requirements Traceability
|
||||
|
||||
| Requirement | Summary | Components | Flows |
|
||||
|-------------|---------|------------|-------|
|
||||
| 1.1–1.5 | seed from trusted pairs, dedupe, origin corpus, maintainer command | Export, Store | export |
|
||||
| 2.1–2.5 | structural shapes, closed guard kinds, no text, extra-absent degradation | Shape | export |
|
||||
| 3.1–3.6 | proposer at suspicion, merge, candidates, budget, absent from quick | VariantSearch | MAP |
|
||||
| 4.1–4.4 | leave-one-out per slice/profile/mechanism, artifact, not a gate | LOO | leave-one-out |
|
||||
| 5.1–5.5 | lever, validator, proposer, gate unchanged, journal | Lever | improve |
|
||||
| 6.1–6.3 | reports, manifest counters, suspicion label | reports | — |
|
||||
| 7.1–7.3 | offline tests, gates unchanged, extra-free green | tests | — |
|
||||
|
||||
## Components and Interfaces
|
||||
|
||||
| Component | Domain | Intent | Req | Dependencies | Contracts |
|
||||
|-----------|--------|--------|-----|--------------|-----------|
|
||||
| Shape | semantic | derive and compare shapes | 2 | FileIR, facts | Service |
|
||||
| Export | maintainer | pairs → records | 1 | PairCase, Shape, Store | Batch |
|
||||
| Store | semantic | additive fields, dedupe | 1.3, 1.4 | JSONL | State |
|
||||
| VariantSearch | MAP | propose at suspicion | 3 | Shape, Store, overlay | Service |
|
||||
| LOO | maintainer | detection rate of the corpus | 4 | Export, VariantSearch, pair rules | Batch |
|
||||
| Lever | improve | admit/retract records | 5 | validator, gate | Batch |
|
||||
|
||||
### Shape
|
||||
|
||||
```python
|
||||
GUARD_KINDS = ("null_test", "bounds_test", "allowlist_test", "auth_check", "parameterized_call", "type_change", "none")
|
||||
SOURCE_KINDS = ("fact_source", "parameter", "container_read")
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Shape:
|
||||
language: str
|
||||
sink_name: str # trailing callee segment
|
||||
arity: int
|
||||
source_positions: tuple[int, ...]
|
||||
source_kinds: tuple[str, ...]
|
||||
guard: str # GUARD_KINDS
|
||||
mechanism: str # closed vocabulary id from the pair label
|
||||
|
||||
def key(self) -> str: ... # stable, text-free
|
||||
|
||||
def derive_shape(vuln: FileIR, fixed: FileIR, *, function: str, sink: str | None, line: int | None, mechanism: str, facts: SemanticFacts) -> Shape | None: ...
|
||||
def match_shapes(ir: FileIR, shapes: Sequence[Shape], facts: SemanticFacts) -> list[VariantHit]: ...
|
||||
```
|
||||
|
||||
Invariants: shapes carry no path, line, or literal beyond identifiers; `derive_shape` returns None when the labeled sink call site is not found.
|
||||
|
||||
### Store additions
|
||||
|
||||
`Mechanism` gains `origin: str = "sandbox"`, `review_tier: str = ""`, `pairs: tuple[str, ...] = ()`, `shape: dict[str, object] | None = None`, `guard: str = "none"`. `append_from_pair(store, shape, *, summary, cwe, pair, provenance, tier)` dedupes on `shape.key()` and extends `pairs`. Old rows load with defaults.
|
||||
|
||||
### VariantSearch
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class VariantHit:
|
||||
path: str
|
||||
line: int
|
||||
mechanism_id: str
|
||||
sink_name: str
|
||||
source_kind: str
|
||||
|
||||
def search_tree(root: Path, targets: Sequence[FileTarget], store: MechanismStore, facts: SemanticFacts, *, max_mechanisms: int) -> list[VariantHit]: ...
|
||||
def hits_to_findings(hits, records: Sequence[OverlayRecord]) -> tuple[list[StaticFinding], list[OverlayRecord]]: ... # merge with overlay flows
|
||||
```
|
||||
|
||||
Finding: id `variant:<mechanism_id>:<path>:<line>`, evidence `suspicion`, tags `["variant", "mechanism:<id>"]`, rationale names the mechanism summary and pair provenance.
|
||||
|
||||
### LOO
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class LooOutcome:
|
||||
pair: str
|
||||
detected: bool
|
||||
silent: bool
|
||||
found_by: tuple[str, ...] # mechanism ids (and their pairs) that hit inside the labeled function
|
||||
|
||||
def evaluate_loo(cases: Sequence[PairCase], *, facts: SemanticFacts) -> LooResult: ... # per slice/profile/mechanism metrics
|
||||
```
|
||||
|
||||
Only `seeded | reviewed` pairs seed; every trusted pair is held out once. `known_limit` pairs are excluded.
|
||||
|
||||
### Lever
|
||||
|
||||
`MechanismEdit(lever="mechanisms", action="admit"|"retract", mechanism_id, source="loo"|"export", rationale)`. Validator: id must exist in the exporter's candidate set; shape guard in `GUARD_KINDS`; no free text fields beyond rationale. Proposer: admit candidates whose leave-one-out hit recovers a currently missed holdout pair in some profile; retract admitted records that leak on holdout fixed sides. Gate unchanged; revert restores `mechanisms.jsonl` byte for byte.
|
||||
|
||||
## Data Models
|
||||
|
||||
Store row additions: `origin`, `review_tier`, `pairs`, `shape`, `guard`. Manifest: `"variants": {"mechanisms_searched": n, "files_searched": m, "findings": k, "merged_into_overlay": j}`. LOO artifact `loo.json`: `per_slice`, `per_profile`, `per_mechanism`, `outcomes[]` with `found_by`.
|
||||
|
||||
## Error Handling
|
||||
|
||||
| Condition | Response |
|
||||
|-----------|----------|
|
||||
| Extra absent for a language | shape derivation and search skipped for that language; degradation `variants_language_unsupported` |
|
||||
| Labeled sink not found in vuln IR | pair skipped from export with a reason in the export report |
|
||||
| Store row lacks `shape` (sandbox origin) | not used for search; still used for prove ordering |
|
||||
| Admission leaks on holdout | round rejected, store reverted |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- Unit: derive_shape on a Python pair (request → `os.system`, fix adds allowlist) yields sink `system`, arity 1, source position 0, guard `allowlist_test`; match_shapes finds a variant in another file and ignores constant arguments; shape has no path or literal; `<anon>` sinks resolve to enclosing named function for LOO scoring.
|
||||
- Integration: export over the sast slice writes deduped rows with `origin=corpus`; variant search on the split-sink Python fixture emits a `variant` finding merged into the overlay coverage record; LOO on a three-pair toy slice reports which pair taught which.
|
||||
- Improve: admitting a record that recovers a holdout pair is accepted; one that leaks is rejected with byte-identical store.
|
||||
- Gates: detection, map, local pair gates unchanged; extra-free green with semantic tests skipping.
|
||||
|
||||
## Security Considerations
|
||||
|
||||
Shapes carry no code text; store rows are data. Variant findings never execute anything; sandbox candidates follow the existing safety check.
|
||||
|
||||
## Performance
|
||||
|
||||
Search is O(call sites × shapes) per file with a name index on `sink_name`; bounded by MAP budget and `max_mechanisms`. LOO parses each excerpt once and reuses IR.
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
1. Store fields and exporter; export the sast and vibe-py trusted pairs; record counts.
|
||||
2. Variant search behind `[variants].enabled` (default true in standard); manifest counters.
|
||||
3. LOO command and roadmap table.
|
||||
4. Lever and proposer.
|
||||
Reference in New Issue
Block a user