mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-21 22:19:29 +00:00
docs(specs): kick off corpus-seeded-mechanisms and reachability-flow-model (unapproved)
Two engine specs drafted after the pair-corpus baseline, awaiting owner approval before implementation: - corpus-seeded-mechanisms: structural shapes derived from trusted pairs, export, variant search merged with overlay flows, leave-one-out measurement, and a closed `mechanisms` lever for the improve loop. - reachability-flow-model: call graph over FunctionIR, reachability from entry points to sinks, hunter client preferring HarnessX (task 4.1b). Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
co-authored by
Claude Fable 5.1
Claude Fable 5.1
parent
3eba6e10a0
commit
7d238f2400
@@ -0,0 +1,48 @@
|
||||
# Brief: corpus-seeded-mechanisms
|
||||
|
||||
## Problem
|
||||
|
||||
The pair corpus is only a ruler. `pair-corpus-honesty` made it honest and is growing it past the license and review walls, but nothing in the tool learns from a labeled pair: the detector on an operator's tree is the same regex inventory plus intra-file taint whether the corpus holds 11 pairs or 1,100. The mechanism store that Clearwing uses as a library (`semantic/mechanisms.py`, JSONL plus an embedding cache) exists here but is filled only from sandbox-proven findings, so it is empty in practice and only reorders prove candidates.
|
||||
|
||||
Measured 2026-09-05: vibe-py 13/35 pair-correct, vfc-js 1/16, sast 4/10 achievable, vfc 0/176. Recall on real web code is where the tool is weakest, and web bug classes repeat structurally across projects (identity from the request body, `exec` of a request-derived string, path joined from a parameter, RLS or CORS left open).
|
||||
|
||||
## Desired Outcome
|
||||
|
||||
Every `seeded` or `reviewed` pair becomes a mechanism record: the source shape, the sink statement shape, and the guard the fix added. A structural variant search runs those shapes over any scanned tree and proposes matches, which the overlay and the sandbox then verify. Leave-one-out recall on the corpus is the corpus's own detection rate and rises as the corpus grows. The improve loop gains a lever that admits mechanisms, so it can raise recall instead of only defending it.
|
||||
|
||||
Principle written into the roadmap: **recall first under a false-positive ceiling, per provenance profile, and every reviewed pair becomes a searchable mechanism.**
|
||||
|
||||
## Approach
|
||||
|
||||
Derive a mechanism record from a pair by parsing the vulnerable excerpt with the existing `FileIR`, taking the sink call site the label names, the identifiers that carry the source, and the guard statements that appear only on the fixed side. Normalize into a language-scoped shape: sink callee name, argument arity, which argument positions carry source-derived identifiers, and the guard kind. Variant search walks a scanned tree's `FileIR`, finds call sites with the same shape whose arguments carry a fact source or a parameter, and emits `variant` findings at `suspicion` carrying the mechanism id; when the overlay already has a flow to that sink, the record is `coverage` with the mechanism id attached. Leave-one-out evaluation seeds from all pairs but one and scores the held-out pair. The improve loop proposes admitting a candidate mechanism when it recovers a missed holdout pair; the existing accept gate and the per-profile holdout clause decide.
|
||||
|
||||
## Scope
|
||||
|
||||
- **In**: mechanism export from pairs; shape normalization; variant search as a proposer in MAP; leave-one-out measurement command and payload; `mechanisms` improve lever with validator rules; store hygiene (tier, provenance, dedupe, version); offline tests.
|
||||
- **Out**: LLM-authored shapes; changing dispositions or evidence rungs; embedding retrieval changes (existing prove-order reuse stays); inter-file flow (reachability spec); any merge gate; vendoring more pairs (corpus spec).
|
||||
|
||||
## Boundary Candidates
|
||||
|
||||
- Shape extraction (reads FileIR) vs overlay dispositions (unchanged)
|
||||
- Mechanism store schema (additive fields) vs prove-order retrieval (unchanged consumer)
|
||||
- Improve lever (admit/retract mechanisms) vs rule/policy levers (unchanged)
|
||||
|
||||
## Out of Boundary
|
||||
|
||||
- Taint semantics, facts, CST walker (`overlay-ir-completeness`, `guard-dominance-regime`)
|
||||
- Call graph, summaries, hunter (`reachability-flow-model`)
|
||||
- Pair scorer and catalogs (`pair-corpus-honesty`), except one read-only consumer of `PairCase`
|
||||
|
||||
## Upstream / Downstream
|
||||
|
||||
- **Upstream**: `PairCase`/`load_pair_catalog` with `review_tier`; `FileIR`; `MechanismStore`; `adjudicate`; `run_improvement` gate with per-profile holdout.
|
||||
- **Downstream**: `reachability-flow-model` attaches paths to variant findings; `guard-dominance-regime` consumes the guard shapes as sanitizer candidates; reports show mechanism ids on findings.
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
|
||||
- **Extends**: `propose-adjudicate-prove` mechanism memory (additive fields, second write path), `three-stage-scan` MAP stage (one more proposer), `harnessx-self-improving-rulesets` improve loop (one more bounded lever).
|
||||
- **Depends**: `pair-corpus-honesty` Req 9 (tiers) for which pairs may seed.
|
||||
|
||||
## Constraints
|
||||
|
||||
Core `dependencies = []`. Variant findings never exceed `suspicion` without overlay or sandbox agreement. Mechanisms seed only from `seeded` or `reviewed` pairs. JSONL stays authoritative; caches are rebuildable. Leave-one-out is reported, never gated. No target code executes outside the sandbox.
|
||||
@@ -0,0 +1,255 @@
|
||||
# Design Document
|
||||
|
||||
## Overview
|
||||
|
||||
**Purpose.** Turn trusted pairs into mechanism records, search scanned trees for structural variants, measure the corpus's own detection rate leave-one-out, and give the improve loop a recall-raising lever.
|
||||
|
||||
**Users.** Maintainers export and measure. AppSec engineers get variant findings that name the known mechanism and its known fix. The improve loop admits or retracts mechanisms under the existing gate.
|
||||
|
||||
**Impact.** `semantic/mechanisms.py` gains additive fields and a second writer. New `semantic/variants.py` (shapes and matching) and `semantic/loo.py` (leave-one-out). MAP runs variant search after overlay. `improve` gains the `mechanisms` lever. Reports show mechanism ids. No disposition, gate, or sandbox change.
|
||||
|
||||
### Goals
|
||||
|
||||
- One mechanism record per distinct shape, seeded from `seeded` and `reviewed` pairs.
|
||||
- Variant findings on real trees at `suspicion`, merged with overlay flows when present.
|
||||
- Leave-one-out recall, silence, and Youden per slice, profile, and mechanism.
|
||||
- `mechanisms` lever under the existing accept gate and per-profile holdout.
|
||||
|
||||
### Non-Goals
|
||||
|
||||
- LLM-authored shapes, regex growth, disposition or ladder changes, inter-file flow, embedding changes, new gates.
|
||||
|
||||
## Boundary Commitments
|
||||
|
||||
### This Spec Owns
|
||||
|
||||
- Shape derivation and matching (`variants.py`).
|
||||
- Additive mechanism fields, corpus writer, dedupe (`mechanisms.py`).
|
||||
- Variant proposer in MAP and manifest counters.
|
||||
- Leave-one-out runner and artifact (`loo.py`, `pairs --loo`).
|
||||
- `mechanisms` lever, its validator rules, proposer, journal fields.
|
||||
- Report rendering of mechanism ids.
|
||||
|
||||
### Out of Boundary
|
||||
|
||||
- `OverlayRecord` fields beyond an optional `mechanism_id`; dispositions; evidence rungs.
|
||||
- Catalog schema, scorer rules, harvest (`pair-corpus-honesty`).
|
||||
- Taint, facts, walker; call graph; hunter.
|
||||
- Rule and policy levers; ledger format.
|
||||
|
||||
### Allowed Dependencies
|
||||
|
||||
- `FileIR`, `parse_file`, `SemanticFacts`/`load_facts` (read), `adjudicate` output, `MechanismStore`, `PairCase`/`load_pair_catalog`/`select_slice`, pair scorer matching helpers (read), `run_round` gate, `EvolveValidator`, journal, reports.
|
||||
|
||||
### Revalidation Triggers
|
||||
|
||||
- Variant findings stamped above `suspicion` without overlay or sandbox agreement.
|
||||
- Seeding from `advisory` or `title` pairs.
|
||||
- Free-form shape text accepted by the validator.
|
||||
- Leave-one-out used as a gate.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
PAIRS[trusted pairs: seeded | reviewed] --> EXPORT[derive_shape per pair]
|
||||
EXPORT --> STORE[mechanisms.jsonl origin=corpus]
|
||||
STORE --> VS[variant search in MAP]
|
||||
IR[FileIR of scanned tree] --> VS
|
||||
FACTS[facts: sources] --> VS
|
||||
VS --> VF[variant findings: suspicion, mechanism_id]
|
||||
OV[overlay records] --> MERGE[merge when same call site has a flow]
|
||||
VF --> MERGE
|
||||
MERGE --> CAND[regress candidates after promotions]
|
||||
PAIRS --> LOO[leave-one-out runner]
|
||||
STORE -. temp store per held-out pair .-> LOO
|
||||
LOO --> ART[loo.json + roadmap table]
|
||||
LOO --> PROP[improve proposer: admit/retract]
|
||||
PROP --> GATE[accept gate + per-profile holdout]
|
||||
GATE --> STORE
|
||||
```
|
||||
|
||||
**Key decisions**
|
||||
|
||||
- Shapes are closed structures derived from `FileIR`, never text patterns.
|
||||
- Variant search proposes at `suspicion`; overlay or sandbox raises.
|
||||
- Leave-one-out is the corpus's detection rate and is reported only.
|
||||
- The lever edits record ids from the exporter; validator rejects anything else.
|
||||
|
||||
### Technology Stack
|
||||
|
||||
| Layer | Choice | Role | Notes |
|
||||
|-------|--------|------|-------|
|
||||
| Shapes | stdlib dataclasses over `FileIR` | derive, match | no new parser |
|
||||
| Store | existing JSONL + cache | corpus and sandbox records | additive fields |
|
||||
| Search | MAP stage | proposer | file budget as rank |
|
||||
| LOO | temp dirs + existing pair materialization | measurement | artifact only |
|
||||
| Improve | existing lever machinery | admit/retract | same gate |
|
||||
|
||||
## File Structure Plan
|
||||
|
||||
```
|
||||
src/openultrasast/
|
||||
├── semantic/
|
||||
│ ├── mechanisms.py # Mechanism += origin, review_tier, pairs, shape, guard; append_from_pair; dedupe_by_shape
|
||||
│ ├── variants.py # Shape, GuardKind, derive_shape(vuln_ir, fixed_ir, label, facts) -> Shape|None; match_shapes(file_ir, shapes, facts) -> list[VariantHit]
|
||||
│ └── loo.py # evaluate_loo(cases, *, store_factory) -> LooResult; payload
|
||||
├── findings.py # origin "variant" via tags/finding_id prefix "variant:" (no schema change)
|
||||
├── cli.py # MAP: variant search after overlay; manifest counters; `pairs --loo`; `mechanisms export --slice`
|
||||
├── improve/validator.py# MechanismEdit lever "mechanisms"
|
||||
├── improve/evolve.py # proposer from loo misses/leaks; gate unchanged
|
||||
└── reports.py # mechanism id, guard, provenance on findings
|
||||
tests/
|
||||
├── test_variants.py, test_loo.py, test_mechanism_export.py, test_improve.py (lever)
|
||||
```
|
||||
|
||||
### Modified Files
|
||||
|
||||
- `semantic/overlay.py` — `OverlayRecord.mechanism_id: str | None = None` (additive).
|
||||
- `regress/candidate.py` — variant findings eligible after overlay promotions (reuses forced-hotspot path).
|
||||
- `config.py` — `[variants] enabled=true, max_mechanisms=500`.
|
||||
|
||||
## System Flows
|
||||
|
||||
### Export
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A[load trusted pairs] --> B[parse vuln + fixed excerpts]
|
||||
B --> C[find labeled sink call site]
|
||||
C --> D[source positions via binds]
|
||||
B --> E[guard kind: statements only on fixed side]
|
||||
D --> F[Shape]
|
||||
E --> F
|
||||
F --> G[dedupe by shape; append with pairs provenance]
|
||||
```
|
||||
|
||||
### Variant search in MAP
|
||||
|
||||
For each parsed file within the MAP budget: for each call site whose trailing callee name equals a shape's sink name and whose arity matches, check the shape's source positions: the argument carries a fact source, a function parameter, or a container read (bind whose value is a subscript or `.get(`). On match emit a `variant:<mechanism_id>:<path>:<line>` finding at `suspicion`. If an overlay record exists at that path and line with a flow, set its `mechanism_id` and do not emit a separate finding.
|
||||
|
||||
### Leave-one-out
|
||||
|
||||
For each trusted pair P in the slice: temp store from all trusted pairs except P; materialize P's sides; run variant search on both; detection if a variant hit lies inside P's labeled function and names P's mechanism id or matches P's sink; leak if any hit on the fixed side. Aggregate per slice, profile, mechanism; record which mechanism found P.
|
||||
|
||||
## Requirements Traceability
|
||||
|
||||
| Requirement | Summary | Components | Flows |
|
||||
|-------------|---------|------------|-------|
|
||||
| 1.1–1.5 | seed from trusted pairs, dedupe, origin corpus, maintainer command | Export, Store | export |
|
||||
| 2.1–2.5 | structural shapes, closed guard kinds, no text, extra-absent degradation | Shape | export |
|
||||
| 3.1–3.6 | proposer at suspicion, merge, candidates, budget, absent from quick | VariantSearch | MAP |
|
||||
| 4.1–4.4 | leave-one-out per slice/profile/mechanism, artifact, not a gate | LOO | leave-one-out |
|
||||
| 5.1–5.5 | lever, validator, proposer, gate unchanged, journal | Lever | improve |
|
||||
| 6.1–6.3 | reports, manifest counters, suspicion label | reports | — |
|
||||
| 7.1–7.3 | offline tests, gates unchanged, extra-free green | tests | — |
|
||||
|
||||
## Components and Interfaces
|
||||
|
||||
| Component | Domain | Intent | Req | Dependencies | Contracts |
|
||||
|-----------|--------|--------|-----|--------------|-----------|
|
||||
| Shape | semantic | derive and compare shapes | 2 | FileIR, facts | Service |
|
||||
| Export | maintainer | pairs → records | 1 | PairCase, Shape, Store | Batch |
|
||||
| Store | semantic | additive fields, dedupe | 1.3, 1.4 | JSONL | State |
|
||||
| VariantSearch | MAP | propose at suspicion | 3 | Shape, Store, overlay | Service |
|
||||
| LOO | maintainer | detection rate of the corpus | 4 | Export, VariantSearch, pair rules | Batch |
|
||||
| Lever | improve | admit/retract records | 5 | validator, gate | Batch |
|
||||
|
||||
### Shape
|
||||
|
||||
```python
|
||||
GUARD_KINDS = ("null_test", "bounds_test", "allowlist_test", "auth_check", "parameterized_call", "type_change", "none")
|
||||
SOURCE_KINDS = ("fact_source", "parameter", "container_read")
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Shape:
|
||||
language: str
|
||||
sink_name: str # trailing callee segment
|
||||
arity: int
|
||||
source_positions: tuple[int, ...]
|
||||
source_kinds: tuple[str, ...]
|
||||
guard: str # GUARD_KINDS
|
||||
mechanism: str # closed vocabulary id from the pair label
|
||||
|
||||
def key(self) -> str: ... # stable, text-free
|
||||
|
||||
def derive_shape(vuln: FileIR, fixed: FileIR, *, function: str, sink: str | None, line: int | None, mechanism: str, facts: SemanticFacts) -> Shape | None: ...
|
||||
def match_shapes(ir: FileIR, shapes: Sequence[Shape], facts: SemanticFacts) -> list[VariantHit]: ...
|
||||
```
|
||||
|
||||
Invariants: shapes carry no path, line, or literal beyond identifiers; `derive_shape` returns None when the labeled sink call site is not found.
|
||||
|
||||
### Store additions
|
||||
|
||||
`Mechanism` gains `origin: str = "sandbox"`, `review_tier: str = ""`, `pairs: tuple[str, ...] = ()`, `shape: dict[str, object] | None = None`, `guard: str = "none"`. `append_from_pair(store, shape, *, summary, cwe, pair, provenance, tier)` dedupes on `shape.key()` and extends `pairs`. Old rows load with defaults.
|
||||
|
||||
### VariantSearch
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class VariantHit:
|
||||
path: str
|
||||
line: int
|
||||
mechanism_id: str
|
||||
sink_name: str
|
||||
source_kind: str
|
||||
|
||||
def search_tree(root: Path, targets: Sequence[FileTarget], store: MechanismStore, facts: SemanticFacts, *, max_mechanisms: int) -> list[VariantHit]: ...
|
||||
def hits_to_findings(hits, records: Sequence[OverlayRecord]) -> tuple[list[StaticFinding], list[OverlayRecord]]: ... # merge with overlay flows
|
||||
```
|
||||
|
||||
Finding: id `variant:<mechanism_id>:<path>:<line>`, evidence `suspicion`, tags `["variant", "mechanism:<id>"]`, rationale names the mechanism summary and pair provenance.
|
||||
|
||||
### LOO
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class LooOutcome:
|
||||
pair: str
|
||||
detected: bool
|
||||
silent: bool
|
||||
found_by: tuple[str, ...] # mechanism ids (and their pairs) that hit inside the labeled function
|
||||
|
||||
def evaluate_loo(cases: Sequence[PairCase], *, facts: SemanticFacts) -> LooResult: ... # per slice/profile/mechanism metrics
|
||||
```
|
||||
|
||||
Only `seeded | reviewed` pairs seed; every trusted pair is held out once. `known_limit` pairs are excluded.
|
||||
|
||||
### Lever
|
||||
|
||||
`MechanismEdit(lever="mechanisms", action="admit"|"retract", mechanism_id, source="loo"|"export", rationale)`. Validator: id must exist in the exporter's candidate set; shape guard in `GUARD_KINDS`; no free text fields beyond rationale. Proposer: admit candidates whose leave-one-out hit recovers a currently missed holdout pair in some profile; retract admitted records that leak on holdout fixed sides. Gate unchanged; revert restores `mechanisms.jsonl` byte for byte.
|
||||
|
||||
## Data Models
|
||||
|
||||
Store row additions: `origin`, `review_tier`, `pairs`, `shape`, `guard`. Manifest: `"variants": {"mechanisms_searched": n, "files_searched": m, "findings": k, "merged_into_overlay": j}`. LOO artifact `loo.json`: `per_slice`, `per_profile`, `per_mechanism`, `outcomes[]` with `found_by`.
|
||||
|
||||
## Error Handling
|
||||
|
||||
| Condition | Response |
|
||||
|-----------|----------|
|
||||
| Extra absent for a language | shape derivation and search skipped for that language; degradation `variants_language_unsupported` |
|
||||
| Labeled sink not found in vuln IR | pair skipped from export with a reason in the export report |
|
||||
| Store row lacks `shape` (sandbox origin) | not used for search; still used for prove ordering |
|
||||
| Admission leaks on holdout | round rejected, store reverted |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- Unit: derive_shape on a Python pair (request → `os.system`, fix adds allowlist) yields sink `system`, arity 1, source position 0, guard `allowlist_test`; match_shapes finds a variant in another file and ignores constant arguments; shape has no path or literal; `<anon>` sinks resolve to enclosing named function for LOO scoring.
|
||||
- Integration: export over the sast slice writes deduped rows with `origin=corpus`; variant search on the split-sink Python fixture emits a `variant` finding merged into the overlay coverage record; LOO on a three-pair toy slice reports which pair taught which.
|
||||
- Improve: admitting a record that recovers a holdout pair is accepted; one that leaks is rejected with byte-identical store.
|
||||
- Gates: detection, map, local pair gates unchanged; extra-free green with semantic tests skipping.
|
||||
|
||||
## Security Considerations
|
||||
|
||||
Shapes carry no code text; store rows are data. Variant findings never execute anything; sandbox candidates follow the existing safety check.
|
||||
|
||||
## Performance
|
||||
|
||||
Search is O(call sites × shapes) per file with a name index on `sink_name`; bounded by MAP budget and `max_mechanisms`. LOO parses each excerpt once and reuses IR.
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
1. Store fields and exporter; export the sast and vibe-py trusted pairs; record counts.
|
||||
2. Variant search behind `[variants].enabled` (default true in standard); manifest counters.
|
||||
3. LOO command and roadmap table.
|
||||
4. Lever and proposer.
|
||||
@@ -0,0 +1,93 @@
|
||||
# Requirements Document
|
||||
|
||||
## Introduction
|
||||
|
||||
OpenUltraSAST's labeled pair corpus measures the detector but does not improve it, and its mechanism memory is empty because only sandbox-proven findings may write to it. This spec turns every trusted pair into a mechanism record, adds a structural variant search that proposes matches on any scanned tree, measures the corpus's own detection rate leave-one-out, and gives the improve loop a lever that can raise recall under the existing false-positive ceiling and per-profile holdout gate.
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: mechanism export from pairs; shape normalization per language; variant search proposer in MAP; leave-one-out evaluation; `mechanisms` improve lever; store schema additions; reports showing mechanism ids; offline tests.
|
||||
- **Out of scope**: taint, facts, walker, dispositions, evidence rungs; embedding retrieval changes; inter-file flow; LLM-authored shapes; catalogs and scorer changes beyond reading `PairCase`; any merge gate.
|
||||
- **Adjacent expectations**: `Mechanism` gains additive fields only; `order_promotions` keeps working on old records; the improve loop's existing clauses and the per-profile holdout clause apply unchanged; `local` remains the only slice that can fail CI.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement 1: Mechanisms seeded from trusted pairs
|
||||
|
||||
**Objective:** As a maintainer, I want each reviewed pair to become a mechanism record, so that the corpus is a library the detector can search, not only a ruler.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When a pair has `review_tier` `seeded` or `reviewed` and its vulnerable excerpt parses, the system shall derive one mechanism record carrying mechanism id, CWE, language, provenance, review tier, pair name, the sink shape, the source shape, and the guard shape the fixed side added.
|
||||
2. The system shall not seed a mechanism from `advisory` or `title` pairs, from `known_limit` pairs, or from pairs whose vulnerable excerpt is `parse_failed`.
|
||||
3. When the same shape is derived from several pairs, the system shall store one record and list every pair in its provenance.
|
||||
4. Seeded records shall be written to the same append-only JSONL store as sandbox-proven mechanisms, tagged with `origin = "corpus"`, and the existing embedding cache shall remain rebuildable from it.
|
||||
5. The system shall expose the export as a maintainer command that runs offline on the vendored catalog.
|
||||
|
||||
### Requirement 2: Shapes are structural and language-scoped
|
||||
|
||||
**Objective:** As an engineer, I want mechanism shapes that match code structure rather than text, so that variants across projects are found without regex growth.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. A sink shape shall consist of the callee name (trailing segment), the argument count, and the argument positions that carry source-derived identifiers, derived from `FileIR` call sites.
|
||||
2. A source shape shall be either a fact source id, a function parameter, or a container read, derived from the vulnerable excerpt's binds.
|
||||
3. A guard shape shall be one of a closed set: `null_test`, `bounds_test`, `allowlist_test`, `auth_check`, `parameterized_call`, `type_change`, `none`, derived from statements present on the fixed side and absent on the vulnerable side.
|
||||
4. Shapes shall never contain literal source text longer than an identifier; paths, line numbers, and secrets shall not appear in a shape.
|
||||
5. When the semantic extra is absent for a language, the system shall skip shape derivation for that language with a recorded degradation and shall not fall back to regex.
|
||||
|
||||
### Requirement 3: Variant search proposes, it does not decide
|
||||
|
||||
**Objective:** As an AppSec engineer, I want a scanned tree searched for structural variants of known mechanisms, so that recall rises with the corpus while evidence stays honest.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When a standard scan runs and the mechanism store holds records for the file's language, the system shall search each parsed file's call sites for sink shapes whose source-position arguments carry a fact source, a parameter, or a container read.
|
||||
2. A match shall produce a finding with origin `variant`, the mechanism id, evidence level `suspicion`, and the pair provenance in its rationale.
|
||||
3. When the overlay already records a source-to-sink flow at the same call site, the overlay record shall carry the mechanism id and the variant finding shall be merged into it rather than duplicated.
|
||||
4. Variant findings shall be eligible for the sandbox candidate set under the existing safety check and caps, ranked after overlay promotions.
|
||||
5. The system shall bound variant search by the existing MAP file budget and shall record how many files and mechanisms were searched in the manifest.
|
||||
6. Variant search shall never demote, never raise evidence above `suspicion` on its own, and shall be absent from `quick`.
|
||||
|
||||
### Requirement 4: Leave-one-out is the corpus's detection rate
|
||||
|
||||
**Objective:** As a maintainer, I want to know whether adding pairs raises recall, so that corpus growth is measured by what it teaches the detector.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When the maintainer runs the leave-one-out command on a slice, the system shall, for each trusted pair, seed a temporary store from every other trusted pair, run variant search on the held-out vulnerable and fixed excerpts, and score the pair with the pair scorer's rules.
|
||||
2. The command shall report leave-one-out recall, silence on the fixed side, and Youden per slice, per provenance profile, and per mechanism id.
|
||||
3. The report shall list, per held-out pair, which mechanism record found it, so that a maintainer can see which pairs teach and which only test.
|
||||
4. Leave-one-out results shall be written as a JSON artifact and recorded in the roadmap table; they shall never be a merge condition.
|
||||
|
||||
### Requirement 5: A recall-raising lever in the improve loop
|
||||
|
||||
**Objective:** As a maintainer, I want the improve loop to be able to admit mechanisms, so that it can raise recall and not only shadow noisy rules.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The improve loop shall gain a `mechanisms` lever whose edits admit or retract a candidate mechanism record into the scan-time store.
|
||||
2. The validator shall accept only records derived by the shape exporter from trusted pairs, with a closed guard kind and no literal text beyond identifiers; free-form patterns shall be rejected.
|
||||
3. The proposer shall suggest admitting a candidate mechanism when it recovers a currently missed holdout pair in a profile, and retracting one when it leaks on holdout fixed sides.
|
||||
4. When a `mechanisms` edit is evaluated, the existing accept gate and the per-profile holdout clause shall apply unchanged; rejection shall revert the store byte for byte.
|
||||
5. The journal shall record each mechanism edit with its pair provenance and the metrics before and after.
|
||||
|
||||
### Requirement 6: Reports name the mechanism
|
||||
|
||||
**Objective:** As an operator, I want a finding to say which known mechanism it resembles and where that mechanism was learned, so that a fix can follow the known fix.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When a finding carries a mechanism id, the markdown and SARIF reports shall show the mechanism summary, the guard shape the known fix used, and the pair provenance.
|
||||
2. The manifest shall record the number of variant findings and the mechanisms searched.
|
||||
3. Reports shall label variant findings as suspicions unless the overlay or the sandbox raised them.
|
||||
|
||||
### Requirement 7: Offline tests and unchanged gates
|
||||
|
||||
**Objective:** As a maintainer, I want the feature proven without network, models, or a sandbox, so that it lands under the existing CI.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. Shape derivation, variant matching, leave-one-out, and the lever shall be unit-tested on local fixtures; the semantic-marked tests shall skip without the extra.
|
||||
2. The stage-1 detection gate, the split-sink map gate, and the local pair gate shall be unchanged.
|
||||
3. The extra-free suite shall stay green.
|
||||
@@ -0,0 +1,85 @@
|
||||
# Research & Design Decisions
|
||||
|
||||
## Summary
|
||||
|
||||
- **Feature**: `corpus-seeded-mechanisms`
|
||||
- **Discovery Scope**: Extension of mechanism memory, MAP stage, pair eval, and the improve loop
|
||||
- **Key Findings**:
|
||||
- `semantic/mechanisms.py` already has `Mechanism`, `MechanismStore` (append-only JSONL), `append_mechanism`, embedding cache, hard filters by language/CWE/tags, and `order_promotions`. It is written only by `cli._append_proven_mechanisms` after a sandbox trigger, so it is empty on every real tree.
|
||||
- Clearwing's recall lever is the variant loop: a proven mechanism becomes a structural search over the messy tree. OpenUltraSAST copied the store and the retrieval, not the loop.
|
||||
- `FileIR` call sites already carry callee name, argument texts, argument identifier lists, and constant flags; binds carry source text and names. That is enough for a shape without new parsing.
|
||||
- The pair scorer (`pair-corpus-honesty`) gives labeled function, mechanism id, provenance, and soon a review tier; the fixed excerpt gives the guard the fix added by diffing statement kinds.
|
||||
- The improve loop has bounded levers with a validator, journal, byte-for-byte revert, and now a per-profile holdout clause; a `mechanisms` lever fits without changing the gate.
|
||||
- Measured 2026-09-05: vibe-py 13/35, vfc-js 1/16, sast 4/10 achievable. Web classes repeat structurally (request-derived string into `exec`/`execute`, identity from the body, `os.path.join` with a parameter, permissive defaults), which is what a shape search catches.
|
||||
|
||||
## Research Log
|
||||
|
||||
### Mechanism store today
|
||||
|
||||
- **Context**: Can the existing store carry corpus-derived records?
|
||||
- **Sources Consulted**: `semantic/mechanisms.py`, `cli.py` `_append_proven_mechanisms`, `propose-adjudicate-prove` design §Mechanism.
|
||||
- **Findings**: `Mechanism` fields are summary, cwe, language, tags, keywords, what_made_it_exploitable, source_finding_id, source_repo. Identity is the pattern. JSONL is authoritative; vectors are a rebuildable cache keyed by id and model.
|
||||
- **Implications**: add `origin` (`sandbox` | `corpus`), `review_tier`, `pairs`, `shape` (structured), `guard` as additive fields with defaults so old rows load. A second writer (`append_from_pair`) beside the sandbox writer.
|
||||
|
||||
### Shape derivation from FileIR
|
||||
|
||||
- **Context**: Avoid regex and LLM authoring.
|
||||
- **Sources Consulted**: `semantic/ir.py` (`CallSite.arg_names`, `arg_is_constant`, `Bind.names`, `FunctionIR.params`), `semantic/taint.py` (`_source_in`, parameter seeding).
|
||||
- **Findings**: for the labeled function, the sink call site is the call whose trailing name equals the label's sink or whose line equals the label's line; source positions are argument indexes whose identifiers trace by binds to a fact source, a parameter, or a container read. Guard kind comes from statements on the fixed side absent on the vulnerable side: `if` with `== NULL`/`is None`/`!x` → `null_test`; comparison with `<`, `>`, `len(` → `bounds_test`; membership in a literal collection or `startswith`/regex → `allowlist_test`; identity read from an auth context (`request.state`, `req.user`, `session`) → `auth_check`; parameterized second argument → `parameterized_call`; changed declared type → `type_change`.
|
||||
- **Implications**: `semantic/variants.py` implements `derive_shape(pair_ir_vuln, pair_ir_fixed, label)` and `match_shape(file_ir, shape)`. Guard classification is heuristic and closed; unknown → `none`.
|
||||
|
||||
### Where variant search sits
|
||||
|
||||
- **Context**: Propose, adjudicate, prove roles must stay separate.
|
||||
- **Sources Consulted**: `three-stage-scan` design (MAP stage, hunter suspicion), `propose-adjudicate-prove` (overlay records, coverage), `regress/candidate.py`.
|
||||
- **Findings**: MAP already runs overlay and the tool hunter; variant search is a third proposer that reads `FileIR` and the store. Findings at `suspicion`, origin `variant`; when the overlay has a flow at that call site, merge (attach mechanism id to the overlay record). Candidate eligibility follows the hunter-eligibility path being added in `reachability-flow-model` or the sev-5 forced-hotspot path today.
|
||||
- **Implications**: no disposition change; one new finding origin; manifest counters.
|
||||
|
||||
### Leave-one-out
|
||||
|
||||
- **Context**: Prove that corpus growth raises recall, honestly.
|
||||
- **Findings**: pair eval materializes each side into a temp tree already; a leave-one-out runner seeds a temp store from the other trusted pairs and runs variant search on both sides, scoring with the pair scorer's detection and leak rules.
|
||||
- **Implications**: `pairs.py` gains a read-only consumer path (`evaluate_loo`) or a sibling module `semantic/loo.py` to keep the scorer's boundary; report per slice, profile, mechanism; artifact only.
|
||||
|
||||
## Architecture Pattern Evaluation
|
||||
|
||||
| Option | Description | Strengths | Risks / Limitations | Notes |
|
||||
|--------|-------------|-----------|---------------------|-------|
|
||||
| Embedding retrieval only | Rank existing promotions by mechanism similarity | Already exists | Cannot propose anything new; store empty | Rejected as recall lever |
|
||||
| LLM writes Semgrep rules from pairs | Model authors patterns | Flexible | Unbounded pattern text, unverifiable, violates no-model-authored-patterns | Rejected |
|
||||
| Structural shapes from FileIR plus variant search | Closed shape vocabulary, deterministic match | Zero-dep, bounded, testable leave-one-out | Shapes coarse; guard heuristics | **Selected** |
|
||||
| Fine-tuned classifier on pairs | ML detector | Learns anything | Opaque, no fix guidance, data too small | Rejected |
|
||||
|
||||
## Design Decisions
|
||||
|
||||
### Decision: Seed only from seeded and reviewed tiers
|
||||
|
||||
- **Context**: `title` pairs are unreviewed guesses; `advisory` pairs have correct fixes but unlabeled mechanisms.
|
||||
- **Selected Approach**: export reads `review_tier`; `advisory`/`title` may be scored leave-one-out as held-out targets but never seed.
|
||||
- **Trade-offs**: fewer seeds at first; honest about what the detector was taught.
|
||||
|
||||
### Decision: Variant is a proposer at suspicion
|
||||
|
||||
- **Context**: Evidence honesty.
|
||||
- **Selected Approach**: origin `variant`, `suspicion`; merge with overlay flow when present; sandbox may raise.
|
||||
- **Trade-offs**: variant-only hits do not fail CI; correct.
|
||||
|
||||
### Decision: Lever admits records, not shapes
|
||||
|
||||
- **Context**: Keep the validator closed.
|
||||
- **Selected Approach**: `mechanisms` lever edits reference record ids produced by the exporter; free-form shapes are rejected.
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
- Coarse shapes over-match (`exec(x)` everywhere) — source-position requirement plus fact/parameter/container check; report leak rate on fixed sides leave-one-out; the per-profile gate rejects admissions that leak.
|
||||
- Guard heuristics misclassify — closed set with `none`; reported, not used for demotion.
|
||||
- Store growth — dedupe by shape; one record per shape with pair provenance list.
|
||||
- Leave-one-out cost — one temp store per pair over tens of pairs; fine; cache parsed IR per excerpt.
|
||||
|
||||
## References
|
||||
|
||||
- `semantic/mechanisms.py`, `semantic/ir.py`, `semantic/taint.py`
|
||||
- `.kiro/specs/propose-adjudicate-prove/design.md` §Mechanism (Clearwing reference architecture)
|
||||
- `.kiro/specs/pair-corpus-honesty/` (review tiers, pointer slices, scorer rules)
|
||||
- `.kiro/specs/harnessx-self-improving-rulesets/design.md` (bounded levers, accept gate)
|
||||
- Clearwing sourcehunt mechanism memory and variant loop (reference, not a dependency)
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"feature_name": "corpus-seeded-mechanisms",
|
||||
"created_at": "2026-09-05T06:18:31Z",
|
||||
"updated_at": "2026-09-05T06:18:31Z",
|
||||
"language": "en",
|
||||
"phase": "tasks-generated",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
"approved": false
|
||||
},
|
||||
"design": {
|
||||
"generated": true,
|
||||
"approved": false
|
||||
},
|
||||
"tasks": {
|
||||
"generated": true,
|
||||
"approved": false
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": false
|
||||
}
|
||||
@@ -0,0 +1,84 @@
|
||||
# Implementation Plan
|
||||
|
||||
- [ ] 1. Foundation: shapes and store fields
|
||||
- [ ] 1.1 Define structural shapes and derive them from a pair
|
||||
- `Shape` with language, sink name, arity, source positions and kinds, closed guard kind, mechanism id; `derive_shape` finds the labeled sink call site in the vulnerable `FileIR`, traces source positions through binds to a fact source, parameter, or container read, and classifies the guard from statements present only on the fixed side.
|
||||
- Shapes contain no path, line, or literal beyond identifiers.
|
||||
- Observable: a Python pair (request input into `os.system`, fix adds an allowlist) yields sink `system`, arity 1, source position 0, guard `allowlist_test`; an unlabeled sink returns None; the shape key is stable across runs.
|
||||
- _Requirements: 2.1, 2.2, 2.3, 2.4_
|
||||
- _Boundary: Shape_
|
||||
|
||||
- [ ] 1.2 Add additive mechanism fields and a corpus writer with dedupe
|
||||
- `Mechanism` gains origin, review_tier, pairs, shape, guard with defaults; `append_from_pair` dedupes by shape key and extends provenance; old JSONL rows load unchanged.
|
||||
- Observable: appending the same shape from two pairs yields one row listing both pairs; a pre-existing sandbox row loads with `origin = "sandbox"`.
|
||||
- _Requirements: 1.3, 1.4_
|
||||
- _Boundary: Store_
|
||||
- _Depends: 1.1_
|
||||
|
||||
- [ ] 2. Core: export and search
|
||||
- [ ] 2.1 Export mechanisms from trusted pairs as a maintainer command
|
||||
- Only `seeded` and `reviewed`, non-known-limit, parse-ok pairs seed; the command runs offline over the vendored catalog and reports skipped pairs with reasons; extra-absent languages are skipped with a degradation.
|
||||
- Observable: `ousast mechanisms export --slice sast` writes corpus rows and reports the skipped `title`/`advisory`/known-limit pairs.
|
||||
- _Requirements: 1.1, 1.2, 1.5, 2.5_
|
||||
- _Boundary: Export_
|
||||
- _Depends: 1.2_
|
||||
|
||||
- [ ] 2.2 Search a tree for variants and merge with overlay flows
|
||||
- For each parsed file within the MAP budget, call sites matching a shape's sink name, arity, and source positions (fact source, parameter, container read) produce `variant` findings at `suspicion` with the mechanism id; when the overlay has a flow at the same call site, its record gains `mechanism_id` and no duplicate finding is emitted.
|
||||
- Observable: the split-sink Python fixture yields a variant finding merged into the overlay coverage record; a constant-argument call does not match; `quick` emits no variants.
|
||||
- _Requirements: 3.1, 3.2, 3.3, 3.6_
|
||||
- _Boundary: VariantSearch_
|
||||
- _Depends: 2.1_
|
||||
|
||||
- [ ] 2.3 Make variant findings sandbox-eligible and count them in the manifest
|
||||
- Variant findings enter the candidate set after overlay promotions under the existing safety check and caps; the manifest records mechanisms searched, files searched, findings, and merges.
|
||||
- Observable: with the fake sandbox a variant finding runs; the manifest `variants` block is present on standard scans and absent on quick.
|
||||
- _Requirements: 3.4, 3.5, 6.2_
|
||||
- _Depends: 2.2_
|
||||
|
||||
- [ ] 3. Measurement: leave-one-out
|
||||
- [ ] 3.1 Evaluate the corpus leave-one-out
|
||||
- For each trusted pair, seed a temporary store from the others, search both excerpts, and score detection and silence with the pair rules; report per slice, profile, mechanism, and per pair which mechanism found it; write `loo.json`.
|
||||
- Observable: `ousast pairs --slice vibe-py --loo --json` reports recall, silence, Youden per slice/profile/mechanism and `found_by` per pair; the run is offline.
|
||||
- _Requirements: 4.1, 4.2, 4.3, 4.4_
|
||||
- _Boundary: LOO_
|
||||
- _Depends: 2.2_
|
||||
|
||||
- [ ] 3.2 Record the leave-one-out baseline
|
||||
- Roadmap gains a leave-one-out table per slice next to the pair baselines; not a gate.
|
||||
- Observable: roadmap table present; gates unchanged.
|
||||
- _Requirements: 4.4, 7.2_
|
||||
- _Depends: 3.1_
|
||||
|
||||
- [ ] 4. Integration: improve lever
|
||||
- [ ] 4.1 Add the mechanisms lever with a closed validator
|
||||
- `MechanismEdit` admit/retract by record id from the exporter's candidate set; validator rejects free-form shapes and unknown guard kinds; journal records pair provenance and metrics.
|
||||
- Observable: a free-form edit is rejected; an admit edit of an exported record passes validation and appears in the journal.
|
||||
- _Requirements: 5.1, 5.2, 5.5_
|
||||
- _Boundary: Lever_
|
||||
- _Depends: 2.1_
|
||||
|
||||
- [ ] 4.2 Propose admissions from leave-one-out misses and retractions from leaks under the existing gate
|
||||
- Admit candidates that recover a currently missed holdout pair; retract admitted records that leak on holdout fixed sides; existing accept gate and per-profile clause decide; rejection reverts the store byte for byte.
|
||||
- Observable: a scripted candidate that recovers a holdout pair is accepted and persisted; one that leaks on a fixed side is rejected and the store is unchanged.
|
||||
- _Requirements: 5.3, 5.4_
|
||||
- _Depends: 4.1, 3.1_
|
||||
|
||||
- [ ] 5. Integration: reports
|
||||
- [ ] 5.1 Show mechanism id, known fix guard, and provenance on findings
|
||||
- Markdown and SARIF show mechanism summary, guard shape, and pair provenance for findings carrying a mechanism id; variant findings are labeled suspicions unless raised.
|
||||
- Observable: a standard scan report on the split-sink fixture names the mechanism and its guard; SARIF properties carry the id.
|
||||
- _Requirements: 6.1, 6.3_
|
||||
- _Depends: 2.2_
|
||||
|
||||
- [ ] 6. Validation
|
||||
- [ ] 6.1 Prove offline tests and unchanged gates
|
||||
- Unit and integration tests run without network, model, or sandbox; semantic-marked tests skip without the extra; detection, map, and local pair gates unchanged; extra-free suite green.
|
||||
- Observable: both pytest matrices green; gate entrypoints unchanged.
|
||||
- _Requirements: 7.1, 7.2, 7.3_
|
||||
- _Depends: 2.3, 3.1, 4.2, 5.1_
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- Seeding reads `PairCase.review_tier` from `pair-corpus-honesty` task 8.1; until it lands, sast and vibe-py rows default to `advisory`/`seeded` per that design and only vibe-py seeds.
|
||||
- Guard classification is heuristic and closed; `none` is allowed and reported, never used to demote.
|
||||
Reference in New Issue
Block a user