mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-19 21:19:28 +00:00
docs: plain-English pass over the active specs
Ran the humanizer rules over 79 prose files in the active specs (brief, design, requirements, research and narrative docs). Removed em dashes, significance inflation, forced rule-of-three, bold-label list items and filler. Skipped tasks.md checklists and the retired specs (harnessx-*, openrouter-sast-harness, unseen-repo-evaluation). Every number, path, command, hash and identifier was preserved: checked each file against HEAD for drift in numbers, inline code and code fences, zero drift across all 79 files. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
Marius Ciepluch
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
b08f372022
commit
39a1873e5b
@@ -2,7 +2,7 @@
|
||||
|
||||
## Problem
|
||||
|
||||
OpenUltraSAST already has three scan stages (`quick` inventory, `standard` map, `deep` regression), but **only the first stage actually decides**. Pattern rules propose sinks; the map ranks files; the sandbox can run recipes. Nothing in the middle answers “does this sink receive attacker-controlled data, or is it a Juliet-style good twin / dead assignment / hardcoded command?”
|
||||
OpenUltraSAST already has three scan stages (`quick` inventory, `standard` map, `deep` regression), but **only the first stage actually decides**. Pattern rules propose sinks; the map ranks files; the sandbox can run recipes. Nothing in the middle answers "does this sink receive attacker-controlled data, or is it a Juliet-style good twin / dead assignment / hardcoded command?"
|
||||
|
||||
Calibration corpora (cheat-sheet fixtures, OWASP Benchmark true/false files, Juliet `bad`/`good`, CVE pairs) are **nice**: one CWE per file, named sinks, labeled counterparts. Real working trees are not: mixed languages, generated code, unparseable files, framework wrappers, comments, one-hop assignments (`sql = f"…{x}"; cur.execute(sql)`), and no ground-truth labels.
|
||||
|
||||
@@ -12,17 +12,17 @@ Measured honesty on the labeled SAST slice is about **+9% Youden**. Widening reg
|
||||
|
||||
Keep three roles, never collapse them:
|
||||
|
||||
1. **Propose** — pattern rules + optional SARIF. Stage 1. Does not decide worth-fixing.
|
||||
2. **Adjudicate** — tree-sitter IR (multi-language) + taint facts; optional Joern CPG sidecar; Python stdlib parse only as last-resort fallback. Promote, demote, or **unadjudicated**.
|
||||
3. **Budget** — ranker + **mechanism memory** (JSONL + OpenRouter embeddings, hard CWE/language/tag filters) **before** prove. Not a verdict. Not TF-IDF. Not raw file-slice embeddings.
|
||||
4. **Prove** — sandbox only on promoted, reachable, budgeted candidates.
|
||||
1. **Propose**, pattern rules + optional SARIF. Stage 1. Does not decide worth-fixing.
|
||||
2. **Adjudicate**, tree-sitter IR (multi-language) + taint facts; optional Joern CPG sidecar; Python stdlib parse only as last-resort fallback. Promote, demote, or **unadjudicated**.
|
||||
3. **Budget**, ranker + **mechanism memory** (JSONL + OpenRouter embeddings, hard CWE/language/tag filters) **before** prove. It is a budget rather than a verdict; it uses neither TF-IDF nor raw file-slice embeddings.
|
||||
4. **Prove**, sandbox only on promoted, reachable, budgeted candidates.
|
||||
|
||||
The primary in-scope target is a **messy working tree the operator is writing**, not a labeled benchmark. Known-vuln corpora remain calibration scoreboards (smoke 90/10, pair Youden, GitHub silent-on-fix). They do not define success on unlabeled code.
|
||||
|
||||
## Scope
|
||||
|
||||
- Semantic overlay as MAP-stage adjudication; **tree-sitter is the prio engine** (CLI/API, fail-closed). Joern optional. Embeddings tune prove order.
|
||||
- Explicit `unadjudicated` when parse or flow fails — do not treat silence as safe.
|
||||
- Explicit `unadjudicated` when parse or flow fails. Do not treat silence as safe.
|
||||
- Sandbox prove only after promote.
|
||||
- Pair eval scores the overlay on the sast slice; smoke gate stays inventory-only.
|
||||
|
||||
|
||||
@@ -10,10 +10,10 @@
|
||||
|
||||
### Goals
|
||||
|
||||
- Map every current scan module onto propose, adjudicate, budget, or prove — rank/embed is not a verdict.
|
||||
- Map every current scan module onto propose, adjudicate, budget, or prove (rank/embed is not a verdict).
|
||||
- Treat messy unlabeled trees as the default; labeled corpora as calibration.
|
||||
- Fail closed: cannot parse or cannot follow flow → `unadjudicated`, not demote.
|
||||
- **Prioritize the semantic engine now**: tree-sitter IR for in-scope languages; Joern optional CPG sidecar; stdlib Python parse only as last resort.
|
||||
- Prioritize the semantic engine now: tree-sitter IR for in-scope languages; Joern optional CPG sidecar; stdlib Python parse only as last resort.
|
||||
- Tunable embeddings + existing ranker order **prove** candidates.
|
||||
- SAST pair slice scores the overlay; smoke gate stays inventory-only.
|
||||
|
||||
@@ -51,11 +51,11 @@
|
||||
- Preprocess, mapping, rank (`heuristic_rank` 0.5/0.2/0.3), `quick_scan_findings`, complexity map, `ousast index` / json-local vectorstore, tool hunter, sandbox runner, pair eval, reports.
|
||||
- Host binaries when present: `tree-sitter`, `joern` / `joern-parse` (probed, never required for inventory).
|
||||
- Optional embedding model from existing `[embeddings]` config.
|
||||
- Clearwing as an **oracle** for tree-sitter callgraph + taint-pattern tables + mechanism retrieval — not a dependency.
|
||||
- Clearwing as an **oracle** for tree-sitter callgraph + taint-pattern tables + mechanism retrieval, not a dependency.
|
||||
|
||||
### Revalidation Triggers
|
||||
|
||||
- Adding a fourth disposition that implies “safe.”
|
||||
- Adding a fourth disposition that implies "safe."
|
||||
- Demoting when flow is incomplete.
|
||||
- Stamping overlay above `static_corroboration`.
|
||||
- Selecting sandbox candidates from demoted or unadjudicated sets.
|
||||
@@ -84,7 +84,7 @@ Concrete mapping of **what exists today** onto the three roles:
|
||||
| Prove | REGRESS / `deep` | `regress` + sandbox | Recipes on ranked findings | Recipes on **promoted then budgeted** findings |
|
||||
| Calibrate | CI | `gate.py` / `pairs` | Cheat-sheet vs Youden | Inventory vs overlay, unchanged split |
|
||||
|
||||
**Selected pattern.** Propose → adjudicate → **budget** → prove. Clearwing’s tree-sitter callgraph + taint tables + mechanism memory are the oracle, not a library import. Embeddings never demote.
|
||||
**Selected pattern.** Propose → adjudicate → **budget** → prove. Clearwing's tree-sitter callgraph + taint tables + mechanism memory are the oracle, not a library import. Embeddings never demote.
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
@@ -161,13 +161,13 @@ src/openultrasast/ruleset/semantic/
|
||||
|
||||
### Modified Files
|
||||
|
||||
- `stages.py` / `cli.py` — overlay in MAP; prove-budget before regress.
|
||||
- `findings.py` — inventory unchanged.
|
||||
- `regress/candidate.py` — `promote` only, then budget order.
|
||||
- `rank.py` / `index.py` / `vectorstore.py` — reuse; do not invent a second store.
|
||||
- `reports.py` — disposition columns.
|
||||
- `pairs.py` — sast slice uses overlay.
|
||||
- `mcp.py` — still no `deep`.
|
||||
- `stages.py` / `cli.py`, overlay in MAP; prove-budget before regress.
|
||||
- `findings.py`, inventory unchanged.
|
||||
- `regress/candidate.py`, `promote` only, then budget order.
|
||||
- `rank.py` / `index.py` / `vectorstore.py`, reuse; do not invent a second store.
|
||||
- `reports.py`, disposition columns.
|
||||
- `pairs.py`, sast slice uses overlay.
|
||||
- `mcp.py`, still no `deep`.
|
||||
|
||||
## System Flows
|
||||
|
||||
@@ -301,11 +301,11 @@ Identity is the *pattern*, not a file slice:
|
||||
| what_made_it_exploitable | why a near-miss became a bug |
|
||||
| source_finding_id, source_repo | provenance only |
|
||||
|
||||
Persistence: append-only JSONL (portable). Recall is **not** Clearwing’s TF-IDF ladder.
|
||||
Persistence: append-only JSONL (portable). Recall is **not** Clearwing's TF-IDF ladder.
|
||||
|
||||
1. **Filter** — language, CWE, tags (exact overlap). This is a hard prefilter, not a ranker.
|
||||
2. **Rank** — OpenRouter `/embeddings` (`OpenRouterEmbeddingClient`, existing `[embeddings].model`) over `summary + what_made_it_exploitable + tags`. Cosine against the promotion’s overlay text. Cache vectors in json-local `mechanisms` namespace, keyed by mechanism id + model id.
|
||||
3. **Degrade** — no key / no model / embed failure → heuristic ranker only, degradation recorded. Never bag-of-words cosine.
|
||||
1. **Filter**, language, CWE, tags (exact overlap). This is a hard prefilter, not a ranker.
|
||||
2. **Rank**, OpenRouter `/embeddings` (`OpenRouterEmbeddingClient`, existing `[embeddings].model`) over `summary + what_made_it_exploitable + tags`. Cosine against the promotion's overlay text. Cache vectors in json-local `mechanisms` namespace, keyed by mechanism id + model id.
|
||||
3. **Degrade**, no key / no model / embed failure → heuristic ranker only, degradation recorded. Never bag-of-words cosine.
|
||||
|
||||
Query for a promotion: embed the overlay rationale (sources, sink, CWE). Result is a **boost to prove order**, never a demote.
|
||||
|
||||
@@ -327,7 +327,7 @@ Repo-code chunks (`ousast index` 80-line slices) stay for hunter/verifier retrie
|
||||
|
||||
- Unit: parse fail → unadjudicated; constant `exec("ls")` → demote; `eval(request.args)` → promote; `sql = f"…{x}"; execute(sql)` → coverage or promote.
|
||||
- Pair: `--slice sast` uses overlay; `juliet-c-cwe134-printf` stays PASS; `owasp-python-deser` stays PASS; Juliet `goodG2B` exec/strcpy must **demote or unadjudicated**, not inventory-leak counted as pair fail for overlay.
|
||||
- Unlabeled: tmp tree with a syntax-error file next to a clean `eval(request.args)` — error file unadjudicated, clean file promoted; no expected.toml required.
|
||||
- Unlabeled: tmp tree with a syntax-error file next to a clean `eval(request.args)`. Error file unadjudicated, clean file promoted; no expected.toml required.
|
||||
- Gate: `python -m openultrasast.gate` still inventory-only.
|
||||
- Deep: candidate list excludes demoted.
|
||||
|
||||
@@ -346,4 +346,4 @@ Repo-code chunks (`ousast index` 80-line slices) stay for hunter/verifier retrie
|
||||
|
||||
- Default `standard` runs overlay when facts load; if facts missing, degrade as Requirement 7.5.
|
||||
- Existing `quick` CI unchanged.
|
||||
- Pair gate local 100% remains inventory-or-overlay-equivalent on fixture counterparts (those files are still “nice”; unlabeled tests are additional).
|
||||
- Pair gate local 100% remains inventory-or-overlay-equivalent on fixture counterparts (those files are still "nice"; unlabeled tests are additional).
|
||||
|
||||
@@ -4,15 +4,15 @@
|
||||
|
||||
## Introduction
|
||||
|
||||
OpenUltraSAST users scan **working trees they are writing**: mixed files, incomplete edits, generated code, framework wrappers, and no labels. Today the harness still *decides* with pattern inventory. The complexity map ranks; the sandbox can run; neither adjudicates “this sink is fed by a source” versus “this sink is hardcoded, sanitized, or unreadable.”
|
||||
OpenUltraSAST users scan **working trees they are writing**: mixed files, incomplete edits, generated code, framework wrappers, and no labels. Today the harness still *decides* with pattern inventory. The complexity map ranks; the sandbox can run; neither adjudicates "this sink is fed by a source" versus "this sink is hardcoded, sanitized, or unreadable."
|
||||
|
||||
This feature splits the scan into three operator-visible roles that must not collapse: **regex proposes, semantics adjudicates, sandbox proves.** Labeled suites (cheat-sheet fixtures, OWASP Benchmark, Juliet, CVE pairs) remain **calibration**. They are not the definition of in-scope code.
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: Role split on `quick` / `standard` / `deep`; overlay dispositions; tree-sitter (and optional Joern) as the semantic engine; tunable retrieval/rank **before prove**; `unadjudicated` on messy files; sandbox only after promote; calibration vs unlabeled working trees.
|
||||
- **Out of scope**: Deleting pattern rules; making an LLM the adjudicator; rewriting CWE policy or project-score formula; requiring Joern or an embedding model for `quick`; auto-generated flow facts from the improve loop; treating benchmark Youden as a merge gate that can unblock a smoke-gate breach. Embedding rank is a **budget**, not a fourth verdict.
|
||||
- **Adjacent expectations**: Stage plan, complexity map, Docker sandbox, pair catalog, and 90/10 detection gate from `three-stage-scan` keep their contracts. This spec overlays adjudication on that pipeline; it does not replace stage accounting, sandbox flags, or `LANGUAGE_MANIFESTS`.
|
||||
- In scope: Role split on `quick` / `standard` / `deep`; overlay dispositions; tree-sitter (and optional Joern) as the semantic engine; tunable retrieval/rank **before prove**; `unadjudicated` on messy files; sandbox only after promote; calibration vs unlabeled working trees.
|
||||
- Out of scope: Deleting pattern rules; making an LLM the adjudicator; rewriting CWE policy or project-score formula; requiring Joern or an embedding model for `quick`; auto-generated flow facts from the improve loop; treating benchmark Youden as a merge gate that can unblock a smoke-gate breach. Embedding rank is a **budget**, not a fourth verdict.
|
||||
- Adjacent expectations: Stage plan, complexity map, Docker sandbox, pair catalog, and 90/10 detection gate from `three-stage-scan` keep their contracts. This spec overlays adjudication on that pipeline; it does not replace stage accounting, sandbox flags, or `LANGUAGE_MANIFESTS`.
|
||||
|
||||
## Requirements
|
||||
|
||||
@@ -36,9 +36,9 @@ This feature splits the scan into three operator-visible roles that must not col
|
||||
|
||||
1. When a scanned file is incomplete, generated, minified, or otherwise not interpretable as structured source, the system shall leave matching inventory proposals **unadjudicated** and shall record that reason on the finding.
|
||||
2. When no available semantic engine can follow data from a source to a sink for that file (parse failure, missing grammar, cross-file call, dynamic dispatch, or missing callee), the system shall leave the proposal unadjudicated rather than demote it.
|
||||
3. When OpenUltraSAST scans a working tree with no benchmark manifest and no expected-finding labels, the system shall still emit proposals, dispositions, and (in `deep`) proofs without requiring a counterpart “fixed” tree.
|
||||
3. When OpenUltraSAST scans a working tree with no benchmark manifest and no expected-finding labels, the system shall still emit proposals, dispositions, and (in `deep`) proofs without requiring a counterpart "fixed" tree.
|
||||
4. If the operator reads a report for an unadjudicated proposal, then the system shall present it as unresolved inventory, not as a verified vulnerability and not as a cleared finding.
|
||||
5. The system shall not require a file to resemble an OWASP Benchmark or Juliet test case in order to produce a proposal.
|
||||
5. The system shall not require a file to resemble an OWASP Benchmark or Juliet test case to produce a proposal.
|
||||
|
||||
### Requirement 3: Adjudication promote, demote, coverage
|
||||
|
||||
@@ -46,8 +46,8 @@ This feature splits the scan into three operator-visible roles that must not col
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When a proposal’s sink is reachable from an identified untrusted source without an identified sanitizer in the adjudicated region, the system shall **promote** the proposal.
|
||||
2. When a proposal’s sink is fed only by a constant, a sanitizer, or an identified non-attacker source in the adjudicated region, the system shall **demote** the proposal and shall not list it as a promoted finding.
|
||||
1. When a proposal's sink is reachable from an identified untrusted source without an identified sanitizer in the adjudicated region, the system shall **promote** the proposal.
|
||||
2. When a proposal's sink is fed only by a constant, a sanitizer, or an identified non-attacker source in the adjudicated region, the system shall **demote** the proposal and shall not list it as a promoted finding.
|
||||
3. When adjudication finds an untrusted source flowing to a dangerous sink that inventory did not propose, the system shall emit a **coverage** finding at corroboration-or-lower evidence, tagged as overlay-originated.
|
||||
4. When both a source-to-sink path and a sanitizer appear to apply, the system shall not promote unless the sanitizer does not dominate that path.
|
||||
5. The system shall persist promote, demote, unadjudicated, and coverage outcomes in the scan artifacts with the proposal identifier they refer to.
|
||||
@@ -62,7 +62,7 @@ This feature splits the scan into three operator-visible roles that must not col
|
||||
2. When the pair-efficiency evaluation runs on the SAST slice, the system shall score **adjudicated** outcomes (promote on the vulnerable side, demote or silent on the false or good side), not raw inventory hits.
|
||||
3. When overlay changes raise SAST Youden and also cause the stage-1 detection gate to fail, the system shall treat the detection-gate failure as blocking.
|
||||
4. The system shall keep labeled pair fixtures and SAST-slice cases out of the stage-1 cheat-sheet corpus map.
|
||||
5. The system shall not require a labeled counterpart tree in order to scan an operator working tree.
|
||||
5. The system shall not require a labeled counterpart tree to scan an operator working tree.
|
||||
|
||||
### Requirement 5: Evidence and models stay honest
|
||||
|
||||
@@ -74,7 +74,7 @@ This feature splits the scan into three operator-visible roles that must not col
|
||||
2. When the tool hunter emits a finding, the system shall keep that finding at `suspicion` unless an independent corroborating stage has already raised it.
|
||||
3. If an overlay or hunter result claims an evidence level above corroboration without a sandbox or verifier step that earned it, then the system shall reject that claim.
|
||||
4. The system shall not use a language model as the adjudication engine that decides promote or demote.
|
||||
5. Where CWE policy applies, the system shall resolve severity from the central CWE policy, not from the overlay or the proposing rule’s private severity string.
|
||||
5. Where CWE policy applies, the system shall resolve severity from the central CWE policy, not from the overlay or the proposing rule's private severity string.
|
||||
|
||||
### Requirement 6: Sandbox proves only promoted work
|
||||
|
||||
@@ -106,7 +106,7 @@ This feature splits the scan into three operator-visible roles that must not col
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When a tree-sitter grammar for the file’s language is available, the system shall use that parse as the primary intra-file IR for adjudication.
|
||||
1. When a tree-sitter grammar for the file's language is available, the system shall use that parse as the primary intra-file IR for adjudication.
|
||||
2. When tree-sitter is unavailable for that language, the system shall try a stdlib parse only for Python and shall otherwise leave the proposal unadjudicated with language_unsupported or parse_failed.
|
||||
3. Where Joern is installed and the language is one Joern can export, the system shall be allowed to attach Joern flow facts as **additional** corroboration; Joern absence shall not fail `quick` or `standard`.
|
||||
4. If Joern and tree-sitter disagree or Joern returns incomplete paths, then the system shall not demote on that disagreement; it shall promote only on a complete dominating path or leave the item unadjudicated.
|
||||
@@ -124,6 +124,6 @@ This feature splits the scan into three operator-visible roles that must not col
|
||||
3. The system shall not demote or drop a promotion solely because mechanism similarity is low.
|
||||
4. When the embedding index is missing or stale, the system shall rebuild it from the mechanism log via OpenRouter embeddings; if that rebuild fails, the system shall record the degradation and continue on heuristic order.
|
||||
5. The system shall keep hard filters operator-tunable (language, CWE, tags) without changing CWE severity policy.
|
||||
6. When a finding is sandbox-proven triggerable (or otherwise verified at corroboration-or-higher with a prove step), the system shall append an abstract mechanism record to the mechanism log: summary, CWE, language, tags, what made it exploitable — without file:line as the identity.
|
||||
6. When a finding is sandbox-proven triggerable (or otherwise verified at corroboration-or-higher with a prove step), the system shall append an abstract mechanism record to the mechanism log: summary, CWE, language, tags, what made it exploitable, without file:line as the identity.
|
||||
7. The system shall treat the append-only mechanism log as the source of truth; any vector index shall be a rebuildable cache of that log.
|
||||
8. The system shall not persist mechanisms from inventory-only, demoted, or unadjudicated proposals.
|
||||
|
||||
@@ -2,9 +2,9 @@
|
||||
|
||||
## Summary
|
||||
|
||||
- **Feature**: `propose-adjudicate-prove`
|
||||
- **Discovery Scope**: Extension of `three-stage-scan` (MAP grows an overlay; STATIC and REGRESS stay)
|
||||
- **Key Findings**:
|
||||
- Feature: `propose-adjudicate-prove`
|
||||
- Discovery Scope: Extension of `three-stage-scan` (MAP grows an overlay; STATIC and REGRESS stay)
|
||||
- Key Findings:
|
||||
- Inventory already decides; MAP ranks; REGRESS can run. Nothing adjudicates flow.
|
||||
- Labeled SAST slice Youden is about +9%. Leaks are OWASP-false / Juliet `goodG2B` that still contain the sink. Misses are one-hop assignments and sink aliases.
|
||||
- Clearwing already does the split we want: tree-sitter callgraph + pattern taint tables, then ranker (surface/influence/reachability 0.5/0.2/0.3), then mechanism-memory embeddings. OpenUltraSAST already copied the rank formula and has `json-local` vectors; they are not wired as prove-budget.
|
||||
@@ -15,55 +15,55 @@
|
||||
|
||||
### Labeled vs unlabeled code
|
||||
|
||||
- **Context**: Operator: known-vuln corpora are “nice”; written code will not be.
|
||||
- **Sources Consulted**: In-tree `LANGUAGE_MANIFESTS`, `benchmarks/pairs/`, OWASP Benchmark true/false files, Juliet `bad`/`goodG2B`/`goodB2G`, CWE-Bench-Java catalog.
|
||||
- **Findings**:
|
||||
- Context: Operator: known-vuln corpora are "nice"; written code will not be.
|
||||
- Sources Consulted: In-tree `LANGUAGE_MANIFESTS`, `benchmarks/pairs/`, OWASP Benchmark true/false files, Juliet `bad`/`goodG2B`/`goodB2G`, CWE-Bench-Java catalog.
|
||||
- Findings:
|
||||
- Cheat-sheets: one CWE, named sink, often same-line source.
|
||||
- OWASP: true and false files both often contain `exec` / `eval`; the difference is source constancy or infeasible branches.
|
||||
- Juliet `goodG2B`: hardcoded source, **same sink**.
|
||||
- Juliet `goodB2G`: same source, **sanitized sink** (`printf("%s")`).
|
||||
- Working trees: syntax errors, generated bundles, wrappers (`helpers.db_sqlite.execute`), no labels.
|
||||
- **Implications**: Overlay scored on sast slice; unlabeled scans must not pretend demote. Unadjudicated is a first-class disposition.
|
||||
- Implications: Overlay scored on sast slice; unlabeled scans must not pretend demote. Unadjudicated is a first-class disposition.
|
||||
|
||||
### Engine choice
|
||||
|
||||
- **Context**: Operator prefers tree-sitter + Joern look, embeddings before prove, still v1-focused.
|
||||
- **Sources Consulted**: `clearwing/sourcehunt/callgraph.py`, `taint.py`, `ranker.py`, `mechanism_memory.py`; `openultrasast/rank.py`, `vectorstore.py`, `index.py`.
|
||||
- **Findings**:
|
||||
- Context: Operator prefers tree-sitter + Joern look, embeddings before prove, still v1-focused.
|
||||
- Sources Consulted: `clearwing/sourcehunt/callgraph.py`, `taint.py`, `ranker.py`, `mechanism_memory.py`; `openultrasast/rank.py`, `vectorstore.py`, `index.py`.
|
||||
- Findings:
|
||||
- Clearwing tree-sitter is optional: missing grammar disables callgraph, does not crash.
|
||||
- Clearwing taint is intra-procedural, pattern tables for sources/sinks — same shape as our fact TOML.
|
||||
- Clearwing taint is intra-procedural, pattern tables for sources/sinks, same shape as our fact TOML.
|
||||
- Clearwing ranker fills reachability from the callgraph; taint_paths boost surface. Embeddings retrieve *mechanisms*, they do not decide taint.
|
||||
- Our `heuristic_rank` already uses surface·0.5 + influence·0.2 + reachability·0.3. Vector store bakeoff selected `json-local`. `VALID_NAMESPACES` already includes `mechanisms` but nothing writes Clearwing-style records into it.
|
||||
|
||||
### Clearwing vector-store reference (real-world bugs)
|
||||
|
||||
- **Context**: Operator: Clearwing’s store is effective on real bugs, not textbook files.
|
||||
- **Sources Consulted**: `clearwing/sourcehunt/mechanism_memory.py`, `data/memory/semantic_memory.py`, `docs/architecture.md` steps 7–8; `openultrasast/index.py` namespaces.
|
||||
- **Findings**:
|
||||
- Store **mechanisms**, not findings and not 80-line file windows. Example: “length field trusted before alloc; 16-bit value widened to size_t.”
|
||||
- Context: Operator: Clearwing's store is effective on real bugs, not textbook files.
|
||||
- Sources Consulted: `clearwing/sourcehunt/mechanism_memory.py`, `data/memory/semantic_memory.py`, `docs/architecture.md` steps 7–8; `openultrasast/index.py` namespaces.
|
||||
- Findings:
|
||||
- Store **mechanisms**, not findings and not 80-line file windows. Example: "length field trusted before alloc; 16-bit value widened to size_t."
|
||||
- JSONL is the source of truth (`~/.clearwing/sourcehunt/mechanisms.jsonl`). Chroma/sqlite embeddings are ephemeral caches rebuilt from JSONL.
|
||||
- Write path is **verified findings only** (after adversarial verify / crash). Suspicion never seeds the store.
|
||||
- Clearwing’s JSONL + verified-write + metadata filters are the reference. Its TF-IDF default is **not**: bag-of-words will not match paraphrased real bugs (`length field trusted before alloc` vs `user-controlled size_t in malloc`).
|
||||
- Clearwing's JSONL + verified-write + metadata filters are the reference. Its TF-IDF default is **not**: bag-of-words will not match paraphrased real bugs (`length field trusted before alloc` vs `user-controlled size_t in malloc`).
|
||||
- OpenUltraSAST already has `OpenRouterEmbeddingClient` and json-local cosine. Use that as the only semantic ranker; CWE/language/tags are hard filters. No TF-IDF, no Chroma, no local MiniLM.
|
||||
- Query is language + file tags + optional free text; same-language boost +0.5. Top-N injects hunter context and variant-grep seeds — it does not decide taint.
|
||||
- Query is language + file tags + optional free text; same-language boost +0.5. Top-N injects hunter context and variant-grep seeds. It does not decide taint.
|
||||
- Variant loop then greps the *messy* tree for structural matches of that mechanism. That is how it leaves Juliet behind.
|
||||
- **Implications**: Copy JSONL + verified-write + metadata filters. Rank with OpenRouter embeddings only. Do not copy TF-IDF. Do not embed 80-line file windows as the prove store. Seed from sandbox-proven mechanisms, not OWASP `BenchmarkTest00007`.
|
||||
- Implications: Copy JSONL + verified-write + metadata filters. Rank with OpenRouter embeddings only. Do not copy TF-IDF. Do not embed 80-line file windows as the prove store. Seed from sandbox-proven mechanisms, not OWASP `BenchmarkTest00007`.
|
||||
- Joern CLI (`joern-parse` / scripts) is the only practical CPG without a Scala runtime in-process.
|
||||
- **Implications**: Probe tree-sitter like Docker. Joern sidecar cannot demote. Embeddings order prove only.
|
||||
- Implications: Probe tree-sitter like Docker. Joern sidecar cannot demote. Embeddings order prove only.
|
||||
|
||||
### Decision: Python intra-file first
|
||||
|
||||
- **Context**: Zero-dep core vs multi-language trees.
|
||||
- **Selected**: **Superseded.** Tree-sitter is the prio engine; stdlib `ast` is Python last resort; `quick` stays zero-dep.
|
||||
- **Rationale**: Operator working trees are mixed-language; Python-only adjudication would leave JS/C/Java as perpetual unadjudicated.
|
||||
- **Trade-offs**: `standard` needs a grammar or CLI on PATH; missing tools → unadjudicated, not skip the scan.
|
||||
- Context: Zero-dep core vs multi-language trees.
|
||||
- Selected: **Superseded.** Tree-sitter is the prio engine; stdlib `ast` is Python last resort; `quick` stays zero-dep.
|
||||
- Rationale: Operator working trees are mixed-language; Python-only adjudication would leave JS/C/Java as perpetual unadjudicated.
|
||||
- Trade-offs: `standard` needs a grammar or CLI on PATH; missing tools → unadjudicated, not skip the scan.
|
||||
|
||||
### Evidence ladder
|
||||
|
||||
- **Context**: Three-stage-scan forbids hunter JSON as corroboration.
|
||||
- **Sources Consulted**: `verification.EvidenceLevel`, `tool_hunter` suspicion rule, `worth_fixing` gate.
|
||||
- **Findings**: Overlay can justify promote at `static_corroboration`. It cannot skip to crash or exploit. Worth-fixing stays sandbox + reachability.
|
||||
- **Implications**: Disposition is not an evidence rung. Do not add a seventh ladder value.
|
||||
- Context: Three-stage-scan forbids hunter JSON as corroboration.
|
||||
- Sources Consulted: `verification.EvidenceLevel`, `tool_hunter` suspicion rule, `worth_fixing` gate.
|
||||
- Findings: Overlay can justify promote at `static_corroboration`. It cannot skip to crash or exploit. Worth-fixing stays sandbox + reachability.
|
||||
- Implications: Disposition is not an evidence rung. Do not add a seventh ladder value.
|
||||
|
||||
## Architecture Pattern Evaluation
|
||||
|
||||
@@ -78,28 +78,28 @@
|
||||
|
||||
### Decision: Unadjudicated is not demote
|
||||
|
||||
- **Context**: Messy files and cross-file flow.
|
||||
- **Alternatives**: Silent drop; treat as safe; force demote.
|
||||
- **Selected**: Explicit `unadjudicated` with reason.
|
||||
- **Rationale**: Operator writing code must not read silence as “clean.”
|
||||
- **Trade-offs**: Reports stay noisy until languages and inter-file land.
|
||||
- **Follow-up**: Count unadjudicated rate on unlabeled trees as an honesty metric.
|
||||
- Context: Messy files and cross-file flow.
|
||||
- Alternatives: Silent drop; treat as safe; force demote.
|
||||
- Selected: Explicit `unadjudicated` with reason.
|
||||
- Rationale: Operator writing code must not read silence as "clean."
|
||||
- Trade-offs: Reports stay noisy until languages and inter-file land.
|
||||
- Follow-up: Count unadjudicated rate on unlabeled trees as an honesty metric.
|
||||
|
||||
### Decision: Score overlay on sast slice, inventory on smoke gate
|
||||
|
||||
- **Context**: +9% Youden vs 93% cheat-sheet recall.
|
||||
- **Selected**: Two gates stay separate; overlay Youden cannot unblock smoke failure.
|
||||
- **Rationale**: Same as detection-gate vs score in the improve loop.
|
||||
- Context: +9% Youden vs 93% cheat-sheet recall.
|
||||
- Selected: Two gates stay separate; overlay Youden cannot unblock smoke failure.
|
||||
- Rationale: Same as detection-gate vs score in the improve loop.
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
- False demote on real bugs (infeasible-branch OWASP tricks, wrappers) — never demote when flow is incomplete; only demote when the engine *has* a dominating constant or sanitizer.
|
||||
- Youden chase via extra sinks — each coverage rule needs a false/good counterpart that must stay demoted or silent.
|
||||
- Parse failures on operator WIP — unadjudicated, `quick` still works.
|
||||
- False demote on real bugs (infeasible-branch OWASP tricks, wrappers). Never demote when flow is incomplete; only demote when the engine *has* a dominating constant or sanitizer.
|
||||
- Youden chase via extra sinks. Each coverage rule needs a false/good counterpart that must stay demoted or silent.
|
||||
- Parse failures on operator WIP, unadjudicated, `quick` still works.
|
||||
|
||||
## References
|
||||
|
||||
- `.kiro/specs/three-stage-scan/design.md` — stage plan, sandbox, hunter suspicion
|
||||
- `benchmarks/pairs/sast/README.md` — OWASP/Juliet slice
|
||||
- `.kiro/specs/three-stage-scan/design.md`, stage plan, sandbox, hunter suspicion
|
||||
- `benchmarks/pairs/sast/README.md`, OWASP/Juliet slice
|
||||
- NIST Juliet `goodG2B` vs `goodB2G` templates
|
||||
- OWASP Benchmark expectedresults true/false protocol
|
||||
|
||||
Reference in New Issue
Block a user