mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-21 05:59:29 +00:00
docs: plain-English pass over the active specs
Ran the humanizer rules over 79 prose files in the active specs (brief, design, requirements, research and narrative docs). Removed em dashes, significance inflation, forced rule-of-three, bold-label list items and filler. Skipped tasks.md checklists and the retired specs (harnessx-*, openrouter-sast-harness, unseen-repo-evaluation). Every number, path, command, hash and identifier was preserved: checked each file against HEAD for drift in numbers, inline code and code fences, zero drift across all 79 files. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
b08f372022
commit
39a1873e5b
@@ -2,9 +2,9 @@
|
||||
|
||||
## Summary
|
||||
|
||||
- **Feature**: `propose-adjudicate-prove`
|
||||
- **Discovery Scope**: Extension of `three-stage-scan` (MAP grows an overlay; STATIC and REGRESS stay)
|
||||
- **Key Findings**:
|
||||
- Feature: `propose-adjudicate-prove`
|
||||
- Discovery Scope: Extension of `three-stage-scan` (MAP grows an overlay; STATIC and REGRESS stay)
|
||||
- Key Findings:
|
||||
- Inventory already decides; MAP ranks; REGRESS can run. Nothing adjudicates flow.
|
||||
- Labeled SAST slice Youden is about +9%. Leaks are OWASP-false / Juliet `goodG2B` that still contain the sink. Misses are one-hop assignments and sink aliases.
|
||||
- Clearwing already does the split we want: tree-sitter callgraph + pattern taint tables, then ranker (surface/influence/reachability 0.5/0.2/0.3), then mechanism-memory embeddings. OpenUltraSAST already copied the rank formula and has `json-local` vectors; they are not wired as prove-budget.
|
||||
@@ -15,55 +15,55 @@
|
||||
|
||||
### Labeled vs unlabeled code
|
||||
|
||||
- **Context**: Operator: known-vuln corpora are “nice”; written code will not be.
|
||||
- **Sources Consulted**: In-tree `LANGUAGE_MANIFESTS`, `benchmarks/pairs/`, OWASP Benchmark true/false files, Juliet `bad`/`goodG2B`/`goodB2G`, CWE-Bench-Java catalog.
|
||||
- **Findings**:
|
||||
- Context: Operator: known-vuln corpora are "nice"; written code will not be.
|
||||
- Sources Consulted: In-tree `LANGUAGE_MANIFESTS`, `benchmarks/pairs/`, OWASP Benchmark true/false files, Juliet `bad`/`goodG2B`/`goodB2G`, CWE-Bench-Java catalog.
|
||||
- Findings:
|
||||
- Cheat-sheets: one CWE, named sink, often same-line source.
|
||||
- OWASP: true and false files both often contain `exec` / `eval`; the difference is source constancy or infeasible branches.
|
||||
- Juliet `goodG2B`: hardcoded source, **same sink**.
|
||||
- Juliet `goodB2G`: same source, **sanitized sink** (`printf("%s")`).
|
||||
- Working trees: syntax errors, generated bundles, wrappers (`helpers.db_sqlite.execute`), no labels.
|
||||
- **Implications**: Overlay scored on sast slice; unlabeled scans must not pretend demote. Unadjudicated is a first-class disposition.
|
||||
- Implications: Overlay scored on sast slice; unlabeled scans must not pretend demote. Unadjudicated is a first-class disposition.
|
||||
|
||||
### Engine choice
|
||||
|
||||
- **Context**: Operator prefers tree-sitter + Joern look, embeddings before prove, still v1-focused.
|
||||
- **Sources Consulted**: `clearwing/sourcehunt/callgraph.py`, `taint.py`, `ranker.py`, `mechanism_memory.py`; `openultrasast/rank.py`, `vectorstore.py`, `index.py`.
|
||||
- **Findings**:
|
||||
- Context: Operator prefers tree-sitter + Joern look, embeddings before prove, still v1-focused.
|
||||
- Sources Consulted: `clearwing/sourcehunt/callgraph.py`, `taint.py`, `ranker.py`, `mechanism_memory.py`; `openultrasast/rank.py`, `vectorstore.py`, `index.py`.
|
||||
- Findings:
|
||||
- Clearwing tree-sitter is optional: missing grammar disables callgraph, does not crash.
|
||||
- Clearwing taint is intra-procedural, pattern tables for sources/sinks — same shape as our fact TOML.
|
||||
- Clearwing taint is intra-procedural, pattern tables for sources/sinks, same shape as our fact TOML.
|
||||
- Clearwing ranker fills reachability from the callgraph; taint_paths boost surface. Embeddings retrieve *mechanisms*, they do not decide taint.
|
||||
- Our `heuristic_rank` already uses surface·0.5 + influence·0.2 + reachability·0.3. Vector store bakeoff selected `json-local`. `VALID_NAMESPACES` already includes `mechanisms` but nothing writes Clearwing-style records into it.
|
||||
|
||||
### Clearwing vector-store reference (real-world bugs)
|
||||
|
||||
- **Context**: Operator: Clearwing’s store is effective on real bugs, not textbook files.
|
||||
- **Sources Consulted**: `clearwing/sourcehunt/mechanism_memory.py`, `data/memory/semantic_memory.py`, `docs/architecture.md` steps 7–8; `openultrasast/index.py` namespaces.
|
||||
- **Findings**:
|
||||
- Store **mechanisms**, not findings and not 80-line file windows. Example: “length field trusted before alloc; 16-bit value widened to size_t.”
|
||||
- Context: Operator: Clearwing's store is effective on real bugs, not textbook files.
|
||||
- Sources Consulted: `clearwing/sourcehunt/mechanism_memory.py`, `data/memory/semantic_memory.py`, `docs/architecture.md` steps 7–8; `openultrasast/index.py` namespaces.
|
||||
- Findings:
|
||||
- Store **mechanisms**, not findings and not 80-line file windows. Example: "length field trusted before alloc; 16-bit value widened to size_t."
|
||||
- JSONL is the source of truth (`~/.clearwing/sourcehunt/mechanisms.jsonl`). Chroma/sqlite embeddings are ephemeral caches rebuilt from JSONL.
|
||||
- Write path is **verified findings only** (after adversarial verify / crash). Suspicion never seeds the store.
|
||||
- Clearwing’s JSONL + verified-write + metadata filters are the reference. Its TF-IDF default is **not**: bag-of-words will not match paraphrased real bugs (`length field trusted before alloc` vs `user-controlled size_t in malloc`).
|
||||
- Clearwing's JSONL + verified-write + metadata filters are the reference. Its TF-IDF default is **not**: bag-of-words will not match paraphrased real bugs (`length field trusted before alloc` vs `user-controlled size_t in malloc`).
|
||||
- OpenUltraSAST already has `OpenRouterEmbeddingClient` and json-local cosine. Use that as the only semantic ranker; CWE/language/tags are hard filters. No TF-IDF, no Chroma, no local MiniLM.
|
||||
- Query is language + file tags + optional free text; same-language boost +0.5. Top-N injects hunter context and variant-grep seeds — it does not decide taint.
|
||||
- Query is language + file tags + optional free text; same-language boost +0.5. Top-N injects hunter context and variant-grep seeds. It does not decide taint.
|
||||
- Variant loop then greps the *messy* tree for structural matches of that mechanism. That is how it leaves Juliet behind.
|
||||
- **Implications**: Copy JSONL + verified-write + metadata filters. Rank with OpenRouter embeddings only. Do not copy TF-IDF. Do not embed 80-line file windows as the prove store. Seed from sandbox-proven mechanisms, not OWASP `BenchmarkTest00007`.
|
||||
- Implications: Copy JSONL + verified-write + metadata filters. Rank with OpenRouter embeddings only. Do not copy TF-IDF. Do not embed 80-line file windows as the prove store. Seed from sandbox-proven mechanisms, not OWASP `BenchmarkTest00007`.
|
||||
- Joern CLI (`joern-parse` / scripts) is the only practical CPG without a Scala runtime in-process.
|
||||
- **Implications**: Probe tree-sitter like Docker. Joern sidecar cannot demote. Embeddings order prove only.
|
||||
- Implications: Probe tree-sitter like Docker. Joern sidecar cannot demote. Embeddings order prove only.
|
||||
|
||||
### Decision: Python intra-file first
|
||||
|
||||
- **Context**: Zero-dep core vs multi-language trees.
|
||||
- **Selected**: **Superseded.** Tree-sitter is the prio engine; stdlib `ast` is Python last resort; `quick` stays zero-dep.
|
||||
- **Rationale**: Operator working trees are mixed-language; Python-only adjudication would leave JS/C/Java as perpetual unadjudicated.
|
||||
- **Trade-offs**: `standard` needs a grammar or CLI on PATH; missing tools → unadjudicated, not skip the scan.
|
||||
- Context: Zero-dep core vs multi-language trees.
|
||||
- Selected: **Superseded.** Tree-sitter is the prio engine; stdlib `ast` is Python last resort; `quick` stays zero-dep.
|
||||
- Rationale: Operator working trees are mixed-language; Python-only adjudication would leave JS/C/Java as perpetual unadjudicated.
|
||||
- Trade-offs: `standard` needs a grammar or CLI on PATH; missing tools → unadjudicated, not skip the scan.
|
||||
|
||||
### Evidence ladder
|
||||
|
||||
- **Context**: Three-stage-scan forbids hunter JSON as corroboration.
|
||||
- **Sources Consulted**: `verification.EvidenceLevel`, `tool_hunter` suspicion rule, `worth_fixing` gate.
|
||||
- **Findings**: Overlay can justify promote at `static_corroboration`. It cannot skip to crash or exploit. Worth-fixing stays sandbox + reachability.
|
||||
- **Implications**: Disposition is not an evidence rung. Do not add a seventh ladder value.
|
||||
- Context: Three-stage-scan forbids hunter JSON as corroboration.
|
||||
- Sources Consulted: `verification.EvidenceLevel`, `tool_hunter` suspicion rule, `worth_fixing` gate.
|
||||
- Findings: Overlay can justify promote at `static_corroboration`. It cannot skip to crash or exploit. Worth-fixing stays sandbox + reachability.
|
||||
- Implications: Disposition is not an evidence rung. Do not add a seventh ladder value.
|
||||
|
||||
## Architecture Pattern Evaluation
|
||||
|
||||
@@ -78,28 +78,28 @@
|
||||
|
||||
### Decision: Unadjudicated is not demote
|
||||
|
||||
- **Context**: Messy files and cross-file flow.
|
||||
- **Alternatives**: Silent drop; treat as safe; force demote.
|
||||
- **Selected**: Explicit `unadjudicated` with reason.
|
||||
- **Rationale**: Operator writing code must not read silence as “clean.”
|
||||
- **Trade-offs**: Reports stay noisy until languages and inter-file land.
|
||||
- **Follow-up**: Count unadjudicated rate on unlabeled trees as an honesty metric.
|
||||
- Context: Messy files and cross-file flow.
|
||||
- Alternatives: Silent drop; treat as safe; force demote.
|
||||
- Selected: Explicit `unadjudicated` with reason.
|
||||
- Rationale: Operator writing code must not read silence as "clean."
|
||||
- Trade-offs: Reports stay noisy until languages and inter-file land.
|
||||
- Follow-up: Count unadjudicated rate on unlabeled trees as an honesty metric.
|
||||
|
||||
### Decision: Score overlay on sast slice, inventory on smoke gate
|
||||
|
||||
- **Context**: +9% Youden vs 93% cheat-sheet recall.
|
||||
- **Selected**: Two gates stay separate; overlay Youden cannot unblock smoke failure.
|
||||
- **Rationale**: Same as detection-gate vs score in the improve loop.
|
||||
- Context: +9% Youden vs 93% cheat-sheet recall.
|
||||
- Selected: Two gates stay separate; overlay Youden cannot unblock smoke failure.
|
||||
- Rationale: Same as detection-gate vs score in the improve loop.
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
- False demote on real bugs (infeasible-branch OWASP tricks, wrappers) — never demote when flow is incomplete; only demote when the engine *has* a dominating constant or sanitizer.
|
||||
- Youden chase via extra sinks — each coverage rule needs a false/good counterpart that must stay demoted or silent.
|
||||
- Parse failures on operator WIP — unadjudicated, `quick` still works.
|
||||
- False demote on real bugs (infeasible-branch OWASP tricks, wrappers). Never demote when flow is incomplete; only demote when the engine *has* a dominating constant or sanitizer.
|
||||
- Youden chase via extra sinks. Each coverage rule needs a false/good counterpart that must stay demoted or silent.
|
||||
- Parse failures on operator WIP, unadjudicated, `quick` still works.
|
||||
|
||||
## References
|
||||
|
||||
- `.kiro/specs/three-stage-scan/design.md` — stage plan, sandbox, hunter suspicion
|
||||
- `benchmarks/pairs/sast/README.md` — OWASP/Juliet slice
|
||||
- `.kiro/specs/three-stage-scan/design.md`, stage plan, sandbox, hunter suspicion
|
||||
- `benchmarks/pairs/sast/README.md`, OWASP/Juliet slice
|
||||
- NIST Juliet `goodG2B` vs `goodB2G` templates
|
||||
- OWASP Benchmark expectedresults true/false protocol
|
||||
|
||||
Reference in New Issue
Block a user