Marius CiepluchandClaude Opus 4.8
39a1873e5b
Ran the humanizer rules over 79 prose files in the active specs (brief, design, requirements, research and narrative docs). Removed em dashes, significance inflation, forced rule-of-three, bold-label list items and filler. Skipped tasks.md checklists and the retired specs (harnessx-*, openrouter-sast-harness, unseen-repo-evaluation). Every number, path, command, hash and identifier was preserved: checked each file against HEAD for drift in numbers, inline code and code fences, zero drift across all 79 files. Co-Authored-By: Claude Opus 4.8 <[email protected]>
2.4 KiB
Brief: regex proposes, semantics adjudicates, sandbox proves
Problem
OpenUltraSAST already has three scan stages (quick inventory, standard map, deep regression), but only the first stage actually decides. Pattern rules propose sinks; the map ranks files; the sandbox can run recipes. Nothing in the middle answers "does this sink receive attacker-controlled data, or is it a Juliet-style good twin / dead assignment / hardcoded command?"
Calibration corpora (cheat-sheet fixtures, OWASP Benchmark true/false files, Juliet bad/good, CVE pairs) are nice: one CWE per file, named sinks, labeled counterparts. Real working trees are not: mixed languages, generated code, unparseable files, framework wrappers, comments, one-hop assignments (sql = f"…{x}"; cur.execute(sql)), and no ground-truth labels.
Measured honesty on the labeled SAST slice is about +9% Youden. Widening regex to chase that number will fire on every Runtime.exec and leak on OWASP-false / Juliet goodG2B.
Approach
Keep three roles, never collapse them:
- Propose, pattern rules + optional SARIF. Stage 1. Does not decide worth-fixing.
- Adjudicate, tree-sitter IR (multi-language) + taint facts; optional Joern CPG sidecar; Python stdlib parse only as last-resort fallback. Promote, demote, or unadjudicated.
- Budget, ranker + mechanism memory (JSONL + OpenRouter embeddings, hard CWE/language/tag filters) before prove. It is a budget rather than a verdict; it uses neither TF-IDF nor raw file-slice embeddings.
- Prove, sandbox only on promoted, reachable, budgeted candidates.
The primary in-scope target is a messy working tree the operator is writing, not a labeled benchmark. Known-vuln corpora remain calibration scoreboards (smoke 90/10, pair Youden, GitHub silent-on-fix). They do not define success on unlabeled code.
Scope
- Semantic overlay as MAP-stage adjudication; tree-sitter is the prio engine (CLI/API, fail-closed). Joern optional. Embeddings tune prove order.
- Explicit
unadjudicatedwhen parse or flow fails. Do not treat silence as safe. - Sandbox prove only after promote.
- Pair eval scores the overlay on the sast slice; smoke gate stays inventory-only.
Out of scope
- Replacing pattern rules.
- LLM as parse/taint engine.
- Requiring Joern or embeddings for
quick. - Using embedding similarity to silently drop promotions.
- Auto-writing flow facts from the improve loop.