mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
docs(spec): add OpenRouter SAST harness spec
This commit is contained in:
@@ -0,0 +1,308 @@
|
||||
# Assessment: OpenUltraSAST Kickoff
|
||||
|
||||
## Executive Assessment
|
||||
|
||||
OpenUltraSAST should be built as a harness runtime specialized for security analysis, not as a direct Clearwing clone. The strongest design is to treat Clearwing's sourcehunt pipeline as a proven reference flow, HarnessX as the runtime abstraction, OpenUltraCode as the verification and fusion discipline, OpenRouter as the provider plane, and Trail of Bits skills as targeted domain expertise.
|
||||
|
||||
The minimum viable system is not deep fuzzing. The minimum viable system is an auditable quick scan that can rank files, generate structured findings, eliminate obvious false positives, enforce evidence levels, and emit reports. Deep Docker-backed validation should come after the evidence model, trace model, false-positive loop, and report model are stable.
|
||||
|
||||
## Why This Project Is Viable
|
||||
|
||||
Clearwing demonstrates that a SAST + LLM hybrid can be useful when it avoids treating model output as truth. Its best transferable ideas are:
|
||||
|
||||
- Static preprocessing before LLM analysis.
|
||||
- File ranking by surface, influence, and reachability.
|
||||
- Tiered hunter budgets instead of equal attention for every file.
|
||||
- Independent adversarial verification.
|
||||
- Evidence ladder that separates suspicion from crash reproduction and patch validation.
|
||||
- Docker isolation for builds, fuzzing, PoCs, and patches.
|
||||
- SARIF, Markdown, and JSON artifacts.
|
||||
|
||||
The most important product differentiator should be false-positive elimination. A scanner that finds many plausible issues but cannot separate evidence-backed findings from model hallucinations will not be trusted. Ranking, retrieval, verification, and HarnessX-style adjustment should all be optimized around reducing false positives without hiding the audit trail.
|
||||
|
||||
HarnessX provides the missing generalization: the scanner should be a composed harness with typed processors and traces, not a fixed pipeline that becomes hard to evolve. This matters because SAST workflows vary heavily by language, build system, vulnerability class, and available evidence.
|
||||
|
||||
OpenUltraCode provides the missing discipline: high-risk findings and proposed fixes need bounded plans, adversarial review, reconciliation, explicit model disclosure, and verification gates. This is directly applicable to false-positive control and to avoiding fixes that look plausible but introduce regressions.
|
||||
|
||||
Trail of Bits skills provide mapping discipline as much as they provide tools. CodeQL thinking maps interprocedural flows, Semgrep maps repeatable patterns and variants, differential review maps changed attack surface, and sharp-edge review maps APIs and defaults that make misuse likely. OpenUltraSAST should convert those disciplines into structured tasks and evidence, not only prompt inspiration.
|
||||
|
||||
## Recommended Initial Product Shape
|
||||
|
||||
The first release should support this workflow:
|
||||
|
||||
```text
|
||||
ousast scan /path/to/repo --mode quick
|
||||
```
|
||||
|
||||
It should produce:
|
||||
|
||||
```text
|
||||
.openultrasast/runs/<scan-id>/manifest.json
|
||||
.openultrasast/runs/<scan-id>/preprocess/file_targets.json
|
||||
.openultrasast/runs/<scan-id>/rank/ranking.json
|
||||
.openultrasast/runs/<scan-id>/findings.json
|
||||
.openultrasast/runs/<scan-id>/report.md
|
||||
```
|
||||
|
||||
The key product promise should be:
|
||||
|
||||
```text
|
||||
OpenUltraSAST tells you where security review should start, what it suspects, what evidence exists, what evidence is missing, and which findings have survived independent verification.
|
||||
```
|
||||
|
||||
## Core Architecture Recommendation
|
||||
|
||||
Use a Python implementation initially. Clearwing and HarnessX are both Python, Docker and SARIF libraries are mature, and Trail of Bits tooling integration will be simpler. OpenCode integration can happen through MCP and command files rather than requiring the core scanner to be TypeScript.
|
||||
|
||||
Recommended modules:
|
||||
|
||||
```text
|
||||
openultrasast/config.py
|
||||
openultrasast/cli.py
|
||||
openultrasast/models.py
|
||||
openultrasast/provider/openrouter.py
|
||||
openultrasast/preprocess/
|
||||
openultrasast/index/
|
||||
openultrasast/rank/
|
||||
openultrasast/harness/
|
||||
openultrasast/hunters/
|
||||
openultrasast/verify/
|
||||
openultrasast/fusion/
|
||||
openultrasast/sandbox/
|
||||
openultrasast/reports/
|
||||
openultrasast/mcp/
|
||||
```
|
||||
|
||||
## Build vs. Reuse Decisions
|
||||
|
||||
| Area | Recommendation | Reason |
|
||||
| --- | --- | --- |
|
||||
| Harness runtime | Build minimal HarnessX-inspired core first | Direct HarnessX dependency may add unstable surface before needs are proven |
|
||||
| OpenRouter chat | Build direct OpenAI-compatible client | Need explicit model IDs, cost metadata, and role controls |
|
||||
| OpenRouter embeddings | Build direct client | Embeddings are central and should not be hidden behind chat abstractions |
|
||||
| Vector store | Choose one local default | Avoid abstracting before retrieval needs are proven |
|
||||
| Docker sandbox | Build focused wrapper | Security defaults are project-specific |
|
||||
| Trail of Bits skills | Index descriptors, mapping tasks, and snippets | Avoid prompt bloat while preserving CodeQL/Semgrep/differential/sharp-edge discipline |
|
||||
| Clearwing | Reference only | Independence is a project requirement |
|
||||
|
||||
## Configuration Shape
|
||||
|
||||
Model names and the vector store should be selected before implementation. The config shape should look like this:
|
||||
|
||||
```toml
|
||||
[models]
|
||||
ranker = "<openrouter-ranker-model>"
|
||||
hunter = "<openrouter-hunter-model>"
|
||||
verifier = "<openrouter-verifier-model>"
|
||||
patcher = "<openrouter-patcher-model>"
|
||||
|
||||
[embeddings]
|
||||
model = "<openrouter-embedding-model>"
|
||||
store = "<selected-local-vector-store>"
|
||||
|
||||
[sandbox]
|
||||
network = false
|
||||
workspace_readonly = true
|
||||
memory_mb = 2048
|
||||
timeout_seconds = 300
|
||||
pids_limit = 512
|
||||
|
||||
[dynamic]
|
||||
enabled = false
|
||||
network_scope = []
|
||||
|
||||
[evidence]
|
||||
minimum_report_verified = "static_corroboration"
|
||||
minimum_exploit = "crash_reproduced"
|
||||
minimum_patch = "root_cause_explained"
|
||||
```
|
||||
|
||||
The actual model names should be explicit before implementation. Avoid baking in placeholder defaults that silently pick a moving target.
|
||||
|
||||
## Vector Store Decision
|
||||
|
||||
Choosing the vector store is a core design decision, not plumbing. It directly affects ranking quality and false-positive rates because poor retrieval contaminates hunter and verifier context.
|
||||
|
||||
Selection criteria:
|
||||
|
||||
- Strong metadata filters for path, language, symbol, repository snapshot, evidence level, and vulnerability class.
|
||||
- Incremental indexing for changed files.
|
||||
- Local persistence that works inside opencode workflows.
|
||||
- Deterministic export/import for reproducible scan artifacts.
|
||||
- Acceptable query latency on medium repositories.
|
||||
- Clean storage of OpenRouter embedding model metadata.
|
||||
- Ability to combine vector retrieval with lexical or structural filters.
|
||||
|
||||
The first milestone should include a short bakeoff on fixtures before locking the default store.
|
||||
|
||||
## Most Important Engineering Constraint
|
||||
|
||||
Evidence transitions must be enforced in code. A model can recommend a transition, but the harness should decide whether the required artifact exists.
|
||||
|
||||
Examples:
|
||||
|
||||
- `suspicion -> static_corroboration`: requires static evidence, independent verifier agreement, or a matching analyzer finding.
|
||||
- `static_corroboration -> crash_reproduced`: requires stored crash/reproducer artifact.
|
||||
- `crash_reproduced -> root_cause_explained`: requires verifier explanation tying crash to source and impact.
|
||||
- `root_cause_explained -> exploit_demonstrated`: requires exploit or trigger artifact beyond accidental crash.
|
||||
- `root_cause_explained -> patch_validated`: requires patch diff and passing validation command.
|
||||
|
||||
False-positive elimination should be handled the same way: a model can recommend demotion or suppression, but the harness should require a scoped reason and counter-evidence.
|
||||
|
||||
False-positive reasons should include:
|
||||
|
||||
- No attacker-controlled path.
|
||||
- Code is unreachable in supported configuration.
|
||||
- Static rule matched a safe wrapper or sanitizer.
|
||||
- Model confused similar symbols or files.
|
||||
- Finding duplicates an existing issue.
|
||||
- Impact is not security-relevant.
|
||||
- Dynamic probe or test disproved triggerability.
|
||||
|
||||
## OpenRouter Design Needs
|
||||
|
||||
OpenRouter should be treated as two capabilities:
|
||||
|
||||
1. Chat/completion routing for reasoning roles.
|
||||
2. Embedding endpoint for retrieval.
|
||||
|
||||
Every model call should record:
|
||||
|
||||
```text
|
||||
role
|
||||
model_id
|
||||
provider metadata
|
||||
prompt hash
|
||||
response hash
|
||||
cost estimate
|
||||
temperature and parameters
|
||||
fallback/degradation status
|
||||
```
|
||||
|
||||
The project should reject ambiguous model routing for reproducible scans. If a role uses auto routing, the manifest must record the actual model selected by OpenRouter if available.
|
||||
|
||||
## Docker Design Needs
|
||||
|
||||
Docker is not just convenience. It is the trust boundary between arbitrary target repositories and the user's host.
|
||||
|
||||
Minimum safety profile:
|
||||
|
||||
- No hidden network in analysis containers.
|
||||
- Read-only source mount.
|
||||
- Writable scratch only.
|
||||
- Timeouts around every command.
|
||||
- Memory, CPU, and pids limits.
|
||||
- Dropped capabilities by default.
|
||||
- Separate writable copy for patch validation.
|
||||
|
||||
Deep mode should not be implemented until this boundary exists.
|
||||
|
||||
## Dynamic Analysis Strategy
|
||||
|
||||
Network verification is not out of scope. It is just not the core idea of the harness. OpenUltraSAST should remain SAST-first, but it should use dynamic analysis when runtime evidence is the right way to confirm a finding.
|
||||
|
||||
Useful dynamic checks include:
|
||||
|
||||
- Starting a target service in a sandbox.
|
||||
- Using netcat-style TCP probes against declared local ports.
|
||||
- Running local HTTP or protocol smoke tests.
|
||||
- Executing a bounded PoC against a sandboxed service.
|
||||
- Using Clearwing-style network checks as validation patterns when a source finding has a runtime network surface.
|
||||
|
||||
The rule is not "no network ever." The rule is "no hidden or open-ended network activity." Dynamic probes must be explicit, scoped, logged, time-bounded, and tied to evidence advancement.
|
||||
|
||||
## Trail Of Bits Skill Strategy
|
||||
|
||||
The skills are best used as a routing corpus and checklist library. The harness should select compact snippets by target profile.
|
||||
|
||||
Good examples:
|
||||
|
||||
- C parser with pointer arithmetic: `c-review`, `address-sanitizer`, `libfuzzer`, `harness-writing`.
|
||||
- Python parser: `atheris`, `property-based-testing`.
|
||||
- Crypto code: `constant-time-analysis`, `wycheproof`, `vector-forge`, `zeroize-audit`.
|
||||
- SARIF flow: `sarif-parsing`, `semgrep`, `codeql`.
|
||||
- Changed code or proposed fixes: `differential-review`, `graph-evolution`.
|
||||
- Dangerous APIs, defaults, or configuration: `sharp-edges`, `insecure-defaults`.
|
||||
- Smart contracts: use chain-specific scanners only when the language and framework match.
|
||||
|
||||
Bad pattern:
|
||||
|
||||
```text
|
||||
Paste all security skill instructions into every hunter prompt.
|
||||
```
|
||||
|
||||
That would increase cost, reduce precision, and mix irrelevant assumptions into reviews.
|
||||
|
||||
## Fix Authoring And Audit Strategy
|
||||
|
||||
UltraCode should define the patch lifecycle. A patch is not complete when a model emits a diff. A patch is complete only after a bounded plan, minimal implementation, adversarial review, reconciliation, and fresh verification.
|
||||
|
||||
Fix stages:
|
||||
|
||||
```text
|
||||
verified finding -> bounded fix plan -> minimal patch -> sandbox validation -> adversarial fix audit -> reconciliation -> fresh verification -> patch artifact
|
||||
```
|
||||
|
||||
Trail of Bits disciplines should audit the fix:
|
||||
|
||||
- Differential review checks whether the patch changed more attack surface than intended.
|
||||
- Semgrep checks whether the vulnerable pattern or variants remain.
|
||||
- CodeQL checks whether taint/dataflow still reaches the sink when applicable.
|
||||
- Sharp-edge review checks whether the fix leaves a misleading or misuse-prone API behind.
|
||||
|
||||
Fusion should always be used to go deeper when the task requires it: risky fixes, contradictory evidence, difficult remediation tradeoffs, high-impact reports, unresolved verifier disagreement, or explicit high-assurance review. It should not be spent on routine low-risk patches when ordinary evidence is already sufficient.
|
||||
|
||||
## Kiro Spec Completion Criteria
|
||||
|
||||
The current spec is ready to drive implementation when these decisions are made:
|
||||
|
||||
1. Language/runtime: Python is recommended.
|
||||
2. Vector store: choose Chroma, LanceDB, Qdrant local, or SQLite vector extension.
|
||||
3. Initial OpenRouter chat models by role.
|
||||
4. Initial OpenRouter embedding model.
|
||||
5. First fixture repository or synthetic vulnerable examples.
|
||||
6. Whether to depend directly on HarnessX or implement a minimal compatible harness core.
|
||||
|
||||
## Recommended MVP Cut
|
||||
|
||||
MVP should include:
|
||||
|
||||
- Local path scan.
|
||||
- File enumeration and static tags.
|
||||
- OpenRouter ranker.
|
||||
- Quick-mode hunter.
|
||||
- Evidence ladder with enforced transitions.
|
||||
- Independent verifier.
|
||||
- JSON and Markdown reports.
|
||||
- Manifest with model IDs and prompt hashes.
|
||||
|
||||
MVP should exclude:
|
||||
|
||||
- Deep fuzzing and broad dynamic analysis.
|
||||
- Auto-patching.
|
||||
- Fusion panels for routine MVP findings.
|
||||
- Full MCP server.
|
||||
- Multi-repository campaigns.
|
||||
- Training bridge.
|
||||
|
||||
MVP should still model the future fix lifecycle in the data model, because evidence, reports, and patch artifacts need stable IDs before patching exists.
|
||||
|
||||
These exclusions keep the first milestone testable and prevent architecture from hardening around unverified assumptions.
|
||||
|
||||
## Implementation Order Rationale
|
||||
|
||||
Do not start with Docker or broad fusion. Those are expensive and only pay off after the data model and evidence model are correct. Start with the smallest useful loop:
|
||||
|
||||
```text
|
||||
preprocess -> rank -> quick finding -> verify -> report
|
||||
```
|
||||
|
||||
Once this loop works on fixtures, add embeddings. Once retrieval is stable, add sandboxed commands. Once sandboxing is stable, add deep fuzzing. Add fusion as soon as tasks require deeper adjudication than the normal verifier can provide.
|
||||
|
||||
## Immediate Next Questions
|
||||
|
||||
1. Which OpenRouter chat models should be used for ranker, hunter, verifier, and patcher roles?
|
||||
2. Which OpenRouter embedding model should be the default?
|
||||
3. Which vector store should be selected for the first implementation?
|
||||
4. Should the project start with synthetic fixtures or scan a real open-source target first?
|
||||
5. Should the first implementation include MCP, or should MCP wait until the CLI and artifacts stabilize?
|
||||
@@ -0,0 +1,614 @@
|
||||
# Design: OpenRouter SAST Harness
|
||||
|
||||
## Architecture Summary
|
||||
|
||||
OpenUltraSAST is a harness-oriented SAST system, not a monolithic scanner. The core runtime composes processors, model roles, retrieval, tools, sandbox policy, traces, verification gates, and reporting into a reproducible scan harness.
|
||||
|
||||
The central quality objective is false-positive elimination. The harness should not merely find more possible bugs; it should repeatedly narrow suspicion into evidence-backed findings and learn from rejected claims.
|
||||
|
||||
Conceptual lineage:
|
||||
|
||||
- HarnessX: harness primitives, event hooks, processor contracts, trace-driven evolution.
|
||||
- OpenUltraCode: bounded context, fix authoring discipline, adversarial review, reconciliation, fusion, verification evidence.
|
||||
- Clearwing sourcehunt: preprocess, rank, tiered hunters, evidence ladder, adversarial verifier, patch oracle, variant loop, reports.
|
||||
- Trail of Bits skills: domain-specific mapping, analysis, audit, and verification methods selected at the right stage, not embedded globally.
|
||||
|
||||
## High-Level Flow
|
||||
|
||||
```text
|
||||
repo path
|
||||
-> intake snapshot
|
||||
-> preprocess and static tags
|
||||
-> graph and optional SARIF ingest
|
||||
-> embeddings and retrieval index
|
||||
-> file/function ranking
|
||||
-> tiered hunter pool
|
||||
-> independent verifier
|
||||
-> optional fusion adjudication
|
||||
-> optional reproducer/exploit triage
|
||||
-> optional patch oracle
|
||||
-> reports and artifacts
|
||||
```
|
||||
|
||||
## Main Components
|
||||
|
||||
### CLI
|
||||
|
||||
Initial command surface:
|
||||
|
||||
```text
|
||||
ousast scan <path> --mode quick|standard|deep --config openultrasast.toml
|
||||
ousast status <scan-id>
|
||||
ousast findings <scan-id>
|
||||
ousast report <scan-id> --format markdown|json|sarif
|
||||
ousast patch <finding-id> --propose
|
||||
ousast index <path> --rebuild
|
||||
```
|
||||
|
||||
The CLI should be thin. It loads config, creates a scan, starts the harness runtime, and writes artifacts.
|
||||
|
||||
### MCP Server
|
||||
|
||||
MCP should expose only stable project operations:
|
||||
|
||||
```text
|
||||
openultrasast.scan
|
||||
openultrasast.status
|
||||
openultrasast.findings
|
||||
openultrasast.get_finding
|
||||
openultrasast.evidence
|
||||
openultrasast.artifacts
|
||||
openultrasast.propose_patch
|
||||
openultrasast.export_report
|
||||
```
|
||||
|
||||
The MCP server must not expose arbitrary shell, Docker, or internal hunter tools. Those stay inside controlled harness stages.
|
||||
|
||||
### Harness Runtime
|
||||
|
||||
The runtime owns the scan lifecycle and event pipeline. It should follow the HarnessX separation:
|
||||
|
||||
```text
|
||||
ModelConfig: role -> OpenRouter model, parameters, budget, fallback policy
|
||||
HarnessConfig: processors, tools, memory, sandbox, trace, control policy
|
||||
ScanConfig: target path, mode, scope, excludes, report options
|
||||
```
|
||||
|
||||
Required event hooks:
|
||||
|
||||
```text
|
||||
task_start
|
||||
stage_start
|
||||
before_model
|
||||
after_model
|
||||
before_tool
|
||||
after_tool
|
||||
stage_end
|
||||
task_end
|
||||
```
|
||||
|
||||
Each processor declares the event types it handles and the state fields it reads or writes. Strict contract mode fails the scan when processors violate schemas. Warn mode records degraded state.
|
||||
|
||||
### OpenRouter Provider
|
||||
|
||||
The provider layer should implement two explicit clients:
|
||||
|
||||
```text
|
||||
OpenRouterChatClient
|
||||
OpenRouterEmbeddingClient
|
||||
```
|
||||
|
||||
Chat roles:
|
||||
|
||||
```text
|
||||
ranker
|
||||
hunter
|
||||
verifier
|
||||
fusion_panel_a
|
||||
fusion_panel_b
|
||||
fusion_decider
|
||||
patcher
|
||||
reporter
|
||||
```
|
||||
|
||||
Provider rules:
|
||||
|
||||
- No silent model substitution.
|
||||
- Every model response records model ID, provider metadata when available, token counts, cost estimate, prompt hash, and response hash.
|
||||
- Fallbacks must be configured per role and reported as degradations.
|
||||
- Embedding model ID must be stored in the vector index metadata.
|
||||
|
||||
### Retrieval Store
|
||||
|
||||
The retrieval layer maintains separate namespaces:
|
||||
|
||||
```text
|
||||
repo_code
|
||||
repo_docs
|
||||
static_findings
|
||||
mechanisms
|
||||
trailofbits_skills
|
||||
scan_traces
|
||||
false_positive_learnings
|
||||
```
|
||||
|
||||
Chunk metadata should include:
|
||||
|
||||
```json
|
||||
{
|
||||
"repo_id": "...",
|
||||
"snapshot": "...",
|
||||
"path": "src/parser.c",
|
||||
"symbol": "parse_frame",
|
||||
"language": "c",
|
||||
"chunk_type": "function",
|
||||
"start_line": 120,
|
||||
"end_line": 220,
|
||||
"tags": ["parser", "memory_unsafe"],
|
||||
"embedding_model": "openrouter/..."
|
||||
}
|
||||
```
|
||||
|
||||
Retrieval packages must be bounded by role. A verifier gets the finding, relevant source, static corroboration, and counter-evidence candidates. It does not get hunter chain-of-thought.
|
||||
|
||||
#### Vector Store Selection Criteria
|
||||
|
||||
The vector store is a core ranking and false-positive-control dependency. The first implementation should choose one default after evaluating:
|
||||
|
||||
```text
|
||||
metadata filtering precision
|
||||
incremental reindexing by repository snapshot
|
||||
local persistence and portability
|
||||
query latency on medium repositories
|
||||
embedding model metadata support
|
||||
hybrid lexical/vector retrieval support or integration path
|
||||
deterministic export/import for scan reproducibility
|
||||
operational simplicity for opencode users
|
||||
```
|
||||
|
||||
Ranking depends on retrieval quality. Poor metadata filters will contaminate verifier context and increase false positives. The store must make it easy to retrieve "same symbol, same package, same vulnerability class, same evidence level" rather than semantically similar but irrelevant code.
|
||||
|
||||
### Preprocessor
|
||||
|
||||
The preprocessor produces `FileTarget` records. It should start simple and grow by plugins.
|
||||
|
||||
Initial fields:
|
||||
|
||||
```text
|
||||
path
|
||||
absolute_path
|
||||
language
|
||||
loc
|
||||
tags
|
||||
imports_by
|
||||
transitive_callers
|
||||
static_hints
|
||||
sarif_hints
|
||||
reachability_hints
|
||||
has_fuzz_entry_point
|
||||
defines_security_constants
|
||||
```
|
||||
|
||||
Initial tags:
|
||||
|
||||
```text
|
||||
memory_unsafe
|
||||
parser
|
||||
crypto
|
||||
deserialization
|
||||
syscall_entry
|
||||
network_entry
|
||||
filesystem_entry
|
||||
fuzzable
|
||||
privileged
|
||||
```
|
||||
|
||||
### Static Analysis Mapping Layer
|
||||
|
||||
The static analysis layer turns tool output and skill discipline into a map for the harness. It is not only a source of finished findings.
|
||||
|
||||
Primary mapping sources:
|
||||
|
||||
```text
|
||||
Semgrep: syntactic and lightweight semantic patterns, custom rule iteration, variant seeds
|
||||
CodeQL: interprocedural dataflow, taint paths, structural vulnerability queries
|
||||
Differential review: changed attack surfaces, regression-prone patches, new trust-boundary crossings
|
||||
Sharp-edge review: unsafe APIs, insecure defaults, confusing configuration, misuse-prone interfaces
|
||||
SARIF parsing: normalized evidence, rule metadata, locations, severity, fingerprints
|
||||
```
|
||||
|
||||
Mapping outputs:
|
||||
|
||||
```text
|
||||
static_hints on FileTarget
|
||||
sarif_hints on FileTarget
|
||||
taint_paths and source/sink pairs
|
||||
variant_search_seeds
|
||||
changed_surface records
|
||||
sharp_edge records
|
||||
evidence candidates for verifier
|
||||
```
|
||||
|
||||
This layer should use Trail of Bits skills as task templates and checklists. For example, the CodeQL discipline supplies query/dataflow thinking, Semgrep supplies pattern and variant loops, differential review supplies changed-code blast-radius analysis, and sharp-edge review supplies API misuse mapping.
|
||||
|
||||
### Ranker
|
||||
|
||||
Ranking should combine heuristics, graph features, static hints, and model judgment.
|
||||
|
||||
Scores:
|
||||
|
||||
```text
|
||||
surface: direct vulnerability likelihood, 1-5
|
||||
influence: downstream blast radius, 1-5
|
||||
reachability: attacker-controlled path likelihood, 1-5
|
||||
priority: surface * 0.5 + influence * 0.2 + reachability * 0.3
|
||||
```
|
||||
|
||||
The ranker should emit concise rationales:
|
||||
|
||||
```json
|
||||
{
|
||||
"path": "src/parser.c",
|
||||
"surface": 5,
|
||||
"influence": 4,
|
||||
"reachability": 4,
|
||||
"priority": 4.5,
|
||||
"rationale": "C parser with external input, pointer arithmetic, and many callers",
|
||||
"model_id": "openrouter/...",
|
||||
"static_boosts": ["memory_unsafe", "parser", "sarif_high"]
|
||||
}
|
||||
```
|
||||
|
||||
Ranking is not only prioritization; it is a false-positive filter. The ranker should use verifier outcomes and dynamic evidence outcomes to calibrate which surfaces deserve attention. Repeated rejected patterns should be demoted unless new evidence appears.
|
||||
|
||||
Ranking metrics:
|
||||
|
||||
```text
|
||||
verified findings per tier
|
||||
false positives per tier
|
||||
false-positive reason distribution
|
||||
time-to-verification per tier
|
||||
retrieval hit quality per role
|
||||
missed known-vulnerable fixture targets
|
||||
```
|
||||
|
||||
### False-Positive Elimination Loop
|
||||
|
||||
HarnessX-style adjustment is the mechanism that reduces false positives over time. Each scan produces traces, verifier decisions, rejected hypotheses, static analyzer deltas, retrieval packages, and evidence transitions. The harness should use those outputs to adjust future runs.
|
||||
|
||||
Adjustment targets:
|
||||
|
||||
```text
|
||||
ranker score calibration
|
||||
retrieval filters and namespace weights
|
||||
hunter prompt constraints
|
||||
skill routing rules
|
||||
verifier tie-breaker templates
|
||||
static mapping seeds
|
||||
variant search patterns
|
||||
finding deduplication rules
|
||||
```
|
||||
|
||||
False-positive records:
|
||||
|
||||
```text
|
||||
finding_id
|
||||
rejection_reason
|
||||
counter_evidence
|
||||
scope of suppression or demotion
|
||||
affected ranker/static/retrieval inputs
|
||||
verifier model_id
|
||||
human_override if present
|
||||
```
|
||||
|
||||
False-positive learning must be scoped. A rejected claim in one framework, version, path, or call pattern should not globally suppress a vulnerability class.
|
||||
|
||||
### Hunter Pool
|
||||
|
||||
The hunter pool schedules targets by priority tier and depth.
|
||||
|
||||
Suggested tiers:
|
||||
|
||||
```text
|
||||
A: priority >= 3.0, most budget
|
||||
B: priority >= 2.0, moderate budget
|
||||
C: priority < 2.0, opportunistic budget
|
||||
```
|
||||
|
||||
Hunter inputs:
|
||||
|
||||
```text
|
||||
target metadata
|
||||
bounded source context
|
||||
static hints
|
||||
retrieved related code
|
||||
selected Trail of Bits skill snippets
|
||||
sandbox capabilities
|
||||
budget
|
||||
evidence requirements
|
||||
```
|
||||
|
||||
Hunter outputs must be structured and evidence-oriented. They should not rely on hidden reasoning.
|
||||
|
||||
### Verifier
|
||||
|
||||
The verifier runs independently from the hunter. It receives only auditable facts:
|
||||
|
||||
```text
|
||||
finding claim
|
||||
source excerpts
|
||||
static evidence
|
||||
retrieved relevant context
|
||||
reproducer or crash artifacts when available
|
||||
negative evidence candidates
|
||||
```
|
||||
|
||||
Verifier output:
|
||||
|
||||
```json
|
||||
{
|
||||
"is_real": true,
|
||||
"severity": "high",
|
||||
"confidence": 0.82,
|
||||
"evidence_level": "static_corroboration",
|
||||
"pro_case": "...",
|
||||
"counter_case": "...",
|
||||
"tie_breaker": "Build a reproducer that drives len past buffer capacity",
|
||||
"required_next_step": "attempt_crash_reproduction"
|
||||
}
|
||||
```
|
||||
|
||||
### Fusion Adjudication
|
||||
|
||||
Fusion is the mandatory deepening mechanism whenever the task requires more reasoning depth than the normal ranker, hunter, verifier, and static mapping loop can provide.
|
||||
|
||||
Fusion triggers:
|
||||
|
||||
- Critical or high severity findings.
|
||||
- Findings with verifier disagreement.
|
||||
- Findings that gate patching or disclosure.
|
||||
- Findings where static and semantic evidence conflict.
|
||||
- Findings where a patch could introduce regression risk.
|
||||
- Findings where multiple skills disagree on exploitability or remediation.
|
||||
- User or task requests for deeper analysis, ultrathink-style review, or high-assurance adjudication.
|
||||
|
||||
Fusion protocol:
|
||||
|
||||
```text
|
||||
panel A independently reviews bounded evidence
|
||||
panel B independently reviews bounded evidence
|
||||
panels critique each other
|
||||
panels revise
|
||||
panels vote/rank
|
||||
decider issues final disposition
|
||||
reconciler records accepted/rejected/blocked items
|
||||
verifier records fresh evidence
|
||||
```
|
||||
|
||||
Fusion should not be forced for routine low-severity findings when ordinary evidence is already sufficient. It must run when the task requires deeper analysis, when evidence does not converge, or when the decision gates a risky fix, exploit claim, disclosure, or high-impact report.
|
||||
|
||||
### Docker Sandbox
|
||||
|
||||
Sandbox defaults:
|
||||
|
||||
```text
|
||||
network: none
|
||||
workspace: read-only
|
||||
scratch: writable tmpfs or volume
|
||||
cap_drop: all
|
||||
cap_add: SYS_PTRACE only when sanitizer/debugger needs it
|
||||
memory: bounded
|
||||
cpu: bounded
|
||||
pids: bounded
|
||||
runtime: docker default, optional gVisor
|
||||
```
|
||||
|
||||
Sandbox use cases:
|
||||
|
||||
```text
|
||||
static tool execution
|
||||
build system probing
|
||||
sanitizer builds
|
||||
fuzz harness compilation
|
||||
PoC execution
|
||||
local service startup
|
||||
netcat-style local socket probes
|
||||
protocol smoke tests
|
||||
patch validation in writable copy
|
||||
```
|
||||
|
||||
Patch validation must use a writable copy, never the user's working tree by default.
|
||||
|
||||
### Dynamic Analysis Boundary
|
||||
|
||||
OpenUltraSAST is SAST-first, but dynamic analysis is in scope when it provides necessary evidence. The harness may run controlled runtime checks such as service startup, local HTTP/TCP probes, netcat-style reachability checks, PoC execution, and Clearwing-style network validation patterns.
|
||||
|
||||
Dynamic analysis rules:
|
||||
|
||||
```text
|
||||
disabled by default
|
||||
enabled only by explicit config or task-required evidence
|
||||
scoped to declared hosts, ports, containers, and commands
|
||||
logged as artifacts
|
||||
bounded by timeout and resource limits
|
||||
never treated as an open-ended network pentest loop
|
||||
```
|
||||
|
||||
Dynamic evidence can support transitions such as `static_corroboration -> crash_reproduced`, `static_corroboration -> root_cause_explained`, or `root_cause_explained -> exploit_demonstrated` when the artifact actually demonstrates the claim.
|
||||
|
||||
### Trail Of Bits Skill Integration
|
||||
|
||||
Trail of Bits skills should be indexed and selected by routing metadata.
|
||||
|
||||
Examples:
|
||||
|
||||
```text
|
||||
C/C++ memory target -> c-review, address-sanitizer, libfuzzer, harness-writing
|
||||
Python parser target -> atheris, property-based-testing
|
||||
Rust target -> cargo-fuzz, zeroize-audit when secrets are present
|
||||
Crypto target -> constant-time-analysis, wycheproof, vector-forge
|
||||
SARIF processing -> sarif-parsing
|
||||
Semgrep mapping loop -> semgrep, semgrep-rule-creator, semgrep-rule-variant-creator
|
||||
CodeQL mapping loop -> codeql, sarif-parsing
|
||||
Changed-code review -> differential-review, graph-evolution
|
||||
API footgun review -> sharp-edges, insecure-defaults
|
||||
Smart contract target -> chain-specific scanner skill
|
||||
```
|
||||
|
||||
Do not paste entire skills into every prompt. Store descriptors, retrieval chunks, and short operating checklists. Pull only the relevant snippets into a role-specific context package.
|
||||
|
||||
### UltraCode Fix Workflow
|
||||
|
||||
OpenUltraCode discipline should govern fix authoring and fix audit. The patch oracle is not a single prompt that writes code and declares success.
|
||||
|
||||
Fix workflow:
|
||||
|
||||
```text
|
||||
intake verified finding and evidence
|
||||
create bounded fix plan
|
||||
generate minimal patch in writable sandbox or worktree
|
||||
run affected checks and reproducer
|
||||
run adversarial fix review
|
||||
reconcile accepted/rejected/deferred review items
|
||||
run fresh verification
|
||||
emit patch diff and validation report
|
||||
```
|
||||
|
||||
Fix audit inputs:
|
||||
|
||||
```text
|
||||
original finding evidence
|
||||
patch diff
|
||||
differential analysis of changed code
|
||||
Semgrep/CodeQL result delta when available
|
||||
tests and reproducer output
|
||||
sharp-edge regression checklist when API/config behavior changes
|
||||
```
|
||||
|
||||
The workflow can propose a fix without applying it. It can only mark a fix as ready when the adversarial review is reconciled and fresh verification evidence exists.
|
||||
|
||||
### Data Model
|
||||
|
||||
Core entities:
|
||||
|
||||
```text
|
||||
Scan
|
||||
RepositorySnapshot
|
||||
HarnessRun
|
||||
ProcessorTrace
|
||||
ModelCall
|
||||
FileTarget
|
||||
Finding
|
||||
Evidence
|
||||
Artifact
|
||||
PatchProposal
|
||||
VerificationDecision
|
||||
FusionDecision
|
||||
FalsePositiveLearning
|
||||
RankingCalibration
|
||||
```
|
||||
|
||||
Finding fields:
|
||||
|
||||
```text
|
||||
id
|
||||
scan_id
|
||||
title
|
||||
description
|
||||
severity
|
||||
confidence
|
||||
status
|
||||
evidence_level
|
||||
path
|
||||
start_line
|
||||
end_line
|
||||
symbol
|
||||
vulnerability_class
|
||||
attack_scenario
|
||||
impact
|
||||
remediation
|
||||
discovered_by
|
||||
verified_by
|
||||
model_ids
|
||||
artifact_refs
|
||||
created_at
|
||||
updated_at
|
||||
```
|
||||
|
||||
Evidence levels:
|
||||
|
||||
```text
|
||||
suspicion
|
||||
static_corroboration
|
||||
crash_reproduced
|
||||
root_cause_explained
|
||||
exploit_demonstrated
|
||||
patch_validated
|
||||
```
|
||||
|
||||
### Artifact Layout
|
||||
|
||||
Suggested scan directory:
|
||||
|
||||
```text
|
||||
.openultrasast/runs/<scan-id>/
|
||||
manifest.json
|
||||
config.resolved.json
|
||||
repo_snapshot.json
|
||||
preprocess/
|
||||
index/
|
||||
rank/
|
||||
traces/
|
||||
model_calls/
|
||||
sandbox/
|
||||
findings.json
|
||||
report.md
|
||||
report.sarif
|
||||
patches/
|
||||
```
|
||||
|
||||
## Implementation Strategy
|
||||
|
||||
Build the system in vertical slices:
|
||||
|
||||
1. Quick scan: intake, preprocess, rank, LLM/static finding generation, JSON/Markdown report.
|
||||
2. Verification: independent verifier, evidence ladder enforcement, SARIF export.
|
||||
3. Retrieval: OpenRouter embeddings, persistent index, bounded context packages.
|
||||
4. Static mapping: Semgrep/CodeQL SARIF ingest, differential mapping, sharp-edge records.
|
||||
5. False-positive loop: rejected finding taxonomy, ranking calibration, retrieval adjustment, and scoped suppressions.
|
||||
6. Sandbox: Docker command runner, static tools, read-only source, scratch volume.
|
||||
7. Dynamic evidence: controlled service startup, local probes, PoC execution, and artifact capture.
|
||||
8. Deep mode: sanitizer builds, fuzz harness generation, crash artifacts.
|
||||
9. Patch mode: UltraCode-style fix authoring, adversarial fix audit, validation loop.
|
||||
10. MCP/opencode integration: narrow tools and commands.
|
||||
11. Fusion: mandatory deepening for task-required hard findings, contradictory evidence, and risky fixes.
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
1. Evidence level is a state machine, not a label the model can freely assign.
|
||||
2. Verifier context excludes hunter reasoning to reduce contamination.
|
||||
3. OpenRouter model selection is role-scoped and auditable.
|
||||
4. Embeddings and vector-store metadata filtering are part of the runtime, not an offline optional add-on.
|
||||
5. Docker isolation is mandatory for execution-heavy modes.
|
||||
6. Trail of Bits skills are retrieved as scoped expertise, not global prompt bloat.
|
||||
7. MCP surface is narrow and project-oriented.
|
||||
8. Fixes are authored and audited through an UltraCode-style workflow, not a one-shot patch prompt.
|
||||
9. Fusion is the required deepening path when the task demands high-assurance reasoning, not a replacement for routine analysis.
|
||||
10. Harness adjustment loops must reduce repeated false positives without hiding auditable evidence.
|
||||
|
||||
## Risks
|
||||
|
||||
1. OpenRouter embedding availability and model-specific response formats may vary.
|
||||
2. Deep mode can become expensive without strict budget controls.
|
||||
3. Docker sandbox escape risk requires conservative defaults and no-network execution.
|
||||
4. LLM findings can inflate false positives unless the evidence ladder is enforced in code.
|
||||
5. Multi-language callgraph quality will vary; ranking must tolerate partial graph data.
|
||||
6. Skill retrieval can overfit prompts if descriptors are too broad.
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. Which local vector store should be the default: Chroma, LanceDB, SQLite vector extension, or Qdrant local?
|
||||
2. Should the first implementation be pure Python like Clearwing/HarnessX, or TypeScript to align with more opencode extension tooling?
|
||||
3. Should OpenUltraSAST vendor a minimal HarnessX-inspired runtime or depend directly on HarnessX?
|
||||
4. What minimum OpenRouter embedding model should be required for acceptable code retrieval?
|
||||
5. Should patch proposals be applied only through git worktrees?
|
||||
@@ -0,0 +1,173 @@
|
||||
# Requirements: OpenRouter SAST Harness
|
||||
|
||||
## Overview
|
||||
|
||||
OpenUltraSAST shall be an independent OpenCode-driven security harness for finding, validating, reporting, and optionally fixing source-code vulnerabilities at scale. It shall use OpenRouter-selected chat models for reasoning, OpenRouter embeddings for semantic retrieval, Docker for isolated analysis, and an explicit evidence ladder to prevent AI suspicion from being reported as verified security impact.
|
||||
|
||||
The project shall use Clearwing only as an implementation oracle. It shall not import Clearwing internals, fork Clearwing, or expose Clearwing's broad MCP surface.
|
||||
|
||||
## Goals
|
||||
|
||||
1. Provide a repeatable SAST harness that can scan local repositories and produce ranked, evidence-scored findings.
|
||||
2. Make false-positive elimination a primary product goal by separating suspicion, corroboration, verification, reproduction, and patch validation.
|
||||
3. Combine static analysis, semantic retrieval, LLM hunters, adversarial verification, and patch validation into one opencode-native workflow.
|
||||
4. Keep model routing explicit through OpenRouter, with per-role model configuration and no silent substitution.
|
||||
5. Preserve Docker isolation as a first-class security boundary for code execution, harness compilation, fuzzing, and patch validation.
|
||||
6. Integrate Trail of Bits skills as scoped security expertise selected by language, framework, vulnerability class, mapping need, or verification need.
|
||||
7. Emit machine-readable artifacts suitable for CI, triage, and later training or harness evolution.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
1. Do not make network pentesting the core workflow.
|
||||
2. Do not reimplement all of Clearwing.
|
||||
3. Do not report LLM-only claims as verified vulnerabilities.
|
||||
4. Do not require a local embedding model.
|
||||
5. Do not depend on a single OpenRouter model.
|
||||
6. Do not mutate user repositories by default.
|
||||
|
||||
## Functional Requirements
|
||||
|
||||
### Repository Intake
|
||||
|
||||
1. The system shall accept a local repository path as the primary scan target.
|
||||
2. The system should later support git URLs, but initial implementation may require local checkout.
|
||||
3. The system shall snapshot basic repository metadata: root path, commit hash when available, language mix, file count, and ignored paths.
|
||||
4. The system shall respect common ignore files such as `.gitignore`, `.ignore`, and explicit harness config excludes.
|
||||
|
||||
### Preprocessing
|
||||
|
||||
1. The system shall enumerate source files and assign initial language, LOC, and path metadata.
|
||||
2. The system shall tag files using lightweight static heuristics, including memory-unsafe code, parser surfaces, crypto surfaces, authentication boundaries, syscall or process boundaries, serialization boundaries, and fuzzable entry points.
|
||||
3. The system shall build a dependency or call graph where practical.
|
||||
4. The system shall ingest optional static analyzer output such as Semgrep SARIF or CodeQL SARIF.
|
||||
5. The system shall preserve preprocessing artifacts for later ranking, retrieval, and audit.
|
||||
|
||||
### Static Analysis And Mapping Discipline
|
||||
|
||||
1. The system shall treat Semgrep, CodeQL, differential review, and sharp-edge/API review as mapping disciplines, not just optional external scanners.
|
||||
2. The system shall ingest Semgrep and CodeQL output as SARIF and normalize it into static hints, evidence candidates, and variant-search seeds.
|
||||
3. The system shall support differential analysis between a base and head revision to identify changed attack surfaces, newly reachable code, and regression-prone fixes.
|
||||
4. The system shall support sharp-edge mapping for APIs, configuration, defaults, cryptographic interfaces, and dangerous framework patterns that create predictable misuse.
|
||||
5. The system shall preserve analyzer provenance so each finding can cite which static tool, rule, diff, or skill-derived checklist contributed evidence.
|
||||
|
||||
### OpenRouter Embeddings And Retrieval
|
||||
|
||||
1. The system shall use OpenRouter embedding endpoints for semantic indexing.
|
||||
2. The system shall chunk source code, docs, static findings, prior mechanisms, and Trail of Bits skill descriptors into separate collections or namespaces.
|
||||
3. The system shall store vectors in a local persistent vector database by default.
|
||||
4. The system shall support reusing an existing index when the repository snapshot has not changed.
|
||||
5. The system shall retrieve bounded context packages for rankers, hunters, verifiers, and patchers.
|
||||
6. The system shall evaluate vector-store candidates against ranking quality, metadata filtering, incremental indexing, local persistence, cost, and reproducibility before selecting a default.
|
||||
7. The vector store shall support precise metadata filters for path, language, symbol, vulnerability class, evidence level, scan ID, and repository snapshot.
|
||||
|
||||
### Ranking
|
||||
|
||||
1. The system shall score candidate files on at least three axes: surface, influence, and reachability.
|
||||
2. The system shall compute a composite priority using an explicit formula, initially `surface * 0.5 + influence * 0.2 + reachability * 0.3`.
|
||||
3. The system shall keep low-surface high-influence files eligible for analysis.
|
||||
4. The system shall combine static heuristics, dependency graph data, embeddings, and OpenRouter LLM scoring.
|
||||
5. The system shall record score rationales and model identity.
|
||||
6. The system shall measure ranking quality with fixture repositories and prior scan outcomes, including false-positive rate by tier and missed-true-positive rate where ground truth exists.
|
||||
7. The system shall feed verifier outcomes back into ranking calibration so repeated false-positive patterns lose priority over time.
|
||||
|
||||
### False-Positive Elimination Loop
|
||||
|
||||
1. The system shall track every rejected, contradicted, duplicate, unsupported, and unverified finding as false-positive learning data.
|
||||
2. The system shall record why a finding failed: unreachable path, missing attacker control, sanitizer disproved, static rule mismatch, incorrect model assumption, duplicate finding, or insufficient impact.
|
||||
3. The system shall use HarnessX-style adjustment loops to update prompts, retrieval filters, ranking calibration, skill routing, verifier tie-breakers, and static mapping seeds from false-positive outcomes.
|
||||
4. The system shall keep false-positive learnings auditable and reversible; learned suppressions shall cite evidence and scope.
|
||||
5. The system shall prefer evidence-gated demotion over deletion so analysts can inspect why a suspected issue was suppressed or downgraded.
|
||||
|
||||
### Harness Execution
|
||||
|
||||
1. The system shall model the scan as a HarnessX-style harness composed of processors, tools, memory, sandbox, trace, and model roles.
|
||||
2. The system shall expose stage hooks for task start, stage start, before model, after model, before tool, after tool, stage end, and task end.
|
||||
3. The system shall contract-check processor inputs and outputs in strict mode for CI and warn mode for local exploration.
|
||||
4. The system shall serialize harness configuration, processor versions, model roles, and prompt hashes into scan artifacts.
|
||||
|
||||
### Hunter Pool
|
||||
|
||||
1. The system shall fan out analysis across ranked targets using budgeted tiers.
|
||||
2. The system shall support at least three analysis depths: quick, standard, and deep.
|
||||
3. Quick mode shall rely on static analysis, ranking, and bounded LLM review without code execution.
|
||||
4. Standard mode shall use sandboxed tool-calling hunters and adversarial verification.
|
||||
5. Deep mode shall add harness generation, sanitizer builds, fuzzing, dynamic reproduction, crash reproduction, and patch validation where applicable.
|
||||
6. The system shall record every hunter trajectory as structured JSONL.
|
||||
|
||||
### Evidence Ladder
|
||||
|
||||
1. Every finding shall carry exactly one current evidence level.
|
||||
2. Evidence levels shall be ordered as: `suspicion`, `static_corroboration`, `crash_reproduced`, `root_cause_explained`, `exploit_demonstrated`, `patch_validated`.
|
||||
3. Findings below `static_corroboration` shall not be reported as verified.
|
||||
4. Exploit or PoC generation shall require at least `crash_reproduced` or equivalent trigger evidence.
|
||||
5. Auto-patching shall require at least `root_cause_explained` and shall only advance to `patch_validated` after tests or reproducer validation pass.
|
||||
|
||||
### Verification And Fusion
|
||||
|
||||
1. The system shall verify findings using independent context that excludes hunter chain-of-thought and raw hunter reasoning.
|
||||
2. The verifier shall steel-man both the vulnerability case and the false-positive case.
|
||||
3. The verifier shall output explicit tie-breaker evidence requirements.
|
||||
4. The system shall support OpenUltraCode-style fusion for task-required deepening, high-impact findings, uncertain findings, risky fixes, or unresolved disagreement, using two panels and an explicit decider.
|
||||
5. Fusion output shall disclose model IDs, panel roles, degradations, warnings, votes, and final decision source.
|
||||
6. Every adversarial finding shall receive a disposition: accepted, rejected, mitigated, deferred, or blocked.
|
||||
7. Fusion shall be mandatory whenever the task requires deeper reasoning than the normal ranker, hunter, verifier, and static mapping loop can provide.
|
||||
8. Fusion shall not be forced for routine low-risk issues unless the task explicitly asks for deeper analysis or the evidence remains unresolved.
|
||||
|
||||
### Docker Sandbox
|
||||
|
||||
1. The system shall run untrusted build, test, fuzz, and PoC commands inside Docker by default.
|
||||
2. Analysis containers shall have network disabled by default.
|
||||
3. Source mounts shall be read-only except in explicit patch workspaces.
|
||||
4. Containers shall use a writable scratch volume or tmpfs.
|
||||
5. Containers shall enforce CPU, memory, pids, timeout, and capability limits.
|
||||
6. The system shall support sanitizer variants, initially ASan and UBSan for C/C++ targets.
|
||||
7. The system should support gVisor or another hardened runtime when available.
|
||||
|
||||
### Dynamic Analysis
|
||||
|
||||
1. The system shall support optional dynamic analysis when runtime evidence is needed to verify, reproduce, or de-risk a finding.
|
||||
2. Dynamic analysis may include service startup, local socket checks, netcat-style probes, protocol smoke tests, PoC execution, or Clearwing-style network validation patterns.
|
||||
3. Dynamic network access shall be disabled by default and enabled only through explicit scan configuration or task-required evidence collection.
|
||||
4. Dynamic probes shall run in sandboxed environments with declared network scope, timeouts, command logs, and captured artifacts.
|
||||
5. Dynamic analysis shall contribute evidence but shall not turn OpenUltraSAST into a general network pentest agent.
|
||||
|
||||
### Patch Oracle
|
||||
|
||||
1. The system shall optionally generate minimal defensive patches for verified findings.
|
||||
2. The patch workflow shall follow OpenUltraCode discipline: intake, bounded plan, implementation, adversarial review, reconciliation, and fresh verification.
|
||||
3. The patch oracle shall target the smallest safe change.
|
||||
4. The patch oracle shall not rewrite unrelated code or change public signatures unless required and approved.
|
||||
5. Patch validation shall rerun the reproducer, affected tests, or relevant build checks in a writable sandbox.
|
||||
6. Patch artifacts shall be emitted as diffs, not silently applied to the user's repository by default.
|
||||
7. The system shall audit proposed fixes for regression risk using differential analysis and relevant Trail of Bits skills before marking them ready.
|
||||
|
||||
### Reporting
|
||||
|
||||
1. The system shall emit JSON, SARIF, Markdown, and a manifest per scan.
|
||||
2. Reports shall sort findings by severity, evidence level, reachability, and confidence.
|
||||
3. Reports shall include model IDs, static evidence, verifier results, reproduction evidence, patch status, and artifact paths.
|
||||
4. Reports shall distinguish suspected, verified, exploited, and patch-validated findings.
|
||||
|
||||
### MCP And OpenCode Integration
|
||||
|
||||
1. The project shall expose a narrow MCP server for OpenCode integration.
|
||||
2. MCP tools shall focus on project-level operations: scan, status, findings, evidence, artifacts, patch proposal, and report export.
|
||||
3. The MCP server shall not expose arbitrary internal tools as public MCP methods.
|
||||
4. OpenCode commands shall invoke the harness with explicit model role configuration and visible assumptions.
|
||||
|
||||
## Non-Functional Requirements
|
||||
|
||||
1. The system shall be deterministic where possible: fixed config, artifact manifests, prompt hashes, and model identifiers.
|
||||
2. The system shall degrade visibly when optional tools, Docker, embeddings, or specific models are unavailable.
|
||||
3. The system shall avoid hidden network access from analysis containers; any dynamic network access shall be explicit, scoped, logged, and artifact-backed.
|
||||
4. The system shall support CI usage with bounded runtime and cost budgets.
|
||||
5. The system shall keep findings auditable from report back to source, trace, model call, verifier decision, and artifact.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
1. A user can run a local repository scan in quick mode and receive ranked static/LLM findings with JSON and Markdown reports.
|
||||
2. A user can run standard mode and receive adversarially verified findings with explicit evidence levels.
|
||||
3. A user can run deep mode on a small C/C++ parser target and receive sandboxed build/fuzz artifacts when the target is fuzzable.
|
||||
4. A finding cannot be marked verified unless it reaches at least `static_corroboration`.
|
||||
5. A patch cannot be marked validated unless a sandboxed validation command passes.
|
||||
6. A scan manifest records repository snapshot, harness config, OpenRouter model IDs, embedding model, prompt hashes, and artifact paths.
|
||||
@@ -0,0 +1,184 @@
|
||||
# Tasks: OpenRouter SAST Harness
|
||||
|
||||
## Phase 0: Project Skeleton
|
||||
|
||||
- [ ] Create Python package skeleton with CLI entry point `ousast`.
|
||||
- [ ] Add config loading for `openultrasast.toml` and environment variables.
|
||||
- [ ] Add resolved config artifact writer.
|
||||
- [ ] Add scan run directory creation under `.openultrasast/runs/<scan-id>/`.
|
||||
- [ ] Add basic tests for config loading and run directory creation.
|
||||
|
||||
## Phase 1: Intake And Preprocess
|
||||
|
||||
- [ ] Implement repository snapshot collection.
|
||||
- [ ] Implement source file enumeration with ignore handling.
|
||||
- [ ] Implement language detection from extension and shebang.
|
||||
- [ ] Implement LOC counting.
|
||||
- [ ] Implement initial tag heuristics for parser, memory unsafe, crypto, auth boundary, deserialization, syscall, network, filesystem, and fuzzable targets.
|
||||
- [ ] Emit `preprocess/file_targets.json`.
|
||||
- [ ] Add tests for tag heuristics on small fixture repositories.
|
||||
|
||||
## Phase 2: Ranking
|
||||
|
||||
- [ ] Define `FileTarget` and ranking schemas.
|
||||
- [ ] Implement heuristic score floors and boosts.
|
||||
- [ ] Implement OpenRouter chat client for ranker role.
|
||||
- [ ] Implement chunked ranker calls with JSON response validation.
|
||||
- [ ] Compute composite priority.
|
||||
- [ ] Emit ranking rationales and model metadata.
|
||||
- [ ] Add tests for composite priority and malformed model response handling.
|
||||
|
||||
## Phase 3: Quick Scan Reports
|
||||
|
||||
- [ ] Implement static finding schema.
|
||||
- [ ] Implement quick-mode hunter prompt and bounded file context.
|
||||
- [ ] Implement structured finding extraction.
|
||||
- [ ] Enforce initial evidence level `suspicion` unless static corroboration exists.
|
||||
- [ ] Emit `findings.json`.
|
||||
- [ ] Emit Markdown report.
|
||||
- [ ] Add smoke test against a fixture repository with one known insecure pattern.
|
||||
|
||||
## Phase 4: OpenRouter Embeddings
|
||||
|
||||
- [ ] Implement OpenRouter embedding client.
|
||||
- [ ] Run a vector-store bakeoff for metadata filtering, local persistence, incremental indexing, export/import, and retrieval quality.
|
||||
- [ ] Select default local vector store based on bakeoff results.
|
||||
- [ ] Implement code chunking with path, symbol, and line metadata.
|
||||
- [ ] Implement namespaces for repo code, docs, static findings, mechanisms, skills, and traces.
|
||||
- [ ] Implement index reuse keyed by repository snapshot and embedding model.
|
||||
- [ ] Implement bounded retrieval package builder per role.
|
||||
- [ ] Add tests for chunk metadata and retrieval filters.
|
||||
|
||||
## Phase 4A: Ranking Quality And False-Positive Loop
|
||||
|
||||
- [ ] Define false-positive reason taxonomy.
|
||||
- [ ] Add false-positive learning records for rejected, duplicate, unsupported, contradicted, and unverified findings.
|
||||
- [ ] Track ranking metrics by tier: verified findings, false positives, time to verification, and missed fixture vulnerabilities.
|
||||
- [ ] Feed verifier outcomes into ranking calibration.
|
||||
- [ ] Feed rejected findings into retrieval filter adjustment and prompt constraints.
|
||||
- [ ] Add tests proving scoped false-positive demotion does not globally suppress the vulnerability class.
|
||||
|
||||
## Phase 5: Harness Runtime
|
||||
|
||||
- [ ] Define harness event model.
|
||||
- [ ] Define processor interface and contract checks.
|
||||
- [ ] Implement strict and warn modes.
|
||||
- [ ] Add trace writer for events and processor state changes.
|
||||
- [ ] Serialize harness config, prompt hashes, processor versions, and model roles.
|
||||
- [ ] Add tests for processor contract violations.
|
||||
|
||||
## Phase 5A: Static Mapping Disciplines
|
||||
|
||||
- [ ] Add SARIF ingestion for Semgrep and CodeQL outputs.
|
||||
- [ ] Normalize SARIF rules, locations, severity, fingerprints, and provenance into static hints.
|
||||
- [ ] Add Semgrep mapping task records for pattern, variant, and rule-authoring loops.
|
||||
- [ ] Add CodeQL mapping task records for source, sink, sanitizer, and path explanations.
|
||||
- [ ] Add differential mapping records for changed files, changed functions, and changed trust-boundary crossings.
|
||||
- [ ] Add sharp-edge records for insecure defaults, dangerous APIs, confusing config, and misuse-prone interfaces.
|
||||
- [ ] Add tests for mapping analyzer output to `FileTarget` hints and verifier evidence candidates.
|
||||
|
||||
## Phase 6: Verification
|
||||
|
||||
- [ ] Define evidence ladder state machine.
|
||||
- [ ] Implement independent verifier prompt and schema.
|
||||
- [ ] Ensure verifier context excludes hunter reasoning.
|
||||
- [ ] Implement pro-case, counter-case, tie-breaker, and required-next-step fields.
|
||||
- [ ] Enforce that verified report status requires at least `static_corroboration`.
|
||||
- [ ] Add tests for invalid evidence transitions.
|
||||
|
||||
## Phase 7: Docker Sandbox
|
||||
|
||||
- [ ] Implement Docker availability probe.
|
||||
- [ ] Implement no-network read-only workspace sandbox runner.
|
||||
- [ ] Add explicit scoped-network sandbox mode for task-required dynamic probes.
|
||||
- [ ] Add scratch volume or tmpfs support.
|
||||
- [ ] Add CPU, memory, pids, timeout, and capability limits.
|
||||
- [ ] Add command execution artifact capture.
|
||||
- [ ] Add sanitizer build environment for C/C++ fixtures.
|
||||
- [ ] Add tests that verify workspace remains read-only.
|
||||
|
||||
## Phase 7A: Dynamic Evidence
|
||||
|
||||
- [ ] Implement dynamic analysis configuration with `enabled = false` by default.
|
||||
- [ ] Implement service startup command records with declared ports and timeouts.
|
||||
- [ ] Implement netcat-style local socket probe tool inside sandbox scope.
|
||||
- [ ] Implement HTTP or protocol smoke-test artifact capture.
|
||||
- [ ] Tie dynamic artifacts to evidence transitions without allowing model-only upgrades.
|
||||
- [ ] Add tests that reject undeclared network targets.
|
||||
|
||||
## Phase 8: Standard Mode Hunter Pool
|
||||
|
||||
- [ ] Implement tier assignment by priority.
|
||||
- [ ] Implement budget allocation and per-target limits.
|
||||
- [ ] Implement parallel hunter scheduling.
|
||||
- [ ] Persist hunter trajectories as JSONL.
|
||||
- [ ] Integrate retrieval packages and selected skill snippets.
|
||||
- [ ] Integrate verifier after hunter output.
|
||||
- [ ] Add fixture scan demonstrating at least one verified static-corroborated finding.
|
||||
|
||||
## Phase 9: SARIF And CI Output
|
||||
|
||||
- [ ] Implement SARIF report writer.
|
||||
- [ ] Implement machine-readable manifest.
|
||||
- [ ] Add severity, confidence, evidence level, model IDs, and artifact refs to reports.
|
||||
- [ ] Add CI-friendly exit code policy.
|
||||
- [ ] Add tests for SARIF schema validity.
|
||||
|
||||
## Phase 10: Deep Mode
|
||||
|
||||
- [ ] Implement fuzzable target detection beyond existing fuzz entry points.
|
||||
- [ ] Implement fuzz harness generation prompt and schema.
|
||||
- [ ] Compile generated harnesses in sandbox.
|
||||
- [ ] Run libFuzzer or equivalent with sanitizer instrumentation.
|
||||
- [ ] Capture crashes and minimization artifacts.
|
||||
- [ ] Advance findings to `crash_reproduced` only with artifact evidence.
|
||||
- [ ] Add a small C/C++ parser fixture with reproducible sanitizer crash.
|
||||
|
||||
## Phase 11: Patch Oracle
|
||||
|
||||
- [ ] Implement patch proposal schema.
|
||||
- [ ] Implement OpenUltraCode-style fix lifecycle states: intake, plan, implement, adversarial_review, reconcile, verify, ready.
|
||||
- [ ] Run patching in a writable sandbox copy or git worktree.
|
||||
- [ ] Generate minimal diffs only.
|
||||
- [ ] Rerun reproducer and affected checks.
|
||||
- [ ] Run differential fix audit against the generated patch.
|
||||
- [ ] Run Semgrep or CodeQL delta checks when corresponding evidence exists.
|
||||
- [ ] Run sharp-edge fix audit when API, config, auth, crypto, or defaults are changed.
|
||||
- [ ] Advance findings to `patch_validated` only after validation passes.
|
||||
- [ ] Add tests that reject patches changing unrelated files.
|
||||
|
||||
## Phase 12: MCP And OpenCode Commands
|
||||
|
||||
- [ ] Implement narrow MCP server.
|
||||
- [ ] Add `openultrasast.scan`, `status`, `findings`, `get_finding`, `evidence`, `artifacts`, `propose_patch`, and `export_report` tools.
|
||||
- [ ] Add opencode command docs for quick, standard, deep, verify, and patch workflows.
|
||||
- [ ] Ensure MCP tools never expose arbitrary shell execution.
|
||||
- [ ] Add integration test for MCP tool listing and scan status.
|
||||
|
||||
## Phase 13: Fusion
|
||||
|
||||
- [ ] Implement fusion trigger policy.
|
||||
- [ ] Trigger fusion whenever a task requires deeper reasoning, including difficult findings, contradictory evidence, high-impact decisions, and risky fixes.
|
||||
- [ ] Implement panel A, panel B, and decider role config.
|
||||
- [ ] Implement cross-critique and revised answer flow.
|
||||
- [ ] Implement vote/rank output schema.
|
||||
- [ ] Implement reconciliation dispositions.
|
||||
- [ ] Record model IDs, degradations, warnings, and final decision source.
|
||||
- [ ] Add tests for fusion output validation.
|
||||
|
||||
## Phase 14: Skill Index
|
||||
|
||||
- [ ] Build Trail of Bits skill descriptor inventory.
|
||||
- [ ] Chunk selected skill operating guidance into retrieval namespace.
|
||||
- [ ] Implement skill router by language, tags, vulnerability class, and stage.
|
||||
- [ ] Add prompt budget caps for skill snippets.
|
||||
- [ ] Add tests for routing C parser targets to memory/fuzzing skills and crypto targets to constant-time/vector skills.
|
||||
|
||||
## Phase 15: Hardening And Release Readiness
|
||||
|
||||
- [ ] Add cost budget enforcement.
|
||||
- [ ] Add model call retry and backoff policies.
|
||||
- [ ] Add redaction for secrets in traces and reports.
|
||||
- [ ] Add docs for threat model and sandbox limits.
|
||||
- [ ] Add end-to-end examples.
|
||||
- [ ] Add lint, typecheck, test, and CI gates.
|
||||
Reference in New Issue
Block a user