marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-18 04:29:27 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(harness): add runtime and reachability mapping

Browse Source
This commit is contained in:
Marius Ciepluch
2026-06-18 07:57:06 +02:00
parent 33757db14a
commit f8c690999a
16 changed files with 1339 additions and 53 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/specs/openrouter-sast-harness/design.md
+34 -2
View File
@@ -19,7 +19,7 @@ Conceptual lineage:
repo path
-> intake snapshot
-> preprocess and static tags
-> graph and optional SARIF ingest
-> graph, SARIF ingest, and entry-point reachability mapping
-> embeddings and retrieval index
-> file/function ranking
-> tiered hunter pool
@@ -212,6 +212,7 @@ Primary mapping sources:
```text
Semgrep: syntactic and lightweight semantic patterns, custom rule iteration, variant seeds
CodeQL: interprocedural dataflow, taint paths, structural vulnerability queries
Entry-point analysis: routes, CLI/process boundaries, parser inputs, smart-contract state-changing entry points, callbacks, privileged surfaces
Differential review: changed attack surfaces, regression-prone patches, new trust-boundary crossings
Sharp-edge review: unsafe APIs, insecure defaults, confusing configuration, misuse-prone interfaces
SARIF parsing: normalized evidence, rule metadata, locations, severity, fingerprints
@@ -226,10 +227,41 @@ taint_paths and source/sink pairs
variant_search_seeds
changed_surface records
sharp_edge records
entry_point records
reachability_hints on FileTarget
evidence candidates for verifier
```
This layer should use Trail of Bits skills as task templates and checklists. For example, the CodeQL discipline supplies query/dataflow thinking, Semgrep supplies pattern and variant loops, differential review supplies changed-code blast-radius analysis, and sharp-edge review supplies API misuse mapping.
This layer should use Trail of Bits skills as task templates and checklists. For example, the CodeQL discipline supplies query/dataflow thinking, Semgrep supplies pattern and variant loops, entry-point analysis supplies attacker-controlled surface and access classification, differential review supplies changed-code blast-radius analysis, and sharp-edge review supplies API misuse mapping.
#### Entry-Point Reachability Mapping
Entry-point mapping is a Phase 5A prerequisite for useful ranking and false-positive reduction. Pattern matches and static analyzer alerts are not enough; the harness must identify whether the affected function or symbol is plausibly reachable from attacker-controlled input.
Initial entry-point kinds:
```text
route: HTTP/framework route handlers and request callbacks
cli: process entry points, argparse/click handlers, argv consumers, main functions
parser: parser or deserialization inputs that consume untrusted data
fuzz: explicit fuzz entry points such as LLVMFuzzerTestOneInput
smart_contract_state_change: externally callable non-view contract functions
contract_callback: callbacks and hook implementations invoked by other contracts
privileged_surface: auth boundaries, admin operations, filesystem/process/syscall boundaries
```
Initial access levels:
```text
public
authenticated
role-restricted
contract-only/callback
local-only
review-required
```
Each entry-point record should include path, function name when known, start line, end line, name, kind, access level, trust boundary, access evidence, condition evidence, provenance, and rationale. Access level distinguishes unauthenticated/public from authenticated, role-restricted, local-only, callback-only, and review-required surfaces. Condition evidence captures feature flags, rollout toggles, pause guards, experiment gates, and runtime guard expressions. The preprocessor attaches matching records to `FileTarget.reachability_hints`. Findings use the source line to inherit only function-level entry points whose range contains the finding. The ranker uses these hints to raise file review priority for files containing public routes, parser/fuzz inputs, authenticated surfaces, contract callbacks, and role-restricted privileged surfaces. A finding outside a reachable function should remain eligible for review, but should not inherit the entry point's priority without call-graph evidence.
### Ranker
.kiro/specs/openrouter-sast-harness/requirements.md
+40 -10
View File
@@ -16,6 +16,30 @@ The project shall use Clearwing only as an implementation oracle. It shall not i
6. Integrate Trail of Bits skills as scoped security expertise selected by language, framework, vulnerability class, mapping need, or verification need.
7. Emit machine-readable artifacts suitable for CI, triage, and later training or harness evolution.
## Completeness Model
Implementation progress shall be reported against functional harness capability, not raw checklist percentage. A task may be complete as scaffold while the product capability remains incomplete.
Completion levels:
1. `scaffold`: schemas, CLI shape, local artifacts, tests, or isolated primitives exist.
2. `usable`: the feature runs end-to-end for a narrow local fixture without manual wiring.
3. `integrated`: the feature participates in the scan harness with traces, evidence, reports, and degradation handling.
4. `production`: the feature is robust enough for real repositories, CI budgets, failure recovery, and analyst audit.
Progress reports shall identify both checklist progress and functional completeness. The project shall not describe OpenUltraSAST as a working harness until the `usable_harness_mvp` gate is satisfied.
Functional gates:
1. `scaffold_quick_scan`: quick scan can enumerate files, rank targets, emit basic static findings, and write JSON/Markdown artifacts. This gate does not constitute a security harness.
2. `usable_harness_mvp`: harness runtime, event tracing, static mapping ingestion, evidence ladder enforcement, independent verification, and JSON/Markdown/SARIF reporting work together on fixture repositories.
3. `standard_security_harness`: standard mode runs ranked hunters with retrieval packages, selected security skills, independent verification, false-positive learning, and auditable traces.
4. `sandboxed_dynamic_harness`: Docker sandbox, scoped dynamic probes, sanitizer/build artifacts, and evidence transitions work without hidden network or host mutation.
5. `fix_validation_harness`: patch proposals, adversarial fix audit, differential/static delta checks, and sandbox validation are integrated and cannot mark patches validated without passing evidence.
6. `opencode_product`: MCP tools, opencode commands/skills, fusion deepening, CI-ready reports, cost controls, and hardening are complete enough for routine opencode-driven use.
The current implementation state shall be documented as `scaffold_quick_scan` until later gates are explicitly verified.
## Non-Goals
1. Do not make network pentesting the core workflow.
@@ -46,9 +70,13 @@ The project shall use Clearwing only as an implementation oracle. It shall not i
1. The system shall treat Semgrep, CodeQL, differential review, and sharp-edge/API review as mapping disciplines, not just optional external scanners.
2. The system shall ingest Semgrep and CodeQL output as SARIF and normalize it into static hints, evidence candidates, and variant-search seeds.
3. The system shall support differential analysis between a base and head revision to identify changed attack surfaces, newly reachable code, and regression-prone fixes.
4. The system shall support sharp-edge mapping for APIs, configuration, defaults, cryptographic interfaces, and dangerous framework patterns that create predictable misuse.
5. The system shall preserve analyzer provenance so each finding can cite which static tool, rule, diff, or skill-derived checklist contributed evidence.
3. The system shall support function-level entry-point and reachability analysis to identify attacker-controlled inputs, externally callable functions, public routes, CLI/process boundaries, parser entry points, smart-contract state-changing entry points, callbacks, and privileged/admin-only surfaces.
4. The system shall classify entry points by access level and trust boundary, including public/unrestricted, authenticated, role-restricted, contract-only/callback, local-only, and unknown/review-required.
5. The system shall preserve condition evidence for entry points, including feature flags, rollout toggles, pause guards, experiment gates, and runtime guard expressions that affect whether a threat actor can reach the code.
6. The system shall use function-level reachability modeling to prioritize candidate bugs inside reachable entry-point functions and to demote pattern-only findings that lack a plausible attacker-controlled path.
6. The system shall support differential analysis between a base and head revision to identify changed attack surfaces, newly reachable code, and regression-prone fixes.
7. The system shall support sharp-edge mapping for APIs, configuration, defaults, cryptographic interfaces, and dangerous framework patterns that create predictable misuse.
8. The system shall preserve analyzer provenance so each finding can cite which static tool, rule, diff, entry-point analysis, or skill-derived checklist contributed evidence.
### OpenRouter Embeddings And Retrieval
@@ -65,7 +93,7 @@ The project shall use Clearwing only as an implementation oracle. It shall not i
1. The system shall score candidate files on at least three axes: surface, influence, and reachability.
2. The system shall compute a composite priority using an explicit formula, initially `surface * 0.5 + influence * 0.2 + reachability * 0.3`.
3. The system shall keep low-surface high-influence files eligible for analysis.
4. The system shall combine static heuristics, dependency graph data, embeddings, and OpenRouter LLM scoring.
4. The system shall combine static heuristics, dependency graph data, entry-point reachability, embeddings, and OpenRouter LLM scoring.
5. The system shall record score rationales and model identity.
6. The system shall measure ranking quality with fixture repositories and prior scan outcomes, including false-positive rate by tier and missed-true-positive rate where ground truth exists.
7. The system shall feed verifier outcomes back into ranking calibration so repeated false-positive patterns lose priority over time.
@@ -165,9 +193,11 @@ The project shall use Clearwing only as an implementation oracle. It shall not i
## Acceptance Criteria
1. A user can run a local repository scan in quick mode and receive ranked static/LLM findings with JSON and Markdown reports.
2. A user can run standard mode and receive adversarially verified findings with explicit evidence levels.
3. A user can run deep mode on a small C/C++ parser target and receive sandboxed build/fuzz artifacts when the target is fuzzable.
4. A finding cannot be marked verified unless it reaches at least `static_corroboration`.
5. A patch cannot be marked validated unless a sandboxed validation command passes.
6. A scan manifest records repository snapshot, harness config, OpenRouter model IDs, embedding model, prompt hashes, and artifact paths.
1. `scaffold_quick_scan`: A user can run a local repository scan in quick mode and receive ranked static findings with JSON and Markdown reports. This may use heuristics and static patterns only.
2. `usable_harness_mvp`: A user can run quick or standard fixture scans through the harness runtime and inspect event traces, normalized static mapping evidence, verifier decisions, and report artifacts.
3. `standard_security_harness`: A user can run standard mode and receive adversarially verified findings with explicit evidence levels, selected retrieval context, skill routing, and false-positive learning records.
4. `sandboxed_dynamic_harness`: A user can run deep mode on a small C/C++ parser target and receive sandboxed build/fuzz artifacts when the target is fuzzable.
5. `fix_validation_harness`: A patch cannot be marked validated unless a sandboxed validation command passes and the fix audit has no accepted blocking findings.
6. `opencode_product`: A user can drive scans, triage, evidence inspection, report export, fusion, and patch proposal through narrow MCP/OpenCode interfaces without exposing arbitrary shell execution.
7. A finding cannot be marked verified unless it reaches at least `static_corroboration` through enforced evidence transitions.
8. A scan manifest records repository snapshot, harness config, OpenRouter model IDs, embedding model, prompt hashes, processor versions, evidence transitions, and artifact paths.
.kiro/specs/openrouter-sast-harness/tasks.md
+89 -26
View File
@@ -1,6 +1,19 @@
# Tasks: OpenRouter SAST Harness
## Phase 0: Project Skeleton
## Completion Gates
Checklist completion is not product completion. Each phase now carries a functional gate label:
- `scaffold`: isolated primitives, schemas, local artifacts, or tests exist.
- `usable`: the feature works end-to-end for fixtures without manual wiring.
- `integrated`: the feature participates in the harness runtime, traces, evidence, and reports.
- `production`: the feature is robust for real repositories, CI budgets, and analyst audit.
Current verified gate: `scaffold_quick_scan`.
Do not report OpenUltraSAST as a working security harness until `usable_harness_mvp` is checked.
## Phase 0: Project Skeleton (`scaffold`, complete)
- [x] Create Python package skeleton with CLI entry point `ousast`.
- [x] Add config loading for `openultrasast.toml` and environment variables.
@@ -58,26 +71,51 @@
- [x] Feed rejected findings into retrieval filter adjustment and prompt constraints.
- [x] Add tests proving scoped false-positive demotion does not globally suppress the vulnerability class.
## Phase 5: Harness Runtime
## Gate: `scaffold_quick_scan`
- [ ] Define harness event model.
- [ ] Define processor interface and contract checks.
- [ ] Implement strict and warn modes.
- [ ] Add trace writer for events and processor state changes.
- [ ] Serialize harness config, prompt hashes, processor versions, and model roles.
- [ ] Add tests for processor contract violations.
- [x] Quick scan can enumerate files, rank targets, emit basic static findings, and write JSON/Markdown artifacts.
- [x] Quick scan is uv-managed and covered by unit tests.
- [ ] Quick scan includes real LLM hunter execution. This is intentionally deferred to standard mode.
- [ ] Quick scan includes SARIF output. This is intentionally deferred to Phase 9.
- [ ] Quick scan includes enforced verifier decisions. This is intentionally deferred to Phase 6.
## Gate: `usable_harness_mvp`
- [x] Harness runtime drives scan stages instead of direct CLI orchestration.
- [x] Event traces are emitted for every scan stage.
- [x] Static mapping ingestion feeds FileTarget hints and verifier evidence candidates.
- [x] Entry-point reachability feeds FileTarget reachability hints and ranking priority.
- [ ] Evidence ladder state machine prevents invalid verified statuses.
- [ ] Independent verifier runs on fixture findings without hunter reasoning.
- [ ] JSON, Markdown, SARIF, and manifest artifacts share finding IDs and evidence references.
- [ ] A fixture scan demonstrates one accepted static-corroborated finding and one rejected false positive.
## Phase 5: Harness Runtime (`usable_harness_mvp` prerequisite)
- [x] Define harness event model.
- [x] Define processor interface and contract checks.
- [x] Implement strict and warn modes.
- [x] Add trace writer for events and processor state changes.
- [x] Serialize harness config, prompt hashes, processor versions, and model roles.
- [x] Add tests for processor contract violations.
## Phase 5A: Static Mapping Disciplines
- [ ] Add SARIF ingestion for Semgrep and CodeQL outputs.
- [ ] Normalize SARIF rules, locations, severity, fingerprints, and provenance into static hints.
- [ ] Add Semgrep mapping task records for pattern, variant, and rule-authoring loops.
- [ ] Add CodeQL mapping task records for source, sink, sanitizer, and path explanations.
- [ ] Add differential mapping records for changed files, changed functions, and changed trust-boundary crossings.
- [ ] Add sharp-edge records for insecure defaults, dangerous APIs, confusing config, and misuse-prone interfaces.
- [ ] Add tests for mapping analyzer output to `FileTarget` hints and verifier evidence candidates.
- [x] Add SARIF ingestion for Semgrep and CodeQL outputs.
- [x] Normalize SARIF rules, locations, severity, fingerprints, and provenance into static hints.
- [x] Add Semgrep mapping task records for pattern, variant, and rule-authoring loops.
- [x] Add CodeQL mapping task records for source, sink, sanitizer, and path explanations.
- [x] Add entry-point mapping records for routes, CLI/process boundaries, parser entry points, smart-contract state-changing entry points, callbacks, and privileged surfaces.
- [x] Classify entry points by access level: public, authenticated, role-restricted, contract-only/callback, local-only, or review-required.
- [x] Map entry points to function-level source ranges and access evidence.
- [x] Preserve conditional reachability evidence for feature flags, rollout toggles, pause guards, and runtime guard expressions.
- [x] Attach reachability hints to FileTargets and ranking inputs.
- [x] Annotate findings with function-level reachability status before prioritizing fixes.
- [x] Add differential mapping records for changed files, changed functions, and changed trust-boundary crossings.
- [x] Add sharp-edge records for insecure defaults, dangerous APIs, confusing config, and misuse-prone interfaces.
- [x] Add tests for mapping analyzer output to `FileTarget` hints and verifier evidence candidates.
## Phase 6: Verification
## Phase 6: Verification (`usable_harness_mvp` prerequisite)
- [ ] Define evidence ladder state machine.
- [ ] Implement independent verifier prompt and schema.
@@ -86,7 +124,14 @@
- [ ] Enforce that verified report status requires at least `static_corroboration`.
- [ ] Add tests for invalid evidence transitions.
## Phase 7: Docker Sandbox
## Gate: `standard_security_harness`
- [ ] Standard mode uses ranked hunter scheduling, retrieval packages, selected skill snippets, and independent verification.
- [ ] Every hunter trajectory is persisted and linked to finding artifacts.
- [ ] False-positive learning updates ranking calibration from verifier outcomes.
- [ ] Fusion can be triggered for high-impact, contradictory, or difficult findings.
## Phase 7: Docker Sandbox (`sandboxed_dynamic_harness` prerequisite)
- [ ] Implement Docker availability probe.
- [ ] Implement no-network read-only workspace sandbox runner.
@@ -97,7 +142,7 @@
- [ ] Add sanitizer build environment for C/C++ fixtures.
- [ ] Add tests that verify workspace remains read-only.
## Phase 7A: Dynamic Evidence
## Phase 7A: Dynamic Evidence (`sandboxed_dynamic_harness` prerequisite)
- [ ] Implement dynamic analysis configuration with `enabled = false` by default.
- [ ] Implement service startup command records with declared ports and timeouts.
@@ -106,7 +151,7 @@
- [ ] Tie dynamic artifacts to evidence transitions without allowing model-only upgrades.
- [ ] Add tests that reject undeclared network targets.
## Phase 8: Standard Mode Hunter Pool
## Phase 8: Standard Mode Hunter Pool (`standard_security_harness` prerequisite)
- [ ] Implement tier assignment by priority.
- [ ] Implement budget allocation and per-target limits.
@@ -116,7 +161,7 @@
- [ ] Integrate verifier after hunter output.
- [ ] Add fixture scan demonstrating at least one verified static-corroborated finding.
## Phase 9: SARIF And CI Output
## Phase 9: SARIF And CI Output (`usable_harness_mvp` prerequisite)
- [ ] Implement SARIF report writer.
- [ ] Implement machine-readable manifest.
@@ -124,7 +169,13 @@
- [ ] Add CI-friendly exit code policy.
- [ ] Add tests for SARIF schema validity.
## Phase 10: Deep Mode
## Gate: `sandboxed_dynamic_harness`
- [ ] Docker sandbox runs build/test/fuzz/PoC commands with no network by default.
- [ ] Scoped dynamic probes require explicit declared targets and emit artifacts.
- [ ] Sanitizer/fuzz artifacts can advance evidence only through valid transitions.
## Phase 10: Deep Mode (`sandboxed_dynamic_harness` prerequisite)
- [ ] Implement fuzzable target detection beyond existing fuzz entry points.
- [ ] Implement fuzz harness generation prompt and schema.
@@ -134,7 +185,13 @@
- [ ] Advance findings to `crash_reproduced` only with artifact evidence.
- [ ] Add a small C/C++ parser fixture with reproducible sanitizer crash.
## Phase 11: Patch Oracle
## Gate: `fix_validation_harness`
- [ ] Patch proposals are generated as diffs and not silently applied.
- [ ] Fix lifecycle records intake, plan, implementation, adversarial review, reconciliation, verification, and ready states.
- [ ] Patch validation requires sandboxed checks and accepted adversarial findings must be resolved.
## Phase 11: Patch Oracle (`fix_validation_harness` prerequisite)
- [ ] Implement patch proposal schema.
- [ ] Implement OpenUltraCode-style fix lifecycle states: intake, plan, implement, adversarial_review, reconcile, verify, ready.
@@ -147,7 +204,13 @@
- [ ] Advance findings to `patch_validated` only after validation passes.
- [ ] Add tests that reject patches changing unrelated files.
## Phase 12: MCP And OpenCode Commands
## Gate: `opencode_product`
- [ ] Narrow MCP tools cover scan, status, findings, evidence, artifacts, patch proposal, and report export.
- [ ] OpenCode workflows can drive scan, triage, fusion, and patch audit without arbitrary shell exposure.
- [ ] Cost controls, retries, redaction, docs, examples, and CI gates are in place.
## Phase 12: MCP And OpenCode Commands (`opencode_product` prerequisite)
- [x] Add project-local opencode skills for scan, triage, fix audit, and implementation steering.
- [ ] Implement narrow MCP server.
@@ -156,7 +219,7 @@
- [ ] Ensure MCP tools never expose arbitrary shell execution.
- [ ] Add integration test for MCP tool listing and scan status.
## Phase 13: Fusion
## Phase 13: Fusion (`standard_security_harness` and `opencode_product` prerequisite)
- [ ] Implement fusion trigger policy.
- [ ] Trigger fusion whenever a task requires deeper reasoning, including difficult findings, contradictory evidence, high-impact decisions, and risky fixes.
@@ -167,7 +230,7 @@
- [ ] Record model IDs, degradations, warnings, and final decision source.
- [ ] Add tests for fusion output validation.
## Phase 14: Skill Index
## Phase 14: Skill Index (`standard_security_harness` prerequisite)
- [ ] Build Trail of Bits skill descriptor inventory.
- [ ] Chunk selected skill operating guidance into retrieval namespace.
@@ -175,7 +238,7 @@
- [ ] Add prompt budget caps for skill snippets.
- [ ] Add tests for routing C parser targets to memory/fuzzing skills and crypto targets to constant-time/vector skills.
## Phase 15: Hardening And Release Readiness
## Phase 15: Hardening And Release Readiness (`production` prerequisite)
- [ ] Add cost budget enforcement.
- [ ] Add model call retry and backoff policies.