marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-22 06:29:29 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(harness): add runtime and reachability mapping

Browse Source
This commit is contained in:
Marius Ciepluch
2026-06-18 07:57:06 +02:00
parent 33757db14a
commit f8c690999a
16 changed files with 1339 additions and 53 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/specs/openrouter-sast-harness/tasks.md
+89 -26
View File
@@ -1,6 +1,19 @@
# Tasks: OpenRouter SAST Harness
## Phase 0: Project Skeleton
## Completion Gates
Checklist completion is not product completion. Each phase now carries a functional gate label:
- `scaffold`: isolated primitives, schemas, local artifacts, or tests exist.
- `usable`: the feature works end-to-end for fixtures without manual wiring.
- `integrated`: the feature participates in the harness runtime, traces, evidence, and reports.
- `production`: the feature is robust for real repositories, CI budgets, and analyst audit.
Current verified gate: `scaffold_quick_scan`.
Do not report OpenUltraSAST as a working security harness until `usable_harness_mvp` is checked.
## Phase 0: Project Skeleton (`scaffold`, complete)
- [x] Create Python package skeleton with CLI entry point `ousast`.
- [x] Add config loading for `openultrasast.toml` and environment variables.
@@ -58,26 +71,51 @@
- [x] Feed rejected findings into retrieval filter adjustment and prompt constraints.
- [x] Add tests proving scoped false-positive demotion does not globally suppress the vulnerability class.
## Phase 5: Harness Runtime
## Gate: `scaffold_quick_scan`
- [ ] Define harness event model.
- [ ] Define processor interface and contract checks.
- [ ] Implement strict and warn modes.
- [ ] Add trace writer for events and processor state changes.
- [ ] Serialize harness config, prompt hashes, processor versions, and model roles.
- [ ] Add tests for processor contract violations.
- [x] Quick scan can enumerate files, rank targets, emit basic static findings, and write JSON/Markdown artifacts.
- [x] Quick scan is uv-managed and covered by unit tests.
- [ ] Quick scan includes real LLM hunter execution. This is intentionally deferred to standard mode.
- [ ] Quick scan includes SARIF output. This is intentionally deferred to Phase 9.
- [ ] Quick scan includes enforced verifier decisions. This is intentionally deferred to Phase 6.
## Gate: `usable_harness_mvp`
- [x] Harness runtime drives scan stages instead of direct CLI orchestration.
- [x] Event traces are emitted for every scan stage.
- [x] Static mapping ingestion feeds FileTarget hints and verifier evidence candidates.
- [x] Entry-point reachability feeds FileTarget reachability hints and ranking priority.
- [ ] Evidence ladder state machine prevents invalid verified statuses.
- [ ] Independent verifier runs on fixture findings without hunter reasoning.
- [ ] JSON, Markdown, SARIF, and manifest artifacts share finding IDs and evidence references.
- [ ] A fixture scan demonstrates one accepted static-corroborated finding and one rejected false positive.
## Phase 5: Harness Runtime (`usable_harness_mvp` prerequisite)
- [x] Define harness event model.
- [x] Define processor interface and contract checks.
- [x] Implement strict and warn modes.
- [x] Add trace writer for events and processor state changes.
- [x] Serialize harness config, prompt hashes, processor versions, and model roles.
- [x] Add tests for processor contract violations.
## Phase 5A: Static Mapping Disciplines
- [ ] Add SARIF ingestion for Semgrep and CodeQL outputs.
- [ ] Normalize SARIF rules, locations, severity, fingerprints, and provenance into static hints.
- [ ] Add Semgrep mapping task records for pattern, variant, and rule-authoring loops.
- [ ] Add CodeQL mapping task records for source, sink, sanitizer, and path explanations.
- [ ] Add differential mapping records for changed files, changed functions, and changed trust-boundary crossings.
- [ ] Add sharp-edge records for insecure defaults, dangerous APIs, confusing config, and misuse-prone interfaces.
- [ ] Add tests for mapping analyzer output to `FileTarget` hints and verifier evidence candidates.
- [x] Add SARIF ingestion for Semgrep and CodeQL outputs.
- [x] Normalize SARIF rules, locations, severity, fingerprints, and provenance into static hints.
- [x] Add Semgrep mapping task records for pattern, variant, and rule-authoring loops.
- [x] Add CodeQL mapping task records for source, sink, sanitizer, and path explanations.
- [x] Add entry-point mapping records for routes, CLI/process boundaries, parser entry points, smart-contract state-changing entry points, callbacks, and privileged surfaces.
- [x] Classify entry points by access level: public, authenticated, role-restricted, contract-only/callback, local-only, or review-required.
- [x] Map entry points to function-level source ranges and access evidence.
- [x] Preserve conditional reachability evidence for feature flags, rollout toggles, pause guards, and runtime guard expressions.
- [x] Attach reachability hints to FileTargets and ranking inputs.
- [x] Annotate findings with function-level reachability status before prioritizing fixes.
- [x] Add differential mapping records for changed files, changed functions, and changed trust-boundary crossings.
- [x] Add sharp-edge records for insecure defaults, dangerous APIs, confusing config, and misuse-prone interfaces.
- [x] Add tests for mapping analyzer output to `FileTarget` hints and verifier evidence candidates.
## Phase 6: Verification
## Phase 6: Verification (`usable_harness_mvp` prerequisite)
- [ ] Define evidence ladder state machine.
- [ ] Implement independent verifier prompt and schema.
@@ -86,7 +124,14 @@
- [ ] Enforce that verified report status requires at least `static_corroboration`.
- [ ] Add tests for invalid evidence transitions.
## Phase 7: Docker Sandbox
## Gate: `standard_security_harness`
- [ ] Standard mode uses ranked hunter scheduling, retrieval packages, selected skill snippets, and independent verification.
- [ ] Every hunter trajectory is persisted and linked to finding artifacts.
- [ ] False-positive learning updates ranking calibration from verifier outcomes.
- [ ] Fusion can be triggered for high-impact, contradictory, or difficult findings.
## Phase 7: Docker Sandbox (`sandboxed_dynamic_harness` prerequisite)
- [ ] Implement Docker availability probe.
- [ ] Implement no-network read-only workspace sandbox runner.
@@ -97,7 +142,7 @@
- [ ] Add sanitizer build environment for C/C++ fixtures.
- [ ] Add tests that verify workspace remains read-only.
## Phase 7A: Dynamic Evidence
## Phase 7A: Dynamic Evidence (`sandboxed_dynamic_harness` prerequisite)
- [ ] Implement dynamic analysis configuration with `enabled = false` by default.
- [ ] Implement service startup command records with declared ports and timeouts.
@@ -106,7 +151,7 @@
- [ ] Tie dynamic artifacts to evidence transitions without allowing model-only upgrades.
- [ ] Add tests that reject undeclared network targets.
## Phase 8: Standard Mode Hunter Pool
## Phase 8: Standard Mode Hunter Pool (`standard_security_harness` prerequisite)
- [ ] Implement tier assignment by priority.
- [ ] Implement budget allocation and per-target limits.
@@ -116,7 +161,7 @@
- [ ] Integrate verifier after hunter output.
- [ ] Add fixture scan demonstrating at least one verified static-corroborated finding.
## Phase 9: SARIF And CI Output
## Phase 9: SARIF And CI Output (`usable_harness_mvp` prerequisite)
- [ ] Implement SARIF report writer.
- [ ] Implement machine-readable manifest.
@@ -124,7 +169,13 @@
- [ ] Add CI-friendly exit code policy.
- [ ] Add tests for SARIF schema validity.
## Phase 10: Deep Mode
## Gate: `sandboxed_dynamic_harness`
- [ ] Docker sandbox runs build/test/fuzz/PoC commands with no network by default.
- [ ] Scoped dynamic probes require explicit declared targets and emit artifacts.
- [ ] Sanitizer/fuzz artifacts can advance evidence only through valid transitions.
## Phase 10: Deep Mode (`sandboxed_dynamic_harness` prerequisite)
- [ ] Implement fuzzable target detection beyond existing fuzz entry points.
- [ ] Implement fuzz harness generation prompt and schema.
@@ -134,7 +185,13 @@
- [ ] Advance findings to `crash_reproduced` only with artifact evidence.
- [ ] Add a small C/C++ parser fixture with reproducible sanitizer crash.
## Phase 11: Patch Oracle
## Gate: `fix_validation_harness`
- [ ] Patch proposals are generated as diffs and not silently applied.
- [ ] Fix lifecycle records intake, plan, implementation, adversarial review, reconciliation, verification, and ready states.
- [ ] Patch validation requires sandboxed checks and accepted adversarial findings must be resolved.
## Phase 11: Patch Oracle (`fix_validation_harness` prerequisite)
- [ ] Implement patch proposal schema.
- [ ] Implement OpenUltraCode-style fix lifecycle states: intake, plan, implement, adversarial_review, reconcile, verify, ready.
@@ -147,7 +204,13 @@
- [ ] Advance findings to `patch_validated` only after validation passes.
- [ ] Add tests that reject patches changing unrelated files.
## Phase 12: MCP And OpenCode Commands
## Gate: `opencode_product`
- [ ] Narrow MCP tools cover scan, status, findings, evidence, artifacts, patch proposal, and report export.
- [ ] OpenCode workflows can drive scan, triage, fusion, and patch audit without arbitrary shell exposure.
- [ ] Cost controls, retries, redaction, docs, examples, and CI gates are in place.
## Phase 12: MCP And OpenCode Commands (`opencode_product` prerequisite)
- [x] Add project-local opencode skills for scan, triage, fix audit, and implementation steering.
- [ ] Implement narrow MCP server.
@@ -156,7 +219,7 @@
- [ ] Ensure MCP tools never expose arbitrary shell execution.
- [ ] Add integration test for MCP tool listing and scan status.
## Phase 13: Fusion
## Phase 13: Fusion (`standard_security_harness` and `opencode_product` prerequisite)
- [ ] Implement fusion trigger policy.
- [ ] Trigger fusion whenever a task requires deeper reasoning, including difficult findings, contradictory evidence, high-impact decisions, and risky fixes.
@@ -167,7 +230,7 @@
- [ ] Record model IDs, degradations, warnings, and final decision source.
- [ ] Add tests for fusion output validation.
## Phase 14: Skill Index
## Phase 14: Skill Index (`standard_security_harness` prerequisite)
- [ ] Build Trail of Bits skill descriptor inventory.
- [ ] Chunk selected skill operating guidance into retrieval namespace.
@@ -175,7 +238,7 @@
- [ ] Add prompt budget caps for skill snippets.
- [ ] Add tests for routing C parser targets to memory/fuzzing skills and crypto targets to constant-time/vector skills.
## Phase 15: Hardening And Release Readiness
## Phase 15: Hardening And Release Readiness (`production` prerequisite)
- [ ] Add cost budget enforcement.
- [ ] Add model call retry and backoff policies.