Open_Ultra_SAST/.kiro/specs/openrouter-sast-harness/tasks.md

Tasks: OpenRouter SAST Harness

Completion Gates

Checklist completion is not product completion. Each phase now carries a functional gate label:

• scaffold: isolated primitives, schemas, local artifacts, or tests exist.
• usable: the feature works end-to-end for fixtures without manual wiring.
• integrated: the feature participates in the harness runtime, traces, evidence, and reports.
• production: the feature is robust for real repositories, CI budgets, and analyst audit.

Current verified gate: usable_harness_mvp.

Do not report OpenUltraSAST as a working security harness until usable_harness_mvp is checked.

Phase 0: Project Skeleton (scaffold, complete)

• Create Python package skeleton with CLI entry point ousast.
• Add config loading for openultrasast.toml and environment variables.
• Add resolved config artifact writer.
• Add scan run directory creation under .openultrasast/runs/<scan-id>/.
• Add basic tests for config loading and run directory creation.

Phase 1: Intake And Preprocess

• Implement repository snapshot collection.
• Implement source file enumeration with ignore handling.
• Implement language detection from extension and shebang.
• Implement LOC counting.
• Implement initial tag heuristics for parser, memory unsafe, crypto, auth boundary, deserialization, syscall, network, filesystem, and fuzzable targets.
• Emit preprocess/file_targets.json.
• Add tests for tag heuristics on small fixture repositories.

Phase 2: Ranking

• Define FileTarget and ranking schemas.
• Implement heuristic score floors and boosts.
• Implement OpenRouter chat client for ranker role.
• Implement chunked ranker calls with JSON response validation.
• Compute composite priority.
• Emit ranking rationales and model metadata.
• Add tests for composite priority and malformed model response handling.

Phase 3: Quick Scan Reports

• Implement static finding schema.
• Implement quick-mode hunter prompt and bounded file context.
• Implement structured finding extraction.
• Enforce initial evidence level suspicion unless static corroboration exists.
• Emit findings.json.
• Emit Markdown report.
• Add smoke test against a fixture repository with one known insecure pattern.

Phase 4: OpenRouter Embeddings

• Implement OpenRouter embedding client.
• Run a vector-store bakeoff for metadata filtering, local persistence, incremental indexing, export/import, and retrieval quality.
• Select default local vector store based on bakeoff results.
• Implement code chunking with path, symbol, and line metadata.
• Implement namespaces for repo code, docs, static findings, mechanisms, skills, and traces.
• Implement index reuse keyed by repository snapshot and embedding model.
• Implement bounded retrieval package builder per role.
• Add tests for chunk metadata and retrieval filters.

Phase 4A: Ranking Quality And False-Positive Loop

• Define false-positive reason taxonomy.
• Add false-positive learning records for rejected, duplicate, unsupported, contradicted, and unverified findings.
• Track ranking metrics by tier: verified findings, false positives, time to verification, and missed fixture vulnerabilities.
• Feed verifier outcomes into ranking calibration.
• Feed rejected findings into retrieval filter adjustment and prompt constraints.
• Add tests proving scoped false-positive demotion does not globally suppress the vulnerability class.

Gate: scaffold_quick_scan

• Quick scan can enumerate files, rank targets, emit basic static findings, and write JSON/Markdown artifacts.
• Quick scan is uv-managed and covered by unit tests.
• Quick scan includes real LLM hunter execution. This is intentionally deferred to standard mode.
• Quick scan includes SARIF output.
• Quick scan includes enforced verifier decisions.

Gate: usable_harness_mvp

• Harness runtime drives scan stages instead of direct CLI orchestration.
• Event traces are emitted for every scan stage.
• Static mapping ingestion feeds FileTarget hints and verifier evidence candidates.
• Entry-point reachability feeds FileTarget reachability hints and ranking priority.
• Evidence ladder state machine prevents invalid verified statuses.
• Independent verifier runs on fixture findings without hunter reasoning.
• JSON, Markdown, SARIF, and manifest artifacts share finding IDs and evidence references.
• A fixture scan demonstrates one accepted static-corroborated finding and one rejected false positive.

Phase 5: Harness Runtime (usable_harness_mvp prerequisite)

• Define harness event model.
• Define processor interface and contract checks.
• Implement strict and warn modes.
• Add trace writer for events and processor state changes.
• Serialize harness config, prompt hashes, processor versions, and model roles.
• Add tests for processor contract violations.

Phase 5A: Static Mapping Disciplines

• Add SARIF ingestion for Semgrep and CodeQL outputs.
• Normalize SARIF rules, locations, severity, fingerprints, and provenance into static hints.
• Add Semgrep mapping task records for pattern, variant, and rule-authoring loops.
• Add CodeQL mapping task records for source, sink, sanitizer, and path explanations.
• Add entry-point mapping records for routes, CLI/process boundaries, parser entry points, smart-contract state-changing entry points, callbacks, and privileged surfaces.
• Classify entry points by access level: public, authenticated, role-restricted, contract-only/callback, local-only, or review-required.
• Map entry points to function-level source ranges and access evidence.
• Preserve conditional reachability evidence for feature flags, rollout toggles, pause guards, and runtime guard expressions.
• Attach reachability hints to FileTargets and ranking inputs.
• Annotate findings with function-level reachability status before prioritizing fixes.
• Add differential mapping records for changed files, changed functions, and changed trust-boundary crossings.
• Add sharp-edge records for insecure defaults, dangerous APIs, confusing config, and misuse-prone interfaces.
• Add tests for mapping analyzer output to FileTarget hints and verifier evidence candidates.

Phase 6: Verification (usable_harness_mvp prerequisite)

• Define evidence ladder state machine.
• Implement independent verifier prompt and schema.
• Ensure verifier context excludes hunter reasoning.
• Implement pro-case, counter-case, tie-breaker, and required-next-step fields.
• Enforce that verified report status requires at least static_corroboration.
• Add tests for invalid evidence transitions.

Gate: standard_security_harness

• Standard mode uses ranked hunter scheduling, retrieval packages, selected skill snippets, and independent verification.
• Every hunter trajectory is persisted and linked to finding artifacts.
• False-positive learning updates ranking calibration from verifier outcomes.
• Fusion can be triggered for high-impact, contradictory, or difficult findings.
• Standard mode uses real language-aware LLM hunters instead of reusing quick static findings.

Gate: benchmark_grounded_finder

• Benchmark manifests exist for JavaScript/Node/web, Python/web, Java, and C/C++ vulnerable projects.
• Benchmark runner executes quick, standard, and deep modes as supported and preserves benchmark artifacts.
• Benchmark reports include recall, missed expected findings, false-positive reasons, runtime, and external baseline deltas.
• Missed benchmark findings feed calibration records for rules, ranking, retrieval, hunters, SARIF ingestion, dynamic probes, or skill routing.
• At least one realistic vulnerable project and one curated CWE corpus are tracked per supported language family.

Phase 6A: Benchmark Corpus And Scoreboard (benchmark_grounded_finder prerequisite)

• Define benchmark manifest schema with source, language, framework, setup, expected vulnerabilities, known noise, scan modes, and optional baselines.
• Implement benchmark run directory creation under .openultrasast/benchmarks/<benchmark-run-id>/.
• Implement finding-to-ground-truth matching by CWE, rule ID, file, function, sink, and evidence text.
• Implement benchmark metrics for recall, precision when possible, false-positive reasons, missed findings, runtime, model usage, and artifact links.
• Add benchmark fixtures or manifests for JavaScript/Node/web, Python/web, Java, and C/C++.
• Add external baseline ingestion for SARIF or normalized findings from Clearwing, Semgrep, CodeQL, clang-tidy, Bandit, npm audit, or language-appropriate tools.
• Add tests proving benchmark misses are recorded as calibration inputs without automatically suppressing unrelated findings.

Gate: explain_and_fix_workflow

• Explain mode can explain a finding or benchmark miss with source evidence, reachability status, missing evidence, secure pattern, and prevention guidance.
• Explain mode supports concise, learner, and reviewer audience levels without changing evidence levels.
• Interactive fixing selects relevant security skills and proposes a minimal patch plan without silently mutating the target repository.
• Fix guidance links to validation commands and required evidence transitions.

Phase 6B: Explain Mode And Interactive Fixing (explain_and_fix_workflow prerequisite)

• Add explain artifact schema for finding ID, source refs, evidence refs, audience level, prevention guidance, selected skills, and validation advice.
• Add ousast explain <finding-id> --scan <scan-id> CLI command.
• Add explain support for benchmark misses and imported SARIF findings.
• Implement language/vulnerability skill routing for JavaScript/Node/web, Python/web, Java, and C/C++.
• Add interactive fixing plan schema with confirmation gates, selected skills, proposed patch scope, validation commands, and evidence requirements.
• Ensure interactive fixing cannot mark patches validated unless the patch oracle and sandbox validation gates pass.
• Add tests that explanation does not upgrade evidence level and patch guidance does not mutate repositories by default.

Phase 7: Docker Sandbox (sandboxed_dynamic_harness prerequisite)

• Implement Docker availability probe.
• Implement no-network read-only workspace sandbox runner.
• Add explicit scoped-network sandbox mode for task-required dynamic probes.
• Add scratch volume or tmpfs support.
• Add CPU, memory, pids, timeout, and capability limits.
• Add command execution artifact capture.
• Add sanitizer build environment for C/C++ fixtures.
• Add tests that verify workspace remains read-only.

Phase 7A: Dynamic Evidence (sandboxed_dynamic_harness prerequisite)

• Implement dynamic analysis configuration with enabled = false by default.
• Implement service startup command records with declared ports and timeouts.
• Implement netcat-style local socket probe tool inside sandbox scope.
• Implement HTTP or protocol smoke-test artifact capture.
• Tie dynamic artifacts to evidence transitions without allowing model-only upgrades.
• Add tests that reject undeclared network targets.

Phase 8: Standard Mode Hunter Pool (standard_security_harness prerequisite)

• Implement tier assignment by priority.
• Implement budget allocation and per-target limits.
• Implement parallel hunter scheduling.
• Persist hunter trajectories as JSONL.
• Integrate retrieval packages and selected skill snippets.
• Integrate verifier after hunter output.
• Add fixture scan demonstrating at least one verified static-corroborated finding.

Phase 9: SARIF And CI Output (usable_harness_mvp prerequisite)

• Implement SARIF report writer.
• Implement machine-readable manifest.
• Add severity, confidence, evidence level, model IDs, and artifact refs to reports.
• Add CI-friendly exit code policy.
• Add tests for SARIF schema validity.

Gate: sandboxed_dynamic_harness

• Docker sandbox runs build/test/fuzz/PoC commands with no network by default.
• Scoped dynamic probes require explicit declared targets and emit artifacts.
• Sanitizer/fuzz artifacts can advance evidence only through valid transitions.

Phase 10: Deep Mode (sandboxed_dynamic_harness prerequisite)

• Implement fuzzable target detection beyond existing fuzz entry points.
• Implement fuzz harness generation prompt and schema.
• Compile generated harnesses in sandbox.
• Run libFuzzer or equivalent with sanitizer instrumentation.
• Capture crashes and minimization artifacts.
• Advance findings to crash_reproduced only with artifact evidence.
• Add a small C/C++ parser fixture with reproducible sanitizer crash.

Gate: fix_validation_harness

• Patch proposals are generated as diffs and not silently applied.
• Fix lifecycle records intake, plan, implementation, adversarial review, reconciliation, verification, and ready states.
• Patch validation requires sandboxed checks and accepted adversarial findings must be resolved.

Phase 11: Patch Oracle (fix_validation_harness prerequisite)

• Implement patch proposal schema.
• Implement OpenUltraCode-style fix lifecycle states: intake, plan, implement, adversarial_review, reconcile, verify, ready.
• Run patching in a writable sandbox copy or git worktree.
• Generate minimal diffs only.
• Rerun reproducer and affected checks.
• Run differential fix audit against the generated patch.
• Run Semgrep or CodeQL delta checks when corresponding evidence exists.
• Run sharp-edge fix audit when API, config, auth, crypto, or defaults are changed.
• Advance findings to patch_validated only after validation passes.
• Add tests that reject patches changing unrelated files.

Gate: opencode_product

• Narrow MCP tools cover scan, status, findings, evidence, artifacts, patch proposal, and report export.
• OpenCode workflows can drive scan, triage, fusion, and patch audit without arbitrary shell exposure.
• Cost controls, retries, redaction, docs, examples, and CI gates are in place.
• OpenCode workflows can drive benchmark runs, finding explanations, and interactive fix proposals.

Phase 12: MCP And OpenCode Commands (opencode_product prerequisite)

Implemented in mcp.py (zero-dep stdio JSON-RPC server) + the ousast mcp CLI subcommand; tests in tests/test_mcp.py; README "MCP server" section documents it.

• Add project-local opencode skills for scan, triage, fix audit, and implementation steering.
• Implement narrow MCP server.
• Add openultrasast.scan, status, findings, get_finding, evidence, artifacts, benchmark, explain, propose_patch, and export_report tools.
• Add opencode command docs for quick, standard, deep, verify, and patch workflows. (README; deep/patch noted as roadmap.)
• Ensure MCP tools never expose arbitrary shell execution. (Surface-safety test asserts no shell/exec/command tool.)
• Add integration test for MCP tool listing and scan status.

Phase 13: Fusion (standard_security_harness and opencode_product prerequisite)

Implemented in fusion.py (zero-dep deterministic engine + lazy LLM panels), wired into the standard-mode CLI (fusion.json + manifest fusion block), config [fusion]. Tests in tests/test_fusion.py.

• Implement fusion trigger policy.
• Trigger fusion whenever a task requires deeper reasoning, including difficult findings, contradictory evidence, high-impact decisions, and risky fixes.
• Implement panel A, panel B, and decider role config.
• Implement cross-critique and revised answer flow.
• Implement vote/rank output schema.
• Implement reconciliation dispositions.
• Record model IDs, degradations, warnings, and final decision source.
• Add tests for fusion output validation.

Phase 14: Skill Index (standard_security_harness prerequisite)

Implemented in skills.py (descriptor inventory + scoring router + retrieval-namespace chunking + budget-capped context); the HarnessX hunter prompt consumes the routed skill context; tests in tests/test_skills.py.

• Build Trail of Bits skill descriptor inventory.
• Chunk selected skill operating guidance into retrieval namespace.
• Implement skill router by language, tags, vulnerability class, and stage.
• Add prompt budget caps for skill snippets.
• Add tests for routing C parser targets to memory/fuzzing skills and crypto targets to constant-time/vector skills.

Phase 15: Hardening And Release Readiness (production prerequisite)

Implemented: redaction.py (traces/reports), provider retry/backoff in provider/openrouter.py, [hardening] config (redact_secrets + max_findings budget; agentic spend bounded by [harnessx]), docs/threat-model.md, docs/examples.md. Tests in tests/test_redaction.py, tests/test_provider_retry.py, tests/test_hardening_pipeline.py. CI gates pre-existed (ruff/mypy/pytest/detection gate/dagger).

• Add cost budget enforcement.
• Add model call retry and backoff policies.
• Add redaction for secrets in traces and reports.
• Add docs for threat model and sandbox limits.
• Add end-to-end examples.
• Add lint, typecheck, test, and CI gates.