chore: add kiro spec-driven tooling and steering docs

Add the Kiro-style agentic SDLC tooling used to drive this project: the .opencode/skills/kiro-* skill set (discovery, spec init/requirements/design/ tasks, validation, implementation, review, steering), the .kiro/settings templates, and AGENTS.md describing the workflow. Update the spec docs (assessment/design/requirements/tasks) to current state. Scope tests/test_opencode_skills.py to the project's own openultrasast-* skills so the third-party kiro-* tooling (which uses a different frontmatter convention) no longer breaks the frontmatter check. Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-10-19 04:59:28 +00:00 · 2026-06-21 14:28:19 +02:00
parent 8fe8911020
commit 6699f1f8f9
58 changed files with 5509 additions and 2 deletions
@@ -34,6 +34,15 @@ The first release should support this workflow:
 ousast scan /path/to/repo --mode quick
 ```

+The first realistic quality milestone should add benchmark-grounded evaluation before claiming language coverage:
+
+```text
+ousast benchmark benchmarks/javascript-node.toml --mode standard
+ousast benchmark benchmarks/python-web.toml --mode standard
+ousast benchmark benchmarks/java-web.toml --mode standard
+ousast benchmark benchmarks/c-cpp.toml --mode quick
+```
+
 It should produce:

 ```text
@@ -50,6 +59,12 @@ The key product promise should be:
 OpenUltraSAST tells you where security review should start, what it suspects, what evidence exists, what evidence is missing, and which findings have survived independent verification.
 ```

+For coder education, the complementary promise should be:
+
+```text
+OpenUltraSAST explains why the pattern is dangerous, what evidence proves or weakens the finding, how to fix it safely, and how to avoid recreating the same bug class.
+```
+
 ## Core Architecture Recommendation

 Use a Python implementation initially. Clearwing and HarnessX are both Python, Docker and SARIF libraries are mature, and Trail of Bits tooling integration will be simpler. OpenCode integration can happen through MCP and command files rather than requiring the core scanner to be TypeScript.
@@ -149,6 +164,8 @@ Examples:

 False-positive elimination should be handled the same way: a model can recommend demotion or suppression, but the harness should require a scoped reason and counter-evidence.

+Benchmark grounding should be handled with the same discipline. A scanner that passes tiny fixtures but misses NodeGoat, PyGoat, OWASP Benchmark/WebGoat, Juliet, or small sanitizer-backed parser targets is not a realistic finder. Language support should be claimed only after benchmark runs show useful recall and auditable false-positive behavior on realistic vulnerable projects.
+
 False-positive reasons should include:

 - No attacker-controlled path.
@@ -25,11 +25,24 @@ repo path
  -> tiered hunter pool
  -> independent verifier
  -> optional fusion adjudication
+  -> optional explain and guided fix workflow
  -> optional reproducer/exploit triage
  -> optional patch oracle
  -> reports and artifacts
 ```

+Benchmark runs wrap the same scan pipeline instead of using a separate detector path:
+
+```text
+benchmark manifest
+  -> acquire or locate vulnerable project
+  -> run quick, standard, or deep scan
+  -> normalize OpenUltraSAST findings and optional external baselines
+  -> match expected vulnerabilities by CWE, file, function, sink, and evidence
+  -> record recall, precision where possible, misses, false-positive reasons, and runtime
+  -> feed misses and rejected findings into calibration artifacts
+```
+
 ## Main Components

 ### CLI
@@ -38,6 +51,8 @@ Initial command surface:

 ```text
 ousast scan <path> --mode quick|standard|deep --config openultrasast.toml
+ousast benchmark <manifest> --mode quick|standard|deep
+ousast explain <finding-id> --scan <scan-id> --audience concise|learner|reviewer
 ousast status <scan-id>
 ousast findings <scan-id>
 ousast report <scan-id> --format markdown|json|sarif
@@ -58,6 +73,8 @@ openultrasast.findings
 openultrasast.get_finding
 openultrasast.evidence
 openultrasast.artifacts
+openultrasast.benchmark
+openultrasast.explain
 openultrasast.propose_patch
 openultrasast.export_report
 ```
@@ -234,6 +251,63 @@ evidence candidates for verifier

 This layer should use Trail of Bits skills as task templates and checklists. For example, the CodeQL discipline supplies query/dataflow thinking, Semgrep supplies pattern and variant loops, entry-point analysis supplies attacker-controlled surface and access classification, differential review supplies changed-code blast-radius analysis, and sharp-edge review supplies API misuse mapping.

+### Benchmark Layer
+
+The benchmark layer is the quality control system for a realistic grounded finder. It prevents the scanner from passing narrow fixtures while missing obvious bugs in vulnerable projects.
+
+Benchmark manifest shape:
+
+```json
+{
+  "name": "nodegoat",
+  "language": "javascript",
+  "frameworks": ["node", "express"],
+  "source": {"type": "local", "path": "../benchmarks/nodegoat"},
+  "setup": {"commands": [], "requires_network": false},
+  "modes": ["quick", "standard"],
+  "expected": [
+    {
+      "cwe": "CWE-89",
+      "class": "sql_injection",
+      "path": "app/routes/*.js",
+      "evidence": "user input reaches database query"
+    }
+  ],
+  "known_noise": [
+    {"reason": "training-only route", "scope": "docs/**"}
+  ],
+  "baselines": ["semgrep", "codeql", "clearwing"]
+}
+```
+
+Initial benchmark families:
+
+```text
+javascript_node_web: NodeGoat, DVNA, OWASP Juice Shop slices, vulnerable Express fixtures
+python_web: PyGoat, vulnerable Flask/Django apps, pickle/deserialization and command-injection fixtures
+java_web: OWASP Benchmark, WebGoat, Juliet Java, Spring MVC injection/auth fixtures
+c_cpp: Juliet C/C++, Damn Vulnerable C/C++ Program, vulnerable parsers, ASan/UBSan crash fixtures
+```
+
+Benchmark metrics:
+
+```text
+expected_findings_total
+matched_findings_total
+missed_findings_total
+unsupported_classes
+false_positive_total
+false_positive_reasons
+recall_by_cwe
+recall_by_language
+precision_when_ground_truth_allows
+runtime_seconds
+model_calls_and_cost
+external_baseline_delta
+```
+
+Benchmark misses are first-class calibration inputs. A miss should record the source location, expected CWE, which stage failed to surface it, whether an external baseline found it, and the next candidate improvement: static rule, SARIF source, entry-point mapping, CodeQL/Semgrep discipline, LLM hunter prompt, retrieval package, dynamic reproducer, or skill routing.
+
 #### Entry-Point Reachability Mapping

 Entry-point mapping is a Phase 5A prerequisite for useful ranking and false-positive reduction. Pattern matches and static analyzer alerts are not enough; the harness must identify whether the affected function or symbol is plausibly reachable from attacker-controlled input.
@@ -362,6 +436,15 @@ evidence requirements

 Hunter outputs must be structured and evidence-oriented. They should not rely on hidden reasoning.

+Hunters must be language- and framework-aware. The default hunter prompts should not ask one generic model to find every vulnerability type. Initial hunter families should cover:
+
+```text
+javascript_node_web: injection, XSS, SSRF, prototype pollution, insecure deserialization, auth/session bugs, dependency misuse
+python_web: command injection, SQL/template injection, deserialization, path traversal, Flask/Django misconfiguration, auth/session bugs
+java_web: servlet/Spring injection, XXE, deserialization, path traversal, authorization bypass, crypto/API misuse
+c_cpp: memory corruption, integer overflow, format strings, command execution, parser bugs, lifetime and ownership mistakes
+```
+
 ### Verifier

 The verifier runs independently from the hunter. It receives only auditable facts:
@@ -468,6 +551,36 @@ never treated as an open-ended network pentest loop

 Dynamic evidence can support transitions such as `static_corroboration -> crash_reproduced`, `static_corroboration -> root_cause_explained`, or `root_cause_explained -> exploit_demonstrated` when the artifact actually demonstrates the claim.

+### Explain Mode And Interactive Fixing
+
+Explain mode turns a finding or benchmark miss into a prevention-oriented explanation. It should be useful to a coder who wants to understand the mistake, a vibecoder who needs safe guidance, and a reviewer who needs evidence boundaries.
+
+Explain input types:
+
+```text
+finding_id within a scan
+benchmark miss ID
+vulnerability class plus source location
+SARIF result ID imported into a scan
+```
+
+Explain output sections:
+
+```text
+what was found
+why this pattern is dangerous
+what attacker-controlled path or reachability evidence exists
+what evidence is missing
+how to fix it safely
+how to avoid recreating it
+which validation command or artifact would prove the fix
+which security skills were used for guidance
+```
+
+Explain mode must not inflate evidence. If a result is only a pattern match, it says so. If reachability is unknown, it names the missing call graph, route, CLI, parser, or dynamic artifact needed to prioritize the fix.
+
+Interactive fixing builds on explain mode and the patch oracle. It should ask for confirmation before patch proposal, select skills by language and vulnerability class, produce a minimal diff, run or list the required validation checks, and preserve all decisions as artifacts. It must not silently apply patches to the user's repository by default.
+
 ### Trail Of Bits Skill Integration

 Trail of Bits skills should be indexed and selected by routing metadata.
@@ -15,6 +15,8 @@ The project shall use Clearwing only as an implementation oracle. It shall not i
 5. Preserve Docker isolation as a first-class security boundary for code execution, harness compilation, fuzzing, and patch validation.
 6. Integrate Trail of Bits skills as scoped security expertise selected by language, framework, vulnerability class, mapping need, or verification need.
 7. Emit machine-readable artifacts suitable for CI, triage, and later training or harness evolution.
+8. Build detector quality against grounded vulnerable-project benchmarks for JavaScript/Node/web, Python/web, Java, and C/C++ instead of relying on toy fixtures or pattern-only tests.
+9. Help coders and vibecoders understand and fix findings through an evidence-aware explain mode that teaches the secure pattern, not only flags the insecure one.

 ## Completeness Model

@@ -37,6 +39,8 @@ Functional gates:
 4. `sandboxed_dynamic_harness`: Docker sandbox, scoped dynamic probes, sanitizer/build artifacts, and evidence transitions work without hidden network or host mutation.
 5. `fix_validation_harness`: patch proposals, adversarial fix audit, differential/static delta checks, and sandbox validation are integrated and cannot mark patches validated without passing evidence.
 6. `opencode_product`: MCP tools, opencode commands/skills, fusion deepening, CI-ready reports, cost controls, and hardening are complete enough for routine opencode-driven use.
+7. `benchmark_grounded_finder`: benchmark manifests, expected vulnerabilities, tool comparison baselines, recall/precision metrics, and missed-finding regression tracking exist for JavaScript, Python, Java, and C/C++ vulnerable projects.
+8. `explain_and_fix_workflow`: users can ask for an evidence-grounded explanation of a finding, receive prevention guidance, and enter an interactive fixing workflow that routes to the appropriate security skills without silently applying patches.

 The current implementation state shall be documented as `usable_harness_mvp` until later gates are explicitly verified.

@@ -78,6 +82,19 @@ The current implementation state shall be documented as `usable_harness_mvp` unt
 7. The system shall support sharp-edge mapping for APIs, configuration, defaults, cryptographic interfaces, and dangerous framework patterns that create predictable misuse.
 8. The system shall preserve analyzer provenance so each finding can cite which static tool, rule, diff, entry-point analysis, or skill-derived checklist contributed evidence.

+### Benchmark-Grounded Detection
+
+1. The system shall maintain benchmark manifests for realistic vulnerable projects, not only synthetic single-file fixtures.
+2. The initial benchmark matrix shall cover JavaScript/Node/web, Python/web, Java, and C/C++.
+3. Benchmarks shall include vulnerable training projects and curated CWE samples such as NodeGoat, DVNA or similar Node/Express apps, OWASP Juice Shop slices, PyGoat or vulnerable Flask/Django samples, OWASP Benchmark/WebGoat/Juliet Java, Juliet C/C++, vulnerable parser fixtures, and small sanitizer-backed C/C++ crash targets.
+4. Each benchmark manifest shall record repository path or source, language, framework, setup commands, scan modes to run, expected vulnerability classes, expected locations when known, and known noisy/non-security patterns.
+5. The benchmark runner shall emit recall, precision where ground truth allows it, false-positive reason distribution, missed expected findings, unsupported vulnerability classes, runtime, model usage, and artifact paths.
+6. Benchmark results shall compare quick, standard, and deep modes separately so rule coverage, LLM hunting, and dynamic validation are not conflated.
+7. Benchmark runs shall support comparison against external baselines such as Clearwing, Semgrep, CodeQL, clang-tidy, npm audit, Bandit, or language-appropriate tools when their outputs are available as SARIF or normalized findings.
+8. Missed benchmark vulnerabilities shall become regression records that feed rule expansion, ranking calibration, retrieval packages, hunter prompts, and skill routing.
+9. A detector shall not be considered realistic for a language until it demonstrates useful recall on at least one realistic benchmark project and one curated CWE corpus for that language.
+10. Benchmark artifacts shall be preserved under `.openultrasast/benchmarks/<benchmark-run-id>/` with links to the scan artifacts that produced each metric.
+
 ### OpenRouter Embeddings And Retrieval

 1. The system shall use OpenRouter embedding endpoints for semantic indexing.
@@ -121,6 +138,8 @@ The current implementation state shall be documented as `usable_harness_mvp` unt
 4. Standard mode shall use sandboxed tool-calling hunters and adversarial verification.
 5. Deep mode shall add harness generation, sanitizer builds, fuzzing, dynamic reproduction, crash reproduction, and patch validation where applicable.
 6. The system shall record every hunter trajectory as structured JSONL.
+7. Standard-mode hunters shall be evaluated against benchmark misses before being treated as production-quality detectors.
+8. Hunter prompts shall be language- and framework-aware for JavaScript/Node/web, Python/web, Java, and C/C++ instead of using one generic vulnerability prompt for all targets.

 ### Evidence Ladder

@@ -169,6 +188,18 @@ The current implementation state shall be documented as `usable_harness_mvp` unt
 6. Patch artifacts shall be emitted as diffs, not silently applied to the user's repository by default.
 7. The system shall audit proposed fixes for regression risk using differential analysis and relevant Trail of Bits skills before marking them ready.

+### Explain Mode And Interactive Fixing
+
+1. The system shall expose an explain workflow for a finding, vulnerability class, or benchmark miss.
+2. Explain mode shall describe the vulnerable pattern, why it matters, what attacker-controlled path or evidence exists, what evidence is missing, and what secure coding habit would have prevented it.
+3. Explain mode shall distinguish education from verification; an explanation shall not upgrade evidence level unless the required evidence artifact already exists.
+4. Explain mode shall cite source lines, finding artifacts, verifier results, SARIF/static evidence, dynamic evidence, benchmark metadata, and relevant skill snippets where available.
+5. Explain mode shall support audience levels such as concise, learner, and reviewer so it can serve coders, vibecoders, and security reviewers without changing the underlying evidence.
+6. Interactive fixing shall guide the user through a bounded plan, relevant security skill selection, minimal patch proposal, adversarial review, and validation commands.
+7. Interactive fixing shall route by language and vulnerability class, for example C/C++ memory issues to `c-review`, `address-sanitizer`, `libfuzzer`, and `harness-writing`; JavaScript web issues to web/security mapping and dependency checks; Python web issues to injection/deserialization/auth/defaults checks; and Java issues to OWASP Benchmark/WebGoat-style dataflow and framework checks.
+8. Interactive fixing shall never silently apply patches to the user's repository by default. It shall produce a proposed diff and required validation plan first.
+9. Explain and fix outputs shall emphasize prevention patterns so users learn how not to recreate the same vulnerability class in future code.
+
 ### Reporting

 1. The system shall emit JSON, SARIF, Markdown, and a manifest per scan.
@@ -182,6 +213,7 @@ The current implementation state shall be documented as `usable_harness_mvp` unt
 2. MCP tools shall focus on project-level operations: scan, status, findings, evidence, artifacts, patch proposal, and report export.
 3. The MCP server shall not expose arbitrary internal tools as public MCP methods.
 4. OpenCode commands shall invoke the harness with explicit model role configuration and visible assumptions.
+5. MCP and OpenCode integration shall include narrow operations for benchmark status, finding explanation, and interactive patch proposal without exposing arbitrary shell execution.

 ## Non-Functional Requirements

@@ -201,3 +233,5 @@ The current implementation state shall be documented as `usable_harness_mvp` unt
 6. `opencode_product`: A user can drive scans, triage, evidence inspection, report export, fusion, and patch proposal through narrow MCP/OpenCode interfaces without exposing arbitrary shell execution.
 7. A finding cannot be marked verified unless it reaches at least `static_corroboration` through enforced evidence transitions.
 8. A scan manifest records repository snapshot, harness config, OpenRouter model IDs, embedding model, prompt hashes, processor versions, evidence transitions, and artifact paths.
+9. `benchmark_grounded_finder`: A user can run a benchmark manifest for JavaScript, Python, Java, or C/C++ and receive language-level recall, missed finding, false-positive, runtime, and artifact reports.
+10. `explain_and_fix_workflow`: A user can request an explanation for a finding or benchmark miss and receive evidence-cited prevention guidance plus an optional interactive fix plan without automatic repository mutation.
@@ -130,6 +130,42 @@ Do not report OpenUltraSAST as a working security harness until `usable_harness_
 - [x] Every hunter trajectory is persisted and linked to finding artifacts.
 - [ ] False-positive learning updates ranking calibration from verifier outcomes.
 - [ ] Fusion can be triggered for high-impact, contradictory, or difficult findings.
+- [ ] Standard mode uses real language-aware LLM hunters instead of reusing quick static findings.
+
+## Gate: `benchmark_grounded_finder`
+
+- [ ] Benchmark manifests exist for JavaScript/Node/web, Python/web, Java, and C/C++ vulnerable projects.
+- [ ] Benchmark runner executes quick, standard, and deep modes as supported and preserves benchmark artifacts.
+- [ ] Benchmark reports include recall, missed expected findings, false-positive reasons, runtime, and external baseline deltas.
+- [ ] Missed benchmark findings feed calibration records for rules, ranking, retrieval, hunters, SARIF ingestion, dynamic probes, or skill routing.
+- [ ] At least one realistic vulnerable project and one curated CWE corpus are tracked per supported language family.
+
+## Phase 6A: Benchmark Corpus And Scoreboard (`benchmark_grounded_finder` prerequisite)
+
+- [x] Define benchmark manifest schema with source, language, framework, setup, expected vulnerabilities, known noise, scan modes, and optional baselines.
+- [x] Implement benchmark run directory creation under `.openultrasast/benchmarks/<benchmark-run-id>/`.
+- [x] Implement finding-to-ground-truth matching by CWE, rule ID, file, function, sink, and evidence text.
+- [x] Implement benchmark metrics for recall, precision when possible, false-positive reasons, missed findings, runtime, model usage, and artifact links.
+- [x] Add benchmark fixtures or manifests for JavaScript/Node/web, Python/web, Java, and C/C++.
+- [x] Add external baseline ingestion for SARIF or normalized findings from Clearwing, Semgrep, CodeQL, clang-tidy, Bandit, npm audit, or language-appropriate tools.
+- [x] Add tests proving benchmark misses are recorded as calibration inputs without automatically suppressing unrelated findings.
+
+## Gate: `explain_and_fix_workflow`
+
+- [ ] Explain mode can explain a finding or benchmark miss with source evidence, reachability status, missing evidence, secure pattern, and prevention guidance.
+- [ ] Explain mode supports concise, learner, and reviewer audience levels without changing evidence levels.
+- [ ] Interactive fixing selects relevant security skills and proposes a minimal patch plan without silently mutating the target repository.
+- [ ] Fix guidance links to validation commands and required evidence transitions.
+
+## Phase 6B: Explain Mode And Interactive Fixing (`explain_and_fix_workflow` prerequisite)
+
+- [ ] Add explain artifact schema for finding ID, source refs, evidence refs, audience level, prevention guidance, selected skills, and validation advice.
+- [ ] Add `ousast explain <finding-id> --scan <scan-id>` CLI command.
+- [ ] Add explain support for benchmark misses and imported SARIF findings.
+- [ ] Implement language/vulnerability skill routing for JavaScript/Node/web, Python/web, Java, and C/C++.
+- [ ] Add interactive fixing plan schema with confirmation gates, selected skills, proposed patch scope, validation commands, and evidence requirements.
+- [ ] Ensure interactive fixing cannot mark patches validated unless the patch oracle and sandbox validation gates pass.
+- [ ] Add tests that explanation does not upgrade evidence level and patch guidance does not mutate repositories by default.

 ## Phase 7: Docker Sandbox (`sandboxed_dynamic_harness` prerequisite)

@@ -209,12 +245,13 @@ Do not report OpenUltraSAST as a working security harness until `usable_harness_
 - [ ] Narrow MCP tools cover scan, status, findings, evidence, artifacts, patch proposal, and report export.
 - [ ] OpenCode workflows can drive scan, triage, fusion, and patch audit without arbitrary shell exposure.
 - [ ] Cost controls, retries, redaction, docs, examples, and CI gates are in place.
+- [ ] OpenCode workflows can drive benchmark runs, finding explanations, and interactive fix proposals.

 ## Phase 12: MCP And OpenCode Commands (`opencode_product` prerequisite)

 - [x] Add project-local opencode skills for scan, triage, fix audit, and implementation steering.
 - [ ] Implement narrow MCP server.
- [ ] Add `openultrasast.scan`, `status`, `findings`, `get_finding`, `evidence`, `artifacts`, `propose_patch`, and `export_report` tools.
+- [ ] Add `openultrasast.scan`, `status`, `findings`, `get_finding`, `evidence`, `artifacts`, `benchmark`, `explain`, `propose_patch`, and `export_report` tools.
 - [ ] Add opencode command docs for quick, standard, deep, verify, and patch workflows.
 - [ ] Ensure MCP tools never expose arbitrary shell execution.
 - [ ] Add integration test for MCP tool listing and scan status.