docs: drop the skills set from the detector requirement and refresh stale framing

The maintainer settled the conflict the group 2 review escalated: a detector
configuration has no skills set. The skill router belongs to the tool's earlier
shape, and a family's prompt, checklist, tools and counterexamples already carry
what a round evolves.

The README described the project as an editor-agent harness whose skills steer
it, and pointed users at the maintainer's spec directory. The tool is a CLI now
and the agent integrations are optional, so the opening says that, the skills
section describes what it actually is, and the spec pointer is gone from the
user-facing text.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-06 10:36:31 +02:00
co-authored by Claude Opus 5
parent 18ee3a2e77
commit c14ac8344f
3 changed files with 28 additions and 31 deletions
+1 -1
View File
@@ -79,7 +79,7 @@ OpenUltraSAST's detection numbers cannot be trusted and cannot be improved. The
#### Acceptance Criteria
1. The Learning Harness shall hold one detector configuration per family consisting of a prompt under a stated length cap, a skills set, a curated tool set, a step budget, a cost budget, an evidence checklist and a set of hard negatives and counterexamples.
1. The Learning Harness shall hold one detector configuration per family consisting of a prompt under a stated length cap, a curated tool set, a step budget, a cost budget, an evidence checklist and a set of hard negatives and counterexamples.
2. While a round evolves one family, the Learning Harness shall keep every other family's configuration unchanged.
3. When a code region is classified, the Learning Harness shall run the detector of every admitted family plus a generalist detector, and shall record on each finding the family, the detector configuration version and the verifier outcome.
4. The Learning Harness shall give a detector the whole file of a region and the files its curated tools resolve, not only an excerpt around a hotspot.
+2 -2
View File
@@ -1,7 +1,7 @@
{
"feature_name": "learning-harness",
"created_at": "2026-09-06T06:32:41Z",
"updated_at": "2026-09-06T08:20:56Z",
"updated_at": "2026-09-06T08:34:55Z",
"language": "en",
"phase": "implementation",
"approvals": {
@@ -26,5 +26,5 @@
"tree-sitter-overlay-extra",
"harnessx-self-improving-rulesets"
],
"approval_note": "2026-09-06 requirements approved by the maintainer in session ('approve the requirements and run the design phase'); 2026-09-06 design approved by the maintainer in session ('approve the design and generate the tasks'); 2026-09-06 tasks approved by the maintainer in session ('approve the tasks and go on with the implementation'); 2026-09-06 requirements amended by the maintainer in session ('amend the requirements to untrusted_destination and note the flat taxonomy'): Req 1.1 names untrusted destination instead of server-side request forgery, and Req 2.7 and 3.7 require every agreement and family figure to state that hierarchical partial credit could not apply while the taxonomy is flat"
"approval_note": "2026-09-06 requirements approved by the maintainer in session ('approve the requirements and run the design phase'); 2026-09-06 design approved by the maintainer in session ('approve the design and generate the tasks'); 2026-09-06 tasks approved by the maintainer in session ('approve the tasks and go on with the implementation'); 2026-09-06 requirements amended by the maintainer in session ('amend the requirements to untrusted_destination and note the flat taxonomy'): Req 1.1 names untrusted destination instead of server-side request forgery, and Req 2.7 and 3.7 require every agreement and family figure to state that hierarchical partial credit could not apply while the taxonomy is flat; 2026-09-06 Req 6.1 amended by the maintainer in session ('drop the skills set from the requirement'): a detector configuration has no skills set, because the skill router belongs to the tool's earlier shape and a family's prompt, checklist, tools and counterexamples already carry what a round evolves"
}
+25 -28
View File
@@ -1,16 +1,15 @@
# OpenUltraSAST
OpenUltraSAST is an independent OpenCode security harness. It combines
HarnessX-style harness composition, OpenUltraCode verification discipline,
OpenRouter-hosted models and embeddings, Docker-isolated code analysis, and
Trail of Bits security skills into a SAST workflow built around one goal:
eliminating false positives by turning suspicion into evidence-backed findings
and learning from rejected claims.
OpenUltraSAST is a command-line security analysis harness built around one
goal: eliminating false positives by turning a suspicion into an
evidence-backed finding, and learning from the claims that were rejected. A
scan combines static rules, a semantic overlay, optional model-driven hunting
and Docker-isolated proof, and every claim carries the evidence that earned its
place.
The full specification lives in `.kiro/specs/openrouter-sast-harness/`
(`requirements.md`, `design.md`, `tasks.md`, `assessment.md`). Clearwing is used
as an implementation oracle for proven source-hunting ideas, not as a dependency
or fork target.
The project began as an editor-agent skill and is now a CLI first. Agent
integrations remain, but nothing requires one: `ousast` is the interface, and
the model-driven parts are optional extras that degrade visibly when absent.
> **Maturity legend:** ✅ implemented and tested · 🟡 primitive exists, not yet
> auto-wired into the scan loop · 🧭 designed, on the roadmap. The current
@@ -115,7 +114,7 @@ No penalty → 100; one reachable severity-5 finding → ~43. The **reachability
**Artifacts** ✅: the `score` stage writes `score.json` (project score, `max_severity`, `penalty_total`, `by_category`, `out_of_scope_dynamic_only`, `unmapped_cwe`, gate verdict) and merges the same block into `manifest.json`. Scoring is zero-dependency (stdlib only). One mapping detail: verycode has no CWE-120 (generic buffer copy), so the C/C++ memory-unsafe rules listed under [Detection coverage](#detection-coverage) carry **CWE-121** (Stack-Based Buffer Overflow, severity 5), which the policy does govern.
🧭 This is the first implemented slice (**Phase 1**) of the `.kiro/specs/harnessx-self-improving-rulesets/` spec: a central, policy-governed severity model and a project score. Rules-as-data and the full HarnessX self-improvement loop over rulesets are later phases on that roadmap.
🧭 This is the first implemented slice of the self-improving ruleset work: a central, policy-governed severity model and a project score. Rules-as-data and the full self-improvement loop over rulesets are later phases.
## Running the benchmarks
@@ -163,20 +162,18 @@ The project goal is **≥90% recall and <10% false positives** per language. The
gate lives in `tests/test_detection_benchmarks.py`, so a rule change that drops
recall or raises false positives fails CI.
## OpenCode commands, fusion, and ultra workflows
## Agent integrations, fusion, and ultra workflows
OpenUltraSAST runs from the command line. You can call the `ousast` CLI directly,
or run [OpenCode](https://opencode.ai) in the repository and let the agent run
the harness. OpenCode loads the project skills below, then executes `ousast` plus
the triage/fix workflow.
The `ousast` CLI is the interface. An agent that can run shell commands needs
nothing else:
```bash
opencode run "scan this repo with ousast in quick mode and triage the findings"
# or drive the CLI yourself:
uv run ousast scan . --mode quick --fail-on verified
```
Project skills that steer the harness from OpenCode (`.opencode/skills/`):
For [OpenCode](https://opencode.ai) there are optional project skills that
describe the workflows above, so the agent drives the same CLI
(`.opencode/skills/`):
| Skill | Purpose |
| --- | --- |
@@ -227,9 +224,9 @@ decider_model = "gpt-4o" # optional: disclosed in the decision's model IDs
high_assurance = false # true forces fusion on every finding
```
> The `kiro-*` skills and `AGENTS.md` in this repo are the maintainer's
> spec-driven development tooling. They are not part of using OpenUltraSAST and
> can be ignored by users.
> The `kiro-*` skills, `AGENTS.md` and the `.kiro/` directory are the
> maintainer's spec-driven development tooling. They are not part of using
> OpenUltraSAST and can be ignored by users.
## Evidence ladder: how a false positive is eliminated
@@ -282,8 +279,8 @@ overflow the regex engine cannot see:
```
The `next_improvement_candidate` routes the gap to the stage that should close
it: a static rule, SARIF source, entry-point mapping, retrieval package,
language-aware hunter prompt, dynamic reproducer, or skill route. Once a miss is
it: a static rule, SARIF source, entry-point mapping, retrieval package, hunter
prompt, dynamic reproducer, or skill route. Once a miss is
addressed, it stays closed because the recall/precision gate runs in CI.
`external_baseline_deltas.json` additionally shows where another tool found a
vulnerability OpenUltraSAST missed (or vice-versa).
@@ -395,10 +392,10 @@ becomes the richer *proposer* that plugs into this same validated, gated
machinery — the safety contract (bounded levers, replay/novelty/journal gates,
hard acceptance gate, byte-for-byte revert) is identical either way.
Beyond these loops, `openultrasast-triage` still lets OpenCode adjust prompt
constraints, retrieval filters, skill routing, and benchmark-miss triage. Those
`RankingCalibration` fields are already produced and will be consumed directly
once the language-aware LLM hunters land (`standard_security_harness`).
Beyond these loops, the `openultrasast-triage` skill lets an agent adjust
prompt constraints, retrieval filters, skill routing and benchmark-miss triage.
Those `RankingCalibration` fields are already produced and will be consumed
directly once the per-family detectors land.
## Security & hardening