mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
docs: drop the skills set from the detector requirement and refresh stale framing
The maintainer settled the conflict the group 2 review escalated: a detector configuration has no skills set. The skill router belongs to the tool's earlier shape, and a family's prompt, checklist, tools and counterexamples already carry what a round evolves. The README described the project as an editor-agent harness whose skills steer it, and pointed users at the maintainer's spec directory. The tool is a CLI now and the agent integrations are optional, so the opening says that, the skills section describes what it actually is, and the spec pointer is gone from the user-facing text. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
co-authored by
Claude Opus 5
parent
18ee3a2e77
commit
c14ac8344f
@@ -79,7 +79,7 @@ OpenUltraSAST's detection numbers cannot be trusted and cannot be improved. The
|
|||||||
|
|
||||||
#### Acceptance Criteria
|
#### Acceptance Criteria
|
||||||
|
|
||||||
1. The Learning Harness shall hold one detector configuration per family consisting of a prompt under a stated length cap, a skills set, a curated tool set, a step budget, a cost budget, an evidence checklist and a set of hard negatives and counterexamples.
|
1. The Learning Harness shall hold one detector configuration per family consisting of a prompt under a stated length cap, a curated tool set, a step budget, a cost budget, an evidence checklist and a set of hard negatives and counterexamples.
|
||||||
2. While a round evolves one family, the Learning Harness shall keep every other family's configuration unchanged.
|
2. While a round evolves one family, the Learning Harness shall keep every other family's configuration unchanged.
|
||||||
3. When a code region is classified, the Learning Harness shall run the detector of every admitted family plus a generalist detector, and shall record on each finding the family, the detector configuration version and the verifier outcome.
|
3. When a code region is classified, the Learning Harness shall run the detector of every admitted family plus a generalist detector, and shall record on each finding the family, the detector configuration version and the verifier outcome.
|
||||||
4. The Learning Harness shall give a detector the whole file of a region and the files its curated tools resolve, not only an excerpt around a hotspot.
|
4. The Learning Harness shall give a detector the whole file of a region and the files its curated tools resolve, not only an excerpt around a hotspot.
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
{
|
{
|
||||||
"feature_name": "learning-harness",
|
"feature_name": "learning-harness",
|
||||||
"created_at": "2026-09-06T06:32:41Z",
|
"created_at": "2026-09-06T06:32:41Z",
|
||||||
"updated_at": "2026-09-06T08:20:56Z",
|
"updated_at": "2026-09-06T08:34:55Z",
|
||||||
"language": "en",
|
"language": "en",
|
||||||
"phase": "implementation",
|
"phase": "implementation",
|
||||||
"approvals": {
|
"approvals": {
|
||||||
@@ -26,5 +26,5 @@
|
|||||||
"tree-sitter-overlay-extra",
|
"tree-sitter-overlay-extra",
|
||||||
"harnessx-self-improving-rulesets"
|
"harnessx-self-improving-rulesets"
|
||||||
],
|
],
|
||||||
"approval_note": "2026-09-06 requirements approved by the maintainer in session ('approve the requirements and run the design phase'); 2026-09-06 design approved by the maintainer in session ('approve the design and generate the tasks'); 2026-09-06 tasks approved by the maintainer in session ('approve the tasks and go on with the implementation'); 2026-09-06 requirements amended by the maintainer in session ('amend the requirements to untrusted_destination and note the flat taxonomy'): Req 1.1 names untrusted destination instead of server-side request forgery, and Req 2.7 and 3.7 require every agreement and family figure to state that hierarchical partial credit could not apply while the taxonomy is flat"
|
"approval_note": "2026-09-06 requirements approved by the maintainer in session ('approve the requirements and run the design phase'); 2026-09-06 design approved by the maintainer in session ('approve the design and generate the tasks'); 2026-09-06 tasks approved by the maintainer in session ('approve the tasks and go on with the implementation'); 2026-09-06 requirements amended by the maintainer in session ('amend the requirements to untrusted_destination and note the flat taxonomy'): Req 1.1 names untrusted destination instead of server-side request forgery, and Req 2.7 and 3.7 require every agreement and family figure to state that hierarchical partial credit could not apply while the taxonomy is flat; 2026-09-06 Req 6.1 amended by the maintainer in session ('drop the skills set from the requirement'): a detector configuration has no skills set, because the skill router belongs to the tool's earlier shape and a family's prompt, checklist, tools and counterexamples already carry what a round evolves"
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,16 +1,15 @@
|
|||||||
# OpenUltraSAST
|
# OpenUltraSAST
|
||||||
|
|
||||||
OpenUltraSAST is an independent OpenCode security harness. It combines
|
OpenUltraSAST is a command-line security analysis harness built around one
|
||||||
HarnessX-style harness composition, OpenUltraCode verification discipline,
|
goal: eliminating false positives by turning a suspicion into an
|
||||||
OpenRouter-hosted models and embeddings, Docker-isolated code analysis, and
|
evidence-backed finding, and learning from the claims that were rejected. A
|
||||||
Trail of Bits security skills into a SAST workflow built around one goal:
|
scan combines static rules, a semantic overlay, optional model-driven hunting
|
||||||
eliminating false positives by turning suspicion into evidence-backed findings
|
and Docker-isolated proof, and every claim carries the evidence that earned its
|
||||||
and learning from rejected claims.
|
place.
|
||||||
|
|
||||||
The full specification lives in `.kiro/specs/openrouter-sast-harness/`
|
The project began as an editor-agent skill and is now a CLI first. Agent
|
||||||
(`requirements.md`, `design.md`, `tasks.md`, `assessment.md`). Clearwing is used
|
integrations remain, but nothing requires one: `ousast` is the interface, and
|
||||||
as an implementation oracle for proven source-hunting ideas, not as a dependency
|
the model-driven parts are optional extras that degrade visibly when absent.
|
||||||
or fork target.
|
|
||||||
|
|
||||||
> **Maturity legend:** ✅ implemented and tested · 🟡 primitive exists, not yet
|
> **Maturity legend:** ✅ implemented and tested · 🟡 primitive exists, not yet
|
||||||
> auto-wired into the scan loop · 🧭 designed, on the roadmap. The current
|
> auto-wired into the scan loop · 🧭 designed, on the roadmap. The current
|
||||||
@@ -115,7 +114,7 @@ No penalty → 100; one reachable severity-5 finding → ~43. The **reachability
|
|||||||
|
|
||||||
**Artifacts** ✅: the `score` stage writes `score.json` (project score, `max_severity`, `penalty_total`, `by_category`, `out_of_scope_dynamic_only`, `unmapped_cwe`, gate verdict) and merges the same block into `manifest.json`. Scoring is zero-dependency (stdlib only). One mapping detail: verycode has no CWE-120 (generic buffer copy), so the C/C++ memory-unsafe rules listed under [Detection coverage](#detection-coverage) carry **CWE-121** (Stack-Based Buffer Overflow, severity 5), which the policy does govern.
|
**Artifacts** ✅: the `score` stage writes `score.json` (project score, `max_severity`, `penalty_total`, `by_category`, `out_of_scope_dynamic_only`, `unmapped_cwe`, gate verdict) and merges the same block into `manifest.json`. Scoring is zero-dependency (stdlib only). One mapping detail: verycode has no CWE-120 (generic buffer copy), so the C/C++ memory-unsafe rules listed under [Detection coverage](#detection-coverage) carry **CWE-121** (Stack-Based Buffer Overflow, severity 5), which the policy does govern.
|
||||||
|
|
||||||
🧭 This is the first implemented slice (**Phase 1**) of the `.kiro/specs/harnessx-self-improving-rulesets/` spec: a central, policy-governed severity model and a project score. Rules-as-data and the full HarnessX self-improvement loop over rulesets are later phases on that roadmap.
|
🧭 This is the first implemented slice of the self-improving ruleset work: a central, policy-governed severity model and a project score. Rules-as-data and the full self-improvement loop over rulesets are later phases.
|
||||||
|
|
||||||
## Running the benchmarks
|
## Running the benchmarks
|
||||||
|
|
||||||
@@ -163,20 +162,18 @@ The project goal is **≥90% recall and <10% false positives** per language. The
|
|||||||
gate lives in `tests/test_detection_benchmarks.py`, so a rule change that drops
|
gate lives in `tests/test_detection_benchmarks.py`, so a rule change that drops
|
||||||
recall or raises false positives fails CI.
|
recall or raises false positives fails CI.
|
||||||
|
|
||||||
## OpenCode commands, fusion, and ultra workflows
|
## Agent integrations, fusion, and ultra workflows
|
||||||
|
|
||||||
OpenUltraSAST runs from the command line. You can call the `ousast` CLI directly,
|
The `ousast` CLI is the interface. An agent that can run shell commands needs
|
||||||
or run [OpenCode](https://opencode.ai) in the repository and let the agent run
|
nothing else:
|
||||||
the harness. OpenCode loads the project skills below, then executes `ousast` plus
|
|
||||||
the triage/fix workflow.
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
opencode run "scan this repo with ousast in quick mode and triage the findings"
|
|
||||||
# or drive the CLI yourself:
|
|
||||||
uv run ousast scan . --mode quick --fail-on verified
|
uv run ousast scan . --mode quick --fail-on verified
|
||||||
```
|
```
|
||||||
|
|
||||||
Project skills that steer the harness from OpenCode (`.opencode/skills/`):
|
For [OpenCode](https://opencode.ai) there are optional project skills that
|
||||||
|
describe the workflows above, so the agent drives the same CLI
|
||||||
|
(`.opencode/skills/`):
|
||||||
|
|
||||||
| Skill | Purpose |
|
| Skill | Purpose |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
@@ -227,9 +224,9 @@ decider_model = "gpt-4o" # optional: disclosed in the decision's model IDs
|
|||||||
high_assurance = false # true forces fusion on every finding
|
high_assurance = false # true forces fusion on every finding
|
||||||
```
|
```
|
||||||
|
|
||||||
> The `kiro-*` skills and `AGENTS.md` in this repo are the maintainer's
|
> The `kiro-*` skills, `AGENTS.md` and the `.kiro/` directory are the
|
||||||
> spec-driven development tooling. They are not part of using OpenUltraSAST and
|
> maintainer's spec-driven development tooling. They are not part of using
|
||||||
> can be ignored by users.
|
> OpenUltraSAST and can be ignored by users.
|
||||||
|
|
||||||
## Evidence ladder: how a false positive is eliminated
|
## Evidence ladder: how a false positive is eliminated
|
||||||
|
|
||||||
@@ -282,8 +279,8 @@ overflow the regex engine cannot see:
|
|||||||
```
|
```
|
||||||
|
|
||||||
The `next_improvement_candidate` routes the gap to the stage that should close
|
The `next_improvement_candidate` routes the gap to the stage that should close
|
||||||
it: a static rule, SARIF source, entry-point mapping, retrieval package,
|
it: a static rule, SARIF source, entry-point mapping, retrieval package, hunter
|
||||||
language-aware hunter prompt, dynamic reproducer, or skill route. Once a miss is
|
prompt, dynamic reproducer, or skill route. Once a miss is
|
||||||
addressed, it stays closed because the recall/precision gate runs in CI.
|
addressed, it stays closed because the recall/precision gate runs in CI.
|
||||||
`external_baseline_deltas.json` additionally shows where another tool found a
|
`external_baseline_deltas.json` additionally shows where another tool found a
|
||||||
vulnerability OpenUltraSAST missed (or vice-versa).
|
vulnerability OpenUltraSAST missed (or vice-versa).
|
||||||
@@ -395,10 +392,10 @@ becomes the richer *proposer* that plugs into this same validated, gated
|
|||||||
machinery — the safety contract (bounded levers, replay/novelty/journal gates,
|
machinery — the safety contract (bounded levers, replay/novelty/journal gates,
|
||||||
hard acceptance gate, byte-for-byte revert) is identical either way.
|
hard acceptance gate, byte-for-byte revert) is identical either way.
|
||||||
|
|
||||||
Beyond these loops, `openultrasast-triage` still lets OpenCode adjust prompt
|
Beyond these loops, the `openultrasast-triage` skill lets an agent adjust
|
||||||
constraints, retrieval filters, skill routing, and benchmark-miss triage. Those
|
prompt constraints, retrieval filters, skill routing and benchmark-miss triage.
|
||||||
`RankingCalibration` fields are already produced and will be consumed directly
|
Those `RankingCalibration` fields are already produced and will be consumed
|
||||||
once the language-aware LLM hunters land (`standard_security_harness`).
|
directly once the per-family detectors land.
|
||||||
|
|
||||||
## Security & hardening
|
## Security & hardening
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user