mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-22 06:29:29 +00:00
chore(skills): apply the 2026-10-02 review -- current CLI and evidence, merges, removals, steering names
openultrasast-scan: evidence ladder as verification.py defines it, with the levels nothing produces today; --mode prerequisites, --fail-on, recorded degradations, run artifacts, ousast mcp; retired [evidence] threshold language removed. openultrasast-triage: reasons verified against calibration.py; deterministic fusion; adjudications as labels, plane verdicts as features, rule changes through improve or a registered experiment; fix-audit merged as a fix-and-re-scan section. kiro: spec-quick and spec-batch removed; validate-gap and validate-design merged into spec-design (--gap, --review; the review gate stays); steering-custom merged into steering; core steering names overview.md, roadmap.md, safety-net.md; instrument rule in debug and verify-completion; repo gates named in validate-impl. AGENTS.md, README skills paragraph and tests/test_agent_skills.py (.agents/skills, two product skills, drift tests) updated. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
5010b15ffc
commit
16d9ef7110
@@ -1,23 +1,29 @@
|
||||
---
|
||||
name: openultrasast-triage
|
||||
description: OpenUltraSAST false-positive elimination, ranking calibration, evidence ladder, verifier triage, and finding adjudication. Use when reviewing findings, reducing false positives, or deciding whether evidence is enough.
|
||||
description: OpenUltraSAST finding triage - false-positive adjudication with the fixed reason taxonomy, the evidence ladder, what a verdict may feed (labels, never rule edits), and fixing then re-scanning. Use when reviewing findings, reducing false positives, deciding whether evidence is enough, or auditing a fix for a finding.
|
||||
---
|
||||
|
||||
# OpenUltraSAST Triage
|
||||
|
||||
Use this skill when triaging OpenUltraSAST findings from opencode.
|
||||
Use this skill when triaging OpenUltraSAST findings or auditing a fix for one.
|
||||
|
||||
## Triage Loop
|
||||
|
||||
1. Identify the finding, path, rule, severity, confidence, and evidence level.
|
||||
2. Separate deterministic evidence from model assumptions.
|
||||
3. Check attacker control, reachability, sanitizer behavior, duplicates, impact, and contradictory evidence.
|
||||
4. If rejected or unresolved, record the false-positive reason and scope.
|
||||
5. Prefer scoped demotion over deleting a vulnerability class globally.
|
||||
6. Feed the outcome into ranking calibration and prompt constraints.
|
||||
1. Identify the finding: id, path, rule, CWE and policy severity, confidence, evidence level and
|
||||
verification status (`findings.json`, `verification.json` in the run directory).
|
||||
2. Separate deterministic evidence (rule match, graph entailment, attached reachability) from model
|
||||
assumptions.
|
||||
3. Check attacker control, reachability, sanitizer behaviour, duplicates, impact, and contradictory
|
||||
evidence.
|
||||
4. If rejected or unresolved, record the false-positive reason and its scope.
|
||||
5. Prefer scoped demotion over deleting a vulnerability class globally: a rejection in `auth/`
|
||||
demotes `auth/...`, never the class (`calibration.py`).
|
||||
6. Record the adjudication. Do not edit a rule because of one finding (see below).
|
||||
|
||||
## False-Positive Reasons
|
||||
|
||||
The taxonomy is `FalsePositiveReason` in `src/openultrasast/calibration.py`; use these values exactly:
|
||||
|
||||
- `unreachable_path`
|
||||
- `missing_attacker_control`
|
||||
- `sanitizer_disproved`
|
||||
@@ -31,4 +37,39 @@ Use this skill when triaging OpenUltraSAST findings from opencode.
|
||||
|
||||
## Decision Standard
|
||||
|
||||
If evidence is uncertain, mark the finding as requiring next evidence instead of escalating severity. Difficult or high-impact disagreements should trigger fusion/deeper reasoning when available.
|
||||
If evidence is uncertain, mark the finding `needs_evidence` and name the next evidence instead of
|
||||
escalating severity. Fusion (`fusion.json`, standard mode) is deterministic: it adjudicates between
|
||||
the quick-rule and MAP panels by fixed triggers and is not a second opinion from a model. Independent
|
||||
model agreement exists only in the plane's verify passes a and b, with pass c on disputes, and the
|
||||
`agree` step; those verdicts arrive as features of the decision engine, never as a verification
|
||||
status.
|
||||
|
||||
## What an Adjudication Feeds
|
||||
|
||||
- Recorded adjudications (a per-finding verdict with a reason) become decision-engine labels through
|
||||
the fail-closed source list `src/openultrasast/learn/sources.toml` (`[[source]] id =
|
||||
"adjudications"`). Aggregate counts label nothing; only per-finding verdicts do.
|
||||
- Plane verdicts (`agreed`, `rejected`, `disputed`) are features, never labels. "Rejected by the
|
||||
plane" is not a negative.
|
||||
- Rule changes go through the ledger: `ousast improve` (the validator and the detection gate, holdout
|
||||
pairs per round) or a registered experiment (`ousast learn experiment register|units|run|analyse`).
|
||||
Never a hand edit of `rules.toml` per miss or per false positive.
|
||||
|
||||
## Fix and Re-scan
|
||||
|
||||
1. Intake: restate the finding id, evidence level, affected path, and the acceptance criterion.
|
||||
2. Plan: the smallest defensive change that addresses the root cause.
|
||||
3. Implement: modify only the files the finding requires.
|
||||
4. Adversarial review: look for bypasses, regressions, false assumptions, and unrelated edits. Use a
|
||||
differential review for changes in auth, crypto, config, parsers, and sandbox behaviour.
|
||||
5. Reconcile: fix each review finding or document why it does not apply.
|
||||
6. Verify: a re-scan of the checkout in the same mode, or a replay of the change with
|
||||
`ousast pre-push --base <before> --head <after> --artifact <file> <repo>`, shows the finding gone
|
||||
and no new findings. That re-scan is the evidence. Nothing in the tool records `patch_validated`;
|
||||
do not claim it.
|
||||
|
||||
Guardrails:
|
||||
|
||||
- Do not rewrite modules when a local fix is enough.
|
||||
- Do not change public APIs unless the vulnerability requires it.
|
||||
- Do not call a finding fixed from model reasoning alone; the re-scan or replay decides.
|
||||
|
||||
Reference in New Issue
Block a user