diff --git a/README.md b/README.md index 343b331..1d2ca01 100644 --- a/README.md +++ b/README.md @@ -3,8 +3,8 @@ OpenUltraSAST is an independent OpenCode security harness. It combines HarnessX-style harness composition, OpenUltraCode verification discipline, OpenRouter-hosted models and embeddings, Docker-isolated code analysis, and -Trail of Bits security skills into a SAST workflow whose primary product goal is -**false-positive elimination** โ€” turning suspicion into evidence-backed findings +Trail of Bits security skills into a SAST workflow built around one goal: +eliminating false positives by turning suspicion into evidence-backed findings and learning from rejected claims. The full specification lives in `.kiro/specs/openrouter-sast-harness/` @@ -12,7 +12,7 @@ The full specification lives in `.kiro/specs/openrouter-sast-harness/` as an implementation oracle for proven source-hunting ideas, not as a dependency or fork target. -> **Maturity legend** โ€” โœ… implemented and tested ยท ๐ŸŸก primitive exists, not yet +> **Maturity legend:** โœ… implemented and tested ยท ๐ŸŸก primitive exists, not yet > auto-wired into the scan loop ยท ๐Ÿงญ designed, on the roadmap. The current > verified gate is `usable_harness_mvp`. @@ -52,13 +52,13 @@ repo โ”€โ”€โ–ถ preprocess (language, LOC, tags, fuzz entry points) โ”€โ”€โ–ถ report.md + report.sarif + manifest.json (shared finding IDs) ``` -- **quick** โœ… โ€” deterministic, language-scoped pattern rules + reachability + +- **quick** โœ…: deterministic, language-scoped pattern rules + reachability + verification. No model calls, fully reproducible. -- **standard** โœ… scheduling / ๐Ÿงญ LLM hunters โ€” runs the tiered hunter pool +- **standard** โœ… scheduling / ๐Ÿงญ LLM hunters: runs the tiered hunter pool (A/B/C budgets, retrieval context, selected Trail of Bits skill snippets, JSONL trajectories) and verification; language-aware LLM hunters are the next upgrade (today standard reuses the deterministic findings per target). -- **deep** ๐Ÿงญ โ€” sandboxed build/fuzz/dynamic reproduction (not yet implemented). +- **deep** ๐Ÿงญ: sandboxed build/fuzz/dynamic reproduction (not yet implemented). ## Detection coverage @@ -114,16 +114,16 @@ SQLi, prototype pollution) so recall is not self-fulfilling. | Java | 100% (4/4) | 0.0% | | **Overall** | **93.6% (44/47)** | **0.0%** | -The project goal โ€” **โ‰ฅ90% recall and <10% false positives** per language โ€” is -enforced as a regression gate in `tests/test_detection_benchmarks.py`, so a rule -change that drops recall or raises false positives fails CI. +The project goal is **โ‰ฅ90% recall and <10% false positives** per language. The +gate lives in `tests/test_detection_benchmarks.py`, so a rule change that drops +recall or raises false positives fails CI. ## OpenCode commands, fusion, and ultra workflows -OpenUltraSAST is driven from the command line. You can call the `ousast` CLI -directly, or run [OpenCode](https://opencode.ai) in the repository and let the -agent run the harness for you โ€” OpenCode is invoked from the command line, loads -the project skills below, and executes `ousast` plus the triage/fix workflow. +OpenUltraSAST runs from the command line. You can call the `ousast` CLI directly, +or run [OpenCode](https://opencode.ai) in the repository and let the agent run +the harness. OpenCode loads the project skills below, then executes `ousast` plus +the triage/fix workflow. ```bash opencode run "scan this repo with ousast in quick mode and triage the findings" @@ -139,13 +139,13 @@ Project skills that steer the harness from OpenCode (`.opencode/skills/`): | `openultrasast-triage` โœ… | false-positive elimination, ranking calibration, verifier adjudication | | `openultrasast-fix-audit` โœ… | OpenUltraCode fix lifecycle and adversarial fix review | -**Ultra workflows (OpenUltraCode discipline)** โ€” fixes are never a one-shot -patch prompt. `openultrasast-fix-audit` runs the bounded lifecycle +**Ultra workflows (OpenUltraCode discipline):** fixes are never a one-shot patch +prompt. `openultrasast-fix-audit` runs the bounded lifecycle `intake โ†’ plan โ†’ minimal patch โ†’ adversarial review โ†’ reconcile โ†’ fresh verification โ†’ ready`, and a patch can only reach `patch_validated` after sandboxed validation passes with no accepted blocking findings. -**Fusion (deepening) ๐Ÿงญ** โ€” when a finding needs more reasoning than the normal +**Fusion (deepening) ๐Ÿงญ:** when a finding needs more reasoning than the normal ranker โ†’ hunter โ†’ verifier โ†’ mapping loop provides (critical/high severity, verifier disagreement, conflicting static vs semantic evidence, risky fixes, or an explicit high-assurance request), fusion runs two independent panels that @@ -184,9 +184,9 @@ When a finding is still rejected after review, `openultrasast-triage` records a taxonomy (`unreachable_path`, `missing_attacker_control`, `sanitizer_disproved`, `static_rule_mismatch`, `incorrect_model_assumption`, `duplicate`, `insufficient_impact`, `unsupported`, `contradicted`, `unverified`), evidence, -and a **scope**. Scoping is the safety property: a rejected `syscall_entry` claim -in `auth/` demotes only `auth/โ€ฆ`, never the whole vulnerability class โ€” proven by -`tests/test_calibration.py::test_scoped_false_positive_demotion_does_not_suppress_class_globally`. +and a **scope**. Scoping keeps the learning narrow: a rejected `syscall_entry` +claim in `auth/` demotes only `auth/โ€ฆ`, never the whole vulnerability class. This +is covered by `tests/test_calibration.py::test_scoped_false_positive_demotion_does_not_suppress_class_globally`. Demotion (with an audit trail) is preferred over deletion, so an analyst can always see *why* something was downgraded. @@ -209,7 +209,7 @@ overflow the regex engine cannot see: ``` The `next_improvement_candidate` routes the gap to the stage that should close -it โ€” a static rule, SARIF source, entry-point mapping, retrieval package, +it: a static rule, SARIF source, entry-point mapping, retrieval package, language-aware hunter prompt, dynamic reproducer, or skill route. Once a miss is addressed, it stays closed because the recall/precision gate runs in CI. `external_baseline_deltas.json` additionally shows where another tool found a @@ -218,11 +218,10 @@ vulnerability OpenUltraSAST missed (or vice-versa). ## The HarnessX self-improving cycle Each scan is a composed harness of typed processors with read/write **state -contracts** (strict mode fails the scan on a violation; warn mode records a -degradation) and a full event trace. The loop is now **closed automatically -inside the pipeline** โ€” no agent required: every scan persists its verifier -rejections as scoped learnings and the next scan loads them and demotes those -scopes before findings are produced. +contracts**. Strict mode fails the scan on a violation; warn mode records a +degradation. The loop now runs inside the pipeline without an agent: every scan +persists verifier rejections as scoped learnings, and the next scan loads them +and demotes those scopes before findings are produced. ``` โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ automatic ledger (no agent in the loop) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” @@ -249,11 +248,11 @@ priority is demoted rather than the finding deleted, the audit trail (`applied_calibrations.json`, the ledger) shows exactly why a surface lost attention. Covered by `tests/test_pipeline_calibration.py`. -Beyond the automatic ranking loop, **OpenCode still steers the richer -adjustments** โ€” prompt constraints, retrieval filters, skill routing, and -benchmark-miss triage โ€” via `openultrasast-triage`; those `RankingCalibration` -fields are already produced and will be consumed directly once the language-aware -LLM hunters land (`standard_security_harness`). +Beyond the automatic ranking loop, `openultrasast-triage` still lets OpenCode +adjust prompt constraints, retrieval filters, skill routing, and benchmark-miss +triage. Those `RankingCalibration` fields are already produced and will be +consumed directly once the language-aware LLM hunters land +(`standard_security_harness`). ## Development diff --git a/src/openultrasast/calibration.py b/src/openultrasast/calibration.py index 6226769..b304794 100644 --- a/src/openultrasast/calibration.py +++ b/src/openultrasast/calibration.py @@ -9,8 +9,7 @@ from .findings import StaticFinding from .rank import RankingScore from .verification import VerificationResult, VerificationStatus -# Calibration may push a repeatedly-rejected scope below the natural ranking -# minimum (1.0) so accumulated false positives keep losing priority over runs. +# Repeated rejections can push a scope below the normal 1.0 ranking floor. MIN_CALIBRATED_PRIORITY = 0.1 diff --git a/src/openultrasast/cli.py b/src/openultrasast/cli.py index 7d5dd83..e3d5a61 100644 --- a/src/openultrasast/cli.py +++ b/src/openultrasast/cli.py @@ -177,8 +177,7 @@ def _persist_calibration_feedback( findings: list[StaticFinding], verifications: list[VerificationResult], ) -> list[FalsePositiveLearning]: - # Turn this run's non-accepted verifier outcomes into scoped learnings and - # merge them into the persistent ledger so the next scan demotes those scopes. + # Store rejected or held findings as scoped learnings for the next scan. new_learnings = learnings_from_verifications(findings, verifications) merged = merge_false_positive_learnings(prior_learnings, new_learnings) write_false_positive_learnings(merged, ledger_path) diff --git a/src/openultrasast/findings.py b/src/openultrasast/findings.py index 078c596..f180c3c 100644 --- a/src/openultrasast/findings.py +++ b/src/openultrasast/findings.py @@ -69,8 +69,8 @@ PATTERN_RULES = ( title="Unsafe C format string needs review", severity="high", tags=("memory_unsafe",), - # Format function whose format argument is a bare identifier (not a string - # literal), i.e. the classic format-string sink. printf("...") is ignored. + # Flag format functions that receive a variable format string. + # Calls with string literals, such as printf("..."), are ignored. pattern=re.compile(r"\b(printf|vprintf|vsprintf|vsnprintf|vfprintf|syslog)\s*\(\s*[A-Za-z_]"), cwe="CWE-134", languages=C_LANGUAGES, @@ -144,8 +144,8 @@ PATTERN_RULES = ( title="Python SQL string composition needs review", severity="high", tags=("injection",), - # .execute(...) where the argument is built with %-formatting, .format(), - # string concatenation, or an f-string. Parameterised %s placeholders pass. + # Flag .execute(...) calls built with %-formatting, .format(), string + # concatenation, or an f-string. Parameterised %s placeholders pass. pattern=re.compile(r"(\.execute)\s*\(\s*[^\n]*(?:[\"']\s*%\s*[(A-Za-z_]|\.format\s*\(|[\"']\s*\+|\bf[\"'])"), cwe="CWE-89", languages=PYTHON_LANGUAGES, @@ -203,8 +203,8 @@ PATTERN_RULES = ( title="Node child_process command execution needs review", severity="high", tags=("syscall_entry",), - # exec/execSync invoke a shell; execFile/spawn with an argv array do not and - # are the safe alternatives, so they are intentionally excluded here. + # exec and execSync invoke a shell. execFile and spawn with an argv array + # do not, so this rule leaves them alone. pattern=re.compile(r"\b(exec|execSync)\s*\("), cwe="CWE-78", languages=JS_LANGUAGES, @@ -316,8 +316,8 @@ PATTERN_RULES = ( title="Java SQL string composition needs review", severity="high", tags=("injection",), - # A string literal that begins with a SQL keyword and is concatenated with - # untrusted data. Parameterised queries (placeholders, no '+') are ignored. + # Flag SQL strings that start with a statement keyword and then concatenate + # more data. Parameterised queries with placeholders are ignored. pattern=re.compile(r"(?i)\"\s*(select|insert|update|delete)\b[^\"]*\"\s*\+"), cwe="CWE-89", languages=JAVA_LANGUAGES, @@ -346,7 +346,7 @@ PATTERN_RULES = ( title="Groovy template unescaped output needs review", severity="high", tags=("injection",), - # Unescaped output of a bound variable; string literals are not flagged. + # Flag unescaped bound variables. String literals are ignored. pattern=re.compile(r"(yieldUnescaped)\s+[A-Za-z_]"), cwe="CWE-79", languages=GROOVY_LANGUAGES,