docs: humanize README and code comments

This commit is contained in:
Marius Ciepluch
2026-06-21 21:02:18 +02:00
parent 0c0137e591
commit 0d24033f32
4 changed files with 40 additions and 43 deletions
+29 -30
View File
@@ -3,8 +3,8 @@
OpenUltraSAST is an independent OpenCode security harness. It combines OpenUltraSAST is an independent OpenCode security harness. It combines
HarnessX-style harness composition, OpenUltraCode verification discipline, HarnessX-style harness composition, OpenUltraCode verification discipline,
OpenRouter-hosted models and embeddings, Docker-isolated code analysis, and OpenRouter-hosted models and embeddings, Docker-isolated code analysis, and
Trail of Bits security skills into a SAST workflow whose primary product goal is Trail of Bits security skills into a SAST workflow built around one goal:
**false-positive elimination** — turning suspicion into evidence-backed findings eliminating false positives by turning suspicion into evidence-backed findings
and learning from rejected claims. and learning from rejected claims.
The full specification lives in `.kiro/specs/openrouter-sast-harness/` The full specification lives in `.kiro/specs/openrouter-sast-harness/`
@@ -12,7 +12,7 @@ The full specification lives in `.kiro/specs/openrouter-sast-harness/`
as an implementation oracle for proven source-hunting ideas, not as a dependency as an implementation oracle for proven source-hunting ideas, not as a dependency
or fork target. or fork target.
> **Maturity legend** — ✅ implemented and tested · 🟡 primitive exists, not yet > **Maturity legend:** ✅ implemented and tested · 🟡 primitive exists, not yet
> auto-wired into the scan loop · 🧭 designed, on the roadmap. The current > auto-wired into the scan loop · 🧭 designed, on the roadmap. The current
> verified gate is `usable_harness_mvp`. > verified gate is `usable_harness_mvp`.
@@ -52,13 +52,13 @@ repo ──▶ preprocess (language, LOC, tags, fuzz entry points)
──▶ report.md + report.sarif + manifest.json (shared finding IDs) ──▶ report.md + report.sarif + manifest.json (shared finding IDs)
``` ```
- **quick** ✅ — deterministic, language-scoped pattern rules + reachability + - **quick** ✅: deterministic, language-scoped pattern rules + reachability +
verification. No model calls, fully reproducible. verification. No model calls, fully reproducible.
- **standard** ✅ scheduling / 🧭 LLM hunters — runs the tiered hunter pool - **standard** ✅ scheduling / 🧭 LLM hunters: runs the tiered hunter pool
(A/B/C budgets, retrieval context, selected Trail of Bits skill snippets, (A/B/C budgets, retrieval context, selected Trail of Bits skill snippets,
JSONL trajectories) and verification; language-aware LLM hunters are the next JSONL trajectories) and verification; language-aware LLM hunters are the next
upgrade (today standard reuses the deterministic findings per target). upgrade (today standard reuses the deterministic findings per target).
- **deep** 🧭 — sandboxed build/fuzz/dynamic reproduction (not yet implemented). - **deep** 🧭: sandboxed build/fuzz/dynamic reproduction (not yet implemented).
## Detection coverage ## Detection coverage
@@ -114,16 +114,16 @@ SQLi, prototype pollution) so recall is not self-fulfilling.
| Java | 100% (4/4) | 0.0% | | Java | 100% (4/4) | 0.0% |
| **Overall** | **93.6% (44/47)** | **0.0%** | | **Overall** | **93.6% (44/47)** | **0.0%** |
The project goal — **≥90% recall and <10% false positives** per language — is The project goal is **≥90% recall and <10% false positives** per language. The
enforced as a regression gate in `tests/test_detection_benchmarks.py`, so a rule gate lives in `tests/test_detection_benchmarks.py`, so a rule change that drops
change that drops recall or raises false positives fails CI. recall or raises false positives fails CI.
## OpenCode commands, fusion, and ultra workflows ## OpenCode commands, fusion, and ultra workflows
OpenUltraSAST is driven from the command line. You can call the `ousast` CLI OpenUltraSAST runs from the command line. You can call the `ousast` CLI directly,
directly, or run [OpenCode](https://opencode.ai) in the repository and let the or run [OpenCode](https://opencode.ai) in the repository and let the agent run
agent run the harness for you — OpenCode is invoked from the command line, loads the harness. OpenCode loads the project skills below, then executes `ousast` plus
the project skills below, and executes `ousast` plus the triage/fix workflow. the triage/fix workflow.
```bash ```bash
opencode run "scan this repo with ousast in quick mode and triage the findings" opencode run "scan this repo with ousast in quick mode and triage the findings"
@@ -139,13 +139,13 @@ Project skills that steer the harness from OpenCode (`.opencode/skills/`):
| `openultrasast-triage` ✅ | false-positive elimination, ranking calibration, verifier adjudication | | `openultrasast-triage` ✅ | false-positive elimination, ranking calibration, verifier adjudication |
| `openultrasast-fix-audit` ✅ | OpenUltraCode fix lifecycle and adversarial fix review | | `openultrasast-fix-audit` ✅ | OpenUltraCode fix lifecycle and adversarial fix review |
**Ultra workflows (OpenUltraCode discipline)** — fixes are never a one-shot **Ultra workflows (OpenUltraCode discipline):** fixes are never a one-shot patch
patch prompt. `openultrasast-fix-audit` runs the bounded lifecycle prompt. `openultrasast-fix-audit` runs the bounded lifecycle
`intake → plan → minimal patch → adversarial review → reconcile → fresh `intake → plan → minimal patch → adversarial review → reconcile → fresh
verification → ready`, and a patch can only reach `patch_validated` after verification → ready`, and a patch can only reach `patch_validated` after
sandboxed validation passes with no accepted blocking findings. sandboxed validation passes with no accepted blocking findings.
**Fusion (deepening) 🧭** — when a finding needs more reasoning than the normal **Fusion (deepening) 🧭:** when a finding needs more reasoning than the normal
ranker → hunter → verifier → mapping loop provides (critical/high severity, ranker → hunter → verifier → mapping loop provides (critical/high severity,
verifier disagreement, conflicting static vs semantic evidence, risky fixes, or verifier disagreement, conflicting static vs semantic evidence, risky fixes, or
an explicit high-assurance request), fusion runs two independent panels that an explicit high-assurance request), fusion runs two independent panels that
@@ -184,9 +184,9 @@ When a finding is still rejected after review, `openultrasast-triage` records a
taxonomy (`unreachable_path`, `missing_attacker_control`, `sanitizer_disproved`, taxonomy (`unreachable_path`, `missing_attacker_control`, `sanitizer_disproved`,
`static_rule_mismatch`, `incorrect_model_assumption`, `duplicate`, `static_rule_mismatch`, `incorrect_model_assumption`, `duplicate`,
`insufficient_impact`, `unsupported`, `contradicted`, `unverified`), evidence, `insufficient_impact`, `unsupported`, `contradicted`, `unverified`), evidence,
and a **scope**. Scoping is the safety property: a rejected `syscall_entry` claim and a **scope**. Scoping keeps the learning narrow: a rejected `syscall_entry`
in `auth/` demotes only `auth/…`, never the whole vulnerability class — proven by claim in `auth/` demotes only `auth/…`, never the whole vulnerability class. This
`tests/test_calibration.py::test_scoped_false_positive_demotion_does_not_suppress_class_globally`. is covered by `tests/test_calibration.py::test_scoped_false_positive_demotion_does_not_suppress_class_globally`.
Demotion (with an audit trail) is preferred over deletion, so an analyst can Demotion (with an audit trail) is preferred over deletion, so an analyst can
always see *why* something was downgraded. always see *why* something was downgraded.
@@ -209,7 +209,7 @@ overflow the regex engine cannot see:
``` ```
The `next_improvement_candidate` routes the gap to the stage that should close The `next_improvement_candidate` routes the gap to the stage that should close
it — a static rule, SARIF source, entry-point mapping, retrieval package, it: a static rule, SARIF source, entry-point mapping, retrieval package,
language-aware hunter prompt, dynamic reproducer, or skill route. Once a miss is language-aware hunter prompt, dynamic reproducer, or skill route. Once a miss is
addressed, it stays closed because the recall/precision gate runs in CI. addressed, it stays closed because the recall/precision gate runs in CI.
`external_baseline_deltas.json` additionally shows where another tool found a `external_baseline_deltas.json` additionally shows where another tool found a
@@ -218,11 +218,10 @@ vulnerability OpenUltraSAST missed (or vice-versa).
## The HarnessX self-improving cycle ## The HarnessX self-improving cycle
Each scan is a composed harness of typed processors with read/write **state Each scan is a composed harness of typed processors with read/write **state
contracts** (strict mode fails the scan on a violation; warn mode records a contracts**. Strict mode fails the scan on a violation; warn mode records a
degradation) and a full event trace. The loop is now **closed automatically degradation. The loop now runs inside the pipeline without an agent: every scan
inside the pipeline** — no agent required: every scan persists its verifier persists verifier rejections as scoped learnings, and the next scan loads them
rejections as scoped learnings and the next scan loads them and demotes those and demotes those scopes before findings are produced.
scopes before findings are produced.
``` ```
┌──────────── automatic ledger (no agent in the loop) ───────────┐ ┌──────────── automatic ledger (no agent in the loop) ───────────┐
@@ -249,11 +248,11 @@ priority is demoted rather than the finding deleted, the audit trail
(`applied_calibrations.json`, the ledger) shows exactly why a surface lost (`applied_calibrations.json`, the ledger) shows exactly why a surface lost
attention. Covered by `tests/test_pipeline_calibration.py`. attention. Covered by `tests/test_pipeline_calibration.py`.
Beyond the automatic ranking loop, **OpenCode still steers the richer Beyond the automatic ranking loop, `openultrasast-triage` still lets OpenCode
adjustments** — prompt constraints, retrieval filters, skill routing, and adjust prompt constraints, retrieval filters, skill routing, and benchmark-miss
benchmark-miss triage — via `openultrasast-triage`; those `RankingCalibration` triage. Those `RankingCalibration` fields are already produced and will be
fields are already produced and will be consumed directly once the language-aware consumed directly once the language-aware LLM hunters land
LLM hunters land (`standard_security_harness`). (`standard_security_harness`).
## Development ## Development
+1 -2
View File
@@ -9,8 +9,7 @@ from .findings import StaticFinding
from .rank import RankingScore from .rank import RankingScore
from .verification import VerificationResult, VerificationStatus from .verification import VerificationResult, VerificationStatus
# Calibration may push a repeatedly-rejected scope below the natural ranking # Repeated rejections can push a scope below the normal 1.0 ranking floor.
# minimum (1.0) so accumulated false positives keep losing priority over runs.
MIN_CALIBRATED_PRIORITY = 0.1 MIN_CALIBRATED_PRIORITY = 0.1
+1 -2
View File
@@ -177,8 +177,7 @@ def _persist_calibration_feedback(
findings: list[StaticFinding], findings: list[StaticFinding],
verifications: list[VerificationResult], verifications: list[VerificationResult],
) -> list[FalsePositiveLearning]: ) -> list[FalsePositiveLearning]:
# Turn this run's non-accepted verifier outcomes into scoped learnings and # Store rejected or held findings as scoped learnings for the next scan.
# merge them into the persistent ledger so the next scan demotes those scopes.
new_learnings = learnings_from_verifications(findings, verifications) new_learnings = learnings_from_verifications(findings, verifications)
merged = merge_false_positive_learnings(prior_learnings, new_learnings) merged = merge_false_positive_learnings(prior_learnings, new_learnings)
write_false_positive_learnings(merged, ledger_path) write_false_positive_learnings(merged, ledger_path)
+9 -9
View File
@@ -69,8 +69,8 @@ PATTERN_RULES = (
title="Unsafe C format string needs review", title="Unsafe C format string needs review",
severity="high", severity="high",
tags=("memory_unsafe",), tags=("memory_unsafe",),
# Format function whose format argument is a bare identifier (not a string # Flag format functions that receive a variable format string.
# literal), i.e. the classic format-string sink. printf("...") is ignored. # Calls with string literals, such as printf("..."), are ignored.
pattern=re.compile(r"\b(printf|vprintf|vsprintf|vsnprintf|vfprintf|syslog)\s*\(\s*[A-Za-z_]"), pattern=re.compile(r"\b(printf|vprintf|vsprintf|vsnprintf|vfprintf|syslog)\s*\(\s*[A-Za-z_]"),
cwe="CWE-134", cwe="CWE-134",
languages=C_LANGUAGES, languages=C_LANGUAGES,
@@ -144,8 +144,8 @@ PATTERN_RULES = (
title="Python SQL string composition needs review", title="Python SQL string composition needs review",
severity="high", severity="high",
tags=("injection",), tags=("injection",),
# .execute(...) where the argument is built with %-formatting, .format(), # Flag .execute(...) calls built with %-formatting, .format(), string
# string concatenation, or an f-string. Parameterised %s placeholders pass. # concatenation, or an f-string. Parameterised %s placeholders pass.
pattern=re.compile(r"(\.execute)\s*\(\s*[^\n]*(?:[\"']\s*%\s*[(A-Za-z_]|\.format\s*\(|[\"']\s*\+|\bf[\"'])"), pattern=re.compile(r"(\.execute)\s*\(\s*[^\n]*(?:[\"']\s*%\s*[(A-Za-z_]|\.format\s*\(|[\"']\s*\+|\bf[\"'])"),
cwe="CWE-89", cwe="CWE-89",
languages=PYTHON_LANGUAGES, languages=PYTHON_LANGUAGES,
@@ -203,8 +203,8 @@ PATTERN_RULES = (
title="Node child_process command execution needs review", title="Node child_process command execution needs review",
severity="high", severity="high",
tags=("syscall_entry",), tags=("syscall_entry",),
# exec/execSync invoke a shell; execFile/spawn with an argv array do not and # exec and execSync invoke a shell. execFile and spawn with an argv array
# are the safe alternatives, so they are intentionally excluded here. # do not, so this rule leaves them alone.
pattern=re.compile(r"\b(exec|execSync)\s*\("), pattern=re.compile(r"\b(exec|execSync)\s*\("),
cwe="CWE-78", cwe="CWE-78",
languages=JS_LANGUAGES, languages=JS_LANGUAGES,
@@ -316,8 +316,8 @@ PATTERN_RULES = (
title="Java SQL string composition needs review", title="Java SQL string composition needs review",
severity="high", severity="high",
tags=("injection",), tags=("injection",),
# A string literal that begins with a SQL keyword and is concatenated with # Flag SQL strings that start with a statement keyword and then concatenate
# untrusted data. Parameterised queries (placeholders, no '+') are ignored. # more data. Parameterised queries with placeholders are ignored.
pattern=re.compile(r"(?i)\"\s*(select|insert|update|delete)\b[^\"]*\"\s*\+"), pattern=re.compile(r"(?i)\"\s*(select|insert|update|delete)\b[^\"]*\"\s*\+"),
cwe="CWE-89", cwe="CWE-89",
languages=JAVA_LANGUAGES, languages=JAVA_LANGUAGES,
@@ -346,7 +346,7 @@ PATTERN_RULES = (
title="Groovy template unescaped output needs review", title="Groovy template unescaped output needs review",
severity="high", severity="high",
tags=("injection",), tags=("injection",),
# Unescaped output of a bound variable; string literals are not flagged. # Flag unescaped bound variables. String literals are ignored.
pattern=re.compile(r"(yieldUnescaped)\s+[A-Za-z_]"), pattern=re.compile(r"(yieldUnescaped)\s+[A-Za-z_]"),
cwe="CWE-79", cwe="CWE-79",
languages=GROOVY_LANGUAGES, languages=GROOVY_LANGUAGES,