OpenUltraSAST

v1.2.0-alpha.1: experimental pre-push safety net. Integration and census repairs are verified; useful hook coverage, latency and independent capability qualification remain NO-GO. No real hook capability is enabled. See release notes and spec status.

OpenUltraSAST is a command-line security analysis harness built around one goal: eliminating false positives by turning a suspicion into an evidence-backed finding, and learning from the claims that were rejected. A scan combines static rules, a semantic overlay, optional model-driven hunting and Docker-isolated proof, and every claim carries the evidence that earned its place.

The project began as an editor-agent skill and is now a CLI first. Agent integrations remain, but nothing requires one: ousast is the interface, and the model-driven parts are optional extras that degrade visibly when absent.

Maturity legend: ✅ implemented and tested · 🟡 primitive exists, not yet auto-wired into the scan loop · 🧭 designed, on the roadmap. The current verified gate is usable_harness_mvp.

Install and quickstart

uv sync --group dev

# Scan a local repository (quick mode = deterministic, no model calls)
uv run ousast scan /path/to/repo --mode quick

# Build an embedding-ready chunk index
uv run ousast index /path/to/repo

# Score the detector against a benchmark manifest
uv run ousast benchmark benchmarks/manifests/python-vulnerable.toml --mode quick

# Differential pair eval: fire on vuln, stay silent on the fix (local + real GitHub VFCs)
uv run ousast pairs
uv run ousast pairs --slice sast   # OWASP Benchmark Java/Python + Juliet
uv run ousast pairs --slice vibe-py --pointers   # nightly: also harvest non-vendored pointer pairs into ~/.cache
uv run ousast mechanisms export --slice vibe-py   # seed mechanism candidates from trusted pairs (offline); `ousast improve` admits them under the holdout gate
uv run ousast pairs --slice vibe-py --loo        # leave-one-out recall of the corpus as a teacher
uv run python -m openultrasast.pair_gate

# Run the bounded self-improvement loop over the ruleset (benchmark-driven)
uv run ousast improve benchmarks/manifests/python-vulnerable.toml --dry-run

Every run writes auditable artifacts under .openultrasast/runs/<scan-id>/ (manifest.json, findings.json, verification.json, report.md, report.sarif, trace/events.jsonl, ranking/preprocess/mapping JSON). --fail-on findings|verified gives CI-friendly exit codes.

How a scan works

The CLI is thin; the HarnessRuntime owns the lifecycle and emits an event trace for every stage (task_start … stage_start/stage_end … task_end).

repo ──▶ preprocess (language, LOC, tags, fuzz entry points)
     ──▶ static mapping (Semgrep/CodeQL SARIF ingest, normalized hints)
     ──▶ entry-point + reachability mapping (routes, CLI, parsers, contracts)
     ──▶ rank (surface·0.5 + influence·0.2 + reachability·0.3)
     ──▶ findings  ─ quick: language-scoped pattern rules
     │             └ standard: tiered hunter pool (retrieval + skill snippets)
     ──▶ independent verifier (evidence ladder enforced)
     ──▶ report.md + report.sarif + manifest.json (shared finding IDs)

• quick ✅: deterministic, language-scoped pattern rules + reachability + verification. No model calls, fully reproducible.
• standard ✅ scheduling / 🧭 LLM hunters: runs the tiered hunter pool (A/B/C budgets, retrieval context, selected Trail of Bits skill snippets, JSONL trajectories) and verification; language-aware LLM hunters are the next upgrade (today standard reuses the deterministic findings per target).
• deep 🧭: sandboxed build/fuzz/dynamic reproduction (not yet implemented).

Detection coverage

Language-scoped rule families (rules only run against matching languages, which removes cross-language false positives):

Language	Classes (CWE)
C / C++	buffer overflow (120), format string (134), command injection (78)
Python	command injection (78), code injection (95), deserialization (502), SQLi (89), path traversal (22), SSRF (918), weak hash (327), SSTI (94), insecure default (489)
JavaScript / TS	command injection (78), code injection (95), reflected/DOM XSS (79), SQLi (89), path traversal (22), SSRF (918), weak hash (327), deserialization (502)
Java + Groovy templates	command injection (78), SQLi via concatenation (89), weak hash (327), deserialization (502), unescaped template XSS (79)

Cheat-sheet fixtures live under benchmarks/fixtures/ and feed the 90/10 smoke gate. Efficiency is measured separately on isolated vuln-vs-fixed pairs (ousast pairs, python -m openultrasast.pair_gate): the harness must fire on the vulnerable snapshot and stay silent after the patch. Local fixture counterparts are a CI gate; vendored GitHub VFCs (SVEN, CWE-Bench-Java, NVD patterns) are the honesty dashboard and the miss/fp signals for ousast improve. Dataset pointers (CVEfixes, DiverseVul, PrimeVul, MegaVul, MoreFixes, Vul4J, GitHub Advisory Database, OSV) are in benchmarks/pairs/datasets.toml.

Project scores and central CWE policy

Severity used to be whatever string a rule set on itself. It is now governed centrally: one CWE, one severity, decided by policy — not by the rule that fired.

Central CWE policy ✅ (policy/verycode.py): a vendored CWE_Score.tsv (verycode-policies, ~167 CWEs) is the single source of truth. Each CWE carries a flaw category, a severity (0-5), and static / dynamic scope flags. load_policy() reads it positionally (the upstream file ships CRLF and a trailing space in the Flaw Severity header), and resolve_severity() resolves a finding's severity exclusively from policy, keyed on CWE — any legacy rule-local severity is discarded. CWEs that are not static resolve to 0 and are report-only, never scored.

Fail-loud startup ✅: assert_rules_resolve(PATTERN_RULES, policy) runs as the policy_check stage right after policy_load. If any enabled rule names a CWE the policy does not govern, it raises PolicyError and the scan aborts before doing work — you cannot ship a rule whose severity nobody decided.

0-100 project score ✅ (scoring/project_score.py): each finding's penalty is severity weight × reachability multiplier, and the score is an exponential decay of the total (100 · e^(−total/k), k=60).

Severity	Weight (SEV_WEIGHT)		Reachability	Multiplier (REACH_MULT)
5	50		reachable	1.0
4	25		inferred-file-surface	0.6
3	10		unknown	0.4
2 / 1 / 0	2 / 1 / 0

No penalty → 100; one reachable severity-5 finding → ~43. The reachability multiplier is the false-positive calibration knob: a confirmed FP lowers a finding's effective reachability instead of deleting the rule, so the score moves without losing the detection.

Two-condition gate ✅: a finding that is both severity-5 and reachable always fails the gate. The score < min_score threshold (default min_score=80) only fails when blocking is enabled — so scoring is advisory-first by default and turns into a CI gate when you opt in.

Artifacts ✅: the score stage writes score.json (project score, max_severity, penalty_total, by_category, out_of_scope_dynamic_only, unmapped_cwe, gate verdict) and merges the same block into manifest.json. Scoring is zero-dependency (stdlib only). One mapping detail: verycode has no CWE-120 (generic buffer copy), so the C/C++ memory-unsafe rules listed under Detection coverage carry CWE-121 (Stack-Based Buffer Overflow, severity 5), which the policy does govern.

🧭 This is the first implemented slice of the self-improving ruleset work: a central, policy-governed severity model and a project score. Rules-as-data and the full self-improvement loop over rulesets are later phases.

Running the benchmarks

Benchmarks wrap the same scan pipeline and score it against ground-truth manifests in benchmarks/manifests/. Each manifest lists the known vulnerabilities (CWE, file, line, sink, evidence) plus optional external baselines; fixtures live in benchmarks/fixtures/ and include safe-API files so precision is measured honestly.

# One language
uv run ousast benchmark benchmarks/manifests/cpp-damn-vulnerable.toml --mode quick

# All in-scope corpora
for m in c-cpp-smoke cpp-damn-vulnerable \
         python-web-smoke python-vulnerable \
         javascript-node-web-smoke javascript-vulnerable \
         java-web-smoke java-spring-boot-vulnerable; do
  uv run ousast benchmark "benchmarks/manifests/$m.toml" --mode quick
done

Each run writes .openultrasast/benchmarks/<id>/: benchmark_result.json (recall, precision, misses, false positives, runtime), calibration_records.json (every miss as a next-improvement candidate), and external_baseline_deltas.json (tool-vs-tool comparison).

Latest results (--mode quick)

The corpora are modeled on Damn Vulnerable C/C++, PyGoat/DVPWA and NodeGoat, plus the real kiview/damn-vulnerable-spring-boot-app vendored verbatim (MIT). Three vulnerabilities are deliberately left in the ground truth that pattern matching cannot reach (integer overflow, second-order SQLi, prototype pollution) so recall is not self-fulfilling.

Language	Recall	False-positive rate
C / C++	93.3% (14/15)	0.0%
Python	93.3% (14/15)	0.0%
JavaScript	92.3% (12/13)	0.0%
Java	100% (4/4)	0.0%
Overall	93.6% (44/47)	0.0%

The project goal is ≥90% recall and <10% false positives per language. The gate lives in tests/test_detection_benchmarks.py, so a rule change that drops recall or raises false positives fails CI.

Agent integrations, fusion, and ultra workflows

The ousast CLI is the interface. An agent that can run shell commands needs nothing else:

uv run ousast scan . --mode quick --fail-on verified

For OpenCode there are optional project skills that describe the workflows above, so the agent drives the same CLI (.opencode/skills/):

Skill	Purpose
openultrasast-scan ✅	run/plan ousast scan, indexing, evidence-gated analysis
openultrasast-triage ✅	false-positive elimination, ranking calibration, verifier adjudication
openultrasast-fix-audit ✅	OpenUltraCode fix lifecycle and adversarial fix review

MCP server (narrow surface) ✅

ousast mcp runs a zero-dependency MCP server over stdio (newline-delimited JSON-RPC) exposing only stable, project-level operations — never arbitrary shell, Docker, or internal hunter tools. Point an MCP client (OpenCode, Claude Desktop, an IDE) at it:

// e.g. an MCP client config entry
{ "command": "uv", "args": ["run", "ousast", "mcp"] }

Tools: openultrasast.scan, status, findings, get_finding, evidence, artifacts, benchmark, explain, propose_patch (degrades visibly — the patch oracle is a later phase), and export_report. The scan/benchmark tools run the bounded analysis pipeline; no tool accepts a free-form command. A typical flow: scan {path} → returns a run_dir → findings {run_dir} → explain {run_dir, finding_id}.

Ultra workflows (OpenUltraCode discipline): fixes are never a one-shot patch prompt. openultrasast-fix-audit runs the bounded lifecycle intake → plan → minimal patch → adversarial review → reconcile → fresh verification → ready, and a patch can only reach patch_validated after sandboxed validation passes with no accepted blocking findings.

Fusion (deepening) ✅: when a finding needs more reasoning than the normal ranker → hunter → verifier → mapping loop provides (critical/high severity, verifier disagreement, conflicting static vs semantic evidence, risky fixes, or an explicit high-assurance request), fusion runs two independent panels that steel-man the vulnerability and false-positive cases, vote, and a decider issues the disposition (accepted / rejected / mitigated / deferred / blocked) with votes and degradations disclosed. It runs automatically on triggered findings in --mode standard, writing fusion.json and a fusion block in the manifest. Fusion is deterministic (fusion.py); independent LLM agreement on a candidate is the plane's agree task (see The agentic plane (ax)):

[fusion]
enabled        = true       # default; runs on triggered findings in standard mode
high_assurance = false      # true forces fusion on every finding

The kiro-* skills, AGENTS.md and the .kiro/ directory are the maintainer's spec-driven development tooling. They are not part of using OpenUltraSAST and can be ignored by users.

Evidence ladder: how a false positive is eliminated

Evidence level is a state machine, not a label a model may assign. The verifier runs on independent context (it never sees hunter reasoning) and enforces:

suspicion ─▶ static_corroboration ─▶ crash_reproduced ─▶ root_cause_explained
          ─▶ exploit_demonstrated ─▶ patch_validated

A finding is rejected or held back unless the artifact for the next level actually exists:

• below static_corroboration → NEEDS_EVIDENCE (a model/heuristic suspicion is never reported as verified);
• static_corroboration but reachability unknown → REJECTED with a tie-breaker demanding call-graph / route / CLI / parser / dynamic evidence (this is what kills "pattern matched but nothing attacker-controlled reaches it" findings);
• static_corroboration and function-level reachability → ACCEPTED.

When a finding is still rejected after review, openultrasast-triage records a scoped false-positive learning (calibration.py): a reason from a fixed taxonomy (unreachable_path, missing_attacker_control, sanitizer_disproved, static_rule_mismatch, incorrect_model_assumption, duplicate, insufficient_impact, unsupported, contradicted, unverified), evidence, and a scope. Scoping keeps the learning narrow: a rejected syscall_entry claim in auth/ demotes only auth/…, never the whole vulnerability class. This is covered by tests/test_calibration.py::test_scoped_false_positive_demotion_does_not_suppress_class_globally. Demotion (with an audit trail) is preferred over deletion, so an analyst can always see why something was downgraded.

How a false negative is eliminated

Misses are first-class signals, surfaced by the benchmark layer rather than hidden. Every expected vulnerability that no finding matched becomes a BenchmarkCalibrationRecord in calibration_records.json, e.g. the integer overflow the regex engine cannot see:

{
  "cwe": "CWE-190",
  "vulnerability_class": "integer overflow",
  "path": "src/buffer.cpp",
  "failed_stage": "benchmark_ground_truth_matching",
  "next_improvement_candidate": "rules_or_language_hunter",
  "reason": "no OpenUltraSAST finding matched the expected benchmark vulnerability"
}

The next_improvement_candidate routes the gap to the stage that should close it: a static rule, SARIF source, entry-point mapping, retrieval package, hunter prompt, dynamic reproducer, or skill route. Once a miss is addressed, it stays closed because the recall/precision gate runs in CI. external_baseline_deltas.json additionally shows where another tool found a vulnerability OpenUltraSAST missed (or vice-versa).

The agentic plane (ax)

The core install is zero-dependency and --mode quick never calls a model. Model work that spans a whole repository runs on the agentic plane: a Run manifest of tasks executed on ax (Kubernetes with Agent Substrate), one isolated actor per task. What runs there today: repo-facts, two independent verify passes, agree with a 2-of-3 tie-break on disputed candidates, and remember, which writes each run's per-candidate rows into the plane memory.

uv run ousast plane doctor                                   # kind, Agent Substrate, ax controller, runner image
uv run ousast plane run plane/runs/validation-46.yaml        # a rerun skips tasks already done
uv run ousast plane status <run>                             # per-task status and token attribution
uv run ousast plane remember <run>                           # ingest the run's memory rows (OUSAST_MEMORY)

Every task binds its own Model and its own budget (usd, calls), and a task that reaches its ceiling stops as unfinished instead of overspending. Egress is deny-by-default per task; the provider key travels only in the task's start request and is never written to an artifact. Bring-up, the egress policy and the host requirements are in ops/ax/README.md.

In a local ousast scan --mode standard, [models] hunter still enables the MAP-stage tool hunter; verification and fusion are deterministic.

The earlier optional agentic extra, HarnessX, was retired 2026-09-30, and a leftover [harnessx] section (retired 2026-09-30) is ignored with one warning. [models] verifier, [fusion] panel_model and [fusion] decider_model fail with a message naming the plane replacement. See RELEASE_NOTES.md.

The self-improving cycle

Each scan is a composed harness of typed processors with read/write state contracts. Strict mode fails the scan on a violation; warn mode records a degradation. The loop now runs inside the pipeline without an agent: every scan persists verifier rejections as scoped learnings, and the next scan loads them and demotes those scopes before findings are produced.

        ┌──────────── automatic ledger (no agent in the loop) ───────────┐
        ▼                                                                 │
  scan ──▶ verify ──▶ non-accepted findings ──▶ scoped learnings ─────────┤
        │             (.openultrasast/calibration/                        │  loaded by
        │              false_positive_learnings.json)                     │  the next
  next scan ──▶ rank ──▶ calibrate (demote rejected scopes) ──▶ findings ──┘  scan

What runs automatically each scan (✅, calibration.py + cli.py):

1. calibrate stage (after rank, before findings) loads the persistent ledger and demotes the priority of every scope that previously produced a rejected finding (calibrate_rankings), writing applied_calibrations.json.
2. record_calibration stage (after verify) turns this run's non-accepted verifier outcomes into scoped FalsePositiveLearning records (learnings_from_verifications) and merges them into the ledger, de-duplicated by finding ID so a repeat rejection does not compound without bound.

Demotion is scoped and reversible: a rejection in lib.py demotes only that scope; an accepted (reachable) finding in app.py is never touched; and because priority is demoted rather than the finding deleted, the audit trail (applied_calibrations.json, the ledger) shows exactly why a surface lost attention. Covered by tests/test_pipeline_calibration.py.

Rule-level loop: ousast improve

The ranking loop above adjusts attention. A second, bounded loop adjusts the governed ruleset itself from benchmark feedback — and writes its result to the same loop-owned ledger a scan reads, so the loop closes end-to-end:

# Preview what the loop would change (writes nothing)
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run

# Apply: accepted rounds land in <target>/.openultrasast/calibration/rule_policy.json,
# which the next `ousast scan`/`benchmark` of that target loads automatically.
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml

Each round (improve/evolve.py): benchmark with the current ledger → per-rule signals (rule_signals.json) → propose bounded edits → validate → replay smoke → re-benchmark → a hard acceptance gate → accept or revert byte-for-byte. The loop may only pull two levers (rule status, score constants); it can never edit a rule's pattern text or the authoritative 0–5 CWE severity, and must stage enabled → shadow before disabled. A round is accepted only if recall ≥ 90% AND FP < 10% AND the project score did not regress AND no matched finding was lost — otherwise the ledger is never written. Auto-shadowing removes a precision-dragger's false-positive cost without deleting it (recall is preserved); persistent misses are nominated through the signal file, leaving pattern authoring to a human pull request. The project score is the optimization reward; the recall/FP gate is a separate hard constraint that is never folded into it — and the same gate runs in CI (python -m openultrasast.gate), so a loop result that breached it could never merge. Covered by tests/test_improve.py and tests/test_cli_improve.py.

This is the deterministic substrate of the self-improvement loop. The plane memory is the richer proposer that plugs into it: with --memory, ousast improve also reads the rows that plane runs stored per repository and pin, and two deterministic rules turn them into rule-status proposals: a rule whose alerts are repeatedly false across repositories is shadowed, and a shadow rule that keeps hitting agreed, declared vulnerable sites is re-enabled. Rows from holdout pairs, from the gated manifest's own cases and from the qualifying population are dropped before any rule sees them, so the loop never learns from the cases it is judged on. Each proposal records the memory rows it came from (memory_proposals.jsonl next to the journal), and it goes through the same validator and gate as every other edit.

uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run --memory

The whole loop (alerts, measure, propose, improve) can also run as a plane Run (generated by ousast plane workspaces --validation-set ... --loop), with its own tasks, Models and budgets; see ops/ax/README.md.

Beyond these loops, the openultrasast-triage skill lets an agent adjust prompt constraints, retrieval filters, skill routing and benchmark-miss triage. Those RankingCalibration fields are already produced and will be consumed directly once the per-family detectors land.

Security & hardening

OpenUltraSAST treats scanned code as untrusted input and never executes it (quick and standard modes). Scan artifacts are scrubbed of credentials before they are written ([hardening] redact_secrets, on by default), agentic spend and output size are bounded (per-task usd/calls budgets on the plane, [hardening] max_findings), provider calls retry transient failures with backoff, and missing capabilities degrade visibly in the manifest rather than silently. See docs/threat-model.md for the full trust boundaries and sandbox limits, and docs/examples.md for end-to-end walkthroughs.

Development

uv run pytest                 # tests, incl. the 90/10 detection gate
uv run ruff check .           # lint
uv run ruff format --check .  # format
uv run mypy src/openultrasast # types
uv run python -m openultrasast.gate  # standalone detection gate
uv run python -m openultrasast.pair_gate  # local vuln-vs-fix efficiency + GitHub honesty
uv run python dagger/ci.py    # full containerized CI pipeline