Open_Ultra_SAST/docs/architecture.md

Architecture

How a scan works, how a finding earns its place, how severity and the project score are decided, and how the ruleset improves itself. State as of 2026-10-02; module paths are under src/openultrasast/.

The scan pipeline

The CLI is thin; HarnessRuntime owns the lifecycle and writes an event trace for every stage (trace/events.jsonl). The stages a mode requests are MODE_STAGES in stages.py: quick runs STATIC, standard adds MAP, deep adds REGRESS.

policy_load ─▶ ruleset_load ─▶ policy_check            (fail loud on an ungoverned CWE)
  ─▶ static_mapping (SARIF ingest) ─▶ preprocess ─▶ entry_point_mapping
  ─▶ rank ─▶ calibrate (demote previously rejected scopes)
  ─▶ findings   quick, deep: quick_findings (language-scoped pattern rules)
  │             standard: hunter_pool (the same rules, scheduled per ranked target)
  ─▶ verify ─▶ record_calibration ─▶ fusion (standard only, deterministic) ─▶ score
  ─▶ MAP (standard, deep)
  │     map          complexity map and hotspots
  │     overlay      semantic overlay over the findings (tree-sitter with the `semantic` extra)
  │     obligations  absence bugs: an obligated operation with no governing guard
  │     model        one Joern CPG per repository; taint, dominance and configuration arbiters
  │     tool hunter  only with `[models] hunter`
  ─▶ REGRESS (deep)  sandboxed regression snippets ─▶ verdicts.json, worth-fixing
  ─▶ report.md, report.sarif, manifest.json

The model layer (model/, cpg/) is the core of standard. It builds one code property graph per scan with Joern (joern-parse, then batched joern --script queries from cpg/queries/*.sc) and dispatches each question to the arbiter of its spec type: taint (source to sink with no discharging sanitizer), dominance (does a guard govern the obligated operation) and configuration (constant abstraction over a value). Taint facts ship for C, Java, JavaScript, PHP and Python (ruleset/semantic/*.toml). Without Joern the layer records cpg_unavailable and the scan is otherwise unchanged; the Docker image ships Joern and php-cli (which the PHP frontend needs).

The model layer reports on a rung ladder:

execution_confirmed   a test or sandbox reproduced it
model_entailed        the graph decided: a flow with no discharge, an obligation with no
                      guard, a permissive value
model_corroborated    the graph found the shape; a judge agreed on the residual question
suspicion             proposed, nothing corroborated it

The LLM (model/pipeline.py, residual_question) is asked only the residual question the graph cannot settle, for a flow the graph already found; it can confirm or contradict, never create a flow. When the graph resolved nothing, the enumerator's candidates (at most MAX_JUDGED_CANDIDATES, 8) are put to it, and anything it affirms is reported at suspicion. Without a provider key neither question is asked and only the graph's entailed findings are reported.

REGRESS (regress/, sandbox/) runs only when docker info succeeds. Each promoted candidate gets a small language recipe (Python, JavaScript, C) that loads the candidate's source inside a container with no network, a read-only /workspace, a non-root user and the [sandbox] limits. A structural deny-list rejects snippets mentioning sockets, curl, host networking or writes under /workspace before anything runs.

Evidence ladder: how a false positive is eliminated

Evidence level is a state machine, not a label a model may assign. The verifier (verification.py) runs on independent context and enforces:

suspicion ─▶ static_corroboration ─▶ crash_reproduced ─▶ root_cause_explained
          ─▶ exploit_demonstrated ─▶ patch_validated

• below static_corroboration: NEEDS_EVIDENCE;
• static_corroboration with reachability unknown: REJECTED, with a tie-breaker demanding call-graph, route, CLI, parser or dynamic evidence;
• static_corroboration with function-level reachability: ACCEPTED.

A rejected finding becomes a scoped false-positive learning (calibration.py) with a reason from a fixed taxonomy (unreachable_path, missing_attacker_control, sanitizer_disproved, static_rule_mismatch, incorrect_model_assumption, duplicate, insufficient_impact, unsupported, contradicted, unverified) and a scope. A rejection in auth/ demotes only auth/..., never the whole class (tests/test_calibration.py::test_scoped_false_positive_demotion_does_not_suppress_class_globally).

How a false negative is surfaced

Every expected vulnerability that no finding matched becomes a BenchmarkCalibrationRecord in calibration_records.json, with a next_improvement_candidate (a static rule, SARIF source, entry-point mapping, retrieval package, hunter prompt, dynamic reproducer or skill route). Once a miss is closed the detection gate keeps it closed in CI.

Central CWE policy and the project score

Policy (policy/verycode.py): a vendored CWE_Score.tsv (verycode-policies) is the single source of truth. Each CWE carries a flaw category, a severity (0-5) and static/dynamic scope flags. A finding's severity comes only from policy, keyed on CWE. CWEs that are not static resolve to 0 and are report-only. assert_rules_resolve runs as the policy_check stage: an enabled rule naming an ungoverned CWE aborts the scan. Verycode has no CWE-120, so the C/C++ buffer rules carry CWE-121.

Score (scoring/project_score.py): each finding's penalty is SEV_WEIGHT[severity] x REACH_MULT[reachability], and the score is 100 * e^(-total/k) with k = 60.

Severity	Weight		Reachability	Multiplier
5	50		reachable	1.0
4	25		inferred-file-surface	0.6
3	10		unknown	0.4
2 / 1 / 0	2 / 1 / 0

A confirmed false positive lowers a finding's reachability multiplier instead of deleting the rule. A severity-5 reachable finding always fails the score gate; score < min_score (default 80) fails it only with [score] blocking = true. The score stage writes score.json and merges it into manifest.json.

Self-improvement

Scoped calibration (every scan)

scan ─▶ verify ─▶ non-accepted findings ─▶ scoped learnings
        (.openultrasast/calibration/false_positive_learnings.json)
next scan ─▶ rank ─▶ calibrate (demote rejected scopes) ─▶ findings

record_calibration (after verify) merges this run's non-accepted outcomes into the ledger, de-duplicated by finding ID; calibrate (after rank) demotes the priority of every scope that produced a rejection and writes applied_calibrations.json. Demotion is scoped and reversible (tests/test_pipeline_calibration.py).

Rule-level loop: ousast improve

Each round (improve/evolve.py): benchmark with the current ledger, per-rule signals (rule_signals.json), bounded proposals, validation (improve/validator.py), replay smoke, re-benchmark, and a hard gate. The levers are rule status (enabled -> shadow before disabled) and score constants; the loop can never edit pattern text or the authoritative CWE severity. A round is accepted only if recall >= 90%, FP < 10%, the project score did not regress and no matched finding was lost; otherwise the ledger is not written. With --pair-catalog, every round is also gated per provenance profile on the catalog's holdout pairs (--no-pair-gate skips this and is not for accepted ledgers). The same detection gate runs in CI (python -m openultrasast.gate).

Proposals from plane memory

With --memory [STORE], ousast improve also reads the rows plane runs stored per repository and pin (improve/memory.py). Two deterministic rules turn them into rule-status proposals: a rule whose alerts are repeatedly false across repositories is shadowed, and a shadow rule that keeps hitting agreed, declared vulnerable sites is re-enabled. Rows from holdout pairs, from the gated manifest's own cases and from any --qualify-population are dropped before any rule sees them. Each proposal records the rows it came from (memory_proposals.jsonl next to the journal) and goes through the same validator and gate.

The loop as a plane Run

ousast plane workspaces <population> --validation-set ... --loop appends the improvement loop to a generated Run: alerts per case (the quick scan on the vulnerable and fixed pins), then the steps snapshot (task memory-snapshot), measure, propose and improve (plane/tasks/loop.py), each model-free with a zero budget. The snapshot applies the same train-on-test guard before any row leaves the store. A recorded live run: benchmarks/measurements/2026-09-30-harnessx-removal-loop/.

For PHP, which quick mode covered with zero rules when that run was made, ousast plane alerts-engine produces a Run's alerts from the engine image on the host instead (benchmarks/measurements/2026-09-30-php-engine-alerts/).