mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
docs: describe the system as of 2026-10-02 (ax plane, AI decision engine, PHP, v3)
Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
7b17eacd44
commit
32a2cb8bcf
@@ -1,6 +1,37 @@
|
||||
# Overview — what this is, and what is actually true of it
|
||||
|
||||
**Current direction, 2026-09-12:** an actionable pre-push security safety net for AI-accelerated
|
||||
## Current state, 2026-10-02
|
||||
|
||||
Everything below this section is dated history; where it disagrees with this section, this section wins.
|
||||
User-facing documentation: `README.md`, `docs/architecture.md`, `docs/evaluation.md`, `docs/decision-engine.md`,
|
||||
`ops/ax/README.md`.
|
||||
|
||||
- **Scanning.** `ousast scan --mode quick|standard|deep`. Quick rules for C/C++, Python, JavaScript/TypeScript,
|
||||
Java/Groovy and PHP (PHP numbers are in-sample only: `benchmarks/measurements/2026-09-30-php-quick-rules/`).
|
||||
Standard adds the MAP stage with the Joern model layer (shipped in the Docker image); deep adds sandboxed
|
||||
regression snippets when Docker is available. Framework knowledge is tagged as optional priors
|
||||
(`ruleset/frameworks.toml`, `priors=`). `ousast pre-push` compares head against base; the capability registry
|
||||
is empty, rollout NO-GO.
|
||||
- **Agentic plane.** google/ax over Agent Substrate on kind is the only executor of model work over whole
|
||||
repositories (`ousast plane run|status|doctor|workspaces|remember|alerts-engine|harvest`). Credentials travel
|
||||
only in the start request through `atenet-router`; egress is per task; budgets are per task; rows go to the
|
||||
memory store (`OUSAST_MEMORY`: FileStore, or MinioStore on the maintainer's MinIO).
|
||||
- **Removed.** HarnessX (retired 2026-09-30): its config keys fail, a `[harnessx]` section warns. The LLM judge
|
||||
of the verifier and the LLM fusion panels are gone; fusion is deterministic.
|
||||
- **Improvement.** `ousast improve --memory` proposes rule-status edits from plane memory through the unchanged
|
||||
validator and gate; the loop also runs as a plane Run.
|
||||
- **Decision engine** (`learned-decision-engine`, implementation, not adopted). First slice, injection only:
|
||||
ADVISORY recall 0.90, precision 0.54, BLOCK not offered
|
||||
(`benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`).
|
||||
- **Evaluation.** Populations v1 and v2 are spent and failed every M4 gate (`benchmarks/independent/results-v1.json`,
|
||||
`results-v2.json`); v3 (PHP) is frozen and untouched for the one-time final check. The plane increment met its
|
||||
gate on the second measurement with a tie-break chosen after the first (`benchmarks/independent/plane-increment-2.json`).
|
||||
- **Providers.** DeepSeek for LLM calls, OpenRouter only for OpenAI embeddings; keys in `.env`, which never
|
||||
overrides an exported shell variable.
|
||||
|
||||
## History
|
||||
|
||||
**Direction, 2026-09-12:** an actionable pre-push security safety net for AI-accelerated
|
||||
development across supported ecosystems. See `safety-net.md` and the approved `pre-push-safety-net`
|
||||
spec. The ranker remains the scope mechanism; third-party vendor code stays outside targets and
|
||||
graphs. Advisory must meet the same actionability bar as blocking.
|
||||
@@ -177,4 +208,4 @@ function-level pairs could not see any of them, and a repository showed all nine
|
||||
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax (`ai-service-plane`),
|
||||
> and evolutionary proposals come from the plane memory. The HarnessX specs in the table above are history.
|
||||
|
||||
97 source modules, 90 load-bearing and 7 standalone capabilities; 814 tests.
|
||||
As of 2026-09-09: 97 source modules, 90 load-bearing and 7 standalone capabilities; 814 tests.
|
||||
|
||||
@@ -1,6 +1,17 @@
|
||||
# Roadmap
|
||||
|
||||
## Current release path — 2026-09-14
|
||||
## Status, 2026-10-02
|
||||
|
||||
The milestone table below dates from 2026-09-14. Since then (`.kiro/specs/pre-push-safety-net/release-milestones.md`):
|
||||
M1a and M1b are met, M2 is measured NO-GO, and M3 is next, with M4 population preparation in parallel; none of
|
||||
M1-M5 is verified and rollout remains NO-GO. Populations v1 and v2 were evaluated and failed every M4 gate
|
||||
(`benchmarks/independent/results-v1.json`, `results-v2.json`); v3 (PHP) is frozen for the one-time final check.
|
||||
|
||||
Work since then moved to: the ax plane as the only agentic executor (`ai-service-plane`, HarnessX retired
|
||||
2026-09-30 by `harnessx-removal`), and the learned decision engine (`learned-decision-engine`, task 6 in
|
||||
progress, not adopted). See `overview.md` for the current-state summary.
|
||||
|
||||
## Release path as accepted 2026-09-14
|
||||
|
||||
Build an actionable pre-push security safety net for AI-accelerated development.
|
||||
The maintainer accepted the following order: **prove one useful detection, make it
|
||||
|
||||
@@ -1,159 +1,78 @@
|
||||
# OpenUltraSAST
|
||||
|
||||
**v1.2.0-alpha.1:** experimental pre-push safety net. Integration and census repairs are
|
||||
verified; useful hook coverage, latency and independent capability qualification remain
|
||||
**NO-GO**. No real hook capability is enabled. See [release notes](RELEASE_NOTES.md) and
|
||||
[spec status](.kiro/specs/pre-push-safety-net/status.md).
|
||||
**v1.2.0-alpha.1 (Python package `1.2.0a1`), state as of 2026-10-02.** An experimental static
|
||||
security analyser. The pre-push safety net is **NO-GO** for rollout: no hook capability is
|
||||
enabled and no independent population has passed the qualification gates (see
|
||||
[docs/evaluation.md](docs/evaluation.md)). See the [release notes](RELEASE_NOTES.md).
|
||||
|
||||
OpenUltraSAST is a command-line security analysis harness built around one
|
||||
goal: eliminating false positives by turning a suspicion into an
|
||||
evidence-backed finding, and learning from the claims that were rejected. A
|
||||
scan combines static rules, a semantic overlay, optional model-driven hunting
|
||||
and Docker-isolated proof, and every claim carries the evidence that earned its
|
||||
place.
|
||||
OpenUltraSAST separates what it can prove from what it merely suspects, and says which is which.
|
||||
A scan combines language-scoped pattern rules, a semantic overlay, a Joern-based taint engine
|
||||
and optional model calls; every finding carries the evidence that earned its place, and a scan
|
||||
that could not look at something records that as a degradation instead of reporting a clean
|
||||
result.
|
||||
|
||||
The project began as an editor-agent skill and is now a CLI first. Agent
|
||||
integrations remain, but nothing requires one: `ousast` is the interface, and
|
||||
the model-driven parts are optional extras that degrade visibly when absent.
|
||||
|
||||
> **Maturity legend:** ✅ implemented and tested · 🟡 primitive exists, not yet
|
||||
> auto-wired into the scan loop · 🧭 designed, on the roadmap. The current
|
||||
> verified gate is `usable_harness_mvp`.
|
||||
`ousast` is the interface. Model-driven work over whole repositories runs on a separate
|
||||
agentic plane (google/ax), and a learned decision engine that weighs all the signals is in
|
||||
development; neither is needed for a scan.
|
||||
|
||||
## Install and quickstart
|
||||
|
||||
```bash
|
||||
uv sync --group dev
|
||||
|
||||
# Scan a local repository (quick mode = deterministic, no model calls)
|
||||
# Deterministic scan, no model calls
|
||||
uv run ousast scan /path/to/repo --mode quick
|
||||
|
||||
# Build an embedding-ready chunk index
|
||||
uv run ousast index /path/to/repo
|
||||
|
||||
# Score the detector against a benchmark manifest
|
||||
uv run ousast benchmark benchmarks/manifests/python-vulnerable.toml --mode quick
|
||||
|
||||
# Differential pair eval: fire on vuln, stay silent on the fix (local + real GitHub VFCs)
|
||||
uv run ousast pairs
|
||||
uv run ousast pairs --slice sast # OWASP Benchmark Java/Python + Juliet
|
||||
uv run ousast pairs --slice vibe-py --pointers # nightly: also harvest non-vendored pointer pairs into ~/.cache
|
||||
uv run ousast mechanisms export --slice vibe-py # seed mechanism candidates from trusted pairs (offline); `ousast improve` admits them under the holdout gate
|
||||
uv run ousast pairs --slice vibe-py --loo # leave-one-out recall of the corpus as a teacher
|
||||
uv run python -m openultrasast.pair_gate
|
||||
|
||||
# Run the bounded self-improvement loop over the ruleset (benchmark-driven)
|
||||
uv run ousast improve benchmarks/manifests/python-vulnerable.toml --dry-run
|
||||
# CI: exit 1 on any evidence-verified finding
|
||||
uv run ousast scan . --mode quick --fail-on verified
|
||||
```
|
||||
|
||||
Every run writes auditable artifacts under `.openultrasast/runs/<scan-id>/`
|
||||
(`manifest.json`, `findings.json`, `verification.json`, `report.md`,
|
||||
`report.sarif`, `trace/events.jsonl`, ranking/preprocess/mapping JSON).
|
||||
`--fail-on findings|verified` gives CI-friendly exit codes.
|
||||
The core install depends only on PyYAML. Optional extras: `semantic` (tree-sitter grammars
|
||||
for the overlay) and `minio` (the plane's MinIO memory store).
|
||||
|
||||
## How a scan works
|
||||
For `standard` and `deep` scans, use the Docker image, which ships Joern (with `php-cli` for
|
||||
the PHP frontend) next to the tool. Source the shell wrapper once and `ousast` runs in the
|
||||
container with your directory mounted read-only:
|
||||
|
||||
The CLI is thin; the `HarnessRuntime` owns the lifecycle and emits an event
|
||||
trace for every stage (`task_start … stage_start/stage_end … task_end`).
|
||||
|
||||
```
|
||||
repo ──▶ preprocess (language, LOC, tags, fuzz entry points)
|
||||
──▶ static mapping (Semgrep/CodeQL SARIF ingest, normalized hints)
|
||||
──▶ entry-point + reachability mapping (routes, CLI, parsers, contracts)
|
||||
──▶ rank (surface·0.5 + influence·0.2 + reachability·0.3)
|
||||
──▶ findings ─ quick: language-scoped pattern rules
|
||||
│ └ standard: tiered hunter pool (retrieval + skill snippets)
|
||||
──▶ independent verifier (evidence ladder enforced)
|
||||
──▶ report.md + report.sarif + manifest.json (shared finding IDs)
|
||||
```bash
|
||||
source /path/to/OpenUltraSAST/ops/shell/ousast.sh # PowerShell: ops/shell/ousast.ps1
|
||||
ousast scan . --mode standard
|
||||
```
|
||||
|
||||
- **quick** ✅: deterministic, language-scoped pattern rules + reachability +
|
||||
verification. No model calls, fully reproducible.
|
||||
- **standard** ✅ scheduling / 🧭 LLM hunters: runs the tiered hunter pool
|
||||
(A/B/C budgets, retrieval context, selected Trail of Bits skill snippets,
|
||||
JSONL trajectories) and verification; language-aware LLM hunters are the next
|
||||
upgrade (today standard reuses the deterministic findings per target).
|
||||
- **deep** 🧭: sandboxed build/fuzz/dynamic reproduction (not yet implemented).
|
||||
Details, including a pinned host install of Joern, are in [ops/README.md](ops/README.md).
|
||||
|
||||
## Detection coverage
|
||||
Every run writes auditable artifacts under `<target>/.openultrasast/runs/<scan-id>/`:
|
||||
`manifest.json`, `findings.json`, `verification.json`, `score.json`, `report.md`,
|
||||
`report.sarif` and `trace/events.jsonl`, plus the stage artifacts of the mode.
|
||||
|
||||
Language-scoped rule families (rules only run against matching languages, which
|
||||
removes cross-language false positives):
|
||||
## Scan modes
|
||||
|
||||
| Mode | What runs | Needs |
|
||||
| --- | --- | --- |
|
||||
| `quick` | Language-scoped pattern rules, entry-point reachability hints, ranking, verification, scoring. Deterministic and reproducible. | nothing |
|
||||
| `standard` | `quick` plus the MAP stage: complexity map, semantic overlay, authorization obligations, and the model layer, which builds one Joern code property graph per repository and decides taint, guard-dominance and configuration questions on it. An LLM is asked only the residual question the graph cannot settle (a `suspicion`); `[models] hunter` adds the tool hunter. | Joern for the model layer (else a `cpg_unavailable` degradation); a provider key for the LLM parts (else they are skipped and recorded) |
|
||||
| `deep` | The MAP stage of `standard` plus REGRESS: promoted candidates are loaded by a small snippet inside a Docker sandbox (no network, read-only source, non-root, memory/pid/time limits) and get a `triggerable` / `not_triggerable` / ... verdict. | a working `docker` (else a `sandbox_unavailable` degradation) |
|
||||
|
||||
`--fail-on never|findings|verified|worth-fixing` sets the exit code: `findings` fails on any
|
||||
reported finding, `verified` on any evidence-verified one, `worth-fixing` on a deep-mode
|
||||
verdict that is worth fixing. `--config openultrasast.toml` loads settings.
|
||||
|
||||
## Detection coverage (quick rules)
|
||||
|
||||
Rules run only against files of their language. The bundled ruleset lives in
|
||||
`src/openultrasast/ruleset/<language>/rules.toml`:
|
||||
|
||||
| Language | Classes (CWE) |
|
||||
| --- | --- |
|
||||
| C / C++ | buffer overflow (120), format string (134), command injection (78) |
|
||||
| Python | command injection (78), code injection (95), deserialization (502), SQLi (89), path traversal (22), SSRF (918), weak hash (327), SSTI (94), insecure default (489) |
|
||||
| JavaScript / TS | command injection (78), code injection (95), reflected/DOM XSS (79), SQLi (89), path traversal (22), SSRF (918), weak hash (327), deserialization (502) |
|
||||
| Java + Groovy templates | command injection (78), SQLi via concatenation (89), weak hash (327), deserialization (502), unescaped template XSS (79) |
|
||||
| C / C++ | stack buffer overflow (121), format string (134), command injection (78) |
|
||||
| Python | command injection (78), code injection (95), SSTI (94), deserialization (502), SQLi (89), path traversal (22), SSRF (918), weak hash (327), insecure default (489) |
|
||||
| JavaScript / TS | command injection (78), code injection (95), XSS (79), SQLi (89), path traversal (22), SSRF (918), weak hash (327), deserialization (502) |
|
||||
| Java + Groovy templates | command injection (78), SQLi (89), weak hash (327), deserialization (502), unescaped template XSS (79) |
|
||||
| PHP | SQLi (89), command injection (78), file inclusion (98), path (22), open redirect (601), SSRF (918); code eval (95), unserialize (502) and echo XSS (79) ship as `shadow` rules |
|
||||
|
||||
Cheat-sheet fixtures live under `benchmarks/fixtures/` and feed the 90/10 smoke
|
||||
gate. **Efficiency** is measured separately on isolated vuln-vs-fixed pairs
|
||||
(`ousast pairs`, `python -m openultrasast.pair_gate`): the harness must fire on
|
||||
the vulnerable snapshot and stay silent after the patch. Local fixture
|
||||
counterparts are a CI gate; vendored GitHub VFCs (SVEN, CWE-Bench-Java, NVD
|
||||
patterns) are the honesty dashboard and the miss/fp signals for `ousast improve`.
|
||||
Dataset pointers (CVEfixes, DiverseVul, PrimeVul, MegaVul, MoreFixes, Vul4J,
|
||||
GitHub Advisory Database, OSV) are in `benchmarks/pairs/datasets.toml`.
|
||||
|
||||
## Project scores and central CWE policy
|
||||
|
||||
Severity used to be whatever string a rule set on itself. It is now governed centrally: one CWE, one severity, decided by policy — not by the rule that fired.
|
||||
|
||||
**Central CWE policy** ✅ (`policy/verycode.py`): a vendored `CWE_Score.tsv` (verycode-policies, ~167 CWEs) is the single source of truth. Each CWE carries a flaw category, a `severity` (0-5), and `static` / `dynamic` scope flags. `load_policy()` reads it positionally (the upstream file ships CRLF and a trailing space in the `Flaw Severity ` header), and `resolve_severity()` resolves a finding's severity *exclusively* from policy, keyed on CWE — any legacy rule-local severity is discarded. CWEs that are not `static` resolve to `0` and are report-only, never scored.
|
||||
|
||||
**Fail-loud startup** ✅: `assert_rules_resolve(PATTERN_RULES, policy)` runs as the `policy_check` stage right after `policy_load`. If any *enabled* rule names a CWE the policy does not govern, it raises `PolicyError` and the scan aborts before doing work — you cannot ship a rule whose severity nobody decided.
|
||||
|
||||
**0-100 project score** ✅ (`scoring/project_score.py`): each finding's penalty is `severity weight × reachability multiplier`, and the score is an exponential decay of the total (`100 · e^(−total/k)`, `k=60`).
|
||||
|
||||
| Severity | Weight (`SEV_WEIGHT`) | | Reachability | Multiplier (`REACH_MULT`) |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| 5 | 50 | | `reachable` | 1.0 |
|
||||
| 4 | 25 | | `inferred-file-surface` | 0.6 |
|
||||
| 3 | 10 | | `unknown` | 0.4 |
|
||||
| 2 / 1 / 0 | 2 / 1 / 0 | | | |
|
||||
|
||||
No penalty → 100; one reachable severity-5 finding → ~43. The **reachability multiplier is the false-positive calibration knob**: a confirmed FP lowers a finding's effective reachability instead of deleting the rule, so the score moves without losing the detection.
|
||||
|
||||
**Two-condition gate** ✅: a finding that is both severity-5 *and* `reachable` **always** fails the gate. The `score < min_score` threshold (default `min_score=80`) only fails when `blocking` is enabled — so scoring is **advisory-first by default** and turns into a CI gate when you opt in.
|
||||
|
||||
**Artifacts** ✅: the `score` stage writes `score.json` (project score, `max_severity`, `penalty_total`, `by_category`, `out_of_scope_dynamic_only`, `unmapped_cwe`, gate verdict) and merges the same block into `manifest.json`. Scoring is zero-dependency (stdlib only). One mapping detail: verycode has no CWE-120 (generic buffer copy), so the C/C++ memory-unsafe rules listed under [Detection coverage](#detection-coverage) carry **CWE-121** (Stack-Based Buffer Overflow, severity 5), which the policy does govern.
|
||||
|
||||
🧭 This is the first implemented slice of the self-improving ruleset work: a central, policy-governed severity model and a project score. Rules-as-data and the full self-improvement loop over rulesets are later phases.
|
||||
|
||||
## Running the benchmarks
|
||||
|
||||
Benchmarks wrap the same scan pipeline and score it against ground-truth
|
||||
manifests in `benchmarks/manifests/`. Each manifest lists the *known*
|
||||
vulnerabilities (CWE, file, line, sink, evidence) plus optional external
|
||||
baselines; fixtures live in `benchmarks/fixtures/` and include safe-API files so
|
||||
precision is measured honestly.
|
||||
|
||||
```bash
|
||||
# One language
|
||||
uv run ousast benchmark benchmarks/manifests/cpp-damn-vulnerable.toml --mode quick
|
||||
|
||||
# All in-scope corpora
|
||||
for m in c-cpp-smoke cpp-damn-vulnerable \
|
||||
python-web-smoke python-vulnerable \
|
||||
javascript-node-web-smoke javascript-vulnerable \
|
||||
java-web-smoke java-spring-boot-vulnerable; do
|
||||
uv run ousast benchmark "benchmarks/manifests/$m.toml" --mode quick
|
||||
done
|
||||
```
|
||||
|
||||
Each run writes `.openultrasast/benchmarks/<id>/`:
|
||||
`benchmark_result.json` (recall, precision, misses, false positives, runtime),
|
||||
`calibration_records.json` (every miss as a next-improvement candidate), and
|
||||
`external_baseline_deltas.json` (tool-vs-tool comparison).
|
||||
|
||||
### Latest results (`--mode quick`)
|
||||
|
||||
The corpora are modeled on Damn Vulnerable C/C++, PyGoat/DVPWA and NodeGoat,
|
||||
plus the real [`kiview/damn-vulnerable-spring-boot-app`](https://github.com/kiview/damn-vulnerable-spring-boot-app)
|
||||
vendored verbatim (MIT). Three vulnerabilities are deliberately left in the
|
||||
ground truth that pattern matching *cannot* reach (integer overflow, second-order
|
||||
SQLi, prototype pollution) so recall is not self-fulfilling.
|
||||
The detection gate (`uv run python -m openultrasast.gate`, also `tests/test_detection_benchmarks.py`)
|
||||
enforces the project goal of **at least 90% recall and under 10% false positives** per language
|
||||
on the bundled corpora in `benchmarks/manifests/`. Its output on this tree:
|
||||
|
||||
| Language | Recall | False-positive rate |
|
||||
| --- | --- | --- |
|
||||
@@ -163,259 +82,152 @@ SQLi, prototype pollution) so recall is not self-fulfilling.
|
||||
| Java | 100% (4/4) | 0.0% |
|
||||
| **Overall** | **93.6% (44/47)** | **0.0%** |
|
||||
|
||||
The project goal is **≥90% recall and <10% false positives** per language. The
|
||||
gate lives in `tests/test_detection_benchmarks.py`, so a rule change that drops
|
||||
recall or raises false positives fails CI.
|
||||
These corpora are cheat-sheet fixtures modelled on deliberately vulnerable applications. They
|
||||
show that a rule change did not break a known detection; they are not a recall estimate on
|
||||
real code. Real-code results are in [docs/evaluation.md](docs/evaluation.md).
|
||||
|
||||
## Agent integrations, fusion, and ultra workflows
|
||||
**PHP is not in the gate, and its numbers are in-sample only.** The PHP rules were written and
|
||||
tightened while looking at the development corpus they were measured on
|
||||
(`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`): 13 of 14 expected
|
||||
findings on the `php-vulnerable` fixture with no false positive, 2 false positives on
|
||||
`php-benign`, and per-rule precision lower bounds from 0.03 to 1.0 on the real pins. Treat
|
||||
them as development numbers, not as a holdout result.
|
||||
|
||||
The `ousast` CLI is the interface. An agent that can run shell commands needs
|
||||
nothing else:
|
||||
**Framework knowledge is optional.** Rule and taint-fact entries that know a framework or
|
||||
library (WordPress, Flask, Django, Express, Spring, ...) carry a `framework` or `library` tag
|
||||
naming a row of `src/openultrasast/ruleset/frameworks.toml`. The loaders take `priors=`
|
||||
(`"all"`, the default and today's behaviour; `"off"`, language-level knowledge only; or a set
|
||||
of ids), so their contribution can be measured and switched off.
|
||||
|
||||
## Pre-push checks (experimental)
|
||||
|
||||
`ousast pre-push` analyses the commits a `git push` would publish, or an explicit
|
||||
`--base`/`--head` pair, and compares the head against its base so that only new or worsened
|
||||
defects are candidates. `--mode advisory` (the default) never blocks; `--mode blocking` and
|
||||
`--incomplete-coverage block` opt into enforcement. The artifact must be written outside the
|
||||
analysed repository.
|
||||
|
||||
```bash
|
||||
uv run ousast scan . --mode quick --fail-on verified
|
||||
uv run ousast pre-push /path/to/repo --base BASE --head HEAD \
|
||||
--artifact /outside/the/repo/result.json
|
||||
```
|
||||
|
||||
For [OpenCode](https://opencode.ai) there are optional project skills that
|
||||
describe the workflows above, so the agent drives the same CLI
|
||||
(`.opencode/skills/`):
|
||||
The default capability registry is empty, so no normal alert is emitted today; every
|
||||
candidate stays diagnostic in the artifact. Installation into a repository's hooks
|
||||
(`ops/install-pre-push`), caching and the optional witness model are documented in
|
||||
[ops/README.md](ops/README.md).
|
||||
|
||||
| Skill | Purpose |
|
||||
| --- | --- |
|
||||
| `openultrasast-scan` ✅ | run/plan `ousast scan`, indexing, evidence-gated analysis |
|
||||
| `openultrasast-triage` ✅ | false-positive elimination, ranking calibration, verifier adjudication |
|
||||
| `openultrasast-fix-audit` ✅ | OpenUltraCode fix lifecycle and adversarial fix review |
|
||||
## Model providers and keys
|
||||
|
||||
### MCP server (narrow surface) ✅
|
||||
|
||||
`ousast mcp` runs a zero-dependency MCP server over stdio (newline-delimited
|
||||
JSON-RPC) exposing only stable, project-level operations — **never** arbitrary
|
||||
shell, Docker, or internal hunter tools. Point an MCP client (OpenCode, Claude
|
||||
Desktop, an IDE) at it:
|
||||
|
||||
```jsonc
|
||||
// e.g. an MCP client config entry
|
||||
{ "command": "uv", "args": ["run", "ousast", "mcp"] }
|
||||
```
|
||||
|
||||
Tools: `openultrasast.scan`, `status`, `findings`, `get_finding`, `evidence`,
|
||||
`artifacts`, `benchmark`, `explain`, `propose_patch` (degrades visibly — the patch
|
||||
oracle is a later phase), and `export_report`. The `scan`/`benchmark` tools run the
|
||||
bounded analysis pipeline; no tool accepts a free-form command. A typical flow:
|
||||
`scan {path}` → returns a `run_dir` → `findings {run_dir}` → `explain {run_dir, finding_id}`.
|
||||
|
||||
**Ultra workflows (OpenUltraCode discipline):** fixes are never a one-shot patch
|
||||
prompt. `openultrasast-fix-audit` runs the bounded lifecycle
|
||||
`intake → plan → minimal patch → adversarial review → reconcile → fresh
|
||||
verification → ready`, and a patch can only reach `patch_validated` after
|
||||
sandboxed validation passes with no accepted blocking findings.
|
||||
|
||||
**Fusion (deepening) ✅:** when a finding needs more reasoning than the normal
|
||||
ranker → hunter → verifier → mapping loop provides (critical/high severity,
|
||||
verifier disagreement, conflicting static vs semantic evidence, risky fixes, or
|
||||
an explicit high-assurance request), fusion runs two independent panels that
|
||||
steel-man the vulnerability and false-positive cases, vote, and a decider issues
|
||||
the disposition (accepted / rejected / mitigated / deferred / blocked) with votes
|
||||
and degradations disclosed. It runs automatically on triggered findings
|
||||
in `--mode standard`, writing `fusion.json` and a `fusion` block in the manifest.
|
||||
Fusion is deterministic (`fusion.py`); independent LLM agreement on a candidate
|
||||
is the plane's `agree` task (see [The agentic plane (ax)](#the-agentic-plane-ax)):
|
||||
|
||||
```toml
|
||||
[fusion]
|
||||
enabled = true # default; runs on triggered findings in standard mode
|
||||
high_assurance = false # true forces fusion on every finding
|
||||
```
|
||||
|
||||
> The `kiro-*` skills, `AGENTS.md` and the `.kiro/` directory are the
|
||||
> maintainer's spec-driven development tooling. They are not part of using
|
||||
> OpenUltraSAST and can be ignored by users.
|
||||
|
||||
## Evidence ladder: how a false positive is eliminated
|
||||
|
||||
Evidence level is a state machine, not a label a model may assign. The verifier
|
||||
runs on independent context (it never sees hunter reasoning) and enforces:
|
||||
|
||||
```
|
||||
suspicion ─▶ static_corroboration ─▶ crash_reproduced ─▶ root_cause_explained
|
||||
─▶ exploit_demonstrated ─▶ patch_validated
|
||||
```
|
||||
|
||||
A finding is rejected or held back unless the *artifact* for the next level
|
||||
actually exists:
|
||||
|
||||
- below `static_corroboration` → `NEEDS_EVIDENCE` (a model/heuristic suspicion is
|
||||
never reported as verified);
|
||||
- `static_corroboration` but **reachability unknown** → `REJECTED` with a
|
||||
tie-breaker demanding call-graph / route / CLI / parser / dynamic evidence
|
||||
(this is what kills "pattern matched but nothing attacker-controlled reaches
|
||||
it" findings);
|
||||
- `static_corroboration` **and** function-level reachability → `ACCEPTED`.
|
||||
|
||||
When a finding is still rejected after review, `openultrasast-triage` records a
|
||||
**scoped false-positive learning** (`calibration.py`): a reason from a fixed
|
||||
taxonomy (`unreachable_path`, `missing_attacker_control`, `sanitizer_disproved`,
|
||||
`static_rule_mismatch`, `incorrect_model_assumption`, `duplicate`,
|
||||
`insufficient_impact`, `unsupported`, `contradicted`, `unverified`), evidence,
|
||||
and a **scope**. Scoping keeps the learning narrow: a rejected `syscall_entry`
|
||||
claim in `auth/` demotes only `auth/…`, never the whole vulnerability class. This
|
||||
is covered by `tests/test_calibration.py::test_scoped_false_positive_demotion_does_not_suppress_class_globally`.
|
||||
Demotion (with an audit trail) is preferred over deletion, so an analyst can
|
||||
always see *why* something was downgraded.
|
||||
|
||||
## How a false negative is eliminated
|
||||
|
||||
Misses are first-class signals, surfaced by the benchmark layer rather than
|
||||
hidden. Every expected vulnerability that no finding matched becomes a
|
||||
`BenchmarkCalibrationRecord` in `calibration_records.json`, e.g. the integer
|
||||
overflow the regex engine cannot see:
|
||||
|
||||
```json
|
||||
{
|
||||
"cwe": "CWE-190",
|
||||
"vulnerability_class": "integer overflow",
|
||||
"path": "src/buffer.cpp",
|
||||
"failed_stage": "benchmark_ground_truth_matching",
|
||||
"next_improvement_candidate": "rules_or_language_hunter",
|
||||
"reason": "no OpenUltraSAST finding matched the expected benchmark vulnerability"
|
||||
}
|
||||
```
|
||||
|
||||
The `next_improvement_candidate` routes the gap to the stage that should close
|
||||
it: a static rule, SARIF source, entry-point mapping, retrieval package, hunter
|
||||
prompt, dynamic reproducer, or skill route. Once a miss is
|
||||
addressed, it stays closed because the recall/precision gate runs in CI.
|
||||
`external_baseline_deltas.json` additionally shows where another tool found a
|
||||
vulnerability OpenUltraSAST missed (or vice-versa).
|
||||
|
||||
## The agentic plane (ax)
|
||||
|
||||
The core install is zero-dependency and `--mode quick` never calls a model. Model
|
||||
work that spans a whole repository runs on the **agentic plane**: a Run manifest
|
||||
of tasks executed on [ax](ops/ax/README.md) (Kubernetes with Agent Substrate), one
|
||||
isolated actor per task. What runs there today: `repo-facts`, two independent
|
||||
`verify` passes, `agree` with a 2-of-3 tie-break on disputed candidates, and
|
||||
`remember`, which writes each run's per-candidate rows into the plane memory.
|
||||
Keys live in `.env` in the working directory (gitignored, never committed):
|
||||
|
||||
```bash
|
||||
uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image
|
||||
uv run ousast plane run plane/runs/validation-46.yaml # a rerun skips tasks already done
|
||||
uv run ousast plane status <run> # per-task status and token attribution
|
||||
uv run ousast plane remember <run> # ingest the run's memory rows (OUSAST_MEMORY)
|
||||
DEEPSEEK_API_KEY=... # every LLM call: model layer, tool hunter, plane tasks
|
||||
OPENROUTER_API_KEY=... # only OpenAI embeddings (openai/text-embedding-3-small)
|
||||
```
|
||||
|
||||
Every task binds its own Model and its own budget (`usd`, `calls`), and a task
|
||||
that reaches its ceiling stops as `unfinished` instead of overspending. Egress is
|
||||
deny-by-default per task; the provider key travels only in the task's start
|
||||
request and is never written to an artifact. Bring-up, the egress policy and the
|
||||
host requirements are in [ops/ax/README.md](ops/ax/README.md).
|
||||
The project uses DeepSeek (`deepseek-flash`) for LLM calls and OpenRouter only for embeddings.
|
||||
Set `DEEPSEEK_API_KEY`: without it, chat calls fall back to OpenRouter when that key is set.
|
||||
|
||||
In a local `ousast scan --mode standard`, `[models] hunter` still enables the
|
||||
MAP-stage tool hunter; verification and fusion are deterministic.
|
||||
**`.env` never overrides a variable already exported in your shell.** A stale
|
||||
`export DEEPSEEK_API_KEY=...` in your profile wins over the `.env` file silently; `unset` it
|
||||
when a key change in `.env` seems to have no effect.
|
||||
|
||||
> The earlier optional agentic extra, HarnessX, was retired 2026-09-30, and a
|
||||
> leftover `[harnessx]` section (retired 2026-09-30) is ignored with one warning.
|
||||
> `[models] verifier`, `[fusion] panel_model` and `[fusion] decider_model` fail with
|
||||
> a message naming the plane replacement. See [RELEASE_NOTES.md](RELEASE_NOTES.md).
|
||||
## Benchmarks and pair evaluation
|
||||
|
||||
## The self-improving cycle
|
||||
```bash
|
||||
# Score the scan against a ground-truth manifest
|
||||
uv run ousast benchmark benchmarks/manifests/python-vulnerable.toml --mode quick
|
||||
|
||||
Each scan is a composed harness of typed processors with read/write **state
|
||||
contracts**. Strict mode fails the scan on a violation; warn mode records a
|
||||
degradation. The loop now runs inside the pipeline without an agent: every scan
|
||||
persists verifier rejections as scoped learnings, and the next scan loads them
|
||||
and demotes those scopes before findings are produced.
|
||||
|
||||
```
|
||||
┌──────────── automatic ledger (no agent in the loop) ───────────┐
|
||||
▼ │
|
||||
scan ──▶ verify ──▶ non-accepted findings ──▶ scoped learnings ─────────┤
|
||||
│ (.openultrasast/calibration/ │ loaded by
|
||||
│ false_positive_learnings.json) │ the next
|
||||
next scan ──▶ rank ──▶ calibrate (demote rejected scopes) ──▶ findings ──┘ scan
|
||||
# Vulnerable-vs-fixed pairs: fire on the vulnerable side, stay silent on the fix
|
||||
uv run ousast pairs
|
||||
uv run ousast pairs --slice sast # OWASP Benchmark + Juliet
|
||||
uv run ousast pairs --slice vibe-py --pointers # also fetch non-vendored pointer pairs (network)
|
||||
uv run python -m openultrasast.pair_gate # CI: local pairs must all pass
|
||||
```
|
||||
|
||||
What runs automatically each scan (✅, `calibration.py` + `cli.py`):
|
||||
|
||||
1. **`calibrate` stage** (after `rank`, before findings) loads the persistent
|
||||
ledger and demotes the priority of every scope that previously produced a
|
||||
rejected finding (`calibrate_rankings`), writing `applied_calibrations.json`.
|
||||
2. **`record_calibration` stage** (after `verify`) turns this run's non-accepted
|
||||
verifier outcomes into scoped `FalsePositiveLearning` records
|
||||
(`learnings_from_verifications`) and merges them into the ledger, de-duplicated
|
||||
by finding ID so a repeat rejection does not compound without bound.
|
||||
|
||||
Demotion is **scoped and reversible**: a rejection in `lib.py` demotes only that
|
||||
scope; an accepted (reachable) finding in `app.py` is never touched; and because
|
||||
priority is demoted rather than the finding deleted, the audit trail
|
||||
(`applied_calibrations.json`, the ledger) shows exactly why a surface lost
|
||||
attention. Covered by `tests/test_pipeline_calibration.py`.
|
||||
Each benchmark run writes `.openultrasast/benchmarks/<id>/` (`benchmark_result.json`,
|
||||
`calibration_records.json` with every miss, `external_baseline_deltas.json`). The pair
|
||||
catalogs live in `benchmarks/pairs/`.
|
||||
|
||||
### Rule-level loop: `ousast improve`
|
||||
## Self-improvement
|
||||
|
||||
The ranking loop above adjusts *attention*. A second, bounded loop adjusts the
|
||||
**governed ruleset itself** from benchmark feedback — and writes its result to the
|
||||
same loop-owned ledger a scan reads, so the loop closes end-to-end:
|
||||
`ousast improve` runs a bounded loop over the governed ruleset: benchmark, propose edits to
|
||||
rule status and score constants only, validate, re-benchmark, and accept a round only if
|
||||
recall stays at least 90%, false positives under 10%, the project score does not regress and
|
||||
no matched finding is lost. Accepted rounds land in
|
||||
`<target>/.openultrasast/calibration/rule_policy.json`, which the next scan of that target loads.
|
||||
|
||||
```bash
|
||||
# Preview what the loop would change (writes nothing)
|
||||
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run
|
||||
|
||||
# Apply: accepted rounds land in <target>/.openultrasast/calibration/rule_policy.json,
|
||||
# which the next `ousast scan`/`benchmark` of that target loads automatically.
|
||||
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml
|
||||
```
|
||||
|
||||
Each round (`improve/evolve.py`): benchmark with the current ledger → per-rule
|
||||
signals (`rule_signals.json`) → propose **bounded** edits → validate → replay
|
||||
smoke → re-benchmark → a **hard acceptance gate** → accept or revert
|
||||
byte-for-byte. The loop may only pull two levers (rule status, score constants);
|
||||
it can never edit a rule's pattern text or the authoritative 0–5 CWE severity, and
|
||||
must stage `enabled → shadow` before `disabled`. A round is accepted only if
|
||||
**recall ≥ 90% AND FP < 10% AND the project score did not regress AND no matched
|
||||
finding was lost** — otherwise the ledger is never written. Auto-shadowing removes
|
||||
a precision-dragger's false-positive cost without deleting it (recall is
|
||||
preserved); persistent misses are *nominated* through the signal file, leaving
|
||||
pattern authoring to a human pull request. The project score is the optimization
|
||||
reward; the recall/FP gate is a separate hard constraint that is never folded into
|
||||
it — and the same gate runs in CI (`python -m openultrasast.gate`), so a loop
|
||||
result that breached it could never merge. Covered by `tests/test_improve.py` and
|
||||
`tests/test_cli_improve.py`.
|
||||
|
||||
This is the deterministic substrate of the self-improvement loop. The **plane
|
||||
memory** is the richer proposer that plugs into it: with `--memory`, `ousast
|
||||
improve` also reads the rows that plane runs stored per repository and pin, and two
|
||||
deterministic rules turn them into rule-status proposals: a rule whose alerts are
|
||||
repeatedly false across repositories is shadowed, and a shadow rule that keeps
|
||||
hitting agreed, declared vulnerable sites is re-enabled. Rows from holdout pairs,
|
||||
from the gated manifest's own cases and from the qualifying population are dropped
|
||||
before any rule sees them, so the loop never learns from the cases it is judged on. Each
|
||||
proposal records the memory rows it came from (`memory_proposals.jsonl` next to the
|
||||
journal), and it goes through the same validator and gate as every other edit.
|
||||
|
||||
```bash
|
||||
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run --memory
|
||||
```
|
||||
|
||||
The whole loop (alerts, measure, propose, improve) can also run as a plane Run
|
||||
(generated by `ousast plane workspaces --validation-set ... --loop`), with its own tasks, Models
|
||||
and budgets; see [ops/ax/README.md](ops/ax/README.md).
|
||||
With `--memory`, proposals also come from the rows plane runs stored (see below), through the
|
||||
same validator and gate; rows from holdout pairs, from the gated manifest's own cases and from
|
||||
any `--qualify-population` are dropped first. Details:
|
||||
[docs/architecture.md](docs/architecture.md#self-improvement).
|
||||
|
||||
Beyond these loops, the `openultrasast-triage` skill lets an agent adjust
|
||||
prompt constraints, retrieval filters, skill routing and benchmark-miss triage.
|
||||
Those `RankingCalibration` fields are already produced and will be consumed
|
||||
directly once the per-family detectors land.
|
||||
## The agentic plane (maintainers)
|
||||
|
||||
## Security & hardening
|
||||
Model work over whole repositories runs as a Run manifest on [google/ax](ops/ax/README.md)
|
||||
over Agent Substrate (a kind cluster on the maintainer's host), one isolated actor per task.
|
||||
Each task binds its own Model and its own `usd`/`calls` budget, egress is deny-by-default per
|
||||
task, and the provider key travels only in the task's start request.
|
||||
|
||||
OpenUltraSAST treats scanned code as **untrusted input and never executes it** (quick
|
||||
and standard modes). Scan artifacts are scrubbed of credentials before they are written
|
||||
(`[hardening] redact_secrets`, on by default), agentic spend and output size are bounded
|
||||
(per-task `usd`/`calls` budgets on the plane, `[hardening] max_findings`), provider calls
|
||||
retry transient failures with backoff, and missing capabilities degrade visibly in the
|
||||
manifest rather than silently. See [docs/threat-model.md](docs/threat-model.md) for the
|
||||
full trust boundaries and sandbox limits, and [docs/examples.md](docs/examples.md) for
|
||||
end-to-end walkthroughs.
|
||||
```bash
|
||||
uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image
|
||||
uv run ousast plane run plane/runs/validation-46.yaml # a rerun skips tasks already done
|
||||
uv run ousast plane status validation-46 # per-task status and token attribution
|
||||
uv run ousast plane remember validation-46 # ingest the run's rows into the memory store
|
||||
```
|
||||
|
||||
`ousast plane workspaces`, `alerts-engine` and `harvest` generate Runs and inputs. Bring-up,
|
||||
the task catalogue, the memory store (`OUSAST_MEMORY`) and the checklist for moving the plane
|
||||
to another Kubernetes cluster are in [ops/ax/README.md](ops/ax/README.md).
|
||||
|
||||
> HarnessX, the earlier optional agentic extra, was retired 2026-09-30, and a leftover
|
||||
> `[harnessx]` section (retired 2026-09-30) is ignored with one warning. `[models] verifier`,
|
||||
> `[fusion] panel_model` and `[fusion] decider_model` fail with a message naming the plane
|
||||
> replacement. The LLM judge and the LLM fusion panels were retired with it; fusion is
|
||||
> deterministic. See [RELEASE_NOTES.md](RELEASE_NOTES.md).
|
||||
|
||||
## Decision engine (in development, not adopted)
|
||||
|
||||
A learned decision engine is being built to replace hand-tuned detection edits: each
|
||||
instrument (quick rules, the Joern engine, model sink classification, verify passes, repository
|
||||
facts) produces signals, never verdicts, and a compiled AI classifier with local memory turns
|
||||
them into a calibrated probability per candidate with BLOCK and ADVISORY operating points.
|
||||
No scan, pre-push or report uses it yet. Status and the first measured slice:
|
||||
[docs/decision-engine.md](docs/decision-engine.md).
|
||||
|
||||
## Agent integrations
|
||||
|
||||
An agent that can run shell commands needs nothing but the CLI. For
|
||||
[OpenCode](https://opencode.ai) there are optional project skills in `.opencode/skills/`:
|
||||
`openultrasast-scan`, `openultrasast-triage` and `openultrasast-fix-audit`.
|
||||
|
||||
`ousast mcp` runs a narrow MCP server over stdio (newline-delimited JSON-RPC) with ten tools:
|
||||
`openultrasast.scan`, `status`, `findings`, `get_finding`, `evidence`, `artifacts`,
|
||||
`benchmark`, `explain`, `propose_patch` (degrades visibly) and `export_report`. No tool runs a
|
||||
shell, Docker or a free-form command.
|
||||
|
||||
```jsonc
|
||||
{ "command": "uv", "args": ["run", "ousast", "mcp"] }
|
||||
```
|
||||
|
||||
> The `kiro-*` skills, `AGENTS.md` and `.kiro/` are the maintainer's development tooling and
|
||||
> can be ignored by users.
|
||||
|
||||
## Further reading
|
||||
|
||||
- [docs/architecture.md](docs/architecture.md): the scan pipeline, evidence ladder, CWE policy
|
||||
and project score, and the self-improving loops.
|
||||
- [docs/evaluation.md](docs/evaluation.md): independent populations, the qualification gates,
|
||||
pair corpora and the plane increment results.
|
||||
- [docs/decision-engine.md](docs/decision-engine.md): the learned decision engine.
|
||||
- [docs/threat-model.md](docs/threat-model.md): trust boundaries, sandbox and hardening.
|
||||
- [docs/examples.md](docs/examples.md): end-to-end walkthroughs.
|
||||
|
||||
## Development
|
||||
|
||||
@@ -424,7 +236,11 @@ uv run pytest # tests, incl. the 90/10 detection gate
|
||||
uv run ruff check . # lint
|
||||
uv run ruff format --check . # format
|
||||
uv run mypy src/openultrasast # types
|
||||
uv run python -m openultrasast.gate # standalone detection gate
|
||||
uv run python -m openultrasast.pair_gate # local vuln-vs-fix efficiency + GitHub honesty
|
||||
uv run python dagger/ci.py # full containerized CI pipeline
|
||||
uv run python -m openultrasast.gate # standalone detection gate
|
||||
uv run python -m openultrasast.pair_gate # local vuln-vs-fix pairs
|
||||
uv run python dagger/ci.py # containerized CI pipeline
|
||||
```
|
||||
|
||||
Maintainer commands not covered above: `ousast repos` (pinned known-vulnerable checkouts),
|
||||
`ousast model candidates` (what the candidate enumerator can reach) and `ousast learn`
|
||||
(the decision engine's data; see [docs/decision-engine.md](docs/decision-engine.md)).
|
||||
|
||||
@@ -1,5 +1,39 @@
|
||||
# Unreleased
|
||||
|
||||
## PHP quick-mode rules (2026-09-30)
|
||||
|
||||
`ruleset/php/rules.toml` adds 18 PHP rules (SQL injection, command injection, file inclusion,
|
||||
path, open redirect, SSRF; code eval, unserialize and echo XSS as `shadow`). They were written
|
||||
on the development corpus, so their numbers are in-sample only
|
||||
(`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`); PHP is not part of
|
||||
the detection gate. `ousast plane alerts-engine` produces a Run's `alerts` for PHP from the
|
||||
Joern engine image on the host.
|
||||
|
||||
## Framework knowledge as optional priors (2026-09-30)
|
||||
|
||||
Rule and taint-fact entries that know a framework or library carry a `framework`/`library`
|
||||
tag naming a row of `ruleset/frameworks.toml`; the loaders take `priors` (`"all"` by default,
|
||||
unchanged behaviour; `"off"`; or a set of ids). Scan output is unchanged.
|
||||
|
||||
## Decision engine, in development (2026-10-01)
|
||||
|
||||
The learned decision engine (signals from every instrument, labels from ground truth, an AI
|
||||
classifier with local memory compiled in house) is in development and **not adopted**: no scan,
|
||||
pre-push check or report uses it. New maintainer commands: `ousast learn labels|memory|compile|
|
||||
evaluate|curve|audit-leaks` and `ousast plane harvest`; new plane tasks `features` and `roles`;
|
||||
the `advisory-fixes` pair source (97 advisory fix commits). First slice (injection, 110
|
||||
out-of-repository candidates): ADVISORY recall 0.90, precision 0.54, BLOCK not offered
|
||||
(`benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`;
|
||||
[docs/decision-engine.md](docs/decision-engine.md)).
|
||||
|
||||
## Documentation (2026-10-02)
|
||||
|
||||
The README is reduced to user-facing material; scan internals moved to
|
||||
[docs/architecture.md](docs/architecture.md) and measured results to
|
||||
[docs/evaluation.md](docs/evaluation.md). Removed references to commands that no longer exist
|
||||
(`ousast mechanisms export`, `ousast pairs --loo`) and corrected the description of `deep`
|
||||
mode, which runs sandboxed regression snippets when Docker is available.
|
||||
|
||||
## HarnessX removed; the ax plane is the agentic path (2026-09-30)
|
||||
|
||||
HarnessX, the optional agentic extra (`openultrasast[harnessx]`), is removed with its code,
|
||||
|
||||
@@ -0,0 +1,160 @@
|
||||
# Architecture
|
||||
|
||||
How a scan works, how a finding earns its place, how severity and the project score are
|
||||
decided, and how the ruleset improves itself. State as of 2026-10-02; module paths are under
|
||||
`src/openultrasast/`.
|
||||
|
||||
## The scan pipeline
|
||||
|
||||
The CLI is thin; `HarnessRuntime` owns the lifecycle and writes an event trace for every stage
|
||||
(`trace/events.jsonl`). The stages a mode requests are `MODE_STAGES` in `stages.py`:
|
||||
`quick` runs STATIC, `standard` adds MAP, `deep` adds REGRESS.
|
||||
|
||||
```
|
||||
policy_load ─▶ ruleset_load ─▶ policy_check (fail loud on an ungoverned CWE)
|
||||
─▶ static_mapping (SARIF ingest) ─▶ preprocess ─▶ entry_point_mapping
|
||||
─▶ rank ─▶ calibrate (demote previously rejected scopes)
|
||||
─▶ findings quick, deep: quick_findings (language-scoped pattern rules)
|
||||
│ standard: hunter_pool (the same rules, scheduled per ranked target)
|
||||
─▶ verify ─▶ record_calibration ─▶ fusion (standard only, deterministic) ─▶ score
|
||||
─▶ MAP (standard, deep)
|
||||
│ map complexity map and hotspots
|
||||
│ overlay semantic overlay over the findings (tree-sitter with the `semantic` extra)
|
||||
│ obligations absence bugs: an obligated operation with no governing guard
|
||||
│ model one Joern CPG per repository; taint, dominance and configuration arbiters
|
||||
│ tool hunter only with `[models] hunter`
|
||||
─▶ REGRESS (deep) sandboxed regression snippets ─▶ verdicts.json, worth-fixing
|
||||
─▶ report.md, report.sarif, manifest.json
|
||||
```
|
||||
|
||||
**The model layer** (`model/`, `cpg/`) is the core of `standard`. It builds one code property
|
||||
graph per scan with Joern (`joern-parse`, then batched `joern --script` queries from
|
||||
`cpg/queries/*.sc`) and dispatches each question to the arbiter of its spec type: taint
|
||||
(source to sink with no discharging sanitizer), dominance (does a guard govern the obligated
|
||||
operation) and configuration (constant abstraction over a value). Taint facts ship for C,
|
||||
Java, JavaScript, PHP and Python (`ruleset/semantic/*.toml`). Without Joern the layer records
|
||||
`cpg_unavailable` and the scan is otherwise unchanged; the Docker image ships Joern and
|
||||
`php-cli` (which the PHP frontend needs).
|
||||
|
||||
The model layer reports on a rung ladder:
|
||||
|
||||
```
|
||||
execution_confirmed a test or sandbox reproduced it
|
||||
model_entailed the graph decided: a flow with no discharge, an obligation with no
|
||||
guard, a permissive value
|
||||
model_corroborated the graph found the shape; a judge agreed on the residual question
|
||||
suspicion proposed, nothing corroborated it
|
||||
```
|
||||
|
||||
The LLM (`model/judge.py`) is asked only the residual question the graph cannot settle, for a
|
||||
flow the graph already found; it can confirm or contradict, never create a flow. Without a
|
||||
provider key only the `suspicion` band goes unasked.
|
||||
|
||||
**REGRESS** (`regress/`, `sandbox/`) runs only when `docker info` succeeds. Each promoted
|
||||
candidate gets a small language recipe (Python, JavaScript, C) that loads the candidate's
|
||||
source inside a container with no network, a read-only `/workspace`, a non-root user and the
|
||||
`[sandbox]` limits. A structural deny-list rejects snippets mentioning sockets, curl, host
|
||||
networking or writes under `/workspace` before anything runs.
|
||||
|
||||
## Evidence ladder: how a false positive is eliminated
|
||||
|
||||
Evidence level is a state machine, not a label a model may assign. The verifier
|
||||
(`verification.py`) runs on independent context and enforces:
|
||||
|
||||
```
|
||||
suspicion ─▶ static_corroboration ─▶ crash_reproduced ─▶ root_cause_explained
|
||||
─▶ exploit_demonstrated ─▶ patch_validated
|
||||
```
|
||||
|
||||
- below `static_corroboration`: `NEEDS_EVIDENCE`;
|
||||
- `static_corroboration` with reachability unknown: `REJECTED`, with a tie-breaker demanding
|
||||
call-graph, route, CLI, parser or dynamic evidence;
|
||||
- `static_corroboration` with function-level reachability: `ACCEPTED`.
|
||||
|
||||
A rejected finding becomes a **scoped false-positive learning** (`calibration.py`) with a
|
||||
reason from a fixed taxonomy (`unreachable_path`, `missing_attacker_control`,
|
||||
`sanitizer_disproved`, `static_rule_mismatch`, `incorrect_model_assumption`, `duplicate`,
|
||||
`insufficient_impact`, `unsupported`, `contradicted`, `unverified`) and a scope. A rejection in
|
||||
`auth/` demotes only `auth/...`, never the whole class
|
||||
(`tests/test_calibration.py::test_scoped_false_positive_demotion_does_not_suppress_class_globally`).
|
||||
|
||||
## How a false negative is surfaced
|
||||
|
||||
Every expected vulnerability that no finding matched becomes a `BenchmarkCalibrationRecord` in
|
||||
`calibration_records.json`, with a `next_improvement_candidate` (a static rule, SARIF source,
|
||||
entry-point mapping, retrieval package, hunter prompt, dynamic reproducer or skill route).
|
||||
Once a miss is closed the detection gate keeps it closed in CI.
|
||||
|
||||
## Central CWE policy and the project score
|
||||
|
||||
**Policy** (`policy/verycode.py`): a vendored `CWE_Score.tsv` (verycode-policies) is the single
|
||||
source of truth. Each CWE carries a flaw category, a severity (0-5) and `static`/`dynamic` scope
|
||||
flags. A finding's severity comes only from policy, keyed on CWE. CWEs that are not `static`
|
||||
resolve to 0 and are report-only. `assert_rules_resolve` runs as the `policy_check` stage: an
|
||||
enabled rule naming an ungoverned CWE aborts the scan. Verycode has no CWE-120, so the C/C++
|
||||
buffer rules carry CWE-121.
|
||||
|
||||
**Score** (`scoring/project_score.py`): each finding's penalty is
|
||||
`SEV_WEIGHT[severity] x REACH_MULT[reachability]`, and the score is `100 * e^(-total/k)` with
|
||||
`k = 60`.
|
||||
|
||||
| Severity | Weight | | Reachability | Multiplier |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| 5 | 50 | | `reachable` | 1.0 |
|
||||
| 4 | 25 | | `inferred-file-surface` | 0.6 |
|
||||
| 3 | 10 | | `unknown` | 0.4 |
|
||||
| 2 / 1 / 0 | 2 / 1 / 0 | | | |
|
||||
|
||||
A confirmed false positive lowers a finding's reachability multiplier instead of deleting the
|
||||
rule. A severity-5 reachable finding always fails the score gate; `score < min_score`
|
||||
(default 80) fails it only with `[score] blocking = true`. The `score` stage writes
|
||||
`score.json` and merges it into `manifest.json`.
|
||||
|
||||
## Self-improvement
|
||||
|
||||
### Scoped calibration (every scan)
|
||||
|
||||
```
|
||||
scan ─▶ verify ─▶ non-accepted findings ─▶ scoped learnings
|
||||
(.openultrasast/calibration/false_positive_learnings.json)
|
||||
next scan ─▶ rank ─▶ calibrate (demote rejected scopes) ─▶ findings
|
||||
```
|
||||
|
||||
`record_calibration` (after `verify`) merges this run's non-accepted outcomes into the ledger,
|
||||
de-duplicated by finding ID; `calibrate` (after `rank`) demotes the priority of every scope
|
||||
that produced a rejection and writes `applied_calibrations.json`. Demotion is scoped and
|
||||
reversible (`tests/test_pipeline_calibration.py`).
|
||||
|
||||
### Rule-level loop: `ousast improve`
|
||||
|
||||
Each round (`improve/evolve.py`): benchmark with the current ledger, per-rule signals
|
||||
(`rule_signals.json`), bounded proposals, validation (`improve/validator.py`), replay smoke,
|
||||
re-benchmark, and a hard gate. The levers are rule status (`enabled -> shadow` before
|
||||
`disabled`) and score constants; the loop can never edit pattern text
|
||||
or the authoritative CWE severity. A round is accepted only if recall >= 90%, FP < 10%, the
|
||||
project score did not regress and no matched finding was lost; otherwise the ledger is not
|
||||
written. With `--pair-catalog`, every round is also gated per provenance profile on the
|
||||
catalog's holdout pairs (`--no-pair-gate` skips this and is not for accepted ledgers). The same
|
||||
detection gate runs in CI (`python -m openultrasast.gate`).
|
||||
|
||||
### Proposals from plane memory
|
||||
|
||||
With `--memory [STORE]`, `ousast improve` also reads the rows plane runs stored per repository
|
||||
and pin (`improve/memory.py`). Two deterministic rules turn them into rule-status proposals: a
|
||||
rule whose alerts are repeatedly false across repositories is shadowed, and a shadow rule that
|
||||
keeps hitting agreed, declared vulnerable sites is re-enabled. Rows from holdout pairs, from the
|
||||
gated manifest's own cases and from any `--qualify-population` are dropped before any rule sees
|
||||
them. Each proposal records the rows it came from (`memory_proposals.jsonl` next to the
|
||||
journal) and goes through the same validator and gate.
|
||||
|
||||
### The loop as a plane Run
|
||||
|
||||
`ousast plane workspaces <population> --validation-set ... --loop` appends the improvement loop
|
||||
to a generated Run: `alerts` per case (the quick scan on the vulnerable and fixed pins), then
|
||||
the steps `snapshot` (task `memory-snapshot`), `measure`, `propose` and `improve` (`plane/tasks/loop.py`), each model-free
|
||||
with a zero budget. The snapshot applies the same train-on-test guard before any row leaves the
|
||||
store. A recorded live run: `benchmarks/measurements/2026-09-30-harnessx-removal-loop/`.
|
||||
|
||||
For PHP, which quick mode covered with zero rules when that run was made, `ousast plane
|
||||
alerts-engine` produces a Run's `alerts` from the engine image on the host instead
|
||||
(`benchmarks/measurements/2026-09-30-php-engine-alerts/`).
|
||||
@@ -0,0 +1,85 @@
|
||||
# The learned decision engine (in development)
|
||||
|
||||
**Status 2026-10-02: in development, not adopted.** No scan, pre-push check or report uses it;
|
||||
every user-facing path is today's deterministic one until a compiled program is adopted. The
|
||||
design is the `learned-decision-engine` specification (maintainer tooling); the code is in
|
||||
`src/openultrasast/learn/`.
|
||||
|
||||
## Why
|
||||
|
||||
Every detection gain so far was a hand edit made after a miss: a rule, a taint vocabulary entry,
|
||||
a guard, a prompt line, a tie-break. Each was tuned on the cases that exposed it, and engine
|
||||
tuning measured on population v1 did not transfer to v2 (see [evaluation.md](evaluation.md)).
|
||||
The engine replaces that with a decision learned from labelled outcomes and checked out of
|
||||
repository.
|
||||
|
||||
## Design
|
||||
|
||||
- **Signals, not verdicts.** Every instrument (quick rules, the Joern engine, model sink and
|
||||
role classification, the plane's verify passes and agreement, repository facts such as
|
||||
callers) contributes a feature record per candidate `(path, function, family)`, including an
|
||||
explicit state when it did not run or failed. None decides alone.
|
||||
- **Labels from ground truth.** `ousast learn labels` derives labels only from sources on a
|
||||
fail-closed allow-list (`learn/sources.toml`): the vulnerable and fixed sides of pairs, the
|
||||
declared sites and fix ranges of spent populations, benign deltas and recorded adjudications.
|
||||
"Rejected by the plane" is never a negative. Reserved populations (v3) are not on the list.
|
||||
- **An AI classifier with local memory, compiled in house in the DSPy style.** The program is an
|
||||
instruction, a few demonstrations and up to six retrieved labelled examples, then the
|
||||
candidate's code excerpt and signals; the model (`deepseek-flash`) answers
|
||||
vulnerable / not vulnerable / unsure with a rationale, sampled `k` times, giving a score `s`.
|
||||
Compilation bootstraps demonstrations and searches instructions on a compile split, scored by
|
||||
a pre-registered balanced Brier score (`learn/compile.toml`).
|
||||
- **Retrieval** from the memory store: nearest neighbours by Gower distance on the signal
|
||||
profile, re-ranked by cosine similarity of code embeddings (OpenAI `text-embedding-3-small`
|
||||
through OpenRouter, cached by excerpt hash), under a label/family/repository balance rule. An
|
||||
evaluation-boundary filter keeps a candidate's own repository group out of what it is shown.
|
||||
- **Repository-grouped evaluation.** The compile split (25% of repository groups) is never
|
||||
evaluated. The rest is evaluated by outer grouped 5-fold cross-validation: the Platt
|
||||
calibration map is cross-fitted on the other folds and the operating points are chosen on the
|
||||
other folds (nested). One compile per profile, not one per fold. Intervals are 95%
|
||||
repository-cluster bootstrap.
|
||||
- **Two operating points.** BLOCK is offered only when calibration holds (slope CI contains 1,
|
||||
intercept CI contains 0, ECE <= 0.05) and a fold reaches the required precision; otherwise the
|
||||
engine is ADVISORY only. An `unsure` answer can never BLOCK.
|
||||
|
||||
## First measured slice (2026-10-01)
|
||||
|
||||
Record: `benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json` (counts
|
||||
only). Family **injection** only; memory of 724 static examples from all families; input profile
|
||||
v1 with the engine signals and function length withheld (a leak audit found them separating the
|
||||
sides of a pair by construction: `2026-10-01-decision-engine-leak-audit/`); framework priors off.
|
||||
|
||||
| Measure | Value (95% repository-cluster CI) |
|
||||
| --- | --- |
|
||||
| Candidates evaluated | 110 in 29 repository groups, 58 positive |
|
||||
| ADVISORY recall | 0.90 (0.82 - 0.97) |
|
||||
| ADVISORY precision | 0.54 (0.44 - 0.65) |
|
||||
| AUC of the score `s` | 0.74 (0.65 - 0.85) |
|
||||
| Majority "vulnerable" verdict | precision 0.86 (0.72 - 1.0), recall 0.43 (0.29 - 0.62) |
|
||||
| BLOCK | not offered: ECE 0.064 > 0.05, and every fold reported `precision_unreachable` |
|
||||
| Compiled program | balanced Brier 0.128 on the validation split vs 0.199 for the baseline instruction |
|
||||
| Canary agreement on re-run | 0.83 (30 candidates) |
|
||||
| Spend | $0.91 metered at list price for DeepSeek, $0.0040 for embeddings; $0.0032 per evaluated candidate |
|
||||
|
||||
The other evaluable families are not evaluated yet. Leave-one-source-out, leave-one-framework-out
|
||||
and learning curves, experiments, integration into scan and pre-push, adoption, and the one-time
|
||||
v3 check are still ahead.
|
||||
|
||||
## Commands
|
||||
|
||||
`ousast learn` is a maintainer command group over the engine's data. Paid steps take
|
||||
`--budget-usd`; `--replay-only` answers every call from the response cache at no cost.
|
||||
|
||||
| Command | Does |
|
||||
| --- | --- |
|
||||
| `ousast learn labels` | labels from the allow-listed sources (host only); `--snapshot` writes the counts-only record |
|
||||
| `ousast learn memory build` | label + feature record + excerpt -> `example` rows (no model) |
|
||||
| `ousast learn memory embed` | embed the stored excerpts (OpenRouter, metered, cached) |
|
||||
| `ousast learn compile` | compile the program on the compile split |
|
||||
| `ousast learn evaluate --program ID` | outer-fold evaluation, cross-fitted calibration, nested operating points |
|
||||
| `ousast learn curve --program ID` | the metric over memory size on a fixed candidate subset |
|
||||
| `ousast learn audit-leaks` | how well each signal alone separates the sides of a pair (no model) |
|
||||
|
||||
The memory store is the plane's (`OUSAST_MEMORY`; see [ops/ax/README.md](../ops/ax/README.md)).
|
||||
The harvest that produced the plane signals runs as plane Runs generated by
|
||||
`ousast plane harvest` (`benchmarks/measurements/2026-10-01-decision-engine-harvest/record.json`).
|
||||
@@ -0,0 +1,84 @@
|
||||
# Evaluation
|
||||
|
||||
What has been measured on code the tool was not tuned on, and how. State as of 2026-10-02.
|
||||
Every number below is quoted from a committed record; the path is given with it.
|
||||
|
||||
## Instruments
|
||||
|
||||
| Instrument | Measures | Where |
|
||||
| --- | --- | --- |
|
||||
| Detection gate | per-language recall/FP on the bundled cheat-sheet corpora (goal: >= 90% recall, < 10% FP) | `python -m openultrasast.gate`, `benchmarks/manifests/` |
|
||||
| Pair corpus | fire on the vulnerable function, stay silent on its fix | `ousast pairs`, `benchmarks/pairs/` |
|
||||
| Independent populations | real repositories at vulnerable, fixed and benign pins, scored under a pre-registered protocol | `benchmarks/independent/` |
|
||||
|
||||
The detection gate and the local pairs are CI gates. They catch regressions; they do not
|
||||
estimate recall on real code, because the rules were written against them.
|
||||
|
||||
## Pair corpora
|
||||
|
||||
`benchmarks/pairs/catalog.toml` and the slice directories (`local`, `github`, `sast`, `vfc`,
|
||||
`vibe-py`, `vfc-js`, `agent-vfc`, `owasp`) are runnable with `ousast pairs --slice <name>`; each
|
||||
pair carries a provenance profile and a train/holdout split (`--profile`, `--split`). Pointer
|
||||
pairs are not vendored: `--pointers` fetches them into `~/.cache/openultrasast/pairs/`.
|
||||
`benchmarks/pairs/README.md` documents the schema and the review tiers.
|
||||
|
||||
**`advisory-fixes`** (`benchmarks/pairs/advisory-fixes/`, 2026-10-01) holds 97 pointer pairs,
|
||||
one per repository: a public CVE/GHSA fix commit whose advisory-named function is declared on
|
||||
both sides and changed by the fix. It was assembled to give the decision engine real negatives
|
||||
for families the corpus had few of (access control, deserialization, output encoding, path,
|
||||
untrusted destination). It is a label source of the decision engine
|
||||
(`src/openultrasast/learn/sources.toml`), not an `ousast pairs` slice; the research log and
|
||||
every rejection are in `research-2026-10-01.md`.
|
||||
|
||||
## Independent populations and the M4 gates
|
||||
|
||||
The pre-push safety net's qualification milestone (M4) requires, together: at least 95%
|
||||
actionable precision, at least 90% supported recall, at least 95% supported-check completion,
|
||||
the runtime gates, and zero fixed-side or benign false alerts. A population is frozen before
|
||||
any of its cases is scanned, scored once under a pre-registered protocol, and then **spent**:
|
||||
it may inform changes, but it can no longer qualify anything.
|
||||
|
||||
| Population | Status | Result | Record |
|
||||
| --- | --- | --- | --- |
|
||||
| v1 (11 cases) | spent | engine: recall 1/11, precision 1/12, 4 fixed-side and 1 benign false alert; every M4 gate failed. Eight of eleven cases completed no question at the vulnerable pin. | `benchmarks/independent/results-v1.json` |
|
||||
| v2 (17 cases) | spent | engine: recall 0/17, 3 benign alerts, completion 13,206 of 108,374. Tool hunter: recall 1/17, precision 2/14, 1 fixed-side and 16 benign alerts, $8.88. No M4 gate met by either. | `results-v2.json`, `results-v2-hunter.json` |
|
||||
| v2, model-driven pipeline | exploratory (v2 already spent) | stopped after 7 of 17 cases when the account emptied: recall 4/7; precision not adjudicated; no gate measurable | `results-v2-model-pipeline.json` |
|
||||
| v3 (PHP, 16 cases) | frozen, untouched | none: it is the one-time final check, run once under `protocol-v3.md` after a prediction is committed | `population-v3-php.toml`, `freeze-v3-php.json` |
|
||||
|
||||
v3 is reserved: development and analysis work never reads its cases, and
|
||||
`tests/test_independent_population.py` fails when anything in the development tree (a recipe,
|
||||
catalog, manifest or measurement) names one of its repositories.
|
||||
|
||||
## The plane increment
|
||||
|
||||
The first increment of the agentic plane ran the model-driven pipeline's validation set (46
|
||||
candidates in 15 cases of v2, 20 declared vulnerable sites) on google/ax and compared it with
|
||||
the script pipeline on the same cases, candidates, pins and recorded triage. The pre-registered
|
||||
gate: at least 16 of 20 declared sites agreed, at a cost per candidate under $0.022.
|
||||
|
||||
- **Increment 1** (2026-09-29, `benchmarks/independent/plane-increment-1.json`): 14 of 20
|
||||
declared sites agreed against the reference's 16; $0.0194 per candidate with recorded triage.
|
||||
**NOT MET** on recall.
|
||||
*Correction (2026-09-30, in the record):* the first version read agree's
|
||||
`declared_sites_matched` (coverage by candidates, 19) as "found in either pass" and concluded
|
||||
detection was unchanged. Recounted from the agreed rows, the plane's passes flagged 17 of 20
|
||||
declared sites in at least one pass against the reference's 19.
|
||||
- **Increment 2** (2026-09-30, `benchmarks/independent/plane-increment-2.json`): after the design
|
||||
revisit, a tie-break pass c on the 10 disputed candidates with 2-of-3 agreement. 16 of 20
|
||||
declared sites agreed (reference 16) at $0.0205 per candidate over 46 ($0.02195 over the 43
|
||||
after triage, against the reference's $0.022). **MET**, with two caveats the record carries:
|
||||
the tie-break was chosen after seeing the first measurement on the same validation set, so
|
||||
this compares plane with script and qualifies nothing on an independent population; and the
|
||||
plane's passes still flag fewer declared sites than the reference (17 vs 19). Precision is
|
||||
not part of this gate and remains the open problem.
|
||||
|
||||
## The decision engine
|
||||
|
||||
The first measured slice of the learned decision engine (injection family, out-of-repository
|
||||
evaluation) is described with its numbers in [decision-engine.md](decision-engine.md).
|
||||
|
||||
## Removal of the earlier agentic extra
|
||||
|
||||
The removal of the earlier agentic extra (2026-09-30) was checked for byte-identical
|
||||
deterministic outputs (pair replay, quick benchmark, standard scan, `improve --dry-run`):
|
||||
`benchmarks/measurements/2026-09-30-harnessx-removal-baseline/` and `-equality/`.
|
||||
+24
-15
@@ -13,7 +13,18 @@ jq . "$run/score.json" # 0–100 project score + gate verdict
|
||||
"$run/report.sarif" # SARIF for code-scanning / IDEs
|
||||
```
|
||||
|
||||
## 2. CI gate
|
||||
## 2. Standard scan with the engine (Docker)
|
||||
|
||||
```bash
|
||||
source ops/shell/ousast.sh # once per shell; builds the image on first use
|
||||
ousast scan . --mode standard # Joern model layer, container network off: no model calls
|
||||
ousast-with-judge scan . --mode standard # network on, so the model is asked (needs DEEPSEEK_API_KEY)
|
||||
```
|
||||
|
||||
Without a key the graph still decides; only the `suspicion` band goes unasked, and the manifest
|
||||
records the degradation.
|
||||
|
||||
## 3. CI gate
|
||||
|
||||
```bash
|
||||
# Fail the build on any evidence-verified finding:
|
||||
@@ -23,23 +34,21 @@ uv run ousast scan . --mode quick --fail-on verified
|
||||
uv run python -m openultrasast.gate
|
||||
```
|
||||
|
||||
## 3. Benchmark a known-vulnerable corpus
|
||||
## 4. Benchmark a known-vulnerable corpus
|
||||
|
||||
```bash
|
||||
uv run ousast benchmark benchmarks/manifests/python-vulnerable.toml --mode quick
|
||||
# -> expected / matched / missed counts + a per-rule recommendation delta
|
||||
```
|
||||
|
||||
## 4. Vuln vs fixed pair eval (efficiency, not cheat-sheet recall)
|
||||
## 5. Vuln vs fixed pair eval (efficiency, not cheat-sheet recall)
|
||||
|
||||
```bash
|
||||
uv run ousast pairs # local fixtures + vendored GitHub VFCs
|
||||
uv run ousast pairs --slice github --json # honesty dashboard as JSON
|
||||
uv run ousast pairs --slice sast # OWASP Benchmark + Juliet Youden (TPR-FPR)
|
||||
uv run ousast pairs --slice vibe-py --pointers # nightly: pointer pairs via the local cache (network)
|
||||
uv run ousast mechanisms export --slice vibe-py # maintainer: seed mechanism candidates from trusted pairs (offline)
|
||||
uv run ousast improve bench.toml --pair-catalog benchmarks/pairs/catalog.toml # the mechanisms lever admits candidates the holdout pairs prove; the scan then searches them
|
||||
uv run ousast pairs --slice vibe-py --loo # leave-one-out: the corpus's own detection rate (reports/loo.json)
|
||||
uv run ousast pairs --split holdout --profile human # filter by declared split and provenance profile
|
||||
uv run python -m openultrasast.pair_gate # CI: local pairs must all pass
|
||||
```
|
||||
|
||||
@@ -48,7 +57,7 @@ Each case is two isolated trees (vuln file vs patched file, same relative path).
|
||||
fix. Misses and fix-side leaks become `miss`/`fp` signals for `ousast improve`.
|
||||
The catalog and public dataset pointers live in `benchmarks/pairs/`.
|
||||
|
||||
## 5. Self-improve the ruleset from benchmark feedback
|
||||
## 6. Self-improve the ruleset from benchmark feedback
|
||||
|
||||
```bash
|
||||
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run # preview
|
||||
@@ -57,16 +66,16 @@ uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml
|
||||
# which the next scan/benchmark of that target loads automatically.
|
||||
```
|
||||
|
||||
## 6. Running the agentic plane
|
||||
## 7. Running the agentic plane
|
||||
|
||||
Model work over whole repositories runs as a Run manifest on ax, one isolated actor
|
||||
per task (bring-up: [ops/ax/README.md](../ops/ax/README.md)). The provider key is
|
||||
read from the environment (or `.env`) by `ousast` and sent only in each task's start
|
||||
request.
|
||||
read from the environment (or `.env`, which never overrides an exported variable) by
|
||||
`ousast` and sent only in each task's start request.
|
||||
|
||||
```bash
|
||||
uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image
|
||||
uv run ousast plane run plane/runs/validation-46.yaml # repo-facts, verify a/b(/c), agree, remember per case
|
||||
uv run ousast plane run plane/runs/validation-46.yaml # per case: repo-facts, verify a/b/c, agree, final, features, remember
|
||||
uv run ousast plane status validation-46 # per-task status and token attribution
|
||||
uv run ousast plane remember validation-46 # re-ingest the run's memory rows (OUSAST_MEMORY)
|
||||
|
||||
@@ -76,14 +85,14 @@ uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dr
|
||||
|
||||
Each task in the Run carries its own Model and `budget: {usd, calls}`; a task that
|
||||
reaches its ceiling stops as `unfinished` and resumes on a rerun with a larger
|
||||
budget. A local `ousast scan --mode standard` stays deterministic apart from the
|
||||
optional `[models] hunter` tool hunter.
|
||||
budget. `ousast plane status` prints the token attribution table: calls, prompt,
|
||||
cache-hit and output tokens and USD per task.
|
||||
|
||||
## 7. Drive it from an MCP client (OpenCode / IDE)
|
||||
## 8. Drive it from an MCP client (OpenCode / IDE)
|
||||
|
||||
```jsonc
|
||||
{ "command": "uv", "args": ["run", "ousast", "mcp"] }
|
||||
```
|
||||
|
||||
Then: `openultrasast.scan {path}` → `run_dir` → `openultrasast.findings {run_dir}` →
|
||||
`openultrasast.explain {run_dir, finding_id}`. See the README "MCP server" section.
|
||||
`openultrasast.explain {run_dir, finding_id}`. See the README section "Agent integrations".
|
||||
|
||||
+28
-21
@@ -5,18 +5,23 @@ it does not, and the controls that keep a scan safe to run in CI and safe to sha
|
||||
|
||||
## Trust boundaries
|
||||
|
||||
- **Scanned code is untrusted input, never executed.** Quick and standard modes read
|
||||
source as text (regex rules, evidence ladder, optional LLM review). They never import,
|
||||
build, or run the target. Only `deep` mode would execute target-derived code, and only
|
||||
inside the sandbox below — and deep mode is not yet implemented (it exits with a clear
|
||||
message rather than running anything).
|
||||
- **Scanned code is untrusted input.** Quick and standard modes never import, build or run
|
||||
the target: quick reads source as text, and standard additionally parses it into a code
|
||||
property graph with Joern (`joern-parse`, a parser, not a build) and may send code excerpts
|
||||
to the configured model. Only `deep` mode executes target-derived code: its REGRESS stage
|
||||
loads promoted candidates inside the Docker sandbox below, and it runs only when `docker
|
||||
info` succeeds (otherwise the stage is skipped and recorded as `sandbox_unavailable`).
|
||||
- **The ruleset and CWE policy are trusted, governed data.** They change only through
|
||||
reviewed commits or the bounded self-improvement loop (`ousast improve`), which can
|
||||
flip rule *status* and tune score constants but can never edit pattern text or the
|
||||
authoritative 0–5 severity. See the self-improving cycle in the README.
|
||||
- **Provider credentials are the operator's.** The zero-dependency core makes no network
|
||||
calls. A local scan calls a model only for the optional `[models] hunter` tool hunter
|
||||
in standard mode. Agentic work runs on the plane (`ousast plane run`): `ousast` reads
|
||||
authoritative 0–5 severity. See [architecture.md](architecture.md#self-improvement).
|
||||
- **Provider credentials are the operator's.** The core install (PyYAML only) makes no
|
||||
network calls on its own. A local scan calls a model only in standard or deep mode, and
|
||||
only when a key is available: the model layer asks the residual question the graph cannot
|
||||
settle, and `[models] hunter` enables the tool hunter. LLM calls go to DeepSeek
|
||||
(`DEEPSEEK_API_KEY`); OpenRouter (`OPENROUTER_API_KEY`) serves embeddings. Keys come from
|
||||
the environment or a gitignored `.env`, which never overrides an exported variable. The
|
||||
pre-push check calls a model only with an explicit `--model-config`. Agentic work runs on the plane (`ousast plane run`): `ousast` reads
|
||||
the variable a task's Model names (`secretKey.key`) from the operator's environment or
|
||||
`.env` and sends the key only in that task's start request, through Agent Substrate's
|
||||
router to that task's actor. It is not baked into an image or task template, and it is
|
||||
@@ -54,7 +59,7 @@ LLM/embedding calls retry transient failures (HTTP 429/5xx, connection errors, t
|
||||
with exponential backoff; non-transient errors (e.g. 4xx, malformed JSON) fail fast.
|
||||
|
||||
### Visible degradation & determinism
|
||||
When an optional capability is missing (the extra, a model, a provider key), the scan
|
||||
When an optional capability is missing (Joern, Docker, a model, a provider key), the scan
|
||||
falls back to its deterministic equivalent and records a `degradations` entry in the
|
||||
manifest — it is never silently downgraded. Runs are reproducible: fixed config,
|
||||
artifact manifests, prompt hashes, and model identifiers are recorded.
|
||||
@@ -63,18 +68,20 @@ artifact manifests, prompt hashes, and model identifiers are recorded.
|
||||
`ousast mcp` exposes only the ten narrow project tools. No tool runs arbitrary shell,
|
||||
Docker, or internal hunter tools, and no tool accepts a free-form command argument.
|
||||
|
||||
## Sandbox limits (deep mode, when implemented)
|
||||
## Sandbox (deep mode)
|
||||
|
||||
Dynamic analysis runs under the bounds in `[sandbox]` (`config.SandboxConfig`):
|
||||
The REGRESS stage (`regress/`, `sandbox/runner.py`) runs each snippet with `docker run`
|
||||
and never mounts the host Docker socket:
|
||||
|
||||
| Control | Default |
|
||||
| Control | Setting |
|
||||
|---|---|
|
||||
| network | disabled (`network = false`) |
|
||||
| workspace | read-only (`workspace_readonly = true`) |
|
||||
| memory | 2048 MB (`memory_mb`) |
|
||||
| pids | 512 (`pids_limit`) |
|
||||
| timeout | 300 s (`timeout_seconds`) |
|
||||
| network | `--network none`, always; a host-network argument is refused |
|
||||
| source | the target bind-mounted read-only at `/workspace`; root filesystem `--read-only` |
|
||||
| scratch | a tmpfs at `/scratch` for the snippet |
|
||||
| user | non-root `65534:65534`, `--cap-drop ALL`, `no-new-privileges` |
|
||||
| memory | `[sandbox] memory_mb`, default 2048 MB |
|
||||
| pids | `[sandbox] pids_limit`, default 512 |
|
||||
| timeout | `[sandbox] timeout_seconds`, default 300 s |
|
||||
|
||||
Dynamic probes additionally declare their network scope, log commands, and capture
|
||||
artifacts; a hardened runtime (e.g. gVisor) is used when available. Until deep mode
|
||||
ships, no target code is executed at all.
|
||||
Before anything runs, a structural deny-list (`regress/safety.py`) rejects a snippet that
|
||||
mentions sockets, curl, host networking or writes under `/workspace`.
|
||||
|
||||
+11
-6
@@ -1,7 +1,7 @@
|
||||
# ops
|
||||
|
||||
Infrastructure for the optional engines. Nothing here is a project dependency: the core install is
|
||||
`dependencies = []` and every engine below degrades to a recorded reason when it is absent.
|
||||
Infrastructure for the optional engines. Nothing here is a project dependency: the core install needs only
|
||||
PyYAML, and every engine below degrades to a recorded reason when it is absent.
|
||||
|
||||
## Use it as a command
|
||||
|
||||
@@ -22,7 +22,7 @@ Then, from any directory:
|
||||
```bash
|
||||
ousast scan .
|
||||
ousast scan src/api
|
||||
ousast-with-judge scan . # opt into the LLM judge: needs network and a key
|
||||
ousast-with-judge scan . --mode standard # container network on, so the model layer may ask its question (needs DEEPSEEK_API_KEY)
|
||||
```
|
||||
|
||||
The wrapper rewrites paths under your working directory to their mounted equivalents, so `src/api` means what
|
||||
@@ -48,7 +48,8 @@ TARGET=/path/to/repo docker compose run --rm ousast scan /target
|
||||
The target is mounted read-only; `/work` is a tmpfs so CPG scratch never outlives the run. The LLM endpoint
|
||||
is optional and passed through the environment, never baked into a layer — without it the model layer still
|
||||
entails and only the `suspicion` band goes unasked. `network_mode` defaults to `none`; set it to `bridge`
|
||||
only when you want the judge.
|
||||
(`OUSAST_NETWORK=bridge`, which `ousast-with-judge` does) only when the model layer should reach the model
|
||||
endpoint. Compose passes `DEEPSEEK_API_KEY` and `OPENROUTER_API_KEY` through from the environment.
|
||||
|
||||
**pyinfra (for a persistent host install).** Same engine, same pin, same checksum, but on the machine:
|
||||
|
||||
@@ -146,7 +147,8 @@ No Joern server mode, no workspace/project management, no `joern-scan`, no overl
|
||||
## Languages
|
||||
|
||||
Joern ships frontends for C/C++, Java (source and bytecode), JavaScript/TypeScript, Python, Kotlin, Go, C#,
|
||||
PHP, ABAP and Ghidra binaries. We currently model **python, javascript, java, c**.
|
||||
PHP, ABAP and Ghidra binaries. Taint facts ship for **python, javascript, java, c and php**
|
||||
(`src/openultrasast/ruleset/semantic/*.toml`).
|
||||
|
||||
**PHP works but needs a PHP interpreter**: `php2cpg` drives PHP-Parser and shells out to `php`, so on a host
|
||||
without it a PHP tree fails to build with an opaque "Process exited with code 1". Set `php_frontend: True` in
|
||||
@@ -372,4 +374,7 @@ read under the same analysis deadline.
|
||||
|
||||
## The agentic service plane on ax
|
||||
|
||||
Model-driven pipeline stages run as tasks on google/ax over Agent Substrate (`.kiro/specs/ai-service-plane/`). Bring-up, the smoke Run, the measured footprint and the lessons from the live cluster are in [`ops/ax/README.md`](ax/README.md); `ousast plane run|status|doctor|workspaces` drives it.
|
||||
Model-driven work over whole repositories runs as tasks on google/ax over Agent Substrate. Bring-up, the task
|
||||
catalogue, the memory store, the measured footprint, the lessons from the live cluster and the checklist for a
|
||||
separate Kubernetes cluster are in [`ops/ax/README.md`](ax/README.md);
|
||||
`ousast plane run|status|doctor|workspaces|remember|alerts-engine|harvest` drives it.
|
||||
|
||||
+69
-1
@@ -14,6 +14,74 @@ single-node kind cluster. ax is the only executor; `ousast plane run` submits, r
|
||||
Tools live in `~/go/bin` (kind, ax, ko, kubectl-ate) and `~/.local/bin` (kubectl); sources and rendered files
|
||||
in `~/.cache/ousast/ax-src/` (the runner digest pin is `runner-image` there).
|
||||
|
||||
## How a Run executes
|
||||
|
||||
A Run (`plane/runs/*.yaml`, kind `openultrasast.io/v1alpha1 Run`) is a DAG of steps; each step names an ax Task
|
||||
(`plane/tasks/`), its inputs and outputs, and its own `budget: {usd, calls}`. Every Task binds its Workspaces
|
||||
(`plane/workspaces/`) and one ax Model (`plane/models/`): `deepseek-flash` for chat, `openrouter-embedding`
|
||||
(OpenAI `text-embedding-3-small` through OpenRouter) for embeddings. `ousast plane run` (`plane/reconciler.py`)
|
||||
applies Workspaces before the Tasks that bind them, resumes each Task, sends its start signal, collects its
|
||||
artifacts and records state under `$OUSAST_RESULTS/plane/<run>/` (default `~/ousast-results/plane/`); a rerun
|
||||
skips tasks already done, and `--workers` bounds the tasks in flight.
|
||||
|
||||
**Runner contract** (`plane/runner.py`, PID 1 of the runner image). ax hands the Task in `AX_TASK_YAML` and the
|
||||
Workspaces in `AX_WORKSPACES_YAML`; the runner serves `/healthz` and `/readyz` (503 until every workspace is
|
||||
materialised). Agent Substrate boots each template once as a golden actor to snapshot it, so the command never
|
||||
starts by itself: it waits for `POST /ousast/v1/start` with `{run, task, credentials}`, which the reconciler sends
|
||||
through Substrate's `atenet-router` to that actor only (`plane/router.py`). **The provider key exists only in that
|
||||
request**: the runner keeps it in memory, passes it to the command's environment, redacts it from echoed stderr
|
||||
and never writes it. The reconciler reads the variable the Model's `secretKey.key` names from the operator's
|
||||
environment or `.env` (which never overrides an exported variable). After the command exits, the runner posts
|
||||
`OUSAST_OUTPUT_DIR` as a tar to the receiver; that delivery is the task's completion, and a completion marker
|
||||
stops a resumed actor from repeating billed work.
|
||||
|
||||
**Budgets.** The metered client refuses the next call once a task's `usd` or `calls` ceiling is reached; the task
|
||||
ends `unfinished` and resumes on a rerun with a larger budget. An account or authentication error ends it
|
||||
`failed`. Model-free tasks run with `{usd: 0, calls: 0}`, so a stray call fails loudly.
|
||||
|
||||
**Egress** (`plane/egress.py`). One EgressPolicy per actor through the `atenet-egress` gateway, deny by default,
|
||||
hostnames only: plain HTTP to the artifact receiver, TLS passthrough to the source hosts of the bound Workspaces
|
||||
and to the hosts the bound Model declares (`openultrasast.io/egress-hosts`). The policy is deleted with the actor.
|
||||
|
||||
**Token attribution.** `ousast plane status <run>` prints per task its status and `calls`, `prompt`,
|
||||
`cache_hit` and `output` tokens and `usd` from the tasks' `summary.json`, and writes `<run dir>/attribution.json`;
|
||||
`--units` adds per-unit rows.
|
||||
|
||||
## Tasks
|
||||
|
||||
| Task | Model | Does |
|
||||
|---|---|---|
|
||||
| `repo-facts` | none | functions per product file, and each candidate's call sites in other files |
|
||||
| `verify` (passes a, b, c) | `deepseek-flash` | batched tool hunt per file over the candidates, with their known callers |
|
||||
| `agree` | none | a and b agreed or disputed; after pass c on the disputed, 2-of-3 (the step `final`) |
|
||||
| `features` | none | one feature record per candidate for the decision engine |
|
||||
| `remember` | none | the case's artifacts as memory rows (`facts`, `verdict`, `unit_cost`, `alert`, `coverage`, `features`) |
|
||||
| `alerts` | none | the quick scan on the vulnerable and fixed pins (loop Runs) |
|
||||
| `loop` | none | the steps `snapshot`, `measure`, `propose`, `improve`: the improvement loop as a Run |
|
||||
| `roles` | `deepseek-flash` | per-repository source, sink and sanitizer roles without a vocabulary (decision-engine harvest) |
|
||||
|
||||
Generators: `ousast plane workspaces <population> --validation-set ...` writes the per-case chain
|
||||
(`facts -> va, vb -> agree -> vc -> final -> features -> remember`, plus `alerts` and the loop with `--loop`);
|
||||
`ousast plane harvest --labels ... --plane DIR` writes the decision engine's harvest Runs; `ousast plane
|
||||
alerts-engine` produces a Run's `alerts` from the Joern engine image on the host, for PHP and other languages
|
||||
quick mode does not cover.
|
||||
|
||||
## Memory store
|
||||
|
||||
`plane/memory.py`. One store keyed by repository and pin holds what runs learned: `index.jsonl`, `facts/` by
|
||||
content hash, `repos/<host>__<owner>__<name>/<pin>.jsonl` rows, and content-addressed blobs (excerpts, embeddings,
|
||||
cached model responses, compiled programs). `ousast plane remember <run>` ingests a run's rows; a repeated ingest
|
||||
is skipped by the index. A facts entry is reused for the same repository, pin, candidates and runner image.
|
||||
|
||||
`OUSAST_MEMORY` selects the backend:
|
||||
|
||||
- `file:///path` (`FileStore`), default `$OUSAST_RESULTS/plane/memory`; it refuses to write below 1 GiB free;
|
||||
- `minio://<bucket>[/<prefix>]` (`MinioStore`, the `minio` extra), used on the maintainer's MinIO. Endpoint and
|
||||
credentials come from `.env` or the environment (`MINIO_ENDPOINT`, `MINIO_ACCESS_KEY`, `MINIO_SECRET_KEY`,
|
||||
`MINIO_SECURE`), never from a manifest, and are never printed.
|
||||
|
||||
`ousast improve --memory [STORE]` and the `ousast learn` commands (`--memory`) read the same store.
|
||||
|
||||
## Measured footprint (2026-09-29, idle, two workers)
|
||||
|
||||
| Item | Value |
|
||||
@@ -62,7 +130,7 @@ are; these parts assume this laptop and must change first (checked 2026-09-30):
|
||||
| ax's snapshot bucket: `AX_SNAPSHOTS_BUCKET` in ax's `deploy/ax-server.yaml` points at the ax authors' GCS bucket | ax deploy manifest | your own bucket, set before deploying ax |
|
||||
| Egress gateway applied by hand (agentgateway variant, no Rust build) | this README | the Substrate-installed gateway; per-task EgressPolicies work unchanged |
|
||||
| Worker pool of 2 x 1 CPU / 1.5 GiB for the 7 GB host | `workerpool.yaml.tmpl` | sized to the cluster; `--workers` to match |
|
||||
| Plane memory store and results under `~/ousast-results/` on the host | reconciler, memory layer (harnessx-removal Req 6) | a persistent volume or bucket shared by the reconciler |
|
||||
| Plane memory store and results under `~/ousast-results/` on the host | `reconciler.py` (`OUSAST_RESULTS`), `memory.py` (`OUSAST_MEMORY`) | a persistent volume or bucket shared by the reconciler; the `minio://` store already works against any reachable MinIO |
|
||||
|
||||
The provider key already travels only in the start request through `atenet-router`, which works the same through
|
||||
a port-forward to any cluster.
|
||||
|
||||
Reference in New Issue
Block a user