Open_Ultra_SAST/README.md

# OpenUltraSAST

OpenUltraSAST is an experimental static security analyser. It aims to check security on
repositories it has never seen. It scans a checkout or the commits a `git push` would publish. It
distinguishes what it can prove from what it suspects.

A scan combines up to four parts:

- language-scoped pattern rules;
- a tree-sitter overlay (a syntax tree of each file, built by the tree-sitter parser);
- a Joern code property graph (a graph of the program's syntax, control flow and data flow);
- optionally, a model. It is asked only the question the graph cannot settle.

Every finding includes its supporting evidence. When a scan cannot check something, it records
a degradation instead of reporting a clean result.

Where a repository's application can be built and run, an experimental proof arm goes further. It
demonstrates a finding by executing it. The issue triggers on the vulnerable revision and not on
the fixed one, which makes the finding reproducible. This arm is in pilot testing and is being
qualified on repositories the tool has never seen; see [Status](#status).

`ousast` is the interface. Model work over whole repositories runs on a separate agentic plane
(google/ax). A learned decision engine is in development. A scan needs neither of them.

Each section in this file links to the corresponding page on the documentation site.
The site is published at <https://norandom.github.io/OpenUltraSAST/>. Its sources are in `docs/` and
`mkdocs.yml`; to preview it locally, run `uv sync --extra docs && uv run mkdocs serve`.

## Status

**v2.0.1, state as of 2026-10-08.** The pre-push safety net is **not yet released**. Blocking on
the static analyser by hand is **NO-GO** against every M4 gate, and the project no longer tries to
meet those gates that way.

- No blocking hook capability is enabled; the pre-push hook is advisory only.
- Two independent populations (case sets frozen before any scan) have been spent on the static
  engine: recall was 1 of 11, then 0 of 17. The learned decision engine ranks findings but does not block for any
  family.
- Static abstract interpretation could not prove the precision needed to block a push. Current
  work focuses on a **dynamic proof arm**. It builds and runs a repository's app and
  demonstrates a finding by a differential: the issue triggers on the vulnerable revision and not
  on the fixed one. Execution establishes the finding as ground truth, so it can justify a block
  without calibration. Pilot demonstrations exist, including one executed on the cluster; the arm
  is being qualified, not shipped.
- The release gate is a **measured** catch-rate and false-alarm number on repositories the tool
  has never seen (the unseen-repo evaluation). Until that number
  exists, treat all output as advisory.

Every detection change must show an improvement in a controlled measurement on held-out repositories.
The current status and the record behind each figure are in
[docs/where-we-stand.md](docs/where-we-stand.md). For developers, the pre-push check and its
honest limits: [docs/pre-push-for-developers.md](docs/pre-push-for-developers.md); the
catch–prove–fix–re-verify loop: [docs/use-case-fix-and-reverify.md](docs/use-case-fix-and-reverify.md).
The release history is in [RELEASE_NOTES.md](RELEASE_NOTES.md).

## Install and first scan

Pick the tier you need:

1. Tier 1: a deterministic scan of a checkout, or a CI gate.
2. Tier 2: the Joern engine (`standard`, `deep`).
3. Tier 3, for maintainers: model work over whole repositories on the plane.
4. Tier 4 does not exist yet.

### Tier 1: quick scan, no Docker

The core install depends only on PyYAML. From a checkout, `uv sync` (or `pip install .`) installs
the `ousast` entry point. `uv sync --group dev` adds the test and lint tools.

```bash
uv sync
uv run ousast scan /path/to/repo --mode quick
uv run ousast scan . --mode quick --fail-on verified   # CI: exit 1 on any evidence-verified finding
```

This runs the language-scoped pattern rules, entry-point reachability hints, ranking,
verification and scoring. It is deterministic and reproducible. It needs no network and no keys.

The same install gives you the advisory pre-push hook for any repository. It runs the quick rules
on the lines a push changes, names every check it could not run, and never blocks a push:

```bash
cd /path/to/your/repository && ousast pre-push install    # remove: ousast pre-push uninstall
```

What it cannot do:

- There is no engine. It cannot follow a value across statements or files.
- The PHP quick rules are a low-precision, in-sample development set. Per-rule precision lower
  bounds run from 0.03 to 1.0 on the real pins
  (`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`).

The optional `semantic` extra (`uv sync --extra semantic`) installs the tree-sitter grammars for
the overlay.

### Tier 2: standard and deep scans (Docker)

`standard` and `deep` need Joern. The Docker image includes the tool and
Joern v4.0.625. It also includes `php-cli`, which the PHP frontend needs. Each release publishes the image
to GHCR. It is big: a JRE plus about 2 GB of Joern. Pull it instead of building, and tag it with the name the
shell wrapper uses:

```bash
docker pull ghcr.io/norandom/openultrasast:2.0.1
docker tag ghcr.io/norandom/openultrasast:2.0.1 openultrasast:dev   # or build: docker compose build
```

The shell wrapper runs `ousast` in the container. It builds the image only if `openultrasast:dev`
is missing. Your directory is mounted read-only and the container network is off. For the pre-push
hook, use `ousast pre-push install --docker`: Git cannot call a shell function, so it installs the
script `ousast-docker` beside the hook.

```bash
source /path/to/OpenUltraSAST/ops/shell/ousast.sh   # PowerShell: ops/shell/ousast.ps1
ousast scan . --mode standard                       # the graph decides; no model is asked
ousast-with-judge scan . --mode standard            # network on, the model is asked (DEEPSEEK_API_KEY)
```

`docker compose build` and `TARGET=/path/to/repo docker compose run --rm ousast scan /target` do
the same without the wrapper.

`deep` adds REGRESS. REGRESS runs each reproduction snippet with `docker run` from wherever the
scan runs. The engine container has no Docker socket. So run `deep` in one of two ways:

- from a host install with Joern on the PATH (`pyinfra @local ops/joern.py`, pinned and checksum-
  verified) and a working `docker`;
- or accept the recorded `sandbox_unavailable` degradation.

Details, including the host install, are in [ops/README.md](ops/README.md).

Keys and store settings live in `.env` in the working directory. The file is gitignored;
`.env.example` lists the names.

- Every LLM call uses `DEEPSEEK_API_KEY`.
- `OPENROUTER_API_KEY` is used for embeddings only (`ousast learn memory embed`).
- `.env` never overrides a variable already exported in your shell. If a key change seems to have
  no effect, `unset` the stale export.
- The plane's memory store defaults to a local file store under `~/ousast-results/plane/memory`.
- An S3-compatible bucket is optional: `uv sync --extra s3`, then `OUSAST_MEMORY=s3://<bucket>`
  with `S3_ENDPOINT` and the AWS credentials in `.env`. Nothing in a scan needs it.

### Tier 3: the plane on ax (maintainers)

Tier 1 and 2 need none of this. The current profile is a single-node kind cluster on the maintainer's
host.

- `ops/ax/up.sh` needs `docker`, `go`, `kubectl`, `kind`, `ko` and `ax` on the PATH. It refuses to
  start when one is missing.
- It builds Agent Substrate and ax from their checkouts under `~/.cache/ousast/ax-src/`.
- Measured idle footprint with two workers: the kind node takes 1.7 GiB of memory (27% of the
  7.7 GiB host) and 4.9 GB of disk for images. The build uses several GB more for Go caches.
- The worker pool is sized for that 7 GB host.
- The verify and roles tasks need `DEEPSEEK_API_KEY`.
- Memory lives in the local file store or the S3 store. RustFS is the tested server.

See [ops/ax/README.md](ops/ax/README.md) for bring-up and doctor instructions, along with the smoke Run.

### Tier 4: Kubernetes (k3s), planned

Nothing has been deployed to any cluster other than that kind cluster. The target is two profiles
of one code base: `kind` locally and `k3s` in production. The same `ousast plane run` would
submit a Run in a remote execution mode. It needs:

- a kubeconfig context or the cluster's ingress (and an `imagePullSecret` on the worker
  ServiceAccount for a private package);
- runner and engine images published to GHCR and pinned by digest;
- the S3 store as the artifact path, with no receiver on a laptop.

The plan and required code changes, with file paths, are in
[docs/deployment.md](docs/deployment.md#production-topology-ax-on-a-kubernetes-cluster).

## Scan modes

There are three modes. Each adds to the one before.

- `quick` runs the pattern rules, hints, ranking, verification and score. It needs nothing.
- `standard` adds the MAP stage: complexity map, semantic overlay, authorization obligations and
  the Joern model layer. The model layer builds one code property graph per repository. It decides
  taint, guard-dominance and configuration questions on that graph. A model is asked only the
  residual question. Without a key, that question is skipped and only the graph's entailed
  findings are reported.
- `deep` adds REGRESS. Promoted candidates are loaded by a small snippet in a Docker sandbox. Each
  gets a `triggerable` or `not_triggerable` verdict.

A missing engine is a recorded degradation (`cpg_unavailable`, `sandbox_unavailable`), never a
clean result.

`--fail-on never|findings|verified|worth-fixing` sets the exit code. `--config openultrasast.toml`
loads settings. Every run writes these files under `<target>/.openultrasast/runs/<scan-id>/`:
`manifest.json`, `findings.json`, `verification.json`, `score.json`, `report.md`, `report.sarif`
and `trace/events.jsonl`.

`ousast pre-push` (experimental) analyses the commits a push would publish against their base.
First, the quick rules run on the changed lines. Then the engine compares head with base, and only
new or worsened defects become candidates. It is advisory by default, and it prints one line per
check it could not run. Its capability registry is empty, so it emits no normal alert today. An
engine finding the push introduced is shown as advisory. Install, settings, skip reasons and a CI
recipe: [docs/scanning.md](docs/scanning.md#first-run-install-the-hook-on-any-repository).

Flows and the per-language rule table: [docs/scanning.md](docs/scanning.md).

## How detection works

The tool uses three representations of code, from cheap to expensive:

1. Text: regex quick rules, one statement at a time.
2. Tree: the tree-sitter CST (concrete syntax tree) and an IR (intermediate representation) per
   function. The overlay and the obligations use it.
3. Graph: the Joern CPG. It is the only one that follows a value across statements, calls and
   files.

Taint analysis uses fact tables and queries on the graph to produce verdicts. Absence bugs involve
operations that should have been guarded. The tool finds them through obligations and dominance
checks, plus a configuration census. A model answers only the residual question.

- [docs/detection-techniques.md](docs/detection-techniques.md) explains each technique, with the
  code it comes from and what it misses.
- [docs/architecture.md](docs/architecture.md) gives the stage order and the evidence ladder (a
  state machine, not a label a model may assign). It also covers the central CWE policy, the
  project score and the self-improving loops (`ousast improve`, with `--memory` for proposals from
  plane rows).

Quick rules cover C/C++, Python, JavaScript/TypeScript, Java with Groovy templates, and PHP. Each
rule runs only against files of its language.

The detection gate (`uv run python -m openultrasast.gate`) enforces at least 90% recall and under
10% false positives per language on the bundled cheat-sheet corpora. On this tree it reports
93.62% recall (44/47) and 0.0% false positives. Those corpora show that a rule change did not
break a known detection. They are not a recall estimate on real code, and PHP is not in the gate.

Framework knowledge (WordPress, Flask, Django, Express, Spring) is tagged as an optional prior in
`src/openultrasast/ruleset/frameworks.toml`. Its contribution can be measured and switched off.

## The plane on ax

Model-driven work over whole repositories runs as a Run manifest. A Run is a DAG of ax Tasks. Each
Task runs in its own gVisor actor (an isolated sandbox) on Agent Substrate. ax is the only agentic
executor; there is no local subprocess path.

- Each Task binds at most one Model and its own `usd`/`calls` budget.
- Egress is deny-by-default per Task, by hostname only.
- The provider key travels only in the Task's start request, never in a manifest or an image.

Measured on the first increment: the plane agreed on 16 of 20 declared sites at $0.0205 per
candidate.

```bash
uv run ousast plane doctor                               # kind, Agent Substrate, ax controller, runner image
uv run ousast plane run plane/runs/validation-46.yaml    # a rerun skips tasks already done; --workers bounds tasks in flight
uv run ousast plane status validation-46                 # per-task status and the token attribution table
uv run ousast plane remember validation-46               # ingest the run's rows into the memory store
```

`workspaces`, `harvest`, `alerts-engine` and `memory-normalise` generate Runs, their inputs and
the store's rewrites. The flow and the task catalogue: [docs/plane.md](docs/plane.md). What runs
where today, and what a separate cluster would take: [docs/deployment.md](docs/deployment.md).

HarnessX, the earlier agentic extra, was retired 2026-09-30. A `[harnessx]` section is ignored
with one warning. `[models] verifier`, `[fusion] panel_model` and `[fusion] decider_model` fail and
name the plane replacement. Config keys that nothing ever read fail the same way. Fusion is
deterministic.

## Memory

One store holds what plane runs and the decision engine learned. It is keyed by repository and pin.
It holds two kinds of data:

- rows: facts, verdicts, unit costs, alerts, coverage and feature records;
- content-addressed blobs: excerpts, embeddings, cached responses and compiled programs.

`OUSAST_MEMORY` selects the backend. It is either a local file store or `s3://<bucket>` on any
S3-compatible server. The S3 server must support versioning, lifecycle rules, object tags and S3
Select. RustFS is the tested server.

An admin sets the bucket up once. The store verifies it at every start. If anything is missing, it
refuses to run and names each missing piece. It does not fall back.

- Layout and the read and write paths: [docs/memory.md](docs/memory.md).
- The bucket policy and admin steps: [docs/rustfs.md](docs/rustfs.md).
- How stored rows are meant to turn into better decisions:
  [docs/memory-and-detection.md](docs/memory-and-detection.md).

## Decision engine and evaluation

A learned decision engine is being built to replace hand-tuned detection edits. Its rules are:

- Every instrument produces signals, never verdicts.
- Labels come from ground truth only.
- A compiled AI classifier with local memory turns retrieved similar cases and the candidate's
  signals into a calibrated probability. It has BLOCK and ADVISORY operating points.

It is **in development, not adopted**. No scan, pre-push check or report uses it.

It was measured on six families, on repositories the compiled program was not shown:

- The pooled AUC (area under the ROC curve) of its score runs from 0.77 (injection) to 0.96
  (access control).
- The stricter within-pair AUC is 0.72 to 0.93 across the same families.
- BLOCK is offered for no family
  (`benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`).

Maintainer surface: `ousast learn labels|memory|compile|evaluate|curve|experiment|audit-leaks`.
The design and evaluation slices are in [docs/decision-engine.md](docs/decision-engine.md), along with the A/B protocol.

[docs/evaluation.md](docs/evaluation.md) explains how anything is measured on code the tool was not
tuned on. It covers the pair corpora, the independent populations and the M4 gates. It also covers
the fold discipline that keeps learned decisions out of their own training data. Population v3
(PHP, 15 cases) is frozen as the one-time final check. Nothing in the tree may read it.

```bash
uv run ousast benchmark benchmarks/manifests/python-vulnerable.toml --mode quick
uv run ousast pairs --slice sast                     # fire on the vulnerable side, stay silent on the fix
uv run python -m openultrasast.pair_gate             # CI: local pairs must all pass
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run --memory
```

## Token economics

Model tokens are the one running cost. At runtime the plane routes them:

- one Model and one budget per Task;
- a metered client that refuses the next call at the ceiling;
- repository facts computed without a model and reused;
- blobs replayed at $0, with no new model call;
- triage before the hunt;
- a tie-break only on disputed candidates.

In development, 85.5% of the maintainers' coding-session tokens were tool input and output rather
than reasoning. That is down from 90%. The remedies in `benchmarks/dev/` target that share.
Mechanisms, the measured costs and the before-and-after table:
[docs/token-ergonomics.md](docs/token-ergonomics.md).

## Examples, threat model

- End-to-end walkthroughs, from a quick scan to an MCP client: [docs/examples.md](docs/examples.md).
- What the tool trusts and what it does not: [docs/threat-model.md](docs/threat-model.md). This
  covers scanned code as untrusted input, secret redaction, per-task egress, budgets and the
  `deep` sandbox.

## Contributing and maintainer tooling

```bash
uv run pytest                 # tests, incl. the 90/10 detection gate
uv run ruff check .           # lint
uv run ruff format --check .  # format
uv run mypy src/openultrasast # types
uv run python -m openultrasast.gate       # standalone detection gate
uv run python -m openultrasast.pair_gate  # local vuln-vs-fix pairs
uv run python dagger/ci.py    # containerized CI pipeline
```

An agent that can run shell commands needs nothing but the CLI. For [OpenCode](https://opencode.ai)
there are two project skills in `.agents/skills/`:

- `openultrasast-scan`: modes and their prerequisites, `--fail-on`, the evidence ladder, recorded
  degradations and run artifacts.
- `openultrasast-triage`: false-positive reasons, what an adjudication may feed, and fixing and
  re-scanning.

`ousast mcp` runs a narrow MCP (Model Context Protocol) server over stdio. It has ten tools:
`openultrasast.scan`, `status`, `findings`, `get_finding`, `evidence`, `artifacts`, `benchmark`,
`explain`, `propose_patch` and `export_report`. No tool runs a shell, Docker or a free-form
command.

```jsonc
{ "command": "uv", "args": ["run", "ousast", "mcp"] }
```

The `kiro-*` skills, `AGENTS.md` and the Kiro-style specifications under `.kiro/` are the
maintainer's development tooling. Users can ignore them. Other maintainer commands:

- `ousast repos`: pinned known-vulnerable checkouts.
- `ousast model candidates`: what the candidate enumerator can reach.

## Licence

OpenUltraSAST is licensed under the [Apache License 2.0](LICENSE). Third-party material keeps its
own terms:

- Joern (Apache-2.0) is downloaded at image build time.
- The tree-sitter wheels of the `semantic` extra are MIT.
- The published benchmark cases were licence-checked one by one. v1's OpenCVE case is BUSL-1.1, see [RELEASE_NOTES.md](RELEASE_NOTES.md).

## Engine traces on kube-ax

`benchmarks/learn/engine_trace.py --executor ax` submits AX v0.3.1 Tasks with
`ax apply -f -` and removes them with `ax delete task NAME`. The operator supplies
kubeconfig and AX access; the dispatcher creates no Kubernetes resources or secrets.
The `ousast-engine-task` GHCR package **must be made Public once** in its package
settings so the actors can pull it without registry credentials.

The task image contains the current engine and worker code. Rebuild it after code
changes and pass its digest with `--image`; no analyzer download is needed. The host
prepares each pin as usual and uploads source and prepared questions under
`engine-queue/<run>/<pin>/`. Only object-scoped presigned URLs and deadline parameters
enter Task env. Credentials stay on the coordinator. Configure `OUSAST_MEMORY=s3://…`
and the store credentials in the coordinator's `.env` or environment, with
`S3_ENDPOINT=https://files.because-security.com`. Presigned links expire after the
pin deadline plus ten minutes. IP and HTTP endpoints are rejected: the operator's
actor EgressPolicy must allow files.because-security.com:443.

The result object marks completion, even while AX reports Running. Failed phases and
host deadlines produce explicit failure/timeout rows. Task and staging-object cleanup
runs on success and failure; cleanup failure stops further dispatch. Per-pin files,
`--rerun-status`, resume and `progress.json` retain the existing trace-runner behavior.
AX results include `executor: "ax"`, `worker_ip`, and the image digest. Placement is
random; `--parallel` defaults to two AX tasks and cannot exceed two.

AX workers use `--heap-profile cluster` with
`-Xms1700m -Xmx1700m -XX:MaxMetaspaceSize=256m -XX:ReservedCodeCacheSize=128m -XX:MaxDirectMemorySize=256m -Xss512k -XX:ActiveProcessorCount=1 -XX:+UseG1GC`
on every frontend/query JVM; instruments record the flags. A `JVM out of memory`
result is automatically re-queued to one local Docker lane with `--heap-profile vm`
(2560m), recording `fallback: "vm-oom"` and the AX attempt. `mixed-ax` also uses this
one Docker lane for overflow, prioritizing pending OOM fallbacks. Thus a mixed run has
up to two AX tasks plus one Docker container. Use `--docker-image` for the locally
available VM image; Docker never pulls it implicitly. A STOP file prevents new
launches; interrupted fallback queues are recovered on resume.

Coordinator commands, from this worktree (the AX binary and kubeconfig remain
configurable). Build/push the actual uncommitted tree and read its digest:

```bash
cd /home/mc/Source/OpenUltraSAST/.claude/worktrees/engine-ax
export PATH=/home/mc/Source/OpenUltraSAST/.venv/bin:$PATH
export PYTHONPATH="$PWD/src"
TASK_TAG="ghcr.io/norandom/ousast-engine-task:sha-$(git rev-parse --short=7 HEAD)"
docker buildx build --platform linux/amd64 -f plane/Dockerfile.engine-task \
  -t "$TASK_TAG" -t ghcr.io/norandom/ousast-engine-task:main \
  --push --metadata-file /tmp/ousast-engine-task-build.json .
export TASK_IMAGE="ghcr.io/norandom/ousast-engine-task@$(python -c 'import json; print(json.load(open("/tmp/ousast-engine-task-build.json"))["containerimage.digest"])')"
printf '%s\n' "$TASK_IMAGE"
docker build -f Dockerfile -t openultrasast:engine-ax .
```

The Engine image workflow also builds/pushes both images and prints each digest;
CI needs the changes published before dispatch. The local build above includes
uncommitted changes without publishing a git commit.

With `KUBECONFIG` pointing to the operator's kube-ax config and store settings in
`.env`, run a one-pin smoke using the default experiment units and local example
memory (or add `--examples /path/to/frozen-examples.jsonl`). Use the `ax` binary at
`$HOME/go/bin/ax`. `$HOME/.local/ax-v0.3.1/ax` hits the AX-lane dispatch blocker:

```bash
: "${KUBECONFIG:?Set KUBECONFIG to the operator kube-ax configuration}"
python benchmarks/learn/engine_trace.py --executor ax \
  --ax-bin "$HOME/go/bin/ax" --kubeconfig "$KUBECONFIG" \
  --atespace default --image "$TASK_IMAGE" --docker-image openultrasast:engine-ax \
  --parallel 1 --limit 1 --deadline 900 \
  --out "$HOME/ousast-results/plane/engine-ax-smoke"
```

Then run the mixed lanes (repeat to resume; defaults re-run failed/timeout pins):

```bash
python benchmarks/learn/engine_trace.py --executor mixed-ax \
  --ax-bin "$HOME/go/bin/ax" --kubeconfig "$KUBECONFIG" \
  --atespace default --image "$TASK_IMAGE" --docker-image openultrasast:engine-ax \
  --parallel 2 --deadline 900 --rerun-status failed,timeout \
  --out "$HOME/ousast-results/plane/engine-ax-mixed"
```