docs: describe the system as of 2026-10-02 (ax plane, AI decision engine, PHP, v3)

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-10-02 07:08:44 +02:00
co-authored by Claude Opus 5.5
parent 7b17eacd44
commit 32a2cb8bcf
11 changed files with 715 additions and 405 deletions
+24 -15
View File
@@ -13,7 +13,18 @@ jq . "$run/score.json" # 0–100 project score + gate verdict
"$run/report.sarif" # SARIF for code-scanning / IDEs
```
## 2. CI gate
## 2. Standard scan with the engine (Docker)
```bash
source ops/shell/ousast.sh # once per shell; builds the image on first use
ousast scan . --mode standard # Joern model layer, container network off: no model calls
ousast-with-judge scan . --mode standard # network on, so the model is asked (needs DEEPSEEK_API_KEY)
```
Without a key the graph still decides; only the `suspicion` band goes unasked, and the manifest
records the degradation.
## 3. CI gate
```bash
# Fail the build on any evidence-verified finding:
@@ -23,23 +34,21 @@ uv run ousast scan . --mode quick --fail-on verified
uv run python -m openultrasast.gate
```
## 3. Benchmark a known-vulnerable corpus
## 4. Benchmark a known-vulnerable corpus
```bash
uv run ousast benchmark benchmarks/manifests/python-vulnerable.toml --mode quick
# -> expected / matched / missed counts + a per-rule recommendation delta
```
## 4. Vuln vs fixed pair eval (efficiency, not cheat-sheet recall)
## 5. Vuln vs fixed pair eval (efficiency, not cheat-sheet recall)
```bash
uv run ousast pairs # local fixtures + vendored GitHub VFCs
uv run ousast pairs --slice github --json # honesty dashboard as JSON
uv run ousast pairs --slice sast # OWASP Benchmark + Juliet Youden (TPR-FPR)
uv run ousast pairs --slice vibe-py --pointers # nightly: pointer pairs via the local cache (network)
uv run ousast mechanisms export --slice vibe-py # maintainer: seed mechanism candidates from trusted pairs (offline)
uv run ousast improve bench.toml --pair-catalog benchmarks/pairs/catalog.toml # the mechanisms lever admits candidates the holdout pairs prove; the scan then searches them
uv run ousast pairs --slice vibe-py --loo # leave-one-out: the corpus's own detection rate (reports/loo.json)
uv run ousast pairs --split holdout --profile human # filter by declared split and provenance profile
uv run python -m openultrasast.pair_gate # CI: local pairs must all pass
```
@@ -48,7 +57,7 @@ Each case is two isolated trees (vuln file vs patched file, same relative path).
fix. Misses and fix-side leaks become `miss`/`fp` signals for `ousast improve`.
The catalog and public dataset pointers live in `benchmarks/pairs/`.
## 5. Self-improve the ruleset from benchmark feedback
## 6. Self-improve the ruleset from benchmark feedback
```bash
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run # preview
@@ -57,16 +66,16 @@ uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml
# which the next scan/benchmark of that target loads automatically.
```
## 6. Running the agentic plane
## 7. Running the agentic plane
Model work over whole repositories runs as a Run manifest on ax, one isolated actor
per task (bring-up: [ops/ax/README.md](../ops/ax/README.md)). The provider key is
read from the environment (or `.env`) by `ousast` and sent only in each task's start
request.
read from the environment (or `.env`, which never overrides an exported variable) by
`ousast` and sent only in each task's start request.
```bash
uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image
uv run ousast plane run plane/runs/validation-46.yaml # repo-facts, verify a/b(/c), agree, remember per case
uv run ousast plane run plane/runs/validation-46.yaml # per case: repo-facts, verify a/b/c, agree, final, features, remember
uv run ousast plane status validation-46 # per-task status and token attribution
uv run ousast plane remember validation-46 # re-ingest the run's memory rows (OUSAST_MEMORY)
@@ -76,14 +85,14 @@ uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dr
Each task in the Run carries its own Model and `budget: {usd, calls}`; a task that
reaches its ceiling stops as `unfinished` and resumes on a rerun with a larger
budget. A local `ousast scan --mode standard` stays deterministic apart from the
optional `[models] hunter` tool hunter.
budget. `ousast plane status` prints the token attribution table: calls, prompt,
cache-hit and output tokens and USD per task.
## 7. Drive it from an MCP client (OpenCode / IDE)
## 8. Drive it from an MCP client (OpenCode / IDE)
```jsonc
{ "command": "uv", "args": ["run", "ousast", "mcp"] }
```
Then: `openultrasast.scan {path}` → `run_dir` → `openultrasast.findings {run_dir}` →
`openultrasast.explain {run_dir, finding_id}`. See the README "MCP server" section.
`openultrasast.explain {run_dir, finding_id}`. See the README section "Agent integrations".