mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 11:49:28 +00:00
Marius CiepluchandClaude Opus 5.5
f4bd984fda
README: "Configuring the HarnessX agentic plane" becomes "The agentic plane (ax)" (ousast plane run|status|doctor|remember, per-task Model, budget and egress, pointer to ops/ax/README.md, one tagged retirement line); "The HarnessX self-improving cycle" becomes "The self-improving cycle", and its MetaAgent paragraph is replaced by the memory proposer (`ousast improve --memory`, provenance, train-on-test guard). The fusion panel_model/decider_model lines and the [harnessx] spend caps are gone. The README reference test now passes without its strict xfail. docs/examples.md section 6 runs the plane; docs/threat-model.md describes the per-task EgressPolicy, the usd/calls budgets and the start-request credential path. RELEASE_NOTES.md gets an Unreleased entry: HarnessX removed, the retired keys and their errors, the silent loss of the second hunter for users who set [models] hunter, and the plane as the agentic path. Dated retirement notes (2026-09-30) head the requirements, design and brief of harnessx-self-improving-rulesets, learning-harness, three-stage-scan and openrouter-sast-harness, and follow the spec table in steering/overview.md; history is unchanged. All tasks are checked, so spec.json moves to implementation-complete. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
90 lines
4.0 KiB
Markdown
90 lines
4.0 KiB
Markdown
# End-to-end examples
|
||
|
||
Each example is self-contained. Artifacts land under
|
||
`<target>/.openultrasast/runs/<scan-id>/` and `.openultrasast/` is gitignored.
|
||
|
||
## 1. Quick scan a local repo (deterministic, no keys)
|
||
|
||
```bash
|
||
uv run ousast scan /path/to/code --mode quick
|
||
run=/path/to/code/.openultrasast/runs/<scan-id>
|
||
cat "$run/report.md" # findings + evidence (secrets redacted)
|
||
jq . "$run/score.json" # 0–100 project score + gate verdict
|
||
"$run/report.sarif" # SARIF for code-scanning / IDEs
|
||
```
|
||
|
||
## 2. CI gate
|
||
|
||
```bash
|
||
# Fail the build on any evidence-verified finding:
|
||
uv run ousast scan . --mode quick --fail-on verified
|
||
|
||
# The hard detection gate (>=90% recall, <10% FP), score-independent:
|
||
uv run python -m openultrasast.gate
|
||
```
|
||
|
||
## 3. Benchmark a known-vulnerable corpus
|
||
|
||
```bash
|
||
uv run ousast benchmark benchmarks/manifests/python-vulnerable.toml --mode quick
|
||
# -> expected / matched / missed counts + a per-rule recommendation delta
|
||
```
|
||
|
||
## 4. Vuln vs fixed pair eval (efficiency, not cheat-sheet recall)
|
||
|
||
```bash
|
||
uv run ousast pairs # local fixtures + vendored GitHub VFCs
|
||
uv run ousast pairs --slice github --json # honesty dashboard as JSON
|
||
uv run ousast pairs --slice sast # OWASP Benchmark + Juliet Youden (TPR-FPR)
|
||
uv run ousast pairs --slice vibe-py --pointers # nightly: pointer pairs via the local cache (network)
|
||
uv run ousast mechanisms export --slice vibe-py # maintainer: seed mechanism candidates from trusted pairs (offline)
|
||
uv run ousast improve bench.toml --pair-catalog benchmarks/pairs/catalog.toml # the mechanisms lever admits candidates the holdout pairs prove; the scan then searches them
|
||
uv run ousast pairs --slice vibe-py --loo # leave-one-out: the corpus's own detection rate (reports/loo.json)
|
||
uv run python -m openultrasast.pair_gate # CI: local pairs must all pass
|
||
```
|
||
|
||
Each case is two isolated trees (vuln file vs patched file, same relative path).
|
||
`pair_correct` means the harness fired on the vuln side and stayed silent on the
|
||
fix. Misses and fix-side leaks become `miss`/`fp` signals for `ousast improve`.
|
||
The catalog and public dataset pointers live in `benchmarks/pairs/`.
|
||
|
||
## 5. Self-improve the ruleset from benchmark feedback
|
||
|
||
```bash
|
||
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run # preview
|
||
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml # apply
|
||
# Accepted edits land in <target>/.openultrasast/calibration/rule_policy.json,
|
||
# which the next scan/benchmark of that target loads automatically.
|
||
```
|
||
|
||
## 6. Running the agentic plane
|
||
|
||
Model work over whole repositories runs as a Run manifest on ax, one isolated actor
|
||
per task (bring-up: [ops/ax/README.md](../ops/ax/README.md)). The provider key is
|
||
read from the environment (or `.env`) by `ousast` and sent only in each task's start
|
||
request.
|
||
|
||
```bash
|
||
uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image
|
||
uv run ousast plane run plane/runs/validation-46.yaml # repo-facts, verify a/b(/c), agree, remember per case
|
||
uv run ousast plane status validation-46 # per-task status and token attribution
|
||
uv run ousast plane remember validation-46 # re-ingest the run's memory rows (OUSAST_MEMORY)
|
||
|
||
# Feed the stored rows back into the ruleset loop (same validator and gate)
|
||
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run --memory
|
||
```
|
||
|
||
Each task in the Run carries its own Model and `budget: {usd, calls}`; a task that
|
||
reaches its ceiling stops as `unfinished` and resumes on a rerun with a larger
|
||
budget. A local `ousast scan --mode standard` stays deterministic apart from the
|
||
optional `[models] hunter` tool hunter.
|
||
|
||
## 7. Drive it from an MCP client (OpenCode / IDE)
|
||
|
||
```jsonc
|
||
{ "command": "uv", "args": ["run", "ousast", "mcp"] }
|
||
```
|
||
|
||
Then: `openultrasast.scan {path}` → `run_dir` → `openultrasast.findings {run_dir}` →
|
||
`openultrasast.explain {run_dir, finding_id}`. See the README "MCP server" section.
|