mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 11:49:28 +00:00

Files

T

Marius CiepluchandClaude Opus 5.5 f4bd984fda docs(harnessx-removal): the plane replaces HarnessX in README, docs and specs (task 8)

README: "Configuring the HarnessX agentic plane" becomes "The agentic
plane (ax)" (ousast plane run|status|doctor|remember, per-task Model,
budget and egress, pointer to ops/ax/README.md, one tagged retirement
line); "The HarnessX self-improving cycle" becomes "The self-improving
cycle", and its MetaAgent paragraph is replaced by the memory proposer
(`ousast improve --memory`, provenance, train-on-test guard). The fusion
panel_model/decider_model lines and the [harnessx] spend caps are gone.
The README reference test now passes without its strict xfail.

docs/examples.md section 6 runs the plane; docs/threat-model.md describes
the per-task EgressPolicy, the usd/calls budgets and the start-request
credential path. RELEASE_NOTES.md gets an Unreleased entry: HarnessX
removed, the retired keys and their errors, the silent loss of the
second hunter for users who set [models] hunter, and the plane as the
agentic path.

Dated retirement notes (2026-09-30) head the requirements, design and
brief of harnessx-self-improving-rulesets, learning-harness,
three-stage-scan and openrouter-sast-harness, and follow the spec table
in steering/overview.md; history is unchanged. All tasks are checked, so
spec.json moves to implementation-complete.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo

2026-09-30 12:45:29 +02:00

4.0 KiB

Raw Blame History

End-to-end examples

Each example is self-contained. Artifacts land under <target>/.openultrasast/runs/<scan-id>/ and .openultrasast/ is gitignored.

1. Quick scan a local repo (deterministic, no keys)

uv run ousast scan /path/to/code --mode quick
run=/path/to/code/.openultrasast/runs/<scan-id>
cat  "$run/report.md"       # findings + evidence (secrets redacted)
jq . "$run/score.json"      # 0–100 project score + gate verdict
"$run/report.sarif"         # SARIF for code-scanning / IDEs

2. CI gate

# Fail the build on any evidence-verified finding:
uv run ousast scan . --mode quick --fail-on verified

# The hard detection gate (>=90% recall, <10% FP), score-independent:
uv run python -m openultrasast.gate

3. Benchmark a known-vulnerable corpus

uv run ousast benchmark benchmarks/manifests/python-vulnerable.toml --mode quick
# -> expected / matched / missed counts + a per-rule recommendation delta

4. Vuln vs fixed pair eval (efficiency, not cheat-sheet recall)

uv run ousast pairs                              # local fixtures + vendored GitHub VFCs
uv run ousast pairs --slice github --json        # honesty dashboard as JSON
uv run ousast pairs --slice sast                 # OWASP Benchmark + Juliet Youden (TPR-FPR)
uv run ousast pairs --slice vibe-py --pointers   # nightly: pointer pairs via the local cache (network)
uv run ousast mechanisms export --slice vibe-py   # maintainer: seed mechanism candidates from trusted pairs (offline)
uv run ousast improve bench.toml --pair-catalog benchmarks/pairs/catalog.toml   # the mechanisms lever admits candidates the holdout pairs prove; the scan then searches them
uv run ousast pairs --slice vibe-py --loo        # leave-one-out: the corpus's own detection rate (reports/loo.json)
uv run python -m openultrasast.pair_gate         # CI: local pairs must all pass

Each case is two isolated trees (vuln file vs patched file, same relative path). pair_correct means the harness fired on the vuln side and stayed silent on the fix. Misses and fix-side leaks become miss/fp signals for ousast improve. The catalog and public dataset pointers live in benchmarks/pairs/.

5. Self-improve the ruleset from benchmark feedback

uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run  # preview
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml            # apply
# Accepted edits land in <target>/.openultrasast/calibration/rule_policy.json,
# which the next scan/benchmark of that target loads automatically.

6. Running the agentic plane

Model work over whole repositories runs as a Run manifest on ax, one isolated actor per task (bring-up: ops/ax/README.md). The provider key is read from the environment (or .env) by ousast and sent only in each task's start request.

uv run ousast plane doctor                               # kind, Agent Substrate, ax controller, runner image
uv run ousast plane run plane/runs/validation-46.yaml    # repo-facts, verify a/b(/c), agree, remember per case
uv run ousast plane status validation-46                 # per-task status and token attribution
uv run ousast plane remember validation-46               # re-ingest the run's memory rows (OUSAST_MEMORY)

# Feed the stored rows back into the ruleset loop (same validator and gate)
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run --memory

Each task in the Run carries its own Model and budget: {usd, calls}; a task that reaches its ceiling stops as unfinished and resumes on a rerun with a larger budget. A local ousast scan --mode standard stays deterministic apart from the optional [models] hunter tool hunter.

7. Drive it from an MCP client (OpenCode / IDE)

{ "command": "uv", "args": ["run", "ousast", "mcp"] }

Then: openultrasast.scan {path} → run_dir → openultrasast.findings {run_dir} → openultrasast.explain {run_dir, finding_id}. See the README "MCP server" section.

4.0 KiB Raw Blame History Unescape Escape