mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
feat(harnessx): configurable provider + per-task budget caps
Add a [harnessx] config block so the agentic plane can be steered from
openultrasast.toml, and make the LLM provider selectable (the hunter and verifier
previously hardcoded AnthropicProvider).
- config.HarnessxConfig: provider ("anthropic"|"openai"|"litellm", default
anthropic), max_cost_usd (2.0), token_threshold (120000); parsed from [harnessx].
- harness_ext.build_provider: single lazy, capability-guarded provider factory.
The provider name is validated before any import, so an unknown choice fails with
a clear ValueError even when the extra is absent; each provider reads its own
standard key env var (openai -> OPENAI_API_KEY, anthropic -> ANTHROPIC_API_KEY).
- HxScanOrchestrator and verify_findings_dispatch take a provider and route through
build_provider; cli threads config.harnessx.{provider,max_cost_usd,token_threshold}
into both the hunter and the verifier.
- README: "Configuring the HarnessX agentic plane" section (install, models,
provider, budgets, fallback/degradation).
Test isolation: convert the "harnessx not in sys.modules" import-guards to a
hermetic subprocess fixture (tests/conftest.py assert_cold_of_harnessx) and
simulate extra-absent in the CLI dispatch tests, so the suite is deterministic
whether or not the optional extra is installed (it now is, locally). Verified the
OpenAI path constructs end-to-end (build_provider -> OpenAIProvider; orchestrator
-> Harness) with the budget threaded through.
137 passed, 3 skipped; ruff + mypy clean; detection gate PASS.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
760df7c809
commit
e255eb264f
@@ -243,6 +243,43 @@ addressed, it stays closed because the recall/precision gate runs in CI.
|
||||
`external_baseline_deltas.json` additionally shows where another tool found a
|
||||
vulnerability OpenUltraSAST missed (or vice-versa).
|
||||
|
||||
## Configuring the HarnessX agentic plane
|
||||
|
||||
The LLM hunter pool and the llm-judge verifier are optional. The core install is
|
||||
zero-dependency and `--mode quick` never calls a model. To enable the agentic
|
||||
plane, install the extra, name the models, and pick a provider:
|
||||
|
||||
```bash
|
||||
uv sync --extra harnessx # SHA-pinned optional dependency
|
||||
export OPENAI_API_KEY=sk-... # or ANTHROPIC_API_KEY, matching the provider
|
||||
uv run ousast scan /path/to/code --mode standard --config openultrasast.toml
|
||||
```
|
||||
|
||||
```toml
|
||||
# openultrasast.toml
|
||||
[models]
|
||||
hunter = "gpt-4o" # LLM hunter pool (empty -> deterministic hunter)
|
||||
verifier = "gpt-4o" # llm-judge verifier (empty -> structural verifier)
|
||||
|
||||
[harnessx]
|
||||
provider = "openai" # "anthropic" (default) | "openai" | "litellm"
|
||||
max_cost_usd = 2.0 # per-task spend cap
|
||||
token_threshold = 120000 # per-task token budget
|
||||
```
|
||||
|
||||
The provider selects the HarnessX backend for **both** the hunter and the
|
||||
verifier; each reads its own standard key env var (`openai` → `OPENAI_API_KEY`,
|
||||
`anthropic` → `ANTHROPIC_API_KEY`). The HarnessX path activates only when the
|
||||
extra is installed, a model is configured, and the mode is `standard` — all three
|
||||
gated through one capability seam (`harness_ext.has_harnessx`). If any condition
|
||||
fails, the scan transparently falls back to the deterministic hunter / structural
|
||||
verifier and records a `degradations` entry in `manifest.json`, so a run is never
|
||||
silently downgraded:
|
||||
|
||||
```bash
|
||||
jq '.degradations' "$run/manifest.json" # null = HarnessX ran; entries = fell back
|
||||
```
|
||||
|
||||
## The HarnessX self-improving cycle
|
||||
|
||||
Each scan is a composed harness of typed processors with read/write **state
|
||||
|
||||
Reference in New Issue
Block a user