feat(harnessx): configurable provider + per-task budget caps

Add a [harnessx] config block so the agentic plane can be steered from
openultrasast.toml, and make the LLM provider selectable (the hunter and verifier
previously hardcoded AnthropicProvider).

- config.HarnessxConfig: provider ("anthropic"|"openai"|"litellm", default
  anthropic), max_cost_usd (2.0), token_threshold (120000); parsed from [harnessx].
- harness_ext.build_provider: single lazy, capability-guarded provider factory.
  The provider name is validated before any import, so an unknown choice fails with
  a clear ValueError even when the extra is absent; each provider reads its own
  standard key env var (openai -> OPENAI_API_KEY, anthropic -> ANTHROPIC_API_KEY).
- HxScanOrchestrator and verify_findings_dispatch take a provider and route through
  build_provider; cli threads config.harnessx.{provider,max_cost_usd,token_threshold}
  into both the hunter and the verifier.
- README: "Configuring the HarnessX agentic plane" section (install, models,
  provider, budgets, fallback/degradation).

Test isolation: convert the "harnessx not in sys.modules" import-guards to a
hermetic subprocess fixture (tests/conftest.py assert_cold_of_harnessx) and
simulate extra-absent in the CLI dispatch tests, so the suite is deterministic
whether or not the optional extra is installed (it now is, locally). Verified the
OpenAI path constructs end-to-end (build_provider -> OpenAIProvider; orchestrator
-> Harness) with the budget threaded through.

137 passed, 3 skipped; ruff + mypy clean; detection gate PASS.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-06-22 17:59:44 +02:00
co-authored by Claude Opus 4.8
parent 760df7c809
commit e255eb264f
12 changed files with 240 additions and 40 deletions
+37
View File
@@ -243,6 +243,43 @@ addressed, it stays closed because the recall/precision gate runs in CI.
`external_baseline_deltas.json` additionally shows where another tool found a
vulnerability OpenUltraSAST missed (or vice-versa).
## Configuring the HarnessX agentic plane
The LLM hunter pool and the llm-judge verifier are optional. The core install is
zero-dependency and `--mode quick` never calls a model. To enable the agentic
plane, install the extra, name the models, and pick a provider:
```bash
uv sync --extra harnessx # SHA-pinned optional dependency
export OPENAI_API_KEY=sk-... # or ANTHROPIC_API_KEY, matching the provider
uv run ousast scan /path/to/code --mode standard --config openultrasast.toml
```
```toml
# openultrasast.toml
[models]
hunter = "gpt-4o" # LLM hunter pool (empty -> deterministic hunter)
verifier = "gpt-4o" # llm-judge verifier (empty -> structural verifier)
[harnessx]
provider = "openai" # "anthropic" (default) | "openai" | "litellm"
max_cost_usd = 2.0 # per-task spend cap
token_threshold = 120000 # per-task token budget
```
The provider selects the HarnessX backend for **both** the hunter and the
verifier; each reads its own standard key env var (`openai` → `OPENAI_API_KEY`,
`anthropic` → `ANTHROPIC_API_KEY`). The HarnessX path activates only when the
extra is installed, a model is configured, and the mode is `standard` — all three
gated through one capability seam (`harness_ext.has_harnessx`). If any condition
fails, the scan transparently falls back to the deterministic hunter / structural
verifier and records a `degradations` entry in `manifest.json`, so a run is never
silently downgraded:
```bash
jq '.degradations' "$run/manifest.json" # null = HarnessX ran; entries = fell back
```
## The HarnessX self-improving cycle
Each scan is a composed harness of typed processors with read/write **state