mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 11:49:28 +00:00
feat(harnessx): configurable provider + per-task budget caps
Add a [harnessx] config block so the agentic plane can be steered from
openultrasast.toml, and make the LLM provider selectable (the hunter and verifier
previously hardcoded AnthropicProvider).
- config.HarnessxConfig: provider ("anthropic"|"openai"|"litellm", default
anthropic), max_cost_usd (2.0), token_threshold (120000); parsed from [harnessx].
- harness_ext.build_provider: single lazy, capability-guarded provider factory.
The provider name is validated before any import, so an unknown choice fails with
a clear ValueError even when the extra is absent; each provider reads its own
standard key env var (openai -> OPENAI_API_KEY, anthropic -> ANTHROPIC_API_KEY).
- HxScanOrchestrator and verify_findings_dispatch take a provider and route through
build_provider; cli threads config.harnessx.{provider,max_cost_usd,token_threshold}
into both the hunter and the verifier.
- README: "Configuring the HarnessX agentic plane" section (install, models,
provider, budgets, fallback/degradation).
Test isolation: convert the "harnessx not in sys.modules" import-guards to a
hermetic subprocess fixture (tests/conftest.py assert_cold_of_harnessx) and
simulate extra-absent in the CLI dispatch tests, so the suite is deterministic
whether or not the optional extra is installed (it now is, locally). Verified the
OpenAI path constructs end-to-end (build_provider -> OpenAIProvider; orchestrator
-> Harness) with the budget threaded through.
137 passed, 3 skipped; ruff + mypy clean; detection gate PASS.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
760df7c809
commit
e255eb264f
@@ -1,8 +1,10 @@
|
||||
"""CLI capability-gated dispatch + graceful degradation (Phase 3 task 6.5).
|
||||
|
||||
These run in the default (extra-absent) environment: standard mode with models
|
||||
configured must fall back to the deterministic hunter/structural verifier and
|
||||
record a degradation; with no models configured the manifest stays byte-identical.
|
||||
Simulates the extra-absent environment (via a monkeypatched capability probe so the
|
||||
behaviour is deterministic whether or not the optional extra is installed): standard
|
||||
mode with models configured must fall back to the deterministic hunter/structural
|
||||
verifier and record a degradation; with no models configured the manifest stays
|
||||
byte-identical.
|
||||
"""
|
||||
|
||||
import json
|
||||
@@ -20,6 +22,7 @@ def _run(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, config_body: str | Non
|
||||
repo = tmp_path / "repo"
|
||||
repo.mkdir()
|
||||
(repo / "app.py").write_text(_VULN)
|
||||
monkeypatch.setattr(cli, "has_harnessx", lambda: False) # simulate the extra being absent
|
||||
monkeypatch.setenv("OPENULTRASAST_RUNS_DIR", ".runs")
|
||||
argv = ["scan", str(repo), "--mode", "standard"]
|
||||
if config_body is not None:
|
||||
@@ -32,7 +35,6 @@ def _run(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, config_body: str | Non
|
||||
|
||||
|
||||
def test_degradation_recorded_when_models_set_but_extra_absent(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
assert not cli.has_harnessx() # this CI env has no harnessx extra
|
||||
manifest = _run(tmp_path, monkeypatch, '[models]\nhunter = "anthropic/claude"\nverifier = "anthropic/claude"\n')
|
||||
|
||||
degradations = manifest.get("degradations", [])
|
||||
|
||||
Reference in New Issue
Block a user