Add a [harnessx] config block so the agentic plane can be steered from
openultrasast.toml, and make the LLM provider selectable (the hunter and verifier
previously hardcoded AnthropicProvider).
- config.HarnessxConfig: provider ("anthropic"|"openai"|"litellm", default
anthropic), max_cost_usd (2.0), token_threshold (120000); parsed from [harnessx].
- harness_ext.build_provider: single lazy, capability-guarded provider factory.
The provider name is validated before any import, so an unknown choice fails with
a clear ValueError even when the extra is absent; each provider reads its own
standard key env var (openai -> OPENAI_API_KEY, anthropic -> ANTHROPIC_API_KEY).
- HxScanOrchestrator and verify_findings_dispatch take a provider and route through
build_provider; cli threads config.harnessx.{provider,max_cost_usd,token_threshold}
into both the hunter and the verifier.
- README: "Configuring the HarnessX agentic plane" section (install, models,
provider, budgets, fallback/degradation).
Test isolation: convert the "harnessx not in sys.modules" import-guards to a
hermetic subprocess fixture (tests/conftest.py assert_cold_of_harnessx) and
simulate extra-absent in the CLI dispatch tests, so the suite is deterministic
whether or not the optional extra is installed (it now is, locally). Verified the
OpenAI path constructs end-to-end (build_provider -> OpenAIProvider; orchestrator
-> Harness) with the budget threaded through.
137 passed, 3 skipped; ruff + mypy clean; detection gate PASS.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Expose the bounded, gated self-improvement loop as a first-class subcommand so it
can be steered from the command line / opencode.
- `ousast improve <manifest>` resolves the target from the benchmark manifest,
runs the loop to convergence, and writes accepted edits to the loop-owned ledger
at <target>/.openultrasast/calibration/rule_policy.json — exactly where `scan`
and `benchmark` read it, so a subsequent scan of that target automatically
picks up the improved ruleset (the loop closes end-to-end).
- Flags: --max-rounds, --recall-floor, --fp-ceiling, --ledger, --journal,
--ruleset-dir, and --dry-run (runs against a throwaway ledger, touches nothing).
- Per-round output: reason, recall/fp/score deltas, and the edits applied.
- tests/test_cli_improve.py: accept path writes the loop-owned ledger (noisy rule
auto-shadowed), --dry-run never touches it, missing manifest fails loudly.
- README: quickstart entry + a "Rule-level loop: ousast improve" subsection
documenting the bounded levers, hard acceptance gate, and the HarnessX MetaAgent
as the optional proposer over the same gated machinery.
131 passed, 3 skipped; ruff + mypy clean; detection gate PASS.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
CI: the dagger job repeatedly failed (and once hung 10 min) pulling the engine
image from registry.dagger.io. Pin the dagger CLI to 0.21.7 (matching dagger-io
and the SDK's CLI_VERSION so the engine tag is deterministic), pre-pull
registry.dagger.io/engine:v0.21.7 with a dependency-free retry loop (5 attempts,
90s/pull, 20s backoff) so the action's `docker image inspect` finds it cached and
skips its own flaky pull, and add a 20-minute job timeout so a stalled pull can
no longer hang the run. The canonical `checks` job is unchanged.
Docs: add a "Project scores and central CWE policy" README section documenting
the implemented Phase 1 — central verycode CWE policy, policy-governed severity
(one CWE = one severity), the fail-loud invariant, the 0-100 project score and
two-condition advisory-first gate, the score.json/manifest artifacts, and the
CWE-120 -> CWE-121 remap.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Close the HarnessX self-improving loop without an agent in the loop. Every scan
now feeds its verifier outcomes back into the next scan's ranking:
- New `calibrate` stage (after rank, before findings) loads a persistent ledger
of scoped false-positive learnings and demotes every previously-rejected scope
(`calibrate_rankings`), writing `calibration/applied_calibrations.json`.
- New `record_calibration` stage (after verify) converts this run's non-accepted
verifier outcomes into scoped `FalsePositiveLearning` records
(`learnings_from_verifications`) and merges them into
`.openultrasast/calibration/false_positive_learnings.json`, de-duplicated by
finding ID so a repeat rejection does not compound without bound.
- Lower the calibration floor (MIN_CALIBRATED_PRIORITY = 0.1) so accumulated
rejections can demote a scope below the natural ranking minimum.
Demotion is scoped and reversible: a rejection in one path never touches an
accepted finding in another, and priority is demoted rather than the finding
deleted so the audit trail explains every downgrade. First run is a no-op
(empty ledger), so existing single-run behavior is unchanged.
Adds tests/test_pipeline_calibration.py (two-run demotion, order-independent),
documents the realized loop in the README, and checks off the
standard_security_harness milestone in tasks.md.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
- OpenUltraSAST is driven from the command line (ousast CLI, or OpenCode invoked
on the repo running the harness for you); show both.
- Mark the kiro-* skills and AGENTS.md as maintainer-only SDLC tooling, not part
of using OpenUltraSAST; drop them from the user-facing skill list.
- State that OpenCode is the agent steering the HarnessX self-improving cycle
today (reads verifier/benchmark signals, applies scoped calibration via the
triage skill, re-runs the gate); autonomous in-pipeline feedback is the next
milestone.
- Remove a duplicated Fusion section.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Expand the README with: how to run the per-language benchmarks and the latest
measured results (C/C++ 93.3%, Python 93.3%, JavaScript 92.3%, Java 100%;
overall 93.6% recall, 0% false positives); the OpenCode skills, the ultra
(OpenUltraCode) fix lifecycle, and fusion deepening; how a false positive is
eliminated via the evidence-ladder verifier and scoped calibration; how a false
negative becomes a benchmark calibration record routed to a next-improvement
candidate; and how the HarnessX self-improving cycle is realized. Maturity is
marked (implemented / primitive / roadmap) to stay accurate to the current gate.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
