feat(cli): add ousast improve — run the self-improvement loop from the CLI

Expose the bounded, gated self-improvement loop as a first-class subcommand so it
can be steered from the command line / opencode.

- `ousast improve <manifest>` resolves the target from the benchmark manifest,
  runs the loop to convergence, and writes accepted edits to the loop-owned ledger
  at <target>/.openultrasast/calibration/rule_policy.json — exactly where `scan`
  and `benchmark` read it, so a subsequent scan of that target automatically
  picks up the improved ruleset (the loop closes end-to-end).
- Flags: --max-rounds, --recall-floor, --fp-ceiling, --ledger, --journal,
  --ruleset-dir, and --dry-run (runs against a throwaway ledger, touches nothing).
- Per-round output: reason, recall/fp/score deltas, and the edits applied.
- tests/test_cli_improve.py: accept path writes the loop-owned ledger (noisy rule
  auto-shadowed), --dry-run never touches it, missing manifest fails loudly.
- README: quickstart entry + a "Rule-level loop: ousast improve" subsection
  documenting the bounded levers, hard acceptance gate, and the HarnessX MetaAgent
  as the optional proposer over the same gated machinery.

131 passed, 3 skipped; ruff + mypy clean; detection gate PASS.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-06-22 17:36:58 +02:00
co-authored by Claude Opus 4.8
parent 649bd42fc3
commit 760df7c809
3 changed files with 199 additions and 5 deletions
+44 -5
View File
@@ -29,6 +29,9 @@ uv run ousast index /path/to/repo
# Score the detector against a benchmark manifest
uv run ousast benchmark benchmarks/manifests/python-vulnerable.toml --mode quick
# Run the bounded self-improvement loop over the ruleset (benchmark-driven)
uv run ousast improve benchmarks/manifests/python-vulnerable.toml --dry-run
```
Every run writes auditable artifacts under `.openultrasast/runs/<scan-id>/`
@@ -273,11 +276,47 @@ priority is demoted rather than the finding deleted, the audit trail
(`applied_calibrations.json`, the ledger) shows exactly why a surface lost
attention. Covered by `tests/test_pipeline_calibration.py`.
Beyond the automatic ranking loop, `openultrasast-triage` still lets OpenCode
adjust prompt constraints, retrieval filters, skill routing, and benchmark-miss
triage. Those `RankingCalibration` fields are already produced and will be
consumed directly once the language-aware LLM hunters land
(`standard_security_harness`).
### Rule-level loop: `ousast improve`
The ranking loop above adjusts *attention*. A second, bounded loop adjusts the
**governed ruleset itself** from benchmark feedback — and writes its result to the
same loop-owned ledger a scan reads, so the loop closes end-to-end:
```bash
# Preview what the loop would change (writes nothing)
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run
# Apply: accepted rounds land in <target>/.openultrasast/calibration/rule_policy.json,
# which the next `ousast scan`/`benchmark` of that target loads automatically.
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml
```
Each round (`improve/evolve.py`): benchmark with the current ledger → per-rule
signals (`rule_signals.json`) → propose **bounded** edits → validate → replay
smoke → re-benchmark → a **hard acceptance gate** → accept or revert
byte-for-byte. The loop may only pull two levers (rule status, score constants);
it can never edit a rule's pattern text or the authoritative 0–5 CWE severity, and
must stage `enabled → shadow` before `disabled`. A round is accepted only if
**recall ≥ 90% AND FP < 10% AND the project score did not regress AND no matched
finding was lost** — otherwise the ledger is never written. Auto-shadowing removes
a precision-dragger's false-positive cost without deleting it (recall is
preserved); persistent misses are *nominated* through the signal file, leaving
pattern authoring to a human pull request. The project score is the optimization
reward; the recall/FP gate is a separate hard constraint that is never folded into
it — and the same gate runs in CI (`python -m openultrasast.gate`), so a loop
result that breached it could never merge. Covered by `tests/test_improve.py` and
`tests/test_cli_improve.py`.
This is the deterministic substrate of the self-improvement loop. When the
optional `openultrasast[harnessx]` extra is present, the HarnessX `MetaAgent`
becomes the richer *proposer* that plugs into this same validated, gated
machinery — the safety contract (bounded levers, replay/novelty/journal gates,
hard acceptance gate, byte-for-byte revert) is identical either way.
Beyond these loops, `openultrasast-triage` still lets OpenCode adjust prompt
constraints, retrieval filters, skill routing, and benchmark-miss triage. Those
`RankingCalibration` fields are already produced and will be consumed directly
once the language-aware LLM hunters land (`standard_security_harness`).
## Development