feat(calibration): automatic in-pipeline false-positive feedback

Close the HarnessX self-improving loop without an agent in the loop. Every scan
now feeds its verifier outcomes back into the next scan's ranking:

- New `calibrate` stage (after rank, before findings) loads a persistent ledger
  of scoped false-positive learnings and demotes every previously-rejected scope
  (`calibrate_rankings`), writing `calibration/applied_calibrations.json`.
- New `record_calibration` stage (after verify) converts this run's non-accepted
  verifier outcomes into scoped `FalsePositiveLearning` records
  (`learnings_from_verifications`) and merges them into
  `.openultrasast/calibration/false_positive_learnings.json`, de-duplicated by
  finding ID so a repeat rejection does not compound without bound.
- Lower the calibration floor (MIN_CALIBRATED_PRIORITY = 0.1) so accumulated
  rejections can demote a scope below the natural ranking minimum.

Demotion is scoped and reversible: a rejection in one path never touches an
accepted finding in another, and priority is demoted rather than the finding
deleted so the audit trail explains every downgrade. First run is a no-op
(empty ledger), so existing single-run behavior is unchanged.

Adds tests/test_pipeline_calibration.py (two-run demotion, order-independent),
documents the realized loop in the README, and checks off the
standard_security_harness milestone in tasks.md.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-06-21 20:44:08 +02:00
co-authored by Claude Opus 4.8
parent d9eff9aade
commit 9c4af4f546
5 changed files with 213 additions and 30 deletions
+29 -25
View File
@@ -219,37 +219,41 @@ vulnerability OpenUltraSAST missed (or vice-versa).
Each scan is a composed harness of typed processors with read/write **state
contracts** (strict mode fails the scan on a violation; warn mode records a
degradation) and a full event trace. That trace plus the verifier and benchmark
outputs are the feedback that tunes the *next* run — and **OpenCode is the agent
that steers it**: running the triage and fix skills, it reads the harness's
signals and applies the adjustments back into config.
degradation) and a full event trace. The loop is now **closed automatically
inside the pipeline** — no agent required: every scan persists its verifier
rejections as scoped learnings and the next scan loads them and demotes those
scopes before findings are produced.
```
┌──────────────────── OpenCode steers (triage / fix skills) ─────┐
┌──────────── automatic ledger (no agent in the loop) ───────────┐
▼ │
scan ──▶ traces (events.jsonl) + verifier decisions + benchmark misses │
│ │
├─ rejected findings ─▶ scoped false-positive learnings ─────────┤
├─ verifier outcomes ─▶ ranking calibration / demotion ──────────┤ applied
├─ benchmark misses ─▶ calibration records (next candidate) ─────┤ back
└─ ranking metrics ─▶ retrieval filters · prompt constraints ───┘ into
· skill routing · variant seeds config
scan ──▶ verify ──▶ non-accepted findings ──▶ scoped learnings ─────────┤
│ (.openultrasast/calibration/ │ loaded by
│ false_positive_learnings.json) │ the next
next scan ──▶ rank ──▶ calibrate (demote rejected scopes) ──▶ findings ──┘ scan
```
What the harness exposes today (✅): the event/trace model and processor
contracts; the evidence-ladder verifier; the scoped false-positive learning
taxonomy with ranking calibration and prompt/retrieval-filter adjustments
(`calibration.py`); ranking metrics by tier; and benchmark miss records.
What runs automatically each scan (✅, `calibration.py` + `cli.py`):
**How the loop is closed today** — OpenCode is the controller. Invoked from the
command line on the repo, it runs a scan, reads the verifier decisions and
benchmark `calibration_records.json`, and via `openultrasast-triage` records the
scoped learnings and recalibrates ranking, then re-runs the benchmark gate to
confirm the change helped without regressing other findings. So the
self-improving cycle is real now with OpenCode as the steering agent; making the
harness apply that feedback **autonomously inside the pipeline** — recalibrating
on the following run without an agent in the loop — is the next milestone
(`standard_security_harness`).
1. **`calibrate` stage** (after `rank`, before findings) loads the persistent
ledger and demotes the priority of every scope that previously produced a
rejected finding (`calibrate_rankings`), writing `applied_calibrations.json`.
2. **`record_calibration` stage** (after `verify`) turns this run's non-accepted
verifier outcomes into scoped `FalsePositiveLearning` records
(`learnings_from_verifications`) and merges them into the ledger, de-duplicated
by finding ID so a repeat rejection does not compound without bound.
Demotion is **scoped and reversible**: a rejection in `lib.py` demotes only that
scope; an accepted (reachable) finding in `app.py` is never touched; and because
priority is demoted rather than the finding deleted, the audit trail
(`applied_calibrations.json`, the ledger) shows exactly why a surface lost
attention. Covered by `tests/test_pipeline_calibration.py`.
Beyond the automatic ranking loop, **OpenCode still steers the richer
adjustments** — prompt constraints, retrieval filters, skill routing, and
benchmark-miss triage — via `openultrasast-triage`; those `RankingCalibration`
fields are already produced and will be consumed directly once the language-aware
LLM hunters land (`standard_security_harness`).
## Development