marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(calibration): automatic in-pipeline false-positive feedback

Browse Source 
Close the HarnessX self-improving loop without an agent in the loop. Every scan
now feeds its verifier outcomes back into the next scan's ranking:

- New `calibrate` stage (after rank, before findings) loads a persistent ledger
  of scoped false-positive learnings and demotes every previously-rejected scope
  (`calibrate_rankings`), writing `calibration/applied_calibrations.json`.
- New `record_calibration` stage (after verify) converts this run's non-accepted
  verifier outcomes into scoped `FalsePositiveLearning` records
  (`learnings_from_verifications`) and merges them into
  `.openultrasast/calibration/false_positive_learnings.json`, de-duplicated by
  finding ID so a repeat rejection does not compound without bound.
- Lower the calibration floor (MIN_CALIBRATED_PRIORITY = 0.1) so accumulated
  rejections can demote a scope below the natural ranking minimum.

Demotion is scoped and reversible: a rejection in one path never touches an
accepted finding in another, and priority is demoted rather than the finding
deleted so the audit trail explains every downgrade. First run is a no-op
(empty ledger), so existing single-run behavior is unchanged.

Adds tests/test_pipeline_calibration.py (two-run demotion, order-independent),
documents the realized loop in the README, and checks off the
standard_security_harness milestone in tasks.md.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-06-21 20:44:08 +02:00
co-authored by Claude Opus 4.8
parent d9eff9aade
commit 9c4af4f546
5 changed files with 213 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/specs/openrouter-sast-harness/tasks.md
+1 -1
View File
@@ -128,7 +128,7 @@ Do not report OpenUltraSAST as a working security harness until `usable_harness_
- [x] Standard mode uses ranked hunter scheduling, retrieval packages, selected skill snippets, and independent verification.
- [x] Every hunter trajectory is persisted and linked to finding artifacts.
- [ ] False-positive learning updates ranking calibration from verifier outcomes.
- [x] False-positive learning updates ranking calibration from verifier outcomes.
- [ ] Fusion can be triggered for high-impact, contradictory, or difficult findings.
- [ ] Standard mode uses real language-aware LLM hunters instead of reusing quick static findings.