marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(pair-corpus-honesty): reject improve rounds that regress any provenance profile on holdout

Browse Source 
Task 6.3.

- improve/evolve.py: evaluate_profiles and profile_regressions score the
  holdout pair split per provenance profile every round; a drop in pair
  pass rate or Youden beyond --profile-tolerance rejects the round with
  reason profile_regression:<profile>; profiles under
  --min-holdout-pairs are reported, not gated; journal records both.
- cli improve: --pair-catalog, --no-pair-gate, --profile-tolerance,
  --min-holdout-pairs; a missing catalog fails loud instead of skipping
  the gate.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-05 10:45:43 +02:00
co-authored by Claude Fable 5.1
parent 25fa7aa7e1
commit 9e7aef6a6b
4 changed files with 217 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/test_cli_improve.py
+9
View File
@@ -58,3 +58,12 @@ def test_improve_rejects_missing_manifest(tmp_path: Path) -> None:
with pytest.raises(SystemExit):
main(["improve", str(tmp_path / "nope.toml")])
def test_improve_fails_loud_when_pair_catalog_is_missing(tmp_path: Path) -> None:
import pytest
_, ruleset_dir = _scaffold(tmp_path)
manifest = tmp_path / "bench.toml"
with pytest.raises(SystemExit, match="pair catalog"):
main(["improve", str(manifest), "--ruleset-dir", str(ruleset_dir), "--dry-run", "--pair-catalog", str(tmp_path / "missing.toml")])