marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 19:59:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(pair-corpus-honesty): profile gate scores only seeded and reviewed pairs

Browse Source 
Task 8.2 (Req 9.3).

- improve/evolve.py: evaluate_profiles filters holdout pairs to
  GATING_TIERS before computing per-profile metrics, so advisory and
  title pairs are scored by `pairs` but never gate a round; a regression
  on reviewed pairs still rejects with profile_regression:<profile>.
- tests: a regression confined to title pairs is accepted; the same
  regression on reviewed pairs is rejected and the title pairs are
  excluded from the gated profile rather than outvoted.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-05 10:45:43 +02:00
co-authored by Claude Fable 5.1
parent b1c5f36076
commit 25f1512eb9
2 changed files with 49 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/openultrasast/improve/evolve.py
+6 -3
View File
@@ -57,10 +57,13 @@ def evaluate_profiles(
pair_cases: Sequence[object],
rules: tuple[PatternRule, ...],
) -> dict[str, dict[str, float]]:
"""Pair-corpus metrics per provenance profile under ``rules`` (holdout split, no inventory sidecar)."""
from ..pairs import PairCase, evaluate_catalog
"""Pair-corpus metrics per provenance profile under ``rules`` (holdout split, no inventory sidecar).
cases = [case for case in pair_cases if isinstance(case, PairCase)]
Only ``seeded`` and ``reviewed`` pairs gate (Req 9.3); ``advisory`` and ``title`` pairs are scored by ``pairs``, never here.
"""
from ..pairs import GATING_TIERS, PairCase, evaluate_catalog, select_tier
cases = list(select_tier([case for case in pair_cases if isinstance(case, PairCase)], GATING_TIERS))
if not cases:
return {}
result = evaluate_catalog(cases, ruleset=rules, inventory_sidecar=False)