mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-19 04:59:28 +00:00
feat(learning-harness): declared twins and a fixture-difficulty check
The five byte-identical vibe-py twins now say so in the catalog instead of only hashing to it, and a declared limit that names one of the unscorable vocabulary's own reasons keeps that spelling, so `identical_twin` stays one fact. Leave-one-out over vibe-py consequently runs on 30 rows, not 35. The difficulty check measures three axes the corpus varies on: binding hops from the identity source to the constrained use, registration distance, and guard distance. Measured first, and it failed: the shared obligations fixture reached (1, 0, 2) where the vibe-py absence rows reach (5, 2, 2). Every fixture handler carried its own route decorator, so nothing in the suite exercised a handler registered from another document and no binding chain was longer than one hop -- the checker was passing on a simpler world than the one it meets. `from_spec` closes that with the connexion shape those rows really have. The verdict is committed beside the numbers. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
co-authored by
Claude Opus 5
parent
1843ef9a02
commit
d1faa04771
@@ -195,14 +195,16 @@ def _obligation_facts() -> ObligationFacts:
|
||||
def _skip_reason(case: PairCase) -> str | None:
|
||||
from ..learning.split import is_teacher
|
||||
|
||||
if not is_teacher(case) and case.review_tier in GATING_TIERS and case.vendored and not case.known_limit:
|
||||
# learning-harness Req 5.1: only train-split, scorable pairs teach. Say which of the two it was: two
|
||||
# vibe-py rows are train-split identical twins, and calling them `split:train` is simply untrue.
|
||||
return f"unscorable:{case.unscorable}" if case.unscorable else f"split:{case.split}"
|
||||
if case.review_tier not in GATING_TIERS:
|
||||
return f"tier:{case.review_tier}"
|
||||
if case.unscorable:
|
||||
# One spelling per corpus fact. `identical_twin` declared in the catalog and `identical_twin` computed from
|
||||
# the bytes are the same reason, and calling one of them `split:train` is simply untrue (Req 5.1, 10.3).
|
||||
return f"unscorable:{case.unscorable}"
|
||||
if case.known_limit:
|
||||
return f"known_limit:{case.known_limit}"
|
||||
if not is_teacher(case) and case.vendored:
|
||||
return f"split:{case.split}"
|
||||
if not case.vuln_file.is_file() or not case.fixed_file.is_file():
|
||||
return "pointer_pair_not_cached" if not case.vendored else "excerpt_missing"
|
||||
return None
|
||||
|
||||
Reference in New Issue
Block a user