mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 11:49:28 +00:00
feat(learning-harness): declared twins and a fixture-difficulty check
The five byte-identical vibe-py twins now say so in the catalog instead of only hashing to it, and a declared limit that names one of the unscorable vocabulary's own reasons keeps that spelling, so `identical_twin` stays one fact. Leave-one-out over vibe-py consequently runs on 30 rows, not 35. The difficulty check measures three axes the corpus varies on: binding hops from the identity source to the constrained use, registration distance, and guard distance. Measured first, and it failed: the shared obligations fixture reached (1, 0, 2) where the vibe-py absence rows reach (5, 2, 2). Every fixture handler carried its own route decorator, so nothing in the suite exercised a handler registered from another document and no binding chain was longer than one hop -- the checker was passing on a simpler world than the one it meets. `from_spec` closes that with the connexion shape those rows really have. The verdict is committed beside the numbers. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
co-authored by
Claude Opus 5
parent
1843ef9a02
commit
d1faa04771
+3
-1
@@ -103,7 +103,9 @@ def test_cli_pairs_loo_writes_the_artifact_offline(tmp_path: Path, monkeypatch:
|
||||
with redirect_stdout(buf):
|
||||
assert main(["pairs", "--slice", "vibe-py", "--loo", "--json", "--loo-out", str(tmp_path / "loo.json")]) == 0
|
||||
payload = json.loads(buf.getvalue())
|
||||
assert payload["per_slice"]["vibe-py"]["pairs"] == 35 and "found_by" in payload["outcomes"][0]
|
||||
# 35 vendored rows less the five byte-identical twins, which task 4.2 declared unscorable in the catalog: a
|
||||
# pair whose two sides are the same code cannot be held out against itself.
|
||||
assert payload["per_slice"]["vibe-py"]["pairs"] == 30 and "found_by" in payload["outcomes"][0]
|
||||
assert json.loads((tmp_path / "loo.json").read_text()) == payload
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user