marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-20 13:39:27 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(learning-harness): declared twins and a fixture-difficulty check

Browse Source 
The five byte-identical vibe-py twins now say so in the catalog instead of
only hashing to it, and a declared limit that names one of the unscorable
vocabulary's own reasons keeps that spelling, so `identical_twin` stays one
fact. Leave-one-out over vibe-py consequently runs on 30 rows, not 35.

The difficulty check measures three axes the corpus varies on: binding hops
from the identity source to the constrained use, registration distance, and
guard distance. Measured first, and it failed: the shared obligations
fixture reached (1, 0, 2) where the vibe-py absence rows reach (5, 2, 2).
Every fixture handler carried its own route decorator, so nothing in the
suite exercised a handler registered from another document and no binding
chain was longer than one hop -- the checker was passing on a simpler world
than the one it meets. `from_spec` closes that with the connexion shape
those rows really have. The verdict is committed beside the numbers.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-06 13:09:56 +02:00
co-authored by Claude Opus 5
parent 1843ef9a02
commit d1faa04771
10 changed files with 393 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benchmarks/pairs/vibe-py/catalog.toml
+5
View File
@@ -2324,6 +2324,7 @@ fix_policy = "silent"
provenance = "human"
split = "holdout"
review_tier = "seeded"
known_limit = "identical_twin"
[[pair.expected]]
cwe = "CWE-89"
@@ -3207,6 +3208,7 @@ fix_policy = "silent"
provenance = "human"
split = "train"
review_tier = "seeded"
known_limit = "identical_twin"
[[pair.expected]]
cwe = "CWE-1336"
@@ -3236,6 +3238,7 @@ fix_policy = "silent"
provenance = "human"
split = "holdout"
review_tier = "seeded"
known_limit = "identical_twin"
[[pair.expected]]
cwe = "CWE-79"
@@ -3354,6 +3357,7 @@ fix_policy = "silent"
provenance = "human"
split = "holdout"
review_tier = "seeded"
known_limit = "identical_twin"
[[pair.context]]
relpath = "openapi_specs/openapi3.yml"
@@ -3656,6 +3660,7 @@ fix_policy = "silent"
provenance = "human"
split = "train"
review_tier = "seeded"
known_limit = "identical_twin"
[[pair.expected]]
cwe = "CWE-502"