Files
Open_Ultra_SAST/tests
Marius CiepluchandClaude Opus 5 346cc703c3 test(learning-harness): a flaky harness must not be mistaken for a flaky detector
The round-zero test decided which materialized side it was looking at by
searching the whole path for a substring, and the random temporary directory
name sometimes contained it. The detector then "found" the bug on the fixed side
too, the pair flipped, and the model comparison failed about one run in eight.

The check is now the directory name alone. This matters beyond the test: a noise
floor exists to measure how much a detector disagrees with itself, and noise
coming from the harness would be recorded as the detector's own and inflate
every family's regression budget.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
2026-09-06 11:45:00 +02:00
..