marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-17 03:59:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(learning-harness): a slice is part of what identifies a baseline

Browse Source 
Keyed by model alone, the memory-safety run over the vfc slice silently
replaced the run over the web families, and the published table would have
shown one of them as if it were everything. The artifact key and the
comparison label now carry the slice selection when it is not the default.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-06 14:39:43 +02:00
co-authored by Claude Opus 5
parent c1ef3ca539
commit ce89b0dbe0
4 changed files with 58 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/test_learning_cutoff.py
+3 -1
View File
@@ -208,6 +208,8 @@ def test_a_baseline_over_a_named_slice_measures_that_slice(tmp_path: Path, monke
)
== 0
)
report = json.loads((tmp_path / "l" / "baseline" / "m" / "report.json").read_text())
# A run over a named slice is its own baseline: keyed by model *and* slice, so it cannot overwrite the run
# over the web families.
report = json.loads((tmp_path / "l" / "baseline" / "m-vfc" / "report.json").read_text())
assert report["slices"] == ["vfc"]
assert report["metrics"]["injection"]["scorable"] == 1