marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-19 04:59:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(ci): run ruff format, and record run 5 at 90% pair-correct

Browse Source 
CI was red on `ruff format --check`: 23 files. I ran `ruff check` after every
change this session and never once ran `ruff format --check`, though CI runs
both. Lint passing is not formatting passing, and I treated one as evidence for
the other for the whole session.

Formatting applied. The full CI sequence now reproduces locally: format, lint,
mypy over 94 files, 735 tests, three gates byte-identical, compileall, and the
semantic job's 16 marked tests.

Also records run 5 of the pipeline measurement, which the vampi fix lifted as
predicted:

                       hunter (K=5)   pipeline (K=1)
  pair-correct              76.7%          90.0%
  leak                      12.3%           6.7%
  detection                    --          96.7%
  cost                     ~$0.69        $0.0068

27 of 30 pairs. Five runs, each step from a measurement contradicting an
assumption: 73.3 (no pipeline path) -> 73.3 (routing fixed, IDENTICAL, which
exposed the discarded verdict) -> 80.0 (residual question matched to its
arbiter) -> 86.7 (aiohttp settings) -> 90.0 (dominance question no longer
contradicting its own evidence).

One miss left: dsvpwa-dsvpwa-main, whose setting is an argparse --ssl default
rather than a security-setting call. Out of reach before the settings work and
still is; contorting the arbiter around one pair would be worse than the miss.

The caveat is recorded beside the number: 30 pairs of one slice, and the slice
iterated against all session. The +16% overfitting gap against vfc-js is why
contributor-scan Req 8.1/8.2 exist.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-08 19:46:58 +02:00
co-authored by Claude Opus 5
parent 14ca0a96ce
commit fe45081b43
24 changed files with 397 additions and 213 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/test_model_calibrate.py
+14 -10
View File
@@ -48,12 +48,14 @@ def test_flagging_both_sides_equally_is_a_gap_not_a_success() -> None:
def test_gaps_are_grouped_by_family_with_their_reasons() -> None:
from openultrasast.model.calibrate import calibrate
report = calibrate([
_outcome("a", family="path", vuln="none", reason="sink_not_in_table"),
_outcome("b", family="path", vuln="none", reason="sink_not_in_table"),
_outcome("c", family="injection", vuln="none", reason="source_unmodelled"),
_outcome("d", family="injection"),
])
report = calibrate(
[
_outcome("a", family="path", vuln="none", reason="sink_not_in_table"),
_outcome("b", family="path", vuln="none", reason="sink_not_in_table"),
_outcome("c", family="injection", vuln="none", reason="source_unmodelled"),
_outcome("d", family="injection"),
]
)
by_family = report.gaps_by_family()
assert set(by_family) == {"path", "injection"}
assert by_family["path"]["pairs"] == 2
@@ -65,10 +67,12 @@ def test_the_report_is_per_slice_and_names_the_deciding_one() -> None:
"""Req 9.3/11.1: the real-world slice decides; a development slice must not stand in for the corpus."""
from openultrasast.model.calibrate import calibrate
report = calibrate([
_outcome("a", slice_name="vibe-py"),
_outcome("b", slice_name="vfc-js", vuln="none", reason="sink_not_in_table"),
])
report = calibrate(
[
_outcome("a", slice_name="vibe-py"),
_outcome("b", slice_name="vfc-js", vuln="none", reason="sink_not_in_table"),
]
)
assert set(report.per_slice) == {"vibe-py", "vfc-js"}
assert report.per_slice["vibe-py"]["covered"] == 1
assert report.per_slice["vfc-js"]["gaps"] == 1