mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 11:49:28 +00:00
Marius CiepluchandClaude Opus 5
8c387306be
The pair corpus scores excerpts: a vulnerable function and its fixed twin, handed to the engine with the answer already narrowed down. Nothing in it exercises a repository -- region ranking, the budget, one CPG over thousands of files, or whether a verdict that holds on fifteen lines still holds when the source, the sink and the guard live in three modules. benchmarks/repos/<name>.toml names a real checkout: url, a full 40-char commit SHA, licence, and every known vulnerability with its file, function and line. `ousast repos` lists them; `--fetch` materialises one; CI does neither, and conftest clears the switch and redirects the cache so a developer with warm checkouts cannot read them by accident. Three rules the loader enforces rather than documents: - A branch or tag is rejected. Either would make the same recipe mean different code next month while the baseline it produced kept the old numbers. - `in_scope` must be stated per vulnerability, never defaulted. It decides whether "not found" is a miss or a non-question, which is the most important thing about the result. - A recipe that declares `detection` must name something in scope to detect. `ousast repos` does not trust a recipe either: against a materialised checkout it checks HEAD against the pin, that the licence file is there, and that each declared function appears above its declared line. That earned its place immediately -- the first libpng recipe named LICENSE.md, which is the filename at HEAD today and not at the pinned commit, where it is still LICENSE. Two checkouts, verified against the real repositories: - vampi (MIT, 520 lines) is the DETECTION checkout. Excerpts of it are already in the pair corpus, including the update_password miss diagnosed earlier, so a whole-repository run says directly whether a verdict survives the move off the excerpt. Its routes register through an OpenAPI operationId, so it exercises registration-based entry-point mapping rather than the decorator path. - libpng (libpng-2.0, 91k lines of C) is the ENVELOPE checkout, pinned at the parent of the commit that added the missing clamp. Its CVE is marked out of scope: an out-of-bounds read is not a family this engine arbitrates, and recording "not found" against it as a miss is the easiest way to make a measurement lie. Nothing is vendored; the recipe records the licence so an artifact can say what it ran against. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
52 lines
2.1 KiB
Python
52 lines
2.1 KiB
Python
"""Shared test fixtures."""
|
|
|
|
import subprocess
|
|
import sys
|
|
|
|
import pytest
|
|
|
|
_HARNESSX_GUARD = (
|
|
"\nimport sys as _sys\n"
|
|
"_leaked = sorted(m for m in _sys.modules if m == 'harnessx' or m.startswith('harnessx.'))\n"
|
|
"assert not _leaked, 'leaked harnessx modules: ' + repr(_leaked)\n"
|
|
)
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _pointer_pairs_offline(monkeypatch: pytest.MonkeyPatch, tmp_path_factory: pytest.TempPathFactory) -> None:
|
|
"""Req 10.4: the default test run never touches the network for pointer pairs, whatever the operator's shell exports."""
|
|
monkeypatch.delenv("OPENULTRASAST_PAIRS_NETWORK", raising=False)
|
|
monkeypatch.setenv("OPENULTRASAST_PAIR_CACHE", str(tmp_path_factory.mktemp("pair-cache")))
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _repos_offline(monkeypatch: pytest.MonkeyPatch, tmp_path_factory: pytest.TempPathFactory) -> None:
|
|
"""Req 4.1: pinned repository recipes never fetch during a test run, whatever the operator's shell exports.
|
|
|
|
The cache is redirected as well as the switch cleared, so a developer with warm checkouts and a test that
|
|
forgets to pass an explicit root still cannot read them by accident.
|
|
"""
|
|
monkeypatch.delenv("OPENULTRASAST_REPOS_NETWORK", raising=False)
|
|
monkeypatch.setenv("OPENULTRASAST_REPO_CACHE", str(tmp_path_factory.mktemp("repo-cache")))
|
|
|
|
|
|
@pytest.fixture
|
|
def assert_cold_of_harnessx(): # type: ignore[no-untyped-def]
|
|
"""Run ``code`` in a fresh interpreter and assert it imported no ``harnessx`` module.
|
|
|
|
Hermetic by construction: the lazy-import guarantee is verified the same way
|
|
whether or not the optional extra is installed in this environment (a global
|
|
``sys.modules`` check would be polluted by other tests once the extra is present).
|
|
"""
|
|
|
|
def _check(code: str) -> str:
|
|
proc = subprocess.run(
|
|
[sys.executable, "-c", code + _HARNESSX_GUARD],
|
|
capture_output=True,
|
|
text=True,
|
|
)
|
|
assert proc.returncode == 0, f"exit={proc.returncode}\nstdout={proc.stdout}\nstderr={proc.stderr}"
|
|
return proc.stdout
|
|
|
|
return _check
|