mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 11:49:28 +00:00
test(learning-harness): the gate baseline and the degradation matrix
The three gate outputs are captured at 71efc9c -- the commit before the
first line of this feature -- and a test reproduces them byte for byte in
both matrices. Byte-identity was something a reviewer checked by hand
against a worktree; it is now mechanical and runs in CI.
Six absences are each exercised once with the reason they record: HarnessX,
the chat endpoint (the run still writes its artifact), the family
configurations, the network for pointer pairs, the semantic extra (named,
not guessed), and the sandbox -- so a claim nothing could run is not
confused with a claim that failed.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
co-authored by
Claude Opus 5
Claude Opus 5
parent
3e71e9e88b
commit
75ae9ee47b
@@ -156,7 +156,7 @@
|
||||
- _Requirements: 10.5_
|
||||
|
||||
- [ ] 5. Validation
|
||||
- [ ] 5.1 Gates, matrices and degradation paths
|
||||
- [x] 5.1 Gates, matrices and degradation paths
|
||||
- Detection, map and local pair gate outputs byte-identical to the pre-feature commit; the extra-free suite green with semantic and HarnessX tests skipping; each optional dependency's absence exercised once with its recorded reason.
|
||||
- Observable: the three gate outputs diff empty against the recorded baseline and both test matrices pass.
|
||||
- _Requirements: 11.4, 11.5_
|
||||
@@ -237,3 +237,4 @@
|
||||
- **The five twins declare their limit.** A content hash is evidence; the catalog row is the maintainer standing behind it, and without one a re-harvest that changes a byte turns a known-unscorable row back into a scored one with no review. A declared limit that names one of the vocabulary's own reasons keeps that spelling, so `identical_twin` declared and `identical_twin` computed stay one fact instead of splitting across two buckets in every aggregation. `semantic.seed._skip_reason` was reordered for the same reason: an unscorable row reports `unscorable:<reason>` before either the known-limit or the split branch can claim it. Consequence, recorded because it moves a published number: leave-one-out over vibe-py now runs on **30 rows, not 35** — a pair whose two sides are the same code cannot be held out against itself.
|
||||
- **The fixture-difficulty check** (`learning/difficulty.py`), on three axes the corpus really varies on: binding hops from the identity source to the constrained use, registration distance (0 decorator on the handler, 1 statement elsewhere in the file, 2 another document), and guard distance (0 decorator, 1 call in the body, 2 nothing visible). The comparison is per axis over the whole set, because a suite covers a bug shape with several handlers between them. **Measured first, and it failed: the shared obligations fixture reached (1, 0, 2) where the vibe-py absence rows reach (5, 2, 2).** Every shared handler carried its own route decorator, so nothing in the suite exercised a handler registered from another document and no chain was longer than one binding — the checker was passing on a simpler world than the one it meets. `from_spec` was added to the fixture: the connexion shape those rows really have, a token validated by a call rather than a decorator and five bindings between the token and the constraint. The verdict is committed as `benchmarks/measurements/2026-09-06-fixture-difficulty.json` with both reaches and the history of the first measurement. One existing expectation moved with it (`from_spec` reads the user row and then the book: two real operations).
|
||||
- Task 4.3 (2026-09-06, RED first: 10 failing tests in `tests/test_learning_cutoff.py`; then green, 11). The catalog generator fills `fix_date` from the recipe at the precision the recipe has: a full ISO day when one was recorded, otherwise the year. Every harvested recipe carries a year and almost none carries a day, and writing `-01-01` would turn a year into a claim about a month. A row therefore counts as post-cutoff only when its **earliest possible** date is after the cutoff, so a coarse date can under-claim the clean slice but never over-claim it — and over-claiming is the direction that would turn contamination into a result. `PairFamilyScore` carries the date, `aggregate(cutoff=...)` reports the same numbers over the post-cutoff rows beside the whole number and never instead of it, undated rows are counted on their own, and `publish` prints the slice with the cutoff it used. Without a configured cutoff every scoreboard says so rather than inventing one; `[learning].cutoff_date` has no default because guessing a model's training cutoff and publishing a contamination number from the guess would be worse than reporting none. Corpus dates as harvested: vibe-py all 2026, agent-vfc 67 of 69 in 2026, vfc spread 2014-2026, vfc-js 2014-2021.
|
||||
- Task 5.1 (2026-09-06, RED first: 3 of 9 tests in `tests/test_degradation_matrix.py`; then green, 8 after one test was deleted rather than kept). The three gate outputs are captured at `71efc9c` — the commit before the first line of this feature — as `benchmarks/measurements/2026-09-06-gate-baseline.txt`, and a test reproduces them byte for byte on every run, in both matrices. Until now byte-identity was something a reviewer checked by hand against a worktree; it is now mechanical and runs in CI, which is what makes it a gate rather than a habit. Six absences are each exercised once with the reason they record: HarnessX (`harnessx_unavailable`), the chat endpoint (`learning_endpoint_unavailable`, and the run still writes its artifact), the family configurations (`learning_configs_unavailable`), the network for pointer pairs (`pointer_pair_skipped`), the semantic extra (`language_unsupported` rather than a guess), and the sandbox (`sandbox_unavailable`, so a claim nothing could run is not confused with a claim that failed). One test written for this task was **deleted before the commit**: it asserted that this file contained a `without_<capability>` name per capability, which measures the file's own naming and nothing about the code.
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
# Gate baseline for the learning-harness spec (task 5.1, Req 11.5).
|
||||
# Captured at 71efc9c, the commit before the first line of this feature, by running in order:
|
||||
# python -m openultrasast.gate
|
||||
# python -m openultrasast.map_gate
|
||||
# python -m openultrasast.pair_gate
|
||||
# Every commit of this spec must reproduce this file byte for byte. The three gates are the only place
|
||||
# where a detector change is allowed to be invisible: if any of them moves, something that was never
|
||||
# supposed to change did.
|
||||
# --- output begins ---
|
||||
detection gate: recall 93.62% (44/47), FP 0.00% (0/44)
|
||||
c_cpp recall 93.33% (14/15) FP 0.00% (0/14)
|
||||
java recall 100.00% (4/4) FP 0.00% (0/4)
|
||||
javascript recall 92.31% (12/13) FP 0.00% (0/12)
|
||||
python recall 93.33% (14/15) FP 0.00% (0/14)
|
||||
detection gate: PASS
|
||||
split-sink map gate: PASS
|
||||
pair gate: pair_pass 100.00% (3/3), vuln_recall 100.00%, silent_fix 100.00%, Youden +100.00%, labeled_recall 100.00% (28/28), fix_leaks 0
|
||||
local pair_pass 100.00% (3/3) vuln 100.00% silent 100.00% Youden +100.00% labeled 100.00%
|
||||
PASS local-python-vulnerable vuln 12/12 fix_findings=0 leaks=0
|
||||
PASS local-javascript-vulnerable vuln 10/10 fix_findings=0 leaks=0
|
||||
PASS local-cpp-buffer vuln 6/6 fix_findings=0 leaks=0
|
||||
pair gate: PASS (local slice)
|
||||
@@ -0,0 +1,144 @@
|
||||
"""What every optional dependency's absence looks like, and that the gates never move (task 5.1, Req 11.4, 11.5).
|
||||
|
||||
The core install has no dependencies. Every capability beyond it is optional, and the rule for all of them is the
|
||||
same: when it is missing the run says so by name and finishes, rather than crashing or — worse — quietly producing
|
||||
a smaller number that reads like a result. Each absence is exercised once here, with the reason it records.
|
||||
|
||||
The gate baseline is the other half. Three gates are the only place in this project where a detector change is
|
||||
allowed to be invisible; if one of them moves, something that was never supposed to change did.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
BASELINE = Path("benchmarks/measurements/2026-09-06-gate-baseline.txt")
|
||||
MARKER = "# --- output begins ---\n"
|
||||
|
||||
|
||||
def test_the_three_gate_outputs_match_the_committed_pre_feature_baseline() -> None:
|
||||
"""Req 11.5. The baseline was captured at `71efc9c`, before the first line of this feature."""
|
||||
assert BASELINE.is_file()
|
||||
expected = BASELINE.read_text().split(MARKER, 1)[1]
|
||||
produced = "".join(
|
||||
subprocess.run( # noqa: S603 — this repository's own modules, no shell
|
||||
[sys.executable, "-m", f"openultrasast.{module}"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
).stdout
|
||||
for module in ("gate", "map_gate", "pair_gate")
|
||||
)
|
||||
assert produced == expected, "a gate moved; diff this against benchmarks/measurements/2026-09-06-gate-baseline.txt"
|
||||
|
||||
|
||||
def test_the_baseline_records_where_it_came_from() -> None:
|
||||
head = BASELINE.read_text().split(MARKER, 1)[0]
|
||||
assert "71efc9c" in head and "openultrasast.pair_gate" in head
|
||||
|
||||
|
||||
# --- one optional dependency at a time ---------------------------------------
|
||||
|
||||
|
||||
def test_without_harnessx_the_meta_agent_proposer_proposes_nothing_and_says_why(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
from openultrasast.learning import proposer as proposer_module
|
||||
from openultrasast.learning.detectors import FamilyConfig
|
||||
from openultrasast.learning.proposer import FailureFacts, HarnessXProposer
|
||||
|
||||
monkeypatch.setattr(proposer_module, "has_harnessx", lambda: False)
|
||||
(tmp_path / "injection").mkdir()
|
||||
facts = FailureFacts(
|
||||
family="injection",
|
||||
config=FamilyConfig(
|
||||
family="injection", version="0", prompt="p", checklist="c", tools=("read_file",), max_steps=1, max_cost_usd=0.1, max_chars=100
|
||||
),
|
||||
)
|
||||
proposer = HarnessXProposer(configs_dir=tmp_path, model="m")
|
||||
assert proposer.propose(facts) is None
|
||||
assert proposer.reason == "harnessx_unavailable"
|
||||
|
||||
|
||||
def test_without_a_chat_endpoint_the_learning_run_records_it_and_still_writes_its_artifact(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
from openultrasast.cli import main
|
||||
|
||||
for name in ("OPENULTRASAST_HUNTER_CLIENT", "DEEPSEEK_API_KEY", "OPENROUTER_API_KEY"):
|
||||
monkeypatch.delenv(name, raising=False)
|
||||
(tmp_path / "a-v.py").write_text("def run():\n return 1\n")
|
||||
(tmp_path / "a-f.py").write_text("def run():\n return 1\n")
|
||||
(tmp_path / "pairs.toml").write_text(
|
||||
'[[pair]]\nname = "a"\nslice = "vibe-py"\nlanguage = "python"\nvuln = "a-v.py"\nfixed = "a-f.py"\n'
|
||||
'relpath = "app.py"\nsplit = "train"\nreview_tier = "seeded"\n\n'
|
||||
'[[pair.expected]]\ncwe = "CWE-78"\nclass = "x"\npath = "app.py"\nfunction = "run"\nmechanism = "other"\nfamily = "injection"\n'
|
||||
)
|
||||
printed: list[str] = []
|
||||
monkeypatch.setattr("builtins.print", lambda *args, **kwargs: printed.append(" ".join(str(item) for item in args)))
|
||||
assert main(["learning", "baseline", "--catalog", str(tmp_path / "pairs.toml"), "--out", str(tmp_path / "l"), "--model", "none"]) == 0
|
||||
assert any("learning_endpoint_unavailable" in line for line in printed)
|
||||
assert (tmp_path / "l" / "baseline" / "none" / "report.json").is_file()
|
||||
|
||||
|
||||
def test_without_family_configurations_a_scan_records_the_reason_rather_than_the_traceback(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
from openultrasast.cli import _run_family_detectors
|
||||
|
||||
class _Settings:
|
||||
configs_dir = "does/not/exist"
|
||||
families_path = None
|
||||
|
||||
class _Runtime:
|
||||
def __init__(self) -> None:
|
||||
self.state: dict[str, list[dict[str, object]]] = {"degradations": []}
|
||||
|
||||
runtime = _Runtime()
|
||||
findings, payload, cited = _run_family_detectors(
|
||||
root=tmp_path,
|
||||
hotspots=[],
|
||||
cases=None,
|
||||
settings=_Settings(),
|
||||
config=None,
|
||||
runtime=runtime, # type: ignore[arg-type]
|
||||
)
|
||||
assert (findings, payload, cited) == ([], {}, {})
|
||||
assert [item["reason"] for item in runtime.state["degradations"]] == ["learning_configs_unavailable"]
|
||||
|
||||
|
||||
def test_without_the_network_a_pointer_pair_is_skipped_by_name(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
from openultrasast.pairs import evaluate_catalog, load_pair_catalog, select_slice
|
||||
|
||||
monkeypatch.delenv("OPENULTRASAST_PAIRS_NETWORK", raising=False)
|
||||
pointers = [case for case in select_slice(load_pair_catalog(), "vibe-py") if not case.vendored][:2]
|
||||
assert pointers, "the vibe-py slice carries pointer rows; without them this proves nothing"
|
||||
result = evaluate_catalog(pointers, pointers=False)
|
||||
assert {item["reason"] for item in result.degradations} == {"pointer_pair_skipped"}
|
||||
assert result.outcomes == ()
|
||||
|
||||
|
||||
def test_without_the_semantic_extra_an_unsupported_language_is_named_not_guessed(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
from openultrasast.semantic import extra as extra_module
|
||||
from openultrasast.semantic.ir import parse_file
|
||||
|
||||
monkeypatch.setattr(extra_module, "has_semantic_extra", lambda: False)
|
||||
ir = parse_file("app.go", "package main\n\nfunc main() {}\n", "go")
|
||||
assert ir.reason == "language_unsupported" and ir.parse_ok is False
|
||||
|
||||
|
||||
def test_without_a_sandbox_the_regress_stage_is_skipped_by_name(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""A claim that could not be proved because nothing could run it is not the same as a claim that failed."""
|
||||
import json
|
||||
|
||||
from openultrasast.cli import main
|
||||
|
||||
monkeypatch.setenv("OPENULTRASAST_SANDBOX_PROBE", "0")
|
||||
(tmp_path / "app.py").write_text("import os\n\n\ndef run(cmd):\n os.system(cmd)\n")
|
||||
assert main(["scan", str(tmp_path), "--mode", "deep"]) in {0, 1}
|
||||
runs = sorted((tmp_path / ".openultrasast" / "runs").iterdir())
|
||||
manifest = json.loads((runs[-1] / "manifest.json").read_text())
|
||||
reasons = {item["reason"] for item in manifest.get("degradations", [])}
|
||||
assert "sandbox_unavailable" in reasons
|
||||
Reference in New Issue
Block a user