From 75ae9ee47b05b70ba81b7874c43120c8e994d6e8 Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Sun, 6 Sep 2026 13:20:23 +0200 Subject: [PATCH] test(learning-harness): the gate baseline and the degradation matrix The three gate outputs are captured at 71efc9c -- the commit before the first line of this feature -- and a test reproduces them byte for byte in both matrices. Byte-identity was something a reviewer checked by hand against a worktree; it is now mechanical and runs in CI. Six absences are each exercised once with the reason they record: HarnessX, the chat endpoint (the run still writes its artifact), the family configurations, the network for pointer pairs, the semantic extra (named, not guessed), and the sandbox -- so a claim nothing could run is not confused with a claim that failed. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG --- .kiro/specs/learning-harness/tasks.md | 3 +- .../measurements/2026-09-06-gate-baseline.txt | 22 +++ tests/test_degradation_matrix.py | 144 ++++++++++++++++++ 3 files changed, 168 insertions(+), 1 deletion(-) create mode 100644 benchmarks/measurements/2026-09-06-gate-baseline.txt create mode 100644 tests/test_degradation_matrix.py diff --git a/.kiro/specs/learning-harness/tasks.md b/.kiro/specs/learning-harness/tasks.md index 0740a8c..e981e73 100644 --- a/.kiro/specs/learning-harness/tasks.md +++ b/.kiro/specs/learning-harness/tasks.md @@ -156,7 +156,7 @@ - _Requirements: 10.5_ - [ ] 5. Validation -- [ ] 5.1 Gates, matrices and degradation paths +- [x] 5.1 Gates, matrices and degradation paths - Detection, map and local pair gate outputs byte-identical to the pre-feature commit; the extra-free suite green with semantic and HarnessX tests skipping; each optional dependency's absence exercised once with its recorded reason. - Observable: the three gate outputs diff empty against the recorded baseline and both test matrices pass. - _Requirements: 11.4, 11.5_ @@ -237,3 +237,4 @@ - **The five twins declare their limit.** A content hash is evidence; the catalog row is the maintainer standing behind it, and without one a re-harvest that changes a byte turns a known-unscorable row back into a scored one with no review. A declared limit that names one of the vocabulary's own reasons keeps that spelling, so `identical_twin` declared and `identical_twin` computed stay one fact instead of splitting across two buckets in every aggregation. `semantic.seed._skip_reason` was reordered for the same reason: an unscorable row reports `unscorable:` before either the known-limit or the split branch can claim it. Consequence, recorded because it moves a published number: leave-one-out over vibe-py now runs on **30 rows, not 35** — a pair whose two sides are the same code cannot be held out against itself. - **The fixture-difficulty check** (`learning/difficulty.py`), on three axes the corpus really varies on: binding hops from the identity source to the constrained use, registration distance (0 decorator on the handler, 1 statement elsewhere in the file, 2 another document), and guard distance (0 decorator, 1 call in the body, 2 nothing visible). The comparison is per axis over the whole set, because a suite covers a bug shape with several handlers between them. **Measured first, and it failed: the shared obligations fixture reached (1, 0, 2) where the vibe-py absence rows reach (5, 2, 2).** Every shared handler carried its own route decorator, so nothing in the suite exercised a handler registered from another document and no chain was longer than one binding — the checker was passing on a simpler world than the one it meets. `from_spec` was added to the fixture: the connexion shape those rows really have, a token validated by a call rather than a decorator and five bindings between the token and the constraint. The verdict is committed as `benchmarks/measurements/2026-09-06-fixture-difficulty.json` with both reaches and the history of the first measurement. One existing expectation moved with it (`from_spec` reads the user row and then the book: two real operations). - Task 4.3 (2026-09-06, RED first: 10 failing tests in `tests/test_learning_cutoff.py`; then green, 11). The catalog generator fills `fix_date` from the recipe at the precision the recipe has: a full ISO day when one was recorded, otherwise the year. Every harvested recipe carries a year and almost none carries a day, and writing `-01-01` would turn a year into a claim about a month. A row therefore counts as post-cutoff only when its **earliest possible** date is after the cutoff, so a coarse date can under-claim the clean slice but never over-claim it — and over-claiming is the direction that would turn contamination into a result. `PairFamilyScore` carries the date, `aggregate(cutoff=...)` reports the same numbers over the post-cutoff rows beside the whole number and never instead of it, undated rows are counted on their own, and `publish` prints the slice with the cutoff it used. Without a configured cutoff every scoreboard says so rather than inventing one; `[learning].cutoff_date` has no default because guessing a model's training cutoff and publishing a contamination number from the guess would be worse than reporting none. Corpus dates as harvested: vibe-py all 2026, agent-vfc 67 of 69 in 2026, vfc spread 2014-2026, vfc-js 2014-2021. +- Task 5.1 (2026-09-06, RED first: 3 of 9 tests in `tests/test_degradation_matrix.py`; then green, 8 after one test was deleted rather than kept). The three gate outputs are captured at `71efc9c` — the commit before the first line of this feature — as `benchmarks/measurements/2026-09-06-gate-baseline.txt`, and a test reproduces them byte for byte on every run, in both matrices. Until now byte-identity was something a reviewer checked by hand against a worktree; it is now mechanical and runs in CI, which is what makes it a gate rather than a habit. Six absences are each exercised once with the reason they record: HarnessX (`harnessx_unavailable`), the chat endpoint (`learning_endpoint_unavailable`, and the run still writes its artifact), the family configurations (`learning_configs_unavailable`), the network for pointer pairs (`pointer_pair_skipped`), the semantic extra (`language_unsupported` rather than a guess), and the sandbox (`sandbox_unavailable`, so a claim nothing could run is not confused with a claim that failed). One test written for this task was **deleted before the commit**: it asserted that this file contained a `without_` name per capability, which measures the file's own naming and nothing about the code. diff --git a/benchmarks/measurements/2026-09-06-gate-baseline.txt b/benchmarks/measurements/2026-09-06-gate-baseline.txt new file mode 100644 index 0000000..41e3bb3 --- /dev/null +++ b/benchmarks/measurements/2026-09-06-gate-baseline.txt @@ -0,0 +1,22 @@ +# Gate baseline for the learning-harness spec (task 5.1, Req 11.5). +# Captured at 71efc9c, the commit before the first line of this feature, by running in order: +# python -m openultrasast.gate +# python -m openultrasast.map_gate +# python -m openultrasast.pair_gate +# Every commit of this spec must reproduce this file byte for byte. The three gates are the only place +# where a detector change is allowed to be invisible: if any of them moves, something that was never +# supposed to change did. +# --- output begins --- +detection gate: recall 93.62% (44/47), FP 0.00% (0/44) + c_cpp recall 93.33% (14/15) FP 0.00% (0/14) + java recall 100.00% (4/4) FP 0.00% (0/4) + javascript recall 92.31% (12/13) FP 0.00% (0/12) + python recall 93.33% (14/15) FP 0.00% (0/14) +detection gate: PASS +split-sink map gate: PASS +pair gate: pair_pass 100.00% (3/3), vuln_recall 100.00%, silent_fix 100.00%, Youden +100.00%, labeled_recall 100.00% (28/28), fix_leaks 0 + local pair_pass 100.00% (3/3) vuln 100.00% silent 100.00% Youden +100.00% labeled 100.00% + PASS local-python-vulnerable vuln 12/12 fix_findings=0 leaks=0 + PASS local-javascript-vulnerable vuln 10/10 fix_findings=0 leaks=0 + PASS local-cpp-buffer vuln 6/6 fix_findings=0 leaks=0 +pair gate: PASS (local slice) diff --git a/tests/test_degradation_matrix.py b/tests/test_degradation_matrix.py new file mode 100644 index 0000000..48767e6 --- /dev/null +++ b/tests/test_degradation_matrix.py @@ -0,0 +1,144 @@ +"""What every optional dependency's absence looks like, and that the gates never move (task 5.1, Req 11.4, 11.5). + +The core install has no dependencies. Every capability beyond it is optional, and the rule for all of them is the +same: when it is missing the run says so by name and finishes, rather than crashing or — worse — quietly producing +a smaller number that reads like a result. Each absence is exercised once here, with the reason it records. + +The gate baseline is the other half. Three gates are the only place in this project where a detector change is +allowed to be invisible; if one of them moves, something that was never supposed to change did. +""" + +from __future__ import annotations + +import subprocess +import sys +from pathlib import Path + +import pytest + +BASELINE = Path("benchmarks/measurements/2026-09-06-gate-baseline.txt") +MARKER = "# --- output begins ---\n" + + +def test_the_three_gate_outputs_match_the_committed_pre_feature_baseline() -> None: + """Req 11.5. The baseline was captured at `71efc9c`, before the first line of this feature.""" + assert BASELINE.is_file() + expected = BASELINE.read_text().split(MARKER, 1)[1] + produced = "".join( + subprocess.run( # noqa: S603 — this repository's own modules, no shell + [sys.executable, "-m", f"openultrasast.{module}"], + capture_output=True, + text=True, + check=False, + ).stdout + for module in ("gate", "map_gate", "pair_gate") + ) + assert produced == expected, "a gate moved; diff this against benchmarks/measurements/2026-09-06-gate-baseline.txt" + + +def test_the_baseline_records_where_it_came_from() -> None: + head = BASELINE.read_text().split(MARKER, 1)[0] + assert "71efc9c" in head and "openultrasast.pair_gate" in head + + +# --- one optional dependency at a time --------------------------------------- + + +def test_without_harnessx_the_meta_agent_proposer_proposes_nothing_and_says_why(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + from openultrasast.learning import proposer as proposer_module + from openultrasast.learning.detectors import FamilyConfig + from openultrasast.learning.proposer import FailureFacts, HarnessXProposer + + monkeypatch.setattr(proposer_module, "has_harnessx", lambda: False) + (tmp_path / "injection").mkdir() + facts = FailureFacts( + family="injection", + config=FamilyConfig( + family="injection", version="0", prompt="p", checklist="c", tools=("read_file",), max_steps=1, max_cost_usd=0.1, max_chars=100 + ), + ) + proposer = HarnessXProposer(configs_dir=tmp_path, model="m") + assert proposer.propose(facts) is None + assert proposer.reason == "harnessx_unavailable" + + +def test_without_a_chat_endpoint_the_learning_run_records_it_and_still_writes_its_artifact( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + from openultrasast.cli import main + + for name in ("OPENULTRASAST_HUNTER_CLIENT", "DEEPSEEK_API_KEY", "OPENROUTER_API_KEY"): + monkeypatch.delenv(name, raising=False) + (tmp_path / "a-v.py").write_text("def run():\n return 1\n") + (tmp_path / "a-f.py").write_text("def run():\n return 1\n") + (tmp_path / "pairs.toml").write_text( + '[[pair]]\nname = "a"\nslice = "vibe-py"\nlanguage = "python"\nvuln = "a-v.py"\nfixed = "a-f.py"\n' + 'relpath = "app.py"\nsplit = "train"\nreview_tier = "seeded"\n\n' + '[[pair.expected]]\ncwe = "CWE-78"\nclass = "x"\npath = "app.py"\nfunction = "run"\nmechanism = "other"\nfamily = "injection"\n' + ) + printed: list[str] = [] + monkeypatch.setattr("builtins.print", lambda *args, **kwargs: printed.append(" ".join(str(item) for item in args))) + assert main(["learning", "baseline", "--catalog", str(tmp_path / "pairs.toml"), "--out", str(tmp_path / "l"), "--model", "none"]) == 0 + assert any("learning_endpoint_unavailable" in line for line in printed) + assert (tmp_path / "l" / "baseline" / "none" / "report.json").is_file() + + +def test_without_family_configurations_a_scan_records_the_reason_rather_than_the_traceback( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + from openultrasast.cli import _run_family_detectors + + class _Settings: + configs_dir = "does/not/exist" + families_path = None + + class _Runtime: + def __init__(self) -> None: + self.state: dict[str, list[dict[str, object]]] = {"degradations": []} + + runtime = _Runtime() + findings, payload, cited = _run_family_detectors( + root=tmp_path, + hotspots=[], + cases=None, + settings=_Settings(), + config=None, + runtime=runtime, # type: ignore[arg-type] + ) + assert (findings, payload, cited) == ([], {}, {}) + assert [item["reason"] for item in runtime.state["degradations"]] == ["learning_configs_unavailable"] + + +def test_without_the_network_a_pointer_pair_is_skipped_by_name(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + from openultrasast.pairs import evaluate_catalog, load_pair_catalog, select_slice + + monkeypatch.delenv("OPENULTRASAST_PAIRS_NETWORK", raising=False) + pointers = [case for case in select_slice(load_pair_catalog(), "vibe-py") if not case.vendored][:2] + assert pointers, "the vibe-py slice carries pointer rows; without them this proves nothing" + result = evaluate_catalog(pointers, pointers=False) + assert {item["reason"] for item in result.degradations} == {"pointer_pair_skipped"} + assert result.outcomes == () + + +def test_without_the_semantic_extra_an_unsupported_language_is_named_not_guessed(monkeypatch: pytest.MonkeyPatch) -> None: + from openultrasast.semantic import extra as extra_module + from openultrasast.semantic.ir import parse_file + + monkeypatch.setattr(extra_module, "has_semantic_extra", lambda: False) + ir = parse_file("app.go", "package main\n\nfunc main() {}\n", "go") + assert ir.reason == "language_unsupported" and ir.parse_ok is False + + +def test_without_a_sandbox_the_regress_stage_is_skipped_by_name(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """A claim that could not be proved because nothing could run it is not the same as a claim that failed.""" + import json + + from openultrasast.cli import main + + monkeypatch.setenv("OPENULTRASAST_SANDBOX_PROBE", "0") + (tmp_path / "app.py").write_text("import os\n\n\ndef run(cmd):\n os.system(cmd)\n") + assert main(["scan", str(tmp_path), "--mode", "deep"]) in {0, 1} + runs = sorted((tmp_path / ".openultrasast" / "runs").iterdir()) + manifest = json.loads((runs[-1] / "manifest.json").read_text()) + reasons = {item["reason"] for item in manifest.get("degradations", [])} + assert "sandbox_unavailable" in reasons