diff --git a/.kiro/specs/learning-harness/tasks.md b/.kiro/specs/learning-harness/tasks.md index e981e73..d2bb94b 100644 --- a/.kiro/specs/learning-harness/tasks.md +++ b/.kiro/specs/learning-harness/tasks.md @@ -238,3 +238,8 @@ - **The fixture-difficulty check** (`learning/difficulty.py`), on three axes the corpus really varies on: binding hops from the identity source to the constrained use, registration distance (0 decorator on the handler, 1 statement elsewhere in the file, 2 another document), and guard distance (0 decorator, 1 call in the body, 2 nothing visible). The comparison is per axis over the whole set, because a suite covers a bug shape with several handlers between them. **Measured first, and it failed: the shared obligations fixture reached (1, 0, 2) where the vibe-py absence rows reach (5, 2, 2).** Every shared handler carried its own route decorator, so nothing in the suite exercised a handler registered from another document and no chain was longer than one binding — the checker was passing on a simpler world than the one it meets. `from_spec` was added to the fixture: the connexion shape those rows really have, a token validated by a call rather than a decorator and five bindings between the token and the constraint. The verdict is committed as `benchmarks/measurements/2026-09-06-fixture-difficulty.json` with both reaches and the history of the first measurement. One existing expectation moved with it (`from_spec` reads the user row and then the book: two real operations). - Task 4.3 (2026-09-06, RED first: 10 failing tests in `tests/test_learning_cutoff.py`; then green, 11). The catalog generator fills `fix_date` from the recipe at the precision the recipe has: a full ISO day when one was recorded, otherwise the year. Every harvested recipe carries a year and almost none carries a day, and writing `-01-01` would turn a year into a claim about a month. A row therefore counts as post-cutoff only when its **earliest possible** date is after the cutoff, so a coarse date can under-claim the clean slice but never over-claim it — and over-claiming is the direction that would turn contamination into a result. `PairFamilyScore` carries the date, `aggregate(cutoff=...)` reports the same numbers over the post-cutoff rows beside the whole number and never instead of it, undated rows are counted on their own, and `publish` prints the slice with the cutoff it used. Without a configured cutoff every scoreboard says so rather than inventing one; `[learning].cutoff_date` has no default because guessing a model's training cutoff and publishing a contamination number from the guess would be worse than reporting none. Corpus dates as harvested: vibe-py all 2026, agent-vfc 67 of 69 in 2026, vfc spread 2014-2026, vfc-js 2014-2021. - Task 5.1 (2026-09-06, RED first: 3 of 9 tests in `tests/test_degradation_matrix.py`; then green, 8 after one test was deleted rather than kept). The three gate outputs are captured at `71efc9c` — the commit before the first line of this feature — as `benchmarks/measurements/2026-09-06-gate-baseline.txt`, and a test reproduces them byte for byte on every run, in both matrices. Until now byte-identity was something a reviewer checked by hand against a worktree; it is now mechanical and runs in CI, which is what makes it a gate rather than a habit. Six absences are each exercised once with the reason they record: HarnessX (`harnessx_unavailable`), the chat endpoint (`learning_endpoint_unavailable`, and the run still writes its artifact), the family configurations (`learning_configs_unavailable`), the network for pointer pairs (`pointer_pair_skipped`), the semantic extra (`language_unsupported` rather than a guess), and the sandbox (`sandbox_unavailable`, so a claim nothing could run is not confused with a claim that failed). One test written for this task was **deleted before the commit**: it asserted that this file contained a `without_` name per capability, which measures the file's own naming and nothing about the code. +- Review round 4 (2026-09-06) APPROVED 4.1, 4.2, 4.3; REJECTED the round-3 remediation on three counts. The reviewer re-derived the RED phase for group 4 (24 of 26 for 4.1, 3 for 4.2, 11 of 11 for 4.3 — **every count in the notes above was understated**, corrected here), killed 18 of 19 mutants, verified the 4.1 observable independently (2 of 10 access-control rows named at `7a33756`, 9 of 10 at `3e71e9e`, the tenth stating its reason), confirmed `_python_route_access` changed no access level anywhere in the corpus and `.mjs` changed exactly one snapshot, and reproduced the fixture-difficulty artifact number for number including its failing arm. Remediated RED-first: + - **The meta-agent proposer still could not propose.** `MetaAgent(inner_model=...)` takes a `ModelConfig`; it was handed a model name, a `# type: ignore` hid that from mypy, and a bare `except Exception` turned the resulting `AttributeError` into the degradation reason `meta_agent_failed` — which reads exactly like the model declining. Every other HarnessX composition site in the repository builds the config properly. `_meta_agent_class` and `_model_config` are now seams a test can pin without the extra, and a `TypeError`/`AttributeError`/`ImportError` from the call is re-raised: "we called the agent wrong" is not "the agent had nothing to propose". + - **`ousast pairs --hunter` crashed in its documented form.** The K floor landed with `--k-runs` still defaulting to 1, so the flag's absence raised `ValueError` from inside `evaluate_catalog`. The default is `None` (three), and a breach exits naming the flag. The covering test always passed `--k-runs 4`, which is why nothing noticed. + - **28 rows resolved a span on the vulnerable side and none on the fixed side** (12% of the corpus), so every fixed-side finding read as outside the labeled function and the pair scored as silence whatever the detector said — the same flattering direction as the range defect, one-seventh the size. `fix_function` reached only the harvester; it now reaches the catalog and `PairCase`, `_spans` uses it for the fixed side, and the catalog generator derives it from the fixed excerpt itself for the 25 vibe-py trap rows that never declared one. A row that still resolves no fixed-side span reports `unresolved_label:fixed` instead of scoring silent. Measured after: **0 rows**. + - Suggestions applied: one spelling per corpus fact everywhere (`PairCase.unscorable` verbatim, no invented prefix); leave-one-out records non-teachers withheld for tier or an unscorable reason, not only holdout refusals; the hygiene test asserts the declared twins still hash equal, so a declaration that stopped being true would not be invisible; `is_post_cutoff` refuses a cutoff coarser than a day, which was the one over-claim path that existed, and month precision is pinned; `UNSCORABLE_REASONS` covers the whole vocabulary rather than `identical_twin` alone; the zero-run score carries its date; `_pairs_named_by` filters by substring before the regex. diff --git a/benchmarks/pairs/catalog_gen.py b/benchmarks/pairs/catalog_gen.py index f673cbe..c5474d1 100644 --- a/benchmarks/pairs/catalog_gen.py +++ b/benchmarks/pairs/catalog_gen.py @@ -93,6 +93,9 @@ def catalog_text(slice_name: str, recipes: list[dict[str, object]]) -> str: lines.append(f'reviewer = "{_q(reviewer)}"') if recipe.get("known_limit"): lines.append(f'known_limit = "{_q(recipe["known_limit"])}"') + if recipe.get("fix_function"): + # Req 3.2: the fixed side of a trap pair is a different handler, and the scorer has to know which one + lines.append(f'fix_function = "{_q(recipe["fix_function"])}"') fix_date = _fix_date(recipe) if fix_date: lines.append(f'fix_date = "{_q(fix_date)}"') # Req 10.5: the post-cutoff slice needs a date to sort by @@ -399,6 +402,27 @@ def _stamp_function(path: Path, function: str) -> None: path.write_text(updated) +def derive_fix_function(slice_root: Path, recipe: dict[str, object]) -> str: + """The function the fixed excerpt actually holds, when it is not the labeled one. + + Real-Vuln-Benchmark traps take their fixed side from a different, correctly guarded handler of the same + snapshot. Without this the fixed side resolves no span, every finding there counts as outside the labeled + function, and the pair scores as silence instead of as a leak — always in the direction that flatters the + number (learning-harness Req 3.2).""" + _vuln, fixed = excerpt_rel(recipe) + path = slice_root / fixed + if not path.is_file(): + return "" + ranges = _ranges(path, _language(recipe)) + if not ranges: + return "" + labeled = str(recipe.get("function", "") or "") + names = [name for name, _start, _end in ranges] + if labeled in names or len(set(names)) != 1: + return "" # the fixed side holds the labeled function, or holds several and the recipe must say which + return names[0] + + def generate(slice_name: str, *, prune: bool = False) -> tuple[Path, int]: slice_root = ROOT / slice_name loaded = [dict(item) for item in tomllib.loads((slice_root / "recipes.toml").read_text()).get("recipe", [])] @@ -419,6 +443,10 @@ def generate(slice_name: str, *, prune: bool = False) -> tuple[Path, int]: recipe["function"] = derived _stamp_function(slice_root / vuln, derived) _stamp_function(slice_root / fixed, derived) + if not recipe.get("fix_function"): + derived = derive_fix_function(slice_root, recipe) + if derived: + recipe["fix_function"] = derived recipes.append(recipe) # `reviewer = "pending"` rows load at tier `title` (Req 9.4); no separate queue file else: print(f"skip {recipe['name']}: excerpt missing (harvest failed or not fetched)") diff --git a/benchmarks/pairs/vfc/catalog.toml b/benchmarks/pairs/vfc/catalog.toml index f1d9ed5..c92986c 100644 --- a/benchmarks/pairs/vfc/catalog.toml +++ b/benchmarks/pairs/vfc/catalog.toml @@ -492,6 +492,7 @@ fix_policy = "silent" provenance = "human" split = "holdout" review_tier = "advisory" +fix_function = "OSSL_TRACE_BEGIN" fix_date = "2025" [[pair.expected]] diff --git a/benchmarks/pairs/vibe-py/catalog.toml b/benchmarks/pairs/vibe-py/catalog.toml index 4d8a4fa..b46251d 100644 --- a/benchmarks/pairs/vibe-py/catalog.toml +++ b/benchmarks/pairs/vibe-py/catalog.toml @@ -2153,6 +2153,7 @@ fix_policy = "silent" provenance = "human" split = "holdout" review_tier = "seeded" +fix_function = "mutate" fix_date = "2026" [[pair.expected]] @@ -2183,6 +2184,7 @@ fix_policy = "silent" provenance = "human" split = "train" review_tier = "seeded" +fix_function = "mutate" fix_date = "2026" [[pair.expected]] @@ -2213,6 +2215,7 @@ fix_policy = "silent" provenance = "human" split = "holdout" review_tier = "seeded" +fix_function = "parse_field" fix_date = "2026" [[pair.expected]] @@ -2243,6 +2246,7 @@ fix_policy = "silent" provenance = "human" split = "train" review_tier = "seeded" +fix_function = "forgot_password" fix_date = "2026" [[pair.expected]] @@ -2273,6 +2277,7 @@ fix_policy = "silent" provenance = "human" split = "holdout" review_tier = "seeded" +fix_function = "parse_field" fix_date = "2026" [[pair.expected]] @@ -2333,6 +2338,7 @@ fix_policy = "silent" provenance = "human" split = "holdout" review_tier = "seeded" +fix_function = "run" fix_date = "2026" [[pair.expected]] @@ -2363,6 +2369,7 @@ fix_policy = "silent" provenance = "human" split = "train" review_tier = "seeded" +fix_function = "run" fix_date = "2026" [[pair.expected]] @@ -2424,6 +2431,7 @@ fix_policy = "silent" provenance = "human" split = "train" review_tier = "seeded" +fix_function = "get_by_username" fix_date = "2026" [[pair.expected]] @@ -2454,6 +2462,7 @@ fix_policy = "silent" provenance = "human" split = "holdout" review_tier = "seeded" +fix_function = "get_by_username" fix_date = "2026" [[pair.expected]] @@ -2484,6 +2493,7 @@ fix_policy = "silent" provenance = "human" split = "train" review_tier = "seeded" +fix_function = "get_by_username" fix_date = "2026" [[pair.expected]] @@ -2514,6 +2524,7 @@ fix_policy = "silent" provenance = "human" split = "holdout" review_tier = "seeded" +fix_function = "do_login" fix_date = "2026" [[pair.expected]] @@ -2544,6 +2555,7 @@ fix_policy = "silent" provenance = "human" split = "train" review_tier = "seeded" +fix_function = "do_login" fix_date = "2026" [[pair.expected]] @@ -2574,6 +2586,7 @@ fix_policy = "silent" provenance = "human" split = "holdout" review_tier = "seeded" +fix_function = "do_signup" fix_date = "2026" [[pair.expected]] @@ -3366,6 +3379,7 @@ fix_policy = "silent" provenance = "human" split = "train" review_tier = "seeded" +fix_function = "post" fix_date = "2026" [[pair.expected]] @@ -3397,6 +3411,7 @@ fix_policy = "silent" provenance = "human" split = "holdout" review_tier = "seeded" +fix_function = "post" fix_date = "2026" [[pair.expected]] @@ -3428,6 +3443,7 @@ fix_policy = "silent" provenance = "human" split = "train" review_tier = "seeded" +fix_function = "login" fix_date = "2026" [[pair.expected]] @@ -3495,6 +3511,7 @@ fix_policy = "silent" provenance = "human" split = "train" review_tier = "seeded" +fix_function = "register_user" fix_date = "2026" [[pair.expected]] @@ -3525,6 +3542,7 @@ fix_policy = "silent" provenance = "human" split = "holdout" review_tier = "seeded" +fix_function = "register_user" fix_date = "2026" [[pair.context]] @@ -3561,6 +3579,7 @@ fix_policy = "silent" provenance = "human" split = "train" review_tier = "seeded" +fix_function = "nosql_return_users_from_username" fix_date = "2026" [[pair.expected]] @@ -3591,6 +3610,7 @@ fix_policy = "silent" provenance = "human" split = "holdout" review_tier = "seeded" +fix_function = "nosql_return_users_from_username" fix_date = "2026" [[pair.expected]] @@ -3621,6 +3641,7 @@ fix_policy = "silent" provenance = "human" split = "train" review_tier = "seeded" +fix_function = "nosql_return_users_from_username" fix_date = "2026" [[pair.expected]] @@ -3651,6 +3672,7 @@ fix_policy = "silent" provenance = "human" split = "holdout" review_tier = "seeded" +fix_function = "get_post" fix_date = "2026" [[pair.expected]] @@ -3681,6 +3703,7 @@ fix_policy = "silent" provenance = "human" split = "train" review_tier = "seeded" +fix_function = "get_post" fix_date = "2026" [[pair.expected]] @@ -3711,6 +3734,7 @@ fix_policy = "silent" provenance = "human" split = "train" review_tier = "seeded" +fix_function = "_execute" fix_date = "2026" [[pair.expected]] @@ -3741,6 +3765,7 @@ fix_policy = "silent" provenance = "human" split = "holdout" review_tier = "seeded" +fix_function = "_execute" fix_date = "2026" [[pair.expected]] @@ -3802,6 +3827,7 @@ fix_policy = "silent" provenance = "human" split = "holdout" review_tier = "seeded" +fix_function = "get_posts" fix_date = "2026" [[pair.expected]] @@ -3832,6 +3858,7 @@ fix_policy = "silent" provenance = "human" split = "train" review_tier = "seeded" +fix_function = "get_posts" fix_date = "2026" [[pair.expected]] @@ -3862,6 +3889,7 @@ fix_policy = "silent" provenance = "human" split = "holdout" review_tier = "seeded" +fix_function = "get_posts" fix_date = "2026" [[pair.expected]] diff --git a/src/openultrasast/cli.py b/src/openultrasast/cli.py index 5e01bd8..fd582fc 100644 --- a/src/openultrasast/cli.py +++ b/src/openultrasast/cli.py @@ -189,7 +189,9 @@ def main(argv: list[str] | None = None) -> int: ) pairs.add_argument("--loo-out", type=Path, default=None, help="where to write loo.json (default: reports/loo.json)") pairs.add_argument("--json", action="store_true", help="print the pair scoreboard as JSON") - pairs.add_argument("--k-runs", type=int, default=1, help="runs per pair on the hunter path; the family scorer needs at least three") + pairs.add_argument( + "--k-runs", type=int, default=None, help="runs per pair on the hunter path; never fewer than three, and three by default" + ) subparsers.add_parser("mcp", help="run the narrow MCP server over stdio for OpenCode integration") mechanisms = subparsers.add_parser("mechanisms", help="maintainer: mechanism memory seeded from trusted pairs") @@ -982,7 +984,10 @@ def _pairs( from .learning.families import load_families scan = make_hunter_scan(client, model, taxonomy=load_families()) - result = evaluate_catalog(cases, hunter=scan, pointers=True if pointers else None, k_runs=k_runs) + try: + result = evaluate_catalog(cases, hunter=scan, pointers=True if pointers else None, k_runs=k_runs) + except ValueError as exc: + raise SystemExit(f"--k-runs: {exc}") from exc if json_out: print(json.dumps(result_payload(result), indent=2, sort_keys=True)) return 0 diff --git a/src/openultrasast/learning/proposer.py b/src/openultrasast/learning/proposer.py index d17a4d7..3fce6ce 100644 --- a/src/openultrasast/learning/proposer.py +++ b/src/openultrasast/learning/proposer.py @@ -201,6 +201,10 @@ class HarnessXProposer: before = _directory_text(workspace) try: hypothesis = call(workspace=workspace, facts=facts, model=self.model) + except (TypeError, AttributeError, ImportError): + # "we called the agent wrong" is not "the agent declined", and folding one into the other is how a + # broken composition reads as a quiet round for as long as nobody looks. + raise except Exception as exc: # noqa: BLE001 — an agent that failed proposed nothing, and the round says why self.reason = f"meta_agent_failed: {type(exc).__name__}" return None @@ -239,6 +243,26 @@ def _directory_text(directory: Path) -> dict[str, str]: return out +def _meta_agent_class() -> type: + """`MetaAgent` behind `harness_ext`. A seam, so the argument contract can be pinned without the extra.""" + from ..harness_ext import require_harnessx + + require_harnessx() + from harnessx.meta_harness.agent import MetaAgent # type: ignore[import-not-found] + + return MetaAgent # type: ignore[no-any-return] + + +def _model_config(model: str, provider: str = "anthropic") -> object: + """`MetaAgent(inner_model=...)` takes a `ModelConfig`, never a model name; the name goes to the provider.""" + from ..harness_ext import build_provider, require_harnessx + + require_harnessx() + from harnessx.core.model_config import ModelConfig # type: ignore[import-not-found] + + return ModelConfig(main=build_provider(model, provider)) + + def _meta_agent_call() -> MetaAgentCall: """Compose `MetaAgent.evolve` behind `harness_ext`, writing only inside the scratch copy. @@ -252,12 +276,10 @@ def _meta_agent_call() -> MetaAgentCall: def call(*, workspace: Path, facts: FailureFacts, model: str) -> str: import asyncio - from harnessx.meta_harness.agent import MetaAgent # type: ignore[import-not-found] - trajectories = workspace.parent / "trajectories" trajectories.mkdir(parents=True, exist_ok=True) (trajectories / "facts.json").write_text(facts_prompt(facts), encoding="utf-8") - agent = MetaAgent(inner_model=model, allowed_write_roots=(workspace,)) # type: ignore[arg-type] + agent = _meta_agent_class()(inner_model=_model_config(model), allowed_write_roots=(workspace,)) output = workspace.parent / "meta-out" output.mkdir(parents=True, exist_ok=True) asyncio.run(agent.evolve(current_config=workspace, trajectories_dir=trajectories, output_dir=output)) diff --git a/src/openultrasast/learning/rounds.py b/src/openultrasast/learning/rounds.py index cc5adc3..0971d39 100644 --- a/src/openultrasast/learning/rounds.py +++ b/src/openultrasast/learning/rounds.py @@ -457,11 +457,13 @@ def _pairs_named_by(proposal: Proposal, cases: Sequence[PairCase]) -> tuple[str, A memory entry that quotes a held-out pair is the loop learning the answer instead of the shape, and it looks exactly like a legitimate hard negative until the holdout number is read as if it meant something.""" haystack = "\n".join([*proposal.predicted_affected, *proposal.predicted_at_risk, *proposal.change.values()]) - named = [] - for case in cases: - name = str(getattr(case, "name", "")) - if name and re.search(rf"(? bool: A year-precision row counts only when the whole year is after the cutoff, so a coarse date can under-claim the clean slice but never over-claim it — and over-claiming is the one that would turn contamination into a result. """ - if not fix_date or not cutoff: - return False - earliest = fix_date if len(fix_date) == 10 else f"{fix_date[:4]}-01-01" + if not fix_date or len(cutoff) != 10: + return False # a cutoff coarser than a day is the one over-claim path there is, so it is refused outright + earliest = fix_date if len(fix_date) == 10 else f"{fix_date}-01-01"[:10] return earliest > cutoff @@ -96,7 +96,22 @@ def score_pair_family( unscorable_reason=reason, ) spans = _spans(case, ranges) - fixed_spans = _spans(case, fixed_ranges) if fixed_ranges is not None else spans + if fixed_ranges is None: + fixed_spans = spans + else: + # A trap pair's fixed side holds a different, correctly guarded handler; `fix_function` names it. Without a + # span there, every fixed-side finding reads as outside the labeled function and the pair scores as silence. + fixed_spans = _spans(case, fixed_ranges, function=str(getattr(case, "fix_function", "") or "")) + if not fixed_spans: + return PairFamilyScore( + pair=name, + family=family, + slice=slice_name, + fix_date=str(getattr(case, "fix_date", "") or ""), + runs=(), + outcome="unscorable", + unscorable_reason="unresolved_label:fixed", + ) outcomes: list[FamilyOutcome] = [] other_vuln = other_fixed = fabricated_vuln = fabricated_fixed = 0 rung: Rung = "suspicion" @@ -112,7 +127,15 @@ def score_pair_family( rung = _best_rung(rung, hits_vuln) outcomes.append(_outcome(bool(hits_vuln), bool(hits_fixed))) if not outcomes: - return PairFamilyScore(pair=name, family=family, slice=slice_name, runs=(), outcome="unscorable", unscorable_reason="no_runs") + return PairFamilyScore( + pair=name, + family=family, + slice=slice_name, + fix_date=str(getattr(case, "fix_date", "") or ""), + runs=(), + outcome="unscorable", + unscorable_reason="no_runs", + ) majority = Counter(outcomes).most_common(1)[0][0] return PairFamilyScore( pair=name, @@ -170,9 +193,9 @@ def unscorable_reason( return None -def _spans(case: object, ranges: FunctionRanges) -> tuple[tuple[str, int, int], ...]: - """The (path, start, end) spans of the labeled functions, from the parsed vulnerable side.""" - wanted = {str(getattr(row, "function", "") or "") for row in getattr(case, "expected", ()) or ()} +def _spans(case: object, ranges: FunctionRanges, *, function: str = "") -> tuple[tuple[str, int, int], ...]: + """The (path, start, end) spans of the labeled functions. ``function`` overrides the label for the fixed side.""" + wanted = {function} if function else {str(getattr(row, "function", "") or "") for row in getattr(case, "expected", ()) or ()} wanted.discard("") found: list[tuple[str, int, int]] = [] for path, entries in ranges.items(): diff --git a/src/openultrasast/pairs.py b/src/openultrasast/pairs.py index 5c14e77..e25b146 100644 --- a/src/openultrasast/pairs.py +++ b/src/openultrasast/pairs.py @@ -131,6 +131,12 @@ class PairCase: unscorable: str | None = None # computed: identical_twin, known_limit: (learning-harness Req 4.1) split_declared: bool = True # False when the catalog row named no split, so the split helper may assign one fix_date: str = "" # ISO date of the fixing commit; the catalog generator fills it, the split helper orders by it + # The function the *fixed* side actually holds, when it is not the labeled one. Real-Vuln-Benchmark traps take + # their fixed side from a different, correctly guarded handler of the same snapshot, and without this the fixed + # side resolves no span at all: every finding there reads as outside the labeled function, so the pair scores as + # silence rather than as a leak (learning-harness Req 3.2, 4.1). + fix_function: str = "" + recipe: tuple[tuple[str, object], ...] = () # pointer pairs carry their harvest recipe instead of excerpts # (relpath, vuln document, fixed document): files that register the handler from outside its own module — an # OpenAPI operation, a router table. Materialized beside the excerpt so the mapper can name the handler at all @@ -292,6 +298,7 @@ def _load_catalog_file(path: Path) -> tuple[PairCase, ...]: unscorable=_unscorable_reason(known_limit, vuln_file, fixed_file, vendored=vendored), split_declared="split" in item, fix_date=str(item.get("fix_date", "")), + fix_function=str(item.get("fix_function", "")), review_tier=review_tier, reviewer=reviewer, vendored=vendored, @@ -625,8 +632,10 @@ def _body_digest(path: Path) -> str | None: return hashlib.sha256(body.encode()).hexdigest() if body else None -# The reasons a row may carry verbatim from the catalog, because the loader computes them under the same names. -UNSCORABLE_REASONS = frozenset({"identical_twin"}) +# The reasons a row may carry verbatim from the catalog, because something else computes them under the same +# names. A maintainer declaring one of these is stating the fact the computation would have found, so the two must +# not end up as two buckets in the same report (learning-harness Req 4.1, 10.3). +UNSCORABLE_REASONS = frozenset({"identical_twin", "unsupported_language", "unresolved_label", "missing_context"}) def _unscorable_reason(known_limit: object, vuln_file: Path, fixed_file: Path, *, vendored: bool) -> str | None: diff --git a/src/openultrasast/semantic/loo.py b/src/openultrasast/semantic/loo.py index 90cbd5a..3c1c67b 100644 --- a/src/openultrasast/semantic/loo.py +++ b/src/openultrasast/semantic/loo.py @@ -93,20 +93,31 @@ def evaluate_loo(cases: Sequence[PairCase], *, facts: SemanticFacts | None = Non targets: list[PairCase] = [] skipped: list[tuple[str, str]] = [] for case in cases: - if case.known_limit: - skipped.append((case.name, f"known_limit:{case.known_limit}")) + if case.unscorable or case.known_limit: + # `PairCase.unscorable` verbatim when the loader computed one: one spelling per corpus fact, and it + # already carries the `known_limit:` prefix when the limit is a maintainer's own word rather than one + # of the vocabulary's. A case built by hand may carry only the declaration. + skipped.append((case.name, case.unscorable or f"known_limit:{case.known_limit}")) elif not case.vuln_file.is_file() or not case.fixed_file.is_file(): skipped.append((case.name, "pointer_pair_not_cached" if not case.vendored else "excerpt_missing")) else: targets.append(case) - from ..learning.split import is_teacher, refuse_if_holdout + from ..learning.split import GATING_TIERS, holdout_names, is_teacher, refuse_if_holdout # learning-harness Req 5.1: every pair is still scored, but only a train-split teacher may seed a store. lessons = {case.name: pair_lessons(case, loaded) for case in targets if is_teacher(case)} outcomes = [_hold_out(case, targets, lessons, loaded) for case in targets] - # Req 5.2: the pairs that were kept out of the teaching set are named, not silently dropped. - refusal = refuse_if_holdout([case.name for case in targets if case.name not in lessons], targets) + # Req 5.2: the holdout pairs kept out of the teaching set are named, not silently dropped. Non-teachers held + # back for their tier or their unscorable reason are already in `skipped` with that reason. + excluded = [case.name for case in targets if case.name not in lessons] + refusal = refuse_if_holdout(excluded, targets) refusals = (refusal.degradation(),) if refusal is not None else () + withheld = tuple( + (case.name, "tier" if case.review_tier not in GATING_TIERS else "unscorable") + for case in targets + if case.name in set(excluded) - set(holdout_names(targets)) + ) + skipped.extend((name, f"not_a_teacher:{why}") for name, why in withheld) return LooResult( outcomes=tuple(outcomes), per_slice=_group(outcomes, lambda item: (item.slice,)), diff --git a/src/openultrasast/semantic/seed.py b/src/openultrasast/semantic/seed.py index 97ffd71..c3e039e 100644 --- a/src/openultrasast/semantic/seed.py +++ b/src/openultrasast/semantic/seed.py @@ -197,12 +197,12 @@ def _skip_reason(case: PairCase) -> str | None: if case.review_tier not in GATING_TIERS: return f"tier:{case.review_tier}" - if case.unscorable: - # One spelling per corpus fact. `identical_twin` declared in the catalog and `identical_twin` computed from - # the bytes are the same reason, and calling one of them `split:train` is simply untrue (Req 5.1, 10.3). - return f"unscorable:{case.unscorable}" - if case.known_limit: - return f"known_limit:{case.known_limit}" + if case.unscorable or case.known_limit: + # `PairCase.unscorable` verbatim when the loader computed one, the same string every other path reports: + # `identical_twin` declared in the catalog and `identical_twin` computed from the bytes are one fact, and + # calling either of them `split:train` is simply untrue. A case built by hand may carry only the + # declaration (Req 5.1, 10.3). + return case.unscorable or f"known_limit:{case.known_limit}" if not is_teacher(case) and case.vendored: return f"split:{case.split}" if not case.vuln_file.is_file() or not case.fixed_file.is_file(): diff --git a/tests/test_learning_cutoff.py b/tests/test_learning_cutoff.py index 63398e0..853026b 100644 --- a/tests/test_learning_cutoff.py +++ b/tests/test_learning_cutoff.py @@ -44,6 +44,9 @@ def test_every_vendored_row_carries_the_date_its_recipe_recorded() -> None: ("2026", "2026-03-01", False), # a year-precision row that might be before it is not claimed ("2026-04-02", "2026-03-01", True), ("2024", "2025-07-01", False), + ("2026-05", "2026-03-01", True), # month precision: the earliest day of May is after 1 March + ("2026-05", "2026-06-01", False), # and before 1 June, so the row is not claimed + ("2025", "2025", False), # a cutoff coarser than a day is refused rather than compared ("", "2025-07-01", False), ], ) diff --git a/tests/test_learning_refusals.py b/tests/test_learning_refusals.py index 237d3e7..16ca7fe 100644 --- a/tests/test_learning_refusals.py +++ b/tests/test_learning_refusals.py @@ -45,7 +45,7 @@ def test_a_train_split_twin_is_refused_for_being_a_twin_not_for_its_split(tmp_pa `vulnpy-deserialization-do-pickle-load` are train-split identical twins.""" from openultrasast.semantic.seed import _skip_reason - assert _skip_reason(_case(tmp_path, "twin", split="train", unscorable="identical_twin")) == "unscorable:identical_twin" + assert _skip_reason(_case(tmp_path, "twin", split="train", unscorable="identical_twin")) == "identical_twin" assert _skip_reason(_case(tmp_path, "held", split="holdout")) == "split:holdout" assert _skip_reason(_case(tmp_path, "good", split="train")) is None diff --git a/tests/test_learning_round4_evidence.py b/tests/test_learning_round4_evidence.py new file mode 100644 index 0000000..bb971b4 --- /dev/null +++ b/tests/test_learning_round4_evidence.py @@ -0,0 +1,247 @@ +"""Three defects the round-4 review found, one test each (learning-harness, Req 3.2, 4.1, 9.1). + +Two of them were the same shape as defects already rejected once: a proposer that cannot propose because the call +is wrong and the error is swallowed, and a K-run floor that turned a documented command into a traceback. The +third is the fixed-side half of the range defect — 28 rows resolve a span on the vulnerable side and none on the +fixed side, so every finding there reads as outside the labeled function and the pair scores as silence. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from openultrasast.benchmark import ExpectedFinding +from openultrasast.findings import StaticFinding +from openultrasast.harness_ext import has_harnessx +from openultrasast.learning.families import load_families +from openultrasast.pairs import DEFAULT_CATALOG, PairCase, load_pair_catalog, select_vendored +from openultrasast.semantic.extra import has_semantic_extra + +VULN = "from flask import request\nimport os\n\n\ndef run():\n os.system(request.args.get('cmd'))\n" +FIXED_OTHER = "from flask import request\n\n\ndef unrelated():\n return 'ok'\n" + + +def _finding(line: int) -> StaticFinding: + return StaticFinding( + finding_id=f"tool-hunter:app.py:{line}", + path="app.py", + title="t", + severity="high", + confidence="low", + evidence_level="suspicion", + rationale="r", + line=line, + function_name="run", + reachability_status="unknown", + reachability_evidence=[], + reachability_conditions=[], + tags=["tool-hunter", "family:injection"], + ranking_priority=0.0, + ) + + +# --- the meta-agent call ----------------------------------------------------- + + +@pytest.mark.skipif(not has_harnessx(), reason="the composition seam only runs when the extra is importable") +def test_the_meta_agent_is_built_with_a_model_config_not_a_model_name(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """`MetaAgent(inner_model=...)` wants a `ModelConfig`; a string reaches `self.inner_model.agentic(...)` and dies. + + Every other HarnessX composition site in this repository builds the config properly. Here the mistake was + invisible because a `# type: ignore` hid it from mypy and a bare `except Exception` turned it into the + degradation reason `meta_agent_failed`, which reads exactly like the model declining.""" + from openultrasast.learning import proposer as proposer_module + + seen: dict[str, object] = {} + + class _Agent: + def __init__(self, **kwargs: object) -> None: + seen.update(kwargs) + + async def evolve(self, **kwargs: object) -> Path: + seen["evolve"] = kwargs + workspace = kwargs["current_config"] + assert isinstance(workspace, Path) + (workspace / "checklist.md").write_text("- from the meta agent\n") + return workspace + + monkeypatch.setattr(proposer_module, "_meta_agent_class", lambda: _Agent) + monkeypatch.setattr(proposer_module, "_model_config", lambda model: f"ModelConfig({model})") + (tmp_path / "injection").mkdir() + (tmp_path / "injection" / "checklist.md").write_text("- old\n") + proposal = proposer_module.HarnessXProposer(configs_dir=tmp_path, model="m").propose(_facts()) + assert proposal is not None and proposal.change == {"checklist.md": "- from the meta agent\n"} + assert seen["inner_model"] == "ModelConfig(m)", "the agent was handed a model name where it wanted a config" + assert seen["allowed_write_roots"] != (tmp_path / "injection",), "the live family directory is never a write root" + + +@pytest.mark.skipif(not has_harnessx(), reason="the composition seam only runs when the extra is importable") +def test_a_meta_agent_built_wrongly_is_not_reported_as_a_declined_proposal(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """ "We called the agent wrong" and "the agent had nothing to propose" are different facts.""" + from openultrasast.learning import proposer as proposer_module + + def _boom() -> object: + raise TypeError("inner_model must be a ModelConfig") + + monkeypatch.setattr(proposer_module, "_meta_agent_class", _boom) + (tmp_path / "injection").mkdir() + proposer = proposer_module.HarnessXProposer(configs_dir=tmp_path, model="m") + with pytest.raises(TypeError, match="ModelConfig"): + proposer.propose(_facts()) + + +def _facts(): # type: ignore[no-untyped-def] + from openultrasast.learning.detectors import FamilyConfig + from openultrasast.learning.proposer import FailureFacts + + return FailureFacts( + family="injection", + config=FamilyConfig( + family="injection", + version="0", + prompt="p", + checklist="- old", + tools=("read_file",), + max_steps=1, + max_cost_usd=0.1, + max_chars=100, + ), + ) + + +# --- the pairs command in its documented form -------------------------------- + + +def test_the_pairs_hunter_path_runs_without_a_k_runs_flag(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """The floor is a default, not a trap: `ousast pairs --hunter` must work as documented.""" + from openultrasast.cli import main + + (tmp_path / "a-v.py").write_text(VULN) + (tmp_path / "a-f.py").write_text(FIXED_OTHER) + (tmp_path / "pairs.toml").write_text( + '[[pair]]\nname = "a"\nslice = "vibe-py"\nlanguage = "python"\nvuln = "a-v.py"\nfixed = "a-f.py"\n' + 'relpath = "app.py"\nsplit = "train"\nreview_tier = "seeded"\n\n' + '[[pair.expected]]\ncwe = "CWE-78"\nclass = "x"\npath = "app.py"\nfunction = "run"\nmechanism = "other"\nfamily = "injection"\n' + ) + monkeypatch.setenv("OPENULTRASAST_HUNTER_CLIENT", "scripted") + assert main(["pairs", "--catalog", str(tmp_path / "pairs.toml"), "--hunter", "--hunter-model", "scripted"]) == 0 + + +def test_a_k_below_the_floor_names_the_flag_instead_of_raising(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + from openultrasast.cli import main + + (tmp_path / "pairs.toml").write_text("") + monkeypatch.setenv("OPENULTRASAST_HUNTER_CLIENT", "scripted") + with pytest.raises(SystemExit, match="k-runs"): + main(["pairs", "--catalog", str(tmp_path / "pairs.toml"), "--hunter", "--hunter-model", "scripted", "--k-runs", "1"]) + + +# --- the fixed side of a trap pair ------------------------------------------- + + +def test_a_trap_pair_is_scored_against_the_function_its_fixed_side_really_holds(tmp_path: Path) -> None: + """A Real-Vuln-Benchmark trap takes its fixed side from a different, correctly guarded handler. + + With no span for it, every fixed-side finding counts as outside the labeled function and the pair scores as + silence — the same flattering direction as the range defect, for the 28 rows that resolve one side only.""" + from openultrasast.learning.scoring import score_pair_family + + (tmp_path / "v.py").write_text(VULN) + (tmp_path / "f.py").write_text(FIXED_OTHER) + case = PairCase( + name="trap", + slice="vibe-py", + language="python", + origin="test", + vuln_file=tmp_path / "v.py", + fixed_file=tmp_path / "f.py", + relpath="app.py", + expected=( + ExpectedFinding( + cwe="CWE-78", vulnerability_class="x", path="app.py", evidence="", function="run", mechanism="other", family="injection" + ), + ), + min_recall=1.0, + fix_policy="silent", + fix_function="unrelated", + ) + score = score_pair_family( + case, + "injection", + [[_finding(6)]], + [[_finding(5)]], + ranges={"app.py": (("run", 5, 6),)}, + fixed_ranges={"app.py": (("unrelated", 4, 5),)}, + taxonomy=load_families(), + ) + assert score.outcome == "both_flagged", "the finding is inside the fixed side's own handler; that is a leak" + + +def test_a_row_whose_fixed_side_resolves_no_span_says_so_instead_of_scoring_silent(tmp_path: Path) -> None: + from openultrasast.learning.scoring import score_pair_family + + (tmp_path / "v.py").write_text(VULN) + (tmp_path / "f.py").write_text(FIXED_OTHER) + case = PairCase( + name="orphan", + slice="vibe-py", + language="python", + origin="test", + vuln_file=tmp_path / "v.py", + fixed_file=tmp_path / "f.py", + relpath="app.py", + expected=( + ExpectedFinding( + cwe="CWE-78", vulnerability_class="x", path="app.py", evidence="", function="run", mechanism="other", family="injection" + ), + ), + min_recall=1.0, + fix_policy="silent", + ) + score = score_pair_family( + case, + "injection", + [[_finding(6)]], + [[]], + ranges={"app.py": (("run", 5, 6),)}, + fixed_ranges={"app.py": (("unrelated", 4, 5),)}, + taxonomy=load_families(), + ) + assert score.outcome == "unscorable" and score.unscorable_reason == "unresolved_label:fixed" + + +def test_the_trap_rows_carry_their_fixed_side_function_in_the_catalog() -> None: + """`fix_function` lived only in the recipes; the scorer could not see which handler the fixed side holds.""" + cases = {case.name: case for case in select_vendored(load_pair_catalog(DEFAULT_CATALOG))} + assert cases["threatbyte-api-v1-get"].fix_function == "post" + assert cases["threatbyte-api-v1-delete"].fix_function == "post" + assert cases["vampi-users-update-password"].fix_function == "register_user" + assert cases["vampi-books-get-by-title"].fix_function == "", "a row whose two sides are one function names none" + + +@pytest.mark.semantic +@pytest.mark.skipif(not has_semantic_extra(), reason="resolving a span on both sides needs the tree-sitter grammars") +def test_no_vendored_row_resolves_one_side_and_not_the_other() -> None: + """The corpus-wide form of the same defect: 28 rows resolved a vulnerable span and no fixed-side span. + + Every one of them therefore scored the fixed side as silent by construction, whatever the detector said. The + catalog generator now derives `fix_function` from the fixed excerpt itself, so the span exists and the leak is + visible.""" + import tempfile + + from openultrasast.learning.scoring import _spans + from openultrasast.pairs import _materialize_side, _overlay_scan + + orphans = [] + for case in select_vendored(load_pair_catalog(DEFAULT_CATALOG)): + if case.unscorable: + continue + with tempfile.TemporaryDirectory() as scratch: + vuln = _materialize_side(Path(scratch) / "vuln", case, side="vuln") + fixed = _materialize_side(Path(scratch) / "fixed", case, side="fixed") + vuln_ranges, fixed_ranges = _overlay_scan(vuln).ranges, _overlay_scan(fixed).ranges + if _spans(case, vuln_ranges) and not _spans(case, fixed_ranges, function=case.fix_function): + orphans.append(case.name) + assert orphans == [], f"rows whose fixed side can only ever score as silence: {orphans}" diff --git a/tests/test_loo.py b/tests/test_loo.py index c60c96f..8975adf 100644 --- a/tests/test_loo.py +++ b/tests/test_loo.py @@ -89,7 +89,9 @@ def test_leave_one_out_seeds_only_trusted_pairs_and_skips_known_limit(tmp_path: assert set(by_name) == {"a", "b"} # known_limit excluded entirely assert not by_name["a"].detected # b is advisory: it cannot teach a assert by_name["b"].detected # a (seeded) teaches b, which is scored as a held-out target - assert result.skipped == (("c", "known_limit:engine"),) + # `b` is excluded from teaching for its tier and says so; without that line a pair that silently stopped + # teaching would look identical to one that never could (Req 5.2). + assert result.skipped == (("c", "known_limit:engine"), ("b", "not_a_teacher:tier")) def test_cli_pairs_loo_writes_the_artifact_offline(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: diff --git a/tests/test_obligation_export.py b/tests/test_obligation_export.py index deaa1ff..7846569 100644 --- a/tests/test_obligation_export.py +++ b/tests/test_obligation_export.py @@ -133,7 +133,7 @@ def test_the_train_split_of_vibe_py_teaches_no_obligation_shape_today(tmp_path: assert refused["threatbyte-api-v1-get"] == "split:holdout" # Its own reason, not its split: this row is a byte-identical twin that happens to sit on the holdout side, # and reporting `split:holdout` would hide a corpus fact behind a split fact (Req 5.2). - assert refused["vampi-books-get-by-title"] == "unscorable:identical_twin" + assert refused["vampi-books-get-by-title"] == "identical_twin" flow_rows = [row for row in store.load() if row.shape is not None and row.shape.get("family", "sink") == "sink"] assert report.records == len(flow_rows) and len(flow_rows) >= 1 # the flow family still teaches from the train split diff --git a/tests/test_pair_hygiene.py b/tests/test_pair_hygiene.py index eaa0876..f5d9971 100644 --- a/tests/test_pair_hygiene.py +++ b/tests/test_pair_hygiene.py @@ -26,6 +26,13 @@ def test_identical_twins_are_unscorable_and_no_row_is_dropped() -> None: assert per_slice == VENDORED_PER_SLICE # nothing dropped by the new accounting twins = {case.name for case in cases if case.unscorable == "identical_twin"} assert twins == IDENTICAL_TWINS + # The declaration and the bytes must still agree. A declared limit that stopped being true after a re-harvest + # would otherwise be invisible: the mirror image of the risk declaring it was meant to close. + from openultrasast.pairs import _body_digest + + declared = [case for case in cases if case.known_limit == "identical_twin"] + assert {case.name for case in declared} == IDENTICAL_TWINS + assert all(_body_digest(case.vuln_file) == _body_digest(case.fixed_file) for case in declared) assert all(case.unscorable is None for case in cases if case.name not in IDENTICAL_TWINS and not case.known_limit)