fix(learning-harness): the three defects the group 2 review found

Task 2.3: two of the five unscorable reasons had no code path, so an unparsable
row was reported as a corpus labelling defect when the true cause was a tooling
gap. They have different owners and different fixes, and conflating them is the
dishonesty this spec exists to remove. The design's unscorable_reason is
reinstated with the parse result as an input, and a handler that arrived without
its registration is now named as missing context for the families whose bug
shape is a missing check.

Task 2.6: a verification stamped its own word for proof onto the shared evidence
level, which is not a value the project's ladder defines, and three existing
helpers raised on it. Nothing in production called it yet, so the suite stayed
green while the wiring in the next group would have broken a scan. A
verification now writes only its tag, and a caller that wants to move a claim up
the ladder goes through an explicit mapping that can only produce a level the
ladder has.

Task 2.7: the injection canary planted its row in a private in-memory database
the target could never reach, so it could not fire; the fake sandbox handed the
token back regardless of the snippet, which is why nothing noticed. It is now an
execution canary whose marker only an interpreter could write, and each template
is tested by running it for real against a vulnerable fixture and its fixed
twin. That test immediately caught a second flaw: a handler that merely echoes
its input printed the oracle token and satisfied the oracle without any
vulnerability, so an injected payload now carries a value unrelated to the token
the oracle looks for.

Also from the review: a configuration with no tools is refused instead of
producing a detector that silently reports nothing, the classifier abstains when
its fallback call fails too, and the archive docstring says which version wins.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-06 10:32:02 +02:00
co-authored by Claude Opus 5
parent e44d45f488
commit b0dba26468
10 changed files with 268 additions and 37 deletions
+33
View File
@@ -150,3 +150,36 @@ def test_the_rung_is_the_best_verifier_outcome_among_the_detections(tmp_path: Pa
assert score.outcome == "pair_correct" and score.rung == "static_corroboration"
plain = score_pair_family(_case(tmp_path), "access_control", [[_finding("access_control", 12)]], [[]], ranges=RANGES, taxonomy=taxonomy)
assert plain.rung == "suspicion"
def test_the_unscorable_reason_tells_a_tooling_gap_from_a_corpus_defect(tmp_path: Path) -> None:
"""Req 4.1: an unparsable row is a tooling gap; a row whose labeled function no range names is a corpus defect."""
from openultrasast.learning.scoring import unscorable_reason
case = _case(tmp_path)
assert unscorable_reason(case, parse_ok=True, ranges=RANGES) is None
assert unscorable_reason(case, parse_ok=False, ranges={}) == "unsupported_language"
assert unscorable_reason(case, parse_ok=False, ranges=RANGES) == "unsupported_language" # the parse decides, not the ranges
assert unscorable_reason(case, parse_ok=True, ranges={}) == "unresolved_label"
assert unscorable_reason(_case(tmp_path, unscorable="identical_twin"), parse_ok=True, ranges=RANGES) == "identical_twin"
twin = _case(tmp_path, unscorable="known_limit:java hash")
assert unscorable_reason(twin, parse_ok=False, ranges={}) == "known_limit:java hash" # a declared limit wins
def test_a_row_whose_registration_context_is_missing_is_named_as_such(tmp_path: Path) -> None:
"""Req 4.1: an access-control row whose handler arrived without its registration cannot be scored for that reason."""
from openultrasast.learning.scoring import unscorable_reason
case = _case(tmp_path, family="access_control")
assert unscorable_reason(case, parse_ok=True, ranges=RANGES, entry_points=("leaky",)) is None
assert unscorable_reason(case, parse_ok=True, ranges=RANGES, entry_points=()) == "missing_context"
# only families whose bug shape needs the registration are judged on it
assert unscorable_reason(_case(tmp_path, family="injection"), parse_ok=True, ranges=RANGES, entry_points=()) is None
def test_an_unparsable_side_is_scored_unscorable_rather_than_missed(tmp_path: Path) -> None:
from openultrasast.learning.scoring import score_pair_family
taxonomy = load_families()
score = score_pair_family(_case(tmp_path), "access_control", [[]], [[]], ranges={}, taxonomy=taxonomy, parse_ok=False)
assert score.outcome == "unscorable" and score.unscorable_reason == "unsupported_language"