fix(model-grounded-detection): the dominance question contradicted its own evidence

vampi-users-update-password scored both_silent while the arbiter returned
model_corroborated on the vulnerable side and nothing on the fixed one -- so the
finding existed and the pipeline threw it away.

The residual question was written for the ENTAILED case, but entailed returns
before the judge is ever consulted, so the question is only ever asked for
CORROBORATED. It asserted "in a file where sibling handlers are guarded" and
asked about "the guard its siblings have", printed directly above a witness
reading "and no guarded sibling" -- because corroborated means precisely that
no sibling is guarded. The judge was handed a prompt contradicting its own
evidence line and reasonably answered no.

The question now describes what corroborated actually is: an obligated
operation no guard governs, with no sibling evidence either way, so the model
cannot tell from consistency alone whether the guard is missing or the module
is deliberately public. That is the real open question and the one worth asking.

Verified end to end against a live endpoint: vuln model_corroborated, fixed
silent, the pair splits.

The taint and config questions were checked for the same defect and are
correct -- each describes the corroborated case it is actually asked about.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-08 19:22:55 +02:00
co-authored by Claude Opus 5
parent c74e101d10
commit 3fe7b9ded7
2 changed files with 27 additions and 4 deletions
+18
View File
@@ -195,3 +195,21 @@ def test_a_corroborated_absence_finding_survives_a_judge_that_agrees() -> None:
{"operation": "q.filter_by(id=n)", "opLine": "3", "opMethod": "safe", "dominatingGuards": ["current_user"]}]
findings = scan_region(_cpg(rows), _dominance_spec(), function="leaky", client=_Client(), model="m", candidates=())
assert len(findings) == 1 and findings[0].rung is Rung.ENTAILED
def test_the_dominance_question_describes_the_evidence_the_model_actually_has() -> None:
"""The residual question is only ever asked for CORROBORATED, never ENTAILED -- entailed returns early.
It was written for the entailed case, so it asserted "in a file where sibling handlers are guarded" and
asked about "the guard its siblings have", directly above a witness reading "and no guarded sibling".
The judge, handed a prompt contradicting its own evidence line, answered no and the finding was dropped:
vampi-users-update-password scored both_silent while the arbiter returned corroborated on the vulnerable
side and nothing on the fixed one.
"""
from openultrasast.model.pipeline import residual_question
witness = "update_password line 16: obligated operation with no discharging guard, and no guarded sibling"
q = residual_question(_dominance_spec(), witness, "update_password")
assert "siblings have" not in q, "the corroborated case has no guarded siblings to appeal to"
assert "sibling handlers are guarded" not in q
assert "guard" in q.lower() and "public" in q.lower(), "it must still ask whether the operation needs a guard"