feat(model-grounded-detection): group 3, the evidence ladder and the judge

3.1 adds StaticFinding.rung beside evidence_level rather than renaming it. The
existing evidence_level carries `static_corroboration`, emitted by the flat IR
overlay -- a path Req 4.4 demotes to the suspicion band. Restating that as
model_corroborated would claim a CPG arbitrated something it never saw, and
rewriting every legacy finding to suspicion would be a product-visible
regression nobody asked for. So `rung` defaults to suspicion and is raised only
by ladder.at_rung(finding, verdict), which is the structural form of Req 5.4:
nothing rises above suspicion without a Verdict behind it. The witness travels
with the rung, because a rung without its evidence is just a louder assertion.
It reaches markdown, JSON and SARIF.

3.2 is the judge, and it is defined as much by what it lacks. Three exhaustive
outcomes: the model entails (report it, never call the model), the model
contradicts (drop the claim, record why), the model cannot decide (report
suspicion). The LLM gets one bounded typed question with tools=[] so it cannot
search for the site, asked only about the residual the graph cannot settle --
whether a sanitizer on a REAL flow suffices. Its answer can only confirm a flow
the CPG already found; it can never conjure one.

Req 8.5 is asserted structurally: a test parses judge.py and fails on any AST
name like k_runs/majority/vote/average/acceptance/floor/budget, any for/while
loop, or an import from the deleted round machinery. That test first matched
the word "votes" in the module's own docstring, which is why it now inspects
code rather than text.

Verified live against a real CPG, not only scripted rows:
  vuln   rung=model_entailed   request.args["name"] -> db.execute(...)
  fixed  rung=suspicion        no source reaches this sink
  model calls made: 0

Both sides arbitrated by the graph with the LLM never consulted -- the thing
the prior architecture could not do at any price.

678 passing, mypy/ruff clean, gates byte-identical, zero orphaned modules.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG

src/openultrasast/findings.py

@@ -29,6 +29,10 @@ class StaticFinding:
     ranking_priority: float
     status: str = "enabled"
     proposed_snippet: str | None = None
+    # What ESTABLISHED this finding (model-grounded-detection Req 5). Defaults to `suspicion` and is raised
+    # only by `model.ladder.at_rung` with a Verdict behind it, so nothing can claim the model arbitrated a
+    # claim the model never saw. Distinct from `evidence_level`, which is the older overlay pipeline's word.
+    rung: str = "suspicion"
 
 
 SEVERITY_LABEL = {5: "critical", 4: "high", 3: "medium", 2: "low", 1: "low", 0: "info"}

src/openultrasast/model/audit.py

@@ -33,6 +33,7 @@ SPEC_OWNED = {
     "model.audit": "model-grounded-detection Req 3: this classifier; run by the maintainer, not by the scan",
     "model.ladder": "model-grounded-detection Req 5: the evidence ladder every finding carries",
     "model.taint": "model-grounded-detection Req 6.1: taint reachability over the CPG, the flow-family arbiter",
+    "model.judge": "model-grounded-detection Req 8: one bounded question, checked against the model",
     "cpg": "model-grounded-detection Req 4: the CPG seam package",
     "cpg.backend": "model-grounded-detection Req 4.1/4.5: the single Joern subprocess boundary",
     "cpg.capability": "model-grounded-detection Req 4.2/4.3: the engine probe that degrades to suspicion",

src/openultrasast/model/judge.py

@@ -0,0 +1,128 @@
+"""The LLM proposes; the model disposes (model-grounded-detection, Req 8).
+
+This module is where the prior architecture's premise is inverted. There, the LLM was an oracle whose noise
+had to be averaged over K runs, floored per family, and gated by a sign test. Here it answers one bounded,
+typed question about a candidate it was *handed*, and its answer is checked against the CPG. Three outcomes,
+and they are exhaustive:
+
+    the model entails it       report `model_entailed`, and never call the model at all
+    the model contradicts it   drop the claim, recording the contradiction
+    the model cannot decide    report `suspicion`, honestly
+
+Nothing here averages, votes, retries for agreement, or maintains a budget: a deterministic arbiter removes
+the need for every one of those. The LLM is consulted only for the residual the graph cannot settle — whether
+a sanitizer on a real flow is actually sufficient — and even that answer only ever *confirms* a flow the CPG
+already found. It can never conjure one.
+"""
+
+from __future__ import annotations
+
+import json
+import logging
+from collections.abc import Mapping
+from typing import Any
+
+from ..cpg.backend import CpgResult
+from ..redaction import redact_secrets
+from .ladder import Rung, Verdict
+from .specs import TaintSpec
+from .taint import verdict as taint_verdict
+
+logger = logging.getLogger(__name__)
+
+QUESTION = """You are judging ONE candidate a static model already found. Do not look for other issues.
+
+A dataflow analysis resolved this path in {language} code:
+
+  source: {source}
+  sink:   {sink}   (line {line}, function {function})
+  a sanitizer appears on the path: {sanitized}
+
+The graph has established that the value reaches the sink. The only open question is whether the sanitizer on
+that path actually makes it safe for the {family} weakness class.
+
+Answer ONLY with JSON: {{"vulnerable": true|false, "why": "<one sentence>"}}"""
+
+
+def judge(
+    cpg: CpgResult,
+    spec: TaintSpec,
+    *,
+    function: str = "",
+    client: Any | None = None,
+    model: str = "",
+    parameter_sources: bool = False,
+) -> Verdict:
+    """Arbitrate one candidate. Always returns a verdict; ``suspicion`` is a real answer, not a failure."""
+    answer = taint_verdict(cpg, spec, function=function, parameter_sources=parameter_sources)
+
+    if answer is not None and answer.rung is Rung.ENTAILED:
+        return answer  # the graph decided; the model is not consulted
+
+    if answer is None:
+        # No resolved flow. Whatever anyone claims about this site, the model saw nothing to support it.
+        return Verdict(
+            rung=Rung.SUSPICION,
+            family=spec.family,
+            contradiction="the model resolved no flow: no source reaches this sink",
+        )
+
+    # A real flow through a sanitizer. Whether that sanitizer suffices is a semantic judgement the graph
+    # cannot make, so it is the one thing the LLM is asked -- about evidence the model itself produced.
+    if client is None:
+        return Verdict(
+            rung=Rung.SUSPICION,
+            family=spec.family,
+            witness=answer.witness,
+            contradiction="no chat endpoint: the sanitizer's sufficiency was never judged",
+        )
+
+    said_vulnerable = _ask(client, model, spec, answer.witness, function)
+    if said_vulnerable is None:
+        return Verdict(rung=Rung.SUSPICION, family=spec.family, witness=answer.witness,
+                       contradiction="the judge returned no usable answer")
+    if not said_vulnerable:
+        return Verdict(rung=Rung.SUSPICION, family=spec.family, witness=answer.witness,
+                       contradiction="the judge held the sanitizer on this path sufficient")
+    return answer  # CORROBORATED: a real flow the judge agrees is not neutralised
+
+
+def _ask(client: Any, model: str, spec: TaintSpec, witness: str, function: str) -> bool | None:
+    """One question, one answer. No tools, no retries for agreement, no second opinion."""
+    source, _, sink = witness.partition(" -> ")
+    line = ""
+    if "(line " in sink:
+        sink, _, tail = sink.partition(" (line ")
+        line = tail.split(",")[0].strip(") ")
+    prompt = redact_secrets(
+        QUESTION.format(
+            language=spec.language, source=source.strip(), sink=sink.strip(), line=line or "?",
+            function=function or "?", sanitized="yes", family=spec.family,
+        )
+    )
+    try:
+        response = client.complete(
+            model=model,
+            messages=[{"role": "user", "content": prompt}],
+            tools=[],  # the candidate is given; the model never searches for it
+            json_object=True,
+        )
+    except Exception as exc:  # noqa: BLE001 -- an endpoint failure is a degradation, never a verdict
+        logger.warning("judge call failed: %s", exc)
+        return None
+    return _vulnerable(getattr(response, "content", None))
+
+
+def _vulnerable(content: object) -> bool | None:
+    if not isinstance(content, str) or not content.strip():
+        return None
+    try:
+        payload = json.loads(content)
+    except json.JSONDecodeError:
+        return None
+    if isinstance(payload, Mapping) and isinstance(payload.get("vulnerable"), bool):
+        return bool(payload["vulnerable"])
+    return None
+
+
+__all__ = ["QUESTION", "judge"]

src/openultrasast/model/ladder.py

@@ -18,6 +18,12 @@ from __future__ import annotations
 
 from dataclasses import dataclass
 from enum import StrEnum
+from typing import TYPE_CHECKING, TypeVar
+
+if TYPE_CHECKING:
+    from ..findings import StaticFinding
+
+_Finding = TypeVar("_Finding", bound="StaticFinding")
 
 
 class Rung(StrEnum):
@@ -45,4 +51,20 @@ class Verdict:
     contradiction: str = ""  # why an LLM claim was dropped, when it was
 
 
-__all__ = ["Rung", "Verdict", "at_or_above"]
+def at_rung(finding: _Finding, verdict: Verdict | None) -> _Finding:
+    """Return ``finding`` carrying ``verdict``'s rung and witness, or unchanged when there is no verdict.
+
+    This is the only way a finding rises above ``suspicion`` (Req 5.4). The witness travels with the rung
+    because a rung without the evidence that justifies it is just a louder assertion.
+    """
+    from dataclasses import replace
+
+    if verdict is None or verdict.rung is Rung.SUSPICION:
+        return finding
+    rationale = finding.rationale
+    if verdict.witness and verdict.witness not in rationale:
+        rationale = f"{rationale} [{verdict.rung.value}: {verdict.witness}]".strip()
+    return replace(finding, rung=verdict.rung.value, rationale=rationale)
+
+
+__all__ = ["Rung", "Verdict", "at_or_above", "at_rung"]

src/openultrasast/reports.py

@@ -46,6 +46,7 @@ def write_markdown_report(
                 f"- Severity: `{finding.severity}`",
                 f"- Confidence: `{finding.confidence}`",
                 f"- Evidence level: `{finding.evidence_level}`",
+                f"- Rung: `{finding.rung}`",
                 f"- Verification: `{verification.status if verification else 'not_run'}`",
                 f"- Reachability: `{finding.reachability_status}`",
                 f"- Conditions: `{', '.join(finding.reachability_conditions) or 'none'}`",
@@ -218,6 +219,7 @@ def write_manifest(
                 "line": finding.line,
                 "severity": finding.severity,
                 "evidence_level": finding.evidence_level,
+                "rung": finding.rung,
                 "verification_status": verification_by_id[finding.finding_id].status
                 if finding.finding_id in verification_by_id
                 else "not_run",
@@ -293,6 +295,7 @@ def _sarif_result(
         "severity": finding.severity,
         "confidence": finding.confidence,
         "evidence_level": finding.evidence_level,
+        "rung": finding.rung,
         "verification": asdict(verification) if verification else None,
         "reachability_status": finding.reachability_status,
         "reachability_conditions": finding.reachability_conditions,