feat(model-grounded-detection): group 5, corpus calibration and honest reporting

5.1 inverts what the corpus is for. Previously a pair the detector missed was a
DETECTOR failure, to be averaged away over K runs and chased with a prompt
change. The arbiter is deterministic: it will miss that pair the same way every
time, and the only thing that moves it is better modelling. So a pair the model
cannot split is a NAMED GAP IN THE MODEL, recorded with its reason, and every
gap carries one even when the caller supplies none -- a bare gap teaches
nothing.

`covered` means the model reached a strictly higher rung on the vulnerable side
than on its fix. Flagging both sides equally is not coverage: it distinguishes
nothing, which is exactly what group 2 measured taint reachability doing on
every parameterised SQL pair. The harvest seam takes the fix commit and derives
the vulnerable side as its parent, with the fix diff as the oracle -- nothing
hand-labelled.

5.2 keeps three disciplines from the deleted publish layer, each because its
absence caused a real error here:

  * per-slice rows always -- `render` RAISES without them. A mixed headline
    once hid agent-written code scoring 44% against the development slice's 83%.
  * the overfitting gap beside the headline, and ABSENT rather than zero when a
    slice is missing: reading an unmeasured slice as "no overfitting" is the
    flattering failure, not a neutral one.
  * the rung on every finding, so a suspicion cannot be read as a result.

Run end to end against the committed ceiling artifact, the report says things
worth not hiding: agent-vfc 0/28 covered, vibe-py 40% against vfc-js 24%, an
overfitting gap of +16%, and gaps grouped by family with their reasons --
injection 32 (21 no verdict, 11 distinguished nothing), path 8 (all no
verdict), untrusted_destination 5 (4 entailed on BOTH sides).

713 passing, mypy/ruff clean, gates byte-identical, zero orphaned modules.
A full re-measure with the closure-scoping fix is in flight; these numbers are
the pre-fix baseline and will move.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG

src/openultrasast/model/audit.py

@@ -36,6 +36,8 @@ SPEC_OWNED = {
     "model.judge": "model-grounded-detection Req 8: one bounded question, checked against the model",
     "model.dominance": "model-grounded-detection Req 6.2: guard dominance, the arbiter for bugs that never crash",
     "model.config_value": "model-grounded-detection Req 6.3: constant abstraction for the configuration families",
+    "model.calibrate": "model-grounded-detection Req 9: the corpus as the model's calibration set",
+    "model.report": "model-grounded-detection Req 9.3/11.1: per-slice reporting with the overfitting gap",
     "cpg": "model-grounded-detection Req 4: the CPG seam package",
     "cpg.backend": "model-grounded-detection Req 4.1/4.5: the single Joern subprocess boundary",
     "cpg.capability": "model-grounded-detection Req 4.2/4.3: the engine probe that degrades to suspicion",

src/openultrasast/model/calibrate.py

@@ -0,0 +1,175 @@
+"""The corpus calibrates the model, not an LLM vote (model-grounded-detection, Req 9).
+
+This is the inversion the whole feature rests on. In the prior architecture a pair the detector missed was a
+*detector* failure, to be averaged away over K runs and chased with a prompt change. Here the arbiter is
+deterministic: it will miss that pair the same way every time, and the only thing that moves it is better
+modelling. So a pair the model cannot split is a **named gap in the model**, recorded with its reason, and
+the report's job is to make those gaps legible by family and by slice rather than to produce a score.
+
+`covered` means the model reached a strictly higher rung on the vulnerable side than on its fix. Flagging
+both sides equally is not coverage — it distinguishes nothing, which is exactly the failure the group-2
+measurement found taint reachability making on every parameterised SQL pair.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Iterable, Mapping, Sequence
+from dataclasses import dataclass, field
+from typing import TypedDict
+
+
+class FamilyGaps(TypedDict):
+    """Gaps for one family: how many pairs, and the tally of reasons behind them."""
+
+    pairs: int
+    reasons: dict[str, int]
+
+# Ascending, matching `ladder.Rung`. A pair is split when the vulnerable side outranks its fix.
+_ORDER = {"none": 0, "": 0, "suspicion": 0, "model_corroborated": 1, "model_entailed": 2, "execution_confirmed": 3}
+
+
+@dataclass(frozen=True)
+class PairOutcome:
+    """What the model made of one vulnerable/fixed pair."""
+
+    pair: str
+    family: str
+    slice: str
+    vuln_rung: str
+    fixed_rung: str
+    reason: str = ""
+
+
+@dataclass(frozen=True)
+class ModelGap:
+    """A pair the model could not split, and why. Not a miss to average — a gap to close."""
+
+    pair: str
+    family: str
+    slice: str
+    reason: str
+
+
+@dataclass(frozen=True)
+class HarvestRecipe:
+    """Where a pair comes from. The fix commit's parent is the vulnerable side, by construction."""
+
+    repo: str
+    fix_commit: str
+    relpath: str
+    function: str = ""
+
+    @property
+    def parent_ref(self) -> str:
+        return f"{self.fix_commit}^"
+
+
+@dataclass(frozen=True)
+class CalibrationReport:
+    covered: tuple[str, ...]
+    gaps: tuple[ModelGap, ...]
+    per_slice: Mapping[str, Mapping[str, int]]
+    per_family: Mapping[str, Mapping[str, int]] = field(default_factory=dict)
+
+    def gaps_by_family(self) -> dict[str, FamilyGaps]:
+        """Gaps grouped by family, each with a tally of the reasons behind them."""
+        out: dict[str, FamilyGaps] = {}
+        for gap in self.gaps:
+            block = out.setdefault(gap.family, {"pairs": 0, "reasons": {}})
+            block["pairs"] += 1
+            block["reasons"][gap.reason] = block["reasons"].get(gap.reason, 0) + 1
+        return out
+
+    def to_markdown(self) -> str:
+        """Per-slice rows first; there is no headline here that is not backed by them (Req 9.3, 11.1)."""
+        lines = ["## Model calibration", "", "Coverage is a pair the model *splits*: a strictly higher rung on",
+                 "the vulnerable side than on its fix. Everything else is a named gap.", "",
+                 "| slice | pairs | covered | gaps | coverage |", "|---|---|---|---|---|"]
+        for slice_name, block in sorted(self.per_slice.items()):
+            pairs = block["pairs"] or 1
+            lines.append(
+                f"| {slice_name} | {block['pairs']} | {block['covered']} | {block['gaps']} | "
+                f"{block['covered'] / pairs:.0%} |"
+            )
+        by_family = self.gaps_by_family()
+        if by_family:
+            lines += ["", "### Gaps by family", ""]
+            for family, gaps in sorted(by_family.items()):
+                detail = ", ".join(f"{k} x{v}" for k, v in sorted(gaps["reasons"].items()))
+                lines.append(f"- `{family}`: {gaps['pairs']} pair(s) — {detail}")
+        return "\n".join(lines) + "\n"
+
+
+def calibrate(outcomes: Iterable[PairOutcome]) -> CalibrationReport:
+    """Split the corpus into what the model covers and what it cannot, with reasons."""
+    covered: list[str] = []
+    gaps: list[ModelGap] = []
+    per_slice: dict[str, dict[str, int]] = {}
+    per_family: dict[str, dict[str, int]] = {}
+
+    for outcome in outcomes:
+        split = _ORDER.get(outcome.vuln_rung, 0) > _ORDER.get(outcome.fixed_rung, 0)
+        for key, store in ((outcome.slice, per_slice), (outcome.family, per_family)):
+            block = store.setdefault(key, {"pairs": 0, "covered": 0, "gaps": 0})
+            block["pairs"] += 1
+            block["covered" if split else "gaps"] += 1
+        if split:
+            covered.append(outcome.pair)
+        else:
+            gaps.append(ModelGap(pair=outcome.pair, family=outcome.family, slice=outcome.slice,
+                                 reason=outcome.reason or _default_reason(outcome)))
+    return CalibrationReport(
+        covered=tuple(sorted(covered)),
+        gaps=tuple(sorted(gaps, key=lambda g: (g.family, g.pair))),
+        per_slice=per_slice,
+        per_family=per_family,
+    )
+
+
+def _default_reason(outcome: PairOutcome) -> str:
+    """Every gap carries a reason, even when the caller supplied none — a bare gap teaches nothing."""
+    if _ORDER.get(outcome.vuln_rung, 0) == 0:
+        return "no_verdict_on_the_vulnerable_side"
+    if outcome.vuln_rung == outcome.fixed_rung:
+        return f"both_sides_reached_{outcome.vuln_rung}: the model distinguished nothing"
+    return "fixed_side_outranked_the_vulnerable_one"
+
+
+def harvest_recipe(entry: Mapping[str, object]) -> HarvestRecipe:
+    """A pair from CVE history: the fix commit and its parent, with the fix diff as the oracle (Req 9.4).
+
+    Nothing here is hand-labelled. The recipe names where to fetch both sides; what makes the pair *true* is
+    the fix diff itself, read by the model rather than annotated by us.
+    """
+    return HarvestRecipe(
+        repo=str(entry.get("repo", "")),
+        fix_commit=str(entry.get("fix_commit", "")),
+        relpath=str(entry.get("relpath", "")),
+        function=str(entry.get("function", "") or ""),
+    )
+
+
+def outcomes_from_rows(rows: Sequence[Mapping[str, object]]) -> list[PairOutcome]:
+    """Adapt the ceiling artifact's rows, so calibration reads the measurement rather than re-running it."""
+    return [
+        PairOutcome(
+            pair=str(row.get("pair", "")),
+            family=str(row.get("family", "")),
+            slice=str(row.get("slice", "")),
+            vuln_rung=str(row.get("vuln_rung", "none")),
+            fixed_rung=str(row.get("fixed_rung", "none")),
+            reason=str(row.get("miss_cause", "") or ""),
+        )
+        for row in rows
+    ]
+
+
+__all__ = [
+    "CalibrationReport",
+    "HarvestRecipe",
+    "ModelGap",
+    "PairOutcome",
+    "calibrate",
+    "harvest_recipe",
+    "outcomes_from_rows",
+]

src/openultrasast/model/report.py

@@ -0,0 +1,83 @@
+"""Reporting that cannot flatter itself (model-grounded-detection, Req 9.3, 11.1).
+
+Three disciplines survive from the deleted publish/scoring layer, and each is here because its absence caused
+a real error in this project's history rather than because it is tidy:
+
+* **Per-slice rows, always.** A headline over mixed slices once hid that agent-written code scored 44.4%
+  while the hand-written development slice scored 83.3%. `render` refuses to produce anything without rows.
+* **The overfitting gap beside the headline.** The distance between the development slice and the deciding
+  real-world one is stated, never left for the reader to compute. When a slice is missing the gap is
+  *absent*, not zero — reading a missing measurement as "no overfitting" is the flattering failure.
+* **The rung on every finding.** A finding's strength is what established it; omitting the rung invites a
+  reader to treat a suspicion as a result.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+from dataclasses import dataclass
+
+from ..findings import StaticFinding
+
+
+@dataclass(frozen=True)
+class SliceRow:
+    """One slice's coverage. `covered` is pairs the model *split*, not pairs it flagged."""
+
+    slice: str
+    pairs: int
+    covered: int
+
+    @property
+    def rate(self) -> float:
+        return self.covered / self.pairs if self.pairs else 0.0
+
+
+def overfitting_gap(rows: Sequence[SliceRow], *, development: str = "", deciding: str = "") -> float | None:
+    """Development-slice coverage minus the deciding slice's, or ``None`` when either is absent.
+
+    ``None`` rather than ``0.0`` on purpose: a slice that was never measured must not read as agreement
+    between them.
+    """
+    by_name = {row.slice: row for row in rows}
+    if development not in by_name or deciding not in by_name:
+        return None
+    return by_name[development].rate - by_name[deciding].rate
+
+
+def render(rows: Sequence[SliceRow], *, development: str = "", deciding: str = "") -> str:
+    """The report. Raises when there are no per-slice rows, because a bare aggregate is the thing to prevent."""
+    if not rows:
+        raise ValueError("a report needs per-slice rows: there is no headline without them")
+
+    lines = ["## Model coverage", "", "| slice | pairs | covered | coverage |", "|---|---|---|---|"]
+    for row in sorted(rows, key=lambda r: r.slice):
+        lines.append(f"| {row.slice} | {row.pairs} | {row.covered} | {row.rate:.0%} |")
+
+    total_pairs = sum(row.pairs for row in rows)
+    total_covered = sum(row.covered for row in rows)
+    lines += ["", f"Across every slice: {total_covered}/{total_pairs} = {total_covered / total_pairs:.0%} "
+                  "— read the rows above it, not this line."]
+
+    gap = overfitting_gap(rows, development=development, deciding=deciding)
+    if gap is not None:
+        lines += [
+            "",
+            f"**Overfitting gap: {gap:+.0%}** — `{development}` (development) minus `{deciding}`, "
+            f"which is the deciding real-world slice.",
+        ]
+    elif development or deciding:
+        lines += ["", f"**Overfitting gap: not measurable** — `{development}` or `{deciding}` is absent from this run."]
+    return "\n".join(lines) + "\n"
+
+
+def render_findings(findings: Sequence[StaticFinding]) -> str:
+    """Findings with the rung that established each one (Req 5.3)."""
+    lines = ["## Findings", "", "| finding | location | rung | severity |", "|---|---|---|---|"]
+    for finding in findings:
+        where = f"{finding.path}:{finding.line}" if finding.line else finding.path
+        lines.append(f"| {finding.title} | {where} | `{finding.rung}` | {finding.severity} |")
+    return "\n".join(lines) + "\n"
+
+
+__all__ = ["SliceRow", "overfitting_gap", "render", "render_findings"]