feat(plane): honest coverage in alerts; PHP alerts from the engine on the host

Quick mode has rules only for c/cpp, groovy, java, javascript/typescript and
python, so every PHP case's "0 alerts" read as clean. The `alerts` task now
records per language of the files read whether quick mode covers it
(derived from the bundled ruleset, not a list): coverage quick | none, with
file counts per pin, and `uncovered`. `remember` turns that into `coverage`
memory rows (a new kind; the Run passes the alerts summary.json as input),
loop-measure reports coverage per language, and improve/memory.py's M1/M2
skip a repository+pin whose coverage for the rule's language is only
`none` (counted under `coverage_none` in the proposal's excluded).

`ousast plane alerts-engine <Run.yaml> --source <frozen export>`
(plane/engine_alerts.py) does the alerts task on the host for cases whose
files include a language the engine covers but quick mode does not (PHP):
exports both pins from the case cache, runs the quick scan and the
openultrasast:dev engine container (finding_dump.py, --network none,
--memory 3g, one at a time, per-pin deadline), writes engine findings in
the alerts.jsonl row shape (rule_id engine:<family>, source engine,
in_fix_range from case.json) with a summary (files read, questions,
completed, seconds, degradations per pin), and marks the task done via
reconciler.mark_done. No result, no files read, or questions == 0 fails
loudly and leaves the run untouched. Refuses a git checkout as source.

Runner image rebuilt and re-pinned (sha256:ebb40a9a...); validation-46
regenerated (digest only). Live, recorded in
benchmarks/measurements/2026-09-30-php-engine-alerts/: froxlor 366 files,
130/130 questions per pin, 6 alerts per pin, 0 in the fix range; wp-gopay
19 files, 101/101 questions, 0 alerts. The loop: 32 run rows (coverage 8,
php=engine), 0 proposals, gate no_proposals, $0.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo

src/openultrasast/plane/tasks/alerts.py

@@ -15,6 +15,15 @@ of ``case.json`` on the vulnerable pin, the new-side ``fixed_ranges`` on the fix
 none). A fixed-pin alert inside the fix is a false alert on code the fix wrote: rule M1 of ``improve/memory.py``
 counts it.
 
+**Coverage** (an honest zero): quick mode has rules only for the languages its ruleset declares
+(:func:`quick_languages`, derived from the bundled ruleset directory, never listed by hand). ``summary.json`` records
+per language of the files the scan read whether quick mode covers it -- ``coverage: {language: {"coverage": "quick" |
+"none", "files": {role: n}}}`` -- and ``uncovered``, the languages with ``none``. Zero alerts on an uncovered
+language is not a clean result: nothing could have fired. ``remember`` turns the coverage into ``coverage`` rows,
+and M1/M2 skip a repository and pin whose coverage for a rule's language is ``none``. The deterministic engine covers
+more languages (:func:`engine_languages`, PHP among them); ``ousast plane alerts-engine`` (:mod:`..engine_alerts`)
+produces those alerts on the host, where the engine image runs, and marks this task done.
+
 Env: ``OUSAST_OUTPUT_DIR``, ``OUSAST_WORKSPACE_DIR`` (vulnerable checkout), ``OUSAST_FIXED_DIR`` (fixed checkout),
 ``OUSAST_INPUT_CASE`` (``case.json``), ``OUSAST_VULNERABLE_PIN``, ``OUSAST_FIXED_PIN``. A pin whose checkout yields no
 source file fails the task: an unread tree and a silent one are indistinguishable afterwards. Exit 0 done, 2 failed.
@@ -26,7 +35,7 @@ import json
 import os
 import sys
 import traceback
-from collections.abc import Mapping, Sequence
+from collections.abc import Iterable, Mapping, Sequence
 from pathlib import Path
 from typing import Any
 
@@ -34,14 +43,55 @@ from ...findings import quick_scan_findings
 from ...mapping import analyze_entry_points, attach_reachability_hints
 from ...preprocess import detect_language, preprocess_repository
 from ...rank import rank_targets
+from ...ruleset import DEFAULT_RULESET_DIR, load_ruleset
 from .repo_facts import DECLARATION, GLOBAL, enclosing
 
 ROLES = ("vulnerable", "fixed")
 DIRS = ("OUSAST_WORKSPACE_DIR", "OUSAST_FIXED_DIR")
 PINS = ("OUSAST_VULNERABLE_PIN", "OUSAST_FIXED_PIN")
+ANY_LANGUAGE = "*"  # a rule that names no language runs on every file
 Ranges = Mapping[str, Sequence[Sequence[int]]]
 
 
+def quick_languages(directory: Path = DEFAULT_RULESET_DIR) -> frozenset[str]:
+    """The languages quick mode has a rule for: the ``languages`` of every rule the ruleset directory holds that is
+    not disabled (``*`` when a rule names none: it runs on every file)."""
+    covered: set[str] = set()
+    for rule in load_ruleset(directory):
+        if rule.status != "disabled":
+            covered.update(rule.languages or (ANY_LANGUAGE,))
+    return frozenset(covered)
+
+
+def engine_languages(directory: Path = DEFAULT_RULESET_DIR) -> frozenset[str]:
+    """The languages the deterministic engine has a semantic model for: one ``semantic/<language>.toml`` each."""
+    return frozenset(path.stem for path in (directory / "semantic").glob("*.toml"))
+
+
+def covers(covered: Iterable[str], language: str) -> bool:
+    languages = set(covered)
+    return ANY_LANGUAGE in languages or language in languages
+
+
+def coverage(files: Mapping[str, Mapping[str, int]], quick: Iterable[str], engine: Iterable[str] = ()) -> dict[str, dict[str, Any]]:
+    """Per language of the files read (``{role: {language: n}}``): ``quick`` when quick mode has a rule for it,
+    ``engine`` when the engine's alerts were produced for it, else ``none``, with the file count per role."""
+    quick, engine = set(quick), set(engine)
+    result: dict[str, dict[str, Any]] = {}
+    for role in sorted(files):
+        for language, number in files[role].items():
+            if language == "unknown":
+                continue
+            kind = "quick" if covers(quick, language) else "engine" if language in engine else "none"
+            entry = result.setdefault(language, {"coverage": kind, "files": {}})
+            entry["files"][role] = int(number)
+    return dict(sorted(result.items()))
+
+
+def uncovered(cov: Mapping[str, Mapping[str, Any]]) -> list[str]:
+    return sorted(language for language, entry in cov.items() if entry.get("coverage") == "none")
+
+
 def _lines(path: Path) -> list[str]:
     try:
         return path.read_text(encoding="utf-8", errors="replace").splitlines()
@@ -55,24 +105,35 @@ def in_range(ranges: Ranges | None, path: str, line: int | None) -> bool | None:
     return line is not None and any(int(lo) <= line <= int(hi) for lo, hi in ranges.get(path, []))
 
 
-def scan(root: Path, role: str, pin: str, ranges: Ranges | None) -> tuple[list[dict[str, Any]], dict[str, int]]:
-    """(rows, what was read) of one pin: the quick-mode findings as alert rows, one per rule, file, line."""
+def function_at(root: Path, path: str, line: int | None, fallback: str | None, cache: dict[str, list[str]]) -> str:
+    """The enclosing declaration of ``path:line`` by the ``repo-facts`` patterns, else ``fallback`` (``<global>``)."""
+    lines = cache.setdefault(path, _lines(root / path))
+    language = detect_language(root / path)
+    if language in DECLARATION and line and 0 < line <= len(lines):
+        return enclosing(lines, line - 1, language)
+    return fallback or GLOBAL
+
+
+def scan(root: Path, role: str, pin: str, ranges: Ranges | None) -> tuple[list[dict[str, Any]], dict[str, Any]]:
+    """(rows, what was read) of one pin: the quick-mode findings as alert rows, one per rule, file, line; what was read
+    counts files and bytes, and files per language."""
     _, targets = preprocess_repository(root)
     targets = attach_reachability_hints(targets, analyze_entry_points(root, targets))
     findings = quick_scan_findings(root, targets, rank_targets(targets))
-    read = {"files": len(targets), "bytes": sum((root / t.path).stat().st_size for t in targets if (root / t.path).is_file())}
+    languages: dict[str, int] = {}
+    for target in targets:
+        languages[target.language] = languages.get(target.language, 0) + 1
+    read = {
+        "files": len(targets), "bytes": sum((root / t.path).stat().st_size for t in targets if (root / t.path).is_file()),
+        "languages": dict(sorted(languages.items())),
+    }  # fmt: skip
     cache: dict[str, list[str]] = {}
     rows: dict[tuple[str, str, int | None], dict[str, Any]] = {}
     for finding in findings:
         rule_id = finding.finding_id.split(":", 1)[0]
-        lines = cache.setdefault(finding.path, _lines(root / finding.path))
-        language = detect_language(root / finding.path)
-        if language in DECLARATION and finding.line and 0 < finding.line <= len(lines):
-            function = enclosing(lines, finding.line - 1, language)
-        else:
-            function = finding.function_name or GLOBAL
         row = {
-            "rule_id": rule_id, "rule_status": finding.status, "path": finding.path, "line": finding.line, "function": function,
+            "rule_id": rule_id, "rule_status": finding.status, "path": finding.path, "line": finding.line,
+            "function": function_at(root, finding.path, finding.line, finding.function_name, cache),
             "pin_role": role, "pin": pin, "in_fix_range": in_range(ranges, finding.path, finding.line),
         }  # fmt: skip
         rows.setdefault((rule_id, finding.path, finding.line), row)
@@ -80,10 +141,14 @@ def scan(root: Path, role: str, pin: str, ranges: Ranges | None) -> tuple[list[d
     return ordered, read
 
 
-def _summary(output_dir: Path, summary: Mapping[str, Any]) -> None:
+def write_summary(output_dir: Path, summary: Mapping[str, Any]) -> None:
     (output_dir / "summary.json").write_text(json.dumps(summary, indent=2, sort_keys=True) + "\n", encoding="utf-8")
 
 
+def write_rows(output_dir: Path, rows: Iterable[Mapping[str, Any]]) -> None:
+    (output_dir / "alerts.jsonl").write_text("".join(json.dumps(r, sort_keys=True) + "\n" for r in rows), encoding="utf-8")
+
+
 def main(environ: Mapping[str, str] | None = None) -> int:
     env = environ if environ is not None else os.environ
     output = Path(env["OUSAST_OUTPUT_DIR"]) if env.get("OUSAST_OUTPUT_DIR") else None
@@ -95,6 +160,7 @@ def main(environ: Mapping[str, str] | None = None) -> int:
         case = json.loads(Path(env["OUSAST_INPUT_CASE"]).read_text(encoding="utf-8")) if env.get("OUSAST_INPUT_CASE") else {}
         ranges = {"vulnerable": case.get("ranges"), "fixed": case.get("fixed_ranges")}
         rows: list[dict[str, Any]] = []
+        files: dict[str, dict[str, int]] = {}
         for role, directory, pin in zip(ROLES, DIRS, PINS, strict=True):
             root = Path(env.get(directory) or "")
             if not env.get(directory) or not root.is_dir():
@@ -102,21 +168,27 @@ def main(environ: Mapping[str, str] | None = None) -> int:
             found, read = scan(root, role, str(env.get(pin) or ""), ranges[role])
             if read["files"] == 0:
                 raise ValueError(f"the {role} checkout {root} has no source file the scan reads: an unread tree is not a clean one")
+            files[role] = read.pop("languages")
             summary.setdefault("read", {})[role] = read
             summary.setdefault("alerts", {})[role] = len(found)
+            summary.setdefault("pins", {})[role] = str(env.get(pin) or "")
             rows.extend(found)
             summary["units_done"] += 1
-        (output / "alerts.jsonl").write_text("".join(json.dumps(r, sort_keys=True) + "\n" for r in rows), encoding="utf-8")
+        quick = quick_languages()
+        summary["quick_languages"] = sorted(quick)
+        summary["coverage"] = coverage(files, quick)
+        summary["uncovered"] = uncovered(summary["coverage"])
+        write_rows(output, rows)
         summary["status"] = "done"
     except Exception:  # noqa: BLE001 -- a crash is `failed` with its traceback
         summary["reason"] = traceback.format_exc()[-2000:]
     if output is not None:
-        _summary(output, summary)
-    print(json.dumps({k: summary.get(k) for k in ("status", "alerts", "read")}), file=sys.stderr)
+        write_summary(output, summary)
+    print(json.dumps({k: summary.get(k) for k in ("status", "alerts", "read", "uncovered")}), file=sys.stderr)
     return 0 if summary["status"] == "done" else 2
 
 
-__all__ = ["in_range", "main", "scan"]
+__all__ = ["coverage", "covers", "engine_languages", "function_at", "in_range", "main", "quick_languages", "scan", "uncovered"]
 
 
 if __name__ == "__main__":

src/openultrasast/plane/tasks/loop.py

@@ -119,6 +119,10 @@ def metrics(rows: Sequence[Mapping[str, Any]]) -> dict[str, Any]:
         tally_a = alerts.setdefault(str(a.get("rule_id")), {"vulnerable": 0, "fixed": 0, "fixed_in_fix_range": 0})
         tally_a[role] += 1
         tally_a["fixed_in_fix_range"] += role == "fixed" and a.get("in_fix_range") is True
+    coverage: dict[str, dict[str, int]] = {}  # language -> pins per coverage kind: a `none` pin's zero alerts mean nothing
+    for c in (r for r in rows if r["kind"] == "coverage"):
+        tally_c = coverage.setdefault(str(c.get("language")), {"quick": 0, "engine": 0, "none": 0})
+        tally_c[str(c.get("coverage"))] = tally_c.get(str(c.get("coverage")), 0) + 1
     return {
         "cases": len({(v["repo"], v["pin"]) for v in verdicts}),
         "candidates": len(verdicts),
@@ -130,6 +134,7 @@ def metrics(rows: Sequence[Mapping[str, Any]]) -> dict[str, Any]:
         "usd_per_candidate": None if usd is None or not verdicts else round(usd / len(verdicts), 6),
         "dispute_rate": {f: {"candidates": n, "disputed": d, "rate": round(d / n, 4)} for f, (n, d) in sorted(families.items())},
         "alerts": dict(sorted(alerts.items())),
+        "coverage": dict(sorted(coverage.items())),
     }
 
 

src/openultrasast/plane/tasks/remember.py

@@ -2,8 +2,9 @@
 
 No model, budget ``{usd: 0, calls: 0}``, one per case. It reads the case's `facts.json`, the `units.jsonl` of the
 verify passes a, b and c, the final `agreed.json` (its per-candidate rows and `disputed` list) and, when present,
-`alerts.jsonl`, and writes `memory.jsonl` -- rows of the kinds `facts`, `verdict`, `unit_cost` and `alert` (the
-fifth kind, `proposal_outcome`, comes from the loop) -- and passes `facts.json` through. Task code interprets the
+`alerts.jsonl` with the alerts' `summary.json`, and writes `memory.jsonl` -- rows of the kinds `facts`, `verdict`,
+`unit_cost`, `alert` and `coverage` (per language and pin: whether any rule could have fired there; the kind
+`proposal_outcome` comes from the loop) -- and passes `facts.json` through. Task code interprets the
 artifacts, so the reconciler and the runner stay content-blind (ai-service-plane Req 3.4).
 
 Every row carries `id` (deterministic: :func:`..memory.row_id`), `kind`, `repo`, `pin`, `run`, `task`,
@@ -12,7 +13,8 @@ Every row carries `id` (deterministic: :func:`..memory.row_id`), `kind`, `repo`,
 Env contract in a Run: `OUSAST_OUTPUT_DIR`, `OUSAST_RUN`, `OUSAST_TASK`, `OUSAST_INPUT_FACTS` (and `OUSAST_INPUT_FACTS_SUMMARY`,
 whose reused or recorded `image` the facts row carries; without one named, the row is omitted),
 `OUSAST_INPUT_PASS_A`/`_B`, optional `OUSAST_INPUT_PASS_C`, `OUSAST_INPUT_SUMMARY_A`/`_B`/`_C` (the passes'
-`summary.json`, for the model), `OUSAST_INPUT_AGREED`, `OUSAST_INPUT_ALERTS`; the repository and pin from the bound
+`summary.json`, for the model), `OUSAST_INPUT_AGREED`, `OUSAST_INPUT_ALERTS` (and `OUSAST_INPUT_ALERTS_SUMMARY`,
+the coverage); the repository and pin from the bound
 Workspace (`AX_WORKSPACES_YAML`, `OUSAST_GIT_PINS`), the image from `AX_TASK_YAML`, the population and split from
 `OUSAST_POPULATION`/`OUSAST_SPLIT` (the generator copies the Run's `openultrasast.io/population` and
 `openultrasast.io/split` annotations). Exit 0 done, 2 failed.
@@ -74,6 +76,26 @@ def _final(row: Mapping[str, Any], disputed: set[str]) -> str:
     return "disputed" if str(row["candidate"]) in disputed else "rejected"
 
 
+def coverage_rows(ctx: Context, alerts_summary: Mapping[str, Any] | None) -> list[dict[str, Any]]:
+    """One ``coverage`` row per language and pin of the alerts' ``summary.json``: ``quick``, ``engine`` or ``none``
+    (no rule could fire, so no alert there is evidence of anything), with the files read in that language. A summary
+    from before coverage was recorded yields none: unknown stays unknown."""
+    if not alerts_summary or not isinstance(alerts_summary.get("coverage"), Mapping):
+        return []
+    recorded = alerts_summary.get("pins")
+    pins: Mapping[str, Any] = recorded if isinstance(recorded, Mapping) else {}
+    rows: list[dict[str, Any]] = []
+    for language, entry in sorted(alerts_summary["coverage"].items()):
+        for role, files in sorted((entry.get("files") or {}).items()):
+            fields: dict[str, Any] = {"language": language, "coverage": entry.get("coverage"), "files": files, "pin_role": role}
+            if pins.get(role):
+                fields["pin"] = str(pins[role])
+            if alerts_summary.get("source"):
+                fields["source"] = str(alerts_summary["source"])
+            rows.append(ctx.row("coverage", f"{language}:{role}", **fields))
+    return rows
+
+
 def rows_for(
     ctx: Context,
     *,
@@ -82,8 +104,9 @@ def rows_for(
     models: Mapping[str, str | None],
     agreed: Mapping[str, Any] | None,
     alerts: Sequence[Mapping[str, Any]] = (),
+    alerts_summary: Mapping[str, Any] | None = None,
 ) -> list[dict[str, Any]]:
-    """The memory rows of one case: facts, verdicts, unit costs, alerts."""
+    """The memory rows of one case: facts, verdicts, unit costs, alerts, and the alerts' coverage per language and pin."""
     rows: list[dict[str, Any]] = []
     if facts is not None:
         loaded = json.loads(facts)
@@ -115,8 +138,11 @@ def rows_for(
             fields["in_fix_range"] = alert["in_fix_range"]
         if alert.get("pin"):  # a fixed-pin alert is a row of the pin it was raised on, not of the case's vulnerable pin
             fields["pin"] = str(alert["pin"])
+        if alert.get("source"):  # `engine` when `ousast plane alerts-engine` produced it on the host
+            fields["source"] = str(alert["source"])
         subject = f"{fields['rule_id']}:{fields['path']}:{fields['line']}" + (":fixed" if fields.get("pin_role") == "fixed" else "")
         rows.append(ctx.row("alert", subject, **fields))
+    rows.extend(coverage_rows(ctx, alerts_summary))
     return [validate_row(r, f"{ctx.task} {r['kind']}") for r in rows]
 
 
@@ -185,7 +211,9 @@ def case_rows(base: Path, case: str, *, population: str, split: str) -> tuple[li
     passes = {p: _jsonl(base / f"{case}-v{p}" / "units.jsonl") for p in PASSES}
     models = {p: _model(_json(base / f"{case}-v{p}" / "summary.json")) for p in PASSES}
     alerts = _jsonl(base / f"{case}-alerts" / "alerts.jsonl")
-    return rows_for(ctx, facts=facts, passes=passes, models=models, agreed=agreed, alerts=alerts), facts
+    alerts_summary = _json(base / f"{case}-alerts" / "summary.json")
+    rows = rows_for(ctx, facts=facts, passes=passes, models=models, agreed=agreed, alerts=alerts, alerts_summary=alerts_summary)
+    return rows, facts
 
 
 def remember_run(run_dir: Path, store: MemoryStore, *, population: str = UNKNOWN, split: str = UNKNOWN) -> list[IngestResult]:
@@ -246,7 +274,8 @@ def main(environ: Mapping[str, str] | None = None) -> int:
         passes = {p: _units(_path(env, f"OUSAST_INPUT_PASS_{p.upper()}")) for p in PASSES}
         models = {p: _model(_json(_path(env, f"OUSAST_INPUT_SUMMARY_{p.upper()}"))) for p in PASSES}
         agreed = _json(_path(env, "OUSAST_INPUT_AGREED"))
-        rows = rows_for(ctx, facts=None, passes=passes, models=models, agreed=agreed, alerts=_jsonl(_path(env, "OUSAST_INPUT_ALERTS")))
+        alerts, alerts_summary = _jsonl(_path(env, "OUSAST_INPUT_ALERTS")), _json(_path(env, "OUSAST_INPUT_ALERTS_SUMMARY"))
+        rows = rows_for(ctx, facts=None, passes=passes, models=models, agreed=agreed, alerts=alerts, alerts_summary=alerts_summary)
         facts_image = _facts_image(env, ctx.image)
         if facts is not None and facts_image is not None:  # the facts row carries the image the facts were computed on
             facts_ctx = Context(ctx.run, ctx.task, ctx.repo, ctx.pin, facts_image, ctx.population, ctx.split)
@@ -265,7 +294,7 @@ def main(environ: Mapping[str, str] | None = None) -> int:
     return 0 if summary["status"] == "done" else 2
 
 
-__all__ = ["Context", "candidates_digest", "case_rows", "context_from", "main", "remember_run", "rows_for"]
+__all__ = ["Context", "candidates_digest", "case_rows", "context_from", "coverage_rows", "main", "remember_run", "rows_for"]
 
 
 if __name__ == "__main__":