feat(plane): honest coverage in alerts; PHP alerts from the engine on the host

Quick mode has rules only for c/cpp, groovy, java, javascript/typescript and
python, so every PHP case's "0 alerts" read as clean. The `alerts` task now
records per language of the files read whether quick mode covers it
(derived from the bundled ruleset, not a list): coverage quick | none, with
file counts per pin, and `uncovered`. `remember` turns that into `coverage`
memory rows (a new kind; the Run passes the alerts summary.json as input),
loop-measure reports coverage per language, and improve/memory.py's M1/M2
skip a repository+pin whose coverage for the rule's language is only
`none` (counted under `coverage_none` in the proposal's excluded).

`ousast plane alerts-engine <Run.yaml> --source <frozen export>`
(plane/engine_alerts.py) does the alerts task on the host for cases whose
files include a language the engine covers but quick mode does not (PHP):
exports both pins from the case cache, runs the quick scan and the
openultrasast:dev engine container (finding_dump.py, --network none,
--memory 3g, one at a time, per-pin deadline), writes engine findings in
the alerts.jsonl row shape (rule_id engine:<family>, source engine,
in_fix_range from case.json) with a summary (files read, questions,
completed, seconds, degradations per pin), and marks the task done via
reconciler.mark_done. No result, no files read, or questions == 0 fails
loudly and leaves the run untouched. Refuses a git checkout as source.

Runner image rebuilt and re-pinned (sha256:ebb40a9a...); validation-46
regenerated (digest only). Live, recorded in
benchmarks/measurements/2026-09-30-php-engine-alerts/: froxlor 366 files,
130/130 questions per pin, 6 alerts per pin, 0 in the fix range; wp-gopay
19 files, 101/101 questions, 0 alerts. The loop: 32 run rows (coverage 8,
php=engine), 0 proposals, gate no_proposals, $0.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-09-30 13:44:32 +02:00
co-authored by Claude Opus 5.5
parent f4bd984fda
commit 45d02ecf20
35 changed files with 1949 additions and 142 deletions
+32
View File
@@ -274,6 +274,18 @@ def _main(argv: list[str] | None) -> int:
plane_remember.add_argument("--plane", type=Path, default=Path("plane"), help="where runs/<run>.yaml names population and split")
plane_remember.add_argument("--population", help="the population the run measured (default: the Run's annotation)")
plane_remember.add_argument("--split", help="the split of that population (default: the Run's annotation)")
plane_engine = plane_sub.add_parser(
"alerts-engine", help="a Run's `alerts` for languages quick mode does not cover (PHP), from the engine image on this host"
)
plane_engine.add_argument("run_manifest", type=Path)
plane_engine.add_argument(
"--source", type=Path, required=True, help="a frozen export: git archive HEAD src benchmarks/push/finding_dump.py"
)
plane_engine.add_argument("--cases", default="", help="comma-separated case ids (default: every alerts case of the Run)")
plane_engine.add_argument("--deadline", type=float, default=1800.0, help="the engine's scan deadline per pin, seconds")
plane_engine.add_argument("--image", default="openultrasast:dev")
plane_engine.add_argument("--repos", type=Path, help="the case cache (default ~/.cache/openultrasast/independent)")
plane_engine.add_argument("--work", type=Path, help="where pins are exported and results land (default: a temporary directory)")
plane_ws = plane_sub.add_parser("workspaces", help="one Workspace manifest per case pin of a population file")
plane_ws.add_argument("population", type=Path)
plane_ws.add_argument("--out", type=Path, default=Path("plane/workspaces"))
@@ -1318,6 +1330,8 @@ def _plane(args: argparse.Namespace) -> int:
if args.plane_command == "status":
print(reconciler.status(args.run, units=args.units))
return 0
if args.plane_command == "alerts-engine":
return _plane_alerts_engine(args)
if args.plane_command == "doctor":
checks = reconciler.doctor()
for name, ok, text in checks:
@@ -1332,6 +1346,24 @@ def _plane(args: argparse.Namespace) -> int:
return 0
def _plane_alerts_engine(args: argparse.Namespace) -> int:
from .plane import engine_alerts
only = [c.strip() for c in args.cases.split(",") if c.strip()]
try:
results = engine_alerts.alerts_engine(
args.run_manifest, source=args.source, only=only, work=args.work, cache=args.repos or engine_alerts.CACHE,
deadline=args.deadline, image=args.image,
) # fmt: skip
except (engine_alerts.EngineAlertsError, OSError, ValueError, subprocess.CalledProcessError) as exc:
print(f"alerts-engine failed: {exc}", file=sys.stderr)
return 2
keys = ("case", "status", "reason", "alerts", "alerts_engine", "in_fix_range", "engine", "uncovered", "seconds")
for result in results:
print(json.dumps({k: result[k] for k in keys if k in result}, sort_keys=True))
return 0
def _plane_memory(
step: str, run_manifest: Path | None, *, run: str | None = None, population: str | None = None, split: str | None = None
) -> int: