feat(ruleset): PHP quick-mode rules from the engine's vocabulary, measured on the development corpus

Quick mode had no PHP rules, so a PHP quick scan read the files and reported a plausible zero.
ruleset/php/rules.toml adds 13 line/statement rules built on semantic/php.toml's sinks, sources and
sanitizers (sql/wpdb, command, code_eval, unserialize, echo, file_inclusion, filesystem,
outbound_request, redirect). 9 are enabled and 4 are shadow (sql-call-composition, code-eval,
unserialize, xss-echo-input). A rule is shadow when it fired on the corpus but never at a known site.
precision_estimate is the measured lower bound.

Measured in-sample on the development corpus only (pmpro introduce/repair/benign, population-v1 PHP
cases; population-v2 was not read). Known site hit on the vulnerable pin in 4 of 5 cases (LearnPress
missed). The hit disappears on the fixed pin for pmpro and YesWiki. Numbers, per-pin files/bytes read
and the engine comparison are in benchmarks/measurements/2026-09-30-php-quick-rules/.

- fixtures php-vulnerable / php-benign plus manifests: ousast benchmark covers PHP
  (9/14 reported, 13/14 including shadow; the benign twin records 2 known false alerts)
- PHP `#` comments are comments; `#[...]` attributes are not
- the alerts task now reports PHP as quick coverage; alerts-engine keeps running the engine for PHP
  next to the quick rules (ENGINE_ALONGSIDE_QUICK)
- other languages' benchmark findings and metrics are unchanged

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-09-30 15:05:47 +02:00
co-authored by Claude Opus 5.5
parent 45d02ecf20
commit 7927d6e539
16 changed files with 1757 additions and 17 deletions
+16 -7
View File
@@ -1,10 +1,12 @@
"""Engine alerts for the languages quick mode does not cover, produced on the host (harnessx-removal Req 6.3).
"""Engine alerts for the languages quick mode does not cover, and for PHP beside its quick rules
(``ENGINE_ALONGSIDE_QUICK``), produced on the host (harnessx-removal Req 6.3).
The loop's ``alerts`` task runs the quick-mode scan, which has rules only for the languages its ruleset declares; a
PHP case yields zero alerts that mean nothing (``coverage: none``). PHP detection comes from the Joern taint engine
(``ruleset/semantic/php.toml``), which runs in the ``openultrasast:dev`` image, not in the runner sandbox. So
``ousast plane alerts-engine <Run.yaml>`` does the ``alerts`` task's work on the host for every case whose files
include a language quick mode does not cover but the engine does:
case in any other language yields zero alerts that mean nothing (``coverage: none``). PHP has quick rules since
2026-09-30 (``ruleset/php/rules.toml``, line patterns over the engine's vocabulary), but its detection of record is
still the Joern taint engine (``ruleset/semantic/php.toml``), which runs in the ``openultrasast:dev`` image, not in
the runner sandbox. So ``ousast plane alerts-engine <Run.yaml>`` does the ``alerts`` task's work on the host for
every case whose files include a language the engine models and quick mode does not cover, or PHP:
- export the case's vulnerable and fixed pins from the case cache (``git archive``, as
``benchmarks/independent/evaluate.py`` does), run the quick scan on each (:func:`.tasks.alerts.scan`) and the
@@ -42,6 +44,10 @@ IMAGE = "openultrasast:dev"
MEMORY = "3g"
DEADLINE = 1800.0
GRACE = 600.0 # the container's own startup and teardown beyond the scan deadline
# Languages whose alerts of record come from the engine even though quick mode has rules for them. PHP's quick rules
# (ruleset/php, 2026-09-30) are a line layer with a low measured precision lower bound on the development corpus, so
# the engine keeps running for PHP beside them, and the coverage reports the engine for it.
ENGINE_ALONGSIDE_QUICK = frozenset({"php"})
CACHE = Path.home() / ".cache" / "openultrasast" / "independent"
DUMP = Path("benchmarks") / "push" / "finding_dump.py"
_READ = re.compile(r"^read (\d+) files, (\d+) bytes", re.MULTILINE)
@@ -193,7 +199,10 @@ def produce(
files[role] = read.pop("languages")
if read["files"] == 0:
raise EngineAlertsError(f"{case.case_id} {role}: the quick scan read no file of {checkout}")
needed |= {lang for lang in files[role] if lang != "unknown" and not covers(quick, lang) and lang in engine}
needed |= {
lang for lang in files[role]
if lang != "unknown" and lang in engine and (not covers(quick, lang) or lang in ENGINE_ALONGSIDE_QUICK)
} # fmt: skip
if not needed:
return {"case": case.case_id, "status": "skipped", "reason": "quick mode covers every language read", "languages": files}
out_dir = work / "out"
@@ -216,7 +225,7 @@ def produce(
summary["in_fix_range"][role] = sum(1 for r in found if r["in_fix_range"] is True)
keys = ("files", "bytes", "questions", "completed", "seconds", "degradations", "exit")
summary["engine"][role] = {**{k: record.get(k) for k in keys}, "findings": len(record.get("findings") or [])}
cov = coverage(files, quick, needed)
cov = coverage(files, set(quick) - needed, needed) # a language the engine ran for is reported as the engine's
summary.update(
status="done", units_done=len(ROLES), units_total=len(ROLES), usd=0, calls=0, usage={}, model=None, source="engine",
host=True, family=case.family, image=_image_id(image, runner), deadline=deadline, quick_languages=sorted(quick),