mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-19 04:59:28 +00:00
feat(ruleset): PHP quick-mode rules from the engine's vocabulary, measured on the development corpus
Quick mode had no PHP rules, so a PHP quick scan read the files and reported a plausible zero. ruleset/php/rules.toml adds 13 line/statement rules built on semantic/php.toml's sinks, sources and sanitizers (sql/wpdb, command, code_eval, unserialize, echo, file_inclusion, filesystem, outbound_request, redirect). 9 are enabled and 4 are shadow (sql-call-composition, code-eval, unserialize, xss-echo-input). A rule is shadow when it fired on the corpus but never at a known site. precision_estimate is the measured lower bound. Measured in-sample on the development corpus only (pmpro introduce/repair/benign, population-v1 PHP cases; population-v2 was not read). Known site hit on the vulnerable pin in 4 of 5 cases (LearnPress missed). The hit disappears on the fixed pin for pmpro and YesWiki. Numbers, per-pin files/bytes read and the engine comparison are in benchmarks/measurements/2026-09-30-php-quick-rules/. - fixtures php-vulnerable / php-benign plus manifests: ousast benchmark covers PHP (9/14 reported, 13/14 including shadow; the benign twin records 2 known false alerts) - PHP `#` comments are comments; `#[...]` attributes are not - the alerts task now reports PHP as quick coverage; alerts-engine keeps running the engine for PHP next to the quick rules (ENGINE_ALONGSIDE_QUICK) - other languages' benchmark findings and metrics are unchanged Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
45d02ecf20
commit
7927d6e539
@@ -1,10 +1,12 @@
|
||||
"""Engine alerts for the languages quick mode does not cover, produced on the host (harnessx-removal Req 6.3).
|
||||
"""Engine alerts for the languages quick mode does not cover, and for PHP beside its quick rules
|
||||
(``ENGINE_ALONGSIDE_QUICK``), produced on the host (harnessx-removal Req 6.3).
|
||||
|
||||
The loop's ``alerts`` task runs the quick-mode scan, which has rules only for the languages its ruleset declares; a
|
||||
PHP case yields zero alerts that mean nothing (``coverage: none``). PHP detection comes from the Joern taint engine
|
||||
(``ruleset/semantic/php.toml``), which runs in the ``openultrasast:dev`` image, not in the runner sandbox. So
|
||||
``ousast plane alerts-engine <Run.yaml>`` does the ``alerts`` task's work on the host for every case whose files
|
||||
include a language quick mode does not cover but the engine does:
|
||||
case in any other language yields zero alerts that mean nothing (``coverage: none``). PHP has quick rules since
|
||||
2026-09-30 (``ruleset/php/rules.toml``, line patterns over the engine's vocabulary), but its detection of record is
|
||||
still the Joern taint engine (``ruleset/semantic/php.toml``), which runs in the ``openultrasast:dev`` image, not in
|
||||
the runner sandbox. So ``ousast plane alerts-engine <Run.yaml>`` does the ``alerts`` task's work on the host for
|
||||
every case whose files include a language the engine models and quick mode does not cover, or PHP:
|
||||
|
||||
- export the case's vulnerable and fixed pins from the case cache (``git archive``, as
|
||||
``benchmarks/independent/evaluate.py`` does), run the quick scan on each (:func:`.tasks.alerts.scan`) and the
|
||||
@@ -42,6 +44,10 @@ IMAGE = "openultrasast:dev"
|
||||
MEMORY = "3g"
|
||||
DEADLINE = 1800.0
|
||||
GRACE = 600.0 # the container's own startup and teardown beyond the scan deadline
|
||||
# Languages whose alerts of record come from the engine even though quick mode has rules for them. PHP's quick rules
|
||||
# (ruleset/php, 2026-09-30) are a line layer with a low measured precision lower bound on the development corpus, so
|
||||
# the engine keeps running for PHP beside them, and the coverage reports the engine for it.
|
||||
ENGINE_ALONGSIDE_QUICK = frozenset({"php"})
|
||||
CACHE = Path.home() / ".cache" / "openultrasast" / "independent"
|
||||
DUMP = Path("benchmarks") / "push" / "finding_dump.py"
|
||||
_READ = re.compile(r"^read (\d+) files, (\d+) bytes", re.MULTILINE)
|
||||
@@ -193,7 +199,10 @@ def produce(
|
||||
files[role] = read.pop("languages")
|
||||
if read["files"] == 0:
|
||||
raise EngineAlertsError(f"{case.case_id} {role}: the quick scan read no file of {checkout}")
|
||||
needed |= {lang for lang in files[role] if lang != "unknown" and not covers(quick, lang) and lang in engine}
|
||||
needed |= {
|
||||
lang for lang in files[role]
|
||||
if lang != "unknown" and lang in engine and (not covers(quick, lang) or lang in ENGINE_ALONGSIDE_QUICK)
|
||||
} # fmt: skip
|
||||
if not needed:
|
||||
return {"case": case.case_id, "status": "skipped", "reason": "quick mode covers every language read", "languages": files}
|
||||
out_dir = work / "out"
|
||||
@@ -216,7 +225,7 @@ def produce(
|
||||
summary["in_fix_range"][role] = sum(1 for r in found if r["in_fix_range"] is True)
|
||||
keys = ("files", "bytes", "questions", "completed", "seconds", "degradations", "exit")
|
||||
summary["engine"][role] = {**{k: record.get(k) for k in keys}, "findings": len(record.get("findings") or [])}
|
||||
cov = coverage(files, quick, needed)
|
||||
cov = coverage(files, set(quick) - needed, needed) # a language the engine ran for is reported as the engine's
|
||||
summary.update(
|
||||
status="done", units_done=len(ROLES), units_total=len(ROLES), usd=0, calls=0, usage={}, model=None, source="engine",
|
||||
host=True, family=case.family, image=_image_id(image, runner), deadline=deadline, quick_languages=sorted(quick),
|
||||
|
||||
Reference in New Issue
Block a user