mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 11:49:28 +00:00
feat(plane): honest coverage in alerts; PHP alerts from the engine on the host
Quick mode has rules only for c/cpp, groovy, java, javascript/typescript and python, so every PHP case's "0 alerts" read as clean. The `alerts` task now records per language of the files read whether quick mode covers it (derived from the bundled ruleset, not a list): coverage quick | none, with file counts per pin, and `uncovered`. `remember` turns that into `coverage` memory rows (a new kind; the Run passes the alerts summary.json as input), loop-measure reports coverage per language, and improve/memory.py's M1/M2 skip a repository+pin whose coverage for the rule's language is only `none` (counted under `coverage_none` in the proposal's excluded). `ousast plane alerts-engine <Run.yaml> --source <frozen export>` (plane/engine_alerts.py) does the alerts task on the host for cases whose files include a language the engine covers but quick mode does not (PHP): exports both pins from the case cache, runs the quick scan and the openultrasast:dev engine container (finding_dump.py, --network none, --memory 3g, one at a time, per-pin deadline), writes engine findings in the alerts.jsonl row shape (rule_id engine:<family>, source engine, in_fix_range from case.json) with a summary (files read, questions, completed, seconds, degradations per pin), and marks the task done via reconciler.mark_done. No result, no files read, or questions == 0 fails loudly and leaves the run untouched. Refuses a git checkout as source. Runner image rebuilt and re-pinned (sha256:ebb40a9a...); validation-46 regenerated (digest only). Live, recorded in benchmarks/measurements/2026-09-30-php-engine-alerts/: froxlor 366 files, 130/130 questions per pin, 6 alerts per pin, 0 in the fix range; wp-gopay 19 files, 101/101 questions, 0 alerts. The loop: 32 run rows (coverage 8, php=engine), 0 proposals, gate no_proposals, $0. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
f4bd984fda
commit
45d02ecf20
@@ -149,6 +149,34 @@ def test_m2_stays_silent_on_a_fixed_pin_alert_or_rejected_candidates() -> None:
|
||||
assert propose([*site_hits(), *rejected], rules) == []
|
||||
|
||||
|
||||
# ---- coverage: no rule could fire, so no evidence either way -------------------------------------------------------
|
||||
|
||||
|
||||
def covered(repo: str, pin: str, kind: str, language: str = "python", role: str = "vulnerable") -> dict[str, Any]:
|
||||
return _row("coverage", repo, pin, f"{language}:{role}:{kind}", language=language, coverage=kind, files=3, pin_role=role)
|
||||
|
||||
|
||||
def test_m1_skips_a_repository_and_pin_whose_coverage_for_the_rules_language_is_none() -> None:
|
||||
blind = [*false_alerts(), covered("r2", P["a"], "none")]
|
||||
assert propose(blind, [_rule("noisy")]) == [], "r2's alert could not be evidence: one repository is left"
|
||||
wider = [*false_alerts(repos=("r1", "r1", "r2", "r3")), covered("r3", P["a"], "none")]
|
||||
[proposal] = propose(wider, [_rule("noisy")])
|
||||
assert proposal.provenance["counts"] == {"false_alerts": 3, "repos": 2, "agreed_hits": 0}
|
||||
assert proposal.provenance["excluded"]["coverage_none"] == {f"github.com/o/r3@{P['a']}": 1}
|
||||
|
||||
|
||||
def test_coverage_none_of_another_language_or_beside_engine_coverage_is_no_skip() -> None:
|
||||
other = [*false_alerts(), covered("r2", P["a"], "none", language="php")]
|
||||
assert len(propose(other, [_rule("noisy")])) == 1, "the rule's language (python) was covered there"
|
||||
engine = [*false_alerts(), covered("r2", P["a"], "none"), covered("r2", P["a"], "engine")]
|
||||
assert len(propose(engine, [_rule("noisy")])) == 1, "a later run covered the pin: its alerts are evidence"
|
||||
|
||||
|
||||
def test_m2_skips_a_blind_repository_and_pin() -> None:
|
||||
rules = [_rule("quiet", status="shadow"), _rule("other")]
|
||||
assert propose([*site_hits(), covered("r2", P["b"], "none")], rules) == []
|
||||
|
||||
|
||||
# ---- M3: advisory only ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user