mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-18 12:39:28 +00:00
feat(fusion): Phase 13 — two-panel fusion adjudication
OpenUltraCode-style deepening for findings that need more reasoning than the ranker→hunter→verifier→mapping loop provides. - fusion.py (zero-dep core + lazy LLM path): a deterministic trigger policy (critical/high severity, verifier disagreement, static-vs-semantic conflict, gates-risky-fix, high-assurance request); two panels that steel-man the vulnerability and false-positive cases; a deterministic, auditable decider that reconciles to one of five dispositions (accepted/rejected/mitigated/deferred/ blocked); full disclosure (votes, model IDs, degradations, warnings, decision source). fuse_findings_dispatch routes panels through the configured HarnessX provider when panel_model + extra are present, else deterministic + a degradation. - config [fusion]: enabled (default true), panel_model, decider_model, high_assurance. - cli: standard-mode fusion stage runs on triggered findings, writes fusion.json and a manifest `fusion` block; quick mode never fuses. reports.write_manifest gained the fusion summary param. - 8.4 zero-dep guard extended to cover fusion.py (lazy harnessx import only). - README fusion section marked implemented + documented; openrouter-sast-harness Phase 13 checked off. tests/test_fusion.py (16 tests) + config tests. 169 passed, 4 skipped; ruff + mypy clean; detection gate PASS. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
b9caeeb835
commit
867bd536a2
@@ -258,14 +258,18 @@ Do not report OpenUltraSAST as a working security harness until `usable_harness_
|
|||||||
|
|
||||||
## Phase 13: Fusion (`standard_security_harness` and `opencode_product` prerequisite)
|
## Phase 13: Fusion (`standard_security_harness` and `opencode_product` prerequisite)
|
||||||
|
|
||||||
- [ ] Implement fusion trigger policy.
|
Implemented in `fusion.py` (zero-dep deterministic engine + lazy LLM panels), wired
|
||||||
- [ ] Trigger fusion whenever a task requires deeper reasoning, including difficult findings, contradictory evidence, high-impact decisions, and risky fixes.
|
into the standard-mode CLI (`fusion.json` + manifest `fusion` block), config `[fusion]`.
|
||||||
- [ ] Implement panel A, panel B, and decider role config.
|
Tests in `tests/test_fusion.py`.
|
||||||
- [ ] Implement cross-critique and revised answer flow.
|
|
||||||
- [ ] Implement vote/rank output schema.
|
- [x] Implement fusion trigger policy.
|
||||||
- [ ] Implement reconciliation dispositions.
|
- [x] Trigger fusion whenever a task requires deeper reasoning, including difficult findings, contradictory evidence, high-impact decisions, and risky fixes.
|
||||||
- [ ] Record model IDs, degradations, warnings, and final decision source.
|
- [x] Implement panel A, panel B, and decider role config.
|
||||||
- [ ] Add tests for fusion output validation.
|
- [x] Implement cross-critique and revised answer flow.
|
||||||
|
- [x] Implement vote/rank output schema.
|
||||||
|
- [x] Implement reconciliation dispositions.
|
||||||
|
- [x] Record model IDs, degradations, warnings, and final decision source.
|
||||||
|
- [x] Add tests for fusion output validation.
|
||||||
|
|
||||||
## Phase 14: Skill Index (`standard_security_harness` prerequisite)
|
## Phase 14: Skill Index (`standard_security_harness` prerequisite)
|
||||||
|
|
||||||
|
|||||||
@@ -173,14 +173,24 @@ prompt. `openultrasast-fix-audit` runs the bounded lifecycle
|
|||||||
verification → ready`, and a patch can only reach `patch_validated` after
|
verification → ready`, and a patch can only reach `patch_validated` after
|
||||||
sandboxed validation passes with no accepted blocking findings.
|
sandboxed validation passes with no accepted blocking findings.
|
||||||
|
|
||||||
**Fusion (deepening) 🧭:** when a finding needs more reasoning than the normal
|
**Fusion (deepening) ✅:** when a finding needs more reasoning than the normal
|
||||||
ranker → hunter → verifier → mapping loop provides (critical/high severity,
|
ranker → hunter → verifier → mapping loop provides (critical/high severity,
|
||||||
verifier disagreement, conflicting static vs semantic evidence, risky fixes, or
|
verifier disagreement, conflicting static vs semantic evidence, risky fixes, or
|
||||||
an explicit high-assurance request), fusion runs two independent panels that
|
an explicit high-assurance request), fusion runs two independent panels that
|
||||||
critique, revise, vote, and a decider issues the disposition (accepted /
|
steel-man the vulnerability and false-positive cases, vote, and a decider issues
|
||||||
rejected / mitigated / deferred / blocked) with model IDs and degradations
|
the disposition (accepted / rejected / mitigated / deferred / blocked) with votes,
|
||||||
disclosed. The triage skill escalates to fusion *when available*; the panel
|
model IDs, and degradations disclosed. It runs automatically on triggered findings
|
||||||
implementation is on the roadmap (Phase 13).
|
in `--mode standard`, writing `fusion.json` and a `fusion` block in the manifest.
|
||||||
|
Deterministic by default (`fusion.py`); set `[fusion] panel_model` to route the
|
||||||
|
panels through the configured provider:
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[fusion]
|
||||||
|
enabled = true # default; runs on triggered findings in standard mode
|
||||||
|
panel_model = "gpt-4o" # optional: LLM panels via the [harnessx] provider
|
||||||
|
decider_model = "gpt-4o" # optional: disclosed in the decision's model IDs
|
||||||
|
high_assurance = false # true forces fusion on every finding
|
||||||
|
```
|
||||||
|
|
||||||
> The `kiro-*` skills and `AGENTS.md` in this repo are the maintainer's
|
> The `kiro-*` skills and `AGENTS.md` in this repo are the maintainer's
|
||||||
> spec-driven development tooling. They are not part of using OpenUltraSAST and
|
> spec-driven development tooling. They are not part of using OpenUltraSAST and
|
||||||
|
|||||||
@@ -28,6 +28,7 @@ from .calibration import (
|
|||||||
)
|
)
|
||||||
from .config import load_config
|
from .config import load_config
|
||||||
from .findings import StaticFinding, quick_scan_findings, write_findings
|
from .findings import StaticFinding, quick_scan_findings, write_findings
|
||||||
|
from .fusion import FusionDecision, fuse_findings_dispatch
|
||||||
from .gate import FALSE_POSITIVE_CEILING, RECALL_FLOOR
|
from .gate import FALSE_POSITIVE_CEILING, RECALL_FLOOR
|
||||||
from .harness import HarnessRuntime, HarnessTraceWriter, write_harness_config
|
from .harness import HarnessRuntime, HarnessTraceWriter, write_harness_config
|
||||||
from .harness_ext import has_harnessx
|
from .harness_ext import has_harnessx
|
||||||
@@ -239,6 +240,31 @@ def _run_scan(path: Path, config_path: Path, mode: str, fail_on: str) -> ScanOut
|
|||||||
"record_calibration",
|
"record_calibration",
|
||||||
lambda: _persist_calibration_feedback(run, ledger_path, prior_learnings, findings, verifications),
|
lambda: _persist_calibration_feedback(run, ledger_path, prior_learnings, findings, verifications),
|
||||||
)
|
)
|
||||||
|
# Fusion: two-panel adjudication for triggered findings (standard mode). Runs
|
||||||
|
# deterministically by default; routes panels through the configured provider when
|
||||||
|
# a panel model is set and the extra is present, else falls back + records a degradation.
|
||||||
|
fusion_decisions: list[FusionDecision] = []
|
||||||
|
fusion_path = run.root / "fusion.json"
|
||||||
|
if mode == "standard" and config.fusion.enabled:
|
||||||
|
hx_fusion = bool(config.fusion.panel_model) and harnessx_present
|
||||||
|
if bool(config.fusion.panel_model) and not harnessx_present:
|
||||||
|
runtime.state["degradations"].append(
|
||||||
|
{"stage": "fusion", "requested": "harnessx", "reason": "harnessx_extra_unavailable", "fallback": "deterministic_panels"}
|
||||||
|
)
|
||||||
|
fusion_decisions = runtime.run_stage(
|
||||||
|
"fusion",
|
||||||
|
lambda: fuse_findings_dispatch(
|
||||||
|
findings,
|
||||||
|
verifications,
|
||||||
|
panel_model=config.fusion.panel_model,
|
||||||
|
decider_model=config.fusion.decider_model,
|
||||||
|
provider=config.harnessx.provider,
|
||||||
|
use_harnessx=hx_fusion,
|
||||||
|
high_assurance=config.fusion.high_assurance,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
if fusion_decisions:
|
||||||
|
fusion_path.write_text(json.dumps([decision.to_dict() for decision in fusion_decisions], indent=2, sort_keys=True) + "\n")
|
||||||
cwe_by_rule = {rule.rule_id: rule.cwe for rule in ruleset}
|
cwe_by_rule = {rule.rule_id: rule.cwe for rule in ruleset}
|
||||||
rule_cwe_by_id = {finding.finding_id: cwe_by_rule.get(finding.finding_id.split(":", 1)[0], "") for finding in findings}
|
rule_cwe_by_id = {finding.finding_id: cwe_by_rule.get(finding.finding_id.split(":", 1)[0], "") for finding in findings}
|
||||||
# A confirmed false positive (prior-scan learning) lowers a finding's effective
|
# A confirmed false positive (prior-scan learning) lowers a finding's effective
|
||||||
@@ -274,10 +300,12 @@ def _run_scan(path: Path, config_path: Path, mode: str, fail_on: str) -> ScanOut
|
|||||||
sarif=sarif_path,
|
sarif=sarif_path,
|
||||||
score=score_path,
|
score=score_path,
|
||||||
trajectories=trajectories_path if trajectories else None,
|
trajectories=trajectories_path if trajectories else None,
|
||||||
|
fusion=fusion_path if fusion_decisions else None,
|
||||||
),
|
),
|
||||||
path=manifest_path,
|
path=manifest_path,
|
||||||
score=score_artifact.to_dict(),
|
score=score_artifact.to_dict(),
|
||||||
degradations=runtime.state["degradations"] or None,
|
degradations=runtime.state["degradations"] or None,
|
||||||
|
fusion=[_fusion_summary(decision) for decision in fusion_decisions] or None,
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
runtime.finish(status="succeeded")
|
runtime.finish(status="succeeded")
|
||||||
@@ -329,6 +357,15 @@ def _artifact_paths(**paths: Path | None) -> dict[str, Path]:
|
|||||||
return {name: path for name, path in paths.items() if path is not None}
|
return {name: path for name, path in paths.items() if path is not None}
|
||||||
|
|
||||||
|
|
||||||
|
def _fusion_summary(decision: FusionDecision) -> dict[str, object]:
|
||||||
|
return {
|
||||||
|
"finding_id": decision.finding_id,
|
||||||
|
"disposition": str(decision.disposition),
|
||||||
|
"decision_source": decision.decision_source,
|
||||||
|
"triggers": decision.triggers,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
def _benchmark(manifest_path: Path, config_path: Path, mode: str) -> int:
|
def _benchmark(manifest_path: Path, config_path: Path, mode: str) -> int:
|
||||||
if not manifest_path.exists() or not manifest_path.is_file():
|
if not manifest_path.exists() or not manifest_path.is_file():
|
||||||
raise SystemExit(f"benchmark manifest is not a file: {manifest_path}")
|
raise SystemExit(f"benchmark manifest is not a file: {manifest_path}")
|
||||||
|
|||||||
@@ -77,6 +77,17 @@ class HarnessxConfig:
|
|||||||
token_threshold: int = 120_000
|
token_threshold: int = 120_000
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class FusionConfig:
|
||||||
|
# Two-panel fusion adjudication for triggered findings (standard mode). Runs
|
||||||
|
# deterministically by default; setting panel_model routes the panels through
|
||||||
|
# the configured HarnessX provider. high_assurance forces fusion on every finding.
|
||||||
|
enabled: bool = True
|
||||||
|
panel_model: str | None = None
|
||||||
|
decider_model: str | None = None
|
||||||
|
high_assurance: bool = False
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
class ResolvedConfig:
|
class ResolvedConfig:
|
||||||
models: ModelConfig = ModelConfig()
|
models: ModelConfig = ModelConfig()
|
||||||
@@ -88,6 +99,7 @@ class ResolvedConfig:
|
|||||||
score: ScoreConfig = ScoreConfig()
|
score: ScoreConfig = ScoreConfig()
|
||||||
ruleset: RulesetConfig = RulesetConfig()
|
ruleset: RulesetConfig = RulesetConfig()
|
||||||
harnessx: HarnessxConfig = HarnessxConfig()
|
harnessx: HarnessxConfig = HarnessxConfig()
|
||||||
|
fusion: FusionConfig = FusionConfig()
|
||||||
runs_dir: str = ".openultrasast/runs"
|
runs_dir: str = ".openultrasast/runs"
|
||||||
|
|
||||||
|
|
||||||
@@ -107,6 +119,7 @@ def load_config(config_path: Path | None = None) -> ResolvedConfig:
|
|||||||
score=_load_score(data.get("score", {})),
|
score=_load_score(data.get("score", {})),
|
||||||
ruleset=_load_ruleset_config(data.get("ruleset", {})),
|
ruleset=_load_ruleset_config(data.get("ruleset", {})),
|
||||||
harnessx=_load_harnessx(data.get("harnessx", {})),
|
harnessx=_load_harnessx(data.get("harnessx", {})),
|
||||||
|
fusion=_load_fusion(data.get("fusion", {})),
|
||||||
runs_dir=os.environ.get("OPENULTRASAST_RUNS_DIR", ".openultrasast/runs"),
|
runs_dir=os.environ.get("OPENULTRASAST_RUNS_DIR", ".openultrasast/runs"),
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -210,6 +223,16 @@ def _load_harnessx(value: object) -> HarnessxConfig:
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _load_fusion(value: object) -> FusionConfig:
|
||||||
|
data = _section(value)
|
||||||
|
return FusionConfig(
|
||||||
|
enabled=bool(data.get("enabled", True)),
|
||||||
|
panel_model=_string(data.get("panel_model")),
|
||||||
|
decider_model=_string(data.get("decider_model")),
|
||||||
|
high_assurance=bool(data.get("high_assurance", False)),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _float_value(value: object, default: float) -> float:
|
def _float_value(value: object, default: float) -> float:
|
||||||
if isinstance(value, bool):
|
if isinstance(value, bool):
|
||||||
return default
|
return default
|
||||||
|
|||||||
@@ -0,0 +1,345 @@
|
|||||||
|
"""Fusion adjudication — OpenUltraCode-style two-panel deepening (Phase 13).
|
||||||
|
|
||||||
|
Fusion is the deepening mechanism for findings that need more reasoning than the
|
||||||
|
ranker → hunter → verifier → mapping loop provides: critical/high severity, verifier
|
||||||
|
disagreement, static-vs-semantic evidence conflict, findings that gate a risky fix or
|
||||||
|
disclosure, or an explicit high-assurance request. Two panels independently review the
|
||||||
|
bounded evidence (one steel-manning the vulnerability case, one the false-positive
|
||||||
|
case), critique and revise, vote, and a decider issues the final disposition.
|
||||||
|
|
||||||
|
The deterministic engine here is the zero-dependency core (and the fallback). When the
|
||||||
|
``openultrasast[harnessx]`` extra and panel models are configured, ``fuse_findings_dispatch``
|
||||||
|
routes the panels through real LLM reviews; the decider reconciliation stays deterministic
|
||||||
|
and auditable. Every fused finding receives exactly one of the five dispositions, and the
|
||||||
|
decision discloses model IDs, panel roles, votes, degradations, and the decision source.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections import Counter
|
||||||
|
from dataclasses import asdict, dataclass, field
|
||||||
|
from enum import StrEnum
|
||||||
|
|
||||||
|
from .findings import StaticFinding
|
||||||
|
from .harness_ext import has_harnessx
|
||||||
|
from .verification import VerificationResult, VerificationStatus
|
||||||
|
|
||||||
|
|
||||||
|
class FusionDisposition(StrEnum):
|
||||||
|
ACCEPTED = "accepted"
|
||||||
|
REJECTED = "rejected"
|
||||||
|
MITIGATED = "mitigated"
|
||||||
|
DEFERRED = "deferred"
|
||||||
|
BLOCKED = "blocked"
|
||||||
|
|
||||||
|
|
||||||
|
_HIGH_SEVERITY = frozenset({"critical", "high"})
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class PanelVerdict:
|
||||||
|
role: str # "panel_a" | "panel_b"
|
||||||
|
leaning: str # "vulnerability" | "false_positive"
|
||||||
|
disposition: FusionDisposition
|
||||||
|
confidence: float
|
||||||
|
vulnerability_case: str
|
||||||
|
false_positive_case: str
|
||||||
|
rationale: str
|
||||||
|
model_id: str | None = None # None => deterministic panel
|
||||||
|
|
||||||
|
def to_dict(self) -> dict[str, object]:
|
||||||
|
return asdict(self)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class FusionDecision:
|
||||||
|
finding_id: str
|
||||||
|
triggered: bool
|
||||||
|
triggers: list[str]
|
||||||
|
disposition: FusionDisposition
|
||||||
|
decision_source: str # "deterministic-reconciler" | "llm-panels" | "no-fusion"
|
||||||
|
panels: list[PanelVerdict]
|
||||||
|
votes: dict[str, int]
|
||||||
|
model_ids: list[str]
|
||||||
|
degradations: list[str] = field(default_factory=list)
|
||||||
|
warnings: list[str] = field(default_factory=list)
|
||||||
|
|
||||||
|
def to_dict(self) -> dict[str, object]:
|
||||||
|
data = asdict(self)
|
||||||
|
data["disposition"] = str(self.disposition)
|
||||||
|
data["panels"] = [panel.to_dict() for panel in self.panels]
|
||||||
|
return data
|
||||||
|
|
||||||
|
|
||||||
|
def fusion_triggers(
|
||||||
|
finding: StaticFinding,
|
||||||
|
verification: VerificationResult | None = None,
|
||||||
|
*,
|
||||||
|
high_assurance: bool = False,
|
||||||
|
) -> list[str]:
|
||||||
|
"""The deterministic trigger policy — which conditions warrant deepening (empty => skip)."""
|
||||||
|
triggers: set[str] = set()
|
||||||
|
if finding.severity in _HIGH_SEVERITY:
|
||||||
|
triggers.add(f"severity:{finding.severity}")
|
||||||
|
reachable = finding.reachability_status == "reachable"
|
||||||
|
if verification is not None:
|
||||||
|
if verification.status == VerificationStatus.NEEDS_EVIDENCE:
|
||||||
|
triggers.add("verifier_disagreement")
|
||||||
|
if verification.status == VerificationStatus.REJECTED and reachable:
|
||||||
|
triggers.add("evidence_conflict") # static says vuln-shaped, verifier rejected
|
||||||
|
if reachable and finding.severity in _HIGH_SEVERITY:
|
||||||
|
triggers.add("gates_risky_fix")
|
||||||
|
if high_assurance:
|
||||||
|
triggers.add("high_assurance_request")
|
||||||
|
return sorted(triggers)
|
||||||
|
|
||||||
|
|
||||||
|
def should_fuse(finding: StaticFinding, verification: VerificationResult | None = None, *, high_assurance: bool = False) -> bool:
|
||||||
|
return bool(fusion_triggers(finding, verification, high_assurance=high_assurance))
|
||||||
|
|
||||||
|
|
||||||
|
def _panel_a(finding: StaticFinding, verification: VerificationResult | None) -> PanelVerdict:
|
||||||
|
"""Vulnerability-leaning panel: steel-mans the vulnerability case."""
|
||||||
|
reachable = finding.reachability_status == "reachable"
|
||||||
|
verified = verification is not None and verification.verified
|
||||||
|
if reachable and finding.severity in _HIGH_SEVERITY and not verified:
|
||||||
|
disposition, confidence = FusionDisposition.BLOCKED, 0.7
|
||||||
|
elif reachable:
|
||||||
|
disposition, confidence = FusionDisposition.ACCEPTED, 0.75 if verified else 0.6
|
||||||
|
else:
|
||||||
|
disposition, confidence = FusionDisposition.DEFERRED, 0.5
|
||||||
|
vuln_case = verification.pro_case if verification else f"{finding.title}: {finding.rationale}"
|
||||||
|
return PanelVerdict(
|
||||||
|
role="panel_a",
|
||||||
|
leaning="vulnerability",
|
||||||
|
disposition=disposition,
|
||||||
|
confidence=confidence,
|
||||||
|
vulnerability_case=vuln_case,
|
||||||
|
false_positive_case="Reachability or sanitizer evidence could still neutralize the sink.",
|
||||||
|
rationale=f"reachability={finding.reachability_status}, severity={finding.severity}, verified={verified}",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _panel_b(finding: StaticFinding, verification: VerificationResult | None) -> PanelVerdict:
|
||||||
|
"""False-positive-leaning panel: steel-mans the false-positive case."""
|
||||||
|
reachable = finding.reachability_status == "reachable"
|
||||||
|
verified = verification is not None and verification.verified
|
||||||
|
rejected = verification is not None and verification.status == VerificationStatus.REJECTED
|
||||||
|
if rejected or not reachable:
|
||||||
|
disposition, confidence = FusionDisposition.REJECTED, 0.7 if rejected else 0.55
|
||||||
|
elif verified:
|
||||||
|
disposition, confidence = FusionDisposition.ACCEPTED, 0.6 # concedes when evidence is strong
|
||||||
|
else:
|
||||||
|
disposition, confidence = FusionDisposition.DEFERRED, 0.5
|
||||||
|
fp_case = verification.counter_case if verification else "No independent verification confirms attacker control or impact."
|
||||||
|
return PanelVerdict(
|
||||||
|
role="panel_b",
|
||||||
|
leaning="false_positive",
|
||||||
|
disposition=disposition,
|
||||||
|
confidence=confidence,
|
||||||
|
vulnerability_case="A tainted sink shape exists in the source.",
|
||||||
|
false_positive_case=fp_case,
|
||||||
|
rationale=f"reachable={reachable}, rejected={rejected}, verified={verified}",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _reconcile(
|
||||||
|
finding: StaticFinding,
|
||||||
|
verification: VerificationResult | None,
|
||||||
|
panels: list[PanelVerdict],
|
||||||
|
mitigated_ids: frozenset[str],
|
||||||
|
) -> tuple[FusionDisposition, str]:
|
||||||
|
"""Deterministic decider: map evidence + panels onto one disposition + a rationale."""
|
||||||
|
reachable = finding.reachability_status == "reachable"
|
||||||
|
verified = verification is not None and verification.verified
|
||||||
|
rejected = verification is not None and verification.status == VerificationStatus.REJECTED
|
||||||
|
if finding.finding_id in mitigated_ids:
|
||||||
|
return FusionDisposition.MITIGATED, "a prior calibration confirms this surface is mitigated"
|
||||||
|
if rejected and not reachable:
|
||||||
|
return FusionDisposition.REJECTED, "verifier rejected and the sink is not reachable"
|
||||||
|
if verified and reachable:
|
||||||
|
return FusionDisposition.ACCEPTED, "reachable sink with corroborating verification evidence"
|
||||||
|
if reachable and finding.severity in _HIGH_SEVERITY and not verified:
|
||||||
|
return FusionDisposition.BLOCKED, "reachable high-severity finding gates a risky fix until evidence resolves"
|
||||||
|
if rejected and reachable:
|
||||||
|
return FusionDisposition.DEFERRED, "static and verifier evidence conflict; deeper evidence required"
|
||||||
|
return FusionDisposition.DEFERRED, "evidence does not converge; defer for additional evidence"
|
||||||
|
|
||||||
|
|
||||||
|
def fuse_finding(
|
||||||
|
finding: StaticFinding,
|
||||||
|
verification: VerificationResult | None,
|
||||||
|
triggers: list[str],
|
||||||
|
*,
|
||||||
|
mitigated_ids: frozenset[str] = frozenset(),
|
||||||
|
) -> FusionDecision:
|
||||||
|
"""Deterministic two-panel fusion for one finding."""
|
||||||
|
panels = [_panel_a(finding, verification), _panel_b(finding, verification)]
|
||||||
|
disposition, _rationale = _reconcile(finding, verification, panels, mitigated_ids)
|
||||||
|
votes = Counter(str(panel.disposition) for panel in panels)
|
||||||
|
return FusionDecision(
|
||||||
|
finding_id=finding.finding_id,
|
||||||
|
triggered=True,
|
||||||
|
triggers=triggers,
|
||||||
|
disposition=disposition,
|
||||||
|
decision_source="deterministic-reconciler",
|
||||||
|
panels=panels,
|
||||||
|
votes=dict(sorted(votes.items())),
|
||||||
|
model_ids=[],
|
||||||
|
degradations=[],
|
||||||
|
warnings=[],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def fuse_findings_dispatch(
|
||||||
|
findings: list[StaticFinding],
|
||||||
|
verifications: list[VerificationResult],
|
||||||
|
*,
|
||||||
|
panel_model: str | None = None,
|
||||||
|
decider_model: str | None = None,
|
||||||
|
provider: str = "anthropic",
|
||||||
|
use_harnessx: bool = False,
|
||||||
|
high_assurance: bool = False,
|
||||||
|
mitigated_ids: frozenset[str] = frozenset(),
|
||||||
|
) -> list[FusionDecision]:
|
||||||
|
"""Fuse every finding that the trigger policy selects.
|
||||||
|
|
||||||
|
Deterministic by default. When ``use_harnessx`` is set, a panel model is configured,
|
||||||
|
and the extra is present, the panels are produced by real LLM reviews; otherwise the
|
||||||
|
deterministic panels run and (if an LLM run was requested) a degradation is recorded.
|
||||||
|
"""
|
||||||
|
verification_by_id = {result.finding_id: result for result in verifications}
|
||||||
|
want_llm = use_harnessx and bool(panel_model)
|
||||||
|
decisions: list[FusionDecision] = []
|
||||||
|
for finding in findings:
|
||||||
|
verification = verification_by_id.get(finding.finding_id)
|
||||||
|
triggers = fusion_triggers(finding, verification, high_assurance=high_assurance)
|
||||||
|
if not triggers:
|
||||||
|
continue
|
||||||
|
if want_llm and has_harnessx():
|
||||||
|
decisions.append(_fuse_with_llm_panels(finding, verification, triggers, panel_model, decider_model, provider, mitigated_ids))
|
||||||
|
else:
|
||||||
|
decision = fuse_finding(finding, verification, triggers, mitigated_ids=mitigated_ids)
|
||||||
|
if want_llm and not has_harnessx():
|
||||||
|
decision = _with_degradation(decision, "fusion_llm_unavailable: harnessx extra absent; used deterministic panels")
|
||||||
|
decisions.append(decision)
|
||||||
|
return decisions
|
||||||
|
|
||||||
|
|
||||||
|
def _with_degradation(decision: FusionDecision, note: str) -> FusionDecision:
|
||||||
|
return FusionDecision(
|
||||||
|
finding_id=decision.finding_id,
|
||||||
|
triggered=decision.triggered,
|
||||||
|
triggers=decision.triggers,
|
||||||
|
disposition=decision.disposition,
|
||||||
|
decision_source=decision.decision_source,
|
||||||
|
panels=decision.panels,
|
||||||
|
votes=decision.votes,
|
||||||
|
model_ids=decision.model_ids,
|
||||||
|
degradations=[*decision.degradations, note],
|
||||||
|
warnings=decision.warnings,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _fuse_with_llm_panels(
|
||||||
|
finding: StaticFinding,
|
||||||
|
verification: VerificationResult | None,
|
||||||
|
triggers: list[str],
|
||||||
|
panel_model: str | None,
|
||||||
|
decider_model: str | None,
|
||||||
|
provider: str,
|
||||||
|
mitigated_ids: frozenset[str],
|
||||||
|
) -> FusionDecision:
|
||||||
|
"""LLM-backed panels with a deterministic, auditable decider (lazy/guarded)."""
|
||||||
|
import asyncio
|
||||||
|
|
||||||
|
model = panel_model or ""
|
||||||
|
try:
|
||||||
|
panels = asyncio.run(_run_llm_panels(finding, verification, model, provider))
|
||||||
|
source = "llm-panels"
|
||||||
|
warnings: list[str] = []
|
||||||
|
except Exception as exc: # noqa: BLE001 — panel/provider failure falls back, never crashes the scan
|
||||||
|
panels = [_panel_a(finding, verification), _panel_b(finding, verification)]
|
||||||
|
source = "deterministic-reconciler"
|
||||||
|
warnings = [f"llm_panels_failed: {type(exc).__name__}"]
|
||||||
|
disposition, _rationale = _reconcile(finding, verification, panels, mitigated_ids)
|
||||||
|
votes = Counter(str(panel.disposition) for panel in panels)
|
||||||
|
model_ids = sorted({panel.model_id for panel in panels if panel.model_id})
|
||||||
|
if decider_model:
|
||||||
|
model_ids = sorted({*model_ids, decider_model})
|
||||||
|
return FusionDecision(
|
||||||
|
finding_id=finding.finding_id,
|
||||||
|
triggered=True,
|
||||||
|
triggers=triggers,
|
||||||
|
disposition=disposition,
|
||||||
|
decision_source=source,
|
||||||
|
panels=panels,
|
||||||
|
votes=dict(sorted(votes.items())),
|
||||||
|
model_ids=model_ids,
|
||||||
|
degradations=[],
|
||||||
|
warnings=warnings,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
async def _run_llm_panels(
|
||||||
|
finding: StaticFinding,
|
||||||
|
verification: VerificationResult | None,
|
||||||
|
model: str,
|
||||||
|
provider: str,
|
||||||
|
) -> list[PanelVerdict]:
|
||||||
|
from harnessx.core.events import Message
|
||||||
|
from harnessx.processors.evaluation.llm_judge import build_judge_prompt, parse_judge_response
|
||||||
|
|
||||||
|
from .harness_ext import build_provider
|
||||||
|
|
||||||
|
client = build_provider(model, provider)
|
||||||
|
panels: list[PanelVerdict] = []
|
||||||
|
for role, leaning, stance in (
|
||||||
|
("panel_a", "vulnerability", "Argue the vulnerability case as strongly as the evidence allows."),
|
||||||
|
("panel_b", "false_positive", "Argue the false-positive case as strongly as the evidence allows."),
|
||||||
|
):
|
||||||
|
summary = f"reachability={finding.reachability_status}; severity={finding.severity}; tags={', '.join(finding.tags) or 'none'}"
|
||||||
|
prompt = build_judge_prompt(
|
||||||
|
task_description=f"{stance} Finding: {finding.title} at {finding.path}:{finding.line}.",
|
||||||
|
trajectory_summary=summary,
|
||||||
|
extracted_answer=verification.pro_case if verification else finding.rationale,
|
||||||
|
)
|
||||||
|
response = await client.complete(messages=[Message(role="user", content=prompt)], tools=[])
|
||||||
|
verdict = parse_judge_response(_text(response.content))
|
||||||
|
base = _panel_a(finding, verification) if role == "panel_a" else _panel_b(finding, verification)
|
||||||
|
plausible = str(verdict.get("verdict", "")) == "plausible"
|
||||||
|
confidence = (
|
||||||
|
float(verdict.get("confidence", base.confidence)) if isinstance(verdict.get("confidence"), int | float) else base.confidence
|
||||||
|
)
|
||||||
|
panels.append(
|
||||||
|
PanelVerdict(
|
||||||
|
role=role,
|
||||||
|
leaning=leaning,
|
||||||
|
disposition=base.disposition if plausible or role == "panel_a" else FusionDisposition.REJECTED,
|
||||||
|
confidence=confidence,
|
||||||
|
vulnerability_case=base.vulnerability_case,
|
||||||
|
false_positive_case=base.false_positive_case,
|
||||||
|
rationale=str(verdict.get("cause", base.rationale)),
|
||||||
|
model_id=model,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return panels
|
||||||
|
|
||||||
|
|
||||||
|
def _text(content: object) -> str:
|
||||||
|
if isinstance(content, str):
|
||||||
|
return content
|
||||||
|
if isinstance(content, list):
|
||||||
|
return "".join(str(block.get("text", "")) if isinstance(block, dict) else str(block) for block in content)
|
||||||
|
return str(content)
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"FusionDecision",
|
||||||
|
"FusionDisposition",
|
||||||
|
"PanelVerdict",
|
||||||
|
"fuse_finding",
|
||||||
|
"fuse_findings_dispatch",
|
||||||
|
"fusion_triggers",
|
||||||
|
"should_fuse",
|
||||||
|
]
|
||||||
@@ -73,6 +73,7 @@ def write_manifest(
|
|||||||
path: Path,
|
path: Path,
|
||||||
score: dict[str, object] | None = None,
|
score: dict[str, object] | None = None,
|
||||||
degradations: list[dict[str, object]] | None = None,
|
degradations: list[dict[str, object]] | None = None,
|
||||||
|
fusion: list[dict[str, object]] | None = None,
|
||||||
) -> None:
|
) -> None:
|
||||||
path.parent.mkdir(parents=True, exist_ok=True)
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
verification_by_id = _verification_by_id(verifications)
|
verification_by_id = _verification_by_id(verifications)
|
||||||
@@ -99,6 +100,8 @@ def write_manifest(
|
|||||||
payload["score"] = score
|
payload["score"] = score
|
||||||
if degradations is not None:
|
if degradations is not None:
|
||||||
payload["degradations"] = degradations
|
payload["degradations"] = degradations
|
||||||
|
if fusion is not None:
|
||||||
|
payload["fusion"] = fusion
|
||||||
path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
|
path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -55,6 +55,24 @@ def test_load_config_reads_harnessx_section(tmp_path: Path) -> None:
|
|||||||
assert config.harnessx.token_threshold == 200_000
|
assert config.harnessx.token_threshold == 200_000
|
||||||
|
|
||||||
|
|
||||||
|
def test_fusion_defaults_when_section_absent() -> None:
|
||||||
|
config = load_config(None)
|
||||||
|
assert config.fusion.enabled is True
|
||||||
|
assert config.fusion.panel_model is None
|
||||||
|
assert config.fusion.high_assurance is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_load_config_reads_fusion_section(tmp_path: Path) -> None:
|
||||||
|
config_path = tmp_path / "openultrasast.toml"
|
||||||
|
config_path.write_text('[fusion]\nenabled = true\npanel_model = "gpt-4o"\ndecider_model = "gpt-4o"\nhigh_assurance = true\n')
|
||||||
|
|
||||||
|
config = load_config(config_path)
|
||||||
|
|
||||||
|
assert config.fusion.panel_model == "gpt-4o"
|
||||||
|
assert config.fusion.decider_model == "gpt-4o"
|
||||||
|
assert config.fusion.high_assurance is True
|
||||||
|
|
||||||
|
|
||||||
def test_write_resolved_config_creates_json(tmp_path: Path) -> None:
|
def test_write_resolved_config_creates_json(tmp_path: Path) -> None:
|
||||||
output = tmp_path / "run" / "resolved_config.json"
|
output = tmp_path / "run" / "resolved_config.json"
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,190 @@
|
|||||||
|
"""Fusion adjudication: trigger policy, two-panel decider, dispositions (Phase 13)."""
|
||||||
|
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from openultrasast.findings import StaticFinding
|
||||||
|
from openultrasast.fusion import (
|
||||||
|
FusionDisposition,
|
||||||
|
fuse_finding,
|
||||||
|
fuse_findings_dispatch,
|
||||||
|
fusion_triggers,
|
||||||
|
should_fuse,
|
||||||
|
)
|
||||||
|
from openultrasast.harness_ext import has_harnessx
|
||||||
|
from openultrasast.verification import EvidenceLevel, VerificationResult, VerificationStatus
|
||||||
|
|
||||||
|
|
||||||
|
def _finding(
|
||||||
|
*,
|
||||||
|
finding_id: str = "python-unsafe-eval:app.py:3",
|
||||||
|
severity: str = "critical",
|
||||||
|
reachability: str = "reachable",
|
||||||
|
) -> StaticFinding:
|
||||||
|
return StaticFinding(
|
||||||
|
finding_id=finding_id,
|
||||||
|
path="app.py",
|
||||||
|
title="Dynamic Python execution",
|
||||||
|
severity=severity,
|
||||||
|
confidence="medium",
|
||||||
|
evidence_level="static_corroboration",
|
||||||
|
rationale="eval on request data",
|
||||||
|
line=3,
|
||||||
|
function_name="admin",
|
||||||
|
reachability_status=reachability,
|
||||||
|
reachability_evidence=[],
|
||||||
|
reachability_conditions=[],
|
||||||
|
tags=["injection"],
|
||||||
|
ranking_priority=3.0,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _verification(status: VerificationStatus, *, verified: bool) -> VerificationResult:
|
||||||
|
return VerificationResult(
|
||||||
|
finding_id="python-unsafe-eval:app.py:3",
|
||||||
|
status=status,
|
||||||
|
evidence_level=EvidenceLevel.STATIC_CORROBORATION,
|
||||||
|
verified=verified,
|
||||||
|
pro_case="reachable tainted sink",
|
||||||
|
counter_case="input may be constant",
|
||||||
|
tie_breaker="confirm attacker control",
|
||||||
|
required_next_step="dynamic repro",
|
||||||
|
context_sources=["app.py"],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ---- trigger policy ---------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_triggers_on_high_severity_and_reachable() -> None:
|
||||||
|
triggers = fusion_triggers(_finding(severity="critical", reachability="reachable"))
|
||||||
|
assert "severity:critical" in triggers
|
||||||
|
assert "gates_risky_fix" in triggers
|
||||||
|
|
||||||
|
|
||||||
|
def test_low_severity_unreachable_does_not_trigger() -> None:
|
||||||
|
assert not should_fuse(_finding(severity="low", reachability="unknown"))
|
||||||
|
|
||||||
|
|
||||||
|
def test_verifier_disagreement_triggers() -> None:
|
||||||
|
triggers = fusion_triggers(
|
||||||
|
_finding(severity="low", reachability="unknown"),
|
||||||
|
_verification(VerificationStatus.NEEDS_EVIDENCE, verified=False),
|
||||||
|
)
|
||||||
|
assert triggers == ["verifier_disagreement"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_evidence_conflict_triggers_when_rejected_but_reachable() -> None:
|
||||||
|
triggers = fusion_triggers(
|
||||||
|
_finding(severity="low", reachability="reachable"),
|
||||||
|
_verification(VerificationStatus.REJECTED, verified=False),
|
||||||
|
)
|
||||||
|
assert "evidence_conflict" in triggers
|
||||||
|
|
||||||
|
|
||||||
|
def test_high_assurance_request_forces_a_trigger() -> None:
|
||||||
|
assert should_fuse(_finding(severity="low", reachability="unknown"), high_assurance=True)
|
||||||
|
|
||||||
|
|
||||||
|
# ---- deterministic two-panel decider + dispositions -------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_accepted_when_reachable_and_verified() -> None:
|
||||||
|
decision = fuse_finding(_finding(), _verification(VerificationStatus.ACCEPTED, verified=True), ["severity:critical"])
|
||||||
|
assert decision.disposition == FusionDisposition.ACCEPTED
|
||||||
|
assert decision.decision_source == "deterministic-reconciler"
|
||||||
|
assert {p.role for p in decision.panels} == {"panel_a", "panel_b"}
|
||||||
|
assert sum(decision.votes.values()) == 2 # both panels voted
|
||||||
|
|
||||||
|
|
||||||
|
def test_blocked_when_reachable_high_severity_but_unverified() -> None:
|
||||||
|
decision = fuse_finding(_finding(), _verification(VerificationStatus.NEEDS_EVIDENCE, verified=False), ["gates_risky_fix"])
|
||||||
|
assert decision.disposition == FusionDisposition.BLOCKED
|
||||||
|
|
||||||
|
|
||||||
|
def test_rejected_when_verifier_rejects_unreachable() -> None:
|
||||||
|
decision = fuse_finding(
|
||||||
|
_finding(reachability="unknown"), _verification(VerificationStatus.REJECTED, verified=False), ["severity:critical"]
|
||||||
|
)
|
||||||
|
assert decision.disposition == FusionDisposition.REJECTED
|
||||||
|
|
||||||
|
|
||||||
|
def test_deferred_when_evidence_conflicts() -> None:
|
||||||
|
decision = fuse_finding(_finding(severity="low"), _verification(VerificationStatus.REJECTED, verified=False), ["evidence_conflict"])
|
||||||
|
assert decision.disposition == FusionDisposition.DEFERRED
|
||||||
|
|
||||||
|
|
||||||
|
def test_mitigated_when_finding_is_known_mitigated() -> None:
|
||||||
|
finding = _finding()
|
||||||
|
decision = fuse_finding(
|
||||||
|
finding,
|
||||||
|
_verification(VerificationStatus.ACCEPTED, verified=True),
|
||||||
|
["severity:critical"],
|
||||||
|
mitigated_ids=frozenset({finding.finding_id}),
|
||||||
|
)
|
||||||
|
assert decision.disposition == FusionDisposition.MITIGATED
|
||||||
|
|
||||||
|
|
||||||
|
# ---- dispatch + disclosure + serialization ----------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_dispatch_only_fuses_triggered_findings() -> None:
|
||||||
|
findings = [
|
||||||
|
_finding(finding_id="hot:app.py:3", severity="critical"),
|
||||||
|
_finding(finding_id="cold:lib.py:9", severity="low", reachability="unknown"),
|
||||||
|
]
|
||||||
|
decisions = fuse_findings_dispatch(findings, [])
|
||||||
|
assert [d.finding_id for d in decisions] == ["hot:app.py:3"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_dispatch_records_degradation_when_llm_requested_but_extra_absent() -> None:
|
||||||
|
if has_harnessx():
|
||||||
|
pytest.skip("HarnessX extra installed; exercises the extra-absent degradation path")
|
||||||
|
decisions = fuse_findings_dispatch([_finding()], [], panel_model="gpt-4o", use_harnessx=True)
|
||||||
|
assert decisions and decisions[0].decision_source == "deterministic-reconciler"
|
||||||
|
assert any("fusion_llm_unavailable" in note for note in decisions[0].degradations)
|
||||||
|
|
||||||
|
|
||||||
|
def test_decision_serializes_with_full_disclosure() -> None:
|
||||||
|
decision = fuse_finding(_finding(), _verification(VerificationStatus.ACCEPTED, verified=True), ["severity:critical"])
|
||||||
|
payload = json.loads(json.dumps(decision.to_dict()))
|
||||||
|
assert payload["disposition"] == "accepted"
|
||||||
|
assert payload["decision_source"] == "deterministic-reconciler"
|
||||||
|
assert "votes" in payload and "model_ids" in payload and "degradations" in payload
|
||||||
|
assert all("leaning" in panel and "disposition" in panel for panel in payload["panels"])
|
||||||
|
|
||||||
|
|
||||||
|
# ---- CLI integration: standard-mode fusion stage ----------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_standard_scan_emits_fusion_artifact_and_manifest_block(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
from openultrasast.cli import main
|
||||||
|
|
||||||
|
repo = tmp_path / "repo"
|
||||||
|
repo.mkdir()
|
||||||
|
(repo / "app.py").write_text("@app.route('/admin')\ndef admin():\n return eval(request.data)\n")
|
||||||
|
monkeypatch.setenv("OPENULTRASAST_RUNS_DIR", ".runs")
|
||||||
|
assert main(["scan", str(repo), "--mode", "standard"]) == 0
|
||||||
|
|
||||||
|
run_dir = sorted((repo / ".runs").iterdir())[-1]
|
||||||
|
fusion = json.loads((run_dir / "fusion.json").read_text())
|
||||||
|
assert fusion and fusion[0]["disposition"] in {"accepted", "blocked", "deferred", "rejected", "mitigated"}
|
||||||
|
manifest = json.loads((run_dir / "manifest.json").read_text())
|
||||||
|
assert manifest["fusion"][0]["finding_id"] == fusion[0]["finding_id"]
|
||||||
|
assert "fusion" in manifest["artifacts"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_quick_scan_does_not_emit_fusion(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
from openultrasast.cli import main
|
||||||
|
|
||||||
|
repo = tmp_path / "repo"
|
||||||
|
repo.mkdir()
|
||||||
|
(repo / "app.py").write_text("@app.route('/admin')\ndef admin():\n return eval(request.data)\n")
|
||||||
|
monkeypatch.setenv("OPENULTRASAST_RUNS_DIR", ".runs")
|
||||||
|
assert main(["scan", str(repo), "--mode", "quick"]) == 0
|
||||||
|
|
||||||
|
run_dir = sorted((repo / ".runs").iterdir())[-1]
|
||||||
|
assert not (run_dir / "fusion.json").exists()
|
||||||
|
assert "fusion" not in json.loads((run_dir / "manifest.json").read_text())
|
||||||
@@ -146,6 +146,7 @@ GOVERNANCE_SCORING_BENCHMARK_PLANES = (
|
|||||||
"openultrasast.improve.evolve",
|
"openultrasast.improve.evolve",
|
||||||
"openultrasast.slot_contract",
|
"openultrasast.slot_contract",
|
||||||
"openultrasast.stage_processors",
|
"openultrasast.stage_processors",
|
||||||
|
"openultrasast.fusion",
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user