mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
feat(slots): Phase 3 task 6.3 — model-disabled, slot-contracted deterministic stages
Express the deterministic scan stages as model-disabled processors carrying work in
named slots under a read/write allow-list validated on every lifecycle hook.
- slot_contract.py (zero-dep): SlotContractError, SlotContractMixin (reads_slots/
writes_slots/skip_model=True, validates reads pre-hook and writes post-hook),
SlotPipeline (runs stages in order, enforces the model-disabled invariant, emits
stage_start/stage_end lifecycle hooks). Preserves the ProcessorSpec.reads/writes
provenance discipline without forcing deterministic stages through HarnessX's agent
loop ("bridge, do not map").
- stage_processors.py: Preprocess/EntryPoint/Rank/QuickFindings/PolicyScoring/Score as
skip_model slot processors; run_quick_pipeline is byte-identical to the quick path.
host_under_harnessx is the lazy, capability-gated bridge that disables the model on
HarnessX's BeforeModelEvent.skip_model (proven against the real API when the extra
is present).
- 8.4 zero-dep guard hardened: now hermetic (subprocess) and covers the two new
modules; the module-level import scan flags only column-0 imports so the intended
lazy import inside host_under_harnessx is allowed.
The sync HarnessRuntime stays the top-level driver (design "stays through Phase 4");
these slot processors are the model-disabled representation, ready for Phase 5.
tests/test_slot_contract.py (8 tests). 153 passed, 3 skipped; ruff + mypy clean;
detection gate PASS.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
bebfc72cf7
commit
b9caeeb835
@@ -1,7 +1,7 @@
|
|||||||
{
|
{
|
||||||
"feature_name": "harnessx-self-improving-rulesets",
|
"feature_name": "harnessx-self-improving-rulesets",
|
||||||
"created_at": "2026-06-22T10:07:18Z",
|
"created_at": "2026-06-22T10:07:18Z",
|
||||||
"updated_at": "2026-06-22T16:20:00Z",
|
"updated_at": "2026-06-22T16:45:00Z",
|
||||||
"language": "en",
|
"language": "en",
|
||||||
"phase": "implementation",
|
"phase": "implementation",
|
||||||
"approvals": {
|
"approvals": {
|
||||||
|
|||||||
@@ -138,7 +138,7 @@ This plan splits the fused regex layer into three governed owners (rules detect,
|
|||||||
- _Boundary: Packaging Configuration_
|
- _Boundary: Packaging Configuration_
|
||||||
- _Depends: 5.1_
|
- _Depends: 5.1_
|
||||||
|
|
||||||
- [ ] 6. Adopt HarnessX at the hunter loop and verifier
|
- [x] 6. Adopt HarnessX at the hunter loop and verifier
|
||||||
- [x] 6.1 Run the hunter pool stage on a real HarnessX agent loop
|
- [x] 6.1 Run the hunter pool stage on a real HarnessX agent loop
|
||||||
- Run the hunter pool stage on a real HarnessX run loop with the control, tool, and observability processors registered in the hunter sub-harness, invoked from the existing hunter-pool stage callback.
|
- Run the hunter pool stage on a real HarnessX run loop with the control, tool, and observability processors registered in the hunter sub-harness, invoked from the existing hunter-pool stage callback.
|
||||||
- Confine all asynchronous execution to this single stage so the deterministic stages remain synchronous.
|
- Confine all asynchronous execution to this single stage so the deterministic stages remain synchronous.
|
||||||
@@ -154,10 +154,11 @@ This plan splits the fused regex layer into three governed owners (rules detect,
|
|||||||
- _Boundary: HxScanOrchestrator_
|
- _Boundary: HxScanOrchestrator_
|
||||||
- _Depends: 6.1_
|
- _Depends: 6.1_
|
||||||
|
|
||||||
- [ ] 6.3 (P) Run deterministic stages as model-disabled HarnessX processors with slot contracts
|
- [x] 6.3 (P) Run deterministic stages as model-disabled HarnessX processors with slot contracts
|
||||||
- Run each deterministic stage under HarnessX with model invocation disabled so no deterministic stage incurs a model call.
|
- Run each deterministic stage under HarnessX with model invocation disabled so no deterministic stage incurs a model call.
|
||||||
- Preserve each stage's declared read/write slot allow-list and validate slot access on every lifecycle hook.
|
- Preserve each stage's declared read/write slot allow-list and validate slot access on every lifecycle hook.
|
||||||
- Observable completion: deterministic stages execute under HarnessX with no model calls and a slot-contract violation is raised on undeclared slot access, proven by a slot-contract test.
|
- Observable completion: deterministic stages execute under HarnessX with no model calls and a slot-contract violation is raised on undeclared slot access, proven by a slot-contract test.
|
||||||
|
- Implemented zero-dep in `slot_contract.py` (`SlotContractMixin`/`SlotPipeline`, validates reads/writes per lifecycle hook, enforces `skip_model`) + `stage_processors.py` (the deterministic stages as `skip_model` slot processors; `run_quick_pipeline` is byte-identical to the quick path). Per "bridge, do not map", the sync `HarnessRuntime` stays the top-level driver and these slot processors are the model-disabled representation; `host_under_harnessx` is the lazy, capability-gated bridge that disables the model on HarnessX's `BeforeModelEvent.skip_model`. Module names differ from the design's `harness/{slot_contract,processors}.py` because `harness` is a module, not a package. Tests: `tests/test_slot_contract.py`.
|
||||||
- _Requirements: 1.5, 1.6_
|
- _Requirements: 1.5, 1.6_
|
||||||
- _Boundary: HxScanOrchestrator_
|
- _Boundary: HxScanOrchestrator_
|
||||||
- _Depends: 6.1_
|
- _Depends: 6.1_
|
||||||
|
|||||||
@@ -0,0 +1,93 @@
|
|||||||
|
"""Slot-contract discipline for model-disabled deterministic stages (task 6.3).
|
||||||
|
|
||||||
|
The deterministic scan stages carry their work in named *slots* and run with model
|
||||||
|
invocation disabled (``skip_model = True``). Each stage declares a read/write slot
|
||||||
|
allow-list that is validated on every lifecycle hook, preserving the
|
||||||
|
``ProcessorSpec.reads/writes`` provenance discipline of the sync ``HarnessRuntime``
|
||||||
|
(the top-level driver) — without forcing the deterministic stages through HarnessX's
|
||||||
|
agent loop ("bridge, do not map", per the spec design).
|
||||||
|
|
||||||
|
This is the zero-dependency substrate. When the optional ``openultrasast[harnessx]``
|
||||||
|
extra is present, the same model-disabled processors can be hosted under a HarnessX
|
||||||
|
``MultiHookProcessor`` (see ``stage_processors.host_under_harnessx``); the
|
||||||
|
``skip_model`` marker maps onto HarnessX's ``BeforeModelEvent.skip_model`` so no
|
||||||
|
deterministic stage incurs a model call.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Mapping
|
||||||
|
from typing import Any, Protocol
|
||||||
|
|
||||||
|
from .harness import HarnessContractError
|
||||||
|
|
||||||
|
|
||||||
|
class SlotContractError(HarnessContractError):
|
||||||
|
"""Raised when a stage reads or writes a slot outside its declared allow-list."""
|
||||||
|
|
||||||
|
|
||||||
|
class SlotContractMixin:
|
||||||
|
"""A model-disabled stage that carries work in slots under a read/write allow-list."""
|
||||||
|
|
||||||
|
name: str = "slot-stage"
|
||||||
|
version: str = "1"
|
||||||
|
reads_slots: tuple[str, ...] = ()
|
||||||
|
writes_slots: tuple[str, ...] = ()
|
||||||
|
skip_model: bool = True # deterministic: never invokes a model
|
||||||
|
|
||||||
|
def run(self, slots: Mapping[str, Any]) -> dict[str, Any]:
|
||||||
|
raise NotImplementedError
|
||||||
|
|
||||||
|
def validate_reads(self, slots: Mapping[str, Any]) -> None:
|
||||||
|
missing = sorted(slot for slot in self.reads_slots if slot not in slots)
|
||||||
|
if missing:
|
||||||
|
raise SlotContractError(f"{self.name}: missing read slots: {', '.join(missing)}")
|
||||||
|
|
||||||
|
def validate_writes(self, updates: Mapping[str, Any]) -> None:
|
||||||
|
undeclared = sorted(set(updates) - set(self.writes_slots))
|
||||||
|
if undeclared:
|
||||||
|
raise SlotContractError(f"{self.name}: undeclared write slots: {', '.join(undeclared)}")
|
||||||
|
|
||||||
|
def execute(self, slots: Mapping[str, Any]) -> dict[str, Any]:
|
||||||
|
"""Validate the read allow-list, run the stage, validate the write allow-list."""
|
||||||
|
self.validate_reads(slots) # pre-hook
|
||||||
|
updates = self.run(slots) or {}
|
||||||
|
if not isinstance(updates, dict):
|
||||||
|
raise SlotContractError(f"{self.name}: stage returned a non-dict update")
|
||||||
|
self.validate_writes(updates) # post-hook
|
||||||
|
return updates
|
||||||
|
|
||||||
|
|
||||||
|
class _Emitter(Protocol):
|
||||||
|
def emit(self, event_type: Any, stage: str | None, payload: dict[str, Any]) -> None: ...
|
||||||
|
|
||||||
|
|
||||||
|
class SlotPipeline:
|
||||||
|
"""Runs slot processors in order, validating contracts and enforcing model-disable."""
|
||||||
|
|
||||||
|
def __init__(self, processors: list[SlotContractMixin]) -> None:
|
||||||
|
for proc in processors:
|
||||||
|
if not proc.skip_model:
|
||||||
|
raise SlotContractError(f"{proc.name}: deterministic stage must be model-disabled (skip_model=True)")
|
||||||
|
self._processors = processors
|
||||||
|
|
||||||
|
@property
|
||||||
|
def processors(self) -> list[SlotContractMixin]:
|
||||||
|
return list(self._processors)
|
||||||
|
|
||||||
|
def run(self, initial_slots: Mapping[str, Any], *, runtime: _Emitter | None = None) -> dict[str, Any]:
|
||||||
|
"""Execute every stage; with ``runtime`` set, emit stage_start/stage_end lifecycle hooks."""
|
||||||
|
slots: dict[str, Any] = dict(initial_slots)
|
||||||
|
for proc in self._processors:
|
||||||
|
if runtime is not None:
|
||||||
|
runtime.emit(
|
||||||
|
"stage_start", proc.name, {"reads": list(proc.reads_slots), "writes": list(proc.writes_slots), "skip_model": True}
|
||||||
|
)
|
||||||
|
updates = proc.execute(slots)
|
||||||
|
slots.update(updates)
|
||||||
|
if runtime is not None:
|
||||||
|
runtime.emit("stage_end", proc.name, {"status": "succeeded", "wrote": sorted(updates)})
|
||||||
|
return slots
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = ["SlotContractError", "SlotContractMixin", "SlotPipeline"]
|
||||||
@@ -0,0 +1,170 @@
|
|||||||
|
"""Deterministic scan stages as model-disabled, slot-contracted processors (task 6.3).
|
||||||
|
|
||||||
|
Each stage is a :class:`slot_contract.SlotContractMixin` with ``skip_model = True`` and
|
||||||
|
a declared read/write slot allow-list. Running them through a :class:`SlotPipeline`
|
||||||
|
reproduces the deterministic quick-scan path byte-for-byte while enforcing the slot
|
||||||
|
contract on every lifecycle hook. The hunter and verifier stages are *not* here —
|
||||||
|
they are the agentic plane (``hunter_harness``/``verify_judge``); this module is the
|
||||||
|
deterministic, model-free remainder.
|
||||||
|
|
||||||
|
``host_under_harnessx`` is the optional, lazy bridge: it wraps a model-disabled stage
|
||||||
|
as a HarnessX ``MultiHookProcessor`` that disables the model call on ``before_model``.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Mapping
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from .findings import quick_scan_findings
|
||||||
|
from .mapping import analyze_entry_points, attach_reachability_hints
|
||||||
|
from .policy import load_policy
|
||||||
|
from .preprocess import preprocess_repository
|
||||||
|
from .rank import rank_targets
|
||||||
|
from .ruleset import DEFAULT_RULESET_DIR, load_ruleset
|
||||||
|
from .scoring import build_score_artifact
|
||||||
|
from .slot_contract import SlotContractMixin, SlotPipeline
|
||||||
|
|
||||||
|
|
||||||
|
class PreprocessProcessor(SlotContractMixin):
|
||||||
|
name = "preprocess"
|
||||||
|
reads_slots = ("root",)
|
||||||
|
writes_slots = ("snapshot", "targets")
|
||||||
|
|
||||||
|
def run(self, slots: Mapping[str, Any]) -> dict[str, Any]:
|
||||||
|
snapshot, targets = preprocess_repository(slots["root"])
|
||||||
|
return {"snapshot": snapshot, "targets": targets}
|
||||||
|
|
||||||
|
|
||||||
|
class EntryPointProcessor(SlotContractMixin):
|
||||||
|
name = "entry_point"
|
||||||
|
reads_slots = ("root", "targets")
|
||||||
|
writes_slots = ("targets", "entry_points")
|
||||||
|
|
||||||
|
def run(self, slots: Mapping[str, Any]) -> dict[str, Any]:
|
||||||
|
entry_points = analyze_entry_points(slots["root"], slots["targets"])
|
||||||
|
targets = attach_reachability_hints(slots["targets"], entry_points)
|
||||||
|
return {"targets": targets, "entry_points": entry_points}
|
||||||
|
|
||||||
|
|
||||||
|
class RankProcessor(SlotContractMixin):
|
||||||
|
name = "rank"
|
||||||
|
reads_slots = ("targets",)
|
||||||
|
writes_slots = ("rankings",)
|
||||||
|
|
||||||
|
def run(self, slots: Mapping[str, Any]) -> dict[str, Any]:
|
||||||
|
return {"rankings": rank_targets(slots["targets"])}
|
||||||
|
|
||||||
|
|
||||||
|
class QuickFindingsProcessor(SlotContractMixin):
|
||||||
|
name = "quick_findings"
|
||||||
|
reads_slots = ("root", "targets", "rankings", "ruleset", "policy")
|
||||||
|
writes_slots = ("findings",)
|
||||||
|
|
||||||
|
def run(self, slots: Mapping[str, Any]) -> dict[str, Any]:
|
||||||
|
findings = quick_scan_findings(slots["root"], slots["targets"], slots["rankings"], slots["ruleset"], slots["policy"])
|
||||||
|
return {"findings": findings}
|
||||||
|
|
||||||
|
|
||||||
|
class PolicyScoringProcessor(SlotContractMixin):
|
||||||
|
name = "policy_scoring"
|
||||||
|
reads_slots = ("findings",)
|
||||||
|
writes_slots = ("reported_findings", "shadow_findings")
|
||||||
|
|
||||||
|
def run(self, slots: Mapping[str, Any]) -> dict[str, Any]:
|
||||||
|
findings = slots["findings"]
|
||||||
|
shadow = [finding for finding in findings if finding.status == "shadow"]
|
||||||
|
reported = [finding for finding in findings if finding.status != "shadow"]
|
||||||
|
return {"reported_findings": reported, "shadow_findings": shadow}
|
||||||
|
|
||||||
|
|
||||||
|
class ScoreProcessor(SlotContractMixin):
|
||||||
|
name = "score"
|
||||||
|
reads_slots = ("reported_findings", "ruleset", "policy")
|
||||||
|
writes_slots = ("score",)
|
||||||
|
|
||||||
|
def run(self, slots: Mapping[str, Any]) -> dict[str, Any]:
|
||||||
|
cwe_by_rule = {rule.rule_id: rule.cwe for rule in slots["ruleset"]}
|
||||||
|
reported = slots["reported_findings"]
|
||||||
|
rule_cwe_by_id = {f.finding_id: cwe_by_rule.get(f.finding_id.split(":", 1)[0], "") for f in reported}
|
||||||
|
return {"score": build_score_artifact(reported, rule_cwe_by_id, slots["policy"])}
|
||||||
|
|
||||||
|
|
||||||
|
def deterministic_processors() -> list[SlotContractMixin]:
|
||||||
|
"""The ordered deterministic stages (preprocess → entry-point → rank → findings → policy → score)."""
|
||||||
|
return [
|
||||||
|
PreprocessProcessor(),
|
||||||
|
EntryPointProcessor(),
|
||||||
|
RankProcessor(),
|
||||||
|
QuickFindingsProcessor(),
|
||||||
|
PolicyScoringProcessor(),
|
||||||
|
ScoreProcessor(),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def build_deterministic_pipeline() -> SlotPipeline:
|
||||||
|
return SlotPipeline(deterministic_processors())
|
||||||
|
|
||||||
|
|
||||||
|
def run_quick_pipeline(
|
||||||
|
root: Path | str,
|
||||||
|
*,
|
||||||
|
ruleset: Any = None,
|
||||||
|
policy: Any = None,
|
||||||
|
runtime: Any = None,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
"""Run the deterministic quick-scan pipeline and return the final slot map.
|
||||||
|
|
||||||
|
Defaults to the bundled ruleset and the vendored policy, which makes the
|
||||||
|
``reported_findings`` slot byte-identical to ``findings.quick_scan_findings`` on
|
||||||
|
the same target.
|
||||||
|
"""
|
||||||
|
resolved_ruleset = ruleset if ruleset is not None else load_ruleset(DEFAULT_RULESET_DIR)
|
||||||
|
resolved_policy = policy if policy is not None else load_policy()
|
||||||
|
initial = {"root": Path(root), "ruleset": resolved_ruleset, "policy": resolved_policy}
|
||||||
|
return build_deterministic_pipeline().run(initial, runtime=runtime)
|
||||||
|
|
||||||
|
|
||||||
|
def host_under_harnessx(processor: SlotContractMixin) -> Any:
|
||||||
|
"""Host a model-disabled deterministic stage under a HarnessX ``MultiHookProcessor``.
|
||||||
|
|
||||||
|
Lazy + capability-guarded (imports ``harnessx`` only when called). On the
|
||||||
|
``before_model`` hook the hosted stage yields a ``BeforeModelEvent`` with
|
||||||
|
``skip_model=True``, so no deterministic stage incurs a model call when it runs on
|
||||||
|
the HarnessX runloop. Raises :class:`HarnessXUnavailableError` when the extra is absent.
|
||||||
|
"""
|
||||||
|
import dataclasses
|
||||||
|
|
||||||
|
from .harness_ext import require_harnessx
|
||||||
|
|
||||||
|
require_harnessx()
|
||||||
|
from harnessx.core.events import BeforeModelEvent
|
||||||
|
from harnessx.core.processor import MultiHookProcessor
|
||||||
|
|
||||||
|
class _ModelDisabledStage(MultiHookProcessor): # type: ignore[misc, valid-type]
|
||||||
|
_singleton_group = "ousast_deterministic"
|
||||||
|
_order = 50
|
||||||
|
|
||||||
|
def __init__(self, stage: SlotContractMixin) -> None:
|
||||||
|
self.stage = stage
|
||||||
|
self.name = stage.name
|
||||||
|
|
||||||
|
async def on_before_model(self, event: BeforeModelEvent): # type: ignore[no-untyped-def]
|
||||||
|
yield dataclasses.replace(event, skip_model=True, synthetic_output=f"[deterministic stage {self.stage.name}: model disabled]")
|
||||||
|
|
||||||
|
return _ModelDisabledStage(processor)
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"EntryPointProcessor",
|
||||||
|
"PolicyScoringProcessor",
|
||||||
|
"PreprocessProcessor",
|
||||||
|
"QuickFindingsProcessor",
|
||||||
|
"RankProcessor",
|
||||||
|
"ScoreProcessor",
|
||||||
|
"build_deterministic_pipeline",
|
||||||
|
"deterministic_processors",
|
||||||
|
"host_under_harnessx",
|
||||||
|
"run_quick_pipeline",
|
||||||
|
]
|
||||||
+10
-10
@@ -6,9 +6,7 @@
|
|||||||
8.4 — the governance/scoring/benchmark planes never import the HarnessX extra.
|
8.4 — the governance/scoring/benchmark planes never import the HarnessX extra.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import importlib
|
|
||||||
import json
|
import json
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
@@ -146,17 +144,20 @@ GOVERNANCE_SCORING_BENCHMARK_PLANES = (
|
|||||||
"openultrasast.ruleset.store",
|
"openultrasast.ruleset.store",
|
||||||
"openultrasast.improve",
|
"openultrasast.improve",
|
||||||
"openultrasast.improve.evolve",
|
"openultrasast.improve.evolve",
|
||||||
|
"openultrasast.slot_contract",
|
||||||
|
"openultrasast.stage_processors",
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_governance_scoring_benchmark_planes_never_import_harnessx() -> None:
|
def test_governance_scoring_benchmark_planes_never_import_harnessx(assert_cold_of_harnessx) -> None: # type: ignore[no-untyped-def]
|
||||||
for module in GOVERNANCE_SCORING_BENCHMARK_PLANES:
|
# Hermetic: a fresh interpreter imports every plane and must pull no harnessx.
|
||||||
importlib.import_module(module)
|
imports = "\n".join(f"import {module}" for module in GOVERNANCE_SCORING_BENCHMARK_PLANES)
|
||||||
leaked = sorted(name for name in sys.modules if name == "harnessx" or name.startswith("harnessx."))
|
assert_cold_of_harnessx(imports)
|
||||||
assert leaked == [], f"governance/scoring/benchmark planes leaked HarnessX imports: {leaked}"
|
|
||||||
|
|
||||||
|
|
||||||
def test_planes_have_no_module_level_harnessx_import() -> None:
|
def test_planes_have_no_module_level_harnessx_import() -> None:
|
||||||
|
# Flags only column-0 (module-level) imports; an indented lazy import inside a
|
||||||
|
# capability-guarded function (e.g. stage_processors.host_under_harnessx) is fine.
|
||||||
src = Path("src/openultrasast")
|
src = Path("src/openultrasast")
|
||||||
offenders = []
|
offenders = []
|
||||||
for module in GOVERNANCE_SCORING_BENCHMARK_PLANES:
|
for module in GOVERNANCE_SCORING_BENCHMARK_PLANES:
|
||||||
@@ -166,7 +167,6 @@ def test_planes_have_no_module_level_harnessx_import() -> None:
|
|||||||
if path is None:
|
if path is None:
|
||||||
continue
|
continue
|
||||||
for line in path.read_text().splitlines():
|
for line in path.read_text().splitlines():
|
||||||
stripped = line.strip()
|
if line.startswith(("import harnessx", "from harnessx")):
|
||||||
if stripped.startswith(("import harnessx", "from harnessx")):
|
offenders.append(f"{path}: {line}")
|
||||||
offenders.append(f"{path}: {stripped}")
|
|
||||||
assert offenders == [], f"plane modules import HarnessX at module scope: {offenders}"
|
assert offenders == [], f"plane modules import HarnessX at module scope: {offenders}"
|
||||||
|
|||||||
@@ -0,0 +1,133 @@
|
|||||||
|
"""Model-disabled, slot-contracted deterministic stages (Phase 3 task 6.3).
|
||||||
|
|
||||||
|
Proves: deterministic stages carry work in declared slots, a slot-contract violation
|
||||||
|
is raised on undeclared slot access, the pipeline is model-disabled, the slot pipeline
|
||||||
|
reproduces the deterministic quick-scan path byte-for-byte, lifecycle hooks fire, and
|
||||||
|
(when the extra is present) a stage can run under HarnessX with the model disabled.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from collections.abc import Mapping
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from openultrasast.gate import _quick_scan
|
||||||
|
from openultrasast.harness_ext import has_harnessx
|
||||||
|
from openultrasast.slot_contract import SlotContractError, SlotContractMixin, SlotPipeline
|
||||||
|
from openultrasast.stage_processors import (
|
||||||
|
PreprocessProcessor,
|
||||||
|
build_deterministic_pipeline,
|
||||||
|
deterministic_processors,
|
||||||
|
host_under_harnessx,
|
||||||
|
run_quick_pipeline,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class _GoodStage(SlotContractMixin):
|
||||||
|
name = "good"
|
||||||
|
reads_slots = ("a",)
|
||||||
|
writes_slots = ("b",)
|
||||||
|
|
||||||
|
def run(self, slots: Mapping[str, Any]) -> dict[str, Any]:
|
||||||
|
return {"b": slots["a"] + 1}
|
||||||
|
|
||||||
|
|
||||||
|
class _UndeclaredWriteStage(SlotContractMixin):
|
||||||
|
name = "bad_write"
|
||||||
|
reads_slots = ()
|
||||||
|
writes_slots = ("declared",)
|
||||||
|
|
||||||
|
def run(self, slots: Mapping[str, Any]) -> dict[str, Any]:
|
||||||
|
return {"undeclared": 1}
|
||||||
|
|
||||||
|
|
||||||
|
class _MissingReadStage(SlotContractMixin):
|
||||||
|
name = "bad_read"
|
||||||
|
reads_slots = ("required",)
|
||||||
|
writes_slots = ()
|
||||||
|
|
||||||
|
def run(self, slots: Mapping[str, Any]) -> dict[str, Any]:
|
||||||
|
return {}
|
||||||
|
|
||||||
|
|
||||||
|
# ---- slot contract enforcement (1.5) ----------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_declared_slots_pass() -> None:
|
||||||
|
assert _GoodStage().execute({"a": 1}) == {"b": 2}
|
||||||
|
|
||||||
|
|
||||||
|
def test_undeclared_write_is_rejected() -> None:
|
||||||
|
with pytest.raises(SlotContractError, match="undeclared write slots: undeclared"):
|
||||||
|
_UndeclaredWriteStage().execute({})
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_read_is_rejected() -> None:
|
||||||
|
with pytest.raises(SlotContractError, match="missing read slots: required"):
|
||||||
|
_MissingReadStage().execute({})
|
||||||
|
|
||||||
|
|
||||||
|
# ---- model-disabled invariant (1.6) -----------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_all_deterministic_stages_are_model_disabled() -> None:
|
||||||
|
assert all(proc.skip_model for proc in deterministic_processors())
|
||||||
|
|
||||||
|
|
||||||
|
def test_pipeline_rejects_a_model_enabled_stage() -> None:
|
||||||
|
enabled = _GoodStage()
|
||||||
|
enabled.skip_model = False
|
||||||
|
with pytest.raises(SlotContractError, match="model-disabled"):
|
||||||
|
SlotPipeline([enabled])
|
||||||
|
|
||||||
|
|
||||||
|
# ---- parity with the deterministic quick-scan path --------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_pipeline_findings_match_quick_scan_baseline() -> None:
|
||||||
|
target = Path("benchmarks/fixtures/python-vulnerable")
|
||||||
|
slots = run_quick_pipeline(target)
|
||||||
|
assert slots["reported_findings"] == _quick_scan(target)
|
||||||
|
assert "score" in slots # the score slot is produced too
|
||||||
|
|
||||||
|
|
||||||
|
# ---- lifecycle hooks fire per stage -----------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
class _Recorder:
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self.events: list[tuple[str, str | None]] = []
|
||||||
|
|
||||||
|
def emit(self, event_type: Any, stage: str | None, payload: dict[str, Any]) -> None:
|
||||||
|
self.events.append((event_type, stage))
|
||||||
|
|
||||||
|
|
||||||
|
def test_pipeline_emits_lifecycle_hooks_per_stage() -> None:
|
||||||
|
recorder = _Recorder()
|
||||||
|
run_quick_pipeline(Path("benchmarks/fixtures/python-vulnerable"), runtime=recorder)
|
||||||
|
names = [proc.name for proc in build_deterministic_pipeline().processors]
|
||||||
|
for name in names:
|
||||||
|
assert ("stage_start", name) in recorder.events
|
||||||
|
assert ("stage_end", name) in recorder.events
|
||||||
|
|
||||||
|
|
||||||
|
# ---- optional: run under HarnessX with the model disabled (1.6) --------------
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.skipif(not has_harnessx(), reason="requires the harnessx extra")
|
||||||
|
def test_host_under_harnessx_disables_the_model() -> None: # pragma: no cover - needs the extra
|
||||||
|
import asyncio
|
||||||
|
|
||||||
|
from harnessx.core.events import BeforeModelEvent
|
||||||
|
from harnessx.core.processor import MultiHookProcessor
|
||||||
|
|
||||||
|
hosted = host_under_harnessx(PreprocessProcessor())
|
||||||
|
assert isinstance(hosted, MultiHookProcessor)
|
||||||
|
|
||||||
|
async def drive() -> list[Any]:
|
||||||
|
event = BeforeModelEvent(run_id="test", step_id=0)
|
||||||
|
return [out async for out in hosted.on_before_model(event)]
|
||||||
|
|
||||||
|
emitted = asyncio.run(drive())
|
||||||
|
assert emitted and emitted[-1].skip_model is True
|
||||||
Reference in New Issue
Block a user