marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-19 13:10:03 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(learning-harness): verifier registry and the sandbox oracle callback

Browse Source 
Task 2.6. A detector's claim is a suspicion until something other than the model
agrees with it, and what counts as agreement depends on the family. The regress
verdict now takes an optional oracle over the sandbox result: when a family
supplies one it decides triggerability instead of the exit code, because a crash
and a demonstrated injection are different evidence and one family's oracle must
not answer for another. Without an oracle the exit code decides exactly as
before, and an inconclusive run stays inconclusive either way.

The registry answers by the verifier kind the taxonomy declares. Access control
gets static corroboration when the obligations checker looks at the same
function and agrees the obligation is undischarged, and refuses to corroborate a
handler that owes nothing. A family whose verifier does not exist yet reports at
suspicion rather than pretending, which is the honest default rather than a gap.
A verification is written onto a copy of the claim as a tag, never onto the
original.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-06 10:04:07 +02:00
co-authored by Claude Opus 5
parent 5628052b51
commit 9801d5407b
3 changed files with 242 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/openultrasast/regress/verdict.py
+16 -1
View File
@@ -2,6 +2,7 @@
from __future__ import annotations
from collections.abc import Callable
from dataclasses import dataclass
from ..sandbox import SandboxResult
@@ -21,6 +22,8 @@ SANITIZER_ABORT = "sanitizer_abort"
CRASH = "crash"
NONZERO_EXIT = "nonzero_exit"
EXIT_ZERO = "exit_zero"
ORACLE_FIRED = "oracle_fired"
ORACLE_QUIET = "oracle_quiet"
WORTH_FIXING_REACHABILITY = frozenset({"reachable", "inferred-file-surface"})
@@ -65,8 +68,14 @@ def verdict_from_result(
sandbox_missing: bool = False,
safety_rejected: bool = False,
covering_test: bool = False,
oracle: Callable[[SandboxResult], bool] | None = None,
) -> RegressionVerdict:
"""Map a sandbox observation onto one of the four regression verdicts."""
"""Map a sandbox observation onto one of the four regression verdicts.
``oracle`` is how a family says what "the vulnerability fired" means for it: when one is given it
decides triggerability instead of the exit code, because a crash and a demonstrated injection are
not the same evidence. An inconclusive run stays inconclusive either way.
"""
if recipe_missing:
return RegressionVerdict(verdict=INCONCLUSIVE, reason=MISSING_RECIPE)
if sandbox_missing:
@@ -79,6 +88,12 @@ def verdict_from_result(
return RegressionVerdict(verdict=INCONCLUSIVE, reason=TIMEOUT)
if covering_test and result.exit_code == 0:
return RegressionVerdict(verdict=ALREADY_COVERED, reason=COVERED_TEST_PASSED)
if oracle is not None:
return (
RegressionVerdict(verdict=TRIGGERABLE, reason=ORACLE_FIRED)
if oracle(result)
else RegressionVerdict(verdict=NOT_TRIGGERABLE, reason=ORACLE_QUIET)
)
if result.exit_code != 0:
return RegressionVerdict(verdict=TRIGGERABLE, reason=_nonzero_reason(result))
return RegressionVerdict(verdict=NOT_TRIGGERABLE, reason=EXIT_ZERO)