mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-18 04:29:27 +00:00
fix(learning-harness): the three defects the group 2 review found
Task 2.3: two of the five unscorable reasons had no code path, so an unparsable row was reported as a corpus labelling defect when the true cause was a tooling gap. They have different owners and different fixes, and conflating them is the dishonesty this spec exists to remove. The design's unscorable_reason is reinstated with the parse result as an input, and a handler that arrived without its registration is now named as missing context for the families whose bug shape is a missing check. Task 2.6: a verification stamped its own word for proof onto the shared evidence level, which is not a value the project's ladder defines, and three existing helpers raised on it. Nothing in production called it yet, so the suite stayed green while the wiring in the next group would have broken a scan. A verification now writes only its tag, and a caller that wants to move a claim up the ladder goes through an explicit mapping that can only produce a level the ladder has. Task 2.7: the injection canary planted its row in a private in-memory database the target could never reach, so it could not fire; the fake sandbox handed the token back regardless of the snippet, which is why nothing noticed. It is now an execution canary whose marker only an interpreter could write, and each template is tested by running it for real against a vulnerable fixture and its fixed twin. That test immediately caught a second flaw: a handler that merely echoes its input printed the oracle token and satisfied the oracle without any vulnerability, so an injected payload now carries a value unrelated to the token the oracle looks for. Also from the review: a configuration with no tools is refused instead of producing a detector that silently reports nothing, the classifier abstains when its fallback call fails too, and the archive docstring says which version wins. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
co-authored by
Claude Opus 5
parent
e44d45f488
commit
b0dba26468
@@ -19,9 +19,11 @@ from typing import Literal, Protocol
|
||||
|
||||
from ..findings import StaticFinding
|
||||
from ..sandbox import SandboxResult
|
||||
from ..verification import EvidenceLevel
|
||||
from .families import Family
|
||||
|
||||
Rung = Literal["suspicion", "static_corroboration", "proven"]
|
||||
RUNGS: tuple[Rung, ...] = ("suspicion", "static_corroboration", "proven")
|
||||
VerifierKind = Literal["canary", "static", "none"]
|
||||
|
||||
|
||||
@@ -93,12 +95,37 @@ def verifier_for(family: Family, *, canary: Verifier | None = None) -> Verifier:
|
||||
|
||||
|
||||
def apply_verification(claim: StaticFinding, verification: Verification) -> StaticFinding:
|
||||
"""A copy of the claim carrying what the verifier said. The original is never mutated."""
|
||||
"""A copy of the claim carrying what the verifier said, as a tag.
|
||||
|
||||
The evidence ladder belongs to another boundary and this spec does not change it, so nothing here
|
||||
writes ``evidence_level``. "proven" is this spec's word, not a rung the ladder defines, and stamping
|
||||
it onto the field made the project's own verification helpers raise. A caller that wants to move a
|
||||
claim up the ladder does it through :func:`raise_to`, which can only produce a level the ladder has.
|
||||
"""
|
||||
tags = [tag for tag in claim.tags if not tag.startswith("verifier:")] + [f"verifier:{verification.rung}"]
|
||||
return replace(claim, tags=tags, evidence_level=verification.rung)
|
||||
return replace(claim, tags=tags)
|
||||
|
||||
|
||||
def evidence_level_for(rung: Rung) -> EvidenceLevel:
|
||||
"""The project's evidence level a verifier rung corresponds to. The only bridge between the two vocabularies."""
|
||||
return _LADDER[rung]
|
||||
|
||||
|
||||
def raise_to(claim: StaticFinding, rung: Rung) -> StaticFinding:
|
||||
"""A copy of the claim at the ladder level this rung corresponds to. Never a value the ladder lacks."""
|
||||
return replace(claim, evidence_level=str(evidence_level_for(rung)))
|
||||
|
||||
|
||||
_LADDER: dict[Rung, EvidenceLevel] = {
|
||||
"suspicion": EvidenceLevel.SUSPICION,
|
||||
"static_corroboration": EvidenceLevel.STATIC_CORROBORATION,
|
||||
# A canary that fired in the sandbox demonstrated the vulnerability, which is what this level means here.
|
||||
"proven": EvidenceLevel.EXPLOIT_DEMONSTRATED,
|
||||
}
|
||||
|
||||
|
||||
__all__ = [
|
||||
"RUNGS",
|
||||
"NoVerifier",
|
||||
"Rung",
|
||||
"StaticVerifier",
|
||||
@@ -106,6 +133,8 @@ __all__ = [
|
||||
"Verifier",
|
||||
"VerifierKind",
|
||||
"apply_verification",
|
||||
"evidence_level_for",
|
||||
"raise_to",
|
||||
"token_oracle",
|
||||
"verifier_for",
|
||||
]
|
||||
|
||||
Reference in New Issue
Block a user