mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-18 12:39:28 +00:00
fix(learning-harness): the round-3 review findings, remediated RED-first
Eight of nine group-3 tasks were rejected. Each defect was hiding a number, and the numbers are why the loop was not learning: - The fixed side was judged against the *vulnerable* side's function ranges. 191 of 227 pairs with a resolvable labeled span (84%) have different spans on the two sides, so a leak was recorded as silence, always in the direction that flatters Youden. - Round zero measured every family with the generalist. `family:unknown` earns no credit against any real family, so every family but `unknown` baselined at a structural zero and every noise floor with it. - The cross-family sweep counted absolute failures instead of negative flips. On a corpus where most pairs fail at round zero that rejects every round for damage it did not do. Unscorable rows voted too. - K runs was declared and never threaded; `pairs --k-runs` was parsed and discarded, and its only test grepped `--help`. - The meta-agent proposer was a stub, so `learning round` could never propose anything. It now composes MetaAgent.evolve against a scratch copy of the family directory, and `--proposals FILE` runs a recorded sequence offline. - `refused_holdout` was a vocabulary word no code path could produce, and the covering test accepted either outcome. - Cost was metered once, after everything had been paid for; a stage that raised left no record at all. - The corrected lever number was published beside the earlier one rather than in place of it, and the test named for that property never checked. - Refusals were `continue` statements: `Refusal.degradation()` existed and nothing called it. A train-split identical twin was reported as `split:train`, which is untrue. Test repairs that each changed a number: two split tests measured an `x = 1` fixture that teaches nothing whatever the rule says; two CLI scoreboard tests asserted against all-zero output produced by a stub that answers twice and puts its finding outside the labeled function. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
co-authored by
Claude Opus 5
parent
c2d9c4ad01
commit
1843ef9a02
@@ -175,8 +175,9 @@ class ChatClient(Protocol):
|
||||
class ScriptedChatClient:
|
||||
"""Deterministic ChatClient for tests and OPENULTRASAST_HUNTER_CLIENT injection."""
|
||||
|
||||
def __init__(self, turns: list[ChatResponse]) -> None:
|
||||
def __init__(self, turns: list[ChatResponse], *, cycle: bool = False) -> None:
|
||||
self._turns = turns
|
||||
self._cycle = cycle # a stub that runs out of answers turns every multi-run test into a silent zero
|
||||
self.calls: list[dict[str, object]] = []
|
||||
|
||||
def complete(
|
||||
@@ -192,7 +193,9 @@ class ScriptedChatClient:
|
||||
self.calls.append({"model": model, "messages": list(messages), "tools": tools, "timeout_seconds": timeout_seconds, **options})
|
||||
index = len(self.calls) - 1
|
||||
if index >= len(self._turns):
|
||||
return ChatResponse(content="")
|
||||
if not self._cycle or not self._turns:
|
||||
return ChatResponse(content="")
|
||||
index %= len(self._turns)
|
||||
return self._turns[index]
|
||||
|
||||
|
||||
@@ -527,6 +530,10 @@ def _finding_from_item(
|
||||
severity = item.get("severity")
|
||||
snippet = item.get("snippet")
|
||||
proposed_snippet = snippet if isinstance(snippet, str) and snippet else None
|
||||
family = item.get("family")
|
||||
# The family is the only key the class-aware scorer credits, and it is the model's own answer: a fabricated
|
||||
# name is counted as noise there rather than silently dropped here.
|
||||
family_tags = [f"family:{family}"] if isinstance(family, str) and family.strip() else []
|
||||
return StaticFinding(
|
||||
finding_id=f"{_FINDING_ID_PREFIX}{relative}:{line_no if line_no is not None else 0}",
|
||||
path=relative,
|
||||
@@ -540,7 +547,7 @@ def _finding_from_item(
|
||||
reachability_status="unknown",
|
||||
reachability_evidence=[],
|
||||
reachability_conditions=[],
|
||||
tags=list(tags),
|
||||
tags=[*tags, *family_tags],
|
||||
ranking_priority=float(scores.get(relative, 0.0)),
|
||||
proposed_snippet=proposed_snippet,
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user