feat(learning-harness): canary snippets and the second judge

Task 2.7. A canary is how a family says the vulnerability fired. Injection
plants a row in an in-memory database that only an unconstrained query returns,
file path plants a file under the scratch mount that only a traversal reaches,
and deserialization plants an object that only an unsafe load reconstructs. Each
prints its token on that path and no other, and the token is derived from the
claim, so one family's oracle can never answer for another and one claim's
canary can never answer for the next. A snippet that cannot reach the function
prints nothing, which is a suspicion rather than a proof.

Every template passes the existing snippet safety check before it can run, and
the candidate is always imported from the read-only source mount.

Nothing reaches proven on a machine's say-so alone. A second, independent
judgement reads the snippet and what the sandbox printed and either confirms or
drops the claim back to corroboration; a judge that cannot answer never promotes
anything. The judge is told the claim and the output, never the answer key.
Templates and oracle code live in the package, outside every directory a round
may write.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-06 10:06:57 +02:00
co-authored by Claude Opus 5
parent 9801d5407b
commit 0c2b9ec685
3 changed files with 340 additions and 8 deletions
+7 -8
View File
@@ -3,7 +3,7 @@
from __future__ import annotations
import json
from collections.abc import Mapping, Sequence
from collections.abc import Callable, Mapping, Sequence
from dataclasses import asdict, dataclass, replace
from pathlib import Path
@@ -131,24 +131,23 @@ class RegressionRunner:
sandbox_limits: SandboxConfig,
*,
repo_root: Path,
oracle: Callable[[SandboxResult], bool] | None = None,
) -> RegressionVerdict:
job = recipe_for(language, snippet, image, sandbox_limits, repo_root=repo_root)
if job is None:
return verdict_from_result(recipe_missing=True)
return self.run_snippet(snippet, job)
return self.run_snippet(snippet, job, oracle=oracle)
def run_snippet(self, snippet: str, job: SandboxJob | None) -> RegressionVerdict:
def run_snippet(
self, snippet: str, job: SandboxJob | None, *, oracle: Callable[[SandboxResult], bool] | None = None
) -> RegressionVerdict:
if job is None:
return verdict_from_result(recipe_missing=True)
try:
check_snippet_safety(snippet)
except UnsafeSnippetError:
return verdict_from_result(safety_rejected=True)
return _verdict_from_result(self._sandbox.run(job))
def _verdict_from_result(result: SandboxResult) -> RegressionVerdict:
return verdict_from_result(result)
return verdict_from_result(self._sandbox.run(job), oracle=oracle)
def snippet_for(language: str, path: str, function_name: str | None) -> str: