feat(learning-harness): publish every number from the artifacts that produced it

Task 3.8. A number in a roadmap that nobody can regenerate is a claim, not a
measurement. One entry point now writes them all from the committed artifacts
and puts the denominator beside each: how many pairs could be scored, why the
rest could not, which taxonomy version scored them, the noise the family shows
against itself, and what the run cost per pair it got right. Every table says
whether hierarchical partial credit could apply, which today it cannot.

The roadmap section is replaced in place between markers rather than appended,
so the file cannot accumulate a stale number beside a fresh one, and running the
command twice without new artifacts changes nothing. With no baseline recorded
it says so instead of printing an empty table.

The corrected mechanism-lever number travels with the rest: detected five of
seventeen at a negative Youden once only train-split pairs teach, against seven
when every pair taught, with the difference named as what holdout pairs stopped
teaching. Round zero now records what it spent so the cost column has a source.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-06 11:59:13 +02:00
co-authored by Claude Opus 5
parent b9552518b9
commit b8ad42c57e
4 changed files with 363 additions and 0 deletions
+4
View File
@@ -69,6 +69,7 @@ class BaselineReport:
floors: dict[str, NoiseFloor] = field(default_factory=dict)
metrics: dict[str, dict[str, object]] = field(default_factory=dict)
scores: tuple[PairFamilyScore, ...] = ()
cost_usd: float = 0.0
def to_dict(self) -> dict[str, object]:
return {
@@ -76,6 +77,7 @@ class BaselineReport:
"k_runs": self.k_runs,
"taxonomy_version": self.taxonomy_version,
"slices": list(self.slices),
"cost_usd": self.cost_usd,
"floors": {name: floor.to_dict() for name, floor in sorted(self.floors.items())},
"metrics": {name: dict(block) for name, block in sorted(self.metrics.items())},
}
@@ -92,6 +94,7 @@ def run_baseline(
slices: Sequence[str] = DEFAULT_SLICES,
k_runs: int = DEFAULT_K_RUNS,
prompt: str | None = None,
spent_usd: Callable[[], float] | None = None,
) -> BaselineReport:
"""Clone one detector into every family, measure each family against itself, and write the artifacts."""
from ..tool_hunter import _SYSTEM_PROMPT
@@ -110,6 +113,7 @@ def run_baseline(
floors=floors,
metrics={name: block.to_dict() for name, block in metrics.items()},
scores=tuple(scores),
cost_usd=spent_usd() if spent_usd is not None else 0.0,
)
_write(out_dir, report)
return report