bench(learn): injection first slice -- compiled AI classifier, out-of-repo evaluation (decision-engine 6.2, 6.6)

6.2: the 724 static examples' 616 distinct excerpts embedded with openai/text-embedding-3-small via
OpenRouter, 198,819 tokens, $0.0040 metered (ceiling $1).

6.6, family injection only, memory = every family (at most 2 contrast examples per prompt), input
profile v1: smoke fixed k = 5 (repeat prefix-cache hit share 0.96); one compile (program c1237a49,
proposed instruction 5, balanced Brier 0.128 on 13 C_val candidates vs 0.199 baseline); outer grouped
5-fold, 110 out-of-repository candidates in 29 groups: ADVISORY recall 0.90 [0.82, 0.97], precision
0.54 [0.44, 0.65], AUC 0.74; BLOCK not offered (ECE 0.064 > 0.05; no fold reached a Wilson lower bound
of 0.95); canary re-ask agreement 0.83. $0.91 metered at list price; DeepSeek balance $18.20 -> $17.90.

Instrument fixes found on the way, each with a test:
- memory build: pair sides join the record stored under their harvest unit pin (--units), and only the
  built profile's records are read (static and plane shared a key; 85 -> 724 examples);
- retrieval: stage 1 per side (family / contrast) and at most 2 contrast examples, never padded;
- program: the JSON answer contract is part of the signature, appended to proposed instructions
  (DeepSeek JSON mode refused a prompt without "json");
- compile: candidate_families for a one-family compile; compile.toml k = 5 from the smoke rule.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-10-01 20:40:41 +02:00
co-authored by Claude Opus 5.5
parent 68294e9db3
commit 7b17eacd44
13 changed files with 692 additions and 46 deletions
+11
View File
@@ -15,6 +15,8 @@ from learn_fixtures import ScriptedChat, by_id, corpus, x_record
from openultrasast.learn import program as program_module
from openultrasast.learn.folds import outer_folds
from openultrasast.learn.program import (
ANSWER_FORMAT,
BASELINE_INSTRUCTION,
PARSE_FAILED,
Answer,
Caller,
@@ -121,6 +123,15 @@ def test_the_prefix_is_byte_identical_across_candidates() -> None:
assert "Demonstration 1:" in a.prefix and a.messages[1] != b.messages[1]
def test_every_instruction_carries_the_answer_contract() -> None:
"""A proposed instruction rarely restates the JSON contract; the prefix adds it, once, so JSON mode is accepted
(DeepSeek refuses a prompt without the word "json") and the answer parses. The baseline prefix is unchanged."""
proposed, _ = program(instruction="Weigh the sanitiser before the sink.").prepare(candidate(), FOLD)
baseline, _ = program().prepare(candidate(), FOLD)
assert proposed.prefix.startswith("Weigh the sanitiser before the sink.\n") and proposed.prefix.count(ANSWER_FORMAT) == 1
assert baseline.prefix.count(ANSWER_FORMAT) == 1 and baseline.prefix.startswith(BASELINE_INSTRUCTION)
def test_parsing_valid_invalid_and_out_of_range_citations() -> None:
lines = frozenset({40, 41})
ok = parse_answer(