mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
bench(learn): injection first slice -- compiled AI classifier, out-of-repo evaluation (decision-engine 6.2, 6.6)
6.2: the 724 static examples' 616 distinct excerpts embedded with openai/text-embedding-3-small via OpenRouter, 198,819 tokens, $0.0040 metered (ceiling $1). 6.6, family injection only, memory = every family (at most 2 contrast examples per prompt), input profile v1: smoke fixed k = 5 (repeat prefix-cache hit share 0.96); one compile (program c1237a49, proposed instruction 5, balanced Brier 0.128 on 13 C_val candidates vs 0.199 baseline); outer grouped 5-fold, 110 out-of-repository candidates in 29 groups: ADVISORY recall 0.90 [0.82, 0.97], precision 0.54 [0.44, 0.65], AUC 0.74; BLOCK not offered (ECE 0.064 > 0.05; no fold reached a Wilson lower bound of 0.95); canary re-ask agreement 0.83. $0.91 metered at list price; DeepSeek balance $18.20 -> $17.90. Instrument fixes found on the way, each with a test: - memory build: pair sides join the record stored under their harvest unit pin (--units), and only the built profile's records are read (static and plane shared a key; 85 -> 724 examples); - retrieval: stage 1 per side (family / contrast) and at most 2 contrast examples, never padded; - program: the JSON answer contract is part of the signature, appended to proposed instructions (DeepSeek JSON mode refused a prompt without "json"); - compile: candidate_families for a one-family compile; compile.toml k = 5 from the smoke rule. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
68294e9db3
commit
7b17eacd44
@@ -15,6 +15,8 @@ from learn_fixtures import ScriptedChat, by_id, corpus, x_record
|
||||
from openultrasast.learn import program as program_module
|
||||
from openultrasast.learn.folds import outer_folds
|
||||
from openultrasast.learn.program import (
|
||||
ANSWER_FORMAT,
|
||||
BASELINE_INSTRUCTION,
|
||||
PARSE_FAILED,
|
||||
Answer,
|
||||
Caller,
|
||||
@@ -121,6 +123,15 @@ def test_the_prefix_is_byte_identical_across_candidates() -> None:
|
||||
assert "Demonstration 1:" in a.prefix and a.messages[1] != b.messages[1]
|
||||
|
||||
|
||||
def test_every_instruction_carries_the_answer_contract() -> None:
|
||||
"""A proposed instruction rarely restates the JSON contract; the prefix adds it, once, so JSON mode is accepted
|
||||
(DeepSeek refuses a prompt without the word "json") and the answer parses. The baseline prefix is unchanged."""
|
||||
proposed, _ = program(instruction="Weigh the sanitiser before the sink.").prepare(candidate(), FOLD)
|
||||
baseline, _ = program().prepare(candidate(), FOLD)
|
||||
assert proposed.prefix.startswith("Weigh the sanitiser before the sink.\n") and proposed.prefix.count(ANSWER_FORMAT) == 1
|
||||
assert baseline.prefix.count(ANSWER_FORMAT) == 1 and baseline.prefix.startswith(BASELINE_INSTRUCTION)
|
||||
|
||||
|
||||
def test_parsing_valid_invalid_and_out_of_range_citations() -> None:
|
||||
lines = frozenset({40, 41})
|
||||
ok = parse_answer(
|
||||
|
||||
Reference in New Issue
Block a user