marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

bench(learn): injection first slice -- compiled AI classifier, out-of-repo evaluation (decision-engine 6.2, 6.6)

Browse Source 
6.2: the 724 static examples' 616 distinct excerpts embedded with openai/text-embedding-3-small via
OpenRouter, 198,819 tokens, $0.0040 metered (ceiling $1).

6.6, family injection only, memory = every family (at most 2 contrast examples per prompt), input
profile v1: smoke fixed k = 5 (repeat prefix-cache hit share 0.96); one compile (program c1237a49,
proposed instruction 5, balanced Brier 0.128 on 13 C_val candidates vs 0.199 baseline); outer grouped
5-fold, 110 out-of-repository candidates in 29 groups: ADVISORY recall 0.90 [0.82, 0.97], precision
0.54 [0.44, 0.65], AUC 0.74; BLOCK not offered (ECE 0.064 > 0.05; no fold reached a Wilson lower bound
of 0.95); canary re-ask agreement 0.83. $0.91 metered at list price; DeepSeek balance $18.20 -> $17.90.

Instrument fixes found on the way, each with a test:
- memory build: pair sides join the record stored under their harvest unit pin (--units), and only the
  built profile's records are read (static and plane shared a key; 85 -> 724 examples);
- retrieval: stage 1 per side (family / contrast) and at most 2 contrast examples, never padded;
- program: the JSON answer contract is part of the signature, appended to proposed instructions
  (DeepSeek JSON mode refused a prompt without "json");
- compile: candidate_families for a one-family compile; compile.toml k = 5 from the smoke rule.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-10-01 20:40:41 +02:00
co-authored by Claude Opus 5.5
parent 68294e9db3
commit 7b17eacd44
13 changed files with 692 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/test_learn_examples.py
+16
View File
@@ -18,6 +18,7 @@ from openultrasast.learn.examples import (
build_examples,
label_pin,
load_examples,
pair_pins,
record_key,
records_by_key,
)
@@ -155,3 +156,18 @@ def test_a_fixed_side_without_the_function_is_a_label_defect_not_an_unread_side(
assert count.absent_fixed_side == 1 and count.no_excerpt == 0 and build.failing() == []
unread = build_examples([fixed], records_by_key([feature(fixed)]), lambda lab, role: None, sources=SOURCES)
assert unread.sources["population-v1"].unread == 1 and unread.failing() == ["population-v1"]
def test_pair_sides_join_the_record_stored_under_their_harvest_unit_pin() -> None:
"""The harvest stores a pair side's record under its unit pin (the blob sha1 of the side's excerpt), not under
:func:`label_pin`; ``pins`` from the harvest ``units.json`` joins them, and without it the record is not found."""
lab = label(source="pairs", source_ref="case-9", pin="")
unit_pin = "a" * 40
stored = {**feature(lab), "pin": unit_pin}
units = {"h-1": {"source": "pairs", "ref": "case-9", "side": "vulnerable", "pin": unit_pin, "repo": lab["repo"]},
"h-2": {"source": "population-v1", "ref": "case-9", "side": "vulnerable", "pin": PIN, "repo": lab["repo"]}} # fmt: skip
pins = pair_pins(units)
assert pins == {("case-9", "vulnerable"): unit_pin}
assert build_examples([lab], records_by_key([stored]), lines_of, sources=SOURCES).summary()["examples"] == 0
build = build_examples([lab], records_by_key([stored]), lines_of, sources=SOURCES, pins=pins)
assert build.summary()["examples"] == 1 and build.rows[0]["pin"] == unit_pin