Decision-engine task 6.7 (partial). After the advisory-fix harvest (memory 724 -> 904 static examples),
one compile per evaluable family and an outer grouped 5-fold evaluation of that family's own candidates,
input profile v1, priors off, k = 5, cross-fitted Platt, nested operating points (repository-cluster
bootstrap 95%):
family n AUC(s) ADVISORY R / P 'vulnerable' P / R ECE canary
injection 110 0.77 [0.67, 0.88] 0.88 / 0.57 0.84 / 0.36 0.093 0.77
untrusted_dest. 62 0.85 [0.76, 0.94] 0.84 / 0.73 0.96 / 0.66 0.071 0.87
path 35 0.89 [0.79, 0.97] 0.86 / 0.69 0.89 / 0.76 0.119 0.90
output_encoding 44 0.91 [0.82, 0.97] 0.86 / 0.81 0.94 / 0.55 0.045 0.90
deserialization 46 0.90 [0.82, 0.97] 0.83 / 0.76 0.93 / 0.83 0.095 0.93
access_control 37 0.96 [0.90, 1.00] 0.91 / 0.91 1.00 / 0.61 0.076 0.83
BLOCK offered for none: calibration fails everywhere except output_encoding, where no fold reaches a
Wilson lower bound of 0.95. Within paired groups (a function and its own fix) AUC is lower, 0.72-0.93:
every single-label group in the new families holds positives only. Injection learning-curve point,
paired (same program, candidates, folds): memory 724 -> 904, AUC 0.76 -> 0.77 (no injection example
was added). $2.88 metered DeepSeek (ceiling $8; balance 17.73 -> 16.73), $0.0009 embeddings.
Instrument fix: compiled demonstrations now follow the near-duplicate rule (cos >= 0.98) that retrieval
and the pre-call assertion apply; the assertion had stopped the untrusted_destination evaluation.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
6.2: the 724 static examples' 616 distinct excerpts embedded with openai/text-embedding-3-small via
OpenRouter, 198,819 tokens, $0.0040 metered (ceiling $1).
6.6, family injection only, memory = every family (at most 2 contrast examples per prompt), input
profile v1: smoke fixed k = 5 (repeat prefix-cache hit share 0.96); one compile (program c1237a49,
proposed instruction 5, balanced Brier 0.128 on 13 C_val candidates vs 0.199 baseline); outer grouped
5-fold, 110 out-of-repository candidates in 29 groups: ADVISORY recall 0.90 [0.82, 0.97], precision
0.54 [0.44, 0.65], AUC 0.74; BLOCK not offered (ECE 0.064 > 0.05; no fold reached a Wilson lower bound
of 0.95); canary re-ask agreement 0.83. $0.91 metered at list price; DeepSeek balance $18.20 -> $17.90.
Instrument fixes found on the way, each with a test:
- memory build: pair sides join the record stored under their harvest unit pin (--units), and only the
built profile's records are read (static and plane shared a key; 85 -> 724 examples);
- retrieval: stage 1 per side (family / contrast) and at most 2 contrast examples, never padded;
- program: the JSON answer contract is part of the signature, appended to proposed instructions
(DeepSeek JSON mode refused a prompt without "json");
- compile: candidate_families for a one-family compile; compile.toml k = 5 from the smoke rule.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer decision "Neutralise both for v1".
1. Phantom candidates. A pair side's candidates are now only the functions that
side declares (plane/harvest.py `declares`, the excerpt builder's
function_bounds); a labelled function absent from a side is listed under
`absent_on_side` and never asked, and a side declaring none of its verify
family's functions is no verify unit. The recorded harvest units held 58
phantom verify candidates (53 fixed, 5 vulnerable sides). build_features.py
skips label rows whose side does not declare the function and drops their
earlier rows (MemoryStore.drop_rows); rebuilt the pair records from the
recorded signals, no model call: 126 label rows absent_on_side (117 fixed,
9 vulnerable) -- access_control 13, config_secrets 4, deserialization 20,
injection 28, memory 4, output_encoding 16, path 15, untrusted_destination
26 -- so 252 feature rows dropped (1712 -> 1460); no other row changed.
2. Engine withheld. Input profiles in learn/schema.py (`v1` withholds the
engine instrument, `full` withholds nothing); ProgramSpec.inputs defaults to
v1 and compile.toml records it. Neither the engine's state nor its values
reach the candidate, examples, demonstrations or the instruction proposer's
error summary, and the Gower retrieval distance leaves the engine block out.
Stored records are unchanged. design.md Risks records the decision and that
the engine returns only through a pre-registered A/B on assumed-benign and
benign-control negatives.
3. `ousast learn audit-leaks` (learn/leaks.py): per feature and per instrument
state, separation = (pairs the signal alone puts on the right side - on the
wrong side) / pairs, over all pairs and per family; flagged at >= 0.20 over
>= 10 pairs. Counts under
benchmarks/measurements/2026-10-01-decision-engine-leak-audit/ (before and
after the rebuild). After: facts.function_lines (0.80 untrusted_destination,
0.75 config_secrets, 0.26 injection; higher on fixed), verify.turns_mean
0.67 config_secrets, eng.* 0.35-0.40 output_encoding, verify.flag_a/votes/
site_offset and agree.final 0.20-0.30 on 10-12 pair families. Before the
rebuild the phantom fixed sides drove state.source/state.roles/ms.flagged
to 1.0 and state.engine to 0.95 on deserialization.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
learn/retrieve.py: stage 1 is a per-instrument Gower distance over the
profile's allow-listed features with explicit missing (both ran: mean
feature distance; differing states 1; same non-ran state 0; null in one 1;
priors dropped with priors=off), n1 = 40 with seeded tie-breaks; stage 2
re-ranks by r = lam(1 - D) + (1 - lam)cos, signals_only without the
target's vector; the balance rule takes k_ret = 6 (<= k/2 per label, <= 2
per group, <= 2 other families; `short` recorded). eligible() is the only
way into a prompt: not the target's group, not an evaluated group, not the
held-out source or framework, and outside deployment not a near-duplicate
(cos >= 0.98). assert_boundary repeats the check on rendered ids;
demonstrations of a held-out framework swap to their alternates;
deployment_fold excludes a scanned repository that is a corpus group;
seed_memory never hands a case its own group.
learn/folds.py: compile split (25% of groups per primary family, 60/40
boot/val), outer grouped 5-fold stratified by family and label mix,
leave-one-source-out, leave-one-framework-out (insufficient data below 10
groups), assert_disjoint and a folds digest; all seeded.
Tests: test_learn_retrieve.py (hand-computed Gower, lam 0/0.5, balance,
near-duplicates, test_boundary over outer/LOSO/LOFO folds with
demonstrations), test_learn_folds.py.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
