From 7b17eacd44fa22ba56d96da3af1491cef2331987 Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Thu, 1 Oct 2026 20:40:41 +0200 Subject: [PATCH] bench(learn): injection first slice -- compiled AI classifier, out-of-repo evaluation (decision-engine 6.2, 6.6) 6.2: the 724 static examples' 616 distinct excerpts embedded with openai/text-embedding-3-small via OpenRouter, 198,819 tokens, $0.0040 metered (ceiling $1). 6.6, family injection only, memory = every family (at most 2 contrast examples per prompt), input profile v1: smoke fixed k = 5 (repeat prefix-cache hit share 0.96); one compile (program c1237a49, proposed instruction 5, balanced Brier 0.128 on 13 C_val candidates vs 0.199 baseline); outer grouped 5-fold, 110 out-of-repository candidates in 29 groups: ADVISORY recall 0.90 [0.82, 0.97], precision 0.54 [0.44, 0.65], AUC 0.74; BLOCK not offered (ECE 0.064 > 0.05; no fold reached a Wilson lower bound of 0.95); canary re-ask agreement 0.83. $0.91 metered at list price; DeepSeek balance $18.20 -> $17.90. Instrument fixes found on the way, each with a test: - memory build: pair sides join the record stored under their harvest unit pin (--units), and only the built profile's records are read (static and plane shared a key; 85 -> 724 examples); - retrieval: stage 1 per side (family / contrast) and at most 2 contrast examples, never padded; - program: the JSON answer contract is part of the signature, appended to proposed instructions (DeepSeek JSON mode refused a prompt without "json"); - compile: candidate_families for a one-family compile; compile.toml k = 5 from the smoke rule. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo --- .kiro/specs/learned-decision-engine/design.md | 24 +- .kiro/specs/learned-decision-engine/tasks.md | 12 +- .../record.json | 524 ++++++++++++++++++ src/openultrasast/learn/cli.py | 9 +- src/openultrasast/learn/compile.py | 9 +- src/openultrasast/learn/compile.toml | 4 +- src/openultrasast/learn/examples.py | 19 +- src/openultrasast/learn/program.py | 21 +- src/openultrasast/learn/retrieve.py | 45 +- tests/test_learn_compile.py | 23 +- tests/test_learn_examples.py | 16 + tests/test_learn_program.py | 11 + tests/test_learn_retrieve.py | 21 +- 13 files changed, 692 insertions(+), 46 deletions(-) create mode 100644 benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json diff --git a/.kiro/specs/learned-decision-engine/design.md b/.kiro/specs/learned-decision-engine/design.md index 994223e..1753d11 100644 --- a/.kiro/specs/learned-decision-engine/design.md +++ b/.kiro/specs/learned-decision-engine/design.md @@ -529,7 +529,8 @@ explicit rather than imputed: block). Prior features (`FeatureSpec.prior`) are dropped when `priors = off`. Weights are a compile-time setting only if an A/B shows a gain; the default is uniform. - Candidates in the eligible pool (after the boundary filter) are ranked by `D`, ties by `sha256(example_id + - seed)`; the first `n1 = 40` go to stage 2. + seed)`; the first `n1 = 40` of the candidate's family and the first `n1` of the other families go to stage 2 + (per side since 2026-10-01, so a minority family reaches stage 2; Risks, "First paid slice"). **Stage 2: code-embedding re-rank.** Cosine similarity of the candidate's excerpt embedding with each stage-1 example's. The re-rank score is `r = lam * (1 - D) + (1 - lam) * cos`, with `lam` a compiled setting in {0, 0.5} @@ -1205,6 +1206,27 @@ start only after the maintainer confirms their budget against the balance read j Pre-registering that and recording "inconclusive" is the intended behaviour. The failure mode to avoid is extending an experiment after looking. +- **First paid slice, 2026-10-01 (tasks 6.2, 6.6; injection only).** Four instrument defects surfaced before any + number was believed, each fixed with a test: + - *The memory build joined no pair.* The harvest stores a pair side's record under its unit pin (the blob sha1 of + the side's excerpt), not under `label_pin`; `ousast learn memory build --units ` joins them. + And a candidate's `static` and `plane` records share a key, so reading both let one overwrite the other and + silently drop about half the examples; the build now reads its own profile only. 85 -> 724 examples. + - *Contrast was unbounded.* The balance rule padded a short pick with other families (up to 5 of 6), and stage 1 + took the 40 nearest of all families, so a minority family often never reached stage 2. Now stage 1 takes the + `n1` nearest of the candidate's family and the `n1` nearest others, and at most 2 contrast examples are shown + whenever the pool holds the family (maintainer: "allow 2 contrast"); never padded. + - *Proposed instructions dropped the answer contract*, and DeepSeek's JSON mode refuses a prompt without the word + "json" (HTTP 400, the first compile stopped there). The contract (`ANSWER_FORMAT`) is now part of the signature + and appended to any instruction that lacks it; the baseline prefix is byte-identical. + - The adjudication source fails the 10% excerpt gate: 9 of its 43 candidates are anonymous functions (`N`, + ``) that `function_span` cannot find by name; the clones were read. Open. + The memory is small: 724 static examples (only harvested candidates have feature records; the 97 advisory-fix pairs + have none yet), injection 140 (75/65), so `C_val` had 13 injection candidates and the compile metric is noisy. + The measured ranking signal is real but weak at the recall target: AUC 0.74 out of repository, ADVISORY at recall + 0.90 has precision 0.54 (prevalence 0.53); the majority `vulnerable` verdict alone has precision 0.86, recall 0.43. + Canary re-ask agreement at temperature 0 was 0.83, below the 0.9 the adoption check requires. + ## Maintainer decisions at design approval (2026-09-30) - Design approved as written. diff --git a/.kiro/specs/learned-decision-engine/tasks.md b/.kiro/specs/learned-decision-engine/tasks.md index adb19b7..e268a3b 100644 --- a/.kiro/specs/learned-decision-engine/tasks.md +++ b/.kiro/specs/learned-decision-engine/tasks.md @@ -54,6 +54,8 @@ a model or embedding call. `plane/budget.py`, usage kept by `parse_embedding_response`; cache `embeddings//.json`; `ousast learn memory embed`; scripted-client tests. - **(budget go-ahead)** embedding the store: ~5.0 M tokens, ~$0.10, ceiling $1 (OpenRouter account). + Done 2026-10-01: 616 distinct excerpts of the 724 static examples, 198,819 tokens, 20 calls, $0.0040 metered + (`benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`). - _Requirements: 3.2_ - [x] 6.3 Retrieval and the evaluation boundary (no model) - `learn/retrieve.py`: per-instrument Gower distance with explicit missing, `n1 = 40`, embedding re-rank @@ -82,13 +84,21 @@ a model or embedding call. - Tests: `test_learn_compile.py` (`test_reproducible_from_cache` on both backends), `test_learn_calibrate.py`, `test_learn_decide.py`. - _Requirements: 3.3, 3.5, 5.2, 7.3, 7.4_ - - [ ] 6.6 First paid slice **(budget go-ahead: ~$5 expected, ceiling $7)** + - [x] 6.6 First paid slice **(budget go-ahead: ~$5 expected, ceiling $7)** - Smoke run on 20 candidates (~$0.10): real token counts, prefix-cache hit share, parse rate; it fixes `k` (5 if samples 2..k hit the cache for >= 80% of their prompt tokens, else 1) before anything else is spent. - One compile per profile with data (`static` first; ~$2.2, ceiling $3). - Outer-fold evaluation, priors off, evaluable families only (675 candidates: ~$2.0 at k = 1, ~$3.0 at k = 5), with the 30-candidate canary set; report under `benchmarks/measurements/-decision-engine-v1/` (counts and intervals only, no identities). + - Done 2026-10-01 for family **injection only** (the maintainer's scope for this slice; memory holds every family, + at most 2 contrast examples per prompt): smoke fixed `k = 5` (repeat hit share 0.96); program `c1237a49...` + (proposed instruction 5, balanced Brier 0.128 on 13 `C_val` candidates vs 0.199 baseline); 110 out-of-repository + candidates in 29 groups: ADVISORY recall 0.90 [0.82, 0.97], precision 0.54 [0.44, 0.65]; AUC of `s` 0.74 [0.65, + 0.85]; BLOCK not offered (ECE 0.064 > 0.05, and no fold reached a Wilson lower bound of 0.95); canary agreement + 0.83; $0.91 metered at list price (provider balance $18.20 -> $17.90). Record: + `benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`. The other evaluable families + are not evaluated yet. - _Requirements: 3.1-3.5, 5.2, 7.3, 7.4_ - [ ] 6.7 Second paid slice **(budget go-ahead: ~$7 expected, ceiling $10; needs a top-up)** - Leave-one-source-out (~$1.4-2.1), leave-one-framework-out where a framework reaches 10 groups ($0 today), diff --git a/benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json b/benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json new file mode 100644 index 0000000..0c23cca --- /dev/null +++ b/benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json @@ -0,0 +1,524 @@ +{ + "calibration": { + "ece": 0.0636, + "holds": false, + "intercept": -0.0179, + "intercept_ci": [ + -0.5344, + 0.4288 + ], + "n": 110, + "offered": { + "advisory": { + "offered": true, + "reason": null + }, + "block": { + "offered": false, + "reason": "calibration" + } + }, + "reliability": [ + { + "mean_p": 0.2891, + "n": 22, + "positive_rate": 0.3636 + }, + { + "mean_p": 0.4015, + "n": 22, + "positive_rate": 0.3182 + }, + { + "mean_p": 0.4867, + "n": 22, + "positive_rate": 0.4545 + }, + { + "mean_p": 0.6075, + "n": 22, + "positive_rate": 0.6818 + }, + { + "mean_p": 0.8718, + "n": 22, + "positive_rate": 0.8182 + } + ], + "rule": "holds when the slope CI contains 1, the intercept CI contains 0 and ECE <= 0.05", + "slope": 0.9155, + "slope_ci": [ + 0.5808, + 1.6419 + ] + }, + "canary": { + "agreement": 0.8333333333333334, + "canary": 30, + "mean_abs_s_diff": 0.03833333333333333, + "reparsed": 30, + "set_digest": "f0888f459e9daa9175c8d3ba8df75631f66f52598d38904cda8ef6ed9c22de15", + "spend": null + }, + "compile_metric": { + "baseline_instruction": 0.198998, + "c_boot_candidates": 17, + "c_val_candidates": 13, + "chosen": 0.127636, + "configurations": [ + { + "demo_set": 0, + "instruction": 0, + "prompt_tokens": 2271, + "score": 0.198998, + "secondary": { + "recall_at_precision_0.90": 0.571429, + "unsure_share": 0.230769 + } + }, + { + "demo_set": 0, + "instruction": 1, + "prompt_tokens": 2248, + "score": 0.170483, + "secondary": { + "recall_at_precision_0.90": 0.571429, + "unsure_share": 0.307692 + } + }, + { + "demo_set": 0, + "instruction": 2, + "prompt_tokens": 2239, + "score": 0.145871, + "secondary": { + "recall_at_precision_0.90": 0.714286, + "unsure_share": 0.307692 + } + }, + { + "demo_set": 0, + "instruction": 3, + "prompt_tokens": 2236, + "score": 0.17766, + "secondary": { + "recall_at_precision_0.90": 0.428571, + "unsure_share": 0.692308 + } + }, + { + "demo_set": 0, + "instruction": 4, + "prompt_tokens": 2222, + "score": 0.19376, + "secondary": { + "recall_at_precision_0.90": 0.571429, + "unsure_share": 0.692308 + } + }, + { + "demo_set": 0, + "instruction": 5, + "prompt_tokens": 2238, + "score": 0.127636, + "secondary": { + "recall_at_precision_0.90": 0.714286, + "unsure_share": 0.0 + } + }, + { + "demo_set": 0, + "instruction": 6, + "prompt_tokens": 2238, + "score": 0.156588, + "secondary": { + "recall_at_precision_0.90": 0.571429, + "unsure_share": 0.384615 + } + }, + { + "demo_set": 1, + "instruction": 5, + "prompt_tokens": 2137, + "score": 0.169213, + "secondary": { + "recall_at_precision_0.90": 0.571429, + "unsure_share": 0.0 + } + }, + { + "demo_set": 2, + "instruction": 5, + "prompt_tokens": 2118, + "score": 0.129923, + "secondary": { + "recall_at_precision_0.90": 0.714286, + "unsure_share": 0.076923 + } + }, + { + "demo_set": 3, + "instruction": 5, + "prompt_tokens": 2455, + "score": 0.128863, + "secondary": { + "recall_at_precision_0.90": 0.857143, + "unsure_share": 0.0 + } + } + ], + "name": "balanced_brier_v1 (lower is better)", + "on": "C_val, injection candidates" + }, + "created": "2026-10-01", + "embeddings_6_2": { + "calls": 20, + "ceiling_usd": 1.0, + "embedded": 616, + "examples_with_vector": 724, + "excerpts": 616, + "metered_usd": 0.003976, + "model": "openai/text-embedding-3-small", + "route": "OpenRouter", + "tokens": 198819 + }, + "evaluation": { + "advisory": { + "precision": { + "ci": [ + 0.4444444444444444, + 0.6477272727272727 + ], + "value": 0.5416666666666666 + }, + "recall": { + "ci": [ + 0.8245614035087719, + 0.967741935483871 + ], + "value": 0.896551724137931 + } + }, + "advisory_threshold_by_fold": { + "outer-0": 0.2933, + "outer-1": 0.3203, + "outer-2": 0.2933, + "outer-3": 0.3479, + "outer-4": 0.3432 + }, + "auc_raw_score": { + "ci": [ + 0.6451, + 0.8527 + ], + "value": 0.7382 + }, + "block": { + "offered": false, + "reason": "calibration" + }, + "block_reason_by_fold": { + "outer-0": "precision_unreachable", + "outer-1": "precision_unreachable", + "outer-2": "precision_unreachable", + "outer-3": "precision_unreachable", + "outer-4": "precision_unreachable" + }, + "by_language": { + "javascript": { + "advisory": { + "precision": { + "ci": [ + 0.3333333333333333, + 0.6226415094339622 + ], + "value": 0.5 + }, + "recall": { + "ci": [ + 0.6153846153846154, + 0.9375 + ], + "value": 0.8095238095238095 + } + }, + "block": { + "recall": { + "ci": [ + 0.0, + 0.0 + ], + "value": 0.0 + } + }, + "candidates": 41, + "groups": 15, + "status": "pooled" + }, + "php": { + "status": "insufficient data" + }, + "python": { + "status": "insufficient data" + }, + "typescript": { + "status": "insufficient data" + } + }, + "candidates": 110, + "compile_groups": 25, + "folds": { + "outer-0": { + "groups": 15, + "status": "ok" + }, + "outer-1": { + "groups": 15, + "status": "ok" + }, + "outer-2": { + "groups": 15, + "status": "ok" + }, + "outer-3": { + "groups": 14, + "status": "ok" + }, + "outer-4": { + "groups": 14, + "status": "ok" + } + }, + "groups": 29, + "intervals": "repository-cluster bootstrap 95%", + "majority_vulnerable_verdict": { + "precision": { + "ci": [ + 0.7241, + 1.0 + ], + "value": 0.8621 + }, + "recall": { + "ci": [ + 0.2917, + 0.6222 + ], + "value": 0.431 + } + }, + "parse_failed_share": 0.0, + "positives": 58, + "status": "ok", + "undecided": 0, + "verdict_by_label": { + "0:not_vulnerable": 42, + "0:unsure": 6, + "0:vulnerable": 4, + "1:not_vulnerable": 27, + "1:unsure": 6, + "1:vulnerable": 25 + } + }, + "instruments": { + "balance": { + "ok": 110 + }, + "candidates": 110, + "candidates_with_vector": 110, + "contrast_per_candidate": { + "0": 24, + "1": 24, + "2": 62 + }, + "demos_per_candidate": { + "3": 110 + }, + "distinct_s": 52, + "examples_per_candidate": { + "6": 110 + }, + "excerpt_bytes_min_mean_max": [ + 38, + 1166, + 1975 + ], + "modes": { + "signals+embeddings": 110 + }, + "near_duplicates_dropped": 0, + "no_excerpt": 0, + "parse_failed": 0, + "pool_mean": 636.4, + "prompt_chars_mean": 25480 + }, + "labels": { + "same_counts_as": "2026-10-01-decision-engine-labels-advisory-fixes/labels.json", + "sha256": "92f1856c90aaebfaaf0a738c3e4e3a5a730135391b2ea5a3ca2b5987db7430ee" + }, + "memory_build": { + "by_source": { + "adjudications": { + "absent_fixed_side": 0, + "examples": 34, + "labels": 43, + "no_excerpt": 9, + "no_record": 0, + "recorded": 43, + "unread": 0 + }, + "dev-php": { + "absent_fixed_side": 0, + "examples": 3, + "labels": 3, + "no_excerpt": 0, + "no_record": 0, + "recorded": 3, + "unread": 0 + }, + "pairs": { + "absent_fixed_side": 0, + "examples": 560, + "labels": 1544, + "no_excerpt": 0, + "no_record": 984, + "recorded": 560, + "unread": 0 + }, + "population-v1": { + "absent_fixed_side": 0, + "examples": 85, + "labels": 170, + "no_excerpt": 0, + "no_record": 85, + "recorded": 85, + "unread": 0 + }, + "population-v2": { + "absent_fixed_side": 0, + "examples": 42, + "labels": 86, + "no_excerpt": 1, + "no_record": 43, + "recorded": 43, + "unread": 0 + } + }, + "examples": 724, + "examples_by_family_label": { + "access_control:1": 13, + "config_secrets:0": 13, + "config_secrets:1": 18, + "deserialization:1": 20, + "injection:0": 65, + "injection:1": 75, + "memory:0": 171, + "memory:1": 171, + "output_encoding:0": 40, + "output_encoding:1": 56, + "path:0": 1, + "path:1": 13, + "unknown:0": 2, + "unknown:1": 2, + "untrusted_destination:0": 19, + "untrusted_destination:1": 45 + }, + "examples_by_source_label": { + "adjudications:0": 32, + "adjudications:1": 2, + "dev-php:1": 3, + "pairs:0": 226, + "pairs:1": 334, + "population-v1:0": 35, + "population-v1:1": 50, + "population-v2:0": 18, + "population-v2:1": 24 + }, + "excerpt_bytes_total": 817773, + "excerpts_read": 724, + "failing_reason": "adjudications: 9 of 43 recorded candidates are anonymous functions (N, ) that function_span cannot locate by name; the clones were read (unread 0). Exit 2 recorded, rows kept.", + "failing_sources": [ + "adjudications" + ], + "profile": "static", + "skipped": { + "conditional label (condition not evaluated)": 4743 + } + }, + "note": "COUNTS ONLY. Decision-engine tasks 6.2 and 6.6, family INJECTION only: memory = every family's static examples (retrieval may show up to 2 contrast examples of other families), candidates = injection. Input profile v1 (engine and facts.function_lines withheld), priors off. Out-of-repository: compile split (25% of groups) never evaluated; outer grouped 5-fold over the rest; Platt map cross-fitted on the other folds; operating points nested.", + "program": { + "bootstrap": { + "candidates": 17, + "kept": 11, + "rejected": { + "not confident": 1, + "rationale names an identity": 1, + "wrong or unsure": 4 + } + }, + "classify": { + "confidence": "vote_mean", + "inputs": "v1", + "k": 5, + "max_output_tokens": 300, + "model": "deepseek-flash", + "model_params_digest": "f31055fe086f97cf687b9f2198637556ec8b9eee5624f8004a178c502166735a", + "temperature": [ + 0.0, + 0.7 + ], + "withheld": [ + "engine", + "facts.function_lines" + ] + }, + "data": { + "examples": 724, + "groups": 98, + "sources": [ + "adjudications", + "dev-php", + "pairs", + "population-v1", + "population-v2" + ] + }, + "demos": 3, + "id": "c1237a49e19d2477aa8521b182271fa9d6eae165d391aa0f2a4c4c6d6a825a8b", + "instruction_source": "proposed:5" + }, + "replay_check": "evaluate re-run replay-only on the committed code: 550 responses replayed, $0, identical overall and calibration", + "smoke": { + "candidates": 17, + "k5_repeat_hit_share_measured": 0.96, + "k_fixed": 5, + "k_rule": "k = 5 when samples 2..k hit the prefix cache for >= 0.8 of their prompt tokens", + "parse_failed": 0 + }, + "spend": { + "balance_note": "the provider charged ~1/3 of the list-price meter (balance moved $0.30 for $0.91 metered): DeepSeek's off-peak discount window, presumably; the meter prices at list", + "ceilings_usd": { + "6_2": 1.0, + "6_6": 7.0 + }, + "cost_per_candidate_evaluate_usd": 0.003215, + "deepseek_balance_usd": { + "after": 17.9, + "before": 18.2 + }, + "deepseek_tokens": { + "completion_tokens": 155480, + "prompt_cache_hit_tokens": 8601840, + "prompt_tokens": 9936337 + }, + "metered_at_list_price_usd": { + "canary": 0.010158, + "compile": 0.429194, + "compile_first_attempt_failed_400": 0.049047, + "deepseek_total_6_6": 0.912838, + "embed_6_2_openrouter": 0.003976, + "evaluate": 0.353626, + "smoke_deepseek": 0.070812 + } + } +} diff --git a/src/openultrasast/learn/cli.py b/src/openultrasast/learn/cli.py index e69df4c..f8eadfd 100644 --- a/src/openultrasast/learn/cli.py +++ b/src/openultrasast/learn/cli.py @@ -25,6 +25,7 @@ def add_commands(learn_sub: Any) -> None: build.add_argument("--cache", type=Path, help="clones and checkouts (default ~/.cache/openultrasast)") build.add_argument("--memory", help="the store (default OUSAST_MEMORY, then results_root()/memory)") build.add_argument("--profile", choices=("static", "plane"), default="static") + build.add_argument("--units", type=Path, help="the harvest units.json: the pin each pair side's record is stored under") build.add_argument("--dry-run", action="store_true", help="count, write nothing") embed = memory_sub.add_parser("embed", help="embed the stored excerpts (OpenRouter, metered; cached by excerpt sha)") embed.add_argument("--memory", help="the store (default OUSAST_MEMORY, then results_root()/memory)") @@ -209,19 +210,21 @@ def _memory_embed(args: argparse.Namespace) -> int: def _memory_build(args: argparse.Namespace) -> int: """Exit 2 on a refused label (``origin: user``, a source outside the list) or a source with >10% unread excerpts.""" from ..plane.memory import open_store - from .examples import ExampleBuildError, SideReader, build_examples, read_jsonl, records_by_key + from .examples import ExampleBuildError, SideReader, build_examples, pair_pins, read_jsonl, records_by_key from .labels import DEFAULT_CACHE, DEFAULT_SOURCES, load_sources sources = load_sources(args.sources or DEFAULT_SOURCES) labels = read_jsonl(args.labels) store = None if args.dry_run else open_store(args.memory) - rows = [row for path in args.features for row in read_jsonl(path)] + rows = [row for path in args.features for row in read_jsonl(path) if row.get("profile", args.profile) == args.profile] if store is not None: - rows += [r.row for r in store.rows(kind="features")] + # one profile only: the static and plane records of a candidate share a key, and the other would overwrite it + rows += [r.row for r in store.rows(kind="features") if r.row.get("profile") == args.profile] reader = SideReader(args.root, args.cache or DEFAULT_CACHE, sources, labels) try: build = build_examples( labels, records_by_key(rows), reader, sources=sources, store=store, profile=args.profile, license_of=reader.license, + pins=pair_pins(json.loads(args.units.read_text(encoding="utf-8"))) if args.units is not None else None, ) # fmt: skip except ExampleBuildError as exc: print(f"learn memory build: {exc}", file=sys.stderr) diff --git a/src/openultrasast/learn/compile.py b/src/openultrasast/learn/compile.py index aa340a8..2f7d46b 100644 --- a/src/openultrasast/learn/compile.py +++ b/src/openultrasast/learn/compile.py @@ -354,12 +354,14 @@ def compile_program( vectors: Mapping[str, Sequence[float]] | None = None, identities: Callable[[Example], Iterable[str]] | None = None, families: Sequence[str] = (), + candidate_families: Sequence[str] = (), store: MemoryStore | None = None, created: str = "", code_commit: str = "", ) -> CompileResult: """Compile one profile's program from ``memory`` (module docstring). Deterministic given the memory, the spec and - the response cache.""" + the response cache. ``candidate_families`` limits the bootstrap and validation candidates to those families (a + one-family slice); retrieval still draws on every family of the compile split, for contrast examples.""" seed = spec.seed pool = [e for e in memory if e.profile == profile] split: CompileSplit = compile_split(pool, fraction=float(spec["split"].get("compile_fraction", 0.25)), @@ -368,8 +370,9 @@ def compile_program( assert_disjoint(split, evaluation) compile_memory = [e for e in pool if e.group in split.groups] fold = Fold("compile", "compile", frozenset(), phase="compile") - boot = _order([e for e in compile_memory if e.group in split.boot], seed, "boot")[: int(spec["bootstrap"].get("max_candidates", 150))] - val = _order([e for e in compile_memory if e.group in split.val], seed, "val")[: int(spec["search"].get("val_candidates", 60))] + asked = [e for e in compile_memory if not candidate_families or e.family in candidate_families] + boot = _order([e for e in asked if e.group in split.boot], seed, "boot")[: int(spec["bootstrap"].get("max_candidates", 150))] + val = _order([e for e in asked if e.group in split.val], seed, "val")[: int(spec["search"].get("val_candidates", 60))] if not boot or not val: raise CompileError(f"compile split too small: {len(boot)} bootstrap and {len(val)} validation candidates") retrieval, classify = spec["retrieval"], spec["classify"] diff --git a/src/openultrasast/learn/compile.toml b/src/openultrasast/learn/compile.toml index 8b3206b..d9f8211 100644 --- a/src/openultrasast/learn/compile.toml +++ b/src/openultrasast/learn/compile.toml @@ -29,7 +29,9 @@ val_candidates = 60 [classify] model = "deepseek-flash" -k = 1 # k and lam are A/B variants (design section 6), not searched here +k = 5 # fixed by the smoke run before any compile spend (task 6.6): samples 2..5 hit the + # provider prefix cache for 0.96 of their prompt tokens (>= 0.8), 2026-10-01. k and lam + # remain A/B variants (design section 6), not searched here max_output_tokens = 300 inputs = "v1" # the input profile: the engine instrument is withheld from prompts and retrieval (2026-10-01) diff --git a/src/openultrasast/learn/examples.py b/src/openultrasast/learn/examples.py index c439419..c24842e 100644 --- a/src/openultrasast/learn/examples.py +++ b/src/openultrasast/learn/examples.py @@ -22,7 +22,9 @@ Memory examples carry **no rationale** (code, signals, roles, family and label o ``fix_function`` (a different handler of the same snapshot). Pair labels have no commit pin (the corpus holds excerpts); their envelope pin is :func:`label_pin`, a stable -40-hex digest of the pair and its side, so the vulnerable and the fixed side stay distinct rows. +40-hex digest of the pair and its side, so the vulnerable and the fixed side stay distinct rows. The harvest keyed a +pair side's feature record by its unit's pin instead (the blob sha1 of the side's excerpt, ``plane/harvest.py``): +:func:`pair_pins` reads that mapping from the harvest ``units.json`` and the build uses it when given. """ from __future__ import annotations @@ -62,6 +64,11 @@ def label_pin(label: Mapping[str, Any]) -> str: return hashlib.sha1(subject.encode("utf-8")).hexdigest() # noqa: S324 -- a name, not a security digest +def pair_pins(units: Mapping[str, Mapping[str, Any]]) -> dict[tuple[str, str], str]: + """Harvest ``units.json`` -> ``(case reference, side) -> pin`` for the pair units (the pin their records carry).""" + return {(str(u["ref"]), str(u["side"])): str(u["pin"]) for u in units.values() if u.get("source") == "pairs"} + + def record_key(repo: str, pin: str, candidate: str, family: str, unit: str) -> tuple[str, str, str, str, str]: return repo_key(repo), pin, candidate, family, unit @@ -127,9 +134,11 @@ def build_examples( license_of: Callable[[Mapping[str, Any]], str] | None = None, function_of: Callable[[Mapping[str, Any], str], str | None] | None = None, bounds: Bounds = EXAMPLE, + pins: Mapping[tuple[str, str], str] | None = None, ) -> MemoryBuild: """Example rows for the verified labels that have a feature record (``records`` by :func:`record_key`) and an - excerpt; with ``store`` the excerpt blobs and the rows are written. Delta units append the base->head diff.""" + excerpt; with ``store`` the excerpt blobs and the rows are written. Delta units append the base->head diff. + ``pins`` (:func:`pair_pins`) gives a pair side the pin its harvest record was stored under.""" rows = [_as_dict(label) for label in labels] allowed = {s.id for s in sources.sources} | {d for d, kind in DERIVED_SOURCES.items() if any(s.kind == kind for s in sources.sources)} for row in rows: @@ -145,7 +154,9 @@ def build_examples( continue count = build.sources.setdefault(str(label["source"]), SourceCount()) count.labels += 1 - pin = label_pin(label) + side = (str(label.get("source_ref")), str(label.get("pin_role") or "vulnerable")) + pin = (pins or {}).get(side) if label.get("source") == "pairs" else None + pin = pin or label_pin(label) candidate, family, unit = str(label["candidate"]), str(label["family"]), str(label["unit"]) record = records.get(record_key(str(label["repo"]), pin, candidate, family, unit)) if record is None or record.get("profile") != profile: @@ -307,5 +318,5 @@ def records_by_key(rows: Iterable[Mapping[str, Any]]) -> dict[tuple[str, str, st __all__ = [ "EXAMPLE_KIND", "NO_EXCERPT_LIMIT", "Example", "ExampleBuildError", "MemoryBuild", "SideReader", "build_examples", - "label_pin", "label_set_digest", "load_examples", "read_jsonl", "record_key", "records_by_key", + "label_pin", "label_set_digest", "load_examples", "pair_pins", "read_jsonl", "record_key", "records_by_key", ] # fmt: skip diff --git a/src/openultrasast/learn/program.py b/src/openultrasast/learn/program.py index 777a455..284e394 100644 --- a/src/openultrasast/learn/program.py +++ b/src/openultrasast/learn/program.py @@ -87,16 +87,21 @@ SIGNATURE = Signature( ) ) -BASELINE_INSTRUCTION = ( - "You decide whether a function contains an exploitable security vulnerability of the given family.\n" - "You see the function's code (numbered lines), the signals static instruments emitted for it, and the roles\n" - "inferred for its calls. Signals are evidence, not verdicts: rules over-report, and a missing instrument is not a\n" - "negative. Similar labelled functions from other repositories follow as examples; reason about this code.\n" +# The answer contract is part of the signature, not of the searched instruction: a proposed instruction that leaves it +# out still gets it (``render_prefix``), so every prompt asks for JSON (DeepSeek's JSON mode refuses a prompt that +# never says "json") and every answer can parse. +ANSWER_FORMAT = ( 'Answer with one JSON object: {"verdict": "vulnerable" | "not_vulnerable" | "unsure", "family": , "confidence": , "rationale": ,\n' '"cited_lines": []}. Cite at least one line. Say "unsure"\n' "when the code does not decide it." ) +BASELINE_INSTRUCTION = ( + "You decide whether a function contains an exploitable security vulnerability of the given family.\n" + "You see the function's code (numbered lines), the signals static instruments emitted for it, and the roles\n" + "inferred for its calls. Signals are evidence, not verdicts: rules over-report, and a missing instrument is not a\n" + "negative. Similar labelled functions from other repositories follow as examples; reason about this code.\n" + ANSWER_FORMAT +) # --- rendering --------------------------------------------------------------------------------------------------------- @@ -232,7 +237,8 @@ ExcerptText = Callable[[str], str | None] def render_prefix(spec: ProgramSpec, demos: Sequence[Demo], examples: Mapping[str, Example], excerpt_text: ExcerptText) -> str: """The instruction and the demonstrations: identical for every candidate of one program and fold.""" - parts = [spec.instruction.strip()] + instruction = spec.instruction.strip() + parts = [instruction if ANSWER_FORMAT in instruction else f"{instruction}\n{ANSWER_FORMAT}"] for number, demo in enumerate(demos, 1): example = examples[demo.example_id] code = excerpt_text(demo.excerpt_sha) or "" @@ -481,7 +487,8 @@ class Program: __all__ = [ - "BASELINE_INSTRUCTION", "PARSE_FAILED", "SIGNATURE", "UNSURE", "VERDICTS", "Answer", "Caller", "Candidate", "Decision", "Demo", + "ANSWER_FORMAT", "BASELINE_INSTRUCTION", "PARSE_FAILED", "SIGNATURE", "UNSURE", "VERDICTS", "Answer", "Caller", "Candidate", + "Decision", "Demo", "Program", "ProgramSpec", "Prompt", "ReplayMiss", "aggregate", "classify", "label_word", "line_numbers", "params_digest", "parse_answer", "render", "render_prefix", "render_roles", "render_signals", "request_key", ] # fmt: skip diff --git a/src/openultrasast/learn/retrieve.py b/src/openultrasast/learn/retrieve.py index 8119f16..c68e4ee 100644 --- a/src/openultrasast/learn/retrieve.py +++ b/src/openultrasast/learn/retrieve.py @@ -11,12 +11,13 @@ profile, computed per instrument block so that missing is explicit, never impute - ``D`` is the uniform mean over the profile's blocks; prior features are dropped with ``priors = "off"``, and the blocks of instruments the input profile withholds (``inputs``, default ``v1``: the engine) are left out entirely. -Ties break by ``sha256(example id + seed)``; the first ``n1 = 40`` go to stage 2, a cosine re-rank of the code +Ties break by ``sha256(example id + seed)``; the first ``n1 = 40`` of the candidate's family and the first ``n1`` +of the other families go to stage 2, a cosine re-rank of the code embeddings, ``r = lam (1 - D) + (1 - lam) cos``. Without the target's vector the whole retrieval is ``signals_only``; an example without a vector scores ``cos = 1 - D`` (its signal similarity, so it keeps its stage-1 standing). The **balance rule** then takes ``k_ret = 6``: at most ``k/2`` per label, at most 2 per -repository group, at most 2 from other families while the candidate's family has examples left; ``short`` when -the pool cannot fill it. +repository group, at most 2 contrast examples from other families whenever the pool holds the candidate's family +(never padded with more); ``short`` when the pool cannot fill it. **The boundary.** :func:`eligible` is the only way an example enters a prompt -- retrieved examples and compiled demonstrations alike: not the target's group, not a group the fold evaluates, not the held-out source, not the @@ -43,6 +44,7 @@ LAM = 0.5 NEAR_DUPLICATE_COS = 0.98 UNBOUNDED_CAP = 10.0 PER_GROUP = 2 +CONTRAST = 2 # contrast examples from other families, at most, while the pool holds the target's family class BoundaryViolation(AssertionError): @@ -153,35 +155,26 @@ def _tie(example_id: str, seed: int) -> str: def balance(ranked: Sequence[Example], family: str, k: int = K_RET) -> tuple[list[Example], str]: - """Take up to ``k`` in rank order: <= k/2 per label, <= 2 per group, <= 2 other families while the target's family - has examples left; then, if short, fill from other families under the label and group caps.""" - per_label, other_cap = k // 2, min(2, k) + """Take up to ``k`` in rank order: <= k/2 per label, <= 2 per group, and at most 2 contrast examples from other + families whenever the pool holds any example of the target's family (maintainer, 2026-10-01: "allow 2 contrast"). + A pool without the target's family takes from the others under the label and group caps. ``short`` when the + pool cannot fill ``k`` or a label minimum: what exists is taken, never padded with more contrast.""" + per_label = k // 2 + other_cap = min(CONTRAST, k) if any(e.family == family for e in ranked) else k taken: list[Example] = [] labels: dict[int, int] = {} groups: dict[str, int] = {} - - def fits(example: Example) -> bool: - return labels.get(example.label, 0) < per_label and groups.get(example.group, 0) < PER_GROUP - - def take(example: Example) -> None: - taken.append(example) - labels[example.label] = labels.get(example.label, 0) + 1 - groups[example.group] = groups.get(example.group, 0) + 1 - others = 0 for example in ranked: if len(taken) == k: break if example.family != family and others >= other_cap: continue - if fits(example): + if labels.get(example.label, 0) < per_label and groups.get(example.group, 0) < PER_GROUP: others += example.family != family - take(example) - for example in ranked: - if len(taken) == k: - break - if example.id not in {e.id for e in taken} and fits(example): - take(example) + taken.append(example) + labels[example.label] = labels.get(example.label, 0) + 1 + groups[example.group] = groups.get(example.group, 0) + 1 pool_labels = {e.label for e in ranked} short = len(taken) < k or any(labels.get(lab, 0) < min(2, per_label) for lab in (0, 1) if lab in pool_labels) return taken, "short" if short else "ok" @@ -202,10 +195,14 @@ def retrieve( ) -> Retrieval: """The ``k`` balanced examples for ``target`` from the eligible part of ``pool`` (module docstring).""" candidates = [e for e in pool if e.profile == target.profile and eligible(e, target, fold)] - scored = sorted( + ranked_all = sorted( ((gower(target.x, target.instruments, e.x, e.instruments, target.profile, priors=priors, inputs=inputs), e) for e in candidates), key=lambda pair: (pair[0], _tie(pair[1].id, seed)), - )[:n1] + ) + # stage 1 per side: the n1 nearest of the target's family and the n1 nearest contrast examples, so a family that is a + # minority of the memory still reaches stage 2 (the balance rule then takes at most 2 contrast examples) + own = [pair for pair in ranked_all if pair[1].family == target.family][:n1] + scored = own + [pair for pair in ranked_all if pair[1].family != target.family][:n1] mode = "signals+embeddings" if target.vector is not None and vectors else "signals_only" near = without = 0 reranked: list[tuple[float, str, Example]] = [] diff --git a/tests/test_learn_compile.py b/tests/test_learn_compile.py index 72a63fc..bd9674f 100644 --- a/tests/test_learn_compile.py +++ b/tests/test_learn_compile.py @@ -6,7 +6,7 @@ byte for byte at $0 (``test_reproducible_from_cache``); the permissive-licence f from __future__ import annotations import json -from collections.abc import Iterator, Mapping, Sequence +from collections.abc import Iterable, Iterator, Mapping, Sequence from pathlib import Path import pytest @@ -30,6 +30,7 @@ from openultrasast.learn.compile import ( permissive, program_id, ) +from openultrasast.learn.examples import Example from openultrasast.learn.folds import Fold, FoldError from openultrasast.learn.program import Caller, Demo from openultrasast.plane.budget import MeteredClient @@ -191,3 +192,23 @@ def test_only_permissively_licensed_demonstrations_are_packaged() -> None: assert [d["example_id"] for d in packaged["demos"]] == [mit.id, apache.id] assert packaged["program_id"] == program_id(packaged) != artifact["program_id"] assert packaged["packaged"]["from"] == artifact["program_id"] + + +def test_a_one_family_compile_asks_only_that_family(monkeypatch: pytest.MonkeyPatch) -> None: + """``candidate_families`` limits the bootstrap and validation candidates; the compile memory keeps every family + (retrieval's contrast examples), so the artifact's data counts are those of the whole pool.""" + asked: dict[str, set[str]] = {} + real = compile_module._order + + def spy(items: Iterable[Example], seed: int, salt: str) -> list[Example]: + items = list(items) + asked.setdefault(salt, set()).update(e.family for e in items) + return real(items, seed, salt) + + monkeypatch.setattr(compile_module, "_order", spy) + result = compile_program(MEMORY, _caller(ScriptedChat(answer)), excerpt_text, spec=SPEC, candidate_families=("injection",)) + assert result.status == "done" and result.artifact is not None + assert asked["boot"] == asked["val"] == {"injection"} + assert result.artifact["data"]["examples"] == len(MEMORY) + with pytest.raises(CompileError, match="compile split too small"): + compile_program(MEMORY, _caller(ScriptedChat(answer)), excerpt_text, spec=SPEC, candidate_families=("memory",)) diff --git a/tests/test_learn_examples.py b/tests/test_learn_examples.py index 22407a7..e60fe6a 100644 --- a/tests/test_learn_examples.py +++ b/tests/test_learn_examples.py @@ -18,6 +18,7 @@ from openultrasast.learn.examples import ( build_examples, label_pin, load_examples, + pair_pins, record_key, records_by_key, ) @@ -155,3 +156,18 @@ def test_a_fixed_side_without_the_function_is_a_label_defect_not_an_unread_side( assert count.absent_fixed_side == 1 and count.no_excerpt == 0 and build.failing() == [] unread = build_examples([fixed], records_by_key([feature(fixed)]), lambda lab, role: None, sources=SOURCES) assert unread.sources["population-v1"].unread == 1 and unread.failing() == ["population-v1"] + + +def test_pair_sides_join_the_record_stored_under_their_harvest_unit_pin() -> None: + """The harvest stores a pair side's record under its unit pin (the blob sha1 of the side's excerpt), not under + :func:`label_pin`; ``pins`` from the harvest ``units.json`` joins them, and without it the record is not found.""" + lab = label(source="pairs", source_ref="case-9", pin="") + unit_pin = "a" * 40 + stored = {**feature(lab), "pin": unit_pin} + units = {"h-1": {"source": "pairs", "ref": "case-9", "side": "vulnerable", "pin": unit_pin, "repo": lab["repo"]}, + "h-2": {"source": "population-v1", "ref": "case-9", "side": "vulnerable", "pin": PIN, "repo": lab["repo"]}} # fmt: skip + pins = pair_pins(units) + assert pins == {("case-9", "vulnerable"): unit_pin} + assert build_examples([lab], records_by_key([stored]), lines_of, sources=SOURCES).summary()["examples"] == 0 + build = build_examples([lab], records_by_key([stored]), lines_of, sources=SOURCES, pins=pins) + assert build.summary()["examples"] == 1 and build.rows[0]["pin"] == unit_pin diff --git a/tests/test_learn_program.py b/tests/test_learn_program.py index b9185f6..f2d297a 100644 --- a/tests/test_learn_program.py +++ b/tests/test_learn_program.py @@ -15,6 +15,8 @@ from learn_fixtures import ScriptedChat, by_id, corpus, x_record from openultrasast.learn import program as program_module from openultrasast.learn.folds import outer_folds from openultrasast.learn.program import ( + ANSWER_FORMAT, + BASELINE_INSTRUCTION, PARSE_FAILED, Answer, Caller, @@ -121,6 +123,15 @@ def test_the_prefix_is_byte_identical_across_candidates() -> None: assert "Demonstration 1:" in a.prefix and a.messages[1] != b.messages[1] +def test_every_instruction_carries_the_answer_contract() -> None: + """A proposed instruction rarely restates the JSON contract; the prefix adds it, once, so JSON mode is accepted + (DeepSeek refuses a prompt without the word "json") and the answer parses. The baseline prefix is unchanged.""" + proposed, _ = program(instruction="Weigh the sanitiser before the sink.").prepare(candidate(), FOLD) + baseline, _ = program().prepare(candidate(), FOLD) + assert proposed.prefix.startswith("Weigh the sanitiser before the sink.\n") and proposed.prefix.count(ANSWER_FORMAT) == 1 + assert baseline.prefix.count(ANSWER_FORMAT) == 1 and baseline.prefix.startswith(BASELINE_INSTRUCTION) + + def test_parsing_valid_invalid_and_out_of_range_citations() -> None: lines = frozenset({40, 41}) ok = parse_answer( diff --git a/tests/test_learn_retrieve.py b/tests/test_learn_retrieve.py index 8e02b8a..74fae50 100644 --- a/tests/test_learn_retrieve.py +++ b/tests/test_learn_retrieve.py @@ -8,7 +8,7 @@ from dataclasses import replace import pytest from learn_fixtures import by_id, corpus, example, x_record -from openultrasast.learn.folds import compile_split, leave_one_framework_out, leave_one_source_out, outer_folds +from openultrasast.learn.folds import Fold, compile_split, leave_one_framework_out, leave_one_source_out, outer_folds from openultrasast.learn.retrieve import ( BoundaryViolation, Target, @@ -102,6 +102,25 @@ def test_balance_caps_labels_groups_and_other_families() -> None: crowded = [example(f"c{i}", group="one/repo", label=i % 2) for i in range(10)] taken, state = balance(crowded, "injection", 6) assert len(taken) == 2 and state == "short" # at most 2 per repository group + scarce = [example("own", group="own/repo", label=1)] + [ + example(f"o{i}", group=f"o{i}/r", label=i % 2, family="path") for i in range(10) + ] + taken, state = balance(scarce, "injection", 6) + assert sum(e.family != "injection" for e in taken) == 2 and len(taken) == 3 and state == "short" # never padded + absent = [example(f"a{i}", group=f"a{i}/r", label=i % 2, family="path") for i in range(10)] + taken, state = balance(absent, "injection", 6) + assert len(taken) == 6 and state == "ok" # no example of the family at all: the others fill under the caps + + +def test_a_minority_family_reaches_stage_two_and_contrast_stays_at_two() -> None: + """Stage 1 keeps the n1 nearest of the target's family beside the n1 nearest others: 50 contrast examples nearer + in signals cannot push the family out, and the balance rule takes at most 2 of them.""" + others = [example(f"o{i}", group=f"o{i}/r", label=i % 2, family="path", hits=0) for i in range(50)] + own = [example(f"own{i}", group=f"own{i}/r", label=i % 2, hits=6) for i in range(4)] + fold = Fold("compile", "compile", frozenset(), phase="compile") + got = retrieve(_target(hits=0), [*others, *own], fold, n1=40, k=6) + assert sum(e.family == "injection" for e in got.examples) == 4 + assert sum(e.family != "injection" for e in got.examples) == 2 def test_near_duplicates_drop_in_evaluation_and_stay_in_deployment() -> None: