mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
bench(learn): injection first slice -- compiled AI classifier, out-of-repo evaluation (decision-engine 6.2, 6.6)
6.2: the 724 static examples' 616 distinct excerpts embedded with openai/text-embedding-3-small via OpenRouter, 198,819 tokens, $0.0040 metered (ceiling $1). 6.6, family injection only, memory = every family (at most 2 contrast examples per prompt), input profile v1: smoke fixed k = 5 (repeat prefix-cache hit share 0.96); one compile (program c1237a49, proposed instruction 5, balanced Brier 0.128 on 13 C_val candidates vs 0.199 baseline); outer grouped 5-fold, 110 out-of-repository candidates in 29 groups: ADVISORY recall 0.90 [0.82, 0.97], precision 0.54 [0.44, 0.65], AUC 0.74; BLOCK not offered (ECE 0.064 > 0.05; no fold reached a Wilson lower bound of 0.95); canary re-ask agreement 0.83. $0.91 metered at list price; DeepSeek balance $18.20 -> $17.90. Instrument fixes found on the way, each with a test: - memory build: pair sides join the record stored under their harvest unit pin (--units), and only the built profile's records are read (static and plane shared a key; 85 -> 724 examples); - retrieval: stage 1 per side (family / contrast) and at most 2 contrast examples, never padded; - program: the JSON answer contract is part of the signature, appended to proposed instructions (DeepSeek JSON mode refused a prompt without "json"); - compile: candidate_families for a one-family compile; compile.toml k = 5 from the smoke rule. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
68294e9db3
commit
7b17eacd44
@@ -529,7 +529,8 @@ explicit rather than imputed:
|
||||
block). Prior features (`FeatureSpec.prior`) are dropped when `priors = off`. Weights are a compile-time
|
||||
setting only if an A/B shows a gain; the default is uniform.
|
||||
- Candidates in the eligible pool (after the boundary filter) are ranked by `D`, ties by `sha256(example_id +
|
||||
seed)`; the first `n1 = 40` go to stage 2.
|
||||
seed)`; the first `n1 = 40` of the candidate's family and the first `n1` of the other families go to stage 2
|
||||
(per side since 2026-10-01, so a minority family reaches stage 2; Risks, "First paid slice").
|
||||
|
||||
**Stage 2: code-embedding re-rank.** Cosine similarity of the candidate's excerpt embedding with each stage-1
|
||||
example's. The re-rank score is `r = lam * (1 - D) + (1 - lam) * cos`, with `lam` a compiled setting in {0, 0.5}
|
||||
@@ -1205,6 +1206,27 @@ start only after the maintainer confirms their budget against the balance read j
|
||||
Pre-registering that and recording "inconclusive" is the intended behaviour. The failure mode to avoid is
|
||||
extending an experiment after looking.
|
||||
|
||||
- **First paid slice, 2026-10-01 (tasks 6.2, 6.6; injection only).** Four instrument defects surfaced before any
|
||||
number was believed, each fixed with a test:
|
||||
- *The memory build joined no pair.* The harvest stores a pair side's record under its unit pin (the blob sha1 of
|
||||
the side's excerpt), not under `label_pin`; `ousast learn memory build --units <harvest units.json>` joins them.
|
||||
And a candidate's `static` and `plane` records share a key, so reading both let one overwrite the other and
|
||||
silently drop about half the examples; the build now reads its own profile only. 85 -> 724 examples.
|
||||
- *Contrast was unbounded.* The balance rule padded a short pick with other families (up to 5 of 6), and stage 1
|
||||
took the 40 nearest of all families, so a minority family often never reached stage 2. Now stage 1 takes the
|
||||
`n1` nearest of the candidate's family and the `n1` nearest others, and at most 2 contrast examples are shown
|
||||
whenever the pool holds the family (maintainer: "allow 2 contrast"); never padded.
|
||||
- *Proposed instructions dropped the answer contract*, and DeepSeek's JSON mode refuses a prompt without the word
|
||||
"json" (HTTP 400, the first compile stopped there). The contract (`ANSWER_FORMAT`) is now part of the signature
|
||||
and appended to any instruction that lacks it; the baseline prefix is byte-identical.
|
||||
- The adjudication source fails the 10% excerpt gate: 9 of its 43 candidates are anonymous functions (`<lambda>N`,
|
||||
`<module>`) that `function_span` cannot find by name; the clones were read. Open.
|
||||
The memory is small: 724 static examples (only harvested candidates have feature records; the 97 advisory-fix pairs
|
||||
have none yet), injection 140 (75/65), so `C_val` had 13 injection candidates and the compile metric is noisy.
|
||||
The measured ranking signal is real but weak at the recall target: AUC 0.74 out of repository, ADVISORY at recall
|
||||
0.90 has precision 0.54 (prevalence 0.53); the majority `vulnerable` verdict alone has precision 0.86, recall 0.43.
|
||||
Canary re-ask agreement at temperature 0 was 0.83, below the 0.9 the adoption check requires.
|
||||
|
||||
## Maintainer decisions at design approval (2026-09-30)
|
||||
|
||||
- Design approved as written.
|
||||
|
||||
@@ -54,6 +54,8 @@ a model or embedding call.
|
||||
`plane/budget.py`, usage kept by `parse_embedding_response`; cache `embeddings/<model>/<excerpt_sha>.json`;
|
||||
`ousast learn memory embed`; scripted-client tests.
|
||||
- **(budget go-ahead)** embedding the store: ~5.0 M tokens, ~$0.10, ceiling $1 (OpenRouter account).
|
||||
Done 2026-10-01: 616 distinct excerpts of the 724 static examples, 198,819 tokens, 20 calls, $0.0040 metered
|
||||
(`benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`).
|
||||
- _Requirements: 3.2_
|
||||
- [x] 6.3 Retrieval and the evaluation boundary (no model)
|
||||
- `learn/retrieve.py`: per-instrument Gower distance with explicit missing, `n1 = 40`, embedding re-rank
|
||||
@@ -82,13 +84,21 @@ a model or embedding call.
|
||||
- Tests: `test_learn_compile.py` (`test_reproducible_from_cache` on both backends), `test_learn_calibrate.py`,
|
||||
`test_learn_decide.py`.
|
||||
- _Requirements: 3.3, 3.5, 5.2, 7.3, 7.4_
|
||||
- [ ] 6.6 First paid slice **(budget go-ahead: ~$5 expected, ceiling $7)**
|
||||
- [x] 6.6 First paid slice **(budget go-ahead: ~$5 expected, ceiling $7)**
|
||||
- Smoke run on 20 candidates (~$0.10): real token counts, prefix-cache hit share, parse rate; it fixes `k`
|
||||
(5 if samples 2..k hit the cache for >= 80% of their prompt tokens, else 1) before anything else is spent.
|
||||
- One compile per profile with data (`static` first; ~$2.2, ceiling $3).
|
||||
- Outer-fold evaluation, priors off, evaluable families only (675 candidates: ~$2.0 at k = 1, ~$3.0 at k = 5),
|
||||
with the 30-candidate canary set; report under `benchmarks/measurements/<date>-decision-engine-v1/` (counts
|
||||
and intervals only, no identities).
|
||||
- Done 2026-10-01 for family **injection only** (the maintainer's scope for this slice; memory holds every family,
|
||||
at most 2 contrast examples per prompt): smoke fixed `k = 5` (repeat hit share 0.96); program `c1237a49...`
|
||||
(proposed instruction 5, balanced Brier 0.128 on 13 `C_val` candidates vs 0.199 baseline); 110 out-of-repository
|
||||
candidates in 29 groups: ADVISORY recall 0.90 [0.82, 0.97], precision 0.54 [0.44, 0.65]; AUC of `s` 0.74 [0.65,
|
||||
0.85]; BLOCK not offered (ECE 0.064 > 0.05, and no fold reached a Wilson lower bound of 0.95); canary agreement
|
||||
0.83; $0.91 metered at list price (provider balance $18.20 -> $17.90). Record:
|
||||
`benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`. The other evaluable families
|
||||
are not evaluated yet.
|
||||
- _Requirements: 3.1-3.5, 5.2, 7.3, 7.4_
|
||||
- [ ] 6.7 Second paid slice **(budget go-ahead: ~$7 expected, ceiling $10; needs a top-up)**
|
||||
- Leave-one-source-out (~$1.4-2.1), leave-one-framework-out where a framework reaches 10 groups ($0 today),
|
||||
|
||||
@@ -0,0 +1,524 @@
|
||||
{
|
||||
"calibration": {
|
||||
"ece": 0.0636,
|
||||
"holds": false,
|
||||
"intercept": -0.0179,
|
||||
"intercept_ci": [
|
||||
-0.5344,
|
||||
0.4288
|
||||
],
|
||||
"n": 110,
|
||||
"offered": {
|
||||
"advisory": {
|
||||
"offered": true,
|
||||
"reason": null
|
||||
},
|
||||
"block": {
|
||||
"offered": false,
|
||||
"reason": "calibration"
|
||||
}
|
||||
},
|
||||
"reliability": [
|
||||
{
|
||||
"mean_p": 0.2891,
|
||||
"n": 22,
|
||||
"positive_rate": 0.3636
|
||||
},
|
||||
{
|
||||
"mean_p": 0.4015,
|
||||
"n": 22,
|
||||
"positive_rate": 0.3182
|
||||
},
|
||||
{
|
||||
"mean_p": 0.4867,
|
||||
"n": 22,
|
||||
"positive_rate": 0.4545
|
||||
},
|
||||
{
|
||||
"mean_p": 0.6075,
|
||||
"n": 22,
|
||||
"positive_rate": 0.6818
|
||||
},
|
||||
{
|
||||
"mean_p": 0.8718,
|
||||
"n": 22,
|
||||
"positive_rate": 0.8182
|
||||
}
|
||||
],
|
||||
"rule": "holds when the slope CI contains 1, the intercept CI contains 0 and ECE <= 0.05",
|
||||
"slope": 0.9155,
|
||||
"slope_ci": [
|
||||
0.5808,
|
||||
1.6419
|
||||
]
|
||||
},
|
||||
"canary": {
|
||||
"agreement": 0.8333333333333334,
|
||||
"canary": 30,
|
||||
"mean_abs_s_diff": 0.03833333333333333,
|
||||
"reparsed": 30,
|
||||
"set_digest": "f0888f459e9daa9175c8d3ba8df75631f66f52598d38904cda8ef6ed9c22de15",
|
||||
"spend": null
|
||||
},
|
||||
"compile_metric": {
|
||||
"baseline_instruction": 0.198998,
|
||||
"c_boot_candidates": 17,
|
||||
"c_val_candidates": 13,
|
||||
"chosen": 0.127636,
|
||||
"configurations": [
|
||||
{
|
||||
"demo_set": 0,
|
||||
"instruction": 0,
|
||||
"prompt_tokens": 2271,
|
||||
"score": 0.198998,
|
||||
"secondary": {
|
||||
"recall_at_precision_0.90": 0.571429,
|
||||
"unsure_share": 0.230769
|
||||
}
|
||||
},
|
||||
{
|
||||
"demo_set": 0,
|
||||
"instruction": 1,
|
||||
"prompt_tokens": 2248,
|
||||
"score": 0.170483,
|
||||
"secondary": {
|
||||
"recall_at_precision_0.90": 0.571429,
|
||||
"unsure_share": 0.307692
|
||||
}
|
||||
},
|
||||
{
|
||||
"demo_set": 0,
|
||||
"instruction": 2,
|
||||
"prompt_tokens": 2239,
|
||||
"score": 0.145871,
|
||||
"secondary": {
|
||||
"recall_at_precision_0.90": 0.714286,
|
||||
"unsure_share": 0.307692
|
||||
}
|
||||
},
|
||||
{
|
||||
"demo_set": 0,
|
||||
"instruction": 3,
|
||||
"prompt_tokens": 2236,
|
||||
"score": 0.17766,
|
||||
"secondary": {
|
||||
"recall_at_precision_0.90": 0.428571,
|
||||
"unsure_share": 0.692308
|
||||
}
|
||||
},
|
||||
{
|
||||
"demo_set": 0,
|
||||
"instruction": 4,
|
||||
"prompt_tokens": 2222,
|
||||
"score": 0.19376,
|
||||
"secondary": {
|
||||
"recall_at_precision_0.90": 0.571429,
|
||||
"unsure_share": 0.692308
|
||||
}
|
||||
},
|
||||
{
|
||||
"demo_set": 0,
|
||||
"instruction": 5,
|
||||
"prompt_tokens": 2238,
|
||||
"score": 0.127636,
|
||||
"secondary": {
|
||||
"recall_at_precision_0.90": 0.714286,
|
||||
"unsure_share": 0.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"demo_set": 0,
|
||||
"instruction": 6,
|
||||
"prompt_tokens": 2238,
|
||||
"score": 0.156588,
|
||||
"secondary": {
|
||||
"recall_at_precision_0.90": 0.571429,
|
||||
"unsure_share": 0.384615
|
||||
}
|
||||
},
|
||||
{
|
||||
"demo_set": 1,
|
||||
"instruction": 5,
|
||||
"prompt_tokens": 2137,
|
||||
"score": 0.169213,
|
||||
"secondary": {
|
||||
"recall_at_precision_0.90": 0.571429,
|
||||
"unsure_share": 0.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"demo_set": 2,
|
||||
"instruction": 5,
|
||||
"prompt_tokens": 2118,
|
||||
"score": 0.129923,
|
||||
"secondary": {
|
||||
"recall_at_precision_0.90": 0.714286,
|
||||
"unsure_share": 0.076923
|
||||
}
|
||||
},
|
||||
{
|
||||
"demo_set": 3,
|
||||
"instruction": 5,
|
||||
"prompt_tokens": 2455,
|
||||
"score": 0.128863,
|
||||
"secondary": {
|
||||
"recall_at_precision_0.90": 0.857143,
|
||||
"unsure_share": 0.0
|
||||
}
|
||||
}
|
||||
],
|
||||
"name": "balanced_brier_v1 (lower is better)",
|
||||
"on": "C_val, injection candidates"
|
||||
},
|
||||
"created": "2026-10-01",
|
||||
"embeddings_6_2": {
|
||||
"calls": 20,
|
||||
"ceiling_usd": 1.0,
|
||||
"embedded": 616,
|
||||
"examples_with_vector": 724,
|
||||
"excerpts": 616,
|
||||
"metered_usd": 0.003976,
|
||||
"model": "openai/text-embedding-3-small",
|
||||
"route": "OpenRouter",
|
||||
"tokens": 198819
|
||||
},
|
||||
"evaluation": {
|
||||
"advisory": {
|
||||
"precision": {
|
||||
"ci": [
|
||||
0.4444444444444444,
|
||||
0.6477272727272727
|
||||
],
|
||||
"value": 0.5416666666666666
|
||||
},
|
||||
"recall": {
|
||||
"ci": [
|
||||
0.8245614035087719,
|
||||
0.967741935483871
|
||||
],
|
||||
"value": 0.896551724137931
|
||||
}
|
||||
},
|
||||
"advisory_threshold_by_fold": {
|
||||
"outer-0": 0.2933,
|
||||
"outer-1": 0.3203,
|
||||
"outer-2": 0.2933,
|
||||
"outer-3": 0.3479,
|
||||
"outer-4": 0.3432
|
||||
},
|
||||
"auc_raw_score": {
|
||||
"ci": [
|
||||
0.6451,
|
||||
0.8527
|
||||
],
|
||||
"value": 0.7382
|
||||
},
|
||||
"block": {
|
||||
"offered": false,
|
||||
"reason": "calibration"
|
||||
},
|
||||
"block_reason_by_fold": {
|
||||
"outer-0": "precision_unreachable",
|
||||
"outer-1": "precision_unreachable",
|
||||
"outer-2": "precision_unreachable",
|
||||
"outer-3": "precision_unreachable",
|
||||
"outer-4": "precision_unreachable"
|
||||
},
|
||||
"by_language": {
|
||||
"javascript": {
|
||||
"advisory": {
|
||||
"precision": {
|
||||
"ci": [
|
||||
0.3333333333333333,
|
||||
0.6226415094339622
|
||||
],
|
||||
"value": 0.5
|
||||
},
|
||||
"recall": {
|
||||
"ci": [
|
||||
0.6153846153846154,
|
||||
0.9375
|
||||
],
|
||||
"value": 0.8095238095238095
|
||||
}
|
||||
},
|
||||
"block": {
|
||||
"recall": {
|
||||
"ci": [
|
||||
0.0,
|
||||
0.0
|
||||
],
|
||||
"value": 0.0
|
||||
}
|
||||
},
|
||||
"candidates": 41,
|
||||
"groups": 15,
|
||||
"status": "pooled"
|
||||
},
|
||||
"php": {
|
||||
"status": "insufficient data"
|
||||
},
|
||||
"python": {
|
||||
"status": "insufficient data"
|
||||
},
|
||||
"typescript": {
|
||||
"status": "insufficient data"
|
||||
}
|
||||
},
|
||||
"candidates": 110,
|
||||
"compile_groups": 25,
|
||||
"folds": {
|
||||
"outer-0": {
|
||||
"groups": 15,
|
||||
"status": "ok"
|
||||
},
|
||||
"outer-1": {
|
||||
"groups": 15,
|
||||
"status": "ok"
|
||||
},
|
||||
"outer-2": {
|
||||
"groups": 15,
|
||||
"status": "ok"
|
||||
},
|
||||
"outer-3": {
|
||||
"groups": 14,
|
||||
"status": "ok"
|
||||
},
|
||||
"outer-4": {
|
||||
"groups": 14,
|
||||
"status": "ok"
|
||||
}
|
||||
},
|
||||
"groups": 29,
|
||||
"intervals": "repository-cluster bootstrap 95%",
|
||||
"majority_vulnerable_verdict": {
|
||||
"precision": {
|
||||
"ci": [
|
||||
0.7241,
|
||||
1.0
|
||||
],
|
||||
"value": 0.8621
|
||||
},
|
||||
"recall": {
|
||||
"ci": [
|
||||
0.2917,
|
||||
0.6222
|
||||
],
|
||||
"value": 0.431
|
||||
}
|
||||
},
|
||||
"parse_failed_share": 0.0,
|
||||
"positives": 58,
|
||||
"status": "ok",
|
||||
"undecided": 0,
|
||||
"verdict_by_label": {
|
||||
"0:not_vulnerable": 42,
|
||||
"0:unsure": 6,
|
||||
"0:vulnerable": 4,
|
||||
"1:not_vulnerable": 27,
|
||||
"1:unsure": 6,
|
||||
"1:vulnerable": 25
|
||||
}
|
||||
},
|
||||
"instruments": {
|
||||
"balance": {
|
||||
"ok": 110
|
||||
},
|
||||
"candidates": 110,
|
||||
"candidates_with_vector": 110,
|
||||
"contrast_per_candidate": {
|
||||
"0": 24,
|
||||
"1": 24,
|
||||
"2": 62
|
||||
},
|
||||
"demos_per_candidate": {
|
||||
"3": 110
|
||||
},
|
||||
"distinct_s": 52,
|
||||
"examples_per_candidate": {
|
||||
"6": 110
|
||||
},
|
||||
"excerpt_bytes_min_mean_max": [
|
||||
38,
|
||||
1166,
|
||||
1975
|
||||
],
|
||||
"modes": {
|
||||
"signals+embeddings": 110
|
||||
},
|
||||
"near_duplicates_dropped": 0,
|
||||
"no_excerpt": 0,
|
||||
"parse_failed": 0,
|
||||
"pool_mean": 636.4,
|
||||
"prompt_chars_mean": 25480
|
||||
},
|
||||
"labels": {
|
||||
"same_counts_as": "2026-10-01-decision-engine-labels-advisory-fixes/labels.json",
|
||||
"sha256": "92f1856c90aaebfaaf0a738c3e4e3a5a730135391b2ea5a3ca2b5987db7430ee"
|
||||
},
|
||||
"memory_build": {
|
||||
"by_source": {
|
||||
"adjudications": {
|
||||
"absent_fixed_side": 0,
|
||||
"examples": 34,
|
||||
"labels": 43,
|
||||
"no_excerpt": 9,
|
||||
"no_record": 0,
|
||||
"recorded": 43,
|
||||
"unread": 0
|
||||
},
|
||||
"dev-php": {
|
||||
"absent_fixed_side": 0,
|
||||
"examples": 3,
|
||||
"labels": 3,
|
||||
"no_excerpt": 0,
|
||||
"no_record": 0,
|
||||
"recorded": 3,
|
||||
"unread": 0
|
||||
},
|
||||
"pairs": {
|
||||
"absent_fixed_side": 0,
|
||||
"examples": 560,
|
||||
"labels": 1544,
|
||||
"no_excerpt": 0,
|
||||
"no_record": 984,
|
||||
"recorded": 560,
|
||||
"unread": 0
|
||||
},
|
||||
"population-v1": {
|
||||
"absent_fixed_side": 0,
|
||||
"examples": 85,
|
||||
"labels": 170,
|
||||
"no_excerpt": 0,
|
||||
"no_record": 85,
|
||||
"recorded": 85,
|
||||
"unread": 0
|
||||
},
|
||||
"population-v2": {
|
||||
"absent_fixed_side": 0,
|
||||
"examples": 42,
|
||||
"labels": 86,
|
||||
"no_excerpt": 1,
|
||||
"no_record": 43,
|
||||
"recorded": 43,
|
||||
"unread": 0
|
||||
}
|
||||
},
|
||||
"examples": 724,
|
||||
"examples_by_family_label": {
|
||||
"access_control:1": 13,
|
||||
"config_secrets:0": 13,
|
||||
"config_secrets:1": 18,
|
||||
"deserialization:1": 20,
|
||||
"injection:0": 65,
|
||||
"injection:1": 75,
|
||||
"memory:0": 171,
|
||||
"memory:1": 171,
|
||||
"output_encoding:0": 40,
|
||||
"output_encoding:1": 56,
|
||||
"path:0": 1,
|
||||
"path:1": 13,
|
||||
"unknown:0": 2,
|
||||
"unknown:1": 2,
|
||||
"untrusted_destination:0": 19,
|
||||
"untrusted_destination:1": 45
|
||||
},
|
||||
"examples_by_source_label": {
|
||||
"adjudications:0": 32,
|
||||
"adjudications:1": 2,
|
||||
"dev-php:1": 3,
|
||||
"pairs:0": 226,
|
||||
"pairs:1": 334,
|
||||
"population-v1:0": 35,
|
||||
"population-v1:1": 50,
|
||||
"population-v2:0": 18,
|
||||
"population-v2:1": 24
|
||||
},
|
||||
"excerpt_bytes_total": 817773,
|
||||
"excerpts_read": 724,
|
||||
"failing_reason": "adjudications: 9 of 43 recorded candidates are anonymous functions (<lambda>N, <module>) that function_span cannot locate by name; the clones were read (unread 0). Exit 2 recorded, rows kept.",
|
||||
"failing_sources": [
|
||||
"adjudications"
|
||||
],
|
||||
"profile": "static",
|
||||
"skipped": {
|
||||
"conditional label (condition not evaluated)": 4743
|
||||
}
|
||||
},
|
||||
"note": "COUNTS ONLY. Decision-engine tasks 6.2 and 6.6, family INJECTION only: memory = every family's static examples (retrieval may show up to 2 contrast examples of other families), candidates = injection. Input profile v1 (engine and facts.function_lines withheld), priors off. Out-of-repository: compile split (25% of groups) never evaluated; outer grouped 5-fold over the rest; Platt map cross-fitted on the other folds; operating points nested.",
|
||||
"program": {
|
||||
"bootstrap": {
|
||||
"candidates": 17,
|
||||
"kept": 11,
|
||||
"rejected": {
|
||||
"not confident": 1,
|
||||
"rationale names an identity": 1,
|
||||
"wrong or unsure": 4
|
||||
}
|
||||
},
|
||||
"classify": {
|
||||
"confidence": "vote_mean",
|
||||
"inputs": "v1",
|
||||
"k": 5,
|
||||
"max_output_tokens": 300,
|
||||
"model": "deepseek-flash",
|
||||
"model_params_digest": "f31055fe086f97cf687b9f2198637556ec8b9eee5624f8004a178c502166735a",
|
||||
"temperature": [
|
||||
0.0,
|
||||
0.7
|
||||
],
|
||||
"withheld": [
|
||||
"engine",
|
||||
"facts.function_lines"
|
||||
]
|
||||
},
|
||||
"data": {
|
||||
"examples": 724,
|
||||
"groups": 98,
|
||||
"sources": [
|
||||
"adjudications",
|
||||
"dev-php",
|
||||
"pairs",
|
||||
"population-v1",
|
||||
"population-v2"
|
||||
]
|
||||
},
|
||||
"demos": 3,
|
||||
"id": "c1237a49e19d2477aa8521b182271fa9d6eae165d391aa0f2a4c4c6d6a825a8b",
|
||||
"instruction_source": "proposed:5"
|
||||
},
|
||||
"replay_check": "evaluate re-run replay-only on the committed code: 550 responses replayed, $0, identical overall and calibration",
|
||||
"smoke": {
|
||||
"candidates": 17,
|
||||
"k5_repeat_hit_share_measured": 0.96,
|
||||
"k_fixed": 5,
|
||||
"k_rule": "k = 5 when samples 2..k hit the prefix cache for >= 0.8 of their prompt tokens",
|
||||
"parse_failed": 0
|
||||
},
|
||||
"spend": {
|
||||
"balance_note": "the provider charged ~1/3 of the list-price meter (balance moved $0.30 for $0.91 metered): DeepSeek's off-peak discount window, presumably; the meter prices at list",
|
||||
"ceilings_usd": {
|
||||
"6_2": 1.0,
|
||||
"6_6": 7.0
|
||||
},
|
||||
"cost_per_candidate_evaluate_usd": 0.003215,
|
||||
"deepseek_balance_usd": {
|
||||
"after": 17.9,
|
||||
"before": 18.2
|
||||
},
|
||||
"deepseek_tokens": {
|
||||
"completion_tokens": 155480,
|
||||
"prompt_cache_hit_tokens": 8601840,
|
||||
"prompt_tokens": 9936337
|
||||
},
|
||||
"metered_at_list_price_usd": {
|
||||
"canary": 0.010158,
|
||||
"compile": 0.429194,
|
||||
"compile_first_attempt_failed_400": 0.049047,
|
||||
"deepseek_total_6_6": 0.912838,
|
||||
"embed_6_2_openrouter": 0.003976,
|
||||
"evaluate": 0.353626,
|
||||
"smoke_deepseek": 0.070812
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -25,6 +25,7 @@ def add_commands(learn_sub: Any) -> None:
|
||||
build.add_argument("--cache", type=Path, help="clones and checkouts (default ~/.cache/openultrasast)")
|
||||
build.add_argument("--memory", help="the store (default OUSAST_MEMORY, then results_root()/memory)")
|
||||
build.add_argument("--profile", choices=("static", "plane"), default="static")
|
||||
build.add_argument("--units", type=Path, help="the harvest units.json: the pin each pair side's record is stored under")
|
||||
build.add_argument("--dry-run", action="store_true", help="count, write nothing")
|
||||
embed = memory_sub.add_parser("embed", help="embed the stored excerpts (OpenRouter, metered; cached by excerpt sha)")
|
||||
embed.add_argument("--memory", help="the store (default OUSAST_MEMORY, then results_root()/memory)")
|
||||
@@ -209,19 +210,21 @@ def _memory_embed(args: argparse.Namespace) -> int:
|
||||
def _memory_build(args: argparse.Namespace) -> int:
|
||||
"""Exit 2 on a refused label (``origin: user``, a source outside the list) or a source with >10% unread excerpts."""
|
||||
from ..plane.memory import open_store
|
||||
from .examples import ExampleBuildError, SideReader, build_examples, read_jsonl, records_by_key
|
||||
from .examples import ExampleBuildError, SideReader, build_examples, pair_pins, read_jsonl, records_by_key
|
||||
from .labels import DEFAULT_CACHE, DEFAULT_SOURCES, load_sources
|
||||
|
||||
sources = load_sources(args.sources or DEFAULT_SOURCES)
|
||||
labels = read_jsonl(args.labels)
|
||||
store = None if args.dry_run else open_store(args.memory)
|
||||
rows = [row for path in args.features for row in read_jsonl(path)]
|
||||
rows = [row for path in args.features for row in read_jsonl(path) if row.get("profile", args.profile) == args.profile]
|
||||
if store is not None:
|
||||
rows += [r.row for r in store.rows(kind="features")]
|
||||
# one profile only: the static and plane records of a candidate share a key, and the other would overwrite it
|
||||
rows += [r.row for r in store.rows(kind="features") if r.row.get("profile") == args.profile]
|
||||
reader = SideReader(args.root, args.cache or DEFAULT_CACHE, sources, labels)
|
||||
try:
|
||||
build = build_examples(
|
||||
labels, records_by_key(rows), reader, sources=sources, store=store, profile=args.profile, license_of=reader.license,
|
||||
pins=pair_pins(json.loads(args.units.read_text(encoding="utf-8"))) if args.units is not None else None,
|
||||
) # fmt: skip
|
||||
except ExampleBuildError as exc:
|
||||
print(f"learn memory build: {exc}", file=sys.stderr)
|
||||
|
||||
@@ -354,12 +354,14 @@ def compile_program(
|
||||
vectors: Mapping[str, Sequence[float]] | None = None,
|
||||
identities: Callable[[Example], Iterable[str]] | None = None,
|
||||
families: Sequence[str] = (),
|
||||
candidate_families: Sequence[str] = (),
|
||||
store: MemoryStore | None = None,
|
||||
created: str = "",
|
||||
code_commit: str = "",
|
||||
) -> CompileResult:
|
||||
"""Compile one profile's program from ``memory`` (module docstring). Deterministic given the memory, the spec and
|
||||
the response cache."""
|
||||
the response cache. ``candidate_families`` limits the bootstrap and validation candidates to those families (a
|
||||
one-family slice); retrieval still draws on every family of the compile split, for contrast examples."""
|
||||
seed = spec.seed
|
||||
pool = [e for e in memory if e.profile == profile]
|
||||
split: CompileSplit = compile_split(pool, fraction=float(spec["split"].get("compile_fraction", 0.25)),
|
||||
@@ -368,8 +370,9 @@ def compile_program(
|
||||
assert_disjoint(split, evaluation)
|
||||
compile_memory = [e for e in pool if e.group in split.groups]
|
||||
fold = Fold("compile", "compile", frozenset(), phase="compile")
|
||||
boot = _order([e for e in compile_memory if e.group in split.boot], seed, "boot")[: int(spec["bootstrap"].get("max_candidates", 150))]
|
||||
val = _order([e for e in compile_memory if e.group in split.val], seed, "val")[: int(spec["search"].get("val_candidates", 60))]
|
||||
asked = [e for e in compile_memory if not candidate_families or e.family in candidate_families]
|
||||
boot = _order([e for e in asked if e.group in split.boot], seed, "boot")[: int(spec["bootstrap"].get("max_candidates", 150))]
|
||||
val = _order([e for e in asked if e.group in split.val], seed, "val")[: int(spec["search"].get("val_candidates", 60))]
|
||||
if not boot or not val:
|
||||
raise CompileError(f"compile split too small: {len(boot)} bootstrap and {len(val)} validation candidates")
|
||||
retrieval, classify = spec["retrieval"], spec["classify"]
|
||||
|
||||
@@ -29,7 +29,9 @@ val_candidates = 60
|
||||
|
||||
[classify]
|
||||
model = "deepseek-flash"
|
||||
k = 1 # k and lam are A/B variants (design section 6), not searched here
|
||||
k = 5 # fixed by the smoke run before any compile spend (task 6.6): samples 2..5 hit the
|
||||
# provider prefix cache for 0.96 of their prompt tokens (>= 0.8), 2026-10-01. k and lam
|
||||
# remain A/B variants (design section 6), not searched here
|
||||
max_output_tokens = 300
|
||||
inputs = "v1" # the input profile: the engine instrument is withheld from prompts and retrieval (2026-10-01)
|
||||
|
||||
|
||||
@@ -22,7 +22,9 @@ Memory examples carry **no rationale** (code, signals, roles, family and label o
|
||||
``fix_function`` (a different handler of the same snapshot).
|
||||
|
||||
Pair labels have no commit pin (the corpus holds excerpts); their envelope pin is :func:`label_pin`, a stable
|
||||
40-hex digest of the pair and its side, so the vulnerable and the fixed side stay distinct rows.
|
||||
40-hex digest of the pair and its side, so the vulnerable and the fixed side stay distinct rows. The harvest keyed a
|
||||
pair side's feature record by its unit's pin instead (the blob sha1 of the side's excerpt, ``plane/harvest.py``):
|
||||
:func:`pair_pins` reads that mapping from the harvest ``units.json`` and the build uses it when given.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -62,6 +64,11 @@ def label_pin(label: Mapping[str, Any]) -> str:
|
||||
return hashlib.sha1(subject.encode("utf-8")).hexdigest() # noqa: S324 -- a name, not a security digest
|
||||
|
||||
|
||||
def pair_pins(units: Mapping[str, Mapping[str, Any]]) -> dict[tuple[str, str], str]:
|
||||
"""Harvest ``units.json`` -> ``(case reference, side) -> pin`` for the pair units (the pin their records carry)."""
|
||||
return {(str(u["ref"]), str(u["side"])): str(u["pin"]) for u in units.values() if u.get("source") == "pairs"}
|
||||
|
||||
|
||||
def record_key(repo: str, pin: str, candidate: str, family: str, unit: str) -> tuple[str, str, str, str, str]:
|
||||
return repo_key(repo), pin, candidate, family, unit
|
||||
|
||||
@@ -127,9 +134,11 @@ def build_examples(
|
||||
license_of: Callable[[Mapping[str, Any]], str] | None = None,
|
||||
function_of: Callable[[Mapping[str, Any], str], str | None] | None = None,
|
||||
bounds: Bounds = EXAMPLE,
|
||||
pins: Mapping[tuple[str, str], str] | None = None,
|
||||
) -> MemoryBuild:
|
||||
"""Example rows for the verified labels that have a feature record (``records`` by :func:`record_key`) and an
|
||||
excerpt; with ``store`` the excerpt blobs and the rows are written. Delta units append the base->head diff."""
|
||||
excerpt; with ``store`` the excerpt blobs and the rows are written. Delta units append the base->head diff.
|
||||
``pins`` (:func:`pair_pins`) gives a pair side the pin its harvest record was stored under."""
|
||||
rows = [_as_dict(label) for label in labels]
|
||||
allowed = {s.id for s in sources.sources} | {d for d, kind in DERIVED_SOURCES.items() if any(s.kind == kind for s in sources.sources)}
|
||||
for row in rows:
|
||||
@@ -145,7 +154,9 @@ def build_examples(
|
||||
continue
|
||||
count = build.sources.setdefault(str(label["source"]), SourceCount())
|
||||
count.labels += 1
|
||||
pin = label_pin(label)
|
||||
side = (str(label.get("source_ref")), str(label.get("pin_role") or "vulnerable"))
|
||||
pin = (pins or {}).get(side) if label.get("source") == "pairs" else None
|
||||
pin = pin or label_pin(label)
|
||||
candidate, family, unit = str(label["candidate"]), str(label["family"]), str(label["unit"])
|
||||
record = records.get(record_key(str(label["repo"]), pin, candidate, family, unit))
|
||||
if record is None or record.get("profile") != profile:
|
||||
@@ -307,5 +318,5 @@ def records_by_key(rows: Iterable[Mapping[str, Any]]) -> dict[tuple[str, str, st
|
||||
|
||||
__all__ = [
|
||||
"EXAMPLE_KIND", "NO_EXCERPT_LIMIT", "Example", "ExampleBuildError", "MemoryBuild", "SideReader", "build_examples",
|
||||
"label_pin", "label_set_digest", "load_examples", "read_jsonl", "record_key", "records_by_key",
|
||||
"label_pin", "label_set_digest", "load_examples", "pair_pins", "read_jsonl", "record_key", "records_by_key",
|
||||
] # fmt: skip
|
||||
|
||||
@@ -87,16 +87,21 @@ SIGNATURE = Signature(
|
||||
)
|
||||
)
|
||||
|
||||
BASELINE_INSTRUCTION = (
|
||||
"You decide whether a function contains an exploitable security vulnerability of the given family.\n"
|
||||
"You see the function's code (numbered lines), the signals static instruments emitted for it, and the roles\n"
|
||||
"inferred for its calls. Signals are evidence, not verdicts: rules over-report, and a missing instrument is not a\n"
|
||||
"negative. Similar labelled functions from other repositories follow as examples; reason about this code.\n"
|
||||
# The answer contract is part of the signature, not of the searched instruction: a proposed instruction that leaves it
|
||||
# out still gets it (``render_prefix``), so every prompt asks for JSON (DeepSeek's JSON mode refuses a prompt that
|
||||
# never says "json") and every answer can parse.
|
||||
ANSWER_FORMAT = (
|
||||
'Answer with one JSON object: {"verdict": "vulnerable" | "not_vulnerable" | "unsure", "family": <family or\n'
|
||||
'"none">, "confidence": <probability in [0, 1] that your verdict is right>, "rationale": <at most 3 sentences>,\n'
|
||||
'"cited_lines": [<line numbers of the code that support the verdict>]}. Cite at least one line. Say "unsure"\n'
|
||||
"when the code does not decide it."
|
||||
)
|
||||
BASELINE_INSTRUCTION = (
|
||||
"You decide whether a function contains an exploitable security vulnerability of the given family.\n"
|
||||
"You see the function's code (numbered lines), the signals static instruments emitted for it, and the roles\n"
|
||||
"inferred for its calls. Signals are evidence, not verdicts: rules over-report, and a missing instrument is not a\n"
|
||||
"negative. Similar labelled functions from other repositories follow as examples; reason about this code.\n" + ANSWER_FORMAT
|
||||
)
|
||||
|
||||
|
||||
# --- rendering ---------------------------------------------------------------------------------------------------------
|
||||
@@ -232,7 +237,8 @@ ExcerptText = Callable[[str], str | None]
|
||||
|
||||
def render_prefix(spec: ProgramSpec, demos: Sequence[Demo], examples: Mapping[str, Example], excerpt_text: ExcerptText) -> str:
|
||||
"""The instruction and the demonstrations: identical for every candidate of one program and fold."""
|
||||
parts = [spec.instruction.strip()]
|
||||
instruction = spec.instruction.strip()
|
||||
parts = [instruction if ANSWER_FORMAT in instruction else f"{instruction}\n{ANSWER_FORMAT}"]
|
||||
for number, demo in enumerate(demos, 1):
|
||||
example = examples[demo.example_id]
|
||||
code = excerpt_text(demo.excerpt_sha) or ""
|
||||
@@ -481,7 +487,8 @@ class Program:
|
||||
|
||||
|
||||
__all__ = [
|
||||
"BASELINE_INSTRUCTION", "PARSE_FAILED", "SIGNATURE", "UNSURE", "VERDICTS", "Answer", "Caller", "Candidate", "Decision", "Demo",
|
||||
"ANSWER_FORMAT", "BASELINE_INSTRUCTION", "PARSE_FAILED", "SIGNATURE", "UNSURE", "VERDICTS", "Answer", "Caller", "Candidate",
|
||||
"Decision", "Demo",
|
||||
"Program", "ProgramSpec", "Prompt", "ReplayMiss", "aggregate", "classify", "label_word", "line_numbers",
|
||||
"params_digest", "parse_answer", "render", "render_prefix", "render_roles", "render_signals", "request_key",
|
||||
] # fmt: skip
|
||||
|
||||
@@ -11,12 +11,13 @@ profile, computed per instrument block so that missing is explicit, never impute
|
||||
- ``D`` is the uniform mean over the profile's blocks; prior features are dropped with ``priors = "off"``, and the blocks
|
||||
of instruments the input profile withholds (``inputs``, default ``v1``: the engine) are left out entirely.
|
||||
|
||||
Ties break by ``sha256(example id + seed)``; the first ``n1 = 40`` go to stage 2, a cosine re-rank of the code
|
||||
Ties break by ``sha256(example id + seed)``; the first ``n1 = 40`` of the candidate's family and the first ``n1``
|
||||
of the other families go to stage 2, a cosine re-rank of the code
|
||||
embeddings, ``r = lam (1 - D) + (1 - lam) cos``. Without the target's vector the whole retrieval is
|
||||
``signals_only``; an example without a vector scores ``cos = 1 - D`` (its signal similarity, so it keeps its
|
||||
stage-1 standing). The **balance rule** then takes ``k_ret = 6``: at most ``k/2`` per label, at most 2 per
|
||||
repository group, at most 2 from other families while the candidate's family has examples left; ``short`` when
|
||||
the pool cannot fill it.
|
||||
repository group, at most 2 contrast examples from other families whenever the pool holds the candidate's family
|
||||
(never padded with more); ``short`` when the pool cannot fill it.
|
||||
|
||||
**The boundary.** :func:`eligible` is the only way an example enters a prompt -- retrieved examples and compiled
|
||||
demonstrations alike: not the target's group, not a group the fold evaluates, not the held-out source, not the
|
||||
@@ -43,6 +44,7 @@ LAM = 0.5
|
||||
NEAR_DUPLICATE_COS = 0.98
|
||||
UNBOUNDED_CAP = 10.0
|
||||
PER_GROUP = 2
|
||||
CONTRAST = 2 # contrast examples from other families, at most, while the pool holds the target's family
|
||||
|
||||
|
||||
class BoundaryViolation(AssertionError):
|
||||
@@ -153,35 +155,26 @@ def _tie(example_id: str, seed: int) -> str:
|
||||
|
||||
|
||||
def balance(ranked: Sequence[Example], family: str, k: int = K_RET) -> tuple[list[Example], str]:
|
||||
"""Take up to ``k`` in rank order: <= k/2 per label, <= 2 per group, <= 2 other families while the target's family
|
||||
has examples left; then, if short, fill from other families under the label and group caps."""
|
||||
per_label, other_cap = k // 2, min(2, k)
|
||||
"""Take up to ``k`` in rank order: <= k/2 per label, <= 2 per group, and at most 2 contrast examples from other
|
||||
families whenever the pool holds any example of the target's family (maintainer, 2026-10-01: "allow 2 contrast").
|
||||
A pool without the target's family takes from the others under the label and group caps. ``short`` when the
|
||||
pool cannot fill ``k`` or a label minimum: what exists is taken, never padded with more contrast."""
|
||||
per_label = k // 2
|
||||
other_cap = min(CONTRAST, k) if any(e.family == family for e in ranked) else k
|
||||
taken: list[Example] = []
|
||||
labels: dict[int, int] = {}
|
||||
groups: dict[str, int] = {}
|
||||
|
||||
def fits(example: Example) -> bool:
|
||||
return labels.get(example.label, 0) < per_label and groups.get(example.group, 0) < PER_GROUP
|
||||
|
||||
def take(example: Example) -> None:
|
||||
taken.append(example)
|
||||
labels[example.label] = labels.get(example.label, 0) + 1
|
||||
groups[example.group] = groups.get(example.group, 0) + 1
|
||||
|
||||
others = 0
|
||||
for example in ranked:
|
||||
if len(taken) == k:
|
||||
break
|
||||
if example.family != family and others >= other_cap:
|
||||
continue
|
||||
if fits(example):
|
||||
if labels.get(example.label, 0) < per_label and groups.get(example.group, 0) < PER_GROUP:
|
||||
others += example.family != family
|
||||
take(example)
|
||||
for example in ranked:
|
||||
if len(taken) == k:
|
||||
break
|
||||
if example.id not in {e.id for e in taken} and fits(example):
|
||||
take(example)
|
||||
taken.append(example)
|
||||
labels[example.label] = labels.get(example.label, 0) + 1
|
||||
groups[example.group] = groups.get(example.group, 0) + 1
|
||||
pool_labels = {e.label for e in ranked}
|
||||
short = len(taken) < k or any(labels.get(lab, 0) < min(2, per_label) for lab in (0, 1) if lab in pool_labels)
|
||||
return taken, "short" if short else "ok"
|
||||
@@ -202,10 +195,14 @@ def retrieve(
|
||||
) -> Retrieval:
|
||||
"""The ``k`` balanced examples for ``target`` from the eligible part of ``pool`` (module docstring)."""
|
||||
candidates = [e for e in pool if e.profile == target.profile and eligible(e, target, fold)]
|
||||
scored = sorted(
|
||||
ranked_all = sorted(
|
||||
((gower(target.x, target.instruments, e.x, e.instruments, target.profile, priors=priors, inputs=inputs), e) for e in candidates),
|
||||
key=lambda pair: (pair[0], _tie(pair[1].id, seed)),
|
||||
)[:n1]
|
||||
)
|
||||
# stage 1 per side: the n1 nearest of the target's family and the n1 nearest contrast examples, so a family that is a
|
||||
# minority of the memory still reaches stage 2 (the balance rule then takes at most 2 contrast examples)
|
||||
own = [pair for pair in ranked_all if pair[1].family == target.family][:n1]
|
||||
scored = own + [pair for pair in ranked_all if pair[1].family != target.family][:n1]
|
||||
mode = "signals+embeddings" if target.vector is not None and vectors else "signals_only"
|
||||
near = without = 0
|
||||
reranked: list[tuple[float, str, Example]] = []
|
||||
|
||||
@@ -6,7 +6,7 @@ byte for byte at $0 (``test_reproducible_from_cache``); the permissive-licence f
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections.abc import Iterator, Mapping, Sequence
|
||||
from collections.abc import Iterable, Iterator, Mapping, Sequence
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
@@ -30,6 +30,7 @@ from openultrasast.learn.compile import (
|
||||
permissive,
|
||||
program_id,
|
||||
)
|
||||
from openultrasast.learn.examples import Example
|
||||
from openultrasast.learn.folds import Fold, FoldError
|
||||
from openultrasast.learn.program import Caller, Demo
|
||||
from openultrasast.plane.budget import MeteredClient
|
||||
@@ -191,3 +192,23 @@ def test_only_permissively_licensed_demonstrations_are_packaged() -> None:
|
||||
assert [d["example_id"] for d in packaged["demos"]] == [mit.id, apache.id]
|
||||
assert packaged["program_id"] == program_id(packaged) != artifact["program_id"]
|
||||
assert packaged["packaged"]["from"] == artifact["program_id"]
|
||||
|
||||
|
||||
def test_a_one_family_compile_asks_only_that_family(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""``candidate_families`` limits the bootstrap and validation candidates; the compile memory keeps every family
|
||||
(retrieval's contrast examples), so the artifact's data counts are those of the whole pool."""
|
||||
asked: dict[str, set[str]] = {}
|
||||
real = compile_module._order
|
||||
|
||||
def spy(items: Iterable[Example], seed: int, salt: str) -> list[Example]:
|
||||
items = list(items)
|
||||
asked.setdefault(salt, set()).update(e.family for e in items)
|
||||
return real(items, seed, salt)
|
||||
|
||||
monkeypatch.setattr(compile_module, "_order", spy)
|
||||
result = compile_program(MEMORY, _caller(ScriptedChat(answer)), excerpt_text, spec=SPEC, candidate_families=("injection",))
|
||||
assert result.status == "done" and result.artifact is not None
|
||||
assert asked["boot"] == asked["val"] == {"injection"}
|
||||
assert result.artifact["data"]["examples"] == len(MEMORY)
|
||||
with pytest.raises(CompileError, match="compile split too small"):
|
||||
compile_program(MEMORY, _caller(ScriptedChat(answer)), excerpt_text, spec=SPEC, candidate_families=("memory",))
|
||||
|
||||
@@ -18,6 +18,7 @@ from openultrasast.learn.examples import (
|
||||
build_examples,
|
||||
label_pin,
|
||||
load_examples,
|
||||
pair_pins,
|
||||
record_key,
|
||||
records_by_key,
|
||||
)
|
||||
@@ -155,3 +156,18 @@ def test_a_fixed_side_without_the_function_is_a_label_defect_not_an_unread_side(
|
||||
assert count.absent_fixed_side == 1 and count.no_excerpt == 0 and build.failing() == []
|
||||
unread = build_examples([fixed], records_by_key([feature(fixed)]), lambda lab, role: None, sources=SOURCES)
|
||||
assert unread.sources["population-v1"].unread == 1 and unread.failing() == ["population-v1"]
|
||||
|
||||
|
||||
def test_pair_sides_join_the_record_stored_under_their_harvest_unit_pin() -> None:
|
||||
"""The harvest stores a pair side's record under its unit pin (the blob sha1 of the side's excerpt), not under
|
||||
:func:`label_pin`; ``pins`` from the harvest ``units.json`` joins them, and without it the record is not found."""
|
||||
lab = label(source="pairs", source_ref="case-9", pin="")
|
||||
unit_pin = "a" * 40
|
||||
stored = {**feature(lab), "pin": unit_pin}
|
||||
units = {"h-1": {"source": "pairs", "ref": "case-9", "side": "vulnerable", "pin": unit_pin, "repo": lab["repo"]},
|
||||
"h-2": {"source": "population-v1", "ref": "case-9", "side": "vulnerable", "pin": PIN, "repo": lab["repo"]}} # fmt: skip
|
||||
pins = pair_pins(units)
|
||||
assert pins == {("case-9", "vulnerable"): unit_pin}
|
||||
assert build_examples([lab], records_by_key([stored]), lines_of, sources=SOURCES).summary()["examples"] == 0
|
||||
build = build_examples([lab], records_by_key([stored]), lines_of, sources=SOURCES, pins=pins)
|
||||
assert build.summary()["examples"] == 1 and build.rows[0]["pin"] == unit_pin
|
||||
|
||||
@@ -15,6 +15,8 @@ from learn_fixtures import ScriptedChat, by_id, corpus, x_record
|
||||
from openultrasast.learn import program as program_module
|
||||
from openultrasast.learn.folds import outer_folds
|
||||
from openultrasast.learn.program import (
|
||||
ANSWER_FORMAT,
|
||||
BASELINE_INSTRUCTION,
|
||||
PARSE_FAILED,
|
||||
Answer,
|
||||
Caller,
|
||||
@@ -121,6 +123,15 @@ def test_the_prefix_is_byte_identical_across_candidates() -> None:
|
||||
assert "Demonstration 1:" in a.prefix and a.messages[1] != b.messages[1]
|
||||
|
||||
|
||||
def test_every_instruction_carries_the_answer_contract() -> None:
|
||||
"""A proposed instruction rarely restates the JSON contract; the prefix adds it, once, so JSON mode is accepted
|
||||
(DeepSeek refuses a prompt without the word "json") and the answer parses. The baseline prefix is unchanged."""
|
||||
proposed, _ = program(instruction="Weigh the sanitiser before the sink.").prepare(candidate(), FOLD)
|
||||
baseline, _ = program().prepare(candidate(), FOLD)
|
||||
assert proposed.prefix.startswith("Weigh the sanitiser before the sink.\n") and proposed.prefix.count(ANSWER_FORMAT) == 1
|
||||
assert baseline.prefix.count(ANSWER_FORMAT) == 1 and baseline.prefix.startswith(BASELINE_INSTRUCTION)
|
||||
|
||||
|
||||
def test_parsing_valid_invalid_and_out_of_range_citations() -> None:
|
||||
lines = frozenset({40, 41})
|
||||
ok = parse_answer(
|
||||
|
||||
@@ -8,7 +8,7 @@ from dataclasses import replace
|
||||
import pytest
|
||||
from learn_fixtures import by_id, corpus, example, x_record
|
||||
|
||||
from openultrasast.learn.folds import compile_split, leave_one_framework_out, leave_one_source_out, outer_folds
|
||||
from openultrasast.learn.folds import Fold, compile_split, leave_one_framework_out, leave_one_source_out, outer_folds
|
||||
from openultrasast.learn.retrieve import (
|
||||
BoundaryViolation,
|
||||
Target,
|
||||
@@ -102,6 +102,25 @@ def test_balance_caps_labels_groups_and_other_families() -> None:
|
||||
crowded = [example(f"c{i}", group="one/repo", label=i % 2) for i in range(10)]
|
||||
taken, state = balance(crowded, "injection", 6)
|
||||
assert len(taken) == 2 and state == "short" # at most 2 per repository group
|
||||
scarce = [example("own", group="own/repo", label=1)] + [
|
||||
example(f"o{i}", group=f"o{i}/r", label=i % 2, family="path") for i in range(10)
|
||||
]
|
||||
taken, state = balance(scarce, "injection", 6)
|
||||
assert sum(e.family != "injection" for e in taken) == 2 and len(taken) == 3 and state == "short" # never padded
|
||||
absent = [example(f"a{i}", group=f"a{i}/r", label=i % 2, family="path") for i in range(10)]
|
||||
taken, state = balance(absent, "injection", 6)
|
||||
assert len(taken) == 6 and state == "ok" # no example of the family at all: the others fill under the caps
|
||||
|
||||
|
||||
def test_a_minority_family_reaches_stage_two_and_contrast_stays_at_two() -> None:
|
||||
"""Stage 1 keeps the n1 nearest of the target's family beside the n1 nearest others: 50 contrast examples nearer
|
||||
in signals cannot push the family out, and the balance rule takes at most 2 of them."""
|
||||
others = [example(f"o{i}", group=f"o{i}/r", label=i % 2, family="path", hits=0) for i in range(50)]
|
||||
own = [example(f"own{i}", group=f"own{i}/r", label=i % 2, hits=6) for i in range(4)]
|
||||
fold = Fold("compile", "compile", frozenset(), phase="compile")
|
||||
got = retrieve(_target(hits=0), [*others, *own], fold, n1=40, k=6)
|
||||
assert sum(e.family == "injection" for e in got.examples) == 4
|
||||
assert sum(e.family != "injection" for e in got.examples) == 2
|
||||
|
||||
|
||||
def test_near_duplicates_drop_in_evaluation_and_stay_in_deployment() -> None:
|
||||
|
||||
Reference in New Issue
Block a user