mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
feat(learn): the classifier program and its response cache (decision-engine 6.4)
learn/program.py: the declared signature (code, signals, roles, family in; verdict, family, confidence, rationale with cited lines out), the baseline instruction, rendering as [system: instruction + demonstrations] -- byte-identical for every candidate, so the provider's prefix cache can hit -- then [user: retrieved examples with their labels, the candidate without one]. Signals pass validate_x first (no EXCLUDED_FIELDS, missing instruments in words). Answers are one JSON object; one that does not parse or cites no line inside the excerpt is asked once more at temperature 0 and otherwise becomes unsure/parse_failed, never not_vulnerable. Classify(k): sample 1 at 0.0, the rest at 0.7; score s = mean p_j; majority verdict, ties unsure; a majority unsure cannot BLOCK. Caller caches each response as responses/<sha>.json keyed by (model, Model parameters digest, messages, temperature, sample, json_object) and replays at $0; spend is priced from the recorded usage once per distinct request, so a replay reports the same usd. Program.decide asserts the evaluation boundary on the rendered ids before any call. MeteredClient.complete passes temperature/logprobs through (only when set); DeepSeekChatClient sends temperature beside thinking=disabled. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
f5180a14ec
commit
ea61dc1bb3
@@ -61,7 +61,7 @@ a model or embedding call.
|
||||
- Tests: `test_learn_retrieve.py` including `test_boundary` (every rendered example and demonstration eligible,
|
||||
for all fold kinds; `memory.seed` excludes the case's group), `test_learn_folds.py`.
|
||||
- _Requirements: 3.2, 3.4, 7.1, 8.3_
|
||||
- [ ] 6.4 The program (no model in tests)
|
||||
- [x] 6.4 The program (no model in tests)
|
||||
- `learn/program.py`: the signature, prompt rendering (instruction + demonstrations as a byte-identical prefix,
|
||||
then retrieved examples, then the candidate), JSON parsing with one retry -> `unsure, parse_failed`,
|
||||
`Classify(k)` with temperature 0 then 0.7, the score `s` and majority verdict; `responses/<sha>.json` cache
|
||||
|
||||
Reference in New Issue
Block a user