mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
spec(learned-decision-engine): design and tasks revised for the AI classifier
In-house DSPy-style program (retrieve -> classify(k)) on the metered client; example store with signal-profile Gower neighbours re-ranked by code embeddings; balanced Brier metric, bootstrap few-shot and instruction search on a compile split; Platt calibration on other folds; cached replay; tasks 6.1-6.7 with paid steps flagged for go-ahead. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
2d69a8b677
commit
b6a49fa4dc
File diff suppressed because it is too large
Load Diff
@@ -1,9 +1,9 @@
|
||||
{
|
||||
"feature_name": "learned-decision-engine",
|
||||
"created_at": "2026-09-30T14:00:00Z",
|
||||
"updated_at": "2026-10-01T09:00:00Z",
|
||||
"updated_at": "2026-10-01T10:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "requirements-amended",
|
||||
"phase": "design-generated",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
|
||||
@@ -1,9 +1,12 @@
|
||||
# Implementation Plan
|
||||
|
||||
Order follows the design's commit sequence. Every task is a commit gated on the full suite's own exit code,
|
||||
`ruff check`, `ruff format --check` and `mypy`. No reported number changes until task 9 adopts a model; without an
|
||||
adopted model every path is today's. Tasks 5, 7 and 10 spend model money or host hours: each starts only after
|
||||
the maintainer confirms its budget.
|
||||
`ruff check`, `ruff format --check` and `mypy`. No reported number changes until task 9 adopts a compiled program;
|
||||
without an adopted program every path is today's. Requirement 3 changed on 2026-10-01 (an AI classifier with local
|
||||
memory, compiled DSPy-style in-house); tasks 1-5 are unaffected, tasks 6-10 are re-planned for it. Sub-tasks marked
|
||||
**(budget go-ahead)** spend model money or host hours: each starts only after the maintainer confirms its budget
|
||||
against the provider balance read just before it, and none starts with a ceiling above that balance. No test makes
|
||||
a model or embedding call.
|
||||
|
||||
- [x] 1. Schema and features
|
||||
- `learn/schema.py` (closed, typed allow-list; no identities; missing != zero), `learn/features.py` (host
|
||||
@@ -28,35 +31,96 @@ the maintainer confirms its budget.
|
||||
commits (separate, never merged); a first label snapshot committed as counts only.
|
||||
- _Requirements: 2.1, 2.2, 2.3, 7.1_
|
||||
|
||||
- [ ] 5. Harvest (needs budget go-ahead)
|
||||
- [ ] 5. Harvest (in progress)
|
||||
- A plane Run of verify a/b on the verify-family pairs and development cases (~$6, ceiling $10), model roles on
|
||||
the candidate files (ceiling $10), engine features for the pair corpus on the host (frozen source, one
|
||||
container at a time, background).
|
||||
- _Requirements: 1.1_
|
||||
|
||||
- [ ] 6. Train, calibrate, evaluate
|
||||
- `learn/{folds,train,calibrate,evaluate}.py` (repository-grouped, leave-one-source-out, leave-one-framework-out,
|
||||
nested tuning), the model blob API in `plane/memory.py`, `ousast learn train|evaluate|curve`, the first
|
||||
evaluation report with priors off under `benchmarks/measurements/<date>-decision-engine-v1/`.
|
||||
- _Requirements: 3.1-3.5, 5.1, 5.2, 7.3, 7.4, 8.3_
|
||||
- [ ] 6. Program, local memory and compilation (design sections 4-5)
|
||||
- [ ] 6.1 Excerpts and the example store (no model)
|
||||
- `learn/excerpt.py` (function span at the label's pin, bounds 80/4,000 for candidates and 40/2,000 for
|
||||
examples, numbered lines, no path/repository/commit, advisory ids and security wording in comments
|
||||
redacted, delta diff); `learn/examples.py` and `ousast learn memory build` (label + feature record +
|
||||
excerpt -> `example` row; refuses `origin: user` and sources outside `sources.toml`; exits 2 when more than
|
||||
10% of a source has no excerpt); `example` added to `KINDS`; `put_blob`/`get_blob` on both backends.
|
||||
- Tests: `test_learn_excerpt.py`, example-builder refusals, blob round trip on FileStore (MinIO gated).
|
||||
- _Requirements: 3.2, 3.6, 7.2_
|
||||
- [ ] 6.2 Metered embeddings and the cache
|
||||
- `plane/models/openrouter-embedding.yaml` (`input_per_m: 0.02`), `MeteredEmbeddingClient` in
|
||||
`plane/budget.py`, usage kept by `parse_embedding_response`; cache `embeddings/<model>/<excerpt_sha>.json`;
|
||||
`ousast learn memory embed`; scripted-client tests.
|
||||
- **(budget go-ahead)** embedding the store: ~5.0 M tokens, ~$0.10, ceiling $1 (OpenRouter account).
|
||||
- _Requirements: 3.2_
|
||||
- [ ] 6.3 Retrieval and the evaluation boundary (no model)
|
||||
- `learn/retrieve.py`: per-instrument Gower distance with explicit missing, `n1 = 40`, embedding re-rank
|
||||
`r = lam(1 - D) + (1 - lam)cos`, `signals_only` mode, balance rule (`k_ret = 6`), `eligible(example, target,
|
||||
fold)`, near-duplicate drop in evaluation, deployment self-exclusion.
|
||||
- `learn/folds.py`: compile split (25% of groups), outer grouped 5-fold, leave-one-source-out,
|
||||
leave-one-framework-out.
|
||||
- Tests: `test_learn_retrieve.py` including `test_boundary` (every rendered example and demonstration eligible,
|
||||
for all fold kinds; `memory.seed` excludes the case's group), `test_learn_folds.py`.
|
||||
- _Requirements: 3.2, 3.4, 7.1, 8.3_
|
||||
- [ ] 6.4 The program (no model in tests)
|
||||
- `learn/program.py`: the signature, prompt rendering (instruction + demonstrations as a byte-identical prefix,
|
||||
then retrieved examples, then the candidate), JSON parsing with one retry -> `unsure, parse_failed`,
|
||||
`Classify(k)` with temperature 0 then 0.7, the score `s` and majority verdict; `responses/<sha>.json` cache
|
||||
and replay; `temperature`/`logprobs` pass-through in `MeteredClient.complete` and `DeepSeekChatClient._call`.
|
||||
- Tests: `test_learn_program.py` (no `EXCLUDED_FIELDS` in prompts, prefix identity, parsing, `unsure` cannot
|
||||
BLOCK, budget ceiling, replay).
|
||||
- _Requirements: 3.1, 3.3_
|
||||
- [ ] 6.5 Compile, calibrate, evaluate (no model in tests)
|
||||
- `learn/compile.toml` (pre-registered metric: balanced Brier score; seeds; budget), `learn/compile.py`
|
||||
(bootstrap few-shot on `C_boot`, 6 proposed + baseline instructions scored on `C_val`, 4 demonstration sets,
|
||||
alternates for leave-one-framework-out, artifact `programs/<sha>.json`), `learn/calibrate.py` (cross-fitted
|
||||
Platt, reliability/slope/intercept/ECE, ADVISORY-only downgrade), `learn/evaluate.py` (operating points from
|
||||
the other folds, repository-cluster bootstrap, strata, "insufficient data"), `ousast learn
|
||||
compile|evaluate|curve`.
|
||||
- Tests: `test_learn_compile.py` (`test_reproducible_from_cache` on both backends), `test_learn_calibrate.py`,
|
||||
`test_learn_decide.py`.
|
||||
- _Requirements: 3.3, 3.5, 5.2, 7.3, 7.4_
|
||||
- [ ] 6.6 First paid slice **(budget go-ahead: ~$5 expected, ceiling $7)**
|
||||
- Smoke run on 20 candidates (~$0.10): real token counts, prefix-cache hit share, parse rate; it fixes `k`
|
||||
(5 if samples 2..k hit the cache for >= 80% of their prompt tokens, else 1) before anything else is spent.
|
||||
- One compile per profile with data (`static` first; ~$2.2, ceiling $3).
|
||||
- Outer-fold evaluation, priors off, evaluable families only (675 candidates: ~$2.0 at k = 1, ~$3.0 at k = 5),
|
||||
with the 30-candidate canary set; report under `benchmarks/measurements/<date>-decision-engine-v1/` (counts
|
||||
and intervals only, no identities).
|
||||
- _Requirements: 3.1-3.5, 5.2, 7.3, 7.4_
|
||||
- [ ] 6.7 Second paid slice **(budget go-ahead: ~$7 expected, ceiling $10; needs a top-up)**
|
||||
- Leave-one-source-out (~$1.4-2.1), leave-one-framework-out where a framework reaches 10 groups ($0 today),
|
||||
learning curves over memory size (0/25/50/100%) and number of sources (~$5.2), recorded beside 6.6.
|
||||
- _Requirements: 5.1, 5.2, 8.3_
|
||||
|
||||
- [ ] 7. Experiments (exp-001 needs budget go-ahead)
|
||||
- [ ] 7. Experiments (exp-001 and program variants need budget go-ahead)
|
||||
- `learn/experiments.py`, `register`, `experiment-outcome`, Run annotation checks, `OUSAST_VERIFY_CALLERS`;
|
||||
exp-001 (known-callers line on/off, ~$7.7 expected, ceiling $15) registered, run and recorded -- an
|
||||
inconclusive result is pre-registered as likely.
|
||||
- _Requirements: 4.1-4.4_
|
||||
- Program-variant experiments (paired on candidates, repository cluster, arm A replayed from cache at $0): the
|
||||
machinery and its test; the first variant (`k = 5` vs `k = 1`, or the confidence source) **(budget go-ahead:
|
||||
~$0.9-1.35 per arm-B pass of 300 candidates, ceiling $2 each)**.
|
||||
- _Requirements: 3.6, 4.1-4.4_
|
||||
|
||||
- [ ] 8. Integration
|
||||
- The `decide` task, the `scan` decision stage and `dropped_by_engine.json`, `pre-push` admission with the engine
|
||||
(structural admission reasons unchanged), report fields, the fallback line, the opt-in adaptation layer.
|
||||
- `plane/tasks/embed.py` (Model `openrouter-embedding`) and `plane/tasks/decide.py` (Model `deepseek-flash`,
|
||||
metered, exits 0/2/3), the host seed of the adopted program and the case's boundary-filtered memory; the `scan`
|
||||
decision stage and `dropped_by_engine.json`; `pre-push` admission with the engine (structural admission reasons
|
||||
unchanged; concurrency, `undecided: deadline`, a per-push budget); report fields (p, operating point,
|
||||
program id, verdict, rationale, cited lines); the fallback line (no program, no key, budget exhausted,
|
||||
insufficient data, calibration failed); the opt-in adaptation layer with user-local memory.
|
||||
- Tests use scripted clients; a plane fixture Run proves `decide` never receives an example of its own case.
|
||||
- _Requirements: 6.1-6.3, 7.3, 7.5_
|
||||
|
||||
- [ ] 9. Adopt
|
||||
- `ousast learn adopt`, packaged models for the families that pass, the priors-on/off experiment per framework
|
||||
with a recorded decision.
|
||||
- _Requirements: 3.2, 8.2_
|
||||
- `ousast learn adopt`; the packaged `ruleset/decision/program-<profile>.json` with only licence-cleared
|
||||
demonstrations (the maintainer decides which licences qualify), re-evaluated as its own variant; the canary
|
||||
drift check before adoption; the priors-on/off experiment per framework with a recorded decision
|
||||
**(budget go-ahead per experiment, ~$1-2 each)**.
|
||||
- _Requirements: 3.3, 8.2_
|
||||
|
||||
- [ ] 10. The one-time v3 check
|
||||
- Commit `prediction-v3.json` first; then v3 is run under protocol v3 and `result-v3-engine.json` is committed.
|
||||
Nothing between the two commits may touch the engine, features, priors or thresholds.
|
||||
- [ ] 10. The one-time v3 check **(budget go-ahead for the engine's v3 spend, ~$0.0045 per candidate)**
|
||||
- Commit `prediction-v3.json` first (program id, memory snapshot digest, model ids, operating points, predicted
|
||||
intervals, spend ceiling from the owner's candidate count); then v3 is run under protocol v3 and
|
||||
`result-v3-engine.json` is committed. Nothing between the two commits may touch the program, memory snapshot,
|
||||
features, priors or thresholds. This spec's code never opens v3 files.
|
||||
- _Requirements: 5.3_
|
||||
|
||||
Reference in New Issue
Block a user