Open_Ultra_SAST/.kiro/specs/learned-decision-engine/tasks.md

# Implementation Plan

Order follows the design's commit sequence. Every task is a commit gated on the full suite's own exit code,
`ruff check`, `ruff format --check` and `mypy`. No reported number changes until task 9 adopts a compiled program;
without an adopted program every path is today's. Requirement 3 changed on 2026-10-01 (an AI classifier with local
memory, compiled DSPy-style in-house); tasks 1-5 are unaffected, tasks 6-10 are re-planned for it. Sub-tasks marked
**(budget go-ahead)** spend model money or host hours: each starts only after the maintainer confirms its budget
against the provider balance read just before it, and none starts with a ceiling above that balance. No test makes
a model or embedding call.

Direction 2026-10-04: `.kiro/steering/roadmap.md` (section of 2026-10-04, phases G0-G5) is the authority. Every
change is judged on the unseen-repository pool (G0, spec `unseen-repo-evaluation`) by catch rate and false-alarm
rate per push. Within-pair AUC stays a development signal. Tasks 11-14 below follow that roadmap.

- [x] 1. Schema and features
  - `learn/schema.py` (closed, typed allow-list; no identities; missing != zero), `learn/features.py` (host
    builder), new memory `KINDS`, `plane/tasks/features.py` and the `remember` input; tests enforcing the
    allow-list and the exclusion of label-carrying fields.
  - _Requirements: 1.1, 1.2, 1.3, 7.2_

- [x] 2. Framework tags
  - `ruleset/frameworks.toml`; tag and split every framework/library entry listed in design section 2; `priors=`
    in the semantic and quick loaders (default `all`, so today's scan is unchanged -- the golden benchmark test
    stays byte-identical); a lint test that no untagged framework entry remains.
  - _Requirements: 8.2, 8.4_

- [x] 3. Role inference
  - `learn/roles.py` (deterministic wrapper inference), `plane/tasks/roles.py` (model roles, the `model_sinks.py`
    approach), the vocabulary overlay for the engine; fixture tests with no model calls.
  - _Requirements: 8.1_

- [x] 4. Labels
  - `learn/sources.toml` (fail-closed allowed sources, v3 excluded unread), `learn/labels.py`, `ousast learn
    labels`, repository-group normalisation and CVE merge, the `assumed_benign` source mined from non-security
    commits (separate, never merged); a first label snapshot committed as counts only.
  - _Requirements: 2.1, 2.2, 2.3, 7.1_

- [x] 5. Harvest (needs budget go-ahead)
  - A plane Run of verify a/b on the verify-family pairs and development cases (~$6, ceiling $10), model roles on
    the candidate files (ceiling $10), engine features for the pair corpus on the host (frozen source, one
    container at a time, background).
  - Done 2026-10-01 (`benchmarks/measurements/2026-10-01-decision-engine-harvest/record.json`): verify $2.54 and roles
    $1.40 by attribution. The engine job finished the same day (682 of 686 pair sides; the 2 family-unknown pairs
    not run) and the feature records were rebuilt; dev-php pins have no engine run (`missing`).
  - Advisory-fix pairs harvested 2026-10-02 (`benchmarks/measurements/2026-10-02-decision-engine-harvest-advisory/record.json`):
    194 sides, verify a/b on the 34 untrusted_destination sides and model roles on all ($0.38 metered, ceiling $5),
    engine on the host (54 ran, 58 asked no question, 82 no language model); static memory 724 -> 904 examples. The
    leak audit flags nothing new in the static profile; the plane profile newly flags `verify.flag_b` and `ms.operation`.
  - _Requirements: 1.1_

- [ ] 6. Program, local memory and compilation (design sections 4-5)
  - Status 2026-10-04: 6.1-6.6 are done. 6.7 is partial: leave-one-source-out, the 0/25/50% memory-size points and
    the source-count curve are not run. Around this task the experiment machinery grew: rule arms
    (`precision_bound_block`, `pooled_block_gate` with a family floor) in `learn/experiments.py`, the response-cache
    fill with a hard ceiling and a dry-run estimate in `learn/fill.py`, and the in-function slice in `learn/slice.py`.
    These ran exp-003, exp-004 and exp-005 (7.5-7.7). Memory now holds 710 complete fix pairs (7.9).
  - [x] 6.1 Excerpts and the example store (no model)
    - `learn/excerpt.py` (function span at the label's pin, bounds 80/4,000 for candidates and 40/2,000 for
      examples, numbered lines, no path/repository/commit, advisory ids and security wording in comments
      redacted, delta diff); `learn/examples.py` and `ousast learn memory build` (label + feature record +
      excerpt -> `example` row; refuses `origin: user` and sources outside `sources.toml`; exits 2 when more than
      10% of a source has no excerpt); `example` added to `KINDS`; `put_blob`/`get_blob` on both backends.
    - Tests: `test_learn_excerpt.py`, example-builder refusals, blob round trip on FileStore (S3 gated).
    - _Requirements: 3.2, 3.6, 7.2_
  - [x] 6.2 Metered embeddings and the cache
    - `plane/models/openrouter-embedding.yaml` (`input_per_m: 0.02`), `MeteredEmbeddingClient` in
      `plane/budget.py`, usage kept by `parse_embedding_response`; cache `embeddings/<model>/<excerpt_sha>.json`;
      `ousast learn memory embed`; scripted-client tests.
    - **(budget go-ahead)** embedding the store: ~5.0 M tokens, ~$0.10, ceiling $1 (OpenRouter account).
      Done 2026-10-01: 616 distinct excerpts of the 724 static examples, 198,819 tokens, 20 calls, $0.0040 metered
      (`benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`).
    - _Requirements: 3.2_
  - [x] 6.3 Retrieval and the evaluation boundary (no model)
    - `learn/retrieve.py`: per-instrument Gower distance with explicit missing, `n1 = 40`, embedding re-rank
      `r = lam(1 - D) + (1 - lam)cos`, `signals_only` mode, balance rule (`k_ret = 6`), `eligible(example, target,
      fold)`, near-duplicate drop in evaluation, deployment self-exclusion.
    - `learn/folds.py`: compile split (25% of groups), outer grouped 5-fold, leave-one-source-out,
      leave-one-framework-out.
    - Tests: `test_learn_retrieve.py` including `test_boundary` (every rendered example and demonstration eligible,
      for all fold kinds; `memory.seed` excludes the case's group), `test_learn_folds.py`.
    - _Requirements: 3.2, 3.4, 7.1, 8.3_
  - [x] 6.4 The program (no model in tests)
    - `learn/program.py`: the signature, prompt rendering (instruction + demonstrations as a byte-identical prefix,
      then retrieved examples, then the candidate), JSON parsing with one retry -> `unsure, parse_failed`,
      `Classify(k)` with temperature 0 then 0.7, the score `s` and majority verdict; `responses/<sha>.json` cache
      and replay; `temperature`/`logprobs` pass-through in `MeteredClient.complete` and `DeepSeekChatClient._call`.
    - Tests: `test_learn_program.py` (no `EXCLUDED_FIELDS` in prompts, prefix identity, parsing, `unsure` cannot
      BLOCK, budget ceiling, replay).
    - _Requirements: 3.1, 3.3_
  - [x] 6.5 Compile, calibrate, evaluate (no model in tests)
    - `learn/compile.toml` (pre-registered metric: balanced Brier score; seeds; budget), `learn/compile.py`
      (bootstrap few-shot on `C_boot`, 6 proposed + baseline instructions scored on `C_val`, 4 demonstration sets,
      alternates for leave-one-framework-out, artifact `programs/<sha>.json`), `learn/calibrate.py` (cross-fitted
      Platt, reliability/slope/intercept/ECE, ADVISORY-only downgrade), `learn/evaluate.py` (operating points from
      the other folds, repository-cluster bootstrap, strata, "insufficient data"), `ousast learn
      compile|evaluate|curve`.
    - Tests: `test_learn_compile.py` (`test_reproducible_from_cache` on both backends), `test_learn_calibrate.py`,
      `test_learn_decide.py`.
    - _Requirements: 3.3, 3.5, 5.2, 7.3, 7.4_
  - [x] 6.6 First paid slice **(budget go-ahead: ~$5 expected, ceiling $7)**
    - Smoke run on 20 candidates (~$0.10): real token counts, prefix-cache hit share, parse rate; it fixes `k`
      (5 if samples 2..k hit the cache for >= 80% of their prompt tokens, else 1) before anything else is spent.
    - One compile per profile with data (`static` first; ~$2.2, ceiling $3).
    - Outer-fold evaluation, priors off, evaluable families only (675 candidates: ~$2.0 at k = 1, ~$3.0 at k = 5),
      with the 30-candidate canary set; report under `benchmarks/measurements/<date>-decision-engine-v1/` (counts
      and intervals only, no identities).
    - Done 2026-10-01 for family **injection only** (the maintainer's scope for this slice; memory holds every family,
      at most 2 contrast examples per prompt): smoke fixed `k = 5` (repeat hit share 0.96); program `c1237a49...`
      (proposed instruction 5, balanced Brier 0.128 on 13 `C_val` candidates vs 0.199 baseline); 110 out-of-repository
      candidates in 29 groups: ADVISORY recall 0.90 [0.82, 0.97], precision 0.54 [0.44, 0.65]; AUC of `s` 0.74 [0.65,
      0.85]; BLOCK not offered (ECE 0.064 > 0.05, and no fold reached a Wilson lower bound of 0.95); canary agreement
      0.83; $0.91 metered at list price (provider balance $18.20 -> $17.90). Record:
      `benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`. The other evaluable families
      are not evaluated yet.
    - _Requirements: 3.1-3.5, 5.2, 7.3, 7.4_
  - [ ] 6.7 Second paid slice **(budget go-ahead: ~$7 expected, ceiling $10; needs a top-up)**
    - Leave-one-source-out (~$1.4-2.1), leave-one-framework-out where a framework reaches 10 groups ($0 today),
      learning curves over memory size (0/25/50/100%) and number of sources (~$5.2), recorded beside 6.6.
    - Partial 2026-10-02 (`benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`): after the
      advisory-fix harvest (memory 724 -> 904), one compile per evaluable family and its outer-fold evaluation for all
      six (injection, untrusted_destination, path, output_encoding, deserialization, access_control); out-of-repository
      AUC 0.77-0.96, BLOCK offered for none (calibration holds only for output_encoding, where no fold reaches the
      Wilson bound); one memory-size point for injection (724 vs 904 examples, paired: AUC 0.76 vs 0.77). $2.88
      metered (ceiling $8). Not done: leave-one-source-out, the 0/25/50% curve points, the source-count curve.
    - _Requirements: 5.1, 5.2, 8.3_

- [ ] 7. Experiments (exp-001 and program variants need budget go-ahead)
  - [x] 7.1 Machinery (no model in tests): `learn/experiments.py` -- YAML manifests under `plane/experiments/<id>.yaml`
    (hypothesis, arms as program-spec overrides or plane `task_env`, unit, pairing, repository cluster, metrics with
    `[label=]` selectors and targets, MDE, two looks with O'Brien-Fleming boundaries, budget ceiling, families, folds,
    frozen units file), `register` (`experiment` row with both digests; refuses an uncommitted, modified or re-declared
    manifest), `units` (freeze the families' evaluation candidates, paired by candidate across pins; no model), `run`
    (paired arms in a seeded repository order, a seeded coin per unit for which arm goes first, `arm_outcome` rows,
    resumable, arm A replay-only at $0, canary replicates under a fresh cache salt, the first look during the run),
    `analyse` (paired repository bootstrap, exact McNemar, the looks, adopt / reject / equivalent / inconclusive,
    `experiment_result` row + `benchmarks/experiments/<id>/result.json`), `ousast learn experiment
    register|units|run|analyse`; `OUSAST_VERIFY_CALLERS=off` in `plane/tasks/verify.py`; the retrieval-ensemble
    sampling variant (`ProgramSpec.sampling = retrieval_ensemble`: `k` samples at temperature 0 over `k` disjoint
    retrieved example sets; default unchanged). Tests: `test_learn_experiments.py` (18, scripted clients, FileStore).
  - [x] 7.2 Registered, not run: `exp-001-known-callers` (plane arms, `flag_rate[label=1]`, MDE 0.15, ceiling $15;
    units `pending` -- model-free candidate generation on the 41 development repositories is not built, and plane Run
    annotation checks / the `experiment-outcome` task are not written) and `exp-002-retrieval-ensemble` (arm A today's
    k = 5 temperature sampling, arm B the retrieval ensemble; primary canary agreement (target 0.90) and within-pair
    AUC, secondary usd per candidate, AUC, ADVISORY recall/precision; families injection and access_control on the
    slice-2 candidates; ceiling $4: A $1, B $3). Both registered in `s3://sast-memory` at fbc3220
    (`benchmarks/experiments/<id>/registration.json`). exp-002's units are `pending` too: on 2026-10-02 every example
    row in the store carried null plane-only instruments (`agree`, `model_sinks`, `verify`) and `load_examples` raised
    on them, so `ousast learn experiment units` could not freeze the candidates.
  - [ ] 7.3 exp-001 run and recorded (~$7.7 expected, ceiling $15) -- an inconclusive result is pre-registered as
    likely; needs the frozen units file first.
  - [x] 7.4 exp-002 run and recorded 2026-10-02 (budget go-ahead ~$2 expected, ceiling $4): **reject B**. 160 paired
    units (injection 119, access_control 41; 56 repositories; 108 units in 54 candidate pairs), both arms, two
    replicates each, no early stop. Primary: canary agreement A 0.875 -> B 0.906 (diff +0.03, CI [-0.03, +0.10],
    spans; B meets the 0.90 target), within-pair AUC A 0.815 -> B 0.741 (diff -0.07, CI [-0.13, -0.02], **against B**).
    Secondary: AUC -0.02 (CI [-0.04, -0.003], against), ADVISORY recall -0.06 (spans; McNemar 7/2, p = 0.18),
    precision +0.01 (spans), usd per candidate A $0.0021 -> B $0.0051 (CI [+0.0027, +0.0032], against: the five
    disjoint example sets share only the prompt prefix in DeepSeek's cache). Priced usage A $0.68 (1206 paid calls,
    395 replayed), B $1.63 (1350 paid, 250 replayed). Result: `benchmarks/experiments/exp-002-retrieval-ensemble/result.json`,
    `benchmarks/measurements/2026-10-02-exp-002/record.json`. Three repairs on the way (each its own commit): example
    rows' null plane-only features (1f525ff), the 788 embedding vectors absent from RustFS (copied from the file
    store), the result row's column-type collision under S3 Select (780caaa). Further program variants (`k = 5` vs
    `k = 1`, the confidence source) as new manifests.
  - [x] 7.5 exp-003 run and recorded 2026-10-03 ($0, replay only): **inconclusive**. Selective blocking (a per-family
    raw-score threshold chosen on training folds by its Wilson lower bound) offers no BLOCK on held-out paired units
    in any family; the incumbent stays. 211 of 353 frozen units evaluated (142 replay misses). Proving 0.95 needs at
    least 73 clean flags. Result: `benchmarks/experiments/exp-003-precision-bound-blocking/result.json`.
  - [x] 7.6 exp-004 run and recorded 2026-10-03: **inconclusive**. One pooled BLOCK threshold over all families with a
    family floor (Req 6.4). No outer fold found a threshold, pooled or per family; B offers no BLOCK. 501 units,
    replay at $0 after a cache fill of $1.24 (ceiling $2; the dry-run estimate over-priced 4.5x). Result and record:
    `benchmarks/experiments/exp-004-pooled-block-gate/result.json`, `record.json`.
  - [x] 7.7 exp-005 run and recorded 2026-10-03: **inconclusive**. The in-function def-use slice in the prompt of the
    six slice-2 programs. Within-pair AUC 0.762 -> 0.746 (CI [-0.063, 0.032], spans 0); unsure rate 0.29 -> 0.24
    (favours B); usd per candidate slightly higher. A $0 leak audit came first
    (`benchmarks/measurements/2026-10-03-graph-slice-leak-audit/record.json`). Result:
    `benchmarks/experiments/exp-005-graph-slice/result.json`.
  - [x] 7.8 Negatives harvest 2026-10-03 ($0.0014 embeddings, no chat-model call): 98 real fix pairs whose function
    survives the fix, from reviewed advisories, as pointer-pair corpus `advisory-fixes-2`; memory 937 -> 1133
    examples. Record: `benchmarks/measurements/2026-10-03-decision-engine-negatives-harvest/record.json`.
  - [x] 7.9 Harvest 3, 2026-10-04 ($0.003 embeddings, no chat-model call): 251 pairs as `advisory-fixes-3`; complete
    pairs in memory 459 -> 710. Record: `benchmarks/measurements/2026-10-04-decision-engine-harvest-3/record.json`.
  - [x] 7.10 Joern trace run over exp-005's 501 units and its leak audit, 2026-10-04 ($0): no leak flagged. In the 105
    pairs where both sides were asked, a path appears on both sides in 12, on the vulnerable side only in 2, on the
    fixed side only in 0, and on neither in 91. The engine cannot see what a fix changed. Record:
    `benchmarks/measurements/2026-10-04-joern-slice-leak-audit/record.json`.
  - _Requirements: 3.6, 4.1-4.4_

- [ ] 8. Integration
  - `plane/tasks/embed.py` (Model `openrouter-embedding`) and `plane/tasks/decide.py` (Model `deepseek-flash`,
    metered, exits 0/2/3), the host seed of the adopted program and the case's boundary-filtered memory; the `scan`
    decision stage and `dropped_by_engine.json`; `pre-push` admission with the engine (structural admission reasons
    unchanged; concurrency, `undecided: deadline`, a per-push budget); report fields (p, operating point,
    program id, verdict, rationale, cited lines); the fallback line (no program, no key, budget exhausted,
    insufficient data, calibration failed); the opt-in adaptation layer with user-local memory.
  - Tests use scripted clients; a plane fixture Run proves `decide` never receives an example of its own case.
  - _Requirements: 6.1-6.3, 7.3, 7.5_

- [ ] 9. Adopt
  - `ousast learn adopt`; the packaged `ruleset/decision/program-<profile>.json` with only licence-cleared
    demonstrations (the maintainer decides which licences qualify), re-evaluated as its own variant; the canary
    drift check before adoption; the priors-on/off experiment per framework with a recorded decision
    **(budget go-ahead per experiment, ~$1-2 each)**.
  - _Requirements: 3.3, 8.2_

- [ ] 10. The one-time v3 check **(budget go-ahead for the engine's v3 spend, ~$0.0045 per candidate)**
  - Commit `prediction-v3.json` first (program id, memory snapshot digest, model ids, operating points, predicted
    intervals, spend ceiling from the owner's candidate count); then v3 is run under protocol v3 and
    `result-v3-engine.json` is committed. Nothing between the two commits may touch the program, memory snapshot,
    features, priors or thresholds. This spec's code never opens v3 files.
  - _Requirements: 5.3_

- [ ] 11. Fix mechanisms (plan approved 2026-10-04; see `fix-mechanisms-plan.md`)
  - Roadmap G2 arm (`.kiro/steering/roadmap.md`, section of 2026-10-04): one candidate improvement, not the spine.
    It is judged on the unseen-repository pool (G0, spec `unseen-repo-evaluation`) by catch rate and false-alarm
    rate per push. **Paused after 11.1 until the G0 pool exists.**
  - [ ] 11.1 Mine candidate discharge facts from the 710 fix diffs (sanitizer calls, guarding conditions, safe sink
    arguments, safe replacements); deterministic, $0. In progress.
  - [ ] 11.2 Admission by held-out separation per outer fold: a candidate turns "path on both sides" into "path on the
    vulnerable side only" on folds it was not mined from, removes no held-out vulnerable path, has at least 2
    proposing repositories, and is admitted per language.
  - [ ] 11.3 Engine re-run on the affected pins (kube-ax lanes plus the VM lane).
  - [ ] 11.4 Registered experiment, with the detection gates green, before `ruleset/semantic/*.toml` changes.
  - [ ] 11.5 Coverage increment: source and sink additions for pairs where neither side has a path (later).
  - [ ] 11.6 TypeScript admission in the engine (branch `engine-typescript`).
  - _Requirements: 4.1-4.4, 7.1, 8.1_

- [ ] 12. exp-006: Joern witness slice in the prompt **(budget go-ahead, about $2-4)**
  - Planned, roadmap G1/G2. Against the current prompt, after a $0 leak audit, on re-frozen units. Judged on the
    unseen-repository pool, not on within-pair AUC alone.
  - _Requirements: 4.1-4.4, 7.1_

- [ ] 13. exp-007: the combiner ($0 replay)
  - Planned, roadmap G1/G2. Three or four weights (score, engine entailment, neighbour agreement) over cached
    responses. Judged on the unseen-repository pool.
  - _Requirements: 4.1-4.4, 7.1_

- [ ] 14. Re-run the pooled block gate on re-frozen units
  - Planned, roadmap G3. The exp-004 design with the family floor (Req 6.4), together with the pool's false-alarm
    rate per family.
  - _Requirements: 6.4, 7.3_

- [ ] 15. Trained classifier arm (maintainer decision 2026-10-04: train a model, not hand-set profiles; judged on the
  unseen-repo pool, roadmap G1)
  - [ ] 15.1 Features: code embedding of the excerpt (stored vectors) + every allow-listed signal; no hand-picked
    profile. Automated screening: a signal that separates pair sides by itself on training folds above the leak
    threshold is dropped for that fold (the leak audit as a filter, fitted per fold).
  - [ ] 15.2 Models: regularised logistic regression and gradient-boosted trees, fitted per repository-held-out outer
    fold with inner-fold hyper-parameter search; per-family and pooled variants.
  - [ ] 15.3 Scores on held-out folds: within-pair AUC, pooled AUC, top-k precision, Wilson bound at the pooled
    threshold; compared with the current LLM programs on the same units.
  - [ ] 15.4 Scored on the unseen-repo pool when G0 exists: catch rate at fixed false-alarm budgets.
  - _Requirements: 3 (revised 2026-10-04 by maintainer decision), 6, 7_

- [ ] 16. Jointly optimised LLM program arm (judged on the unseen-repo pool)
  - Search input fields, retrieval parameters, k, instruction and demonstrations together (DSPy MIPRO-style) on
    training folds; no hand-set input profile. Budget go-ahead before paid search.
  - _Requirements: 3, 6, 7_

- [ ] 17. Triage objective (Req 6.5, amended 2026-10-05 with `search-with-proof`)
  - [ ] 17.1 Metric: top-K recall per push (K = 1, 3, 5) of the true vulnerable function among changed functions,
    on held-out folds and on the unseen-repo pool; added to the pool scorer next to catch and false alarm.
  - [ ] 17.2 Score the existing arms on it ($0 replay): trained logistic (cheapest per candidate, likely default),
    LLM programs (exp-005 A), engine-evidence ordering, and an averaged ensemble.
  - [ ] 17.3 Re-scope 14 (pooled block gate) to families without a demonstration oracle; 15.4 and 16 report top-K
    recall as their primary unseen-pool number for families with an oracle.
  - _Requirements: 6.5, 7_