Open_Ultra_SAST/.kiro/specs/hunter-learning-loop/research.md

# Research and design decisions: hunter-learning-loop

## Summary
- **Feature**: `hunter-learning-loop`
- **Discovery scope**: Extension of the learn harness and the valid benchmark; one new runtime dependency (DSPy).
- **Maintainer direction, 2026-10-13**: "the core we need to evolve is feature engineering ... that is the deciding
  factor here"; "labels for me are a pandas task"; DSPy for the hunter program first, statistics libraries second,
  no OpenAI SDK on its own ("unless dspy pulls it in"; it does, through litellm); lizard against its alternatives is worth
  pursuing; labels and folds are a pandas and scikit-learn task; Semgrep OSS is a good idea. Each of those is a
  feature producer or a feature consumer, and the feature table decides between them.
- **Key findings**:
  - The pilot (benchmarks/measurements/2026-10-12-valid-benchmark-pilot) measured the hunter at 5 to 6 of 12 and 54
    false alarms per 100 pushes after the classifier; 250 of 343 "vulnerable" suspicions on ordinary pushes sit more
    than 10 lines from the diff. The hunter never sees the diff (01-flowcharts/U6-hunter-client.md).
  - The harness already has the pieces the loop needs: labels (`learn/labels.py`), repository folds and leak audit
    (`learn/folds.py`, `learn/leaks.py`), paired experiments with cluster bootstrap and McNemar
    (`learn/experiments.py`), the cached caller (`learn/program.py`), and the runner that joins hunter output to
    frozen changes (`benchmarks/valid/hunter_arm.py`). What it lacks is a feature table and a program runtime that
    can be optimised rather than hand-edited.
  - `source.facts_for` (P1, commit 062a7b7) is the first feature producer: language, enclosing function, span,
    callers, entry points, from the repository only.

## Research log

### DSPy as the program runtime
- **Context**: requirements R3 to R6 need a signature with switchable inputs, demonstrations, instruction
  proposals, a scorer-driven compile, and metered, replayable model calls.
- **Sources**: local install, `dspy==3.4.1` into the research venv (`uv pip install --dry-run dspy`: 42 packages,
  among them `litellm 1.105`, `openai 2.54`, `gepa 0.1.4`, `diskcache`, `json_repair`, `tenacity`, `pydantic`);
  API introspection of the installed package (`dspy.LM(model, cache=True, num_retries=3, ...)`,
  `dspy.ReAct(signature, tools, max_iters=20)`, `dspy.Predict`, `dspy.Signature` with `InputField`/`OutputField`,
  `dspy.Evaluate(devset, metric, num_threads)`, `dspy.BootstrapFewShot`, `dspy.MIPROv2(metric, auto=...)`,
  `dspy.GEPA(metric, auto, reflection_lm, candidate_selection_strategy="pareto")`, `dspy.track_usage()`,
  `dspy.Module.save(path)`).
- **Findings**: `dspy.LM` reaches DeepSeek and OpenRouter through litellm with an OpenAI-compatible base URL and the
  key from the environment; the project's model id `deepseek-flash` is not in litellm's price table, so prices are
  registered from our table (`model/endpoint.py:55`). DSPy caches responses on disk (`diskcache`) keyed by the
  request; a run's cache directory is settable, so a run can be replayed from its own directory. `dspy.ReAct` is
  the tool-calling loop the hunter already is (read_file, grep_repo, find_refs). GEPA is the optimiser the
  learning-harness brief described (per-instance Pareto archive, reflective edits); MIPROv2 is the instruction and
  demo search. Neither applies our acceptance rule; that stays outside DSPy (R6).
- **Implications**: the hunter program is written as a DSPy module from the start; the classifier
  (`learn/program.py`) stays as is for this spec (R7.5). The hook's urllib client is untouched: the hook has no LLM
  tier yet, and the core install stays PyYAML-only; DSPy is a research extra.

### The feature table (labels and features as a pandas task)
- **Context**: R1, R3, R4 need one row per candidate location with named feature columns, a label, provenance, and
  the ability to switch columns off per ablation arm.
- **Findings**: `learn/features.py` builds feature records as nested dicts and parts; `learn/labels.py` (979 lines)
  joins sources by hand; `benchmarks/unseen/score.py` joins records by id. All of it is group-by, merge and
  dedup work. pandas and scikit-learn (grouped folds, calibration, metrics) are what the maintainer expects here and are pulled
  by none of the current extras.
- **Implications**: the loop's contract is a DataFrame (persisted as JSONL or parquet) with a fixed column schema;
  producers add columns, ablation arms drop columns, the program renders a row into its prompt. Existing dict-based
  builders are consumed, not rewritten, in this spec; migrating labels to pandas is part of the statistics item,
  not this one.

### Feature producers and the tools under evaluation
- **Context**: the annotation model (R3) lists the facts; the maintainer wants the producers chosen by what gives
  the best features.
- **Candidates**: `source.facts_for` (ours; P1), the semantic overlay facts (roles, def-use, obligations;
  `semantic/`), `mapping.analyze_entry_points`, lizard (function spans, nesting, cyclomatic complexity; pure
  Python), universal-ctags (spans across many languages; a binary), Semgrep OSS or Opengrep (pattern
  and taint findings as features; also a candidate to replace the quick-rule engine, a separate decision).
- **Result, lizard (benchmarks/measurements/2026-10-13-span-engines/REPORT.md)**: on 610 pair files and 570 labelled
  functions, lizard matches our parser engine line for line on C, Python and Java (end-line difference 0.00,
  name-set Jaccard 0.97 to 1.00) and survives the 56 excerpts the cpp grammar rejects, but loses 6 of 44 labelled
  JavaScript functions to naming (`$` identifiers, `function anonymous`, nested named functions, assigned function
  expressions), gets TypeScript end lines wrong on 22% of labels (one by 81 lines), has no Perl reader, and is the
  slowest of the four engines. Verdict: keep ours; lizard's per-function cyclomatic complexity and nesting are a
  candidate feature column, not a span engine (its nesting correlates with `measure_nesting` at Spearman 0.6, so
  it is not a drop-in there either).
- **Result, tree-sitter-language-pack**: the same tree-sitter, 503 of 503 spans identical with our parser engine,
  1.4x faster (it caches parsers), one 6 MB MIT wheel in place of six grammar wheels, and it adds Go, Ruby, PHP
  and Perl grammars that ours handles by regex today. Adopt it as the grammar packaging in `semantic/extra.py`;
  map `.c` to the c grammar first (20 of the 56 rejections clear); cache the parser. `tree-sitter-languages` is
  ABI-broken under tree-sitter 0.26; universal-ctags has no pip build.
- **Result, Semgrep CE and Opengrep (benchmarks/measurements/2026-10-13-semgrep-brief/BRIEF.md)**: the engine is
  LGPL-2.1 and shelling out carries no obligation, but every Semgrep-maintained registry rule has been under the
  Semgrep Rules License since 2024-12-13 (internal use only, no distribution, not in competing products), so
  nothing but our own converted rules could ship. Measured here: 4.7 s startup floor and about 5 to 10 ms per
  file after it (the regex tier runs in-process under a second); converted regex and taint rules fire with
  traces for Python, PHP and JavaScript; C propagation through `strcpy` does not (propagators are Pro); Groovy has
  no parser; a Semgrep sanitizer silenced the `ORDER BY` shape our `quoted_only` flag refuses on purpose; guards,
  dispatch facts, obligations, shadow status and emit floors have no equivalent. A swap would retire about 1,250
  lines. Recommendation taken from the brief: option C first, a Semgrep arm on the valid pilot
  (`benchmarks/valid/semgrep_arm.py`, our rules converted, scored by the same scorer) before any decision on the
  quick tier; adopting it later means amending the "no second engine" rule in safety-net.md with that measurement.
  For this spec, Semgrep findings are a candidate feature column like any other producer.
- **Research needed**: Semgrep registry rule licence terms for an open-source tool.

### Training data apart from the pilot
- **Context**: R2. The pilot (252 changes, 12 positives) is held out.
- **Findings**: `benchmarks/valid/catalog.py` and `verify.py` produce fix-site positives from OSV advisories in
  about 12 accepted per 32 advisories; the used-set guard (`benchmarks/unseen/*`) excludes training-only sources.
  `learn/folds.py` groups by repository; `learn/leaks.py` audits per-signal separation.
- **Implications**: about 100 advisories give about 40 positives; negatives at 20 per positive from the same
  repositories; folds by repository; the pilot is never read by the loop.

## Architecture pattern evaluation
| Option | Description | Strengths | Risks | Notes |
|---|---|---|---|---|
| A. Extend `learn/program.py` with a hunter signature | one harness, our caller and cache | no new dependency | the compile loop has no train-vs-holdout rule, no tool loop, no optimiser beyond demos+instructions; everything the maintainer asked for (statistical mutation of prompts) is to build | rejected |
| B. New `hunt/` package on DSPy, feature table as the contract, harness reused for labels, folds, leaks, paired analysis | the optimisers, the tool loop, caching and usage come with the library; our code is the feature producers, the table, the acceptance rule, the scoring | one heavy research dependency; two caches (DSPy's and `Caller`'s) until the classifier migrates | selected |
| C. Rewrite the classifier onto DSPy too | one runtime | large rewrite of a parked line | later, if B earns it |

## Design decisions
### Decision: DSPy runs the hunter program; the acceptance rule stays ours
- Alternatives: A above; a hand-written optimiser.
- Selected: `dspy.ReAct` over a `HuntSignature` whose inputs are the rendered feature row plus the changed hunks;
  `dspy.GEPA` or `MIPROv2` propose candidates; `dspy.Evaluate` scores them on compile folds with our metric; the
  candidate is accepted only by our rule on held-out folds (R6), implemented outside DSPy.
- Trade-offs: DSPy's cache is request-keyed, not content-addressed; a run pins its cache directory and copies it
  with the run record. Cost comes from `dspy.track_usage()` with our prices registered in litellm.

### Decision: the feature table is the contract between producers, ablation and the program
- Selected: one row per (change, path, line) candidate with columns `feature.<producer>.<name>`, `label`,
  `provenance.*`, `fold`; producers are functions `(root, change, path, line) -> dict`; an ablation arm is a
  column mask recorded in the manifest; the program sees only the unmasked columns, rendered in a fixed order.
- Trade-offs: a table per run is larger than nested records; it is also what pandas, scipy and the maintainer read.

### Decision: producers are a fixed list, switched by name, no registry
- `source.facts_for`, `semantic` facts (roles, def-use, obligations), `mapping` entry points, `diff` hunks; lizard and
  Semgrep/Opengrep join the list only after their evaluation reports.

### Decision: objective and metric
- Search stage: recall over verified fix sites (a suspicion within the scorer's tolerance of the fix site with the
  right family) and suspicions per ordinary push; the DSPy metric returns the per-change score and, for GEPA, a
  text feedback naming the missed site or the off-diff suspicions. No threshold is tuned (R4.7); the curve is
  reported (R4.8).

## Risks and mitigations
- DSPy or litellm upgrades change behaviour: pin `dspy==3.4.*`, record versions in every run artifact.
- A producer reads a label by accident: producers take the repository and a location only; a test feeds a
  repository with a planted advisory id and asserts no feature carries it (R3.7).
- Train-on-test: the pilot path is denied to the loop by construction (the runner refuses the pilot manifest
  digest), and every artifact lists the data digests it read (R2.4).
- Cost: budgets per run (R9.1); DSPy cache on; `--replay-only` reruns at no cost.

## References
- DSPy 3.4.1 (installed; https://dspy.ai), GEPA (https://github.com/gepa-ai/gepa), litellm pricing registration.
- PATHFINDER-2026-10-13/01-flowcharts/U6-hunter-client.md, U7-learn-memory-tasks.md; 02-duplication-report.md D1, D4.
- benchmarks/measurements/2026-10-12-valid-benchmark-pilot/NOTES.md (numbers and the filter sweep).