Open_Ultra_SAST/.kiro/specs/hunter-learning-loop/research.md

Research and design decisions: hunter-learning-loop

Summary

• Feature: hunter-learning-loop
• Discovery scope: Extension of the learn harness and the valid benchmark; one new runtime dependency (DSPy).
• Maintainer direction, 2026-10-13: "the core we need to evolve is feature engineering ... that is the deciding factor here"; "labels for me are a pandas task"; DSPy for the hunter program first, statistics libraries second, no OpenAI SDK on its own ("unless dspy pulls it in"; it does, through litellm); lizard against its alternatives is worth pursuing; labels and folds are a pandas and scikit-learn task; Semgrep OSS is a good idea. Each of those is a feature producer or a feature consumer, and the feature table decides between them.
• Key findings:
  • The pilot (benchmarks/measurements/2026-10-12-valid-benchmark-pilot) measured the hunter at 5 to 6 of 12 and 54 false alarms per 100 pushes after the classifier; 250 of 343 "vulnerable" suspicions on ordinary pushes sit more than 10 lines from the diff. The hunter never sees the diff (01-flowcharts/U6-hunter-client.md).
  • The harness already has the pieces the loop needs: labels (learn/labels.py), repository folds and leak audit (learn/folds.py, learn/leaks.py), paired experiments with cluster bootstrap and McNemar (learn/experiments.py), the cached caller (learn/program.py), and the runner that joins hunter output to frozen changes (benchmarks/valid/hunter_arm.py). What it lacks is a feature table and a program runtime that can be optimised rather than hand-edited.
  • source.facts_for (P1, commit 062a7b7) is the first feature producer: language, enclosing function, span, callers, entry points, from the repository only.

Research log

DSPy as the program runtime

• Context: requirements R3 to R6 need a signature with switchable inputs, demonstrations, instruction proposals, a scorer-driven compile, and metered, replayable model calls.
• Sources: local install, dspy==3.4.1 into the research venv (uv pip install --dry-run dspy: 42 packages, among them litellm 1.105, openai 2.54, gepa 0.1.4, diskcache, json_repair, tenacity, pydantic); API introspection of the installed package (dspy.LM(model, cache=True, num_retries=3, ...), dspy.ReAct(signature, tools, max_iters=20), dspy.Predict, dspy.Signature with InputField/OutputField, dspy.Evaluate(devset, metric, num_threads), dspy.BootstrapFewShot, dspy.MIPROv2(metric, auto=...), dspy.GEPA(metric, auto, reflection_lm, candidate_selection_strategy="pareto"), dspy.track_usage(), dspy.Module.save(path)).
• Findings: dspy.LM reaches DeepSeek and OpenRouter through litellm with an OpenAI-compatible base URL and the key from the environment; the project's model id deepseek-flash is not in litellm's price table, so prices are registered from our table (model/endpoint.py:55). DSPy caches responses on disk (diskcache) keyed by the request; a run's cache directory is settable, so a run can be replayed from its own directory. dspy.ReAct is the tool-calling loop the hunter already is (read_file, grep_repo, find_refs). GEPA is the optimiser the learning-harness brief described (per-instance Pareto archive, reflective edits); MIPROv2 is the instruction and demo search. Neither applies our acceptance rule; that stays outside DSPy (R6).
• Implications: the hunter program is written as a DSPy module from the start; the classifier (learn/program.py) stays as is for this spec (R7.5). The hook's urllib client is untouched: the hook has no LLM tier yet, and the core install stays PyYAML-only; DSPy is a research extra.

The feature table (labels and features as a pandas task)

• Context: R1, R3, R4 need one row per candidate location with named feature columns, a label, provenance, and the ability to switch columns off per ablation arm.
• Findings: learn/features.py builds feature records as nested dicts and parts; learn/labels.py (979 lines) joins sources by hand; benchmarks/unseen/score.py joins records by id. All of it is group-by, merge and dedup work. pandas and scikit-learn (grouped folds, calibration, metrics) are what the maintainer expects here and are pulled by none of the current extras.
• Implications: the loop's contract is a DataFrame (persisted as JSONL or parquet) with a fixed column schema; producers add columns, ablation arms drop columns, the program renders a row into its prompt. Existing dict-based builders are consumed, not rewritten, in this spec; migrating labels to pandas is part of the statistics item, not this one.

Feature producers and the tools under evaluation

• Context: the annotation model (R3) lists the facts; the maintainer wants the producers chosen by what gives the best features.
• Candidates: source.facts_for (ours; P1), the semantic overlay facts (roles, def-use, obligations; semantic/), mapping.analyze_entry_points, lizard (function spans, nesting, cyclomatic complexity; pure Python), universal-ctags (spans across many languages; a binary), Semgrep OSS or Opengrep (pattern and taint findings as features; also a candidate to replace the quick-rule engine, a separate decision).
• Status: a measured comparison of lizard against our two engines on the pair corpus is running (~/ousast-results/fossil/span-eval/REPORT.md); a licensing and capability brief for Semgrep CE and Opengrep is being written (~/ousast-results/fossil/semgrep-brief.md). Both land as producers behind the same contract; the ablation decides whether their columns stay.
• Research needed: Semgrep registry rule licence terms for an open-source tool.

Training data apart from the pilot

• Context: R2. The pilot (252 changes, 12 positives) is held out.
• Findings: benchmarks/valid/catalog.py and verify.py produce fix-site positives from OSV advisories in about 12 accepted per 32 advisories; the used-set guard (benchmarks/unseen/*) excludes training-only sources. learn/folds.py groups by repository; learn/leaks.py audits per-signal separation.
• Implications: about 100 advisories give about 40 positives; negatives at 20 per positive from the same repositories; folds by repository; the pilot is never read by the loop.

Architecture pattern evaluation

Option	Description	Strengths	Risks	Notes
A. Extend learn/program.py with a hunter signature	one harness, our caller and cache	no new dependency	the compile loop has no train-vs-holdout rule, no tool loop, no optimiser beyond demos+instructions; everything the maintainer asked for (statistical mutation of prompts) is to build	rejected
B. New hunt/ package on DSPy, feature table as the contract, harness reused for labels, folds, leaks, paired analysis	the optimisers, the tool loop, caching and usage come with the library; our code is the feature producers, the table, the acceptance rule, the scoring	one heavy research dependency; two caches (DSPy's and Caller's) until the classifier migrates	selected
C. Rewrite the classifier onto DSPy too	one runtime	large rewrite of a parked line	later, if B earns it

Design decisions

Decision: DSPy runs the hunter program; the acceptance rule stays ours

• Alternatives: A above; a hand-written optimiser.
• Selected: dspy.ReAct over a HuntSignature whose inputs are the rendered feature row plus the changed hunks; dspy.GEPA or MIPROv2 propose candidates; dspy.Evaluate scores them on compile folds with our metric; the candidate is accepted only by our rule on held-out folds (R6), implemented outside DSPy.
• Trade-offs: DSPy's cache is request-keyed, not content-addressed; a run pins its cache directory and copies it with the run record. Cost comes from dspy.track_usage() with our prices registered in litellm.

Decision: the feature table is the contract between producers, ablation and the program

• Selected: one row per (change, path, line) candidate with columns feature.<producer>.<name>, label, provenance.*, fold; producers are functions (root, change, path, line) -> dict; an ablation arm is a column mask recorded in the manifest; the program sees only the unmasked columns, rendered in a fixed order.
• Trade-offs: a table per run is larger than nested records; it is also what pandas, scipy and the maintainer read.

Decision: producers are a fixed list, switched by name, no registry

• source.facts_for, semantic facts (roles, def-use, obligations), mapping entry points, diff hunks; lizard and Semgrep/Opengrep join the list only after their evaluation reports.

Decision: objective and metric

• Search stage: recall over verified fix sites (a suspicion within the scorer's tolerance of the fix site with the right family) and suspicions per ordinary push; the DSPy metric returns the per-change score and, for GEPA, a text feedback naming the missed site or the off-diff suspicions. No threshold is tuned (R4.7); the curve is reported (R4.8).

Risks and mitigations

• DSPy or litellm upgrades change behaviour: pin dspy==3.4.*, record versions in every run artifact.
• A producer reads a label by accident: producers take the repository and a location only; a test feeds a repository with a planted advisory id and asserts no feature carries it (R3.7).
• Train-on-test: the pilot path is denied to the loop by construction (the runner refuses the pilot manifest digest), and every artifact lists the data digests it read (R2.4).
• Cost: budgets per run (R9.1); DSPy cache on; --replay-only reruns at no cost.

References

• DSPy 3.4.1 (installed; https://dspy.ai), GEPA (https://github.com/gepa-ai/gepa), litellm pricing registration.
• PATHFINDER-2026-10-13/01-flowcharts/U6-hunter-client.md, U7-learn-memory-tasks.md; 02-duplication-report.md D1, D4.
• benchmarks/measurements/2026-10-12-valid-benchmark-pilot/NOTES.md (numbers and the filter sweep).