diff --git a/.kiro/specs/hunter-learning-loop/design.md b/.kiro/specs/hunter-learning-loop/design.md new file mode 100644 index 0000000..d64d68a --- /dev/null +++ b/.kiro/specs/hunter-learning-loop/design.md @@ -0,0 +1,204 @@ +# Design: hunter-learning-loop + +## Overview +**Purpose**: turn the LLM hunter from a hand-written prompt into a measured search stage whose inputs are an +engineered feature table and whose prompt is optimised by a program runtime, judged on held-out folds. +**Users**: the maintainer runs the loop locally on the research host; the pre-push safety net consumes its output +later (not in this spec). +**Impact**: adds a research package `src/openultrasast/hunt/` and one research dependency (DSPy); reuses the +learn harness for labels, folds, leak audit and paired analysis; reuses `benchmarks/valid/hunter_arm.py` as the +runner and `benchmarks/unseen/score.py` plus `benchmarks/valid/metrics.py` as the only scorekeepers. The hook, +its urllib client and the core install are untouched. + +### Goals +- A feature table with named, switchable columns is the contract between producers, ablation and the program (3, 4). +- The hunter is a DSPy program over that table plus the changed hunks; its instruction and demonstrations are + proposed by DSPy optimisers and accepted only by our rule on held-out folds (5, 6). +- Every annotation's worth is a paired, pre-registered ablation with intervals; the importance table is the output + the maintainer reads (4). +- Search recall and suspicions per push improve on the held-out pilot, reported with the pilot's own scorer (7, 8). + +### Non-Goals +- Thresholds or operating points (4.7); the hook's LLM tier (pre-push-safety-net); the proof arm (search-with-proof); + retraining the classifier (7.5); replacing the quick-rule engine with Semgrep (separate decision, brief pending); + migrating `learn/labels.py` to pandas (statistics item, separate). + +## Boundary Commitments +### This spec owns +- `src/openultrasast/hunt/`: the feature table schema, the producer list, the hunt program, the ablation runner, + the acceptance rule, the run artifact. +- The training-set recipe (which advisories, folds, digests) and its records under `benchmarks/measurements/`. +- The importance table and the before/after report on the pilot. +### Out of boundary +- Label semantics (`benchmarks/valid/catalog.py`, `verify.py`, `learn/labels.py`): consumed as they are. +- The scorer and metrics (`benchmarks/unseen/score.py`, `benchmarks/valid/metrics.py`): never edited to fit. +- The hook, `tool_hunter.py`'s shipped prompt, the classifier program, Semgrep adoption, lizard adoption (each a + producer candidate that enters only through the producer contract after its own evaluation). +### Allowed dependencies +- `openultrasast.source`, `semantic` facts, `mapping` entry points, `learn.folds`, `learn.leaks`, + `learn.experiments` (paired bootstrap, McNemar), `learn.labels` readers, `benchmarks.valid.hunter_arm` (runner), + `benchmarks.unseen.score`; `dspy` (research extra `[hunt]`), `pandas` (table), `scipy`/`statsmodels` once the + statistics item lands (until then `learn.experiments`). +- Dependency direction: data (manifests, labels) → producers → feature table → program → runner → scorer → report. + Nothing to the right imports to the left; producers never import `hunt.program`; nothing in `hunt/` imports + `push/` or `cli.py`. +### Revalidation triggers +- The feature table schema version changes; a producer's output shape changes; the scorer's tolerance changes; + DSPy minor version changes; the pilot manifest digest changes. + +## Architecture +### Existing architecture analysis +The hunter (`tool_hunter.run_tool_hunter`) is a tool loop with a four-sentence prompt and no diff +(01-flowcharts/U6-hunter-client.md). The runner `hunter_arm.run` already walks a frozen manifest, checks out +each head, proves bytes read, runs the hunter, classifies each finding with the decision-engine classifier and +writes one record per change; `combine` joins with the replay. The harness has grouped folds, a leak audit and +paired analysis. The missing pieces are the table, the switchable inputs and the optimisable program. + +### Pattern and boundary map +```mermaid +flowchart LR + M[(frozen manifests
training set, pilot held out)] --> P + L[(labels
fix sites, ordinary pushes)] --> T + subgraph producers + P1[source.facts_for] + P2[semantic facts
roles, def-use, obligations] + P3[mapping entry points] + P4[diff hunks] + P5[candidates: lizard, Semgrep
after evaluation] + end + M --> P[hunt.producers.compute] + P --> P1 & P2 & P3 & P4 & P5 + P1 & P2 & P3 & P4 & P5 --> T[(feature table
hunt.table)] + T --> A[hunt.ablation
column mask per arm] + A --> G[hunt.program
dspy.ReAct + HuntSignature] + G --> R[hunter_arm.run
records per change] + R --> S[unseen.score + valid.metrics] + S --> I[importance table,
acceptance decision, report] + O[dspy optimisers
GEPA, MIPROv2] --> G + I --> O +``` +- Selected pattern: a table-centred pipeline; the program is a replaceable consumer of the table. +- Existing patterns preserved: the runner's record shape and resumability; the scorer; folds by repository; the + instrument proof (bytes read, base and head verified). + +### Technology stack (changed layers only) +| Layer | Tool | Role | +|---|---|---| +| Program runtime | `dspy==3.4.*` (pulls litellm, openai, gepa, diskcache) as extra `[hunt]` | signature, ReAct tool loop, optimisers, cache, usage | +| Model access | `dspy.LM("openai/deepseek-flash", api_base=DEEPSEEK_BASE_URL, api_key=env)` | one client for the research side; prices registered from `model/endpoint.py:55` | +| Table | `pandas` as extra `[hunt]` | the feature table, masks, joins with labels and scores | +| Statistics | `learn.experiments` now; `scipy`/`statsmodels` after the statistics item | paired bootstrap, McNemar | + +## File structure plan +``` +src/openultrasast/hunt/ + __init__.py + table.py FeatureTable: schema v1, columns, masks, read/write (jsonl + parquet), digest + producers.py PRODUCERS: fixed list of (name, fn, columns, needs); compute(root, change) -> rows + program.py HuntSignature, HuntProgram(dspy.Module: ReAct + tools), render_row, parse_output + metric.py search_metric(example, prediction) -> score, feedback text (GEPA), per-change outcome + ablation.py Manifest (pre-registered), arms as masks, run(arm) -> records, analyse -> importance table + accept.py the acceptance rule (6.1-6.8) over compile folds and held-out folds + artifact.py RunArtifact: program id, schema version, masks, data digests, usage, versions +benchmarks/valid/hunter_arm.py --program and --mask options; everything else unchanged +benchmarks/valid/training_set.py sourcing recipe apart from the pilot (catalog + verify + assemble, pilot excluded) +tests/test_hunt_*.py table, producers (planted-advisory test), program with a scripted LM, metric, + ablation masks, acceptance rule, artifact +``` + +## Components and interfaces +| Component | Intent | Requirements | Depends on | Contract | +|---|---|---|---|---| +| FeatureTable | one row per candidate location with named columns | 3.1, 3.5, 4.2 | pandas | `columns: feature..`, `label`, `provenance.*`, `fold`, `schema_version` | +| producers | compute facts from the repository only | 3.2, 3.3, 3.4, 3.7 | source, semantic, mapping | `compute(root, change) -> list[dict]`; absent feature = `None` + `provenance..reason` | +| HuntProgram | the search | 3.5, 3.6, 5.1 | dspy, producers (rendered row) | input: rendered row + hunks; output: list of `{path, line, family ∈ FAMILY_IDS, confidence, rationale, cited_lines}` | +| search_metric | per-change score and feedback | 4.3, 7.1 | unseen.score tolerance | hit on fix site with right family = 1; ordinary push suspicions counted; feedback text names misses | +| ablation | arms as masks, paired runs | 4.1 to 4.6, 4.8 | table, program, runner, experiments | manifest: table digest, masks, folds, seed, budget, rule | +| accept | train and holdout rule | 6.1 to 6.8 | ablation results | returns accepted|rejected with both fold results and the regression budget check | +| RunArtifact | the record of a run | 5.1, 5.2, 5.5, 9.5 | dspy save | content-addressed JSON + the saved program + the DSPy cache directory copied | + +### FeatureTable schema v1 (one row per candidate location) +| Column group | Columns | Producer | +|---|---|---| +| key | `change_id`, `repo`, `head`, `path`, `line`, `kind` (vulnerability or ordinary), `fold` | manifest | +| label | `label` (1 fix site, 0 ordinary, null unknown), `family`, `fix_span` | labels | +| diff | `feature.diff.hunks` (list of (start, end) at head), `feature.diff.on_changed_line`, `feature.diff.distance` | producers.diff | +| source | `feature.source.function`, `.span`, `.callers` (list of (path, line)), `.entry_points`, `.language` | source.facts_for | +| semantic | `feature.semantic.roles` (source/sink/sanitizer per identifier), `.def_use` (binds and calls in the function), `.obligations` | semantic facts | +| taxonomy | `feature.taxonomy.families` (closed list with one-line definitions) | ruleset/families.toml | +| provenance | `provenance..{version, ok, reason, bytes_read}` | each producer | +A candidate location is every function the change touched at head (ordinary pushes) or the labelled fix site plus +every other touched function (positives; the non-site rows carry `label` null, 1.3). + +### HuntSignature +Inputs (each an `InputField`, present only when its column is unmasked): `change_summary`, `hunks`, +`function_text`, `callers`, `entry_points`, `roles`, `def_use`, `obligations`, `families`. Output: `findings` +(JSON list, the classifier's answer structure). Tools (ReAct, `max_iters` bounded by the budget): `read_file`, +`grep_repo`, `find_refs` from `hunter_tools`, path-clamped. An answer with a family outside `FAMILY_IDS` or a +cited line outside the shown text is recorded `unknown` and scores against the program (3.6, 3.7). + +### Optimisation and acceptance (one change per round, 6.8) +1. Compile folds and held-out folds come from `learn.folds` by repository (2.5); the pilot is refused by digest. +2. A round changes one thing: a producer mask (ablation), an instruction (GEPA or MIPROv2 proposal), or a + demonstration set (BootstrapFewShot). DSPy proposes; `search_metric` scores on compile folds. +3. `accept` scores the candidate on held-out folds, per family, with intervals from the paired bootstrap; keeps + it only if compile and held-out are both non-negative, one strictly positive, no family below its regression + budget, instruction under the length cap; ties rejected; rejected candidates kept visible to the next round; + held-out never shown to the proposer (6.2 to 6.7). +4. The artifact records both fold results and the rule (6.7). + +## System flows +```mermaid +sequenceDiagram + participant M as maintainer + participant A as hunt.ablation + participant T as FeatureTable + participant G as HuntProgram (dspy) + participant R as hunter_arm.run + participant S as score + metrics + M->>A: register manifest (table digest, masks, folds, budget, rule) + A->>T: load table, apply mask for arm + A->>G: build program (artifact or baseline) + A->>R: run arm over compile folds (resumable records) + R->>S: score per change + A->>A: paired analysis vs comparison arm (bootstrap by repo, McNemar) + A->>M: importance table, decision (accept.py), artifact id +``` + +## Data flow and storage +- Tables, records and DSPy caches live outside the checkout under `$OUSAST_RESULTS/hunt//` (9.4); the run + artifact and the importance table are copied into `benchmarks/measurements/-hunt-/`. +- Budgets: money and calls per run through `dspy.track_usage()` and litellm prices (9.1, 9.5); errors recorded by + type (9.2); records resumable by `(change_id, input_digest, program_id, mask)` (9.3). + +## Error handling +- A producer that cannot compute writes `provenance..ok=false` with a reason; the row stays (3.4). +- A change with nothing readable is an instrument failure, never a clean result (8.4); more than 5% invalidates + the run (8.5). +- A parse rejection in `source.ranges` is recorded as a provenance reason, not swallowed. + +## Testing strategy +- Unit: table masks and digests; each producer on a fixture repository; the planted-advisory test (3.7); the metric + on scorer fixtures; the acceptance rule on synthetic fold results (every clause of 6.1 to 6.8). +- Program: a scripted `dspy.LM` (DSPy's dummy LM) drives ReAct through the three tools; no network in tests. +- Integration: one ablation round on a five-change fixture manifest with the scripted LM; the artifact round-trips. +- Live, budgeted: the pilot run once per accepted program (8.6), recorded under `benchmarks/measurements/`. + +## Requirements traceability +| Requirement | Components | +|---|---| +| 1.1, 1.2, 1.3, 1.4, 1.5, 1.6 | FeatureTable label columns, search_metric, training_set.py (verification record next to the label) | +| 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7 | training_set.py, learn.folds, learn.leaks audit over table columns, RunArtifact digests, pilot refusal in ablation | +| 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8 | FeatureTable schema v1, producers (planted-advisory test for 3.7), HuntSignature, program output check | +| 4.1, 4.2, 4.3, 4.4, 4.5, 4.6, 4.7, 4.8 | ablation (manifest, masks, paired runs, importance table, curve), search_metric | +| 5.1, 5.2, 5.3, 5.4, 5.5, 5.6 | HuntProgram, RunArtifact, DSPy cache per run, scorer and labels outside the editable surface, length cap in accept | +| 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 6.7, 6.8 | accept | +| 7.1, 7.2, 7.3, 7.4, 7.5 | search_metric (search stage), hunter_arm classification (second stage, unchanged), report | +| 8.1, 8.2, 8.3, 8.4, 8.5, 8.6 | hunter_arm `--program`, measurements record, instrument proof in records | +| 9.1, 9.2, 9.3, 9.4, 9.5 | RunArtifact, dspy.track_usage, error types, results directory, resumable records | + +## First increment (what task group 1 builds) +`hunt/table.py`, `hunt/producers.py` with the four existing producers, `hunt/program.py` with the baseline +signature equal to today's hunter prompt plus the hunks, `hunt/metric.py`, `--program` in `hunter_arm.py`, the +scripted-LM tests, and one baseline run over the 12 pilot positives compared with the pilot's hunter records to +prove the plumbing before any optimisation. diff --git a/.kiro/specs/hunter-learning-loop/research.md b/.kiro/specs/hunter-learning-loop/research.md new file mode 100644 index 0000000..b8ec410 --- /dev/null +++ b/.kiro/specs/hunter-learning-loop/research.md @@ -0,0 +1,122 @@ +# Research and design decisions: hunter-learning-loop + +## Summary +- **Feature**: `hunter-learning-loop` +- **Discovery scope**: Extension of the learn harness and the valid benchmark; one new runtime dependency (DSPy). +- **Maintainer direction, 2026-10-13**: "the core we need to evolve is feature engineering ... that is the deciding + factor here"; "labels for me are a pandas task"; DSPy for the hunter program first, statistics libraries second, + no OpenAI SDK on its own ("unless dspy pulls it in"; it does, through litellm); lizard against its alternatives is worth + pursuing; labels and folds are a pandas and scikit-learn task; Semgrep OSS is a good idea. Each of those is a + feature producer or a feature consumer, and the feature table decides between them. +- **Key findings**: + - The pilot (benchmarks/measurements/2026-10-12-valid-benchmark-pilot) measured the hunter at 5 to 6 of 12 and 54 + false alarms per 100 pushes after the classifier; 250 of 343 "vulnerable" suspicions on ordinary pushes sit more + than 10 lines from the diff. The hunter never sees the diff (01-flowcharts/U6-hunter-client.md). + - The harness already has the pieces the loop needs: labels (`learn/labels.py`), repository folds and leak audit + (`learn/folds.py`, `learn/leaks.py`), paired experiments with cluster bootstrap and McNemar + (`learn/experiments.py`), the cached caller (`learn/program.py`), and the runner that joins hunter output to + frozen changes (`benchmarks/valid/hunter_arm.py`). What it lacks is a feature table and a program runtime that + can be optimised rather than hand-edited. + - `source.facts_for` (P1, commit 062a7b7) is the first feature producer: language, enclosing function, span, + callers, entry points, from the repository only. + +## Research log + +### DSPy as the program runtime +- **Context**: requirements R3 to R6 need a signature with switchable inputs, demonstrations, instruction + proposals, a scorer-driven compile, and metered, replayable model calls. +- **Sources**: local install, `dspy==3.4.1` into the research venv (`uv pip install --dry-run dspy`: 42 packages, + among them `litellm 1.105`, `openai 2.54`, `gepa 0.1.4`, `diskcache`, `json_repair`, `tenacity`, `pydantic`); + API introspection of the installed package (`dspy.LM(model, cache=True, num_retries=3, ...)`, + `dspy.ReAct(signature, tools, max_iters=20)`, `dspy.Predict`, `dspy.Signature` with `InputField`/`OutputField`, + `dspy.Evaluate(devset, metric, num_threads)`, `dspy.BootstrapFewShot`, `dspy.MIPROv2(metric, auto=...)`, + `dspy.GEPA(metric, auto, reflection_lm, candidate_selection_strategy="pareto")`, `dspy.track_usage()`, + `dspy.Module.save(path)`). +- **Findings**: `dspy.LM` reaches DeepSeek and OpenRouter through litellm with an OpenAI-compatible base URL and the + key from the environment; the project's model id `deepseek-flash` is not in litellm's price table, so prices are + registered from our table (`model/endpoint.py:55`). DSPy caches responses on disk (`diskcache`) keyed by the + request; a run's cache directory is settable, so a run can be replayed from its own directory. `dspy.ReAct` is + the tool-calling loop the hunter already is (read_file, grep_repo, find_refs). GEPA is the optimiser the + learning-harness brief described (per-instance Pareto archive, reflective edits); MIPROv2 is the instruction and + demo search. Neither applies our acceptance rule; that stays outside DSPy (R6). +- **Implications**: the hunter program is written as a DSPy module from the start; the classifier + (`learn/program.py`) stays as is for this spec (R7.5). The hook's urllib client is untouched: the hook has no LLM + tier yet, and the core install stays PyYAML-only; DSPy is a research extra. + +### The feature table (labels and features as a pandas task) +- **Context**: R1, R3, R4 need one row per candidate location with named feature columns, a label, provenance, and + the ability to switch columns off per ablation arm. +- **Findings**: `learn/features.py` builds feature records as nested dicts and parts; `learn/labels.py` (979 lines) + joins sources by hand; `benchmarks/unseen/score.py` joins records by id. All of it is group-by, merge and + dedup work. pandas and scikit-learn (grouped folds, calibration, metrics) are what the maintainer expects here and are pulled + by none of the current extras. +- **Implications**: the loop's contract is a DataFrame (persisted as JSONL or parquet) with a fixed column schema; + producers add columns, ablation arms drop columns, the program renders a row into its prompt. Existing dict-based + builders are consumed, not rewritten, in this spec; migrating labels to pandas is part of the statistics item, + not this one. + +### Feature producers and the tools under evaluation +- **Context**: the annotation model (R3) lists the facts; the maintainer wants the producers chosen by what gives + the best features. +- **Candidates**: `source.facts_for` (ours; P1), the semantic overlay facts (roles, def-use, obligations; + `semantic/`), `mapping.analyze_entry_points`, lizard (function spans, nesting, cyclomatic complexity; pure + Python), universal-ctags (spans across many languages; a binary), Semgrep OSS or Opengrep (pattern + and taint findings as features; also a candidate to replace the quick-rule engine, a separate decision). +- **Status**: a measured comparison of lizard against our two engines on the pair corpus is running + (`~/ousast-results/fossil/span-eval/REPORT.md`); a licensing and capability brief for Semgrep CE and Opengrep is + being written (`~/ousast-results/fossil/semgrep-brief.md`). Both land as producers behind the same contract; + the ablation decides whether their columns stay. +- **Research needed**: Semgrep registry rule licence terms for an open-source tool. + +### Training data apart from the pilot +- **Context**: R2. The pilot (252 changes, 12 positives) is held out. +- **Findings**: `benchmarks/valid/catalog.py` and `verify.py` produce fix-site positives from OSV advisories in + about 12 accepted per 32 advisories; the used-set guard (`benchmarks/unseen/*`) excludes training-only sources. + `learn/folds.py` groups by repository; `learn/leaks.py` audits per-signal separation. +- **Implications**: about 100 advisories give about 40 positives; negatives at 20 per positive from the same + repositories; folds by repository; the pilot is never read by the loop. + +## Architecture pattern evaluation +| Option | Description | Strengths | Risks | Notes | +|---|---|---|---|---| +| A. Extend `learn/program.py` with a hunter signature | one harness, our caller and cache | no new dependency | the compile loop has no train-vs-holdout rule, no tool loop, no optimiser beyond demos+instructions; everything the maintainer asked for (statistical mutation of prompts) is to build | rejected | +| B. New `hunt/` package on DSPy, feature table as the contract, harness reused for labels, folds, leaks, paired analysis | the optimisers, the tool loop, caching and usage come with the library; our code is the feature producers, the table, the acceptance rule, the scoring | one heavy research dependency; two caches (DSPy's and `Caller`'s) until the classifier migrates | selected | +| C. Rewrite the classifier onto DSPy too | one runtime | large rewrite of a parked line | later, if B earns it | + +## Design decisions +### Decision: DSPy runs the hunter program; the acceptance rule stays ours +- Alternatives: A above; a hand-written optimiser. +- Selected: `dspy.ReAct` over a `HuntSignature` whose inputs are the rendered feature row plus the changed hunks; + `dspy.GEPA` or `MIPROv2` propose candidates; `dspy.Evaluate` scores them on compile folds with our metric; the + candidate is accepted only by our rule on held-out folds (R6), implemented outside DSPy. +- Trade-offs: DSPy's cache is request-keyed, not content-addressed; a run pins its cache directory and copies it + with the run record. Cost comes from `dspy.track_usage()` with our prices registered in litellm. + +### Decision: the feature table is the contract between producers, ablation and the program +- Selected: one row per (change, path, line) candidate with columns `feature..`, `label`, + `provenance.*`, `fold`; producers are functions `(root, change, path, line) -> dict`; an ablation arm is a + column mask recorded in the manifest; the program sees only the unmasked columns, rendered in a fixed order. +- Trade-offs: a table per run is larger than nested records; it is also what pandas, scipy and the maintainer read. + +### Decision: producers are a fixed list, switched by name, no registry +- `source.facts_for`, `semantic` facts (roles, def-use, obligations), `mapping` entry points, `diff` hunks; lizard and + Semgrep/Opengrep join the list only after their evaluation reports. + +### Decision: objective and metric +- Search stage: recall over verified fix sites (a suspicion within the scorer's tolerance of the fix site with the + right family) and suspicions per ordinary push; the DSPy metric returns the per-change score and, for GEPA, a + text feedback naming the missed site or the off-diff suspicions. No threshold is tuned (R4.7); the curve is + reported (R4.8). + +## Risks and mitigations +- DSPy or litellm upgrades change behaviour: pin `dspy==3.4.*`, record versions in every run artifact. +- A producer reads a label by accident: producers take the repository and a location only; a test feeds a + repository with a planted advisory id and asserts no feature carries it (R3.7). +- Train-on-test: the pilot path is denied to the loop by construction (the runner refuses the pilot manifest + digest), and every artifact lists the data digests it read (R2.4). +- Cost: budgets per run (R9.1); DSPy cache on; `--replay-only` reruns at no cost. + +## References +- DSPy 3.4.1 (installed; https://dspy.ai), GEPA (https://github.com/gepa-ai/gepa), litellm pricing registration. +- PATHFINDER-2026-10-13/01-flowcharts/U6-hunter-client.md, U7-learn-memory-tasks.md; 02-duplication-report.md D1, D4. +- benchmarks/measurements/2026-10-12-valid-benchmark-pilot/NOTES.md (numbers and the filter sweep). diff --git a/.kiro/specs/hunter-learning-loop/spec.json b/.kiro/specs/hunter-learning-loop/spec.json index 1027cf0..a309d28 100644 --- a/.kiro/specs/hunter-learning-loop/spec.json +++ b/.kiro/specs/hunter-learning-loop/spec.json @@ -1,16 +1,17 @@ { "feature_name": "hunter-learning-loop", "created_at": "2026-10-13T09:05:00+02:00", - "updated_at": "2026-10-13T09:20:00+02:00", + "updated_at": "2026-10-13T13:30:00+02:00", "language": "en", - "phase": "requirements-generated", + "phase": "design-generated", "approvals": { "requirements": { "generated": true, - "approved": false + "approved": true, + "note": "maintainer 2026-10-13: '4 first' (DSPy for the hunter program)" }, "design": { - "generated": false, + "generated": true, "approved": false }, "tasks": { @@ -19,4 +20,4 @@ } }, "ready_for_implementation": false -} +} \ No newline at end of file