diff --git a/.kiro/specs/hunter-learning-loop/requirements.md b/.kiro/specs/hunter-learning-loop/requirements.md new file mode 100644 index 0000000..ed0d98f --- /dev/null +++ b/.kiro/specs/hunter-learning-loop/requirements.md @@ -0,0 +1,272 @@ +# Requirements Document + +## Project Description (Input) + +Maintainer, 2026-10-13: "the hunter would then need a dspy process now, so that we iteratively refine our +llm integration. we can distinct positives from false positives. and dspy can mutate prompts with its +statistical tools. the key is to produce enough information for the llm to use it correctly." Then: +"threshold based assessments come later. right now the search is too bad. if I cut a threshold at 50:50 +that is coinflip optimization. what we need rather than that is an annotate model for the llm, with +statistical ablation." And: "the answer to our problem is feature engineering + math." Before that: +"proof should be a resort to face with certainty. before our search problem needs to be solved to reduce +what we have to proof." + +Who has the problem: the pre-push safety net. The valid benchmark pilot +(`benchmarks/measurements/2026-10-12-valid-benchmark-pilot/NOTES.md`) measured the LLM tool hunter at +5 of 12 verified positives found and 54 false alarms per 100 ordinary pushes after the decision-engine +classifier. The hook as shipped found 0 of 12. + +Current situation: the hunter's search prompt is a hand-written string, its family answer is free text, +and it never sees the diff. Of 343 ordinary-push suspicions the classifier called `vulnerable`, 250 sit +more than 10 lines from any changed line: the hunter re-reports the repo's old code on every push. Every +precision gain so far was a hand rule or a threshold picked after looking at the misses, in-sample. A +threshold on today's output is a coin flip dressed as a decision. The learning harness that compiles the +decision-engine classifier (bootstrapped demonstrations, instruction proposals, grouped folds, a leak +audit, paired A/B experiments with cluster bootstrap, a cached and replayable model caller) exists, but +the hunter does not run on it, and the compile step accepts the best score on one validation split with +no train-against-holdout check, no per-family regression check, and no prompt length cap. + +What should change: the search gets better before anything is thresholded. The hunter receives an +annotation model: a named, versioned set of facts about the push computed from the repository (changed +hunks with line numbers, the enclosing function and its callers, entry-point evidence, source, sink and +sanitizer roles, framework facts, def-use inside the function, guards and obligations, closed taxonomy +families with the classifier's answer format). Which facts the model needs is decided by statistical +ablation: each annotation is switched on and off in pre-registered, paired experiments over the same +changes, and its effect on search recall over fix sites and on suspicions per ordinary push is reported +per family with intervals. An annotation or a prompt change is kept only when it helps on held-out folds. +Labels come from the benchmark's rule: a suspicion is a true positive when it lands on a verified fix +site, a false positive when it sits on an ordinary push. Training data is sourced separately (about 100 +advisories through the fix-site catalog and verifier, about 40 accepted positives) with repository-level +folds and the leak audit; the 252-change pilot stays held out and is only used to report the gain. +Thresholds and operating points come after the search is good; this spec reports the whole recall +against false-alarm curve and sets none. The proof arm runs only on what survives. + +## Introduction + +This spec turns the LLM hunter from a hand-written prompt into a measured search stage. It owns four +things: the labelled data the loop learns from, the annotation model the hunter is given about a push, +the ablation method that decides which annotations stay, and the rule by which a changed prompt or +annotation set is kept or thrown away. It reports in the benchmark's own terms: recall, false alarms per +100 pushes, precision, per family, with intervals. It sets no threshold. + +## Boundary Context + +- **In scope**: labelling hunter suspicions from fix-site positives and ordinary pushes; sourcing a + training set separate from the held-out pilot; the annotation model and its producers; statistical + ablation of annotations; the hunter as a versioned, replayable compiled program; the acceptance rule + for a prompt, demonstration or annotation-set change; re-measuring the pilot and reporting the gain. +- **Out of scope**: choosing confidence or scope thresholds and operating points (later, once the search + is measured good); wiring the compiled hunter into the pre-push hook (`pre-push-safety-net`); the proof + arm itself (`search-with-proof`, `proof-arm-opencode`); building new deterministic analyses such as a + whole-repo call graph or taint engine (`engine-change-scoped-analysis` owns those; this spec consumes + what exists); growing the benchmark beyond what this loop needs (`valid-detection-benchmark`). +- **Adjacent expectations**: the fix-site catalog and verifier produce the labels; the benchmark scorer + and metrics are the only scorekeepers; the decision-engine classifier stays as the second stage and is + not retrained here; the earlier freeze on hunter changes ("no hunter prompt, tool or configuration may + change before that measurement") is lifted by the pilot measurement, which is the baseline. + +## Requirements + +### Requirement 1: Labels from fix sites and ordinary pushes + +**Objective:** As the maintainer, I want every hunter suspicion labelled true or false by the benchmark's +own rule, so that the loop learns from facts and not from my reading of the misses. + +#### Acceptance Criteria +1. When a hunter suspicion names a path and line inside a verified fix-site function of a positive change, + or within the benchmark scorer's location tolerance of it, and its family equals the change's family, + the Learning Loop shall label it a true positive. +2. When a hunter suspicion is raised on an ordinary push, the Learning Loop shall label it a false positive. +3. When a hunter suspicion is raised on a positive change but outside every verified fix site of that + change, the Learning Loop shall label it unknown and shall exclude it from training and scoring. +4. The Learning Loop shall label at the level of the suspicion (path, line, family), and shall also derive + the push-level outcome (caught, alarmed) with the benchmark scorer, so that both the search stage and + the push-level metric are measurable from the same records. +5. When a positive change is used for training, the Learning Loop shall record its fix-site verification + record (model checks and human decision) next to the label, so that a label can be traced to its + evidence. +6. If a label's source cannot be traced to a verified fix site or a frozen ordinary draw, the Learning + Loop shall reject it. + +### Requirement 2: A training set apart from the held-out pilot + +**Objective:** As the maintainer, I want the loop to learn on data the benchmark never reports on, so that +the pilot number stays a measurement. + +#### Acceptance Criteria +1. The Learning Loop shall source its training positives through the fix-site catalog and verifier from + advisories and repositories that are not in the held-out pilot, and shall record the sourcing run. +2. The Learning Loop shall draw its training negatives as ordinary pushes from the training repositories + only, at the benchmark's ratio of at least 20 ordinary pushes per positive change. +3. The training set shall contain at least 40 accepted positive changes across at least two languages + before the first ablation result is reported; a run on fewer is marked a dry run. +4. While an ablation, compile or evaluation runs, the Learning Loop shall never read the held-out pilot's + changes, labels or records, and every artifact shall record which data digests it did read. +5. The Learning Loop shall split the training set by repository group into folds, so that no repository + appears on both sides of a split. +6. Before the first ablation, the Learning Loop shall run the leak audit over every annotation and shall + report any annotation that separates positives from negatives on its own above the audit's threshold, + so that an annotation that encodes the label is found before it is credited. +7. If a training repository is later found in the held-out pilot or in any benchmark report, the Learning + Loop shall mark every run that used it as contaminated and shall exclude those runs from any reported + gain. + +### Requirement 3: The annotation model + +**Objective:** As the maintainer, I want the facts the hunter sees about a push to be a named, versioned +set that each can be switched on or off, so that "enough information for the model" is something we can +measure rather than guess. + +#### Acceptance Criteria +1. The Learning Loop shall define the annotation model as a versioned schema of named annotations, each + with a producer, a bound on its size, and a digest, and shall record the schema version on every + hunter run. +2. The Learning Loop shall compute every annotation from the repository at the push's revisions and from + the tool's own framework knowledge only; no annotation shall read a label, an advisory, a CVE or GHSA + identifier, a fix commit, or security-worded commit text. +3. The annotation model shall include at least: the changed hunks with head-side line numbers; the + enclosing function's full text at head; the names and locations of that function's callers; entry-point + evidence (an in-repo route, handler, command or public API that reaches the function); source, sink and + sanitizer roles for the identifiers in the function; the framework facts that apply to the file; + def-use evidence inside the function; the guards and obligations on the changed path; and the closed + list of taxonomy families with a one-line definition each. +4. When an annotation's producer cannot compute it for a push, the Learning Loop shall record the + annotation as absent with the reason, and shall not substitute a default that could be read as a fact. +5. The Learning Loop shall render the enabled annotations into the hunter's input in a fixed order with + fixed labels, so that two runs with the same annotation set and the same push produce the same input. +6. The Learning Loop shall require the hunter's family answer to come from the closed list; an answer + outside the list is recorded as `unknown` and counts against the program. +7. The Learning Loop shall require the hunter's answer in the same fixed structure the classifier uses + (verdict, family, confidence, rationale, cited lines), and shall reject an answer whose cited lines + fall outside the text it was shown. +8. When the hunter reports a suspicion, the Learning Loop shall record whether it lies on a changed line, + within 10 lines of one, or farther away, so that scope is a measured property of every suspicion and + not a filter. + +### Requirement 4: Statistical ablation decides what the model needs + +**Objective:** As the maintainer, I want each annotation's worth measured by switching it off, so that the +input the hunter gets is built from effects with intervals, not from intuition. + +#### Acceptance Criteria +1. Before an ablation runs, the Learning Loop shall register it: the annotation set under test, the + annotations switched off, the folds, the metric, the sample size, the budget and the decision rule. +2. The Learning Loop shall run every ablation arm over the same changes as its comparison arm, paired by + change, with the same program, model and sampling, so that the only difference between arms is the + annotation set. +3. For each ablation, the Learning Loop shall report the paired effect on search recall over fix sites + and on suspicions per ordinary push, per family and pooled, with a 95% interval from a bootstrap + clustered by repository and a paired test on the per-change outcomes. +4. The Learning Loop shall report, after a full ablation round, an importance table: for every + annotation, its effect on recall and on suspicions per ordinary push when removed from the full set and + when added to the baseline, with intervals and sample counts. +5. The Learning Loop shall keep an annotation in the hunter's input only when its removal lowers search + recall or raises suspicions per ordinary push on held-out folds with an interval that excludes zero, + or when its removal has no measurable effect and it is needed by an annotation that does. +6. If an annotation raises suspicions per ordinary push without raising recall, the Learning Loop shall + drop it and shall record the result where the next round can see it. +7. The Learning Loop shall not use a confidence cut, a scope cut or any other threshold on the hunter's + output as the quantity being optimised; confidence is reported, not tuned, in this spec. +8. The Learning Loop shall report the full recall against suspicions-per-push curve of each annotation + set, so that a later operating-point decision has the whole curve and not one point of it. + +### Requirement 5: The hunter as a compiled, replayable program + +**Objective:** As the maintainer, I want every version of the hunter's prompt, demonstrations and +annotation set to be an artifact I can name, diff, replay and roll back, so that an improvement is a fact +and not a memory. + +#### Acceptance Criteria +1. The Learning Loop shall store each compiled hunter as a single content-addressed artifact holding the + instruction, the demonstrations, the annotation schema version and enabled set, the model, the data + digests, every scored configuration and the spend. +2. When the hunter runs, the Learning Loop shall record the program id and the annotation set on every + suspicion it emits. +3. The Learning Loop shall route every hunter model call through a cached, metered caller, so that a + rerun of the same program on the same inputs reproduces the same answers at no cost. +4. The Learning Loop shall keep the scorer, the labels and the held-out data outside every surface the + compile step may edit, so that a proposal cannot read its own answer key. +5. If a run exhausts its budget before finishing, the Learning Loop shall write no program artifact and + shall report the run as unfinished with the spend so far. +6. The Learning Loop shall cap the instruction length per family, and shall reject a proposal that + exceeds the cap regardless of its score. + +### Requirement 6: Accepting or rejecting a change + +**Objective:** As the maintainer, I want a prompt, demonstration or annotation-set change kept only when +it helps on data it was not tuned on, so that the loop cannot overfit its way to a better number. + +#### Acceptance Criteria +1. When a candidate program is scored, the Learning Loop shall score it on the compile folds and on the + held-out folds separately, per family, and shall report both. +2. The Learning Loop shall accept a candidate only when its compile-fold and held-out-fold results are + both non-negative against the current program and at least one is strictly positive. +3. If any family's held-out result falls below its regression budget, the Learning Loop shall reject the + candidate, whatever the aggregate shows. +4. If a candidate ties the current program within the pre-registered tolerance, the Learning Loop shall + reject it. +5. The Learning Loop shall report every result with its sample count and a 95% interval, and shall not + accept a candidate on a point estimate alone. +6. The Learning Loop shall keep rejected candidates and their scores where the next proposal round can + see them, and shall never show the held-out results to the proposer. +7. When a candidate is accepted, the Learning Loop shall record the comparison (both programs' ids, both + fold results, the decision rule) in the new program's artifact. +8. The Learning Loop shall change one thing per round (one annotation, or one instruction, or one + demonstration set), so that an accepted gain is attributable. + +### Requirement 7: Two stages, reported for what each owns + +**Objective:** As the maintainer, I want the search stage and the classifier stage measured separately, +so that we fix the search problem before paying for proof. + +#### Acceptance Criteria +1. The Learning Loop shall report the search stage on recall over verified fix sites (did a suspicion land + on the fix site with the right family) and on suspicions per ordinary push, per family. +2. The Learning Loop shall report the classifier stage on precision over the suspicions the search stage + emitted, and on the push-level false alarms per 100 ordinary pushes after it, per family. +3. The Learning Loop shall report the combined push-level result as recall, false alarms per 100 pushes + and precision with 95% intervals, in the benchmark's names and no coined ones. +4. The Learning Loop shall report the gate distance on every run: the recall reached at 10 false alarms + per 100 pushes and the false alarms at 90% recall, read off the curve, as a report and not as a tuned + operating point. +5. The Learning Loop shall not retrain or recompile the classifier stage within this spec; a classifier + change is a separate experiment. + +### Requirement 8: Measuring the gain on the pilot + +**Objective:** As the maintainer, I want one number that says what the loop bought, measured on the data +it never saw, so that the decision to put the hunter on pre-push rests on evidence. + +#### Acceptance Criteria +1. When a compiled hunter is accepted, the Learning Loop shall run it once over the held-out pilot's 252 + changes with the same scorer and metrics as the pilot, and shall write the result next to the pilot's + measurement. +2. The Learning Loop shall report the gain as the before-and-after of search recall, suspicions per + ordinary push, push-level recall, false alarms per 100 pushes and precision, per family, with + intervals, against the pilot's recorded hunter numbers. +3. The Learning Loop shall report, for the same run, the number of suspicions the proof arm would have to + handle per 100 pushes, so that proof's workload is a measured count. +4. The Learning Loop shall prove its instrument in every pilot run: for each change it shall record bytes + read per changed file and the verified base and head, and shall mark a change with nothing readable as + an instrument failure rather than a clean result. +5. If a pilot run's instrument failures exceed 5% of changes, the Learning Loop shall report the run as + invalid and shall not quote its numbers. +6. The Learning Loop shall run the pilot at most once per accepted program, and shall record each run, so + that the pilot is not used as a tuning surface. + +### Requirement 9: Cost, secrets and unattended runs + +**Objective:** As the operator, I want ablation and compile runs to stay inside a budget, keep the model +key out of every record, and finish without me, so that the loop can run overnight on the research host. + +#### Acceptance Criteria +1. The Learning Loop shall take a pre-registered budget in money and calls per run and shall stop at the + first exceeded bound. +2. The Learning Loop shall never write a model key, or an exception message that could carry one, into a + record, a log or an artifact; an error is recorded by its type. +3. When a run is interrupted, the Learning Loop shall resume from its recorded progress without repeating + finished model calls. +4. The Learning Loop shall write its outputs outside the repository checkout and shall keep each record + once, so that an overnight run cannot fill the host's disk with duplicates. +5. The Learning Loop shall report, at the end of every run, the spend, the calls, the replay hits and the + wall time. diff --git a/.kiro/specs/hunter-learning-loop/spec.json b/.kiro/specs/hunter-learning-loop/spec.json new file mode 100644 index 0000000..1027cf0 --- /dev/null +++ b/.kiro/specs/hunter-learning-loop/spec.json @@ -0,0 +1,22 @@ +{ + "feature_name": "hunter-learning-loop", + "created_at": "2026-10-13T09:05:00+02:00", + "updated_at": "2026-10-13T09:20:00+02:00", + "language": "en", + "phase": "requirements-generated", + "approvals": { + "requirements": { + "generated": true, + "approved": false + }, + "design": { + "generated": false, + "approved": false + }, + "tasks": { + "generated": false, + "approved": false + } + }, + "ready_for_implementation": false +}