Open_Ultra_SAST/.kiro/specs/hunter-learning-loop/requirements.md

# Requirements Document

## Project Description (Input)

Maintainer, 2026-10-13: "the hunter would then need a dspy process now, so that we iteratively refine our
llm integration. we can distinct positives from false positives. and dspy can mutate prompts with its
statistical tools. the key is to produce enough information for the llm to use it correctly." Then:
"threshold based assessments come later. right now the search is too bad. if I cut a threshold at 50:50
that is coinflip optimization. what we need rather than that is an annotate model for the llm, with
statistical ablation." And: "the answer to our problem is feature engineering + math." Before that:
"proof should be a resort to face with certainty. before our search problem needs to be solved to reduce
what we have to proof."

Who has the problem: the pre-push safety net. The valid benchmark pilot
(`benchmarks/measurements/2026-10-12-valid-benchmark-pilot/NOTES.md`) measured the LLM tool hunter at
5 of 12 verified positives found and 54 false alarms per 100 ordinary pushes after the decision-engine
classifier. The hook as shipped found 0 of 12.

Current situation: the hunter's search prompt is a hand-written string, its family answer is free text,
and it never sees the diff. Of 343 ordinary-push suspicions the classifier called `vulnerable`, 250 sit
more than 10 lines from any changed line: the hunter re-reports the repo's old code on every push. Every
precision gain so far was a hand rule or a threshold picked after looking at the misses, in-sample. A
threshold on today's output is a coin flip dressed as a decision. The learning harness that compiles the
decision-engine classifier (bootstrapped demonstrations, instruction proposals, grouped folds, a leak
audit, paired A/B experiments with cluster bootstrap, a cached and replayable model caller) exists, but
the hunter does not run on it, and the compile step accepts the best score on one validation split with
no train-against-holdout check, no per-family regression check, and no prompt length cap.

What should change: the search gets better before anything is thresholded. The hunter receives an
annotation model: a named, versioned set of facts about the push computed from the repository (changed
hunks with line numbers, the enclosing function and its callers, entry-point evidence, source, sink and
sanitizer roles, framework facts, def-use inside the function, guards and obligations, closed taxonomy
families with the classifier's answer format). Which facts the model needs is decided by statistical
ablation: each annotation is switched on and off in pre-registered, paired experiments over the same
changes, and its effect on search recall over fix sites and on suspicions per ordinary push is reported
per family with intervals. An annotation or a prompt change is kept only when it helps on held-out folds.
Labels come from the benchmark's rule: a suspicion is a true positive when it lands on a verified fix
site, a false positive when it sits on an ordinary push. Training data is sourced separately (about 100
advisories through the fix-site catalog and verifier, about 40 accepted positives) with repository-level
folds and the leak audit; the 252-change pilot stays held out and is only used to report the gain.
Thresholds and operating points come after the search is good; this spec reports the whole recall
against false-alarm curve and sets none. The proof arm runs only on what survives.

## Introduction

This spec turns the LLM hunter from a hand-written prompt into a measured search stage. It owns four
things: the labelled data the loop learns from, the annotation model the hunter is given about a push,
the ablation method that decides which annotations stay, and the rule by which a changed prompt or
annotation set is kept or thrown away. It reports in the benchmark's own terms: recall, false alarms per
100 pushes, precision, per family, with intervals. It sets no threshold.

## Boundary Context

- **In scope**: labelling hunter suspicions from fix-site positives and ordinary pushes; sourcing a
  training set separate from the held-out pilot; the annotation model and its producers; statistical
  ablation of annotations; the hunter as a versioned, replayable compiled program; the acceptance rule
  for a prompt, demonstration or annotation-set change; re-measuring the pilot and reporting the gain.
- **Out of scope**: choosing confidence or scope thresholds and operating points (later, once the search
  is measured good); wiring the compiled hunter into the pre-push hook (`pre-push-safety-net`); the proof
  arm itself (`search-with-proof`, `proof-arm-opencode`); building new deterministic analyses such as a
  whole-repo call graph or taint engine (`engine-change-scoped-analysis` owns those; this spec consumes
  what exists); growing the benchmark beyond what this loop needs (`valid-detection-benchmark`).
- **Adjacent expectations**: the fix-site catalog and verifier produce the labels; the benchmark scorer
  and metrics are the only scorekeepers; the decision-engine classifier stays as the second stage and is
  not retrained here; the earlier freeze on hunter changes ("no hunter prompt, tool or configuration may
  change before that measurement") is lifted by the pilot measurement, which is the baseline.

## Requirements

### Requirement 1: Labels from fix sites and ordinary pushes

**Objective:** As the maintainer, I want every hunter suspicion labelled true or false by the benchmark's
own rule, so that the loop learns from facts and not from my reading of the misses.

#### Acceptance Criteria
1. When a hunter suspicion names a path and line inside a verified fix-site function of a positive change,
   or within the benchmark scorer's location tolerance of it, and its family equals the change's family,
   the Learning Loop shall label it a true positive.
2. When a hunter suspicion is raised on an ordinary push, the Learning Loop shall label it a false positive.
3. When a hunter suspicion is raised on a positive change but outside every verified fix site of that
   change, the Learning Loop shall label it unknown and shall exclude it from training and scoring.
4. The Learning Loop shall label at the level of the suspicion (path, line, family), and shall also derive
   the push-level outcome (caught, alarmed) with the benchmark scorer, so that both the search stage and
   the push-level metric are measurable from the same records.
5. When a positive change is used for training, the Learning Loop shall record its fix-site verification
   record (model checks and human decision) next to the label, so that a label can be traced to its
   evidence.
6. If a label's source cannot be traced to a verified fix site or a frozen ordinary draw, the Learning
   Loop shall reject it.

### Requirement 2: A training set apart from the held-out pilot

**Objective:** As the maintainer, I want the loop to learn on data the benchmark never reports on, so that
the pilot number stays a measurement.

#### Acceptance Criteria
1. The Learning Loop shall source its training positives through the fix-site catalog and verifier from
   advisories and repositories that are not in the held-out pilot, and shall record the sourcing run.
2. The Learning Loop shall draw its training negatives as ordinary pushes from the training repositories
   only, at the benchmark's ratio of at least 20 ordinary pushes per positive change.
3. The training set shall contain at least 40 accepted positive changes across at least two languages
   before the first ablation result is reported; a run on fewer is marked a dry run.
4. While an ablation, compile or evaluation runs, the Learning Loop shall never read the held-out pilot's
   changes, labels or records, and every artifact shall record which data digests it did read.
5. The Learning Loop shall split the training set by repository group into folds, so that no repository
   appears on both sides of a split.
6. Before the first ablation, the Learning Loop shall run the leak audit over every annotation and shall
   report any annotation that separates positives from negatives on its own above the audit's threshold,
   so that an annotation that encodes the label is found before it is credited.
7. If a training repository is later found in the held-out pilot or in any benchmark report, the Learning
   Loop shall mark every run that used it as contaminated and shall exclude those runs from any reported
   gain.

### Requirement 3: The annotation model

**Objective:** As the maintainer, I want the facts the hunter sees about a push to be a named, versioned
set that each can be switched on or off, so that "enough information for the model" is something we can
measure rather than guess.

#### Acceptance Criteria
1. The Learning Loop shall define the annotation model as a versioned schema of named annotations, each
   with a producer, a bound on its size, and a digest, and shall record the schema version on every
   hunter run.
2. The Learning Loop shall compute every annotation from the repository at the push's revisions and from
   the tool's own framework knowledge only; no annotation shall read a label, an advisory, a CVE or GHSA
   identifier, a fix commit, or security-worded commit text.
3. The annotation model shall include at least: the changed hunks with head-side line numbers; the
   enclosing function's full text at head; the names and locations of that function's callers; entry-point
   evidence (an in-repo route, handler, command or public API that reaches the function); source, sink and
   sanitizer roles for the identifiers in the function; the framework facts that apply to the file;
   def-use evidence inside the function; the guards and obligations on the changed path; and the closed
   list of taxonomy families with a one-line definition each.
4. When an annotation's producer cannot compute it for a push, the Learning Loop shall record the
   annotation as absent with the reason, and shall not substitute a default that could be read as a fact.
5. The Learning Loop shall render the enabled annotations into the hunter's input in a fixed order with
   fixed labels, so that two runs with the same annotation set and the same push produce the same input.
6. The Learning Loop shall require the hunter's family answer to come from the closed list; an answer
   outside the list is recorded as `unknown` and counts against the program.
7. The Learning Loop shall require the hunter's answer in the same fixed structure the classifier uses
   (verdict, family, confidence, rationale, cited lines), and shall reject an answer whose cited lines
   fall outside the text it was shown.
8. When the hunter reports a suspicion, the Learning Loop shall record whether it lies on a changed line,
   within 10 lines of one, or farther away, so that scope is a measured property of every suspicion and
   not a filter.

### Requirement 4: Statistical ablation decides what the model needs

**Objective:** As the maintainer, I want each annotation's worth measured by switching it off, so that the
input the hunter gets is built from effects with intervals, not from intuition.

#### Acceptance Criteria
1. Before an ablation runs, the Learning Loop shall register it: the annotation set under test, the
   annotations switched off, the folds, the metric, the sample size, the budget and the decision rule.
2. The Learning Loop shall run every ablation arm over the same changes as its comparison arm, paired by
   change, with the same program, model and sampling, so that the only difference between arms is the
   annotation set.
3. For each ablation, the Learning Loop shall report the paired effect on search recall over fix sites
   and on suspicions per ordinary push, per family and pooled, with a 95% interval from a bootstrap
   clustered by repository and a paired test on the per-change outcomes.
4. The Learning Loop shall report, after a full ablation round, an importance table: for every
   annotation, its effect on recall and on suspicions per ordinary push when removed from the full set and
   when added to the baseline, with intervals and sample counts.
5. The Learning Loop shall keep an annotation in the hunter's input only when its removal lowers search
   recall or raises suspicions per ordinary push on held-out folds with an interval that excludes zero,
   or when its removal has no measurable effect and it is needed by an annotation that does.
6. If an annotation raises suspicions per ordinary push without raising recall, the Learning Loop shall
   drop it and shall record the result where the next round can see it.
7. The Learning Loop shall not use a confidence cut, a scope cut or any other threshold on the hunter's
   output as the quantity being optimised; confidence is reported, not tuned, in this spec.
8. The Learning Loop shall report the full recall against suspicions-per-push curve of each annotation
   set, so that a later operating-point decision has the whole curve and not one point of it.

### Requirement 5: The hunter as a compiled, replayable program

**Objective:** As the maintainer, I want every version of the hunter's prompt, demonstrations and
annotation set to be an artifact I can name, diff, replay and roll back, so that an improvement is a fact
and not a memory.

#### Acceptance Criteria
1. The Learning Loop shall store each compiled hunter as a single content-addressed artifact holding the
   instruction, the demonstrations, the annotation schema version and enabled set, the model, the data
   digests, every scored configuration and the spend.
2. When the hunter runs, the Learning Loop shall record the program id and the annotation set on every
   suspicion it emits.
3. The Learning Loop shall route every hunter model call through a cached, metered caller, so that a
   rerun of the same program on the same inputs reproduces the same answers at no cost.
4. The Learning Loop shall keep the scorer, the labels and the held-out data outside every surface the
   compile step may edit, so that a proposal cannot read its own answer key.
5. If a run exhausts its budget before finishing, the Learning Loop shall write no program artifact and
   shall report the run as unfinished with the spend so far.
6. The Learning Loop shall cap the instruction length per family, and shall reject a proposal that
   exceeds the cap regardless of its score.

### Requirement 6: Accepting or rejecting a change

**Objective:** As the maintainer, I want a prompt, demonstration or annotation-set change kept only when
it helps on data it was not tuned on, so that the loop cannot overfit its way to a better number.

#### Acceptance Criteria
1. When a candidate program is scored, the Learning Loop shall score it on the compile folds and on the
   held-out folds separately, per family, and shall report both.
2. The Learning Loop shall accept a candidate only when its compile-fold and held-out-fold results are
   both non-negative against the current program and at least one is strictly positive.
3. If any family's held-out result falls below its regression budget, the Learning Loop shall reject the
   candidate, whatever the aggregate shows.
4. If a candidate ties the current program within the pre-registered tolerance, the Learning Loop shall
   reject it.
5. The Learning Loop shall report every result with its sample count and a 95% interval, and shall not
   accept a candidate on a point estimate alone.
6. The Learning Loop shall keep rejected candidates and their scores where the next proposal round can
   see them, and shall never show the held-out results to the proposer.
7. When a candidate is accepted, the Learning Loop shall record the comparison (both programs' ids, both
   fold results, the decision rule) in the new program's artifact.
8. The Learning Loop shall change one thing per round (one annotation, or one instruction, or one
   demonstration set), so that an accepted gain is attributable.

### Requirement 7: Two stages, reported for what each owns

**Objective:** As the maintainer, I want the search stage and the classifier stage measured separately,
so that we fix the search problem before paying for proof.

#### Acceptance Criteria
1. The Learning Loop shall report the search stage on recall over verified fix sites (did a suspicion land
   on the fix site with the right family) and on suspicions per ordinary push, per family.
2. The Learning Loop shall report the classifier stage on precision over the suspicions the search stage
   emitted, and on the push-level false alarms per 100 ordinary pushes after it, per family.
3. The Learning Loop shall report the combined push-level result as recall, false alarms per 100 pushes
   and precision with 95% intervals, in the benchmark's names and no coined ones.
4. The Learning Loop shall report the gate distance on every run: the recall reached at 10 false alarms
   per 100 pushes and the false alarms at 90% recall, read off the curve, as a report and not as a tuned
   operating point.
5. The Learning Loop shall not retrain or recompile the classifier stage within this spec; a classifier
   change is a separate experiment.

### Requirement 8: Measuring the gain on the pilot

**Objective:** As the maintainer, I want one number that says what the loop bought, measured on the data
it never saw, so that the decision to put the hunter on pre-push rests on evidence.

#### Acceptance Criteria
1. When a compiled hunter is accepted, the Learning Loop shall run it once over the held-out pilot's 252
   changes with the same scorer and metrics as the pilot, and shall write the result next to the pilot's
   measurement.
2. The Learning Loop shall report the gain as the before-and-after of search recall, suspicions per
   ordinary push, push-level recall, false alarms per 100 pushes and precision, per family, with
   intervals, against the pilot's recorded hunter numbers.
3. The Learning Loop shall report, for the same run, the number of suspicions the proof arm would have to
   handle per 100 pushes, so that proof's workload is a measured count.
4. The Learning Loop shall prove its instrument in every pilot run: for each change it shall record bytes
   read per changed file and the verified base and head, and shall mark a change with nothing readable as
   an instrument failure rather than a clean result.
5. If a pilot run's instrument failures exceed 5% of changes, the Learning Loop shall report the run as
   invalid and shall not quote its numbers.
6. The Learning Loop shall run the pilot at most once per accepted program, and shall record each run, so
   that the pilot is not used as a tuning surface.

### Requirement 9: Cost, secrets and unattended runs

**Objective:** As the operator, I want ablation and compile runs to stay inside a budget, keep the model
key out of every record, and finish without me, so that the loop can run overnight on the research host.

#### Acceptance Criteria
1. The Learning Loop shall take a pre-registered budget in money and calls per run and shall stop at the
   first exceeded bound.
2. The Learning Loop shall never write a model key, or an exception message that could carry one, into a
   record, a log or an artifact; an error is recorded by its type.
3. When a run is interrupted, the Learning Loop shall resume from its recorded progress without repeating
   finished model calls.
4. The Learning Loop shall write its outputs outside the repository checkout and shall keep each record
   once, so that an overnight run cannot fill the host's disk with duplicates.
5. The Learning Loop shall report, at the end of every run, the spend, the calls, the replay hits and the
   wall time.