docs(constrained-detector): requirements, with real-world code as the decider

Renamed from detector-optimizer: the brief's own conclusion is that the
optimizer is not the next thing to build, and a spec should be named for what
it delivers.

Seven requirements, 37 criteria. The load-bearing ones:

Req 1 -- candidates come from the semantic IR, never the pattern ruleset,
and the enumerator's ceiling is published as the limit on recall. Measured
before building: ruleset 12.1%, IR 96.6%.

Req 2 -- one bounded typed question per candidate, no tools, no choice of
what to look at, and a test asserts two runs make an identical sequence of
model calls.

Req 3 is the maintainer's emphasis and it decides acceptance. The slices
harvested from real projects decide; hand-written vibe-py is a development
slice that cannot on its own justify a change; a change that improves vibe-py
and regresses a deciding slice is rejected by name; and the overfitting gap
-- 83.3% against 44.4% at baseline -- is published beside every headline
number. The candidate ceiling is measured separately on real-world slices,
because their functions are larger and the enumerator's behaviour there is
not implied by its behaviour on teaching code.

Req 5 -- the K-run and noise-floor machinery is re-measured against the
constrained detector and deleted if the floor collapses, naming the
learning-harness requirements that would be amended.

Not approved; design not started.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG

.kiro/specs/constrained-detector/brief.md

@@ -0,0 +1,340 @@
+# Brief: the detector optimizer
+
+## Problem
+
+`learning-harness` built the measurement layer honestly — a closed family taxonomy, a class-aware pair
+scorer, per-family detector configurations, a split enforced by construction, verifier-gated reporting,
+and a journal with attribution. What it did **not** build is a search. `run_learning_round` is a
+hand-rolled hill-climber: one proposal, one minibatch, one holdout check, a threshold rule, accept or
+revert. Every defect four review rounds found in it was a defect of that improvised loop rather than of
+the measurement underneath it.
+
+The maintainer's judgement, and the reason for this spec: the remaining work is not a sequence of
+patches to that loop. It is adopting an interface and an algorithm that already exist, are published,
+and are better than what we would arrive at by fixing ours one finding at a time.
+
+## What the current loop gets wrong, measured
+
+Round zero on vibe-py, `deepseek-v4-flash`, K = 5, $0.69, 23/30 pair-correct:
+
+| family | scorable | recall | silence | Youden | noise floor |
+|---|---|---|---|---|---|
+| injection | 20 | 0.75 | 0.90 | +0.65 | **11** |
+| access_control | 3 | 1.00 | 1.00 | +1.00 | 3 |
+| config_secrets | 3 | 1.00 | 1.00 | +1.00 | 1 |
+| deserialization | 2 | 1.00 | 1.00 | +1.00 | 0 |
+| output_encoding | 1 | 1.00 | 1.00 | +1.00 | 1 |
+| path | 1 | 1.00 | 1.00 | +1.00 | 1 |
+
+Four defects follow from the loop's shape, not from its bugs:
+
+1. **The acceptance rule cannot discriminate.** `decide` compares raw deltas: any regression on train or
+   holdout rejects, a tie rejects. The measured floor says 11 of 20 injection pairs disagree with
+   *themselves* across five runs, so a delta over a minibatch of 8 is mostly noise. The spec built
+   `sign_test` and `reliable_change` for exactly this and the rule calls neither.
+2. **The holdout is an input to the decision.** `decide` reads `holdout_delta`. Req 5.4 keeps the
+   *proposer* blind to the holdout, and then the *selector* reads it every round. Repeated rounds
+   selecting on holdout performance is search-time overfitting of the test set — the same leak class
+   this project has already measured once, one level up.
+3. **Noise is fought with repetition instead of diversity.** K = 5 spends five rollouts per pair per
+   side estimating variance. The published alternative spends that budget on more candidates and lets a
+   frontier keep whatever wins somewhere.
+4. **The proposer is told a score, not what went wrong.** `FailureFacts` carries outcomes, run strings
+   and counts. The single largest lever in the published work is textual feedback from the *evaluation
+   trace* — and our evaluator produces exactly the right material (which finding landed on which line,
+   whether it fell inside the labeled span, whether the fixed side leaked, which family was claimed)
+   and then throws all of it away.
+
+## What others built
+
+**GEPA — Reflective Prompt Evolution (Agrawal et al., arXiv:2507.19457, ICLR 2026 oral).** Algorithm 1,
+read from the paper rather than from a summary:
+
+- Split `D_train` into `D_feedback` and `D_pareto`. Initialise the candidate pool `P = [Φ]` and score
+  the seed on every instance of `D_pareto`, giving a score **matrix** `S[candidate][instance]`.
+- Each iteration: select a candidate from the Pareto frontier; select a module round-robin; run it on a
+  minibatch drawn from `D_feedback`; collect scores, traces and **textual feedback** via a feedback
+  function `μ_f`; have a reflection LM rewrite that module's prompt; re-run the mutated system on the
+  *same* minibatch.
+- **Accept if and only if the minibatch average improved.** Only then is the candidate scored on all of
+  `D_pareto` and added to the pool with its parent recorded.
+- Return the candidate with the best average on `D_pareto`.
+
+Algorithm 2, the selection: for each instance take the best score any candidate achieved; keep the
+candidates that achieve it; drop strictly dominated ones; sample a survivor with probability
+proportional to the number of instances it leads on.
+
+Three things about that are directly relevant to us. The acceptance gate is **cheap and noisy on
+purpose** — a single minibatch comparison — because the frontier, not the gate, is what protects
+against a bad accept: anything that wins on one instance is retained. Selection is **stochastic and
+diversity-preserving**, which is what stops the search collapsing onto one lineage. And the expensive
+full evaluation is spent only on candidates that already passed the cheap gate.
+
+Reported: 93% on MATH with a DSPy `ChainOfThought` program against 67% unoptimised, +10% over MIPROv2
+and +20% over GRPO with **35× fewer rollouts**; on IFBench, optimal prompts after 678 rollouts against
+GRPO's 24,000. Sample efficiency is the claim that matters for us, because our rollouts cost money.
+
+**DSPy's optimizer interface.** `MIPROv2` proposes instruction and demonstration candidates and searches
+them with Bayesian optimisation (Optuna TPE), scoring on minibatches and running a full validation pass
+on the incumbent every `minibatch_full_eval_steps` — an explicit two-fidelity design for a noisy metric.
+`dspy.GEPA` takes `metric`, `reflection_lm`, a budget (`auto` / `max_full_evals` / `max_metric_calls`),
+`reflection_minibatch_size`, `candidate_selection_strategy` (`pareto` | `current_best`) and
+`track_stats`, and its metric returns `Prediction(score, feedback)` rather than a float.
+
+**SIMBA — Stochastic Introspective Mini-Batch Ascent (`dspy.SIMBA`).** Two of its ideas fit our corpus
+better than GEPA's do. It runs `num_candidates` (6 by default) program variants per example, buckets the
+trajectories, and ranks examples by **max-to-min score gap** — deliberately spending the budget on the
+instances where variants disagree most. On our corpus that is not a nuisance, it is the training signal:
+the 11 injection pairs that disagree with themselves are precisely the pairs where a rule change can flip
+an outcome, and uniform minibatch sampling would spend most of its budget on the 9 that never move. It
+also alternates two strategies rather than round-robining blindly — `append_a_demo` (a successful
+trajectory becomes a demonstration) and `append_a_rule` (the model introspects on the bucket and writes a
+rule) — which is exactly the split our levers already have between `counterexamples`/`hard_negatives` and
+`checklist`/`prompt`. Candidates are sampled by softmax over historical scores at temperature 0.2, a
+cheaper third option beside GEPA's `pareto` and `current_best`.
+
+**The reflection meta-prompt** (GEPA Appendix C) is short and tells us what the feedback function must
+produce. It shows the reflector the current instruction, then "inputs, outputs and feedback" for the
+minibatch, and asks it to infer the task, extract "all niche and domain specific factual information ...
+as a lot of it may not be available to the assistant in the future", and name any generalizable strategy.
+That is a specification for `μ_f`, not just a prompt: feedback has to carry the domain facts a detector
+cannot re-derive — which sink, which span, which side leaked — rather than a verdict.
+
+**GEPA's merge** is gated strictly: two candidates are crossed only when they share a common ancestor,
+optimised *disjoint* sets of prompts, are both Pareto-optimal, and both beat the ancestor's aggregate.
+Worth having once we have per-lever candidates, and worth not having before then.
+
+**The standalone `gepa` library** (MIT) exposes `gepa.optimize(seed_candidate, trainset, valset,
+task_lm, reflection_lm, max_metric_calls)` over a `GEPAAdapter` protocol of three methods —
+`evaluate`, `make_reflective_dataset`, `propose_new_texts` — with a seed candidate that is just a dict
+of component name to text. That adapter shape is the interface we want whether or not we take the
+dependency: it is the smallest surface that separates *what a detector is* from *how it is searched*.
+
+## What this spec proposes
+
+Restate the detector as an optimisable program and put a published optimiser behind it.
+
+- **A detector is a candidate**: a dict of named text components (`prompt`, `checklist`,
+  `hard_negatives`, `counterexamples`) plus its frozen non-text settings (tools, budgets). That is
+  already what `FamilyConfig` holds; it becomes the seed candidate.
+- **The metric returns a score and a reason.** `score_pair_family` gains a feedback string built from
+  what the evaluation already knows. This is the highest-value change in the spec and it is independent
+  of which optimiser sits on top.
+- **`D_pareto` is carved out of the train split.** The holdout leaves the acceptance rule entirely and
+  is scored once, for reporting, when the search is over. This is a requirements change to Req 9.5 and
+  the most important honesty consequence of the redesign.
+- **An adapter protocol** (`evaluate`, `make_reflective_dataset`, `propose_new_texts`) with two backends:
+  our own implementation of Algorithms 1 and 2 by default, so the core install keeps
+  `dependencies = []`, and `gepa` behind an optional extra so its improvements arrive for free.
+- **The Pareto pool replaces the accept/revert tree.** `Archive` already records per-pair winners; it
+  becomes the frontier, and the journal's parent records become the ancestry GEPA needs.
+
+## Where the noise actually is, and why the architecture may be wrong
+
+The maintainer's objection, and I think it is correct: the harness treats the detector as a black box
+emitting a noisy scalar, and everything follows from that — K runs, majority vote, a measured floor, a
+regression budget, a sign test. That is the stance you take when you cannot see inside the thing you are
+measuring. We can see inside it.
+
+The variance is not in the judgement. `temperature: 0` is sent and thinking is disabled, and 11 of 20
+injection pairs still disagree with themselves. A traced run shows why: over four steps the model chose
+different files to read and different grep patterns, and reached its conclusion from different context
+each time. The freedom is in the **trajectory**, and we built statistics to average over it instead of
+removing it.
+
+This is exactly what DSPy exists to argue about, and three of its constructs attack the source rather
+than the symptom.
+
+**Constraints as predicates, enforced at generation time** (`dspy.Assert`/`Suggest`, arXiv:2312.13382).
+A constraint is a boolean function of the output; on failure the past output *and the error* are injected
+into a retry. We already have the predicates — is the reported line inside a parsed function, does the
+named sink occur at that line, is the claimed family in the taxonomy, did the run cite a tool result —
+and we apply every one of them **after the fact, at scoring time, to decide whether to count a finding**.
+Applying them at generation time to decide whether to *accept* one is free, needs no dependency, and is
+the single highest-value change available. The paper reports near-total intrinsic constraint satisfaction
+and 5–15% downstream gains.
+
+**Typed decomposition instead of one free-running loop.** Our detector is a single ReAct loop that decides
+which files to read, what to grep, when to stop and what to report; every one of those is a variance
+source. DSPy's thesis is that each step gets a signature — a typed input/output contract — and a little
+control flow composes them. Ours would be roughly: locate candidate sink sites; trace whether an untrusted
+value reaches each; check for a guard between source and sink; judge. **Several of those steps need no
+model at all** — the CST, the entry-point mapper, `flows` and `obligations` already answer them
+deterministically, and we are currently paying a model to re-derive facts we hold.
+
+**Best-of-N against a checkable reward, instead of a majority vote** (`dspy.Refine`, `BestOfN`). Both
+spend K samples. A majority vote assumes the modal answer is right and discards the disagreement; Refine
+ranks the N attempts by an explicit reward and feeds a hint from the failures into the next attempt. We
+have checkable rewards for free. The same budget, used as evidence rather than as averaging.
+
+**Why this matters more for our models, not less.** GEPA and SIMBA are demonstrated with a strong
+reflection model over a weaker task model, and their gains come from evolving instructions. A cheap model
+fails on an under-specified task and does well on a narrow typed one. So the cheaper the detector model,
+the more of the return sits in decomposition and constraints and the less in prompt evolution — which is
+the opposite of the order this spec originally proposed.
+
+### The experiment, run (2026-09-06, `benchmarks/measurements/2026-09-06-trajectory-divergence.json`)
+
+Twelve injection pairs, both sides, K = 5, temperature 0, thinking disabled, recording the tool-call
+sequence of every run. $0.23.
+
+**The intended statistic could not be computed. Across 120 runs there was not one repeated trajectory** —
+4.67 distinct trajectories per vulnerable side and 4.92 per fixed side, out of 5, and 78 distinct tool-*name*
+sequences ignoring arguments entirely. The detector is not sampling one measurement five times. It is running
+five different investigations and voting on the results.
+
+Asked as a gradient instead, the relationship is monotone and clean:
+
+| trajectory similarity (Jaccard over tool calls) | run pairs | disagreed on the outcome |
+|---|---|---|
+| high (≥ 0.75) | 2 | 0 (0.0%) |
+| mid (0.4–0.75) | 24 | 1 (4.2%) |
+| low (< 0.4) | 94 | 29 (30.9%) |
+
+Three more facts from the same run:
+
+- **The variance is a search problem, and the search is `grep_repo`.** 555 grep calls against 181 `read_file`,
+  71 `entry_points`, 43 `find_refs`, 13 `read_definition`. The model spends its budget guessing patterns to
+  locate a sink — while the CST, the entry-point mapper, `flows` and `obligations` already know where the
+  sinks are.
+- **All of the instability is on the vulnerable side.** Every one of the twelve pairs produced exactly one
+  distinct finding set on the fixed side. The detector is reliably silent on the fix and unreliably vocal on
+  the bug, so the noise is entirely in detection and none of it in false positives.
+- **Findings converge far more than trajectories do.** 78 distinct trajectories, but one to three distinct
+  finding sets per pair. Many different investigations reach the same answer; six of twelve pairs are stable
+  despite never once repeating a trajectory.
+
+**What this does to the noise architecture.** The floor is not measuring the detector's uncertainty about a
+question. It is measuring how often an unconstrained search happens to reach the sink. A majority vote over
+five runs is therefore not variance reduction — it is an unweighted ensemble of five arbitrary search
+procedures — and a regression budget derived from it is a budget for the search's luck. K runs, the floor,
+the budget and the sign test are all statistical treatment for a structural problem, and the 0% and 4.2%
+rows of that table say what happens when the structure is fixed instead.
+
+## Reflection on the approved specification
+
+Read against that measurement, `learning-harness` divides cleanly into a part that stands and a part that
+was answering the wrong question.
+
+**Stands, and is worth more now than before.** The closed taxonomy; class-aware pair scoring; the split
+enforced by construction; the unscorable vocabulary and its denominators; verifier-gated reporting; the
+corpus repairs; the publication machinery; the per-pair archive. None of it depends on the noise framing,
+and all of it is what made this experiment possible to run and cost $0.23. A measurement layer that can
+answer "is our own instability structural?" in twenty minutes is the thing that was missing before.
+
+**Was answering the wrong question.**
+
+- **Req 3.5, K ≥ 3 runs per pair.** The premise is variance reduction over a repeated measurement. There is
+  no repeated measurement: 120 runs, 120 trajectories. K runs is an *ensemble of distinct search
+  procedures*, and a majority vote over it is an unweighted ensemble, not an estimate. It should either be
+  renamed and published as an ensemble — which changes what every family number means — or dropped once the
+  search is constrained.
+- **Req 8.2, the noise floor as the regression budget.** The floor measures how often an unconstrained
+  search happens to reach the sink. Deriving a regression budget from it means the worse the search wanders,
+  the more collateral damage a round is licensed to do. That is backwards, and it is why injection's budget
+  is 11 of 20.
+- **Req 9.5, the acceptance rule.** Already flagged for reading the holdout every round. Now also: it
+  compares deltas between two ensemble votes, and calls the difference an improvement.
+- **The sign test and `reliable_change`.** Added this session in good faith; they test whether an ensemble's
+  vote moved, which is not the question either.
+
+**What the data adds that neither spec anticipated.**
+
+- **The instability is entirely one-sided.** All twelve pairs produced exactly one finding set on the fixed
+  side. Specificity is solid and recall is the noisy half, so the leak ceiling and the false-positive
+  machinery are sound and it is the recall numbers that carry the caveat.
+- **`grep_repo` is the variance engine** — 555 calls against 181 `read_file` and 71 `entry_points`. The
+  single highest-leverage change in either spec is not a prompt, a lever or an optimiser: **stop asking the
+  model to locate the sink.** The CST, the entry-point mapper, `flows` and `obligations` already know. Give
+  the detector the candidate sites and ask it the question only a model can answer — does untrusted input
+  reach this one, and is there a guard.
+- **The loop is partly self-correcting**, so this should be a large win rather than a marginal one: 78
+  distinct trajectories collapse to one to three finding sets per pair, and half the pairs are already
+  stable despite never repeating an investigation.
+
+**Limits of the experiment, stated so the conclusion is not over-read.** Twelve pairs, one family, one
+slice, one model. The high-similarity bucket holds only two run pairs, so its 0% is decoration; the
+defensible number is the mid bucket's 4.2% over 24 run pairs against 30.9% over 94. And the direction of
+causation is inferred, not established — similar trajectories may be a symptom of easy pairs rather than a
+cause of agreement. The way to settle that is the constrained detector itself: fix the investigation, and
+if the floor does not collapse, this reading was wrong.
+
+## The constraint that outranks the algorithm
+
+Both algorithms assume a dataset. GEPA splits `D_train` into `D_feedback` and `D_pareto`; SIMBA's default
+batch size is 32. **Our injection family has 12 train teachers** — fewer than one SIMBA batch, and after a
+GEPA split roughly six instances on each side. Every other family has one to three.
+
+So the honest statement of what this spec can and cannot buy: GEPA and SIMBA are sample-efficient in
+*rollouts*, not in *instances*. They will not shortcut a corpus of twelve. The optimiser is worth building
+because the current loop is the wrong shape and will stay wrong at any corpus size, but the binding
+constraint on results is the train split, and it will remain so after this spec ships. Two things move it,
+neither of which is an algorithm: reviewing agent-vfc rows above tier `title` (29 rows, currently unable to
+teach anything), and harvesting more pairs per family.
+
+## Scope
+
+In: the search over a family's text components, the feedback function, the candidate pool and its
+selection, the budget accounting, and the artifacts that make a run reproducible.
+
+Out, and deliberately: the taxonomy, the classifier, the pair scorer's *outcome* rules, the verifier
+boundary, the write roots, redaction, the three gates. Those are the measurement, and a search whose
+objective is not trustworthy is worse than no search. Nothing here relaxes them — in particular the
+proposer stays blind to the holdout, verifier code and gold labels stay outside every write root, and
+no pattern or policy is written by a model.
+
+## Decisions for the maintainer
+
+0. **Is the optimiser even the next thing to build?** Proposed no, and this reverses the order above.
+   Constraints at generation time, typed decomposition and best-of-N attack the variance at its source; an
+   optimiser only tunes text around a program whose freedom is the problem. Proposed sequence: measure where
+   the variance is, constrain generation, decompose the loop, then optimise — and at a trainset of twelve the
+   optimiser that fits is `BootstrapFewShot` (collect traces that scored well, install them as demonstrations
+   in a lever we already have) rather than an evolutionary search with nothing to search over.
+
+0b. **Adopt DSPy itself?** Proposed no. Our detector is not a DSPy program — it is an agentic tool loop over
+   a repository with curated tools, redaction on every prompt, path clamping, a "no findings unless a tool
+   was called" rule and a final-answer turn, several of which are security properties rather than
+   conveniences. Handing that loop to `dspy.ReAct` and the calls to litellm would put them behind someone
+   else's abstraction, and DSPy could only ever be an optional extra under `dependencies = []`, which means
+   two code paths for the thing that *is* the product. What DSPy would genuinely buy — signatures, adapters,
+   multi-module composition — we barely need at one module per family. The optimisers are available without
+   the framework: `gepa` ships standalone under MIT, and SIMBA's two ideas are a hundred lines of logic.
+   Proposed instead: take the interface shape and the algorithms, not the framework.
+
+1. **Reimplement Algorithms 1 and 2, take `gepa` as an extra, or both?** Both is proposed: ours by
+   default (zero-dependency, and the security properties are ours to own), `gepa` behind an extra for
+   comparison. The cost is one adapter and one conformance test that both backends must pass.
+2. **Drop the holdout from the acceptance rule?** Proposed yes, and it amends approved Req 9.5. The
+   holdout becomes what its name says.
+3. **Sample instances by disagreement (SIMBA) or uniformly (GEPA)?** Proposed SIMBA's ranking. Our measured
+   floor makes it the better fit, and it turns the noise floor from an obstacle into an instance sampler.
+
+4. **K = 1 inside the search, K ≥ 3 only for published numbers?** Proposed yes. That is roughly a 5×
+   budget reclaim spent on more candidates, which is where the sample efficiency comes from.
+5. **Does the reflection model differ from the detector model?** The published work uses a stronger
+   reflector than the task model. We have `deepseek-v4-pro` configured as the judge; proposed to reuse it.
+6. **Is `access_control` still the family we care most about?** It has one train teacher. Until agent-vfc
+   rows are reviewed above tier `title`, the optimiser cannot work on it, whatever its algorithm.
+
+## Open questions
+
+- Does `gepa` run against an OpenAI-shaped base URL that is not OpenAI (our DeepSeek endpoint), and what
+  does it pull in? Both must be checked before the extra is offered, under the no-undeclared-dependency rule.
+- Our "module" granularity: is a module a family, or a lever within a family? Round-robin over levers is
+  the closer analogue, but the levers are not independent — a checklist edit and a prompt edit interact.
+- The feedback function's failure mode is a reflection model that writes plausible prose about a trace it
+  misread. What check makes a feedback string answerable rather than persuasive?
+- GEPA's acceptance is a single noisy comparison. Our floor says a single comparison is close to a coin
+  flip on this corpus. Does the frontier really absorb that at n ≈ 20 instances, or is our corpus simply
+  too small for the algorithm's assumptions?
+
+Sources: [GEPA (arXiv:2507.19457)](https://arxiv.org/abs/2507.19457) ·
+[DSPy GEPA overview](https://dspy.ai/api/optimizers/GEPA/overview/) ·
+[DSPy MIPROv2](https://dspy.ai/api/optimizers/MIPROv2/) ·
+[gepa-ai/gepa](https://github.com/gepa-ai/gepa) ·
+[DSPy SIMBA](https://dspy.ai/api/optimizers/SIMBA/) ·
+[dspy/teleprompt/simba.py](https://github.com/stanfordnlp/dspy/blob/main/dspy/teleprompt/simba.py) ·
+[DSPy Assertions (arXiv:2312.13382)](https://arxiv.org/pdf/2312.13382) ·
+[dspy.Refine](https://dspy.ai/api/modules/Refine/) ·
+[DSPy (arXiv:2310.03714)](https://arxiv.org/pdf/2310.03714)

.kiro/specs/constrained-detector/requirements.md

@@ -0,0 +1,170 @@
+# Requirements Document
+
+## Introduction
+
+The detector's instability is structural, not statistical. Measured on 2026-09-06 over twelve injection pairs at
+temperature 0 with thinking disabled, **120 runs produced 120 distinct tool-call trajectories** — not one repeat —
+and outcome disagreement tracks trajectory divergence (30.9% at low similarity against 4.2% at mid). The model
+spends its budget searching: 555 `grep_repo` calls against 71 `entry_points`, guessing where sinks are while the
+semantic IR already knows. So `learning-harness`'s K runs are an ensemble of distinct search procedures rather than
+repeated measurements, and the noise floor it derives a regression budget from measures how often an unconstrained
+search happens to reach the sink.
+
+This feature removes the freedom instead of averaging over it. A deterministic pass enumerates the candidate sites
+inside the labeled region from the IR; the model is asked one bounded, typed question per candidate with no tools
+and no choice of what to look at; and the predicates the scorer already applies after the fact are enforced at
+generation time with a bounded retry. The candidate generator was measured before anything was built: the pattern
+ruleset caps recall at 12.1%, below the 64.9% the unconstrained loop already reaches, while the semantic IR caps it
+at 96.6% with a median of ten sites per labeled function.
+
+Baseline to beat, `deepseek-v4-flash`, K = 5, 57 scorable web pairs: **detection 64.9%, leak 12.3%, pair-correct
+52.6%** — and 83.3% detection on hand-written vibe-py against 44.4% on agent-written agent-vfc, which is the gap
+this feature must close rather than widen. Context and measurements: `brief.md`,
+`benchmarks/measurements/2026-09-06-trajectory-divergence.json`, `2026-09-06-candidate-ceiling.json`.
+
+## Boundary Context
+
+- **In scope**: candidate enumeration from the semantic IR, per family shape; the typed, toolless judgment step;
+  constraint predicates enforced at generation time with a bounded retry; the evidence that decides whether the
+  K-run and noise-floor machinery still earns its place; per-slice reporting that keeps real-world code in front.
+- **Out of scope**: the family taxonomy, the classifier, the pair scorer's outcome rules, the split, the verifier
+  boundary and write roots, redaction, the three gates, and the publication machinery — all of which stand and are
+  what made this feature's measurements possible. Also out: prompt optimisation, reflective evolution and any
+  optimiser (deferred to a later spec, on this feature's evidence); finding bugs in JavaScript interpreters, WebGL
+  renderers or browser engines; memory-safety fuzzing for C and C++; fine-tuning model weights; adopting DSPy as a
+  dependency.
+- **Adjacent expectations**: `learning-harness` keeps owning the taxonomy, the scorer, the split and publication;
+  its Req 3.5 (K runs), 8.2 (noise floor) and 9.5 (acceptance) are amended by this feature only on evidence it
+  produces, never on argument. The pair corpus keeps owning labels, tiers and splits. The unconstrained tool hunter
+  stays as the generalist and as the baseline this feature is measured against.
+
+## Requirements
+
+### Requirement 1: Candidates come from the semantic IR, and the enumerator's ceiling is published
+
+**Objective:** As a maintainer, I want the sites a detector judges to be enumerated deterministically, so that the
+model stops searching and the limit of what it can possibly find is a number rather than a hope.
+
+#### Acceptance Criteria
+
+1. The Constrained Detector shall enumerate candidate sites for a region from the semantic IR — call sites and
+   bindings — and shall not use the pattern ruleset as its candidate generator.
+2. The Constrained Detector shall shape candidates by family: a call site for injection, path, deserialization,
+   untrusted destination and prototype; an operation on a protected resource with the guards in scope for access
+   control; a configuration assignment or literal for permissive configuration and secrets.
+3. The Constrained Detector shall publish, per family and per slice, the fraction of labeled functions for which at
+   least one candidate was enumerated, and shall name that fraction as the ceiling on its recall.
+4. If the semantic IR cannot parse a side, then the Constrained Detector shall record `unsupported_language` and
+   shall score nothing for that pair rather than falling back to a search.
+5. If a labeled function yields no candidate site, then the Constrained Detector shall record `no_candidate` and
+   report that pair as unscorable with that reason rather than as a miss.
+6. While enumerating, the Constrained Detector shall bound the candidates handed to one judgment call and shall
+   record the bound and the number discarded when a region exceeds it.
+
+### Requirement 2: One bounded, typed question per candidate, with no search
+
+**Objective:** As a maintainer, I want the model to answer only what a model is needed for, so that two runs of the
+same detector on the same input take the same path.
+
+#### Acceptance Criteria
+
+1. The Constrained Detector shall call the model with the candidate site, its enclosing function, the binding chain
+   the IR resolved for its arguments, and the guards in scope, and shall offer it no tools.
+2. The Constrained Detector shall require a typed answer per candidate: a verdict, the expression it believes
+   carries untrusted input, the guard it found or the statement that there is none, and the family.
+3. The Constrained Detector shall derive every finding from a typed answer and shall never parse a finding out of
+   free prose.
+4. When the detector is run twice on the same input with the same configuration, the sequence of model calls shall
+   be identical, and a test shall assert this.
+5. The Constrained Detector shall record, for every claim, the candidate that produced it, so that a claim is
+   traceable to the site it was asked about.
+
+### Requirement 3: Real-world code decides, and the gap to teaching code is published
+
+**Objective:** As a maintainer, I want changes judged on code from real projects rather than on hand-written
+teaching examples, so that the tool is not tuned for cases nobody has.
+
+#### Acceptance Criteria
+
+1. The Constrained Detector shall report every number per slice and shall never publish an aggregate across slices
+   without the per-slice numbers beside it.
+2. The Constrained Detector shall designate the slices harvested from real projects — agent-vfc, vfc-js, vfc and
+   github — as the deciding slices, and the hand-written vibe-py slice as a development slice that cannot on its
+   own justify accepting a change.
+3. If a change improves the development slice and regresses any deciding slice, then the Constrained Detector shall
+   reject it, and the rejection shall name the slice that regressed.
+4. The Constrained Detector shall publish the **overfitting gap** — development-slice detection minus deciding-slice
+   detection — beside every headline number, and shall state that the baseline gap is 83.3% against 44.4%.
+5. The Constrained Detector shall measure and publish the candidate ceiling separately for the real-world slices,
+   because the functions there are larger than the teaching corpus's and the enumerator's behaviour on them is not
+   implied by its behaviour on vibe-py.
+6. While the deciding slices carry no pair a learning path may teach from, the Constrained Detector shall state that
+   with every number it publishes rather than presenting the development slice as if it were the corpus.
+7. The Constrained Detector shall keep memory safety measured and non-gating, and shall not let the 176 vfc rows
+   dominate an aggregate that a reader would take as the tool's web-taxonomy performance.
+
+### Requirement 4: The predicates that score a finding also constrain it
+
+**Objective:** As a maintainer, I want the checks that decide whether a finding counts applied while it is being
+made, so that a violation is corrected rather than silently discarded.
+
+#### Acceptance Criteria
+
+1. The Constrained Detector shall express as predicates the checks the scorer already applies: the reported line
+   falls inside a parsed function, the named sink occurs at that line, the claimed family is in the taxonomy, and
+   the claimed source expression appears in the enclosing function.
+2. When a predicate fails, the Constrained Detector shall retry the judgment once with the failed answer and the
+   predicate that rejected it, and shall not retry more than once per candidate.
+3. If a retry also fails a predicate, then the Constrained Detector shall discard the claim and record the predicate
+   that rejected it, and shall count that discard in a published tally.
+4. The Constrained Detector shall apply the same predicates at generation time and at scoring time from one
+   definition, so the two cannot drift apart.
+
+### Requirement 5: The noise machinery is re-measured, and kept only if it earns its place
+
+**Objective:** As a maintainer, I want the K runs, the noise floor and the regression budget re-measured against a
+constrained detector, so that machinery built for an unconstrained one is deleted rather than maintained.
+
+#### Acceptance Criteria
+
+1. The Constrained Detector shall re-run the trajectory measurement against itself and shall publish the resulting
+   distinct-trajectory count per pair.
+2. The Constrained Detector shall re-measure the per-family noise floor at the same K as the baseline and shall
+   publish it beside the baseline's floor.
+3. If the re-measured floor is zero for a family, then the Constrained Detector shall report that family's K-run
+   requirement and its regression budget as unnecessary and shall name the `learning-harness` requirements that
+   would be amended.
+4. The Constrained Detector shall state, wherever a K-run number is published, whether the runs were repeated
+   measurements or an ensemble of distinct procedures.
+
+### Requirement 6: The constrained detector is measured against the unconstrained one, on the same corpus
+
+**Objective:** As a maintainer, I want one comparison that says whether this was worth building.
+
+#### Acceptance Criteria
+
+1. The Constrained Detector shall be scored by the existing class-aware pair scorer, with the same denominators,
+   the same unscorable vocabulary and the same split, and shall introduce no scoring rule of its own.
+2. The Constrained Detector shall publish detection, leak and pair-correct rates against the recorded baseline of
+   64.9%, 12.3% and 52.6%, per slice and in total.
+3. The Constrained Detector shall publish its cost per correct pair against the baseline's $0.030 on vibe-py and
+   $0.136 on agent-vfc.
+4. If the constrained detector's detection rate is below the unconstrained loop's on any deciding slice, then that
+   shall be published as the headline of the comparison rather than as a footnote.
+5. The Constrained Detector shall keep the unconstrained tool hunter available and shall not remove it while it
+   scores higher on any deciding slice.
+
+### Requirement 7: Degradation, budget and boundaries
+
+**Objective:** As a maintainer, I want the same operational contract the rest of the tool has.
+
+#### Acceptance Criteria
+
+1. The Constrained Detector shall run behind the existing chat endpoint resolution and shall record
+   `learning_endpoint_unavailable` and score nothing rather than fall back to another provider.
+2. The Constrained Detector shall meter its own spend through the client's meter and shall stop when a configured
+   cap is crossed, recording what it had spent.
+3. The Constrained Detector shall leave the detection, map and local pair gate outputs byte-identical.
+4. The Constrained Detector shall keep the core install at zero dependencies, and every optional capability it uses
+   shall degrade to a recorded reason without its extra.
+5. The Constrained Detector shall pass every prompt and every persisted trajectory through redaction.

.kiro/specs/constrained-detector/spec.json

@@ -0,0 +1,26 @@
+{
+  "feature_name": "constrained-detector",
+  "created_at": "2026-09-06T14:30:00Z",
+  "updated_at": "2026-09-06T16:00:00Z",
+  "language": "en",
+  "phase": "requirements",
+  "approvals": {
+    "requirements": {
+      "generated": true,
+      "approved": false
+    },
+    "design": {
+      "generated": false,
+      "approved": false
+    },
+    "tasks": {
+      "generated": false,
+      "approved": false
+    }
+  },
+  "ready_for_implementation": false,
+  "depends_on": [
+    "learning-harness"
+  ],
+  "approval_note": "2026-09-06 brief drafted at the maintainer's direction ('I'd rather spend time to design the exact DSPy style interface and evolutionary / statistical prompt optimization than fixing piece by piece'). Not approved; requirements not generated.; 2026-09-06 renamed from detector-optimizer to constrained-detector after the trajectory measurement: the brief's own conclusion is that the optimiser is not the next thing to build, and the spec should be named for what it delivers; 2026-09-06 requirements generated at the maintainer's direction, with the maintainer's emphasis recorded as Requirement 3 ('don't forget the emphasis on the real world bugs from the larger projects to avoid overfitting. we otherwise benchmark for cases that are not relevant')"
+}