Open_Ultra_SAST/.kiro/specs/constrained-detector/requirements.md

# Requirements Document

> **Status 2026-10-02:** superseded by `model-grounded-detection` (spec.json `superseded_by`). 4 of 20 tasks were done and the tasks were never approved; none will be. Kept as decision history; nothing in src, tests or CI points here.

## Introduction

The detector's instability is structural, not statistical. Measured on 2026-09-06 over twelve injection pairs at
temperature 0 with thinking disabled, **120 runs produced 120 distinct tool-call trajectories**, not one repeat,
and outcome disagreement tracks trajectory divergence (30.9% at low similarity against 4.2% at mid). The model
spends its budget searching: 555 `grep_repo` calls against 71 `entry_points`, guessing where sinks are while the
semantic IR already knows. So `learning-harness`'s K runs are an ensemble of distinct search procedures rather than
repeated measurements, and the noise floor it derives a regression budget from measures how often an unconstrained
search happens to reach the sink.

This feature removes the freedom instead of averaging over it. A deterministic pass enumerates the candidate sites
inside the labeled region from the IR; the model is asked one bounded, typed question per candidate with no tools
and no choice of what to look at; and the predicates the scorer already applies after the fact are enforced at
generation time with a bounded retry. The candidate generator was measured before anything was built: the pattern
ruleset caps recall at 12.1%, below the 64.9% the unconstrained loop already reaches, while the semantic IR caps it
at 96.6% with a median of ten sites per labeled function.

Baseline to beat, `deepseek-v4-flash`, K = 5, 57 scorable web pairs: **detection 64.9%, leak 12.3%, pair-correct
52.6%**, and 83.3% detection on hand-written vibe-py against 44.4% on agent-written agent-vfc, which is the gap
this feature must close rather than widen. Context and measurements: `brief.md`,
`benchmarks/measurements/2026-09-06-trajectory-divergence.json`, `2026-09-06-candidate-ceiling.json`.

## Boundary Context

- In scope: candidate enumeration from the semantic IR, per family shape; the typed, toolless judgment step;
  constraint predicates enforced at generation time with a bounded retry; the evidence that decides whether the
  K-run and noise-floor machinery still earns its place; per-slice reporting that keeps real-world code in front.
- Out of scope: the family taxonomy, the classifier, the pair scorer's outcome rules, the split, the verifier
  boundary and write roots, redaction, the three gates, and the publication machinery, all of which stand and are
  what made this feature's measurements possible. Also out: prompt optimisation, reflective evolution and any
  optimiser (deferred to a later spec, on this feature's evidence); finding bugs in JavaScript interpreters, WebGL
  renderers or browser engines; memory-safety fuzzing for C and C++; fine-tuning model weights; adopting DSPy as a
  dependency.
- Adjacent expectations: `learning-harness` keeps owning the taxonomy, the scorer, the split and publication;
  its Req 3.5 (K runs), 8.2 (noise floor) and 9.5 (acceptance) are amended by this feature only on evidence it
  produces, never on argument. The pair corpus keeps owning labels, tiers and splits. The unconstrained tool hunter
  stays as the generalist and as the baseline this feature is measured against.

## Requirements

### Requirement 1: Candidates come from the semantic IR, and the enumerator's ceiling is published

**Objective:** As a maintainer, I want the sites a detector judges to be enumerated deterministically, so that the
model stops searching and the limit of what it can possibly find is a number rather than a hope.

#### Acceptance Criteria

1. The Constrained Detector shall enumerate candidate sites for a region from the semantic IR (call sites and
   bindings), and shall not use the pattern ruleset as its candidate generator.
2. The Constrained Detector shall shape candidates by family: a call site for injection, path, deserialization,
   untrusted destination and prototype; an operation on a protected resource with the guards in scope for access
   control; a configuration assignment or literal for permissive configuration and secrets.
3. The Constrained Detector shall publish, per family and per slice, the fraction of labeled functions for which at
   least one candidate was enumerated, and shall name that fraction as the ceiling on its recall.
4. If the semantic IR cannot parse a side, then the Constrained Detector shall record `unsupported_language` and
   shall score nothing for that pair rather than falling back to a search.
5. If a labeled function yields no candidate site, then the Constrained Detector shall record `no_candidate` and
   report that pair as unscorable with that reason rather than as a miss.
6. While enumerating, the Constrained Detector shall bound the candidates handed to one judgment call and shall
   record the bound and the number discarded when a region exceeds it.

### Requirement 2: One bounded, typed question per candidate, with no search

**Objective:** As a maintainer, I want the model to answer only what a model is needed for, so that two runs of the
same detector on the same input take the same path.

#### Acceptance Criteria

1. The Constrained Detector shall call the model with the candidate site, its enclosing function, the binding chain
   the IR resolved for its arguments, and the guards in scope, and shall offer it no tools.
2. The Constrained Detector shall require a typed answer per candidate: a verdict, the expression it believes
   carries untrusted input, the guard it found or the statement that there is none, and the family.
3. The Constrained Detector shall derive every finding from a typed answer and shall never parse a finding out of
   free prose.
4. When the detector is run twice on the same input with the same configuration, the sequence of model calls shall
   be identical, and a test shall assert this.
5. The Constrained Detector shall record, for every claim, the candidate that produced it, so that a claim is
   traceable to the site it was asked about.

### Requirement 3: Real-world code decides, and the gap to teaching code is published

**Objective:** As a maintainer, I want changes judged on code from real projects rather than on hand-written
teaching examples, so that the tool is not tuned for cases nobody has.

#### Acceptance Criteria

1. The Constrained Detector shall report every number per slice and shall never publish an aggregate across slices
   without the per-slice numbers beside it.
2. The Constrained Detector shall designate the slices harvested from real projects (agent-vfc, vfc-js, vfc and
   github) as the deciding slices, and the hand-written vibe-py slice as a development slice that cannot on its
   own justify accepting a change.
3. If a change improves the development slice and regresses any deciding slice, then the Constrained Detector shall
   reject it, and the rejection shall name the slice that regressed.
4. The Constrained Detector shall publish the **overfitting gap** (development-slice detection minus deciding-slice
   detection) beside every headline number, and shall state that the baseline gap is 83.3% against 44.4%.
5. The Constrained Detector shall measure and publish the candidate ceiling separately for the real-world slices,
   because the functions there are larger than the teaching corpus's and the enumerator's behaviour on them is not
   implied by its behaviour on vibe-py.
6. While the deciding slices carry no pair a learning path may teach from, the Constrained Detector shall state that
   with every number it publishes rather than presenting the development slice as if it were the corpus.
7. The Constrained Detector shall keep memory safety measured and non-gating, and shall not let the 176 vfc rows
   dominate an aggregate that a reader would take as the tool's web-taxonomy performance.

### Requirement 4: The predicates that score a finding also constrain it

**Objective:** As a maintainer, I want the checks that decide whether a finding counts applied while it is being
made, so that a violation is corrected rather than silently discarded.

#### Acceptance Criteria

1. The Constrained Detector shall express as predicates the checks the scorer already applies: the reported line
   falls inside a parsed function, the named sink occurs at that line, the claimed family is in the taxonomy, and
   the claimed source expression appears in the enclosing function.
2. When a predicate fails, the Constrained Detector shall retry the judgment once with the failed answer and the
   predicate that rejected it, and shall not retry more than once per candidate.
3. If a retry also fails a predicate, then the Constrained Detector shall discard the claim and record the predicate
   that rejected it, and shall count that discard in a published tally.
4. The Constrained Detector shall apply the same predicates at generation time and at scoring time from one
   definition, so the two cannot drift apart.

### Requirement 5: The noise machinery is re-measured, and kept only if it earns its place

**Objective:** As a maintainer, I want the K runs, the noise floor and the regression budget re-measured against a
constrained detector, so that machinery built for an unconstrained one is deleted rather than maintained.

#### Acceptance Criteria

1. The Constrained Detector shall re-run the trajectory measurement against itself and shall publish the resulting
   distinct-trajectory count per pair.
2. The Constrained Detector shall re-measure the per-family noise floor at the same K as the baseline and shall
   publish it beside the baseline's floor.
3. If the re-measured floor is zero for a family, then the Constrained Detector shall report that family's K-run
   requirement and its regression budget as unnecessary and shall name the `learning-harness` requirements that
   would be amended.
4. The Constrained Detector shall state, wherever a K-run number is published, whether the runs were repeated
   measurements or an ensemble of distinct procedures.

### Requirement 6: The constrained detector is measured against the unconstrained one, on the same corpus

**Objective:** As a maintainer, I want one comparison that says whether this was worth building.

#### Acceptance Criteria

1. The Constrained Detector shall be scored by the existing class-aware pair scorer, with the same denominators,
   the same unscorable vocabulary and the same split, and shall introduce no scoring rule of its own.
2. The Constrained Detector shall publish detection, leak and pair-correct rates against the recorded baseline of
   64.9%, 12.3% and 52.6%, per slice and in total.
3. The Constrained Detector shall publish its cost per correct pair against the baseline's $0.030 on vibe-py and
   $0.136 on agent-vfc.
4. If the constrained detector's detection rate is below the unconstrained loop's on any deciding slice, then that
   shall be published as the headline of the comparison rather than as a footnote.
5. The Constrained Detector shall keep the unconstrained tool hunter available and shall not remove it while it
   scores higher on any deciding slice.

### Requirement 7: Degradation, budget and boundaries

**Objective:** As a maintainer, I want the same operational contract the rest of the tool has.

#### Acceptance Criteria

1. The Constrained Detector shall run behind the existing chat endpoint resolution and shall record
   `learning_endpoint_unavailable` and score nothing rather than fall back to another provider.
2. The Constrained Detector shall meter its own spend through the client's meter and shall stop when a configured
   cap is crossed, recording what it had spent.
3. The Constrained Detector shall leave the detection, map and local pair gate outputs byte-identical.
4. The Constrained Detector shall keep the core install at zero dependencies, and every optional capability it uses
   shall degrade to a recorded reason without its extra.
5. The Constrained Detector shall pass every prompt and every persisted trajectory through redaction.