Open_Ultra_SAST/.kiro/specs/model-grounded-detection/requirements.md

# Requirements Document: model-grounded-detection

> **Status 2026-10-02:** complete, 16 of 16 tasks (spec.json phase corrected to `implementation-complete`). On 2026-10-02 three of its modules were deleted because nothing imported them: `model/judge.py` (Req 8; `model/pipeline.py` does the judging), `model/calibrate.py` (Req 9; the decision engine's `learn/calibrate.py` is the live calibration) and `model/execution.py`. Deleting `execution.py` closes the Req 10 execution-tier option: a future execution tier is a new design, not this seam.

## Introduction

This feature supersedes `learning-harness` and `constrained-detector`. It replaces the assumption that the LLM
is a noisy oracle to be averaged, floored and gated with an architecture in which a **deterministic program
model arbitrates the LLM's claims**. The arbiter is a Code Property Graph (CPG), an intermediate language
merging AST, control-flow graph, program-dependence graph and call graph, over which taint reachability and
guard dominance produce verdicts the LLM's assertion is no longer needed for. The measurements that force this
design are committed: the candidate ceiling (96.6% of labeled functions have an enumerable site), and the
entailment ceiling (a deterministic model *decides* only 2.2% today on our flat AST-projection IR, ~24-35%
with deeper taint, the distance to a real CPG). Context: `brief.md`,
`benchmarks/measurements/2026-09-06-trajectory-divergence.json`, `2026-09-06-candidate-ceiling-all-slices.json`,
`2026-09-08-model-entailment-ceiling.json`.

The project has an outgrowth problem: 99 source modules and 832 tests, a large fraction implementing the noise
architecture this feature discards. **Subtraction comes first.** The dead layer is removed and the test suite
reduced to what protects surviving behavior before the CPG arbiter is adopted, because building on top of an
overgrown, superseded surface is how this session logged eighteen defects.

## Boundary Context

- In scope: removing the noise architecture and the superseded arbiter; reducing the test suite to
  surviving behavior; auditing the whole tree for orphaned modules; adopting a CPG engine (Joern) as the
  model-layer substrate behind an optional extra; the evidence ladder as the verdict type; taint + dominance
  verdicts; porting our taxonomy and source/sink/sanitizer/guard models to the CPG; the LLM as a proposer
  checked against the model; the corpus as the model's calibration set; per-slice reporting.
- Out of scope: building a dataflow/CFG engine by hand (the CPG is adopted, §Req 4); the execution tier
  itself (deferred to Clearwing, `execution_confirmed` rung only, Req 10); prompt optimisation, reflective
  evolution, any optimiser (deleted, not deferred); fine-tuning weights.
- North star, not v1: pure `model_entailed`-as-arbiter. The entailment ceiling says the model decides a
  minority today; v1 is the *layered* arbiter (entail where it can, corroborate the taint band, honest
  suspicion for the rest). This is stated so the requirements are not read as promising a soundness the
  measurement denies.

## Requirements

### Requirement 1: Subtract the noise architecture

**Objective:** As a maintainer, I want the machinery built to average and gate an LLM oracle removed entirely,
so that nothing is maintained, tested, or built upon that this feature's measurements say is the wrong shape.

#### Acceptance Criteria

1. The system shall remove the round loop, the per-family noise floor, the regression budget, the acceptance
   rule, the sign test, the evolve journal and the meta-agent proposer, concretely `learning/rounds.py`,
   `learning/acceptance.py`, `learning/proposer.py`, `learning/journal.py`, and the K-run / floor / sign-test
   machinery in `learning/scoring.py`.
2. The system shall remove the mechanism-optimisation loop (the mechanism store and lever in
   `improve/evolve.py`, and the shape-search machinery in `semantic/mechanisms.py` and the optimisation
   surface of `semantic/variants.py`), while preserving the *security models* those modules carried (the
   guard vocabulary and source/sink classification) for porting under Req 7.
2b. The system shall remove the per-family evolvable detector configuration and its canary/verifier surface
   (`learning/detectors.py` config-evolution parts, `learning/canaries.py`, `learning/verifiers.py`,
   `learning/difficulty.py`) except where a component is retained by name under Req 6, 7 or 8.
3. When a module is removed, no surviving module shall import it, and a static check shall confirm no dangling
   import remains.
4. The detection, map and local pair gate outputs shall stay byte-identical to the committed baseline after
   the removal, since the removed modules are absent from the gate path.
5. The system shall record, for every removed module, one line stating what replaced it or why nothing did.

### Requirement 2: Reduce the test suite to what protects surviving behavior

**Objective:** As a maintainer, I want the test suite to shrink with the code, so that tests stop being a
ratchet that pins deleted mechanisms and the suite is once again a description of what matters.

#### Acceptance Criteria

1. When a mechanism is removed under Req 1, the tests that existed only to pin its behavior shall be removed
   with it, in the same change, not left skipped or xfailed.
2. Every surviving test shall map to a surviving requirement (of this spec or of a still-shipping feature)
   and a test that maps to nothing shall be removed.
3. The reduced suite shall be smaller than the current 832 tests, and the change shall report the count before
   and after with the requirement each removed test cluster served.
4. The system shall keep as the non-negotiable floor: the three gate-baseline tests, the redaction tests, the
   determinism test (Req 6.4), the per-slice honesty tests, and the two ceiling measurements' regeneration
   tests.
5. A test that asserts a shape "does nothing" or greps `--help`, or that passes with its capability absent,
   shall be removed rather than kept. The session logged several; none survive.

### Requirement 3: Audit the whole tree for outgrowth

**Objective:** As a maintainer, I want every module classified, so that orphaned code from earlier phases is
found and removed rather than carried indefinitely.

#### Acceptance Criteria

1. The system shall classify every source module as one of: load-bearing for the model-grounded architecture;
   a retained standalone capability with its own users; or orphaned.
2. If a module is orphaned (no importer, no CLI entry, no gate dependency, no surviving requirement), then the
   system shall remove it and record the classification.
3. The system shall name, for the audit, the older-phase subsystems whose status is not obvious
   (`fusion.py`, `mcp.py`, `skills.py`, `harness_ext.py`, `hunter_harness.py`) and give each an explicit
   keep-or-remove decision with its reason, rather than leaving them unexamined.
4. The audit result shall be committed as a manifest that a later reader can check the tree against.

### Requirement 4: Adopt a CPG engine as the arbiter substrate

**Objective:** As a maintainer, I want the model layer built on an adopted Code Property Graph engine, not on
a hand-built dataflow engine, so that the CFG and PDG our flat IR lacks come from a mature substrate.

#### Acceptance Criteria

1. The system shall compile a target to a Code Property Graph via an adopted engine (Joern, Apache-2.0),
   producing AST, control-flow, program-dependence and call-graph relations over one queryable graph.
2. The CPG engine shall sit behind an optional extra; the core install shall remain zero-dependency
   (`dependencies = []`).
3. Without the CPG engine, the system shall degrade every model verdict to `suspicion` and record the reason,
   never crash and never fall back to the flat IR as if it were an arbiter.
4. The system shall not build a control-flow or program-dependence engine of its own; the flat tree-sitter IR
   is retained only for candidate enumeration in the `suspicion` band (Req 8.4).
5. The system shall record the named integration cost (the engine is JVM/Scala; access is via its server mode
   or an MCP bridge, not an in-process call) and isolate it behind one seam.

### Requirement 5: The evidence ladder is the verdict type

**Objective:** As a maintainer, I want every finding to carry what established it, so that a deterministic
model result is never confused with an LLM suspicion.

#### Acceptance Criteria

1. The system shall assign every finding one rung: `suspicion` (a site was enumerated / the LLM flagged it),
   `model_corroborated` (the LLM's claim is consistent with the CPG, the source it names reaches the sink
   through resolved dataflow, or a guard its siblings carry is absent), `model_entailed` (the CPG alone
   establishes it, independent of trusting the LLM), or `execution_confirmed` (a sandbox reproduced it).
2. The system shall establish `model_corroborated` and `model_entailed` from the CPG deterministically, with
   no model call in the establishing step.
3. The system shall report the rung on every finding in every output format.
4. The system shall never report a claim above `suspicion` on the strength of the LLM alone.

### Requirement 6: Deterministic verdicts from taint reachability and guard dominance

**Objective:** As a maintainer, I want the model verdicts to be reproducible, so that the run-to-run noise the
project measured is removed on the band the model covers.

#### Acceptance Criteria

1. The system shall establish a flow-family verdict by taint reachability over the CPG: a source reaching the
   labeled sink through data-dependence, minus any sanitizer node on the path.
2. The system shall establish an access-control (absence) verdict by dominance over the CFG: whether a
   discharging guard dominates the operation on every path.
3. The system shall establish a configuration verdict by constant/abstract-value evaluation of the binding.
4. When the system runs the model layer twice over the same CPG, the sequence of verdicts shall be identical,
   and a test shall assert it.
5. The system shall report, per family and per slice, the fraction of the corpus it can `model_entail` and
   `model_corroborate` (the regeneration of the committed entailment ceiling), and shall name where it falls
   back to `suspicion`.

### Requirement 7: Port the taxonomy and security models onto the CPG

**Objective:** As a maintainer, I want our security knowledge (the classes we target and how their bugs look)
expressed as queries over the adopted CPG, so that the contribution is the modelling, not the engine.

#### Acceptance Criteria

1. The system shall retain the closed family taxonomy as the routing key, unchanged in meaning.
2. The system shall express the source, sink and sanitizer models for the web/logic CWEs as CPG taint
   specifications, seeded from the models the deleted modules carried.
3. The system shall express the obligation / absence model (an operation on a protected resource, and the
   guards that discharge it) as CPG dominance queries, as the arbiter for the classes that never crash.
4. The system shall keep the verifier boundary: the models and gold labels stay outside anything an LLM or a
   future optimiser may write.

### Requirement 8: The LLM proposes; the model disposes

**Objective:** As a maintainer, I want the LLM's role reduced to the residual judgement the model cannot make,
with its claim checked against the model, so that there is no oracle to average.

#### Acceptance Criteria

1. The system shall ask the LLM one bounded, typed question per candidate, offering it no freedom to search
   for the site.
2. When the model contradicts the LLM's claim, the system shall drop the claim and record the contradiction.
3. When the model corroborates the LLM's claim, the system shall advance the finding to at least
   `model_corroborated`.
4. When the model can neither confirm nor contradict, the system shall report the claim at `suspicion`, and
   this band is where a future execution tier earns its place.
5. The system shall contain no K-run averaging, no noise floor, no acceptance rule, no evolve loop. The model
   check replaces all of them.

### Requirement 9: The corpus is the model's calibration set

**Objective:** As a maintainer, I want the corpus to measure the model's precision rather than train an LLM
vote, so that a gap is a gap in the model, named and fixable.

#### Acceptance Criteria

1. The system shall use the vulnerable-parent / fix-commit pair as ground truth for whether the model can
   distinguish the two. The fix is what the model should be able to see.
2. If the model cannot distinguish a pair's two sides, then the system shall record it as a named model gap
   with the reason, not as a detector miss to be averaged away.
3. The system shall keep the per-slice split with the real-world slices deciding, and shall not present the
   development slice as the corpus.
4. The system shall support harvesting pairs at repository scale from CVE history, with the fix diff read by
   the model as the oracle, rather than hand-assigned labels.

### Requirement 10: The execution tier is deferred and adopted, not built

**Objective:** As a maintainer, I want dynamic confirmation available for the classes a static model cannot
arbitrate, without building or requiring it at v1.

#### Acceptance Criteria

1. The system shall treat `execution_confirmed` as an optional rung reached only for classes where the model
   layer cannot decide, memory-safety in C first.
2. When execution confirmation is used, the system shall adopt it from Clearwing (the existing fork), not
   reimplement the sandbox lifecycle, container pooling, sanitizer images or PoC stability.
3. The system shall not require the execution tier for any web/logic class at v1.

### Requirement 11: Reporting and operational contract

**Objective:** As a maintainer, I want the honesty disciplines that survived and the same operational contract
as the rest of the tool.

#### Acceptance Criteria

1. The system shall report every number per slice, publish the overfitting gap beside every headline number,
   and carry the evidence rung on every finding.
2. The system shall leave the detection, map and local pair gate outputs byte-identical to the committed
   baseline.
3. The system shall keep the core install at zero dependencies; the CPG engine, the LLM endpoint and any
   execution tier are optional extras that degrade to a recorded reason without them.
4. The system shall pass every prompt and every persisted artifact through redaction.