From d08dea19b82d65371c1006f60bc103e79eda24c4 Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Tue, 13 Oct 2026 13:01:32 +0200 Subject: [PATCH] spec(hunter-learning-loop): a pattern engine's miss is never evidence A Semgrep, quick-rule or engine match may enter the feature table as positive evidence; its absence is never a feature, filter, demotion or gate on a hunter suspicion, and no acceptance rule counts it. The Semgrep arm is a comparison only and may at most replace the regex quick tier. Co-Authored-By: Claude Opus 5.5 --- .kiro/breadth.toml | 2 +- .kiro/specs/hunter-learning-loop/design.md | 7 +++++++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/.kiro/breadth.toml b/.kiro/breadth.toml index 1a88f45..7ff32a2 100644 --- a/.kiro/breadth.toml +++ b/.kiro/breadth.toml @@ -257,5 +257,5 @@ closes = "U3: move under benchmarks/push or delete if only tests use them" status = "review" modules = ["findings.py", "ruleset", "semantic (taint, overlay, prove_filter)"] note = "maintainer 2026-10-13: 'semgrep oss is a good idea'; brief in benchmarks/measurements/2026-10-13-semgrep-brief/ says registry rules are not shippable, our converted rules are; 4.7 s startup floor" -closes = "run the Semgrep arm on the valid pilot (benchmarks/valid/semgrep_arm.py); if it scores at least as well, retire findings.py, ruleset/*.py and the taint overlay (~1,250 lines) and amend safety-net.md's second-engine rule with the measurement" +closes = "run the Semgrep arm on the valid pilot (benchmarks/valid/semgrep_arm.py) as a comparison; it may replace the regex quick tier only (same job: pattern matches), never act as a filter on other tiers. A Semgrep miss is never evidence (maintainer 2026-10-13). If it matches the quick tier at least as well, retire findings.py and ruleset/*.py and amend safety-net.md with the measurement" diff --git a/.kiro/specs/hunter-learning-loop/design.md b/.kiro/specs/hunter-learning-loop/design.md index d64d68a..a26e06f 100644 --- a/.kiro/specs/hunter-learning-loop/design.md +++ b/.kiro/specs/hunter-learning-loop/design.md @@ -106,6 +106,13 @@ tests/test_hunt_*.py table, producers (planted-advisory test), progr ablation masks, acceptance rule, artifact ``` +### Rule: a pattern engine's miss is never evidence +Maintainer, 2026-10-13: "what does not work is that we dismiss things just because semgrep cannot find the bug. +if semgrep was enough we'd use it." A Semgrep (or quick-rule, or engine) match is positive evidence and may enter +the table as a feature ("matched here, rule X"). The absence of a match is not recorded as a feature, is never a +filter, a demotion, a down-weight or a gate on a hunter suspicion, and no acceptance rule may count it. The +Semgrep arm on the pilot is a comparison arm only. + ## Components and interfaces | Component | Intent | Requirements | Depends on | Contract | |---|---|---|---|---|