Marius CiepluchandClaude Opus 4.8 43ef254134 feat(benchmark): valid detection benchmark spec + offline modules (groups 1-4) ...

Builds the benchmark from .kiro/specs/valid-detection-benchmark/brief.md:
sound labels from the fix (no SZZ), app-shaped verified positives,
ordinary negatives from the same repos, whole tool on unseen repos,
standard metrics.

Spec: requirements.md, design.md, tasks.md. Reuses the existing sourcing,
fix-site labelling, negative drawing, replay, and Wilson scoring; drops
the SZZ layer.

New offline modules under benchmarks/valid/:
- catalog.py  positives labelled at the fix site (never introducing/last_touch)
- verify.py   one LLM call per case (vulnerable? fix addresses it? reachable
              in-repo?) + human spot-check; this is the app-shaped filter.
              Honors the instrument rule: bytes read per function, reject on
              an empty read.
- assemble.py >=20x ordinary negatives + a frozen, deterministic manifest with
              the held-out guard
- metrics.py  precision (PPV), false alarms per 100 pushes, exploit-verified
              rate, and the plain headline, layered on score.py

36 new offline tests, 495 regression tests green. No network, no harvest,
no model calls in tests. Group 5 (the live pilot) is not started.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

2026-10-12 18:55:42 +02:00

..

ai-service-plane

docs: plain-English pass over the active specs

2026-10-12 18:16:41 +02:00

authorization-obligations

docs: plain-English pass over the active specs

2026-10-12 18:16:41 +02:00

constrained-detector

docs: plain-English pass over the active specs

2026-10-12 18:16:41 +02:00

contributor-scan

docs: plain-English pass over the active specs

2026-10-12 18:16:41 +02:00

corpus-seeded-mechanisms

docs: plain-English pass over the active specs

2026-10-12 18:16:41 +02:00

engine-change-scoped-analysis

docs: plain-English pass over the active specs

2026-10-12 18:16:41 +02:00

finding-feedback-loop

docs: plain-English pass over the active specs

2026-10-12 18:16:41 +02:00

flow-aware-ranking

docs: plain-English pass over the active specs

2026-10-12 18:16:41 +02:00

harnessx-removal

refactor(memory): S3-generic store on boto3; RustFS is the tested server (no MinIO naming)

2026-10-02 10:15:19 +02:00

harnessx-self-improving-rulesets

docs(kiro): archive the superseded roadmap; status notes on specs

2026-10-02 08:19:48 +02:00

ir-consolidation

docs: plain-English pass over the active specs

2026-10-12 18:16:41 +02:00

learned-decision-engine

docs: plain-English pass over the active specs

2026-10-12 18:16:41 +02:00

learning-harness

docs: plain-English pass over the active specs

2026-10-12 18:16:41 +02:00

mcp-shift-left-agent

docs: plain-English pass over the active specs

2026-10-12 18:16:41 +02:00

model-grounded-detection

docs: plain-English pass over the active specs

2026-10-12 18:16:41 +02:00

openrouter-sast-harness

docs(kiro): archive the superseded roadmap; status notes on specs

2026-10-02 08:19:48 +02:00

pair-corpus-honesty

docs: plain-English pass over the active specs

2026-10-12 18:16:41 +02:00

plane-on-kubernetes

docs: plain-English pass over the active specs

2026-10-12 18:16:41 +02:00

pre-push-safety-net

docs: plain-English pass over the active specs

2026-10-12 18:16:41 +02:00

proof-arm-opencode

docs: plain-English pass over the active specs

2026-10-12 18:16:41 +02:00

propose-adjudicate-prove

docs: plain-English pass over the active specs

2026-10-12 18:16:41 +02:00

reachability-flow-model

docs: plain-English pass over the active specs

2026-10-12 18:16:41 +02:00

real-world-vfc-slice

docs: plain-English pass over the active specs

2026-10-12 18:16:41 +02:00

search-with-proof

docs: plain-English pass over the active specs

2026-10-12 18:16:41 +02:00

three-stage-scan

docs: plain-English pass over the active specs

2026-10-12 18:16:41 +02:00

tree-sitter-overlay-extra

docs: plain-English pass over the active specs

2026-10-12 18:16:41 +02:00

unseen-repo-evaluation

push(snapshot): run-aware base/head line-correspondence (stop discarding intra-run anchors)

2026-10-11 10:12:03 +02:00

valid-detection-benchmark

feat(benchmark): valid detection benchmark spec + offline modules (groups 1-4)

2026-10-12 18:55:42 +02:00

zero-with-a-reason

docs: plain-English pass over the active specs

2026-10-12 18:16:41 +02:00