measure(g2): CORRECT pilot diagnosis -- engine produced zero vuln-family findings; anchoring is a red herring

Inspecting head.findings (not ranker evidence) overturns the previous commit's read:
the dataflow/taint tier produced ZERO findings in the vuln families (path/injection/deser)
on all 5 repos -- only config_secrets (banks 1, socketio 2; launch-editor/graphiti/bedrock 0).
The tier1/score2.0 'sink' I cited was the RANKER's static candidate, not a confirmed taint
finding. taint_evidence_not_applicable = no source->sink flow was traced (banks: sink found,
no source). So ambiguous_line_correspondence is downstream and moot -- there are no
vuln-family findings to anchor, and the region-level witness fix (Option A) is refuted before
implementation. The LLM adjudication tier was disabled (model_calls=0).

Reprioritised: #1 LLM-tier pilot L1/L2 (now primary, staged, costs money), #2 taint
source/sink coverage + region ranking, #3 change-anchoring (only after tiers yield findings).
Instrument rule: caught the ranker-vs-finding misread before building the wrong fix.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

.kiro/specs/engine-change-scoped-analysis/brief.md

@@ -232,32 +232,38 @@ Five public repos picked for flow-shaped families and small size (engine's best
 **Result: regex 1/5, engine 0/5.** On a clean host, with budget to spare on most, the shipped engine **added no
 catches over the regex floor.**
 
-**Why (measured, traced — not a budget wall):**
+**Why (measured, traced — CORRECTED after inspecting `head.findings`, not ranker evidence):**
 1. **The budget hypothesis is REFUTED.** banks, graphiti and socketio **finished their region queue** (deferred=0)
    and still emitted no finding. Change-scoping to fit budget would not have changed the outcome.
-2. **The engine ranked the vuln region in every repo** (`vuln_in_analyses=true`) and on 2 of 5 (socketio deser,
+2. **The engine produced ZERO vuln-family findings (path/injection/deser) on all 5.** Actual `head.findings`:
-   launch-editor) the **dataflow found the right sink** (tier=1, score=2.0). So detection partially worked.
+   banks 1 (config_secrets), socketio 2 (config_secrets), launch-editor / graphiti / bedrock **0 findings each**.
-3. **The dominant blocker is change-anchoring, not taint.** All 5 hit `ambiguous_line_correspondence` →
+   The dataflow/taint tier traced **no source→sink flow** for the vuln families (`taint_evidence_not_applicable`;
-   `witness_unresolved` → `coverage_status=incomplete`, and the engine **withholds any finding it cannot anchor
+   e.g. banks found the `path.open` sink but **no source reaches it**). **Correction:** the tier=1 / score=2.0
-   one-to-one to the changed lines** — suppressing even the scored sinks in (2). This is `snapshot.py`'s line
+   "sink" seen in the ranker's `normalized_evidence` is the RANKER's static sink candidate, NOT a confirmed taint
-   correspondence marking the diff ambiguous when the same content line appears in multiple runs (moved/repeated
+   finding — an earlier draft of this section misread it as detection partially working. It was not.
-   lines). **Confirmed NOT a shallow-clone replay artifact** — correspondence is computed from both full blobs, which
+3. **`ambiguous_line_correspondence` is a downstream RED HERRING for these vulns.** It fires on all 5, but there
-   depth-1 fetches of each SHA provide. The merged line-correspondence fix (c39fbb8) does not resolve these cases.
+   are **no vuln-family findings to anchor**, so a region-level witness fallback (the proposed "Option A") would
-4. **Family ranking also misses** on some (graphiti injection score 0; banks path found the sink but no source).
+   change nothing here. Witness/anchoring only touched the config_secrets findings that did appear.
-5. **Measurement caveat:** the pool's SZZ last-writer change definition yields **large/moved diffs** where exact
+4. **The LLM adjudication tier was DISABLED** (`model_calls=0`, `cost_usd=0`, `model_assistance={}` on all 5). The
-   change-anchoring is inherently ambiguous (the pool documents this bias). Part of the 0/5 is this corpus property
+   rungs reached: `suspicion=0, model_corroborated=0, model_entailed=0–2, execution_confirmed=0`. The tier designed
-   interacting with the engine's change-anchored design; a cleaner probe uses tight vuln-introducing diffs.
+   to confirm a vuln the static dataflow cannot trace **never ran**.
+5. **Ranking also under-promotes the vuln region** (tier 0 for most families; banks path family stuck low). So even
+   where a sink exists, the region is not analysed deeply enough to attempt a flow.
+6. **Measurement caveat:** the pool's SZZ last-writer change definition yields large/moved diffs; a cleaner probe
+   uses tight vuln-introducing diffs. Secondary to (2)/(4).
 
-**Reprioritised engine levers (budget/change-scoping demoted):**
+**Reprioritised engine levers (CORRECTED — anchoring is NOT the lever):**
-- **#1 change-anchoring robustness.** `ambiguous_line_correspondence`/`witness_unresolved` suppressed findings on
+- **#1 the LLM adjudication tier (disabled in the push path) — now the primary lever.** regex+dataflow = 1/5 with
-  5/5, including 2 with a scored sink. Make the engine able to report a change-adjacent finding when exact 1:1
+  the model OFF; the static tiers produce no vuln-family findings on these shapes. The product thesis
-  anchoring is impossible (region-level witness, not line-level) — the single highest-leverage fix.
+  ([[search-with-proof]]) is that the LLM tier catches what regex+dataflow miss. The pilot is the empirical case to
-- **#2 family ranking / taint coverage** so the correct family wins and source→sink connects on these shapes.
+  turn it on: **L1** model latency + calls/push, **L2** recall LIFT on the same 5 (and beyond). Costs money — stage
-- **#3 the LLM adjudication tier (disabled in the push path)** — the arm designed to confirm what the dataflow
+  it (5–10 sizing → ~10–20 pilot → full only on lift).
-  finds but cannot anchor. The pilot is the empirical case for L1/L2: regex+dataflow caught 1/5 here; the question
+- **#2 taint source/sink coverage + region ranking** so the dataflow recognises these vuln sources and promotes the
-  is whether the LLM tier lifts it. **This is now the strongest argument for prioritising the LLM arm.**
+  vuln region (banks path: sink found, source unrecognised; graphiti injection: score 0).
+- **#3 change-anchoring robustness** — real, but only matters AFTER the static/LLM tiers produce vuln-family
+  findings to anchor. Deprioritised until (1)/(2) yield findings.
 
-**Release read:** the pre-push net is NOT releasable on detection yet. Clean host, generous budget: regex 1/5,
+**Release read:** the pre-push net is NOT releasable on detection. Clean host, generous budget, LLM off: regex 1/5,
-engine 0/5 on flow-shaped unseen vulns. The engine is close (ranks the region, finds sinks) but its change-anchoring
+engine 0/5. The static tiers (regex + Joern dataflow) do not detect these unseen vulns; the decisive untested lever
-gate and family ranking block the catch. Next: a change-anchoring fix + an LLM-tier pilot (L2 recall lift), not more
+is the **LLM adjudication tier**. Next: an **LLM-tier pilot (L1/L2)** on these 5 repos — does enabling the model
-budget work.
+produce vuln-family findings regex+dataflow miss, and at what $/latency.

.kiro/steering/roadmap.md

@@ -119,20 +119,22 @@ against a realistic pre-push window (≤120 s target, 300 s hard):
     launch-editor, score 2.0), but **withholds any finding it cannot anchor 1:1 to changed lines**: all 5 hit
     `ambiguous_line_correspondence` → `witness_unresolved` → `coverage_status=incomplete`, suppressing even the
     scored sinks. Confirmed NOT a shallow-clone artifact. **Release read: the net is NOT releasable on detection.**
-  - **Reprioritised engine levers (budget/change-scoping DEMOTED):** #1 **change-anchoring robustness** (report a
+  - **CORRECTED after inspecting `head.findings`:** the engine produced **ZERO vuln-family findings** (path/
-    region-level / change-adjacent finding when exact 1:1 line anchoring fails — suppressed findings on 5/5); #2
+    injection/deser) on all 5 — only config_secrets. The tier1/score2.0 "sink" was the RANKER's static candidate,
-    **family ranking / taint coverage** (correct family must win, source→sink must connect); #3 the **LLM
+    not a taint finding. `ambiguous_line_correspondence` is a downstream red herring (nothing to anchor), so the
-    adjudication tier** — the pilot is the empirical case to prioritise it (L1/L2): does the LLM confirm what the
+    region-level witness fix (old "Option A") was **refuted before implementation**. The **LLM tier was disabled**
-    dataflow finds but cannot anchor?
+    (model_calls=0). **Reprioritised levers:** #1 the **LLM adjudication tier** (disabled) — now primary: run the
+    LLM pilot L1/L2, staged (costs money). #2 taint source/sink coverage + region ranking. #3 change-anchoring
+    (only matters once tiers produce vuln-family findings — deprioritised).
   - LLM: L1 model latency and calls per push; L2 the recall LIFT of the LLM tier over regex+dataflow on the pool.
   - Deliverable: a barrier table (budget × tier combo → fits? measured recall/FP) → the minimal releasable config.
 
 - **Change-scoped engine analysis — DEMOTED again by the pilot.** Branch `engine-change-scoped-evidence-pass`
   (unmerged). Fitting the budget does not produce catches (3/5 pilot repos finished the queue with zero findings),
   so this is not on the catch critical path. Keep only as a latency optimisation once detection works.
-- **Line-correspondence (merged c39fbb8 but INSUFFICIENT).** The pilot shows `ambiguous_line_correspondence` still
+- **Line-correspondence (merged c39fbb8).** `ambiguous_line_correspondence` fires on 5/5 realistic diffs, but the
-  fires on 5/5 realistic diffs and is now the **#1 detection blocker** — the engine withholds anchored findings.
+  pilot's corrected diagnosis shows it is NOT the detection blocker here — the static tiers produce no vuln-family
-  Needs a region-level witness fallback when 1:1 line anchoring is impossible.
+  findings to anchor in the first place. A region-level witness fallback only matters once #1/#2 yield findings.
 
 **G3. Decide BLOCK per family** on the pool's false-alarm rate and the pooled precision bound (Req 6.4).