From a5a37374a79ba49b37c7b77e39979baf777537a5 Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Mon, 12 Oct 2026 08:53:38 +0200 Subject: [PATCH] measure(g2): engine-on pilot -- regex 1/5, engine 0/5 on clean host; budget hypothesis refuted Five public pool repos, vuln change each, clean host, 300s. The shipped engine added NO catches over the regex floor. 3/5 repos finished their region queue (deferred=0) and still emitted no finding, so change-scoping is not the path to catch. The engine ranked the vuln region in all 5 and found the right sink on 2 (score 2.0), but withholds any finding it cannot anchor 1:1 to changed lines -- all 5 hit ambiguous_line_correspondence -> witness_unresolved -> coverage incomplete, suppressing even the scored sinks. Confirmed not a shallow-clone artifact (correspondence is computed from both blobs). Reprioritised engine levers: #1 change-anchoring robustness (region-level witness fallback), #2 family ranking/taint coverage, #3 the LLM adjudication tier (the pilot is the case for it). Release read: the pre-push net is NOT releasable on detection yet. Co-Authored-By: Claude Opus 4.8 --- .../engine-change-scoped-analysis/brief.md | 45 +++++++++++++++++++ .kiro/steering/roadmap.md | 26 +++++++---- 2 files changed, 63 insertions(+), 8 deletions(-) diff --git a/.kiro/specs/engine-change-scoped-analysis/brief.md b/.kiro/specs/engine-change-scoped-analysis/brief.md index 8822c45..9e13b6f 100644 --- a/.kiro/specs/engine-change-scoped-analysis/brief.md +++ b/.kiro/specs/engine-change-scoped-analysis/brief.md @@ -216,3 +216,48 @@ conflated with host contention from the running baseline replay: CATCH vulns regex misses?* That needs an **engine-on unseen pilot** (a handful of pool repos replayed with the engine arm enabled on a clean host) scored against record 07's regex-only floor (catch 3/92). That is the decisive next experiment, ahead of E3/E4/L1/L2. + +## ENGINE-ON PILOT RESULTS (measured 2026-10-12, CLEAN host, 5 public pool repos, vuln change each, 300 s) + +Five public repos picked for flow-shaped families and small size (engine's best chance to finish and catch): + +| repo | fam | size | REGEX on vuln | ENGINE finding | cov | vuln region ranked | best taint | selected/deferred | wall | +|---|---|---|---|---|---|---|---|---|---| +| masci/banks | path | 2f | ✗ | **none** | incomplete | yes | sink found, no source (score 0) | 281 / **0** | 236 s | +| vitejs/launch-editor | injection | 1f | ✗ | **none** | incomplete | yes | tier1 score2.0 sink (wrong fam) | 13 / 38 | 261 s | +| getzep/graphiti | injection | 5f | ✗ | **none** | incomplete | yes | injection score 0 | 882 / **0** | 304 s | +| aws/bedrock-agentcore | injection | 3f | ✗ | **none** | incomplete | yes | — | 356 / 1009 | 305 s | +| miguelgrinberg/python-socketio | deser | 10f | **✓ (n=2)** | **none** | incomplete | yes | **tier1 score2.0 sink found** | 136 / **0** | 257 s | + +**Result: regex 1/5, engine 0/5.** On a clean host, with budget to spare on most, the shipped engine **added no +catches over the regex floor.** + +**Why (measured, traced — not a budget wall):** +1. **The budget hypothesis is REFUTED.** banks, graphiti and socketio **finished their region queue** (deferred=0) + and still emitted no finding. Change-scoping to fit budget would not have changed the outcome. +2. **The engine ranked the vuln region in every repo** (`vuln_in_analyses=true`) and on 2 of 5 (socketio deser, + launch-editor) the **dataflow found the right sink** (tier=1, score=2.0). So detection partially worked. +3. **The dominant blocker is change-anchoring, not taint.** All 5 hit `ambiguous_line_correspondence` → + `witness_unresolved` → `coverage_status=incomplete`, and the engine **withholds any finding it cannot anchor + one-to-one to the changed lines** — suppressing even the scored sinks in (2). This is `snapshot.py`'s line + correspondence marking the diff ambiguous when the same content line appears in multiple runs (moved/repeated + lines). **Confirmed NOT a shallow-clone replay artifact** — correspondence is computed from both full blobs, which + depth-1 fetches of each SHA provide. The merged line-correspondence fix (c39fbb8) does not resolve these cases. +4. **Family ranking also misses** on some (graphiti injection score 0; banks path found the sink but no source). +5. **Measurement caveat:** the pool's SZZ last-writer change definition yields **large/moved diffs** where exact + change-anchoring is inherently ambiguous (the pool documents this bias). Part of the 0/5 is this corpus property + interacting with the engine's change-anchored design; a cleaner probe uses tight vuln-introducing diffs. + +**Reprioritised engine levers (budget/change-scoping demoted):** +- **#1 change-anchoring robustness.** `ambiguous_line_correspondence`/`witness_unresolved` suppressed findings on + 5/5, including 2 with a scored sink. Make the engine able to report a change-adjacent finding when exact 1:1 + anchoring is impossible (region-level witness, not line-level) — the single highest-leverage fix. +- **#2 family ranking / taint coverage** so the correct family wins and source→sink connects on these shapes. +- **#3 the LLM adjudication tier (disabled in the push path)** — the arm designed to confirm what the dataflow + finds but cannot anchor. The pilot is the empirical case for L1/L2: regex+dataflow caught 1/5 here; the question + is whether the LLM tier lifts it. **This is now the strongest argument for prioritising the LLM arm.** + +**Release read:** the pre-push net is NOT releasable on detection yet. Clean host, generous budget: regex 1/5, +engine 0/5 on flow-shaped unseen vulns. The engine is close (ranks the region, finds sinks) but its change-anchoring +gate and family ranking block the catch. Next: a change-anchoring fix + an LLM-tier pilot (L2 recall lift), not more +budget work. diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index bd3f2aa..7260407 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -112,17 +112,27 @@ against a realistic pre-push window (≤120 s target, 300 s hard): `taint.sc`'s ~160 s per-batch flow work, so `build(~30)+evidence(~225)≈255 s` of 300 s defers 1009/1365 regions. Lever = **change-scope the family/file set** the taint batch walks (NOT session/preload). E3 analysis per-pair cost; E4 build vs repo size remain. - - **DECISIVE next experiment (ahead of E3/E4/L1/L2): engine-on unseen PILOT.** Clean host selects 356 regions — - does it CATCH what regex misses? Replay a handful of pool repos with the engine arm enabled, clean host, score - vs record-07's regex-only floor (catch 3/92). This is the G2 payoff measurement. + - **ENGINE-ON PILOT DONE (2026-10-12, clean host, 5 public repos) — see brief "ENGINE-ON PILOT RESULTS".** + **regex 1/5, engine 0/5.** The engine added NO catches over the regex floor. The **budget hypothesis is + REFUTED**: 3/5 repos finished their queue (deferred=0) and still emitted no finding — so change-scoping is NOT + the path to catch. The engine ranked the vuln region in all 5 and found the right sink on 2 (socketio deser, + launch-editor, score 2.0), but **withholds any finding it cannot anchor 1:1 to changed lines**: all 5 hit + `ambiguous_line_correspondence` → `witness_unresolved` → `coverage_status=incomplete`, suppressing even the + scored sinks. Confirmed NOT a shallow-clone artifact. **Release read: the net is NOT releasable on detection.** + - **Reprioritised engine levers (budget/change-scoping DEMOTED):** #1 **change-anchoring robustness** (report a + region-level / change-adjacent finding when exact 1:1 line anchoring fails — suppressed findings on 5/5); #2 + **family ranking / taint coverage** (correct family must win, source→sink must connect); #3 the **LLM + adjudication tier** — the pilot is the empirical case to prioritise it (L1/L2): does the LLM confirm what the + dataflow finds but cannot anchor? - LLM: L1 model latency and calls per push; L2 the recall LIFT of the LLM tier over regex+dataflow on the pool. - Deliverable: a barrier table (budget × tier combo → fits? measured recall/FP) → the minimal releasable config. -- **Change-scoped engine analysis — RECONSIDERED: relevant after all, on a clean host.** Branch - `engine-change-scoped-evidence-pass` (unmerged). E1/E2 show the barrier is the taint batch's per-batch work, so - scoping the **family/file set** it walks (not merely the pair list) is the lever; replace the adaptive portioning - with a fixed cap (cost is per-batch-fixed, not per-pair). Pair up with the engine-on pilot. -- **Line-correspondence (done, necessary-not-sufficient).** `snapshot.py` run-aware anchoring (commit c39fbb8). +- **Change-scoped engine analysis — DEMOTED again by the pilot.** Branch `engine-change-scoped-evidence-pass` + (unmerged). Fitting the budget does not produce catches (3/5 pilot repos finished the queue with zero findings), + so this is not on the catch critical path. Keep only as a latency optimisation once detection works. +- **Line-correspondence (merged c39fbb8 but INSUFFICIENT).** The pilot shows `ambiguous_line_correspondence` still + fires on 5/5 realistic diffs and is now the **#1 detection blocker** — the engine withholds anchored findings. + Needs a region-level witness fallback when 1:1 line anchoring is impossible. **G3. Decide BLOCK per family** on the pool's false-alarm rate and the pooled precision bound (Req 6.4).