From 4c4b7d375a619e71e24465bb984c3b990d307ece Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Mon, 12 Oct 2026 08:20:16 +0200 Subject: [PATCH] measure(g2): E1/E2 clean-host results -- the engine 'query=0 wall' was contention, not intrinsic On a clean host the shipped engine runs dataflow (query 0->35s, selected 0->356 regions). The evidence/taint batch (~225s) is invariant to warm session and graph preload; importCpg is cheap (~35s). The real barrier is taint.sc's ~160s per-batch flow work, so build+evidence ~255s of a 300s budget defers 1009/1365 regions. Lever = change-scope the family/file set the taint batch walks, not session/preload. Corrects the prior ~200s graph-load wall diagnosis. Next decisive experiment: engine-on unseen pilot vs record-07 regex floor (catch 3/92). Co-Authored-By: Claude Opus 4.8 --- .../engine-change-scoped-analysis/brief.md | 43 +++++++++++++++++++ .kiro/steering/roadmap.md | 26 +++++++---- 2 files changed, 60 insertions(+), 9 deletions(-) diff --git a/.kiro/specs/engine-change-scoped-analysis/brief.md b/.kiro/specs/engine-change-scoped-analysis/brief.md index ef17233..8822c45 100644 --- a/.kiro/specs/engine-change-scoped-analysis/brief.md +++ b/.kiro/specs/engine-change-scoped-analysis/brief.md @@ -173,3 +173,46 @@ Sequence: finish G0 (record 07) → **E1/E2 first** (they decide whether the war barrier) → E3/E4 → L1/L2. Each on a clean host (the session times out contended). The change-scoped branch (`engine-change-scoped-evidence-pass`, unmerged) only trims the tier-2 per-pair tail and is a late add-on, not the lever. + +## E1/E2 RESULTS (measured 2026-10-12, CLEAN host, bedrock-agentcore, 3-file change, 300 s budget, head side) + +Four clean-host replays settle the engine barrier and **correct the earlier "~200 s graph-load wall"**, which was +conflated with host contention from the running baseline replay: + +| Run | build | evidence (taint batch) | dominance | query_s | selected | deferred | +|---|---|---|---|---|---|---| +| contended (prior) | 48–127 | 243.8 | — | **0** | **0** | 1365 | +| disposable (shipped, clean) | 27.9 | 225.1 | 34.8 | 34.8 | **356** | 1009 | +| warm session `SESSION=1` (clean) | 32.9 | 228.6 | 25.9 | 25.9 | **356** | 1009 | +| session + graph preload (clean) | 30.4 | 223.2 | 32.5 | 32.5 | **356** | 1009 | + +**Findings (each traced to a measured cause, not assumed):** + +1. **The engine arm was walled by CONTENTION, not an intrinsic floor.** The prior `query=0 / selected=0 / all + 1365 deferred` was measured while the slice-1 baseline replay contended for the 7 GB host. On a clean host the + **shipped (disposable) engine already runs dataflow**: `query_seconds` 0 → ~35 s, `selected` 0 → **356 regions**, + dominance flow queries complete. This is the first time the engine arm produced analysis on this change. +2. **The evidence/taint batch (~225 s) is INVARIANT to session and to graph preload** (225.1 / 228.6 / 223.2 across + the three clean runs). Warm session and preload-at-build do **not** reduce it. So the dominant cost is intrinsic + to the taint batch, not graph I/O. +3. **Graph-load is cheap — it is NOT the wall.** The `dominance` query is a full cold query *including* + `importCpg`, and it costs only ~35 s. So `importCpg` is tens of seconds, not ~150 s. **This corrects the earlier + note** that attributed the ~150 s fixed to "first query JVM-start + graph-load + census." The ~160 s fixed lives + inside `taint.sc`'s own per-batch flow work (seed/sink memo fill over the distinct `(family, file)` combinations + it walks), paid lazily as the first pairs process — independent of the raw pair count. +4. **The warm session is not the lever.** It starts fine on a clean host (the contended `session_startup_timeout` + was contention-only), but gives ~no benefit here: subsequent queries (dominance) are already cheap disposable, + and the evidence pass is a single batch call, so there is nothing for the session to amortise. +5. **The barrier, restated:** `build(~30) + evidence(~225) ≈ 255 s` of a 300 s budget, so 1009/1365 regions defer + as `deadline_exhausted`; only 356 are analysed. The lever is **reducing the taint batch's work** — change-scope + the pair/family set so the batch fits budget — NOT session, preload, or a bigger JVM. The parked branch + (`engine-change-scoped-evidence-pass`) is therefore **relevant after all, on a clean host**, provided it scopes + the family/file set `taint.sc` walks (not merely the pair list); its adaptive portioning still needs replacing + with a fixed cap (per-call-fixed cost, not per-pair). +6. **Side note:** the `base`-side CPG build failed (`cpg_build_failed`, 0.9 s) in the preload run — a separate + intermittent base-revision build issue; the `head` side built and analysed normally. Track separately. + +**What E1/E2 did NOT answer (the real G2 question):** on a clean host the engine selects 356 regions — *does it +CATCH vulns regex misses?* That needs an **engine-on unseen pilot** (a handful of pool repos replayed with the +engine arm enabled on a clean host) scored against record 07's regex-only floor (catch 3/92). That is the decisive +next experiment, ahead of E3/E4/L1/L2. diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index b46b729..bd3f2aa 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -102,18 +102,26 @@ Active G2 workstreams (2026-10-11; progress tracked in `.kiro/specs/engine-chang **#1 priority after record 07: get real (semantic) detection under the pre-push budget.** The record-07 baseline is **regex only** — verified: the push arms hardcode `model="disabled"`, `max_model_calls=0`, and the engine never ran (`engine_covered=0`); measured `catch≈2.5%`. Pattern rules alone will never be enough (else `grep` would do). -The two tiers that carry detection both sat out: the **engine** (Joern dataflow, blocked by ~200 s fixed overhead) -and the **LLM adjudication** (disabled in the push path). The workstream is an **efficiency barrier analysis** — a -per-tier budget ledger against a realistic pre-push window (≤120 s target, 300 s hard): - - Engine: E1 split the ~150 s evidence-fixed (JVM/graph-load/census); E2 does a warm session with the graph - preloaded at build kill it (decisive); E3 analysis per-pair cost; E4 build vs repo size. Levers: warm session + - preload → cached census → change-scope the per-pair tail. +The two tiers that carry detection both sat out: the **engine** (Joern dataflow) and the **LLM adjudication** +(disabled in the push path). The workstream is an **efficiency barrier analysis** — a per-tier budget ledger +against a realistic pre-push window (≤120 s target, 300 s hard): + - **Engine: E1/E2 DONE (2026-10-12, clean host) — see brief "E1/E2 RESULTS".** CORRECTION: the engine's + `query=0` wall was **host contention** (the baseline replay), not an intrinsic floor. On a clean host the + shipped engine **runs dataflow** (query 0→35 s, selected 0→**356 regions**). The evidence/taint batch (~225 s) + is **invariant to session and preload**; graph-load is cheap (~35 s incl `importCpg`). The real barrier is + `taint.sc`'s ~160 s per-batch flow work, so `build(~30)+evidence(~225)≈255 s` of 300 s defers 1009/1365 + regions. Lever = **change-scope the family/file set** the taint batch walks (NOT session/preload). E3 analysis + per-pair cost; E4 build vs repo size remain. + - **DECISIVE next experiment (ahead of E3/E4/L1/L2): engine-on unseen PILOT.** Clean host selects 356 regions — + does it CATCH what regex misses? Replay a handful of pool repos with the engine arm enabled, clean host, score + vs record-07's regex-only floor (catch 3/92). This is the G2 payoff measurement. - LLM: L1 model latency and calls per push; L2 the recall LIFT of the LLM tier over regex+dataflow on the pool. - Deliverable: a barrier table (budget × tier combo → fits? measured recall/FP) → the minimal releasable config. - Each measured on a **clean host** (the session times out contended). Sequence E1/E2 first. -- **Change-scoped engine analysis (late add-on, not the lever).** Branch `engine-change-scoped-evidence-pass` - (unmerged): trims only the ~86 s per-pair tier tail; does NOT cross the fixed-cost barrier. Reconsider after E1/E2. +- **Change-scoped engine analysis — RECONSIDERED: relevant after all, on a clean host.** Branch + `engine-change-scoped-evidence-pass` (unmerged). E1/E2 show the barrier is the taint batch's per-batch work, so + scoping the **family/file set** it walks (not merely the pair list) is the lever; replace the adaptive portioning + with a fixed cap (cost is per-batch-fixed, not per-pair). Pair up with the engine-on pilot. - **Line-correspondence (done, necessary-not-sufficient).** `snapshot.py` run-aware anchoring (commit c39fbb8). **G3. Decide BLOCK per family** on the pool's false-alarm rate and the pooled precision bound (Req 6.4).