From e716e3beab8b01f63ac59dd49586b5904231e2a4 Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Sun, 11 Oct 2026 15:05:51 +0200 Subject: [PATCH] docs(g2): record-07 baseline is regex-only; prioritize semantic detection + barrier analysis Verified on the records: the push replay arms hardcode model="disabled", max_model_calls=0, cost_usd=0 (NO LLM) and engine_covered=0 (engine never ran, ~200s wall). So the measured catch~2.5% is the quick pattern-rule tier ALONE -- grep-with-structure. Pattern rules alone will never be enough. Reprioritize G2 after record 07 to an efficiency barrier analysis over the two tiers that carry detection and both sat out: - engine (Joern dataflow): E1 split the ~150s evidence-fixed (JVM/graph-load/ census); E2 does a warm session + graph preload at build kill it (decisive); E3 analysis per-pair; E4 build vs repo size. - LLM adjudication: L1 model latency + calls per push; L2 recall lift over regex+dataflow on the pool. Deliverable: a barrier table (budget x tier combo -> fits? measured recall/FP) for a realistic pre-push window (<=120s target, 300s hard) -> minimal releasable config. Change-scoping branch is a late add-on (trims only the per-pair tail), not the lever. Also fixed a stray tag in the brief. Co-Authored-By: Claude Opus 4.8 --- .../engine-change-scoped-analysis/brief.md | 37 ++++++++++++++++++- .kiro/steering/roadmap.md | 28 ++++++++------ 2 files changed, 53 insertions(+), 12 deletions(-) diff --git a/.kiro/specs/engine-change-scoped-analysis/brief.md b/.kiro/specs/engine-change-scoped-analysis/brief.md index bff1d59..ef17233 100644 --- a/.kiro/specs/engine-change-scoped-analysis/brief.md +++ b/.kiro/specs/engine-change-scoped-analysis/brief.md @@ -137,4 +137,39 @@ detection is being lost, not in Joern's capability. here is findings moving from `comparison_unknown`/deferred to a decided novelty → advisory engine finding. - Implement directly and verify on the LIVE instrument (a fake-client/unit pass is not evidence the live engine changed); the kube-lane and store endpoints are operator-owned infra, not assumptions to make here. - + +## Efficiency barrier analysis (post-record-07 priority) + +**What the record-07 baseline actually measured (verified on the records): regex only.** The default and +long_deadline replay arms both hardcode `model="disabled"`, `max_model_calls=0`, `cost_usd=0` — **no LLM is +called** — and the engine never completed (`engine_covered=0`, the ~200 s wall). So the measured `catch≈2.5%` is +the **quick pattern-rule tier alone**: essentially grep-with-structure. This is the FLOOR, not the product, and it +confirms the maintainer's point — pattern rules alone will never be enough, or `grep` would do. The product's +detection lives in two tiers that did NOT run: + +| Tier | Mechanism | Measured cost (uncontended, mid-size repo) | Ran? | Catch | +|---|---|---|---|---| +| 1 quick rules | regex/pattern over changed lines | ~seconds | yes | 2.5% (floor) | +| 2 engine (Joern) | CPG build + taint/dataflow | build 48 s + evidence (150 s fixed + 93 ms/pair) + analysis (unmeasured) | no (walled) | unknown | +| 3 LLM adjudication | model judgment / search-with-proof (hunter, fusion, decision engine) | model latency + $/call (unmeasured in the push path) | no (disabled) | unknown | + +**The barrier:** a releasable net needs tier 2 and/or tier 3 to run and catch within a realistic pre-push window — +target **≤60–120 s** (developer patience), hard ceiling 300 s. The analysis is a budget ledger per tier against +that window, and the measurements each needs: + +- **Engine barrier (tier 2).** Fixed floor `build(48) + evidence_fixed(~150) ≈ 198 s` precedes any dataflow answer. + Measure (clean host): **E1** split the ~150 s into JVM-start / graph-load / census; **E2** does a warm session + with the graph **preloaded at build** eliminate `evidence_fixed` on the first query (the decisive architectural + test)?; **E3** analysis (`reachableByFlows`) per-pair cost + warm reuse; **E4** build cost vs repo size. + Levers in priority order: warm session + graph preload → cached census → change-scope the per-pair tail. +- **LLM barrier (tier 3).** Unmeasured in the push path. Measure: **L1** model round-trip latency per call + (DeepSeek) and calls needed per push for the decision/proof stage; **L2** the recall LIFT of tier 3 over tiers + 1+2 on the pool — does the LLM catch what regex and dataflow miss, and at what $ and latency per push? +- **Deliverable:** a **barrier table** — for budget ∈ {60, 120, 300, 600} s × each tier combination, does it FIT + and what is the MEASURED recall/false-alarm on the pool — identifying the minimal viable config for a releasable + net. 600 s unblocks the benchmark arm but is not a pre-push deadline; the product target is tiers 2+3 under ~120 s. + +Sequence: finish G0 (record 07) → **E1/E2 first** (they decide whether the warm session alone crosses the engine +barrier) → E3/E4 → L1/L2. Each on a clean host (the session times out contended). The change-scoped branch +(`engine-change-scoped-evidence-pass`, unmerged) only trims the tier-2 per-pair tail and is a late add-on, not the +lever. diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index 6efed37..b46b729 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -97,18 +97,24 @@ Candidates, each admitted only if it improves the pool metric on a fresh slice: (approved plan, now one arm, not the spine), engine coverage and TypeScript, decision-engine prompt and retrieval changes, more labelled pairs from more distinct repositories. -Active G2 workstreams (2026-10-11, opened while driving G0; progress tracked in their briefs): +Active G2 workstreams (2026-10-11; progress tracked in `.kiro/specs/engine-change-scoped-analysis/brief.md`): -- **Change-scoped engine analysis** (`.kiro/specs/engine-change-scoped-analysis/brief.md`). Live diagnosis on the - frozen pool showed the engine's "no finding" on unseen changes is usually *analysis-never-runs*: the engine ranks - repo-wide candidates and `deferred: deadline_exhausted` before analyzing anything (even at 300 s), and the ranked - sites are unrelated to the pushed change. Lever: seed analysis from the change's declarations + dataflow - neighbourhood (`rank.py`/`mapping.py`) so it runs within budget. Increment 0 (confirm CPG-build-vs-analysis cost - split) gates the rest. Scale measurement needs the kube lane. -- **Line-correspondence (done, necessary-not-sufficient).** `snapshot.py` discarded base↔head anchors on - non-globally-unique lines → `comparison_unknown` → suppressed findings; fixed to run-aware anchoring - (commit c39fbb8, 163 tests). Removes one cause; coverage still needs change-scoped analysis + the capability - registry (Task 8.3). +**#1 priority after record 07: get real (semantic) detection under the pre-push budget.** The record-07 baseline +is **regex only** — verified: the push arms hardcode `model="disabled"`, `max_model_calls=0`, and the engine never +ran (`engine_covered=0`); measured `catch≈2.5%`. Pattern rules alone will never be enough (else `grep` would do). +The two tiers that carry detection both sat out: the **engine** (Joern dataflow, blocked by ~200 s fixed overhead) +and the **LLM adjudication** (disabled in the push path). The workstream is an **efficiency barrier analysis** — a +per-tier budget ledger against a realistic pre-push window (≤120 s target, 300 s hard): + - Engine: E1 split the ~150 s evidence-fixed (JVM/graph-load/census); E2 does a warm session with the graph + preloaded at build kill it (decisive); E3 analysis per-pair cost; E4 build vs repo size. Levers: warm session + + preload → cached census → change-scope the per-pair tail. + - LLM: L1 model latency and calls per push; L2 the recall LIFT of the LLM tier over regex+dataflow on the pool. + - Deliverable: a barrier table (budget × tier combo → fits? measured recall/FP) → the minimal releasable config. + Each measured on a **clean host** (the session times out contended). Sequence E1/E2 first. + +- **Change-scoped engine analysis (late add-on, not the lever).** Branch `engine-change-scoped-evidence-pass` + (unmerged): trims only the ~86 s per-pair tier tail; does NOT cross the fixed-cost barrier. Reconsider after E1/E2. +- **Line-correspondence (done, necessary-not-sufficient).** `snapshot.py` run-aware anchoring (commit c39fbb8). **G3. Decide BLOCK per family** on the pool's false-alarm rate and the pooled precision bound (Req 6.4).